2022-06-29T15:49:12.983 INFO:root:teuthology version: 1.1.1.dev379+gf58709e3 2022-06-29T15:49:12.985 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/filestore-xfs overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests} --verbose --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2022-06-29_13:31:31-fs-wip-yuri5-testing-2022-06-27-0923-octopus-distro-default-smithi/6905536 --name yuriw-2022-06-29_13:31:31-fs-wip-yuri5-testing-2022-06-27-0923-octopus-distro-default-smithi -- /home/teuthworker/archive/yuriw-2022-06-29_13:31:31-fs-wip-yuri5-testing-2022-06-27-0923-octopus-distro-default-smithi/6905536/orig.config.yaml 2022-06-29T15:49:13.038 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-06-29T15:49:13.126 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2022-06-29_13:31:31-fs-wip-yuri5-testing-2022-06-27-0923-octopus-distro-default-smithi/6905536 branch: wip-yuri5-testing-2022-06-27-0923-octopus description: fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/filestore-xfs overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests} email: null first_in_suite: false job_id: '6905536' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2022-06-29_13:31:31-fs-wip-yuri5-testing-2022-06-27-0923-octopus-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 30 - machine: disk: 200 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri5-testing-2022-06-27-0923-octopus ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 15m rados osd op timeout: 15m mds: debug mds: 20 debug ms: 1 mds bal frag: true mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 15m rados osd op timeout: 15m mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: debug ms: 1 debug osd: 20 osd objectstore: filestore osd op complaint time: 180 osd sloppy crc: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - evicting unresponsive client - POOL_APP_NOT_ENABLED - has not responded to cap revoke by MDS for over - MDS_CLIENT_LATE_RELEASE - responding to mclientcaps sha1: f25ffac72e5a9e63860ff7deb6767a11f0559a16 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: osd objectstore: filestore osd sloppy crc: true filestore: true fs: xfs install: ceph: flavor: default sha1: f25ffac72e5a9e63860ff7deb6767a11f0559a16 workunit: branch: wip-yuri5-testing-2022-06-27-0923-octopus sha1: f25ffac72e5a9e63860ff7deb6767a11f0559a16 owner: scheduled_yuriw@teuthology priority: 79 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - osd.0 - osd.1 - osd.2 - osd.3 - - mon.b - mon.c - mgr.x - mds.b - osd.4 - osd.5 - osd.6 - osd.7 - - client.0 - - client.1 - - client.2 sha1: f25ffac72e5a9e63860ff7deb6767a11f0559a16 sleep_before_teardown: 0 suite: fs suite_branch: wip-yuri5-testing-2022-06-27-0923-octopus suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_f25ffac72e5a9e63860ff7deb6767a11f0559a16/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: f25ffac72e5a9e63860ff7deb6767a11f0559a16 targets: smithi060.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9bKV0cYlD5Ztq2Vrgcz9CTMAg5n3wz65YAQxp3+E5raqdTUpiioI+STpkSTjuKoCaNUJR8i4SSIWR7L3GbEXTCLhxPbzfRUE1FyjkrEdpE+q4QaTR2DAmjA8YuVW6HpwNYZ2xXMQ5feo5nYpppuDlWhvbkxlO/z2H4j+0Q8CecsPYZJurwTR+I58g5HXne3j7+j/ILdrLECVcpR66zu9barMyggbuzayVtkm2/9Cr8Uf8uiDQ8Yr/GYf4O6m6xujMTyz1RMD6iBt+HRY3ehTbO/8Ye2Tgc8aD9inQYmkyM6WknqZXiGiGrQwmU3Oaf+vNKlZPnxqSsVBs4FOnAB8HAa3CQtFys1ewnS/vz1WwmKObuUscA0KAIjKra5iKWHyapxNchNp2w1Ld/wlxs3jIQXtc4B4avlUnmbu/zgaXzjHO9DXnJLD7FVodQq7AzXRmwZ7/nyaaTQRm+jT0dQN1pBlGkwc1meMnz+sHov5tSQdqazcHxHGW8a+PRzU0G2U= smithi092.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3Ibg4PZeSlmo48ptCaxvpuBOAw0+Af5PwJS6gDagEPp5oADGdQ+vmdcZ/1/s2fGUxbW2I+s8qXaiJ0H+G79+QfPiP5yp0rzYg8skRREeszYGq359WyVPtvyMr43r4hI2NA219Gm20VP+yq5xz7p0jjmmIpCJp8V4m+yn+b6b3KQrx7o1mvZBtWDzffYMqfsrk3hgsoNqmvfork2Mdjgoopbf/9ogZTQAyN/P2TXytXhynyEqcHmq2YtFK8/tLCk630TiBwMWFuYpIPnjfI92m9R5FMknh2RF5EUciaXpuy1XRLdBcUbh0v6ctUNIcqXGRfptvNgpXMMgllS1uH2AU11/OZJdN0gT1iIvyABEyIhlfhynroU/HGdOWYARx+BVGQn1+3mS0LEhDY0dO2i4PIfgU8gLAa04ZydV5jKAJ8hwdqdUkLEnEbYwItfKU2jZWvBHSN59OSfbyk8Y10RXYRSPo8YERSdwNTsnnXO10J9QghHtKrJrMTvcE60yHS28= smithi121.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDI9oI+7ZKro2egyqE0uCd7A/BzWqCqe3EGnbfaOgeXmZIvLHOwA1KBUfekiZeJaw8Zz1H2dh3SYM+EVw2GcDNg6pwwPvzT0DGpdbIA1RilHWRLdl/fBlsvB8EvaMwWUPRyrdzGJQz7DvX2M6NXDO1+0eyWqFoUDZjTLHcfxeP5cdJ1SNzZmU7ipDpokBHzGG6lhaX/O1z8+vqNAbHvHpP7f5sm64zkzyxkPKIr+69BopGh9EodlAws8Gl1GAYYHbpat8QzqhU0G1Yp0nEYhaaVGSpzde1ugcYzd3lI+dHU5jo14DcG4BZQnGCoirss8TQmoE42/zMM4CBrDxesjWJDdsBnWQUGv0XpNA6+qJMEL4kfwUlcWDInrDO/o6ZPA3KHDuDIQyMbm4coUknqVC+qMZSFFeNiRdKQv9d6QdqrdYimUHcx55+Xav4mCaU9oHIWPJriCPODmTgNE/h/4mVK/qdt362K2fCW1FmOVmKUM75b/24Lxx8iwH5/qpVKKLc= smithi156.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOu7iEksoPH1SegwSXhYdmM3MEuif4oXSs5xGa0ysSWEQeHSq09GGqlm411tAkA1N4eQaC6egKXAY0TEjqAEosEqL0ucIR2p7NqJxKR8fojaCmXjGSoQx9u3r1vFq+xUqTX2SGlcxc6mbMLd+IJv9OySpOv93hCGkRV+pifIUekV3HKu8QsC5mwKwrWYXaY5SciMXYNKazErxsL39YB2lmIE/yOOkPXVYqXi4f1sVswl2X+d+jRuwplRhNUhzcO2WTGMj4E0RBfTyAifrZxBEWNVvilXiyr2WsXLL4NR26cVU6aToSSWCFRfzogmKJEaqAd4d+ind2tD5azv9GxGs6uLy5W5NHkmm12XY/nb0yf7rxNpgqfHxKtiBykDjRuFZQ6BVY7BlDZR2yc6q/6A55AWJVFLAkIJC4J1wGIqgCc+UkcnQQIOMJlOfO3SvhuyuyVeM7+rfoesyq5D4wcxN9B/WdYNyhT+BiGZO6diljZ9XlWnL4EZOCNB+MpQ1pgEM= smithi197.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCUG7FwrrcHhr8+opukEkrYkAmxYaY4/q5LZbQ0ovMPzIsy8yPjC636BiRSzKJQZhDE9ykoupIMQQGXGVmuyJHZ2vJKlXFig3uMO7Nc4csD2qHreEU0UYQERzPe2vDQw5Fe8h4U+VzEtw9UM+CBUfg1iVV9TNAs4v8/yYCTj7MVSEJuMzzgyRsfWuVQ1zIGjnThph+okCi9EPWeVQ/Mh8vzkWgtBvCEj2k/9o0ruvKS9eKZzyKCxe0G0O8sCezp/BQDDLUPxwf6tV+2xYqNJELNsMBPAmuCkhT5wxaRPAuJ+Y9LeP0KsmONfis6vHw2gXY523PeH3lUS5Lw9d5PLrrjM2eGsHDPhCiSBxR4sSkq5ip03u5vXw57BnlcrBJX9OaHkGl7RZFz1+gDZZ9jx372W7vzRkmeWIvUSIwTOMADk7VZUj8q++4lX8/IUyHtis9tks5+Drk/O/rJ2z/uXv5bs3lhQcnAQfI1B9nnNb7r7SRU/8Wv678ZPWYLVGK3HWk= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell rpm: - python3-cephfs extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: null - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_misc teuthology_branch: main teuthology_sha1: f58709e3fda190343157063e5edb0d7a36a41afc timestamp: 2022-06-29_13:31:31 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1713 2022-06-29T15:49:13.127 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_f25ffac72e5a9e63860ff7deb6767a11f0559a16/qa; will attempt to use it 2022-06-29T15:49:13.128 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_f25ffac72e5a9e63860ff7deb6767a11f0559a16/qa/tasks 2022-06-29T15:49:13.128 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-06-29T15:49:13.130 INFO:teuthology.task.internal:Checking packages... 2022-06-29T15:49:13.152 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'f25ffac72e5a9e63860ff7deb6767a11f0559a16' 2022-06-29T15:49:13.152 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-06-29T15:49:13.153 INFO:teuthology.packaging:ref: None 2022-06-29T15:49:13.153 INFO:teuthology.packaging:tag: None 2022-06-29T15:49:13.153 INFO:teuthology.packaging:branch: wip-yuri5-testing-2022-06-27-0923-octopus 2022-06-29T15:49:13.154 INFO:teuthology.packaging:sha1: f25ffac72e5a9e63860ff7deb6767a11f0559a16 2022-06-29T15:49:13.154 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri5-testing-2022-06-27-0923-octopus 2022-06-29T15:49:13.337 INFO:teuthology.task.internal:Found packages for ceph version 15.2.16-304-gf25ffac7-1focal 2022-06-29T15:49:13.338 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-06-29T15:49:13.345 INFO:teuthology.task.internal:no buildpackages task found 2022-06-29T15:49:13.346 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-06-29T15:49:13.386 INFO:teuthology.task.internal:Saving configuration 2022-06-29T15:49:13.412 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-06-29T15:49:13.419 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-06-29T15:49:13.447 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi060.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-06-29_13:31:31-fs-wip-yuri5-testing-2022-06-27-0923-octopus-distro-default-smithi/6905536', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-06-29 15:43:51.723916', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMWilk793OwOHBET+ulnWwksub05k6w96byimgGonEpS'} 2022-06-29T15:49:13.477 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi092.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-06-29_13:31:31-fs-wip-yuri5-testing-2022-06-27-0923-octopus-distro-default-smithi/6905536', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-06-29 15:43:51.721719', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDuK8bKTTgnBJHFJsrXslffpMI8akRVIdKXtjA/wBEho'} 2022-06-29T15:49:13.504 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-06-29_13:31:31-fs-wip-yuri5-testing-2022-06-27-0923-octopus-distro-default-smithi/6905536', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-06-29 15:43:51.722828', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAhPvjwGU4CiEMTvaPqZdatqj3/d9G//MI9m8a3CU3IF'} 2022-06-29T15:49:13.531 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-06-29_13:31:31-fs-wip-yuri5-testing-2022-06-27-0923-octopus-distro-default-smithi/6905536', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-06-29 15:43:51.724998', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJWBejt9L7P5FjX+RXU3T42BGDr+IrfDT4xTFFC3DyuC'} 2022-06-29T15:49:13.557 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi197.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-06-29_13:31:31-fs-wip-yuri5-testing-2022-06-27-0923-octopus-distro-default-smithi/6905536', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-06-29 15:43:51.719987', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJRliM+7Dpz7CJP7ZQoDnErLoliengqSWwL4AJxmYCPB'} 2022-06-29T15:49:13.557 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-06-29T15:49:13.565 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-06-29T15:49:13.565 INFO:teuthology.task.internal:roles: ubuntu@smithi092.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2022-06-29T15:49:13.566 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['client.0'] 2022-06-29T15:49:13.566 INFO:teuthology.task.internal:roles: ubuntu@smithi156.front.sepia.ceph.com - ['client.1'] 2022-06-29T15:49:13.567 INFO:teuthology.task.internal:roles: ubuntu@smithi197.front.sepia.ceph.com - ['client.2'] 2022-06-29T15:49:13.567 INFO:teuthology.run_tasks:Running task console_log... 2022-06-29T15:49:13.800 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ffbe3e978c8>, signals=[15]) 2022-06-29T15:49:13.801 INFO:teuthology.run_tasks:Running task internal.connect... 2022-06-29T15:49:13.809 INFO:teuthology.task.internal:Opening connections... 2022-06-29T15:49:13.809 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi060.front.sepia.ceph.com 2022-06-29T15:49:13.811 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-06-29T15:49:13.922 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi092.front.sepia.ceph.com 2022-06-29T15:49:13.923 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-06-29T15:49:14.034 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi121.front.sepia.ceph.com 2022-06-29T15:49:14.035 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-06-29T15:49:14.141 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi156.front.sepia.ceph.com 2022-06-29T15:49:14.142 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-06-29T15:49:14.248 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi197.front.sepia.ceph.com 2022-06-29T15:49:14.250 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi197.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-06-29T15:49:14.369 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-06-29T15:49:14.378 DEBUG:teuthology.orchestra.run.smithi060:> uname -m 2022-06-29T15:49:14.386 INFO:teuthology.orchestra.run.smithi060.stdout:x86_64 2022-06-29T15:49:14.386 DEBUG:teuthology.orchestra.run.smithi060:> cat /etc/os-release 2022-06-29T15:49:14.432 INFO:teuthology.orchestra.run.smithi060.stdout:NAME="Ubuntu" 2022-06-29T15:49:14.433 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-06-29T15:49:14.433 INFO:teuthology.orchestra.run.smithi060.stdout:ID=ubuntu 2022-06-29T15:49:14.433 INFO:teuthology.orchestra.run.smithi060.stdout:ID_LIKE=debian 2022-06-29T15:49:14.434 INFO:teuthology.orchestra.run.smithi060.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-06-29T15:49:14.434 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_ID="20.04" 2022-06-29T15:49:14.434 INFO:teuthology.orchestra.run.smithi060.stdout:HOME_URL="https://www.ubuntu.com/" 2022-06-29T15:49:14.434 INFO:teuthology.orchestra.run.smithi060.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-06-29T15:49:14.435 INFO:teuthology.orchestra.run.smithi060.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-06-29T15:49:14.435 INFO:teuthology.orchestra.run.smithi060.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-06-29T15:49:14.435 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_CODENAME=focal 2022-06-29T15:49:14.435 INFO:teuthology.orchestra.run.smithi060.stdout:UBUNTU_CODENAME=focal 2022-06-29T15:49:14.437 INFO:teuthology.lock.ops:Updating smithi060.front.sepia.ceph.com on lock server 2022-06-29T15:49:14.462 DEBUG:teuthology.orchestra.run.smithi092:> uname -m 2022-06-29T15:49:14.469 INFO:teuthology.orchestra.run.smithi092.stdout:x86_64 2022-06-29T15:49:14.470 DEBUG:teuthology.orchestra.run.smithi092:> cat /etc/os-release 2022-06-29T15:49:14.518 INFO:teuthology.orchestra.run.smithi092.stdout:NAME="Ubuntu" 2022-06-29T15:49:14.519 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-06-29T15:49:14.519 INFO:teuthology.orchestra.run.smithi092.stdout:ID=ubuntu 2022-06-29T15:49:14.519 INFO:teuthology.orchestra.run.smithi092.stdout:ID_LIKE=debian 2022-06-29T15:49:14.519 INFO:teuthology.orchestra.run.smithi092.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-06-29T15:49:14.520 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION_ID="20.04" 2022-06-29T15:49:14.520 INFO:teuthology.orchestra.run.smithi092.stdout:HOME_URL="https://www.ubuntu.com/" 2022-06-29T15:49:14.520 INFO:teuthology.orchestra.run.smithi092.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-06-29T15:49:14.520 INFO:teuthology.orchestra.run.smithi092.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-06-29T15:49:14.521 INFO:teuthology.orchestra.run.smithi092.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-06-29T15:49:14.521 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION_CODENAME=focal 2022-06-29T15:49:14.521 INFO:teuthology.orchestra.run.smithi092.stdout:UBUNTU_CODENAME=focal 2022-06-29T15:49:14.521 INFO:teuthology.lock.ops:Updating smithi092.front.sepia.ceph.com on lock server 2022-06-29T15:49:14.543 DEBUG:teuthology.orchestra.run.smithi121:> uname -m 2022-06-29T15:49:14.551 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2022-06-29T15:49:14.551 DEBUG:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2022-06-29T15:49:14.600 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="Ubuntu" 2022-06-29T15:49:14.600 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-06-29T15:49:14.600 INFO:teuthology.orchestra.run.smithi121.stdout:ID=ubuntu 2022-06-29T15:49:14.601 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE=debian 2022-06-29T15:49:14.601 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-06-29T15:49:14.601 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="20.04" 2022-06-29T15:49:14.601 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://www.ubuntu.com/" 2022-06-29T15:49:14.602 INFO:teuthology.orchestra.run.smithi121.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-06-29T15:49:14.602 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-06-29T15:49:14.602 INFO:teuthology.orchestra.run.smithi121.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-06-29T15:49:14.602 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_CODENAME=focal 2022-06-29T15:49:14.602 INFO:teuthology.orchestra.run.smithi121.stdout:UBUNTU_CODENAME=focal 2022-06-29T15:49:14.603 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2022-06-29T15:49:14.625 DEBUG:teuthology.orchestra.run.smithi156:> uname -m 2022-06-29T15:49:14.633 INFO:teuthology.orchestra.run.smithi156.stdout:x86_64 2022-06-29T15:49:14.634 DEBUG:teuthology.orchestra.run.smithi156:> cat /etc/os-release 2022-06-29T15:49:14.683 INFO:teuthology.orchestra.run.smithi156.stdout:NAME="Ubuntu" 2022-06-29T15:49:14.684 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-06-29T15:49:14.684 INFO:teuthology.orchestra.run.smithi156.stdout:ID=ubuntu 2022-06-29T15:49:14.684 INFO:teuthology.orchestra.run.smithi156.stdout:ID_LIKE=debian 2022-06-29T15:49:14.685 INFO:teuthology.orchestra.run.smithi156.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-06-29T15:49:14.685 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION_ID="20.04" 2022-06-29T15:49:14.685 INFO:teuthology.orchestra.run.smithi156.stdout:HOME_URL="https://www.ubuntu.com/" 2022-06-29T15:49:14.685 INFO:teuthology.orchestra.run.smithi156.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-06-29T15:49:14.685 INFO:teuthology.orchestra.run.smithi156.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-06-29T15:49:14.686 INFO:teuthology.orchestra.run.smithi156.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-06-29T15:49:14.686 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION_CODENAME=focal 2022-06-29T15:49:14.686 INFO:teuthology.orchestra.run.smithi156.stdout:UBUNTU_CODENAME=focal 2022-06-29T15:49:14.687 INFO:teuthology.lock.ops:Updating smithi156.front.sepia.ceph.com on lock server 2022-06-29T15:49:14.717 DEBUG:teuthology.orchestra.run.smithi197:> uname -m 2022-06-29T15:49:14.725 INFO:teuthology.orchestra.run.smithi197.stdout:x86_64 2022-06-29T15:49:14.726 DEBUG:teuthology.orchestra.run.smithi197:> cat /etc/os-release 2022-06-29T15:49:14.773 INFO:teuthology.orchestra.run.smithi197.stdout:NAME="Ubuntu" 2022-06-29T15:49:14.773 INFO:teuthology.orchestra.run.smithi197.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-06-29T15:49:14.773 INFO:teuthology.orchestra.run.smithi197.stdout:ID=ubuntu 2022-06-29T15:49:14.774 INFO:teuthology.orchestra.run.smithi197.stdout:ID_LIKE=debian 2022-06-29T15:49:14.774 INFO:teuthology.orchestra.run.smithi197.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-06-29T15:49:14.774 INFO:teuthology.orchestra.run.smithi197.stdout:VERSION_ID="20.04" 2022-06-29T15:49:14.775 INFO:teuthology.orchestra.run.smithi197.stdout:HOME_URL="https://www.ubuntu.com/" 2022-06-29T15:49:14.775 INFO:teuthology.orchestra.run.smithi197.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-06-29T15:49:14.775 INFO:teuthology.orchestra.run.smithi197.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-06-29T15:49:14.775 INFO:teuthology.orchestra.run.smithi197.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-06-29T15:49:14.775 INFO:teuthology.orchestra.run.smithi197.stdout:VERSION_CODENAME=focal 2022-06-29T15:49:14.776 INFO:teuthology.orchestra.run.smithi197.stdout:UBUNTU_CODENAME=focal 2022-06-29T15:49:14.776 INFO:teuthology.lock.ops:Updating smithi197.front.sepia.ceph.com on lock server 2022-06-29T15:49:14.799 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-06-29T15:49:14.813 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-06-29T15:49:14.855 INFO:teuthology.task.internal:Checking for old test directory... 2022-06-29T15:49:14.856 DEBUG:teuthology.orchestra.run.smithi060:> test '!' -e /home/ubuntu/cephtest 2022-06-29T15:49:14.860 DEBUG:teuthology.orchestra.run.smithi092:> test '!' -e /home/ubuntu/cephtest 2022-06-29T15:49:14.864 DEBUG:teuthology.orchestra.run.smithi121:> test '!' -e /home/ubuntu/cephtest 2022-06-29T15:49:14.868 DEBUG:teuthology.orchestra.run.smithi156:> test '!' -e /home/ubuntu/cephtest 2022-06-29T15:49:14.870 DEBUG:teuthology.orchestra.run.smithi197:> test '!' -e /home/ubuntu/cephtest 2022-06-29T15:49:14.877 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-06-29T15:49:14.891 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-06-29T15:49:14.891 DEBUG:teuthology.orchestra.run.smithi060:> test -z $(ls -A /var/lib/ceph) 2022-06-29T15:49:14.895 DEBUG:teuthology.orchestra.run.smithi092:> test -z $(ls -A /var/lib/ceph) 2022-06-29T15:49:14.899 DEBUG:teuthology.orchestra.run.smithi121:> test -z $(ls -A /var/lib/ceph) 2022-06-29T15:49:14.902 DEBUG:teuthology.orchestra.run.smithi156:> test -z $(ls -A /var/lib/ceph) 2022-06-29T15:49:14.924 DEBUG:teuthology.orchestra.run.smithi197:> test -z $(ls -A /var/lib/ceph) 2022-06-29T15:49:14.941 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-06-29T15:49:15.072 INFO:teuthology.run_tasks:Running task kernel... 2022-06-29T15:49:15.087 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-06-29T15:49:15.087 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}, 'client.1': {'kdb': True, 'sha1': 'distro'}, 'client.2': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2022-06-29T15:49:15.088 DEBUG:teuthology.orchestra.run.smithi060:> test -f /run/.containerenv -o -f /.dockerenv 2022-06-29T15:49:15.089 DEBUG:teuthology.orchestra.run.smithi092:> test -f /run/.containerenv -o -f /.dockerenv 2022-06-29T15:49:15.090 DEBUG:teuthology.orchestra.run.smithi121:> test -f /run/.containerenv -o -f /.dockerenv 2022-06-29T15:49:15.091 DEBUG:teuthology.orchestra.run.smithi156:> test -f /run/.containerenv -o -f /.dockerenv 2022-06-29T15:49:15.091 DEBUG:teuthology.orchestra.run.smithi197:> test -f /run/.containerenv -o -f /.dockerenv 2022-06-29T15:49:15.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T15:49:15.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T15:49:15.099 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2022-06-29T15:49:15.099 DEBUG:teuthology.orchestra.run.smithi156:> uname -r 2022-06-29T15:49:15.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T15:49:15.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T15:49:15.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T15:49:15.103 DEBUG:teuthology.orchestra.run.smithi197:> uname -r 2022-06-29T15:49:15.104 DEBUG:teuthology.orchestra.run.smithi060:> uname -r 2022-06-29T15:49:15.104 DEBUG:teuthology.orchestra.run.smithi092:> uname -r 2022-06-29T15:49:15.109 INFO:teuthology.orchestra.run.smithi121.stdout:5.4.0-120-generic 2022-06-29T15:49:15.109 INFO:teuthology.task.kernel:Running kernel on smithi121: 5.4.0-120-generic 2022-06-29T15:49:15.109 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-get clean 2022-06-29T15:49:15.113 INFO:teuthology.orchestra.run.smithi060.stdout:5.4.0-120-generic 2022-06-29T15:49:15.114 INFO:teuthology.orchestra.run.smithi197.stdout:5.4.0-120-generic 2022-06-29T15:49:15.114 INFO:teuthology.orchestra.run.smithi156.stdout:5.4.0-120-generic 2022-06-29T15:49:15.114 INFO:teuthology.task.kernel:Running kernel on smithi060: 5.4.0-120-generic 2022-06-29T15:49:15.115 DEBUG:teuthology.orchestra.run.smithi060:> sudo apt-get clean 2022-06-29T15:49:15.115 INFO:teuthology.task.kernel:Running kernel on smithi197: 5.4.0-120-generic 2022-06-29T15:49:15.115 DEBUG:teuthology.orchestra.run.smithi197:> sudo apt-get clean 2022-06-29T15:49:15.116 INFO:teuthology.task.kernel:Running kernel on smithi156: 5.4.0-120-generic 2022-06-29T15:49:15.116 DEBUG:teuthology.orchestra.run.smithi156:> sudo apt-get clean 2022-06-29T15:49:15.118 INFO:teuthology.orchestra.run.smithi092.stdout:5.4.0-120-generic 2022-06-29T15:49:15.118 INFO:teuthology.task.kernel:Running kernel on smithi092: 5.4.0-120-generic 2022-06-29T15:49:15.118 DEBUG:teuthology.orchestra.run.smithi092:> sudo apt-get clean 2022-06-29T15:49:15.225 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-get update 2022-06-29T15:49:15.230 DEBUG:teuthology.orchestra.run.smithi060:> sudo apt-get update 2022-06-29T15:49:15.239 DEBUG:teuthology.orchestra.run.smithi156:> sudo apt-get update 2022-06-29T15:49:15.254 DEBUG:teuthology.orchestra.run.smithi197:> sudo apt-get update 2022-06-29T15:49:15.266 DEBUG:teuthology.orchestra.run.smithi092:> sudo apt-get update 2022-06-29T15:49:15.412 INFO:teuthology.orchestra.run.smithi197.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-06-29T15:49:15.430 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-06-29T15:49:15.449 INFO:teuthology.orchestra.run.smithi092.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-06-29T15:49:15.479 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-06-29T15:49:15.488 INFO:teuthology.orchestra.run.smithi060.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-06-29T15:49:15.488 INFO:teuthology.orchestra.run.smithi060.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-06-29T15:49:15.495 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-06-29T15:49:15.508 INFO:teuthology.orchestra.run.smithi156.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-06-29T15:49:15.513 INFO:teuthology.orchestra.run.smithi197.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-06-29T15:49:15.520 INFO:teuthology.orchestra.run.smithi092.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-06-29T15:49:15.552 INFO:teuthology.orchestra.run.smithi060.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-06-29T15:49:15.569 INFO:teuthology.orchestra.run.smithi121.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-06-29T15:49:15.579 INFO:teuthology.orchestra.run.smithi156.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-06-29T15:49:15.593 INFO:teuthology.orchestra.run.smithi197.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-06-29T15:49:15.706 INFO:teuthology.orchestra.run.smithi197.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [459 kB] 2022-06-29T15:49:15.810 INFO:teuthology.orchestra.run.smithi197.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,583 kB] 2022-06-29T15:49:15.814 INFO:teuthology.orchestra.run.smithi092.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [685 kB] 2022-06-29T15:49:15.832 INFO:teuthology.orchestra.run.smithi121.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-06-29T15:49:15.833 INFO:teuthology.orchestra.run.smithi156.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-06-29T15:49:15.834 INFO:teuthology.orchestra.run.smithi092.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-06-29T15:49:15.855 INFO:teuthology.orchestra.run.smithi060.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,935 kB] 2022-06-29T15:49:15.898 INFO:teuthology.orchestra.run.smithi197.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [268 kB] 2022-06-29T15:49:15.906 INFO:teuthology.orchestra.run.smithi121.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-06-29T15:49:15.907 INFO:teuthology.orchestra.run.smithi197.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [10.6 kB] 2022-06-29T15:49:15.908 INFO:teuthology.orchestra.run.smithi197.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [25.3 kB] 2022-06-29T15:49:15.908 INFO:teuthology.orchestra.run.smithi197.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1,042 kB] 2022-06-29T15:49:15.914 INFO:teuthology.orchestra.run.smithi156.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-06-29T15:49:15.930 INFO:teuthology.orchestra.run.smithi197.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [148 kB] 2022-06-29T15:49:15.931 INFO:teuthology.orchestra.run.smithi092.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,935 kB] 2022-06-29T15:49:15.932 INFO:teuthology.orchestra.run.smithi197.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [572 B] 2022-06-29T15:49:15.932 INFO:teuthology.orchestra.run.smithi197.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [554 kB] 2022-06-29T15:49:15.936 INFO:teuthology.orchestra.run.smithi197.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [708 kB] 2022-06-29T15:49:15.939 INFO:teuthology.orchestra.run.smithi197.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-06-29T15:49:15.945 INFO:teuthology.orchestra.run.smithi197.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.6 kB] 2022-06-29T15:49:15.956 INFO:teuthology.orchestra.run.smithi060.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [685 kB] 2022-06-29T15:49:15.965 INFO:teuthology.orchestra.run.smithi060.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [350 kB] 2022-06-29T15:49:15.969 INFO:teuthology.orchestra.run.smithi060.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [15.6 kB] 2022-06-29T15:49:15.969 INFO:teuthology.orchestra.run.smithi060.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,119 kB] 2022-06-29T15:49:15.976 INFO:teuthology.orchestra.run.smithi092.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [350 kB] 2022-06-29T15:49:15.980 INFO:teuthology.orchestra.run.smithi092.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [15.6 kB] 2022-06-29T15:49:15.980 INFO:teuthology.orchestra.run.smithi092.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [26.6 kB] 2022-06-29T15:49:15.981 INFO:teuthology.orchestra.run.smithi092.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,119 kB] 2022-06-29T15:49:15.983 INFO:teuthology.orchestra.run.smithi060.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [26.6 kB] 2022-06-29T15:49:15.984 INFO:teuthology.orchestra.run.smithi060.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [159 kB] 2022-06-29T15:49:15.984 INFO:teuthology.orchestra.run.smithi060.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [592 B] 2022-06-29T15:49:15.985 INFO:teuthology.orchestra.run.smithi060.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [677 kB] 2022-06-29T15:49:15.985 INFO:teuthology.orchestra.run.smithi060.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [924 kB] 2022-06-29T15:49:15.991 INFO:teuthology.orchestra.run.smithi060.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [208 kB] 2022-06-29T15:49:15.992 INFO:teuthology.orchestra.run.smithi092.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [159 kB] 2022-06-29T15:49:15.992 INFO:teuthology.orchestra.run.smithi060.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.9 kB] 2022-06-29T15:49:15.993 INFO:teuthology.orchestra.run.smithi092.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [592 B] 2022-06-29T15:49:15.994 INFO:teuthology.orchestra.run.smithi092.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [677 kB] 2022-06-29T15:49:15.999 INFO:teuthology.orchestra.run.smithi092.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [924 kB] 2022-06-29T15:49:16.007 INFO:teuthology.orchestra.run.smithi092.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [208 kB] 2022-06-29T15:49:16.009 INFO:teuthology.orchestra.run.smithi092.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.9 kB] 2022-06-29T15:49:16.009 INFO:teuthology.orchestra.run.smithi060.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [459 kB] 2022-06-29T15:49:16.053 INFO:teuthology.orchestra.run.smithi121.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [685 kB] 2022-06-29T15:49:16.063 INFO:teuthology.orchestra.run.smithi156.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [685 kB] 2022-06-29T15:49:16.075 INFO:teuthology.orchestra.run.smithi060.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,583 kB] 2022-06-29T15:49:16.079 INFO:teuthology.orchestra.run.smithi197.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [685 kB] 2022-06-29T15:49:16.115 INFO:teuthology.orchestra.run.smithi060.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [268 kB] 2022-06-29T15:49:16.119 INFO:teuthology.orchestra.run.smithi060.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [10.6 kB] 2022-06-29T15:49:16.119 INFO:teuthology.orchestra.run.smithi060.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [25.3 kB] 2022-06-29T15:49:16.120 INFO:teuthology.orchestra.run.smithi060.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1,042 kB] 2022-06-29T15:49:16.132 INFO:teuthology.orchestra.run.smithi060.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [148 kB] 2022-06-29T15:49:16.133 INFO:teuthology.orchestra.run.smithi060.stdout:Get:24 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [572 B] 2022-06-29T15:49:16.134 INFO:teuthology.orchestra.run.smithi060.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [554 kB] 2022-06-29T15:49:16.138 INFO:teuthology.orchestra.run.smithi060.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [708 kB] 2022-06-29T15:49:16.144 INFO:teuthology.orchestra.run.smithi060.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.6 kB] 2022-06-29T15:49:16.286 INFO:teuthology.orchestra.run.smithi121.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,935 kB] 2022-06-29T15:49:16.287 INFO:teuthology.orchestra.run.smithi197.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,935 kB] 2022-06-29T15:49:16.322 INFO:teuthology.orchestra.run.smithi092.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,583 kB] 2022-06-29T15:49:16.330 INFO:teuthology.orchestra.run.smithi121.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,583 kB] 2022-06-29T15:49:16.341 INFO:teuthology.orchestra.run.smithi156.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [459 kB] 2022-06-29T15:49:16.380 INFO:teuthology.orchestra.run.smithi156.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,935 kB] 2022-06-29T15:49:16.417 INFO:teuthology.orchestra.run.smithi121.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [350 kB] 2022-06-29T15:49:16.431 INFO:teuthology.orchestra.run.smithi121.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [15.6 kB] 2022-06-29T15:49:16.431 INFO:teuthology.orchestra.run.smithi121.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [26.6 kB] 2022-06-29T15:49:16.432 INFO:teuthology.orchestra.run.smithi121.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,119 kB] 2022-06-29T15:49:16.441 INFO:teuthology.orchestra.run.smithi197.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [350 kB] 2022-06-29T15:49:16.456 INFO:teuthology.orchestra.run.smithi197.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [15.6 kB] 2022-06-29T15:49:16.456 INFO:teuthology.orchestra.run.smithi197.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [26.6 kB] 2022-06-29T15:49:16.457 INFO:teuthology.orchestra.run.smithi197.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,119 kB] 2022-06-29T15:49:16.466 INFO:teuthology.orchestra.run.smithi121.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [159 kB] 2022-06-29T15:49:16.470 INFO:teuthology.orchestra.run.smithi121.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [592 B] 2022-06-29T15:49:16.471 INFO:teuthology.orchestra.run.smithi121.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [677 kB] 2022-06-29T15:49:16.488 INFO:teuthology.orchestra.run.smithi121.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [924 kB] 2022-06-29T15:49:16.491 INFO:teuthology.orchestra.run.smithi197.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [159 kB] 2022-06-29T15:49:16.496 INFO:teuthology.orchestra.run.smithi197.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [592 B] 2022-06-29T15:49:16.496 INFO:teuthology.orchestra.run.smithi197.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [924 kB] 2022-06-29T15:49:16.508 INFO:teuthology.orchestra.run.smithi121.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [208 kB] 2022-06-29T15:49:16.519 INFO:teuthology.orchestra.run.smithi197.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [677 kB] 2022-06-29T15:49:16.522 INFO:teuthology.orchestra.run.smithi121.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.9 kB] 2022-06-29T15:49:16.533 INFO:teuthology.orchestra.run.smithi197.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [208 kB] 2022-06-29T15:49:16.541 INFO:teuthology.orchestra.run.smithi156.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [350 kB] 2022-06-29T15:49:16.547 INFO:teuthology.orchestra.run.smithi197.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.9 kB] 2022-06-29T15:49:16.553 INFO:teuthology.orchestra.run.smithi156.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [15.6 kB] 2022-06-29T15:49:16.554 INFO:teuthology.orchestra.run.smithi156.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [26.6 kB] 2022-06-29T15:49:16.554 INFO:teuthology.orchestra.run.smithi156.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,119 kB] 2022-06-29T15:49:16.594 INFO:teuthology.orchestra.run.smithi156.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [159 kB] 2022-06-29T15:49:16.599 INFO:teuthology.orchestra.run.smithi156.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [592 B] 2022-06-29T15:49:16.600 INFO:teuthology.orchestra.run.smithi156.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [677 kB] 2022-06-29T15:49:16.619 INFO:teuthology.orchestra.run.smithi156.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,583 kB] 2022-06-29T15:49:16.621 INFO:teuthology.orchestra.run.smithi156.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [924 kB] 2022-06-29T15:49:16.642 INFO:teuthology.orchestra.run.smithi156.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [208 kB] 2022-06-29T15:49:16.645 INFO:teuthology.orchestra.run.smithi156.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.9 kB] 2022-06-29T15:49:16.739 INFO:teuthology.orchestra.run.smithi092.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [459 kB] 2022-06-29T15:49:16.746 INFO:teuthology.orchestra.run.smithi121.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [459 kB] 2022-06-29T15:49:16.753 INFO:teuthology.orchestra.run.smithi092.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [268 kB] 2022-06-29T15:49:16.760 INFO:teuthology.orchestra.run.smithi121.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [268 kB] 2022-06-29T15:49:16.763 INFO:teuthology.orchestra.run.smithi092.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [10.6 kB] 2022-06-29T15:49:16.763 INFO:teuthology.orchestra.run.smithi092.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [25.3 kB] 2022-06-29T15:49:16.764 INFO:teuthology.orchestra.run.smithi092.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1,042 kB] 2022-06-29T15:49:16.770 INFO:teuthology.orchestra.run.smithi121.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [10.6 kB] 2022-06-29T15:49:16.771 INFO:teuthology.orchestra.run.smithi121.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [25.3 kB] 2022-06-29T15:49:16.771 INFO:teuthology.orchestra.run.smithi121.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1,042 kB] 2022-06-29T15:49:16.792 INFO:teuthology.orchestra.run.smithi156.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [268 kB] 2022-06-29T15:49:16.805 INFO:teuthology.orchestra.run.smithi156.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [10.6 kB] 2022-06-29T15:49:16.806 INFO:teuthology.orchestra.run.smithi156.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [25.3 kB] 2022-06-29T15:49:16.807 INFO:teuthology.orchestra.run.smithi156.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1,042 kB] 2022-06-29T15:49:16.823 INFO:teuthology.orchestra.run.smithi092.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [148 kB] 2022-06-29T15:49:16.828 INFO:teuthology.orchestra.run.smithi092.stdout:Get:24 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [572 B] 2022-06-29T15:49:16.828 INFO:teuthology.orchestra.run.smithi092.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [554 kB] 2022-06-29T15:49:16.830 INFO:teuthology.orchestra.run.smithi121.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [148 kB] 2022-06-29T15:49:16.835 INFO:teuthology.orchestra.run.smithi121.stdout:Get:24 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [572 B] 2022-06-29T15:49:16.836 INFO:teuthology.orchestra.run.smithi121.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [554 kB] 2022-06-29T15:49:16.846 INFO:teuthology.orchestra.run.smithi092.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [708 kB] 2022-06-29T15:49:16.854 INFO:teuthology.orchestra.run.smithi121.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [708 kB] 2022-06-29T15:49:16.858 INFO:teuthology.orchestra.run.smithi156.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [148 kB] 2022-06-29T15:49:16.864 INFO:teuthology.orchestra.run.smithi156.stdout:Get:24 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [572 B] 2022-06-29T15:49:16.865 INFO:teuthology.orchestra.run.smithi156.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [708 kB] 2022-06-29T15:49:16.870 INFO:teuthology.orchestra.run.smithi092.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.6 kB] 2022-06-29T15:49:16.877 INFO:teuthology.orchestra.run.smithi121.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.6 kB] 2022-06-29T15:49:16.886 INFO:teuthology.orchestra.run.smithi156.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [554 kB] 2022-06-29T15:49:16.899 INFO:teuthology.orchestra.run.smithi156.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.6 kB] 2022-06-29T15:49:20.150 INFO:teuthology.orchestra.run.smithi197.stdout:Fetched 11.3 MB in 2s (6,350 kB/s) 2022-06-29T15:49:20.326 INFO:teuthology.orchestra.run.smithi092.stdout:Fetched 11.3 MB in 2s (6,101 kB/s) 2022-06-29T15:49:20.498 INFO:teuthology.orchestra.run.smithi060.stdout:Fetched 11.3 MB in 2s (6,178 kB/s) 2022-06-29T15:49:20.526 INFO:teuthology.orchestra.run.smithi121.stdout:Fetched 11.3 MB in 2s (4,932 kB/s) 2022-06-29T15:49:20.628 INFO:teuthology.orchestra.run.smithi156.stdout:Fetched 11.3 MB in 2s (4,711 kB/s) 2022-06-29T15:49:21.335 INFO:teuthology.orchestra.run.smithi197.stdout:Reading package lists... 2022-06-29T15:49:21.358 DEBUG:teuthology.orchestra.run.smithi197:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-06-29T15:49:21.426 INFO:teuthology.orchestra.run.smithi197.stdout:Reading package lists... 2022-06-29T15:49:21.581 INFO:teuthology.orchestra.run.smithi197.stdout:Building dependency tree... 2022-06-29T15:49:21.581 INFO:teuthology.orchestra.run.smithi197.stdout:Reading state information... 2022-06-29T15:49:21.651 INFO:teuthology.orchestra.run.smithi092.stdout:Reading package lists... 2022-06-29T15:49:21.674 DEBUG:teuthology.orchestra.run.smithi092:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-06-29T15:49:21.725 INFO:teuthology.orchestra.run.smithi197.stdout:The following additional packages will be installed: 2022-06-29T15:49:21.725 INFO:teuthology.orchestra.run.smithi197.stdout: linux-generic linux-headers-5.4.0-121 linux-headers-5.4.0-121-generic 2022-06-29T15:49:21.725 INFO:teuthology.orchestra.run.smithi197.stdout: linux-headers-generic linux-image-5.4.0-121-generic 2022-06-29T15:49:21.726 INFO:teuthology.orchestra.run.smithi197.stdout: linux-modules-5.4.0-121-generic linux-modules-extra-5.4.0-121-generic 2022-06-29T15:49:21.726 INFO:teuthology.orchestra.run.smithi197.stdout:Suggested packages: 2022-06-29T15:49:21.726 INFO:teuthology.orchestra.run.smithi197.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2022-06-29T15:49:21.727 INFO:teuthology.orchestra.run.smithi197.stdout:Recommended packages: 2022-06-29T15:49:21.727 INFO:teuthology.orchestra.run.smithi197.stdout: thermald 2022-06-29T15:49:21.746 INFO:teuthology.orchestra.run.smithi092.stdout:Reading package lists... 2022-06-29T15:49:21.777 INFO:teuthology.orchestra.run.smithi197.stdout:The following NEW packages will be installed: 2022-06-29T15:49:21.777 INFO:teuthology.orchestra.run.smithi197.stdout: linux-headers-5.4.0-121 linux-headers-5.4.0-121-generic 2022-06-29T15:49:21.778 INFO:teuthology.orchestra.run.smithi197.stdout: linux-image-5.4.0-121-generic linux-modules-5.4.0-121-generic 2022-06-29T15:49:21.778 INFO:teuthology.orchestra.run.smithi197.stdout: linux-modules-extra-5.4.0-121-generic 2022-06-29T15:49:21.779 INFO:teuthology.orchestra.run.smithi197.stdout:The following packages will be upgraded: 2022-06-29T15:49:21.779 INFO:teuthology.orchestra.run.smithi197.stdout: linux-generic linux-headers-generic linux-image-generic 2022-06-29T15:49:21.792 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2022-06-29T15:49:21.813 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2022-06-29T15:49:21.815 DEBUG:teuthology.orchestra.run.smithi060:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-06-29T15:49:21.829 INFO:teuthology.orchestra.run.smithi197.stdout:3 upgraded, 5 newly installed, 0 to remove and 17 not upgraded. 2022-06-29T15:49:21.830 INFO:teuthology.orchestra.run.smithi197.stdout:Need to get 77.1 MB of archives. 2022-06-29T15:49:21.830 INFO:teuthology.orchestra.run.smithi197.stdout:After this operation, 380 MB of additional disk space will be used. 2022-06-29T15:49:21.830 INFO:teuthology.orchestra.run.smithi197.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-121-generic amd64 5.4.0-121.137 [15.0 MB] 2022-06-29T15:49:21.838 DEBUG:teuthology.orchestra.run.smithi121:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-06-29T15:49:21.882 INFO:teuthology.orchestra.run.smithi156.stdout:Reading package lists... 2022-06-29T15:49:21.883 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2022-06-29T15:49:21.907 DEBUG:teuthology.orchestra.run.smithi156:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-06-29T15:49:21.908 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2022-06-29T15:49:21.922 INFO:teuthology.orchestra.run.smithi092.stdout:Building dependency tree... 2022-06-29T15:49:21.923 INFO:teuthology.orchestra.run.smithi092.stdout:Reading state information... 2022-06-29T15:49:21.979 INFO:teuthology.orchestra.run.smithi156.stdout:Reading package lists... 2022-06-29T15:49:22.064 INFO:teuthology.orchestra.run.smithi060.stdout:Building dependency tree... 2022-06-29T15:49:22.066 INFO:teuthology.orchestra.run.smithi060.stdout:Reading state information... 2022-06-29T15:49:22.082 INFO:teuthology.orchestra.run.smithi121.stdout:Building dependency tree... 2022-06-29T15:49:22.083 INFO:teuthology.orchestra.run.smithi121.stdout:Reading state information... 2022-06-29T15:49:22.084 INFO:teuthology.orchestra.run.smithi092.stdout:The following additional packages will be installed: 2022-06-29T15:49:22.084 INFO:teuthology.orchestra.run.smithi092.stdout: linux-generic linux-headers-5.4.0-121 linux-headers-5.4.0-121-generic 2022-06-29T15:49:22.084 INFO:teuthology.orchestra.run.smithi092.stdout: linux-headers-generic linux-image-5.4.0-121-generic 2022-06-29T15:49:22.084 INFO:teuthology.orchestra.run.smithi092.stdout: linux-modules-5.4.0-121-generic linux-modules-extra-5.4.0-121-generic 2022-06-29T15:49:22.085 INFO:teuthology.orchestra.run.smithi092.stdout:Suggested packages: 2022-06-29T15:49:22.085 INFO:teuthology.orchestra.run.smithi092.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2022-06-29T15:49:22.085 INFO:teuthology.orchestra.run.smithi092.stdout:Recommended packages: 2022-06-29T15:49:22.086 INFO:teuthology.orchestra.run.smithi092.stdout: thermald 2022-06-29T15:49:22.126 INFO:teuthology.orchestra.run.smithi197.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-121-generic amd64 5.4.0-121.137 [10.5 MB] 2022-06-29T15:49:22.151 INFO:teuthology.orchestra.run.smithi092.stdout:The following NEW packages will be installed: 2022-06-29T15:49:22.151 INFO:teuthology.orchestra.run.smithi092.stdout: linux-headers-5.4.0-121 linux-headers-5.4.0-121-generic 2022-06-29T15:49:22.152 INFO:teuthology.orchestra.run.smithi092.stdout: linux-image-5.4.0-121-generic linux-modules-5.4.0-121-generic 2022-06-29T15:49:22.152 INFO:teuthology.orchestra.run.smithi092.stdout: linux-modules-extra-5.4.0-121-generic 2022-06-29T15:49:22.153 INFO:teuthology.orchestra.run.smithi092.stdout:The following packages will be upgraded: 2022-06-29T15:49:22.153 INFO:teuthology.orchestra.run.smithi092.stdout: linux-generic linux-headers-generic linux-image-generic 2022-06-29T15:49:22.155 INFO:teuthology.orchestra.run.smithi156.stdout:Building dependency tree... 2022-06-29T15:49:22.156 INFO:teuthology.orchestra.run.smithi156.stdout:Reading state information... 2022-06-29T15:49:22.221 INFO:teuthology.orchestra.run.smithi197.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-121-generic amd64 5.4.0-121.137 [39.2 MB] 2022-06-29T15:49:22.231 INFO:teuthology.orchestra.run.smithi060.stdout:The following additional packages will be installed: 2022-06-29T15:49:22.232 INFO:teuthology.orchestra.run.smithi060.stdout: linux-generic linux-headers-5.4.0-121 linux-headers-5.4.0-121-generic 2022-06-29T15:49:22.232 INFO:teuthology.orchestra.run.smithi060.stdout: linux-headers-generic linux-image-5.4.0-121-generic 2022-06-29T15:49:22.232 INFO:teuthology.orchestra.run.smithi060.stdout: linux-modules-5.4.0-121-generic linux-modules-extra-5.4.0-121-generic 2022-06-29T15:49:22.233 INFO:teuthology.orchestra.run.smithi060.stdout:Suggested packages: 2022-06-29T15:49:22.233 INFO:teuthology.orchestra.run.smithi060.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2022-06-29T15:49:22.233 INFO:teuthology.orchestra.run.smithi060.stdout:Recommended packages: 2022-06-29T15:49:22.233 INFO:teuthology.orchestra.run.smithi060.stdout: thermald 2022-06-29T15:49:22.242 INFO:teuthology.orchestra.run.smithi121.stdout:The following additional packages will be installed: 2022-06-29T15:49:22.243 INFO:teuthology.orchestra.run.smithi121.stdout: linux-generic linux-headers-5.4.0-121 linux-headers-5.4.0-121-generic 2022-06-29T15:49:22.243 INFO:teuthology.orchestra.run.smithi121.stdout: linux-headers-generic linux-image-5.4.0-121-generic 2022-06-29T15:49:22.243 INFO:teuthology.orchestra.run.smithi121.stdout: linux-modules-5.4.0-121-generic linux-modules-extra-5.4.0-121-generic 2022-06-29T15:49:22.244 INFO:teuthology.orchestra.run.smithi121.stdout:Suggested packages: 2022-06-29T15:49:22.245 INFO:teuthology.orchestra.run.smithi121.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2022-06-29T15:49:22.245 INFO:teuthology.orchestra.run.smithi121.stdout:Recommended packages: 2022-06-29T15:49:22.245 INFO:teuthology.orchestra.run.smithi121.stdout: thermald 2022-06-29T15:49:22.282 INFO:teuthology.orchestra.run.smithi060.stdout:The following NEW packages will be installed: 2022-06-29T15:49:22.283 INFO:teuthology.orchestra.run.smithi060.stdout: linux-headers-5.4.0-121 linux-headers-5.4.0-121-generic 2022-06-29T15:49:22.283 INFO:teuthology.orchestra.run.smithi060.stdout: linux-image-5.4.0-121-generic linux-modules-5.4.0-121-generic 2022-06-29T15:49:22.283 INFO:teuthology.orchestra.run.smithi060.stdout: linux-modules-extra-5.4.0-121-generic 2022-06-29T15:49:22.284 INFO:teuthology.orchestra.run.smithi060.stdout:The following packages will be upgraded: 2022-06-29T15:49:22.285 INFO:teuthology.orchestra.run.smithi060.stdout: linux-generic linux-headers-generic linux-image-generic 2022-06-29T15:49:22.311 INFO:teuthology.orchestra.run.smithi156.stdout:The following additional packages will be installed: 2022-06-29T15:49:22.312 INFO:teuthology.orchestra.run.smithi156.stdout: linux-generic linux-headers-5.4.0-121 linux-headers-5.4.0-121-generic 2022-06-29T15:49:22.312 INFO:teuthology.orchestra.run.smithi156.stdout: linux-headers-generic linux-image-5.4.0-121-generic 2022-06-29T15:49:22.312 INFO:teuthology.orchestra.run.smithi156.stdout: linux-modules-5.4.0-121-generic linux-modules-extra-5.4.0-121-generic 2022-06-29T15:49:22.313 INFO:teuthology.orchestra.run.smithi156.stdout:Suggested packages: 2022-06-29T15:49:22.313 INFO:teuthology.orchestra.run.smithi156.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2022-06-29T15:49:22.313 INFO:teuthology.orchestra.run.smithi156.stdout:Recommended packages: 2022-06-29T15:49:22.313 INFO:teuthology.orchestra.run.smithi156.stdout: thermald 2022-06-29T15:49:22.314 INFO:teuthology.orchestra.run.smithi121.stdout:The following NEW packages will be installed: 2022-06-29T15:49:22.314 INFO:teuthology.orchestra.run.smithi121.stdout: linux-headers-5.4.0-121 linux-headers-5.4.0-121-generic 2022-06-29T15:49:22.315 INFO:teuthology.orchestra.run.smithi121.stdout: linux-image-5.4.0-121-generic linux-modules-5.4.0-121-generic 2022-06-29T15:49:22.315 INFO:teuthology.orchestra.run.smithi121.stdout: linux-modules-extra-5.4.0-121-generic 2022-06-29T15:49:22.315 INFO:teuthology.orchestra.run.smithi121.stdout:The following packages will be upgraded: 2022-06-29T15:49:22.316 INFO:teuthology.orchestra.run.smithi121.stdout: linux-generic linux-headers-generic linux-image-generic 2022-06-29T15:49:22.329 INFO:teuthology.orchestra.run.smithi092.stdout:3 upgraded, 5 newly installed, 0 to remove and 17 not upgraded. 2022-06-29T15:49:22.329 INFO:teuthology.orchestra.run.smithi092.stdout:Need to get 77.1 MB of archives. 2022-06-29T15:49:22.329 INFO:teuthology.orchestra.run.smithi092.stdout:After this operation, 380 MB of additional disk space will be used. 2022-06-29T15:49:22.330 INFO:teuthology.orchestra.run.smithi092.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-121-generic amd64 5.4.0-121.137 [15.0 MB] 2022-06-29T15:49:22.366 INFO:teuthology.orchestra.run.smithi156.stdout:The following NEW packages will be installed: 2022-06-29T15:49:22.366 INFO:teuthology.orchestra.run.smithi156.stdout: linux-headers-5.4.0-121 linux-headers-5.4.0-121-generic 2022-06-29T15:49:22.366 INFO:teuthology.orchestra.run.smithi156.stdout: linux-image-5.4.0-121-generic linux-modules-5.4.0-121-generic 2022-06-29T15:49:22.367 INFO:teuthology.orchestra.run.smithi156.stdout: linux-modules-extra-5.4.0-121-generic 2022-06-29T15:49:22.368 INFO:teuthology.orchestra.run.smithi121.stdout:3 upgraded, 5 newly installed, 0 to remove and 17 not upgraded. 2022-06-29T15:49:22.368 INFO:teuthology.orchestra.run.smithi121.stdout:Need to get 77.1 MB of archives. 2022-06-29T15:49:22.368 INFO:teuthology.orchestra.run.smithi121.stdout:After this operation, 380 MB of additional disk space will be used. 2022-06-29T15:49:22.368 INFO:teuthology.orchestra.run.smithi121.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-121-generic amd64 5.4.0-121.137 [15.0 MB] 2022-06-29T15:49:22.369 INFO:teuthology.orchestra.run.smithi156.stdout:The following packages will be upgraded: 2022-06-29T15:49:22.369 INFO:teuthology.orchestra.run.smithi156.stdout: linux-generic linux-headers-generic linux-image-generic 2022-06-29T15:49:22.424 INFO:teuthology.orchestra.run.smithi156.stdout:3 upgraded, 5 newly installed, 0 to remove and 17 not upgraded. 2022-06-29T15:49:22.424 INFO:teuthology.orchestra.run.smithi156.stdout:Need to get 77.1 MB of archives. 2022-06-29T15:49:22.424 INFO:teuthology.orchestra.run.smithi156.stdout:After this operation, 380 MB of additional disk space will be used. 2022-06-29T15:49:22.425 INFO:teuthology.orchestra.run.smithi156.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-121-generic amd64 5.4.0-121.137 [15.0 MB] 2022-06-29T15:49:22.671 INFO:teuthology.orchestra.run.smithi060.stdout:3 upgraded, 5 newly installed, 0 to remove and 17 not upgraded. 2022-06-29T15:49:22.671 INFO:teuthology.orchestra.run.smithi060.stdout:Need to get 77.1 MB of archives. 2022-06-29T15:49:22.672 INFO:teuthology.orchestra.run.smithi060.stdout:After this operation, 380 MB of additional disk space will be used. 2022-06-29T15:49:22.672 INFO:teuthology.orchestra.run.smithi060.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-121-generic amd64 5.4.0-121.137 [15.0 MB] 2022-06-29T15:49:22.675 INFO:teuthology.orchestra.run.smithi121.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-121-generic amd64 5.4.0-121.137 [10.5 MB] 2022-06-29T15:49:22.686 INFO:teuthology.orchestra.run.smithi197.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.121.122 [1,896 B] 2022-06-29T15:49:22.687 INFO:teuthology.orchestra.run.smithi197.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.121.122 [2,560 B] 2022-06-29T15:49:22.687 INFO:teuthology.orchestra.run.smithi197.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-121 all 5.4.0-121.137 [11.0 MB] 2022-06-29T15:49:22.772 INFO:teuthology.orchestra.run.smithi121.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-121-generic amd64 5.4.0-121.137 [39.2 MB] 2022-06-29T15:49:22.780 INFO:teuthology.orchestra.run.smithi197.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-121-generic amd64 5.4.0-121.137 [1,380 kB] 2022-06-29T15:49:22.790 INFO:teuthology.orchestra.run.smithi197.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.121.122 [2,424 B] 2022-06-29T15:49:22.810 INFO:teuthology.orchestra.run.smithi156.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-121-generic amd64 5.4.0-121.137 [10.5 MB] 2022-06-29T15:49:22.930 INFO:teuthology.orchestra.run.smithi156.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-121-generic amd64 5.4.0-121.137 [39.2 MB] 2022-06-29T15:49:23.243 INFO:teuthology.orchestra.run.smithi197.stdout:Fetched 77.1 MB in 1s (76.6 MB/s) 2022-06-29T15:49:23.249 INFO:teuthology.orchestra.run.smithi121.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.121.122 [1,896 B] 2022-06-29T15:49:23.249 INFO:teuthology.orchestra.run.smithi121.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.121.122 [2,560 B] 2022-06-29T15:49:23.250 INFO:teuthology.orchestra.run.smithi121.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-121 all 5.4.0-121.137 [11.0 MB] 2022-06-29T15:49:23.344 INFO:teuthology.orchestra.run.smithi121.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-121-generic amd64 5.4.0-121.137 [1,380 kB] 2022-06-29T15:49:23.355 INFO:teuthology.orchestra.run.smithi121.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.121.122 [2,424 B] 2022-06-29T15:49:23.450 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package linux-modules-5.4.0-121-generic. 2022-06-29T15:49:23.476 INFO:teuthology.orchestra.run.smithi092.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-121-generic amd64 5.4.0-121.137 [10.5 MB] 2022-06-29T15:49:23.582 INFO:teuthology.orchestra.run.smithi156.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.121.122 [1,896 B] 2022-06-29T15:49:23.583 INFO:teuthology.orchestra.run.smithi156.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.121.122 [2,560 B] 2022-06-29T15:49:23.583 INFO:teuthology.orchestra.run.smithi156.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-121 all 5.4.0-121.137 [11.0 MB] 2022-06-29T15:49:23.733 INFO:teuthology.orchestra.run.smithi156.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-121-generic amd64 5.4.0-121.137 [1,380 kB] 2022-06-29T15:49:23.743 INFO:teuthology.orchestra.run.smithi092.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-121-generic amd64 5.4.0-121.137 [39.2 MB] 2022-06-29T15:49:23.750 INFO:teuthology.orchestra.run.smithi156.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.121.122 [2,424 B] 2022-06-29T15:49:23.772 INFO:teuthology.orchestra.run.smithi121.stdout:Fetched 77.1 MB in 1s (74.5 MB/s) 2022-06-29T15:49:23.799 INFO:teuthology.orchestra.run.smithi060.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-121-generic amd64 5.4.0-121.137 [10.5 MB] 2022-06-29T15:49:23.952 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package linux-modules-5.4.0-121-generic. 2022-06-29T15:49:24.063 INFO:teuthology.orchestra.run.smithi060.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-121-generic amd64 5.4.0-121.137 [39.2 MB] 2022-06-29T15:49:24.183 INFO:teuthology.orchestra.run.smithi156.stdout:Fetched 77.1 MB in 1s (55.9 MB/s) 2022-06-29T15:49:24.378 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package linux-modules-5.4.0-121-generic. 2022-06-29T15:49:25.015 INFO:teuthology.orchestra.run.smithi092.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.121.122 [1,896 B] 2022-06-29T15:49:25.015 INFO:teuthology.orchestra.run.smithi092.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.121.122 [2,560 B] 2022-06-29T15:49:25.016 INFO:teuthology.orchestra.run.smithi092.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-121 all 5.4.0-121.137 [11.0 MB] 2022-06-29T15:49:25.043 INFO:teuthology.orchestra.run.smithi197.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102341 files and directories currently installed.) 2022-06-29T15:49:25.047 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../0-linux-modules-5.4.0-121-generic_5.4.0-121.137_amd64.deb ... 2022-06-29T15:49:25.182 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking linux-modules-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:25.284 INFO:teuthology.orchestra.run.smithi060.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.121.122 [1,896 B] 2022-06-29T15:49:25.284 INFO:teuthology.orchestra.run.smithi060.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.121.122 [2,560 B] 2022-06-29T15:49:25.284 INFO:teuthology.orchestra.run.smithi060.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-121 all 5.4.0-121.137 [11.0 MB] 2022-06-29T15:49:25.530 INFO:teuthology.orchestra.run.smithi121.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102341 files and directories currently installed.) 2022-06-29T15:49:25.535 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../0-linux-modules-5.4.0-121-generic_5.4.0-121.137_amd64.deb ... 2022-06-29T15:49:25.561 INFO:teuthology.orchestra.run.smithi060.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-121-generic amd64 5.4.0-121.137 [1,380 kB] 2022-06-29T15:49:25.607 INFO:teuthology.orchestra.run.smithi060.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.121.122 [2,424 B] 2022-06-29T15:49:25.661 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-modules-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:25.901 INFO:teuthology.orchestra.run.smithi092.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-121-generic amd64 5.4.0-121.137 [1,380 kB] 2022-06-29T15:49:25.917 INFO:teuthology.orchestra.run.smithi156.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102341 files and directories currently installed.) 2022-06-29T15:49:25.922 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../0-linux-modules-5.4.0-121-generic_5.4.0-121.137_amd64.deb ... 2022-06-29T15:49:25.978 INFO:teuthology.orchestra.run.smithi092.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.121.122 [2,424 B] 2022-06-29T15:49:26.051 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking linux-modules-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:26.058 INFO:teuthology.orchestra.run.smithi060.stdout:Fetched 77.1 MB in 3s (23.2 MB/s) 2022-06-29T15:49:26.246 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package linux-modules-5.4.0-121-generic. 2022-06-29T15:49:26.422 INFO:teuthology.orchestra.run.smithi092.stdout:Fetched 77.1 MB in 4s (20.2 MB/s) 2022-06-29T15:49:26.602 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package linux-modules-5.4.0-121-generic. 2022-06-29T15:49:27.028 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package linux-image-5.4.0-121-generic. 2022-06-29T15:49:27.042 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../1-linux-image-5.4.0-121-generic_5.4.0-121.137_amd64.deb ... 2022-06-29T15:49:27.139 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking linux-image-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:27.505 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package linux-image-5.4.0-121-generic. 2022-06-29T15:49:27.519 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../1-linux-image-5.4.0-121-generic_5.4.0-121.137_amd64.deb ... 2022-06-29T15:49:27.612 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-image-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:27.818 INFO:teuthology.orchestra.run.smithi060.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102341 files and directories currently installed.) 2022-06-29T15:49:27.823 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../0-linux-modules-5.4.0-121-generic_5.4.0-121.137_amd64.deb ... 2022-06-29T15:49:27.944 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package linux-image-5.4.0-121-generic. 2022-06-29T15:49:27.944 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking linux-modules-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:27.956 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../1-linux-image-5.4.0-121-generic_5.4.0-121.137_amd64.deb ... 2022-06-29T15:49:28.061 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking linux-image-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:28.189 INFO:teuthology.orchestra.run.smithi092.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102341 files and directories currently installed.) 2022-06-29T15:49:28.194 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../0-linux-modules-5.4.0-121-generic_5.4.0-121.137_amd64.deb ... 2022-06-29T15:49:28.253 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-121-generic. 2022-06-29T15:49:28.265 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-121-generic_5.4.0-121.137_amd64.deb ... 2022-06-29T15:49:28.302 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking linux-modules-extra-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:28.419 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking linux-modules-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:28.684 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-121-generic. 2022-06-29T15:49:28.699 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-121-generic_5.4.0-121.137_amd64.deb ... 2022-06-29T15:49:28.742 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-modules-extra-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:29.126 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-121-generic. 2022-06-29T15:49:29.139 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-121-generic_5.4.0-121.137_amd64.deb ... 2022-06-29T15:49:29.176 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking linux-modules-extra-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:29.859 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package linux-image-5.4.0-121-generic. 2022-06-29T15:49:29.872 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../1-linux-image-5.4.0-121-generic_5.4.0-121.137_amd64.deb ... 2022-06-29T15:49:29.964 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking linux-image-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:30.433 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package linux-image-5.4.0-121-generic. 2022-06-29T15:49:30.446 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../1-linux-image-5.4.0-121-generic_5.4.0-121.137_amd64.deb ... 2022-06-29T15:49:30.537 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking linux-image-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:31.068 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-121-generic. 2022-06-29T15:49:31.083 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-121-generic_5.4.0-121.137_amd64.deb ... 2022-06-29T15:49:31.125 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking linux-modules-extra-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:31.599 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-121-generic. 2022-06-29T15:49:31.612 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-121-generic_5.4.0-121.137_amd64.deb ... 2022-06-29T15:49:31.649 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking linux-modules-extra-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:33.104 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../3-linux-generic_5.4.0.121.122_amd64.deb ... 2022-06-29T15:49:33.206 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking linux-generic (5.4.0.121.122) over (5.4.0.120.121) ... 2022-06-29T15:49:33.378 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../3-linux-generic_5.4.0.121.122_amd64.deb ... 2022-06-29T15:49:33.501 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-generic (5.4.0.121.122) over (5.4.0.120.121) ... 2022-06-29T15:49:33.550 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.121.122_amd64.deb ... 2022-06-29T15:49:33.676 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking linux-image-generic (5.4.0.121.122) over (5.4.0.120.121) ... 2022-06-29T15:49:33.851 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.121.122_amd64.deb ... 2022-06-29T15:49:33.980 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-image-generic (5.4.0.121.122) over (5.4.0.120.121) ... 2022-06-29T15:49:34.058 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../3-linux-generic_5.4.0.121.122_amd64.deb ... 2022-06-29T15:49:34.181 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking linux-generic (5.4.0.121.122) over (5.4.0.120.121) ... 2022-06-29T15:49:34.282 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package linux-headers-5.4.0-121. 2022-06-29T15:49:34.296 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package linux-headers-5.4.0-121. 2022-06-29T15:49:34.297 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../5-linux-headers-5.4.0-121_5.4.0-121.137_all.deb ... 2022-06-29T15:49:34.311 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../5-linux-headers-5.4.0-121_5.4.0-121.137_all.deb ... 2022-06-29T15:49:34.340 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-headers-5.4.0-121 (5.4.0-121.137) ... 2022-06-29T15:49:34.379 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking linux-headers-5.4.0-121 (5.4.0-121.137) ... 2022-06-29T15:49:34.525 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.121.122_amd64.deb ... 2022-06-29T15:49:34.677 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking linux-image-generic (5.4.0.121.122) over (5.4.0.120.121) ... 2022-06-29T15:49:34.997 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package linux-headers-5.4.0-121. 2022-06-29T15:49:35.013 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../5-linux-headers-5.4.0-121_5.4.0-121.137_all.deb ... 2022-06-29T15:49:35.055 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking linux-headers-5.4.0-121 (5.4.0-121.137) ... 2022-06-29T15:49:35.796 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../3-linux-generic_5.4.0.121.122_amd64.deb ... 2022-06-29T15:49:35.921 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking linux-generic (5.4.0.121.122) over (5.4.0.120.121) ... 2022-06-29T15:49:36.288 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.121.122_amd64.deb ... 2022-06-29T15:49:36.391 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking linux-image-generic (5.4.0.121.122) over (5.4.0.120.121) ... 2022-06-29T15:49:36.678 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package linux-headers-5.4.0-121. 2022-06-29T15:49:36.691 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../5-linux-headers-5.4.0-121_5.4.0-121.137_all.deb ... 2022-06-29T15:49:36.727 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking linux-headers-5.4.0-121 (5.4.0-121.137) ... 2022-06-29T15:49:36.867 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../3-linux-generic_5.4.0.121.122_amd64.deb ... 2022-06-29T15:49:37.011 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking linux-generic (5.4.0.121.122) over (5.4.0.120.121) ... 2022-06-29T15:49:37.377 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.121.122_amd64.deb ... 2022-06-29T15:49:37.515 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking linux-image-generic (5.4.0.121.122) over (5.4.0.120.121) ... 2022-06-29T15:49:37.851 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package linux-headers-5.4.0-121. 2022-06-29T15:49:37.867 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../5-linux-headers-5.4.0-121_5.4.0-121.137_all.deb ... 2022-06-29T15:49:37.943 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking linux-headers-5.4.0-121 (5.4.0-121.137) ... 2022-06-29T15:49:39.534 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package linux-headers-5.4.0-121-generic. 2022-06-29T15:49:39.555 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../6-linux-headers-5.4.0-121-generic_5.4.0-121.137_amd64.deb ... 2022-06-29T15:49:39.600 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-headers-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:39.628 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package linux-headers-5.4.0-121-generic. 2022-06-29T15:49:39.650 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../6-linux-headers-5.4.0-121-generic_5.4.0-121.137_amd64.deb ... 2022-06-29T15:49:39.694 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking linux-headers-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:40.324 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package linux-headers-5.4.0-121-generic. 2022-06-29T15:49:40.346 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../6-linux-headers-5.4.0-121-generic_5.4.0-121.137_amd64.deb ... 2022-06-29T15:49:40.382 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking linux-headers-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:42.091 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package linux-headers-5.4.0-121-generic. 2022-06-29T15:49:42.112 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../6-linux-headers-5.4.0-121-generic_5.4.0-121.137_amd64.deb ... 2022-06-29T15:49:42.149 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking linux-headers-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:42.285 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.121.122_amd64.deb ... 2022-06-29T15:49:42.413 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking linux-headers-generic (5.4.0.121.122) over (5.4.0.120.121) ... 2022-06-29T15:49:42.755 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up linux-modules-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:43.139 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.121.122_amd64.deb ... 2022-06-29T15:49:43.216 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.121.122_amd64.deb ... 2022-06-29T15:49:43.267 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-headers-generic (5.4.0.121.122) over (5.4.0.120.121) ... 2022-06-29T15:49:43.345 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking linux-headers-generic (5.4.0.121.122) over (5.4.0.120.121) ... 2022-06-29T15:49:43.482 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package linux-headers-5.4.0-121-generic. 2022-06-29T15:49:43.504 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../6-linux-headers-5.4.0-121-generic_5.4.0-121.137_amd64.deb ... 2022-06-29T15:49:43.540 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking linux-headers-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:43.605 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-modules-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:43.679 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up linux-modules-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:43.985 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up linux-headers-5.4.0-121 (5.4.0-121.137) ... 2022-06-29T15:49:44.111 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up linux-image-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:44.794 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.121.122_amd64.deb ... 2022-06-29T15:49:44.831 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-headers-5.4.0-121 (5.4.0-121.137) ... 2022-06-29T15:49:44.894 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up linux-headers-5.4.0-121 (5.4.0-121.137) ... 2022-06-29T15:49:44.955 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking linux-headers-generic (5.4.0.121.122) over (5.4.0.120.121) ... 2022-06-29T15:49:44.957 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-image-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:45.012 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up linux-image-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:45.272 INFO:teuthology.orchestra.run.smithi197.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-121-generic 2022-06-29T15:49:45.273 INFO:teuthology.orchestra.run.smithi197.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-121-generic 2022-06-29T15:49:45.388 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up linux-modules-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:45.422 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up linux-headers-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:45.531 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up linux-headers-generic (5.4.0.121.122) ... 2022-06-29T15:49:45.641 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up linux-modules-extra-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:46.105 INFO:teuthology.orchestra.run.smithi121.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-121-generic 2022-06-29T15:49:46.105 INFO:teuthology.orchestra.run.smithi121.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-121-generic 2022-06-29T15:49:46.169 INFO:teuthology.orchestra.run.smithi156.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-121-generic 2022-06-29T15:49:46.170 INFO:teuthology.orchestra.run.smithi156.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-121-generic 2022-06-29T15:49:46.226 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-headers-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:46.306 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up linux-headers-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:46.345 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.121.122_amd64.deb ... 2022-06-29T15:49:46.352 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-headers-generic (5.4.0.121.122) ... 2022-06-29T15:49:46.432 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up linux-headers-generic (5.4.0.121.122) ... 2022-06-29T15:49:46.467 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking linux-headers-generic (5.4.0.121.122) over (5.4.0.120.121) ... 2022-06-29T15:49:46.478 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-modules-extra-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:46.566 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up linux-modules-extra-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:46.635 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up linux-headers-5.4.0-121 (5.4.0-121.137) ... 2022-06-29T15:49:46.761 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up linux-image-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:46.817 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up linux-image-generic (5.4.0.121.122) ... 2022-06-29T15:49:46.861 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up linux-modules-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:46.952 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up linux-generic (5.4.0.121.122) ... 2022-06-29T15:49:47.103 INFO:teuthology.orchestra.run.smithi197.stdout:Processing triggers for linux-image-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:47.156 INFO:teuthology.orchestra.run.smithi197.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-06-29T15:49:47.156 INFO:teuthology.orchestra.run.smithi197.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-121-generic 2022-06-29T15:49:47.662 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-image-generic (5.4.0.121.122) ... 2022-06-29T15:49:47.768 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up linux-image-generic (5.4.0.121.122) ... 2022-06-29T15:49:47.813 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-generic (5.4.0.121.122) ... 2022-06-29T15:49:47.894 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up linux-generic (5.4.0.121.122) ... 2022-06-29T15:49:47.937 INFO:teuthology.orchestra.run.smithi060.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-121-generic 2022-06-29T15:49:47.938 INFO:teuthology.orchestra.run.smithi060.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-121-generic 2022-06-29T15:49:47.940 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for linux-image-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:47.992 INFO:teuthology.orchestra.run.smithi121.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-06-29T15:49:47.993 INFO:teuthology.orchestra.run.smithi121.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-121-generic 2022-06-29T15:49:48.021 INFO:teuthology.orchestra.run.smithi156.stdout:Processing triggers for linux-image-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:48.072 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up linux-headers-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:48.072 INFO:teuthology.orchestra.run.smithi156.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-06-29T15:49:48.073 INFO:teuthology.orchestra.run.smithi156.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-121-generic 2022-06-29T15:49:48.089 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up linux-headers-5.4.0-121 (5.4.0-121.137) ... 2022-06-29T15:49:48.215 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up linux-image-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:49.092 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up linux-headers-generic (5.4.0.121.122) ... 2022-06-29T15:49:49.628 INFO:teuthology.orchestra.run.smithi092.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-121-generic 2022-06-29T15:49:49.629 INFO:teuthology.orchestra.run.smithi092.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-121-generic 2022-06-29T15:49:49.666 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up linux-modules-extra-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:49.759 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up linux-headers-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:49.918 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up linux-headers-generic (5.4.0.121.122) ... 2022-06-29T15:49:50.069 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up linux-modules-extra-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:50.876 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up linux-image-generic (5.4.0.121.122) ... 2022-06-29T15:49:50.985 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up linux-generic (5.4.0.121.122) ... 2022-06-29T15:49:51.112 INFO:teuthology.orchestra.run.smithi060.stdout:Processing triggers for linux-image-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:51.175 INFO:teuthology.orchestra.run.smithi060.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-06-29T15:49:51.176 INFO:teuthology.orchestra.run.smithi060.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-121-generic 2022-06-29T15:49:51.270 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up linux-image-generic (5.4.0.121.122) ... 2022-06-29T15:49:51.380 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up linux-generic (5.4.0.121.122) ... 2022-06-29T15:49:51.515 INFO:teuthology.orchestra.run.smithi092.stdout:Processing triggers for linux-image-5.4.0-121-generic (5.4.0-121.137) ... 2022-06-29T15:49:51.567 INFO:teuthology.orchestra.run.smithi092.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-06-29T15:49:51.567 INFO:teuthology.orchestra.run.smithi092.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-121-generic 2022-06-29T15:50:09.104 INFO:teuthology.orchestra.run.smithi197.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-06-29T15:50:09.105 INFO:teuthology.orchestra.run.smithi197.stdout:Sourcing file `/etc/default/grub' 2022-06-29T15:50:09.115 INFO:teuthology.orchestra.run.smithi197.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-06-29T15:50:09.124 INFO:teuthology.orchestra.run.smithi197.stdout:Generating grub configuration file ... 2022-06-29T15:50:09.394 INFO:teuthology.orchestra.run.smithi197.stdout:Found linux image: /boot/vmlinuz-5.4.0-121-generic 2022-06-29T15:50:09.419 INFO:teuthology.orchestra.run.smithi197.stdout:Found initrd image: /boot/initrd.img-5.4.0-121-generic 2022-06-29T15:50:09.824 INFO:teuthology.orchestra.run.smithi156.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-06-29T15:50:09.825 INFO:teuthology.orchestra.run.smithi156.stdout:Sourcing file `/etc/default/grub' 2022-06-29T15:50:09.833 INFO:teuthology.orchestra.run.smithi156.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-06-29T15:50:09.840 INFO:teuthology.orchestra.run.smithi197.stdout:Found linux image: /boot/vmlinuz-5.4.0-120-generic 2022-06-29T15:50:09.852 INFO:teuthology.orchestra.run.smithi197.stdout:Found initrd image: /boot/initrd.img-5.4.0-120-generic 2022-06-29T15:50:09.862 INFO:teuthology.orchestra.run.smithi156.stdout:Generating grub configuration file ... 2022-06-29T15:50:10.182 INFO:teuthology.orchestra.run.smithi156.stdout:Found linux image: /boot/vmlinuz-5.4.0-121-generic 2022-06-29T15:50:10.204 INFO:teuthology.orchestra.run.smithi156.stdout:Found initrd image: /boot/initrd.img-5.4.0-121-generic 2022-06-29T15:50:10.494 INFO:teuthology.orchestra.run.smithi121.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-06-29T15:50:10.495 INFO:teuthology.orchestra.run.smithi121.stdout:Sourcing file `/etc/default/grub' 2022-06-29T15:50:10.501 INFO:teuthology.orchestra.run.smithi121.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-06-29T15:50:10.526 INFO:teuthology.orchestra.run.smithi121.stdout:Generating grub configuration file ... 2022-06-29T15:50:10.582 INFO:teuthology.orchestra.run.smithi156.stdout:Found linux image: /boot/vmlinuz-5.4.0-120-generic 2022-06-29T15:50:10.594 INFO:teuthology.orchestra.run.smithi156.stdout:Found initrd image: /boot/initrd.img-5.4.0-120-generic 2022-06-29T15:50:10.798 INFO:teuthology.orchestra.run.smithi121.stdout:Found linux image: /boot/vmlinuz-5.4.0-121-generic 2022-06-29T15:50:10.819 INFO:teuthology.orchestra.run.smithi121.stdout:Found initrd image: /boot/initrd.img-5.4.0-121-generic 2022-06-29T15:50:11.194 INFO:teuthology.orchestra.run.smithi121.stdout:Found linux image: /boot/vmlinuz-5.4.0-120-generic 2022-06-29T15:50:11.205 INFO:teuthology.orchestra.run.smithi121.stdout:Found initrd image: /boot/initrd.img-5.4.0-120-generic 2022-06-29T15:50:12.544 INFO:teuthology.orchestra.run.smithi197.stdout:done 2022-06-29T15:50:12.891 DEBUG:teuthology.orchestra.run.smithi197:> dpkg -s linux-image-generic 2022-06-29T15:50:12.928 INFO:teuthology.orchestra.run.smithi197.stdout:Package: linux-image-generic 2022-06-29T15:50:12.928 INFO:teuthology.orchestra.run.smithi197.stdout:Status: install ok installed 2022-06-29T15:50:12.929 INFO:teuthology.orchestra.run.smithi197.stdout:Priority: optional 2022-06-29T15:50:12.929 INFO:teuthology.orchestra.run.smithi197.stdout:Section: kernel 2022-06-29T15:50:12.929 INFO:teuthology.orchestra.run.smithi197.stdout:Installed-Size: 19 2022-06-29T15:50:12.929 INFO:teuthology.orchestra.run.smithi197.stdout:Maintainer: Ubuntu Kernel Team 2022-06-29T15:50:12.930 INFO:teuthology.orchestra.run.smithi197.stdout:Architecture: amd64 2022-06-29T15:50:12.930 INFO:teuthology.orchestra.run.smithi197.stdout:Source: linux-meta 2022-06-29T15:50:12.931 INFO:teuthology.orchestra.run.smithi197.stdout:Version: 5.4.0.121.122 2022-06-29T15:50:12.931 INFO:teuthology.orchestra.run.smithi197.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-06-29T15:50:12.931 INFO:teuthology.orchestra.run.smithi197.stdout:Depends: linux-image-5.4.0-121-generic, linux-modules-extra-5.4.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2022-06-29T15:50:12.931 INFO:teuthology.orchestra.run.smithi197.stdout:Recommends: thermald 2022-06-29T15:50:12.932 INFO:teuthology.orchestra.run.smithi197.stdout:Description: Generic Linux kernel image 2022-06-29T15:50:12.932 INFO:teuthology.orchestra.run.smithi197.stdout: This package will always depend on the latest generic kernel image 2022-06-29T15:50:12.932 INFO:teuthology.orchestra.run.smithi197.stdout: available. 2022-06-29T15:50:12.932 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-120-generic Expected: 5.4.0-121-generic 2022-06-29T15:50:12.933 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-06-29T15:50:12.933 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-06-29T15:50:12.933 INFO:teuthology.task.kernel:Installing distro kernel on client.2... 2022-06-29T15:50:12.934 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi197.front.sepia.ceph.com, path=None, version=distro) 2022-06-29T15:50:12.934 DEBUG:teuthology.orchestra.run.smithi197:> sudo apt-get clean 2022-06-29T15:50:12.986 DEBUG:teuthology.orchestra.run.smithi197:> sudo apt-get update 2022-06-29T15:50:13.153 INFO:teuthology.orchestra.run.smithi197.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-06-29T15:50:13.165 INFO:teuthology.orchestra.run.smithi197.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-06-29T15:50:13.176 INFO:teuthology.orchestra.run.smithi197.stdout:Hit:3 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-06-29T15:50:13.205 INFO:teuthology.orchestra.run.smithi197.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-06-29T15:50:13.215 INFO:teuthology.orchestra.run.smithi156.stdout:done 2022-06-29T15:50:13.507 INFO:teuthology.orchestra.run.smithi092.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-06-29T15:50:13.507 INFO:teuthology.orchestra.run.smithi092.stdout:Sourcing file `/etc/default/grub' 2022-06-29T15:50:13.509 INFO:teuthology.orchestra.run.smithi092.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-06-29T15:50:13.524 INFO:teuthology.orchestra.run.smithi092.stdout:Generating grub configuration file ... 2022-06-29T15:50:13.576 DEBUG:teuthology.orchestra.run.smithi156:> dpkg -s linux-image-generic 2022-06-29T15:50:13.602 INFO:teuthology.orchestra.run.smithi156.stdout:Package: linux-image-generic 2022-06-29T15:50:13.602 INFO:teuthology.orchestra.run.smithi156.stdout:Status: install ok installed 2022-06-29T15:50:13.603 INFO:teuthology.orchestra.run.smithi156.stdout:Priority: optional 2022-06-29T15:50:13.603 INFO:teuthology.orchestra.run.smithi156.stdout:Section: kernel 2022-06-29T15:50:13.603 INFO:teuthology.orchestra.run.smithi156.stdout:Installed-Size: 19 2022-06-29T15:50:13.603 INFO:teuthology.orchestra.run.smithi156.stdout:Maintainer: Ubuntu Kernel Team 2022-06-29T15:50:13.604 INFO:teuthology.orchestra.run.smithi156.stdout:Architecture: amd64 2022-06-29T15:50:13.604 INFO:teuthology.orchestra.run.smithi156.stdout:Source: linux-meta 2022-06-29T15:50:13.604 INFO:teuthology.orchestra.run.smithi156.stdout:Version: 5.4.0.121.122 2022-06-29T15:50:13.605 INFO:teuthology.orchestra.run.smithi156.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-06-29T15:50:13.605 INFO:teuthology.orchestra.run.smithi156.stdout:Depends: linux-image-5.4.0-121-generic, linux-modules-extra-5.4.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2022-06-29T15:50:13.605 INFO:teuthology.orchestra.run.smithi156.stdout:Recommends: thermald 2022-06-29T15:50:13.605 INFO:teuthology.orchestra.run.smithi156.stdout:Description: Generic Linux kernel image 2022-06-29T15:50:13.606 INFO:teuthology.orchestra.run.smithi156.stdout: This package will always depend on the latest generic kernel image 2022-06-29T15:50:13.606 INFO:teuthology.orchestra.run.smithi156.stdout: available. 2022-06-29T15:50:13.606 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-120-generic Expected: 5.4.0-121-generic 2022-06-29T15:50:13.606 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-06-29T15:50:13.607 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-06-29T15:50:13.607 INFO:teuthology.task.kernel:Installing distro kernel on client.1... 2022-06-29T15:50:13.607 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi156.front.sepia.ceph.com, path=None, version=distro) 2022-06-29T15:50:13.608 DEBUG:teuthology.orchestra.run.smithi156:> sudo apt-get clean 2022-06-29T15:50:13.659 DEBUG:teuthology.orchestra.run.smithi156:> sudo apt-get update 2022-06-29T15:50:13.799 INFO:teuthology.orchestra.run.smithi156.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-06-29T15:50:13.806 INFO:teuthology.orchestra.run.smithi156.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-06-29T15:50:13.813 INFO:teuthology.orchestra.run.smithi060.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-06-29T15:50:13.813 INFO:teuthology.orchestra.run.smithi060.stdout:Sourcing file `/etc/default/grub' 2022-06-29T15:50:13.822 INFO:teuthology.orchestra.run.smithi060.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-06-29T15:50:13.823 INFO:teuthology.orchestra.run.smithi156.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-06-29T15:50:13.831 INFO:teuthology.orchestra.run.smithi092.stdout:Found linux image: /boot/vmlinuz-5.4.0-121-generic 2022-06-29T15:50:13.838 INFO:teuthology.orchestra.run.smithi121.stdout:done 2022-06-29T15:50:13.847 INFO:teuthology.orchestra.run.smithi060.stdout:Generating grub configuration file ... 2022-06-29T15:50:13.858 INFO:teuthology.orchestra.run.smithi092.stdout:Found initrd image: /boot/initrd.img-5.4.0-121-generic 2022-06-29T15:50:13.905 INFO:teuthology.orchestra.run.smithi156.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-06-29T15:50:14.103 INFO:teuthology.orchestra.run.smithi060.stdout:Found linux image: /boot/vmlinuz-5.4.0-121-generic 2022-06-29T15:50:14.131 INFO:teuthology.orchestra.run.smithi060.stdout:Found initrd image: /boot/initrd.img-5.4.0-121-generic 2022-06-29T15:50:14.202 DEBUG:teuthology.orchestra.run.smithi121:> dpkg -s linux-image-generic 2022-06-29T15:50:14.243 INFO:teuthology.orchestra.run.smithi121.stdout:Package: linux-image-generic 2022-06-29T15:50:14.244 INFO:teuthology.orchestra.run.smithi121.stdout:Status: install ok installed 2022-06-29T15:50:14.244 INFO:teuthology.orchestra.run.smithi121.stdout:Priority: optional 2022-06-29T15:50:14.244 INFO:teuthology.orchestra.run.smithi121.stdout:Section: kernel 2022-06-29T15:50:14.245 INFO:teuthology.orchestra.run.smithi121.stdout:Installed-Size: 19 2022-06-29T15:50:14.245 INFO:teuthology.orchestra.run.smithi121.stdout:Maintainer: Ubuntu Kernel Team 2022-06-29T15:50:14.245 INFO:teuthology.orchestra.run.smithi121.stdout:Architecture: amd64 2022-06-29T15:50:14.245 INFO:teuthology.orchestra.run.smithi121.stdout:Source: linux-meta 2022-06-29T15:50:14.245 INFO:teuthology.orchestra.run.smithi121.stdout:Version: 5.4.0.121.122 2022-06-29T15:50:14.246 INFO:teuthology.orchestra.run.smithi121.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-06-29T15:50:14.246 INFO:teuthology.orchestra.run.smithi121.stdout:Depends: linux-image-5.4.0-121-generic, linux-modules-extra-5.4.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2022-06-29T15:50:14.246 INFO:teuthology.orchestra.run.smithi121.stdout:Recommends: thermald 2022-06-29T15:50:14.246 INFO:teuthology.orchestra.run.smithi121.stdout:Description: Generic Linux kernel image 2022-06-29T15:50:14.247 INFO:teuthology.orchestra.run.smithi121.stdout: This package will always depend on the latest generic kernel image 2022-06-29T15:50:14.247 INFO:teuthology.orchestra.run.smithi121.stdout: available. 2022-06-29T15:50:14.247 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-120-generic Expected: 5.4.0-121-generic 2022-06-29T15:50:14.248 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-06-29T15:50:14.248 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-06-29T15:50:14.248 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2022-06-29T15:50:14.249 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi121.front.sepia.ceph.com, path=None, version=distro) 2022-06-29T15:50:14.249 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-get clean 2022-06-29T15:50:14.293 INFO:teuthology.orchestra.run.smithi092.stdout:Found linux image: /boot/vmlinuz-5.4.0-120-generic 2022-06-29T15:50:14.305 INFO:teuthology.orchestra.run.smithi092.stdout:Found initrd image: /boot/initrd.img-5.4.0-120-generic 2022-06-29T15:50:14.309 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-get update 2022-06-29T15:50:14.461 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-06-29T15:50:14.520 INFO:teuthology.orchestra.run.smithi060.stdout:Found linux image: /boot/vmlinuz-5.4.0-120-generic 2022-06-29T15:50:14.532 INFO:teuthology.orchestra.run.smithi060.stdout:Found initrd image: /boot/initrd.img-5.4.0-120-generic 2022-06-29T15:50:14.553 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-06-29T15:50:14.633 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-06-29T15:50:14.713 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-06-29T15:50:14.736 INFO:teuthology.orchestra.run.smithi197.stdout:Reading package lists... 2022-06-29T15:50:14.759 DEBUG:teuthology.orchestra.run.smithi197:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-06-29T15:50:14.825 INFO:teuthology.orchestra.run.smithi197.stdout:Reading package lists... 2022-06-29T15:50:15.017 INFO:teuthology.orchestra.run.smithi197.stdout:Building dependency tree... 2022-06-29T15:50:15.018 INFO:teuthology.orchestra.run.smithi197.stdout:Reading state information... 2022-06-29T15:50:15.212 INFO:teuthology.orchestra.run.smithi197.stdout:linux-image-generic is already the newest version (5.4.0.121.122). 2022-06-29T15:50:15.212 INFO:teuthology.orchestra.run.smithi197.stdout:0 upgraded, 0 newly installed, 0 to remove and 17 not upgraded. 2022-06-29T15:50:15.213 DEBUG:teuthology.orchestra.run.smithi197:> dpkg -s linux-image-generic 2022-06-29T15:50:15.237 INFO:teuthology.orchestra.run.smithi197.stdout:Package: linux-image-generic 2022-06-29T15:50:15.238 INFO:teuthology.orchestra.run.smithi197.stdout:Status: install ok installed 2022-06-29T15:50:15.238 INFO:teuthology.orchestra.run.smithi197.stdout:Priority: optional 2022-06-29T15:50:15.238 INFO:teuthology.orchestra.run.smithi197.stdout:Section: kernel 2022-06-29T15:50:15.238 INFO:teuthology.orchestra.run.smithi197.stdout:Installed-Size: 19 2022-06-29T15:50:15.239 INFO:teuthology.orchestra.run.smithi197.stdout:Maintainer: Ubuntu Kernel Team 2022-06-29T15:50:15.239 INFO:teuthology.orchestra.run.smithi197.stdout:Architecture: amd64 2022-06-29T15:50:15.239 INFO:teuthology.orchestra.run.smithi197.stdout:Source: linux-meta 2022-06-29T15:50:15.239 INFO:teuthology.orchestra.run.smithi197.stdout:Version: 5.4.0.121.122 2022-06-29T15:50:15.240 INFO:teuthology.orchestra.run.smithi197.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-06-29T15:50:15.240 INFO:teuthology.orchestra.run.smithi197.stdout:Depends: linux-image-5.4.0-121-generic, linux-modules-extra-5.4.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2022-06-29T15:50:15.240 INFO:teuthology.orchestra.run.smithi197.stdout:Recommends: thermald 2022-06-29T15:50:15.240 INFO:teuthology.orchestra.run.smithi197.stdout:Description: Generic Linux kernel image 2022-06-29T15:50:15.240 INFO:teuthology.orchestra.run.smithi197.stdout: This package will always depend on the latest generic kernel image 2022-06-29T15:50:15.241 INFO:teuthology.orchestra.run.smithi197.stdout: available. 2022-06-29T15:50:15.241 DEBUG:teuthology.orchestra.run.smithi197:> mktemp 2022-06-29T15:50:15.247 INFO:teuthology.orchestra.run.smithi197.stdout:/tmp/tmp.aZ4QgM2J0X 2022-06-29T15:50:15.248 DEBUG:teuthology.orchestra.run.smithi197:> sudo cp /boot/grub/grub.cfg /tmp/tmp.aZ4QgM2J0X 2022-06-29T15:50:15.307 DEBUG:teuthology.orchestra.run.smithi197:> sudo chmod 0666 /tmp/tmp.aZ4QgM2J0X 2022-06-29T15:50:15.347 DEBUG:teuthology.orchestra.remote:smithi197:/tmp/tmp.aZ4QgM2J0X is 9KB 2022-06-29T15:50:15.362 DEBUG:teuthology.orchestra.run.smithi197:> rm -fr /tmp/tmp.aZ4QgM2J0X 2022-06-29T15:50:15.369 DEBUG:teuthology.orchestra.run.smithi197:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-06-29T15:50:15.402 INFO:teuthology.orchestra.run.smithi156.stdout:Reading package lists... 2022-06-29T15:50:15.425 DEBUG:teuthology.orchestra.run.smithi156:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-06-29T15:50:15.427 DEBUG:teuthology.orchestra.run.smithi197:> set -ex 2022-06-29T15:50:15.428 DEBUG:teuthology.orchestra.run.smithi197:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-06-29T15:50:15.428 DEBUG:teuthology.orchestra.run.smithi197:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-06-29T15:50:15.487 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-121-generic 2022-06-29T15:50:15.488 DEBUG:teuthology.orchestra.run.smithi197:> sudo update-grub 2022-06-29T15:50:15.492 INFO:teuthology.orchestra.run.smithi156.stdout:Reading package lists... 2022-06-29T15:50:15.667 INFO:teuthology.orchestra.run.smithi156.stdout:Building dependency tree... 2022-06-29T15:50:15.668 INFO:teuthology.orchestra.run.smithi156.stdout:Reading state information... 2022-06-29T15:50:15.853 INFO:teuthology.orchestra.run.smithi156.stdout:linux-image-generic is already the newest version (5.4.0.121.122). 2022-06-29T15:50:15.853 INFO:teuthology.orchestra.run.smithi156.stdout:0 upgraded, 0 newly installed, 0 to remove and 17 not upgraded. 2022-06-29T15:50:15.854 DEBUG:teuthology.orchestra.run.smithi156:> dpkg -s linux-image-generic 2022-06-29T15:50:15.870 INFO:teuthology.orchestra.run.smithi156.stdout:Package: linux-image-generic 2022-06-29T15:50:15.870 INFO:teuthology.orchestra.run.smithi156.stdout:Status: install ok installed 2022-06-29T15:50:15.871 INFO:teuthology.orchestra.run.smithi156.stdout:Priority: optional 2022-06-29T15:50:15.871 INFO:teuthology.orchestra.run.smithi156.stdout:Section: kernel 2022-06-29T15:50:15.871 INFO:teuthology.orchestra.run.smithi156.stdout:Installed-Size: 19 2022-06-29T15:50:15.871 INFO:teuthology.orchestra.run.smithi156.stdout:Maintainer: Ubuntu Kernel Team 2022-06-29T15:50:15.872 INFO:teuthology.orchestra.run.smithi156.stdout:Architecture: amd64 2022-06-29T15:50:15.872 INFO:teuthology.orchestra.run.smithi156.stdout:Source: linux-meta 2022-06-29T15:50:15.872 INFO:teuthology.orchestra.run.smithi156.stdout:Version: 5.4.0.121.122 2022-06-29T15:50:15.872 INFO:teuthology.orchestra.run.smithi156.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-06-29T15:50:15.872 INFO:teuthology.orchestra.run.smithi156.stdout:Depends: linux-image-5.4.0-121-generic, linux-modules-extra-5.4.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2022-06-29T15:50:15.873 INFO:teuthology.orchestra.run.smithi156.stdout:Recommends: thermald 2022-06-29T15:50:15.873 INFO:teuthology.orchestra.run.smithi156.stdout:Description: Generic Linux kernel image 2022-06-29T15:50:15.873 INFO:teuthology.orchestra.run.smithi156.stdout: This package will always depend on the latest generic kernel image 2022-06-29T15:50:15.873 INFO:teuthology.orchestra.run.smithi156.stdout: available. 2022-06-29T15:50:15.874 DEBUG:teuthology.orchestra.run.smithi156:> mktemp 2022-06-29T15:50:15.880 INFO:teuthology.orchestra.run.smithi156.stdout:/tmp/tmp.YHMMHmhZyz 2022-06-29T15:50:15.881 DEBUG:teuthology.orchestra.run.smithi156:> sudo cp /boot/grub/grub.cfg /tmp/tmp.YHMMHmhZyz 2022-06-29T15:50:15.940 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0666 /tmp/tmp.YHMMHmhZyz 2022-06-29T15:50:15.981 DEBUG:teuthology.orchestra.remote:smithi156:/tmp/tmp.YHMMHmhZyz is 9KB 2022-06-29T15:50:15.996 DEBUG:teuthology.orchestra.run.smithi156:> rm -fr /tmp/tmp.YHMMHmhZyz 2022-06-29T15:50:16.004 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-06-29T15:50:16.053 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2022-06-29T15:50:16.060 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2022-06-29T15:50:16.060 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-06-29T15:50:16.061 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-06-29T15:50:16.077 DEBUG:teuthology.orchestra.run.smithi121:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-06-29T15:50:16.092 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-121-generic 2022-06-29T15:50:16.092 DEBUG:teuthology.orchestra.run.smithi156:> sudo update-grub 2022-06-29T15:50:16.144 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2022-06-29T15:50:16.324 INFO:teuthology.orchestra.run.smithi121.stdout:Building dependency tree... 2022-06-29T15:50:16.325 INFO:teuthology.orchestra.run.smithi121.stdout:Reading state information... 2022-06-29T15:50:16.520 INFO:teuthology.orchestra.run.smithi121.stdout:linux-image-generic is already the newest version (5.4.0.121.122). 2022-06-29T15:50:16.520 INFO:teuthology.orchestra.run.smithi121.stdout:0 upgraded, 0 newly installed, 0 to remove and 17 not upgraded. 2022-06-29T15:50:16.522 DEBUG:teuthology.orchestra.run.smithi121:> dpkg -s linux-image-generic 2022-06-29T15:50:16.545 INFO:teuthology.orchestra.run.smithi121.stdout:Package: linux-image-generic 2022-06-29T15:50:16.546 INFO:teuthology.orchestra.run.smithi121.stdout:Status: install ok installed 2022-06-29T15:50:16.546 INFO:teuthology.orchestra.run.smithi121.stdout:Priority: optional 2022-06-29T15:50:16.546 INFO:teuthology.orchestra.run.smithi121.stdout:Section: kernel 2022-06-29T15:50:16.547 INFO:teuthology.orchestra.run.smithi121.stdout:Installed-Size: 19 2022-06-29T15:50:16.547 INFO:teuthology.orchestra.run.smithi121.stdout:Maintainer: Ubuntu Kernel Team 2022-06-29T15:50:16.548 INFO:teuthology.orchestra.run.smithi121.stdout:Architecture: amd64 2022-06-29T15:50:16.548 INFO:teuthology.orchestra.run.smithi121.stdout:Source: linux-meta 2022-06-29T15:50:16.549 INFO:teuthology.orchestra.run.smithi121.stdout:Version: 5.4.0.121.122 2022-06-29T15:50:16.549 INFO:teuthology.orchestra.run.smithi121.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-06-29T15:50:16.550 INFO:teuthology.orchestra.run.smithi121.stdout:Depends: linux-image-5.4.0-121-generic, linux-modules-extra-5.4.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2022-06-29T15:50:16.550 INFO:teuthology.orchestra.run.smithi121.stdout:Recommends: thermald 2022-06-29T15:50:16.550 INFO:teuthology.orchestra.run.smithi121.stdout:Description: Generic Linux kernel image 2022-06-29T15:50:16.551 INFO:teuthology.orchestra.run.smithi121.stdout: This package will always depend on the latest generic kernel image 2022-06-29T15:50:16.551 INFO:teuthology.orchestra.run.smithi121.stdout: available. 2022-06-29T15:50:16.552 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2022-06-29T15:50:16.596 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.8CZNHdxO9e 2022-06-29T15:50:16.597 DEBUG:teuthology.orchestra.run.smithi121:> sudo cp /boot/grub/grub.cfg /tmp/tmp.8CZNHdxO9e 2022-06-29T15:50:16.613 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.8CZNHdxO9e 2022-06-29T15:50:16.691 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.8CZNHdxO9e is 9KB 2022-06-29T15:50:16.706 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.8CZNHdxO9e 2022-06-29T15:50:16.713 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-06-29T15:50:16.769 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-06-29T15:50:16.770 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-06-29T15:50:16.770 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-06-29T15:50:16.813 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-121-generic 2022-06-29T15:50:16.814 DEBUG:teuthology.orchestra.run.smithi121:> sudo update-grub 2022-06-29T15:50:16.937 INFO:teuthology.orchestra.run.smithi197.stderr:Sourcing file `/etc/default/grub' 2022-06-29T15:50:16.939 INFO:teuthology.orchestra.run.smithi197.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-06-29T15:50:16.945 INFO:teuthology.orchestra.run.smithi197.stderr:Generating grub configuration file ... 2022-06-29T15:50:17.002 INFO:teuthology.orchestra.run.smithi092.stdout:done 2022-06-29T15:50:17.154 INFO:teuthology.orchestra.run.smithi060.stdout:done 2022-06-29T15:50:17.157 INFO:teuthology.orchestra.run.smithi197.stderr:Found linux image: /boot/vmlinuz-5.4.0-121-generic 2022-06-29T15:50:17.168 INFO:teuthology.orchestra.run.smithi197.stderr:Found initrd image: /boot/initrd.img-5.4.0-121-generic 2022-06-29T15:50:17.371 INFO:teuthology.orchestra.run.smithi156.stderr:Sourcing file `/etc/default/grub' 2022-06-29T15:50:17.372 INFO:teuthology.orchestra.run.smithi156.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-06-29T15:50:17.381 INFO:teuthology.orchestra.run.smithi156.stderr:Generating grub configuration file ... 2022-06-29T15:50:17.399 DEBUG:teuthology.orchestra.run.smithi092:> dpkg -s linux-image-generic 2022-06-29T15:50:17.425 INFO:teuthology.orchestra.run.smithi092.stdout:Package: linux-image-generic 2022-06-29T15:50:17.425 INFO:teuthology.orchestra.run.smithi092.stdout:Status: install ok installed 2022-06-29T15:50:17.425 INFO:teuthology.orchestra.run.smithi092.stdout:Priority: optional 2022-06-29T15:50:17.426 INFO:teuthology.orchestra.run.smithi092.stdout:Section: kernel 2022-06-29T15:50:17.426 INFO:teuthology.orchestra.run.smithi092.stdout:Installed-Size: 19 2022-06-29T15:50:17.426 INFO:teuthology.orchestra.run.smithi092.stdout:Maintainer: Ubuntu Kernel Team 2022-06-29T15:50:17.426 INFO:teuthology.orchestra.run.smithi092.stdout:Architecture: amd64 2022-06-29T15:50:17.427 INFO:teuthology.orchestra.run.smithi092.stdout:Source: linux-meta 2022-06-29T15:50:17.427 INFO:teuthology.orchestra.run.smithi092.stdout:Version: 5.4.0.121.122 2022-06-29T15:50:17.427 INFO:teuthology.orchestra.run.smithi092.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-06-29T15:50:17.427 INFO:teuthology.orchestra.run.smithi092.stdout:Depends: linux-image-5.4.0-121-generic, linux-modules-extra-5.4.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2022-06-29T15:50:17.427 INFO:teuthology.orchestra.run.smithi092.stdout:Recommends: thermald 2022-06-29T15:50:17.428 INFO:teuthology.orchestra.run.smithi092.stdout:Description: Generic Linux kernel image 2022-06-29T15:50:17.428 INFO:teuthology.orchestra.run.smithi092.stdout: This package will always depend on the latest generic kernel image 2022-06-29T15:50:17.428 INFO:teuthology.orchestra.run.smithi092.stdout: available. 2022-06-29T15:50:17.429 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-120-generic Expected: 5.4.0-121-generic 2022-06-29T15:50:17.429 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-06-29T15:50:17.429 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-06-29T15:50:17.429 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2022-06-29T15:50:17.430 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi092.front.sepia.ceph.com, path=None, version=distro) 2022-06-29T15:50:17.430 DEBUG:teuthology.orchestra.run.smithi092:> sudo apt-get clean 2022-06-29T15:50:17.520 DEBUG:teuthology.orchestra.run.smithi092:> sudo apt-get update 2022-06-29T15:50:17.552 DEBUG:teuthology.orchestra.run.smithi060:> dpkg -s linux-image-generic 2022-06-29T15:50:17.554 INFO:teuthology.orchestra.run.smithi197.stderr:Found linux image: /boot/vmlinuz-5.4.0-120-generic 2022-06-29T15:50:17.565 INFO:teuthology.orchestra.run.smithi197.stderr:Found initrd image: /boot/initrd.img-5.4.0-120-generic 2022-06-29T15:50:17.575 INFO:teuthology.orchestra.run.smithi060.stdout:Package: linux-image-generic 2022-06-29T15:50:17.576 INFO:teuthology.orchestra.run.smithi060.stdout:Status: install ok installed 2022-06-29T15:50:17.576 INFO:teuthology.orchestra.run.smithi060.stdout:Priority: optional 2022-06-29T15:50:17.576 INFO:teuthology.orchestra.run.smithi060.stdout:Section: kernel 2022-06-29T15:50:17.576 INFO:teuthology.orchestra.run.smithi060.stdout:Installed-Size: 19 2022-06-29T15:50:17.577 INFO:teuthology.orchestra.run.smithi060.stdout:Maintainer: Ubuntu Kernel Team 2022-06-29T15:50:17.577 INFO:teuthology.orchestra.run.smithi060.stdout:Architecture: amd64 2022-06-29T15:50:17.577 INFO:teuthology.orchestra.run.smithi060.stdout:Source: linux-meta 2022-06-29T15:50:17.577 INFO:teuthology.orchestra.run.smithi060.stdout:Version: 5.4.0.121.122 2022-06-29T15:50:17.578 INFO:teuthology.orchestra.run.smithi060.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-06-29T15:50:17.578 INFO:teuthology.orchestra.run.smithi060.stdout:Depends: linux-image-5.4.0-121-generic, linux-modules-extra-5.4.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2022-06-29T15:50:17.578 INFO:teuthology.orchestra.run.smithi060.stdout:Recommends: thermald 2022-06-29T15:50:17.578 INFO:teuthology.orchestra.run.smithi060.stdout:Description: Generic Linux kernel image 2022-06-29T15:50:17.579 INFO:teuthology.orchestra.run.smithi060.stdout: This package will always depend on the latest generic kernel image 2022-06-29T15:50:17.579 INFO:teuthology.orchestra.run.smithi060.stdout: available. 2022-06-29T15:50:17.579 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-120-generic Expected: 5.4.0-121-generic 2022-06-29T15:50:17.580 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-06-29T15:50:17.580 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-06-29T15:50:17.581 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2022-06-29T15:50:17.581 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi060.front.sepia.ceph.com, path=None, version=distro) 2022-06-29T15:50:17.581 DEBUG:teuthology.orchestra.run.smithi060:> sudo apt-get clean 2022-06-29T15:50:17.638 INFO:teuthology.orchestra.run.smithi156.stderr:Found linux image: /boot/vmlinuz-5.4.0-121-generic 2022-06-29T15:50:17.650 INFO:teuthology.orchestra.run.smithi156.stderr:Found initrd image: /boot/initrd.img-5.4.0-121-generic 2022-06-29T15:50:17.655 DEBUG:teuthology.orchestra.run.smithi060:> sudo apt-get update 2022-06-29T15:50:17.656 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-06-29T15:50:17.764 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-06-29T15:50:17.790 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-06-29T15:50:17.790 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-06-29T15:50:17.795 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-06-29T15:50:17.812 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-06-29T15:50:17.844 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-06-29T15:50:17.925 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-06-29T15:50:18.040 INFO:teuthology.orchestra.run.smithi156.stderr:Found linux image: /boot/vmlinuz-5.4.0-120-generic 2022-06-29T15:50:18.051 INFO:teuthology.orchestra.run.smithi156.stderr:Found initrd image: /boot/initrd.img-5.4.0-120-generic 2022-06-29T15:50:18.120 INFO:teuthology.orchestra.run.smithi121.stderr:Sourcing file `/etc/default/grub' 2022-06-29T15:50:18.121 INFO:teuthology.orchestra.run.smithi121.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-06-29T15:50:18.132 INFO:teuthology.orchestra.run.smithi121.stderr:Generating grub configuration file ... 2022-06-29T15:50:18.360 INFO:teuthology.orchestra.run.smithi121.stderr:Found linux image: /boot/vmlinuz-5.4.0-121-generic 2022-06-29T15:50:18.371 INFO:teuthology.orchestra.run.smithi121.stderr:Found initrd image: /boot/initrd.img-5.4.0-121-generic 2022-06-29T15:50:18.728 INFO:teuthology.orchestra.run.smithi121.stderr:Found linux image: /boot/vmlinuz-5.4.0-120-generic 2022-06-29T15:50:18.739 INFO:teuthology.orchestra.run.smithi121.stderr:Found initrd image: /boot/initrd.img-5.4.0-120-generic 2022-06-29T15:50:19.200 INFO:teuthology.orchestra.run.smithi197.stderr:done 2022-06-29T15:50:19.202 DEBUG:teuthology.orchestra.run.smithi197:> sudo shutdown -r now 2022-06-29T15:50:19.240 INFO:teuthology.orchestra.run.smithi092.stdout:Reading package lists... 2022-06-29T15:50:19.264 DEBUG:teuthology.orchestra.run.smithi092:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-06-29T15:50:19.334 INFO:teuthology.orchestra.run.smithi092.stdout:Reading package lists... 2022-06-29T15:50:19.414 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2022-06-29T15:50:19.437 DEBUG:teuthology.orchestra.run.smithi060:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-06-29T15:50:19.507 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2022-06-29T15:50:19.511 INFO:teuthology.orchestra.run.smithi092.stdout:Building dependency tree... 2022-06-29T15:50:19.512 INFO:teuthology.orchestra.run.smithi092.stdout:Reading state information... 2022-06-29T15:50:19.568 INFO:teuthology.orchestra.run.smithi156.stderr:done 2022-06-29T15:50:19.570 DEBUG:teuthology.orchestra.run.smithi156:> sudo shutdown -r now 2022-06-29T15:50:19.687 INFO:teuthology.orchestra.run.smithi060.stdout:Building dependency tree... 2022-06-29T15:50:19.687 INFO:teuthology.orchestra.run.smithi060.stdout:Reading state information... 2022-06-29T15:50:19.703 INFO:teuthology.orchestra.run.smithi092.stdout:linux-image-generic is already the newest version (5.4.0.121.122). 2022-06-29T15:50:19.703 INFO:teuthology.orchestra.run.smithi092.stdout:0 upgraded, 0 newly installed, 0 to remove and 17 not upgraded. 2022-06-29T15:50:19.704 DEBUG:teuthology.orchestra.run.smithi092:> dpkg -s linux-image-generic 2022-06-29T15:50:19.724 INFO:teuthology.orchestra.run.smithi092.stdout:Package: linux-image-generic 2022-06-29T15:50:19.724 INFO:teuthology.orchestra.run.smithi092.stdout:Status: install ok installed 2022-06-29T15:50:19.724 INFO:teuthology.orchestra.run.smithi092.stdout:Priority: optional 2022-06-29T15:50:19.725 INFO:teuthology.orchestra.run.smithi092.stdout:Section: kernel 2022-06-29T15:50:19.725 INFO:teuthology.orchestra.run.smithi092.stdout:Installed-Size: 19 2022-06-29T15:50:19.725 INFO:teuthology.orchestra.run.smithi092.stdout:Maintainer: Ubuntu Kernel Team 2022-06-29T15:50:19.725 INFO:teuthology.orchestra.run.smithi092.stdout:Architecture: amd64 2022-06-29T15:50:19.726 INFO:teuthology.orchestra.run.smithi092.stdout:Source: linux-meta 2022-06-29T15:50:19.726 INFO:teuthology.orchestra.run.smithi092.stdout:Version: 5.4.0.121.122 2022-06-29T15:50:19.726 INFO:teuthology.orchestra.run.smithi092.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-06-29T15:50:19.726 INFO:teuthology.orchestra.run.smithi092.stdout:Depends: linux-image-5.4.0-121-generic, linux-modules-extra-5.4.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2022-06-29T15:50:19.727 INFO:teuthology.orchestra.run.smithi092.stdout:Recommends: thermald 2022-06-29T15:50:19.727 INFO:teuthology.orchestra.run.smithi092.stdout:Description: Generic Linux kernel image 2022-06-29T15:50:19.727 INFO:teuthology.orchestra.run.smithi092.stdout: This package will always depend on the latest generic kernel image 2022-06-29T15:50:19.727 INFO:teuthology.orchestra.run.smithi092.stdout: available. 2022-06-29T15:50:19.728 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2022-06-29T15:50:19.732 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.xjZZBXE9pN 2022-06-29T15:50:19.733 DEBUG:teuthology.orchestra.run.smithi092:> sudo cp /boot/grub/grub.cfg /tmp/tmp.xjZZBXE9pN 2022-06-29T15:50:19.790 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.xjZZBXE9pN 2022-06-29T15:50:19.829 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.xjZZBXE9pN is 9KB 2022-06-29T15:50:19.844 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.xjZZBXE9pN 2022-06-29T15:50:19.851 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-06-29T15:50:19.887 INFO:teuthology.orchestra.run.smithi060.stdout:linux-image-generic is already the newest version (5.4.0.121.122). 2022-06-29T15:50:19.888 INFO:teuthology.orchestra.run.smithi060.stdout:0 upgraded, 0 newly installed, 0 to remove and 17 not upgraded. 2022-06-29T15:50:19.890 DEBUG:teuthology.orchestra.run.smithi060:> dpkg -s linux-image-generic 2022-06-29T15:50:19.905 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2022-06-29T15:50:19.906 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-06-29T15:50:19.907 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-06-29T15:50:19.912 INFO:teuthology.orchestra.run.smithi060.stdout:Package: linux-image-generic 2022-06-29T15:50:19.912 INFO:teuthology.orchestra.run.smithi060.stdout:Status: install ok installed 2022-06-29T15:50:19.912 INFO:teuthology.orchestra.run.smithi060.stdout:Priority: optional 2022-06-29T15:50:19.913 INFO:teuthology.orchestra.run.smithi060.stdout:Section: kernel 2022-06-29T15:50:19.913 INFO:teuthology.orchestra.run.smithi060.stdout:Installed-Size: 19 2022-06-29T15:50:19.913 INFO:teuthology.orchestra.run.smithi060.stdout:Maintainer: Ubuntu Kernel Team 2022-06-29T15:50:19.913 INFO:teuthology.orchestra.run.smithi060.stdout:Architecture: amd64 2022-06-29T15:50:19.914 INFO:teuthology.orchestra.run.smithi060.stdout:Source: linux-meta 2022-06-29T15:50:19.914 INFO:teuthology.orchestra.run.smithi060.stdout:Version: 5.4.0.121.122 2022-06-29T15:50:19.914 INFO:teuthology.orchestra.run.smithi060.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-06-29T15:50:19.914 INFO:teuthology.orchestra.run.smithi060.stdout:Depends: linux-image-5.4.0-121-generic, linux-modules-extra-5.4.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2022-06-29T15:50:19.915 INFO:teuthology.orchestra.run.smithi060.stdout:Recommends: thermald 2022-06-29T15:50:19.915 INFO:teuthology.orchestra.run.smithi060.stdout:Description: Generic Linux kernel image 2022-06-29T15:50:19.915 INFO:teuthology.orchestra.run.smithi060.stdout: This package will always depend on the latest generic kernel image 2022-06-29T15:50:19.915 INFO:teuthology.orchestra.run.smithi060.stdout: available. 2022-06-29T15:50:19.916 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2022-06-29T15:50:19.922 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.5YuOHMHLCg 2022-06-29T15:50:19.922 DEBUG:teuthology.orchestra.run.smithi060:> sudo cp /boot/grub/grub.cfg /tmp/tmp.5YuOHMHLCg 2022-06-29T15:50:19.959 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-121-generic 2022-06-29T15:50:19.959 DEBUG:teuthology.orchestra.run.smithi092:> sudo update-grub 2022-06-29T15:50:19.981 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.5YuOHMHLCg 2022-06-29T15:50:20.020 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.5YuOHMHLCg is 9KB 2022-06-29T15:50:20.033 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.5YuOHMHLCg 2022-06-29T15:50:20.040 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-06-29T15:50:20.097 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2022-06-29T15:50:20.097 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-06-29T15:50:20.097 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-06-29T15:50:20.145 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-121-generic 2022-06-29T15:50:20.145 DEBUG:teuthology.orchestra.run.smithi060:> sudo update-grub 2022-06-29T15:50:20.271 INFO:teuthology.orchestra.run.smithi121.stderr:done 2022-06-29T15:50:20.273 DEBUG:teuthology.orchestra.run.smithi121:> sudo shutdown -r now 2022-06-29T15:50:21.286 INFO:teuthology.orchestra.run.smithi092.stderr:Sourcing file `/etc/default/grub' 2022-06-29T15:50:21.295 INFO:teuthology.orchestra.run.smithi092.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-06-29T15:50:21.306 INFO:teuthology.orchestra.run.smithi092.stderr:Generating grub configuration file ... 2022-06-29T15:50:21.501 INFO:teuthology.orchestra.run.smithi060.stderr:Sourcing file `/etc/default/grub' 2022-06-29T15:50:21.503 INFO:teuthology.orchestra.run.smithi060.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-06-29T15:50:21.512 INFO:teuthology.orchestra.run.smithi060.stderr:Generating grub configuration file ... 2022-06-29T15:50:21.588 INFO:teuthology.orchestra.run.smithi092.stderr:Found linux image: /boot/vmlinuz-5.4.0-121-generic 2022-06-29T15:50:21.600 INFO:teuthology.orchestra.run.smithi092.stderr:Found initrd image: /boot/initrd.img-5.4.0-121-generic 2022-06-29T15:50:21.756 INFO:teuthology.orchestra.run.smithi060.stderr:Found linux image: /boot/vmlinuz-5.4.0-121-generic 2022-06-29T15:50:21.768 INFO:teuthology.orchestra.run.smithi060.stderr:Found initrd image: /boot/initrd.img-5.4.0-121-generic 2022-06-29T15:50:21.993 INFO:teuthology.orchestra.run.smithi092.stderr:Found linux image: /boot/vmlinuz-5.4.0-120-generic 2022-06-29T15:50:22.003 INFO:teuthology.orchestra.run.smithi092.stderr:Found initrd image: /boot/initrd.img-5.4.0-120-generic 2022-06-29T15:50:22.119 INFO:teuthology.orchestra.run.smithi060.stderr:Found linux image: /boot/vmlinuz-5.4.0-120-generic 2022-06-29T15:50:22.131 INFO:teuthology.orchestra.run.smithi060.stderr:Found initrd image: /boot/initrd.img-5.4.0-120-generic 2022-06-29T15:50:23.605 INFO:teuthology.orchestra.run.smithi092.stderr:done 2022-06-29T15:50:23.607 DEBUG:teuthology.orchestra.run.smithi092:> sudo shutdown -r now 2022-06-29T15:50:23.699 INFO:teuthology.orchestra.run.smithi060.stderr:done 2022-06-29T15:50:23.701 DEBUG:teuthology.orchestra.run.smithi060:> sudo shutdown -r now 2022-06-29T15:50:49.230 INFO:teuthology.task.kernel:Checking client client.2 for new kernel version... 2022-06-29T15:50:49.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi197.front.sepia.ceph.com' 2022-06-29T15:50:49.232 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi197.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-06-29T15:50:49.574 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2022-06-29T15:50:49.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2022-06-29T15:50:49.575 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-06-29T15:50:50.277 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2022-06-29T15:50:50.277 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2022-06-29T15:50:50.278 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-06-29T15:50:53.614 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2022-06-29T15:50:53.614 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2022-06-29T15:50:53.615 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-06-29T15:50:53.704 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2022-06-29T15:50:53.705 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2022-06-29T15:50:53.706 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-06-29T15:51:07.774 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.197 2022-06-29T15:51:08.030 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2022-06-29T15:51:08.798 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2022-06-29T15:51:12.122 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.92 2022-06-29T15:51:12.122 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.60 2022-06-29T15:51:37.790 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi197.front.sepia.ceph.com' 2022-06-29T15:51:37.791 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi197.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-06-29T15:51:38.032 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2022-06-29T15:51:38.033 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-06-29T15:51:38.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2022-06-29T15:51:38.800 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-06-29T15:51:42.126 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2022-06-29T15:51:42.127 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-06-29T15:51:42.128 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2022-06-29T15:51:42.129 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-06-29T15:51:49.242 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.60 2022-06-29T15:51:53.082 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.197 2022-06-29T15:51:56.410 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2022-06-29T15:51:57.178 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2022-06-29T15:51:58.458 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.92 2022-06-29T15:52:19.264 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2022-06-29T15:52:19.265 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-06-29T15:52:19.494 DEBUG:teuthology.orchestra.run.smithi060:> true 2022-06-29T15:52:20.113 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi060.front.sepia.ceph.com' 2022-06-29T15:52:20.113 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-121-generic"... 2022-06-29T15:52:20.114 DEBUG:teuthology.orchestra.run.smithi060:> uname -r 2022-06-29T15:52:20.121 INFO:teuthology.orchestra.run.smithi060.stdout:5.4.0-121-generic 2022-06-29T15:52:20.121 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-121-generic vs 5.4.0-121-generic 2022-06-29T15:52:20.121 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-06-29T15:52:20.122 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-06-29T15:52:21.122 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2022-06-29T15:52:21.123 DEBUG:teuthology.orchestra.run.smithi060:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-06-29T15:52:21.210 INFO:teuthology.orchestra.run.smithi060.stdout:ttyS1 2022-06-29T15:52:21.231 DEBUG:teuthology.parallel:result is None 2022-06-29T15:52:23.083 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi197.front.sepia.ceph.com' 2022-06-29T15:52:23.084 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi197.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-06-29T15:52:23.323 DEBUG:teuthology.orchestra.run.smithi197:> true 2022-06-29T15:52:23.836 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi197.front.sepia.ceph.com' 2022-06-29T15:52:23.836 INFO:teuthology.task.kernel:Checking kernel version of client.2, want "5.4.0-121-generic"... 2022-06-29T15:52:23.837 DEBUG:teuthology.orchestra.run.smithi197:> uname -r 2022-06-29T15:52:23.843 INFO:teuthology.orchestra.run.smithi197.stdout:5.4.0-121-generic 2022-06-29T15:52:23.844 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-121-generic vs 5.4.0-121-generic 2022-06-29T15:52:23.844 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-06-29T15:52:23.844 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-06-29T15:52:24.846 INFO:teuthology.task.kernel:Enabling kdb on client.2... 2022-06-29T15:52:24.846 DEBUG:teuthology.orchestra.run.smithi197:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-06-29T15:52:24.925 INFO:teuthology.orchestra.run.smithi197.stdout:ttyS1 2022-06-29T15:52:24.952 DEBUG:teuthology.parallel:result is None 2022-06-29T15:52:26.411 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2022-06-29T15:52:26.412 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-06-29T15:52:26.644 DEBUG:teuthology.orchestra.run.smithi156:> true 2022-06-29T15:52:27.179 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2022-06-29T15:52:27.180 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-06-29T15:52:27.282 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi156.front.sepia.ceph.com' 2022-06-29T15:52:27.283 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "5.4.0-121-generic"... 2022-06-29T15:52:27.283 DEBUG:teuthology.orchestra.run.smithi156:> uname -r 2022-06-29T15:52:27.289 INFO:teuthology.orchestra.run.smithi156.stdout:5.4.0-121-generic 2022-06-29T15:52:27.290 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-121-generic vs 5.4.0-121-generic 2022-06-29T15:52:27.290 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-06-29T15:52:27.290 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-06-29T15:52:27.404 DEBUG:teuthology.orchestra.run.smithi121:> true 2022-06-29T15:52:28.013 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2022-06-29T15:52:28.013 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.4.0-121-generic"... 2022-06-29T15:52:28.014 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2022-06-29T15:52:28.020 INFO:teuthology.orchestra.run.smithi121.stdout:5.4.0-121-generic 2022-06-29T15:52:28.021 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-121-generic vs 5.4.0-121-generic 2022-06-29T15:52:28.021 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-06-29T15:52:28.021 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-06-29T15:52:28.291 INFO:teuthology.task.kernel:Enabling kdb on client.1... 2022-06-29T15:52:28.291 DEBUG:teuthology.orchestra.run.smithi156:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-06-29T15:52:28.381 INFO:teuthology.orchestra.run.smithi156.stdout:ttyS1 2022-06-29T15:52:28.411 DEBUG:teuthology.parallel:result is None 2022-06-29T15:52:28.459 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2022-06-29T15:52:28.460 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-06-29T15:52:28.685 DEBUG:teuthology.orchestra.run.smithi092:> true 2022-06-29T15:52:29.022 INFO:teuthology.task.kernel:Enabling kdb on client.0... 2022-06-29T15:52:29.023 DEBUG:teuthology.orchestra.run.smithi121:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-06-29T15:52:29.097 INFO:teuthology.orchestra.run.smithi121.stdout:ttyS1 2022-06-29T15:52:29.126 DEBUG:teuthology.parallel:result is None 2022-06-29T15:52:29.223 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi092.front.sepia.ceph.com' 2022-06-29T15:52:29.223 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.4.0-121-generic"... 2022-06-29T15:52:29.224 DEBUG:teuthology.orchestra.run.smithi092:> uname -r 2022-06-29T15:52:29.230 INFO:teuthology.orchestra.run.smithi092.stdout:5.4.0-121-generic 2022-06-29T15:52:29.231 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-121-generic vs 5.4.0-121-generic 2022-06-29T15:52:29.231 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-06-29T15:52:29.231 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-06-29T15:52:30.233 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2022-06-29T15:52:30.233 DEBUG:teuthology.orchestra.run.smithi092:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-06-29T15:52:30.295 INFO:teuthology.orchestra.run.smithi092.stdout:ttyS1 2022-06-29T15:52:30.323 DEBUG:teuthology.parallel:result is None 2022-06-29T15:52:30.324 INFO:teuthology.run_tasks:Running task internal.base... 2022-06-29T15:52:30.332 INFO:teuthology.task.internal:Creating test directory... 2022-06-29T15:52:30.333 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-06-29T15:52:30.336 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-06-29T15:52:30.338 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-06-29T15:52:30.341 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-06-29T15:52:30.344 DEBUG:teuthology.orchestra.run.smithi197:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-06-29T15:52:30.353 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-06-29T15:52:30.384 INFO:teuthology.run_tasks:Running task internal.archive... 2022-06-29T15:52:30.435 INFO:teuthology.task.internal:Creating archive directory... 2022-06-29T15:52:30.436 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-06-29T15:52:30.439 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-06-29T15:52:30.441 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-06-29T15:52:30.443 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-06-29T15:52:30.446 DEBUG:teuthology.orchestra.run.smithi197:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-06-29T15:52:30.469 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-06-29T15:52:30.476 INFO:teuthology.task.internal:Enabling coredump saving... 2022-06-29T15:52:30.476 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-06-29T15:52:30.494 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-06-29T15:52:30.497 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-06-29T15:52:30.499 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-06-29T15:52:30.502 DEBUG:teuthology.orchestra.run.smithi197:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-06-29T15:52:30.516 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-06-29T15:52:30.520 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-06-29T15:52:30.520 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-06-29T15:52:30.527 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-06-29T15:52:30.527 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-06-29T15:52:30.530 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-06-29T15:52:30.531 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-06-29T15:52:30.533 INFO:teuthology.orchestra.run.smithi197.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-06-29T15:52:30.538 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-06-29T15:52:30.543 INFO:teuthology.orchestra.run.smithi197.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-06-29T15:52:30.545 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-06-29T15:52:30.552 INFO:teuthology.task.internal:Configuring sudo... 2022-06-29T15:52:30.553 DEBUG:teuthology.orchestra.run.smithi060:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-06-29T15:52:30.556 DEBUG:teuthology.orchestra.run.smithi092:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-06-29T15:52:30.578 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-06-29T15:52:30.581 DEBUG:teuthology.orchestra.run.smithi156:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-06-29T15:52:30.583 DEBUG:teuthology.orchestra.run.smithi197:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-06-29T15:52:30.600 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-06-29T15:52:30.610 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-06-29T15:52:30.611 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-06-29T15:52:30.614 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-06-29T15:52:30.617 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-06-29T15:52:30.641 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-06-29T15:52:30.643 DEBUG:teuthology.orchestra.run.smithi197:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-06-29T15:52:30.651 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-06-29T15:52:30.659 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-06-29T15:52:30.707 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2022-06-29T15:52:30.708 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-06-29T15:52:30.734 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-06-29T15:52:30.741 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-06-29T15:52:30.749 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2022-06-29T15:52:30.749 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-06-29T15:52:30.768 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-06-29T15:52:30.776 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-06-29T15:52:30.830 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-06-29T15:52:30.830 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-06-29T15:52:30.857 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-06-29T15:52:30.865 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-06-29T15:52:30.872 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2022-06-29T15:52:30.873 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-06-29T15:52:30.911 DEBUG:teuthology.orchestra.run.smithi197:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-06-29T15:52:30.919 DEBUG:teuthology.orchestra.run.smithi197:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-06-29T15:52:30.968 DEBUG:teuthology.orchestra.run.smithi197:> set -ex 2022-06-29T15:52:30.968 DEBUG:teuthology.orchestra.run.smithi197:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-06-29T15:52:30.988 DEBUG:teuthology.orchestra.run.smithi060:> sudo service rsyslog restart 2022-06-29T15:52:30.991 DEBUG:teuthology.orchestra.run.smithi092:> sudo service rsyslog restart 2022-06-29T15:52:30.993 DEBUG:teuthology.orchestra.run.smithi121:> sudo service rsyslog restart 2022-06-29T15:52:30.996 DEBUG:teuthology.orchestra.run.smithi156:> sudo service rsyslog restart 2022-06-29T15:52:30.998 DEBUG:teuthology.orchestra.run.smithi197:> sudo service rsyslog restart 2022-06-29T15:52:31.053 INFO:teuthology.run_tasks:Running task internal.timer... 2022-06-29T15:52:31.062 INFO:teuthology.task.internal:Starting timer... 2022-06-29T15:52:31.063 INFO:teuthology.run_tasks:Running task pcp... 2022-06-29T15:52:31.112 INFO:teuthology.run_tasks:Running task selinux... 2022-06-29T15:52:31.160 DEBUG:teuthology.task.selinux:Excluding smithi060: OS 'ubuntu' does not support SELinux 2022-06-29T15:52:31.161 DEBUG:teuthology.task.selinux:Excluding smithi092: OS 'ubuntu' does not support SELinux 2022-06-29T15:52:31.161 DEBUG:teuthology.task.selinux:Excluding smithi121: OS 'ubuntu' does not support SELinux 2022-06-29T15:52:31.161 DEBUG:teuthology.task.selinux:Excluding smithi156: OS 'ubuntu' does not support SELinux 2022-06-29T15:52:31.162 DEBUG:teuthology.task.selinux:Excluding smithi197: OS 'ubuntu' does not support SELinux 2022-06-29T15:52:31.162 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-06-29T15:52:31.162 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-06-29T15:52:31.162 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-06-29T15:52:31.163 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-06-29T15:52:31.203 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2022-06-29T15:52:31.211 INFO:teuthology.repo_utils:Fetching main from origin 2022-06-29T15:52:31.243 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main to origin/main 2022-06-29T15:52:31.263 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-06-29T15:52:31.265 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi060.front.sepia.ceph.com,smithi092.front.sepia.ceph.com,smithi121.front.sepia.ceph.com,smithi156.front.sepia.ceph.com,smithi197.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main/cephlab.yml 2022-06-29T15:55:25.644 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi060.front.sepia.ceph.com'), Remote(name='ubuntu@smithi092.front.sepia.ceph.com'), Remote(name='ubuntu@smithi121.front.sepia.ceph.com'), Remote(name='ubuntu@smithi156.front.sepia.ceph.com'), Remote(name='ubuntu@smithi197.front.sepia.ceph.com')] 2022-06-29T15:55:25.647 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2022-06-29T15:55:25.649 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-06-29T15:55:25.756 DEBUG:teuthology.orchestra.run.smithi060:> true 2022-06-29T15:55:25.881 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2022-06-29T15:55:25.881 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-06-29T15:55:25.977 DEBUG:teuthology.orchestra.run.smithi092:> true 2022-06-29T15:55:26.096 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2022-06-29T15:55:26.097 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-06-29T15:55:26.200 DEBUG:teuthology.orchestra.run.smithi121:> true 2022-06-29T15:55:26.321 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2022-06-29T15:55:26.322 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-06-29T15:55:26.431 DEBUG:teuthology.orchestra.run.smithi156:> true 2022-06-29T15:55:26.551 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi197.front.sepia.ceph.com' 2022-06-29T15:55:26.552 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi197.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-06-29T15:55:26.650 DEBUG:teuthology.orchestra.run.smithi197:> true 2022-06-29T15:55:26.772 INFO:teuthology.run_tasks:Running task clock... 2022-06-29T15:55:26.785 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-06-29T15:55:26.786 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-06-29T15:55:26.788 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-06-29T15:55:26.791 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-06-29T15:55:26.792 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-06-29T15:55:26.794 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-06-29T15:55:26.795 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-06-29T15:55:26.797 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-06-29T15:55:26.797 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-06-29T15:55:26.800 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-06-29T15:55:26.800 DEBUG:teuthology.orchestra.run.smithi197:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-06-29T15:55:26.821 INFO:teuthology.orchestra.run.smithi060.stdout:29 Jun 15:55:26 ntpd[8069]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-06-29T15:55:26.821 INFO:teuthology.orchestra.run.smithi060.stdout:29 Jun 15:55:26 ntpd[8069]: Command line: ntpd -gq 2022-06-29T15:55:26.822 INFO:teuthology.orchestra.run.smithi060.stdout:29 Jun 15:55:26 ntpd[8069]: proto: precision = 0.184 usec (-22) 2022-06-29T15:55:26.823 INFO:teuthology.orchestra.run.smithi060.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-06-29T15:55:26.823 INFO:teuthology.orchestra.run.smithi060.stdout:29 Jun 15:55:26 ntpd[8069]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-06-29T15:55:26.824 INFO:teuthology.orchestra.run.smithi060.stderr:restrict ::: KOD does nothing without LIMITED. 2022-06-29T15:55:26.825 INFO:teuthology.orchestra.run.smithi060.stdout:29 Jun 15:55:26 ntpd[8069]: restrict ::: KOD does nothing without LIMITED. 2022-06-29T15:55:26.826 INFO:teuthology.orchestra.run.smithi060.stdout:29 Jun 15:55:26 ntpd[8069]: Listen and drop on 0 v6wildcard [::]:123 2022-06-29T15:55:26.826 INFO:teuthology.orchestra.run.smithi060.stdout:29 Jun 15:55:26 ntpd[8069]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-06-29T15:55:26.826 INFO:teuthology.orchestra.run.smithi060.stdout:29 Jun 15:55:26 ntpd[8069]: Listen normally on 2 lo 127.0.0.1:123 2022-06-29T15:55:26.827 INFO:teuthology.orchestra.run.smithi060.stdout:29 Jun 15:55:26 ntpd[8069]: Listen normally on 3 ens1f0 172.21.15.60:123 2022-06-29T15:55:26.827 INFO:teuthology.orchestra.run.smithi060.stdout:29 Jun 15:55:26 ntpd[8069]: Listen normally on 4 lo [::1]:123 2022-06-29T15:55:26.827 INFO:teuthology.orchestra.run.smithi060.stdout:29 Jun 15:55:26 ntpd[8069]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1560%4]:123 2022-06-29T15:55:26.827 INFO:teuthology.orchestra.run.smithi060.stdout:29 Jun 15:55:26 ntpd[8069]: Listening on routing socket on fd #22 for interface updates 2022-06-29T15:55:26.828 INFO:teuthology.orchestra.run.smithi092.stdout:29 Jun 15:55:26 ntpd[8075]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-06-29T15:55:26.829 INFO:teuthology.orchestra.run.smithi092.stdout:29 Jun 15:55:26 ntpd[8075]: Command line: ntpd -gq 2022-06-29T15:55:26.829 INFO:teuthology.orchestra.run.smithi092.stdout:29 Jun 15:55:26 ntpd[8075]: proto: precision = 0.069 usec (-24) 2022-06-29T15:55:26.829 INFO:teuthology.orchestra.run.smithi121.stdout:29 Jun 15:55:26 ntpd[8090]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-06-29T15:55:26.830 INFO:teuthology.orchestra.run.smithi121.stdout:29 Jun 15:55:26 ntpd[8090]: Command line: ntpd -gq 2022-06-29T15:55:26.830 INFO:teuthology.orchestra.run.smithi121.stdout:29 Jun 15:55:26 ntpd[8090]: proto: precision = 0.128 usec (-23) 2022-06-29T15:55:26.830 INFO:teuthology.orchestra.run.smithi092.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-06-29T15:55:26.831 INFO:teuthology.orchestra.run.smithi121.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-06-29T15:55:26.831 INFO:teuthology.orchestra.run.smithi092.stdout:29 Jun 15:55:26 ntpd[8075]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-06-29T15:55:26.832 INFO:teuthology.orchestra.run.smithi121.stdout:29 Jun 15:55:26 ntpd[8090]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-06-29T15:55:26.833 INFO:teuthology.orchestra.run.smithi092.stderr:restrict ::: KOD does nothing without LIMITED. 2022-06-29T15:55:26.833 INFO:teuthology.orchestra.run.smithi121.stderr:restrict ::: KOD does nothing without LIMITED. 2022-06-29T15:55:26.836 INFO:teuthology.orchestra.run.smithi197.stdout:29 Jun 15:55:26 ntpd[8137]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-06-29T15:55:26.836 INFO:teuthology.orchestra.run.smithi197.stdout:29 Jun 15:55:26 ntpd[8137]: Command line: ntpd -gq 2022-06-29T15:55:26.836 INFO:teuthology.orchestra.run.smithi197.stdout:29 Jun 15:55:26 ntpd[8137]: proto: precision = 0.123 usec (-23) 2022-06-29T15:55:26.837 INFO:teuthology.orchestra.run.smithi156.stdout:29 Jun 15:55:26 ntpd[8051]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-06-29T15:55:26.837 INFO:teuthology.orchestra.run.smithi156.stdout:29 Jun 15:55:26 ntpd[8051]: Command line: ntpd -gq 2022-06-29T15:55:26.837 INFO:teuthology.orchestra.run.smithi156.stdout:29 Jun 15:55:26 ntpd[8051]: proto: precision = 0.090 usec (-23) 2022-06-29T15:55:26.838 INFO:teuthology.orchestra.run.smithi092.stdout:29 Jun 15:55:26 ntpd[8075]: restrict ::: KOD does nothing without LIMITED. 2022-06-29T15:55:26.838 INFO:teuthology.orchestra.run.smithi092.stdout:29 Jun 15:55:26 ntpd[8075]: Listen and drop on 0 v6wildcard [::]:123 2022-06-29T15:55:26.838 INFO:teuthology.orchestra.run.smithi092.stdout:29 Jun 15:55:26 ntpd[8075]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-06-29T15:55:26.839 INFO:teuthology.orchestra.run.smithi092.stdout:29 Jun 15:55:26 ntpd[8075]: Listen normally on 2 lo 127.0.0.1:123 2022-06-29T15:55:26.839 INFO:teuthology.orchestra.run.smithi092.stdout:29 Jun 15:55:26 ntpd[8075]: Listen normally on 3 enp3s0f1 172.21.15.92:123 2022-06-29T15:55:26.839 INFO:teuthology.orchestra.run.smithi092.stdout:29 Jun 15:55:26 ntpd[8075]: Listen normally on 4 lo [::1]:123 2022-06-29T15:55:26.839 INFO:teuthology.orchestra.run.smithi092.stdout:29 Jun 15:55:26 ntpd[8075]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:cd45%5]:123 2022-06-29T15:55:26.840 INFO:teuthology.orchestra.run.smithi092.stdout:29 Jun 15:55:26 ntpd[8075]: Listening on routing socket on fd #22 for interface updates 2022-06-29T15:55:26.840 INFO:teuthology.orchestra.run.smithi121.stdout:29 Jun 15:55:26 ntpd[8090]: restrict ::: KOD does nothing without LIMITED. 2022-06-29T15:55:26.840 INFO:teuthology.orchestra.run.smithi121.stdout:29 Jun 15:55:26 ntpd[8090]: Listen and drop on 0 v6wildcard [::]:123 2022-06-29T15:55:26.840 INFO:teuthology.orchestra.run.smithi121.stdout:29 Jun 15:55:26 ntpd[8090]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-06-29T15:55:26.840 INFO:teuthology.orchestra.run.smithi121.stdout:29 Jun 15:55:26 ntpd[8090]: Listen normally on 2 lo 127.0.0.1:123 2022-06-29T15:55:26.841 INFO:teuthology.orchestra.run.smithi121.stdout:29 Jun 15:55:26 ntpd[8090]: Listen normally on 3 enp3s0f1 172.21.15.121:123 2022-06-29T15:55:26.841 INFO:teuthology.orchestra.run.smithi121.stdout:29 Jun 15:55:26 ntpd[8090]: Listen normally on 4 lo [::1]:123 2022-06-29T15:55:26.841 INFO:teuthology.orchestra.run.smithi121.stdout:29 Jun 15:55:26 ntpd[8090]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:cba3%5]:123 2022-06-29T15:55:26.841 INFO:teuthology.orchestra.run.smithi121.stdout:29 Jun 15:55:26 ntpd[8090]: Listening on routing socket on fd #22 for interface updates 2022-06-29T15:55:26.842 INFO:teuthology.orchestra.run.smithi197.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-06-29T15:55:26.842 INFO:teuthology.orchestra.run.smithi156.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-06-29T15:55:26.843 INFO:teuthology.orchestra.run.smithi197.stdout:29 Jun 15:55:26 ntpd[8137]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-06-29T15:55:26.843 INFO:teuthology.orchestra.run.smithi197.stdout:29 Jun 15:55:26 ntpd[8137]: restrict ::: KOD does nothing without LIMITED. 2022-06-29T15:55:26.844 INFO:teuthology.orchestra.run.smithi197.stderr:restrict ::: KOD does nothing without LIMITED. 2022-06-29T15:55:26.844 INFO:teuthology.orchestra.run.smithi156.stdout:29 Jun 15:55:26 ntpd[8051]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-06-29T15:55:26.844 INFO:teuthology.orchestra.run.smithi156.stdout:29 Jun 15:55:26 ntpd[8051]: restrict ::: KOD does nothing without LIMITED. 2022-06-29T15:55:26.845 INFO:teuthology.orchestra.run.smithi156.stderr:restrict ::: KOD does nothing without LIMITED. 2022-06-29T15:55:26.848 INFO:teuthology.orchestra.run.smithi197.stdout:29 Jun 15:55:26 ntpd[8137]: Listen and drop on 0 v6wildcard [::]:123 2022-06-29T15:55:26.848 INFO:teuthology.orchestra.run.smithi197.stdout:29 Jun 15:55:26 ntpd[8137]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-06-29T15:55:26.848 INFO:teuthology.orchestra.run.smithi197.stdout:29 Jun 15:55:26 ntpd[8137]: Listen normally on 2 lo 127.0.0.1:123 2022-06-29T15:55:26.848 INFO:teuthology.orchestra.run.smithi197.stdout:29 Jun 15:55:26 ntpd[8137]: Listen normally on 3 enp3s0f1 172.21.15.197:123 2022-06-29T15:55:26.849 INFO:teuthology.orchestra.run.smithi197.stdout:29 Jun 15:55:26 ntpd[8137]: Listen normally on 4 lo [::1]:123 2022-06-29T15:55:26.849 INFO:teuthology.orchestra.run.smithi197.stdout:29 Jun 15:55:26 ntpd[8137]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7d1f%5]:123 2022-06-29T15:55:26.849 INFO:teuthology.orchestra.run.smithi197.stdout:29 Jun 15:55:26 ntpd[8137]: Listening on routing socket on fd #22 for interface updates 2022-06-29T15:55:26.849 INFO:teuthology.orchestra.run.smithi156.stdout:29 Jun 15:55:26 ntpd[8051]: Listen and drop on 0 v6wildcard [::]:123 2022-06-29T15:55:26.850 INFO:teuthology.orchestra.run.smithi156.stdout:29 Jun 15:55:26 ntpd[8051]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-06-29T15:55:26.850 INFO:teuthology.orchestra.run.smithi156.stdout:29 Jun 15:55:26 ntpd[8051]: Listen normally on 2 lo 127.0.0.1:123 2022-06-29T15:55:26.850 INFO:teuthology.orchestra.run.smithi156.stdout:29 Jun 15:55:26 ntpd[8051]: Listen normally on 3 enp3s0f1 172.21.15.156:123 2022-06-29T15:55:26.850 INFO:teuthology.orchestra.run.smithi156.stdout:29 Jun 15:55:26 ntpd[8051]: Listen normally on 4 lo [::1]:123 2022-06-29T15:55:26.851 INFO:teuthology.orchestra.run.smithi156.stdout:29 Jun 15:55:26 ntpd[8051]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6dfd%5]:123 2022-06-29T15:55:26.851 INFO:teuthology.orchestra.run.smithi156.stdout:29 Jun 15:55:26 ntpd[8051]: Listening on routing socket on fd #22 for interface updates 2022-06-29T15:55:27.822 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:27 ntpd[8069]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:27.822 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:27 ntpd[8069]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:27.825 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:27 ntpd[8090]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:27.825 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:27 ntpd[8090]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:27.826 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:27 ntpd[8090]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:27.826 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:27 ntpd[8090]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:27.826 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:27 ntpd[8090]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:27.826 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:27 ntpd[8090]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:27.827 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:27 ntpd[8090]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:27.827 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:27 ntpd[8090]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:27.828 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:27 ntpd[8075]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:27.828 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:27 ntpd[8075]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:27.829 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:27 ntpd[8075]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:27.829 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:27 ntpd[8075]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:27.829 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:27 ntpd[8075]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:27.830 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:27 ntpd[8075]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:27.833 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:27 ntpd[8051]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:27.833 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:27 ntpd[8051]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:27.834 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:27 ntpd[8051]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:27.834 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:27 ntpd[8051]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:27.834 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:27 ntpd[8051]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:27.834 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:27 ntpd[8051]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:27.835 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:27 ntpd[8137]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:27.836 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:27 ntpd[8137]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:27.836 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:27 ntpd[8137]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:27.836 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:27 ntpd[8137]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:27.837 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:27 ntpd[8137]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:27.837 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:27 ntpd[8137]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:28.833 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:28 ntpd[8051]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:28.834 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:28 ntpd[8051]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:28.834 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:28 ntpd[8051]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:28.834 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:28 ntpd[8051]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:28.834 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:28 ntpd[8051]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:28.835 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:28 ntpd[8051]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:28.835 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:28 ntpd[8137]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:28.836 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:28 ntpd[8137]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:29.827 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:29 ntpd[8090]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:29.827 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:29 ntpd[8090]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:29.828 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:29 ntpd[8090]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:29.828 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:29 ntpd[8090]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:29.828 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:29 ntpd[8090]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:29.828 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:29 ntpd[8090]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:29.829 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:29 ntpd[8090]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:29.829 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:29 ntpd[8090]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:29.830 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:29 ntpd[8090]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:29.830 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:29 ntpd[8090]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:29.830 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:29 ntpd[8090]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:29.831 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:29 ntpd[8090]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:29.831 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:29 ntpd[8090]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:29.831 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:29 ntpd[8090]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:29.831 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:29 ntpd[8090]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:29.831 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:29 ntpd[8090]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:29.832 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:29 ntpd[8090]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:29.832 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:29 ntpd[8090]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:29.833 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:29 ntpd[8075]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:29.833 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:29 ntpd[8075]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:29.833 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:29 ntpd[8075]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:29.834 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:29 ntpd[8075]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:29.834 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:29 ntpd[8075]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:29.834 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:29 ntpd[8075]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:29.837 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:29 ntpd[8137]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:29.838 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:29 ntpd[8137]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:29.838 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:29 ntpd[8137]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:29.838 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:29 ntpd[8137]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:29.838 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:29 ntpd[8137]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:29.838 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:29 ntpd[8137]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:29.839 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:29 ntpd[8137]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:29.839 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:29 ntpd[8137]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:29.839 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:29 ntpd[8137]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:29.839 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:29 ntpd[8137]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:29.840 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:29 ntpd[8137]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:29.840 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:29 ntpd[8137]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:29.840 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:29 ntpd[8051]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:29.840 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:29 ntpd[8051]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:29.840 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:29 ntpd[8051]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:29.841 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:29 ntpd[8051]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:29.841 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:29 ntpd[8051]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:29.841 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:29 ntpd[8051]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:29.842 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:29 ntpd[8051]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:29.842 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:29 ntpd[8051]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:29.842 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:29 ntpd[8051]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:29.842 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:29 ntpd[8051]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:29.842 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:29 ntpd[8051]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:29.843 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:29 ntpd[8051]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:30.823 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:30 ntpd[8069]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:30.823 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:30 ntpd[8069]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:30.823 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:30 ntpd[8069]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:30.823 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:30 ntpd[8069]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:30.824 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:30 ntpd[8069]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:30.824 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:30 ntpd[8069]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:30.824 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:30 ntpd[8069]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:30.824 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:30 ntpd[8069]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:30.825 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:30 ntpd[8069]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:30.825 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:30 ntpd[8069]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:30.825 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:30 ntpd[8069]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:30.825 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:30 ntpd[8069]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:30.828 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:30 ntpd[8075]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:30.829 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:30 ntpd[8075]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:30.829 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:30 ntpd[8075]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:30.829 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:30 ntpd[8075]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:30.829 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:30 ntpd[8075]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:30.829 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:30 ntpd[8075]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:30.830 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:30 ntpd[8075]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:30.830 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:30 ntpd[8075]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:30.833 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:30 ntpd[8051]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:30.833 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:30 ntpd[8051]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:30.834 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:30 ntpd[8051]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:30.834 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:30 ntpd[8051]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:30.834 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:30 ntpd[8051]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:30.834 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:30 ntpd[8051]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:30.835 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:30 ntpd[8137]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:30.836 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:30 ntpd[8137]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:30.836 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:30 ntpd[8137]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:30.836 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:30 ntpd[8137]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:30.836 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:30 ntpd[8137]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:30.837 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:30 ntpd[8137]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:31.822 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:31 ntpd[8069]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:31.822 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:31 ntpd[8069]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:31.823 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:31 ntpd[8069]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:31.823 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:31 ntpd[8069]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:31.823 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:31 ntpd[8069]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:31.823 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:31 ntpd[8069]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:31.826 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:31 ntpd[8090]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:31.827 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:31 ntpd[8090]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:31.827 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:31 ntpd[8090]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:31.827 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:31 ntpd[8090]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:31.827 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:31 ntpd[8090]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:31.827 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:31 ntpd[8090]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:31.828 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:31 ntpd[8090]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:31.828 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:31 ntpd[8090]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:31.828 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:31 ntpd[8090]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:31.829 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:31 ntpd[8090]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:31.829 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:31 ntpd[8090]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:31.829 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:31 ntpd[8090]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:31.829 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:31 ntpd[8090]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:31.830 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:31 ntpd[8090]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:31.830 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:31 ntpd[8090]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:31.830 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:31 ntpd[8090]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:31.830 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:31 ntpd[8090]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:31.831 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:31 ntpd[8090]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:31.832 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:31 ntpd[8075]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:31.832 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:31 ntpd[8075]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:31.833 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:31 ntpd[8075]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:31.833 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:31 ntpd[8075]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:31.833 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:31 ntpd[8075]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:31.834 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:31 ntpd[8075]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:31.834 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:31 ntpd[8075]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:31.834 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:31 ntpd[8075]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:31.834 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:31 ntpd[8075]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:31.834 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:31 ntpd[8075]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:31.835 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:31 ntpd[8075]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:31.835 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:31 ntpd[8075]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:31.838 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:31 ntpd[8137]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:31.838 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:31 ntpd[8137]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:31.839 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:31 ntpd[8137]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:31.839 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:31 ntpd[8137]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:31.839 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:31 ntpd[8137]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:31.839 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:31 ntpd[8137]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:31.839 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:31 ntpd[8137]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:31.840 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:31 ntpd[8137]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:31.840 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:31 ntpd[8137]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:31.840 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:31 ntpd[8137]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:31.840 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:31 ntpd[8137]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:31.841 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:31 ntpd[8137]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:31.841 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:31 ntpd[8051]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:31.841 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:31 ntpd[8051]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:31.842 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:31 ntpd[8051]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:31.842 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:31 ntpd[8051]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:31.842 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:31 ntpd[8051]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:31.842 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:31 ntpd[8051]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:31.843 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:31 ntpd[8051]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:31.843 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:31 ntpd[8051]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:31.843 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:31 ntpd[8051]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:31.843 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:31 ntpd[8051]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:31.844 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:31 ntpd[8051]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:31.844 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:31 ntpd[8051]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:31.844 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:31 ntpd[8051]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:31.844 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:31 ntpd[8051]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:32.823 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:32 ntpd[8069]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:32.823 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:32 ntpd[8069]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:32.823 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:32 ntpd[8069]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:32.824 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:32 ntpd[8069]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:32.824 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:32 ntpd[8069]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:32.824 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:32 ntpd[8069]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:32.824 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:32 ntpd[8069]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:32.824 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:32 ntpd[8069]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:32.825 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:32 ntpd[8069]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:32.825 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:32 ntpd[8069]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:32.825 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:32 ntpd[8069]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:32.825 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:32 ntpd[8069]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:32.828 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:32 ntpd[8075]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:32.828 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:32 ntpd[8075]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:32.829 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:32 ntpd[8075]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:32.829 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:32 ntpd[8075]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:32.829 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:32 ntpd[8075]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:32.829 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:32 ntpd[8075]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:32.834 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:32 ntpd[8051]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:32.834 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:32 ntpd[8051]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:32.834 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:32 ntpd[8051]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:32.835 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:32 ntpd[8051]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:32.835 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:32 ntpd[8051]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:32.835 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:32 ntpd[8051]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:32.835 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:32 ntpd[8137]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:32.836 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:32 ntpd[8137]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:32.836 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:32 ntpd[8137]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:32.836 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:32 ntpd[8137]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:32.836 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:32 ntpd[8137]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:32.836 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:32 ntpd[8137]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:33.822 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:33 ntpd[8069]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:33.822 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:33 ntpd[8069]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:33.823 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:33 ntpd[8069]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:33.823 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:33 ntpd[8069]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:33.823 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:33 ntpd[8069]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:33.823 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:33 ntpd[8069]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:33.826 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:33 ntpd[8090]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:33.826 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:33 ntpd[8090]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:33.827 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:33 ntpd[8090]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:33.827 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:33 ntpd[8090]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:33.827 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:33 ntpd[8090]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:33.827 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:33 ntpd[8090]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:33.828 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:33 ntpd[8090]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:33.828 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:33 ntpd[8090]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:33.828 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:33 ntpd[8090]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:33.828 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:33 ntpd[8090]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:33.829 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:33 ntpd[8090]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:33.829 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:33 ntpd[8090]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:33.829 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:33 ntpd[8090]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:33.829 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:33 ntpd[8090]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:33.830 INFO:teuthology.orchestra.run.smithi121.stdout:29 Jun 15:55:33 ntpd[8090]: ntpd: time slew +0.002104 s 2022-06-29T15:55:33.830 INFO:teuthology.orchestra.run.smithi121.stdout:ntpd: time slew +0.002104s 2022-06-29T15:55:33.830 INFO:teuthology.orchestra.run.smithi121.stderr:29 Jun 15:55:33 ntpd[8090]: can't open /var/log/ntpstats/loopstats.20220629: Permission denied 2022-06-29T15:55:33.832 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:33 ntpd[8075]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:33.832 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:33 ntpd[8075]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:33.833 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:33 ntpd[8075]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:33.833 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:33 ntpd[8075]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:33.833 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:33 ntpd[8075]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:33.834 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:33 ntpd[8075]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:33.834 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:33 ntpd[8075]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:33.834 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:33 ntpd[8075]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:33.834 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:33 ntpd[8075]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:33.835 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:33 ntpd[8075]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:33.835 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:33 ntpd[8075]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:33.835 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:33 ntpd[8075]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:33.835 INFO:teuthology.orchestra.run.smithi092.stdout:29 Jun 15:55:33 ntpd[8075]: ntpd: time slew +0.002721 s 2022-06-29T15:55:33.836 INFO:teuthology.orchestra.run.smithi092.stdout:ntpd: time slew +0.002721s 2022-06-29T15:55:33.836 INFO:teuthology.orchestra.run.smithi092.stderr:29 Jun 15:55:33 ntpd[8075]: can't open /var/log/ntpstats/loopstats.20220629: Permission denied 2022-06-29T15:55:33.839 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:33 ntpd[8137]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:33.839 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:33 ntpd[8137]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:33.839 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:33 ntpd[8137]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:33.839 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:33 ntpd[8137]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:33.840 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:33 ntpd[8137]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:33.840 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:33 ntpd[8137]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:33.840 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:33 ntpd[8137]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:33.840 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:33 ntpd[8137]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:33.841 INFO:teuthology.orchestra.run.smithi197.stdout:29 Jun 15:55:33 ntpd[8137]: ntpd: time slew +0.001788 s 2022-06-29T15:55:33.841 INFO:teuthology.orchestra.run.smithi197.stdout:ntpd: time slew +0.001788s 2022-06-29T15:55:33.842 INFO:teuthology.orchestra.run.smithi197.stderr:29 Jun 15:55:33 ntpd[8137]: can't open /var/log/ntpstats/loopstats.20220629: Permission denied 2022-06-29T15:55:33.843 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:33 ntpd[8051]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:33.843 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:33 ntpd[8051]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:33.843 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:33 ntpd[8051]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:33.843 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:33 ntpd[8051]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:33.844 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:33 ntpd[8051]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:33.844 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:33 ntpd[8051]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:33.844 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:33 ntpd[8051]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:33.844 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:33 ntpd[8051]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:33.845 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:33 ntpd[8051]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:33.845 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:33 ntpd[8051]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:33.845 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:33 ntpd[8051]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:33.845 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:33 ntpd[8051]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:33.846 INFO:teuthology.orchestra.run.smithi156.stdout:29 Jun 15:55:33 ntpd[8051]: ntpd: time slew -0.000249 s 2022-06-29T15:55:33.846 INFO:teuthology.orchestra.run.smithi156.stdout:ntpd: time slew -0.000249s 2022-06-29T15:55:33.847 INFO:teuthology.orchestra.run.smithi156.stderr:29 Jun 15:55:33 ntpd[8051]: can't open /var/log/ntpstats/loopstats.20220629: Permission denied 2022-06-29T15:55:33.886 INFO:teuthology.orchestra.run.smithi121.stdout: remote refid st t when poll reach delay offset jitter 2022-06-29T15:55:33.887 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================== 2022-06-29T15:55:33.887 INFO:teuthology.orchestra.run.smithi121.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-06-29T15:55:33.887 INFO:teuthology.orchestra.run.smithi121.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-06-29T15:55:33.888 INFO:teuthology.orchestra.run.smithi121.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-06-29T15:55:33.888 INFO:teuthology.orchestra.run.smithi121.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-06-29T15:55:33.890 INFO:teuthology.orchestra.run.smithi156.stdout: remote refid st t when poll reach delay offset jitter 2022-06-29T15:55:33.890 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================== 2022-06-29T15:55:33.890 INFO:teuthology.orchestra.run.smithi156.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-06-29T15:55:33.890 INFO:teuthology.orchestra.run.smithi156.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-06-29T15:55:33.891 INFO:teuthology.orchestra.run.smithi156.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-06-29T15:55:33.891 INFO:teuthology.orchestra.run.smithi156.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-06-29T15:55:33.891 INFO:teuthology.orchestra.run.smithi092.stdout: remote refid st t when poll reach delay offset jitter 2022-06-29T15:55:33.891 INFO:teuthology.orchestra.run.smithi092.stdout:============================================================================== 2022-06-29T15:55:33.892 INFO:teuthology.orchestra.run.smithi092.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-06-29T15:55:33.892 INFO:teuthology.orchestra.run.smithi092.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-06-29T15:55:33.892 INFO:teuthology.orchestra.run.smithi092.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-06-29T15:55:33.892 INFO:teuthology.orchestra.run.smithi092.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-06-29T15:55:33.894 INFO:teuthology.orchestra.run.smithi197.stdout: remote refid st t when poll reach delay offset jitter 2022-06-29T15:55:33.894 INFO:teuthology.orchestra.run.smithi197.stdout:============================================================================== 2022-06-29T15:55:33.895 INFO:teuthology.orchestra.run.smithi197.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-06-29T15:55:33.895 INFO:teuthology.orchestra.run.smithi197.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-06-29T15:55:33.895 INFO:teuthology.orchestra.run.smithi197.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-06-29T15:55:33.896 INFO:teuthology.orchestra.run.smithi197.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-06-29T15:55:34.823 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:34 ntpd[8069]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:34.823 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:34 ntpd[8069]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:34.823 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:34 ntpd[8069]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:34.823 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:34 ntpd[8069]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:34.824 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:34 ntpd[8069]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:34.824 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:34 ntpd[8069]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:34.824 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:34 ntpd[8069]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:34.824 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:34 ntpd[8069]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:34.825 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:34 ntpd[8069]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:34.825 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:34 ntpd[8069]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:34.825 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:34 ntpd[8069]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:34.825 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:34 ntpd[8069]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:35.822 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:35 ntpd[8069]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:35.822 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:35 ntpd[8069]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:35.822 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:35 ntpd[8069]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:35.823 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:35 ntpd[8069]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:35.823 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:35 ntpd[8069]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:35.823 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:35 ntpd[8069]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:36.822 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:36 ntpd[8069]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:36.823 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:36 ntpd[8069]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:36.823 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:36 ntpd[8069]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:36.823 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:36 ntpd[8069]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:36.823 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:36 ntpd[8069]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-06-29T15:55:36.824 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:36 ntpd[8069]: can't open /var/log/ntpstats/rawstats.20220629: Permission denied 2022-06-29T15:55:36.824 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:36 ntpd[8069]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-06-29T15:55:36.824 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:36 ntpd[8069]: can't open /var/log/ntpstats/peerstats.20220629: Permission denied 2022-06-29T15:55:36.825 INFO:teuthology.orchestra.run.smithi060.stdout:29 Jun 15:55:36 ntpd[8069]: ntpd: time slew -0.003475 s 2022-06-29T15:55:36.825 INFO:teuthology.orchestra.run.smithi060.stdout:ntpd: time slew -0.003475s 2022-06-29T15:55:36.825 INFO:teuthology.orchestra.run.smithi060.stderr:29 Jun 15:55:36 ntpd[8069]: can't open /var/log/ntpstats/loopstats.20220629: Permission denied 2022-06-29T15:55:36.882 INFO:teuthology.orchestra.run.smithi060.stdout: remote refid st t when poll reach delay offset jitter 2022-06-29T15:55:36.883 INFO:teuthology.orchestra.run.smithi060.stdout:============================================================================== 2022-06-29T15:55:36.883 INFO:teuthology.orchestra.run.smithi060.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-06-29T15:55:36.883 INFO:teuthology.orchestra.run.smithi060.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-06-29T15:55:36.884 INFO:teuthology.orchestra.run.smithi060.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-06-29T15:55:36.884 INFO:teuthology.orchestra.run.smithi060.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-06-29T15:55:36.885 INFO:teuthology.run_tasks:Running task install... 2022-06-29T15:55:36.896 DEBUG:teuthology.task.install:project ceph 2022-06-29T15:55:36.897 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'f25ffac72e5a9e63860ff7deb6767a11f0559a16'}} 2022-06-29T15:55:36.897 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': 'f25ffac72e5a9e63860ff7deb6767a11f0559a16'} 2022-06-29T15:55:36.898 INFO:teuthology.task.install:Using flavor: default 2022-06-29T15:55:36.904 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-06-29T15:55:36.905 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']} 2022-06-29T15:55:36.906 DEBUG:teuthology.orchestra.run.smithi060:> sudo lsb_release -is 2022-06-29T15:55:36.955 INFO:teuthology.orchestra.run.smithi060.stdout:Ubuntu 2022-06-29T15:55:36.961 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-06-29T15:55:36.961 DEBUG:teuthology.orchestra.run.smithi092:> sudo lsb_release -is 2022-06-29T15:55:36.962 DEBUG:teuthology.orchestra.run.smithi060:> sudo apt-key list | grep Ceph 2022-06-29T15:55:37.013 INFO:teuthology.orchestra.run.smithi092.stdout:Ubuntu 2022-06-29T15:55:37.019 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-06-29T15:55:37.019 DEBUG:teuthology.orchestra.run.smithi121:> sudo lsb_release -is 2022-06-29T15:55:37.020 DEBUG:teuthology.orchestra.run.smithi092:> sudo apt-key list | grep Ceph 2022-06-29T15:55:37.059 INFO:teuthology.orchestra.run.smithi060.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-06-29T15:55:37.071 INFO:teuthology.orchestra.run.smithi121.stdout:Ubuntu 2022-06-29T15:55:37.077 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-06-29T15:55:37.078 DEBUG:teuthology.orchestra.run.smithi156:> sudo lsb_release -is 2022-06-29T15:55:37.079 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-key list | grep Ceph 2022-06-29T15:55:37.118 INFO:teuthology.orchestra.run.smithi060.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-06-29T15:55:37.118 INFO:teuthology.orchestra.run.smithi060.stdout:uid [ unknown] Ceph.com (release key) 2022-06-29T15:55:37.119 DEBUG:teuthology.orchestra.run.smithi060:> sudo lsb_release -is 2022-06-29T15:55:37.121 INFO:teuthology.orchestra.run.smithi092.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-06-29T15:55:37.129 INFO:teuthology.orchestra.run.smithi156.stdout:Ubuntu 2022-06-29T15:55:37.135 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-06-29T15:55:37.135 DEBUG:teuthology.orchestra.run.smithi197:> sudo lsb_release -is 2022-06-29T15:55:37.136 DEBUG:teuthology.orchestra.run.smithi156:> sudo apt-key list | grep Ceph 2022-06-29T15:55:37.170 INFO:teuthology.orchestra.run.smithi121.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-06-29T15:55:37.170 INFO:teuthology.orchestra.run.smithi060.stdout:Ubuntu 2022-06-29T15:55:37.176 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-06-29T15:55:37.177 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell on remote deb x86_64 2022-06-29T15:55:37.177 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev on remote deb x86_64 2022-06-29T15:55:37.178 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=f25ffac72e5a9e63860ff7deb6767a11f0559a16 2022-06-29T15:55:37.187 INFO:teuthology.orchestra.run.smithi092.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-06-29T15:55:37.187 INFO:teuthology.orchestra.run.smithi092.stdout:uid [ unknown] Ceph.com (release key) 2022-06-29T15:55:37.188 DEBUG:teuthology.orchestra.run.smithi092:> sudo lsb_release -is 2022-06-29T15:55:37.189 INFO:teuthology.orchestra.run.smithi197.stdout:Ubuntu 2022-06-29T15:55:37.192 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-06-29T15:55:37.193 DEBUG:teuthology.orchestra.run.smithi197:> sudo apt-key list | grep Ceph 2022-06-29T15:55:37.232 INFO:teuthology.orchestra.run.smithi121.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-06-29T15:55:37.233 INFO:teuthology.orchestra.run.smithi121.stdout:uid [ unknown] Ceph.com (release key) 2022-06-29T15:55:37.234 INFO:teuthology.orchestra.run.smithi156.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-06-29T15:55:37.235 DEBUG:teuthology.orchestra.run.smithi121:> sudo lsb_release -is 2022-06-29T15:55:37.236 INFO:teuthology.orchestra.run.smithi092.stdout:Ubuntu 2022-06-29T15:55:37.240 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-06-29T15:55:37.240 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell on remote deb x86_64 2022-06-29T15:55:37.240 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev on remote deb x86_64 2022-06-29T15:55:37.241 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=f25ffac72e5a9e63860ff7deb6767a11f0559a16 2022-06-29T15:55:37.283 INFO:teuthology.orchestra.run.smithi121.stdout:Ubuntu 2022-06-29T15:55:37.288 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-06-29T15:55:37.289 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell on remote deb x86_64 2022-06-29T15:55:37.289 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev on remote deb x86_64 2022-06-29T15:55:37.290 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=f25ffac72e5a9e63860ff7deb6767a11f0559a16 2022-06-29T15:55:37.293 INFO:teuthology.orchestra.run.smithi197.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-06-29T15:55:37.294 INFO:teuthology.orchestra.run.smithi156.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-06-29T15:55:37.294 INFO:teuthology.orchestra.run.smithi156.stdout:uid [ unknown] Ceph.com (release key) 2022-06-29T15:55:37.295 DEBUG:teuthology.orchestra.run.smithi156:> sudo lsb_release -is 2022-06-29T15:55:37.346 INFO:teuthology.orchestra.run.smithi156.stdout:Ubuntu 2022-06-29T15:55:37.350 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default/ 2022-06-29T15:55:37.351 INFO:teuthology.task.install.deb:Package version is 15.2.16-304-gf25ffac7-1focal 2022-06-29T15:55:37.357 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-06-29T15:55:37.358 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell on remote deb x86_64 2022-06-29T15:55:37.358 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev on remote deb x86_64 2022-06-29T15:55:37.359 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=f25ffac72e5a9e63860ff7deb6767a11f0559a16 2022-06-29T15:55:37.360 INFO:teuthology.orchestra.run.smithi197.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-06-29T15:55:37.360 INFO:teuthology.orchestra.run.smithi197.stdout:uid [ unknown] Ceph.com (release key) 2022-06-29T15:55:37.361 DEBUG:teuthology.orchestra.run.smithi197:> sudo lsb_release -is 2022-06-29T15:55:37.412 INFO:teuthology.orchestra.run.smithi197.stdout:Ubuntu 2022-06-29T15:55:37.417 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-06-29T15:55:37.418 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell on remote deb x86_64 2022-06-29T15:55:37.418 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev on remote deb x86_64 2022-06-29T15:55:37.419 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=f25ffac72e5a9e63860ff7deb6767a11f0559a16 2022-06-29T15:55:37.424 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default/ 2022-06-29T15:55:37.424 INFO:teuthology.task.install.deb:Package version is 15.2.16-304-gf25ffac7-1focal 2022-06-29T15:55:37.461 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default/ 2022-06-29T15:55:37.462 INFO:teuthology.task.install.deb:Package version is 15.2.16-304-gf25ffac7-1focal 2022-06-29T15:55:37.487 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2022-06-29T15:55:37.488 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-06-29T15:55:37.514 DEBUG:teuthology.orchestra.run.smithi060:> sudo apt-get update 2022-06-29T15:55:37.535 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default/ 2022-06-29T15:55:37.535 INFO:teuthology.task.install.deb:Package version is 15.2.16-304-gf25ffac7-1focal 2022-06-29T15:55:37.540 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2022-06-29T15:55:37.541 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-06-29T15:55:37.558 DEBUG:teuthology.orchestra.run.smithi092:> sudo apt-get update 2022-06-29T15:55:37.571 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default/ 2022-06-29T15:55:37.571 INFO:teuthology.task.install.deb:Package version is 15.2.16-304-gf25ffac7-1focal 2022-06-29T15:55:37.576 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-06-29T15:55:37.576 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-06-29T15:55:37.595 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-get update 2022-06-29T15:55:37.653 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2022-06-29T15:55:37.653 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-06-29T15:55:37.670 DEBUG:teuthology.orchestra.run.smithi156:> sudo apt-get update 2022-06-29T15:55:37.704 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-06-29T15:55:37.738 DEBUG:teuthology.orchestra.run.smithi197:> set -ex 2022-06-29T15:55:37.739 DEBUG:teuthology.orchestra.run.smithi197:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-06-29T15:55:37.756 DEBUG:teuthology.orchestra.run.smithi197:> sudo apt-get update 2022-06-29T15:55:37.788 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-06-29T15:55:37.814 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-06-29T15:55:37.818 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-06-29T15:55:37.819 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-06-29T15:55:37.849 INFO:teuthology.orchestra.run.smithi121.stdout:Ign:2 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal InRelease 2022-06-29T15:55:37.851 INFO:teuthology.orchestra.run.smithi060.stdout:Ign:3 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal InRelease 2022-06-29T15:55:37.852 INFO:teuthology.orchestra.run.smithi092.stdout:Ign:3 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal InRelease 2022-06-29T15:55:37.861 INFO:teuthology.orchestra.run.smithi156.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-06-29T15:55:37.870 INFO:teuthology.orchestra.run.smithi156.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-06-29T15:55:37.872 INFO:teuthology.orchestra.run.smithi121.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal Release [7,680 B] 2022-06-29T15:55:37.874 INFO:teuthology.orchestra.run.smithi060.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal Release [7,680 B] 2022-06-29T15:55:37.876 INFO:teuthology.orchestra.run.smithi092.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal Release [7,680 B] 2022-06-29T15:55:37.888 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-06-29T15:55:37.890 INFO:teuthology.orchestra.run.smithi156.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-06-29T15:55:37.894 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-06-29T15:55:37.897 INFO:teuthology.orchestra.run.smithi121.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal Release.gpg 2022-06-29T15:55:37.897 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal InRelease 2022-06-29T15:55:37.899 INFO:teuthology.orchestra.run.smithi060.stdout:Ign:6 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal Release.gpg 2022-06-29T15:55:37.902 INFO:teuthology.orchestra.run.smithi092.stdout:Ign:6 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal Release.gpg 2022-06-29T15:55:37.905 INFO:teuthology.orchestra.run.smithi197.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-06-29T15:55:37.914 INFO:teuthology.orchestra.run.smithi197.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-06-29T15:55:37.918 INFO:teuthology.orchestra.run.smithi156.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal InRelease 2022-06-29T15:55:37.920 INFO:teuthology.orchestra.run.smithi121.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 Packages [15.8 kB] 2022-06-29T15:55:37.922 INFO:teuthology.orchestra.run.smithi060.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main i386 Packages [3,062 B] 2022-06-29T15:55:37.925 INFO:teuthology.orchestra.run.smithi092.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 Packages [15.8 kB] 2022-06-29T15:55:37.934 INFO:teuthology.orchestra.run.smithi197.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-06-29T15:55:37.942 INFO:teuthology.orchestra.run.smithi156.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal Release [7,680 B] 2022-06-29T15:55:37.960 INFO:teuthology.orchestra.run.smithi060.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 Packages [15.8 kB] 2022-06-29T15:55:37.963 INFO:teuthology.orchestra.run.smithi197.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal InRelease 2022-06-29T15:55:37.964 INFO:teuthology.orchestra.run.smithi092.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main i386 Packages [3,062 B] 2022-06-29T15:55:37.967 INFO:teuthology.orchestra.run.smithi121.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main i386 Packages [3,062 B] 2022-06-29T15:55:37.967 INFO:teuthology.orchestra.run.smithi156.stdout:Ign:6 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal Release.gpg 2022-06-29T15:55:37.968 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-06-29T15:55:37.970 INFO:teuthology.orchestra.run.smithi156.stdout:Hit:7 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-06-29T15:55:37.974 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-06-29T15:55:37.977 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-06-29T15:55:37.986 INFO:teuthology.orchestra.run.smithi197.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal Release [7,680 B] 2022-06-29T15:55:37.990 INFO:teuthology.orchestra.run.smithi156.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 Packages [15.8 kB] 2022-06-29T15:55:38.011 INFO:teuthology.orchestra.run.smithi197.stdout:Ign:6 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal Release.gpg 2022-06-29T15:55:38.015 INFO:teuthology.orchestra.run.smithi197.stdout:Hit:7 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-06-29T15:55:38.035 INFO:teuthology.orchestra.run.smithi197.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main i386 Packages [3,062 B] 2022-06-29T15:55:38.042 INFO:teuthology.orchestra.run.smithi156.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main i386 Packages [3,062 B] 2022-06-29T15:55:38.061 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-06-29T15:55:38.083 INFO:teuthology.orchestra.run.smithi197.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 Packages [15.8 kB] 2022-06-29T15:55:38.380 INFO:teuthology.orchestra.run.smithi060.stdout:Fetched 26.5 kB in 1s (40.4 kB/s) 2022-06-29T15:55:38.419 INFO:teuthology.orchestra.run.smithi121.stdout:Fetched 26.5 kB in 1s (43.2 kB/s) 2022-06-29T15:55:38.469 INFO:teuthology.orchestra.run.smithi092.stdout:Fetched 26.5 kB in 1s (35.2 kB/s) 2022-06-29T15:55:38.526 INFO:teuthology.orchestra.run.smithi156.stdout:Fetched 26.5 kB in 1s (41.2 kB/s) 2022-06-29T15:55:38.552 INFO:teuthology.orchestra.run.smithi197.stdout:Fetched 26.5 kB in 1s (41.9 kB/s) 2022-06-29T15:55:39.292 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2022-06-29T15:55:39.294 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2022-06-29T15:55:39.316 DEBUG:teuthology.orchestra.run.smithi060:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.16-304-gf25ffac7-1focal cephadm=15.2.16-304-gf25ffac7-1focal ceph-mds=15.2.16-304-gf25ffac7-1focal ceph-mgr=15.2.16-304-gf25ffac7-1focal ceph-common=15.2.16-304-gf25ffac7-1focal ceph-fuse=15.2.16-304-gf25ffac7-1focal ceph-test=15.2.16-304-gf25ffac7-1focal radosgw=15.2.16-304-gf25ffac7-1focal python3-rados=15.2.16-304-gf25ffac7-1focal python3-rgw=15.2.16-304-gf25ffac7-1focal python3-cephfs=15.2.16-304-gf25ffac7-1focal python3-rbd=15.2.16-304-gf25ffac7-1focal libcephfs2=15.2.16-304-gf25ffac7-1focal libcephfs-dev=15.2.16-304-gf25ffac7-1focal librados2=15.2.16-304-gf25ffac7-1focal librbd1=15.2.16-304-gf25ffac7-1focal rbd-fuse=15.2.16-304-gf25ffac7-1focal python3-cephfs=15.2.16-304-gf25ffac7-1focal cephfs-shell=15.2.16-304-gf25ffac7-1focal 2022-06-29T15:55:39.318 DEBUG:teuthology.orchestra.run.smithi121:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.16-304-gf25ffac7-1focal cephadm=15.2.16-304-gf25ffac7-1focal ceph-mds=15.2.16-304-gf25ffac7-1focal ceph-mgr=15.2.16-304-gf25ffac7-1focal ceph-common=15.2.16-304-gf25ffac7-1focal ceph-fuse=15.2.16-304-gf25ffac7-1focal ceph-test=15.2.16-304-gf25ffac7-1focal radosgw=15.2.16-304-gf25ffac7-1focal python3-rados=15.2.16-304-gf25ffac7-1focal python3-rgw=15.2.16-304-gf25ffac7-1focal python3-cephfs=15.2.16-304-gf25ffac7-1focal python3-rbd=15.2.16-304-gf25ffac7-1focal libcephfs2=15.2.16-304-gf25ffac7-1focal libcephfs-dev=15.2.16-304-gf25ffac7-1focal librados2=15.2.16-304-gf25ffac7-1focal librbd1=15.2.16-304-gf25ffac7-1focal rbd-fuse=15.2.16-304-gf25ffac7-1focal python3-cephfs=15.2.16-304-gf25ffac7-1focal cephfs-shell=15.2.16-304-gf25ffac7-1focal 2022-06-29T15:55:39.343 INFO:teuthology.orchestra.run.smithi092.stdout:Reading package lists... 2022-06-29T15:55:39.365 DEBUG:teuthology.orchestra.run.smithi092:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.16-304-gf25ffac7-1focal cephadm=15.2.16-304-gf25ffac7-1focal ceph-mds=15.2.16-304-gf25ffac7-1focal ceph-mgr=15.2.16-304-gf25ffac7-1focal ceph-common=15.2.16-304-gf25ffac7-1focal ceph-fuse=15.2.16-304-gf25ffac7-1focal ceph-test=15.2.16-304-gf25ffac7-1focal radosgw=15.2.16-304-gf25ffac7-1focal python3-rados=15.2.16-304-gf25ffac7-1focal python3-rgw=15.2.16-304-gf25ffac7-1focal python3-cephfs=15.2.16-304-gf25ffac7-1focal python3-rbd=15.2.16-304-gf25ffac7-1focal libcephfs2=15.2.16-304-gf25ffac7-1focal libcephfs-dev=15.2.16-304-gf25ffac7-1focal librados2=15.2.16-304-gf25ffac7-1focal librbd1=15.2.16-304-gf25ffac7-1focal rbd-fuse=15.2.16-304-gf25ffac7-1focal python3-cephfs=15.2.16-304-gf25ffac7-1focal cephfs-shell=15.2.16-304-gf25ffac7-1focal 2022-06-29T15:55:39.384 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2022-06-29T15:55:39.387 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2022-06-29T15:55:39.418 INFO:teuthology.orchestra.run.smithi156.stdout:Reading package lists... 2022-06-29T15:55:39.428 INFO:teuthology.orchestra.run.smithi197.stdout:Reading package lists... 2022-06-29T15:55:39.430 INFO:teuthology.orchestra.run.smithi092.stdout:Reading package lists... 2022-06-29T15:55:39.441 DEBUG:teuthology.orchestra.run.smithi156:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.16-304-gf25ffac7-1focal cephadm=15.2.16-304-gf25ffac7-1focal ceph-mds=15.2.16-304-gf25ffac7-1focal ceph-mgr=15.2.16-304-gf25ffac7-1focal ceph-common=15.2.16-304-gf25ffac7-1focal ceph-fuse=15.2.16-304-gf25ffac7-1focal ceph-test=15.2.16-304-gf25ffac7-1focal radosgw=15.2.16-304-gf25ffac7-1focal python3-rados=15.2.16-304-gf25ffac7-1focal python3-rgw=15.2.16-304-gf25ffac7-1focal python3-cephfs=15.2.16-304-gf25ffac7-1focal python3-rbd=15.2.16-304-gf25ffac7-1focal libcephfs2=15.2.16-304-gf25ffac7-1focal libcephfs-dev=15.2.16-304-gf25ffac7-1focal librados2=15.2.16-304-gf25ffac7-1focal librbd1=15.2.16-304-gf25ffac7-1focal rbd-fuse=15.2.16-304-gf25ffac7-1focal python3-cephfs=15.2.16-304-gf25ffac7-1focal cephfs-shell=15.2.16-304-gf25ffac7-1focal 2022-06-29T15:55:39.450 DEBUG:teuthology.orchestra.run.smithi197:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.16-304-gf25ffac7-1focal cephadm=15.2.16-304-gf25ffac7-1focal ceph-mds=15.2.16-304-gf25ffac7-1focal ceph-mgr=15.2.16-304-gf25ffac7-1focal ceph-common=15.2.16-304-gf25ffac7-1focal ceph-fuse=15.2.16-304-gf25ffac7-1focal ceph-test=15.2.16-304-gf25ffac7-1focal radosgw=15.2.16-304-gf25ffac7-1focal python3-rados=15.2.16-304-gf25ffac7-1focal python3-rgw=15.2.16-304-gf25ffac7-1focal python3-cephfs=15.2.16-304-gf25ffac7-1focal python3-rbd=15.2.16-304-gf25ffac7-1focal libcephfs2=15.2.16-304-gf25ffac7-1focal libcephfs-dev=15.2.16-304-gf25ffac7-1focal librados2=15.2.16-304-gf25ffac7-1focal librbd1=15.2.16-304-gf25ffac7-1focal rbd-fuse=15.2.16-304-gf25ffac7-1focal python3-cephfs=15.2.16-304-gf25ffac7-1focal cephfs-shell=15.2.16-304-gf25ffac7-1focal 2022-06-29T15:55:39.511 INFO:teuthology.orchestra.run.smithi156.stdout:Reading package lists... 2022-06-29T15:55:39.514 INFO:teuthology.orchestra.run.smithi197.stdout:Reading package lists... 2022-06-29T15:55:39.543 INFO:teuthology.orchestra.run.smithi121.stdout:Building dependency tree... 2022-06-29T15:55:39.544 INFO:teuthology.orchestra.run.smithi121.stdout:Reading state information... 2022-06-29T15:55:39.565 INFO:teuthology.orchestra.run.smithi060.stdout:Building dependency tree... 2022-06-29T15:55:39.566 INFO:teuthology.orchestra.run.smithi060.stdout:Reading state information... 2022-06-29T15:55:39.586 INFO:teuthology.orchestra.run.smithi092.stdout:Building dependency tree... 2022-06-29T15:55:39.587 INFO:teuthology.orchestra.run.smithi092.stdout:Reading state information... 2022-06-29T15:55:39.669 INFO:teuthology.orchestra.run.smithi197.stdout:Building dependency tree... 2022-06-29T15:55:39.670 INFO:teuthology.orchestra.run.smithi197.stdout:Reading state information... 2022-06-29T15:55:39.670 INFO:teuthology.orchestra.run.smithi156.stdout:Building dependency tree... 2022-06-29T15:55:39.671 INFO:teuthology.orchestra.run.smithi156.stdout:Reading state information... 2022-06-29T15:55:39.682 INFO:teuthology.orchestra.run.smithi121.stdout:The following packages were automatically installed and are no longer required: 2022-06-29T15:55:39.683 INFO:teuthology.orchestra.run.smithi121.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-06-29T15:55:39.683 INFO:teuthology.orchestra.run.smithi121.stdout:Use 'sudo apt autoremove' to remove them. 2022-06-29T15:55:39.683 INFO:teuthology.orchestra.run.smithi121.stdout:The following additional packages will be installed: 2022-06-29T15:55:39.684 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-06-29T15:55:39.684 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-06-29T15:55:39.684 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon ceph-osd jq libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2022-06-29T15:55:39.684 INFO:teuthology.orchestra.run.smithi121.stdout: libleveldb1d liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 2022-06-29T15:55:39.685 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka1 librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-06-29T15:55:39.686 INFO:teuthology.orchestra.run.smithi121.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-06-29T15:55:39.686 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2022-06-29T15:55:39.686 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-cryptography 2022-06-29T15:55:39.686 INFO:teuthology.orchestra.run.smithi121.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2022-06-29T15:55:39.687 INFO:teuthology.orchestra.run.smithi121.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2022-06-29T15:55:39.687 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2022-06-29T15:55:39.687 INFO:teuthology.orchestra.run.smithi121.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2022-06-29T15:55:39.688 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2022-06-29T15:55:39.688 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2022-06-29T15:55:39.688 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2022-06-29T15:55:39.688 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2022-06-29T15:55:39.689 INFO:teuthology.orchestra.run.smithi121.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2022-06-29T15:55:39.689 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2022-06-29T15:55:39.689 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug socat 2022-06-29T15:55:39.689 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet 2022-06-29T15:55:39.690 INFO:teuthology.orchestra.run.smithi121.stdout:Suggested packages: 2022-06-29T15:55:39.690 INFO:teuthology.orchestra.run.smithi121.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2022-06-29T15:55:39.690 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2022-06-29T15:55:39.690 INFO:teuthology.orchestra.run.smithi121.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2022-06-29T15:55:39.691 INFO:teuthology.orchestra.run.smithi121.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2022-06-29T15:55:39.691 INFO:teuthology.orchestra.run.smithi121.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2022-06-29T15:55:39.691 INFO:teuthology.orchestra.run.smithi121.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2022-06-29T15:55:39.691 INFO:teuthology.orchestra.run.smithi121.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2022-06-29T15:55:39.691 INFO:teuthology.orchestra.run.smithi121.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2022-06-29T15:55:39.692 INFO:teuthology.orchestra.run.smithi121.stdout:Recommended packages: 2022-06-29T15:55:39.692 INFO:teuthology.orchestra.run.smithi121.stdout: btrfs-tools 2022-06-29T15:55:39.725 INFO:teuthology.orchestra.run.smithi092.stdout:The following packages were automatically installed and are no longer required: 2022-06-29T15:55:39.727 INFO:teuthology.orchestra.run.smithi092.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-06-29T15:55:39.727 INFO:teuthology.orchestra.run.smithi092.stdout:Use 'sudo apt autoremove' to remove them. 2022-06-29T15:55:39.727 INFO:teuthology.orchestra.run.smithi092.stdout:The following additional packages will be installed: 2022-06-29T15:55:39.727 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-06-29T15:55:39.728 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-06-29T15:55:39.728 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mon ceph-osd jq libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2022-06-29T15:55:39.728 INFO:teuthology.orchestra.run.smithi092.stdout: libleveldb1d liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 2022-06-29T15:55:39.728 INFO:teuthology.orchestra.run.smithi092.stdout: librdkafka1 librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-06-29T15:55:39.729 INFO:teuthology.orchestra.run.smithi092.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-06-29T15:55:39.729 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2022-06-29T15:55:39.729 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-cryptography 2022-06-29T15:55:39.729 INFO:teuthology.orchestra.run.smithi092.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2022-06-29T15:55:39.730 INFO:teuthology.orchestra.run.smithi092.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2022-06-29T15:55:39.730 INFO:teuthology.orchestra.run.smithi092.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2022-06-29T15:55:39.730 INFO:teuthology.orchestra.run.smithi092.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2022-06-29T15:55:39.730 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2022-06-29T15:55:39.730 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2022-06-29T15:55:39.731 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2022-06-29T15:55:39.731 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2022-06-29T15:55:39.731 INFO:teuthology.orchestra.run.smithi092.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2022-06-29T15:55:39.731 INFO:teuthology.orchestra.run.smithi092.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2022-06-29T15:55:39.732 INFO:teuthology.orchestra.run.smithi092.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug socat 2022-06-29T15:55:39.732 INFO:teuthology.orchestra.run.smithi092.stdout: xmlstarlet 2022-06-29T15:55:39.733 INFO:teuthology.orchestra.run.smithi092.stdout:Suggested packages: 2022-06-29T15:55:39.733 INFO:teuthology.orchestra.run.smithi092.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2022-06-29T15:55:39.733 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2022-06-29T15:55:39.733 INFO:teuthology.orchestra.run.smithi092.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2022-06-29T15:55:39.734 INFO:teuthology.orchestra.run.smithi092.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2022-06-29T15:55:39.734 INFO:teuthology.orchestra.run.smithi092.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2022-06-29T15:55:39.734 INFO:teuthology.orchestra.run.smithi092.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2022-06-29T15:55:39.734 INFO:teuthology.orchestra.run.smithi092.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2022-06-29T15:55:39.734 INFO:teuthology.orchestra.run.smithi092.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2022-06-29T15:55:39.735 INFO:teuthology.orchestra.run.smithi092.stdout:Recommended packages: 2022-06-29T15:55:39.735 INFO:teuthology.orchestra.run.smithi092.stdout: btrfs-tools 2022-06-29T15:55:39.737 INFO:teuthology.orchestra.run.smithi060.stdout:The following packages were automatically installed and are no longer required: 2022-06-29T15:55:39.737 INFO:teuthology.orchestra.run.smithi060.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-06-29T15:55:39.737 INFO:teuthology.orchestra.run.smithi060.stdout:Use 'sudo apt autoremove' to remove them. 2022-06-29T15:55:39.738 INFO:teuthology.orchestra.run.smithi060.stdout:The following additional packages will be installed: 2022-06-29T15:55:39.738 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-06-29T15:55:39.738 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-06-29T15:55:39.738 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mon ceph-osd jq libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2022-06-29T15:55:39.739 INFO:teuthology.orchestra.run.smithi060.stdout: libleveldb1d liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 2022-06-29T15:55:39.739 INFO:teuthology.orchestra.run.smithi060.stdout: librdkafka1 librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-06-29T15:55:39.739 INFO:teuthology.orchestra.run.smithi060.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-06-29T15:55:39.739 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2022-06-29T15:55:39.740 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-cryptography 2022-06-29T15:55:39.740 INFO:teuthology.orchestra.run.smithi060.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2022-06-29T15:55:39.740 INFO:teuthology.orchestra.run.smithi060.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2022-06-29T15:55:39.740 INFO:teuthology.orchestra.run.smithi060.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2022-06-29T15:55:39.740 INFO:teuthology.orchestra.run.smithi060.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2022-06-29T15:55:39.741 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2022-06-29T15:55:39.741 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2022-06-29T15:55:39.741 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2022-06-29T15:55:39.741 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2022-06-29T15:55:39.741 INFO:teuthology.orchestra.run.smithi060.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2022-06-29T15:55:39.742 INFO:teuthology.orchestra.run.smithi060.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2022-06-29T15:55:39.742 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug socat 2022-06-29T15:55:39.742 INFO:teuthology.orchestra.run.smithi060.stdout: xmlstarlet 2022-06-29T15:55:39.742 INFO:teuthology.orchestra.run.smithi060.stdout:Suggested packages: 2022-06-29T15:55:39.743 INFO:teuthology.orchestra.run.smithi060.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2022-06-29T15:55:39.743 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2022-06-29T15:55:39.743 INFO:teuthology.orchestra.run.smithi060.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2022-06-29T15:55:39.743 INFO:teuthology.orchestra.run.smithi060.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2022-06-29T15:55:39.743 INFO:teuthology.orchestra.run.smithi060.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2022-06-29T15:55:39.744 INFO:teuthology.orchestra.run.smithi060.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2022-06-29T15:55:39.744 INFO:teuthology.orchestra.run.smithi060.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2022-06-29T15:55:39.744 INFO:teuthology.orchestra.run.smithi060.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2022-06-29T15:55:39.745 INFO:teuthology.orchestra.run.smithi060.stdout:Recommended packages: 2022-06-29T15:55:39.745 INFO:teuthology.orchestra.run.smithi060.stdout: btrfs-tools 2022-06-29T15:55:39.787 INFO:teuthology.orchestra.run.smithi121.stdout:The following NEW packages will be installed: 2022-06-29T15:55:39.787 INFO:teuthology.orchestra.run.smithi121.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2022-06-29T15:55:39.788 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-06-29T15:55:39.788 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-06-29T15:55:39.789 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon ceph-osd ceph-test cephadm cephfs-shell jq libcephfs-dev libcephfs2 2022-06-29T15:55:39.790 INFO:teuthology.orchestra.run.smithi121.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2022-06-29T15:55:39.790 INFO:teuthology.orchestra.run.smithi121.stdout: liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 2022-06-29T15:55:39.790 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl python3-atomicwrites 2022-06-29T15:55:39.790 INFO:teuthology.orchestra.run.smithi121.stdout: python3-attr python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2022-06-29T15:55:39.791 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cherrypy3 2022-06-29T15:55:39.791 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cmd2 python3-colorama python3-cryptography python3-decorator 2022-06-29T15:55:39.791 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2022-06-29T15:55:39.791 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2022-06-29T15:55:39.792 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2022-06-29T15:55:39.792 INFO:teuthology.orchestra.run.smithi121.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2022-06-29T15:55:39.792 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2022-06-29T15:55:39.792 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules python3-pyinotify python3-pyperclip python3-pytest 2022-06-29T15:55:39.792 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2022-06-29T15:55:39.793 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-06-29T15:55:39.793 INFO:teuthology.orchestra.run.smithi121.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-06-29T15:55:39.793 INFO:teuthology.orchestra.run.smithi121.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-06-29T15:55:39.793 INFO:teuthology.orchestra.run.smithi121.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-06-29T15:55:39.793 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2022-06-29T15:55:39.794 INFO:teuthology.orchestra.run.smithi121.stdout:The following packages will be upgraded: 2022-06-29T15:55:39.794 INFO:teuthology.orchestra.run.smithi121.stdout: librados2 librbd1 2022-06-29T15:55:39.808 INFO:teuthology.orchestra.run.smithi197.stdout:The following packages were automatically installed and are no longer required: 2022-06-29T15:55:39.813 INFO:teuthology.orchestra.run.smithi156.stdout:The following packages were automatically installed and are no longer required: 2022-06-29T15:55:39.813 INFO:teuthology.orchestra.run.smithi156.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-06-29T15:55:39.813 INFO:teuthology.orchestra.run.smithi156.stdout:Use 'sudo apt autoremove' to remove them. 2022-06-29T15:55:39.813 INFO:teuthology.orchestra.run.smithi156.stdout:The following additional packages will be installed: 2022-06-29T15:55:39.814 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-06-29T15:55:39.814 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-06-29T15:55:39.814 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon ceph-osd jq libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2022-06-29T15:55:39.814 INFO:teuthology.orchestra.run.smithi156.stdout: libleveldb1d liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 2022-06-29T15:55:39.815 INFO:teuthology.orchestra.run.smithi156.stdout: librdkafka1 librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-06-29T15:55:39.815 INFO:teuthology.orchestra.run.smithi156.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-06-29T15:55:39.815 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2022-06-29T15:55:39.815 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-cryptography 2022-06-29T15:55:39.815 INFO:teuthology.orchestra.run.smithi156.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2022-06-29T15:55:39.816 INFO:teuthology.orchestra.run.smithi156.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2022-06-29T15:55:39.816 INFO:teuthology.orchestra.run.smithi156.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2022-06-29T15:55:39.816 INFO:teuthology.orchestra.run.smithi156.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2022-06-29T15:55:39.816 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2022-06-29T15:55:39.817 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2022-06-29T15:55:39.817 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2022-06-29T15:55:39.817 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2022-06-29T15:55:39.817 INFO:teuthology.orchestra.run.smithi156.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2022-06-29T15:55:39.817 INFO:teuthology.orchestra.run.smithi156.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2022-06-29T15:55:39.818 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug socat 2022-06-29T15:55:39.818 INFO:teuthology.orchestra.run.smithi156.stdout: xmlstarlet 2022-06-29T15:55:39.818 INFO:teuthology.orchestra.run.smithi156.stdout:Suggested packages: 2022-06-29T15:55:39.818 INFO:teuthology.orchestra.run.smithi156.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2022-06-29T15:55:39.819 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2022-06-29T15:55:39.819 INFO:teuthology.orchestra.run.smithi156.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2022-06-29T15:55:39.819 INFO:teuthology.orchestra.run.smithi156.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2022-06-29T15:55:39.819 INFO:teuthology.orchestra.run.smithi156.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2022-06-29T15:55:39.819 INFO:teuthology.orchestra.run.smithi156.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2022-06-29T15:55:39.820 INFO:teuthology.orchestra.run.smithi156.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2022-06-29T15:55:39.820 INFO:teuthology.orchestra.run.smithi156.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2022-06-29T15:55:39.820 INFO:teuthology.orchestra.run.smithi156.stdout:Recommended packages: 2022-06-29T15:55:39.820 INFO:teuthology.orchestra.run.smithi156.stdout: btrfs-tools 2022-06-29T15:55:39.822 INFO:teuthology.orchestra.run.smithi197.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-06-29T15:55:39.822 INFO:teuthology.orchestra.run.smithi197.stdout:Use 'sudo apt autoremove' to remove them. 2022-06-29T15:55:39.822 INFO:teuthology.orchestra.run.smithi197.stdout:The following additional packages will be installed: 2022-06-29T15:55:39.823 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-06-29T15:55:39.823 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-06-29T15:55:39.823 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-mon ceph-osd jq libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2022-06-29T15:55:39.823 INFO:teuthology.orchestra.run.smithi197.stdout: libleveldb1d liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 2022-06-29T15:55:39.823 INFO:teuthology.orchestra.run.smithi197.stdout: librdkafka1 librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-06-29T15:55:39.824 INFO:teuthology.orchestra.run.smithi197.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-06-29T15:55:39.824 INFO:teuthology.orchestra.run.smithi197.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2022-06-29T15:55:39.824 INFO:teuthology.orchestra.run.smithi197.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-cryptography 2022-06-29T15:55:39.824 INFO:teuthology.orchestra.run.smithi197.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2022-06-29T15:55:39.825 INFO:teuthology.orchestra.run.smithi197.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2022-06-29T15:55:39.825 INFO:teuthology.orchestra.run.smithi197.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2022-06-29T15:55:39.825 INFO:teuthology.orchestra.run.smithi197.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2022-06-29T15:55:39.825 INFO:teuthology.orchestra.run.smithi197.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2022-06-29T15:55:39.825 INFO:teuthology.orchestra.run.smithi197.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2022-06-29T15:55:39.826 INFO:teuthology.orchestra.run.smithi197.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2022-06-29T15:55:39.826 INFO:teuthology.orchestra.run.smithi197.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2022-06-29T15:55:39.826 INFO:teuthology.orchestra.run.smithi197.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2022-06-29T15:55:39.826 INFO:teuthology.orchestra.run.smithi197.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2022-06-29T15:55:39.827 INFO:teuthology.orchestra.run.smithi197.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug socat 2022-06-29T15:55:39.827 INFO:teuthology.orchestra.run.smithi197.stdout: xmlstarlet 2022-06-29T15:55:39.827 INFO:teuthology.orchestra.run.smithi197.stdout:Suggested packages: 2022-06-29T15:55:39.827 INFO:teuthology.orchestra.run.smithi197.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2022-06-29T15:55:39.828 INFO:teuthology.orchestra.run.smithi197.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2022-06-29T15:55:39.828 INFO:teuthology.orchestra.run.smithi197.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2022-06-29T15:55:39.828 INFO:teuthology.orchestra.run.smithi197.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2022-06-29T15:55:39.828 INFO:teuthology.orchestra.run.smithi197.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2022-06-29T15:55:39.828 INFO:teuthology.orchestra.run.smithi197.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2022-06-29T15:55:39.829 INFO:teuthology.orchestra.run.smithi197.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2022-06-29T15:55:39.829 INFO:teuthology.orchestra.run.smithi197.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2022-06-29T15:55:39.829 INFO:teuthology.orchestra.run.smithi197.stdout:Recommended packages: 2022-06-29T15:55:39.830 INFO:teuthology.orchestra.run.smithi197.stdout: btrfs-tools 2022-06-29T15:55:39.834 INFO:teuthology.orchestra.run.smithi092.stdout:The following NEW packages will be installed: 2022-06-29T15:55:39.834 INFO:teuthology.orchestra.run.smithi092.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2022-06-29T15:55:39.834 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-06-29T15:55:39.835 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-06-29T15:55:39.835 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mon ceph-osd ceph-test cephadm cephfs-shell jq libcephfs-dev libcephfs2 2022-06-29T15:55:39.835 INFO:teuthology.orchestra.run.smithi092.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2022-06-29T15:55:39.836 INFO:teuthology.orchestra.run.smithi092.stdout: liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 2022-06-29T15:55:39.836 INFO:teuthology.orchestra.run.smithi092.stdout: librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl python3-atomicwrites 2022-06-29T15:55:39.836 INFO:teuthology.orchestra.run.smithi092.stdout: python3-attr python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2022-06-29T15:55:39.836 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cherrypy3 2022-06-29T15:55:39.837 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cmd2 python3-colorama python3-cryptography python3-decorator 2022-06-29T15:55:39.837 INFO:teuthology.orchestra.run.smithi092.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2022-06-29T15:55:39.837 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2022-06-29T15:55:39.837 INFO:teuthology.orchestra.run.smithi092.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2022-06-29T15:55:39.838 INFO:teuthology.orchestra.run.smithi092.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2022-06-29T15:55:39.838 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2022-06-29T15:55:39.838 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-modules python3-pyinotify python3-pyperclip python3-pytest 2022-06-29T15:55:39.838 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2022-06-29T15:55:39.839 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-06-29T15:55:39.839 INFO:teuthology.orchestra.run.smithi092.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-06-29T15:55:39.839 INFO:teuthology.orchestra.run.smithi092.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-06-29T15:55:39.839 INFO:teuthology.orchestra.run.smithi092.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-06-29T15:55:39.840 INFO:teuthology.orchestra.run.smithi092.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2022-06-29T15:55:39.841 INFO:teuthology.orchestra.run.smithi092.stdout:The following packages will be upgraded: 2022-06-29T15:55:39.841 INFO:teuthology.orchestra.run.smithi092.stdout: librados2 librbd1 2022-06-29T15:55:39.843 INFO:teuthology.orchestra.run.smithi060.stdout:The following NEW packages will be installed: 2022-06-29T15:55:39.844 INFO:teuthology.orchestra.run.smithi060.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2022-06-29T15:55:39.844 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-06-29T15:55:39.845 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-06-29T15:55:39.845 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mon ceph-osd ceph-test cephadm cephfs-shell jq libcephfs-dev libcephfs2 2022-06-29T15:55:39.845 INFO:teuthology.orchestra.run.smithi060.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2022-06-29T15:55:39.845 INFO:teuthology.orchestra.run.smithi060.stdout: liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 2022-06-29T15:55:39.846 INFO:teuthology.orchestra.run.smithi060.stdout: librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl python3-atomicwrites 2022-06-29T15:55:39.846 INFO:teuthology.orchestra.run.smithi060.stdout: python3-attr python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2022-06-29T15:55:39.846 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cherrypy3 2022-06-29T15:55:39.846 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cmd2 python3-colorama python3-cryptography python3-decorator 2022-06-29T15:55:39.847 INFO:teuthology.orchestra.run.smithi060.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2022-06-29T15:55:39.847 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2022-06-29T15:55:39.847 INFO:teuthology.orchestra.run.smithi060.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2022-06-29T15:55:39.847 INFO:teuthology.orchestra.run.smithi060.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2022-06-29T15:55:39.848 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2022-06-29T15:55:39.848 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-modules python3-pyinotify python3-pyperclip python3-pytest 2022-06-29T15:55:39.848 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2022-06-29T15:55:39.848 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-06-29T15:55:39.849 INFO:teuthology.orchestra.run.smithi060.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-06-29T15:55:39.849 INFO:teuthology.orchestra.run.smithi060.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-06-29T15:55:39.849 INFO:teuthology.orchestra.run.smithi060.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-06-29T15:55:39.850 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2022-06-29T15:55:39.850 INFO:teuthology.orchestra.run.smithi060.stdout:The following packages will be upgraded: 2022-06-29T15:55:39.850 INFO:teuthology.orchestra.run.smithi060.stdout: librados2 librbd1 2022-06-29T15:55:39.910 INFO:teuthology.orchestra.run.smithi060.stdout:2 upgraded, 101 newly installed, 0 to remove and 7 not upgraded. 2022-06-29T15:55:39.910 INFO:teuthology.orchestra.run.smithi060.stdout:Need to get 153 MB of archives. 2022-06-29T15:55:39.910 INFO:teuthology.orchestra.run.smithi060.stdout:After this operation, 699 MB of additional disk space will be used. 2022-06-29T15:55:39.911 INFO:teuthology.orchestra.run.smithi060.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyperclip all 1.7.0-1 [10.5 kB] 2022-06-29T15:55:39.915 INFO:teuthology.orchestra.run.smithi156.stdout:The following NEW packages will be installed: 2022-06-29T15:55:39.916 INFO:teuthology.orchestra.run.smithi156.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2022-06-29T15:55:39.916 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-06-29T15:55:39.916 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-06-29T15:55:39.919 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon ceph-osd ceph-test cephadm cephfs-shell jq libcephfs-dev libcephfs2 2022-06-29T15:55:39.919 INFO:teuthology.orchestra.run.smithi156.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2022-06-29T15:55:39.920 INFO:teuthology.orchestra.run.smithi156.stdout: liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 2022-06-29T15:55:39.920 INFO:teuthology.orchestra.run.smithi156.stdout: librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl python3-atomicwrites 2022-06-29T15:55:39.920 INFO:teuthology.orchestra.run.smithi156.stdout: python3-attr python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2022-06-29T15:55:39.921 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cherrypy3 2022-06-29T15:55:39.921 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cmd2 python3-colorama python3-cryptography python3-decorator 2022-06-29T15:55:39.921 INFO:teuthology.orchestra.run.smithi156.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2022-06-29T15:55:39.921 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2022-06-29T15:55:39.922 INFO:teuthology.orchestra.run.smithi156.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2022-06-29T15:55:39.922 INFO:teuthology.orchestra.run.smithi156.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2022-06-29T15:55:39.922 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2022-06-29T15:55:39.922 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-modules python3-pyinotify python3-pyperclip python3-pytest 2022-06-29T15:55:39.923 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2022-06-29T15:55:39.923 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-06-29T15:55:39.923 INFO:teuthology.orchestra.run.smithi156.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-06-29T15:55:39.924 INFO:teuthology.orchestra.run.smithi156.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-06-29T15:55:39.924 INFO:teuthology.orchestra.run.smithi156.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-06-29T15:55:39.925 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2022-06-29T15:55:39.925 INFO:teuthology.orchestra.run.smithi156.stdout:The following packages will be upgraded: 2022-06-29T15:55:39.926 INFO:teuthology.orchestra.run.smithi156.stdout: librados2 librbd1 2022-06-29T15:55:39.927 INFO:teuthology.orchestra.run.smithi197.stdout:The following NEW packages will be installed: 2022-06-29T15:55:39.927 INFO:teuthology.orchestra.run.smithi197.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2022-06-29T15:55:39.927 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-06-29T15:55:39.928 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-06-29T15:55:39.928 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-mon ceph-osd ceph-test cephadm cephfs-shell jq libcephfs-dev libcephfs2 2022-06-29T15:55:39.928 INFO:teuthology.orchestra.run.smithi197.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2022-06-29T15:55:39.929 INFO:teuthology.orchestra.run.smithi197.stdout: liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 2022-06-29T15:55:39.929 INFO:teuthology.orchestra.run.smithi197.stdout: librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl python3-atomicwrites 2022-06-29T15:55:39.929 INFO:teuthology.orchestra.run.smithi197.stdout: python3-attr python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2022-06-29T15:55:39.929 INFO:teuthology.orchestra.run.smithi197.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cherrypy3 2022-06-29T15:55:39.930 INFO:teuthology.orchestra.run.smithi197.stdout: python3-cmd2 python3-colorama python3-cryptography python3-decorator 2022-06-29T15:55:39.930 INFO:teuthology.orchestra.run.smithi197.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2022-06-29T15:55:39.930 INFO:teuthology.orchestra.run.smithi197.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2022-06-29T15:55:39.930 INFO:teuthology.orchestra.run.smithi197.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2022-06-29T15:55:39.931 INFO:teuthology.orchestra.run.smithi197.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2022-06-29T15:55:39.931 INFO:teuthology.orchestra.run.smithi197.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2022-06-29T15:55:39.931 INFO:teuthology.orchestra.run.smithi197.stdout: python3-pyasn1-modules python3-pyinotify python3-pyperclip python3-pytest 2022-06-29T15:55:39.931 INFO:teuthology.orchestra.run.smithi197.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2022-06-29T15:55:39.932 INFO:teuthology.orchestra.run.smithi197.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-06-29T15:55:39.932 INFO:teuthology.orchestra.run.smithi197.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-06-29T15:55:39.932 INFO:teuthology.orchestra.run.smithi197.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-06-29T15:55:39.932 INFO:teuthology.orchestra.run.smithi197.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-06-29T15:55:39.933 INFO:teuthology.orchestra.run.smithi197.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2022-06-29T15:55:39.933 INFO:teuthology.orchestra.run.smithi197.stdout:The following packages will be upgraded: 2022-06-29T15:55:39.933 INFO:teuthology.orchestra.run.smithi197.stdout: librados2 librbd1 2022-06-29T15:55:39.934 INFO:teuthology.orchestra.run.smithi060.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-06-29T15:55:39.938 INFO:teuthology.orchestra.run.smithi060.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cmd2 all 0.8.5-2.1 [49.6 kB] 2022-06-29T15:55:39.946 INFO:teuthology.orchestra.run.smithi121.stdout:2 upgraded, 101 newly installed, 0 to remove and 7 not upgraded. 2022-06-29T15:55:39.947 INFO:teuthology.orchestra.run.smithi121.stdout:Need to get 153 MB of archives. 2022-06-29T15:55:39.947 INFO:teuthology.orchestra.run.smithi121.stdout:After this operation, 699 MB of additional disk space will be used. 2022-06-29T15:55:39.947 INFO:teuthology.orchestra.run.smithi121.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 15.2.16-304-gf25ffac7-1focal [1,699 kB] 2022-06-29T15:55:39.967 INFO:teuthology.orchestra.run.smithi060.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-06-29T15:55:39.977 INFO:teuthology.orchestra.run.smithi121.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyperclip all 1.7.0-1 [10.5 kB] 2022-06-29T15:55:39.978 INFO:teuthology.orchestra.run.smithi156.stdout:2 upgraded, 101 newly installed, 0 to remove and 7 not upgraded. 2022-06-29T15:55:39.979 INFO:teuthology.orchestra.run.smithi156.stdout:Need to get 153 MB of archives. 2022-06-29T15:55:39.979 INFO:teuthology.orchestra.run.smithi156.stdout:After this operation, 699 MB of additional disk space will be used. 2022-06-29T15:55:39.979 INFO:teuthology.orchestra.run.smithi156.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyperclip all 1.7.0-1 [10.5 kB] 2022-06-29T15:55:39.992 INFO:teuthology.orchestra.run.smithi092.stdout:2 upgraded, 101 newly installed, 0 to remove and 7 not upgraded. 2022-06-29T15:55:39.993 INFO:teuthology.orchestra.run.smithi092.stdout:Need to get 153 MB of archives. 2022-06-29T15:55:39.993 INFO:teuthology.orchestra.run.smithi092.stdout:After this operation, 699 MB of additional disk space will be used. 2022-06-29T15:55:39.993 INFO:teuthology.orchestra.run.smithi092.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 15.2.16-304-gf25ffac7-1focal [1,699 kB] 2022-06-29T15:55:39.998 INFO:teuthology.orchestra.run.smithi060.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 15.2.16-304-gf25ffac7-1focal [1,699 kB] 2022-06-29T15:55:39.998 INFO:teuthology.orchestra.run.smithi156.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-06-29T15:55:39.999 INFO:teuthology.orchestra.run.smithi156.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cmd2 all 0.8.5-2.1 [49.6 kB] 2022-06-29T15:55:40.006 INFO:teuthology.orchestra.run.smithi060.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-06-29T15:55:40.009 INFO:teuthology.orchestra.run.smithi060.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-06-29T15:55:40.014 INFO:teuthology.orchestra.run.smithi060.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-06-29T15:55:40.023 INFO:teuthology.orchestra.run.smithi092.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyperclip all 1.7.0-1 [10.5 kB] 2022-06-29T15:55:40.024 INFO:teuthology.orchestra.run.smithi156.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-06-29T15:55:40.040 INFO:teuthology.orchestra.run.smithi060.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2022-06-29T15:55:40.045 INFO:teuthology.orchestra.run.smithi060.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-06-29T15:55:40.046 INFO:teuthology.orchestra.run.smithi060.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-06-29T15:55:40.051 INFO:teuthology.orchestra.run.smithi060.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-06-29T15:55:40.052 INFO:teuthology.orchestra.run.smithi060.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-06-29T15:55:40.053 INFO:teuthology.orchestra.run.smithi060.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-06-29T15:55:40.053 INFO:teuthology.orchestra.run.smithi156.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-06-29T15:55:40.055 INFO:teuthology.orchestra.run.smithi156.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-06-29T15:55:40.058 INFO:teuthology.orchestra.run.smithi156.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-06-29T15:55:40.059 INFO:teuthology.orchestra.run.smithi121.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-06-29T15:55:40.060 INFO:teuthology.orchestra.run.smithi060.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jwt all 1.7.1-2ubuntu2 [17.4 kB] 2022-06-29T15:55:40.060 INFO:teuthology.orchestra.run.smithi060.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-06-29T15:55:40.060 INFO:teuthology.orchestra.run.smithi060.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-06-29T15:55:40.069 INFO:teuthology.orchestra.run.smithi060.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-06-29T15:55:40.070 INFO:teuthology.orchestra.run.smithi060.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-06-29T15:55:40.070 INFO:teuthology.orchestra.run.smithi060.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-06-29T15:55:40.072 INFO:teuthology.orchestra.run.smithi156.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 15.2.16-304-gf25ffac7-1focal [1,699 kB] 2022-06-29T15:55:40.080 INFO:teuthology.orchestra.run.smithi197.stdout:2 upgraded, 101 newly installed, 0 to remove and 7 not upgraded. 2022-06-29T15:55:40.080 INFO:teuthology.orchestra.run.smithi197.stdout:Need to get 153 MB of archives. 2022-06-29T15:55:40.080 INFO:teuthology.orchestra.run.smithi197.stdout:After this operation, 699 MB of additional disk space will be used. 2022-06-29T15:55:40.081 INFO:teuthology.orchestra.run.smithi197.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 15.2.16-304-gf25ffac7-1focal [1,699 kB] 2022-06-29T15:55:40.081 INFO:teuthology.orchestra.run.smithi156.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2022-06-29T15:55:40.085 INFO:teuthology.orchestra.run.smithi156.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-06-29T15:55:40.086 INFO:teuthology.orchestra.run.smithi156.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-06-29T15:55:40.092 INFO:teuthology.orchestra.run.smithi156.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-06-29T15:55:40.092 INFO:teuthology.orchestra.run.smithi156.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-06-29T15:55:40.093 INFO:teuthology.orchestra.run.smithi156.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-06-29T15:55:40.095 INFO:teuthology.orchestra.run.smithi060.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-06-29T15:55:40.096 INFO:teuthology.orchestra.run.smithi121.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cmd2 all 0.8.5-2.1 [49.6 kB] 2022-06-29T15:55:40.097 INFO:teuthology.orchestra.run.smithi156.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jwt all 1.7.1-2ubuntu2 [17.4 kB] 2022-06-29T15:55:40.097 INFO:teuthology.orchestra.run.smithi156.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-06-29T15:55:40.098 INFO:teuthology.orchestra.run.smithi060.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-06-29T15:55:40.098 INFO:teuthology.orchestra.run.smithi156.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-06-29T15:55:40.099 INFO:teuthology.orchestra.run.smithi060.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-06-29T15:55:40.100 INFO:teuthology.orchestra.run.smithi060.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2022-06-29T15:55:40.102 INFO:teuthology.orchestra.run.smithi060.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-06-29T15:55:40.103 INFO:teuthology.orchestra.run.smithi060.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-06-29T15:55:40.105 INFO:teuthology.orchestra.run.smithi092.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-06-29T15:55:40.106 INFO:teuthology.orchestra.run.smithi197.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyperclip all 1.7.0-1 [10.5 kB] 2022-06-29T15:55:40.106 INFO:teuthology.orchestra.run.smithi156.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-06-29T15:55:40.107 INFO:teuthology.orchestra.run.smithi156.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-06-29T15:55:40.108 INFO:teuthology.orchestra.run.smithi060.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-06-29T15:55:40.108 INFO:teuthology.orchestra.run.smithi156.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-06-29T15:55:40.115 INFO:teuthology.orchestra.run.smithi060.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-06-29T15:55:40.116 INFO:teuthology.orchestra.run.smithi060.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-06-29T15:55:40.116 INFO:teuthology.orchestra.run.smithi060.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-06-29T15:55:40.123 INFO:teuthology.orchestra.run.smithi156.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-06-29T15:55:40.125 INFO:teuthology.orchestra.run.smithi156.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-06-29T15:55:40.126 INFO:teuthology.orchestra.run.smithi156.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-06-29T15:55:40.127 INFO:teuthology.orchestra.run.smithi156.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2022-06-29T15:55:40.129 INFO:teuthology.orchestra.run.smithi156.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-06-29T15:55:40.130 INFO:teuthology.orchestra.run.smithi156.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-06-29T15:55:40.134 INFO:teuthology.orchestra.run.smithi156.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-06-29T15:55:40.140 INFO:teuthology.orchestra.run.smithi060.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-06-29T15:55:40.141 INFO:teuthology.orchestra.run.smithi156.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-06-29T15:55:40.141 INFO:teuthology.orchestra.run.smithi156.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-06-29T15:55:40.142 INFO:teuthology.orchestra.run.smithi092.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cmd2 all 0.8.5-2.1 [49.6 kB] 2022-06-29T15:55:40.142 INFO:teuthology.orchestra.run.smithi156.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-06-29T15:55:40.143 INFO:teuthology.orchestra.run.smithi060.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2022-06-29T15:55:40.146 INFO:teuthology.orchestra.run.smithi060.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-06-29T15:55:40.146 INFO:teuthology.orchestra.run.smithi060.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-06-29T15:55:40.149 INFO:teuthology.orchestra.run.smithi060.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-06-29T15:55:40.150 INFO:teuthology.orchestra.run.smithi060.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-06-29T15:55:40.151 INFO:teuthology.orchestra.run.smithi060.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-06-29T15:55:40.157 INFO:teuthology.orchestra.run.smithi156.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-06-29T15:55:40.160 INFO:teuthology.orchestra.run.smithi060.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-06-29T15:55:40.160 INFO:teuthology.orchestra.run.smithi156.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2022-06-29T15:55:40.161 INFO:teuthology.orchestra.run.smithi060.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-06-29T15:55:40.161 INFO:teuthology.orchestra.run.smithi156.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-06-29T15:55:40.162 INFO:teuthology.orchestra.run.smithi156.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-06-29T15:55:40.164 INFO:teuthology.orchestra.run.smithi156.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-06-29T15:55:40.166 INFO:teuthology.orchestra.run.smithi156.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-06-29T15:55:40.167 INFO:teuthology.orchestra.run.smithi156.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-06-29T15:55:40.174 INFO:teuthology.orchestra.run.smithi156.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-06-29T15:55:40.174 INFO:teuthology.orchestra.run.smithi156.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-06-29T15:55:40.178 INFO:teuthology.orchestra.run.smithi121.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 15.2.16-304-gf25ffac7-1focal [3,257 kB] 2022-06-29T15:55:40.188 INFO:teuthology.orchestra.run.smithi197.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-06-29T15:55:40.226 INFO:teuthology.orchestra.run.smithi197.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cmd2 all 0.8.5-2.1 [49.6 kB] 2022-06-29T15:55:40.229 INFO:teuthology.orchestra.run.smithi121.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-06-29T15:55:40.253 INFO:teuthology.orchestra.run.smithi156.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-06-29T15:55:40.260 INFO:teuthology.orchestra.run.smithi092.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 15.2.16-304-gf25ffac7-1focal [3,257 kB] 2022-06-29T15:55:40.269 INFO:teuthology.orchestra.run.smithi156.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-06-29T15:55:40.270 INFO:teuthology.orchestra.run.smithi156.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-06-29T15:55:40.277 INFO:teuthology.orchestra.run.smithi092.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-06-29T15:55:40.282 INFO:teuthology.orchestra.run.smithi060.stdout:Get:40 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 15.2.16-304-gf25ffac7-1focal [3,257 kB] 2022-06-29T15:55:40.284 INFO:teuthology.orchestra.run.smithi156.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-06-29T15:55:40.284 INFO:teuthology.orchestra.run.smithi156.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-06-29T15:55:40.285 INFO:teuthology.orchestra.run.smithi156.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-06-29T15:55:40.313 INFO:teuthology.orchestra.run.smithi060.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-06-29T15:55:40.346 INFO:teuthology.orchestra.run.smithi156.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-06-29T15:55:40.346 INFO:teuthology.orchestra.run.smithi156.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-06-29T15:55:40.347 INFO:teuthology.orchestra.run.smithi156.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-06-29T15:55:40.348 INFO:teuthology.orchestra.run.smithi156.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-06-29T15:55:40.349 INFO:teuthology.orchestra.run.smithi060.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-06-29T15:55:40.349 INFO:teuthology.orchestra.run.smithi060.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-06-29T15:55:40.350 INFO:teuthology.orchestra.run.smithi156.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-06-29T15:55:40.350 INFO:teuthology.orchestra.run.smithi156.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-06-29T15:55:40.352 INFO:teuthology.orchestra.run.smithi156.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-06-29T15:55:40.357 INFO:teuthology.orchestra.run.smithi156.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-06-29T15:55:40.357 INFO:teuthology.orchestra.run.smithi197.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-06-29T15:55:40.362 INFO:teuthology.orchestra.run.smithi121.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-06-29T15:55:40.363 INFO:teuthology.orchestra.run.smithi156.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-06-29T15:55:40.372 INFO:teuthology.orchestra.run.smithi121.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-06-29T15:55:40.376 INFO:teuthology.orchestra.run.smithi060.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-06-29T15:55:40.376 INFO:teuthology.orchestra.run.smithi060.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-06-29T15:55:40.377 INFO:teuthology.orchestra.run.smithi156.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-06-29T15:55:40.378 INFO:teuthology.orchestra.run.smithi156.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-06-29T15:55:40.378 INFO:teuthology.orchestra.run.smithi060.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-06-29T15:55:40.378 INFO:teuthology.orchestra.run.smithi060.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-06-29T15:55:40.379 INFO:teuthology.orchestra.run.smithi060.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-06-29T15:55:40.379 INFO:teuthology.orchestra.run.smithi060.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-06-29T15:55:40.380 INFO:teuthology.orchestra.run.smithi060.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-06-29T15:55:40.381 INFO:teuthology.orchestra.run.smithi156.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-06-29T15:55:40.383 INFO:teuthology.orchestra.run.smithi156.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-06-29T15:55:40.386 INFO:teuthology.orchestra.run.smithi156.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-colorama all 0.4.3-1build1 [23.9 kB] 2022-06-29T15:55:40.386 INFO:teuthology.orchestra.run.smithi156.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-06-29T15:55:40.390 INFO:teuthology.orchestra.run.smithi121.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-06-29T15:55:40.391 INFO:teuthology.orchestra.run.smithi156.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-06-29T15:55:40.391 INFO:teuthology.orchestra.run.smithi156.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-06-29T15:55:40.391 INFO:teuthology.orchestra.run.smithi156.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-06-29T15:55:40.394 INFO:teuthology.orchestra.run.smithi060.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-06-29T15:55:40.394 INFO:teuthology.orchestra.run.smithi060.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-06-29T15:55:40.398 INFO:teuthology.orchestra.run.smithi156.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-06-29T15:55:40.399 INFO:teuthology.orchestra.run.smithi156.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-06-29T15:55:40.409 INFO:teuthology.orchestra.run.smithi092.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-06-29T15:55:40.415 INFO:teuthology.orchestra.run.smithi156.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-06-29T15:55:40.416 INFO:teuthology.orchestra.run.smithi156.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-06-29T15:55:40.416 INFO:teuthology.orchestra.run.smithi156.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-06-29T15:55:40.417 INFO:teuthology.orchestra.run.smithi156.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-06-29T15:55:40.417 INFO:teuthology.orchestra.run.smithi092.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-06-29T15:55:40.418 INFO:teuthology.orchestra.run.smithi156.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-06-29T15:55:40.418 INFO:teuthology.orchestra.run.smithi156.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-06-29T15:55:40.419 INFO:teuthology.orchestra.run.smithi156.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-06-29T15:55:40.420 INFO:teuthology.orchestra.run.smithi156.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-06-29T15:55:40.425 INFO:teuthology.orchestra.run.smithi060.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-06-29T15:55:40.430 INFO:teuthology.orchestra.run.smithi060.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-06-29T15:55:40.433 INFO:teuthology.orchestra.run.smithi092.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-06-29T15:55:40.433 INFO:teuthology.orchestra.run.smithi060.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-06-29T15:55:40.434 INFO:teuthology.orchestra.run.smithi060.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-06-29T15:55:40.435 INFO:teuthology.orchestra.run.smithi060.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-06-29T15:55:40.438 INFO:teuthology.orchestra.run.smithi060.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-06-29T15:55:40.439 INFO:teuthology.orchestra.run.smithi060.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-06-29T15:55:40.442 INFO:teuthology.orchestra.run.smithi060.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-colorama all 0.4.3-1build1 [23.9 kB] 2022-06-29T15:55:40.445 INFO:teuthology.orchestra.run.smithi060.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-06-29T15:55:40.448 INFO:teuthology.orchestra.run.smithi060.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-06-29T15:55:40.475 INFO:teuthology.orchestra.run.smithi060.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-06-29T15:55:40.476 INFO:teuthology.orchestra.run.smithi060.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-06-29T15:55:40.477 INFO:teuthology.orchestra.run.smithi060.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-06-29T15:55:40.479 INFO:teuthology.orchestra.run.smithi060.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-06-29T15:55:40.492 INFO:teuthology.orchestra.run.smithi121.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2022-06-29T15:55:40.496 INFO:teuthology.orchestra.run.smithi197.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-06-29T15:55:40.497 INFO:teuthology.orchestra.run.smithi060.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-06-29T15:55:40.497 INFO:teuthology.orchestra.run.smithi060.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-06-29T15:55:40.498 INFO:teuthology.orchestra.run.smithi060.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-06-29T15:55:40.498 INFO:teuthology.orchestra.run.smithi060.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-06-29T15:55:40.498 INFO:teuthology.orchestra.run.smithi060.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-06-29T15:55:40.499 INFO:teuthology.orchestra.run.smithi060.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-06-29T15:55:40.507 INFO:teuthology.orchestra.run.smithi197.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-06-29T15:55:40.514 INFO:teuthology.orchestra.run.smithi121.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-06-29T15:55:40.519 INFO:teuthology.orchestra.run.smithi060.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-06-29T15:55:40.520 INFO:teuthology.orchestra.run.smithi121.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-06-29T15:55:40.524 INFO:teuthology.orchestra.run.smithi197.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-06-29T15:55:40.524 INFO:teuthology.orchestra.run.smithi060.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-06-29T15:55:40.528 INFO:teuthology.orchestra.run.smithi121.stdout:Get:13 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 15.2.16-304-gf25ffac7-1focal [463 kB] 2022-06-29T15:55:40.538 INFO:teuthology.orchestra.run.smithi092.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2022-06-29T15:55:40.545 INFO:teuthology.orchestra.run.smithi121.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-06-29T15:55:40.548 INFO:teuthology.orchestra.run.smithi121.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-06-29T15:55:40.552 INFO:teuthology.orchestra.run.smithi121.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-06-29T15:55:40.559 INFO:teuthology.orchestra.run.smithi092.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-06-29T15:55:40.565 INFO:teuthology.orchestra.run.smithi092.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-06-29T15:55:40.567 INFO:teuthology.orchestra.run.smithi121.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jwt all 1.7.1-2ubuntu2 [17.4 kB] 2022-06-29T15:55:40.568 INFO:teuthology.orchestra.run.smithi121.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-06-29T15:55:40.570 INFO:teuthology.orchestra.run.smithi121.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-06-29T15:55:40.580 INFO:teuthology.orchestra.run.smithi121.stdout:Get:20 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 15.2.16-304-gf25ffac7-1focal [366 kB] 2022-06-29T15:55:40.591 INFO:teuthology.orchestra.run.smithi092.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-06-29T15:55:40.595 INFO:teuthology.orchestra.run.smithi092.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-06-29T15:55:40.598 INFO:teuthology.orchestra.run.smithi092.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-06-29T15:55:40.608 INFO:teuthology.orchestra.run.smithi121.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-06-29T15:55:40.611 INFO:teuthology.orchestra.run.smithi121.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-06-29T15:55:40.612 INFO:teuthology.orchestra.run.smithi121.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-06-29T15:55:40.615 INFO:teuthology.orchestra.run.smithi121.stdout:Get:24 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 15.2.16-304-gf25ffac7-1focal [20.9 kB] 2022-06-29T15:55:40.616 INFO:teuthology.orchestra.run.smithi092.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jwt all 1.7.1-2ubuntu2 [17.4 kB] 2022-06-29T15:55:40.616 INFO:teuthology.orchestra.run.smithi121.stdout:Get:25 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 15.2.16-304-gf25ffac7-1focal [156 kB] 2022-06-29T15:55:40.617 INFO:teuthology.orchestra.run.smithi092.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-06-29T15:55:40.617 INFO:teuthology.orchestra.run.smithi092.stdout:Get:18 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 15.2.16-304-gf25ffac7-1focal [463 kB] 2022-06-29T15:55:40.618 INFO:teuthology.orchestra.run.smithi092.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-06-29T15:55:40.627 INFO:teuthology.orchestra.run.smithi197.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2022-06-29T15:55:40.632 INFO:teuthology.orchestra.run.smithi092.stdout:Get:20 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 15.2.16-304-gf25ffac7-1focal [366 kB] 2022-06-29T15:55:40.639 INFO:teuthology.orchestra.run.smithi121.stdout:Get:26 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 15.2.16-304-gf25ffac7-1focal [29.4 kB] 2022-06-29T15:55:40.640 INFO:teuthology.orchestra.run.smithi121.stdout:Get:27 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 15.2.16-304-gf25ffac7-1focal [364 kB] 2022-06-29T15:55:40.648 INFO:teuthology.orchestra.run.smithi197.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-06-29T15:55:40.653 INFO:teuthology.orchestra.run.smithi092.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-06-29T15:55:40.654 INFO:teuthology.orchestra.run.smithi197.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-06-29T15:55:40.655 INFO:teuthology.orchestra.run.smithi060.stdout:Get:75 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 15.2.16-304-gf25ffac7-1focal [463 kB] 2022-06-29T15:55:40.655 INFO:teuthology.orchestra.run.smithi092.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-06-29T15:55:40.656 INFO:teuthology.orchestra.run.smithi092.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-06-29T15:55:40.679 INFO:teuthology.orchestra.run.smithi197.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-06-29T15:55:40.684 INFO:teuthology.orchestra.run.smithi197.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-06-29T15:55:40.687 INFO:teuthology.orchestra.run.smithi092.stdout:Get:24 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 15.2.16-304-gf25ffac7-1focal [20.9 kB] 2022-06-29T15:55:40.687 INFO:teuthology.orchestra.run.smithi197.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-06-29T15:55:40.688 INFO:teuthology.orchestra.run.smithi092.stdout:Get:25 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 15.2.16-304-gf25ffac7-1focal [156 kB] 2022-06-29T15:55:40.688 INFO:teuthology.orchestra.run.smithi121.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-06-29T15:55:40.691 INFO:teuthology.orchestra.run.smithi121.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-06-29T15:55:40.692 INFO:teuthology.orchestra.run.smithi121.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-06-29T15:55:40.692 INFO:teuthology.orchestra.run.smithi121.stdout:Get:31 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 15.2.16-304-gf25ffac7-1focal [3,145 kB] 2022-06-29T15:55:40.693 INFO:teuthology.orchestra.run.smithi092.stdout:Get:26 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 15.2.16-304-gf25ffac7-1focal [29.4 kB] 2022-06-29T15:55:40.694 INFO:teuthology.orchestra.run.smithi092.stdout:Get:27 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 15.2.16-304-gf25ffac7-1focal [364 kB] 2022-06-29T15:55:40.695 INFO:teuthology.orchestra.run.smithi121.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2022-06-29T15:55:40.699 INFO:teuthology.orchestra.run.smithi121.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-06-29T15:55:40.700 INFO:teuthology.orchestra.run.smithi121.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-06-29T15:55:40.702 INFO:teuthology.orchestra.run.smithi197.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jwt all 1.7.1-2ubuntu2 [17.4 kB] 2022-06-29T15:55:40.704 INFO:teuthology.orchestra.run.smithi197.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-06-29T15:55:40.705 INFO:teuthology.orchestra.run.smithi197.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-06-29T15:55:40.714 INFO:teuthology.orchestra.run.smithi060.stdout:Get:76 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 15.2.16-304-gf25ffac7-1focal [366 kB] 2022-06-29T15:55:40.718 INFO:teuthology.orchestra.run.smithi121.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-06-29T15:55:40.733 INFO:teuthology.orchestra.run.smithi092.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-06-29T15:55:40.736 INFO:teuthology.orchestra.run.smithi092.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-06-29T15:55:40.737 INFO:teuthology.orchestra.run.smithi092.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-06-29T15:55:40.740 INFO:teuthology.orchestra.run.smithi197.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-06-29T15:55:40.741 INFO:teuthology.orchestra.run.smithi092.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2022-06-29T15:55:40.743 INFO:teuthology.orchestra.run.smithi197.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-06-29T15:55:40.744 INFO:teuthology.orchestra.run.smithi197.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-06-29T15:55:40.745 INFO:teuthology.orchestra.run.smithi092.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-06-29T15:55:40.746 INFO:teuthology.orchestra.run.smithi092.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-06-29T15:55:40.750 INFO:teuthology.orchestra.run.smithi092.stdout:Get:34 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 15.2.16-304-gf25ffac7-1focal [3,145 kB] 2022-06-29T15:55:40.763 INFO:teuthology.orchestra.run.smithi092.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-06-29T15:55:40.767 INFO:teuthology.orchestra.run.smithi121.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-06-29T15:55:40.768 INFO:teuthology.orchestra.run.smithi060.stdout:Get:77 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 15.2.16-304-gf25ffac7-1focal [20.9 kB] 2022-06-29T15:55:40.769 INFO:teuthology.orchestra.run.smithi121.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-06-29T15:55:40.770 INFO:teuthology.orchestra.run.smithi121.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-06-29T15:55:40.773 INFO:teuthology.orchestra.run.smithi060.stdout:Get:78 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 15.2.16-304-gf25ffac7-1focal [156 kB] 2022-06-29T15:55:40.778 INFO:teuthology.orchestra.run.smithi060.stdout:Get:79 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 15.2.16-304-gf25ffac7-1focal [29.4 kB] 2022-06-29T15:55:40.780 INFO:teuthology.orchestra.run.smithi060.stdout:Get:80 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 15.2.16-304-gf25ffac7-1focal [364 kB] 2022-06-29T15:55:40.813 INFO:teuthology.orchestra.run.smithi092.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-06-29T15:55:40.813 INFO:teuthology.orchestra.run.smithi092.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-06-29T15:55:40.815 INFO:teuthology.orchestra.run.smithi092.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-06-29T15:55:40.820 INFO:teuthology.orchestra.run.smithi197.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-06-29T15:55:40.824 INFO:teuthology.orchestra.run.smithi197.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-06-29T15:55:40.825 INFO:teuthology.orchestra.run.smithi197.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-06-29T15:55:40.828 INFO:teuthology.orchestra.run.smithi197.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2022-06-29T15:55:40.832 INFO:teuthology.orchestra.run.smithi197.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-06-29T15:55:40.833 INFO:teuthology.orchestra.run.smithi197.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-06-29T15:55:40.835 INFO:teuthology.orchestra.run.smithi060.stdout:Get:81 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 15.2.16-304-gf25ffac7-1focal [3,145 kB] 2022-06-29T15:55:40.848 INFO:teuthology.orchestra.run.smithi121.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-06-29T15:55:40.851 INFO:teuthology.orchestra.run.smithi197.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-06-29T15:55:40.853 INFO:teuthology.orchestra.run.smithi156.stdout:Get:74 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 15.2.16-304-gf25ffac7-1focal [3,257 kB] 2022-06-29T15:55:40.854 INFO:teuthology.orchestra.run.smithi121.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2022-06-29T15:55:40.858 INFO:teuthology.orchestra.run.smithi121.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-06-29T15:55:40.859 INFO:teuthology.orchestra.run.smithi121.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-06-29T15:55:40.861 INFO:teuthology.orchestra.run.smithi197.stdout:Get:28 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 15.2.16-304-gf25ffac7-1focal [3,257 kB] 2022-06-29T15:55:40.868 INFO:teuthology.orchestra.run.smithi121.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-06-29T15:55:40.868 INFO:teuthology.orchestra.run.smithi121.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-06-29T15:55:40.868 INFO:teuthology.orchestra.run.smithi121.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-06-29T15:55:40.893 INFO:teuthology.orchestra.run.smithi092.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-06-29T15:55:40.899 INFO:teuthology.orchestra.run.smithi092.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2022-06-29T15:55:40.901 INFO:teuthology.orchestra.run.smithi197.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-06-29T15:55:40.902 INFO:teuthology.orchestra.run.smithi197.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-06-29T15:55:40.903 INFO:teuthology.orchestra.run.smithi092.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-06-29T15:55:40.903 INFO:teuthology.orchestra.run.smithi092.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-06-29T15:55:40.904 INFO:teuthology.orchestra.run.smithi197.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-06-29T15:55:40.906 INFO:teuthology.orchestra.run.smithi092.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-06-29T15:55:40.910 INFO:teuthology.orchestra.run.smithi092.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-06-29T15:55:40.913 INFO:teuthology.orchestra.run.smithi092.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-06-29T15:55:40.928 INFO:teuthology.orchestra.run.smithi121.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-06-29T15:55:40.930 INFO:teuthology.orchestra.run.smithi121.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-06-29T15:55:40.973 INFO:teuthology.orchestra.run.smithi092.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-06-29T15:55:40.974 INFO:teuthology.orchestra.run.smithi092.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-06-29T15:55:40.981 INFO:teuthology.orchestra.run.smithi197.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-06-29T15:55:40.988 INFO:teuthology.orchestra.run.smithi197.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2022-06-29T15:55:40.992 INFO:teuthology.orchestra.run.smithi197.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-06-29T15:55:40.992 INFO:teuthology.orchestra.run.smithi197.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-06-29T15:55:40.995 INFO:teuthology.orchestra.run.smithi197.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-06-29T15:55:40.998 INFO:teuthology.orchestra.run.smithi197.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-06-29T15:55:41.001 INFO:teuthology.orchestra.run.smithi197.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-06-29T15:55:41.053 INFO:teuthology.orchestra.run.smithi121.stdout:Get:48 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 15.2.16-304-gf25ffac7-1focal [116 kB] 2022-06-29T15:55:41.056 INFO:teuthology.orchestra.run.smithi121.stdout:Get:49 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 15.2.16-304-gf25ffac7-1focal [361 kB] 2022-06-29T15:55:41.061 INFO:teuthology.orchestra.run.smithi197.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-06-29T15:55:41.062 INFO:teuthology.orchestra.run.smithi197.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-06-29T15:55:41.064 INFO:teuthology.orchestra.run.smithi121.stdout:Get:50 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 15.2.16-304-gf25ffac7-1focal [20.1 MB] 2022-06-29T15:55:41.123 INFO:teuthology.orchestra.run.smithi092.stdout:Get:48 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 15.2.16-304-gf25ffac7-1focal [116 kB] 2022-06-29T15:55:41.125 INFO:teuthology.orchestra.run.smithi092.stdout:Get:49 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 15.2.16-304-gf25ffac7-1focal [361 kB] 2022-06-29T15:55:41.179 INFO:teuthology.orchestra.run.smithi092.stdout:Get:50 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 15.2.16-304-gf25ffac7-1focal [20.1 MB] 2022-06-29T15:55:41.245 INFO:teuthology.orchestra.run.smithi060.stdout:Get:82 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 15.2.16-304-gf25ffac7-1focal [116 kB] 2022-06-29T15:55:41.248 INFO:teuthology.orchestra.run.smithi060.stdout:Get:83 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 15.2.16-304-gf25ffac7-1focal [361 kB] 2022-06-29T15:55:41.260 INFO:teuthology.orchestra.run.smithi121.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-06-29T15:55:41.263 INFO:teuthology.orchestra.run.smithi060.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 15.2.16-304-gf25ffac7-1focal [20.1 MB] 2022-06-29T15:55:41.311 INFO:teuthology.orchestra.run.smithi092.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-06-29T15:55:41.323 INFO:teuthology.orchestra.run.smithi121.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-06-29T15:55:41.325 INFO:teuthology.orchestra.run.smithi121.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-06-29T15:55:41.366 INFO:teuthology.orchestra.run.smithi121.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-06-29T15:55:41.366 INFO:teuthology.orchestra.run.smithi121.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-06-29T15:55:41.367 INFO:teuthology.orchestra.run.smithi092.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-06-29T15:55:41.369 INFO:teuthology.orchestra.run.smithi121.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-06-29T15:55:41.370 INFO:teuthology.orchestra.run.smithi092.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-06-29T15:55:41.391 INFO:teuthology.orchestra.run.smithi121.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-06-29T15:55:41.392 INFO:teuthology.orchestra.run.smithi121.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-06-29T15:55:41.393 INFO:teuthology.orchestra.run.smithi121.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-06-29T15:55:41.394 INFO:teuthology.orchestra.run.smithi121.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-06-29T15:55:41.409 INFO:teuthology.orchestra.run.smithi092.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-06-29T15:55:41.409 INFO:teuthology.orchestra.run.smithi092.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-06-29T15:55:41.415 INFO:teuthology.orchestra.run.smithi197.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-06-29T15:55:41.424 INFO:teuthology.orchestra.run.smithi121.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-06-29T15:55:41.425 INFO:teuthology.orchestra.run.smithi121.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-06-29T15:55:41.435 INFO:teuthology.orchestra.run.smithi092.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-06-29T15:55:41.437 INFO:teuthology.orchestra.run.smithi092.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-06-29T15:55:41.439 INFO:teuthology.orchestra.run.smithi092.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-06-29T15:55:41.440 INFO:teuthology.orchestra.run.smithi092.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-06-29T15:55:41.440 INFO:teuthology.orchestra.run.smithi092.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-06-29T15:55:41.464 INFO:teuthology.orchestra.run.smithi197.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-06-29T15:55:41.466 INFO:teuthology.orchestra.run.smithi197.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-06-29T15:55:41.475 INFO:teuthology.orchestra.run.smithi092.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-06-29T15:55:41.476 INFO:teuthology.orchestra.run.smithi092.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-06-29T15:55:41.505 INFO:teuthology.orchestra.run.smithi121.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-06-29T15:55:41.510 INFO:teuthology.orchestra.run.smithi121.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-06-29T15:55:41.513 INFO:teuthology.orchestra.run.smithi121.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-06-29T15:55:41.517 INFO:teuthology.orchestra.run.smithi121.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-06-29T15:55:41.519 INFO:teuthology.orchestra.run.smithi121.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-06-29T15:55:41.521 INFO:teuthology.orchestra.run.smithi121.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-06-29T15:55:41.523 INFO:teuthology.orchestra.run.smithi121.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-06-29T15:55:41.525 INFO:teuthology.orchestra.run.smithi121.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-colorama all 0.4.3-1build1 [23.9 kB] 2022-06-29T15:55:41.542 INFO:teuthology.orchestra.run.smithi197.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-06-29T15:55:41.543 INFO:teuthology.orchestra.run.smithi197.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-06-29T15:55:41.544 INFO:teuthology.orchestra.run.smithi197.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-06-29T15:55:41.544 INFO:teuthology.orchestra.run.smithi197.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-06-29T15:55:41.545 INFO:teuthology.orchestra.run.smithi197.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-06-29T15:55:41.546 INFO:teuthology.orchestra.run.smithi197.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-06-29T15:55:41.546 INFO:teuthology.orchestra.run.smithi197.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-06-29T15:55:41.556 INFO:teuthology.orchestra.run.smithi092.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-06-29T15:55:41.562 INFO:teuthology.orchestra.run.smithi092.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-06-29T15:55:41.566 INFO:teuthology.orchestra.run.smithi092.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-06-29T15:55:41.569 INFO:teuthology.orchestra.run.smithi092.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-06-29T15:55:41.570 INFO:teuthology.orchestra.run.smithi092.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-06-29T15:55:41.572 INFO:teuthology.orchestra.run.smithi092.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-06-29T15:55:41.573 INFO:teuthology.orchestra.run.smithi092.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-06-29T15:55:41.575 INFO:teuthology.orchestra.run.smithi092.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-colorama all 0.4.3-1build1 [23.9 kB] 2022-06-29T15:55:41.580 INFO:teuthology.orchestra.run.smithi197.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-06-29T15:55:41.581 INFO:teuthology.orchestra.run.smithi197.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-06-29T15:55:41.586 INFO:teuthology.orchestra.run.smithi121.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-06-29T15:55:41.592 INFO:teuthology.orchestra.run.smithi121.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-06-29T15:55:41.636 INFO:teuthology.orchestra.run.smithi092.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-06-29T15:55:41.644 INFO:teuthology.orchestra.run.smithi092.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-06-29T15:55:41.661 INFO:teuthology.orchestra.run.smithi197.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-06-29T15:55:41.667 INFO:teuthology.orchestra.run.smithi197.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-06-29T15:55:41.668 INFO:teuthology.orchestra.run.smithi121.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-06-29T15:55:41.669 INFO:teuthology.orchestra.run.smithi121.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-06-29T15:55:41.670 INFO:teuthology.orchestra.run.smithi121.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-06-29T15:55:41.670 INFO:teuthology.orchestra.run.smithi197.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-06-29T15:55:41.672 INFO:teuthology.orchestra.run.smithi121.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-06-29T15:55:41.674 INFO:teuthology.orchestra.run.smithi197.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-06-29T15:55:41.676 INFO:teuthology.orchestra.run.smithi197.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-06-29T15:55:41.678 INFO:teuthology.orchestra.run.smithi197.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-06-29T15:55:41.679 INFO:teuthology.orchestra.run.smithi197.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-06-29T15:55:41.681 INFO:teuthology.orchestra.run.smithi197.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-colorama all 0.4.3-1build1 [23.9 kB] 2022-06-29T15:55:41.690 INFO:teuthology.orchestra.run.smithi121.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-06-29T15:55:41.691 INFO:teuthology.orchestra.run.smithi121.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-06-29T15:55:41.691 INFO:teuthology.orchestra.run.smithi121.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-06-29T15:55:41.692 INFO:teuthology.orchestra.run.smithi121.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-06-29T15:55:41.716 INFO:teuthology.orchestra.run.smithi092.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-06-29T15:55:41.717 INFO:teuthology.orchestra.run.smithi092.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-06-29T15:55:41.718 INFO:teuthology.orchestra.run.smithi092.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-06-29T15:55:41.720 INFO:teuthology.orchestra.run.smithi092.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-06-29T15:55:41.737 INFO:teuthology.orchestra.run.smithi092.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-06-29T15:55:41.738 INFO:teuthology.orchestra.run.smithi092.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-06-29T15:55:41.739 INFO:teuthology.orchestra.run.smithi092.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-06-29T15:55:41.739 INFO:teuthology.orchestra.run.smithi092.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-06-29T15:55:41.742 INFO:teuthology.orchestra.run.smithi197.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-06-29T15:55:41.748 INFO:teuthology.orchestra.run.smithi121.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-06-29T15:55:41.749 INFO:teuthology.orchestra.run.smithi197.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-06-29T15:55:41.750 INFO:teuthology.orchestra.run.smithi121.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-06-29T15:55:41.796 INFO:teuthology.orchestra.run.smithi092.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-06-29T15:55:41.798 INFO:teuthology.orchestra.run.smithi092.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-06-29T15:55:41.822 INFO:teuthology.orchestra.run.smithi197.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-06-29T15:55:41.823 INFO:teuthology.orchestra.run.smithi197.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-06-29T15:55:41.824 INFO:teuthology.orchestra.run.smithi197.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-06-29T15:55:41.825 INFO:teuthology.orchestra.run.smithi197.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-06-29T15:55:41.827 INFO:teuthology.orchestra.run.smithi121.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-06-29T15:55:41.831 INFO:teuthology.orchestra.run.smithi121.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-06-29T15:55:41.843 INFO:teuthology.orchestra.run.smithi197.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-06-29T15:55:41.844 INFO:teuthology.orchestra.run.smithi197.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-06-29T15:55:41.844 INFO:teuthology.orchestra.run.smithi197.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-06-29T15:55:41.845 INFO:teuthology.orchestra.run.smithi197.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-06-29T15:55:41.876 INFO:teuthology.orchestra.run.smithi092.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-06-29T15:55:41.878 INFO:teuthology.orchestra.run.smithi092.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-06-29T15:55:41.902 INFO:teuthology.orchestra.run.smithi197.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-06-29T15:55:41.904 INFO:teuthology.orchestra.run.smithi197.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-06-29T15:55:41.982 INFO:teuthology.orchestra.run.smithi197.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-06-29T15:55:41.985 INFO:teuthology.orchestra.run.smithi197.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-06-29T15:55:42.136 INFO:teuthology.orchestra.run.smithi197.stdout:Get:75 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 15.2.16-304-gf25ffac7-1focal [463 kB] 2022-06-29T15:55:42.246 INFO:teuthology.orchestra.run.smithi156.stdout:Get:75 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 15.2.16-304-gf25ffac7-1focal [463 kB] 2022-06-29T15:55:42.252 INFO:teuthology.orchestra.run.smithi197.stdout:Get:76 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 15.2.16-304-gf25ffac7-1focal [366 kB] 2022-06-29T15:55:42.394 INFO:teuthology.orchestra.run.smithi197.stdout:Get:77 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 15.2.16-304-gf25ffac7-1focal [20.9 kB] 2022-06-29T15:55:42.395 INFO:teuthology.orchestra.run.smithi197.stdout:Get:78 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 15.2.16-304-gf25ffac7-1focal [156 kB] 2022-06-29T15:55:42.453 INFO:teuthology.orchestra.run.smithi156.stdout:Get:76 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 15.2.16-304-gf25ffac7-1focal [366 kB] 2022-06-29T15:55:42.462 INFO:teuthology.orchestra.run.smithi197.stdout:Get:79 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 15.2.16-304-gf25ffac7-1focal [29.4 kB] 2022-06-29T15:55:42.464 INFO:teuthology.orchestra.run.smithi197.stdout:Get:80 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 15.2.16-304-gf25ffac7-1focal [364 kB] 2022-06-29T15:55:42.597 INFO:teuthology.orchestra.run.smithi156.stdout:Get:77 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 15.2.16-304-gf25ffac7-1focal [20.9 kB] 2022-06-29T15:55:42.598 INFO:teuthology.orchestra.run.smithi156.stdout:Get:78 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 15.2.16-304-gf25ffac7-1focal [156 kB] 2022-06-29T15:55:42.603 INFO:teuthology.orchestra.run.smithi197.stdout:Get:81 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 15.2.16-304-gf25ffac7-1focal [3,145 kB] 2022-06-29T15:55:42.667 INFO:teuthology.orchestra.run.smithi156.stdout:Get:79 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 15.2.16-304-gf25ffac7-1focal [29.4 kB] 2022-06-29T15:55:42.669 INFO:teuthology.orchestra.run.smithi156.stdout:Get:80 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 15.2.16-304-gf25ffac7-1focal [364 kB] 2022-06-29T15:55:42.816 INFO:teuthology.orchestra.run.smithi156.stdout:Get:81 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 15.2.16-304-gf25ffac7-1focal [3,145 kB] 2022-06-29T15:55:43.306 INFO:teuthology.orchestra.run.smithi092.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 15.2.16-304-gf25ffac7-1focal [5,344 kB] 2022-06-29T15:55:43.532 INFO:teuthology.orchestra.run.smithi060.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 15.2.16-304-gf25ffac7-1focal [5,344 kB] 2022-06-29T15:55:43.718 INFO:teuthology.orchestra.run.smithi092.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 15.2.16-304-gf25ffac7-1focal [163 kB] 2022-06-29T15:55:43.734 INFO:teuthology.orchestra.run.smithi197.stdout:Get:82 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 15.2.16-304-gf25ffac7-1focal [116 kB] 2022-06-29T15:55:43.737 INFO:teuthology.orchestra.run.smithi197.stdout:Get:83 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 15.2.16-304-gf25ffac7-1focal [361 kB] 2022-06-29T15:55:43.776 INFO:teuthology.orchestra.run.smithi092.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 15.2.16-304-gf25ffac7-1focal [1,402 kB] 2022-06-29T15:55:43.875 INFO:teuthology.orchestra.run.smithi092.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 15.2.16-304-gf25ffac7-1focal [6,164 kB] 2022-06-29T15:55:43.894 INFO:teuthology.orchestra.run.smithi197.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 15.2.16-304-gf25ffac7-1focal [20.1 MB] 2022-06-29T15:55:44.050 INFO:teuthology.orchestra.run.smithi156.stdout:Get:82 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 15.2.16-304-gf25ffac7-1focal [116 kB] 2022-06-29T15:55:44.077 INFO:teuthology.orchestra.run.smithi060.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 15.2.16-304-gf25ffac7-1focal [163 kB] 2022-06-29T15:55:44.082 INFO:teuthology.orchestra.run.smithi060.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 15.2.16-304-gf25ffac7-1focal [1,402 kB] 2022-06-29T15:55:44.128 INFO:teuthology.orchestra.run.smithi156.stdout:Get:83 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 15.2.16-304-gf25ffac7-1focal [361 kB] 2022-06-29T15:55:44.216 INFO:teuthology.orchestra.run.smithi156.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 15.2.16-304-gf25ffac7-1focal [20.1 MB] 2022-06-29T15:55:44.242 INFO:teuthology.orchestra.run.smithi060.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 15.2.16-304-gf25ffac7-1focal [6,164 kB] 2022-06-29T15:55:44.303 INFO:teuthology.orchestra.run.smithi121.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 15.2.16-304-gf25ffac7-1focal [5,344 kB] 2022-06-29T15:55:44.571 INFO:teuthology.orchestra.run.smithi092.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 15.2.16-304-gf25ffac7-1focal [24.0 MB] 2022-06-29T15:55:44.836 INFO:teuthology.orchestra.run.smithi060.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 15.2.16-304-gf25ffac7-1focal [24.0 MB] 2022-06-29T15:55:45.076 INFO:teuthology.orchestra.run.smithi121.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 15.2.16-304-gf25ffac7-1focal [163 kB] 2022-06-29T15:55:45.082 INFO:teuthology.orchestra.run.smithi121.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 15.2.16-304-gf25ffac7-1focal [1,402 kB] 2022-06-29T15:55:45.311 INFO:teuthology.orchestra.run.smithi121.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 15.2.16-304-gf25ffac7-1focal [6,164 kB] 2022-06-29T15:55:46.310 INFO:teuthology.orchestra.run.smithi121.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 15.2.16-304-gf25ffac7-1focal [24.0 MB] 2022-06-29T15:55:47.259 INFO:teuthology.orchestra.run.smithi060.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 15.2.16-304-gf25ffac7-1focal [4,032 B] 2022-06-29T15:55:47.260 INFO:teuthology.orchestra.run.smithi060.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 15.2.16-304-gf25ffac7-1focal [538 kB] 2022-06-29T15:55:47.269 INFO:teuthology.orchestra.run.smithi060.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 15.2.16-304-gf25ffac7-1focal [2,007 kB] 2022-06-29T15:55:47.443 INFO:teuthology.orchestra.run.smithi060.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 15.2.16-304-gf25ffac7-1focal [53.2 kB] 2022-06-29T15:55:47.445 INFO:teuthology.orchestra.run.smithi060.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 15.2.16-304-gf25ffac7-1focal [59.1 kB] 2022-06-29T15:55:47.447 INFO:teuthology.orchestra.run.smithi060.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 15.2.16-304-gf25ffac7-1focal [3,218 kB] 2022-06-29T15:55:47.581 INFO:teuthology.orchestra.run.smithi092.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 15.2.16-304-gf25ffac7-1focal [4,032 B] 2022-06-29T15:55:47.582 INFO:teuthology.orchestra.run.smithi092.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 15.2.16-304-gf25ffac7-1focal [538 kB] 2022-06-29T15:55:47.592 INFO:teuthology.orchestra.run.smithi092.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 15.2.16-304-gf25ffac7-1focal [2,007 kB] 2022-06-29T15:55:47.767 INFO:teuthology.orchestra.run.smithi060.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-cloud all 15.2.16-304-gf25ffac7-1focal [34.4 kB] 2022-06-29T15:55:47.768 INFO:teuthology.orchestra.run.smithi060.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 15.2.16-304-gf25ffac7-1focal [7,866 kB] 2022-06-29T15:55:47.817 INFO:teuthology.orchestra.run.smithi092.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 15.2.16-304-gf25ffac7-1focal [53.2 kB] 2022-06-29T15:55:47.819 INFO:teuthology.orchestra.run.smithi092.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 15.2.16-304-gf25ffac7-1focal [59.1 kB] 2022-06-29T15:55:47.821 INFO:teuthology.orchestra.run.smithi092.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 15.2.16-304-gf25ffac7-1focal [3,218 kB] 2022-06-29T15:55:48.220 INFO:teuthology.orchestra.run.smithi092.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-cloud all 15.2.16-304-gf25ffac7-1focal [34.4 kB] 2022-06-29T15:55:48.221 INFO:teuthology.orchestra.run.smithi092.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 15.2.16-304-gf25ffac7-1focal [7,866 kB] 2022-06-29T15:55:48.527 INFO:teuthology.orchestra.run.smithi060.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 15.2.16-304-gf25ffac7-1focal [4,144 B] 2022-06-29T15:55:48.528 INFO:teuthology.orchestra.run.smithi060.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 15.2.16-304-gf25ffac7-1focal [40.2 MB] 2022-06-29T15:55:49.114 INFO:teuthology.orchestra.run.smithi092.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 15.2.16-304-gf25ffac7-1focal [4,144 B] 2022-06-29T15:55:49.114 INFO:teuthology.orchestra.run.smithi092.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 15.2.16-304-gf25ffac7-1focal [40.2 MB] 2022-06-29T15:55:50.304 INFO:teuthology.orchestra.run.smithi121.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 15.2.16-304-gf25ffac7-1focal [4,032 B] 2022-06-29T15:55:50.304 INFO:teuthology.orchestra.run.smithi121.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 15.2.16-304-gf25ffac7-1focal [538 kB] 2022-06-29T15:55:50.315 INFO:teuthology.orchestra.run.smithi121.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 15.2.16-304-gf25ffac7-1focal [2,007 kB] 2022-06-29T15:55:50.603 INFO:teuthology.orchestra.run.smithi121.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 15.2.16-304-gf25ffac7-1focal [53.2 kB] 2022-06-29T15:55:50.605 INFO:teuthology.orchestra.run.smithi121.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 15.2.16-304-gf25ffac7-1focal [59.1 kB] 2022-06-29T15:55:50.607 INFO:teuthology.orchestra.run.smithi121.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 15.2.16-304-gf25ffac7-1focal [3,218 kB] 2022-06-29T15:55:50.855 INFO:teuthology.orchestra.run.smithi197.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 15.2.16-304-gf25ffac7-1focal [5,344 kB] 2022-06-29T15:55:51.080 INFO:teuthology.orchestra.run.smithi121.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-cloud all 15.2.16-304-gf25ffac7-1focal [34.4 kB] 2022-06-29T15:55:51.111 INFO:teuthology.orchestra.run.smithi121.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 15.2.16-304-gf25ffac7-1focal [7,866 kB] 2022-06-29T15:55:51.791 INFO:teuthology.orchestra.run.smithi156.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 15.2.16-304-gf25ffac7-1focal [5,344 kB] 2022-06-29T15:55:52.230 INFO:teuthology.orchestra.run.smithi121.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 15.2.16-304-gf25ffac7-1focal [4,144 B] 2022-06-29T15:55:52.231 INFO:teuthology.orchestra.run.smithi121.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 15.2.16-304-gf25ffac7-1focal [40.2 MB] 2022-06-29T15:55:52.484 INFO:teuthology.orchestra.run.smithi060.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 cephfs-shell all 15.2.16-304-gf25ffac7-1focal [15.9 kB] 2022-06-29T15:55:52.485 INFO:teuthology.orchestra.run.smithi060.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 15.2.16-304-gf25ffac7-1focal [18.1 kB] 2022-06-29T15:55:52.485 INFO:teuthology.orchestra.run.smithi060.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 15.2.16-304-gf25ffac7-1focal [9,668 kB] 2022-06-29T15:55:52.503 INFO:teuthology.orchestra.run.smithi197.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 15.2.16-304-gf25ffac7-1focal [163 kB] 2022-06-29T15:55:52.570 INFO:teuthology.orchestra.run.smithi197.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 15.2.16-304-gf25ffac7-1focal [1,402 kB] 2022-06-29T15:55:52.936 INFO:teuthology.orchestra.run.smithi197.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 15.2.16-304-gf25ffac7-1focal [6,164 kB] 2022-06-29T15:55:53.194 INFO:teuthology.orchestra.run.smithi060.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 15.2.16-304-gf25ffac7-1focal [60.9 kB] 2022-06-29T15:55:53.364 INFO:teuthology.orchestra.run.smithi156.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 15.2.16-304-gf25ffac7-1focal [163 kB] 2022-06-29T15:55:53.384 INFO:teuthology.orchestra.run.smithi156.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 15.2.16-304-gf25ffac7-1focal [1,402 kB] 2022-06-29T15:55:53.663 INFO:teuthology.orchestra.run.smithi156.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 15.2.16-304-gf25ffac7-1focal [6,164 kB] 2022-06-29T15:55:53.852 INFO:teuthology.orchestra.run.smithi060.stdout:Fetched 153 MB in 13s (11.4 MB/s) 2022-06-29T15:55:54.119 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pyperclip. 2022-06-29T15:55:54.174 INFO:teuthology.orchestra.run.smithi060.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 138771 files and directories currently installed.) 2022-06-29T15:55:54.178 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../000-python3-pyperclip_1.7.0-1_all.deb ... 2022-06-29T15:55:54.208 INFO:teuthology.orchestra.run.smithi197.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 15.2.16-304-gf25ffac7-1focal [24.0 MB] 2022-06-29T15:55:54.234 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pyperclip (1.7.0-1) ... 2022-06-29T15:55:54.321 INFO:teuthology.orchestra.run.smithi092.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 cephfs-shell all 15.2.16-304-gf25ffac7-1focal [15.9 kB] 2022-06-29T15:55:54.321 INFO:teuthology.orchestra.run.smithi092.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 15.2.16-304-gf25ffac7-1focal [18.1 kB] 2022-06-29T15:55:54.322 INFO:teuthology.orchestra.run.smithi092.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 15.2.16-304-gf25ffac7-1focal [9,668 kB] 2022-06-29T15:55:54.528 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-wcwidth. 2022-06-29T15:55:54.544 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../001-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-06-29T15:55:54.587 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-06-29T15:55:54.881 INFO:teuthology.orchestra.run.smithi156.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 15.2.16-304-gf25ffac7-1focal [24.0 MB] 2022-06-29T15:55:54.923 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-cmd2. 2022-06-29T15:55:54.939 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../002-python3-cmd2_0.8.5-2.1_all.deb ... 2022-06-29T15:55:54.981 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-cmd2 (0.8.5-2.1) ... 2022-06-29T15:55:55.183 INFO:teuthology.orchestra.run.smithi092.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 15.2.16-304-gf25ffac7-1focal [60.9 kB] 2022-06-29T15:55:55.325 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-06-29T15:55:55.344 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../003-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-06-29T15:55:55.417 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-06-29T15:55:55.695 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../004-librbd1_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:55:55.820 INFO:teuthology.orchestra.run.smithi092.stdout:Fetched 153 MB in 15s (9,961 kB/s) 2022-06-29T15:55:55.821 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking librbd1 (15.2.16-304-gf25ffac7-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-06-29T15:55:55.964 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-pyperclip. 2022-06-29T15:55:56.018 INFO:teuthology.orchestra.run.smithi092.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 138771 files and directories currently installed.) 2022-06-29T15:55:56.022 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../000-python3-pyperclip_1.7.0-1_all.deb ... 2022-06-29T15:55:56.081 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-pyperclip (1.7.0-1) ... 2022-06-29T15:55:56.194 INFO:teuthology.orchestra.run.smithi121.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 cephfs-shell all 15.2.16-304-gf25ffac7-1focal [15.9 kB] 2022-06-29T15:55:56.194 INFO:teuthology.orchestra.run.smithi121.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 15.2.16-304-gf25ffac7-1focal [18.1 kB] 2022-06-29T15:55:56.195 INFO:teuthology.orchestra.run.smithi121.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 15.2.16-304-gf25ffac7-1focal [9,668 kB] 2022-06-29T15:55:56.361 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-wcwidth. 2022-06-29T15:55:56.379 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../001-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-06-29T15:55:56.382 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../005-librados2_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:55:56.415 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-06-29T15:55:56.508 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking librados2 (15.2.16-304-gf25ffac7-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-06-29T15:55:56.652 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-cmd2. 2022-06-29T15:55:56.670 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../002-python3-cmd2_0.8.5-2.1_all.deb ... 2022-06-29T15:55:56.672 INFO:teuthology.orchestra.run.smithi121.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 15.2.16-304-gf25ffac7-1focal [60.9 kB] 2022-06-29T15:55:56.701 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-cmd2 (0.8.5-2.1) ... 2022-06-29T15:55:57.054 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-06-29T15:55:57.071 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../003-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-06-29T15:55:57.154 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-06-29T15:55:57.162 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libcephfs2. 2022-06-29T15:55:57.178 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../006-libcephfs2_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:55:57.212 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libcephfs2 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:55:57.257 INFO:teuthology.orchestra.run.smithi121.stdout:Fetched 153 MB in 17s (9,083 kB/s) 2022-06-29T15:55:57.381 INFO:teuthology.orchestra.run.smithi197.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 15.2.16-304-gf25ffac7-1focal [4,032 B] 2022-06-29T15:55:57.382 INFO:teuthology.orchestra.run.smithi197.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 15.2.16-304-gf25ffac7-1focal [538 kB] 2022-06-29T15:55:57.391 INFO:teuthology.orchestra.run.smithi197.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 15.2.16-304-gf25ffac7-1focal [2,007 kB] 2022-06-29T15:55:57.399 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pyperclip. 2022-06-29T15:55:57.454 INFO:teuthology.orchestra.run.smithi121.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 138771 files and directories currently installed.) 2022-06-29T15:55:57.458 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../000-python3-pyperclip_1.7.0-1_all.deb ... 2022-06-29T15:55:57.498 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pyperclip (1.7.0-1) ... 2022-06-29T15:55:57.499 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../004-librbd1_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:55:57.514 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-rados. 2022-06-29T15:55:57.530 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../007-python3-rados_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:55:57.546 INFO:teuthology.orchestra.run.smithi197.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 15.2.16-304-gf25ffac7-1focal [53.2 kB] 2022-06-29T15:55:57.547 INFO:teuthology.orchestra.run.smithi197.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 15.2.16-304-gf25ffac7-1focal [59.1 kB] 2022-06-29T15:55:57.564 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-rados (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:55:57.565 INFO:teuthology.orchestra.run.smithi197.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 15.2.16-304-gf25ffac7-1focal [3,218 kB] 2022-06-29T15:55:57.625 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking librbd1 (15.2.16-304-gf25ffac7-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-06-29T15:55:57.744 INFO:teuthology.orchestra.run.smithi156.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 15.2.16-304-gf25ffac7-1focal [4,032 B] 2022-06-29T15:55:57.745 INFO:teuthology.orchestra.run.smithi156.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 15.2.16-304-gf25ffac7-1focal [538 kB] 2022-06-29T15:55:57.755 INFO:teuthology.orchestra.run.smithi156.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 15.2.16-304-gf25ffac7-1focal [2,007 kB] 2022-06-29T15:55:57.767 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-wcwidth. 2022-06-29T15:55:57.787 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../001-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-06-29T15:55:57.797 INFO:teuthology.orchestra.run.smithi197.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-cloud all 15.2.16-304-gf25ffac7-1focal [34.4 kB] 2022-06-29T15:55:57.798 INFO:teuthology.orchestra.run.smithi197.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 15.2.16-304-gf25ffac7-1focal [7,866 kB] 2022-06-29T15:55:57.817 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-06-29T15:55:57.892 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-06-29T15:55:57.906 INFO:teuthology.orchestra.run.smithi156.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 15.2.16-304-gf25ffac7-1focal [53.2 kB] 2022-06-29T15:55:57.908 INFO:teuthology.orchestra.run.smithi156.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 15.2.16-304-gf25ffac7-1focal [59.1 kB] 2022-06-29T15:55:57.909 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../008-python3-ceph-argparse_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:55:57.924 INFO:teuthology.orchestra.run.smithi156.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 15.2.16-304-gf25ffac7-1focal [3,218 kB] 2022-06-29T15:55:57.950 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-ceph-argparse (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:55:58.104 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-cmd2. 2022-06-29T15:55:58.119 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../002-python3-cmd2_0.8.5-2.1_all.deb ... 2022-06-29T15:55:58.157 INFO:teuthology.orchestra.run.smithi156.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-cloud all 15.2.16-304-gf25ffac7-1focal [34.4 kB] 2022-06-29T15:55:58.162 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-cmd2 (0.8.5-2.1) ... 2022-06-29T15:55:58.165 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../005-librados2_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:55:58.170 INFO:teuthology.orchestra.run.smithi156.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 15.2.16-304-gf25ffac7-1focal [7,866 kB] 2022-06-29T15:55:58.278 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-cephfs. 2022-06-29T15:55:58.287 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking librados2 (15.2.16-304-gf25ffac7-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-06-29T15:55:58.296 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../009-python3-cephfs_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:55:58.336 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-cephfs (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:55:58.374 INFO:teuthology.orchestra.run.smithi197.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 15.2.16-304-gf25ffac7-1focal [4,144 B] 2022-06-29T15:55:58.375 INFO:teuthology.orchestra.run.smithi197.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 15.2.16-304-gf25ffac7-1focal [40.2 MB] 2022-06-29T15:55:58.540 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-06-29T15:55:58.555 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../003-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-06-29T15:55:58.640 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-06-29T15:55:58.664 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-ceph-common. 2022-06-29T15:55:58.681 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../010-python3-ceph-common_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:55:58.723 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-ceph-common (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:55:58.741 INFO:teuthology.orchestra.run.smithi156.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 15.2.16-304-gf25ffac7-1focal [4,144 B] 2022-06-29T15:55:58.742 INFO:teuthology.orchestra.run.smithi156.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 15.2.16-304-gf25ffac7-1focal [40.2 MB] 2022-06-29T15:55:58.941 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libcephfs2. 2022-06-29T15:55:58.957 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../006-libcephfs2_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:55:58.979 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../004-librbd1_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:55:58.991 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libcephfs2 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:55:59.034 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-prettytable. 2022-06-29T15:55:59.050 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../011-python3-prettytable_0.7.2-5_all.deb ... 2022-06-29T15:55:59.084 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-06-29T15:55:59.087 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking librbd1 (15.2.16-304-gf25ffac7-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-06-29T15:55:59.260 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-rados. 2022-06-29T15:55:59.276 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../007-python3-rados_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:55:59.320 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-rbd. 2022-06-29T15:55:59.335 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-rados (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:55:59.337 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../012-python3-rbd_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:55:59.378 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-rbd (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:55:59.623 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../005-librados2_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:55:59.713 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-06-29T15:55:59.729 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../008-python3-ceph-argparse_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:55:59.764 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-06-29T15:55:59.771 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-ceph-argparse (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:55:59.781 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../013-librabbitmq4_0.10.0-1_amd64.deb ... 2022-06-29T15:55:59.783 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking librados2 (15.2.16-304-gf25ffac7-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-06-29T15:55:59.822 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-06-29T15:56:00.074 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-cephfs. 2022-06-29T15:56:00.093 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../009-python3-cephfs_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:00.132 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-cephfs (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:00.133 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-06-29T15:56:00.152 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../014-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-06-29T15:56:00.192 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-06-29T15:56:00.410 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-ceph-common. 2022-06-29T15:56:00.420 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libcephfs2. 2022-06-29T15:56:00.425 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../010-python3-ceph-common_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:00.436 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../006-libcephfs2_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:00.462 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-ceph-common (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:00.470 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libcephfs2 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:00.578 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package librgw2. 2022-06-29T15:56:00.596 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../015-librgw2_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:00.636 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking librgw2 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:00.781 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-prettytable. 2022-06-29T15:56:00.801 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../011-python3-prettytable_0.7.2-5_all.deb ... 2022-06-29T15:56:00.806 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-rados. 2022-06-29T15:56:00.823 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../007-python3-rados_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:00.840 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-06-29T15:56:00.864 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-rados (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:01.134 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-rbd. 2022-06-29T15:56:01.153 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../012-python3-rbd_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:01.192 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-rbd (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:01.214 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-rgw. 2022-06-29T15:56:01.225 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-06-29T15:56:01.232 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../016-python3-rgw_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:01.244 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../008-python3-ceph-argparse_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:01.272 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-rgw (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:01.284 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-ceph-argparse (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:01.516 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libleveldb1d:amd64. 2022-06-29T15:56:01.533 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../017-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2022-06-29T15:56:01.537 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-06-29T15:56:01.553 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-cephfs. 2022-06-29T15:56:01.554 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../013-librabbitmq4_0.10.0-1_amd64.deb ... 2022-06-29T15:56:01.566 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-06-29T15:56:01.569 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../009-python3-cephfs_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:01.594 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-06-29T15:56:01.636 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-cephfs (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:01.906 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-ceph-common. 2022-06-29T15:56:01.924 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../010-python3-ceph-common_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:01.948 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package liboath0. 2022-06-29T15:56:01.956 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-06-29T15:56:01.962 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../018-liboath0_2.6.1-1.3_amd64.deb ... 2022-06-29T15:56:01.964 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-ceph-common (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:01.975 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../014-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-06-29T15:56:02.002 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-06-29T15:56:02.014 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-06-29T15:56:02.275 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-prettytable. 2022-06-29T15:56:02.291 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../011-python3-prettytable_0.7.2-5_all.deb ... 2022-06-29T15:56:02.299 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libradosstriper1. 2022-06-29T15:56:02.313 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../019-libradosstriper1_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:02.334 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-06-29T15:56:02.355 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libradosstriper1 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:02.358 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package librgw2. 2022-06-29T15:56:02.374 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../015-librgw2_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:02.411 INFO:teuthology.orchestra.run.smithi197.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 cephfs-shell all 15.2.16-304-gf25ffac7-1focal [15.9 kB] 2022-06-29T15:56:02.412 INFO:teuthology.orchestra.run.smithi197.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 15.2.16-304-gf25ffac7-1focal [18.1 kB] 2022-06-29T15:56:02.413 INFO:teuthology.orchestra.run.smithi197.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 15.2.16-304-gf25ffac7-1focal [9,668 kB] 2022-06-29T15:56:02.416 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking librgw2 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:02.611 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-rbd. 2022-06-29T15:56:02.627 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../012-python3-rbd_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:02.678 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-rbd (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:02.750 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-common. 2022-06-29T15:56:02.766 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../020-ceph-common_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:02.799 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-common (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:02.952 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-rgw. 2022-06-29T15:56:02.971 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../016-python3-rgw_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:03.010 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-rgw (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:03.073 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-06-29T15:56:03.089 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../013-librabbitmq4_0.10.0-1_amd64.deb ... 2022-06-29T15:56:03.131 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-06-29T15:56:03.338 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libleveldb1d:amd64. 2022-06-29T15:56:03.358 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../017-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2022-06-29T15:56:03.396 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-06-29T15:56:03.434 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-06-29T15:56:03.450 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../014-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-06-29T15:56:03.484 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-06-29T15:56:03.741 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package liboath0. 2022-06-29T15:56:03.758 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../018-liboath0_2.6.1-1.3_amd64.deb ... 2022-06-29T15:56:03.799 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-06-29T15:56:03.828 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package librgw2. 2022-06-29T15:56:03.848 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../015-librgw2_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:03.887 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking librgw2 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:03.976 INFO:teuthology.orchestra.run.smithi156.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 cephfs-shell all 15.2.16-304-gf25ffac7-1focal [15.9 kB] 2022-06-29T15:56:03.977 INFO:teuthology.orchestra.run.smithi156.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 15.2.16-304-gf25ffac7-1focal [18.1 kB] 2022-06-29T15:56:03.977 INFO:teuthology.orchestra.run.smithi156.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 15.2.16-304-gf25ffac7-1focal [9,668 kB] 2022-06-29T15:56:04.051 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libradosstriper1. 2022-06-29T15:56:04.067 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../019-libradosstriper1_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:04.101 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libradosstriper1 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:04.102 INFO:teuthology.orchestra.run.smithi197.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 15.2.16-304-gf25ffac7-1focal [60.9 kB] 2022-06-29T15:56:04.422 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-rgw. 2022-06-29T15:56:04.438 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../016-python3-rgw_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:04.481 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-rgw (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:04.530 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-common. 2022-06-29T15:56:04.545 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../020-ceph-common_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:04.587 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-common (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:04.690 INFO:teuthology.orchestra.run.smithi197.stdout:Fetched 153 MB in 24s (6,340 kB/s) 2022-06-29T15:56:04.800 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libleveldb1d:amd64. 2022-06-29T15:56:04.815 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../017-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2022-06-29T15:56:04.850 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-06-29T15:56:04.854 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-pyperclip. 2022-06-29T15:56:04.897 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-base. 2022-06-29T15:56:04.907 INFO:teuthology.orchestra.run.smithi197.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 138771 files and directories currently installed.) 2022-06-29T15:56:04.911 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../000-python3-pyperclip_1.7.0-1_all.deb ... 2022-06-29T15:56:04.913 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../021-ceph-base_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:04.952 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-pyperclip (1.7.0-1) ... 2022-06-29T15:56:04.966 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-base (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:05.195 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package liboath0. 2022-06-29T15:56:05.210 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../018-liboath0_2.6.1-1.3_amd64.deb ... 2022-06-29T15:56:05.254 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-06-29T15:56:05.255 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-wcwidth. 2022-06-29T15:56:05.271 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../001-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-06-29T15:56:05.284 INFO:teuthology.orchestra.run.smithi156.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-yuri5-testing-2022-06-27-0923-octopus/f25ffac72e5a9e63860ff7deb6767a11f0559a16/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 15.2.16-304-gf25ffac7-1focal [60.9 kB] 2022-06-29T15:56:05.338 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-06-29T15:56:05.597 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libradosstriper1. 2022-06-29T15:56:05.614 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../019-libradosstriper1_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:05.616 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-cmd2. 2022-06-29T15:56:05.633 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../002-python3-cmd2_0.8.5-2.1_all.deb ... 2022-06-29T15:56:05.656 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libradosstriper1 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:05.666 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-cmd2 (0.8.5-2.1) ... 2022-06-29T15:56:05.868 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-cryptography. 2022-06-29T15:56:05.873 INFO:teuthology.orchestra.run.smithi156.stdout:Fetched 153 MB in 25s (6,045 kB/s) 2022-06-29T15:56:05.888 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../022-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-06-29T15:56:05.926 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-06-29T15:56:05.986 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-06-29T15:56:06.004 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../003-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-06-29T15:56:06.021 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-pyperclip. 2022-06-29T15:56:06.069 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-06-29T15:56:06.076 INFO:teuthology.orchestra.run.smithi156.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 138771 files and directories currently installed.) 2022-06-29T15:56:06.080 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../000-python3-pyperclip_1.7.0-1_all.deb ... 2022-06-29T15:56:06.117 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-common. 2022-06-29T15:56:06.133 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../020-ceph-common_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:06.144 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-pyperclip (1.7.0-1) ... 2022-06-29T15:56:06.175 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-common (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:06.374 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-openssl. 2022-06-29T15:56:06.389 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../023-python3-openssl_19.0.0-1build1_all.deb ... 2022-06-29T15:56:06.422 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../004-librbd1_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:06.430 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-06-29T15:56:06.431 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-wcwidth. 2022-06-29T15:56:06.446 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../001-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-06-29T15:56:06.505 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-06-29T15:56:06.548 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking librbd1 (15.2.16-304-gf25ffac7-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-06-29T15:56:06.659 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-base. 2022-06-29T15:56:06.676 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../021-ceph-base_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:06.699 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-06-29T15:56:06.719 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../024-ceph-mgr-modules-core_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:06.720 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-base (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:06.742 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-cmd2. 2022-06-29T15:56:06.760 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../002-python3-cmd2_0.8.5-2.1_all.deb ... 2022-06-29T15:56:06.792 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-modules-core (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:06.816 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-cmd2 (0.8.5-2.1) ... 2022-06-29T15:56:07.086 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../005-librados2_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:07.178 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-06-29T15:56:07.194 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../003-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-06-29T15:56:07.211 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking librados2 (15.2.16-304-gf25ffac7-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-06-29T15:56:07.278 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-06-29T15:56:07.454 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-bcrypt. 2022-06-29T15:56:07.470 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../025-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-06-29T15:56:07.520 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-06-29T15:56:07.656 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-cryptography. 2022-06-29T15:56:07.656 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../004-librbd1_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:07.672 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../022-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-06-29T15:56:07.713 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-06-29T15:56:07.782 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking librbd1 (15.2.16-304-gf25ffac7-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-06-29T15:56:07.839 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-cherrypy3. 2022-06-29T15:56:07.857 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../026-python3-cherrypy3_8.9.1-6_all.deb ... 2022-06-29T15:56:07.864 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package libcephfs2. 2022-06-29T15:56:07.882 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../006-libcephfs2_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:07.897 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-06-29T15:56:07.922 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking libcephfs2 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:08.134 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-openssl. 2022-06-29T15:56:08.153 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../023-python3-openssl_19.0.0-1build1_all.deb ... 2022-06-29T15:56:08.192 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-06-29T15:56:08.214 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-base. 2022-06-29T15:56:08.230 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../021-ceph-base_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:08.250 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-rados. 2022-06-29T15:56:08.267 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../007-python3-rados_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:08.282 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-base (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:08.284 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-jwt. 2022-06-29T15:56:08.302 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../027-python3-jwt_1.7.1-2ubuntu2_all.deb ... 2022-06-29T15:56:08.308 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-rados (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:08.319 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../005-librados2_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:08.342 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2) ... 2022-06-29T15:56:08.445 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking librados2 (15.2.16-304-gf25ffac7-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-06-29T15:56:08.478 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-06-29T15:56:08.495 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../024-ceph-mgr-modules-core_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:08.545 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-mgr-modules-core (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:08.655 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-logutils. 2022-06-29T15:56:08.661 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-06-29T15:56:08.671 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../028-python3-logutils_0.3.3-7_all.deb ... 2022-06-29T15:56:08.677 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../008-python3-ceph-argparse_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:08.711 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-ceph-argparse (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:08.713 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-06-29T15:56:08.947 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-cephfs. 2022-06-29T15:56:08.957 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-bcrypt. 2022-06-29T15:56:08.962 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../009-python3-cephfs_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:08.975 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../025-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-06-29T15:56:08.997 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-cephfs (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:09.015 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-06-29T15:56:09.040 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-markupsafe. 2022-06-29T15:56:09.060 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../029-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-06-29T15:56:09.098 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-06-29T15:56:09.149 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package libcephfs2. 2022-06-29T15:56:09.160 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-cryptography. 2022-06-29T15:56:09.165 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../006-libcephfs2_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:09.176 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../022-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-06-29T15:56:09.207 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking libcephfs2 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:09.218 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-06-29T15:56:09.333 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-ceph-common. 2022-06-29T15:56:09.352 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../010-python3-ceph-common_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:09.360 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-cherrypy3. 2022-06-29T15:56:09.375 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../026-python3-cherrypy3_8.9.1-6_all.deb ... 2022-06-29T15:56:09.391 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-ceph-common (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:09.393 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-mako. 2022-06-29T15:56:09.412 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../030-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-06-29T15:56:09.444 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-06-29T15:56:09.459 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-06-29T15:56:09.493 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-rados. 2022-06-29T15:56:09.510 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../007-python3-rados_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:09.543 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-rados (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:09.690 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-openssl. 2022-06-29T15:56:09.706 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../023-python3-openssl_19.0.0-1build1_all.deb ... 2022-06-29T15:56:09.747 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-06-29T15:56:09.752 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-prettytable. 2022-06-29T15:56:09.769 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../011-python3-prettytable_0.7.2-5_all.deb ... 2022-06-29T15:56:09.811 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-06-29T15:56:09.812 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-jwt. 2022-06-29T15:56:09.828 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../027-python3-jwt_1.7.1-2ubuntu2_all.deb ... 2022-06-29T15:56:09.831 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-simplegeneric. 2022-06-29T15:56:09.849 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../031-python3-simplegeneric_0.8.1-3_all.deb ... 2022-06-29T15:56:09.871 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2) ... 2022-06-29T15:56:09.889 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-06-29T15:56:09.921 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-06-29T15:56:09.937 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../008-python3-ceph-argparse_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:09.979 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-ceph-argparse (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:10.025 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-06-29T15:56:10.042 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../024-ceph-mgr-modules-core_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:10.083 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mgr-modules-core (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:10.088 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-rbd. 2022-06-29T15:56:10.105 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../012-python3-rbd_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:10.146 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-rbd (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:10.197 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-singledispatch. 2022-06-29T15:56:10.199 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-logutils. 2022-06-29T15:56:10.214 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../032-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-06-29T15:56:10.215 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../028-python3-logutils_0.3.3-7_all.deb ... 2022-06-29T15:56:10.272 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-06-29T15:56:10.290 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-06-29T15:56:10.307 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-cephfs. 2022-06-29T15:56:10.323 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../009-python3-cephfs_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:10.390 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-cephfs (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:10.424 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-06-29T15:56:10.444 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../013-librabbitmq4_0.10.0-1_amd64.deb ... 2022-06-29T15:56:10.446 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-bcrypt. 2022-06-29T15:56:10.462 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../025-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-06-29T15:56:10.492 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-06-29T15:56:10.513 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-06-29T15:56:10.566 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-webob. 2022-06-29T15:56:10.585 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../033-python3-webob_1%3a1.8.5-2_all.deb ... 2022-06-29T15:56:10.593 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-markupsafe. 2022-06-29T15:56:10.608 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../029-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-06-29T15:56:10.624 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-06-29T15:56:10.659 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-06-29T15:56:10.726 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-ceph-common. 2022-06-29T15:56:10.746 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../010-python3-ceph-common_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:10.776 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-ceph-common (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:10.817 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-cherrypy3. 2022-06-29T15:56:10.833 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../026-python3-cherrypy3_8.9.1-6_all.deb ... 2022-06-29T15:56:10.844 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-06-29T15:56:10.861 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../014-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-06-29T15:56:10.875 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-06-29T15:56:10.902 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-06-29T15:56:10.947 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-mako. 2022-06-29T15:56:10.962 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../030-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-06-29T15:56:10.969 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-soupsieve. 2022-06-29T15:56:10.988 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../034-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-06-29T15:56:11.005 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-06-29T15:56:11.027 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-06-29T15:56:11.121 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-prettytable. 2022-06-29T15:56:11.138 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../011-python3-prettytable_0.7.2-5_all.deb ... 2022-06-29T15:56:11.179 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-06-29T15:56:11.236 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-jwt. 2022-06-29T15:56:11.256 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../027-python3-jwt_1.7.1-2ubuntu2_all.deb ... 2022-06-29T15:56:11.272 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package librgw2. 2022-06-29T15:56:11.282 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-bs4. 2022-06-29T15:56:11.289 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../015-librgw2_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:11.294 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2) ... 2022-06-29T15:56:11.301 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../035-python3-bs4_4.8.2-1_all.deb ... 2022-06-29T15:56:11.330 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking librgw2 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:11.340 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-06-29T15:56:11.346 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-simplegeneric. 2022-06-29T15:56:11.366 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../031-python3-simplegeneric_0.8.1-3_all.deb ... 2022-06-29T15:56:11.404 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-06-29T15:56:11.466 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-rbd. 2022-06-29T15:56:11.481 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../012-python3-rbd_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:11.524 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-rbd (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:11.665 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-logutils. 2022-06-29T15:56:11.682 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../028-python3-logutils_0.3.3-7_all.deb ... 2022-06-29T15:56:11.691 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-singledispatch. 2022-06-29T15:56:11.709 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../032-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-06-29T15:56:11.719 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-waitress. 2022-06-29T15:56:11.723 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-06-29T15:56:11.734 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../036-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2022-06-29T15:56:11.780 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2022-06-29T15:56:11.849 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-rgw. 2022-06-29T15:56:11.864 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../016-python3-rgw_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:11.894 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-06-29T15:56:11.907 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-rgw (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:11.911 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../013-librabbitmq4_0.10.0-1_amd64.deb ... 2022-06-29T15:56:11.943 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-06-29T15:56:12.141 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-06-29T15:56:12.210 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package libleveldb1d:amd64. 2022-06-29T15:56:12.227 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../017-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2022-06-29T15:56:12.268 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-06-29T15:56:12.437 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-markupsafe. 2022-06-29T15:56:12.456 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../029-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-06-29T15:56:12.739 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-tempita. 2022-06-29T15:56:12.756 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../037-python3-tempita_0.5.2-6_all.deb ... 2022-06-29T15:56:12.796 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-06-29T15:56:12.895 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-06-29T15:56:13.069 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-webob. 2022-06-29T15:56:13.084 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../033-python3-webob_1%3a1.8.5-2_all.deb ... 2022-06-29T15:56:13.105 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-06-29T15:56:13.107 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-paste. 2022-06-29T15:56:13.121 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../014-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-06-29T15:56:13.124 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../038-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-06-29T15:56:13.126 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-06-29T15:56:13.164 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-06-29T15:56:13.198 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-06-29T15:56:13.228 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-mako. 2022-06-29T15:56:13.245 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../030-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-06-29T15:56:13.271 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package liboath0. 2022-06-29T15:56:13.286 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-06-29T15:56:13.291 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../018-liboath0_2.6.1-1.3_amd64.deb ... 2022-06-29T15:56:13.329 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-06-29T15:56:13.454 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-soupsieve. 2022-06-29T15:56:13.471 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../034-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-06-29T15:56:13.492 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package librgw2. 2022-06-29T15:56:13.509 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../015-librgw2_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:13.513 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-06-29T15:56:13.537 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-06-29T15:56:13.554 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../039-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-06-29T15:56:13.575 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking librgw2 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:13.595 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-06-29T15:56:13.615 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package libradosstriper1. 2022-06-29T15:56:13.631 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../019-libradosstriper1_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:13.640 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-simplegeneric. 2022-06-29T15:56:13.657 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../031-python3-simplegeneric_0.8.1-3_all.deb ... 2022-06-29T15:56:13.665 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking libradosstriper1 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:13.698 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-06-29T15:56:13.832 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-bs4. 2022-06-29T15:56:13.848 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../035-python3-bs4_4.8.2-1_all.deb ... 2022-06-29T15:56:13.890 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-06-29T15:56:13.939 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pastedeploy. 2022-06-29T15:56:13.955 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../040-python3-pastedeploy_2.1.0-1_all.deb ... 2022-06-29T15:56:13.997 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-06-29T15:56:14.043 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-singledispatch. 2022-06-29T15:56:14.061 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../032-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-06-29T15:56:14.101 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-06-29T15:56:14.119 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package ceph-common. 2022-06-29T15:56:14.134 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../020-ceph-common_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:14.144 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-rgw. 2022-06-29T15:56:14.161 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../016-python3-rgw_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:14.177 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking ceph-common (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:14.194 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-rgw (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:14.242 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-waitress. 2022-06-29T15:56:14.259 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../036-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2022-06-29T15:56:14.292 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-webtest. 2022-06-29T15:56:14.303 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2022-06-29T15:56:14.312 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../041-python3-webtest_2.0.32-2_all.deb ... 2022-06-29T15:56:14.350 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-06-29T15:56:14.404 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-webob. 2022-06-29T15:56:14.419 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../033-python3-webob_1%3a1.8.5-2_all.deb ... 2022-06-29T15:56:14.495 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-06-29T15:56:14.547 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package libleveldb1d:amd64. 2022-06-29T15:56:14.563 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../017-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2022-06-29T15:56:14.605 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-06-29T15:56:14.662 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-tempita. 2022-06-29T15:56:14.679 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../037-python3-tempita_0.5.2-6_all.deb ... 2022-06-29T15:56:14.712 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pecan. 2022-06-29T15:56:14.720 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-06-29T15:56:14.729 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../042-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-06-29T15:56:14.770 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-06-29T15:56:14.823 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-soupsieve. 2022-06-29T15:56:14.843 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../034-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-06-29T15:56:14.881 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-06-29T15:56:14.992 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package liboath0. 2022-06-29T15:56:15.007 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-paste. 2022-06-29T15:56:15.011 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../018-liboath0_2.6.1-1.3_amd64.deb ... 2022-06-29T15:56:15.025 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../038-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-06-29T15:56:15.058 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-06-29T15:56:15.064 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-06-29T15:56:15.117 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-werkzeug. 2022-06-29T15:56:15.136 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../043-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-06-29T15:56:15.174 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-06-29T15:56:15.192 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-bs4. 2022-06-29T15:56:15.212 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../035-python3-bs4_4.8.2-1_all.deb ... 2022-06-29T15:56:15.250 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-06-29T15:56:15.361 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package libradosstriper1. 2022-06-29T15:56:15.376 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-06-29T15:56:15.380 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../019-libradosstriper1_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:15.392 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../039-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-06-29T15:56:15.411 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking libradosstriper1 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:15.459 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-06-29T15:56:15.561 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr. 2022-06-29T15:56:15.577 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../044-ceph-mgr_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:15.619 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:15.620 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-waitress. 2022-06-29T15:56:15.637 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../036-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2022-06-29T15:56:15.673 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2022-06-29T15:56:15.762 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-pastedeploy. 2022-06-29T15:56:15.779 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../040-python3-pastedeploy_2.1.0-1_all.deb ... 2022-06-29T15:56:15.820 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-06-29T15:56:15.856 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package ceph-common. 2022-06-29T15:56:15.875 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../020-ceph-common_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:15.914 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking ceph-common (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:16.019 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-tempita. 2022-06-29T15:56:16.036 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../037-python3-tempita_0.5.2-6_all.deb ... 2022-06-29T15:56:16.077 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-06-29T15:56:16.114 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mon. 2022-06-29T15:56:16.130 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../045-ceph-mon_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:16.140 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-webtest. 2022-06-29T15:56:16.158 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../041-python3-webtest_2.0.32-2_all.deb ... 2022-06-29T15:56:16.172 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mon (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:16.206 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-06-29T15:56:16.207 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package ceph-base. 2022-06-29T15:56:16.222 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../021-ceph-base_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:16.276 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking ceph-base (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:16.396 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-paste. 2022-06-29T15:56:16.415 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../038-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-06-29T15:56:16.462 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-06-29T15:56:16.526 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-pecan. 2022-06-29T15:56:16.545 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../042-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-06-29T15:56:16.584 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-06-29T15:56:16.766 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-06-29T15:56:16.784 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../039-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-06-29T15:56:16.824 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-06-29T15:56:16.914 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-werkzeug. 2022-06-29T15:56:16.933 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../043-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-06-29T15:56:16.972 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-06-29T15:56:17.185 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-osd. 2022-06-29T15:56:17.186 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pastedeploy. 2022-06-29T15:56:17.201 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../040-python3-pastedeploy_2.1.0-1_all.deb ... 2022-06-29T15:56:17.203 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../046-ceph-osd_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:17.228 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-cryptography. 2022-06-29T15:56:17.242 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-osd (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:17.245 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../022-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-06-29T15:56:17.252 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-06-29T15:56:17.287 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-06-29T15:56:17.367 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-mgr. 2022-06-29T15:56:17.384 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../044-ceph-mgr_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:17.425 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-mgr (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:17.588 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-webtest. 2022-06-29T15:56:17.607 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../041-python3-webtest_2.0.32-2_all.deb ... 2022-06-29T15:56:17.646 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-06-29T15:56:17.765 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-openssl. 2022-06-29T15:56:17.781 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../023-python3-openssl_19.0.0-1build1_all.deb ... 2022-06-29T15:56:17.823 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-06-29T15:56:17.937 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-mon. 2022-06-29T15:56:17.954 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../045-ceph-mon_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:17.987 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package ceph-base. 2022-06-29T15:56:17.991 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pecan. 2022-06-29T15:56:17.994 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-mon (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:18.006 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../021-ceph-base_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:18.011 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../042-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-06-29T15:56:18.049 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-06-29T15:56:18.058 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking ceph-base (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:18.118 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-06-29T15:56:18.136 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../024-ceph-mgr-modules-core_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:18.185 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking ceph-mgr-modules-core (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:18.421 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-werkzeug. 2022-06-29T15:56:18.437 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../043-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-06-29T15:56:18.479 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-06-29T15:56:18.572 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-bcrypt. 2022-06-29T15:56:18.589 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../025-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-06-29T15:56:18.630 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-06-29T15:56:18.858 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mgr. 2022-06-29T15:56:18.874 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../044-ceph-mgr_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:18.908 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mgr (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:18.934 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-cherrypy3. 2022-06-29T15:56:18.950 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../026-python3-cherrypy3_8.9.1-6_all.deb ... 2022-06-29T15:56:18.957 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-osd. 2022-06-29T15:56:18.972 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../046-ceph-osd_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:19.015 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-osd (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:19.026 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-06-29T15:56:19.242 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-cryptography. 2022-06-29T15:56:19.257 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../022-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-06-29T15:56:19.300 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-06-29T15:56:19.380 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-jwt. 2022-06-29T15:56:19.394 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mon. 2022-06-29T15:56:19.400 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../027-python3-jwt_1.7.1-2ubuntu2_all.deb ... 2022-06-29T15:56:19.413 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../045-ceph-mon_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:19.453 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mon (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:19.463 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2) ... 2022-06-29T15:56:19.606 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph. 2022-06-29T15:56:19.625 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../047-ceph_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:19.656 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:19.775 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-logutils. 2022-06-29T15:56:19.787 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-openssl. 2022-06-29T15:56:19.795 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../028-python3-logutils_0.3.3-7_all.deb ... 2022-06-29T15:56:19.805 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../023-python3-openssl_19.0.0-1build1_all.deb ... 2022-06-29T15:56:19.846 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-06-29T15:56:19.883 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-06-29T15:56:19.967 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-fuse. 2022-06-29T15:56:19.987 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../048-ceph-fuse_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:20.025 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-fuse (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:20.123 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-06-29T15:56:20.143 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../024-ceph-mgr-modules-core_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:20.182 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking ceph-mgr-modules-core (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:20.213 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-markupsafe. 2022-06-29T15:56:20.228 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../029-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-06-29T15:56:20.271 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-06-29T15:56:20.429 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mds. 2022-06-29T15:56:20.448 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../049-ceph-mds_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:20.473 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-osd. 2022-06-29T15:56:20.487 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mds (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:20.489 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../046-ceph-osd_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:20.531 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-osd (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:20.562 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-bcrypt. 2022-06-29T15:56:20.580 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../025-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-06-29T15:56:20.595 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-mako. 2022-06-29T15:56:20.613 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../030-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-06-29T15:56:20.620 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-06-29T15:56:20.654 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-06-29T15:56:20.965 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-cherrypy3. 2022-06-29T15:56:20.981 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../026-python3-cherrypy3_8.9.1-6_all.deb ... 2022-06-29T15:56:21.007 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package cephadm. 2022-06-29T15:56:21.007 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-simplegeneric. 2022-06-29T15:56:21.023 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../050-cephadm_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:21.024 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-06-29T15:56:21.024 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../031-python3-simplegeneric_0.8.1-3_all.deb ... 2022-06-29T15:56:21.065 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking cephadm (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:21.065 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-06-29T15:56:21.388 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-jwt. 2022-06-29T15:56:21.393 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-singledispatch. 2022-06-29T15:56:21.405 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../027-python3-jwt_1.7.1-2ubuntu2_all.deb ... 2022-06-29T15:56:21.409 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../032-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-06-29T15:56:21.418 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-jinja2. 2022-06-29T15:56:21.420 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph. 2022-06-29T15:56:21.434 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../051-python3-jinja2_2.10.1-2_all.deb ... 2022-06-29T15:56:21.435 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../047-ceph_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:21.447 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2) ... 2022-06-29T15:56:21.451 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-06-29T15:56:21.476 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2022-06-29T15:56:21.478 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:21.793 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-logutils. 2022-06-29T15:56:21.796 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-webob. 2022-06-29T15:56:21.806 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-fuse. 2022-06-29T15:56:21.809 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../028-python3-logutils_0.3.3-7_all.deb ... 2022-06-29T15:56:21.813 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../033-python3-webob_1%3a1.8.5-2_all.deb ... 2022-06-29T15:56:21.823 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../048-ceph-fuse_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:21.847 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-06-29T15:56:21.854 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2022-06-29T15:56:21.862 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-06-29T15:56:21.864 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-fuse (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:21.870 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../052-ceph-mgr-cephadm_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:21.912 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-cephadm (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:22.160 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-markupsafe. 2022-06-29T15:56:22.179 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../029-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-06-29T15:56:22.232 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-repoze.lru. 2022-06-29T15:56:22.243 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-06-29T15:56:22.248 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../053-python3-repoze.lru_0.7-2_all.deb ... 2022-06-29T15:56:22.257 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-soupsieve. 2022-06-29T15:56:22.259 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-mds. 2022-06-29T15:56:22.273 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../034-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-06-29T15:56:22.277 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../049-ceph-mds_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:22.290 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-06-29T15:56:22.315 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-06-29T15:56:22.317 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-mds (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:22.546 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-mako. 2022-06-29T15:56:22.564 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../030-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-06-29T15:56:22.568 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libjs-underscore. 2022-06-29T15:56:22.586 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../054-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-06-29T15:56:22.618 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-bs4. 2022-06-29T15:56:22.620 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-06-29T15:56:22.634 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../035-python3-bs4_4.8.2-1_all.deb ... 2022-06-29T15:56:22.635 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-06-29T15:56:22.693 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-06-29T15:56:22.811 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package cephadm. 2022-06-29T15:56:22.827 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../050-cephadm_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:22.870 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking cephadm (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:22.870 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph. 2022-06-29T15:56:22.885 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../047-ceph_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:22.928 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:22.929 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-06-29T15:56:22.948 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../055-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-06-29T15:56:22.987 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-06-29T15:56:22.999 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-simplegeneric. 2022-06-29T15:56:23.017 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../031-python3-simplegeneric_0.8.1-3_all.deb ... 2022-06-29T15:56:23.037 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-waitress. 2022-06-29T15:56:23.056 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../036-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2022-06-29T15:56:23.057 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-06-29T15:56:23.132 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2022-06-29T15:56:23.198 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-jinja2. 2022-06-29T15:56:23.215 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../051-python3-jinja2_2.10.1-2_all.deb ... 2022-06-29T15:56:23.256 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2022-06-29T15:56:23.348 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-routes. 2022-06-29T15:56:23.365 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../056-python3-routes_2.4.1-2_all.deb ... 2022-06-29T15:56:23.368 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-singledispatch. 2022-06-29T15:56:23.388 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../032-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-06-29T15:56:23.423 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-06-29T15:56:23.427 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-06-29T15:56:23.551 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2022-06-29T15:56:23.569 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../052-ceph-mgr-cephadm_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:23.589 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-fuse. 2022-06-29T15:56:23.606 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../048-ceph-fuse_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:23.608 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-mgr-cephadm (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:23.647 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-fuse (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:23.755 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-webob. 2022-06-29T15:56:23.760 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-06-29T15:56:23.772 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../033-python3-webob_1%3a1.8.5-2_all.deb ... 2022-06-29T15:56:23.777 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:23.784 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-tempita. 2022-06-29T15:56:23.801 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../037-python3-tempita_0.5.2-6_all.deb ... 2022-06-29T15:56:23.813 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-06-29T15:56:23.834 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-dashboard (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:23.842 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-06-29T15:56:23.961 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-repoze.lru. 2022-06-29T15:56:23.978 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../053-python3-repoze.lru_0.7-2_all.deb ... 2022-06-29T15:56:24.020 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-06-29T15:56:24.059 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mds. 2022-06-29T15:56:24.077 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../049-ceph-mds_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:24.117 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mds (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:24.170 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-paste. 2022-06-29T15:56:24.174 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-soupsieve. 2022-06-29T15:56:24.188 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../038-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-06-29T15:56:24.190 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../034-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-06-29T15:56:24.228 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-06-29T15:56:24.232 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-06-29T15:56:24.256 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libjs-underscore. 2022-06-29T15:56:24.272 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../054-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-06-29T15:56:24.314 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-06-29T15:56:24.515 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-06-29T15:56:24.532 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../039-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-06-29T15:56:24.573 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-06-29T15:56:24.594 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-bs4. 2022-06-29T15:56:24.611 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../035-python3-bs4_4.8.2-1_all.deb ... 2022-06-29T15:56:24.612 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package cephadm. 2022-06-29T15:56:24.625 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-06-29T15:56:24.631 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../050-cephadm_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:24.644 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../055-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-06-29T15:56:24.657 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-06-29T15:56:24.670 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking cephadm (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:24.683 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-06-29T15:56:24.859 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2022-06-29T15:56:24.877 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../058-ceph-mgr-diskprediction-cloud_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:24.892 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-pastedeploy. 2022-06-29T15:56:24.909 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../040-python3-pastedeploy_2.1.0-1_all.deb ... 2022-06-29T15:56:24.925 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:24.950 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-06-29T15:56:24.986 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-routes. 2022-06-29T15:56:25.005 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../056-python3-routes_2.4.1-2_all.deb ... 2022-06-29T15:56:25.023 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-jinja2. 2022-06-29T15:56:25.039 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../051-python3-jinja2_2.10.1-2_all.deb ... 2022-06-29T15:56:25.044 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-06-29T15:56:25.045 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-waitress. 2022-06-29T15:56:25.063 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../036-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2022-06-29T15:56:25.081 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2022-06-29T15:56:25.113 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2022-06-29T15:56:25.262 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-decorator. 2022-06-29T15:56:25.279 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-webtest. 2022-06-29T15:56:25.280 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../059-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-06-29T15:56:25.294 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../041-python3-webtest_2.0.32-2_all.deb ... 2022-06-29T15:56:25.319 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-06-29T15:56:25.337 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-06-29T15:56:25.393 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2022-06-29T15:56:25.409 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../052-ceph-mgr-cephadm_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:25.422 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-tempita. 2022-06-29T15:56:25.423 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-06-29T15:56:25.438 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../037-python3-tempita_0.5.2-6_all.deb ... 2022-06-29T15:56:25.441 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:25.451 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mgr-cephadm (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:25.472 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-06-29T15:56:25.483 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-mgr-dashboard (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:25.656 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-06-29T15:56:25.674 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../060-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-06-29T15:56:25.682 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-pecan. 2022-06-29T15:56:25.698 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../042-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-06-29T15:56:25.722 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-06-29T15:56:25.740 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-06-29T15:56:25.792 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-paste. 2022-06-29T15:56:25.796 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-repoze.lru. 2022-06-29T15:56:25.808 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../038-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-06-29T15:56:25.812 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../053-python3-repoze.lru_0.7-2_all.deb ... 2022-06-29T15:56:25.841 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-06-29T15:56:25.854 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-06-29T15:56:26.050 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-scipy. 2022-06-29T15:56:26.069 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../061-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-06-29T15:56:26.117 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-06-29T15:56:26.120 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-werkzeug. 2022-06-29T15:56:26.124 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libjs-underscore. 2022-06-29T15:56:26.137 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../043-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-06-29T15:56:26.141 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../054-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-06-29T15:56:26.178 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-06-29T15:56:26.182 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-06-29T15:56:26.186 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-06-29T15:56:26.207 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../039-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-06-29T15:56:26.245 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-06-29T15:56:26.390 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2022-06-29T15:56:26.409 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../058-ceph-mgr-diskprediction-cloud_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:26.456 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:26.485 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-06-29T15:56:26.504 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../055-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-06-29T15:56:26.531 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-pastedeploy. 2022-06-29T15:56:26.543 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-06-29T15:56:26.550 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../040-python3-pastedeploy_2.1.0-1_all.deb ... 2022-06-29T15:56:26.581 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-06-29T15:56:26.590 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package ceph-mgr. 2022-06-29T15:56:26.605 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../044-ceph-mgr_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:26.648 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking ceph-mgr (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:26.776 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-decorator. 2022-06-29T15:56:26.792 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../059-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-06-29T15:56:26.830 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-routes. 2022-06-29T15:56:26.834 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-06-29T15:56:26.848 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../056-python3-routes_2.4.1-2_all.deb ... 2022-06-29T15:56:26.879 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-06-29T15:56:26.909 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-webtest. 2022-06-29T15:56:26.925 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../041-python3-webtest_2.0.32-2_all.deb ... 2022-06-29T15:56:26.967 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-06-29T15:56:27.135 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package ceph-mon. 2022-06-29T15:56:27.152 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../045-ceph-mon_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:27.170 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-06-29T15:56:27.187 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../060-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-06-29T15:56:27.193 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking ceph-mon (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:27.228 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-06-29T15:56:27.268 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-06-29T15:56:27.284 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:27.312 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-pecan. 2022-06-29T15:56:27.326 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mgr-dashboard (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:27.328 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../042-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-06-29T15:56:27.370 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-06-29T15:56:27.531 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-scipy. 2022-06-29T15:56:27.550 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../061-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-06-29T15:56:27.589 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-06-29T15:56:27.677 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-06-29T15:56:27.694 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../062-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-06-29T15:56:27.735 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-06-29T15:56:27.750 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-werkzeug. 2022-06-29T15:56:27.768 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../043-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-06-29T15:56:27.809 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-06-29T15:56:28.222 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package ceph-osd. 2022-06-29T15:56:28.229 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package ceph-mgr. 2022-06-29T15:56:28.238 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../046-ceph-osd_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:28.247 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../044-ceph-mgr_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:28.279 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking ceph-osd (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:28.287 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking ceph-mgr (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:28.347 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-joblib. 2022-06-29T15:56:28.359 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2022-06-29T15:56:28.367 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../063-python3-joblib_0.14.0-3_all.deb ... 2022-06-29T15:56:28.375 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../058-ceph-mgr-diskprediction-cloud_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:28.413 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-06-29T15:56:28.443 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:28.779 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-decorator. 2022-06-29T15:56:28.795 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../059-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-06-29T15:56:28.824 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package ceph-mon. 2022-06-29T15:56:28.834 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-sklearn. 2022-06-29T15:56:28.840 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../045-ceph-mon_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:28.845 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-06-29T15:56:28.850 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../064-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-06-29T15:56:28.882 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking ceph-mon (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:28.893 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-06-29T15:56:29.140 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-06-29T15:56:29.157 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-06-29T15:56:29.159 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../060-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-06-29T15:56:29.175 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../062-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-06-29T15:56:29.215 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-06-29T15:56:29.232 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-06-29T15:56:29.560 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-scipy. 2022-06-29T15:56:29.577 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../061-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-06-29T15:56:29.618 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-06-29T15:56:29.634 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-06-29T15:56:29.650 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../065-ceph-mgr-diskprediction-local_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:29.692 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:29.802 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-joblib. 2022-06-29T15:56:29.819 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../063-python3-joblib_0.14.0-3_all.deb ... 2022-06-29T15:56:29.868 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-06-29T15:56:29.902 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package ceph-osd. 2022-06-29T15:56:29.920 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../046-ceph-osd_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:29.986 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking ceph-osd (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:30.231 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-sklearn. 2022-06-29T15:56:30.247 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../064-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-06-29T15:56:30.288 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-06-29T15:56:30.610 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package ceph. 2022-06-29T15:56:30.626 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../047-ceph_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:30.677 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking ceph (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:30.812 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-cachetools. 2022-06-29T15:56:30.829 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../066-python3-cachetools_4.0.0-1_all.deb ... 2022-06-29T15:56:30.870 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-06-29T15:56:30.989 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-06-29T15:56:31.009 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../065-ceph-mgr-diskprediction-local_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:31.030 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package ceph-fuse. 2022-06-29T15:56:31.046 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../048-ceph-fuse_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:31.047 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:31.061 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-06-29T15:56:31.080 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../062-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-06-29T15:56:31.087 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking ceph-fuse (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:31.127 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-06-29T15:56:31.198 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pyasn1. 2022-06-29T15:56:31.214 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-06-29T15:56:31.256 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-06-29T15:56:31.517 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package ceph-mds. 2022-06-29T15:56:31.533 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../049-ceph-mds_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:31.574 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking ceph-mds (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:31.634 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-06-29T15:56:31.651 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-06-29T15:56:31.693 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-06-29T15:56:31.756 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-joblib. 2022-06-29T15:56:31.773 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../063-python3-joblib_0.14.0-3_all.deb ... 2022-06-29T15:56:31.814 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-06-29T15:56:32.020 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-rsa. 2022-06-29T15:56:32.037 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../069-python3-rsa_4.0-3_all.deb ... 2022-06-29T15:56:32.079 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-rsa (4.0-3) ... 2022-06-29T15:56:32.086 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package cephadm. 2022-06-29T15:56:32.104 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../050-cephadm_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:32.158 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-cachetools. 2022-06-29T15:56:32.160 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking cephadm (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:32.178 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../066-python3-cachetools_4.0.0-1_all.deb ... 2022-06-29T15:56:32.202 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-sklearn. 2022-06-29T15:56:32.216 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-06-29T15:56:32.218 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../064-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-06-29T15:56:32.259 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-06-29T15:56:32.275 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package ceph. 2022-06-29T15:56:32.291 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../047-ceph_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:32.333 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking ceph (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:32.432 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-google-auth. 2022-06-29T15:56:32.449 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../070-python3-google-auth_1.5.1-2_all.deb ... 2022-06-29T15:56:32.477 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-pyasn1. 2022-06-29T15:56:32.480 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-jinja2. 2022-06-29T15:56:32.490 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-06-29T15:56:32.494 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-06-29T15:56:32.497 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../051-python3-jinja2_2.10.1-2_all.deb ... 2022-06-29T15:56:32.535 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-06-29T15:56:32.538 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2022-06-29T15:56:32.678 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package ceph-fuse. 2022-06-29T15:56:32.694 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../048-ceph-fuse_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:32.736 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking ceph-fuse (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:32.827 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-blinker. 2022-06-29T15:56:32.847 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../071-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-06-29T15:56:32.855 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-06-29T15:56:32.871 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-06-29T15:56:32.885 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-06-29T15:56:32.909 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2022-06-29T15:56:32.913 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-06-29T15:56:32.925 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../052-ceph-mgr-cephadm_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:32.966 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking ceph-mgr-cephadm (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:32.993 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-06-29T15:56:33.012 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../065-ceph-mgr-diskprediction-local_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:33.051 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:33.139 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package ceph-mds. 2022-06-29T15:56:33.156 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../049-ceph-mds_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:33.197 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking ceph-mds (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:33.221 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-oauthlib. 2022-06-29T15:56:33.240 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../072-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-06-29T15:56:33.258 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-rsa. 2022-06-29T15:56:33.275 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../069-python3-rsa_4.0-3_all.deb ... 2022-06-29T15:56:33.279 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-06-29T15:56:33.316 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-rsa (4.0-3) ... 2022-06-29T15:56:33.319 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-repoze.lru. 2022-06-29T15:56:33.337 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../053-python3-repoze.lru_0.7-2_all.deb ... 2022-06-29T15:56:33.378 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-06-29T15:56:33.586 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-google-auth. 2022-06-29T15:56:33.603 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../070-python3-google-auth_1.5.1-2_all.deb ... 2022-06-29T15:56:33.608 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-06-29T15:56:33.626 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../073-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-06-29T15:56:33.636 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-06-29T15:56:33.656 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package libjs-underscore. 2022-06-29T15:56:33.673 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../054-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-06-29T15:56:33.674 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-06-29T15:56:33.709 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package cephadm. 2022-06-29T15:56:33.713 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-06-29T15:56:33.726 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../050-cephadm_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:33.768 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking cephadm (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:33.981 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-blinker. 2022-06-29T15:56:33.998 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../071-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-06-29T15:56:34.017 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-06-29T15:56:34.033 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../055-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-06-29T15:56:34.039 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-06-29T15:56:34.044 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-websocket. 2022-06-29T15:56:34.064 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../074-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-06-29T15:56:34.076 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-06-29T15:56:34.095 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-jinja2. 2022-06-29T15:56:34.102 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-06-29T15:56:34.112 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../051-python3-jinja2_2.10.1-2_all.deb ... 2022-06-29T15:56:34.153 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2022-06-29T15:56:34.205 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-cachetools. 2022-06-29T15:56:34.221 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../066-python3-cachetools_4.0.0-1_all.deb ... 2022-06-29T15:56:34.263 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-06-29T15:56:34.343 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-oauthlib. 2022-06-29T15:56:34.359 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../072-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-06-29T15:56:34.401 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-06-29T15:56:34.403 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-routes. 2022-06-29T15:56:34.414 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-kubernetes. 2022-06-29T15:56:34.420 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../056-python3-routes_2.4.1-2_all.deb ... 2022-06-29T15:56:34.430 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../075-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-06-29T15:56:34.469 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-06-29T15:56:34.497 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-06-29T15:56:34.507 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2022-06-29T15:56:34.526 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../052-ceph-mgr-cephadm_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:34.566 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pyasn1. 2022-06-29T15:56:34.573 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking ceph-mgr-cephadm (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:34.582 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-06-29T15:56:34.649 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-06-29T15:56:34.755 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-06-29T15:56:34.772 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../073-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-06-29T15:56:34.813 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-06-29T15:56:34.814 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-06-29T15:56:34.832 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:34.872 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking ceph-mgr-dashboard (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:34.935 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-repoze.lru. 2022-06-29T15:56:34.951 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../053-python3-repoze.lru_0.7-2_all.deb ... 2022-06-29T15:56:34.993 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-06-29T15:56:34.994 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-06-29T15:56:35.010 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-06-29T15:56:35.035 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-06-29T15:56:35.053 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:35.078 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-06-29T15:56:35.093 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-k8sevents (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:35.099 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-websocket. 2022-06-29T15:56:35.118 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../074-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-06-29T15:56:35.157 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-06-29T15:56:35.280 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package libjs-underscore. 2022-06-29T15:56:35.296 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../054-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-06-29T15:56:35.338 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-06-29T15:56:35.410 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-kubernetes. 2022-06-29T15:56:35.413 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-rsa. 2022-06-29T15:56:35.427 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../075-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-06-29T15:56:35.432 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../069-python3-rsa_4.0-3_all.deb ... 2022-06-29T15:56:35.455 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libonig5:amd64. 2022-06-29T15:56:35.471 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../077-libonig5_6.9.4-1_amd64.deb ... 2022-06-29T15:56:35.477 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-06-29T15:56:35.480 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-rsa (4.0-3) ... 2022-06-29T15:56:35.512 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-06-29T15:56:35.641 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-06-29T15:56:35.657 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../055-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-06-29T15:56:35.699 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-06-29T15:56:35.826 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-google-auth. 2022-06-29T15:56:35.842 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../070-python3-google-auth_1.5.1-2_all.deb ... 2022-06-29T15:56:35.847 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2022-06-29T15:56:35.866 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../058-ceph-mgr-diskprediction-cloud_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:35.900 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-06-29T15:56:35.905 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:35.924 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libjq1:amd64. 2022-06-29T15:56:35.941 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../078-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-06-29T15:56:35.992 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-06-29T15:56:36.027 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-routes. 2022-06-29T15:56:36.044 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../056-python3-routes_2.4.1-2_all.deb ... 2022-06-29T15:56:36.065 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-06-29T15:56:36.081 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:36.110 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-06-29T15:56:36.123 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-mgr-k8sevents (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:36.200 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-decorator. 2022-06-29T15:56:36.219 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../059-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-06-29T15:56:36.258 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-06-29T15:56:36.263 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-blinker. 2022-06-29T15:56:36.277 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package jq. 2022-06-29T15:56:36.281 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../071-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-06-29T15:56:36.294 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../079-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-06-29T15:56:36.321 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-06-29T15:56:36.335 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-06-29T15:56:36.451 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libonig5:amd64. 2022-06-29T15:56:36.463 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-06-29T15:56:36.470 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../077-libonig5_6.9.4-1_amd64.deb ... 2022-06-29T15:56:36.480 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:36.509 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-06-29T15:56:36.522 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking ceph-mgr-dashboard (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:36.596 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package socat. 2022-06-29T15:56:36.602 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-06-29T15:56:36.613 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../080-socat_1.7.3.3-2_amd64.deb ... 2022-06-29T15:56:36.618 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../060-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-06-29T15:56:36.657 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-oauthlib. 2022-06-29T15:56:36.661 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-06-29T15:56:36.671 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking socat (1.7.3.3-2) ... 2022-06-29T15:56:36.674 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../072-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-06-29T15:56:36.715 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-06-29T15:56:36.887 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libjq1:amd64. 2022-06-29T15:56:36.906 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../078-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-06-29T15:56:36.945 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-06-29T15:56:37.005 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-scipy. 2022-06-29T15:56:37.021 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../061-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-06-29T15:56:37.063 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-06-29T15:56:37.070 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-06-29T15:56:37.088 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../073-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-06-29T15:56:37.128 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-06-29T15:56:37.211 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-06-29T15:56:37.227 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../081-libxslt1.1_1.1.34-4_amd64.deb ... 2022-06-29T15:56:37.239 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package jq. 2022-06-29T15:56:37.255 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../079-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-06-29T15:56:37.276 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-06-29T15:56:37.314 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-06-29T15:56:37.439 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-websocket. 2022-06-29T15:56:37.458 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../074-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-06-29T15:56:37.519 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-06-29T15:56:37.564 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2022-06-29T15:56:37.575 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package socat. 2022-06-29T15:56:37.581 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../058-ceph-mgr-diskprediction-cloud_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:37.588 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package xmlstarlet. 2022-06-29T15:56:37.593 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../080-socat_1.7.3.3-2_amd64.deb ... 2022-06-29T15:56:37.604 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../082-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-06-29T15:56:37.621 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:37.633 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking socat (1.7.3.3-2) ... 2022-06-29T15:56:37.654 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-06-29T15:56:37.843 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-kubernetes. 2022-06-29T15:56:37.859 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../075-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-06-29T15:56:37.903 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-06-29T15:56:37.992 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-decorator. 2022-06-29T15:56:38.009 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../059-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-06-29T15:56:38.050 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-06-29T15:56:38.131 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-06-29T15:56:38.147 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../081-libxslt1.1_1.1.34-4_amd64.deb ... 2022-06-29T15:56:38.189 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-06-29T15:56:38.358 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-test. 2022-06-29T15:56:38.377 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../083-ceph-test_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:38.395 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-06-29T15:56:38.412 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../060-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-06-29T15:56:38.453 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-06-29T15:56:38.475 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package xmlstarlet. 2022-06-29T15:56:38.492 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../082-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-06-29T15:56:38.509 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-06-29T15:56:38.525 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-test (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:38.528 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:38.533 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-06-29T15:56:38.540 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-06-29T15:56:38.557 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../062-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-06-29T15:56:38.572 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mgr-k8sevents (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:38.598 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-06-29T15:56:38.848 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-scipy. 2022-06-29T15:56:38.865 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../061-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-06-29T15:56:38.887 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-test. 2022-06-29T15:56:38.905 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../083-ceph-test_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:38.906 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-06-29T15:56:38.945 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-test (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:39.017 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libonig5:amd64. 2022-06-29T15:56:39.033 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../077-libonig5_6.9.4-1_amd64.deb ... 2022-06-29T15:56:39.075 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-06-29T15:56:39.244 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-joblib. 2022-06-29T15:56:39.261 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../063-python3-joblib_0.14.0-3_all.deb ... 2022-06-29T15:56:39.302 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-06-29T15:56:39.428 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libjq1:amd64. 2022-06-29T15:56:39.445 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../078-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-06-29T15:56:39.503 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-06-29T15:56:39.722 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-sklearn. 2022-06-29T15:56:39.738 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../064-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-06-29T15:56:39.780 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-06-29T15:56:39.789 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package jq. 2022-06-29T15:56:39.806 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../079-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-06-29T15:56:39.847 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-06-29T15:56:40.150 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package socat. 2022-06-29T15:56:40.166 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../080-socat_1.7.3.3-2_amd64.deb ... 2022-06-29T15:56:40.216 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking socat (1.7.3.3-2) ... 2022-06-29T15:56:40.438 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-06-29T15:56:40.457 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../065-ceph-mgr-diskprediction-local_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:40.496 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:40.508 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-06-29T15:56:40.525 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../062-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-06-29T15:56:40.566 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-06-29T15:56:40.689 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-06-29T15:56:40.708 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../081-libxslt1.1_1.1.34-4_amd64.deb ... 2022-06-29T15:56:40.747 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-06-29T15:56:41.042 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package xmlstarlet. 2022-06-29T15:56:41.061 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../082-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-06-29T15:56:41.100 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-06-29T15:56:41.145 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-joblib. 2022-06-29T15:56:41.161 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../063-python3-joblib_0.14.0-3_all.deb ... 2022-06-29T15:56:41.228 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-06-29T15:56:41.463 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-test. 2022-06-29T15:56:41.483 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../083-ceph-test_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:41.521 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-test (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:41.599 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-sklearn. 2022-06-29T15:56:41.619 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../064-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-06-29T15:56:41.650 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-cachetools. 2022-06-29T15:56:41.657 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-06-29T15:56:41.668 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../066-python3-cachetools_4.0.0-1_all.deb ... 2022-06-29T15:56:41.717 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-06-29T15:56:41.979 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-colorama. 2022-06-29T15:56:41.995 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../084-python3-colorama_0.4.3-1build1_all.deb ... 2022-06-29T15:56:42.036 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-pyasn1. 2022-06-29T15:56:42.037 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-colorama (0.4.3-1build1) ... 2022-06-29T15:56:42.053 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-06-29T15:56:42.094 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-06-29T15:56:42.340 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package cephfs-shell. 2022-06-29T15:56:42.357 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../085-cephfs-shell_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:42.383 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-06-29T15:56:42.398 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking cephfs-shell (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:42.403 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../065-ceph-mgr-diskprediction-local_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:42.408 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-colorama. 2022-06-29T15:56:42.426 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../084-python3-colorama_0.4.3-1build1_all.deb ... 2022-06-29T15:56:42.449 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:42.457 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-06-29T15:56:42.457 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-colorama (0.4.3-1build1) ... 2022-06-29T15:56:42.473 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-06-29T15:56:42.515 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-06-29T15:56:42.669 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package cephfs-shell. 2022-06-29T15:56:42.689 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../085-cephfs-shell_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:42.718 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking cephfs-shell (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:42.810 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libcephfs-dev. 2022-06-29T15:56:42.826 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../086-libcephfs-dev_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:42.835 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-rsa. 2022-06-29T15:56:42.851 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../069-python3-rsa_4.0-3_all.deb ... 2022-06-29T15:56:42.868 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libcephfs-dev (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:42.910 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-rsa (4.0-3) ... 2022-06-29T15:56:43.007 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libcephfs-dev. 2022-06-29T15:56:43.026 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../086-libcephfs-dev_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:43.096 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libcephfs-dev (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:43.222 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-google-auth. 2022-06-29T15:56:43.237 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../070-python3-google-auth_1.5.1-2_all.deb ... 2022-06-29T15:56:43.280 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-06-29T15:56:43.296 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package nvme-cli. 2022-06-29T15:56:43.313 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../087-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-06-29T15:56:43.380 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-06-29T15:56:43.553 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-cachetools. 2022-06-29T15:56:43.571 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../066-python3-cachetools_4.0.0-1_all.deb ... 2022-06-29T15:56:43.611 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-06-29T15:56:43.650 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-blinker. 2022-06-29T15:56:43.666 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../071-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-06-29T15:56:43.708 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-06-29T15:56:43.844 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-atomicwrites. 2022-06-29T15:56:43.864 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../088-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-06-29T15:56:43.902 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-06-29T15:56:43.989 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-pyasn1. 2022-06-29T15:56:44.006 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-06-29T15:56:44.036 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-oauthlib. 2022-06-29T15:56:44.053 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../072-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-06-29T15:56:44.058 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package nvme-cli. 2022-06-29T15:56:44.063 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-06-29T15:56:44.074 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../087-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-06-29T15:56:44.094 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-06-29T15:56:44.134 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-06-29T15:56:44.188 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-attr. 2022-06-29T15:56:44.208 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../089-python3-attr_19.3.0-2_all.deb ... 2022-06-29T15:56:44.246 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-06-29T15:56:44.431 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-06-29T15:56:44.434 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-06-29T15:56:44.449 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../073-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-06-29T15:56:44.454 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-06-29T15:56:44.489 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-06-29T15:56:44.492 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-06-29T15:56:44.521 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-atomicwrites. 2022-06-29T15:56:44.537 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../088-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-06-29T15:56:44.579 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-06-29T15:56:44.616 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-webencodings. 2022-06-29T15:56:44.634 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../090-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-06-29T15:56:44.674 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-06-29T15:56:44.834 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-websocket. 2022-06-29T15:56:44.837 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-rsa. 2022-06-29T15:56:44.850 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../074-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-06-29T15:56:44.853 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../069-python3-rsa_4.0-3_all.deb ... 2022-06-29T15:56:44.891 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-attr. 2022-06-29T15:56:44.892 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-06-29T15:56:44.895 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-rsa (4.0-3) ... 2022-06-29T15:56:44.910 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../089-python3-attr_19.3.0-2_all.deb ... 2022-06-29T15:56:44.949 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-06-29T15:56:44.950 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-colorama. 2022-06-29T15:56:44.969 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-html5lib. 2022-06-29T15:56:44.969 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../084-python3-colorama_0.4.3-1build1_all.deb ... 2022-06-29T15:56:44.988 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../091-python3-html5lib_1.0.1-2_all.deb ... 2022-06-29T15:56:45.000 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-colorama (0.4.3-1build1) ... 2022-06-29T15:56:45.027 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-06-29T15:56:45.215 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-google-auth. 2022-06-29T15:56:45.229 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-kubernetes. 2022-06-29T15:56:45.231 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../070-python3-google-auth_1.5.1-2_all.deb ... 2022-06-29T15:56:45.248 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../075-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-06-29T15:56:45.255 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package cephfs-shell. 2022-06-29T15:56:45.273 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-06-29T15:56:45.274 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../085-cephfs-shell_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:45.287 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-06-29T15:56:45.293 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-webencodings. 2022-06-29T15:56:45.313 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../090-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-06-29T15:56:45.319 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking cephfs-shell (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:45.368 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-06-29T15:56:45.397 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-06-29T15:56:45.415 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../092-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-06-29T15:56:45.455 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-06-29T15:56:45.572 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libcephfs-dev. 2022-06-29T15:56:45.592 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../086-libcephfs-dev_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:45.619 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-blinker. 2022-06-29T15:56:45.630 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libcephfs-dev (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:45.639 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../071-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-06-29T15:56:45.677 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-06-29T15:56:45.704 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-html5lib. 2022-06-29T15:56:45.720 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../091-python3-html5lib_1.0.1-2_all.deb ... 2022-06-29T15:56:45.762 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-06-29T15:56:45.869 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-06-29T15:56:45.885 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:45.927 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking ceph-mgr-k8sevents (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:45.975 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-packaging. 2022-06-29T15:56:45.993 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../093-python3-packaging_20.3-1_all.deb ... 2022-06-29T15:56:46.009 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package nvme-cli. 2022-06-29T15:56:46.013 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-oauthlib. 2022-06-29T15:56:46.025 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../087-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-06-29T15:56:46.029 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../072-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-06-29T15:56:46.049 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-packaging (20.3-1) ... 2022-06-29T15:56:46.071 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-06-29T15:56:46.084 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-06-29T15:56:46.107 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-06-29T15:56:46.123 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../092-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-06-29T15:56:46.165 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-06-29T15:56:46.280 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package libonig5:amd64. 2022-06-29T15:56:46.297 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../077-libonig5_6.9.4-1_amd64.deb ... 2022-06-29T15:56:46.338 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-06-29T15:56:46.371 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pastescript. 2022-06-29T15:56:46.388 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../094-python3-pastescript_2.0.2-4_all.deb ... 2022-06-29T15:56:46.429 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-06-29T15:56:46.459 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-06-29T15:56:46.464 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-atomicwrites. 2022-06-29T15:56:46.475 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../073-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-06-29T15:56:46.483 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../088-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-06-29T15:56:46.522 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-06-29T15:56:46.525 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-06-29T15:56:46.652 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-packaging. 2022-06-29T15:56:46.672 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../093-python3-packaging_20.3-1_all.deb ... 2022-06-29T15:56:46.710 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-packaging (20.3-1) ... 2022-06-29T15:56:46.724 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package libjq1:amd64. 2022-06-29T15:56:46.741 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../078-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-06-29T15:56:46.783 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-06-29T15:56:46.783 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pluggy. 2022-06-29T15:56:46.804 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../095-python3-pluggy_0.13.0-2_all.deb ... 2022-06-29T15:56:46.841 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-06-29T15:56:46.862 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-websocket. 2022-06-29T15:56:46.867 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-attr. 2022-06-29T15:56:46.878 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../074-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-06-29T15:56:46.883 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../089-python3-attr_19.3.0-2_all.deb ... 2022-06-29T15:56:46.920 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-06-29T15:56:46.925 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-06-29T15:56:46.980 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-pastescript. 2022-06-29T15:56:46.998 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../094-python3-pastescript_2.0.2-4_all.deb ... 2022-06-29T15:56:47.029 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-06-29T15:56:47.103 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package jq. 2022-06-29T15:56:47.123 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../079-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-06-29T15:56:47.136 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-psutil. 2022-06-29T15:56:47.156 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../096-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-06-29T15:56:47.160 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-06-29T15:56:47.211 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-06-29T15:56:47.257 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-kubernetes. 2022-06-29T15:56:47.270 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-webencodings. 2022-06-29T15:56:47.273 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../075-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-06-29T15:56:47.288 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../090-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-06-29T15:56:47.315 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-06-29T15:56:47.328 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-06-29T15:56:47.391 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-pluggy. 2022-06-29T15:56:47.410 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../095-python3-pluggy_0.13.0-2_all.deb ... 2022-06-29T15:56:47.413 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package socat. 2022-06-29T15:56:47.432 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../080-socat_1.7.3.3-2_amd64.deb ... 2022-06-29T15:56:47.449 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-06-29T15:56:47.471 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking socat (1.7.3.3-2) ... 2022-06-29T15:56:47.548 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-py. 2022-06-29T15:56:47.568 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../097-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-06-29T15:56:47.589 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-html5lib. 2022-06-29T15:56:47.605 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../091-python3-html5lib_1.0.1-2_all.deb ... 2022-06-29T15:56:47.606 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-06-29T15:56:47.639 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-06-29T15:56:47.761 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-psutil. 2022-06-29T15:56:47.778 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../096-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-06-29T15:56:47.819 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-06-29T15:56:47.895 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-06-29T15:56:47.913 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:47.926 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pyinotify. 2022-06-29T15:56:47.945 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../098-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-06-29T15:56:47.967 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-06-29T15:56:47.970 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking ceph-mgr-k8sevents (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:47.983 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../092-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-06-29T15:56:47.984 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-06-29T15:56:48.011 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-06-29T15:56:48.025 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-06-29T15:56:48.031 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../081-libxslt1.1_1.1.34-4_amd64.deb ... 2022-06-29T15:56:48.069 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-06-29T15:56:48.172 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-py. 2022-06-29T15:56:48.190 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../097-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-06-29T15:56:48.230 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-06-29T15:56:48.298 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package libonig5:amd64. 2022-06-29T15:56:48.316 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../077-libonig5_6.9.4-1_amd64.deb ... 2022-06-29T15:56:48.329 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pytest. 2022-06-29T15:56:48.346 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../099-python3-pytest_4.6.9-1_all.deb ... 2022-06-29T15:56:48.356 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-06-29T15:56:48.364 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package xmlstarlet. 2022-06-29T15:56:48.380 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../082-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-06-29T15:56:48.387 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-06-29T15:56:48.422 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-06-29T15:56:48.533 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-pyinotify. 2022-06-29T15:56:48.537 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-packaging. 2022-06-29T15:56:48.553 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../098-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-06-29T15:56:48.553 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../093-python3-packaging_20.3-1_all.deb ... 2022-06-29T15:56:48.591 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-06-29T15:56:48.595 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-packaging (20.3-1) ... 2022-06-29T15:56:48.741 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-simplejson. 2022-06-29T15:56:48.758 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../100-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-06-29T15:56:48.759 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package libjq1:amd64. 2022-06-29T15:56:48.776 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../078-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-06-29T15:56:48.776 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package ceph-test. 2022-06-29T15:56:48.795 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../083-ceph-test_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:48.824 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-06-29T15:56:48.843 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking ceph-test (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:48.903 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-pytest. 2022-06-29T15:56:48.919 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../099-python3-pytest_4.6.9-1_all.deb ... 2022-06-29T15:56:48.923 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pastescript. 2022-06-29T15:56:48.944 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../094-python3-pastescript_2.0.2-4_all.deb ... 2022-06-29T15:56:48.980 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-06-29T15:56:48.981 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-06-29T15:56:49.216 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-06-29T15:56:49.295 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package radosgw. 2022-06-29T15:56:49.311 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../101-radosgw_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:49.322 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python3-simplejson. 2022-06-29T15:56:49.335 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pluggy. 2022-06-29T15:56:49.339 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../100-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-06-29T15:56:49.351 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../095-python3-pluggy_0.13.0-2_all.deb ... 2022-06-29T15:56:49.361 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking radosgw (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:49.380 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-06-29T15:56:49.393 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-06-29T15:56:49.512 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package jq. 2022-06-29T15:56:49.529 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../079-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-06-29T15:56:49.570 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-06-29T15:56:49.721 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-psutil. 2022-06-29T15:56:49.738 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../096-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-06-29T15:56:49.780 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-06-29T15:56:49.834 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package radosgw. 2022-06-29T15:56:49.848 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package socat. 2022-06-29T15:56:49.850 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../101-radosgw_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:49.865 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../080-socat_1.7.3.3-2_amd64.deb ... 2022-06-29T15:56:49.892 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking radosgw (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:49.906 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking socat (1.7.3.3-2) ... 2022-06-29T15:56:50.125 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-py. 2022-06-29T15:56:50.142 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../097-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-06-29T15:56:50.208 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-06-29T15:56:50.389 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-06-29T15:56:50.406 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../081-libxslt1.1_1.1.34-4_amd64.deb ... 2022-06-29T15:56:50.440 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package rbd-fuse. 2022-06-29T15:56:50.458 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../102-rbd-fuse_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:50.481 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-06-29T15:56:50.498 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking rbd-fuse (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:50.527 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pyinotify. 2022-06-29T15:56:50.548 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../098-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-06-29T15:56:50.586 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-06-29T15:56:50.759 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package xmlstarlet. 2022-06-29T15:56:50.776 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../082-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-06-29T15:56:50.806 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-attr (19.3.0-2) ... 2022-06-29T15:56:50.842 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-06-29T15:56:50.930 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pytest. 2022-06-29T15:56:50.947 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../099-python3-pytest_4.6.9-1_all.deb ... 2022-06-29T15:56:50.962 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package rbd-fuse. 2022-06-29T15:56:50.981 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../102-rbd-fuse_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:50.988 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-06-29T15:56:51.020 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking rbd-fuse (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:51.144 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-06-29T15:56:51.205 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package ceph-test. 2022-06-29T15:56:51.221 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../083-ceph-test_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:51.270 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-06-29T15:56:51.271 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking ceph-test (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:51.311 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-attr (19.3.0-2) ... 2022-06-29T15:56:51.392 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-simplejson. 2022-06-29T15:56:51.396 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-06-29T15:56:51.409 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../100-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-06-29T15:56:51.459 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-06-29T15:56:51.565 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up cephadm (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:51.666 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-06-29T15:56:51.792 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-06-29T15:56:51.913 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package radosgw. 2022-06-29T15:56:51.918 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-06-29T15:56:51.932 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../101-radosgw_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:51.970 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking radosgw (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:52.251 INFO:teuthology.orchestra.run.smithi060.stdout:Adding system user cephadm....done 2022-06-29T15:56:52.328 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up cephadm (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:52.440 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-colorama. 2022-06-29T15:56:52.456 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../084-python3-colorama_0.4.3-1build1_all.deb ... 2022-06-29T15:56:52.463 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2022-06-29T15:56:52.498 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-colorama (0.4.3-1build1) ... 2022-06-29T15:56:52.784 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package cephfs-shell. 2022-06-29T15:56:52.800 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../085-cephfs-shell_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:52.822 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-06-29T15:56:52.842 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking cephfs-shell (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:52.998 INFO:teuthology.orchestra.run.smithi092.stdout:Adding system user cephadm....done 2022-06-29T15:56:53.015 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package rbd-fuse. 2022-06-29T15:56:53.031 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../102-rbd-fuse_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:53.074 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking rbd-fuse (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:53.132 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-06-29T15:56:53.136 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package libcephfs-dev. 2022-06-29T15:56:53.153 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../086-libcephfs-dev_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:53.176 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2022-06-29T15:56:53.195 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking libcephfs-dev (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:53.266 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-06-29T15:56:53.527 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-06-29T15:56:53.533 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-attr (19.3.0-2) ... 2022-06-29T15:56:53.599 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package nvme-cli. 2022-06-29T15:56:53.601 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-06-29T15:56:53.619 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../087-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-06-29T15:56:53.682 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-06-29T15:56:53.811 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-06-29T15:56:53.878 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-06-29T15:56:53.937 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-06-29T15:56:54.028 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-jwt (1.7.1-2ubuntu2) ... 2022-06-29T15:56:54.085 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-atomicwrites. 2022-06-29T15:56:54.104 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../088-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-06-29T15:56:54.145 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-06-29T15:56:54.232 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-06-29T15:56:54.289 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-06-29T15:56:54.321 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-06-29T15:56:54.605 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-06-29T15:56:54.614 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-colorama (0.4.3-1build1) ... 2022-06-29T15:56:54.701 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-colorama. 2022-06-29T15:56:54.708 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-jwt (1.7.1-2ubuntu2) ... 2022-06-29T15:56:54.720 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../084-python3-colorama_0.4.3-1build1_all.deb ... 2022-06-29T15:56:54.759 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-colorama (0.4.3-1build1) ... 2022-06-29T15:56:54.857 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up cephadm (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:54.882 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-attr. 2022-06-29T15:56:54.898 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../089-python3-attr_19.3.0-2_all.deb ... 2022-06-29T15:56:54.924 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-06-29T15:56:54.940 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-06-29T15:56:55.037 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package cephfs-shell. 2022-06-29T15:56:55.042 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-06-29T15:56:55.055 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../085-cephfs-shell_15.2.16-304-gf25ffac7-1focal_all.deb ... 2022-06-29T15:56:55.095 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking cephfs-shell (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:55.233 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-06-29T15:56:55.251 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-webencodings. 2022-06-29T15:56:55.271 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../090-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-06-29T15:56:55.310 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-06-29T15:56:55.335 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-colorama (0.4.3-1build1) ... 2022-06-29T15:56:55.373 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package libcephfs-dev. 2022-06-29T15:56:55.390 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../086-libcephfs-dev_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:56:55.431 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking libcephfs-dev (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:55.526 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-ceph-argparse (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:55.527 INFO:teuthology.orchestra.run.smithi121.stdout:Adding system user cephadm....done 2022-06-29T15:56:55.643 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-06-29T15:56:55.646 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-html5lib. 2022-06-29T15:56:55.664 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../091-python3-html5lib_1.0.1-2_all.deb ... 2022-06-29T15:56:55.704 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-06-29T15:56:55.713 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2022-06-29T15:56:55.827 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-06-29T15:56:55.851 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package nvme-cli. 2022-06-29T15:56:55.871 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../087-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-06-29T15:56:55.912 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-06-29T15:56:55.927 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-06-29T15:56:55.978 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pyperclip (1.7.0-1) ... 2022-06-29T15:56:56.065 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-06-29T15:56:56.074 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-06-29T15:56:56.092 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../092-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-06-29T15:56:56.132 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-06-29T15:56:56.196 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-ceph-argparse (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:56.271 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-06-29T15:56:56.356 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-atomicwrites. 2022-06-29T15:56:56.373 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../088-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-06-29T15:56:56.383 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-06-29T15:56:56.414 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-06-29T15:56:56.514 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-06-29T15:56:56.525 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-06-29T15:56:56.556 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-06-29T15:56:56.619 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-packaging. 2022-06-29T15:56:56.635 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../093-python3-packaging_20.3-1_all.deb ... 2022-06-29T15:56:56.649 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-pyperclip (1.7.0-1) ... 2022-06-29T15:56:56.677 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-packaging (20.3-1) ... 2022-06-29T15:56:56.750 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-attr. 2022-06-29T15:56:56.768 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../089-python3-attr_19.3.0-2_all.deb ... 2022-06-29T15:56:56.809 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-06-29T15:56:56.852 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-06-29T15:56:56.865 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-06-29T15:56:56.950 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-06-29T15:56:57.039 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-pastescript. 2022-06-29T15:56:57.055 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../094-python3-pastescript_2.0.2-4_all.deb ... 2022-06-29T15:56:57.097 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-06-29T15:56:57.170 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-webencodings. 2022-06-29T15:56:57.186 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../090-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-06-29T15:56:57.228 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-06-29T15:56:57.233 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-06-29T15:56:57.242 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-06-29T15:56:57.271 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-jwt (1.7.1-2ubuntu2) ... 2022-06-29T15:56:57.450 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-pluggy. 2022-06-29T15:56:57.466 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../095-python3-pluggy_0.13.0-2_all.deb ... 2022-06-29T15:56:57.508 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-06-29T15:56:57.551 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-06-29T15:56:57.555 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-06-29T15:56:57.560 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-06-29T15:56:57.564 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-html5lib. 2022-06-29T15:56:57.581 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../091-python3-html5lib_1.0.1-2_all.deb ... 2022-06-29T15:56:57.623 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-06-29T15:56:57.829 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-psutil. 2022-06-29T15:56:57.848 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../096-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-06-29T15:56:57.882 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-colorama (0.4.3-1build1) ... 2022-06-29T15:56:57.886 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-06-29T15:56:57.894 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-06-29T15:56:57.928 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-06-29T15:56:58.026 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-06-29T15:56:58.046 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../092-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-06-29T15:56:58.101 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-06-29T15:56:58.174 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-06-29T15:56:58.212 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-06-29T15:56:58.229 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2022-06-29T15:56:58.282 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-py. 2022-06-29T15:56:58.300 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../097-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-06-29T15:56:58.339 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-06-29T15:56:58.459 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-06-29T15:56:58.547 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-06-29T15:56:58.563 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-06-29T15:56:58.604 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-packaging. 2022-06-29T15:56:58.623 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../093-python3-packaging_20.3-1_all.deb ... 2022-06-29T15:56:58.662 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-packaging (20.3-1) ... 2022-06-29T15:56:58.685 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-pyinotify. 2022-06-29T15:56:58.701 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../098-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-06-29T15:56:58.760 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-06-29T15:56:58.760 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-ceph-argparse (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:56:58.848 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2022-06-29T15:56:58.856 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-packaging (20.3-1) ... 2022-06-29T15:56:58.990 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-pastescript. 2022-06-29T15:56:59.006 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../094-python3-pastescript_2.0.2-4_all.deb ... 2022-06-29T15:56:59.048 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-06-29T15:56:59.069 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-06-29T15:56:59.071 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-pytest. 2022-06-29T15:56:59.088 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../099-python3-pytest_4.6.9-1_all.deb ... 2022-06-29T15:56:59.129 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-06-29T15:56:59.149 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-06-29T15:56:59.196 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pyperclip (1.7.0-1) ... 2022-06-29T15:56:59.216 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-06-29T15:56:59.275 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-06-29T15:56:59.444 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-pluggy. 2022-06-29T15:56:59.461 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../095-python3-pluggy_0.13.0-2_all.deb ... 2022-06-29T15:56:59.488 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-06-29T15:56:59.491 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package python3-simplejson. 2022-06-29T15:56:59.500 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-packaging (20.3-1) ... 2022-06-29T15:56:59.508 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../100-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-06-29T15:56:59.535 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-06-29T15:56:59.549 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-06-29T15:56:59.568 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-06-29T15:56:59.818 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-06-29T15:56:59.823 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-06-29T15:56:59.872 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-psutil. 2022-06-29T15:56:59.891 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../096-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-06-29T15:56:59.930 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-06-29T15:56:59.944 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-06-29T15:56:59.978 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-cmd2 (0.8.5-2.1) ... 2022-06-29T15:57:00.012 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package radosgw. 2022-06-29T15:57:00.029 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../101-radosgw_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:57:00.103 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking radosgw (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:00.116 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-06-29T15:57:00.229 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-06-29T15:57:00.283 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-py. 2022-06-29T15:57:00.287 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-06-29T15:57:00.300 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../097-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-06-29T15:57:00.341 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-06-29T15:57:00.517 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-06-29T15:57:00.630 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-cmd2 (0.8.5-2.1) ... 2022-06-29T15:57:00.647 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-06-29T15:57:00.704 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-pyinotify. 2022-06-29T15:57:00.720 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../098-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-06-29T15:57:00.762 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-06-29T15:57:00.818 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-06-29T15:57:00.923 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-06-29T15:57:01.016 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up socat (1.7.3.3-2) ... 2022-06-29T15:57:01.031 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-pytest. 2022-06-29T15:57:01.048 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../099-python3-pytest_4.6.9-1_all.deb ... 2022-06-29T15:57:01.081 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-06-29T15:57:01.142 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-ceph-common (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:01.148 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package rbd-fuse. 2022-06-29T15:57:01.167 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../102-rbd-fuse_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:57:01.170 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-06-29T15:57:01.198 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking rbd-fuse (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:01.299 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-06-29T15:57:01.418 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package python3-simplejson. 2022-06-29T15:57:01.437 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../100-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-06-29T15:57:01.443 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-06-29T15:57:01.476 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-06-29T15:57:01.479 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2022-06-29T15:57:01.507 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-attr (19.3.0-2) ... 2022-06-29T15:57:01.668 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up socat (1.7.3.3-2) ... 2022-06-29T15:57:01.794 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-ceph-common (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:01.820 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-06-29T15:57:01.830 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-06-29T15:57:01.853 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-06-29T15:57:01.871 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package radosgw. 2022-06-29T15:57:01.888 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../101-radosgw_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:57:01.921 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking radosgw (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:01.946 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-06-29T15:57:02.004 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-06-29T15:57:02.095 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-06-29T15:57:02.121 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-06-29T15:57:02.165 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-packaging (20.3-1) ... 2022-06-29T15:57:02.264 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-06-29T15:57:02.289 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up cephadm (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:02.390 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-06-29T15:57:02.438 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-06-29T15:57:02.466 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-06-29T15:57:02.573 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-06-29T15:57:02.592 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-06-29T15:57:02.717 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-06-29T15:57:02.882 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-06-29T15:57:02.893 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-06-29T15:57:02.968 INFO:teuthology.orchestra.run.smithi197.stdout:Adding system user cephadm....done 2022-06-29T15:57:02.999 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package rbd-fuse. 2022-06-29T15:57:03.001 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-06-29T15:57:03.016 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../102-rbd-fuse_15.2.16-304-gf25ffac7-1focal_amd64.deb ... 2022-06-29T15:57:03.024 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-06-29T15:57:03.074 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking rbd-fuse (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:03.129 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2022-06-29T15:57:03.294 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-06-29T15:57:03.303 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-cmd2 (0.8.5-2.1) ... 2022-06-29T15:57:03.373 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-attr (19.3.0-2) ... 2022-06-29T15:57:03.376 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-06-29T15:57:03.445 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-06-29T15:57:03.464 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-06-29T15:57:03.630 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-06-29T15:57:03.652 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-06-29T15:57:03.704 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-06-29T15:57:03.715 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-06-29T15:57:03.730 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-06-29T15:57:03.841 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-06-29T15:57:03.855 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-06-29T15:57:03.978 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-06-29T15:57:03.981 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-06-29T15:57:03.989 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-06-29T15:57:04.031 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-06-29T15:57:04.104 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-06-29T15:57:04.160 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-06-29T15:57:04.207 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up cephadm (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:04.357 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-06-29T15:57:04.392 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up socat (1.7.3.3-2) ... 2022-06-29T15:57:04.405 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-06-29T15:57:04.518 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-ceph-common (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:04.587 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-jwt (1.7.1-2ubuntu2) ... 2022-06-29T15:57:04.654 INFO:teuthology.orchestra.run.smithi060.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-06-29T15:57:04.707 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-06-29T15:57:04.819 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-06-29T15:57:04.871 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-06-29T15:57:04.909 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-06-29T15:57:04.919 INFO:teuthology.orchestra.run.smithi156.stdout:Adding system user cephadm....done 2022-06-29T15:57:05.041 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-06-29T15:57:05.061 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-06-29T15:57:05.088 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2022-06-29T15:57:05.164 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-colorama (0.4.3-1build1) ... 2022-06-29T15:57:05.204 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-06-29T15:57:05.344 INFO:teuthology.orchestra.run.smithi092.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-06-29T15:57:05.371 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-06-29T15:57:05.387 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-06-29T15:57:05.432 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-06-29T15:57:05.465 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-06-29T15:57:05.593 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-06-29T15:57:05.681 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-06-29T15:57:05.691 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-06-29T15:57:05.737 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-06-29T15:57:05.766 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-06-29T15:57:05.780 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-06-29T15:57:05.807 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-06-29T15:57:05.906 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-06-29T15:57:05.917 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-06-29T15:57:05.967 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-ceph-argparse (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:06.054 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-06-29T15:57:06.057 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-06-29T15:57:06.176 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-06-29T15:57:06.252 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-06-29T15:57:06.277 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-06-29T15:57:06.383 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-06-29T15:57:06.403 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-pyperclip (1.7.0-1) ... 2022-06-29T15:57:06.405 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-06-29T15:57:06.460 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-06-29T15:57:06.548 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-06-29T15:57:06.671 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-jwt (1.7.1-2ubuntu2) ... 2022-06-29T15:57:06.687 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-06-29T15:57:06.699 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-06-29T15:57:06.710 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up librados2 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:06.753 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-06-29T15:57:06.844 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-06-29T15:57:06.904 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-06-29T15:57:06.955 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-06-29T15:57:06.972 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-06-29T15:57:07.025 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-06-29T15:57:07.180 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-06-29T15:57:07.256 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-06-29T15:57:07.273 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-colorama (0.4.3-1build1) ... 2022-06-29T15:57:07.335 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up librados2 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:07.436 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-06-29T15:57:07.481 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-06-29T15:57:07.557 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-06-29T15:57:07.616 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-06-29T15:57:07.799 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-06-29T15:57:07.859 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-06-29T15:57:07.875 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-06-29T15:57:08.088 INFO:teuthology.orchestra.run.smithi121.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-06-29T15:57:08.193 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-ceph-argparse (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:08.227 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-06-29T15:57:08.343 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-06-29T15:57:08.367 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up librgw2 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:08.469 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-06-29T15:57:08.486 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-06-29T15:57:08.493 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-06-29T15:57:08.511 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2022-06-29T15:57:08.612 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-pyperclip (1.7.0-1) ... 2022-06-29T15:57:08.635 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-06-29T15:57:08.804 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-06-29T15:57:08.841 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up librgw2 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:08.871 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-06-29T15:57:08.921 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-06-29T15:57:08.975 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-06-29T15:57:09.012 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-06-29T15:57:09.093 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-06-29T15:57:09.156 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-06-29T15:57:09.163 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-packaging (20.3-1) ... 2022-06-29T15:57:09.231 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-06-29T15:57:09.307 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-06-29T15:57:09.440 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-06-29T15:57:09.466 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-06-29T15:57:09.478 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-06-29T15:57:09.532 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-06-29T15:57:09.540 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-06-29T15:57:09.689 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-06-29T15:57:09.801 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-06-29T15:57:09.804 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-06-29T15:57:09.808 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-06-29T15:57:09.908 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-06-29T15:57:09.998 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-06-29T15:57:10.085 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up librados2 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:10.090 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-06-29T15:57:10.209 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-06-29T15:57:10.227 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-cmd2 (0.8.5-2.1) ... 2022-06-29T15:57:10.244 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-06-29T15:57:10.341 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libcephfs2 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:10.417 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libcephfs2 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:10.475 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-06-29T15:57:10.543 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-06-29T15:57:10.545 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-06-29T15:57:10.553 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-06-29T15:57:10.802 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libradosstriper1 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:10.837 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2022-06-29T15:57:10.895 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libradosstriper1 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:10.913 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-06-29T15:57:10.962 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-06-29T15:57:11.021 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-06-29T15:57:11.171 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-06-29T15:57:11.265 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up socat (1.7.3.3-2) ... 2022-06-29T15:57:11.297 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-06-29T15:57:11.339 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-06-29T15:57:11.366 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-ceph-common (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:11.523 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-packaging (20.3-1) ... 2022-06-29T15:57:11.667 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-06-29T15:57:11.701 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up librgw2 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:11.826 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-06-29T15:57:11.849 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-06-29T15:57:11.952 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-06-29T15:57:11.975 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-06-29T15:57:12.027 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-06-29T15:57:12.162 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-06-29T15:57:12.268 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-06-29T15:57:12.321 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-06-29T15:57:12.385 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up librbd1 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:12.394 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up librbd1 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:12.421 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-06-29T15:57:12.520 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-06-29T15:57:12.522 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-06-29T15:57:12.528 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-06-29T15:57:12.638 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-06-29T15:57:12.647 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-06-29T15:57:12.671 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-06-29T15:57:12.678 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-cmd2 (0.8.5-2.1) ... 2022-06-29T15:57:12.772 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-06-29T15:57:12.830 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-06-29T15:57:12.857 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-06-29T15:57:12.917 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-06-29T15:57:12.979 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-06-29T15:57:13.109 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-06-29T15:57:13.142 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-06-29T15:57:13.148 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-06-29T15:57:13.242 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-rsa (4.0-3) ... 2022-06-29T15:57:13.251 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libcephfs2 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:13.274 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-rsa (4.0-3) ... 2022-06-29T15:57:13.377 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-06-29T15:57:13.380 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-06-29T15:57:13.435 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-06-29T15:57:13.543 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-06-29T15:57:13.576 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-06-29T15:57:13.585 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-06-29T15:57:13.696 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libradosstriper1 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:13.740 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up socat (1.7.3.3-2) ... 2022-06-29T15:57:13.822 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-06-29T15:57:13.861 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-mgr-modules-core (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:13.866 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-ceph-common (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:13.870 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-06-29T15:57:13.885 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-modules-core (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:14.020 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-fuse (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:14.020 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-fuse (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:14.131 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-06-29T15:57:14.179 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-06-29T15:57:14.193 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-06-29T15:57:14.247 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-06-29T15:57:14.247 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-06-29T15:57:14.270 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-06-29T15:57:14.270 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-06-29T15:57:14.506 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-06-29T15:57:14.578 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-06-29T15:57:14.738 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libcephfs-dev (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:14.745 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-06-29T15:57:14.833 INFO:teuthology.orchestra.run.smithi197.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-06-29T15:57:14.864 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-06-29T15:57:14.888 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libcephfs-dev (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:15.014 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-06-29T15:57:15.038 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-06-29T15:57:15.092 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-06-29T15:57:15.137 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up librbd1 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:15.164 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-06-29T15:57:15.190 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-06-29T15:57:15.226 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-06-29T15:57:15.262 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-06-29T15:57:15.333 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-06-29T15:57:15.388 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-06-29T15:57:15.491 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-06-29T15:57:15.515 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-06-29T15:57:15.525 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-06-29T15:57:15.544 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-06-29T15:57:15.648 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-06-29T15:57:15.784 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-06-29T15:57:15.808 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-06-29T15:57:15.902 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-rados (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:15.912 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-06-29T15:57:15.917 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-rsa (4.0-3) ... 2022-06-29T15:57:16.003 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-rbd (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:16.019 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-rados (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:16.038 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-06-29T15:57:16.085 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-06-29T15:57:16.113 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up rbd-fuse (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:16.145 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-rbd (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:16.164 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-06-29T15:57:16.211 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-06-29T15:57:16.214 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-routes (2.4.1-2) ... 2022-06-29T15:57:16.218 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-06-29T15:57:16.271 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up rbd-fuse (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:16.397 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-routes (2.4.1-2) ... 2022-06-29T15:57:16.495 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-06-29T15:57:16.511 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mgr-modules-core (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:16.524 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-06-29T15:57:16.616 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-06-29T15:57:16.637 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-fuse (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:16.796 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-06-29T15:57:16.800 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-06-29T15:57:16.825 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up librados2 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:16.861 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-06-29T15:57:16.861 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-06-29T15:57:16.934 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-rgw (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:16.960 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-06-29T15:57:17.072 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-06-29T15:57:17.077 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-cephfs (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:17.110 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-rgw (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:17.253 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-cephfs (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:17.347 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libcephfs-dev (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:17.361 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up cephfs-shell (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:17.390 INFO:teuthology.orchestra.run.smithi156.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-06-29T15:57:17.463 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-06-29T15:57:17.523 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-06-29T15:57:17.546 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up cephfs-shell (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:17.641 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-06-29T15:57:17.672 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-06-29T15:57:17.768 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-06-29T15:57:17.816 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-06-29T15:57:18.119 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-06-29T15:57:18.399 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-common (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:18.424 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up librgw2 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:18.476 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-06-29T15:57:18.495 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-06-29T15:57:18.592 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-common (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:18.602 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-06-29T15:57:18.646 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-06-29T15:57:18.726 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-06-29T15:57:18.772 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-06-29T15:57:18.854 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-rados (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:18.963 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-rbd (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:18.968 INFO:teuthology.orchestra.run.smithi060.stdout:Adding group ceph....done 2022-06-29T15:57:19.097 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up rbd-fuse (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:19.103 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-06-29T15:57:19.132 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-06-29T15:57:19.190 INFO:teuthology.orchestra.run.smithi092.stdout:Adding group ceph....done 2022-06-29T15:57:19.223 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-routes (2.4.1-2) ... 2022-06-29T15:57:19.362 INFO:teuthology.orchestra.run.smithi060.stdout:Adding system user ceph....done 2022-06-29T15:57:19.412 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-06-29T15:57:19.424 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up librados2 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:19.530 INFO:teuthology.orchestra.run.smithi092.stdout:Adding system user ceph....done 2022-06-29T15:57:19.547 INFO:teuthology.orchestra.run.smithi060.stdout:Setting system user ceph properties....done 2022-06-29T15:57:19.568 INFO:teuthology.orchestra.run.smithi060.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-06-29T15:57:19.574 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-06-29T15:57:19.618 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-06-29T15:57:19.714 INFO:teuthology.orchestra.run.smithi092.stdout:Setting system user ceph properties....done 2022-06-29T15:57:19.720 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-06-29T15:57:19.730 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-06-29T15:57:19.734 INFO:teuthology.orchestra.run.smithi092.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-06-29T15:57:19.883 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-06-29T15:57:19.919 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-rgw (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:20.050 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-06-29T15:57:20.070 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-cephfs (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:20.082 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up libcephfs2 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:20.208 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-06-29T15:57:20.214 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-06-29T15:57:20.363 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up cephfs-shell (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:20.497 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-06-29T15:57:20.526 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-test (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:20.543 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up libradosstriper1 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:20.678 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-06-29T15:57:20.685 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up radosgw (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:20.735 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-test (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:20.861 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up radosgw (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:20.948 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-06-29T15:57:20.948 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-06-29T15:57:21.004 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-06-29T15:57:21.023 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up librgw2 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:21.124 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-06-29T15:57:21.133 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-06-29T15:57:21.134 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-06-29T15:57:21.250 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-06-29T15:57:21.442 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-common (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:21.637 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-base (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:21.690 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-06-29T15:57:21.813 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-base (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:21.988 INFO:teuthology.orchestra.run.smithi121.stdout:Adding group ceph....done 2022-06-29T15:57:21.993 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-06-29T15:57:21.994 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-06-29T15:57:22.035 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up librbd1 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:22.183 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-06-29T15:57:22.194 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-06-29T15:57:22.312 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-06-29T15:57:22.313 INFO:teuthology.orchestra.run.smithi121.stdout:Adding system user ceph....done 2022-06-29T15:57:22.320 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-06-29T15:57:22.429 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-06-29T15:57:22.456 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mds (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:22.514 INFO:teuthology.orchestra.run.smithi121.stdout:Setting system user ceph properties....done 2022-06-29T15:57:22.538 INFO:teuthology.orchestra.run.smithi121.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-06-29T15:57:22.657 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-mds (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:22.672 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up libcephfs2 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:22.684 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-06-29T15:57:22.689 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-06-29T15:57:22.690 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-06-29T15:57:22.731 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-06-29T15:57:22.798 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-06-29T15:57:22.832 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-rsa (4.0-3) ... 2022-06-29T15:57:22.881 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-06-29T15:57:22.881 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-06-29T15:57:22.979 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-06-29T15:57:23.125 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up libradosstriper1 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:23.125 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-06-29T15:57:23.141 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:23.250 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-06-29T15:57:23.358 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-mgr (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:23.373 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-06-29T15:57:23.374 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-06-29T15:57:23.426 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up ceph-mgr-modules-core (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:23.452 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-test (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:23.578 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up radosgw (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:23.585 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up ceph-fuse (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:23.586 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-06-29T15:57:23.611 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-06-29T15:57:23.611 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-06-29T15:57:23.801 INFO:teuthology.orchestra.run.smithi197.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-06-29T15:57:23.802 INFO:teuthology.orchestra.run.smithi197.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-06-29T15:57:23.825 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:23.848 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-06-29T15:57:23.849 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-06-29T15:57:23.968 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-osd (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:24.101 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:24.246 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-osd (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:24.287 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up libcephfs-dev (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:24.404 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-06-29T15:57:24.505 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-base (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:24.528 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-06-29T15:57:24.528 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-06-29T15:57:24.590 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up librbd1 (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:24.708 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-06-29T15:57:24.714 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-06-29T15:57:24.745 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-06-29T15:57:24.745 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-06-29T15:57:24.854 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-06-29T15:57:24.862 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-06-29T15:57:24.979 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-k8sevents (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:24.985 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-06-29T15:57:25.015 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-06-29T15:57:25.114 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-diskprediction-local (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:25.221 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-mgr-k8sevents (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:25.256 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mon (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:25.295 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-06-29T15:57:25.352 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-mgr-diskprediction-local (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:25.415 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mds (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:25.421 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-rsa (4.0-3) ... 2022-06-29T15:57:25.482 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-mon (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:25.490 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-06-29T15:57:25.491 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-06-29T15:57:25.642 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-06-29T15:57:25.643 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-06-29T15:57:25.690 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-06-29T15:57:25.691 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-06-29T15:57:25.714 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-06-29T15:57:25.743 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-rados (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:25.869 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-rbd (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:25.949 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-cephadm (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:25.981 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up ceph-mgr-modules-core (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:26.011 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up rbd-fuse (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:26.082 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up ceph-fuse (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:26.092 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:26.100 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mgr (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:26.137 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-routes (2.4.1-2) ... 2022-06-29T15:57:26.158 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-mgr-cephadm (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:26.243 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-dashboard (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:26.293 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:26.302 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-06-29T15:57:26.302 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-06-29T15:57:26.332 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-06-29T15:57:26.332 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-06-29T15:57:26.387 INFO:teuthology.orchestra.run.smithi060.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2022-06-29T15:57:26.418 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-mgr-dashboard (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:26.507 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-06-29T15:57:26.587 INFO:teuthology.orchestra.run.smithi092.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2022-06-29T15:57:26.662 INFO:teuthology.orchestra.run.smithi060.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-06-29T15:57:26.776 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:26.783 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-rgw (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:26.842 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up libcephfs-dev (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:26.888 INFO:teuthology.orchestra.run.smithi092.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-06-29T15:57:26.894 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-osd (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:26.934 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-cephfs (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:27.027 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-06-29T15:57:27.235 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up cephfs-shell (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:27.329 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-06-29T15:57:27.365 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-06-29T15:57:27.366 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-06-29T15:57:27.386 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-06-29T15:57:27.854 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mgr-k8sevents (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:27.971 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mgr-diskprediction-local (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:27.980 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-06-29T15:57:28.131 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mon (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:28.298 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up ceph-common (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:28.358 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-06-29T15:57:28.358 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-06-29T15:57:28.371 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-rados (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:28.475 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-rbd (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:28.593 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up rbd-fuse (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:28.760 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-routes (2.4.1-2) ... 2022-06-29T15:57:28.807 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mgr-cephadm (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:28.809 INFO:teuthology.orchestra.run.smithi197.stdout:Adding group ceph....done 2022-06-29T15:57:28.950 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:29.095 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mgr-dashboard (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:29.147 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-06-29T15:57:29.219 INFO:teuthology.orchestra.run.smithi197.stdout:Adding system user ceph....done 2022-06-29T15:57:29.228 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2022-06-29T15:57:29.378 INFO:teuthology.orchestra.run.smithi197.stdout:Setting system user ceph properties....done 2022-06-29T15:57:29.399 INFO:teuthology.orchestra.run.smithi197.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-06-29T15:57:29.473 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-rgw (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:29.512 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-06-29T15:57:29.515 INFO:teuthology.orchestra.run.smithi060.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-06-29T15:57:29.547 INFO:teuthology.orchestra.run.smithi197.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-06-29T15:57:29.574 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-cephfs (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:29.657 INFO:teuthology.orchestra.run.smithi092.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-06-29T15:57:29.841 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up cephfs-shell (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:29.893 INFO:teuthology.orchestra.run.smithi197.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-06-29T15:57:29.946 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-06-29T15:57:30.032 INFO:teuthology.orchestra.run.smithi060.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-06-29T15:57:30.036 DEBUG:teuthology.orchestra.run.smithi060:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev 2022-06-29T15:57:30.110 INFO:teuthology.orchestra.run.smithi092.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-06-29T15:57:30.112 DEBUG:teuthology.orchestra.run.smithi092:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev 2022-06-29T15:57:30.114 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2022-06-29T15:57:30.184 INFO:teuthology.orchestra.run.smithi092.stdout:Reading package lists... 2022-06-29T15:57:30.296 INFO:teuthology.orchestra.run.smithi060.stdout:Building dependency tree... 2022-06-29T15:57:30.297 INFO:teuthology.orchestra.run.smithi060.stdout:Reading state information... 2022-06-29T15:57:30.363 INFO:teuthology.orchestra.run.smithi092.stdout:Building dependency tree... 2022-06-29T15:57:30.364 INFO:teuthology.orchestra.run.smithi092.stdout:Reading state information... 2022-06-29T15:57:30.375 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up ceph-test (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:30.465 INFO:teuthology.orchestra.run.smithi060.stdout:libtool-bin is already the newest version (2.4.6-14). 2022-06-29T15:57:30.465 INFO:teuthology.orchestra.run.smithi060.stdout:xfslibs-dev is already the newest version (5.3.0-1ubuntu2). 2022-06-29T15:57:30.465 INFO:teuthology.orchestra.run.smithi060.stdout:uuid-dev is already the newest version (2.34-0.1ubuntu9.3). 2022-06-29T15:57:30.466 INFO:teuthology.orchestra.run.smithi060.stdout:The following packages were automatically installed and are no longer required: 2022-06-29T15:57:30.467 INFO:teuthology.orchestra.run.smithi060.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-06-29T15:57:30.467 INFO:teuthology.orchestra.run.smithi060.stdout:Use 'sudo apt autoremove' to remove them. 2022-06-29T15:57:30.467 INFO:teuthology.orchestra.run.smithi060.stdout:The following additional packages will be installed: 2022-06-29T15:57:30.468 INFO:teuthology.orchestra.run.smithi060.stdout: libfl-dev libfl2 liblzo2-2 libssl1.1 2022-06-29T15:57:30.468 INFO:teuthology.orchestra.run.smithi060.stdout:Suggested packages: 2022-06-29T15:57:30.468 INFO:teuthology.orchestra.run.smithi060.stdout: bison-doc flex-doc indent-doc libssl-doc 2022-06-29T15:57:30.492 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up radosgw (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:30.504 INFO:teuthology.orchestra.run.smithi060.stdout:The following NEW packages will be installed: 2022-06-29T15:57:30.505 INFO:teuthology.orchestra.run.smithi060.stdout: bison dump flex indent libaio-dev libelf-dev libfl-dev libfl2 liblzo2-2 2022-06-29T15:57:30.505 INFO:teuthology.orchestra.run.smithi060.stdout: libssl-dev 2022-06-29T15:57:30.506 INFO:teuthology.orchestra.run.smithi060.stdout:The following packages will be upgraded: 2022-06-29T15:57:30.507 INFO:teuthology.orchestra.run.smithi060.stdout: libssl1.1 2022-06-29T15:57:30.519 INFO:teuthology.orchestra.run.smithi092.stdout:libtool-bin is already the newest version (2.4.6-14). 2022-06-29T15:57:30.519 INFO:teuthology.orchestra.run.smithi092.stdout:xfslibs-dev is already the newest version (5.3.0-1ubuntu2). 2022-06-29T15:57:30.520 INFO:teuthology.orchestra.run.smithi092.stdout:uuid-dev is already the newest version (2.34-0.1ubuntu9.3). 2022-06-29T15:57:30.520 INFO:teuthology.orchestra.run.smithi092.stdout:The following packages were automatically installed and are no longer required: 2022-06-29T15:57:30.521 INFO:teuthology.orchestra.run.smithi092.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-06-29T15:57:30.522 INFO:teuthology.orchestra.run.smithi092.stdout:Use 'sudo apt autoremove' to remove them. 2022-06-29T15:57:30.522 INFO:teuthology.orchestra.run.smithi092.stdout:The following additional packages will be installed: 2022-06-29T15:57:30.523 INFO:teuthology.orchestra.run.smithi092.stdout: libfl-dev libfl2 liblzo2-2 libssl1.1 2022-06-29T15:57:30.523 INFO:teuthology.orchestra.run.smithi092.stdout:Suggested packages: 2022-06-29T15:57:30.524 INFO:teuthology.orchestra.run.smithi092.stdout: bison-doc flex-doc indent-doc libssl-doc 2022-06-29T15:57:30.556 INFO:teuthology.orchestra.run.smithi092.stdout:The following NEW packages will be installed: 2022-06-29T15:57:30.556 INFO:teuthology.orchestra.run.smithi092.stdout: bison dump flex indent libaio-dev libelf-dev libfl-dev libfl2 liblzo2-2 2022-06-29T15:57:30.557 INFO:teuthology.orchestra.run.smithi092.stdout: libssl-dev 2022-06-29T15:57:30.558 INFO:teuthology.orchestra.run.smithi092.stdout:The following packages will be upgraded: 2022-06-29T15:57:30.559 INFO:teuthology.orchestra.run.smithi092.stdout: libssl1.1 2022-06-29T15:57:30.683 INFO:teuthology.orchestra.run.smithi060.stdout:1 upgraded, 10 newly installed, 0 to remove and 6 not upgraded. 2022-06-29T15:57:30.683 INFO:teuthology.orchestra.run.smithi060.stdout:Need to get 4,234 kB of archives. 2022-06-29T15:57:30.683 INFO:teuthology.orchestra.run.smithi060.stdout:After this operation, 12.4 MB of additional disk space will be used. 2022-06-29T15:57:30.684 INFO:teuthology.orchestra.run.smithi060.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 flex amd64 2.6.4-6.2 [317 kB] 2022-06-29T15:57:30.737 INFO:teuthology.orchestra.run.smithi092.stdout:1 upgraded, 10 newly installed, 0 to remove and 6 not upgraded. 2022-06-29T15:57:30.738 INFO:teuthology.orchestra.run.smithi092.stdout:Need to get 4,234 kB of archives. 2022-06-29T15:57:30.739 INFO:teuthology.orchestra.run.smithi092.stdout:After this operation, 12.4 MB of additional disk space will be used. 2022-06-29T15:57:30.739 INFO:teuthology.orchestra.run.smithi092.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 flex amd64 2.6.4-6.2 [317 kB] 2022-06-29T15:57:30.856 INFO:teuthology.orchestra.run.smithi197.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-06-29T15:57:30.856 INFO:teuthology.orchestra.run.smithi197.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-06-29T15:57:30.871 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up ceph-common (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:31.155 INFO:teuthology.orchestra.run.smithi060.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl1.1 amd64 1.1.1f-1ubuntu2.15 [1,321 kB] 2022-06-29T15:57:31.216 INFO:teuthology.orchestra.run.smithi092.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl1.1 amd64 1.1.1f-1ubuntu2.15 [1,321 kB] 2022-06-29T15:57:31.358 INFO:teuthology.orchestra.run.smithi060.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 bison amd64 2:3.5.1+dfsg-1 [657 kB] 2022-06-29T15:57:31.407 INFO:teuthology.orchestra.run.smithi060.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB] 2022-06-29T15:57:31.412 INFO:teuthology.orchestra.run.smithi060.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/universe amd64 dump amd64 0.4b46-6 [130 kB] 2022-06-29T15:57:31.416 INFO:teuthology.orchestra.run.smithi060.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 indent amd64 2.2.12-1 [86.9 kB] 2022-06-29T15:57:31.420 INFO:teuthology.orchestra.run.smithi060.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 libaio-dev amd64 0.3.112-5 [13.7 kB] 2022-06-29T15:57:31.420 INFO:teuthology.orchestra.run.smithi060.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 libelf-dev amd64 0.176-1.1build1 [57.0 kB] 2022-06-29T15:57:31.421 INFO:teuthology.orchestra.run.smithi092.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 bison amd64 2:3.5.1+dfsg-1 [657 kB] 2022-06-29T15:57:31.423 INFO:teuthology.orchestra.run.smithi060.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl2 amd64 2.6.4-6.2 [11.5 kB] 2022-06-29T15:57:31.423 INFO:teuthology.orchestra.run.smithi060.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl-dev amd64 2.6.4-6.2 [6,316 B] 2022-06-29T15:57:31.424 INFO:teuthology.orchestra.run.smithi156.stdout:Adding group ceph....done 2022-06-29T15:57:31.424 INFO:teuthology.orchestra.run.smithi060.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.15 [1,584 kB] 2022-06-29T15:57:31.469 INFO:teuthology.orchestra.run.smithi092.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB] 2022-06-29T15:57:31.470 INFO:teuthology.orchestra.run.smithi092.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/universe amd64 dump amd64 0.4b46-6 [130 kB] 2022-06-29T15:57:31.478 INFO:teuthology.orchestra.run.smithi092.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 indent amd64 2.2.12-1 [86.9 kB] 2022-06-29T15:57:31.481 INFO:teuthology.orchestra.run.smithi092.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 libaio-dev amd64 0.3.112-5 [13.7 kB] 2022-06-29T15:57:31.482 INFO:teuthology.orchestra.run.smithi092.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 libelf-dev amd64 0.176-1.1build1 [57.0 kB] 2022-06-29T15:57:31.484 INFO:teuthology.orchestra.run.smithi092.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl2 amd64 2.6.4-6.2 [11.5 kB] 2022-06-29T15:57:31.485 INFO:teuthology.orchestra.run.smithi092.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl-dev amd64 2.6.4-6.2 [6,316 B] 2022-06-29T15:57:31.485 INFO:teuthology.orchestra.run.smithi092.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.15 [1,584 kB] 2022-06-29T15:57:31.654 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up ceph-base (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:31.790 INFO:teuthology.orchestra.run.smithi060.stdout:Preconfiguring packages ... 2022-06-29T15:57:31.825 INFO:teuthology.orchestra.run.smithi156.stdout:Adding system user ceph....done 2022-06-29T15:57:31.842 INFO:teuthology.orchestra.run.smithi092.stdout:Preconfiguring packages ... 2022-06-29T15:57:31.955 INFO:teuthology.orchestra.run.smithi060.stdout:Fetched 4,234 kB in 1s (4,378 kB/s) 2022-06-29T15:57:31.976 INFO:teuthology.orchestra.run.smithi092.stdout:Fetched 4,234 kB in 1s (4,343 kB/s) 2022-06-29T15:57:32.026 INFO:teuthology.orchestra.run.smithi156.stdout:Setting system user ceph properties....done 2022-06-29T15:57:32.038 INFO:teuthology.orchestra.run.smithi197.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-06-29T15:57:32.051 INFO:teuthology.orchestra.run.smithi156.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-06-29T15:57:32.072 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package flex. 2022-06-29T15:57:32.114 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package flex. 2022-06-29T15:57:32.129 INFO:teuthology.orchestra.run.smithi060.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 146195 files and directories currently installed.) 2022-06-29T15:57:32.134 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../00-flex_2.6.4-6.2_amd64.deb ... 2022-06-29T15:57:32.170 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking flex (2.6.4-6.2) ... 2022-06-29T15:57:32.171 INFO:teuthology.orchestra.run.smithi092.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 146195 files and directories currently installed.) 2022-06-29T15:57:32.175 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../00-flex_2.6.4-6.2_amd64.deb ... 2022-06-29T15:57:32.203 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-06-29T15:57:32.221 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking flex (2.6.4-6.2) ... 2022-06-29T15:57:32.381 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-06-29T15:57:32.531 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up ceph-mds (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:32.531 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-06-29T15:57:32.750 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../01-libssl1.1_1.1.1f-1ubuntu2.15_amd64.deb ... 2022-06-29T15:57:32.760 INFO:teuthology.orchestra.run.smithi197.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-06-29T15:57:32.760 INFO:teuthology.orchestra.run.smithi197.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-06-29T15:57:32.802 INFO:teuthology.orchestra.run.smithi121.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-06-29T15:57:32.805 DEBUG:teuthology.orchestra.run.smithi121:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev 2022-06-29T15:57:32.817 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../01-libssl1.1_1.1.1f-1ubuntu2.15_amd64.deb ... 2022-06-29T15:57:32.874 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2022-06-29T15:57:32.926 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libssl1.1:amd64 (1.1.1f-1ubuntu2.15) over (1.1.1f-1ubuntu2.13) ... 2022-06-29T15:57:32.993 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libssl1.1:amd64 (1.1.1f-1ubuntu2.15) over (1.1.1f-1ubuntu2.13) ... 2022-06-29T15:57:32.999 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up ceph-test (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:33.060 INFO:teuthology.orchestra.run.smithi121.stdout:Building dependency tree... 2022-06-29T15:57:33.060 INFO:teuthology.orchestra.run.smithi121.stdout:Reading state information... 2022-06-29T15:57:33.123 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up radosgw (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:33.218 INFO:teuthology.orchestra.run.smithi121.stdout:libtool-bin is already the newest version (2.4.6-14). 2022-06-29T15:57:33.219 INFO:teuthology.orchestra.run.smithi121.stdout:xfslibs-dev is already the newest version (5.3.0-1ubuntu2). 2022-06-29T15:57:33.219 INFO:teuthology.orchestra.run.smithi121.stdout:uuid-dev is already the newest version (2.34-0.1ubuntu9.3). 2022-06-29T15:57:33.220 INFO:teuthology.orchestra.run.smithi121.stdout:The following packages were automatically installed and are no longer required: 2022-06-29T15:57:33.222 INFO:teuthology.orchestra.run.smithi121.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-06-29T15:57:33.222 INFO:teuthology.orchestra.run.smithi121.stdout:Use 'sudo apt autoremove' to remove them. 2022-06-29T15:57:33.222 INFO:teuthology.orchestra.run.smithi121.stdout:The following additional packages will be installed: 2022-06-29T15:57:33.223 INFO:teuthology.orchestra.run.smithi121.stdout: libfl-dev libfl2 liblzo2-2 libssl1.1 2022-06-29T15:57:33.223 INFO:teuthology.orchestra.run.smithi121.stdout:Suggested packages: 2022-06-29T15:57:33.224 INFO:teuthology.orchestra.run.smithi121.stdout: bison-doc flex-doc indent-doc libssl-doc 2022-06-29T15:57:33.254 INFO:teuthology.orchestra.run.smithi121.stdout:The following NEW packages will be installed: 2022-06-29T15:57:33.255 INFO:teuthology.orchestra.run.smithi121.stdout: bison dump flex indent libaio-dev libelf-dev libfl-dev libfl2 liblzo2-2 2022-06-29T15:57:33.256 INFO:teuthology.orchestra.run.smithi121.stdout: libssl-dev 2022-06-29T15:57:33.256 INFO:teuthology.orchestra.run.smithi121.stdout:The following packages will be upgraded: 2022-06-29T15:57:33.258 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up ceph-mgr (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:33.258 INFO:teuthology.orchestra.run.smithi121.stdout: libssl1.1 2022-06-29T15:57:33.308 INFO:teuthology.orchestra.run.smithi121.stdout:1 upgraded, 10 newly installed, 0 to remove and 6 not upgraded. 2022-06-29T15:57:33.308 INFO:teuthology.orchestra.run.smithi121.stdout:Need to get 4,234 kB of archives. 2022-06-29T15:57:33.309 INFO:teuthology.orchestra.run.smithi121.stdout:After this operation, 12.4 MB of additional disk space will be used. 2022-06-29T15:57:33.309 INFO:teuthology.orchestra.run.smithi121.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 flex amd64 2.6.4-6.2 [317 kB] 2022-06-29T15:57:33.392 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-06-29T15:57:33.392 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-06-29T15:57:33.404 INFO:teuthology.orchestra.run.smithi121.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl1.1 amd64 1.1.1f-1ubuntu2.15 [1,321 kB] 2022-06-29T15:57:33.413 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package bison. 2022-06-29T15:57:33.432 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../02-bison_2%3a3.5.1+dfsg-1_amd64.deb ... 2022-06-29T15:57:33.445 INFO:teuthology.orchestra.run.smithi121.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 bison amd64 2:3.5.1+dfsg-1 [657 kB] 2022-06-29T15:57:33.456 INFO:teuthology.orchestra.run.smithi121.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB] 2022-06-29T15:57:33.456 INFO:teuthology.orchestra.run.smithi121.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/universe amd64 dump amd64 0.4b46-6 [130 kB] 2022-06-29T15:57:33.457 INFO:teuthology.orchestra.run.smithi121.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 indent amd64 2.2.12-1 [86.9 kB] 2022-06-29T15:57:33.458 INFO:teuthology.orchestra.run.smithi121.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 libaio-dev amd64 0.3.112-5 [13.7 kB] 2022-06-29T15:57:33.459 INFO:teuthology.orchestra.run.smithi121.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 libelf-dev amd64 0.176-1.1build1 [57.0 kB] 2022-06-29T15:57:33.460 INFO:teuthology.orchestra.run.smithi121.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl2 amd64 2.6.4-6.2 [11.5 kB] 2022-06-29T15:57:33.460 INFO:teuthology.orchestra.run.smithi121.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl-dev amd64 2.6.4-6.2 [6,316 B] 2022-06-29T15:57:33.460 INFO:teuthology.orchestra.run.smithi121.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.15 [1,584 kB] 2022-06-29T15:57:33.479 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking bison (2:3.5.1+dfsg-1) ... 2022-06-29T15:57:33.503 INFO:teuthology.orchestra.run.smithi197.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-06-29T15:57:33.504 INFO:teuthology.orchestra.run.smithi197.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-06-29T15:57:33.547 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package bison. 2022-06-29T15:57:33.565 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../02-bison_2%3a3.5.1+dfsg-1_amd64.deb ... 2022-06-29T15:57:33.610 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking bison (2:3.5.1+dfsg-1) ... 2022-06-29T15:57:33.770 INFO:teuthology.orchestra.run.smithi121.stdout:Preconfiguring packages ... 2022-06-29T15:57:33.890 INFO:teuthology.orchestra.run.smithi121.stdout:Fetched 4,234 kB in 0s (20.0 MB/s) 2022-06-29T15:57:33.901 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package liblzo2-2:amd64. 2022-06-29T15:57:33.920 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../03-liblzo2-2_2.10-2_amd64.deb ... 2022-06-29T15:57:33.950 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:33.959 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking liblzo2-2:amd64 (2.10-2) ... 2022-06-29T15:57:34.029 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package flex. 2022-06-29T15:57:34.051 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package liblzo2-2:amd64. 2022-06-29T15:57:34.067 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../03-liblzo2-2_2.10-2_amd64.deb ... 2022-06-29T15:57:34.068 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up ceph-osd (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:34.086 INFO:teuthology.orchestra.run.smithi121.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 146195 files and directories currently installed.) 2022-06-29T15:57:34.091 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../00-flex_2.6.4-6.2_amd64.deb ... 2022-06-29T15:57:34.100 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up ceph-base (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:34.110 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking liblzo2-2:amd64 (2.10-2) ... 2022-06-29T15:57:34.144 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking flex (2.6.4-6.2) ... 2022-06-29T15:57:34.271 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package dump. 2022-06-29T15:57:34.289 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../04-dump_0.4b46-6_amd64.deb ... 2022-06-29T15:57:34.321 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking dump (0.4b46-6) ... 2022-06-29T15:57:34.454 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package dump. 2022-06-29T15:57:34.470 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../04-dump_0.4b46-6_amd64.deb ... 2022-06-29T15:57:34.496 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-06-29T15:57:34.521 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking dump (0.4b46-6) ... 2022-06-29T15:57:34.565 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package indent. 2022-06-29T15:57:34.570 INFO:teuthology.orchestra.run.smithi197.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-06-29T15:57:34.571 INFO:teuthology.orchestra.run.smithi197.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-06-29T15:57:34.584 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../05-indent_2.2.12-1_amd64.deb ... 2022-06-29T15:57:34.624 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking indent (2.2.12-1) ... 2022-06-29T15:57:34.690 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../01-libssl1.1_1.1.1f-1ubuntu2.15_amd64.deb ... 2022-06-29T15:57:34.833 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libssl1.1:amd64 (1.1.1f-1ubuntu2.15) over (1.1.1f-1ubuntu2.13) ... 2022-06-29T15:57:34.883 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package indent. 2022-06-29T15:57:34.903 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../05-indent_2.2.12-1_amd64.deb ... 2022-06-29T15:57:34.941 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking indent (2.2.12-1) ... 2022-06-29T15:57:34.969 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up ceph-mds (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:35.020 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up ceph-mgr-k8sevents (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:35.035 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libaio-dev:amd64. 2022-06-29T15:57:35.053 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../06-libaio-dev_0.3.112-5_amd64.deb ... 2022-06-29T15:57:35.094 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libaio-dev:amd64 (0.3.112-5) ... 2022-06-29T15:57:35.179 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up ceph-mgr-diskprediction-local (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:35.209 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-06-29T15:57:35.210 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-06-29T15:57:35.322 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up ceph-mon (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:35.328 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libaio-dev:amd64. 2022-06-29T15:57:35.330 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libelf-dev:amd64. 2022-06-29T15:57:35.348 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../06-libaio-dev_0.3.112-5_amd64.deb ... 2022-06-29T15:57:35.349 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../07-libelf-dev_0.176-1.1build1_amd64.deb ... 2022-06-29T15:57:35.378 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package bison. 2022-06-29T15:57:35.380 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libelf-dev:amd64 (0.176-1.1build1) ... 2022-06-29T15:57:35.386 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libaio-dev:amd64 (0.3.112-5) ... 2022-06-29T15:57:35.394 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../02-bison_2%3a3.5.1+dfsg-1_amd64.deb ... 2022-06-29T15:57:35.441 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking bison (2:3.5.1+dfsg-1) ... 2022-06-29T15:57:35.548 INFO:teuthology.orchestra.run.smithi197.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-06-29T15:57:35.548 INFO:teuthology.orchestra.run.smithi197.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-06-29T15:57:35.662 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up ceph-mgr (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:35.690 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libelf-dev:amd64. 2022-06-29T15:57:35.708 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../07-libelf-dev_0.176-1.1build1_amd64.deb ... 2022-06-29T15:57:35.717 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libfl2:amd64. 2022-06-29T15:57:35.738 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../08-libfl2_2.6.4-6.2_amd64.deb ... 2022-06-29T15:57:35.748 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libelf-dev:amd64 (0.176-1.1build1) ... 2022-06-29T15:57:35.775 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libfl2:amd64 (2.6.4-6.2) ... 2022-06-29T15:57:35.868 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-06-29T15:57:35.868 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-06-29T15:57:35.925 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package liblzo2-2:amd64. 2022-06-29T15:57:35.941 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../03-liblzo2-2_2.10-2_amd64.deb ... 2022-06-29T15:57:35.983 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking liblzo2-2:amd64 (2.10-2) ... 2022-06-29T15:57:36.027 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libfl-dev:amd64. 2022-06-29T15:57:36.045 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../09-libfl-dev_2.6.4-6.2_amd64.deb ... 2022-06-29T15:57:36.082 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up ceph-mgr-cephadm (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:36.086 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libfl-dev:amd64 (2.6.4-6.2) ... 2022-06-29T15:57:36.134 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libfl2:amd64. 2022-06-29T15:57:36.151 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../08-libfl2_2.6.4-6.2_amd64.deb ... 2022-06-29T15:57:36.194 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libfl2:amd64 (2.6.4-6.2) ... 2022-06-29T15:57:36.218 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up ceph (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:36.344 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package dump. 2022-06-29T15:57:36.351 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up ceph-mgr-dashboard (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:36.364 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:36.367 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../04-dump_0.4b46-6_amd64.deb ... 2022-06-29T15:57:36.372 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libssl-dev:amd64. 2022-06-29T15:57:36.389 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../10-libssl-dev_1.1.1f-1ubuntu2.15_amd64.deb ... 2022-06-29T15:57:36.402 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking dump (0.4b46-6) ... 2022-06-29T15:57:36.430 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.15) ... 2022-06-29T15:57:36.446 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libfl-dev:amd64. 2022-06-29T15:57:36.463 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../09-libfl-dev_2.6.4-6.2_amd64.deb ... 2022-06-29T15:57:36.495 INFO:teuthology.orchestra.run.smithi197.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2022-06-29T15:57:36.506 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up ceph-osd (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:36.529 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libfl-dev:amd64 (2.6.4-6.2) ... 2022-06-29T15:57:36.648 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package indent. 2022-06-29T15:57:36.665 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../05-indent_2.2.12-1_amd64.deb ... 2022-06-29T15:57:36.697 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking indent (2.2.12-1) ... 2022-06-29T15:57:36.765 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libssl-dev:amd64. 2022-06-29T15:57:36.784 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../10-libssl-dev_1.1.1f-1ubuntu2.15_amd64.deb ... 2022-06-29T15:57:36.824 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.15) ... 2022-06-29T15:57:36.829 INFO:teuthology.orchestra.run.smithi197.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-06-29T15:57:36.932 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up flex (2.6.4-6.2) ... 2022-06-29T15:57:36.992 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libaio-dev:amd64. 2022-06-29T15:57:37.001 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-06-29T15:57:37.001 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-06-29T15:57:37.008 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../06-libaio-dev_0.3.112-5_amd64.deb ... 2022-06-29T15:57:37.044 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libssl1.1:amd64 (1.1.1f-1ubuntu2.15) ... 2022-06-29T15:57:37.050 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libaio-dev:amd64 (0.3.112-5) ... 2022-06-29T15:57:37.292 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up flex (2.6.4-6.2) ... 2022-06-29T15:57:37.337 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up liblzo2-2:amd64 (2.10-2) ... 2022-06-29T15:57:37.345 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libelf-dev:amd64. 2022-06-29T15:57:37.364 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../07-libelf-dev_0.176-1.1build1_amd64.deb ... 2022-06-29T15:57:37.403 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libelf-dev:amd64 (0.176-1.1build1) ... 2022-06-29T15:57:37.445 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libssl1.1:amd64 (1.1.1f-1ubuntu2.15) ... 2022-06-29T15:57:37.455 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libaio-dev:amd64 (0.3.112-5) ... 2022-06-29T15:57:37.483 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up ceph-mgr-k8sevents (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:37.614 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.15) ... 2022-06-29T15:57:37.634 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up ceph-mgr-diskprediction-local (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:37.723 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libfl2:amd64. 2022-06-29T15:57:37.739 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../08-libfl2_2.6.4-6.2_amd64.deb ... 2022-06-29T15:57:37.740 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libelf-dev:amd64 (0.176-1.1build1) ... 2022-06-29T15:57:37.763 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up liblzo2-2:amd64 (2.10-2) ... 2022-06-29T15:57:37.785 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up ceph-mon (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:37.797 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libfl2:amd64 (2.6.4-6.2) ... 2022-06-29T15:57:37.866 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libfl2:amd64 (2.6.4-6.2) ... 2022-06-29T15:57:37.914 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libaio-dev:amd64 (0.3.112-5) ... 2022-06-29T15:57:37.992 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up dump (0.4b46-6) ... 2022-06-29T15:57:38.015 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.15) ... 2022-06-29T15:57:38.036 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-06-29T15:57:38.036 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-06-29T15:57:38.050 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libfl-dev:amd64. 2022-06-29T15:57:38.068 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../09-libfl-dev_2.6.4-6.2_amd64.deb ... 2022-06-29T15:57:38.087 INFO:teuthology.orchestra.run.smithi060.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2022-06-29T15:57:38.108 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libfl-dev:amd64 (2.6.4-6.2) ... 2022-06-29T15:57:38.116 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libelf-dev:amd64 (0.176-1.1build1) ... 2022-06-29T15:57:38.143 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up bison (2:3.5.1+dfsg-1) ... 2022-06-29T15:57:38.216 INFO:teuthology.orchestra.run.smithi060.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2022-06-29T15:57:38.259 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libfl2:amd64 (2.6.4-6.2) ... 2022-06-29T15:57:38.304 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up indent (2.2.12-1) ... 2022-06-29T15:57:38.385 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up dump (0.4b46-6) ... 2022-06-29T15:57:38.403 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libssl-dev:amd64. 2022-06-29T15:57:38.420 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../10-libssl-dev_1.1.1f-1ubuntu2.15_amd64.deb ... 2022-06-29T15:57:38.421 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libfl-dev:amd64 (2.6.4-6.2) ... 2022-06-29T15:57:38.461 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.15) ... 2022-06-29T15:57:38.480 INFO:teuthology.orchestra.run.smithi092.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2022-06-29T15:57:38.503 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up ceph-mgr-cephadm (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:38.545 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up bison (2:3.5.1+dfsg-1) ... 2022-06-29T15:57:38.548 INFO:teuthology.orchestra.run.smithi060.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-06-29T15:57:38.635 INFO:teuthology.orchestra.run.smithi092.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2022-06-29T15:57:38.682 INFO:teuthology.orchestra.run.smithi060.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-06-29T15:57:38.687 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up ceph (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:38.697 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up indent (2.2.12-1) ... 2022-06-29T15:57:38.797 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up ceph-mgr-dashboard (15.2.16-304-gf25ffac7-1focal) ... 2022-06-29T15:57:38.823 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libfl-dev:amd64 (2.6.4-6.2) ... 2022-06-29T15:57:38.933 INFO:teuthology.orchestra.run.smithi092.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-06-29T15:57:38.934 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up flex (2.6.4-6.2) ... 2022-06-29T15:57:38.936 INFO:teuthology.orchestra.run.smithi156.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2022-06-29T15:57:39.074 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libssl1.1:amd64 (1.1.1f-1ubuntu2.15) ... 2022-06-29T15:57:39.084 INFO:teuthology.orchestra.run.smithi092.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-06-29T15:57:39.232 INFO:teuthology.orchestra.run.smithi156.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-06-29T15:57:39.384 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up liblzo2-2:amd64 (2.10-2) ... 2022-06-29T15:57:39.510 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libaio-dev:amd64 (0.3.112-5) ... 2022-06-29T15:57:39.582 INFO:teuthology.orchestra.run.smithi197.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-06-29T15:57:39.636 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.15) ... 2022-06-29T15:57:39.787 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libelf-dev:amd64 (0.176-1.1build1) ... 2022-06-29T15:57:39.913 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libfl2:amd64 (2.6.4-6.2) ... 2022-06-29T15:57:40.010 INFO:teuthology.orchestra.run.smithi197.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-06-29T15:57:40.012 DEBUG:teuthology.orchestra.run.smithi197:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev 2022-06-29T15:57:40.056 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up dump (0.4b46-6) ... 2022-06-29T15:57:40.082 INFO:teuthology.orchestra.run.smithi197.stdout:Reading package lists... 2022-06-29T15:57:40.151 INFO:teuthology.orchestra.run.smithi121.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2022-06-29T15:57:40.232 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up bison (2:3.5.1+dfsg-1) ... 2022-06-29T15:57:40.259 INFO:teuthology.orchestra.run.smithi197.stdout:Building dependency tree... 2022-06-29T15:57:40.260 INFO:teuthology.orchestra.run.smithi197.stdout:Reading state information... 2022-06-29T15:57:40.322 INFO:teuthology.orchestra.run.smithi121.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2022-06-29T15:57:40.393 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up indent (2.2.12-1) ... 2022-06-29T15:57:40.419 INFO:teuthology.orchestra.run.smithi197.stdout:libtool-bin is already the newest version (2.4.6-14). 2022-06-29T15:57:40.419 INFO:teuthology.orchestra.run.smithi197.stdout:xfslibs-dev is already the newest version (5.3.0-1ubuntu2). 2022-06-29T15:57:40.419 INFO:teuthology.orchestra.run.smithi197.stdout:uuid-dev is already the newest version (2.34-0.1ubuntu9.3). 2022-06-29T15:57:40.420 INFO:teuthology.orchestra.run.smithi197.stdout:The following packages were automatically installed and are no longer required: 2022-06-29T15:57:40.421 INFO:teuthology.orchestra.run.smithi197.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-06-29T15:57:40.422 INFO:teuthology.orchestra.run.smithi197.stdout:Use 'sudo apt autoremove' to remove them. 2022-06-29T15:57:40.422 INFO:teuthology.orchestra.run.smithi197.stdout:The following additional packages will be installed: 2022-06-29T15:57:40.423 INFO:teuthology.orchestra.run.smithi197.stdout: libfl-dev libfl2 liblzo2-2 libssl1.1 2022-06-29T15:57:40.423 INFO:teuthology.orchestra.run.smithi197.stdout:Suggested packages: 2022-06-29T15:57:40.424 INFO:teuthology.orchestra.run.smithi197.stdout: bison-doc flex-doc indent-doc libssl-doc 2022-06-29T15:57:40.454 INFO:teuthology.orchestra.run.smithi197.stdout:The following NEW packages will be installed: 2022-06-29T15:57:40.454 INFO:teuthology.orchestra.run.smithi197.stdout: bison dump flex indent libaio-dev libelf-dev libfl-dev libfl2 liblzo2-2 2022-06-29T15:57:40.455 INFO:teuthology.orchestra.run.smithi197.stdout: libssl-dev 2022-06-29T15:57:40.456 INFO:teuthology.orchestra.run.smithi197.stdout:The following packages will be upgraded: 2022-06-29T15:57:40.457 INFO:teuthology.orchestra.run.smithi197.stdout: libssl1.1 2022-06-29T15:57:40.501 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libfl-dev:amd64 (2.6.4-6.2) ... 2022-06-29T15:57:40.509 INFO:teuthology.orchestra.run.smithi197.stdout:1 upgraded, 10 newly installed, 0 to remove and 6 not upgraded. 2022-06-29T15:57:40.509 INFO:teuthology.orchestra.run.smithi197.stdout:Need to get 4,234 kB of archives. 2022-06-29T15:57:40.509 INFO:teuthology.orchestra.run.smithi197.stdout:After this operation, 12.4 MB of additional disk space will be used. 2022-06-29T15:57:40.509 INFO:teuthology.orchestra.run.smithi197.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 flex amd64 2.6.4-6.2 [317 kB] 2022-06-29T15:57:40.608 INFO:teuthology.orchestra.run.smithi197.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl1.1 amd64 1.1.1f-1ubuntu2.15 [1,321 kB] 2022-06-29T15:57:40.648 INFO:teuthology.orchestra.run.smithi197.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 bison amd64 2:3.5.1+dfsg-1 [657 kB] 2022-06-29T15:57:40.659 INFO:teuthology.orchestra.run.smithi197.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB] 2022-06-29T15:57:40.660 INFO:teuthology.orchestra.run.smithi197.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/universe amd64 dump amd64 0.4b46-6 [130 kB] 2022-06-29T15:57:40.661 INFO:teuthology.orchestra.run.smithi197.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 indent amd64 2.2.12-1 [86.9 kB] 2022-06-29T15:57:40.664 INFO:teuthology.orchestra.run.smithi197.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 libaio-dev amd64 0.3.112-5 [13.7 kB] 2022-06-29T15:57:40.664 INFO:teuthology.orchestra.run.smithi197.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 libelf-dev amd64 0.176-1.1build1 [57.0 kB] 2022-06-29T15:57:40.664 INFO:teuthology.orchestra.run.smithi197.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl2 amd64 2.6.4-6.2 [11.5 kB] 2022-06-29T15:57:40.664 INFO:teuthology.orchestra.run.smithi197.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl-dev amd64 2.6.4-6.2 [6,316 B] 2022-06-29T15:57:40.665 INFO:teuthology.orchestra.run.smithi197.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.15 [1,584 kB] 2022-06-29T15:57:40.678 INFO:teuthology.orchestra.run.smithi060.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2022-06-29T15:57:40.687 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-06-29T15:57:40.805 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-06-29T15:57:40.979 INFO:teuthology.orchestra.run.smithi197.stdout:Preconfiguring packages ... 2022-06-29T15:57:40.996 INFO:teuthology.orchestra.run.smithi092.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2022-06-29T15:57:41.114 INFO:teuthology.orchestra.run.smithi197.stdout:Fetched 4,234 kB in 0s (19.8 MB/s) 2022-06-29T15:57:41.355 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package flex. 2022-06-29T15:57:41.404 INFO:teuthology.orchestra.run.smithi060.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-06-29T15:57:41.407 DEBUG:teuthology.parallel:result is None 2022-06-29T15:57:41.413 INFO:teuthology.orchestra.run.smithi197.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 146195 files and directories currently installed.) 2022-06-29T15:57:41.417 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../00-flex_2.6.4-6.2_amd64.deb ... 2022-06-29T15:57:41.462 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking flex (2.6.4-6.2) ... 2022-06-29T15:57:41.750 INFO:teuthology.orchestra.run.smithi092.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-06-29T15:57:41.753 DEBUG:teuthology.parallel:result is None 2022-06-29T15:57:42.011 INFO:teuthology.orchestra.run.smithi156.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-06-29T15:57:42.084 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../01-libssl1.1_1.1.1f-1ubuntu2.15_amd64.deb ... 2022-06-29T15:57:42.286 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking libssl1.1:amd64 (1.1.1f-1ubuntu2.15) over (1.1.1f-1ubuntu2.13) ... 2022-06-29T15:57:42.467 INFO:teuthology.orchestra.run.smithi156.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-06-29T15:57:42.470 DEBUG:teuthology.orchestra.run.smithi156:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev 2022-06-29T15:57:42.544 INFO:teuthology.orchestra.run.smithi156.stdout:Reading package lists... 2022-06-29T15:57:42.721 INFO:teuthology.orchestra.run.smithi156.stdout:Building dependency tree... 2022-06-29T15:57:42.722 INFO:teuthology.orchestra.run.smithi156.stdout:Reading state information... 2022-06-29T15:57:42.757 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2022-06-29T15:57:42.832 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package bison. 2022-06-29T15:57:42.848 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../02-bison_2%3a3.5.1+dfsg-1_amd64.deb ... 2022-06-29T15:57:42.879 INFO:teuthology.orchestra.run.smithi156.stdout:libtool-bin is already the newest version (2.4.6-14). 2022-06-29T15:57:42.880 INFO:teuthology.orchestra.run.smithi156.stdout:xfslibs-dev is already the newest version (5.3.0-1ubuntu2). 2022-06-29T15:57:42.880 INFO:teuthology.orchestra.run.smithi156.stdout:uuid-dev is already the newest version (2.34-0.1ubuntu9.3). 2022-06-29T15:57:42.880 INFO:teuthology.orchestra.run.smithi156.stdout:The following packages were automatically installed and are no longer required: 2022-06-29T15:57:42.881 INFO:teuthology.orchestra.run.smithi156.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-06-29T15:57:42.881 INFO:teuthology.orchestra.run.smithi156.stdout:Use 'sudo apt autoremove' to remove them. 2022-06-29T15:57:42.882 INFO:teuthology.orchestra.run.smithi156.stdout:The following additional packages will be installed: 2022-06-29T15:57:42.882 INFO:teuthology.orchestra.run.smithi156.stdout: libfl-dev libfl2 liblzo2-2 libssl1.1 2022-06-29T15:57:42.882 INFO:teuthology.orchestra.run.smithi156.stdout:Suggested packages: 2022-06-29T15:57:42.883 INFO:teuthology.orchestra.run.smithi156.stdout: bison-doc flex-doc indent-doc libssl-doc 2022-06-29T15:57:42.894 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking bison (2:3.5.1+dfsg-1) ... 2022-06-29T15:57:42.915 INFO:teuthology.orchestra.run.smithi156.stdout:The following NEW packages will be installed: 2022-06-29T15:57:42.916 INFO:teuthology.orchestra.run.smithi156.stdout: bison dump flex indent libaio-dev libelf-dev libfl-dev libfl2 liblzo2-2 2022-06-29T15:57:42.916 INFO:teuthology.orchestra.run.smithi156.stdout: libssl-dev 2022-06-29T15:57:42.917 INFO:teuthology.orchestra.run.smithi156.stdout:The following packages will be upgraded: 2022-06-29T15:57:42.918 INFO:teuthology.orchestra.run.smithi156.stdout: libssl1.1 2022-06-29T15:57:43.096 INFO:teuthology.orchestra.run.smithi156.stdout:1 upgraded, 10 newly installed, 0 to remove and 6 not upgraded. 2022-06-29T15:57:43.096 INFO:teuthology.orchestra.run.smithi156.stdout:Need to get 4,234 kB of archives. 2022-06-29T15:57:43.096 INFO:teuthology.orchestra.run.smithi156.stdout:After this operation, 12.4 MB of additional disk space will be used. 2022-06-29T15:57:43.097 INFO:teuthology.orchestra.run.smithi156.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 flex amd64 2.6.4-6.2 [317 kB] 2022-06-29T15:57:43.286 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package liblzo2-2:amd64. 2022-06-29T15:57:43.303 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../03-liblzo2-2_2.10-2_amd64.deb ... 2022-06-29T15:57:43.353 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking liblzo2-2:amd64 (2.10-2) ... 2022-06-29T15:57:43.462 INFO:teuthology.orchestra.run.smithi121.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-06-29T15:57:43.464 DEBUG:teuthology.parallel:result is None 2022-06-29T15:57:43.569 INFO:teuthology.orchestra.run.smithi156.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl1.1 amd64 1.1.1f-1ubuntu2.15 [1,321 kB] 2022-06-29T15:57:43.722 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package dump. 2022-06-29T15:57:43.740 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../04-dump_0.4b46-6_amd64.deb ... 2022-06-29T15:57:43.773 INFO:teuthology.orchestra.run.smithi156.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 bison amd64 2:3.5.1+dfsg-1 [657 kB] 2022-06-29T15:57:43.780 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking dump (0.4b46-6) ... 2022-06-29T15:57:43.821 INFO:teuthology.orchestra.run.smithi156.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB] 2022-06-29T15:57:43.823 INFO:teuthology.orchestra.run.smithi156.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/universe amd64 dump amd64 0.4b46-6 [130 kB] 2022-06-29T15:57:43.829 INFO:teuthology.orchestra.run.smithi156.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 indent amd64 2.2.12-1 [86.9 kB] 2022-06-29T15:57:43.833 INFO:teuthology.orchestra.run.smithi156.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 libaio-dev amd64 0.3.112-5 [13.7 kB] 2022-06-29T15:57:43.834 INFO:teuthology.orchestra.run.smithi156.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 libelf-dev amd64 0.176-1.1build1 [57.0 kB] 2022-06-29T15:57:43.837 INFO:teuthology.orchestra.run.smithi156.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl2 amd64 2.6.4-6.2 [11.5 kB] 2022-06-29T15:57:43.838 INFO:teuthology.orchestra.run.smithi156.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl-dev amd64 2.6.4-6.2 [6,316 B] 2022-06-29T15:57:43.838 INFO:teuthology.orchestra.run.smithi156.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.15 [1,584 kB] 2022-06-29T15:57:44.067 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package indent. 2022-06-29T15:57:44.084 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../05-indent_2.2.12-1_amd64.deb ... 2022-06-29T15:57:44.134 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking indent (2.2.12-1) ... 2022-06-29T15:57:44.193 INFO:teuthology.orchestra.run.smithi156.stdout:Preconfiguring packages ... 2022-06-29T15:57:44.330 INFO:teuthology.orchestra.run.smithi156.stdout:Fetched 4,234 kB in 1s (4,367 kB/s) 2022-06-29T15:57:44.378 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package libaio-dev:amd64. 2022-06-29T15:57:44.394 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../06-libaio-dev_0.3.112-5_amd64.deb ... 2022-06-29T15:57:44.428 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking libaio-dev:amd64 (0.3.112-5) ... 2022-06-29T15:57:44.467 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package flex. 2022-06-29T15:57:44.528 INFO:teuthology.orchestra.run.smithi156.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 146195 files and directories currently installed.) 2022-06-29T15:57:44.533 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../00-flex_2.6.4-6.2_amd64.deb ... 2022-06-29T15:57:44.598 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking flex (2.6.4-6.2) ... 2022-06-29T15:57:44.698 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package libelf-dev:amd64. 2022-06-29T15:57:44.715 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../07-libelf-dev_0.176-1.1build1_amd64.deb ... 2022-06-29T15:57:44.756 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking libelf-dev:amd64 (0.176-1.1build1) ... 2022-06-29T15:57:45.126 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package libfl2:amd64. 2022-06-29T15:57:45.144 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../08-libfl2_2.6.4-6.2_amd64.deb ... 2022-06-29T15:57:45.184 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking libfl2:amd64 (2.6.4-6.2) ... 2022-06-29T15:57:45.204 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../01-libssl1.1_1.1.1f-1ubuntu2.15_amd64.deb ... 2022-06-29T15:57:45.387 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking libssl1.1:amd64 (1.1.1f-1ubuntu2.15) over (1.1.1f-1ubuntu2.13) ... 2022-06-29T15:57:45.412 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package libfl-dev:amd64. 2022-06-29T15:57:45.428 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../09-libfl-dev_2.6.4-6.2_amd64.deb ... 2022-06-29T15:57:45.461 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking libfl-dev:amd64 (2.6.4-6.2) ... 2022-06-29T15:57:45.689 INFO:teuthology.orchestra.run.smithi197.stdout:Selecting previously unselected package libssl-dev:amd64. 2022-06-29T15:57:45.709 INFO:teuthology.orchestra.run.smithi197.stdout:Preparing to unpack .../10-libssl-dev_1.1.1f-1ubuntu2.15_amd64.deb ... 2022-06-29T15:57:45.739 INFO:teuthology.orchestra.run.smithi197.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.15) ... 2022-06-29T15:57:46.033 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package bison. 2022-06-29T15:57:46.050 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../02-bison_2%3a3.5.1+dfsg-1_amd64.deb ... 2022-06-29T15:57:46.096 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking bison (2:3.5.1+dfsg-1) ... 2022-06-29T15:57:46.242 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up flex (2.6.4-6.2) ... 2022-06-29T15:57:46.464 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up libssl1.1:amd64 (1.1.1f-1ubuntu2.15) ... 2022-06-29T15:57:46.554 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package liblzo2-2:amd64. 2022-06-29T15:57:46.572 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../03-liblzo2-2_2.10-2_amd64.deb ... 2022-06-29T15:57:46.612 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking liblzo2-2:amd64 (2.10-2) ... 2022-06-29T15:57:46.788 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up liblzo2-2:amd64 (2.10-2) ... 2022-06-29T15:57:46.914 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up libaio-dev:amd64 (0.3.112-5) ... 2022-06-29T15:57:46.990 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package dump. 2022-06-29T15:57:47.008 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../04-dump_0.4b46-6_amd64.deb ... 2022-06-29T15:57:47.057 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.15) ... 2022-06-29T15:57:47.074 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking dump (0.4b46-6) ... 2022-06-29T15:57:47.182 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up libelf-dev:amd64 (0.176-1.1build1) ... 2022-06-29T15:57:47.308 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up libfl2:amd64 (2.6.4-6.2) ... 2022-06-29T15:57:47.338 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package indent. 2022-06-29T15:57:47.355 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../05-indent_2.2.12-1_amd64.deb ... 2022-06-29T15:57:47.421 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking indent (2.2.12-1) ... 2022-06-29T15:57:47.459 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up dump (0.4b46-6) ... 2022-06-29T15:57:47.555 INFO:teuthology.orchestra.run.smithi197.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2022-06-29T15:57:47.619 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up bison (2:3.5.1+dfsg-1) ... 2022-06-29T15:57:47.682 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package libaio-dev:amd64. 2022-06-29T15:57:47.702 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../06-libaio-dev_0.3.112-5_amd64.deb ... 2022-06-29T15:57:47.709 INFO:teuthology.orchestra.run.smithi197.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2022-06-29T15:57:47.773 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking libaio-dev:amd64 (0.3.112-5) ... 2022-06-29T15:57:47.779 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up indent (2.2.12-1) ... 2022-06-29T15:57:47.930 INFO:teuthology.orchestra.run.smithi197.stdout:Setting up libfl-dev:amd64 (2.6.4-6.2) ... 2022-06-29T15:57:48.032 INFO:teuthology.orchestra.run.smithi197.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-06-29T15:57:48.052 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package libelf-dev:amd64. 2022-06-29T15:57:48.068 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../07-libelf-dev_0.176-1.1build1_amd64.deb ... 2022-06-29T15:57:48.126 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking libelf-dev:amd64 (0.176-1.1build1) ... 2022-06-29T15:57:48.175 INFO:teuthology.orchestra.run.smithi197.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-06-29T15:57:48.430 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package libfl2:amd64. 2022-06-29T15:57:48.450 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../08-libfl2_2.6.4-6.2_amd64.deb ... 2022-06-29T15:57:48.488 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking libfl2:amd64 (2.6.4-6.2) ... 2022-06-29T15:57:48.724 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package libfl-dev:amd64. 2022-06-29T15:57:48.743 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../09-libfl-dev_2.6.4-6.2_amd64.deb ... 2022-06-29T15:57:48.782 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking libfl-dev:amd64 (2.6.4-6.2) ... 2022-06-29T15:57:49.068 INFO:teuthology.orchestra.run.smithi156.stdout:Selecting previously unselected package libssl-dev:amd64. 2022-06-29T15:57:49.084 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing to unpack .../10-libssl-dev_1.1.1f-1ubuntu2.15_amd64.deb ... 2022-06-29T15:57:49.126 INFO:teuthology.orchestra.run.smithi156.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.15) ... 2022-06-29T15:57:49.659 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up flex (2.6.4-6.2) ... 2022-06-29T15:57:49.782 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up libssl1.1:amd64 (1.1.1f-1ubuntu2.15) ... 2022-06-29T15:57:50.125 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up liblzo2-2:amd64 (2.10-2) ... 2022-06-29T15:57:50.135 INFO:teuthology.orchestra.run.smithi197.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2022-06-29T15:57:50.276 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up libaio-dev:amd64 (0.3.112-5) ... 2022-06-29T15:57:50.402 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.15) ... 2022-06-29T15:57:50.528 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up libelf-dev:amd64 (0.176-1.1build1) ... 2022-06-29T15:57:50.696 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up libfl2:amd64 (2.6.4-6.2) ... 2022-06-29T15:57:50.822 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up dump (0.4b46-6) ... 2022-06-29T15:57:50.886 INFO:teuthology.orchestra.run.smithi197.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-06-29T15:57:50.889 DEBUG:teuthology.parallel:result is None 2022-06-29T15:57:50.917 INFO:teuthology.orchestra.run.smithi156.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2022-06-29T15:57:51.127 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up bison (2:3.5.1+dfsg-1) ... 2022-06-29T15:57:51.728 INFO:teuthology.orchestra.run.smithi156.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2022-06-29T15:57:51.851 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up indent (2.2.12-1) ... 2022-06-29T15:57:51.986 INFO:teuthology.orchestra.run.smithi156.stdout:Setting up libfl-dev:amd64 (2.6.4-6.2) ... 2022-06-29T15:57:52.146 INFO:teuthology.orchestra.run.smithi156.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-06-29T15:57:52.306 INFO:teuthology.orchestra.run.smithi156.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-06-29T15:57:54.333 INFO:teuthology.orchestra.run.smithi156.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2022-06-29T15:57:55.112 INFO:teuthology.orchestra.run.smithi156.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-06-29T15:57:55.114 DEBUG:teuthology.parallel:result is None 2022-06-29T15:57:55.115 DEBUG:teuthology.orchestra.run.smithi060:> sudo lsb_release -is 2022-06-29T15:57:55.167 INFO:teuthology.orchestra.run.smithi060.stdout:Ubuntu 2022-06-29T15:57:55.172 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-06-29T15:57:55.173 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=f25ffac72e5a9e63860ff7deb6767a11f0559a16 2022-06-29T15:57:55.355 DEBUG:teuthology.orchestra.run.smithi060:> dpkg-query -W -f '${Version}' ceph 2022-06-29T15:57:55.378 INFO:teuthology.orchestra.run.smithi060.stdout:15.2.16-304-gf25ffac7-1focal 2022-06-29T15:57:55.379 INFO:teuthology.packaging:The installed version of ceph is 15.2.16-304-gf25ffac7-1focal 2022-06-29T15:57:55.380 INFO:teuthology.task.install:The correct ceph version 15.2.16-304-gf25ffac7-1focal is installed. 2022-06-29T15:57:55.381 DEBUG:teuthology.orchestra.run.smithi092:> sudo lsb_release -is 2022-06-29T15:57:55.433 INFO:teuthology.orchestra.run.smithi092.stdout:Ubuntu 2022-06-29T15:57:55.439 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-06-29T15:57:55.440 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=f25ffac72e5a9e63860ff7deb6767a11f0559a16 2022-06-29T15:57:55.607 DEBUG:teuthology.orchestra.run.smithi092:> dpkg-query -W -f '${Version}' ceph 2022-06-29T15:57:55.630 INFO:teuthology.orchestra.run.smithi092.stdout:15.2.16-304-gf25ffac7-1focal 2022-06-29T15:57:55.631 INFO:teuthology.packaging:The installed version of ceph is 15.2.16-304-gf25ffac7-1focal 2022-06-29T15:57:55.631 INFO:teuthology.task.install:The correct ceph version 15.2.16-304-gf25ffac7-1focal is installed. 2022-06-29T15:57:55.632 DEBUG:teuthology.orchestra.run.smithi121:> sudo lsb_release -is 2022-06-29T15:57:55.686 INFO:teuthology.orchestra.run.smithi121.stdout:Ubuntu 2022-06-29T15:57:55.691 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-06-29T15:57:55.692 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=f25ffac72e5a9e63860ff7deb6767a11f0559a16 2022-06-29T15:57:55.853 DEBUG:teuthology.orchestra.run.smithi121:> dpkg-query -W -f '${Version}' ceph 2022-06-29T15:57:55.877 INFO:teuthology.orchestra.run.smithi121.stdout:15.2.16-304-gf25ffac7-1focal 2022-06-29T15:57:55.878 INFO:teuthology.packaging:The installed version of ceph is 15.2.16-304-gf25ffac7-1focal 2022-06-29T15:57:55.878 INFO:teuthology.task.install:The correct ceph version 15.2.16-304-gf25ffac7-1focal is installed. 2022-06-29T15:57:55.879 DEBUG:teuthology.orchestra.run.smithi156:> sudo lsb_release -is 2022-06-29T15:57:55.932 INFO:teuthology.orchestra.run.smithi156.stdout:Ubuntu 2022-06-29T15:57:55.937 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-06-29T15:57:55.938 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=f25ffac72e5a9e63860ff7deb6767a11f0559a16 2022-06-29T15:57:56.106 DEBUG:teuthology.orchestra.run.smithi156:> dpkg-query -W -f '${Version}' ceph 2022-06-29T15:57:56.129 INFO:teuthology.orchestra.run.smithi156.stdout:15.2.16-304-gf25ffac7-1focal 2022-06-29T15:57:56.130 INFO:teuthology.packaging:The installed version of ceph is 15.2.16-304-gf25ffac7-1focal 2022-06-29T15:57:56.130 INFO:teuthology.task.install:The correct ceph version 15.2.16-304-gf25ffac7-1focal is installed. 2022-06-29T15:57:56.131 DEBUG:teuthology.orchestra.run.smithi197:> sudo lsb_release -is 2022-06-29T15:57:56.184 INFO:teuthology.orchestra.run.smithi197.stdout:Ubuntu 2022-06-29T15:57:56.189 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-06-29T15:57:56.189 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=f25ffac72e5a9e63860ff7deb6767a11f0559a16 2022-06-29T15:57:56.354 DEBUG:teuthology.orchestra.run.smithi197:> dpkg-query -W -f '${Version}' ceph 2022-06-29T15:57:56.377 INFO:teuthology.orchestra.run.smithi197.stdout:15.2.16-304-gf25ffac7-1focal 2022-06-29T15:57:56.377 INFO:teuthology.packaging:The installed version of ceph is 15.2.16-304-gf25ffac7-1focal 2022-06-29T15:57:56.378 INFO:teuthology.task.install:The correct ceph version 15.2.16-304-gf25ffac7-1focal is installed. 2022-06-29T15:57:56.378 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-06-29T15:57:56.379 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2022-06-29T15:57:56.379 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-06-29T15:57:56.396 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2022-06-29T15:57:56.397 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-06-29T15:57:56.415 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-06-29T15:57:56.415 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-06-29T15:57:56.433 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2022-06-29T15:57:56.433 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-06-29T15:57:56.451 DEBUG:teuthology.orchestra.run.smithi197:> set -ex 2022-06-29T15:57:56.451 DEBUG:teuthology.orchestra.run.smithi197:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-06-29T15:57:56.468 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-06-29T15:57:56.469 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2022-06-29T15:57:56.470 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/daemon-helper 2022-06-29T15:57:56.487 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-06-29T15:57:56.546 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2022-06-29T15:57:56.546 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/daemon-helper 2022-06-29T15:57:56.563 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-06-29T15:57:56.622 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-06-29T15:57:56.623 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/daemon-helper 2022-06-29T15:57:56.640 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-06-29T15:57:56.697 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2022-06-29T15:57:56.697 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/daemon-helper 2022-06-29T15:57:56.715 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-06-29T15:57:56.725 DEBUG:teuthology.orchestra.run.smithi197:> set -ex 2022-06-29T15:57:56.726 DEBUG:teuthology.orchestra.run.smithi197:> sudo dd of=/usr/bin/daemon-helper 2022-06-29T15:57:56.743 DEBUG:teuthology.orchestra.run.smithi197:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-06-29T15:57:56.753 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-06-29T15:57:56.753 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2022-06-29T15:57:56.754 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/adjust-ulimits 2022-06-29T15:57:56.771 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-06-29T15:57:56.829 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2022-06-29T15:57:56.830 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/adjust-ulimits 2022-06-29T15:57:56.847 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-06-29T15:57:56.904 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-06-29T15:57:56.905 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/adjust-ulimits 2022-06-29T15:57:56.922 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-06-29T15:57:56.933 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2022-06-29T15:57:56.933 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/adjust-ulimits 2022-06-29T15:57:56.951 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-06-29T15:57:57.008 DEBUG:teuthology.orchestra.run.smithi197:> set -ex 2022-06-29T15:57:57.008 DEBUG:teuthology.orchestra.run.smithi197:> sudo dd of=/usr/bin/adjust-ulimits 2022-06-29T15:57:57.025 DEBUG:teuthology.orchestra.run.smithi197:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-06-29T15:57:57.083 INFO:teuthology.run_tasks:Running task ceph... 2022-06-29T15:57:57.149 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2022-06-29T15:57:57.150 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 777 /var/log/ceph 2022-06-29T15:57:57.155 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 777 /var/log/ceph 2022-06-29T15:57:57.159 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 777 /var/log/ceph 2022-06-29T15:57:57.162 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 777 /var/log/ceph 2022-06-29T15:57:57.166 DEBUG:teuthology.orchestra.run.smithi197:> sudo chmod 777 /var/log/ceph 2022-06-29T15:57:57.183 INFO:tasks.ceph:Disabling ceph logrotate... 2022-06-29T15:57:57.184 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/logrotate.d/ceph 2022-06-29T15:57:57.215 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /etc/logrotate.d/ceph 2022-06-29T15:57:57.219 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/logrotate.d/ceph 2022-06-29T15:57:57.223 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /etc/logrotate.d/ceph 2022-06-29T15:57:57.226 DEBUG:teuthology.orchestra.run.smithi197:> sudo rm -f -- /etc/logrotate.d/ceph 2022-06-29T15:57:57.243 INFO:tasks.ceph:Creating extra log directories... 2022-06-29T15:57:57.244 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-06-29T15:57:57.248 DEBUG:teuthology.orchestra.run.smithi092:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-06-29T15:57:57.251 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-06-29T15:57:57.279 DEBUG:teuthology.orchestra.run.smithi156:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-06-29T15:57:57.281 DEBUG:teuthology.orchestra.run.smithi197:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-06-29T15:57:57.303 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2022-06-29T15:57:57.304 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2022-06-29T15:57:57.304 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2022-06-29T15:57:57.305 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2022-06-29T15:57:57.306 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2022-06-29T15:57:57.315 DEBUG:teuthology.orchestra.run.smithi060:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2022-06-29T15:57:57.355 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2022-06-29T15:57:57.356 DEBUG:teuthology.orchestra.run.smithi092:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2022-06-29T15:57:57.363 DEBUG:teuthology.orchestra.run.smithi092:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2022-06-29T15:57:57.402 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-06-29T15:57:57.403 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2022-06-29T15:57:57.411 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2022-06-29T15:57:57.450 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2022-06-29T15:57:57.450 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2022-06-29T15:57:57.458 DEBUG:teuthology.orchestra.run.smithi156:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2022-06-29T15:57:57.538 DEBUG:teuthology.orchestra.run.smithi197:> set -ex 2022-06-29T15:57:57.539 DEBUG:teuthology.orchestra.run.smithi197:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2022-06-29T15:57:57.547 DEBUG:teuthology.orchestra.run.smithi197:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2022-06-29T15:57:57.587 INFO:tasks.ceph:Creating ceph cluster ceph... 2022-06-29T15:57:57.587 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'filestore', 'osd op complaint time': 180, 'osd sloppy crc': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'evicting unresponsive client', 'POOL_APP_NOT_ENABLED', 'has not responded to cap revoke by MDS for over', 'MDS_CLIENT_LATE_RELEASE', 'responding to mclientcaps'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2022-06-29T15:57:57.588 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2022-06-29_13:31:31-fs-wip-yuri5-testing-2022-06-27-0923-octopus-distro-default-smithi/6905536', 'branch': 'wip-yuri5-testing-2022-06-27-0923-octopus', 'description': 'fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/filestore-xfs overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests}', 'email': None, 'first_in_suite': False, 'job_id': '6905536', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'yuriw-2022-06-29_13:31:31-fs-wip-yuri5-testing-2022-06-27-0923-octopus-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 30}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri5-testing-2022-06-27-0923-octopus'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'filestore', 'osd op complaint time': 180, 'osd sloppy crc': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'evicting unresponsive client', 'POOL_APP_NOT_ENABLED', 'has not responded to cap revoke by MDS for over', 'MDS_CLIENT_LATE_RELEASE', 'responding to mclientcaps'], 'sha1': 'f25ffac72e5a9e63860ff7deb6767a11f0559a16'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'osd objectstore': 'filestore', 'osd sloppy crc': True}}, 'filestore': True, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'f25ffac72e5a9e63860ff7deb6767a11f0559a16'}}, 'workunit': {'branch': 'wip-yuri5-testing-2022-06-27-0923-octopus', 'sha1': 'f25ffac72e5a9e63860ff7deb6767a11f0559a16'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 79, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'], ['client.0'], ['client.1'], ['client.2']], 'sha1': 'f25ffac72e5a9e63860ff7deb6767a11f0559a16', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'wip-yuri5-testing-2022-06-27-0923-octopus', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_f25ffac72e5a9e63860ff7deb6767a11f0559a16/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'f25ffac72e5a9e63860ff7deb6767a11f0559a16', 'targets': {'smithi060.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9bKV0cYlD5Ztq2Vrgcz9CTMAg5n3wz65YAQxp3+E5raqdTUpiioI+STpkSTjuKoCaNUJR8i4SSIWR7L3GbEXTCLhxPbzfRUE1FyjkrEdpE+q4QaTR2DAmjA8YuVW6HpwNYZ2xXMQ5feo5nYpppuDlWhvbkxlO/z2H4j+0Q8CecsPYZJurwTR+I58g5HXne3j7+j/ILdrLECVcpR66zu9barMyggbuzayVtkm2/9Cr8Uf8uiDQ8Yr/GYf4O6m6xujMTyz1RMD6iBt+HRY3ehTbO/8Ye2Tgc8aD9inQYmkyM6WknqZXiGiGrQwmU3Oaf+vNKlZPnxqSsVBs4FOnAB8HAa3CQtFys1ewnS/vz1WwmKObuUscA0KAIjKra5iKWHyapxNchNp2w1Ld/wlxs3jIQXtc4B4avlUnmbu/zgaXzjHO9DXnJLD7FVodQq7AzXRmwZ7/nyaaTQRm+jT0dQN1pBlGkwc1meMnz+sHov5tSQdqazcHxHGW8a+PRzU0G2U=', 'smithi092.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3Ibg4PZeSlmo48ptCaxvpuBOAw0+Af5PwJS6gDagEPp5oADGdQ+vmdcZ/1/s2fGUxbW2I+s8qXaiJ0H+G79+QfPiP5yp0rzYg8skRREeszYGq359WyVPtvyMr43r4hI2NA219Gm20VP+yq5xz7p0jjmmIpCJp8V4m+yn+b6b3KQrx7o1mvZBtWDzffYMqfsrk3hgsoNqmvfork2Mdjgoopbf/9ogZTQAyN/P2TXytXhynyEqcHmq2YtFK8/tLCk630TiBwMWFuYpIPnjfI92m9R5FMknh2RF5EUciaXpuy1XRLdBcUbh0v6ctUNIcqXGRfptvNgpXMMgllS1uH2AU11/OZJdN0gT1iIvyABEyIhlfhynroU/HGdOWYARx+BVGQn1+3mS0LEhDY0dO2i4PIfgU8gLAa04ZydV5jKAJ8hwdqdUkLEnEbYwItfKU2jZWvBHSN59OSfbyk8Y10RXYRSPo8YERSdwNTsnnXO10J9QghHtKrJrMTvcE60yHS28=', 'smithi121.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDI9oI+7ZKro2egyqE0uCd7A/BzWqCqe3EGnbfaOgeXmZIvLHOwA1KBUfekiZeJaw8Zz1H2dh3SYM+EVw2GcDNg6pwwPvzT0DGpdbIA1RilHWRLdl/fBlsvB8EvaMwWUPRyrdzGJQz7DvX2M6NXDO1+0eyWqFoUDZjTLHcfxeP5cdJ1SNzZmU7ipDpokBHzGG6lhaX/O1z8+vqNAbHvHpP7f5sm64zkzyxkPKIr+69BopGh9EodlAws8Gl1GAYYHbpat8QzqhU0G1Yp0nEYhaaVGSpzde1ugcYzd3lI+dHU5jo14DcG4BZQnGCoirss8TQmoE42/zMM4CBrDxesjWJDdsBnWQUGv0XpNA6+qJMEL4kfwUlcWDInrDO/o6ZPA3KHDuDIQyMbm4coUknqVC+qMZSFFeNiRdKQv9d6QdqrdYimUHcx55+Xav4mCaU9oHIWPJriCPODmTgNE/h/4mVK/qdt362K2fCW1FmOVmKUM75b/24Lxx8iwH5/qpVKKLc=', 'smithi156.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOu7iEksoPH1SegwSXhYdmM3MEuif4oXSs5xGa0ysSWEQeHSq09GGqlm411tAkA1N4eQaC6egKXAY0TEjqAEosEqL0ucIR2p7NqJxKR8fojaCmXjGSoQx9u3r1vFq+xUqTX2SGlcxc6mbMLd+IJv9OySpOv93hCGkRV+pifIUekV3HKu8QsC5mwKwrWYXaY5SciMXYNKazErxsL39YB2lmIE/yOOkPXVYqXi4f1sVswl2X+d+jRuwplRhNUhzcO2WTGMj4E0RBfTyAifrZxBEWNVvilXiyr2WsXLL4NR26cVU6aToSSWCFRfzogmKJEaqAd4d+ind2tD5azv9GxGs6uLy5W5NHkmm12XY/nb0yf7rxNpgqfHxKtiBykDjRuFZQ6BVY7BlDZR2yc6q/6A55AWJVFLAkIJC4J1wGIqgCc+UkcnQQIOMJlOfO3SvhuyuyVeM7+rfoesyq5D4wcxN9B/WdYNyhT+BiGZO6diljZ9XlWnL4EZOCNB+MpQ1pgEM=', 'smithi197.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCUG7FwrrcHhr8+opukEkrYkAmxYaY4/q5LZbQ0ovMPzIsy8yPjC636BiRSzKJQZhDE9ykoupIMQQGXGVmuyJHZ2vJKlXFig3uMO7Nc4csD2qHreEU0UYQERzPe2vDQw5Fe8h4U+VzEtw9UM+CBUfg1iVV9TNAs4v8/yYCTj7MVSEJuMzzgyRsfWuVQ1zIGjnThph+okCi9EPWeVQ/Mh8vzkWgtBvCEj2k/9o0ruvKS9eKZzyKCxe0G0O8sCezp/BQDDLUPxwf6tV+2xYqNJELNsMBPAmuCkhT5wxaRPAuJ+Y9LeP0KsmONfis6vHw2gXY523PeH3lUS5Lw9d5PLrrjM2eGsHDPhCiSBxR4sSkq5ip03u5vXw57BnlcrBJX9OaHkGl7RZFz1+gDZZ9jx372W7vzRkmeWIvUSIwTOMADk7VZUj8q++4lX8/IUyHtis9tks5+Drk/O/rJ2z/uXv5bs3lhQcnAQfI1B9nnNb7r7SRU/8Wv678ZPWYLVGK3HWk='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': 'f25ffac72e5a9e63860ff7deb6767a11f0559a16'}}, {'ceph': None}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_misc']}}], 'teuthology_branch': 'main', 'teuthology_sha1': 'f58709e3fda190343157063e5edb0d7a36a41afc', 'timestamp': '2022-06-29_13:31:31', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.1713'} 2022-06-29T15:57:57.589 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2022-06-29T15:57:57.592 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2022-06-29T15:57:57.595 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2022-06-29T15:57:57.598 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2022-06-29T15:57:57.600 DEBUG:teuthology.orchestra.run.smithi197:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2022-06-29T15:57:57.635 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -d -m0777 -- /var/run/ceph 2022-06-29T15:57:57.643 DEBUG:teuthology.orchestra.run.smithi092:> sudo install -d -m0777 -- /var/run/ceph 2022-06-29T15:57:57.645 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m0777 -- /var/run/ceph 2022-06-29T15:57:57.648 DEBUG:teuthology.orchestra.run.smithi156:> sudo install -d -m0777 -- /var/run/ceph 2022-06-29T15:57:57.650 DEBUG:teuthology.orchestra.run.smithi197:> sudo install -d -m0777 -- /var/run/ceph 2022-06-29T15:57:57.668 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2022-06-29T15:57:57.668 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/scratch_devs of=/dev/stdout 2022-06-29T15:57:57.675 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-06-29T15:57:57.675 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-06-29T15:57:57.683 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-06-29T15:57:57.683 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-06-29T15:57:57.684 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 6h/6d Inode: 374 Links: 1 2022-06-29T15:57:57.684 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-06-29T15:57:57.684 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2022-06-29 15:54:39.771998010 +0000 2022-06-29T15:57:57.684 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2022-06-29 15:54:39.767998010 +0000 2022-06-29T15:57:57.685 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2022-06-29 15:54:39.767998010 +0000 2022-06-29T15:57:57.685 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: - 2022-06-29T15:57:57.694 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2022-06-29T15:57:57.694 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2022-06-29T15:57:57.694 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000317783 s, 1.6 MB/s 2022-06-29T15:57:57.699 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-06-29T15:57:57.748 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-06-29T15:57:57.748 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-06-29T15:57:57.748 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 6h/6d Inode: 382 Links: 1 2022-06-29T15:57:57.749 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-06-29T15:57:57.749 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2022-06-29 15:54:40.631998001 +0000 2022-06-29T15:57:57.749 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2022-06-29 15:54:40.291998005 +0000 2022-06-29T15:57:57.749 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2022-06-29 15:54:40.291998005 +0000 2022-06-29T15:57:57.750 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: - 2022-06-29T15:57:57.758 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2022-06-29T15:57:57.758 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2022-06-29T15:57:57.759 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000281804 s, 1.8 MB/s 2022-06-29T15:57:57.762 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-06-29T15:57:57.769 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-06-29T15:57:57.769 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-06-29T15:57:57.770 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 6h/6d Inode: 385 Links: 1 2022-06-29T15:57:57.770 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-06-29T15:57:57.770 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2022-06-29 15:54:41.319997993 +0000 2022-06-29T15:57:57.771 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2022-06-29 15:54:40.839997999 +0000 2022-06-29T15:57:57.771 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2022-06-29 15:54:40.839997999 +0000 2022-06-29T15:57:57.771 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: - 2022-06-29T15:57:57.780 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2022-06-29T15:57:57.780 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2022-06-29T15:57:57.780 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000295266 s, 1.7 MB/s 2022-06-29T15:57:57.785 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-06-29T15:57:57.793 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-06-29T15:57:57.793 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-06-29T15:57:57.794 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 6h/6d Inode: 394 Links: 1 2022-06-29T15:57:57.794 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-06-29T15:57:57.794 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2022-06-29 15:54:41.847997988 +0000 2022-06-29T15:57:57.795 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2022-06-29 15:54:41.543997991 +0000 2022-06-29T15:57:57.795 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2022-06-29 15:54:41.543997991 +0000 2022-06-29T15:57:57.796 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: - 2022-06-29T15:57:57.802 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2022-06-29T15:57:57.802 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2022-06-29T15:57:57.802 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000289653 s, 1.8 MB/s 2022-06-29T15:57:57.807 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-06-29T15:57:57.807 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2022-06-29T15:57:57.807 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/scratch_devs of=/dev/stdout 2022-06-29T15:57:57.814 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-06-29T15:57:57.815 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-06-29T15:57:57.823 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-06-29T15:57:57.823 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-06-29T15:57:57.824 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 6h/6d Inode: 362 Links: 1 2022-06-29T15:57:57.824 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-06-29T15:57:57.824 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2022-06-29 15:54:42.875996085 +0000 2022-06-29T15:57:57.825 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2022-06-29 15:54:42.587996091 +0000 2022-06-29T15:57:57.825 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2022-06-29 15:54:42.587996091 +0000 2022-06-29T15:57:57.826 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: - 2022-06-29T15:57:57.833 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2022-06-29T15:57:57.834 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2022-06-29T15:57:57.834 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000432344 s, 1.2 MB/s 2022-06-29T15:57:57.838 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-06-29T15:57:57.846 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-06-29T15:57:57.846 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-06-29T15:57:57.846 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 6h/6d Inode: 367 Links: 1 2022-06-29T15:57:57.847 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-06-29T15:57:57.847 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2022-06-29 15:54:43.411996073 +0000 2022-06-29T15:57:57.847 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2022-06-29 15:54:43.099996080 +0000 2022-06-29T15:57:57.847 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2022-06-29 15:54:43.099996080 +0000 2022-06-29T15:57:57.848 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: - 2022-06-29T15:57:57.857 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2022-06-29T15:57:57.857 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2022-06-29T15:57:57.857 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000419221 s, 1.2 MB/s 2022-06-29T15:57:57.861 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-06-29T15:57:57.869 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-06-29T15:57:57.869 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-06-29T15:57:57.869 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 6h/6d Inode: 373 Links: 1 2022-06-29T15:57:57.869 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-06-29T15:57:57.870 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2022-06-29 15:54:43.619996068 +0000 2022-06-29T15:57:57.870 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2022-06-29 15:54:43.615996069 +0000 2022-06-29T15:57:57.870 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2022-06-29 15:54:43.615996069 +0000 2022-06-29T15:57:57.870 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: - 2022-06-29T15:57:57.875 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2022-06-29T15:57:57.875 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2022-06-29T15:57:57.875 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.00038114 s, 1.3 MB/s 2022-06-29T15:57:57.880 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-06-29T15:57:57.931 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-06-29T15:57:57.932 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-06-29T15:57:57.932 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 6h/6d Inode: 382 Links: 1 2022-06-29T15:57:57.932 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-06-29T15:57:57.933 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2022-06-29 15:54:44.175996056 +0000 2022-06-29T15:57:57.933 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2022-06-29 15:54:44.171996056 +0000 2022-06-29T15:57:57.933 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2022-06-29 15:54:44.171996056 +0000 2022-06-29T15:57:57.934 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: - 2022-06-29T15:57:57.942 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2022-06-29T15:57:57.943 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2022-06-29T15:57:57.943 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000415015 s, 1.2 MB/s 2022-06-29T15:57:57.947 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-06-29T15:57:57.948 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi060.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi092.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2022-06-29T15:57:57.949 INFO:tasks.ceph:Generating config... 2022-06-29T15:57:57.952 INFO:tasks.ceph:[client] client mount timeout = 600 2022-06-29T15:57:57.952 INFO:tasks.ceph:[client] debug client = 20 2022-06-29T15:57:57.953 INFO:tasks.ceph:[client] debug ms = 1 2022-06-29T15:57:57.954 INFO:tasks.ceph:[client] rados mon op timeout = 15m 2022-06-29T15:57:57.954 INFO:tasks.ceph:[client] rados osd op timeout = 15m 2022-06-29T15:57:57.955 INFO:tasks.ceph:[mds] debug mds = 20 2022-06-29T15:57:57.955 INFO:tasks.ceph:[mds] debug ms = 1 2022-06-29T15:57:57.956 INFO:tasks.ceph:[mds] mds bal frag = True 2022-06-29T15:57:57.956 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2022-06-29T15:57:57.956 INFO:tasks.ceph:[mds] mds bal merge size = 5 2022-06-29T15:57:57.956 INFO:tasks.ceph:[mds] mds bal split bits = 3 2022-06-29T15:57:57.957 INFO:tasks.ceph:[mds] mds bal split size = 100 2022-06-29T15:57:57.957 INFO:tasks.ceph:[mds] mds debug frag = True 2022-06-29T15:57:57.957 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2022-06-29T15:57:57.958 INFO:tasks.ceph:[mds] mds op complaint time = 180 2022-06-29T15:57:57.958 INFO:tasks.ceph:[mds] mds verify scatter = True 2022-06-29T15:57:57.958 INFO:tasks.ceph:[mds] osd op complaint time = 180 2022-06-29T15:57:57.959 INFO:tasks.ceph:[mds] rados mon op timeout = 15m 2022-06-29T15:57:57.959 INFO:tasks.ceph:[mds] rados osd op timeout = 15m 2022-06-29T15:57:57.959 INFO:tasks.ceph:[mgr] debug mgr = 20 2022-06-29T15:57:57.960 INFO:tasks.ceph:[mgr] debug ms = 1 2022-06-29T15:57:57.960 INFO:tasks.ceph:[mon] debug mon = 20 2022-06-29T15:57:57.960 INFO:tasks.ceph:[mon] debug ms = 1 2022-06-29T15:57:57.961 INFO:tasks.ceph:[mon] debug paxos = 20 2022-06-29T15:57:57.961 INFO:tasks.ceph:[mon] mon op complaint time = 120 2022-06-29T15:57:57.961 INFO:tasks.ceph:[osd] debug ms = 1 2022-06-29T15:57:57.961 INFO:tasks.ceph:[osd] debug osd = 20 2022-06-29T15:57:57.962 INFO:tasks.ceph:[osd] osd objectstore = filestore 2022-06-29T15:57:57.962 INFO:tasks.ceph:[osd] osd op complaint time = 180 2022-06-29T15:57:57.962 INFO:tasks.ceph:[osd] osd sloppy crc = True 2022-06-29T15:57:57.963 INFO:tasks.ceph:Setting up mon.a... 2022-06-29T15:57:57.963 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2022-06-29T15:57:58.011 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/ceph.keyring 2022-06-29T15:57:58.014 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2022-06-29T15:57:58.045 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-06-29T15:57:58.104 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.60'), ('mon.b', '172.21.15.92'), ('mon.c', '[v2:172.21.15.92:3301,v1:172.21.15.92:6790]')] 2022-06-29T15:57:58.105 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.60 --add b 172.21.15.92 --addv c '[v2:172.21.15.92:3301,v1:172.21.15.92:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2022-06-29T15:57:58.138 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2022-06-29T15:57:58.139 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: generated fsid e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85 2022-06-29T15:57:58.139 INFO:teuthology.orchestra.run.smithi060.stdout:epoch 0 2022-06-29T15:57:58.139 INFO:teuthology.orchestra.run.smithi060.stdout:fsid e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85 2022-06-29T15:57:58.139 INFO:teuthology.orchestra.run.smithi060.stdout:last_changed 2022-06-29T15:57:58.134926+0000 2022-06-29T15:57:58.140 INFO:teuthology.orchestra.run.smithi060.stdout:created 2022-06-29T15:57:58.134926+0000 2022-06-29T15:57:58.140 INFO:teuthology.orchestra.run.smithi060.stdout:min_mon_release 0 (unknown) 2022-06-29T15:57:58.140 INFO:teuthology.orchestra.run.smithi060.stdout:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2022-06-29T15:57:58.140 INFO:teuthology.orchestra.run.smithi060.stdout:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2022-06-29T15:57:58.141 INFO:teuthology.orchestra.run.smithi060.stdout:2: [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] mon.c 2022-06-29T15:57:58.141 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2022-06-29T15:57:58.142 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85... 2022-06-29T15:57:58.144 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-06-29T15:57:58.148 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-06-29T15:57:58.151 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-06-29T15:57:58.154 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-06-29T15:57:58.157 DEBUG:teuthology.orchestra.run.smithi197:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-06-29T15:57:58.180 INFO:teuthology.orchestra.run.smithi060.stdout:[global] 2022-06-29T15:57:58.180 INFO:teuthology.orchestra.run.smithi060.stdout: chdir = "" 2022-06-29T15:57:58.180 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-06-29T15:57:58.181 INFO:teuthology.orchestra.run.smithi060.stdout: auth supported = cephx 2022-06-29T15:57:58.181 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:57:58.181 INFO:teuthology.orchestra.run.smithi060.stdout: filestore xattr use omap = true 2022-06-29T15:57:58.181 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:57:58.182 INFO:teuthology.orchestra.run.smithi060.stdout: mon clock drift allowed = 1.000 2022-06-29T15:57:58.182 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:57:58.182 INFO:teuthology.orchestra.run.smithi060.stdout: osd crush chooseleaf type = 0 2022-06-29T15:57:58.182 INFO:teuthology.orchestra.run.smithi060.stdout: auth debug = true 2022-06-29T15:57:58.183 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:57:58.183 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on old message = true 2022-06-29T15:57:58.183 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on bug = true 2022-06-29T15:57:58.184 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:57:58.184 INFO:teuthology.orchestra.run.smithi060.stdout: mon max pg per osd = 10000 # >= luminous 2022-06-29T15:57:58.184 INFO:teuthology.orchestra.run.smithi060.stdout: mon pg warn max object skew = 0 2022-06-29T15:57:58.184 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:57:58.185 INFO:teuthology.orchestra.run.smithi060.stdout: # disable pg_autoscaler by default for new pools 2022-06-29T15:57:58.185 INFO:teuthology.orchestra.run.smithi060.stdout: osd_pool_default_pg_autoscale_mode = off 2022-06-29T15:57:58.185 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:57:58.185 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default size = 2 2022-06-29T15:57:58.185 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:57:58.186 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow primary affinity = true 2022-06-29T15:57:58.186 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow pg remap = true 2022-06-29T15:57:58.186 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on legacy crush tunables = false 2022-06-29T15:57:58.186 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on crush straw calc version zero = false 2022-06-29T15:57:58.187 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on no sortbitwise = false 2022-06-29T15:57:58.187 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on osd down out interval zero = false 2022-06-29T15:57:58.187 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on too few osds = false 2022-06-29T15:57:58.187 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-06-29T15:57:58.188 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_no_redundancy = false 2022-06-29T15:57:58.188 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:57:58.189 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-06-29T15:57:58.189 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:57:58.189 INFO:teuthology.orchestra.run.smithi060.stdout: osd default data pool replay window = 5 2022-06-29T15:57:58.189 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:57:58.190 INFO:teuthology.orchestra.run.smithi060.stdout: mon allow pool delete = true 2022-06-29T15:57:58.190 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:57:58.190 INFO:teuthology.orchestra.run.smithi060.stdout: mon cluster log file level = debug 2022-06-29T15:57:58.191 INFO:teuthology.orchestra.run.smithi060.stdout: debug asserts on shutdown = true 2022-06-29T15:57:58.191 INFO:teuthology.orchestra.run.smithi060.stdout: mon health detail to clog = false 2022-06-29T15:57:58.191 INFO:teuthology.orchestra.run.smithi060.stdout: mon host = "172.21.15.60,172.21.15.92,[v2:172.21.15.92:3301,v1:172.21.15.92:6790]" 2022-06-29T15:57:58.191 INFO:teuthology.orchestra.run.smithi060.stdout: fsid = e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85 2022-06-29T15:57:58.192 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:57:58.192 INFO:teuthology.orchestra.run.smithi060.stdout:[osd] 2022-06-29T15:57:58.192 INFO:teuthology.orchestra.run.smithi060.stdout: osd journal size = 100 2022-06-29T15:57:58.192 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:57:58.193 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub load threshold = 5.0 2022-06-29T15:57:58.193 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub max interval = 600 2022-06-29T15:57:58.193 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:57:58.193 INFO:teuthology.orchestra.run.smithi060.stdout: osd recover clone overlap = true 2022-06-29T15:57:58.193 INFO:teuthology.orchestra.run.smithi060.stdout: osd recovery max chunk = 1048576 2022-06-29T15:57:58.194 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:57:58.194 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug shutdown = true 2022-06-29T15:57:58.194 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug op order = true 2022-06-29T15:57:58.194 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug verify stray on activate = true 2022-06-29T15:57:58.195 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:57:58.195 INFO:teuthology.orchestra.run.smithi060.stdout: osd open classes on start = true 2022-06-29T15:57:58.195 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug pg log writeout = true 2022-06-29T15:57:58.195 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:57:58.196 INFO:teuthology.orchestra.run.smithi060.stdout: osd deep scrub update digest min age = 30 2022-06-29T15:57:58.196 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:57:58.196 INFO:teuthology.orchestra.run.smithi060.stdout: osd map max advance = 10 2022-06-29T15:57:58.196 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:57:58.196 INFO:teuthology.orchestra.run.smithi060.stdout: journal zero on create = true 2022-06-29T15:57:58.197 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:57:58.197 INFO:teuthology.orchestra.run.smithi060.stdout: filestore ondisk finisher threads = 3 2022-06-29T15:57:58.197 INFO:teuthology.orchestra.run.smithi060.stdout: filestore apply finisher threads = 3 2022-06-29T15:57:58.197 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:57:58.197 INFO:teuthology.orchestra.run.smithi060.stdout: bdev debug aio = true 2022-06-29T15:57:58.198 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug misdirected ops = true 2022-06-29T15:57:58.198 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2022-06-29T15:57:58.198 INFO:teuthology.orchestra.run.smithi060.stdout: debug osd = 20 2022-06-29T15:57:58.199 INFO:teuthology.orchestra.run.smithi060.stdout: osd objectstore = filestore 2022-06-29T15:57:58.199 INFO:teuthology.orchestra.run.smithi060.stdout: osd op complaint time = 180 2022-06-29T15:57:58.199 INFO:teuthology.orchestra.run.smithi060.stdout: osd sloppy crc = True 2022-06-29T15:57:58.199 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:57:58.199 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr] 2022-06-29T15:57:58.200 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2022-06-29T15:57:58.200 INFO:teuthology.orchestra.run.smithi060.stdout: debug mgr = 20 2022-06-29T15:57:58.200 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2022-06-29T15:57:58.200 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2022-06-29T15:57:58.201 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2022-06-29T15:57:58.201 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2022-06-29T15:57:58.201 INFO:teuthology.orchestra.run.smithi060.stdout: mgr/telemetry/nag = false 2022-06-29T15:57:58.201 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:57:58.201 INFO:teuthology.orchestra.run.smithi060.stdout:[mon] 2022-06-29T15:57:58.202 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2022-06-29T15:57:58.202 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2022-06-29T15:57:58.202 INFO:teuthology.orchestra.run.smithi060.stdout: debug paxos = 20 2022-06-29T15:57:58.202 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2022-06-29T15:57:58.203 INFO:teuthology.orchestra.run.smithi060.stdout: mon data avail warn = 5 2022-06-29T15:57:58.203 INFO:teuthology.orchestra.run.smithi060.stdout: mon mgr mkfs grace = 240 2022-06-29T15:57:58.203 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2022-06-29T15:57:58.203 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd reporter subtree level = osd 2022-06-29T15:57:58.204 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd prime pg temp = true 2022-06-29T15:57:58.204 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2022-06-29T15:57:58.204 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:57:58.204 INFO:teuthology.orchestra.run.smithi060.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-06-29T15:57:58.205 INFO:teuthology.orchestra.run.smithi060.stdout: auth mon ticket ttl = 660 # 11m 2022-06-29T15:57:58.205 INFO:teuthology.orchestra.run.smithi060.stdout: auth service ticket ttl = 240 # 4m 2022-06-29T15:57:58.205 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:57:58.205 INFO:teuthology.orchestra.run.smithi060.stdout: # don't complain about insecure global_id in the test suite 2022-06-29T15:57:58.206 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-06-29T15:57:58.206 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-06-29T15:57:58.206 INFO:teuthology.orchestra.run.smithi060.stdout: mon op complaint time = 120 2022-06-29T15:57:58.206 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:57:58.207 INFO:teuthology.orchestra.run.smithi060.stdout:[client] 2022-06-29T15:57:58.207 INFO:teuthology.orchestra.run.smithi060.stdout: rgw cache enabled = true 2022-06-29T15:57:58.207 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable ops log = true 2022-06-29T15:57:58.207 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable usage log = true 2022-06-29T15:57:58.208 INFO:teuthology.orchestra.run.smithi060.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-06-29T15:57:58.208 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-06-29T15:57:58.208 INFO:teuthology.orchestra.run.smithi060.stdout: client mount timeout = 600 2022-06-29T15:57:58.208 INFO:teuthology.orchestra.run.smithi060.stdout: debug client = 20 2022-06-29T15:57:58.208 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2022-06-29T15:57:58.209 INFO:teuthology.orchestra.run.smithi060.stdout: rados mon op timeout = 15m 2022-06-29T15:57:58.209 INFO:teuthology.orchestra.run.smithi060.stdout: rados osd op timeout = 15m 2022-06-29T15:57:58.209 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.a] 2022-06-29T15:57:58.209 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.b] 2022-06-29T15:57:58.210 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.c] 2022-06-29T15:57:58.210 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.a] 2022-06-29T15:57:58.210 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.b] 2022-06-29T15:57:58.210 INFO:teuthology.orchestra.run.smithi060.stdout:[mds] 2022-06-29T15:57:58.211 INFO:teuthology.orchestra.run.smithi060.stdout: debug mds = 20 2022-06-29T15:57:58.211 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2022-06-29T15:57:58.211 INFO:teuthology.orchestra.run.smithi060.stdout: mds bal frag = True 2022-06-29T15:57:58.211 INFO:teuthology.orchestra.run.smithi060.stdout: mds bal fragment size max = 10000 2022-06-29T15:57:58.212 INFO:teuthology.orchestra.run.smithi060.stdout: mds bal merge size = 5 2022-06-29T15:57:58.212 INFO:teuthology.orchestra.run.smithi060.stdout: mds bal split bits = 3 2022-06-29T15:57:58.212 INFO:teuthology.orchestra.run.smithi060.stdout: mds bal split size = 100 2022-06-29T15:57:58.212 INFO:teuthology.orchestra.run.smithi060.stdout: mds debug frag = True 2022-06-29T15:57:58.213 INFO:teuthology.orchestra.run.smithi060.stdout: mds debug scatterstat = True 2022-06-29T15:57:58.213 INFO:teuthology.orchestra.run.smithi060.stdout: mds op complaint time = 180 2022-06-29T15:57:58.213 INFO:teuthology.orchestra.run.smithi060.stdout: mds verify scatter = True 2022-06-29T15:57:58.213 INFO:teuthology.orchestra.run.smithi060.stdout: osd op complaint time = 180 2022-06-29T15:57:58.213 INFO:teuthology.orchestra.run.smithi060.stdout: rados mon op timeout = 15m 2022-06-29T15:57:58.214 INFO:teuthology.orchestra.run.smithi060.stdout: rados osd op timeout = 15m 2022-06-29T15:57:58.219 INFO:teuthology.orchestra.run.smithi197.stdout:[global] 2022-06-29T15:57:58.219 INFO:teuthology.orchestra.run.smithi197.stdout: chdir = "" 2022-06-29T15:57:58.220 INFO:teuthology.orchestra.run.smithi197.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-06-29T15:57:58.220 INFO:teuthology.orchestra.run.smithi197.stdout: auth supported = cephx 2022-06-29T15:57:58.221 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T15:57:58.221 INFO:teuthology.orchestra.run.smithi197.stdout: filestore xattr use omap = true 2022-06-29T15:57:58.222 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T15:57:58.222 INFO:teuthology.orchestra.run.smithi197.stdout: mon clock drift allowed = 1.000 2022-06-29T15:57:58.222 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T15:57:58.223 INFO:teuthology.orchestra.run.smithi197.stdout: osd crush chooseleaf type = 0 2022-06-29T15:57:58.223 INFO:teuthology.orchestra.run.smithi197.stdout: auth debug = true 2022-06-29T15:57:58.223 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T15:57:58.224 INFO:teuthology.orchestra.run.smithi197.stdout: ms die on old message = true 2022-06-29T15:57:58.224 INFO:teuthology.orchestra.run.smithi197.stdout: ms die on bug = true 2022-06-29T15:57:58.224 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T15:57:58.224 INFO:teuthology.orchestra.run.smithi197.stdout: mon max pg per osd = 10000 # >= luminous 2022-06-29T15:57:58.225 INFO:teuthology.orchestra.run.smithi197.stdout: mon pg warn max object skew = 0 2022-06-29T15:57:58.225 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T15:57:58.225 INFO:teuthology.orchestra.run.smithi197.stdout: # disable pg_autoscaler by default for new pools 2022-06-29T15:57:58.225 INFO:teuthology.orchestra.run.smithi197.stdout: osd_pool_default_pg_autoscale_mode = off 2022-06-29T15:57:58.226 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T15:57:58.226 INFO:teuthology.orchestra.run.smithi197.stdout: osd pool default size = 2 2022-06-29T15:57:58.226 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T15:57:58.226 INFO:teuthology.orchestra.run.smithi197.stdout: mon osd allow primary affinity = true 2022-06-29T15:57:58.226 INFO:teuthology.orchestra.run.smithi197.stdout: mon osd allow pg remap = true 2022-06-29T15:57:58.227 INFO:teuthology.orchestra.run.smithi197.stdout: mon warn on legacy crush tunables = false 2022-06-29T15:57:58.227 INFO:teuthology.orchestra.run.smithi197.stdout: mon warn on crush straw calc version zero = false 2022-06-29T15:57:58.227 INFO:teuthology.orchestra.run.smithi197.stdout: mon warn on no sortbitwise = false 2022-06-29T15:57:58.227 INFO:teuthology.orchestra.run.smithi197.stdout: mon warn on osd down out interval zero = false 2022-06-29T15:57:58.228 INFO:teuthology.orchestra.run.smithi197.stdout: mon warn on too few osds = false 2022-06-29T15:57:58.228 INFO:teuthology.orchestra.run.smithi197.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-06-29T15:57:58.228 INFO:teuthology.orchestra.run.smithi197.stdout: mon_warn_on_pool_no_redundancy = false 2022-06-29T15:57:58.228 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T15:57:58.229 INFO:teuthology.orchestra.run.smithi197.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-06-29T15:57:58.229 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T15:57:58.229 INFO:teuthology.orchestra.run.smithi197.stdout: osd default data pool replay window = 5 2022-06-29T15:57:58.229 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T15:57:58.229 INFO:teuthology.orchestra.run.smithi197.stdout: mon allow pool delete = true 2022-06-29T15:57:58.230 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T15:57:58.230 INFO:teuthology.orchestra.run.smithi197.stdout: mon cluster log file level = debug 2022-06-29T15:57:58.230 INFO:teuthology.orchestra.run.smithi197.stdout: debug asserts on shutdown = true 2022-06-29T15:57:58.230 INFO:teuthology.orchestra.run.smithi197.stdout: mon health detail to clog = false 2022-06-29T15:57:58.231 INFO:teuthology.orchestra.run.smithi197.stdout: mon host = "172.21.15.60,172.21.15.92,[v2:172.21.15.92:3301,v1:172.21.15.92:6790]" 2022-06-29T15:57:58.231 INFO:teuthology.orchestra.run.smithi197.stdout: fsid = e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85 2022-06-29T15:57:58.231 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T15:57:58.231 INFO:teuthology.orchestra.run.smithi197.stdout:[osd] 2022-06-29T15:57:58.231 INFO:teuthology.orchestra.run.smithi197.stdout: osd journal size = 100 2022-06-29T15:57:58.232 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T15:57:58.232 INFO:teuthology.orchestra.run.smithi197.stdout: osd scrub load threshold = 5.0 2022-06-29T15:57:58.232 INFO:teuthology.orchestra.run.smithi197.stdout: osd scrub max interval = 600 2022-06-29T15:57:58.232 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T15:57:58.233 INFO:teuthology.orchestra.run.smithi197.stdout: osd recover clone overlap = true 2022-06-29T15:57:58.233 INFO:teuthology.orchestra.run.smithi197.stdout: osd recovery max chunk = 1048576 2022-06-29T15:57:58.233 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T15:57:58.233 INFO:teuthology.orchestra.run.smithi197.stdout: osd debug shutdown = true 2022-06-29T15:57:58.234 INFO:teuthology.orchestra.run.smithi197.stdout: osd debug op order = true 2022-06-29T15:57:58.234 INFO:teuthology.orchestra.run.smithi197.stdout: osd debug verify stray on activate = true 2022-06-29T15:57:58.234 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T15:57:58.234 INFO:teuthology.orchestra.run.smithi197.stdout: osd open classes on start = true 2022-06-29T15:57:58.235 INFO:teuthology.orchestra.run.smithi197.stdout: osd debug pg log writeout = true 2022-06-29T15:57:58.235 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T15:57:58.235 INFO:teuthology.orchestra.run.smithi197.stdout: osd deep scrub update digest min age = 30 2022-06-29T15:57:58.236 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T15:57:58.236 INFO:teuthology.orchestra.run.smithi197.stdout: osd map max advance = 10 2022-06-29T15:57:58.236 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T15:57:58.237 INFO:teuthology.orchestra.run.smithi197.stdout: journal zero on create = true 2022-06-29T15:57:58.237 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T15:57:58.237 INFO:teuthology.orchestra.run.smithi197.stdout: filestore ondisk finisher threads = 3 2022-06-29T15:57:58.237 INFO:teuthology.orchestra.run.smithi197.stdout: filestore apply finisher threads = 3 2022-06-29T15:57:58.238 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T15:57:58.238 INFO:teuthology.orchestra.run.smithi197.stdout: bdev debug aio = true 2022-06-29T15:57:58.238 INFO:teuthology.orchestra.run.smithi197.stdout: osd debug misdirected ops = true 2022-06-29T15:57:58.238 INFO:teuthology.orchestra.run.smithi197.stdout: debug ms = 1 2022-06-29T15:57:58.238 INFO:teuthology.orchestra.run.smithi197.stdout: debug osd = 20 2022-06-29T15:57:58.239 INFO:teuthology.orchestra.run.smithi197.stdout: osd objectstore = filestore 2022-06-29T15:57:58.239 INFO:teuthology.orchestra.run.smithi197.stdout: osd op complaint time = 180 2022-06-29T15:57:58.239 INFO:teuthology.orchestra.run.smithi197.stdout: osd sloppy crc = True 2022-06-29T15:57:58.240 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T15:57:58.240 INFO:teuthology.orchestra.run.smithi197.stdout:[mgr] 2022-06-29T15:57:58.240 INFO:teuthology.orchestra.run.smithi197.stdout: debug ms = 1 2022-06-29T15:57:58.240 INFO:teuthology.orchestra.run.smithi197.stdout: debug mgr = 20 2022-06-29T15:57:58.240 INFO:teuthology.orchestra.run.smithi197.stdout: debug mon = 20 2022-06-29T15:57:58.241 INFO:teuthology.orchestra.run.smithi197.stdout: debug auth = 20 2022-06-29T15:57:58.241 INFO:teuthology.orchestra.run.smithi197.stdout: mon reweight min pgs per osd = 4 2022-06-29T15:57:58.241 INFO:teuthology.orchestra.run.smithi197.stdout: mon reweight min bytes per osd = 10 2022-06-29T15:57:58.241 INFO:teuthology.orchestra.run.smithi197.stdout: mgr/telemetry/nag = false 2022-06-29T15:57:58.242 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T15:57:58.242 INFO:teuthology.orchestra.run.smithi197.stdout:[mon] 2022-06-29T15:57:58.242 INFO:teuthology.orchestra.run.smithi197.stdout: debug ms = 1 2022-06-29T15:57:58.242 INFO:teuthology.orchestra.run.smithi197.stdout: debug mon = 20 2022-06-29T15:57:58.243 INFO:teuthology.orchestra.run.smithi197.stdout: debug paxos = 20 2022-06-29T15:57:58.243 INFO:teuthology.orchestra.run.smithi197.stdout: debug auth = 20 2022-06-29T15:57:58.243 INFO:teuthology.orchestra.run.smithi197.stdout: mon data avail warn = 5 2022-06-29T15:57:58.243 INFO:teuthology.orchestra.run.smithi197.stdout: mon mgr mkfs grace = 240 2022-06-29T15:57:58.244 INFO:teuthology.orchestra.run.smithi197.stdout: mon reweight min pgs per osd = 4 2022-06-29T15:57:58.244 INFO:teuthology.orchestra.run.smithi197.stdout: mon osd reporter subtree level = osd 2022-06-29T15:57:58.244 INFO:teuthology.orchestra.run.smithi197.stdout: mon osd prime pg temp = true 2022-06-29T15:57:58.244 INFO:teuthology.orchestra.run.smithi197.stdout: mon reweight min bytes per osd = 10 2022-06-29T15:57:58.245 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T15:57:58.245 INFO:teuthology.orchestra.run.smithi197.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-06-29T15:57:58.245 INFO:teuthology.orchestra.run.smithi197.stdout: auth mon ticket ttl = 660 # 11m 2022-06-29T15:57:58.245 INFO:teuthology.orchestra.run.smithi197.stdout: auth service ticket ttl = 240 # 4m 2022-06-29T15:57:58.245 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T15:57:58.246 INFO:teuthology.orchestra.run.smithi197.stdout: # don't complain about insecure global_id in the test suite 2022-06-29T15:57:58.246 INFO:teuthology.orchestra.run.smithi197.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-06-29T15:57:58.246 INFO:teuthology.orchestra.run.smithi197.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-06-29T15:57:58.246 INFO:teuthology.orchestra.run.smithi197.stdout: mon op complaint time = 120 2022-06-29T15:57:58.247 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T15:57:58.247 INFO:teuthology.orchestra.run.smithi197.stdout:[client] 2022-06-29T15:57:58.247 INFO:teuthology.orchestra.run.smithi197.stdout: rgw cache enabled = true 2022-06-29T15:57:58.247 INFO:teuthology.orchestra.run.smithi197.stdout: rgw enable ops log = true 2022-06-29T15:57:58.248 INFO:teuthology.orchestra.run.smithi197.stdout: rgw enable usage log = true 2022-06-29T15:57:58.248 INFO:teuthology.orchestra.run.smithi197.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-06-29T15:57:58.248 INFO:teuthology.orchestra.run.smithi197.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-06-29T15:57:58.248 INFO:teuthology.orchestra.run.smithi197.stdout: client mount timeout = 600 2022-06-29T15:57:58.248 INFO:teuthology.orchestra.run.smithi197.stdout: debug client = 20 2022-06-29T15:57:58.249 INFO:teuthology.orchestra.run.smithi197.stdout: debug ms = 1 2022-06-29T15:57:58.249 INFO:teuthology.orchestra.run.smithi197.stdout: rados mon op timeout = 15m 2022-06-29T15:57:58.249 INFO:teuthology.orchestra.run.smithi197.stdout: rados osd op timeout = 15m 2022-06-29T15:57:58.249 INFO:teuthology.orchestra.run.smithi197.stdout:[mon.a] 2022-06-29T15:57:58.250 INFO:teuthology.orchestra.run.smithi197.stdout:[mon.b] 2022-06-29T15:57:58.250 INFO:teuthology.orchestra.run.smithi197.stdout:[mon.c] 2022-06-29T15:57:58.250 INFO:teuthology.orchestra.run.smithi197.stdout:[mds.a] 2022-06-29T15:57:58.250 INFO:teuthology.orchestra.run.smithi197.stdout:[mds.b] 2022-06-29T15:57:58.250 INFO:teuthology.orchestra.run.smithi197.stdout:[mds] 2022-06-29T15:57:58.251 INFO:teuthology.orchestra.run.smithi197.stdout: debug mds = 20 2022-06-29T15:57:58.251 INFO:teuthology.orchestra.run.smithi197.stdout: debug ms = 1 2022-06-29T15:57:58.251 INFO:teuthology.orchestra.run.smithi197.stdout: mds bal frag = True 2022-06-29T15:57:58.251 INFO:teuthology.orchestra.run.smithi197.stdout: mds bal fragment size max = 10000 2022-06-29T15:57:58.252 INFO:teuthology.orchestra.run.smithi197.stdout: mds bal merge size = 5 2022-06-29T15:57:58.252 INFO:teuthology.orchestra.run.smithi197.stdout: mds bal split bits = 3 2022-06-29T15:57:58.252 INFO:teuthology.orchestra.run.smithi197.stdout: mds bal split size = 100 2022-06-29T15:57:58.252 INFO:teuthology.orchestra.run.smithi197.stdout: mds debug frag = True 2022-06-29T15:57:58.252 INFO:teuthology.orchestra.run.smithi197.stdout: mds debug scatterstat = True 2022-06-29T15:57:58.253 INFO:teuthology.orchestra.run.smithi197.stdout: mds op complaint time = 180 2022-06-29T15:57:58.253 INFO:teuthology.orchestra.run.smithi197.stdout: mds verify scatter = True 2022-06-29T15:57:58.253 INFO:teuthology.orchestra.run.smithi197.stdout: osd op complaint time = 180 2022-06-29T15:57:58.253 INFO:teuthology.orchestra.run.smithi197.stdout: rados mon op timeout = 15m 2022-06-29T15:57:58.254 INFO:teuthology.orchestra.run.smithi197.stdout: rados osd op timeout = 15m 2022-06-29T15:57:58.254 INFO:teuthology.orchestra.run.smithi121.stdout:[global] 2022-06-29T15:57:58.254 INFO:teuthology.orchestra.run.smithi121.stdout: chdir = "" 2022-06-29T15:57:58.255 INFO:teuthology.orchestra.run.smithi121.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-06-29T15:57:58.255 INFO:teuthology.orchestra.run.smithi121.stdout: auth supported = cephx 2022-06-29T15:57:58.255 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T15:57:58.256 INFO:teuthology.orchestra.run.smithi121.stdout: filestore xattr use omap = true 2022-06-29T15:57:58.256 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T15:57:58.257 INFO:teuthology.orchestra.run.smithi121.stdout: mon clock drift allowed = 1.000 2022-06-29T15:57:58.257 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T15:57:58.258 INFO:teuthology.orchestra.run.smithi121.stdout: osd crush chooseleaf type = 0 2022-06-29T15:57:58.258 INFO:teuthology.orchestra.run.smithi121.stdout: auth debug = true 2022-06-29T15:57:58.258 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T15:57:58.259 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on old message = true 2022-06-29T15:57:58.259 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on bug = true 2022-06-29T15:57:58.260 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T15:57:58.260 INFO:teuthology.orchestra.run.smithi121.stdout: mon max pg per osd = 10000 # >= luminous 2022-06-29T15:57:58.261 INFO:teuthology.orchestra.run.smithi121.stdout: mon pg warn max object skew = 0 2022-06-29T15:57:58.261 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T15:57:58.262 INFO:teuthology.orchestra.run.smithi121.stdout: # disable pg_autoscaler by default for new pools 2022-06-29T15:57:58.262 INFO:teuthology.orchestra.run.smithi121.stdout: osd_pool_default_pg_autoscale_mode = off 2022-06-29T15:57:58.263 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T15:57:58.263 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default size = 2 2022-06-29T15:57:58.263 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T15:57:58.264 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow primary affinity = true 2022-06-29T15:57:58.264 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow pg remap = true 2022-06-29T15:57:58.264 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on legacy crush tunables = false 2022-06-29T15:57:58.265 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on crush straw calc version zero = false 2022-06-29T15:57:58.265 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on no sortbitwise = false 2022-06-29T15:57:58.266 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on osd down out interval zero = false 2022-06-29T15:57:58.266 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on too few osds = false 2022-06-29T15:57:58.266 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-06-29T15:57:58.267 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_no_redundancy = false 2022-06-29T15:57:58.267 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T15:57:58.267 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-06-29T15:57:58.267 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T15:57:58.268 INFO:teuthology.orchestra.run.smithi121.stdout: osd default data pool replay window = 5 2022-06-29T15:57:58.268 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T15:57:58.268 INFO:teuthology.orchestra.run.smithi121.stdout: mon allow pool delete = true 2022-06-29T15:57:58.268 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T15:57:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: mon cluster log file level = debug 2022-06-29T15:57:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: debug asserts on shutdown = true 2022-06-29T15:57:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: mon health detail to clog = false 2022-06-29T15:57:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: mon host = "172.21.15.60,172.21.15.92,[v2:172.21.15.92:3301,v1:172.21.15.92:6790]" 2022-06-29T15:57:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: fsid = e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85 2022-06-29T15:57:58.270 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T15:57:58.270 INFO:teuthology.orchestra.run.smithi121.stdout:[osd] 2022-06-29T15:57:58.271 INFO:teuthology.orchestra.run.smithi121.stdout: osd journal size = 100 2022-06-29T15:57:58.271 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T15:57:58.272 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub load threshold = 5.0 2022-06-29T15:57:58.272 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub max interval = 600 2022-06-29T15:57:58.272 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T15:57:58.273 INFO:teuthology.orchestra.run.smithi121.stdout: osd recover clone overlap = true 2022-06-29T15:57:58.273 INFO:teuthology.orchestra.run.smithi121.stdout: osd recovery max chunk = 1048576 2022-06-29T15:57:58.274 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T15:57:58.274 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug shutdown = true 2022-06-29T15:57:58.274 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug op order = true 2022-06-29T15:57:58.275 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify stray on activate = true 2022-06-29T15:57:58.275 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T15:57:58.276 INFO:teuthology.orchestra.run.smithi121.stdout: osd open classes on start = true 2022-06-29T15:57:58.276 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug pg log writeout = true 2022-06-29T15:57:58.277 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T15:57:58.277 INFO:teuthology.orchestra.run.smithi121.stdout: osd deep scrub update digest min age = 30 2022-06-29T15:57:58.277 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T15:57:58.277 INFO:teuthology.orchestra.run.smithi121.stdout: osd map max advance = 10 2022-06-29T15:57:58.277 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T15:57:58.278 INFO:teuthology.orchestra.run.smithi121.stdout: journal zero on create = true 2022-06-29T15:57:58.278 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T15:57:58.278 INFO:teuthology.orchestra.run.smithi121.stdout: filestore ondisk finisher threads = 3 2022-06-29T15:57:58.278 INFO:teuthology.orchestra.run.smithi121.stdout: filestore apply finisher threads = 3 2022-06-29T15:57:58.279 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T15:57:58.279 INFO:teuthology.orchestra.run.smithi121.stdout: bdev debug aio = true 2022-06-29T15:57:58.279 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug misdirected ops = true 2022-06-29T15:57:58.279 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2022-06-29T15:57:58.280 INFO:teuthology.orchestra.run.smithi121.stdout: debug osd = 20 2022-06-29T15:57:58.280 INFO:teuthology.orchestra.run.smithi121.stdout: osd objectstore = filestore 2022-06-29T15:57:58.280 INFO:teuthology.orchestra.run.smithi121.stdout: osd op complaint time = 180 2022-06-29T15:57:58.280 INFO:teuthology.orchestra.run.smithi121.stdout: osd sloppy crc = True 2022-06-29T15:57:58.281 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T15:57:58.281 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr] 2022-06-29T15:57:58.281 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2022-06-29T15:57:58.281 INFO:teuthology.orchestra.run.smithi121.stdout: debug mgr = 20 2022-06-29T15:57:58.282 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2022-06-29T15:57:58.282 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2022-06-29T15:57:58.282 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2022-06-29T15:57:58.283 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2022-06-29T15:57:58.283 INFO:teuthology.orchestra.run.smithi121.stdout: mgr/telemetry/nag = false 2022-06-29T15:57:58.283 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T15:57:58.283 INFO:teuthology.orchestra.run.smithi121.stdout:[mon] 2022-06-29T15:57:58.283 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2022-06-29T15:57:58.284 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2022-06-29T15:57:58.284 INFO:teuthology.orchestra.run.smithi121.stdout: debug paxos = 20 2022-06-29T15:57:58.284 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2022-06-29T15:57:58.284 INFO:teuthology.orchestra.run.smithi121.stdout: mon data avail warn = 5 2022-06-29T15:57:58.285 INFO:teuthology.orchestra.run.smithi121.stdout: mon mgr mkfs grace = 240 2022-06-29T15:57:58.285 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2022-06-29T15:57:58.285 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd reporter subtree level = osd 2022-06-29T15:57:58.285 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd prime pg temp = true 2022-06-29T15:57:58.285 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2022-06-29T15:57:58.286 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T15:57:58.286 INFO:teuthology.orchestra.run.smithi121.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-06-29T15:57:58.286 INFO:teuthology.orchestra.run.smithi121.stdout: auth mon ticket ttl = 660 # 11m 2022-06-29T15:57:58.286 INFO:teuthology.orchestra.run.smithi121.stdout: auth service ticket ttl = 240 # 4m 2022-06-29T15:57:58.287 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T15:57:58.287 INFO:teuthology.orchestra.run.smithi121.stdout: # don't complain about insecure global_id in the test suite 2022-06-29T15:57:58.287 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-06-29T15:57:58.287 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-06-29T15:57:58.288 INFO:teuthology.orchestra.run.smithi121.stdout: mon op complaint time = 120 2022-06-29T15:57:58.288 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T15:57:58.288 INFO:teuthology.orchestra.run.smithi121.stdout:[client] 2022-06-29T15:57:58.288 INFO:teuthology.orchestra.run.smithi121.stdout: rgw cache enabled = true 2022-06-29T15:57:58.288 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable ops log = true 2022-06-29T15:57:58.289 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable usage log = true 2022-06-29T15:57:58.289 INFO:teuthology.orchestra.run.smithi121.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-06-29T15:57:58.289 INFO:teuthology.orchestra.run.smithi121.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-06-29T15:57:58.289 INFO:teuthology.orchestra.run.smithi121.stdout: client mount timeout = 600 2022-06-29T15:57:58.290 INFO:teuthology.orchestra.run.smithi121.stdout: debug client = 20 2022-06-29T15:57:58.290 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2022-06-29T15:57:58.290 INFO:teuthology.orchestra.run.smithi121.stdout: rados mon op timeout = 15m 2022-06-29T15:57:58.290 INFO:teuthology.orchestra.run.smithi121.stdout: rados osd op timeout = 15m 2022-06-29T15:57:58.290 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.a] 2022-06-29T15:57:58.291 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.b] 2022-06-29T15:57:58.291 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.c] 2022-06-29T15:57:58.291 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.a] 2022-06-29T15:57:58.291 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.b] 2022-06-29T15:57:58.292 INFO:teuthology.orchestra.run.smithi121.stdout:[mds] 2022-06-29T15:57:58.292 INFO:teuthology.orchestra.run.smithi121.stdout: debug mds = 20 2022-06-29T15:57:58.292 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2022-06-29T15:57:58.292 INFO:teuthology.orchestra.run.smithi121.stdout: mds bal frag = True 2022-06-29T15:57:58.293 INFO:teuthology.orchestra.run.smithi121.stdout: mds bal fragment size max = 10000 2022-06-29T15:57:58.293 INFO:teuthology.orchestra.run.smithi121.stdout: mds bal merge size = 5 2022-06-29T15:57:58.293 INFO:teuthology.orchestra.run.smithi121.stdout: mds bal split bits = 3 2022-06-29T15:57:58.293 INFO:teuthology.orchestra.run.smithi121.stdout: mds bal split size = 100 2022-06-29T15:57:58.293 INFO:teuthology.orchestra.run.smithi121.stdout: mds debug frag = True 2022-06-29T15:57:58.294 INFO:teuthology.orchestra.run.smithi121.stdout: mds debug scatterstat = True 2022-06-29T15:57:58.294 INFO:teuthology.orchestra.run.smithi121.stdout: mds op complaint time = 180 2022-06-29T15:57:58.294 INFO:teuthology.orchestra.run.smithi121.stdout: mds verify scatter = True 2022-06-29T15:57:58.294 INFO:teuthology.orchestra.run.smithi121.stdout: osd op complaint time = 180 2022-06-29T15:57:58.295 INFO:teuthology.orchestra.run.smithi121.stdout: rados mon op timeout = 15m 2022-06-29T15:57:58.295 INFO:teuthology.orchestra.run.smithi121.stdout: rados osd op timeout = 15m 2022-06-29T15:57:58.296 INFO:teuthology.orchestra.run.smithi156.stdout:[global] 2022-06-29T15:57:58.296 INFO:teuthology.orchestra.run.smithi156.stdout: chdir = "" 2022-06-29T15:57:58.296 INFO:teuthology.orchestra.run.smithi156.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-06-29T15:57:58.296 INFO:teuthology.orchestra.run.smithi156.stdout: auth supported = cephx 2022-06-29T15:57:58.297 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T15:57:58.297 INFO:teuthology.orchestra.run.smithi156.stdout: filestore xattr use omap = true 2022-06-29T15:57:58.297 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T15:57:58.297 INFO:teuthology.orchestra.run.smithi156.stdout: mon clock drift allowed = 1.000 2022-06-29T15:57:58.298 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T15:57:58.298 INFO:teuthology.orchestra.run.smithi156.stdout: osd crush chooseleaf type = 0 2022-06-29T15:57:58.298 INFO:teuthology.orchestra.run.smithi156.stdout: auth debug = true 2022-06-29T15:57:58.298 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T15:57:58.299 INFO:teuthology.orchestra.run.smithi156.stdout: ms die on old message = true 2022-06-29T15:57:58.299 INFO:teuthology.orchestra.run.smithi156.stdout: ms die on bug = true 2022-06-29T15:57:58.300 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T15:57:58.300 INFO:teuthology.orchestra.run.smithi156.stdout: mon max pg per osd = 10000 # >= luminous 2022-06-29T15:57:58.300 INFO:teuthology.orchestra.run.smithi156.stdout: mon pg warn max object skew = 0 2022-06-29T15:57:58.301 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T15:57:58.301 INFO:teuthology.orchestra.run.smithi156.stdout: # disable pg_autoscaler by default for new pools 2022-06-29T15:57:58.301 INFO:teuthology.orchestra.run.smithi156.stdout: osd_pool_default_pg_autoscale_mode = off 2022-06-29T15:57:58.301 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T15:57:58.302 INFO:teuthology.orchestra.run.smithi156.stdout: osd pool default size = 2 2022-06-29T15:57:58.302 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T15:57:58.302 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd allow primary affinity = true 2022-06-29T15:57:58.302 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd allow pg remap = true 2022-06-29T15:57:58.303 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on legacy crush tunables = false 2022-06-29T15:57:58.303 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on crush straw calc version zero = false 2022-06-29T15:57:58.303 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on no sortbitwise = false 2022-06-29T15:57:58.303 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on osd down out interval zero = false 2022-06-29T15:57:58.304 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on too few osds = false 2022-06-29T15:57:58.304 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-06-29T15:57:58.304 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_pool_no_redundancy = false 2022-06-29T15:57:58.304 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T15:57:58.304 INFO:teuthology.orchestra.run.smithi156.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-06-29T15:57:58.305 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T15:57:58.305 INFO:teuthology.orchestra.run.smithi156.stdout: osd default data pool replay window = 5 2022-06-29T15:57:58.305 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T15:57:58.305 INFO:teuthology.orchestra.run.smithi156.stdout: mon allow pool delete = true 2022-06-29T15:57:58.306 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T15:57:58.306 INFO:teuthology.orchestra.run.smithi156.stdout: mon cluster log file level = debug 2022-06-29T15:57:58.306 INFO:teuthology.orchestra.run.smithi156.stdout: debug asserts on shutdown = true 2022-06-29T15:57:58.306 INFO:teuthology.orchestra.run.smithi156.stdout: mon health detail to clog = false 2022-06-29T15:57:58.307 INFO:teuthology.orchestra.run.smithi156.stdout: mon host = "172.21.15.60,172.21.15.92,[v2:172.21.15.92:3301,v1:172.21.15.92:6790]" 2022-06-29T15:57:58.307 INFO:teuthology.orchestra.run.smithi156.stdout: fsid = e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85 2022-06-29T15:57:58.307 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T15:57:58.307 INFO:teuthology.orchestra.run.smithi156.stdout:[osd] 2022-06-29T15:57:58.308 INFO:teuthology.orchestra.run.smithi156.stdout: osd journal size = 100 2022-06-29T15:57:58.308 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T15:57:58.308 INFO:teuthology.orchestra.run.smithi156.stdout: osd scrub load threshold = 5.0 2022-06-29T15:57:58.308 INFO:teuthology.orchestra.run.smithi156.stdout: osd scrub max interval = 600 2022-06-29T15:57:58.308 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T15:57:58.309 INFO:teuthology.orchestra.run.smithi156.stdout: osd recover clone overlap = true 2022-06-29T15:57:58.309 INFO:teuthology.orchestra.run.smithi156.stdout: osd recovery max chunk = 1048576 2022-06-29T15:57:58.309 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T15:57:58.309 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug shutdown = true 2022-06-29T15:57:58.309 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug op order = true 2022-06-29T15:57:58.310 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug verify stray on activate = true 2022-06-29T15:57:58.310 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T15:57:58.310 INFO:teuthology.orchestra.run.smithi156.stdout: osd open classes on start = true 2022-06-29T15:57:58.310 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug pg log writeout = true 2022-06-29T15:57:58.311 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T15:57:58.311 INFO:teuthology.orchestra.run.smithi156.stdout: osd deep scrub update digest min age = 30 2022-06-29T15:57:58.311 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T15:57:58.312 INFO:teuthology.orchestra.run.smithi156.stdout: osd map max advance = 10 2022-06-29T15:57:58.312 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T15:57:58.312 INFO:teuthology.orchestra.run.smithi156.stdout: journal zero on create = true 2022-06-29T15:57:58.312 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T15:57:58.312 INFO:teuthology.orchestra.run.smithi156.stdout: filestore ondisk finisher threads = 3 2022-06-29T15:57:58.313 INFO:teuthology.orchestra.run.smithi156.stdout: filestore apply finisher threads = 3 2022-06-29T15:57:58.313 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T15:57:58.313 INFO:teuthology.orchestra.run.smithi156.stdout: bdev debug aio = true 2022-06-29T15:57:58.313 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug misdirected ops = true 2022-06-29T15:57:58.314 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2022-06-29T15:57:58.314 INFO:teuthology.orchestra.run.smithi156.stdout: debug osd = 20 2022-06-29T15:57:58.314 INFO:teuthology.orchestra.run.smithi156.stdout: osd objectstore = filestore 2022-06-29T15:57:58.314 INFO:teuthology.orchestra.run.smithi156.stdout: osd op complaint time = 180 2022-06-29T15:57:58.314 INFO:teuthology.orchestra.run.smithi156.stdout: osd sloppy crc = True 2022-06-29T15:57:58.315 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T15:57:58.315 INFO:teuthology.orchestra.run.smithi156.stdout:[mgr] 2022-06-29T15:57:58.315 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2022-06-29T15:57:58.315 INFO:teuthology.orchestra.run.smithi156.stdout: debug mgr = 20 2022-06-29T15:57:58.316 INFO:teuthology.orchestra.run.smithi156.stdout: debug mon = 20 2022-06-29T15:57:58.316 INFO:teuthology.orchestra.run.smithi156.stdout: debug auth = 20 2022-06-29T15:57:58.316 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min pgs per osd = 4 2022-06-29T15:57:58.316 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min bytes per osd = 10 2022-06-29T15:57:58.316 INFO:teuthology.orchestra.run.smithi156.stdout: mgr/telemetry/nag = false 2022-06-29T15:57:58.317 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T15:57:58.317 INFO:teuthology.orchestra.run.smithi156.stdout:[mon] 2022-06-29T15:57:58.317 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2022-06-29T15:57:58.317 INFO:teuthology.orchestra.run.smithi156.stdout: debug mon = 20 2022-06-29T15:57:58.318 INFO:teuthology.orchestra.run.smithi156.stdout: debug paxos = 20 2022-06-29T15:57:58.318 INFO:teuthology.orchestra.run.smithi156.stdout: debug auth = 20 2022-06-29T15:57:58.318 INFO:teuthology.orchestra.run.smithi156.stdout: mon data avail warn = 5 2022-06-29T15:57:58.318 INFO:teuthology.orchestra.run.smithi156.stdout: mon mgr mkfs grace = 240 2022-06-29T15:57:58.318 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min pgs per osd = 4 2022-06-29T15:57:58.319 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd reporter subtree level = osd 2022-06-29T15:57:58.319 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd prime pg temp = true 2022-06-29T15:57:58.319 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min bytes per osd = 10 2022-06-29T15:57:58.319 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T15:57:58.320 INFO:teuthology.orchestra.run.smithi156.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-06-29T15:57:58.320 INFO:teuthology.orchestra.run.smithi156.stdout: auth mon ticket ttl = 660 # 11m 2022-06-29T15:57:58.320 INFO:teuthology.orchestra.run.smithi156.stdout: auth service ticket ttl = 240 # 4m 2022-06-29T15:57:58.320 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T15:57:58.321 INFO:teuthology.orchestra.run.smithi156.stdout: # don't complain about insecure global_id in the test suite 2022-06-29T15:57:58.321 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-06-29T15:57:58.321 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-06-29T15:57:58.321 INFO:teuthology.orchestra.run.smithi156.stdout: mon op complaint time = 120 2022-06-29T15:57:58.322 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T15:57:58.322 INFO:teuthology.orchestra.run.smithi156.stdout:[client] 2022-06-29T15:57:58.322 INFO:teuthology.orchestra.run.smithi156.stdout: rgw cache enabled = true 2022-06-29T15:57:58.322 INFO:teuthology.orchestra.run.smithi156.stdout: rgw enable ops log = true 2022-06-29T15:57:58.323 INFO:teuthology.orchestra.run.smithi156.stdout: rgw enable usage log = true 2022-06-29T15:57:58.323 INFO:teuthology.orchestra.run.smithi156.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-06-29T15:57:58.323 INFO:teuthology.orchestra.run.smithi156.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-06-29T15:57:58.323 INFO:teuthology.orchestra.run.smithi156.stdout: client mount timeout = 600 2022-06-29T15:57:58.324 INFO:teuthology.orchestra.run.smithi156.stdout: debug client = 20 2022-06-29T15:57:58.324 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2022-06-29T15:57:58.324 INFO:teuthology.orchestra.run.smithi156.stdout: rados mon op timeout = 15m 2022-06-29T15:57:58.324 INFO:teuthology.orchestra.run.smithi156.stdout: rados osd op timeout = 15m 2022-06-29T15:57:58.325 INFO:teuthology.orchestra.run.smithi156.stdout:[mon.a] 2022-06-29T15:57:58.325 INFO:teuthology.orchestra.run.smithi156.stdout:[mon.b] 2022-06-29T15:57:58.325 INFO:teuthology.orchestra.run.smithi156.stdout:[mon.c] 2022-06-29T15:57:58.325 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.a] 2022-06-29T15:57:58.325 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.b] 2022-06-29T15:57:58.326 INFO:teuthology.orchestra.run.smithi156.stdout:[mds] 2022-06-29T15:57:58.326 INFO:teuthology.orchestra.run.smithi156.stdout: debug mds = 20 2022-06-29T15:57:58.326 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2022-06-29T15:57:58.326 INFO:teuthology.orchestra.run.smithi156.stdout: mds bal frag = True 2022-06-29T15:57:58.327 INFO:teuthology.orchestra.run.smithi156.stdout: mds bal fragment size max = 10000 2022-06-29T15:57:58.327 INFO:teuthology.orchestra.run.smithi156.stdout: mds bal merge size = 5 2022-06-29T15:57:58.327 INFO:teuthology.orchestra.run.smithi156.stdout: mds bal split bits = 3 2022-06-29T15:57:58.327 INFO:teuthology.orchestra.run.smithi156.stdout: mds bal split size = 100 2022-06-29T15:57:58.327 INFO:teuthology.orchestra.run.smithi156.stdout: mds debug frag = True 2022-06-29T15:57:58.328 INFO:teuthology.orchestra.run.smithi156.stdout: mds debug scatterstat = True 2022-06-29T15:57:58.328 INFO:teuthology.orchestra.run.smithi156.stdout: mds op complaint time = 180 2022-06-29T15:57:58.328 INFO:teuthology.orchestra.run.smithi156.stdout: mds verify scatter = True 2022-06-29T15:57:58.328 INFO:teuthology.orchestra.run.smithi156.stdout: osd op complaint time = 180 2022-06-29T15:57:58.329 INFO:teuthology.orchestra.run.smithi156.stdout: rados mon op timeout = 15m 2022-06-29T15:57:58.329 INFO:teuthology.orchestra.run.smithi156.stdout: rados osd op timeout = 15m 2022-06-29T15:57:58.329 INFO:teuthology.orchestra.run.smithi092.stdout:[global] 2022-06-29T15:57:58.329 INFO:teuthology.orchestra.run.smithi092.stdout: chdir = "" 2022-06-29T15:57:58.330 INFO:teuthology.orchestra.run.smithi092.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-06-29T15:57:58.330 INFO:teuthology.orchestra.run.smithi092.stdout: auth supported = cephx 2022-06-29T15:57:58.330 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T15:57:58.330 INFO:teuthology.orchestra.run.smithi092.stdout: filestore xattr use omap = true 2022-06-29T15:57:58.331 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T15:57:58.331 INFO:teuthology.orchestra.run.smithi092.stdout: mon clock drift allowed = 1.000 2022-06-29T15:57:58.331 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T15:57:58.331 INFO:teuthology.orchestra.run.smithi092.stdout: osd crush chooseleaf type = 0 2022-06-29T15:57:58.331 INFO:teuthology.orchestra.run.smithi092.stdout: auth debug = true 2022-06-29T15:57:58.332 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T15:57:58.332 INFO:teuthology.orchestra.run.smithi092.stdout: ms die on old message = true 2022-06-29T15:57:58.332 INFO:teuthology.orchestra.run.smithi092.stdout: ms die on bug = true 2022-06-29T15:57:58.333 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T15:57:58.333 INFO:teuthology.orchestra.run.smithi092.stdout: mon max pg per osd = 10000 # >= luminous 2022-06-29T15:57:58.333 INFO:teuthology.orchestra.run.smithi092.stdout: mon pg warn max object skew = 0 2022-06-29T15:57:58.334 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T15:57:58.334 INFO:teuthology.orchestra.run.smithi092.stdout: # disable pg_autoscaler by default for new pools 2022-06-29T15:57:58.334 INFO:teuthology.orchestra.run.smithi092.stdout: osd_pool_default_pg_autoscale_mode = off 2022-06-29T15:57:58.335 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T15:57:58.335 INFO:teuthology.orchestra.run.smithi092.stdout: osd pool default size = 2 2022-06-29T15:57:58.336 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T15:57:58.336 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd allow primary affinity = true 2022-06-29T15:57:58.337 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd allow pg remap = true 2022-06-29T15:57:58.337 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on legacy crush tunables = false 2022-06-29T15:57:58.338 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on crush straw calc version zero = false 2022-06-29T15:57:58.338 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on no sortbitwise = false 2022-06-29T15:57:58.338 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on osd down out interval zero = false 2022-06-29T15:57:58.339 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on too few osds = false 2022-06-29T15:57:58.339 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-06-29T15:57:58.340 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_pool_no_redundancy = false 2022-06-29T15:57:58.340 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T15:57:58.340 INFO:teuthology.orchestra.run.smithi092.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-06-29T15:57:58.340 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T15:57:58.341 INFO:teuthology.orchestra.run.smithi092.stdout: osd default data pool replay window = 5 2022-06-29T15:57:58.341 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T15:57:58.342 INFO:teuthology.orchestra.run.smithi092.stdout: mon allow pool delete = true 2022-06-29T15:57:58.342 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T15:57:58.342 INFO:teuthology.orchestra.run.smithi092.stdout: mon cluster log file level = debug 2022-06-29T15:57:58.342 INFO:teuthology.orchestra.run.smithi092.stdout: debug asserts on shutdown = true 2022-06-29T15:57:58.343 INFO:teuthology.orchestra.run.smithi092.stdout: mon health detail to clog = false 2022-06-29T15:57:58.343 INFO:teuthology.orchestra.run.smithi092.stdout: mon host = "172.21.15.60,172.21.15.92,[v2:172.21.15.92:3301,v1:172.21.15.92:6790]" 2022-06-29T15:57:58.343 INFO:teuthology.orchestra.run.smithi092.stdout: fsid = e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85 2022-06-29T15:57:58.343 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T15:57:58.343 INFO:teuthology.orchestra.run.smithi092.stdout:[osd] 2022-06-29T15:57:58.344 INFO:teuthology.orchestra.run.smithi092.stdout: osd journal size = 100 2022-06-29T15:57:58.344 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T15:57:58.344 INFO:teuthology.orchestra.run.smithi092.stdout: osd scrub load threshold = 5.0 2022-06-29T15:57:58.344 INFO:teuthology.orchestra.run.smithi092.stdout: osd scrub max interval = 600 2022-06-29T15:57:58.345 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T15:57:58.345 INFO:teuthology.orchestra.run.smithi092.stdout: osd recover clone overlap = true 2022-06-29T15:57:58.345 INFO:teuthology.orchestra.run.smithi092.stdout: osd recovery max chunk = 1048576 2022-06-29T15:57:58.345 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T15:57:58.346 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug shutdown = true 2022-06-29T15:57:58.346 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug op order = true 2022-06-29T15:57:58.346 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug verify stray on activate = true 2022-06-29T15:57:58.346 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T15:57:58.346 INFO:teuthology.orchestra.run.smithi092.stdout: osd open classes on start = true 2022-06-29T15:57:58.347 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug pg log writeout = true 2022-06-29T15:57:58.347 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T15:57:58.347 INFO:teuthology.orchestra.run.smithi092.stdout: osd deep scrub update digest min age = 30 2022-06-29T15:57:58.347 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T15:57:58.348 INFO:teuthology.orchestra.run.smithi092.stdout: osd map max advance = 10 2022-06-29T15:57:58.348 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T15:57:58.348 INFO:teuthology.orchestra.run.smithi092.stdout: journal zero on create = true 2022-06-29T15:57:58.348 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T15:57:58.348 INFO:teuthology.orchestra.run.smithi092.stdout: filestore ondisk finisher threads = 3 2022-06-29T15:57:58.349 INFO:teuthology.orchestra.run.smithi092.stdout: filestore apply finisher threads = 3 2022-06-29T15:57:58.349 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T15:57:58.349 INFO:teuthology.orchestra.run.smithi092.stdout: bdev debug aio = true 2022-06-29T15:57:58.349 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug misdirected ops = true 2022-06-29T15:57:58.350 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2022-06-29T15:57:58.350 INFO:teuthology.orchestra.run.smithi092.stdout: debug osd = 20 2022-06-29T15:57:58.350 INFO:teuthology.orchestra.run.smithi092.stdout: osd objectstore = filestore 2022-06-29T15:57:58.350 INFO:teuthology.orchestra.run.smithi092.stdout: osd op complaint time = 180 2022-06-29T15:57:58.351 INFO:teuthology.orchestra.run.smithi092.stdout: osd sloppy crc = True 2022-06-29T15:57:58.351 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T15:57:58.351 INFO:teuthology.orchestra.run.smithi092.stdout:[mgr] 2022-06-29T15:57:58.351 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2022-06-29T15:57:58.351 INFO:teuthology.orchestra.run.smithi092.stdout: debug mgr = 20 2022-06-29T15:57:58.352 INFO:teuthology.orchestra.run.smithi092.stdout: debug mon = 20 2022-06-29T15:57:58.352 INFO:teuthology.orchestra.run.smithi092.stdout: debug auth = 20 2022-06-29T15:57:58.352 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min pgs per osd = 4 2022-06-29T15:57:58.352 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min bytes per osd = 10 2022-06-29T15:57:58.353 INFO:teuthology.orchestra.run.smithi092.stdout: mgr/telemetry/nag = false 2022-06-29T15:57:58.353 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T15:57:58.353 INFO:teuthology.orchestra.run.smithi092.stdout:[mon] 2022-06-29T15:57:58.353 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2022-06-29T15:57:58.353 INFO:teuthology.orchestra.run.smithi092.stdout: debug mon = 20 2022-06-29T15:57:58.354 INFO:teuthology.orchestra.run.smithi092.stdout: debug paxos = 20 2022-06-29T15:57:58.354 INFO:teuthology.orchestra.run.smithi092.stdout: debug auth = 20 2022-06-29T15:57:58.354 INFO:teuthology.orchestra.run.smithi092.stdout: mon data avail warn = 5 2022-06-29T15:57:58.354 INFO:teuthology.orchestra.run.smithi092.stdout: mon mgr mkfs grace = 240 2022-06-29T15:57:58.355 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min pgs per osd = 4 2022-06-29T15:57:58.355 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd reporter subtree level = osd 2022-06-29T15:57:58.355 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd prime pg temp = true 2022-06-29T15:57:58.355 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min bytes per osd = 10 2022-06-29T15:57:58.356 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T15:57:58.356 INFO:teuthology.orchestra.run.smithi092.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-06-29T15:57:58.356 INFO:teuthology.orchestra.run.smithi092.stdout: auth mon ticket ttl = 660 # 11m 2022-06-29T15:57:58.357 INFO:teuthology.orchestra.run.smithi092.stdout: auth service ticket ttl = 240 # 4m 2022-06-29T15:57:58.357 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T15:57:58.357 INFO:teuthology.orchestra.run.smithi092.stdout: # don't complain about insecure global_id in the test suite 2022-06-29T15:57:58.357 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-06-29T15:57:58.358 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-06-29T15:57:58.358 INFO:teuthology.orchestra.run.smithi092.stdout: mon op complaint time = 120 2022-06-29T15:57:58.358 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T15:57:58.359 INFO:teuthology.orchestra.run.smithi092.stdout:[client] 2022-06-29T15:57:58.359 INFO:teuthology.orchestra.run.smithi092.stdout: rgw cache enabled = true 2022-06-29T15:57:58.359 INFO:teuthology.orchestra.run.smithi092.stdout: rgw enable ops log = true 2022-06-29T15:57:58.359 INFO:teuthology.orchestra.run.smithi092.stdout: rgw enable usage log = true 2022-06-29T15:57:58.359 INFO:teuthology.orchestra.run.smithi092.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-06-29T15:57:58.360 INFO:teuthology.orchestra.run.smithi092.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-06-29T15:57:58.360 INFO:teuthology.orchestra.run.smithi092.stdout: client mount timeout = 600 2022-06-29T15:57:58.360 INFO:teuthology.orchestra.run.smithi092.stdout: debug client = 20 2022-06-29T15:57:58.360 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2022-06-29T15:57:58.361 INFO:teuthology.orchestra.run.smithi092.stdout: rados mon op timeout = 15m 2022-06-29T15:57:58.361 INFO:teuthology.orchestra.run.smithi092.stdout: rados osd op timeout = 15m 2022-06-29T15:57:58.361 INFO:teuthology.orchestra.run.smithi092.stdout:[mon.a] 2022-06-29T15:57:58.361 INFO:teuthology.orchestra.run.smithi092.stdout:[mon.b] 2022-06-29T15:57:58.362 INFO:teuthology.orchestra.run.smithi092.stdout:[mon.c] 2022-06-29T15:57:58.362 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.a] 2022-06-29T15:57:58.362 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.b] 2022-06-29T15:57:58.362 INFO:teuthology.orchestra.run.smithi092.stdout:[mds] 2022-06-29T15:57:58.363 INFO:teuthology.orchestra.run.smithi092.stdout: debug mds = 20 2022-06-29T15:57:58.363 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2022-06-29T15:57:58.363 INFO:teuthology.orchestra.run.smithi092.stdout: mds bal frag = True 2022-06-29T15:57:58.363 INFO:teuthology.orchestra.run.smithi092.stdout: mds bal fragment size max = 10000 2022-06-29T15:57:58.364 INFO:teuthology.orchestra.run.smithi092.stdout: mds bal merge size = 5 2022-06-29T15:57:58.364 INFO:teuthology.orchestra.run.smithi092.stdout: mds bal split bits = 3 2022-06-29T15:57:58.364 INFO:teuthology.orchestra.run.smithi092.stdout: mds bal split size = 100 2022-06-29T15:57:58.364 INFO:teuthology.orchestra.run.smithi092.stdout: mds debug frag = True 2022-06-29T15:57:58.365 INFO:teuthology.orchestra.run.smithi092.stdout: mds debug scatterstat = True 2022-06-29T15:57:58.365 INFO:teuthology.orchestra.run.smithi092.stdout: mds op complaint time = 180 2022-06-29T15:57:58.365 INFO:teuthology.orchestra.run.smithi092.stdout: mds verify scatter = True 2022-06-29T15:57:58.365 INFO:teuthology.orchestra.run.smithi092.stdout: osd op complaint time = 180 2022-06-29T15:57:58.366 INFO:teuthology.orchestra.run.smithi092.stdout: rados mon op timeout = 15m 2022-06-29T15:57:58.366 INFO:teuthology.orchestra.run.smithi092.stdout: rados osd op timeout = 15m 2022-06-29T15:57:58.367 INFO:tasks.ceph:Creating admin key on mon.a... 2022-06-29T15:57:58.367 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2022-06-29T15:57:58.415 INFO:tasks.ceph:Copying monmap to all nodes... 2022-06-29T15:57:58.428 DEBUG:teuthology.orchestra.remote:smithi060:/etc/ceph/ceph.keyring is 206B 2022-06-29T15:57:58.451 DEBUG:teuthology.orchestra.remote:smithi060:/home/ubuntu/cephtest/ceph.monmap is 379B 2022-06-29T15:57:58.465 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi060.front.sepia.ceph.com 2022-06-29T15:57:58.466 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2022-06-29T15:57:58.466 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/ceph/ceph.keyring 2022-06-29T15:57:58.467 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-06-29T15:57:58.495 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2022-06-29T15:57:58.496 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/ceph.monmap 2022-06-29T15:57:58.544 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi092.front.sepia.ceph.com 2022-06-29T15:57:58.544 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2022-06-29T15:57:58.545 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/ceph/ceph.keyring 2022-06-29T15:57:58.545 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-06-29T15:57:58.573 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2022-06-29T15:57:58.574 DEBUG:teuthology.orchestra.run.smithi092:> dd of=/home/ubuntu/cephtest/ceph.monmap 2022-06-29T15:57:58.580 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi121.front.sepia.ceph.com 2022-06-29T15:57:58.581 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-06-29T15:57:58.581 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.keyring 2022-06-29T15:57:58.581 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-06-29T15:57:58.609 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-06-29T15:57:58.610 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/home/ubuntu/cephtest/ceph.monmap 2022-06-29T15:57:58.663 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi156.front.sepia.ceph.com 2022-06-29T15:57:58.664 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2022-06-29T15:57:58.664 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/ceph/ceph.keyring 2022-06-29T15:57:58.665 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-06-29T15:57:58.692 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2022-06-29T15:57:58.693 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/home/ubuntu/cephtest/ceph.monmap 2022-06-29T15:57:58.698 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi197.front.sepia.ceph.com 2022-06-29T15:57:58.699 DEBUG:teuthology.orchestra.run.smithi197:> set -ex 2022-06-29T15:57:58.699 DEBUG:teuthology.orchestra.run.smithi197:> sudo dd of=/etc/ceph/ceph.keyring 2022-06-29T15:57:58.700 DEBUG:teuthology.orchestra.run.smithi197:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-06-29T15:57:58.727 DEBUG:teuthology.orchestra.run.smithi197:> set -ex 2022-06-29T15:57:58.727 DEBUG:teuthology.orchestra.run.smithi197:> dd of=/home/ubuntu/cephtest/ceph.monmap 2022-06-29T15:57:58.774 INFO:tasks.ceph:Setting up mon nodes... 2022-06-29T15:57:58.775 INFO:tasks.ceph:Setting up mgr nodes... 2022-06-29T15:57:58.775 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2022-06-29T15:57:58.819 INFO:teuthology.orchestra.run.smithi060.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2022-06-29T15:57:58.822 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2022-06-29T15:57:58.874 INFO:teuthology.orchestra.run.smithi092.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2022-06-29T15:57:58.878 INFO:tasks.ceph:Setting up mds nodes... 2022-06-29T15:57:58.879 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2022-06-29T15:57:58.926 INFO:teuthology.orchestra.run.smithi060.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2022-06-29T15:57:58.927 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2022-06-29T15:57:58.945 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2022-06-29T15:57:58.987 INFO:teuthology.orchestra.run.smithi092.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2022-06-29T15:57:58.990 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2022-06-29T15:57:59.047 INFO:tasks.ceph_client:Setting up client nodes... 2022-06-29T15:57:59.048 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2022-06-29T15:57:59.100 INFO:teuthology.orchestra.run.smithi121.stdout:creating /etc/ceph/ceph.client.0.keyring 2022-06-29T15:57:59.113 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2022-06-29T15:57:59.155 INFO:teuthology.orchestra.run.smithi156.stdout:creating /etc/ceph/ceph.client.1.keyring 2022-06-29T15:57:59.169 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2022-06-29T15:57:59.221 INFO:teuthology.orchestra.run.smithi197.stdout:creating /etc/ceph/ceph.client.2.keyring 2022-06-29T15:57:59.235 INFO:tasks.ceph:Running mkfs on osd nodes... 2022-06-29T15:57:59.236 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi060.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi092.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2022-06-29T15:57:59.236 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2022-06-29T15:57:59.253 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-06-29T15:57:59.253 INFO:tasks.ceph:role: osd.0 2022-06-29T15:57:59.253 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi060.front.sepia.ceph.com 2022-06-29T15:57:59.254 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2022-06-29T15:57:59.295 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2022-06-29T15:57:59.296 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2022-06-29T15:57:59.296 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-06-29T15:57:59.296 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 2022-06-29T15:57:59.297 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-06-29T15:57:59.297 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2022-06-29T15:57:59.297 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-06-29T15:57:59.297 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-06-29T15:57:59.298 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-06-29T15:57:59.298 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-06-29T15:57:59.299 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2022-06-29T15:57:59.300 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2022-06-29T15:57:59.362 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2022-06-29T15:57:59.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T15:57:59.372 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: /sbin/restorecon: command not found 2022-06-29T15:57:59.373 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2022-06-29T15:57:59.384 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-06-29T15:57:59.385 INFO:tasks.ceph:role: osd.1 2022-06-29T15:57:59.385 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi060.front.sepia.ceph.com 2022-06-29T15:57:59.385 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2022-06-29T15:57:59.469 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2022-06-29T15:57:59.470 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2022-06-29T15:57:59.470 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-06-29T15:57:59.471 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 2022-06-29T15:57:59.471 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-06-29T15:57:59.472 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2022-06-29T15:57:59.472 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-06-29T15:57:59.473 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-06-29T15:57:59.473 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-06-29T15:57:59.474 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-06-29T15:57:59.476 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2022-06-29T15:57:59.477 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2022-06-29T15:57:59.501 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2022-06-29T15:57:59.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T15:57:59.554 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: /sbin/restorecon: command not found 2022-06-29T15:57:59.555 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2022-06-29T15:57:59.568 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-06-29T15:57:59.569 INFO:tasks.ceph:role: osd.2 2022-06-29T15:57:59.570 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi060.front.sepia.ceph.com 2022-06-29T15:57:59.570 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2022-06-29T15:57:59.614 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2022-06-29T15:57:59.615 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2022-06-29T15:57:59.615 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-06-29T15:57:59.616 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 2022-06-29T15:57:59.616 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-06-29T15:57:59.617 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2022-06-29T15:57:59.617 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-06-29T15:57:59.618 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-06-29T15:57:59.618 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-06-29T15:57:59.618 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-06-29T15:57:59.621 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2022-06-29T15:57:59.621 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2022-06-29T15:57:59.683 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2022-06-29T15:57:59.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T15:57:59.724 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: /sbin/restorecon: command not found 2022-06-29T15:57:59.725 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2022-06-29T15:57:59.785 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-06-29T15:57:59.786 INFO:tasks.ceph:role: osd.3 2022-06-29T15:57:59.786 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi060.front.sepia.ceph.com 2022-06-29T15:57:59.786 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2022-06-29T15:57:59.830 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2022-06-29T15:57:59.831 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2022-06-29T15:57:59.831 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-06-29T15:57:59.831 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 2022-06-29T15:57:59.832 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-06-29T15:57:59.832 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2022-06-29T15:57:59.832 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-06-29T15:57:59.832 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-06-29T15:57:59.833 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-06-29T15:57:59.833 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-06-29T15:57:59.834 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2022-06-29T15:57:59.835 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2022-06-29T15:57:59.851 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2022-06-29T15:57:59.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T15:57:59.905 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: /sbin/restorecon: command not found 2022-06-29T15:57:59.905 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-06-29T15:57:59.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:57:59.953+0000 7efffa7e2d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2022-06-29T15:57:59.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:57:59.953+0000 7efffa7e2d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2022-06-29T15:57:59.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:57:59.961+0000 7efffa7e2d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-06-29T15:57:59.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:57:59.961+0000 7efffa7e2d80 -1 journal FileJournal::_open_file : zeroing journal 2022-06-29T15:58:00.045 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:00.037+0000 7efffa7e2d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-06-29T15:58:00.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:00.041+0000 7efffa7e2d80 -1 journal do_read_entry(4096): bad header magic 2022-06-29T15:58:00.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:00.041+0000 7efffa7e2d80 -1 journal do_read_entry(4096): bad header magic 2022-06-29T15:58:00.059 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2022-06-29T15:58:00.075 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-06-29T15:58:00.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:00.101+0000 7fd1bd1e6d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2022-06-29T15:58:00.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:00.101+0000 7fd1bd1e6d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2022-06-29T15:58:00.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:00.113+0000 7fd1bd1e6d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-06-29T15:58:00.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:00.113+0000 7fd1bd1e6d80 -1 journal FileJournal::_open_file : zeroing journal 2022-06-29T15:58:00.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:00.189+0000 7fd1bd1e6d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-06-29T15:58:00.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:00.189+0000 7fd1bd1e6d80 -1 journal do_read_entry(4096): bad header magic 2022-06-29T15:58:00.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:00.189+0000 7fd1bd1e6d80 -1 journal do_read_entry(4096): bad header magic 2022-06-29T15:58:00.209 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2022-06-29T15:58:00.271 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-06-29T15:58:00.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:00.301+0000 7f47921ead80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2022-06-29T15:58:00.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:00.301+0000 7f47921ead80 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2022-06-29T15:58:00.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:00.313+0000 7f47921ead80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-06-29T15:58:00.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:00.313+0000 7f47921ead80 -1 journal FileJournal::_open_file : zeroing journal 2022-06-29T15:58:00.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:00.389+0000 7f47921ead80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-06-29T15:58:00.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:00.389+0000 7f47921ead80 -1 journal do_read_entry(4096): bad header magic 2022-06-29T15:58:00.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:00.389+0000 7f47921ead80 -1 journal do_read_entry(4096): bad header magic 2022-06-29T15:58:00.410 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2022-06-29T15:58:00.428 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-06-29T15:58:00.511 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:00.505+0000 7ff217670d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2022-06-29T15:58:00.511 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:00.505+0000 7ff217670d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2022-06-29T15:58:00.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:00.513+0000 7ff217670d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-06-29T15:58:00.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:00.513+0000 7ff217670d80 -1 journal FileJournal::_open_file : zeroing journal 2022-06-29T15:58:00.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:00.589+0000 7ff217670d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-06-29T15:58:00.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:00.589+0000 7ff217670d80 -1 journal do_read_entry(4096): bad header magic 2022-06-29T15:58:00.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:00.589+0000 7ff217670d80 -1 journal do_read_entry(4096): bad header magic 2022-06-29T15:58:00.610 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2022-06-29T15:58:00.625 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2022-06-29T15:58:00.642 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-06-29T15:58:00.643 INFO:tasks.ceph:role: osd.4 2022-06-29T15:58:00.643 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi092.front.sepia.ceph.com 2022-06-29T15:58:00.644 DEBUG:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2022-06-29T15:58:01.170 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2022-06-29T15:58:01.171 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2022-06-29T15:58:01.171 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-06-29T15:58:01.171 INFO:teuthology.orchestra.run.smithi092.stdout: = reflink=1 2022-06-29T15:58:01.171 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-06-29T15:58:01.172 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2022-06-29T15:58:01.172 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-06-29T15:58:01.172 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-06-29T15:58:01.172 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-06-29T15:58:01.172 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-06-29T15:58:01.174 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2022-06-29T15:58:01.174 DEBUG:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2022-06-29T15:58:01.269 DEBUG:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2022-06-29T15:58:01.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T15:58:01.282 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: /sbin/restorecon: command not found 2022-06-29T15:58:01.283 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2022-06-29T15:58:01.293 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-06-29T15:58:01.294 INFO:tasks.ceph:role: osd.5 2022-06-29T15:58:01.294 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi092.front.sepia.ceph.com 2022-06-29T15:58:01.294 DEBUG:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2022-06-29T15:58:01.828 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2022-06-29T15:58:01.828 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2022-06-29T15:58:01.828 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-06-29T15:58:01.828 INFO:teuthology.orchestra.run.smithi092.stdout: = reflink=1 2022-06-29T15:58:01.829 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-06-29T15:58:01.829 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2022-06-29T15:58:01.829 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-06-29T15:58:01.829 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-06-29T15:58:01.830 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-06-29T15:58:01.830 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-06-29T15:58:01.831 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2022-06-29T15:58:01.831 DEBUG:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2022-06-29T15:58:01.872 DEBUG:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2022-06-29T15:58:01.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T15:58:01.884 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: /sbin/restorecon: command not found 2022-06-29T15:58:01.885 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2022-06-29T15:58:01.897 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-06-29T15:58:01.897 INFO:tasks.ceph:role: osd.6 2022-06-29T15:58:01.898 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi092.front.sepia.ceph.com 2022-06-29T15:58:01.898 DEBUG:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2022-06-29T15:58:02.404 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2022-06-29T15:58:02.405 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2022-06-29T15:58:02.405 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-06-29T15:58:02.405 INFO:teuthology.orchestra.run.smithi092.stdout: = reflink=1 2022-06-29T15:58:02.406 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-06-29T15:58:02.406 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2022-06-29T15:58:02.407 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-06-29T15:58:02.407 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-06-29T15:58:02.408 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-06-29T15:58:02.408 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-06-29T15:58:02.410 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2022-06-29T15:58:02.411 DEBUG:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2022-06-29T15:58:02.475 DEBUG:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2022-06-29T15:58:02.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T15:58:02.487 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: /sbin/restorecon: command not found 2022-06-29T15:58:02.488 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2022-06-29T15:58:02.500 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-06-29T15:58:02.501 INFO:tasks.ceph:role: osd.7 2022-06-29T15:58:02.502 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi092.front.sepia.ceph.com 2022-06-29T15:58:02.503 DEBUG:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2022-06-29T15:58:02.997 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2022-06-29T15:58:02.997 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2022-06-29T15:58:02.998 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-06-29T15:58:02.998 INFO:teuthology.orchestra.run.smithi092.stdout: = reflink=1 2022-06-29T15:58:02.998 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-06-29T15:58:02.999 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2022-06-29T15:58:02.999 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-06-29T15:58:02.999 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-06-29T15:58:02.999 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-06-29T15:58:03.000 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-06-29T15:58:03.001 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2022-06-29T15:58:03.001 DEBUG:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2022-06-29T15:58:03.063 DEBUG:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2022-06-29T15:58:03.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T15:58:03.075 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: /sbin/restorecon: command not found 2022-06-29T15:58:03.076 DEBUG:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-06-29T15:58:03.128 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:03.124+0000 7f4e0babed80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2022-06-29T15:58:03.130 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:03.128+0000 7f4e0babed80 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2022-06-29T15:58:03.151 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:03.148+0000 7f4e0babed80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-06-29T15:58:03.152 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:03.148+0000 7f4e0babed80 -1 journal FileJournal::_open_file : zeroing journal 2022-06-29T15:58:03.396 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:03.392+0000 7f4e0babed80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-06-29T15:58:03.400 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:03.396+0000 7f4e0babed80 -1 journal do_read_entry(4096): bad header magic 2022-06-29T15:58:03.400 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:03.396+0000 7f4e0babed80 -1 journal do_read_entry(4096): bad header magic 2022-06-29T15:58:03.444 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2022-06-29T15:58:03.459 DEBUG:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-06-29T15:58:03.542 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:03.540+0000 7f92e6b8bd80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2022-06-29T15:58:03.546 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:03.544+0000 7f92e6b8bd80 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2022-06-29T15:58:03.581 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:03.576+0000 7f92e6b8bd80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-06-29T15:58:03.582 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:03.580+0000 7f92e6b8bd80 -1 journal FileJournal::_open_file : zeroing journal 2022-06-29T15:58:03.844 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:03.840+0000 7f92e6b8bd80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-06-29T15:58:03.848 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:03.844+0000 7f92e6b8bd80 -1 journal do_read_entry(4096): bad header magic 2022-06-29T15:58:03.849 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:03.844+0000 7f92e6b8bd80 -1 journal do_read_entry(4096): bad header magic 2022-06-29T15:58:03.888 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2022-06-29T15:58:03.904 DEBUG:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-06-29T15:58:03.940 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:03.936+0000 7fe39ae8bd80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2022-06-29T15:58:03.941 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:03.940+0000 7fe39ae8bd80 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2022-06-29T15:58:03.980 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:03.976+0000 7fe39ae8bd80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-06-29T15:58:03.981 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:03.976+0000 7fe39ae8bd80 -1 journal FileJournal::_open_file : zeroing journal 2022-06-29T15:58:04.196 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:04.192+0000 7fe39ae8bd80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-06-29T15:58:04.199 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:04.196+0000 7fe39ae8bd80 -1 journal do_read_entry(4096): bad header magic 2022-06-29T15:58:04.199 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:04.196+0000 7fe39ae8bd80 -1 journal do_read_entry(4096): bad header magic 2022-06-29T15:58:04.213 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2022-06-29T15:58:04.271 DEBUG:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-06-29T15:58:04.309 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:04.308+0000 7fe368757d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2022-06-29T15:58:04.310 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:04.308+0000 7fe368757d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2022-06-29T15:58:04.322 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:04.320+0000 7fe368757d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-06-29T15:58:04.323 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:04.320+0000 7fe368757d80 -1 journal FileJournal::_open_file : zeroing journal 2022-06-29T15:58:04.469 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:04.464+0000 7fe368757d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-06-29T15:58:04.469 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:04.468+0000 7fe368757d80 -1 journal do_read_entry(4096): bad header magic 2022-06-29T15:58:04.470 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:04.468+0000 7fe368757d80 -1 journal do_read_entry(4096): bad header magic 2022-06-29T15:58:04.484 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2022-06-29T15:58:04.499 INFO:tasks.ceph:Reading keys from all nodes... 2022-06-29T15:58:04.499 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2022-06-29T15:58:04.507 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.8pRGZN9et8 2022-06-29T15:58:04.508 DEBUG:teuthology.orchestra.run.smithi060:> sudo cp /var/lib/ceph/mgr/ceph-y/keyring /tmp/tmp.8pRGZN9et8 2022-06-29T15:58:04.523 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.8pRGZN9et8 2022-06-29T15:58:04.590 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.8pRGZN9et8 is 56B 2022-06-29T15:58:04.643 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.8pRGZN9et8 2022-06-29T15:58:04.651 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2022-06-29T15:58:04.699 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.CP1H35qjvq 2022-06-29T15:58:04.700 DEBUG:teuthology.orchestra.run.smithi060:> sudo cp /var/lib/ceph/mds/ceph-a/keyring /tmp/tmp.CP1H35qjvq 2022-06-29T15:58:04.718 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.CP1H35qjvq 2022-06-29T15:58:04.781 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.CP1H35qjvq is 56B 2022-06-29T15:58:04.798 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.CP1H35qjvq 2022-06-29T15:58:04.808 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2022-06-29T15:58:04.855 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.RbtXqpZLeV 2022-06-29T15:58:04.856 DEBUG:teuthology.orchestra.run.smithi060:> sudo cp /var/lib/ceph/osd/ceph-0/keyring /tmp/tmp.RbtXqpZLeV 2022-06-29T15:58:04.871 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.RbtXqpZLeV 2022-06-29T15:58:04.892 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.RbtXqpZLeV is 56B 2022-06-29T15:58:04.908 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.RbtXqpZLeV 2022-06-29T15:58:04.916 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2022-06-29T15:58:04.963 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.xGui3JreW4 2022-06-29T15:58:04.964 DEBUG:teuthology.orchestra.run.smithi060:> sudo cp /var/lib/ceph/osd/ceph-1/keyring /tmp/tmp.xGui3JreW4 2022-06-29T15:58:04.978 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.xGui3JreW4 2022-06-29T15:58:05.003 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.xGui3JreW4 is 56B 2022-06-29T15:58:05.016 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.xGui3JreW4 2022-06-29T15:58:05.023 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2022-06-29T15:58:05.071 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.AeJFZlosMZ 2022-06-29T15:58:05.071 DEBUG:teuthology.orchestra.run.smithi060:> sudo cp /var/lib/ceph/osd/ceph-2/keyring /tmp/tmp.AeJFZlosMZ 2022-06-29T15:58:05.089 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.AeJFZlosMZ 2022-06-29T15:58:05.153 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.AeJFZlosMZ is 56B 2022-06-29T15:58:05.169 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.AeJFZlosMZ 2022-06-29T15:58:05.176 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2022-06-29T15:58:05.223 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.jR1cxVNO5z 2022-06-29T15:58:05.224 DEBUG:teuthology.orchestra.run.smithi060:> sudo cp /var/lib/ceph/osd/ceph-3/keyring /tmp/tmp.jR1cxVNO5z 2022-06-29T15:58:05.239 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.jR1cxVNO5z 2022-06-29T15:58:05.308 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.jR1cxVNO5z is 56B 2022-06-29T15:58:05.323 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.jR1cxVNO5z 2022-06-29T15:58:05.330 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2022-06-29T15:58:05.337 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.xZ6SiXcXaH 2022-06-29T15:58:05.338 DEBUG:teuthology.orchestra.run.smithi092:> sudo cp /var/lib/ceph/mgr/ceph-x/keyring /tmp/tmp.xZ6SiXcXaH 2022-06-29T15:58:05.355 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.xZ6SiXcXaH 2022-06-29T15:58:05.420 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.xZ6SiXcXaH is 56B 2022-06-29T15:58:05.435 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.xZ6SiXcXaH 2022-06-29T15:58:05.443 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2022-06-29T15:58:05.491 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.if2kmgqRs8 2022-06-29T15:58:05.492 DEBUG:teuthology.orchestra.run.smithi092:> sudo cp /var/lib/ceph/mds/ceph-b/keyring /tmp/tmp.if2kmgqRs8 2022-06-29T15:58:05.509 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.if2kmgqRs8 2022-06-29T15:58:05.576 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.if2kmgqRs8 is 56B 2022-06-29T15:58:05.592 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.if2kmgqRs8 2022-06-29T15:58:05.599 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2022-06-29T15:58:05.647 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.Pe4Umo0dC3 2022-06-29T15:58:05.647 DEBUG:teuthology.orchestra.run.smithi092:> sudo cp /var/lib/ceph/osd/ceph-4/keyring /tmp/tmp.Pe4Umo0dC3 2022-06-29T15:58:05.663 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.Pe4Umo0dC3 2022-06-29T15:58:05.728 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.Pe4Umo0dC3 is 56B 2022-06-29T15:58:05.744 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.Pe4Umo0dC3 2022-06-29T15:58:05.751 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2022-06-29T15:58:05.799 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.RVetRpzysm 2022-06-29T15:58:05.799 DEBUG:teuthology.orchestra.run.smithi092:> sudo cp /var/lib/ceph/osd/ceph-5/keyring /tmp/tmp.RVetRpzysm 2022-06-29T15:58:05.816 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.RVetRpzysm 2022-06-29T15:58:05.837 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.RVetRpzysm is 56B 2022-06-29T15:58:05.852 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.RVetRpzysm 2022-06-29T15:58:05.859 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2022-06-29T15:58:05.907 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.ZJRnjlLUjr 2022-06-29T15:58:05.908 DEBUG:teuthology.orchestra.run.smithi092:> sudo cp /var/lib/ceph/osd/ceph-6/keyring /tmp/tmp.ZJRnjlLUjr 2022-06-29T15:58:05.924 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.ZJRnjlLUjr 2022-06-29T15:58:05.988 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.ZJRnjlLUjr is 56B 2022-06-29T15:58:06.003 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.ZJRnjlLUjr 2022-06-29T15:58:06.010 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2022-06-29T15:58:06.018 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.AldQ0eMTH4 2022-06-29T15:58:06.018 DEBUG:teuthology.orchestra.run.smithi092:> sudo cp /var/lib/ceph/osd/ceph-7/keyring /tmp/tmp.AldQ0eMTH4 2022-06-29T15:58:06.079 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.AldQ0eMTH4 2022-06-29T15:58:06.103 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.AldQ0eMTH4 is 56B 2022-06-29T15:58:06.116 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.AldQ0eMTH4 2022-06-29T15:58:06.135 DEBUG:teuthology.orchestra.remote:smithi121:/etc/ceph/ceph.client.0.keyring is 59B 2022-06-29T15:58:06.163 DEBUG:teuthology.orchestra.remote:smithi156:/etc/ceph/ceph.client.1.keyring is 59B 2022-06-29T15:58:06.191 DEBUG:teuthology.orchestra.remote:smithi197:/etc/ceph/ceph.client.2.keyring is 59B 2022-06-29T15:58:06.215 INFO:tasks.ceph:Adding keys to all mons... 2022-06-29T15:58:06.216 DEBUG:teuthology.orchestra.run.smithi060:> sudo tee -a /etc/ceph/ceph.keyring 2022-06-29T15:58:06.218 DEBUG:teuthology.orchestra.run.smithi092:> sudo tee -a /etc/ceph/ceph.keyring 2022-06-29T15:58:06.231 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr.y] 2022-06-29T15:58:06.231 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQCGdrxi3LmaMBAAN+0mZrdKXqSXOipVTA1drw== 2022-06-29T15:58:06.232 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.a] 2022-06-29T15:58:06.232 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQCGdrxivv7JNhAAnA291UxlB/SfgITOipP00A== 2022-06-29T15:58:06.232 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.0] 2022-06-29T15:58:06.232 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQCHdrxiZYPxOBAAIkweWJlTJIXgxnlagkGdXA== 2022-06-29T15:58:06.233 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.1] 2022-06-29T15:58:06.233 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQCIdrxiS2JGBhAAJn0UfAxZI3IVFyD5QBdb/Q== 2022-06-29T15:58:06.233 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.2] 2022-06-29T15:58:06.233 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQCIdrxiPhkfEhAApH7r8OUJzW7fZNM6XVv1dg== 2022-06-29T15:58:06.234 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.3] 2022-06-29T15:58:06.234 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQCIdrxit/47HhAA1vyU6h7hjPG+op9qP/rVYQ== 2022-06-29T15:58:06.234 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr.x] 2022-06-29T15:58:06.235 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQCGdrxi2LZJNBAA3MKb/pXYbQCsoRx8EQfGbw== 2022-06-29T15:58:06.235 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.b] 2022-06-29T15:58:06.236 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQCGdrxiRSr/OhAAfFQTF2YHvzCv5wXBVCA5WQ== 2022-06-29T15:58:06.236 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.4] 2022-06-29T15:58:06.236 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQCLdrxiaKzKBxAAbIwCF9ag7ezsNy11hi0BEw== 2022-06-29T15:58:06.236 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.5] 2022-06-29T15:58:06.237 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQCLdrxiJQV/IBAASBBDb6UFgEtXILe4fcT/xA== 2022-06-29T15:58:06.237 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.6] 2022-06-29T15:58:06.237 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQCLdrxi6D07OBAA2FWiZ+mNK+jFMp/kPZnbQQ== 2022-06-29T15:58:06.237 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.7] 2022-06-29T15:58:06.238 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQCMdrxinEOcEhAA562ybpzGHNQfWIq8q7zw9w== 2022-06-29T15:58:06.238 INFO:teuthology.orchestra.run.smithi060.stdout:[client.0] 2022-06-29T15:58:06.238 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQCHdrxiHXPwBRAAkOOwlYJvIW7dqVFdlXXoqw== 2022-06-29T15:58:06.238 INFO:teuthology.orchestra.run.smithi060.stdout:[client.1] 2022-06-29T15:58:06.239 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQCHdrxibe5JCRAAGBJbi9t09w0836JqrAT7+w== 2022-06-29T15:58:06.239 INFO:teuthology.orchestra.run.smithi060.stdout:[client.2] 2022-06-29T15:58:06.239 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQCHdrxiK3dBDRAA7XINPSOPQDfOrUYEpHC+gw== 2022-06-29T15:58:06.241 INFO:teuthology.orchestra.run.smithi092.stdout:[mgr.y] 2022-06-29T15:58:06.241 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCGdrxi3LmaMBAAN+0mZrdKXqSXOipVTA1drw== 2022-06-29T15:58:06.242 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.a] 2022-06-29T15:58:06.242 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCGdrxivv7JNhAAnA291UxlB/SfgITOipP00A== 2022-06-29T15:58:06.242 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.0] 2022-06-29T15:58:06.242 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCHdrxiZYPxOBAAIkweWJlTJIXgxnlagkGdXA== 2022-06-29T15:58:06.243 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.1] 2022-06-29T15:58:06.243 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCIdrxiS2JGBhAAJn0UfAxZI3IVFyD5QBdb/Q== 2022-06-29T15:58:06.243 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.2] 2022-06-29T15:58:06.243 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCIdrxiPhkfEhAApH7r8OUJzW7fZNM6XVv1dg== 2022-06-29T15:58:06.244 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.3] 2022-06-29T15:58:06.244 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCIdrxit/47HhAA1vyU6h7hjPG+op9qP/rVYQ== 2022-06-29T15:58:06.244 INFO:teuthology.orchestra.run.smithi092.stdout:[mgr.x] 2022-06-29T15:58:06.244 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCGdrxi2LZJNBAA3MKb/pXYbQCsoRx8EQfGbw== 2022-06-29T15:58:06.244 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.b] 2022-06-29T15:58:06.245 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCGdrxiRSr/OhAAfFQTF2YHvzCv5wXBVCA5WQ== 2022-06-29T15:58:06.245 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.4] 2022-06-29T15:58:06.245 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCLdrxiaKzKBxAAbIwCF9ag7ezsNy11hi0BEw== 2022-06-29T15:58:06.245 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.5] 2022-06-29T15:58:06.246 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCLdrxiJQV/IBAASBBDb6UFgEtXILe4fcT/xA== 2022-06-29T15:58:06.246 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.6] 2022-06-29T15:58:06.246 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCLdrxi6D07OBAA2FWiZ+mNK+jFMp/kPZnbQQ== 2022-06-29T15:58:06.246 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.7] 2022-06-29T15:58:06.247 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCMdrxinEOcEhAA562ybpzGHNQfWIq8q7zw9w== 2022-06-29T15:58:06.247 INFO:teuthology.orchestra.run.smithi092.stdout:[client.0] 2022-06-29T15:58:06.247 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCHdrxiHXPwBRAAkOOwlYJvIW7dqVFdlXXoqw== 2022-06-29T15:58:06.247 INFO:teuthology.orchestra.run.smithi092.stdout:[client.1] 2022-06-29T15:58:06.247 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCHdrxibe5JCRAAGBJbi9t09w0836JqrAT7+w== 2022-06-29T15:58:06.248 INFO:teuthology.orchestra.run.smithi092.stdout:[client.2] 2022-06-29T15:58:06.248 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCHdrxiK3dBDRAA7XINPSOPQDfOrUYEpHC+gw== 2022-06-29T15:58:06.248 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-06-29T15:58:06.287 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-06-29T15:58:06.334 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-06-29T15:58:06.337 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-06-29T15:58:06.378 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-06-29T15:58:06.419 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-06-29T15:58:06.462 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-06-29T15:58:06.465 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-06-29T15:58:06.507 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-06-29T15:58:06.511 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-06-29T15:58:06.552 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-06-29T15:58:06.595 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-06-29T15:58:06.638 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-06-29T15:58:06.641 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-06-29T15:58:06.685 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-06-29T15:58:06.727 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-06-29T15:58:06.772 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-06-29T15:58:06.775 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-06-29T15:58:06.855 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-06-29T15:58:06.857 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-06-29T15:58:06.899 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-06-29T15:58:06.943 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-06-29T15:58:06.986 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-06-29T15:58:06.989 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-06-29T15:58:07.030 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-06-29T15:58:07.071 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-06-29T15:58:07.117 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-06-29T15:58:07.120 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-06-29T15:58:07.163 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-06-29T15:58:07.166 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-06-29T15:58:07.211 INFO:tasks.ceph:Running mkfs on mon nodes... 2022-06-29T15:58:07.211 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2022-06-29T15:58:07.229 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2022-06-29T15:58:07.351 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2022-06-29T15:58:07.369 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2022-06-29T15:58:07.387 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2022-06-29T15:58:07.445 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2022-06-29T15:58:07.460 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2022-06-29T15:58:07.517 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2022-06-29T15:58:07.578 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2022-06-29T15:58:07.592 DEBUG:teuthology.orchestra.run.smithi060:> rm -- /home/ubuntu/cephtest/ceph.monmap 2022-06-29T15:58:07.594 DEBUG:teuthology.orchestra.run.smithi092:> rm -- /home/ubuntu/cephtest/ceph.monmap 2022-06-29T15:58:07.599 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2022-06-29T15:58:07.600 INFO:tasks.ceph.mon.a:Restarting daemon 2022-06-29T15:58:07.601 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2022-06-29T15:58:07.643 INFO:tasks.ceph.mon.a:Started 2022-06-29T15:58:07.643 INFO:tasks.ceph.mon.b:Restarting daemon 2022-06-29T15:58:07.644 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2022-06-29T15:58:07.646 INFO:tasks.ceph.mon.b:Started 2022-06-29T15:58:07.647 INFO:tasks.ceph.mon.c:Restarting daemon 2022-06-29T15:58:07.647 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2022-06-29T15:58:07.649 INFO:tasks.ceph.mon.c:Started 2022-06-29T15:58:07.650 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2022-06-29T15:58:07.650 INFO:tasks.ceph.mgr.y:Restarting daemon 2022-06-29T15:58:07.651 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2022-06-29T15:58:07.653 INFO:tasks.ceph.mgr.y:Started 2022-06-29T15:58:07.654 INFO:tasks.ceph.mgr.x:Restarting daemon 2022-06-29T15:58:07.654 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2022-06-29T15:58:07.657 INFO:tasks.ceph.mgr.x:Started 2022-06-29T15:58:07.657 INFO:tasks.ceph:Setting crush tunables to default 2022-06-29T15:58:07.657 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph osd crush tunables default 2022-06-29T15:58:07.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.781+0000 7fccc29e1700 1 Processor -- start 2022-06-29T15:58:07.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.781+0000 7fccc29e1700 1 -- start start 2022-06-29T15:58:07.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.785+0000 7fccc29e1700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fccbc0b61d0 0x7fccbc0b0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:07.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.785+0000 7fccc29e1700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fccbc13f790 0x7fccbc13bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:07.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.785+0000 7fccc29e1700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fccbc13fc00 0x7fccbc13c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:07.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.785+0000 7fccc29e1700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccbc142820 con 0x7fccbc0b61d0 2022-06-29T15:58:07.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.785+0000 7fccc29e1700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccbc062bb0 con 0x7fccbc13f790 2022-06-29T15:58:07.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.785+0000 7fccc29e1700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccbc0b65b0 con 0x7fccbc13fc00 2022-06-29T15:58:07.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.785+0000 7fccbbfff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fccbc0b61d0 0x7fccbc0b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:07.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.785+0000 7fccb3fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fccbc13f790 0x7fccbc13bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:07.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.785+0000 7fccbbfff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fccbc0b61d0 0x7fccbc0b0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44458/0 (socket says 172.21.15.60:44458) 2022-06-29T15:58:07.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.785+0000 7fccbbfff700 1 -- 172.21.15.60:0/1552449561 learned_addr learned my addr 172.21.15.60:0/1552449561 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:07.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.785+0000 7fccc0f7e700 1 --2- 172.21.15.60:0/1552449561 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fccbc13fc00 0x7fccbc13c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:07.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.785+0000 7fccb3fff700 1 -- 172.21.15.60:0/1552449561 >> v2:172.21.15.92:3300/0 conn(0x7fccbc13f790 msgr2=0x7fccbc13bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2022-06-29T15:58:07.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.785+0000 7fccb3fff700 1 -- 172.21.15.60:0/1552449561 >> v2:172.21.15.92:3300/0 conn(0x7fccbc13f790 msgr2=0x7fccbc13bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-06-29T15:58:07.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.785+0000 7fccb3fff700 1 --2- 172.21.15.60:0/1552449561 >> v2:172.21.15.92:3300/0 conn(0x7fccbc13f790 0x7fccbc13bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-06-29T15:58:07.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.785+0000 7fccb3fff700 1 --2- 172.21.15.60:0/1552449561 >> v2:172.21.15.92:3300/0 conn(0x7fccbc13f790 0x7fccbc13bc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2022-06-29T15:58:07.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.785+0000 7fccc0f7e700 1 -- 172.21.15.60:0/1552449561 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fccbc13fc00 msgr2=0x7fccbc13c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2022-06-29T15:58:07.797 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.785+0000 7fccc0f7e700 1 -- 172.21.15.60:0/1552449561 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fccbc13fc00 msgr2=0x7fccbc13c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-06-29T15:58:07.797 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.785+0000 7fccc0f7e700 1 --2- 172.21.15.60:0/1552449561 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fccbc13fc00 0x7fccbc13c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-06-29T15:58:07.797 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.785+0000 7fccc0f7e700 1 --2- 172.21.15.60:0/1552449561 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fccbc13fc00 0x7fccbc13c130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2022-06-29T15:58:07.797 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.785+0000 7fccbbfff700 1 -- 172.21.15.60:0/1552449561 >> v2:172.21.15.60:3300/0 conn(0x7fccbc0b61d0 msgr2=0x7fccbc0b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2022-06-29T15:58:07.797 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.785+0000 7fccbbfff700 1 -- 172.21.15.60:0/1552449561 >> v2:172.21.15.60:3300/0 conn(0x7fccbc0b61d0 msgr2=0x7fccbc0b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-06-29T15:58:07.798 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.785+0000 7fccbbfff700 1 --2- 172.21.15.60:0/1552449561 >> v2:172.21.15.60:3300/0 conn(0x7fccbc0b61d0 0x7fccbc0b0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-06-29T15:58:07.798 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.785+0000 7fccbbfff700 1 --2- 172.21.15.60:0/1552449561 >> v2:172.21.15.60:3300/0 conn(0x7fccbc0b61d0 0x7fccbc0b0540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2022-06-29T15:58:07.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.989+0000 7fccbbfff700 1 --2- 172.21.15.60:0/1552449561 >> v2:172.21.15.60:3300/0 conn(0x7fccbc0b61d0 0x7fccbc0b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:07.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.989+0000 7fccb3fff700 1 --2- 172.21.15.60:0/1552449561 >> v2:172.21.15.92:3300/0 conn(0x7fccbc13f790 0x7fccbc13bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:07.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.989+0000 7fccc0f7e700 1 --2- 172.21.15.60:0/1552449561 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fccbc13fc00 0x7fccbc13c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:07.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.989+0000 7fccb3fff700 1 -- 172.21.15.60:0/1552449561 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fccbc13fc00 msgr2=0x7fccbc13c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:07.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.989+0000 7fccb3fff700 1 --2- 172.21.15.60:0/1552449561 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fccbc13fc00 0x7fccbc13c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:07.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.989+0000 7fccb3fff700 1 -- 172.21.15.60:0/1552449561 >> v2:172.21.15.60:3300/0 conn(0x7fccbc0b61d0 msgr2=0x7fccbc0b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:07.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.989+0000 7fccb3fff700 1 --2- 172.21.15.60:0/1552449561 >> v2:172.21.15.60:3300/0 conn(0x7fccbc0b61d0 0x7fccbc0b0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:07.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.989+0000 7fccb3fff700 1 -- 172.21.15.60:0/1552449561 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccbc13f050 con 0x7fccbc13f790 2022-06-29T15:58:07.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.989+0000 7fccb3fff700 1 --2- 172.21.15.60:0/1552449561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fccbc13f790 0x7fccbc13bc40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fccbc0acf50 tx=0x7fccac006f50).ready entity=mon.1 client_cookie=a9631a10acfeb535 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:07.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.989+0000 7fccbb7fe700 1 -- 172.21.15.60:0/1552449561 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fccac00f620 con 0x7fccbc13f790 2022-06-29T15:58:07.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.989+0000 7fccbb7fe700 1 -- 172.21.15.60:0/1552449561 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccac00f780 con 0x7fccbc13f790 2022-06-29T15:58:07.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.989+0000 7fccbb7fe700 1 -- 172.21.15.60:0/1552449561 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fccac00fae0 con 0x7fccbc13f790 2022-06-29T15:58:08.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.989+0000 7fccc29e1700 1 -- 172.21.15.60:0/1552449561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fccbc13f790 msgr2=0x7fccbc13bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:08.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.989+0000 7fccc29e1700 1 --2- 172.21.15.60:0/1552449561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fccbc13f790 0x7fccbc13bc40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fccbc0acf50 tx=0x7fccac006f50).stop 2022-06-29T15:58:08.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.989+0000 7fccc29e1700 1 -- 172.21.15.60:0/1552449561 shutdown_connections 2022-06-29T15:58:08.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.989+0000 7fccc29e1700 1 --2- 172.21.15.60:0/1552449561 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fccbc13fc00 0x7fccbc13c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:08.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.989+0000 7fccc29e1700 1 --2- 172.21.15.60:0/1552449561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fccbc13f790 0x7fccbc13bc40 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:08.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.989+0000 7fccc29e1700 1 --2- 172.21.15.60:0/1552449561 >> v2:172.21.15.60:3300/0 conn(0x7fccbc0b61d0 0x7fccbc0b0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:08.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.989+0000 7fccc29e1700 1 -- 172.21.15.60:0/1552449561 >> 172.21.15.60:0/1552449561 conn(0x7fccbc063b50 msgr2=0x7fccbc0b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:08.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.993+0000 7fccc29e1700 1 -- 172.21.15.60:0/1552449561 shutdown_connections 2022-06-29T15:58:08.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.993+0000 7fccc29e1700 1 -- 172.21.15.60:0/1552449561 wait complete. 2022-06-29T15:58:08.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.993+0000 7fccc29e1700 1 Processor -- start 2022-06-29T15:58:08.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.993+0000 7fccc29e1700 1 -- start start 2022-06-29T15:58:08.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.993+0000 7fccc29e1700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fccbc0b61d0 0x7fccbc13db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:08.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.993+0000 7fccc29e1700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fccbc13f790 0x7fccbc13b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:08.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.993+0000 7fccc29e1700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fccbc13fc00 0x7fccbc13b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:08.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.993+0000 7fccc29e1700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccbc1132b0 con 0x7fccbc13f790 2022-06-29T15:58:08.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.993+0000 7fccc29e1700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccbc113550 con 0x7fccbc13fc00 2022-06-29T15:58:08.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.993+0000 7fccc29e1700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccbc1137f0 con 0x7fccbc0b61d0 2022-06-29T15:58:08.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.993+0000 7fccbbfff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fccbc0b61d0 0x7fccbc13db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:08.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.997+0000 7fccbbfff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fccbc0b61d0 0x7fccbc13db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:57576/0 (socket says 172.21.15.60:57576) 2022-06-29T15:58:08.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.997+0000 7fccbbfff700 1 -- 172.21.15.60:0/2619355066 learned_addr learned my addr 172.21.15.60:0/2619355066 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:08.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.997+0000 7fccb3fff700 1 --2- 172.21.15.60:0/2619355066 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fccbc13f790 0x7fccbc13b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:08.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.997+0000 7fccc0f7e700 1 --2- 172.21.15.60:0/2619355066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fccbc13fc00 0x7fccbc13b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:08.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.997+0000 7fccbbfff700 1 -- 172.21.15.60:0/2619355066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fccbc13fc00 msgr2=0x7fccbc13b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:08.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.997+0000 7fccbbfff700 1 --2- 172.21.15.60:0/2619355066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fccbc13fc00 0x7fccbc13b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:08.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.997+0000 7fccbbfff700 1 -- 172.21.15.60:0/2619355066 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fccbc13f790 msgr2=0x7fccbc13b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:08.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.997+0000 7fccbbfff700 1 --2- 172.21.15.60:0/2619355066 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fccbc13f790 0x7fccbc13b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:08.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.997+0000 7fccbbfff700 1 -- 172.21.15.60:0/2619355066 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccac00c3e0 con 0x7fccbc0b61d0 2022-06-29T15:58:08.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.997+0000 7fccbbfff700 1 --2- 172.21.15.60:0/2619355066 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fccbc0b61d0 0x7fccbc13db80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fcca400e000 tx=0x7fcca4008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:08.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.997+0000 7fccb9ffb700 1 -- 172.21.15.60:0/2619355066 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcca400daa0 con 0x7fccbc0b61d0 2022-06-29T15:58:08.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.997+0000 7fccb9ffb700 1 -- 172.21.15.60:0/2619355066 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca400dc00 con 0x7fccbc0b61d0 2022-06-29T15:58:08.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.997+0000 7fccc29e1700 1 -- 172.21.15.60:0/2619355066 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccbc063fd0 con 0x7fccbc0b61d0 2022-06-29T15:58:08.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.997+0000 7fccb9ffb700 1 -- 172.21.15.60:0/2619355066 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcca4011450 con 0x7fccbc0b61d0 2022-06-29T15:58:08.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.997+0000 7fccc29e1700 1 -- 172.21.15.60:0/2619355066 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccbc13ba00 con 0x7fccbc0b61d0 2022-06-29T15:58:08.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.997+0000 7fccb9ffb700 1 -- 172.21.15.60:0/2619355066 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 1) v1 ==== 345+0+0 (secure 0 0 0) 0x7fcca40061f0 con 0x7fccbc0b61d0 2022-06-29T15:58:08.016 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.997+0000 7fccb9ffb700 1 -- 172.21.15.60:0/2619355066 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fcca4005b30 con 0x7fccbc0b61d0 2022-06-29T15:58:08.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:07.997+0000 7fccc29e1700 1 -- 172.21.15.60:0/2619355066 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccbc13f2f0 con 0x7fccbc0b61d0 2022-06-29T15:58:08.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:08.001+0000 7fccb9ffb700 1 -- 172.21.15.60:0/2619355066 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67674 (secure 0 0 0) 0x7fcca4005cf0 con 0x7fccbc0b61d0 2022-06-29T15:58:08.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:08.065+0000 7fccc29e1700 1 -- 172.21.15.60:0/2619355066 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fccbc0573d0 con 0x7fccbc0b61d0 2022-06-29T15:58:08.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:08.073+0000 7fccb9ffb700 1 -- 172.21.15.60:0/2619355066 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fcca40115b0 con 0x7fccbc0b61d0 2022-06-29T15:58:08.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:08.073+0000 7fccc29e1700 1 -- 172.21.15.60:0/2619355066 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fccbc0b61d0 msgr2=0x7fccbc13db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:08.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:08.073+0000 7fccc29e1700 1 --2- 172.21.15.60:0/2619355066 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fccbc0b61d0 0x7fccbc13db80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fcca400e000 tx=0x7fcca4008040).stop 2022-06-29T15:58:08.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:08.073+0000 7fccc29e1700 1 -- 172.21.15.60:0/2619355066 shutdown_connections 2022-06-29T15:58:08.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:08.073+0000 7fccc29e1700 1 --2- 172.21.15.60:0/2619355066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fccbc13fc00 0x7fccbc13b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:08.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:08.073+0000 7fccc29e1700 1 --2- 172.21.15.60:0/2619355066 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fccbc13f790 0x7fccbc13b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:08.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:08.073+0000 7fccc29e1700 1 --2- 172.21.15.60:0/2619355066 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fccbc0b61d0 0x7fccbc13db80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:08.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:08.073+0000 7fccc29e1700 1 -- 172.21.15.60:0/2619355066 >> 172.21.15.60:0/2619355066 conn(0x7fccbc063b50 msgr2=0x7fccbc0b5690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:08.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:08.073+0000 7fccc29e1700 1 -- 172.21.15.60:0/2619355066 shutdown_connections 2022-06-29T15:58:08.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:08.073+0000 7fccc29e1700 1 -- 172.21.15.60:0/2619355066 wait complete. 2022-06-29T15:58:08.083 INFO:teuthology.orchestra.run.smithi060.stderr:adjusted tunables profile to default 2022-06-29T15:58:08.093 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2022-06-29T15:58:08.093 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2022-06-29T15:58:08.098 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.RSFkOOklWr 2022-06-29T15:58:08.099 DEBUG:teuthology.orchestra.run.smithi060:> sudo cp /var/lib/ceph/osd/ceph-0/fsid /tmp/tmp.RSFkOOklWr 2022-06-29T15:58:08.112 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.RSFkOOklWr 2022-06-29T15:58:08.133 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.RSFkOOklWr is 37B 2022-06-29T15:58:08.183 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.RSFkOOklWr 2022-06-29T15:58:08.191 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2022-06-29T15:58:08.239 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.VfX4F3If6k 2022-06-29T15:58:08.240 DEBUG:teuthology.orchestra.run.smithi060:> sudo cp /var/lib/ceph/osd/ceph-1/fsid /tmp/tmp.VfX4F3If6k 2022-06-29T15:58:08.256 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.VfX4F3If6k 2022-06-29T15:58:08.320 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.VfX4F3If6k is 37B 2022-06-29T15:58:08.335 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.VfX4F3If6k 2022-06-29T15:58:08.341 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2022-06-29T15:58:08.387 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.zn0UzBsQjU 2022-06-29T15:58:08.387 DEBUG:teuthology.orchestra.run.smithi060:> sudo cp /var/lib/ceph/osd/ceph-2/fsid /tmp/tmp.zn0UzBsQjU 2022-06-29T15:58:08.405 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.zn0UzBsQjU 2022-06-29T15:58:08.428 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.zn0UzBsQjU is 37B 2022-06-29T15:58:08.441 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.zn0UzBsQjU 2022-06-29T15:58:08.448 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2022-06-29T15:58:08.495 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.R6jefA2cW5 2022-06-29T15:58:08.496 DEBUG:teuthology.orchestra.run.smithi060:> sudo cp /var/lib/ceph/osd/ceph-3/fsid /tmp/tmp.R6jefA2cW5 2022-06-29T15:58:08.510 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.R6jefA2cW5 2022-06-29T15:58:08.580 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.R6jefA2cW5 is 37B 2022-06-29T15:58:08.595 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.R6jefA2cW5 2022-06-29T15:58:08.601 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2022-06-29T15:58:08.609 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.HTh6IBSL0b 2022-06-29T15:58:08.609 DEBUG:teuthology.orchestra.run.smithi092:> sudo cp /var/lib/ceph/osd/ceph-4/fsid /tmp/tmp.HTh6IBSL0b 2022-06-29T15:58:08.674 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.HTh6IBSL0b 2022-06-29T15:58:08.743 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.HTh6IBSL0b is 37B 2022-06-29T15:58:08.757 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.HTh6IBSL0b 2022-06-29T15:58:08.764 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2022-06-29T15:58:08.810 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.tTbAzpzuNn 2022-06-29T15:58:08.811 DEBUG:teuthology.orchestra.run.smithi092:> sudo cp /var/lib/ceph/osd/ceph-5/fsid /tmp/tmp.tTbAzpzuNn 2022-06-29T15:58:08.827 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.tTbAzpzuNn 2022-06-29T15:58:08.892 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.tTbAzpzuNn is 37B 2022-06-29T15:58:08.906 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.tTbAzpzuNn 2022-06-29T15:58:08.912 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2022-06-29T15:58:08.917 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.AFTWLOF6k8 2022-06-29T15:58:08.918 DEBUG:teuthology.orchestra.run.smithi092:> sudo cp /var/lib/ceph/osd/ceph-6/fsid /tmp/tmp.AFTWLOF6k8 2022-06-29T15:58:08.978 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.AFTWLOF6k8 2022-06-29T15:58:09.003 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.AFTWLOF6k8 is 37B 2022-06-29T15:58:09.015 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.AFTWLOF6k8 2022-06-29T15:58:09.022 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2022-06-29T15:58:09.070 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.zm99L6rOoQ 2022-06-29T15:58:09.071 DEBUG:teuthology.orchestra.run.smithi092:> sudo cp /var/lib/ceph/osd/ceph-7/fsid /tmp/tmp.zm99L6rOoQ 2022-06-29T15:58:09.084 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.zm99L6rOoQ 2022-06-29T15:58:09.149 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.zm99L6rOoQ is 37B 2022-06-29T15:58:09.163 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.zm99L6rOoQ 2022-06-29T15:58:09.170 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new 11e7ab9e-fce2-470f-8195-ea4c3c52df22 0 2022-06-29T15:58:09.312 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.308+0000 7f7e6a32f700 1 Processor -- start 2022-06-29T15:58:09.313 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.308+0000 7f7e6a32f700 1 -- start start 2022-06-29T15:58:09.315 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.312+0000 7f7e6a32f700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7e640afa00 0x7f7e6413bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:09.315 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.312+0000 7f7e6a32f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7e6413ef20 0x7f7e6413f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:09.315 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.312+0000 7f7e6a32f700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e640afde0 con 0x7f7e640b1650 2022-06-29T15:58:09.316 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.312+0000 7f7e6a32f700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e640ae6a0 con 0x7f7e640afa00 2022-06-29T15:58:09.316 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.312+0000 7f7e6a32f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e64142900 con 0x7f7e6413ef20 2022-06-29T15:58:09.316 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.312+0000 7f7e688cc700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7e6413ef20 0x7f7e6413f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:09.316 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.312+0000 7f7e637fe700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f7e640b1650 0x7f7e6413c150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:49930/0 (socket says 172.21.15.92:49930) 2022-06-29T15:58:09.317 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.312+0000 7f7e637fe700 1 -- 172.21.15.92:0/29067613 learned_addr learned my addr 172.21.15.92:0/29067613 (peer_addr_for_me v1:172.21.15.92:0/0) 2022-06-29T15:58:09.317 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.312+0000 7f7e63fff700 1 --2- 172.21.15.92:0/29067613 >> v2:172.21.15.60:3300/0 conn(0x7f7e640afa00 0x7f7e6413bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:09.318 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.312+0000 7f7e62ffd700 1 -- 172.21.15.92:0/29067613 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96225541 0 0) 0x7f7e640afde0 con 0x7f7e640b1650 2022-06-29T15:58:09.318 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.312+0000 7f7e62ffd700 1 -- 172.21.15.92:0/29067613 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e50004370 con 0x7f7e640b1650 2022-06-29T15:58:09.318 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.312+0000 7f7e688cc700 1 -- 172.21.15.92:0/29067613 >> v2:172.21.15.60:3300/0 conn(0x7f7e640afa00 msgr2=0x7f7e6413bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:09.325 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.312+0000 7f7e688cc700 1 --2- 172.21.15.92:0/29067613 >> v2:172.21.15.60:3300/0 conn(0x7f7e640afa00 0x7f7e6413bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:09.326 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.312+0000 7f7e688cc700 1 -- 172.21.15.92:0/29067613 >> v1:172.21.15.92:6789/0 conn(0x7f7e640b1650 legacy=0x7f7e6413c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:09.326 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.312+0000 7f7e688cc700 1 -- 172.21.15.92:0/29067613 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e64142ba0 con 0x7f7e6413ef20 2022-06-29T15:58:09.326 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.312+0000 7f7e688cc700 1 --2- 172.21.15.92:0/29067613 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7e6413ef20 0x7f7e6413f390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f7e5c0032f0 tx=0x7f7e5c0091d0).ready entity=mon.2 client_cookie=2f17d8fed0efea51 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:09.326 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.312+0000 7f7e62ffd700 1 -- 172.21.15.92:0/29067613 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e5c00b600 con 0x7f7e6413ef20 2022-06-29T15:58:09.327 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.312+0000 7f7e62ffd700 1 -- 172.21.15.92:0/29067613 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e5c00b760 con 0x7f7e6413ef20 2022-06-29T15:58:09.327 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.312+0000 7f7e62ffd700 1 -- 172.21.15.92:0/29067613 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e5c00b8d0 con 0x7f7e6413ef20 2022-06-29T15:58:09.327 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.312+0000 7f7e6a32f700 1 -- 172.21.15.92:0/29067613 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7e6413ef20 msgr2=0x7f7e6413f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:09.328 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.312+0000 7f7e6a32f700 1 --2- 172.21.15.92:0/29067613 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7e6413ef20 0x7f7e6413f390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f7e5c0032f0 tx=0x7f7e5c0091d0).stop 2022-06-29T15:58:09.328 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.312+0000 7f7e6a32f700 1 -- 172.21.15.92:0/29067613 shutdown_connections 2022-06-29T15:58:09.328 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.312+0000 7f7e6a32f700 1 --2- 172.21.15.92:0/29067613 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7e6413ef20 0x7f7e6413f390 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:09.328 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.312+0000 7f7e6a32f700 1 --2- 172.21.15.92:0/29067613 >> v2:172.21.15.60:3300/0 conn(0x7f7e640afa00 0x7f7e6413bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:09.329 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.312+0000 7f7e6a32f700 1 -- 172.21.15.92:0/29067613 >> 172.21.15.92:0/29067613 conn(0x7f7e6405f0b0 msgr2=0x7f7e6405f490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:09.329 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.312+0000 7f7e6a32f700 1 -- 172.21.15.92:0/29067613 shutdown_connections 2022-06-29T15:58:09.329 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.312+0000 7f7e6a32f700 1 -- 172.21.15.92:0/29067613 wait complete. 2022-06-29T15:58:09.329 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.316+0000 7f7e6a32f700 1 Processor -- start 2022-06-29T15:58:09.330 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.316+0000 7f7e6a32f700 1 -- start start 2022-06-29T15:58:09.330 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.316+0000 7f7e6a32f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e640afa00 0x7f7e64066ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:09.330 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.316+0000 7f7e6a32f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e640b1650 0x7f7e6413c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:09.330 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.316+0000 7f7e6a32f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7e6413ef20 0x7f7e6414fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:09.331 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.316+0000 7f7e6a32f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e64067390 con 0x7f7e640afa00 2022-06-29T15:58:09.331 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.316+0000 7f7e6a32f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e640657a0 con 0x7f7e640b1650 2022-06-29T15:58:09.331 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.316+0000 7f7e6a32f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e64065a40 con 0x7f7e6413ef20 2022-06-29T15:58:09.331 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.316+0000 7f7e688cc700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7e6413ef20 0x7f7e6414fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:09.332 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.316+0000 7f7e637fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e640b1650 0x7f7e6413c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:09.332 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.316+0000 7f7e688cc700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7e6413ef20 0x7f7e6414fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.92:59928/0 (socket says 172.21.15.92:59928) 2022-06-29T15:58:09.332 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.316+0000 7f7e688cc700 1 -- 172.21.15.92:0/3286191837 learned_addr learned my addr 172.21.15.92:0/3286191837 (peer_addr_for_me v2:172.21.15.92:0/0) 2022-06-29T15:58:09.332 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.316+0000 7f7e63fff700 1 --2- 172.21.15.92:0/3286191837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e640afa00 0x7f7e64066ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:09.332 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.316+0000 7f7e688cc700 1 -- 172.21.15.92:0/3286191837 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e640b1650 msgr2=0x7f7e6413c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:09.333 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.316+0000 7f7e688cc700 1 --2- 172.21.15.92:0/3286191837 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e640b1650 0x7f7e6413c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:09.333 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.316+0000 7f7e688cc700 1 -- 172.21.15.92:0/3286191837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e640afa00 msgr2=0x7f7e64066ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:09.333 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.316+0000 7f7e688cc700 1 --2- 172.21.15.92:0/3286191837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e640afa00 0x7f7e64066ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:09.333 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.316+0000 7f7e688cc700 1 -- 172.21.15.92:0/3286191837 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e5c009470 con 0x7f7e6413ef20 2022-06-29T15:58:09.334 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.316+0000 7f7e688cc700 1 --2- 172.21.15.92:0/3286191837 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7e6413ef20 0x7f7e6414fdf0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f7e5c0032f0 tx=0x7f7e5c016000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:09.334 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.316+0000 7f7e617fa700 1 -- 172.21.15.92:0/3286191837 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e5c00b600 con 0x7f7e6413ef20 2022-06-29T15:58:09.334 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.316+0000 7f7e617fa700 1 -- 172.21.15.92:0/3286191837 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e5c011e20 con 0x7f7e6413ef20 2022-06-29T15:58:09.334 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.316+0000 7f7e6a32f700 1 -- 172.21.15.92:0/3286191837 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e64065ce0 con 0x7f7e6413ef20 2022-06-29T15:58:09.335 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.316+0000 7f7e617fa700 1 -- 172.21.15.92:0/3286191837 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e5c0205f0 con 0x7f7e6413ef20 2022-06-29T15:58:09.335 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.316+0000 7f7e6a32f700 1 -- 172.21.15.92:0/3286191837 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e6414f680 con 0x7f7e6413ef20 2022-06-29T15:58:09.335 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.320+0000 7f7e617fa700 1 -- 172.21.15.92:0/3286191837 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 1) v1 ==== 345+0+0 (secure 0 0 0) 0x7f7e5c027030 con 0x7f7e6413ef20 2022-06-29T15:58:09.335 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.320+0000 7f7e617fa700 1 -- 172.21.15.92:0/3286191837 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 754+0+0 (secure 0 0 0) 0x7f7e5c02a7b0 con 0x7f7e6413ef20 2022-06-29T15:58:09.336 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.320+0000 7f7e6a32f700 1 -- 172.21.15.92:0/3286191837 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e6410f1b0 con 0x7f7e6413ef20 2022-06-29T15:58:09.336 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.324+0000 7f7e617fa700 1 -- 172.21.15.92:0/3286191837 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67674 (secure 0 0 0) 0x7f7e6410f1b0 con 0x7f7e6413ef20 2022-06-29T15:58:09.393 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.392+0000 7f7e6a32f700 1 -- 172.21.15.92:0/3286191837 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd new", "uuid": "11e7ab9e-fce2-470f-8195-ea4c3c52df22", "id": 0} v 0) v1 -- 0x7f7e64152ca0 con 0x7f7e6413ef20 2022-06-29T15:58:09.403 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.400+0000 7f7e617fa700 1 -- 172.21.15.92:0/3286191837 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "11e7ab9e-fce2-470f-8195-ea4c3c52df22", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7e64152ca0 con 0x7f7e6413ef20 2022-06-29T15:58:09.403 INFO:teuthology.orchestra.run.smithi092.stdout:0 2022-06-29T15:58:09.405 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.400+0000 7f7e6a32f700 1 -- 172.21.15.92:0/3286191837 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7e6413ef20 msgr2=0x7f7e6414fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:09.405 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.400+0000 7f7e6a32f700 1 --2- 172.21.15.92:0/3286191837 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7e6413ef20 0x7f7e6414fdf0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f7e5c0032f0 tx=0x7f7e5c016000).stop 2022-06-29T15:58:09.405 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.400+0000 7f7e6a32f700 1 -- 172.21.15.92:0/3286191837 shutdown_connections 2022-06-29T15:58:09.405 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.400+0000 7f7e6a32f700 1 --2- 172.21.15.92:0/3286191837 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7e6413ef20 0x7f7e6414fdf0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:09.406 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.400+0000 7f7e6a32f700 1 --2- 172.21.15.92:0/3286191837 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e640b1650 0x7f7e6413c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:09.406 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.400+0000 7f7e6a32f700 1 --2- 172.21.15.92:0/3286191837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e640afa00 0x7f7e64066ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:09.406 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.400+0000 7f7e6a32f700 1 -- 172.21.15.92:0/3286191837 >> 172.21.15.92:0/3286191837 conn(0x7f7e6405f0b0 msgr2=0x7f7e6405f490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:09.406 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.400+0000 7f7e6a32f700 1 -- 172.21.15.92:0/3286191837 shutdown_connections 2022-06-29T15:58:09.407 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.400+0000 7f7e6a32f700 1 -- 172.21.15.92:0/3286191837 wait complete. 2022-06-29T15:58:09.418 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new ca43f136-6f69-4431-a157-fbe073fb585c 1 2022-06-29T15:58:09.518 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.516+0000 7fb950c14700 1 Processor -- start 2022-06-29T15:58:09.519 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.516+0000 7fb950c14700 1 -- start start 2022-06-29T15:58:09.521 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.516+0000 7fb950c14700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb94c13fbf0 0x7fb94c140020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:09.521 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.516+0000 7fb950c14700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb94c0ad140 con 0x7fb94c0ad520 2022-06-29T15:58:09.521 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.516+0000 7fb950c14700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb94c0b24f0 con 0x7fb94c0acd60 2022-06-29T15:58:09.522 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.516+0000 7fb950c14700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb94c0649e0 con 0x7fb94c13fbf0 2022-06-29T15:58:09.522 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.516+0000 7fb94b7fe700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fb94c0acd60 0x7fb94c143e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:49940/0 (socket says 172.21.15.92:49940) 2022-06-29T15:58:09.522 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.516+0000 7fb94b7fe700 1 -- 172.21.15.92:0/1667549557 learned_addr learned my addr 172.21.15.92:0/1667549557 (peer_addr_for_me v1:172.21.15.92:0/0) 2022-06-29T15:58:09.522 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.516+0000 7fb94bfff700 1 --2- 172.21.15.92:0/1667549557 >> v2:172.21.15.60:3300/0 conn(0x7fb94c13fbf0 0x7fb94c140020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:09.523 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.516+0000 7fb94a7fc700 1 -- 172.21.15.92:0/1667549557 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 391900540 0 0) 0x7fb94c0b24f0 con 0x7fb94c0acd60 2022-06-29T15:58:09.523 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.516+0000 7fb94a7fc700 1 -- 172.21.15.92:0/1667549557 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb938003260 con 0x7fb94c0acd60 2022-06-29T15:58:09.523 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.516+0000 7fb94bfff700 1 -- 172.21.15.92:0/1667549557 >> v1:172.21.15.92:6789/0 conn(0x7fb94c0acd60 legacy=0x7fb94c143e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:09.523 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.516+0000 7fb94bfff700 1 -- 172.21.15.92:0/1667549557 >> v1:172.21.15.60:6789/0 conn(0x7fb94c0ad520 legacy=0x7fb94c065ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:09.524 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.516+0000 7fb94bfff700 1 -- 172.21.15.92:0/1667549557 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb94c065640 con 0x7fb94c13fbf0 2022-06-29T15:58:09.524 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb94bfff700 1 --2- 172.21.15.92:0/1667549557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb94c13fbf0 0x7fb94c140020 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fb9440032f0 tx=0x7fb9440091d0).ready entity=mon.0 client_cookie=247d8c36cb0b13fb server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:09.524 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb94a7fc700 1 -- 172.21.15.92:0/1667549557 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb94400b6c0 con 0x7fb94c13fbf0 2022-06-29T15:58:09.524 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb94a7fc700 1 -- 172.21.15.92:0/1667549557 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb94400b820 con 0x7fb94c13fbf0 2022-06-29T15:58:09.525 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb94a7fc700 1 -- 172.21.15.92:0/1667549557 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb94400b990 con 0x7fb94c13fbf0 2022-06-29T15:58:09.528 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb950c14700 1 -- 172.21.15.92:0/1667549557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb94c13fbf0 msgr2=0x7fb94c140020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:09.529 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb950c14700 1 --2- 172.21.15.92:0/1667549557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb94c13fbf0 0x7fb94c140020 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fb9440032f0 tx=0x7fb9440091d0).stop 2022-06-29T15:58:09.529 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb950c14700 1 -- 172.21.15.92:0/1667549557 shutdown_connections 2022-06-29T15:58:09.529 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb950c14700 1 --2- 172.21.15.92:0/1667549557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb94c13fbf0 0x7fb94c140020 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:09.529 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb950c14700 1 -- 172.21.15.92:0/1667549557 >> 172.21.15.92:0/1667549557 conn(0x7fb94c0b6fb0 msgr2=0x7fb94c0b7390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:09.530 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb950c14700 1 -- 172.21.15.92:0/1667549557 shutdown_connections 2022-06-29T15:58:09.530 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb950c14700 1 -- 172.21.15.92:0/1667549557 wait complete. 2022-06-29T15:58:09.530 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb950c14700 1 Processor -- start 2022-06-29T15:58:09.530 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb950c14700 1 -- start start 2022-06-29T15:58:09.530 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb950c14700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb94c0acd60 0x7fb94c14fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:09.531 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb950c14700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb94c0ad520 0x7fb94c13e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:09.531 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb950c14700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb94c13fbf0 0x7fb94c1535b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:09.531 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb950c14700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c110780 con 0x7fb94c13fbf0 2022-06-29T15:58:09.531 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb950c14700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c0b14b0 con 0x7fb94c0acd60 2022-06-29T15:58:09.532 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb950c14700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c153150 con 0x7fb94c0ad520 2022-06-29T15:58:09.532 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb94b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb94c0acd60 0x7fb94c14fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:09.532 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb94affd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb94c0ad520 0x7fb94c13e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:09.532 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb94affd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb94c0ad520 0x7fb94c13e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.92:59938/0 (socket says 172.21.15.92:59938) 2022-06-29T15:58:09.532 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb94b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb94c0acd60 0x7fb94c14fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:45858/0 (socket says 172.21.15.92:45858) 2022-06-29T15:58:09.533 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb94affd700 1 -- 172.21.15.92:0/1631095432 learned_addr learned my addr 172.21.15.92:0/1631095432 (peer_addr_for_me v2:172.21.15.92:0/0) 2022-06-29T15:58:09.533 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb94bfff700 1 --2- 172.21.15.92:0/1631095432 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb94c13fbf0 0x7fb94c1535b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:09.533 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb94b7fe700 1 -- 172.21.15.92:0/1631095432 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb94c0ad520 msgr2=0x7fb94c13e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:09.533 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb94b7fe700 1 --2- 172.21.15.92:0/1631095432 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb94c0ad520 0x7fb94c13e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:09.534 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb94b7fe700 1 -- 172.21.15.92:0/1631095432 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb94c13fbf0 msgr2=0x7fb94c1535b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:09.534 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb94b7fe700 1 --2- 172.21.15.92:0/1631095432 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb94c13fbf0 0x7fb94c1535b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:09.534 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb94b7fe700 1 -- 172.21.15.92:0/1631095432 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb944009470 con 0x7fb94c0acd60 2022-06-29T15:58:09.534 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb94affd700 1 --2- 172.21.15.92:0/1631095432 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb94c0ad520 0x7fb94c13e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:09.534 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb94b7fe700 1 --2- 172.21.15.92:0/1631095432 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb94c0acd60 0x7fb94c14fae0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fb93c001110 tx=0x7fb93c007a20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:09.535 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb948ff9700 1 -- 172.21.15.92:0/1631095432 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb93c00ce80 con 0x7fb94c0acd60 2022-06-29T15:58:09.535 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb948ff9700 1 -- 172.21.15.92:0/1631095432 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb93c0070d0 con 0x7fb94c0acd60 2022-06-29T15:58:09.535 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb950c14700 1 -- 172.21.15.92:0/1631095432 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb94c0aed00 con 0x7fb94c0acd60 2022-06-29T15:58:09.535 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb948ff9700 1 -- 172.21.15.92:0/1631095432 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb93c007430 con 0x7fb94c0acd60 2022-06-29T15:58:09.536 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.520+0000 7fb950c14700 1 -- 172.21.15.92:0/1631095432 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb94c0af6d0 con 0x7fb94c0acd60 2022-06-29T15:58:09.536 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.524+0000 7fb948ff9700 1 -- 172.21.15.92:0/1631095432 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 1) v1 ==== 345+0+0 (secure 0 0 0) 0x7fb93c01a060 con 0x7fb94c0acd60 2022-06-29T15:58:09.536 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.524+0000 7fb948ff9700 1 -- 172.21.15.92:0/1631095432 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 869+0+0 (secure 0 0 0) 0x7fb93c005cc0 con 0x7fb94c0acd60 2022-06-29T15:58:09.536 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.524+0000 7fb950c14700 1 -- 172.21.15.92:0/1631095432 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb94c0b0560 con 0x7fb94c0acd60 2022-06-29T15:58:09.537 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.528+0000 7fb948ff9700 1 -- 172.21.15.92:0/1631095432 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67674 (secure 0 0 0) 0x7fb94c0b0560 con 0x7fb94c0acd60 2022-06-29T15:58:09.597 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.592+0000 7fb950c14700 1 -- 172.21.15.92:0/1631095432 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ca43f136-6f69-4431-a157-fbe073fb585c", "id": 1} v 0) v1 -- 0x7fb94c0573d0 con 0x7fb94c0acd60 2022-06-29T15:58:09.605 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.600+0000 7fb948ff9700 1 -- 172.21.15.92:0/1631095432 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ca43f136-6f69-4431-a157-fbe073fb585c", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb94c0aed00 con 0x7fb94c0acd60 2022-06-29T15:58:09.606 INFO:teuthology.orchestra.run.smithi092.stdout:1 2022-06-29T15:58:09.608 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.604+0000 7fb950c14700 1 -- 172.21.15.92:0/1631095432 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb94c0acd60 msgr2=0x7fb94c14fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:09.608 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.604+0000 7fb950c14700 1 --2- 172.21.15.92:0/1631095432 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb94c0acd60 0x7fb94c14fae0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fb93c001110 tx=0x7fb93c007a20).stop 2022-06-29T15:58:09.609 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.604+0000 7fb950c14700 1 -- 172.21.15.92:0/1631095432 shutdown_connections 2022-06-29T15:58:09.609 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.604+0000 7fb950c14700 1 --2- 172.21.15.92:0/1631095432 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb94c13fbf0 0x7fb94c1535b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:09.610 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.604+0000 7fb950c14700 1 --2- 172.21.15.92:0/1631095432 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb94c0ad520 0x7fb94c13e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:09.610 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.604+0000 7fb950c14700 1 --2- 172.21.15.92:0/1631095432 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb94c0acd60 0x7fb94c14fae0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:09.610 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.604+0000 7fb950c14700 1 -- 172.21.15.92:0/1631095432 >> 172.21.15.92:0/1631095432 conn(0x7fb94c0b6fb0 msgr2=0x7fb94c141310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:09.611 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.604+0000 7fb950c14700 1 -- 172.21.15.92:0/1631095432 shutdown_connections 2022-06-29T15:58:09.611 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.604+0000 7fb950c14700 1 -- 172.21.15.92:0/1631095432 wait complete. 2022-06-29T15:58:09.619 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new 74fac45f-5f07-44ec-a9cb-2644c123fd4a 2 2022-06-29T15:58:09.715 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.712+0000 7fdf7afe3700 1 Processor -- start 2022-06-29T15:58:09.715 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.712+0000 7fdf7afe3700 1 -- start start 2022-06-29T15:58:09.730 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.712+0000 7fdf7afe3700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdf740b3000 0x7fdf74146ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:09.731 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.712+0000 7fdf7afe3700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf7405fea0 con 0x7fdf74142c20 2022-06-29T15:58:09.731 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.712+0000 7fdf7afe3700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf740607f0 con 0x7fdf740b0990 2022-06-29T15:58:09.732 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.712+0000 7fdf7afe3700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf74060a90 con 0x7fdf740b3000 2022-06-29T15:58:09.732 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.712+0000 7fdf79fe1700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fdf740b0990 0x7fdf7413c6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:49952/0 (socket says 172.21.15.92:49952) 2022-06-29T15:58:09.733 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.712+0000 7fdf79fe1700 1 -- 172.21.15.92:0/1584434670 learned_addr learned my addr 172.21.15.92:0/1584434670 (peer_addr_for_me v1:172.21.15.92:0/0) 2022-06-29T15:58:09.733 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.712+0000 7fdf7a7e2700 1 --2- 172.21.15.92:0/1584434670 >> v2:172.21.15.60:3300/0 conn(0x7fdf740b3000 0x7fdf74146ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:09.733 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.712+0000 7fdf78fdf700 1 -- 172.21.15.92:0/1584434670 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893849309 0 0) 0x7fdf740607f0 con 0x7fdf740b0990 2022-06-29T15:58:09.734 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.712+0000 7fdf78fdf700 1 -- 172.21.15.92:0/1584434670 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf5c003260 con 0x7fdf740b0990 2022-06-29T15:58:09.734 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.712+0000 7fdf78fdf700 1 -- 172.21.15.92:0/1584434670 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2161574091 0 0) 0x7fdf5c003260 con 0x7fdf740b0990 2022-06-29T15:58:09.735 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.712+0000 7fdf78fdf700 1 -- 172.21.15.92:0/1584434670 >> v2:172.21.15.60:3300/0 conn(0x7fdf740b3000 msgr2=0x7fdf74146ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:09.735 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.712+0000 7fdf78fdf700 1 --2- 172.21.15.92:0/1584434670 >> v2:172.21.15.60:3300/0 conn(0x7fdf740b3000 0x7fdf74146ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:09.736 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.712+0000 7fdf78fdf700 1 -- 172.21.15.92:0/1584434670 >> v1:172.21.15.60:6789/0 conn(0x7fdf74142c20 legacy=0x7fdf74146490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:09.736 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.712+0000 7fdf78fdf700 1 -- 172.21.15.92:0/1584434670 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf74060d30 con 0x7fdf740b0990 2022-06-29T15:58:09.737 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.712+0000 7fdf78fdf700 1 -- 172.21.15.92:0/1584434670 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7fdf70001660 con 0x7fdf740b0990 2022-06-29T15:58:09.737 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.712+0000 7fdf78fdf700 1 -- 172.21.15.92:0/1584434670 >> v1:172.21.15.92:6789/0 conn(0x7fdf740b0990 legacy=0x7fdf7413c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:09.738 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.712+0000 7fdf78fdf700 1 --2- 172.21.15.92:0/1584434670 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf5c003750 0x7fdf5c003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:09.738 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.712+0000 7fdf78fdf700 1 --2- 172.21.15.92:0/1584434670 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf5c004500 0x7fdf5c004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:09.738 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.712+0000 7fdf78fdf700 1 --2- 172.21.15.92:0/1584434670 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf5c004e70 0x7fdf5c00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:09.739 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.712+0000 7fdf78fdf700 1 -- 172.21.15.92:0/1584434670 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf74060a90 con 0x7fdf5c004e70 2022-06-29T15:58:09.739 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.712+0000 7fdf78fdf700 1 -- 172.21.15.92:0/1584434670 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c00bd20 con 0x7fdf5c003750 2022-06-29T15:58:09.740 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.712+0000 7fdf78fdf700 1 -- 172.21.15.92:0/1584434670 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c00bee0 con 0x7fdf5c004500 2022-06-29T15:58:09.740 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.712+0000 7fdf79fe1700 1 --2- 172.21.15.92:0/1584434670 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf5c003750 0x7fdf5c003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:09.741 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.716+0000 7fdf797e0700 1 --2- 172.21.15.92:0/1584434670 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf5c004500 0x7fdf5c004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:09.741 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.716+0000 7fdf7a7e2700 1 --2- 172.21.15.92:0/1584434670 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf5c004e70 0x7fdf5c00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:09.742 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.716+0000 7fdf79fe1700 1 -- 172.21.15.92:0/1584434670 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf5c004500 msgr2=0x7fdf5c004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:09.742 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.716+0000 7fdf79fe1700 1 --2- 172.21.15.92:0/1584434670 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf5c004500 0x7fdf5c004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:09.743 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.716+0000 7fdf79fe1700 1 -- 172.21.15.92:0/1584434670 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf5c004e70 msgr2=0x7fdf5c00b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:09.743 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.716+0000 7fdf79fe1700 1 --2- 172.21.15.92:0/1584434670 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf5c004e70 0x7fdf5c00b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:09.743 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.716+0000 7fdf79fe1700 1 -- 172.21.15.92:0/1584434670 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf5c003260 con 0x7fdf5c003750 2022-06-29T15:58:09.744 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.716+0000 7fdf7a7e2700 1 --2- 172.21.15.92:0/1584434670 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf5c004e70 0x7fdf5c00b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:58:09.744 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.716+0000 7fdf7a7e2700 1 -- 172.21.15.92:0/1584434670 reap_dead start 2022-06-29T15:58:09.745 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.716+0000 7fdf79fe1700 1 --2- 172.21.15.92:0/1584434670 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf5c003750 0x7fdf5c003b70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fdf70001370 tx=0x7fdf740b3480).ready entity=mon.1 client_cookie=5cb3e30d5930f017 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:09.745 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.716+0000 7fdf78fdf700 1 -- 172.21.15.92:0/1584434670 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf700070d0 con 0x7fdf5c003750 2022-06-29T15:58:09.746 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.716+0000 7fdf78fdf700 1 -- 172.21.15.92:0/1584434670 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf70007230 con 0x7fdf5c003750 2022-06-29T15:58:09.746 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.716+0000 7fdf78fdf700 1 -- 172.21.15.92:0/1584434670 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf7000c8f0 con 0x7fdf5c003750 2022-06-29T15:58:09.747 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.716+0000 7fdf7afe3700 1 -- 172.21.15.92:0/1584434670 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf5c003750 msgr2=0x7fdf5c003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:09.747 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.716+0000 7fdf7afe3700 1 --2- 172.21.15.92:0/1584434670 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf5c003750 0x7fdf5c003b70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fdf70001370 tx=0x7fdf740b3480).stop 2022-06-29T15:58:09.748 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.716+0000 7fdf7afe3700 1 -- 172.21.15.92:0/1584434670 shutdown_connections 2022-06-29T15:58:09.748 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.716+0000 7fdf7afe3700 1 --2- 172.21.15.92:0/1584434670 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf5c003750 0x7fdf5c003b70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:09.749 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.716+0000 7fdf7afe3700 1 -- 172.21.15.92:0/1584434670 >> 172.21.15.92:0/1584434670 conn(0x7fdf740637c0 msgr2=0x7fdf74063ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:09.749 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.716+0000 7fdf7afe3700 1 -- 172.21.15.92:0/1584434670 shutdown_connections 2022-06-29T15:58:09.750 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.716+0000 7fdf7afe3700 1 -- 172.21.15.92:0/1584434670 wait complete. 2022-06-29T15:58:09.750 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.716+0000 7fdf7afe3700 1 Processor -- start 2022-06-29T15:58:09.750 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.716+0000 7fdf7afe3700 1 -- start start 2022-06-29T15:58:09.751 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.716+0000 7fdf7afe3700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf5c003750 0x7fdf74066d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:09.751 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.716+0000 7fdf7afe3700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf74067210 0x7fdf740659d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:09.752 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.716+0000 7fdf7afe3700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf74065ec0 0x7fdf74066370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:09.752 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.716+0000 7fdf7afe3700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf740afbe0 con 0x7fdf5c003750 2022-06-29T15:58:09.753 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.716+0000 7fdf7afe3700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf740adb30 con 0x7fdf74067210 2022-06-29T15:58:09.753 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.716+0000 7fdf7afe3700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf740ae930 con 0x7fdf74065ec0 2022-06-29T15:58:09.754 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.716+0000 7fdf7a7e2700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf74065ec0 0x7fdf74066370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:09.754 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.716+0000 7fdf797e0700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf74067210 0x7fdf740659d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:09.755 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.716+0000 7fdf797e0700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf74067210 0x7fdf740659d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:45878/0 (socket says 172.21.15.92:45878) 2022-06-29T15:58:09.755 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.716+0000 7fdf797e0700 1 -- 172.21.15.92:0/2122253546 learned_addr learned my addr 172.21.15.92:0/2122253546 (peer_addr_for_me v2:172.21.15.92:0/0) 2022-06-29T15:58:09.755 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.716+0000 7fdf79fe1700 1 --2- 172.21.15.92:0/2122253546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf5c003750 0x7fdf74066d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:09.756 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.720+0000 7fdf7a7e2700 1 -- 172.21.15.92:0/2122253546 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf74067210 msgr2=0x7fdf740659d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:09.756 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.720+0000 7fdf7a7e2700 1 --2- 172.21.15.92:0/2122253546 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf74067210 0x7fdf740659d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:09.757 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.720+0000 7fdf7a7e2700 1 -- 172.21.15.92:0/2122253546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf5c003750 msgr2=0x7fdf74066d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:09.757 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.720+0000 7fdf7a7e2700 1 --2- 172.21.15.92:0/2122253546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf5c003750 0x7fdf74066d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:09.758 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.720+0000 7fdf7a7e2700 1 -- 172.21.15.92:0/2122253546 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf740ad5b0 con 0x7fdf74065ec0 2022-06-29T15:58:09.758 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.720+0000 7fdf797e0700 1 --2- 172.21.15.92:0/2122253546 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf74067210 0x7fdf740659d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:09.758 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.720+0000 7fdf7a7e2700 1 --2- 172.21.15.92:0/2122253546 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf74065ec0 0x7fdf74066370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fdf740afac0 tx=0x7fdf6c008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:09.759 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.720+0000 7fdf62ffd700 1 -- 172.21.15.92:0/2122253546 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf6c011ed0 con 0x7fdf74065ec0 2022-06-29T15:58:09.759 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.720+0000 7fdf62ffd700 1 -- 172.21.15.92:0/2122253546 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf6c013550 con 0x7fdf74065ec0 2022-06-29T15:58:09.760 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.720+0000 7fdf7afe3700 1 -- 172.21.15.92:0/2122253546 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf741554b0 con 0x7fdf74065ec0 2022-06-29T15:58:09.760 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.720+0000 7fdf62ffd700 1 -- 172.21.15.92:0/2122253546 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf6c0138b0 con 0x7fdf74065ec0 2022-06-29T15:58:09.761 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.720+0000 7fdf7afe3700 1 -- 172.21.15.92:0/2122253546 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf74155900 con 0x7fdf74065ec0 2022-06-29T15:58:09.761 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.720+0000 7fdf62ffd700 1 -- 172.21.15.92:0/2122253546 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 1) v1 ==== 345+0+0 (secure 0 0 0) 0x7fdf6c01d030 con 0x7fdf74065ec0 2022-06-29T15:58:09.762 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.720+0000 7fdf62ffd700 1 -- 172.21.15.92:0/2122253546 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 984+0+0 (secure 0 0 0) 0x7fdf6c015890 con 0x7fdf74065ec0 2022-06-29T15:58:09.762 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.720+0000 7fdf7afe3700 1 -- 172.21.15.92:0/2122253546 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf740adcf0 con 0x7fdf74065ec0 2022-06-29T15:58:09.762 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.724+0000 7fdf62ffd700 1 -- 172.21.15.92:0/2122253546 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67674 (secure 0 0 0) 0x7fdf6c01b070 con 0x7fdf74065ec0 2022-06-29T15:58:09.791 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.788+0000 7fdf7afe3700 1 -- 172.21.15.92:0/2122253546 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd new", "uuid": "74fac45f-5f07-44ec-a9cb-2644c123fd4a", "id": 2} v 0) v1 -- 0x7fdf740675f0 con 0x7fdf74065ec0 2022-06-29T15:58:09.798 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.796+0000 7fdf62ffd700 1 -- 172.21.15.92:0/2122253546 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "74fac45f-5f07-44ec-a9cb-2644c123fd4a", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fdf6c015440 con 0x7fdf74065ec0 2022-06-29T15:58:09.799 INFO:teuthology.orchestra.run.smithi092.stdout:2 2022-06-29T15:58:09.801 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.796+0000 7fdf7afe3700 1 -- 172.21.15.92:0/2122253546 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf74065ec0 msgr2=0x7fdf74066370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:09.801 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.796+0000 7fdf7afe3700 1 --2- 172.21.15.92:0/2122253546 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf74065ec0 0x7fdf74066370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fdf740afac0 tx=0x7fdf6c008040).stop 2022-06-29T15:58:09.801 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.796+0000 7fdf7afe3700 1 -- 172.21.15.92:0/2122253546 shutdown_connections 2022-06-29T15:58:09.802 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.796+0000 7fdf7afe3700 1 --2- 172.21.15.92:0/2122253546 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdf74065ec0 0x7fdf74066370 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:09.802 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.796+0000 7fdf7afe3700 1 --2- 172.21.15.92:0/2122253546 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf74067210 0x7fdf740659d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:09.802 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.796+0000 7fdf7afe3700 1 --2- 172.21.15.92:0/2122253546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdf5c003750 0x7fdf74066d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:09.802 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.796+0000 7fdf7afe3700 1 -- 172.21.15.92:0/2122253546 >> 172.21.15.92:0/2122253546 conn(0x7fdf740637c0 msgr2=0x7fdf7413fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:09.803 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.796+0000 7fdf7afe3700 1 -- 172.21.15.92:0/2122253546 shutdown_connections 2022-06-29T15:58:09.803 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.796+0000 7fdf7afe3700 1 -- 172.21.15.92:0/2122253546 wait complete. 2022-06-29T15:58:09.813 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new 2581d351-973a-4b4e-85ca-3658f0068765 3 2022-06-29T15:58:09.914 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.912+0000 7f16c9244700 1 Processor -- start 2022-06-29T15:58:09.914 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.912+0000 7f16c9244700 1 -- start start 2022-06-29T15:58:09.917 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.912+0000 7f16c9244700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f16c40b3000 0x7f16c4146ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:09.917 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.912+0000 7f16c9244700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16c405fea0 con 0x7f16c4142c20 2022-06-29T15:58:09.918 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.912+0000 7f16c9244700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16c40607f0 con 0x7f16c40b0990 2022-06-29T15:58:09.918 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.912+0000 7f16c9244700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16c4060a90 con 0x7f16c40b3000 2022-06-29T15:58:09.918 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.912+0000 7f16c8a43700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f16c40b3000 0x7f16c4146ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:09.918 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.912+0000 7f16c3fff700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f16c40b0990 0x7f16c413c6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:49970/0 (socket says 172.21.15.92:49970) 2022-06-29T15:58:09.919 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.912+0000 7f16c3fff700 1 -- 172.21.15.92:0/2027253157 learned_addr learned my addr 172.21.15.92:0/2027253157 (peer_addr_for_me v1:172.21.15.92:0/0) 2022-06-29T15:58:09.919 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.912+0000 7f16c8a43700 1 -- 172.21.15.92:0/2027253157 >> v1:172.21.15.92:6789/0 conn(0x7f16c40b0990 legacy=0x7f16c413c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:09.919 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.912+0000 7f16c8a43700 1 -- 172.21.15.92:0/2027253157 >> v1:172.21.15.60:6789/0 conn(0x7f16c4142c20 legacy=0x7f16c4146490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:09.919 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.912+0000 7f16c8a43700 1 -- 172.21.15.92:0/2027253157 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c4060d30 con 0x7f16c40b3000 2022-06-29T15:58:09.920 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.912+0000 7f16c8a43700 1 --2- 172.21.15.92:0/2027253157 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16c40b3000 0x7f16c4146ba0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f16c40b06b0 tx=0x7f16bc0091d0).ready entity=mon.1 client_cookie=a98146e8a6aa33c4 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:09.920 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.912+0000 7f16c2ffd700 1 -- 172.21.15.92:0/2027253157 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16bc009ea0 con 0x7f16c40b3000 2022-06-29T15:58:09.920 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.912+0000 7f16c2ffd700 1 -- 172.21.15.92:0/2027253157 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16bc00b860 con 0x7f16c40b3000 2022-06-29T15:58:09.921 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.912+0000 7f16c2ffd700 1 -- 172.21.15.92:0/2027253157 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16bc00b9d0 con 0x7f16c40b3000 2022-06-29T15:58:09.921 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.916+0000 7f16c9244700 1 -- 172.21.15.92:0/2027253157 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16c40b3000 msgr2=0x7f16c4146ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:09.921 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.916+0000 7f16c9244700 1 --2- 172.21.15.92:0/2027253157 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16c40b3000 0x7f16c4146ba0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f16c40b06b0 tx=0x7f16bc0091d0).stop 2022-06-29T15:58:09.922 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.916+0000 7f16c9244700 1 -- 172.21.15.92:0/2027253157 shutdown_connections 2022-06-29T15:58:09.923 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.916+0000 7f16c9244700 1 --2- 172.21.15.92:0/2027253157 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16c40b3000 0x7f16c4146ba0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:09.924 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.916+0000 7f16c9244700 1 -- 172.21.15.92:0/2027253157 >> 172.21.15.92:0/2027253157 conn(0x7f16c40637c0 msgr2=0x7f16c4063ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:09.924 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.916+0000 7f16c9244700 1 -- 172.21.15.92:0/2027253157 shutdown_connections 2022-06-29T15:58:09.925 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.916+0000 7f16c9244700 1 -- 172.21.15.92:0/2027253157 wait complete. 2022-06-29T15:58:09.932 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.916+0000 7f16c9244700 1 Processor -- start 2022-06-29T15:58:09.932 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.916+0000 7f16c9244700 1 -- start start 2022-06-29T15:58:09.933 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.916+0000 7f16c9244700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16c40b0990 0x7f16c4066dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:09.933 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.916+0000 7f16c9244700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16c40b3000 0x7f16c413fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:09.934 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.916+0000 7f16c9244700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16c40672c0 0x7f16c413ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:09.934 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.916+0000 7f16c9244700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c40af790 con 0x7f16c40672c0 2022-06-29T15:58:09.935 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.916+0000 7f16c9244700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c40ae6d0 con 0x7f16c40b0990 2022-06-29T15:58:09.935 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.916+0000 7f16c9244700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c4065220 con 0x7f16c40b3000 2022-06-29T15:58:09.935 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.916+0000 7f16c37fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16c40b3000 0x7f16c413fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:09.936 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.916+0000 7f16c3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16c40b0990 0x7f16c4066dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:09.936 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.916+0000 7f16c37fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16c40b3000 0x7f16c413fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.92:59968/0 (socket says 172.21.15.92:59968) 2022-06-29T15:58:09.937 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.916+0000 7f16c37fe700 1 -- 172.21.15.92:0/2860987995 learned_addr learned my addr 172.21.15.92:0/2860987995 (peer_addr_for_me v2:172.21.15.92:0/0) 2022-06-29T15:58:09.937 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.916+0000 7f16c8a43700 1 --2- 172.21.15.92:0/2860987995 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16c40672c0 0x7f16c413ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:09.938 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.916+0000 7f16c37fe700 1 -- 172.21.15.92:0/2860987995 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16c40b0990 msgr2=0x7f16c4066dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:09.938 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.916+0000 7f16c37fe700 1 --2- 172.21.15.92:0/2860987995 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16c40b0990 0x7f16c4066dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:09.939 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.916+0000 7f16c37fe700 1 -- 172.21.15.92:0/2860987995 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16c40672c0 msgr2=0x7f16c413ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:09.939 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.916+0000 7f16c37fe700 1 --2- 172.21.15.92:0/2860987995 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16c40672c0 0x7f16c413ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:09.940 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.916+0000 7f16c37fe700 1 -- 172.21.15.92:0/2860987995 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16bc009470 con 0x7f16c40b3000 2022-06-29T15:58:09.940 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.916+0000 7f16c8a43700 1 --2- 172.21.15.92:0/2860987995 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16c40672c0 0x7f16c413ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:58:09.940 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.920+0000 7f16c37fe700 1 --2- 172.21.15.92:0/2860987995 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16c40b3000 0x7f16c413fae0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f16c40af8d0 tx=0x7f16b8006d60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:09.941 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.920+0000 7f16c17fa700 1 -- 172.21.15.92:0/2860987995 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16b8006030 con 0x7f16c40b3000 2022-06-29T15:58:09.941 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.920+0000 7f16c17fa700 1 -- 172.21.15.92:0/2860987995 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b8006190 con 0x7f16c40b3000 2022-06-29T15:58:09.941 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.920+0000 7f16c9244700 1 -- 172.21.15.92:0/2860987995 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16c40ae900 con 0x7f16c40b3000 2022-06-29T15:58:09.941 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.920+0000 7f16c17fa700 1 -- 172.21.15.92:0/2860987995 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16b80064f0 con 0x7f16c40b3000 2022-06-29T15:58:09.941 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.920+0000 7f16c9244700 1 -- 172.21.15.92:0/2860987995 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16c40ae2d0 con 0x7f16c40b3000 2022-06-29T15:58:09.942 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.920+0000 7f16c17fa700 1 -- 172.21.15.92:0/2860987995 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 1) v1 ==== 345+0+0 (secure 0 0 0) 0x7f16b801a060 con 0x7f16c40b3000 2022-06-29T15:58:09.942 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.920+0000 7f16c17fa700 1 -- 172.21.15.92:0/2860987995 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1099+0+0 (secure 0 0 0) 0x7f16b8005c70 con 0x7f16c40b3000 2022-06-29T15:58:09.942 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.920+0000 7f16c9244700 1 -- 172.21.15.92:0/2860987995 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16c40af480 con 0x7f16c40b3000 2022-06-29T15:58:09.942 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.924+0000 7f16c17fa700 1 -- 172.21.15.92:0/2860987995 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67674 (secure 0 0 0) 0x7f16b8018070 con 0x7f16c40b3000 2022-06-29T15:58:09.992 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.988+0000 7f16c9244700 1 -- 172.21.15.92:0/2860987995 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd new", "uuid": "2581d351-973a-4b4e-85ca-3658f0068765", "id": 3} v 0) v1 -- 0x7f16c40573d0 con 0x7f16c40b3000 2022-06-29T15:58:10.000 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:09.996+0000 7f16c17fa700 1 -- 172.21.15.92:0/2860987995 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2581d351-973a-4b4e-85ca-3658f0068765", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f16b8005e30 con 0x7f16c40b3000 2022-06-29T15:58:10.000 INFO:teuthology.orchestra.run.smithi092.stdout:3 2022-06-29T15:58:10.001 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.000+0000 7f16c9244700 1 -- 172.21.15.92:0/2860987995 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16c40b3000 msgr2=0x7f16c413fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:10.002 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.000+0000 7f16c9244700 1 --2- 172.21.15.92:0/2860987995 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16c40b3000 0x7f16c413fae0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f16c40af8d0 tx=0x7f16b8006d60).stop 2022-06-29T15:58:10.004 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.000+0000 7f16c9244700 1 -- 172.21.15.92:0/2860987995 shutdown_connections 2022-06-29T15:58:10.004 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.000+0000 7f16c9244700 1 --2- 172.21.15.92:0/2860987995 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16c40672c0 0x7f16c413ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.004 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.000+0000 7f16c9244700 1 --2- 172.21.15.92:0/2860987995 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16c40b3000 0x7f16c413fae0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.005 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.000+0000 7f16c9244700 1 --2- 172.21.15.92:0/2860987995 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16c40b0990 0x7f16c4066dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.005 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.000+0000 7f16c9244700 1 -- 172.21.15.92:0/2860987995 >> 172.21.15.92:0/2860987995 conn(0x7f16c40637c0 msgr2=0x7f16c40b4e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:10.005 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.000+0000 7f16c9244700 1 -- 172.21.15.92:0/2860987995 shutdown_connections 2022-06-29T15:58:10.005 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.000+0000 7f16c9244700 1 -- 172.21.15.92:0/2860987995 wait complete. 2022-06-29T15:58:10.014 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new 3d234db9-87cd-4fe9-814f-0240a287d8b4 4 2022-06-29T15:58:10.108 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.104+0000 7ffbaba99700 1 Processor -- start 2022-06-29T15:58:10.109 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.104+0000 7ffbaba99700 1 -- start start 2022-06-29T15:58:10.111 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.108+0000 7ffbaba99700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffba40aed60 0x7ffba4147890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:10.112 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.108+0000 7ffbaba99700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffba40afb10 0x7ffba40b0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:10.112 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.108+0000 7ffbaba99700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba40ac8f0 con 0x7ffba41471a0 2022-06-29T15:58:10.112 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.108+0000 7ffbaba99700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffba40acb90 con 0x7ffba40afb10 2022-06-29T15:58:10.112 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.108+0000 7ffbaba99700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffba40b02d0 con 0x7ffba40aed60 2022-06-29T15:58:10.113 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.108+0000 7ffba9835700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffba40aed60 0x7ffba4147890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:10.113 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.108+0000 7ffba9034700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7ffba41471a0 0x7ffba4147d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:49984/0 (socket says 172.21.15.92:49984) 2022-06-29T15:58:10.113 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.108+0000 7ffba9034700 1 -- 172.21.15.92:0/2767141800 learned_addr learned my addr 172.21.15.92:0/2767141800 (peer_addr_for_me v1:172.21.15.92:0/0) 2022-06-29T15:58:10.113 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.108+0000 7ffbaa036700 1 --2- 172.21.15.92:0/2767141800 >> v2:172.21.15.60:3300/0 conn(0x7ffba40afb10 0x7ffba40b0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:10.114 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.108+0000 7ffba9835700 1 -- 172.21.15.92:0/2767141800 >> v2:172.21.15.60:3300/0 conn(0x7ffba40afb10 msgr2=0x7ffba40b0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:10.114 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.108+0000 7ffba9835700 1 --2- 172.21.15.92:0/2767141800 >> v2:172.21.15.60:3300/0 conn(0x7ffba40afb10 0x7ffba40b0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.114 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.108+0000 7ffba9835700 1 -- 172.21.15.92:0/2767141800 >> v1:172.21.15.92:6789/0 conn(0x7ffba41471a0 legacy=0x7ffba4147d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:10.114 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.108+0000 7ffba8833700 1 -- 172.21.15.92:0/2767141800 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3489786783 0 0) 0x7ffba40ac8f0 con 0x7ffba41471a0 2022-06-29T15:58:10.114 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.108+0000 7ffba9835700 1 -- 172.21.15.92:0/2767141800 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba40ac5b0 con 0x7ffba40aed60 2022-06-29T15:58:10.115 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.108+0000 7ffbaa036700 1 --2- 172.21.15.92:0/2767141800 >> v2:172.21.15.60:3300/0 conn(0x7ffba40afb10 0x7ffba40b0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:58:10.115 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.108+0000 7ffba9835700 1 --2- 172.21.15.92:0/2767141800 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffba40aed60 0x7ffba4147890 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7ffba0002f70 tx=0x7ffba0006df0).ready entity=mon.2 client_cookie=35505c35886ac1f3 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:10.115 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.108+0000 7ffba8833700 1 -- 172.21.15.92:0/2767141800 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffba000d570 con 0x7ffba40aed60 2022-06-29T15:58:10.115 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.108+0000 7ffba8833700 1 -- 172.21.15.92:0/2767141800 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffba0000e90 con 0x7ffba40aed60 2022-06-29T15:58:10.116 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.108+0000 7ffba8833700 1 -- 172.21.15.92:0/2767141800 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffba000d8c0 con 0x7ffba40aed60 2022-06-29T15:58:10.116 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.108+0000 7ffbaba99700 1 -- 172.21.15.92:0/2767141800 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffba40aed60 msgr2=0x7ffba4147890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:10.116 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.108+0000 7ffbaba99700 1 --2- 172.21.15.92:0/2767141800 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffba40aed60 0x7ffba4147890 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7ffba0002f70 tx=0x7ffba0006df0).stop 2022-06-29T15:58:10.116 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.108+0000 7ffbaba99700 1 -- 172.21.15.92:0/2767141800 shutdown_connections 2022-06-29T15:58:10.116 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.108+0000 7ffbaba99700 1 --2- 172.21.15.92:0/2767141800 >> v2:172.21.15.60:3300/0 conn(0x7ffba40afb10 0x7ffba40b0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.117 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.108+0000 7ffbaba99700 1 --2- 172.21.15.92:0/2767141800 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffba40aed60 0x7ffba4147890 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.117 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.108+0000 7ffbaba99700 1 -- 172.21.15.92:0/2767141800 >> 172.21.15.92:0/2767141800 conn(0x7ffba40b4f60 msgr2=0x7ffba40b5370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:10.117 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.108+0000 7ffbaba99700 1 -- 172.21.15.92:0/2767141800 shutdown_connections 2022-06-29T15:58:10.117 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.108+0000 7ffbaba99700 1 -- 172.21.15.92:0/2767141800 wait complete. 2022-06-29T15:58:10.117 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.108+0000 7ffbaba99700 1 Processor -- start 2022-06-29T15:58:10.118 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.108+0000 7ffbaba99700 1 -- start start 2022-06-29T15:58:10.121 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.108+0000 7ffbaba99700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffba40aed60 0x7ffba414ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:10.121 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.108+0000 7ffbaba99700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffba40afb10 0x7ffba414a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:10.121 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.112+0000 7ffbaba99700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffba4063d40 0x7ffba40b0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:10.121 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.112+0000 7ffbaba99700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba40637f0 con 0x7ffba4063d40 2022-06-29T15:58:10.122 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.112+0000 7ffbaba99700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba4063a20 con 0x7ffba40aed60 2022-06-29T15:58:10.122 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.112+0000 7ffbaba99700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffba414b7a0 con 0x7ffba40afb10 2022-06-29T15:58:10.122 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.112+0000 7ffba9835700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffba40aed60 0x7ffba414ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:10.122 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.112+0000 7ffba9034700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffba40afb10 0x7ffba414a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:10.122 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.112+0000 7ffba9835700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffba40aed60 0x7ffba414ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:45900/0 (socket says 172.21.15.92:45900) 2022-06-29T15:58:10.123 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.112+0000 7ffba9835700 1 -- 172.21.15.92:0/1749581267 learned_addr learned my addr 172.21.15.92:0/1749581267 (peer_addr_for_me v2:172.21.15.92:0/0) 2022-06-29T15:58:10.123 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.112+0000 7ffbaa036700 1 --2- 172.21.15.92:0/1749581267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffba4063d40 0x7ffba40b0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:10.123 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.112+0000 7ffba9835700 1 -- 172.21.15.92:0/1749581267 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffba40afb10 msgr2=0x7ffba414a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:10.124 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.112+0000 7ffba9835700 1 --2- 172.21.15.92:0/1749581267 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffba40afb10 0x7ffba414a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.124 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.112+0000 7ffba9835700 1 -- 172.21.15.92:0/1749581267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffba4063d40 msgr2=0x7ffba40b0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:10.124 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.112+0000 7ffba9835700 1 --2- 172.21.15.92:0/1749581267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffba4063d40 0x7ffba40b0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.124 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.112+0000 7ffba9835700 1 -- 172.21.15.92:0/1749581267 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba000b180 con 0x7ffba40aed60 2022-06-29T15:58:10.124 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.112+0000 7ffba9835700 1 --2- 172.21.15.92:0/1749581267 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffba40aed60 0x7ffba414ab10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7ffba0002f70 tx=0x7ffba0015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:10.125 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.112+0000 7ffb92ffd700 1 -- 172.21.15.92:0/1749581267 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffba000d8f0 con 0x7ffba40aed60 2022-06-29T15:58:10.125 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.112+0000 7ffb92ffd700 1 -- 172.21.15.92:0/1749581267 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffba0011b40 con 0x7ffba40aed60 2022-06-29T15:58:10.125 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.112+0000 7ffbaba99700 1 -- 172.21.15.92:0/1749581267 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffba414ba40 con 0x7ffba40aed60 2022-06-29T15:58:10.125 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.112+0000 7ffb92ffd700 1 -- 172.21.15.92:0/1749581267 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffba0011ea0 con 0x7ffba40aed60 2022-06-29T15:58:10.126 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.112+0000 7ffbaba99700 1 -- 172.21.15.92:0/1749581267 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffba4065d60 con 0x7ffba40aed60 2022-06-29T15:58:10.126 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.112+0000 7ffb92ffd700 1 -- 172.21.15.92:0/1749581267 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 1) v1 ==== 345+0+0 (secure 0 0 0) 0x7ffba0025030 con 0x7ffba40aed60 2022-06-29T15:58:10.126 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.112+0000 7ffb92ffd700 1 -- 172.21.15.92:0/1749581267 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1214+0+0 (secure 0 0 0) 0x7ffba001d8f0 con 0x7ffba40aed60 2022-06-29T15:58:10.126 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.112+0000 7ffbaba99700 1 -- 172.21.15.92:0/1749581267 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffba4065f80 con 0x7ffba40aed60 2022-06-29T15:58:10.126 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.116+0000 7ffb92ffd700 1 -- 172.21.15.92:0/1749581267 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67674 (secure 0 0 0) 0x7ffba0023070 con 0x7ffba40aed60 2022-06-29T15:58:10.182 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.180+0000 7ffbaba99700 1 -- 172.21.15.92:0/1749581267 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd new", "uuid": "3d234db9-87cd-4fe9-814f-0240a287d8b4", "id": 4} v 0) v1 -- 0x7ffba4065b40 con 0x7ffba40aed60 2022-06-29T15:58:10.189 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.184+0000 7ffb92ffd700 1 -- 172.21.15.92:0/1749581267 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3d234db9-87cd-4fe9-814f-0240a287d8b4", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7ffba001db90 con 0x7ffba40aed60 2022-06-29T15:58:10.190 INFO:teuthology.orchestra.run.smithi092.stdout:4 2022-06-29T15:58:10.191 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.188+0000 7ffbaba99700 1 -- 172.21.15.92:0/1749581267 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffba40aed60 msgr2=0x7ffba414ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:10.192 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.188+0000 7ffbaba99700 1 --2- 172.21.15.92:0/1749581267 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffba40aed60 0x7ffba414ab10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7ffba0002f70 tx=0x7ffba0015000).stop 2022-06-29T15:58:10.192 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.188+0000 7ffbaba99700 1 -- 172.21.15.92:0/1749581267 shutdown_connections 2022-06-29T15:58:10.192 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.188+0000 7ffbaba99700 1 --2- 172.21.15.92:0/1749581267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffba4063d40 0x7ffba40b0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.193 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.188+0000 7ffbaba99700 1 --2- 172.21.15.92:0/1749581267 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffba40afb10 0x7ffba414a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.193 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.188+0000 7ffbaba99700 1 --2- 172.21.15.92:0/1749581267 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffba40aed60 0x7ffba414ab10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.193 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.188+0000 7ffbaba99700 1 -- 172.21.15.92:0/1749581267 >> 172.21.15.92:0/1749581267 conn(0x7ffba40b4f60 msgr2=0x7ffba40b43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:10.193 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.188+0000 7ffbaba99700 1 -- 172.21.15.92:0/1749581267 shutdown_connections 2022-06-29T15:58:10.195 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.188+0000 7ffbaba99700 1 -- 172.21.15.92:0/1749581267 wait complete. 2022-06-29T15:58:10.203 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new e4bf1104-ff5c-438b-b608-8cee31c163f6 5 2022-06-29T15:58:10.308 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.304+0000 7fdaf3226700 1 Processor -- start 2022-06-29T15:58:10.309 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.304+0000 7fdaf3226700 1 -- start start 2022-06-29T15:58:10.311 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.308+0000 7fdaf3226700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdaec0b2bc0 0x7fdaec0ae990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:10.311 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.308+0000 7fdaf3226700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdaec1392e0 0x7fdaec139750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:10.311 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.308+0000 7fdaf3226700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaec0b2fa0 con 0x7fdaec0b0b10 2022-06-29T15:58:10.311 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.308+0000 7fdaf3226700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaec0ae250 con 0x7fdaec1392e0 2022-06-29T15:58:10.312 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.308+0000 7fdaf3226700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaec0ae4f0 con 0x7fdaec0b2bc0 2022-06-29T15:58:10.312 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.308+0000 7fdaf2224700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdaec0b2bc0 0x7fdaec0ae990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:10.312 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.308+0000 7fdaf1a23700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fdaec0b0b10 0x7fdaec05f0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:49996/0 (socket says 172.21.15.92:49996) 2022-06-29T15:58:10.312 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.308+0000 7fdaf1a23700 1 -- 172.21.15.92:0/476384079 learned_addr learned my addr 172.21.15.92:0/476384079 (peer_addr_for_me v1:172.21.15.92:0/0) 2022-06-29T15:58:10.313 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.308+0000 7fdaf2a25700 1 --2- 172.21.15.92:0/476384079 >> v2:172.21.15.60:3300/0 conn(0x7fdaec1392e0 0x7fdaec139750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:10.313 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.308+0000 7fdaf1222700 1 -- 172.21.15.92:0/476384079 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 364568762 0 0) 0x7fdaec0b2fa0 con 0x7fdaec0b0b10 2022-06-29T15:58:10.313 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.308+0000 7fdaf1222700 1 -- 172.21.15.92:0/476384079 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdad4003260 con 0x7fdaec0b0b10 2022-06-29T15:58:10.313 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.308+0000 7fdaf2224700 1 -- 172.21.15.92:0/476384079 >> v2:172.21.15.60:3300/0 conn(0x7fdaec1392e0 msgr2=0x7fdaec139750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:10.314 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.308+0000 7fdaf2224700 1 --2- 172.21.15.92:0/476384079 >> v2:172.21.15.60:3300/0 conn(0x7fdaec1392e0 0x7fdaec139750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.314 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.308+0000 7fdaf2224700 1 -- 172.21.15.92:0/476384079 >> v1:172.21.15.92:6789/0 conn(0x7fdaec0b0b10 legacy=0x7fdaec05f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:10.314 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.308+0000 7fdaf2224700 1 -- 172.21.15.92:0/476384079 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaec0aee80 con 0x7fdaec0b2bc0 2022-06-29T15:58:10.314 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.308+0000 7fdaf2224700 1 --2- 172.21.15.92:0/476384079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdaec0b2bc0 0x7fdaec0ae990 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fdae8002f70 tx=0x7fdae8006df0).ready entity=mon.1 client_cookie=a503253acef868dc server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:10.315 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.308+0000 7fdaf1222700 1 -- 172.21.15.92:0/476384079 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdae800d4f0 con 0x7fdaec0b2bc0 2022-06-29T15:58:10.315 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.308+0000 7fdaf1222700 1 -- 172.21.15.92:0/476384079 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdae8000e90 con 0x7fdaec0b2bc0 2022-06-29T15:58:10.315 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.308+0000 7fdaf1222700 1 -- 172.21.15.92:0/476384079 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdae800d840 con 0x7fdaec0b2bc0 2022-06-29T15:58:10.318 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.308+0000 7fdaf3226700 1 -- 172.21.15.92:0/476384079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdaec0b2bc0 msgr2=0x7fdaec0ae990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:10.319 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.308+0000 7fdaf3226700 1 --2- 172.21.15.92:0/476384079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdaec0b2bc0 0x7fdaec0ae990 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fdae8002f70 tx=0x7fdae8006df0).stop 2022-06-29T15:58:10.319 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.308+0000 7fdaf3226700 1 -- 172.21.15.92:0/476384079 shutdown_connections 2022-06-29T15:58:10.319 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.308+0000 7fdaf3226700 1 --2- 172.21.15.92:0/476384079 >> v2:172.21.15.60:3300/0 conn(0x7fdaec1392e0 0x7fdaec139750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.320 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.308+0000 7fdaf3226700 1 --2- 172.21.15.92:0/476384079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdaec0b2bc0 0x7fdaec0ae990 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.320 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.308+0000 7fdaf3226700 1 -- 172.21.15.92:0/476384079 >> 172.21.15.92:0/476384079 conn(0x7fdaec0637c0 msgr2=0x7fdaec063ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:10.320 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.308+0000 7fdaf3226700 1 -- 172.21.15.92:0/476384079 shutdown_connections 2022-06-29T15:58:10.320 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.308+0000 7fdaf3226700 1 -- 172.21.15.92:0/476384079 wait complete. 2022-06-29T15:58:10.320 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.312+0000 7fdaf3226700 1 Processor -- start 2022-06-29T15:58:10.321 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.312+0000 7fdaf3226700 1 -- start start 2022-06-29T15:58:10.321 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.312+0000 7fdaf3226700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdaec0b0b10 0x7fdaec065a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:10.321 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.312+0000 7fdaf3226700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdaec0b2bc0 0x7fdaec13b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:10.321 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.312+0000 7fdaf3226700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdaec1392e0 0x7fdaec13b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:10.321 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.312+0000 7fdaf3226700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaec141ec0 con 0x7fdaec1392e0 2022-06-29T15:58:10.322 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.312+0000 7fdaf3226700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaec066100 con 0x7fdaec0b0b10 2022-06-29T15:58:10.322 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.312+0000 7fdaf3226700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaec0663a0 con 0x7fdaec0b2bc0 2022-06-29T15:58:10.322 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.312+0000 7fdaf2224700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdaec0b0b10 0x7fdaec065a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:10.322 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.312+0000 7fdaf1a23700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdaec0b2bc0 0x7fdaec13b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:10.323 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.312+0000 7fdaf2224700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdaec0b0b10 0x7fdaec065a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:45912/0 (socket says 172.21.15.92:45912) 2022-06-29T15:58:10.323 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.312+0000 7fdaf2224700 1 -- 172.21.15.92:0/4273854351 learned_addr learned my addr 172.21.15.92:0/4273854351 (peer_addr_for_me v2:172.21.15.92:0/0) 2022-06-29T15:58:10.323 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.312+0000 7fdaf2a25700 1 --2- 172.21.15.92:0/4273854351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdaec1392e0 0x7fdaec13b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:10.323 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.312+0000 7fdaf2224700 1 -- 172.21.15.92:0/4273854351 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdaec0b2bc0 msgr2=0x7fdaec13b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:10.324 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.312+0000 7fdaf2224700 1 --2- 172.21.15.92:0/4273854351 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdaec0b2bc0 0x7fdaec13b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.324 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.312+0000 7fdaf2224700 1 -- 172.21.15.92:0/4273854351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdaec1392e0 msgr2=0x7fdaec13b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:10.324 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.312+0000 7fdaf2224700 1 --2- 172.21.15.92:0/4273854351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdaec1392e0 0x7fdaec13b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.324 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.312+0000 7fdaf2224700 1 -- 172.21.15.92:0/4273854351 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae800b180 con 0x7fdaec0b0b10 2022-06-29T15:58:10.324 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.312+0000 7fdaf2a25700 1 --2- 172.21.15.92:0/4273854351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdaec1392e0 0x7fdaec13b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:58:10.325 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.312+0000 7fdaf2224700 1 --2- 172.21.15.92:0/4273854351 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdaec0b0b10 0x7fdaec065a40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fdaec064ff0 tx=0x7fdae8014000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:10.325 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.312+0000 7fdadb7fe700 1 -- 172.21.15.92:0/4273854351 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdae8013e50 con 0x7fdaec0b0b10 2022-06-29T15:58:10.325 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.312+0000 7fdadb7fe700 1 -- 172.21.15.92:0/4273854351 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdae800de60 con 0x7fdaec0b0b10 2022-06-29T15:58:10.325 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.312+0000 7fdaf3226700 1 -- 172.21.15.92:0/4273854351 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaec1414a0 con 0x7fdaec0b0b10 2022-06-29T15:58:10.326 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.312+0000 7fdadb7fe700 1 -- 172.21.15.92:0/4273854351 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdae801e5f0 con 0x7fdaec0b0b10 2022-06-29T15:58:10.326 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.312+0000 7fdaf3226700 1 -- 172.21.15.92:0/4273854351 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaec138a90 con 0x7fdaec0b0b10 2022-06-29T15:58:10.326 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.312+0000 7fdadb7fe700 1 -- 172.21.15.92:0/4273854351 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 1) v1 ==== 345+0+0 (secure 0 0 0) 0x7fdae801c070 con 0x7fdaec0b0b10 2022-06-29T15:58:10.326 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.312+0000 7fdadb7fe700 1 -- 172.21.15.92:0/4273854351 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1329+0+0 (secure 0 0 0) 0x7fdae8011970 con 0x7fdaec0b0b10 2022-06-29T15:58:10.326 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.316+0000 7fdaf3226700 1 -- 172.21.15.92:0/4273854351 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaec066560 con 0x7fdaec0b0b10 2022-06-29T15:58:10.327 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.316+0000 7fdadb7fe700 1 -- 172.21.15.92:0/4273854351 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67674 (secure 0 0 0) 0x7fdae8023070 con 0x7fdaec0b0b10 2022-06-29T15:58:10.388 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.384+0000 7fdaf3226700 1 -- 172.21.15.92:0/4273854351 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e4bf1104-ff5c-438b-b608-8cee31c163f6", "id": 5} v 0) v1 -- 0x7fdaec065f30 con 0x7fdaec0b0b10 2022-06-29T15:58:10.395 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.392+0000 7fdadb7fe700 1 -- 172.21.15.92:0/4273854351 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e4bf1104-ff5c-438b-b608-8cee31c163f6", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fdae8019030 con 0x7fdaec0b0b10 2022-06-29T15:58:10.396 INFO:teuthology.orchestra.run.smithi092.stdout:5 2022-06-29T15:58:10.397 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.392+0000 7fdaf3226700 1 -- 172.21.15.92:0/4273854351 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdaec0b0b10 msgr2=0x7fdaec065a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:10.399 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.392+0000 7fdaf3226700 1 --2- 172.21.15.92:0/4273854351 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdaec0b0b10 0x7fdaec065a40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fdaec064ff0 tx=0x7fdae8014000).stop 2022-06-29T15:58:10.399 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.392+0000 7fdaf3226700 1 -- 172.21.15.92:0/4273854351 shutdown_connections 2022-06-29T15:58:10.400 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.392+0000 7fdaf3226700 1 --2- 172.21.15.92:0/4273854351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdaec1392e0 0x7fdaec13b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.400 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.392+0000 7fdaf3226700 1 --2- 172.21.15.92:0/4273854351 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdaec0b2bc0 0x7fdaec13b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.400 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.392+0000 7fdaf3226700 1 --2- 172.21.15.92:0/4273854351 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdaec0b0b10 0x7fdaec065a40 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.400 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.392+0000 7fdaf3226700 1 -- 172.21.15.92:0/4273854351 >> 172.21.15.92:0/4273854351 conn(0x7fdaec0637c0 msgr2=0x7fdaec0b48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:10.401 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.396+0000 7fdaf3226700 1 -- 172.21.15.92:0/4273854351 shutdown_connections 2022-06-29T15:58:10.401 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.396+0000 7fdaf3226700 1 -- 172.21.15.92:0/4273854351 wait complete. 2022-06-29T15:58:10.408 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new 8564dd25-27ab-4203-b54e-af6d3aa05857 6 2022-06-29T15:58:10.504 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.500+0000 7f0b65663700 1 Processor -- start 2022-06-29T15:58:10.504 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.500+0000 7f0b65663700 1 -- start start 2022-06-29T15:58:10.507 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.504+0000 7f0b65663700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0b600b3d30 0x7f0b60146ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:10.508 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.504+0000 7f0b65663700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0b60061d90 0x7f0b60142b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:10.508 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.504+0000 7f0b65663700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b600b4f50 con 0x7f0b600b9d70 2022-06-29T15:58:10.508 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.504+0000 7f0b65663700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b600ba150 con 0x7f0b600b3d30 2022-06-29T15:58:10.509 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.504+0000 7f0b65663700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b600b5180 con 0x7f0b60061d90 2022-06-29T15:58:10.509 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.504+0000 7f0b5f7fe700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0b60061d90 0x7f0b60142b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:10.509 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.504+0000 7f0b5f7fe700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0b60061d90 0x7f0b60142b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:45922/0 (socket says 172.21.15.92:45922) 2022-06-29T15:58:10.509 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.504+0000 7f0b5f7fe700 1 -- 172.21.15.92:0/2443461614 learned_addr learned my addr 172.21.15.92:0/2443461614 (peer_addr_for_me v2:172.21.15.92:0/0) 2022-06-29T15:58:10.509 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.504+0000 7f0b5e7fc700 1 --2- 172.21.15.92:0/2443461614 >> v2:172.21.15.60:3300/0 conn(0x7f0b600b3d30 0x7f0b60146ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:10.510 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.504+0000 7f0b5f7fe700 1 -- 172.21.15.92:0/2443461614 >> v2:172.21.15.60:3300/0 conn(0x7f0b600b3d30 msgr2=0x7f0b60146ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:10.510 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.504+0000 7f0b5f7fe700 1 --2- 172.21.15.92:0/2443461614 >> v2:172.21.15.60:3300/0 conn(0x7f0b600b3d30 0x7f0b60146ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.510 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.504+0000 7f0b5f7fe700 1 -- 172.21.15.92:0/2443461614 >> v1:172.21.15.60:6789/0 conn(0x7f0b600b9d70 legacy=0x7f0b601465d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:10.511 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.504+0000 7f0b5f7fe700 1 -- 172.21.15.92:0/2443461614 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b601461a0 con 0x7f0b60061d90 2022-06-29T15:58:10.511 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.504+0000 7f0b5f7fe700 1 --2- 172.21.15.92:0/2443461614 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b60061d90 0x7f0b60142b70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f0b580032f0 tx=0x7f0b58009290).ready entity=mon.1 client_cookie=2b9f59ef2715fc3 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:10.511 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.504+0000 7f0b5dffb700 1 -- 172.21.15.92:0/2443461614 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b5800b870 con 0x7f0b60061d90 2022-06-29T15:58:10.511 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.504+0000 7f0b5dffb700 1 -- 172.21.15.92:0/2443461614 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b5800b9d0 con 0x7f0b60061d90 2022-06-29T15:58:10.512 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.504+0000 7f0b5dffb700 1 -- 172.21.15.92:0/2443461614 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b5800bb40 con 0x7f0b60061d90 2022-06-29T15:58:10.512 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.504+0000 7f0b65663700 1 -- 172.21.15.92:0/2443461614 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b60061d90 msgr2=0x7f0b60142b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:10.512 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.504+0000 7f0b65663700 1 --2- 172.21.15.92:0/2443461614 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b60061d90 0x7f0b60142b70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f0b580032f0 tx=0x7f0b58009290).stop 2022-06-29T15:58:10.512 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.504+0000 7f0b65663700 1 -- 172.21.15.92:0/2443461614 shutdown_connections 2022-06-29T15:58:10.512 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.504+0000 7f0b65663700 1 --2- 172.21.15.92:0/2443461614 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b60061d90 0x7f0b60142b70 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.513 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.504+0000 7f0b65663700 1 --2- 172.21.15.92:0/2443461614 >> v2:172.21.15.60:3300/0 conn(0x7f0b600b3d30 0x7f0b60146ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.513 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.504+0000 7f0b65663700 1 -- 172.21.15.92:0/2443461614 >> 172.21.15.92:0/2443461614 conn(0x7f0b60062e70 msgr2=0x7f0b600b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:10.513 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.504+0000 7f0b65663700 1 -- 172.21.15.92:0/2443461614 shutdown_connections 2022-06-29T15:58:10.514 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.504+0000 7f0b65663700 1 -- 172.21.15.92:0/2443461614 wait complete. 2022-06-29T15:58:10.517 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.504+0000 7f0b65663700 1 Processor -- start 2022-06-29T15:58:10.517 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.504+0000 7f0b65663700 1 -- start start 2022-06-29T15:58:10.518 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.508+0000 7f0b65663700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b60061d90 0x7f0b600adcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:10.518 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.508+0000 7f0b65663700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b600b3d30 0x7f0b60155330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:10.518 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.508+0000 7f0b65663700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b600b9d70 0x7f0b60112940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:10.518 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.508+0000 7f0b65663700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b60112510 con 0x7f0b600b3d30 2022-06-29T15:58:10.519 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.508+0000 7f0b65663700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b60148ed0 con 0x7f0b600b9d70 2022-06-29T15:58:10.519 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.508+0000 7f0b65663700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b60149140 con 0x7f0b60061d90 2022-06-29T15:58:10.519 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.508+0000 7f0b5effd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b60061d90 0x7f0b600adcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:10.520 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.508+0000 7f0b5f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b600b9d70 0x7f0b60112940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:10.520 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.508+0000 7f0b5effd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b60061d90 0x7f0b600adcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.92:60002/0 (socket says 172.21.15.92:60002) 2022-06-29T15:58:10.520 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.508+0000 7f0b5effd700 1 -- 172.21.15.92:0/1043221385 learned_addr learned my addr 172.21.15.92:0/1043221385 (peer_addr_for_me v2:172.21.15.92:0/0) 2022-06-29T15:58:10.520 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.508+0000 7f0b5e7fc700 1 --2- 172.21.15.92:0/1043221385 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b600b3d30 0x7f0b60155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:10.520 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.508+0000 7f0b5effd700 1 -- 172.21.15.92:0/1043221385 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b600b9d70 msgr2=0x7f0b60112940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:10.521 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.508+0000 7f0b5effd700 1 --2- 172.21.15.92:0/1043221385 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b600b9d70 0x7f0b60112940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.521 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.508+0000 7f0b5effd700 1 -- 172.21.15.92:0/1043221385 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b600b3d30 msgr2=0x7f0b60155330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:10.521 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.508+0000 7f0b5effd700 1 --2- 172.21.15.92:0/1043221385 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b600b3d30 0x7f0b60155330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.521 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.508+0000 7f0b5effd700 1 -- 172.21.15.92:0/1043221385 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b58009530 con 0x7f0b60061d90 2022-06-29T15:58:10.522 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.508+0000 7f0b5f7fe700 1 --2- 172.21.15.92:0/1043221385 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b600b9d70 0x7f0b60112940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:10.522 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.508+0000 7f0b5effd700 1 --2- 172.21.15.92:0/1043221385 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b60061d90 0x7f0b600adcf0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f0b60063560 tx=0x7f0b50007c00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:10.522 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.508+0000 7f0b4ffff700 1 -- 172.21.15.92:0/1043221385 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b50007e70 con 0x7f0b60061d90 2022-06-29T15:58:10.522 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.508+0000 7f0b4ffff700 1 -- 172.21.15.92:0/1043221385 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b50007030 con 0x7f0b60061d90 2022-06-29T15:58:10.522 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.508+0000 7f0b65663700 1 -- 172.21.15.92:0/1043221385 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b60146d60 con 0x7f0b60061d90 2022-06-29T15:58:10.523 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.508+0000 7f0b4ffff700 1 -- 172.21.15.92:0/1043221385 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b50007390 con 0x7f0b60061d90 2022-06-29T15:58:10.523 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.508+0000 7f0b65663700 1 -- 172.21.15.92:0/1043221385 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b601471b0 con 0x7f0b60061d90 2022-06-29T15:58:10.523 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.508+0000 7f0b4ffff700 1 -- 172.21.15.92:0/1043221385 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 1) v1 ==== 345+0+0 (secure 0 0 0) 0x7f0b50010030 con 0x7f0b60061d90 2022-06-29T15:58:10.524 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.508+0000 7f0b4ffff700 1 -- 172.21.15.92:0/1043221385 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1444+0+0 (secure 0 0 0) 0x7f0b50006be0 con 0x7f0b60061d90 2022-06-29T15:58:10.524 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.508+0000 7f0b65663700 1 -- 172.21.15.92:0/1043221385 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b60112650 con 0x7f0b60061d90 2022-06-29T15:58:10.524 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.512+0000 7f0b4ffff700 1 -- 172.21.15.92:0/1043221385 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67674 (secure 0 0 0) 0x7f0b50013410 con 0x7f0b60061d90 2022-06-29T15:58:10.583 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.580+0000 7f0b65663700 1 -- 172.21.15.92:0/1043221385 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd new", "uuid": "8564dd25-27ab-4203-b54e-af6d3aa05857", "id": 6} v 0) v1 -- 0x7f0b600573d0 con 0x7f0b60061d90 2022-06-29T15:58:10.593 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.592+0000 7f0b4ffff700 1 -- 172.21.15.92:0/1043221385 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8564dd25-27ab-4203-b54e-af6d3aa05857", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0b500135b0 con 0x7f0b60061d90 2022-06-29T15:58:10.594 INFO:teuthology.orchestra.run.smithi092.stdout:6 2022-06-29T15:58:10.596 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.592+0000 7f0b65663700 1 -- 172.21.15.92:0/1043221385 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b60061d90 msgr2=0x7f0b600adcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:10.596 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.592+0000 7f0b65663700 1 --2- 172.21.15.92:0/1043221385 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b60061d90 0x7f0b600adcf0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f0b60063560 tx=0x7f0b50007c00).stop 2022-06-29T15:58:10.597 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.592+0000 7f0b65663700 1 -- 172.21.15.92:0/1043221385 shutdown_connections 2022-06-29T15:58:10.597 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.592+0000 7f0b65663700 1 --2- 172.21.15.92:0/1043221385 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b600b9d70 0x7f0b60112940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.598 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.592+0000 7f0b65663700 1 --2- 172.21.15.92:0/1043221385 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b600b3d30 0x7f0b60155330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.598 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.592+0000 7f0b65663700 1 --2- 172.21.15.92:0/1043221385 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b60061d90 0x7f0b600adcf0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.598 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.592+0000 7f0b65663700 1 -- 172.21.15.92:0/1043221385 >> 172.21.15.92:0/1043221385 conn(0x7f0b60062e70 msgr2=0x7f0b60143130 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:10.599 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.592+0000 7f0b65663700 1 -- 172.21.15.92:0/1043221385 shutdown_connections 2022-06-29T15:58:10.599 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.592+0000 7f0b65663700 1 -- 172.21.15.92:0/1043221385 wait complete. 2022-06-29T15:58:10.606 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new bd513a9c-3b19-4314-9897-d300b5c897bb 7 2022-06-29T15:58:10.695 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.692+0000 7fddb6b3a700 1 Processor -- start 2022-06-29T15:58:10.696 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.692+0000 7fddb6b3a700 1 -- start start 2022-06-29T15:58:10.703 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.692+0000 7fddb6b3a700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fddb005f2c0 0x7fddb005f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:10.704 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.692+0000 7fddb6b3a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fddb005ff90 0x7fddb00aede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:10.704 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.692+0000 7fddb6b3a700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fddb005c3f0 0x7fddb005c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:10.704 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.692+0000 7fddb6b3a700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddb005bd20 con 0x7fddb005c3f0 2022-06-29T15:58:10.705 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.692+0000 7fddb6b3a700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddb01438e0 con 0x7fddb005f2c0 2022-06-29T15:58:10.705 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.692+0000 7fddb6b3a700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddb005bf50 con 0x7fddb005ff90 2022-06-29T15:58:10.705 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.692+0000 7fddb5b38700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fddb005f2c0 0x7fddb005f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:10.705 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.692+0000 7fddb5337700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fddb005ff90 0x7fddb00aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:10.706 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.692+0000 7fddb5b38700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fddb005f2c0 0x7fddb005f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:45930/0 (socket says 172.21.15.92:45930) 2022-06-29T15:58:10.706 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.692+0000 7fddb5b38700 1 -- 172.21.15.92:0/3654056756 learned_addr learned my addr 172.21.15.92:0/3654056756 (peer_addr_for_me v2:172.21.15.92:0/0) 2022-06-29T15:58:10.706 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.692+0000 7fddb6339700 1 --2- 172.21.15.92:0/3654056756 >> v2:172.21.15.60:3300/0 conn(0x7fddb005c3f0 0x7fddb005c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:10.706 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.692+0000 7fddb5b38700 1 -- 172.21.15.92:0/3654056756 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fddb005ff90 msgr2=0x7fddb00aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:10.707 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.692+0000 7fddb5b38700 1 --2- 172.21.15.92:0/3654056756 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fddb005ff90 0x7fddb00aede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.707 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.692+0000 7fddb5b38700 1 -- 172.21.15.92:0/3654056756 >> v2:172.21.15.60:3300/0 conn(0x7fddb005c3f0 msgr2=0x7fddb005c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:10.707 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.692+0000 7fddb5b38700 1 --2- 172.21.15.92:0/3654056756 >> v2:172.21.15.60:3300/0 conn(0x7fddb005c3f0 0x7fddb005c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.707 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.692+0000 7fddb5b38700 1 -- 172.21.15.92:0/3654056756 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb00af2d0 con 0x7fddb005f2c0 2022-06-29T15:58:10.708 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.692+0000 7fddb6339700 1 --2- 172.21.15.92:0/3654056756 >> v2:172.21.15.60:3300/0 conn(0x7fddb005c3f0 0x7fddb005c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:58:10.708 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.692+0000 7fddb5337700 1 --2- 172.21.15.92:0/3654056756 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fddb005ff90 0x7fddb00aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:10.708 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.696+0000 7fddb5b38700 1 --2- 172.21.15.92:0/3654056756 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fddb005f2c0 0x7fddb005f6a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fddac002f70 tx=0x7fddac006eb0).ready entity=mon.1 client_cookie=d8c9fad885c2baf1 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:10.708 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.696+0000 7fddb4b36700 1 -- 172.21.15.92:0/3654056756 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fddac00d570 con 0x7fddb005f2c0 2022-06-29T15:58:10.709 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.696+0000 7fddb4b36700 1 -- 172.21.15.92:0/3654056756 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddac000e90 con 0x7fddb005f2c0 2022-06-29T15:58:10.709 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.696+0000 7fddb4b36700 1 -- 172.21.15.92:0/3654056756 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fddac00d900 con 0x7fddb005f2c0 2022-06-29T15:58:10.709 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.696+0000 7fddb6b3a700 1 -- 172.21.15.92:0/3654056756 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fddb005f2c0 msgr2=0x7fddb005f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:10.709 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.696+0000 7fddb6b3a700 1 --2- 172.21.15.92:0/3654056756 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fddb005f2c0 0x7fddb005f6a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fddac002f70 tx=0x7fddac006eb0).stop 2022-06-29T15:58:10.710 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.696+0000 7fddb6b3a700 1 -- 172.21.15.92:0/3654056756 shutdown_connections 2022-06-29T15:58:10.710 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.696+0000 7fddb6b3a700 1 --2- 172.21.15.92:0/3654056756 >> v2:172.21.15.60:3300/0 conn(0x7fddb005c3f0 0x7fddb005c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.710 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.696+0000 7fddb6b3a700 1 --2- 172.21.15.92:0/3654056756 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fddb005ff90 0x7fddb00aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.710 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.696+0000 7fddb6b3a700 1 --2- 172.21.15.92:0/3654056756 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fddb005f2c0 0x7fddb005f6a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.711 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.696+0000 7fddb6b3a700 1 -- 172.21.15.92:0/3654056756 >> 172.21.15.92:0/3654056756 conn(0x7fddb00b91e0 msgr2=0x7fddb00b95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:10.711 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.696+0000 7fddb6b3a700 1 -- 172.21.15.92:0/3654056756 shutdown_connections 2022-06-29T15:58:10.711 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.696+0000 7fddb6b3a700 1 -- 172.21.15.92:0/3654056756 wait complete. 2022-06-29T15:58:10.711 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.696+0000 7fddb6b3a700 1 Processor -- start 2022-06-29T15:58:10.712 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.696+0000 7fddb6b3a700 1 -- start start 2022-06-29T15:58:10.712 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.696+0000 7fddb6b3a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fddb005c3f0 0x7fddb009e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:10.712 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.696+0000 7fddb6b3a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddb005f2c0 0x7fddb0155450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:10.712 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.696+0000 7fddb6b3a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fddb005ff90 0x7fddb0155940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:10.713 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.696+0000 7fddb6b3a700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb009cc00 con 0x7fddb005f2c0 2022-06-29T15:58:10.713 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.696+0000 7fddb6b3a700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb009ea70 con 0x7fddb005ff90 2022-06-29T15:58:10.713 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.696+0000 7fddb6b3a700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddb009c4c0 con 0x7fddb005c3f0 2022-06-29T15:58:10.713 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.696+0000 7fddb5b38700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fddb005c3f0 0x7fddb009e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:10.714 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.696+0000 7fddb6339700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fddb005ff90 0x7fddb0155940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:10.714 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.696+0000 7fddb6339700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fddb005ff90 0x7fddb0155940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:45940/0 (socket says 172.21.15.92:45940) 2022-06-29T15:58:10.714 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.696+0000 7fddb6339700 1 -- 172.21.15.92:0/2415187618 learned_addr learned my addr 172.21.15.92:0/2415187618 (peer_addr_for_me v2:172.21.15.92:0/0) 2022-06-29T15:58:10.714 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.696+0000 7fddb5337700 1 --2- 172.21.15.92:0/2415187618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddb005f2c0 0x7fddb0155450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:10.715 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.700+0000 7fddb5b38700 1 -- 172.21.15.92:0/2415187618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fddb005ff90 msgr2=0x7fddb0155940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:10.715 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.700+0000 7fddb5b38700 1 --2- 172.21.15.92:0/2415187618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fddb005ff90 0x7fddb0155940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.715 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.700+0000 7fddb5b38700 1 -- 172.21.15.92:0/2415187618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddb005f2c0 msgr2=0x7fddb0155450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:10.715 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.700+0000 7fddb5b38700 1 --2- 172.21.15.92:0/2415187618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddb005f2c0 0x7fddb0155450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.716 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.700+0000 7fddb5b38700 1 -- 172.21.15.92:0/2415187618 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda000b180 con 0x7fddb005c3f0 2022-06-29T15:58:10.716 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.700+0000 7fddb5b38700 1 --2- 172.21.15.92:0/2415187618 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fddb005c3f0 0x7fddb009e140 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fddac002f70 tx=0x7fddac015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:10.716 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.700+0000 7fdd9effd700 1 -- 172.21.15.92:0/2415187618 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fddac0068b0 con 0x7fddb005c3f0 2022-06-29T15:58:10.716 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.700+0000 7fdd9effd700 1 -- 172.21.15.92:0/2415187618 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddac006b40 con 0x7fddb005c3f0 2022-06-29T15:58:10.717 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.700+0000 7fddb6b3a700 1 -- 172.21.15.92:0/2415187618 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddac00b1e0 con 0x7fddb005c3f0 2022-06-29T15:58:10.717 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.700+0000 7fdd9effd700 1 -- 172.21.15.92:0/2415187618 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fddac011d30 con 0x7fddb005c3f0 2022-06-29T15:58:10.717 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.700+0000 7fddb6b3a700 1 -- 172.21.15.92:0/2415187618 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddb009e630 con 0x7fddb005c3f0 2022-06-29T15:58:10.717 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.700+0000 7fdd9effd700 1 -- 172.21.15.92:0/2415187618 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 1) v1 ==== 345+0+0 (secure 0 0 0) 0x7fddac01a040 con 0x7fddb005c3f0 2022-06-29T15:58:10.718 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.700+0000 7fdd9effd700 1 -- 172.21.15.92:0/2415187618 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1559+0+0 (secure 0 0 0) 0x7fddac01e630 con 0x7fddb005c3f0 2022-06-29T15:58:10.718 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.700+0000 7fddb6b3a700 1 -- 172.21.15.92:0/2415187618 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddb00573d0 con 0x7fddb005c3f0 2022-06-29T15:58:10.718 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.704+0000 7fdd9effd700 1 -- 172.21.15.92:0/2415187618 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67674 (secure 0 0 0) 0x7fddac027450 con 0x7fddb005c3f0 2022-06-29T15:58:10.766 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.764+0000 7fddb6b3a700 1 -- 172.21.15.92:0/2415187618 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd new", "uuid": "bd513a9c-3b19-4314-9897-d300b5c897bb", "id": 7} v 0) v1 -- 0x7fddb009e850 con 0x7fddb005c3f0 2022-06-29T15:58:10.772 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.768+0000 7fdd9effd700 1 -- 172.21.15.92:0/2415187618 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bd513a9c-3b19-4314-9897-d300b5c897bb", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fddac02d020 con 0x7fddb005c3f0 2022-06-29T15:58:10.773 INFO:teuthology.orchestra.run.smithi092.stdout:7 2022-06-29T15:58:10.774 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.772+0000 7fddb6b3a700 1 -- 172.21.15.92:0/2415187618 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fddb005c3f0 msgr2=0x7fddb009e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:10.774 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.772+0000 7fddb6b3a700 1 --2- 172.21.15.92:0/2415187618 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fddb005c3f0 0x7fddb009e140 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fddac002f70 tx=0x7fddac015000).stop 2022-06-29T15:58:10.775 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.772+0000 7fddb6b3a700 1 -- 172.21.15.92:0/2415187618 shutdown_connections 2022-06-29T15:58:10.775 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.772+0000 7fddb6b3a700 1 --2- 172.21.15.92:0/2415187618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fddb005ff90 0x7fddb0155940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.775 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.772+0000 7fddb6b3a700 1 --2- 172.21.15.92:0/2415187618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddb005f2c0 0x7fddb0155450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.775 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.772+0000 7fddb6b3a700 1 --2- 172.21.15.92:0/2415187618 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fddb005c3f0 0x7fddb009e140 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:10.776 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.772+0000 7fddb6b3a700 1 -- 172.21.15.92:0/2415187618 >> 172.21.15.92:0/2415187618 conn(0x7fddb00b91e0 msgr2=0x7fddb0140cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:10.776 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.772+0000 7fddb6b3a700 1 -- 172.21.15.92:0/2415187618 shutdown_connections 2022-06-29T15:58:10.776 INFO:teuthology.orchestra.run.smithi092.stderr:2022-06-29T15:58:10.772+0000 7fddb6b3a700 1 -- 172.21.15.92:0/2415187618 wait complete. 2022-06-29T15:58:10.784 INFO:tasks.ceph.osd.0:Restarting daemon 2022-06-29T15:58:10.785 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2022-06-29T15:58:10.787 INFO:tasks.ceph.osd.0:Started 2022-06-29T15:58:10.788 INFO:tasks.ceph.osd.1:Restarting daemon 2022-06-29T15:58:10.788 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2022-06-29T15:58:10.792 INFO:tasks.ceph.osd.1:Started 2022-06-29T15:58:10.792 INFO:tasks.ceph.osd.2:Restarting daemon 2022-06-29T15:58:10.793 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2022-06-29T15:58:10.795 INFO:tasks.ceph.osd.2:Started 2022-06-29T15:58:10.796 INFO:tasks.ceph.osd.3:Restarting daemon 2022-06-29T15:58:10.797 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2022-06-29T15:58:10.800 INFO:tasks.ceph.osd.3:Started 2022-06-29T15:58:10.800 INFO:tasks.ceph.osd.4:Restarting daemon 2022-06-29T15:58:10.801 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2022-06-29T15:58:10.805 INFO:tasks.ceph.osd.4:Started 2022-06-29T15:58:10.805 INFO:tasks.ceph.osd.5:Restarting daemon 2022-06-29T15:58:10.806 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2022-06-29T15:58:10.808 INFO:tasks.ceph.osd.5:Started 2022-06-29T15:58:10.808 INFO:tasks.ceph.osd.6:Restarting daemon 2022-06-29T15:58:10.809 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2022-06-29T15:58:10.811 INFO:tasks.ceph.osd.6:Started 2022-06-29T15:58:10.811 INFO:tasks.ceph.osd.7:Restarting daemon 2022-06-29T15:58:10.812 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2022-06-29T15:58:10.815 INFO:tasks.ceph.osd.7:Started 2022-06-29T15:58:10.815 INFO:tasks.ceph:Waiting for OSDs to come up 2022-06-29T15:58:10.853 INFO:tasks.ceph.osd.3.smithi060.stderr:2022-06-29T15:58:10.849+0000 7f820dd04d80 -1 Falling back to public interface 2022-06-29T15:58:10.864 INFO:tasks.ceph.osd.2.smithi060.stderr:2022-06-29T15:58:10.857+0000 7f6e6350dd80 -1 Falling back to public interface 2022-06-29T15:58:10.865 INFO:tasks.ceph.osd.1.smithi060.stderr:2022-06-29T15:58:10.861+0000 7f4809520d80 -1 Falling back to public interface 2022-06-29T15:58:10.867 INFO:tasks.ceph.osd.0.smithi060.stderr:2022-06-29T15:58:10.861+0000 7f532188dd80 -1 Falling back to public interface 2022-06-29T15:58:10.868 INFO:tasks.ceph.osd.3.smithi060.stderr:2022-06-29T15:58:10.861+0000 7f820dd04d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-06-29T15:58:10.868 INFO:tasks.ceph.osd.3.smithi060.stderr:2022-06-29T15:58:10.861+0000 7f820dd04d80 -1 journal do_read_entry(8192): bad header magic 2022-06-29T15:58:10.868 INFO:tasks.ceph.osd.3.smithi060.stderr:2022-06-29T15:58:10.861+0000 7f820dd04d80 -1 journal do_read_entry(8192): bad header magic 2022-06-29T15:58:10.871 INFO:tasks.ceph.osd.3.smithi060.stderr:2022-06-29T15:58:10.865+0000 7f820dd04d80 -1 osd.3 0 log_to_monitors {default=true} 2022-06-29T15:58:10.876 INFO:tasks.ceph.osd.7.smithi092.stderr:2022-06-29T15:58:10.872+0000 7f552c39bd80 -1 Falling back to public interface 2022-06-29T15:58:10.879 INFO:tasks.ceph.osd.1.smithi060.stderr:2022-06-29T15:58:10.873+0000 7f4809520d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-06-29T15:58:10.879 INFO:tasks.ceph.osd.1.smithi060.stderr:2022-06-29T15:58:10.873+0000 7f4809520d80 -1 journal do_read_entry(8192): bad header magic 2022-06-29T15:58:10.879 INFO:tasks.ceph.osd.1.smithi060.stderr:2022-06-29T15:58:10.873+0000 7f4809520d80 -1 journal do_read_entry(8192): bad header magic 2022-06-29T15:58:10.880 INFO:tasks.ceph.osd.2.smithi060.stderr:2022-06-29T15:58:10.873+0000 7f6e6350dd80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-06-29T15:58:10.880 INFO:tasks.ceph.osd.2.smithi060.stderr:2022-06-29T15:58:10.873+0000 7f6e6350dd80 -1 journal do_read_entry(8192): bad header magic 2022-06-29T15:58:10.881 INFO:tasks.ceph.osd.2.smithi060.stderr:2022-06-29T15:58:10.873+0000 7f6e6350dd80 -1 journal do_read_entry(8192): bad header magic 2022-06-29T15:58:10.883 INFO:tasks.ceph.osd.1.smithi060.stderr:2022-06-29T15:58:10.877+0000 7f4809520d80 -1 osd.1 0 log_to_monitors {default=true} 2022-06-29T15:58:10.884 INFO:tasks.ceph.osd.2.smithi060.stderr:2022-06-29T15:58:10.877+0000 7f6e6350dd80 -1 osd.2 0 log_to_monitors {default=true} 2022-06-29T15:58:10.887 INFO:tasks.ceph.osd.0.smithi060.stderr:2022-06-29T15:58:10.881+0000 7f532188dd80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-06-29T15:58:10.887 INFO:tasks.ceph.osd.0.smithi060.stderr:2022-06-29T15:58:10.881+0000 7f532188dd80 -1 journal do_read_entry(8192): bad header magic 2022-06-29T15:58:10.887 INFO:tasks.ceph.osd.0.smithi060.stderr:2022-06-29T15:58:10.881+0000 7f532188dd80 -1 journal do_read_entry(8192): bad header magic 2022-06-29T15:58:10.888 INFO:tasks.ceph.osd.4.smithi092.stderr:2022-06-29T15:58:10.884+0000 7f3a9d409d80 -1 Falling back to public interface 2022-06-29T15:58:10.889 INFO:tasks.ceph.osd.5.smithi092.stderr:2022-06-29T15:58:10.884+0000 7fb3658a4d80 -1 Falling back to public interface 2022-06-29T15:58:10.889 INFO:tasks.ceph.osd.6.smithi092.stderr:2022-06-29T15:58:10.884+0000 7fcbb5fe9d80 -1 Falling back to public interface 2022-06-29T15:58:10.891 INFO:tasks.ceph.osd.0.smithi060.stderr:2022-06-29T15:58:10.885+0000 7f532188dd80 -1 osd.0 0 log_to_monitors {default=true} 2022-06-29T15:58:10.892 INFO:tasks.ceph.osd.7.smithi092.stderr:2022-06-29T15:58:10.888+0000 7f552c39bd80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-06-29T15:58:10.893 INFO:tasks.ceph.osd.7.smithi092.stderr:2022-06-29T15:58:10.888+0000 7f552c39bd80 -1 journal do_read_entry(8192): bad header magic 2022-06-29T15:58:10.893 INFO:tasks.ceph.osd.7.smithi092.stderr:2022-06-29T15:58:10.888+0000 7f552c39bd80 -1 journal do_read_entry(8192): bad header magic 2022-06-29T15:58:10.897 INFO:tasks.ceph.osd.7.smithi092.stderr:2022-06-29T15:58:10.896+0000 7f552c39bd80 -1 osd.7 0 log_to_monitors {default=true} 2022-06-29T15:58:10.901 INFO:tasks.ceph.osd.5.smithi092.stderr:2022-06-29T15:58:10.900+0000 7fb3658a4d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-06-29T15:58:10.902 INFO:tasks.ceph.osd.5.smithi092.stderr:2022-06-29T15:58:10.900+0000 7fb3658a4d80 -1 journal do_read_entry(8192): bad header magic 2022-06-29T15:58:10.902 INFO:tasks.ceph.osd.5.smithi092.stderr:2022-06-29T15:58:10.900+0000 7fb3658a4d80 -1 journal do_read_entry(8192): bad header magic 2022-06-29T15:58:10.903 INFO:tasks.ceph.osd.4.smithi092.stderr:2022-06-29T15:58:10.900+0000 7f3a9d409d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-06-29T15:58:10.903 INFO:tasks.ceph.osd.4.smithi092.stderr:2022-06-29T15:58:10.900+0000 7f3a9d409d80 -1 journal do_read_entry(8192): bad header magic 2022-06-29T15:58:10.903 INFO:tasks.ceph.osd.4.smithi092.stderr:2022-06-29T15:58:10.900+0000 7f3a9d409d80 -1 journal do_read_entry(8192): bad header magic 2022-06-29T15:58:10.904 INFO:tasks.ceph.osd.6.smithi092.stderr:2022-06-29T15:58:10.900+0000 7fcbb5fe9d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-06-29T15:58:10.904 INFO:tasks.ceph.osd.6.smithi092.stderr:2022-06-29T15:58:10.900+0000 7fcbb5fe9d80 -1 journal do_read_entry(8192): bad header magic 2022-06-29T15:58:10.904 INFO:tasks.ceph.osd.6.smithi092.stderr:2022-06-29T15:58:10.900+0000 7fcbb5fe9d80 -1 journal do_read_entry(8192): bad header magic 2022-06-29T15:58:10.906 INFO:tasks.ceph.osd.5.smithi092.stderr:2022-06-29T15:58:10.904+0000 7fb3658a4d80 -1 osd.5 0 log_to_monitors {default=true} 2022-06-29T15:58:10.906 INFO:tasks.ceph.osd.4.smithi092.stderr:2022-06-29T15:58:10.904+0000 7f3a9d409d80 -1 osd.4 0 log_to_monitors {default=true} 2022-06-29T15:58:10.908 INFO:tasks.ceph.osd.6.smithi092.stderr:2022-06-29T15:58:10.904+0000 7fcbb5fe9d80 -1 osd.6 0 log_to_monitors {default=true} 2022-06-29T15:58:11.620 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2022-06-29T15:58:11.719 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.713+0000 7ff39bfff700 1 Processor -- start 2022-06-29T15:58:11.719 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.713+0000 7ff39bfff700 1 -- start start 2022-06-29T15:58:11.723 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.713+0000 7ff39bfff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff39c142550 0x7ff39c146780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:11.723 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.713+0000 7ff39bfff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff39c0b63f0 con 0x7ff39c0aef80 2022-06-29T15:58:11.724 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.713+0000 7ff39bfff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff39c1429a0 con 0x7ff39c0b6010 2022-06-29T15:58:11.724 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.713+0000 7ff39bfff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff39c0aeda0 con 0x7ff39c142550 2022-06-29T15:58:11.724 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.717+0000 7ff39a7fc700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff39c0aef80 0x7ff39c146070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52630/0 (socket says 172.21.15.60:52630) 2022-06-29T15:58:11.724 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.717+0000 7ff39a7fc700 1 -- 172.21.15.60:0/372590046 learned_addr learned my addr 172.21.15.60:0/372590046 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:11.725 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.717+0000 7ff39b7fe700 1 --2- 172.21.15.60:0/372590046 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff39c142550 0x7ff39c146780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:11.725 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.717+0000 7ff399ffb700 1 -- 172.21.15.60:0/372590046 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3902344344 0 0) 0x7ff39c1429a0 con 0x7ff39c0b6010 2022-06-29T15:58:11.725 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.717+0000 7ff399ffb700 1 -- 172.21.15.60:0/372590046 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff380003260 con 0x7ff39c0b6010 2022-06-29T15:58:11.725 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.717+0000 7ff399ffb700 1 -- 172.21.15.60:0/372590046 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2787625447 0 0) 0x7ff39c0b63f0 con 0x7ff39c0aef80 2022-06-29T15:58:11.725 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.717+0000 7ff399ffb700 1 -- 172.21.15.60:0/372590046 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff39c1429a0 con 0x7ff39c0aef80 2022-06-29T15:58:11.726 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.717+0000 7ff39b7fe700 1 -- 172.21.15.60:0/372590046 >> v1:172.21.15.92:6789/0 conn(0x7ff39c0b6010 legacy=0x7ff39c141e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:11.726 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.717+0000 7ff39b7fe700 1 -- 172.21.15.60:0/372590046 >> v1:172.21.15.60:6789/0 conn(0x7ff39c0aef80 legacy=0x7ff39c146070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:11.726 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.717+0000 7ff39b7fe700 1 -- 172.21.15.60:0/372590046 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff39c0ae3c0 con 0x7ff39c142550 2022-06-29T15:58:11.726 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.717+0000 7ff39b7fe700 1 --2- 172.21.15.60:0/372590046 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff39c142550 0x7ff39c146780 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7ff3940032f0 tx=0x7ff3940091d0).ready entity=mon.2 client_cookie=7777533f981951a4 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:11.727 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.717+0000 7ff399ffb700 1 -- 172.21.15.60:0/372590046 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff39400b640 con 0x7ff39c142550 2022-06-29T15:58:11.727 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.717+0000 7ff399ffb700 1 -- 172.21.15.60:0/372590046 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff39400b7a0 con 0x7ff39c142550 2022-06-29T15:58:11.728 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.717+0000 7ff399ffb700 1 -- 172.21.15.60:0/372590046 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff39400b910 con 0x7ff39c142550 2022-06-29T15:58:11.731 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.717+0000 7ff39bfff700 1 -- 172.21.15.60:0/372590046 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff39c142550 msgr2=0x7ff39c146780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:11.732 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.717+0000 7ff39bfff700 1 --2- 172.21.15.60:0/372590046 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff39c142550 0x7ff39c146780 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7ff3940032f0 tx=0x7ff3940091d0).stop 2022-06-29T15:58:11.732 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.717+0000 7ff39bfff700 1 -- 172.21.15.60:0/372590046 shutdown_connections 2022-06-29T15:58:11.732 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.717+0000 7ff39bfff700 1 --2- 172.21.15.60:0/372590046 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff39c142550 0x7ff39c146780 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:11.733 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.717+0000 7ff39bfff700 1 -- 172.21.15.60:0/372590046 >> 172.21.15.60:0/372590046 conn(0x7ff39c063650 msgr2=0x7ff39c0b55c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:11.733 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.717+0000 7ff39bfff700 1 -- 172.21.15.60:0/372590046 shutdown_connections 2022-06-29T15:58:11.733 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.717+0000 7ff39bfff700 1 -- 172.21.15.60:0/372590046 wait complete. 2022-06-29T15:58:11.734 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.721+0000 7ff39bfff700 1 Processor -- start 2022-06-29T15:58:11.734 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.721+0000 7ff39bfff700 1 -- start start 2022-06-29T15:58:11.734 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.721+0000 7ff39bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff39c0aef80 0x7ff39c13eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:11.734 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.721+0000 7ff39bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff39c0b6010 0x7ff39c065200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:11.735 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.721+0000 7ff39bfff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff39c142550 0x7ff39c0656f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:11.735 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.721+0000 7ff39bfff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff39c1461c0 con 0x7ff39c0b6010 2022-06-29T15:58:11.735 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.721+0000 7ff39bfff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff39c145f20 con 0x7ff39c0aef80 2022-06-29T15:58:11.735 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.721+0000 7ff39bfff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff39c146c80 con 0x7ff39c142550 2022-06-29T15:58:11.735 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.721+0000 7ff39a7fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff39c0b6010 0x7ff39c065200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:11.736 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.721+0000 7ff39a7fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff39c0b6010 0x7ff39c065200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44550/0 (socket says 172.21.15.60:44550) 2022-06-29T15:58:11.736 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.721+0000 7ff39a7fc700 1 -- 172.21.15.60:0/474293927 learned_addr learned my addr 172.21.15.60:0/474293927 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:11.736 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.721+0000 7ff39affd700 1 --2- 172.21.15.60:0/474293927 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff39c0aef80 0x7ff39c13eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:11.736 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.721+0000 7ff39b7fe700 1 --2- 172.21.15.60:0/474293927 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff39c142550 0x7ff39c0656f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:11.737 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.721+0000 7ff39a7fc700 1 -- 172.21.15.60:0/474293927 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff39c142550 msgr2=0x7ff39c0656f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:11.737 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.721+0000 7ff39a7fc700 1 --2- 172.21.15.60:0/474293927 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff39c142550 0x7ff39c0656f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:11.737 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.721+0000 7ff39a7fc700 1 -- 172.21.15.60:0/474293927 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff39c0aef80 msgr2=0x7ff39c13eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:11.737 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.721+0000 7ff39a7fc700 1 --2- 172.21.15.60:0/474293927 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff39c0aef80 0x7ff39c13eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:11.738 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.721+0000 7ff39a7fc700 1 -- 172.21.15.60:0/474293927 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff394009470 con 0x7ff39c0b6010 2022-06-29T15:58:11.738 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.721+0000 7ff39b7fe700 1 --2- 172.21.15.60:0/474293927 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff39c142550 0x7ff39c0656f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T15:58:11.738 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.721+0000 7ff39a7fc700 1 --2- 172.21.15.60:0/474293927 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff39c0b6010 0x7ff39c065200 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7ff390001110 tx=0x7ff390007aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:11.738 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.721+0000 7ff38bfff700 1 -- 172.21.15.60:0/474293927 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3900070d0 con 0x7ff39c0b6010 2022-06-29T15:58:11.739 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.721+0000 7ff38bfff700 1 -- 172.21.15.60:0/474293927 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff390007230 con 0x7ff39c0b6010 2022-06-29T15:58:11.739 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.721+0000 7ff39bfff700 1 -- 172.21.15.60:0/474293927 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff39c145ba0 con 0x7ff39c0b6010 2022-06-29T15:58:11.739 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.721+0000 7ff38bfff700 1 -- 172.21.15.60:0/474293927 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff390005610 con 0x7ff39c0b6010 2022-06-29T15:58:11.739 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.721+0000 7ff39bfff700 1 -- 172.21.15.60:0/474293927 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff39c067950 con 0x7ff39c0b6010 2022-06-29T15:58:11.740 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.725+0000 7ff38bfff700 1 -- 172.21.15.60:0/474293927 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 2) v1 ==== 32894+0+0 (secure 0 0 0) 0x7ff390010070 con 0x7ff39c0b6010 2022-06-29T15:58:11.740 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.725+0000 7ff38bfff700 1 -- 172.21.15.60:0/474293927 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1674+0+0 (secure 0 0 0) 0x7ff390039c00 con 0x7ff39c0b6010 2022-06-29T15:58:11.740 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.725+0000 7ff39bfff700 1 -- 172.21.15.60:0/474293927 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff39c146e40 con 0x7ff39c0b6010 2022-06-29T15:58:11.741 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.729+0000 7ff38bfff700 1 -- 172.21.15.60:0/474293927 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67674 (secure 0 0 0) 0x7ff39001a8e0 con 0x7ff39c0b6010 2022-06-29T15:58:11.797 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.789+0000 7ff39bfff700 1 -- 172.21.15.60:0/474293927 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff39c0573d0 con 0x7ff39c0b6010 2022-06-29T15:58:11.797 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.793+0000 7ff38bfff700 1 -- 172.21.15.60:0/474293927 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+8254 (secure 0 0 0) 0x7ff390036090 con 0x7ff39c0b6010 2022-06-29T15:58:11.798 INFO:teuthology.misc.health.smithi060.stdout: 2022-06-29T15:58:11.798 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":11,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T15:58:11.766061+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T15:58:11.799 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.793+0000 7ff39bfff700 1 -- 172.21.15.60:0/474293927 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff39c0b6010 msgr2=0x7ff39c065200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:11.800 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.793+0000 7ff39bfff700 1 --2- 172.21.15.60:0/474293927 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff39c0b6010 0x7ff39c065200 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7ff390001110 tx=0x7ff390007aa0).stop 2022-06-29T15:58:11.800 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.793+0000 7ff39bfff700 1 -- 172.21.15.60:0/474293927 shutdown_connections 2022-06-29T15:58:11.800 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.793+0000 7ff39bfff700 1 --2- 172.21.15.60:0/474293927 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff39c142550 0x7ff39c0656f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:11.800 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.793+0000 7ff39bfff700 1 --2- 172.21.15.60:0/474293927 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff39c0b6010 0x7ff39c065200 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:11.801 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.793+0000 7ff39bfff700 1 --2- 172.21.15.60:0/474293927 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff39c0aef80 0x7ff39c13eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:11.801 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.793+0000 7ff39bfff700 1 -- 172.21.15.60:0/474293927 >> 172.21.15.60:0/474293927 conn(0x7ff39c063650 msgr2=0x7ff39c148a80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:11.801 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.793+0000 7ff39bfff700 1 -- 172.21.15.60:0/474293927 shutdown_connections 2022-06-29T15:58:11.801 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:11.793+0000 7ff39bfff700 1 -- 172.21.15.60:0/474293927 wait complete. 2022-06-29T15:58:11.809 DEBUG:teuthology.misc:0 of 8 OSDs are up 2022-06-29T15:58:12.789 INFO:tasks.ceph.osd.3.smithi060.stderr:2022-06-29T15:58:12.781+0000 7f81f7dc7700 -1 osd.3 0 waiting for initial osdmap 2022-06-29T15:58:12.790 INFO:tasks.ceph.osd.1.smithi060.stderr:2022-06-29T15:58:12.785+0000 7f47f35e3700 -1 osd.1 0 waiting for initial osdmap 2022-06-29T15:58:12.790 INFO:tasks.ceph.osd.0.smithi060.stderr:2022-06-29T15:58:12.785+0000 7f530b950700 -1 osd.0 0 waiting for initial osdmap 2022-06-29T15:58:12.791 INFO:tasks.ceph.osd.2.smithi060.stderr:2022-06-29T15:58:12.785+0000 7f6e4d5d0700 -1 osd.2 0 waiting for initial osdmap 2022-06-29T15:58:12.795 INFO:tasks.ceph.osd.7.smithi092.stderr:2022-06-29T15:58:12.792+0000 7f551645e700 -1 osd.7 0 waiting for initial osdmap 2022-06-29T15:58:12.796 INFO:tasks.ceph.osd.6.smithi092.stderr:2022-06-29T15:58:12.792+0000 7fcba00ac700 -1 osd.6 0 waiting for initial osdmap 2022-06-29T15:58:12.796 INFO:tasks.ceph.osd.4.smithi092.stderr:2022-06-29T15:58:12.792+0000 7f3a874cc700 -1 osd.4 0 waiting for initial osdmap 2022-06-29T15:58:12.797 INFO:tasks.ceph.osd.5.smithi092.stderr:2022-06-29T15:58:12.792+0000 7fb34f967700 -1 osd.5 0 waiting for initial osdmap 2022-06-29T15:58:12.798 INFO:tasks.ceph.osd.1.smithi060.stderr:2022-06-29T15:58:12.789+0000 7f4802a66700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-06-29T15:58:12.799 INFO:tasks.ceph.osd.3.smithi060.stderr:2022-06-29T15:58:12.789+0000 7f820724a700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-06-29T15:58:12.800 INFO:tasks.ceph.osd.2.smithi060.stderr:2022-06-29T15:58:12.789+0000 7f6e5ca53700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-06-29T15:58:12.800 INFO:tasks.ceph.osd.6.smithi092.stderr:2022-06-29T15:58:12.796+0000 7fcbaf52f700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-06-29T15:58:12.801 INFO:tasks.ceph.osd.5.smithi092.stderr:2022-06-29T15:58:12.800+0000 7fb35edea700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-06-29T15:58:12.814 INFO:tasks.ceph.osd.0.smithi060.stderr:2022-06-29T15:58:12.809+0000 7f531add3700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-06-29T15:58:12.829 INFO:tasks.ceph.osd.4.smithi092.stderr:2022-06-29T15:58:12.828+0000 7f3a9694f700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-06-29T15:58:12.841 INFO:tasks.ceph.osd.7.smithi092.stderr:2022-06-29T15:58:12.840+0000 7f55258e1700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-06-29T15:58:18.615 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2022-06-29T15:58:18.722 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.713+0000 7f1ab2ab8700 1 Processor -- start 2022-06-29T15:58:18.723 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.713+0000 7f1ab2ab8700 1 -- start start 2022-06-29T15:58:18.723 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.713+0000 7f1ab2ab8700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1aac0b6010 0x7f1aac0b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:18.723 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.713+0000 7f1ab2ab8700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1aac0abba0 0x7f1aac145d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:18.724 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.713+0000 7f1ab2ab8700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aac0b63f0 con 0x7f1aac0aacd0 2022-06-29T15:58:18.724 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.713+0000 7f1ab2ab8700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aac0b1ec0 con 0x7f1aac0abba0 2022-06-29T15:58:18.725 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.713+0000 7f1ab2ab8700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aac14b3e0 con 0x7f1aac0b6010 2022-06-29T15:58:18.725 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.713+0000 7f1ab22b7700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1aac0abba0 0x7f1aac145d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:18.726 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.713+0000 7f1ab22b7700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1aac0abba0 0x7f1aac145d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44758/0 (socket says 172.21.15.60:44758) 2022-06-29T15:58:18.726 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.713+0000 7f1ab22b7700 1 -- 172.21.15.60:0/1166261452 learned_addr learned my addr 172.21.15.60:0/1166261452 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:18.727 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.713+0000 7f1ab1ab6700 1 --2- 172.21.15.60:0/1166261452 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1aac0b6010 0x7f1aac0b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:18.727 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.713+0000 7f1ab22b7700 1 -- 172.21.15.60:0/1166261452 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1aac0b6010 msgr2=0x7f1aac0b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:18.727 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.713+0000 7f1ab0ab4700 1 -- 172.21.15.60:0/1166261452 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2238071176 0 0) 0x7f1aac0b63f0 con 0x7f1aac0aacd0 2022-06-29T15:58:18.728 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.713+0000 7f1ab22b7700 1 --2- 172.21.15.60:0/1166261452 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1aac0b6010 0x7f1aac0b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:18.728 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.713+0000 7f1ab22b7700 1 -- 172.21.15.60:0/1166261452 >> v1:172.21.15.92:6789/0 conn(0x7f1aac0aacd0 legacy=0x7f1aac0b17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:18.729 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.713+0000 7f1ab22b7700 1 -- 172.21.15.60:0/1166261452 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aac146dc0 con 0x7f1aac0abba0 2022-06-29T15:58:18.729 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.713+0000 7f1ab22b7700 1 --2- 172.21.15.60:0/1166261452 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1aac0abba0 0x7f1aac145d10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f1aa40032f0 tx=0x7f1aa4009290).ready entity=mon.0 client_cookie=72cf492b425e2716 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:18.730 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.713+0000 7f1ab0ab4700 1 -- 172.21.15.60:0/1166261452 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1aa400b8a0 con 0x7f1aac0abba0 2022-06-29T15:58:18.730 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.713+0000 7f1ab0ab4700 1 -- 172.21.15.60:0/1166261452 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa400ba00 con 0x7f1aac0abba0 2022-06-29T15:58:18.731 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.713+0000 7f1ab0ab4700 1 -- 172.21.15.60:0/1166261452 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1aa400bb70 con 0x7f1aac0abba0 2022-06-29T15:58:18.731 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.713+0000 7f1ab2ab8700 1 -- 172.21.15.60:0/1166261452 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1aac0abba0 msgr2=0x7f1aac145d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:18.732 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.713+0000 7f1ab2ab8700 1 --2- 172.21.15.60:0/1166261452 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1aac0abba0 0x7f1aac145d10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f1aa40032f0 tx=0x7f1aa4009290).stop 2022-06-29T15:58:18.732 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.713+0000 7f1ab2ab8700 1 -- 172.21.15.60:0/1166261452 shutdown_connections 2022-06-29T15:58:18.732 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.713+0000 7f1ab2ab8700 1 --2- 172.21.15.60:0/1166261452 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1aac0abba0 0x7f1aac145d10 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:18.733 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.713+0000 7f1ab2ab8700 1 --2- 172.21.15.60:0/1166261452 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1aac0b6010 0x7f1aac0b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:18.733 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.713+0000 7f1ab2ab8700 1 -- 172.21.15.60:0/1166261452 >> 172.21.15.60:0/1166261452 conn(0x7f1aac063650 msgr2=0x7f1aac0b55c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:18.734 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.713+0000 7f1ab2ab8700 1 -- 172.21.15.60:0/1166261452 shutdown_connections 2022-06-29T15:58:18.734 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.713+0000 7f1ab2ab8700 1 -- 172.21.15.60:0/1166261452 wait complete. 2022-06-29T15:58:18.744 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.717+0000 7f1ab2ab8700 1 Processor -- start 2022-06-29T15:58:18.745 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.717+0000 7f1ab2ab8700 1 -- start start 2022-06-29T15:58:18.745 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.717+0000 7f1ab2ab8700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1aac0aacd0 0x7f1aac149850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:18.746 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.717+0000 7f1ab2ab8700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1aac0abba0 0x7f1aac1407a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:18.746 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.717+0000 7f1ab2ab8700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1aac0b6010 0x7f1aac140c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:18.746 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.717+0000 7f1ab2ab8700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aac145a40 con 0x7f1aac0aacd0 2022-06-29T15:58:18.747 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.717+0000 7f1ab2ab8700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aac143f20 con 0x7f1aac0abba0 2022-06-29T15:58:18.747 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.717+0000 7f1ab2ab8700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aac144150 con 0x7f1aac0b6010 2022-06-29T15:58:18.748 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.717+0000 7f1ab1ab6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1aac0aacd0 0x7f1aac149850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:18.748 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.717+0000 7f1ab1ab6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1aac0aacd0 0x7f1aac149850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44762/0 (socket says 172.21.15.60:44762) 2022-06-29T15:58:18.749 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.717+0000 7f1ab1ab6700 1 -- 172.21.15.60:0/3583023527 learned_addr learned my addr 172.21.15.60:0/3583023527 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:18.749 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.717+0000 7f1ab22b7700 1 --2- 172.21.15.60:0/3583023527 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1aac0b6010 0x7f1aac140c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:18.749 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.717+0000 7f1ab12b5700 1 --2- 172.21.15.60:0/3583023527 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1aac0abba0 0x7f1aac1407a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:18.750 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.721+0000 7f1ab1ab6700 1 -- 172.21.15.60:0/3583023527 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1aac0b6010 msgr2=0x7f1aac140c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:18.750 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.721+0000 7f1ab1ab6700 1 --2- 172.21.15.60:0/3583023527 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1aac0b6010 0x7f1aac140c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:18.751 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.721+0000 7f1ab1ab6700 1 -- 172.21.15.60:0/3583023527 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1aac0abba0 msgr2=0x7f1aac1407a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:18.751 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.721+0000 7f1ab1ab6700 1 --2- 172.21.15.60:0/3583023527 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1aac0abba0 0x7f1aac1407a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:18.752 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.721+0000 7f1ab1ab6700 1 -- 172.21.15.60:0/3583023527 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa4009530 con 0x7f1aac0aacd0 2022-06-29T15:58:18.752 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.721+0000 7f1ab1ab6700 1 --2- 172.21.15.60:0/3583023527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1aac0aacd0 0x7f1aac149850 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f1aa800c000 tx=0x7f1aa8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:18.753 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.721+0000 7f1a9affd700 1 -- 172.21.15.60:0/3583023527 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1aa8011750 con 0x7f1aac0aacd0 2022-06-29T15:58:18.753 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.721+0000 7f1a9affd700 1 -- 172.21.15.60:0/3583023527 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa80118b0 con 0x7f1aac0aacd0 2022-06-29T15:58:18.754 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.721+0000 7f1ab2ab8700 1 -- 172.21.15.60:0/3583023527 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aac1443f0 con 0x7f1aac0aacd0 2022-06-29T15:58:18.754 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.721+0000 7f1a9affd700 1 -- 172.21.15.60:0/3583023527 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1aa8011c10 con 0x7f1aac0aacd0 2022-06-29T15:58:18.754 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.721+0000 7f1ab2ab8700 1 -- 172.21.15.60:0/3583023527 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aac141180 con 0x7f1aac0aacd0 2022-06-29T15:58:18.755 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.721+0000 7f1ab2ab8700 1 -- 172.21.15.60:0/3583023527 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aac0573d0 con 0x7f1aac0aacd0 2022-06-29T15:58:18.755 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.721+0000 7f1a9affd700 1 -- 172.21.15.60:0/3583023527 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1aa8011d70 con 0x7f1aac0aacd0 2022-06-29T15:58:18.756 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.721+0000 7f1a9affd700 1 --2- 172.21.15.60:0/3583023527 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1a8c053aa0 0x7f1a8c055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:18.756 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.721+0000 7f1a9affd700 1 -- 172.21.15.60:0/3583023527 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4900+0+0 (secure 0 0 0) 0x7f1aa806be20 con 0x7f1aac0aacd0 2022-06-29T15:58:18.757 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.725+0000 7f1ab12b5700 1 --2- 172.21.15.60:0/3583023527 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1a8c053aa0 0x7f1a8c055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:18.757 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.725+0000 7f1ab12b5700 1 --2- 172.21.15.60:0/3583023527 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1a8c053aa0 0x7f1a8c055f60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f1a9c00d000 tx=0x7f1a9c007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:18.758 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.729+0000 7f1a9affd700 1 -- 172.21.15.60:0/3583023527 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1aa8043170 con 0x7f1aac0aacd0 2022-06-29T15:58:18.865 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.857+0000 7f1ab2ab8700 1 -- 172.21.15.60:0/3583023527 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1aac05f1b0 con 0x7f1aac0aacd0 2022-06-29T15:58:18.866 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.861+0000 7f1a9affd700 1 -- 172.21.15.60:0/3583023527 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+11805 (secure 0 0 0) 0x7f1aa8046790 con 0x7f1aac0aacd0 2022-06-29T15:58:18.866 INFO:teuthology.misc.health.smithi060.stdout: 2022-06-29T15:58:18.867 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":15,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T15:58:15.808401+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T15:58:18.869 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.861+0000 7f1ab2ab8700 1 -- 172.21.15.60:0/3583023527 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1a8c053aa0 msgr2=0x7f1a8c055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:18.869 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.861+0000 7f1ab2ab8700 1 --2- 172.21.15.60:0/3583023527 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1a8c053aa0 0x7f1a8c055f60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f1a9c00d000 tx=0x7f1a9c007a00).stop 2022-06-29T15:58:18.869 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.861+0000 7f1ab2ab8700 1 -- 172.21.15.60:0/3583023527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1aac0aacd0 msgr2=0x7f1aac149850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:18.869 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.861+0000 7f1ab2ab8700 1 --2- 172.21.15.60:0/3583023527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1aac0aacd0 0x7f1aac149850 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f1aa800c000 tx=0x7f1aa8006df0).stop 2022-06-29T15:58:18.870 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.861+0000 7f1ab2ab8700 1 -- 172.21.15.60:0/3583023527 shutdown_connections 2022-06-29T15:58:18.870 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.861+0000 7f1ab2ab8700 1 --2- 172.21.15.60:0/3583023527 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1a8c053aa0 0x7f1a8c055f60 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:18.870 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.861+0000 7f1ab2ab8700 1 --2- 172.21.15.60:0/3583023527 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1aac0b6010 0x7f1aac140c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:18.870 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.861+0000 7f1ab2ab8700 1 --2- 172.21.15.60:0/3583023527 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1aac0abba0 0x7f1aac1407a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:18.871 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.861+0000 7f1ab2ab8700 1 --2- 172.21.15.60:0/3583023527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1aac0aacd0 0x7f1aac149850 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:18.871 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.861+0000 7f1ab2ab8700 1 -- 172.21.15.60:0/3583023527 >> 172.21.15.60:0/3583023527 conn(0x7f1aac063650 msgr2=0x7f1aac0b33f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:18.871 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.861+0000 7f1ab2ab8700 1 -- 172.21.15.60:0/3583023527 shutdown_connections 2022-06-29T15:58:18.871 INFO:teuthology.misc.health.smithi060.stderr:2022-06-29T15:58:18.861+0000 7f1ab2ab8700 1 -- 172.21.15.60:0/3583023527 wait complete. 2022-06-29T15:58:18.879 DEBUG:teuthology.misc:8 of 8 OSDs are up 2022-06-29T15:58:18.879 INFO:tasks.ceph:Creating RBD pool 2022-06-29T15:58:18.880 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph osd pool create rbd 8 2022-06-29T15:58:18.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.977+0000 7ffb0b279700 1 Processor -- start 2022-06-29T15:58:18.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.977+0000 7ffb0b279700 1 -- start start 2022-06-29T15:58:18.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.977+0000 7ffb0b279700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ffb040b28e0 0x7ffb040b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:18.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.977+0000 7ffb0b279700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffb04060760 0x7ffb04140180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:18.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.977+0000 7ffb0b279700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb04140670 con 0x7ffb04065450 2022-06-29T15:58:18.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.977+0000 7ffb0b279700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb0413fa40 con 0x7ffb04060760 2022-06-29T15:58:18.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.977+0000 7ffb0b279700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb0413fce0 con 0x7ffb040b28e0 2022-06-29T15:58:18.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.977+0000 7ffb0aa78700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffb04060760 0x7ffb04140180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:18.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.977+0000 7ffb0aa78700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffb04060760 0x7ffb04140180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44772/0 (socket says 172.21.15.60:44772) 2022-06-29T15:58:18.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.977+0000 7ffb0aa78700 1 -- 172.21.15.60:0/536522659 learned_addr learned my addr 172.21.15.60:0/536522659 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:18.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.977+0000 7ffb0a277700 1 --2- 172.21.15.60:0/536522659 >> v2:172.21.15.92:3300/0 conn(0x7ffb040b28e0 0x7ffb040b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:18.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.977+0000 7ffb09275700 1 -- 172.21.15.60:0/536522659 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3068744822 0 0) 0x7ffb04140670 con 0x7ffb04065450 2022-06-29T15:58:18.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.977+0000 7ffb09275700 1 -- 172.21.15.60:0/536522659 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaec003260 con 0x7ffb04065450 2022-06-29T15:58:18.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.981+0000 7ffb0aa78700 1 -- 172.21.15.60:0/536522659 >> v2:172.21.15.92:3300/0 conn(0x7ffb040b28e0 msgr2=0x7ffb040b2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:18.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.981+0000 7ffb0aa78700 1 --2- 172.21.15.60:0/536522659 >> v2:172.21.15.92:3300/0 conn(0x7ffb040b28e0 0x7ffb040b2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:18.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.981+0000 7ffb0aa78700 1 -- 172.21.15.60:0/536522659 >> v1:172.21.15.92:6789/0 conn(0x7ffb04065450 legacy=0x7ffb04139340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:18.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.981+0000 7ffb0aa78700 1 -- 172.21.15.60:0/536522659 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb04140860 con 0x7ffb04060760 2022-06-29T15:58:18.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.981+0000 7ffb0a277700 1 --2- 172.21.15.60:0/536522659 >> v2:172.21.15.92:3300/0 conn(0x7ffb040b28e0 0x7ffb040b2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:18.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.981+0000 7ffb0aa78700 1 --2- 172.21.15.60:0/536522659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb04060760 0x7ffb04140180 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7ffafc0032f0 tx=0x7ffafc009290).ready entity=mon.0 client_cookie=78acb44a42300110 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:18.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.981+0000 7ffb09275700 1 -- 172.21.15.60:0/536522659 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffafc00f8a0 con 0x7ffb04060760 2022-06-29T15:58:18.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.981+0000 7ffb09275700 1 -- 172.21.15.60:0/536522659 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffafc00fa00 con 0x7ffb04060760 2022-06-29T15:58:18.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.981+0000 7ffb09275700 1 -- 172.21.15.60:0/536522659 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffafc00fd60 con 0x7ffb04060760 2022-06-29T15:58:18.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.981+0000 7ffb0b279700 1 -- 172.21.15.60:0/536522659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb04060760 msgr2=0x7ffb04140180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:18.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.981+0000 7ffb0b279700 1 --2- 172.21.15.60:0/536522659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb04060760 0x7ffb04140180 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7ffafc0032f0 tx=0x7ffafc009290).stop 2022-06-29T15:58:18.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.981+0000 7ffb0b279700 1 -- 172.21.15.60:0/536522659 shutdown_connections 2022-06-29T15:58:18.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.981+0000 7ffb0b279700 1 --2- 172.21.15.60:0/536522659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb04060760 0x7ffb04140180 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:18.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.981+0000 7ffb0b279700 1 --2- 172.21.15.60:0/536522659 >> v2:172.21.15.92:3300/0 conn(0x7ffb040b28e0 0x7ffb040b2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:18.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.981+0000 7ffb0b279700 1 -- 172.21.15.60:0/536522659 >> 172.21.15.60:0/536522659 conn(0x7ffb04061c80 msgr2=0x7ffb040637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:18.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.981+0000 7ffb0b279700 1 -- 172.21.15.60:0/536522659 shutdown_connections 2022-06-29T15:58:18.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.981+0000 7ffb0b279700 1 -- 172.21.15.60:0/536522659 wait complete. 2022-06-29T15:58:18.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.981+0000 7ffb0b279700 1 Processor -- start 2022-06-29T15:58:18.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.981+0000 7ffb0b279700 1 -- start start 2022-06-29T15:58:18.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.985+0000 7ffb0b279700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb04060760 0x7ffb040afe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:18.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.985+0000 7ffb0b279700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb04065450 0x7ffb0405cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:18.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.985+0000 7ffb0b279700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb040b28e0 0x7ffb0413d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:18.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.985+0000 7ffb0b279700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0413f5a0 con 0x7ffb04065450 2022-06-29T15:58:18.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.985+0000 7ffb0b279700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb040ad770 con 0x7ffb04060760 2022-06-29T15:58:18.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.985+0000 7ffb0b279700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0413f340 con 0x7ffb040b28e0 2022-06-29T15:58:18.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.985+0000 7ffb09a76700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb04065450 0x7ffb0405cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:18.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.985+0000 7ffb0aa78700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb040b28e0 0x7ffb0413d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:18.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.985+0000 7ffb0a277700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb04060760 0x7ffb040afe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:18.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.985+0000 7ffb09a76700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb04065450 0x7ffb0405cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44776/0 (socket says 172.21.15.60:44776) 2022-06-29T15:58:18.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.985+0000 7ffb0aa78700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb040b28e0 0x7ffb0413d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:57870/0 (socket says 172.21.15.60:57870) 2022-06-29T15:58:19.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.985+0000 7ffb09a76700 1 -- 172.21.15.60:0/2281868284 learned_addr learned my addr 172.21.15.60:0/2281868284 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:19.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.985+0000 7ffb0aa78700 1 -- 172.21.15.60:0/2281868284 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb04060760 msgr2=0x7ffb040afe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:19.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.985+0000 7ffb0aa78700 1 --2- 172.21.15.60:0/2281868284 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb04060760 0x7ffb040afe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:19.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.985+0000 7ffb0aa78700 1 -- 172.21.15.60:0/2281868284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb04065450 msgr2=0x7ffb0405cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:19.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.985+0000 7ffb0aa78700 1 --2- 172.21.15.60:0/2281868284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb04065450 0x7ffb0405cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:19.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.985+0000 7ffb0aa78700 1 -- 172.21.15.60:0/2281868284 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf400b180 con 0x7ffb040b28e0 2022-06-29T15:58:19.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.985+0000 7ffb0aa78700 1 --2- 172.21.15.60:0/2281868284 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb040b28e0 0x7ffb0413d300 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7ffafc0032f0 tx=0x7ffafc019000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:19.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.985+0000 7ffaf37fe700 1 -- 172.21.15.60:0/2281868284 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffafc00f8a0 con 0x7ffb040b28e0 2022-06-29T15:58:19.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.985+0000 7ffaf37fe700 1 -- 172.21.15.60:0/2281868284 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffafc019450 con 0x7ffb040b28e0 2022-06-29T15:58:19.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.985+0000 7ffb0b279700 1 -- 172.21.15.60:0/2281868284 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffafc009530 con 0x7ffb040b28e0 2022-06-29T15:58:19.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.985+0000 7ffaf37fe700 1 -- 172.21.15.60:0/2281868284 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffafc0197b0 con 0x7ffb040b28e0 2022-06-29T15:58:19.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.985+0000 7ffb0b279700 1 -- 172.21.15.60:0/2281868284 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb0413e430 con 0x7ffb040b28e0 2022-06-29T15:58:19.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.985+0000 7ffb0b279700 1 -- 172.21.15.60:0/2281868284 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb0413cce0 con 0x7ffb040b28e0 2022-06-29T15:58:19.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.989+0000 7ffaf37fe700 1 -- 172.21.15.60:0/2281868284 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ffafc016030 con 0x7ffb040b28e0 2022-06-29T15:58:19.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.989+0000 7ffaf37fe700 1 --2- 172.21.15.60:0/2281868284 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ffae4053a70 0x7ffae4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:19.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.989+0000 7ffaf37fe700 1 -- 172.21.15.60:0/2281868284 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4900+0+0 (secure 0 0 0) 0x7ffafc06cfa0 con 0x7ffb040b28e0 2022-06-29T15:58:19.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.989+0000 7ffb0a277700 1 --2- 172.21.15.60:0/2281868284 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ffae4053a70 0x7ffae4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:19.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.989+0000 7ffb0a277700 1 --2- 172.21.15.60:0/2281868284 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ffae4053a70 0x7ffae4055f30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7ffaf400b000 tx=0x7ffaf4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:19.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:18.997+0000 7ffaf37fe700 1 -- 172.21.15.60:0/2281868284 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ffafc044400 con 0x7ffb040b28e0 2022-06-29T15:58:19.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:19.129+0000 7ffb0b279700 1 -- 172.21.15.60:0/2281868284 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7ffb0413e630 con 0x7ffb040b28e0 2022-06-29T15:58:19.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:19.981+0000 7ffaf37fe700 1 -- 172.21.15.60:0/2281868284 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v16) v1 ==== 109+0+0 (secure 0 0 0) 0x7ffafc047a20 con 0x7ffb040b28e0 2022-06-29T15:58:19.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:19.985+0000 7ffb0b279700 1 -- 172.21.15.60:0/2281868284 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ffae4053a70 msgr2=0x7ffae4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:19.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:19.985+0000 7ffb0b279700 1 --2- 172.21.15.60:0/2281868284 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ffae4053a70 0x7ffae4055f30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7ffaf400b000 tx=0x7ffaf4006df0).stop 2022-06-29T15:58:19.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:19.985+0000 7ffb0b279700 1 -- 172.21.15.60:0/2281868284 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb040b28e0 msgr2=0x7ffb0413d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:19.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:19.985+0000 7ffb0b279700 1 --2- 172.21.15.60:0/2281868284 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb040b28e0 0x7ffb0413d300 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7ffafc0032f0 tx=0x7ffafc019000).stop 2022-06-29T15:58:19.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:19.985+0000 7ffb0b279700 1 -- 172.21.15.60:0/2281868284 shutdown_connections 2022-06-29T15:58:19.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:19.985+0000 7ffb0b279700 1 --2- 172.21.15.60:0/2281868284 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ffae4053a70 0x7ffae4055f30 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:19.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:19.985+0000 7ffb0b279700 1 --2- 172.21.15.60:0/2281868284 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffb040b28e0 0x7ffb0413d300 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:19.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:19.985+0000 7ffb0b279700 1 --2- 172.21.15.60:0/2281868284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb04065450 0x7ffb0405cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:19.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:19.985+0000 7ffb0b279700 1 --2- 172.21.15.60:0/2281868284 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb04060760 0x7ffb040afe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:19.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:19.985+0000 7ffb0b279700 1 -- 172.21.15.60:0/2281868284 >> 172.21.15.60:0/2281868284 conn(0x7ffb04061c80 msgr2=0x7ffb0405f2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:19.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:19.985+0000 7ffb0b279700 1 -- 172.21.15.60:0/2281868284 shutdown_connections 2022-06-29T15:58:19.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:19.985+0000 7ffb0b279700 1 -- 172.21.15.60:0/2281868284 wait complete. 2022-06-29T15:58:19.996 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'rbd' created 2022-06-29T15:58:20.005 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2022-06-29T15:58:20.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.113+0000 7f9e666cd700 1 Processor -- start 2022-06-29T15:58:20.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.113+0000 7f9e666cd700 1 -- start start 2022-06-29T15:58:20.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.113+0000 7f9e666cd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e600b9d80 0x7f9e6005bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:20.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.113+0000 7f9e666cd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9e600b3fc0 0x7f9e6005c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:20.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.113+0000 7f9e666cd700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9e600b0a20 0x7f9e6005c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:20.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.113+0000 7f9e666cd700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e600b3090 con 0x7f9e600b3fc0 2022-06-29T15:58:20.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.113+0000 7f9e666cd700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e6005cbf0 con 0x7f9e600b0a20 2022-06-29T15:58:20.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.113+0000 7f9e666cd700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e600b2ab0 con 0x7f9e600b9d80 2022-06-29T15:58:20.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.113+0000 7f9e5f7fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9e600b3fc0 0x7f9e6005c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:20.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.113+0000 7f9e5f7fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9e600b3fc0 0x7f9e6005c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44782/0 (socket says 172.21.15.60:44782) 2022-06-29T15:58:20.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.113+0000 7f9e5f7fe700 1 -- 172.21.15.60:0/4029191403 learned_addr learned my addr 172.21.15.60:0/4029191403 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:20.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.113+0000 7f9e5ffff700 1 --2- 172.21.15.60:0/4029191403 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e600b9d80 0x7f9e6005bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:20.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.117+0000 7f9e64c6a700 1 --2- 172.21.15.60:0/4029191403 >> v2:172.21.15.92:3300/0 conn(0x7f9e600b0a20 0x7f9e6005c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:20.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.117+0000 7f9e5f7fe700 1 -- 172.21.15.60:0/4029191403 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e600b9d80 msgr2=0x7f9e6005bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:20.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.117+0000 7f9e5f7fe700 1 --2- 172.21.15.60:0/4029191403 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e600b9d80 0x7f9e6005bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:20.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.117+0000 7f9e5f7fe700 1 -- 172.21.15.60:0/4029191403 >> v2:172.21.15.92:3300/0 conn(0x7f9e600b0a20 msgr2=0x7f9e6005c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:20.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.117+0000 7f9e5f7fe700 1 --2- 172.21.15.60:0/4029191403 >> v2:172.21.15.92:3300/0 conn(0x7f9e600b0a20 0x7f9e6005c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:20.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.117+0000 7f9e5f7fe700 1 -- 172.21.15.60:0/4029191403 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e600ba160 con 0x7f9e600b3fc0 2022-06-29T15:58:20.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.117+0000 7f9e5ffff700 1 --2- 172.21.15.60:0/4029191403 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e600b9d80 0x7f9e6005bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:58:20.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.117+0000 7f9e5f7fe700 1 --2- 172.21.15.60:0/4029191403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e600b3fc0 0x7f9e6005c210 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f9e600b5bc0 tx=0x7f9e54006eb0).ready entity=mon.0 client_cookie=45626d8f6c8567c0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:20.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.117+0000 7f9e5effd700 1 -- 172.21.15.60:0/4029191403 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e5400d570 con 0x7f9e600b3fc0 2022-06-29T15:58:20.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.117+0000 7f9e5effd700 1 -- 172.21.15.60:0/4029191403 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e54000e90 con 0x7f9e600b3fc0 2022-06-29T15:58:20.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.117+0000 7f9e5effd700 1 -- 172.21.15.60:0/4029191403 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e5400d900 con 0x7f9e600b3fc0 2022-06-29T15:58:20.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.117+0000 7f9e666cd700 1 -- 172.21.15.60:0/4029191403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e600b3fc0 msgr2=0x7f9e6005c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:20.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.117+0000 7f9e666cd700 1 --2- 172.21.15.60:0/4029191403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e600b3fc0 0x7f9e6005c210 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f9e600b5bc0 tx=0x7f9e54006eb0).stop 2022-06-29T15:58:20.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.117+0000 7f9e666cd700 1 -- 172.21.15.60:0/4029191403 shutdown_connections 2022-06-29T15:58:20.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.117+0000 7f9e666cd700 1 --2- 172.21.15.60:0/4029191403 >> v2:172.21.15.92:3300/0 conn(0x7f9e600b0a20 0x7f9e6005c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:20.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.117+0000 7f9e666cd700 1 --2- 172.21.15.60:0/4029191403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e600b3fc0 0x7f9e6005c210 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:20.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.117+0000 7f9e666cd700 1 --2- 172.21.15.60:0/4029191403 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e600b9d80 0x7f9e6005bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:20.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.117+0000 7f9e666cd700 1 -- 172.21.15.60:0/4029191403 >> 172.21.15.60:0/4029191403 conn(0x7f9e60062e70 msgr2=0x7f9e600b9330 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:20.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.117+0000 7f9e666cd700 1 -- 172.21.15.60:0/4029191403 shutdown_connections 2022-06-29T15:58:20.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.117+0000 7f9e666cd700 1 -- 172.21.15.60:0/4029191403 wait complete. 2022-06-29T15:58:20.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.117+0000 7f9e666cd700 1 Processor -- start 2022-06-29T15:58:20.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.117+0000 7f9e666cd700 1 -- start start 2022-06-29T15:58:20.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.121+0000 7f9e666cd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e600b0a20 0x7f9e600ad390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:20.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.121+0000 7f9e666cd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e600b3fc0 0x7f9e60154f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:20.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.121+0000 7f9e666cd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e600b9d80 0x7f9e60115a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:20.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.121+0000 7f9e666cd700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e60114a50 con 0x7f9e600b9d80 2022-06-29T15:58:20.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.121+0000 7f9e666cd700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e60113480 con 0x7f9e600b0a20 2022-06-29T15:58:20.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.121+0000 7f9e666cd700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e60155420 con 0x7f9e600b3fc0 2022-06-29T15:58:20.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.121+0000 7f9e5f7fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e600b3fc0 0x7f9e60154f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:20.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.121+0000 7f9e64c6a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e600b9d80 0x7f9e60115a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:20.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.121+0000 7f9e5f7fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e600b3fc0 0x7f9e60154f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:57884/0 (socket says 172.21.15.60:57884) 2022-06-29T15:58:20.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.121+0000 7f9e5f7fe700 1 -- 172.21.15.60:0/4209144652 learned_addr learned my addr 172.21.15.60:0/4209144652 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:20.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.121+0000 7f9e5ffff700 1 --2- 172.21.15.60:0/4209144652 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e600b0a20 0x7f9e600ad390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:20.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.121+0000 7f9e5f7fe700 1 -- 172.21.15.60:0/4209144652 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e600b0a20 msgr2=0x7f9e600ad390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:20.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.121+0000 7f9e5f7fe700 1 --2- 172.21.15.60:0/4209144652 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e600b0a20 0x7f9e600ad390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:20.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.121+0000 7f9e5f7fe700 1 -- 172.21.15.60:0/4209144652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e600b9d80 msgr2=0x7f9e60115a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:20.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.121+0000 7f9e5f7fe700 1 --2- 172.21.15.60:0/4209144652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e600b9d80 0x7f9e60115a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:20.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.121+0000 7f9e5f7fe700 1 -- 172.21.15.60:0/4209144652 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5400b1e0 con 0x7f9e600b3fc0 2022-06-29T15:58:20.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.121+0000 7f9e5f7fe700 1 --2- 172.21.15.60:0/4209144652 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e600b3fc0 0x7f9e60154f30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f9e60115430 tx=0x7f9e54015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:20.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.121+0000 7f9e5d7fa700 1 -- 172.21.15.60:0/4209144652 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e54006be0 con 0x7f9e600b3fc0 2022-06-29T15:58:20.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.121+0000 7f9e5d7fa700 1 -- 172.21.15.60:0/4209144652 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e54002f70 con 0x7f9e600b3fc0 2022-06-29T15:58:20.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.121+0000 7f9e5d7fa700 1 -- 172.21.15.60:0/4209144652 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e5400dea0 con 0x7f9e600b3fc0 2022-06-29T15:58:20.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.121+0000 7f9e666cd700 1 -- 172.21.15.60:0/4209144652 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e60115630 con 0x7f9e600b3fc0 2022-06-29T15:58:20.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.121+0000 7f9e666cd700 1 -- 172.21.15.60:0/4209144652 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e600afcf0 con 0x7f9e600b3fc0 2022-06-29T15:58:20.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.125+0000 7f9e666cd700 1 -- 172.21.15.60:0/4209144652 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e600573d0 con 0x7f9e600b3fc0 2022-06-29T15:58:20.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.125+0000 7f9e5d7fa700 1 -- 172.21.15.60:0/4209144652 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9e5400f070 con 0x7f9e600b3fc0 2022-06-29T15:58:20.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.125+0000 7f9e5d7fa700 1 --2- 172.21.15.60:0/4209144652 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9e40053aa0 0x7f9e40055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:20.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.125+0000 7f9e5d7fa700 1 -- 172.21.15.60:0/4209144652 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f9e5406de20 con 0x7f9e600b3fc0 2022-06-29T15:58:20.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.125+0000 7f9e5ffff700 1 --2- 172.21.15.60:0/4209144652 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9e40053aa0 0x7f9e40055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:20.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.125+0000 7f9e5ffff700 1 --2- 172.21.15.60:0/4209144652 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9e40053aa0 0x7f9e40055f60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f9e5000c000 tx=0x7f9e50008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:20.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.133+0000 7f9e5d7fa700 1 -- 172.21.15.60:0/4209144652 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9e54048ae0 con 0x7f9e600b3fc0 2022-06-29T15:58:20.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.261+0000 7f9e666cd700 1 -- 172.21.15.60:0/4209144652 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f9e6005d990 con 0x7f9e600b3fc0 2022-06-29T15:58:20.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.989+0000 7f9e5d7fa700 1 -- 172.21.15.60:0/4209144652 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v17) v1 ==== 173+0+0 (secure 0 0 0) 0x7f9e540486d0 con 0x7f9e600b3fc0 2022-06-29T15:58:21.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.997+0000 7f9e666cd700 1 -- 172.21.15.60:0/4209144652 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9e40053aa0 msgr2=0x7f9e40055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:21.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.997+0000 7f9e666cd700 1 --2- 172.21.15.60:0/4209144652 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9e40053aa0 0x7f9e40055f60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f9e5000c000 tx=0x7f9e50008040).stop 2022-06-29T15:58:21.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.997+0000 7f9e666cd700 1 -- 172.21.15.60:0/4209144652 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e600b3fc0 msgr2=0x7f9e60154f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:21.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.997+0000 7f9e666cd700 1 --2- 172.21.15.60:0/4209144652 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e600b3fc0 0x7f9e60154f30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f9e60115430 tx=0x7f9e54015000).stop 2022-06-29T15:58:21.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.997+0000 7f9e666cd700 1 -- 172.21.15.60:0/4209144652 shutdown_connections 2022-06-29T15:58:21.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.997+0000 7f9e666cd700 1 --2- 172.21.15.60:0/4209144652 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9e40053aa0 0x7f9e40055f60 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.997+0000 7f9e666cd700 1 --2- 172.21.15.60:0/4209144652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e600b9d80 0x7f9e60115a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.997+0000 7f9e666cd700 1 --2- 172.21.15.60:0/4209144652 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e600b3fc0 0x7f9e60154f30 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.997+0000 7f9e666cd700 1 --2- 172.21.15.60:0/4209144652 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e600b0a20 0x7f9e600ad390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.997+0000 7f9e666cd700 1 -- 172.21.15.60:0/4209144652 >> 172.21.15.60:0/4209144652 conn(0x7f9e60062e70 msgr2=0x7f9e600b88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:21.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.997+0000 7f9e666cd700 1 -- 172.21.15.60:0/4209144652 shutdown_connections 2022-06-29T15:58:21.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:20.997+0000 7f9e666cd700 1 -- 172.21.15.60:0/4209144652 wait complete. 2022-06-29T15:58:21.006 INFO:teuthology.orchestra.run.smithi060.stderr:enabled application 'rbd' on pool 'rbd' 2022-06-29T15:58:21.015 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2022-06-29T15:58:21.016 INFO:tasks.ceph.mds.a:Restarting daemon 2022-06-29T15:58:21.016 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-06-29T15:58:21.019 INFO:tasks.ceph.mds.a:Started 2022-06-29T15:58:21.020 INFO:tasks.ceph.mds.b:Restarting daemon 2022-06-29T15:58:21.020 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-06-29T15:58:21.023 INFO:tasks.ceph.mds.b:Started 2022-06-29T15:58:21.023 INFO:tasks.ceph:Setting up CephFS filesystem... 2022-06-29T15:58:21.024 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T15:58:21.089 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2022-06-29T15:58:21.097 INFO:tasks.ceph.mds.b.smithi092.stdout:starting mds.b at 2022-06-29T15:58:21.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.113+0000 7efd8ad2c700 1 Processor -- start 2022-06-29T15:58:21.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.113+0000 7efd8ad2c700 1 -- start start 2022-06-29T15:58:21.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.117+0000 7efd8ad2c700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7efd84140410 0x7efd84065250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:21.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.117+0000 7efd8ad2c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd84064b50 0x7efd84128290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:21.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.117+0000 7efd8ad2c700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd840b5d30 con 0x7efd840b5950 2022-06-29T15:58:21.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.117+0000 7efd8ad2c700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd84066fa0 con 0x7efd84140410 2022-06-29T15:58:21.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.117+0000 7efd8ad2c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd84143960 con 0x7efd84064b50 2022-06-29T15:58:21.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.117+0000 7efd83fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7efd84140410 0x7efd84065250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:21.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.117+0000 7efd892c9700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd84064b50 0x7efd84128290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:21.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.117+0000 7efd83fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7efd84140410 0x7efd84065250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:50842/0 (socket says 172.21.15.60:50842) 2022-06-29T15:58:21.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.117+0000 7efd88ac8700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7efd840b5950 0x7efd8413fd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:57144/0 (socket says 172.21.15.60:57144) 2022-06-29T15:58:21.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.117+0000 7efd83fff700 1 -- 172.21.15.60:0/386189296 learned_addr learned my addr 172.21.15.60:0/386189296 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:21.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.117+0000 7efd83fff700 1 -- 172.21.15.60:0/386189296 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd84064b50 msgr2=0x7efd84128290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:21.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.117+0000 7efd83fff700 1 --2- 172.21.15.60:0/386189296 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd84064b50 0x7efd84128290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.117+0000 7efd83fff700 1 -- 172.21.15.60:0/386189296 >> v1:172.21.15.92:6789/0 conn(0x7efd840b5950 legacy=0x7efd8413fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:21.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.117+0000 7efd83fff700 1 -- 172.21.15.60:0/386189296 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd84143c00 con 0x7efd84140410 2022-06-29T15:58:21.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.117+0000 7efd892c9700 1 --2- 172.21.15.60:0/386189296 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd84064b50 0x7efd84128290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:21.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.117+0000 7efd83fff700 1 --2- 172.21.15.60:0/386189296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd84140410 0x7efd84065250 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7efd74002f70 tx=0x7efd74006eb0).ready entity=mon.1 client_cookie=42d90b35637aa77c server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:21.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.117+0000 7efd837fe700 1 -- 172.21.15.60:0/386189296 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd7400d570 con 0x7efd84140410 2022-06-29T15:58:21.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.117+0000 7efd837fe700 1 -- 172.21.15.60:0/386189296 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd74000e90 con 0x7efd84140410 2022-06-29T15:58:21.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.117+0000 7efd837fe700 1 -- 172.21.15.60:0/386189296 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd7400d900 con 0x7efd84140410 2022-06-29T15:58:21.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.121+0000 7efd8ad2c700 1 -- 172.21.15.60:0/386189296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd84140410 msgr2=0x7efd84065250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:21.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.121+0000 7efd8ad2c700 1 --2- 172.21.15.60:0/386189296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd84140410 0x7efd84065250 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7efd74002f70 tx=0x7efd74006eb0).stop 2022-06-29T15:58:21.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.121+0000 7efd8ad2c700 1 -- 172.21.15.60:0/386189296 shutdown_connections 2022-06-29T15:58:21.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.121+0000 7efd8ad2c700 1 --2- 172.21.15.60:0/386189296 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd84064b50 0x7efd84128290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.121+0000 7efd8ad2c700 1 --2- 172.21.15.60:0/386189296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd84140410 0x7efd84065250 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.121+0000 7efd8ad2c700 1 -- 172.21.15.60:0/386189296 >> 172.21.15.60:0/386189296 conn(0x7efd840632c0 msgr2=0x7efd840b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:21.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.121+0000 7efd8ad2c700 1 -- 172.21.15.60:0/386189296 shutdown_connections 2022-06-29T15:58:21.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.121+0000 7efd8ad2c700 1 -- 172.21.15.60:0/386189296 wait complete. 2022-06-29T15:58:21.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.121+0000 7efd8ad2c700 1 Processor -- start 2022-06-29T15:58:21.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.121+0000 7efd8ad2c700 1 -- start start 2022-06-29T15:58:21.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.125+0000 7efd8ad2c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd84064b50 0x7efd840b06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:21.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.125+0000 7efd8ad2c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd840b5950 0x7efd84155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:21.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.125+0000 7efd8ad2c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd84140410 0x7efd84155c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:21.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.125+0000 7efd8ad2c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd840afbb0 con 0x7efd84140410 2022-06-29T15:58:21.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.125+0000 7efd8ad2c700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8412ab30 con 0x7efd84064b50 2022-06-29T15:58:21.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.125+0000 7efd8ad2c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd8412add0 con 0x7efd840b5950 2022-06-29T15:58:21.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.125+0000 7efd83fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd840b5950 0x7efd84155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:21.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.125+0000 7efd892c9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd84140410 0x7efd84155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:21.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.125+0000 7efd88ac8700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd84064b50 0x7efd840b06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:21.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.125+0000 7efd83fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd840b5950 0x7efd84155720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:57912/0 (socket says 172.21.15.60:57912) 2022-06-29T15:58:21.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.125+0000 7efd83fff700 1 -- 172.21.15.60:0/1929988927 learned_addr learned my addr 172.21.15.60:0/1929988927 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:21.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.125+0000 7efd83fff700 1 -- 172.21.15.60:0/1929988927 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd84064b50 msgr2=0x7efd840b06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:21.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.125+0000 7efd83fff700 1 --2- 172.21.15.60:0/1929988927 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd84064b50 0x7efd840b06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.125+0000 7efd83fff700 1 -- 172.21.15.60:0/1929988927 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd84140410 msgr2=0x7efd84155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:21.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.125+0000 7efd83fff700 1 --2- 172.21.15.60:0/1929988927 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd84140410 0x7efd84155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.125+0000 7efd83fff700 1 -- 172.21.15.60:0/1929988927 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd7c009470 con 0x7efd840b5950 2022-06-29T15:58:21.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.125+0000 7efd83fff700 1 --2- 172.21.15.60:0/1929988927 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd840b5950 0x7efd84155720 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7efd74002f70 tx=0x7efd74015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:21.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.125+0000 7efd81ffb700 1 -- 172.21.15.60:0/1929988927 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd740117e0 con 0x7efd840b5950 2022-06-29T15:58:21.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.125+0000 7efd81ffb700 1 -- 172.21.15.60:0/1929988927 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd74011940 con 0x7efd840b5950 2022-06-29T15:58:21.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.125+0000 7efd8ad2c700 1 -- 172.21.15.60:0/1929988927 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd7400b1e0 con 0x7efd840b5950 2022-06-29T15:58:21.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.125+0000 7efd81ffb700 1 -- 172.21.15.60:0/1929988927 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd74011ca0 con 0x7efd840b5950 2022-06-29T15:58:21.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.125+0000 7efd8ad2c700 1 -- 172.21.15.60:0/1929988927 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd84156100 con 0x7efd840b5950 2022-06-29T15:58:21.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.125+0000 7efd8ad2c700 1 -- 172.21.15.60:0/1929988927 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd8413f860 con 0x7efd840b5950 2022-06-29T15:58:21.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.129+0000 7efd81ffb700 1 -- 172.21.15.60:0/1929988927 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7efd74025030 con 0x7efd840b5950 2022-06-29T15:58:21.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.129+0000 7efd81ffb700 1 --2- 172.21.15.60:0/1929988927 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd5c053b00 0x7efd5c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:21.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.129+0000 7efd81ffb700 1 -- 172.21.15.60:0/1929988927 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5253+0+0 (secure 0 0 0) 0x7efd7406ce00 con 0x7efd840b5950 2022-06-29T15:58:21.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.129+0000 7efd88ac8700 1 --2- 172.21.15.60:0/1929988927 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd5c053b00 0x7efd5c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:21.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.129+0000 7efd88ac8700 1 --2- 172.21.15.60:0/1929988927 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd5c053b00 0x7efd5c055fc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7efd6c00d000 tx=0x7efd6c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:21.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.137+0000 7efd81ffb700 1 -- 172.21.15.60:0/1929988927 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efd74044050 con 0x7efd840b5950 2022-06-29T15:58:21.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.261+0000 7efd8ad2c700 1 -- 172.21.15.60:0/1929988927 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd840573d0 con 0x7efd840b5950 2022-06-29T15:58:21.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.265+0000 7efd81ffb700 1 -- 172.21.15.60:0/1929988927 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+13191 (secure 0 0 0) 0x7efd74023020 con 0x7efd840b5950 2022-06-29T15:58:21.270 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:58:21.271 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":17,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T15:58:20.977638+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T15:58:21.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.265+0000 7efd8ad2c700 1 -- 172.21.15.60:0/1929988927 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd5c053b00 msgr2=0x7efd5c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:21.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.265+0000 7efd8ad2c700 1 --2- 172.21.15.60:0/1929988927 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd5c053b00 0x7efd5c055fc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7efd6c00d000 tx=0x7efd6c009040).stop 2022-06-29T15:58:21.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.265+0000 7efd8ad2c700 1 -- 172.21.15.60:0/1929988927 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd840b5950 msgr2=0x7efd84155720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:21.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.265+0000 7efd8ad2c700 1 --2- 172.21.15.60:0/1929988927 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd840b5950 0x7efd84155720 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7efd74002f70 tx=0x7efd74015000).stop 2022-06-29T15:58:21.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.265+0000 7efd8ad2c700 1 -- 172.21.15.60:0/1929988927 shutdown_connections 2022-06-29T15:58:21.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.265+0000 7efd8ad2c700 1 --2- 172.21.15.60:0/1929988927 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd5c053b00 0x7efd5c055fc0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.265+0000 7efd8ad2c700 1 --2- 172.21.15.60:0/1929988927 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd84140410 0x7efd84155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.265+0000 7efd8ad2c700 1 --2- 172.21.15.60:0/1929988927 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd840b5950 0x7efd84155720 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.265+0000 7efd8ad2c700 1 --2- 172.21.15.60:0/1929988927 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd84064b50 0x7efd840b06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.265+0000 7efd8ad2c700 1 -- 172.21.15.60:0/1929988927 >> 172.21.15.60:0/1929988927 conn(0x7efd840632c0 msgr2=0x7efd840b1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:21.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.265+0000 7efd8ad2c700 1 -- 172.21.15.60:0/1929988927 shutdown_connections 2022-06-29T15:58:21.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.265+0000 7efd8ad2c700 1 -- 172.21.15.60:0/1929988927 wait complete. 2022-06-29T15:58:21.290 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-06-29T15:58:21.291 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T15:58:21.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.397+0000 7f2e362fe700 1 Processor -- start 2022-06-29T15:58:21.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.397+0000 7f2e362fe700 1 -- start start 2022-06-29T15:58:21.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.401+0000 7f2e362fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2e300b1a90 0x7f2e30066110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:21.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.401+0000 7f2e362fe700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e300ad350 con 0x7f2e30066600 2022-06-29T15:58:21.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.401+0000 7f2e362fe700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e300b2490 con 0x7f2e300acf70 2022-06-29T15:58:21.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.401+0000 7f2e362fe700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e30065c70 con 0x7f2e300b1a90 2022-06-29T15:58:21.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.401+0000 7f2e3489b700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2e30066600 0x7f2e30141c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52914/0 (socket says 172.21.15.60:52914) 2022-06-29T15:58:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.401+0000 7f2e3489b700 1 -- 172.21.15.60:0/1183043592 learned_addr learned my addr 172.21.15.60:0/1183043592 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.401+0000 7f2e2f7fe700 1 --2- 172.21.15.60:0/1183043592 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2e300b1a90 0x7f2e30066110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.401+0000 7f2e2effd700 1 -- 172.21.15.60:0/1183043592 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2788897409 0 0) 0x7f2e300ad350 con 0x7f2e30066600 2022-06-29T15:58:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.401+0000 7f2e2effd700 1 -- 172.21.15.60:0/1183043592 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e1c003260 con 0x7f2e30066600 2022-06-29T15:58:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.401+0000 7f2e2effd700 1 -- 172.21.15.60:0/1183043592 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3359348237 0 0) 0x7f2e300b2490 con 0x7f2e300acf70 2022-06-29T15:58:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.401+0000 7f2e2effd700 1 -- 172.21.15.60:0/1183043592 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e300ad350 con 0x7f2e300acf70 2022-06-29T15:58:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.401+0000 7f2e2f7fe700 1 -- 172.21.15.60:0/1183043592 >> v1:172.21.15.92:6789/0 conn(0x7f2e300acf70 legacy=0x7f2e30146200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:21.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.401+0000 7f2e2f7fe700 1 -- 172.21.15.60:0/1183043592 >> v1:172.21.15.60:6789/0 conn(0x7f2e30066600 legacy=0x7f2e30141c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:21.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.401+0000 7f2e2f7fe700 1 -- 172.21.15.60:0/1183043592 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e300b2110 con 0x7f2e300b1a90 2022-06-29T15:58:21.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.401+0000 7f2e2f7fe700 1 --2- 172.21.15.60:0/1183043592 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2e300b1a90 0x7f2e30066110 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f2e24002f70 tx=0x7f2e24006df0).ready entity=mon.2 client_cookie=1d2ad27f581876ed server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:21.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.401+0000 7f2e2effd700 1 -- 172.21.15.60:0/1183043592 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e2400d5c0 con 0x7f2e300b1a90 2022-06-29T15:58:21.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.401+0000 7f2e2effd700 1 -- 172.21.15.60:0/1183043592 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e24000e90 con 0x7f2e300b1a90 2022-06-29T15:58:21.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.401+0000 7f2e2effd700 1 -- 172.21.15.60:0/1183043592 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e2400d910 con 0x7f2e300b1a90 2022-06-29T15:58:21.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.401+0000 7f2e362fe700 1 -- 172.21.15.60:0/1183043592 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2e300b1a90 msgr2=0x7f2e30066110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:21.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.401+0000 7f2e362fe700 1 --2- 172.21.15.60:0/1183043592 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2e300b1a90 0x7f2e30066110 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f2e24002f70 tx=0x7f2e24006df0).stop 2022-06-29T15:58:21.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.401+0000 7f2e362fe700 1 -- 172.21.15.60:0/1183043592 shutdown_connections 2022-06-29T15:58:21.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.401+0000 7f2e362fe700 1 --2- 172.21.15.60:0/1183043592 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2e300b1a90 0x7f2e30066110 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.401+0000 7f2e362fe700 1 -- 172.21.15.60:0/1183043592 >> 172.21.15.60:0/1183043592 conn(0x7f2e30064ae0 msgr2=0x7f2e30064ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:21.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.401+0000 7f2e362fe700 1 -- 172.21.15.60:0/1183043592 shutdown_connections 2022-06-29T15:58:21.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.401+0000 7f2e362fe700 1 -- 172.21.15.60:0/1183043592 wait complete. 2022-06-29T15:58:21.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.405+0000 7f2e362fe700 1 Processor -- start 2022-06-29T15:58:21.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.405+0000 7f2e362fe700 1 -- start start 2022-06-29T15:58:21.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.405+0000 7f2e362fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2e30066600 0x7f2e300b6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:21.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.405+0000 7f2e362fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e300acf70 0x7f2e30155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:21.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.405+0000 7f2e362fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e300b1a90 0x7f2e30143410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:21.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.405+0000 7f2e362fe700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e30141b90 con 0x7f2e300acf70 2022-06-29T15:58:21.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.405+0000 7f2e362fe700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e30141e30 con 0x7f2e300b1a90 2022-06-29T15:58:21.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.405+0000 7f2e362fe700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e301420d0 con 0x7f2e30066600 2022-06-29T15:58:21.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.405+0000 7f2e2f7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e300acf70 0x7f2e30155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:21.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.405+0000 7f2e2f7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e300acf70 0x7f2e30155cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44832/0 (socket says 172.21.15.60:44832) 2022-06-29T15:58:21.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.405+0000 7f2e2f7fe700 1 -- 172.21.15.60:0/2979783572 learned_addr learned my addr 172.21.15.60:0/2979783572 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:21.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.405+0000 7f2e3489b700 1 --2- 172.21.15.60:0/2979783572 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e300b1a90 0x7f2e30143410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:21.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.405+0000 7f2e2ffff700 1 --2- 172.21.15.60:0/2979783572 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2e30066600 0x7f2e300b6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:21.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.409+0000 7f2e2f7fe700 1 -- 172.21.15.60:0/2979783572 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2e30066600 msgr2=0x7f2e300b6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:21.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.409+0000 7f2e2f7fe700 1 --2- 172.21.15.60:0/2979783572 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2e30066600 0x7f2e300b6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.409+0000 7f2e2f7fe700 1 -- 172.21.15.60:0/2979783572 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e300b1a90 msgr2=0x7f2e30143410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:21.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.409+0000 7f2e2f7fe700 1 --2- 172.21.15.60:0/2979783572 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e300b1a90 0x7f2e30143410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.409+0000 7f2e2f7fe700 1 -- 172.21.15.60:0/2979783572 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e2400b180 con 0x7f2e300acf70 2022-06-29T15:58:21.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.409+0000 7f2e2f7fe700 1 --2- 172.21.15.60:0/2979783572 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e300acf70 0x7f2e30155cc0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f2e24002f70 tx=0x7f2e24014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:21.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.409+0000 7f2e2d7fa700 1 -- 172.21.15.60:0/2979783572 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e24013de0 con 0x7f2e300acf70 2022-06-29T15:58:21.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.409+0000 7f2e2d7fa700 1 -- 172.21.15.60:0/2979783572 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e2401e400 con 0x7f2e300acf70 2022-06-29T15:58:21.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.409+0000 7f2e2d7fa700 1 -- 172.21.15.60:0/2979783572 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e2401e760 con 0x7f2e300acf70 2022-06-29T15:58:21.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.409+0000 7f2e362fe700 1 -- 172.21.15.60:0/2979783572 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e301457b0 con 0x7f2e300acf70 2022-06-29T15:58:21.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.409+0000 7f2e362fe700 1 -- 172.21.15.60:0/2979783572 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e300ae530 con 0x7f2e300acf70 2022-06-29T15:58:21.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.409+0000 7f2e362fe700 1 -- 172.21.15.60:0/2979783572 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e301561b0 con 0x7f2e300acf70 2022-06-29T15:58:21.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.409+0000 7f2e2d7fa700 1 -- 172.21.15.60:0/2979783572 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2e2401c070 con 0x7f2e300acf70 2022-06-29T15:58:21.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.409+0000 7f2e2d7fa700 1 --2- 172.21.15.60:0/2979783572 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2e10053b30 0x7f2e10055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:21.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.409+0000 7f2e2d7fa700 1 -- 172.21.15.60:0/2979783572 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f2e2406c520 con 0x7f2e300acf70 2022-06-29T15:58:21.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.409+0000 7f2e2ffff700 1 --2- 172.21.15.60:0/2979783572 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2e10053b30 0x7f2e10055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:21.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.413+0000 7f2e2ffff700 1 --2- 172.21.15.60:0/2979783572 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2e10053b30 0x7f2e10055ff0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f2e2000d000 tx=0x7f2e20007a20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:21.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.417+0000 7f2e2d7fa700 1 -- 172.21.15.60:0/2979783572 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2e24043770 con 0x7f2e300acf70 2022-06-29T15:58:21.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.545+0000 7f2e362fe700 1 -- 172.21.15.60:0/2979783572 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f2e300573d0 con 0x7f2e300acf70 2022-06-29T15:58:21.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.545+0000 7f2e2d7fa700 1 -- 172.21.15.60:0/2979783572 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v17) v1 ==== 110+0+10 (secure 0 0 0) 0x7f2e24023020 con 0x7f2e300acf70 2022-06-29T15:58:21.564 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T15:58:21.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.549+0000 7f2e362fe700 1 -- 172.21.15.60:0/2979783572 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2e10053b30 msgr2=0x7f2e10055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:21.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.549+0000 7f2e362fe700 1 --2- 172.21.15.60:0/2979783572 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2e10053b30 0x7f2e10055ff0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f2e2000d000 tx=0x7f2e20007a20).stop 2022-06-29T15:58:21.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.549+0000 7f2e362fe700 1 -- 172.21.15.60:0/2979783572 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e300acf70 msgr2=0x7f2e30155cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:21.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.549+0000 7f2e362fe700 1 --2- 172.21.15.60:0/2979783572 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e300acf70 0x7f2e30155cc0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f2e24002f70 tx=0x7f2e24014000).stop 2022-06-29T15:58:21.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.549+0000 7f2e362fe700 1 -- 172.21.15.60:0/2979783572 shutdown_connections 2022-06-29T15:58:21.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.549+0000 7f2e362fe700 1 --2- 172.21.15.60:0/2979783572 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2e10053b30 0x7f2e10055ff0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.549+0000 7f2e362fe700 1 --2- 172.21.15.60:0/2979783572 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e300b1a90 0x7f2e30143410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.549+0000 7f2e362fe700 1 --2- 172.21.15.60:0/2979783572 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2e300acf70 0x7f2e30155cc0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.549+0000 7f2e362fe700 1 --2- 172.21.15.60:0/2979783572 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2e30066600 0x7f2e300b6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.549+0000 7f2e362fe700 1 -- 172.21.15.60:0/2979783572 >> 172.21.15.60:0/2979783572 conn(0x7f2e30064ae0 msgr2=0x7f2e300b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:21.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.549+0000 7f2e362fe700 1 -- 172.21.15.60:0/2979783572 shutdown_connections 2022-06-29T15:58:21.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.549+0000 7f2e362fe700 1 -- 172.21.15.60:0/2979783572 wait complete. 2022-06-29T15:58:21.569 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T15:58:21.679 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.673+0000 7f16ddd18700 1 Processor -- start 2022-06-29T15:58:21.680 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.673+0000 7f16ddd18700 1 -- start start 2022-06-29T15:58:21.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.673+0000 7f16ddd18700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f16d805c5b0 0x7f16d805c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:21.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.673+0000 7f16ddd18700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f16d805fee0 0x7f16d8142a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:21.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.673+0000 7f16ddd18700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16d80b0a40 con 0x7f16d8060610 2022-06-29T15:58:21.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.673+0000 7f16ddd18700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16d80b0850 con 0x7f16d805c5b0 2022-06-29T15:58:21.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.673+0000 7f16ddd18700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16d80b0ce0 con 0x7f16d805fee0 2022-06-29T15:58:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.673+0000 7f16dcd16700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f16d805c5b0 0x7f16d805c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.677+0000 7f16d7fff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f16d8060610 0x7f16d813c740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52926/0 (socket says 172.21.15.60:52926) 2022-06-29T15:58:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.677+0000 7f16d7fff700 1 -- 172.21.15.60:0/4155015549 learned_addr learned my addr 172.21.15.60:0/4155015549 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.677+0000 7f16dd517700 1 --2- 172.21.15.60:0/4155015549 >> v2:172.21.15.92:3300/0 conn(0x7f16d805fee0 0x7f16d8142a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.677+0000 7f16dcd16700 1 -- 172.21.15.60:0/4155015549 >> v2:172.21.15.92:3300/0 conn(0x7f16d805fee0 msgr2=0x7f16d8142a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:21.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.677+0000 7f16dcd16700 1 --2- 172.21.15.60:0/4155015549 >> v2:172.21.15.92:3300/0 conn(0x7f16d805fee0 0x7f16d8142a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.677+0000 7f16dcd16700 1 -- 172.21.15.60:0/4155015549 >> v1:172.21.15.60:6789/0 conn(0x7f16d8060610 legacy=0x7f16d813c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:21.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.677+0000 7f16dcd16700 1 -- 172.21.15.60:0/4155015549 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d813ce50 con 0x7f16d805c5b0 2022-06-29T15:58:21.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.677+0000 7f16dcd16700 1 --2- 172.21.15.60:0/4155015549 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16d805c5b0 0x7f16d805c990 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f16c8002f70 tx=0x7f16c8006df0).ready entity=mon.0 client_cookie=2a3b68b61bc2153a server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:21.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.677+0000 7f16d77fe700 1 -- 172.21.15.60:0/4155015549 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16c800d570 con 0x7f16d805c5b0 2022-06-29T15:58:21.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.677+0000 7f16d77fe700 1 -- 172.21.15.60:0/4155015549 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16c8000e90 con 0x7f16d805c5b0 2022-06-29T15:58:21.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.677+0000 7f16d77fe700 1 -- 172.21.15.60:0/4155015549 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16c800d8c0 con 0x7f16d805c5b0 2022-06-29T15:58:21.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.677+0000 7f16ddd18700 1 -- 172.21.15.60:0/4155015549 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16d805c5b0 msgr2=0x7f16d805c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:21.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.677+0000 7f16ddd18700 1 --2- 172.21.15.60:0/4155015549 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16d805c5b0 0x7f16d805c990 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f16c8002f70 tx=0x7f16c8006df0).stop 2022-06-29T15:58:21.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.677+0000 7f16ddd18700 1 -- 172.21.15.60:0/4155015549 shutdown_connections 2022-06-29T15:58:21.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.677+0000 7f16ddd18700 1 --2- 172.21.15.60:0/4155015549 >> v2:172.21.15.92:3300/0 conn(0x7f16d805fee0 0x7f16d8142a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.677+0000 7f16ddd18700 1 --2- 172.21.15.60:0/4155015549 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16d805c5b0 0x7f16d805c990 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.677+0000 7f16ddd18700 1 -- 172.21.15.60:0/4155015549 >> 172.21.15.60:0/4155015549 conn(0x7f16d8061c80 msgr2=0x7f16d80637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:21.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.677+0000 7f16ddd18700 1 -- 172.21.15.60:0/4155015549 shutdown_connections 2022-06-29T15:58:21.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.677+0000 7f16ddd18700 1 -- 172.21.15.60:0/4155015549 wait complete. 2022-06-29T15:58:21.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.681+0000 7f16ddd18700 1 Processor -- start 2022-06-29T15:58:21.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.681+0000 7f16ddd18700 1 -- start start 2022-06-29T15:58:21.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.681+0000 7f16ddd18700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16d805c5b0 0x7f16d80b4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:21.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.681+0000 7f16ddd18700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16d805fee0 0x7f16d8139b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:21.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.681+0000 7f16ddd18700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16d8060610 0x7f16d813a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:21.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.681+0000 7f16ddd18700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d81563c0 con 0x7f16d805c5b0 2022-06-29T15:58:21.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.681+0000 7f16ddd18700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d80b0210 con 0x7f16d8060610 2022-06-29T15:58:21.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.681+0000 7f16ddd18700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16d8155f60 con 0x7f16d805fee0 2022-06-29T15:58:21.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.681+0000 7f16dcd16700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16d805c5b0 0x7f16d80b4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:21.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.681+0000 7f16dcd16700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16d805c5b0 0x7f16d80b4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44844/0 (socket says 172.21.15.60:44844) 2022-06-29T15:58:21.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.681+0000 7f16dcd16700 1 -- 172.21.15.60:0/3403542598 learned_addr learned my addr 172.21.15.60:0/3403542598 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:21.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.681+0000 7f16d7fff700 1 --2- 172.21.15.60:0/3403542598 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16d805fee0 0x7f16d8139b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:21.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.681+0000 7f16dd517700 1 --2- 172.21.15.60:0/3403542598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16d8060610 0x7f16d813a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:21.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.681+0000 7f16dcd16700 1 -- 172.21.15.60:0/3403542598 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16d805fee0 msgr2=0x7f16d8139b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:21.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.681+0000 7f16dcd16700 1 --2- 172.21.15.60:0/3403542598 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16d805fee0 0x7f16d8139b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.681+0000 7f16dcd16700 1 -- 172.21.15.60:0/3403542598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16d8060610 msgr2=0x7f16d813a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:21.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.681+0000 7f16dcd16700 1 --2- 172.21.15.60:0/3403542598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16d8060610 0x7f16d813a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.681+0000 7f16dcd16700 1 -- 172.21.15.60:0/3403542598 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c800b180 con 0x7f16d805c5b0 2022-06-29T15:58:21.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.681+0000 7f16d7fff700 1 --2- 172.21.15.60:0/3403542598 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16d805fee0 0x7f16d8139b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:58:21.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.681+0000 7f16dd517700 1 --2- 172.21.15.60:0/3403542598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16d8060610 0x7f16d813a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:58:21.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.681+0000 7f16dcd16700 1 --2- 172.21.15.60:0/3403542598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16d805c5b0 0x7f16d80b4b40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f16c8002f70 tx=0x7f16c8014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:21.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.681+0000 7f16d5ffb700 1 -- 172.21.15.60:0/3403542598 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16c8016070 con 0x7f16d805c5b0 2022-06-29T15:58:21.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.681+0000 7f16d5ffb700 1 -- 172.21.15.60:0/3403542598 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16c800d8f0 con 0x7f16d805c5b0 2022-06-29T15:58:21.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.681+0000 7f16ddd18700 1 -- 172.21.15.60:0/3403542598 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16d80aff70 con 0x7f16d805c5b0 2022-06-29T15:58:21.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.681+0000 7f16d5ffb700 1 -- 172.21.15.60:0/3403542598 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16c801e450 con 0x7f16d805c5b0 2022-06-29T15:58:21.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.681+0000 7f16ddd18700 1 -- 172.21.15.60:0/3403542598 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16d8142830 con 0x7f16d805c5b0 2022-06-29T15:58:21.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.685+0000 7f16d5ffb700 1 -- 172.21.15.60:0/3403542598 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f16c800f030 con 0x7f16d805c5b0 2022-06-29T15:58:21.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.685+0000 7f16d5ffb700 1 --2- 172.21.15.60:0/3403542598 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f16bc053b00 0x7f16bc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:21.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.685+0000 7f16d5ffb700 1 -- 172.21.15.60:0/3403542598 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f16c806ca00 con 0x7f16d805c5b0 2022-06-29T15:58:21.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.685+0000 7f16ddd18700 1 -- 172.21.15.60:0/3403542598 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16d80573d0 con 0x7f16d805c5b0 2022-06-29T15:58:21.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.685+0000 7f16d7fff700 1 --2- 172.21.15.60:0/3403542598 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f16bc053b00 0x7f16bc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:21.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.685+0000 7f16d7fff700 1 --2- 172.21.15.60:0/3403542598 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f16bc053b00 0x7f16bc055fc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f16c000b000 tx=0x7f16c0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:21.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.693+0000 7f16d5ffb700 1 -- 172.21.15.60:0/3403542598 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f16c8023020 con 0x7f16d805c5b0 2022-06-29T15:58:21.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.821+0000 7f16ddd18700 1 -- 172.21.15.60:0/3403542598 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f16d80b5ba0 con 0x7f16d805c5b0 2022-06-29T15:58:21.829 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.821+0000 7f16d5ffb700 1 -- 172.21.15.60:0/3403542598 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v17) v1 ==== 92+0+10 (secure 0 0 0) 0x7f16c8023020 con 0x7f16d805c5b0 2022-06-29T15:58:21.829 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T15:58:21.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.825+0000 7f16ddd18700 1 -- 172.21.15.60:0/3403542598 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f16bc053b00 msgr2=0x7f16bc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:21.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.825+0000 7f16ddd18700 1 --2- 172.21.15.60:0/3403542598 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f16bc053b00 0x7f16bc055fc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f16c000b000 tx=0x7f16c0009040).stop 2022-06-29T15:58:21.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.825+0000 7f16ddd18700 1 -- 172.21.15.60:0/3403542598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16d805c5b0 msgr2=0x7f16d80b4b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:21.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.825+0000 7f16ddd18700 1 --2- 172.21.15.60:0/3403542598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16d805c5b0 0x7f16d80b4b40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f16c8002f70 tx=0x7f16c8014000).stop 2022-06-29T15:58:21.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.825+0000 7f16ddd18700 1 -- 172.21.15.60:0/3403542598 shutdown_connections 2022-06-29T15:58:21.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.825+0000 7f16ddd18700 1 --2- 172.21.15.60:0/3403542598 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f16bc053b00 0x7f16bc055fc0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.825+0000 7f16ddd18700 1 --2- 172.21.15.60:0/3403542598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16d8060610 0x7f16d813a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.825+0000 7f16ddd18700 1 --2- 172.21.15.60:0/3403542598 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16d805fee0 0x7f16d8139b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.825+0000 7f16ddd18700 1 --2- 172.21.15.60:0/3403542598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16d805c5b0 0x7f16d80b4b40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.825+0000 7f16ddd18700 1 -- 172.21.15.60:0/3403542598 >> 172.21.15.60:0/3403542598 conn(0x7f16d8061c80 msgr2=0x7f16d80668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:21.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.825+0000 7f16ddd18700 1 -- 172.21.15.60:0/3403542598 shutdown_connections 2022-06-29T15:58:21.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.825+0000 7f16ddd18700 1 -- 172.21.15.60:0/3403542598 wait complete. 2022-06-29T15:58:21.843 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-06-29T15:58:21.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facfe142700 1 Processor -- start 2022-06-29T15:58:21.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facfe142700 1 -- start start 2022-06-29T15:58:21.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facfe142700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7facf80b12e0 0x7facf81475b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:21.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facfe142700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf80b65b0 con 0x7facf80b61d0 2022-06-29T15:58:21.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facfe142700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf80adf20 con 0x7facf8147aa0 2022-06-29T15:58:21.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facfe142700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facf8147110 con 0x7facf80b12e0 2022-06-29T15:58:21.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facf77fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7facf80b61d0 0x7facf805f0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52938/0 (socket says 172.21.15.60:52938) 2022-06-29T15:58:21.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facf77fe700 1 -- 172.21.15.60:0/3593520511 learned_addr learned my addr 172.21.15.60:0/3593520511 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:21.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facf6ffd700 1 --2- 172.21.15.60:0/3593520511 >> v2:172.21.15.92:3300/0 conn(0x7facf80b12e0 0x7facf81475b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:21.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facf67fc700 1 -- 172.21.15.60:0/3593520511 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2750052008 0 0) 0x7facf80b65b0 con 0x7facf80b61d0 2022-06-29T15:58:21.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facf67fc700 1 -- 172.21.15.60:0/3593520511 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facdc003260 con 0x7facf80b61d0 2022-06-29T15:58:21.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facf67fc700 1 -- 172.21.15.60:0/3593520511 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1219742706 0 0) 0x7facdc003260 con 0x7facf80b61d0 2022-06-29T15:58:21.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facf67fc700 1 -- 172.21.15.60:0/3593520511 >> v2:172.21.15.92:3300/0 conn(0x7facf80b12e0 msgr2=0x7facf81475b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:21.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facf67fc700 1 --2- 172.21.15.60:0/3593520511 >> v2:172.21.15.92:3300/0 conn(0x7facf80b12e0 0x7facf81475b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facf67fc700 1 -- 172.21.15.60:0/3593520511 >> v1:172.21.15.92:6789/0 conn(0x7facf8147aa0 legacy=0x7facf80af290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:21.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facf67fc700 1 -- 172.21.15.60:0/3593520511 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf8146e30 con 0x7facf80b61d0 2022-06-29T15:58:21.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facf67fc700 1 -- 172.21.15.60:0/3593520511 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7face0001660 con 0x7facf80b61d0 2022-06-29T15:58:21.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facf67fc700 1 -- 172.21.15.60:0/3593520511 >> v1:172.21.15.60:6789/0 conn(0x7facf80b61d0 legacy=0x7facf805f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:21.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facf67fc700 1 --2- 172.21.15.60:0/3593520511 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facdc003750 0x7facdc003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:21.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facf67fc700 1 --2- 172.21.15.60:0/3593520511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7facdc004500 0x7facdc004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:21.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facf67fc700 1 --2- 172.21.15.60:0/3593520511 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7facdc004e70 0x7facdc00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:21.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facf67fc700 1 -- 172.21.15.60:0/3593520511 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf8147110 con 0x7facdc003750 2022-06-29T15:58:21.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facf67fc700 1 -- 172.21.15.60:0/3593520511 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facdc00bdd0 con 0x7facdc004500 2022-06-29T15:58:21.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facf67fc700 1 -- 172.21.15.60:0/3593520511 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facdc00bf60 con 0x7facdc004e70 2022-06-29T15:58:21.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facf77fe700 1 --2- 172.21.15.60:0/3593520511 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facdc003750 0x7facdc003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:21.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facf6ffd700 1 --2- 172.21.15.60:0/3593520511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7facdc004500 0x7facdc004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:21.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facf7fff700 1 --2- 172.21.15.60:0/3593520511 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7facdc004e70 0x7facdc00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:21.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facf77fe700 1 -- 172.21.15.60:0/3593520511 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7facdc004e70 msgr2=0x7facdc00b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:21.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facf77fe700 1 --2- 172.21.15.60:0/3593520511 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7facdc004e70 0x7facdc00b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facf77fe700 1 -- 172.21.15.60:0/3593520511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7facdc004500 msgr2=0x7facdc004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:21.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facf77fe700 1 --2- 172.21.15.60:0/3593520511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7facdc004500 0x7facdc004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facf77fe700 1 -- 172.21.15.60:0/3593520511 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facdc003260 con 0x7facdc003750 2022-06-29T15:58:21.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facf7fff700 1 --2- 172.21.15.60:0/3593520511 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7facdc004e70 0x7facdc00b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T15:58:21.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facf6ffd700 1 --2- 172.21.15.60:0/3593520511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7facdc004500 0x7facdc004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:58:21.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facf7fff700 1 -- 172.21.15.60:0/3593520511 reap_dead start 2022-06-29T15:58:21.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facf77fe700 1 --2- 172.21.15.60:0/3593520511 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facdc003750 0x7facdc003b70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7face0001370 tx=0x7facf80ad0e0).ready entity=mon.0 client_cookie=9a92a34b37f5406c server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:21.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facf67fc700 1 -- 172.21.15.60:0/3593520511 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7face00070d0 con 0x7facdc003750 2022-06-29T15:58:21.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facf67fc700 1 -- 172.21.15.60:0/3593520511 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7face0007230 con 0x7facdc003750 2022-06-29T15:58:21.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.945+0000 7facf67fc700 1 -- 172.21.15.60:0/3593520511 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7face000c8f0 con 0x7facdc003750 2022-06-29T15:58:21.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.949+0000 7facfe142700 1 -- 172.21.15.60:0/3593520511 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facdc003750 msgr2=0x7facdc003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:21.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.949+0000 7facfe142700 1 --2- 172.21.15.60:0/3593520511 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facdc003750 0x7facdc003b70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7face0001370 tx=0x7facf80ad0e0).stop 2022-06-29T15:58:21.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.949+0000 7facfe142700 1 -- 172.21.15.60:0/3593520511 shutdown_connections 2022-06-29T15:58:21.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.949+0000 7facfe142700 1 --2- 172.21.15.60:0/3593520511 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facdc003750 0x7facdc003b70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.949+0000 7facfe142700 1 -- 172.21.15.60:0/3593520511 >> 172.21.15.60:0/3593520511 conn(0x7facf8063b50 msgr2=0x7facf80b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:21.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.949+0000 7facfe142700 1 -- 172.21.15.60:0/3593520511 shutdown_connections 2022-06-29T15:58:21.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.949+0000 7facfe142700 1 -- 172.21.15.60:0/3593520511 wait complete. 2022-06-29T15:58:21.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.949+0000 7facfe142700 1 Processor -- start 2022-06-29T15:58:21.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.949+0000 7facfe142700 1 -- start start 2022-06-29T15:58:21.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.949+0000 7facfe142700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facdc003750 0x7facf814ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:21.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.949+0000 7facfe142700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7facf8148340 0x7facf81564b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:21.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.949+0000 7facfe142700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7facf81569a0 0x7facf8156e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:21.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.949+0000 7facfe142700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf81498f0 con 0x7facdc003750 2022-06-29T15:58:21.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.949+0000 7facfe142700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf8065d30 con 0x7facf81569a0 2022-06-29T15:58:21.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.949+0000 7facfe142700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf8065100 con 0x7facf8148340 2022-06-29T15:58:21.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.949+0000 7facf77fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facdc003750 0x7facf814ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:21.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.949+0000 7facf77fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facdc003750 0x7facf814ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44864/0 (socket says 172.21.15.60:44864) 2022-06-29T15:58:21.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.949+0000 7facf77fe700 1 -- 172.21.15.60:0/465911404 learned_addr learned my addr 172.21.15.60:0/465911404 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:21.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.949+0000 7facf7fff700 1 --2- 172.21.15.60:0/465911404 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7facf81569a0 0x7facf8156e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:21.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.949+0000 7facf6ffd700 1 --2- 172.21.15.60:0/465911404 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7facf8148340 0x7facf81564b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:21.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.953+0000 7facf77fe700 1 -- 172.21.15.60:0/465911404 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7facf8148340 msgr2=0x7facf81564b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:21.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.953+0000 7facf77fe700 1 --2- 172.21.15.60:0/465911404 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7facf8148340 0x7facf81564b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.953+0000 7facf77fe700 1 -- 172.21.15.60:0/465911404 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7facf81569a0 msgr2=0x7facf8156e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:21.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.953+0000 7facf77fe700 1 --2- 172.21.15.60:0/465911404 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7facf81569a0 0x7facf8156e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:21.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.953+0000 7facf77fe700 1 -- 172.21.15.60:0/465911404 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf8149b20 con 0x7facdc003750 2022-06-29T15:58:21.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.953+0000 7facf7fff700 1 --2- 172.21.15.60:0/465911404 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7facf81569a0 0x7facf8156e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:21.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.953+0000 7facf6ffd700 1 --2- 172.21.15.60:0/465911404 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7facf8148340 0x7facf81564b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:21.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.953+0000 7facf77fe700 1 --2- 172.21.15.60:0/465911404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facdc003750 0x7facf814ac40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7face0001370 tx=0x7face0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:21.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.953+0000 7facf4ff9700 1 -- 172.21.15.60:0/465911404 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7face0004710 con 0x7facdc003750 2022-06-29T15:58:21.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.953+0000 7facf4ff9700 1 -- 172.21.15.60:0/465911404 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7face000ced0 con 0x7facdc003750 2022-06-29T15:58:21.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.953+0000 7facfe142700 1 -- 172.21.15.60:0/465911404 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facf8064b80 con 0x7facdc003750 2022-06-29T15:58:21.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.953+0000 7facf4ff9700 1 -- 172.21.15.60:0/465911404 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7face0004a60 con 0x7facdc003750 2022-06-29T15:58:21.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.953+0000 7facfe142700 1 -- 172.21.15.60:0/465911404 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facf8159850 con 0x7facdc003750 2022-06-29T15:58:21.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.953+0000 7facfe142700 1 -- 172.21.15.60:0/465911404 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facf80573d0 con 0x7facdc003750 2022-06-29T15:58:21.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.953+0000 7facf4ff9700 1 -- 172.21.15.60:0/465911404 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7face0027070 con 0x7facdc003750 2022-06-29T15:58:21.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.953+0000 7facf4ff9700 1 --2- 172.21.15.60:0/465911404 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7facd0053b00 0x7facd0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:21.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.953+0000 7facf4ff9700 1 -- 172.21.15.60:0/465911404 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5253+0+0 (secure 0 0 0) 0x7face006c620 con 0x7facdc003750 2022-06-29T15:58:21.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.953+0000 7facf6ffd700 1 --2- 172.21.15.60:0/465911404 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7facd0053b00 0x7facd0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:21.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.957+0000 7facf6ffd700 1 --2- 172.21.15.60:0/465911404 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7facd0053b00 0x7facd0055fc0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7facf000b000 tx=0x7facf0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:21.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:21.961+0000 7facf4ff9700 1 -- 172.21.15.60:0/465911404 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7face0043870 con 0x7facdc003750 2022-06-29T15:58:22.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.097+0000 7facfe142700 1 -- 172.21.15.60:0/465911404 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7facf80686b0 con 0x7facdc003750 2022-06-29T15:58:22.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.097+0000 7facf4ff9700 1 -- 172.21.15.60:0/465911404 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v18) v1 ==== 84+0+143 (secure 0 0 0) 0x7face0046e90 con 0x7facdc003750 2022-06-29T15:58:22.105 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:58:22.105 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2022-06-29T15:58:22.105 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:22.106 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2022-06-29T15:58:22.106 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": "device_health_metrics" 2022-06-29T15:58:22.106 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:22.107 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:22.107 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 2, 2022-06-29T15:58:22.107 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": "rbd" 2022-06-29T15:58:22.107 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:22.108 INFO:teuthology.orchestra.run.smithi060.stdout:] 2022-06-29T15:58:22.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.101+0000 7facfe142700 1 -- 172.21.15.60:0/465911404 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7facd0053b00 msgr2=0x7facd0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:22.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.101+0000 7facfe142700 1 --2- 172.21.15.60:0/465911404 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7facd0053b00 0x7facd0055fc0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7facf000b000 tx=0x7facf0008040).stop 2022-06-29T15:58:22.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.101+0000 7facfe142700 1 -- 172.21.15.60:0/465911404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facdc003750 msgr2=0x7facf814ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:22.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.101+0000 7facfe142700 1 --2- 172.21.15.60:0/465911404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facdc003750 0x7facf814ac40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7face0001370 tx=0x7face0015000).stop 2022-06-29T15:58:22.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.101+0000 7facfe142700 1 -- 172.21.15.60:0/465911404 shutdown_connections 2022-06-29T15:58:22.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.101+0000 7facfe142700 1 --2- 172.21.15.60:0/465911404 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7facd0053b00 0x7facd0055fc0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:22.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.101+0000 7facfe142700 1 --2- 172.21.15.60:0/465911404 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7facf81569a0 0x7facf8156e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:22.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.101+0000 7facfe142700 1 --2- 172.21.15.60:0/465911404 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7facf8148340 0x7facf81564b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:22.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.101+0000 7facfe142700 1 --2- 172.21.15.60:0/465911404 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7facdc003750 0x7facf814ac40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:22.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.101+0000 7facfe142700 1 -- 172.21.15.60:0/465911404 >> 172.21.15.60:0/465911404 conn(0x7facf8063b50 msgr2=0x7facf8143d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:22.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.101+0000 7facfe142700 1 -- 172.21.15.60:0/465911404 shutdown_connections 2022-06-29T15:58:22.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.101+0000 7facfe142700 1 -- 172.21.15.60:0/465911404 wait complete. 2022-06-29T15:58:22.122 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-06-29T15:58:22.123 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-06-29T15:58:22.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.221+0000 7f762ac6e700 1 Processor -- start 2022-06-29T15:58:22.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.221+0000 7f762ac6e700 1 -- start start 2022-06-29T15:58:22.230 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.225+0000 7f762ac6e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7624062260 0x7f76241400a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:22.230 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.225+0000 7f762ac6e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7624061620 0x7f7624140590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:22.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.225+0000 7f762ac6e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7624144020 con 0x7f76240b6a50 2022-06-29T15:58:22.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.225+0000 7f762ac6e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76240b6e30 con 0x7f7624062260 2022-06-29T15:58:22.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.225+0000 7f762ac6e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76240b2370 con 0x7f7624061620 2022-06-29T15:58:22.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.225+0000 7f7628a0a700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f76240b6a50 0x7f76241446f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52958/0 (socket says 172.21.15.60:52958) 2022-06-29T15:58:22.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.225+0000 7f7628a0a700 1 -- 172.21.15.60:0/62833594 learned_addr learned my addr 172.21.15.60:0/62833594 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:22.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.225+0000 7f7623fff700 1 --2- 172.21.15.60:0/62833594 >> v2:172.21.15.92:3300/0 conn(0x7f7624062260 0x7f76241400a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:22.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.225+0000 7f762920b700 1 --2- 172.21.15.60:0/62833594 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7624061620 0x7f7624140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:22.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.225+0000 7f76237fe700 1 -- 172.21.15.60:0/62833594 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1995948867 0 0) 0x7f7624144020 con 0x7f76240b6a50 2022-06-29T15:58:22.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.225+0000 7f76237fe700 1 -- 172.21.15.60:0/62833594 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7608003260 con 0x7f76240b6a50 2022-06-29T15:58:22.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.225+0000 7f762920b700 1 -- 172.21.15.60:0/62833594 >> v2:172.21.15.92:3300/0 conn(0x7f7624062260 msgr2=0x7f76241400a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:22.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.225+0000 7f762920b700 1 --2- 172.21.15.60:0/62833594 >> v2:172.21.15.92:3300/0 conn(0x7f7624062260 0x7f76241400a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:22.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.225+0000 7f762920b700 1 -- 172.21.15.60:0/62833594 >> v1:172.21.15.60:6789/0 conn(0x7f76240b6a50 legacy=0x7f76241446f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:22.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.225+0000 7f762920b700 1 -- 172.21.15.60:0/62833594 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7624144250 con 0x7f7624061620 2022-06-29T15:58:22.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.225+0000 7f7623fff700 1 --2- 172.21.15.60:0/62833594 >> v2:172.21.15.92:3300/0 conn(0x7f7624062260 0x7f76241400a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:22.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.225+0000 7f762920b700 1 --2- 172.21.15.60:0/62833594 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7624061620 0x7f7624140590 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f761c0032f0 tx=0x7f761c0091d0).ready entity=mon.2 client_cookie=5f6558a66faa3c14 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:22.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.225+0000 7f76237fe700 1 -- 172.21.15.60:0/62833594 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f761c00b600 con 0x7f7624061620 2022-06-29T15:58:22.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.225+0000 7f76237fe700 1 -- 172.21.15.60:0/62833594 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f761c00b760 con 0x7f7624061620 2022-06-29T15:58:22.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.225+0000 7f76237fe700 1 -- 172.21.15.60:0/62833594 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f761c00b8d0 con 0x7f7624061620 2022-06-29T15:58:22.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.225+0000 7f762ac6e700 1 -- 172.21.15.60:0/62833594 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7624061620 msgr2=0x7f7624140590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:22.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.225+0000 7f762ac6e700 1 --2- 172.21.15.60:0/62833594 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7624061620 0x7f7624140590 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f761c0032f0 tx=0x7f761c0091d0).stop 2022-06-29T15:58:22.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.225+0000 7f762ac6e700 1 -- 172.21.15.60:0/62833594 shutdown_connections 2022-06-29T15:58:22.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.225+0000 7f762ac6e700 1 --2- 172.21.15.60:0/62833594 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7624061620 0x7f7624140590 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:22.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.225+0000 7f762ac6e700 1 --2- 172.21.15.60:0/62833594 >> v2:172.21.15.92:3300/0 conn(0x7f7624062260 0x7f76241400a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:22.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.225+0000 7f762ac6e700 1 -- 172.21.15.60:0/62833594 >> 172.21.15.60:0/62833594 conn(0x7f76240643e0 msgr2=0x7f76240b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:22.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.225+0000 7f762ac6e700 1 -- 172.21.15.60:0/62833594 shutdown_connections 2022-06-29T15:58:22.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.225+0000 7f762ac6e700 1 -- 172.21.15.60:0/62833594 wait complete. 2022-06-29T15:58:22.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.229+0000 7f762ac6e700 1 Processor -- start 2022-06-29T15:58:22.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.229+0000 7f762ac6e700 1 -- start start 2022-06-29T15:58:22.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.229+0000 7f762ac6e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7624061620 0x7f7624066600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:22.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.229+0000 7f762ac6e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7624062260 0x7f76240ad440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:22.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.229+0000 7f762ac6e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76240b6a50 0x7f7624091460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:22.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.229+0000 7f762ac6e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76240aef30 con 0x7f76240b6a50 2022-06-29T15:58:22.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.229+0000 7f762ac6e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76240ae6f0 con 0x7f7624061620 2022-06-29T15:58:22.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.229+0000 7f762ac6e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76240ae990 con 0x7f7624062260 2022-06-29T15:58:22.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.229+0000 7f762920b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76240b6a50 0x7f7624091460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:22.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.229+0000 7f762920b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76240b6a50 0x7f7624091460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44882/0 (socket says 172.21.15.60:44882) 2022-06-29T15:58:22.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.229+0000 7f762920b700 1 -- 172.21.15.60:0/560526081 learned_addr learned my addr 172.21.15.60:0/560526081 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:22.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.229+0000 7f7623fff700 1 --2- 172.21.15.60:0/560526081 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7624062260 0x7f76240ad440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:22.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.229+0000 7f7628a0a700 1 --2- 172.21.15.60:0/560526081 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7624061620 0x7f7624066600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:22.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.229+0000 7f762920b700 1 -- 172.21.15.60:0/560526081 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7624062260 msgr2=0x7f76240ad440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:22.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.229+0000 7f762920b700 1 --2- 172.21.15.60:0/560526081 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7624062260 0x7f76240ad440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:22.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.229+0000 7f762920b700 1 -- 172.21.15.60:0/560526081 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7624061620 msgr2=0x7f7624066600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:22.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.229+0000 7f762920b700 1 --2- 172.21.15.60:0/560526081 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7624061620 0x7f7624066600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:22.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.229+0000 7f762920b700 1 -- 172.21.15.60:0/560526081 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761000b180 con 0x7f76240b6a50 2022-06-29T15:58:22.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.229+0000 7f762920b700 1 --2- 172.21.15.60:0/560526081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76240b6a50 0x7f7624091460 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f761c0032f0 tx=0x7f761c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:22.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.229+0000 7f7621ffb700 1 -- 172.21.15.60:0/560526081 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f761c00b600 con 0x7f76240b6a50 2022-06-29T15:58:22.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.233+0000 7f7621ffb700 1 -- 172.21.15.60:0/560526081 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f761c011b70 con 0x7f76240b6a50 2022-06-29T15:58:22.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.233+0000 7f762ac6e700 1 -- 172.21.15.60:0/560526081 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f761c009470 con 0x7f76240b6a50 2022-06-29T15:58:22.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.233+0000 7f7621ffb700 1 -- 172.21.15.60:0/560526081 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f761c01f3f0 con 0x7f76240b6a50 2022-06-29T15:58:22.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.233+0000 7f762ac6e700 1 -- 172.21.15.60:0/560526081 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76240b0110 con 0x7f76240b6a50 2022-06-29T15:58:22.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.233+0000 7f762ac6e700 1 -- 172.21.15.60:0/560526081 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76240573d0 con 0x7f76240b6a50 2022-06-29T15:58:22.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.233+0000 7f7621ffb700 1 -- 172.21.15.60:0/560526081 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f761c026030 con 0x7f76240b6a50 2022-06-29T15:58:22.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.233+0000 7f7621ffb700 1 --2- 172.21.15.60:0/560526081 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7604053b00 0x7f7604055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:22.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.233+0000 7f7621ffb700 1 -- 172.21.15.60:0/560526081 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f761c06c5c0 con 0x7f76240b6a50 2022-06-29T15:58:22.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.233+0000 7f7628a0a700 1 --2- 172.21.15.60:0/560526081 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7604053b00 0x7f7604055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:22.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.237+0000 7f7628a0a700 1 --2- 172.21.15.60:0/560526081 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7604053b00 0x7f7604055fc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f761800d000 tx=0x7f7618009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:22.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.241+0000 7f7621ffb700 1 -- 172.21.15.60:0/560526081 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f761c0471a0 con 0x7f76240b6a50 2022-06-29T15:58:22.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:22.365+0000 7f762ac6e700 1 -- 172.21.15.60:0/560526081 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f76240ace80 con 0x7f76240b6a50 2022-06-29T15:58:23.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.009+0000 7f7621ffb700 1 -- 172.21.15.60:0/560526081 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v19) v1 ==== 133+0+0 (secure 0 0 0) 0x7f761c046d90 con 0x7f76240b6a50 2022-06-29T15:58:23.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.013+0000 7f762ac6e700 1 -- 172.21.15.60:0/560526081 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7604053b00 msgr2=0x7f7604055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:23.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.013+0000 7f762ac6e700 1 --2- 172.21.15.60:0/560526081 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7604053b00 0x7f7604055fc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f761800d000 tx=0x7f7618009040).stop 2022-06-29T15:58:23.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.013+0000 7f762ac6e700 1 -- 172.21.15.60:0/560526081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76240b6a50 msgr2=0x7f7624091460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:23.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.013+0000 7f762ac6e700 1 --2- 172.21.15.60:0/560526081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76240b6a50 0x7f7624091460 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f761c0032f0 tx=0x7f761c016000).stop 2022-06-29T15:58:23.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.017+0000 7f762ac6e700 1 -- 172.21.15.60:0/560526081 shutdown_connections 2022-06-29T15:58:23.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.017+0000 7f762ac6e700 1 --2- 172.21.15.60:0/560526081 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7604053b00 0x7f7604055fc0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:23.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.017+0000 7f762ac6e700 1 --2- 172.21.15.60:0/560526081 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76240b6a50 0x7f7624091460 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:23.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.017+0000 7f762ac6e700 1 --2- 172.21.15.60:0/560526081 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7624062260 0x7f76240ad440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:23.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.017+0000 7f762ac6e700 1 --2- 172.21.15.60:0/560526081 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7624061620 0x7f7624066600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:23.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.017+0000 7f762ac6e700 1 -- 172.21.15.60:0/560526081 >> 172.21.15.60:0/560526081 conn(0x7f76240643e0 msgr2=0x7f76240ac0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:23.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.017+0000 7f762ac6e700 1 -- 172.21.15.60:0/560526081 shutdown_connections 2022-06-29T15:58:23.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.017+0000 7f762ac6e700 1 -- 172.21.15.60:0/560526081 wait complete. 2022-06-29T15:58:23.025 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2022-06-29T15:58:23.034 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-06-29T15:58:23.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.137+0000 7f7f46b1f700 1 Processor -- start 2022-06-29T15:58:23.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.137+0000 7f7f46b1f700 1 -- start start 2022-06-29T15:58:23.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.141+0000 7f7f46b1f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f400adc80 0x7f7f400af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:23.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.141+0000 7f7f46b1f700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f400b65b0 con 0x7f7f40060b60 2022-06-29T15:58:23.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.141+0000 7f7f46b1f700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f400605e0 con 0x7f7f400b61d0 2022-06-29T15:58:23.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.141+0000 7f7f46b1f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f40060880 con 0x7f7f400adc80 2022-06-29T15:58:23.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.141+0000 7f7f450bc700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f7f40060b60 0x7f7f4013c010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52976/0 (socket says 172.21.15.60:52976) 2022-06-29T15:58:23.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.141+0000 7f7f450bc700 1 -- 172.21.15.60:0/4209427340 learned_addr learned my addr 172.21.15.60:0/4209427340 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:23.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.141+0000 7f7f37fff700 1 --2- 172.21.15.60:0/4209427340 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f400adc80 0x7f7f400af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:23.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.141+0000 7f7f377fe700 1 -- 172.21.15.60:0/4209427340 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2345837562 0 0) 0x7f7f400b65b0 con 0x7f7f40060b60 2022-06-29T15:58:23.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.141+0000 7f7f377fe700 1 -- 172.21.15.60:0/4209427340 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f2c003260 con 0x7f7f40060b60 2022-06-29T15:58:23.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.141+0000 7f7f377fe700 1 -- 172.21.15.60:0/4209427340 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3798525789 0 0) 0x7f7f400605e0 con 0x7f7f400b61d0 2022-06-29T15:58:23.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.141+0000 7f7f377fe700 1 -- 172.21.15.60:0/4209427340 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f400b65b0 con 0x7f7f400b61d0 2022-06-29T15:58:23.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.141+0000 7f7f37fff700 1 -- 172.21.15.60:0/4209427340 >> v1:172.21.15.92:6789/0 conn(0x7f7f400b61d0 legacy=0x7f7f40142a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:23.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.141+0000 7f7f37fff700 1 -- 172.21.15.60:0/4209427340 >> v1:172.21.15.60:6789/0 conn(0x7f7f40060b60 legacy=0x7f7f4013c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:23.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.141+0000 7f7f37fff700 1 -- 172.21.15.60:0/4209427340 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f40067340 con 0x7f7f400adc80 2022-06-29T15:58:23.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.141+0000 7f7f37fff700 1 --2- 172.21.15.60:0/4209427340 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f400adc80 0x7f7f400af8d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f7f30002f70 tx=0x7f7f30006df0).ready entity=mon.2 client_cookie=7a7e37a8fef957be server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:23.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.141+0000 7f7f377fe700 1 -- 172.21.15.60:0/4209427340 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f3000d5c0 con 0x7f7f400adc80 2022-06-29T15:58:23.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.141+0000 7f7f377fe700 1 -- 172.21.15.60:0/4209427340 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f30000e90 con 0x7f7f400adc80 2022-06-29T15:58:23.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.141+0000 7f7f377fe700 1 -- 172.21.15.60:0/4209427340 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f3000d910 con 0x7f7f400adc80 2022-06-29T15:58:23.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.141+0000 7f7f46b1f700 1 -- 172.21.15.60:0/4209427340 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f400adc80 msgr2=0x7f7f400af8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:23.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.141+0000 7f7f46b1f700 1 --2- 172.21.15.60:0/4209427340 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f400adc80 0x7f7f400af8d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f7f30002f70 tx=0x7f7f30006df0).stop 2022-06-29T15:58:23.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.141+0000 7f7f46b1f700 1 -- 172.21.15.60:0/4209427340 shutdown_connections 2022-06-29T15:58:23.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.141+0000 7f7f46b1f700 1 --2- 172.21.15.60:0/4209427340 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f400adc80 0x7f7f400af8d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:23.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.141+0000 7f7f46b1f700 1 -- 172.21.15.60:0/4209427340 >> 172.21.15.60:0/4209427340 conn(0x7f7f40063b50 msgr2=0x7f7f400b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:23.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.141+0000 7f7f46b1f700 1 -- 172.21.15.60:0/4209427340 shutdown_connections 2022-06-29T15:58:23.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.145+0000 7f7f46b1f700 1 -- 172.21.15.60:0/4209427340 wait complete. 2022-06-29T15:58:23.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.145+0000 7f7f46b1f700 1 Processor -- start 2022-06-29T15:58:23.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.145+0000 7f7f46b1f700 1 -- start start 2022-06-29T15:58:23.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.149+0000 7f7f46b1f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f40060b60 0x7f7f4013d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:23.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.149+0000 7f7f46b1f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f400adc80 0x7f7f400b4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:23.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.149+0000 7f7f46b1f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f400b61d0 0x7f7f4013b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:23.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.149+0000 7f7f46b1f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f40065960 con 0x7f7f400b61d0 2022-06-29T15:58:23.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.149+0000 7f7f46b1f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f40065c00 con 0x7f7f40060b60 2022-06-29T15:58:23.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.149+0000 7f7f46b1f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f400b4660 con 0x7f7f400adc80 2022-06-29T15:58:23.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.149+0000 7f7f450bc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f400b61d0 0x7f7f4013b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:23.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.149+0000 7f7f37fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f400adc80 0x7f7f400b4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:23.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.149+0000 7f7f37fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f400adc80 0x7f7f400b4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:57988/0 (socket says 172.21.15.60:57988) 2022-06-29T15:58:23.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.149+0000 7f7f450bc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f400b61d0 0x7f7f4013b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44896/0 (socket says 172.21.15.60:44896) 2022-06-29T15:58:23.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.149+0000 7f7f37fff700 1 -- 172.21.15.60:0/1344000751 learned_addr learned my addr 172.21.15.60:0/1344000751 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:23.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.149+0000 7f7f448bb700 1 --2- 172.21.15.60:0/1344000751 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f40060b60 0x7f7f4013d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:23.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.149+0000 7f7f37fff700 1 -- 172.21.15.60:0/1344000751 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f40060b60 msgr2=0x7f7f4013d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:23.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.149+0000 7f7f37fff700 1 --2- 172.21.15.60:0/1344000751 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f40060b60 0x7f7f4013d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:23.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.149+0000 7f7f37fff700 1 -- 172.21.15.60:0/1344000751 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f400b61d0 msgr2=0x7f7f4013b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:23.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.149+0000 7f7f37fff700 1 --2- 172.21.15.60:0/1344000751 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f400b61d0 0x7f7f4013b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:23.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.149+0000 7f7f37fff700 1 -- 172.21.15.60:0/1344000751 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f3000b180 con 0x7f7f400adc80 2022-06-29T15:58:23.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.149+0000 7f7f37fff700 1 --2- 172.21.15.60:0/1344000751 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f400adc80 0x7f7f400b4170 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f7f30002f70 tx=0x7f7f30014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:23.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.149+0000 7f7f35ffb700 1 -- 172.21.15.60:0/1344000751 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f30016070 con 0x7f7f400adc80 2022-06-29T15:58:23.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.149+0000 7f7f35ffb700 1 -- 172.21.15.60:0/1344000751 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f3000de40 con 0x7f7f400adc80 2022-06-29T15:58:23.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.149+0000 7f7f46b1f700 1 -- 172.21.15.60:0/1344000751 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f40141c10 con 0x7f7f400adc80 2022-06-29T15:58:23.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.149+0000 7f7f35ffb700 1 -- 172.21.15.60:0/1344000751 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f3001e640 con 0x7f7f400adc80 2022-06-29T15:58:23.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.149+0000 7f7f46b1f700 1 -- 172.21.15.60:0/1344000751 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f4013e880 con 0x7f7f400adc80 2022-06-29T15:58:23.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.149+0000 7f7f46b1f700 1 -- 172.21.15.60:0/1344000751 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f401421e0 con 0x7f7f400adc80 2022-06-29T15:58:23.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.153+0000 7f7f35ffb700 1 -- 172.21.15.60:0/1344000751 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7f3000f030 con 0x7f7f400adc80 2022-06-29T15:58:23.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.153+0000 7f7f35ffb700 1 --2- 172.21.15.60:0/1344000751 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7f20053a70 0x7f7f20055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:23.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.153+0000 7f7f35ffb700 1 -- 172.21.15.60:0/1344000751 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f7f3006ca50 con 0x7f7f400adc80 2022-06-29T15:58:23.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.153+0000 7f7f448bb700 1 --2- 172.21.15.60:0/1344000751 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7f20053a70 0x7f7f20055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:23.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.153+0000 7f7f448bb700 1 --2- 172.21.15.60:0/1344000751 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7f20053a70 0x7f7f20055f30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f7f3c00d000 tx=0x7f7f3c007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:23.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.161+0000 7f7f35ffb700 1 -- 172.21.15.60:0/1344000751 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7f401421e0 con 0x7f7f400adc80 2022-06-29T15:58:23.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:23.289+0000 7f7f46b1f700 1 -- 172.21.15.60:0/1344000751 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f7f401421e0 con 0x7f7f400adc80 2022-06-29T15:58:24.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.017+0000 7f7f35ffb700 1 -- 172.21.15.60:0/1344000751 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v20) v1 ==== 125+0+0 (secure 0 0 0) 0x7f7f401421e0 con 0x7f7f400adc80 2022-06-29T15:58:24.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.025+0000 7f7f46b1f700 1 -- 172.21.15.60:0/1344000751 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7f20053a70 msgr2=0x7f7f20055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:24.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.025+0000 7f7f46b1f700 1 --2- 172.21.15.60:0/1344000751 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7f20053a70 0x7f7f20055f30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f7f3c00d000 tx=0x7f7f3c007a00).stop 2022-06-29T15:58:24.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.025+0000 7f7f46b1f700 1 -- 172.21.15.60:0/1344000751 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f400adc80 msgr2=0x7f7f400b4170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:24.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.025+0000 7f7f46b1f700 1 --2- 172.21.15.60:0/1344000751 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f400adc80 0x7f7f400b4170 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f7f30002f70 tx=0x7f7f30014000).stop 2022-06-29T15:58:24.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.025+0000 7f7f46b1f700 1 -- 172.21.15.60:0/1344000751 shutdown_connections 2022-06-29T15:58:24.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.025+0000 7f7f46b1f700 1 --2- 172.21.15.60:0/1344000751 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7f20053a70 0x7f7f20055f30 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:24.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.025+0000 7f7f46b1f700 1 --2- 172.21.15.60:0/1344000751 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f400b61d0 0x7f7f4013b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:24.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.025+0000 7f7f46b1f700 1 --2- 172.21.15.60:0/1344000751 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f400adc80 0x7f7f400b4170 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:24.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.025+0000 7f7f46b1f700 1 --2- 172.21.15.60:0/1344000751 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f40060b60 0x7f7f4013d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:24.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.025+0000 7f7f46b1f700 1 -- 172.21.15.60:0/1344000751 >> 172.21.15.60:0/1344000751 conn(0x7f7f40063b50 msgr2=0x7f7f401431d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:24.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.025+0000 7f7f46b1f700 1 -- 172.21.15.60:0/1344000751 shutdown_connections 2022-06-29T15:58:24.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.025+0000 7f7f46b1f700 1 -- 172.21.15.60:0/1344000751 wait complete. 2022-06-29T15:58:24.033 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2022-06-29T15:58:24.042 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-06-29T15:58:24.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.145+0000 7f528e4be700 1 Processor -- start 2022-06-29T15:58:24.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.145+0000 7f528e4be700 1 -- start start 2022-06-29T15:58:24.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.149+0000 7f528e4be700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f52880b2340 0x7f52880b3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:24.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.149+0000 7f528e4be700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52880ba150 con 0x7f52880b9d70 2022-06-29T15:58:24.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.149+0000 7f528e4be700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52880b4a10 con 0x7f52880b3cc0 2022-06-29T15:58:24.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.149+0000 7f528e4be700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52880b4f30 con 0x7f52880b2340 2022-06-29T15:58:24.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.149+0000 7f5287fff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f52880b9d70 0x7f52880b1c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52986/0 (socket says 172.21.15.60:52986) 2022-06-29T15:58:24.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.149+0000 7f52877fe700 1 --2- 172.21.15.60:0/1920532216 >> v2:172.21.15.60:3300/0 conn(0x7f52880b2340 0x7f52880b3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:24.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.149+0000 7f5287fff700 1 -- 172.21.15.60:0/1920532216 learned_addr learned my addr 172.21.15.60:0/1920532216 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:24.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.149+0000 7f5286ffd700 1 -- 172.21.15.60:0/1920532216 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1309979764 0 0) 0x7f52880b4a10 con 0x7f52880b3cc0 2022-06-29T15:58:24.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.149+0000 7f52877fe700 1 -- 172.21.15.60:0/1920532216 >> v1:172.21.15.92:6789/0 conn(0x7f52880b3cc0 legacy=0x7f5288142190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:24.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.149+0000 7f52877fe700 1 -- 172.21.15.60:0/1920532216 >> v1:172.21.15.60:6789/0 conn(0x7f52880b9d70 legacy=0x7f52880b1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:24.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.149+0000 7f52877fe700 1 -- 172.21.15.60:0/1920532216 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52880b2dc0 con 0x7f52880b2340 2022-06-29T15:58:24.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.149+0000 7f52877fe700 1 --2- 172.21.15.60:0/1920532216 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52880b2340 0x7f52880b3260 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f5278001110 tx=0x7f527c006df0).ready entity=mon.0 client_cookie=63a8ae569ffa56f9 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:24.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.149+0000 7f5286ffd700 1 -- 172.21.15.60:0/1920532216 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f527c00d5c0 con 0x7f52880b2340 2022-06-29T15:58:24.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.149+0000 7f5286ffd700 1 -- 172.21.15.60:0/1920532216 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f527c000e90 con 0x7f52880b2340 2022-06-29T15:58:24.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.149+0000 7f5286ffd700 1 -- 172.21.15.60:0/1920532216 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f527c00d910 con 0x7f52880b2340 2022-06-29T15:58:24.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.149+0000 7f528e4be700 1 -- 172.21.15.60:0/1920532216 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52880b2340 msgr2=0x7f52880b3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:24.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.149+0000 7f528e4be700 1 --2- 172.21.15.60:0/1920532216 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52880b2340 0x7f52880b3260 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f5278001110 tx=0x7f527c006df0).stop 2022-06-29T15:58:24.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.149+0000 7f528e4be700 1 -- 172.21.15.60:0/1920532216 shutdown_connections 2022-06-29T15:58:24.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.149+0000 7f528e4be700 1 --2- 172.21.15.60:0/1920532216 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52880b2340 0x7f52880b3260 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:24.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.149+0000 7f528e4be700 1 -- 172.21.15.60:0/1920532216 >> 172.21.15.60:0/1920532216 conn(0x7f5288062cd0 msgr2=0x7f52880b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:24.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.149+0000 7f528e4be700 1 -- 172.21.15.60:0/1920532216 shutdown_connections 2022-06-29T15:58:24.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.149+0000 7f528e4be700 1 -- 172.21.15.60:0/1920532216 wait complete. 2022-06-29T15:58:24.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.153+0000 7f528e4be700 1 Processor -- start 2022-06-29T15:58:24.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.153+0000 7f528e4be700 1 -- start start 2022-06-29T15:58:24.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.153+0000 7f528e4be700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52880b2340 0x7f52880addb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:24.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.153+0000 7f528e4be700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52880b3cc0 0x7f5288141130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:24.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.153+0000 7f528e4be700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f52880b9d70 0x7f528813ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:24.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.153+0000 7f528e4be700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52880af720 con 0x7f52880b2340 2022-06-29T15:58:24.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.153+0000 7f528e4be700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52881417b0 con 0x7f52880b3cc0 2022-06-29T15:58:24.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.153+0000 7f528e4be700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5288141a50 con 0x7f52880b9d70 2022-06-29T15:58:24.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.153+0000 7f5287fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52880b2340 0x7f52880addb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:24.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.153+0000 7f5287fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52880b2340 0x7f52880addb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44906/0 (socket says 172.21.15.60:44906) 2022-06-29T15:58:24.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.153+0000 7f5287fff700 1 -- 172.21.15.60:0/2617679168 learned_addr learned my addr 172.21.15.60:0/2617679168 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:24.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.153+0000 7f528ca5b700 1 --2- 172.21.15.60:0/2617679168 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f52880b9d70 0x7f528813ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:24.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.153+0000 7f5287fff700 1 -- 172.21.15.60:0/2617679168 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f52880b9d70 msgr2=0x7f528813ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:24.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.153+0000 7f52877fe700 1 --2- 172.21.15.60:0/2617679168 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52880b3cc0 0x7f5288141130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:24.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.153+0000 7f5287fff700 1 --2- 172.21.15.60:0/2617679168 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f52880b9d70 0x7f528813ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:24.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.153+0000 7f5287fff700 1 -- 172.21.15.60:0/2617679168 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52880b3cc0 msgr2=0x7f5288141130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:24.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.153+0000 7f5287fff700 1 --2- 172.21.15.60:0/2617679168 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52880b3cc0 0x7f5288141130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:24.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.153+0000 7f5287fff700 1 -- 172.21.15.60:0/2617679168 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f527c00b180 con 0x7f52880b2340 2022-06-29T15:58:24.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.153+0000 7f528ca5b700 1 --2- 172.21.15.60:0/2617679168 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f52880b9d70 0x7f528813ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T15:58:24.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.153+0000 7f52877fe700 1 --2- 172.21.15.60:0/2617679168 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52880b3cc0 0x7f5288141130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T15:58:24.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.153+0000 7f5287fff700 1 --2- 172.21.15.60:0/2617679168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52880b2340 0x7f52880addb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f5278000bc0 tx=0x7f5278007bf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:24.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.157+0000 7f52857fa700 1 -- 172.21.15.60:0/2617679168 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f52780072e0 con 0x7f52880b2340 2022-06-29T15:58:24.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.157+0000 7f52857fa700 1 -- 172.21.15.60:0/2617679168 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5278007440 con 0x7f52880b2340 2022-06-29T15:58:24.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.157+0000 7f528e4be700 1 -- 172.21.15.60:0/2617679168 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f528813f1d0 con 0x7f52880b2340 2022-06-29T15:58:24.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.157+0000 7f52857fa700 1 -- 172.21.15.60:0/2617679168 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5278005800 con 0x7f52880b2340 2022-06-29T15:58:24.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.157+0000 7f528e4be700 1 -- 172.21.15.60:0/2617679168 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52880aedd0 con 0x7f52880b2340 2022-06-29T15:58:24.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.157+0000 7f528e4be700 1 -- 172.21.15.60:0/2617679168 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52880573d0 con 0x7f52880b2340 2022-06-29T15:58:24.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.157+0000 7f52857fa700 1 -- 172.21.15.60:0/2617679168 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f527801a060 con 0x7f52880b2340 2022-06-29T15:58:24.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.157+0000 7f52857fa700 1 --2- 172.21.15.60:0/2617679168 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5268053a10 0x7f5268055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:24.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.157+0000 7f52857fa700 1 -- 172.21.15.60:0/2617679168 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f527806b770 con 0x7f52880b2340 2022-06-29T15:58:24.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.157+0000 7f52877fe700 1 --2- 172.21.15.60:0/2617679168 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5268053a10 0x7f5268055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:24.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.161+0000 7f52877fe700 1 --2- 172.21.15.60:0/2617679168 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5268053a10 0x7f5268055ed0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f527c015000 tx=0x7f527c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:24.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.169+0000 7f52857fa700 1 -- 172.21.15.60:0/2617679168 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5278042700 con 0x7f52880b2340 2022-06-29T15:58:24.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.329+0000 7f528e4be700 1 -- 172.21.15.60:0/2617679168 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f528805d990 con 0x7f52880b2340 2022-06-29T15:58:24.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.341+0000 7f52857fa700 1 -- 172.21.15.60:0/2617679168 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 3 and data pool 4 v3) v1 ==== 187+0+0 (secure 0 0 0) 0x7f5278018020 con 0x7f52880b2340 2022-06-29T15:58:24.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.345+0000 7f528e4be700 1 -- 172.21.15.60:0/2617679168 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5268053a10 msgr2=0x7f5268055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:24.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.345+0000 7f528e4be700 1 --2- 172.21.15.60:0/2617679168 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5268053a10 0x7f5268055ed0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f527c015000 tx=0x7f527c014000).stop 2022-06-29T15:58:24.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.345+0000 7f528e4be700 1 -- 172.21.15.60:0/2617679168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52880b2340 msgr2=0x7f52880addb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:24.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.345+0000 7f528e4be700 1 --2- 172.21.15.60:0/2617679168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52880b2340 0x7f52880addb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f5278000bc0 tx=0x7f5278007bf0).stop 2022-06-29T15:58:24.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.353+0000 7f528e4be700 1 -- 172.21.15.60:0/2617679168 shutdown_connections 2022-06-29T15:58:24.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.353+0000 7f528e4be700 1 --2- 172.21.15.60:0/2617679168 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5268053a10 0x7f5268055ed0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:24.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.353+0000 7f528e4be700 1 --2- 172.21.15.60:0/2617679168 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f52880b9d70 0x7f528813ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:24.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.353+0000 7f528e4be700 1 --2- 172.21.15.60:0/2617679168 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52880b3cc0 0x7f5288141130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:24.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.353+0000 7f528e4be700 1 --2- 172.21.15.60:0/2617679168 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52880b2340 0x7f52880addb0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:24.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.353+0000 7f528e4be700 1 -- 172.21.15.60:0/2617679168 >> 172.21.15.60:0/2617679168 conn(0x7f5288062cd0 msgr2=0x7f52880b50f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:24.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.353+0000 7f528e4be700 1 -- 172.21.15.60:0/2617679168 shutdown_connections 2022-06-29T15:58:24.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.353+0000 7f528e4be700 1 -- 172.21.15.60:0/2617679168 wait complete. 2022-06-29T15:58:24.360 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 3 and data pool 4 2022-06-29T15:58:24.369 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T15:58:24.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.457+0000 7f37a9007700 1 Processor -- start 2022-06-29T15:58:24.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.457+0000 7f37a9007700 1 -- start start 2022-06-29T15:58:24.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.457+0000 7f37a9007700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f37a40b2130 0x7f37a4146100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:24.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.457+0000 7f37a9007700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f37a40aed00 0x7f37a41465f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:24.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.457+0000 7f37a9007700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37a405d450 con 0x7f37a405d070 2022-06-29T15:58:24.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.457+0000 7f37a9007700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37a40adf90 con 0x7f37a40aed00 2022-06-29T15:58:24.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.457+0000 7f37a9007700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37a40b25d0 con 0x7f37a40b2130 2022-06-29T15:58:24.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.457+0000 7f37a8806700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f37a40aed00 0x7f37a41465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:24.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.457+0000 7f37a8806700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f37a40aed00 0x7f37a41465f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44926/0 (socket says 172.21.15.60:44926) 2022-06-29T15:58:24.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.457+0000 7f37a8806700 1 -- 172.21.15.60:0/1464301410 learned_addr learned my addr 172.21.15.60:0/1464301410 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:24.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.461+0000 7f37a37fe700 1 --2- 172.21.15.60:0/1464301410 >> v2:172.21.15.92:3300/0 conn(0x7f37a40b2130 0x7f37a4146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:24.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.461+0000 7f37a8806700 1 -- 172.21.15.60:0/1464301410 >> v2:172.21.15.92:3300/0 conn(0x7f37a40b2130 msgr2=0x7f37a4146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:24.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.461+0000 7f37a8806700 1 --2- 172.21.15.60:0/1464301410 >> v2:172.21.15.92:3300/0 conn(0x7f37a40b2130 0x7f37a4146100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:24.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.461+0000 7f37a8806700 1 -- 172.21.15.60:0/1464301410 >> v1:172.21.15.92:6789/0 conn(0x7f37a405d070 legacy=0x7f37a414aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:24.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.461+0000 7f37a8806700 1 -- 172.21.15.60:0/1464301410 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a414a350 con 0x7f37a40aed00 2022-06-29T15:58:24.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.461+0000 7f37a8806700 1 --2- 172.21.15.60:0/1464301410 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37a40aed00 0x7f37a41465f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f3794002f70 tx=0x7f3794006eb0).ready entity=mon.0 client_cookie=97d9dc9360dba471 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:24.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.461+0000 7f37a2ffd700 1 -- 172.21.15.60:0/1464301410 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f379400d5c0 con 0x7f37a40aed00 2022-06-29T15:58:24.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.461+0000 7f37a2ffd700 1 -- 172.21.15.60:0/1464301410 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3794000e90 con 0x7f37a40aed00 2022-06-29T15:58:24.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.461+0000 7f37a2ffd700 1 -- 172.21.15.60:0/1464301410 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f379400d950 con 0x7f37a40aed00 2022-06-29T15:58:24.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.461+0000 7f37a9007700 1 -- 172.21.15.60:0/1464301410 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37a40aed00 msgr2=0x7f37a41465f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:24.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.461+0000 7f37a9007700 1 --2- 172.21.15.60:0/1464301410 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37a40aed00 0x7f37a41465f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f3794002f70 tx=0x7f3794006eb0).stop 2022-06-29T15:58:24.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.461+0000 7f37a9007700 1 -- 172.21.15.60:0/1464301410 shutdown_connections 2022-06-29T15:58:24.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.461+0000 7f37a9007700 1 --2- 172.21.15.60:0/1464301410 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37a40aed00 0x7f37a41465f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:24.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.461+0000 7f37a9007700 1 --2- 172.21.15.60:0/1464301410 >> v2:172.21.15.92:3300/0 conn(0x7f37a40b2130 0x7f37a4146100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:24.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.461+0000 7f37a9007700 1 -- 172.21.15.60:0/1464301410 >> 172.21.15.60:0/1464301410 conn(0x7f37a40b5890 msgr2=0x7f37a40b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:24.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.461+0000 7f37a9007700 1 -- 172.21.15.60:0/1464301410 shutdown_connections 2022-06-29T15:58:24.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.461+0000 7f37a9007700 1 -- 172.21.15.60:0/1464301410 wait complete. 2022-06-29T15:58:24.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.461+0000 7f37a9007700 1 Processor -- start 2022-06-29T15:58:24.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.461+0000 7f37a9007700 1 -- start start 2022-06-29T15:58:24.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.465+0000 7f37a9007700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37a405d070 0x7f37a4116c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:24.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.465+0000 7f37a9007700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37a40aed00 0x7f37a41424a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:24.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.465+0000 7f37a9007700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37a40b2130 0x7f37a4142990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:24.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.465+0000 7f37a9007700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a4142e80 con 0x7f37a405d070 2022-06-29T15:58:24.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.465+0000 7f37a9007700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a4065800 con 0x7f37a40aed00 2022-06-29T15:58:24.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.465+0000 7f37a9007700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37a4066e10 con 0x7f37a40b2130 2022-06-29T15:58:24.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.465+0000 7f37a3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37a405d070 0x7f37a4116c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:24.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.465+0000 7f37a3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37a405d070 0x7f37a4116c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44930/0 (socket says 172.21.15.60:44930) 2022-06-29T15:58:24.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.465+0000 7f37a3fff700 1 -- 172.21.15.60:0/2394910153 learned_addr learned my addr 172.21.15.60:0/2394910153 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:24.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.465+0000 7f37a8806700 1 --2- 172.21.15.60:0/2394910153 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37a40b2130 0x7f37a4142990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:24.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.465+0000 7f37a37fe700 1 --2- 172.21.15.60:0/2394910153 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37a40aed00 0x7f37a41424a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:24.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.465+0000 7f37a3fff700 1 -- 172.21.15.60:0/2394910153 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37a40b2130 msgr2=0x7f37a4142990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:24.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.465+0000 7f37a3fff700 1 --2- 172.21.15.60:0/2394910153 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37a40b2130 0x7f37a4142990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:24.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.465+0000 7f37a3fff700 1 -- 172.21.15.60:0/2394910153 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37a40aed00 msgr2=0x7f37a41424a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:24.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.465+0000 7f37a3fff700 1 --2- 172.21.15.60:0/2394910153 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37a40aed00 0x7f37a41424a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:24.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.465+0000 7f37a3fff700 1 -- 172.21.15.60:0/2394910153 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f379400b1e0 con 0x7f37a405d070 2022-06-29T15:58:24.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.465+0000 7f37a3fff700 1 --2- 172.21.15.60:0/2394910153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37a405d070 0x7f37a4116c50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f37a4119280 tx=0x7f379c001400).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:24.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.465+0000 7f37a17fa700 1 -- 172.21.15.60:0/2394910153 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f379c00e4a0 con 0x7f37a405d070 2022-06-29T15:58:24.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.465+0000 7f37a17fa700 1 -- 172.21.15.60:0/2394910153 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f379c00e600 con 0x7f37a405d070 2022-06-29T15:58:24.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.465+0000 7f37a9007700 1 -- 172.21.15.60:0/2394910153 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37a4065d90 con 0x7f37a405d070 2022-06-29T15:58:24.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.465+0000 7f37a17fa700 1 -- 172.21.15.60:0/2394910153 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f379c00e960 con 0x7f37a405d070 2022-06-29T15:58:24.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.465+0000 7f37a9007700 1 -- 172.21.15.60:0/2394910153 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37a4119560 con 0x7f37a405d070 2022-06-29T15:58:24.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.465+0000 7f37a9007700 1 -- 172.21.15.60:0/2394910153 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37a40659c0 con 0x7f37a405d070 2022-06-29T15:58:24.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.469+0000 7f37a17fa700 1 -- 172.21.15.60:0/2394910153 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f379c019030 con 0x7f37a405d070 2022-06-29T15:58:24.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.469+0000 7f37a17fa700 1 --2- 172.21.15.60:0/2394910153 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3788053a10 0x7f3788055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:24.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.469+0000 7f37a17fa700 1 -- 172.21.15.60:0/2394910153 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f379c06b580 con 0x7f37a405d070 2022-06-29T15:58:24.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.469+0000 7f37a37fe700 1 --2- 172.21.15.60:0/2394910153 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3788053a10 0x7f3788055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:24.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.469+0000 7f37a37fe700 1 --2- 172.21.15.60:0/2394910153 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3788053a10 0x7f3788055ed0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f378c00b000 tx=0x7f378c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:24.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.473+0000 7f37a17fa700 1 -- 172.21.15.60:0/2394910153 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f379c0424a0 con 0x7f37a405d070 2022-06-29T15:58:24.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.593+0000 7f37a9007700 1 -- 172.21.15.60:0/2394910153 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37a414a5f0 con 0x7f37a405d070 2022-06-29T15:58:24.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.597+0000 7f37a17fa700 1 -- 172.21.15.60:0/2394910153 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16051 (secure 0 0 0) 0x7f379c045ac0 con 0x7f37a405d070 2022-06-29T15:58:24.603 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:58:24.603 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":21,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T15:58:24.334438+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-06-29T15:58:22.424395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-06-29T15:58:23.343438+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T15:58:24.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.597+0000 7f37a9007700 1 -- 172.21.15.60:0/2394910153 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3788053a10 msgr2=0x7f3788055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:24.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.597+0000 7f37a9007700 1 --2- 172.21.15.60:0/2394910153 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3788053a10 0x7f3788055ed0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f378c00b000 tx=0x7f378c006df0).stop 2022-06-29T15:58:24.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.597+0000 7f37a9007700 1 -- 172.21.15.60:0/2394910153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37a405d070 msgr2=0x7f37a4116c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:24.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.597+0000 7f37a9007700 1 --2- 172.21.15.60:0/2394910153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37a405d070 0x7f37a4116c50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f37a4119280 tx=0x7f379c001400).stop 2022-06-29T15:58:24.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.601+0000 7f37a9007700 1 -- 172.21.15.60:0/2394910153 shutdown_connections 2022-06-29T15:58:24.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.601+0000 7f37a9007700 1 --2- 172.21.15.60:0/2394910153 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3788053a10 0x7f3788055ed0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:24.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.601+0000 7f37a9007700 1 --2- 172.21.15.60:0/2394910153 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f37a40b2130 0x7f37a4142990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:24.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.601+0000 7f37a9007700 1 --2- 172.21.15.60:0/2394910153 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37a40aed00 0x7f37a41424a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:24.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.601+0000 7f37a9007700 1 --2- 172.21.15.60:0/2394910153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37a405d070 0x7f37a4116c50 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:24.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.601+0000 7f37a9007700 1 -- 172.21.15.60:0/2394910153 >> 172.21.15.60:0/2394910153 conn(0x7f37a40b5890 msgr2=0x7f37a40b44c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:24.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.601+0000 7f37a9007700 1 -- 172.21.15.60:0/2394910153 shutdown_connections 2022-06-29T15:58:24.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.601+0000 7f37a9007700 1 -- 172.21.15.60:0/2394910153 wait complete. 2022-06-29T15:58:24.620 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T15:58:24.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.713+0000 7fcb91da9700 1 Processor -- start 2022-06-29T15:58:24.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.713+0000 7fcb91da9700 1 -- start start 2022-06-29T15:58:24.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.713+0000 7fcb91da9700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb8c05d1f0 0x7fcb8c0b1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:24.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.713+0000 7fcb91da9700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcb8c0b3cc0 0x7fcb8c0b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:24.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.713+0000 7fcb91da9700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb8c0b2ce0 con 0x7fcb8c0b9d70 2022-06-29T15:58:24.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.713+0000 7fcb91da9700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb8c0b4d10 con 0x7fcb8c0b3cc0 2022-06-29T15:58:24.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.713+0000 7fcb91da9700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c0ba150 con 0x7fcb8c05d1f0 2022-06-29T15:58:24.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.713+0000 7fcb8affd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb8c05d1f0 0x7fcb8c0b1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:24.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.713+0000 7fcb8bfff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcb8c0b3cc0 0x7fcb8c0b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:24.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.717+0000 7fcb8affd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb8c05d1f0 0x7fcb8c0b1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:58034/0 (socket says 172.21.15.60:58034) 2022-06-29T15:58:24.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.717+0000 7fcb8affd700 1 -- 172.21.15.60:0/1655398022 learned_addr learned my addr 172.21.15.60:0/1655398022 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:24.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.717+0000 7fcb8b7fe700 1 --1- 172.21.15.60:0/1655398022 >> v1:172.21.15.92:6789/0 conn(0x7fcb8c0b9d70 0x7fcb8c0b3140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:57272/0 (socket says 172.21.15.60:57272) 2022-06-29T15:58:24.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.717+0000 7fcb8affd700 1 -- 172.21.15.60:0/1655398022 >> v2:172.21.15.92:3300/0 conn(0x7fcb8c0b3cc0 msgr2=0x7fcb8c0b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:24.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.717+0000 7fcb8affd700 1 --2- 172.21.15.60:0/1655398022 >> v2:172.21.15.92:3300/0 conn(0x7fcb8c0b3cc0 0x7fcb8c0b2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:24.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.717+0000 7fcb8affd700 1 -- 172.21.15.60:0/1655398022 >> v1:172.21.15.92:6789/0 conn(0x7fcb8c0b9d70 legacy=0x7fcb8c0b3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:24.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.717+0000 7fcb8affd700 1 -- 172.21.15.60:0/1655398022 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb8c0b4a70 con 0x7fcb8c05d1f0 2022-06-29T15:58:24.728 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.717+0000 7fcb8affd700 1 --2- 172.21.15.60:0/1655398022 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb8c05d1f0 0x7fcb8c0b1b50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fcb7c002f70 tx=0x7fcb7c006eb0).ready entity=mon.2 client_cookie=96d89654ef8b59d8 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:24.728 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.717+0000 7fcb8a7fc700 1 -- 172.21.15.60:0/1655398022 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb7c00d570 con 0x7fcb8c05d1f0 2022-06-29T15:58:24.728 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.717+0000 7fcb8a7fc700 1 -- 172.21.15.60:0/1655398022 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb7c000e90 con 0x7fcb8c05d1f0 2022-06-29T15:58:24.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.717+0000 7fcb8a7fc700 1 -- 172.21.15.60:0/1655398022 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb7c00d900 con 0x7fcb8c05d1f0 2022-06-29T15:58:24.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.717+0000 7fcb91da9700 1 -- 172.21.15.60:0/1655398022 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb8c05d1f0 msgr2=0x7fcb8c0b1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:24.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.717+0000 7fcb91da9700 1 --2- 172.21.15.60:0/1655398022 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb8c05d1f0 0x7fcb8c0b1b50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fcb7c002f70 tx=0x7fcb7c006eb0).stop 2022-06-29T15:58:24.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.717+0000 7fcb91da9700 1 -- 172.21.15.60:0/1655398022 shutdown_connections 2022-06-29T15:58:24.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.717+0000 7fcb91da9700 1 --2- 172.21.15.60:0/1655398022 >> v2:172.21.15.92:3300/0 conn(0x7fcb8c0b3cc0 0x7fcb8c0b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:24.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.717+0000 7fcb91da9700 1 --2- 172.21.15.60:0/1655398022 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb8c05d1f0 0x7fcb8c0b1b50 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:24.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.717+0000 7fcb91da9700 1 -- 172.21.15.60:0/1655398022 >> 172.21.15.60:0/1655398022 conn(0x7fcb8c062ce0 msgr2=0x7fcb8c0b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:24.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.717+0000 7fcb91da9700 1 -- 172.21.15.60:0/1655398022 shutdown_connections 2022-06-29T15:58:24.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.717+0000 7fcb91da9700 1 -- 172.21.15.60:0/1655398022 wait complete. 2022-06-29T15:58:24.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.717+0000 7fcb91da9700 1 Processor -- start 2022-06-29T15:58:24.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.717+0000 7fcb91da9700 1 -- start start 2022-06-29T15:58:24.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.721+0000 7fcb91da9700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb8c05d1f0 0x7fcb8c152a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:24.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.721+0000 7fcb91da9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb8c0b3cc0 0x7fcb8c1460b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:24.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.721+0000 7fcb91da9700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb8c0b9d70 0x7fcb8c0aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:24.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.721+0000 7fcb91da9700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c0ad770 con 0x7fcb8c0b3cc0 2022-06-29T15:58:24.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.721+0000 7fcb91da9700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c146740 con 0x7fcb8c05d1f0 2022-06-29T15:58:24.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.721+0000 7fcb91da9700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c0af840 con 0x7fcb8c0b9d70 2022-06-29T15:58:24.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.721+0000 7fcb8affd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb8c0b3cc0 0x7fcb8c1460b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:24.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.721+0000 7fcb8b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb8c05d1f0 0x7fcb8c152a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:24.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.721+0000 7fcb8b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb8c05d1f0 0x7fcb8c152a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:50974/0 (socket says 172.21.15.60:50974) 2022-06-29T15:58:24.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.721+0000 7fcb8affd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb8c0b3cc0 0x7fcb8c1460b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44946/0 (socket says 172.21.15.60:44946) 2022-06-29T15:58:24.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.721+0000 7fcb8b7fe700 1 -- 172.21.15.60:0/15034034 learned_addr learned my addr 172.21.15.60:0/15034034 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:24.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.721+0000 7fcb8bfff700 1 --2- 172.21.15.60:0/15034034 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb8c0b9d70 0x7fcb8c0aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:24.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.721+0000 7fcb8b7fe700 1 -- 172.21.15.60:0/15034034 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb8c0b9d70 msgr2=0x7fcb8c0aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:24.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.721+0000 7fcb8b7fe700 1 --2- 172.21.15.60:0/15034034 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb8c0b9d70 0x7fcb8c0aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:24.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.721+0000 7fcb8b7fe700 1 -- 172.21.15.60:0/15034034 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb8c0b3cc0 msgr2=0x7fcb8c1460b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:24.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.721+0000 7fcb8b7fe700 1 --2- 172.21.15.60:0/15034034 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb8c0b3cc0 0x7fcb8c1460b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:24.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.721+0000 7fcb8b7fe700 1 -- 172.21.15.60:0/15034034 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb7c00b1e0 con 0x7fcb8c05d1f0 2022-06-29T15:58:24.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.721+0000 7fcb8affd700 1 --2- 172.21.15.60:0/15034034 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb8c0b3cc0 0x7fcb8c1460b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:24.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.721+0000 7fcb8b7fe700 1 --2- 172.21.15.60:0/15034034 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb8c05d1f0 0x7fcb8c152a60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fcb8c0ae0d0 tx=0x7fcb80007c40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:24.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.721+0000 7fcb88ff9700 1 -- 172.21.15.60:0/15034034 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb800070d0 con 0x7fcb8c05d1f0 2022-06-29T15:58:24.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.721+0000 7fcb88ff9700 1 -- 172.21.15.60:0/15034034 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb80007230 con 0x7fcb8c05d1f0 2022-06-29T15:58:24.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.721+0000 7fcb91da9700 1 -- 172.21.15.60:0/15034034 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb8c0acf10 con 0x7fcb8c05d1f0 2022-06-29T15:58:24.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.721+0000 7fcb88ff9700 1 -- 172.21.15.60:0/15034034 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb80005550 con 0x7fcb8c05d1f0 2022-06-29T15:58:24.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.721+0000 7fcb91da9700 1 -- 172.21.15.60:0/15034034 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb8c1469d0 con 0x7fcb8c05d1f0 2022-06-29T15:58:24.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.721+0000 7fcb91da9700 1 -- 172.21.15.60:0/15034034 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb8c145370 con 0x7fcb8c05d1f0 2022-06-29T15:58:24.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.725+0000 7fcb88ff9700 1 -- 172.21.15.60:0/15034034 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcb80010070 con 0x7fcb8c05d1f0 2022-06-29T15:58:24.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.725+0000 7fcb88ff9700 1 --2- 172.21.15.60:0/15034034 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcb6c053a70 0x7fcb6c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:24.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.725+0000 7fcb88ff9700 1 -- 172.21.15.60:0/15034034 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcb8006bec0 con 0x7fcb8c05d1f0 2022-06-29T15:58:24.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.725+0000 7fcb8affd700 1 --2- 172.21.15.60:0/15034034 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcb6c053a70 0x7fcb6c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:24.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.725+0000 7fcb8affd700 1 --2- 172.21.15.60:0/15034034 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcb6c053a70 0x7fcb6c055f30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fcb7c01c000 tx=0x7fcb7c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:24.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.733+0000 7fcb88ff9700 1 -- 172.21.15.60:0/15034034 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcb80042cd0 con 0x7fcb8c05d1f0 2022-06-29T15:58:24.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.861+0000 7fcb91da9700 1 -- 172.21.15.60:0/15034034 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb8c0afa00 con 0x7fcb8c05d1f0 2022-06-29T15:58:24.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.865+0000 7fcb88ff9700 1 -- 172.21.15.60:0/15034034 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16051 (secure 0 0 0) 0x7fcb800462f0 con 0x7fcb8c05d1f0 2022-06-29T15:58:24.871 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:58:24.872 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":21,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T15:58:24.334438+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-06-29T15:58:22.424395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-06-29T15:58:23.343438+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T15:58:24.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.865+0000 7fcb91da9700 1 -- 172.21.15.60:0/15034034 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcb6c053a70 msgr2=0x7fcb6c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:24.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.865+0000 7fcb91da9700 1 --2- 172.21.15.60:0/15034034 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcb6c053a70 0x7fcb6c055f30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fcb7c01c000 tx=0x7fcb7c015000).stop 2022-06-29T15:58:24.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.865+0000 7fcb91da9700 1 -- 172.21.15.60:0/15034034 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb8c05d1f0 msgr2=0x7fcb8c152a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:24.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.865+0000 7fcb91da9700 1 --2- 172.21.15.60:0/15034034 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb8c05d1f0 0x7fcb8c152a60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fcb8c0ae0d0 tx=0x7fcb80007c40).stop 2022-06-29T15:58:24.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.865+0000 7fcb91da9700 1 -- 172.21.15.60:0/15034034 shutdown_connections 2022-06-29T15:58:24.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.865+0000 7fcb91da9700 1 --2- 172.21.15.60:0/15034034 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcb6c053a70 0x7fcb6c055f30 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:24.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.865+0000 7fcb91da9700 1 --2- 172.21.15.60:0/15034034 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb8c0b9d70 0x7fcb8c0aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:24.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.865+0000 7fcb91da9700 1 --2- 172.21.15.60:0/15034034 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb8c0b3cc0 0x7fcb8c1460b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:24.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.865+0000 7fcb91da9700 1 --2- 172.21.15.60:0/15034034 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb8c05d1f0 0x7fcb8c152a60 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:24.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.865+0000 7fcb91da9700 1 -- 172.21.15.60:0/15034034 >> 172.21.15.60:0/15034034 conn(0x7fcb8c062ce0 msgr2=0x7fcb8c0b1d10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:24.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.865+0000 7fcb91da9700 1 -- 172.21.15.60:0/15034034 shutdown_connections 2022-06-29T15:58:24.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.869+0000 7fcb91da9700 1 -- 172.21.15.60:0/15034034 wait complete. 2022-06-29T15:58:24.885 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-06-29T15:58:24.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.985+0000 7fe58e710700 1 Processor -- start 2022-06-29T15:58:24.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.985+0000 7fe58e710700 1 -- start start 2022-06-29T15:58:24.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.985+0000 7fe58e710700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe5880b2120 0x7fe5880ab880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:24.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.985+0000 7fe58e710700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe58814b380 con 0x7fe58805cf40 2022-06-29T15:58:24.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.985+0000 7fe58e710700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe58805cb40 con 0x7fe58814af10 2022-06-29T15:58:24.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.985+0000 7fe58e710700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5880b1720 con 0x7fe5880b2120 2022-06-29T15:58:24.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.989+0000 7fe58d70e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe5880b2120 0x7fe5880ab880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:24.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.989+0000 7fe58cf0d700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe58805cf40 0x7fe58805bd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53040/0 (socket says 172.21.15.60:53040) 2022-06-29T15:58:24.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.989+0000 7fe58cf0d700 1 -- 172.21.15.60:0/3934480338 learned_addr learned my addr 172.21.15.60:0/3934480338 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:24.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.989+0000 7fe58d70e700 1 -- 172.21.15.60:0/3934480338 >> v1:172.21.15.92:6789/0 conn(0x7fe58814af10 legacy=0x7fe58805c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:24.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.989+0000 7fe58d70e700 1 -- 172.21.15.60:0/3934480338 >> v1:172.21.15.60:6789/0 conn(0x7fe58805cf40 legacy=0x7fe58805bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:24.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.989+0000 7fe57ffff700 1 -- 172.21.15.60:0/3934480338 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1348698564 0 0) 0x7fe58814b380 con 0x7fe58805cf40 2022-06-29T15:58:24.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.989+0000 7fe58d70e700 1 -- 172.21.15.60:0/3934480338 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe58814a530 con 0x7fe5880b2120 2022-06-29T15:58:24.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.989+0000 7fe58d70e700 1 --2- 172.21.15.60:0/3934480338 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5880b2120 0x7fe5880ab880 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fe5800032f0 tx=0x7fe5800091d0).ready entity=mon.0 client_cookie=707d889fc13519e2 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:24.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.989+0000 7fe57ffff700 1 -- 172.21.15.60:0/3934480338 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe580009ea0 con 0x7fe5880b2120 2022-06-29T15:58:24.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.989+0000 7fe57ffff700 1 -- 172.21.15.60:0/3934480338 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe58000b8a0 con 0x7fe5880b2120 2022-06-29T15:58:24.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.989+0000 7fe57ffff700 1 -- 172.21.15.60:0/3934480338 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe58000ba10 con 0x7fe5880b2120 2022-06-29T15:58:24.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.989+0000 7fe58e710700 1 -- 172.21.15.60:0/3934480338 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5880b2120 msgr2=0x7fe5880ab880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:24.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.989+0000 7fe58e710700 1 --2- 172.21.15.60:0/3934480338 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5880b2120 0x7fe5880ab880 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fe5800032f0 tx=0x7fe5800091d0).stop 2022-06-29T15:58:25.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.989+0000 7fe58e710700 1 -- 172.21.15.60:0/3934480338 shutdown_connections 2022-06-29T15:58:25.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.989+0000 7fe58e710700 1 --2- 172.21.15.60:0/3934480338 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5880b2120 0x7fe5880ab880 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:25.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.989+0000 7fe58e710700 1 -- 172.21.15.60:0/3934480338 >> 172.21.15.60:0/3934480338 conn(0x7fe5880b58a0 msgr2=0x7fe5880b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:25.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.989+0000 7fe58e710700 1 -- 172.21.15.60:0/3934480338 shutdown_connections 2022-06-29T15:58:25.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.989+0000 7fe58e710700 1 -- 172.21.15.60:0/3934480338 wait complete. 2022-06-29T15:58:25.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.989+0000 7fe58e710700 1 Processor -- start 2022-06-29T15:58:25.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.989+0000 7fe58e710700 1 -- start start 2022-06-29T15:58:25.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.993+0000 7fe58e710700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe58805cf40 0x7fe5880afd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:25.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.993+0000 7fe58e710700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5880b2120 0x7fe5880b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:25.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.993+0000 7fe58e710700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe58814af10 0x7fe5880b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:25.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.993+0000 7fe58e710700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe588145bf0 con 0x7fe58814af10 2022-06-29T15:58:25.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.993+0000 7fe58e710700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe588099e50 con 0x7fe5880b2120 2022-06-29T15:58:25.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.993+0000 7fe58e710700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe588098380 con 0x7fe58805cf40 2022-06-29T15:58:25.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.993+0000 7fe58df0f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe58814af10 0x7fe5880b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:25.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.993+0000 7fe58df0f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe58814af10 0x7fe5880b30f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44962/0 (socket says 172.21.15.60:44962) 2022-06-29T15:58:25.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.993+0000 7fe58df0f700 1 -- 172.21.15.60:0/280103601 learned_addr learned my addr 172.21.15.60:0/280103601 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:25.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.993+0000 7fe58d70e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe58805cf40 0x7fe5880afd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:25.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.993+0000 7fe58cf0d700 1 --2- 172.21.15.60:0/280103601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5880b2120 0x7fe5880b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:25.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.993+0000 7fe58df0f700 1 -- 172.21.15.60:0/280103601 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe58805cf40 msgr2=0x7fe5880afd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:25.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.993+0000 7fe58df0f700 1 --2- 172.21.15.60:0/280103601 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe58805cf40 0x7fe5880afd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:25.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.993+0000 7fe58df0f700 1 -- 172.21.15.60:0/280103601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5880b2120 msgr2=0x7fe5880b2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:25.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.993+0000 7fe58df0f700 1 --2- 172.21.15.60:0/280103601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5880b2120 0x7fe5880b2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:25.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.993+0000 7fe58df0f700 1 -- 172.21.15.60:0/280103601 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe580009470 con 0x7fe58814af10 2022-06-29T15:58:25.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.993+0000 7fe58df0f700 1 --2- 172.21.15.60:0/280103601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe58814af10 0x7fe5880b30f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fe5880b4d20 tx=0x7fe584007c20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:25.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.993+0000 7fe57e7fc700 1 -- 172.21.15.60:0/280103601 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe584007030 con 0x7fe58814af10 2022-06-29T15:58:25.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.993+0000 7fe57e7fc700 1 -- 172.21.15.60:0/280103601 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe584007190 con 0x7fe58814af10 2022-06-29T15:58:25.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.993+0000 7fe58e710700 1 -- 172.21.15.60:0/280103601 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5880b3f80 con 0x7fe58814af10 2022-06-29T15:58:25.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.993+0000 7fe57e7fc700 1 -- 172.21.15.60:0/280103601 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe584006510 con 0x7fe58814af10 2022-06-29T15:58:25.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.993+0000 7fe58e710700 1 -- 172.21.15.60:0/280103601 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe588097ee0 con 0x7fe58814af10 2022-06-29T15:58:25.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.993+0000 7fe57e7fc700 1 -- 172.21.15.60:0/280103601 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe584010030 con 0x7fe58814af10 2022-06-29T15:58:25.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.993+0000 7fe57e7fc700 1 --2- 172.21.15.60:0/280103601 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe5700539e0 0x7fe570055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:25.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.993+0000 7fe58e710700 1 -- 172.21.15.60:0/280103601 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5880981f0 con 0x7fe58814af10 2022-06-29T15:58:25.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.993+0000 7fe57e7fc700 1 -- 172.21.15.60:0/280103601 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe58406ba90 con 0x7fe58814af10 2022-06-29T15:58:25.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.997+0000 7fe58d70e700 1 --2- 172.21.15.60:0/280103601 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe5700539e0 0x7fe570055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:25.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:24.997+0000 7fe58d70e700 1 --2- 172.21.15.60:0/280103601 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe5700539e0 0x7fe570055ea0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fe58001b000 tx=0x7fe580016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:25.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.001+0000 7fe57e7fc700 1 -- 172.21.15.60:0/280103601 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe584042910 con 0x7fe58814af10 2022-06-29T15:58:25.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.165+0000 7fe58e710700 1 -- 172.21.15.60:0/280103601 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe588098fa0 con 0x7fe58814af10 2022-06-29T15:58:25.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.349+0000 7fe57e7fc700 1 -- 172.21.15.60:0/280103601 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v4) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe588098fa0 con 0x7fe58814af10 2022-06-29T15:58:25.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.349+0000 7fe58e710700 1 -- 172.21.15.60:0/280103601 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe5700539e0 msgr2=0x7fe570055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:25.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.349+0000 7fe58e710700 1 --2- 172.21.15.60:0/280103601 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe5700539e0 0x7fe570055ea0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fe58001b000 tx=0x7fe580016000).stop 2022-06-29T15:58:25.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.349+0000 7fe58e710700 1 -- 172.21.15.60:0/280103601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe58814af10 msgr2=0x7fe5880b30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:25.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.349+0000 7fe58e710700 1 --2- 172.21.15.60:0/280103601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe58814af10 0x7fe5880b30f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fe5880b4d20 tx=0x7fe584007c20).stop 2022-06-29T15:58:25.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.349+0000 7fe58e710700 1 -- 172.21.15.60:0/280103601 shutdown_connections 2022-06-29T15:58:25.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.349+0000 7fe58e710700 1 --2- 172.21.15.60:0/280103601 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe5700539e0 0x7fe570055ea0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:25.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.349+0000 7fe58e710700 1 --2- 172.21.15.60:0/280103601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe58814af10 0x7fe5880b30f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:25.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.349+0000 7fe58e710700 1 --2- 172.21.15.60:0/280103601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5880b2120 0x7fe5880b2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:25.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.349+0000 7fe58e710700 1 --2- 172.21.15.60:0/280103601 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe58805cf40 0x7fe5880afd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:25.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.349+0000 7fe58e710700 1 -- 172.21.15.60:0/280103601 >> 172.21.15.60:0/280103601 conn(0x7fe5880b58a0 msgr2=0x7fe588147240 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:25.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.349+0000 7fe58e710700 1 -- 172.21.15.60:0/280103601 shutdown_connections 2022-06-29T15:58:25.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.349+0000 7fe58e710700 1 -- 172.21.15.60:0/280103601 wait complete. 2022-06-29T15:58:25.370 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T15:58:25.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.473+0000 7fc2adedd700 1 Processor -- start 2022-06-29T15:58:25.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.473+0000 7fc2adedd700 1 -- start start 2022-06-29T15:58:25.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.473+0000 7fc2adedd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc2a805d070 0x7fc2a814aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:25.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.473+0000 7fc2adedd700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc2a814af80 0x7fc2a80ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:25.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.473+0000 7fc2adedd700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a805d450 con 0x7fc2a80b2130 2022-06-29T15:58:25.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.473+0000 7fc2adedd700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2a80adf90 con 0x7fc2a805d070 2022-06-29T15:58:25.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.473+0000 7fc2adedd700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2a80b25d0 con 0x7fc2a814af80 2022-06-29T15:58:25.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.473+0000 7fc29ffff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc2a80b2130 0x7fc2a80aca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53054/0 (socket says 172.21.15.60:53054) 2022-06-29T15:58:25.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.473+0000 7fc2acedb700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc2a805d070 0x7fc2a814aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:25.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.473+0000 7fc29ffff700 1 -- 172.21.15.60:0/3831645825 learned_addr learned my addr 172.21.15.60:0/3831645825 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:25.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.473+0000 7fc2ad6dc700 1 --2- 172.21.15.60:0/3831645825 >> v2:172.21.15.92:3300/0 conn(0x7fc2a814af80 0x7fc2a80ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:25.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.473+0000 7fc29f7fe700 1 -- 172.21.15.60:0/3831645825 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2587191880 0 0) 0x7fc2a805d450 con 0x7fc2a80b2130 2022-06-29T15:58:25.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.473+0000 7fc29f7fe700 1 -- 172.21.15.60:0/3831645825 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc294003260 con 0x7fc2a80b2130 2022-06-29T15:58:25.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.473+0000 7fc2acedb700 1 -- 172.21.15.60:0/3831645825 >> v2:172.21.15.92:3300/0 conn(0x7fc2a814af80 msgr2=0x7fc2a80ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:25.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.473+0000 7fc2acedb700 1 --2- 172.21.15.60:0/3831645825 >> v2:172.21.15.92:3300/0 conn(0x7fc2a814af80 0x7fc2a80ad130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:25.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.473+0000 7fc2acedb700 1 -- 172.21.15.60:0/3831645825 >> v1:172.21.15.60:6789/0 conn(0x7fc2a80b2130 legacy=0x7fc2a80aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:25.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.473+0000 7fc2acedb700 1 -- 172.21.15.60:0/3831645825 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a814a350 con 0x7fc2a805d070 2022-06-29T15:58:25.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.473+0000 7fc2acedb700 1 --2- 172.21.15.60:0/3831645825 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2a805d070 0x7fc2a814aa90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fc2a4002f70 tx=0x7fc2a4006df0).ready entity=mon.0 client_cookie=f527c0e035347000 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:25.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.473+0000 7fc29f7fe700 1 -- 172.21.15.60:0/3831645825 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2a400d4f0 con 0x7fc2a805d070 2022-06-29T15:58:25.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.473+0000 7fc29f7fe700 1 -- 172.21.15.60:0/3831645825 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2a4000e90 con 0x7fc2a805d070 2022-06-29T15:58:25.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.473+0000 7fc29f7fe700 1 -- 172.21.15.60:0/3831645825 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2a400d840 con 0x7fc2a805d070 2022-06-29T15:58:25.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.473+0000 7fc2adedd700 1 -- 172.21.15.60:0/3831645825 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2a805d070 msgr2=0x7fc2a814aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:25.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.473+0000 7fc2adedd700 1 --2- 172.21.15.60:0/3831645825 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2a805d070 0x7fc2a814aa90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fc2a4002f70 tx=0x7fc2a4006df0).stop 2022-06-29T15:58:25.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.473+0000 7fc2adedd700 1 -- 172.21.15.60:0/3831645825 shutdown_connections 2022-06-29T15:58:25.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.473+0000 7fc2adedd700 1 --2- 172.21.15.60:0/3831645825 >> v2:172.21.15.92:3300/0 conn(0x7fc2a814af80 0x7fc2a80ad130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:25.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.473+0000 7fc2adedd700 1 --2- 172.21.15.60:0/3831645825 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2a805d070 0x7fc2a814aa90 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:25.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.473+0000 7fc2adedd700 1 -- 172.21.15.60:0/3831645825 >> 172.21.15.60:0/3831645825 conn(0x7fc2a80b5890 msgr2=0x7fc2a80b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:25.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.473+0000 7fc2adedd700 1 -- 172.21.15.60:0/3831645825 shutdown_connections 2022-06-29T15:58:25.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.473+0000 7fc2adedd700 1 -- 172.21.15.60:0/3831645825 wait complete. 2022-06-29T15:58:25.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.477+0000 7fc2adedd700 1 Processor -- start 2022-06-29T15:58:25.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.477+0000 7fc2adedd700 1 -- start start 2022-06-29T15:58:25.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.477+0000 7fc2adedd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2a805d070 0x7fc2a80650d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:25.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.477+0000 7fc2adedd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2a80b2130 0x7fc2a80655c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:25.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.477+0000 7fc2adedd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2a814af80 0x7fc2a81192d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:25.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.477+0000 7fc2adedd700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a8156f90 con 0x7fc2a814af80 2022-06-29T15:58:25.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.477+0000 7fc2adedd700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a80665c0 con 0x7fc2a805d070 2022-06-29T15:58:25.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.477+0000 7fc2adedd700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a8066860 con 0x7fc2a80b2130 2022-06-29T15:58:25.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.477+0000 7fc2ad6dc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2a814af80 0x7fc2a81192d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:25.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.477+0000 7fc2ad6dc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2a814af80 0x7fc2a81192d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44976/0 (socket says 172.21.15.60:44976) 2022-06-29T15:58:25.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.477+0000 7fc29ffff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2a80b2130 0x7fc2a80655c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:25.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.477+0000 7fc2ad6dc700 1 -- 172.21.15.60:0/2009214571 learned_addr learned my addr 172.21.15.60:0/2009214571 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:25.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.477+0000 7fc2acedb700 1 --2- 172.21.15.60:0/2009214571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2a805d070 0x7fc2a80650d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:25.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.481+0000 7fc2ad6dc700 1 -- 172.21.15.60:0/2009214571 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2a80b2130 msgr2=0x7fc2a80655c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:25.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.481+0000 7fc2ad6dc700 1 --2- 172.21.15.60:0/2009214571 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2a80b2130 0x7fc2a80655c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:25.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.481+0000 7fc2ad6dc700 1 -- 172.21.15.60:0/2009214571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2a805d070 msgr2=0x7fc2a80650d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:25.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.481+0000 7fc2ad6dc700 1 --2- 172.21.15.60:0/2009214571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2a805d070 0x7fc2a80650d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:25.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.481+0000 7fc2ad6dc700 1 -- 172.21.15.60:0/2009214571 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a400b180 con 0x7fc2a814af80 2022-06-29T15:58:25.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.481+0000 7fc2acedb700 1 --2- 172.21.15.60:0/2009214571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2a805d070 0x7fc2a80650d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:25.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.481+0000 7fc2ad6dc700 1 --2- 172.21.15.60:0/2009214571 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2a814af80 0x7fc2a81192d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fc2a000c000 tx=0x7fc2a00091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:25.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.481+0000 7fc29dffb700 1 -- 172.21.15.60:0/2009214571 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2a0012600 con 0x7fc2a814af80 2022-06-29T15:58:25.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.481+0000 7fc2adedd700 1 -- 172.21.15.60:0/2009214571 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a8066230 con 0x7fc2a814af80 2022-06-29T15:58:25.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.481+0000 7fc29dffb700 1 -- 172.21.15.60:0/2009214571 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2a0012760 con 0x7fc2a814af80 2022-06-29T15:58:25.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.481+0000 7fc29dffb700 1 -- 172.21.15.60:0/2009214571 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2a0012ac0 con 0x7fc2a814af80 2022-06-29T15:58:25.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.481+0000 7fc2adedd700 1 -- 172.21.15.60:0/2009214571 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a8156d40 con 0x7fc2a814af80 2022-06-29T15:58:25.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.481+0000 7fc2adedd700 1 -- 172.21.15.60:0/2009214571 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2a814a5f0 con 0x7fc2a814af80 2022-06-29T15:58:25.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.481+0000 7fc29dffb700 1 -- 172.21.15.60:0/2009214571 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc2a0022070 con 0x7fc2a814af80 2022-06-29T15:58:25.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.485+0000 7fc29dffb700 1 --2- 172.21.15.60:0/2009214571 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc288053a70 0x7fc288055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:25.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.485+0000 7fc29dffb700 1 -- 172.21.15.60:0/2009214571 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc2a006cac0 con 0x7fc2a814af80 2022-06-29T15:58:25.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.485+0000 7fc2acedb700 1 --2- 172.21.15.60:0/2009214571 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc288053a70 0x7fc288055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:25.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.485+0000 7fc2acedb700 1 --2- 172.21.15.60:0/2009214571 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc288053a70 0x7fc288055f30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fc2a401a000 tx=0x7fc2a4014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:25.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.485+0000 7fc29dffb700 1 -- 172.21.15.60:0/2009214571 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc2a00438e0 con 0x7fc2a814af80 2022-06-29T15:58:25.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.649+0000 7fc2adedd700 1 -- 172.21.15.60:0/2009214571 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2a8066a20 con 0x7fc2a814af80 2022-06-29T15:58:25.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.649+0000 7fc29dffb700 1 -- 172.21.15.60:0/2009214571 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2134 (secure 0 0 0) 0x7fc2a0029870 con 0x7fc2a814af80 2022-06-29T15:58:25.656 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:58:25.657 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":4,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.92:6835/2121698334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2121698334},{"type":"v1","addr":"172.21.15.92:6835","nonce":2121698334}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":4,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T15:58:24.334770+0000","modified":"2022-06-29T15:58:25.338504+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"a","rank":0,"incarnation":3,"state":"up:active","state_seq":2,"addr":"172.21.15.60:6833/3164364318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3164364318},{"type":"v1","addr":"172.21.15.60:6833","nonce":3164364318}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-29T15:58:25.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.653+0000 7fc2adedd700 1 -- 172.21.15.60:0/2009214571 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc288053a70 msgr2=0x7fc288055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:25.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.653+0000 7fc2adedd700 1 --2- 172.21.15.60:0/2009214571 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc288053a70 0x7fc288055f30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fc2a401a000 tx=0x7fc2a4014000).stop 2022-06-29T15:58:25.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.653+0000 7fc2adedd700 1 -- 172.21.15.60:0/2009214571 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2a814af80 msgr2=0x7fc2a81192d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:25.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.653+0000 7fc2adedd700 1 --2- 172.21.15.60:0/2009214571 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2a814af80 0x7fc2a81192d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fc2a000c000 tx=0x7fc2a00091d0).stop 2022-06-29T15:58:25.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.653+0000 7fc2adedd700 1 -- 172.21.15.60:0/2009214571 shutdown_connections 2022-06-29T15:58:25.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.653+0000 7fc2adedd700 1 --2- 172.21.15.60:0/2009214571 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc288053a70 0x7fc288055f30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:25.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.653+0000 7fc2adedd700 1 --2- 172.21.15.60:0/2009214571 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2a814af80 0x7fc2a81192d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:25.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.653+0000 7fc2adedd700 1 --2- 172.21.15.60:0/2009214571 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2a80b2130 0x7fc2a80655c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:25.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.653+0000 7fc2adedd700 1 --2- 172.21.15.60:0/2009214571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2a805d070 0x7fc2a80650d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:25.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.653+0000 7fc2adedd700 1 -- 172.21.15.60:0/2009214571 >> 172.21.15.60:0/2009214571 conn(0x7fc2a80b5890 msgr2=0x7fc2a8144650 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:25.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.653+0000 7fc2adedd700 1 -- 172.21.15.60:0/2009214571 shutdown_connections 2022-06-29T15:58:25.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.653+0000 7fc2adedd700 1 -- 172.21.15.60:0/2009214571 wait complete. 2022-06-29T15:58:25.662 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 4 2022-06-29T15:58:25.670 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T15:58:25.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.769+0000 7fd95dc6c700 1 Processor -- start 2022-06-29T15:58:25.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.769+0000 7fd95dc6c700 1 -- start start 2022-06-29T15:58:25.778 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.769+0000 7fd95dc6c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd95805d070 0x7fd95814aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:25.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.769+0000 7fd95dc6c700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd95805d450 con 0x7fd95814af80 2022-06-29T15:58:25.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.769+0000 7fd95dc6c700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9580ad840 con 0x7fd9580b2130 2022-06-29T15:58:25.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.769+0000 7fd95dc6c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9580adf90 con 0x7fd95805d070 2022-06-29T15:58:25.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.773+0000 7fd95d46b700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd95814af80 0x7fd9580ad130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53068/0 (socket says 172.21.15.60:53068) 2022-06-29T15:58:25.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.773+0000 7fd95d46b700 1 -- 172.21.15.60:0/3428017142 learned_addr learned my addr 172.21.15.60:0/3428017142 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:25.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.773+0000 7fd95cc6a700 1 --2- 172.21.15.60:0/3428017142 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd95805d070 0x7fd95814aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:25.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.773+0000 7fd9577fe700 1 -- 172.21.15.60:0/3428017142 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2411316709 0 0) 0x7fd95805d450 con 0x7fd95814af80 2022-06-29T15:58:25.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.773+0000 7fd9577fe700 1 -- 172.21.15.60:0/3428017142 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd944003260 con 0x7fd95814af80 2022-06-29T15:58:25.781 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.773+0000 7fd95cc6a700 1 -- 172.21.15.60:0/3428017142 >> v1:172.21.15.92:6789/0 conn(0x7fd9580b2130 legacy=0x7fd9580aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:25.781 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.773+0000 7fd95cc6a700 1 -- 172.21.15.60:0/3428017142 >> v1:172.21.15.60:6789/0 conn(0x7fd95814af80 legacy=0x7fd9580ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:25.781 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.773+0000 7fd95cc6a700 1 -- 172.21.15.60:0/3428017142 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9580b25d0 con 0x7fd95805d070 2022-06-29T15:58:25.781 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.773+0000 7fd95cc6a700 1 --2- 172.21.15.60:0/3428017142 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd95805d070 0x7fd95814aa90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fd9500032f0 tx=0x7fd9500091d0).ready entity=mon.2 client_cookie=3b560992adc5bda8 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:25.782 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.773+0000 7fd9577fe700 1 -- 172.21.15.60:0/3428017142 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd95000b6c0 con 0x7fd95805d070 2022-06-29T15:58:25.782 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.773+0000 7fd9577fe700 1 -- 172.21.15.60:0/3428017142 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd95000b820 con 0x7fd95805d070 2022-06-29T15:58:25.783 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.773+0000 7fd9577fe700 1 -- 172.21.15.60:0/3428017142 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd95000b990 con 0x7fd95805d070 2022-06-29T15:58:25.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.773+0000 7fd95dc6c700 1 -- 172.21.15.60:0/3428017142 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd95805d070 msgr2=0x7fd95814aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:25.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.773+0000 7fd95dc6c700 1 --2- 172.21.15.60:0/3428017142 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd95805d070 0x7fd95814aa90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fd9500032f0 tx=0x7fd9500091d0).stop 2022-06-29T15:58:25.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.773+0000 7fd95dc6c700 1 -- 172.21.15.60:0/3428017142 shutdown_connections 2022-06-29T15:58:25.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.773+0000 7fd95dc6c700 1 --2- 172.21.15.60:0/3428017142 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd95805d070 0x7fd95814aa90 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:25.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.773+0000 7fd95dc6c700 1 -- 172.21.15.60:0/3428017142 >> 172.21.15.60:0/3428017142 conn(0x7fd9580b5890 msgr2=0x7fd9580b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:25.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.773+0000 7fd95dc6c700 1 -- 172.21.15.60:0/3428017142 shutdown_connections 2022-06-29T15:58:25.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.773+0000 7fd95dc6c700 1 -- 172.21.15.60:0/3428017142 wait complete. 2022-06-29T15:58:25.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.777+0000 7fd95dc6c700 1 Processor -- start 2022-06-29T15:58:25.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.777+0000 7fd95dc6c700 1 -- start start 2022-06-29T15:58:25.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.777+0000 7fd95dc6c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd95805d070 0x7fd958063400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:25.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.777+0000 7fd95dc6c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9580b2130 0x7fd958159760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:25.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.777+0000 7fd95dc6c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd95814af80 0x7fd958159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:25.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.777+0000 7fd95dc6c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd958065700 con 0x7fd95814af80 2022-06-29T15:58:25.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.777+0000 7fd95dc6c700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd958116780 con 0x7fd95805d070 2022-06-29T15:58:25.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.777+0000 7fd95dc6c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd958065930 con 0x7fd9580b2130 2022-06-29T15:58:25.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.777+0000 7fd95d46b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd95814af80 0x7fd958159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:25.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.777+0000 7fd95d46b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd95814af80 0x7fd958159c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44990/0 (socket says 172.21.15.60:44990) 2022-06-29T15:58:25.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.777+0000 7fd95d46b700 1 -- 172.21.15.60:0/1582538982 learned_addr learned my addr 172.21.15.60:0/1582538982 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:25.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.777+0000 7fd95cc6a700 1 --2- 172.21.15.60:0/1582538982 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd95805d070 0x7fd958063400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:25.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.777+0000 7fd957fff700 1 --2- 172.21.15.60:0/1582538982 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9580b2130 0x7fd958159760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:25.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.781+0000 7fd95d46b700 1 -- 172.21.15.60:0/1582538982 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9580b2130 msgr2=0x7fd958159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:25.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.781+0000 7fd95d46b700 1 --2- 172.21.15.60:0/1582538982 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9580b2130 0x7fd958159760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:25.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.781+0000 7fd95d46b700 1 -- 172.21.15.60:0/1582538982 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd95805d070 msgr2=0x7fd958063400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:25.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.781+0000 7fd95d46b700 1 --2- 172.21.15.60:0/1582538982 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd95805d070 0x7fd958063400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:25.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.781+0000 7fd95d46b700 1 -- 172.21.15.60:0/1582538982 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd950009470 con 0x7fd95814af80 2022-06-29T15:58:25.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.781+0000 7fd95d46b700 1 --2- 172.21.15.60:0/1582538982 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd95814af80 0x7fd958159c50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fd948001110 tx=0x7fd9480079e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:25.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.781+0000 7fd955ffb700 1 -- 172.21.15.60:0/1582538982 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd94800cd60 con 0x7fd95814af80 2022-06-29T15:58:25.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.781+0000 7fd955ffb700 1 -- 172.21.15.60:0/1582538982 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9480070d0 con 0x7fd95814af80 2022-06-29T15:58:25.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.781+0000 7fd95dc6c700 1 -- 172.21.15.60:0/1582538982 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9581169e0 con 0x7fd95814af80 2022-06-29T15:58:25.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.781+0000 7fd955ffb700 1 -- 172.21.15.60:0/1582538982 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd948007430 con 0x7fd95814af80 2022-06-29T15:58:25.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.781+0000 7fd95dc6c700 1 -- 172.21.15.60:0/1582538982 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd958116e80 con 0x7fd95814af80 2022-06-29T15:58:25.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.781+0000 7fd95dc6c700 1 -- 172.21.15.60:0/1582538982 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd95814a350 con 0x7fd95814af80 2022-06-29T15:58:25.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.781+0000 7fd955ffb700 1 -- 172.21.15.60:0/1582538982 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd948010070 con 0x7fd95814af80 2022-06-29T15:58:25.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.781+0000 7fd955ffb700 1 --2- 172.21.15.60:0/1582538982 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd938053aa0 0x7fd938055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:25.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.781+0000 7fd955ffb700 1 -- 172.21.15.60:0/1582538982 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd94806b2c0 con 0x7fd95814af80 2022-06-29T15:58:25.797 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.781+0000 7fd95cc6a700 1 --2- 172.21.15.60:0/1582538982 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd938053aa0 0x7fd938055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:25.797 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.785+0000 7fd95cc6a700 1 --2- 172.21.15.60:0/1582538982 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd938053aa0 0x7fd938055f60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fd95001b000 tx=0x7fd950016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:25.797 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.789+0000 7fd955ffb700 1 -- 172.21.15.60:0/1582538982 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd9480421e0 con 0x7fd95814af80 2022-06-29T15:58:25.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.913+0000 7fd95dc6c700 1 -- 172.21.15.60:0/1582538982 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd958116b70 con 0x7fd95814af80 2022-06-29T15:58:25.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.917+0000 7fd955ffb700 1 -- 172.21.15.60:0/1582538982 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16051 (secure 0 0 0) 0x7fd948045800 con 0x7fd95814af80 2022-06-29T15:58:25.922 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:58:25.922 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":22,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T15:58:25.338104+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-06-29T15:58:22.424395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-06-29T15:58:23.343438+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T15:58:25.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.917+0000 7fd95dc6c700 1 -- 172.21.15.60:0/1582538982 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd938053aa0 msgr2=0x7fd938055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:25.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.917+0000 7fd95dc6c700 1 --2- 172.21.15.60:0/1582538982 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd938053aa0 0x7fd938055f60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fd95001b000 tx=0x7fd950016000).stop 2022-06-29T15:58:25.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.917+0000 7fd95dc6c700 1 -- 172.21.15.60:0/1582538982 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd95814af80 msgr2=0x7fd958159c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:25.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.917+0000 7fd95dc6c700 1 --2- 172.21.15.60:0/1582538982 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd95814af80 0x7fd958159c50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fd948001110 tx=0x7fd9480079e0).stop 2022-06-29T15:58:25.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.917+0000 7fd95dc6c700 1 -- 172.21.15.60:0/1582538982 shutdown_connections 2022-06-29T15:58:25.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.917+0000 7fd95dc6c700 1 --2- 172.21.15.60:0/1582538982 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd938053aa0 0x7fd938055f60 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:25.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.917+0000 7fd95dc6c700 1 --2- 172.21.15.60:0/1582538982 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd95814af80 0x7fd958159c50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:25.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.917+0000 7fd95dc6c700 1 --2- 172.21.15.60:0/1582538982 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9580b2130 0x7fd958159760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:25.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.917+0000 7fd95dc6c700 1 --2- 172.21.15.60:0/1582538982 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd95805d070 0x7fd958063400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:25.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.917+0000 7fd95dc6c700 1 -- 172.21.15.60:0/1582538982 >> 172.21.15.60:0/1582538982 conn(0x7fd9580b5890 msgr2=0x7fd9580aca20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:25.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.917+0000 7fd95dc6c700 1 -- 172.21.15.60:0/1582538982 shutdown_connections 2022-06-29T15:58:25.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:25.917+0000 7fd95dc6c700 1 -- 172.21.15.60:0/1582538982 wait complete. 2022-06-29T15:58:25.936 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T15:58:25.937 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2022-06-29T15:58:26.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.029+0000 7f6839b21700 1 Processor -- start 2022-06-29T15:58:26.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.029+0000 7f6839b21700 1 -- start start 2022-06-29T15:58:26.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.033+0000 7f6839b21700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f683405d1f0 0x7f68340b1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:26.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.033+0000 7f6839b21700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68340b3cc0 0x7f68340b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:26.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.033+0000 7f6839b21700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68340b2ce0 con 0x7f68340b9d70 2022-06-29T15:58:26.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.033+0000 7f6839b21700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68340b4d10 con 0x7f683405d1f0 2022-06-29T15:58:26.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.033+0000 7f6839b21700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68340ba150 con 0x7f68340b3cc0 2022-06-29T15:58:26.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.033+0000 7f6832ffd700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f683405d1f0 0x7f68340b1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:26.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.033+0000 7f6833fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68340b3cc0 0x7f68340b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:26.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.033+0000 7f6832ffd700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f683405d1f0 0x7f68340b1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:51024/0 (socket says 172.21.15.60:51024) 2022-06-29T15:58:26.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.033+0000 7f6832ffd700 1 -- 172.21.15.60:0/4187166964 learned_addr learned my addr 172.21.15.60:0/4187166964 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:26.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.033+0000 7f6832ffd700 1 -- 172.21.15.60:0/4187166964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68340b3cc0 msgr2=0x7f68340b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:26.042 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.033+0000 7f6832ffd700 1 --2- 172.21.15.60:0/4187166964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68340b3cc0 0x7f68340b2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.042 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.033+0000 7f6832ffd700 1 -- 172.21.15.60:0/4187166964 >> v1:172.21.15.92:6789/0 conn(0x7f68340b9d70 legacy=0x7f68340b3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:26.042 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.033+0000 7f6832ffd700 1 -- 172.21.15.60:0/4187166964 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68340b4a70 con 0x7f683405d1f0 2022-06-29T15:58:26.042 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.033+0000 7f6833fff700 1 --2- 172.21.15.60:0/4187166964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68340b3cc0 0x7f68340b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:26.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.033+0000 7f6832ffd700 1 --2- 172.21.15.60:0/4187166964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f683405d1f0 0x7f68340b1b50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f68280032f0 tx=0x7f6828009290).ready entity=mon.1 client_cookie=e81d3fb7901311c5 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:26.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.033+0000 7f68327fc700 1 -- 172.21.15.60:0/4187166964 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f682800b8a0 con 0x7f683405d1f0 2022-06-29T15:58:26.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.033+0000 7f68327fc700 1 -- 172.21.15.60:0/4187166964 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f682800ba00 con 0x7f683405d1f0 2022-06-29T15:58:26.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.033+0000 7f68327fc700 1 -- 172.21.15.60:0/4187166964 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f682800bb70 con 0x7f683405d1f0 2022-06-29T15:58:26.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.033+0000 7f6839b21700 1 -- 172.21.15.60:0/4187166964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f683405d1f0 msgr2=0x7f68340b1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:26.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.033+0000 7f6839b21700 1 --2- 172.21.15.60:0/4187166964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f683405d1f0 0x7f68340b1b50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f68280032f0 tx=0x7f6828009290).stop 2022-06-29T15:58:26.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.033+0000 7f6839b21700 1 -- 172.21.15.60:0/4187166964 shutdown_connections 2022-06-29T15:58:26.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.033+0000 7f6839b21700 1 --2- 172.21.15.60:0/4187166964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68340b3cc0 0x7f68340b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.033+0000 7f6839b21700 1 --2- 172.21.15.60:0/4187166964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f683405d1f0 0x7f68340b1b50 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.033+0000 7f6839b21700 1 -- 172.21.15.60:0/4187166964 >> 172.21.15.60:0/4187166964 conn(0x7f6834062ce0 msgr2=0x7f68340b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:26.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.033+0000 7f6839b21700 1 -- 172.21.15.60:0/4187166964 shutdown_connections 2022-06-29T15:58:26.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.033+0000 7f6839b21700 1 -- 172.21.15.60:0/4187166964 wait complete. 2022-06-29T15:58:26.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.037+0000 7f6839b21700 1 Processor -- start 2022-06-29T15:58:26.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.037+0000 7f6839b21700 1 -- start start 2022-06-29T15:58:26.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.037+0000 7f6839b21700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f683405d1f0 0x7f68340af920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:26.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.037+0000 7f6839b21700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68340b3cc0 0x7f68341552d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:26.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.037+0000 7f6839b21700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68340b9d70 0x7f68341557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:26.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.037+0000 7f6839b21700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68340ae870 con 0x7f68340b3cc0 2022-06-29T15:58:26.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.037+0000 7f6839b21700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6834146bd0 con 0x7f683405d1f0 2022-06-29T15:58:26.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.037+0000 7f6839b21700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68341467d0 con 0x7f68340b9d70 2022-06-29T15:58:26.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.037+0000 7f6832ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68340b3cc0 0x7f68341552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:26.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.037+0000 7f6833fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68340b9d70 0x7f68341557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:26.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.037+0000 7f6832ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68340b3cc0 0x7f68341552d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45004/0 (socket says 172.21.15.60:45004) 2022-06-29T15:58:26.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.037+0000 7f6832ffd700 1 -- 172.21.15.60:0/3925458251 learned_addr learned my addr 172.21.15.60:0/3925458251 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:26.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.037+0000 7f68337fe700 1 --2- 172.21.15.60:0/3925458251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f683405d1f0 0x7f68340af920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:26.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.037+0000 7f6832ffd700 1 -- 172.21.15.60:0/3925458251 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68340b9d70 msgr2=0x7f68341557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:26.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.037+0000 7f6832ffd700 1 --2- 172.21.15.60:0/3925458251 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68340b9d70 0x7f68341557c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.037+0000 7f6832ffd700 1 -- 172.21.15.60:0/3925458251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f683405d1f0 msgr2=0x7f68340af920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:26.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.037+0000 7f6832ffd700 1 --2- 172.21.15.60:0/3925458251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f683405d1f0 0x7f68340af920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.037+0000 7f6832ffd700 1 -- 172.21.15.60:0/3925458251 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f682c00b180 con 0x7f68340b3cc0 2022-06-29T15:58:26.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.037+0000 7f6833fff700 1 --2- 172.21.15.60:0/3925458251 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68340b9d70 0x7f68341557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:26.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.037+0000 7f6832ffd700 1 --2- 172.21.15.60:0/3925458251 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68340b3cc0 0x7f68341552d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f68280032f0 tx=0x7f6828016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:26.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.037+0000 7f6830ff9700 1 -- 172.21.15.60:0/3925458251 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f682800b8a0 con 0x7f68340b3cc0 2022-06-29T15:58:26.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.037+0000 7f6830ff9700 1 -- 172.21.15.60:0/3925458251 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6828021480 con 0x7f68340b3cc0 2022-06-29T15:58:26.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.037+0000 7f6839b21700 1 -- 172.21.15.60:0/3925458251 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6828009530 con 0x7f68340b3cc0 2022-06-29T15:58:26.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.037+0000 7f6830ff9700 1 -- 172.21.15.60:0/3925458251 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68280217e0 con 0x7f68340b3cc0 2022-06-29T15:58:26.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.037+0000 7f6839b21700 1 -- 172.21.15.60:0/3925458251 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6834151fa0 con 0x7f68340b3cc0 2022-06-29T15:58:26.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.041+0000 7f6839b21700 1 -- 172.21.15.60:0/3925458251 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68340573d0 con 0x7f68340b3cc0 2022-06-29T15:58:26.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.041+0000 7f6830ff9700 1 -- 172.21.15.60:0/3925458251 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f682800f070 con 0x7f68340b3cc0 2022-06-29T15:58:26.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.041+0000 7f6830ff9700 1 --2- 172.21.15.60:0/3925458251 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6814053a70 0x7f6814055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:26.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.041+0000 7f6830ff9700 1 -- 172.21.15.60:0/3925458251 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f682806d0e0 con 0x7f68340b3cc0 2022-06-29T15:58:26.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.041+0000 7f68337fe700 1 --2- 172.21.15.60:0/3925458251 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6814053a70 0x7f6814055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:26.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.041+0000 7f68337fe700 1 --2- 172.21.15.60:0/3925458251 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6814053a70 0x7f6814055f30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f682400d000 tx=0x7f6824007b50).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:26.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.049+0000 7f6830ff9700 1 -- 172.21.15.60:0/3925458251 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f68280440b0 con 0x7f68340b3cc0 2022-06-29T15:58:26.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.173+0000 7f6839b21700 1 -- 172.21.15.60:0/3925458251 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f683405d990 con 0x7f68340b3cc0 2022-06-29T15:58:26.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.177+0000 7f6830ff9700 1 -- 172.21.15.60:0/3925458251 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16051 (secure 0 0 0) 0x7f6828015070 con 0x7f68340b3cc0 2022-06-29T15:58:26.182 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:58:26.183 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":22,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T15:58:25.338104+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-06-29T15:58:22.424395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-06-29T15:58:23.343438+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T15:58:26.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.177+0000 7f6839b21700 1 -- 172.21.15.60:0/3925458251 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6814053a70 msgr2=0x7f6814055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:26.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.177+0000 7f6839b21700 1 --2- 172.21.15.60:0/3925458251 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6814053a70 0x7f6814055f30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f682400d000 tx=0x7f6824007b50).stop 2022-06-29T15:58:26.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.177+0000 7f6839b21700 1 -- 172.21.15.60:0/3925458251 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68340b3cc0 msgr2=0x7f68341552d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:26.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.177+0000 7f6839b21700 1 --2- 172.21.15.60:0/3925458251 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68340b3cc0 0x7f68341552d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f68280032f0 tx=0x7f6828016000).stop 2022-06-29T15:58:26.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.177+0000 7f6839b21700 1 -- 172.21.15.60:0/3925458251 shutdown_connections 2022-06-29T15:58:26.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.177+0000 7f6839b21700 1 --2- 172.21.15.60:0/3925458251 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6814053a70 0x7f6814055f30 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.177+0000 7f6839b21700 1 --2- 172.21.15.60:0/3925458251 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68340b9d70 0x7f68341557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.177+0000 7f6839b21700 1 --2- 172.21.15.60:0/3925458251 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68340b3cc0 0x7f68341552d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.177+0000 7f6839b21700 1 --2- 172.21.15.60:0/3925458251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f683405d1f0 0x7f68340af920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.177+0000 7f6839b21700 1 -- 172.21.15.60:0/3925458251 >> 172.21.15.60:0/3925458251 conn(0x7f6834062ce0 msgr2=0x7f683413ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:26.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.177+0000 7f6839b21700 1 -- 172.21.15.60:0/3925458251 shutdown_connections 2022-06-29T15:58:26.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.177+0000 7f6839b21700 1 -- 172.21.15.60:0/3925458251 wait complete. 2022-06-29T15:58:26.197 INFO:tasks.ceph.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-29T15:58:22.424395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-06-29T15:58:23.343438+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-29T15:58:26.198 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T15:58:26.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.301+0000 7f7cb3fff700 1 Processor -- start 2022-06-29T15:58:26.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.301+0000 7f7cb3fff700 1 -- start start 2022-06-29T15:58:26.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.301+0000 7f7cb3fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7cb40b2120 0x7f7cb40ab880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:26.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.301+0000 7f7cb3fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7cb405cf40 0x7f7cb414af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:26.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.301+0000 7f7cb3fff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cb414a530 con 0x7f7cb405bd20 2022-06-29T15:58:26.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.301+0000 7f7cb3fff700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cb40b1720 con 0x7f7cb40b2120 2022-06-29T15:58:26.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.301+0000 7f7cb3fff700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cb414a7d0 con 0x7f7cb405cf40 2022-06-29T15:58:26.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.301+0000 7f7cb2ffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7cb40b2120 0x7f7cb40ab880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:26.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.301+0000 7f7cb37fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f7cb405bd20 0x7f7cb405c190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53098/0 (socket says 172.21.15.60:53098) 2022-06-29T15:58:26.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.305+0000 7f7cb37fe700 1 -- 172.21.15.60:0/3216549625 learned_addr learned my addr 172.21.15.60:0/3216549625 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:26.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.305+0000 7f7cb27fc700 1 --2- 172.21.15.60:0/3216549625 >> v2:172.21.15.92:3300/0 conn(0x7f7cb405cf40 0x7f7cb414af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:26.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.305+0000 7f7cb1ffb700 1 -- 172.21.15.60:0/3216549625 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3389308756 0 0) 0x7f7cb414a530 con 0x7f7cb405bd20 2022-06-29T15:58:26.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.305+0000 7f7cb1ffb700 1 -- 172.21.15.60:0/3216549625 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c98003260 con 0x7f7cb405bd20 2022-06-29T15:58:26.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.305+0000 7f7cb2ffd700 1 -- 172.21.15.60:0/3216549625 >> v2:172.21.15.92:3300/0 conn(0x7f7cb405cf40 msgr2=0x7f7cb414af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:26.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.305+0000 7f7cb2ffd700 1 --2- 172.21.15.60:0/3216549625 >> v2:172.21.15.92:3300/0 conn(0x7f7cb405cf40 0x7f7cb414af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.305+0000 7f7cb2ffd700 1 -- 172.21.15.60:0/3216549625 >> v1:172.21.15.60:6789/0 conn(0x7f7cb405bd20 legacy=0x7f7cb405c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:26.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.305+0000 7f7cb2ffd700 1 -- 172.21.15.60:0/3216549625 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb414aa70 con 0x7f7cb40b2120 2022-06-29T15:58:26.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.305+0000 7f7cb2ffd700 1 --2- 172.21.15.60:0/3216549625 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7cb40b2120 0x7f7cb40ab880 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f7ca8002f70 tx=0x7f7ca8006df0).ready entity=mon.0 client_cookie=8368d0411edfc820 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:26.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.305+0000 7f7cb1ffb700 1 -- 172.21.15.60:0/3216549625 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ca800d4f0 con 0x7f7cb40b2120 2022-06-29T15:58:26.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.305+0000 7f7cb1ffb700 1 -- 172.21.15.60:0/3216549625 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ca8000e90 con 0x7f7cb40b2120 2022-06-29T15:58:26.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.305+0000 7f7cb1ffb700 1 -- 172.21.15.60:0/3216549625 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ca800d840 con 0x7f7cb40b2120 2022-06-29T15:58:26.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.305+0000 7f7cb3fff700 1 -- 172.21.15.60:0/3216549625 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7cb40b2120 msgr2=0x7f7cb40ab880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:26.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.305+0000 7f7cb3fff700 1 --2- 172.21.15.60:0/3216549625 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7cb40b2120 0x7f7cb40ab880 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f7ca8002f70 tx=0x7f7ca8006df0).stop 2022-06-29T15:58:26.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.305+0000 7f7cb3fff700 1 -- 172.21.15.60:0/3216549625 shutdown_connections 2022-06-29T15:58:26.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.305+0000 7f7cb3fff700 1 --2- 172.21.15.60:0/3216549625 >> v2:172.21.15.92:3300/0 conn(0x7f7cb405cf40 0x7f7cb414af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.305+0000 7f7cb3fff700 1 --2- 172.21.15.60:0/3216549625 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7cb40b2120 0x7f7cb40ab880 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.305+0000 7f7cb3fff700 1 -- 172.21.15.60:0/3216549625 >> 172.21.15.60:0/3216549625 conn(0x7f7cb40b58a0 msgr2=0x7f7cb40b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:26.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.305+0000 7f7cb3fff700 1 -- 172.21.15.60:0/3216549625 shutdown_connections 2022-06-29T15:58:26.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.305+0000 7f7cb3fff700 1 -- 172.21.15.60:0/3216549625 wait complete. 2022-06-29T15:58:26.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.309+0000 7f7cb3fff700 1 Processor -- start 2022-06-29T15:58:26.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.309+0000 7f7cb3fff700 1 -- start start 2022-06-29T15:58:26.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.309+0000 7f7cb3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7cb405bd20 0x7f7cb4063920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:26.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.309+0000 7f7cb3fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7cb405cf40 0x7f7cb40b0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:26.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.309+0000 7f7cb3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7cb40b2120 0x7f7cb4065750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:26.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.309+0000 7f7cb3fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb4064c70 con 0x7f7cb405bd20 2022-06-29T15:58:26.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.309+0000 7f7cb3fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb4063f50 con 0x7f7cb40b2120 2022-06-29T15:58:26.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.309+0000 7f7cb3fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb40641f0 con 0x7f7cb405cf40 2022-06-29T15:58:26.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.309+0000 7f7cb2ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7cb405bd20 0x7f7cb4063920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:26.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.309+0000 7f7cb2ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7cb405bd20 0x7f7cb4063920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45014/0 (socket says 172.21.15.60:45014) 2022-06-29T15:58:26.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.309+0000 7f7cb2ffd700 1 -- 172.21.15.60:0/377879377 learned_addr learned my addr 172.21.15.60:0/377879377 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:26.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.309+0000 7f7cb37fe700 1 --2- 172.21.15.60:0/377879377 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7cb40b2120 0x7f7cb4065750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:26.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.309+0000 7f7cb27fc700 1 --2- 172.21.15.60:0/377879377 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7cb405cf40 0x7f7cb40b0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:26.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.309+0000 7f7cb2ffd700 1 -- 172.21.15.60:0/377879377 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7cb405cf40 msgr2=0x7f7cb40b0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:26.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.309+0000 7f7cb2ffd700 1 --2- 172.21.15.60:0/377879377 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7cb405cf40 0x7f7cb40b0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.309+0000 7f7cb2ffd700 1 -- 172.21.15.60:0/377879377 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7cb40b2120 msgr2=0x7f7cb4065750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:26.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.309+0000 7f7cb2ffd700 1 --2- 172.21.15.60:0/377879377 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7cb40b2120 0x7f7cb4065750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.309+0000 7f7cb2ffd700 1 -- 172.21.15.60:0/377879377 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca800b180 con 0x7f7cb405bd20 2022-06-29T15:58:26.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.309+0000 7f7cb2ffd700 1 --2- 172.21.15.60:0/377879377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7cb405bd20 0x7f7cb4063920 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f7ca8002f70 tx=0x7f7ca8014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:26.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.309+0000 7f7ca3fff700 1 -- 172.21.15.60:0/377879377 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ca8016070 con 0x7f7cb405bd20 2022-06-29T15:58:26.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.309+0000 7f7ca3fff700 1 -- 172.21.15.60:0/377879377 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ca80138d0 con 0x7f7cb405bd20 2022-06-29T15:58:26.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.309+0000 7f7cb3fff700 1 -- 172.21.15.60:0/377879377 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cb40996f0 con 0x7f7cb405bd20 2022-06-29T15:58:26.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.309+0000 7f7ca3fff700 1 -- 172.21.15.60:0/377879377 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ca8013c30 con 0x7f7cb405bd20 2022-06-29T15:58:26.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.309+0000 7f7cb3fff700 1 -- 172.21.15.60:0/377879377 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cb4065e60 con 0x7f7cb405bd20 2022-06-29T15:58:26.326 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.313+0000 7f7cb3fff700 1 -- 172.21.15.60:0/377879377 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cb409af10 con 0x7f7cb405bd20 2022-06-29T15:58:26.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.313+0000 7f7ca3fff700 1 -- 172.21.15.60:0/377879377 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7ca8025030 con 0x7f7cb405bd20 2022-06-29T15:58:26.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.313+0000 7f7ca3fff700 1 --2- 172.21.15.60:0/377879377 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7c90053a70 0x7f7c90055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:26.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.313+0000 7f7ca3fff700 1 -- 172.21.15.60:0/377879377 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7ca806ccf0 con 0x7f7cb405bd20 2022-06-29T15:58:26.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.313+0000 7f7cb27fc700 1 --2- 172.21.15.60:0/377879377 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7c90053a70 0x7f7c90055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:26.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.313+0000 7f7cb27fc700 1 --2- 172.21.15.60:0/377879377 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7c90053a70 0x7f7c90055f30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f7ca400b000 tx=0x7f7ca40091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:26.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.317+0000 7f7ca3fff700 1 -- 172.21.15.60:0/377879377 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7cb409af10 con 0x7f7cb405bd20 2022-06-29T15:58:26.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.445+0000 7f7cb3fff700 1 -- 172.21.15.60:0/377879377 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f7cb409af10 con 0x7f7cb405bd20 2022-06-29T15:58:26.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.449+0000 7f7ca3fff700 1 -- 172.21.15.60:0/377879377 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v23) v1 ==== 110+0+10 (secure 0 0 0) 0x7f7cb409af10 con 0x7f7cb405bd20 2022-06-29T15:58:26.454 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T15:58:26.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.449+0000 7f7cb3fff700 1 -- 172.21.15.60:0/377879377 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7c90053a70 msgr2=0x7f7c90055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:26.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.449+0000 7f7cb3fff700 1 --2- 172.21.15.60:0/377879377 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7c90053a70 0x7f7c90055f30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f7ca400b000 tx=0x7f7ca40091d0).stop 2022-06-29T15:58:26.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.449+0000 7f7cb3fff700 1 -- 172.21.15.60:0/377879377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7cb405bd20 msgr2=0x7f7cb4063920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:26.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.449+0000 7f7cb3fff700 1 --2- 172.21.15.60:0/377879377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7cb405bd20 0x7f7cb4063920 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f7ca8002f70 tx=0x7f7ca8014000).stop 2022-06-29T15:58:26.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.449+0000 7f7cb3fff700 1 -- 172.21.15.60:0/377879377 shutdown_connections 2022-06-29T15:58:26.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.449+0000 7f7cb3fff700 1 --2- 172.21.15.60:0/377879377 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7c90053a70 0x7f7c90055f30 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.449+0000 7f7cb3fff700 1 --2- 172.21.15.60:0/377879377 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7cb40b2120 0x7f7cb4065750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.449+0000 7f7cb3fff700 1 --2- 172.21.15.60:0/377879377 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7cb405cf40 0x7f7cb40b0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.449+0000 7f7cb3fff700 1 --2- 172.21.15.60:0/377879377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7cb405bd20 0x7f7cb4063920 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.449+0000 7f7cb3fff700 1 -- 172.21.15.60:0/377879377 >> 172.21.15.60:0/377879377 conn(0x7f7cb40b58a0 msgr2=0x7f7cb4146010 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:26.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.449+0000 7f7cb3fff700 1 -- 172.21.15.60:0/377879377 shutdown_connections 2022-06-29T15:58:26.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.449+0000 7f7cb3fff700 1 -- 172.21.15.60:0/377879377 wait complete. 2022-06-29T15:58:26.467 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T15:58:26.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.577+0000 7f4b42a14700 1 Processor -- start 2022-06-29T15:58:26.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.577+0000 7f4b42a14700 1 -- start start 2022-06-29T15:58:26.587 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.581+0000 7f4b42a14700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4b3c143910 0x7f4b3c0ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:26.587 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.581+0000 7f4b42a14700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4b3c0ad620 0x7f4b3c13fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:26.587 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.581+0000 7f4b42a14700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b3c05fee0 con 0x7f4b3c05fb00 2022-06-29T15:58:26.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.581+0000 7f4b42a14700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b3c0b1f00 con 0x7f4b3c0ad620 2022-06-29T15:58:26.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.581+0000 7f4b42a14700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b3c0af1d0 con 0x7f4b3c143910 2022-06-29T15:58:26.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.581+0000 7f4b42213700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4b3c0ad620 0x7f4b3c13fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:26.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.581+0000 7f4b42213700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4b3c0ad620 0x7f4b3c13fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45026/0 (socket says 172.21.15.60:45026) 2022-06-29T15:58:26.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.581+0000 7f4b42213700 1 -- 172.21.15.60:0/1322065283 learned_addr learned my addr 172.21.15.60:0/1322065283 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:26.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.581+0000 7f4b41211700 1 --2- 172.21.15.60:0/1322065283 >> v2:172.21.15.92:3300/0 conn(0x7f4b3c143910 0x7f4b3c0ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:26.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.581+0000 7f4b42213700 1 -- 172.21.15.60:0/1322065283 >> v2:172.21.15.92:3300/0 conn(0x7f4b3c143910 msgr2=0x7f4b3c0ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:26.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.581+0000 7f4b42213700 1 --2- 172.21.15.60:0/1322065283 >> v2:172.21.15.92:3300/0 conn(0x7f4b3c143910 0x7f4b3c0ad130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.581+0000 7f4b42213700 1 -- 172.21.15.60:0/1322065283 >> v1:172.21.15.92:6789/0 conn(0x7f4b3c05fb00 legacy=0x7f4b3c0aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:26.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.581+0000 7f4b42213700 1 -- 172.21.15.60:0/1322065283 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b3c0af470 con 0x7f4b3c0ad620 2022-06-29T15:58:26.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.581+0000 7f4b41211700 1 --2- 172.21.15.60:0/1322065283 >> v2:172.21.15.92:3300/0 conn(0x7f4b3c143910 0x7f4b3c0ad130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:58:26.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.581+0000 7f4b42213700 1 --2- 172.21.15.60:0/1322065283 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b3c0ad620 0x7f4b3c13fe00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f4b340032f0 tx=0x7f4b34009290).ready entity=mon.0 client_cookie=7d9080ddbdb8a337 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:26.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.581+0000 7f4b40a10700 1 -- 172.21.15.60:0/1322065283 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b3400b870 con 0x7f4b3c0ad620 2022-06-29T15:58:26.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.581+0000 7f4b40a10700 1 -- 172.21.15.60:0/1322065283 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b3400b9d0 con 0x7f4b3c0ad620 2022-06-29T15:58:26.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.581+0000 7f4b40a10700 1 -- 172.21.15.60:0/1322065283 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b3400bb40 con 0x7f4b3c0ad620 2022-06-29T15:58:26.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.581+0000 7f4b42a14700 1 -- 172.21.15.60:0/1322065283 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b3c0ad620 msgr2=0x7f4b3c13fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:26.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.581+0000 7f4b42a14700 1 --2- 172.21.15.60:0/1322065283 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b3c0ad620 0x7f4b3c13fe00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f4b340032f0 tx=0x7f4b34009290).stop 2022-06-29T15:58:26.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.581+0000 7f4b42a14700 1 -- 172.21.15.60:0/1322065283 shutdown_connections 2022-06-29T15:58:26.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.581+0000 7f4b42a14700 1 --2- 172.21.15.60:0/1322065283 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b3c0ad620 0x7f4b3c13fe00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.581+0000 7f4b42a14700 1 --2- 172.21.15.60:0/1322065283 >> v2:172.21.15.92:3300/0 conn(0x7f4b3c143910 0x7f4b3c0ad130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.581+0000 7f4b42a14700 1 -- 172.21.15.60:0/1322065283 >> 172.21.15.60:0/1322065283 conn(0x7f4b3c063a60 msgr2=0x7f4b3c05f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:26.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.581+0000 7f4b42a14700 1 -- 172.21.15.60:0/1322065283 shutdown_connections 2022-06-29T15:58:26.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.581+0000 7f4b42a14700 1 -- 172.21.15.60:0/1322065283 wait complete. 2022-06-29T15:58:26.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.585+0000 7f4b42a14700 1 Processor -- start 2022-06-29T15:58:26.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.585+0000 7f4b42a14700 1 -- start start 2022-06-29T15:58:26.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.585+0000 7f4b42a14700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b3c05fb00 0x7f4b3c116110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:26.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.585+0000 7f4b42a14700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4b3c0ad620 0x7f4b3c13fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:26.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.585+0000 7f4b42a14700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b3c143910 0x7f4b3c155780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:26.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.585+0000 7f4b42a14700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3c13cd40 con 0x7f4b3c05fb00 2022-06-29T15:58:26.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.585+0000 7f4b42a14700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3c155350 con 0x7f4b3c143910 2022-06-29T15:58:26.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.585+0000 7f4b42a14700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3c0664e0 con 0x7f4b3c0ad620 2022-06-29T15:58:26.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.585+0000 7f4b41a12700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b3c05fb00 0x7f4b3c116110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:26.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.585+0000 7f4b41a12700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b3c05fb00 0x7f4b3c116110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45028/0 (socket says 172.21.15.60:45028) 2022-06-29T15:58:26.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.585+0000 7f4b41a12700 1 -- 172.21.15.60:0/1219834198 learned_addr learned my addr 172.21.15.60:0/1219834198 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:26.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.585+0000 7f4b41211700 1 --2- 172.21.15.60:0/1219834198 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4b3c0ad620 0x7f4b3c13fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:26.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.585+0000 7f4b42213700 1 --2- 172.21.15.60:0/1219834198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b3c143910 0x7f4b3c155780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:26.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.585+0000 7f4b41a12700 1 -- 172.21.15.60:0/1219834198 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4b3c0ad620 msgr2=0x7f4b3c13fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:26.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.585+0000 7f4b41a12700 1 --2- 172.21.15.60:0/1219834198 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4b3c0ad620 0x7f4b3c13fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.585+0000 7f4b41a12700 1 -- 172.21.15.60:0/1219834198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b3c143910 msgr2=0x7f4b3c155780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:26.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.585+0000 7f4b41a12700 1 --2- 172.21.15.60:0/1219834198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b3c143910 0x7f4b3c155780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.585+0000 7f4b41a12700 1 -- 172.21.15.60:0/1219834198 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b34009530 con 0x7f4b3c05fb00 2022-06-29T15:58:26.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.585+0000 7f4b42213700 1 --2- 172.21.15.60:0/1219834198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b3c143910 0x7f4b3c155780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:58:26.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.585+0000 7f4b41a12700 1 --2- 172.21.15.60:0/1219834198 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b3c05fb00 0x7f4b3c116110 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f4b3c13ce80 tx=0x7f4b38007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:26.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.585+0000 7f4b2effd700 1 -- 172.21.15.60:0/1219834198 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b38012400 con 0x7f4b3c05fb00 2022-06-29T15:58:26.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.585+0000 7f4b2effd700 1 -- 172.21.15.60:0/1219834198 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b38012560 con 0x7f4b3c05fb00 2022-06-29T15:58:26.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.585+0000 7f4b42a14700 1 -- 172.21.15.60:0/1219834198 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b3c0656f0 con 0x7f4b3c05fb00 2022-06-29T15:58:26.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.585+0000 7f4b2effd700 1 -- 172.21.15.60:0/1219834198 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b380128c0 con 0x7f4b3c05fb00 2022-06-29T15:58:26.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.585+0000 7f4b42a14700 1 -- 172.21.15.60:0/1219834198 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b3c066160 con 0x7f4b3c05fb00 2022-06-29T15:58:26.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.589+0000 7f4b42a14700 1 -- 172.21.15.60:0/1219834198 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b3c0573d0 con 0x7f4b3c05fb00 2022-06-29T15:58:26.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.589+0000 7f4b2effd700 1 -- 172.21.15.60:0/1219834198 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4b38019030 con 0x7f4b3c05fb00 2022-06-29T15:58:26.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.589+0000 7f4b2effd700 1 --2- 172.21.15.60:0/1219834198 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4b1c053a70 0x7f4b1c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:26.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.589+0000 7f4b2effd700 1 -- 172.21.15.60:0/1219834198 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4b3806b2a0 con 0x7f4b3c05fb00 2022-06-29T15:58:26.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.589+0000 7f4b41211700 1 --2- 172.21.15.60:0/1219834198 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4b1c053a70 0x7f4b1c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:26.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.593+0000 7f4b41211700 1 --2- 172.21.15.60:0/1219834198 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4b1c053a70 0x7f4b1c055f30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f4b3000c000 tx=0x7f4b30008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:26.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.601+0000 7f4b2effd700 1 -- 172.21.15.60:0/1219834198 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4b380420b0 con 0x7f4b3c05fb00 2022-06-29T15:58:26.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.729+0000 7f4b42a14700 1 -- 172.21.15.60:0/1219834198 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f4b3c1554e0 con 0x7f4b3c05fb00 2022-06-29T15:58:26.734 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.729+0000 7f4b2effd700 1 -- 172.21.15.60:0/1219834198 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4b380456d0 con 0x7f4b3c05fb00 2022-06-29T15:58:26.735 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T15:58:26.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.729+0000 7f4b42a14700 1 -- 172.21.15.60:0/1219834198 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4b1c053a70 msgr2=0x7f4b1c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:26.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.729+0000 7f4b42a14700 1 --2- 172.21.15.60:0/1219834198 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4b1c053a70 0x7f4b1c055f30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f4b3000c000 tx=0x7f4b30008040).stop 2022-06-29T15:58:26.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.729+0000 7f4b42a14700 1 -- 172.21.15.60:0/1219834198 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b3c05fb00 msgr2=0x7f4b3c116110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:26.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.729+0000 7f4b42a14700 1 --2- 172.21.15.60:0/1219834198 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b3c05fb00 0x7f4b3c116110 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f4b3c13ce80 tx=0x7f4b38007c50).stop 2022-06-29T15:58:26.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.729+0000 7f4b42a14700 1 -- 172.21.15.60:0/1219834198 shutdown_connections 2022-06-29T15:58:26.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.729+0000 7f4b42a14700 1 --2- 172.21.15.60:0/1219834198 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4b1c053a70 0x7f4b1c055f30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.729+0000 7f4b42a14700 1 --2- 172.21.15.60:0/1219834198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b3c143910 0x7f4b3c155780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.729+0000 7f4b42a14700 1 --2- 172.21.15.60:0/1219834198 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4b3c0ad620 0x7f4b3c13fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.729+0000 7f4b42a14700 1 --2- 172.21.15.60:0/1219834198 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b3c05fb00 0x7f4b3c116110 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.729+0000 7f4b42a14700 1 -- 172.21.15.60:0/1219834198 >> 172.21.15.60:0/1219834198 conn(0x7f4b3c063a60 msgr2=0x7f4b3c05f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:26.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.729+0000 7f4b42a14700 1 -- 172.21.15.60:0/1219834198 shutdown_connections 2022-06-29T15:58:26.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.729+0000 7f4b42a14700 1 -- 172.21.15.60:0/1219834198 wait complete. 2022-06-29T15:58:26.748 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-29T15:58:26.860 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.853+0000 7f1551f32700 1 Processor -- start 2022-06-29T15:58:26.860 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.853+0000 7f1551f32700 1 -- start start 2022-06-29T15:58:26.861 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.857+0000 7f1551f32700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f154c05fb00 0x7f154c0aec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:26.862 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.857+0000 7f1551f32700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f154c067b30 0x7f154c13fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:26.863 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.857+0000 7f1551f32700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f154c140300 0x7f154c0b2060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:26.863 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.857+0000 7f1551f32700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f154c05fee0 con 0x7f154c140300 2022-06-29T15:58:26.863 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.857+0000 7f1551f32700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f154c0b0df0 con 0x7f154c05fb00 2022-06-29T15:58:26.863 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.857+0000 7f1551f32700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f154c0af580 con 0x7f154c067b30 2022-06-29T15:58:26.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.857+0000 7f1551731700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f154c140300 0x7f154c0b2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:26.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.857+0000 7f1551731700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f154c140300 0x7f154c0b2060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45040/0 (socket says 172.21.15.60:45040) 2022-06-29T15:58:26.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.857+0000 7f1551731700 1 -- 172.21.15.60:0/3776810879 learned_addr learned my addr 172.21.15.60:0/3776810879 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:26.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.857+0000 7f154bfff700 1 --2- 172.21.15.60:0/3776810879 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f154c067b30 0x7f154c13fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:26.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.857+0000 7f1550f30700 1 --2- 172.21.15.60:0/3776810879 >> v2:172.21.15.92:3300/0 conn(0x7f154c05fb00 0x7f154c0aec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:26.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.857+0000 7f1551731700 1 -- 172.21.15.60:0/3776810879 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f154c067b30 msgr2=0x7f154c13fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:26.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.857+0000 7f1551731700 1 --2- 172.21.15.60:0/3776810879 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f154c067b30 0x7f154c13fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.857+0000 7f1551731700 1 -- 172.21.15.60:0/3776810879 >> v2:172.21.15.92:3300/0 conn(0x7f154c05fb00 msgr2=0x7f154c0aec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:26.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.857+0000 7f1551731700 1 --2- 172.21.15.60:0/3776810879 >> v2:172.21.15.92:3300/0 conn(0x7f154c05fb00 0x7f154c0aec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.857+0000 7f1551731700 1 -- 172.21.15.60:0/3776810879 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f154c0af820 con 0x7f154c140300 2022-06-29T15:58:26.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.857+0000 7f1550f30700 1 --2- 172.21.15.60:0/3776810879 >> v2:172.21.15.92:3300/0 conn(0x7f154c05fb00 0x7f154c0aec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:26.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.857+0000 7f1551731700 1 --2- 172.21.15.60:0/3776810879 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f154c140300 0x7f154c0b2060 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f15440032f0 tx=0x7f1544009290).ready entity=mon.0 client_cookie=c80e3d309d70df37 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:26.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.857+0000 7f154b7fe700 1 -- 172.21.15.60:0/3776810879 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f154400b680 con 0x7f154c140300 2022-06-29T15:58:26.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.857+0000 7f154b7fe700 1 -- 172.21.15.60:0/3776810879 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f154400b7e0 con 0x7f154c140300 2022-06-29T15:58:26.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.857+0000 7f154b7fe700 1 -- 172.21.15.60:0/3776810879 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f154400b950 con 0x7f154c140300 2022-06-29T15:58:26.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.857+0000 7f1551f32700 1 -- 172.21.15.60:0/3776810879 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f154c140300 msgr2=0x7f154c0b2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:26.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.857+0000 7f1551f32700 1 --2- 172.21.15.60:0/3776810879 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f154c140300 0x7f154c0b2060 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f15440032f0 tx=0x7f1544009290).stop 2022-06-29T15:58:26.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.857+0000 7f1551f32700 1 -- 172.21.15.60:0/3776810879 shutdown_connections 2022-06-29T15:58:26.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.857+0000 7f1551f32700 1 --2- 172.21.15.60:0/3776810879 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f154c140300 0x7f154c0b2060 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.857+0000 7f1551f32700 1 --2- 172.21.15.60:0/3776810879 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f154c067b30 0x7f154c13fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.857+0000 7f1551f32700 1 --2- 172.21.15.60:0/3776810879 >> v2:172.21.15.92:3300/0 conn(0x7f154c05fb00 0x7f154c0aec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.857+0000 7f1551f32700 1 -- 172.21.15.60:0/3776810879 >> 172.21.15.60:0/3776810879 conn(0x7f154c063a60 msgr2=0x7f154c05f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:26.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.857+0000 7f1551f32700 1 -- 172.21.15.60:0/3776810879 shutdown_connections 2022-06-29T15:58:26.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.857+0000 7f1551f32700 1 -- 172.21.15.60:0/3776810879 wait complete. 2022-06-29T15:58:26.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.861+0000 7f1551f32700 1 Processor -- start 2022-06-29T15:58:26.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.861+0000 7f1551f32700 1 -- start start 2022-06-29T15:58:26.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.861+0000 7f1551f32700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f154c05fb00 0x7f154c143a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:26.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.861+0000 7f1551f32700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f154c140300 0x7f154c0acd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:26.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.861+0000 7f1551f32700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f154c142b10 0x7f154c1446e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:26.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.861+0000 7f1551f32700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154c066070 con 0x7f154c140300 2022-06-29T15:58:26.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.861+0000 7f1551f32700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154c0651f0 con 0x7f154c05fb00 2022-06-29T15:58:26.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.861+0000 7f1551f32700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f154c1443d0 con 0x7f154c142b10 2022-06-29T15:58:26.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.861+0000 7f154bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f154c140300 0x7f154c0acd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:26.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.861+0000 7f154bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f154c140300 0x7f154c0acd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45044/0 (socket says 172.21.15.60:45044) 2022-06-29T15:58:26.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.861+0000 7f154bfff700 1 -- 172.21.15.60:0/1208753527 learned_addr learned my addr 172.21.15.60:0/1208753527 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:26.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.861+0000 7f1550f30700 1 --2- 172.21.15.60:0/1208753527 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f154c05fb00 0x7f154c143a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:26.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.861+0000 7f1551731700 1 --2- 172.21.15.60:0/1208753527 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f154c142b10 0x7f154c1446e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:26.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.861+0000 7f154bfff700 1 -- 172.21.15.60:0/1208753527 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f154c142b10 msgr2=0x7f154c1446e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:26.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.861+0000 7f154bfff700 1 --2- 172.21.15.60:0/1208753527 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f154c142b10 0x7f154c1446e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.861+0000 7f154bfff700 1 -- 172.21.15.60:0/1208753527 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f154c05fb00 msgr2=0x7f154c143a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:26.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.861+0000 7f154bfff700 1 --2- 172.21.15.60:0/1208753527 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f154c05fb00 0x7f154c143a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:26.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.861+0000 7f154bfff700 1 -- 172.21.15.60:0/1208753527 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f154000b180 con 0x7f154c140300 2022-06-29T15:58:26.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.861+0000 7f1551731700 1 --2- 172.21.15.60:0/1208753527 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f154c142b10 0x7f154c1446e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:58:26.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.861+0000 7f154bfff700 1 --2- 172.21.15.60:0/1208753527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f154c140300 0x7f154c0acd60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f153800c000 tx=0x7f1538006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:26.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.861+0000 7f1549ffb700 1 -- 172.21.15.60:0/1208753527 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1538010780 con 0x7f154c140300 2022-06-29T15:58:26.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.861+0000 7f1551f32700 1 -- 172.21.15.60:0/1208753527 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1544009530 con 0x7f154c140300 2022-06-29T15:58:26.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.861+0000 7f1549ffb700 1 -- 172.21.15.60:0/1208753527 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15380108e0 con 0x7f154c140300 2022-06-29T15:58:26.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.861+0000 7f1549ffb700 1 -- 172.21.15.60:0/1208753527 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1538010c40 con 0x7f154c140300 2022-06-29T15:58:26.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.861+0000 7f1551f32700 1 -- 172.21.15.60:0/1208753527 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f154c066d20 con 0x7f154c140300 2022-06-29T15:58:26.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.865+0000 7f1549ffb700 1 -- 172.21.15.60:0/1208753527 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f153801c460 con 0x7f154c140300 2022-06-29T15:58:26.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.865+0000 7f1549ffb700 1 --2- 172.21.15.60:0/1208753527 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f152c053b30 0x7f152c055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:26.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.865+0000 7f1549ffb700 1 -- 172.21.15.60:0/1208753527 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f153806c370 con 0x7f154c140300 2022-06-29T15:58:26.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.865+0000 7f1551f32700 1 -- 172.21.15.60:0/1208753527 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f154c066ee0 con 0x7f154c140300 2022-06-29T15:58:26.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.865+0000 7f1550f30700 1 --2- 172.21.15.60:0/1208753527 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f152c053b30 0x7f152c055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:26.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.865+0000 7f1550f30700 1 --2- 172.21.15.60:0/1208753527 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f152c053b30 0x7f152c055ff0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f154000b000 tx=0x7f1540006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:26.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.869+0000 7f1549ffb700 1 -- 172.21.15.60:0/1208753527 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1538043190 con 0x7f154c140300 2022-06-29T15:58:27.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.993+0000 7f1551f32700 1 -- 172.21.15.60:0/1208753527 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f154c0573d0 con 0x7f154c140300 2022-06-29T15:58:27.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.993+0000 7f1549ffb700 1 -- 172.21.15.60:0/1208753527 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7f15380467b0 con 0x7f154c140300 2022-06-29T15:58:27.001 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T15:58:27.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.997+0000 7f1551f32700 1 -- 172.21.15.60:0/1208753527 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f152c053b30 msgr2=0x7f152c055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:27.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.997+0000 7f1551f32700 1 --2- 172.21.15.60:0/1208753527 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f152c053b30 0x7f152c055ff0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f154000b000 tx=0x7f1540006df0).stop 2022-06-29T15:58:27.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.997+0000 7f1551f32700 1 -- 172.21.15.60:0/1208753527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f154c140300 msgr2=0x7f154c0acd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:27.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.997+0000 7f1551f32700 1 --2- 172.21.15.60:0/1208753527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f154c140300 0x7f154c0acd60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f153800c000 tx=0x7f1538006df0).stop 2022-06-29T15:58:27.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.997+0000 7f1551f32700 1 -- 172.21.15.60:0/1208753527 shutdown_connections 2022-06-29T15:58:27.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.997+0000 7f1551f32700 1 --2- 172.21.15.60:0/1208753527 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f152c053b30 0x7f152c055ff0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.997+0000 7f1551f32700 1 --2- 172.21.15.60:0/1208753527 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f154c142b10 0x7f154c1446e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.997+0000 7f1551f32700 1 --2- 172.21.15.60:0/1208753527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f154c140300 0x7f154c0acd60 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.997+0000 7f1551f32700 1 --2- 172.21.15.60:0/1208753527 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f154c05fb00 0x7f154c143a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.997+0000 7f1551f32700 1 -- 172.21.15.60:0/1208753527 >> 172.21.15.60:0/1208753527 conn(0x7f154c063a60 msgr2=0x7f154c13fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:27.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.997+0000 7f1551f32700 1 -- 172.21.15.60:0/1208753527 shutdown_connections 2022-06-29T15:58:27.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:26.997+0000 7f1551f32700 1 -- 172.21.15.60:0/1208753527 wait complete. 2022-06-29T15:58:27.014 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-29T15:58:27.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.117+0000 7f756dc5c700 1 Processor -- start 2022-06-29T15:58:27.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.117+0000 7f756dc5c700 1 -- start start 2022-06-29T15:58:27.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.117+0000 7f756dc5c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f756805fb00 0x7f75680ae890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:27.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.117+0000 7f756dc5c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75680b1c00 0x7f7568067250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:27.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.117+0000 7f756dc5c700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75680ad240 con 0x7f75680b0fa0 2022-06-29T15:58:27.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.117+0000 7f756dc5c700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f756813ce60 con 0x7f756805fb00 2022-06-29T15:58:27.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.117+0000 7f756dc5c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f756805fee0 con 0x7f75680b1c00 2022-06-29T15:58:27.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.117+0000 7f756cc5a700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f756805fb00 0x7f75680ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:27.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.117+0000 7f756cc5a700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f756805fb00 0x7f75680ae890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45050/0 (socket says 172.21.15.60:45050) 2022-06-29T15:58:27.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.117+0000 7f756cc5a700 1 -- 172.21.15.60:0/407695700 learned_addr learned my addr 172.21.15.60:0/407695700 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:27.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.117+0000 7f756d45b700 1 --2- 172.21.15.60:0/407695700 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75680b1c00 0x7f7568067250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:27.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.117+0000 7f756cc5a700 1 -- 172.21.15.60:0/407695700 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75680b1c00 msgr2=0x7f7568067250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:27.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.117+0000 7f756cc5a700 1 --2- 172.21.15.60:0/407695700 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75680b1c00 0x7f7568067250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.117+0000 7f756cc5a700 1 -- 172.21.15.60:0/407695700 >> v1:172.21.15.92:6789/0 conn(0x7f75680b0fa0 legacy=0x7f75680b14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:27.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.117+0000 7f756cc5a700 1 -- 172.21.15.60:0/407695700 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7568143180 con 0x7f756805fb00 2022-06-29T15:58:27.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.117+0000 7f756d45b700 1 --2- 172.21.15.60:0/407695700 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75680b1c00 0x7f7568067250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:58:27.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.117+0000 7f756cc5a700 1 --2- 172.21.15.60:0/407695700 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f756805fb00 0x7f75680ae890 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f755c002f70 tx=0x7f755c006f40).ready entity=mon.0 client_cookie=1c53e4041dee9583 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:27.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.117+0000 7f75677fe700 1 -- 172.21.15.60:0/407695700 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f755c00c900 con 0x7f756805fb00 2022-06-29T15:58:27.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.121+0000 7f75677fe700 1 -- 172.21.15.60:0/407695700 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f755c000e90 con 0x7f756805fb00 2022-06-29T15:58:27.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.121+0000 7f75677fe700 1 -- 172.21.15.60:0/407695700 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f755c00cc90 con 0x7f756805fb00 2022-06-29T15:58:27.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.121+0000 7f756dc5c700 1 -- 172.21.15.60:0/407695700 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f756805fb00 msgr2=0x7f75680ae890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:27.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.121+0000 7f756dc5c700 1 --2- 172.21.15.60:0/407695700 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f756805fb00 0x7f75680ae890 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f755c002f70 tx=0x7f755c006f40).stop 2022-06-29T15:58:27.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.121+0000 7f756dc5c700 1 -- 172.21.15.60:0/407695700 shutdown_connections 2022-06-29T15:58:27.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.121+0000 7f756dc5c700 1 --2- 172.21.15.60:0/407695700 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75680b1c00 0x7f7568067250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.121+0000 7f756dc5c700 1 --2- 172.21.15.60:0/407695700 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f756805fb00 0x7f75680ae890 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.121+0000 7f756dc5c700 1 -- 172.21.15.60:0/407695700 >> 172.21.15.60:0/407695700 conn(0x7f7568063a60 msgr2=0x7f756805f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:27.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.121+0000 7f756dc5c700 1 -- 172.21.15.60:0/407695700 shutdown_connections 2022-06-29T15:58:27.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.121+0000 7f756dc5c700 1 -- 172.21.15.60:0/407695700 wait complete. 2022-06-29T15:58:27.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.121+0000 7f756dc5c700 1 Processor -- start 2022-06-29T15:58:27.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.121+0000 7f756dc5c700 1 -- start start 2022-06-29T15:58:27.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.121+0000 7f756dc5c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f756805fb00 0x7f7568115780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:27.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.121+0000 7f756dc5c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75680b0fa0 0x7f7568115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:27.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.121+0000 7f756dc5c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75680b1c00 0x7f756813b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:27.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.121+0000 7f756dc5c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7568066fc0 con 0x7f75680b0fa0 2022-06-29T15:58:27.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.121+0000 7f756dc5c700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7568156070 con 0x7f75680b1c00 2022-06-29T15:58:27.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.121+0000 7f756dc5c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7568156310 con 0x7f756805fb00 2022-06-29T15:58:27.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.121+0000 7f7567fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75680b0fa0 0x7f7568115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:27.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.121+0000 7f7567fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75680b0fa0 0x7f7568115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45056/0 (socket says 172.21.15.60:45056) 2022-06-29T15:58:27.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.121+0000 7f7567fff700 1 -- 172.21.15.60:0/477595376 learned_addr learned my addr 172.21.15.60:0/477595376 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:27.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.121+0000 7f756cc5a700 1 --2- 172.21.15.60:0/477595376 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f756805fb00 0x7f7568115780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:27.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.121+0000 7f756d45b700 1 --2- 172.21.15.60:0/477595376 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75680b1c00 0x7f756813b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:27.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.121+0000 7f7567fff700 1 -- 172.21.15.60:0/477595376 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f756805fb00 msgr2=0x7f7568115780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:27.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.121+0000 7f7567fff700 1 --2- 172.21.15.60:0/477595376 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f756805fb00 0x7f7568115780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.121+0000 7f7567fff700 1 -- 172.21.15.60:0/477595376 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75680b1c00 msgr2=0x7f756813b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:27.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.121+0000 7f7567fff700 1 --2- 172.21.15.60:0/477595376 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75680b1c00 0x7f756813b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.121+0000 7f7567fff700 1 -- 172.21.15.60:0/477595376 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f755c00a360 con 0x7f75680b0fa0 2022-06-29T15:58:27.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.121+0000 7f7567fff700 1 --2- 172.21.15.60:0/477595376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75680b0fa0 0x7f7568115c70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f7568062600 tx=0x7f7554007be0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:27.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.125+0000 7f7565ffb700 1 -- 172.21.15.60:0/477595376 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7554007030 con 0x7f75680b0fa0 2022-06-29T15:58:27.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.125+0000 7f7565ffb700 1 -- 172.21.15.60:0/477595376 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7554007190 con 0x7f75680b0fa0 2022-06-29T15:58:27.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.125+0000 7f7565ffb700 1 -- 172.21.15.60:0/477595376 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7554006510 con 0x7f75680b0fa0 2022-06-29T15:58:27.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.125+0000 7f756dc5c700 1 -- 172.21.15.60:0/477595376 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7568066cf0 con 0x7f75680b0fa0 2022-06-29T15:58:27.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.125+0000 7f756dc5c700 1 -- 172.21.15.60:0/477595376 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75680675d0 con 0x7f75680b0fa0 2022-06-29T15:58:27.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.125+0000 7f756dc5c700 1 -- 172.21.15.60:0/477595376 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75680573d0 con 0x7f75680b0fa0 2022-06-29T15:58:27.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.125+0000 7f7565ffb700 1 -- 172.21.15.60:0/477595376 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7554006670 con 0x7f75680b0fa0 2022-06-29T15:58:27.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.125+0000 7f7565ffb700 1 --2- 172.21.15.60:0/477595376 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7548053b00 0x7f7548055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:27.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.125+0000 7f7565ffb700 1 -- 172.21.15.60:0/477595376 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f755406b840 con 0x7f75680b0fa0 2022-06-29T15:58:27.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.125+0000 7f756cc5a700 1 --2- 172.21.15.60:0/477595376 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7548053b00 0x7f7548055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:27.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.129+0000 7f756cc5a700 1 --2- 172.21.15.60:0/477595376 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7548053b00 0x7f7548055fc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f755c01b000 tx=0x7f75680ad130).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:27.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.133+0000 7f7565ffb700 1 -- 172.21.15.60:0/477595376 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f75540425f0 con 0x7f75680b0fa0 2022-06-29T15:58:27.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.257+0000 7f756dc5c700 1 -- 172.21.15.60:0/477595376 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7568068560 con 0x7f75680b0fa0 2022-06-29T15:58:27.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.257+0000 7f7565ffb700 1 -- 172.21.15.60:0/477595376 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7f7554045c10 con 0x7f75680b0fa0 2022-06-29T15:58:27.264 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T15:58:27.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.261+0000 7f756dc5c700 1 -- 172.21.15.60:0/477595376 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7548053b00 msgr2=0x7f7548055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:27.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.261+0000 7f756dc5c700 1 --2- 172.21.15.60:0/477595376 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7548053b00 0x7f7548055fc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f755c01b000 tx=0x7f75680ad130).stop 2022-06-29T15:58:27.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.261+0000 7f756dc5c700 1 -- 172.21.15.60:0/477595376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75680b0fa0 msgr2=0x7f7568115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:27.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.261+0000 7f756dc5c700 1 --2- 172.21.15.60:0/477595376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75680b0fa0 0x7f7568115c70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f7568062600 tx=0x7f7554007be0).stop 2022-06-29T15:58:27.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.261+0000 7f756dc5c700 1 -- 172.21.15.60:0/477595376 shutdown_connections 2022-06-29T15:58:27.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.261+0000 7f756dc5c700 1 --2- 172.21.15.60:0/477595376 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7548053b00 0x7f7548055fc0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.261+0000 7f756dc5c700 1 --2- 172.21.15.60:0/477595376 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75680b1c00 0x7f756813b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.261+0000 7f756dc5c700 1 --2- 172.21.15.60:0/477595376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75680b0fa0 0x7f7568115c70 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.261+0000 7f756dc5c700 1 --2- 172.21.15.60:0/477595376 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f756805fb00 0x7f7568115780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.261+0000 7f756dc5c700 1 -- 172.21.15.60:0/477595376 >> 172.21.15.60:0/477595376 conn(0x7f7568063a60 msgr2=0x7f75680b1470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:27.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.261+0000 7f756dc5c700 1 -- 172.21.15.60:0/477595376 shutdown_connections 2022-06-29T15:58:27.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.261+0000 7f756dc5c700 1 -- 172.21.15.60:0/477595376 wait complete. 2022-06-29T15:58:27.277 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-06-29T15:58:27.277 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2022-06-29T15:58:27.278 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2022-06-29T15:58:27.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.373+0000 7fb8e085e700 1 Processor -- start 2022-06-29T15:58:27.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.373+0000 7fb8e085e700 1 -- start start 2022-06-29T15:58:27.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.373+0000 7fb8e085e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb8dc0b5eb0 0x7fb8dc142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:27.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.373+0000 7fb8e085e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb8dc0b2c50 0x7fb8dc0b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:27.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.373+0000 7fb8e085e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8dc0b0cf0 con 0x7fb8dc0b48b0 2022-06-29T15:58:27.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.373+0000 7fb8e085e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8dc0b54e0 con 0x7fb8dc0b2c50 2022-06-29T15:58:27.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.373+0000 7fb8e085e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8dc0b4d10 con 0x7fb8dc0b5eb0 2022-06-29T15:58:27.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.373+0000 7fb8db59e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb8dc0b2c50 0x7fb8dc0b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:27.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.373+0000 7fb8da59c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb8dc0b48b0 0x7fb8dc0b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53152/0 (socket says 172.21.15.60:53152) 2022-06-29T15:58:27.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.373+0000 7fb8da59c700 1 -- 172.21.15.60:0/2583204321 learned_addr learned my addr 172.21.15.60:0/2583204321 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:27.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.377+0000 7fb8dad9d700 1 --2- 172.21.15.60:0/2583204321 >> v2:172.21.15.92:3300/0 conn(0x7fb8dc0b5eb0 0x7fb8dc142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:27.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.377+0000 7fb8db59e700 1 -- 172.21.15.60:0/2583204321 >> v2:172.21.15.92:3300/0 conn(0x7fb8dc0b5eb0 msgr2=0x7fb8dc142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:27.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.377+0000 7fb8db59e700 1 --2- 172.21.15.60:0/2583204321 >> v2:172.21.15.92:3300/0 conn(0x7fb8dc0b5eb0 0x7fb8dc142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.377+0000 7fb8db59e700 1 -- 172.21.15.60:0/2583204321 >> v1:172.21.15.60:6789/0 conn(0x7fb8dc0b48b0 legacy=0x7fb8dc0b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:27.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.377+0000 7fb8d9d9b700 1 -- 172.21.15.60:0/2583204321 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333936855 0 0) 0x7fb8d0001110 con 0x7fb8dc0b48b0 2022-06-29T15:58:27.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.377+0000 7fb8db59e700 1 -- 172.21.15.60:0/2583204321 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8dc0b6290 con 0x7fb8dc0b2c50 2022-06-29T15:58:27.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.377+0000 7fb8db59e700 1 --2- 172.21.15.60:0/2583204321 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8dc0b2c50 0x7fb8dc0b30c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fb8cc002f70 tx=0x7fb8cc006df0).ready entity=mon.0 client_cookie=6538c2dd4e211af4 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:27.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.377+0000 7fb8d9d9b700 1 -- 172.21.15.60:0/2583204321 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8cc00d570 con 0x7fb8dc0b2c50 2022-06-29T15:58:27.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.377+0000 7fb8d9d9b700 1 -- 172.21.15.60:0/2583204321 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8cc000e90 con 0x7fb8dc0b2c50 2022-06-29T15:58:27.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.377+0000 7fb8d9d9b700 1 -- 172.21.15.60:0/2583204321 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8cc00d8c0 con 0x7fb8dc0b2c50 2022-06-29T15:58:27.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.377+0000 7fb8e085e700 1 -- 172.21.15.60:0/2583204321 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8dc0b2c50 msgr2=0x7fb8dc0b30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:27.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.377+0000 7fb8e085e700 1 --2- 172.21.15.60:0/2583204321 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8dc0b2c50 0x7fb8dc0b30c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fb8cc002f70 tx=0x7fb8cc006df0).stop 2022-06-29T15:58:27.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.377+0000 7fb8e085e700 1 -- 172.21.15.60:0/2583204321 shutdown_connections 2022-06-29T15:58:27.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.377+0000 7fb8e085e700 1 --2- 172.21.15.60:0/2583204321 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8dc0b2c50 0x7fb8dc0b30c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.377+0000 7fb8e085e700 1 --2- 172.21.15.60:0/2583204321 >> v2:172.21.15.92:3300/0 conn(0x7fb8dc0b5eb0 0x7fb8dc142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.377+0000 7fb8e085e700 1 -- 172.21.15.60:0/2583204321 >> 172.21.15.60:0/2583204321 conn(0x7fb8dc062090 msgr2=0x7fb8dc062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:27.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.377+0000 7fb8e085e700 1 -- 172.21.15.60:0/2583204321 shutdown_connections 2022-06-29T15:58:27.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.377+0000 7fb8e085e700 1 -- 172.21.15.60:0/2583204321 wait complete. 2022-06-29T15:58:27.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.377+0000 7fb8e085e700 1 Processor -- start 2022-06-29T15:58:27.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.377+0000 7fb8e085e700 1 -- start start 2022-06-29T15:58:27.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.381+0000 7fb8e085e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8dc0b2c50 0x7fb8dc146bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:27.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.381+0000 7fb8e085e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8dc0b48b0 0x7fb8dc149b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:27.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.381+0000 7fb8e085e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8dc0b5eb0 0x7fb8dc145a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:27.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.381+0000 7fb8e085e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8dc113710 con 0x7fb8dc0b5eb0 2022-06-29T15:58:27.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.381+0000 7fb8e085e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8dc155330 con 0x7fb8dc0b2c50 2022-06-29T15:58:27.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.381+0000 7fb8e085e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8dc113470 con 0x7fb8dc0b48b0 2022-06-29T15:58:27.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.381+0000 7fb8db59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8dc0b5eb0 0x7fb8dc145a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:27.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.381+0000 7fb8db59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8dc0b5eb0 0x7fb8dc145a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45072/0 (socket says 172.21.15.60:45072) 2022-06-29T15:58:27.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.381+0000 7fb8db59e700 1 -- 172.21.15.60:0/424220731 learned_addr learned my addr 172.21.15.60:0/424220731 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:27.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.381+0000 7fb8da59c700 1 --2- 172.21.15.60:0/424220731 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8dc0b48b0 0x7fb8dc149b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:27.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.381+0000 7fb8dad9d700 1 --2- 172.21.15.60:0/424220731 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8dc0b2c50 0x7fb8dc146bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:27.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.381+0000 7fb8db59e700 1 -- 172.21.15.60:0/424220731 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8dc0b48b0 msgr2=0x7fb8dc149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:27.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.381+0000 7fb8db59e700 1 --2- 172.21.15.60:0/424220731 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8dc0b48b0 0x7fb8dc149b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.381+0000 7fb8db59e700 1 -- 172.21.15.60:0/424220731 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8dc0b2c50 msgr2=0x7fb8dc146bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:27.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.381+0000 7fb8db59e700 1 --2- 172.21.15.60:0/424220731 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8dc0b2c50 0x7fb8dc146bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.381+0000 7fb8db59e700 1 -- 172.21.15.60:0/424220731 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8cc00b180 con 0x7fb8dc0b5eb0 2022-06-29T15:58:27.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.381+0000 7fb8db59e700 1 --2- 172.21.15.60:0/424220731 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8dc0b5eb0 0x7fb8dc145a00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fb8cc002f70 tx=0x7fb8cc014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:27.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.381+0000 7fb8cbfff700 1 -- 172.21.15.60:0/424220731 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8cc016070 con 0x7fb8dc0b5eb0 2022-06-29T15:58:27.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.381+0000 7fb8cbfff700 1 -- 172.21.15.60:0/424220731 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8cc013850 con 0x7fb8dc0b5eb0 2022-06-29T15:58:27.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.381+0000 7fb8e085e700 1 -- 172.21.15.60:0/424220731 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8dc145ef0 con 0x7fb8dc0b5eb0 2022-06-29T15:58:27.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.381+0000 7fb8cbfff700 1 -- 172.21.15.60:0/424220731 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8cc013bb0 con 0x7fb8dc0b5eb0 2022-06-29T15:58:27.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.381+0000 7fb8e085e700 1 -- 172.21.15.60:0/424220731 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8dc144600 con 0x7fb8dc0b5eb0 2022-06-29T15:58:27.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.381+0000 7fb8e085e700 1 -- 172.21.15.60:0/424220731 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8dc1554f0 con 0x7fb8dc0b5eb0 2022-06-29T15:58:27.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.381+0000 7fb8cbfff700 1 -- 172.21.15.60:0/424220731 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb8cc025030 con 0x7fb8dc0b5eb0 2022-06-29T15:58:27.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.385+0000 7fb8cbfff700 1 --2- 172.21.15.60:0/424220731 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb8b8053a70 0x7fb8b8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:27.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.385+0000 7fb8cbfff700 1 -- 172.21.15.60:0/424220731 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb8cc06cff0 con 0x7fb8dc0b5eb0 2022-06-29T15:58:27.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.385+0000 7fb8dad9d700 1 --2- 172.21.15.60:0/424220731 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb8b8053a70 0x7fb8b8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:27.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.385+0000 7fb8dad9d700 1 --2- 172.21.15.60:0/424220731 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb8b8053a70 0x7fb8b8055f30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fb8d400b000 tx=0x7fb8d40091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:27.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.389+0000 7fb8cbfff700 1 -- 172.21.15.60:0/424220731 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb8dc1554f0 con 0x7fb8dc0b5eb0 2022-06-29T15:58:27.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.545+0000 7fb8e085e700 1 -- 172.21.15.60:0/424220731 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fb8dc1554f0 con 0x7fb8dc0b5eb0 2022-06-29T15:58:27.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.549+0000 7fb8cbfff700 1 -- 172.21.15.60:0/424220731 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v4) v1 ==== 74+0+131776 (secure 0 0 0) 0x7fb8dc1554f0 con 0x7fb8dc0b5eb0 2022-06-29T15:58:27.559 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:58:27.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.553+0000 7fb8e085e700 1 -- 172.21.15.60:0/424220731 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb8b8053a70 msgr2=0x7fb8b8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:27.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.553+0000 7fb8e085e700 1 --2- 172.21.15.60:0/424220731 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb8b8053a70 0x7fb8b8055f30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fb8d400b000 tx=0x7fb8d40091d0).stop 2022-06-29T15:58:27.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.553+0000 7fb8e085e700 1 -- 172.21.15.60:0/424220731 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8dc0b5eb0 msgr2=0x7fb8dc145a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:27.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.553+0000 7fb8e085e700 1 --2- 172.21.15.60:0/424220731 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8dc0b5eb0 0x7fb8dc145a00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fb8cc002f70 tx=0x7fb8cc014000).stop 2022-06-29T15:58:27.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.553+0000 7fb8e085e700 1 -- 172.21.15.60:0/424220731 shutdown_connections 2022-06-29T15:58:27.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.553+0000 7fb8e085e700 1 --2- 172.21.15.60:0/424220731 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb8b8053a70 0x7fb8b8055f30 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.553+0000 7fb8e085e700 1 --2- 172.21.15.60:0/424220731 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8dc0b5eb0 0x7fb8dc145a00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.553+0000 7fb8e085e700 1 --2- 172.21.15.60:0/424220731 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8dc0b48b0 0x7fb8dc149b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.553+0000 7fb8e085e700 1 --2- 172.21.15.60:0/424220731 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8dc0b2c50 0x7fb8dc146bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.553+0000 7fb8e085e700 1 -- 172.21.15.60:0/424220731 >> 172.21.15.60:0/424220731 conn(0x7fb8dc062090 msgr2=0x7fb8dc143570 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:27.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.553+0000 7fb8e085e700 1 -- 172.21.15.60:0/424220731 shutdown_connections 2022-06-29T15:58:27.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.553+0000 7fb8e085e700 1 -- 172.21.15.60:0/424220731 wait complete. 2022-06-29T15:58:27.573 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":4,"active_gid":4103,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":12872},{"type":"v1","addr":"172.21.15.92:6833","nonce":12872}]},"active_addr":"172.21.15.92:6833/12872","active_change":"2022-06-29T15:58:10.903372+0000","active_mgr_features":4540138292840890367,"available":true,"standbys":[{"gid":4114,"name":"y","mgr_features":4540138292840890367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.92:0","nonce":1331235526}]},{"addrvec":[{"type":"v2","addr":"172.21.15.92:0","nonce":2731129411}]},{"addrvec":[{"type":"v2","addr":"172.21.15.92:0","nonce":2739161638}]}]}} 2022-06-29T15:58:27.579 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2022-06-29T15:58:27.579 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2022-06-29T15:58:27.579 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T15:58:27.590 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T15:58:27.627 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T15:58:27.630 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T15:58:27.633 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T15:58:27.636 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T15:58:27.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.681+0000 7f0b8892f700 1 Processor -- start 2022-06-29T15:58:27.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.681+0000 7f0b8892f700 1 -- start start 2022-06-29T15:58:27.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.681+0000 7f0b8892f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b840b28b0 0x7f0b8413f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:27.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.681+0000 7f0b8892f700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0b841438a0 0x7f0b840b2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:27.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.681+0000 7f0b8892f700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b84142d60 con 0x7f0b84060650 2022-06-29T15:58:27.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.681+0000 7f0b8892f700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b840b3180 con 0x7f0b841438a0 2022-06-29T15:58:27.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.681+0000 7f0b8892f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b840654f0 con 0x7f0b840b28b0 2022-06-29T15:58:27.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.681+0000 7f0b83fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0b841438a0 0x7f0b840b2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:27.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.681+0000 7f0b837fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b840b28b0 0x7f0b8413f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:27.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.681+0000 7f0b7bfff700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f0b84060650 0x7f0b84143190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:57412/0 (socket says 172.21.15.60:57412) 2022-06-29T15:58:27.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.681+0000 7f0b7bfff700 1 -- 172.21.15.60:0/2451639390 learned_addr learned my addr 172.21.15.60:0/2451639390 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:27.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.681+0000 7f0b837fe700 1 -- 172.21.15.60:0/2451639390 >> v2:172.21.15.92:3300/0 conn(0x7f0b841438a0 msgr2=0x7f0b840b2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:27.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.681+0000 7f0b837fe700 1 --2- 172.21.15.60:0/2451639390 >> v2:172.21.15.92:3300/0 conn(0x7f0b841438a0 0x7f0b840b2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.696 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.681+0000 7f0b837fe700 1 -- 172.21.15.60:0/2451639390 >> v1:172.21.15.92:6789/0 conn(0x7f0b84060650 legacy=0x7f0b84143190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:27.696 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.681+0000 7f0b837fe700 1 -- 172.21.15.60:0/2451639390 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b84065790 con 0x7f0b840b28b0 2022-06-29T15:58:27.696 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.681+0000 7f0b82ffd700 1 -- 172.21.15.60:0/2451639390 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3874402029 0 0) 0x7f0b84142d60 con 0x7f0b84060650 2022-06-29T15:58:27.696 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.681+0000 7f0b83fff700 1 --2- 172.21.15.60:0/2451639390 >> v2:172.21.15.92:3300/0 conn(0x7f0b841438a0 0x7f0b840b2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:27.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.681+0000 7f0b837fe700 1 --2- 172.21.15.60:0/2451639390 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b840b28b0 0x7f0b8413f690 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f0b74002f70 tx=0x7f0b74006df0).ready entity=mon.2 client_cookie=c430b17020e2f41b server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:27.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.681+0000 7f0b82ffd700 1 -- 172.21.15.60:0/2451639390 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b7400d570 con 0x7f0b840b28b0 2022-06-29T15:58:27.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.681+0000 7f0b82ffd700 1 -- 172.21.15.60:0/2451639390 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b74000e90 con 0x7f0b840b28b0 2022-06-29T15:58:27.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.681+0000 7f0b82ffd700 1 -- 172.21.15.60:0/2451639390 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b7400d8c0 con 0x7f0b840b28b0 2022-06-29T15:58:27.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.685+0000 7f0b8892f700 1 -- 172.21.15.60:0/2451639390 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b840b28b0 msgr2=0x7f0b8413f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:27.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.685+0000 7f0b8892f700 1 --2- 172.21.15.60:0/2451639390 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b840b28b0 0x7f0b8413f690 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f0b74002f70 tx=0x7f0b74006df0).stop 2022-06-29T15:58:27.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.685+0000 7f0b8892f700 1 -- 172.21.15.60:0/2451639390 shutdown_connections 2022-06-29T15:58:27.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.685+0000 7f0b8892f700 1 --2- 172.21.15.60:0/2451639390 >> v2:172.21.15.92:3300/0 conn(0x7f0b841438a0 0x7f0b840b2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.685+0000 7f0b8892f700 1 --2- 172.21.15.60:0/2451639390 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b840b28b0 0x7f0b8413f690 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.685+0000 7f0b8892f700 1 -- 172.21.15.60:0/2451639390 >> 172.21.15.60:0/2451639390 conn(0x7f0b840637c0 msgr2=0x7f0b84063ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:27.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.685+0000 7f0b8892f700 1 -- 172.21.15.60:0/2451639390 shutdown_connections 2022-06-29T15:58:27.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.685+0000 7f0b8892f700 1 -- 172.21.15.60:0/2451639390 wait complete. 2022-06-29T15:58:27.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.685+0000 7f0b8892f700 1 Processor -- start 2022-06-29T15:58:27.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.685+0000 7f0b8892f700 1 -- start start 2022-06-29T15:58:27.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.685+0000 7f0b8892f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b84060650 0x7f0b84151e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:27.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.685+0000 7f0b8892f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b840b28b0 0x7f0b8413ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:27.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.685+0000 7f0b8892f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b841438a0 0x7f0b8413dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:27.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.689+0000 7f0b8892f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8413b520 con 0x7f0b840b28b0 2022-06-29T15:58:27.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.689+0000 7f0b8892f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8413b7c0 con 0x7f0b84060650 2022-06-29T15:58:27.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.689+0000 7f0b8892f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8413ba60 con 0x7f0b841438a0 2022-06-29T15:58:27.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.689+0000 7f0b7bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b840b28b0 0x7f0b8413ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:27.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.689+0000 7f0b837fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b84060650 0x7f0b84151e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:27.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.689+0000 7f0b7bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b840b28b0 0x7f0b8413ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45086/0 (socket says 172.21.15.60:45086) 2022-06-29T15:58:27.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.689+0000 7f0b83fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b841438a0 0x7f0b8413dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:27.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.689+0000 7f0b7bfff700 1 -- 172.21.15.60:0/3502398742 learned_addr learned my addr 172.21.15.60:0/3502398742 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:27.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.689+0000 7f0b83fff700 1 -- 172.21.15.60:0/3502398742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b84060650 msgr2=0x7f0b84151e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:27.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.689+0000 7f0b83fff700 1 --2- 172.21.15.60:0/3502398742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b84060650 0x7f0b84151e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.689+0000 7f0b83fff700 1 -- 172.21.15.60:0/3502398742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b840b28b0 msgr2=0x7f0b8413ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:27.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.689+0000 7f0b83fff700 1 --2- 172.21.15.60:0/3502398742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b840b28b0 0x7f0b8413ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.689+0000 7f0b83fff700 1 -- 172.21.15.60:0/3502398742 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7c009470 con 0x7f0b841438a0 2022-06-29T15:58:27.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.689+0000 7f0b837fe700 1 --2- 172.21.15.60:0/3502398742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b84060650 0x7f0b84151e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:27.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.689+0000 7f0b7bfff700 1 --2- 172.21.15.60:0/3502398742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b840b28b0 0x7f0b8413ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:27.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.689+0000 7f0b83fff700 1 --2- 172.21.15.60:0/3502398742 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b841438a0 0x7f0b8413dad0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f0b7c00c000 tx=0x7f0b7c0091d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:27.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.689+0000 7f0b817fa700 1 -- 172.21.15.60:0/3502398742 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b7c00c610 con 0x7f0b841438a0 2022-06-29T15:58:27.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.689+0000 7f0b817fa700 1 -- 172.21.15.60:0/3502398742 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b7c00c770 con 0x7f0b841438a0 2022-06-29T15:58:27.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.689+0000 7f0b8892f700 1 -- 172.21.15.60:0/3502398742 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b7400b180 con 0x7f0b841438a0 2022-06-29T15:58:27.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.689+0000 7f0b817fa700 1 -- 172.21.15.60:0/3502398742 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b7c00cad0 con 0x7f0b841438a0 2022-06-29T15:58:27.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.689+0000 7f0b8892f700 1 -- 172.21.15.60:0/3502398742 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b8413f170 con 0x7f0b841438a0 2022-06-29T15:58:27.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.689+0000 7f0b8892f700 1 -- 172.21.15.60:0/3502398742 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b840573d0 con 0x7f0b841438a0 2022-06-29T15:58:27.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.693+0000 7f0b817fa700 1 -- 172.21.15.60:0/3502398742 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0b7c00cc30 con 0x7f0b841438a0 2022-06-29T15:58:27.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.693+0000 7f0b817fa700 1 --2- 172.21.15.60:0/3502398742 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0b68053a70 0x7f0b68055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:27.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.693+0000 7f0b817fa700 1 -- 172.21.15.60:0/3502398742 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0b7c06c170 con 0x7f0b841438a0 2022-06-29T15:58:27.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.693+0000 7f0b837fe700 1 --2- 172.21.15.60:0/3502398742 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0b68053a70 0x7f0b68055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:27.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.693+0000 7f0b837fe700 1 --2- 172.21.15.60:0/3502398742 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0b68053a70 0x7f0b68055f30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f0b7401a000 tx=0x7f0b74014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:27.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.701+0000 7f0b817fa700 1 -- 172.21.15.60:0/3502398742 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0b7c043090 con 0x7f0b841438a0 2022-06-29T15:58:27.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.829+0000 7f0b8892f700 1 -- 172.21.15.60:0/3502398742 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b8413bba0 con 0x7f0b841438a0 2022-06-29T15:58:27.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.829+0000 7f0b817fa700 1 -- 172.21.15.60:0/3502398742 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16038 (secure 0 0 0) 0x7f0b7c0466b0 con 0x7f0b841438a0 2022-06-29T15:58:27.836 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:58:27.836 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":23,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T15:58:26.345644+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-06-29T15:58:22.424395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-06-29T15:58:23.343438+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T15:58:27.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.833+0000 7f0b8892f700 1 -- 172.21.15.60:0/3502398742 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0b68053a70 msgr2=0x7f0b68055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:27.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.833+0000 7f0b8892f700 1 --2- 172.21.15.60:0/3502398742 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0b68053a70 0x7f0b68055f30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f0b7401a000 tx=0x7f0b74014000).stop 2022-06-29T15:58:27.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.833+0000 7f0b8892f700 1 -- 172.21.15.60:0/3502398742 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b841438a0 msgr2=0x7f0b8413dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:27.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.833+0000 7f0b8892f700 1 --2- 172.21.15.60:0/3502398742 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b841438a0 0x7f0b8413dad0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f0b7c00c000 tx=0x7f0b7c0091d0).stop 2022-06-29T15:58:27.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.833+0000 7f0b8892f700 1 -- 172.21.15.60:0/3502398742 shutdown_connections 2022-06-29T15:58:27.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.833+0000 7f0b8892f700 1 --2- 172.21.15.60:0/3502398742 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0b68053a70 0x7f0b68055f30 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.833+0000 7f0b8892f700 1 --2- 172.21.15.60:0/3502398742 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b841438a0 0x7f0b8413dad0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.833+0000 7f0b8892f700 1 --2- 172.21.15.60:0/3502398742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b840b28b0 0x7f0b8413ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.833+0000 7f0b8892f700 1 --2- 172.21.15.60:0/3502398742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b84060650 0x7f0b84151e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.833+0000 7f0b8892f700 1 -- 172.21.15.60:0/3502398742 >> 172.21.15.60:0/3502398742 conn(0x7f0b840637c0 msgr2=0x7f0b840659e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:27.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.833+0000 7f0b8892f700 1 -- 172.21.15.60:0/3502398742 shutdown_connections 2022-06-29T15:58:27.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.833+0000 7f0b8892f700 1 -- 172.21.15.60:0/3502398742 wait complete. 2022-06-29T15:58:27.850 INFO:tasks.ceph.ceph_manager.ceph:all up! 2022-06-29T15:58:27.850 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T15:58:27.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.949+0000 7f1ae1aa4700 1 Processor -- start 2022-06-29T15:58:27.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.949+0000 7f1ae1aa4700 1 -- start start 2022-06-29T15:58:27.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.949+0000 7f1ae1aa4700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1adc0adc80 0x7f1adc0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:27.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.949+0000 7f1ae1aa4700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1adc060b60 0x7f1adc143180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:27.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.949+0000 7f1ae1aa4700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1adc0b65b0 con 0x7f1adc0b61d0 2022-06-29T15:58:27.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.949+0000 7f1ae1aa4700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1adc0605e0 con 0x7f1adc060b60 2022-06-29T15:58:27.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.949+0000 7f1ae1aa4700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1adc060880 con 0x7f1adc0adc80 2022-06-29T15:58:27.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.949+0000 7f1ad3fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1adc0adc80 0x7f1adc0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:27.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.949+0000 7f1ad3fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1adc0adc80 0x7f1adc0af8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:58186/0 (socket says 172.21.15.60:58186) 2022-06-29T15:58:27.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.949+0000 7f1ad3fff700 1 -- 172.21.15.60:0/2807003956 learned_addr learned my addr 172.21.15.60:0/2807003956 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:27.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.949+0000 7f1adbfff700 1 --2- 172.21.15.60:0/2807003956 >> v2:172.21.15.92:3300/0 conn(0x7f1adc060b60 0x7f1adc143180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:27.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.949+0000 7f1adbfff700 1 -- 172.21.15.60:0/2807003956 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1adc0adc80 msgr2=0x7f1adc0af8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:27.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.949+0000 7f1adbfff700 1 --2- 172.21.15.60:0/2807003956 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1adc0adc80 0x7f1adc0af8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.949+0000 7f1adbfff700 1 -- 172.21.15.60:0/2807003956 >> v1:172.21.15.92:6789/0 conn(0x7f1adc0b61d0 legacy=0x7f1adc142a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:27.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.949+0000 7f1adbfff700 1 -- 172.21.15.60:0/2807003956 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1adc067340 con 0x7f1adc060b60 2022-06-29T15:58:27.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.949+0000 7f1ad3fff700 1 --2- 172.21.15.60:0/2807003956 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1adc0adc80 0x7f1adc0af8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:27.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.949+0000 7f1adbfff700 1 --2- 172.21.15.60:0/2807003956 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1adc060b60 0x7f1adc143180 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f1ad40032f0 tx=0x7f1ad40091d0).ready entity=mon.1 client_cookie=a6ff482867ba4db2 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:27.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.949+0000 7f1adaffd700 1 -- 172.21.15.60:0/2807003956 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ad400b6c0 con 0x7f1adc060b60 2022-06-29T15:58:27.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.949+0000 7f1adaffd700 1 -- 172.21.15.60:0/2807003956 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad400b820 con 0x7f1adc060b60 2022-06-29T15:58:27.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.949+0000 7f1adaffd700 1 -- 172.21.15.60:0/2807003956 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ad400b990 con 0x7f1adc060b60 2022-06-29T15:58:27.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.953+0000 7f1ae1aa4700 1 -- 172.21.15.60:0/2807003956 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1adc060b60 msgr2=0x7f1adc143180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:27.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.953+0000 7f1ae1aa4700 1 --2- 172.21.15.60:0/2807003956 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1adc060b60 0x7f1adc143180 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f1ad40032f0 tx=0x7f1ad40091d0).stop 2022-06-29T15:58:27.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.953+0000 7f1ae1aa4700 1 -- 172.21.15.60:0/2807003956 shutdown_connections 2022-06-29T15:58:27.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.953+0000 7f1ae1aa4700 1 --2- 172.21.15.60:0/2807003956 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1adc060b60 0x7f1adc143180 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.953+0000 7f1ae1aa4700 1 --2- 172.21.15.60:0/2807003956 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1adc0adc80 0x7f1adc0af8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.953+0000 7f1ae1aa4700 1 -- 172.21.15.60:0/2807003956 >> 172.21.15.60:0/2807003956 conn(0x7f1adc063b50 msgr2=0x7f1adc0b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:27.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.953+0000 7f1ae1aa4700 1 -- 172.21.15.60:0/2807003956 shutdown_connections 2022-06-29T15:58:27.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.953+0000 7f1ae1aa4700 1 -- 172.21.15.60:0/2807003956 wait complete. 2022-06-29T15:58:27.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.953+0000 7f1ae1aa4700 1 Processor -- start 2022-06-29T15:58:27.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.953+0000 7f1ae1aa4700 1 -- start start 2022-06-29T15:58:27.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.953+0000 7f1ae1aa4700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1adc060b60 0x7f1adc13cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:27.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.953+0000 7f1ae1aa4700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1adc0adc80 0x7f1adc13de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:27.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.953+0000 7f1ae1aa4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1adc0b61d0 0x7f1adc13e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:27.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.953+0000 7f1ae1aa4700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1adc112f00 con 0x7f1adc0b61d0 2022-06-29T15:58:27.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.953+0000 7f1ae1aa4700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1adc1131a0 con 0x7f1adc060b60 2022-06-29T15:58:27.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.953+0000 7f1ae1aa4700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1adc066eb0 con 0x7f1adc0adc80 2022-06-29T15:58:27.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.953+0000 7f1adbfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1adc0b61d0 0x7f1adc13e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:27.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.953+0000 7f1ad3fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1adc0adc80 0x7f1adc13de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:27.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.953+0000 7f1ad3fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1adc0adc80 0x7f1adc13de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:58192/0 (socket says 172.21.15.60:58192) 2022-06-29T15:58:27.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.953+0000 7f1ad3fff700 1 -- 172.21.15.60:0/43077773 learned_addr learned my addr 172.21.15.60:0/43077773 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:27.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.953+0000 7f1adbfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1adc0b61d0 0x7f1adc13e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45102/0 (socket says 172.21.15.60:45102) 2022-06-29T15:58:27.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.953+0000 7f1adb7fe700 1 --2- 172.21.15.60:0/43077773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1adc060b60 0x7f1adc13cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:27.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.957+0000 7f1adbfff700 1 -- 172.21.15.60:0/43077773 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1adc0adc80 msgr2=0x7f1adc13de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:27.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.957+0000 7f1adbfff700 1 --2- 172.21.15.60:0/43077773 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1adc0adc80 0x7f1adc13de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.957+0000 7f1adbfff700 1 -- 172.21.15.60:0/43077773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1adc060b60 msgr2=0x7f1adc13cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:27.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.957+0000 7f1adbfff700 1 --2- 172.21.15.60:0/43077773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1adc060b60 0x7f1adc13cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:27.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.957+0000 7f1adbfff700 1 -- 172.21.15.60:0/43077773 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac400b1e0 con 0x7f1adc0b61d0 2022-06-29T15:58:27.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.957+0000 7f1adb7fe700 1 --2- 172.21.15.60:0/43077773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1adc060b60 0x7f1adc13cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:58:27.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.957+0000 7f1adbfff700 1 --2- 172.21.15.60:0/43077773 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1adc0b61d0 0x7f1adc13e310 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f1ad40032f0 tx=0x7f1ad4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:27.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.957+0000 7f1ad97fa700 1 -- 172.21.15.60:0/43077773 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ad400b990 con 0x7f1adc0b61d0 2022-06-29T15:58:27.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.957+0000 7f1ad97fa700 1 -- 172.21.15.60:0/43077773 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad4021630 con 0x7f1adc0b61d0 2022-06-29T15:58:27.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.957+0000 7f1ae1aa4700 1 -- 172.21.15.60:0/43077773 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ad4009470 con 0x7f1adc0b61d0 2022-06-29T15:58:27.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.957+0000 7f1ad97fa700 1 -- 172.21.15.60:0/43077773 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ad400b6c0 con 0x7f1adc0b61d0 2022-06-29T15:58:27.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.957+0000 7f1ae1aa4700 1 -- 172.21.15.60:0/43077773 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1adc13c940 con 0x7f1adc0b61d0 2022-06-29T15:58:27.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.957+0000 7f1ae1aa4700 1 -- 172.21.15.60:0/43077773 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1adc0573d0 con 0x7f1adc0b61d0 2022-06-29T15:58:27.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.957+0000 7f1ad97fa700 1 -- 172.21.15.60:0/43077773 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1ad400f070 con 0x7f1adc0b61d0 2022-06-29T15:58:27.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.957+0000 7f1ad97fa700 1 --2- 172.21.15.60:0/43077773 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1ac0053aa0 0x7f1ac0055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:27.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.961+0000 7f1ad97fa700 1 -- 172.21.15.60:0/43077773 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1ad406d380 con 0x7f1adc0b61d0 2022-06-29T15:58:27.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.961+0000 7f1adb7fe700 1 --2- 172.21.15.60:0/43077773 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1ac0053aa0 0x7f1ac0055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:27.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.961+0000 7f1adb7fe700 1 --2- 172.21.15.60:0/43077773 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1ac0053aa0 0x7f1ac0055f60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f1acc00b000 tx=0x7f1acc009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:27.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:27.965+0000 7f1ad97fa700 1 -- 172.21.15.60:0/43077773 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1ad4044190 con 0x7f1adc0b61d0 2022-06-29T15:58:28.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.093+0000 7f1ae1aa4700 1 -- 172.21.15.60:0/43077773 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1adc113310 con 0x7f1adc0b61d0 2022-06-29T15:58:28.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.093+0000 7f1ad97fa700 1 -- 172.21.15.60:0/43077773 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16038 (secure 0 0 0) 0x7f1ad4015020 con 0x7f1adc0b61d0 2022-06-29T15:58:28.099 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:58:28.100 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":23,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T15:58:26.345644+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-06-29T15:58:22.424395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-06-29T15:58:23.343438+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T15:58:28.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.093+0000 7f1ae1aa4700 1 -- 172.21.15.60:0/43077773 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1ac0053aa0 msgr2=0x7f1ac0055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.093+0000 7f1ae1aa4700 1 --2- 172.21.15.60:0/43077773 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1ac0053aa0 0x7f1ac0055f60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f1acc00b000 tx=0x7f1acc009040).stop 2022-06-29T15:58:28.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.093+0000 7f1ae1aa4700 1 -- 172.21.15.60:0/43077773 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1adc0b61d0 msgr2=0x7f1adc13e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.093+0000 7f1ae1aa4700 1 --2- 172.21.15.60:0/43077773 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1adc0b61d0 0x7f1adc13e310 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f1ad40032f0 tx=0x7f1ad4016000).stop 2022-06-29T15:58:28.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.097+0000 7f1ae1aa4700 1 -- 172.21.15.60:0/43077773 shutdown_connections 2022-06-29T15:58:28.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.097+0000 7f1ae1aa4700 1 --2- 172.21.15.60:0/43077773 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1ac0053aa0 0x7f1ac0055f60 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:28.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.097+0000 7f1ae1aa4700 1 --2- 172.21.15.60:0/43077773 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1adc0b61d0 0x7f1adc13e310 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:28.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.097+0000 7f1ae1aa4700 1 --2- 172.21.15.60:0/43077773 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1adc0adc80 0x7f1adc13de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:28.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.097+0000 7f1ae1aa4700 1 --2- 172.21.15.60:0/43077773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1adc060b60 0x7f1adc13cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:28.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.097+0000 7f1ae1aa4700 1 -- 172.21.15.60:0/43077773 >> 172.21.15.60:0/43077773 conn(0x7f1adc063b50 msgr2=0x7f1adc13a860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:28.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.097+0000 7f1ae1aa4700 1 -- 172.21.15.60:0/43077773 shutdown_connections 2022-06-29T15:58:28.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.097+0000 7f1ae1aa4700 1 -- 172.21.15.60:0/43077773 wait complete. 2022-06-29T15:58:28.113 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2022-06-29T15:58:28.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.205+0000 7f104759e700 1 Processor -- start 2022-06-29T15:58:28.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.205+0000 7f104759e700 1 -- start start 2022-06-29T15:58:28.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.209+0000 7f104759e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10480acd60 0x7f1048146020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.209+0000 7f104759e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f10480b2f70 0x7f1048146510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.209+0000 7f104759e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10480ad140 con 0x7f104814a680 2022-06-29T15:58:28.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.209+0000 7f104759e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1048062440 con 0x7f10480b2f70 2022-06-29T15:58:28.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.209+0000 7f104759e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f104814b200 con 0x7f10480acd60 2022-06-29T15:58:28.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.209+0000 7f1045d9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f10480b2f70 0x7f1048146510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.209+0000 7f104659c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10480acd60 0x7f1048146020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.209+0000 7f104659c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10480acd60 0x7f1048146020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:58200/0 (socket says 172.21.15.60:58200) 2022-06-29T15:58:28.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.209+0000 7f1046d9d700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f104814a680 0x7f104814aaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:57444/0 (socket says 172.21.15.60:57444) 2022-06-29T15:58:28.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.209+0000 7f104659c700 1 -- 172.21.15.60:0/4002278717 learned_addr learned my addr 172.21.15.60:0/4002278717 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:28.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.209+0000 7f104559a700 1 -- 172.21.15.60:0/4002278717 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1575175196 0 0) 0x7f10480ad140 con 0x7f104814a680 2022-06-29T15:58:28.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.209+0000 7f104659c700 1 -- 172.21.15.60:0/4002278717 >> v2:172.21.15.60:3300/0 conn(0x7f10480b2f70 msgr2=0x7f1048146510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:28.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.209+0000 7f104659c700 1 --2- 172.21.15.60:0/4002278717 >> v2:172.21.15.60:3300/0 conn(0x7f10480b2f70 0x7f1048146510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:28.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.209+0000 7f104659c700 1 -- 172.21.15.60:0/4002278717 >> v1:172.21.15.92:6789/0 conn(0x7f104814a680 legacy=0x7f104814aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.209+0000 7f104659c700 1 -- 172.21.15.60:0/4002278717 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10480632d0 con 0x7f10480acd60 2022-06-29T15:58:28.230 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.209+0000 7f1045d9b700 1 --2- 172.21.15.60:0/4002278717 >> v2:172.21.15.60:3300/0 conn(0x7f10480b2f70 0x7f1048146510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:28.230 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.209+0000 7f104659c700 1 --2- 172.21.15.60:0/4002278717 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10480acd60 0x7f1048146020 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f10400032f0 tx=0x7f1040009290).ready entity=mon.2 client_cookie=800c37a6c4049884 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:28.230 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.209+0000 7f104559a700 1 -- 172.21.15.60:0/4002278717 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f104000b8a0 con 0x7f10480acd60 2022-06-29T15:58:28.230 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.209+0000 7f104559a700 1 -- 172.21.15.60:0/4002278717 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f104000ba00 con 0x7f10480acd60 2022-06-29T15:58:28.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.209+0000 7f104559a700 1 -- 172.21.15.60:0/4002278717 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f104000bb70 con 0x7f10480acd60 2022-06-29T15:58:28.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.209+0000 7f104759e700 1 -- 172.21.15.60:0/4002278717 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10480acd60 msgr2=0x7f1048146020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.209+0000 7f104759e700 1 --2- 172.21.15.60:0/4002278717 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10480acd60 0x7f1048146020 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f10400032f0 tx=0x7f1040009290).stop 2022-06-29T15:58:28.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.209+0000 7f104759e700 1 -- 172.21.15.60:0/4002278717 shutdown_connections 2022-06-29T15:58:28.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.209+0000 7f104759e700 1 --2- 172.21.15.60:0/4002278717 >> v2:172.21.15.60:3300/0 conn(0x7f10480b2f70 0x7f1048146510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:28.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.209+0000 7f104759e700 1 --2- 172.21.15.60:0/4002278717 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f10480acd60 0x7f1048146020 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:28.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.209+0000 7f104759e700 1 -- 172.21.15.60:0/4002278717 >> 172.21.15.60:0/4002278717 conn(0x7f10480b6f80 msgr2=0x7f10480b7390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:28.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.209+0000 7f104759e700 1 -- 172.21.15.60:0/4002278717 shutdown_connections 2022-06-29T15:58:28.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.209+0000 7f104759e700 1 -- 172.21.15.60:0/4002278717 wait complete. 2022-06-29T15:58:28.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.213+0000 7f104759e700 1 Processor -- start 2022-06-29T15:58:28.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.213+0000 7f104759e700 1 -- start start 2022-06-29T15:58:28.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.213+0000 7f104759e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10480acd60 0x7f1048159f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.213+0000 7f104759e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10480b2f70 0x7f10481448f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.213+0000 7f104759e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f104814a680 0x7f10480af250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.213+0000 7f104759e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1048144730 con 0x7f10480b2f70 2022-06-29T15:58:28.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.213+0000 7f104759e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104815a410 con 0x7f10480acd60 2022-06-29T15:58:28.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.213+0000 7f104759e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1048159540 con 0x7f104814a680 2022-06-29T15:58:28.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.213+0000 7f1045d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10480b2f70 0x7f10481448f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.213+0000 7f1045d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10480b2f70 0x7f10481448f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45116/0 (socket says 172.21.15.60:45116) 2022-06-29T15:58:28.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.213+0000 7f1045d9b700 1 -- 172.21.15.60:0/316696146 learned_addr learned my addr 172.21.15.60:0/316696146 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:28.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.213+0000 7f1046d9d700 1 --2- 172.21.15.60:0/316696146 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f104814a680 0x7f10480af250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.213+0000 7f104659c700 1 --2- 172.21.15.60:0/316696146 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10480acd60 0x7f1048159f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.213+0000 7f1046d9d700 1 -- 172.21.15.60:0/316696146 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10480acd60 msgr2=0x7f1048159f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.213+0000 7f1046d9d700 1 --2- 172.21.15.60:0/316696146 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10480acd60 0x7f1048159f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:28.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.213+0000 7f1046d9d700 1 -- 172.21.15.60:0/316696146 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10480b2f70 msgr2=0x7f10481448f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.213+0000 7f1046d9d700 1 --2- 172.21.15.60:0/316696146 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10480b2f70 0x7f10481448f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:28.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.213+0000 7f1046d9d700 1 -- 172.21.15.60:0/316696146 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f103000b180 con 0x7f104814a680 2022-06-29T15:58:28.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.213+0000 7f104659c700 1 --2- 172.21.15.60:0/316696146 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10480acd60 0x7f1048159f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:28.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.213+0000 7f1045d9b700 1 --2- 172.21.15.60:0/316696146 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f10480b2f70 0x7f10481448f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:28.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.213+0000 7f1046d9d700 1 --2- 172.21.15.60:0/316696146 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f104814a680 0x7f10480af250 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f1038000bc0 tx=0x7f1038007980).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:28.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.213+0000 7f103f7fe700 1 -- 172.21.15.60:0/316696146 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10380072e0 con 0x7f104814a680 2022-06-29T15:58:28.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.213+0000 7f103f7fe700 1 -- 172.21.15.60:0/316696146 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f103800cb30 con 0x7f104814a680 2022-06-29T15:58:28.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.213+0000 7f104759e700 1 -- 172.21.15.60:0/316696146 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1040009530 con 0x7f104814a680 2022-06-29T15:58:28.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.213+0000 7f103f7fe700 1 -- 172.21.15.60:0/316696146 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f103800ce90 con 0x7f104814a680 2022-06-29T15:58:28.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.213+0000 7f104759e700 1 -- 172.21.15.60:0/316696146 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10480649d0 con 0x7f104814a680 2022-06-29T15:58:28.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.217+0000 7f104759e700 1 -- 172.21.15.60:0/316696146 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1048159c80 con 0x7f104814a680 2022-06-29T15:58:28.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.217+0000 7f103f7fe700 1 -- 172.21.15.60:0/316696146 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f103801a060 con 0x7f104814a680 2022-06-29T15:58:28.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.217+0000 7f103f7fe700 1 --2- 172.21.15.60:0/316696146 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f102c053b00 0x7f102c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.217+0000 7f103f7fe700 1 -- 172.21.15.60:0/316696146 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f103806bf60 con 0x7f104814a680 2022-06-29T15:58:28.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.217+0000 7f104659c700 1 --2- 172.21.15.60:0/316696146 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f102c053b00 0x7f102c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.217+0000 7f103f7fe700 1 --2- 172.21.15.60:0/316696146 >> [v2:172.21.15.60:6816/12890,v1:172.21.15.60:6825/12890] conn(0x7f102c05acf0 0x7f102c05d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.217+0000 7f103f7fe700 1 -- 172.21.15.60:0/316696146 --> [v2:172.21.15.60:6816/12890,v1:172.21.15.60:6825/12890] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f102c05d730 con 0x7f102c05acf0 2022-06-29T15:58:28.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.217+0000 7f103f7fe700 1 -- 172.21.15.60:0/316696146 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f103806c230 con 0x7f104814a680 2022-06-29T15:58:28.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.217+0000 7f1045d9b700 1 --2- 172.21.15.60:0/316696146 >> [v2:172.21.15.60:6816/12890,v1:172.21.15.60:6825/12890] conn(0x7f102c05acf0 0x7f102c05d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.217+0000 7f104659c700 1 --2- 172.21.15.60:0/316696146 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f102c053b00 0x7f102c055fc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f104001b000 tx=0x7f1040016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:28.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.217+0000 7f1045d9b700 1 --2- 172.21.15.60:0/316696146 >> [v2:172.21.15.60:6816/12890,v1:172.21.15.60:6825/12890] conn(0x7f102c05acf0 0x7f102c05d150 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:28.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.221+0000 7f103f7fe700 1 -- 172.21.15.60:0/316696146 <== osd.0 v2:172.21.15.60:6816/12890 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f102c05d730 con 0x7f102c05acf0 2022-06-29T15:58:28.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.237+0000 7f104759e700 1 -- 172.21.15.60:0/316696146 --> [v2:172.21.15.60:6816/12890,v1:172.21.15.60:6825/12890] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1048064fe0 con 0x7f102c05acf0 2022-06-29T15:58:28.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.237+0000 7f103f7fe700 1 -- 172.21.15.60:0/316696146 <== osd.0 v2:172.21.15.60:6816/12890 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1030014410 con 0x7f102c05acf0 2022-06-29T15:58:28.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.237+0000 7f104759e700 1 -- 172.21.15.60:0/316696146 >> [v2:172.21.15.60:6816/12890,v1:172.21.15.60:6825/12890] conn(0x7f102c05acf0 msgr2=0x7f102c05d150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.237+0000 7f104759e700 1 --2- 172.21.15.60:0/316696146 >> [v2:172.21.15.60:6816/12890,v1:172.21.15.60:6825/12890] conn(0x7f102c05acf0 0x7f102c05d150 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:28.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.237+0000 7f104759e700 1 -- 172.21.15.60:0/316696146 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f102c053b00 msgr2=0x7f102c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.237+0000 7f104759e700 1 --2- 172.21.15.60:0/316696146 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f102c053b00 0x7f102c055fc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f104001b000 tx=0x7f1040016000).stop 2022-06-29T15:58:28.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.237+0000 7f104759e700 1 -- 172.21.15.60:0/316696146 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f104814a680 msgr2=0x7f10480af250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.237+0000 7f104759e700 1 --2- 172.21.15.60:0/316696146 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f104814a680 0x7f10480af250 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f1038000bc0 tx=0x7f1038007980).stop 2022-06-29T15:58:28.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.237+0000 7f1046d9d700 1 -- 172.21.15.60:0/316696146 reap_dead start 2022-06-29T15:58:28.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.237+0000 7f104759e700 1 -- 172.21.15.60:0/316696146 shutdown_connections 2022-06-29T15:58:28.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.237+0000 7f104759e700 1 -- 172.21.15.60:0/316696146 >> 172.21.15.60:0/316696146 conn(0x7f10480b6f80 msgr2=0x7f10480b1920 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:28.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.237+0000 7f104759e700 1 -- 172.21.15.60:0/316696146 shutdown_connections 2022-06-29T15:58:28.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.237+0000 7f104759e700 1 -- 172.21.15.60:0/316696146 wait complete. 2022-06-29T15:58:28.260 INFO:teuthology.orchestra.run.smithi060.stdout:55834574853 2022-06-29T15:58:28.260 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2022-06-29T15:58:28.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.361+0000 7f5ff0b79700 1 Processor -- start 2022-06-29T15:58:28.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.361+0000 7f5ff0b79700 1 -- start start 2022-06-29T15:58:28.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.361+0000 7f5ff0b79700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5fec0b5eb0 0x7f5fec142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.361+0000 7f5ff0b79700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5fec0b2c50 0x7f5fec0b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.361+0000 7f5ff0b79700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fec0b0cf0 con 0x7f5fec0b48b0 2022-06-29T15:58:28.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.361+0000 7f5ff0b79700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fec0b54e0 con 0x7f5fec0b2c50 2022-06-29T15:58:28.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.361+0000 7f5ff0b79700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fec0b4d10 con 0x7f5fec0b5eb0 2022-06-29T15:58:28.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.365+0000 7f5febfff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5fec0b2c50 0x7f5fec0b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.365+0000 7f5febfff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5fec0b2c50 0x7f5fec0b30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45126/0 (socket says 172.21.15.60:45126) 2022-06-29T15:58:28.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.365+0000 7f5febfff700 1 -- 172.21.15.60:0/140842532 learned_addr learned my addr 172.21.15.60:0/140842532 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:28.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.365+0000 7f5feb7fe700 1 --2- 172.21.15.60:0/140842532 >> v2:172.21.15.92:3300/0 conn(0x7f5fec0b5eb0 0x7f5fec142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.365+0000 7f5febfff700 1 -- 172.21.15.60:0/140842532 >> v2:172.21.15.92:3300/0 conn(0x7f5fec0b5eb0 msgr2=0x7f5fec142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:28.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.365+0000 7f5febfff700 1 --2- 172.21.15.60:0/140842532 >> v2:172.21.15.92:3300/0 conn(0x7f5fec0b5eb0 0x7f5fec142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:28.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.365+0000 7f5febfff700 1 -- 172.21.15.60:0/140842532 >> v1:172.21.15.92:6789/0 conn(0x7f5fec0b48b0 legacy=0x7f5fec0b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.365+0000 7f5febfff700 1 -- 172.21.15.60:0/140842532 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fec0b6290 con 0x7f5fec0b2c50 2022-06-29T15:58:28.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.365+0000 7f5febfff700 1 --2- 172.21.15.60:0/140842532 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fec0b2c50 0x7f5fec0b30c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f5fe40032f0 tx=0x7f5fe4009290).ready entity=mon.0 client_cookie=585f4666cc927005 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:28.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.365+0000 7f5feaffd700 1 -- 172.21.15.60:0/140842532 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fe400b870 con 0x7f5fec0b2c50 2022-06-29T15:58:28.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.365+0000 7f5feaffd700 1 -- 172.21.15.60:0/140842532 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fe400b9d0 con 0x7f5fec0b2c50 2022-06-29T15:58:28.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.365+0000 7f5feaffd700 1 -- 172.21.15.60:0/140842532 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fe400bb40 con 0x7f5fec0b2c50 2022-06-29T15:58:28.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.365+0000 7f5ff0b79700 1 -- 172.21.15.60:0/140842532 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fec0b2c50 msgr2=0x7f5fec0b30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.365+0000 7f5ff0b79700 1 --2- 172.21.15.60:0/140842532 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fec0b2c50 0x7f5fec0b30c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f5fe40032f0 tx=0x7f5fe4009290).stop 2022-06-29T15:58:28.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.365+0000 7f5ff0b79700 1 -- 172.21.15.60:0/140842532 shutdown_connections 2022-06-29T15:58:28.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.365+0000 7f5ff0b79700 1 --2- 172.21.15.60:0/140842532 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fec0b2c50 0x7f5fec0b30c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:28.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.365+0000 7f5ff0b79700 1 --2- 172.21.15.60:0/140842532 >> v2:172.21.15.92:3300/0 conn(0x7f5fec0b5eb0 0x7f5fec142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:28.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.365+0000 7f5ff0b79700 1 -- 172.21.15.60:0/140842532 >> 172.21.15.60:0/140842532 conn(0x7f5fec062090 msgr2=0x7f5fec062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:28.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.365+0000 7f5ff0b79700 1 -- 172.21.15.60:0/140842532 shutdown_connections 2022-06-29T15:58:28.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.365+0000 7f5ff0b79700 1 -- 172.21.15.60:0/140842532 wait complete. 2022-06-29T15:58:28.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.369+0000 7f5ff0b79700 1 Processor -- start 2022-06-29T15:58:28.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.369+0000 7f5ff0b79700 1 -- start start 2022-06-29T15:58:28.382 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.369+0000 7f5ff0b79700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fec0b2c50 0x7f5fec0b8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.382 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.369+0000 7f5ff0b79700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fec0b48b0 0x7f5fec13fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.382 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.369+0000 7f5ff0b79700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fec0b5eb0 0x7f5fec1400a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.369+0000 7f5ff0b79700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec152ea0 con 0x7f5fec0b2c50 2022-06-29T15:58:28.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.369+0000 7f5ff0b79700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec144180 con 0x7f5fec0b5eb0 2022-06-29T15:58:28.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.369+0000 7f5ff0b79700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec143ee0 con 0x7f5fec0b48b0 2022-06-29T15:58:28.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.369+0000 7f5feb7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fec0b2c50 0x7f5fec0b8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.369+0000 7f5feb7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fec0b2c50 0x7f5fec0b8230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45128/0 (socket says 172.21.15.60:45128) 2022-06-29T15:58:28.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.369+0000 7f5feb7fe700 1 -- 172.21.15.60:0/2656420846 learned_addr learned my addr 172.21.15.60:0/2656420846 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:28.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.369+0000 7f5fe3fff700 1 --2- 172.21.15.60:0/2656420846 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fec0b48b0 0x7f5fec13fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.369+0000 7f5febfff700 1 --2- 172.21.15.60:0/2656420846 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fec0b5eb0 0x7f5fec1400a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.373+0000 7f5feb7fe700 1 -- 172.21.15.60:0/2656420846 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fec0b48b0 msgr2=0x7f5fec13fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.373+0000 7f5feb7fe700 1 --2- 172.21.15.60:0/2656420846 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fec0b48b0 0x7f5fec13fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:28.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.373+0000 7f5feb7fe700 1 -- 172.21.15.60:0/2656420846 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fec0b5eb0 msgr2=0x7f5fec1400a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.373+0000 7f5feb7fe700 1 --2- 172.21.15.60:0/2656420846 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fec0b5eb0 0x7f5fec1400a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:28.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.373+0000 7f5feb7fe700 1 -- 172.21.15.60:0/2656420846 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe4009530 con 0x7f5fec0b2c50 2022-06-29T15:58:28.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.373+0000 7f5fe3fff700 1 --2- 172.21.15.60:0/2656420846 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fec0b48b0 0x7f5fec13fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:28.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.373+0000 7f5febfff700 1 --2- 172.21.15.60:0/2656420846 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fec0b5eb0 0x7f5fec1400a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:28.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.373+0000 7f5feb7fe700 1 --2- 172.21.15.60:0/2656420846 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fec0b2c50 0x7f5fec0b8230 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f5fdc0030b0 tx=0x7f5fdc008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:28.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.373+0000 7f5fe97fa700 1 -- 172.21.15.60:0/2656420846 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fdc011780 con 0x7f5fec0b2c50 2022-06-29T15:58:28.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.373+0000 7f5fe97fa700 1 -- 172.21.15.60:0/2656420846 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fdc0118e0 con 0x7f5fec0b2c50 2022-06-29T15:58:28.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.373+0000 7f5ff0b79700 1 -- 172.21.15.60:0/2656420846 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fec145b80 con 0x7f5fec0b2c50 2022-06-29T15:58:28.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.373+0000 7f5fe97fa700 1 -- 172.21.15.60:0/2656420846 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fdc011c40 con 0x7f5fec0b2c50 2022-06-29T15:58:28.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.373+0000 7f5ff0b79700 1 -- 172.21.15.60:0/2656420846 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fec153040 con 0x7f5fec0b2c50 2022-06-29T15:58:28.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.373+0000 7f5ff0b79700 1 -- 172.21.15.60:0/2656420846 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5fec145d10 con 0x7f5fec0b2c50 2022-06-29T15:58:28.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.373+0000 7f5fe97fa700 1 -- 172.21.15.60:0/2656420846 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5fdc012420 con 0x7f5fec0b2c50 2022-06-29T15:58:28.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.373+0000 7f5fe97fa700 1 --2- 172.21.15.60:0/2656420846 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5fd0053b00 0x7f5fd0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.373+0000 7f5fe97fa700 1 -- 172.21.15.60:0/2656420846 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5fdc06bbc0 con 0x7f5fec0b2c50 2022-06-29T15:58:28.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.377+0000 7f5fe97fa700 1 --2- 172.21.15.60:0/2656420846 >> [v2:172.21.15.60:6815/12892,v1:172.21.15.60:6818/12892] conn(0x7f5fd005ae60 0x7f5fd005d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.377+0000 7f5fe97fa700 1 -- 172.21.15.60:0/2656420846 --> [v2:172.21.15.60:6815/12892,v1:172.21.15.60:6818/12892] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5fd005d8a0 con 0x7f5fd005ae60 2022-06-29T15:58:28.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.377+0000 7f5fe97fa700 1 -- 172.21.15.60:0/2656420846 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5fdc071220 con 0x7f5fec0b2c50 2022-06-29T15:58:28.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.377+0000 7f5fe3fff700 1 --2- 172.21.15.60:0/2656420846 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5fd0053b00 0x7f5fd0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.377+0000 7f5febfff700 1 --2- 172.21.15.60:0/2656420846 >> [v2:172.21.15.60:6815/12892,v1:172.21.15.60:6818/12892] conn(0x7f5fd005ae60 0x7f5fd005d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.377+0000 7f5febfff700 1 --2- 172.21.15.60:0/2656420846 >> [v2:172.21.15.60:6815/12892,v1:172.21.15.60:6818/12892] conn(0x7f5fd005ae60 0x7f5fd005d2c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:28.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.377+0000 7f5fe3fff700 1 --2- 172.21.15.60:0/2656420846 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5fd0053b00 0x7f5fd0055fc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f5fd400d000 tx=0x7f5fd4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:28.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.377+0000 7f5fe97fa700 1 -- 172.21.15.60:0/2656420846 <== osd.1 v2:172.21.15.60:6815/12892 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f5fd005d8a0 con 0x7f5fd005ae60 2022-06-29T15:58:28.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.389+0000 7f5ff0b79700 1 -- 172.21.15.60:0/2656420846 --> [v2:172.21.15.60:6815/12892,v1:172.21.15.60:6818/12892] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5fec0573d0 con 0x7f5fd005ae60 2022-06-29T15:58:28.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.389+0000 7f5fe97fa700 1 -- 172.21.15.60:0/2656420846 <== osd.1 v2:172.21.15.60:6815/12892 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5fec0573d0 con 0x7f5fd005ae60 2022-06-29T15:58:28.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.389+0000 7f5ff0b79700 1 -- 172.21.15.60:0/2656420846 >> [v2:172.21.15.60:6815/12892,v1:172.21.15.60:6818/12892] conn(0x7f5fd005ae60 msgr2=0x7f5fd005d2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.389+0000 7f5ff0b79700 1 --2- 172.21.15.60:0/2656420846 >> [v2:172.21.15.60:6815/12892,v1:172.21.15.60:6818/12892] conn(0x7f5fd005ae60 0x7f5fd005d2c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:28.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.389+0000 7f5ff0b79700 1 -- 172.21.15.60:0/2656420846 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5fd0053b00 msgr2=0x7f5fd0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.389+0000 7f5ff0b79700 1 --2- 172.21.15.60:0/2656420846 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5fd0053b00 0x7f5fd0055fc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f5fd400d000 tx=0x7f5fd4009040).stop 2022-06-29T15:58:28.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.389+0000 7f5ff0b79700 1 -- 172.21.15.60:0/2656420846 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fec0b2c50 msgr2=0x7f5fec0b8230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.389+0000 7f5ff0b79700 1 --2- 172.21.15.60:0/2656420846 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fec0b2c50 0x7f5fec0b8230 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f5fdc0030b0 tx=0x7f5fdc008040).stop 2022-06-29T15:58:28.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.389+0000 7f5febfff700 1 -- 172.21.15.60:0/2656420846 reap_dead start 2022-06-29T15:58:28.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.393+0000 7f5ff0b79700 1 -- 172.21.15.60:0/2656420846 shutdown_connections 2022-06-29T15:58:28.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.393+0000 7f5ff0b79700 1 -- 172.21.15.60:0/2656420846 >> 172.21.15.60:0/2656420846 conn(0x7f5fec062090 msgr2=0x7f5fec143530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:28.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.393+0000 7f5ff0b79700 1 -- 172.21.15.60:0/2656420846 shutdown_connections 2022-06-29T15:58:28.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.393+0000 7f5ff0b79700 1 -- 172.21.15.60:0/2656420846 wait complete. 2022-06-29T15:58:28.408 INFO:teuthology.orchestra.run.smithi060.stdout:55834574852 2022-06-29T15:58:28.408 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2022-06-29T15:58:28.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.509+0000 7fd894f64700 1 Processor -- start 2022-06-29T15:58:28.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.509+0000 7fd894f64700 1 -- start start 2022-06-29T15:58:28.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.513+0000 7fd894f64700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8901426b0 0x7fd8900b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.513+0000 7fd894f64700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd890143a40 con 0x7fd8900ae7c0 2022-06-29T15:58:28.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.513+0000 7fd894f64700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8900aee60 con 0x7fd8900b4010 2022-06-29T15:58:28.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.513+0000 7fd894f64700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8900af100 con 0x7fd8901426b0 2022-06-29T15:58:28.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.513+0000 7fd88e59c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd8900ae7c0 0x7fd890141fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53224/0 (socket says 172.21.15.60:53224) 2022-06-29T15:58:28.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.513+0000 7fd88e59c700 1 -- 172.21.15.60:0/1978526404 learned_addr learned my addr 172.21.15.60:0/1978526404 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:28.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.513+0000 7fd88ed9d700 1 --2- 172.21.15.60:0/1978526404 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8901426b0 0x7fd8900b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.513+0000 7fd88d59a700 1 -- 172.21.15.60:0/1978526404 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3514539706 0 0) 0x7fd890143a40 con 0x7fd8900ae7c0 2022-06-29T15:58:28.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.513+0000 7fd88d59a700 1 -- 172.21.15.60:0/1978526404 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd874003260 con 0x7fd8900ae7c0 2022-06-29T15:58:28.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.513+0000 7fd88d59a700 1 -- 172.21.15.60:0/1978526404 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3326790600 0 0) 0x7fd8900aee60 con 0x7fd8900b4010 2022-06-29T15:58:28.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.513+0000 7fd88d59a700 1 -- 172.21.15.60:0/1978526404 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd890143a40 con 0x7fd8900b4010 2022-06-29T15:58:28.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.513+0000 7fd88ed9d700 1 -- 172.21.15.60:0/1978526404 >> v1:172.21.15.92:6789/0 conn(0x7fd8900b4010 legacy=0x7fd8900b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.513+0000 7fd88ed9d700 1 -- 172.21.15.60:0/1978526404 >> v1:172.21.15.60:6789/0 conn(0x7fd8900ae7c0 legacy=0x7fd890141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.513+0000 7fd88ed9d700 1 -- 172.21.15.60:0/1978526404 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8900aeba0 con 0x7fd8901426b0 2022-06-29T15:58:28.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.513+0000 7fd88ed9d700 1 --2- 172.21.15.60:0/1978526404 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8901426b0 0x7fd8900b4dd0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fd8880032f0 tx=0x7fd8880091d0).ready entity=mon.2 client_cookie=26dd7f1c1412bd42 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:28.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.513+0000 7fd88d59a700 1 -- 172.21.15.60:0/1978526404 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd88800b640 con 0x7fd8901426b0 2022-06-29T15:58:28.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.513+0000 7fd88d59a700 1 -- 172.21.15.60:0/1978526404 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd88800b7a0 con 0x7fd8901426b0 2022-06-29T15:58:28.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.513+0000 7fd88d59a700 1 -- 172.21.15.60:0/1978526404 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd88800b910 con 0x7fd8901426b0 2022-06-29T15:58:28.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.513+0000 7fd894f64700 1 -- 172.21.15.60:0/1978526404 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8901426b0 msgr2=0x7fd8900b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.513+0000 7fd894f64700 1 --2- 172.21.15.60:0/1978526404 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8901426b0 0x7fd8900b4dd0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fd8880032f0 tx=0x7fd8880091d0).stop 2022-06-29T15:58:28.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.513+0000 7fd894f64700 1 -- 172.21.15.60:0/1978526404 shutdown_connections 2022-06-29T15:58:28.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.513+0000 7fd894f64700 1 --2- 172.21.15.60:0/1978526404 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8901426b0 0x7fd8900b4dd0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:28.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.513+0000 7fd894f64700 1 -- 172.21.15.60:0/1978526404 >> 172.21.15.60:0/1978526404 conn(0x7fd8900b9340 msgr2=0x7fd8900b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:28.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.513+0000 7fd894f64700 1 -- 172.21.15.60:0/1978526404 shutdown_connections 2022-06-29T15:58:28.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.513+0000 7fd894f64700 1 -- 172.21.15.60:0/1978526404 wait complete. 2022-06-29T15:58:28.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.517+0000 7fd894f64700 1 Processor -- start 2022-06-29T15:58:28.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.517+0000 7fd894f64700 1 -- start start 2022-06-29T15:58:28.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.517+0000 7fd894f64700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd8900ae7c0 0x7fd8900b4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.517+0000 7fd894f64700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8900b4010 0x7fd8900af610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.517+0000 7fd894f64700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8901426b0 0x7fd8900afb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.517+0000 7fd894f64700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd890146a30 con 0x7fd8900b4010 2022-06-29T15:58:28.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.517+0000 7fd894f64700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd890146c60 con 0x7fd8900ae7c0 2022-06-29T15:58:28.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.517+0000 7fd894f64700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd890145a70 con 0x7fd8901426b0 2022-06-29T15:58:28.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.517+0000 7fd88dd9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8900b4010 0x7fd8900af610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.517+0000 7fd88dd9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8900b4010 0x7fd8900af610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45146/0 (socket says 172.21.15.60:45146) 2022-06-29T15:58:28.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.517+0000 7fd88dd9b700 1 -- 172.21.15.60:0/1501057810 learned_addr learned my addr 172.21.15.60:0/1501057810 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:28.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.517+0000 7fd88e59c700 1 --2- 172.21.15.60:0/1501057810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd8900ae7c0 0x7fd8900b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.517+0000 7fd88ed9d700 1 --2- 172.21.15.60:0/1501057810 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8901426b0 0x7fd8900afb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.517+0000 7fd88dd9b700 1 -- 172.21.15.60:0/1501057810 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8901426b0 msgr2=0x7fd8900afb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.517+0000 7fd88dd9b700 1 --2- 172.21.15.60:0/1501057810 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8901426b0 0x7fd8900afb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:28.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.517+0000 7fd88dd9b700 1 -- 172.21.15.60:0/1501057810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd8900ae7c0 msgr2=0x7fd8900b4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.517+0000 7fd88dd9b700 1 --2- 172.21.15.60:0/1501057810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd8900ae7c0 0x7fd8900b4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:28.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.517+0000 7fd88dd9b700 1 -- 172.21.15.60:0/1501057810 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd888009470 con 0x7fd8900b4010 2022-06-29T15:58:28.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.517+0000 7fd88dd9b700 1 --2- 172.21.15.60:0/1501057810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8900b4010 0x7fd8900af610 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fd880001110 tx=0x7fd880007ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:28.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.517+0000 7fd87f7fe700 1 -- 172.21.15.60:0/1501057810 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd880012400 con 0x7fd8900b4010 2022-06-29T15:58:28.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.517+0000 7fd87f7fe700 1 -- 172.21.15.60:0/1501057810 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd880012560 con 0x7fd8900b4010 2022-06-29T15:58:28.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.517+0000 7fd87f7fe700 1 -- 172.21.15.60:0/1501057810 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8800128c0 con 0x7fd8900b4010 2022-06-29T15:58:28.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.517+0000 7fd894f64700 1 -- 172.21.15.60:0/1501057810 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd890145d10 con 0x7fd8900b4010 2022-06-29T15:58:28.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.517+0000 7fd894f64700 1 -- 172.21.15.60:0/1501057810 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd890145fc0 con 0x7fd8900b4010 2022-06-29T15:58:28.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.521+0000 7fd894f64700 1 -- 172.21.15.60:0/1501057810 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd8900b2280 con 0x7fd8900b4010 2022-06-29T15:58:28.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.521+0000 7fd87f7fe700 1 -- 172.21.15.60:0/1501057810 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd880019030 con 0x7fd8900b4010 2022-06-29T15:58:28.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.521+0000 7fd87f7fe700 1 --2- 172.21.15.60:0/1501057810 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd86c053b00 0x7fd86c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.521+0000 7fd87f7fe700 1 -- 172.21.15.60:0/1501057810 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd88006b420 con 0x7fd8900b4010 2022-06-29T15:58:28.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.521+0000 7fd87f7fe700 1 --2- 172.21.15.60:0/1501057810 >> [v2:172.21.15.60:6808/12891,v1:172.21.15.60:6809/12891] conn(0x7fd86c05ade0 0x7fd86c05d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.521+0000 7fd87f7fe700 1 -- 172.21.15.60:0/1501057810 --> [v2:172.21.15.60:6808/12891,v1:172.21.15.60:6809/12891] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd86c05d820 con 0x7fd86c05ade0 2022-06-29T15:58:28.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.521+0000 7fd87f7fe700 1 -- 172.21.15.60:0/1501057810 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd88006d9a0 con 0x7fd8900b4010 2022-06-29T15:58:28.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.521+0000 7fd88e59c700 1 --2- 172.21.15.60:0/1501057810 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd86c053b00 0x7fd86c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.521+0000 7fd88ed9d700 1 --2- 172.21.15.60:0/1501057810 >> [v2:172.21.15.60:6808/12891,v1:172.21.15.60:6809/12891] conn(0x7fd86c05ade0 0x7fd86c05d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.539 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.521+0000 7fd88ed9d700 1 --2- 172.21.15.60:0/1501057810 >> [v2:172.21.15.60:6808/12891,v1:172.21.15.60:6809/12891] conn(0x7fd86c05ade0 0x7fd86c05d240 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:28.539 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.521+0000 7fd88e59c700 1 --2- 172.21.15.60:0/1501057810 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd86c053b00 0x7fd86c055fc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fd88400d000 tx=0x7fd884009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:28.539 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.525+0000 7fd87f7fe700 1 -- 172.21.15.60:0/1501057810 <== osd.2 v2:172.21.15.60:6808/12891 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7fd86c05d820 con 0x7fd86c05ade0 2022-06-29T15:58:28.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.541+0000 7fd894f64700 1 -- 172.21.15.60:0/1501057810 --> [v2:172.21.15.60:6808/12891,v1:172.21.15.60:6809/12891] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd8900b1e40 con 0x7fd86c05ade0 2022-06-29T15:58:28.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.541+0000 7fd87f7fe700 1 -- 172.21.15.60:0/1501057810 <== osd.2 v2:172.21.15.60:6808/12891 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd888006cc0 con 0x7fd86c05ade0 2022-06-29T15:58:28.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.541+0000 7fd894f64700 1 -- 172.21.15.60:0/1501057810 >> [v2:172.21.15.60:6808/12891,v1:172.21.15.60:6809/12891] conn(0x7fd86c05ade0 msgr2=0x7fd86c05d240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.541+0000 7fd894f64700 1 --2- 172.21.15.60:0/1501057810 >> [v2:172.21.15.60:6808/12891,v1:172.21.15.60:6809/12891] conn(0x7fd86c05ade0 0x7fd86c05d240 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:28.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.541+0000 7fd894f64700 1 -- 172.21.15.60:0/1501057810 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd86c053b00 msgr2=0x7fd86c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.541+0000 7fd894f64700 1 --2- 172.21.15.60:0/1501057810 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd86c053b00 0x7fd86c055fc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fd88400d000 tx=0x7fd884009040).stop 2022-06-29T15:58:28.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.541+0000 7fd894f64700 1 -- 172.21.15.60:0/1501057810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8900b4010 msgr2=0x7fd8900af610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.541+0000 7fd894f64700 1 --2- 172.21.15.60:0/1501057810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8900b4010 0x7fd8900af610 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fd880001110 tx=0x7fd880007ae0).stop 2022-06-29T15:58:28.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.541+0000 7fd88ed9d700 1 -- 172.21.15.60:0/1501057810 reap_dead start 2022-06-29T15:58:28.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.541+0000 7fd894f64700 1 -- 172.21.15.60:0/1501057810 shutdown_connections 2022-06-29T15:58:28.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.541+0000 7fd894f64700 1 -- 172.21.15.60:0/1501057810 >> 172.21.15.60:0/1501057810 conn(0x7fd8900b9340 msgr2=0x7fd890141d70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:28.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.541+0000 7fd894f64700 1 -- 172.21.15.60:0/1501057810 shutdown_connections 2022-06-29T15:58:28.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.541+0000 7fd894f64700 1 -- 172.21.15.60:0/1501057810 wait complete. 2022-06-29T15:58:28.564 INFO:teuthology.orchestra.run.smithi060.stdout:55834574854 2022-06-29T15:58:28.564 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2022-06-29T15:58:28.676 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.669+0000 7f9bb4dac700 1 Processor -- start 2022-06-29T15:58:28.676 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.669+0000 7f9bb4dac700 1 -- start start 2022-06-29T15:58:28.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.669+0000 7f9bb4dac700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9bb005d070 0x7f9bb014aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.669+0000 7f9bb4dac700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bb014af80 0x7f9bb00ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.669+0000 7f9bb4dac700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bb005d450 con 0x7f9bb00b2130 2022-06-29T15:58:28.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.669+0000 7f9bb4dac700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bb00adf90 con 0x7f9bb005d070 2022-06-29T15:58:28.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.669+0000 7f9bb4dac700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb00b25d0 con 0x7f9bb014af80 2022-06-29T15:58:28.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.669+0000 7f9baf7fe700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9bb005d070 0x7f9bb014aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.673+0000 7f9baffff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bb014af80 0x7f9bb00ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.673+0000 7f9baf7fe700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9bb005d070 0x7f9bb014aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:51184/0 (socket says 172.21.15.60:51184) 2022-06-29T15:58:28.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.673+0000 7f9baf7fe700 1 -- 172.21.15.60:0/3834419966 learned_addr learned my addr 172.21.15.60:0/3834419966 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:28.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.673+0000 7f9baeffd700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f9bb00b2130 0x7f9bb00aca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:57490/0 (socket says 172.21.15.60:57490) 2022-06-29T15:58:28.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.673+0000 7f9baf7fe700 1 -- 172.21.15.60:0/3834419966 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bb014af80 msgr2=0x7f9bb00ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:28.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.673+0000 7f9baf7fe700 1 --2- 172.21.15.60:0/3834419966 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bb014af80 0x7f9bb00ad130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:28.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.673+0000 7f9baf7fe700 1 -- 172.21.15.60:0/3834419966 >> v1:172.21.15.92:6789/0 conn(0x7f9bb00b2130 legacy=0x7f9bb00aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.673+0000 7f9baf7fe700 1 -- 172.21.15.60:0/3834419966 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb014a350 con 0x7f9bb005d070 2022-06-29T15:58:28.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.673+0000 7f9baf7fe700 1 --2- 172.21.15.60:0/3834419966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bb005d070 0x7f9bb014aa90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f9ba80032f0 tx=0x7f9ba8009290).ready entity=mon.1 client_cookie=9e86a7eebade463d server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:28.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.673+0000 7f9bae7fc700 1 -- 172.21.15.60:0/3834419966 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ba800b8a0 con 0x7f9bb005d070 2022-06-29T15:58:28.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.673+0000 7f9bae7fc700 1 -- 172.21.15.60:0/3834419966 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ba800ba00 con 0x7f9bb005d070 2022-06-29T15:58:28.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.673+0000 7f9bae7fc700 1 -- 172.21.15.60:0/3834419966 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ba800bb70 con 0x7f9bb005d070 2022-06-29T15:58:28.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.673+0000 7f9bb4dac700 1 -- 172.21.15.60:0/3834419966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bb005d070 msgr2=0x7f9bb014aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.673+0000 7f9bb4dac700 1 --2- 172.21.15.60:0/3834419966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bb005d070 0x7f9bb014aa90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f9ba80032f0 tx=0x7f9ba8009290).stop 2022-06-29T15:58:28.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.673+0000 7f9bb4dac700 1 -- 172.21.15.60:0/3834419966 shutdown_connections 2022-06-29T15:58:28.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.673+0000 7f9bb4dac700 1 --2- 172.21.15.60:0/3834419966 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bb014af80 0x7f9bb00ad130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:28.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.673+0000 7f9bb4dac700 1 --2- 172.21.15.60:0/3834419966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bb005d070 0x7f9bb014aa90 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:28.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.673+0000 7f9bb4dac700 1 -- 172.21.15.60:0/3834419966 >> 172.21.15.60:0/3834419966 conn(0x7f9bb00b5890 msgr2=0x7f9bb00b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:28.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.673+0000 7f9bb4dac700 1 -- 172.21.15.60:0/3834419966 shutdown_connections 2022-06-29T15:58:28.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.673+0000 7f9bb4dac700 1 -- 172.21.15.60:0/3834419966 wait complete. 2022-06-29T15:58:28.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.673+0000 7f9bb4dac700 1 Processor -- start 2022-06-29T15:58:28.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.673+0000 7f9bb4dac700 1 -- start start 2022-06-29T15:58:28.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.677+0000 7f9bb4dac700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bb005d070 0x7f9bb01598e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.677+0000 7f9bb4dac700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bb00b2130 0x7f9bb00663d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.677+0000 7f9bb4dac700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bb014af80 0x7f9bb00b45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.677+0000 7f9bb4dac700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb0065650 con 0x7f9bb005d070 2022-06-29T15:58:28.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.677+0000 7f9bb4dac700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb00658f0 con 0x7f9bb014af80 2022-06-29T15:58:28.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.677+0000 7f9bb4dac700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb00b4aa0 con 0x7f9bb00b2130 2022-06-29T15:58:28.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.677+0000 7f9baf7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bb005d070 0x7f9bb01598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.677+0000 7f9baf7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bb005d070 0x7f9bb01598e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45160/0 (socket says 172.21.15.60:45160) 2022-06-29T15:58:28.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.677+0000 7f9baf7fe700 1 -- 172.21.15.60:0/1708657767 learned_addr learned my addr 172.21.15.60:0/1708657767 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:28.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.677+0000 7f9baffff700 1 --2- 172.21.15.60:0/1708657767 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bb014af80 0x7f9bb00b45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.677+0000 7f9baeffd700 1 --2- 172.21.15.60:0/1708657767 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bb00b2130 0x7f9bb00663d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.677+0000 7f9baf7fe700 1 -- 172.21.15.60:0/1708657767 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bb00b2130 msgr2=0x7f9bb00663d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.677+0000 7f9baf7fe700 1 --2- 172.21.15.60:0/1708657767 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bb00b2130 0x7f9bb00663d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:28.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.677+0000 7f9baf7fe700 1 -- 172.21.15.60:0/1708657767 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bb014af80 msgr2=0x7f9bb00b45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.677+0000 7f9baf7fe700 1 --2- 172.21.15.60:0/1708657767 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bb014af80 0x7f9bb00b45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:28.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.677+0000 7f9baf7fe700 1 -- 172.21.15.60:0/1708657767 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba8009530 con 0x7f9bb005d070 2022-06-29T15:58:28.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.677+0000 7f9baf7fe700 1 --2- 172.21.15.60:0/1708657767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bb005d070 0x7f9bb01598e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f9ba80032f0 tx=0x7f9ba8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:28.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.677+0000 7f9bacff9700 1 -- 172.21.15.60:0/1708657767 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ba800b8a0 con 0x7f9bb005d070 2022-06-29T15:58:28.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.677+0000 7f9bacff9700 1 -- 172.21.15.60:0/1708657767 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ba8021440 con 0x7f9bb005d070 2022-06-29T15:58:28.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.677+0000 7f9bb4dac700 1 -- 172.21.15.60:0/1708657767 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bb0066b60 con 0x7f9bb005d070 2022-06-29T15:58:28.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.677+0000 7f9bacff9700 1 -- 172.21.15.60:0/1708657767 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ba80217a0 con 0x7f9bb005d070 2022-06-29T15:58:28.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.677+0000 7f9bb4dac700 1 -- 172.21.15.60:0/1708657767 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bb0155760 con 0x7f9bb005d070 2022-06-29T15:58:28.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.677+0000 7f9bb4dac700 1 -- 172.21.15.60:0/1708657767 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9bb014a5f0 con 0x7f9bb005d070 2022-06-29T15:58:28.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.681+0000 7f9bacff9700 1 -- 172.21.15.60:0/1708657767 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9ba800f070 con 0x7f9bb005d070 2022-06-29T15:58:28.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.681+0000 7f9bacff9700 1 --2- 172.21.15.60:0/1708657767 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9b90053aa0 0x7f9b90055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.681+0000 7f9bacff9700 1 -- 172.21.15.60:0/1708657767 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9ba806d0a0 con 0x7f9bb005d070 2022-06-29T15:58:28.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.681+0000 7f9bacff9700 1 --2- 172.21.15.60:0/1708657767 >> [v2:172.21.15.60:6800/12889,v1:172.21.15.60:6801/12889] conn(0x7f9b9005ae00 0x7f9b9005d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.681+0000 7f9bacff9700 1 -- 172.21.15.60:0/1708657767 --> [v2:172.21.15.60:6800/12889,v1:172.21.15.60:6801/12889] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b9005d840 con 0x7f9b9005ae00 2022-06-29T15:58:28.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.681+0000 7f9bacff9700 1 -- 172.21.15.60:0/1708657767 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ba806f730 con 0x7f9bb005d070 2022-06-29T15:58:28.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.681+0000 7f9baeffd700 1 --2- 172.21.15.60:0/1708657767 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9b90053aa0 0x7f9b90055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.681+0000 7f9baffff700 1 --2- 172.21.15.60:0/1708657767 >> [v2:172.21.15.60:6800/12889,v1:172.21.15.60:6801/12889] conn(0x7f9b9005ae00 0x7f9b9005d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.681+0000 7f9baeffd700 1 --2- 172.21.15.60:0/1708657767 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9b90053aa0 0x7f9b90055f60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f9ba400d000 tx=0x7f9ba4007b80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:28.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.681+0000 7f9baffff700 1 --2- 172.21.15.60:0/1708657767 >> [v2:172.21.15.60:6800/12889,v1:172.21.15.60:6801/12889] conn(0x7f9b9005ae00 0x7f9b9005d260 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:28.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.681+0000 7f9bacff9700 1 -- 172.21.15.60:0/1708657767 <== osd.3 v2:172.21.15.60:6800/12889 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f9b9005d840 con 0x7f9b9005ae00 2022-06-29T15:58:28.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.701+0000 7f9bb4dac700 1 -- 172.21.15.60:0/1708657767 --> [v2:172.21.15.60:6800/12889,v1:172.21.15.60:6801/12889] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9bb0065440 con 0x7f9b9005ae00 2022-06-29T15:58:28.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.701+0000 7f9bacff9700 1 -- 172.21.15.60:0/1708657767 <== osd.3 v2:172.21.15.60:6800/12889 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9ba0000e90 con 0x7f9b9005ae00 2022-06-29T15:58:28.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.701+0000 7f9bb4dac700 1 -- 172.21.15.60:0/1708657767 >> [v2:172.21.15.60:6800/12889,v1:172.21.15.60:6801/12889] conn(0x7f9b9005ae00 msgr2=0x7f9b9005d260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.701+0000 7f9bb4dac700 1 --2- 172.21.15.60:0/1708657767 >> [v2:172.21.15.60:6800/12889,v1:172.21.15.60:6801/12889] conn(0x7f9b9005ae00 0x7f9b9005d260 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:28.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.701+0000 7f9bb4dac700 1 -- 172.21.15.60:0/1708657767 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9b90053aa0 msgr2=0x7f9b90055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.701+0000 7f9bb4dac700 1 --2- 172.21.15.60:0/1708657767 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9b90053aa0 0x7f9b90055f60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f9ba400d000 tx=0x7f9ba4007b80).stop 2022-06-29T15:58:28.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.701+0000 7f9bb4dac700 1 -- 172.21.15.60:0/1708657767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bb005d070 msgr2=0x7f9bb01598e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.701+0000 7f9bb4dac700 1 --2- 172.21.15.60:0/1708657767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bb005d070 0x7f9bb01598e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f9ba80032f0 tx=0x7f9ba8016000).stop 2022-06-29T15:58:28.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.701+0000 7f9baffff700 1 -- 172.21.15.60:0/1708657767 reap_dead start 2022-06-29T15:58:28.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.701+0000 7f9bb4dac700 1 -- 172.21.15.60:0/1708657767 shutdown_connections 2022-06-29T15:58:28.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.701+0000 7f9bb4dac700 1 -- 172.21.15.60:0/1708657767 >> 172.21.15.60:0/1708657767 conn(0x7f9bb00b5890 msgr2=0x7f9bb0146f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:28.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.701+0000 7f9bb4dac700 1 -- 172.21.15.60:0/1708657767 shutdown_connections 2022-06-29T15:58:28.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.701+0000 7f9bb4dac700 1 -- 172.21.15.60:0/1708657767 wait complete. 2022-06-29T15:58:28.720 INFO:teuthology.orchestra.run.smithi060.stdout:55834574853 2022-06-29T15:58:28.720 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2022-06-29T15:58:28.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.829+0000 7f0c77fff700 1 Processor -- start 2022-06-29T15:58:28.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.829+0000 7f0c77fff700 1 -- start start 2022-06-29T15:58:28.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.829+0000 7f0c77fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0c780aca20 0x7f0c780ace00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.829+0000 7f0c77fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0c780b3270 0x7f0c78062cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.829+0000 7f0c77fff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c780adc80 con 0x7f0c78143030 2022-06-29T15:58:28.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.829+0000 7f0c77fff700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c780ad740 con 0x7f0c780b3270 2022-06-29T15:58:28.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.829+0000 7f0c77fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c780ad9e0 con 0x7f0c780aca20 2022-06-29T15:58:28.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.829+0000 7f0c76ffd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0c780aca20 0x7f0c780ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.829+0000 7f0c6e7fc700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0c780b3270 0x7f0c78062cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.829+0000 7f0c777fe700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f0c78143030 0x7f0c781461c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:57508/0 (socket says 172.21.15.60:57508) 2022-06-29T15:58:28.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.829+0000 7f0c777fe700 1 -- 172.21.15.60:0/4221607772 learned_addr learned my addr 172.21.15.60:0/4221607772 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:28.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.829+0000 7f0c76ffd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0c780aca20 0x7f0c780ace00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:58264/0 (socket says 172.21.15.60:58264) 2022-06-29T15:58:28.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.829+0000 7f0c76ffd700 1 -- 172.21.15.60:0/4221607772 >> v2:172.21.15.92:3300/0 conn(0x7f0c780b3270 msgr2=0x7f0c78062cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:28.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.829+0000 7f0c76ffd700 1 --2- 172.21.15.60:0/4221607772 >> v2:172.21.15.92:3300/0 conn(0x7f0c780b3270 0x7f0c78062cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:28.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.829+0000 7f0c76ffd700 1 -- 172.21.15.60:0/4221607772 >> v1:172.21.15.92:6789/0 conn(0x7f0c78143030 legacy=0x7f0c781461c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.829+0000 7f0c76ffd700 1 -- 172.21.15.60:0/4221607772 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c78142920 con 0x7f0c780aca20 2022-06-29T15:58:28.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.829+0000 7f0c6e7fc700 1 --2- 172.21.15.60:0/4221607772 >> v2:172.21.15.92:3300/0 conn(0x7f0c780b3270 0x7f0c78062cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:28.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.829+0000 7f0c76ffd700 1 --2- 172.21.15.60:0/4221607772 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0c780aca20 0x7f0c780ace00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f0c68002f70 tx=0x7f0c68006df0).ready entity=mon.2 client_cookie=5fe248d202ae2d4d server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:28.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.829+0000 7f0c767fc700 1 -- 172.21.15.60:0/4221607772 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c6800d570 con 0x7f0c780aca20 2022-06-29T15:58:28.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.829+0000 7f0c767fc700 1 -- 172.21.15.60:0/4221607772 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c68000e90 con 0x7f0c780aca20 2022-06-29T15:58:28.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.829+0000 7f0c767fc700 1 -- 172.21.15.60:0/4221607772 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c6800d8c0 con 0x7f0c780aca20 2022-06-29T15:58:28.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.829+0000 7f0c77fff700 1 -- 172.21.15.60:0/4221607772 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0c780aca20 msgr2=0x7f0c780ace00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.843 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.829+0000 7f0c77fff700 1 --2- 172.21.15.60:0/4221607772 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0c780aca20 0x7f0c780ace00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f0c68002f70 tx=0x7f0c68006df0).stop 2022-06-29T15:58:28.843 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.829+0000 7f0c77fff700 1 -- 172.21.15.60:0/4221607772 shutdown_connections 2022-06-29T15:58:28.843 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.829+0000 7f0c77fff700 1 --2- 172.21.15.60:0/4221607772 >> v2:172.21.15.92:3300/0 conn(0x7f0c780b3270 0x7f0c78062cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:28.843 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.829+0000 7f0c77fff700 1 --2- 172.21.15.60:0/4221607772 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0c780aca20 0x7f0c780ace00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:28.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.829+0000 7f0c77fff700 1 -- 172.21.15.60:0/4221607772 >> 172.21.15.60:0/4221607772 conn(0x7f0c780b6f80 msgr2=0x7f0c780b7390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:28.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.829+0000 7f0c77fff700 1 -- 172.21.15.60:0/4221607772 shutdown_connections 2022-06-29T15:58:28.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.829+0000 7f0c77fff700 1 -- 172.21.15.60:0/4221607772 wait complete. 2022-06-29T15:58:28.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.833+0000 7f0c77fff700 1 Processor -- start 2022-06-29T15:58:28.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.833+0000 7f0c77fff700 1 -- start start 2022-06-29T15:58:28.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.833+0000 7f0c77fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0c780aca20 0x7f0c78152d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.833+0000 7f0c77fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0c780b3270 0x7f0c78064ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.833+0000 7f0c77fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c78143030 0x7f0c78065390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.833+0000 7f0c77fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c78065880 con 0x7f0c780aca20 2022-06-29T15:58:28.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.833+0000 7f0c77fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c78153210 con 0x7f0c78143030 2022-06-29T15:58:28.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.833+0000 7f0c77fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c780aeab0 con 0x7f0c780b3270 2022-06-29T15:58:28.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.833+0000 7f0c76ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0c780aca20 0x7f0c78152d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.833+0000 7f0c6e7fc700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0c780b3270 0x7f0c78064ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.833+0000 7f0c76ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0c780aca20 0x7f0c78152d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45176/0 (socket says 172.21.15.60:45176) 2022-06-29T15:58:28.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.833+0000 7f0c76ffd700 1 -- 172.21.15.60:0/852147477 learned_addr learned my addr 172.21.15.60:0/852147477 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:28.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.833+0000 7f0c777fe700 1 --2- 172.21.15.60:0/852147477 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c78143030 0x7f0c78065390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.833+0000 7f0c6e7fc700 1 -- 172.21.15.60:0/852147477 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c78143030 msgr2=0x7f0c78065390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.833+0000 7f0c6e7fc700 1 --2- 172.21.15.60:0/852147477 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c78143030 0x7f0c78065390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:28.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.833+0000 7f0c6e7fc700 1 -- 172.21.15.60:0/852147477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0c780aca20 msgr2=0x7f0c78152d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.833+0000 7f0c6e7fc700 1 --2- 172.21.15.60:0/852147477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0c780aca20 0x7f0c78152d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:28.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.833+0000 7f0c6e7fc700 1 -- 172.21.15.60:0/852147477 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c6000b180 con 0x7f0c780b3270 2022-06-29T15:58:28.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.833+0000 7f0c6e7fc700 1 --2- 172.21.15.60:0/852147477 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0c780b3270 0x7f0c78064ea0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f0c6000c000 tx=0x7f0c60006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:28.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.837+0000 7f0c74ff9700 1 -- 172.21.15.60:0/852147477 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c60010780 con 0x7f0c780b3270 2022-06-29T15:58:28.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.837+0000 7f0c74ff9700 1 -- 172.21.15.60:0/852147477 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c600108e0 con 0x7f0c780b3270 2022-06-29T15:58:28.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.837+0000 7f0c77fff700 1 -- 172.21.15.60:0/852147477 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c6800b180 con 0x7f0c780b3270 2022-06-29T15:58:28.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.837+0000 7f0c74ff9700 1 -- 172.21.15.60:0/852147477 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c60010c40 con 0x7f0c780b3270 2022-06-29T15:58:28.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.837+0000 7f0c77fff700 1 -- 172.21.15.60:0/852147477 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c780ae810 con 0x7f0c780b3270 2022-06-29T15:58:28.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.837+0000 7f0c77fff700 1 -- 172.21.15.60:0/852147477 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0c780af7e0 con 0x7f0c780b3270 2022-06-29T15:58:28.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.837+0000 7f0c74ff9700 1 -- 172.21.15.60:0/852147477 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0c600334d0 con 0x7f0c780b3270 2022-06-29T15:58:28.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.837+0000 7f0c74ff9700 1 --2- 172.21.15.60:0/852147477 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0c5c053b30 0x7f0c5c055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.837+0000 7f0c74ff9700 1 -- 172.21.15.60:0/852147477 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0c6006c510 con 0x7f0c780b3270 2022-06-29T15:58:28.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.837+0000 7f0c76ffd700 1 --2- 172.21.15.60:0/852147477 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0c5c053b30 0x7f0c5c055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.837+0000 7f0c74ff9700 1 --2- 172.21.15.60:0/852147477 >> [v2:172.21.15.92:6808/13249,v1:172.21.15.92:6810/13249] conn(0x7f0c5c05ae90 0x7f0c5c05d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.837+0000 7f0c74ff9700 1 -- 172.21.15.60:0/852147477 --> [v2:172.21.15.92:6808/13249,v1:172.21.15.92:6810/13249] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0c5c05d8d0 con 0x7f0c5c05ae90 2022-06-29T15:58:28.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.837+0000 7f0c74ff9700 1 -- 172.21.15.60:0/852147477 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c6006c7e0 con 0x7f0c780b3270 2022-06-29T15:58:28.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.841+0000 7f0c777fe700 1 --2- 172.21.15.60:0/852147477 >> [v2:172.21.15.92:6808/13249,v1:172.21.15.92:6810/13249] conn(0x7f0c5c05ae90 0x7f0c5c05d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:28.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.841+0000 7f0c76ffd700 1 --2- 172.21.15.60:0/852147477 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0c5c053b30 0x7f0c5c055ff0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f0c6801a000 tx=0x7f0c68014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:28.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.841+0000 7f0c777fe700 1 --2- 172.21.15.60:0/852147477 >> [v2:172.21.15.92:6808/13249,v1:172.21.15.92:6810/13249] conn(0x7f0c5c05ae90 0x7f0c5c05d2f0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:28.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.841+0000 7f0c74ff9700 1 -- 172.21.15.60:0/852147477 <== osd.4 v2:172.21.15.92:6808/13249 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f0c5c05d8d0 con 0x7f0c5c05ae90 2022-06-29T15:58:28.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.861+0000 7f0c77fff700 1 -- 172.21.15.60:0/852147477 --> [v2:172.21.15.92:6808/13249,v1:172.21.15.92:6810/13249] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0c780b5170 con 0x7f0c5c05ae90 2022-06-29T15:58:28.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.861+0000 7f0c74ff9700 1 -- 172.21.15.60:0/852147477 <== osd.4 v2:172.21.15.92:6808/13249 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0c780b5170 con 0x7f0c5c05ae90 2022-06-29T15:58:28.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.861+0000 7f0c77fff700 1 -- 172.21.15.60:0/852147477 >> [v2:172.21.15.92:6808/13249,v1:172.21.15.92:6810/13249] conn(0x7f0c5c05ae90 msgr2=0x7f0c5c05d2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.861+0000 7f0c77fff700 1 --2- 172.21.15.60:0/852147477 >> [v2:172.21.15.92:6808/13249,v1:172.21.15.92:6810/13249] conn(0x7f0c5c05ae90 0x7f0c5c05d2f0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:28.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.861+0000 7f0c77fff700 1 -- 172.21.15.60:0/852147477 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0c5c053b30 msgr2=0x7f0c5c055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.861+0000 7f0c77fff700 1 --2- 172.21.15.60:0/852147477 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0c5c053b30 0x7f0c5c055ff0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f0c6801a000 tx=0x7f0c68014000).stop 2022-06-29T15:58:28.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.861+0000 7f0c77fff700 1 -- 172.21.15.60:0/852147477 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0c780b3270 msgr2=0x7f0c78064ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:28.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.861+0000 7f0c77fff700 1 --2- 172.21.15.60:0/852147477 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0c780b3270 0x7f0c78064ea0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f0c6000c000 tx=0x7f0c60006df0).stop 2022-06-29T15:58:28.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.861+0000 7f0c777fe700 1 -- 172.21.15.60:0/852147477 reap_dead start 2022-06-29T15:58:28.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.861+0000 7f0c77fff700 1 -- 172.21.15.60:0/852147477 shutdown_connections 2022-06-29T15:58:28.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.861+0000 7f0c77fff700 1 -- 172.21.15.60:0/852147477 >> 172.21.15.60:0/852147477 conn(0x7f0c780b6f80 msgr2=0x7f0c780b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:28.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.861+0000 7f0c77fff700 1 -- 172.21.15.60:0/852147477 shutdown_connections 2022-06-29T15:58:28.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.861+0000 7f0c77fff700 1 -- 172.21.15.60:0/852147477 wait complete. 2022-06-29T15:58:28.880 INFO:teuthology.orchestra.run.smithi060.stdout:55834574854 2022-06-29T15:58:28.881 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2022-06-29T15:58:28.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.989+0000 7f721d056700 1 Processor -- start 2022-06-29T15:58:28.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.989+0000 7f721d056700 1 -- start start 2022-06-29T15:58:28.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.989+0000 7f721d056700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f72180aed00 0x7f7218146810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:28.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.989+0000 7f721d056700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f721805d450 con 0x7f721805d070 2022-06-29T15:58:28.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.989+0000 7f721d056700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72180b25d0 con 0x7f72180b2130 2022-06-29T15:58:28.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.989+0000 7f721d056700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72180adf90 con 0x7f72180aed00 2022-06-29T15:58:28.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.989+0000 7f7217fff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f721805d070 0x7f721814aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53272/0 (socket says 172.21.15.60:53272) 2022-06-29T15:58:29.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.989+0000 7f7217fff700 1 -- 172.21.15.60:0/3891315001 learned_addr learned my addr 172.21.15.60:0/3891315001 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:29.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.989+0000 7f721c855700 1 --2- 172.21.15.60:0/3891315001 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f72180aed00 0x7f7218146810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.989+0000 7f7216ffd700 1 -- 172.21.15.60:0/3891315001 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 330789822 0 0) 0x7f72180b25d0 con 0x7f72180b2130 2022-06-29T15:58:29.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.989+0000 7f7216ffd700 1 -- 172.21.15.60:0/3891315001 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71f8003260 con 0x7f72180b2130 2022-06-29T15:58:29.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.989+0000 7f7216ffd700 1 -- 172.21.15.60:0/3891315001 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 669999265 0 0) 0x7f721805d450 con 0x7f721805d070 2022-06-29T15:58:29.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.989+0000 7f7216ffd700 1 -- 172.21.15.60:0/3891315001 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72180b25d0 con 0x7f721805d070 2022-06-29T15:58:29.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.989+0000 7f721c855700 1 -- 172.21.15.60:0/3891315001 >> v1:172.21.15.92:6789/0 conn(0x7f72180b2130 legacy=0x7f7218146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:29.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.989+0000 7f721c855700 1 -- 172.21.15.60:0/3891315001 >> v1:172.21.15.60:6789/0 conn(0x7f721805d070 legacy=0x7f721814aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:29.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.989+0000 7f721c855700 1 -- 172.21.15.60:0/3891315001 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721814a350 con 0x7f72180aed00 2022-06-29T15:58:29.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.989+0000 7f721c855700 1 --2- 172.21.15.60:0/3891315001 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f72180aed00 0x7f7218146810 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f72100032f0 tx=0x7f72100091d0).ready entity=mon.2 client_cookie=968a9eb9f34d75ba server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:29.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.989+0000 7f7216ffd700 1 -- 172.21.15.60:0/3891315001 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f721000b640 con 0x7f72180aed00 2022-06-29T15:58:29.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.989+0000 7f7216ffd700 1 -- 172.21.15.60:0/3891315001 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f721000b7a0 con 0x7f72180aed00 2022-06-29T15:58:29.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.989+0000 7f7216ffd700 1 -- 172.21.15.60:0/3891315001 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f721000b910 con 0x7f72180aed00 2022-06-29T15:58:29.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.993+0000 7f721d056700 1 -- 172.21.15.60:0/3891315001 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f72180aed00 msgr2=0x7f7218146810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:29.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.993+0000 7f721d056700 1 --2- 172.21.15.60:0/3891315001 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f72180aed00 0x7f7218146810 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f72100032f0 tx=0x7f72100091d0).stop 2022-06-29T15:58:29.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.993+0000 7f721d056700 1 -- 172.21.15.60:0/3891315001 shutdown_connections 2022-06-29T15:58:29.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.993+0000 7f721d056700 1 --2- 172.21.15.60:0/3891315001 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f72180aed00 0x7f7218146810 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.993+0000 7f721d056700 1 -- 172.21.15.60:0/3891315001 >> 172.21.15.60:0/3891315001 conn(0x7f72180b5890 msgr2=0x7f72180b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:29.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.993+0000 7f721d056700 1 -- 172.21.15.60:0/3891315001 shutdown_connections 2022-06-29T15:58:29.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.993+0000 7f721d056700 1 -- 172.21.15.60:0/3891315001 wait complete. 2022-06-29T15:58:29.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.993+0000 7f721d056700 1 Processor -- start 2022-06-29T15:58:29.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.993+0000 7f721d056700 1 -- start start 2022-06-29T15:58:29.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.997+0000 7f721d056700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f721805d070 0x7f72180661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.997+0000 7f721d056700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f72180aed00 0x7f72180b3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.997+0000 7f721d056700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72180b2130 0x7f7218063470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.997+0000 7f721d056700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7218066d00 con 0x7f72180b2130 2022-06-29T15:58:29.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.997+0000 7f721d056700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721814c5e0 con 0x7f72180aed00 2022-06-29T15:58:29.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.997+0000 7f721d056700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f721814d090 con 0x7f721805d070 2022-06-29T15:58:29.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.997+0000 7f721c855700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72180b2130 0x7f7218063470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.997+0000 7f721c855700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72180b2130 0x7f7218063470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45194/0 (socket says 172.21.15.60:45194) 2022-06-29T15:58:29.016 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.997+0000 7f721c855700 1 -- 172.21.15.60:0/2635353935 learned_addr learned my addr 172.21.15.60:0/2635353935 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:29.016 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.997+0000 7f7217fff700 1 --2- 172.21.15.60:0/2635353935 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f721805d070 0x7f72180661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.016 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.997+0000 7f72177fe700 1 --2- 172.21.15.60:0/2635353935 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f72180aed00 0x7f72180b3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.997+0000 7f721c855700 1 -- 172.21.15.60:0/2635353935 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f721805d070 msgr2=0x7f72180661e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:29.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.997+0000 7f721c855700 1 --2- 172.21.15.60:0/2635353935 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f721805d070 0x7f72180661e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.997+0000 7f721c855700 1 -- 172.21.15.60:0/2635353935 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f72180aed00 msgr2=0x7f72180b3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:29.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.997+0000 7f721c855700 1 --2- 172.21.15.60:0/2635353935 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f72180aed00 0x7f72180b3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.997+0000 7f721c855700 1 -- 172.21.15.60:0/2635353935 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7210009470 con 0x7f72180b2130 2022-06-29T15:58:29.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.997+0000 7f72177fe700 1 --2- 172.21.15.60:0/2635353935 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f72180aed00 0x7f72180b3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:58:29.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.997+0000 7f721c855700 1 --2- 172.21.15.60:0/2635353935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72180b2130 0x7f7218063470 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f72100032f0 tx=0x7f7210016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:29.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.997+0000 7f72157fa700 1 -- 172.21.15.60:0/2635353935 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f721000b910 con 0x7f72180b2130 2022-06-29T15:58:29.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.997+0000 7f72157fa700 1 -- 172.21.15.60:0/2635353935 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f721001f3f0 con 0x7f72180b2130 2022-06-29T15:58:29.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.997+0000 7f721d056700 1 -- 172.21.15.60:0/2635353935 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7218066700 con 0x7f72180b2130 2022-06-29T15:58:29.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.997+0000 7f72157fa700 1 -- 172.21.15.60:0/2635353935 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f721000b640 con 0x7f72180b2130 2022-06-29T15:58:29.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.997+0000 7f721d056700 1 -- 172.21.15.60:0/2635353935 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7218063960 con 0x7f72180b2130 2022-06-29T15:58:29.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:28.997+0000 7f721d056700 1 -- 172.21.15.60:0/2635353935 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f721814d2b0 con 0x7f72180b2130 2022-06-29T15:58:29.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.001+0000 7f72157fa700 1 -- 172.21.15.60:0/2635353935 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f721000f070 con 0x7f72180b2130 2022-06-29T15:58:29.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.001+0000 7f72157fa700 1 --2- 172.21.15.60:0/2635353935 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f71fc053a70 0x7f71fc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.001+0000 7f72157fa700 1 -- 172.21.15.60:0/2635353935 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f721006ca30 con 0x7f72180b2130 2022-06-29T15:58:29.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.001+0000 7f72157fa700 1 --2- 172.21.15.60:0/2635353935 >> [v2:172.21.15.92:6809/13248,v1:172.21.15.92:6811/13248] conn(0x7f71fc05ae20 0x7f71fc05d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.001+0000 7f72157fa700 1 -- 172.21.15.60:0/2635353935 --> [v2:172.21.15.92:6809/13248,v1:172.21.15.92:6811/13248] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f71fc05d860 con 0x7f71fc05ae20 2022-06-29T15:58:29.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.001+0000 7f72157fa700 1 -- 172.21.15.60:0/2635353935 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f721006eea0 con 0x7f72180b2130 2022-06-29T15:58:29.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.001+0000 7f7217fff700 1 --2- 172.21.15.60:0/2635353935 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f71fc053a70 0x7f71fc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.001+0000 7f72177fe700 1 --2- 172.21.15.60:0/2635353935 >> [v2:172.21.15.92:6809/13248,v1:172.21.15.92:6811/13248] conn(0x7f71fc05ae20 0x7f71fc05d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.001+0000 7f7217fff700 1 --2- 172.21.15.60:0/2635353935 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f71fc053a70 0x7f71fc055f30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f720000b000 tx=0x7f7200009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:29.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.001+0000 7f72177fe700 1 --2- 172.21.15.60:0/2635353935 >> [v2:172.21.15.92:6809/13248,v1:172.21.15.92:6811/13248] conn(0x7f71fc05ae20 0x7f71fc05d280 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:29.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.001+0000 7f72157fa700 1 -- 172.21.15.60:0/2635353935 <== osd.5 v2:172.21.15.92:6809/13248 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f71fc05d860 con 0x7f71fc05ae20 2022-06-29T15:58:29.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.013+0000 7f721d056700 1 -- 172.21.15.60:0/2635353935 --> [v2:172.21.15.92:6809/13248,v1:172.21.15.92:6811/13248] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f721814c970 con 0x7f71fc05ae20 2022-06-29T15:58:29.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.017+0000 7f72157fa700 1 -- 172.21.15.60:0/2635353935 <== osd.5 v2:172.21.15.92:6809/13248 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7208003640 con 0x7f71fc05ae20 2022-06-29T15:58:29.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.017+0000 7f721d056700 1 -- 172.21.15.60:0/2635353935 >> [v2:172.21.15.92:6809/13248,v1:172.21.15.92:6811/13248] conn(0x7f71fc05ae20 msgr2=0x7f71fc05d280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:29.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.017+0000 7f721d056700 1 --2- 172.21.15.60:0/2635353935 >> [v2:172.21.15.92:6809/13248,v1:172.21.15.92:6811/13248] conn(0x7f71fc05ae20 0x7f71fc05d280 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.017+0000 7f721d056700 1 -- 172.21.15.60:0/2635353935 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f71fc053a70 msgr2=0x7f71fc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:29.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.017+0000 7f721d056700 1 --2- 172.21.15.60:0/2635353935 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f71fc053a70 0x7f71fc055f30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f720000b000 tx=0x7f7200009040).stop 2022-06-29T15:58:29.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.017+0000 7f721d056700 1 -- 172.21.15.60:0/2635353935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72180b2130 msgr2=0x7f7218063470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:29.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.017+0000 7f721d056700 1 --2- 172.21.15.60:0/2635353935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72180b2130 0x7f7218063470 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f72100032f0 tx=0x7f7210016000).stop 2022-06-29T15:58:29.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.017+0000 7f721c855700 1 -- 172.21.15.60:0/2635353935 reap_dead start 2022-06-29T15:58:29.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.017+0000 7f721d056700 1 -- 172.21.15.60:0/2635353935 shutdown_connections 2022-06-29T15:58:29.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.017+0000 7f721d056700 1 -- 172.21.15.60:0/2635353935 >> 172.21.15.60:0/2635353935 conn(0x7f72180b5890 msgr2=0x7f72180b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:29.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.017+0000 7f721d056700 1 -- 172.21.15.60:0/2635353935 shutdown_connections 2022-06-29T15:58:29.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.017+0000 7f721d056700 1 -- 172.21.15.60:0/2635353935 wait complete. 2022-06-29T15:58:29.034 INFO:teuthology.orchestra.run.smithi060.stdout:55834574852 2022-06-29T15:58:29.035 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2022-06-29T15:58:29.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.133+0000 7fe309b5b700 1 Processor -- start 2022-06-29T15:58:29.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.133+0000 7fe309b5b700 1 -- start start 2022-06-29T15:58:29.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.137+0000 7fe309b5b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe3041426b0 0x7fe3040b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.137+0000 7fe309b5b700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe304143a40 con 0x7fe3040b4010 2022-06-29T15:58:29.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.137+0000 7fe309b5b700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3040aee60 con 0x7fe3040ae7c0 2022-06-29T15:58:29.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.137+0000 7fe309b5b700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3040af100 con 0x7fe3041426b0 2022-06-29T15:58:29.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.137+0000 7fe303fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe3041426b0 0x7fe3040b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.137+0000 7fe303fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe3041426b0 0x7fe3040b4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45206/0 (socket says 172.21.15.60:45206) 2022-06-29T15:58:29.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.137+0000 7fe303fff700 1 -- 172.21.15.60:0/1091750946 learned_addr learned my addr 172.21.15.60:0/1091750946 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:29.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.137+0000 7fe302ffd700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe3040b4010 0x7fe3040b46c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53290/0 (socket says 172.21.15.60:53290) 2022-06-29T15:58:29.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.137+0000 7fe303fff700 1 -- 172.21.15.60:0/1091750946 >> v1:172.21.15.92:6789/0 conn(0x7fe3040ae7c0 legacy=0x7fe304141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:29.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.137+0000 7fe303fff700 1 -- 172.21.15.60:0/1091750946 >> v1:172.21.15.60:6789/0 conn(0x7fe3040b4010 legacy=0x7fe3040b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:29.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.137+0000 7fe303fff700 1 -- 172.21.15.60:0/1091750946 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3040aeba0 con 0x7fe3041426b0 2022-06-29T15:58:29.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.137+0000 7fe3027fc700 1 -- 172.21.15.60:0/1091750946 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 533768869 0 0) 0x7fe304143a40 con 0x7fe3040b4010 2022-06-29T15:58:29.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.137+0000 7fe303fff700 1 --2- 172.21.15.60:0/1091750946 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3041426b0 0x7fe3040b4dd0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fe304060a90 tx=0x7fe2f4006eb0).ready entity=mon.0 client_cookie=888d4c1046c6843c server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:29.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.137+0000 7fe3027fc700 1 -- 172.21.15.60:0/1091750946 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe2f400d610 con 0x7fe3041426b0 2022-06-29T15:58:29.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.137+0000 7fe3027fc700 1 -- 172.21.15.60:0/1091750946 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f4000e90 con 0x7fe3041426b0 2022-06-29T15:58:29.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.137+0000 7fe3027fc700 1 -- 172.21.15.60:0/1091750946 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe2f400d960 con 0x7fe3041426b0 2022-06-29T15:58:29.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.137+0000 7fe309b5b700 1 -- 172.21.15.60:0/1091750946 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3041426b0 msgr2=0x7fe3040b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:29.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.137+0000 7fe309b5b700 1 --2- 172.21.15.60:0/1091750946 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3041426b0 0x7fe3040b4dd0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fe304060a90 tx=0x7fe2f4006eb0).stop 2022-06-29T15:58:29.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.137+0000 7fe309b5b700 1 -- 172.21.15.60:0/1091750946 shutdown_connections 2022-06-29T15:58:29.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.137+0000 7fe309b5b700 1 --2- 172.21.15.60:0/1091750946 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3041426b0 0x7fe3040b4dd0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.137+0000 7fe309b5b700 1 -- 172.21.15.60:0/1091750946 >> 172.21.15.60:0/1091750946 conn(0x7fe3040b9340 msgr2=0x7fe3040b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:29.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.137+0000 7fe309b5b700 1 -- 172.21.15.60:0/1091750946 shutdown_connections 2022-06-29T15:58:29.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.137+0000 7fe309b5b700 1 -- 172.21.15.60:0/1091750946 wait complete. 2022-06-29T15:58:29.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.141+0000 7fe309b5b700 1 Processor -- start 2022-06-29T15:58:29.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.141+0000 7fe309b5b700 1 -- start start 2022-06-29T15:58:29.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.141+0000 7fe309b5b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3040ae7c0 0x7fe304152050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.141+0000 7fe309b5b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3040b4010 0x7fe3040b8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.141+0000 7fe309b5b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3041426b0 0x7fe3041527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.141+0000 7fe309b5b700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3041556c0 con 0x7fe3040b4010 2022-06-29T15:58:29.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.141+0000 7fe309b5b700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe304115130 con 0x7fe3040ae7c0 2022-06-29T15:58:29.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.141+0000 7fe309b5b700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3041153d0 con 0x7fe3041426b0 2022-06-29T15:58:29.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.141+0000 7fe302ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3040b4010 0x7fe3040b8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.141+0000 7fe303fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3041426b0 0x7fe3041527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.141+0000 7fe302ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3040b4010 0x7fe3040b8700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45210/0 (socket says 172.21.15.60:45210) 2022-06-29T15:58:29.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.141+0000 7fe302ffd700 1 -- 172.21.15.60:0/3191575466 learned_addr learned my addr 172.21.15.60:0/3191575466 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:29.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.141+0000 7fe3037fe700 1 --2- 172.21.15.60:0/3191575466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3040ae7c0 0x7fe304152050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.141+0000 7fe303fff700 1 -- 172.21.15.60:0/3191575466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3040ae7c0 msgr2=0x7fe304152050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:29.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.141+0000 7fe303fff700 1 --2- 172.21.15.60:0/3191575466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3040ae7c0 0x7fe304152050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.141+0000 7fe303fff700 1 -- 172.21.15.60:0/3191575466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3040b4010 msgr2=0x7fe3040b8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:29.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.141+0000 7fe303fff700 1 --2- 172.21.15.60:0/3191575466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3040b4010 0x7fe3040b8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.141+0000 7fe303fff700 1 -- 172.21.15.60:0/3191575466 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f400b1e0 con 0x7fe3041426b0 2022-06-29T15:58:29.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.141+0000 7fe303fff700 1 --2- 172.21.15.60:0/3191575466 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3041426b0 0x7fe3041527b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fe304060a90 tx=0x7fe2f4015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:29.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.141+0000 7fe300ff9700 1 -- 172.21.15.60:0/3191575466 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe2f4002f70 con 0x7fe3041426b0 2022-06-29T15:58:29.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.141+0000 7fe300ff9700 1 -- 172.21.15.60:0/3191575466 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f400bec0 con 0x7fe3041426b0 2022-06-29T15:58:29.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.141+0000 7fe300ff9700 1 -- 172.21.15.60:0/3191575466 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe2f4011d70 con 0x7fe3041426b0 2022-06-29T15:58:29.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.141+0000 7fe309b5b700 1 -- 172.21.15.60:0/3191575466 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3040b2740 con 0x7fe3041426b0 2022-06-29T15:58:29.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.141+0000 7fe309b5b700 1 -- 172.21.15.60:0/3191575466 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3040b31f0 con 0x7fe3041426b0 2022-06-29T15:58:29.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.145+0000 7fe309b5b700 1 -- 172.21.15.60:0/3191575466 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe304155850 con 0x7fe3041426b0 2022-06-29T15:58:29.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.145+0000 7fe300ff9700 1 -- 172.21.15.60:0/3191575466 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe2f401a040 con 0x7fe3041426b0 2022-06-29T15:58:29.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.145+0000 7fe300ff9700 1 --2- 172.21.15.60:0/3191575466 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe2e8053b00 0x7fe2e8055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.145+0000 7fe300ff9700 1 -- 172.21.15.60:0/3191575466 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe2f406cf50 con 0x7fe3041426b0 2022-06-29T15:58:29.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.145+0000 7fe3037fe700 1 --2- 172.21.15.60:0/3191575466 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe2e8053b00 0x7fe2e8055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.145+0000 7fe300ff9700 1 --2- 172.21.15.60:0/3191575466 >> [v2:172.21.15.92:6816/13250,v1:172.21.15.92:6819/13250] conn(0x7fe2e805ade0 0x7fe2e805d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.145+0000 7fe300ff9700 1 -- 172.21.15.60:0/3191575466 --> [v2:172.21.15.92:6816/13250,v1:172.21.15.92:6819/13250] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2e805d820 con 0x7fe2e805ade0 2022-06-29T15:58:29.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.145+0000 7fe300ff9700 1 -- 172.21.15.60:0/3191575466 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe2f406f4f0 con 0x7fe3041426b0 2022-06-29T15:58:29.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.145+0000 7fe302ffd700 1 --2- 172.21.15.60:0/3191575466 >> [v2:172.21.15.92:6816/13250,v1:172.21.15.92:6819/13250] conn(0x7fe2e805ade0 0x7fe2e805d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.145+0000 7fe3037fe700 1 --2- 172.21.15.60:0/3191575466 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe2e8053b00 0x7fe2e8055fc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fe2fc00d000 tx=0x7fe2fc007ef0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:29.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.145+0000 7fe302ffd700 1 --2- 172.21.15.60:0/3191575466 >> [v2:172.21.15.92:6816/13250,v1:172.21.15.92:6819/13250] conn(0x7fe2e805ade0 0x7fe2e805d240 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:29.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.149+0000 7fe300ff9700 1 -- 172.21.15.60:0/3191575466 <== osd.6 v2:172.21.15.92:6816/13250 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7fe2e805d820 con 0x7fe2e805ade0 2022-06-29T15:58:29.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.169+0000 7fe309b5b700 1 -- 172.21.15.60:0/3191575466 --> [v2:172.21.15.92:6816/13250,v1:172.21.15.92:6819/13250] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe3040b2dc0 con 0x7fe2e805ade0 2022-06-29T15:58:29.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.169+0000 7fe300ff9700 1 -- 172.21.15.60:0/3191575466 <== osd.6 v2:172.21.15.92:6816/13250 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe3040b2dc0 con 0x7fe2e805ade0 2022-06-29T15:58:29.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.169+0000 7fe309b5b700 1 -- 172.21.15.60:0/3191575466 >> [v2:172.21.15.92:6816/13250,v1:172.21.15.92:6819/13250] conn(0x7fe2e805ade0 msgr2=0x7fe2e805d240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:29.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.169+0000 7fe309b5b700 1 --2- 172.21.15.60:0/3191575466 >> [v2:172.21.15.92:6816/13250,v1:172.21.15.92:6819/13250] conn(0x7fe2e805ade0 0x7fe2e805d240 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.169+0000 7fe309b5b700 1 -- 172.21.15.60:0/3191575466 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe2e8053b00 msgr2=0x7fe2e8055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:29.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.169+0000 7fe309b5b700 1 --2- 172.21.15.60:0/3191575466 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe2e8053b00 0x7fe2e8055fc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fe2fc00d000 tx=0x7fe2fc007ef0).stop 2022-06-29T15:58:29.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.169+0000 7fe309b5b700 1 -- 172.21.15.60:0/3191575466 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3041426b0 msgr2=0x7fe3041527b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:29.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.169+0000 7fe309b5b700 1 --2- 172.21.15.60:0/3191575466 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3041426b0 0x7fe3041527b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fe304060a90 tx=0x7fe2f4015000).stop 2022-06-29T15:58:29.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.169+0000 7fe303fff700 1 -- 172.21.15.60:0/3191575466 reap_dead start 2022-06-29T15:58:29.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.169+0000 7fe309b5b700 1 -- 172.21.15.60:0/3191575466 shutdown_connections 2022-06-29T15:58:29.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.169+0000 7fe309b5b700 1 -- 172.21.15.60:0/3191575466 >> 172.21.15.60:0/3191575466 conn(0x7fe3040b9340 msgr2=0x7fe3041403a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:29.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.169+0000 7fe309b5b700 1 -- 172.21.15.60:0/3191575466 shutdown_connections 2022-06-29T15:58:29.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.169+0000 7fe309b5b700 1 -- 172.21.15.60:0/3191575466 wait complete. 2022-06-29T15:58:29.188 INFO:teuthology.orchestra.run.smithi060.stdout:55834574854 2022-06-29T15:58:29.188 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2022-06-29T15:58:29.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.293+0000 7fc37beef700 1 Processor -- start 2022-06-29T15:58:29.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.293+0000 7fc37beef700 1 -- start start 2022-06-29T15:58:29.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.293+0000 7fc37beef700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc3740ae7c0 0x7fc374141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.293+0000 7fc37beef700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3740b4010 0x7fc374142490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.293+0000 7fc37beef700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc374146160 0x7fc3741465f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.293+0000 7fc37beef700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc374143a40 con 0x7fc3740ae7c0 2022-06-29T15:58:29.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.293+0000 7fc37beef700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3740aee60 con 0x7fc374146160 2022-06-29T15:58:29.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.293+0000 7fc37beef700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3740af100 con 0x7fc3740b4010 2022-06-29T15:58:29.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.293+0000 7fc379c8b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc3740ae7c0 0x7fc374141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.293+0000 7fc379c8b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc3740ae7c0 0x7fc374141fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45218/0 (socket says 172.21.15.60:45218) 2022-06-29T15:58:29.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.293+0000 7fc379c8b700 1 -- 172.21.15.60:0/3938682686 learned_addr learned my addr 172.21.15.60:0/3938682686 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:29.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.293+0000 7fc37a48c700 1 --2- 172.21.15.60:0/3938682686 >> v2:172.21.15.92:3300/0 conn(0x7fc374146160 0x7fc3741465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.293+0000 7fc37948a700 1 --2- 172.21.15.60:0/3938682686 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3740b4010 0x7fc374142490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.293+0000 7fc379c8b700 1 -- 172.21.15.60:0/3938682686 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3740b4010 msgr2=0x7fc374142490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:29.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.293+0000 7fc379c8b700 1 --2- 172.21.15.60:0/3938682686 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3740b4010 0x7fc374142490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.293+0000 7fc379c8b700 1 -- 172.21.15.60:0/3938682686 >> v2:172.21.15.92:3300/0 conn(0x7fc374146160 msgr2=0x7fc3741465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:29.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.293+0000 7fc379c8b700 1 --2- 172.21.15.60:0/3938682686 >> v2:172.21.15.92:3300/0 conn(0x7fc374146160 0x7fc3741465f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.293+0000 7fc379c8b700 1 -- 172.21.15.60:0/3938682686 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3740aeba0 con 0x7fc3740ae7c0 2022-06-29T15:58:29.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.293+0000 7fc37948a700 1 --2- 172.21.15.60:0/3938682686 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3740b4010 0x7fc374142490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:58:29.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.293+0000 7fc379c8b700 1 --2- 172.21.15.60:0/3938682686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3740ae7c0 0x7fc374141fa0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fc370002f70 tx=0x7fc370006eb0).ready entity=mon.0 client_cookie=d5a696cd81562b8b server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:29.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.293+0000 7fc378c89700 1 -- 172.21.15.60:0/3938682686 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc37000d570 con 0x7fc3740ae7c0 2022-06-29T15:58:29.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.293+0000 7fc378c89700 1 -- 172.21.15.60:0/3938682686 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc370000e90 con 0x7fc3740ae7c0 2022-06-29T15:58:29.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.293+0000 7fc378c89700 1 -- 172.21.15.60:0/3938682686 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc37000d900 con 0x7fc3740ae7c0 2022-06-29T15:58:29.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.297+0000 7fc37beef700 1 -- 172.21.15.60:0/3938682686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3740ae7c0 msgr2=0x7fc374141fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:29.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.297+0000 7fc37beef700 1 --2- 172.21.15.60:0/3938682686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3740ae7c0 0x7fc374141fa0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fc370002f70 tx=0x7fc370006eb0).stop 2022-06-29T15:58:29.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.297+0000 7fc37beef700 1 -- 172.21.15.60:0/3938682686 shutdown_connections 2022-06-29T15:58:29.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.297+0000 7fc37beef700 1 --2- 172.21.15.60:0/3938682686 >> v2:172.21.15.92:3300/0 conn(0x7fc374146160 0x7fc3741465f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.297+0000 7fc37beef700 1 --2- 172.21.15.60:0/3938682686 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3740b4010 0x7fc374142490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.297+0000 7fc37beef700 1 --2- 172.21.15.60:0/3938682686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3740ae7c0 0x7fc374141fa0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.297+0000 7fc37beef700 1 -- 172.21.15.60:0/3938682686 >> 172.21.15.60:0/3938682686 conn(0x7fc3740b9340 msgr2=0x7fc3740b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:29.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.297+0000 7fc37beef700 1 -- 172.21.15.60:0/3938682686 shutdown_connections 2022-06-29T15:58:29.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.297+0000 7fc37beef700 1 -- 172.21.15.60:0/3938682686 wait complete. 2022-06-29T15:58:29.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.297+0000 7fc37beef700 1 Processor -- start 2022-06-29T15:58:29.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.297+0000 7fc37beef700 1 -- start start 2022-06-29T15:58:29.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.297+0000 7fc37beef700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3740ae7c0 0x7fc3740b0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.297+0000 7fc37beef700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3740b4010 0x7fc3740b1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.297+0000 7fc37beef700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc374146160 0x7fc374151f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.297+0000 7fc37beef700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3741133d0 con 0x7fc374146160 2022-06-29T15:58:29.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.297+0000 7fc37beef700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3740b0a00 con 0x7fc3740b4010 2022-06-29T15:58:29.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.297+0000 7fc37beef700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc374113600 con 0x7fc3740ae7c0 2022-06-29T15:58:29.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.297+0000 7fc37a48c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc374146160 0x7fc374151f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.297+0000 7fc379c8b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3740ae7c0 0x7fc3740b0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.297+0000 7fc37a48c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc374146160 0x7fc374151f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45226/0 (socket says 172.21.15.60:45226) 2022-06-29T15:58:29.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.297+0000 7fc379c8b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3740ae7c0 0x7fc3740b0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:58318/0 (socket says 172.21.15.60:58318) 2022-06-29T15:58:29.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.297+0000 7fc37a48c700 1 -- 172.21.15.60:0/3871747122 learned_addr learned my addr 172.21.15.60:0/3871747122 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:29.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.301+0000 7fc37948a700 1 --2- 172.21.15.60:0/3871747122 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3740b4010 0x7fc3740b1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.301+0000 7fc37a48c700 1 -- 172.21.15.60:0/3871747122 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3740ae7c0 msgr2=0x7fc3740b0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:29.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.301+0000 7fc37a48c700 1 --2- 172.21.15.60:0/3871747122 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3740ae7c0 0x7fc3740b0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.301+0000 7fc37a48c700 1 -- 172.21.15.60:0/3871747122 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3740b4010 msgr2=0x7fc3740b1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:29.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.301+0000 7fc37a48c700 1 --2- 172.21.15.60:0/3871747122 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3740b4010 0x7fc3740b1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.301+0000 7fc37a48c700 1 -- 172.21.15.60:0/3871747122 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc37000b1e0 con 0x7fc374146160 2022-06-29T15:58:29.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.301+0000 7fc379c8b700 1 --2- 172.21.15.60:0/3871747122 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3740ae7c0 0x7fc3740b0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:29.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.301+0000 7fc37a48c700 1 --2- 172.21.15.60:0/3871747122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc374146160 0x7fc374151f60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fc3740b0b90 tx=0x7fc36c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:29.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.301+0000 7fc362ffd700 1 -- 172.21.15.60:0/3871747122 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc36c0127e0 con 0x7fc374146160 2022-06-29T15:58:29.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.301+0000 7fc362ffd700 1 -- 172.21.15.60:0/3871747122 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc36c012940 con 0x7fc374146160 2022-06-29T15:58:29.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.301+0000 7fc37beef700 1 -- 172.21.15.60:0/3871747122 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3740b23b0 con 0x7fc374146160 2022-06-29T15:58:29.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.301+0000 7fc362ffd700 1 -- 172.21.15.60:0/3871747122 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc36c012ca0 con 0x7fc374146160 2022-06-29T15:58:29.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.301+0000 7fc37beef700 1 -- 172.21.15.60:0/3871747122 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3740affc0 con 0x7fc374146160 2022-06-29T15:58:29.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.301+0000 7fc37beef700 1 -- 172.21.15.60:0/3871747122 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc3740b21f0 con 0x7fc374146160 2022-06-29T15:58:29.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.301+0000 7fc362ffd700 1 -- 172.21.15.60:0/3871747122 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc36c013480 con 0x7fc374146160 2022-06-29T15:58:29.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.301+0000 7fc362ffd700 1 --2- 172.21.15.60:0/3871747122 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc354053a70 0x7fc354055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.301+0000 7fc362ffd700 1 -- 172.21.15.60:0/3871747122 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc36c06c1b0 con 0x7fc374146160 2022-06-29T15:58:29.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.301+0000 7fc362ffd700 1 --2- 172.21.15.60:0/3871747122 >> [v2:172.21.15.92:6800/13247,v1:172.21.15.92:6801/13247] conn(0x7fc35405add0 0x7fc35405d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.301+0000 7fc362ffd700 1 -- 172.21.15.60:0/3871747122 --> [v2:172.21.15.92:6800/13247,v1:172.21.15.92:6801/13247] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc35405d810 con 0x7fc35405add0 2022-06-29T15:58:29.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.301+0000 7fc362ffd700 1 -- 172.21.15.60:0/3871747122 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc36c06e840 con 0x7fc374146160 2022-06-29T15:58:29.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.305+0000 7fc379c8b700 1 --2- 172.21.15.60:0/3871747122 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc354053a70 0x7fc354055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.305+0000 7fc37948a700 1 --2- 172.21.15.60:0/3871747122 >> [v2:172.21.15.92:6800/13247,v1:172.21.15.92:6801/13247] conn(0x7fc35405add0 0x7fc35405d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.305+0000 7fc379c8b700 1 --2- 172.21.15.60:0/3871747122 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc354053a70 0x7fc354055f30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fc37001c000 tx=0x7fc370015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:29.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.305+0000 7fc37948a700 1 --2- 172.21.15.60:0/3871747122 >> [v2:172.21.15.92:6800/13247,v1:172.21.15.92:6801/13247] conn(0x7fc35405add0 0x7fc35405d230 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:29.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.305+0000 7fc362ffd700 1 -- 172.21.15.60:0/3871747122 <== osd.7 v2:172.21.15.92:6800/13247 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7fc35405d810 con 0x7fc35405add0 2022-06-29T15:58:29.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.321+0000 7fc37beef700 1 -- 172.21.15.60:0/3871747122 --> [v2:172.21.15.92:6800/13247,v1:172.21.15.92:6801/13247] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc374152770 con 0x7fc35405add0 2022-06-29T15:58:29.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.321+0000 7fc362ffd700 1 -- 172.21.15.60:0/3871747122 <== osd.7 v2:172.21.15.92:6800/13247 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc374152770 con 0x7fc35405add0 2022-06-29T15:58:29.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.325+0000 7fc37beef700 1 -- 172.21.15.60:0/3871747122 >> [v2:172.21.15.92:6800/13247,v1:172.21.15.92:6801/13247] conn(0x7fc35405add0 msgr2=0x7fc35405d230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:29.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.325+0000 7fc37beef700 1 --2- 172.21.15.60:0/3871747122 >> [v2:172.21.15.92:6800/13247,v1:172.21.15.92:6801/13247] conn(0x7fc35405add0 0x7fc35405d230 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.325+0000 7fc37beef700 1 -- 172.21.15.60:0/3871747122 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc354053a70 msgr2=0x7fc354055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:29.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.325+0000 7fc37beef700 1 --2- 172.21.15.60:0/3871747122 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc354053a70 0x7fc354055f30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fc37001c000 tx=0x7fc370015000).stop 2022-06-29T15:58:29.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.325+0000 7fc37beef700 1 -- 172.21.15.60:0/3871747122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc374146160 msgr2=0x7fc374151f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:29.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.325+0000 7fc37beef700 1 --2- 172.21.15.60:0/3871747122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc374146160 0x7fc374151f60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fc3740b0b90 tx=0x7fc36c008040).stop 2022-06-29T15:58:29.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.325+0000 7fc37a48c700 1 -- 172.21.15.60:0/3871747122 reap_dead start 2022-06-29T15:58:29.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.325+0000 7fc37beef700 1 -- 172.21.15.60:0/3871747122 shutdown_connections 2022-06-29T15:58:29.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.325+0000 7fc37beef700 1 -- 172.21.15.60:0/3871747122 >> 172.21.15.60:0/3871747122 conn(0x7fc3740b9340 msgr2=0x7fc3740b7240 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:29.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.325+0000 7fc37beef700 1 -- 172.21.15.60:0/3871747122 shutdown_connections 2022-06-29T15:58:29.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.325+0000 7fc37beef700 1 -- 172.21.15.60:0/3871747122 wait complete. 2022-06-29T15:58:29.341 INFO:teuthology.orchestra.run.smithi060.stdout:55834574853 2022-06-29T15:58:29.341 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2022-06-29T15:58:29.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.449+0000 7fcd33fff700 1 Processor -- start 2022-06-29T15:58:29.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.449+0000 7fcd33fff700 1 -- start start 2022-06-29T15:58:29.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.449+0000 7fcd33fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcd3405d070 0x7fcd3414aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.449+0000 7fcd33fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcd340b2130 0x7fcd340aca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.449+0000 7fcd33fff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd3405d450 con 0x7fcd3414af80 2022-06-29T15:58:29.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.449+0000 7fcd33fff700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd340adf90 con 0x7fcd3405d070 2022-06-29T15:58:29.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.449+0000 7fcd33fff700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd340b25d0 con 0x7fcd340b2130 2022-06-29T15:58:29.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.449+0000 7fcd32ffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcd3405d070 0x7fcd3414aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.449+0000 7fcd337fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcd3414af80 0x7fcd340acf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53324/0 (socket says 172.21.15.60:53324) 2022-06-29T15:58:29.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.449+0000 7fcd337fe700 1 -- 172.21.15.60:0/1535878472 learned_addr learned my addr 172.21.15.60:0/1535878472 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:29.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.453+0000 7fcd327fc700 1 --2- 172.21.15.60:0/1535878472 >> v2:172.21.15.92:3300/0 conn(0x7fcd340b2130 0x7fcd340aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.453+0000 7fcd32ffd700 1 -- 172.21.15.60:0/1535878472 >> v2:172.21.15.92:3300/0 conn(0x7fcd340b2130 msgr2=0x7fcd340aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:29.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.453+0000 7fcd32ffd700 1 --2- 172.21.15.60:0/1535878472 >> v2:172.21.15.92:3300/0 conn(0x7fcd340b2130 0x7fcd340aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.453+0000 7fcd32ffd700 1 -- 172.21.15.60:0/1535878472 >> v1:172.21.15.60:6789/0 conn(0x7fcd3414af80 legacy=0x7fcd340acf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:29.461 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.453+0000 7fcd32ffd700 1 -- 172.21.15.60:0/1535878472 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd3414a350 con 0x7fcd3405d070 2022-06-29T15:58:29.461 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.453+0000 7fcd327fc700 1 --2- 172.21.15.60:0/1535878472 >> v2:172.21.15.92:3300/0 conn(0x7fcd340b2130 0x7fcd340aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:29.461 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.453+0000 7fcd31ffb700 1 -- 172.21.15.60:0/1535878472 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 327880193 0 0) 0x7fcd3405d450 con 0x7fcd3414af80 2022-06-29T15:58:29.461 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.453+0000 7fcd32ffd700 1 --2- 172.21.15.60:0/1535878472 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd3405d070 0x7fcd3414aa90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fcd24002f70 tx=0x7fcd24006df0).ready entity=mon.0 client_cookie=b83871afa280b708 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:29.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.453+0000 7fcd31ffb700 1 -- 172.21.15.60:0/1535878472 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd2400d570 con 0x7fcd3405d070 2022-06-29T15:58:29.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.453+0000 7fcd31ffb700 1 -- 172.21.15.60:0/1535878472 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd24000e90 con 0x7fcd3405d070 2022-06-29T15:58:29.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.453+0000 7fcd31ffb700 1 -- 172.21.15.60:0/1535878472 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd2400d8c0 con 0x7fcd3405d070 2022-06-29T15:58:29.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.453+0000 7fcd33fff700 1 -- 172.21.15.60:0/1535878472 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd3405d070 msgr2=0x7fcd3414aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:29.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.453+0000 7fcd33fff700 1 --2- 172.21.15.60:0/1535878472 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd3405d070 0x7fcd3414aa90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fcd24002f70 tx=0x7fcd24006df0).stop 2022-06-29T15:58:29.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.453+0000 7fcd33fff700 1 -- 172.21.15.60:0/1535878472 shutdown_connections 2022-06-29T15:58:29.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.453+0000 7fcd33fff700 1 --2- 172.21.15.60:0/1535878472 >> v2:172.21.15.92:3300/0 conn(0x7fcd340b2130 0x7fcd340aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.453+0000 7fcd33fff700 1 --2- 172.21.15.60:0/1535878472 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd3405d070 0x7fcd3414aa90 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.453+0000 7fcd33fff700 1 -- 172.21.15.60:0/1535878472 >> 172.21.15.60:0/1535878472 conn(0x7fcd340b5890 msgr2=0x7fcd340b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:29.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.453+0000 7fcd33fff700 1 -- 172.21.15.60:0/1535878472 shutdown_connections 2022-06-29T15:58:29.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.453+0000 7fcd33fff700 1 -- 172.21.15.60:0/1535878472 wait complete. 2022-06-29T15:58:29.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.457+0000 7fcd33fff700 1 Processor -- start 2022-06-29T15:58:29.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.457+0000 7fcd33fff700 1 -- start start 2022-06-29T15:58:29.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.457+0000 7fcd33fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd3405d070 0x7fcd340b4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.457+0000 7fcd33fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd340b2130 0x7fcd34064ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.457+0000 7fcd33fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd3414af80 0x7fcd34139d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.457+0000 7fcd33fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd340653b0 con 0x7fcd3414af80 2022-06-29T15:58:29.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.457+0000 7fcd33fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3413b530 con 0x7fcd3405d070 2022-06-29T15:58:29.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.457+0000 7fcd33fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd34065f50 con 0x7fcd340b2130 2022-06-29T15:58:29.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.457+0000 7fcd337fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd3414af80 0x7fcd34139d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.457+0000 7fcd327fc700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd340b2130 0x7fcd34064ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.457+0000 7fcd337fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd3414af80 0x7fcd34139d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45244/0 (socket says 172.21.15.60:45244) 2022-06-29T15:58:29.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.457+0000 7fcd32ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd3405d070 0x7fcd340b4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.457+0000 7fcd337fe700 1 -- 172.21.15.60:0/2527082909 learned_addr learned my addr 172.21.15.60:0/2527082909 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:29.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.457+0000 7fcd337fe700 1 -- 172.21.15.60:0/2527082909 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd340b2130 msgr2=0x7fcd34064ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:29.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.457+0000 7fcd337fe700 1 --2- 172.21.15.60:0/2527082909 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd340b2130 0x7fcd34064ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.457+0000 7fcd337fe700 1 -- 172.21.15.60:0/2527082909 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd3405d070 msgr2=0x7fcd340b4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:29.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.457+0000 7fcd337fe700 1 --2- 172.21.15.60:0/2527082909 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd3405d070 0x7fcd340b4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.457+0000 7fcd337fe700 1 -- 172.21.15.60:0/2527082909 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1c00b180 con 0x7fcd3414af80 2022-06-29T15:58:29.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.457+0000 7fcd337fe700 1 --2- 172.21.15.60:0/2527082909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd3414af80 0x7fcd34139d10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fcd2c001250 tx=0x7fcd2c001360).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:29.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.457+0000 7fcd2bfff700 1 -- 172.21.15.60:0/2527082909 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd2c005ba0 con 0x7fcd3414af80 2022-06-29T15:58:29.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.457+0000 7fcd2bfff700 1 -- 172.21.15.60:0/2527082909 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd2c005d00 con 0x7fcd3414af80 2022-06-29T15:58:29.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.457+0000 7fcd33fff700 1 -- 172.21.15.60:0/2527082909 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd2400b180 con 0x7fcd3414af80 2022-06-29T15:58:29.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.457+0000 7fcd2bfff700 1 -- 172.21.15.60:0/2527082909 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd2c004a90 con 0x7fcd3414af80 2022-06-29T15:58:29.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.457+0000 7fcd33fff700 1 -- 172.21.15.60:0/2527082909 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd341428b0 con 0x7fcd3414af80 2022-06-29T15:58:29.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.461+0000 7fcd33fff700 1 -- 172.21.15.60:0/2527082909 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd3414a5f0 con 0x7fcd3414af80 2022-06-29T15:58:29.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.461+0000 7fcd2bfff700 1 -- 172.21.15.60:0/2527082909 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcd2c004bf0 con 0x7fcd3414af80 2022-06-29T15:58:29.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.461+0000 7fcd2bfff700 1 --2- 172.21.15.60:0/2527082909 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcd18053aa0 0x7fcd18055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.461+0000 7fcd2bfff700 1 -- 172.21.15.60:0/2527082909 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcd2c06b9f0 con 0x7fcd3414af80 2022-06-29T15:58:29.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.461+0000 7fcd32ffd700 1 --2- 172.21.15.60:0/2527082909 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcd18053aa0 0x7fcd18055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.465+0000 7fcd32ffd700 1 --2- 172.21.15.60:0/2527082909 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcd18053aa0 0x7fcd18055f60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fcd2401a000 tx=0x7fcd24014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:29.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.469+0000 7fcd2bfff700 1 -- 172.21.15.60:0/2527082909 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcd2c042890 con 0x7fcd3414af80 2022-06-29T15:58:29.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.597+0000 7fcd33fff700 1 -- 172.21.15.60:0/2527082909 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fcd340654f0 con 0x7fcd3414af80 2022-06-29T15:58:29.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.597+0000 7fcd2bfff700 1 -- 172.21.15.60:0/2527082909 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcd2c045eb0 con 0x7fcd3414af80 2022-06-29T15:58:29.605 INFO:teuthology.orchestra.run.smithi060.stdout:55834574853 2022-06-29T15:58:29.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.601+0000 7fcd33fff700 1 -- 172.21.15.60:0/2527082909 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcd18053aa0 msgr2=0x7fcd18055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:29.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.601+0000 7fcd33fff700 1 --2- 172.21.15.60:0/2527082909 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcd18053aa0 0x7fcd18055f60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fcd2401a000 tx=0x7fcd24014000).stop 2022-06-29T15:58:29.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.601+0000 7fcd33fff700 1 -- 172.21.15.60:0/2527082909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd3414af80 msgr2=0x7fcd34139d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:29.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.601+0000 7fcd33fff700 1 --2- 172.21.15.60:0/2527082909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd3414af80 0x7fcd34139d10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fcd2c001250 tx=0x7fcd2c001360).stop 2022-06-29T15:58:29.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.601+0000 7fcd33fff700 1 -- 172.21.15.60:0/2527082909 shutdown_connections 2022-06-29T15:58:29.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.601+0000 7fcd33fff700 1 --2- 172.21.15.60:0/2527082909 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcd18053aa0 0x7fcd18055f60 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.601+0000 7fcd33fff700 1 --2- 172.21.15.60:0/2527082909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd3414af80 0x7fcd34139d10 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.601+0000 7fcd33fff700 1 --2- 172.21.15.60:0/2527082909 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd340b2130 0x7fcd34064ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.601+0000 7fcd33fff700 1 --2- 172.21.15.60:0/2527082909 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd3405d070 0x7fcd340b4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.601+0000 7fcd33fff700 1 -- 172.21.15.60:0/2527082909 >> 172.21.15.60:0/2527082909 conn(0x7fcd340b5890 msgr2=0x7fcd34142f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:29.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.601+0000 7fcd33fff700 1 -- 172.21.15.60:0/2527082909 shutdown_connections 2022-06-29T15:58:29.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.601+0000 7fcd33fff700 1 -- 172.21.15.60:0/2527082909 wait complete. 2022-06-29T15:58:29.621 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.0 2022-06-29T15:58:29.622 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2022-06-29T15:58:29.728 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.721+0000 7f28eb5ae700 1 Processor -- start 2022-06-29T15:58:29.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.721+0000 7f28eb5ae700 1 -- start start 2022-06-29T15:58:29.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.725+0000 7f28eb5ae700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f28e40b6a50 0x7f28e41461e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.725+0000 7f28eb5ae700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f28e405f0b0 0x7f28e4142310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.725+0000 7f28eb5ae700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28e4142800 0x7f28e405ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.725+0000 7f28eb5ae700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28e40b6e30 con 0x7f28e40b6a50 2022-06-29T15:58:29.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.725+0000 7f28eb5ae700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28e40b0dd0 con 0x7f28e405f0b0 2022-06-29T15:58:29.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.725+0000 7f28eb5ae700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28e40b1070 con 0x7f28e4142800 2022-06-29T15:58:29.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.725+0000 7f28e934a700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f28e40b6a50 0x7f28e41461e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.725+0000 7f28e934a700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f28e40b6a50 0x7f28e41461e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45248/0 (socket says 172.21.15.60:45248) 2022-06-29T15:58:29.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.725+0000 7f28e934a700 1 -- 172.21.15.60:0/2041222718 learned_addr learned my addr 172.21.15.60:0/2041222718 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:29.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.725+0000 7f28e8b49700 1 --2- 172.21.15.60:0/2041222718 >> v2:172.21.15.92:3300/0 conn(0x7f28e405f0b0 0x7f28e4142310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.725+0000 7f28e9b4b700 1 --2- 172.21.15.60:0/2041222718 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28e4142800 0x7f28e405ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.725+0000 7f28e934a700 1 -- 172.21.15.60:0/2041222718 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28e4142800 msgr2=0x7f28e405ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:29.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.725+0000 7f28e934a700 1 --2- 172.21.15.60:0/2041222718 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28e4142800 0x7f28e405ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.725+0000 7f28e934a700 1 -- 172.21.15.60:0/2041222718 >> v2:172.21.15.92:3300/0 conn(0x7f28e405f0b0 msgr2=0x7f28e4142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:29.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.725+0000 7f28e934a700 1 --2- 172.21.15.60:0/2041222718 >> v2:172.21.15.92:3300/0 conn(0x7f28e405f0b0 0x7f28e4142310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.725+0000 7f28e934a700 1 -- 172.21.15.60:0/2041222718 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e405c5b0 con 0x7f28e40b6a50 2022-06-29T15:58:29.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.725+0000 7f28e934a700 1 --2- 172.21.15.60:0/2041222718 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28e40b6a50 0x7f28e41461e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f28e0002f70 tx=0x7f28e0006eb0).ready entity=mon.0 client_cookie=61c981aae19b6755 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:29.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.725+0000 7f28dbfff700 1 -- 172.21.15.60:0/2041222718 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f28e000d570 con 0x7f28e40b6a50 2022-06-29T15:58:29.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.725+0000 7f28dbfff700 1 -- 172.21.15.60:0/2041222718 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e0000e90 con 0x7f28e40b6a50 2022-06-29T15:58:29.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.725+0000 7f28dbfff700 1 -- 172.21.15.60:0/2041222718 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f28e000d900 con 0x7f28e40b6a50 2022-06-29T15:58:29.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.725+0000 7f28eb5ae700 1 -- 172.21.15.60:0/2041222718 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28e40b6a50 msgr2=0x7f28e41461e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:29.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.725+0000 7f28eb5ae700 1 --2- 172.21.15.60:0/2041222718 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28e40b6a50 0x7f28e41461e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f28e0002f70 tx=0x7f28e0006eb0).stop 2022-06-29T15:58:29.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.725+0000 7f28eb5ae700 1 -- 172.21.15.60:0/2041222718 shutdown_connections 2022-06-29T15:58:29.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.725+0000 7f28eb5ae700 1 --2- 172.21.15.60:0/2041222718 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28e4142800 0x7f28e405ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.725+0000 7f28eb5ae700 1 --2- 172.21.15.60:0/2041222718 >> v2:172.21.15.92:3300/0 conn(0x7f28e405f0b0 0x7f28e4142310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.725+0000 7f28eb5ae700 1 --2- 172.21.15.60:0/2041222718 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28e40b6a50 0x7f28e41461e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.725+0000 7f28eb5ae700 1 -- 172.21.15.60:0/2041222718 >> 172.21.15.60:0/2041222718 conn(0x7f28e40643e0 msgr2=0x7f28e40b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:29.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.725+0000 7f28eb5ae700 1 -- 172.21.15.60:0/2041222718 shutdown_connections 2022-06-29T15:58:29.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.725+0000 7f28eb5ae700 1 -- 172.21.15.60:0/2041222718 wait complete. 2022-06-29T15:58:29.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.729+0000 7f28eb5ae700 1 Processor -- start 2022-06-29T15:58:29.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.729+0000 7f28eb5ae700 1 -- start start 2022-06-29T15:58:29.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.729+0000 7f28eb5ae700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f28e405f0b0 0x7f28e40661a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.729+0000 7f28eb5ae700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28e40b6a50 0x7f28e4147320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.729+0000 7f28eb5ae700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28e4142800 0x7f28e4147810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.729+0000 7f28eb5ae700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e4144700 con 0x7f28e40b6a50 2022-06-29T15:58:29.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.729+0000 7f28eb5ae700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e4065350 con 0x7f28e405f0b0 2022-06-29T15:58:29.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.729+0000 7f28eb5ae700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28e41462e0 con 0x7f28e4142800 2022-06-29T15:58:29.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.729+0000 7f28e8b49700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28e40b6a50 0x7f28e4147320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.729+0000 7f28e8b49700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28e40b6a50 0x7f28e4147320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45256/0 (socket says 172.21.15.60:45256) 2022-06-29T15:58:29.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.729+0000 7f28e8b49700 1 -- 172.21.15.60:0/369649835 learned_addr learned my addr 172.21.15.60:0/369649835 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:29.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.729+0000 7f28e934a700 1 --2- 172.21.15.60:0/369649835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f28e405f0b0 0x7f28e40661a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.729+0000 7f28e9b4b700 1 --2- 172.21.15.60:0/369649835 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28e4142800 0x7f28e4147810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.729+0000 7f28e8b49700 1 -- 172.21.15.60:0/369649835 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28e4142800 msgr2=0x7f28e4147810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:29.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.729+0000 7f28e8b49700 1 --2- 172.21.15.60:0/369649835 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28e4142800 0x7f28e4147810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.729+0000 7f28e8b49700 1 -- 172.21.15.60:0/369649835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f28e405f0b0 msgr2=0x7f28e40661a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:29.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.729+0000 7f28e8b49700 1 --2- 172.21.15.60:0/369649835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f28e405f0b0 0x7f28e40661a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.729+0000 7f28e8b49700 1 -- 172.21.15.60:0/369649835 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e000b1e0 con 0x7f28e40b6a50 2022-06-29T15:58:29.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.729+0000 7f28e8b49700 1 --2- 172.21.15.60:0/369649835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28e40b6a50 0x7f28e4147320 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f28d400c000 tx=0x7f28d4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:29.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.729+0000 7f28da7fc700 1 -- 172.21.15.60:0/369649835 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f28d4010780 con 0x7f28e40b6a50 2022-06-29T15:58:29.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.729+0000 7f28da7fc700 1 -- 172.21.15.60:0/369649835 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28d40108e0 con 0x7f28e40b6a50 2022-06-29T15:58:29.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.733+0000 7f28eb5ae700 1 -- 172.21.15.60:0/369649835 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28e40655b0 con 0x7f28e40b6a50 2022-06-29T15:58:29.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.733+0000 7f28da7fc700 1 -- 172.21.15.60:0/369649835 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f28d4010c40 con 0x7f28e40b6a50 2022-06-29T15:58:29.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.733+0000 7f28eb5ae700 1 -- 172.21.15.60:0/369649835 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28e4148fa0 con 0x7f28e40b6a50 2022-06-29T15:58:29.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.733+0000 7f28da7fc700 1 -- 172.21.15.60:0/369649835 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f28d401c460 con 0x7f28e40b6a50 2022-06-29T15:58:29.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.733+0000 7f28eb5ae700 1 -- 172.21.15.60:0/369649835 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28e4066d00 con 0x7f28e40b6a50 2022-06-29T15:58:29.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.733+0000 7f28da7fc700 1 --2- 172.21.15.60:0/369649835 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f28cc053a70 0x7f28cc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.733+0000 7f28da7fc700 1 -- 172.21.15.60:0/369649835 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f28d406c260 con 0x7f28e40b6a50 2022-06-29T15:58:29.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.733+0000 7f28e934a700 1 --2- 172.21.15.60:0/369649835 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f28cc053a70 0x7f28cc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.733+0000 7f28e934a700 1 --2- 172.21.15.60:0/369649835 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f28cc053a70 0x7f28cc055f30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f28e001c000 tx=0x7f28e0015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:29.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.737+0000 7f28da7fc700 1 -- 172.21.15.60:0/369649835 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f28d4043120 con 0x7f28e40b6a50 2022-06-29T15:58:29.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.865+0000 7f28eb5ae700 1 -- 172.21.15.60:0/369649835 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f28e40573d0 con 0x7f28e40b6a50 2022-06-29T15:58:29.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.865+0000 7f28da7fc700 1 -- 172.21.15.60:0/369649835 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f28d4046740 con 0x7f28e40b6a50 2022-06-29T15:58:29.871 INFO:teuthology.orchestra.run.smithi060.stdout:55834574852 2022-06-29T15:58:29.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.865+0000 7f28eb5ae700 1 -- 172.21.15.60:0/369649835 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f28cc053a70 msgr2=0x7f28cc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:29.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.865+0000 7f28eb5ae700 1 --2- 172.21.15.60:0/369649835 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f28cc053a70 0x7f28cc055f30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f28e001c000 tx=0x7f28e0015000).stop 2022-06-29T15:58:29.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.865+0000 7f28eb5ae700 1 -- 172.21.15.60:0/369649835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28e40b6a50 msgr2=0x7f28e4147320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:29.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.865+0000 7f28eb5ae700 1 --2- 172.21.15.60:0/369649835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28e40b6a50 0x7f28e4147320 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f28d400c000 tx=0x7f28d4006df0).stop 2022-06-29T15:58:29.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.865+0000 7f28eb5ae700 1 -- 172.21.15.60:0/369649835 shutdown_connections 2022-06-29T15:58:29.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.865+0000 7f28eb5ae700 1 --2- 172.21.15.60:0/369649835 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f28cc053a70 0x7f28cc055f30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.865+0000 7f28eb5ae700 1 --2- 172.21.15.60:0/369649835 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f28e4142800 0x7f28e4147810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.865+0000 7f28eb5ae700 1 --2- 172.21.15.60:0/369649835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f28e40b6a50 0x7f28e4147320 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.865+0000 7f28eb5ae700 1 --2- 172.21.15.60:0/369649835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f28e405f0b0 0x7f28e40661a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:29.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.865+0000 7f28eb5ae700 1 -- 172.21.15.60:0/369649835 >> 172.21.15.60:0/369649835 conn(0x7f28e40643e0 msgr2=0x7f28e4142c50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:29.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.865+0000 7f28eb5ae700 1 -- 172.21.15.60:0/369649835 shutdown_connections 2022-06-29T15:58:29.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.865+0000 7f28eb5ae700 1 -- 172.21.15.60:0/369649835 wait complete. 2022-06-29T15:58:29.886 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.1 2022-06-29T15:58:29.887 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2022-06-29T15:58:29.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.989+0000 7f05ceb4f700 1 Processor -- start 2022-06-29T15:58:29.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.989+0000 7f05ceb4f700 1 -- start start 2022-06-29T15:58:29.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.989+0000 7f05ceb4f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05c80b48b0 0x7f05c8142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.989+0000 7f05ceb4f700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f05c80b2c50 0x7f05c80b3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:29.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.989+0000 7f05ceb4f700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05c80b0cf0 con 0x7f05c80b5eb0 2022-06-29T15:58:29.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.989+0000 7f05ceb4f700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05c80b54e0 con 0x7f05c80b2c50 2022-06-29T15:58:29.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.989+0000 7f05ceb4f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05c80b4d10 con 0x7f05c80b48b0 2022-06-29T15:58:29.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.989+0000 7f05cd34c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05c80b48b0 0x7f05c8142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.989+0000 7f05ce34e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f05c80b2c50 0x7f05c80b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:29.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.989+0000 7f05cdb4d700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f05c80b5eb0 0x7f05c80b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:57596/0 (socket says 172.21.15.60:57596) 2022-06-29T15:58:29.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.989+0000 7f05cdb4d700 1 -- 172.21.15.60:0/2436287659 learned_addr learned my addr 172.21.15.60:0/2436287659 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:29.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.989+0000 7f05ccb4b700 1 -- 172.21.15.60:0/2436287659 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3939301915 0 0) 0x7f05c0001490 con 0x7f05c80b5eb0 2022-06-29T15:58:29.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.989+0000 7f05ccb4b700 1 -- 172.21.15.60:0/2436287659 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05b4003260 con 0x7f05c80b5eb0 2022-06-29T15:58:29.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.989+0000 7f05cd34c700 1 -- 172.21.15.60:0/2436287659 >> v2:172.21.15.92:3300/0 conn(0x7f05c80b2c50 msgr2=0x7f05c80b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:30.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.989+0000 7f05cd34c700 1 --2- 172.21.15.60:0/2436287659 >> v2:172.21.15.92:3300/0 conn(0x7f05c80b2c50 0x7f05c80b3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.989+0000 7f05cd34c700 1 -- 172.21.15.60:0/2436287659 >> v1:172.21.15.92:6789/0 conn(0x7f05c80b5eb0 legacy=0x7f05c80b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:30.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.989+0000 7f05cd34c700 1 -- 172.21.15.60:0/2436287659 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c80b6290 con 0x7f05c80b48b0 2022-06-29T15:58:30.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.989+0000 7f05cd34c700 1 --2- 172.21.15.60:0/2436287659 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05c80b48b0 0x7f05c8142a20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f05bc002f70 tx=0x7f05bc006df0).ready entity=mon.2 client_cookie=20127e4abbf9453b server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:30.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.993+0000 7f05ccb4b700 1 -- 172.21.15.60:0/2436287659 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05bc00d4a0 con 0x7f05c80b48b0 2022-06-29T15:58:30.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.993+0000 7f05ccb4b700 1 -- 172.21.15.60:0/2436287659 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05bc000e90 con 0x7f05c80b48b0 2022-06-29T15:58:30.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.993+0000 7f05ccb4b700 1 -- 172.21.15.60:0/2436287659 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05bc00d7f0 con 0x7f05c80b48b0 2022-06-29T15:58:30.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.993+0000 7f05ceb4f700 1 -- 172.21.15.60:0/2436287659 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05c80b48b0 msgr2=0x7f05c8142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:30.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.993+0000 7f05ceb4f700 1 --2- 172.21.15.60:0/2436287659 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05c80b48b0 0x7f05c8142a20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f05bc002f70 tx=0x7f05bc006df0).stop 2022-06-29T15:58:30.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.993+0000 7f05ceb4f700 1 -- 172.21.15.60:0/2436287659 shutdown_connections 2022-06-29T15:58:30.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.993+0000 7f05ceb4f700 1 --2- 172.21.15.60:0/2436287659 >> v2:172.21.15.92:3300/0 conn(0x7f05c80b2c50 0x7f05c80b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.993+0000 7f05ceb4f700 1 --2- 172.21.15.60:0/2436287659 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05c80b48b0 0x7f05c8142a20 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.993+0000 7f05ceb4f700 1 -- 172.21.15.60:0/2436287659 >> 172.21.15.60:0/2436287659 conn(0x7f05c8062090 msgr2=0x7f05c8062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:30.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.993+0000 7f05ceb4f700 1 -- 172.21.15.60:0/2436287659 shutdown_connections 2022-06-29T15:58:30.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.993+0000 7f05ceb4f700 1 -- 172.21.15.60:0/2436287659 wait complete. 2022-06-29T15:58:30.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.993+0000 7f05ceb4f700 1 Processor -- start 2022-06-29T15:58:30.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.993+0000 7f05ceb4f700 1 -- start start 2022-06-29T15:58:30.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.997+0000 7f05ceb4f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05c80b2c50 0x7f05c81408b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:30.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.997+0000 7f05ceb4f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05c80b48b0 0x7f05c8144030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:30.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.997+0000 7f05ceb4f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05c80b5eb0 0x7f05c80af1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:30.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.997+0000 7f05ceb4f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c8144520 con 0x7f05c80b5eb0 2022-06-29T15:58:30.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.997+0000 7f05ceb4f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c813f330 con 0x7f05c80b2c50 2022-06-29T15:58:30.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.997+0000 7f05ceb4f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05c8152860 con 0x7f05c80b48b0 2022-06-29T15:58:30.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.997+0000 7f05ce34e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05c80b5eb0 0x7f05c80af1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:30.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.997+0000 7f05ce34e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05c80b5eb0 0x7f05c80af1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45270/0 (socket says 172.21.15.60:45270) 2022-06-29T15:58:30.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.997+0000 7f05ce34e700 1 -- 172.21.15.60:0/1060074524 learned_addr learned my addr 172.21.15.60:0/1060074524 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:30.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.997+0000 7f05cd34c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05c80b48b0 0x7f05c8144030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:30.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.997+0000 7f05cdb4d700 1 --2- 172.21.15.60:0/1060074524 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05c80b2c50 0x7f05c81408b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:30.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.997+0000 7f05ce34e700 1 -- 172.21.15.60:0/1060074524 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05c80b48b0 msgr2=0x7f05c8144030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:30.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.997+0000 7f05ce34e700 1 --2- 172.21.15.60:0/1060074524 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05c80b48b0 0x7f05c8144030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.997+0000 7f05ce34e700 1 -- 172.21.15.60:0/1060074524 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05c80b2c50 msgr2=0x7f05c81408b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:30.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.997+0000 7f05ce34e700 1 --2- 172.21.15.60:0/1060074524 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05c80b2c50 0x7f05c81408b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.997+0000 7f05ce34e700 1 -- 172.21.15.60:0/1060074524 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05bc00b180 con 0x7f05c80b5eb0 2022-06-29T15:58:30.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.997+0000 7f05ce34e700 1 --2- 172.21.15.60:0/1060074524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05c80b5eb0 0x7f05c80af1c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f05b800c000 tx=0x7f05b8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:30.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.997+0000 7f05c6ffd700 1 -- 172.21.15.60:0/1060074524 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05b8011750 con 0x7f05c80b5eb0 2022-06-29T15:58:30.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.997+0000 7f05c6ffd700 1 -- 172.21.15.60:0/1060074524 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b80118b0 con 0x7f05c80b5eb0 2022-06-29T15:58:30.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.997+0000 7f05ceb4f700 1 -- 172.21.15.60:0/1060074524 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05c8152b00 con 0x7f05c80b5eb0 2022-06-29T15:58:30.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.997+0000 7f05c6ffd700 1 -- 172.21.15.60:0/1060074524 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05b8011c10 con 0x7f05c80b5eb0 2022-06-29T15:58:30.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:29.997+0000 7f05ceb4f700 1 -- 172.21.15.60:0/1060074524 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05c80acc50 con 0x7f05c80b5eb0 2022-06-29T15:58:30.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.001+0000 7f05c6ffd700 1 -- 172.21.15.60:0/1060074524 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f05b8011d70 con 0x7f05c80b5eb0 2022-06-29T15:58:30.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.001+0000 7f05ceb4f700 1 -- 172.21.15.60:0/1060074524 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05c80573d0 con 0x7f05c80b5eb0 2022-06-29T15:58:30.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.001+0000 7f05c6ffd700 1 --2- 172.21.15.60:0/1060074524 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f05a8053b00 0x7f05a8055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:30.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.001+0000 7f05c6ffd700 1 -- 172.21.15.60:0/1060074524 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f05b806c350 con 0x7f05c80b5eb0 2022-06-29T15:58:30.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.001+0000 7f05cdb4d700 1 --2- 172.21.15.60:0/1060074524 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f05a8053b00 0x7f05a8055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:30.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.001+0000 7f05cdb4d700 1 --2- 172.21.15.60:0/1060074524 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f05a8053b00 0x7f05a8055fc0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f05c000d000 tx=0x7f05c0000f40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:30.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.005+0000 7f05c6ffd700 1 -- 172.21.15.60:0/1060074524 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f05b80430e0 con 0x7f05c80b5eb0 2022-06-29T15:58:30.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.133+0000 7f05ceb4f700 1 -- 172.21.15.60:0/1060074524 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f05c80b3c80 con 0x7f05c80b5eb0 2022-06-29T15:58:30.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.133+0000 7f05c6ffd700 1 -- 172.21.15.60:0/1060074524 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f05b8046700 con 0x7f05c80b5eb0 2022-06-29T15:58:30.139 INFO:teuthology.orchestra.run.smithi060.stdout:55834574854 2022-06-29T15:58:30.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.133+0000 7f05ceb4f700 1 -- 172.21.15.60:0/1060074524 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f05a8053b00 msgr2=0x7f05a8055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:30.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.133+0000 7f05ceb4f700 1 --2- 172.21.15.60:0/1060074524 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f05a8053b00 0x7f05a8055fc0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f05c000d000 tx=0x7f05c0000f40).stop 2022-06-29T15:58:30.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.133+0000 7f05ceb4f700 1 -- 172.21.15.60:0/1060074524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05c80b5eb0 msgr2=0x7f05c80af1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:30.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.133+0000 7f05ceb4f700 1 --2- 172.21.15.60:0/1060074524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05c80b5eb0 0x7f05c80af1c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f05b800c000 tx=0x7f05b8006df0).stop 2022-06-29T15:58:30.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.133+0000 7f05ceb4f700 1 -- 172.21.15.60:0/1060074524 shutdown_connections 2022-06-29T15:58:30.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.133+0000 7f05ceb4f700 1 --2- 172.21.15.60:0/1060074524 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f05a8053b00 0x7f05a8055fc0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.133+0000 7f05ceb4f700 1 --2- 172.21.15.60:0/1060074524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05c80b5eb0 0x7f05c80af1c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.133+0000 7f05ceb4f700 1 --2- 172.21.15.60:0/1060074524 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05c80b48b0 0x7f05c8144030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.133+0000 7f05ceb4f700 1 --2- 172.21.15.60:0/1060074524 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05c80b2c50 0x7f05c81408b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.133+0000 7f05ceb4f700 1 -- 172.21.15.60:0/1060074524 >> 172.21.15.60:0/1060074524 conn(0x7f05c8062090 msgr2=0x7f05c80b6460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:30.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.133+0000 7f05ceb4f700 1 -- 172.21.15.60:0/1060074524 shutdown_connections 2022-06-29T15:58:30.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.133+0000 7f05ceb4f700 1 -- 172.21.15.60:0/1060074524 wait complete. 2022-06-29T15:58:30.151 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.2 2022-06-29T15:58:30.152 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2022-06-29T15:58:30.255 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.249+0000 7f8362815700 1 Processor -- start 2022-06-29T15:58:30.255 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.249+0000 7f8362815700 1 -- start start 2022-06-29T15:58:30.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.249+0000 7f8362815700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f835c0b2130 0x7f835c146100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:30.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.249+0000 7f8362815700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f835c0aed00 0x7f835c1465f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:30.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.249+0000 7f8362815700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835c05d450 con 0x7f835c05d070 2022-06-29T15:58:30.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.249+0000 7f8362815700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f835c0adf90 con 0x7f835c0b2130 2022-06-29T15:58:30.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.249+0000 7f8362815700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f835c0b25d0 con 0x7f835c0aed00 2022-06-29T15:58:30.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.249+0000 7f8361012700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f835c0b2130 0x7f835c146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:30.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.249+0000 7f8361813700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f835c05d070 0x7f835c14aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:57608/0 (socket says 172.21.15.60:57608) 2022-06-29T15:58:30.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.249+0000 7f8361012700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f835c0b2130 0x7f835c146100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45278/0 (socket says 172.21.15.60:45278) 2022-06-29T15:58:30.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.249+0000 7f8362014700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f835c0aed00 0x7f835c1465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:30.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.249+0000 7f8361813700 1 -- 172.21.15.60:0/756233072 learned_addr learned my addr 172.21.15.60:0/756233072 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:30.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.249+0000 7f8361012700 1 -- 172.21.15.60:0/756233072 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f835c0aed00 msgr2=0x7f835c1465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:30.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.249+0000 7f8361012700 1 --2- 172.21.15.60:0/756233072 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f835c0aed00 0x7f835c1465f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.249+0000 7f8360811700 1 -- 172.21.15.60:0/756233072 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 921699576 0 0) 0x7f835c05d450 con 0x7f835c05d070 2022-06-29T15:58:30.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.249+0000 7f8361012700 1 -- 172.21.15.60:0/756233072 >> v1:172.21.15.92:6789/0 conn(0x7f835c05d070 legacy=0x7f835c14aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:30.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.249+0000 7f8361012700 1 -- 172.21.15.60:0/756233072 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f835c14a350 con 0x7f835c0b2130 2022-06-29T15:58:30.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.253+0000 7f8361012700 1 --2- 172.21.15.60:0/756233072 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f835c0b2130 0x7f835c146100 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f834c002f70 tx=0x7f834c006df0).ready entity=mon.0 client_cookie=259a71e9e6f013dd server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:30.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.253+0000 7f8360811700 1 -- 172.21.15.60:0/756233072 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f834c00d570 con 0x7f835c0b2130 2022-06-29T15:58:30.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.253+0000 7f8360811700 1 -- 172.21.15.60:0/756233072 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f834c000e90 con 0x7f835c0b2130 2022-06-29T15:58:30.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.253+0000 7f8360811700 1 -- 172.21.15.60:0/756233072 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f834c00d8c0 con 0x7f835c0b2130 2022-06-29T15:58:30.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.253+0000 7f8362815700 1 -- 172.21.15.60:0/756233072 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f835c0b2130 msgr2=0x7f835c146100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:30.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.253+0000 7f8362815700 1 --2- 172.21.15.60:0/756233072 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f835c0b2130 0x7f835c146100 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f834c002f70 tx=0x7f834c006df0).stop 2022-06-29T15:58:30.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.253+0000 7f8362815700 1 -- 172.21.15.60:0/756233072 shutdown_connections 2022-06-29T15:58:30.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.253+0000 7f8362815700 1 --2- 172.21.15.60:0/756233072 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f835c0aed00 0x7f835c1465f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.253+0000 7f8362815700 1 --2- 172.21.15.60:0/756233072 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f835c0b2130 0x7f835c146100 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.253+0000 7f8362815700 1 -- 172.21.15.60:0/756233072 >> 172.21.15.60:0/756233072 conn(0x7f835c0b5890 msgr2=0x7f835c0b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:30.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.253+0000 7f8362815700 1 -- 172.21.15.60:0/756233072 shutdown_connections 2022-06-29T15:58:30.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.253+0000 7f8362815700 1 -- 172.21.15.60:0/756233072 wait complete. 2022-06-29T15:58:30.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.253+0000 7f8362815700 1 Processor -- start 2022-06-29T15:58:30.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.253+0000 7f8362815700 1 -- start start 2022-06-29T15:58:30.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.253+0000 7f8362815700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f835c05d070 0x7f835c066810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:30.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.253+0000 7f8362815700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f835c0aed00 0x7f835c0653c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:30.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.253+0000 7f8362815700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f835c0b2130 0x7f835c116780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:30.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.253+0000 7f8362815700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f835c0662e0 con 0x7f835c0aed00 2022-06-29T15:58:30.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.253+0000 7f8362815700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f835c117a30 con 0x7f835c05d070 2022-06-29T15:58:30.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.253+0000 7f8362815700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f835c118510 con 0x7f835c0b2130 2022-06-29T15:58:30.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.253+0000 7f8361012700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f835c0aed00 0x7f835c0653c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:30.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.257+0000 7f8361012700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f835c0aed00 0x7f835c0653c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45280/0 (socket says 172.21.15.60:45280) 2022-06-29T15:58:30.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.257+0000 7f8361012700 1 -- 172.21.15.60:0/526290591 learned_addr learned my addr 172.21.15.60:0/526290591 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:30.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.257+0000 7f8362014700 1 --2- 172.21.15.60:0/526290591 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f835c0b2130 0x7f835c116780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:30.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.257+0000 7f8361813700 1 --2- 172.21.15.60:0/526290591 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f835c05d070 0x7f835c066810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:30.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.257+0000 7f8361012700 1 -- 172.21.15.60:0/526290591 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f835c0b2130 msgr2=0x7f835c116780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:30.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.257+0000 7f8361012700 1 --2- 172.21.15.60:0/526290591 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f835c0b2130 0x7f835c116780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.257+0000 7f8361012700 1 -- 172.21.15.60:0/526290591 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f835c05d070 msgr2=0x7f835c066810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:30.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.257+0000 7f8361012700 1 --2- 172.21.15.60:0/526290591 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f835c05d070 0x7f835c066810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.257+0000 7f8361012700 1 -- 172.21.15.60:0/526290591 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f834c00b180 con 0x7f835c0aed00 2022-06-29T15:58:30.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.257+0000 7f8361012700 1 --2- 172.21.15.60:0/526290591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f835c0aed00 0x7f835c0653c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f834c002f70 tx=0x7f834c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:30.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.257+0000 7f834affd700 1 -- 172.21.15.60:0/526290591 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f834c00dea0 con 0x7f835c0aed00 2022-06-29T15:58:30.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.257+0000 7f834affd700 1 -- 172.21.15.60:0/526290591 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f834c013e90 con 0x7f835c0aed00 2022-06-29T15:58:30.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.257+0000 7f8362815700 1 -- 172.21.15.60:0/526290591 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f835c1189e0 con 0x7f835c0aed00 2022-06-29T15:58:30.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.257+0000 7f834affd700 1 -- 172.21.15.60:0/526290591 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f834c01e5f0 con 0x7f835c0aed00 2022-06-29T15:58:30.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.257+0000 7f8362815700 1 -- 172.21.15.60:0/526290591 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f835c066d00 con 0x7f835c0aed00 2022-06-29T15:58:30.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.257+0000 7f8362815700 1 -- 172.21.15.60:0/526290591 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f835c14a5f0 con 0x7f835c0aed00 2022-06-29T15:58:30.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.257+0000 7f834affd700 1 -- 172.21.15.60:0/526290591 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f834c025030 con 0x7f835c0aed00 2022-06-29T15:58:30.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.257+0000 7f834affd700 1 --2- 172.21.15.60:0/526290591 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f833c053b00 0x7f833c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:30.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.257+0000 7f834affd700 1 -- 172.21.15.60:0/526290591 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f834c06cce0 con 0x7f835c0aed00 2022-06-29T15:58:30.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.257+0000 7f8361813700 1 --2- 172.21.15.60:0/526290591 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f833c053b00 0x7f833c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:30.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.261+0000 7f8361813700 1 --2- 172.21.15.60:0/526290591 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f833c053b00 0x7f833c055fc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f835400d000 tx=0x7f8354007da0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:30.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.265+0000 7f834affd700 1 -- 172.21.15.60:0/526290591 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f834c043b60 con 0x7f835c0aed00 2022-06-29T15:58:30.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.393+0000 7f8362815700 1 -- 172.21.15.60:0/526290591 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f835c0573d0 con 0x7f835c0aed00 2022-06-29T15:58:30.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.393+0000 7f834affd700 1 -- 172.21.15.60:0/526290591 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f834c023070 con 0x7f835c0aed00 2022-06-29T15:58:30.446 INFO:teuthology.orchestra.run.smithi060.stdout:55834574853 2022-06-29T15:58:30.448 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.397+0000 7f8362815700 1 -- 172.21.15.60:0/526290591 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f833c053b00 msgr2=0x7f833c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:30.449 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.397+0000 7f8362815700 1 --2- 172.21.15.60:0/526290591 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f833c053b00 0x7f833c055fc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f835400d000 tx=0x7f8354007da0).stop 2022-06-29T15:58:30.449 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.397+0000 7f8362815700 1 -- 172.21.15.60:0/526290591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f835c0aed00 msgr2=0x7f835c0653c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:30.449 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.397+0000 7f8362815700 1 --2- 172.21.15.60:0/526290591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f835c0aed00 0x7f835c0653c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f834c002f70 tx=0x7f834c014000).stop 2022-06-29T15:58:30.449 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.397+0000 7f8362815700 1 -- 172.21.15.60:0/526290591 shutdown_connections 2022-06-29T15:58:30.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.397+0000 7f8362815700 1 --2- 172.21.15.60:0/526290591 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f833c053b00 0x7f833c055fc0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.397+0000 7f8362815700 1 --2- 172.21.15.60:0/526290591 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f835c0b2130 0x7f835c116780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.397+0000 7f8362815700 1 --2- 172.21.15.60:0/526290591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f835c0aed00 0x7f835c0653c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.397+0000 7f8362815700 1 --2- 172.21.15.60:0/526290591 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f835c05d070 0x7f835c066810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.397+0000 7f8362815700 1 -- 172.21.15.60:0/526290591 >> 172.21.15.60:0/526290591 conn(0x7f835c0b5890 msgr2=0x7f835c0aca20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:30.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.397+0000 7f8362815700 1 -- 172.21.15.60:0/526290591 shutdown_connections 2022-06-29T15:58:30.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.397+0000 7f8362815700 1 -- 172.21.15.60:0/526290591 wait complete. 2022-06-29T15:58:30.451 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.3 2022-06-29T15:58:30.452 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2022-06-29T15:58:30.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.553+0000 7f14c3188700 1 Processor -- start 2022-06-29T15:58:30.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.553+0000 7f14c3188700 1 -- start start 2022-06-29T15:58:30.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.553+0000 7f14c3188700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f14bc066b80 0x7f14bc05f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:30.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.553+0000 7f14c3188700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f14bc0b2b50 0x7f14bc140e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:30.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.553+0000 7f14c3188700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14bc1407e0 con 0x7f14bc0b3060 2022-06-29T15:58:30.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.553+0000 7f14c3188700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14bc1441b0 con 0x7f14bc066b80 2022-06-29T15:58:30.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.553+0000 7f14c3188700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14bc140a20 con 0x7f14bc0b2b50 2022-06-29T15:58:30.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.553+0000 7f14c2186700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f14bc066b80 0x7f14bc05f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:30.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.553+0000 7f14c2186700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f14bc066b80 0x7f14bc05f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45288/0 (socket says 172.21.15.60:45288) 2022-06-29T15:58:30.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.553+0000 7f14c2186700 1 -- 172.21.15.60:0/1038859330 learned_addr learned my addr 172.21.15.60:0/1038859330 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:30.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.553+0000 7f14c2987700 1 --2- 172.21.15.60:0/1038859330 >> v2:172.21.15.92:3300/0 conn(0x7f14bc0b2b50 0x7f14bc140e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:30.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.553+0000 7f14c2186700 1 -- 172.21.15.60:0/1038859330 >> v2:172.21.15.92:3300/0 conn(0x7f14bc0b2b50 msgr2=0x7f14bc140e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:30.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.553+0000 7f14c2186700 1 --2- 172.21.15.60:0/1038859330 >> v2:172.21.15.92:3300/0 conn(0x7f14bc0b2b50 0x7f14bc140e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.553+0000 7f14c2186700 1 -- 172.21.15.60:0/1038859330 >> v1:172.21.15.60:6789/0 conn(0x7f14bc0b3060 legacy=0x7f14bc13a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:30.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.553+0000 7f14c2186700 1 -- 172.21.15.60:0/1038859330 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14bc05f9e0 con 0x7f14bc066b80 2022-06-29T15:58:30.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.553+0000 7f14c2186700 1 --2- 172.21.15.60:0/1038859330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14bc066b80 0x7f14bc05f4f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f14b8002f70 tx=0x7f14b8006eb0).ready entity=mon.0 client_cookie=ec87a098c6663a56 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:30.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.557+0000 7f14c1184700 1 -- 172.21.15.60:0/1038859330 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14b800d610 con 0x7f14bc066b80 2022-06-29T15:58:30.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.557+0000 7f14c1184700 1 -- 172.21.15.60:0/1038859330 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b8000e90 con 0x7f14bc066b80 2022-06-29T15:58:30.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.557+0000 7f14c1184700 1 -- 172.21.15.60:0/1038859330 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14b800d9a0 con 0x7f14bc066b80 2022-06-29T15:58:30.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.557+0000 7f14c3188700 1 -- 172.21.15.60:0/1038859330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14bc066b80 msgr2=0x7f14bc05f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:30.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.557+0000 7f14c3188700 1 --2- 172.21.15.60:0/1038859330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14bc066b80 0x7f14bc05f4f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f14b8002f70 tx=0x7f14b8006eb0).stop 2022-06-29T15:58:30.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.557+0000 7f14c3188700 1 -- 172.21.15.60:0/1038859330 shutdown_connections 2022-06-29T15:58:30.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.557+0000 7f14c3188700 1 --2- 172.21.15.60:0/1038859330 >> v2:172.21.15.92:3300/0 conn(0x7f14bc0b2b50 0x7f14bc140e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.557+0000 7f14c3188700 1 --2- 172.21.15.60:0/1038859330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14bc066b80 0x7f14bc05f4f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.557+0000 7f14c3188700 1 -- 172.21.15.60:0/1038859330 >> 172.21.15.60:0/1038859330 conn(0x7f14bc061c80 msgr2=0x7f14bc0637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:30.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.557+0000 7f14c3188700 1 -- 172.21.15.60:0/1038859330 shutdown_connections 2022-06-29T15:58:30.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.557+0000 7f14c3188700 1 -- 172.21.15.60:0/1038859330 wait complete. 2022-06-29T15:58:30.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.561+0000 7f14c3188700 1 Processor -- start 2022-06-29T15:58:30.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.561+0000 7f14c3188700 1 -- start start 2022-06-29T15:58:30.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.561+0000 7f14c3188700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14bc066b80 0x7f14bc0ad670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:30.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.561+0000 7f14c3188700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14bc0b2b50 0x7f14bc1379b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:30.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.561+0000 7f14c3188700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14bc0b3060 0x7f14bc137ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:30.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.561+0000 7f14c3188700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14bc0b02f0 con 0x7f14bc066b80 2022-06-29T15:58:30.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.561+0000 7f14c3188700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14bc13e710 con 0x7f14bc0b3060 2022-06-29T15:58:30.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.561+0000 7f14c3188700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14bc13e9b0 con 0x7f14bc0b2b50 2022-06-29T15:58:30.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.561+0000 7f14c2186700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14bc066b80 0x7f14bc0ad670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:30.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.561+0000 7f14c2186700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14bc066b80 0x7f14bc0ad670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45294/0 (socket says 172.21.15.60:45294) 2022-06-29T15:58:30.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.561+0000 7f14c2186700 1 -- 172.21.15.60:0/4256183178 learned_addr learned my addr 172.21.15.60:0/4256183178 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:30.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.561+0000 7f14c1985700 1 --2- 172.21.15.60:0/4256183178 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14bc0b2b50 0x7f14bc1379b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:30.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.561+0000 7f14c2987700 1 --2- 172.21.15.60:0/4256183178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14bc0b3060 0x7f14bc137ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:30.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.561+0000 7f14c2186700 1 -- 172.21.15.60:0/4256183178 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14bc0b2b50 msgr2=0x7f14bc1379b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:30.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.561+0000 7f14c2186700 1 --2- 172.21.15.60:0/4256183178 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14bc0b2b50 0x7f14bc1379b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.561+0000 7f14c2186700 1 -- 172.21.15.60:0/4256183178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14bc0b3060 msgr2=0x7f14bc137ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:30.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.561+0000 7f14c2186700 1 --2- 172.21.15.60:0/4256183178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14bc0b3060 0x7f14bc137ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.561+0000 7f14c2186700 1 -- 172.21.15.60:0/4256183178 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b800b1e0 con 0x7f14bc066b80 2022-06-29T15:58:30.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.561+0000 7f14c2186700 1 --2- 172.21.15.60:0/4256183178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14bc066b80 0x7f14bc0ad670 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f14b8002f70 tx=0x7f14b8015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:30.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.561+0000 7f14ab7fe700 1 -- 172.21.15.60:0/4256183178 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14b8011860 con 0x7f14bc066b80 2022-06-29T15:58:30.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.561+0000 7f14ab7fe700 1 -- 172.21.15.60:0/4256183178 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b80119c0 con 0x7f14bc066b80 2022-06-29T15:58:30.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.561+0000 7f14ab7fe700 1 -- 172.21.15.60:0/4256183178 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14b8011d20 con 0x7f14bc066b80 2022-06-29T15:58:30.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.561+0000 7f14c3188700 1 -- 172.21.15.60:0/4256183178 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14bc14f810 con 0x7f14bc066b80 2022-06-29T15:58:30.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.561+0000 7f14c3188700 1 -- 172.21.15.60:0/4256183178 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14bc0aff70 con 0x7f14bc066b80 2022-06-29T15:58:30.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.565+0000 7f14c3188700 1 -- 172.21.15.60:0/4256183178 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14bc0573d0 con 0x7f14bc066b80 2022-06-29T15:58:30.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.565+0000 7f14ab7fe700 1 -- 172.21.15.60:0/4256183178 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f14b801a040 con 0x7f14bc066b80 2022-06-29T15:58:30.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.565+0000 7f14ab7fe700 1 --2- 172.21.15.60:0/4256183178 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f149c053b30 0x7f149c055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:30.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.565+0000 7f14ab7fe700 1 -- 172.21.15.60:0/4256183178 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f14b806cc80 con 0x7f14bc066b80 2022-06-29T15:58:30.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.565+0000 7f14c1985700 1 --2- 172.21.15.60:0/4256183178 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f149c053b30 0x7f149c055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:30.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.565+0000 7f14c1985700 1 --2- 172.21.15.60:0/4256183178 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f149c053b30 0x7f149c055ff0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f14ac00d000 tx=0x7f14ac007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:30.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.573+0000 7f14ab7fe700 1 -- 172.21.15.60:0/4256183178 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f14b8043ba0 con 0x7f14bc066b80 2022-06-29T15:58:30.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.701+0000 7f14c3188700 1 -- 172.21.15.60:0/4256183178 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f14bc0aec90 con 0x7f14bc066b80 2022-06-29T15:58:30.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.701+0000 7f14ab7fe700 1 -- 172.21.15.60:0/4256183178 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f14b80471c0 con 0x7f14bc066b80 2022-06-29T15:58:30.708 INFO:teuthology.orchestra.run.smithi060.stdout:55834574854 2022-06-29T15:58:30.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.701+0000 7f14c3188700 1 -- 172.21.15.60:0/4256183178 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f149c053b30 msgr2=0x7f149c055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:30.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.701+0000 7f14c3188700 1 --2- 172.21.15.60:0/4256183178 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f149c053b30 0x7f149c055ff0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f14ac00d000 tx=0x7f14ac007b90).stop 2022-06-29T15:58:30.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.701+0000 7f14c3188700 1 -- 172.21.15.60:0/4256183178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14bc066b80 msgr2=0x7f14bc0ad670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:30.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.701+0000 7f14c3188700 1 --2- 172.21.15.60:0/4256183178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14bc066b80 0x7f14bc0ad670 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f14b8002f70 tx=0x7f14b8015000).stop 2022-06-29T15:58:30.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.701+0000 7f14c3188700 1 -- 172.21.15.60:0/4256183178 shutdown_connections 2022-06-29T15:58:30.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.701+0000 7f14c3188700 1 --2- 172.21.15.60:0/4256183178 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f149c053b30 0x7f149c055ff0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.701+0000 7f14c3188700 1 --2- 172.21.15.60:0/4256183178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14bc0b3060 0x7f14bc137ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.701+0000 7f14c3188700 1 --2- 172.21.15.60:0/4256183178 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14bc0b2b50 0x7f14bc1379b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.701+0000 7f14c3188700 1 --2- 172.21.15.60:0/4256183178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14bc066b80 0x7f14bc0ad670 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.701+0000 7f14c3188700 1 -- 172.21.15.60:0/4256183178 >> 172.21.15.60:0/4256183178 conn(0x7f14bc061c80 msgr2=0x7f14bc0b4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:30.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.701+0000 7f14c3188700 1 -- 172.21.15.60:0/4256183178 shutdown_connections 2022-06-29T15:58:30.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.701+0000 7f14c3188700 1 -- 172.21.15.60:0/4256183178 wait complete. 2022-06-29T15:58:30.719 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.4 2022-06-29T15:58:30.720 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2022-06-29T15:58:30.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.821+0000 7fc076d01700 1 Processor -- start 2022-06-29T15:58:30.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.821+0000 7fc076d01700 1 -- start start 2022-06-29T15:58:30.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.821+0000 7fc076d01700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc0700b3c70 0x7fc070065ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:30.829 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.821+0000 7fc076d01700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc070065820 con 0x7fc07013fa70 2022-06-29T15:58:30.829 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.821+0000 7fc076d01700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0701405d0 con 0x7fc07005bf30 2022-06-29T15:58:30.829 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.821+0000 7fc076d01700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0700b4050 con 0x7fc0700b3c70 2022-06-29T15:58:30.829 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.821+0000 7fc076500700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc07013fa70 0x7fc07013fec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53392/0 (socket says 172.21.15.60:53392) 2022-06-29T15:58:30.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.821+0000 7fc076500700 1 -- 172.21.15.60:0/3031761424 learned_addr learned my addr 172.21.15.60:0/3031761424 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:30.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.821+0000 7fc075cff700 1 --2- 172.21.15.60:0/3031761424 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc0700b3c70 0x7fc070065ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:30.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.821+0000 7fc074cfd700 1 -- 172.21.15.60:0/3031761424 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2332487361 0 0) 0x7fc070065820 con 0x7fc07013fa70 2022-06-29T15:58:30.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.821+0000 7fc074cfd700 1 -- 172.21.15.60:0/3031761424 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc058003260 con 0x7fc07013fa70 2022-06-29T15:58:30.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.821+0000 7fc074cfd700 1 -- 172.21.15.60:0/3031761424 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 304693237 0 0) 0x7fc0701405d0 con 0x7fc07005bf30 2022-06-29T15:58:30.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.821+0000 7fc074cfd700 1 -- 172.21.15.60:0/3031761424 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc070065820 con 0x7fc07005bf30 2022-06-29T15:58:30.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.821+0000 7fc075cff700 1 -- 172.21.15.60:0/3031761424 >> v1:172.21.15.92:6789/0 conn(0x7fc07005bf30 legacy=0x7fc070143cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:30.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.821+0000 7fc075cff700 1 -- 172.21.15.60:0/3031761424 >> v1:172.21.15.60:6789/0 conn(0x7fc07013fa70 legacy=0x7fc07013fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:30.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.821+0000 7fc075cff700 1 -- 172.21.15.60:0/3031761424 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc07005c4e0 con 0x7fc0700b3c70 2022-06-29T15:58:30.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.821+0000 7fc075cff700 1 --2- 172.21.15.60:0/3031761424 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc0700b3c70 0x7fc070065ef0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fc064002f70 tx=0x7fc064006df0).ready entity=mon.2 client_cookie=40e8b9ebda6a89c0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:30.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.821+0000 7fc074cfd700 1 -- 172.21.15.60:0/3031761424 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc06400d5c0 con 0x7fc0700b3c70 2022-06-29T15:58:30.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.821+0000 7fc074cfd700 1 -- 172.21.15.60:0/3031761424 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc064000e90 con 0x7fc0700b3c70 2022-06-29T15:58:30.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.821+0000 7fc074cfd700 1 -- 172.21.15.60:0/3031761424 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc06400d910 con 0x7fc0700b3c70 2022-06-29T15:58:30.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.825+0000 7fc076d01700 1 -- 172.21.15.60:0/3031761424 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc0700b3c70 msgr2=0x7fc070065ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:30.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.825+0000 7fc076d01700 1 --2- 172.21.15.60:0/3031761424 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc0700b3c70 0x7fc070065ef0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fc064002f70 tx=0x7fc064006df0).stop 2022-06-29T15:58:30.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.825+0000 7fc076d01700 1 -- 172.21.15.60:0/3031761424 shutdown_connections 2022-06-29T15:58:30.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.825+0000 7fc076d01700 1 --2- 172.21.15.60:0/3031761424 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc0700b3c70 0x7fc070065ef0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.843 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.825+0000 7fc076d01700 1 -- 172.21.15.60:0/3031761424 >> 172.21.15.60:0/3031761424 conn(0x7fc070064810 msgr2=0x7fc070064bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:30.843 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.825+0000 7fc076d01700 1 -- 172.21.15.60:0/3031761424 shutdown_connections 2022-06-29T15:58:30.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.825+0000 7fc076d01700 1 -- 172.21.15.60:0/3031761424 wait complete. 2022-06-29T15:58:30.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.825+0000 7fc076d01700 1 Processor -- start 2022-06-29T15:58:30.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.825+0000 7fc076d01700 1 -- start start 2022-06-29T15:58:30.845 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.829+0000 7fc076d01700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc07005bf30 0x7fc0700b07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:30.845 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.829+0000 7fc076d01700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc0700b3c70 0x7fc0700b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:30.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.829+0000 7fc076d01700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc07013fa70 0x7fc07013d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:30.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.829+0000 7fc076d01700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0700ae5a0 con 0x7fc07013fa70 2022-06-29T15:58:30.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.829+0000 7fc076d01700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0700af4d0 con 0x7fc07005bf30 2022-06-29T15:58:30.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.829+0000 7fc076d01700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc070112190 con 0x7fc0700b3c70 2022-06-29T15:58:30.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.829+0000 7fc076500700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc07013fa70 0x7fc07013d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:30.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.829+0000 7fc076500700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc07013fa70 0x7fc07013d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45312/0 (socket says 172.21.15.60:45312) 2022-06-29T15:58:30.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.829+0000 7fc076500700 1 -- 172.21.15.60:0/1788829040 learned_addr learned my addr 172.21.15.60:0/1788829040 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:30.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.829+0000 7fc0754fe700 1 --2- 172.21.15.60:0/1788829040 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc0700b3c70 0x7fc0700b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:30.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.829+0000 7fc075cff700 1 --2- 172.21.15.60:0/1788829040 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc07005bf30 0x7fc0700b07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:30.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.829+0000 7fc076500700 1 -- 172.21.15.60:0/1788829040 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc0700b3c70 msgr2=0x7fc0700b0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:30.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.829+0000 7fc076500700 1 --2- 172.21.15.60:0/1788829040 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc0700b3c70 0x7fc0700b0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.829+0000 7fc076500700 1 -- 172.21.15.60:0/1788829040 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc07005bf30 msgr2=0x7fc0700b07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:30.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.829+0000 7fc076500700 1 --2- 172.21.15.60:0/1788829040 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc07005bf30 0x7fc0700b07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.829+0000 7fc076500700 1 -- 172.21.15.60:0/1788829040 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc06400b180 con 0x7fc07013fa70 2022-06-29T15:58:30.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.829+0000 7fc075cff700 1 --2- 172.21.15.60:0/1788829040 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc07005bf30 0x7fc0700b07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:58:30.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.829+0000 7fc076500700 1 --2- 172.21.15.60:0/1788829040 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc07013fa70 0x7fc07013d730 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fc068001110 tx=0x7fc068007a80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:30.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.829+0000 7fc062ffd700 1 -- 172.21.15.60:0/1788829040 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0680070d0 con 0x7fc07013fa70 2022-06-29T15:58:30.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.829+0000 7fc062ffd700 1 -- 172.21.15.60:0/1788829040 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc068007230 con 0x7fc07013fa70 2022-06-29T15:58:30.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.829+0000 7fc062ffd700 1 -- 172.21.15.60:0/1788829040 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc068005610 con 0x7fc07013fa70 2022-06-29T15:58:30.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.829+0000 7fc076d01700 1 -- 172.21.15.60:0/1788829040 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0700af030 con 0x7fc07013fa70 2022-06-29T15:58:30.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.829+0000 7fc076d01700 1 -- 172.21.15.60:0/1788829040 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc070111e80 con 0x7fc07013fa70 2022-06-29T15:58:30.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.829+0000 7fc076d01700 1 -- 172.21.15.60:0/1788829040 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0700af1c0 con 0x7fc07013fa70 2022-06-29T15:58:30.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.833+0000 7fc062ffd700 1 -- 172.21.15.60:0/1788829040 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc068010070 con 0x7fc07013fa70 2022-06-29T15:58:30.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.833+0000 7fc062ffd700 1 --2- 172.21.15.60:0/1788829040 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc050053b00 0x7fc050055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:30.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.833+0000 7fc062ffd700 1 -- 172.21.15.60:0/1788829040 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc06806b740 con 0x7fc07013fa70 2022-06-29T15:58:30.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.833+0000 7fc075cff700 1 --2- 172.21.15.60:0/1788829040 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc050053b00 0x7fc050055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:30.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.833+0000 7fc075cff700 1 --2- 172.21.15.60:0/1788829040 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc050053b00 0x7fc050055fc0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fc064018000 tx=0x7fc064014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:30.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.837+0000 7fc062ffd700 1 -- 172.21.15.60:0/1788829040 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc0680424d0 con 0x7fc07013fa70 2022-06-29T15:58:30.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.965+0000 7fc076d01700 1 -- 172.21.15.60:0/1788829040 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fc0700573d0 con 0x7fc07013fa70 2022-06-29T15:58:30.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.965+0000 7fc062ffd700 1 -- 172.21.15.60:0/1788829040 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc068045af0 con 0x7fc07013fa70 2022-06-29T15:58:30.974 INFO:teuthology.orchestra.run.smithi060.stdout:55834574851 2022-06-29T15:58:30.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.969+0000 7fc076d01700 1 -- 172.21.15.60:0/1788829040 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc050053b00 msgr2=0x7fc050055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:30.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.969+0000 7fc076d01700 1 --2- 172.21.15.60:0/1788829040 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc050053b00 0x7fc050055fc0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fc064018000 tx=0x7fc064014000).stop 2022-06-29T15:58:30.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.969+0000 7fc076d01700 1 -- 172.21.15.60:0/1788829040 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc07013fa70 msgr2=0x7fc07013d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:30.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.969+0000 7fc076d01700 1 --2- 172.21.15.60:0/1788829040 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc07013fa70 0x7fc07013d730 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fc068001110 tx=0x7fc068007a80).stop 2022-06-29T15:58:30.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.969+0000 7fc076d01700 1 -- 172.21.15.60:0/1788829040 shutdown_connections 2022-06-29T15:58:30.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.969+0000 7fc076d01700 1 --2- 172.21.15.60:0/1788829040 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc050053b00 0x7fc050055fc0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.969+0000 7fc076d01700 1 --2- 172.21.15.60:0/1788829040 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc07013fa70 0x7fc07013d730 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.969+0000 7fc076d01700 1 --2- 172.21.15.60:0/1788829040 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc0700b3c70 0x7fc0700b0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.969+0000 7fc076d01700 1 --2- 172.21.15.60:0/1788829040 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc07005bf30 0x7fc0700b07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:30.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.969+0000 7fc076d01700 1 -- 172.21.15.60:0/1788829040 >> 172.21.15.60:0/1788829040 conn(0x7fc070064810 msgr2=0x7fc070145db0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:30.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.969+0000 7fc076d01700 1 -- 172.21.15.60:0/1788829040 shutdown_connections 2022-06-29T15:58:30.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:30.969+0000 7fc076d01700 1 -- 172.21.15.60:0/1788829040 wait complete. 2022-06-29T15:58:30.986 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.5 2022-06-29T15:58:31.987 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2022-06-29T15:58:32.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.097+0000 7f5249dfa700 1 Processor -- start 2022-06-29T15:58:32.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.097+0000 7f5249dfa700 1 -- start start 2022-06-29T15:58:32.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.101+0000 7f5249dfa700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f524405d1f0 0x7f52440b1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:32.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.101+0000 7f5249dfa700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f52440b3cc0 0x7f52440b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:32.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.101+0000 7f5249dfa700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52440b2ce0 con 0x7f52440b9d70 2022-06-29T15:58:32.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.101+0000 7f5249dfa700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52440b4d10 con 0x7f524405d1f0 2022-06-29T15:58:32.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.101+0000 7f5249dfa700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52440ba150 con 0x7f52440b3cc0 2022-06-29T15:58:32.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.101+0000 7f5242ffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f524405d1f0 0x7f52440b1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:32.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.101+0000 7f5242ffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f524405d1f0 0x7f52440b1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45318/0 (socket says 172.21.15.60:45318) 2022-06-29T15:58:32.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.101+0000 7f5242ffd700 1 -- 172.21.15.60:0/4274819668 learned_addr learned my addr 172.21.15.60:0/4274819668 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:32.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.101+0000 7f5243fff700 1 --2- 172.21.15.60:0/4274819668 >> v2:172.21.15.92:3300/0 conn(0x7f52440b3cc0 0x7f52440b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:32.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.101+0000 7f5242ffd700 1 -- 172.21.15.60:0/4274819668 >> v2:172.21.15.92:3300/0 conn(0x7f52440b3cc0 msgr2=0x7f52440b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:32.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.101+0000 7f5242ffd700 1 --2- 172.21.15.60:0/4274819668 >> v2:172.21.15.92:3300/0 conn(0x7f52440b3cc0 0x7f52440b2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:32.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.101+0000 7f5242ffd700 1 -- 172.21.15.60:0/4274819668 >> v1:172.21.15.92:6789/0 conn(0x7f52440b9d70 legacy=0x7f52440b3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:32.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.101+0000 7f5242ffd700 1 -- 172.21.15.60:0/4274819668 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52440b4a70 con 0x7f524405d1f0 2022-06-29T15:58:32.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.101+0000 7f5242ffd700 1 --2- 172.21.15.60:0/4274819668 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f524405d1f0 0x7f52440b1b50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f5234002f70 tx=0x7f5234006eb0).ready entity=mon.0 client_cookie=be2a338d9097614d server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:32.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.101+0000 7f52427fc700 1 -- 172.21.15.60:0/4274819668 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f523400d5c0 con 0x7f524405d1f0 2022-06-29T15:58:32.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.101+0000 7f52427fc700 1 -- 172.21.15.60:0/4274819668 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5234000e90 con 0x7f524405d1f0 2022-06-29T15:58:32.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.101+0000 7f52427fc700 1 -- 172.21.15.60:0/4274819668 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f523400d950 con 0x7f524405d1f0 2022-06-29T15:58:32.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.101+0000 7f5249dfa700 1 -- 172.21.15.60:0/4274819668 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f524405d1f0 msgr2=0x7f52440b1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:32.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.101+0000 7f5249dfa700 1 --2- 172.21.15.60:0/4274819668 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f524405d1f0 0x7f52440b1b50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f5234002f70 tx=0x7f5234006eb0).stop 2022-06-29T15:58:32.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.101+0000 7f5249dfa700 1 -- 172.21.15.60:0/4274819668 shutdown_connections 2022-06-29T15:58:32.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.101+0000 7f5249dfa700 1 --2- 172.21.15.60:0/4274819668 >> v2:172.21.15.92:3300/0 conn(0x7f52440b3cc0 0x7f52440b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:32.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.101+0000 7f5249dfa700 1 --2- 172.21.15.60:0/4274819668 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f524405d1f0 0x7f52440b1b50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:32.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.101+0000 7f5249dfa700 1 -- 172.21.15.60:0/4274819668 >> 172.21.15.60:0/4274819668 conn(0x7f5244062ce0 msgr2=0x7f52440b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:32.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.101+0000 7f5249dfa700 1 -- 172.21.15.60:0/4274819668 shutdown_connections 2022-06-29T15:58:32.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.101+0000 7f5249dfa700 1 -- 172.21.15.60:0/4274819668 wait complete. 2022-06-29T15:58:32.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.105+0000 7f5249dfa700 1 Processor -- start 2022-06-29T15:58:32.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.105+0000 7f5249dfa700 1 -- start start 2022-06-29T15:58:32.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.105+0000 7f5249dfa700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f524405d1f0 0x7f5244142a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:32.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.105+0000 7f5249dfa700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52440b3cc0 0x7f524413c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:32.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.105+0000 7f5249dfa700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52440b9d70 0x7f524413cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:32.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.105+0000 7f5249dfa700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5244152f10 con 0x7f52440b3cc0 2022-06-29T15:58:32.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.105+0000 7f5249dfa700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f524413d190 con 0x7f52440b9d70 2022-06-29T15:58:32.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.105+0000 7f5249dfa700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5244152010 con 0x7f524405d1f0 2022-06-29T15:58:32.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.105+0000 7f5242ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52440b3cc0 0x7f524413c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:32.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.105+0000 7f5242ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52440b3cc0 0x7f524413c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45322/0 (socket says 172.21.15.60:45322) 2022-06-29T15:58:32.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.105+0000 7f5242ffd700 1 -- 172.21.15.60:0/3176168116 learned_addr learned my addr 172.21.15.60:0/3176168116 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:32.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.105+0000 7f52437fe700 1 --2- 172.21.15.60:0/3176168116 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f524405d1f0 0x7f5244142a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:32.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.105+0000 7f5243fff700 1 --2- 172.21.15.60:0/3176168116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52440b9d70 0x7f524413cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:32.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.105+0000 7f5242ffd700 1 -- 172.21.15.60:0/3176168116 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f524405d1f0 msgr2=0x7f5244142a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:32.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.105+0000 7f5242ffd700 1 --2- 172.21.15.60:0/3176168116 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f524405d1f0 0x7f5244142a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:32.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.105+0000 7f5242ffd700 1 -- 172.21.15.60:0/3176168116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52440b9d70 msgr2=0x7f524413cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:32.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.105+0000 7f5242ffd700 1 --2- 172.21.15.60:0/3176168116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52440b9d70 0x7f524413cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:32.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.105+0000 7f5242ffd700 1 -- 172.21.15.60:0/3176168116 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f523400b1e0 con 0x7f52440b3cc0 2022-06-29T15:58:32.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.105+0000 7f5242ffd700 1 --2- 172.21.15.60:0/3176168116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52440b3cc0 0x7f524413c7b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f5234002f70 tx=0x7f5234015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:32.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.105+0000 7f5240ff9700 1 -- 172.21.15.60:0/3176168116 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5234011990 con 0x7f52440b3cc0 2022-06-29T15:58:32.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.105+0000 7f5240ff9700 1 -- 172.21.15.60:0/3176168116 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5234011af0 con 0x7f52440b3cc0 2022-06-29T15:58:32.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.105+0000 7f5240ff9700 1 -- 172.21.15.60:0/3176168116 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5234011e50 con 0x7f52440b3cc0 2022-06-29T15:58:32.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.105+0000 7f5249dfa700 1 -- 172.21.15.60:0/3176168116 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52441522b0 con 0x7f52440b3cc0 2022-06-29T15:58:32.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.105+0000 7f5249dfa700 1 -- 172.21.15.60:0/3176168116 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5244151d00 con 0x7f52440b3cc0 2022-06-29T15:58:32.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.109+0000 7f5249dfa700 1 -- 172.21.15.60:0/3176168116 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52440573d0 con 0x7f52440b3cc0 2022-06-29T15:58:32.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.109+0000 7f5240ff9700 1 -- 172.21.15.60:0/3176168116 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f523401a040 con 0x7f52440b3cc0 2022-06-29T15:58:32.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.109+0000 7f5240ff9700 1 --2- 172.21.15.60:0/3176168116 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5224053a70 0x7f5224055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:32.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.109+0000 7f5240ff9700 1 -- 172.21.15.60:0/3176168116 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f523406cf40 con 0x7f52440b3cc0 2022-06-29T15:58:32.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.109+0000 7f52437fe700 1 --2- 172.21.15.60:0/3176168116 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5224053a70 0x7f5224055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:32.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.109+0000 7f52437fe700 1 --2- 172.21.15.60:0/3176168116 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5224053a70 0x7f5224055f30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f523c00b000 tx=0x7f523c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:32.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.113+0000 7f5240ff9700 1 -- 172.21.15.60:0/3176168116 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5234043d50 con 0x7f52440b3cc0 2022-06-29T15:58:32.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.245+0000 7f5249dfa700 1 -- 172.21.15.60:0/3176168116 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f524405d990 con 0x7f52440b3cc0 2022-06-29T15:58:32.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.245+0000 7f5240ff9700 1 -- 172.21.15.60:0/3176168116 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5234023020 con 0x7f52440b3cc0 2022-06-29T15:58:32.253 INFO:teuthology.orchestra.run.smithi060.stdout:55834574853 2022-06-29T15:58:32.255 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.249+0000 7f5249dfa700 1 -- 172.21.15.60:0/3176168116 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5224053a70 msgr2=0x7f5224055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:32.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.249+0000 7f5249dfa700 1 --2- 172.21.15.60:0/3176168116 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5224053a70 0x7f5224055f30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f523c00b000 tx=0x7f523c009040).stop 2022-06-29T15:58:32.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.249+0000 7f5249dfa700 1 -- 172.21.15.60:0/3176168116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52440b3cc0 msgr2=0x7f524413c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:32.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.249+0000 7f5249dfa700 1 --2- 172.21.15.60:0/3176168116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52440b3cc0 0x7f524413c7b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f5234002f70 tx=0x7f5234015000).stop 2022-06-29T15:58:32.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.249+0000 7f5249dfa700 1 -- 172.21.15.60:0/3176168116 shutdown_connections 2022-06-29T15:58:32.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.249+0000 7f5249dfa700 1 --2- 172.21.15.60:0/3176168116 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5224053a70 0x7f5224055f30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:32.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.249+0000 7f5249dfa700 1 --2- 172.21.15.60:0/3176168116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52440b9d70 0x7f524413cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:32.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.249+0000 7f5249dfa700 1 --2- 172.21.15.60:0/3176168116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52440b3cc0 0x7f524413c7b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:32.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.249+0000 7f5249dfa700 1 --2- 172.21.15.60:0/3176168116 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f524405d1f0 0x7f5244142a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:32.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.249+0000 7f5249dfa700 1 -- 172.21.15.60:0/3176168116 >> 172.21.15.60:0/3176168116 conn(0x7f5244062ce0 msgr2=0x7f52440b20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:32.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.249+0000 7f5249dfa700 1 -- 172.21.15.60:0/3176168116 shutdown_connections 2022-06-29T15:58:32.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.249+0000 7f5249dfa700 1 -- 172.21.15.60:0/3176168116 wait complete. 2022-06-29T15:58:32.266 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574853 for osd.5 2022-06-29T15:58:32.266 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2022-06-29T15:58:32.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.365+0000 7fb7df59e700 1 Processor -- start 2022-06-29T15:58:32.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.365+0000 7fb7df59e700 1 -- start start 2022-06-29T15:58:32.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.365+0000 7fb7df59e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb7e00b3360 0x7fb7e0142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:32.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.365+0000 7fb7df59e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7e00b0cf0 con 0x7fb7e00b5eb0 2022-06-29T15:58:32.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.365+0000 7fb7df59e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7e00b4d10 con 0x7fb7e00b48b0 2022-06-29T15:58:32.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.365+0000 7fb7df59e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7e00b54e0 con 0x7fb7e00b3360 2022-06-29T15:58:32.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.365+0000 7fb7de59c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb7e00b5eb0 0x7fb7e00b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53416/0 (socket says 172.21.15.60:53416) 2022-06-29T15:58:32.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.365+0000 7fb7de59c700 1 -- 172.21.15.60:0/2852455496 learned_addr learned my addr 172.21.15.60:0/2852455496 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:32.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.365+0000 7fb7ded9d700 1 --2- 172.21.15.60:0/2852455496 >> v2:172.21.15.92:3300/0 conn(0x7fb7e00b3360 0x7fb7e0142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:32.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.365+0000 7fb7dd59a700 1 -- 172.21.15.60:0/2852455496 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2245315965 0 0) 0x7fb7d0001110 con 0x7fb7e00b5eb0 2022-06-29T15:58:32.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.365+0000 7fb7dd59a700 1 -- 172.21.15.60:0/2852455496 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7c4003260 con 0x7fb7e00b5eb0 2022-06-29T15:58:32.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.365+0000 7fb7ded9d700 1 -- 172.21.15.60:0/2852455496 >> v1:172.21.15.92:6789/0 conn(0x7fb7e00b48b0 legacy=0x7fb7e00b2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:32.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.365+0000 7fb7ded9d700 1 -- 172.21.15.60:0/2852455496 >> v1:172.21.15.60:6789/0 conn(0x7fb7e00b5eb0 legacy=0x7fb7e00b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:32.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.365+0000 7fb7ded9d700 1 -- 172.21.15.60:0/2852455496 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e00b6290 con 0x7fb7e00b3360 2022-06-29T15:58:32.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.369+0000 7fb7ded9d700 1 --2- 172.21.15.60:0/2852455496 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7e00b3360 0x7fb7e0142a20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fb7d80032f0 tx=0x7fb7d80091d0).ready entity=mon.1 client_cookie=b8c7416860142c6f server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:32.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.369+0000 7fb7dd59a700 1 -- 172.21.15.60:0/2852455496 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb7d800b6c0 con 0x7fb7e00b3360 2022-06-29T15:58:32.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.369+0000 7fb7dd59a700 1 -- 172.21.15.60:0/2852455496 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7d800b820 con 0x7fb7e00b3360 2022-06-29T15:58:32.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.369+0000 7fb7dd59a700 1 -- 172.21.15.60:0/2852455496 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb7d800b990 con 0x7fb7e00b3360 2022-06-29T15:58:32.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.369+0000 7fb7df59e700 1 -- 172.21.15.60:0/2852455496 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7e00b3360 msgr2=0x7fb7e0142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:32.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.369+0000 7fb7df59e700 1 --2- 172.21.15.60:0/2852455496 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7e00b3360 0x7fb7e0142a20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fb7d80032f0 tx=0x7fb7d80091d0).stop 2022-06-29T15:58:32.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.369+0000 7fb7df59e700 1 -- 172.21.15.60:0/2852455496 shutdown_connections 2022-06-29T15:58:32.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.369+0000 7fb7df59e700 1 --2- 172.21.15.60:0/2852455496 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7e00b3360 0x7fb7e0142a20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:32.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.369+0000 7fb7df59e700 1 -- 172.21.15.60:0/2852455496 >> 172.21.15.60:0/2852455496 conn(0x7fb7e0062090 msgr2=0x7fb7e0062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:32.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.369+0000 7fb7df59e700 1 -- 172.21.15.60:0/2852455496 shutdown_connections 2022-06-29T15:58:32.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.369+0000 7fb7df59e700 1 -- 172.21.15.60:0/2852455496 wait complete. 2022-06-29T15:58:32.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.369+0000 7fb7df59e700 1 Processor -- start 2022-06-29T15:58:32.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.369+0000 7fb7df59e700 1 -- start start 2022-06-29T15:58:32.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.373+0000 7fb7df59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7e00b3360 0x7fb7e0116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:32.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.373+0000 7fb7df59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7e00b48b0 0x7fb7e01166f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:32.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.373+0000 7fb7df59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb7e00b5eb0 0x7fb7e0151f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:32.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.373+0000 7fb7df59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e0152db0 con 0x7fb7e00b48b0 2022-06-29T15:58:32.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.373+0000 7fb7df59e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e01127d0 con 0x7fb7e00b3360 2022-06-29T15:58:32.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.373+0000 7fb7df59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e0112c70 con 0x7fb7e00b5eb0 2022-06-29T15:58:32.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.373+0000 7fb7ddd9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7e00b48b0 0x7fb7e01166f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:32.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.373+0000 7fb7ddd9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7e00b48b0 0x7fb7e01166f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45340/0 (socket says 172.21.15.60:45340) 2022-06-29T15:58:32.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.373+0000 7fb7ddd9b700 1 -- 172.21.15.60:0/3368202226 learned_addr learned my addr 172.21.15.60:0/3368202226 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:32.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.373+0000 7fb7ded9d700 1 --2- 172.21.15.60:0/3368202226 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb7e00b5eb0 0x7fb7e0151f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:32.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.373+0000 7fb7de59c700 1 --2- 172.21.15.60:0/3368202226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7e00b3360 0x7fb7e0116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:32.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.373+0000 7fb7ddd9b700 1 -- 172.21.15.60:0/3368202226 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb7e00b5eb0 msgr2=0x7fb7e0151f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:32.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.373+0000 7fb7ddd9b700 1 --2- 172.21.15.60:0/3368202226 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb7e00b5eb0 0x7fb7e0151f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:32.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.373+0000 7fb7ddd9b700 1 -- 172.21.15.60:0/3368202226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7e00b3360 msgr2=0x7fb7e0116200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:32.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.373+0000 7fb7ddd9b700 1 --2- 172.21.15.60:0/3368202226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7e00b3360 0x7fb7e0116200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:32.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.373+0000 7fb7ddd9b700 1 -- 172.21.15.60:0/3368202226 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d8009470 con 0x7fb7e00b48b0 2022-06-29T15:58:32.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.373+0000 7fb7ddd9b700 1 --2- 172.21.15.60:0/3368202226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7e00b48b0 0x7fb7e01166f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fb7e0062fa0 tx=0x7fb7d4007be0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:32.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.373+0000 7fb7cf7fe700 1 -- 172.21.15.60:0/3368202226 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb7d4007030 con 0x7fb7e00b48b0 2022-06-29T15:58:32.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.373+0000 7fb7cf7fe700 1 -- 172.21.15.60:0/3368202226 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7d4007190 con 0x7fb7e00b48b0 2022-06-29T15:58:32.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.373+0000 7fb7cf7fe700 1 -- 172.21.15.60:0/3368202226 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb7d4006510 con 0x7fb7e00b48b0 2022-06-29T15:58:32.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.373+0000 7fb7df59e700 1 -- 172.21.15.60:0/3368202226 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7e0116be0 con 0x7fb7e00b48b0 2022-06-29T15:58:32.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.373+0000 7fb7df59e700 1 -- 172.21.15.60:0/3368202226 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7e0152470 con 0x7fb7e00b48b0 2022-06-29T15:58:32.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.373+0000 7fb7df59e700 1 -- 172.21.15.60:0/3368202226 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7e0142f10 con 0x7fb7e00b48b0 2022-06-29T15:58:32.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.377+0000 7fb7cf7fe700 1 -- 172.21.15.60:0/3368202226 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb7d4010030 con 0x7fb7e00b48b0 2022-06-29T15:58:32.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.377+0000 7fb7cf7fe700 1 --2- 172.21.15.60:0/3368202226 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb7bc053a70 0x7fb7bc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:32.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.377+0000 7fb7cf7fe700 1 -- 172.21.15.60:0/3368202226 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb7d406baf0 con 0x7fb7e00b48b0 2022-06-29T15:58:32.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.377+0000 7fb7de59c700 1 --2- 172.21.15.60:0/3368202226 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb7bc053a70 0x7fb7bc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:32.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.377+0000 7fb7de59c700 1 --2- 172.21.15.60:0/3368202226 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb7bc053a70 0x7fb7bc055f30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fb7d000d000 tx=0x7fb7d00078a0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:32.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.385+0000 7fb7cf7fe700 1 -- 172.21.15.60:0/3368202226 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb7d4042900 con 0x7fb7e00b48b0 2022-06-29T15:58:32.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.513+0000 7fb7df59e700 1 -- 172.21.15.60:0/3368202226 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fb7e0112990 con 0x7fb7e00b48b0 2022-06-29T15:58:32.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.513+0000 7fb7cf7fe700 1 -- 172.21.15.60:0/3368202226 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb7d4045f20 con 0x7fb7e00b48b0 2022-06-29T15:58:32.521 INFO:teuthology.orchestra.run.smithi060.stdout:55834574855 2022-06-29T15:58:32.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.517+0000 7fb7df59e700 1 -- 172.21.15.60:0/3368202226 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb7bc053a70 msgr2=0x7fb7bc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:32.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.517+0000 7fb7df59e700 1 --2- 172.21.15.60:0/3368202226 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb7bc053a70 0x7fb7bc055f30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fb7d000d000 tx=0x7fb7d00078a0).stop 2022-06-29T15:58:32.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.517+0000 7fb7df59e700 1 -- 172.21.15.60:0/3368202226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7e00b48b0 msgr2=0x7fb7e01166f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:32.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.517+0000 7fb7df59e700 1 --2- 172.21.15.60:0/3368202226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7e00b48b0 0x7fb7e01166f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fb7e0062fa0 tx=0x7fb7d4007be0).stop 2022-06-29T15:58:32.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.517+0000 7fb7df59e700 1 -- 172.21.15.60:0/3368202226 shutdown_connections 2022-06-29T15:58:32.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.517+0000 7fb7df59e700 1 --2- 172.21.15.60:0/3368202226 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb7bc053a70 0x7fb7bc055f30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:32.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.517+0000 7fb7df59e700 1 --2- 172.21.15.60:0/3368202226 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb7e00b5eb0 0x7fb7e0151f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:32.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.517+0000 7fb7df59e700 1 --2- 172.21.15.60:0/3368202226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7e00b48b0 0x7fb7e01166f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:32.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.517+0000 7fb7df59e700 1 --2- 172.21.15.60:0/3368202226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7e00b3360 0x7fb7e0116200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:32.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.517+0000 7fb7df59e700 1 -- 172.21.15.60:0/3368202226 >> 172.21.15.60:0/3368202226 conn(0x7fb7e0062090 msgr2=0x7fb7e01410a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:32.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.517+0000 7fb7df59e700 1 -- 172.21.15.60:0/3368202226 shutdown_connections 2022-06-29T15:58:32.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.517+0000 7fb7df59e700 1 -- 172.21.15.60:0/3368202226 wait complete. 2022-06-29T15:58:32.534 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.6 2022-06-29T15:58:32.534 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2022-06-29T15:58:32.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.637+0000 7f9a6712a700 1 Processor -- start 2022-06-29T15:58:32.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.637+0000 7f9a6712a700 1 -- start start 2022-06-29T15:58:32.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.637+0000 7f9a6712a700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9a6005fb00 0x7f9a600aea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:32.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.637+0000 7f9a6712a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a600aef40 0x7f9a600b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:32.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.637+0000 7f9a6712a700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a6005fee0 con 0x7f9a600b1650 2022-06-29T15:58:32.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.637+0000 7f9a6712a700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a600b0870 con 0x7f9a6005fb00 2022-06-29T15:58:32.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.637+0000 7f9a6712a700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a600b0b10 con 0x7f9a600aef40 2022-06-29T15:58:32.646 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.637+0000 7f9a66929700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9a600b1650 0x7f9a6014aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53434/0 (socket says 172.21.15.60:53434) 2022-06-29T15:58:32.646 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.637+0000 7f9a66929700 1 -- 172.21.15.60:0/3674052274 learned_addr learned my addr 172.21.15.60:0/3674052274 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:32.646 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.637+0000 7f9a65927700 1 --2- 172.21.15.60:0/3674052274 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a600aef40 0x7f9a600b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:32.646 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.637+0000 7f9a66128700 1 --2- 172.21.15.60:0/3674052274 >> v2:172.21.15.92:3300/0 conn(0x7f9a6005fb00 0x7f9a600aea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:32.647 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.637+0000 7f9a65126700 1 -- 172.21.15.60:0/3674052274 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362165030 0 0) 0x7f9a6005fee0 con 0x7f9a600b1650 2022-06-29T15:58:32.647 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.637+0000 7f9a65126700 1 -- 172.21.15.60:0/3674052274 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a4c003260 con 0x7f9a600b1650 2022-06-29T15:58:32.647 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.637+0000 7f9a66128700 1 -- 172.21.15.60:0/3674052274 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a600aef40 msgr2=0x7f9a600b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:32.647 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.637+0000 7f9a66128700 1 --2- 172.21.15.60:0/3674052274 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a600aef40 0x7f9a600b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:32.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.637+0000 7f9a66128700 1 -- 172.21.15.60:0/3674052274 >> v1:172.21.15.60:6789/0 conn(0x7f9a600b1650 legacy=0x7f9a6014aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:32.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.637+0000 7f9a66128700 1 -- 172.21.15.60:0/3674052274 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a60060b20 con 0x7f9a6005fb00 2022-06-29T15:58:32.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.637+0000 7f9a65927700 1 --2- 172.21.15.60:0/3674052274 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a600aef40 0x7f9a600b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:32.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.637+0000 7f9a66128700 1 --2- 172.21.15.60:0/3674052274 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a6005fb00 0x7f9a600aea50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f9a5c002f70 tx=0x7f9a5c006df0).ready entity=mon.1 client_cookie=64a5bebd6885891d server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:32.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.637+0000 7f9a65126700 1 -- 172.21.15.60:0/3674052274 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a5c00d4a0 con 0x7f9a6005fb00 2022-06-29T15:58:32.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.637+0000 7f9a65126700 1 -- 172.21.15.60:0/3674052274 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a5c000e90 con 0x7f9a6005fb00 2022-06-29T15:58:32.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.637+0000 7f9a65126700 1 -- 172.21.15.60:0/3674052274 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a5c00d7f0 con 0x7f9a6005fb00 2022-06-29T15:58:32.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.641+0000 7f9a6712a700 1 -- 172.21.15.60:0/3674052274 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a6005fb00 msgr2=0x7f9a600aea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:32.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.641+0000 7f9a6712a700 1 --2- 172.21.15.60:0/3674052274 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a6005fb00 0x7f9a600aea50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f9a5c002f70 tx=0x7f9a5c006df0).stop 2022-06-29T15:58:32.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.641+0000 7f9a6712a700 1 -- 172.21.15.60:0/3674052274 shutdown_connections 2022-06-29T15:58:32.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.641+0000 7f9a6712a700 1 --2- 172.21.15.60:0/3674052274 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a600aef40 0x7f9a600b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:32.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.641+0000 7f9a6712a700 1 --2- 172.21.15.60:0/3674052274 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a6005fb00 0x7f9a600aea50 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:32.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.641+0000 7f9a6712a700 1 -- 172.21.15.60:0/3674052274 >> 172.21.15.60:0/3674052274 conn(0x7f9a60063a60 msgr2=0x7f9a6005f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:32.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.641+0000 7f9a6712a700 1 -- 172.21.15.60:0/3674052274 shutdown_connections 2022-06-29T15:58:32.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.641+0000 7f9a6712a700 1 -- 172.21.15.60:0/3674052274 wait complete. 2022-06-29T15:58:32.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.641+0000 7f9a6712a700 1 Processor -- start 2022-06-29T15:58:32.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.641+0000 7f9a6712a700 1 -- start start 2022-06-29T15:58:32.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.645+0000 7f9a6712a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a6005fb00 0x7f9a6014ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:32.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.645+0000 7f9a6712a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a600aef40 0x7f9a60159650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:32.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.645+0000 7f9a6712a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a600b1650 0x7f9a60065910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:32.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.645+0000 7f9a6712a700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a60063f10 con 0x7f9a600aef40 2022-06-29T15:58:32.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.645+0000 7f9a6712a700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6014d5d0 con 0x7f9a6005fb00 2022-06-29T15:58:32.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.645+0000 7f9a6712a700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a60064140 con 0x7f9a600b1650 2022-06-29T15:58:32.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.645+0000 7f9a65927700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a600aef40 0x7f9a60159650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:32.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.645+0000 7f9a65927700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a600aef40 0x7f9a60159650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45352/0 (socket says 172.21.15.60:45352) 2022-06-29T15:58:32.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.645+0000 7f9a65927700 1 -- 172.21.15.60:0/2078607932 learned_addr learned my addr 172.21.15.60:0/2078607932 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:32.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.645+0000 7f9a66929700 1 --2- 172.21.15.60:0/2078607932 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a600b1650 0x7f9a60065910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:32.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.645+0000 7f9a66128700 1 --2- 172.21.15.60:0/2078607932 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a6005fb00 0x7f9a6014ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:32.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.645+0000 7f9a65927700 1 -- 172.21.15.60:0/2078607932 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a600b1650 msgr2=0x7f9a60065910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:32.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.645+0000 7f9a65927700 1 --2- 172.21.15.60:0/2078607932 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a600b1650 0x7f9a60065910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:32.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.645+0000 7f9a65927700 1 -- 172.21.15.60:0/2078607932 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a6005fb00 msgr2=0x7f9a6014ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:32.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.645+0000 7f9a65927700 1 --2- 172.21.15.60:0/2078607932 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a6005fb00 0x7f9a6014ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:32.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.645+0000 7f9a65927700 1 -- 172.21.15.60:0/2078607932 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a58009470 con 0x7f9a600aef40 2022-06-29T15:58:32.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.645+0000 7f9a65927700 1 --2- 172.21.15.60:0/2078607932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a600aef40 0x7f9a60159650 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f9a5800c000 tx=0x7f9a580091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:32.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.645+0000 7f9a577fe700 1 -- 172.21.15.60:0/2078607932 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a5800c850 con 0x7f9a600aef40 2022-06-29T15:58:32.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.645+0000 7f9a6712a700 1 -- 172.21.15.60:0/2078607932 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a5c00b180 con 0x7f9a600aef40 2022-06-29T15:58:32.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.645+0000 7f9a577fe700 1 -- 172.21.15.60:0/2078607932 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a5800c9b0 con 0x7f9a600aef40 2022-06-29T15:58:32.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.645+0000 7f9a6712a700 1 -- 172.21.15.60:0/2078607932 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a6010f650 con 0x7f9a600aef40 2022-06-29T15:58:32.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.645+0000 7f9a577fe700 1 -- 172.21.15.60:0/2078607932 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a5800cd10 con 0x7f9a600aef40 2022-06-29T15:58:32.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.645+0000 7f9a6712a700 1 -- 172.21.15.60:0/2078607932 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a600573d0 con 0x7f9a600aef40 2022-06-29T15:58:32.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.645+0000 7f9a577fe700 1 -- 172.21.15.60:0/2078607932 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9a58022070 con 0x7f9a600aef40 2022-06-29T15:58:32.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.649+0000 7f9a577fe700 1 --2- 172.21.15.60:0/2078607932 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9a40053b00 0x7f9a40055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:32.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.649+0000 7f9a577fe700 1 -- 172.21.15.60:0/2078607932 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9a5806c690 con 0x7f9a600aef40 2022-06-29T15:58:32.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.649+0000 7f9a66128700 1 --2- 172.21.15.60:0/2078607932 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9a40053b00 0x7f9a40055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:32.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.649+0000 7f9a66128700 1 --2- 172.21.15.60:0/2078607932 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9a40053b00 0x7f9a40055fc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f9a5c01a000 tx=0x7f9a5c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:32.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.653+0000 7f9a577fe700 1 -- 172.21.15.60:0/2078607932 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9a58043510 con 0x7f9a600aef40 2022-06-29T15:58:32.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.781+0000 7f9a6712a700 1 -- 172.21.15.60:0/2078607932 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f9a60064420 con 0x7f9a600aef40 2022-06-29T15:58:32.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.781+0000 7f9a577fe700 1 -- 172.21.15.60:0/2078607932 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9a58016090 con 0x7f9a600aef40 2022-06-29T15:58:32.787 INFO:teuthology.orchestra.run.smithi060.stdout:55834574854 2022-06-29T15:58:32.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.781+0000 7f9a6712a700 1 -- 172.21.15.60:0/2078607932 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9a40053b00 msgr2=0x7f9a40055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:32.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.781+0000 7f9a6712a700 1 --2- 172.21.15.60:0/2078607932 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9a40053b00 0x7f9a40055fc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f9a5c01a000 tx=0x7f9a5c014000).stop 2022-06-29T15:58:32.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.781+0000 7f9a6712a700 1 -- 172.21.15.60:0/2078607932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a600aef40 msgr2=0x7f9a60159650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:32.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.781+0000 7f9a6712a700 1 --2- 172.21.15.60:0/2078607932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a600aef40 0x7f9a60159650 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f9a5800c000 tx=0x7f9a580091d0).stop 2022-06-29T15:58:32.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.781+0000 7f9a6712a700 1 -- 172.21.15.60:0/2078607932 shutdown_connections 2022-06-29T15:58:32.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.781+0000 7f9a6712a700 1 --2- 172.21.15.60:0/2078607932 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9a40053b00 0x7f9a40055fc0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:32.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.781+0000 7f9a6712a700 1 --2- 172.21.15.60:0/2078607932 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a600b1650 0x7f9a60065910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:32.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.781+0000 7f9a6712a700 1 --2- 172.21.15.60:0/2078607932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a600aef40 0x7f9a60159650 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:32.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.781+0000 7f9a6712a700 1 --2- 172.21.15.60:0/2078607932 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a6005fb00 0x7f9a6014ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:32.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.781+0000 7f9a6712a700 1 -- 172.21.15.60:0/2078607932 >> 172.21.15.60:0/2078607932 conn(0x7f9a60063a60 msgr2=0x7f9a600b4d20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:32.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.781+0000 7f9a6712a700 1 -- 172.21.15.60:0/2078607932 shutdown_connections 2022-06-29T15:58:32.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.781+0000 7f9a6712a700 1 -- 172.21.15.60:0/2078607932 wait complete. 2022-06-29T15:58:32.800 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.7 2022-06-29T15:58:32.801 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2022-06-29T15:58:32.801 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-06-29T15:58:32.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.901+0000 7f2d42a75700 1 Processor -- start 2022-06-29T15:58:32.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.901+0000 7f2d42a75700 1 -- start start 2022-06-29T15:58:32.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.901+0000 7f2d42a75700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d3c066d20 0x7f2d3c140810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:32.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.901+0000 7f2d42a75700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2d3c144ad0 0x7f2d3c13d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:32.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.901+0000 7f2d42a75700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d3c0b23a0 con 0x7f2d3c0b3110 2022-06-29T15:58:32.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.901+0000 7f2d42a75700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d3c067100 con 0x7f2d3c144ad0 2022-06-29T15:58:32.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.901+0000 7f2d42a75700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3c140d00 con 0x7f2d3c066d20 2022-06-29T15:58:32.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.901+0000 7f2d42274700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2d3c144ad0 0x7f2d3c13d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:32.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.901+0000 7f2d41a73700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d3c066d20 0x7f2d3c140810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:32.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.901+0000 7f2d41272700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f2d3c0b3110 0x7f2d3c1443c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:57692/0 (socket says 172.21.15.60:57692) 2022-06-29T15:58:32.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.901+0000 7f2d41272700 1 -- 172.21.15.60:0/2022256555 learned_addr learned my addr 172.21.15.60:0/2022256555 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:32.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.905+0000 7f2d42274700 1 -- 172.21.15.60:0/2022256555 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d3c066d20 msgr2=0x7f2d3c140810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:32.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.905+0000 7f2d42274700 1 --2- 172.21.15.60:0/2022256555 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d3c066d20 0x7f2d3c140810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:32.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.905+0000 7f2d42274700 1 -- 172.21.15.60:0/2022256555 >> v1:172.21.15.92:6789/0 conn(0x7f2d3c0b3110 legacy=0x7f2d3c1443c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:32.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.905+0000 7f2d42274700 1 -- 172.21.15.60:0/2022256555 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d3c0b2100 con 0x7f2d3c144ad0 2022-06-29T15:58:32.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.905+0000 7f2d41a73700 1 --2- 172.21.15.60:0/2022256555 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d3c066d20 0x7f2d3c140810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:32.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.905+0000 7f2d42274700 1 --2- 172.21.15.60:0/2022256555 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d3c144ad0 0x7f2d3c13d8b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f2d340032f0 tx=0x7f2d340091d0).ready entity=mon.1 client_cookie=bda049e3d627d79b server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:32.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.905+0000 7f2d40a71700 1 -- 172.21.15.60:0/2022256555 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d3400b6c0 con 0x7f2d3c144ad0 2022-06-29T15:58:32.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.905+0000 7f2d40a71700 1 -- 172.21.15.60:0/2022256555 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d3400b820 con 0x7f2d3c144ad0 2022-06-29T15:58:32.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.905+0000 7f2d40a71700 1 -- 172.21.15.60:0/2022256555 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d3400b990 con 0x7f2d3c144ad0 2022-06-29T15:58:32.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.905+0000 7f2d42a75700 1 -- 172.21.15.60:0/2022256555 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d3c144ad0 msgr2=0x7f2d3c13d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:32.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.905+0000 7f2d42a75700 1 --2- 172.21.15.60:0/2022256555 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d3c144ad0 0x7f2d3c13d8b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f2d340032f0 tx=0x7f2d340091d0).stop 2022-06-29T15:58:32.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.905+0000 7f2d42a75700 1 -- 172.21.15.60:0/2022256555 shutdown_connections 2022-06-29T15:58:32.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.905+0000 7f2d42a75700 1 --2- 172.21.15.60:0/2022256555 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d3c144ad0 0x7f2d3c13d8b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:32.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.905+0000 7f2d42a75700 1 --2- 172.21.15.60:0/2022256555 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d3c066d20 0x7f2d3c140810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:32.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.905+0000 7f2d42a75700 1 -- 172.21.15.60:0/2022256555 >> 172.21.15.60:0/2022256555 conn(0x7f2d3c061c80 msgr2=0x7f2d3c0637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:32.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.905+0000 7f2d42a75700 1 -- 172.21.15.60:0/2022256555 shutdown_connections 2022-06-29T15:58:32.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.905+0000 7f2d42a75700 1 -- 172.21.15.60:0/2022256555 wait complete. 2022-06-29T15:58:32.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.905+0000 7f2d42a75700 1 Processor -- start 2022-06-29T15:58:32.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.905+0000 7f2d42a75700 1 -- start start 2022-06-29T15:58:32.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.909+0000 7f2d42a75700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d3c066d20 0x7f2d3c14fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:32.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.909+0000 7f2d42a75700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d3c0b3110 0x7f2d3c13fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:32.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.909+0000 7f2d42a75700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d3c144ad0 0x7f2d3c13dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:32.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.909+0000 7f2d42a75700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3c150b90 con 0x7f2d3c144ad0 2022-06-29T15:58:32.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.909+0000 7f2d42a75700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3c13efa0 con 0x7f2d3c066d20 2022-06-29T15:58:32.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.909+0000 7f2d42a75700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3c13f240 con 0x7f2d3c0b3110 2022-06-29T15:58:32.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.909+0000 7f2d42274700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d3c144ad0 0x7f2d3c13dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:32.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.909+0000 7f2d42274700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d3c144ad0 0x7f2d3c13dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45368/0 (socket says 172.21.15.60:45368) 2022-06-29T15:58:32.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.909+0000 7f2d42274700 1 -- 172.21.15.60:0/2723355011 learned_addr learned my addr 172.21.15.60:0/2723355011 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:32.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.909+0000 7f2d41272700 1 --2- 172.21.15.60:0/2723355011 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d3c0b3110 0x7f2d3c13fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:32.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.909+0000 7f2d41a73700 1 --2- 172.21.15.60:0/2723355011 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d3c066d20 0x7f2d3c14fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:32.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.909+0000 7f2d42274700 1 -- 172.21.15.60:0/2723355011 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d3c0b3110 msgr2=0x7f2d3c13fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:32.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.909+0000 7f2d42274700 1 --2- 172.21.15.60:0/2723355011 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d3c0b3110 0x7f2d3c13fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:32.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.909+0000 7f2d42274700 1 -- 172.21.15.60:0/2723355011 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d3c066d20 msgr2=0x7f2d3c14fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:32.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.909+0000 7f2d42274700 1 --2- 172.21.15.60:0/2723355011 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d3c066d20 0x7f2d3c14fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:32.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.909+0000 7f2d42274700 1 -- 172.21.15.60:0/2723355011 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2c00b180 con 0x7f2d3c144ad0 2022-06-29T15:58:32.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.909+0000 7f2d41a73700 1 --2- 172.21.15.60:0/2723355011 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d3c066d20 0x7f2d3c14fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:58:32.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.909+0000 7f2d42274700 1 --2- 172.21.15.60:0/2723355011 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d3c144ad0 0x7f2d3c13dec0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f2d340032f0 tx=0x7f2d34016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:32.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.909+0000 7f2d2affd700 1 -- 172.21.15.60:0/2723355011 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d34011e20 con 0x7f2d3c144ad0 2022-06-29T15:58:32.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.909+0000 7f2d2affd700 1 -- 172.21.15.60:0/2723355011 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d34021440 con 0x7f2d3c144ad0 2022-06-29T15:58:32.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.909+0000 7f2d2affd700 1 -- 172.21.15.60:0/2723355011 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d340217a0 con 0x7f2d3c144ad0 2022-06-29T15:58:32.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.909+0000 7f2d42a75700 1 -- 172.21.15.60:0/2723355011 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d34009470 con 0x7f2d3c144ad0 2022-06-29T15:58:32.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.909+0000 7f2d42a75700 1 -- 172.21.15.60:0/2723355011 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d3c13f780 con 0x7f2d3c144ad0 2022-06-29T15:58:32.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.909+0000 7f2d42a75700 1 -- 172.21.15.60:0/2723355011 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d3c13f940 con 0x7f2d3c144ad0 2022-06-29T15:58:32.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.909+0000 7f2d2affd700 1 -- 172.21.15.60:0/2723355011 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2d34027020 con 0x7f2d3c144ad0 2022-06-29T15:58:32.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.909+0000 7f2d2affd700 1 --2- 172.21.15.60:0/2723355011 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2d1c053a70 0x7f2d1c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:32.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.913+0000 7f2d2affd700 1 -- 172.21.15.60:0/2723355011 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f2d3406d420 con 0x7f2d3c144ad0 2022-06-29T15:58:32.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.913+0000 7f2d41a73700 1 --2- 172.21.15.60:0/2723355011 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2d1c053a70 0x7f2d1c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:32.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.913+0000 7f2d41a73700 1 --2- 172.21.15.60:0/2723355011 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2d1c053a70 0x7f2d1c055f30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f2d2c00b000 tx=0x7f2d2c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:32.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:32.917+0000 7f2d2affd700 1 -- 172.21.15.60:0/2723355011 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2d340441f0 con 0x7f2d3c144ad0 2022-06-29T15:58:33.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.041+0000 7f2d42a75700 1 -- 172.21.15.60:0/2723355011 --> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2d3c13dc60 con 0x7f2d1c053a70 2022-06-29T15:58:33.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.049+0000 7f2d2affd700 1 -- 172.21.15.60:0/2723355011 <== mgr.4103 v2:172.21.15.92:6832/12872 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+82243 (secure 0 0 0) 0x7f2d2c00bdc0 con 0x7f2d1c053a70 2022-06-29T15:58:33.056 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:58:33.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.049+0000 7f2d42a75700 1 -- 172.21.15.60:0/2723355011 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2d1c053a70 msgr2=0x7f2d1c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:33.058 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.049+0000 7f2d42a75700 1 --2- 172.21.15.60:0/2723355011 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2d1c053a70 0x7f2d1c055f30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f2d2c00b000 tx=0x7f2d2c008040).stop 2022-06-29T15:58:33.058 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.049+0000 7f2d42a75700 1 -- 172.21.15.60:0/2723355011 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d3c144ad0 msgr2=0x7f2d3c13dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:33.058 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.049+0000 7f2d42a75700 1 --2- 172.21.15.60:0/2723355011 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d3c144ad0 0x7f2d3c13dec0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f2d340032f0 tx=0x7f2d34016000).stop 2022-06-29T15:58:33.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.049+0000 7f2d42a75700 1 -- 172.21.15.60:0/2723355011 shutdown_connections 2022-06-29T15:58:33.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.049+0000 7f2d42a75700 1 --2- 172.21.15.60:0/2723355011 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2d1c053a70 0x7f2d1c055f30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:33.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.049+0000 7f2d42a75700 1 --2- 172.21.15.60:0/2723355011 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d3c144ad0 0x7f2d3c13dec0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:33.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.049+0000 7f2d42a75700 1 --2- 172.21.15.60:0/2723355011 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d3c0b3110 0x7f2d3c13fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:33.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.049+0000 7f2d42a75700 1 --2- 172.21.15.60:0/2723355011 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d3c066d20 0x7f2d3c14fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:33.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.049+0000 7f2d42a75700 1 -- 172.21.15.60:0/2723355011 >> 172.21.15.60:0/2723355011 conn(0x7f2d3c061c80 msgr2=0x7f2d3c142e80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:33.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.049+0000 7f2d42a75700 1 -- 172.21.15.60:0/2723355011 shutdown_connections 2022-06-29T15:58:33.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.049+0000 7f2d42a75700 1 -- 172.21.15.60:0/2723355011 wait complete. 2022-06-29T15:58:33.061 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2022-06-29T15:58:33.068 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2022-06-29T15:58:32.922515+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2304,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":50,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":6367304,"kb_used_data":743227840,"kb_used_omap":606,"kb_used_meta":0,"kb_avail":743227736,"statfs":{"total":767585320960,"available":761065201664,"internally_reserved":106496,"allocated":761065308160,"data_stored":761065308160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":620628,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002492"},"pg_stats":[{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.363078+0000","last_change":"2022-06-29T15:58:25.362652+0000","last_active":"2022-06-29T15:58:25.363078+0000","last_peered":"2022-06-29T15:58:25.363078+0000","last_clean":"2022-06-29T15:58:25.363078+0000","last_became_active":"2022-06-29T15:58:25.362256+0000","last_became_peered":"2022-06-29T15:58:25.362256+0000","last_unstale":"2022-06-29T15:58:25.363078+0000","last_undegraded":"2022-06-29T15:58:25.363078+0000","last_fullsized":"2022-06-29T15:58:25.363078+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:24.007232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:24.007232+0000","last_clean_scrub_stamp":"2022-06-29T15:58:24.007232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"14'2","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.356622+0000","last_change":"2022-06-29T15:58:14.849417+0000","last_active":"2022-06-29T15:58:25.356622+0000","last_peered":"2022-06-29T15:58:25.356622+0000","last_clean":"2022-06-29T15:58:25.356622+0000","last_became_active":"2022-06-29T15:58:14.848454+0000","last_became_peered":"2022-06-29T15:58:14.848454+0000","last_unstale":"2022-06-29T15:58:25.356622+0000","last_undegraded":"2022-06-29T15:58:25.356622+0000","last_fullsized":"2022-06-29T15:58:25.356622+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:11.766061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:11.766061+0000","last_clean_scrub_stamp":"2022-06-29T15:58:11.766061+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.592731+0000","last_change":"2022-06-29T15:58:21.003987+0000","last_active":"2022-06-29T15:58:25.592731+0000","last_peered":"2022-06-29T15:58:25.592731+0000","last_clean":"2022-06-29T15:58:25.592731+0000","last_became_active":"2022-06-29T15:58:21.003537+0000","last_became_peered":"2022-06-29T15:58:21.003537+0000","last_unstale":"2022-06-29T15:58:25.592731+0000","last_undegraded":"2022-06-29T15:58:25.592731+0000","last_fullsized":"2022-06-29T15:58:25.592731+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_clean_scrub_stamp":"2022-06-29T15:58:19.966881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"21'1","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.573939+0000","last_change":"2022-06-29T15:58:24.022968+0000","last_active":"2022-06-29T15:58:25.573939+0000","last_peered":"2022-06-29T15:58:25.573939+0000","last_clean":"2022-06-29T15:58:25.573939+0000","last_became_active":"2022-06-29T15:58:24.022660+0000","last_became_peered":"2022-06-29T15:58:24.022660+0000","last_unstale":"2022-06-29T15:58:25.573939+0000","last_undegraded":"2022-06-29T15:58:25.573939+0000","last_fullsized":"2022-06-29T15:58:25.573939+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:22.997265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:22.997265+0000","last_clean_scrub_stamp":"2022-06-29T15:58:22.997265+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.586362+0000","last_change":"2022-06-29T15:58:24.346157+0000","last_active":"2022-06-29T15:58:25.586362+0000","last_peered":"2022-06-29T15:58:25.586362+0000","last_clean":"2022-06-29T15:58:25.586362+0000","last_became_active":"2022-06-29T15:58:24.345683+0000","last_became_peered":"2022-06-29T15:58:24.345683+0000","last_unstale":"2022-06-29T15:58:25.586362+0000","last_undegraded":"2022-06-29T15:58:25.586362+0000","last_fullsized":"2022-06-29T15:58:25.586362+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:24.007232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:24.007232+0000","last_clean_scrub_stamp":"2022-06-29T15:58:24.007232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.356502+0000","last_change":"2022-06-29T15:58:21.001629+0000","last_active":"2022-06-29T15:58:25.356502+0000","last_peered":"2022-06-29T15:58:25.356502+0000","last_clean":"2022-06-29T15:58:25.356502+0000","last_became_active":"2022-06-29T15:58:21.000885+0000","last_became_peered":"2022-06-29T15:58:21.000885+0000","last_unstale":"2022-06-29T15:58:25.356502+0000","last_undegraded":"2022-06-29T15:58:25.356502+0000","last_fullsized":"2022-06-29T15:58:25.356502+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_clean_scrub_stamp":"2022-06-29T15:58:19.966881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.586092+0000","last_change":"2022-06-29T15:58:24.020809+0000","last_active":"2022-06-29T15:58:25.586092+0000","last_peered":"2022-06-29T15:58:25.586092+0000","last_clean":"2022-06-29T15:58:25.586092+0000","last_became_active":"2022-06-29T15:58:24.020202+0000","last_became_peered":"2022-06-29T15:58:24.020202+0000","last_unstale":"2022-06-29T15:58:25.586092+0000","last_undegraded":"2022-06-29T15:58:25.586092+0000","last_fullsized":"2022-06-29T15:58:25.586092+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:22.997265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:22.997265+0000","last_clean_scrub_stamp":"2022-06-29T15:58:22.997265+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.585809+0000","last_change":"2022-06-29T15:58:24.354169+0000","last_active":"2022-06-29T15:58:25.585809+0000","last_peered":"2022-06-29T15:58:25.585809+0000","last_clean":"2022-06-29T15:58:25.585809+0000","last_became_active":"2022-06-29T15:58:24.353817+0000","last_became_peered":"2022-06-29T15:58:24.353817+0000","last_unstale":"2022-06-29T15:58:25.585809+0000","last_undegraded":"2022-06-29T15:58:25.585809+0000","last_fullsized":"2022-06-29T15:58:25.585809+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:24.007232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:24.007232+0000","last_clean_scrub_stamp":"2022-06-29T15:58:24.007232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.348062+0000","last_change":"2022-06-29T15:58:20.997373+0000","last_active":"2022-06-29T15:58:25.348062+0000","last_peered":"2022-06-29T15:58:25.348062+0000","last_clean":"2022-06-29T15:58:25.348062+0000","last_became_active":"2022-06-29T15:58:20.997032+0000","last_became_peered":"2022-06-29T15:58:20.997032+0000","last_unstale":"2022-06-29T15:58:25.348062+0000","last_undegraded":"2022-06-29T15:58:25.348062+0000","last_fullsized":"2022-06-29T15:58:25.348062+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_clean_scrub_stamp":"2022-06-29T15:58:19.966881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"21'8","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.585296+0000","last_change":"2022-06-29T15:58:24.023775+0000","last_active":"2022-06-29T15:58:25.585296+0000","last_peered":"2022-06-29T15:58:25.585296+0000","last_clean":"2022-06-29T15:58:25.585296+0000","last_became_active":"2022-06-29T15:58:24.023397+0000","last_became_peered":"2022-06-29T15:58:24.023397+0000","last_unstale":"2022-06-29T15:58:25.585296+0000","last_undegraded":"2022-06-29T15:58:25.585296+0000","last_fullsized":"2022-06-29T15:58:25.585296+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:22.997265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:22.997265+0000","last_clean_scrub_stamp":"2022-06-29T15:58:22.997265+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.585601+0000","last_change":"2022-06-29T15:58:24.352933+0000","last_active":"2022-06-29T15:58:25.585601+0000","last_peered":"2022-06-29T15:58:25.585601+0000","last_clean":"2022-06-29T15:58:25.585601+0000","last_became_active":"2022-06-29T15:58:24.352641+0000","last_became_peered":"2022-06-29T15:58:24.352641+0000","last_unstale":"2022-06-29T15:58:25.585601+0000","last_undegraded":"2022-06-29T15:58:25.585601+0000","last_fullsized":"2022-06-29T15:58:25.585601+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:24.007232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:24.007232+0000","last_clean_scrub_stamp":"2022-06-29T15:58:24.007232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.592797+0000","last_change":"2022-06-29T15:58:21.005059+0000","last_active":"2022-06-29T15:58:25.592797+0000","last_peered":"2022-06-29T15:58:25.592797+0000","last_clean":"2022-06-29T15:58:25.592797+0000","last_became_active":"2022-06-29T15:58:21.004623+0000","last_became_peered":"2022-06-29T15:58:21.004623+0000","last_unstale":"2022-06-29T15:58:25.592797+0000","last_undegraded":"2022-06-29T15:58:25.592797+0000","last_fullsized":"2022-06-29T15:58:25.592797+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_clean_scrub_stamp":"2022-06-29T15:58:19.966881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"21'5","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.357136+0000","last_change":"2022-06-29T15:58:24.027287+0000","last_active":"2022-06-29T15:58:25.357136+0000","last_peered":"2022-06-29T15:58:25.357136+0000","last_clean":"2022-06-29T15:58:25.357136+0000","last_became_active":"2022-06-29T15:58:24.026492+0000","last_became_peered":"2022-06-29T15:58:24.026492+0000","last_unstale":"2022-06-29T15:58:25.357136+0000","last_undegraded":"2022-06-29T15:58:25.357136+0000","last_fullsized":"2022-06-29T15:58:25.357136+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:22.997265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:22.997265+0000","last_clean_scrub_stamp":"2022-06-29T15:58:22.997265+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.585249+0000","last_change":"2022-06-29T15:58:24.353076+0000","last_active":"2022-06-29T15:58:25.585249+0000","last_peered":"2022-06-29T15:58:25.585249+0000","last_clean":"2022-06-29T15:58:25.585249+0000","last_became_active":"2022-06-29T15:58:24.352673+0000","last_became_peered":"2022-06-29T15:58:24.352673+0000","last_unstale":"2022-06-29T15:58:25.585249+0000","last_undegraded":"2022-06-29T15:58:25.585249+0000","last_fullsized":"2022-06-29T15:58:25.585249+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:24.007232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:24.007232+0000","last_clean_scrub_stamp":"2022-06-29T15:58:24.007232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"21'6","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.592777+0000","last_change":"2022-06-29T15:58:24.028652+0000","last_active":"2022-06-29T15:58:25.592777+0000","last_peered":"2022-06-29T15:58:25.592777+0000","last_clean":"2022-06-29T15:58:25.592777+0000","last_became_active":"2022-06-29T15:58:24.028163+0000","last_became_peered":"2022-06-29T15:58:24.028163+0000","last_unstale":"2022-06-29T15:58:25.592777+0000","last_undegraded":"2022-06-29T15:58:25.592777+0000","last_fullsized":"2022-06-29T15:58:25.592777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:22.997265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:22.997265+0000","last_clean_scrub_stamp":"2022-06-29T15:58:22.997265+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.586172+0000","last_change":"2022-06-29T15:58:21.801445+0000","last_active":"2022-06-29T15:58:25.586172+0000","last_peered":"2022-06-29T15:58:25.586172+0000","last_clean":"2022-06-29T15:58:25.586172+0000","last_became_active":"2022-06-29T15:58:21.800792+0000","last_became_peered":"2022-06-29T15:58:21.800792+0000","last_unstale":"2022-06-29T15:58:25.586172+0000","last_undegraded":"2022-06-29T15:58:25.586172+0000","last_fullsized":"2022-06-29T15:58:25.586172+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_clean_scrub_stamp":"2022-06-29T15:58:19.966881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.586062+0000","last_change":"2022-06-29T15:58:24.351819+0000","last_active":"2022-06-29T15:58:25.586062+0000","last_peered":"2022-06-29T15:58:25.586062+0000","last_clean":"2022-06-29T15:58:25.586062+0000","last_became_active":"2022-06-29T15:58:24.351535+0000","last_became_peered":"2022-06-29T15:58:24.351535+0000","last_unstale":"2022-06-29T15:58:25.586062+0000","last_undegraded":"2022-06-29T15:58:25.586062+0000","last_fullsized":"2022-06-29T15:58:25.586062+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:24.007232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:24.007232+0000","last_clean_scrub_stamp":"2022-06-29T15:58:24.007232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"21'8","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.585896+0000","last_change":"2022-06-29T15:58:24.024424+0000","last_active":"2022-06-29T15:58:25.585896+0000","last_peered":"2022-06-29T15:58:25.585896+0000","last_clean":"2022-06-29T15:58:25.585896+0000","last_became_active":"2022-06-29T15:58:24.024076+0000","last_became_peered":"2022-06-29T15:58:24.024076+0000","last_unstale":"2022-06-29T15:58:25.585896+0000","last_undegraded":"2022-06-29T15:58:25.585896+0000","last_fullsized":"2022-06-29T15:58:25.585896+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:22.997265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:22.997265+0000","last_clean_scrub_stamp":"2022-06-29T15:58:22.997265+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1535,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.356376+0000","last_change":"2022-06-29T15:58:21.807952+0000","last_active":"2022-06-29T15:58:25.356376+0000","last_peered":"2022-06-29T15:58:25.356376+0000","last_clean":"2022-06-29T15:58:25.356376+0000","last_became_active":"2022-06-29T15:58:21.807346+0000","last_became_peered":"2022-06-29T15:58:21.807346+0000","last_unstale":"2022-06-29T15:58:25.356376+0000","last_undegraded":"2022-06-29T15:58:25.356376+0000","last_fullsized":"2022-06-29T15:58:25.356376+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_clean_scrub_stamp":"2022-06-29T15:58:19.966881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.573873+0000","last_change":"2022-06-29T15:58:24.347840+0000","last_active":"2022-06-29T15:58:25.573873+0000","last_peered":"2022-06-29T15:58:25.573873+0000","last_clean":"2022-06-29T15:58:25.573873+0000","last_became_active":"2022-06-29T15:58:24.347424+0000","last_became_peered":"2022-06-29T15:58:24.347424+0000","last_unstale":"2022-06-29T15:58:25.573873+0000","last_undegraded":"2022-06-29T15:58:25.573873+0000","last_fullsized":"2022-06-29T15:58:25.573873+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:24.007232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:24.007232+0000","last_clean_scrub_stamp":"2022-06-29T15:58:24.007232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"21'7","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.573968+0000","last_change":"2022-06-29T15:58:24.021541+0000","last_active":"2022-06-29T15:58:25.573968+0000","last_peered":"2022-06-29T15:58:25.573968+0000","last_clean":"2022-06-29T15:58:25.573968+0000","last_became_active":"2022-06-29T15:58:24.021230+0000","last_became_peered":"2022-06-29T15:58:24.021230+0000","last_unstale":"2022-06-29T15:58:25.573968+0000","last_undegraded":"2022-06-29T15:58:25.573968+0000","last_fullsized":"2022-06-29T15:58:25.573968+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:22.997265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:22.997265+0000","last_clean_scrub_stamp":"2022-06-29T15:58:22.997265+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.585702+0000","last_change":"2022-06-29T15:58:20.996366+0000","last_active":"2022-06-29T15:58:25.585702+0000","last_peered":"2022-06-29T15:58:25.585702+0000","last_clean":"2022-06-29T15:58:25.585702+0000","last_became_active":"2022-06-29T15:58:20.995883+0000","last_became_peered":"2022-06-29T15:58:20.995883+0000","last_unstale":"2022-06-29T15:58:25.585702+0000","last_undegraded":"2022-06-29T15:58:25.585702+0000","last_fullsized":"2022-06-29T15:58:25.585702+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_clean_scrub_stamp":"2022-06-29T15:58:19.966881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.357215+0000","last_change":"2022-06-29T15:58:24.358772+0000","last_active":"2022-06-29T15:58:25.357215+0000","last_peered":"2022-06-29T15:58:25.357215+0000","last_clean":"2022-06-29T15:58:25.357215+0000","last_became_active":"2022-06-29T15:58:24.358128+0000","last_became_peered":"2022-06-29T15:58:24.358128+0000","last_unstale":"2022-06-29T15:58:25.357215+0000","last_undegraded":"2022-06-29T15:58:25.357215+0000","last_fullsized":"2022-06-29T15:58:25.357215+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:24.007232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:24.007232+0000","last_clean_scrub_stamp":"2022-06-29T15:58:24.007232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"22'4","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:26.092891+0000","last_change":"2022-06-29T15:58:24.022399+0000","last_active":"2022-06-29T15:58:26.092891+0000","last_peered":"2022-06-29T15:58:26.092891+0000","last_clean":"2022-06-29T15:58:26.092891+0000","last_became_active":"2022-06-29T15:58:24.022030+0000","last_became_peered":"2022-06-29T15:58:24.022030+0000","last_unstale":"2022-06-29T15:58:26.092891+0000","last_undegraded":"2022-06-29T15:58:26.092891+0000","last_fullsized":"2022-06-29T15:58:26.092891+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:22.997265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:22.997265+0000","last_clean_scrub_stamp":"2022-06-29T15:58:22.997265+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.360692+0000","last_change":"2022-06-29T15:58:21.003020+0000","last_active":"2022-06-29T15:58:25.360692+0000","last_peered":"2022-06-29T15:58:25.360692+0000","last_clean":"2022-06-29T15:58:25.360692+0000","last_became_active":"2022-06-29T15:58:21.002353+0000","last_became_peered":"2022-06-29T15:58:21.002353+0000","last_unstale":"2022-06-29T15:58:25.360692+0000","last_undegraded":"2022-06-29T15:58:25.360692+0000","last_fullsized":"2022-06-29T15:58:25.360692+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_clean_scrub_stamp":"2022-06-29T15:58:19.966881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":2304,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":16,"acting":16,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795912,"kb_used_data":92903480,"kb_used_omap":75,"kb_used_meta":0,"kb_avail":92903468,"statfs":{"total":95948165120,"available":95133151232,"internally_reserved":12288,"allocated":95133163520,"data_stored":95133163520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77622,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92600000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88800000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795904,"kb_used_data":92903488,"kb_used_omap":46,"kb_used_meta":0,"kb_avail":92903476,"statfs":{"total":95948165120,"available":95133159424,"internally_reserved":12288,"allocated":95133171712,"data_stored":95133171712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47791,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92800000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1259999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4319999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3440000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":8,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795912,"kb_used_data":92903480,"kb_used_omap":73,"kb_used_meta":0,"kb_avail":92903468,"statfs":{"total":95948165120,"available":95133151232,"internally_reserved":12288,"allocated":95133163520,"data_stored":95133163520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74884,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.179}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0880000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795908,"kb_used_data":92903484,"kb_used_omap":61,"kb_used_meta":0,"kb_avail":92903472,"statfs":{"total":95948165120,"available":95133155328,"internally_reserved":12288,"allocated":95133167616,"data_stored":95133167616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63068,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0549999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.006}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795920,"kb_used_data":92903472,"kb_used_omap":69,"kb_used_meta":0,"kb_avail":92903460,"statfs":{"total":95948165120,"available":95133143040,"internally_reserved":12288,"allocated":95133155328,"data_stored":95133155328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71238,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0620000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2490000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.117}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.163}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795916,"kb_used_data":92903476,"kb_used_omap":73,"kb_used_meta":0,"kb_avail":92903464,"statfs":{"total":95948165120,"available":95133147136,"internally_reserved":12288,"allocated":95133159424,"data_stored":95133159424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74833,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.679}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.506}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4530000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.361}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6120000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795920,"kb_used_data":92903476,"kb_used_omap":107,"kb_used_meta":0,"kb_avail":92903460,"statfs":{"total":95948165120,"available":95133143040,"internally_reserved":16384,"allocated":95133159424,"data_stored":95133159424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":109785,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.752}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94499999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96799999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.02}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":8,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795912,"kb_used_data":92903484,"kb_used_omap":99,"kb_used_meta":0,"kb_avail":92903468,"statfs":{"total":95948165120,"available":95133151232,"internally_reserved":16384,"allocated":95133167616,"data_stored":95133167616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101407,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1279999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]}]}]}} 2022-06-29T15:58:33.071 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-06-29T15:58:33.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.169+0000 7f8a2e717700 1 Processor -- start 2022-06-29T15:58:33.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.169+0000 7f8a2e717700 1 -- start start 2022-06-29T15:58:33.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.169+0000 7f8a2e717700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8a280b1c00 0x7f8a280ae890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:33.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.169+0000 7f8a2e717700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8a280b0fa0 0x7f8a28067250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:33.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.169+0000 7f8a2e717700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a280ad240 con 0x7f8a2805fb00 2022-06-29T15:58:33.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.169+0000 7f8a2e717700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a2813ce60 con 0x7f8a280b0fa0 2022-06-29T15:58:33.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.169+0000 7f8a2e717700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2805fee0 con 0x7f8a280b1c00 2022-06-29T15:58:33.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.169+0000 7f8a2cf14700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8a280b1c00 0x7f8a280ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:33.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.169+0000 7f8a2cf14700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8a280b1c00 0x7f8a280ae890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:58468/0 (socket says 172.21.15.60:58468) 2022-06-29T15:58:33.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.169+0000 7f8a2cf14700 1 -- 172.21.15.60:0/3356797548 learned_addr learned my addr 172.21.15.60:0/3356797548 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:33.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.169+0000 7f8a2df16700 1 --2- 172.21.15.60:0/3356797548 >> v2:172.21.15.92:3300/0 conn(0x7f8a280b0fa0 0x7f8a28067250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:33.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.169+0000 7f8a2cf14700 1 -- 172.21.15.60:0/3356797548 >> v2:172.21.15.92:3300/0 conn(0x7f8a280b0fa0 msgr2=0x7f8a28067250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:33.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.169+0000 7f8a2cf14700 1 --2- 172.21.15.60:0/3356797548 >> v2:172.21.15.92:3300/0 conn(0x7f8a280b0fa0 0x7f8a28067250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:33.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.169+0000 7f8a1ffff700 1 -- 172.21.15.60:0/3356797548 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4018714104 0 0) 0x7f8a280ad240 con 0x7f8a2805fb00 2022-06-29T15:58:33.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.169+0000 7f8a2cf14700 1 -- 172.21.15.60:0/3356797548 >> v1:172.21.15.92:6789/0 conn(0x7f8a2805fb00 legacy=0x7f8a280b14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:33.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.169+0000 7f8a2cf14700 1 -- 172.21.15.60:0/3356797548 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a28143180 con 0x7f8a280b1c00 2022-06-29T15:58:33.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.169+0000 7f8a2cf14700 1 --2- 172.21.15.60:0/3356797548 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8a280b1c00 0x7f8a280ae890 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f8a18002f70 tx=0x7f8a18006f40).ready entity=mon.2 client_cookie=176de3948139201c server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:33.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.169+0000 7f8a1ffff700 1 -- 172.21.15.60:0/3356797548 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a1800da80 con 0x7f8a280b1c00 2022-06-29T15:58:33.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.169+0000 7f8a1ffff700 1 -- 172.21.15.60:0/3356797548 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a18000e90 con 0x7f8a280b1c00 2022-06-29T15:58:33.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.169+0000 7f8a1ffff700 1 -- 172.21.15.60:0/3356797548 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a1800ddd0 con 0x7f8a280b1c00 2022-06-29T15:58:33.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.169+0000 7f8a2e717700 1 -- 172.21.15.60:0/3356797548 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8a280b1c00 msgr2=0x7f8a280ae890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:33.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.169+0000 7f8a2e717700 1 --2- 172.21.15.60:0/3356797548 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8a280b1c00 0x7f8a280ae890 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f8a18002f70 tx=0x7f8a18006f40).stop 2022-06-29T15:58:33.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.169+0000 7f8a2e717700 1 -- 172.21.15.60:0/3356797548 shutdown_connections 2022-06-29T15:58:33.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.169+0000 7f8a2e717700 1 --2- 172.21.15.60:0/3356797548 >> v2:172.21.15.92:3300/0 conn(0x7f8a280b0fa0 0x7f8a28067250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:33.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.169+0000 7f8a2e717700 1 --2- 172.21.15.60:0/3356797548 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8a280b1c00 0x7f8a280ae890 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:33.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.169+0000 7f8a2e717700 1 -- 172.21.15.60:0/3356797548 >> 172.21.15.60:0/3356797548 conn(0x7f8a28063a60 msgr2=0x7f8a2805f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:33.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.169+0000 7f8a2e717700 1 -- 172.21.15.60:0/3356797548 shutdown_connections 2022-06-29T15:58:33.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.169+0000 7f8a2e717700 1 -- 172.21.15.60:0/3356797548 wait complete. 2022-06-29T15:58:33.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.173+0000 7f8a2e717700 1 Processor -- start 2022-06-29T15:58:33.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.173+0000 7f8a2e717700 1 -- start start 2022-06-29T15:58:33.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.173+0000 7f8a2e717700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8a2805fb00 0x7f8a28152190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:33.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.173+0000 7f8a2e717700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a280b0fa0 0x7f8a2813f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:33.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.173+0000 7f8a2e717700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a280b1c00 0x7f8a2813d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:33.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.173+0000 7f8a2e717700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a28146250 con 0x7f8a280b0fa0 2022-06-29T15:58:33.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.173+0000 7f8a2e717700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2813eb40 con 0x7f8a280b1c00 2022-06-29T15:58:33.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.173+0000 7f8a2e717700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2813ede0 con 0x7f8a2805fb00 2022-06-29T15:58:33.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.173+0000 7f8a2cf14700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a280b0fa0 0x7f8a2813f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:33.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.173+0000 7f8a2cf14700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a280b0fa0 0x7f8a2813f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45380/0 (socket says 172.21.15.60:45380) 2022-06-29T15:58:33.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.173+0000 7f8a2cf14700 1 -- 172.21.15.60:0/2671435929 learned_addr learned my addr 172.21.15.60:0/2671435929 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:33.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.173+0000 7f8a2d715700 1 --2- 172.21.15.60:0/2671435929 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8a2805fb00 0x7f8a28152190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:33.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.173+0000 7f8a2df16700 1 --2- 172.21.15.60:0/2671435929 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a280b1c00 0x7f8a2813d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:33.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.173+0000 7f8a2cf14700 1 -- 172.21.15.60:0/2671435929 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8a2805fb00 msgr2=0x7f8a28152190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:33.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.173+0000 7f8a2cf14700 1 --2- 172.21.15.60:0/2671435929 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8a2805fb00 0x7f8a28152190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:33.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.173+0000 7f8a2cf14700 1 -- 172.21.15.60:0/2671435929 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a280b1c00 msgr2=0x7f8a2813d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:33.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.173+0000 7f8a2cf14700 1 --2- 172.21.15.60:0/2671435929 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a280b1c00 0x7f8a2813d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:33.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.173+0000 7f8a2cf14700 1 -- 172.21.15.60:0/2671435929 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1800b460 con 0x7f8a280b0fa0 2022-06-29T15:58:33.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.173+0000 7f8a2d715700 1 --2- 172.21.15.60:0/2671435929 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8a2805fb00 0x7f8a28152190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:58:33.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.173+0000 7f8a2cf14700 1 --2- 172.21.15.60:0/2671435929 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a280b0fa0 0x7f8a2813f520 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f8a18002f70 tx=0x7f8a280ad130).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:33.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.173+0000 7f8a1e7fc700 1 -- 172.21.15.60:0/2671435929 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a1801d070 con 0x7f8a280b0fa0 2022-06-29T15:58:33.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.177+0000 7f8a1e7fc700 1 -- 172.21.15.60:0/2671435929 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a18018800 con 0x7f8a280b0fa0 2022-06-29T15:58:33.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.177+0000 7f8a2e717700 1 -- 172.21.15.60:0/2671435929 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a2813f080 con 0x7f8a280b0fa0 2022-06-29T15:58:33.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.177+0000 7f8a1e7fc700 1 -- 172.21.15.60:0/2671435929 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a18018b60 con 0x7f8a280b0fa0 2022-06-29T15:58:33.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.177+0000 7f8a2e717700 1 -- 172.21.15.60:0/2671435929 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a2813d970 con 0x7f8a280b0fa0 2022-06-29T15:58:33.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.177+0000 7f8a2e717700 1 -- 172.21.15.60:0/2671435929 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a280573d0 con 0x7f8a280b0fa0 2022-06-29T15:58:33.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.177+0000 7f8a1e7fc700 1 -- 172.21.15.60:0/2671435929 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8a18029070 con 0x7f8a280b0fa0 2022-06-29T15:58:33.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.177+0000 7f8a1e7fc700 1 --2- 172.21.15.60:0/2671435929 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8a100539e0 0x7f8a10055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:33.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.177+0000 7f8a1e7fc700 1 -- 172.21.15.60:0/2671435929 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8a1806d290 con 0x7f8a280b0fa0 2022-06-29T15:58:33.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.177+0000 7f8a2d715700 1 --2- 172.21.15.60:0/2671435929 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8a100539e0 0x7f8a10055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:33.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.181+0000 7f8a2d715700 1 --2- 172.21.15.60:0/2671435929 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8a100539e0 0x7f8a10055ea0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f8a2400b000 tx=0x7f8a24009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:33.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.185+0000 7f8a1e7fc700 1 -- 172.21.15.60:0/2671435929 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8a180440e0 con 0x7f8a280b0fa0 2022-06-29T15:58:33.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.305+0000 7f8a2e717700 1 -- 172.21.15.60:0/2671435929 --> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8a2813e900 con 0x7f8a100539e0 2022-06-29T15:58:33.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.313+0000 7f8a1e7fc700 1 -- 172.21.15.60:0/2671435929 <== mgr.4103 v2:172.21.15.92:6832/12872 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+82243 (secure 0 0 0) 0x7f8a2813e900 con 0x7f8a100539e0 2022-06-29T15:58:33.321 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:58:33.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.317+0000 7f8a2e717700 1 -- 172.21.15.60:0/2671435929 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8a100539e0 msgr2=0x7f8a10055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:33.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.317+0000 7f8a2e717700 1 --2- 172.21.15.60:0/2671435929 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8a100539e0 0x7f8a10055ea0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f8a2400b000 tx=0x7f8a24009040).stop 2022-06-29T15:58:33.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.317+0000 7f8a2e717700 1 -- 172.21.15.60:0/2671435929 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a280b0fa0 msgr2=0x7f8a2813f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:33.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.317+0000 7f8a2e717700 1 --2- 172.21.15.60:0/2671435929 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a280b0fa0 0x7f8a2813f520 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f8a18002f70 tx=0x7f8a280ad130).stop 2022-06-29T15:58:33.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.317+0000 7f8a2e717700 1 -- 172.21.15.60:0/2671435929 shutdown_connections 2022-06-29T15:58:33.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.317+0000 7f8a2e717700 1 --2- 172.21.15.60:0/2671435929 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8a100539e0 0x7f8a10055ea0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:33.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.317+0000 7f8a2e717700 1 --2- 172.21.15.60:0/2671435929 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a280b1c00 0x7f8a2813d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:33.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.317+0000 7f8a2e717700 1 --2- 172.21.15.60:0/2671435929 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a280b0fa0 0x7f8a2813f520 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:33.326 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.317+0000 7f8a2e717700 1 --2- 172.21.15.60:0/2671435929 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8a2805fb00 0x7f8a28152190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:33.326 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.317+0000 7f8a2e717700 1 -- 172.21.15.60:0/2671435929 >> 172.21.15.60:0/2671435929 conn(0x7f8a28063a60 msgr2=0x7f8a280b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:33.326 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.317+0000 7f8a2e717700 1 -- 172.21.15.60:0/2671435929 shutdown_connections 2022-06-29T15:58:33.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.317+0000 7f8a2e717700 1 -- 172.21.15.60:0/2671435929 wait complete. 2022-06-29T15:58:33.327 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2022-06-29T15:58:33.335 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2022-06-29T15:58:32.922515+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2304,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":50,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":6367304,"kb_used_data":743227840,"kb_used_omap":606,"kb_used_meta":0,"kb_avail":743227736,"statfs":{"total":767585320960,"available":761065201664,"internally_reserved":106496,"allocated":761065308160,"data_stored":761065308160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":620628,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002492"},"pg_stats":[{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.363078+0000","last_change":"2022-06-29T15:58:25.362652+0000","last_active":"2022-06-29T15:58:25.363078+0000","last_peered":"2022-06-29T15:58:25.363078+0000","last_clean":"2022-06-29T15:58:25.363078+0000","last_became_active":"2022-06-29T15:58:25.362256+0000","last_became_peered":"2022-06-29T15:58:25.362256+0000","last_unstale":"2022-06-29T15:58:25.363078+0000","last_undegraded":"2022-06-29T15:58:25.363078+0000","last_fullsized":"2022-06-29T15:58:25.363078+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:24.007232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:24.007232+0000","last_clean_scrub_stamp":"2022-06-29T15:58:24.007232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"14'2","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.356622+0000","last_change":"2022-06-29T15:58:14.849417+0000","last_active":"2022-06-29T15:58:25.356622+0000","last_peered":"2022-06-29T15:58:25.356622+0000","last_clean":"2022-06-29T15:58:25.356622+0000","last_became_active":"2022-06-29T15:58:14.848454+0000","last_became_peered":"2022-06-29T15:58:14.848454+0000","last_unstale":"2022-06-29T15:58:25.356622+0000","last_undegraded":"2022-06-29T15:58:25.356622+0000","last_fullsized":"2022-06-29T15:58:25.356622+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:11.766061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:11.766061+0000","last_clean_scrub_stamp":"2022-06-29T15:58:11.766061+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.592731+0000","last_change":"2022-06-29T15:58:21.003987+0000","last_active":"2022-06-29T15:58:25.592731+0000","last_peered":"2022-06-29T15:58:25.592731+0000","last_clean":"2022-06-29T15:58:25.592731+0000","last_became_active":"2022-06-29T15:58:21.003537+0000","last_became_peered":"2022-06-29T15:58:21.003537+0000","last_unstale":"2022-06-29T15:58:25.592731+0000","last_undegraded":"2022-06-29T15:58:25.592731+0000","last_fullsized":"2022-06-29T15:58:25.592731+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_clean_scrub_stamp":"2022-06-29T15:58:19.966881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"21'1","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.573939+0000","last_change":"2022-06-29T15:58:24.022968+0000","last_active":"2022-06-29T15:58:25.573939+0000","last_peered":"2022-06-29T15:58:25.573939+0000","last_clean":"2022-06-29T15:58:25.573939+0000","last_became_active":"2022-06-29T15:58:24.022660+0000","last_became_peered":"2022-06-29T15:58:24.022660+0000","last_unstale":"2022-06-29T15:58:25.573939+0000","last_undegraded":"2022-06-29T15:58:25.573939+0000","last_fullsized":"2022-06-29T15:58:25.573939+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:22.997265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:22.997265+0000","last_clean_scrub_stamp":"2022-06-29T15:58:22.997265+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.586362+0000","last_change":"2022-06-29T15:58:24.346157+0000","last_active":"2022-06-29T15:58:25.586362+0000","last_peered":"2022-06-29T15:58:25.586362+0000","last_clean":"2022-06-29T15:58:25.586362+0000","last_became_active":"2022-06-29T15:58:24.345683+0000","last_became_peered":"2022-06-29T15:58:24.345683+0000","last_unstale":"2022-06-29T15:58:25.586362+0000","last_undegraded":"2022-06-29T15:58:25.586362+0000","last_fullsized":"2022-06-29T15:58:25.586362+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:24.007232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:24.007232+0000","last_clean_scrub_stamp":"2022-06-29T15:58:24.007232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.356502+0000","last_change":"2022-06-29T15:58:21.001629+0000","last_active":"2022-06-29T15:58:25.356502+0000","last_peered":"2022-06-29T15:58:25.356502+0000","last_clean":"2022-06-29T15:58:25.356502+0000","last_became_active":"2022-06-29T15:58:21.000885+0000","last_became_peered":"2022-06-29T15:58:21.000885+0000","last_unstale":"2022-06-29T15:58:25.356502+0000","last_undegraded":"2022-06-29T15:58:25.356502+0000","last_fullsized":"2022-06-29T15:58:25.356502+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_clean_scrub_stamp":"2022-06-29T15:58:19.966881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.586092+0000","last_change":"2022-06-29T15:58:24.020809+0000","last_active":"2022-06-29T15:58:25.586092+0000","last_peered":"2022-06-29T15:58:25.586092+0000","last_clean":"2022-06-29T15:58:25.586092+0000","last_became_active":"2022-06-29T15:58:24.020202+0000","last_became_peered":"2022-06-29T15:58:24.020202+0000","last_unstale":"2022-06-29T15:58:25.586092+0000","last_undegraded":"2022-06-29T15:58:25.586092+0000","last_fullsized":"2022-06-29T15:58:25.586092+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:22.997265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:22.997265+0000","last_clean_scrub_stamp":"2022-06-29T15:58:22.997265+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.585809+0000","last_change":"2022-06-29T15:58:24.354169+0000","last_active":"2022-06-29T15:58:25.585809+0000","last_peered":"2022-06-29T15:58:25.585809+0000","last_clean":"2022-06-29T15:58:25.585809+0000","last_became_active":"2022-06-29T15:58:24.353817+0000","last_became_peered":"2022-06-29T15:58:24.353817+0000","last_unstale":"2022-06-29T15:58:25.585809+0000","last_undegraded":"2022-06-29T15:58:25.585809+0000","last_fullsized":"2022-06-29T15:58:25.585809+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:24.007232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:24.007232+0000","last_clean_scrub_stamp":"2022-06-29T15:58:24.007232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.348062+0000","last_change":"2022-06-29T15:58:20.997373+0000","last_active":"2022-06-29T15:58:25.348062+0000","last_peered":"2022-06-29T15:58:25.348062+0000","last_clean":"2022-06-29T15:58:25.348062+0000","last_became_active":"2022-06-29T15:58:20.997032+0000","last_became_peered":"2022-06-29T15:58:20.997032+0000","last_unstale":"2022-06-29T15:58:25.348062+0000","last_undegraded":"2022-06-29T15:58:25.348062+0000","last_fullsized":"2022-06-29T15:58:25.348062+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_clean_scrub_stamp":"2022-06-29T15:58:19.966881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"21'8","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.585296+0000","last_change":"2022-06-29T15:58:24.023775+0000","last_active":"2022-06-29T15:58:25.585296+0000","last_peered":"2022-06-29T15:58:25.585296+0000","last_clean":"2022-06-29T15:58:25.585296+0000","last_became_active":"2022-06-29T15:58:24.023397+0000","last_became_peered":"2022-06-29T15:58:24.023397+0000","last_unstale":"2022-06-29T15:58:25.585296+0000","last_undegraded":"2022-06-29T15:58:25.585296+0000","last_fullsized":"2022-06-29T15:58:25.585296+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:22.997265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:22.997265+0000","last_clean_scrub_stamp":"2022-06-29T15:58:22.997265+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.585601+0000","last_change":"2022-06-29T15:58:24.352933+0000","last_active":"2022-06-29T15:58:25.585601+0000","last_peered":"2022-06-29T15:58:25.585601+0000","last_clean":"2022-06-29T15:58:25.585601+0000","last_became_active":"2022-06-29T15:58:24.352641+0000","last_became_peered":"2022-06-29T15:58:24.352641+0000","last_unstale":"2022-06-29T15:58:25.585601+0000","last_undegraded":"2022-06-29T15:58:25.585601+0000","last_fullsized":"2022-06-29T15:58:25.585601+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:24.007232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:24.007232+0000","last_clean_scrub_stamp":"2022-06-29T15:58:24.007232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.592797+0000","last_change":"2022-06-29T15:58:21.005059+0000","last_active":"2022-06-29T15:58:25.592797+0000","last_peered":"2022-06-29T15:58:25.592797+0000","last_clean":"2022-06-29T15:58:25.592797+0000","last_became_active":"2022-06-29T15:58:21.004623+0000","last_became_peered":"2022-06-29T15:58:21.004623+0000","last_unstale":"2022-06-29T15:58:25.592797+0000","last_undegraded":"2022-06-29T15:58:25.592797+0000","last_fullsized":"2022-06-29T15:58:25.592797+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_clean_scrub_stamp":"2022-06-29T15:58:19.966881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"21'5","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.357136+0000","last_change":"2022-06-29T15:58:24.027287+0000","last_active":"2022-06-29T15:58:25.357136+0000","last_peered":"2022-06-29T15:58:25.357136+0000","last_clean":"2022-06-29T15:58:25.357136+0000","last_became_active":"2022-06-29T15:58:24.026492+0000","last_became_peered":"2022-06-29T15:58:24.026492+0000","last_unstale":"2022-06-29T15:58:25.357136+0000","last_undegraded":"2022-06-29T15:58:25.357136+0000","last_fullsized":"2022-06-29T15:58:25.357136+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:22.997265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:22.997265+0000","last_clean_scrub_stamp":"2022-06-29T15:58:22.997265+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.585249+0000","last_change":"2022-06-29T15:58:24.353076+0000","last_active":"2022-06-29T15:58:25.585249+0000","last_peered":"2022-06-29T15:58:25.585249+0000","last_clean":"2022-06-29T15:58:25.585249+0000","last_became_active":"2022-06-29T15:58:24.352673+0000","last_became_peered":"2022-06-29T15:58:24.352673+0000","last_unstale":"2022-06-29T15:58:25.585249+0000","last_undegraded":"2022-06-29T15:58:25.585249+0000","last_fullsized":"2022-06-29T15:58:25.585249+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:24.007232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:24.007232+0000","last_clean_scrub_stamp":"2022-06-29T15:58:24.007232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"21'6","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.592777+0000","last_change":"2022-06-29T15:58:24.028652+0000","last_active":"2022-06-29T15:58:25.592777+0000","last_peered":"2022-06-29T15:58:25.592777+0000","last_clean":"2022-06-29T15:58:25.592777+0000","last_became_active":"2022-06-29T15:58:24.028163+0000","last_became_peered":"2022-06-29T15:58:24.028163+0000","last_unstale":"2022-06-29T15:58:25.592777+0000","last_undegraded":"2022-06-29T15:58:25.592777+0000","last_fullsized":"2022-06-29T15:58:25.592777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:22.997265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:22.997265+0000","last_clean_scrub_stamp":"2022-06-29T15:58:22.997265+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.586172+0000","last_change":"2022-06-29T15:58:21.801445+0000","last_active":"2022-06-29T15:58:25.586172+0000","last_peered":"2022-06-29T15:58:25.586172+0000","last_clean":"2022-06-29T15:58:25.586172+0000","last_became_active":"2022-06-29T15:58:21.800792+0000","last_became_peered":"2022-06-29T15:58:21.800792+0000","last_unstale":"2022-06-29T15:58:25.586172+0000","last_undegraded":"2022-06-29T15:58:25.586172+0000","last_fullsized":"2022-06-29T15:58:25.586172+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_clean_scrub_stamp":"2022-06-29T15:58:19.966881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.586062+0000","last_change":"2022-06-29T15:58:24.351819+0000","last_active":"2022-06-29T15:58:25.586062+0000","last_peered":"2022-06-29T15:58:25.586062+0000","last_clean":"2022-06-29T15:58:25.586062+0000","last_became_active":"2022-06-29T15:58:24.351535+0000","last_became_peered":"2022-06-29T15:58:24.351535+0000","last_unstale":"2022-06-29T15:58:25.586062+0000","last_undegraded":"2022-06-29T15:58:25.586062+0000","last_fullsized":"2022-06-29T15:58:25.586062+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:24.007232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:24.007232+0000","last_clean_scrub_stamp":"2022-06-29T15:58:24.007232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"21'8","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.585896+0000","last_change":"2022-06-29T15:58:24.024424+0000","last_active":"2022-06-29T15:58:25.585896+0000","last_peered":"2022-06-29T15:58:25.585896+0000","last_clean":"2022-06-29T15:58:25.585896+0000","last_became_active":"2022-06-29T15:58:24.024076+0000","last_became_peered":"2022-06-29T15:58:24.024076+0000","last_unstale":"2022-06-29T15:58:25.585896+0000","last_undegraded":"2022-06-29T15:58:25.585896+0000","last_fullsized":"2022-06-29T15:58:25.585896+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:22.997265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:22.997265+0000","last_clean_scrub_stamp":"2022-06-29T15:58:22.997265+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1535,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.356376+0000","last_change":"2022-06-29T15:58:21.807952+0000","last_active":"2022-06-29T15:58:25.356376+0000","last_peered":"2022-06-29T15:58:25.356376+0000","last_clean":"2022-06-29T15:58:25.356376+0000","last_became_active":"2022-06-29T15:58:21.807346+0000","last_became_peered":"2022-06-29T15:58:21.807346+0000","last_unstale":"2022-06-29T15:58:25.356376+0000","last_undegraded":"2022-06-29T15:58:25.356376+0000","last_fullsized":"2022-06-29T15:58:25.356376+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_clean_scrub_stamp":"2022-06-29T15:58:19.966881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.573873+0000","last_change":"2022-06-29T15:58:24.347840+0000","last_active":"2022-06-29T15:58:25.573873+0000","last_peered":"2022-06-29T15:58:25.573873+0000","last_clean":"2022-06-29T15:58:25.573873+0000","last_became_active":"2022-06-29T15:58:24.347424+0000","last_became_peered":"2022-06-29T15:58:24.347424+0000","last_unstale":"2022-06-29T15:58:25.573873+0000","last_undegraded":"2022-06-29T15:58:25.573873+0000","last_fullsized":"2022-06-29T15:58:25.573873+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:24.007232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:24.007232+0000","last_clean_scrub_stamp":"2022-06-29T15:58:24.007232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"21'7","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.573968+0000","last_change":"2022-06-29T15:58:24.021541+0000","last_active":"2022-06-29T15:58:25.573968+0000","last_peered":"2022-06-29T15:58:25.573968+0000","last_clean":"2022-06-29T15:58:25.573968+0000","last_became_active":"2022-06-29T15:58:24.021230+0000","last_became_peered":"2022-06-29T15:58:24.021230+0000","last_unstale":"2022-06-29T15:58:25.573968+0000","last_undegraded":"2022-06-29T15:58:25.573968+0000","last_fullsized":"2022-06-29T15:58:25.573968+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:22.997265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:22.997265+0000","last_clean_scrub_stamp":"2022-06-29T15:58:22.997265+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.585702+0000","last_change":"2022-06-29T15:58:20.996366+0000","last_active":"2022-06-29T15:58:25.585702+0000","last_peered":"2022-06-29T15:58:25.585702+0000","last_clean":"2022-06-29T15:58:25.585702+0000","last_became_active":"2022-06-29T15:58:20.995883+0000","last_became_peered":"2022-06-29T15:58:20.995883+0000","last_unstale":"2022-06-29T15:58:25.585702+0000","last_undegraded":"2022-06-29T15:58:25.585702+0000","last_fullsized":"2022-06-29T15:58:25.585702+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_clean_scrub_stamp":"2022-06-29T15:58:19.966881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.357215+0000","last_change":"2022-06-29T15:58:24.358772+0000","last_active":"2022-06-29T15:58:25.357215+0000","last_peered":"2022-06-29T15:58:25.357215+0000","last_clean":"2022-06-29T15:58:25.357215+0000","last_became_active":"2022-06-29T15:58:24.358128+0000","last_became_peered":"2022-06-29T15:58:24.358128+0000","last_unstale":"2022-06-29T15:58:25.357215+0000","last_undegraded":"2022-06-29T15:58:25.357215+0000","last_fullsized":"2022-06-29T15:58:25.357215+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:24.007232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:24.007232+0000","last_clean_scrub_stamp":"2022-06-29T15:58:24.007232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"22'4","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:26.092891+0000","last_change":"2022-06-29T15:58:24.022399+0000","last_active":"2022-06-29T15:58:26.092891+0000","last_peered":"2022-06-29T15:58:26.092891+0000","last_clean":"2022-06-29T15:58:26.092891+0000","last_became_active":"2022-06-29T15:58:24.022030+0000","last_became_peered":"2022-06-29T15:58:24.022030+0000","last_unstale":"2022-06-29T15:58:26.092891+0000","last_undegraded":"2022-06-29T15:58:26.092891+0000","last_fullsized":"2022-06-29T15:58:26.092891+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:22.997265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:22.997265+0000","last_clean_scrub_stamp":"2022-06-29T15:58:22.997265+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-06-29T15:58:25.360692+0000","last_change":"2022-06-29T15:58:21.003020+0000","last_active":"2022-06-29T15:58:25.360692+0000","last_peered":"2022-06-29T15:58:25.360692+0000","last_clean":"2022-06-29T15:58:25.360692+0000","last_became_active":"2022-06-29T15:58:21.002353+0000","last_became_peered":"2022-06-29T15:58:21.002353+0000","last_unstale":"2022-06-29T15:58:25.360692+0000","last_undegraded":"2022-06-29T15:58:25.360692+0000","last_fullsized":"2022-06-29T15:58:25.360692+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_clean_scrub_stamp":"2022-06-29T15:58:19.966881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":2304,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":16,"acting":16,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795912,"kb_used_data":92903480,"kb_used_omap":75,"kb_used_meta":0,"kb_avail":92903468,"statfs":{"total":95948165120,"available":95133151232,"internally_reserved":12288,"allocated":95133163520,"data_stored":95133163520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77622,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92600000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88800000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795904,"kb_used_data":92903488,"kb_used_omap":46,"kb_used_meta":0,"kb_avail":92903476,"statfs":{"total":95948165120,"available":95133159424,"internally_reserved":12288,"allocated":95133171712,"data_stored":95133171712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47791,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92800000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1259999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4319999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3440000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":8,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795912,"kb_used_data":92903480,"kb_used_omap":73,"kb_used_meta":0,"kb_avail":92903468,"statfs":{"total":95948165120,"available":95133151232,"internally_reserved":12288,"allocated":95133163520,"data_stored":95133163520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74884,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.179}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0880000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795908,"kb_used_data":92903484,"kb_used_omap":61,"kb_used_meta":0,"kb_avail":92903472,"statfs":{"total":95948165120,"available":95133155328,"internally_reserved":12288,"allocated":95133167616,"data_stored":95133167616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63068,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0549999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.006}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795920,"kb_used_data":92903472,"kb_used_omap":69,"kb_used_meta":0,"kb_avail":92903460,"statfs":{"total":95948165120,"available":95133143040,"internally_reserved":12288,"allocated":95133155328,"data_stored":95133155328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71238,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0620000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2490000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.117}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.163}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795916,"kb_used_data":92903476,"kb_used_omap":73,"kb_used_meta":0,"kb_avail":92903464,"statfs":{"total":95948165120,"available":95133147136,"internally_reserved":12288,"allocated":95133159424,"data_stored":95133159424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74833,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.679}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.506}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4530000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.361}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6120000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795920,"kb_used_data":92903476,"kb_used_omap":107,"kb_used_meta":0,"kb_avail":92903460,"statfs":{"total":95948165120,"available":95133143040,"internally_reserved":16384,"allocated":95133159424,"data_stored":95133159424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":109785,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.752}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94499999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96799999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.02}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":8,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795912,"kb_used_data":92903484,"kb_used_omap":99,"kb_used_meta":0,"kb_avail":92903468,"statfs":{"total":95948165120,"available":95133151232,"internally_reserved":16384,"allocated":95133167616,"data_stored":95133167616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101407,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1279999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]}]}]}} 2022-06-29T15:58:33.337 INFO:tasks.ceph.ceph_manager.ceph:clean! 2022-06-29T15:58:33.338 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-06-29T15:58:33.338 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2022-06-29T15:58:33.338 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2022-06-29T15:58:33.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.429+0000 7f331fbe0700 1 Processor -- start 2022-06-29T15:58:33.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.429+0000 7f331fbe0700 1 -- start start 2022-06-29T15:58:33.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.433+0000 7f331fbe0700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f33180b61d0 0x7f33180b0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:33.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.433+0000 7f331fbe0700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3318142820 con 0x7f331813f790 2022-06-29T15:58:33.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.433+0000 7f331fbe0700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3318062bb0 con 0x7f331813fc00 2022-06-29T15:58:33.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.433+0000 7f331fbe0700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33180b65b0 con 0x7f33180b61d0 2022-06-29T15:58:33.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.433+0000 7f331d17b700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f331813f790 0x7f331813bc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53474/0 (socket says 172.21.15.60:53474) 2022-06-29T15:58:33.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.433+0000 7f331d17b700 1 -- 172.21.15.60:0/790460355 learned_addr learned my addr 172.21.15.60:0/790460355 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:33.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.433+0000 7f331d97c700 1 --2- 172.21.15.60:0/790460355 >> v2:172.21.15.92:3300/0 conn(0x7f33180b61d0 0x7f33180b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:33.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.433+0000 7f331c97a700 1 -- 172.21.15.60:0/790460355 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3508190535 0 0) 0x7f3318142820 con 0x7f331813f790 2022-06-29T15:58:33.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.433+0000 7f331c97a700 1 -- 172.21.15.60:0/790460355 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3300003260 con 0x7f331813f790 2022-06-29T15:58:33.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.433+0000 7f331c97a700 1 -- 172.21.15.60:0/790460355 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 897324287 0 0) 0x7f3318062bb0 con 0x7f331813fc00 2022-06-29T15:58:33.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.433+0000 7f331c97a700 1 -- 172.21.15.60:0/790460355 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3318142820 con 0x7f331813fc00 2022-06-29T15:58:33.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.433+0000 7f331d97c700 1 -- 172.21.15.60:0/790460355 >> v1:172.21.15.92:6789/0 conn(0x7f331813fc00 legacy=0x7f331813c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:33.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.433+0000 7f331d97c700 1 -- 172.21.15.60:0/790460355 >> v1:172.21.15.60:6789/0 conn(0x7f331813f790 legacy=0x7f331813bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:33.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.433+0000 7f331d97c700 1 -- 172.21.15.60:0/790460355 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331813f050 con 0x7f33180b61d0 2022-06-29T15:58:33.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.433+0000 7f331d97c700 1 --2- 172.21.15.60:0/790460355 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f33180b61d0 0x7f33180b0540 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f33180acf50 tx=0x7f3314006e80).ready entity=mon.1 client_cookie=37edda65e1841027 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:33.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.433+0000 7f331c97a700 1 -- 172.21.15.60:0/790460355 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3314010400 con 0x7f33180b61d0 2022-06-29T15:58:33.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.433+0000 7f331c97a700 1 -- 172.21.15.60:0/790460355 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3314000e90 con 0x7f33180b61d0 2022-06-29T15:58:33.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.433+0000 7f331c97a700 1 -- 172.21.15.60:0/790460355 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3314010750 con 0x7f33180b61d0 2022-06-29T15:58:33.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.433+0000 7f331fbe0700 1 -- 172.21.15.60:0/790460355 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f33180b61d0 msgr2=0x7f33180b0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:33.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.433+0000 7f331fbe0700 1 --2- 172.21.15.60:0/790460355 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f33180b61d0 0x7f33180b0540 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f33180acf50 tx=0x7f3314006e80).stop 2022-06-29T15:58:33.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.437+0000 7f331fbe0700 1 -- 172.21.15.60:0/790460355 shutdown_connections 2022-06-29T15:58:33.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.437+0000 7f331fbe0700 1 --2- 172.21.15.60:0/790460355 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f33180b61d0 0x7f33180b0540 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:33.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.437+0000 7f331fbe0700 1 -- 172.21.15.60:0/790460355 >> 172.21.15.60:0/790460355 conn(0x7f3318063b50 msgr2=0x7f33180b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:33.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.437+0000 7f331fbe0700 1 -- 172.21.15.60:0/790460355 shutdown_connections 2022-06-29T15:58:33.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.437+0000 7f331fbe0700 1 -- 172.21.15.60:0/790460355 wait complete. 2022-06-29T15:58:33.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.437+0000 7f331fbe0700 1 Processor -- start 2022-06-29T15:58:33.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.437+0000 7f331fbe0700 1 -- start start 2022-06-29T15:58:33.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.441+0000 7f331fbe0700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f33180b61d0 0x7f33180afc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:33.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.441+0000 7f331fbe0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f331813f790 0x7f3318066560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:33.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.441+0000 7f331fbe0700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f331813fc00 0x7f3318066a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:33.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.441+0000 7f331fbe0700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33180648f0 con 0x7f331813f790 2022-06-29T15:58:33.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.441+0000 7f331fbe0700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3318064b90 con 0x7f331813fc00 2022-06-29T15:58:33.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.441+0000 7f331fbe0700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33180ae970 con 0x7f33180b61d0 2022-06-29T15:58:33.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.441+0000 7f331d17b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f331813f790 0x7f3318066560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:33.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.441+0000 7f331d17b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f331813f790 0x7f3318066560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45394/0 (socket says 172.21.15.60:45394) 2022-06-29T15:58:33.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.441+0000 7f331d17b700 1 -- 172.21.15.60:0/3149435150 learned_addr learned my addr 172.21.15.60:0/3149435150 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:33.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.441+0000 7f331d97c700 1 --2- 172.21.15.60:0/3149435150 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f33180b61d0 0x7f33180afc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:33.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.441+0000 7f331e17d700 1 --2- 172.21.15.60:0/3149435150 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f331813fc00 0x7f3318066a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:33.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.441+0000 7f331d17b700 1 -- 172.21.15.60:0/3149435150 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f33180b61d0 msgr2=0x7f33180afc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:33.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.441+0000 7f331d17b700 1 --2- 172.21.15.60:0/3149435150 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f33180b61d0 0x7f33180afc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:33.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.441+0000 7f331d17b700 1 -- 172.21.15.60:0/3149435150 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f331813fc00 msgr2=0x7f3318066a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:33.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.441+0000 7f331d17b700 1 --2- 172.21.15.60:0/3149435150 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f331813fc00 0x7f3318066a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:33.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.441+0000 7f331d17b700 1 -- 172.21.15.60:0/3149435150 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331400c310 con 0x7f331813f790 2022-06-29T15:58:33.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.441+0000 7f331e17d700 1 --2- 172.21.15.60:0/3149435150 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f331813fc00 0x7f3318066a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:58:33.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.441+0000 7f331d17b700 1 --2- 172.21.15.60:0/3149435150 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f331813f790 0x7f3318066560 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f33080011d0 tx=0x7f3308007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:33.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.441+0000 7f3306ffd700 1 -- 172.21.15.60:0/3149435150 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f33080070d0 con 0x7f331813f790 2022-06-29T15:58:33.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.441+0000 7f3306ffd700 1 -- 172.21.15.60:0/3149435150 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3308007230 con 0x7f331813f790 2022-06-29T15:58:33.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.441+0000 7f331fbe0700 1 -- 172.21.15.60:0/3149435150 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3318064e30 con 0x7f331813f790 2022-06-29T15:58:33.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.441+0000 7f3306ffd700 1 -- 172.21.15.60:0/3149435150 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3308005610 con 0x7f331813f790 2022-06-29T15:58:33.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.441+0000 7f331fbe0700 1 -- 172.21.15.60:0/3149435150 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3318137fd0 con 0x7f331813f790 2022-06-29T15:58:33.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.445+0000 7f331fbe0700 1 -- 172.21.15.60:0/3149435150 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33180aeb30 con 0x7f331813f790 2022-06-29T15:58:33.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.445+0000 7f3306ffd700 1 -- 172.21.15.60:0/3149435150 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f330801a060 con 0x7f331813f790 2022-06-29T15:58:33.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.445+0000 7f3306ffd700 1 --2- 172.21.15.60:0/3149435150 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f32f8053a70 0x7f32f8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:33.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.445+0000 7f3306ffd700 1 -- 172.21.15.60:0/3149435150 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f330806b6c0 con 0x7f331813f790 2022-06-29T15:58:33.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.445+0000 7f331d97c700 1 --2- 172.21.15.60:0/3149435150 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f32f8053a70 0x7f32f8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:33.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.445+0000 7f331d97c700 1 --2- 172.21.15.60:0/3149435150 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f32f8053a70 0x7f32f8055f30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f331401b000 tx=0x7f3314015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:33.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.449+0000 7f3306ffd700 1 -- 172.21.15.60:0/3149435150 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f33080425e0 con 0x7f331813f790 2022-06-29T15:58:33.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.621+0000 7f331fbe0700 1 -- 172.21.15.60:0/3149435150 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f33180573d0 con 0x7f331813f790 2022-06-29T15:58:33.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.621+0000 7f3306ffd700 1 -- 172.21.15.60:0/3149435150 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+45 (secure 0 0 0) 0x7f3308018080 con 0x7f331813f790 2022-06-29T15:58:33.628 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:58:33.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.625+0000 7f331fbe0700 1 -- 172.21.15.60:0/3149435150 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f32f8053a70 msgr2=0x7f32f8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:33.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.625+0000 7f331fbe0700 1 --2- 172.21.15.60:0/3149435150 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f32f8053a70 0x7f32f8055f30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f331401b000 tx=0x7f3314015000).stop 2022-06-29T15:58:33.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.625+0000 7f331fbe0700 1 -- 172.21.15.60:0/3149435150 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f331813f790 msgr2=0x7f3318066560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:33.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.625+0000 7f331fbe0700 1 --2- 172.21.15.60:0/3149435150 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f331813f790 0x7f3318066560 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f33080011d0 tx=0x7f3308007b50).stop 2022-06-29T15:58:33.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.625+0000 7f331fbe0700 1 -- 172.21.15.60:0/3149435150 shutdown_connections 2022-06-29T15:58:33.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.625+0000 7f331fbe0700 1 --2- 172.21.15.60:0/3149435150 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f32f8053a70 0x7f32f8055f30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:33.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.625+0000 7f331fbe0700 1 --2- 172.21.15.60:0/3149435150 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f331813fc00 0x7f3318066a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:33.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.625+0000 7f331fbe0700 1 --2- 172.21.15.60:0/3149435150 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f331813f790 0x7f3318066560 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:33.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.625+0000 7f331fbe0700 1 --2- 172.21.15.60:0/3149435150 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f33180b61d0 0x7f33180afc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:33.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.625+0000 7f331fbe0700 1 -- 172.21.15.60:0/3149435150 >> 172.21.15.60:0/3149435150 conn(0x7f3318063b50 msgr2=0x7f33181420f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:33.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.625+0000 7f331fbe0700 1 -- 172.21.15.60:0/3149435150 shutdown_connections 2022-06-29T15:58:33.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.625+0000 7f331fbe0700 1 -- 172.21.15.60:0/3149435150 wait complete. 2022-06-29T15:58:33.640 INFO:teuthology.orchestra.run.smithi060.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-06-29T15:58:33.641 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2022-06-29T15:58:33.641 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T15:58:33.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.737+0000 7f1219a73700 1 Processor -- start 2022-06-29T15:58:33.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.737+0000 7f1219a73700 1 -- start start 2022-06-29T15:58:33.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.737+0000 7f1219a73700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f12140b5eb0 0x7f1214142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:33.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.737+0000 7f1219a73700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f12140b2c50 0x7f12140b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:33.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.737+0000 7f1219a73700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12140b0cf0 con 0x7f12140b48b0 2022-06-29T15:58:33.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.737+0000 7f1219a73700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12140b54e0 con 0x7f12140b2c50 2022-06-29T15:58:33.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.737+0000 7f1219a73700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12140b4d10 con 0x7f12140b5eb0 2022-06-29T15:58:33.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.737+0000 7f1219272700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f12140b2c50 0x7f12140b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:33.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.737+0000 7f1219272700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f12140b2c50 0x7f12140b30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45404/0 (socket says 172.21.15.60:45404) 2022-06-29T15:58:33.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.737+0000 7f1219272700 1 -- 172.21.15.60:0/1511344885 learned_addr learned my addr 172.21.15.60:0/1511344885 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:33.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.737+0000 7f1218a71700 1 --2- 172.21.15.60:0/1511344885 >> v2:172.21.15.92:3300/0 conn(0x7f12140b5eb0 0x7f1214142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:33.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.737+0000 7f1219272700 1 -- 172.21.15.60:0/1511344885 >> v2:172.21.15.92:3300/0 conn(0x7f12140b5eb0 msgr2=0x7f1214142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:33.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.737+0000 7f1219272700 1 --2- 172.21.15.60:0/1511344885 >> v2:172.21.15.92:3300/0 conn(0x7f12140b5eb0 0x7f1214142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:33.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.737+0000 7f1219272700 1 -- 172.21.15.60:0/1511344885 >> v1:172.21.15.92:6789/0 conn(0x7f12140b48b0 legacy=0x7f12140b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:33.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.737+0000 7f1219272700 1 -- 172.21.15.60:0/1511344885 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12140b6290 con 0x7f12140b2c50 2022-06-29T15:58:33.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.737+0000 7f1219272700 1 --2- 172.21.15.60:0/1511344885 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12140b2c50 0x7f12140b30c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f120c0032f0 tx=0x7f120c009290).ready entity=mon.0 client_cookie=135633f0b37fac27 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:33.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.737+0000 7f120b7fe700 1 -- 172.21.15.60:0/1511344885 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f120c00b870 con 0x7f12140b2c50 2022-06-29T15:58:33.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.741+0000 7f120b7fe700 1 -- 172.21.15.60:0/1511344885 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f120c00b9d0 con 0x7f12140b2c50 2022-06-29T15:58:33.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.741+0000 7f120b7fe700 1 -- 172.21.15.60:0/1511344885 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f120c00bb40 con 0x7f12140b2c50 2022-06-29T15:58:33.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.741+0000 7f1219a73700 1 -- 172.21.15.60:0/1511344885 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12140b2c50 msgr2=0x7f12140b30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:33.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.741+0000 7f1219a73700 1 --2- 172.21.15.60:0/1511344885 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12140b2c50 0x7f12140b30c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f120c0032f0 tx=0x7f120c009290).stop 2022-06-29T15:58:33.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.741+0000 7f1219a73700 1 -- 172.21.15.60:0/1511344885 shutdown_connections 2022-06-29T15:58:33.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.741+0000 7f1219a73700 1 --2- 172.21.15.60:0/1511344885 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12140b2c50 0x7f12140b30c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:33.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.741+0000 7f1219a73700 1 --2- 172.21.15.60:0/1511344885 >> v2:172.21.15.92:3300/0 conn(0x7f12140b5eb0 0x7f1214142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:33.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.741+0000 7f1219a73700 1 -- 172.21.15.60:0/1511344885 >> 172.21.15.60:0/1511344885 conn(0x7f1214062090 msgr2=0x7f1214062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:33.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.741+0000 7f1219a73700 1 -- 172.21.15.60:0/1511344885 shutdown_connections 2022-06-29T15:58:33.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.741+0000 7f1219a73700 1 -- 172.21.15.60:0/1511344885 wait complete. 2022-06-29T15:58:33.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.741+0000 7f1219a73700 1 Processor -- start 2022-06-29T15:58:33.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.741+0000 7f1219a73700 1 -- start start 2022-06-29T15:58:33.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.741+0000 7f1219a73700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12140b2c50 0x7f1214141e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:33.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.741+0000 7f1219a73700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12140b48b0 0x7f1214155a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:33.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.741+0000 7f1219a73700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12140b5eb0 0x7f1214151c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:33.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.741+0000 7f1219a73700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1214113790 con 0x7f12140b5eb0 2022-06-29T15:58:33.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.741+0000 7f1219a73700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12141552e0 con 0x7f12140b48b0 2022-06-29T15:58:33.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.741+0000 7f1219a73700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1214155580 con 0x7f12140b2c50 2022-06-29T15:58:33.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.741+0000 7f1219272700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12140b5eb0 0x7f1214151c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:33.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.741+0000 7f1218a71700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12140b2c50 0x7f1214141e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:33.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.741+0000 7f1219272700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12140b5eb0 0x7f1214151c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45410/0 (socket says 172.21.15.60:45410) 2022-06-29T15:58:33.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.741+0000 7f1219272700 1 -- 172.21.15.60:0/3101311291 learned_addr learned my addr 172.21.15.60:0/3101311291 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:33.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.741+0000 7f120bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12140b48b0 0x7f1214155a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:33.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.745+0000 7f1219272700 1 -- 172.21.15.60:0/3101311291 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12140b2c50 msgr2=0x7f1214141e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:33.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.745+0000 7f1219272700 1 --2- 172.21.15.60:0/3101311291 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12140b2c50 0x7f1214141e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:33.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.745+0000 7f1219272700 1 -- 172.21.15.60:0/3101311291 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12140b48b0 msgr2=0x7f1214155a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:33.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.745+0000 7f1219272700 1 --2- 172.21.15.60:0/3101311291 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12140b48b0 0x7f1214155a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:33.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.745+0000 7f1219272700 1 -- 172.21.15.60:0/3101311291 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f120c009530 con 0x7f12140b5eb0 2022-06-29T15:58:33.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.745+0000 7f1219272700 1 --2- 172.21.15.60:0/3101311291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12140b5eb0 0x7f1214151c00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f120c0032f0 tx=0x7f120c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:33.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.745+0000 7f1209ffb700 1 -- 172.21.15.60:0/3101311291 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f120c00b870 con 0x7f12140b5eb0 2022-06-29T15:58:33.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.745+0000 7f1209ffb700 1 -- 172.21.15.60:0/3101311291 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f120c021630 con 0x7f12140b5eb0 2022-06-29T15:58:33.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.745+0000 7f1209ffb700 1 -- 172.21.15.60:0/3101311291 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f120c021990 con 0x7f12140b5eb0 2022-06-29T15:58:33.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.745+0000 7f1219a73700 1 -- 172.21.15.60:0/3101311291 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1214113410 con 0x7f12140b5eb0 2022-06-29T15:58:33.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.745+0000 7f1219a73700 1 -- 172.21.15.60:0/3101311291 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1214155f10 con 0x7f12140b5eb0 2022-06-29T15:58:33.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.749+0000 7f1219a73700 1 -- 172.21.15.60:0/3101311291 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12140573d0 con 0x7f12140b5eb0 2022-06-29T15:58:33.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.749+0000 7f1209ffb700 1 -- 172.21.15.60:0/3101311291 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f120c027020 con 0x7f12140b5eb0 2022-06-29T15:58:33.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.749+0000 7f1209ffb700 1 --2- 172.21.15.60:0/3101311291 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f11f4053aa0 0x7f11f4055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:33.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.749+0000 7f1209ffb700 1 -- 172.21.15.60:0/3101311291 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f120c06d150 con 0x7f12140b5eb0 2022-06-29T15:58:33.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.749+0000 7f1218a71700 1 --2- 172.21.15.60:0/3101311291 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f11f4053aa0 0x7f11f4055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:33.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.749+0000 7f1218a71700 1 --2- 172.21.15.60:0/3101311291 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f11f4053aa0 0x7f11f4055f60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f121000b000 tx=0x7f1210006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:33.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.757+0000 7f1209ffb700 1 -- 172.21.15.60:0/3101311291 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f120c044120 con 0x7f12140b5eb0 2022-06-29T15:58:33.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.885+0000 7f1219a73700 1 -- 172.21.15.60:0/3101311291 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1214113600 con 0x7f12140b5eb0 2022-06-29T15:58:33.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.885+0000 7f1209ffb700 1 -- 172.21.15.60:0/3101311291 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16038 (secure 0 0 0) 0x7f120c015020 con 0x7f12140b5eb0 2022-06-29T15:58:33.892 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:58:33.892 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":23,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T15:58:26.345644+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-06-29T15:58:22.424395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-06-29T15:58:23.343438+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T15:58:33.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.889+0000 7f1219a73700 1 -- 172.21.15.60:0/3101311291 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f11f4053aa0 msgr2=0x7f11f4055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:33.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.889+0000 7f1219a73700 1 --2- 172.21.15.60:0/3101311291 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f11f4053aa0 0x7f11f4055f60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f121000b000 tx=0x7f1210006df0).stop 2022-06-29T15:58:33.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.889+0000 7f1219a73700 1 -- 172.21.15.60:0/3101311291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12140b5eb0 msgr2=0x7f1214151c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:33.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.889+0000 7f1219a73700 1 --2- 172.21.15.60:0/3101311291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12140b5eb0 0x7f1214151c00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f120c0032f0 tx=0x7f120c016000).stop 2022-06-29T15:58:33.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.889+0000 7f1219a73700 1 -- 172.21.15.60:0/3101311291 shutdown_connections 2022-06-29T15:58:33.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.889+0000 7f1219a73700 1 --2- 172.21.15.60:0/3101311291 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f11f4053aa0 0x7f11f4055f60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:33.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.889+0000 7f1219a73700 1 --2- 172.21.15.60:0/3101311291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12140b5eb0 0x7f1214151c00 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:33.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.889+0000 7f1219a73700 1 --2- 172.21.15.60:0/3101311291 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12140b48b0 0x7f1214155a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:33.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.889+0000 7f1219a73700 1 --2- 172.21.15.60:0/3101311291 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12140b2c50 0x7f1214141e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:33.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.889+0000 7f1219a73700 1 -- 172.21.15.60:0/3101311291 >> 172.21.15.60:0/3101311291 conn(0x7f1214062090 msgr2=0x7f12140b7040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:33.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.889+0000 7f1219a73700 1 -- 172.21.15.60:0/3101311291 shutdown_connections 2022-06-29T15:58:33.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:33.889+0000 7f1219a73700 1 -- 172.21.15.60:0/3101311291 wait complete. 2022-06-29T15:58:33.905 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-29T15:58:22.424395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-06-29T15:58:23.343438+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-29T15:58:33.906 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T15:58:34.016 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.009+0000 7fbb45e95700 1 Processor -- start 2022-06-29T15:58:34.016 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.009+0000 7fbb45e95700 1 -- start start 2022-06-29T15:58:34.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.009+0000 7fbb45e95700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbb400b3d30 0x7fbb40146ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:34.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.009+0000 7fbb45e95700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbb40061d90 0x7fbb40142b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:34.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.009+0000 7fbb45e95700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb400b4f50 con 0x7fbb400b9d70 2022-06-29T15:58:34.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.009+0000 7fbb45e95700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb400ba150 con 0x7fbb400b3d30 2022-06-29T15:58:34.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.009+0000 7fbb45e95700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb400b5180 con 0x7fbb40061d90 2022-06-29T15:58:34.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.013+0000 7fbb3effd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbb400b3d30 0x7fbb40146ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:34.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.013+0000 7fbb3effd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbb400b3d30 0x7fbb40146ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45416/0 (socket says 172.21.15.60:45416) 2022-06-29T15:58:34.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.013+0000 7fbb3effd700 1 -- 172.21.15.60:0/4001274959 learned_addr learned my addr 172.21.15.60:0/4001274959 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:34.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.013+0000 7fbb3ffff700 1 --2- 172.21.15.60:0/4001274959 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbb40061d90 0x7fbb40142b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:34.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.013+0000 7fbb3effd700 1 -- 172.21.15.60:0/4001274959 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbb40061d90 msgr2=0x7fbb40142b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:34.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.013+0000 7fbb3effd700 1 --2- 172.21.15.60:0/4001274959 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbb40061d90 0x7fbb40142b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.013+0000 7fbb3effd700 1 -- 172.21.15.60:0/4001274959 >> v1:172.21.15.92:6789/0 conn(0x7fbb400b9d70 legacy=0x7fbb401465d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:34.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.013+0000 7fbb3effd700 1 -- 172.21.15.60:0/4001274959 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb401461a0 con 0x7fbb400b3d30 2022-06-29T15:58:34.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.013+0000 7fbb3effd700 1 --2- 172.21.15.60:0/4001274959 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb400b3d30 0x7fbb40146ce0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fbb28002f70 tx=0x7fbb28006eb0).ready entity=mon.0 client_cookie=e79ee315c225646f server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:34.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.013+0000 7fbb3e7fc700 1 -- 172.21.15.60:0/4001274959 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb2800d570 con 0x7fbb400b3d30 2022-06-29T15:58:34.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.013+0000 7fbb3e7fc700 1 -- 172.21.15.60:0/4001274959 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb28000e90 con 0x7fbb400b3d30 2022-06-29T15:58:34.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.013+0000 7fbb3e7fc700 1 -- 172.21.15.60:0/4001274959 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb2800d900 con 0x7fbb400b3d30 2022-06-29T15:58:34.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.013+0000 7fbb45e95700 1 -- 172.21.15.60:0/4001274959 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb400b3d30 msgr2=0x7fbb40146ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:34.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.013+0000 7fbb45e95700 1 --2- 172.21.15.60:0/4001274959 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb400b3d30 0x7fbb40146ce0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fbb28002f70 tx=0x7fbb28006eb0).stop 2022-06-29T15:58:34.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.013+0000 7fbb45e95700 1 -- 172.21.15.60:0/4001274959 shutdown_connections 2022-06-29T15:58:34.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.013+0000 7fbb45e95700 1 --2- 172.21.15.60:0/4001274959 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbb40061d90 0x7fbb40142b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.013+0000 7fbb45e95700 1 --2- 172.21.15.60:0/4001274959 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb400b3d30 0x7fbb40146ce0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.013+0000 7fbb45e95700 1 -- 172.21.15.60:0/4001274959 >> 172.21.15.60:0/4001274959 conn(0x7fbb40062e70 msgr2=0x7fbb400b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:34.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.013+0000 7fbb45e95700 1 -- 172.21.15.60:0/4001274959 shutdown_connections 2022-06-29T15:58:34.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.013+0000 7fbb45e95700 1 -- 172.21.15.60:0/4001274959 wait complete. 2022-06-29T15:58:34.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.013+0000 7fbb45e95700 1 Processor -- start 2022-06-29T15:58:34.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.013+0000 7fbb45e95700 1 -- start start 2022-06-29T15:58:34.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.013+0000 7fbb45e95700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb40061d90 0x7fbb40151d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:34.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.013+0000 7fbb45e95700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbb400b3d30 0x7fbb40149950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:34.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.013+0000 7fbb45e95700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb400b9d70 0x7fbb40113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:34.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.013+0000 7fbb45e95700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb400b7e40 con 0x7fbb400b9d70 2022-06-29T15:58:34.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.013+0000 7fbb45e95700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb400b80e0 con 0x7fbb40061d90 2022-06-29T15:58:34.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.013+0000 7fbb45e95700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb400ae3f0 con 0x7fbb400b3d30 2022-06-29T15:58:34.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.013+0000 7fbb3ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb400b9d70 0x7fbb40113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:34.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.013+0000 7fbb3ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb400b9d70 0x7fbb40113f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45424/0 (socket says 172.21.15.60:45424) 2022-06-29T15:58:34.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.013+0000 7fbb3effd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbb400b3d30 0x7fbb40149950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:34.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.013+0000 7fbb3ffff700 1 -- 172.21.15.60:0/2799352324 learned_addr learned my addr 172.21.15.60:0/2799352324 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:34.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.013+0000 7fbb3f7fe700 1 --2- 172.21.15.60:0/2799352324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb40061d90 0x7fbb40151d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:34.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.017+0000 7fbb3ffff700 1 -- 172.21.15.60:0/2799352324 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbb400b3d30 msgr2=0x7fbb40149950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:34.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.017+0000 7fbb3ffff700 1 --2- 172.21.15.60:0/2799352324 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbb400b3d30 0x7fbb40149950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.017+0000 7fbb3ffff700 1 -- 172.21.15.60:0/2799352324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb40061d90 msgr2=0x7fbb40151d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:34.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.017+0000 7fbb3ffff700 1 --2- 172.21.15.60:0/2799352324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb40061d90 0x7fbb40151d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.017+0000 7fbb3ffff700 1 -- 172.21.15.60:0/2799352324 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb2800b1e0 con 0x7fbb400b9d70 2022-06-29T15:58:34.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.017+0000 7fbb3ffff700 1 --2- 172.21.15.60:0/2799352324 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb400b9d70 0x7fbb40113f50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fbb3000c000 tx=0x7fbb30006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:34.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.017+0000 7fbb3cff9700 1 -- 172.21.15.60:0/2799352324 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb30011750 con 0x7fbb400b9d70 2022-06-29T15:58:34.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.017+0000 7fbb3cff9700 1 -- 172.21.15.60:0/2799352324 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb300118b0 con 0x7fbb400b9d70 2022-06-29T15:58:34.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.017+0000 7fbb45e95700 1 -- 172.21.15.60:0/2799352324 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb400ae6d0 con 0x7fbb400b9d70 2022-06-29T15:58:34.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.017+0000 7fbb3cff9700 1 -- 172.21.15.60:0/2799352324 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb30011c10 con 0x7fbb400b9d70 2022-06-29T15:58:34.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.017+0000 7fbb45e95700 1 -- 172.21.15.60:0/2799352324 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb40140d30 con 0x7fbb400b9d70 2022-06-29T15:58:34.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.017+0000 7fbb45e95700 1 -- 172.21.15.60:0/2799352324 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb400573d0 con 0x7fbb400b9d70 2022-06-29T15:58:34.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.021+0000 7fbb3cff9700 1 -- 172.21.15.60:0/2799352324 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fbb30011d70 con 0x7fbb400b9d70 2022-06-29T15:58:34.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.021+0000 7fbb3cff9700 1 --2- 172.21.15.60:0/2799352324 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbb18053a70 0x7fbb18055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:34.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.021+0000 7fbb3cff9700 1 -- 172.21.15.60:0/2799352324 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbb3006c270 con 0x7fbb400b9d70 2022-06-29T15:58:34.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.021+0000 7fbb3f7fe700 1 --2- 172.21.15.60:0/2799352324 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbb18053a70 0x7fbb18055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:34.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.021+0000 7fbb3f7fe700 1 --2- 172.21.15.60:0/2799352324 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbb18053a70 0x7fbb18055f30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fbb3800d000 tx=0x7fbb38007f10).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:34.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.025+0000 7fbb3cff9700 1 -- 172.21.15.60:0/2799352324 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbb30043190 con 0x7fbb400b9d70 2022-06-29T15:58:34.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.149+0000 7fbb45e95700 1 -- 172.21.15.60:0/2799352324 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fbb4005da40 con 0x7fbb400b9d70 2022-06-29T15:58:34.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.153+0000 7fbb3cff9700 1 -- 172.21.15.60:0/2799352324 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v23) v1 ==== 110+0+10 (secure 0 0 0) 0x7fbb300467b0 con 0x7fbb400b9d70 2022-06-29T15:58:34.161 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T15:58:34.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.153+0000 7fbb45e95700 1 -- 172.21.15.60:0/2799352324 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbb18053a70 msgr2=0x7fbb18055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:34.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.153+0000 7fbb45e95700 1 --2- 172.21.15.60:0/2799352324 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbb18053a70 0x7fbb18055f30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fbb3800d000 tx=0x7fbb38007f10).stop 2022-06-29T15:58:34.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.153+0000 7fbb45e95700 1 -- 172.21.15.60:0/2799352324 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb400b9d70 msgr2=0x7fbb40113f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:34.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.153+0000 7fbb45e95700 1 --2- 172.21.15.60:0/2799352324 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb400b9d70 0x7fbb40113f50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fbb3000c000 tx=0x7fbb30006df0).stop 2022-06-29T15:58:34.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.153+0000 7fbb45e95700 1 -- 172.21.15.60:0/2799352324 shutdown_connections 2022-06-29T15:58:34.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.153+0000 7fbb45e95700 1 --2- 172.21.15.60:0/2799352324 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbb18053a70 0x7fbb18055f30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.153+0000 7fbb45e95700 1 --2- 172.21.15.60:0/2799352324 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb400b9d70 0x7fbb40113f50 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.153+0000 7fbb45e95700 1 --2- 172.21.15.60:0/2799352324 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbb400b3d30 0x7fbb40149950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.153+0000 7fbb45e95700 1 --2- 172.21.15.60:0/2799352324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb40061d90 0x7fbb40151d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.153+0000 7fbb45e95700 1 -- 172.21.15.60:0/2799352324 >> 172.21.15.60:0/2799352324 conn(0x7fbb40062e70 msgr2=0x7fbb4005c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:34.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.153+0000 7fbb45e95700 1 -- 172.21.15.60:0/2799352324 shutdown_connections 2022-06-29T15:58:34.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.153+0000 7fbb45e95700 1 -- 172.21.15.60:0/2799352324 wait complete. 2022-06-29T15:58:34.171 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T15:58:34.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.277+0000 7fc26f4cd700 1 Processor -- start 2022-06-29T15:58:34.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.277+0000 7fc26f4cd700 1 -- start start 2022-06-29T15:58:34.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.277+0000 7fc26f4cd700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc2680b4010 0x7fc2681426b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:34.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.277+0000 7fc26f4cd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc2680b46c0 0x7fc2680b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:34.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.277+0000 7fc26f4cd700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc268143a40 con 0x7fc2680ae7c0 2022-06-29T15:58:34.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.277+0000 7fc26f4cd700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2680aee60 con 0x7fc2680b46c0 2022-06-29T15:58:34.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.277+0000 7fc26f4cd700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2680af100 con 0x7fc2680b4010 2022-06-29T15:58:34.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.277+0000 7fc26da6a700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc2680b46c0 0x7fc2680b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:34.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.277+0000 7fc26da6a700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc2680b46c0 0x7fc2680b4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45432/0 (socket says 172.21.15.60:45432) 2022-06-29T15:58:34.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.277+0000 7fc26da6a700 1 -- 172.21.15.60:0/1512022212 learned_addr learned my addr 172.21.15.60:0/1512022212 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:34.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.277+0000 7fc26ca68700 1 --2- 172.21.15.60:0/1512022212 >> v2:172.21.15.92:3300/0 conn(0x7fc2680b4010 0x7fc2681426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:34.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.277+0000 7fc26da6a700 1 -- 172.21.15.60:0/1512022212 >> v2:172.21.15.92:3300/0 conn(0x7fc2680b4010 msgr2=0x7fc2681426b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:34.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.277+0000 7fc26da6a700 1 --2- 172.21.15.60:0/1512022212 >> v2:172.21.15.92:3300/0 conn(0x7fc2680b4010 0x7fc2681426b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.277+0000 7fc26da6a700 1 -- 172.21.15.60:0/1512022212 >> v1:172.21.15.92:6789/0 conn(0x7fc2680ae7c0 legacy=0x7fc268141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:34.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.277+0000 7fc26da6a700 1 -- 172.21.15.60:0/1512022212 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2680aeba0 con 0x7fc2680b46c0 2022-06-29T15:58:34.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.281+0000 7fc26da6a700 1 --2- 172.21.15.60:0/1512022212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2680b46c0 0x7fc2680b4af0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fc2600032f0 tx=0x7fc260009290).ready entity=mon.0 client_cookie=20d11e9cff1dd558 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:34.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.281+0000 7fc253fff700 1 -- 172.21.15.60:0/1512022212 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc26000b870 con 0x7fc2680b46c0 2022-06-29T15:58:34.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.281+0000 7fc253fff700 1 -- 172.21.15.60:0/1512022212 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc26000b9d0 con 0x7fc2680b46c0 2022-06-29T15:58:34.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.281+0000 7fc253fff700 1 -- 172.21.15.60:0/1512022212 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc26000bb40 con 0x7fc2680b46c0 2022-06-29T15:58:34.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.281+0000 7fc26f4cd700 1 -- 172.21.15.60:0/1512022212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2680b46c0 msgr2=0x7fc2680b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:34.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.281+0000 7fc26f4cd700 1 --2- 172.21.15.60:0/1512022212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2680b46c0 0x7fc2680b4af0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fc2600032f0 tx=0x7fc260009290).stop 2022-06-29T15:58:34.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.281+0000 7fc26f4cd700 1 -- 172.21.15.60:0/1512022212 shutdown_connections 2022-06-29T15:58:34.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.281+0000 7fc26f4cd700 1 --2- 172.21.15.60:0/1512022212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2680b46c0 0x7fc2680b4af0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.281+0000 7fc26f4cd700 1 --2- 172.21.15.60:0/1512022212 >> v2:172.21.15.92:3300/0 conn(0x7fc2680b4010 0x7fc2681426b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.281+0000 7fc26f4cd700 1 -- 172.21.15.60:0/1512022212 >> 172.21.15.60:0/1512022212 conn(0x7fc2680b9340 msgr2=0x7fc2680b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:34.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.281+0000 7fc26f4cd700 1 -- 172.21.15.60:0/1512022212 shutdown_connections 2022-06-29T15:58:34.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.281+0000 7fc26f4cd700 1 -- 172.21.15.60:0/1512022212 wait complete. 2022-06-29T15:58:34.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.285+0000 7fc26f4cd700 1 Processor -- start 2022-06-29T15:58:34.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.285+0000 7fc26f4cd700 1 -- start start 2022-06-29T15:58:34.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.285+0000 7fc26f4cd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2680ae7c0 0x7fc2681443a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:34.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.285+0000 7fc26f4cd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2680b4010 0x7fc26813ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:34.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.285+0000 7fc26f4cd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2680b46c0 0x7fc26813f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:34.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.285+0000 7fc26f4cd700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc268156340 con 0x7fc2680b46c0 2022-06-29T15:58:34.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.285+0000 7fc26f4cd700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2680af660 con 0x7fc2680b4010 2022-06-29T15:58:34.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.285+0000 7fc26f4cd700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2680af900 con 0x7fc2680ae7c0 2022-06-29T15:58:34.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.285+0000 7fc26da6a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2680b46c0 0x7fc26813f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:34.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.285+0000 7fc26da6a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2680b46c0 0x7fc26813f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45438/0 (socket says 172.21.15.60:45438) 2022-06-29T15:58:34.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.285+0000 7fc26d269700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2680ae7c0 0x7fc2681443a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:34.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.285+0000 7fc26da6a700 1 -- 172.21.15.60:0/709157103 learned_addr learned my addr 172.21.15.60:0/709157103 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:34.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.285+0000 7fc26ca68700 1 --2- 172.21.15.60:0/709157103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2680b4010 0x7fc26813ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:34.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.285+0000 7fc26da6a700 1 -- 172.21.15.60:0/709157103 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2680ae7c0 msgr2=0x7fc2681443a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:34.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.285+0000 7fc26da6a700 1 --2- 172.21.15.60:0/709157103 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2680ae7c0 0x7fc2681443a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.285+0000 7fc26da6a700 1 -- 172.21.15.60:0/709157103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2680b4010 msgr2=0x7fc26813ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:34.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.285+0000 7fc26da6a700 1 --2- 172.21.15.60:0/709157103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2680b4010 0x7fc26813ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.285+0000 7fc26da6a700 1 -- 172.21.15.60:0/709157103 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc260009530 con 0x7fc2680b46c0 2022-06-29T15:58:34.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.285+0000 7fc26da6a700 1 --2- 172.21.15.60:0/709157103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2680b46c0 0x7fc26813f120 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fc2600032f0 tx=0x7fc260016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:34.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.289+0000 7fc25affd700 1 -- 172.21.15.60:0/709157103 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc26000b870 con 0x7fc2680b46c0 2022-06-29T15:58:34.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.289+0000 7fc25affd700 1 -- 172.21.15.60:0/709157103 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc260021440 con 0x7fc2680b46c0 2022-06-29T15:58:34.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.289+0000 7fc26f4cd700 1 -- 172.21.15.60:0/709157103 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc26813f610 con 0x7fc2680b46c0 2022-06-29T15:58:34.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.289+0000 7fc25affd700 1 -- 172.21.15.60:0/709157103 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2600217a0 con 0x7fc2680b46c0 2022-06-29T15:58:34.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.289+0000 7fc26f4cd700 1 -- 172.21.15.60:0/709157103 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2680b01a0 con 0x7fc2680b46c0 2022-06-29T15:58:34.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.289+0000 7fc26f4cd700 1 -- 172.21.15.60:0/709157103 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2680573d0 con 0x7fc2680b46c0 2022-06-29T15:58:34.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.289+0000 7fc25affd700 1 -- 172.21.15.60:0/709157103 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc260029070 con 0x7fc2680b46c0 2022-06-29T15:58:34.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.289+0000 7fc25affd700 1 --2- 172.21.15.60:0/709157103 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc244053b00 0x7fc244055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:34.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.289+0000 7fc25affd700 1 -- 172.21.15.60:0/709157103 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc26006cfa0 con 0x7fc2680b46c0 2022-06-29T15:58:34.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.289+0000 7fc26d269700 1 --2- 172.21.15.60:0/709157103 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc244053b00 0x7fc244055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:34.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.289+0000 7fc26d269700 1 --2- 172.21.15.60:0/709157103 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc244053b00 0x7fc244055fc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fc25c00d000 tx=0x7fc25c000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:34.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.293+0000 7fc25affd700 1 -- 172.21.15.60:0/709157103 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc260043e20 con 0x7fc2680b46c0 2022-06-29T15:58:34.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.421+0000 7fc26f4cd700 1 -- 172.21.15.60:0/709157103 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fc26805fdb0 con 0x7fc2680b46c0 2022-06-29T15:58:34.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.421+0000 7fc25affd700 1 -- 172.21.15.60:0/709157103 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc260015030 con 0x7fc2680b46c0 2022-06-29T15:58:34.428 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T15:58:34.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.425+0000 7fc26f4cd700 1 -- 172.21.15.60:0/709157103 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc244053b00 msgr2=0x7fc244055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:34.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.425+0000 7fc26f4cd700 1 --2- 172.21.15.60:0/709157103 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc244053b00 0x7fc244055fc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fc25c00d000 tx=0x7fc25c000bc0).stop 2022-06-29T15:58:34.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.425+0000 7fc26f4cd700 1 -- 172.21.15.60:0/709157103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2680b46c0 msgr2=0x7fc26813f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:34.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.425+0000 7fc26f4cd700 1 --2- 172.21.15.60:0/709157103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2680b46c0 0x7fc26813f120 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fc2600032f0 tx=0x7fc260016000).stop 2022-06-29T15:58:34.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.425+0000 7fc26f4cd700 1 -- 172.21.15.60:0/709157103 shutdown_connections 2022-06-29T15:58:34.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.425+0000 7fc26f4cd700 1 --2- 172.21.15.60:0/709157103 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc244053b00 0x7fc244055fc0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.425+0000 7fc26f4cd700 1 --2- 172.21.15.60:0/709157103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2680b46c0 0x7fc26813f120 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.425+0000 7fc26f4cd700 1 --2- 172.21.15.60:0/709157103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2680b4010 0x7fc26813ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.425+0000 7fc26f4cd700 1 --2- 172.21.15.60:0/709157103 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2680ae7c0 0x7fc2681443a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.425+0000 7fc26f4cd700 1 -- 172.21.15.60:0/709157103 >> 172.21.15.60:0/709157103 conn(0x7fc2680b9340 msgr2=0x7fc268141180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:34.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.425+0000 7fc26f4cd700 1 -- 172.21.15.60:0/709157103 shutdown_connections 2022-06-29T15:58:34.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.425+0000 7fc26f4cd700 1 -- 172.21.15.60:0/709157103 wait complete. 2022-06-29T15:58:34.442 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-29T15:58:34.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.549+0000 7f62395cd700 1 Processor -- start 2022-06-29T15:58:34.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.549+0000 7f62395cd700 1 -- start start 2022-06-29T15:58:34.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.549+0000 7f62395cd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62340b5950 0x7f6234067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:34.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.549+0000 7f62395cd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f623414af10 0x7f623405c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:34.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.549+0000 7f62395cd700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62340acba0 con 0x7f6234063ad0 2022-06-29T15:58:34.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.549+0000 7f62395cd700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f623414b3a0 con 0x7f623414af10 2022-06-29T15:58:34.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.549+0000 7f62395cd700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6234067220 con 0x7f62340b5950 2022-06-29T15:58:34.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.549+0000 7f62337fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f623414af10 0x7f623405c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:34.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.549+0000 7f62337fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f623414af10 0x7f623405c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45446/0 (socket says 172.21.15.60:45446) 2022-06-29T15:58:34.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.549+0000 7f62337fe700 1 -- 172.21.15.60:0/1578380848 learned_addr learned my addr 172.21.15.60:0/1578380848 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:34.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.549+0000 7f6232ffd700 1 --2- 172.21.15.60:0/1578380848 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62340b5950 0x7f6234067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:34.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.549+0000 7f62337fe700 1 -- 172.21.15.60:0/1578380848 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62340b5950 msgr2=0x7f6234067680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:34.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.549+0000 7f62337fe700 1 --2- 172.21.15.60:0/1578380848 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62340b5950 0x7f6234067680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.549+0000 7f62337fe700 1 -- 172.21.15.60:0/1578380848 >> v1:172.21.15.92:6789/0 conn(0x7f6234063ad0 legacy=0x7f623405bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:34.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.549+0000 7f62337fe700 1 -- 172.21.15.60:0/1578380848 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62340b5d30 con 0x7f623414af10 2022-06-29T15:58:34.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.549+0000 7f62337fe700 1 --2- 172.21.15.60:0/1578380848 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f623414af10 0x7f623405c430 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f6224002f70 tx=0x7f6224006eb0).ready entity=mon.0 client_cookie=1921bee98873076d server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:34.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.553+0000 7f6231ffb700 1 -- 172.21.15.60:0/1578380848 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f622400d570 con 0x7f623414af10 2022-06-29T15:58:34.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.553+0000 7f6231ffb700 1 -- 172.21.15.60:0/1578380848 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6224000e90 con 0x7f623414af10 2022-06-29T15:58:34.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.553+0000 7f6231ffb700 1 -- 172.21.15.60:0/1578380848 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f622400d900 con 0x7f623414af10 2022-06-29T15:58:34.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.553+0000 7f62395cd700 1 -- 172.21.15.60:0/1578380848 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f623414af10 msgr2=0x7f623405c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:34.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.553+0000 7f62395cd700 1 --2- 172.21.15.60:0/1578380848 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f623414af10 0x7f623405c430 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f6224002f70 tx=0x7f6224006eb0).stop 2022-06-29T15:58:34.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.553+0000 7f62395cd700 1 -- 172.21.15.60:0/1578380848 shutdown_connections 2022-06-29T15:58:34.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.553+0000 7f62395cd700 1 --2- 172.21.15.60:0/1578380848 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f623414af10 0x7f623405c430 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.553+0000 7f62395cd700 1 --2- 172.21.15.60:0/1578380848 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62340b5950 0x7f6234067680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.553+0000 7f62395cd700 1 -- 172.21.15.60:0/1578380848 >> 172.21.15.60:0/1578380848 conn(0x7f62340632c0 msgr2=0x7f62340b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:34.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.553+0000 7f62395cd700 1 -- 172.21.15.60:0/1578380848 shutdown_connections 2022-06-29T15:58:34.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.553+0000 7f62395cd700 1 -- 172.21.15.60:0/1578380848 wait complete. 2022-06-29T15:58:34.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.553+0000 7f62395cd700 1 Processor -- start 2022-06-29T15:58:34.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.553+0000 7f62395cd700 1 -- start start 2022-06-29T15:58:34.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.553+0000 7f62395cd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6234063ad0 0x7f6234065260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:34.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.553+0000 7f62395cd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62340b5950 0x7f6234148c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:34.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.553+0000 7f62395cd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f623414af10 0x7f62340b0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:34.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.553+0000 7f62395cd700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f623415a0d0 con 0x7f623414af10 2022-06-29T15:58:34.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.553+0000 7f62395cd700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f623415a370 con 0x7f62340b5950 2022-06-29T15:58:34.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.553+0000 7f62395cd700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62340ad740 con 0x7f6234063ad0 2022-06-29T15:58:34.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.557+0000 7f62327fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62340b5950 0x7f6234148c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:34.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.557+0000 7f62337fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f623414af10 0x7f62340b0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:34.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.557+0000 7f62327fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62340b5950 0x7f6234148c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:51480/0 (socket says 172.21.15.60:51480) 2022-06-29T15:58:34.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.557+0000 7f62327fc700 1 -- 172.21.15.60:0/382964726 learned_addr learned my addr 172.21.15.60:0/382964726 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:34.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.557+0000 7f6232ffd700 1 --2- 172.21.15.60:0/382964726 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6234063ad0 0x7f6234065260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:34.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.557+0000 7f62327fc700 1 -- 172.21.15.60:0/382964726 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6234063ad0 msgr2=0x7f6234065260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:34.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.557+0000 7f62327fc700 1 --2- 172.21.15.60:0/382964726 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6234063ad0 0x7f6234065260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.557+0000 7f62327fc700 1 -- 172.21.15.60:0/382964726 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f623414af10 msgr2=0x7f62340b0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:34.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.557+0000 7f62327fc700 1 --2- 172.21.15.60:0/382964726 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f623414af10 0x7f62340b0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.557+0000 7f62327fc700 1 -- 172.21.15.60:0/382964726 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f622400b1e0 con 0x7f62340b5950 2022-06-29T15:58:34.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.557+0000 7f62337fe700 1 --2- 172.21.15.60:0/382964726 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f623414af10 0x7f62340b0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:34.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.557+0000 7f62327fc700 1 --2- 172.21.15.60:0/382964726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62340b5950 0x7f6234148c30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f623405ff90 tx=0x7f621c007c50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:34.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.557+0000 7f622bfff700 1 -- 172.21.15.60:0/382964726 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f621c012400 con 0x7f62340b5950 2022-06-29T15:58:34.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.557+0000 7f622bfff700 1 -- 172.21.15.60:0/382964726 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f621c012560 con 0x7f62340b5950 2022-06-29T15:58:34.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.557+0000 7f62395cd700 1 -- 172.21.15.60:0/382964726 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6234149120 con 0x7f62340b5950 2022-06-29T15:58:34.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.557+0000 7f622bfff700 1 -- 172.21.15.60:0/382964726 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f621c0128c0 con 0x7f62340b5950 2022-06-29T15:58:34.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.557+0000 7f62395cd700 1 -- 172.21.15.60:0/382964726 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62340ad220 con 0x7f62340b5950 2022-06-29T15:58:34.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.557+0000 7f62395cd700 1 -- 172.21.15.60:0/382964726 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6234159f40 con 0x7f62340b5950 2022-06-29T15:58:34.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.557+0000 7f622bfff700 1 -- 172.21.15.60:0/382964726 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f621c019030 con 0x7f62340b5950 2022-06-29T15:58:34.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.561+0000 7f622bfff700 1 --2- 172.21.15.60:0/382964726 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6218053b00 0x7f6218055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:34.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.561+0000 7f622bfff700 1 -- 172.21.15.60:0/382964726 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f621c06b310 con 0x7f62340b5950 2022-06-29T15:58:34.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.561+0000 7f6232ffd700 1 --2- 172.21.15.60:0/382964726 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6218053b00 0x7f6218055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:34.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.561+0000 7f6232ffd700 1 --2- 172.21.15.60:0/382964726 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6218053b00 0x7f6218055fc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f622c00b000 tx=0x7f622c0091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:34.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.565+0000 7f622bfff700 1 -- 172.21.15.60:0/382964726 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f621c042120 con 0x7f62340b5950 2022-06-29T15:58:34.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.697+0000 7f62395cd700 1 -- 172.21.15.60:0/382964726 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f62340573d0 con 0x7f62340b5950 2022-06-29T15:58:34.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.697+0000 7f622bfff700 1 -- 172.21.15.60:0/382964726 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7f621c045740 con 0x7f62340b5950 2022-06-29T15:58:34.704 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T15:58:34.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.701+0000 7f62395cd700 1 -- 172.21.15.60:0/382964726 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6218053b00 msgr2=0x7f6218055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:34.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.701+0000 7f62395cd700 1 --2- 172.21.15.60:0/382964726 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6218053b00 0x7f6218055fc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f622c00b000 tx=0x7f622c0091d0).stop 2022-06-29T15:58:34.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.701+0000 7f62395cd700 1 -- 172.21.15.60:0/382964726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62340b5950 msgr2=0x7f6234148c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:34.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.701+0000 7f62395cd700 1 --2- 172.21.15.60:0/382964726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62340b5950 0x7f6234148c30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f623405ff90 tx=0x7f621c007c50).stop 2022-06-29T15:58:34.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.701+0000 7f62395cd700 1 -- 172.21.15.60:0/382964726 shutdown_connections 2022-06-29T15:58:34.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.701+0000 7f62395cd700 1 --2- 172.21.15.60:0/382964726 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6218053b00 0x7f6218055fc0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.701+0000 7f62395cd700 1 --2- 172.21.15.60:0/382964726 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f623414af10 0x7f62340b0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.701+0000 7f62395cd700 1 --2- 172.21.15.60:0/382964726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62340b5950 0x7f6234148c30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.701+0000 7f62395cd700 1 --2- 172.21.15.60:0/382964726 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6234063ad0 0x7f6234065260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.701+0000 7f62395cd700 1 -- 172.21.15.60:0/382964726 >> 172.21.15.60:0/382964726 conn(0x7f62340632c0 msgr2=0x7f62340b4ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:34.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.701+0000 7f62395cd700 1 -- 172.21.15.60:0/382964726 shutdown_connections 2022-06-29T15:58:34.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.701+0000 7f62395cd700 1 -- 172.21.15.60:0/382964726 wait complete. 2022-06-29T15:58:34.723 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-29T15:58:34.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.829+0000 7fd9b91cb700 1 Processor -- start 2022-06-29T15:58:34.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.829+0000 7fd9b91cb700 1 -- start start 2022-06-29T15:58:34.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.829+0000 7fd9b91cb700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd9b40b6a50 0x7fd9b405fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:34.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.829+0000 7fd9b91cb700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9b4062230 0x7fd9b41465e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:34.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.829+0000 7fd9b91cb700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd9b405f0b0 0x7fd9b4146ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:34.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.829+0000 7fd9b91cb700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9b4146fc0 con 0x7fd9b40b6a50 2022-06-29T15:58:34.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.829+0000 7fd9b91cb700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9b40b6e30 con 0x7fd9b405f0b0 2022-06-29T15:58:34.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.829+0000 7fd9b91cb700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b4098970 con 0x7fd9b4062230 2022-06-29T15:58:34.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.829+0000 7fd9b2d9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd9b40b6a50 0x7fd9b405fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:34.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.829+0000 7fd9b2d9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd9b40b6a50 0x7fd9b405fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45456/0 (socket says 172.21.15.60:45456) 2022-06-29T15:58:34.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.829+0000 7fd9b2d9d700 1 -- 172.21.15.60:0/1527312337 learned_addr learned my addr 172.21.15.60:0/1527312337 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:34.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.829+0000 7fd9b359e700 1 --2- 172.21.15.60:0/1527312337 >> v2:172.21.15.92:3300/0 conn(0x7fd9b405f0b0 0x7fd9b4146ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:34.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.829+0000 7fd9aa59c700 1 --2- 172.21.15.60:0/1527312337 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9b4062230 0x7fd9b41465e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:34.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.829+0000 7fd9b2d9d700 1 -- 172.21.15.60:0/1527312337 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9b4062230 msgr2=0x7fd9b41465e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:34.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.829+0000 7fd9b2d9d700 1 --2- 172.21.15.60:0/1527312337 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9b4062230 0x7fd9b41465e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.829+0000 7fd9b2d9d700 1 -- 172.21.15.60:0/1527312337 >> v2:172.21.15.92:3300/0 conn(0x7fd9b405f0b0 msgr2=0x7fd9b4146ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:34.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.829+0000 7fd9b2d9d700 1 --2- 172.21.15.60:0/1527312337 >> v2:172.21.15.92:3300/0 conn(0x7fd9b405f0b0 0x7fd9b4146ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.829+0000 7fd9b2d9d700 1 -- 172.21.15.60:0/1527312337 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b405fa40 con 0x7fd9b40b6a50 2022-06-29T15:58:34.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.829+0000 7fd9b359e700 1 --2- 172.21.15.60:0/1527312337 >> v2:172.21.15.92:3300/0 conn(0x7fd9b405f0b0 0x7fd9b4146ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:34.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.829+0000 7fd9b2d9d700 1 --2- 172.21.15.60:0/1527312337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9b40b6a50 0x7fd9b405fee0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fd9a4002f70 tx=0x7fd9a4006eb0).ready entity=mon.0 client_cookie=4dfd1fb512661fea server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:34.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.829+0000 7fd9b259c700 1 -- 172.21.15.60:0/1527312337 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9a400d570 con 0x7fd9b40b6a50 2022-06-29T15:58:34.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.829+0000 7fd9b259c700 1 -- 172.21.15.60:0/1527312337 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a4000e90 con 0x7fd9b40b6a50 2022-06-29T15:58:34.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.829+0000 7fd9b259c700 1 -- 172.21.15.60:0/1527312337 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9a400d900 con 0x7fd9b40b6a50 2022-06-29T15:58:34.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.833+0000 7fd9b91cb700 1 -- 172.21.15.60:0/1527312337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9b40b6a50 msgr2=0x7fd9b405fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:34.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.833+0000 7fd9b91cb700 1 --2- 172.21.15.60:0/1527312337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9b40b6a50 0x7fd9b405fee0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fd9a4002f70 tx=0x7fd9a4006eb0).stop 2022-06-29T15:58:34.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.833+0000 7fd9b91cb700 1 -- 172.21.15.60:0/1527312337 shutdown_connections 2022-06-29T15:58:34.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.833+0000 7fd9b91cb700 1 --2- 172.21.15.60:0/1527312337 >> v2:172.21.15.92:3300/0 conn(0x7fd9b405f0b0 0x7fd9b4146ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.833+0000 7fd9b91cb700 1 --2- 172.21.15.60:0/1527312337 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9b4062230 0x7fd9b41465e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.833+0000 7fd9b91cb700 1 --2- 172.21.15.60:0/1527312337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9b40b6a50 0x7fd9b405fee0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.833+0000 7fd9b91cb700 1 -- 172.21.15.60:0/1527312337 >> 172.21.15.60:0/1527312337 conn(0x7fd9b40643e0 msgr2=0x7fd9b40b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:34.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.833+0000 7fd9b91cb700 1 -- 172.21.15.60:0/1527312337 shutdown_connections 2022-06-29T15:58:34.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.833+0000 7fd9b91cb700 1 -- 172.21.15.60:0/1527312337 wait complete. 2022-06-29T15:58:34.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.833+0000 7fd9b91cb700 1 Processor -- start 2022-06-29T15:58:34.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.833+0000 7fd9b91cb700 1 -- start start 2022-06-29T15:58:34.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.833+0000 7fd9b91cb700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9b405f0b0 0x7fd9b4140cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:34.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.833+0000 7fd9b91cb700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9b4062230 0x7fd9b4064f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:34.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.833+0000 7fd9b91cb700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9b40b6a50 0x7fd9b4065430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:34.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.833+0000 7fd9b91cb700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b40ade30 con 0x7fd9b405f0b0 2022-06-29T15:58:34.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.833+0000 7fd9b91cb700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b40ad650 con 0x7fd9b40b6a50 2022-06-29T15:58:34.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.833+0000 7fd9b91cb700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b40ad8f0 con 0x7fd9b4062230 2022-06-29T15:58:34.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.833+0000 7fd9b2d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9b405f0b0 0x7fd9b4140cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:34.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.833+0000 7fd9b2d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9b405f0b0 0x7fd9b4140cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45462/0 (socket says 172.21.15.60:45462) 2022-06-29T15:58:34.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.833+0000 7fd9b2d9d700 1 -- 172.21.15.60:0/4204133542 learned_addr learned my addr 172.21.15.60:0/4204133542 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:34.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.833+0000 7fd9b359e700 1 --2- 172.21.15.60:0/4204133542 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9b40b6a50 0x7fd9b4065430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:34.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.833+0000 7fd9aa59c700 1 --2- 172.21.15.60:0/4204133542 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9b4062230 0x7fd9b4064f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:34.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.837+0000 7fd9b2d9d700 1 -- 172.21.15.60:0/4204133542 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9b4062230 msgr2=0x7fd9b4064f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:34.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.837+0000 7fd9b2d9d700 1 --2- 172.21.15.60:0/4204133542 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9b4062230 0x7fd9b4064f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.837+0000 7fd9b2d9d700 1 -- 172.21.15.60:0/4204133542 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9b40b6a50 msgr2=0x7fd9b4065430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:34.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.837+0000 7fd9b2d9d700 1 --2- 172.21.15.60:0/4204133542 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9b40b6a50 0x7fd9b4065430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.837+0000 7fd9b2d9d700 1 -- 172.21.15.60:0/4204133542 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ac009470 con 0x7fd9b405f0b0 2022-06-29T15:58:34.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.837+0000 7fd9aa59c700 1 --2- 172.21.15.60:0/4204133542 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9b4062230 0x7fd9b4064f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:58:34.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.837+0000 7fd9b2d9d700 1 --2- 172.21.15.60:0/4204133542 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9b405f0b0 0x7fd9b4140cc0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fd9a4002f70 tx=0x7fd9a4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:34.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.837+0000 7fd9b0d99700 1 -- 172.21.15.60:0/4204133542 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9a400dea0 con 0x7fd9b405f0b0 2022-06-29T15:58:34.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.837+0000 7fd9b0d99700 1 -- 172.21.15.60:0/4204133542 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a4011ad0 con 0x7fd9b405f0b0 2022-06-29T15:58:34.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.837+0000 7fd9b91cb700 1 -- 172.21.15.60:0/4204133542 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9a400b1e0 con 0x7fd9b405f0b0 2022-06-29T15:58:34.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.837+0000 7fd9b0d99700 1 -- 172.21.15.60:0/4204133542 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9a4011e30 con 0x7fd9b405f0b0 2022-06-29T15:58:34.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.837+0000 7fd9b91cb700 1 -- 172.21.15.60:0/4204133542 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9b4065920 con 0x7fd9b405f0b0 2022-06-29T15:58:34.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.837+0000 7fd9b91cb700 1 -- 172.21.15.60:0/4204133542 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9b40573d0 con 0x7fd9b405f0b0 2022-06-29T15:58:34.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.837+0000 7fd9b0d99700 1 -- 172.21.15.60:0/4204133542 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd9a4025030 con 0x7fd9b405f0b0 2022-06-29T15:58:34.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.837+0000 7fd9b0d99700 1 --2- 172.21.15.60:0/4204133542 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd998053a70 0x7fd998055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:34.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.837+0000 7fd9b0d99700 1 -- 172.21.15.60:0/4204133542 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd9a406cec0 con 0x7fd9b405f0b0 2022-06-29T15:58:34.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.841+0000 7fd9aa59c700 1 --2- 172.21.15.60:0/4204133542 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd998053a70 0x7fd998055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:34.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.841+0000 7fd9aa59c700 1 --2- 172.21.15.60:0/4204133542 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd998053a70 0x7fd998055f30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fd99c00b000 tx=0x7fd99c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:34.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.845+0000 7fd9b0d99700 1 -- 172.21.15.60:0/4204133542 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd9a4043cd0 con 0x7fd9b405f0b0 2022-06-29T15:58:34.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.977+0000 7fd9b91cb700 1 -- 172.21.15.60:0/4204133542 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd9b4067990 con 0x7fd9b405f0b0 2022-06-29T15:58:34.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.977+0000 7fd9b0d99700 1 -- 172.21.15.60:0/4204133542 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7fd9a4023020 con 0x7fd9b405f0b0 2022-06-29T15:58:34.982 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T15:58:34.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.977+0000 7fd9b91cb700 1 -- 172.21.15.60:0/4204133542 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd998053a70 msgr2=0x7fd998055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:34.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.977+0000 7fd9b91cb700 1 --2- 172.21.15.60:0/4204133542 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd998053a70 0x7fd998055f30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fd99c00b000 tx=0x7fd99c008040).stop 2022-06-29T15:58:34.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.977+0000 7fd9b91cb700 1 -- 172.21.15.60:0/4204133542 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9b405f0b0 msgr2=0x7fd9b4140cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:34.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.977+0000 7fd9b91cb700 1 --2- 172.21.15.60:0/4204133542 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9b405f0b0 0x7fd9b4140cc0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fd9a4002f70 tx=0x7fd9a4015000).stop 2022-06-29T15:58:34.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.977+0000 7fd9b91cb700 1 -- 172.21.15.60:0/4204133542 shutdown_connections 2022-06-29T15:58:34.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.977+0000 7fd9b91cb700 1 --2- 172.21.15.60:0/4204133542 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd998053a70 0x7fd998055f30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.977+0000 7fd9b91cb700 1 --2- 172.21.15.60:0/4204133542 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9b40b6a50 0x7fd9b4065430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.977+0000 7fd9b91cb700 1 --2- 172.21.15.60:0/4204133542 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd9b4062230 0x7fd9b4064f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.977+0000 7fd9b91cb700 1 --2- 172.21.15.60:0/4204133542 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9b405f0b0 0x7fd9b4140cc0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:34.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.977+0000 7fd9b91cb700 1 -- 172.21.15.60:0/4204133542 >> 172.21.15.60:0/4204133542 conn(0x7fd9b40643e0 msgr2=0x7fd9b40ac300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:34.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.977+0000 7fd9b91cb700 1 -- 172.21.15.60:0/4204133542 shutdown_connections 2022-06-29T15:58:34.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:34.977+0000 7fd9b91cb700 1 -- 172.21.15.60:0/4204133542 wait complete. 2022-06-29T15:58:34.996 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T15:58:35.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.093+0000 7f081da35700 1 Processor -- start 2022-06-29T15:58:35.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.093+0000 7f081da35700 1 -- start start 2022-06-29T15:58:35.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.093+0000 7f081da35700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f081805d070 0x7f081814aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:35.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.093+0000 7f081da35700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f081805d450 con 0x7f081814af80 2022-06-29T15:58:35.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.093+0000 7f081da35700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08180ad840 con 0x7f08180b2130 2022-06-29T15:58:35.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.093+0000 7f081da35700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08180adf90 con 0x7f081805d070 2022-06-29T15:58:35.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.093+0000 7f081d234700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f081814af80 0x7f08180ad130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53560/0 (socket says 172.21.15.60:53560) 2022-06-29T15:58:35.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.093+0000 7f081d234700 1 -- 172.21.15.60:0/1294195595 learned_addr learned my addr 172.21.15.60:0/1294195595 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:35.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.093+0000 7f081ca33700 1 --2- 172.21.15.60:0/1294195595 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f081805d070 0x7f081814aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:35.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.093+0000 7f081ca33700 1 -- 172.21.15.60:0/1294195595 >> v1:172.21.15.92:6789/0 conn(0x7f08180b2130 legacy=0x7f08180aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:35.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.093+0000 7f08177fe700 1 -- 172.21.15.60:0/1294195595 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828083318 0 0) 0x7f081805d450 con 0x7f081814af80 2022-06-29T15:58:35.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.093+0000 7f081ca33700 1 -- 172.21.15.60:0/1294195595 >> v1:172.21.15.60:6789/0 conn(0x7f081814af80 legacy=0x7f08180ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:35.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.093+0000 7f081ca33700 1 -- 172.21.15.60:0/1294195595 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08180b25d0 con 0x7f081805d070 2022-06-29T15:58:35.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.093+0000 7f081ca33700 1 --2- 172.21.15.60:0/1294195595 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f081805d070 0x7f081814aa90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f0800002f70 tx=0x7f0800006df0).ready entity=mon.2 client_cookie=3fd0c8e5c928893e server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:35.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.093+0000 7f08177fe700 1 -- 172.21.15.60:0/1294195595 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f080000d5c0 con 0x7f081805d070 2022-06-29T15:58:35.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.093+0000 7f08177fe700 1 -- 172.21.15.60:0/1294195595 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0800000e90 con 0x7f081805d070 2022-06-29T15:58:35.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.093+0000 7f08177fe700 1 -- 172.21.15.60:0/1294195595 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f080000d910 con 0x7f081805d070 2022-06-29T15:58:35.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.097+0000 7f081da35700 1 -- 172.21.15.60:0/1294195595 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f081805d070 msgr2=0x7f081814aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:35.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.097+0000 7f081da35700 1 --2- 172.21.15.60:0/1294195595 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f081805d070 0x7f081814aa90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f0800002f70 tx=0x7f0800006df0).stop 2022-06-29T15:58:35.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.097+0000 7f081da35700 1 -- 172.21.15.60:0/1294195595 shutdown_connections 2022-06-29T15:58:35.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.097+0000 7f081da35700 1 --2- 172.21.15.60:0/1294195595 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f081805d070 0x7f081814aa90 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:35.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.097+0000 7f081da35700 1 -- 172.21.15.60:0/1294195595 >> 172.21.15.60:0/1294195595 conn(0x7f08180b5890 msgr2=0x7f08180b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:35.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.097+0000 7f081da35700 1 -- 172.21.15.60:0/1294195595 shutdown_connections 2022-06-29T15:58:35.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.097+0000 7f081da35700 1 -- 172.21.15.60:0/1294195595 wait complete. 2022-06-29T15:58:35.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.097+0000 7f081da35700 1 Processor -- start 2022-06-29T15:58:35.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.097+0000 7f081da35700 1 -- start start 2022-06-29T15:58:35.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.097+0000 7f081da35700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f081805d070 0x7f0818066750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:35.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.097+0000 7f081da35700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08180b2130 0x7f08180b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:35.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.097+0000 7f081da35700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f081814af80 0x7f08180b46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:35.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.101+0000 7f081da35700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f081815a6d0 con 0x7f081814af80 2022-06-29T15:58:35.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.101+0000 7f081da35700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0818064730 con 0x7f08180b2130 2022-06-29T15:58:35.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.101+0000 7f081da35700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f081815a1d0 con 0x7f081805d070 2022-06-29T15:58:35.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.101+0000 7f081d234700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f081814af80 0x7f08180b46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:35.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.101+0000 7f081d234700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f081814af80 0x7f08180b46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45480/0 (socket says 172.21.15.60:45480) 2022-06-29T15:58:35.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.101+0000 7f081d234700 1 -- 172.21.15.60:0/2072092427 learned_addr learned my addr 172.21.15.60:0/2072092427 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:35.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.101+0000 7f081ca33700 1 --2- 172.21.15.60:0/2072092427 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f081805d070 0x7f0818066750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:35.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.101+0000 7f0817fff700 1 --2- 172.21.15.60:0/2072092427 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08180b2130 0x7f08180b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:35.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.101+0000 7f081d234700 1 -- 172.21.15.60:0/2072092427 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f081805d070 msgr2=0x7f0818066750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:35.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.101+0000 7f081d234700 1 --2- 172.21.15.60:0/2072092427 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f081805d070 0x7f0818066750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:35.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.101+0000 7f081d234700 1 -- 172.21.15.60:0/2072092427 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08180b2130 msgr2=0x7f08180b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:35.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.101+0000 7f081d234700 1 --2- 172.21.15.60:0/2072092427 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08180b2130 0x7f08180b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:35.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.101+0000 7f081d234700 1 -- 172.21.15.60:0/2072092427 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f080000b180 con 0x7f081814af80 2022-06-29T15:58:35.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.101+0000 7f081d234700 1 --2- 172.21.15.60:0/2072092427 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f081814af80 0x7f08180b46e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f0810000bc0 tx=0x7f0810007980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:35.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.101+0000 7f0815ffb700 1 -- 172.21.15.60:0/2072092427 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f081000cda0 con 0x7f081814af80 2022-06-29T15:58:35.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.101+0000 7f0815ffb700 1 -- 172.21.15.60:0/2072092427 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08100070d0 con 0x7f081814af80 2022-06-29T15:58:35.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.101+0000 7f0815ffb700 1 -- 172.21.15.60:0/2072092427 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0810007430 con 0x7f081814af80 2022-06-29T15:58:35.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.101+0000 7f081da35700 1 -- 172.21.15.60:0/2072092427 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f081815a430 con 0x7f081814af80 2022-06-29T15:58:35.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.101+0000 7f081da35700 1 -- 172.21.15.60:0/2072092427 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0818066d60 con 0x7f081814af80 2022-06-29T15:58:35.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.101+0000 7f081da35700 1 -- 172.21.15.60:0/2072092427 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f081814a350 con 0x7f081814af80 2022-06-29T15:58:35.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.101+0000 7f0815ffb700 1 -- 172.21.15.60:0/2072092427 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0810010070 con 0x7f081814af80 2022-06-29T15:58:35.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.101+0000 7f0815ffb700 1 --2- 172.21.15.60:0/2072092427 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f07f0053a70 0x7f07f0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:35.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.105+0000 7f0815ffb700 1 -- 172.21.15.60:0/2072092427 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f081006b420 con 0x7f081814af80 2022-06-29T15:58:35.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.105+0000 7f081ca33700 1 --2- 172.21.15.60:0/2072092427 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f07f0053a70 0x7f07f0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:35.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.105+0000 7f081ca33700 1 --2- 172.21.15.60:0/2072092427 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f07f0053a70 0x7f07f0055f30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f080001c000 tx=0x7f0800015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:35.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.109+0000 7f0815ffb700 1 -- 172.21.15.60:0/2072092427 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0810042340 con 0x7f081814af80 2022-06-29T15:58:35.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.265+0000 7f081da35700 1 -- 172.21.15.60:0/2072092427 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08180573d0 con 0x7f081814af80 2022-06-29T15:58:35.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.265+0000 7f0815ffb700 1 -- 172.21.15.60:0/2072092427 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2134 (secure 0 0 0) 0x7f0810045960 con 0x7f081814af80 2022-06-29T15:58:35.272 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:58:35.272 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":4,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.92:6835/2121698334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2121698334},{"type":"v1","addr":"172.21.15.92:6835","nonce":2121698334}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":4,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T15:58:24.334770+0000","modified":"2022-06-29T15:58:25.338504+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"a","rank":0,"incarnation":3,"state":"up:active","state_seq":2,"addr":"172.21.15.60:6833/3164364318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3164364318},{"type":"v1","addr":"172.21.15.60:6833","nonce":3164364318}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-29T15:58:35.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.269+0000 7f081da35700 1 -- 172.21.15.60:0/2072092427 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f07f0053a70 msgr2=0x7f07f0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:35.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.269+0000 7f081da35700 1 --2- 172.21.15.60:0/2072092427 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f07f0053a70 0x7f07f0055f30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f080001c000 tx=0x7f0800015000).stop 2022-06-29T15:58:35.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.269+0000 7f081da35700 1 -- 172.21.15.60:0/2072092427 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f081814af80 msgr2=0x7f08180b46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:35.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.269+0000 7f081da35700 1 --2- 172.21.15.60:0/2072092427 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f081814af80 0x7f08180b46e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f0810000bc0 tx=0x7f0810007980).stop 2022-06-29T15:58:35.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.269+0000 7f081da35700 1 -- 172.21.15.60:0/2072092427 shutdown_connections 2022-06-29T15:58:35.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.269+0000 7f081da35700 1 --2- 172.21.15.60:0/2072092427 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f07f0053a70 0x7f07f0055f30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:35.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.269+0000 7f081da35700 1 --2- 172.21.15.60:0/2072092427 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f081814af80 0x7f08180b46e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:35.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.269+0000 7f081da35700 1 --2- 172.21.15.60:0/2072092427 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08180b2130 0x7f08180b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:35.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.269+0000 7f081da35700 1 --2- 172.21.15.60:0/2072092427 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f081805d070 0x7f0818066750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:35.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.269+0000 7f081da35700 1 -- 172.21.15.60:0/2072092427 >> 172.21.15.60:0/2072092427 conn(0x7f08180b5890 msgr2=0x7f08181470e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:35.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.269+0000 7f081da35700 1 -- 172.21.15.60:0/2072092427 shutdown_connections 2022-06-29T15:58:35.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.269+0000 7f081da35700 1 -- 172.21.15.60:0/2072092427 wait complete. 2022-06-29T15:58:35.277 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 4 2022-06-29T15:58:35.487 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 4, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T15:58:24.334770+0000', 'modified': '2022-06-29T15:58:25.338504+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4257}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4257': {'gid': 4257, 'name': 'a', 'rank': 0, 'incarnation': 3, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.60:6833/3164364318', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6832', 'nonce': 3164364318}, {'type': 'v1', 'addr': '172.21.15.60:6833', 'nonce': 3164364318}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T15:58:35.487 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T15:58:35.488 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-06-29T15:58:35.604 INFO:teuthology.orchestra.run.smithi060.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":4257,"want_state":"up:active","state":"up:active","rank_uptime":11.253415718999999,"mdsmap_epoch":4,"osdmap_epoch":23,"osdmap_epoch_barrier":21,"uptime":14.503515557} 2022-06-29T15:58:35.605 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 4257, "mdsmap_epoch": 4, "osdmap_epoch": 23, "osdmap_epoch_barrier": 21, "rank_uptime": 11.253415719, "state": "up:active", "uptime": 14.503515557, "want_state": "up:active", "whoami": 0 } 2022-06-29T15:58:35.605 INFO:teuthology.run_tasks:Running task ceph-fuse... 2022-06-29T15:58:35.622 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2022-06-29T15:58:35.623 INFO:tasks.ceph_fuse:config is None 2022-06-29T15:58:35.623 INFO:tasks.ceph_fuse:new config is {'client.0': None, 'client.1': None, 'client.2': None} 2022-06-29T15:58:35.623 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2022-06-29T15:58:35.624 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T15:58:35.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.729+0000 7f078d3b9700 1 Processor -- start 2022-06-29T15:58:35.734 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.729+0000 7f078d3b9700 1 -- start start 2022-06-29T15:58:35.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.729+0000 7f078d3b9700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07880b5950 0x7f0788067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:35.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.729+0000 7f078d3b9700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f078814af10 0x7f078805c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:35.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.729+0000 7f078d3b9700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07880acba0 con 0x7f0788063ad0 2022-06-29T15:58:35.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.729+0000 7f078d3b9700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f078814b3a0 con 0x7f078814af10 2022-06-29T15:58:35.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.729+0000 7f078d3b9700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0788067220 con 0x7f07880b5950 2022-06-29T15:58:35.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.729+0000 7f077e7fc700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f0788063ad0 0x7f078805bd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53570/0 (socket says 172.21.15.60:53570) 2022-06-29T15:58:35.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.729+0000 7f077e7fc700 1 -- 172.21.15.60:0/1409407084 learned_addr learned my addr 172.21.15.60:0/1409407084 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:35.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.729+0000 7f07877fe700 1 --2- 172.21.15.60:0/1409407084 >> v2:172.21.15.60:3300/0 conn(0x7f078814af10 0x7f078805c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:35.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.729+0000 7f0786ffd700 1 --2- 172.21.15.60:0/1409407084 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07880b5950 0x7f0788067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:35.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.729+0000 7f07867fc700 1 -- 172.21.15.60:0/1409407084 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199041866 0 0) 0x7f07880acba0 con 0x7f0788063ad0 2022-06-29T15:58:35.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.729+0000 7f07867fc700 1 -- 172.21.15.60:0/1409407084 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0768003260 con 0x7f0788063ad0 2022-06-29T15:58:35.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.729+0000 7f07877fe700 1 -- 172.21.15.60:0/1409407084 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07880b5950 msgr2=0x7f0788067680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:35.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.729+0000 7f07877fe700 1 --2- 172.21.15.60:0/1409407084 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07880b5950 0x7f0788067680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:35.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.729+0000 7f07877fe700 1 -- 172.21.15.60:0/1409407084 >> v1:172.21.15.60:6789/0 conn(0x7f0788063ad0 legacy=0x7f078805bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:35.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.729+0000 7f07877fe700 1 -- 172.21.15.60:0/1409407084 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07880b5d30 con 0x7f078814af10 2022-06-29T15:58:35.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.729+0000 7f0786ffd700 1 --2- 172.21.15.60:0/1409407084 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07880b5950 0x7f0788067680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:35.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.729+0000 7f07877fe700 1 --2- 172.21.15.60:0/1409407084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f078814af10 0x7f078805c430 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f07800032f0 tx=0x7f07800091d0).ready entity=mon.0 client_cookie=19a54c11ffb58f29 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:35.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.729+0000 7f07867fc700 1 -- 172.21.15.60:0/1409407084 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f078000b600 con 0x7f078814af10 2022-06-29T15:58:35.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.729+0000 7f07867fc700 1 -- 172.21.15.60:0/1409407084 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f078000b760 con 0x7f078814af10 2022-06-29T15:58:35.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.729+0000 7f07867fc700 1 -- 172.21.15.60:0/1409407084 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f078000b8d0 con 0x7f078814af10 2022-06-29T15:58:35.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.729+0000 7f078d3b9700 1 -- 172.21.15.60:0/1409407084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f078814af10 msgr2=0x7f078805c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:35.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.729+0000 7f078d3b9700 1 --2- 172.21.15.60:0/1409407084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f078814af10 0x7f078805c430 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f07800032f0 tx=0x7f07800091d0).stop 2022-06-29T15:58:35.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.729+0000 7f078d3b9700 1 -- 172.21.15.60:0/1409407084 shutdown_connections 2022-06-29T15:58:35.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.729+0000 7f078d3b9700 1 --2- 172.21.15.60:0/1409407084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f078814af10 0x7f078805c430 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:35.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.729+0000 7f078d3b9700 1 --2- 172.21.15.60:0/1409407084 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07880b5950 0x7f0788067680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:35.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.729+0000 7f078d3b9700 1 -- 172.21.15.60:0/1409407084 >> 172.21.15.60:0/1409407084 conn(0x7f07880632c0 msgr2=0x7f07880b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:35.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.733+0000 7f078d3b9700 1 -- 172.21.15.60:0/1409407084 shutdown_connections 2022-06-29T15:58:35.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.733+0000 7f078d3b9700 1 -- 172.21.15.60:0/1409407084 wait complete. 2022-06-29T15:58:35.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.733+0000 7f078d3b9700 1 Processor -- start 2022-06-29T15:58:35.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.733+0000 7f078d3b9700 1 -- start start 2022-06-29T15:58:35.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.733+0000 7f078d3b9700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0788063ad0 0x7f0788145440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:35.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.733+0000 7f078d3b9700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07880b5950 0x7f07880b0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:35.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.733+0000 7f078d3b9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f078814af10 0x7f07880b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:35.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.733+0000 7f078d3b9700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07880b1010 con 0x7f078814af10 2022-06-29T15:58:35.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.733+0000 7f078d3b9700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078815a130 con 0x7f0788063ad0 2022-06-29T15:58:35.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.733+0000 7f078d3b9700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07880ad180 con 0x7f07880b5950 2022-06-29T15:58:35.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.733+0000 7f07877fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f078814af10 0x7f07880b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:35.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.733+0000 7f077e7fc700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07880b5950 0x7f07880b0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:35.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.733+0000 7f07877fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f078814af10 0x7f07880b0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45494/0 (socket says 172.21.15.60:45494) 2022-06-29T15:58:35.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.733+0000 7f0786ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0788063ad0 0x7f0788145440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:35.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.733+0000 7f07877fe700 1 -- 172.21.15.60:0/2692362617 learned_addr learned my addr 172.21.15.60:0/2692362617 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:35.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.737+0000 7f077e7fc700 1 -- 172.21.15.60:0/2692362617 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0788063ad0 msgr2=0x7f0788145440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:35.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.737+0000 7f077e7fc700 1 --2- 172.21.15.60:0/2692362617 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0788063ad0 0x7f0788145440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:35.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.737+0000 7f077e7fc700 1 -- 172.21.15.60:0/2692362617 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f078814af10 msgr2=0x7f07880b0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:35.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.737+0000 7f077e7fc700 1 --2- 172.21.15.60:0/2692362617 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f078814af10 0x7f07880b0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:35.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.737+0000 7f077e7fc700 1 -- 172.21.15.60:0/2692362617 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f077800b180 con 0x7f07880b5950 2022-06-29T15:58:35.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.737+0000 7f077e7fc700 1 --2- 172.21.15.60:0/2692362617 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07880b5950 0x7f07880b0630 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f07880afac0 tx=0x7f07700079e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:35.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.737+0000 7f0784ff9700 1 -- 172.21.15.60:0/2692362617 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f077000cda0 con 0x7f07880b5950 2022-06-29T15:58:35.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.737+0000 7f0784ff9700 1 -- 172.21.15.60:0/2692362617 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07700070d0 con 0x7f07880b5950 2022-06-29T15:58:35.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.737+0000 7f0784ff9700 1 -- 172.21.15.60:0/2692362617 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0770007430 con 0x7f07880b5950 2022-06-29T15:58:35.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.737+0000 7f078d3b9700 1 -- 172.21.15.60:0/2692362617 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0780009470 con 0x7f07880b5950 2022-06-29T15:58:35.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.737+0000 7f078d3b9700 1 -- 172.21.15.60:0/2692362617 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07881572f0 con 0x7f07880b5950 2022-06-29T15:58:35.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.737+0000 7f078d3b9700 1 -- 172.21.15.60:0/2692362617 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07880573d0 con 0x7f07880b5950 2022-06-29T15:58:35.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.737+0000 7f0784ff9700 1 -- 172.21.15.60:0/2692362617 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0770010070 con 0x7f07880b5950 2022-06-29T15:58:35.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.737+0000 7f0784ff9700 1 --2- 172.21.15.60:0/2692362617 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f076c053a70 0x7f076c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:35.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.741+0000 7f0784ff9700 1 -- 172.21.15.60:0/2692362617 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f077006b480 con 0x7f07880b5950 2022-06-29T15:58:35.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.741+0000 7f0786ffd700 1 --2- 172.21.15.60:0/2692362617 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f076c053a70 0x7f076c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:35.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.741+0000 7f0786ffd700 1 --2- 172.21.15.60:0/2692362617 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f076c053a70 0x7f076c055f30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f077800b000 tx=0x7f0778006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:35.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.749+0000 7f0784ff9700 1 -- 172.21.15.60:0/2692362617 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0770042290 con 0x7f07880b5950 2022-06-29T15:58:35.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.877+0000 7f078d3b9700 1 -- 172.21.15.60:0/2692362617 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07880ad2c0 con 0x7f07880b5950 2022-06-29T15:58:35.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.881+0000 7f0784ff9700 1 -- 172.21.15.60:0/2692362617 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16038 (secure 0 0 0) 0x7f0770018070 con 0x7f07880b5950 2022-06-29T15:58:35.885 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:58:35.886 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":23,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T15:58:26.345644+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-06-29T15:58:22.424395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-06-29T15:58:23.343438+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T15:58:35.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.881+0000 7f078d3b9700 1 -- 172.21.15.60:0/2692362617 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f076c053a70 msgr2=0x7f076c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:35.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.881+0000 7f078d3b9700 1 --2- 172.21.15.60:0/2692362617 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f076c053a70 0x7f076c055f30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f077800b000 tx=0x7f0778006df0).stop 2022-06-29T15:58:35.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.881+0000 7f078d3b9700 1 -- 172.21.15.60:0/2692362617 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07880b5950 msgr2=0x7f07880b0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:35.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.881+0000 7f078d3b9700 1 --2- 172.21.15.60:0/2692362617 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07880b5950 0x7f07880b0630 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f07880afac0 tx=0x7f07700079e0).stop 2022-06-29T15:58:35.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.881+0000 7f078d3b9700 1 -- 172.21.15.60:0/2692362617 shutdown_connections 2022-06-29T15:58:35.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.881+0000 7f078d3b9700 1 --2- 172.21.15.60:0/2692362617 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f076c053a70 0x7f076c055f30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:35.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.881+0000 7f078d3b9700 1 --2- 172.21.15.60:0/2692362617 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f078814af10 0x7f07880b0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:35.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.881+0000 7f078d3b9700 1 --2- 172.21.15.60:0/2692362617 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07880b5950 0x7f07880b0630 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:35.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.881+0000 7f078d3b9700 1 --2- 172.21.15.60:0/2692362617 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0788063ad0 0x7f0788145440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:35.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.881+0000 7f078d3b9700 1 -- 172.21.15.60:0/2692362617 >> 172.21.15.60:0/2692362617 conn(0x7f07880632c0 msgr2=0x7f0788142dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:35.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.881+0000 7f078d3b9700 1 -- 172.21.15.60:0/2692362617 shutdown_connections 2022-06-29T15:58:35.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:35.881+0000 7f078d3b9700 1 -- 172.21.15.60:0/2692362617 wait complete. 2022-06-29T15:58:35.905 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-29T15:58:22.424395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-06-29T15:58:23.343438+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-29T15:58:35.906 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T15:58:36.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.013+0000 7f3756c23700 1 Processor -- start 2022-06-29T15:58:36.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.013+0000 7f3756c23700 1 -- start start 2022-06-29T15:58:36.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.013+0000 7f3756c23700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f37500b3d30 0x7f3750146ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:36.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.013+0000 7f3756c23700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3750061d90 0x7f3750142b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:36.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.013+0000 7f3756c23700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37500b4f50 con 0x7f37500b9d70 2022-06-29T15:58:36.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.013+0000 7f3756c23700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37500ba150 con 0x7f37500b3d30 2022-06-29T15:58:36.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.013+0000 7f3756c23700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37500b5180 con 0x7f3750061d90 2022-06-29T15:58:36.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.013+0000 7f3747fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f37500b3d30 0x7f3750146ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:36.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.013+0000 7f3747fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f37500b3d30 0x7f3750146ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45500/0 (socket says 172.21.15.60:45500) 2022-06-29T15:58:36.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.013+0000 7f3747fff700 1 -- 172.21.15.60:0/1608818812 learned_addr learned my addr 172.21.15.60:0/1608818812 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:36.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.013+0000 7f37551c0700 1 --2- 172.21.15.60:0/1608818812 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3750061d90 0x7f3750142b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:36.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.013+0000 7f3747fff700 1 -- 172.21.15.60:0/1608818812 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3750061d90 msgr2=0x7f3750142b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:36.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.013+0000 7f3747fff700 1 --2- 172.21.15.60:0/1608818812 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3750061d90 0x7f3750142b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.013+0000 7f3747fff700 1 -- 172.21.15.60:0/1608818812 >> v1:172.21.15.92:6789/0 conn(0x7f37500b9d70 legacy=0x7f37501465d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:36.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.013+0000 7f3747fff700 1 -- 172.21.15.60:0/1608818812 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37501461a0 con 0x7f37500b3d30 2022-06-29T15:58:36.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.013+0000 7f3747fff700 1 --2- 172.21.15.60:0/1608818812 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37500b3d30 0x7f3750146ce0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f3740002f70 tx=0x7f3740006eb0).ready entity=mon.0 client_cookie=f06f97ae9ae5f9f server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:36.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.013+0000 7f37477fe700 1 -- 172.21.15.60:0/1608818812 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f374000d570 con 0x7f37500b3d30 2022-06-29T15:58:36.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.013+0000 7f37477fe700 1 -- 172.21.15.60:0/1608818812 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3740000e90 con 0x7f37500b3d30 2022-06-29T15:58:36.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.013+0000 7f37477fe700 1 -- 172.21.15.60:0/1608818812 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f374000d900 con 0x7f37500b3d30 2022-06-29T15:58:36.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.013+0000 7f3756c23700 1 -- 172.21.15.60:0/1608818812 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37500b3d30 msgr2=0x7f3750146ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:36.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.013+0000 7f3756c23700 1 --2- 172.21.15.60:0/1608818812 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37500b3d30 0x7f3750146ce0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f3740002f70 tx=0x7f3740006eb0).stop 2022-06-29T15:58:36.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.017+0000 7f3756c23700 1 -- 172.21.15.60:0/1608818812 shutdown_connections 2022-06-29T15:58:36.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.017+0000 7f3756c23700 1 --2- 172.21.15.60:0/1608818812 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3750061d90 0x7f3750142b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.017+0000 7f3756c23700 1 --2- 172.21.15.60:0/1608818812 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37500b3d30 0x7f3750146ce0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.017+0000 7f3756c23700 1 -- 172.21.15.60:0/1608818812 >> 172.21.15.60:0/1608818812 conn(0x7f3750062e70 msgr2=0x7f37500b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:36.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.017+0000 7f3756c23700 1 -- 172.21.15.60:0/1608818812 shutdown_connections 2022-06-29T15:58:36.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.017+0000 7f3756c23700 1 -- 172.21.15.60:0/1608818812 wait complete. 2022-06-29T15:58:36.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.017+0000 7f3756c23700 1 Processor -- start 2022-06-29T15:58:36.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.017+0000 7f3756c23700 1 -- start start 2022-06-29T15:58:36.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.017+0000 7f3756c23700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3750061d90 0x7f3750155390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:36.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.017+0000 7f3756c23700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37500b3d30 0x7f3750148640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:36.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.017+0000 7f3756c23700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37500b9d70 0x7f37501138f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:36.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.017+0000 7f3756c23700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37500ae6c0 con 0x7f37500b3d30 2022-06-29T15:58:36.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.017+0000 7f3756c23700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37500ae8f0 con 0x7f37500b9d70 2022-06-29T15:58:36.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.017+0000 7f3756c23700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37500aeb90 con 0x7f3750061d90 2022-06-29T15:58:36.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.017+0000 7f3747fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37500b3d30 0x7f3750148640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:36.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.017+0000 7f3747fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37500b3d30 0x7f3750148640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45506/0 (socket says 172.21.15.60:45506) 2022-06-29T15:58:36.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.017+0000 7f3747fff700 1 -- 172.21.15.60:0/2135119918 learned_addr learned my addr 172.21.15.60:0/2135119918 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:36.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.017+0000 7f37551c0700 1 --2- 172.21.15.60:0/2135119918 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37500b9d70 0x7f37501138f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:36.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.017+0000 7f37549bf700 1 --2- 172.21.15.60:0/2135119918 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3750061d90 0x7f3750155390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:36.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.017+0000 7f3747fff700 1 -- 172.21.15.60:0/2135119918 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3750061d90 msgr2=0x7f3750155390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:36.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.017+0000 7f3747fff700 1 --2- 172.21.15.60:0/2135119918 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3750061d90 0x7f3750155390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.017+0000 7f3747fff700 1 -- 172.21.15.60:0/2135119918 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37500b9d70 msgr2=0x7f37501138f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:36.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.017+0000 7f3747fff700 1 --2- 172.21.15.60:0/2135119918 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37500b9d70 0x7f37501138f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.017+0000 7f3747fff700 1 -- 172.21.15.60:0/2135119918 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f374000b1e0 con 0x7f37500b3d30 2022-06-29T15:58:36.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.021+0000 7f3747fff700 1 --2- 172.21.15.60:0/2135119918 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37500b3d30 0x7f3750148640 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f3740002f70 tx=0x7f3740015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:36.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.021+0000 7f3745ffb700 1 -- 172.21.15.60:0/2135119918 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f37400068b0 con 0x7f37500b3d30 2022-06-29T15:58:36.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.021+0000 7f3745ffb700 1 -- 172.21.15.60:0/2135119918 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f374000d8f0 con 0x7f37500b3d30 2022-06-29T15:58:36.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.021+0000 7f3756c23700 1 -- 172.21.15.60:0/2135119918 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37500af990 con 0x7f37500b3d30 2022-06-29T15:58:36.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.021+0000 7f3745ffb700 1 -- 172.21.15.60:0/2135119918 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3740011d70 con 0x7f37500b3d30 2022-06-29T15:58:36.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.021+0000 7f3756c23700 1 -- 172.21.15.60:0/2135119918 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37500ad5d0 con 0x7f37500b3d30 2022-06-29T15:58:36.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.021+0000 7f3756c23700 1 -- 172.21.15.60:0/2135119918 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37500aed50 con 0x7f37500b3d30 2022-06-29T15:58:36.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.021+0000 7f3745ffb700 1 -- 172.21.15.60:0/2135119918 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3740025030 con 0x7f37500b3d30 2022-06-29T15:58:36.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.021+0000 7f3745ffb700 1 --2- 172.21.15.60:0/2135119918 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3730053a70 0x7f3730055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:36.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.021+0000 7f3745ffb700 1 -- 172.21.15.60:0/2135119918 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f374006d160 con 0x7f37500b3d30 2022-06-29T15:58:36.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.025+0000 7f37549bf700 1 --2- 172.21.15.60:0/2135119918 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3730053a70 0x7f3730055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:36.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.025+0000 7f37549bf700 1 --2- 172.21.15.60:0/2135119918 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3730053a70 0x7f3730055f30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f374c00d000 tx=0x7f374c007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:36.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.029+0000 7f3745ffb700 1 -- 172.21.15.60:0/2135119918 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3740047e60 con 0x7f37500b3d30 2022-06-29T15:58:36.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.153+0000 7f3756c23700 1 -- 172.21.15.60:0/2135119918 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f37500573d0 con 0x7f37500b3d30 2022-06-29T15:58:36.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.157+0000 7f3745ffb700 1 -- 172.21.15.60:0/2135119918 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v23) v1 ==== 110+0+10 (secure 0 0 0) 0x7f37400230b0 con 0x7f37500b3d30 2022-06-29T15:58:36.161 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T15:58:36.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.157+0000 7f3756c23700 1 -- 172.21.15.60:0/2135119918 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3730053a70 msgr2=0x7f3730055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:36.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.157+0000 7f3756c23700 1 --2- 172.21.15.60:0/2135119918 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3730053a70 0x7f3730055f30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f374c00d000 tx=0x7f374c007b90).stop 2022-06-29T15:58:36.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.157+0000 7f3756c23700 1 -- 172.21.15.60:0/2135119918 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37500b3d30 msgr2=0x7f3750148640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:36.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.157+0000 7f3756c23700 1 --2- 172.21.15.60:0/2135119918 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37500b3d30 0x7f3750148640 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f3740002f70 tx=0x7f3740015000).stop 2022-06-29T15:58:36.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.157+0000 7f3756c23700 1 -- 172.21.15.60:0/2135119918 shutdown_connections 2022-06-29T15:58:36.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.157+0000 7f3756c23700 1 --2- 172.21.15.60:0/2135119918 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3730053a70 0x7f3730055f30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.157+0000 7f3756c23700 1 --2- 172.21.15.60:0/2135119918 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37500b9d70 0x7f37501138f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.157+0000 7f3756c23700 1 --2- 172.21.15.60:0/2135119918 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37500b3d30 0x7f3750148640 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.157+0000 7f3756c23700 1 --2- 172.21.15.60:0/2135119918 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3750061d90 0x7f3750155390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.157+0000 7f3756c23700 1 -- 172.21.15.60:0/2135119918 >> 172.21.15.60:0/2135119918 conn(0x7f3750062e70 msgr2=0x7f375005c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:36.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.157+0000 7f3756c23700 1 -- 172.21.15.60:0/2135119918 shutdown_connections 2022-06-29T15:58:36.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.157+0000 7f3756c23700 1 -- 172.21.15.60:0/2135119918 wait complete. 2022-06-29T15:58:36.179 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T15:58:36.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.281+0000 7f49d9ca7700 1 Processor -- start 2022-06-29T15:58:36.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.281+0000 7f49d9ca7700 1 -- start start 2022-06-29T15:58:36.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.285+0000 7f49d9ca7700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f49d40b3c70 0x7f49d4065a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:36.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.285+0000 7f49d9ca7700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f49d4141860 0x7f49d4141cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:36.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.285+0000 7f49d9ca7700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49d405c2b0 con 0x7f49d4145eb0 2022-06-29T15:58:36.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.285+0000 7f49d9ca7700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49d4066620 con 0x7f49d40b3c70 2022-06-29T15:58:36.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.285+0000 7f49d9ca7700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49d40b4050 con 0x7f49d4141860 2022-06-29T15:58:36.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.285+0000 7f49d8ca5700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f49d40b3c70 0x7f49d4065a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:36.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.285+0000 7f49d3fff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f49d4145eb0 0x7f49d4065f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53598/0 (socket says 172.21.15.60:53598) 2022-06-29T15:58:36.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.285+0000 7f49d3fff700 1 -- 172.21.15.60:0/3218641255 learned_addr learned my addr 172.21.15.60:0/3218641255 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:36.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.285+0000 7f49d94a6700 1 --2- 172.21.15.60:0/3218641255 >> v2:172.21.15.92:3300/0 conn(0x7f49d4141860 0x7f49d4141cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:36.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.285+0000 7f49d37fe700 1 -- 172.21.15.60:0/3218641255 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277542435 0 0) 0x7f49d405c2b0 con 0x7f49d4145eb0 2022-06-29T15:58:36.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.285+0000 7f49d37fe700 1 -- 172.21.15.60:0/3218641255 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49b4003260 con 0x7f49d4145eb0 2022-06-29T15:58:36.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.285+0000 7f49d8ca5700 1 -- 172.21.15.60:0/3218641255 >> v2:172.21.15.92:3300/0 conn(0x7f49d4141860 msgr2=0x7f49d4141cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:36.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.285+0000 7f49d8ca5700 1 --2- 172.21.15.60:0/3218641255 >> v2:172.21.15.92:3300/0 conn(0x7f49d4141860 0x7f49d4141cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.285+0000 7f49d8ca5700 1 -- 172.21.15.60:0/3218641255 >> v1:172.21.15.60:6789/0 conn(0x7f49d4145eb0 legacy=0x7f49d4065f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:36.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.285+0000 7f49d8ca5700 1 -- 172.21.15.60:0/3218641255 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d405d120 con 0x7f49d40b3c70 2022-06-29T15:58:36.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.285+0000 7f49d8ca5700 1 --2- 172.21.15.60:0/3218641255 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49d40b3c70 0x7f49d4065a20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f49bc002f70 tx=0x7f49bc006df0).ready entity=mon.0 client_cookie=b78c2741c69ce88b server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:36.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.285+0000 7f49d37fe700 1 -- 172.21.15.60:0/3218641255 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f49bc00d4f0 con 0x7f49d40b3c70 2022-06-29T15:58:36.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.285+0000 7f49d37fe700 1 -- 172.21.15.60:0/3218641255 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49bc000e90 con 0x7f49d40b3c70 2022-06-29T15:58:36.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.285+0000 7f49d37fe700 1 -- 172.21.15.60:0/3218641255 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f49bc00d840 con 0x7f49d40b3c70 2022-06-29T15:58:36.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.285+0000 7f49d9ca7700 1 -- 172.21.15.60:0/3218641255 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49d40b3c70 msgr2=0x7f49d4065a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:36.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.285+0000 7f49d9ca7700 1 --2- 172.21.15.60:0/3218641255 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49d40b3c70 0x7f49d4065a20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f49bc002f70 tx=0x7f49bc006df0).stop 2022-06-29T15:58:36.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.285+0000 7f49d9ca7700 1 -- 172.21.15.60:0/3218641255 shutdown_connections 2022-06-29T15:58:36.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.285+0000 7f49d9ca7700 1 --2- 172.21.15.60:0/3218641255 >> v2:172.21.15.92:3300/0 conn(0x7f49d4141860 0x7f49d4141cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.285+0000 7f49d9ca7700 1 --2- 172.21.15.60:0/3218641255 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49d40b3c70 0x7f49d4065a20 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.285+0000 7f49d9ca7700 1 -- 172.21.15.60:0/3218641255 >> 172.21.15.60:0/3218641255 conn(0x7f49d4064810 msgr2=0x7f49d4064bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:36.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.285+0000 7f49d9ca7700 1 -- 172.21.15.60:0/3218641255 shutdown_connections 2022-06-29T15:58:36.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.285+0000 7f49d9ca7700 1 -- 172.21.15.60:0/3218641255 wait complete. 2022-06-29T15:58:36.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.289+0000 7f49d9ca7700 1 Processor -- start 2022-06-29T15:58:36.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.289+0000 7f49d9ca7700 1 -- start start 2022-06-29T15:58:36.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.289+0000 7f49d9ca7700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49d40b3c70 0x7f49d4140fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:36.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.289+0000 7f49d9ca7700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49d4141860 0x7f49d41376b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:36.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.289+0000 7f49d9ca7700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f49d4145eb0 0x7f49d41520b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:36.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.293+0000 7f49d9ca7700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d40b5040 con 0x7f49d40b3c70 2022-06-29T15:58:36.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.293+0000 7f49d9ca7700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d40b52e0 con 0x7f49d4141860 2022-06-29T15:58:36.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.293+0000 7f49d9ca7700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49d40b5580 con 0x7f49d4145eb0 2022-06-29T15:58:36.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.293+0000 7f49d8ca5700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49d40b3c70 0x7f49d4140fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:36.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.293+0000 7f49d8ca5700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49d40b3c70 0x7f49d4140fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45518/0 (socket says 172.21.15.60:45518) 2022-06-29T15:58:36.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.293+0000 7f49d8ca5700 1 -- 172.21.15.60:0/7066208 learned_addr learned my addr 172.21.15.60:0/7066208 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:36.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.293+0000 7f49d3fff700 1 --2- 172.21.15.60:0/7066208 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49d4141860 0x7f49d41376b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:36.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.293+0000 7f49d94a6700 1 --2- 172.21.15.60:0/7066208 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f49d4145eb0 0x7f49d41520b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:36.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.293+0000 7f49d8ca5700 1 -- 172.21.15.60:0/7066208 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f49d4145eb0 msgr2=0x7f49d41520b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:36.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.293+0000 7f49d8ca5700 1 --2- 172.21.15.60:0/7066208 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f49d4145eb0 0x7f49d41520b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.293+0000 7f49d8ca5700 1 -- 172.21.15.60:0/7066208 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49d4141860 msgr2=0x7f49d41376b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:36.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.293+0000 7f49d8ca5700 1 --2- 172.21.15.60:0/7066208 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49d4141860 0x7f49d41376b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.293+0000 7f49d8ca5700 1 -- 172.21.15.60:0/7066208 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49bc00b180 con 0x7f49d40b3c70 2022-06-29T15:58:36.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.293+0000 7f49d94a6700 1 --2- 172.21.15.60:0/7066208 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f49d4145eb0 0x7f49d41520b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:58:36.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.293+0000 7f49d8ca5700 1 --2- 172.21.15.60:0/7066208 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49d40b3c70 0x7f49d4140fa0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f49bc002f70 tx=0x7f49bc014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:36.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.293+0000 7f49d1ffb700 1 -- 172.21.15.60:0/7066208 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f49bc016070 con 0x7f49d40b3c70 2022-06-29T15:58:36.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.293+0000 7f49d1ffb700 1 -- 172.21.15.60:0/7066208 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49bc0138d0 con 0x7f49d40b3c70 2022-06-29T15:58:36.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.293+0000 7f49d1ffb700 1 -- 172.21.15.60:0/7066208 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f49bc013c30 con 0x7f49d40b3c70 2022-06-29T15:58:36.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.293+0000 7f49d9ca7700 1 -- 172.21.15.60:0/7066208 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49d40b4d30 con 0x7f49d40b3c70 2022-06-29T15:58:36.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.293+0000 7f49d9ca7700 1 -- 172.21.15.60:0/7066208 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49d4151a90 con 0x7f49d40b3c70 2022-06-29T15:58:36.326 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.293+0000 7f49d9ca7700 1 -- 172.21.15.60:0/7066208 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49d405d3c0 con 0x7f49d40b3c70 2022-06-29T15:58:36.326 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.297+0000 7f49d1ffb700 1 -- 172.21.15.60:0/7066208 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f49bc00f030 con 0x7f49d40b3c70 2022-06-29T15:58:36.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.297+0000 7f49d1ffb700 1 --2- 172.21.15.60:0/7066208 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f49b8053b00 0x7f49b8055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:36.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.297+0000 7f49d1ffb700 1 -- 172.21.15.60:0/7066208 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f49bc06cb00 con 0x7f49d40b3c70 2022-06-29T15:58:36.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.297+0000 7f49d3fff700 1 --2- 172.21.15.60:0/7066208 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f49b8053b00 0x7f49b8055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:36.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.297+0000 7f49d3fff700 1 --2- 172.21.15.60:0/7066208 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f49b8053b00 0x7f49b8055fc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f49c400d000 tx=0x7f49c4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:36.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.301+0000 7f49d1ffb700 1 -- 172.21.15.60:0/7066208 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f49bc043a20 con 0x7f49d40b3c70 2022-06-29T15:58:36.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.437+0000 7f49d9ca7700 1 -- 172.21.15.60:0/7066208 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f49d40573d0 con 0x7f49d40b3c70 2022-06-29T15:58:36.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.437+0000 7f49d1ffb700 1 -- 172.21.15.60:0/7066208 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f49bc0230a0 con 0x7f49d40b3c70 2022-06-29T15:58:36.442 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T15:58:36.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.437+0000 7f49d9ca7700 1 -- 172.21.15.60:0/7066208 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f49b8053b00 msgr2=0x7f49b8055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:36.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.437+0000 7f49d9ca7700 1 --2- 172.21.15.60:0/7066208 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f49b8053b00 0x7f49b8055fc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f49c400d000 tx=0x7f49c4009040).stop 2022-06-29T15:58:36.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.437+0000 7f49d9ca7700 1 -- 172.21.15.60:0/7066208 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49d40b3c70 msgr2=0x7f49d4140fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:36.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.437+0000 7f49d9ca7700 1 --2- 172.21.15.60:0/7066208 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49d40b3c70 0x7f49d4140fa0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f49bc002f70 tx=0x7f49bc014000).stop 2022-06-29T15:58:36.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.437+0000 7f49d9ca7700 1 -- 172.21.15.60:0/7066208 shutdown_connections 2022-06-29T15:58:36.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.437+0000 7f49d9ca7700 1 --2- 172.21.15.60:0/7066208 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f49b8053b00 0x7f49b8055fc0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.437+0000 7f49d9ca7700 1 --2- 172.21.15.60:0/7066208 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f49d4145eb0 0x7f49d41520b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.437+0000 7f49d9ca7700 1 --2- 172.21.15.60:0/7066208 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49d4141860 0x7f49d41376b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.437+0000 7f49d9ca7700 1 --2- 172.21.15.60:0/7066208 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49d40b3c70 0x7f49d4140fa0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.437+0000 7f49d9ca7700 1 -- 172.21.15.60:0/7066208 >> 172.21.15.60:0/7066208 conn(0x7f49d4064810 msgr2=0x7f49d40b4220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:36.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.437+0000 7f49d9ca7700 1 -- 172.21.15.60:0/7066208 shutdown_connections 2022-06-29T15:58:36.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.437+0000 7f49d9ca7700 1 -- 172.21.15.60:0/7066208 wait complete. 2022-06-29T15:58:36.456 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-29T15:58:36.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.557+0000 7fcc4dde2700 1 Processor -- start 2022-06-29T15:58:36.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.557+0000 7fcc4dde2700 1 -- start start 2022-06-29T15:58:36.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.557+0000 7fcc4dde2700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc48067680 0x7fcc4814af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:36.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.557+0000 7fcc4dde2700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc480acba0 con 0x7fcc48063ad0 2022-06-29T15:58:36.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.557+0000 7fcc4dde2700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc48067220 con 0x7fcc480b5950 2022-06-29T15:58:36.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.557+0000 7fcc4dde2700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc480b5d30 con 0x7fcc48067680 2022-06-29T15:58:36.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.557+0000 7fcc3effd700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcc48063ad0 0x7fcc4805c430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53612/0 (socket says 172.21.15.60:53612) 2022-06-29T15:58:36.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.557+0000 7fcc3effd700 1 -- 172.21.15.60:0/594468970 learned_addr learned my addr 172.21.15.60:0/594468970 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:36.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.557+0000 7fcc47fff700 1 --2- 172.21.15.60:0/594468970 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc48067680 0x7fcc4814af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:36.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.561+0000 7fcc46ffd700 1 -- 172.21.15.60:0/594468970 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3570581583 0 0) 0x7fcc480acba0 con 0x7fcc48063ad0 2022-06-29T15:58:36.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.561+0000 7fcc46ffd700 1 -- 172.21.15.60:0/594468970 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc28003260 con 0x7fcc48063ad0 2022-06-29T15:58:36.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.561+0000 7fcc46ffd700 1 -- 172.21.15.60:0/594468970 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2829336011 0 0) 0x7fcc28003260 con 0x7fcc48063ad0 2022-06-29T15:58:36.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.561+0000 7fcc47fff700 1 -- 172.21.15.60:0/594468970 >> v1:172.21.15.92:6789/0 conn(0x7fcc480b5950 legacy=0x7fcc4805bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:36.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.561+0000 7fcc47fff700 1 -- 172.21.15.60:0/594468970 >> v1:172.21.15.60:6789/0 conn(0x7fcc48063ad0 legacy=0x7fcc4805c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:36.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.561+0000 7fcc47fff700 1 -- 172.21.15.60:0/594468970 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4805cb40 con 0x7fcc48067680 2022-06-29T15:58:36.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.561+0000 7fcc47fff700 1 --2- 172.21.15.60:0/594468970 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc48067680 0x7fcc4814af10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fcc30001800 tx=0x7fcc400091d0).ready entity=mon.2 client_cookie=c2febbd837c52cf1 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:36.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.561+0000 7fcc46ffd700 1 -- 172.21.15.60:0/594468970 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc30001660 con 0x7fcc48067680 2022-06-29T15:58:36.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.561+0000 7fcc46ffd700 1 -- 172.21.15.60:0/594468970 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc4000b580 con 0x7fcc48067680 2022-06-29T15:58:36.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.561+0000 7fcc46ffd700 1 -- 172.21.15.60:0/594468970 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc4000b6f0 con 0x7fcc48067680 2022-06-29T15:58:36.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.561+0000 7fcc4dde2700 1 -- 172.21.15.60:0/594468970 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc48067680 msgr2=0x7fcc4814af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:36.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.561+0000 7fcc4dde2700 1 --2- 172.21.15.60:0/594468970 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc48067680 0x7fcc4814af10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fcc30001800 tx=0x7fcc400091d0).stop 2022-06-29T15:58:36.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.561+0000 7fcc4dde2700 1 -- 172.21.15.60:0/594468970 shutdown_connections 2022-06-29T15:58:36.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.561+0000 7fcc4dde2700 1 --2- 172.21.15.60:0/594468970 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc48067680 0x7fcc4814af10 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.561+0000 7fcc4dde2700 1 -- 172.21.15.60:0/594468970 >> 172.21.15.60:0/594468970 conn(0x7fcc480632c0 msgr2=0x7fcc480b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:36.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.561+0000 7fcc4dde2700 1 -- 172.21.15.60:0/594468970 shutdown_connections 2022-06-29T15:58:36.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.561+0000 7fcc4dde2700 1 -- 172.21.15.60:0/594468970 wait complete. 2022-06-29T15:58:36.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.561+0000 7fcc4dde2700 1 Processor -- start 2022-06-29T15:58:36.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.561+0000 7fcc4dde2700 1 -- start start 2022-06-29T15:58:36.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.565+0000 7fcc4dde2700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc48063ad0 0x7fcc481593f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:36.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.565+0000 7fcc4dde2700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc48067680 0x7fcc481471c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:36.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.565+0000 7fcc4dde2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc480b5950 0x7fcc48116ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:36.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.565+0000 7fcc4dde2700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc480b00e0 con 0x7fcc480b5950 2022-06-29T15:58:36.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.565+0000 7fcc4dde2700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc480af470 con 0x7fcc48063ad0 2022-06-29T15:58:36.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.565+0000 7fcc4dde2700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc480afba0 con 0x7fcc48067680 2022-06-29T15:58:36.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.565+0000 7fcc47fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc480b5950 0x7fcc48116ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:36.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.565+0000 7fcc47fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc480b5950 0x7fcc48116ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45536/0 (socket says 172.21.15.60:45536) 2022-06-29T15:58:36.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.565+0000 7fcc47fff700 1 -- 172.21.15.60:0/3039583529 learned_addr learned my addr 172.21.15.60:0/3039583529 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:36.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.565+0000 7fcc3effd700 1 --2- 172.21.15.60:0/3039583529 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc48067680 0x7fcc481471c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:36.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.565+0000 7fcc477fe700 1 --2- 172.21.15.60:0/3039583529 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc48063ad0 0x7fcc481593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:36.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.565+0000 7fcc47fff700 1 -- 172.21.15.60:0/3039583529 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc48067680 msgr2=0x7fcc481471c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:36.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.565+0000 7fcc47fff700 1 --2- 172.21.15.60:0/3039583529 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc48067680 0x7fcc481471c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.565+0000 7fcc47fff700 1 -- 172.21.15.60:0/3039583529 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc48063ad0 msgr2=0x7fcc481593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:36.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.565+0000 7fcc47fff700 1 --2- 172.21.15.60:0/3039583529 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc48063ad0 0x7fcc481593f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.565+0000 7fcc47fff700 1 -- 172.21.15.60:0/3039583529 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc40009470 con 0x7fcc480b5950 2022-06-29T15:58:36.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.565+0000 7fcc47fff700 1 --2- 172.21.15.60:0/3039583529 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc480b5950 0x7fcc48116ba0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fcc30001800 tx=0x7fcc400091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:36.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.565+0000 7fcc457fa700 1 -- 172.21.15.60:0/3039583529 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc40021070 con 0x7fcc480b5950 2022-06-29T15:58:36.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.565+0000 7fcc457fa700 1 -- 172.21.15.60:0/3039583529 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc400032d0 con 0x7fcc480b5950 2022-06-29T15:58:36.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.565+0000 7fcc4dde2700 1 -- 172.21.15.60:0/3039583529 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc480afe40 con 0x7fcc480b5950 2022-06-29T15:58:36.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.565+0000 7fcc457fa700 1 -- 172.21.15.60:0/3039583529 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc40011d70 con 0x7fcc480b5950 2022-06-29T15:58:36.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.565+0000 7fcc4dde2700 1 -- 172.21.15.60:0/3039583529 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc480af690 con 0x7fcc480b5950 2022-06-29T15:58:36.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.565+0000 7fcc4dde2700 1 -- 172.21.15.60:0/3039583529 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc480573d0 con 0x7fcc480b5950 2022-06-29T15:58:36.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.569+0000 7fcc457fa700 1 -- 172.21.15.60:0/3039583529 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcc4000f070 con 0x7fcc480b5950 2022-06-29T15:58:36.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.569+0000 7fcc457fa700 1 --2- 172.21.15.60:0/3039583529 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcc2c053a70 0x7fcc2c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:36.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.569+0000 7fcc457fa700 1 -- 172.21.15.60:0/3039583529 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcc4006d510 con 0x7fcc480b5950 2022-06-29T15:58:36.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.569+0000 7fcc477fe700 1 --2- 172.21.15.60:0/3039583529 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcc2c053a70 0x7fcc2c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:36.586 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.569+0000 7fcc477fe700 1 --2- 172.21.15.60:0/3039583529 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcc2c053a70 0x7fcc2c055f30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fcc3800d000 tx=0x7fcc38007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:36.586 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.573+0000 7fcc457fa700 1 -- 172.21.15.60:0/3039583529 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcc40044320 con 0x7fcc480b5950 2022-06-29T15:58:36.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.701+0000 7fcc4dde2700 1 -- 172.21.15.60:0/3039583529 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcc480684b0 con 0x7fcc480b5950 2022-06-29T15:58:36.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.701+0000 7fcc457fa700 1 -- 172.21.15.60:0/3039583529 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7fcc40047940 con 0x7fcc480b5950 2022-06-29T15:58:36.709 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T15:58:36.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.705+0000 7fcc4dde2700 1 -- 172.21.15.60:0/3039583529 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcc2c053a70 msgr2=0x7fcc2c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:36.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.705+0000 7fcc4dde2700 1 --2- 172.21.15.60:0/3039583529 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcc2c053a70 0x7fcc2c055f30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fcc3800d000 tx=0x7fcc38007b90).stop 2022-06-29T15:58:36.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.705+0000 7fcc4dde2700 1 -- 172.21.15.60:0/3039583529 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc480b5950 msgr2=0x7fcc48116ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:36.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.705+0000 7fcc4dde2700 1 --2- 172.21.15.60:0/3039583529 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc480b5950 0x7fcc48116ba0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fcc30001800 tx=0x7fcc400091d0).stop 2022-06-29T15:58:36.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.705+0000 7fcc4dde2700 1 -- 172.21.15.60:0/3039583529 shutdown_connections 2022-06-29T15:58:36.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.705+0000 7fcc4dde2700 1 --2- 172.21.15.60:0/3039583529 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcc2c053a70 0x7fcc2c055f30 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.705+0000 7fcc4dde2700 1 --2- 172.21.15.60:0/3039583529 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc480b5950 0x7fcc48116ba0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.705+0000 7fcc4dde2700 1 --2- 172.21.15.60:0/3039583529 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc48067680 0x7fcc481471c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.705+0000 7fcc4dde2700 1 --2- 172.21.15.60:0/3039583529 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc48063ad0 0x7fcc481593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.705+0000 7fcc4dde2700 1 -- 172.21.15.60:0/3039583529 >> 172.21.15.60:0/3039583529 conn(0x7fcc480632c0 msgr2=0x7fcc48145f50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:36.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.705+0000 7fcc4dde2700 1 -- 172.21.15.60:0/3039583529 shutdown_connections 2022-06-29T15:58:36.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.705+0000 7fcc4dde2700 1 -- 172.21.15.60:0/3039583529 wait complete. 2022-06-29T15:58:36.721 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-29T15:58:36.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.829+0000 7f2627594700 1 Processor -- start 2022-06-29T15:58:36.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.829+0000 7f2627594700 1 -- start start 2022-06-29T15:58:36.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.829+0000 7f2627594700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f26200b9c20 0x7f26200b0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:36.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.829+0000 7f2627594700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2620146040 0x7f26200b24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:36.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.829+0000 7f2627594700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f26201464b0 0x7f2620141a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:36.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.829+0000 7f2627594700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26200b1330 con 0x7f26200b9c20 2022-06-29T15:58:36.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.829+0000 7f2627594700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26200b1de0 con 0x7f2620146040 2022-06-29T15:58:36.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.829+0000 7f2627594700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26200ba000 con 0x7f26201464b0 2022-06-29T15:58:36.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.829+0000 7f2626592700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f26200b9c20 0x7f26200b0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:36.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.829+0000 7f2626592700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f26200b9c20 0x7f26200b0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45540/0 (socket says 172.21.15.60:45540) 2022-06-29T15:58:36.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.829+0000 7f2626592700 1 -- 172.21.15.60:0/3420081110 learned_addr learned my addr 172.21.15.60:0/3420081110 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:36.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.829+0000 7f2625d91700 1 --2- 172.21.15.60:0/3420081110 >> v2:172.21.15.92:3300/0 conn(0x7f2620146040 0x7f26200b24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:36.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.829+0000 7f2626d93700 1 --2- 172.21.15.60:0/3420081110 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f26201464b0 0x7f2620141a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:36.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.829+0000 7f2625d91700 1 -- 172.21.15.60:0/3420081110 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f26201464b0 msgr2=0x7f2620141a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:36.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.829+0000 7f2625d91700 1 --2- 172.21.15.60:0/3420081110 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f26201464b0 0x7f2620141a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.829+0000 7f2625d91700 1 -- 172.21.15.60:0/3420081110 >> v2:172.21.15.60:3300/0 conn(0x7f26200b9c20 msgr2=0x7f26200b0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:36.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.829+0000 7f2625d91700 1 --2- 172.21.15.60:0/3420081110 >> v2:172.21.15.60:3300/0 conn(0x7f26200b9c20 0x7f26200b0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.829+0000 7f2625d91700 1 -- 172.21.15.60:0/3420081110 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26200b00a0 con 0x7f2620146040 2022-06-29T15:58:36.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.829+0000 7f2626592700 1 --2- 172.21.15.60:0/3420081110 >> v2:172.21.15.60:3300/0 conn(0x7f26200b9c20 0x7f26200b0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:36.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.829+0000 7f2625d91700 1 --2- 172.21.15.60:0/3420081110 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2620146040 0x7f26200b24b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f2610002f70 tx=0x7f2610006df0).ready entity=mon.1 client_cookie=d8a01e4a8dcc4fba server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:36.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.829+0000 7f2625590700 1 -- 172.21.15.60:0/3420081110 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f261000d4a0 con 0x7f2620146040 2022-06-29T15:58:36.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.829+0000 7f2625590700 1 -- 172.21.15.60:0/3420081110 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2610000e90 con 0x7f2620146040 2022-06-29T15:58:36.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.829+0000 7f2625590700 1 -- 172.21.15.60:0/3420081110 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f261000d7f0 con 0x7f2620146040 2022-06-29T15:58:36.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.829+0000 7f2627594700 1 -- 172.21.15.60:0/3420081110 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2620146040 msgr2=0x7f26200b24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:36.845 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.829+0000 7f2627594700 1 --2- 172.21.15.60:0/3420081110 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2620146040 0x7f26200b24b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f2610002f70 tx=0x7f2610006df0).stop 2022-06-29T15:58:36.845 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.829+0000 7f2627594700 1 -- 172.21.15.60:0/3420081110 shutdown_connections 2022-06-29T15:58:36.845 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.829+0000 7f2627594700 1 --2- 172.21.15.60:0/3420081110 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f26201464b0 0x7f2620141a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.845 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.829+0000 7f2627594700 1 --2- 172.21.15.60:0/3420081110 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2620146040 0x7f26200b24b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.829+0000 7f2627594700 1 --2- 172.21.15.60:0/3420081110 >> v2:172.21.15.60:3300/0 conn(0x7f26200b9c20 0x7f26200b0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.829+0000 7f2627594700 1 -- 172.21.15.60:0/3420081110 >> 172.21.15.60:0/3420081110 conn(0x7f2620062cb0 msgr2=0x7f26200b91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:36.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.829+0000 7f2627594700 1 -- 172.21.15.60:0/3420081110 shutdown_connections 2022-06-29T15:58:36.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.833+0000 7f2627594700 1 -- 172.21.15.60:0/3420081110 wait complete. 2022-06-29T15:58:36.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.833+0000 7f2627594700 1 Processor -- start 2022-06-29T15:58:36.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.833+0000 7f2627594700 1 -- start start 2022-06-29T15:58:36.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.833+0000 7f2627594700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f26200b9c20 0x7f26201441e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:36.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.833+0000 7f2627594700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2620146040 0x7f26200b82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:36.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.833+0000 7f2627594700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26201464b0 0x7f262013cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:36.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.833+0000 7f2627594700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2620144ca0 con 0x7f26201464b0 2022-06-29T15:58:36.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.833+0000 7f2627594700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2620144f00 con 0x7f2620146040 2022-06-29T15:58:36.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.833+0000 7f2627594700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26201455d0 con 0x7f26200b9c20 2022-06-29T15:58:36.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.833+0000 7f2626d93700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26201464b0 0x7f262013cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:36.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.833+0000 7f2626592700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f26200b9c20 0x7f26201441e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:36.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.833+0000 7f2625d91700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2620146040 0x7f26200b82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:36.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.833+0000 7f2625d91700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2620146040 0x7f26200b82d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:51576/0 (socket says 172.21.15.60:51576) 2022-06-29T15:58:36.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.833+0000 7f2626592700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f26200b9c20 0x7f26201441e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:58638/0 (socket says 172.21.15.60:58638) 2022-06-29T15:58:36.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.833+0000 7f2625d91700 1 -- 172.21.15.60:0/2525375149 learned_addr learned my addr 172.21.15.60:0/2525375149 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:36.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.833+0000 7f2625d91700 1 -- 172.21.15.60:0/2525375149 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f26200b9c20 msgr2=0x7f26201441e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:36.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.833+0000 7f2625d91700 1 --2- 172.21.15.60:0/2525375149 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f26200b9c20 0x7f26201441e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.833+0000 7f2625d91700 1 -- 172.21.15.60:0/2525375149 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26201464b0 msgr2=0x7f262013cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:36.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.833+0000 7f2625d91700 1 --2- 172.21.15.60:0/2525375149 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26201464b0 0x7f262013cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.833+0000 7f2625d91700 1 -- 172.21.15.60:0/2525375149 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2608009530 con 0x7f2620146040 2022-06-29T15:58:36.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.833+0000 7f2625d91700 1 --2- 172.21.15.60:0/2525375149 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2620146040 0x7f26200b82d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f26200ae060 tx=0x7f2610014000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:36.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.833+0000 7f261f7fe700 1 -- 172.21.15.60:0/2525375149 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2610013c60 con 0x7f2620146040 2022-06-29T15:58:36.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.833+0000 7f261f7fe700 1 -- 172.21.15.60:0/2525375149 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2610013dc0 con 0x7f2620146040 2022-06-29T15:58:36.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.833+0000 7f261f7fe700 1 -- 172.21.15.60:0/2525375149 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f261001e5f0 con 0x7f2620146040 2022-06-29T15:58:36.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.833+0000 7f2627594700 1 -- 172.21.15.60:0/2525375149 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f261000b180 con 0x7f2620146040 2022-06-29T15:58:36.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.833+0000 7f2627594700 1 -- 172.21.15.60:0/2525375149 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2620145870 con 0x7f2620146040 2022-06-29T15:58:36.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.837+0000 7f2627594700 1 -- 172.21.15.60:0/2525375149 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26200573d0 con 0x7f2620146040 2022-06-29T15:58:36.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.837+0000 7f261f7fe700 1 -- 172.21.15.60:0/2525375149 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2610025030 con 0x7f2620146040 2022-06-29T15:58:36.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.837+0000 7f261f7fe700 1 --2- 172.21.15.60:0/2525375149 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f26040539e0 0x7f2604055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:36.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.837+0000 7f261f7fe700 1 -- 172.21.15.60:0/2525375149 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f261006cd10 con 0x7f2620146040 2022-06-29T15:58:36.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.837+0000 7f2626592700 1 --2- 172.21.15.60:0/2525375149 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f26040539e0 0x7f2604055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:36.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.837+0000 7f2626592700 1 --2- 172.21.15.60:0/2525375149 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f26040539e0 0x7f2604055ea0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f260800b000 tx=0x7f2608009290).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:36.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.845+0000 7f261f7fe700 1 -- 172.21.15.60:0/2525375149 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2610072020 con 0x7f2620146040 2022-06-29T15:58:36.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.969+0000 7f2627594700 1 -- 172.21.15.60:0/2525375149 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2620067ae0 con 0x7f2620146040 2022-06-29T15:58:36.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.969+0000 7f261f7fe700 1 -- 172.21.15.60:0/2525375149 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7f2610023020 con 0x7f2620146040 2022-06-29T15:58:36.978 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T15:58:36.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.973+0000 7f2627594700 1 -- 172.21.15.60:0/2525375149 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f26040539e0 msgr2=0x7f2604055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:36.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.973+0000 7f2627594700 1 --2- 172.21.15.60:0/2525375149 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f26040539e0 0x7f2604055ea0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f260800b000 tx=0x7f2608009290).stop 2022-06-29T15:58:36.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.973+0000 7f2627594700 1 -- 172.21.15.60:0/2525375149 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2620146040 msgr2=0x7f26200b82d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:36.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.973+0000 7f2627594700 1 --2- 172.21.15.60:0/2525375149 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2620146040 0x7f26200b82d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f26200ae060 tx=0x7f2610014000).stop 2022-06-29T15:58:36.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.973+0000 7f2627594700 1 -- 172.21.15.60:0/2525375149 shutdown_connections 2022-06-29T15:58:36.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.973+0000 7f2627594700 1 --2- 172.21.15.60:0/2525375149 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f26040539e0 0x7f2604055ea0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.973+0000 7f2627594700 1 --2- 172.21.15.60:0/2525375149 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26201464b0 0x7f262013cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.973+0000 7f2627594700 1 --2- 172.21.15.60:0/2525375149 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2620146040 0x7f26200b82d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.973+0000 7f2627594700 1 --2- 172.21.15.60:0/2525375149 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f26200b9c20 0x7f26201441e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:36.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.973+0000 7f2627594700 1 -- 172.21.15.60:0/2525375149 >> 172.21.15.60:0/2525375149 conn(0x7f2620062cb0 msgr2=0x7f26200b40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:36.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.973+0000 7f2627594700 1 -- 172.21.15.60:0/2525375149 shutdown_connections 2022-06-29T15:58:36.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:36.973+0000 7f2627594700 1 -- 172.21.15.60:0/2525375149 wait complete. 2022-06-29T15:58:36.995 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-06-29T15:58:36.996 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T15:58:37.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.093+0000 7fb373234700 1 Processor -- start 2022-06-29T15:58:37.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.093+0000 7fb373234700 1 -- start start 2022-06-29T15:58:37.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.093+0000 7fb373234700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb36c0b9d70 0x7fb36c0b3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:37.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.093+0000 7fb373234700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb36c0b3cc0 0x7fb36c1420b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:37.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.093+0000 7fb373234700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb36c0b2ce0 con 0x7fb36c05d1f0 2022-06-29T15:58:37.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.093+0000 7fb373234700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb36c0b4d10 con 0x7fb36c0b9d70 2022-06-29T15:58:37.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.093+0000 7fb373234700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb36c0ba150 con 0x7fb36c0b3cc0 2022-06-29T15:58:37.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.093+0000 7fb370fd0700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb36c0b9d70 0x7fb36c0b3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:37.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.093+0000 7fb36bfff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb36c05d1f0 0x7fb36c0b1b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53638/0 (socket says 172.21.15.60:53638) 2022-06-29T15:58:37.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.093+0000 7fb36bfff700 1 -- 172.21.15.60:0/358625774 learned_addr learned my addr 172.21.15.60:0/358625774 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:37.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.093+0000 7fb3717d1700 1 --2- 172.21.15.60:0/358625774 >> v2:172.21.15.92:3300/0 conn(0x7fb36c0b3cc0 0x7fb36c1420b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:37.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.093+0000 7fb36b7fe700 1 -- 172.21.15.60:0/358625774 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2352360767 0 0) 0x7fb36c0b2ce0 con 0x7fb36c05d1f0 2022-06-29T15:58:37.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.093+0000 7fb36b7fe700 1 -- 172.21.15.60:0/358625774 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb34c003260 con 0x7fb36c05d1f0 2022-06-29T15:58:37.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.093+0000 7fb370fd0700 1 -- 172.21.15.60:0/358625774 >> v2:172.21.15.92:3300/0 conn(0x7fb36c0b3cc0 msgr2=0x7fb36c1420b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:37.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.093+0000 7fb370fd0700 1 --2- 172.21.15.60:0/358625774 >> v2:172.21.15.92:3300/0 conn(0x7fb36c0b3cc0 0x7fb36c1420b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:37.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.093+0000 7fb370fd0700 1 -- 172.21.15.60:0/358625774 >> v1:172.21.15.60:6789/0 conn(0x7fb36c05d1f0 legacy=0x7fb36c0b1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:37.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.093+0000 7fb370fd0700 1 -- 172.21.15.60:0/358625774 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36c0b4a70 con 0x7fb36c0b9d70 2022-06-29T15:58:37.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.093+0000 7fb370fd0700 1 --2- 172.21.15.60:0/358625774 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb36c0b9d70 0x7fb36c0b3140 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fb35c002f70 tx=0x7fb35c006df0).ready entity=mon.0 client_cookie=c8b2d5bf87162636 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:37.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.093+0000 7fb36b7fe700 1 -- 172.21.15.60:0/358625774 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb35c00d4f0 con 0x7fb36c0b9d70 2022-06-29T15:58:37.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.093+0000 7fb36b7fe700 1 -- 172.21.15.60:0/358625774 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb35c000e90 con 0x7fb36c0b9d70 2022-06-29T15:58:37.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.093+0000 7fb36b7fe700 1 -- 172.21.15.60:0/358625774 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb35c00d840 con 0x7fb36c0b9d70 2022-06-29T15:58:37.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.097+0000 7fb373234700 1 -- 172.21.15.60:0/358625774 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb36c0b9d70 msgr2=0x7fb36c0b3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:37.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.097+0000 7fb373234700 1 --2- 172.21.15.60:0/358625774 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb36c0b9d70 0x7fb36c0b3140 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fb35c002f70 tx=0x7fb35c006df0).stop 2022-06-29T15:58:37.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.097+0000 7fb373234700 1 -- 172.21.15.60:0/358625774 shutdown_connections 2022-06-29T15:58:37.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.097+0000 7fb373234700 1 --2- 172.21.15.60:0/358625774 >> v2:172.21.15.92:3300/0 conn(0x7fb36c0b3cc0 0x7fb36c1420b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:37.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.097+0000 7fb373234700 1 --2- 172.21.15.60:0/358625774 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb36c0b9d70 0x7fb36c0b3140 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:37.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.097+0000 7fb373234700 1 -- 172.21.15.60:0/358625774 >> 172.21.15.60:0/358625774 conn(0x7fb36c062ce0 msgr2=0x7fb36c0b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:37.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.097+0000 7fb373234700 1 -- 172.21.15.60:0/358625774 shutdown_connections 2022-06-29T15:58:37.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.097+0000 7fb373234700 1 -- 172.21.15.60:0/358625774 wait complete. 2022-06-29T15:58:37.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.097+0000 7fb373234700 1 Processor -- start 2022-06-29T15:58:37.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.097+0000 7fb373234700 1 -- start start 2022-06-29T15:58:37.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.101+0000 7fb373234700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb36c05d1f0 0x7fb36c112e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:37.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.101+0000 7fb373234700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb36c0b3cc0 0x7fb36c144c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:37.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.101+0000 7fb373234700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb36c0b9d70 0x7fb36c0af020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:37.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.101+0000 7fb373234700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c145d30 con 0x7fb36c05d1f0 2022-06-29T15:58:37.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.101+0000 7fb373234700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c145270 con 0x7fb36c0b9d70 2022-06-29T15:58:37.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.101+0000 7fb373234700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c0af510 con 0x7fb36c0b3cc0 2022-06-29T15:58:37.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.101+0000 7fb370fd0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb36c05d1f0 0x7fb36c112e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:37.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.101+0000 7fb370fd0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb36c05d1f0 0x7fb36c112e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45558/0 (socket says 172.21.15.60:45558) 2022-06-29T15:58:37.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.101+0000 7fb370fd0700 1 -- 172.21.15.60:0/4197600289 learned_addr learned my addr 172.21.15.60:0/4197600289 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:37.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.101+0000 7fb36bfff700 1 --2- 172.21.15.60:0/4197600289 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb36c0b3cc0 0x7fb36c144c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:37.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.101+0000 7fb3717d1700 1 --2- 172.21.15.60:0/4197600289 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb36c0b9d70 0x7fb36c0af020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:37.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.101+0000 7fb370fd0700 1 -- 172.21.15.60:0/4197600289 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb36c0b3cc0 msgr2=0x7fb36c144c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:37.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.101+0000 7fb370fd0700 1 --2- 172.21.15.60:0/4197600289 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb36c0b3cc0 0x7fb36c144c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:37.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.101+0000 7fb370fd0700 1 -- 172.21.15.60:0/4197600289 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb36c0b9d70 msgr2=0x7fb36c0af020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:37.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.101+0000 7fb370fd0700 1 --2- 172.21.15.60:0/4197600289 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb36c0b9d70 0x7fb36c0af020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:37.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.101+0000 7fb370fd0700 1 -- 172.21.15.60:0/4197600289 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb35c00b180 con 0x7fb36c05d1f0 2022-06-29T15:58:37.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.101+0000 7fb370fd0700 1 --2- 172.21.15.60:0/4197600289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb36c05d1f0 0x7fb36c112e30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fb35c002f70 tx=0x7fb35c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:37.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.101+0000 7fb369ffb700 1 -- 172.21.15.60:0/4197600289 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb35c00de60 con 0x7fb36c05d1f0 2022-06-29T15:58:37.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.101+0000 7fb369ffb700 1 -- 172.21.15.60:0/4197600289 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb35c01e400 con 0x7fb36c05d1f0 2022-06-29T15:58:37.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.101+0000 7fb373234700 1 -- 172.21.15.60:0/4197600289 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb36c152bb0 con 0x7fb36c05d1f0 2022-06-29T15:58:37.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.101+0000 7fb369ffb700 1 -- 172.21.15.60:0/4197600289 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb35c01e760 con 0x7fb36c05d1f0 2022-06-29T15:58:37.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.101+0000 7fb373234700 1 -- 172.21.15.60:0/4197600289 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb36c0af9f0 con 0x7fb36c05d1f0 2022-06-29T15:58:37.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.101+0000 7fb373234700 1 -- 172.21.15.60:0/4197600289 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb36c145430 con 0x7fb36c05d1f0 2022-06-29T15:58:37.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.101+0000 7fb369ffb700 1 -- 172.21.15.60:0/4197600289 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb35c025030 con 0x7fb36c05d1f0 2022-06-29T15:58:37.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.101+0000 7fb369ffb700 1 --2- 172.21.15.60:0/4197600289 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb350053a70 0x7fb350055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:37.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.101+0000 7fb369ffb700 1 -- 172.21.15.60:0/4197600289 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb35c06cc90 con 0x7fb36c05d1f0 2022-06-29T15:58:37.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.105+0000 7fb36bfff700 1 --2- 172.21.15.60:0/4197600289 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb350053a70 0x7fb350055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:37.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.105+0000 7fb36bfff700 1 --2- 172.21.15.60:0/4197600289 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb350053a70 0x7fb350055f30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fb35400b000 tx=0x7fb354009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:37.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.109+0000 7fb369ffb700 1 -- 172.21.15.60:0/4197600289 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb35c043bb0 con 0x7fb36c05d1f0 2022-06-29T15:58:37.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.277+0000 7fb373234700 1 -- 172.21.15.60:0/4197600289 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb36c0573d0 con 0x7fb36c05d1f0 2022-06-29T15:58:37.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.277+0000 7fb369ffb700 1 -- 172.21.15.60:0/4197600289 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2134 (secure 0 0 0) 0x7fb35c023070 con 0x7fb36c05d1f0 2022-06-29T15:58:37.283 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:58:37.284 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":4,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.92:6835/2121698334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2121698334},{"type":"v1","addr":"172.21.15.92:6835","nonce":2121698334}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":4,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T15:58:24.334770+0000","modified":"2022-06-29T15:58:25.338504+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"a","rank":0,"incarnation":3,"state":"up:active","state_seq":2,"addr":"172.21.15.60:6833/3164364318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3164364318},{"type":"v1","addr":"172.21.15.60:6833","nonce":3164364318}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-29T15:58:37.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.277+0000 7fb373234700 1 -- 172.21.15.60:0/4197600289 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb350053a70 msgr2=0x7fb350055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:37.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.277+0000 7fb373234700 1 --2- 172.21.15.60:0/4197600289 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb350053a70 0x7fb350055f30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fb35400b000 tx=0x7fb354009040).stop 2022-06-29T15:58:37.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.277+0000 7fb373234700 1 -- 172.21.15.60:0/4197600289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb36c05d1f0 msgr2=0x7fb36c112e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:37.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.277+0000 7fb373234700 1 --2- 172.21.15.60:0/4197600289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb36c05d1f0 0x7fb36c112e30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fb35c002f70 tx=0x7fb35c014000).stop 2022-06-29T15:58:37.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.277+0000 7fb373234700 1 -- 172.21.15.60:0/4197600289 shutdown_connections 2022-06-29T15:58:37.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.277+0000 7fb373234700 1 --2- 172.21.15.60:0/4197600289 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb350053a70 0x7fb350055f30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:37.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.277+0000 7fb373234700 1 --2- 172.21.15.60:0/4197600289 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb36c0b9d70 0x7fb36c0af020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:37.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.277+0000 7fb373234700 1 --2- 172.21.15.60:0/4197600289 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb36c0b3cc0 0x7fb36c144c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:37.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.277+0000 7fb373234700 1 --2- 172.21.15.60:0/4197600289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb36c05d1f0 0x7fb36c112e30 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:37.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.277+0000 7fb373234700 1 -- 172.21.15.60:0/4197600289 >> 172.21.15.60:0/4197600289 conn(0x7fb36c062ce0 msgr2=0x7fb36c1437a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:37.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.281+0000 7fb373234700 1 -- 172.21.15.60:0/4197600289 shutdown_connections 2022-06-29T15:58:37.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:37.281+0000 7fb373234700 1 -- 172.21.15.60:0/4197600289 wait complete. 2022-06-29T15:58:37.288 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 4 2022-06-29T15:58:37.498 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 4, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T15:58:24.334770+0000', 'modified': '2022-06-29T15:58:25.338504+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4257}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4257': {'gid': 4257, 'name': 'a', 'rank': 0, 'incarnation': 3, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.60:6833/3164364318', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6832', 'nonce': 3164364318}, {'type': 'v1', 'addr': '172.21.15.60:6833', 'nonce': 3164364318}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T15:58:37.499 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T15:58:37.500 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-06-29T15:58:37.614 INFO:teuthology.orchestra.run.smithi060.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":4257,"want_state":"up:active","state":"up:active","rank_uptime":13.263487380999999,"mdsmap_epoch":4,"osdmap_epoch":23,"osdmap_epoch_barrier":21,"uptime":16.513577023} 2022-06-29T15:58:37.614 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 4257, "mdsmap_epoch": 4, "osdmap_epoch": 23, "osdmap_epoch_barrier": 21, "rank_uptime": 13.263487381, "state": "up:active", "uptime": 16.513577023, "want_state": "up:active", "whoami": 0 } 2022-06-29T15:58:37.615 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-06-29T15:58:37.615 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-06-29T15:58:37.615 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi121.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-06-29T15:58:37.615 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:37.616 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-06-29T15:58:37.625 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T15:58:37.643 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T15:58:37.644 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T15:58:37.698 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T15:58:37.699 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T15:58:37.700 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:37.700 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T15:58:37.707 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-06-29T15:58:37.707 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-06-29T15:58:37.710 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T15:58:37.729 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T15:58:37.730 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T15:58:37.786 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T15:58:37.787 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T15:58:37.788 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:37.788 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T15:58:37.794 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:2022-06-29T15:58:37.789+0000 7facf9b94080 -1 init, newargv = 0x556147b5f290 newargc=9 2022-06-29T15:58:37.794 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[12249]: starting ceph client 2022-06-29T15:58:37.803 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[12249]: starting fuse 2022-06-29T15:58:38.898 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T15:58:38.917 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T15:58:38.918 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T15:58:38.974 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T15:58:38.976 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T15:58:38.976 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:38.977 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T15:58:38.982 INFO:teuthology.orchestra.run.smithi121.stdout:52 2022-06-29T15:58:38.983 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-06-29T15:58:38.983 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:38.983 DEBUG:teuthology.orchestra.run.smithi121:> sudo python3 -c ' 2022-06-29T15:58:38.984 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2022-06-29T15:58:38.984 DEBUG:teuthology.orchestra.run.smithi121:> import re 2022-06-29T15:58:38.984 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-06-29T15:58:38.985 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2022-06-29T15:58:38.985 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T15:58:38.986 DEBUG:teuthology.orchestra.run.smithi121:> def find_socket(client_name): 2022-06-29T15:58:38.986 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-06-29T15:58:38.986 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2022-06-29T15:58:38.987 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T15:58:38.987 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2022-06-29T15:58:38.987 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2022-06-29T15:58:38.988 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2022-06-29T15:58:38.988 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2022-06-29T15:58:38.988 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T15:58:38.989 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2022-06-29T15:58:38.989 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-06-29T15:58:38.989 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2022-06-29T15:58:38.990 DEBUG:teuthology.orchestra.run.smithi121:> return f 2022-06-29T15:58:38.990 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-06-29T15:58:38.990 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T15:58:38.991 DEBUG:teuthology.orchestra.run.smithi121:> print(find_socket("client.0")) 2022-06-29T15:58:38.991 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-06-29T15:58:39.038 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.0.12249.asok 2022-06-29T15:58:39.043 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.12249.asok 2022-06-29T15:58:39.044 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:39.044 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.12249.asok status 2022-06-29T15:58:39.131 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2022-06-29T15:58:39.131 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2022-06-29T15:58:39.131 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "f25ffac72e5a9e63860ff7deb6767a11f0559a16", 2022-06-29T15:58:39.132 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 15.2.16-304-gf25ffac7 (f25ffac72e5a9e63860ff7deb6767a11f0559a16) octopus (stable)", 2022-06-29T15:58:39.132 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "0", 2022-06-29T15:58:39.132 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2022-06-29T15:58:39.132 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-06-29T15:58:39.133 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "12249", 2022-06-29T15:58:39.133 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2022-06-29T15:58:39.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T15:58:39.133 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2022-06-29T15:58:39.134 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2022-06-29T15:58:39.134 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 4639, 2022-06-29T15:58:39.134 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2022-06-29T15:58:39.134 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2022-06-29T15:58:39.135 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2022-06-29T15:58:39.135 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4639 2022-06-29T15:58:39.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T15:58:39.135 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-06-29T15:58:39.136 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-06-29T15:58:39.136 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-06-29T15:58:39.136 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2748841011 2022-06-29T15:58:39.136 INFO:teuthology.orchestra.run.smithi121.stdout: } 2022-06-29T15:58:39.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T15:58:39.137 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-06-29T15:58:39.137 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-06-29T15:58:39.137 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-06-29T15:58:39.137 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2748841011 2022-06-29T15:58:39.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T15:58:39.138 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.4639 172.21.15.121:0/2748841011", 2022-06-29T15:58:39.138 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "172.21.15.121:0/2748841011", 2022-06-29T15:58:39.138 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2022-06-29T15:58:39.139 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 4, 2022-06-29T15:58:39.139 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 23, 2022-06-29T15:58:39.139 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2022-06-29T15:58:39.139 INFO:teuthology.orchestra.run.smithi121.stdout: "blacklisted": false, 2022-06-29T15:58:39.140 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2022-06-29T15:58:39.140 INFO:teuthology.orchestra.run.smithi121.stdout:} 2022-06-29T15:58:39.141 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T15:58:39.255 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.249+0000 7fad6a83e700 1 Processor -- start 2022-06-29T15:58:39.255 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.249+0000 7fad6a83e700 1 -- start start 2022-06-29T15:58:39.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.249+0000 7fad6a83e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad6413a920 0x7fad64143f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:39.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.249+0000 7fad6a83e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fad640ae270 0x7fad640b5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:39.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.249+0000 7fad6a83e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad6413ad00 con 0x7fad640b5b00 2022-06-29T15:58:39.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.249+0000 7fad6a83e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad64144420 con 0x7fad640ae270 2022-06-29T15:58:39.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.249+0000 7fad6a83e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad64144670 con 0x7fad6413a920 2022-06-29T15:58:39.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.249+0000 7fad637fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fad640ae270 0x7fad640b5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:39.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.249+0000 7fad637fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fad640ae270 0x7fad640b5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45566/0 (socket says 172.21.15.60:45566) 2022-06-29T15:58:39.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.249+0000 7fad637fe700 1 -- 172.21.15.60:0/1406615764 learned_addr learned my addr 172.21.15.60:0/1406615764 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:39.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.249+0000 7fad63fff700 1 --2- 172.21.15.60:0/1406615764 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad6413a920 0x7fad64143f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:39.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.249+0000 7fad62ffd700 1 -- 172.21.15.60:0/1406615764 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 425274555 0 0) 0x7fad6413ad00 con 0x7fad640b5b00 2022-06-29T15:58:39.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.249+0000 7fad637fe700 1 -- 172.21.15.60:0/1406615764 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad6413a920 msgr2=0x7fad64143f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:39.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.249+0000 7fad637fe700 1 --2- 172.21.15.60:0/1406615764 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad6413a920 0x7fad64143f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:39.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.249+0000 7fad637fe700 1 -- 172.21.15.60:0/1406615764 >> v1:172.21.15.92:6789/0 conn(0x7fad640b5b00 legacy=0x7fad64140ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:39.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.249+0000 7fad637fe700 1 -- 172.21.15.60:0/1406615764 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad64144910 con 0x7fad640ae270 2022-06-29T15:58:39.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.253+0000 7fad637fe700 1 --2- 172.21.15.60:0/1406615764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad640ae270 0x7fad640b5610 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fad4c002f70 tx=0x7fad4c006eb0).ready entity=mon.0 client_cookie=b74a7a27180c0d79 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:39.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.253+0000 7fad62ffd700 1 -- 172.21.15.60:0/1406615764 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad4c00d570 con 0x7fad640ae270 2022-06-29T15:58:39.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.253+0000 7fad62ffd700 1 -- 172.21.15.60:0/1406615764 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad4c000e90 con 0x7fad640ae270 2022-06-29T15:58:39.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.253+0000 7fad62ffd700 1 -- 172.21.15.60:0/1406615764 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad4c00d8c0 con 0x7fad640ae270 2022-06-29T15:58:39.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.253+0000 7fad6a83e700 1 -- 172.21.15.60:0/1406615764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad640ae270 msgr2=0x7fad640b5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:39.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.253+0000 7fad6a83e700 1 --2- 172.21.15.60:0/1406615764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad640ae270 0x7fad640b5610 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fad4c002f70 tx=0x7fad4c006eb0).stop 2022-06-29T15:58:39.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.253+0000 7fad6a83e700 1 -- 172.21.15.60:0/1406615764 shutdown_connections 2022-06-29T15:58:39.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.253+0000 7fad6a83e700 1 --2- 172.21.15.60:0/1406615764 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad640ae270 0x7fad640b5610 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:39.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.253+0000 7fad6a83e700 1 --2- 172.21.15.60:0/1406615764 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad6413a920 0x7fad64143f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:39.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.253+0000 7fad6a83e700 1 -- 172.21.15.60:0/1406615764 >> 172.21.15.60:0/1406615764 conn(0x7fad640632c0 msgr2=0x7fad640b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:39.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.253+0000 7fad6a83e700 1 -- 172.21.15.60:0/1406615764 shutdown_connections 2022-06-29T15:58:39.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.253+0000 7fad6a83e700 1 -- 172.21.15.60:0/1406615764 wait complete. 2022-06-29T15:58:39.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.257+0000 7fad6a83e700 1 Processor -- start 2022-06-29T15:58:39.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.257+0000 7fad6a83e700 1 -- start start 2022-06-29T15:58:39.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.257+0000 7fad6a83e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad640ae270 0x7fad6413c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:39.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.257+0000 7fad6a83e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad640b5b00 0x7fad640b4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:39.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.257+0000 7fad6a83e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad6413a920 0x7fad6413d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:39.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.257+0000 7fad6a83e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad64154200 con 0x7fad640b5b00 2022-06-29T15:58:39.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.257+0000 7fad6a83e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad64153f60 con 0x7fad6413a920 2022-06-29T15:58:39.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.257+0000 7fad6a83e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad6414fe60 con 0x7fad640ae270 2022-06-29T15:58:39.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.257+0000 7fad637fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad640b5b00 0x7fad640b4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:39.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.257+0000 7fad637fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad640b5b00 0x7fad640b4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45574/0 (socket says 172.21.15.60:45574) 2022-06-29T15:58:39.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.257+0000 7fad637fe700 1 -- 172.21.15.60:0/108710303 learned_addr learned my addr 172.21.15.60:0/108710303 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:39.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.257+0000 7fad63fff700 1 --2- 172.21.15.60:0/108710303 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad640ae270 0x7fad6413c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:39.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.257+0000 7fad68ddb700 1 --2- 172.21.15.60:0/108710303 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad6413a920 0x7fad6413d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:39.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.257+0000 7fad637fe700 1 -- 172.21.15.60:0/108710303 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad640ae270 msgr2=0x7fad6413c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:39.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.257+0000 7fad637fe700 1 --2- 172.21.15.60:0/108710303 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad640ae270 0x7fad6413c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:39.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.257+0000 7fad637fe700 1 -- 172.21.15.60:0/108710303 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad6413a920 msgr2=0x7fad6413d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:39.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.257+0000 7fad637fe700 1 --2- 172.21.15.60:0/108710303 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad6413a920 0x7fad6413d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:39.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.257+0000 7fad637fe700 1 -- 172.21.15.60:0/108710303 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad4c00b1e0 con 0x7fad640b5b00 2022-06-29T15:58:39.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.257+0000 7fad63fff700 1 --2- 172.21.15.60:0/108710303 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad640ae270 0x7fad6413c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:58:39.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.257+0000 7fad68ddb700 1 --2- 172.21.15.60:0/108710303 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad6413a920 0x7fad6413d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:58:39.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.257+0000 7fad637fe700 1 --2- 172.21.15.60:0/108710303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad640b5b00 0x7fad640b4110 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fad4c002f70 tx=0x7fad4c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:39.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.257+0000 7fad617fa700 1 -- 172.21.15.60:0/108710303 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad4c011950 con 0x7fad640b5b00 2022-06-29T15:58:39.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.257+0000 7fad617fa700 1 -- 172.21.15.60:0/108710303 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad4c011ab0 con 0x7fad640b5b00 2022-06-29T15:58:39.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.257+0000 7fad6a83e700 1 -- 172.21.15.60:0/108710303 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad64153c80 con 0x7fad640b5b00 2022-06-29T15:58:39.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.257+0000 7fad617fa700 1 -- 172.21.15.60:0/108710303 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad4c011e10 con 0x7fad640b5b00 2022-06-29T15:58:39.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.257+0000 7fad6a83e700 1 -- 172.21.15.60:0/108710303 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad640b4600 con 0x7fad640b5b00 2022-06-29T15:58:39.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.261+0000 7fad6a83e700 1 -- 172.21.15.60:0/108710303 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad6413cbe0 con 0x7fad640b5b00 2022-06-29T15:58:39.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.261+0000 7fad617fa700 1 -- 172.21.15.60:0/108710303 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fad4c025030 con 0x7fad640b5b00 2022-06-29T15:58:39.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.261+0000 7fad617fa700 1 --2- 172.21.15.60:0/108710303 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fad480539e0 0x7fad48055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:39.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.261+0000 7fad617fa700 1 -- 172.21.15.60:0/108710303 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fad4c06d080 con 0x7fad640b5b00 2022-06-29T15:58:39.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.261+0000 7fad63fff700 1 --2- 172.21.15.60:0/108710303 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fad480539e0 0x7fad48055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:39.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.261+0000 7fad63fff700 1 --2- 172.21.15.60:0/108710303 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fad480539e0 0x7fad48055ea0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fad5400b000 tx=0x7fad54008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:39.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.269+0000 7fad617fa700 1 -- 172.21.15.60:0/108710303 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fad6413cbe0 con 0x7fad640b5b00 2022-06-29T15:58:39.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.393+0000 7fad6a83e700 1 -- 172.21.15.60:0/108710303 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad6413cbe0 con 0x7fad640b5b00 2022-06-29T15:58:39.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.397+0000 7fad617fa700 1 -- 172.21.15.60:0/108710303 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16038 (secure 0 0 0) 0x7fad6413cbe0 con 0x7fad640b5b00 2022-06-29T15:58:39.402 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:58:39.403 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":23,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T15:58:26.345644+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-06-29T15:58:22.424395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-06-29T15:58:23.343438+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T15:58:39.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.397+0000 7fad6a83e700 1 -- 172.21.15.60:0/108710303 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fad480539e0 msgr2=0x7fad48055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:39.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.397+0000 7fad6a83e700 1 --2- 172.21.15.60:0/108710303 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fad480539e0 0x7fad48055ea0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fad5400b000 tx=0x7fad54008040).stop 2022-06-29T15:58:39.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.397+0000 7fad6a83e700 1 -- 172.21.15.60:0/108710303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad640b5b00 msgr2=0x7fad640b4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:39.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.397+0000 7fad6a83e700 1 --2- 172.21.15.60:0/108710303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad640b5b00 0x7fad640b4110 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fad4c002f70 tx=0x7fad4c015000).stop 2022-06-29T15:58:39.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.397+0000 7fad6a83e700 1 -- 172.21.15.60:0/108710303 shutdown_connections 2022-06-29T15:58:39.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.397+0000 7fad6a83e700 1 --2- 172.21.15.60:0/108710303 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fad480539e0 0x7fad48055ea0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:39.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.397+0000 7fad6a83e700 1 --2- 172.21.15.60:0/108710303 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad6413a920 0x7fad6413d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:39.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.397+0000 7fad6a83e700 1 --2- 172.21.15.60:0/108710303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad640b5b00 0x7fad640b4110 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:39.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.397+0000 7fad6a83e700 1 --2- 172.21.15.60:0/108710303 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fad640ae270 0x7fad6413c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:39.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.397+0000 7fad6a83e700 1 -- 172.21.15.60:0/108710303 >> 172.21.15.60:0/108710303 conn(0x7fad640632c0 msgr2=0x7fad6405ca90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:39.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.397+0000 7fad6a83e700 1 -- 172.21.15.60:0/108710303 shutdown_connections 2022-06-29T15:58:39.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.397+0000 7fad6a83e700 1 -- 172.21.15.60:0/108710303 wait complete. 2022-06-29T15:58:39.416 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-29T15:58:22.424395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-06-29T15:58:23.343438+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-29T15:58:39.417 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T15:58:39.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.521+0000 7fe119935700 1 Processor -- start 2022-06-29T15:58:39.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.521+0000 7fe119935700 1 -- start start 2022-06-29T15:58:39.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.521+0000 7fe119935700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe1140b9d70 0x7fe1141465d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:39.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.521+0000 7fe119935700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe114061d90 0x7fe114142b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:39.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.521+0000 7fe119935700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1140b4f50 con 0x7fe1140b3d30 2022-06-29T15:58:39.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.521+0000 7fe119935700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1140ba150 con 0x7fe114061d90 2022-06-29T15:58:39.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.521+0000 7fe119935700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1140b5180 con 0x7fe1140b9d70 2022-06-29T15:58:39.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.521+0000 7fe1137fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe114061d90 0x7fe114142b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:39.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.521+0000 7fe1127fc700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe1140b3d30 0x7fe114146ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53668/0 (socket says 172.21.15.60:53668) 2022-06-29T15:58:39.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.521+0000 7fe1127fc700 1 -- 172.21.15.60:0/2596828044 learned_addr learned my addr 172.21.15.60:0/2596828044 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:39.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.521+0000 7fe112ffd700 1 --2- 172.21.15.60:0/2596828044 >> v2:172.21.15.92:3300/0 conn(0x7fe1140b9d70 0x7fe1141465d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:39.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.521+0000 7fe111ffb700 1 -- 172.21.15.60:0/2596828044 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3011188223 0 0) 0x7fe1140b4f50 con 0x7fe1140b3d30 2022-06-29T15:58:39.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.521+0000 7fe111ffb700 1 -- 172.21.15.60:0/2596828044 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0f8003260 con 0x7fe1140b3d30 2022-06-29T15:58:39.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.521+0000 7fe1137fe700 1 -- 172.21.15.60:0/2596828044 >> v2:172.21.15.92:3300/0 conn(0x7fe1140b9d70 msgr2=0x7fe1141465d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:39.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.521+0000 7fe1137fe700 1 --2- 172.21.15.60:0/2596828044 >> v2:172.21.15.92:3300/0 conn(0x7fe1140b9d70 0x7fe1141465d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:39.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.521+0000 7fe1137fe700 1 -- 172.21.15.60:0/2596828044 >> v1:172.21.15.60:6789/0 conn(0x7fe1140b3d30 legacy=0x7fe114146ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:39.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.521+0000 7fe1137fe700 1 -- 172.21.15.60:0/2596828044 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1141461a0 con 0x7fe114061d90 2022-06-29T15:58:39.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.521+0000 7fe1137fe700 1 --2- 172.21.15.60:0/2596828044 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe114061d90 0x7fe114142b70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fe10c0032f0 tx=0x7fe10c0091d0).ready entity=mon.0 client_cookie=8739d1ffad2ec89b server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:39.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.521+0000 7fe111ffb700 1 -- 172.21.15.60:0/2596828044 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe10c00b600 con 0x7fe114061d90 2022-06-29T15:58:39.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.521+0000 7fe111ffb700 1 -- 172.21.15.60:0/2596828044 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe10c00b760 con 0x7fe114061d90 2022-06-29T15:58:39.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.521+0000 7fe111ffb700 1 -- 172.21.15.60:0/2596828044 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe10c00b8d0 con 0x7fe114061d90 2022-06-29T15:58:39.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.525+0000 7fe119935700 1 -- 172.21.15.60:0/2596828044 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe114061d90 msgr2=0x7fe114142b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:39.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.525+0000 7fe119935700 1 --2- 172.21.15.60:0/2596828044 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe114061d90 0x7fe114142b70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fe10c0032f0 tx=0x7fe10c0091d0).stop 2022-06-29T15:58:39.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.525+0000 7fe119935700 1 -- 172.21.15.60:0/2596828044 shutdown_connections 2022-06-29T15:58:39.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.525+0000 7fe119935700 1 --2- 172.21.15.60:0/2596828044 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe114061d90 0x7fe114142b70 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:39.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.525+0000 7fe119935700 1 --2- 172.21.15.60:0/2596828044 >> v2:172.21.15.92:3300/0 conn(0x7fe1140b9d70 0x7fe1141465d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:39.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.525+0000 7fe119935700 1 -- 172.21.15.60:0/2596828044 >> 172.21.15.60:0/2596828044 conn(0x7fe114062e70 msgr2=0x7fe1140b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:39.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.525+0000 7fe119935700 1 -- 172.21.15.60:0/2596828044 shutdown_connections 2022-06-29T15:58:39.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.525+0000 7fe119935700 1 -- 172.21.15.60:0/2596828044 wait complete. 2022-06-29T15:58:39.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.525+0000 7fe119935700 1 Processor -- start 2022-06-29T15:58:39.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.525+0000 7fe119935700 1 -- start start 2022-06-29T15:58:39.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.525+0000 7fe119935700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe114061d90 0x7fe114149100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:39.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.525+0000 7fe119935700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1140b3d30 0x7fe1140ae120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:39.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.525+0000 7fe119935700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe1140b9d70 0x7fe11413fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:39.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.525+0000 7fe119935700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe114147250 con 0x7fe1140b3d30 2022-06-29T15:58:39.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.525+0000 7fe119935700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe114155470 con 0x7fe114061d90 2022-06-29T15:58:39.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.525+0000 7fe119935700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1140aec80 con 0x7fe1140b9d70 2022-06-29T15:58:39.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.525+0000 7fe1127fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1140b3d30 0x7fe1140ae120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:39.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.525+0000 7fe1127fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1140b3d30 0x7fe1140ae120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45588/0 (socket says 172.21.15.60:45588) 2022-06-29T15:58:39.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.525+0000 7fe1127fc700 1 -- 172.21.15.60:0/1202718370 learned_addr learned my addr 172.21.15.60:0/1202718370 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:39.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.525+0000 7fe112ffd700 1 --2- 172.21.15.60:0/1202718370 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe114061d90 0x7fe114149100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:39.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.525+0000 7fe1137fe700 1 --2- 172.21.15.60:0/1202718370 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe1140b9d70 0x7fe11413fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:39.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.525+0000 7fe1127fc700 1 -- 172.21.15.60:0/1202718370 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe1140b9d70 msgr2=0x7fe11413fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:39.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.525+0000 7fe1127fc700 1 --2- 172.21.15.60:0/1202718370 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe1140b9d70 0x7fe11413fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:39.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.525+0000 7fe1127fc700 1 -- 172.21.15.60:0/1202718370 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe114061d90 msgr2=0x7fe114149100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:39.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.525+0000 7fe1127fc700 1 --2- 172.21.15.60:0/1202718370 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe114061d90 0x7fe114149100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:39.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.525+0000 7fe1127fc700 1 -- 172.21.15.60:0/1202718370 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10c009470 con 0x7fe1140b3d30 2022-06-29T15:58:39.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.525+0000 7fe1137fe700 1 --2- 172.21.15.60:0/1202718370 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe1140b9d70 0x7fe11413fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:58:39.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.525+0000 7fe1127fc700 1 --2- 172.21.15.60:0/1202718370 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1140b3d30 0x7fe1140ae120 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fe108001110 tx=0x7fe108007ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:39.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.525+0000 7fe103fff700 1 -- 172.21.15.60:0/1202718370 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe10800ce20 con 0x7fe1140b3d30 2022-06-29T15:58:39.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.525+0000 7fe103fff700 1 -- 172.21.15.60:0/1202718370 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1080072e0 con 0x7fe1140b3d30 2022-06-29T15:58:39.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.525+0000 7fe119935700 1 -- 172.21.15.60:0/1202718370 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe114146d40 con 0x7fe1140b3d30 2022-06-29T15:58:39.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.525+0000 7fe103fff700 1 -- 172.21.15.60:0/1202718370 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe108005800 con 0x7fe1140b3d30 2022-06-29T15:58:39.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.525+0000 7fe119935700 1 -- 172.21.15.60:0/1202718370 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe114140110 con 0x7fe1140b3d30 2022-06-29T15:58:39.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.529+0000 7fe119935700 1 -- 172.21.15.60:0/1202718370 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe114148090 con 0x7fe1140b3d30 2022-06-29T15:58:39.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.529+0000 7fe103fff700 1 -- 172.21.15.60:0/1202718370 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe108010070 con 0x7fe1140b3d30 2022-06-29T15:58:39.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.529+0000 7fe103fff700 1 --2- 172.21.15.60:0/1202718370 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe0f0053a70 0x7fe0f0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:39.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.529+0000 7fe103fff700 1 -- 172.21.15.60:0/1202718370 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe10806b590 con 0x7fe1140b3d30 2022-06-29T15:58:39.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.529+0000 7fe112ffd700 1 --2- 172.21.15.60:0/1202718370 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe0f0053a70 0x7fe0f0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:39.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.529+0000 7fe112ffd700 1 --2- 172.21.15.60:0/1202718370 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe0f0053a70 0x7fe0f0055f30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fe10400b000 tx=0x7fe104006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:39.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.533+0000 7fe103fff700 1 -- 172.21.15.60:0/1202718370 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe1080424b0 con 0x7fe1140b3d30 2022-06-29T15:58:39.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.657+0000 7fe119935700 1 -- 172.21.15.60:0/1202718370 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fe1140573d0 con 0x7fe1140b3d30 2022-06-29T15:58:39.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.657+0000 7fe103fff700 1 -- 172.21.15.60:0/1202718370 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v23) v1 ==== 110+0+10 (secure 0 0 0) 0x7fe108045ad0 con 0x7fe1140b3d30 2022-06-29T15:58:39.666 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T15:58:39.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.661+0000 7fe119935700 1 -- 172.21.15.60:0/1202718370 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe0f0053a70 msgr2=0x7fe0f0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:39.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.661+0000 7fe119935700 1 --2- 172.21.15.60:0/1202718370 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe0f0053a70 0x7fe0f0055f30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fe10400b000 tx=0x7fe104006df0).stop 2022-06-29T15:58:39.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.661+0000 7fe119935700 1 -- 172.21.15.60:0/1202718370 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1140b3d30 msgr2=0x7fe1140ae120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:39.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.661+0000 7fe119935700 1 --2- 172.21.15.60:0/1202718370 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1140b3d30 0x7fe1140ae120 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fe108001110 tx=0x7fe108007ad0).stop 2022-06-29T15:58:39.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.661+0000 7fe119935700 1 -- 172.21.15.60:0/1202718370 shutdown_connections 2022-06-29T15:58:39.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.661+0000 7fe119935700 1 --2- 172.21.15.60:0/1202718370 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe0f0053a70 0x7fe0f0055f30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:39.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.661+0000 7fe119935700 1 --2- 172.21.15.60:0/1202718370 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe1140b9d70 0x7fe11413fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:39.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.661+0000 7fe119935700 1 --2- 172.21.15.60:0/1202718370 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1140b3d30 0x7fe1140ae120 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:39.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.661+0000 7fe119935700 1 --2- 172.21.15.60:0/1202718370 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe114061d90 0x7fe114149100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:39.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.661+0000 7fe119935700 1 -- 172.21.15.60:0/1202718370 >> 172.21.15.60:0/1202718370 conn(0x7fe114062e70 msgr2=0x7fe1140b9300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:39.672 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.661+0000 7fe119935700 1 -- 172.21.15.60:0/1202718370 shutdown_connections 2022-06-29T15:58:39.672 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.661+0000 7fe119935700 1 -- 172.21.15.60:0/1202718370 wait complete. 2022-06-29T15:58:39.679 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T15:58:39.784 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.777+0000 7f1bbe635700 1 Processor -- start 2022-06-29T15:58:39.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.777+0000 7f1bbe635700 1 -- start start 2022-06-29T15:58:39.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.781+0000 7f1bbe635700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1bb8140410 0x7f1bb8065250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:39.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.781+0000 7f1bbe635700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1bb8064b50 0x7f1bb8128290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:39.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.781+0000 7f1bbe635700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bb80b5d30 con 0x7f1bb80b5950 2022-06-29T15:58:39.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.781+0000 7f1bbe635700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bb8066fa0 con 0x7f1bb8140410 2022-06-29T15:58:39.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.781+0000 7f1bbe635700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bb8143960 con 0x7f1bb8064b50 2022-06-29T15:58:39.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.781+0000 7f1bb77fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1bb8140410 0x7f1bb8065250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:39.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.781+0000 7f1bb7fff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f1bb80b5950 0x7f1bb813fd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53680/0 (socket says 172.21.15.60:53680) 2022-06-29T15:58:39.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.781+0000 7f1bb7fff700 1 -- 172.21.15.60:0/4102320620 learned_addr learned my addr 172.21.15.60:0/4102320620 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:39.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.781+0000 7f1bbcbd2700 1 --2- 172.21.15.60:0/4102320620 >> v2:172.21.15.92:3300/0 conn(0x7f1bb8064b50 0x7f1bb8128290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:39.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.781+0000 7f1bb6ffd700 1 -- 172.21.15.60:0/4102320620 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2989587926 0 0) 0x7f1bb80b5d30 con 0x7f1bb80b5950 2022-06-29T15:58:39.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.781+0000 7f1bb6ffd700 1 -- 172.21.15.60:0/4102320620 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ba4003260 con 0x7f1bb80b5950 2022-06-29T15:58:39.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.781+0000 7f1bbcbd2700 1 -- 172.21.15.60:0/4102320620 >> v2:172.21.15.60:3300/0 conn(0x7f1bb8140410 msgr2=0x7f1bb8065250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:39.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.781+0000 7f1bbcbd2700 1 --2- 172.21.15.60:0/4102320620 >> v2:172.21.15.60:3300/0 conn(0x7f1bb8140410 0x7f1bb8065250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:39.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.781+0000 7f1bbcbd2700 1 -- 172.21.15.60:0/4102320620 >> v1:172.21.15.60:6789/0 conn(0x7f1bb80b5950 legacy=0x7f1bb813fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:39.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.781+0000 7f1bbcbd2700 1 -- 172.21.15.60:0/4102320620 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb8143c00 con 0x7f1bb8064b50 2022-06-29T15:58:39.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.781+0000 7f1bbcbd2700 1 --2- 172.21.15.60:0/4102320620 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1bb8064b50 0x7f1bb8128290 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f1bb00032f0 tx=0x7f1bb00091d0).ready entity=mon.1 client_cookie=b47f69ca5ff9fa5f server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:39.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.781+0000 7f1bb6ffd700 1 -- 172.21.15.60:0/4102320620 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1bb000f8a0 con 0x7f1bb8064b50 2022-06-29T15:58:39.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.781+0000 7f1bb6ffd700 1 -- 172.21.15.60:0/4102320620 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bb000fa00 con 0x7f1bb8064b50 2022-06-29T15:58:39.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.781+0000 7f1bb6ffd700 1 -- 172.21.15.60:0/4102320620 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1bb000fd60 con 0x7f1bb8064b50 2022-06-29T15:58:39.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.781+0000 7f1bbe635700 1 -- 172.21.15.60:0/4102320620 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1bb8064b50 msgr2=0x7f1bb8128290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:39.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.781+0000 7f1bbe635700 1 --2- 172.21.15.60:0/4102320620 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1bb8064b50 0x7f1bb8128290 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f1bb00032f0 tx=0x7f1bb00091d0).stop 2022-06-29T15:58:39.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.781+0000 7f1bbe635700 1 -- 172.21.15.60:0/4102320620 shutdown_connections 2022-06-29T15:58:39.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.781+0000 7f1bbe635700 1 --2- 172.21.15.60:0/4102320620 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1bb8064b50 0x7f1bb8128290 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:39.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.781+0000 7f1bbe635700 1 --2- 172.21.15.60:0/4102320620 >> v2:172.21.15.60:3300/0 conn(0x7f1bb8140410 0x7f1bb8065250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:39.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.781+0000 7f1bbe635700 1 -- 172.21.15.60:0/4102320620 >> 172.21.15.60:0/4102320620 conn(0x7f1bb80632c0 msgr2=0x7f1bb80b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:39.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.781+0000 7f1bbe635700 1 -- 172.21.15.60:0/4102320620 shutdown_connections 2022-06-29T15:58:39.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.781+0000 7f1bbe635700 1 -- 172.21.15.60:0/4102320620 wait complete. 2022-06-29T15:58:39.797 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.785+0000 7f1bbe635700 1 Processor -- start 2022-06-29T15:58:39.797 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.785+0000 7f1bbe635700 1 -- start start 2022-06-29T15:58:39.802 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.785+0000 7f1bbe635700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1bb8064b50 0x7f1bb8066310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:39.802 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.785+0000 7f1bbe635700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bb80b5950 0x7f1bb813d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:39.803 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.785+0000 7f1bbe635700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1bb8140410 0x7f1bb813e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:39.803 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.785+0000 7f1bbe635700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb813c5b0 con 0x7f1bb80b5950 2022-06-29T15:58:39.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.785+0000 7f1bbe635700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb813e2b0 con 0x7f1bb8140410 2022-06-29T15:58:39.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.785+0000 7f1bbe635700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb813ba20 con 0x7f1bb8064b50 2022-06-29T15:58:39.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.785+0000 7f1bb77fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bb80b5950 0x7f1bb813d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:39.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.785+0000 7f1bb77fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bb80b5950 0x7f1bb813d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45602/0 (socket says 172.21.15.60:45602) 2022-06-29T15:58:39.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.785+0000 7f1bb77fe700 1 -- 172.21.15.60:0/2076630812 learned_addr learned my addr 172.21.15.60:0/2076630812 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:39.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.785+0000 7f1bb7fff700 1 --2- 172.21.15.60:0/2076630812 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1bb8064b50 0x7f1bb8066310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:39.807 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.785+0000 7f1bbcbd2700 1 --2- 172.21.15.60:0/2076630812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1bb8140410 0x7f1bb813e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:39.807 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.785+0000 7f1bb77fe700 1 -- 172.21.15.60:0/2076630812 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1bb8064b50 msgr2=0x7f1bb8066310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:39.807 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.785+0000 7f1bb77fe700 1 --2- 172.21.15.60:0/2076630812 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1bb8064b50 0x7f1bb8066310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:39.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.785+0000 7f1bb77fe700 1 -- 172.21.15.60:0/2076630812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1bb8140410 msgr2=0x7f1bb813e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:39.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.785+0000 7f1bb77fe700 1 --2- 172.21.15.60:0/2076630812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1bb8140410 0x7f1bb813e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:39.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.785+0000 7f1bb77fe700 1 -- 172.21.15.60:0/2076630812 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb0009470 con 0x7f1bb80b5950 2022-06-29T15:58:39.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.785+0000 7f1bbcbd2700 1 --2- 172.21.15.60:0/2076630812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1bb8140410 0x7f1bb813e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:39.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.785+0000 7f1bb77fe700 1 --2- 172.21.15.60:0/2076630812 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bb80b5950 0x7f1bb813d2c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f1bac00c000 tx=0x7f1bac006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:39.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.785+0000 7f1bb57fa700 1 -- 172.21.15.60:0/2076630812 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1bac010780 con 0x7f1bb80b5950 2022-06-29T15:58:39.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.785+0000 7f1bb57fa700 1 -- 172.21.15.60:0/2076630812 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bac0108e0 con 0x7f1bb80b5950 2022-06-29T15:58:39.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.785+0000 7f1bb57fa700 1 -- 172.21.15.60:0/2076630812 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1bac010c40 con 0x7f1bb80b5950 2022-06-29T15:58:39.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.785+0000 7f1bbe635700 1 -- 172.21.15.60:0/2076630812 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bb813d7b0 con 0x7f1bb80b5950 2022-06-29T15:58:39.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.785+0000 7f1bbe635700 1 -- 172.21.15.60:0/2076630812 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bb81423a0 con 0x7f1bb80b5950 2022-06-29T15:58:39.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.789+0000 7f1bbe635700 1 -- 172.21.15.60:0/2076630812 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bb813f860 con 0x7f1bb80b5950 2022-06-29T15:58:39.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.789+0000 7f1bb57fa700 1 -- 172.21.15.60:0/2076630812 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1bac01c460 con 0x7f1bb80b5950 2022-06-29T15:58:39.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.789+0000 7f1bb57fa700 1 --2- 172.21.15.60:0/2076630812 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1b98053950 0x7f1b98055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:39.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.789+0000 7f1bb57fa700 1 -- 172.21.15.60:0/2076630812 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1bac06c300 con 0x7f1bb80b5950 2022-06-29T15:58:39.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.789+0000 7f1bb7fff700 1 --2- 172.21.15.60:0/2076630812 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1b98053950 0x7f1b98055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:39.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.793+0000 7f1bb7fff700 1 --2- 172.21.15.60:0/2076630812 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1b98053950 0x7f1b98055e10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f1ba800d000 tx=0x7f1ba8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:39.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.797+0000 7f1bb57fa700 1 -- 172.21.15.60:0/2076630812 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1bac0430b0 con 0x7f1bb80b5950 2022-06-29T15:58:39.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.925+0000 7f1bbe635700 1 -- 172.21.15.60:0/2076630812 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f1bb80573d0 con 0x7f1bb80b5950 2022-06-29T15:58:39.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.925+0000 7f1bb57fa700 1 -- 172.21.15.60:0/2076630812 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f1bac0466d0 con 0x7f1bb80b5950 2022-06-29T15:58:39.933 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T15:58:39.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.929+0000 7f1bbe635700 1 -- 172.21.15.60:0/2076630812 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1b98053950 msgr2=0x7f1b98055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:39.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.929+0000 7f1bbe635700 1 --2- 172.21.15.60:0/2076630812 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1b98053950 0x7f1b98055e10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f1ba800d000 tx=0x7f1ba8009040).stop 2022-06-29T15:58:39.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.929+0000 7f1bbe635700 1 -- 172.21.15.60:0/2076630812 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bb80b5950 msgr2=0x7f1bb813d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:39.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.929+0000 7f1bbe635700 1 --2- 172.21.15.60:0/2076630812 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bb80b5950 0x7f1bb813d2c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f1bac00c000 tx=0x7f1bac006df0).stop 2022-06-29T15:58:39.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.929+0000 7f1bbe635700 1 -- 172.21.15.60:0/2076630812 shutdown_connections 2022-06-29T15:58:39.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.929+0000 7f1bbe635700 1 --2- 172.21.15.60:0/2076630812 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1b98053950 0x7f1b98055e10 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:39.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.929+0000 7f1bbe635700 1 --2- 172.21.15.60:0/2076630812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1bb8140410 0x7f1bb813e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:39.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.929+0000 7f1bbe635700 1 --2- 172.21.15.60:0/2076630812 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1bb80b5950 0x7f1bb813d2c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:39.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.929+0000 7f1bbe635700 1 --2- 172.21.15.60:0/2076630812 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1bb8064b50 0x7f1bb8066310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:39.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.929+0000 7f1bbe635700 1 -- 172.21.15.60:0/2076630812 >> 172.21.15.60:0/2076630812 conn(0x7f1bb80632c0 msgr2=0x7f1bb805bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:39.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.929+0000 7f1bbe635700 1 -- 172.21.15.60:0/2076630812 shutdown_connections 2022-06-29T15:58:39.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:39.929+0000 7f1bbe635700 1 -- 172.21.15.60:0/2076630812 wait complete. 2022-06-29T15:58:39.945 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-29T15:58:40.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.049+0000 7f85a2e11700 1 Processor -- start 2022-06-29T15:58:40.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.049+0000 7f85a2e11700 1 -- start start 2022-06-29T15:58:40.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.049+0000 7f85a2e11700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f859c061620 0x7f859c1407b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:40.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.049+0000 7f85a2e11700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f859c144020 con 0x7f859c0b6a50 2022-06-29T15:58:40.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.049+0000 7f85a2e11700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f859c0b6e30 con 0x7f859c062260 2022-06-29T15:58:40.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.049+0000 7f85a2e11700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f859c0b2370 con 0x7f859c061620 2022-06-29T15:58:40.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.049+0000 7f85a0bad700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f859c0b6a50 0x7f859c1446f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53694/0 (socket says 172.21.15.60:53694) 2022-06-29T15:58:40.058 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.049+0000 7f85a0bad700 1 -- 172.21.15.60:0/1199707050 learned_addr learned my addr 172.21.15.60:0/1199707050 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:40.058 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.049+0000 7f85a13ae700 1 --2- 172.21.15.60:0/1199707050 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f859c061620 0x7f859c1407b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:40.058 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.049+0000 7f859b7fe700 1 -- 172.21.15.60:0/1199707050 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2380514596 0 0) 0x7f859c144020 con 0x7f859c0b6a50 2022-06-29T15:58:40.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.049+0000 7f859b7fe700 1 -- 172.21.15.60:0/1199707050 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8580003260 con 0x7f859c0b6a50 2022-06-29T15:58:40.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.049+0000 7f859b7fe700 1 -- 172.21.15.60:0/1199707050 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2150314358 0 0) 0x7f859c0b6e30 con 0x7f859c062260 2022-06-29T15:58:40.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.049+0000 7f859b7fe700 1 -- 172.21.15.60:0/1199707050 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f859c144020 con 0x7f859c062260 2022-06-29T15:58:40.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.049+0000 7f85a13ae700 1 -- 172.21.15.60:0/1199707050 >> v1:172.21.15.92:6789/0 conn(0x7f859c062260 legacy=0x7f859c1400a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:40.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.049+0000 7f85a13ae700 1 -- 172.21.15.60:0/1199707050 >> v1:172.21.15.60:6789/0 conn(0x7f859c0b6a50 legacy=0x7f859c1446f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:40.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.049+0000 7f85a13ae700 1 -- 172.21.15.60:0/1199707050 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f859c144250 con 0x7f859c061620 2022-06-29T15:58:40.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.049+0000 7f85a13ae700 1 --2- 172.21.15.60:0/1199707050 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f859c061620 0x7f859c1407b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f85940032f0 tx=0x7f85940091d0).ready entity=mon.2 client_cookie=2cbab533f8b2ae36 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:40.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.049+0000 7f859b7fe700 1 -- 172.21.15.60:0/1199707050 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f859400b640 con 0x7f859c061620 2022-06-29T15:58:40.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.049+0000 7f859b7fe700 1 -- 172.21.15.60:0/1199707050 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f859400b7a0 con 0x7f859c061620 2022-06-29T15:58:40.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.049+0000 7f859b7fe700 1 -- 172.21.15.60:0/1199707050 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f859400b910 con 0x7f859c061620 2022-06-29T15:58:40.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.053+0000 7f85a2e11700 1 -- 172.21.15.60:0/1199707050 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f859c061620 msgr2=0x7f859c1407b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:40.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.053+0000 7f85a2e11700 1 --2- 172.21.15.60:0/1199707050 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f859c061620 0x7f859c1407b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f85940032f0 tx=0x7f85940091d0).stop 2022-06-29T15:58:40.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.053+0000 7f85a2e11700 1 -- 172.21.15.60:0/1199707050 shutdown_connections 2022-06-29T15:58:40.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.053+0000 7f85a2e11700 1 --2- 172.21.15.60:0/1199707050 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f859c061620 0x7f859c1407b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:40.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.053+0000 7f85a2e11700 1 -- 172.21.15.60:0/1199707050 >> 172.21.15.60:0/1199707050 conn(0x7f859c0643e0 msgr2=0x7f859c0b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:40.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.053+0000 7f85a2e11700 1 -- 172.21.15.60:0/1199707050 shutdown_connections 2022-06-29T15:58:40.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.053+0000 7f85a2e11700 1 -- 172.21.15.60:0/1199707050 wait complete. 2022-06-29T15:58:40.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.053+0000 7f85a2e11700 1 Processor -- start 2022-06-29T15:58:40.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.053+0000 7f85a2e11700 1 -- start start 2022-06-29T15:58:40.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.053+0000 7f85a2e11700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f859c061620 0x7f859c066030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:40.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.053+0000 7f85a2e11700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f859c062260 0x7f859c0916d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:40.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.053+0000 7f85a2e11700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f859c0b6a50 0x7f859c08f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:40.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.053+0000 7f85a2e11700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859c150da0 con 0x7f859c061620 2022-06-29T15:58:40.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.053+0000 7f85a2e11700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859c090f90 con 0x7f859c062260 2022-06-29T15:58:40.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.053+0000 7f85a2e11700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f859c091230 con 0x7f859c0b6a50 2022-06-29T15:58:40.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.053+0000 7f85a0bad700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f859c061620 0x7f859c066030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:40.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.053+0000 7f85a0bad700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f859c061620 0x7f859c066030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45614/0 (socket says 172.21.15.60:45614) 2022-06-29T15:58:40.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.053+0000 7f85a0bad700 1 -- 172.21.15.60:0/1071271407 learned_addr learned my addr 172.21.15.60:0/1071271407 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:40.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.053+0000 7f85a13ae700 1 --2- 172.21.15.60:0/1071271407 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f859c0b6a50 0x7f859c08f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:40.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.053+0000 7f859bfff700 1 --2- 172.21.15.60:0/1071271407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f859c062260 0x7f859c0916d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:40.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.057+0000 7f85a0bad700 1 -- 172.21.15.60:0/1071271407 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f859c0b6a50 msgr2=0x7f859c08f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:40.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.057+0000 7f85a0bad700 1 --2- 172.21.15.60:0/1071271407 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f859c0b6a50 0x7f859c08f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:40.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.057+0000 7f85a0bad700 1 -- 172.21.15.60:0/1071271407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f859c062260 msgr2=0x7f859c0916d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:40.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.057+0000 7f85a0bad700 1 --2- 172.21.15.60:0/1071271407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f859c062260 0x7f859c0916d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:40.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.057+0000 7f85a0bad700 1 -- 172.21.15.60:0/1071271407 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8594009470 con 0x7f859c061620 2022-06-29T15:58:40.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.057+0000 7f859bfff700 1 --2- 172.21.15.60:0/1071271407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f859c062260 0x7f859c0916d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:58:40.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.057+0000 7f85a13ae700 1 --2- 172.21.15.60:0/1071271407 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f859c0b6a50 0x7f859c08f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:58:40.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.057+0000 7f85a0bad700 1 --2- 172.21.15.60:0/1071271407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f859c061620 0x7f859c066030 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f8590001110 tx=0x7f8590007ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:40.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.057+0000 7f8599ffb700 1 -- 172.21.15.60:0/1071271407 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85900072e0 con 0x7f859c061620 2022-06-29T15:58:40.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.057+0000 7f85a2e11700 1 -- 172.21.15.60:0/1071271407 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f859c091bc0 con 0x7f859c061620 2022-06-29T15:58:40.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.057+0000 7f8599ffb700 1 -- 172.21.15.60:0/1071271407 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8590007440 con 0x7f859c061620 2022-06-29T15:58:40.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.057+0000 7f85a2e11700 1 -- 172.21.15.60:0/1071271407 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f859c08fc60 con 0x7f859c061620 2022-06-29T15:58:40.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.057+0000 7f8599ffb700 1 -- 172.21.15.60:0/1071271407 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85900055f0 con 0x7f859c061620 2022-06-29T15:58:40.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.057+0000 7f85a2e11700 1 -- 172.21.15.60:0/1071271407 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f859c0913f0 con 0x7f859c061620 2022-06-29T15:58:40.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.057+0000 7f8599ffb700 1 -- 172.21.15.60:0/1071271407 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8590010070 con 0x7f859c061620 2022-06-29T15:58:40.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.057+0000 7f8599ffb700 1 --2- 172.21.15.60:0/1071271407 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f857c053a70 0x7f857c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:40.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.057+0000 7f8599ffb700 1 -- 172.21.15.60:0/1071271407 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f859006b510 con 0x7f859c061620 2022-06-29T15:58:40.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.057+0000 7f859bfff700 1 --2- 172.21.15.60:0/1071271407 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f857c053a70 0x7f857c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:40.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.057+0000 7f859bfff700 1 --2- 172.21.15.60:0/1071271407 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f857c053a70 0x7f857c055f30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f858800b000 tx=0x7f8588009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:40.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.061+0000 7f8599ffb700 1 -- 172.21.15.60:0/1071271407 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8590042390 con 0x7f859c061620 2022-06-29T15:58:40.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.189+0000 7f85a2e11700 1 -- 172.21.15.60:0/1071271407 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f859c08feb0 con 0x7f859c061620 2022-06-29T15:58:40.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.189+0000 7f8599ffb700 1 -- 172.21.15.60:0/1071271407 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7f85900459b0 con 0x7f859c061620 2022-06-29T15:58:40.195 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T15:58:40.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.189+0000 7f85a2e11700 1 -- 172.21.15.60:0/1071271407 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f857c053a70 msgr2=0x7f857c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:40.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.189+0000 7f85a2e11700 1 --2- 172.21.15.60:0/1071271407 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f857c053a70 0x7f857c055f30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f858800b000 tx=0x7f8588009040).stop 2022-06-29T15:58:40.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.189+0000 7f85a2e11700 1 -- 172.21.15.60:0/1071271407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f859c061620 msgr2=0x7f859c066030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:40.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.189+0000 7f85a2e11700 1 --2- 172.21.15.60:0/1071271407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f859c061620 0x7f859c066030 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f8590001110 tx=0x7f8590007ad0).stop 2022-06-29T15:58:40.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.189+0000 7f85a2e11700 1 -- 172.21.15.60:0/1071271407 shutdown_connections 2022-06-29T15:58:40.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.189+0000 7f85a2e11700 1 --2- 172.21.15.60:0/1071271407 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f857c053a70 0x7f857c055f30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:40.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.189+0000 7f85a2e11700 1 --2- 172.21.15.60:0/1071271407 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f859c0b6a50 0x7f859c08f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:40.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.189+0000 7f85a2e11700 1 --2- 172.21.15.60:0/1071271407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f859c062260 0x7f859c0916d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:40.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.189+0000 7f85a2e11700 1 --2- 172.21.15.60:0/1071271407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f859c061620 0x7f859c066030 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:40.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.189+0000 7f85a2e11700 1 -- 172.21.15.60:0/1071271407 >> 172.21.15.60:0/1071271407 conn(0x7f859c0643e0 msgr2=0x7f859c0b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:40.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.189+0000 7f85a2e11700 1 -- 172.21.15.60:0/1071271407 shutdown_connections 2022-06-29T15:58:40.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.189+0000 7f85a2e11700 1 -- 172.21.15.60:0/1071271407 wait complete. 2022-06-29T15:58:40.209 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-29T15:58:40.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.317+0000 7f60e359e700 1 Processor -- start 2022-06-29T15:58:40.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.317+0000 7f60e359e700 1 -- start start 2022-06-29T15:58:40.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.321+0000 7f60e359e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60e405c5b0 0x7f60e405c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:40.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.321+0000 7f60e359e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f60e4060610 0x7f60e413c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:40.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.321+0000 7f60e359e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60e40b0a40 con 0x7f60e413cc30 2022-06-29T15:58:40.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.321+0000 7f60e359e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60e40b0850 con 0x7f60e4060610 2022-06-29T15:58:40.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.321+0000 7f60e359e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60e40b0ce0 con 0x7f60e405c5b0 2022-06-29T15:58:40.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.321+0000 7f60e259c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60e405c5b0 0x7f60e405c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:40.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.321+0000 7f60d9d9b700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f60e4060610 0x7f60e413c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:40.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.321+0000 7f60e2d9d700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f60e413cc30 0x7f60e4142a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:57956/0 (socket says 172.21.15.60:57956) 2022-06-29T15:58:40.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.321+0000 7f60e2d9d700 1 -- 172.21.15.60:0/3219280099 learned_addr learned my addr 172.21.15.60:0/3219280099 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:40.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.321+0000 7f60e1d9b700 1 -- 172.21.15.60:0/3219280099 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2074396237 0 0) 0x7f60e40b0a40 con 0x7f60e413cc30 2022-06-29T15:58:40.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.321+0000 7f60e1d9b700 1 -- 172.21.15.60:0/3219280099 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60c8003260 con 0x7f60e413cc30 2022-06-29T15:58:40.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.321+0000 7f60e259c700 1 -- 172.21.15.60:0/3219280099 >> v2:172.21.15.92:3300/0 conn(0x7f60e4060610 msgr2=0x7f60e413c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:40.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.321+0000 7f60e259c700 1 --2- 172.21.15.60:0/3219280099 >> v2:172.21.15.92:3300/0 conn(0x7f60e4060610 0x7f60e413c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:40.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.321+0000 7f60e259c700 1 -- 172.21.15.60:0/3219280099 >> v1:172.21.15.92:6789/0 conn(0x7f60e413cc30 legacy=0x7f60e4142a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:40.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.321+0000 7f60e259c700 1 -- 172.21.15.60:0/3219280099 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e405f350 con 0x7f60e405c5b0 2022-06-29T15:58:40.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.321+0000 7f60e259c700 1 --2- 172.21.15.60:0/3219280099 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60e405c5b0 0x7f60e405c990 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f60d4002f70 tx=0x7f60d4006df0).ready entity=mon.2 client_cookie=7bbc62f52b1a02b server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:40.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.321+0000 7f60e1d9b700 1 -- 172.21.15.60:0/3219280099 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60d400d4a0 con 0x7f60e405c5b0 2022-06-29T15:58:40.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.321+0000 7f60e1d9b700 1 -- 172.21.15.60:0/3219280099 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60d4000e90 con 0x7f60e405c5b0 2022-06-29T15:58:40.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.321+0000 7f60e1d9b700 1 -- 172.21.15.60:0/3219280099 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60d400d7f0 con 0x7f60e405c5b0 2022-06-29T15:58:40.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.321+0000 7f60e359e700 1 -- 172.21.15.60:0/3219280099 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60e405c5b0 msgr2=0x7f60e405c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:40.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.321+0000 7f60e359e700 1 --2- 172.21.15.60:0/3219280099 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60e405c5b0 0x7f60e405c990 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f60d4002f70 tx=0x7f60d4006df0).stop 2022-06-29T15:58:40.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.321+0000 7f60e359e700 1 -- 172.21.15.60:0/3219280099 shutdown_connections 2022-06-29T15:58:40.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.321+0000 7f60e359e700 1 --2- 172.21.15.60:0/3219280099 >> v2:172.21.15.92:3300/0 conn(0x7f60e4060610 0x7f60e413c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:40.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.321+0000 7f60e359e700 1 --2- 172.21.15.60:0/3219280099 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60e405c5b0 0x7f60e405c990 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:40.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.321+0000 7f60e359e700 1 -- 172.21.15.60:0/3219280099 >> 172.21.15.60:0/3219280099 conn(0x7f60e4061c80 msgr2=0x7f60e40637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:40.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.321+0000 7f60e359e700 1 -- 172.21.15.60:0/3219280099 shutdown_connections 2022-06-29T15:58:40.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.321+0000 7f60e359e700 1 -- 172.21.15.60:0/3219280099 wait complete. 2022-06-29T15:58:40.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.325+0000 7f60e359e700 1 Processor -- start 2022-06-29T15:58:40.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.325+0000 7f60e359e700 1 -- start start 2022-06-29T15:58:40.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.325+0000 7f60e359e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f60e405c5b0 0x7f60e41479b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:40.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.325+0000 7f60e359e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60e4060610 0x7f60e413bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:40.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.325+0000 7f60e359e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60e413cc30 0x7f60e413c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:40.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.325+0000 7f60e359e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e4143ed0 con 0x7f60e405c5b0 2022-06-29T15:58:40.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.325+0000 7f60e359e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e413c570 con 0x7f60e413cc30 2022-06-29T15:58:40.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.325+0000 7f60e359e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60e4146440 con 0x7f60e4060610 2022-06-29T15:58:40.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.325+0000 7f60e259c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f60e405c5b0 0x7f60e41479b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:40.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.325+0000 7f60e259c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f60e405c5b0 0x7f60e41479b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45628/0 (socket says 172.21.15.60:45628) 2022-06-29T15:58:40.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.325+0000 7f60e2d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60e413cc30 0x7f60e413c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:40.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.325+0000 7f60e259c700 1 -- 172.21.15.60:0/1900308808 learned_addr learned my addr 172.21.15.60:0/1900308808 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:40.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.325+0000 7f60d9d9b700 1 --2- 172.21.15.60:0/1900308808 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60e4060610 0x7f60e413bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:40.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.329+0000 7f60e259c700 1 -- 172.21.15.60:0/1900308808 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60e4060610 msgr2=0x7f60e413bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:40.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.329+0000 7f60e259c700 1 --2- 172.21.15.60:0/1900308808 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60e4060610 0x7f60e413bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:40.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.329+0000 7f60e259c700 1 -- 172.21.15.60:0/1900308808 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60e413cc30 msgr2=0x7f60e413c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:40.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.329+0000 7f60e259c700 1 --2- 172.21.15.60:0/1900308808 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60e413cc30 0x7f60e413c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:40.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.329+0000 7f60e259c700 1 -- 172.21.15.60:0/1900308808 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d400b180 con 0x7f60e405c5b0 2022-06-29T15:58:40.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.329+0000 7f60e259c700 1 --2- 172.21.15.60:0/1900308808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f60e405c5b0 0x7f60e41479b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f60e413c700 tx=0x7f60d4014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:40.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.329+0000 7f60dbfff700 1 -- 172.21.15.60:0/1900308808 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60d40139c0 con 0x7f60e405c5b0 2022-06-29T15:58:40.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.329+0000 7f60dbfff700 1 -- 172.21.15.60:0/1900308808 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60d4013b20 con 0x7f60e405c5b0 2022-06-29T15:58:40.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.329+0000 7f60e359e700 1 -- 172.21.15.60:0/1900308808 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60e4144100 con 0x7f60e405c5b0 2022-06-29T15:58:40.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.329+0000 7f60dbfff700 1 -- 172.21.15.60:0/1900308808 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60d4013e80 con 0x7f60e405c5b0 2022-06-29T15:58:40.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.329+0000 7f60e359e700 1 -- 172.21.15.60:0/1900308808 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60e41445a0 con 0x7f60e405c5b0 2022-06-29T15:58:40.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.329+0000 7f60e359e700 1 -- 172.21.15.60:0/1900308808 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60e40573d0 con 0x7f60e405c5b0 2022-06-29T15:58:40.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.329+0000 7f60dbfff700 1 -- 172.21.15.60:0/1900308808 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f60d4025030 con 0x7f60e405c5b0 2022-06-29T15:58:40.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.329+0000 7f60dbfff700 1 --2- 172.21.15.60:0/1900308808 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f60bc053aa0 0x7f60bc055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:40.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.329+0000 7f60dbfff700 1 -- 172.21.15.60:0/1900308808 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f60d406cc90 con 0x7f60e405c5b0 2022-06-29T15:58:40.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.329+0000 7f60d9d9b700 1 --2- 172.21.15.60:0/1900308808 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f60bc053aa0 0x7f60bc055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:40.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.333+0000 7f60d9d9b700 1 --2- 172.21.15.60:0/1900308808 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f60bc053aa0 0x7f60bc055f60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f60cc00b000 tx=0x7f60cc006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:40.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.337+0000 7f60dbfff700 1 -- 172.21.15.60:0/1900308808 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f60d4043b50 con 0x7f60e405c5b0 2022-06-29T15:58:40.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.461+0000 7f60e359e700 1 -- 172.21.15.60:0/1900308808 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f60e40b12a0 con 0x7f60e405c5b0 2022-06-29T15:58:40.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.461+0000 7f60dbfff700 1 -- 172.21.15.60:0/1900308808 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7f60d4023070 con 0x7f60e405c5b0 2022-06-29T15:58:40.469 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T15:58:40.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.465+0000 7f60e359e700 1 -- 172.21.15.60:0/1900308808 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f60bc053aa0 msgr2=0x7f60bc055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:40.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.465+0000 7f60e359e700 1 --2- 172.21.15.60:0/1900308808 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f60bc053aa0 0x7f60bc055f60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f60cc00b000 tx=0x7f60cc006df0).stop 2022-06-29T15:58:40.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.465+0000 7f60e359e700 1 -- 172.21.15.60:0/1900308808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f60e405c5b0 msgr2=0x7f60e41479b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:40.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.465+0000 7f60e359e700 1 --2- 172.21.15.60:0/1900308808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f60e405c5b0 0x7f60e41479b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f60e413c700 tx=0x7f60d4014000).stop 2022-06-29T15:58:40.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.465+0000 7f60e359e700 1 -- 172.21.15.60:0/1900308808 shutdown_connections 2022-06-29T15:58:40.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.465+0000 7f60e359e700 1 --2- 172.21.15.60:0/1900308808 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f60bc053aa0 0x7f60bc055f60 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:40.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.465+0000 7f60e359e700 1 --2- 172.21.15.60:0/1900308808 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60e413cc30 0x7f60e413c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:40.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.465+0000 7f60e359e700 1 --2- 172.21.15.60:0/1900308808 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f60e4060610 0x7f60e413bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:40.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.465+0000 7f60e359e700 1 --2- 172.21.15.60:0/1900308808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f60e405c5b0 0x7f60e41479b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:40.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.465+0000 7f60e359e700 1 -- 172.21.15.60:0/1900308808 >> 172.21.15.60:0/1900308808 conn(0x7f60e4061c80 msgr2=0x7f60e40b2140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:40.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.465+0000 7f60e359e700 1 -- 172.21.15.60:0/1900308808 shutdown_connections 2022-06-29T15:58:40.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.465+0000 7f60e359e700 1 -- 172.21.15.60:0/1900308808 wait complete. 2022-06-29T15:58:40.482 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-06-29T15:58:40.483 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T15:58:40.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.577+0000 7f996662f700 1 Processor -- start 2022-06-29T15:58:40.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.577+0000 7f996662f700 1 -- start start 2022-06-29T15:58:40.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.581+0000 7f996662f700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f996005d070 0x7f996014aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:40.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.581+0000 7f996662f700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f996005d450 con 0x7f996014af80 2022-06-29T15:58:40.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.581+0000 7f996662f700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99600ad840 con 0x7f99600b2130 2022-06-29T15:58:40.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.581+0000 7f996662f700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99600adf90 con 0x7f996005d070 2022-06-29T15:58:40.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.581+0000 7f9965e2e700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f996014af80 0x7f99600ad130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53726/0 (socket says 172.21.15.60:53726) 2022-06-29T15:58:40.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.581+0000 7f9965e2e700 1 -- 172.21.15.60:0/3095595480 learned_addr learned my addr 172.21.15.60:0/3095595480 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:40.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.581+0000 7f996562d700 1 --2- 172.21.15.60:0/3095595480 >> v2:172.21.15.92:3300/0 conn(0x7f996005d070 0x7f996014aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:40.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.581+0000 7f9957fff700 1 -- 172.21.15.60:0/3095595480 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1371551352 0 0) 0x7f996005d450 con 0x7f996014af80 2022-06-29T15:58:40.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.581+0000 7f9957fff700 1 -- 172.21.15.60:0/3095595480 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9944003260 con 0x7f996014af80 2022-06-29T15:58:40.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.581+0000 7f9957fff700 1 -- 172.21.15.60:0/3095595480 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3810087002 0 0) 0x7f99600ad840 con 0x7f99600b2130 2022-06-29T15:58:40.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.581+0000 7f9957fff700 1 -- 172.21.15.60:0/3095595480 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f996005d450 con 0x7f99600b2130 2022-06-29T15:58:40.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.581+0000 7f996562d700 1 -- 172.21.15.60:0/3095595480 >> v1:172.21.15.92:6789/0 conn(0x7f99600b2130 legacy=0x7f99600aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:40.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.581+0000 7f996562d700 1 -- 172.21.15.60:0/3095595480 >> v1:172.21.15.60:6789/0 conn(0x7f996014af80 legacy=0x7f99600ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:40.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.581+0000 7f9957fff700 1 -- 172.21.15.60:0/3095595480 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1898194554 0 0) 0x7f9944003260 con 0x7f996014af80 2022-06-29T15:58:40.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.581+0000 7f996562d700 1 -- 172.21.15.60:0/3095595480 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99600b25d0 con 0x7f996005d070 2022-06-29T15:58:40.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.581+0000 7f996562d700 1 --2- 172.21.15.60:0/3095595480 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f996005d070 0x7f996014aa90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f9958001490 tx=0x7f995c006df0).ready entity=mon.1 client_cookie=c9af2bc5700c16f5 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:40.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.581+0000 7f9957fff700 1 -- 172.21.15.60:0/3095595480 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99580019e0 con 0x7f996005d070 2022-06-29T15:58:40.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.581+0000 7f9957fff700 1 -- 172.21.15.60:0/3095595480 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f995c000e90 con 0x7f996005d070 2022-06-29T15:58:40.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.581+0000 7f9957fff700 1 -- 172.21.15.60:0/3095595480 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f995c012460 con 0x7f996005d070 2022-06-29T15:58:40.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.581+0000 7f996662f700 1 -- 172.21.15.60:0/3095595480 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f996005d070 msgr2=0x7f996014aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:40.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.581+0000 7f996662f700 1 --2- 172.21.15.60:0/3095595480 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f996005d070 0x7f996014aa90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f9958001490 tx=0x7f995c006df0).stop 2022-06-29T15:58:40.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.581+0000 7f996662f700 1 -- 172.21.15.60:0/3095595480 shutdown_connections 2022-06-29T15:58:40.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.581+0000 7f996662f700 1 --2- 172.21.15.60:0/3095595480 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f996005d070 0x7f996014aa90 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:40.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.581+0000 7f996662f700 1 -- 172.21.15.60:0/3095595480 >> 172.21.15.60:0/3095595480 conn(0x7f99600b5890 msgr2=0x7f99600b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:40.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.585+0000 7f996662f700 1 -- 172.21.15.60:0/3095595480 shutdown_connections 2022-06-29T15:58:40.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.585+0000 7f996662f700 1 -- 172.21.15.60:0/3095595480 wait complete. 2022-06-29T15:58:40.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.585+0000 7f996662f700 1 Processor -- start 2022-06-29T15:58:40.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.585+0000 7f996662f700 1 -- start start 2022-06-29T15:58:40.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.585+0000 7f996662f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f996005d070 0x7f99600b4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:40.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.585+0000 7f996662f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f99600b2130 0x7f9960156900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:40.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.585+0000 7f996662f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f996014af80 0x7f9960159970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:40.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.585+0000 7f996662f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9960142930 con 0x7f996014af80 2022-06-29T15:58:40.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.585+0000 7f996662f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9960116bb0 con 0x7f996005d070 2022-06-29T15:58:40.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.585+0000 7f996662f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9960116e50 con 0x7f99600b2130 2022-06-29T15:58:40.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.589+0000 7f9965e2e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f996014af80 0x7f9960159970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:40.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.589+0000 7f996562d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f996005d070 0x7f99600b4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:40.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.589+0000 7f9964e2c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f99600b2130 0x7f9960156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:40.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.589+0000 7f996562d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f996005d070 0x7f99600b4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:51672/0 (socket says 172.21.15.60:51672) 2022-06-29T15:58:40.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.589+0000 7f996562d700 1 -- 172.21.15.60:0/3306433103 learned_addr learned my addr 172.21.15.60:0/3306433103 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:40.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.589+0000 7f9965e2e700 1 -- 172.21.15.60:0/3306433103 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f99600b2130 msgr2=0x7f9960156900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:40.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.589+0000 7f9965e2e700 1 --2- 172.21.15.60:0/3306433103 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f99600b2130 0x7f9960156900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:40.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.589+0000 7f9965e2e700 1 -- 172.21.15.60:0/3306433103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f996005d070 msgr2=0x7f99600b4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:40.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.589+0000 7f9965e2e700 1 --2- 172.21.15.60:0/3306433103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f996005d070 0x7f99600b4140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:40.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.589+0000 7f9965e2e700 1 -- 172.21.15.60:0/3306433103 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f995c00b180 con 0x7f996014af80 2022-06-29T15:58:40.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.589+0000 7f996562d700 1 --2- 172.21.15.60:0/3306433103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f996005d070 0x7f99600b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:40.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.589+0000 7f9965e2e700 1 --2- 172.21.15.60:0/3306433103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f996014af80 0x7f9960159970 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f9958001bc0 tx=0x7f9958001960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:40.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.589+0000 7f99567fc700 1 -- 172.21.15.60:0/3306433103 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9958004ab0 con 0x7f996014af80 2022-06-29T15:58:40.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.589+0000 7f99567fc700 1 -- 172.21.15.60:0/3306433103 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9958004c10 con 0x7f996014af80 2022-06-29T15:58:40.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.589+0000 7f996662f700 1 -- 172.21.15.60:0/3306433103 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99601170f0 con 0x7f996014af80 2022-06-29T15:58:40.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.589+0000 7f99567fc700 1 -- 172.21.15.60:0/3306433103 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9958007170 con 0x7f996014af80 2022-06-29T15:58:40.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.589+0000 7f996662f700 1 -- 172.21.15.60:0/3306433103 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9960116780 con 0x7f996014af80 2022-06-29T15:58:40.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.589+0000 7f996662f700 1 -- 172.21.15.60:0/3306433103 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f996014a350 con 0x7f996014af80 2022-06-29T15:58:40.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.593+0000 7f99567fc700 1 -- 172.21.15.60:0/3306433103 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f995801b070 con 0x7f996014af80 2022-06-29T15:58:40.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.593+0000 7f99567fc700 1 --2- 172.21.15.60:0/3306433103 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9948053b00 0x7f9948055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:40.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.593+0000 7f99567fc700 1 -- 172.21.15.60:0/3306433103 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f995806b790 con 0x7f996014af80 2022-06-29T15:58:40.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.593+0000 7f996562d700 1 --2- 172.21.15.60:0/3306433103 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9948053b00 0x7f9948055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:40.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.593+0000 7f996562d700 1 --2- 172.21.15.60:0/3306433103 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9948053b00 0x7f9948055fc0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f995c01a000 tx=0x7f995c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:40.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.597+0000 7f99567fc700 1 -- 172.21.15.60:0/3306433103 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9958042540 con 0x7f996014af80 2022-06-29T15:58:40.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.761+0000 7f996662f700 1 -- 172.21.15.60:0/3306433103 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99600573d0 con 0x7f996014af80 2022-06-29T15:58:40.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.761+0000 7f99567fc700 1 -- 172.21.15.60:0/3306433103 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2134 (secure 0 0 0) 0x7f9958045b60 con 0x7f996014af80 2022-06-29T15:58:40.770 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:58:40.770 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":4,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.92:6835/2121698334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2121698334},{"type":"v1","addr":"172.21.15.92:6835","nonce":2121698334}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":4,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T15:58:24.334770+0000","modified":"2022-06-29T15:58:25.338504+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"a","rank":0,"incarnation":3,"state":"up:active","state_seq":2,"addr":"172.21.15.60:6833/3164364318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3164364318},{"type":"v1","addr":"172.21.15.60:6833","nonce":3164364318}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-29T15:58:40.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.765+0000 7f996662f700 1 -- 172.21.15.60:0/3306433103 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9948053b00 msgr2=0x7f9948055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:40.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.765+0000 7f996662f700 1 --2- 172.21.15.60:0/3306433103 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9948053b00 0x7f9948055fc0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f995c01a000 tx=0x7f995c014000).stop 2022-06-29T15:58:40.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.765+0000 7f996662f700 1 -- 172.21.15.60:0/3306433103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f996014af80 msgr2=0x7f9960159970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:40.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.765+0000 7f996662f700 1 --2- 172.21.15.60:0/3306433103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f996014af80 0x7f9960159970 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f9958001bc0 tx=0x7f9958001960).stop 2022-06-29T15:58:40.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.765+0000 7f996662f700 1 -- 172.21.15.60:0/3306433103 shutdown_connections 2022-06-29T15:58:40.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.765+0000 7f996662f700 1 --2- 172.21.15.60:0/3306433103 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9948053b00 0x7f9948055fc0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:40.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.765+0000 7f996662f700 1 --2- 172.21.15.60:0/3306433103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f996014af80 0x7f9960159970 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:40.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.765+0000 7f996662f700 1 --2- 172.21.15.60:0/3306433103 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f99600b2130 0x7f9960156900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:40.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.765+0000 7f996662f700 1 --2- 172.21.15.60:0/3306433103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f996005d070 0x7f99600b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:40.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.765+0000 7f996662f700 1 -- 172.21.15.60:0/3306433103 >> 172.21.15.60:0/3306433103 conn(0x7f99600b5890 msgr2=0x7f9960143340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:40.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.765+0000 7f996662f700 1 -- 172.21.15.60:0/3306433103 shutdown_connections 2022-06-29T15:58:40.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:40.765+0000 7f996662f700 1 -- 172.21.15.60:0/3306433103 wait complete. 2022-06-29T15:58:40.776 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 4 2022-06-29T15:58:40.989 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 4, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T15:58:24.334770+0000', 'modified': '2022-06-29T15:58:25.338504+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4257}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4257': {'gid': 4257, 'name': 'a', 'rank': 0, 'incarnation': 3, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.60:6833/3164364318', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6832', 'nonce': 3164364318}, {'type': 'v1', 'addr': '172.21.15.60:6833', 'nonce': 3164364318}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T15:58:40.989 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T15:58:40.990 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-06-29T15:58:41.104 INFO:teuthology.orchestra.run.smithi060.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":4257,"want_state":"up:active","state":"up:active","rank_uptime":16.754068178000001,"mdsmap_epoch":4,"osdmap_epoch":23,"osdmap_epoch_barrier":21,"uptime":20.004164512999999} 2022-06-29T15:58:41.105 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 4257, "mdsmap_epoch": 4, "osdmap_epoch": 23, "osdmap_epoch_barrier": 21, "rank_uptime": 16.754068178, "state": "up:active", "uptime": 20.004164513, "want_state": "up:active", "whoami": 0 } 2022-06-29T15:58:41.105 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-06-29T15:58:41.106 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-06-29T15:58:41.106 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi156.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-06-29T15:58:41.106 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:41.107 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-06-29T15:58:41.115 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2022-06-29T15:58:41.134 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T15:58:41.135 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T15:58:41.152 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T15:58:41.154 INFO:teuthology.orchestra.run.smithi156.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T15:58:41.154 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:41.155 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2022-06-29T15:58:41.202 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-06-29T15:58:41.202 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-06-29T15:58:41.205 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2022-06-29T15:58:41.220 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T15:58:41.220 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T15:58:41.279 INFO:teuthology.orchestra.run.smithi156.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T15:58:41.280 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T15:58:41.281 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:41.281 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2022-06-29T15:58:41.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:2022-06-29T15:58:41.285+0000 7f09824f2080 -1 init, newargv = 0x5615ee1c3760 newargc=9 2022-06-29T15:58:41.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[12202]: starting ceph client 2022-06-29T15:58:41.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[12202]: starting fuse 2022-06-29T15:58:42.389 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2022-06-29T15:58:42.408 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T15:58:42.408 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T15:58:42.466 INFO:teuthology.orchestra.run.smithi156.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T15:58:42.467 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T15:58:42.468 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:42.468 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2022-06-29T15:58:42.474 INFO:teuthology.orchestra.run.smithi156.stdout:52 2022-06-29T15:58:42.474 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-06-29T15:58:42.475 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:42.475 DEBUG:teuthology.orchestra.run.smithi156:> sudo python3 -c ' 2022-06-29T15:58:42.476 DEBUG:teuthology.orchestra.run.smithi156:> import glob 2022-06-29T15:58:42.476 DEBUG:teuthology.orchestra.run.smithi156:> import re 2022-06-29T15:58:42.477 DEBUG:teuthology.orchestra.run.smithi156:> import os 2022-06-29T15:58:42.477 DEBUG:teuthology.orchestra.run.smithi156:> import subprocess 2022-06-29T15:58:42.477 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T15:58:42.478 DEBUG:teuthology.orchestra.run.smithi156:> def find_socket(client_name): 2022-06-29T15:58:42.478 DEBUG:teuthology.orchestra.run.smithi156:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-06-29T15:58:42.479 DEBUG:teuthology.orchestra.run.smithi156:> files = glob.glob(asok_path) 2022-06-29T15:58:42.479 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T15:58:42.480 DEBUG:teuthology.orchestra.run.smithi156:> # Given a non-glob path, it better be there 2022-06-29T15:58:42.480 DEBUG:teuthology.orchestra.run.smithi156:> if "*" not in asok_path: 2022-06-29T15:58:42.481 DEBUG:teuthology.orchestra.run.smithi156:> assert(len(files) == 1) 2022-06-29T15:58:42.482 DEBUG:teuthology.orchestra.run.smithi156:> return files[0] 2022-06-29T15:58:42.482 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T15:58:42.483 DEBUG:teuthology.orchestra.run.smithi156:> for f in files: 2022-06-29T15:58:42.484 DEBUG:teuthology.orchestra.run.smithi156:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-06-29T15:58:42.484 DEBUG:teuthology.orchestra.run.smithi156:> if os.path.exists("/proc/{0}".format(pid)): 2022-06-29T15:58:42.485 DEBUG:teuthology.orchestra.run.smithi156:> return f 2022-06-29T15:58:42.486 DEBUG:teuthology.orchestra.run.smithi156:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-06-29T15:58:42.489 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T15:58:42.490 DEBUG:teuthology.orchestra.run.smithi156:> print(find_socket("client.1")) 2022-06-29T15:58:42.490 DEBUG:teuthology.orchestra.run.smithi156:> ' 2022-06-29T15:58:42.534 INFO:teuthology.orchestra.run.smithi156.stdout:/var/run/ceph/ceph-client.1.12202.asok 2022-06-29T15:58:42.545 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.12202.asok 2022-06-29T15:58:42.546 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:42.546 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.12202.asok status 2022-06-29T15:58:42.684 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2022-06-29T15:58:42.684 INFO:teuthology.orchestra.run.smithi156.stdout: "metadata": { 2022-06-29T15:58:42.685 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_sha1": "f25ffac72e5a9e63860ff7deb6767a11f0559a16", 2022-06-29T15:58:42.685 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_version": "ceph version 15.2.16-304-gf25ffac7 (f25ffac72e5a9e63860ff7deb6767a11f0559a16) octopus (stable)", 2022-06-29T15:58:42.685 INFO:teuthology.orchestra.run.smithi156.stdout: "entity_id": "1", 2022-06-29T15:58:42.685 INFO:teuthology.orchestra.run.smithi156.stdout: "hostname": "smithi156", 2022-06-29T15:58:42.686 INFO:teuthology.orchestra.run.smithi156.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-06-29T15:58:42.686 INFO:teuthology.orchestra.run.smithi156.stdout: "pid": "12202", 2022-06-29T15:58:42.686 INFO:teuthology.orchestra.run.smithi156.stdout: "root": "/" 2022-06-29T15:58:42.687 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T15:58:42.687 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_count": 0, 2022-06-29T15:58:42.687 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_pinned_count": 0, 2022-06-29T15:58:42.687 INFO:teuthology.orchestra.run.smithi156.stdout: "id": 4684, 2022-06-29T15:58:42.688 INFO:teuthology.orchestra.run.smithi156.stdout: "inst": { 2022-06-29T15:58:42.688 INFO:teuthology.orchestra.run.smithi156.stdout: "name": { 2022-06-29T15:58:42.688 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "client", 2022-06-29T15:58:42.688 INFO:teuthology.orchestra.run.smithi156.stdout: "num": 4684 2022-06-29T15:58:42.689 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T15:58:42.689 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2022-06-29T15:58:42.689 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2022-06-29T15:58:42.689 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "172.21.15.156:0", 2022-06-29T15:58:42.690 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 2623687735 2022-06-29T15:58:42.690 INFO:teuthology.orchestra.run.smithi156.stdout: } 2022-06-29T15:58:42.690 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T15:58:42.691 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2022-06-29T15:58:42.691 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2022-06-29T15:58:42.691 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "172.21.15.156:0", 2022-06-29T15:58:42.691 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 2623687735 2022-06-29T15:58:42.691 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T15:58:42.692 INFO:teuthology.orchestra.run.smithi156.stdout: "inst_str": "client.4684 172.21.15.156:0/2623687735", 2022-06-29T15:58:42.692 INFO:teuthology.orchestra.run.smithi156.stdout: "addr_str": "172.21.15.156:0/2623687735", 2022-06-29T15:58:42.692 INFO:teuthology.orchestra.run.smithi156.stdout: "inode_count": 1, 2022-06-29T15:58:42.692 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_epoch": 4, 2022-06-29T15:58:42.693 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch": 23, 2022-06-29T15:58:42.693 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch_barrier": 0, 2022-06-29T15:58:42.693 INFO:teuthology.orchestra.run.smithi156.stdout: "blacklisted": false, 2022-06-29T15:58:42.693 INFO:teuthology.orchestra.run.smithi156.stdout: "fs_name": "cephfs" 2022-06-29T15:58:42.694 INFO:teuthology.orchestra.run.smithi156.stdout:} 2022-06-29T15:58:42.696 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T15:58:42.807 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.801+0000 7fde660bd700 1 Processor -- start 2022-06-29T15:58:42.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.801+0000 7fde660bd700 1 -- start start 2022-06-29T15:58:42.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.801+0000 7fde660bd700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fde600b5eb0 0x7fde60142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:42.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.801+0000 7fde660bd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fde600b48b0 0x7fde600b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:42.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.801+0000 7fde660bd700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde600b4d10 con 0x7fde600b2c50 2022-06-29T15:58:42.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.801+0000 7fde660bd700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde600b54e0 con 0x7fde600b48b0 2022-06-29T15:58:42.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.801+0000 7fde660bd700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde600b6290 con 0x7fde600b5eb0 2022-06-29T15:58:42.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.801+0000 7fde648ba700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fde600b48b0 0x7fde600b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:42.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.801+0000 7fde648ba700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fde600b48b0 0x7fde600b05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45650/0 (socket says 172.21.15.60:45650) 2022-06-29T15:58:42.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.801+0000 7fde648ba700 1 -- 172.21.15.60:0/561612137 learned_addr learned my addr 172.21.15.60:0/561612137 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:42.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.801+0000 7fde650bb700 1 --2- 172.21.15.60:0/561612137 >> v2:172.21.15.92:3300/0 conn(0x7fde600b5eb0 0x7fde60142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:42.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.805+0000 7fde648ba700 1 -- 172.21.15.60:0/561612137 >> v2:172.21.15.92:3300/0 conn(0x7fde600b5eb0 msgr2=0x7fde60142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:42.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.805+0000 7fde648ba700 1 --2- 172.21.15.60:0/561612137 >> v2:172.21.15.92:3300/0 conn(0x7fde600b5eb0 0x7fde60142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:42.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.805+0000 7fde648ba700 1 -- 172.21.15.60:0/561612137 >> v1:172.21.15.92:6789/0 conn(0x7fde600b2c50 legacy=0x7fde60145fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:42.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.805+0000 7fde648ba700 1 -- 172.21.15.60:0/561612137 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde60142f10 con 0x7fde600b48b0 2022-06-29T15:58:42.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.805+0000 7fde648ba700 1 --2- 172.21.15.60:0/561612137 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde600b48b0 0x7fde600b05e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fde60062f00 tx=0x7fde54009290).ready entity=mon.0 client_cookie=51db6500ee1c0fda server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:42.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.805+0000 7fde5ffff700 1 -- 172.21.15.60:0/561612137 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde5400b830 con 0x7fde600b48b0 2022-06-29T15:58:42.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.805+0000 7fde5ffff700 1 -- 172.21.15.60:0/561612137 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde5400b990 con 0x7fde600b48b0 2022-06-29T15:58:42.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.805+0000 7fde5ffff700 1 -- 172.21.15.60:0/561612137 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde5400bb40 con 0x7fde600b48b0 2022-06-29T15:58:42.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.805+0000 7fde660bd700 1 -- 172.21.15.60:0/561612137 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde600b48b0 msgr2=0x7fde600b05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:42.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.805+0000 7fde660bd700 1 --2- 172.21.15.60:0/561612137 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde600b48b0 0x7fde600b05e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fde60062f00 tx=0x7fde54009290).stop 2022-06-29T15:58:42.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.805+0000 7fde660bd700 1 -- 172.21.15.60:0/561612137 shutdown_connections 2022-06-29T15:58:42.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.805+0000 7fde660bd700 1 --2- 172.21.15.60:0/561612137 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde600b48b0 0x7fde600b05e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:42.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.805+0000 7fde660bd700 1 --2- 172.21.15.60:0/561612137 >> v2:172.21.15.92:3300/0 conn(0x7fde600b5eb0 0x7fde60142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:42.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.805+0000 7fde660bd700 1 -- 172.21.15.60:0/561612137 >> 172.21.15.60:0/561612137 conn(0x7fde60062090 msgr2=0x7fde60062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:42.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.805+0000 7fde660bd700 1 -- 172.21.15.60:0/561612137 shutdown_connections 2022-06-29T15:58:42.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.805+0000 7fde660bd700 1 -- 172.21.15.60:0/561612137 wait complete. 2022-06-29T15:58:42.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.805+0000 7fde660bd700 1 Processor -- start 2022-06-29T15:58:42.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.809+0000 7fde660bd700 1 -- start start 2022-06-29T15:58:42.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.809+0000 7fde660bd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fde600b2c50 0x7fde600b0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:42.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.809+0000 7fde660bd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde600b48b0 0x7fde60155c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:42.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.809+0000 7fde660bd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fde600b5eb0 0x7fde60114c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:42.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.809+0000 7fde660bd700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde60113300 con 0x7fde600b48b0 2022-06-29T15:58:42.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.809+0000 7fde660bd700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde601462f0 con 0x7fde600b5eb0 2022-06-29T15:58:42.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.809+0000 7fde660bd700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde60145c30 con 0x7fde600b2c50 2022-06-29T15:58:42.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.809+0000 7fde648ba700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde600b48b0 0x7fde60155c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:42.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.809+0000 7fde648ba700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde600b48b0 0x7fde60155c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45658/0 (socket says 172.21.15.60:45658) 2022-06-29T15:58:42.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.809+0000 7fde648ba700 1 -- 172.21.15.60:0/1988737887 learned_addr learned my addr 172.21.15.60:0/1988737887 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:42.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.809+0000 7fde650bb700 1 --2- 172.21.15.60:0/1988737887 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fde600b2c50 0x7fde600b0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:42.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.809+0000 7fde658bc700 1 --2- 172.21.15.60:0/1988737887 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fde600b5eb0 0x7fde60114c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:42.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.809+0000 7fde648ba700 1 -- 172.21.15.60:0/1988737887 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fde600b2c50 msgr2=0x7fde600b0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:42.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.809+0000 7fde648ba700 1 --2- 172.21.15.60:0/1988737887 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fde600b2c50 0x7fde600b0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:42.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.809+0000 7fde648ba700 1 -- 172.21.15.60:0/1988737887 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fde600b5eb0 msgr2=0x7fde60114c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:42.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.809+0000 7fde648ba700 1 --2- 172.21.15.60:0/1988737887 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fde600b5eb0 0x7fde60114c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:42.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.809+0000 7fde648ba700 1 -- 172.21.15.60:0/1988737887 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde54009530 con 0x7fde600b48b0 2022-06-29T15:58:42.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.809+0000 7fde648ba700 1 --2- 172.21.15.60:0/1988737887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde600b48b0 0x7fde60155c00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fde60062f00 tx=0x7fde54016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:42.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.809+0000 7fde5e7fc700 1 -- 172.21.15.60:0/1988737887 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde540032f0 con 0x7fde600b48b0 2022-06-29T15:58:42.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.809+0000 7fde5e7fc700 1 -- 172.21.15.60:0/1988737887 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde54021440 con 0x7fde600b48b0 2022-06-29T15:58:42.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.809+0000 7fde660bd700 1 -- 172.21.15.60:0/1988737887 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde60113530 con 0x7fde600b48b0 2022-06-29T15:58:42.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.809+0000 7fde5e7fc700 1 -- 172.21.15.60:0/1988737887 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde540217a0 con 0x7fde600b48b0 2022-06-29T15:58:42.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.809+0000 7fde660bd700 1 -- 172.21.15.60:0/1988737887 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde60145ed0 con 0x7fde600b48b0 2022-06-29T15:58:42.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.813+0000 7fde660bd700 1 -- 172.21.15.60:0/1988737887 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde600b4f70 con 0x7fde600b48b0 2022-06-29T15:58:42.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.813+0000 7fde5e7fc700 1 -- 172.21.15.60:0/1988737887 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fde54027020 con 0x7fde600b48b0 2022-06-29T15:58:42.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.813+0000 7fde5e7fc700 1 --2- 172.21.15.60:0/1988737887 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fde400539e0 0x7fde40055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:42.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.813+0000 7fde5e7fc700 1 -- 172.21.15.60:0/1988737887 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fde5406d2a0 con 0x7fde600b48b0 2022-06-29T15:58:42.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.813+0000 7fde650bb700 1 --2- 172.21.15.60:0/1988737887 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fde400539e0 0x7fde40055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:42.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.813+0000 7fde650bb700 1 --2- 172.21.15.60:0/1988737887 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fde400539e0 0x7fde40055ea0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fde5000c000 tx=0x7fde50008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:42.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.817+0000 7fde5e7fc700 1 -- 172.21.15.60:0/1988737887 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fde54044160 con 0x7fde600b48b0 2022-06-29T15:58:42.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.941+0000 7fde660bd700 1 -- 172.21.15.60:0/1988737887 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde600573d0 con 0x7fde600b48b0 2022-06-29T15:58:42.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.945+0000 7fde5e7fc700 1 -- 172.21.15.60:0/1988737887 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16038 (secure 0 0 0) 0x7fde54015020 con 0x7fde600b48b0 2022-06-29T15:58:42.949 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:58:42.950 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":23,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T15:58:26.345644+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-06-29T15:58:22.424395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-06-29T15:58:23.343438+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T15:58:42.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.945+0000 7fde660bd700 1 -- 172.21.15.60:0/1988737887 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fde400539e0 msgr2=0x7fde40055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:42.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.945+0000 7fde660bd700 1 --2- 172.21.15.60:0/1988737887 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fde400539e0 0x7fde40055ea0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fde5000c000 tx=0x7fde50008040).stop 2022-06-29T15:58:42.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.945+0000 7fde660bd700 1 -- 172.21.15.60:0/1988737887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde600b48b0 msgr2=0x7fde60155c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:42.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.945+0000 7fde660bd700 1 --2- 172.21.15.60:0/1988737887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde600b48b0 0x7fde60155c00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fde60062f00 tx=0x7fde54016000).stop 2022-06-29T15:58:42.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.945+0000 7fde660bd700 1 -- 172.21.15.60:0/1988737887 shutdown_connections 2022-06-29T15:58:42.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.945+0000 7fde660bd700 1 --2- 172.21.15.60:0/1988737887 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fde400539e0 0x7fde40055ea0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:42.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.945+0000 7fde660bd700 1 --2- 172.21.15.60:0/1988737887 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fde600b5eb0 0x7fde60114c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:42.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.945+0000 7fde660bd700 1 --2- 172.21.15.60:0/1988737887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fde600b48b0 0x7fde60155c00 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:42.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.945+0000 7fde660bd700 1 --2- 172.21.15.60:0/1988737887 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fde600b2c50 0x7fde600b0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:42.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.945+0000 7fde660bd700 1 -- 172.21.15.60:0/1988737887 >> 172.21.15.60:0/1988737887 conn(0x7fde60062090 msgr2=0x7fde6013fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:42.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.945+0000 7fde660bd700 1 -- 172.21.15.60:0/1988737887 shutdown_connections 2022-06-29T15:58:42.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:42.945+0000 7fde660bd700 1 -- 172.21.15.60:0/1988737887 wait complete. 2022-06-29T15:58:42.964 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-29T15:58:22.424395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-06-29T15:58:23.343438+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-29T15:58:42.965 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T15:58:43.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.065+0000 7fd416803700 1 Processor -- start 2022-06-29T15:58:43.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.065+0000 7fd416803700 1 -- start start 2022-06-29T15:58:43.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd416803700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4100b2b40 0x7fd4100af420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:43.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd416803700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4100b2f20 con 0x7fd410140dd0 2022-06-29T15:58:43.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd416803700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4100b1410 con 0x7fd410143e10 2022-06-29T15:58:43.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd416803700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4100aea40 con 0x7fd4100b2b40 2022-06-29T15:58:43.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd416002700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd410140dd0 0x7fd4100b0d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53754/0 (socket says 172.21.15.60:53754) 2022-06-29T15:58:43.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd416002700 1 -- 172.21.15.60:0/4111734872 learned_addr learned my addr 172.21.15.60:0/4111734872 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:43.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd415801700 1 --2- 172.21.15.60:0/4111734872 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4100b2b40 0x7fd4100af420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:43.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd407fff700 1 -- 172.21.15.60:0/4111734872 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1999683439 0 0) 0x7fd4100b2f20 con 0x7fd410140dd0 2022-06-29T15:58:43.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd407fff700 1 -- 172.21.15.60:0/4111734872 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3fc003260 con 0x7fd410140dd0 2022-06-29T15:58:43.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd407fff700 1 -- 172.21.15.60:0/4111734872 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3868520728 0 0) 0x7fd3fc003260 con 0x7fd410140dd0 2022-06-29T15:58:43.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd407fff700 1 -- 172.21.15.60:0/4111734872 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4100b2b40 msgr2=0x7fd4100af420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:43.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd407fff700 1 --2- 172.21.15.60:0/4111734872 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4100b2b40 0x7fd4100af420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd407fff700 1 -- 172.21.15.60:0/4111734872 >> v1:172.21.15.92:6789/0 conn(0x7fd410143e10 legacy=0x7fd4100b05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:43.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd407fff700 1 -- 172.21.15.60:0/4111734872 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4100aece0 con 0x7fd410140dd0 2022-06-29T15:58:43.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd407fff700 1 -- 172.21.15.60:0/4111734872 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7fd408001660 con 0x7fd410140dd0 2022-06-29T15:58:43.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd407fff700 1 -- 172.21.15.60:0/4111734872 >> v1:172.21.15.60:6789/0 conn(0x7fd410140dd0 legacy=0x7fd4100b0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:43.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd407fff700 1 --2- 172.21.15.60:0/4111734872 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3fc003780 0x7fd3fc003b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:43.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd407fff700 1 --2- 172.21.15.60:0/4111734872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3fc004500 0x7fd3fc004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:43.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd407fff700 1 --2- 172.21.15.60:0/4111734872 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4100b2b40 0x7fd3fc004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:43.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd407fff700 1 -- 172.21.15.60:0/4111734872 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4100aea40 con 0x7fd3fc004500 2022-06-29T15:58:43.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd407fff700 1 -- 172.21.15.60:0/4111734872 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc00b750 con 0x7fd3fc003780 2022-06-29T15:58:43.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd407fff700 1 -- 172.21.15.60:0/4111734872 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc00b8c0 con 0x7fd4100b2b40 2022-06-29T15:58:43.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd415000700 1 --2- 172.21.15.60:0/4111734872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3fc004500 0x7fd3fc004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:43.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd415801700 1 --2- 172.21.15.60:0/4111734872 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3fc003780 0x7fd3fc003b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:43.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd416002700 1 --2- 172.21.15.60:0/4111734872 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4100b2b40 0x7fd3fc004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:43.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd415000700 1 -- 172.21.15.60:0/4111734872 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4100b2b40 msgr2=0x7fd3fc004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:43.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd415000700 1 --2- 172.21.15.60:0/4111734872 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4100b2b40 0x7fd3fc004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd415000700 1 -- 172.21.15.60:0/4111734872 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3fc003780 msgr2=0x7fd3fc003b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:43.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd415000700 1 --2- 172.21.15.60:0/4111734872 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3fc003780 0x7fd3fc003b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd415000700 1 -- 172.21.15.60:0/4111734872 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3fc003260 con 0x7fd3fc004500 2022-06-29T15:58:43.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd415000700 1 --2- 172.21.15.60:0/4111734872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3fc004500 0x7fd3fc004980 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fd40c001490 tx=0x7fd40c007d80).ready entity=mon.0 client_cookie=2c5809385e2b449e server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:43.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd407fff700 1 -- 172.21.15.60:0/4111734872 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd40c0017b0 con 0x7fd3fc004500 2022-06-29T15:58:43.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd407fff700 1 -- 172.21.15.60:0/4111734872 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd40c006890 con 0x7fd3fc004500 2022-06-29T15:58:43.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.069+0000 7fd407fff700 1 -- 172.21.15.60:0/4111734872 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd40c006bf0 con 0x7fd3fc004500 2022-06-29T15:58:43.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.073+0000 7fd416803700 1 -- 172.21.15.60:0/4111734872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3fc004500 msgr2=0x7fd3fc004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:43.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.073+0000 7fd416803700 1 --2- 172.21.15.60:0/4111734872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3fc004500 0x7fd3fc004980 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fd40c001490 tx=0x7fd40c007d80).stop 2022-06-29T15:58:43.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.073+0000 7fd416002700 1 -- 172.21.15.60:0/4111734872 reap_dead start 2022-06-29T15:58:43.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.073+0000 7fd416803700 1 -- 172.21.15.60:0/4111734872 shutdown_connections 2022-06-29T15:58:43.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.073+0000 7fd416803700 1 -- 172.21.15.60:0/4111734872 >> 172.21.15.60:0/4111734872 conn(0x7fd4100b6120 msgr2=0x7fd4100b20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:43.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.073+0000 7fd416803700 1 -- 172.21.15.60:0/4111734872 shutdown_connections 2022-06-29T15:58:43.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.073+0000 7fd416803700 1 -- 172.21.15.60:0/4111734872 wait complete. 2022-06-29T15:58:43.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.073+0000 7fd416803700 1 Processor -- start 2022-06-29T15:58:43.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.073+0000 7fd416803700 1 -- start start 2022-06-29T15:58:43.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.073+0000 7fd416803700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3fc004500 0x7fd41013b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:43.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.073+0000 7fd416803700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd410145b00 0x7fd41014f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:43.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.073+0000 7fd416803700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4101449c0 0x7fd410144e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:43.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.073+0000 7fd416803700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd410147250 con 0x7fd4101449c0 2022-06-29T15:58:43.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.073+0000 7fd416803700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4101442f0 con 0x7fd3fc004500 2022-06-29T15:58:43.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.073+0000 7fd416803700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4101463a0 con 0x7fd410145b00 2022-06-29T15:58:43.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.073+0000 7fd416002700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4101449c0 0x7fd410144e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:43.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.073+0000 7fd415801700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3fc004500 0x7fd41013b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:43.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.073+0000 7fd415000700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd410145b00 0x7fd41014f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:43.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.073+0000 7fd416002700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4101449c0 0x7fd410144e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45680/0 (socket says 172.21.15.60:45680) 2022-06-29T15:58:43.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.073+0000 7fd416002700 1 -- 172.21.15.60:0/867656144 learned_addr learned my addr 172.21.15.60:0/867656144 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:43.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.073+0000 7fd415000700 1 -- 172.21.15.60:0/867656144 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3fc004500 msgr2=0x7fd41013b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:43.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.073+0000 7fd415000700 1 --2- 172.21.15.60:0/867656144 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3fc004500 0x7fd41013b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.073+0000 7fd415000700 1 -- 172.21.15.60:0/867656144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4101449c0 msgr2=0x7fd410144e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:43.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.073+0000 7fd415000700 1 --2- 172.21.15.60:0/867656144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4101449c0 0x7fd410144e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.073+0000 7fd415000700 1 -- 172.21.15.60:0/867656144 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd410144520 con 0x7fd410145b00 2022-06-29T15:58:43.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.073+0000 7fd416002700 1 --2- 172.21.15.60:0/867656144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4101449c0 0x7fd410144e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:43.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.073+0000 7fd415000700 1 --2- 172.21.15.60:0/867656144 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd410145b00 0x7fd41014f920 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fd40c001490 tx=0x7fd40c016000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:43.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.073+0000 7fd4067fc700 1 -- 172.21.15.60:0/867656144 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd40c021070 con 0x7fd410145b00 2022-06-29T15:58:43.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.073+0000 7fd4067fc700 1 -- 172.21.15.60:0/867656144 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd40c01b9e0 con 0x7fd410145b00 2022-06-29T15:58:43.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.073+0000 7fd416803700 1 -- 172.21.15.60:0/867656144 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd410148060 con 0x7fd410145b00 2022-06-29T15:58:43.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.073+0000 7fd4067fc700 1 -- 172.21.15.60:0/867656144 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd40c0017b0 con 0x7fd410145b00 2022-06-29T15:58:43.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.073+0000 7fd416803700 1 -- 172.21.15.60:0/867656144 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd410145f60 con 0x7fd410145b00 2022-06-29T15:58:43.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.077+0000 7fd4067fc700 1 -- 172.21.15.60:0/867656144 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd40c02e070 con 0x7fd410145b00 2022-06-29T15:58:43.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.077+0000 7fd416803700 1 -- 172.21.15.60:0/867656144 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4101464e0 con 0x7fd410145b00 2022-06-29T15:58:43.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.077+0000 7fd4067fc700 1 --2- 172.21.15.60:0/867656144 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd3f0053b00 0x7fd3f0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:43.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.077+0000 7fd4067fc700 1 -- 172.21.15.60:0/867656144 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd40c071080 con 0x7fd410145b00 2022-06-29T15:58:43.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.077+0000 7fd415801700 1 --2- 172.21.15.60:0/867656144 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd3f0053b00 0x7fd3f0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:43.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.077+0000 7fd415801700 1 --2- 172.21.15.60:0/867656144 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd3f0053b00 0x7fd3f0055fc0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fd40000b000 tx=0x7fd400006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:43.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.085+0000 7fd4067fc700 1 -- 172.21.15.60:0/867656144 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd40c047e90 con 0x7fd410145b00 2022-06-29T15:58:43.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.217+0000 7fd416803700 1 -- 172.21.15.60:0/867656144 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fd4101446b0 con 0x7fd410145b00 2022-06-29T15:58:43.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.217+0000 7fd4067fc700 1 -- 172.21.15.60:0/867656144 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v23) v1 ==== 110+0+10 (secure 0 0 0) 0x7fd40c04b4b0 con 0x7fd410145b00 2022-06-29T15:58:43.222 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T15:58:43.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.217+0000 7fd416803700 1 -- 172.21.15.60:0/867656144 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd3f0053b00 msgr2=0x7fd3f0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:43.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.217+0000 7fd416803700 1 --2- 172.21.15.60:0/867656144 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd3f0053b00 0x7fd3f0055fc0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fd40000b000 tx=0x7fd400006df0).stop 2022-06-29T15:58:43.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.217+0000 7fd416803700 1 -- 172.21.15.60:0/867656144 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd410145b00 msgr2=0x7fd41014f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:43.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.217+0000 7fd416803700 1 --2- 172.21.15.60:0/867656144 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd410145b00 0x7fd41014f920 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fd40c001490 tx=0x7fd40c016000).stop 2022-06-29T15:58:43.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.217+0000 7fd416803700 1 -- 172.21.15.60:0/867656144 shutdown_connections 2022-06-29T15:58:43.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.217+0000 7fd416803700 1 --2- 172.21.15.60:0/867656144 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd3f0053b00 0x7fd3f0055fc0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.217+0000 7fd416803700 1 --2- 172.21.15.60:0/867656144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4101449c0 0x7fd410144e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.217+0000 7fd416803700 1 --2- 172.21.15.60:0/867656144 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd410145b00 0x7fd41014f920 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.217+0000 7fd416803700 1 --2- 172.21.15.60:0/867656144 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3fc004500 0x7fd41013b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.217+0000 7fd416803700 1 -- 172.21.15.60:0/867656144 >> 172.21.15.60:0/867656144 conn(0x7fd4100b6120 msgr2=0x7fd410143070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:43.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.217+0000 7fd416803700 1 -- 172.21.15.60:0/867656144 shutdown_connections 2022-06-29T15:58:43.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.217+0000 7fd416803700 1 -- 172.21.15.60:0/867656144 wait complete. 2022-06-29T15:58:43.236 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T15:58:43.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.337+0000 7f45ecb13700 1 Processor -- start 2022-06-29T15:58:43.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.337+0000 7f45ecb13700 1 -- start start 2022-06-29T15:58:43.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.337+0000 7f45ecb13700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f45e80b5eb0 0x7f45e8142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:43.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.337+0000 7f45ecb13700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45e80b48b0 0x7f45e80b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:43.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.337+0000 7f45ecb13700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f45e80b2c50 0x7f45e80b3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:43.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.337+0000 7f45ecb13700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45e80b54e0 con 0x7f45e80b2c50 2022-06-29T15:58:43.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.337+0000 7f45ecb13700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45e80b4d10 con 0x7f45e80b5eb0 2022-06-29T15:58:43.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.337+0000 7f45ecb13700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45e80b6290 con 0x7f45e80b48b0 2022-06-29T15:58:43.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.337+0000 7f45e7fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f45e80b2c50 0x7f45e80b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:43.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.337+0000 7f45e7fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f45e80b2c50 0x7f45e80b3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45688/0 (socket says 172.21.15.60:45688) 2022-06-29T15:58:43.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.337+0000 7f45e7fff700 1 -- 172.21.15.60:0/531526506 learned_addr learned my addr 172.21.15.60:0/531526506 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:43.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.337+0000 7f45e6ffd700 1 --2- 172.21.15.60:0/531526506 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45e80b48b0 0x7f45e80b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:43.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.337+0000 7f45e77fe700 1 --2- 172.21.15.60:0/531526506 >> v2:172.21.15.92:3300/0 conn(0x7f45e80b5eb0 0x7f45e8142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:43.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.341+0000 7f45e6ffd700 1 -- 172.21.15.60:0/531526506 >> v2:172.21.15.92:3300/0 conn(0x7f45e80b5eb0 msgr2=0x7f45e8142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:43.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.341+0000 7f45e6ffd700 1 --2- 172.21.15.60:0/531526506 >> v2:172.21.15.92:3300/0 conn(0x7f45e80b5eb0 0x7f45e8142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.341+0000 7f45e6ffd700 1 -- 172.21.15.60:0/531526506 >> v2:172.21.15.60:3300/0 conn(0x7f45e80b2c50 msgr2=0x7f45e80b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:43.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.341+0000 7f45e6ffd700 1 --2- 172.21.15.60:0/531526506 >> v2:172.21.15.60:3300/0 conn(0x7f45e80b2c50 0x7f45e80b3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.341+0000 7f45e6ffd700 1 -- 172.21.15.60:0/531526506 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e8142f10 con 0x7f45e80b48b0 2022-06-29T15:58:43.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.341+0000 7f45e6ffd700 1 --2- 172.21.15.60:0/531526506 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45e80b48b0 0x7f45e80b05e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f45d8002f70 tx=0x7f45d8006df0).ready entity=mon.2 client_cookie=6fec57b24b336c26 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:43.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.341+0000 7f45e67fc700 1 -- 172.21.15.60:0/531526506 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45d800d4a0 con 0x7f45e80b48b0 2022-06-29T15:58:43.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.341+0000 7f45e67fc700 1 -- 172.21.15.60:0/531526506 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45d8000e90 con 0x7f45e80b48b0 2022-06-29T15:58:43.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.341+0000 7f45e67fc700 1 -- 172.21.15.60:0/531526506 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45d800d7f0 con 0x7f45e80b48b0 2022-06-29T15:58:43.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.341+0000 7f45ecb13700 1 -- 172.21.15.60:0/531526506 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45e80b48b0 msgr2=0x7f45e80b05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:43.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.341+0000 7f45ecb13700 1 --2- 172.21.15.60:0/531526506 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45e80b48b0 0x7f45e80b05e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f45d8002f70 tx=0x7f45d8006df0).stop 2022-06-29T15:58:43.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.341+0000 7f45ecb13700 1 -- 172.21.15.60:0/531526506 shutdown_connections 2022-06-29T15:58:43.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.341+0000 7f45ecb13700 1 --2- 172.21.15.60:0/531526506 >> v2:172.21.15.60:3300/0 conn(0x7f45e80b2c50 0x7f45e80b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.341+0000 7f45ecb13700 1 --2- 172.21.15.60:0/531526506 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45e80b48b0 0x7f45e80b05e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.341+0000 7f45ecb13700 1 --2- 172.21.15.60:0/531526506 >> v2:172.21.15.92:3300/0 conn(0x7f45e80b5eb0 0x7f45e8142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.341+0000 7f45ecb13700 1 -- 172.21.15.60:0/531526506 >> 172.21.15.60:0/531526506 conn(0x7f45e8062090 msgr2=0x7f45e8062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:43.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.341+0000 7f45ecb13700 1 -- 172.21.15.60:0/531526506 shutdown_connections 2022-06-29T15:58:43.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.341+0000 7f45ecb13700 1 -- 172.21.15.60:0/531526506 wait complete. 2022-06-29T15:58:43.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.341+0000 7f45ecb13700 1 Processor -- start 2022-06-29T15:58:43.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.341+0000 7f45ecb13700 1 -- start start 2022-06-29T15:58:43.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.345+0000 7f45ecb13700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45e80b2c50 0x7f45e8151b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:43.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.345+0000 7f45ecb13700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45e80b48b0 0x7f45e80af1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:43.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.345+0000 7f45ecb13700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45e80b5eb0 0x7f45e8152170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:43.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.345+0000 7f45ecb13700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e8140140 con 0x7f45e80b2c50 2022-06-29T15:58:43.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.345+0000 7f45ecb13700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e8141170 con 0x7f45e80b48b0 2022-06-29T15:58:43.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.345+0000 7f45ecb13700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45e8140370 con 0x7f45e80b5eb0 2022-06-29T15:58:43.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.345+0000 7f45e77fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45e80b2c50 0x7f45e8151b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:43.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.345+0000 7f45e77fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45e80b2c50 0x7f45e8151b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45690/0 (socket says 172.21.15.60:45690) 2022-06-29T15:58:43.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.345+0000 7f45e77fe700 1 -- 172.21.15.60:0/2717161755 learned_addr learned my addr 172.21.15.60:0/2717161755 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:43.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.345+0000 7f45e7fff700 1 --2- 172.21.15.60:0/2717161755 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45e80b5eb0 0x7f45e8152170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:43.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.345+0000 7f45e6ffd700 1 --2- 172.21.15.60:0/2717161755 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45e80b48b0 0x7f45e80af1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:43.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.345+0000 7f45e77fe700 1 -- 172.21.15.60:0/2717161755 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45e80b5eb0 msgr2=0x7f45e8152170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:43.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.345+0000 7f45e77fe700 1 --2- 172.21.15.60:0/2717161755 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45e80b5eb0 0x7f45e8152170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.345+0000 7f45e77fe700 1 -- 172.21.15.60:0/2717161755 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45e80b48b0 msgr2=0x7f45e80af1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:43.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.345+0000 7f45e77fe700 1 --2- 172.21.15.60:0/2717161755 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45e80b48b0 0x7f45e80af1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.345+0000 7f45e77fe700 1 -- 172.21.15.60:0/2717161755 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d800b180 con 0x7f45e80b2c50 2022-06-29T15:58:43.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.345+0000 7f45e6ffd700 1 --2- 172.21.15.60:0/2717161755 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45e80b48b0 0x7f45e80af1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:43.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.345+0000 7f45e7fff700 1 --2- 172.21.15.60:0/2717161755 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45e80b5eb0 0x7f45e8152170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:43.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.345+0000 7f45e77fe700 1 --2- 172.21.15.60:0/2717161755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45e80b2c50 0x7f45e8151b30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f45d000c000 tx=0x7f45d0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:43.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.345+0000 7f45e4ff9700 1 -- 172.21.15.60:0/2717161755 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45d0011750 con 0x7f45e80b2c50 2022-06-29T15:58:43.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.345+0000 7f45e4ff9700 1 -- 172.21.15.60:0/2717161755 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45d00118b0 con 0x7f45e80b2c50 2022-06-29T15:58:43.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.345+0000 7f45ecb13700 1 -- 172.21.15.60:0/2717161755 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45e8148ca0 con 0x7f45e80b2c50 2022-06-29T15:58:43.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.345+0000 7f45e4ff9700 1 -- 172.21.15.60:0/2717161755 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45d0011c10 con 0x7f45e80b2c50 2022-06-29T15:58:43.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.345+0000 7f45ecb13700 1 -- 172.21.15.60:0/2717161755 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45e8143ee0 con 0x7f45e80b2c50 2022-06-29T15:58:43.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.349+0000 7f45ecb13700 1 -- 172.21.15.60:0/2717161755 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45e80b4f70 con 0x7f45e80b2c50 2022-06-29T15:58:43.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.349+0000 7f45e4ff9700 1 -- 172.21.15.60:0/2717161755 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f45d0011d70 con 0x7f45e80b2c50 2022-06-29T15:58:43.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.349+0000 7f45e4ff9700 1 --2- 172.21.15.60:0/2717161755 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f45cc0539e0 0x7f45cc055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:43.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.349+0000 7f45e4ff9700 1 -- 172.21.15.60:0/2717161755 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f45d006c2c0 con 0x7f45e80b2c50 2022-06-29T15:58:43.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.349+0000 7f45e6ffd700 1 --2- 172.21.15.60:0/2717161755 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f45cc0539e0 0x7f45cc055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:43.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.349+0000 7f45e6ffd700 1 --2- 172.21.15.60:0/2717161755 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f45cc0539e0 0x7f45cc055ea0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f45d801a000 tx=0x7f45d8014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:43.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.353+0000 7f45e4ff9700 1 -- 172.21.15.60:0/2717161755 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f45d00431e0 con 0x7f45e80b2c50 2022-06-29T15:58:43.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.485+0000 7f45ecb13700 1 -- 172.21.15.60:0/2717161755 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f45e80573d0 con 0x7f45e80b2c50 2022-06-29T15:58:43.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.485+0000 7f45e4ff9700 1 -- 172.21.15.60:0/2717161755 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f45d0046800 con 0x7f45e80b2c50 2022-06-29T15:58:43.491 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T15:58:43.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.485+0000 7f45ecb13700 1 -- 172.21.15.60:0/2717161755 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f45cc0539e0 msgr2=0x7f45cc055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:43.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.485+0000 7f45ecb13700 1 --2- 172.21.15.60:0/2717161755 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f45cc0539e0 0x7f45cc055ea0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f45d801a000 tx=0x7f45d8014000).stop 2022-06-29T15:58:43.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.485+0000 7f45ecb13700 1 -- 172.21.15.60:0/2717161755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45e80b2c50 msgr2=0x7f45e8151b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:43.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.485+0000 7f45ecb13700 1 --2- 172.21.15.60:0/2717161755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45e80b2c50 0x7f45e8151b30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f45d000c000 tx=0x7f45d0006df0).stop 2022-06-29T15:58:43.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.485+0000 7f45ecb13700 1 -- 172.21.15.60:0/2717161755 shutdown_connections 2022-06-29T15:58:43.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.485+0000 7f45ecb13700 1 --2- 172.21.15.60:0/2717161755 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f45cc0539e0 0x7f45cc055ea0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.485+0000 7f45ecb13700 1 --2- 172.21.15.60:0/2717161755 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45e80b5eb0 0x7f45e8152170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.485+0000 7f45ecb13700 1 --2- 172.21.15.60:0/2717161755 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45e80b48b0 0x7f45e80af1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.485+0000 7f45ecb13700 1 --2- 172.21.15.60:0/2717161755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45e80b2c50 0x7f45e8151b30 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.485+0000 7f45ecb13700 1 -- 172.21.15.60:0/2717161755 >> 172.21.15.60:0/2717161755 conn(0x7f45e8062090 msgr2=0x7f45e8143100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:43.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.489+0000 7f45ecb13700 1 -- 172.21.15.60:0/2717161755 shutdown_connections 2022-06-29T15:58:43.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.489+0000 7f45ecb13700 1 -- 172.21.15.60:0/2717161755 wait complete. 2022-06-29T15:58:43.504 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-29T15:58:43.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.605+0000 7f3af6e6c700 1 Processor -- start 2022-06-29T15:58:43.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.605+0000 7f3af6e6c700 1 -- start start 2022-06-29T15:58:43.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.605+0000 7f3af6e6c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3af00aca20 0x7f3af00ace00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:43.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.605+0000 7f3af6e6c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3af00b24f0 0x7f3af01468c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:43.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.605+0000 7f3af6e6c700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af0143da0 con 0x7f3af0146db0 2022-06-29T15:58:43.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.605+0000 7f3af6e6c700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3af00adf00 con 0x7f3af00aca20 2022-06-29T15:58:43.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.605+0000 7f3af6e6c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3af005cf40 con 0x7f3af00b24f0 2022-06-29T15:58:43.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.605+0000 7f3af4c08700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3af00aca20 0x7f3af00ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:43.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.605+0000 7f3af4c08700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3af00aca20 0x7f3af00ace00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45698/0 (socket says 172.21.15.60:45698) 2022-06-29T15:58:43.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.605+0000 7f3af4c08700 1 -- 172.21.15.60:0/2246144078 learned_addr learned my addr 172.21.15.60:0/2246144078 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:43.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.605+0000 7f3aeffff700 1 --2- 172.21.15.60:0/2246144078 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3af00b24f0 0x7f3af01468c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:43.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.605+0000 7f3af4c08700 1 -- 172.21.15.60:0/2246144078 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3af00b24f0 msgr2=0x7f3af01468c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:43.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.605+0000 7f3af4c08700 1 --2- 172.21.15.60:0/2246144078 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3af00b24f0 0x7f3af01468c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.605+0000 7f3af4c08700 1 -- 172.21.15.60:0/2246144078 >> v1:172.21.15.92:6789/0 conn(0x7f3af0146db0 legacy=0x7f3af0065cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:43.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.605+0000 7f3aef7fe700 1 -- 172.21.15.60:0/2246144078 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2861228312 0 0) 0x7f3af0143da0 con 0x7f3af0146db0 2022-06-29T15:58:43.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.605+0000 7f3af4c08700 1 -- 172.21.15.60:0/2246144078 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af005d1e0 con 0x7f3af00aca20 2022-06-29T15:58:43.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.605+0000 7f3af4c08700 1 --2- 172.21.15.60:0/2246144078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3af00aca20 0x7f3af00ace00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f3ae4002f70 tx=0x7f3ae4006eb0).ready entity=mon.0 client_cookie=903d314e01ed8a0b server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:43.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.605+0000 7f3aef7fe700 1 -- 172.21.15.60:0/2246144078 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ae400d570 con 0x7f3af00aca20 2022-06-29T15:58:43.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.605+0000 7f3aef7fe700 1 -- 172.21.15.60:0/2246144078 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae4000e90 con 0x7f3af00aca20 2022-06-29T15:58:43.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.605+0000 7f3aef7fe700 1 -- 172.21.15.60:0/2246144078 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ae400d8c0 con 0x7f3af00aca20 2022-06-29T15:58:43.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.605+0000 7f3af6e6c700 1 -- 172.21.15.60:0/2246144078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3af00aca20 msgr2=0x7f3af00ace00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:43.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.605+0000 7f3af6e6c700 1 --2- 172.21.15.60:0/2246144078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3af00aca20 0x7f3af00ace00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f3ae4002f70 tx=0x7f3ae4006eb0).stop 2022-06-29T15:58:43.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.605+0000 7f3af6e6c700 1 -- 172.21.15.60:0/2246144078 shutdown_connections 2022-06-29T15:58:43.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.605+0000 7f3af6e6c700 1 --2- 172.21.15.60:0/2246144078 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3af00b24f0 0x7f3af01468c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.605+0000 7f3af6e6c700 1 --2- 172.21.15.60:0/2246144078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3af00aca20 0x7f3af00ace00 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.605+0000 7f3af6e6c700 1 -- 172.21.15.60:0/2246144078 >> 172.21.15.60:0/2246144078 conn(0x7f3af00b7140 msgr2=0x7f3af00b7520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:43.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.605+0000 7f3af6e6c700 1 -- 172.21.15.60:0/2246144078 shutdown_connections 2022-06-29T15:58:43.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.605+0000 7f3af6e6c700 1 -- 172.21.15.60:0/2246144078 wait complete. 2022-06-29T15:58:43.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.609+0000 7f3af6e6c700 1 Processor -- start 2022-06-29T15:58:43.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.609+0000 7f3af6e6c700 1 -- start start 2022-06-29T15:58:43.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.609+0000 7f3af6e6c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3af00aca20 0x7f3af013e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:43.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.609+0000 7f3af6e6c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3af00b24f0 0x7f3af00b6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:43.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.609+0000 7f3af6e6c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3af0146db0 0x7f3af0155870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:43.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.609+0000 7f3af6e6c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af0062b40 con 0x7f3af0146db0 2022-06-29T15:58:43.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.609+0000 7f3af6e6c700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af0062de0 con 0x7f3af00b24f0 2022-06-29T15:58:43.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.609+0000 7f3af6e6c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3af0063080 con 0x7f3af00aca20 2022-06-29T15:58:43.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.609+0000 7f3af5409700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3af0146db0 0x7f3af0155870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:43.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.609+0000 7f3af5409700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3af0146db0 0x7f3af0155870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45708/0 (socket says 172.21.15.60:45708) 2022-06-29T15:58:43.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.609+0000 7f3af5409700 1 -- 172.21.15.60:0/424348405 learned_addr learned my addr 172.21.15.60:0/424348405 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:43.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.609+0000 7f3aeffff700 1 --2- 172.21.15.60:0/424348405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3af00b24f0 0x7f3af00b6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:43.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.609+0000 7f3af4c08700 1 --2- 172.21.15.60:0/424348405 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3af00aca20 0x7f3af013e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:43.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.609+0000 7f3af5409700 1 -- 172.21.15.60:0/424348405 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3af00aca20 msgr2=0x7f3af013e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:43.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.609+0000 7f3af5409700 1 --2- 172.21.15.60:0/424348405 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3af00aca20 0x7f3af013e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.609+0000 7f3af5409700 1 -- 172.21.15.60:0/424348405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3af00b24f0 msgr2=0x7f3af00b6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:43.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.609+0000 7f3af5409700 1 --2- 172.21.15.60:0/424348405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3af00b24f0 0x7f3af00b6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.609+0000 7f3af5409700 1 -- 172.21.15.60:0/424348405 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae400b1e0 con 0x7f3af0146db0 2022-06-29T15:58:43.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.609+0000 7f3af5409700 1 --2- 172.21.15.60:0/424348405 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3af0146db0 0x7f3af0155870 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f3ae8000f40 tx=0x7f3ae8007f00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:43.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.609+0000 7f3aedffb700 1 -- 172.21.15.60:0/424348405 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ae8006aa0 con 0x7f3af0146db0 2022-06-29T15:58:43.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.609+0000 7f3aedffb700 1 -- 172.21.15.60:0/424348405 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae8006c00 con 0x7f3af0146db0 2022-06-29T15:58:43.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.609+0000 7f3af6e6c700 1 -- 172.21.15.60:0/424348405 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3af01441b0 con 0x7f3af0146db0 2022-06-29T15:58:43.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.609+0000 7f3aedffb700 1 -- 172.21.15.60:0/424348405 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ae8007030 con 0x7f3af0146db0 2022-06-29T15:58:43.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.609+0000 7f3af6e6c700 1 -- 172.21.15.60:0/424348405 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3af01402b0 con 0x7f3af0146db0 2022-06-29T15:58:43.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.613+0000 7f3aedffb700 1 -- 172.21.15.60:0/424348405 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3ae801b060 con 0x7f3af0146db0 2022-06-29T15:58:43.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.613+0000 7f3af6e6c700 1 -- 172.21.15.60:0/424348405 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3af00573d0 con 0x7f3af0146db0 2022-06-29T15:58:43.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.613+0000 7f3aedffb700 1 --2- 172.21.15.60:0/424348405 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3ad00539e0 0x7f3ad0055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:43.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.613+0000 7f3aedffb700 1 -- 172.21.15.60:0/424348405 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3ae806bb60 con 0x7f3af0146db0 2022-06-29T15:58:43.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.613+0000 7f3af4c08700 1 --2- 172.21.15.60:0/424348405 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3ad00539e0 0x7f3ad0055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:43.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.613+0000 7f3af4c08700 1 --2- 172.21.15.60:0/424348405 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3ad00539e0 0x7f3ad0055ea0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f3ae401c000 tx=0x7f3ae4015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:43.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.617+0000 7f3aedffb700 1 -- 172.21.15.60:0/424348405 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3ae8042970 con 0x7f3af0146db0 2022-06-29T15:58:43.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.741+0000 7f3af6e6c700 1 -- 172.21.15.60:0/424348405 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3af01404e0 con 0x7f3af0146db0 2022-06-29T15:58:43.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.745+0000 7f3aedffb700 1 -- 172.21.15.60:0/424348405 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7f3ae8019020 con 0x7f3af0146db0 2022-06-29T15:58:43.750 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T15:58:43.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.745+0000 7f3af6e6c700 1 -- 172.21.15.60:0/424348405 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3ad00539e0 msgr2=0x7f3ad0055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:43.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.745+0000 7f3af6e6c700 1 --2- 172.21.15.60:0/424348405 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3ad00539e0 0x7f3ad0055ea0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f3ae401c000 tx=0x7f3ae4015000).stop 2022-06-29T15:58:43.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.745+0000 7f3af6e6c700 1 -- 172.21.15.60:0/424348405 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3af0146db0 msgr2=0x7f3af0155870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:43.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.745+0000 7f3af6e6c700 1 --2- 172.21.15.60:0/424348405 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3af0146db0 0x7f3af0155870 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f3ae8000f40 tx=0x7f3ae8007f00).stop 2022-06-29T15:58:43.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.745+0000 7f3af6e6c700 1 -- 172.21.15.60:0/424348405 shutdown_connections 2022-06-29T15:58:43.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.745+0000 7f3af6e6c700 1 --2- 172.21.15.60:0/424348405 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3ad00539e0 0x7f3ad0055ea0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.745+0000 7f3af6e6c700 1 --2- 172.21.15.60:0/424348405 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3af0146db0 0x7f3af0155870 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.745+0000 7f3af6e6c700 1 --2- 172.21.15.60:0/424348405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3af00b24f0 0x7f3af00b6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.745+0000 7f3af6e6c700 1 --2- 172.21.15.60:0/424348405 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3af00aca20 0x7f3af013e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.745+0000 7f3af6e6c700 1 -- 172.21.15.60:0/424348405 >> 172.21.15.60:0/424348405 conn(0x7f3af00b7140 msgr2=0x7f3af0142c00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:43.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.745+0000 7f3af6e6c700 1 -- 172.21.15.60:0/424348405 shutdown_connections 2022-06-29T15:58:43.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.745+0000 7f3af6e6c700 1 -- 172.21.15.60:0/424348405 wait complete. 2022-06-29T15:58:43.764 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-29T15:58:43.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.865+0000 7f7f516f2700 1 Processor -- start 2022-06-29T15:58:43.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.865+0000 7f7f516f2700 1 -- start start 2022-06-29T15:58:43.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.869+0000 7f7f516f2700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f4c067a90 0x7f7f4c1400b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:43.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.869+0000 7f7f516f2700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7f4c1405a0 0x7f7f4c143fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:43.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.869+0000 7f7f516f2700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4c064e20 con 0x7f7f4c05fb00 2022-06-29T15:58:43.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.869+0000 7f7f516f2700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f4c0b10e0 con 0x7f7f4c1405a0 2022-06-29T15:58:43.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.869+0000 7f7f516f2700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4c05fee0 con 0x7f7f4c067a90 2022-06-29T15:58:43.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.869+0000 7f7f50ef1700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7f4c1405a0 0x7f7f4c143fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:43.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.869+0000 7f7f50ef1700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7f4c1405a0 0x7f7f4c143fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45716/0 (socket says 172.21.15.60:45716) 2022-06-29T15:58:43.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.869+0000 7f7f50ef1700 1 -- 172.21.15.60:0/2523405183 learned_addr learned my addr 172.21.15.60:0/2523405183 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:43.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.869+0000 7f7f43fff700 1 --2- 172.21.15.60:0/2523405183 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f4c067a90 0x7f7f4c1400b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:43.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.869+0000 7f7f43fff700 1 -- 172.21.15.60:0/2523405183 >> v2:172.21.15.60:3300/0 conn(0x7f7f4c1405a0 msgr2=0x7f7f4c143fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:43.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.869+0000 7f7f43fff700 1 --2- 172.21.15.60:0/2523405183 >> v2:172.21.15.60:3300/0 conn(0x7f7f4c1405a0 0x7f7f4c143fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.869+0000 7f7f43fff700 1 -- 172.21.15.60:0/2523405183 >> v1:172.21.15.92:6789/0 conn(0x7f7f4c05fb00 legacy=0x7f7f4c13f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:43.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.869+0000 7f7f43fff700 1 -- 172.21.15.60:0/2523405183 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f4c060b20 con 0x7f7f4c067a90 2022-06-29T15:58:43.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.869+0000 7f7f43fff700 1 --2- 172.21.15.60:0/2523405183 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f4c067a90 0x7f7f4c1400b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f7f34002f70 tx=0x7f7f34006df0).ready entity=mon.2 client_cookie=39f7892eecc17358 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:43.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.869+0000 7f7f4b7fe700 1 -- 172.21.15.60:0/2523405183 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f3400d570 con 0x7f7f4c067a90 2022-06-29T15:58:43.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.869+0000 7f7f4b7fe700 1 -- 172.21.15.60:0/2523405183 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f34000e90 con 0x7f7f4c067a90 2022-06-29T15:58:43.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.869+0000 7f7f4b7fe700 1 -- 172.21.15.60:0/2523405183 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f3400d8c0 con 0x7f7f4c067a90 2022-06-29T15:58:43.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.869+0000 7f7f516f2700 1 -- 172.21.15.60:0/2523405183 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f4c067a90 msgr2=0x7f7f4c1400b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:43.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.869+0000 7f7f516f2700 1 --2- 172.21.15.60:0/2523405183 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f4c067a90 0x7f7f4c1400b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f7f34002f70 tx=0x7f7f34006df0).stop 2022-06-29T15:58:43.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.869+0000 7f7f516f2700 1 -- 172.21.15.60:0/2523405183 shutdown_connections 2022-06-29T15:58:43.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.869+0000 7f7f516f2700 1 --2- 172.21.15.60:0/2523405183 >> v2:172.21.15.60:3300/0 conn(0x7f7f4c1405a0 0x7f7f4c143fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.869+0000 7f7f516f2700 1 --2- 172.21.15.60:0/2523405183 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f4c067a90 0x7f7f4c1400b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.869+0000 7f7f516f2700 1 -- 172.21.15.60:0/2523405183 >> 172.21.15.60:0/2523405183 conn(0x7f7f4c063a60 msgr2=0x7f7f4c05f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:43.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.869+0000 7f7f516f2700 1 -- 172.21.15.60:0/2523405183 shutdown_connections 2022-06-29T15:58:43.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.869+0000 7f7f516f2700 1 -- 172.21.15.60:0/2523405183 wait complete. 2022-06-29T15:58:43.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.873+0000 7f7f516f2700 1 Processor -- start 2022-06-29T15:58:43.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.873+0000 7f7f516f2700 1 -- start start 2022-06-29T15:58:43.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.873+0000 7f7f516f2700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f4c05fb00 0x7f7f4c067000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:43.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.873+0000 7f7f516f2700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f4c067a90 0x7f7f4c0b0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:43.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.873+0000 7f7f516f2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f4c1405a0 0x7f7f4c1427e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:43.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.873+0000 7f7f516f2700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4c137930 con 0x7f7f4c1405a0 2022-06-29T15:58:43.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.873+0000 7f7f516f2700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4c0668d0 con 0x7f7f4c05fb00 2022-06-29T15:58:43.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.873+0000 7f7f516f2700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4c1420a0 con 0x7f7f4c067a90 2022-06-29T15:58:43.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.873+0000 7f7f50ef1700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f4c1405a0 0x7f7f4c1427e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:43.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.873+0000 7f7f50ef1700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f4c1405a0 0x7f7f4c1427e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45722/0 (socket says 172.21.15.60:45722) 2022-06-29T15:58:43.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.873+0000 7f7f50ef1700 1 -- 172.21.15.60:0/4284770937 learned_addr learned my addr 172.21.15.60:0/4284770937 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:43.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.873+0000 7f7f43fff700 1 --2- 172.21.15.60:0/4284770937 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f4c067a90 0x7f7f4c0b0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:43.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.873+0000 7f7f4bfff700 1 --2- 172.21.15.60:0/4284770937 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f4c05fb00 0x7f7f4c067000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:43.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.873+0000 7f7f43fff700 1 -- 172.21.15.60:0/4284770937 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f4c05fb00 msgr2=0x7f7f4c067000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:43.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.873+0000 7f7f43fff700 1 --2- 172.21.15.60:0/4284770937 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f4c05fb00 0x7f7f4c067000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.873+0000 7f7f43fff700 1 -- 172.21.15.60:0/4284770937 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f4c1405a0 msgr2=0x7f7f4c1427e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:43.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.873+0000 7f7f43fff700 1 --2- 172.21.15.60:0/4284770937 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f4c1405a0 0x7f7f4c1427e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:43.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.873+0000 7f7f43fff700 1 -- 172.21.15.60:0/4284770937 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f3400b180 con 0x7f7f4c067a90 2022-06-29T15:58:43.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.873+0000 7f7f50ef1700 1 --2- 172.21.15.60:0/4284770937 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f4c1405a0 0x7f7f4c1427e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:43.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.877+0000 7f7f43fff700 1 --2- 172.21.15.60:0/4284770937 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f4c067a90 0x7f7f4c0b0820 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f7f4c137a70 tx=0x7f7f34014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:43.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.877+0000 7f7f49ffb700 1 -- 172.21.15.60:0/4284770937 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f34013da0 con 0x7f7f4c067a90 2022-06-29T15:58:43.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.877+0000 7f7f49ffb700 1 -- 172.21.15.60:0/4284770937 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f3400de40 con 0x7f7f4c067a90 2022-06-29T15:58:43.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.877+0000 7f7f516f2700 1 -- 172.21.15.60:0/4284770937 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f4c142340 con 0x7f7f4c067a90 2022-06-29T15:58:43.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.877+0000 7f7f49ffb700 1 -- 172.21.15.60:0/4284770937 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f3401e5f0 con 0x7f7f4c067a90 2022-06-29T15:58:43.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.877+0000 7f7f516f2700 1 -- 172.21.15.60:0/4284770937 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f4c142cd0 con 0x7f7f4c067a90 2022-06-29T15:58:43.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.877+0000 7f7f516f2700 1 -- 172.21.15.60:0/4284770937 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f4c0af740 con 0x7f7f4c067a90 2022-06-29T15:58:43.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.877+0000 7f7f49ffb700 1 -- 172.21.15.60:0/4284770937 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7f34025030 con 0x7f7f4c067a90 2022-06-29T15:58:43.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.877+0000 7f7f49ffb700 1 --2- 172.21.15.60:0/4284770937 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7f30053a70 0x7f7f30055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:43.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.877+0000 7f7f49ffb700 1 -- 172.21.15.60:0/4284770937 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7f3406d180 con 0x7f7f4c067a90 2022-06-29T15:58:43.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.877+0000 7f7f4bfff700 1 --2- 172.21.15.60:0/4284770937 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7f30053a70 0x7f7f30055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:43.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.881+0000 7f7f4bfff700 1 --2- 172.21.15.60:0/4284770937 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7f30053a70 0x7f7f30055f30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f7f3c00d000 tx=0x7f7f3c007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:43.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:43.889+0000 7f7f49ffb700 1 -- 172.21.15.60:0/4284770937 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7f340440e0 con 0x7f7f4c067a90 2022-06-29T15:58:44.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.017+0000 7f7f516f2700 1 -- 172.21.15.60:0/4284770937 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7f4c1424d0 con 0x7f7f4c067a90 2022-06-29T15:58:44.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.017+0000 7f7f49ffb700 1 -- 172.21.15.60:0/4284770937 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7f7f34023020 con 0x7f7f4c067a90 2022-06-29T15:58:44.024 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T15:58:44.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.021+0000 7f7f516f2700 1 -- 172.21.15.60:0/4284770937 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7f30053a70 msgr2=0x7f7f30055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:44.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.021+0000 7f7f516f2700 1 --2- 172.21.15.60:0/4284770937 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7f30053a70 0x7f7f30055f30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f7f3c00d000 tx=0x7f7f3c007b90).stop 2022-06-29T15:58:44.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.021+0000 7f7f516f2700 1 -- 172.21.15.60:0/4284770937 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f4c067a90 msgr2=0x7f7f4c0b0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:44.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.021+0000 7f7f516f2700 1 --2- 172.21.15.60:0/4284770937 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f4c067a90 0x7f7f4c0b0820 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f7f4c137a70 tx=0x7f7f34014000).stop 2022-06-29T15:58:44.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.021+0000 7f7f516f2700 1 -- 172.21.15.60:0/4284770937 shutdown_connections 2022-06-29T15:58:44.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.021+0000 7f7f516f2700 1 --2- 172.21.15.60:0/4284770937 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7f30053a70 0x7f7f30055f30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:44.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.021+0000 7f7f516f2700 1 --2- 172.21.15.60:0/4284770937 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f4c1405a0 0x7f7f4c1427e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:44.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.021+0000 7f7f516f2700 1 --2- 172.21.15.60:0/4284770937 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f4c067a90 0x7f7f4c0b0820 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:44.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.021+0000 7f7f516f2700 1 --2- 172.21.15.60:0/4284770937 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f4c05fb00 0x7f7f4c067000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:44.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.021+0000 7f7f516f2700 1 -- 172.21.15.60:0/4284770937 >> 172.21.15.60:0/4284770937 conn(0x7f7f4c063a60 msgr2=0x7f7f4c13f910 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:44.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.021+0000 7f7f516f2700 1 -- 172.21.15.60:0/4284770937 shutdown_connections 2022-06-29T15:58:44.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.021+0000 7f7f516f2700 1 -- 172.21.15.60:0/4284770937 wait complete. 2022-06-29T15:58:44.043 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-06-29T15:58:44.044 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T15:58:44.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.141+0000 7fabfa83e700 1 Processor -- start 2022-06-29T15:58:44.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.141+0000 7fabfa83e700 1 -- start start 2022-06-29T15:58:44.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabfa83e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fabf4060260 0x7fabf40b6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:44.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabfa83e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf41430e0 con 0x7fabf413f3d0 2022-06-29T15:58:44.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabfa83e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf4143380 con 0x7fabf40b6c00 2022-06-29T15:58:44.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabfa83e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabf4142ef0 con 0x7fabf4060260 2022-06-29T15:58:44.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf3fff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fabf413f3d0 0x7fabf4143820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53814/0 (socket says 172.21.15.60:53814) 2022-06-29T15:58:44.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf3fff700 1 -- 172.21.15.60:0/2417613577 learned_addr learned my addr 172.21.15.60:0/2417613577 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:44.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf37fe700 1 --2- 172.21.15.60:0/2417613577 >> v2:172.21.15.92:3300/0 conn(0x7fabf4060260 0x7fabf40b6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:44.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf2ffd700 1 -- 172.21.15.60:0/2417613577 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2910042680 0 0) 0x7fabf41430e0 con 0x7fabf413f3d0 2022-06-29T15:58:44.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf2ffd700 1 -- 172.21.15.60:0/2417613577 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe0003260 con 0x7fabf413f3d0 2022-06-29T15:58:44.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf2ffd700 1 -- 172.21.15.60:0/2417613577 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 42084087 0 0) 0x7fabf4143380 con 0x7fabf40b6c00 2022-06-29T15:58:44.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf2ffd700 1 -- 172.21.15.60:0/2417613577 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabf41430e0 con 0x7fabf40b6c00 2022-06-29T15:58:44.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf2ffd700 1 -- 172.21.15.60:0/2417613577 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3066004606 0 0) 0x7fabe0003260 con 0x7fabf413f3d0 2022-06-29T15:58:44.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf2ffd700 1 -- 172.21.15.60:0/2417613577 >> v2:172.21.15.92:3300/0 conn(0x7fabf4060260 msgr2=0x7fabf40b6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:44.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf2ffd700 1 --2- 172.21.15.60:0/2417613577 >> v2:172.21.15.92:3300/0 conn(0x7fabf4060260 0x7fabf40b6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:44.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf2ffd700 1 -- 172.21.15.60:0/2417613577 >> v1:172.21.15.92:6789/0 conn(0x7fabf40b6c00 legacy=0x7fabf413f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:44.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf2ffd700 1 -- 172.21.15.60:0/2417613577 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf405fed0 con 0x7fabf413f3d0 2022-06-29T15:58:44.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf2ffd700 1 -- 172.21.15.60:0/2417613577 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7fabe8001660 con 0x7fabf413f3d0 2022-06-29T15:58:44.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf2ffd700 1 -- 172.21.15.60:0/2417613577 >> v1:172.21.15.60:6789/0 conn(0x7fabf413f3d0 legacy=0x7fabf4143820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:44.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf2ffd700 1 --2- 172.21.15.60:0/2417613577 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fabe00038e0 0x7fabe0003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:44.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf2ffd700 1 --2- 172.21.15.60:0/2417613577 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabe0004500 0x7fabe0004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:44.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf2ffd700 1 --2- 172.21.15.60:0/2417613577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fabe0004e70 0x7fabe000d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:44.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf2ffd700 1 -- 172.21.15.60:0/2417613577 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe000dfe0 con 0x7fabe0004e70 2022-06-29T15:58:44.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf2ffd700 1 -- 172.21.15.60:0/2417613577 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe000e120 con 0x7fabe0004500 2022-06-29T15:58:44.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf2ffd700 1 -- 172.21.15.60:0/2417613577 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabe000e290 con 0x7fabe00038e0 2022-06-29T15:58:44.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf37fe700 1 --2- 172.21.15.60:0/2417613577 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabe0004500 0x7fabe0004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:44.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf8ddb700 1 --2- 172.21.15.60:0/2417613577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fabe0004e70 0x7fabe000d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:44.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf3fff700 1 --2- 172.21.15.60:0/2417613577 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fabe00038e0 0x7fabe0003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:44.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf37fe700 1 -- 172.21.15.60:0/2417613577 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fabe00038e0 msgr2=0x7fabe0003ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:44.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf37fe700 1 --2- 172.21.15.60:0/2417613577 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fabe00038e0 0x7fabe0003ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:44.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf37fe700 1 -- 172.21.15.60:0/2417613577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fabe0004e70 msgr2=0x7fabe000d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:44.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf37fe700 1 --2- 172.21.15.60:0/2417613577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fabe0004e70 0x7fabe000d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:44.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf37fe700 1 -- 172.21.15.60:0/2417613577 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe0003260 con 0x7fabe0004500 2022-06-29T15:58:44.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf3fff700 1 --2- 172.21.15.60:0/2417613577 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fabe00038e0 0x7fabe0003ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:44.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf8ddb700 1 --2- 172.21.15.60:0/2417613577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fabe0004e70 0x7fabe000d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:44.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf8ddb700 1 -- 172.21.15.60:0/2417613577 reap_dead start 2022-06-29T15:58:44.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf37fe700 1 --2- 172.21.15.60:0/2417613577 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabe0004500 0x7fabe0004980 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fabe400c000 tx=0x7fabe4006df0).ready entity=mon.1 client_cookie=648fea7bb698a998 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:44.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf2ffd700 1 -- 172.21.15.60:0/2417613577 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabe400e830 con 0x7fabe0004500 2022-06-29T15:58:44.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf2ffd700 1 -- 172.21.15.60:0/2417613577 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe400e990 con 0x7fabe0004500 2022-06-29T15:58:44.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabf2ffd700 1 -- 172.21.15.60:0/2417613577 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabe400ecf0 con 0x7fabe0004500 2022-06-29T15:58:44.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabfa83e700 1 -- 172.21.15.60:0/2417613577 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabe0004500 msgr2=0x7fabe0004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:44.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabfa83e700 1 --2- 172.21.15.60:0/2417613577 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabe0004500 0x7fabe0004980 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fabe400c000 tx=0x7fabe4006df0).stop 2022-06-29T15:58:44.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabfa83e700 1 -- 172.21.15.60:0/2417613577 shutdown_connections 2022-06-29T15:58:44.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabfa83e700 1 --2- 172.21.15.60:0/2417613577 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabe0004500 0x7fabe0004980 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:44.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabfa83e700 1 -- 172.21.15.60:0/2417613577 >> 172.21.15.60:0/2417613577 conn(0x7fabf40643e0 msgr2=0x7fabf40b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:44.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabfa83e700 1 -- 172.21.15.60:0/2417613577 shutdown_connections 2022-06-29T15:58:44.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.145+0000 7fabfa83e700 1 -- 172.21.15.60:0/2417613577 wait complete. 2022-06-29T15:58:44.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.149+0000 7fabfa83e700 1 Processor -- start 2022-06-29T15:58:44.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.149+0000 7fabfa83e700 1 -- start start 2022-06-29T15:58:44.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.149+0000 7fabfa83e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fabe0004500 0x7fabf409a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:44.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.149+0000 7fabfa83e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fabf409bbf0 0x7fabf4099900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:44.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.149+0000 7fabfa83e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabf409a7c0 0x7fabf409ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:44.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.149+0000 7fabfa83e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf409b130 con 0x7fabe0004500 2022-06-29T15:58:44.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.149+0000 7fabfa83e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf40b55c0 con 0x7fabf409a7c0 2022-06-29T15:58:44.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.149+0000 7fabfa83e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabf414e490 con 0x7fabf409bbf0 2022-06-29T15:58:44.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.149+0000 7fabf3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fabe0004500 0x7fabf409a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:44.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.149+0000 7fabf3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fabe0004500 0x7fabf409a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45738/0 (socket says 172.21.15.60:45738) 2022-06-29T15:58:44.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.149+0000 7fabf3fff700 1 -- 172.21.15.60:0/3704211528 learned_addr learned my addr 172.21.15.60:0/3704211528 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:44.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.149+0000 7fabf8ddb700 1 --2- 172.21.15.60:0/3704211528 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabf409a7c0 0x7fabf409ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:44.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.149+0000 7fabf37fe700 1 --2- 172.21.15.60:0/3704211528 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fabf409bbf0 0x7fabf4099900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:44.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.149+0000 7fabf3fff700 1 -- 172.21.15.60:0/3704211528 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fabf409bbf0 msgr2=0x7fabf4099900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:44.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.149+0000 7fabf3fff700 1 --2- 172.21.15.60:0/3704211528 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fabf409bbf0 0x7fabf4099900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:44.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.149+0000 7fabf3fff700 1 -- 172.21.15.60:0/3704211528 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabf409a7c0 msgr2=0x7fabf409ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:44.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.149+0000 7fabf3fff700 1 --2- 172.21.15.60:0/3704211528 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabf409a7c0 0x7fabf409ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:44.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.149+0000 7fabf3fff700 1 -- 172.21.15.60:0/3704211528 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf4099df0 con 0x7fabe0004500 2022-06-29T15:58:44.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.149+0000 7fabf3fff700 1 --2- 172.21.15.60:0/3704211528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fabe0004500 0x7fabf409a180 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fabe800d000 tx=0x7fabf40abb60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:44.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.149+0000 7fabf17fa700 1 -- 172.21.15.60:0/3704211528 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabe8011780 con 0x7fabe0004500 2022-06-29T15:58:44.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.153+0000 7fabf17fa700 1 -- 172.21.15.60:0/3704211528 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe80118e0 con 0x7fabe0004500 2022-06-29T15:58:44.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.153+0000 7fabfa83e700 1 -- 172.21.15.60:0/3704211528 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabf40b5220 con 0x7fabe0004500 2022-06-29T15:58:44.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.153+0000 7fabf17fa700 1 -- 172.21.15.60:0/3704211528 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabe8011c40 con 0x7fabe0004500 2022-06-29T15:58:44.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.153+0000 7fabfa83e700 1 -- 172.21.15.60:0/3704211528 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabf40b5050 con 0x7fabe0004500 2022-06-29T15:58:44.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.153+0000 7fabfa83e700 1 -- 172.21.15.60:0/3704211528 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabf4099f80 con 0x7fabe0004500 2022-06-29T15:58:44.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.153+0000 7fabf17fa700 1 -- 172.21.15.60:0/3704211528 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fabe801b030 con 0x7fabe0004500 2022-06-29T15:58:44.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.153+0000 7fabf17fa700 1 --2- 172.21.15.60:0/3704211528 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fabd4053a70 0x7fabd4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:44.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.153+0000 7fabf17fa700 1 -- 172.21.15.60:0/3704211528 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fabe806bb50 con 0x7fabe0004500 2022-06-29T15:58:44.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.153+0000 7fabf37fe700 1 --2- 172.21.15.60:0/3704211528 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fabd4053a70 0x7fabd4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:44.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.153+0000 7fabf37fe700 1 --2- 172.21.15.60:0/3704211528 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fabd4053a70 0x7fabd4055f30 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fabe401d000 tx=0x7fabe4019000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:44.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.157+0000 7fabf17fa700 1 -- 172.21.15.60:0/3704211528 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fabe8042960 con 0x7fabe0004500 2022-06-29T15:58:44.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.321+0000 7fabfa83e700 1 -- 172.21.15.60:0/3704211528 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabf40b5410 con 0x7fabe0004500 2022-06-29T15:58:44.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.325+0000 7fabf17fa700 1 -- 172.21.15.60:0/3704211528 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2134 (secure 0 0 0) 0x7fabe8019030 con 0x7fabe0004500 2022-06-29T15:58:44.330 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:58:44.330 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":4,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.92:6835/2121698334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2121698334},{"type":"v1","addr":"172.21.15.92:6835","nonce":2121698334}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":4,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T15:58:24.334770+0000","modified":"2022-06-29T15:58:25.338504+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"a","rank":0,"incarnation":3,"state":"up:active","state_seq":2,"addr":"172.21.15.60:6833/3164364318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3164364318},{"type":"v1","addr":"172.21.15.60:6833","nonce":3164364318}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-29T15:58:44.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.325+0000 7fabfa83e700 1 -- 172.21.15.60:0/3704211528 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fabd4053a70 msgr2=0x7fabd4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:44.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.325+0000 7fabfa83e700 1 --2- 172.21.15.60:0/3704211528 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fabd4053a70 0x7fabd4055f30 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fabe401d000 tx=0x7fabe4019000).stop 2022-06-29T15:58:44.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.325+0000 7fabfa83e700 1 -- 172.21.15.60:0/3704211528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fabe0004500 msgr2=0x7fabf409a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:44.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.325+0000 7fabfa83e700 1 --2- 172.21.15.60:0/3704211528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fabe0004500 0x7fabf409a180 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fabe800d000 tx=0x7fabf40abb60).stop 2022-06-29T15:58:44.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.325+0000 7fabfa83e700 1 -- 172.21.15.60:0/3704211528 shutdown_connections 2022-06-29T15:58:44.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.325+0000 7fabfa83e700 1 --2- 172.21.15.60:0/3704211528 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fabd4053a70 0x7fabd4055f30 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:44.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.325+0000 7fabfa83e700 1 --2- 172.21.15.60:0/3704211528 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabf409a7c0 0x7fabf409ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:44.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.325+0000 7fabfa83e700 1 --2- 172.21.15.60:0/3704211528 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fabf409bbf0 0x7fabf4099900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:44.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.325+0000 7fabfa83e700 1 --2- 172.21.15.60:0/3704211528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fabe0004500 0x7fabf409a180 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:44.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.325+0000 7fabfa83e700 1 -- 172.21.15.60:0/3704211528 >> 172.21.15.60:0/3704211528 conn(0x7fabf40643e0 msgr2=0x7fabf40b17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:44.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.325+0000 7fabfa83e700 1 -- 172.21.15.60:0/3704211528 shutdown_connections 2022-06-29T15:58:44.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:44.325+0000 7fabfa83e700 1 -- 172.21.15.60:0/3704211528 wait complete. 2022-06-29T15:58:44.335 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 4 2022-06-29T15:58:44.545 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 4, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T15:58:24.334770+0000', 'modified': '2022-06-29T15:58:25.338504+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4257}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4257': {'gid': 4257, 'name': 'a', 'rank': 0, 'incarnation': 3, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.60:6833/3164364318', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6832', 'nonce': 3164364318}, {'type': 'v1', 'addr': '172.21.15.60:6833', 'nonce': 3164364318}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T15:58:44.546 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T15:58:44.547 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-06-29T15:58:44.661 INFO:teuthology.orchestra.run.smithi060.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":4257,"want_state":"up:active","state":"up:active","rank_uptime":20.311340614999999,"mdsmap_epoch":4,"osdmap_epoch":23,"osdmap_epoch_barrier":21,"uptime":23.561440665999999} 2022-06-29T15:58:44.662 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 4257, "mdsmap_epoch": 4, "osdmap_epoch": 23, "osdmap_epoch_barrier": 21, "rank_uptime": 20.311340615, "state": "up:active", "uptime": 23.561440666, "want_state": "up:active", "whoami": 0 } 2022-06-29T15:58:44.663 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-06-29T15:58:44.663 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2022-06-29T15:58:44.663 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.2 at ubuntu@smithi197.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.2... 2022-06-29T15:58:44.664 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:44.664 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.2) 2022-06-29T15:58:44.673 DEBUG:teuthology.orchestra.run.smithi197:> sudo modprobe fuse 2022-06-29T15:58:44.692 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T15:58:44.692 DEBUG:teuthology.orchestra.run.smithi197:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T15:58:44.709 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T15:58:44.710 INFO:teuthology.orchestra.run.smithi197.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T15:58:44.711 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:44.712 DEBUG:teuthology.orchestra.run.smithi197:> ls /sys/fs/fuse/connections 2022-06-29T15:58:44.717 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-06-29T15:58:44.718 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.2 /home/ubuntu/cephtest/mnt.2) 2022-06-29T15:58:44.760 DEBUG:teuthology.orchestra.run.smithi197:> sudo modprobe fuse 2022-06-29T15:58:44.775 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T15:58:44.776 DEBUG:teuthology.orchestra.run.smithi197:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T15:58:44.786 INFO:teuthology.orchestra.run.smithi197.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T15:58:44.787 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T15:58:44.788 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:44.788 DEBUG:teuthology.orchestra.run.smithi197:> ls /sys/fs/fuse/connections 2022-06-29T15:58:44.845 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi197.stderr:2022-06-29T15:58:44.844+0000 7f882bc5e080 -1 init, newargv = 0x560ab19ce0a0 newargc=9 2022-06-29T15:58:44.845 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi197.stderr:ceph-fuse[12289]: starting ceph client 2022-06-29T15:58:44.853 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi197.stderr:ceph-fuse[12289]: starting fuse 2022-06-29T15:58:45.895 DEBUG:teuthology.orchestra.run.smithi197:> sudo modprobe fuse 2022-06-29T15:58:45.914 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T15:58:45.914 DEBUG:teuthology.orchestra.run.smithi197:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T15:58:45.970 INFO:teuthology.orchestra.run.smithi197.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T15:58:45.971 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T15:58:45.972 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:45.972 DEBUG:teuthology.orchestra.run.smithi197:> ls /sys/fs/fuse/connections 2022-06-29T15:58:45.979 INFO:teuthology.orchestra.run.smithi197.stdout:52 2022-06-29T15:58:45.980 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-06-29T15:58:45.980 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:45.981 DEBUG:teuthology.orchestra.run.smithi197:> sudo python3 -c ' 2022-06-29T15:58:45.981 DEBUG:teuthology.orchestra.run.smithi197:> import glob 2022-06-29T15:58:45.981 DEBUG:teuthology.orchestra.run.smithi197:> import re 2022-06-29T15:58:45.982 DEBUG:teuthology.orchestra.run.smithi197:> import os 2022-06-29T15:58:45.982 DEBUG:teuthology.orchestra.run.smithi197:> import subprocess 2022-06-29T15:58:45.982 DEBUG:teuthology.orchestra.run.smithi197:> 2022-06-29T15:58:45.983 DEBUG:teuthology.orchestra.run.smithi197:> def find_socket(client_name): 2022-06-29T15:58:45.983 DEBUG:teuthology.orchestra.run.smithi197:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2022-06-29T15:58:45.983 DEBUG:teuthology.orchestra.run.smithi197:> files = glob.glob(asok_path) 2022-06-29T15:58:45.984 DEBUG:teuthology.orchestra.run.smithi197:> 2022-06-29T15:58:45.984 DEBUG:teuthology.orchestra.run.smithi197:> # Given a non-glob path, it better be there 2022-06-29T15:58:45.984 DEBUG:teuthology.orchestra.run.smithi197:> if "*" not in asok_path: 2022-06-29T15:58:45.985 DEBUG:teuthology.orchestra.run.smithi197:> assert(len(files) == 1) 2022-06-29T15:58:45.985 DEBUG:teuthology.orchestra.run.smithi197:> return files[0] 2022-06-29T15:58:45.985 DEBUG:teuthology.orchestra.run.smithi197:> 2022-06-29T15:58:45.986 DEBUG:teuthology.orchestra.run.smithi197:> for f in files: 2022-06-29T15:58:45.986 DEBUG:teuthology.orchestra.run.smithi197:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-06-29T15:58:45.986 DEBUG:teuthology.orchestra.run.smithi197:> if os.path.exists("/proc/{0}".format(pid)): 2022-06-29T15:58:45.987 DEBUG:teuthology.orchestra.run.smithi197:> return f 2022-06-29T15:58:45.987 DEBUG:teuthology.orchestra.run.smithi197:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-06-29T15:58:45.987 DEBUG:teuthology.orchestra.run.smithi197:> 2022-06-29T15:58:45.988 DEBUG:teuthology.orchestra.run.smithi197:> print(find_socket("client.2")) 2022-06-29T15:58:45.988 DEBUG:teuthology.orchestra.run.smithi197:> ' 2022-06-29T15:58:46.032 INFO:teuthology.orchestra.run.smithi197.stdout:/var/run/ceph/ceph-client.2.12289.asok 2022-06-29T15:58:46.037 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.12289.asok 2022-06-29T15:58:46.038 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:46.038 DEBUG:teuthology.orchestra.run.smithi197:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.12289.asok status 2022-06-29T15:58:46.127 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2022-06-29T15:58:46.128 INFO:teuthology.orchestra.run.smithi197.stdout: "metadata": { 2022-06-29T15:58:46.128 INFO:teuthology.orchestra.run.smithi197.stdout: "ceph_sha1": "f25ffac72e5a9e63860ff7deb6767a11f0559a16", 2022-06-29T15:58:46.128 INFO:teuthology.orchestra.run.smithi197.stdout: "ceph_version": "ceph version 15.2.16-304-gf25ffac7 (f25ffac72e5a9e63860ff7deb6767a11f0559a16) octopus (stable)", 2022-06-29T15:58:46.129 INFO:teuthology.orchestra.run.smithi197.stdout: "entity_id": "2", 2022-06-29T15:58:46.129 INFO:teuthology.orchestra.run.smithi197.stdout: "hostname": "smithi197", 2022-06-29T15:58:46.129 INFO:teuthology.orchestra.run.smithi197.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2022-06-29T15:58:46.130 INFO:teuthology.orchestra.run.smithi197.stdout: "pid": "12289", 2022-06-29T15:58:46.130 INFO:teuthology.orchestra.run.smithi197.stdout: "root": "/" 2022-06-29T15:58:46.131 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2022-06-29T15:58:46.131 INFO:teuthology.orchestra.run.smithi197.stdout: "dentry_count": 0, 2022-06-29T15:58:46.131 INFO:teuthology.orchestra.run.smithi197.stdout: "dentry_pinned_count": 0, 2022-06-29T15:58:46.132 INFO:teuthology.orchestra.run.smithi197.stdout: "id": 4547, 2022-06-29T15:58:46.132 INFO:teuthology.orchestra.run.smithi197.stdout: "inst": { 2022-06-29T15:58:46.132 INFO:teuthology.orchestra.run.smithi197.stdout: "name": { 2022-06-29T15:58:46.133 INFO:teuthology.orchestra.run.smithi197.stdout: "type": "client", 2022-06-29T15:58:46.133 INFO:teuthology.orchestra.run.smithi197.stdout: "num": 4547 2022-06-29T15:58:46.133 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2022-06-29T15:58:46.134 INFO:teuthology.orchestra.run.smithi197.stdout: "addr": { 2022-06-29T15:58:46.134 INFO:teuthology.orchestra.run.smithi197.stdout: "type": "v1", 2022-06-29T15:58:46.134 INFO:teuthology.orchestra.run.smithi197.stdout: "addr": "172.21.15.197:0", 2022-06-29T15:58:46.135 INFO:teuthology.orchestra.run.smithi197.stdout: "nonce": 3437874647 2022-06-29T15:58:46.135 INFO:teuthology.orchestra.run.smithi197.stdout: } 2022-06-29T15:58:46.135 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2022-06-29T15:58:46.136 INFO:teuthology.orchestra.run.smithi197.stdout: "addr": { 2022-06-29T15:58:46.136 INFO:teuthology.orchestra.run.smithi197.stdout: "type": "v1", 2022-06-29T15:58:46.136 INFO:teuthology.orchestra.run.smithi197.stdout: "addr": "172.21.15.197:0", 2022-06-29T15:58:46.137 INFO:teuthology.orchestra.run.smithi197.stdout: "nonce": 3437874647 2022-06-29T15:58:46.137 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2022-06-29T15:58:46.137 INFO:teuthology.orchestra.run.smithi197.stdout: "inst_str": "client.4547 172.21.15.197:0/3437874647", 2022-06-29T15:58:46.138 INFO:teuthology.orchestra.run.smithi197.stdout: "addr_str": "172.21.15.197:0/3437874647", 2022-06-29T15:58:46.138 INFO:teuthology.orchestra.run.smithi197.stdout: "inode_count": 1, 2022-06-29T15:58:46.138 INFO:teuthology.orchestra.run.smithi197.stdout: "mds_epoch": 4, 2022-06-29T15:58:46.139 INFO:teuthology.orchestra.run.smithi197.stdout: "osd_epoch": 23, 2022-06-29T15:58:46.139 INFO:teuthology.orchestra.run.smithi197.stdout: "osd_epoch_barrier": 0, 2022-06-29T15:58:46.139 INFO:teuthology.orchestra.run.smithi197.stdout: "blacklisted": false, 2022-06-29T15:58:46.140 INFO:teuthology.orchestra.run.smithi197.stdout: "fs_name": "cephfs" 2022-06-29T15:58:46.140 INFO:teuthology.orchestra.run.smithi197.stdout:} 2022-06-29T15:58:46.142 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:46.142 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T15:58:46.143 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T15:58:46.152 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T15:58:46.153 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T15:58:46.153 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:46.154 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-06-29T15:58:46.174 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:46.175 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T15:58:46.175 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T15:58:46.183 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2022-06-29T15:58:46.185 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-06-29T15:58:46.185 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:46.186 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-06-29T15:58:46.211 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:46.211 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T15:58:46.212 DEBUG:teuthology.orchestra.run.smithi197:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T15:58:46.220 INFO:teuthology.orchestra.run.smithi197.stdout:fuseblk 2022-06-29T15:58:46.221 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2022-06-29T15:58:46.222 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:46.222 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.2) 2022-06-29T15:58:46.245 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2022-06-29T15:58:46.272 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T15:58:46.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.381+0000 7efc5f59e700 1 Processor -- start 2022-06-29T15:58:46.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.381+0000 7efc5f59e700 1 -- start start 2022-06-29T15:58:46.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.381+0000 7efc5f59e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efc600b5eb0 0x7efc60142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:46.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.381+0000 7efc5f59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc600b2c50 0x7efc600b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:46.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.381+0000 7efc5f59e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc600b0cf0 con 0x7efc600b48b0 2022-06-29T15:58:46.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.381+0000 7efc5f59e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc600b54e0 con 0x7efc600b5eb0 2022-06-29T15:58:46.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.381+0000 7efc5f59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc600b4d10 con 0x7efc600b2c50 2022-06-29T15:58:46.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.381+0000 7efc5e59c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efc600b5eb0 0x7efc60142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:46.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.381+0000 7efc5e59c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efc600b5eb0 0x7efc60142a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45748/0 (socket says 172.21.15.60:45748) 2022-06-29T15:58:46.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.381+0000 7efc5e59c700 1 -- 172.21.15.60:0/2131735794 learned_addr learned my addr 172.21.15.60:0/2131735794 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:46.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.381+0000 7efc5ed9d700 1 --2- 172.21.15.60:0/2131735794 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc600b2c50 0x7efc600b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:46.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.381+0000 7efc5e59c700 1 -- 172.21.15.60:0/2131735794 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc600b2c50 msgr2=0x7efc600b30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:46.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.381+0000 7efc5e59c700 1 --2- 172.21.15.60:0/2131735794 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc600b2c50 0x7efc600b30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:46.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.381+0000 7efc5e59c700 1 -- 172.21.15.60:0/2131735794 >> v1:172.21.15.92:6789/0 conn(0x7efc600b48b0 legacy=0x7efc600b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:46.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.381+0000 7efc5e59c700 1 -- 172.21.15.60:0/2131735794 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc600b6290 con 0x7efc600b5eb0 2022-06-29T15:58:46.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.381+0000 7efc5e59c700 1 --2- 172.21.15.60:0/2131735794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc600b5eb0 0x7efc60142a20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7efc50002f70 tx=0x7efc50006eb0).ready entity=mon.0 client_cookie=c59b74e22bc2ca83 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:46.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.381+0000 7efc5d59a700 1 -- 172.21.15.60:0/2131735794 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc5000d5c0 con 0x7efc600b5eb0 2022-06-29T15:58:46.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.381+0000 7efc5d59a700 1 -- 172.21.15.60:0/2131735794 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc50000e90 con 0x7efc600b5eb0 2022-06-29T15:58:46.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.381+0000 7efc5d59a700 1 -- 172.21.15.60:0/2131735794 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc5000d950 con 0x7efc600b5eb0 2022-06-29T15:58:46.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.381+0000 7efc5f59e700 1 -- 172.21.15.60:0/2131735794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc600b5eb0 msgr2=0x7efc60142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:46.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.381+0000 7efc5f59e700 1 --2- 172.21.15.60:0/2131735794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc600b5eb0 0x7efc60142a20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7efc50002f70 tx=0x7efc50006eb0).stop 2022-06-29T15:58:46.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.385+0000 7efc5f59e700 1 -- 172.21.15.60:0/2131735794 shutdown_connections 2022-06-29T15:58:46.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.385+0000 7efc5f59e700 1 --2- 172.21.15.60:0/2131735794 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc600b2c50 0x7efc600b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:46.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.385+0000 7efc5f59e700 1 --2- 172.21.15.60:0/2131735794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc600b5eb0 0x7efc60142a20 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:46.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.385+0000 7efc5f59e700 1 -- 172.21.15.60:0/2131735794 >> 172.21.15.60:0/2131735794 conn(0x7efc60062090 msgr2=0x7efc60062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:46.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.385+0000 7efc5f59e700 1 -- 172.21.15.60:0/2131735794 shutdown_connections 2022-06-29T15:58:46.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.385+0000 7efc5f59e700 1 -- 172.21.15.60:0/2131735794 wait complete. 2022-06-29T15:58:46.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.385+0000 7efc5f59e700 1 Processor -- start 2022-06-29T15:58:46.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.385+0000 7efc5f59e700 1 -- start start 2022-06-29T15:58:46.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.385+0000 7efc5f59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc600b2c50 0x7efc600b1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:46.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.385+0000 7efc5f59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc600b48b0 0x7efc601553d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:46.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.385+0000 7efc5f59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc600b5eb0 0x7efc60112a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:46.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.385+0000 7efc5f59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc60145f10 con 0x7efc600b5eb0 2022-06-29T15:58:46.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.385+0000 7efc5f59e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc601125d0 con 0x7efc600b48b0 2022-06-29T15:58:46.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.385+0000 7efc5f59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc60143ee0 con 0x7efc600b2c50 2022-06-29T15:58:46.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.385+0000 7efc5ed9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc600b5eb0 0x7efc60112a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:46.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.385+0000 7efc5e59c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc600b2c50 0x7efc600b1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:46.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.385+0000 7efc5dd9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc600b48b0 0x7efc601553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:46.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.385+0000 7efc5ed9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc600b5eb0 0x7efc60112a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45756/0 (socket says 172.21.15.60:45756) 2022-06-29T15:58:46.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.385+0000 7efc5ed9d700 1 -- 172.21.15.60:0/414977633 learned_addr learned my addr 172.21.15.60:0/414977633 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:46.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.385+0000 7efc5ed9d700 1 -- 172.21.15.60:0/414977633 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc600b2c50 msgr2=0x7efc600b1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:46.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.385+0000 7efc5ed9d700 1 --2- 172.21.15.60:0/414977633 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc600b2c50 0x7efc600b1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:46.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.385+0000 7efc5ed9d700 1 -- 172.21.15.60:0/414977633 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc600b48b0 msgr2=0x7efc601553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:46.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.385+0000 7efc5ed9d700 1 --2- 172.21.15.60:0/414977633 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc600b48b0 0x7efc601553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:46.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.385+0000 7efc5ed9d700 1 -- 172.21.15.60:0/414977633 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc5000b1e0 con 0x7efc600b5eb0 2022-06-29T15:58:46.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.385+0000 7efc5ed9d700 1 --2- 172.21.15.60:0/414977633 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc600b5eb0 0x7efc60112a00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7efc5800a000 tx=0x7efc58008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:46.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.389+0000 7efc577fe700 1 -- 172.21.15.60:0/414977633 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc58011780 con 0x7efc600b5eb0 2022-06-29T15:58:46.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.389+0000 7efc577fe700 1 -- 172.21.15.60:0/414977633 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc580118e0 con 0x7efc600b5eb0 2022-06-29T15:58:46.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.389+0000 7efc577fe700 1 -- 172.21.15.60:0/414977633 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc58011c40 con 0x7efc600b5eb0 2022-06-29T15:58:46.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.389+0000 7efc5f59e700 1 -- 172.21.15.60:0/414977633 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc60144180 con 0x7efc600b5eb0 2022-06-29T15:58:46.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.389+0000 7efc5f59e700 1 -- 172.21.15.60:0/414977633 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc600ae670 con 0x7efc600b5eb0 2022-06-29T15:58:46.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.389+0000 7efc5f59e700 1 -- 172.21.15.60:0/414977633 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc600573d0 con 0x7efc600b5eb0 2022-06-29T15:58:46.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.389+0000 7efc577fe700 1 -- 172.21.15.60:0/414977633 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7efc5801c460 con 0x7efc600b5eb0 2022-06-29T15:58:46.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.389+0000 7efc577fe700 1 --2- 172.21.15.60:0/414977633 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efc44053aa0 0x7efc44055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:46.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.389+0000 7efc577fe700 1 -- 172.21.15.60:0/414977633 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7efc5806c0e0 con 0x7efc600b5eb0 2022-06-29T15:58:46.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.389+0000 7efc5e59c700 1 --2- 172.21.15.60:0/414977633 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efc44053aa0 0x7efc44055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:46.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.393+0000 7efc5e59c700 1 --2- 172.21.15.60:0/414977633 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efc44053aa0 0x7efc44055f60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7efc5001c000 tx=0x7efc50015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:46.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.397+0000 7efc577fe700 1 -- 172.21.15.60:0/414977633 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efc58042e10 con 0x7efc600b5eb0 2022-06-29T15:58:46.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.525+0000 7efc5f59e700 1 -- 172.21.15.60:0/414977633 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc60067a30 con 0x7efc600b5eb0 2022-06-29T15:58:46.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.529+0000 7efc577fe700 1 -- 172.21.15.60:0/414977633 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16038 (secure 0 0 0) 0x7efc58046430 con 0x7efc600b5eb0 2022-06-29T15:58:46.535 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:58:46.535 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":23,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T15:58:26.345644+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-06-29T15:58:22.424395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-06-29T15:58:23.343438+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T15:58:46.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.529+0000 7efc5f59e700 1 -- 172.21.15.60:0/414977633 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efc44053aa0 msgr2=0x7efc44055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:46.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.529+0000 7efc5f59e700 1 --2- 172.21.15.60:0/414977633 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efc44053aa0 0x7efc44055f60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7efc5001c000 tx=0x7efc50015000).stop 2022-06-29T15:58:46.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.529+0000 7efc5f59e700 1 -- 172.21.15.60:0/414977633 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc600b5eb0 msgr2=0x7efc60112a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:46.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.529+0000 7efc5f59e700 1 --2- 172.21.15.60:0/414977633 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc600b5eb0 0x7efc60112a00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7efc5800a000 tx=0x7efc58008040).stop 2022-06-29T15:58:46.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.529+0000 7efc5f59e700 1 -- 172.21.15.60:0/414977633 shutdown_connections 2022-06-29T15:58:46.539 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.529+0000 7efc5f59e700 1 --2- 172.21.15.60:0/414977633 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efc44053aa0 0x7efc44055f60 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:46.539 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.529+0000 7efc5f59e700 1 --2- 172.21.15.60:0/414977633 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc600b5eb0 0x7efc60112a00 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:46.539 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.529+0000 7efc5f59e700 1 --2- 172.21.15.60:0/414977633 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc600b48b0 0x7efc601553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:46.539 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.529+0000 7efc5f59e700 1 --2- 172.21.15.60:0/414977633 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc600b2c50 0x7efc600b1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:46.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.529+0000 7efc5f59e700 1 -- 172.21.15.60:0/414977633 >> 172.21.15.60:0/414977633 conn(0x7efc60062090 msgr2=0x7efc600b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:46.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.529+0000 7efc5f59e700 1 -- 172.21.15.60:0/414977633 shutdown_connections 2022-06-29T15:58:46.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.529+0000 7efc5f59e700 1 -- 172.21.15.60:0/414977633 wait complete. 2022-06-29T15:58:46.548 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-29T15:58:22.424395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-06-29T15:58:23.343438+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-29T15:58:46.549 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T15:58:46.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.657+0000 7f5fd009c700 1 Processor -- start 2022-06-29T15:58:46.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.657+0000 7f5fd009c700 1 -- start start 2022-06-29T15:58:46.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.657+0000 7f5fd009c700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5fc80b9d70 0x7f5fc81465d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:46.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.657+0000 7f5fd009c700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc80b4f50 con 0x7f5fc80b3d30 2022-06-29T15:58:46.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.657+0000 7f5fd009c700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc81461a0 con 0x7f5fc8061d90 2022-06-29T15:58:46.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.657+0000 7f5fd009c700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fc80ba150 con 0x7f5fc80b9d70 2022-06-29T15:58:46.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.657+0000 7f5fcd637700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5fc80b3d30 0x7f5fc8146ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53848/0 (socket says 172.21.15.60:53848) 2022-06-29T15:58:46.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.657+0000 7f5fcd637700 1 -- 172.21.15.60:0/3316320968 learned_addr learned my addr 172.21.15.60:0/3316320968 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:46.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.657+0000 7f5fcde38700 1 --2- 172.21.15.60:0/3316320968 >> v2:172.21.15.92:3300/0 conn(0x7f5fc80b9d70 0x7f5fc81465d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:46.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.657+0000 7f5fcce36700 1 -- 172.21.15.60:0/3316320968 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2636496610 0 0) 0x7f5fc80b4f50 con 0x7f5fc80b3d30 2022-06-29T15:58:46.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.657+0000 7f5fcce36700 1 -- 172.21.15.60:0/3316320968 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fac003260 con 0x7f5fc80b3d30 2022-06-29T15:58:46.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.657+0000 7f5fcde38700 1 -- 172.21.15.60:0/3316320968 >> v1:172.21.15.92:6789/0 conn(0x7f5fc8061d90 legacy=0x7f5fc8142b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:46.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.657+0000 7f5fcde38700 1 -- 172.21.15.60:0/3316320968 >> v1:172.21.15.60:6789/0 conn(0x7f5fc80b3d30 legacy=0x7f5fc8146ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:46.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.657+0000 7f5fcde38700 1 -- 172.21.15.60:0/3316320968 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc80b5180 con 0x7f5fc80b9d70 2022-06-29T15:58:46.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.657+0000 7f5fcde38700 1 --2- 172.21.15.60:0/3316320968 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fc80b9d70 0x7f5fc81465d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f5fc4002f70 tx=0x7f5fc4006df0).ready entity=mon.1 client_cookie=6e43e8bd9fbbf8a0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:46.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.657+0000 7f5fcce36700 1 -- 172.21.15.60:0/3316320968 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fc400d610 con 0x7f5fc80b9d70 2022-06-29T15:58:46.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.657+0000 7f5fcce36700 1 -- 172.21.15.60:0/3316320968 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fc4000e90 con 0x7f5fc80b9d70 2022-06-29T15:58:46.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.657+0000 7f5fcce36700 1 -- 172.21.15.60:0/3316320968 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fc400d960 con 0x7f5fc80b9d70 2022-06-29T15:58:46.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.661+0000 7f5fd009c700 1 -- 172.21.15.60:0/3316320968 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fc80b9d70 msgr2=0x7f5fc81465d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:46.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.661+0000 7f5fd009c700 1 --2- 172.21.15.60:0/3316320968 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fc80b9d70 0x7f5fc81465d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f5fc4002f70 tx=0x7f5fc4006df0).stop 2022-06-29T15:58:46.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.661+0000 7f5fd009c700 1 -- 172.21.15.60:0/3316320968 shutdown_connections 2022-06-29T15:58:46.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.661+0000 7f5fd009c700 1 --2- 172.21.15.60:0/3316320968 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fc80b9d70 0x7f5fc81465d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:46.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.661+0000 7f5fd009c700 1 -- 172.21.15.60:0/3316320968 >> 172.21.15.60:0/3316320968 conn(0x7f5fc8062e70 msgr2=0x7f5fc80b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:46.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.661+0000 7f5fd009c700 1 -- 172.21.15.60:0/3316320968 shutdown_connections 2022-06-29T15:58:46.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.661+0000 7f5fd009c700 1 -- 172.21.15.60:0/3316320968 wait complete. 2022-06-29T15:58:46.674 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.661+0000 7f5fd009c700 1 Processor -- start 2022-06-29T15:58:46.674 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.661+0000 7f5fd009c700 1 -- start start 2022-06-29T15:58:46.674 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.661+0000 7f5fd009c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fc8061d90 0x7f5fc80adf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:46.674 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.661+0000 7f5fd009c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fc80b3d30 0x7f5fc8135350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:46.675 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.661+0000 7f5fd009c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fc80b9d70 0x7f5fc8148000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:46.675 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.661+0000 7f5fd009c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc8147720 con 0x7f5fc80b3d30 2022-06-29T15:58:46.675 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.661+0000 7f5fd009c700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc8147120 con 0x7f5fc80b9d70 2022-06-29T15:58:46.675 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.661+0000 7f5fd009c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc8155200 con 0x7f5fc8061d90 2022-06-29T15:58:46.676 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.665+0000 7f5fcd637700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fc80b3d30 0x7f5fc8135350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:46.676 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.665+0000 7f5fcde38700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fc8061d90 0x7f5fc80adf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:46.676 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.665+0000 7f5fcd637700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fc80b3d30 0x7f5fc8135350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45768/0 (socket says 172.21.15.60:45768) 2022-06-29T15:58:46.676 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.665+0000 7f5fce639700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fc80b9d70 0x7f5fc8148000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:46.677 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.665+0000 7f5fcd637700 1 -- 172.21.15.60:0/2659759911 learned_addr learned my addr 172.21.15.60:0/2659759911 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:46.677 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.665+0000 7f5fcd637700 1 -- 172.21.15.60:0/2659759911 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fc8061d90 msgr2=0x7f5fc80adf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:46.677 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.665+0000 7f5fcd637700 1 --2- 172.21.15.60:0/2659759911 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fc8061d90 0x7f5fc80adf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:46.677 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.665+0000 7f5fcd637700 1 -- 172.21.15.60:0/2659759911 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fc80b9d70 msgr2=0x7f5fc8148000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:46.678 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.665+0000 7f5fcd637700 1 --2- 172.21.15.60:0/2659759911 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fc80b9d70 0x7f5fc8148000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:46.678 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.665+0000 7f5fcd637700 1 -- 172.21.15.60:0/2659759911 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc400b180 con 0x7f5fc80b3d30 2022-06-29T15:58:46.678 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.665+0000 7f5fcd637700 1 --2- 172.21.15.60:0/2659759911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fc80b3d30 0x7f5fc8135350 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f5fb8001110 tx=0x7f5fb8007ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:46.678 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.665+0000 7f5fbeffd700 1 -- 172.21.15.60:0/2659759911 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fb80070d0 con 0x7f5fc80b3d30 2022-06-29T15:58:46.679 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.665+0000 7f5fbeffd700 1 -- 172.21.15.60:0/2659759911 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fb8007230 con 0x7f5fc80b3d30 2022-06-29T15:58:46.679 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.665+0000 7f5fbeffd700 1 -- 172.21.15.60:0/2659759911 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fb8005610 con 0x7f5fc80b3d30 2022-06-29T15:58:46.679 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.665+0000 7f5fd009c700 1 -- 172.21.15.60:0/2659759911 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fc81484f0 con 0x7f5fc80b3d30 2022-06-29T15:58:46.679 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.665+0000 7f5fd009c700 1 -- 172.21.15.60:0/2659759911 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fc8146e30 con 0x7f5fc80b3d30 2022-06-29T15:58:46.680 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.665+0000 7f5fd009c700 1 -- 172.21.15.60:0/2659759911 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fc81472e0 con 0x7f5fc80b3d30 2022-06-29T15:58:46.680 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.669+0000 7f5fbeffd700 1 -- 172.21.15.60:0/2659759911 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5fb8010070 con 0x7f5fc80b3d30 2022-06-29T15:58:46.680 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.669+0000 7f5fbeffd700 1 --2- 172.21.15.60:0/2659759911 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5fb0053a70 0x7f5fb0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:46.680 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.669+0000 7f5fbeffd700 1 -- 172.21.15.60:0/2659759911 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5fb806b5c0 con 0x7f5fc80b3d30 2022-06-29T15:58:46.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.669+0000 7f5fcde38700 1 --2- 172.21.15.60:0/2659759911 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5fb0053a70 0x7f5fb0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:46.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.669+0000 7f5fcde38700 1 --2- 172.21.15.60:0/2659759911 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5fb0053a70 0x7f5fb0055f30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f5fc401a000 tx=0x7f5fc4014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:46.682 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.673+0000 7f5fbeffd700 1 -- 172.21.15.60:0/2659759911 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5fb8042440 con 0x7f5fc80b3d30 2022-06-29T15:58:46.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.805+0000 7f5fd009c700 1 -- 172.21.15.60:0/2659759911 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f5fc81553c0 con 0x7f5fc80b3d30 2022-06-29T15:58:46.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.805+0000 7f5fbeffd700 1 -- 172.21.15.60:0/2659759911 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v23) v1 ==== 110+0+10 (secure 0 0 0) 0x7f5fb8045a60 con 0x7f5fc80b3d30 2022-06-29T15:58:46.810 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T15:58:46.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.805+0000 7f5fd009c700 1 -- 172.21.15.60:0/2659759911 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5fb0053a70 msgr2=0x7f5fb0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:46.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.805+0000 7f5fd009c700 1 --2- 172.21.15.60:0/2659759911 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5fb0053a70 0x7f5fb0055f30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f5fc401a000 tx=0x7f5fc4014000).stop 2022-06-29T15:58:46.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.805+0000 7f5fd009c700 1 -- 172.21.15.60:0/2659759911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fc80b3d30 msgr2=0x7f5fc8135350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:46.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.805+0000 7f5fd009c700 1 --2- 172.21.15.60:0/2659759911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fc80b3d30 0x7f5fc8135350 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f5fb8001110 tx=0x7f5fb8007ad0).stop 2022-06-29T15:58:46.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.805+0000 7f5fd009c700 1 -- 172.21.15.60:0/2659759911 shutdown_connections 2022-06-29T15:58:46.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.805+0000 7f5fd009c700 1 --2- 172.21.15.60:0/2659759911 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5fb0053a70 0x7f5fb0055f30 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:46.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.805+0000 7f5fd009c700 1 --2- 172.21.15.60:0/2659759911 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fc80b9d70 0x7f5fc8148000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:46.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.805+0000 7f5fd009c700 1 --2- 172.21.15.60:0/2659759911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fc80b3d30 0x7f5fc8135350 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:46.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.805+0000 7f5fd009c700 1 --2- 172.21.15.60:0/2659759911 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fc8061d90 0x7f5fc80adf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.805+0000 7f5fd009c700 1 -- 172.21.15.60:0/2659759911 >> 172.21.15.60:0/2659759911 conn(0x7f5fc8062e70 msgr2=0x7f5fc8142940 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.805+0000 7f5fd009c700 1 -- 172.21.15.60:0/2659759911 shutdown_connections 2022-06-29T15:58:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.805+0000 7f5fd009c700 1 -- 172.21.15.60:0/2659759911 wait complete. 2022-06-29T15:58:46.823 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T15:58:46.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.921+0000 7fe56a635700 1 Processor -- start 2022-06-29T15:58:46.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.921+0000 7fe56a635700 1 -- start start 2022-06-29T15:58:46.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.921+0000 7fe56a635700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe5640ae7c0 0x7fe564141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:46.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.921+0000 7fe56a635700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe5640b4010 0x7fe564142490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:46.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.921+0000 7fe56a635700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe564143a40 con 0x7fe564146160 2022-06-29T15:58:46.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.921+0000 7fe56a635700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5640b56d0 con 0x7fe5640ae7c0 2022-06-29T15:58:46.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.921+0000 7fe56a635700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5640aee60 con 0x7fe5640b4010 2022-06-29T15:58:46.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.921+0000 7fe563fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe5640ae7c0 0x7fe564141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:46.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.921+0000 7fe55b7fe700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe5640b4010 0x7fe564142490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:46.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.921+0000 7fe563fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe5640ae7c0 0x7fe564141fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45774/0 (socket says 172.21.15.60:45774) 2022-06-29T15:58:46.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.921+0000 7fe568bd2700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fe564146160 0x7fe564146590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:58112/0 (socket says 172.21.15.60:58112) 2022-06-29T15:58:46.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.921+0000 7fe563fff700 1 -- 172.21.15.60:0/2826030285 learned_addr learned my addr 172.21.15.60:0/2826030285 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:46.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.925+0000 7fe55b7fe700 1 -- 172.21.15.60:0/2826030285 >> v2:172.21.15.60:3300/0 conn(0x7fe5640ae7c0 msgr2=0x7fe564141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:46.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.925+0000 7fe55b7fe700 1 --2- 172.21.15.60:0/2826030285 >> v2:172.21.15.60:3300/0 conn(0x7fe5640ae7c0 0x7fe564141fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:46.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.925+0000 7fe55b7fe700 1 -- 172.21.15.60:0/2826030285 >> v1:172.21.15.92:6789/0 conn(0x7fe564146160 legacy=0x7fe564146590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:46.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.925+0000 7fe55b7fe700 1 -- 172.21.15.60:0/2826030285 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5640af100 con 0x7fe5640b4010 2022-06-29T15:58:46.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.925+0000 7fe5637fe700 1 -- 172.21.15.60:0/2826030285 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4108370361 0 0) 0x7fe564143a40 con 0x7fe564146160 2022-06-29T15:58:46.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.925+0000 7fe563fff700 1 --2- 172.21.15.60:0/2826030285 >> v2:172.21.15.60:3300/0 conn(0x7fe5640ae7c0 0x7fe564141fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:46.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.925+0000 7fe55b7fe700 1 --2- 172.21.15.60:0/2826030285 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5640b4010 0x7fe564142490 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fe54c002f70 tx=0x7fe54c006df0).ready entity=mon.1 client_cookie=9c9438bf1be4d203 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:46.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.925+0000 7fe5637fe700 1 -- 172.21.15.60:0/2826030285 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe54c00d570 con 0x7fe5640b4010 2022-06-29T15:58:46.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.925+0000 7fe5637fe700 1 -- 172.21.15.60:0/2826030285 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe54c000e90 con 0x7fe5640b4010 2022-06-29T15:58:46.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.925+0000 7fe5637fe700 1 -- 172.21.15.60:0/2826030285 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe54c00d8c0 con 0x7fe5640b4010 2022-06-29T15:58:46.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.925+0000 7fe56a635700 1 -- 172.21.15.60:0/2826030285 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5640b4010 msgr2=0x7fe564142490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:46.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.925+0000 7fe56a635700 1 --2- 172.21.15.60:0/2826030285 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5640b4010 0x7fe564142490 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fe54c002f70 tx=0x7fe54c006df0).stop 2022-06-29T15:58:46.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.925+0000 7fe56a635700 1 -- 172.21.15.60:0/2826030285 shutdown_connections 2022-06-29T15:58:46.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.925+0000 7fe56a635700 1 --2- 172.21.15.60:0/2826030285 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5640b4010 0x7fe564142490 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:46.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.925+0000 7fe56a635700 1 --2- 172.21.15.60:0/2826030285 >> v2:172.21.15.60:3300/0 conn(0x7fe5640ae7c0 0x7fe564141fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:46.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.925+0000 7fe56a635700 1 -- 172.21.15.60:0/2826030285 >> 172.21.15.60:0/2826030285 conn(0x7fe5640b9340 msgr2=0x7fe5640b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:46.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.925+0000 7fe56a635700 1 -- 172.21.15.60:0/2826030285 shutdown_connections 2022-06-29T15:58:46.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.925+0000 7fe56a635700 1 -- 172.21.15.60:0/2826030285 wait complete. 2022-06-29T15:58:46.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.929+0000 7fe56a635700 1 Processor -- start 2022-06-29T15:58:46.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.929+0000 7fe56a635700 1 -- start start 2022-06-29T15:58:46.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.929+0000 7fe56a635700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe5640ae7c0 0x7fe5641556d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:46.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.929+0000 7fe56a635700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5640b4010 0x7fe564140110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:46.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.929+0000 7fe56a635700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe564146160 0x7fe56413ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:46.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.929+0000 7fe56a635700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5641565f0 con 0x7fe5640b4010 2022-06-29T15:58:46.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.929+0000 7fe56a635700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5640b03f0 con 0x7fe564146160 2022-06-29T15:58:46.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.929+0000 7fe56a635700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5640b1750 con 0x7fe5640ae7c0 2022-06-29T15:58:46.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.929+0000 7fe55b7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5640b4010 0x7fe564140110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:46.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.929+0000 7fe55b7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5640b4010 0x7fe564140110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45782/0 (socket says 172.21.15.60:45782) 2022-06-29T15:58:46.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.929+0000 7fe55b7fe700 1 -- 172.21.15.60:0/1114835935 learned_addr learned my addr 172.21.15.60:0/1114835935 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:46.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.929+0000 7fe563fff700 1 --2- 172.21.15.60:0/1114835935 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe5640ae7c0 0x7fe5641556d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:46.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.929+0000 7fe568bd2700 1 --2- 172.21.15.60:0/1114835935 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe564146160 0x7fe56413ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:46.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.933+0000 7fe55b7fe700 1 -- 172.21.15.60:0/1114835935 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe5640ae7c0 msgr2=0x7fe5641556d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:46.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.933+0000 7fe55b7fe700 1 --2- 172.21.15.60:0/1114835935 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe5640ae7c0 0x7fe5641556d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:46.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.933+0000 7fe55b7fe700 1 -- 172.21.15.60:0/1114835935 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe564146160 msgr2=0x7fe56413ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:46.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.933+0000 7fe55b7fe700 1 --2- 172.21.15.60:0/1114835935 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe564146160 0x7fe56413ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:46.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.933+0000 7fe55b7fe700 1 -- 172.21.15.60:0/1114835935 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe55400b1e0 con 0x7fe5640b4010 2022-06-29T15:58:46.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.933+0000 7fe55b7fe700 1 --2- 172.21.15.60:0/1114835935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5640b4010 0x7fe564140110 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fe54c002f70 tx=0x7fe54c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:46.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.933+0000 7fe561ffb700 1 -- 172.21.15.60:0/1114835935 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe54c00dea0 con 0x7fe5640b4010 2022-06-29T15:58:46.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.933+0000 7fe561ffb700 1 -- 172.21.15.60:0/1114835935 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe54c013e90 con 0x7fe5640b4010 2022-06-29T15:58:46.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.933+0000 7fe56a635700 1 -- 172.21.15.60:0/1114835935 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe54c00b180 con 0x7fe5640b4010 2022-06-29T15:58:46.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.933+0000 7fe561ffb700 1 -- 172.21.15.60:0/1114835935 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe54c01e5f0 con 0x7fe5640b4010 2022-06-29T15:58:46.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.933+0000 7fe56a635700 1 -- 172.21.15.60:0/1114835935 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5640b08c0 con 0x7fe5640b4010 2022-06-29T15:58:46.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.933+0000 7fe56a635700 1 -- 172.21.15.60:0/1114835935 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5640573d0 con 0x7fe5640b4010 2022-06-29T15:58:46.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.933+0000 7fe561ffb700 1 -- 172.21.15.60:0/1114835935 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe54c025030 con 0x7fe5640b4010 2022-06-29T15:58:46.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.933+0000 7fe561ffb700 1 --2- 172.21.15.60:0/1114835935 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe548053b00 0x7fe548055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:46.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.933+0000 7fe561ffb700 1 -- 172.21.15.60:0/1114835935 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe54c06cb60 con 0x7fe5640b4010 2022-06-29T15:58:46.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.933+0000 7fe563fff700 1 --2- 172.21.15.60:0/1114835935 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe548053b00 0x7fe548055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:46.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.937+0000 7fe563fff700 1 --2- 172.21.15.60:0/1114835935 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe548053b00 0x7fe548055fc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fe55400b000 tx=0x7fe554006eb0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:46.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:46.941+0000 7fe561ffb700 1 -- 172.21.15.60:0/1114835935 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe54c043a80 con 0x7fe5640b4010 2022-06-29T15:58:47.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.069+0000 7fe56a635700 1 -- 172.21.15.60:0/1114835935 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fe564141a00 con 0x7fe5640b4010 2022-06-29T15:58:47.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.069+0000 7fe561ffb700 1 -- 172.21.15.60:0/1114835935 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe54c023070 con 0x7fe5640b4010 2022-06-29T15:58:47.077 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T15:58:47.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.073+0000 7fe56a635700 1 -- 172.21.15.60:0/1114835935 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe548053b00 msgr2=0x7fe548055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:47.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.073+0000 7fe56a635700 1 --2- 172.21.15.60:0/1114835935 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe548053b00 0x7fe548055fc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fe55400b000 tx=0x7fe554006eb0).stop 2022-06-29T15:58:47.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.073+0000 7fe56a635700 1 -- 172.21.15.60:0/1114835935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5640b4010 msgr2=0x7fe564140110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:47.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.073+0000 7fe56a635700 1 --2- 172.21.15.60:0/1114835935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5640b4010 0x7fe564140110 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fe54c002f70 tx=0x7fe54c014000).stop 2022-06-29T15:58:47.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.073+0000 7fe56a635700 1 -- 172.21.15.60:0/1114835935 shutdown_connections 2022-06-29T15:58:47.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.073+0000 7fe56a635700 1 --2- 172.21.15.60:0/1114835935 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe548053b00 0x7fe548055fc0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:47.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.073+0000 7fe56a635700 1 --2- 172.21.15.60:0/1114835935 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe564146160 0x7fe56413ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:47.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.073+0000 7fe56a635700 1 --2- 172.21.15.60:0/1114835935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5640b4010 0x7fe564140110 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:47.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.073+0000 7fe56a635700 1 --2- 172.21.15.60:0/1114835935 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe5640ae7c0 0x7fe5641556d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:47.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.073+0000 7fe56a635700 1 -- 172.21.15.60:0/1114835935 >> 172.21.15.60:0/1114835935 conn(0x7fe5640b9340 msgr2=0x7fe564141110 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:47.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.073+0000 7fe56a635700 1 -- 172.21.15.60:0/1114835935 shutdown_connections 2022-06-29T15:58:47.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.073+0000 7fe56a635700 1 -- 172.21.15.60:0/1114835935 wait complete. 2022-06-29T15:58:47.090 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-29T15:58:47.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.193+0000 7f3ef759e700 1 Processor -- start 2022-06-29T15:58:47.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.193+0000 7f3ef759e700 1 -- start start 2022-06-29T15:58:47.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.193+0000 7f3ef759e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3ef805fb00 0x7f3ef813f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:47.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.193+0000 7f3ef759e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ef8067a90 0x7f3ef813fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:47.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.193+0000 7f3ef759e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ef8064e20 con 0x7f3ef80aec30 2022-06-29T15:58:47.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.193+0000 7f3ef759e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ef80b10e0 con 0x7f3ef805fb00 2022-06-29T15:58:47.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.193+0000 7f3ef759e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef805fee0 con 0x7f3ef8067a90 2022-06-29T15:58:47.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.193+0000 7f3ef659c700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3ef805fb00 0x7f3ef813f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:47.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.193+0000 7f3ef5d9b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ef8067a90 0x7f3ef813fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:47.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.193+0000 7f3ef6d9d700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f3ef80aec30 0x7f3ef80b2030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:58124/0 (socket says 172.21.15.60:58124) 2022-06-29T15:58:47.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.193+0000 7f3ef659c700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3ef805fb00 0x7f3ef813f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:51818/0 (socket says 172.21.15.60:51818) 2022-06-29T15:58:47.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.193+0000 7f3ef659c700 1 -- 172.21.15.60:0/4196880293 learned_addr learned my addr 172.21.15.60:0/4196880293 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:47.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.197+0000 7f3ef5d9b700 1 -- 172.21.15.60:0/4196880293 >> v2:172.21.15.92:3300/0 conn(0x7f3ef805fb00 msgr2=0x7f3ef813f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:47.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.197+0000 7f3ef5d9b700 1 --2- 172.21.15.60:0/4196880293 >> v2:172.21.15.92:3300/0 conn(0x7f3ef805fb00 0x7f3ef813f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:47.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.197+0000 7f3ef5d9b700 1 -- 172.21.15.60:0/4196880293 >> v1:172.21.15.92:6789/0 conn(0x7f3ef80aec30 legacy=0x7f3ef80b2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:47.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.197+0000 7f3ef5d9b700 1 -- 172.21.15.60:0/4196880293 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ef8060b20 con 0x7f3ef8067a90 2022-06-29T15:58:47.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.197+0000 7f3ef5d9b700 1 --2- 172.21.15.60:0/4196880293 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ef8067a90 0x7f3ef813fe90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f3ee4002f70 tx=0x7f3ee4006df0).ready entity=mon.2 client_cookie=3f14e35d6c635588 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:47.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.197+0000 7f3ef559a700 1 -- 172.21.15.60:0/4196880293 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ee400d570 con 0x7f3ef8067a90 2022-06-29T15:58:47.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.197+0000 7f3ef559a700 1 -- 172.21.15.60:0/4196880293 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ee4000e90 con 0x7f3ef8067a90 2022-06-29T15:58:47.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.197+0000 7f3ef559a700 1 -- 172.21.15.60:0/4196880293 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ee400d8c0 con 0x7f3ef8067a90 2022-06-29T15:58:47.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.197+0000 7f3ef759e700 1 -- 172.21.15.60:0/4196880293 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ef8067a90 msgr2=0x7f3ef813fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:47.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.197+0000 7f3ef759e700 1 --2- 172.21.15.60:0/4196880293 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ef8067a90 0x7f3ef813fe90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f3ee4002f70 tx=0x7f3ee4006df0).stop 2022-06-29T15:58:47.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.197+0000 7f3ef759e700 1 -- 172.21.15.60:0/4196880293 shutdown_connections 2022-06-29T15:58:47.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.197+0000 7f3ef759e700 1 --2- 172.21.15.60:0/4196880293 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ef8067a90 0x7f3ef813fe90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:47.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.197+0000 7f3ef759e700 1 --2- 172.21.15.60:0/4196880293 >> v2:172.21.15.92:3300/0 conn(0x7f3ef805fb00 0x7f3ef813f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:47.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.197+0000 7f3ef759e700 1 -- 172.21.15.60:0/4196880293 >> 172.21.15.60:0/4196880293 conn(0x7f3ef8063a60 msgr2=0x7f3ef805f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:47.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.197+0000 7f3ef759e700 1 -- 172.21.15.60:0/4196880293 shutdown_connections 2022-06-29T15:58:47.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.197+0000 7f3ef759e700 1 -- 172.21.15.60:0/4196880293 wait complete. 2022-06-29T15:58:47.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.201+0000 7f3ef759e700 1 Processor -- start 2022-06-29T15:58:47.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.201+0000 7f3ef759e700 1 -- start start 2022-06-29T15:58:47.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.201+0000 7f3ef759e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ef805fb00 0x7f3ef80671e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:47.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.201+0000 7f3ef759e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ef8067a90 0x7f3ef8148fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:47.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.201+0000 7f3ef759e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ef80aec30 0x7f3ef8144700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:47.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.201+0000 7f3ef759e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef8066a20 con 0x7f3ef8067a90 2022-06-29T15:58:47.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.201+0000 7f3ef759e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef80664e0 con 0x7f3ef805fb00 2022-06-29T15:58:47.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.201+0000 7f3ef759e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef8066780 con 0x7f3ef80aec30 2022-06-29T15:58:47.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.201+0000 7f3ef5d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ef8067a90 0x7f3ef8148fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:47.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.201+0000 7f3ef659c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ef805fb00 0x7f3ef80671e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:47.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.201+0000 7f3ef659c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ef805fb00 0x7f3ef80671e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:51824/0 (socket says 172.21.15.60:51824) 2022-06-29T15:58:47.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.201+0000 7f3ef659c700 1 -- 172.21.15.60:0/3639108420 learned_addr learned my addr 172.21.15.60:0/3639108420 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:47.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.201+0000 7f3ef6d9d700 1 --2- 172.21.15.60:0/3639108420 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ef80aec30 0x7f3ef8144700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:47.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.201+0000 7f3ef5d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ef8067a90 0x7f3ef8148fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45796/0 (socket says 172.21.15.60:45796) 2022-06-29T15:58:47.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.201+0000 7f3ef659c700 1 -- 172.21.15.60:0/3639108420 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ef80aec30 msgr2=0x7f3ef8144700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:47.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.201+0000 7f3ef659c700 1 --2- 172.21.15.60:0/3639108420 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ef80aec30 0x7f3ef8144700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:47.230 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.201+0000 7f3ef659c700 1 -- 172.21.15.60:0/3639108420 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ef8067a90 msgr2=0x7f3ef8148fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:47.230 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.201+0000 7f3ef659c700 1 --2- 172.21.15.60:0/3639108420 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ef8067a90 0x7f3ef8148fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:47.230 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.201+0000 7f3ef659c700 1 -- 172.21.15.60:0/3639108420 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee400b180 con 0x7f3ef805fb00 2022-06-29T15:58:47.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.201+0000 7f3ef6d9d700 1 --2- 172.21.15.60:0/3639108420 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ef80aec30 0x7f3ef8144700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:47.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.205+0000 7f3ef659c700 1 --2- 172.21.15.60:0/3639108420 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ef805fb00 0x7f3ef80671e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f3ee800c000 tx=0x7f3ee8008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:47.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.205+0000 7f3eef7fe700 1 -- 172.21.15.60:0/3639108420 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ee8013610 con 0x7f3ef805fb00 2022-06-29T15:58:47.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.205+0000 7f3eef7fe700 1 -- 172.21.15.60:0/3639108420 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ee8013770 con 0x7f3ef805fb00 2022-06-29T15:58:47.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.205+0000 7f3ef759e700 1 -- 172.21.15.60:0/3639108420 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ef8144bf0 con 0x7f3ef805fb00 2022-06-29T15:58:47.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.205+0000 7f3eef7fe700 1 -- 172.21.15.60:0/3639108420 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ee8013ad0 con 0x7f3ef805fb00 2022-06-29T15:58:47.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.205+0000 7f3ef759e700 1 -- 172.21.15.60:0/3639108420 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ef8130150 con 0x7f3ef805fb00 2022-06-29T15:58:47.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.205+0000 7f3ef759e700 1 -- 172.21.15.60:0/3639108420 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ef81444a0 con 0x7f3ef805fb00 2022-06-29T15:58:47.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.205+0000 7f3eef7fe700 1 -- 172.21.15.60:0/3639108420 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3ee8023070 con 0x7f3ef805fb00 2022-06-29T15:58:47.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.209+0000 7f3eef7fe700 1 --2- 172.21.15.60:0/3639108420 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3ed4053a70 0x7f3ed4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:47.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.209+0000 7f3eef7fe700 1 -- 172.21.15.60:0/3639108420 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3ee806c090 con 0x7f3ef805fb00 2022-06-29T15:58:47.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.209+0000 7f3ef5d9b700 1 --2- 172.21.15.60:0/3639108420 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3ed4053a70 0x7f3ed4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:47.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.209+0000 7f3ef5d9b700 1 --2- 172.21.15.60:0/3639108420 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3ed4053a70 0x7f3ed4055f30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f3ee401a000 tx=0x7f3ee4014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:47.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.217+0000 7f3eef7fe700 1 -- 172.21.15.60:0/3639108420 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3ee8042fb0 con 0x7f3ef805fb00 2022-06-29T15:58:47.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.345+0000 7f3ef759e700 1 -- 172.21.15.60:0/3639108420 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3ef80573d0 con 0x7f3ef805fb00 2022-06-29T15:58:47.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.345+0000 7f3eef7fe700 1 -- 172.21.15.60:0/3639108420 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7f3ee80465d0 con 0x7f3ef805fb00 2022-06-29T15:58:47.351 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T15:58:47.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.349+0000 7f3ef759e700 1 -- 172.21.15.60:0/3639108420 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3ed4053a70 msgr2=0x7f3ed4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:47.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.349+0000 7f3ef759e700 1 --2- 172.21.15.60:0/3639108420 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3ed4053a70 0x7f3ed4055f30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f3ee401a000 tx=0x7f3ee4014000).stop 2022-06-29T15:58:47.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.349+0000 7f3ef759e700 1 -- 172.21.15.60:0/3639108420 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ef805fb00 msgr2=0x7f3ef80671e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:47.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.349+0000 7f3ef759e700 1 --2- 172.21.15.60:0/3639108420 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ef805fb00 0x7f3ef80671e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f3ee800c000 tx=0x7f3ee8008040).stop 2022-06-29T15:58:47.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.349+0000 7f3ef759e700 1 -- 172.21.15.60:0/3639108420 shutdown_connections 2022-06-29T15:58:47.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.349+0000 7f3ef759e700 1 --2- 172.21.15.60:0/3639108420 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3ed4053a70 0x7f3ed4055f30 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:47.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.349+0000 7f3ef759e700 1 --2- 172.21.15.60:0/3639108420 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ef80aec30 0x7f3ef8144700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:47.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.349+0000 7f3ef759e700 1 --2- 172.21.15.60:0/3639108420 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ef8067a90 0x7f3ef8148fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:47.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.349+0000 7f3ef759e700 1 --2- 172.21.15.60:0/3639108420 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ef805fb00 0x7f3ef80671e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:47.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.349+0000 7f3ef759e700 1 -- 172.21.15.60:0/3639108420 >> 172.21.15.60:0/3639108420 conn(0x7f3ef8063a60 msgr2=0x7f3ef80b1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:47.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.349+0000 7f3ef759e700 1 -- 172.21.15.60:0/3639108420 shutdown_connections 2022-06-29T15:58:47.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.349+0000 7f3ef759e700 1 -- 172.21.15.60:0/3639108420 wait complete. 2022-06-29T15:58:47.370 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-29T15:58:47.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.473+0000 7fdba5e41700 1 Processor -- start 2022-06-29T15:58:47.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.473+0000 7fdba5e41700 1 -- start start 2022-06-29T15:58:47.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.473+0000 7fdba5e41700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdba00b4010 0x7fdba01426b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:47.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.473+0000 7fdba5e41700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdba00b46c0 0x7fdba00b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:47.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.473+0000 7fdba5e41700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba0143a40 con 0x7fdba00ae7c0 2022-06-29T15:58:47.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.473+0000 7fdba5e41700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdba00aee60 con 0x7fdba00b46c0 2022-06-29T15:58:47.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.473+0000 7fdba5e41700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdba00af100 con 0x7fdba00b4010 2022-06-29T15:58:47.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.473+0000 7fdb9ffff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdba00b46c0 0x7fdba00b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:47.486 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.473+0000 7fdb9f7fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fdba00ae7c0 0x7fdba0141fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53888/0 (socket says 172.21.15.60:53888) 2022-06-29T15:58:47.486 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.473+0000 7fdb9ffff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdba00b46c0 0x7fdba00b4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45804/0 (socket says 172.21.15.60:45804) 2022-06-29T15:58:47.487 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.473+0000 7fdb9f7fe700 1 -- 172.21.15.60:0/952265395 learned_addr learned my addr 172.21.15.60:0/952265395 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:47.487 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.473+0000 7fdb9effd700 1 --2- 172.21.15.60:0/952265395 >> v2:172.21.15.92:3300/0 conn(0x7fdba00b4010 0x7fdba01426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:47.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.477+0000 7fdb9ffff700 1 -- 172.21.15.60:0/952265395 >> v2:172.21.15.92:3300/0 conn(0x7fdba00b4010 msgr2=0x7fdba01426b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:47.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.477+0000 7fdb9ffff700 1 --2- 172.21.15.60:0/952265395 >> v2:172.21.15.92:3300/0 conn(0x7fdba00b4010 0x7fdba01426b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:47.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.477+0000 7fdb9ffff700 1 -- 172.21.15.60:0/952265395 >> v1:172.21.15.60:6789/0 conn(0x7fdba00ae7c0 legacy=0x7fdba0141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:47.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.477+0000 7fdb9e7fc700 1 -- 172.21.15.60:0/952265395 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3316829046 0 0) 0x7fdba0143a40 con 0x7fdba00ae7c0 2022-06-29T15:58:47.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.477+0000 7fdb9ffff700 1 -- 172.21.15.60:0/952265395 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba00aeba0 con 0x7fdba00b46c0 2022-06-29T15:58:47.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.477+0000 7fdb9ffff700 1 --2- 172.21.15.60:0/952265395 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdba00b46c0 0x7fdba00b4af0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fdb980032f0 tx=0x7fdb980091d0).ready entity=mon.0 client_cookie=6fc75eb455197894 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:47.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.477+0000 7fdb9e7fc700 1 -- 172.21.15.60:0/952265395 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb9800b680 con 0x7fdba00b46c0 2022-06-29T15:58:47.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.477+0000 7fdb9e7fc700 1 -- 172.21.15.60:0/952265395 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb9800b7e0 con 0x7fdba00b46c0 2022-06-29T15:58:47.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.477+0000 7fdb9e7fc700 1 -- 172.21.15.60:0/952265395 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb9800b950 con 0x7fdba00b46c0 2022-06-29T15:58:47.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.477+0000 7fdba5e41700 1 -- 172.21.15.60:0/952265395 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdba00b46c0 msgr2=0x7fdba00b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:47.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.477+0000 7fdba5e41700 1 --2- 172.21.15.60:0/952265395 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdba00b46c0 0x7fdba00b4af0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fdb980032f0 tx=0x7fdb980091d0).stop 2022-06-29T15:58:47.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.477+0000 7fdba5e41700 1 -- 172.21.15.60:0/952265395 shutdown_connections 2022-06-29T15:58:47.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.477+0000 7fdba5e41700 1 --2- 172.21.15.60:0/952265395 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdba00b46c0 0x7fdba00b4af0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:47.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.477+0000 7fdba5e41700 1 --2- 172.21.15.60:0/952265395 >> v2:172.21.15.92:3300/0 conn(0x7fdba00b4010 0x7fdba01426b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:47.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.477+0000 7fdba5e41700 1 -- 172.21.15.60:0/952265395 >> 172.21.15.60:0/952265395 conn(0x7fdba00b9340 msgr2=0x7fdba00b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:47.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.477+0000 7fdba5e41700 1 -- 172.21.15.60:0/952265395 shutdown_connections 2022-06-29T15:58:47.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.477+0000 7fdba5e41700 1 -- 172.21.15.60:0/952265395 wait complete. 2022-06-29T15:58:47.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.477+0000 7fdba5e41700 1 Processor -- start 2022-06-29T15:58:47.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.477+0000 7fdba5e41700 1 -- start start 2022-06-29T15:58:47.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.481+0000 7fdba5e41700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdba00ae7c0 0x7fdba00b81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:47.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.481+0000 7fdba5e41700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdba00b4010 0x7fdba0155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:47.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.481+0000 7fdba5e41700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdba00b46c0 0x7fdba0113d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:47.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.481+0000 7fdba5e41700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba013e590 con 0x7fdba00b4010 2022-06-29T15:58:47.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.481+0000 7fdba5e41700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba01128b0 con 0x7fdba00b46c0 2022-06-29T15:58:47.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.481+0000 7fdba5e41700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba0112b50 con 0x7fdba00ae7c0 2022-06-29T15:58:47.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.481+0000 7fdb9effd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdba00b4010 0x7fdba0155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:47.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.481+0000 7fdb9effd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdba00b4010 0x7fdba0155bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45810/0 (socket says 172.21.15.60:45810) 2022-06-29T15:58:47.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.481+0000 7fdb9effd700 1 -- 172.21.15.60:0/3113176402 learned_addr learned my addr 172.21.15.60:0/3113176402 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:47.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.481+0000 7fdb9f7fe700 1 --2- 172.21.15.60:0/3113176402 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdba00ae7c0 0x7fdba00b81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:47.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.481+0000 7fdb9ffff700 1 --2- 172.21.15.60:0/3113176402 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdba00b46c0 0x7fdba0113d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:47.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.481+0000 7fdb9effd700 1 -- 172.21.15.60:0/3113176402 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdba00ae7c0 msgr2=0x7fdba00b81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:47.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.481+0000 7fdb9effd700 1 --2- 172.21.15.60:0/3113176402 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdba00ae7c0 0x7fdba00b81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:47.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.481+0000 7fdb9effd700 1 -- 172.21.15.60:0/3113176402 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdba00b46c0 msgr2=0x7fdba0113d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:47.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.481+0000 7fdb9effd700 1 --2- 172.21.15.60:0/3113176402 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdba00b46c0 0x7fdba0113d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:47.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.481+0000 7fdb9effd700 1 -- 172.21.15.60:0/3113176402 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb98009470 con 0x7fdba00b4010 2022-06-29T15:58:47.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.481+0000 7fdb9effd700 1 --2- 172.21.15.60:0/3113176402 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdba00b4010 0x7fdba0155bd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fdb9000c000 tx=0x7fdb90006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:47.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.481+0000 7fdb9cff9700 1 -- 172.21.15.60:0/3113176402 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb90010780 con 0x7fdba00b4010 2022-06-29T15:58:47.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.481+0000 7fdb9cff9700 1 -- 172.21.15.60:0/3113176402 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb900108e0 con 0x7fdba00b4010 2022-06-29T15:58:47.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.481+0000 7fdb9cff9700 1 -- 172.21.15.60:0/3113176402 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb90010c40 con 0x7fdba00b4010 2022-06-29T15:58:47.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.481+0000 7fdba5e41700 1 -- 172.21.15.60:0/3113176402 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdba0112df0 con 0x7fdba00b4010 2022-06-29T15:58:47.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.481+0000 7fdba5e41700 1 -- 172.21.15.60:0/3113176402 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdba0155420 con 0x7fdba00b4010 2022-06-29T15:58:47.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.485+0000 7fdba5e41700 1 -- 172.21.15.60:0/3113176402 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdba00573d0 con 0x7fdba00b4010 2022-06-29T15:58:47.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.485+0000 7fdb9cff9700 1 -- 172.21.15.60:0/3113176402 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdb9001c460 con 0x7fdba00b4010 2022-06-29T15:58:47.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.485+0000 7fdb9cff9700 1 --2- 172.21.15.60:0/3113176402 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdb840539e0 0x7fdb84055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:47.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.485+0000 7fdb9cff9700 1 -- 172.21.15.60:0/3113176402 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fdb9006c2b0 con 0x7fdba00b4010 2022-06-29T15:58:47.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.485+0000 7fdb9f7fe700 1 --2- 172.21.15.60:0/3113176402 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdb840539e0 0x7fdb84055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:47.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.485+0000 7fdb9f7fe700 1 --2- 172.21.15.60:0/3113176402 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdb840539e0 0x7fdb84055ea0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fdb8800d000 tx=0x7fdb880011a0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:47.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.489+0000 7fdb9cff9700 1 -- 172.21.15.60:0/3113176402 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdb90043170 con 0x7fdba00b4010 2022-06-29T15:58:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.609+0000 7fdba5e41700 1 -- 172.21.15.60:0/3113176402 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdba005fdb0 con 0x7fdba00b4010 2022-06-29T15:58:47.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.613+0000 7fdb9cff9700 1 -- 172.21.15.60:0/3113176402 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7fdb90046790 con 0x7fdba00b4010 2022-06-29T15:58:47.618 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T15:58:47.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.613+0000 7fdba5e41700 1 -- 172.21.15.60:0/3113176402 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdb840539e0 msgr2=0x7fdb84055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:47.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.613+0000 7fdba5e41700 1 --2- 172.21.15.60:0/3113176402 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdb840539e0 0x7fdb84055ea0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fdb8800d000 tx=0x7fdb880011a0).stop 2022-06-29T15:58:47.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.613+0000 7fdba5e41700 1 -- 172.21.15.60:0/3113176402 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdba00b4010 msgr2=0x7fdba0155bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:47.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.613+0000 7fdba5e41700 1 --2- 172.21.15.60:0/3113176402 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdba00b4010 0x7fdba0155bd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fdb9000c000 tx=0x7fdb90006df0).stop 2022-06-29T15:58:47.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.613+0000 7fdba5e41700 1 -- 172.21.15.60:0/3113176402 shutdown_connections 2022-06-29T15:58:47.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.613+0000 7fdba5e41700 1 --2- 172.21.15.60:0/3113176402 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdb840539e0 0x7fdb84055ea0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:47.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.613+0000 7fdba5e41700 1 --2- 172.21.15.60:0/3113176402 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdba00b46c0 0x7fdba0113d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:47.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.613+0000 7fdba5e41700 1 --2- 172.21.15.60:0/3113176402 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdba00b4010 0x7fdba0155bd0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:47.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.613+0000 7fdba5e41700 1 --2- 172.21.15.60:0/3113176402 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdba00ae7c0 0x7fdba00b81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:47.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.613+0000 7fdba5e41700 1 -- 172.21.15.60:0/3113176402 >> 172.21.15.60:0/3113176402 conn(0x7fdba00b9340 msgr2=0x7fdba00b6660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:47.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.613+0000 7fdba5e41700 1 -- 172.21.15.60:0/3113176402 shutdown_connections 2022-06-29T15:58:47.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.613+0000 7fdba5e41700 1 -- 172.21.15.60:0/3113176402 wait complete. 2022-06-29T15:58:47.631 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T15:58:47.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.733+0000 7ff99cb5e700 1 Processor -- start 2022-06-29T15:58:47.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.733+0000 7ff99cb5e700 1 -- start start 2022-06-29T15:58:47.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.733+0000 7ff99cb5e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9980b6a50 0x7ff99805fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:47.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.733+0000 7ff99cb5e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff998146cf0 0x7ff9980aaf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:47.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.733+0000 7ff99cb5e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9980b6e30 con 0x7ff998062230 2022-06-29T15:58:47.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.733+0000 7ff99cb5e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff998098970 con 0x7ff998146cf0 2022-06-29T15:58:47.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.733+0000 7ff99cb5e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff99805fa40 con 0x7ff9980b6a50 2022-06-29T15:58:47.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.733+0000 7ff996d9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff998146cf0 0x7ff9980aaf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:47.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.733+0000 7ff99659c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9980b6a50 0x7ff99805fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:47.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.733+0000 7ff996d9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff998146cf0 0x7ff9980aaf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45820/0 (socket says 172.21.15.60:45820) 2022-06-29T15:58:47.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.733+0000 7ff995d9b700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7ff998062230 0x7ff9981465e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:58152/0 (socket says 172.21.15.60:58152) 2022-06-29T15:58:47.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.733+0000 7ff995d9b700 1 -- 172.21.15.60:0/970558206 learned_addr learned my addr 172.21.15.60:0/970558206 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:47.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.733+0000 7ff996d9d700 1 -- 172.21.15.60:0/970558206 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9980b6a50 msgr2=0x7ff99805fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:47.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.733+0000 7ff996d9d700 1 --2- 172.21.15.60:0/970558206 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9980b6a50 0x7ff99805fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:47.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.733+0000 7ff996d9d700 1 -- 172.21.15.60:0/970558206 >> v1:172.21.15.92:6789/0 conn(0x7ff998062230 legacy=0x7ff9981465e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:47.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.733+0000 7ff99559a700 1 -- 172.21.15.60:0/970558206 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3931635171 0 0) 0x7ff9980b6e30 con 0x7ff998062230 2022-06-29T15:58:47.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.733+0000 7ff996d9d700 1 -- 172.21.15.60:0/970558206 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff99805f0b0 con 0x7ff998146cf0 2022-06-29T15:58:47.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.733+0000 7ff996d9d700 1 --2- 172.21.15.60:0/970558206 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff998146cf0 0x7ff9980aaf60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7ff988002f70 tx=0x7ff988006df0).ready entity=mon.0 client_cookie=b87f3da6f15711b6 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:47.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.733+0000 7ff99559a700 1 -- 172.21.15.60:0/970558206 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff98800d570 con 0x7ff998146cf0 2022-06-29T15:58:47.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.733+0000 7ff99559a700 1 -- 172.21.15.60:0/970558206 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff988000e90 con 0x7ff998146cf0 2022-06-29T15:58:47.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.733+0000 7ff99559a700 1 -- 172.21.15.60:0/970558206 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff98800d8c0 con 0x7ff998146cf0 2022-06-29T15:58:47.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.733+0000 7ff99cb5e700 1 -- 172.21.15.60:0/970558206 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff998146cf0 msgr2=0x7ff9980aaf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:47.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.733+0000 7ff99cb5e700 1 --2- 172.21.15.60:0/970558206 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff998146cf0 0x7ff9980aaf60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7ff988002f70 tx=0x7ff988006df0).stop 2022-06-29T15:58:47.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.733+0000 7ff99cb5e700 1 -- 172.21.15.60:0/970558206 shutdown_connections 2022-06-29T15:58:47.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.733+0000 7ff99cb5e700 1 --2- 172.21.15.60:0/970558206 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff998146cf0 0x7ff9980aaf60 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:47.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.733+0000 7ff99cb5e700 1 --2- 172.21.15.60:0/970558206 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9980b6a50 0x7ff99805fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:47.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.733+0000 7ff99cb5e700 1 -- 172.21.15.60:0/970558206 >> 172.21.15.60:0/970558206 conn(0x7ff9980643e0 msgr2=0x7ff9980b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:47.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.733+0000 7ff99cb5e700 1 -- 172.21.15.60:0/970558206 shutdown_connections 2022-06-29T15:58:47.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.733+0000 7ff99cb5e700 1 -- 172.21.15.60:0/970558206 wait complete. 2022-06-29T15:58:47.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.737+0000 7ff99cb5e700 1 Processor -- start 2022-06-29T15:58:47.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.737+0000 7ff99cb5e700 1 -- start start 2022-06-29T15:58:47.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.737+0000 7ff99cb5e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff998062230 0x7ff99814ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:47.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.737+0000 7ff99cb5e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9980b6a50 0x7ff99809b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:47.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.737+0000 7ff99cb5e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff998146cf0 0x7ff998065de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:47.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.737+0000 7ff99cb5e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99814a4b0 con 0x7ff998146cf0 2022-06-29T15:58:47.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.737+0000 7ff99cb5e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99814a750 con 0x7ff9980b6a50 2022-06-29T15:58:47.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.737+0000 7ff99cb5e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff99814a9f0 con 0x7ff998062230 2022-06-29T15:58:47.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.737+0000 7ff996d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff998146cf0 0x7ff998065de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:47.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.737+0000 7ff99659c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff998062230 0x7ff99814ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:47.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.737+0000 7ff996d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff998146cf0 0x7ff998065de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45826/0 (socket says 172.21.15.60:45826) 2022-06-29T15:58:47.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.737+0000 7ff996d9d700 1 -- 172.21.15.60:0/805712084 learned_addr learned my addr 172.21.15.60:0/805712084 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:47.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.737+0000 7ff995d9b700 1 --2- 172.21.15.60:0/805712084 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9980b6a50 0x7ff99809b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:47.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.737+0000 7ff996d9d700 1 -- 172.21.15.60:0/805712084 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff998062230 msgr2=0x7ff99814ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:47.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.737+0000 7ff996d9d700 1 --2- 172.21.15.60:0/805712084 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff998062230 0x7ff99814ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:47.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.737+0000 7ff996d9d700 1 -- 172.21.15.60:0/805712084 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9980b6a50 msgr2=0x7ff99809b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:47.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.737+0000 7ff996d9d700 1 --2- 172.21.15.60:0/805712084 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9980b6a50 0x7ff99809b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:47.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.737+0000 7ff996d9d700 1 -- 172.21.15.60:0/805712084 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff98800b180 con 0x7ff998146cf0 2022-06-29T15:58:47.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.737+0000 7ff995d9b700 1 --2- 172.21.15.60:0/805712084 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9980b6a50 0x7ff99809b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:58:47.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.737+0000 7ff996d9d700 1 --2- 172.21.15.60:0/805712084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff998146cf0 0x7ff998065de0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7ff988002f70 tx=0x7ff988014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:47.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.737+0000 7ff9877fe700 1 -- 172.21.15.60:0/805712084 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff988006960 con 0x7ff998146cf0 2022-06-29T15:58:47.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.737+0000 7ff9877fe700 1 -- 172.21.15.60:0/805712084 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98800de40 con 0x7ff998146cf0 2022-06-29T15:58:47.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.741+0000 7ff99cb5e700 1 -- 172.21.15.60:0/805712084 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff998148ed0 con 0x7ff998146cf0 2022-06-29T15:58:47.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.741+0000 7ff9877fe700 1 -- 172.21.15.60:0/805712084 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff98801e5f0 con 0x7ff998146cf0 2022-06-29T15:58:47.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.741+0000 7ff99cb5e700 1 -- 172.21.15.60:0/805712084 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9980664f0 con 0x7ff998146cf0 2022-06-29T15:58:47.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.741+0000 7ff99cb5e700 1 -- 172.21.15.60:0/805712084 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff99814abb0 con 0x7ff998146cf0 2022-06-29T15:58:47.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.741+0000 7ff9877fe700 1 -- 172.21.15.60:0/805712084 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff988025030 con 0x7ff998146cf0 2022-06-29T15:58:47.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.741+0000 7ff9877fe700 1 --2- 172.21.15.60:0/805712084 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff974053b30 0x7ff974055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:47.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.741+0000 7ff9877fe700 1 -- 172.21.15.60:0/805712084 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff98806ccc0 con 0x7ff998146cf0 2022-06-29T15:58:47.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.741+0000 7ff99659c700 1 --2- 172.21.15.60:0/805712084 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff974053b30 0x7ff974055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:47.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.745+0000 7ff99659c700 1 --2- 172.21.15.60:0/805712084 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff974053b30 0x7ff974055ff0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7ff98c00b000 tx=0x7ff98c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:47.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.749+0000 7ff9877fe700 1 -- 172.21.15.60:0/805712084 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff988073080 con 0x7ff998146cf0 2022-06-29T15:58:47.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.881+0000 7ff99cb5e700 1 -- 172.21.15.60:0/805712084 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9980573d0 con 0x7ff998146cf0 2022-06-29T15:58:47.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.885+0000 7ff9877fe700 1 -- 172.21.15.60:0/805712084 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16038 (secure 0 0 0) 0x7ff988023070 con 0x7ff998146cf0 2022-06-29T15:58:47.890 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:58:47.890 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":23,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T15:58:26.345644+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-06-29T15:58:22.424395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-06-29T15:58:23.343438+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T15:58:47.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.885+0000 7ff99cb5e700 1 -- 172.21.15.60:0/805712084 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff974053b30 msgr2=0x7ff974055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:47.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.885+0000 7ff99cb5e700 1 --2- 172.21.15.60:0/805712084 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff974053b30 0x7ff974055ff0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7ff98c00b000 tx=0x7ff98c006df0).stop 2022-06-29T15:58:47.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.885+0000 7ff99cb5e700 1 -- 172.21.15.60:0/805712084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff998146cf0 msgr2=0x7ff998065de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:47.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.885+0000 7ff99cb5e700 1 --2- 172.21.15.60:0/805712084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff998146cf0 0x7ff998065de0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7ff988002f70 tx=0x7ff988014000).stop 2022-06-29T15:58:47.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.885+0000 7ff99cb5e700 1 -- 172.21.15.60:0/805712084 shutdown_connections 2022-06-29T15:58:47.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.885+0000 7ff99cb5e700 1 --2- 172.21.15.60:0/805712084 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff974053b30 0x7ff974055ff0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:47.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.885+0000 7ff99cb5e700 1 --2- 172.21.15.60:0/805712084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff998146cf0 0x7ff998065de0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:47.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.885+0000 7ff99cb5e700 1 --2- 172.21.15.60:0/805712084 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9980b6a50 0x7ff99809b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:47.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.885+0000 7ff99cb5e700 1 --2- 172.21.15.60:0/805712084 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff998062230 0x7ff99814ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:47.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.885+0000 7ff99cb5e700 1 -- 172.21.15.60:0/805712084 >> 172.21.15.60:0/805712084 conn(0x7ff9980643e0 msgr2=0x7ff9980ab4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:47.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.885+0000 7ff99cb5e700 1 -- 172.21.15.60:0/805712084 shutdown_connections 2022-06-29T15:58:47.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:47.885+0000 7ff99cb5e700 1 -- 172.21.15.60:0/805712084 wait complete. 2022-06-29T15:58:47.903 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-29T15:58:22.424395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-06-29T15:58:23.343438+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-29T15:58:47.904 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T15:58:48.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.001+0000 7f8168a20700 1 Processor -- start 2022-06-29T15:58:48.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.001+0000 7f8168a20700 1 -- start start 2022-06-29T15:58:48.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.005+0000 7f8168a20700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f816405f2c0 0x7f816405f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:48.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.005+0000 7f8168a20700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f816405c3f0 0x7f816405c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:48.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.005+0000 7f8168a20700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f816405bd20 con 0x7f816405ff90 2022-06-29T15:58:48.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.005+0000 7f8168a20700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81641438e0 con 0x7f816405f2c0 2022-06-29T15:58:48.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.005+0000 7f8168a20700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f816405bf50 con 0x7f816405c3f0 2022-06-29T15:58:48.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.005+0000 7f8162d9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f816405f2c0 0x7f816405f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:48.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.005+0000 7f816259c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f816405ff90 0x7f81640aede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53918/0 (socket says 172.21.15.60:53918) 2022-06-29T15:58:48.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.005+0000 7f816259c700 1 -- 172.21.15.60:0/3280959942 learned_addr learned my addr 172.21.15.60:0/3280959942 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:48.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.005+0000 7f816359e700 1 --2- 172.21.15.60:0/3280959942 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f816405c3f0 0x7f816405c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:48.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.005+0000 7f8162d9d700 1 -- 172.21.15.60:0/3280959942 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f816405c3f0 msgr2=0x7f816405c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:48.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.005+0000 7f8162d9d700 1 --2- 172.21.15.60:0/3280959942 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f816405c3f0 0x7f816405c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.005+0000 7f8162d9d700 1 -- 172.21.15.60:0/3280959942 >> v1:172.21.15.60:6789/0 conn(0x7f816405ff90 legacy=0x7f81640aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:48.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.005+0000 7f8162d9d700 1 -- 172.21.15.60:0/3280959942 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8164143560 con 0x7f816405f2c0 2022-06-29T15:58:48.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.005+0000 7f8162d9d700 1 --2- 172.21.15.60:0/3280959942 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f816405f2c0 0x7f816405f6a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f815c0032f0 tx=0x7f815c0091d0).ready entity=mon.0 client_cookie=f38fa1e0e75f7f0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:48.016 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.005+0000 7f8161d9b700 1 -- 172.21.15.60:0/3280959942 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f815c009ea0 con 0x7f816405f2c0 2022-06-29T15:58:48.016 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.005+0000 7f8161d9b700 1 -- 172.21.15.60:0/3280959942 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f815c00b8a0 con 0x7f816405f2c0 2022-06-29T15:58:48.016 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.005+0000 7f8161d9b700 1 -- 172.21.15.60:0/3280959942 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f815c00ba10 con 0x7f816405f2c0 2022-06-29T15:58:48.016 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.005+0000 7f8168a20700 1 -- 172.21.15.60:0/3280959942 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f816405f2c0 msgr2=0x7f816405f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:48.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.005+0000 7f8168a20700 1 --2- 172.21.15.60:0/3280959942 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f816405f2c0 0x7f816405f6a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f815c0032f0 tx=0x7f815c0091d0).stop 2022-06-29T15:58:48.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.005+0000 7f8168a20700 1 -- 172.21.15.60:0/3280959942 shutdown_connections 2022-06-29T15:58:48.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.005+0000 7f8168a20700 1 --2- 172.21.15.60:0/3280959942 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f816405c3f0 0x7f816405c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.005+0000 7f8168a20700 1 --2- 172.21.15.60:0/3280959942 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f816405f2c0 0x7f816405f6a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.005+0000 7f8168a20700 1 -- 172.21.15.60:0/3280959942 >> 172.21.15.60:0/3280959942 conn(0x7f81640b91e0 msgr2=0x7f81640b95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:48.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.005+0000 7f8168a20700 1 -- 172.21.15.60:0/3280959942 shutdown_connections 2022-06-29T15:58:48.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.005+0000 7f8168a20700 1 -- 172.21.15.60:0/3280959942 wait complete. 2022-06-29T15:58:48.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.009+0000 7f8168a20700 1 Processor -- start 2022-06-29T15:58:48.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.009+0000 7f8168a20700 1 -- start start 2022-06-29T15:58:48.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.009+0000 7f8168a20700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f816405c3f0 0x7f8164155ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:48.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.009+0000 7f8168a20700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f816405f2c0 0x7f8164145600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:48.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.009+0000 7f8168a20700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f816405ff90 0x7f8164145af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:48.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.009+0000 7f8168a20700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81640b0b60 con 0x7f816405c3f0 2022-06-29T15:58:48.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.009+0000 7f8168a20700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81640b1450 con 0x7f816405f2c0 2022-06-29T15:58:48.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.009+0000 7f8168a20700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81640b1610 con 0x7f816405ff90 2022-06-29T15:58:48.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.009+0000 7f8162d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f816405c3f0 0x7f8164155ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:48.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.009+0000 7f8162d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f816405c3f0 0x7f8164155ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45836/0 (socket says 172.21.15.60:45836) 2022-06-29T15:58:48.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.009+0000 7f8162d9d700 1 -- 172.21.15.60:0/1754690775 learned_addr learned my addr 172.21.15.60:0/1754690775 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:48.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.009+0000 7f816259c700 1 --2- 172.21.15.60:0/1754690775 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f816405f2c0 0x7f8164145600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:48.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.009+0000 7f816359e700 1 --2- 172.21.15.60:0/1754690775 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f816405ff90 0x7f8164145af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:48.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.009+0000 7f8162d9d700 1 -- 172.21.15.60:0/1754690775 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f816405ff90 msgr2=0x7f8164145af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:48.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.009+0000 7f8162d9d700 1 --2- 172.21.15.60:0/1754690775 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f816405ff90 0x7f8164145af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.009+0000 7f8162d9d700 1 -- 172.21.15.60:0/1754690775 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f816405f2c0 msgr2=0x7f8164145600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:48.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.009+0000 7f8162d9d700 1 --2- 172.21.15.60:0/1754690775 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f816405f2c0 0x7f8164145600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.009+0000 7f8162d9d700 1 -- 172.21.15.60:0/1754690775 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f815c009470 con 0x7f816405c3f0 2022-06-29T15:58:48.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.009+0000 7f8162d9d700 1 --2- 172.21.15.60:0/1754690775 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f816405c3f0 0x7f8164155ac0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f81640b0d20 tx=0x7f815c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:48.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.009+0000 7f8153fff700 1 -- 172.21.15.60:0/1754690775 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f815c011c00 con 0x7f816405c3f0 2022-06-29T15:58:48.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.009+0000 7f8153fff700 1 -- 172.21.15.60:0/1754690775 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f815c011d60 con 0x7f816405c3f0 2022-06-29T15:58:48.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.009+0000 7f8168a20700 1 -- 172.21.15.60:0/1754690775 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f816409f1b0 con 0x7f816405c3f0 2022-06-29T15:58:48.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.009+0000 7f8153fff700 1 -- 172.21.15.60:0/1754690775 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f815c01f5e0 con 0x7f816405c3f0 2022-06-29T15:58:48.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.009+0000 7f8168a20700 1 -- 172.21.15.60:0/1754690775 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81640b07c0 con 0x7f816405c3f0 2022-06-29T15:58:48.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.009+0000 7f8168a20700 1 -- 172.21.15.60:0/1754690775 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81640573d0 con 0x7f816405c3f0 2022-06-29T15:58:48.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.013+0000 7f8153fff700 1 -- 172.21.15.60:0/1754690775 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f815c026030 con 0x7f816405c3f0 2022-06-29T15:58:48.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.013+0000 7f8153fff700 1 --2- 172.21.15.60:0/1754690775 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8140053b00 0x7f8140055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:48.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.013+0000 7f8153fff700 1 -- 172.21.15.60:0/1754690775 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f815c06cbd0 con 0x7f816405c3f0 2022-06-29T15:58:48.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.013+0000 7f816259c700 1 --2- 172.21.15.60:0/1754690775 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8140053b00 0x7f8140055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:48.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.013+0000 7f816259c700 1 --2- 172.21.15.60:0/1754690775 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8140053b00 0x7f8140055fc0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f815800d000 tx=0x7f8158007be0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:48.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.017+0000 7f8153fff700 1 -- 172.21.15.60:0/1754690775 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f815c0439e0 con 0x7f816405c3f0 2022-06-29T15:58:48.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.141+0000 7f8168a20700 1 -- 172.21.15.60:0/1754690775 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f81640b8bc0 con 0x7f816405c3f0 2022-06-29T15:58:48.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.141+0000 7f8153fff700 1 -- 172.21.15.60:0/1754690775 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v23) v1 ==== 110+0+10 (secure 0 0 0) 0x7f815c0240b0 con 0x7f816405c3f0 2022-06-29T15:58:48.149 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T15:58:48.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.145+0000 7f8168a20700 1 -- 172.21.15.60:0/1754690775 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8140053b00 msgr2=0x7f8140055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:48.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.145+0000 7f8168a20700 1 --2- 172.21.15.60:0/1754690775 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8140053b00 0x7f8140055fc0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f815800d000 tx=0x7f8158007be0).stop 2022-06-29T15:58:48.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.145+0000 7f8168a20700 1 -- 172.21.15.60:0/1754690775 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f816405c3f0 msgr2=0x7f8164155ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:48.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.145+0000 7f8168a20700 1 --2- 172.21.15.60:0/1754690775 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f816405c3f0 0x7f8164155ac0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f81640b0d20 tx=0x7f815c016000).stop 2022-06-29T15:58:48.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.145+0000 7f8168a20700 1 -- 172.21.15.60:0/1754690775 shutdown_connections 2022-06-29T15:58:48.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.145+0000 7f8168a20700 1 --2- 172.21.15.60:0/1754690775 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8140053b00 0x7f8140055fc0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.145+0000 7f8168a20700 1 --2- 172.21.15.60:0/1754690775 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f816405ff90 0x7f8164145af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.145+0000 7f8168a20700 1 --2- 172.21.15.60:0/1754690775 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f816405f2c0 0x7f8164145600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.145+0000 7f8168a20700 1 --2- 172.21.15.60:0/1754690775 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f816405c3f0 0x7f8164155ac0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.145+0000 7f8168a20700 1 -- 172.21.15.60:0/1754690775 >> 172.21.15.60:0/1754690775 conn(0x7f81640b91e0 msgr2=0x7f81640af520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:48.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.145+0000 7f8168a20700 1 -- 172.21.15.60:0/1754690775 shutdown_connections 2022-06-29T15:58:48.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.145+0000 7f8168a20700 1 -- 172.21.15.60:0/1754690775 wait complete. 2022-06-29T15:58:48.161 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T15:58:48.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.265+0000 7fd3f9a6e700 1 Processor -- start 2022-06-29T15:58:48.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.265+0000 7fd3f9a6e700 1 -- start start 2022-06-29T15:58:48.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.265+0000 7fd3f9a6e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd3f40b5eb0 0x7fd3f4142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:48.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.265+0000 7fd3f9a6e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3f40b2c50 0x7fd3f40b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:48.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.265+0000 7fd3f9a6e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f40b0cf0 con 0x7fd3f40b48b0 2022-06-29T15:58:48.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.265+0000 7fd3f9a6e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3f40b54e0 con 0x7fd3f40b5eb0 2022-06-29T15:58:48.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.265+0000 7fd3f9a6e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f40b4d10 con 0x7fd3f40b2c50 2022-06-29T15:58:48.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.265+0000 7fd3f8a6c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd3f40b5eb0 0x7fd3f4142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:48.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.265+0000 7fd3f8a6c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd3f40b5eb0 0x7fd3f4142a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45844/0 (socket says 172.21.15.60:45844) 2022-06-29T15:58:48.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.265+0000 7fd3f8a6c700 1 -- 172.21.15.60:0/1551715351 learned_addr learned my addr 172.21.15.60:0/1551715351 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:48.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.265+0000 7fd3f926d700 1 --2- 172.21.15.60:0/1551715351 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3f40b2c50 0x7fd3f40b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:48.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.265+0000 7fd3f37fe700 1 -- 172.21.15.60:0/1551715351 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3976777275 0 0) 0x7fd3e0001110 con 0x7fd3f40b48b0 2022-06-29T15:58:48.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.265+0000 7fd3f8a6c700 1 -- 172.21.15.60:0/1551715351 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3f40b2c50 msgr2=0x7fd3f40b30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:48.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.265+0000 7fd3f8a6c700 1 --2- 172.21.15.60:0/1551715351 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3f40b2c50 0x7fd3f40b30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.265+0000 7fd3f8a6c700 1 -- 172.21.15.60:0/1551715351 >> v1:172.21.15.92:6789/0 conn(0x7fd3f40b48b0 legacy=0x7fd3f40b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:48.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.265+0000 7fd3f8a6c700 1 -- 172.21.15.60:0/1551715351 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f40b6290 con 0x7fd3f40b5eb0 2022-06-29T15:58:48.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.265+0000 7fd3f926d700 1 --2- 172.21.15.60:0/1551715351 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3f40b2c50 0x7fd3f40b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:48.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.269+0000 7fd3f8a6c700 1 --2- 172.21.15.60:0/1551715351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3f40b5eb0 0x7fd3f4142a20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fd3e8002f70 tx=0x7fd3e8006eb0).ready entity=mon.0 client_cookie=f3f9c03a53934cce server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:48.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.269+0000 7fd3f37fe700 1 -- 172.21.15.60:0/1551715351 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd3e800d570 con 0x7fd3f40b5eb0 2022-06-29T15:58:48.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.269+0000 7fd3f37fe700 1 -- 172.21.15.60:0/1551715351 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3e8000e90 con 0x7fd3f40b5eb0 2022-06-29T15:58:48.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.269+0000 7fd3f37fe700 1 -- 172.21.15.60:0/1551715351 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd3e800d8c0 con 0x7fd3f40b5eb0 2022-06-29T15:58:48.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.269+0000 7fd3f9a6e700 1 -- 172.21.15.60:0/1551715351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3f40b5eb0 msgr2=0x7fd3f4142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:48.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.269+0000 7fd3f9a6e700 1 --2- 172.21.15.60:0/1551715351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3f40b5eb0 0x7fd3f4142a20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fd3e8002f70 tx=0x7fd3e8006eb0).stop 2022-06-29T15:58:48.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.269+0000 7fd3f9a6e700 1 -- 172.21.15.60:0/1551715351 shutdown_connections 2022-06-29T15:58:48.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.269+0000 7fd3f9a6e700 1 --2- 172.21.15.60:0/1551715351 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3f40b2c50 0x7fd3f40b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.269+0000 7fd3f9a6e700 1 --2- 172.21.15.60:0/1551715351 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3f40b5eb0 0x7fd3f4142a20 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.269+0000 7fd3f9a6e700 1 -- 172.21.15.60:0/1551715351 >> 172.21.15.60:0/1551715351 conn(0x7fd3f4062090 msgr2=0x7fd3f4062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:48.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.269+0000 7fd3f9a6e700 1 -- 172.21.15.60:0/1551715351 shutdown_connections 2022-06-29T15:58:48.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.269+0000 7fd3f9a6e700 1 -- 172.21.15.60:0/1551715351 wait complete. 2022-06-29T15:58:48.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.269+0000 7fd3f9a6e700 1 Processor -- start 2022-06-29T15:58:48.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.269+0000 7fd3f9a6e700 1 -- start start 2022-06-29T15:58:48.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.273+0000 7fd3f9a6e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3f40b2c50 0x7fd3f40b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:48.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.273+0000 7fd3f9a6e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3f40b48b0 0x7fd3f4146880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:48.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.273+0000 7fd3f9a6e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3f40b5eb0 0x7fd3f413e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:48.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.273+0000 7fd3f9a6e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f4146450 con 0x7fd3f40b5eb0 2022-06-29T15:58:48.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.273+0000 7fd3f9a6e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f4146d70 con 0x7fd3f40b48b0 2022-06-29T15:58:48.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.273+0000 7fd3f9a6e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f4140780 con 0x7fd3f40b2c50 2022-06-29T15:58:48.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.273+0000 7fd3f926d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3f40b5eb0 0x7fd3f413e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:48.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.273+0000 7fd3f926d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3f40b5eb0 0x7fd3f413e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45854/0 (socket says 172.21.15.60:45854) 2022-06-29T15:58:48.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.273+0000 7fd3f926d700 1 -- 172.21.15.60:0/3034940380 learned_addr learned my addr 172.21.15.60:0/3034940380 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:48.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.273+0000 7fd3f8a6c700 1 --2- 172.21.15.60:0/3034940380 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3f40b2c50 0x7fd3f40b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:48.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.273+0000 7fd3f3fff700 1 --2- 172.21.15.60:0/3034940380 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3f40b48b0 0x7fd3f4146880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:48.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.273+0000 7fd3f926d700 1 -- 172.21.15.60:0/3034940380 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3f40b2c50 msgr2=0x7fd3f40b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:48.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.273+0000 7fd3f926d700 1 --2- 172.21.15.60:0/3034940380 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3f40b2c50 0x7fd3f40b19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.273+0000 7fd3f926d700 1 -- 172.21.15.60:0/3034940380 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3f40b48b0 msgr2=0x7fd3f4146880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:48.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.273+0000 7fd3f926d700 1 --2- 172.21.15.60:0/3034940380 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3f40b48b0 0x7fd3f4146880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.273+0000 7fd3f926d700 1 -- 172.21.15.60:0/3034940380 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ec009470 con 0x7fd3f40b5eb0 2022-06-29T15:58:48.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.273+0000 7fd3f926d700 1 --2- 172.21.15.60:0/3034940380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3f40b5eb0 0x7fd3f413e200 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fd3ec00c000 tx=0x7fd3ec0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:48.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.273+0000 7fd3f1ffb700 1 -- 172.21.15.60:0/3034940380 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd3ec00c850 con 0x7fd3f40b5eb0 2022-06-29T15:58:48.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.273+0000 7fd3f1ffb700 1 -- 172.21.15.60:0/3034940380 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3ec00c9b0 con 0x7fd3f40b5eb0 2022-06-29T15:58:48.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.273+0000 7fd3f9a6e700 1 -- 172.21.15.60:0/3034940380 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3e800b1e0 con 0x7fd3f40b5eb0 2022-06-29T15:58:48.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.273+0000 7fd3f1ffb700 1 -- 172.21.15.60:0/3034940380 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd3ec00cd10 con 0x7fd3f40b5eb0 2022-06-29T15:58:48.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.273+0000 7fd3f9a6e700 1 -- 172.21.15.60:0/3034940380 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3f41482f0 con 0x7fd3f40b5eb0 2022-06-29T15:58:48.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.273+0000 7fd3f9a6e700 1 -- 172.21.15.60:0/3034940380 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3f41405e0 con 0x7fd3f40b5eb0 2022-06-29T15:58:48.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.277+0000 7fd3f1ffb700 1 -- 172.21.15.60:0/3034940380 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd3ec022070 con 0x7fd3f40b5eb0 2022-06-29T15:58:48.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.277+0000 7fd3f1ffb700 1 --2- 172.21.15.60:0/3034940380 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd3d4053a70 0x7fd3d4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:48.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.277+0000 7fd3f1ffb700 1 -- 172.21.15.60:0/3034940380 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd3ec06c680 con 0x7fd3f40b5eb0 2022-06-29T15:58:48.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.277+0000 7fd3f8a6c700 1 --2- 172.21.15.60:0/3034940380 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd3d4053a70 0x7fd3d4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:48.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.277+0000 7fd3f8a6c700 1 --2- 172.21.15.60:0/3034940380 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd3d4053a70 0x7fd3d4055f30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fd3e801c000 tx=0x7fd3e8015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:48.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.281+0000 7fd3f1ffb700 1 -- 172.21.15.60:0/3034940380 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd3f41405e0 con 0x7fd3f40b5eb0 2022-06-29T15:58:48.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.409+0000 7fd3f9a6e700 1 -- 172.21.15.60:0/3034940380 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd3f41405e0 con 0x7fd3f40b5eb0 2022-06-29T15:58:48.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.409+0000 7fd3f1ffb700 1 -- 172.21.15.60:0/3034940380 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd3ec016020 con 0x7fd3f40b5eb0 2022-06-29T15:58:48.423 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T15:58:48.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.413+0000 7fd3f9a6e700 1 -- 172.21.15.60:0/3034940380 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd3d4053a70 msgr2=0x7fd3d4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:48.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.413+0000 7fd3f9a6e700 1 --2- 172.21.15.60:0/3034940380 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd3d4053a70 0x7fd3d4055f30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fd3e801c000 tx=0x7fd3e8015000).stop 2022-06-29T15:58:48.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.413+0000 7fd3f9a6e700 1 -- 172.21.15.60:0/3034940380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3f40b5eb0 msgr2=0x7fd3f413e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:48.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.413+0000 7fd3f9a6e700 1 --2- 172.21.15.60:0/3034940380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3f40b5eb0 0x7fd3f413e200 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fd3ec00c000 tx=0x7fd3ec0091d0).stop 2022-06-29T15:58:48.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.413+0000 7fd3f9a6e700 1 -- 172.21.15.60:0/3034940380 shutdown_connections 2022-06-29T15:58:48.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.413+0000 7fd3f9a6e700 1 --2- 172.21.15.60:0/3034940380 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd3d4053a70 0x7fd3d4055f30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.413+0000 7fd3f9a6e700 1 --2- 172.21.15.60:0/3034940380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3f40b5eb0 0x7fd3f413e200 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.413+0000 7fd3f9a6e700 1 --2- 172.21.15.60:0/3034940380 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3f40b48b0 0x7fd3f4146880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.413+0000 7fd3f9a6e700 1 --2- 172.21.15.60:0/3034940380 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3f40b2c50 0x7fd3f40b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.413+0000 7fd3f9a6e700 1 -- 172.21.15.60:0/3034940380 >> 172.21.15.60:0/3034940380 conn(0x7fd3f4062090 msgr2=0x7fd3f413f650 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:48.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.413+0000 7fd3f9a6e700 1 -- 172.21.15.60:0/3034940380 shutdown_connections 2022-06-29T15:58:48.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.413+0000 7fd3f9a6e700 1 -- 172.21.15.60:0/3034940380 wait complete. 2022-06-29T15:58:48.433 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-29T15:58:48.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.537+0000 7f4398f34700 1 Processor -- start 2022-06-29T15:58:48.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.537+0000 7f4398f34700 1 -- start start 2022-06-29T15:58:48.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.537+0000 7f4398f34700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f43940b50a0 0x7f43940b4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:48.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.537+0000 7f4398f34700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f439405c730 con 0x7f43940b5eb0 2022-06-29T15:58:48.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.537+0000 7f4398f34700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43940b6290 con 0x7f43940b4960 2022-06-29T15:58:48.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.537+0000 7f4398f34700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43940b54e0 con 0x7f43940b50a0 2022-06-29T15:58:48.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.537+0000 7f4393fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f43940b50a0 0x7f43940b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:48.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.537+0000 7f43937fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f43940b5eb0 0x7f43940b0280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53944/0 (socket says 172.21.15.60:53944) 2022-06-29T15:58:48.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.537+0000 7f43937fe700 1 -- 172.21.15.60:0/2288930894 learned_addr learned my addr 172.21.15.60:0/2288930894 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:48.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.537+0000 7f43927fc700 1 -- 172.21.15.60:0/2288930894 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3499229728 0 0) 0x7f43940b6290 con 0x7f43940b4960 2022-06-29T15:58:48.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.537+0000 7f4393fff700 1 -- 172.21.15.60:0/2288930894 >> v1:172.21.15.92:6789/0 conn(0x7f43940b4960 legacy=0x7f4394146250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:48.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.537+0000 7f4393fff700 1 -- 172.21.15.60:0/2288930894 >> v1:172.21.15.60:6789/0 conn(0x7f43940b5eb0 legacy=0x7f43940b0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:48.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.537+0000 7f4393fff700 1 -- 172.21.15.60:0/2288930894 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f439405c960 con 0x7f43940b50a0 2022-06-29T15:58:48.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.537+0000 7f4393fff700 1 --2- 172.21.15.60:0/2288930894 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43940b50a0 0x7f43940b4340 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f4388001110 tx=0x7f438c0091d0).ready entity=mon.0 client_cookie=5424e9b036423b6e server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:48.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.537+0000 7f43927fc700 1 -- 172.21.15.60:0/2288930894 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f438c009ea0 con 0x7f43940b50a0 2022-06-29T15:58:48.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.537+0000 7f43927fc700 1 -- 172.21.15.60:0/2288930894 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f438c00b760 con 0x7f43940b50a0 2022-06-29T15:58:48.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.537+0000 7f43927fc700 1 -- 172.21.15.60:0/2288930894 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f438c00b8d0 con 0x7f43940b50a0 2022-06-29T15:58:48.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.541+0000 7f4398f34700 1 -- 172.21.15.60:0/2288930894 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43940b50a0 msgr2=0x7f43940b4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:48.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.541+0000 7f4398f34700 1 --2- 172.21.15.60:0/2288930894 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43940b50a0 0x7f43940b4340 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f4388001110 tx=0x7f438c0091d0).stop 2022-06-29T15:58:48.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.541+0000 7f4398f34700 1 -- 172.21.15.60:0/2288930894 shutdown_connections 2022-06-29T15:58:48.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.541+0000 7f4398f34700 1 --2- 172.21.15.60:0/2288930894 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43940b50a0 0x7f43940b4340 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.541+0000 7f4398f34700 1 -- 172.21.15.60:0/2288930894 >> 172.21.15.60:0/2288930894 conn(0x7f4394062090 msgr2=0x7f4394062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:48.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.541+0000 7f4398f34700 1 -- 172.21.15.60:0/2288930894 shutdown_connections 2022-06-29T15:58:48.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.541+0000 7f4398f34700 1 -- 172.21.15.60:0/2288930894 wait complete. 2022-06-29T15:58:48.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.541+0000 7f4398f34700 1 Processor -- start 2022-06-29T15:58:48.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.541+0000 7f4398f34700 1 -- start start 2022-06-29T15:58:48.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.541+0000 7f4398f34700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43940b4960 0x7f43940b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:48.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.541+0000 7f4398f34700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43940b50a0 0x7f4394159e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:48.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.541+0000 7f4398f34700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43940b5eb0 0x7f439415a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:48.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.541+0000 7f4398f34700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4394159850 con 0x7f43940b4960 2022-06-29T15:58:48.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.541+0000 7f4398f34700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4394145250 con 0x7f43940b5eb0 2022-06-29T15:58:48.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.541+0000 7f4398f34700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4394144660 con 0x7f43940b50a0 2022-06-29T15:58:48.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.541+0000 7f43937fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43940b4960 0x7f43940b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:48.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.541+0000 7f43937fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43940b4960 0x7f43940b1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45864/0 (socket says 172.21.15.60:45864) 2022-06-29T15:58:48.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.541+0000 7f43937fe700 1 -- 172.21.15.60:0/2057745174 learned_addr learned my addr 172.21.15.60:0/2057745174 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:48.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.541+0000 7f4392ffd700 1 --2- 172.21.15.60:0/2057745174 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43940b50a0 0x7f4394159e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:48.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.541+0000 7f4393fff700 1 --2- 172.21.15.60:0/2057745174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43940b5eb0 0x7f439415a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:48.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.541+0000 7f43937fe700 1 -- 172.21.15.60:0/2057745174 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43940b50a0 msgr2=0x7f4394159e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:48.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.541+0000 7f43937fe700 1 --2- 172.21.15.60:0/2057745174 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43940b50a0 0x7f4394159e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.541+0000 7f43937fe700 1 -- 172.21.15.60:0/2057745174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43940b5eb0 msgr2=0x7f439415a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:48.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.541+0000 7f43937fe700 1 --2- 172.21.15.60:0/2057745174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43940b5eb0 0x7f439415a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.541+0000 7f43937fe700 1 -- 172.21.15.60:0/2057745174 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f438c009470 con 0x7f43940b4960 2022-06-29T15:58:48.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.541+0000 7f43937fe700 1 --2- 172.21.15.60:0/2057745174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43940b4960 0x7f43940b1cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f4388000fe0 tx=0x7f4388007c40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:48.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.541+0000 7f4390ff9700 1 -- 172.21.15.60:0/2057745174 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43880070d0 con 0x7f43940b4960 2022-06-29T15:58:48.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.541+0000 7f4390ff9700 1 -- 172.21.15.60:0/2057745174 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4388007230 con 0x7f43940b4960 2022-06-29T15:58:48.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.541+0000 7f4398f34700 1 -- 172.21.15.60:0/2057745174 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f439415a800 con 0x7f43940b4960 2022-06-29T15:58:48.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.541+0000 7f4390ff9700 1 -- 172.21.15.60:0/2057745174 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4388005610 con 0x7f43940b4960 2022-06-29T15:58:48.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.541+0000 7f4398f34700 1 -- 172.21.15.60:0/2057745174 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43940b21b0 con 0x7f43940b4960 2022-06-29T15:58:48.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.545+0000 7f4398f34700 1 -- 172.21.15.60:0/2057745174 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f439405cc00 con 0x7f43940b4960 2022-06-29T15:58:48.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.545+0000 7f4390ff9700 1 -- 172.21.15.60:0/2057745174 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f438801a060 con 0x7f43940b4960 2022-06-29T15:58:48.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.545+0000 7f4390ff9700 1 --2- 172.21.15.60:0/2057745174 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4374053aa0 0x7f4374055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:48.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.545+0000 7f4390ff9700 1 -- 172.21.15.60:0/2057745174 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f438806b7c0 con 0x7f43940b4960 2022-06-29T15:58:48.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.545+0000 7f4392ffd700 1 --2- 172.21.15.60:0/2057745174 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4374053aa0 0x7f4374055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:48.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.545+0000 7f4392ffd700 1 --2- 172.21.15.60:0/2057745174 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4374053aa0 0x7f4374055f60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f438400d000 tx=0x7f43840079b0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:48.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.553+0000 7f4390ff9700 1 -- 172.21.15.60:0/2057745174 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f43880425d0 con 0x7f43940b4960 2022-06-29T15:58:48.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.673+0000 7f4398f34700 1 -- 172.21.15.60:0/2057745174 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f43940573d0 con 0x7f43940b4960 2022-06-29T15:58:48.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.677+0000 7f4390ff9700 1 -- 172.21.15.60:0/2057745174 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7f4388018020 con 0x7f43940b4960 2022-06-29T15:58:48.682 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T15:58:48.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.677+0000 7f4398f34700 1 -- 172.21.15.60:0/2057745174 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4374053aa0 msgr2=0x7f4374055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:48.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.677+0000 7f4398f34700 1 --2- 172.21.15.60:0/2057745174 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4374053aa0 0x7f4374055f60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f438400d000 tx=0x7f43840079b0).stop 2022-06-29T15:58:48.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.677+0000 7f4398f34700 1 -- 172.21.15.60:0/2057745174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43940b4960 msgr2=0x7f43940b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:48.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.677+0000 7f4398f34700 1 --2- 172.21.15.60:0/2057745174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43940b4960 0x7f43940b1cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f4388000fe0 tx=0x7f4388007c40).stop 2022-06-29T15:58:48.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.677+0000 7f4398f34700 1 -- 172.21.15.60:0/2057745174 shutdown_connections 2022-06-29T15:58:48.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.677+0000 7f4398f34700 1 --2- 172.21.15.60:0/2057745174 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4374053aa0 0x7f4374055f60 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.677+0000 7f4398f34700 1 --2- 172.21.15.60:0/2057745174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43940b5eb0 0x7f439415a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.677+0000 7f4398f34700 1 --2- 172.21.15.60:0/2057745174 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43940b50a0 0x7f4394159e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.677+0000 7f4398f34700 1 --2- 172.21.15.60:0/2057745174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43940b4960 0x7f43940b1cc0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.677+0000 7f4398f34700 1 -- 172.21.15.60:0/2057745174 >> 172.21.15.60:0/2057745174 conn(0x7f4394062090 msgr2=0x7f439414c720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:48.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.677+0000 7f4398f34700 1 -- 172.21.15.60:0/2057745174 shutdown_connections 2022-06-29T15:58:48.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.677+0000 7f4398f34700 1 -- 172.21.15.60:0/2057745174 wait complete. 2022-06-29T15:58:48.701 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-29T15:58:48.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.805+0000 7f6d39f90700 1 Processor -- start 2022-06-29T15:58:48.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.805+0000 7f6d39f90700 1 -- start start 2022-06-29T15:58:48.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.805+0000 7f6d39f90700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6d3412bde0 0x7f6d34142ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:48.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.805+0000 7f6d39f90700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6d34066b20 0x7f6d34142fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:48.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.805+0000 7f6d39f90700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d340ae9a0 con 0x7f6d3405d070 2022-06-29T15:58:48.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.805+0000 7f6d39f90700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d34143b60 con 0x7f6d3412bde0 2022-06-29T15:58:48.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.805+0000 7f6d39f90700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d340aced0 con 0x7f6d34066b20 2022-06-29T15:58:48.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.805+0000 7f6d3978f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6d34066b20 0x7f6d34142fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:48.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.805+0000 7f6d33fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6d3412bde0 0x7f6d34142ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:48.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.805+0000 7f6d3978f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6d34066b20 0x7f6d34142fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:58968/0 (socket says 172.21.15.60:58968) 2022-06-29T15:58:48.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.805+0000 7f6d3978f700 1 -- 172.21.15.60:0/1422711122 learned_addr learned my addr 172.21.15.60:0/1422711122 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:48.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.805+0000 7f6d33fff700 1 -- 172.21.15.60:0/1422711122 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6d34066b20 msgr2=0x7f6d34142fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:48.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.805+0000 7f6d33fff700 1 --2- 172.21.15.60:0/1422711122 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6d34066b20 0x7f6d34142fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.805+0000 7f6d33fff700 1 -- 172.21.15.60:0/1422711122 >> v1:172.21.15.92:6789/0 conn(0x7f6d3405d070 legacy=0x7f6d3412b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:48.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.805+0000 7f6d33fff700 1 -- 172.21.15.60:0/1422711122 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d3405d450 con 0x7f6d3412bde0 2022-06-29T15:58:48.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.805+0000 7f6d33fff700 1 --2- 172.21.15.60:0/1422711122 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d3412bde0 0x7f6d34142ae0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f6d1c002f70 tx=0x7f6d1c006df0).ready entity=mon.1 client_cookie=8f2eb5ab4a79a47d server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:48.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.805+0000 7f6d337fe700 1 -- 172.21.15.60:0/1422711122 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d1c00d570 con 0x7f6d3412bde0 2022-06-29T15:58:48.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.805+0000 7f6d337fe700 1 -- 172.21.15.60:0/1422711122 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d1c000e90 con 0x7f6d3412bde0 2022-06-29T15:58:48.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.805+0000 7f6d337fe700 1 -- 172.21.15.60:0/1422711122 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d1c00d8c0 con 0x7f6d3412bde0 2022-06-29T15:58:48.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.809+0000 7f6d39f90700 1 -- 172.21.15.60:0/1422711122 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d3412bde0 msgr2=0x7f6d34142ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:48.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.809+0000 7f6d39f90700 1 --2- 172.21.15.60:0/1422711122 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d3412bde0 0x7f6d34142ae0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f6d1c002f70 tx=0x7f6d1c006df0).stop 2022-06-29T15:58:48.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.809+0000 7f6d39f90700 1 -- 172.21.15.60:0/1422711122 shutdown_connections 2022-06-29T15:58:48.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.809+0000 7f6d39f90700 1 --2- 172.21.15.60:0/1422711122 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6d34066b20 0x7f6d34142fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.809+0000 7f6d39f90700 1 --2- 172.21.15.60:0/1422711122 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d3412bde0 0x7f6d34142ae0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.809+0000 7f6d39f90700 1 -- 172.21.15.60:0/1422711122 >> 172.21.15.60:0/1422711122 conn(0x7f6d340b5890 msgr2=0x7f6d340b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:48.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.809+0000 7f6d39f90700 1 -- 172.21.15.60:0/1422711122 shutdown_connections 2022-06-29T15:58:48.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.809+0000 7f6d39f90700 1 -- 172.21.15.60:0/1422711122 wait complete. 2022-06-29T15:58:48.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.809+0000 7f6d39f90700 1 Processor -- start 2022-06-29T15:58:48.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.809+0000 7f6d39f90700 1 -- start start 2022-06-29T15:58:48.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.809+0000 7f6d39f90700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6d3405d070 0x7f6d3412d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:48.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.809+0000 7f6d39f90700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d34066b20 0x7f6d34128110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:48.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.809+0000 7f6d39f90700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d3412bde0 0x7f6d340b06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:48.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.809+0000 7f6d39f90700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d340b1ea0 con 0x7f6d3412bde0 2022-06-29T15:58:48.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.809+0000 7f6d39f90700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3412fd30 con 0x7f6d34066b20 2022-06-29T15:58:48.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.809+0000 7f6d39f90700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3412e390 con 0x7f6d3405d070 2022-06-29T15:58:48.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.809+0000 7f6d3978f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d3412bde0 0x7f6d340b06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:48.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.809+0000 7f6d3978f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d3412bde0 0x7f6d340b06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45882/0 (socket says 172.21.15.60:45882) 2022-06-29T15:58:48.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.809+0000 7f6d3978f700 1 -- 172.21.15.60:0/4142107968 learned_addr learned my addr 172.21.15.60:0/4142107968 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:48.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.809+0000 7f6d33fff700 1 --2- 172.21.15.60:0/4142107968 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d34066b20 0x7f6d34128110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:48.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.809+0000 7f6d38f8e700 1 --2- 172.21.15.60:0/4142107968 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6d3405d070 0x7f6d3412d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:48.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.809+0000 7f6d3978f700 1 -- 172.21.15.60:0/4142107968 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6d3405d070 msgr2=0x7f6d3412d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:48.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.809+0000 7f6d3978f700 1 --2- 172.21.15.60:0/4142107968 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6d3405d070 0x7f6d3412d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.813+0000 7f6d3978f700 1 -- 172.21.15.60:0/4142107968 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d34066b20 msgr2=0x7f6d34128110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:48.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.813+0000 7f6d3978f700 1 --2- 172.21.15.60:0/4142107968 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d34066b20 0x7f6d34128110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.813+0000 7f6d3978f700 1 -- 172.21.15.60:0/4142107968 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d1c00b180 con 0x7f6d3412bde0 2022-06-29T15:58:48.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.813+0000 7f6d3978f700 1 --2- 172.21.15.60:0/4142107968 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d3412bde0 0x7f6d340b06b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f6d2c00c000 tx=0x7f6d2c009290).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:48.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.813+0000 7f6d31ffb700 1 -- 172.21.15.60:0/4142107968 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d2c00c850 con 0x7f6d3412bde0 2022-06-29T15:58:48.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.813+0000 7f6d31ffb700 1 -- 172.21.15.60:0/4142107968 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d2c00c9b0 con 0x7f6d3412bde0 2022-06-29T15:58:48.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.813+0000 7f6d39f90700 1 -- 172.21.15.60:0/4142107968 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d3412e5c0 con 0x7f6d3412bde0 2022-06-29T15:58:48.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.813+0000 7f6d31ffb700 1 -- 172.21.15.60:0/4142107968 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d2c00cd10 con 0x7f6d3412bde0 2022-06-29T15:58:48.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.813+0000 7f6d39f90700 1 -- 172.21.15.60:0/4142107968 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d3412da60 con 0x7f6d3412bde0 2022-06-29T15:58:48.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.813+0000 7f6d39f90700 1 -- 172.21.15.60:0/4142107968 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d3412d1b0 con 0x7f6d3412bde0 2022-06-29T15:58:48.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.813+0000 7f6d31ffb700 1 -- 172.21.15.60:0/4142107968 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6d2c022070 con 0x7f6d3412bde0 2022-06-29T15:58:48.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.813+0000 7f6d31ffb700 1 --2- 172.21.15.60:0/4142107968 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6d18053b00 0x7f6d18055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:48.829 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.813+0000 7f6d31ffb700 1 -- 172.21.15.60:0/4142107968 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6d2c06c620 con 0x7f6d3412bde0 2022-06-29T15:58:48.829 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.813+0000 7f6d38f8e700 1 --2- 172.21.15.60:0/4142107968 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6d18053b00 0x7f6d18055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:48.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.817+0000 7f6d38f8e700 1 --2- 172.21.15.60:0/4142107968 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6d18053b00 0x7f6d18055fc0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f6d2400b000 tx=0x7f6d24009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:48.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.821+0000 7f6d31ffb700 1 -- 172.21.15.60:0/4142107968 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6d2c043540 con 0x7f6d3412bde0 2022-06-29T15:58:48.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.945+0000 7f6d39f90700 1 -- 172.21.15.60:0/4142107968 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6d34131510 con 0x7f6d3412bde0 2022-06-29T15:58:48.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.945+0000 7f6d31ffb700 1 -- 172.21.15.60:0/4142107968 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7f6d2c0160a0 con 0x7f6d3412bde0 2022-06-29T15:58:48.951 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T15:58:48.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.949+0000 7f6d39f90700 1 -- 172.21.15.60:0/4142107968 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6d18053b00 msgr2=0x7f6d18055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:48.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.949+0000 7f6d39f90700 1 --2- 172.21.15.60:0/4142107968 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6d18053b00 0x7f6d18055fc0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f6d2400b000 tx=0x7f6d24009040).stop 2022-06-29T15:58:48.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.949+0000 7f6d39f90700 1 -- 172.21.15.60:0/4142107968 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d3412bde0 msgr2=0x7f6d340b06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:48.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.949+0000 7f6d39f90700 1 --2- 172.21.15.60:0/4142107968 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d3412bde0 0x7f6d340b06b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f6d2c00c000 tx=0x7f6d2c009290).stop 2022-06-29T15:58:48.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.949+0000 7f6d39f90700 1 -- 172.21.15.60:0/4142107968 shutdown_connections 2022-06-29T15:58:48.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.949+0000 7f6d39f90700 1 --2- 172.21.15.60:0/4142107968 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6d18053b00 0x7f6d18055fc0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.949+0000 7f6d39f90700 1 --2- 172.21.15.60:0/4142107968 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d3412bde0 0x7f6d340b06b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.949+0000 7f6d39f90700 1 --2- 172.21.15.60:0/4142107968 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d34066b20 0x7f6d34128110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.949+0000 7f6d39f90700 1 --2- 172.21.15.60:0/4142107968 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6d3405d070 0x7f6d3412d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:48.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.949+0000 7f6d39f90700 1 -- 172.21.15.60:0/4142107968 >> 172.21.15.60:0/4142107968 conn(0x7f6d340b5890 msgr2=0x7f6d34142930 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:48.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.949+0000 7f6d39f90700 1 -- 172.21.15.60:0/4142107968 shutdown_connections 2022-06-29T15:58:48.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:48.949+0000 7f6d39f90700 1 -- 172.21.15.60:0/4142107968 wait complete. 2022-06-29T15:58:48.970 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T15:58:49.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.069+0000 7f94c2a38700 1 Processor -- start 2022-06-29T15:58:49.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.069+0000 7f94c2a38700 1 -- start start 2022-06-29T15:58:49.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.069+0000 7f94c2a38700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f94bc0b5eb0 0x7f94bc142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:49.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.069+0000 7f94c2a38700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f94bc0b2c50 0x7f94bc0b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:49.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.069+0000 7f94c2a38700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94bc0b0cf0 con 0x7f94bc0b48b0 2022-06-29T15:58:49.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.069+0000 7f94c2a38700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94bc0b54e0 con 0x7f94bc0b2c50 2022-06-29T15:58:49.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.069+0000 7f94c2a38700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94bc0b4d10 con 0x7f94bc0b5eb0 2022-06-29T15:58:49.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.069+0000 7f94c1235700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f94bc0b48b0 0x7f94bc0b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53976/0 (socket says 172.21.15.60:53976) 2022-06-29T15:58:49.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.069+0000 7f94c2237700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f94bc0b2c50 0x7f94bc0b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:49.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.069+0000 7f94c1235700 1 -- 172.21.15.60:0/4070507170 learned_addr learned my addr 172.21.15.60:0/4070507170 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:49.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.069+0000 7f94c1a36700 1 --2- 172.21.15.60:0/4070507170 >> v2:172.21.15.92:3300/0 conn(0x7f94bc0b5eb0 0x7f94bc142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:49.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.069+0000 7f94c0a34700 1 -- 172.21.15.60:0/4070507170 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3927425564 0 0) 0x7f94b8001490 con 0x7f94bc0b48b0 2022-06-29T15:58:49.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.069+0000 7f94c0a34700 1 -- 172.21.15.60:0/4070507170 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94a8003260 con 0x7f94bc0b48b0 2022-06-29T15:58:49.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.069+0000 7f94c0a34700 1 -- 172.21.15.60:0/4070507170 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2093396245 0 0) 0x7f94a8003260 con 0x7f94bc0b48b0 2022-06-29T15:58:49.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.069+0000 7f94c2237700 1 -- 172.21.15.60:0/4070507170 >> v2:172.21.15.92:3300/0 conn(0x7f94bc0b5eb0 msgr2=0x7f94bc142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:49.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.069+0000 7f94c2237700 1 --2- 172.21.15.60:0/4070507170 >> v2:172.21.15.92:3300/0 conn(0x7f94bc0b5eb0 0x7f94bc142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:49.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.069+0000 7f94c2237700 1 -- 172.21.15.60:0/4070507170 >> v1:172.21.15.60:6789/0 conn(0x7f94bc0b48b0 legacy=0x7f94bc0b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:49.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.069+0000 7f94c2237700 1 -- 172.21.15.60:0/4070507170 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94bc0b6290 con 0x7f94bc0b2c50 2022-06-29T15:58:49.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.069+0000 7f94c2237700 1 --2- 172.21.15.60:0/4070507170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94bc0b2c50 0x7f94bc0b30c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f94b8001d10 tx=0x7f94b4006df0).ready entity=mon.0 client_cookie=ee81047ef69c107e server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:49.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.069+0000 7f94c0a34700 1 -- 172.21.15.60:0/4070507170 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94b8001b70 con 0x7f94bc0b2c50 2022-06-29T15:58:49.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.069+0000 7f94c0a34700 1 -- 172.21.15.60:0/4070507170 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b4000e90 con 0x7f94bc0b2c50 2022-06-29T15:58:49.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.069+0000 7f94c0a34700 1 -- 172.21.15.60:0/4070507170 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94b4012460 con 0x7f94bc0b2c50 2022-06-29T15:58:49.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.073+0000 7f94c2a38700 1 -- 172.21.15.60:0/4070507170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94bc0b2c50 msgr2=0x7f94bc0b30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:49.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.073+0000 7f94c2a38700 1 --2- 172.21.15.60:0/4070507170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94bc0b2c50 0x7f94bc0b30c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f94b8001d10 tx=0x7f94b4006df0).stop 2022-06-29T15:58:49.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.073+0000 7f94c2a38700 1 -- 172.21.15.60:0/4070507170 shutdown_connections 2022-06-29T15:58:49.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.073+0000 7f94c2a38700 1 --2- 172.21.15.60:0/4070507170 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94bc0b2c50 0x7f94bc0b30c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:49.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.073+0000 7f94c2a38700 1 --2- 172.21.15.60:0/4070507170 >> v2:172.21.15.92:3300/0 conn(0x7f94bc0b5eb0 0x7f94bc142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:49.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.073+0000 7f94c2a38700 1 -- 172.21.15.60:0/4070507170 >> 172.21.15.60:0/4070507170 conn(0x7f94bc062090 msgr2=0x7f94bc062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:49.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.073+0000 7f94c2a38700 1 -- 172.21.15.60:0/4070507170 shutdown_connections 2022-06-29T15:58:49.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.073+0000 7f94c2a38700 1 -- 172.21.15.60:0/4070507170 wait complete. 2022-06-29T15:58:49.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.073+0000 7f94c2a38700 1 Processor -- start 2022-06-29T15:58:49.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.073+0000 7f94c2a38700 1 -- start start 2022-06-29T15:58:49.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.073+0000 7f94c2a38700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94bc0b2c50 0x7f94bc13cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:49.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.073+0000 7f94c2a38700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94bc0b48b0 0x7f94bc151ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:49.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.073+0000 7f94c2a38700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94bc0b5eb0 0x7f94bc152390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:49.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.073+0000 7f94c2a38700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc0af290 con 0x7f94bc0b48b0 2022-06-29T15:58:49.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.073+0000 7f94c2a38700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc141290 con 0x7f94bc0b5eb0 2022-06-29T15:58:49.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.073+0000 7f94c2a38700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc155f40 con 0x7f94bc0b2c50 2022-06-29T15:58:49.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.073+0000 7f94c1235700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94bc0b48b0 0x7f94bc151ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:49.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.073+0000 7f94c1235700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94bc0b48b0 0x7f94bc151ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45892/0 (socket says 172.21.15.60:45892) 2022-06-29T15:58:49.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.073+0000 7f94c1235700 1 -- 172.21.15.60:0/2949320433 learned_addr learned my addr 172.21.15.60:0/2949320433 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:49.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.073+0000 7f94c2237700 1 --2- 172.21.15.60:0/2949320433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94bc0b5eb0 0x7f94bc152390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:49.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.073+0000 7f94c1a36700 1 --2- 172.21.15.60:0/2949320433 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94bc0b2c50 0x7f94bc13cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:49.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.073+0000 7f94c1235700 1 -- 172.21.15.60:0/2949320433 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94bc0b2c50 msgr2=0x7f94bc13cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:49.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.073+0000 7f94c1235700 1 --2- 172.21.15.60:0/2949320433 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94bc0b2c50 0x7f94bc13cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:49.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.073+0000 7f94c1235700 1 -- 172.21.15.60:0/2949320433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94bc0b5eb0 msgr2=0x7f94bc152390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:49.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.073+0000 7f94c1235700 1 --2- 172.21.15.60:0/2949320433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94bc0b5eb0 0x7f94bc152390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:49.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.073+0000 7f94c1235700 1 -- 172.21.15.60:0/2949320433 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b400b180 con 0x7f94bc0b48b0 2022-06-29T15:58:49.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.073+0000 7f94c1235700 1 --2- 172.21.15.60:0/2949320433 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94bc0b48b0 0x7f94bc151ea0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f94b8001dd0 tx=0x7f94b8001630).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:49.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.073+0000 7f94b2ffd700 1 -- 172.21.15.60:0/2949320433 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94b8005ba0 con 0x7f94bc0b48b0 2022-06-29T15:58:49.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.077+0000 7f94b2ffd700 1 -- 172.21.15.60:0/2949320433 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b8005d00 con 0x7f94bc0b48b0 2022-06-29T15:58:49.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.077+0000 7f94c2a38700 1 -- 172.21.15.60:0/2949320433 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94bc1561e0 con 0x7f94bc0b48b0 2022-06-29T15:58:49.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.077+0000 7f94b2ffd700 1 -- 172.21.15.60:0/2949320433 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94b8007300 con 0x7f94bc0b48b0 2022-06-29T15:58:49.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.077+0000 7f94c2a38700 1 -- 172.21.15.60:0/2949320433 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94bc0af8a0 con 0x7f94bc0b48b0 2022-06-29T15:58:49.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.077+0000 7f94c2a38700 1 -- 172.21.15.60:0/2949320433 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94bc152b70 con 0x7f94bc0b48b0 2022-06-29T15:58:49.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.077+0000 7f94b2ffd700 1 -- 172.21.15.60:0/2949320433 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f94b8010030 con 0x7f94bc0b48b0 2022-06-29T15:58:49.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.077+0000 7f94b2ffd700 1 --2- 172.21.15.60:0/2949320433 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f949c053b30 0x7f949c055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:49.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.077+0000 7f94b2ffd700 1 -- 172.21.15.60:0/2949320433 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f94b806be20 con 0x7f94bc0b48b0 2022-06-29T15:58:49.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.077+0000 7f94c1a36700 1 --2- 172.21.15.60:0/2949320433 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f949c053b30 0x7f949c055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:49.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.077+0000 7f94c1a36700 1 --2- 172.21.15.60:0/2949320433 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f949c053b30 0x7f949c055ff0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f94ac00b000 tx=0x7f94ac008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:49.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.081+0000 7f94b2ffd700 1 -- 172.21.15.60:0/2949320433 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f94b8042ca0 con 0x7f94bc0b48b0 2022-06-29T15:58:49.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.209+0000 7f94c2a38700 1 -- 172.21.15.60:0/2949320433 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94bc0b0cf0 con 0x7f94bc0b48b0 2022-06-29T15:58:49.216 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.209+0000 7f94b2ffd700 1 -- 172.21.15.60:0/2949320433 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16038 (secure 0 0 0) 0x7f94bc0b0cf0 con 0x7f94bc0b48b0 2022-06-29T15:58:49.217 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:58:49.217 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":23,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T15:58:26.345644+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-06-29T15:58:22.424395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-06-29T15:58:23.343438+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T15:58:49.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.213+0000 7f94c2a38700 1 -- 172.21.15.60:0/2949320433 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f949c053b30 msgr2=0x7f949c055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:49.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.213+0000 7f94c2a38700 1 --2- 172.21.15.60:0/2949320433 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f949c053b30 0x7f949c055ff0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f94ac00b000 tx=0x7f94ac008040).stop 2022-06-29T15:58:49.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.213+0000 7f94c2a38700 1 -- 172.21.15.60:0/2949320433 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94bc0b48b0 msgr2=0x7f94bc151ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:49.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.213+0000 7f94c2a38700 1 --2- 172.21.15.60:0/2949320433 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94bc0b48b0 0x7f94bc151ea0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f94b8001dd0 tx=0x7f94b8001630).stop 2022-06-29T15:58:49.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.213+0000 7f94c2a38700 1 -- 172.21.15.60:0/2949320433 shutdown_connections 2022-06-29T15:58:49.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.213+0000 7f94c2a38700 1 --2- 172.21.15.60:0/2949320433 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f949c053b30 0x7f949c055ff0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:49.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.213+0000 7f94c2a38700 1 --2- 172.21.15.60:0/2949320433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94bc0b5eb0 0x7f94bc152390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:49.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.213+0000 7f94c2a38700 1 --2- 172.21.15.60:0/2949320433 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94bc0b48b0 0x7f94bc151ea0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:49.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.213+0000 7f94c2a38700 1 --2- 172.21.15.60:0/2949320433 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94bc0b2c50 0x7f94bc13cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:49.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.213+0000 7f94c2a38700 1 -- 172.21.15.60:0/2949320433 >> 172.21.15.60:0/2949320433 conn(0x7f94bc062090 msgr2=0x7f94bc1430d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:49.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.213+0000 7f94c2a38700 1 -- 172.21.15.60:0/2949320433 shutdown_connections 2022-06-29T15:58:49.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.213+0000 7f94c2a38700 1 -- 172.21.15.60:0/2949320433 wait complete. 2022-06-29T15:58:49.230 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-29T15:58:22.424395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-06-29T15:58:23.343438+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-29T15:58:49.231 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T15:58:49.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.337+0000 7fc14a4a6700 1 Processor -- start 2022-06-29T15:58:49.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.337+0000 7fc14a4a6700 1 -- start start 2022-06-29T15:58:49.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.337+0000 7fc14a4a6700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc144067930 0x7fc144146f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:49.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.337+0000 7fc14a4a6700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc14414abe0 0x7fc1440aee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:49.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.337+0000 7fc14a4a6700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc144067d10 con 0x7fc1440b18b0 2022-06-29T15:58:49.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.337+0000 7fc14a4a6700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1440b34a0 con 0x7fc144067930 2022-06-29T15:58:49.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.337+0000 7fc14a4a6700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1441474f0 con 0x7fc14414abe0 2022-06-29T15:58:49.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.337+0000 7fc143fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc144067930 0x7fc144146f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:49.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.337+0000 7fc143fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc144067930 0x7fc144146f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45900/0 (socket says 172.21.15.60:45900) 2022-06-29T15:58:49.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.337+0000 7fc143fff700 1 -- 172.21.15.60:0/2948580931 learned_addr learned my addr 172.21.15.60:0/2948580931 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:49.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.337+0000 7fc148a43700 1 --2- 172.21.15.60:0/2948580931 >> v2:172.21.15.92:3300/0 conn(0x7fc14414abe0 0x7fc1440aee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:49.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.337+0000 7fc143fff700 1 -- 172.21.15.60:0/2948580931 >> v2:172.21.15.92:3300/0 conn(0x7fc14414abe0 msgr2=0x7fc1440aee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:49.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.337+0000 7fc143fff700 1 --2- 172.21.15.60:0/2948580931 >> v2:172.21.15.92:3300/0 conn(0x7fc14414abe0 0x7fc1440aee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:49.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.337+0000 7fc143fff700 1 -- 172.21.15.60:0/2948580931 >> v1:172.21.15.92:6789/0 conn(0x7fc1440b18b0 legacy=0x7fc14414a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:49.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.337+0000 7fc143fff700 1 -- 172.21.15.60:0/2948580931 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc144147790 con 0x7fc144067930 2022-06-29T15:58:49.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.337+0000 7fc143fff700 1 --2- 172.21.15.60:0/2948580931 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc144067930 0x7fc144146f00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fc134002f70 tx=0x7fc134006eb0).ready entity=mon.0 client_cookie=344f8dc9d48e1359 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:49.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.337+0000 7fc142ffd700 1 -- 172.21.15.60:0/2948580931 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc13400d5c0 con 0x7fc144067930 2022-06-29T15:58:49.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.337+0000 7fc142ffd700 1 -- 172.21.15.60:0/2948580931 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc134000e90 con 0x7fc144067930 2022-06-29T15:58:49.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.337+0000 7fc142ffd700 1 -- 172.21.15.60:0/2948580931 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc13400d950 con 0x7fc144067930 2022-06-29T15:58:49.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.341+0000 7fc14a4a6700 1 -- 172.21.15.60:0/2948580931 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc144067930 msgr2=0x7fc144146f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:49.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.341+0000 7fc14a4a6700 1 --2- 172.21.15.60:0/2948580931 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc144067930 0x7fc144146f00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fc134002f70 tx=0x7fc134006eb0).stop 2022-06-29T15:58:49.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.341+0000 7fc14a4a6700 1 -- 172.21.15.60:0/2948580931 shutdown_connections 2022-06-29T15:58:49.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.341+0000 7fc14a4a6700 1 --2- 172.21.15.60:0/2948580931 >> v2:172.21.15.92:3300/0 conn(0x7fc14414abe0 0x7fc1440aee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:49.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.341+0000 7fc14a4a6700 1 --2- 172.21.15.60:0/2948580931 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc144067930 0x7fc144146f00 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:49.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.341+0000 7fc14a4a6700 1 -- 172.21.15.60:0/2948580931 >> 172.21.15.60:0/2948580931 conn(0x7fc14405f0b0 msgr2=0x7fc14405f490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:49.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.341+0000 7fc14a4a6700 1 -- 172.21.15.60:0/2948580931 shutdown_connections 2022-06-29T15:58:49.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.341+0000 7fc14a4a6700 1 -- 172.21.15.60:0/2948580931 wait complete. 2022-06-29T15:58:49.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.341+0000 7fc14a4a6700 1 Processor -- start 2022-06-29T15:58:49.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.341+0000 7fc14a4a6700 1 -- start start 2022-06-29T15:58:49.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.345+0000 7fc14a4a6700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc144067930 0x7fc144144040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:49.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.345+0000 7fc14a4a6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1440b18b0 0x7fc144146290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:49.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.345+0000 7fc14a4a6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc14414abe0 0x7fc144062fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:49.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.345+0000 7fc14a4a6700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc144143660 con 0x7fc14414abe0 2022-06-29T15:58:49.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.345+0000 7fc14a4a6700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc144143900 con 0x7fc144067930 2022-06-29T15:58:49.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.345+0000 7fc14a4a6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc144143ba0 con 0x7fc1440b18b0 2022-06-29T15:58:49.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.345+0000 7fc148a43700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc14414abe0 0x7fc144062fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:49.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.345+0000 7fc148a43700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc14414abe0 0x7fc144062fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45910/0 (socket says 172.21.15.60:45910) 2022-06-29T15:58:49.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.345+0000 7fc148a43700 1 -- 172.21.15.60:0/1783126151 learned_addr learned my addr 172.21.15.60:0/1783126151 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:49.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.345+0000 7fc143fff700 1 --2- 172.21.15.60:0/1783126151 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc144067930 0x7fc144144040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:49.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.345+0000 7fc1437fe700 1 --2- 172.21.15.60:0/1783126151 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1440b18b0 0x7fc144146290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:49.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.345+0000 7fc148a43700 1 -- 172.21.15.60:0/1783126151 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1440b18b0 msgr2=0x7fc144146290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:49.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.345+0000 7fc148a43700 1 --2- 172.21.15.60:0/1783126151 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1440b18b0 0x7fc144146290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:49.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.345+0000 7fc148a43700 1 -- 172.21.15.60:0/1783126151 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc144067930 msgr2=0x7fc144144040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:49.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.345+0000 7fc148a43700 1 --2- 172.21.15.60:0/1783126151 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc144067930 0x7fc144144040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:49.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.345+0000 7fc148a43700 1 -- 172.21.15.60:0/1783126151 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc13400b1e0 con 0x7fc14414abe0 2022-06-29T15:58:49.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.345+0000 7fc148a43700 1 --2- 172.21.15.60:0/1783126151 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc14414abe0 0x7fc144062fd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fc13c006e00 tx=0x7fc13c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:49.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.345+0000 7fc1417fa700 1 -- 172.21.15.60:0/1783126151 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc13c00c670 con 0x7fc14414abe0 2022-06-29T15:58:49.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.345+0000 7fc1417fa700 1 -- 172.21.15.60:0/1783126151 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc13c00c7d0 con 0x7fc14414abe0 2022-06-29T15:58:49.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.345+0000 7fc14a4a6700 1 -- 172.21.15.60:0/1783126151 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1440634c0 con 0x7fc14414abe0 2022-06-29T15:58:49.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.345+0000 7fc1417fa700 1 -- 172.21.15.60:0/1783126151 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc13c00cb30 con 0x7fc14414abe0 2022-06-29T15:58:49.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.345+0000 7fc14a4a6700 1 -- 172.21.15.60:0/1783126151 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc144140860 con 0x7fc14414abe0 2022-06-29T15:58:49.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.349+0000 7fc14a4a6700 1 -- 172.21.15.60:0/1783126151 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc144143d60 con 0x7fc14414abe0 2022-06-29T15:58:49.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.349+0000 7fc1417fa700 1 -- 172.21.15.60:0/1783126151 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc13c00cc90 con 0x7fc14414abe0 2022-06-29T15:58:49.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.349+0000 7fc1417fa700 1 --2- 172.21.15.60:0/1783126151 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc124053a70 0x7fc124055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:49.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.349+0000 7fc1417fa700 1 -- 172.21.15.60:0/1783126151 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc13c06c2b0 con 0x7fc14414abe0 2022-06-29T15:58:49.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.349+0000 7fc143fff700 1 --2- 172.21.15.60:0/1783126151 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc124053a70 0x7fc124055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:49.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.349+0000 7fc143fff700 1 --2- 172.21.15.60:0/1783126151 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc124053a70 0x7fc124055f30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fc13401c000 tx=0x7fc134015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:49.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.353+0000 7fc1417fa700 1 -- 172.21.15.60:0/1783126151 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc13c043040 con 0x7fc14414abe0 2022-06-29T15:58:49.486 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.481+0000 7fc14a4a6700 1 -- 172.21.15.60:0/1783126151 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fc1440573d0 con 0x7fc14414abe0 2022-06-29T15:58:49.487 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.481+0000 7fc1417fa700 1 -- 172.21.15.60:0/1783126151 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v23) v1 ==== 110+0+10 (secure 0 0 0) 0x7fc13c046660 con 0x7fc14414abe0 2022-06-29T15:58:49.487 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T15:58:49.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.481+0000 7fc14a4a6700 1 -- 172.21.15.60:0/1783126151 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc124053a70 msgr2=0x7fc124055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:49.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.481+0000 7fc14a4a6700 1 --2- 172.21.15.60:0/1783126151 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc124053a70 0x7fc124055f30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fc13401c000 tx=0x7fc134015000).stop 2022-06-29T15:58:49.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.481+0000 7fc14a4a6700 1 -- 172.21.15.60:0/1783126151 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc14414abe0 msgr2=0x7fc144062fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:49.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.481+0000 7fc14a4a6700 1 --2- 172.21.15.60:0/1783126151 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc14414abe0 0x7fc144062fd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fc13c006e00 tx=0x7fc13c0091d0).stop 2022-06-29T15:58:49.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.485+0000 7fc14a4a6700 1 -- 172.21.15.60:0/1783126151 shutdown_connections 2022-06-29T15:58:49.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.485+0000 7fc14a4a6700 1 --2- 172.21.15.60:0/1783126151 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc124053a70 0x7fc124055f30 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:49.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.485+0000 7fc14a4a6700 1 --2- 172.21.15.60:0/1783126151 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc14414abe0 0x7fc144062fd0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:49.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.485+0000 7fc14a4a6700 1 --2- 172.21.15.60:0/1783126151 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1440b18b0 0x7fc144146290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:49.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.485+0000 7fc14a4a6700 1 --2- 172.21.15.60:0/1783126151 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc144067930 0x7fc144144040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:49.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.485+0000 7fc14a4a6700 1 -- 172.21.15.60:0/1783126151 >> 172.21.15.60:0/1783126151 conn(0x7fc14405f0b0 msgr2=0x7fc1440b4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:49.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.485+0000 7fc14a4a6700 1 -- 172.21.15.60:0/1783126151 shutdown_connections 2022-06-29T15:58:49.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.485+0000 7fc14a4a6700 1 -- 172.21.15.60:0/1783126151 wait complete. 2022-06-29T15:58:49.501 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T15:58:49.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.605+0000 7f6429475700 1 Processor -- start 2022-06-29T15:58:49.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.605+0000 7f6429475700 1 -- start start 2022-06-29T15:58:49.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.609+0000 7f6429475700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64240acd60 0x7f6424144900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:49.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.609+0000 7f6429475700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f64240657d0 0x7f6424129e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:49.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.609+0000 7f6429475700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6424143cf0 con 0x7f64240ad990 2022-06-29T15:58:49.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.609+0000 7f6429475700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6424064a70 con 0x7f64240657d0 2022-06-29T15:58:49.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.609+0000 7f6429475700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64240ad140 con 0x7f64240acd60 2022-06-29T15:58:49.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.609+0000 7f6428c74700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f64240657d0 0x7f6424129e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:49.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.609+0000 7f641bfff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f64240ad990 0x7f64241296f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54002/0 (socket says 172.21.15.60:54002) 2022-06-29T15:58:49.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.609+0000 7f641bfff700 1 -- 172.21.15.60:0/3805275323 learned_addr learned my addr 172.21.15.60:0/3805275323 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:49.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.609+0000 7f6423fff700 1 --2- 172.21.15.60:0/3805275323 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64240acd60 0x7f6424144900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:49.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.609+0000 7f6428c74700 1 -- 172.21.15.60:0/3805275323 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64240acd60 msgr2=0x7f6424144900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:49.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.609+0000 7f6428c74700 1 --2- 172.21.15.60:0/3805275323 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64240acd60 0x7f6424144900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:49.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.609+0000 7f6428c74700 1 -- 172.21.15.60:0/3805275323 >> v1:172.21.15.60:6789/0 conn(0x7f64240ad990 legacy=0x7f64241296f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:49.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.609+0000 7f6428c74700 1 -- 172.21.15.60:0/3805275323 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6424143f20 con 0x7f64240657d0 2022-06-29T15:58:49.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.609+0000 7f6423fff700 1 --2- 172.21.15.60:0/3805275323 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64240acd60 0x7f6424144900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:58:49.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.609+0000 7f6428c74700 1 --2- 172.21.15.60:0/3805275323 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64240657d0 0x7f6424129e00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f641c0032f0 tx=0x7f641c0091d0).ready entity=mon.0 client_cookie=f337a032c512335c server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:49.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.609+0000 7f64237fe700 1 -- 172.21.15.60:0/3805275323 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f641c009ea0 con 0x7f64240657d0 2022-06-29T15:58:49.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.609+0000 7f64237fe700 1 -- 172.21.15.60:0/3805275323 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f641c00b8a0 con 0x7f64240657d0 2022-06-29T15:58:49.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.609+0000 7f64237fe700 1 -- 172.21.15.60:0/3805275323 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f641c00ba10 con 0x7f64240657d0 2022-06-29T15:58:49.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.609+0000 7f6429475700 1 -- 172.21.15.60:0/3805275323 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64240657d0 msgr2=0x7f6424129e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:49.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.609+0000 7f6429475700 1 --2- 172.21.15.60:0/3805275323 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64240657d0 0x7f6424129e00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f641c0032f0 tx=0x7f641c0091d0).stop 2022-06-29T15:58:49.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.609+0000 7f6429475700 1 -- 172.21.15.60:0/3805275323 shutdown_connections 2022-06-29T15:58:49.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.609+0000 7f6429475700 1 --2- 172.21.15.60:0/3805275323 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64240657d0 0x7f6424129e00 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:49.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.609+0000 7f6429475700 1 --2- 172.21.15.60:0/3805275323 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64240acd60 0x7f6424144900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:49.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.609+0000 7f6429475700 1 -- 172.21.15.60:0/3805275323 >> 172.21.15.60:0/3805275323 conn(0x7f64240b6f80 msgr2=0x7f64240b7390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:49.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.609+0000 7f6429475700 1 -- 172.21.15.60:0/3805275323 shutdown_connections 2022-06-29T15:58:49.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.609+0000 7f6429475700 1 -- 172.21.15.60:0/3805275323 wait complete. 2022-06-29T15:58:49.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.613+0000 7f6429475700 1 Processor -- start 2022-06-29T15:58:49.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.613+0000 7f6429475700 1 -- start start 2022-06-29T15:58:49.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.613+0000 7f6429475700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64240657d0 0x7f64240b6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:49.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.613+0000 7f6429475700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64240acd60 0x7f642414fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:49.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.613+0000 7f6429475700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64240ad990 0x7f6424153520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:49.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.613+0000 7f6429475700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64241530c0 con 0x7f64240ad990 2022-06-29T15:58:49.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.613+0000 7f6429475700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642412b310 con 0x7f64240657d0 2022-06-29T15:58:49.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.613+0000 7f6429475700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f642412b070 con 0x7f64240acd60 2022-06-29T15:58:49.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.617+0000 7f6428c74700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64240ad990 0x7f6424153520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:49.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.617+0000 7f6423fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64240657d0 0x7f64240b6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:49.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.617+0000 7f641bfff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64240acd60 0x7f642414fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:49.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.617+0000 7f6428c74700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64240ad990 0x7f6424153520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45924/0 (socket says 172.21.15.60:45924) 2022-06-29T15:58:49.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.617+0000 7f6428c74700 1 -- 172.21.15.60:0/830086135 learned_addr learned my addr 172.21.15.60:0/830086135 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:49.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.617+0000 7f6423fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64240657d0 0x7f64240b6600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:51950/0 (socket says 172.21.15.60:51950) 2022-06-29T15:58:49.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.617+0000 7f6428c74700 1 -- 172.21.15.60:0/830086135 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64240acd60 msgr2=0x7f642414fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:49.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.617+0000 7f6428c74700 1 --2- 172.21.15.60:0/830086135 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64240acd60 0x7f642414fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:49.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.617+0000 7f6428c74700 1 -- 172.21.15.60:0/830086135 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64240657d0 msgr2=0x7f64240b6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:49.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.617+0000 7f6428c74700 1 --2- 172.21.15.60:0/830086135 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64240657d0 0x7f64240b6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:49.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.617+0000 7f6428c74700 1 -- 172.21.15.60:0/830086135 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f641c009470 con 0x7f64240ad990 2022-06-29T15:58:49.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.617+0000 7f6423fff700 1 --2- 172.21.15.60:0/830086135 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64240657d0 0x7f64240b6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:49.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.617+0000 7f6428c74700 1 --2- 172.21.15.60:0/830086135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64240ad990 0x7f6424153520 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f641c0032f0 tx=0x7f641c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:49.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.617+0000 7f6421ffb700 1 -- 172.21.15.60:0/830086135 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f641c021630 con 0x7f64240ad990 2022-06-29T15:58:49.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.617+0000 7f6421ffb700 1 -- 172.21.15.60:0/830086135 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f641c021790 con 0x7f64240ad990 2022-06-29T15:58:49.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.617+0000 7f6429475700 1 -- 172.21.15.60:0/830086135 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f642414fef0 con 0x7f64240ad990 2022-06-29T15:58:49.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.617+0000 7f6421ffb700 1 -- 172.21.15.60:0/830086135 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f641c021af0 con 0x7f64240ad990 2022-06-29T15:58:49.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.617+0000 7f6429475700 1 -- 172.21.15.60:0/830086135 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64240b0ae0 con 0x7f64240ad990 2022-06-29T15:58:49.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.617+0000 7f6429475700 1 -- 172.21.15.60:0/830086135 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6424153230 con 0x7f64240ad990 2022-06-29T15:58:49.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.617+0000 7f6421ffb700 1 -- 172.21.15.60:0/830086135 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f641c00f070 con 0x7f64240ad990 2022-06-29T15:58:49.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.621+0000 7f6421ffb700 1 --2- 172.21.15.60:0/830086135 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6408053a70 0x7f6408055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:49.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.621+0000 7f6421ffb700 1 -- 172.21.15.60:0/830086135 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f641c06dad0 con 0x7f64240ad990 2022-06-29T15:58:49.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.621+0000 7f6423fff700 1 --2- 172.21.15.60:0/830086135 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6408053a70 0x7f6408055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:49.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.621+0000 7f6423fff700 1 --2- 172.21.15.60:0/830086135 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6408053a70 0x7f6408055f30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f641400c000 tx=0x7f6414008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:49.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.629+0000 7f6421ffb700 1 -- 172.21.15.60:0/830086135 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f641c0448e0 con 0x7f64240ad990 2022-06-29T15:58:49.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.757+0000 7f6429475700 1 -- 172.21.15.60:0/830086135 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f64240573d0 con 0x7f64240ad990 2022-06-29T15:58:49.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.757+0000 7f6421ffb700 1 -- 172.21.15.60:0/830086135 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f641c015070 con 0x7f64240ad990 2022-06-29T15:58:49.765 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T15:58:49.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.761+0000 7f6429475700 1 -- 172.21.15.60:0/830086135 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6408053a70 msgr2=0x7f6408055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:49.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.761+0000 7f6429475700 1 --2- 172.21.15.60:0/830086135 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6408053a70 0x7f6408055f30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f641400c000 tx=0x7f6414008040).stop 2022-06-29T15:58:49.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.761+0000 7f6429475700 1 -- 172.21.15.60:0/830086135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64240ad990 msgr2=0x7f6424153520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:49.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.761+0000 7f6429475700 1 --2- 172.21.15.60:0/830086135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64240ad990 0x7f6424153520 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f641c0032f0 tx=0x7f641c016000).stop 2022-06-29T15:58:49.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.761+0000 7f6429475700 1 -- 172.21.15.60:0/830086135 shutdown_connections 2022-06-29T15:58:49.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.761+0000 7f6429475700 1 --2- 172.21.15.60:0/830086135 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6408053a70 0x7f6408055f30 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:49.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.761+0000 7f6429475700 1 --2- 172.21.15.60:0/830086135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64240ad990 0x7f6424153520 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:49.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.761+0000 7f6429475700 1 --2- 172.21.15.60:0/830086135 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64240acd60 0x7f642414fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:49.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.761+0000 7f6429475700 1 --2- 172.21.15.60:0/830086135 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64240657d0 0x7f64240b6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:49.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.761+0000 7f6429475700 1 -- 172.21.15.60:0/830086135 >> 172.21.15.60:0/830086135 conn(0x7f64240b6f80 msgr2=0x7f64241288f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:49.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.761+0000 7f6429475700 1 -- 172.21.15.60:0/830086135 shutdown_connections 2022-06-29T15:58:49.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.761+0000 7f6429475700 1 -- 172.21.15.60:0/830086135 wait complete. 2022-06-29T15:58:49.777 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-29T15:58:49.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.881+0000 7f70295b4700 1 Processor -- start 2022-06-29T15:58:49.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.881+0000 7f70295b4700 1 -- start start 2022-06-29T15:58:49.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.885+0000 7f70295b4700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70240ad990 0x7f7024144900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:49.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.885+0000 7f70295b4700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f70240657d0 0x7f7024129e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:49.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.885+0000 7f70295b4700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7024143cf0 con 0x7f70240acd60 2022-06-29T15:58:49.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.885+0000 7f70295b4700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7024064a70 con 0x7f70240657d0 2022-06-29T15:58:49.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.885+0000 7f70295b4700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70240ad140 con 0x7f70240ad990 2022-06-29T15:58:49.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.885+0000 7f7028db3700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f70240657d0 0x7f7024129e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:49.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.885+0000 7f7023fff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f70240acd60 0x7f70241296f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54014/0 (socket says 172.21.15.60:54014) 2022-06-29T15:58:49.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.885+0000 7f7023fff700 1 -- 172.21.15.60:0/2863050792 learned_addr learned my addr 172.21.15.60:0/2863050792 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:49.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.885+0000 7f70237fe700 1 --2- 172.21.15.60:0/2863050792 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70240ad990 0x7f7024144900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:49.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.885+0000 7f7028db3700 1 -- 172.21.15.60:0/2863050792 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70240ad990 msgr2=0x7f7024144900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:49.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.885+0000 7f7028db3700 1 --2- 172.21.15.60:0/2863050792 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70240ad990 0x7f7024144900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:49.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.885+0000 7f7028db3700 1 -- 172.21.15.60:0/2863050792 >> v1:172.21.15.60:6789/0 conn(0x7f70240acd60 legacy=0x7f70241296f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:49.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.885+0000 7f7028db3700 1 -- 172.21.15.60:0/2863050792 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7024143f20 con 0x7f70240657d0 2022-06-29T15:58:49.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.885+0000 7f7028db3700 1 --2- 172.21.15.60:0/2863050792 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70240657d0 0x7f7024129e00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f7014002f70 tx=0x7f7014006df0).ready entity=mon.0 client_cookie=e226cc79b9dd01bb server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:49.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.885+0000 7f7022ffd700 1 -- 172.21.15.60:0/2863050792 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f701400d570 con 0x7f70240657d0 2022-06-29T15:58:49.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.885+0000 7f7022ffd700 1 -- 172.21.15.60:0/2863050792 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7014000e90 con 0x7f70240657d0 2022-06-29T15:58:49.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.885+0000 7f7022ffd700 1 -- 172.21.15.60:0/2863050792 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f701400d8c0 con 0x7f70240657d0 2022-06-29T15:58:49.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.885+0000 7f70295b4700 1 -- 172.21.15.60:0/2863050792 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70240657d0 msgr2=0x7f7024129e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:49.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.885+0000 7f70295b4700 1 --2- 172.21.15.60:0/2863050792 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70240657d0 0x7f7024129e00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f7014002f70 tx=0x7f7014006df0).stop 2022-06-29T15:58:49.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.885+0000 7f70295b4700 1 -- 172.21.15.60:0/2863050792 shutdown_connections 2022-06-29T15:58:49.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.885+0000 7f70295b4700 1 --2- 172.21.15.60:0/2863050792 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70240657d0 0x7f7024129e00 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:49.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.885+0000 7f70295b4700 1 --2- 172.21.15.60:0/2863050792 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70240ad990 0x7f7024144900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:49.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.885+0000 7f70295b4700 1 -- 172.21.15.60:0/2863050792 >> 172.21.15.60:0/2863050792 conn(0x7f70240b6f80 msgr2=0x7f70240b7390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:49.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.885+0000 7f70295b4700 1 -- 172.21.15.60:0/2863050792 shutdown_connections 2022-06-29T15:58:49.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.885+0000 7f70295b4700 1 -- 172.21.15.60:0/2863050792 wait complete. 2022-06-29T15:58:49.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.889+0000 7f70295b4700 1 Processor -- start 2022-06-29T15:58:49.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.889+0000 7f70295b4700 1 -- start start 2022-06-29T15:58:49.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.889+0000 7f70295b4700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70240657d0 0x7f7024126bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:49.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.889+0000 7f70295b4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70240acd60 0x7f7024153160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:49.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.889+0000 7f70295b4700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70240ad990 0x7f70240aff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:49.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.889+0000 7f70295b4700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7024111020 con 0x7f70240acd60 2022-06-29T15:58:49.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.889+0000 7f70295b4700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70240b0480 con 0x7f70240657d0 2022-06-29T15:58:49.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.889+0000 7f70295b4700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70240b06c0 con 0x7f70240ad990 2022-06-29T15:58:49.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.889+0000 7f70237fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70240acd60 0x7f7024153160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:49.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.889+0000 7f70237fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70240acd60 0x7f7024153160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45938/0 (socket says 172.21.15.60:45938) 2022-06-29T15:58:49.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.889+0000 7f70237fe700 1 -- 172.21.15.60:0/1896419396 learned_addr learned my addr 172.21.15.60:0/1896419396 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:49.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.889+0000 7f7028db3700 1 --2- 172.21.15.60:0/1896419396 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70240ad990 0x7f70240aff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:49.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.889+0000 7f7023fff700 1 --2- 172.21.15.60:0/1896419396 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70240657d0 0x7f7024126bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:49.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.889+0000 7f70237fe700 1 -- 172.21.15.60:0/1896419396 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70240ad990 msgr2=0x7f70240aff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:49.915 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.889+0000 7f70237fe700 1 --2- 172.21.15.60:0/1896419396 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70240ad990 0x7f70240aff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:49.915 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.889+0000 7f70237fe700 1 -- 172.21.15.60:0/1896419396 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70240657d0 msgr2=0x7f7024126bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:49.915 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.889+0000 7f70237fe700 1 --2- 172.21.15.60:0/1896419396 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70240657d0 0x7f7024126bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:49.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.889+0000 7f70237fe700 1 -- 172.21.15.60:0/1896419396 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f701400b180 con 0x7f70240acd60 2022-06-29T15:58:49.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.889+0000 7f70237fe700 1 --2- 172.21.15.60:0/1896419396 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70240acd60 0x7f7024153160 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f700c00c000 tx=0x7f700c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:49.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.889+0000 7f70217fa700 1 -- 172.21.15.60:0/1896419396 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f700c012600 con 0x7f70240acd60 2022-06-29T15:58:49.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.889+0000 7f70217fa700 1 -- 172.21.15.60:0/1896419396 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f700c012760 con 0x7f70240acd60 2022-06-29T15:58:49.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.889+0000 7f70295b4700 1 -- 172.21.15.60:0/1896419396 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70240b0960 con 0x7f70240acd60 2022-06-29T15:58:49.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.889+0000 7f70217fa700 1 -- 172.21.15.60:0/1896419396 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f700c012ac0 con 0x7f70240acd60 2022-06-29T15:58:49.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.889+0000 7f70295b4700 1 -- 172.21.15.60:0/1896419396 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70240afad0 con 0x7f70240acd60 2022-06-29T15:58:49.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.893+0000 7f70295b4700 1 -- 172.21.15.60:0/1896419396 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70240573d0 con 0x7f70240acd60 2022-06-29T15:58:49.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.893+0000 7f70217fa700 1 -- 172.21.15.60:0/1896419396 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f700c018020 con 0x7f70240acd60 2022-06-29T15:58:49.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.893+0000 7f70217fa700 1 --2- 172.21.15.60:0/1896419396 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6ffc053b00 0x7f6ffc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:49.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.893+0000 7f70217fa700 1 -- 172.21.15.60:0/1896419396 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f700c06c550 con 0x7f70240acd60 2022-06-29T15:58:49.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.893+0000 7f7023fff700 1 --2- 172.21.15.60:0/1896419396 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6ffc053b00 0x7f6ffc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:49.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.893+0000 7f7023fff700 1 --2- 172.21.15.60:0/1896419396 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6ffc053b00 0x7f6ffc055fc0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f701c00d000 tx=0x7f701c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:49.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:49.897+0000 7f70217fa700 1 -- 172.21.15.60:0/1896419396 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f700c0432e0 con 0x7f70240acd60 2022-06-29T15:58:50.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.041+0000 7f70295b4700 1 -- 172.21.15.60:0/1896419396 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f70240b7de0 con 0x7f70240acd60 2022-06-29T15:58:50.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.045+0000 7f70217fa700 1 -- 172.21.15.60:0/1896419396 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7f700c046900 con 0x7f70240acd60 2022-06-29T15:58:50.050 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T15:58:50.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.045+0000 7f70295b4700 1 -- 172.21.15.60:0/1896419396 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6ffc053b00 msgr2=0x7f6ffc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:50.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.045+0000 7f70295b4700 1 --2- 172.21.15.60:0/1896419396 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6ffc053b00 0x7f6ffc055fc0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f701c00d000 tx=0x7f701c009040).stop 2022-06-29T15:58:50.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.045+0000 7f70295b4700 1 -- 172.21.15.60:0/1896419396 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70240acd60 msgr2=0x7f7024153160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:50.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.045+0000 7f70295b4700 1 --2- 172.21.15.60:0/1896419396 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70240acd60 0x7f7024153160 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f700c00c000 tx=0x7f700c0091d0).stop 2022-06-29T15:58:50.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.045+0000 7f70295b4700 1 -- 172.21.15.60:0/1896419396 shutdown_connections 2022-06-29T15:58:50.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.045+0000 7f70295b4700 1 --2- 172.21.15.60:0/1896419396 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6ffc053b00 0x7f6ffc055fc0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.045+0000 7f70295b4700 1 --2- 172.21.15.60:0/1896419396 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70240ad990 0x7f70240aff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.045+0000 7f70295b4700 1 --2- 172.21.15.60:0/1896419396 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70240acd60 0x7f7024153160 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.045+0000 7f70295b4700 1 --2- 172.21.15.60:0/1896419396 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70240657d0 0x7f7024126bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.045+0000 7f70295b4700 1 -- 172.21.15.60:0/1896419396 >> 172.21.15.60:0/1896419396 conn(0x7f70240b6f80 msgr2=0x7f70240667c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:50.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.045+0000 7f70295b4700 1 -- 172.21.15.60:0/1896419396 shutdown_connections 2022-06-29T15:58:50.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.045+0000 7f70295b4700 1 -- 172.21.15.60:0/1896419396 wait complete. 2022-06-29T15:58:50.062 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-29T15:58:50.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.169+0000 7f0d4b59e700 1 Processor -- start 2022-06-29T15:58:50.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.169+0000 7f0d4b59e700 1 -- start start 2022-06-29T15:58:50.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.169+0000 7f0d4b59e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0d4c05c5b0 0x7f0d4c05c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:50.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.169+0000 7f0d4b59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d4c05f620 0x7f0d4c146990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:50.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.169+0000 7f0d4b59e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d4c146e80 con 0x7f0d4c0b0c80 2022-06-29T15:58:50.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.169+0000 7f0d4b59e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d4c145fb0 con 0x7f0d4c05c5b0 2022-06-29T15:58:50.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.169+0000 7f0d4b59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c146250 con 0x7f0d4c05f620 2022-06-29T15:58:50.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.169+0000 7f0d4a59c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0d4c05c5b0 0x7f0d4c05c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:50.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.169+0000 7f0d4a59c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0d4c05c5b0 0x7f0d4c05c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45942/0 (socket says 172.21.15.60:45942) 2022-06-29T15:58:50.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.169+0000 7f0d4a59c700 1 -- 172.21.15.60:0/2267525535 learned_addr learned my addr 172.21.15.60:0/2267525535 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:50.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.169+0000 7f0d49d9b700 1 --1- 172.21.15.60:0/2267525535 >> v1:172.21.15.60:6789/0 conn(0x7f0d4c0b0c80 0x7f0d4c13c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54030/0 (socket says 172.21.15.60:54030) 2022-06-29T15:58:50.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.169+0000 7f0d4ad9d700 1 --2- 172.21.15.60:0/2267525535 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d4c05f620 0x7f0d4c146990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:50.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.169+0000 7f0d4a59c700 1 -- 172.21.15.60:0/2267525535 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d4c05f620 msgr2=0x7f0d4c146990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:50.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.169+0000 7f0d4a59c700 1 --2- 172.21.15.60:0/2267525535 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d4c05f620 0x7f0d4c146990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.169+0000 7f0d4959a700 1 -- 172.21.15.60:0/2267525535 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 671535284 0 0) 0x7f0d4c146e80 con 0x7f0d4c0b0c80 2022-06-29T15:58:50.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.169+0000 7f0d4a59c700 1 -- 172.21.15.60:0/2267525535 >> v1:172.21.15.60:6789/0 conn(0x7f0d4c0b0c80 legacy=0x7f0d4c13c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:50.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.169+0000 7f0d4a59c700 1 -- 172.21.15.60:0/2267525535 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d4c1464f0 con 0x7f0d4c05c5b0 2022-06-29T15:58:50.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.169+0000 7f0d4a59c700 1 --2- 172.21.15.60:0/2267525535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d4c05c5b0 0x7f0d4c05c990 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f0d3c002f70 tx=0x7f0d3c006eb0).ready entity=mon.0 client_cookie=75d285e1c95f1a6e server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:50.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.173+0000 7f0d4959a700 1 -- 172.21.15.60:0/2267525535 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d3c00d570 con 0x7f0d4c05c5b0 2022-06-29T15:58:50.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.173+0000 7f0d4959a700 1 -- 172.21.15.60:0/2267525535 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d3c000e90 con 0x7f0d4c05c5b0 2022-06-29T15:58:50.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.173+0000 7f0d4959a700 1 -- 172.21.15.60:0/2267525535 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d3c00d8c0 con 0x7f0d4c05c5b0 2022-06-29T15:58:50.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.173+0000 7f0d4b59e700 1 -- 172.21.15.60:0/2267525535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d4c05c5b0 msgr2=0x7f0d4c05c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:50.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.173+0000 7f0d4b59e700 1 --2- 172.21.15.60:0/2267525535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d4c05c5b0 0x7f0d4c05c990 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f0d3c002f70 tx=0x7f0d3c006eb0).stop 2022-06-29T15:58:50.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.173+0000 7f0d4b59e700 1 -- 172.21.15.60:0/2267525535 shutdown_connections 2022-06-29T15:58:50.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.173+0000 7f0d4b59e700 1 --2- 172.21.15.60:0/2267525535 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d4c05f620 0x7f0d4c146990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.173+0000 7f0d4b59e700 1 --2- 172.21.15.60:0/2267525535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d4c05c5b0 0x7f0d4c05c990 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.173+0000 7f0d4b59e700 1 -- 172.21.15.60:0/2267525535 >> 172.21.15.60:0/2267525535 conn(0x7f0d4c0637c0 msgr2=0x7f0d4c063ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:50.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.173+0000 7f0d4b59e700 1 -- 172.21.15.60:0/2267525535 shutdown_connections 2022-06-29T15:58:50.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.173+0000 7f0d4b59e700 1 -- 172.21.15.60:0/2267525535 wait complete. 2022-06-29T15:58:50.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.173+0000 7f0d4b59e700 1 Processor -- start 2022-06-29T15:58:50.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.173+0000 7f0d4b59e700 1 -- start start 2022-06-29T15:58:50.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.177+0000 7f0d4b59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d4c05c5b0 0x7f0d4c0b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:50.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.177+0000 7f0d4b59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d4c05f620 0x7f0d4c143820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:50.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.177+0000 7f0d4b59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d4c0b0c80 0x7f0d4c155210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:50.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.177+0000 7f0d4b59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c142a00 con 0x7f0d4c0b0c80 2022-06-29T15:58:50.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.177+0000 7f0d4b59e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c0afc50 con 0x7f0d4c05c5b0 2022-06-29T15:58:50.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.177+0000 7f0d4b59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c0b3e30 con 0x7f0d4c05f620 2022-06-29T15:58:50.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.177+0000 7f0d4ad9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d4c0b0c80 0x7f0d4c155210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:50.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.177+0000 7f0d4ad9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d4c0b0c80 0x7f0d4c155210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45952/0 (socket says 172.21.15.60:45952) 2022-06-29T15:58:50.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.177+0000 7f0d4ad9d700 1 -- 172.21.15.60:0/674589825 learned_addr learned my addr 172.21.15.60:0/674589825 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:50.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.177+0000 7f0d49d9b700 1 --2- 172.21.15.60:0/674589825 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d4c05f620 0x7f0d4c143820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:50.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.177+0000 7f0d4a59c700 1 --2- 172.21.15.60:0/674589825 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d4c05c5b0 0x7f0d4c0b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:50.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.177+0000 7f0d4ad9d700 1 -- 172.21.15.60:0/674589825 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d4c05f620 msgr2=0x7f0d4c143820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:50.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.177+0000 7f0d4ad9d700 1 --2- 172.21.15.60:0/674589825 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d4c05f620 0x7f0d4c143820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.177+0000 7f0d4ad9d700 1 -- 172.21.15.60:0/674589825 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d4c05c5b0 msgr2=0x7f0d4c0b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:50.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.177+0000 7f0d4ad9d700 1 --2- 172.21.15.60:0/674589825 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d4c05c5b0 0x7f0d4c0b42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.177+0000 7f0d4ad9d700 1 -- 172.21.15.60:0/674589825 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d3c00b1e0 con 0x7f0d4c0b0c80 2022-06-29T15:58:50.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.177+0000 7f0d4a59c700 1 --2- 172.21.15.60:0/674589825 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d4c05c5b0 0x7f0d4c0b42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:58:50.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.177+0000 7f0d4ad9d700 1 --2- 172.21.15.60:0/674589825 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d4c0b0c80 0x7f0d4c155210 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f0d44006e00 tx=0x7f0d440091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:50.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.177+0000 7f0d437fe700 1 -- 172.21.15.60:0/674589825 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d4400eeb0 con 0x7f0d4c0b0c80 2022-06-29T15:58:50.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.177+0000 7f0d437fe700 1 -- 172.21.15.60:0/674589825 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d4400c850 con 0x7f0d4c0b0c80 2022-06-29T15:58:50.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.177+0000 7f0d4b59e700 1 -- 172.21.15.60:0/674589825 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d4c140780 con 0x7f0d4c0b0c80 2022-06-29T15:58:50.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.177+0000 7f0d437fe700 1 -- 172.21.15.60:0/674589825 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d4400cbb0 con 0x7f0d4c0b0c80 2022-06-29T15:58:50.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.177+0000 7f0d4b59e700 1 -- 172.21.15.60:0/674589825 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d4c0af960 con 0x7f0d4c0b0c80 2022-06-29T15:58:50.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.181+0000 7f0d4b59e700 1 -- 172.21.15.60:0/674589825 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d4c0afe10 con 0x7f0d4c0b0c80 2022-06-29T15:58:50.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.181+0000 7f0d437fe700 1 -- 172.21.15.60:0/674589825 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0d4400cd10 con 0x7f0d4c0b0c80 2022-06-29T15:58:50.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.181+0000 7f0d437fe700 1 --2- 172.21.15.60:0/674589825 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0d30053a70 0x7f0d30055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:50.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.181+0000 7f0d437fe700 1 -- 172.21.15.60:0/674589825 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0d4406c1b0 con 0x7f0d4c0b0c80 2022-06-29T15:58:50.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.181+0000 7f0d4a59c700 1 --2- 172.21.15.60:0/674589825 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0d30053a70 0x7f0d30055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:50.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.181+0000 7f0d4a59c700 1 --2- 172.21.15.60:0/674589825 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0d30053a70 0x7f0d30055f30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f0d3c01a000 tx=0x7f0d3c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:50.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.189+0000 7f0d437fe700 1 -- 172.21.15.60:0/674589825 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0d440430d0 con 0x7f0d4c0b0c80 2022-06-29T15:58:50.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.313+0000 7f0d4b59e700 1 -- 172.21.15.60:0/674589825 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0d4c1413c0 con 0x7f0d4c0b0c80 2022-06-29T15:58:50.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.313+0000 7f0d437fe700 1 -- 172.21.15.60:0/674589825 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7f0d4c1413c0 con 0x7f0d4c0b0c80 2022-06-29T15:58:50.318 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T15:58:50.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.313+0000 7f0d4b59e700 1 -- 172.21.15.60:0/674589825 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0d30053a70 msgr2=0x7f0d30055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:50.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.313+0000 7f0d4b59e700 1 --2- 172.21.15.60:0/674589825 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0d30053a70 0x7f0d30055f30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f0d3c01a000 tx=0x7f0d3c015000).stop 2022-06-29T15:58:50.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.313+0000 7f0d4b59e700 1 -- 172.21.15.60:0/674589825 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d4c0b0c80 msgr2=0x7f0d4c155210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:50.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.313+0000 7f0d4b59e700 1 --2- 172.21.15.60:0/674589825 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d4c0b0c80 0x7f0d4c155210 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f0d44006e00 tx=0x7f0d440091d0).stop 2022-06-29T15:58:50.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.313+0000 7f0d4b59e700 1 -- 172.21.15.60:0/674589825 shutdown_connections 2022-06-29T15:58:50.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.313+0000 7f0d4b59e700 1 --2- 172.21.15.60:0/674589825 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0d30053a70 0x7f0d30055f30 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.313+0000 7f0d4b59e700 1 --2- 172.21.15.60:0/674589825 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d4c0b0c80 0x7f0d4c155210 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.313+0000 7f0d4b59e700 1 --2- 172.21.15.60:0/674589825 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d4c05f620 0x7f0d4c143820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.313+0000 7f0d4b59e700 1 --2- 172.21.15.60:0/674589825 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d4c05c5b0 0x7f0d4c0b42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.313+0000 7f0d4b59e700 1 -- 172.21.15.60:0/674589825 >> 172.21.15.60:0/674589825 conn(0x7f0d4c0637c0 msgr2=0x7f0d4c0b5150 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:50.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.313+0000 7f0d4b59e700 1 -- 172.21.15.60:0/674589825 shutdown_connections 2022-06-29T15:58:50.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.313+0000 7f0d4b59e700 1 -- 172.21.15.60:0/674589825 wait complete. 2022-06-29T15:58:50.331 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T15:58:50.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.425+0000 7f580a038700 1 Processor -- start 2022-06-29T15:58:50.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.425+0000 7f580a038700 1 -- start start 2022-06-29T15:58:50.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.429+0000 7f580a038700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f58040b5950 0x7f5804067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:50.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.429+0000 7f580a038700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5804063ad0 0x7f580414af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:50.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.429+0000 7f580a038700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58040acba0 con 0x7f580405bd20 2022-06-29T15:58:50.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.429+0000 7f580a038700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5804067220 con 0x7f5804063ad0 2022-06-29T15:58:50.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.429+0000 7f580a038700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58040b5d30 con 0x7f58040b5950 2022-06-29T15:58:50.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.429+0000 7f5802ffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5804063ad0 0x7f580414af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:50.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.429+0000 7f5803fff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f580405bd20 0x7f580405c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54046/0 (socket says 172.21.15.60:54046) 2022-06-29T15:58:50.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.429+0000 7f5803fff700 1 -- 172.21.15.60:0/2364216146 learned_addr learned my addr 172.21.15.60:0/2364216146 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:50.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.429+0000 7f58037fe700 1 --2- 172.21.15.60:0/2364216146 >> v2:172.21.15.92:3300/0 conn(0x7f58040b5950 0x7f5804067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:50.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.429+0000 7f58027fc700 1 -- 172.21.15.60:0/2364216146 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3984279415 0 0) 0x7f58040acba0 con 0x7f580405bd20 2022-06-29T15:58:50.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.429+0000 7f58027fc700 1 -- 172.21.15.60:0/2364216146 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57e4003260 con 0x7f580405bd20 2022-06-29T15:58:50.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.429+0000 7f5802ffd700 1 -- 172.21.15.60:0/2364216146 >> v2:172.21.15.92:3300/0 conn(0x7f58040b5950 msgr2=0x7f5804067680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:50.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.429+0000 7f5802ffd700 1 --2- 172.21.15.60:0/2364216146 >> v2:172.21.15.92:3300/0 conn(0x7f58040b5950 0x7f5804067680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.429+0000 7f5802ffd700 1 -- 172.21.15.60:0/2364216146 >> v1:172.21.15.60:6789/0 conn(0x7f580405bd20 legacy=0x7f580405c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:50.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.429+0000 7f5802ffd700 1 -- 172.21.15.60:0/2364216146 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58040ace00 con 0x7f5804063ad0 2022-06-29T15:58:50.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.429+0000 7f5802ffd700 1 --2- 172.21.15.60:0/2364216146 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5804063ad0 0x7f580414af10 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f57ec002f70 tx=0x7f57ec006df0).ready entity=mon.0 client_cookie=3d792ff5c8938177 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:50.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.429+0000 7f58027fc700 1 -- 172.21.15.60:0/2364216146 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57ec00d4f0 con 0x7f5804063ad0 2022-06-29T15:58:50.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.429+0000 7f58027fc700 1 -- 172.21.15.60:0/2364216146 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57ec000e90 con 0x7f5804063ad0 2022-06-29T15:58:50.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.429+0000 7f58027fc700 1 -- 172.21.15.60:0/2364216146 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57ec00d840 con 0x7f5804063ad0 2022-06-29T15:58:50.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.429+0000 7f580a038700 1 -- 172.21.15.60:0/2364216146 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5804063ad0 msgr2=0x7f580414af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:50.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.429+0000 7f580a038700 1 --2- 172.21.15.60:0/2364216146 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5804063ad0 0x7f580414af10 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f57ec002f70 tx=0x7f57ec006df0).stop 2022-06-29T15:58:50.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.429+0000 7f580a038700 1 -- 172.21.15.60:0/2364216146 shutdown_connections 2022-06-29T15:58:50.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.429+0000 7f580a038700 1 --2- 172.21.15.60:0/2364216146 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5804063ad0 0x7f580414af10 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.448 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.429+0000 7f580a038700 1 --2- 172.21.15.60:0/2364216146 >> v2:172.21.15.92:3300/0 conn(0x7f58040b5950 0x7f5804067680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.448 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.429+0000 7f580a038700 1 -- 172.21.15.60:0/2364216146 >> 172.21.15.60:0/2364216146 conn(0x7f58040632c0 msgr2=0x7f58040b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:50.448 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.429+0000 7f580a038700 1 -- 172.21.15.60:0/2364216146 shutdown_connections 2022-06-29T15:58:50.448 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.429+0000 7f580a038700 1 -- 172.21.15.60:0/2364216146 wait complete. 2022-06-29T15:58:50.449 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.433+0000 7f580a038700 1 Processor -- start 2022-06-29T15:58:50.449 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.433+0000 7f580a038700 1 -- start start 2022-06-29T15:58:50.449 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.433+0000 7f580a038700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f580405bd20 0x7f58040b3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:50.449 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.433+0000 7f580a038700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5804063ad0 0x7f58040af070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:50.449 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.433+0000 7f580a038700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f58040b5950 0x7f58041568a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:50.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.433+0000 7f580a038700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5804117ac0 con 0x7f580405bd20 2022-06-29T15:58:50.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.433+0000 7f580a038700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5804156160 con 0x7f58040b5950 2022-06-29T15:58:50.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.433+0000 7f580a038700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5804156400 con 0x7f5804063ad0 2022-06-29T15:58:50.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.433+0000 7f58037fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f580405bd20 0x7f58040b3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:50.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.433+0000 7f58037fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f580405bd20 0x7f58040b3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45962/0 (socket says 172.21.15.60:45962) 2022-06-29T15:58:50.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.433+0000 7f58037fe700 1 -- 172.21.15.60:0/484942767 learned_addr learned my addr 172.21.15.60:0/484942767 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:50.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.433+0000 7f5802ffd700 1 --2- 172.21.15.60:0/484942767 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5804063ad0 0x7f58040af070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:50.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.433+0000 7f5803fff700 1 --2- 172.21.15.60:0/484942767 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f58040b5950 0x7f58041568a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:50.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.433+0000 7f58037fe700 1 -- 172.21.15.60:0/484942767 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5804063ad0 msgr2=0x7f58040af070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:50.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.433+0000 7f58037fe700 1 --2- 172.21.15.60:0/484942767 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5804063ad0 0x7f58040af070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.433+0000 7f58037fe700 1 -- 172.21.15.60:0/484942767 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f58040b5950 msgr2=0x7f58041568a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:50.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.433+0000 7f58037fe700 1 --2- 172.21.15.60:0/484942767 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f58040b5950 0x7f58041568a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.433+0000 7f58037fe700 1 -- 172.21.15.60:0/484942767 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57ec00b180 con 0x7f580405bd20 2022-06-29T15:58:50.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.433+0000 7f5802ffd700 1 --2- 172.21.15.60:0/484942767 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5804063ad0 0x7f58040af070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:58:50.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.437+0000 7f58037fe700 1 --2- 172.21.15.60:0/484942767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f580405bd20 0x7f58040b3cf0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f57fc00c000 tx=0x7f57fc0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:50.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.437+0000 7f5800ff9700 1 -- 172.21.15.60:0/484942767 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57fc00c6a0 con 0x7f580405bd20 2022-06-29T15:58:50.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.437+0000 7f5800ff9700 1 -- 172.21.15.60:0/484942767 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57fc00c800 con 0x7f580405bd20 2022-06-29T15:58:50.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.437+0000 7f580a038700 1 -- 172.21.15.60:0/484942767 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58040b41e0 con 0x7f580405bd20 2022-06-29T15:58:50.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.437+0000 7f5800ff9700 1 -- 172.21.15.60:0/484942767 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57fc00cb60 con 0x7f580405bd20 2022-06-29T15:58:50.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.437+0000 7f580a038700 1 -- 172.21.15.60:0/484942767 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58040af560 con 0x7f580405bd20 2022-06-29T15:58:50.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.437+0000 7f580a038700 1 -- 172.21.15.60:0/484942767 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58041565c0 con 0x7f580405bd20 2022-06-29T15:58:50.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.437+0000 7f5800ff9700 1 -- 172.21.15.60:0/484942767 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f57fc022070 con 0x7f580405bd20 2022-06-29T15:58:50.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.437+0000 7f5800ff9700 1 --2- 172.21.15.60:0/484942767 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f57e8053a70 0x7f57e8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:50.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.437+0000 7f5800ff9700 1 -- 172.21.15.60:0/484942767 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f57fc06c640 con 0x7f580405bd20 2022-06-29T15:58:50.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.437+0000 7f5802ffd700 1 --2- 172.21.15.60:0/484942767 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f57e8053a70 0x7f57e8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:50.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.441+0000 7f5802ffd700 1 --2- 172.21.15.60:0/484942767 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f57e8053a70 0x7f57e8055f30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f57ec018000 tx=0x7f57ec014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:50.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.441+0000 7f5800ff9700 1 -- 172.21.15.60:0/484942767 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f57fc043450 con 0x7f580405bd20 2022-06-29T15:58:50.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.569+0000 7f580a038700 1 -- 172.21.15.60:0/484942767 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58040573d0 con 0x7f580405bd20 2022-06-29T15:58:50.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.569+0000 7f5800ff9700 1 -- 172.21.15.60:0/484942767 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16038 (secure 0 0 0) 0x7f57fc046a70 con 0x7f580405bd20 2022-06-29T15:58:50.577 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:58:50.577 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":23,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T15:58:26.345644+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-06-29T15:58:22.424395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-06-29T15:58:23.343438+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T15:58:50.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.573+0000 7f580a038700 1 -- 172.21.15.60:0/484942767 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f57e8053a70 msgr2=0x7f57e8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:50.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.573+0000 7f580a038700 1 --2- 172.21.15.60:0/484942767 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f57e8053a70 0x7f57e8055f30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f57ec018000 tx=0x7f57ec014000).stop 2022-06-29T15:58:50.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.573+0000 7f580a038700 1 -- 172.21.15.60:0/484942767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f580405bd20 msgr2=0x7f58040b3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:50.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.573+0000 7f580a038700 1 --2- 172.21.15.60:0/484942767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f580405bd20 0x7f58040b3cf0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f57fc00c000 tx=0x7f57fc0091d0).stop 2022-06-29T15:58:50.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.573+0000 7f580a038700 1 -- 172.21.15.60:0/484942767 shutdown_connections 2022-06-29T15:58:50.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.573+0000 7f580a038700 1 --2- 172.21.15.60:0/484942767 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f57e8053a70 0x7f57e8055f30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.573+0000 7f580a038700 1 --2- 172.21.15.60:0/484942767 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f58040b5950 0x7f58041568a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.573+0000 7f580a038700 1 --2- 172.21.15.60:0/484942767 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5804063ad0 0x7f58040af070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.573+0000 7f580a038700 1 --2- 172.21.15.60:0/484942767 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f580405bd20 0x7f58040b3cf0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.573+0000 7f580a038700 1 -- 172.21.15.60:0/484942767 >> 172.21.15.60:0/484942767 conn(0x7f58040632c0 msgr2=0x7f580414a330 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:50.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.573+0000 7f580a038700 1 -- 172.21.15.60:0/484942767 shutdown_connections 2022-06-29T15:58:50.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.573+0000 7f580a038700 1 -- 172.21.15.60:0/484942767 wait complete. 2022-06-29T15:58:50.590 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-29T15:58:22.424395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-06-29T15:58:23.343438+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-29T15:58:50.590 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T15:58:50.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.693+0000 7f6d5aebf700 1 Processor -- start 2022-06-29T15:58:50.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.693+0000 7f6d5aebf700 1 -- start start 2022-06-29T15:58:50.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.693+0000 7f6d5aebf700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6d540b3d30 0x7f6d54146ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:50.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.693+0000 7f6d5aebf700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6d54061d90 0x7f6d54142b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:50.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.693+0000 7f6d5aebf700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d540b4f50 con 0x7f6d540b9d70 2022-06-29T15:58:50.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.693+0000 7f6d5aebf700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d540ba150 con 0x7f6d540b3d30 2022-06-29T15:58:50.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.693+0000 7f6d5aebf700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d540b5180 con 0x7f6d54061d90 2022-06-29T15:58:50.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.693+0000 7f6d53fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6d540b3d30 0x7f6d54146ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:50.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.693+0000 7f6d58c5b700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f6d540b9d70 0x7f6d541465d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:58304/0 (socket says 172.21.15.60:58304) 2022-06-29T15:58:50.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.693+0000 7f6d53fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6d540b3d30 0x7f6d54146ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45974/0 (socket says 172.21.15.60:45974) 2022-06-29T15:58:50.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.693+0000 7f6d58c5b700 1 -- 172.21.15.60:0/1311772943 learned_addr learned my addr 172.21.15.60:0/1311772943 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:50.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.693+0000 7f6d5945c700 1 --2- 172.21.15.60:0/1311772943 >> v2:172.21.15.92:3300/0 conn(0x7f6d54061d90 0x7f6d54142b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:50.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.693+0000 7f6d53fff700 1 -- 172.21.15.60:0/1311772943 >> v2:172.21.15.92:3300/0 conn(0x7f6d54061d90 msgr2=0x7f6d54142b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:50.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.693+0000 7f6d53fff700 1 --2- 172.21.15.60:0/1311772943 >> v2:172.21.15.92:3300/0 conn(0x7f6d54061d90 0x7f6d54142b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.693+0000 7f6d53fff700 1 -- 172.21.15.60:0/1311772943 >> v1:172.21.15.92:6789/0 conn(0x7f6d540b9d70 legacy=0x7f6d541465d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:50.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.693+0000 7f6d537fe700 1 -- 172.21.15.60:0/1311772943 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4212396250 0 0) 0x7f6d540b4f50 con 0x7f6d540b9d70 2022-06-29T15:58:50.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.693+0000 7f6d53fff700 1 -- 172.21.15.60:0/1311772943 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d541461a0 con 0x7f6d540b3d30 2022-06-29T15:58:50.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.697+0000 7f6d53fff700 1 --2- 172.21.15.60:0/1311772943 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d540b3d30 0x7f6d54146ce0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f6d48002f70 tx=0x7f6d48006df0).ready entity=mon.0 client_cookie=33f035846ed2fe60 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:50.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.697+0000 7f6d537fe700 1 -- 172.21.15.60:0/1311772943 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d4800d570 con 0x7f6d540b3d30 2022-06-29T15:58:50.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.697+0000 7f6d537fe700 1 -- 172.21.15.60:0/1311772943 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d48000e90 con 0x7f6d540b3d30 2022-06-29T15:58:50.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.697+0000 7f6d537fe700 1 -- 172.21.15.60:0/1311772943 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d4800d8c0 con 0x7f6d540b3d30 2022-06-29T15:58:50.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.697+0000 7f6d5aebf700 1 -- 172.21.15.60:0/1311772943 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d540b3d30 msgr2=0x7f6d54146ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:50.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.697+0000 7f6d5aebf700 1 --2- 172.21.15.60:0/1311772943 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d540b3d30 0x7f6d54146ce0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f6d48002f70 tx=0x7f6d48006df0).stop 2022-06-29T15:58:50.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.697+0000 7f6d5aebf700 1 -- 172.21.15.60:0/1311772943 shutdown_connections 2022-06-29T15:58:50.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.697+0000 7f6d5aebf700 1 --2- 172.21.15.60:0/1311772943 >> v2:172.21.15.92:3300/0 conn(0x7f6d54061d90 0x7f6d54142b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.697+0000 7f6d5aebf700 1 --2- 172.21.15.60:0/1311772943 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d540b3d30 0x7f6d54146ce0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.697+0000 7f6d5aebf700 1 -- 172.21.15.60:0/1311772943 >> 172.21.15.60:0/1311772943 conn(0x7f6d54062e70 msgr2=0x7f6d540b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:50.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.697+0000 7f6d5aebf700 1 -- 172.21.15.60:0/1311772943 shutdown_connections 2022-06-29T15:58:50.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.697+0000 7f6d5aebf700 1 -- 172.21.15.60:0/1311772943 wait complete. 2022-06-29T15:58:50.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.697+0000 7f6d5aebf700 1 Processor -- start 2022-06-29T15:58:50.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.697+0000 7f6d5aebf700 1 -- start start 2022-06-29T15:58:50.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.701+0000 7f6d5aebf700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6d54061d90 0x7f6d540ae5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:50.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.701+0000 7f6d5aebf700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d540b3d30 0x7f6d54134860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:50.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.701+0000 7f6d5aebf700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d540b9d70 0x7f6d54152620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:50.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.701+0000 7f6d5aebf700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d54151fd0 con 0x7f6d540b9d70 2022-06-29T15:58:50.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.701+0000 7f6d5aebf700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d54139410 con 0x7f6d540b3d30 2022-06-29T15:58:50.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.701+0000 7f6d5aebf700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d54138580 con 0x7f6d54061d90 2022-06-29T15:58:50.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.701+0000 7f6d5945c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d540b9d70 0x7f6d54152620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:50.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.701+0000 7f6d5945c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d540b9d70 0x7f6d54152620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45980/0 (socket says 172.21.15.60:45980) 2022-06-29T15:58:50.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.701+0000 7f6d5945c700 1 -- 172.21.15.60:0/3717361450 learned_addr learned my addr 172.21.15.60:0/3717361450 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:50.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.701+0000 7f6d58c5b700 1 --2- 172.21.15.60:0/3717361450 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6d54061d90 0x7f6d540ae5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:50.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.701+0000 7f6d53fff700 1 --2- 172.21.15.60:0/3717361450 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d540b3d30 0x7f6d54134860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:50.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.701+0000 7f6d5945c700 1 -- 172.21.15.60:0/3717361450 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6d54061d90 msgr2=0x7f6d540ae5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:50.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.701+0000 7f6d5945c700 1 --2- 172.21.15.60:0/3717361450 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6d54061d90 0x7f6d540ae5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.701+0000 7f6d5945c700 1 -- 172.21.15.60:0/3717361450 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d540b3d30 msgr2=0x7f6d54134860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:50.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.701+0000 7f6d5945c700 1 --2- 172.21.15.60:0/3717361450 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d540b3d30 0x7f6d54134860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.701+0000 7f6d5945c700 1 -- 172.21.15.60:0/3717361450 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d4800b180 con 0x7f6d540b9d70 2022-06-29T15:58:50.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.701+0000 7f6d53fff700 1 --2- 172.21.15.60:0/3717361450 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d540b3d30 0x7f6d54134860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:58:50.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.701+0000 7f6d5945c700 1 --2- 172.21.15.60:0/3717361450 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d540b9d70 0x7f6d54152620 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f6d4c006e00 tx=0x7f6d4c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:50.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.701+0000 7f6d51ffb700 1 -- 172.21.15.60:0/3717361450 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d4c00eeb0 con 0x7f6d540b9d70 2022-06-29T15:58:50.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.701+0000 7f6d51ffb700 1 -- 172.21.15.60:0/3717361450 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d4c00c850 con 0x7f6d540b9d70 2022-06-29T15:58:50.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.701+0000 7f6d5aebf700 1 -- 172.21.15.60:0/3717361450 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d5414fc80 con 0x7f6d540b9d70 2022-06-29T15:58:50.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.701+0000 7f6d51ffb700 1 -- 172.21.15.60:0/3717361450 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d4c00cbb0 con 0x7f6d540b9d70 2022-06-29T15:58:50.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.701+0000 7f6d5aebf700 1 -- 172.21.15.60:0/3717361450 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d54139170 con 0x7f6d540b9d70 2022-06-29T15:58:50.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.701+0000 7f6d51ffb700 1 -- 172.21.15.60:0/3717361450 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6d4c00cd10 con 0x7f6d540b9d70 2022-06-29T15:58:50.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.701+0000 7f6d5aebf700 1 -- 172.21.15.60:0/3717361450 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d540573d0 con 0x7f6d540b9d70 2022-06-29T15:58:50.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.705+0000 7f6d51ffb700 1 --2- 172.21.15.60:0/3717361450 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6d34053a70 0x7f6d34055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:50.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.705+0000 7f6d51ffb700 1 -- 172.21.15.60:0/3717361450 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6d4c06c1c0 con 0x7f6d540b9d70 2022-06-29T15:58:50.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.705+0000 7f6d58c5b700 1 --2- 172.21.15.60:0/3717361450 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6d34053a70 0x7f6d34055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:50.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.705+0000 7f6d58c5b700 1 --2- 172.21.15.60:0/3717361450 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6d34053a70 0x7f6d34055f30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f6d4400d000 tx=0x7f6d44007c20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:50.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.709+0000 7f6d51ffb700 1 -- 172.21.15.60:0/3717361450 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6d4c0430e0 con 0x7f6d540b9d70 2022-06-29T15:58:50.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.833+0000 7f6d5aebf700 1 -- 172.21.15.60:0/3717361450 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f6d540b10e0 con 0x7f6d540b9d70 2022-06-29T15:58:50.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.833+0000 7f6d51ffb700 1 -- 172.21.15.60:0/3717361450 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v23) v1 ==== 110+0+10 (secure 0 0 0) 0x7f6d4c046700 con 0x7f6d540b9d70 2022-06-29T15:58:50.840 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T15:58:50.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.833+0000 7f6d5aebf700 1 -- 172.21.15.60:0/3717361450 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6d34053a70 msgr2=0x7f6d34055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:50.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.833+0000 7f6d5aebf700 1 --2- 172.21.15.60:0/3717361450 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6d34053a70 0x7f6d34055f30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f6d4400d000 tx=0x7f6d44007c20).stop 2022-06-29T15:58:50.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.833+0000 7f6d5aebf700 1 -- 172.21.15.60:0/3717361450 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d540b9d70 msgr2=0x7f6d54152620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:50.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.833+0000 7f6d5aebf700 1 --2- 172.21.15.60:0/3717361450 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d540b9d70 0x7f6d54152620 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f6d4c006e00 tx=0x7f6d4c0091d0).stop 2022-06-29T15:58:50.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.833+0000 7f6d5aebf700 1 -- 172.21.15.60:0/3717361450 shutdown_connections 2022-06-29T15:58:50.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.833+0000 7f6d5aebf700 1 --2- 172.21.15.60:0/3717361450 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6d34053a70 0x7f6d34055f30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.833+0000 7f6d5aebf700 1 --2- 172.21.15.60:0/3717361450 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d540b9d70 0x7f6d54152620 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.843 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.833+0000 7f6d5aebf700 1 --2- 172.21.15.60:0/3717361450 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d540b3d30 0x7f6d54134860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.843 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.833+0000 7f6d5aebf700 1 --2- 172.21.15.60:0/3717361450 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6d54061d90 0x7f6d540ae5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.843 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.833+0000 7f6d5aebf700 1 -- 172.21.15.60:0/3717361450 >> 172.21.15.60:0/3717361450 conn(0x7f6d54062e70 msgr2=0x7f6d54142940 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:50.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.833+0000 7f6d5aebf700 1 -- 172.21.15.60:0/3717361450 shutdown_connections 2022-06-29T15:58:50.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.833+0000 7f6d5aebf700 1 -- 172.21.15.60:0/3717361450 wait complete. 2022-06-29T15:58:50.852 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T15:58:50.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.957+0000 7f9e07fff700 1 Processor -- start 2022-06-29T15:58:50.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.957+0000 7f9e07fff700 1 -- start start 2022-06-29T15:58:50.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.957+0000 7f9e07fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9e0805d070 0x7f9e0814aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:50.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.957+0000 7f9e07fff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e0805d450 con 0x7f9e080b2130 2022-06-29T15:58:50.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.957+0000 7f9e07fff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e080ad840 con 0x7f9e0814af80 2022-06-29T15:58:50.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.957+0000 7f9e07fff700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e080adf90 con 0x7f9e0805d070 2022-06-29T15:58:50.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.957+0000 7f9e067fc700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9e080b2130 0x7f9e080aca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54072/0 (socket says 172.21.15.60:54072) 2022-06-29T15:58:50.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.957+0000 7f9e067fc700 1 -- 172.21.15.60:0/764876654 learned_addr learned my addr 172.21.15.60:0/764876654 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:50.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.957+0000 7f9e06ffd700 1 --2- 172.21.15.60:0/764876654 >> v2:172.21.15.92:3300/0 conn(0x7f9e0805d070 0x7f9e0814aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:50.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.957+0000 7f9e05ffb700 1 -- 172.21.15.60:0/764876654 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2486803841 0 0) 0x7f9e0805d450 con 0x7f9e080b2130 2022-06-29T15:58:50.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.957+0000 7f9e05ffb700 1 -- 172.21.15.60:0/764876654 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9de8003260 con 0x7f9e080b2130 2022-06-29T15:58:50.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e05ffb700 1 -- 172.21.15.60:0/764876654 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3060961378 0 0) 0x7f9e080ad840 con 0x7f9e0814af80 2022-06-29T15:58:50.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e05ffb700 1 -- 172.21.15.60:0/764876654 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e0805d450 con 0x7f9e0814af80 2022-06-29T15:58:50.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e05ffb700 1 -- 172.21.15.60:0/764876654 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1125765752 0 0) 0x7f9de8003260 con 0x7f9e080b2130 2022-06-29T15:58:50.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e05ffb700 1 -- 172.21.15.60:0/764876654 >> v2:172.21.15.92:3300/0 conn(0x7f9e0805d070 msgr2=0x7f9e0814aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:50.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e05ffb700 1 --2- 172.21.15.60:0/764876654 >> v2:172.21.15.92:3300/0 conn(0x7f9e0805d070 0x7f9e0814aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e05ffb700 1 -- 172.21.15.60:0/764876654 >> v1:172.21.15.92:6789/0 conn(0x7f9e0814af80 legacy=0x7f9e080ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:50.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e05ffb700 1 -- 172.21.15.60:0/764876654 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e080b25d0 con 0x7f9e080b2130 2022-06-29T15:58:50.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e05ffb700 1 -- 172.21.15.60:0/764876654 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f9df0001660 con 0x7f9e080b2130 2022-06-29T15:58:50.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e05ffb700 1 -- 172.21.15.60:0/764876654 >> v1:172.21.15.60:6789/0 conn(0x7f9e080b2130 legacy=0x7f9e080aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:50.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e05ffb700 1 --2- 172.21.15.60:0/764876654 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9de80038e0 0x7f9de8003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:50.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e05ffb700 1 --2- 172.21.15.60:0/764876654 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9de8004500 0x7f9de8004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:50.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e05ffb700 1 --2- 172.21.15.60:0/764876654 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9de8004e70 0x7f9de800b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:50.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e05ffb700 1 -- 172.21.15.60:0/764876654 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e080adf90 con 0x7f9de80038e0 2022-06-29T15:58:50.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e05ffb700 1 -- 172.21.15.60:0/764876654 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de800bcf0 con 0x7f9de8004e70 2022-06-29T15:58:50.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e05ffb700 1 -- 172.21.15.60:0/764876654 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de800beb0 con 0x7f9de8004500 2022-06-29T15:58:50.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e06ffd700 1 --2- 172.21.15.60:0/764876654 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9de80038e0 0x7f9de8003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:50.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e06ffd700 1 -- 172.21.15.60:0/764876654 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9de8004500 msgr2=0x7f9de8004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:50.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e06ffd700 1 --2- 172.21.15.60:0/764876654 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9de8004500 0x7f9de8004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-06-29T15:58:50.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e06ffd700 1 -- 172.21.15.60:0/764876654 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9de8004e70 msgr2=0x7f9de800b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:50.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e077fe700 1 --2- 172.21.15.60:0/764876654 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9de8004e70 0x7f9de800b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:50.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e06ffd700 1 --2- 172.21.15.60:0/764876654 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9de8004e70 0x7f9de800b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e06ffd700 1 -- 172.21.15.60:0/764876654 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de8003260 con 0x7f9de80038e0 2022-06-29T15:58:50.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e077fe700 1 --2- 172.21.15.60:0/764876654 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9de8004e70 0x7f9de800b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T15:58:50.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e077fe700 1 -- 172.21.15.60:0/764876654 reap_dead start 2022-06-29T15:58:50.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e06ffd700 1 --2- 172.21.15.60:0/764876654 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9de80038e0 0x7f9de8003ce0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f9df800c000 tx=0x7f9df8006df0).ready entity=mon.0 client_cookie=b3525e58fc1995db server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:50.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e05ffb700 1 -- 172.21.15.60:0/764876654 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9df800e800 con 0x7f9de80038e0 2022-06-29T15:58:50.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e05ffb700 1 -- 172.21.15.60:0/764876654 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df800e960 con 0x7f9de80038e0 2022-06-29T15:58:50.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e05ffb700 1 -- 172.21.15.60:0/764876654 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9df800ecc0 con 0x7f9de80038e0 2022-06-29T15:58:50.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e07fff700 1 -- 172.21.15.60:0/764876654 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9de80038e0 msgr2=0x7f9de8003ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:50.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e07fff700 1 --2- 172.21.15.60:0/764876654 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9de80038e0 0x7f9de8003ce0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f9df800c000 tx=0x7f9df8006df0).stop 2022-06-29T15:58:50.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e07fff700 1 -- 172.21.15.60:0/764876654 shutdown_connections 2022-06-29T15:58:50.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e07fff700 1 --2- 172.21.15.60:0/764876654 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9de80038e0 0x7f9de8003ce0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e07fff700 1 -- 172.21.15.60:0/764876654 >> 172.21.15.60:0/764876654 conn(0x7f9e080b5890 msgr2=0x7f9e080b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:50.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e07fff700 1 -- 172.21.15.60:0/764876654 shutdown_connections 2022-06-29T15:58:50.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.961+0000 7f9e07fff700 1 -- 172.21.15.60:0/764876654 wait complete. 2022-06-29T15:58:50.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.965+0000 7f9e07fff700 1 Processor -- start 2022-06-29T15:58:50.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.965+0000 7f9e07fff700 1 -- start start 2022-06-29T15:58:50.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.965+0000 7f9e07fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9de80038e0 0x7f9e081597a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:50.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.965+0000 7f9e07fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e08065510 0x7f9e08159c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:50.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.965+0000 7f9e07fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e08116df0 0x7f9e081561d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:50.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.965+0000 7f9e07fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0815a6f0 con 0x7f9de80038e0 2022-06-29T15:58:50.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.965+0000 7f9e07fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0815a450 con 0x7f9e08065510 2022-06-29T15:58:50.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.965+0000 7f9e07fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0815a180 con 0x7f9e08116df0 2022-06-29T15:58:50.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.965+0000 7f9e06ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9de80038e0 0x7f9e081597a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:50.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.965+0000 7f9e06ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9de80038e0 0x7f9e081597a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45996/0 (socket says 172.21.15.60:45996) 2022-06-29T15:58:50.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.965+0000 7f9e06ffd700 1 -- 172.21.15.60:0/3784424031 learned_addr learned my addr 172.21.15.60:0/3784424031 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:50.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.965+0000 7f9e077fe700 1 --2- 172.21.15.60:0/3784424031 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e08116df0 0x7f9e081561d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:50.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.965+0000 7f9e067fc700 1 --2- 172.21.15.60:0/3784424031 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e08065510 0x7f9e08159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:50.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.965+0000 7f9e06ffd700 1 -- 172.21.15.60:0/3784424031 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e08116df0 msgr2=0x7f9e081561d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:50.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.965+0000 7f9e06ffd700 1 --2- 172.21.15.60:0/3784424031 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e08116df0 0x7f9e081561d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.965+0000 7f9e06ffd700 1 -- 172.21.15.60:0/3784424031 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e08065510 msgr2=0x7f9e08159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:50.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.965+0000 7f9e06ffd700 1 --2- 172.21.15.60:0/3784424031 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e08065510 0x7f9e08159c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:50.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.965+0000 7f9e06ffd700 1 -- 172.21.15.60:0/3784424031 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e081566c0 con 0x7f9de80038e0 2022-06-29T15:58:50.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.965+0000 7f9e06ffd700 1 --2- 172.21.15.60:0/3784424031 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9de80038e0 0x7f9e081597a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f9df800c000 tx=0x7f9df8019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:50.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.965+0000 7f9dfffff700 1 -- 172.21.15.60:0/3784424031 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9df801ccf0 con 0x7f9de80038e0 2022-06-29T15:58:50.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.965+0000 7f9dfffff700 1 -- 172.21.15.60:0/3784424031 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df8000e90 con 0x7f9de80038e0 2022-06-29T15:58:50.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.965+0000 7f9dfffff700 1 -- 172.21.15.60:0/3784424031 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9df80245e0 con 0x7f9de80038e0 2022-06-29T15:58:50.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.965+0000 7f9e07fff700 1 -- 172.21.15.60:0/3784424031 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e08156850 con 0x7f9de80038e0 2022-06-29T15:58:50.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.965+0000 7f9e07fff700 1 -- 172.21.15.60:0/3784424031 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e08156d20 con 0x7f9de80038e0 2022-06-29T15:58:50.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.969+0000 7f9e07fff700 1 -- 172.21.15.60:0/3784424031 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e0814a350 con 0x7f9de80038e0 2022-06-29T15:58:50.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.969+0000 7f9dfffff700 1 -- 172.21.15.60:0/3784424031 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9df8024740 con 0x7f9de80038e0 2022-06-29T15:58:50.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.969+0000 7f9dfffff700 1 --2- 172.21.15.60:0/3784424031 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9dec053aa0 0x7f9dec055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:50.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.969+0000 7f9dfffff700 1 -- 172.21.15.60:0/3784424031 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9df806d700 con 0x7f9de80038e0 2022-06-29T15:58:50.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.969+0000 7f9e067fc700 1 --2- 172.21.15.60:0/3784424031 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9dec053aa0 0x7f9dec055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:50.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.969+0000 7f9e067fc700 1 --2- 172.21.15.60:0/3784424031 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9dec053aa0 0x7f9dec055f60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f9df000d000 tx=0x7f9df0001ab0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:50.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:50.973+0000 7f9dfffff700 1 -- 172.21.15.60:0/3784424031 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9df8044590 con 0x7f9de80038e0 2022-06-29T15:58:51.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.101+0000 7f9e07fff700 1 -- 172.21.15.60:0/3784424031 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f9e08065200 con 0x7f9de80038e0 2022-06-29T15:58:51.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.101+0000 7f9dfffff700 1 -- 172.21.15.60:0/3784424031 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f9df8028020 con 0x7f9de80038e0 2022-06-29T15:58:51.109 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T15:58:51.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.105+0000 7f9e07fff700 1 -- 172.21.15.60:0/3784424031 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9dec053aa0 msgr2=0x7f9dec055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:51.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.105+0000 7f9e07fff700 1 --2- 172.21.15.60:0/3784424031 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9dec053aa0 0x7f9dec055f60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f9df000d000 tx=0x7f9df0001ab0).stop 2022-06-29T15:58:51.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.105+0000 7f9e07fff700 1 -- 172.21.15.60:0/3784424031 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9de80038e0 msgr2=0x7f9e081597a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:51.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.105+0000 7f9e07fff700 1 --2- 172.21.15.60:0/3784424031 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9de80038e0 0x7f9e081597a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f9df800c000 tx=0x7f9df8019000).stop 2022-06-29T15:58:51.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.105+0000 7f9e07fff700 1 -- 172.21.15.60:0/3784424031 shutdown_connections 2022-06-29T15:58:51.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.105+0000 7f9e07fff700 1 --2- 172.21.15.60:0/3784424031 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9dec053aa0 0x7f9dec055f60 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:51.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.105+0000 7f9e07fff700 1 --2- 172.21.15.60:0/3784424031 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e08116df0 0x7f9e081561d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:51.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.105+0000 7f9e07fff700 1 --2- 172.21.15.60:0/3784424031 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e08065510 0x7f9e08159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:51.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.105+0000 7f9e07fff700 1 --2- 172.21.15.60:0/3784424031 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9de80038e0 0x7f9e081597a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:51.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.105+0000 7f9e07fff700 1 -- 172.21.15.60:0/3784424031 >> 172.21.15.60:0/3784424031 conn(0x7f9e080b5890 msgr2=0x7f9e08145ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:51.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.105+0000 7f9e07fff700 1 -- 172.21.15.60:0/3784424031 shutdown_connections 2022-06-29T15:58:51.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.105+0000 7f9e07fff700 1 -- 172.21.15.60:0/3784424031 wait complete. 2022-06-29T15:58:51.122 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-29T15:58:51.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.233+0000 7fb567fff700 1 Processor -- start 2022-06-29T15:58:51.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.233+0000 7fb567fff700 1 -- start start 2022-06-29T15:58:51.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.233+0000 7fb567fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb5680b48b0 0x7fb568142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:51.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.233+0000 7fb567fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb5680b2c50 0x7fb5680b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:51.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.233+0000 7fb567fff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5680b0cf0 con 0x7fb5680b5eb0 2022-06-29T15:58:51.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.233+0000 7fb567fff700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5680b54e0 con 0x7fb5680b48b0 2022-06-29T15:58:51.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.233+0000 7fb567fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5680b4d10 con 0x7fb5680b2c50 2022-06-29T15:58:51.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.233+0000 7fb5667fc700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb5680b48b0 0x7fb568142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:51.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.233+0000 7fb5667fc700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb5680b48b0 0x7fb568142a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46006/0 (socket says 172.21.15.60:46006) 2022-06-29T15:58:51.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.233+0000 7fb5667fc700 1 -- 172.21.15.60:0/2109832592 learned_addr learned my addr 172.21.15.60:0/2109832592 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:51.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.233+0000 7fb5677fe700 1 --2- 172.21.15.60:0/2109832592 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb5680b2c50 0x7fb5680b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:51.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb5667fc700 1 -- 172.21.15.60:0/2109832592 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb5680b2c50 msgr2=0x7fb5680b30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:51.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb5667fc700 1 --2- 172.21.15.60:0/2109832592 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb5680b2c50 0x7fb5680b30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:51.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb5667fc700 1 -- 172.21.15.60:0/2109832592 >> v1:172.21.15.92:6789/0 conn(0x7fb5680b5eb0 legacy=0x7fb5680b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:51.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb5667fc700 1 -- 172.21.15.60:0/2109832592 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5680b6290 con 0x7fb5680b48b0 2022-06-29T15:58:51.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb5667fc700 1 --2- 172.21.15.60:0/2109832592 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5680b48b0 0x7fb568142a20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fb558002f70 tx=0x7fb558006eb0).ready entity=mon.0 client_cookie=6b8523bdb5c803cf server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:51.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb565ffb700 1 -- 172.21.15.60:0/2109832592 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb55800d5c0 con 0x7fb5680b48b0 2022-06-29T15:58:51.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb565ffb700 1 -- 172.21.15.60:0/2109832592 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb558000e90 con 0x7fb5680b48b0 2022-06-29T15:58:51.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb565ffb700 1 -- 172.21.15.60:0/2109832592 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb55800d950 con 0x7fb5680b48b0 2022-06-29T15:58:51.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb567fff700 1 -- 172.21.15.60:0/2109832592 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5680b48b0 msgr2=0x7fb568142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:51.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb567fff700 1 --2- 172.21.15.60:0/2109832592 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5680b48b0 0x7fb568142a20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fb558002f70 tx=0x7fb558006eb0).stop 2022-06-29T15:58:51.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb567fff700 1 -- 172.21.15.60:0/2109832592 shutdown_connections 2022-06-29T15:58:51.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb567fff700 1 --2- 172.21.15.60:0/2109832592 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb5680b2c50 0x7fb5680b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:51.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb567fff700 1 --2- 172.21.15.60:0/2109832592 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5680b48b0 0x7fb568142a20 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:51.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb567fff700 1 -- 172.21.15.60:0/2109832592 >> 172.21.15.60:0/2109832592 conn(0x7fb568062090 msgr2=0x7fb568062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:51.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb567fff700 1 -- 172.21.15.60:0/2109832592 shutdown_connections 2022-06-29T15:58:51.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb567fff700 1 -- 172.21.15.60:0/2109832592 wait complete. 2022-06-29T15:58:51.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb567fff700 1 Processor -- start 2022-06-29T15:58:51.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb567fff700 1 -- start start 2022-06-29T15:58:51.254 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb567fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5680b2c50 0x7fb56813f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:51.254 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb567fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb5680b48b0 0x7fb5680af220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:51.254 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb567fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb5680b5eb0 0x7fb568140720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:51.254 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb567fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb568144060 con 0x7fb5680b2c50 2022-06-29T15:58:51.255 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb567fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5680adb10 con 0x7fb5680b5eb0 2022-06-29T15:58:51.255 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb567fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb56813e1c0 con 0x7fb5680b48b0 2022-06-29T15:58:51.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb566ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5680b2c50 0x7fb56813f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:51.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb566ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5680b2c50 0x7fb56813f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46010/0 (socket says 172.21.15.60:46010) 2022-06-29T15:58:51.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb566ffd700 1 -- 172.21.15.60:0/4175849474 learned_addr learned my addr 172.21.15.60:0/4175849474 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:51.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb5667fc700 1 --2- 172.21.15.60:0/4175849474 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb5680b48b0 0x7fb5680af220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:51.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb5677fe700 1 --2- 172.21.15.60:0/4175849474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb5680b5eb0 0x7fb568140720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:51.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb566ffd700 1 -- 172.21.15.60:0/4175849474 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb5680b48b0 msgr2=0x7fb5680af220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:51.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb566ffd700 1 --2- 172.21.15.60:0/4175849474 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb5680b48b0 0x7fb5680af220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:51.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb566ffd700 1 -- 172.21.15.60:0/4175849474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb5680b5eb0 msgr2=0x7fb568140720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:51.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb566ffd700 1 --2- 172.21.15.60:0/4175849474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb5680b5eb0 0x7fb568140720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:51.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb566ffd700 1 -- 172.21.15.60:0/4175849474 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb55800b1e0 con 0x7fb5680b2c50 2022-06-29T15:58:51.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb5667fc700 1 --2- 172.21.15.60:0/4175849474 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb5680b48b0 0x7fb5680af220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:58:51.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb566ffd700 1 --2- 172.21.15.60:0/4175849474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5680b2c50 0x7fb56813f880 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fb568062f00 tx=0x7fb55c007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:51.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.237+0000 7fb557fff700 1 -- 172.21.15.60:0/4175849474 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb55c00ce30 con 0x7fb5680b2c50 2022-06-29T15:58:51.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.241+0000 7fb567fff700 1 -- 172.21.15.60:0/4175849474 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5680af710 con 0x7fb5680b2c50 2022-06-29T15:58:51.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.241+0000 7fb557fff700 1 -- 172.21.15.60:0/4175849474 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb55c00f470 con 0x7fb5680b2c50 2022-06-29T15:58:51.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.241+0000 7fb567fff700 1 -- 172.21.15.60:0/4175849474 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb56813edd0 con 0x7fb5680b2c50 2022-06-29T15:58:51.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.241+0000 7fb557fff700 1 -- 172.21.15.60:0/4175849474 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb55c00f7d0 con 0x7fb5680b2c50 2022-06-29T15:58:51.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.241+0000 7fb567fff700 1 -- 172.21.15.60:0/4175849474 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5681441d0 con 0x7fb5680b2c50 2022-06-29T15:58:51.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.241+0000 7fb557fff700 1 -- 172.21.15.60:0/4175849474 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb55c019030 con 0x7fb5680b2c50 2022-06-29T15:58:51.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.241+0000 7fb557fff700 1 --2- 172.21.15.60:0/4175849474 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb544053a70 0x7fb544055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:51.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.241+0000 7fb557fff700 1 -- 172.21.15.60:0/4175849474 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb55c06b390 con 0x7fb5680b2c50 2022-06-29T15:58:51.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.241+0000 7fb5667fc700 1 --2- 172.21.15.60:0/4175849474 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb544053a70 0x7fb544055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:51.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.245+0000 7fb5667fc700 1 --2- 172.21.15.60:0/4175849474 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb544053a70 0x7fb544055f30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fb55801a000 tx=0x7fb558015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:51.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.249+0000 7fb557fff700 1 -- 172.21.15.60:0/4175849474 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb5681441d0 con 0x7fb5680b2c50 2022-06-29T15:58:51.382 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.377+0000 7fb567fff700 1 -- 172.21.15.60:0/4175849474 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb5680b0cf0 con 0x7fb5680b2c50 2022-06-29T15:58:51.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.377+0000 7fb557fff700 1 -- 172.21.15.60:0/4175849474 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7fb5680b0cf0 con 0x7fb5680b2c50 2022-06-29T15:58:51.383 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T15:58:51.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.377+0000 7fb567fff700 1 -- 172.21.15.60:0/4175849474 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb544053a70 msgr2=0x7fb544055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:51.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.377+0000 7fb567fff700 1 --2- 172.21.15.60:0/4175849474 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb544053a70 0x7fb544055f30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fb55801a000 tx=0x7fb558015000).stop 2022-06-29T15:58:51.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.377+0000 7fb567fff700 1 -- 172.21.15.60:0/4175849474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5680b2c50 msgr2=0x7fb56813f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:51.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.377+0000 7fb567fff700 1 --2- 172.21.15.60:0/4175849474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5680b2c50 0x7fb56813f880 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fb568062f00 tx=0x7fb55c007c50).stop 2022-06-29T15:58:51.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.377+0000 7fb567fff700 1 -- 172.21.15.60:0/4175849474 shutdown_connections 2022-06-29T15:58:51.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.377+0000 7fb567fff700 1 --2- 172.21.15.60:0/4175849474 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb544053a70 0x7fb544055f30 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:51.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.377+0000 7fb567fff700 1 --2- 172.21.15.60:0/4175849474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb5680b5eb0 0x7fb568140720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:51.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.377+0000 7fb567fff700 1 --2- 172.21.15.60:0/4175849474 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb5680b48b0 0x7fb5680af220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:51.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.377+0000 7fb567fff700 1 --2- 172.21.15.60:0/4175849474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5680b2c50 0x7fb56813f880 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:51.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.377+0000 7fb567fff700 1 -- 172.21.15.60:0/4175849474 >> 172.21.15.60:0/4175849474 conn(0x7fb568062090 msgr2=0x7fb56805bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:51.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.381+0000 7fb567fff700 1 -- 172.21.15.60:0/4175849474 shutdown_connections 2022-06-29T15:58:51.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.381+0000 7fb567fff700 1 -- 172.21.15.60:0/4175849474 wait complete. 2022-06-29T15:58:51.397 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-29T15:58:51.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.505+0000 7f6ea147e700 1 Processor -- start 2022-06-29T15:58:51.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.505+0000 7f6ea147e700 1 -- start start 2022-06-29T15:58:51.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.509+0000 7f6ea147e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e9c0b3d00 0x7f6e9c144580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:51.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.509+0000 7f6ea147e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e9c0b40e0 con 0x7f6e9c0b1f60 2022-06-29T15:58:51.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.509+0000 7f6ea147e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e9c0ad840 con 0x7f6e9c05c470 2022-06-29T15:58:51.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.509+0000 7f6ea147e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c05c280 con 0x7f6e9c0b3d00 2022-06-29T15:58:51.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.509+0000 7f6e9b7fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6e9c0b1f60 0x7f6e9c0aca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54104/0 (socket says 172.21.15.60:54104) 2022-06-29T15:58:51.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.509+0000 7f6e9b7fe700 1 -- 172.21.15.60:0/2105757180 learned_addr learned my addr 172.21.15.60:0/2105757180 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:51.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.509+0000 7f6e9bfff700 1 --2- 172.21.15.60:0/2105757180 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e9c0b3d00 0x7f6e9c144580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:51.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.509+0000 7f6e9affd700 1 -- 172.21.15.60:0/2105757180 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 951925513 0 0) 0x7f6e9c0b40e0 con 0x7f6e9c0b1f60 2022-06-29T15:58:51.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.509+0000 7f6e9affd700 1 -- 172.21.15.60:0/2105757180 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e7c003260 con 0x7f6e9c0b1f60 2022-06-29T15:58:51.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.509+0000 7f6e9bfff700 1 -- 172.21.15.60:0/2105757180 >> v1:172.21.15.92:6789/0 conn(0x7f6e9c05c470 legacy=0x7f6e9c0ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:51.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.509+0000 7f6e9bfff700 1 -- 172.21.15.60:0/2105757180 >> v1:172.21.15.60:6789/0 conn(0x7f6e9c0b1f60 legacy=0x7f6e9c0aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:51.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.509+0000 7f6e9bfff700 1 -- 172.21.15.60:0/2105757180 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e9c143e40 con 0x7f6e9c0b3d00 2022-06-29T15:58:51.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.509+0000 7f6e9bfff700 1 --2- 172.21.15.60:0/2105757180 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e9c0b3d00 0x7f6e9c144580 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f6e84002f70 tx=0x7f6e84006df0).ready entity=mon.2 client_cookie=bd765b6b55013f1b server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:51.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.509+0000 7f6e9affd700 1 -- 172.21.15.60:0/2105757180 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e8400d570 con 0x7f6e9c0b3d00 2022-06-29T15:58:51.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.509+0000 7f6e9affd700 1 -- 172.21.15.60:0/2105757180 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e84000e90 con 0x7f6e9c0b3d00 2022-06-29T15:58:51.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.509+0000 7f6e9affd700 1 -- 172.21.15.60:0/2105757180 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e8400d8c0 con 0x7f6e9c0b3d00 2022-06-29T15:58:51.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.509+0000 7f6ea147e700 1 -- 172.21.15.60:0/2105757180 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e9c0b3d00 msgr2=0x7f6e9c144580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:51.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.509+0000 7f6ea147e700 1 --2- 172.21.15.60:0/2105757180 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e9c0b3d00 0x7f6e9c144580 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f6e84002f70 tx=0x7f6e84006df0).stop 2022-06-29T15:58:51.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.509+0000 7f6ea147e700 1 -- 172.21.15.60:0/2105757180 shutdown_connections 2022-06-29T15:58:51.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.509+0000 7f6ea147e700 1 --2- 172.21.15.60:0/2105757180 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e9c0b3d00 0x7f6e9c144580 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:51.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.509+0000 7f6ea147e700 1 -- 172.21.15.60:0/2105757180 >> 172.21.15.60:0/2105757180 conn(0x7f6e9c064810 msgr2=0x7f6e9c064bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:51.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.509+0000 7f6ea147e700 1 -- 172.21.15.60:0/2105757180 shutdown_connections 2022-06-29T15:58:51.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.509+0000 7f6ea147e700 1 -- 172.21.15.60:0/2105757180 wait complete. 2022-06-29T15:58:51.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.513+0000 7f6ea147e700 1 Processor -- start 2022-06-29T15:58:51.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.513+0000 7f6ea147e700 1 -- start start 2022-06-29T15:58:51.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.513+0000 7f6ea147e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e9c05c470 0x7f6e9c141210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:51.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.513+0000 7f6ea147e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e9c0b1f60 0x7f6e9c14f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:51.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.513+0000 7f6ea147e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e9c0b3d00 0x7f6e9c153490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:51.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.513+0000 7f6ea147e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c113a50 con 0x7f6e9c0b3d00 2022-06-29T15:58:51.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.517+0000 7f6ea147e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c1137b0 con 0x7f6e9c05c470 2022-06-29T15:58:51.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.517+0000 7f6ea147e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c152ff0 con 0x7f6e9c0b1f60 2022-06-29T15:58:51.539 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.517+0000 7f6ea0c7d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e9c0b3d00 0x7f6e9c153490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:51.539 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.517+0000 7f6ea0c7d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e9c0b3d00 0x7f6e9c153490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46028/0 (socket says 172.21.15.60:46028) 2022-06-29T15:58:51.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.517+0000 7f6ea0c7d700 1 -- 172.21.15.60:0/2127509117 learned_addr learned my addr 172.21.15.60:0/2127509117 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:51.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.517+0000 7f6e9b7fe700 1 --2- 172.21.15.60:0/2127509117 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e9c0b1f60 0x7f6e9c14f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:51.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.517+0000 7f6e9bfff700 1 --2- 172.21.15.60:0/2127509117 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e9c05c470 0x7f6e9c141210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:51.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.517+0000 7f6ea0c7d700 1 -- 172.21.15.60:0/2127509117 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e9c0b1f60 msgr2=0x7f6e9c14f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:51.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.517+0000 7f6ea0c7d700 1 --2- 172.21.15.60:0/2127509117 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e9c0b1f60 0x7f6e9c14f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:51.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.517+0000 7f6ea0c7d700 1 -- 172.21.15.60:0/2127509117 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e9c05c470 msgr2=0x7f6e9c141210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:51.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.517+0000 7f6ea0c7d700 1 --2- 172.21.15.60:0/2127509117 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e9c05c470 0x7f6e9c141210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:51.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.517+0000 7f6ea0c7d700 1 -- 172.21.15.60:0/2127509117 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e8400b180 con 0x7f6e9c0b3d00 2022-06-29T15:58:51.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.517+0000 7f6ea0c7d700 1 --2- 172.21.15.60:0/2127509117 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e9c0b3d00 0x7f6e9c153490 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f6e9c05fe30 tx=0x7f6e94007c20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:51.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.517+0000 7f6e997fa700 1 -- 172.21.15.60:0/2127509117 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e94007030 con 0x7f6e9c0b3d00 2022-06-29T15:58:51.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.517+0000 7f6e997fa700 1 -- 172.21.15.60:0/2127509117 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e94007190 con 0x7f6e9c0b3d00 2022-06-29T15:58:51.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.517+0000 7f6ea147e700 1 -- 172.21.15.60:0/2127509117 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e9c14fdc0 con 0x7f6e9c0b3d00 2022-06-29T15:58:51.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.517+0000 7f6e997fa700 1 -- 172.21.15.60:0/2127509117 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e94006510 con 0x7f6e9c0b3d00 2022-06-29T15:58:51.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.517+0000 7f6ea147e700 1 -- 172.21.15.60:0/2127509117 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e9c0b0120 con 0x7f6e9c0b3d00 2022-06-29T15:58:51.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.517+0000 7f6ea147e700 1 -- 172.21.15.60:0/2127509117 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e9c1531b0 con 0x7f6e9c0b3d00 2022-06-29T15:58:51.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.517+0000 7f6e997fa700 1 -- 172.21.15.60:0/2127509117 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6e94010030 con 0x7f6e9c0b3d00 2022-06-29T15:58:51.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.517+0000 7f6e997fa700 1 --2- 172.21.15.60:0/2127509117 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6e80053a70 0x7f6e80055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:51.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.521+0000 7f6e997fa700 1 -- 172.21.15.60:0/2127509117 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6e9406baa0 con 0x7f6e9c0b3d00 2022-06-29T15:58:51.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.521+0000 7f6e9bfff700 1 --2- 172.21.15.60:0/2127509117 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6e80053a70 0x7f6e80055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:51.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.521+0000 7f6e9bfff700 1 --2- 172.21.15.60:0/2127509117 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6e80053a70 0x7f6e80055f30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f6e8401a000 tx=0x7f6e84014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:51.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.529+0000 7f6e997fa700 1 -- 172.21.15.60:0/2127509117 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6e94042920 con 0x7f6e9c0b3d00 2022-06-29T15:58:51.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.657+0000 7f6ea147e700 1 -- 172.21.15.60:0/2127509117 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6e9c0573d0 con 0x7f6e9c0b3d00 2022-06-29T15:58:51.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.657+0000 7f6e997fa700 1 -- 172.21.15.60:0/2127509117 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7f6e94045f40 con 0x7f6e9c0b3d00 2022-06-29T15:58:51.664 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T15:58:51.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.657+0000 7f6ea147e700 1 -- 172.21.15.60:0/2127509117 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6e80053a70 msgr2=0x7f6e80055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:51.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.657+0000 7f6ea147e700 1 --2- 172.21.15.60:0/2127509117 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6e80053a70 0x7f6e80055f30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f6e8401a000 tx=0x7f6e84014000).stop 2022-06-29T15:58:51.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.661+0000 7f6ea147e700 1 -- 172.21.15.60:0/2127509117 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e9c0b3d00 msgr2=0x7f6e9c153490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:51.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.661+0000 7f6ea147e700 1 --2- 172.21.15.60:0/2127509117 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e9c0b3d00 0x7f6e9c153490 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f6e9c05fe30 tx=0x7f6e94007c20).stop 2022-06-29T15:58:51.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.661+0000 7f6ea147e700 1 -- 172.21.15.60:0/2127509117 shutdown_connections 2022-06-29T15:58:51.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.661+0000 7f6ea147e700 1 --2- 172.21.15.60:0/2127509117 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6e80053a70 0x7f6e80055f30 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:51.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.661+0000 7f6ea147e700 1 --2- 172.21.15.60:0/2127509117 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e9c0b3d00 0x7f6e9c153490 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:51.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.661+0000 7f6ea147e700 1 --2- 172.21.15.60:0/2127509117 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e9c0b1f60 0x7f6e9c14f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:51.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.661+0000 7f6ea147e700 1 --2- 172.21.15.60:0/2127509117 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e9c05c470 0x7f6e9c141210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:51.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.661+0000 7f6ea147e700 1 -- 172.21.15.60:0/2127509117 >> 172.21.15.60:0/2127509117 conn(0x7f6e9c064810 msgr2=0x7f6e9c142b70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:51.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.661+0000 7f6ea147e700 1 -- 172.21.15.60:0/2127509117 shutdown_connections 2022-06-29T15:58:51.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.661+0000 7f6ea147e700 1 -- 172.21.15.60:0/2127509117 wait complete. 2022-06-29T15:58:51.685 INFO:tasks.cephfs_test_runner:Starting test: test_drop_cache_command (tasks.cephfs.test_misc.TestCacheDrop) 2022-06-29T15:58:51.687 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command' 2022-06-29T15:58:51.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.869+0000 7f706d7ce700 1 Processor -- start 2022-06-29T15:58:51.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.869+0000 7f706d7ce700 1 -- start start 2022-06-29T15:58:51.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.869+0000 7f706d7ce700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f706805d070 0x7f706812b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:51.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f706d7ce700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70680ae9a0 con 0x7f7068066b20 2022-06-29T15:58:51.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f706d7ce700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f706805d450 con 0x7f706812bbc0 2022-06-29T15:58:51.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f706d7ce700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7068143b60 con 0x7f706805d070 2022-06-29T15:58:51.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f70677fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f7068066b20 0x7f7068142ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54122/0 (socket says 172.21.15.60:54122) 2022-06-29T15:58:51.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f70677fe700 1 -- 172.21.15.60:0/2377077925 learned_addr learned my addr 172.21.15.60:0/2377077925 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:51.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f7067fff700 1 --2- 172.21.15.60:0/2377077925 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f706805d070 0x7f706812b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:51.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f7066ffd700 1 -- 172.21.15.60:0/2377077925 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719202595 0 0) 0x7f70680ae9a0 con 0x7f7068066b20 2022-06-29T15:58:51.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f7066ffd700 1 -- 172.21.15.60:0/2377077925 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7048003260 con 0x7f7068066b20 2022-06-29T15:58:51.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f7066ffd700 1 -- 172.21.15.60:0/2377077925 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1542093178 0 0) 0x7f706805d450 con 0x7f706812bbc0 2022-06-29T15:58:51.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f7066ffd700 1 -- 172.21.15.60:0/2377077925 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70680ae9a0 con 0x7f706812bbc0 2022-06-29T15:58:51.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f7066ffd700 1 -- 172.21.15.60:0/2377077925 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 896297545 0 0) 0x7f7048003260 con 0x7f7068066b20 2022-06-29T15:58:51.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f7066ffd700 1 -- 172.21.15.60:0/2377077925 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f706805d070 msgr2=0x7f706812b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:51.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f7066ffd700 1 --2- 172.21.15.60:0/2377077925 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f706805d070 0x7f706812b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:51.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f7066ffd700 1 -- 172.21.15.60:0/2377077925 >> v1:172.21.15.92:6789/0 conn(0x7f706812bbc0 legacy=0x7f70681431f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:51.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f7066ffd700 1 -- 172.21.15.60:0/2377077925 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7068067240 con 0x7f7068066b20 2022-06-29T15:58:51.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f7066ffd700 1 -- 172.21.15.60:0/2377077925 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f70500019e0 con 0x7f7068066b20 2022-06-29T15:58:51.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f7066ffd700 1 -- 172.21.15.60:0/2377077925 >> v1:172.21.15.60:6789/0 conn(0x7f7068066b20 legacy=0x7f7068142ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:51.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f7066ffd700 1 --2- 172.21.15.60:0/2377077925 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7048003910 0x7f7048003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:51.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f7066ffd700 1 --2- 172.21.15.60:0/2377077925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7048004500 0x7f7048004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:51.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f7066ffd700 1 --2- 172.21.15.60:0/2377077925 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f706805d070 0x7f7048004e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:51.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f7066ffd700 1 -- 172.21.15.60:0/2377077925 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704800b800 con 0x7f7048004500 2022-06-29T15:58:51.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f7066ffd700 1 -- 172.21.15.60:0/2377077925 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704800b940 con 0x7f7048003910 2022-06-29T15:58:51.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f7066ffd700 1 -- 172.21.15.60:0/2377077925 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f704800bad0 con 0x7f706805d070 2022-06-29T15:58:51.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f70677fe700 1 --2- 172.21.15.60:0/2377077925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7048004500 0x7f7048004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:51.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f7067fff700 1 --2- 172.21.15.60:0/2377077925 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7048003910 0x7f7048003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:51.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f706cfcd700 1 --2- 172.21.15.60:0/2377077925 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f706805d070 0x7f7048004e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:51.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f70677fe700 1 -- 172.21.15.60:0/2377077925 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f706805d070 msgr2=0x7f7048004e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:51.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f70677fe700 1 --2- 172.21.15.60:0/2377077925 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f706805d070 0x7f7048004e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:51.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f70677fe700 1 -- 172.21.15.60:0/2377077925 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7048003910 msgr2=0x7f7048003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:51.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f70677fe700 1 --2- 172.21.15.60:0/2377077925 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7048003910 0x7f7048003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:51.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f70677fe700 1 -- 172.21.15.60:0/2377077925 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7048003260 con 0x7f7048004500 2022-06-29T15:58:51.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f706cfcd700 1 --2- 172.21.15.60:0/2377077925 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f706805d070 0x7f7048004e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T15:58:51.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f7067fff700 1 --2- 172.21.15.60:0/2377077925 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7048003910 0x7f7048003d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:51.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f70677fe700 1 --2- 172.21.15.60:0/2377077925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7048004500 0x7f7048004980 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f70480054d0 tx=0x7f7050002080).ready entity=mon.0 client_cookie=7716f36b2b3421fa server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:51.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f7066ffd700 1 -- 172.21.15.60:0/2377077925 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f705000cb10 con 0x7f7048004500 2022-06-29T15:58:51.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f7066ffd700 1 -- 172.21.15.60:0/2377077925 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f705000cc70 con 0x7f7048004500 2022-06-29T15:58:51.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.873+0000 7f7066ffd700 1 -- 172.21.15.60:0/2377077925 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7050005990 con 0x7f7048004500 2022-06-29T15:58:51.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.877+0000 7f706d7ce700 1 -- 172.21.15.60:0/2377077925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7048004500 msgr2=0x7f7048004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:51.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.877+0000 7f706d7ce700 1 --2- 172.21.15.60:0/2377077925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7048004500 0x7f7048004980 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f70480054d0 tx=0x7f7050002080).stop 2022-06-29T15:58:51.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.877+0000 7f706cfcd700 1 -- 172.21.15.60:0/2377077925 reap_dead start 2022-06-29T15:58:51.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.877+0000 7f706d7ce700 1 -- 172.21.15.60:0/2377077925 shutdown_connections 2022-06-29T15:58:51.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.877+0000 7f706d7ce700 1 -- 172.21.15.60:0/2377077925 >> 172.21.15.60:0/2377077925 conn(0x7f70680b5890 msgr2=0x7f70680b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:51.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.877+0000 7f706d7ce700 1 -- 172.21.15.60:0/2377077925 shutdown_connections 2022-06-29T15:58:51.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.877+0000 7f706d7ce700 1 -- 172.21.15.60:0/2377077925 wait complete. 2022-06-29T15:58:51.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.877+0000 7f706d7ce700 1 Processor -- start 2022-06-29T15:58:51.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.877+0000 7f706d7ce700 1 -- start start 2022-06-29T15:58:51.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.881+0000 7f706d7ce700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f706812fce0 0x7f70681300c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:51.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.881+0000 7f706d7ce700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7068142270 0x7f7068155160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:51.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.881+0000 7f706d7ce700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7068155650 0x7f7068155b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:51.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.881+0000 7f706d7ce700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7068155ff0 con 0x7f706812fce0 2022-06-29T15:58:51.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.881+0000 7f706d7ce700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70681305b0 con 0x7f7068155650 2022-06-29T15:58:51.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.881+0000 7f706d7ce700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70680b1c50 con 0x7f7068142270 2022-06-29T15:58:51.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.881+0000 7f7067fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f706812fce0 0x7f70681300c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:51.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.881+0000 7f7067fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f706812fce0 0x7f70681300c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46044/0 (socket says 172.21.15.60:46044) 2022-06-29T15:58:51.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.881+0000 7f7067fff700 1 -- 172.21.15.60:0/1625559783 learned_addr learned my addr 172.21.15.60:0/1625559783 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:51.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.881+0000 7f706cfcd700 1 --2- 172.21.15.60:0/1625559783 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7068155650 0x7f7068155b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:51.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.881+0000 7f70677fe700 1 --2- 172.21.15.60:0/1625559783 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7068142270 0x7f7068155160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:51.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.881+0000 7f7067fff700 1 -- 172.21.15.60:0/1625559783 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7068142270 msgr2=0x7f7068155160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:51.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.881+0000 7f7067fff700 1 --2- 172.21.15.60:0/1625559783 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7068142270 0x7f7068155160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:51.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.881+0000 7f7067fff700 1 -- 172.21.15.60:0/1625559783 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7068155650 msgr2=0x7f7068155b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:51.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.881+0000 7f7067fff700 1 --2- 172.21.15.60:0/1625559783 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7068155650 0x7f7068155b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:51.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.881+0000 7f7067fff700 1 -- 172.21.15.60:0/1625559783 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7068152810 con 0x7f706812fce0 2022-06-29T15:58:51.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.881+0000 7f7067fff700 1 --2- 172.21.15.60:0/1625559783 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f706812fce0 0x7f70681300c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f7068156130 tx=0x7f7058006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:51.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.881+0000 7f70657fa700 1 -- 172.21.15.60:0/1625559783 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f705800fa10 con 0x7f706812fce0 2022-06-29T15:58:51.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.881+0000 7f70657fa700 1 -- 172.21.15.60:0/1625559783 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f705800fb70 con 0x7f706812fce0 2022-06-29T15:58:51.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.881+0000 7f706d7ce700 1 -- 172.21.15.60:0/1625559783 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7068152ab0 con 0x7f706812fce0 2022-06-29T15:58:51.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.881+0000 7f70657fa700 1 -- 172.21.15.60:0/1625559783 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f705800fce0 con 0x7f706812fce0 2022-06-29T15:58:51.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.881+0000 7f706d7ce700 1 -- 172.21.15.60:0/1625559783 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7068128370 con 0x7f706812fce0 2022-06-29T15:58:51.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.881+0000 7f706d7ce700 1 -- 172.21.15.60:0/1625559783 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7068152650 con 0x7f706812fce0 2022-06-29T15:58:51.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.885+0000 7f70657fa700 1 -- 172.21.15.60:0/1625559783 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7058014440 con 0x7f706812fce0 2022-06-29T15:58:51.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.885+0000 7f70657fa700 1 --2- 172.21.15.60:0/1625559783 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f704c0539e0 0x7f704c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:51.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.885+0000 7f70657fa700 1 -- 172.21.15.60:0/1625559783 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f705806ca20 con 0x7f706812fce0 2022-06-29T15:58:51.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.885+0000 7f70677fe700 1 --2- 172.21.15.60:0/1625559783 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f704c0539e0 0x7f704c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:51.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.885+0000 7f70677fe700 1 --2- 172.21.15.60:0/1625559783 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f704c0539e0 0x7f704c055ea0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f7050019000 tx=0x7f7050015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:51.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:51.889+0000 7f70657fa700 1 -- 172.21.15.60:0/1625559783 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f70580437d0 con 0x7f706812fce0 2022-06-29T15:58:52.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.065+0000 7f706d7ce700 1 -- 172.21.15.60:0/1625559783 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command"]} v 0) v1 -- 0x7f70680af650 con 0x7f706812fce0 2022-06-29T15:58:52.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.809+0000 7f70657fa700 1 -- 172.21.15.60:0/1625559783 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command"]}]=0 v43) v1 ==== 142+0+0 (secure 0 0 0) 0x7f7058046df0 con 0x7f706812fce0 2022-06-29T15:58:52.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.813+0000 7f706d7ce700 1 -- 172.21.15.60:0/1625559783 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f704c0539e0 msgr2=0x7f704c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:52.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.813+0000 7f706d7ce700 1 --2- 172.21.15.60:0/1625559783 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f704c0539e0 0x7f704c055ea0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f7050019000 tx=0x7f7050015000).stop 2022-06-29T15:58:52.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.813+0000 7f706d7ce700 1 -- 172.21.15.60:0/1625559783 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f706812fce0 msgr2=0x7f70681300c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:52.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.813+0000 7f706d7ce700 1 --2- 172.21.15.60:0/1625559783 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f706812fce0 0x7f70681300c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f7068156130 tx=0x7f7058006df0).stop 2022-06-29T15:58:52.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.813+0000 7f706d7ce700 1 -- 172.21.15.60:0/1625559783 shutdown_connections 2022-06-29T15:58:52.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.813+0000 7f706d7ce700 1 --2- 172.21.15.60:0/1625559783 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f704c0539e0 0x7f704c055ea0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:52.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.813+0000 7f706d7ce700 1 --2- 172.21.15.60:0/1625559783 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7068155650 0x7f7068155b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:52.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.813+0000 7f706d7ce700 1 --2- 172.21.15.60:0/1625559783 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7068142270 0x7f7068155160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:52.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.813+0000 7f706d7ce700 1 --2- 172.21.15.60:0/1625559783 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f706812fce0 0x7f70681300c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:52.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.813+0000 7f706d7ce700 1 -- 172.21.15.60:0/1625559783 >> 172.21.15.60:0/1625559783 conn(0x7f70680b5890 msgr2=0x7f706805bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:52.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.813+0000 7f706d7ce700 1 -- 172.21.15.60:0/1625559783 shutdown_connections 2022-06-29T15:58:52.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.813+0000 7f706d7ce700 1 -- 172.21.15.60:0/1625559783 wait complete. 2022-06-29T15:58:52.835 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-06-29T15:58:52.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.945+0000 7fb449cf1700 1 Processor -- start 2022-06-29T15:58:52.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.949+0000 7fb449cf1700 1 -- start start 2022-06-29T15:58:52.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.949+0000 7fb449cf1700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb4440ae7c0 0x7fb444141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:52.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.949+0000 7fb449cf1700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb4440b4010 0x7fb444142490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:52.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.949+0000 7fb449cf1700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb444143a40 con 0x7fb444146160 2022-06-29T15:58:52.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.949+0000 7fb449cf1700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4440b56d0 con 0x7fb4440ae7c0 2022-06-29T15:58:52.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.949+0000 7fb449cf1700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4440aee60 con 0x7fb4440b4010 2022-06-29T15:58:52.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.949+0000 7fb4437fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb4440ae7c0 0x7fb444141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:52.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.949+0000 7fb4437fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb4440ae7c0 0x7fb444141fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46052/0 (socket says 172.21.15.60:46052) 2022-06-29T15:58:52.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.949+0000 7fb442ffd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb4440b4010 0x7fb444142490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:52.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.949+0000 7fb4437fe700 1 -- 172.21.15.60:0/2288687454 learned_addr learned my addr 172.21.15.60:0/2288687454 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:52.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.949+0000 7fb4437fe700 1 -- 172.21.15.60:0/2288687454 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb4440b4010 msgr2=0x7fb444142490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:52.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.949+0000 7fb4437fe700 1 --2- 172.21.15.60:0/2288687454 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb4440b4010 0x7fb444142490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:52.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.949+0000 7fb4437fe700 1 -- 172.21.15.60:0/2288687454 >> v1:172.21.15.92:6789/0 conn(0x7fb444146160 legacy=0x7fb4441465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:52.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.949+0000 7fb4437fe700 1 -- 172.21.15.60:0/2288687454 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4440af100 con 0x7fb4440ae7c0 2022-06-29T15:58:52.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.949+0000 7fb4437fe700 1 --2- 172.21.15.60:0/2288687454 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4440ae7c0 0x7fb444141fa0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fb42c002f70 tx=0x7fb42c006eb0).ready entity=mon.0 client_cookie=d19d3f578ee17d4b server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:52.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.949+0000 7fb4427fc700 1 -- 172.21.15.60:0/2288687454 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb42c00d570 con 0x7fb4440ae7c0 2022-06-29T15:58:52.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.949+0000 7fb4427fc700 1 -- 172.21.15.60:0/2288687454 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42c000e90 con 0x7fb4440ae7c0 2022-06-29T15:58:52.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.949+0000 7fb4427fc700 1 -- 172.21.15.60:0/2288687454 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb42c00d900 con 0x7fb4440ae7c0 2022-06-29T15:58:52.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.949+0000 7fb449cf1700 1 -- 172.21.15.60:0/2288687454 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4440ae7c0 msgr2=0x7fb444141fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:52.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.949+0000 7fb449cf1700 1 --2- 172.21.15.60:0/2288687454 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4440ae7c0 0x7fb444141fa0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fb42c002f70 tx=0x7fb42c006eb0).stop 2022-06-29T15:58:52.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.953+0000 7fb449cf1700 1 -- 172.21.15.60:0/2288687454 shutdown_connections 2022-06-29T15:58:52.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.953+0000 7fb449cf1700 1 --2- 172.21.15.60:0/2288687454 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb4440b4010 0x7fb444142490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:52.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.953+0000 7fb449cf1700 1 --2- 172.21.15.60:0/2288687454 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4440ae7c0 0x7fb444141fa0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:52.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.953+0000 7fb449cf1700 1 -- 172.21.15.60:0/2288687454 >> 172.21.15.60:0/2288687454 conn(0x7fb4440b9340 msgr2=0x7fb4440b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:52.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.953+0000 7fb449cf1700 1 -- 172.21.15.60:0/2288687454 shutdown_connections 2022-06-29T15:58:52.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.953+0000 7fb449cf1700 1 -- 172.21.15.60:0/2288687454 wait complete. 2022-06-29T15:58:52.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.953+0000 7fb449cf1700 1 Processor -- start 2022-06-29T15:58:52.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.953+0000 7fb449cf1700 1 -- start start 2022-06-29T15:58:52.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.953+0000 7fb449cf1700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4440ae7c0 0x7fb444142b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:52.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.953+0000 7fb449cf1700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4440b4010 0x7fb444151e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:52.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.953+0000 7fb449cf1700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb444146160 0x7fb4440b0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:52.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.953+0000 7fb449cf1700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb444148590 con 0x7fb4440ae7c0 2022-06-29T15:58:52.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.953+0000 7fb449cf1700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4440b03b0 con 0x7fb4440b4010 2022-06-29T15:58:52.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.953+0000 7fb449cf1700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4440b05e0 con 0x7fb444146160 2022-06-29T15:58:52.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.953+0000 7fb4437fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4440ae7c0 0x7fb444142b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:52.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.953+0000 7fb442ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4440b4010 0x7fb444151e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:52.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.953+0000 7fb4437fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4440ae7c0 0x7fb444142b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46058/0 (socket says 172.21.15.60:46058) 2022-06-29T15:58:52.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.953+0000 7fb442ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4440b4010 0x7fb444151e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:52090/0 (socket says 172.21.15.60:52090) 2022-06-29T15:58:52.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.953+0000 7fb4437fe700 1 -- 172.21.15.60:0/1084319863 learned_addr learned my addr 172.21.15.60:0/1084319863 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:52.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.953+0000 7fb443fff700 1 --2- 172.21.15.60:0/1084319863 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb444146160 0x7fb4440b0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:52.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.953+0000 7fb4437fe700 1 -- 172.21.15.60:0/1084319863 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb444146160 msgr2=0x7fb4440b0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:52.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.953+0000 7fb4437fe700 1 --2- 172.21.15.60:0/1084319863 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb444146160 0x7fb4440b0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:52.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.953+0000 7fb4437fe700 1 -- 172.21.15.60:0/1084319863 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4440b4010 msgr2=0x7fb444151e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:52.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.953+0000 7fb4437fe700 1 --2- 172.21.15.60:0/1084319863 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4440b4010 0x7fb444151e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:52.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.953+0000 7fb4437fe700 1 -- 172.21.15.60:0/1084319863 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb42c00b1e0 con 0x7fb4440ae7c0 2022-06-29T15:58:52.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.957+0000 7fb4437fe700 1 --2- 172.21.15.60:0/1084319863 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4440ae7c0 0x7fb444142b70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fb444148830 tx=0x7fb42c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:52.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.957+0000 7fb440ff9700 1 -- 172.21.15.60:0/1084319863 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb42c006b40 con 0x7fb4440ae7c0 2022-06-29T15:58:52.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.957+0000 7fb449cf1700 1 -- 172.21.15.60:0/1084319863 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4440af9f0 con 0x7fb4440ae7c0 2022-06-29T15:58:52.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.957+0000 7fb440ff9700 1 -- 172.21.15.60:0/1084319863 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42c00d8f0 con 0x7fb4440ae7c0 2022-06-29T15:58:52.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.957+0000 7fb449cf1700 1 -- 172.21.15.60:0/1084319863 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4440b3470 con 0x7fb4440ae7c0 2022-06-29T15:58:52.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.957+0000 7fb440ff9700 1 -- 172.21.15.60:0/1084319863 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb42c011de0 con 0x7fb4440ae7c0 2022-06-29T15:58:52.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.957+0000 7fb449cf1700 1 -- 172.21.15.60:0/1084319863 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4440b36e0 con 0x7fb4440ae7c0 2022-06-29T15:58:52.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.957+0000 7fb440ff9700 1 -- 172.21.15.60:0/1084319863 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb42c025030 con 0x7fb4440ae7c0 2022-06-29T15:58:52.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.957+0000 7fb440ff9700 1 --2- 172.21.15.60:0/1084319863 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb4280539e0 0x7fb428055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:52.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.957+0000 7fb440ff9700 1 -- 172.21.15.60:0/1084319863 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb42c06d180 con 0x7fb4440ae7c0 2022-06-29T15:58:52.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.961+0000 7fb442ffd700 1 --2- 172.21.15.60:0/1084319863 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb4280539e0 0x7fb428055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:52.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.961+0000 7fb442ffd700 1 --2- 172.21.15.60:0/1084319863 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb4280539e0 0x7fb428055ea0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fb43400b000 tx=0x7fb434006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:52.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:52.965+0000 7fb440ff9700 1 -- 172.21.15.60:0/1084319863 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb42c043fc0 con 0x7fb4440ae7c0 2022-06-29T15:58:53.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.089+0000 7fb449cf1700 1 -- 172.21.15.60:0/1084319863 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fb4440573d0 con 0x7fb4440ae7c0 2022-06-29T15:58:53.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.101+0000 7fb440ff9700 1 -- 172.21.15.60:0/1084319863 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v2)=0 v2) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb42c023020 con 0x7fb4440ae7c0 2022-06-29T15:58:53.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.101+0000 7fb449cf1700 1 -- 172.21.15.60:0/1084319863 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb4280539e0 msgr2=0x7fb428055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:53.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.101+0000 7fb449cf1700 1 --2- 172.21.15.60:0/1084319863 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb4280539e0 0x7fb428055ea0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fb43400b000 tx=0x7fb434006df0).stop 2022-06-29T15:58:53.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.101+0000 7fb449cf1700 1 -- 172.21.15.60:0/1084319863 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4440ae7c0 msgr2=0x7fb444142b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:53.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.101+0000 7fb449cf1700 1 --2- 172.21.15.60:0/1084319863 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4440ae7c0 0x7fb444142b70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fb444148830 tx=0x7fb42c015000).stop 2022-06-29T15:58:53.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.101+0000 7fb449cf1700 1 -- 172.21.15.60:0/1084319863 shutdown_connections 2022-06-29T15:58:53.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.101+0000 7fb449cf1700 1 --2- 172.21.15.60:0/1084319863 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb4280539e0 0x7fb428055ea0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:53.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.101+0000 7fb449cf1700 1 --2- 172.21.15.60:0/1084319863 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb444146160 0x7fb4440b0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:53.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.101+0000 7fb449cf1700 1 --2- 172.21.15.60:0/1084319863 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4440b4010 0x7fb444151e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:53.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.101+0000 7fb449cf1700 1 --2- 172.21.15.60:0/1084319863 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4440ae7c0 0x7fb444142b70 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:53.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.101+0000 7fb449cf1700 1 -- 172.21.15.60:0/1084319863 >> 172.21.15.60:0/1084319863 conn(0x7fb4440b9340 msgr2=0x7fb4440b8800 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:53.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.101+0000 7fb449cf1700 1 -- 172.21.15.60:0/1084319863 shutdown_connections 2022-06-29T15:58:53.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.101+0000 7fb449cf1700 1 -- 172.21.15.60:0/1084319863 wait complete. 2022-06-29T15:58:53.121 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-06-29T15:58:53.122 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T15:58:53.139 DEBUG:teuthology.orchestra.run.smithi092:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T15:58:53.162 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T15:58:53.184 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T15:58:53.206 DEBUG:teuthology.orchestra.run.smithi197:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T15:58:53.228 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-06-29T15:58:53.229 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:53.229 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T15:58:53.230 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T15:58:53.238 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T15:58:53.239 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T15:58:53.240 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:53.240 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T15:58:53.241 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T15:58:53.288 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T15:58:53.291 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T15:58:53.292 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:53.293 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T15:58:53.293 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T15:58:53.301 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': Transport endpoint is not connected 2022-06-29T15:58:53.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T15:58:53.303 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.0 2022-06-29T15:58:53.303 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2022-06-29T15:58:53.304 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-06-29T15:58:53.305 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-06-29T15:58:53.381 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:53.382 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T15:58:53.382 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T15:58:53.390 INFO:teuthology.orchestra.run.smithi121.stdout:ext2/ext3 2022-06-29T15:58:53.391 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-06-29T15:58:53.392 INFO:teuthology.orchestra.run:waiting for 900 2022-06-29T15:58:53.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T15:58:53.393 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T15:58:53.408 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:53.408 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T15:58:53.409 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T15:58:53.417 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2022-06-29T15:58:53.419 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-06-29T15:58:53.420 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:53.421 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T15:58:53.421 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T15:58:53.430 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2022-06-29T15:58:53.431 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-06-29T15:58:53.432 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:53.432 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T15:58:53.432 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T15:58:53.478 INFO:teuthology.orchestra.run.smithi156.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': Transport endpoint is not connected 2022-06-29T15:58:53.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T15:58:53.480 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.1 2022-06-29T15:58:53.481 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi156.front.sepia.ceph.com... 2022-06-29T15:58:53.482 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-06-29T15:58:53.482 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-06-29T15:58:53.506 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:53.507 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T15:58:53.507 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T15:58:53.514 INFO:teuthology.orchestra.run.smithi156.stdout:ext2/ext3 2022-06-29T15:58:53.516 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-06-29T15:58:53.516 INFO:teuthology.orchestra.run:waiting for 900 2022-06-29T15:58:53.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T15:58:53.517 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T15:58:53.572 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:53.572 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T15:58:53.573 DEBUG:teuthology.orchestra.run.smithi197:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T15:58:53.581 INFO:teuthology.orchestra.run.smithi197.stdout:fuseblk 2022-06-29T15:58:53.582 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2022-06-29T15:58:53.583 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:53.583 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T15:58:53.584 DEBUG:teuthology.orchestra.run.smithi197:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T15:58:53.591 INFO:teuthology.orchestra.run.smithi197.stdout:fuseblk 2022-06-29T15:58:53.592 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2022-06-29T15:58:53.593 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:53.593 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T15:58:53.594 DEBUG:teuthology.orchestra.run.smithi197:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T15:58:53.642 INFO:teuthology.orchestra.run.smithi197.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': Transport endpoint is not connected 2022-06-29T15:58:53.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T15:58:53.643 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.2 2022-06-29T15:58:53.644 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi197.front.sepia.ceph.com... 2022-06-29T15:58:53.644 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-06-29T15:58:53.645 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.2) 2022-06-29T15:58:53.677 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:58:53.678 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T15:58:53.678 DEBUG:teuthology.orchestra.run.smithi197:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T15:58:53.685 INFO:teuthology.orchestra.run.smithi197.stdout:ext2/ext3 2022-06-29T15:58:53.686 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-06-29T15:58:53.687 INFO:teuthology.orchestra.run:waiting for 900 2022-06-29T15:58:53.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T15:58:53.688 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T15:58:53.744 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-06-29T15:58:53.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.853+0000 7f5ea7fff700 1 Processor -- start 2022-06-29T15:58:53.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.853+0000 7f5ea7fff700 1 -- start start 2022-06-29T15:58:53.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.853+0000 7f5ea7fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5ea805d070 0x7f5ea812b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:53.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.853+0000 7f5ea7fff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea80ae9a0 con 0x7f5ea8066b20 2022-06-29T15:58:53.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.853+0000 7f5ea7fff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea812c050 con 0x7f5ea812bbc0 2022-06-29T15:58:53.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.853+0000 7f5ea7fff700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ea8143b60 con 0x7f5ea805d070 2022-06-29T15:58:53.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea67fc700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5ea8066b20 0x7f5ea8142ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54152/0 (socket says 172.21.15.60:54152) 2022-06-29T15:58:53.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea67fc700 1 -- 172.21.15.60:0/245657904 learned_addr learned my addr 172.21.15.60:0/245657904 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:53.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea6ffd700 1 --2- 172.21.15.60:0/245657904 >> v2:172.21.15.92:3300/0 conn(0x7f5ea805d070 0x7f5ea812b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:53.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea5ffb700 1 -- 172.21.15.60:0/245657904 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1351575550 0 0) 0x7f5ea80ae9a0 con 0x7f5ea8066b20 2022-06-29T15:58:53.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea5ffb700 1 -- 172.21.15.60:0/245657904 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e88003260 con 0x7f5ea8066b20 2022-06-29T15:58:53.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea5ffb700 1 -- 172.21.15.60:0/245657904 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151680600 0 0) 0x7f5ea812c050 con 0x7f5ea812bbc0 2022-06-29T15:58:53.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea5ffb700 1 -- 172.21.15.60:0/245657904 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ea80ae9a0 con 0x7f5ea812bbc0 2022-06-29T15:58:53.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea5ffb700 1 -- 172.21.15.60:0/245657904 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1335495108 0 0) 0x7f5e88003260 con 0x7f5ea8066b20 2022-06-29T15:58:53.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea5ffb700 1 -- 172.21.15.60:0/245657904 >> v2:172.21.15.92:3300/0 conn(0x7f5ea805d070 msgr2=0x7f5ea812b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:53.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea5ffb700 1 --2- 172.21.15.60:0/245657904 >> v2:172.21.15.92:3300/0 conn(0x7f5ea805d070 0x7f5ea812b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:53.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea5ffb700 1 -- 172.21.15.60:0/245657904 >> v1:172.21.15.92:6789/0 conn(0x7f5ea812bbc0 legacy=0x7f5ea81431f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:53.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea5ffb700 1 -- 172.21.15.60:0/245657904 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea805d450 con 0x7f5ea8066b20 2022-06-29T15:58:53.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea5ffb700 1 -- 172.21.15.60:0/245657904 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f5e90001660 con 0x7f5ea8066b20 2022-06-29T15:58:53.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea5ffb700 1 -- 172.21.15.60:0/245657904 >> v1:172.21.15.60:6789/0 conn(0x7f5ea8066b20 legacy=0x7f5ea8142ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:53.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea5ffb700 1 --2- 172.21.15.60:0/245657904 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e880038e0 0x7f5e88003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:53.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea5ffb700 1 --2- 172.21.15.60:0/245657904 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e88004500 0x7f5e88004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:53.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea5ffb700 1 --2- 172.21.15.60:0/245657904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e88004e70 0x7f5e8800b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:53.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea5ffb700 1 -- 172.21.15.60:0/245657904 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8800bcf0 con 0x7f5e88004e70 2022-06-29T15:58:53.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea5ffb700 1 -- 172.21.15.60:0/245657904 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8800be30 con 0x7f5e88004500 2022-06-29T15:58:53.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea5ffb700 1 -- 172.21.15.60:0/245657904 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8800bff0 con 0x7f5e880038e0 2022-06-29T15:58:53.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea77fe700 1 --2- 172.21.15.60:0/245657904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e88004e70 0x7f5e8800b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:53.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea6ffd700 1 --2- 172.21.15.60:0/245657904 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e880038e0 0x7f5e88003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:53.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea67fc700 1 --2- 172.21.15.60:0/245657904 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e88004500 0x7f5e88004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:53.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea67fc700 1 -- 172.21.15.60:0/245657904 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e880038e0 msgr2=0x7f5e88003ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:53.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea67fc700 1 --2- 172.21.15.60:0/245657904 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e880038e0 0x7f5e88003ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:53.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea67fc700 1 -- 172.21.15.60:0/245657904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e88004e70 msgr2=0x7f5e8800b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:53.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea67fc700 1 --2- 172.21.15.60:0/245657904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e88004e70 0x7f5e8800b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:53.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea67fc700 1 -- 172.21.15.60:0/245657904 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e88003260 con 0x7f5e88004500 2022-06-29T15:58:53.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea77fe700 1 --2- 172.21.15.60:0/245657904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e88004e70 0x7f5e8800b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:53.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea77fe700 1 -- 172.21.15.60:0/245657904 reap_dead start 2022-06-29T15:58:53.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea67fc700 1 --2- 172.21.15.60:0/245657904 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e88004500 0x7f5e88004980 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f5e880054d0 tx=0x7f5ea81428e0).ready entity=mon.1 client_cookie=d5e1fd5077f6d6a6 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:53.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea5ffb700 1 -- 172.21.15.60:0/245657904 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e900070d0 con 0x7f5e88004500 2022-06-29T15:58:53.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea5ffb700 1 -- 172.21.15.60:0/245657904 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e90007230 con 0x7f5e88004500 2022-06-29T15:58:53.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea5ffb700 1 -- 172.21.15.60:0/245657904 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e9000c8b0 con 0x7f5e88004500 2022-06-29T15:58:53.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea7fff700 1 -- 172.21.15.60:0/245657904 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e88004500 msgr2=0x7f5e88004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:53.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea7fff700 1 --2- 172.21.15.60:0/245657904 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e88004500 0x7f5e88004980 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f5e880054d0 tx=0x7f5ea81428e0).stop 2022-06-29T15:58:53.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea7fff700 1 -- 172.21.15.60:0/245657904 shutdown_connections 2022-06-29T15:58:53.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea7fff700 1 --2- 172.21.15.60:0/245657904 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e88004500 0x7f5e88004980 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:53.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea7fff700 1 -- 172.21.15.60:0/245657904 >> 172.21.15.60:0/245657904 conn(0x7f5ea80b5890 msgr2=0x7f5ea80b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:53.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea7fff700 1 -- 172.21.15.60:0/245657904 shutdown_connections 2022-06-29T15:58:53.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.857+0000 7f5ea7fff700 1 -- 172.21.15.60:0/245657904 wait complete. 2022-06-29T15:58:53.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.861+0000 7f5ea7fff700 1 Processor -- start 2022-06-29T15:58:53.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.861+0000 7f5ea7fff700 1 -- start start 2022-06-29T15:58:53.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.865+0000 7f5ea7fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e88004500 0x7f5ea8132770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:53.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.865+0000 7f5ea7fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ea812e5e0 0x7f5ea812fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:53.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.865+0000 7f5ea7fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ea81517e0 0x7f5ea812f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:53.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.865+0000 7f5ea7fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea8142420 con 0x7f5ea81517e0 2022-06-29T15:58:53.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.865+0000 7f5ea7fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea81426c0 con 0x7f5ea812e5e0 2022-06-29T15:58:53.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.865+0000 7f5ea7fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea81304d0 con 0x7f5e88004500 2022-06-29T15:58:53.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.865+0000 7f5ea77fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ea81517e0 0x7f5ea812f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:53.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.865+0000 7f5ea67fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ea812e5e0 0x7f5ea812fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:53.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.865+0000 7f5ea77fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ea81517e0 0x7f5ea812f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46082/0 (socket says 172.21.15.60:46082) 2022-06-29T15:58:53.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.865+0000 7f5ea6ffd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e88004500 0x7f5ea8132770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:53.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.865+0000 7f5ea77fe700 1 -- 172.21.15.60:0/1983523128 learned_addr learned my addr 172.21.15.60:0/1983523128 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:53.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.865+0000 7f5ea77fe700 1 -- 172.21.15.60:0/1983523128 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e88004500 msgr2=0x7f5ea8132770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:53.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.865+0000 7f5ea77fe700 1 --2- 172.21.15.60:0/1983523128 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e88004500 0x7f5ea8132770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:53.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.865+0000 7f5ea77fe700 1 -- 172.21.15.60:0/1983523128 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ea812e5e0 msgr2=0x7f5ea812fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:53.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.865+0000 7f5ea77fe700 1 --2- 172.21.15.60:0/1983523128 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ea812e5e0 0x7f5ea812fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:53.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.865+0000 7f5ea77fe700 1 -- 172.21.15.60:0/1983523128 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea812f830 con 0x7f5ea81517e0 2022-06-29T15:58:53.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.865+0000 7f5ea67fc700 1 --2- 172.21.15.60:0/1983523128 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ea812e5e0 0x7f5ea812fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:53.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.865+0000 7f5ea77fe700 1 --2- 172.21.15.60:0/1983523128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ea81517e0 0x7f5ea812f340 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f5e90001840 tx=0x7f5ea0007d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:53.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.865+0000 7f5e9ffff700 1 -- 172.21.15.60:0/1983523128 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ea0006a80 con 0x7f5ea81517e0 2022-06-29T15:58:53.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.865+0000 7f5e9ffff700 1 -- 172.21.15.60:0/1983523128 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ea0006be0 con 0x7f5ea81517e0 2022-06-29T15:58:53.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.865+0000 7f5e9ffff700 1 -- 172.21.15.60:0/1983523128 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ea0007070 con 0x7f5ea81517e0 2022-06-29T15:58:53.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.865+0000 7f5ea7fff700 1 -- 172.21.15.60:0/1983523128 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ea812eb10 con 0x7f5ea81517e0 2022-06-29T15:58:53.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.865+0000 7f5ea7fff700 1 -- 172.21.15.60:0/1983523128 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ea812e3a0 con 0x7f5ea81517e0 2022-06-29T15:58:53.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.865+0000 7f5ea7fff700 1 -- 172.21.15.60:0/1983523128 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ea812eca0 con 0x7f5ea81517e0 2022-06-29T15:58:53.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.869+0000 7f5e9ffff700 1 -- 172.21.15.60:0/1983523128 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5ea00071d0 con 0x7f5ea81517e0 2022-06-29T15:58:53.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.869+0000 7f5e9ffff700 1 --2- 172.21.15.60:0/1983523128 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5e8c053a70 0x7f5e8c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:53.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.869+0000 7f5e9ffff700 1 -- 172.21.15.60:0/1983523128 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5ea006bcc0 con 0x7f5ea81517e0 2022-06-29T15:58:53.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.869+0000 7f5ea6ffd700 1 --2- 172.21.15.60:0/1983523128 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5e8c053a70 0x7f5e8c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:53.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.869+0000 7f5ea6ffd700 1 --2- 172.21.15.60:0/1983523128 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5e8c053a70 0x7f5e8c055f30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f5e9800b000 tx=0x7f5e98006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:53.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.873+0000 7f5e9ffff700 1 -- 172.21.15.60:0/1983523128 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5ea0042be0 con 0x7f5ea81517e0 2022-06-29T15:58:54.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:53.997+0000 7f5ea7fff700 1 -- 172.21.15.60:0/1983523128 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f5ea8130690 con 0x7f5ea81517e0 2022-06-29T15:58:54.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.001+0000 7f5e9ffff700 1 -- 172.21.15.60:0/1983523128 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v23) v1 ==== 81+0+33645 (secure 0 0 0) 0x7f5ea002a380 con 0x7f5ea81517e0 2022-06-29T15:58:54.006 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:58:54.007 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T15:58:54.007 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 23, 2022-06-29T15:58:54.008 INFO:teuthology.orchestra.run.smithi060.stdout: "fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", 2022-06-29T15:58:54.008 INFO:teuthology.orchestra.run.smithi060.stdout: "created": "2022-06-29T15:58:07.965719+0000", 2022-06-29T15:58:54.008 INFO:teuthology.orchestra.run.smithi060.stdout: "modified": "2022-06-29T15:58:26.345644+0000", 2022-06-29T15:58:54.009 INFO:teuthology.orchestra.run.smithi060.stdout: "last_up_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T15:58:54.009 INFO:teuthology.orchestra.run.smithi060.stdout: "last_in_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T15:58:54.009 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-06-29T15:58:54.010 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_num": 5799936, 2022-06-29T15:58:54.010 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_set": [ 2022-06-29T15:58:54.011 INFO:teuthology.orchestra.run.smithi060.stdout: "pglog_hardlimit", 2022-06-29T15:58:54.011 INFO:teuthology.orchestra.run.smithi060.stdout: "purged_snapdirs", 2022-06-29T15:58:54.011 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_deletes", 2022-06-29T15:58:54.012 INFO:teuthology.orchestra.run.smithi060.stdout: "sortbitwise" 2022-06-29T15:58:54.012 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T15:58:54.012 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_version": 4, 2022-06-29T15:58:54.013 INFO:teuthology.orchestra.run.smithi060.stdout: "full_ratio": 0.94999998807907104, 2022-06-29T15:58:54.013 INFO:teuthology.orchestra.run.smithi060.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-06-29T15:58:54.013 INFO:teuthology.orchestra.run.smithi060.stdout: "nearfull_ratio": 0.85000002384185791, 2022-06-29T15:58:54.014 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_snapshot": "", 2022-06-29T15:58:54.014 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_max": 4, 2022-06-29T15:58:54.014 INFO:teuthology.orchestra.run.smithi060.stdout: "max_osd": 8, 2022-06-29T15:58:54.015 INFO:teuthology.orchestra.run.smithi060.stdout: "require_min_compat_client": "jewel", 2022-06-29T15:58:54.015 INFO:teuthology.orchestra.run.smithi060.stdout: "min_compat_client": "jewel", 2022-06-29T15:58:54.015 INFO:teuthology.orchestra.run.smithi060.stdout: "require_osd_release": "octopus", 2022-06-29T15:58:54.016 INFO:teuthology.orchestra.run.smithi060.stdout: "pools": [ 2022-06-29T15:58:54.016 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.017 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2022-06-29T15:58:54.017 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "device_health_metrics", 2022-06-29T15:58:54.017 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:10.961127+0000", 2022-06-29T15:58:54.017 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T15:58:54.018 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T15:58:54.018 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T15:58:54.018 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T15:58:54.018 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T15:58:54.018 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T15:58:54.019 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T15:58:54.019 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T15:58:54.019 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 1, 2022-06-29T15:58:54.019 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 1, 2022-06-29T15:58:54.020 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 1, 2022-06-29T15:58:54.020 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 1, 2022-06-29T15:58:54.020 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 1, 2022-06-29T15:58:54.020 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T15:58:54.020 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T15:58:54.021 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T15:58:54.021 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T15:58:54.021 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T15:58:54.021 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T15:58:54.021 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T15:58:54.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.022 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "15", 2022-06-29T15:58:54.022 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T15:58:54.022 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T15:58:54.023 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T15:58:54.023 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T15:58:54.023 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T15:58:54.023 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T15:58:54.023 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T15:58:54.024 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T15:58:54.024 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T15:58:54.024 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T15:58:54.024 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T15:58:54.024 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T15:58:54.025 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T15:58:54.025 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T15:58:54.025 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T15:58:54.025 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T15:58:54.026 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T15:58:54.026 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T15:58:54.026 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T15:58:54.026 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T15:58:54.026 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T15:58:54.027 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T15:58:54.027 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T15:58:54.027 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T15:58:54.028 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T15:58:54.028 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T15:58:54.028 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.029 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T15:58:54.029 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T15:58:54.029 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T15:58:54.030 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T15:58:54.030 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T15:58:54.030 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T15:58:54.031 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T15:58:54.031 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T15:58:54.031 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T15:58:54.032 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T15:58:54.032 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T15:58:54.032 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T15:58:54.033 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 1 2022-06-29T15:58:54.033 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.034 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T15:58:54.034 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_devicehealth": {} 2022-06-29T15:58:54.034 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.035 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.035 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.035 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 2, 2022-06-29T15:58:54.036 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "rbd", 2022-06-29T15:58:54.036 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:19.184276+0000", 2022-06-29T15:58:54.036 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T15:58:54.037 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T15:58:54.037 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T15:58:54.037 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T15:58:54.038 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T15:58:54.038 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T15:58:54.038 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T15:58:54.039 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T15:58:54.039 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T15:58:54.039 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T15:58:54.040 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T15:58:54.040 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T15:58:54.040 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T15:58:54.041 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T15:58:54.041 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T15:58:54.041 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T15:58:54.042 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T15:58:54.042 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T15:58:54.042 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T15:58:54.043 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T15:58:54.043 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.044 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "18", 2022-06-29T15:58:54.044 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T15:58:54.044 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T15:58:54.045 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T15:58:54.045 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T15:58:54.045 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T15:58:54.046 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T15:58:54.046 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T15:58:54.046 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T15:58:54.047 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T15:58:54.047 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T15:58:54.047 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T15:58:54.048 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T15:58:54.048 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T15:58:54.048 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T15:58:54.049 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T15:58:54.049 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T15:58:54.049 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T15:58:54.050 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T15:58:54.050 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T15:58:54.050 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T15:58:54.051 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T15:58:54.051 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T15:58:54.051 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T15:58:54.052 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T15:58:54.052 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T15:58:54.052 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T15:58:54.053 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.053 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T15:58:54.053 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T15:58:54.053 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T15:58:54.054 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T15:58:54.054 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T15:58:54.054 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T15:58:54.054 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T15:58:54.055 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T15:58:54.055 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T15:58:54.055 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T15:58:54.055 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T15:58:54.055 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T15:58:54.056 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T15:58:54.056 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd": {} 2022-06-29T15:58:54.056 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.056 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.057 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.057 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 3, 2022-06-29T15:58:54.057 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "cephfs_metadata", 2022-06-29T15:58:54.057 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:22.424395+0000", 2022-06-29T15:58:54.058 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T15:58:54.058 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T15:58:54.058 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T15:58:54.058 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T15:58:54.059 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T15:58:54.059 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T15:58:54.059 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T15:58:54.059 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T15:58:54.059 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T15:58:54.060 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T15:58:54.060 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T15:58:54.060 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T15:58:54.060 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T15:58:54.061 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T15:58:54.061 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T15:58:54.061 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T15:58:54.061 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T15:58:54.062 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T15:58:54.062 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T15:58:54.062 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T15:58:54.062 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.063 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "21", 2022-06-29T15:58:54.063 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T15:58:54.063 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T15:58:54.063 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T15:58:54.063 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T15:58:54.064 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T15:58:54.064 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T15:58:54.064 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T15:58:54.065 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T15:58:54.065 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T15:58:54.065 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T15:58:54.065 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T15:58:54.066 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T15:58:54.066 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T15:58:54.066 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T15:58:54.066 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T15:58:54.067 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T15:58:54.067 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T15:58:54.067 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T15:58:54.067 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T15:58:54.068 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T15:58:54.068 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T15:58:54.068 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T15:58:54.068 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T15:58:54.069 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T15:58:54.069 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T15:58:54.069 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T15:58:54.069 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.069 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T15:58:54.070 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T15:58:54.070 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T15:58:54.070 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T15:58:54.070 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T15:58:54.071 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T15:58:54.071 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T15:58:54.071 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T15:58:54.071 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T15:58:54.072 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T15:58:54.072 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T15:58:54.072 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T15:58:54.073 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_bias": 4, 2022-06-29T15:58:54.073 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 16, 2022-06-29T15:58:54.073 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_priority": 5 2022-06-29T15:58:54.073 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.074 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T15:58:54.074 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs": { 2022-06-29T15:58:54.074 INFO:teuthology.orchestra.run.smithi060.stdout: "metadata": "cephfs" 2022-06-29T15:58:54.075 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.075 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.075 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.075 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.076 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 4, 2022-06-29T15:58:54.076 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "cephfs_data", 2022-06-29T15:58:54.076 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:23.343438+0000", 2022-06-29T15:58:54.076 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T15:58:54.077 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T15:58:54.077 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T15:58:54.077 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T15:58:54.077 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T15:58:54.077 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T15:58:54.078 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T15:58:54.078 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T15:58:54.078 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T15:58:54.078 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T15:58:54.079 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T15:58:54.079 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T15:58:54.079 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T15:58:54.079 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T15:58:54.080 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T15:58:54.080 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T15:58:54.080 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T15:58:54.080 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T15:58:54.081 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T15:58:54.081 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T15:58:54.081 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.081 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "23", 2022-06-29T15:58:54.081 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T15:58:54.082 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T15:58:54.082 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T15:58:54.082 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T15:58:54.082 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T15:58:54.083 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T15:58:54.083 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T15:58:54.083 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T15:58:54.083 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T15:58:54.084 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T15:58:54.084 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T15:58:54.084 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T15:58:54.084 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T15:58:54.084 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T15:58:54.085 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T15:58:54.085 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T15:58:54.085 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T15:58:54.085 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T15:58:54.086 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T15:58:54.086 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T15:58:54.086 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T15:58:54.087 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T15:58:54.087 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T15:58:54.087 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T15:58:54.087 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T15:58:54.088 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T15:58:54.088 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.088 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T15:58:54.089 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T15:58:54.089 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T15:58:54.089 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T15:58:54.089 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T15:58:54.090 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T15:58:54.090 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T15:58:54.090 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T15:58:54.090 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T15:58:54.091 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T15:58:54.091 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T15:58:54.091 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T15:58:54.092 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T15:58:54.092 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs": { 2022-06-29T15:58:54.092 INFO:teuthology.orchestra.run.smithi060.stdout: "data": "cephfs" 2022-06-29T15:58:54.093 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.093 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.093 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.094 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T15:58:54.094 INFO:teuthology.orchestra.run.smithi060.stdout: "osds": [ 2022-06-29T15:58:54.094 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.095 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T15:58:54.095 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "11e7ab9e-fce2-470f-8195-ea4c3c52df22", 2022-06-29T15:58:54.095 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T15:58:54.096 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T15:58:54.096 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T15:58:54.096 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T15:58:54.096 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T15:58:54.097 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T15:58:54.097 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T15:58:54.097 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 20, 2022-06-29T15:58:54.098 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T15:58:54.098 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T15:58:54.098 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T15:58:54.098 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T15:58:54.099 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.099 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T15:58:54.099 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6816", 2022-06-29T15:58:54.099 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T15:58:54.100 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.100 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.100 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T15:58:54.101 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6825", 2022-06-29T15:58:54.101 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T15:58:54.101 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.101 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.102 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.102 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T15:58:54.102 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T15:58:54.102 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.103 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T15:58:54.103 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6826", 2022-06-29T15:58:54.103 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T15:58:54.103 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.104 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.104 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T15:58:54.104 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6827", 2022-06-29T15:58:54.105 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T15:58:54.105 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.105 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.105 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.106 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T15:58:54.106 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T15:58:54.106 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.106 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T15:58:54.107 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6830", 2022-06-29T15:58:54.107 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T15:58:54.107 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.107 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.108 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T15:58:54.108 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6831", 2022-06-29T15:58:54.108 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T15:58:54.108 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.109 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.109 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.109 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T15:58:54.109 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T15:58:54.110 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.110 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T15:58:54.110 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6828", 2022-06-29T15:58:54.111 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T15:58:54.111 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.111 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.111 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T15:58:54.111 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6829", 2022-06-29T15:58:54.112 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T15:58:54.112 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.112 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.113 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.113 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6825/12890", 2022-06-29T15:58:54.113 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6827/12890", 2022-06-29T15:58:54.113 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6831/12890", 2022-06-29T15:58:54.114 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6829/12890", 2022-06-29T15:58:54.114 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T15:58:54.114 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T15:58:54.114 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T15:58:54.115 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.115 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.115 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.115 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T15:58:54.116 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "ca43f136-6f69-4431-a157-fbe073fb585c", 2022-06-29T15:58:54.116 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T15:58:54.116 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T15:58:54.116 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T15:58:54.116 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T15:58:54.117 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T15:58:54.117 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T15:58:54.117 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T15:58:54.117 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 20, 2022-06-29T15:58:54.118 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T15:58:54.118 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T15:58:54.118 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T15:58:54.118 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T15:58:54.119 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.119 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T15:58:54.119 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6815", 2022-06-29T15:58:54.119 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T15:58:54.120 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.120 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.120 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T15:58:54.120 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6818", 2022-06-29T15:58:54.120 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T15:58:54.121 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.121 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.121 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.121 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T15:58:54.122 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T15:58:54.122 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.122 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T15:58:54.122 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6819", 2022-06-29T15:58:54.122 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T15:58:54.123 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.123 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.123 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T15:58:54.123 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6820", 2022-06-29T15:58:54.124 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T15:58:54.124 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.124 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.124 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.124 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T15:58:54.125 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T15:58:54.125 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.125 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T15:58:54.125 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6823", 2022-06-29T15:58:54.126 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T15:58:54.126 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.126 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.126 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T15:58:54.126 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6824", 2022-06-29T15:58:54.127 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T15:58:54.127 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.127 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.127 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.128 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T15:58:54.128 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T15:58:54.128 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.128 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T15:58:54.128 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6821", 2022-06-29T15:58:54.129 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T15:58:54.129 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.129 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.129 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T15:58:54.130 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6822", 2022-06-29T15:58:54.130 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T15:58:54.130 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.130 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.131 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6818/12892", 2022-06-29T15:58:54.131 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6820/12892", 2022-06-29T15:58:54.131 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6824/12892", 2022-06-29T15:58:54.131 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6822/12892", 2022-06-29T15:58:54.132 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T15:58:54.132 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T15:58:54.132 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T15:58:54.132 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.132 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.133 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.133 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T15:58:54.133 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "74fac45f-5f07-44ec-a9cb-2644c123fd4a", 2022-06-29T15:58:54.134 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T15:58:54.134 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T15:58:54.134 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T15:58:54.134 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T15:58:54.135 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T15:58:54.135 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T15:58:54.135 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T15:58:54.135 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 16, 2022-06-29T15:58:54.136 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T15:58:54.136 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T15:58:54.136 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T15:58:54.137 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T15:58:54.137 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.137 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T15:58:54.137 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6808", 2022-06-29T15:58:54.137 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T15:58:54.138 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.138 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.138 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T15:58:54.138 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6809", 2022-06-29T15:58:54.139 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T15:58:54.139 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.139 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.139 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.140 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T15:58:54.140 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T15:58:54.140 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.141 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T15:58:54.141 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6810", 2022-06-29T15:58:54.141 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T15:58:54.141 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.142 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.142 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T15:58:54.142 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6811", 2022-06-29T15:58:54.142 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T15:58:54.143 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.143 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.143 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.144 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T15:58:54.144 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T15:58:54.144 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.144 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T15:58:54.145 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6814", 2022-06-29T15:58:54.145 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T15:58:54.145 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.145 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.146 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T15:58:54.146 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6817", 2022-06-29T15:58:54.146 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T15:58:54.146 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.147 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.147 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.147 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T15:58:54.147 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T15:58:54.148 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.148 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T15:58:54.148 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6812", 2022-06-29T15:58:54.148 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T15:58:54.149 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.149 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.149 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T15:58:54.149 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6813", 2022-06-29T15:58:54.150 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T15:58:54.150 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.150 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.151 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.151 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6809/12891", 2022-06-29T15:58:54.151 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6811/12891", 2022-06-29T15:58:54.151 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6817/12891", 2022-06-29T15:58:54.151 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6813/12891", 2022-06-29T15:58:54.152 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T15:58:54.152 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T15:58:54.152 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T15:58:54.152 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.153 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.153 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.153 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T15:58:54.154 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "2581d351-973a-4b4e-85ca-3658f0068765", 2022-06-29T15:58:54.154 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T15:58:54.154 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T15:58:54.154 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T15:58:54.155 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T15:58:54.155 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T15:58:54.155 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T15:58:54.156 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T15:58:54.156 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 20, 2022-06-29T15:58:54.156 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T15:58:54.157 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T15:58:54.157 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T15:58:54.157 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T15:58:54.157 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.158 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T15:58:54.158 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6800", 2022-06-29T15:58:54.158 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T15:58:54.158 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.159 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.159 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T15:58:54.159 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6801", 2022-06-29T15:58:54.159 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T15:58:54.159 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.160 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.160 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.160 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T15:58:54.161 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T15:58:54.161 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.161 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T15:58:54.161 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6802", 2022-06-29T15:58:54.161 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T15:58:54.162 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.162 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.162 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T15:58:54.162 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6803", 2022-06-29T15:58:54.163 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T15:58:54.163 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.163 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.164 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.164 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T15:58:54.164 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T15:58:54.164 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.165 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T15:58:54.165 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6806", 2022-06-29T15:58:54.165 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T15:58:54.165 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.166 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.166 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T15:58:54.166 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6807", 2022-06-29T15:58:54.167 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T15:58:54.167 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.167 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.167 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.168 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T15:58:54.168 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T15:58:54.168 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.169 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T15:58:54.169 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6804", 2022-06-29T15:58:54.169 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T15:58:54.170 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.170 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.170 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T15:58:54.170 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6805", 2022-06-29T15:58:54.171 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T15:58:54.171 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.171 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.172 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.172 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6801/12889", 2022-06-29T15:58:54.172 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6803/12889", 2022-06-29T15:58:54.173 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6807/12889", 2022-06-29T15:58:54.173 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6805/12889", 2022-06-29T15:58:54.173 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T15:58:54.173 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T15:58:54.174 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T15:58:54.174 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.174 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.175 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.175 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T15:58:54.175 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "3d234db9-87cd-4fe9-814f-0240a287d8b4", 2022-06-29T15:58:54.176 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T15:58:54.176 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T15:58:54.176 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T15:58:54.176 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T15:58:54.177 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T15:58:54.177 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T15:58:54.177 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T15:58:54.178 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 20, 2022-06-29T15:58:54.178 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T15:58:54.178 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T15:58:54.178 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T15:58:54.179 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T15:58:54.179 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.179 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T15:58:54.179 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6808", 2022-06-29T15:58:54.180 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T15:58:54.180 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.180 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.181 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T15:58:54.181 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6810", 2022-06-29T15:58:54.181 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T15:58:54.181 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.182 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.182 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.182 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T15:58:54.182 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T15:58:54.183 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.183 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T15:58:54.183 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6812", 2022-06-29T15:58:54.183 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T15:58:54.183 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.184 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.184 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T15:58:54.184 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6814", 2022-06-29T15:58:54.185 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T15:58:54.185 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.185 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.185 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.186 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T15:58:54.186 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T15:58:54.186 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.187 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T15:58:54.187 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6823", 2022-06-29T15:58:54.187 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T15:58:54.187 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.188 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.188 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T15:58:54.188 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6825", 2022-06-29T15:58:54.188 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T15:58:54.189 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.189 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.189 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.190 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T15:58:54.190 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T15:58:54.190 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.190 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T15:58:54.191 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6817", 2022-06-29T15:58:54.191 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T15:58:54.191 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.192 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.192 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T15:58:54.192 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6820", 2022-06-29T15:58:54.192 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T15:58:54.193 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.193 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.193 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.194 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6810/13249", 2022-06-29T15:58:54.194 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6814/13249", 2022-06-29T15:58:54.194 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6825/13249", 2022-06-29T15:58:54.194 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6820/13249", 2022-06-29T15:58:54.195 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T15:58:54.195 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T15:58:54.195 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T15:58:54.195 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.196 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.196 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.196 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T15:58:54.196 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "e4bf1104-ff5c-438b-b608-8cee31c163f6", 2022-06-29T15:58:54.196 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T15:58:54.197 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T15:58:54.197 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T15:58:54.197 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T15:58:54.197 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T15:58:54.198 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T15:58:54.198 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T15:58:54.198 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 19, 2022-06-29T15:58:54.198 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T15:58:54.199 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T15:58:54.199 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T15:58:54.199 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T15:58:54.199 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.199 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T15:58:54.200 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6809", 2022-06-29T15:58:54.200 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T15:58:54.200 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.200 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.201 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T15:58:54.201 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6811", 2022-06-29T15:58:54.201 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T15:58:54.201 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.201 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.202 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.202 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T15:58:54.202 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T15:58:54.202 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.203 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T15:58:54.203 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6813", 2022-06-29T15:58:54.203 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T15:58:54.203 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.204 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.204 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T15:58:54.204 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6815", 2022-06-29T15:58:54.204 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T15:58:54.204 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.205 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.205 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.205 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T15:58:54.205 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T15:58:54.206 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.206 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T15:58:54.206 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6824", 2022-06-29T15:58:54.206 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T15:58:54.206 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.207 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.207 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T15:58:54.207 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6827", 2022-06-29T15:58:54.207 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T15:58:54.208 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.208 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.208 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.208 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T15:58:54.209 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T15:58:54.209 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.209 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T15:58:54.209 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6818", 2022-06-29T15:58:54.209 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T15:58:54.210 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.210 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.210 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T15:58:54.210 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6821", 2022-06-29T15:58:54.211 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T15:58:54.211 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.211 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.211 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.212 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6811/13248", 2022-06-29T15:58:54.212 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6815/13248", 2022-06-29T15:58:54.212 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6827/13248", 2022-06-29T15:58:54.212 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6821/13248", 2022-06-29T15:58:54.212 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T15:58:54.213 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T15:58:54.213 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T15:58:54.213 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.213 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.214 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.214 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T15:58:54.214 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "8564dd25-27ab-4203-b54e-af6d3aa05857", 2022-06-29T15:58:54.214 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T15:58:54.215 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T15:58:54.215 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T15:58:54.215 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T15:58:54.215 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T15:58:54.216 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T15:58:54.216 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T15:58:54.216 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 21, 2022-06-29T15:58:54.216 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T15:58:54.216 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T15:58:54.217 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T15:58:54.217 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T15:58:54.217 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.217 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T15:58:54.218 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6816", 2022-06-29T15:58:54.218 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T15:58:54.218 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.218 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.219 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T15:58:54.219 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6819", 2022-06-29T15:58:54.219 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T15:58:54.219 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.219 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.220 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.220 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T15:58:54.220 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T15:58:54.220 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.221 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T15:58:54.221 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6822", 2022-06-29T15:58:54.221 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T15:58:54.221 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.222 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.222 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T15:58:54.222 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6826", 2022-06-29T15:58:54.222 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T15:58:54.223 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.223 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.223 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.223 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T15:58:54.223 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T15:58:54.224 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.224 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T15:58:54.224 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6830", 2022-06-29T15:58:54.224 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T15:58:54.225 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.225 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.225 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T15:58:54.225 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6831", 2022-06-29T15:58:54.226 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T15:58:54.226 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.226 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.226 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.227 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T15:58:54.227 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T15:58:54.227 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.227 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T15:58:54.228 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6828", 2022-06-29T15:58:54.228 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T15:58:54.228 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.228 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.229 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T15:58:54.229 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6829", 2022-06-29T15:58:54.229 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T15:58:54.229 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.229 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.230 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.230 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6819/13250", 2022-06-29T15:58:54.230 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6826/13250", 2022-06-29T15:58:54.230 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6831/13250", 2022-06-29T15:58:54.231 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6829/13250", 2022-06-29T15:58:54.231 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T15:58:54.231 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T15:58:54.231 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T15:58:54.232 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.232 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.232 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.232 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T15:58:54.232 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "bd513a9c-3b19-4314-9897-d300b5c897bb", 2022-06-29T15:58:54.233 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T15:58:54.233 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T15:58:54.233 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T15:58:54.233 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T15:58:54.234 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T15:58:54.234 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T15:58:54.234 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T15:58:54.234 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 16, 2022-06-29T15:58:54.235 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T15:58:54.235 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T15:58:54.235 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T15:58:54.235 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T15:58:54.235 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.236 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T15:58:54.236 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6800", 2022-06-29T15:58:54.236 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T15:58:54.236 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.237 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.237 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T15:58:54.237 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6801", 2022-06-29T15:58:54.237 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T15:58:54.237 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.238 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.238 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.238 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T15:58:54.238 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T15:58:54.239 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.239 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T15:58:54.239 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6802", 2022-06-29T15:58:54.239 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T15:58:54.240 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.240 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.240 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T15:58:54.240 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6803", 2022-06-29T15:58:54.241 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T15:58:54.241 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.241 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.241 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.242 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T15:58:54.242 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T15:58:54.242 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.242 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T15:58:54.243 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6806", 2022-06-29T15:58:54.243 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T15:58:54.243 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.243 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.243 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T15:58:54.244 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6807", 2022-06-29T15:58:54.244 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T15:58:54.244 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.244 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.245 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.245 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T15:58:54.245 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T15:58:54.245 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.245 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T15:58:54.246 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6804", 2022-06-29T15:58:54.246 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T15:58:54.246 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.246 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.247 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T15:58:54.247 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6805", 2022-06-29T15:58:54.247 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T15:58:54.247 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.248 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.248 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.248 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6801/13247", 2022-06-29T15:58:54.248 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6803/13247", 2022-06-29T15:58:54.248 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6807/13247", 2022-06-29T15:58:54.249 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6805/13247", 2022-06-29T15:58:54.249 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T15:58:54.249 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T15:58:54.249 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T15:58:54.250 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:58:54.250 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.250 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T15:58:54.250 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_xinfo": [ 2022-06-29T15:58:54.250 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.251 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T15:58:54.251 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T15:58:54.251 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T15:58:54.251 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T15:58:54.252 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T15:58:54.252 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T15:58:54.252 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.891342+0000", 2022-06-29T15:58:54.252 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T15:58:54.253 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.253 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.253 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T15:58:54.253 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T15:58:54.253 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T15:58:54.254 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T15:58:54.254 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T15:58:54.254 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T15:58:54.254 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.900866+0000", 2022-06-29T15:58:54.255 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T15:58:54.255 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.255 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.255 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T15:58:54.256 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T15:58:54.256 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T15:58:54.256 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T15:58:54.256 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T15:58:54.256 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T15:58:54.257 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.912755+0000", 2022-06-29T15:58:54.257 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T15:58:54.257 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.257 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.258 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T15:58:54.258 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T15:58:54.258 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T15:58:54.258 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T15:58:54.258 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T15:58:54.259 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T15:58:54.259 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.864332+0000", 2022-06-29T15:58:54.259 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T15:58:54.259 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.260 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.260 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T15:58:54.260 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T15:58:54.260 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T15:58:54.260 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T15:58:54.261 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T15:58:54.261 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T15:58:54.261 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.863756+0000", 2022-06-29T15:58:54.261 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T15:58:54.262 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.262 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.262 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T15:58:54.262 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T15:58:54.263 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T15:58:54.263 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T15:58:54.263 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T15:58:54.263 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T15:58:54.263 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.869041+0000", 2022-06-29T15:58:54.264 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T15:58:54.264 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.264 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.264 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T15:58:54.265 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T15:58:54.265 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T15:58:54.265 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T15:58:54.265 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T15:58:54.265 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T15:58:54.266 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.895043+0000", 2022-06-29T15:58:54.266 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T15:58:54.266 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.266 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:58:54.267 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T15:58:54.267 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T15:58:54.267 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T15:58:54.267 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T15:58:54.268 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T15:58:54.268 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T15:58:54.268 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.889491+0000", 2022-06-29T15:58:54.268 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T15:58:54.268 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.269 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T15:58:54.269 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap": [], 2022-06-29T15:58:54.269 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap_items": [], 2022-06-29T15:58:54.269 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_temp": [], 2022-06-29T15:58:54.270 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_temp": [], 2022-06-29T15:58:54.270 INFO:teuthology.orchestra.run.smithi060.stdout: "blacklist": {}, 2022-06-29T15:58:54.270 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profiles": { 2022-06-29T15:58:54.270 INFO:teuthology.orchestra.run.smithi060.stdout: "default": { 2022-06-29T15:58:54.270 INFO:teuthology.orchestra.run.smithi060.stdout: "crush-failure-domain": "osd", 2022-06-29T15:58:54.271 INFO:teuthology.orchestra.run.smithi060.stdout: "k": "2", 2022-06-29T15:58:54.271 INFO:teuthology.orchestra.run.smithi060.stdout: "m": "1", 2022-06-29T15:58:54.271 INFO:teuthology.orchestra.run.smithi060.stdout: "plugin": "jerasure", 2022-06-29T15:58:54.271 INFO:teuthology.orchestra.run.smithi060.stdout: "ruleset-failure-domain": "osd", 2022-06-29T15:58:54.272 INFO:teuthology.orchestra.run.smithi060.stdout: "technique": "reed_sol_van" 2022-06-29T15:58:54.272 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:58:54.272 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:58:54.272 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps_queue": [], 2022-06-29T15:58:54.272 INFO:teuthology.orchestra.run.smithi060.stdout: "new_removed_snaps": [], 2022-06-29T15:58:54.273 INFO:teuthology.orchestra.run.smithi060.stdout: "new_purged_snaps": [], 2022-06-29T15:58:54.273 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_node_flags": {}, 2022-06-29T15:58:54.273 INFO:teuthology.orchestra.run.smithi060.stdout: "device_class_flags": {} 2022-06-29T15:58:54.273 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T15:58:54.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.001+0000 7f5ea7fff700 1 -- 172.21.15.60:0/1983523128 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5e8c053a70 msgr2=0x7f5e8c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:54.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.001+0000 7f5ea7fff700 1 --2- 172.21.15.60:0/1983523128 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5e8c053a70 0x7f5e8c055f30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f5e9800b000 tx=0x7f5e98006df0).stop 2022-06-29T15:58:54.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.001+0000 7f5ea7fff700 1 -- 172.21.15.60:0/1983523128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ea81517e0 msgr2=0x7f5ea812f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:54.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.001+0000 7f5ea7fff700 1 --2- 172.21.15.60:0/1983523128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ea81517e0 0x7f5ea812f340 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f5e90001840 tx=0x7f5ea0007d80).stop 2022-06-29T15:58:54.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.001+0000 7f5ea7fff700 1 -- 172.21.15.60:0/1983523128 shutdown_connections 2022-06-29T15:58:54.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.001+0000 7f5ea7fff700 1 --2- 172.21.15.60:0/1983523128 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5e8c053a70 0x7f5e8c055f30 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:54.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.001+0000 7f5ea7fff700 1 --2- 172.21.15.60:0/1983523128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ea81517e0 0x7f5ea812f340 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:54.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.001+0000 7f5ea7fff700 1 --2- 172.21.15.60:0/1983523128 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ea812e5e0 0x7f5ea812fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:54.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.001+0000 7f5ea7fff700 1 --2- 172.21.15.60:0/1983523128 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e88004500 0x7f5ea8132770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:54.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.001+0000 7f5ea7fff700 1 -- 172.21.15.60:0/1983523128 >> 172.21.15.60:0/1983523128 conn(0x7f5ea80b5890 msgr2=0x7f5ea8128ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:54.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.001+0000 7f5ea7fff700 1 -- 172.21.15.60:0/1983523128 shutdown_connections 2022-06-29T15:58:54.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.001+0000 7f5ea7fff700 1 -- 172.21.15.60:0/1983523128 wait complete. 2022-06-29T15:58:54.279 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T15:58:54.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.385+0000 7feccf576700 1 Processor -- start 2022-06-29T15:58:54.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.385+0000 7feccf576700 1 -- start start 2022-06-29T15:58:54.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.385+0000 7feccf576700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fecc805f7c0 0x7fecc805fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:54.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.385+0000 7feccf576700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc80ae000 con 0x7fecc8067020 2022-06-29T15:58:54.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.389+0000 7feccf576700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc8146f50 con 0x7fecc80af850 2022-06-29T15:58:54.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.389+0000 7feccf576700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecc80ae230 con 0x7fecc805f7c0 2022-06-29T15:58:54.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.389+0000 7fecced75700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fecc8067020 0x7fecc8067430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54174/0 (socket says 172.21.15.60:54174) 2022-06-29T15:58:54.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.389+0000 7fecced75700 1 -- 172.21.15.60:0/340382592 learned_addr learned my addr 172.21.15.60:0/340382592 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:58:54.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.389+0000 7fecce574700 1 --2- 172.21.15.60:0/340382592 >> v2:172.21.15.92:3300/0 conn(0x7fecc805f7c0 0x7fecc805fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:54.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.389+0000 7feccd572700 1 -- 172.21.15.60:0/340382592 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3810868534 0 0) 0x7fecc80ae000 con 0x7fecc8067020 2022-06-29T15:58:54.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.389+0000 7feccd572700 1 -- 172.21.15.60:0/340382592 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecb0003260 con 0x7fecc8067020 2022-06-29T15:58:54.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.389+0000 7feccd572700 1 -- 172.21.15.60:0/340382592 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1701566311 0 0) 0x7fecc8146f50 con 0x7fecc80af850 2022-06-29T15:58:54.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.389+0000 7feccd572700 1 -- 172.21.15.60:0/340382592 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecc80ae000 con 0x7fecc80af850 2022-06-29T15:58:54.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.389+0000 7fecce574700 1 -- 172.21.15.60:0/340382592 >> v1:172.21.15.92:6789/0 conn(0x7fecc80af850 legacy=0x7fecc80ae6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:54.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.389+0000 7fecce574700 1 -- 172.21.15.60:0/340382592 >> v1:172.21.15.60:6789/0 conn(0x7fecc8067020 legacy=0x7fecc8067430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:54.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.389+0000 7fecce574700 1 -- 172.21.15.60:0/340382592 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc8146bd0 con 0x7fecc805f7c0 2022-06-29T15:58:54.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.389+0000 7fecce574700 1 --2- 172.21.15.60:0/340382592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fecc805f7c0 0x7fecc805fba0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fecc4002f70 tx=0x7fecc4006df0).ready entity=mon.1 client_cookie=4db0baf1e4c1363e server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:54.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.389+0000 7feccd572700 1 -- 172.21.15.60:0/340382592 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fecc400d540 con 0x7fecc805f7c0 2022-06-29T15:58:54.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.389+0000 7feccd572700 1 -- 172.21.15.60:0/340382592 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecc4000e90 con 0x7fecc805f7c0 2022-06-29T15:58:54.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.389+0000 7feccd572700 1 -- 172.21.15.60:0/340382592 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fecc400d890 con 0x7fecc805f7c0 2022-06-29T15:58:54.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.389+0000 7feccf576700 1 -- 172.21.15.60:0/340382592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fecc805f7c0 msgr2=0x7fecc805fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:54.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.389+0000 7feccf576700 1 --2- 172.21.15.60:0/340382592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fecc805f7c0 0x7fecc805fba0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fecc4002f70 tx=0x7fecc4006df0).stop 2022-06-29T15:58:54.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.389+0000 7feccf576700 1 -- 172.21.15.60:0/340382592 shutdown_connections 2022-06-29T15:58:54.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.389+0000 7feccf576700 1 --2- 172.21.15.60:0/340382592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fecc805f7c0 0x7fecc805fba0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:54.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.389+0000 7feccf576700 1 -- 172.21.15.60:0/340382592 >> 172.21.15.60:0/340382592 conn(0x7fecc8063a60 msgr2=0x7fecc805f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:54.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.389+0000 7feccf576700 1 -- 172.21.15.60:0/340382592 shutdown_connections 2022-06-29T15:58:54.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.389+0000 7feccf576700 1 -- 172.21.15.60:0/340382592 wait complete. 2022-06-29T15:58:54.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.393+0000 7feccf576700 1 Processor -- start 2022-06-29T15:58:54.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.393+0000 7feccf576700 1 -- start start 2022-06-29T15:58:54.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.393+0000 7feccf576700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fecc805f7c0 0x7fecc8065730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:54.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.393+0000 7feccf576700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fecc8067020 0x7fecc80b0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:54.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.393+0000 7feccf576700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fecc80af850 0x7fecc8152680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:54.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.393+0000 7feccf576700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc8064ce0 con 0x7fecc80af850 2022-06-29T15:58:54.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.393+0000 7feccf576700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc8065020 con 0x7fecc8067020 2022-06-29T15:58:54.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.393+0000 7feccf576700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecc8152b70 con 0x7fecc805f7c0 2022-06-29T15:58:54.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.393+0000 7fecced75700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fecc80af850 0x7fecc8152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:54.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.393+0000 7fecce574700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fecc805f7c0 0x7fecc8065730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:54.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.393+0000 7fecce574700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fecc805f7c0 0x7fecc8065730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:59186/0 (socket says 172.21.15.60:59186) 2022-06-29T15:58:54.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.393+0000 7fecce574700 1 -- 172.21.15.60:0/2882213446 learned_addr learned my addr 172.21.15.60:0/2882213446 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:54.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.393+0000 7feccdd73700 1 --2- 172.21.15.60:0/2882213446 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fecc8067020 0x7fecc80b0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:54.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.393+0000 7fecce574700 1 -- 172.21.15.60:0/2882213446 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fecc8067020 msgr2=0x7fecc80b0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:54.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.393+0000 7fecce574700 1 --2- 172.21.15.60:0/2882213446 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fecc8067020 0x7fecc80b0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:54.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.393+0000 7fecce574700 1 -- 172.21.15.60:0/2882213446 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fecc80af850 msgr2=0x7fecc8152680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:54.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.393+0000 7fecce574700 1 --2- 172.21.15.60:0/2882213446 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fecc80af850 0x7fecc8152680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:54.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.393+0000 7fecce574700 1 -- 172.21.15.60:0/2882213446 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc400b180 con 0x7fecc805f7c0 2022-06-29T15:58:54.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.393+0000 7fecced75700 1 --2- 172.21.15.60:0/2882213446 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fecc80af850 0x7fecc8152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:54.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.393+0000 7fecce574700 1 --2- 172.21.15.60:0/2882213446 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fecc805f7c0 0x7fecc8065730 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fecc4002f70 tx=0x7fecc4014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:54.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.397+0000 7fecb77fe700 1 -- 172.21.15.60:0/2882213446 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fecc4016070 con 0x7fecc805f7c0 2022-06-29T15:58:54.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.397+0000 7fecb77fe700 1 -- 172.21.15.60:0/2882213446 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecc4013850 con 0x7fecc805f7c0 2022-06-29T15:58:54.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.397+0000 7fecb77fe700 1 -- 172.21.15.60:0/2882213446 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fecc4013bb0 con 0x7fecc805f7c0 2022-06-29T15:58:54.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.397+0000 7feccf576700 1 -- 172.21.15.60:0/2882213446 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecc8145c90 con 0x7fecc805f7c0 2022-06-29T15:58:54.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.397+0000 7feccf576700 1 -- 172.21.15.60:0/2882213446 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecc8064a50 con 0x7fecc805f7c0 2022-06-29T15:58:54.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.397+0000 7feccf576700 1 -- 172.21.15.60:0/2882213446 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecc80aca20 con 0x7fecc805f7c0 2022-06-29T15:58:54.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.397+0000 7fecb77fe700 1 -- 172.21.15.60:0/2882213446 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fecc4025030 con 0x7fecc805f7c0 2022-06-29T15:58:54.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.397+0000 7fecb77fe700 1 --2- 172.21.15.60:0/2882213446 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7feca8053aa0 0x7feca8055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:54.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.397+0000 7fecb77fe700 1 -- 172.21.15.60:0/2882213446 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fecc406cde0 con 0x7fecc805f7c0 2022-06-29T15:58:54.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.397+0000 7feccdd73700 1 --2- 172.21.15.60:0/2882213446 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7feca8053aa0 0x7feca8055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:54.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.397+0000 7feccdd73700 1 --2- 172.21.15.60:0/2882213446 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7feca8053aa0 0x7feca8055f60 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fecb800d000 tx=0x7fecb8007a20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:54.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.409+0000 7fecb77fe700 1 -- 172.21.15.60:0/2882213446 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fecc4043c60 con 0x7fecc805f7c0 2022-06-29T15:58:54.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.573+0000 7feccf576700 1 -- 172.21.15.60:0/2882213446 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fecc80573d0 con 0x7fecc805f7c0 2022-06-29T15:58:54.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.577+0000 7fecb77fe700 1 -- 172.21.15.60:0/2882213446 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2134 (secure 0 0 0) 0x7fecc4023070 con 0x7fecc805f7c0 2022-06-29T15:58:54.582 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:58:54.582 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":4,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.92:6835/2121698334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2121698334},{"type":"v1","addr":"172.21.15.92:6835","nonce":2121698334}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":4,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T15:58:24.334770+0000","modified":"2022-06-29T15:58:25.338504+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"a","rank":0,"incarnation":3,"state":"up:active","state_seq":2,"addr":"172.21.15.60:6833/3164364318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3164364318},{"type":"v1","addr":"172.21.15.60:6833","nonce":3164364318}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-29T15:58:54.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.577+0000 7feccf576700 1 -- 172.21.15.60:0/2882213446 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7feca8053aa0 msgr2=0x7feca8055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:54.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.577+0000 7feccf576700 1 --2- 172.21.15.60:0/2882213446 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7feca8053aa0 0x7feca8055f60 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fecb800d000 tx=0x7fecb8007a20).stop 2022-06-29T15:58:54.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.577+0000 7feccf576700 1 -- 172.21.15.60:0/2882213446 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fecc805f7c0 msgr2=0x7fecc8065730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:54.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.577+0000 7feccf576700 1 --2- 172.21.15.60:0/2882213446 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fecc805f7c0 0x7fecc8065730 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fecc4002f70 tx=0x7fecc4014000).stop 2022-06-29T15:58:54.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.577+0000 7feccf576700 1 -- 172.21.15.60:0/2882213446 shutdown_connections 2022-06-29T15:58:54.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.577+0000 7feccf576700 1 --2- 172.21.15.60:0/2882213446 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7feca8053aa0 0x7feca8055f60 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:54.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.577+0000 7feccf576700 1 --2- 172.21.15.60:0/2882213446 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fecc80af850 0x7fecc8152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:54.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.577+0000 7feccf576700 1 --2- 172.21.15.60:0/2882213446 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fecc8067020 0x7fecc80b0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:54.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.577+0000 7feccf576700 1 --2- 172.21.15.60:0/2882213446 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fecc805f7c0 0x7fecc8065730 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:54.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.577+0000 7feccf576700 1 -- 172.21.15.60:0/2882213446 >> 172.21.15.60:0/2882213446 conn(0x7fecc8063a60 msgr2=0x7fecc80b3530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:54.586 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.577+0000 7feccf576700 1 -- 172.21.15.60:0/2882213446 shutdown_connections 2022-06-29T15:58:54.586 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.577+0000 7feccf576700 1 -- 172.21.15.60:0/2882213446 wait complete. 2022-06-29T15:58:54.586 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 4 2022-06-29T15:58:54.594 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-06-29T15:58:54.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.681+0000 7fcfc1f7a700 1 Processor -- start 2022-06-29T15:58:54.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.681+0000 7fcfc1f7a700 1 -- start start 2022-06-29T15:58:54.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.685+0000 7fcfc1f7a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcfbc0b61d0 0x7fcfbc0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:54.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.685+0000 7fcfc1f7a700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcfbc0adc80 0x7fcfbc142a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:54.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.685+0000 7fcfc1f7a700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfbc0b65b0 con 0x7fcfbc060b60 2022-06-29T15:58:54.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.685+0000 7fcfc1f7a700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfbc0605e0 con 0x7fcfbc0adc80 2022-06-29T15:58:54.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.685+0000 7fcfc1f7a700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfbc060880 con 0x7fcfbc0b61d0 2022-06-29T15:58:54.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.685+0000 7fcfbaffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcfbc0adc80 0x7fcfbc142a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:54.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.685+0000 7fcfbaffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcfbc0adc80 0x7fcfbc142a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46100/0 (socket says 172.21.15.60:46100) 2022-06-29T15:58:54.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.685+0000 7fcfbaffd700 1 -- 172.21.15.60:0/1172709067 learned_addr learned my addr 172.21.15.60:0/1172709067 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:54.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.685+0000 7fcfbb7fe700 1 --2- 172.21.15.60:0/1172709067 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcfbc0b61d0 0x7fcfbc0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:54.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.685+0000 7fcfbaffd700 1 -- 172.21.15.60:0/1172709067 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcfbc0b61d0 msgr2=0x7fcfbc0af8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:54.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.685+0000 7fcfbaffd700 1 --2- 172.21.15.60:0/1172709067 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcfbc0b61d0 0x7fcfbc0af8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:54.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.685+0000 7fcfbaffd700 1 -- 172.21.15.60:0/1172709067 >> v1:172.21.15.92:6789/0 conn(0x7fcfbc060b60 legacy=0x7fcfbc142f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:54.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.685+0000 7fcfbaffd700 1 -- 172.21.15.60:0/1172709067 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfbc067340 con 0x7fcfbc0adc80 2022-06-29T15:58:54.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.685+0000 7fcfbaffd700 1 --2- 172.21.15.60:0/1172709067 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfbc0adc80 0x7fcfbc142a70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fcfa40032f0 tx=0x7fcfa4009290).ready entity=mon.0 client_cookie=cee6695336834fb5 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:54.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.685+0000 7fcfba7fc700 1 -- 172.21.15.60:0/1172709067 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcfa400b8a0 con 0x7fcfbc0adc80 2022-06-29T15:58:54.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.685+0000 7fcfba7fc700 1 -- 172.21.15.60:0/1172709067 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfa400ba00 con 0x7fcfbc0adc80 2022-06-29T15:58:54.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.685+0000 7fcfba7fc700 1 -- 172.21.15.60:0/1172709067 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcfa400bb70 con 0x7fcfbc0adc80 2022-06-29T15:58:54.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.685+0000 7fcfc1f7a700 1 -- 172.21.15.60:0/1172709067 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfbc0adc80 msgr2=0x7fcfbc142a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:54.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.685+0000 7fcfc1f7a700 1 --2- 172.21.15.60:0/1172709067 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfbc0adc80 0x7fcfbc142a70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fcfa40032f0 tx=0x7fcfa4009290).stop 2022-06-29T15:58:54.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.685+0000 7fcfc1f7a700 1 -- 172.21.15.60:0/1172709067 shutdown_connections 2022-06-29T15:58:54.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.685+0000 7fcfc1f7a700 1 --2- 172.21.15.60:0/1172709067 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfbc0adc80 0x7fcfbc142a70 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:54.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.685+0000 7fcfc1f7a700 1 --2- 172.21.15.60:0/1172709067 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcfbc0b61d0 0x7fcfbc0af8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:54.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.685+0000 7fcfc1f7a700 1 -- 172.21.15.60:0/1172709067 >> 172.21.15.60:0/1172709067 conn(0x7fcfbc063b50 msgr2=0x7fcfbc0b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:54.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.685+0000 7fcfc1f7a700 1 -- 172.21.15.60:0/1172709067 shutdown_connections 2022-06-29T15:58:54.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.685+0000 7fcfc1f7a700 1 -- 172.21.15.60:0/1172709067 wait complete. 2022-06-29T15:58:54.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.689+0000 7fcfc1f7a700 1 Processor -- start 2022-06-29T15:58:54.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.689+0000 7fcfc1f7a700 1 -- start start 2022-06-29T15:58:54.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.689+0000 7fcfc1f7a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcfbc060b60 0x7fcfbc149ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:54.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.689+0000 7fcfc1f7a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcfbc0adc80 0x7fcfbc066610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:54.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.689+0000 7fcfc1f7a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfbc0b61d0 0x7fcfbc151c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:54.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.689+0000 7fcfc1f7a700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfbc066b00 con 0x7fcfbc0b61d0 2022-06-29T15:58:54.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.689+0000 7fcfc1f7a700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfbc152180 con 0x7fcfbc0adc80 2022-06-29T15:58:54.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.689+0000 7fcfc1f7a700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfbc066d50 con 0x7fcfbc060b60 2022-06-29T15:58:54.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.689+0000 7fcfbbfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfbc0b61d0 0x7fcfbc151c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:54.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.689+0000 7fcfbb7fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcfbc060b60 0x7fcfbc149ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:54.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.689+0000 7fcfbaffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcfbc0adc80 0x7fcfbc066610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:54.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.689+0000 7fcfbbfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfbc0b61d0 0x7fcfbc151c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46110/0 (socket says 172.21.15.60:46110) 2022-06-29T15:58:54.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.689+0000 7fcfbbfff700 1 -- 172.21.15.60:0/310860804 learned_addr learned my addr 172.21.15.60:0/310860804 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:54.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.689+0000 7fcfbbfff700 1 -- 172.21.15.60:0/310860804 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcfbc060b60 msgr2=0x7fcfbc149ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:54.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.689+0000 7fcfbbfff700 1 --2- 172.21.15.60:0/310860804 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcfbc060b60 0x7fcfbc149ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:54.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.689+0000 7fcfbbfff700 1 -- 172.21.15.60:0/310860804 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcfbc0adc80 msgr2=0x7fcfbc066610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:54.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.689+0000 7fcfbbfff700 1 --2- 172.21.15.60:0/310860804 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcfbc0adc80 0x7fcfbc066610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:54.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.689+0000 7fcfbbfff700 1 -- 172.21.15.60:0/310860804 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa4009530 con 0x7fcfbc0b61d0 2022-06-29T15:58:54.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.689+0000 7fcfbbfff700 1 --2- 172.21.15.60:0/310860804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfbc0b61d0 0x7fcfbc151c90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7fcfbc066c70 tx=0x7fcfb4007bc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:54.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.689+0000 7fcfb8ff9700 1 -- 172.21.15.60:0/310860804 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcfb4012400 con 0x7fcfbc0b61d0 2022-06-29T15:58:54.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.689+0000 7fcfb8ff9700 1 -- 172.21.15.60:0/310860804 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb4012560 con 0x7fcfbc0b61d0 2022-06-29T15:58:54.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.689+0000 7fcfc1f7a700 1 -- 172.21.15.60:0/310860804 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfbc066ff0 con 0x7fcfbc0b61d0 2022-06-29T15:58:54.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.689+0000 7fcfb8ff9700 1 -- 172.21.15.60:0/310860804 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcfb40128c0 con 0x7fcfbc0b61d0 2022-06-29T15:58:54.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.689+0000 7fcfc1f7a700 1 -- 172.21.15.60:0/310860804 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfbc065c30 con 0x7fcfbc0b61d0 2022-06-29T15:58:54.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.693+0000 7fcfc1f7a700 1 -- 172.21.15.60:0/310860804 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfbc0573d0 con 0x7fcfbc0b61d0 2022-06-29T15:58:54.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.693+0000 7fcfb8ff9700 1 -- 172.21.15.60:0/310860804 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcfb4019030 con 0x7fcfbc0b61d0 2022-06-29T15:58:54.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.693+0000 7fcfb8ff9700 1 --2- 172.21.15.60:0/310860804 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcf94053950 0x7fcf94055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:54.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.693+0000 7fcfb8ff9700 1 -- 172.21.15.60:0/310860804 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcfb406b250 con 0x7fcfbc0b61d0 2022-06-29T15:58:54.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.693+0000 7fcfbb7fe700 1 --2- 172.21.15.60:0/310860804 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcf94053950 0x7fcf94055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:54.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.693+0000 7fcfbb7fe700 1 --2- 172.21.15.60:0/310860804 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcf94053950 0x7fcf94055e10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fcfac00c000 tx=0x7fcfac008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:54.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.701+0000 7fcfb8ff9700 1 -- 172.21.15.60:0/310860804 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcfb40420d0 con 0x7fcfbc0b61d0 2022-06-29T15:58:54.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:54.869+0000 7fcfc1f7a700 1 -- 172.21.15.60:0/310860804 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fcfbc0686b0 con 0x7fcfbc0b61d0 2022-06-29T15:58:55.823 INFO:tasks.ceph.mon.a.smithi060.stderr:2022-06-29T15:58:55.817+0000 7fcc1a376700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-06-29T15:58:55.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.829+0000 7fcfb8ff9700 1 -- 172.21.15.60:0/310860804 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v5) v1 ==== 161+0+0 (secure 0 0 0) 0x7fcfb40456f0 con 0x7fcfbc0b61d0 2022-06-29T15:58:55.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.833+0000 7fcfc1f7a700 1 -- 172.21.15.60:0/310860804 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcf94053950 msgr2=0x7fcf94055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:55.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.833+0000 7fcfc1f7a700 1 --2- 172.21.15.60:0/310860804 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcf94053950 0x7fcf94055e10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fcfac00c000 tx=0x7fcfac008040).stop 2022-06-29T15:58:55.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.833+0000 7fcfc1f7a700 1 -- 172.21.15.60:0/310860804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfbc0b61d0 msgr2=0x7fcfbc151c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:55.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.833+0000 7fcfc1f7a700 1 --2- 172.21.15.60:0/310860804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfbc0b61d0 0x7fcfbc151c90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7fcfbc066c70 tx=0x7fcfb4007bc0).stop 2022-06-29T15:58:55.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.833+0000 7fcfc1f7a700 1 -- 172.21.15.60:0/310860804 shutdown_connections 2022-06-29T15:58:55.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.833+0000 7fcfc1f7a700 1 --2- 172.21.15.60:0/310860804 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcf94053950 0x7fcf94055e10 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:55.843 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.833+0000 7fcfc1f7a700 1 --2- 172.21.15.60:0/310860804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfbc0b61d0 0x7fcfbc151c90 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:55.843 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.833+0000 7fcfc1f7a700 1 --2- 172.21.15.60:0/310860804 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcfbc0adc80 0x7fcfbc066610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:55.843 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.833+0000 7fcfc1f7a700 1 --2- 172.21.15.60:0/310860804 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcfbc060b60 0x7fcfbc149ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:55.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.833+0000 7fcfc1f7a700 1 -- 172.21.15.60:0/310860804 >> 172.21.15.60:0/310860804 conn(0x7fcfbc063b50 msgr2=0x7fcfbc142a50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:55.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.833+0000 7fcfc1f7a700 1 -- 172.21.15.60:0/310860804 shutdown_connections 2022-06-29T15:58:55.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.833+0000 7fcfc1f7a700 1 -- 172.21.15.60:0/310860804 wait complete. 2022-06-29T15:58:55.845 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-06-29T15:58:55.857 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T15:58:55.864 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2022-06-29T15:58:55.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.961+0000 7fb52a5cc700 1 Processor -- start 2022-06-29T15:58:55.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.961+0000 7fb52a5cc700 1 -- start start 2022-06-29T15:58:55.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.961+0000 7fb52a5cc700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb52405d070 0x7fb52414aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:55.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.961+0000 7fb52a5cc700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb5240b2130 0x7fb5240aca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:55.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.961+0000 7fb52a5cc700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb52414af80 0x7fb5240acf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:55.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.961+0000 7fb52a5cc700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5240adf90 con 0x7fb52405d070 2022-06-29T15:58:55.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.961+0000 7fb52a5cc700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb52405d450 con 0x7fb5240b2130 2022-06-29T15:58:55.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.961+0000 7fb52a5cc700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5240b25d0 con 0x7fb52414af80 2022-06-29T15:58:55.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.961+0000 7fb5295ca700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb52405d070 0x7fb52414aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:55.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.961+0000 7fb5295ca700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb52405d070 0x7fb52414aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46124/0 (socket says 172.21.15.60:46124) 2022-06-29T15:58:55.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.961+0000 7fb5295ca700 1 -- 172.21.15.60:0/2046008119 learned_addr learned my addr 172.21.15.60:0/2046008119 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:55.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.965+0000 7fb529dcb700 1 --2- 172.21.15.60:0/2046008119 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb52414af80 0x7fb5240acf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:55.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.965+0000 7fb528dc9700 1 --2- 172.21.15.60:0/2046008119 >> v2:172.21.15.92:3300/0 conn(0x7fb5240b2130 0x7fb5240aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:55.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.965+0000 7fb5295ca700 1 -- 172.21.15.60:0/2046008119 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb52414af80 msgr2=0x7fb5240acf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:55.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.965+0000 7fb5295ca700 1 --2- 172.21.15.60:0/2046008119 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb52414af80 0x7fb5240acf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:55.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.965+0000 7fb5295ca700 1 -- 172.21.15.60:0/2046008119 >> v2:172.21.15.92:3300/0 conn(0x7fb5240b2130 msgr2=0x7fb5240aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:55.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.965+0000 7fb5295ca700 1 --2- 172.21.15.60:0/2046008119 >> v2:172.21.15.92:3300/0 conn(0x7fb5240b2130 0x7fb5240aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:55.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.965+0000 7fb5295ca700 1 -- 172.21.15.60:0/2046008119 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb52414a350 con 0x7fb52405d070 2022-06-29T15:58:55.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.965+0000 7fb5295ca700 1 --2- 172.21.15.60:0/2046008119 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb52405d070 0x7fb52414aa90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fb520002f70 tx=0x7fb520006eb0).ready entity=mon.0 client_cookie=1ef8ea8353f575ef server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:55.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.965+0000 7fb51bfff700 1 -- 172.21.15.60:0/2046008119 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb52000d570 con 0x7fb52405d070 2022-06-29T15:58:55.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.965+0000 7fb51bfff700 1 -- 172.21.15.60:0/2046008119 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb520000e90 con 0x7fb52405d070 2022-06-29T15:58:55.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.965+0000 7fb51bfff700 1 -- 172.21.15.60:0/2046008119 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb52000d900 con 0x7fb52405d070 2022-06-29T15:58:55.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.965+0000 7fb52a5cc700 1 -- 172.21.15.60:0/2046008119 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb52405d070 msgr2=0x7fb52414aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:55.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.965+0000 7fb52a5cc700 1 --2- 172.21.15.60:0/2046008119 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb52405d070 0x7fb52414aa90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fb520002f70 tx=0x7fb520006eb0).stop 2022-06-29T15:58:55.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.965+0000 7fb52a5cc700 1 -- 172.21.15.60:0/2046008119 shutdown_connections 2022-06-29T15:58:55.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.965+0000 7fb52a5cc700 1 --2- 172.21.15.60:0/2046008119 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb52414af80 0x7fb5240acf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:55.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.965+0000 7fb52a5cc700 1 --2- 172.21.15.60:0/2046008119 >> v2:172.21.15.92:3300/0 conn(0x7fb5240b2130 0x7fb5240aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:55.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.965+0000 7fb52a5cc700 1 --2- 172.21.15.60:0/2046008119 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb52405d070 0x7fb52414aa90 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:55.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.965+0000 7fb52a5cc700 1 -- 172.21.15.60:0/2046008119 >> 172.21.15.60:0/2046008119 conn(0x7fb5240b5890 msgr2=0x7fb5240b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:55.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.965+0000 7fb52a5cc700 1 -- 172.21.15.60:0/2046008119 shutdown_connections 2022-06-29T15:58:55.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.965+0000 7fb52a5cc700 1 -- 172.21.15.60:0/2046008119 wait complete. 2022-06-29T15:58:55.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.969+0000 7fb52a5cc700 1 Processor -- start 2022-06-29T15:58:55.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.969+0000 7fb52a5cc700 1 -- start start 2022-06-29T15:58:55.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.969+0000 7fb52a5cc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb52405d070 0x7fb524117510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:55.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.969+0000 7fb52a5cc700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb5240b2130 0x7fb5240b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:55.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.969+0000 7fb52a5cc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb52414af80 0x7fb524144000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:55.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.969+0000 7fb52a5cc700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb524066700 con 0x7fb52405d070 2022-06-29T15:58:55.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.969+0000 7fb52a5cc700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb524157280 con 0x7fb52414af80 2022-06-29T15:58:55.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.969+0000 7fb52a5cc700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb524063a90 con 0x7fb5240b2130 2022-06-29T15:58:55.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.969+0000 7fb5295ca700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb52405d070 0x7fb524117510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:55.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.969+0000 7fb5295ca700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb52405d070 0x7fb524117510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46130/0 (socket says 172.21.15.60:46130) 2022-06-29T15:58:55.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.969+0000 7fb5295ca700 1 -- 172.21.15.60:0/842188064 learned_addr learned my addr 172.21.15.60:0/842188064 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:55.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.969+0000 7fb528dc9700 1 --2- 172.21.15.60:0/842188064 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb5240b2130 0x7fb5240b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:55.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.969+0000 7fb529dcb700 1 --2- 172.21.15.60:0/842188064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb52414af80 0x7fb524144000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:55.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.969+0000 7fb5295ca700 1 -- 172.21.15.60:0/842188064 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb5240b2130 msgr2=0x7fb5240b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:55.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.969+0000 7fb5295ca700 1 --2- 172.21.15.60:0/842188064 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb5240b2130 0x7fb5240b44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:55.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.969+0000 7fb5295ca700 1 -- 172.21.15.60:0/842188064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb52414af80 msgr2=0x7fb524144000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:55.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.969+0000 7fb5295ca700 1 --2- 172.21.15.60:0/842188064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb52414af80 0x7fb524144000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:55.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.969+0000 7fb5295ca700 1 -- 172.21.15.60:0/842188064 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb52000b1e0 con 0x7fb52405d070 2022-06-29T15:58:55.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.969+0000 7fb5295ca700 1 --2- 172.21.15.60:0/842188064 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb52405d070 0x7fb524117510 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fb524063c20 tx=0x7fb520015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:55.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.969+0000 7fb51a7fc700 1 -- 172.21.15.60:0/842188064 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5200116e0 con 0x7fb52405d070 2022-06-29T15:58:55.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.969+0000 7fb51a7fc700 1 -- 172.21.15.60:0/842188064 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb520011840 con 0x7fb52405d070 2022-06-29T15:58:55.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.969+0000 7fb52a5cc700 1 -- 172.21.15.60:0/842188064 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb524063d00 con 0x7fb52405d070 2022-06-29T15:58:55.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.969+0000 7fb51a7fc700 1 -- 172.21.15.60:0/842188064 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb520011ba0 con 0x7fb52405d070 2022-06-29T15:58:56.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.969+0000 7fb52a5cc700 1 -- 172.21.15.60:0/842188064 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb524155fd0 con 0x7fb52405d070 2022-06-29T15:58:56.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.973+0000 7fb52a5cc700 1 -- 172.21.15.60:0/842188064 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5241561f0 con 0x7fb52405d070 2022-06-29T15:58:56.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.973+0000 7fb51a7fc700 1 -- 172.21.15.60:0/842188064 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb520025030 con 0x7fb52405d070 2022-06-29T15:58:56.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.973+0000 7fb51a7fc700 1 --2- 172.21.15.60:0/842188064 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb50c053aa0 0x7fb50c055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:56.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.973+0000 7fb51a7fc700 1 -- 172.21.15.60:0/842188064 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fb52006d190 con 0x7fb52405d070 2022-06-29T15:58:56.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.973+0000 7fb528dc9700 1 --2- 172.21.15.60:0/842188064 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb50c053aa0 0x7fb50c055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:56.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.977+0000 7fb528dc9700 1 --2- 172.21.15.60:0/842188064 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb50c053aa0 0x7fb50c055f60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fb51400c000 tx=0x7fb514008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:56.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:55.981+0000 7fb51a7fc700 1 -- 172.21.15.60:0/842188064 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb520043f50 con 0x7fb52405d070 2022-06-29T15:58:56.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.149+0000 7fb52a5cc700 1 -- 172.21.15.60:0/842188064 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5241595d0 con 0x7fb52405d070 2022-06-29T15:58:56.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.149+0000 7fb51a7fc700 1 -- 172.21.15.60:0/842188064 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+1774 (secure 0 0 0) 0x7fb520023090 con 0x7fb52405d070 2022-06-29T15:58:56.155 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:58:56.155 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":5,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4273,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.92:6835/2121698334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2121698334},{"type":"v1","addr":"172.21.15.92:6835","nonce":2121698334}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":5,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T15:58:24.334770+0000","modified":"2022-06-29T15:58:55.820331+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-06-29T15:58:56.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.149+0000 7fb52a5cc700 1 -- 172.21.15.60:0/842188064 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb50c053aa0 msgr2=0x7fb50c055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:56.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.149+0000 7fb52a5cc700 1 --2- 172.21.15.60:0/842188064 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb50c053aa0 0x7fb50c055f60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fb51400c000 tx=0x7fb514008040).stop 2022-06-29T15:58:56.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.149+0000 7fb52a5cc700 1 -- 172.21.15.60:0/842188064 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb52405d070 msgr2=0x7fb524117510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:56.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.149+0000 7fb52a5cc700 1 --2- 172.21.15.60:0/842188064 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb52405d070 0x7fb524117510 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fb524063c20 tx=0x7fb520015000).stop 2022-06-29T15:58:56.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.153+0000 7fb52a5cc700 1 -- 172.21.15.60:0/842188064 shutdown_connections 2022-06-29T15:58:56.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.153+0000 7fb52a5cc700 1 --2- 172.21.15.60:0/842188064 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb50c053aa0 0x7fb50c055f60 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:56.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.153+0000 7fb52a5cc700 1 --2- 172.21.15.60:0/842188064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb52414af80 0x7fb524144000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:56.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.153+0000 7fb52a5cc700 1 --2- 172.21.15.60:0/842188064 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb5240b2130 0x7fb5240b44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:56.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.153+0000 7fb52a5cc700 1 --2- 172.21.15.60:0/842188064 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb52405d070 0x7fb524117510 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:56.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.153+0000 7fb52a5cc700 1 -- 172.21.15.60:0/842188064 >> 172.21.15.60:0/842188064 conn(0x7fb5240b5890 msgr2=0x7fb5241452a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:56.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.153+0000 7fb52a5cc700 1 -- 172.21.15.60:0/842188064 shutdown_connections 2022-06-29T15:58:56.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.153+0000 7fb52a5cc700 1 -- 172.21.15.60:0/842188064 wait complete. 2022-06-29T15:58:56.160 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 5 2022-06-29T15:58:56.168 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-06-29T15:58:56.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.269+0000 7f75491fa700 1 Processor -- start 2022-06-29T15:58:56.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.269+0000 7f75491fa700 1 -- start start 2022-06-29T15:58:56.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.273+0000 7f75491fa700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f75440ae7c0 0x7f7544141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:56.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.273+0000 7f75491fa700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f75440b4010 0x7f7544142490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:56.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.273+0000 7f75491fa700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7544143a40 con 0x7f7544146160 2022-06-29T15:58:56.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.273+0000 7f75491fa700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75440b56d0 con 0x7f75440b4010 2022-06-29T15:58:56.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.273+0000 7f75491fa700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75440aee60 con 0x7f75440ae7c0 2022-06-29T15:58:56.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.273+0000 7f753bfff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f75440b4010 0x7f7544142490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:56.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.273+0000 7f753bfff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f75440b4010 0x7f7544142490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46138/0 (socket says 172.21.15.60:46138) 2022-06-29T15:58:56.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.273+0000 7f753bfff700 1 -- 172.21.15.60:0/1686219793 learned_addr learned my addr 172.21.15.60:0/1686219793 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:56.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.273+0000 7f7542d9d700 1 --2- 172.21.15.60:0/1686219793 >> v2:172.21.15.92:3300/0 conn(0x7f75440ae7c0 0x7f7544141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:56.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.273+0000 7f754259c700 1 -- 172.21.15.60:0/1686219793 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1756718431 0 0) 0x7f7544143a40 con 0x7f7544146160 2022-06-29T15:58:56.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.273+0000 7f754259c700 1 -- 172.21.15.60:0/1686219793 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7524003260 con 0x7f7544146160 2022-06-29T15:58:56.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.273+0000 7f753bfff700 1 -- 172.21.15.60:0/1686219793 >> v2:172.21.15.92:3300/0 conn(0x7f75440ae7c0 msgr2=0x7f7544141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:56.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.273+0000 7f753bfff700 1 --2- 172.21.15.60:0/1686219793 >> v2:172.21.15.92:3300/0 conn(0x7f75440ae7c0 0x7f7544141fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:56.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.273+0000 7f753bfff700 1 -- 172.21.15.60:0/1686219793 >> v1:172.21.15.92:6789/0 conn(0x7f7544146160 legacy=0x7f7544146590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:56.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.273+0000 7f753bfff700 1 -- 172.21.15.60:0/1686219793 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75440af100 con 0x7f75440b4010 2022-06-29T15:58:56.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.273+0000 7f753bfff700 1 --2- 172.21.15.60:0/1686219793 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75440b4010 0x7f7544142490 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f752c002f70 tx=0x7f752c006eb0).ready entity=mon.0 client_cookie=c5afbc0a1209578 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:56.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.273+0000 7f754259c700 1 -- 172.21.15.60:0/1686219793 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f752c00d610 con 0x7f75440b4010 2022-06-29T15:58:56.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.273+0000 7f754259c700 1 -- 172.21.15.60:0/1686219793 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f752c000e90 con 0x7f75440b4010 2022-06-29T15:58:56.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.273+0000 7f754259c700 1 -- 172.21.15.60:0/1686219793 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f752c00d9a0 con 0x7f75440b4010 2022-06-29T15:58:56.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f75491fa700 1 -- 172.21.15.60:0/1686219793 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75440b4010 msgr2=0x7f7544142490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:56.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f75491fa700 1 --2- 172.21.15.60:0/1686219793 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75440b4010 0x7f7544142490 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f752c002f70 tx=0x7f752c006eb0).stop 2022-06-29T15:58:56.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f75491fa700 1 -- 172.21.15.60:0/1686219793 shutdown_connections 2022-06-29T15:58:56.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f75491fa700 1 --2- 172.21.15.60:0/1686219793 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75440b4010 0x7f7544142490 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:56.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f75491fa700 1 --2- 172.21.15.60:0/1686219793 >> v2:172.21.15.92:3300/0 conn(0x7f75440ae7c0 0x7f7544141fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:56.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f75491fa700 1 -- 172.21.15.60:0/1686219793 >> 172.21.15.60:0/1686219793 conn(0x7f75440b9340 msgr2=0x7f75440b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:56.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f75491fa700 1 -- 172.21.15.60:0/1686219793 shutdown_connections 2022-06-29T15:58:56.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f75491fa700 1 -- 172.21.15.60:0/1686219793 wait complete. 2022-06-29T15:58:56.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f75491fa700 1 Processor -- start 2022-06-29T15:58:56.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f75491fa700 1 -- start start 2022-06-29T15:58:56.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f75491fa700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75440ae7c0 0x7f75440af4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:56.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f75491fa700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75440b4010 0x7f7544149ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:56.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f75491fa700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7544146160 0x7f7544156270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:56.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f75491fa700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75440af9d0 con 0x7f75440ae7c0 2022-06-29T15:58:56.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f75491fa700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75440afbf0 con 0x7f7544146160 2022-06-29T15:58:56.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f75491fa700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7544155b30 con 0x7f75440b4010 2022-06-29T15:58:56.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f7542d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75440ae7c0 0x7f75440af4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:56.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f7542d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75440ae7c0 0x7f75440af4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46144/0 (socket says 172.21.15.60:46144) 2022-06-29T15:58:56.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f7542d9d700 1 -- 172.21.15.60:0/984959411 learned_addr learned my addr 172.21.15.60:0/984959411 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:56.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f753bfff700 1 --2- 172.21.15.60:0/984959411 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75440b4010 0x7f7544149ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:56.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f754359e700 1 --2- 172.21.15.60:0/984959411 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7544146160 0x7f7544156270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:56.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f7542d9d700 1 -- 172.21.15.60:0/984959411 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75440b4010 msgr2=0x7f7544149ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:56.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f7542d9d700 1 --2- 172.21.15.60:0/984959411 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75440b4010 0x7f7544149ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:56.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f7542d9d700 1 -- 172.21.15.60:0/984959411 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7544146160 msgr2=0x7f7544156270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:56.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f7542d9d700 1 --2- 172.21.15.60:0/984959411 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7544146160 0x7f7544156270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:56.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f7542d9d700 1 -- 172.21.15.60:0/984959411 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f752c00b1e0 con 0x7f75440ae7c0 2022-06-29T15:58:56.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f753bfff700 1 --2- 172.21.15.60:0/984959411 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75440b4010 0x7f7544149ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T15:58:56.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f754359e700 1 --2- 172.21.15.60:0/984959411 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7544146160 0x7f7544156270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T15:58:56.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f7542d9d700 1 --2- 172.21.15.60:0/984959411 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75440ae7c0 0x7f75440af4e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f75440afb10 tx=0x7f7534006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:56.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f7540d99700 1 -- 172.21.15.60:0/984959411 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f753400c830 con 0x7f75440ae7c0 2022-06-29T15:58:56.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f7540d99700 1 -- 172.21.15.60:0/984959411 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f753400c990 con 0x7f75440ae7c0 2022-06-29T15:58:56.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f75491fa700 1 -- 172.21.15.60:0/984959411 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7544155dd0 con 0x7f75440ae7c0 2022-06-29T15:58:56.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f7540d99700 1 -- 172.21.15.60:0/984959411 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f753400ccf0 con 0x7f75440ae7c0 2022-06-29T15:58:56.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.277+0000 7f75491fa700 1 -- 172.21.15.60:0/984959411 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7544112530 con 0x7f75440ae7c0 2022-06-29T15:58:56.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.281+0000 7f7540d99700 1 -- 172.21.15.60:0/984959411 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f753401c460 con 0x7f75440ae7c0 2022-06-29T15:58:56.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.281+0000 7f75491fa700 1 -- 172.21.15.60:0/984959411 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7544155f60 con 0x7f75440ae7c0 2022-06-29T15:58:56.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.281+0000 7f7540d99700 1 --2- 172.21.15.60:0/984959411 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7528053a70 0x7f7528055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:56.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.281+0000 7f7540d99700 1 -- 172.21.15.60:0/984959411 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f753406c4f0 con 0x7f75440ae7c0 2022-06-29T15:58:56.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.281+0000 7f753bfff700 1 --2- 172.21.15.60:0/984959411 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7528053a70 0x7f7528055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:56.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.281+0000 7f753bfff700 1 --2- 172.21.15.60:0/984959411 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7528053a70 0x7f7528055f30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f752c017000 tx=0x7f752c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:56.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.285+0000 7f7540d99700 1 -- 172.21.15.60:0/984959411 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f75340432e0 con 0x7f75440ae7c0 2022-06-29T15:58:56.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.449+0000 7f75491fa700 1 -- 172.21.15.60:0/984959411 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f75440573d0 con 0x7f75440ae7c0 2022-06-29T15:58:56.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.876+0000 7f7540d99700 1 -- 172.21.15.60:0/984959411 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v6) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7534046900 con 0x7f75440ae7c0 2022-06-29T15:58:56.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.880+0000 7f75491fa700 1 -- 172.21.15.60:0/984959411 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7528053a70 msgr2=0x7f7528055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:56.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.880+0000 7f75491fa700 1 --2- 172.21.15.60:0/984959411 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7528053a70 0x7f7528055f30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f752c017000 tx=0x7f752c015000).stop 2022-06-29T15:58:56.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.880+0000 7f75491fa700 1 -- 172.21.15.60:0/984959411 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75440ae7c0 msgr2=0x7f75440af4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:56.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.880+0000 7f75491fa700 1 --2- 172.21.15.60:0/984959411 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75440ae7c0 0x7f75440af4e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f75440afb10 tx=0x7f7534006df0).stop 2022-06-29T15:58:56.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.880+0000 7f75491fa700 1 -- 172.21.15.60:0/984959411 shutdown_connections 2022-06-29T15:58:56.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.880+0000 7f75491fa700 1 --2- 172.21.15.60:0/984959411 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7528053a70 0x7f7528055f30 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:56.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.880+0000 7f75491fa700 1 --2- 172.21.15.60:0/984959411 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7544146160 0x7f7544156270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:56.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.880+0000 7f75491fa700 1 --2- 172.21.15.60:0/984959411 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75440b4010 0x7f7544149ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:56.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.880+0000 7f75491fa700 1 --2- 172.21.15.60:0/984959411 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75440ae7c0 0x7f75440af4e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:56.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.880+0000 7f75491fa700 1 -- 172.21.15.60:0/984959411 >> 172.21.15.60:0/984959411 conn(0x7f75440b9340 msgr2=0x7f754405bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:56.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.880+0000 7f75491fa700 1 -- 172.21.15.60:0/984959411 shutdown_connections 2022-06-29T15:58:56.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:56.880+0000 7f75491fa700 1 -- 172.21.15.60:0/984959411 wait complete. 2022-06-29T15:58:56.906 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-06-29T15:58:57.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.020+0000 7faca0477700 1 Processor -- start 2022-06-29T15:58:57.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.020+0000 7faca0477700 1 -- start start 2022-06-29T15:58:57.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.024+0000 7faca0477700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fac9813dff0 0x7fac980b2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:57.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.024+0000 7faca0477700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac98061620 0x7fac98140a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:57.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.024+0000 7faca0477700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac98143ff0 con 0x7fac980b6710 2022-06-29T15:58:57.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.024+0000 7faca0477700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac98061e50 con 0x7fac9813dff0 2022-06-29T15:58:57.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.024+0000 7faca0477700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac98144220 con 0x7fac98061620 2022-06-29T15:58:57.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.024+0000 7fac9e213700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fac9813dff0 0x7fac980b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:57.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.024+0000 7fac9e213700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fac9813dff0 0x7fac980b2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46152/0 (socket says 172.21.15.60:46152) 2022-06-29T15:58:57.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.024+0000 7fac9e213700 1 -- 172.21.15.60:0/4247158174 learned_addr learned my addr 172.21.15.60:0/4247158174 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:57.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.024+0000 7fac9da12700 1 --2- 172.21.15.60:0/4247158174 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac98061620 0x7fac98140a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:57.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.024+0000 7fac9e213700 1 -- 172.21.15.60:0/4247158174 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac98061620 msgr2=0x7fac98140a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:57.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.024+0000 7fac9e213700 1 --2- 172.21.15.60:0/4247158174 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac98061620 0x7fac98140a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:57.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.024+0000 7fac9e213700 1 -- 172.21.15.60:0/4247158174 >> v1:172.21.15.92:6789/0 conn(0x7fac980b6710 legacy=0x7fac981446c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:57.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.024+0000 7fac9e213700 1 -- 172.21.15.60:0/4247158174 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac9813e3d0 con 0x7fac9813dff0 2022-06-29T15:58:57.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.024+0000 7fac9e213700 1 --2- 172.21.15.60:0/4247158174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac9813dff0 0x7fac980b2110 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fac94002f70 tx=0x7fac94006eb0).ready entity=mon.0 client_cookie=6d051dac21ab4fca server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:57.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.024+0000 7fac9d211700 1 -- 172.21.15.60:0/4247158174 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac9400d570 con 0x7fac9813dff0 2022-06-29T15:58:57.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.024+0000 7fac9d211700 1 -- 172.21.15.60:0/4247158174 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac94000e90 con 0x7fac9813dff0 2022-06-29T15:58:57.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.024+0000 7fac9d211700 1 -- 172.21.15.60:0/4247158174 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac9400d900 con 0x7fac9813dff0 2022-06-29T15:58:57.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.024+0000 7faca0477700 1 -- 172.21.15.60:0/4247158174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac9813dff0 msgr2=0x7fac980b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:57.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.024+0000 7faca0477700 1 --2- 172.21.15.60:0/4247158174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac9813dff0 0x7fac980b2110 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fac94002f70 tx=0x7fac94006eb0).stop 2022-06-29T15:58:57.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.024+0000 7faca0477700 1 -- 172.21.15.60:0/4247158174 shutdown_connections 2022-06-29T15:58:57.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.024+0000 7faca0477700 1 --2- 172.21.15.60:0/4247158174 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac98061620 0x7fac98140a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:57.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.024+0000 7faca0477700 1 --2- 172.21.15.60:0/4247158174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac9813dff0 0x7fac980b2110 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:57.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.024+0000 7faca0477700 1 -- 172.21.15.60:0/4247158174 >> 172.21.15.60:0/4247158174 conn(0x7fac980643e0 msgr2=0x7fac980b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:57.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.024+0000 7faca0477700 1 -- 172.21.15.60:0/4247158174 shutdown_connections 2022-06-29T15:58:57.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.024+0000 7faca0477700 1 -- 172.21.15.60:0/4247158174 wait complete. 2022-06-29T15:58:57.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.028+0000 7faca0477700 1 Processor -- start 2022-06-29T15:58:57.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.028+0000 7faca0477700 1 -- start start 2022-06-29T15:58:57.042 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.028+0000 7faca0477700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac98061620 0x7fac98065a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:57.042 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.028+0000 7faca0477700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac980b6710 0x7fac9813c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:57.042 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.028+0000 7faca0477700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac9813dff0 0x7fac9813c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:57.042 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.028+0000 7faca0477700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac980aaf60 con 0x7fac9813dff0 2022-06-29T15:58:57.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.028+0000 7faca0477700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac98098b60 con 0x7fac980b6710 2022-06-29T15:58:57.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.028+0000 7faca0477700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac980988c0 con 0x7fac98061620 2022-06-29T15:58:57.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.028+0000 7fac9ea14700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac9813dff0 0x7fac9813c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:57.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.028+0000 7fac9e213700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac98061620 0x7fac98065a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:57.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.028+0000 7fac9ea14700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac9813dff0 0x7fac9813c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46160/0 (socket says 172.21.15.60:46160) 2022-06-29T15:58:57.044 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.028+0000 7fac9ea14700 1 -- 172.21.15.60:0/3126988964 learned_addr learned my addr 172.21.15.60:0/3126988964 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:57.044 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.028+0000 7fac9da12700 1 --2- 172.21.15.60:0/3126988964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac980b6710 0x7fac9813c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:57.044 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.028+0000 7fac9e213700 1 -- 172.21.15.60:0/3126988964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac980b6710 msgr2=0x7fac9813c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:57.044 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.028+0000 7fac9e213700 1 --2- 172.21.15.60:0/3126988964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac980b6710 0x7fac9813c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:57.045 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.028+0000 7fac9e213700 1 -- 172.21.15.60:0/3126988964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac9813dff0 msgr2=0x7fac9813c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:57.045 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.028+0000 7fac9e213700 1 --2- 172.21.15.60:0/3126988964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac9813dff0 0x7fac9813c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:57.045 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.028+0000 7fac9e213700 1 -- 172.21.15.60:0/3126988964 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac9400b1e0 con 0x7fac98061620 2022-06-29T15:58:57.045 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.028+0000 7fac9ea14700 1 --2- 172.21.15.60:0/3126988964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac9813dff0 0x7fac9813c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:58:57.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.032+0000 7fac9e213700 1 --2- 172.21.15.60:0/3126988964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac98061620 0x7fac98065a30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fac94002f70 tx=0x7fac94015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:57.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.032+0000 7fac877fe700 1 -- 172.21.15.60:0/3126988964 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac94011760 con 0x7fac98061620 2022-06-29T15:58:57.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.032+0000 7fac877fe700 1 -- 172.21.15.60:0/3126988964 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac940118c0 con 0x7fac98061620 2022-06-29T15:58:57.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.032+0000 7fac877fe700 1 -- 172.21.15.60:0/3126988964 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac94011c20 con 0x7fac98061620 2022-06-29T15:58:57.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.032+0000 7faca0477700 1 -- 172.21.15.60:0/3126988964 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac9813cb70 con 0x7fac98061620 2022-06-29T15:58:57.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.032+0000 7faca0477700 1 -- 172.21.15.60:0/3126988964 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac98067250 con 0x7fac98061620 2022-06-29T15:58:57.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.032+0000 7faca0477700 1 -- 172.21.15.60:0/3126988964 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac980573d0 con 0x7fac98061620 2022-06-29T15:58:57.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.032+0000 7fac877fe700 1 -- 172.21.15.60:0/3126988964 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fac9401a040 con 0x7fac98061620 2022-06-29T15:58:57.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.032+0000 7fac877fe700 1 --2- 172.21.15.60:0/3126988964 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fac78053950 0x7fac78055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:57.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.032+0000 7fac877fe700 1 -- 172.21.15.60:0/3126988964 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fac9406cca0 con 0x7fac98061620 2022-06-29T15:58:57.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.036+0000 7fac9da12700 1 --2- 172.21.15.60:0/3126988964 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fac78053950 0x7fac78055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:57.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.036+0000 7fac9da12700 1 --2- 172.21.15.60:0/3126988964 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fac78053950 0x7fac78055e10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fac8800b000 tx=0x7fac88006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:57.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.044+0000 7fac877fe700 1 -- 172.21.15.60:0/3126988964 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fac94071070 con 0x7fac98061620 2022-06-29T15:58:57.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.172+0000 7faca0477700 1 -- 172.21.15.60:0/3126988964 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fac98067990 con 0x7fac98061620 2022-06-29T15:58:57.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.888+0000 7fac877fe700 1 -- 172.21.15.60:0/3126988964 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v25) v1 ==== 185+0+0 (secure 0 0 0) 0x7fac94047650 con 0x7fac98061620 2022-06-29T15:58:57.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.892+0000 7faca0477700 1 -- 172.21.15.60:0/3126988964 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fac78053950 msgr2=0x7fac78055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:57.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.892+0000 7faca0477700 1 --2- 172.21.15.60:0/3126988964 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fac78053950 0x7fac78055e10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fac8800b000 tx=0x7fac88006df0).stop 2022-06-29T15:58:57.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.892+0000 7faca0477700 1 -- 172.21.15.60:0/3126988964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac98061620 msgr2=0x7fac98065a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:57.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.892+0000 7faca0477700 1 --2- 172.21.15.60:0/3126988964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac98061620 0x7fac98065a30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fac94002f70 tx=0x7fac94015000).stop 2022-06-29T15:58:57.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.892+0000 7faca0477700 1 -- 172.21.15.60:0/3126988964 shutdown_connections 2022-06-29T15:58:57.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.892+0000 7faca0477700 1 --2- 172.21.15.60:0/3126988964 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fac78053950 0x7fac78055e10 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:57.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.892+0000 7faca0477700 1 --2- 172.21.15.60:0/3126988964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac9813dff0 0x7fac9813c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:57.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.892+0000 7faca0477700 1 --2- 172.21.15.60:0/3126988964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac980b6710 0x7fac9813c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:57.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.892+0000 7faca0477700 1 --2- 172.21.15.60:0/3126988964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac98061620 0x7fac98065a30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:57.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.892+0000 7faca0477700 1 -- 172.21.15.60:0/3126988964 >> 172.21.15.60:0/3126988964 conn(0x7fac980643e0 msgr2=0x7fac980b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:57.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.892+0000 7faca0477700 1 -- 172.21.15.60:0/3126988964 shutdown_connections 2022-06-29T15:58:57.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:57.892+0000 7faca0477700 1 -- 172.21.15.60:0/3126988964 wait complete. 2022-06-29T15:58:57.903 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2022-06-29T15:58:57.913 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-06-29T15:58:58.023 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T15:58:58.026 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T15:58:58.029 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T15:58:58.032 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T15:58:58.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.024+0000 7f14bcb7d700 1 Processor -- start 2022-06-29T15:58:58.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.024+0000 7f14bcb7d700 1 -- start start 2022-06-29T15:58:58.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.028+0000 7f14bcb7d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f14b80b5e90 0x7f14b80b6270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:58.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.028+0000 7f14bcb7d700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f14b8060710 0x7f14b80ae9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:58.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.028+0000 7f14bcb7d700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14b8060c60 con 0x7f14b8142d20 2022-06-29T15:58:58.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.028+0000 7f14bcb7d700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14b80b1d30 con 0x7f14b80b5e90 2022-06-29T15:58:58.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.028+0000 7f14bcb7d700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14b8060e90 con 0x7f14b8060710 2022-06-29T15:58:58.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.028+0000 7f14b659c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f14b80b5e90 0x7f14b80b6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:58.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.028+0000 7f14b659c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f14b80b5e90 0x7f14b80b6270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46166/0 (socket says 172.21.15.60:46166) 2022-06-29T15:58:58.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.028+0000 7f14b659c700 1 -- 172.21.15.60:0/2335518925 learned_addr learned my addr 172.21.15.60:0/2335518925 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:58.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.028+0000 7f14add9b700 1 --2- 172.21.15.60:0/2335518925 >> v2:172.21.15.92:3300/0 conn(0x7f14b8060710 0x7f14b80ae9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:58.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.028+0000 7f14b659c700 1 -- 172.21.15.60:0/2335518925 >> v2:172.21.15.92:3300/0 conn(0x7f14b8060710 msgr2=0x7f14b80ae9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:58.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.028+0000 7f14b659c700 1 --2- 172.21.15.60:0/2335518925 >> v2:172.21.15.92:3300/0 conn(0x7f14b8060710 0x7f14b80ae9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:58.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.028+0000 7f14b659c700 1 -- 172.21.15.60:0/2335518925 >> v1:172.21.15.60:6789/0 conn(0x7f14b8142d20 legacy=0x7f14b81431b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:58.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.028+0000 7f14b659c700 1 -- 172.21.15.60:0/2335518925 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b80ae2e0 con 0x7f14b80b5e90 2022-06-29T15:58:58.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.028+0000 7f14b659c700 1 --2- 172.21.15.60:0/2335518925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14b80b5e90 0x7f14b80b6270 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f14b80b20f0 tx=0x7f14a8006f40).ready entity=mon.0 client_cookie=964fde3a51d09223 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:58.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.028+0000 7f14b5d9b700 1 -- 172.21.15.60:0/2335518925 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14a800da40 con 0x7f14b80b5e90 2022-06-29T15:58:58.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.028+0000 7f14b5d9b700 1 -- 172.21.15.60:0/2335518925 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a8000e90 con 0x7f14b80b5e90 2022-06-29T15:58:58.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.028+0000 7f14b5d9b700 1 -- 172.21.15.60:0/2335518925 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14a800ddd0 con 0x7f14b80b5e90 2022-06-29T15:58:58.042 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T15:58:58.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.028+0000 7f14bcb7d700 1 -- 172.21.15.60:0/2335518925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14b80b5e90 msgr2=0x7f14b80b6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:58.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.028+0000 7f14bcb7d700 1 --2- 172.21.15.60:0/2335518925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14b80b5e90 0x7f14b80b6270 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f14b80b20f0 tx=0x7f14a8006f40).stop 2022-06-29T15:58:58.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.028+0000 7f14bcb7d700 1 -- 172.21.15.60:0/2335518925 shutdown_connections 2022-06-29T15:58:58.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.028+0000 7f14bcb7d700 1 --2- 172.21.15.60:0/2335518925 >> v2:172.21.15.92:3300/0 conn(0x7f14b8060710 0x7f14b80ae9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:58.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.028+0000 7f14bcb7d700 1 --2- 172.21.15.60:0/2335518925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14b80b5e90 0x7f14b80b6270 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:58.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.028+0000 7f14bcb7d700 1 -- 172.21.15.60:0/2335518925 >> 172.21.15.60:0/2335518925 conn(0x7f14b8063b50 msgr2=0x7f14b80b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:58.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.028+0000 7f14bcb7d700 1 -- 172.21.15.60:0/2335518925 shutdown_connections 2022-06-29T15:58:58.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.028+0000 7f14bcb7d700 1 -- 172.21.15.60:0/2335518925 wait complete. 2022-06-29T15:58:58.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.032+0000 7f14bcb7d700 1 Processor -- start 2022-06-29T15:58:58.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.032+0000 7f14bcb7d700 1 -- start start 2022-06-29T15:58:58.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.032+0000 7f14bcb7d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14b8060710 0x7f14b813d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:58.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.032+0000 7f14bcb7d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14b80b5e90 0x7f14b8151780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:58.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.032+0000 7f14bcb7d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14b8142d20 0x7f14b814dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:58.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.032+0000 7f14bcb7d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b8151c70 con 0x7f14b8142d20 2022-06-29T15:58:58.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.032+0000 7f14bcb7d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b813bb80 con 0x7f14b8060710 2022-06-29T15:58:58.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.032+0000 7f14bcb7d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14b813b640 con 0x7f14b80b5e90 2022-06-29T15:58:58.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.032+0000 7f14b6d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14b8142d20 0x7f14b814dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:58.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.032+0000 7f14b6d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14b8142d20 0x7f14b814dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46174/0 (socket says 172.21.15.60:46174) 2022-06-29T15:58:58.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.032+0000 7f14b6d9d700 1 -- 172.21.15.60:0/4008747635 learned_addr learned my addr 172.21.15.60:0/4008747635 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:58.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.032+0000 7f14b659c700 1 --2- 172.21.15.60:0/4008747635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14b8060710 0x7f14b813d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:58.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.032+0000 7f14add9b700 1 --2- 172.21.15.60:0/4008747635 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14b80b5e90 0x7f14b8151780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:58.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.032+0000 7f14b6d9d700 1 -- 172.21.15.60:0/4008747635 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14b80b5e90 msgr2=0x7f14b8151780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:58.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.032+0000 7f14b6d9d700 1 --2- 172.21.15.60:0/4008747635 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14b80b5e90 0x7f14b8151780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:58.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.032+0000 7f14b6d9d700 1 -- 172.21.15.60:0/4008747635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14b8060710 msgr2=0x7f14b813d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:58.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.032+0000 7f14b6d9d700 1 --2- 172.21.15.60:0/4008747635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14b8060710 0x7f14b813d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:58.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.032+0000 7f14b6d9d700 1 -- 172.21.15.60:0/4008747635 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a800b460 con 0x7f14b8142d20 2022-06-29T15:58:58.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.032+0000 7f14b6d9d700 1 --2- 172.21.15.60:0/4008747635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14b8142d20 0x7f14b814dae0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f14b813bd10 tx=0x7f14b00013a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:58.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.032+0000 7f14affff700 1 -- 172.21.15.60:0/4008747635 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14b0006aa0 con 0x7f14b8142d20 2022-06-29T15:58:58.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.032+0000 7f14affff700 1 -- 172.21.15.60:0/4008747635 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b0006c00 con 0x7f14b8142d20 2022-06-29T15:58:58.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.032+0000 7f14bcb7d700 1 -- 172.21.15.60:0/4008747635 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14b813b8e0 con 0x7f14b8142d20 2022-06-29T15:58:58.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.032+0000 7f14affff700 1 -- 172.21.15.60:0/4008747635 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14b0007030 con 0x7f14b8142d20 2022-06-29T15:58:58.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.032+0000 7f14bcb7d700 1 -- 172.21.15.60:0/4008747635 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14b813bf70 con 0x7f14b8142d20 2022-06-29T15:58:58.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.036+0000 7f14bcb7d700 1 -- 172.21.15.60:0/4008747635 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14b80ae550 con 0x7f14b8142d20 2022-06-29T15:58:58.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.036+0000 7f14affff700 1 -- 172.21.15.60:0/4008747635 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f14b0010030 con 0x7f14b8142d20 2022-06-29T15:58:58.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.036+0000 7f14affff700 1 --2- 172.21.15.60:0/4008747635 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f149c053950 0x7f149c055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:58.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.036+0000 7f14affff700 1 -- 172.21.15.60:0/4008747635 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f14b006bc70 con 0x7f14b8142d20 2022-06-29T15:58:58.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.036+0000 7f14b659c700 1 --2- 172.21.15.60:0/4008747635 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f149c053950 0x7f149c055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:58.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.036+0000 7f14b659c700 1 --2- 172.21.15.60:0/4008747635 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f149c053950 0x7f149c055e10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f14a801e000 tx=0x7f14a8006f40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:58.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.040+0000 7f14affff700 1 -- 172.21.15.60:0/4008747635 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f14b0042be0 con 0x7f14b8142d20 2022-06-29T15:58:58.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.164+0000 7f14bcb7d700 1 -- 172.21.15.60:0/4008747635 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f14b80573d0 con 0x7f14b8142d20 2022-06-29T15:58:58.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.899+0000 7f14affff700 1 -- 172.21.15.60:0/4008747635 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v26) v1 ==== 173+0+0 (secure 0 0 0) 0x7f14b0046200 con 0x7f14b8142d20 2022-06-29T15:58:58.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.903+0000 7f14bcb7d700 1 -- 172.21.15.60:0/4008747635 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f149c053950 msgr2=0x7f149c055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:58.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.903+0000 7f14bcb7d700 1 --2- 172.21.15.60:0/4008747635 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f149c053950 0x7f149c055e10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f14a801e000 tx=0x7f14a8006f40).stop 2022-06-29T15:58:58.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.903+0000 7f14bcb7d700 1 -- 172.21.15.60:0/4008747635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14b8142d20 msgr2=0x7f14b814dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:58.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.903+0000 7f14bcb7d700 1 --2- 172.21.15.60:0/4008747635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14b8142d20 0x7f14b814dae0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f14b813bd10 tx=0x7f14b00013a0).stop 2022-06-29T15:58:58.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.903+0000 7f14bcb7d700 1 -- 172.21.15.60:0/4008747635 shutdown_connections 2022-06-29T15:58:58.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.903+0000 7f14bcb7d700 1 --2- 172.21.15.60:0/4008747635 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f149c053950 0x7f149c055e10 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:58.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.903+0000 7f14bcb7d700 1 --2- 172.21.15.60:0/4008747635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14b8142d20 0x7f14b814dae0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:58.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.903+0000 7f14bcb7d700 1 --2- 172.21.15.60:0/4008747635 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14b80b5e90 0x7f14b8151780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:58.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.903+0000 7f14bcb7d700 1 --2- 172.21.15.60:0/4008747635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14b8060710 0x7f14b813d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:58.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.903+0000 7f14bcb7d700 1 -- 172.21.15.60:0/4008747635 >> 172.21.15.60:0/4008747635 conn(0x7f14b8063b50 msgr2=0x7f14b813f140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:58.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.903+0000 7f14bcb7d700 1 -- 172.21.15.60:0/4008747635 shutdown_connections 2022-06-29T15:58:58.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:58.903+0000 7f14bcb7d700 1 -- 172.21.15.60:0/4008747635 wait complete. 2022-06-29T15:58:58.915 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2022-06-29T15:58:58.926 INFO:tasks.ceph.mds.a:Restarting daemon 2022-06-29T15:58:58.927 INFO:tasks.ceph.mds.a:Stopping old one... 2022-06-29T15:58:58.927 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-06-29T15:58:58.928 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T15:58:58.928 INFO:tasks.ceph.mds.b:Restarting daemon 2022-06-29T15:58:58.928 INFO:tasks.ceph.mds.b:Stopping old one... 2022-06-29T15:58:58.929 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-06-29T15:58:58.929 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T15:58:58.942 INFO:tasks.ceph.mds.a:Stopped 2022-06-29T15:58:58.942 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-06-29T15:58:58.945 INFO:tasks.ceph.mds.b:Stopped 2022-06-29T15:58:58.945 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-06-29T15:58:58.948 INFO:tasks.ceph.mds.a:Started 2022-06-29T15:58:58.949 DEBUG:teuthology.parallel:result is None 2022-06-29T15:58:58.990 INFO:tasks.ceph.mds.b:Started 2022-06-29T15:58:58.990 DEBUG:teuthology.parallel:result is None 2022-06-29T15:58:58.991 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-06-29T15:58:59.021 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2022-06-29T15:58:59.065 INFO:tasks.ceph.mds.b.smithi092.stdout:starting mds.b at 2022-06-29T15:58:59.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.091+0000 7fa9c8912700 1 Processor -- start 2022-06-29T15:58:59.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.091+0000 7fa9c8912700 1 -- start start 2022-06-29T15:58:59.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.091+0000 7fa9c8912700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa9c40b5eb0 0x7fa9c4142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:59.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.091+0000 7fa9c8912700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa9c40b48b0 0x7fa9c40b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:59.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.091+0000 7fa9c8912700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa9c40b2c50 0x7fa9c40b3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:59.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.091+0000 7fa9c8912700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9c40b54e0 con 0x7fa9c40b48b0 2022-06-29T15:58:59.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.091+0000 7fa9c8912700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9c40b4d10 con 0x7fa9c40b2c50 2022-06-29T15:58:59.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.091+0000 7fa9c8912700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c40b6290 con 0x7fa9c40b5eb0 2022-06-29T15:58:59.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.091+0000 7fa9baffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa9c40b48b0 0x7fa9c40b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:59.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.091+0000 7fa9baffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa9c40b48b0 0x7fa9c40b05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46194/0 (socket says 172.21.15.60:46194) 2022-06-29T15:58:59.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.091+0000 7fa9baffd700 1 -- 172.21.15.60:0/529369669 learned_addr learned my addr 172.21.15.60:0/529369669 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:59.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.091+0000 7fa9c37fe700 1 --2- 172.21.15.60:0/529369669 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa9c40b5eb0 0x7fa9c4142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:59.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.091+0000 7fa9c3fff700 1 --2- 172.21.15.60:0/529369669 >> v2:172.21.15.92:3300/0 conn(0x7fa9c40b2c50 0x7fa9c40b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:59.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.091+0000 7fa9baffd700 1 -- 172.21.15.60:0/529369669 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa9c40b5eb0 msgr2=0x7fa9c4142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:59.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.091+0000 7fa9baffd700 1 --2- 172.21.15.60:0/529369669 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa9c40b5eb0 0x7fa9c4142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:59.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.091+0000 7fa9baffd700 1 -- 172.21.15.60:0/529369669 >> v2:172.21.15.92:3300/0 conn(0x7fa9c40b2c50 msgr2=0x7fa9c40b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:58:59.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.091+0000 7fa9baffd700 1 --2- 172.21.15.60:0/529369669 >> v2:172.21.15.92:3300/0 conn(0x7fa9c40b2c50 0x7fa9c40b3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:59.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.091+0000 7fa9baffd700 1 -- 172.21.15.60:0/529369669 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c4142f10 con 0x7fa9c40b48b0 2022-06-29T15:58:59.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.091+0000 7fa9baffd700 1 --2- 172.21.15.60:0/529369669 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9c40b48b0 0x7fa9c40b05e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fa9ac002f70 tx=0x7fa9ac006eb0).ready entity=mon.0 client_cookie=b04b94df2309e77c server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:59.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.095+0000 7fa9c2ffd700 1 -- 172.21.15.60:0/529369669 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9ac00d570 con 0x7fa9c40b48b0 2022-06-29T15:58:59.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.095+0000 7fa9c2ffd700 1 -- 172.21.15.60:0/529369669 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9ac000e90 con 0x7fa9c40b48b0 2022-06-29T15:58:59.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.095+0000 7fa9c2ffd700 1 -- 172.21.15.60:0/529369669 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9ac00d900 con 0x7fa9c40b48b0 2022-06-29T15:58:59.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.095+0000 7fa9c8912700 1 -- 172.21.15.60:0/529369669 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9c40b48b0 msgr2=0x7fa9c40b05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:59.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.095+0000 7fa9c8912700 1 --2- 172.21.15.60:0/529369669 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9c40b48b0 0x7fa9c40b05e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fa9ac002f70 tx=0x7fa9ac006eb0).stop 2022-06-29T15:58:59.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.095+0000 7fa9c8912700 1 -- 172.21.15.60:0/529369669 shutdown_connections 2022-06-29T15:58:59.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.095+0000 7fa9c8912700 1 --2- 172.21.15.60:0/529369669 >> v2:172.21.15.92:3300/0 conn(0x7fa9c40b2c50 0x7fa9c40b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:59.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.095+0000 7fa9c8912700 1 --2- 172.21.15.60:0/529369669 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9c40b48b0 0x7fa9c40b05e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:59.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.095+0000 7fa9c8912700 1 --2- 172.21.15.60:0/529369669 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa9c40b5eb0 0x7fa9c4142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:59.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.095+0000 7fa9c8912700 1 -- 172.21.15.60:0/529369669 >> 172.21.15.60:0/529369669 conn(0x7fa9c4062090 msgr2=0x7fa9c4062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:58:59.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.095+0000 7fa9c8912700 1 -- 172.21.15.60:0/529369669 shutdown_connections 2022-06-29T15:58:59.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.095+0000 7fa9c8912700 1 -- 172.21.15.60:0/529369669 wait complete. 2022-06-29T15:58:59.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.095+0000 7fa9c8912700 1 Processor -- start 2022-06-29T15:58:59.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.095+0000 7fa9c8912700 1 -- start start 2022-06-29T15:58:59.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.095+0000 7fa9c8912700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa9c40b2c50 0x7fa9c40b80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:59.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.095+0000 7fa9c8912700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa9c40b48b0 0x7fa9c413a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:59.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.095+0000 7fa9c8912700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9c40b5eb0 0x7fa9c4151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:59.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.095+0000 7fa9c8912700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c41468a0 con 0x7fa9c40b5eb0 2022-06-29T15:58:59.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.095+0000 7fa9c8912700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c4155da0 con 0x7fa9c40b2c50 2022-06-29T15:58:59.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.095+0000 7fa9c8912700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c4146ad0 con 0x7fa9c40b48b0 2022-06-29T15:58:59.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.099+0000 7fa9c3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9c40b5eb0 0x7fa9c4151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:59.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.099+0000 7fa9baffd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa9c40b48b0 0x7fa9c413a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:59.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.099+0000 7fa9c3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9c40b5eb0 0x7fa9c4151a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46202/0 (socket says 172.21.15.60:46202) 2022-06-29T15:58:59.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.099+0000 7fa9c3fff700 1 -- 172.21.15.60:0/2085084779 learned_addr learned my addr 172.21.15.60:0/2085084779 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:58:59.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.099+0000 7fa9c37fe700 1 --2- 172.21.15.60:0/2085084779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa9c40b2c50 0x7fa9c40b80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:59.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.099+0000 7fa9c3fff700 1 -- 172.21.15.60:0/2085084779 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa9c40b48b0 msgr2=0x7fa9c413a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:59.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.099+0000 7fa9c3fff700 1 --2- 172.21.15.60:0/2085084779 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa9c40b48b0 0x7fa9c413a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:59.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.099+0000 7fa9c3fff700 1 -- 172.21.15.60:0/2085084779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa9c40b2c50 msgr2=0x7fa9c40b80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:58:59.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.099+0000 7fa9c3fff700 1 --2- 172.21.15.60:0/2085084779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa9c40b2c50 0x7fa9c40b80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:58:59.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.099+0000 7fa9c3fff700 1 -- 172.21.15.60:0/2085084779 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9ac00b1e0 con 0x7fa9c40b5eb0 2022-06-29T15:58:59.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.099+0000 7fa9c3fff700 1 --2- 172.21.15.60:0/2085084779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9c40b5eb0 0x7fa9c4151a40 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fa9bc00a000 tx=0x7fa9bc008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:59.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.099+0000 7fa9c17fa700 1 -- 172.21.15.60:0/2085084779 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9bc013620 con 0x7fa9c40b5eb0 2022-06-29T15:58:59.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.099+0000 7fa9c17fa700 1 -- 172.21.15.60:0/2085084779 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9bc013780 con 0x7fa9c40b5eb0 2022-06-29T15:58:59.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.099+0000 7fa9c8912700 1 -- 172.21.15.60:0/2085084779 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9c4155a20 con 0x7fa9c40b5eb0 2022-06-29T15:58:59.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.099+0000 7fa9c17fa700 1 -- 172.21.15.60:0/2085084779 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9bc013ae0 con 0x7fa9c40b5eb0 2022-06-29T15:58:59.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.099+0000 7fa9c8912700 1 -- 172.21.15.60:0/2085084779 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9c40ae950 con 0x7fa9c40b5eb0 2022-06-29T15:58:59.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.099+0000 7fa9c8912700 1 -- 172.21.15.60:0/2085084779 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9c40b4f70 con 0x7fa9c40b5eb0 2022-06-29T15:58:59.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.099+0000 7fa9c17fa700 1 -- 172.21.15.60:0/2085084779 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa9bc013c40 con 0x7fa9c40b5eb0 2022-06-29T15:58:59.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.099+0000 7fa9c17fa700 1 --2- 172.21.15.60:0/2085084779 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa9a8053a70 0x7fa9a8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:58:59.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.099+0000 7fa9c17fa700 1 -- 172.21.15.60:0/2085084779 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fa9bc06bd70 con 0x7fa9c40b5eb0 2022-06-29T15:58:59.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.099+0000 7fa9c37fe700 1 --2- 172.21.15.60:0/2085084779 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa9a8053a70 0x7fa9a8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:58:59.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.103+0000 7fa9c37fe700 1 --2- 172.21.15.60:0/2085084779 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa9a8053a70 0x7fa9a8055f30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fa9b400b000 tx=0x7fa9b4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:58:59.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.107+0000 7fa9c17fa700 1 -- 172.21.15.60:0/2085084779 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa9bc042e70 con 0x7fa9c40b5eb0 2022-06-29T15:58:59.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:58:59.239+0000 7fa9c8912700 1 -- 172.21.15.60:0/2085084779 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fa9c4146c90 con 0x7fa9c40b5eb0 2022-06-29T15:59:00.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.095+0000 7fa9c17fa700 1 -- 172.21.15.60:0/2085084779 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v28) v1 ==== 97+0+0 (secure 0 0 0) 0x7fa9bc046490 con 0x7fa9c40b5eb0 2022-06-29T15:59:00.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.099+0000 7fa9c8912700 1 -- 172.21.15.60:0/2085084779 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa9a8053a70 msgr2=0x7fa9a8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:00.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.099+0000 7fa9c8912700 1 --2- 172.21.15.60:0/2085084779 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa9a8053a70 0x7fa9a8055f30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fa9b400b000 tx=0x7fa9b4006df0).stop 2022-06-29T15:59:00.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.099+0000 7fa9c8912700 1 -- 172.21.15.60:0/2085084779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9c40b5eb0 msgr2=0x7fa9c4151a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:00.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.099+0000 7fa9c8912700 1 --2- 172.21.15.60:0/2085084779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9c40b5eb0 0x7fa9c4151a40 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fa9bc00a000 tx=0x7fa9bc008040).stop 2022-06-29T15:59:00.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.099+0000 7fa9c8912700 1 -- 172.21.15.60:0/2085084779 shutdown_connections 2022-06-29T15:59:00.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.099+0000 7fa9c8912700 1 --2- 172.21.15.60:0/2085084779 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa9a8053a70 0x7fa9a8055f30 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:00.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.099+0000 7fa9c8912700 1 --2- 172.21.15.60:0/2085084779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9c40b5eb0 0x7fa9c4151a40 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:00.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.099+0000 7fa9c8912700 1 --2- 172.21.15.60:0/2085084779 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa9c40b48b0 0x7fa9c413a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:00.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.099+0000 7fa9c8912700 1 --2- 172.21.15.60:0/2085084779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa9c40b2c50 0x7fa9c40b80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:00.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.099+0000 7fa9c8912700 1 -- 172.21.15.60:0/2085084779 >> 172.21.15.60:0/2085084779 conn(0x7fa9c4062090 msgr2=0x7fa9c4062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:00.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.099+0000 7fa9c8912700 1 -- 172.21.15.60:0/2085084779 shutdown_connections 2022-06-29T15:59:00.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.099+0000 7fa9c8912700 1 -- 172.21.15.60:0/2085084779 wait complete. 2022-06-29T15:59:00.113 INFO:teuthology.orchestra.run.smithi060.stderr: removed all blacklist entries 2022-06-29T15:59:00.125 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-06-29T15:59:00.126 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-06-29T15:59:00.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.263+0000 7fd8eb167700 1 Processor -- start 2022-06-29T15:59:00.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.263+0000 7fd8eb167700 1 -- start start 2022-06-29T15:59:00.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8eb167700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd8e40b5eb0 0x7fd8e4142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:00.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8eb167700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8e40b48b0 0x7fd8e40b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:00.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8eb167700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8e40b4d10 con 0x7fd8e40b2c50 2022-06-29T15:59:00.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8eb167700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8e40b54e0 con 0x7fd8e40b5eb0 2022-06-29T15:59:00.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8eb167700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e40b6290 con 0x7fd8e40b48b0 2022-06-29T15:59:00.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8ea966700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd8e40b2c50 0x7fd8e40b3090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54298/0 (socket says 172.21.15.60:54298) 2022-06-29T15:59:00.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8ea966700 1 -- 172.21.15.60:0/4212095996 learned_addr learned my addr 172.21.15.60:0/4212095996 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:59:00.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8e9964700 1 --2- 172.21.15.60:0/4212095996 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8e40b48b0 0x7fd8e40b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:00.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8ea165700 1 --2- 172.21.15.60:0/4212095996 >> v2:172.21.15.92:3300/0 conn(0x7fd8e40b5eb0 0x7fd8e4142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:00.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8e9163700 1 -- 172.21.15.60:0/4212095996 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609312193 0 0) 0x7fd8e40b4d10 con 0x7fd8e40b2c50 2022-06-29T15:59:00.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8e9163700 1 -- 172.21.15.60:0/4212095996 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8cc003260 con 0x7fd8e40b2c50 2022-06-29T15:59:00.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8e9163700 1 -- 172.21.15.60:0/4212095996 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2084936503 0 0) 0x7fd8cc003260 con 0x7fd8e40b2c50 2022-06-29T15:59:00.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8e9163700 1 -- 172.21.15.60:0/4212095996 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8e40b48b0 msgr2=0x7fd8e40b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:00.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8e9163700 1 --2- 172.21.15.60:0/4212095996 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8e40b48b0 0x7fd8e40b05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:00.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8e9163700 1 -- 172.21.15.60:0/4212095996 >> v2:172.21.15.92:3300/0 conn(0x7fd8e40b5eb0 msgr2=0x7fd8e4142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:00.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8e9163700 1 --2- 172.21.15.60:0/4212095996 >> v2:172.21.15.92:3300/0 conn(0x7fd8e40b5eb0 0x7fd8e4142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:00.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8e9163700 1 -- 172.21.15.60:0/4212095996 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e4142f10 con 0x7fd8e40b2c50 2022-06-29T15:59:00.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8e9163700 1 -- 172.21.15.60:0/4212095996 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7fd8dc0019e0 con 0x7fd8e40b2c50 2022-06-29T15:59:00.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8e9163700 1 -- 172.21.15.60:0/4212095996 >> v1:172.21.15.60:6789/0 conn(0x7fd8e40b2c50 legacy=0x7fd8e40b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:00.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8e9163700 1 --2- 172.21.15.60:0/4212095996 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8e40b48b0 0x7fd8cc003750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:00.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8e9163700 1 --2- 172.21.15.60:0/4212095996 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd8cc003c80 0x7fd8cc004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:00.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8e9163700 1 --2- 172.21.15.60:0/4212095996 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8cc0049f0 0x7fd8cc004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:00.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8e9163700 1 -- 172.21.15.60:0/4212095996 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e40b54e0 con 0x7fd8cc0049f0 2022-06-29T15:59:00.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8e9163700 1 -- 172.21.15.60:0/4212095996 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e40b6290 con 0x7fd8cc003c80 2022-06-29T15:59:00.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8e9163700 1 -- 172.21.15.60:0/4212095996 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8cc00b750 con 0x7fd8e40b48b0 2022-06-29T15:59:00.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8ea966700 1 --2- 172.21.15.60:0/4212095996 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8cc0049f0 0x7fd8cc004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:00.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8ea165700 1 --2- 172.21.15.60:0/4212095996 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8e40b48b0 0x7fd8cc003750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:00.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8e9964700 1 --2- 172.21.15.60:0/4212095996 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd8cc003c80 0x7fd8cc004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:00.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8ea966700 1 -- 172.21.15.60:0/4212095996 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8e40b48b0 msgr2=0x7fd8cc003750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:00.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8ea966700 1 --2- 172.21.15.60:0/4212095996 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8e40b48b0 0x7fd8cc003750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:00.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8ea966700 1 -- 172.21.15.60:0/4212095996 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd8cc003c80 msgr2=0x7fd8cc004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:00.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8ea966700 1 --2- 172.21.15.60:0/4212095996 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd8cc003c80 0x7fd8cc004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:00.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8ea966700 1 -- 172.21.15.60:0/4212095996 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8cc003260 con 0x7fd8cc0049f0 2022-06-29T15:59:00.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8ea966700 1 --2- 172.21.15.60:0/4212095996 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8cc0049f0 0x7fd8cc004e70 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fd8cc0054d0 tx=0x7fd8dc0021d0).ready entity=mon.0 client_cookie=3260f0662b0a6675 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:00.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8e9163700 1 -- 172.21.15.60:0/4212095996 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8dc00cc70 con 0x7fd8cc0049f0 2022-06-29T15:59:00.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8e9163700 1 -- 172.21.15.60:0/4212095996 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8dc00cdd0 con 0x7fd8cc0049f0 2022-06-29T15:59:00.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.267+0000 7fd8e9163700 1 -- 172.21.15.60:0/4212095996 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8dc005b80 con 0x7fd8cc0049f0 2022-06-29T15:59:00.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.271+0000 7fd8eb167700 1 -- 172.21.15.60:0/4212095996 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8cc0049f0 msgr2=0x7fd8cc004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:00.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.271+0000 7fd8eb167700 1 --2- 172.21.15.60:0/4212095996 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8cc0049f0 0x7fd8cc004e70 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fd8cc0054d0 tx=0x7fd8dc0021d0).stop 2022-06-29T15:59:00.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.271+0000 7fd8ea966700 1 -- 172.21.15.60:0/4212095996 reap_dead start 2022-06-29T15:59:00.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.271+0000 7fd8eb167700 1 -- 172.21.15.60:0/4212095996 shutdown_connections 2022-06-29T15:59:00.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.271+0000 7fd8eb167700 1 -- 172.21.15.60:0/4212095996 >> 172.21.15.60:0/4212095996 conn(0x7fd8e4062090 msgr2=0x7fd8e4062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:00.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.271+0000 7fd8eb167700 1 -- 172.21.15.60:0/4212095996 shutdown_connections 2022-06-29T15:59:00.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.271+0000 7fd8eb167700 1 -- 172.21.15.60:0/4212095996 wait complete. 2022-06-29T15:59:00.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.271+0000 7fd8eb167700 1 Processor -- start 2022-06-29T15:59:00.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.271+0000 7fd8eb167700 1 -- start start 2022-06-29T15:59:00.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.275+0000 7fd8eb167700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8e40b8580 0x7fd8e4146880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:00.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.275+0000 7fd8eb167700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd8e4151ab0 0x7fd8e4140ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:00.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.275+0000 7fd8eb167700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8e4140fc0 0x7fd8e41553d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:00.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.275+0000 7fd8eb167700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e4141470 con 0x7fd8e4140fc0 2022-06-29T15:59:00.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.275+0000 7fd8eb167700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e40b8960 con 0x7fd8e4151ab0 2022-06-29T15:59:00.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.275+0000 7fd8eb167700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e4145a40 con 0x7fd8e40b8580 2022-06-29T15:59:00.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.275+0000 7fd8ea966700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8e4140fc0 0x7fd8e41553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:00.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.275+0000 7fd8ea966700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8e4140fc0 0x7fd8e41553d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46224/0 (socket says 172.21.15.60:46224) 2022-06-29T15:59:00.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.275+0000 7fd8e9964700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd8e4151ab0 0x7fd8e4140ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:00.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.275+0000 7fd8ea966700 1 -- 172.21.15.60:0/277340672 learned_addr learned my addr 172.21.15.60:0/277340672 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:00.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.275+0000 7fd8ea165700 1 --2- 172.21.15.60:0/277340672 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8e40b8580 0x7fd8e4146880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:00.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.275+0000 7fd8ea966700 1 -- 172.21.15.60:0/277340672 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8e40b8580 msgr2=0x7fd8e4146880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:00.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.275+0000 7fd8ea966700 1 --2- 172.21.15.60:0/277340672 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8e40b8580 0x7fd8e4146880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:00.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.275+0000 7fd8ea966700 1 -- 172.21.15.60:0/277340672 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd8e4151ab0 msgr2=0x7fd8e4140ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:00.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.275+0000 7fd8ea966700 1 --2- 172.21.15.60:0/277340672 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd8e4151ab0 0x7fd8e4140ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:00.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.275+0000 7fd8ea966700 1 -- 172.21.15.60:0/277340672 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e4144920 con 0x7fd8e4140fc0 2022-06-29T15:59:00.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.275+0000 7fd8ea966700 1 --2- 172.21.15.60:0/277340672 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8e4140fc0 0x7fd8e41553d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fd8e406a720 tx=0x7fd8dc015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:00.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.275+0000 7fd8d77fe700 1 -- 172.21.15.60:0/277340672 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8dc018030 con 0x7fd8e4140fc0 2022-06-29T15:59:00.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.275+0000 7fd8d77fe700 1 -- 172.21.15.60:0/277340672 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8dc0072c0 con 0x7fd8e4140fc0 2022-06-29T15:59:00.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.275+0000 7fd8eb167700 1 -- 172.21.15.60:0/277340672 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8e4144520 con 0x7fd8e4140fc0 2022-06-29T15:59:00.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.275+0000 7fd8d77fe700 1 -- 172.21.15.60:0/277340672 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8dc00cc70 con 0x7fd8e4140fc0 2022-06-29T15:59:00.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.275+0000 7fd8eb167700 1 -- 172.21.15.60:0/277340672 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8e4155a60 con 0x7fd8e4140fc0 2022-06-29T15:59:00.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.275+0000 7fd8eb167700 1 -- 172.21.15.60:0/277340672 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8e4145bb0 con 0x7fd8e4140fc0 2022-06-29T15:59:00.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.279+0000 7fd8d77fe700 1 -- 172.21.15.60:0/277340672 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd8dc01d070 con 0x7fd8e4140fc0 2022-06-29T15:59:00.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.279+0000 7fd8d77fe700 1 --2- 172.21.15.60:0/277340672 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd8c4053a70 0x7fd8c4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:00.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.279+0000 7fd8d77fe700 1 -- 172.21.15.60:0/277340672 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fd8dc06c0a0 con 0x7fd8e4140fc0 2022-06-29T15:59:00.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.279+0000 7fd8ea165700 1 --2- 172.21.15.60:0/277340672 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd8c4053a70 0x7fd8c4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:00.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.279+0000 7fd8ea165700 1 --2- 172.21.15.60:0/277340672 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd8c4053a70 0x7fd8c4055f30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fd8d800b000 tx=0x7fd8d8006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:00.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.283+0000 7fd8d77fe700 1 -- 172.21.15.60:0/277340672 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd8e4145bb0 con 0x7fd8e4140fc0 2022-06-29T15:59:00.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.447+0000 7fd8eb167700 1 -- 172.21.15.60:0/277340672 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd8e4145bb0 con 0x7fd8e4140fc0 2022-06-29T15:59:00.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.451+0000 7fd8d77fe700 1 -- 172.21.15.60:0/277340672 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v6) v1 ==== 80+0+5627 (secure 0 0 0) 0x7fd8e4145bb0 con 0x7fd8e4140fc0 2022-06-29T15:59:00.460 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:59:00.460 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T15:59:00.460 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_dump": [ 2022-06-29T15:59:00.460 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:59:00.461 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.a", 2022-06-29T15:59:00.461 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxivv7JNhAAnA291UxlB/SfgITOipP00A==", 2022-06-29T15:59:00.461 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T15:59:00.461 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T15:59:00.462 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T15:59:00.462 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T15:59:00.462 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T15:59:00.462 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:59:00.463 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:59:00.463 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:59:00.463 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.b", 2022-06-29T15:59:00.463 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxiRSr/OhAAfFQTF2YHvzCv5wXBVCA5WQ==", 2022-06-29T15:59:00.464 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T15:59:00.464 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T15:59:00.464 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T15:59:00.464 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T15:59:00.465 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T15:59:00.465 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:59:00.466 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:59:00.466 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:59:00.467 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.0", 2022-06-29T15:59:00.467 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiZYPxOBAAIkweWJlTJIXgxnlagkGdXA==", 2022-06-29T15:59:00.467 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T15:59:00.468 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T15:59:00.468 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T15:59:00.469 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T15:59:00.469 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:59:00.470 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:59:00.470 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:59:00.471 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.1", 2022-06-29T15:59:00.471 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxiS2JGBhAAJn0UfAxZI3IVFyD5QBdb/Q==", 2022-06-29T15:59:00.471 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T15:59:00.472 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T15:59:00.472 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T15:59:00.473 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T15:59:00.473 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:59:00.474 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:59:00.474 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:59:00.474 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.2", 2022-06-29T15:59:00.475 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxiPhkfEhAApH7r8OUJzW7fZNM6XVv1dg==", 2022-06-29T15:59:00.475 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T15:59:00.475 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T15:59:00.476 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T15:59:00.476 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T15:59:00.477 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:59:00.477 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:59:00.477 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:59:00.478 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.3", 2022-06-29T15:59:00.478 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxit/47HhAA1vyU6h7hjPG+op9qP/rVYQ==", 2022-06-29T15:59:00.478 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T15:59:00.478 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T15:59:00.479 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T15:59:00.479 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T15:59:00.479 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:59:00.479 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:59:00.480 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:59:00.480 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.4", 2022-06-29T15:59:00.480 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxiaKzKBxAAbIwCF9ag7ezsNy11hi0BEw==", 2022-06-29T15:59:00.480 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T15:59:00.481 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T15:59:00.481 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T15:59:00.481 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T15:59:00.481 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:59:00.481 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:59:00.482 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:59:00.482 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.5", 2022-06-29T15:59:00.482 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxiJQV/IBAASBBDb6UFgEtXILe4fcT/xA==", 2022-06-29T15:59:00.482 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T15:59:00.483 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T15:59:00.483 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T15:59:00.483 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T15:59:00.483 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:59:00.484 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:59:00.484 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:59:00.484 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.6", 2022-06-29T15:59:00.485 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxi6D07OBAA2FWiZ+mNK+jFMp/kPZnbQQ==", 2022-06-29T15:59:00.485 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T15:59:00.485 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T15:59:00.485 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T15:59:00.486 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T15:59:00.486 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:59:00.486 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:59:00.486 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:59:00.487 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.7", 2022-06-29T15:59:00.487 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCMdrxinEOcEhAA562ybpzGHNQfWIq8q7zw9w==", 2022-06-29T15:59:00.487 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T15:59:00.487 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T15:59:00.488 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T15:59:00.488 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T15:59:00.488 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:59:00.488 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:59:00.489 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:59:00.489 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.0", 2022-06-29T15:59:00.489 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiHXPwBRAAkOOwlYJvIW7dqVFdlXXoqw==", 2022-06-29T15:59:00.489 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T15:59:00.489 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T15:59:00.490 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2022-06-29T15:59:00.490 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2022-06-29T15:59:00.490 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2022-06-29T15:59:00.490 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:59:00.491 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:59:00.491 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:59:00.491 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.1", 2022-06-29T15:59:00.491 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxibe5JCRAAGBJbi9t09w0836JqrAT7+w==", 2022-06-29T15:59:00.492 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T15:59:00.492 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T15:59:00.492 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2022-06-29T15:59:00.493 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2022-06-29T15:59:00.493 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2022-06-29T15:59:00.494 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:59:00.494 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:59:00.494 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:59:00.495 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.2", 2022-06-29T15:59:00.495 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiK3dBDRAA7XINPSOPQDfOrUYEpHC+gw==", 2022-06-29T15:59:00.496 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T15:59:00.496 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T15:59:00.496 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow r", 2022-06-29T15:59:00.497 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow rw", 2022-06-29T15:59:00.497 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rwx" 2022-06-29T15:59:00.498 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:59:00.498 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:59:00.499 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:59:00.499 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.admin", 2022-06-29T15:59:00.500 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxieKVcGBAAnjH80a0qWIArz2qwjYHWdQ==", 2022-06-29T15:59:00.500 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T15:59:00.501 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T15:59:00.501 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T15:59:00.501 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T15:59:00.502 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T15:59:00.502 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:59:00.503 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:59:00.503 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:59:00.503 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.bootstrap-mds", 2022-06-29T15:59:00.504 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCPdrxiPn+hORAAeij8EEAJX64eBYiv2DtDtg==", 2022-06-29T15:59:00.504 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T15:59:00.504 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile bootstrap-mds" 2022-06-29T15:59:00.504 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:59:00.505 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:59:00.505 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:59:00.505 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.bootstrap-mgr", 2022-06-29T15:59:00.505 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCPdrxippehORAApHlX39gb7QPew6bSZ9MR6w==", 2022-06-29T15:59:00.506 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T15:59:00.506 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile bootstrap-mgr" 2022-06-29T15:59:00.514 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:59:00.514 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:59:00.515 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:59:00.515 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.bootstrap-osd", 2022-06-29T15:59:00.515 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCPdrxiFqyhORAAEAViYmf24Sx6S4DRRXByQA==", 2022-06-29T15:59:00.515 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T15:59:00.516 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile bootstrap-osd" 2022-06-29T15:59:00.516 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:59:00.516 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:59:00.516 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:59:00.517 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.bootstrap-rbd", 2022-06-29T15:59:00.517 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCPdrxibsChORAAsgW7RVglQ/QayO53YYBPfA==", 2022-06-29T15:59:00.517 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T15:59:00.517 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile bootstrap-rbd" 2022-06-29T15:59:00.517 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:59:00.518 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:59:00.518 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:59:00.518 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.bootstrap-rbd-mirror", 2022-06-29T15:59:00.518 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCPdrxib9ehORAAM7+07dfhHRjiog8K2Wr4wA==", 2022-06-29T15:59:00.519 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T15:59:00.519 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2022-06-29T15:59:00.519 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:59:00.519 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:59:00.520 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:59:00.520 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.bootstrap-rgw", 2022-06-29T15:59:00.520 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCPdrxiJO2hORAAThH1r787kht3/XBRXfpMWQ==", 2022-06-29T15:59:00.520 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T15:59:00.520 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile bootstrap-rgw" 2022-06-29T15:59:00.521 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:59:00.521 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:59:00.521 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:59:00.521 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.x", 2022-06-29T15:59:00.522 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxi2LZJNBAA3MKb/pXYbQCsoRx8EQfGbw==", 2022-06-29T15:59:00.522 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T15:59:00.522 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T15:59:00.522 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2022-06-29T15:59:00.522 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T15:59:00.523 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:59:00.523 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:59:00.524 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:59:00.524 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.y", 2022-06-29T15:59:00.524 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxi3LmaMBAAN+0mZrdKXqSXOipVTA1drw==", 2022-06-29T15:59:00.525 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T15:59:00.525 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T15:59:00.526 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2022-06-29T15:59:00.526 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T15:59:00.527 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:59:00.527 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:59:00.527 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T15:59:00.528 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T15:59:00.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.451+0000 7fd8eb167700 1 -- 172.21.15.60:0/277340672 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd8c4053a70 msgr2=0x7fd8c4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:00.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.451+0000 7fd8eb167700 1 --2- 172.21.15.60:0/277340672 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd8c4053a70 0x7fd8c4055f30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fd8d800b000 tx=0x7fd8d8006df0).stop 2022-06-29T15:59:00.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.451+0000 7fd8eb167700 1 -- 172.21.15.60:0/277340672 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8e4140fc0 msgr2=0x7fd8e41553d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:00.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.451+0000 7fd8eb167700 1 --2- 172.21.15.60:0/277340672 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8e4140fc0 0x7fd8e41553d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fd8e406a720 tx=0x7fd8dc015000).stop 2022-06-29T15:59:00.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.451+0000 7fd8eb167700 1 -- 172.21.15.60:0/277340672 shutdown_connections 2022-06-29T15:59:00.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.451+0000 7fd8eb167700 1 --2- 172.21.15.60:0/277340672 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd8c4053a70 0x7fd8c4055f30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:00.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.451+0000 7fd8eb167700 1 --2- 172.21.15.60:0/277340672 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8e4140fc0 0x7fd8e41553d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:00.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.451+0000 7fd8eb167700 1 --2- 172.21.15.60:0/277340672 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd8e4151ab0 0x7fd8e4140ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:00.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.451+0000 7fd8eb167700 1 --2- 172.21.15.60:0/277340672 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd8e40b8580 0x7fd8e4146880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:00.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.451+0000 7fd8eb167700 1 -- 172.21.15.60:0/277340672 >> 172.21.15.60:0/277340672 conn(0x7fd8e4062090 msgr2=0x7fd8e4141b30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:00.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.451+0000 7fd8eb167700 1 -- 172.21.15.60:0/277340672 shutdown_connections 2022-06-29T15:59:00.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.451+0000 7fd8eb167700 1 -- 172.21.15.60:0/277340672 wait complete. 2022-06-29T15:59:00.535 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2022-06-29T15:59:00.646 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.635+0000 7f029e161700 1 Processor -- start 2022-06-29T15:59:00.646 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.635+0000 7f029e161700 1 -- start start 2022-06-29T15:59:00.647 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.635+0000 7f029e161700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0298063ad0 0x7f0298067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:00.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.635+0000 7f029e161700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f029814af10 0x7f029805c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:00.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.635+0000 7f029e161700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02980acba0 con 0x7f02980b5950 2022-06-29T15:59:00.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.639+0000 7f029e161700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f029814b3a0 con 0x7f0298063ad0 2022-06-29T15:59:00.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.639+0000 7f029e161700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0298067220 con 0x7f029814af10 2022-06-29T15:59:00.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.639+0000 7f0296ffd700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0298063ad0 0x7f0298067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:00.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.639+0000 7f0297fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f029814af10 0x7f029805c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:00.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.639+0000 7f0296ffd700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0298063ad0 0x7f0298067680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:52258/0 (socket says 172.21.15.60:52258) 2022-06-29T15:59:00.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.639+0000 7f0296ffd700 1 -- 172.21.15.60:0/438243585 learned_addr learned my addr 172.21.15.60:0/438243585 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:00.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.639+0000 7f0297fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f029814af10 0x7f029805c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:59326/0 (socket says 172.21.15.60:59326) 2022-06-29T15:59:00.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.639+0000 7f0296ffd700 1 -- 172.21.15.60:0/438243585 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f029814af10 msgr2=0x7f029805c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:00.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.639+0000 7f0296ffd700 1 --2- 172.21.15.60:0/438243585 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f029814af10 0x7f029805c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:00.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.639+0000 7f0296ffd700 1 -- 172.21.15.60:0/438243585 >> v1:172.21.15.92:6789/0 conn(0x7f02980b5950 legacy=0x7f029805bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:00.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.639+0000 7f0296ffd700 1 -- 172.21.15.60:0/438243585 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02980b5d30 con 0x7f0298063ad0 2022-06-29T15:59:00.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.639+0000 7f0296ffd700 1 --2- 172.21.15.60:0/438243585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0298063ad0 0x7f0298067680 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f0288002f70 tx=0x7f0288006eb0).ready entity=mon.1 client_cookie=66be05c6efe43228 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:00.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.639+0000 7f02967fc700 1 -- 172.21.15.60:0/438243585 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f028800d570 con 0x7f0298063ad0 2022-06-29T15:59:00.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.639+0000 7f02967fc700 1 -- 172.21.15.60:0/438243585 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0288000e90 con 0x7f0298063ad0 2022-06-29T15:59:00.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.639+0000 7f02967fc700 1 -- 172.21.15.60:0/438243585 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f028800d900 con 0x7f0298063ad0 2022-06-29T15:59:00.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.639+0000 7f029e161700 1 -- 172.21.15.60:0/438243585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0298063ad0 msgr2=0x7f0298067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:00.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.639+0000 7f029e161700 1 --2- 172.21.15.60:0/438243585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0298063ad0 0x7f0298067680 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f0288002f70 tx=0x7f0288006eb0).stop 2022-06-29T15:59:00.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.639+0000 7f029e161700 1 -- 172.21.15.60:0/438243585 shutdown_connections 2022-06-29T15:59:00.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.639+0000 7f029e161700 1 --2- 172.21.15.60:0/438243585 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f029814af10 0x7f029805c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:00.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.639+0000 7f029e161700 1 --2- 172.21.15.60:0/438243585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0298063ad0 0x7f0298067680 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:00.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.639+0000 7f029e161700 1 -- 172.21.15.60:0/438243585 >> 172.21.15.60:0/438243585 conn(0x7f02980632c0 msgr2=0x7f02980b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:00.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.639+0000 7f029e161700 1 -- 172.21.15.60:0/438243585 shutdown_connections 2022-06-29T15:59:00.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.639+0000 7f029e161700 1 -- 172.21.15.60:0/438243585 wait complete. 2022-06-29T15:59:00.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.643+0000 7f029e161700 1 Processor -- start 2022-06-29T15:59:00.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.643+0000 7f029e161700 1 -- start start 2022-06-29T15:59:00.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.643+0000 7f029e161700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0298063ad0 0x7f0298142780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:00.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.643+0000 7f029e161700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02980b5950 0x7f02980b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:00.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.643+0000 7f029e161700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f029814af10 0x7f02981183b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:00.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.643+0000 7f029e161700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029815a630 con 0x7f02980b5950 2022-06-29T15:59:00.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.643+0000 7f029e161700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02980b0280 con 0x7f029814af10 2022-06-29T15:59:00.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.643+0000 7f029e161700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02981596d0 con 0x7f0298063ad0 2022-06-29T15:59:00.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.643+0000 7f0296ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02980b5950 0x7f02980b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:00.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.643+0000 7f02977fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0298063ad0 0x7f0298142780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:00.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.643+0000 7f0296ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02980b5950 0x7f02980b3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46236/0 (socket says 172.21.15.60:46236) 2022-06-29T15:59:00.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.643+0000 7f02977fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0298063ad0 0x7f0298142780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:59328/0 (socket says 172.21.15.60:59328) 2022-06-29T15:59:00.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.643+0000 7f0296ffd700 1 -- 172.21.15.60:0/3672625462 learned_addr learned my addr 172.21.15.60:0/3672625462 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:00.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.643+0000 7f0297fff700 1 --2- 172.21.15.60:0/3672625462 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f029814af10 0x7f02981183b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:00.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.643+0000 7f0296ffd700 1 -- 172.21.15.60:0/3672625462 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0298063ad0 msgr2=0x7f0298142780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:00.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.643+0000 7f0296ffd700 1 --2- 172.21.15.60:0/3672625462 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0298063ad0 0x7f0298142780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:00.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.643+0000 7f0296ffd700 1 -- 172.21.15.60:0/3672625462 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f029814af10 msgr2=0x7f02981183b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:00.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.643+0000 7f0296ffd700 1 --2- 172.21.15.60:0/3672625462 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f029814af10 0x7f02981183b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:00.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.643+0000 7f0296ffd700 1 -- 172.21.15.60:0/3672625462 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f028800b1e0 con 0x7f02980b5950 2022-06-29T15:59:00.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.643+0000 7f0296ffd700 1 --2- 172.21.15.60:0/3672625462 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02980b5950 0x7f02980b3f80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f02981181e0 tx=0x7f0288015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:00.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.643+0000 7f0294ff9700 1 -- 172.21.15.60:0/3672625462 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0288006b40 con 0x7f02980b5950 2022-06-29T15:59:00.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.643+0000 7f0294ff9700 1 -- 172.21.15.60:0/3672625462 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f028800d8f0 con 0x7f02980b5950 2022-06-29T15:59:00.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.643+0000 7f0294ff9700 1 -- 172.21.15.60:0/3672625462 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0288011d30 con 0x7f02980b5950 2022-06-29T15:59:00.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.643+0000 7f029e161700 1 -- 172.21.15.60:0/3672625462 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02980af0c0 con 0x7f02980b5950 2022-06-29T15:59:00.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.643+0000 7f029e161700 1 -- 172.21.15.60:0/3672625462 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02980af320 con 0x7f02980b5950 2022-06-29T15:59:00.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.647+0000 7f029e161700 1 -- 172.21.15.60:0/3672625462 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02980af4e0 con 0x7f02980b5950 2022-06-29T15:59:00.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.647+0000 7f0294ff9700 1 -- 172.21.15.60:0/3672625462 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0288025030 con 0x7f02980b5950 2022-06-29T15:59:00.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.647+0000 7f0294ff9700 1 --2- 172.21.15.60:0/3672625462 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f027c053b30 0x7f027c055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:00.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.647+0000 7f0294ff9700 1 -- 172.21.15.60:0/3672625462 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f028806ccc0 con 0x7f02980b5950 2022-06-29T15:59:00.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.647+0000 7f02977fe700 1 --2- 172.21.15.60:0/3672625462 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f027c053b30 0x7f027c055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:00.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.647+0000 7f02977fe700 1 --2- 172.21.15.60:0/3672625462 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f027c053b30 0x7f027c055ff0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f028000d000 tx=0x7f0280009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:00.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.651+0000 7f0294ff9700 1 -- 172.21.15.60:0/3672625462 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0288043f10 con 0x7f02980b5950 2022-06-29T15:59:00.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.819+0000 7f029e161700 1 -- 172.21.15.60:0/3672625462 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f02980573d0 con 0x7f02980b5950 2022-06-29T15:59:00.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.823+0000 7f0294ff9700 1 -- 172.21.15.60:0/3672625462 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 updated v7) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0288023020 con 0x7f02980b5950 2022-06-29T15:59:00.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.827+0000 7f029e161700 1 -- 172.21.15.60:0/3672625462 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f027c053b30 msgr2=0x7f027c055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:00.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.827+0000 7f029e161700 1 --2- 172.21.15.60:0/3672625462 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f027c053b30 0x7f027c055ff0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f028000d000 tx=0x7f0280009040).stop 2022-06-29T15:59:00.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.827+0000 7f029e161700 1 -- 172.21.15.60:0/3672625462 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02980b5950 msgr2=0x7f02980b3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:00.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.827+0000 7f029e161700 1 --2- 172.21.15.60:0/3672625462 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02980b5950 0x7f02980b3f80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f02981181e0 tx=0x7f0288015000).stop 2022-06-29T15:59:00.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.827+0000 7f029e161700 1 -- 172.21.15.60:0/3672625462 shutdown_connections 2022-06-29T15:59:00.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.827+0000 7f029e161700 1 --2- 172.21.15.60:0/3672625462 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f027c053b30 0x7f027c055ff0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:00.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.827+0000 7f029e161700 1 --2- 172.21.15.60:0/3672625462 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f029814af10 0x7f02981183b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:00.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.827+0000 7f029e161700 1 --2- 172.21.15.60:0/3672625462 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02980b5950 0x7f02980b3f80 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:00.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.827+0000 7f029e161700 1 --2- 172.21.15.60:0/3672625462 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0298063ad0 0x7f0298142780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:00.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.827+0000 7f029e161700 1 -- 172.21.15.60:0/3672625462 >> 172.21.15.60:0/3672625462 conn(0x7f02980632c0 msgr2=0x7f029814a330 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:00.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.827+0000 7f029e161700 1 -- 172.21.15.60:0/3672625462 shutdown_connections 2022-06-29T15:59:00.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.827+0000 7f029e161700 1 -- 172.21.15.60:0/3672625462 wait complete. 2022-06-29T15:59:00.842 INFO:teuthology.orchestra.run.smithi060.stderr:updated 2022-06-29T15:59:00.851 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2022-06-29T15:59:00.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.951+0000 7fec2b59e700 1 Processor -- start 2022-06-29T15:59:00.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.951+0000 7fec2b59e700 1 -- start start 2022-06-29T15:59:00.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.951+0000 7fec2b59e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fec2c0b48b0 0x7fec2c142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:00.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.951+0000 7fec2b59e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fec2c0b2c50 0x7fec2c0b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:00.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.951+0000 7fec2b59e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec2c0b0cf0 con 0x7fec2c0b5eb0 2022-06-29T15:59:00.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.951+0000 7fec2b59e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec2c0b54e0 con 0x7fec2c0b48b0 2022-06-29T15:59:00.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.951+0000 7fec2b59e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec2c0b4d10 con 0x7fec2c0b2c50 2022-06-29T15:59:00.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.951+0000 7fec29d9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fec2c0b48b0 0x7fec2c142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:00.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.951+0000 7fec29d9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fec2c0b48b0 0x7fec2c142a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46244/0 (socket says 172.21.15.60:46244) 2022-06-29T15:59:00.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.951+0000 7fec29d9b700 1 -- 172.21.15.60:0/1418859284 learned_addr learned my addr 172.21.15.60:0/1418859284 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:00.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.951+0000 7fec2ad9d700 1 --2- 172.21.15.60:0/1418859284 >> v2:172.21.15.92:3300/0 conn(0x7fec2c0b2c50 0x7fec2c0b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:00.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.951+0000 7fec29d9b700 1 -- 172.21.15.60:0/1418859284 >> v2:172.21.15.92:3300/0 conn(0x7fec2c0b2c50 msgr2=0x7fec2c0b30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:00.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.951+0000 7fec29d9b700 1 --2- 172.21.15.60:0/1418859284 >> v2:172.21.15.92:3300/0 conn(0x7fec2c0b2c50 0x7fec2c0b30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:00.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.951+0000 7fec29d9b700 1 -- 172.21.15.60:0/1418859284 >> v1:172.21.15.60:6789/0 conn(0x7fec2c0b5eb0 legacy=0x7fec2c0b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:00.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.951+0000 7fec2959a700 1 -- 172.21.15.60:0/1418859284 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 574451461 0 0) 0x7fec14001110 con 0x7fec2c0b5eb0 2022-06-29T15:59:00.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.951+0000 7fec29d9b700 1 -- 172.21.15.60:0/1418859284 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec2c0b6290 con 0x7fec2c0b48b0 2022-06-29T15:59:00.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.951+0000 7fec2ad9d700 1 --2- 172.21.15.60:0/1418859284 >> v2:172.21.15.92:3300/0 conn(0x7fec2c0b2c50 0x7fec2c0b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T15:59:00.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.951+0000 7fec29d9b700 1 --2- 172.21.15.60:0/1418859284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec2c0b48b0 0x7fec2c142a20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fec240032f0 tx=0x7fec24009290).ready entity=mon.0 client_cookie=7e439b51180763d2 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:00.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.951+0000 7fec2959a700 1 -- 172.21.15.60:0/1418859284 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec2400b870 con 0x7fec2c0b48b0 2022-06-29T15:59:00.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.951+0000 7fec2959a700 1 -- 172.21.15.60:0/1418859284 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec2400b9d0 con 0x7fec2c0b48b0 2022-06-29T15:59:00.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.951+0000 7fec2959a700 1 -- 172.21.15.60:0/1418859284 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec2400bb40 con 0x7fec2c0b48b0 2022-06-29T15:59:00.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.951+0000 7fec2b59e700 1 -- 172.21.15.60:0/1418859284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec2c0b48b0 msgr2=0x7fec2c142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:00.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.951+0000 7fec2b59e700 1 --2- 172.21.15.60:0/1418859284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec2c0b48b0 0x7fec2c142a20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fec240032f0 tx=0x7fec24009290).stop 2022-06-29T15:59:00.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.951+0000 7fec2b59e700 1 -- 172.21.15.60:0/1418859284 shutdown_connections 2022-06-29T15:59:00.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.951+0000 7fec2b59e700 1 --2- 172.21.15.60:0/1418859284 >> v2:172.21.15.92:3300/0 conn(0x7fec2c0b2c50 0x7fec2c0b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:00.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.951+0000 7fec2b59e700 1 --2- 172.21.15.60:0/1418859284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec2c0b48b0 0x7fec2c142a20 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:00.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.951+0000 7fec2b59e700 1 -- 172.21.15.60:0/1418859284 >> 172.21.15.60:0/1418859284 conn(0x7fec2c062090 msgr2=0x7fec2c062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:00.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.951+0000 7fec2b59e700 1 -- 172.21.15.60:0/1418859284 shutdown_connections 2022-06-29T15:59:00.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.951+0000 7fec2b59e700 1 -- 172.21.15.60:0/1418859284 wait complete. 2022-06-29T15:59:00.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.955+0000 7fec2b59e700 1 Processor -- start 2022-06-29T15:59:00.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.955+0000 7fec2b59e700 1 -- start start 2022-06-29T15:59:00.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.955+0000 7fec2b59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec2c0b2c50 0x7fec2c155170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:00.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.955+0000 7fec2b59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec2c0b48b0 0x7fec2c0b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:00.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.955+0000 7fec2b59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec2c0b5eb0 0x7fec2c0aefb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:00.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.955+0000 7fec2b59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c140520 con 0x7fec2c0b48b0 2022-06-29T15:59:00.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.955+0000 7fec2b59e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c140d80 con 0x7fec2c0b5eb0 2022-06-29T15:59:00.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.955+0000 7fec2b59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c0b2390 con 0x7fec2c0b2c50 2022-06-29T15:59:00.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.955+0000 7fec29d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec2c0b48b0 0x7fec2c0b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:00.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.955+0000 7fec29d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec2c0b48b0 0x7fec2c0b1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46250/0 (socket says 172.21.15.60:46250) 2022-06-29T15:59:00.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.955+0000 7fec29d9b700 1 -- 172.21.15.60:0/3609122938 learned_addr learned my addr 172.21.15.60:0/3609122938 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:00.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.955+0000 7fec2ad9d700 1 --2- 172.21.15.60:0/3609122938 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec2c0b5eb0 0x7fec2c0aefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:00.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.955+0000 7fec2a59c700 1 --2- 172.21.15.60:0/3609122938 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec2c0b2c50 0x7fec2c155170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:00.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.955+0000 7fec29d9b700 1 -- 172.21.15.60:0/3609122938 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec2c0b2c50 msgr2=0x7fec2c155170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:00.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.955+0000 7fec29d9b700 1 --2- 172.21.15.60:0/3609122938 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec2c0b2c50 0x7fec2c155170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:00.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.955+0000 7fec29d9b700 1 -- 172.21.15.60:0/3609122938 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec2c0b5eb0 msgr2=0x7fec2c0aefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:00.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.955+0000 7fec29d9b700 1 --2- 172.21.15.60:0/3609122938 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec2c0b5eb0 0x7fec2c0aefb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:00.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.955+0000 7fec29d9b700 1 -- 172.21.15.60:0/3609122938 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec24009530 con 0x7fec2c0b48b0 2022-06-29T15:59:00.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.955+0000 7fec2a59c700 1 --2- 172.21.15.60:0/3609122938 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec2c0b2c50 0x7fec2c155170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:59:00.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.955+0000 7fec2ad9d700 1 --2- 172.21.15.60:0/3609122938 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec2c0b5eb0 0x7fec2c0aefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:59:00.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.955+0000 7fec29d9b700 1 --2- 172.21.15.60:0/3609122938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec2c0b48b0 0x7fec2c0b1ea0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fec240032f0 tx=0x7fec24016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:00.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.955+0000 7fec237fe700 1 -- 172.21.15.60:0/3609122938 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec24021630 con 0x7fec2c0b48b0 2022-06-29T15:59:00.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.955+0000 7fec237fe700 1 -- 172.21.15.60:0/3609122938 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec24021790 con 0x7fec2c0b48b0 2022-06-29T15:59:00.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.955+0000 7fec2b59e700 1 -- 172.21.15.60:0/3609122938 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec2c141220 con 0x7fec2c0b48b0 2022-06-29T15:59:00.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.955+0000 7fec237fe700 1 -- 172.21.15.60:0/3609122938 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec24021af0 con 0x7fec2c0b48b0 2022-06-29T15:59:00.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.955+0000 7fec2b59e700 1 -- 172.21.15.60:0/3609122938 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec2c0ada10 con 0x7fec2c0b48b0 2022-06-29T15:59:00.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.959+0000 7fec2b59e700 1 -- 172.21.15.60:0/3609122938 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec2c0573d0 con 0x7fec2c0b48b0 2022-06-29T15:59:00.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.959+0000 7fec237fe700 1 -- 172.21.15.60:0/3609122938 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fec24027020 con 0x7fec2c0b48b0 2022-06-29T15:59:00.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.959+0000 7fec237fe700 1 --2- 172.21.15.60:0/3609122938 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fec04053a70 0x7fec04055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:00.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.959+0000 7fec237fe700 1 -- 172.21.15.60:0/3609122938 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fec2406d510 con 0x7fec2c0b48b0 2022-06-29T15:59:00.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.959+0000 7fec2a59c700 1 --2- 172.21.15.60:0/3609122938 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fec04053a70 0x7fec04055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:00.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.959+0000 7fec2a59c700 1 --2- 172.21.15.60:0/3609122938 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fec04053a70 0x7fec04055f30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fec1400b000 tx=0x7fec14009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:00.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:00.967+0000 7fec237fe700 1 -- 172.21.15.60:0/3609122938 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fec24044760 con 0x7fec2c0b48b0 2022-06-29T15:59:01.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.135+0000 7fec2b59e700 1 -- 172.21.15.60:0/3609122938 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7fec2c140f40 con 0x7fec2c0b48b0 2022-06-29T15:59:01.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.143+0000 7fec237fe700 1 -- 172.21.15.60:0/3609122938 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 updated v8) v1 ==== 97+0+0 (secure 0 0 0) 0x7fec24015070 con 0x7fec2c0b48b0 2022-06-29T15:59:01.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.147+0000 7fec2b59e700 1 -- 172.21.15.60:0/3609122938 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fec04053a70 msgr2=0x7fec04055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:01.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.147+0000 7fec2b59e700 1 --2- 172.21.15.60:0/3609122938 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fec04053a70 0x7fec04055f30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fec1400b000 tx=0x7fec14009040).stop 2022-06-29T15:59:01.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.147+0000 7fec2b59e700 1 -- 172.21.15.60:0/3609122938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec2c0b48b0 msgr2=0x7fec2c0b1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:01.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.147+0000 7fec2b59e700 1 --2- 172.21.15.60:0/3609122938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec2c0b48b0 0x7fec2c0b1ea0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fec240032f0 tx=0x7fec24016000).stop 2022-06-29T15:59:01.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.147+0000 7fec2b59e700 1 -- 172.21.15.60:0/3609122938 shutdown_connections 2022-06-29T15:59:01.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.147+0000 7fec2b59e700 1 --2- 172.21.15.60:0/3609122938 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fec04053a70 0x7fec04055f30 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:01.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.147+0000 7fec2b59e700 1 --2- 172.21.15.60:0/3609122938 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec2c0b5eb0 0x7fec2c0aefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:01.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.147+0000 7fec2b59e700 1 --2- 172.21.15.60:0/3609122938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec2c0b48b0 0x7fec2c0b1ea0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:01.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.147+0000 7fec2b59e700 1 --2- 172.21.15.60:0/3609122938 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec2c0b2c50 0x7fec2c155170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:01.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.147+0000 7fec2b59e700 1 -- 172.21.15.60:0/3609122938 >> 172.21.15.60:0/3609122938 conn(0x7fec2c062090 msgr2=0x7fec2c143570 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:01.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.147+0000 7fec2b59e700 1 -- 172.21.15.60:0/3609122938 shutdown_connections 2022-06-29T15:59:01.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.147+0000 7fec2b59e700 1 -- 172.21.15.60:0/3609122938 wait complete. 2022-06-29T15:59:01.180 INFO:teuthology.orchestra.run.smithi060.stderr:updated 2022-06-29T15:59:01.180 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2022-06-29T15:59:01.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.275+0000 7f82894f6700 1 Processor -- start 2022-06-29T15:59:01.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.275+0000 7f82894f6700 1 -- start start 2022-06-29T15:59:01.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.275+0000 7f82894f6700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f828405fe60 0x7f828413ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:01.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.275+0000 7f82894f6700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f828405d2c0 con 0x7f82840affe0 2022-06-29T15:59:01.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.275+0000 7f82894f6700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8284060240 con 0x7f8284097b40 2022-06-29T15:59:01.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.275+0000 7f82894f6700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82840b0510 con 0x7f828405fe60 2022-06-29T15:59:01.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.275+0000 7f8283fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f828405fe60 0x7f828413ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:01.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.275+0000 7f82837fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f82840affe0 0x7f828413e2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54344/0 (socket says 172.21.15.60:54344) 2022-06-29T15:59:01.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.275+0000 7f8283fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f828405fe60 0x7f828413ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46256/0 (socket says 172.21.15.60:46256) 2022-06-29T15:59:01.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.275+0000 7f82837fe700 1 -- 172.21.15.60:0/1663775322 learned_addr learned my addr 172.21.15.60:0/1663775322 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:59:01.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.275+0000 7f8283fff700 1 -- 172.21.15.60:0/1663775322 >> v1:172.21.15.92:6789/0 conn(0x7f8284097b40 legacy=0x7f82840ab1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:01.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.275+0000 7f8282ffd700 1 -- 172.21.15.60:0/1663775322 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3240259796 0 0) 0x7f828405d2c0 con 0x7f82840affe0 2022-06-29T15:59:01.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.275+0000 7f8283fff700 1 -- 172.21.15.60:0/1663775322 >> v1:172.21.15.60:6789/0 conn(0x7f82840affe0 legacy=0x7f828413e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:01.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.275+0000 7f8283fff700 1 -- 172.21.15.60:0/1663775322 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82840b07b0 con 0x7f828405fe60 2022-06-29T15:59:01.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.279+0000 7f8283fff700 1 --2- 172.21.15.60:0/1663775322 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f828405fe60 0x7f828413ddb0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f827c0032f0 tx=0x7f827c00ac00).ready entity=mon.0 client_cookie=499fd745135ddb10 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:01.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.279+0000 7f8282ffd700 1 -- 172.21.15.60:0/1663775322 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f827c00f070 con 0x7f828405fe60 2022-06-29T15:59:01.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.279+0000 7f8282ffd700 1 -- 172.21.15.60:0/1663775322 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f827c008de0 con 0x7f828405fe60 2022-06-29T15:59:01.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.279+0000 7f8282ffd700 1 -- 172.21.15.60:0/1663775322 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f827c00a2c0 con 0x7f828405fe60 2022-06-29T15:59:01.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.279+0000 7f82894f6700 1 -- 172.21.15.60:0/1663775322 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f828405fe60 msgr2=0x7f828413ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:01.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.279+0000 7f82894f6700 1 --2- 172.21.15.60:0/1663775322 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f828405fe60 0x7f828413ddb0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f827c0032f0 tx=0x7f827c00ac00).stop 2022-06-29T15:59:01.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.279+0000 7f82894f6700 1 -- 172.21.15.60:0/1663775322 shutdown_connections 2022-06-29T15:59:01.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.279+0000 7f82894f6700 1 --2- 172.21.15.60:0/1663775322 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f828405fe60 0x7f828413ddb0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:01.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.279+0000 7f82894f6700 1 -- 172.21.15.60:0/1663775322 >> 172.21.15.60:0/1663775322 conn(0x7f82840b5180 msgr2=0x7f82840b5560 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:01.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.279+0000 7f82894f6700 1 -- 172.21.15.60:0/1663775322 shutdown_connections 2022-06-29T15:59:01.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.279+0000 7f82894f6700 1 -- 172.21.15.60:0/1663775322 wait complete. 2022-06-29T15:59:01.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.279+0000 7f82894f6700 1 Processor -- start 2022-06-29T15:59:01.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.279+0000 7f82894f6700 1 -- start start 2022-06-29T15:59:01.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.283+0000 7f82894f6700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f828405fe60 0x7f828409a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:01.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.283+0000 7f82894f6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8284097b40 0x7f82840ae5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:01.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.283+0000 7f82894f6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f82840affe0 0x7f82840aeab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:01.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.283+0000 7f82894f6700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8284137570 con 0x7f8284097b40 2022-06-29T15:59:01.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.283+0000 7f82894f6700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82840ad720 con 0x7f828405fe60 2022-06-29T15:59:01.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.283+0000 7f82894f6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82841377a0 con 0x7f82840affe0 2022-06-29T15:59:01.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.283+0000 7f82837fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8284097b40 0x7f82840ae5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:01.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.283+0000 7f82837fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8284097b40 0x7f82840ae5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46264/0 (socket says 172.21.15.60:46264) 2022-06-29T15:59:01.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.283+0000 7f82837fe700 1 -- 172.21.15.60:0/2041651143 learned_addr learned my addr 172.21.15.60:0/2041651143 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:01.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.283+0000 7f8288cf5700 1 --2- 172.21.15.60:0/2041651143 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f82840affe0 0x7f82840aeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:01.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.283+0000 7f8283fff700 1 --2- 172.21.15.60:0/2041651143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f828405fe60 0x7f828409a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:01.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.283+0000 7f82837fe700 1 -- 172.21.15.60:0/2041651143 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f82840affe0 msgr2=0x7f82840aeab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:01.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.283+0000 7f82837fe700 1 --2- 172.21.15.60:0/2041651143 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f82840affe0 0x7f82840aeab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:01.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.283+0000 7f82837fe700 1 -- 172.21.15.60:0/2041651143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f828405fe60 msgr2=0x7f828409a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:01.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.283+0000 7f82837fe700 1 --2- 172.21.15.60:0/2041651143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f828405fe60 0x7f828409a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:01.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.283+0000 7f82837fe700 1 -- 172.21.15.60:0/2041651143 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f827c007040 con 0x7f8284097b40 2022-06-29T15:59:01.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.283+0000 7f82837fe700 1 --2- 172.21.15.60:0/2041651143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8284097b40 0x7f82840ae5c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f8278000bc0 tx=0x7f8278007c90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:01.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.283+0000 7f82817fa700 1 -- 172.21.15.60:0/2041651143 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f827800ce00 con 0x7f8284097b40 2022-06-29T15:59:01.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.283+0000 7f82817fa700 1 -- 172.21.15.60:0/2041651143 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82780070d0 con 0x7f8284097b40 2022-06-29T15:59:01.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.283+0000 7f82894f6700 1 -- 172.21.15.60:0/2041651143 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82840ad980 con 0x7f8284097b40 2022-06-29T15:59:01.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.283+0000 7f82817fa700 1 -- 172.21.15.60:0/2041651143 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8278007430 con 0x7f8284097b40 2022-06-29T15:59:01.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.283+0000 7f82894f6700 1 -- 172.21.15.60:0/2041651143 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f828413c2d0 con 0x7f8284097b40 2022-06-29T15:59:01.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.283+0000 7f82894f6700 1 -- 172.21.15.60:0/2041651143 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f828413d130 con 0x7f8284097b40 2022-06-29T15:59:01.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.287+0000 7f82817fa700 1 -- 172.21.15.60:0/2041651143 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f827801a060 con 0x7f8284097b40 2022-06-29T15:59:01.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.287+0000 7f82817fa700 1 --2- 172.21.15.60:0/2041651143 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8264053a70 0x7f8264055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:01.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.287+0000 7f82817fa700 1 -- 172.21.15.60:0/2041651143 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f827806b120 con 0x7f8284097b40 2022-06-29T15:59:01.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.287+0000 7f8283fff700 1 --2- 172.21.15.60:0/2041651143 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8264053a70 0x7f8264055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:01.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.287+0000 7f8283fff700 1 --2- 172.21.15.60:0/2041651143 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8264053a70 0x7f8264055f30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f827c019000 tx=0x7f82840aa630).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:01.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.291+0000 7f82817fa700 1 -- 172.21.15.60:0/2041651143 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8278042370 con 0x7f8284097b40 2022-06-29T15:59:01.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.459+0000 7f82894f6700 1 -- 172.21.15.60:0/2041651143 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f828405cf40 con 0x7f8284097b40 2022-06-29T15:59:01.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.467+0000 7f82817fa700 1 -- 172.21.15.60:0/2041651143 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 updated v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8278018070 con 0x7f8284097b40 2022-06-29T15:59:01.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.471+0000 7f82894f6700 1 -- 172.21.15.60:0/2041651143 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8264053a70 msgr2=0x7f8264055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:01.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.471+0000 7f82894f6700 1 --2- 172.21.15.60:0/2041651143 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8264053a70 0x7f8264055f30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f827c019000 tx=0x7f82840aa630).stop 2022-06-29T15:59:01.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.471+0000 7f82894f6700 1 -- 172.21.15.60:0/2041651143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8284097b40 msgr2=0x7f82840ae5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:01.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.471+0000 7f82894f6700 1 --2- 172.21.15.60:0/2041651143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8284097b40 0x7f82840ae5c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f8278000bc0 tx=0x7f8278007c90).stop 2022-06-29T15:59:01.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.471+0000 7f82894f6700 1 -- 172.21.15.60:0/2041651143 shutdown_connections 2022-06-29T15:59:01.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.471+0000 7f82894f6700 1 --2- 172.21.15.60:0/2041651143 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8264053a70 0x7f8264055f30 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:01.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.471+0000 7f82894f6700 1 --2- 172.21.15.60:0/2041651143 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f82840affe0 0x7f82840aeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:01.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.471+0000 7f82894f6700 1 --2- 172.21.15.60:0/2041651143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8284097b40 0x7f82840ae5c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:01.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.471+0000 7f82894f6700 1 --2- 172.21.15.60:0/2041651143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f828405fe60 0x7f828409a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:01.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.471+0000 7f82894f6700 1 -- 172.21.15.60:0/2041651143 >> 172.21.15.60:0/2041651143 conn(0x7f82840b5180 msgr2=0x7f8284144a20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:01.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.471+0000 7f82894f6700 1 -- 172.21.15.60:0/2041651143 shutdown_connections 2022-06-29T15:59:01.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.471+0000 7f82894f6700 1 -- 172.21.15.60:0/2041651143 wait complete. 2022-06-29T15:59:01.484 INFO:teuthology.orchestra.run.smithi060.stderr:updated 2022-06-29T15:59:01.493 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2022-06-29T15:59:01.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.587+0000 7fdc153ed700 1 Processor -- start 2022-06-29T15:59:01.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.587+0000 7fdc153ed700 1 -- start start 2022-06-29T15:59:01.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.587+0000 7fdc153ed700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdc100b5950 0x7fdc10067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:01.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.587+0000 7fdc153ed700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc10063ad0 0x7fdc1014af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:01.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.587+0000 7fdc153ed700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc100acba0 con 0x7fdc1005bd20 2022-06-29T15:59:01.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.587+0000 7fdc153ed700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc10067220 con 0x7fdc100b5950 2022-06-29T15:59:01.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.587+0000 7fdc153ed700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc100b5d30 con 0x7fdc10063ad0 2022-06-29T15:59:01.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.587+0000 7fdc0f7fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fdc1005bd20 0x7fdc1005c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54360/0 (socket says 172.21.15.60:54360) 2022-06-29T15:59:01.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.587+0000 7fdc0f7fe700 1 -- 172.21.15.60:0/3219830786 learned_addr learned my addr 172.21.15.60:0/3219830786 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:59:01.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.587+0000 7fdc0e7fc700 1 --2- 172.21.15.60:0/3219830786 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc10063ad0 0x7fdc1014af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:01.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.587+0000 7fdc0effd700 1 --2- 172.21.15.60:0/3219830786 >> v2:172.21.15.92:3300/0 conn(0x7fdc100b5950 0x7fdc10067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:01.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.587+0000 7fdc0dffb700 1 -- 172.21.15.60:0/3219830786 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830416971 0 0) 0x7fdc100acba0 con 0x7fdc1005bd20 2022-06-29T15:59:01.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.587+0000 7fdc0dffb700 1 -- 172.21.15.60:0/3219830786 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbf4003260 con 0x7fdc1005bd20 2022-06-29T15:59:01.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.587+0000 7fdc0e7fc700 1 -- 172.21.15.60:0/3219830786 >> v2:172.21.15.92:3300/0 conn(0x7fdc100b5950 msgr2=0x7fdc10067680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:01.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.587+0000 7fdc0e7fc700 1 --2- 172.21.15.60:0/3219830786 >> v2:172.21.15.92:3300/0 conn(0x7fdc100b5950 0x7fdc10067680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:01.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.587+0000 7fdc0e7fc700 1 -- 172.21.15.60:0/3219830786 >> v1:172.21.15.60:6789/0 conn(0x7fdc1005bd20 legacy=0x7fdc1005c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:01.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.587+0000 7fdc0e7fc700 1 -- 172.21.15.60:0/3219830786 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc100ace00 con 0x7fdc10063ad0 2022-06-29T15:59:01.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.591+0000 7fdc0e7fc700 1 --2- 172.21.15.60:0/3219830786 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc10063ad0 0x7fdc1014af10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fdc00002f70 tx=0x7fdc00006df0).ready entity=mon.2 client_cookie=adf3f4109e6bbde4 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:01.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.591+0000 7fdc0dffb700 1 -- 172.21.15.60:0/3219830786 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc0000d4a0 con 0x7fdc10063ad0 2022-06-29T15:59:01.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.591+0000 7fdc0dffb700 1 -- 172.21.15.60:0/3219830786 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc00000e90 con 0x7fdc10063ad0 2022-06-29T15:59:01.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.591+0000 7fdc0dffb700 1 -- 172.21.15.60:0/3219830786 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc0000d7f0 con 0x7fdc10063ad0 2022-06-29T15:59:01.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.591+0000 7fdc153ed700 1 -- 172.21.15.60:0/3219830786 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc10063ad0 msgr2=0x7fdc1014af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:01.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.591+0000 7fdc153ed700 1 --2- 172.21.15.60:0/3219830786 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc10063ad0 0x7fdc1014af10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fdc00002f70 tx=0x7fdc00006df0).stop 2022-06-29T15:59:01.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.591+0000 7fdc153ed700 1 -- 172.21.15.60:0/3219830786 shutdown_connections 2022-06-29T15:59:01.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.591+0000 7fdc153ed700 1 --2- 172.21.15.60:0/3219830786 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc10063ad0 0x7fdc1014af10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:01.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.591+0000 7fdc153ed700 1 --2- 172.21.15.60:0/3219830786 >> v2:172.21.15.92:3300/0 conn(0x7fdc100b5950 0x7fdc10067680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:01.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.591+0000 7fdc153ed700 1 -- 172.21.15.60:0/3219830786 >> 172.21.15.60:0/3219830786 conn(0x7fdc100632c0 msgr2=0x7fdc100b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:01.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.591+0000 7fdc153ed700 1 -- 172.21.15.60:0/3219830786 shutdown_connections 2022-06-29T15:59:01.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.591+0000 7fdc153ed700 1 -- 172.21.15.60:0/3219830786 wait complete. 2022-06-29T15:59:01.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.591+0000 7fdc153ed700 1 Processor -- start 2022-06-29T15:59:01.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.591+0000 7fdc153ed700 1 -- start start 2022-06-29T15:59:01.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.595+0000 7fdc153ed700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc1005bd20 0x7fdc101594c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:01.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.595+0000 7fdc153ed700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc10063ad0 0x7fdc1014a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:01.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.595+0000 7fdc153ed700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc100b5950 0x7fdc1014c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:01.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.595+0000 7fdc153ed700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc100ae7c0 con 0x7fdc1005bd20 2022-06-29T15:59:01.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.595+0000 7fdc153ed700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc100aea60 con 0x7fdc100b5950 2022-06-29T15:59:01.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.595+0000 7fdc153ed700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc100acfe0 con 0x7fdc10063ad0 2022-06-29T15:59:01.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.595+0000 7fdc0effd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc1005bd20 0x7fdc101594c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:01.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.595+0000 7fdc0e7fc700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc10063ad0 0x7fdc1014a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:01.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.595+0000 7fdc0e7fc700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc10063ad0 0x7fdc1014a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:59372/0 (socket says 172.21.15.60:59372) 2022-06-29T15:59:01.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.595+0000 7fdc0e7fc700 1 -- 172.21.15.60:0/2581707249 learned_addr learned my addr 172.21.15.60:0/2581707249 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:01.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.595+0000 7fdc0f7fe700 1 --2- 172.21.15.60:0/2581707249 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc100b5950 0x7fdc1014c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:01.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.595+0000 7fdc0e7fc700 1 -- 172.21.15.60:0/2581707249 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc100b5950 msgr2=0x7fdc1014c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:01.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.595+0000 7fdc0e7fc700 1 --2- 172.21.15.60:0/2581707249 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc100b5950 0x7fdc1014c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:01.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.595+0000 7fdc0e7fc700 1 -- 172.21.15.60:0/2581707249 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc1005bd20 msgr2=0x7fdc101594c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:01.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.595+0000 7fdc0e7fc700 1 --2- 172.21.15.60:0/2581707249 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc1005bd20 0x7fdc101594c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:01.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.595+0000 7fdc0e7fc700 1 -- 172.21.15.60:0/2581707249 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc0000b180 con 0x7fdc10063ad0 2022-06-29T15:59:01.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.595+0000 7fdc0effd700 1 --2- 172.21.15.60:0/2581707249 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc1005bd20 0x7fdc101594c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:59:01.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.595+0000 7fdc0e7fc700 1 --2- 172.21.15.60:0/2581707249 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc10063ad0 0x7fdc1014a2e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fdc00002f70 tx=0x7fdc00014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:01.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.595+0000 7fdbfffff700 1 -- 172.21.15.60:0/2581707249 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc00016070 con 0x7fdc10063ad0 2022-06-29T15:59:01.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.595+0000 7fdbfffff700 1 -- 172.21.15.60:0/2581707249 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc000137d0 con 0x7fdc10063ad0 2022-06-29T15:59:01.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.595+0000 7fdc153ed700 1 -- 172.21.15.60:0/2581707249 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc100ad1a0 con 0x7fdc10063ad0 2022-06-29T15:59:01.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.595+0000 7fdbfffff700 1 -- 172.21.15.60:0/2581707249 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc00013b30 con 0x7fdc10063ad0 2022-06-29T15:59:01.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.595+0000 7fdc153ed700 1 -- 172.21.15.60:0/2581707249 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc100b0630 con 0x7fdc10063ad0 2022-06-29T15:59:01.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.599+0000 7fdc153ed700 1 -- 172.21.15.60:0/2581707249 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc100aec20 con 0x7fdc10063ad0 2022-06-29T15:59:01.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.599+0000 7fdbfffff700 1 -- 172.21.15.60:0/2581707249 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdc00025030 con 0x7fdc10063ad0 2022-06-29T15:59:01.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.599+0000 7fdbfffff700 1 --2- 172.21.15.60:0/2581707249 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdbec053a70 0x7fdbec055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:01.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.599+0000 7fdbfffff700 1 -- 172.21.15.60:0/2581707249 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fdc0006cb20 con 0x7fdc10063ad0 2022-06-29T15:59:01.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.599+0000 7fdc0effd700 1 --2- 172.21.15.60:0/2581707249 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdbec053a70 0x7fdbec055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:01.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.599+0000 7fdc0effd700 1 --2- 172.21.15.60:0/2581707249 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdbec053a70 0x7fdbec055f30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fdc0400b000 tx=0x7fdc04006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:01.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.611+0000 7fdbfffff700 1 -- 172.21.15.60:0/2581707249 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdc00043d70 con 0x7fdc10063ad0 2022-06-29T15:59:01.784 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.775+0000 7fdc153ed700 1 -- 172.21.15.60:0/2581707249 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7fdc100573d0 con 0x7fdc10063ad0 2022-06-29T15:59:01.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.783+0000 7fdbfffff700 1 -- 172.21.15.60:0/2581707249 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 updated v10) v1 ==== 97+0+0 (secure 0 0 0) 0x7fdc00023070 con 0x7fdc10063ad0 2022-06-29T15:59:01.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.787+0000 7fdc153ed700 1 -- 172.21.15.60:0/2581707249 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdbec053a70 msgr2=0x7fdbec055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:01.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.787+0000 7fdc153ed700 1 --2- 172.21.15.60:0/2581707249 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdbec053a70 0x7fdbec055f30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fdc0400b000 tx=0x7fdc04006df0).stop 2022-06-29T15:59:01.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.787+0000 7fdc153ed700 1 -- 172.21.15.60:0/2581707249 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc10063ad0 msgr2=0x7fdc1014a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:01.797 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.787+0000 7fdc153ed700 1 --2- 172.21.15.60:0/2581707249 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc10063ad0 0x7fdc1014a2e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fdc00002f70 tx=0x7fdc00014000).stop 2022-06-29T15:59:01.797 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.787+0000 7fdc153ed700 1 -- 172.21.15.60:0/2581707249 shutdown_connections 2022-06-29T15:59:01.797 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.787+0000 7fdc153ed700 1 --2- 172.21.15.60:0/2581707249 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdbec053a70 0x7fdbec055f30 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:01.797 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.787+0000 7fdc153ed700 1 --2- 172.21.15.60:0/2581707249 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc100b5950 0x7fdc1014c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:01.798 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.787+0000 7fdc153ed700 1 --2- 172.21.15.60:0/2581707249 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc10063ad0 0x7fdc1014a2e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:01.798 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.787+0000 7fdc153ed700 1 --2- 172.21.15.60:0/2581707249 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc1005bd20 0x7fdc101594c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:01.798 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.787+0000 7fdc153ed700 1 -- 172.21.15.60:0/2581707249 >> 172.21.15.60:0/2581707249 conn(0x7fdc100632c0 msgr2=0x7fdc10145f50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:01.799 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.787+0000 7fdc153ed700 1 -- 172.21.15.60:0/2581707249 shutdown_connections 2022-06-29T15:59:01.799 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.787+0000 7fdc153ed700 1 -- 172.21.15.60:0/2581707249 wait complete. 2022-06-29T15:59:01.800 INFO:teuthology.orchestra.run.smithi060.stderr:updated 2022-06-29T15:59:01.810 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2022-06-29T15:59:01.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.907+0000 7f3478630700 1 Processor -- start 2022-06-29T15:59:01.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.907+0000 7f3478630700 1 -- start start 2022-06-29T15:59:01.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.907+0000 7f3478630700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f34700678f0 0x7f34700b2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:01.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.907+0000 7f3478630700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3470141530 0x7f347013f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:01.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.907+0000 7f3478630700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3470067d60 con 0x7f34700b2020 2022-06-29T15:59:01.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.907+0000 7f3478630700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34700b28f0 con 0x7f34700678f0 2022-06-29T15:59:01.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.907+0000 7f3478630700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3470144a00 con 0x7f3470141530 2022-06-29T15:59:01.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.907+0000 7f34763cc700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f34700b2020 0x7f3470140e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54370/0 (socket says 172.21.15.60:54370) 2022-06-29T15:59:01.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.907+0000 7f34763cc700 1 -- 172.21.15.60:0/3551582019 learned_addr learned my addr 172.21.15.60:0/3551582019 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:59:01.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.907+0000 7f3475bcb700 1 --2- 172.21.15.60:0/3551582019 >> v2:172.21.15.60:3300/0 conn(0x7f34700678f0 0x7f34700b2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:01.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.907+0000 7f3476bcd700 1 --2- 172.21.15.60:0/3551582019 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3470141530 0x7f347013f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:01.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.907+0000 7f3475bcb700 1 -- 172.21.15.60:0/3551582019 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3470141530 msgr2=0x7f347013f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:01.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.907+0000 7f3475bcb700 1 --2- 172.21.15.60:0/3551582019 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3470141530 0x7f347013f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:01.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.907+0000 7f3475bcb700 1 -- 172.21.15.60:0/3551582019 >> v1:172.21.15.60:6789/0 conn(0x7f34700b2020 legacy=0x7f3470140e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:01.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.907+0000 7f3475bcb700 1 -- 172.21.15.60:0/3551582019 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3470144ca0 con 0x7f34700678f0 2022-06-29T15:59:01.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.911+0000 7f3476bcd700 1 --2- 172.21.15.60:0/3551582019 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3470141530 0x7f347013f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:59:01.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.911+0000 7f3475bcb700 1 --2- 172.21.15.60:0/3551582019 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34700678f0 0x7f34700b2400 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f3464002f70 tx=0x7f3464006df0).ready entity=mon.0 client_cookie=f55e1243908d9fd5 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:01.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.911+0000 7f34753ca700 1 -- 172.21.15.60:0/3551582019 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f346400d570 con 0x7f34700678f0 2022-06-29T15:59:01.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.911+0000 7f34753ca700 1 -- 172.21.15.60:0/3551582019 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3464000e90 con 0x7f34700678f0 2022-06-29T15:59:01.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.911+0000 7f34753ca700 1 -- 172.21.15.60:0/3551582019 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f346400d8c0 con 0x7f34700678f0 2022-06-29T15:59:01.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.911+0000 7f3478630700 1 -- 172.21.15.60:0/3551582019 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34700678f0 msgr2=0x7f34700b2400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:01.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.911+0000 7f3478630700 1 --2- 172.21.15.60:0/3551582019 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34700678f0 0x7f34700b2400 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f3464002f70 tx=0x7f3464006df0).stop 2022-06-29T15:59:01.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.911+0000 7f3478630700 1 -- 172.21.15.60:0/3551582019 shutdown_connections 2022-06-29T15:59:01.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.911+0000 7f3478630700 1 --2- 172.21.15.60:0/3551582019 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3470141530 0x7f347013f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:01.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.911+0000 7f3478630700 1 --2- 172.21.15.60:0/3551582019 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34700678f0 0x7f34700b2400 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:01.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.911+0000 7f3478630700 1 -- 172.21.15.60:0/3551582019 >> 172.21.15.60:0/3551582019 conn(0x7f347005f0b0 msgr2=0x7f347005f490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:01.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.911+0000 7f3478630700 1 -- 172.21.15.60:0/3551582019 shutdown_connections 2022-06-29T15:59:01.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.911+0000 7f3478630700 1 -- 172.21.15.60:0/3551582019 wait complete. 2022-06-29T15:59:01.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.915+0000 7f3478630700 1 Processor -- start 2022-06-29T15:59:01.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.915+0000 7f3478630700 1 -- start start 2022-06-29T15:59:01.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.915+0000 7f3478630700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34700678f0 0x7f3470065700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:01.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.915+0000 7f3478630700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f34700b2020 0x7f3470065bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:01.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.915+0000 7f3478630700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3470141530 0x7f347013c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:01.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.915+0000 7f3478630700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3470066680 con 0x7f34700678f0 2022-06-29T15:59:01.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.915+0000 7f3478630700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347013f080 con 0x7f3470141530 2022-06-29T15:59:01.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.915+0000 7f3478630700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3470064b10 con 0x7f34700b2020 2022-06-29T15:59:01.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.915+0000 7f34763cc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34700678f0 0x7f3470065700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:01.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.915+0000 7f34763cc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34700678f0 0x7f3470065700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46290/0 (socket says 172.21.15.60:46290) 2022-06-29T15:59:01.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.915+0000 7f34763cc700 1 -- 172.21.15.60:0/3653552063 learned_addr learned my addr 172.21.15.60:0/3653552063 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:01.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.915+0000 7f3475bcb700 1 --2- 172.21.15.60:0/3653552063 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f34700b2020 0x7f3470065bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:01.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.915+0000 7f3476bcd700 1 --2- 172.21.15.60:0/3653552063 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3470141530 0x7f347013c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:01.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.915+0000 7f34763cc700 1 -- 172.21.15.60:0/3653552063 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f34700b2020 msgr2=0x7f3470065bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:01.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.915+0000 7f34763cc700 1 --2- 172.21.15.60:0/3653552063 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f34700b2020 0x7f3470065bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:01.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.915+0000 7f34763cc700 1 -- 172.21.15.60:0/3653552063 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3470141530 msgr2=0x7f347013c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:01.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.915+0000 7f34763cc700 1 --2- 172.21.15.60:0/3653552063 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3470141530 0x7f347013c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:01.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.915+0000 7f34763cc700 1 -- 172.21.15.60:0/3653552063 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f346400b180 con 0x7f34700678f0 2022-06-29T15:59:01.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.915+0000 7f34763cc700 1 --2- 172.21.15.60:0/3653552063 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34700678f0 0x7f3470065700 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f34700601d0 tx=0x7f346000c890).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:01.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.915+0000 7f346f7fe700 1 -- 172.21.15.60:0/3653552063 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3460005ba0 con 0x7f34700678f0 2022-06-29T15:59:01.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.915+0000 7f346f7fe700 1 -- 172.21.15.60:0/3653552063 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3460005d00 con 0x7f34700678f0 2022-06-29T15:59:01.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.915+0000 7f346f7fe700 1 -- 172.21.15.60:0/3653552063 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34600072c0 con 0x7f34700678f0 2022-06-29T15:59:01.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.915+0000 7f3478630700 1 -- 172.21.15.60:0/3653552063 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34701443e0 con 0x7f34700678f0 2022-06-29T15:59:01.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.919+0000 7f3478630700 1 -- 172.21.15.60:0/3653552063 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34700651a0 con 0x7f34700678f0 2022-06-29T15:59:01.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.919+0000 7f346f7fe700 1 -- 172.21.15.60:0/3653552063 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3460010070 con 0x7f34700678f0 2022-06-29T15:59:01.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.919+0000 7f346f7fe700 1 --2- 172.21.15.60:0/3653552063 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3450053980 0x7f3450055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:01.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.919+0000 7f346f7fe700 1 -- 172.21.15.60:0/3653552063 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f346006b3c0 con 0x7f34700678f0 2022-06-29T15:59:01.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.919+0000 7f3475bcb700 1 --2- 172.21.15.60:0/3653552063 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3450053980 0x7f3450055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:01.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.919+0000 7f3478630700 1 -- 172.21.15.60:0/3653552063 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34700651a0 con 0x7f34700678f0 2022-06-29T15:59:01.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.919+0000 7f3475bcb700 1 --2- 172.21.15.60:0/3653552063 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3450053980 0x7f3450055e40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f346401c000 tx=0x7f3464015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:01.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:01.927+0000 7f346f7fe700 1 -- 172.21.15.60:0/3653552063 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f34700651a0 con 0x7f34700678f0 2022-06-29T15:59:02.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.095+0000 7f3478630700 1 -- 172.21.15.60:0/3653552063 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f34700651a0 con 0x7f34700678f0 2022-06-29T15:59:02.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.103+0000 7f346f7fe700 1 -- 172.21.15.60:0/3653552063 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 updated v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f34700651a0 con 0x7f34700678f0 2022-06-29T15:59:02.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.107+0000 7f3478630700 1 -- 172.21.15.60:0/3653552063 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3450053980 msgr2=0x7f3450055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:02.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.107+0000 7f3478630700 1 --2- 172.21.15.60:0/3653552063 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3450053980 0x7f3450055e40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f346401c000 tx=0x7f3464015000).stop 2022-06-29T15:59:02.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.107+0000 7f3478630700 1 -- 172.21.15.60:0/3653552063 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34700678f0 msgr2=0x7f3470065700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:02.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.107+0000 7f3478630700 1 --2- 172.21.15.60:0/3653552063 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34700678f0 0x7f3470065700 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f34700601d0 tx=0x7f346000c890).stop 2022-06-29T15:59:02.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.107+0000 7f3478630700 1 -- 172.21.15.60:0/3653552063 shutdown_connections 2022-06-29T15:59:02.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.107+0000 7f3478630700 1 --2- 172.21.15.60:0/3653552063 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3450053980 0x7f3450055e40 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:02.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.107+0000 7f3478630700 1 --2- 172.21.15.60:0/3653552063 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3470141530 0x7f347013c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:02.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.107+0000 7f3478630700 1 --2- 172.21.15.60:0/3653552063 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f34700b2020 0x7f3470065bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:02.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.107+0000 7f3478630700 1 --2- 172.21.15.60:0/3653552063 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34700678f0 0x7f3470065700 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:02.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.107+0000 7f3478630700 1 -- 172.21.15.60:0/3653552063 >> 172.21.15.60:0/3653552063 conn(0x7f347005f0b0 msgr2=0x7f347013b040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:02.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.107+0000 7f3478630700 1 -- 172.21.15.60:0/3653552063 shutdown_connections 2022-06-29T15:59:02.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.107+0000 7f3478630700 1 -- 172.21.15.60:0/3653552063 wait complete. 2022-06-29T15:59:02.119 INFO:teuthology.orchestra.run.smithi060.stderr:updated 2022-06-29T15:59:02.129 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2022-06-29T15:59:02.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.227+0000 7f00dd491700 1 Processor -- start 2022-06-29T15:59:02.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.227+0000 7f00dd491700 1 -- start start 2022-06-29T15:59:02.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.231+0000 7f00dd491700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f00d805d1f0 0x7f00d80b1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:02.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.231+0000 7f00dd491700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00d80b3cc0 0x7f00d80b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:02.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.231+0000 7f00dd491700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00d80b2ce0 con 0x7f00d80b9d70 2022-06-29T15:59:02.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.231+0000 7f00dd491700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00d80b4d10 con 0x7f00d805d1f0 2022-06-29T15:59:02.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.231+0000 7f00dd491700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00d80ba150 con 0x7f00d80b3cc0 2022-06-29T15:59:02.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.231+0000 7f00d67fc700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f00d805d1f0 0x7f00d80b1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:02.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.231+0000 7f00d6ffd700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f00d80b9d70 0x7f00d80b3140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:58632/0 (socket says 172.21.15.60:58632) 2022-06-29T15:59:02.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.231+0000 7f00d6ffd700 1 -- 172.21.15.60:0/3628363449 learned_addr learned my addr 172.21.15.60:0/3628363449 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:59:02.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.231+0000 7f00d77fe700 1 --2- 172.21.15.60:0/3628363449 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00d80b3cc0 0x7f00d80b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:02.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.231+0000 7f00d67fc700 1 -- 172.21.15.60:0/3628363449 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00d80b3cc0 msgr2=0x7f00d80b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:02.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.231+0000 7f00d67fc700 1 --2- 172.21.15.60:0/3628363449 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00d80b3cc0 0x7f00d80b2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:02.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.231+0000 7f00d67fc700 1 -- 172.21.15.60:0/3628363449 >> v1:172.21.15.92:6789/0 conn(0x7f00d80b9d70 legacy=0x7f00d80b3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:02.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.231+0000 7f00d67fc700 1 -- 172.21.15.60:0/3628363449 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d80b4a70 con 0x7f00d805d1f0 2022-06-29T15:59:02.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.231+0000 7f00d67fc700 1 --2- 172.21.15.60:0/3628363449 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00d805d1f0 0x7f00d80b1b50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f00cc002f70 tx=0x7f00cc006df0).ready entity=mon.1 client_cookie=32d3fd142bc91b37 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:02.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.231+0000 7f00d5ffb700 1 -- 172.21.15.60:0/3628363449 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f00cc00d570 con 0x7f00d805d1f0 2022-06-29T15:59:02.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.231+0000 7f00d5ffb700 1 -- 172.21.15.60:0/3628363449 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00cc000e90 con 0x7f00d805d1f0 2022-06-29T15:59:02.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.231+0000 7f00d5ffb700 1 -- 172.21.15.60:0/3628363449 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f00cc00d8c0 con 0x7f00d805d1f0 2022-06-29T15:59:02.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.231+0000 7f00dd491700 1 -- 172.21.15.60:0/3628363449 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00d805d1f0 msgr2=0x7f00d80b1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:02.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.231+0000 7f00dd491700 1 --2- 172.21.15.60:0/3628363449 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00d805d1f0 0x7f00d80b1b50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f00cc002f70 tx=0x7f00cc006df0).stop 2022-06-29T15:59:02.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.231+0000 7f00dd491700 1 -- 172.21.15.60:0/3628363449 shutdown_connections 2022-06-29T15:59:02.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.231+0000 7f00dd491700 1 --2- 172.21.15.60:0/3628363449 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00d80b3cc0 0x7f00d80b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:02.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.231+0000 7f00dd491700 1 --2- 172.21.15.60:0/3628363449 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00d805d1f0 0x7f00d80b1b50 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:02.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.231+0000 7f00dd491700 1 -- 172.21.15.60:0/3628363449 >> 172.21.15.60:0/3628363449 conn(0x7f00d8062ce0 msgr2=0x7f00d80b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:02.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.231+0000 7f00dd491700 1 -- 172.21.15.60:0/3628363449 shutdown_connections 2022-06-29T15:59:02.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.231+0000 7f00dd491700 1 -- 172.21.15.60:0/3628363449 wait complete. 2022-06-29T15:59:02.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.235+0000 7f00dd491700 1 Processor -- start 2022-06-29T15:59:02.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.235+0000 7f00dd491700 1 -- start start 2022-06-29T15:59:02.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.235+0000 7f00dd491700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00d805d1f0 0x7f00d81467f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:02.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.235+0000 7f00dd491700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00d80b3cc0 0x7f00d813be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:02.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.235+0000 7f00dd491700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00d80b9d70 0x7f00d8151dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:02.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.235+0000 7f00dd491700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d80ae320 con 0x7f00d805d1f0 2022-06-29T15:59:02.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.235+0000 7f00dd491700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d80ad130 con 0x7f00d80b9d70 2022-06-29T15:59:02.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.235+0000 7f00dd491700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00d80ad3d0 con 0x7f00d80b3cc0 2022-06-29T15:59:02.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.235+0000 7f00d6ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00d805d1f0 0x7f00d81467f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:02.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.235+0000 7f00d77fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00d80b9d70 0x7f00d8151dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:02.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.235+0000 7f00d67fc700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00d80b3cc0 0x7f00d813be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:02.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.235+0000 7f00d77fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00d80b9d70 0x7f00d8151dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:52336/0 (socket says 172.21.15.60:52336) 2022-06-29T15:59:02.254 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.235+0000 7f00d77fe700 1 -- 172.21.15.60:0/1311400556 learned_addr learned my addr 172.21.15.60:0/1311400556 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:02.254 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.235+0000 7f00d6ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00d805d1f0 0x7f00d81467f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46304/0 (socket says 172.21.15.60:46304) 2022-06-29T15:59:02.254 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.235+0000 7f00d77fe700 1 -- 172.21.15.60:0/1311400556 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00d80b3cc0 msgr2=0x7f00d813be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:02.254 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.235+0000 7f00d77fe700 1 --2- 172.21.15.60:0/1311400556 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00d80b3cc0 0x7f00d813be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:02.255 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.235+0000 7f00d77fe700 1 -- 172.21.15.60:0/1311400556 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00d805d1f0 msgr2=0x7f00d81467f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:02.255 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.235+0000 7f00d77fe700 1 --2- 172.21.15.60:0/1311400556 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00d805d1f0 0x7f00d81467f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:02.255 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.235+0000 7f00d77fe700 1 -- 172.21.15.60:0/1311400556 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00cc00b180 con 0x7f00d80b9d70 2022-06-29T15:59:02.255 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.239+0000 7f00d6ffd700 1 --2- 172.21.15.60:0/1311400556 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00d805d1f0 0x7f00d81467f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:59:02.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.239+0000 7f00d77fe700 1 --2- 172.21.15.60:0/1311400556 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00d80b9d70 0x7f00d8151dc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f00d80ae460 tx=0x7f00d00091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:02.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.239+0000 7f00c7fff700 1 -- 172.21.15.60:0/1311400556 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f00d0012600 con 0x7f00d80b9d70 2022-06-29T15:59:02.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.239+0000 7f00c7fff700 1 -- 172.21.15.60:0/1311400556 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00d0012760 con 0x7f00d80b9d70 2022-06-29T15:59:02.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.239+0000 7f00dd491700 1 -- 172.21.15.60:0/1311400556 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00d80af770 con 0x7f00d80b9d70 2022-06-29T15:59:02.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.239+0000 7f00c7fff700 1 -- 172.21.15.60:0/1311400556 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f00d0012ac0 con 0x7f00d80b9d70 2022-06-29T15:59:02.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.239+0000 7f00dd491700 1 -- 172.21.15.60:0/1311400556 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00d80afc10 con 0x7f00d80b9d70 2022-06-29T15:59:02.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.239+0000 7f00dd491700 1 -- 172.21.15.60:0/1311400556 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00d80573d0 con 0x7f00d80b9d70 2022-06-29T15:59:02.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.239+0000 7f00c7fff700 1 -- 172.21.15.60:0/1311400556 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f00d0022070 con 0x7f00d80b9d70 2022-06-29T15:59:02.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.239+0000 7f00c7fff700 1 --2- 172.21.15.60:0/1311400556 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f00b4053a70 0x7f00b4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:02.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.239+0000 7f00c7fff700 1 -- 172.21.15.60:0/1311400556 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f00d006c2e0 con 0x7f00d80b9d70 2022-06-29T15:59:02.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.239+0000 7f00d6ffd700 1 --2- 172.21.15.60:0/1311400556 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f00b4053a70 0x7f00b4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:02.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.243+0000 7f00d6ffd700 1 --2- 172.21.15.60:0/1311400556 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f00b4053a70 0x7f00b4055f30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f00c800d000 tx=0x7f00c8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:02.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.247+0000 7f00c7fff700 1 -- 172.21.15.60:0/1311400556 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f00d00433a0 con 0x7f00d80b9d70 2022-06-29T15:59:02.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.415+0000 7f00dd491700 1 -- 172.21.15.60:0/1311400556 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f00d805d990 con 0x7f00d80b9d70 2022-06-29T15:59:02.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.427+0000 7f00c7fff700 1 -- 172.21.15.60:0/1311400556 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 updated v12) v1 ==== 97+0+0 (secure 0 0 0) 0x7f00d00160a0 con 0x7f00d80b9d70 2022-06-29T15:59:02.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.427+0000 7f00dd491700 1 -- 172.21.15.60:0/1311400556 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f00b4053a70 msgr2=0x7f00b4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:02.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.427+0000 7f00dd491700 1 --2- 172.21.15.60:0/1311400556 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f00b4053a70 0x7f00b4055f30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f00c800d000 tx=0x7f00c8009040).stop 2022-06-29T15:59:02.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.427+0000 7f00dd491700 1 -- 172.21.15.60:0/1311400556 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00d80b9d70 msgr2=0x7f00d8151dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:02.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.427+0000 7f00dd491700 1 --2- 172.21.15.60:0/1311400556 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00d80b9d70 0x7f00d8151dc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f00d80ae460 tx=0x7f00d00091d0).stop 2022-06-29T15:59:02.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.427+0000 7f00dd491700 1 -- 172.21.15.60:0/1311400556 shutdown_connections 2022-06-29T15:59:02.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.427+0000 7f00dd491700 1 --2- 172.21.15.60:0/1311400556 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f00b4053a70 0x7f00b4055f30 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:02.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.427+0000 7f00dd491700 1 --2- 172.21.15.60:0/1311400556 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00d80b9d70 0x7f00d8151dc0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:02.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.427+0000 7f00dd491700 1 --2- 172.21.15.60:0/1311400556 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00d80b3cc0 0x7f00d813be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:02.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.427+0000 7f00dd491700 1 --2- 172.21.15.60:0/1311400556 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00d805d1f0 0x7f00d81467f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:02.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.427+0000 7f00dd491700 1 -- 172.21.15.60:0/1311400556 >> 172.21.15.60:0/1311400556 conn(0x7f00d8062ce0 msgr2=0x7f00d813ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:02.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.431+0000 7f00dd491700 1 -- 172.21.15.60:0/1311400556 shutdown_connections 2022-06-29T15:59:02.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.431+0000 7f00dd491700 1 -- 172.21.15.60:0/1311400556 wait complete. 2022-06-29T15:59:02.442 INFO:teuthology.orchestra.run.smithi060.stderr:updated 2022-06-29T15:59:02.452 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T15:59:02.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.547+0000 7f8d1f59e700 1 Processor -- start 2022-06-29T15:59:02.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.547+0000 7f8d1f59e700 1 -- start start 2022-06-29T15:59:02.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.547+0000 7f8d1f59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8d200b48b0 0x7f8d20142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:02.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.547+0000 7f8d1f59e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8d200b2c50 0x7f8d200b3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:02.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.547+0000 7f8d1f59e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d200b0cf0 con 0x7f8d200b5eb0 2022-06-29T15:59:02.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.547+0000 7f8d1f59e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d200b54e0 con 0x7f8d200b2c50 2022-06-29T15:59:02.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.547+0000 7f8d1f59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d200b4d10 con 0x7f8d200b48b0 2022-06-29T15:59:02.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.547+0000 7f8d1ed9d700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8d200b2c50 0x7f8d200b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:02.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.547+0000 7f8d1dd9b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8d200b48b0 0x7f8d20142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:02.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.547+0000 7f8d1e59c700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f8d200b5eb0 0x7f8d200b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:58646/0 (socket says 172.21.15.60:58646) 2022-06-29T15:59:02.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.547+0000 7f8d1ed9d700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8d200b2c50 0x7f8d200b3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:52346/0 (socket says 172.21.15.60:52346) 2022-06-29T15:59:02.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.547+0000 7f8d1e59c700 1 -- 172.21.15.60:0/3941511598 learned_addr learned my addr 172.21.15.60:0/3941511598 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:59:02.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.547+0000 7f8d1dd9b700 1 -- 172.21.15.60:0/3941511598 >> v2:172.21.15.92:3300/0 conn(0x7f8d200b2c50 msgr2=0x7f8d200b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:02.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.547+0000 7f8d1dd9b700 1 --2- 172.21.15.60:0/3941511598 >> v2:172.21.15.92:3300/0 conn(0x7f8d200b2c50 0x7f8d200b3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:02.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.547+0000 7f8d1dd9b700 1 -- 172.21.15.60:0/3941511598 >> v1:172.21.15.92:6789/0 conn(0x7f8d200b5eb0 legacy=0x7f8d200b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:02.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.547+0000 7f8d1dd9b700 1 -- 172.21.15.60:0/3941511598 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d200b6290 con 0x7f8d200b48b0 2022-06-29T15:59:02.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.547+0000 7f8d1dd9b700 1 --2- 172.21.15.60:0/3941511598 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8d200b48b0 0x7f8d20142a20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f8d14002f70 tx=0x7f8d14006df0).ready entity=mon.2 client_cookie=681bcdd2525da7bc server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:02.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.547+0000 7f8d1d59a700 1 -- 172.21.15.60:0/3941511598 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d1400d570 con 0x7f8d200b48b0 2022-06-29T15:59:02.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.547+0000 7f8d1d59a700 1 -- 172.21.15.60:0/3941511598 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d14000e90 con 0x7f8d200b48b0 2022-06-29T15:59:02.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.547+0000 7f8d1d59a700 1 -- 172.21.15.60:0/3941511598 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d1400d8c0 con 0x7f8d200b48b0 2022-06-29T15:59:02.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.551+0000 7f8d1f59e700 1 -- 172.21.15.60:0/3941511598 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8d200b48b0 msgr2=0x7f8d20142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:02.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.551+0000 7f8d1f59e700 1 --2- 172.21.15.60:0/3941511598 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8d200b48b0 0x7f8d20142a20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f8d14002f70 tx=0x7f8d14006df0).stop 2022-06-29T15:59:02.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.551+0000 7f8d1f59e700 1 -- 172.21.15.60:0/3941511598 shutdown_connections 2022-06-29T15:59:02.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.551+0000 7f8d1f59e700 1 --2- 172.21.15.60:0/3941511598 >> v2:172.21.15.92:3300/0 conn(0x7f8d200b2c50 0x7f8d200b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:02.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.551+0000 7f8d1f59e700 1 --2- 172.21.15.60:0/3941511598 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8d200b48b0 0x7f8d20142a20 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:02.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.551+0000 7f8d1f59e700 1 -- 172.21.15.60:0/3941511598 >> 172.21.15.60:0/3941511598 conn(0x7f8d20062090 msgr2=0x7f8d20062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:02.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.551+0000 7f8d1f59e700 1 -- 172.21.15.60:0/3941511598 shutdown_connections 2022-06-29T15:59:02.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.551+0000 7f8d1f59e700 1 -- 172.21.15.60:0/3941511598 wait complete. 2022-06-29T15:59:02.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.551+0000 7f8d1f59e700 1 Processor -- start 2022-06-29T15:59:02.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.551+0000 7f8d1f59e700 1 -- start start 2022-06-29T15:59:02.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.555+0000 7f8d1f59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d200b2c50 0x7f8d20152150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:02.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.555+0000 7f8d1f59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8d200b48b0 0x7f8d20152770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:02.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.555+0000 7f8d1f59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d200b5eb0 0x7f8d200adba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:02.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.555+0000 7f8d1f59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d20144920 con 0x7f8d200b5eb0 2022-06-29T15:59:02.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.555+0000 7f8d1f59e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d20152c60 con 0x7f8d200b2c50 2022-06-29T15:59:02.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.555+0000 7f8d1f59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d20145220 con 0x7f8d200b48b0 2022-06-29T15:59:02.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.555+0000 7f8d1ed9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d200b5eb0 0x7f8d200adba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:02.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.555+0000 7f8d1ed9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d200b5eb0 0x7f8d200adba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46322/0 (socket says 172.21.15.60:46322) 2022-06-29T15:59:02.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.555+0000 7f8d1ed9d700 1 -- 172.21.15.60:0/1402334801 learned_addr learned my addr 172.21.15.60:0/1402334801 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:02.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.555+0000 7f8d1dd9b700 1 --2- 172.21.15.60:0/1402334801 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8d200b48b0 0x7f8d20152770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:02.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.555+0000 7f8d1e59c700 1 --2- 172.21.15.60:0/1402334801 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d200b2c50 0x7f8d20152150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:02.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.555+0000 7f8d1ed9d700 1 -- 172.21.15.60:0/1402334801 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8d200b48b0 msgr2=0x7f8d20152770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:02.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.555+0000 7f8d1ed9d700 1 --2- 172.21.15.60:0/1402334801 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8d200b48b0 0x7f8d20152770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:02.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.555+0000 7f8d1ed9d700 1 -- 172.21.15.60:0/1402334801 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d200b2c50 msgr2=0x7f8d20152150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:02.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.555+0000 7f8d1ed9d700 1 --2- 172.21.15.60:0/1402334801 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d200b2c50 0x7f8d20152150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:02.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.555+0000 7f8d1ed9d700 1 -- 172.21.15.60:0/1402334801 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d1400b180 con 0x7f8d200b5eb0 2022-06-29T15:59:02.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.555+0000 7f8d1e59c700 1 --2- 172.21.15.60:0/1402334801 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d200b2c50 0x7f8d20152150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:59:02.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.555+0000 7f8d1ed9d700 1 --2- 172.21.15.60:0/1402334801 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d200b5eb0 0x7f8d200adba0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f8d20152df0 tx=0x7f8d180091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:02.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.555+0000 7f8d0f7fe700 1 -- 172.21.15.60:0/1402334801 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d1800eee0 con 0x7f8d200b5eb0 2022-06-29T15:59:02.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.555+0000 7f8d0f7fe700 1 -- 172.21.15.60:0/1402334801 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d1800c850 con 0x7f8d200b5eb0 2022-06-29T15:59:02.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.555+0000 7f8d0f7fe700 1 -- 172.21.15.60:0/1402334801 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d1800cbb0 con 0x7f8d200b5eb0 2022-06-29T15:59:02.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.555+0000 7f8d1f59e700 1 -- 172.21.15.60:0/1402334801 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d20146020 con 0x7f8d200b5eb0 2022-06-29T15:59:02.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.555+0000 7f8d1f59e700 1 -- 172.21.15.60:0/1402334801 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d20144b20 con 0x7f8d200b5eb0 2022-06-29T15:59:02.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.555+0000 7f8d1f59e700 1 -- 172.21.15.60:0/1402334801 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d20145080 con 0x7f8d200b5eb0 2022-06-29T15:59:02.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.555+0000 7f8d0f7fe700 1 -- 172.21.15.60:0/1402334801 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8d1800cd10 con 0x7f8d200b5eb0 2022-06-29T15:59:02.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.559+0000 7f8d0f7fe700 1 --2- 172.21.15.60:0/1402334801 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8cfc053b30 0x7f8cfc055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:02.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.559+0000 7f8d0f7fe700 1 -- 172.21.15.60:0/1402334801 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f8d1806bf30 con 0x7f8d200b5eb0 2022-06-29T15:59:02.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.559+0000 7f8d1e59c700 1 --2- 172.21.15.60:0/1402334801 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8cfc053b30 0x7f8cfc055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:02.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.559+0000 7f8d1e59c700 1 --2- 172.21.15.60:0/1402334801 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8cfc053b30 0x7f8cfc055ff0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f8d1000b000 tx=0x7f8d10009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:02.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.563+0000 7f8d0f7fe700 1 -- 172.21.15.60:0/1402334801 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8d20145080 con 0x7f8d200b5eb0 2022-06-29T15:59:02.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.687+0000 7f8d1f59e700 1 -- 172.21.15.60:0/1402334801 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d20145080 con 0x7f8d200b5eb0 2022-06-29T15:59:02.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.691+0000 7f8d0f7fe700 1 -- 172.21.15.60:0/1402334801 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+13181 (secure 0 0 0) 0x7f8d20145080 con 0x7f8d200b5eb0 2022-06-29T15:59:02.699 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:59:02.700 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":28,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T15:59:00.095360+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T15:59:02.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.691+0000 7f8d1f59e700 1 -- 172.21.15.60:0/1402334801 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8cfc053b30 msgr2=0x7f8cfc055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:02.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.691+0000 7f8d1f59e700 1 --2- 172.21.15.60:0/1402334801 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8cfc053b30 0x7f8cfc055ff0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f8d1000b000 tx=0x7f8d10009040).stop 2022-06-29T15:59:02.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.691+0000 7f8d1f59e700 1 -- 172.21.15.60:0/1402334801 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d200b5eb0 msgr2=0x7f8d200adba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:02.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.691+0000 7f8d1f59e700 1 --2- 172.21.15.60:0/1402334801 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d200b5eb0 0x7f8d200adba0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f8d20152df0 tx=0x7f8d180091d0).stop 2022-06-29T15:59:02.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.695+0000 7f8d1f59e700 1 -- 172.21.15.60:0/1402334801 shutdown_connections 2022-06-29T15:59:02.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.695+0000 7f8d1f59e700 1 --2- 172.21.15.60:0/1402334801 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8cfc053b30 0x7f8cfc055ff0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:02.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.695+0000 7f8d1f59e700 1 --2- 172.21.15.60:0/1402334801 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d200b5eb0 0x7f8d200adba0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:02.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.695+0000 7f8d1f59e700 1 --2- 172.21.15.60:0/1402334801 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8d200b48b0 0x7f8d20152770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:02.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.695+0000 7f8d1f59e700 1 --2- 172.21.15.60:0/1402334801 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d200b2c50 0x7f8d20152150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:02.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.695+0000 7f8d1f59e700 1 -- 172.21.15.60:0/1402334801 >> 172.21.15.60:0/1402334801 conn(0x7f8d20062090 msgr2=0x7f8d20143290 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:02.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.695+0000 7f8d1f59e700 1 -- 172.21.15.60:0/1402334801 shutdown_connections 2022-06-29T15:59:02.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.695+0000 7f8d1f59e700 1 -- 172.21.15.60:0/1402334801 wait complete. 2022-06-29T15:59:02.719 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-06-29T15:59:02.720 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T15:59:02.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.827+0000 7f75f8ae2700 1 Processor -- start 2022-06-29T15:59:02.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.827+0000 7f75f8ae2700 1 -- start start 2022-06-29T15:59:02.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.827+0000 7f75f8ae2700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f75f40b22c0 0x7f75f405bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:02.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.827+0000 7f75f8ae2700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f75f40b02d0 0x7f75f405c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:02.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.827+0000 7f75f8ae2700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f40b1540 con 0x7f75f405c700 2022-06-29T15:59:02.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.827+0000 7f75f8ae2700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75f405cb70 con 0x7f75f40b02d0 2022-06-29T15:59:02.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.827+0000 7f75f8ae2700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75f40b26a0 con 0x7f75f40b22c0 2022-06-29T15:59:02.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.827+0000 7f75f2ffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f75f40b02d0 0x7f75f405c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:02.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.827+0000 7f75f2ffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f75f40b02d0 0x7f75f405c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46328/0 (socket says 172.21.15.60:46328) 2022-06-29T15:59:02.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.827+0000 7f75f2ffd700 1 -- 172.21.15.60:0/2337453182 learned_addr learned my addr 172.21.15.60:0/2337453182 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:02.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.827+0000 7f75f37fe700 1 --2- 172.21.15.60:0/2337453182 >> v2:172.21.15.92:3300/0 conn(0x7f75f40b22c0 0x7f75f405bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:02.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.827+0000 7f75f2ffd700 1 -- 172.21.15.60:0/2337453182 >> v2:172.21.15.92:3300/0 conn(0x7f75f40b22c0 msgr2=0x7f75f405bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:02.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.827+0000 7f75f2ffd700 1 --2- 172.21.15.60:0/2337453182 >> v2:172.21.15.92:3300/0 conn(0x7f75f40b22c0 0x7f75f405bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:02.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.827+0000 7f75f2ffd700 1 -- 172.21.15.60:0/2337453182 >> v1:172.21.15.92:6789/0 conn(0x7f75f405c700 legacy=0x7f75f4140d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:02.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.827+0000 7f75f2ffd700 1 -- 172.21.15.60:0/2337453182 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f409a8b0 con 0x7f75f40b02d0 2022-06-29T15:59:02.843 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.827+0000 7f75f2ffd700 1 --2- 172.21.15.60:0/2337453182 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75f40b02d0 0x7f75f405c210 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f75e4002f70 tx=0x7f75e4006eb0).ready entity=mon.0 client_cookie=4a404385592227da server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:02.843 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.831+0000 7f75f27fc700 1 -- 172.21.15.60:0/2337453182 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75e400d5c0 con 0x7f75f40b02d0 2022-06-29T15:59:02.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.831+0000 7f75f27fc700 1 -- 172.21.15.60:0/2337453182 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75e4000e90 con 0x7f75f40b02d0 2022-06-29T15:59:02.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.831+0000 7f75f27fc700 1 -- 172.21.15.60:0/2337453182 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75e400d950 con 0x7f75f40b02d0 2022-06-29T15:59:02.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.831+0000 7f75f8ae2700 1 -- 172.21.15.60:0/2337453182 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75f40b02d0 msgr2=0x7f75f405c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:02.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.831+0000 7f75f8ae2700 1 --2- 172.21.15.60:0/2337453182 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75f40b02d0 0x7f75f405c210 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f75e4002f70 tx=0x7f75e4006eb0).stop 2022-06-29T15:59:02.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.831+0000 7f75f8ae2700 1 -- 172.21.15.60:0/2337453182 shutdown_connections 2022-06-29T15:59:02.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.831+0000 7f75f8ae2700 1 --2- 172.21.15.60:0/2337453182 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75f40b02d0 0x7f75f405c210 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:02.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.831+0000 7f75f8ae2700 1 --2- 172.21.15.60:0/2337453182 >> v2:172.21.15.92:3300/0 conn(0x7f75f40b22c0 0x7f75f405bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:02.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.831+0000 7f75f8ae2700 1 -- 172.21.15.60:0/2337453182 >> 172.21.15.60:0/2337453182 conn(0x7f75f40b58a0 msgr2=0x7f75f40b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:02.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.831+0000 7f75f8ae2700 1 -- 172.21.15.60:0/2337453182 shutdown_connections 2022-06-29T15:59:02.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.831+0000 7f75f8ae2700 1 -- 172.21.15.60:0/2337453182 wait complete. 2022-06-29T15:59:02.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.831+0000 7f75f8ae2700 1 Processor -- start 2022-06-29T15:59:02.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.831+0000 7f75f8ae2700 1 -- start start 2022-06-29T15:59:02.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.835+0000 7f75f8ae2700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75f405c700 0x7f75f413eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:02.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.835+0000 7f75f8ae2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75f40b02d0 0x7f75f4063cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:02.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.835+0000 7f75f8ae2700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75f40b22c0 0x7f75f40641a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:02.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.835+0000 7f75f8ae2700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f40abf10 con 0x7f75f40b02d0 2022-06-29T15:59:02.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.835+0000 7f75f8ae2700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f40ac170 con 0x7f75f405c700 2022-06-29T15:59:02.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.835+0000 7f75f8ae2700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75f40ac410 con 0x7f75f40b22c0 2022-06-29T15:59:02.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.835+0000 7f75f2ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75f40b02d0 0x7f75f4063cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:02.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.835+0000 7f75f2ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75f40b02d0 0x7f75f4063cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46334/0 (socket says 172.21.15.60:46334) 2022-06-29T15:59:02.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.835+0000 7f75f37fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75f405c700 0x7f75f413eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:02.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.835+0000 7f75f2ffd700 1 -- 172.21.15.60:0/2983406905 learned_addr learned my addr 172.21.15.60:0/2983406905 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:02.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.835+0000 7f75f3fff700 1 --2- 172.21.15.60:0/2983406905 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75f40b22c0 0x7f75f40641a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:02.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.835+0000 7f75f37fe700 1 -- 172.21.15.60:0/2983406905 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75f40b22c0 msgr2=0x7f75f40641a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:02.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.835+0000 7f75f37fe700 1 --2- 172.21.15.60:0/2983406905 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75f40b22c0 0x7f75f40641a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:02.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.835+0000 7f75f37fe700 1 -- 172.21.15.60:0/2983406905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75f40b02d0 msgr2=0x7f75f4063cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:02.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.835+0000 7f75f37fe700 1 --2- 172.21.15.60:0/2983406905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75f40b02d0 0x7f75f4063cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:02.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.835+0000 7f75f37fe700 1 -- 172.21.15.60:0/2983406905 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e400b1e0 con 0x7f75f405c700 2022-06-29T15:59:02.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.835+0000 7f75f37fe700 1 --2- 172.21.15.60:0/2983406905 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75f405c700 0x7f75f413eef0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f75dc00c000 tx=0x7f75dc006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:02.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.835+0000 7f75f0ff9700 1 -- 172.21.15.60:0/2983406905 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75dc011750 con 0x7f75f405c700 2022-06-29T15:59:02.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.835+0000 7f75f0ff9700 1 -- 172.21.15.60:0/2983406905 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75dc0118b0 con 0x7f75f405c700 2022-06-29T15:59:02.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.835+0000 7f75f8ae2700 1 -- 172.21.15.60:0/2983406905 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f413f3e0 con 0x7f75f405c700 2022-06-29T15:59:02.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.835+0000 7f75f0ff9700 1 -- 172.21.15.60:0/2983406905 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75dc011c10 con 0x7f75f405c700 2022-06-29T15:59:02.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.835+0000 7f75f8ae2700 1 -- 172.21.15.60:0/2983406905 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f4064690 con 0x7f75f405c700 2022-06-29T15:59:02.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.835+0000 7f75f8ae2700 1 -- 172.21.15.60:0/2983406905 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75f40ad0c0 con 0x7f75f405c700 2022-06-29T15:59:02.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.835+0000 7f75f0ff9700 1 -- 172.21.15.60:0/2983406905 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f75dc011d70 con 0x7f75f405c700 2022-06-29T15:59:02.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.835+0000 7f75f0ff9700 1 --2- 172.21.15.60:0/2983406905 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f75d8053b00 0x7f75d8055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:02.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.835+0000 7f75f0ff9700 1 -- 172.21.15.60:0/2983406905 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f75dc06bf50 con 0x7f75f405c700 2022-06-29T15:59:02.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.839+0000 7f75f2ffd700 1 --2- 172.21.15.60:0/2983406905 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f75d8053b00 0x7f75d8055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:02.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.839+0000 7f75f2ffd700 1 --2- 172.21.15.60:0/2983406905 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f75d8053b00 0x7f75d8055fc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f75e401c000 tx=0x7f75e4015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:02.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.843+0000 7f75f0ff9700 1 -- 172.21.15.60:0/2983406905 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f75dc0430a0 con 0x7f75f405c700 2022-06-29T15:59:02.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.971+0000 7f75f8ae2700 1 -- 172.21.15.60:0/2983406905 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f75f40b0880 con 0x7f75f405c700 2022-06-29T15:59:02.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.975+0000 7f75f0ff9700 1 -- 172.21.15.60:0/2983406905 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v28) v1 ==== 110+0+10 (secure 0 0 0) 0x7f75dc0466c0 con 0x7f75f405c700 2022-06-29T15:59:02.983 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T15:59:02.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.975+0000 7f75f8ae2700 1 -- 172.21.15.60:0/2983406905 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f75d8053b00 msgr2=0x7f75d8055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:02.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.975+0000 7f75f8ae2700 1 --2- 172.21.15.60:0/2983406905 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f75d8053b00 0x7f75d8055fc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f75e401c000 tx=0x7f75e4015000).stop 2022-06-29T15:59:02.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.975+0000 7f75f8ae2700 1 -- 172.21.15.60:0/2983406905 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75f405c700 msgr2=0x7f75f413eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:02.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.975+0000 7f75f8ae2700 1 --2- 172.21.15.60:0/2983406905 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75f405c700 0x7f75f413eef0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f75dc00c000 tx=0x7f75dc006df0).stop 2022-06-29T15:59:02.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.975+0000 7f75f8ae2700 1 -- 172.21.15.60:0/2983406905 shutdown_connections 2022-06-29T15:59:02.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.975+0000 7f75f8ae2700 1 --2- 172.21.15.60:0/2983406905 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f75d8053b00 0x7f75d8055fc0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:02.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.975+0000 7f75f8ae2700 1 --2- 172.21.15.60:0/2983406905 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75f40b22c0 0x7f75f40641a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:02.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.975+0000 7f75f8ae2700 1 --2- 172.21.15.60:0/2983406905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75f40b02d0 0x7f75f4063cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:02.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.975+0000 7f75f8ae2700 1 --2- 172.21.15.60:0/2983406905 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75f405c700 0x7f75f413eef0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:02.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.975+0000 7f75f8ae2700 1 -- 172.21.15.60:0/2983406905 >> 172.21.15.60:0/2983406905 conn(0x7f75f40b58a0 msgr2=0x7f75f40aa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:02.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.975+0000 7f75f8ae2700 1 -- 172.21.15.60:0/2983406905 shutdown_connections 2022-06-29T15:59:02.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:02.975+0000 7f75f8ae2700 1 -- 172.21.15.60:0/2983406905 wait complete. 2022-06-29T15:59:02.997 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T15:59:03.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.095+0000 7f603b59e700 1 Processor -- start 2022-06-29T15:59:03.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.095+0000 7f603b59e700 1 -- start start 2022-06-29T15:59:03.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.095+0000 7f603b59e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f603c0b48b0 0x7f603c142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:03.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.095+0000 7f603b59e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f603c0b2c50 0x7f603c0b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:03.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.095+0000 7f603b59e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f603c0b0cf0 con 0x7f603c0b5eb0 2022-06-29T15:59:03.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.095+0000 7f603b59e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f603c0b54e0 con 0x7f603c0b2c50 2022-06-29T15:59:03.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.095+0000 7f603b59e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f603c0b4d10 con 0x7f603c0b48b0 2022-06-29T15:59:03.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.095+0000 7f603ad9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f603c0b2c50 0x7f603c0b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:03.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.095+0000 7f603ad9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f603c0b2c50 0x7f603c0b30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46342/0 (socket says 172.21.15.60:46342) 2022-06-29T15:59:03.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.095+0000 7f603ad9d700 1 -- 172.21.15.60:0/3831545358 learned_addr learned my addr 172.21.15.60:0/3831545358 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:03.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.095+0000 7f6039d9b700 1 --2- 172.21.15.60:0/3831545358 >> v2:172.21.15.92:3300/0 conn(0x7f603c0b48b0 0x7f603c142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:03.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.095+0000 7f603ad9d700 1 -- 172.21.15.60:0/3831545358 >> v2:172.21.15.92:3300/0 conn(0x7f603c0b48b0 msgr2=0x7f603c142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:03.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.095+0000 7f603ad9d700 1 --2- 172.21.15.60:0/3831545358 >> v2:172.21.15.92:3300/0 conn(0x7f603c0b48b0 0x7f603c142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:03.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.095+0000 7f603ad9d700 1 -- 172.21.15.60:0/3831545358 >> v1:172.21.15.92:6789/0 conn(0x7f603c0b5eb0 legacy=0x7f603c0b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:03.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.095+0000 7f603ad9d700 1 -- 172.21.15.60:0/3831545358 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f603c0b6290 con 0x7f603c0b2c50 2022-06-29T15:59:03.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.095+0000 7f603959a700 1 -- 172.21.15.60:0/3831545358 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2690129963 0 0) 0x7f602c001490 con 0x7f603c0b5eb0 2022-06-29T15:59:03.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.095+0000 7f603ad9d700 1 --2- 172.21.15.60:0/3831545358 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f603c0b2c50 0x7f603c0b30c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f6034002f70 tx=0x7f6034006eb0).ready entity=mon.0 client_cookie=1705e992ae3c16c3 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:03.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.099+0000 7f603959a700 1 -- 172.21.15.60:0/3831545358 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f603400d5c0 con 0x7f603c0b2c50 2022-06-29T15:59:03.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.099+0000 7f603959a700 1 -- 172.21.15.60:0/3831545358 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6034000e90 con 0x7f603c0b2c50 2022-06-29T15:59:03.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.099+0000 7f603959a700 1 -- 172.21.15.60:0/3831545358 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f603400d910 con 0x7f603c0b2c50 2022-06-29T15:59:03.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.099+0000 7f603b59e700 1 -- 172.21.15.60:0/3831545358 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f603c0b2c50 msgr2=0x7f603c0b30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:03.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.099+0000 7f603b59e700 1 --2- 172.21.15.60:0/3831545358 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f603c0b2c50 0x7f603c0b30c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f6034002f70 tx=0x7f6034006eb0).stop 2022-06-29T15:59:03.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.099+0000 7f603b59e700 1 -- 172.21.15.60:0/3831545358 shutdown_connections 2022-06-29T15:59:03.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.099+0000 7f603b59e700 1 --2- 172.21.15.60:0/3831545358 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f603c0b2c50 0x7f603c0b30c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:03.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.099+0000 7f603b59e700 1 --2- 172.21.15.60:0/3831545358 >> v2:172.21.15.92:3300/0 conn(0x7f603c0b48b0 0x7f603c142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:03.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.099+0000 7f603b59e700 1 -- 172.21.15.60:0/3831545358 >> 172.21.15.60:0/3831545358 conn(0x7f603c062090 msgr2=0x7f603c062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:03.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.099+0000 7f603b59e700 1 -- 172.21.15.60:0/3831545358 shutdown_connections 2022-06-29T15:59:03.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.099+0000 7f603b59e700 1 -- 172.21.15.60:0/3831545358 wait complete. 2022-06-29T15:59:03.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.099+0000 7f603b59e700 1 Processor -- start 2022-06-29T15:59:03.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.099+0000 7f603b59e700 1 -- start start 2022-06-29T15:59:03.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.103+0000 7f603b59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f603c0b2c50 0x7f603c151c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:03.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.103+0000 7f603b59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f603c0b48b0 0x7f603c0aed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:03.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.103+0000 7f603b59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f603c0b5eb0 0x7f603c13fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:03.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.103+0000 7f603b59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603c145db0 con 0x7f603c0b48b0 2022-06-29T15:59:03.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.103+0000 7f603b59e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603c145570 con 0x7f603c0b5eb0 2022-06-29T15:59:03.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.103+0000 7f603b59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603c145060 con 0x7f603c0b2c50 2022-06-29T15:59:03.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.103+0000 7f6039d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f603c0b48b0 0x7f603c0aed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:03.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.103+0000 7f6039d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f603c0b48b0 0x7f603c0aed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46348/0 (socket says 172.21.15.60:46348) 2022-06-29T15:59:03.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.103+0000 7f6039d9b700 1 -- 172.21.15.60:0/1437117679 learned_addr learned my addr 172.21.15.60:0/1437117679 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:03.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.103+0000 7f603a59c700 1 --2- 172.21.15.60:0/1437117679 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f603c0b2c50 0x7f603c151c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:03.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.103+0000 7f603ad9d700 1 --2- 172.21.15.60:0/1437117679 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f603c0b5eb0 0x7f603c13fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:03.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.103+0000 7f6039d9b700 1 -- 172.21.15.60:0/1437117679 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f603c0b2c50 msgr2=0x7f603c151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:03.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.103+0000 7f6039d9b700 1 --2- 172.21.15.60:0/1437117679 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f603c0b2c50 0x7f603c151c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:03.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.103+0000 7f6039d9b700 1 -- 172.21.15.60:0/1437117679 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f603c0b5eb0 msgr2=0x7f603c13fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:03.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.103+0000 7f6039d9b700 1 --2- 172.21.15.60:0/1437117679 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f603c0b5eb0 0x7f603c13fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:03.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.103+0000 7f6039d9b700 1 -- 172.21.15.60:0/1437117679 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f603400b1e0 con 0x7f603c0b48b0 2022-06-29T15:59:03.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.103+0000 7f6039d9b700 1 --2- 172.21.15.60:0/1437117679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f603c0b48b0 0x7f603c0aed70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f602400c000 tx=0x7f6024006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:03.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.103+0000 7f60337fe700 1 -- 172.21.15.60:0/1437117679 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6024010780 con 0x7f603c0b48b0 2022-06-29T15:59:03.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.103+0000 7f60337fe700 1 -- 172.21.15.60:0/1437117679 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f602400b3b0 con 0x7f603c0b48b0 2022-06-29T15:59:03.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.103+0000 7f603b59e700 1 -- 172.21.15.60:0/1437117679 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f603c146c30 con 0x7f603c0b48b0 2022-06-29T15:59:03.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.103+0000 7f60337fe700 1 -- 172.21.15.60:0/1437117679 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6024010ad0 con 0x7f603c0b48b0 2022-06-29T15:59:03.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.103+0000 7f603b59e700 1 -- 172.21.15.60:0/1437117679 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f603c140110 con 0x7f603c0b48b0 2022-06-29T15:59:03.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.103+0000 7f603b59e700 1 -- 172.21.15.60:0/1437117679 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f603c145f20 con 0x7f603c0b48b0 2022-06-29T15:59:03.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.103+0000 7f60337fe700 1 -- 172.21.15.60:0/1437117679 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6024010c30 con 0x7f603c0b48b0 2022-06-29T15:59:03.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.103+0000 7f60337fe700 1 --2- 172.21.15.60:0/1437117679 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6020053aa0 0x7f6020055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:03.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.107+0000 7f60337fe700 1 -- 172.21.15.60:0/1437117679 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f602406bfa0 con 0x7f603c0b48b0 2022-06-29T15:59:03.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.107+0000 7f603a59c700 1 --2- 172.21.15.60:0/1437117679 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6020053aa0 0x7f6020055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:03.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.107+0000 7f603a59c700 1 --2- 172.21.15.60:0/1437117679 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6020053aa0 0x7f6020055f60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f602c00d000 tx=0x7f602c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:03.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.111+0000 7f60337fe700 1 -- 172.21.15.60:0/1437117679 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f603c145f20 con 0x7f603c0b48b0 2022-06-29T15:59:03.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.235+0000 7f603b59e700 1 -- 172.21.15.60:0/1437117679 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f603c1452e0 con 0x7f603c0b48b0 2022-06-29T15:59:03.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.239+0000 7f60337fe700 1 -- 172.21.15.60:0/1437117679 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v28) v1 ==== 92+0+10 (secure 0 0 0) 0x7f60240430f0 con 0x7f603c0b48b0 2022-06-29T15:59:03.247 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T15:59:03.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.239+0000 7f603b59e700 1 -- 172.21.15.60:0/1437117679 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6020053aa0 msgr2=0x7f6020055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:03.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.239+0000 7f603b59e700 1 --2- 172.21.15.60:0/1437117679 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6020053aa0 0x7f6020055f60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f602c00d000 tx=0x7f602c009040).stop 2022-06-29T15:59:03.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.239+0000 7f603b59e700 1 -- 172.21.15.60:0/1437117679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f603c0b48b0 msgr2=0x7f603c0aed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:03.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.239+0000 7f603b59e700 1 --2- 172.21.15.60:0/1437117679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f603c0b48b0 0x7f603c0aed70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f602400c000 tx=0x7f6024006df0).stop 2022-06-29T15:59:03.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.239+0000 7f603b59e700 1 -- 172.21.15.60:0/1437117679 shutdown_connections 2022-06-29T15:59:03.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.239+0000 7f603b59e700 1 --2- 172.21.15.60:0/1437117679 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6020053aa0 0x7f6020055f60 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:03.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.239+0000 7f603b59e700 1 --2- 172.21.15.60:0/1437117679 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f603c0b5eb0 0x7f603c13fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:03.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.239+0000 7f603b59e700 1 --2- 172.21.15.60:0/1437117679 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f603c0b48b0 0x7f603c0aed70 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:03.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.239+0000 7f603b59e700 1 --2- 172.21.15.60:0/1437117679 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f603c0b2c50 0x7f603c151c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:03.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.239+0000 7f603b59e700 1 -- 172.21.15.60:0/1437117679 >> 172.21.15.60:0/1437117679 conn(0x7f603c062090 msgr2=0x7f603c148210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:03.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.239+0000 7f603b59e700 1 -- 172.21.15.60:0/1437117679 shutdown_connections 2022-06-29T15:59:03.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.239+0000 7f603b59e700 1 -- 172.21.15.60:0/1437117679 wait complete. 2022-06-29T15:59:03.260 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-06-29T15:59:03.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.355+0000 7f49c0c5f700 1 Processor -- start 2022-06-29T15:59:03.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.355+0000 7f49c0c5f700 1 -- start start 2022-06-29T15:59:03.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.355+0000 7f49c0c5f700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f49bc0b1f30 0x7f49bc141b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:03.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.355+0000 7f49c0c5f700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49bc05d450 con 0x7f49bc0aee30 2022-06-29T15:59:03.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.355+0000 7f49c0c5f700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49bc062200 con 0x7f49bc05d070 2022-06-29T15:59:03.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.355+0000 7f49c0c5f700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49bc143a90 con 0x7f49bc0b1f30 2022-06-29T15:59:03.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.355+0000 7f49b2ffd700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f49bc0aee30 0x7f49bc140bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54442/0 (socket says 172.21.15.60:54442) 2022-06-29T15:59:03.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.355+0000 7f49b2ffd700 1 -- 172.21.15.60:0/3256635950 learned_addr learned my addr 172.21.15.60:0/3256635950 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:59:03.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.355+0000 7f49bbfff700 1 --2- 172.21.15.60:0/3256635950 >> v2:172.21.15.92:3300/0 conn(0x7f49bc0b1f30 0x7f49bc141b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:03.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.355+0000 7f49baffd700 1 -- 172.21.15.60:0/3256635950 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299611452 0 0) 0x7f49bc05d450 con 0x7f49bc0aee30 2022-06-29T15:59:03.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.355+0000 7f49baffd700 1 -- 172.21.15.60:0/3256635950 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f499c003260 con 0x7f49bc0aee30 2022-06-29T15:59:03.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.355+0000 7f49bbfff700 1 -- 172.21.15.60:0/3256635950 >> v1:172.21.15.92:6789/0 conn(0x7f49bc05d070 legacy=0x7f49bc13a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:03.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.355+0000 7f49bbfff700 1 -- 172.21.15.60:0/3256635950 >> v1:172.21.15.60:6789/0 conn(0x7f49bc0aee30 legacy=0x7f49bc140bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:03.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.355+0000 7f49bbfff700 1 -- 172.21.15.60:0/3256635950 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49bc064dc0 con 0x7f49bc0b1f30 2022-06-29T15:59:03.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.355+0000 7f49bbfff700 1 --2- 172.21.15.60:0/3256635950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49bc0b1f30 0x7f49bc141b70 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f49b40032f0 tx=0x7f49b40091d0).ready entity=mon.1 client_cookie=412677eb847d7600 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:03.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.355+0000 7f49baffd700 1 -- 172.21.15.60:0/3256635950 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f49b400b6c0 con 0x7f49bc0b1f30 2022-06-29T15:59:03.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.355+0000 7f49baffd700 1 -- 172.21.15.60:0/3256635950 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49b400b820 con 0x7f49bc0b1f30 2022-06-29T15:59:03.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.355+0000 7f49baffd700 1 -- 172.21.15.60:0/3256635950 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f49b400b990 con 0x7f49bc0b1f30 2022-06-29T15:59:03.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.355+0000 7f49c0c5f700 1 -- 172.21.15.60:0/3256635950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49bc0b1f30 msgr2=0x7f49bc141b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:03.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.355+0000 7f49c0c5f700 1 --2- 172.21.15.60:0/3256635950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49bc0b1f30 0x7f49bc141b70 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f49b40032f0 tx=0x7f49b40091d0).stop 2022-06-29T15:59:03.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.355+0000 7f49c0c5f700 1 -- 172.21.15.60:0/3256635950 shutdown_connections 2022-06-29T15:59:03.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.355+0000 7f49c0c5f700 1 --2- 172.21.15.60:0/3256635950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49bc0b1f30 0x7f49bc141b70 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:03.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.359+0000 7f49c0c5f700 1 -- 172.21.15.60:0/3256635950 >> 172.21.15.60:0/3256635950 conn(0x7f49bc0b5890 msgr2=0x7f49bc0b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:03.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.359+0000 7f49c0c5f700 1 -- 172.21.15.60:0/3256635950 shutdown_connections 2022-06-29T15:59:03.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.359+0000 7f49c0c5f700 1 -- 172.21.15.60:0/3256635950 wait complete. 2022-06-29T15:59:03.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.359+0000 7f49c0c5f700 1 Processor -- start 2022-06-29T15:59:03.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.359+0000 7f49c0c5f700 1 -- start start 2022-06-29T15:59:03.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.363+0000 7f49c0c5f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f49bc05d070 0x7f49bc1423c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:03.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.363+0000 7f49c0c5f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49bc0aee30 0x7f49bc065320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:03.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.363+0000 7f49c0c5f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49bc0b1f30 0x7f49bc065810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:03.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.363+0000 7f49c0c5f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49bc13fd20 con 0x7f49bc0aee30 2022-06-29T15:59:03.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.363+0000 7f49c0c5f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49bc13f340 con 0x7f49bc0b1f30 2022-06-29T15:59:03.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.363+0000 7f49c0c5f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49bc13f5e0 con 0x7f49bc05d070 2022-06-29T15:59:03.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.363+0000 7f49b2ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49bc0aee30 0x7f49bc065320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:03.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.363+0000 7f49b2ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49bc0aee30 0x7f49bc065320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46362/0 (socket says 172.21.15.60:46362) 2022-06-29T15:59:03.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.363+0000 7f49b2ffd700 1 -- 172.21.15.60:0/535274978 learned_addr learned my addr 172.21.15.60:0/535274978 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:03.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.363+0000 7f49bb7fe700 1 --2- 172.21.15.60:0/535274978 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f49bc05d070 0x7f49bc1423c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:03.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.363+0000 7f49bbfff700 1 --2- 172.21.15.60:0/535274978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49bc0b1f30 0x7f49bc065810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:03.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.363+0000 7f49b2ffd700 1 -- 172.21.15.60:0/535274978 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f49bc05d070 msgr2=0x7f49bc1423c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:03.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.363+0000 7f49b2ffd700 1 --2- 172.21.15.60:0/535274978 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f49bc05d070 0x7f49bc1423c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:03.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.363+0000 7f49b2ffd700 1 -- 172.21.15.60:0/535274978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49bc0b1f30 msgr2=0x7f49bc065810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:03.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.363+0000 7f49b2ffd700 1 --2- 172.21.15.60:0/535274978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49bc0b1f30 0x7f49bc065810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:03.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.363+0000 7f49b2ffd700 1 -- 172.21.15.60:0/535274978 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b4009470 con 0x7f49bc0aee30 2022-06-29T15:59:03.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.363+0000 7f49b2ffd700 1 --2- 172.21.15.60:0/535274978 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49bc0aee30 0x7f49bc065320 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f49a4001110 tx=0x7f49a4007ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:03.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.363+0000 7f49b97fa700 1 -- 172.21.15.60:0/535274978 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f49a40070d0 con 0x7f49bc0aee30 2022-06-29T15:59:03.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.363+0000 7f49b97fa700 1 -- 172.21.15.60:0/535274978 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a4007230 con 0x7f49bc0aee30 2022-06-29T15:59:03.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.363+0000 7f49b97fa700 1 -- 172.21.15.60:0/535274978 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f49a4005610 con 0x7f49bc0aee30 2022-06-29T15:59:03.382 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.363+0000 7f49c0c5f700 1 -- 172.21.15.60:0/535274978 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49bc13f880 con 0x7f49bc0aee30 2022-06-29T15:59:03.382 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.363+0000 7f49c0c5f700 1 -- 172.21.15.60:0/535274978 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49bc13ef10 con 0x7f49bc0aee30 2022-06-29T15:59:03.382 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.363+0000 7f49c0c5f700 1 -- 172.21.15.60:0/535274978 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49bc140260 con 0x7f49bc0aee30 2022-06-29T15:59:03.382 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.367+0000 7f49b97fa700 1 -- 172.21.15.60:0/535274978 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f49a4010070 con 0x7f49bc0aee30 2022-06-29T15:59:03.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.367+0000 7f49b97fa700 1 --2- 172.21.15.60:0/535274978 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f49a0053b00 0x7f49a0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:03.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.367+0000 7f49b97fa700 1 -- 172.21.15.60:0/535274978 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f49a406b240 con 0x7f49bc0aee30 2022-06-29T15:59:03.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.367+0000 7f49bb7fe700 1 --2- 172.21.15.60:0/535274978 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f49a0053b00 0x7f49a0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:03.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.367+0000 7f49bb7fe700 1 --2- 172.21.15.60:0/535274978 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f49a0053b00 0x7f49a0055fc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f49ac00d000 tx=0x7f49ac000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:03.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.371+0000 7f49b97fa700 1 -- 172.21.15.60:0/535274978 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f49bc140260 con 0x7f49bc0aee30 2022-06-29T15:59:03.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.499+0000 7f49c0c5f700 1 -- 172.21.15.60:0/535274978 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f49bc140260 con 0x7f49bc0aee30 2022-06-29T15:59:03.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.499+0000 7f49b97fa700 1 -- 172.21.15.60:0/535274978 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v28) v1 ==== 84+0+143 (secure 0 0 0) 0x7f49bc140260 con 0x7f49bc0aee30 2022-06-29T15:59:03.510 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:59:03.510 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2022-06-29T15:59:03.510 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:59:03.510 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2022-06-29T15:59:03.511 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": "device_health_metrics" 2022-06-29T15:59:03.511 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T15:59:03.511 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T15:59:03.511 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 2, 2022-06-29T15:59:03.512 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": "rbd" 2022-06-29T15:59:03.512 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T15:59:03.512 INFO:teuthology.orchestra.run.smithi060.stdout:] 2022-06-29T15:59:03.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.503+0000 7f49c0c5f700 1 -- 172.21.15.60:0/535274978 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f49a0053b00 msgr2=0x7f49a0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:03.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.503+0000 7f49c0c5f700 1 --2- 172.21.15.60:0/535274978 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f49a0053b00 0x7f49a0055fc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f49ac00d000 tx=0x7f49ac000bc0).stop 2022-06-29T15:59:03.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.503+0000 7f49c0c5f700 1 -- 172.21.15.60:0/535274978 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49bc0aee30 msgr2=0x7f49bc065320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:03.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.503+0000 7f49c0c5f700 1 --2- 172.21.15.60:0/535274978 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49bc0aee30 0x7f49bc065320 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f49a4001110 tx=0x7f49a4007ad0).stop 2022-06-29T15:59:03.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.503+0000 7f49c0c5f700 1 -- 172.21.15.60:0/535274978 shutdown_connections 2022-06-29T15:59:03.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.503+0000 7f49c0c5f700 1 --2- 172.21.15.60:0/535274978 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f49a0053b00 0x7f49a0055fc0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:03.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.503+0000 7f49c0c5f700 1 --2- 172.21.15.60:0/535274978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49bc0b1f30 0x7f49bc065810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:03.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.503+0000 7f49c0c5f700 1 --2- 172.21.15.60:0/535274978 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49bc0aee30 0x7f49bc065320 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:03.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.503+0000 7f49c0c5f700 1 --2- 172.21.15.60:0/535274978 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f49bc05d070 0x7f49bc1423c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:03.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.503+0000 7f49c0c5f700 1 -- 172.21.15.60:0/535274978 >> 172.21.15.60:0/535274978 conn(0x7f49bc0b5890 msgr2=0x7f49bc13a350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:03.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.503+0000 7f49c0c5f700 1 -- 172.21.15.60:0/535274978 shutdown_connections 2022-06-29T15:59:03.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.503+0000 7f49c0c5f700 1 -- 172.21.15.60:0/535274978 wait complete. 2022-06-29T15:59:03.523 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-06-29T15:59:03.524 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-06-29T15:59:03.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.623+0000 7f5e99fa1700 1 Processor -- start 2022-06-29T15:59:03.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.623+0000 7f5e99fa1700 1 -- start start 2022-06-29T15:59:03.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.627+0000 7f5e99fa1700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5e9405d070 0x7f5e940b1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:03.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.627+0000 7f5e99fa1700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e94141b70 0x7f5e94140bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:03.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.627+0000 7f5e99fa1700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e9405d450 con 0x7f5e940aee30 2022-06-29T15:59:03.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.627+0000 7f5e99fa1700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e94143a90 con 0x7f5e9405d070 2022-06-29T15:59:03.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.627+0000 7f5e99fa1700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e94062200 con 0x7f5e94141b70 2022-06-29T15:59:03.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.627+0000 7f5e8bfff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5e940aee30 0x7f5e9413a460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54456/0 (socket says 172.21.15.60:54456) 2022-06-29T15:59:03.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.627+0000 7f5e8bfff700 1 -- 172.21.15.60:0/3831242300 learned_addr learned my addr 172.21.15.60:0/3831242300 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:59:03.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.627+0000 7f5e98f9f700 1 --2- 172.21.15.60:0/3831242300 >> v2:172.21.15.60:3300/0 conn(0x7f5e9405d070 0x7f5e940b1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:03.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.627+0000 7f5e997a0700 1 --2- 172.21.15.60:0/3831242300 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e94141b70 0x7f5e94140bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:03.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.627+0000 7f5e8b7fe700 1 -- 172.21.15.60:0/3831242300 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1797878537 0 0) 0x7f5e9405d450 con 0x7f5e940aee30 2022-06-29T15:59:03.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.627+0000 7f5e8b7fe700 1 -- 172.21.15.60:0/3831242300 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e80003260 con 0x7f5e940aee30 2022-06-29T15:59:03.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.627+0000 7f5e997a0700 1 -- 172.21.15.60:0/3831242300 >> v2:172.21.15.60:3300/0 conn(0x7f5e9405d070 msgr2=0x7f5e940b1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:03.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.627+0000 7f5e997a0700 1 --2- 172.21.15.60:0/3831242300 >> v2:172.21.15.60:3300/0 conn(0x7f5e9405d070 0x7f5e940b1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:03.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.627+0000 7f5e997a0700 1 -- 172.21.15.60:0/3831242300 >> v1:172.21.15.60:6789/0 conn(0x7f5e940aee30 legacy=0x7f5e9413a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:03.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.627+0000 7f5e997a0700 1 -- 172.21.15.60:0/3831242300 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e94064dc0 con 0x7f5e94141b70 2022-06-29T15:59:03.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.627+0000 7f5e997a0700 1 --2- 172.21.15.60:0/3831242300 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e94141b70 0x7f5e94140bb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f5e8c0032f0 tx=0x7f5e8c0091d0).ready entity=mon.2 client_cookie=ab674906e63c6e08 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:03.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.627+0000 7f5e8b7fe700 1 -- 172.21.15.60:0/3831242300 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e8c00b600 con 0x7f5e94141b70 2022-06-29T15:59:03.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.627+0000 7f5e8b7fe700 1 -- 172.21.15.60:0/3831242300 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e8c00b760 con 0x7f5e94141b70 2022-06-29T15:59:03.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.627+0000 7f5e8b7fe700 1 -- 172.21.15.60:0/3831242300 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e8c00b8d0 con 0x7f5e94141b70 2022-06-29T15:59:03.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.627+0000 7f5e99fa1700 1 -- 172.21.15.60:0/3831242300 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e94141b70 msgr2=0x7f5e94140bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:03.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.627+0000 7f5e99fa1700 1 --2- 172.21.15.60:0/3831242300 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e94141b70 0x7f5e94140bb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f5e8c0032f0 tx=0x7f5e8c0091d0).stop 2022-06-29T15:59:03.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.627+0000 7f5e99fa1700 1 -- 172.21.15.60:0/3831242300 shutdown_connections 2022-06-29T15:59:03.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.627+0000 7f5e99fa1700 1 --2- 172.21.15.60:0/3831242300 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e94141b70 0x7f5e94140bb0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:03.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.627+0000 7f5e99fa1700 1 --2- 172.21.15.60:0/3831242300 >> v2:172.21.15.60:3300/0 conn(0x7f5e9405d070 0x7f5e940b1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:03.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.627+0000 7f5e99fa1700 1 -- 172.21.15.60:0/3831242300 >> 172.21.15.60:0/3831242300 conn(0x7f5e940b5890 msgr2=0x7f5e940b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:03.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.627+0000 7f5e99fa1700 1 -- 172.21.15.60:0/3831242300 shutdown_connections 2022-06-29T15:59:03.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.627+0000 7f5e99fa1700 1 -- 172.21.15.60:0/3831242300 wait complete. 2022-06-29T15:59:03.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.631+0000 7f5e99fa1700 1 Processor -- start 2022-06-29T15:59:03.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.631+0000 7f5e99fa1700 1 -- start start 2022-06-29T15:59:03.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.631+0000 7f5e99fa1700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e9405d070 0x7f5e94135880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:03.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.631+0000 7f5e99fa1700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e940aee30 0x7f5e940b3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:03.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.631+0000 7f5e99fa1700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e94141b70 0x7f5e94136550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:03.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.631+0000 7f5e99fa1700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e940b45d0 con 0x7f5e940aee30 2022-06-29T15:59:03.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.631+0000 7f5e99fa1700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e94150530 con 0x7f5e94141b70 2022-06-29T15:59:03.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.631+0000 7f5e99fa1700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e94150250 con 0x7f5e9405d070 2022-06-29T15:59:03.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.631+0000 7f5e8bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e940aee30 0x7f5e940b3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:03.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.631+0000 7f5e8bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e940aee30 0x7f5e940b3010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46376/0 (socket says 172.21.15.60:46376) 2022-06-29T15:59:03.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.631+0000 7f5e8bfff700 1 -- 172.21.15.60:0/2645537053 learned_addr learned my addr 172.21.15.60:0/2645537053 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:03.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.631+0000 7f5e98f9f700 1 --2- 172.21.15.60:0/2645537053 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e9405d070 0x7f5e94135880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:03.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.631+0000 7f5e997a0700 1 --2- 172.21.15.60:0/2645537053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e94141b70 0x7f5e94136550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:03.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.631+0000 7f5e8bfff700 1 -- 172.21.15.60:0/2645537053 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e9405d070 msgr2=0x7f5e94135880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:03.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.631+0000 7f5e8bfff700 1 --2- 172.21.15.60:0/2645537053 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e9405d070 0x7f5e94135880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:03.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.631+0000 7f5e8bfff700 1 -- 172.21.15.60:0/2645537053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e94141b70 msgr2=0x7f5e94136550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:03.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.631+0000 7f5e8bfff700 1 --2- 172.21.15.60:0/2645537053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e94141b70 0x7f5e94136550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:03.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.631+0000 7f5e8bfff700 1 -- 172.21.15.60:0/2645537053 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e8c009470 con 0x7f5e940aee30 2022-06-29T15:59:03.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.631+0000 7f5e997a0700 1 --2- 172.21.15.60:0/2645537053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e94141b70 0x7f5e94136550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:59:03.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.631+0000 7f5e8bfff700 1 --2- 172.21.15.60:0/2645537053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e940aee30 0x7f5e940b3010 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f5e84001110 tx=0x7f5e84007a80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:03.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.635+0000 7f5e89ffb700 1 -- 172.21.15.60:0/2645537053 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e8400ce00 con 0x7f5e940aee30 2022-06-29T15:59:03.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.635+0000 7f5e89ffb700 1 -- 172.21.15.60:0/2645537053 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e84007320 con 0x7f5e940aee30 2022-06-29T15:59:03.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.635+0000 7f5e99fa1700 1 -- 172.21.15.60:0/2645537053 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e940b3500 con 0x7f5e940aee30 2022-06-29T15:59:03.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.635+0000 7f5e89ffb700 1 -- 172.21.15.60:0/2645537053 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e84005220 con 0x7f5e940aee30 2022-06-29T15:59:03.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.635+0000 7f5e99fa1700 1 -- 172.21.15.60:0/2645537053 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e94137c80 con 0x7f5e940aee30 2022-06-29T15:59:03.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.635+0000 7f5e99fa1700 1 -- 172.21.15.60:0/2645537053 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e941506f0 con 0x7f5e940aee30 2022-06-29T15:59:03.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.635+0000 7f5e89ffb700 1 -- 172.21.15.60:0/2645537053 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5e8401b070 con 0x7f5e940aee30 2022-06-29T15:59:03.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.635+0000 7f5e89ffb700 1 --2- 172.21.15.60:0/2645537053 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5e74053aa0 0x7f5e74055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:03.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.635+0000 7f5e89ffb700 1 -- 172.21.15.60:0/2645537053 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f5e8406b1b0 con 0x7f5e940aee30 2022-06-29T15:59:03.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.635+0000 7f5e98f9f700 1 --2- 172.21.15.60:0/2645537053 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5e74053aa0 0x7f5e74055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:03.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.639+0000 7f5e98f9f700 1 --2- 172.21.15.60:0/2645537053 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5e74053aa0 0x7f5e74055f60 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f5e9000b000 tx=0x7f5e90008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:03.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.639+0000 7f5e89ffb700 1 -- 172.21.15.60:0/2645537053 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5e84042400 con 0x7f5e940aee30 2022-06-29T15:59:03.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.763+0000 7f5e99fa1700 1 -- 172.21.15.60:0/2645537053 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f5e940573d0 con 0x7f5e940aee30 2022-06-29T15:59:03.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.931+0000 7f5e89ffb700 1 -- 172.21.15.60:0/2645537053 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v29) v1 ==== 133+0+0 (secure 0 0 0) 0x7f5e84045a20 con 0x7f5e940aee30 2022-06-29T15:59:03.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.939+0000 7f5e99fa1700 1 -- 172.21.15.60:0/2645537053 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5e74053aa0 msgr2=0x7f5e74055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:03.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.939+0000 7f5e99fa1700 1 --2- 172.21.15.60:0/2645537053 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5e74053aa0 0x7f5e74055f60 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f5e9000b000 tx=0x7f5e90008040).stop 2022-06-29T15:59:03.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.939+0000 7f5e99fa1700 1 -- 172.21.15.60:0/2645537053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e940aee30 msgr2=0x7f5e940b3010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:03.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.939+0000 7f5e99fa1700 1 --2- 172.21.15.60:0/2645537053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e940aee30 0x7f5e940b3010 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f5e84001110 tx=0x7f5e84007a80).stop 2022-06-29T15:59:03.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.939+0000 7f5e99fa1700 1 -- 172.21.15.60:0/2645537053 shutdown_connections 2022-06-29T15:59:03.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.939+0000 7f5e99fa1700 1 --2- 172.21.15.60:0/2645537053 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5e74053aa0 0x7f5e74055f60 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:03.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.939+0000 7f5e99fa1700 1 --2- 172.21.15.60:0/2645537053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e94141b70 0x7f5e94136550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:03.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.939+0000 7f5e99fa1700 1 --2- 172.21.15.60:0/2645537053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e940aee30 0x7f5e940b3010 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:03.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.939+0000 7f5e99fa1700 1 --2- 172.21.15.60:0/2645537053 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e9405d070 0x7f5e94135880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:03.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.939+0000 7f5e99fa1700 1 -- 172.21.15.60:0/2645537053 >> 172.21.15.60:0/2645537053 conn(0x7f5e940b5890 msgr2=0x7f5e940b17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:03.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.939+0000 7f5e99fa1700 1 -- 172.21.15.60:0/2645537053 shutdown_connections 2022-06-29T15:59:03.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:03.939+0000 7f5e99fa1700 1 -- 172.21.15.60:0/2645537053 wait complete. 2022-06-29T15:59:03.951 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2022-06-29T15:59:03.961 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-06-29T15:59:04.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.071+0000 7f1e2181b700 1 Processor -- start 2022-06-29T15:59:04.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.071+0000 7f1e2181b700 1 -- start start 2022-06-29T15:59:04.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.071+0000 7f1e2181b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1e1c05d070 0x7f1e1c14aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:04.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.071+0000 7f1e2181b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1e1c14af80 0x7f1e1c0ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:04.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.071+0000 7f1e2181b700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e1c05d450 con 0x7f1e1c0b2130 2022-06-29T15:59:04.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.071+0000 7f1e2181b700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e1c0adf90 con 0x7f1e1c05d070 2022-06-29T15:59:04.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.071+0000 7f1e2181b700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c0b25d0 con 0x7f1e1c14af80 2022-06-29T15:59:04.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.071+0000 7f1e20819700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1e1c05d070 0x7f1e1c14aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:04.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.071+0000 7f1e20819700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1e1c05d070 0x7f1e1c14aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46382/0 (socket says 172.21.15.60:46382) 2022-06-29T15:59:04.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.071+0000 7f1e20819700 1 -- 172.21.15.60:0/3476571687 learned_addr learned my addr 172.21.15.60:0/3476571687 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:04.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.071+0000 7f1e2101a700 1 --2- 172.21.15.60:0/3476571687 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1e1c14af80 0x7f1e1c0ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:04.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.071+0000 7f1e20819700 1 -- 172.21.15.60:0/3476571687 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1e1c14af80 msgr2=0x7f1e1c0ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:04.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.071+0000 7f1e20819700 1 --2- 172.21.15.60:0/3476571687 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1e1c14af80 0x7f1e1c0ad130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:04.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.071+0000 7f1e20819700 1 -- 172.21.15.60:0/3476571687 >> v1:172.21.15.92:6789/0 conn(0x7f1e1c0b2130 legacy=0x7f1e1c0aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:04.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.071+0000 7f1e20819700 1 -- 172.21.15.60:0/3476571687 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e1c14a350 con 0x7f1e1c05d070 2022-06-29T15:59:04.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.071+0000 7f1e20819700 1 --2- 172.21.15.60:0/3476571687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e1c05d070 0x7f1e1c14aa90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f1e0c002f70 tx=0x7f1e0c006eb0).ready entity=mon.0 client_cookie=41b2677ac886e2e5 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:04.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.071+0000 7f1e1b7fe700 1 -- 172.21.15.60:0/3476571687 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e0c00d5c0 con 0x7f1e1c05d070 2022-06-29T15:59:04.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.071+0000 7f1e1b7fe700 1 -- 172.21.15.60:0/3476571687 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e0c000e90 con 0x7f1e1c05d070 2022-06-29T15:59:04.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.071+0000 7f1e1b7fe700 1 -- 172.21.15.60:0/3476571687 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e0c00d950 con 0x7f1e1c05d070 2022-06-29T15:59:04.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.075+0000 7f1e2181b700 1 -- 172.21.15.60:0/3476571687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e1c05d070 msgr2=0x7f1e1c14aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:04.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.075+0000 7f1e2181b700 1 --2- 172.21.15.60:0/3476571687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e1c05d070 0x7f1e1c14aa90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f1e0c002f70 tx=0x7f1e0c006eb0).stop 2022-06-29T15:59:04.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.075+0000 7f1e2181b700 1 -- 172.21.15.60:0/3476571687 shutdown_connections 2022-06-29T15:59:04.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.075+0000 7f1e2181b700 1 --2- 172.21.15.60:0/3476571687 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1e1c14af80 0x7f1e1c0ad130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:04.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.075+0000 7f1e2181b700 1 --2- 172.21.15.60:0/3476571687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e1c05d070 0x7f1e1c14aa90 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:04.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.075+0000 7f1e2181b700 1 -- 172.21.15.60:0/3476571687 >> 172.21.15.60:0/3476571687 conn(0x7f1e1c0b5890 msgr2=0x7f1e1c0b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:04.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.075+0000 7f1e2181b700 1 -- 172.21.15.60:0/3476571687 shutdown_connections 2022-06-29T15:59:04.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.075+0000 7f1e2181b700 1 -- 172.21.15.60:0/3476571687 wait complete. 2022-06-29T15:59:04.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.075+0000 7f1e2181b700 1 Processor -- start 2022-06-29T15:59:04.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.075+0000 7f1e2181b700 1 -- start start 2022-06-29T15:59:04.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.075+0000 7f1e2181b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e1c05d070 0x7f1e1c1598e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:04.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.075+0000 7f1e2181b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e1c0b2130 0x7f1e1c0663d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:04.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.075+0000 7f1e2181b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1e1c14af80 0x7f1e1c0b45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:04.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.075+0000 7f1e2181b700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c065650 con 0x7f1e1c0b2130 2022-06-29T15:59:04.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.075+0000 7f1e2181b700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c0658f0 con 0x7f1e1c05d070 2022-06-29T15:59:04.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.075+0000 7f1e2181b700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c0b4aa0 con 0x7f1e1c14af80 2022-06-29T15:59:04.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.075+0000 7f1e20819700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e1c05d070 0x7f1e1c1598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:04.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.075+0000 7f1e1bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e1c0b2130 0x7f1e1c0663d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:04.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.075+0000 7f1e20819700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e1c05d070 0x7f1e1c1598e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:52418/0 (socket says 172.21.15.60:52418) 2022-06-29T15:59:04.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.075+0000 7f1e20819700 1 -- 172.21.15.60:0/763828536 learned_addr learned my addr 172.21.15.60:0/763828536 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:04.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.075+0000 7f1e2101a700 1 --2- 172.21.15.60:0/763828536 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1e1c14af80 0x7f1e1c0b45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:04.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.079+0000 7f1e1bfff700 1 -- 172.21.15.60:0/763828536 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1e1c14af80 msgr2=0x7f1e1c0b45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:04.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.079+0000 7f1e1bfff700 1 --2- 172.21.15.60:0/763828536 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1e1c14af80 0x7f1e1c0b45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:04.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.079+0000 7f1e1bfff700 1 -- 172.21.15.60:0/763828536 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e1c05d070 msgr2=0x7f1e1c1598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:04.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.079+0000 7f1e1bfff700 1 --2- 172.21.15.60:0/763828536 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e1c05d070 0x7f1e1c1598e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:04.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.079+0000 7f1e1bfff700 1 -- 172.21.15.60:0/763828536 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e0c00b1e0 con 0x7f1e1c0b2130 2022-06-29T15:59:04.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.079+0000 7f1e1bfff700 1 --2- 172.21.15.60:0/763828536 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e1c0b2130 0x7f1e1c0663d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f1e1c061990 tx=0x7f1e04007f10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:04.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.079+0000 7f1e19ffb700 1 -- 172.21.15.60:0/763828536 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e0400f470 con 0x7f1e1c0b2130 2022-06-29T15:59:04.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.079+0000 7f1e19ffb700 1 -- 172.21.15.60:0/763828536 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e0400f5d0 con 0x7f1e1c0b2130 2022-06-29T15:59:04.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.079+0000 7f1e2181b700 1 -- 172.21.15.60:0/763828536 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e1c066b60 con 0x7f1e1c0b2130 2022-06-29T15:59:04.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.079+0000 7f1e19ffb700 1 -- 172.21.15.60:0/763828536 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e0400f930 con 0x7f1e1c0b2130 2022-06-29T15:59:04.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.079+0000 7f1e2181b700 1 -- 172.21.15.60:0/763828536 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e1c155760 con 0x7f1e1c0b2130 2022-06-29T15:59:04.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.079+0000 7f1e19ffb700 1 -- 172.21.15.60:0/763828536 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1e04019030 con 0x7f1e1c0b2130 2022-06-29T15:59:04.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.079+0000 7f1e2181b700 1 -- 172.21.15.60:0/763828536 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e1c14a5f0 con 0x7f1e1c0b2130 2022-06-29T15:59:04.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.079+0000 7f1e19ffb700 1 --2- 172.21.15.60:0/763828536 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1e00053aa0 0x7f1e00055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:04.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.079+0000 7f1e19ffb700 1 -- 172.21.15.60:0/763828536 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f1e0406b560 con 0x7f1e1c0b2130 2022-06-29T15:59:04.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.079+0000 7f1e20819700 1 --2- 172.21.15.60:0/763828536 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1e00053aa0 0x7f1e00055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:04.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.083+0000 7f1e20819700 1 --2- 172.21.15.60:0/763828536 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1e00053aa0 0x7f1e00055f60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f1e0c01c000 tx=0x7f1e0c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:04.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.087+0000 7f1e19ffb700 1 -- 172.21.15.60:0/763828536 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1e04042580 con 0x7f1e1c0b2130 2022-06-29T15:59:04.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.215+0000 7f1e2181b700 1 -- 172.21.15.60:0/763828536 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f1e1c065440 con 0x7f1e1c0b2130 2022-06-29T15:59:04.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.939+0000 7f1e19ffb700 1 -- 172.21.15.60:0/763828536 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v30) v1 ==== 125+0+0 (secure 0 0 0) 0x7f1e04045ba0 con 0x7f1e1c0b2130 2022-06-29T15:59:04.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.943+0000 7f1e2181b700 1 -- 172.21.15.60:0/763828536 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1e00053aa0 msgr2=0x7f1e00055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:04.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.943+0000 7f1e2181b700 1 --2- 172.21.15.60:0/763828536 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1e00053aa0 0x7f1e00055f60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f1e0c01c000 tx=0x7f1e0c015000).stop 2022-06-29T15:59:04.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.943+0000 7f1e2181b700 1 -- 172.21.15.60:0/763828536 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e1c0b2130 msgr2=0x7f1e1c0663d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:04.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.943+0000 7f1e2181b700 1 --2- 172.21.15.60:0/763828536 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e1c0b2130 0x7f1e1c0663d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f1e1c061990 tx=0x7f1e04007f10).stop 2022-06-29T15:59:04.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.943+0000 7f1e2181b700 1 -- 172.21.15.60:0/763828536 shutdown_connections 2022-06-29T15:59:04.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.943+0000 7f1e2181b700 1 --2- 172.21.15.60:0/763828536 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1e00053aa0 0x7f1e00055f60 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:04.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.943+0000 7f1e2181b700 1 --2- 172.21.15.60:0/763828536 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1e1c14af80 0x7f1e1c0b45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:04.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.943+0000 7f1e2181b700 1 --2- 172.21.15.60:0/763828536 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e1c0b2130 0x7f1e1c0663d0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:04.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.943+0000 7f1e2181b700 1 --2- 172.21.15.60:0/763828536 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e1c05d070 0x7f1e1c1598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:04.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.943+0000 7f1e2181b700 1 -- 172.21.15.60:0/763828536 >> 172.21.15.60:0/763828536 conn(0x7f1e1c0b5890 msgr2=0x7f1e1c146f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:04.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.947+0000 7f1e2181b700 1 -- 172.21.15.60:0/763828536 shutdown_connections 2022-06-29T15:59:04.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:04.947+0000 7f1e2181b700 1 -- 172.21.15.60:0/763828536 wait complete. 2022-06-29T15:59:04.958 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2022-06-29T15:59:04.967 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-06-29T15:59:05.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.063+0000 7fcee50a2700 1 Processor -- start 2022-06-29T15:59:05.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.063+0000 7fcee50a2700 1 -- start start 2022-06-29T15:59:05.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.067+0000 7fcee50a2700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcee00acd60 0x7fcee00ad710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:05.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.067+0000 7fcee50a2700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcee00b1b30 0x7fcee00b1f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:05.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.067+0000 7fcee50a2700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee0140900 con 0x7fcee01446a0 2022-06-29T15:59:05.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.067+0000 7fcee50a2700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcee0144be0 con 0x7fcee00acd60 2022-06-29T15:59:05.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.067+0000 7fcee50a2700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee0140ba0 con 0x7fcee00b1b30 2022-06-29T15:59:05.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.067+0000 7fcedf7fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcee01446a0 0x7fcee0141040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54486/0 (socket says 172.21.15.60:54486) 2022-06-29T15:59:05.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.067+0000 7fcedf7fe700 1 -- 172.21.15.60:0/239201236 learned_addr learned my addr 172.21.15.60:0/239201236 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:59:05.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.067+0000 7fcee48a1700 1 --2- 172.21.15.60:0/239201236 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcee00b1b30 0x7fcee00b1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:05.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.067+0000 7fcedffff700 1 --2- 172.21.15.60:0/239201236 >> v2:172.21.15.92:3300/0 conn(0x7fcee00acd60 0x7fcee00ad710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:05.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.067+0000 7fcedeffd700 1 -- 172.21.15.60:0/239201236 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3463066651 0 0) 0x7fcee0140900 con 0x7fcee01446a0 2022-06-29T15:59:05.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.067+0000 7fcedeffd700 1 -- 172.21.15.60:0/239201236 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcec0003260 con 0x7fcee01446a0 2022-06-29T15:59:05.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.067+0000 7fcedffff700 1 -- 172.21.15.60:0/239201236 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcee00b1b30 msgr2=0x7fcee00b1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:05.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.067+0000 7fcedffff700 1 --2- 172.21.15.60:0/239201236 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcee00b1b30 0x7fcee00b1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:05.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.067+0000 7fcedffff700 1 -- 172.21.15.60:0/239201236 >> v1:172.21.15.60:6789/0 conn(0x7fcee01446a0 legacy=0x7fcee0141040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:05.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.067+0000 7fcedffff700 1 -- 172.21.15.60:0/239201236 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee00ad140 con 0x7fcee00acd60 2022-06-29T15:59:05.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.067+0000 7fcedffff700 1 --2- 172.21.15.60:0/239201236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcee00acd60 0x7fcee00ad710 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fcec8002f70 tx=0x7fcec8006df0).ready entity=mon.1 client_cookie=fa67b75cf4d36adf server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:05.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.067+0000 7fcedeffd700 1 -- 172.21.15.60:0/239201236 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcec800d4a0 con 0x7fcee00acd60 2022-06-29T15:59:05.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.067+0000 7fcedeffd700 1 -- 172.21.15.60:0/239201236 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec8000e90 con 0x7fcee00acd60 2022-06-29T15:59:05.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.067+0000 7fcedeffd700 1 -- 172.21.15.60:0/239201236 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcec800d7f0 con 0x7fcee00acd60 2022-06-29T15:59:05.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.067+0000 7fcee50a2700 1 -- 172.21.15.60:0/239201236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcee00acd60 msgr2=0x7fcee00ad710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:05.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.067+0000 7fcee50a2700 1 --2- 172.21.15.60:0/239201236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcee00acd60 0x7fcee00ad710 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fcec8002f70 tx=0x7fcec8006df0).stop 2022-06-29T15:59:05.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.067+0000 7fcee50a2700 1 -- 172.21.15.60:0/239201236 shutdown_connections 2022-06-29T15:59:05.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.067+0000 7fcee50a2700 1 --2- 172.21.15.60:0/239201236 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcee00b1b30 0x7fcee00b1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:05.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.067+0000 7fcee50a2700 1 --2- 172.21.15.60:0/239201236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcee00acd60 0x7fcee00ad710 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:05.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.067+0000 7fcee50a2700 1 -- 172.21.15.60:0/239201236 >> 172.21.15.60:0/239201236 conn(0x7fcee00b6f80 msgr2=0x7fcee00b7390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:05.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.067+0000 7fcee50a2700 1 -- 172.21.15.60:0/239201236 shutdown_connections 2022-06-29T15:59:05.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.067+0000 7fcee50a2700 1 -- 172.21.15.60:0/239201236 wait complete. 2022-06-29T15:59:05.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.071+0000 7fcee50a2700 1 Processor -- start 2022-06-29T15:59:05.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.071+0000 7fcee50a2700 1 -- start start 2022-06-29T15:59:05.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.071+0000 7fcee50a2700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcee00acd60 0x7fcee01452e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:05.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.071+0000 7fcee50a2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcee00b1b30 0x7fcee00ae3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:05.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.071+0000 7fcee50a2700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcee01446a0 0x7fcee0151c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:05.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.071+0000 7fcee50a2700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee0152160 con 0x7fcee00b1b30 2022-06-29T15:59:05.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.071+0000 7fcee50a2700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee01523d0 con 0x7fcee01446a0 2022-06-29T15:59:05.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.071+0000 7fcee50a2700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee0152670 con 0x7fcee00acd60 2022-06-29T15:59:05.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.071+0000 7fcedf7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcee00b1b30 0x7fcee00ae3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:05.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.071+0000 7fcedf7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcee00b1b30 0x7fcee00ae3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46404/0 (socket says 172.21.15.60:46404) 2022-06-29T15:59:05.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.071+0000 7fcedf7fe700 1 -- 172.21.15.60:0/2618219741 learned_addr learned my addr 172.21.15.60:0/2618219741 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:05.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.071+0000 7fcee48a1700 1 --2- 172.21.15.60:0/2618219741 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcee01446a0 0x7fcee0151c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:05.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.071+0000 7fcedffff700 1 --2- 172.21.15.60:0/2618219741 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcee00acd60 0x7fcee01452e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:05.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.071+0000 7fcedf7fe700 1 -- 172.21.15.60:0/2618219741 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcee00acd60 msgr2=0x7fcee01452e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:05.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.071+0000 7fcedf7fe700 1 --2- 172.21.15.60:0/2618219741 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcee00acd60 0x7fcee01452e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:05.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.071+0000 7fcedf7fe700 1 -- 172.21.15.60:0/2618219741 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcee01446a0 msgr2=0x7fcee0151c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:05.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.071+0000 7fcedf7fe700 1 --2- 172.21.15.60:0/2618219741 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcee01446a0 0x7fcee0151c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:05.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.071+0000 7fcedf7fe700 1 -- 172.21.15.60:0/2618219741 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec800b180 con 0x7fcee00b1b30 2022-06-29T15:59:05.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.071+0000 7fcedf7fe700 1 --2- 172.21.15.60:0/2618219741 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcee00b1b30 0x7fcee00ae3e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fced0001110 tx=0x7fced0007a60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:05.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.071+0000 7fcedd7fa700 1 -- 172.21.15.60:0/2618219741 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fced000cdb0 con 0x7fcee00b1b30 2022-06-29T15:59:05.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.071+0000 7fcedd7fa700 1 -- 172.21.15.60:0/2618219741 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced00070d0 con 0x7fcee00b1b30 2022-06-29T15:59:05.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.071+0000 7fcee50a2700 1 -- 172.21.15.60:0/2618219741 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee0152910 con 0x7fcee00b1b30 2022-06-29T15:59:05.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.071+0000 7fcedd7fa700 1 -- 172.21.15.60:0/2618219741 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fced0007430 con 0x7fcee00b1b30 2022-06-29T15:59:05.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.071+0000 7fcee50a2700 1 -- 172.21.15.60:0/2618219741 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee0142df0 con 0x7fcee00b1b30 2022-06-29T15:59:05.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.075+0000 7fcee50a2700 1 -- 172.21.15.60:0/2618219741 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee0152ad0 con 0x7fcee00b1b30 2022-06-29T15:59:05.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.075+0000 7fcedd7fa700 1 -- 172.21.15.60:0/2618219741 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fced001a070 con 0x7fcee00b1b30 2022-06-29T15:59:05.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.075+0000 7fcedd7fa700 1 --2- 172.21.15.60:0/2618219741 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcec4053b00 0x7fcec4055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:05.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.075+0000 7fcedd7fa700 1 -- 172.21.15.60:0/2618219741 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5957+0+0 (secure 0 0 0) 0x7fced006ae30 con 0x7fcee00b1b30 2022-06-29T15:59:05.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.075+0000 7fcedffff700 1 --2- 172.21.15.60:0/2618219741 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcec4053b00 0x7fcec4055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:05.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.079+0000 7fcedffff700 1 --2- 172.21.15.60:0/2618219741 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcec4053b00 0x7fcec4055fc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fcec801a000 tx=0x7fcec8014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:05.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.083+0000 7fcedd7fa700 1 -- 172.21.15.60:0/2618219741 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fced0041cb0 con 0x7fcee00b1b30 2022-06-29T15:59:05.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.251+0000 7fcee50a2700 1 -- 172.21.15.60:0/2618219741 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7fcee00573d0 con 0x7fcee00b1b30 2022-06-29T15:59:05.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.259+0000 7fcedd7fa700 1 -- 172.21.15.60:0/2618219741 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 5 and data pool 6 v8) v1 ==== 187+0+0 (secure 0 0 0) 0x7fced00452d0 con 0x7fcee00b1b30 2022-06-29T15:59:05.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.263+0000 7fcee50a2700 1 -- 172.21.15.60:0/2618219741 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcec4053b00 msgr2=0x7fcec4055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:05.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.263+0000 7fcee50a2700 1 --2- 172.21.15.60:0/2618219741 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcec4053b00 0x7fcec4055fc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fcec801a000 tx=0x7fcec8014000).stop 2022-06-29T15:59:05.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.263+0000 7fcee50a2700 1 -- 172.21.15.60:0/2618219741 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcee00b1b30 msgr2=0x7fcee00ae3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:05.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.263+0000 7fcee50a2700 1 --2- 172.21.15.60:0/2618219741 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcee00b1b30 0x7fcee00ae3e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fced0001110 tx=0x7fced0007a60).stop 2022-06-29T15:59:05.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.263+0000 7fcee50a2700 1 -- 172.21.15.60:0/2618219741 shutdown_connections 2022-06-29T15:59:05.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.263+0000 7fcee50a2700 1 --2- 172.21.15.60:0/2618219741 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcec4053b00 0x7fcec4055fc0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:05.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.263+0000 7fcee50a2700 1 --2- 172.21.15.60:0/2618219741 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcee01446a0 0x7fcee0151c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:05.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.263+0000 7fcee50a2700 1 --2- 172.21.15.60:0/2618219741 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcee00b1b30 0x7fcee00ae3e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:05.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.263+0000 7fcee50a2700 1 --2- 172.21.15.60:0/2618219741 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcee00acd60 0x7fcee01452e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:05.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.263+0000 7fcee50a2700 1 -- 172.21.15.60:0/2618219741 >> 172.21.15.60:0/2618219741 conn(0x7fcee00b6f80 msgr2=0x7fcee00b7360 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:05.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.263+0000 7fcee50a2700 1 -- 172.21.15.60:0/2618219741 shutdown_connections 2022-06-29T15:59:05.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.263+0000 7fcee50a2700 1 -- 172.21.15.60:0/2618219741 wait complete. 2022-06-29T15:59:05.277 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 5 and data pool 6 2022-06-29T15:59:05.290 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T15:59:05.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.375+0000 7f744993b700 1 Processor -- start 2022-06-29T15:59:05.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.375+0000 7f744993b700 1 -- start start 2022-06-29T15:59:05.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.375+0000 7f744993b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f744412bde0 0x7f7444142ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:05.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.375+0000 7f744993b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7444066b20 0x7f7444142fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:05.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.375+0000 7f744993b700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74440ae9a0 con 0x7f744405d070 2022-06-29T15:59:05.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.375+0000 7f744993b700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7444143b60 con 0x7f7444066b20 2022-06-29T15:59:05.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.375+0000 7f744993b700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74440aced0 con 0x7f744412bde0 2022-06-29T15:59:05.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.375+0000 7f744913a700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7444066b20 0x7f7444142fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:05.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.375+0000 7f744913a700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7444066b20 0x7f7444142fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46412/0 (socket says 172.21.15.60:46412) 2022-06-29T15:59:05.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.375+0000 7f744913a700 1 -- 172.21.15.60:0/3089216622 learned_addr learned my addr 172.21.15.60:0/3089216622 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:05.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.375+0000 7f743bfff700 1 --2- 172.21.15.60:0/3089216622 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f744412bde0 0x7f7444142ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:05.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.375+0000 7f744913a700 1 -- 172.21.15.60:0/3089216622 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f744412bde0 msgr2=0x7f7444142ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:05.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.375+0000 7f744913a700 1 --2- 172.21.15.60:0/3089216622 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f744412bde0 0x7f7444142ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:05.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.375+0000 7f744913a700 1 -- 172.21.15.60:0/3089216622 >> v1:172.21.15.92:6789/0 conn(0x7f744405d070 legacy=0x7f744412b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:05.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.375+0000 7f744913a700 1 -- 172.21.15.60:0/3089216622 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f744405d450 con 0x7f7444066b20 2022-06-29T15:59:05.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.375+0000 7f744913a700 1 --2- 172.21.15.60:0/3089216622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7444066b20 0x7f7444142fd0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f743c0032f0 tx=0x7f743c009290).ready entity=mon.0 client_cookie=c148d9902666ea60 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:05.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.375+0000 7f743b7fe700 1 -- 172.21.15.60:0/3089216622 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f743c00b8a0 con 0x7f7444066b20 2022-06-29T15:59:05.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.375+0000 7f743b7fe700 1 -- 172.21.15.60:0/3089216622 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f743c00ba00 con 0x7f7444066b20 2022-06-29T15:59:05.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.375+0000 7f743b7fe700 1 -- 172.21.15.60:0/3089216622 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f743c00bb70 con 0x7f7444066b20 2022-06-29T15:59:05.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.375+0000 7f744993b700 1 -- 172.21.15.60:0/3089216622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7444066b20 msgr2=0x7f7444142fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:05.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.375+0000 7f744993b700 1 --2- 172.21.15.60:0/3089216622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7444066b20 0x7f7444142fd0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f743c0032f0 tx=0x7f743c009290).stop 2022-06-29T15:59:05.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.375+0000 7f744993b700 1 -- 172.21.15.60:0/3089216622 shutdown_connections 2022-06-29T15:59:05.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.375+0000 7f744993b700 1 --2- 172.21.15.60:0/3089216622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7444066b20 0x7f7444142fd0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:05.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.375+0000 7f744993b700 1 --2- 172.21.15.60:0/3089216622 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f744412bde0 0x7f7444142ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:05.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.375+0000 7f744993b700 1 -- 172.21.15.60:0/3089216622 >> 172.21.15.60:0/3089216622 conn(0x7f74440b5890 msgr2=0x7f74440b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:05.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.375+0000 7f744993b700 1 -- 172.21.15.60:0/3089216622 shutdown_connections 2022-06-29T15:59:05.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.375+0000 7f744993b700 1 -- 172.21.15.60:0/3089216622 wait complete. 2022-06-29T15:59:05.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.379+0000 7f744993b700 1 Processor -- start 2022-06-29T15:59:05.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.379+0000 7f744993b700 1 -- start start 2022-06-29T15:59:05.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.379+0000 7f744993b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f744405d070 0x7f7444142010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:05.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.379+0000 7f744993b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7444066b20 0x7f744412aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:05.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.379+0000 7f744993b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f744412bde0 0x7f744412e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:05.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.379+0000 7f744993b700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7444126a90 con 0x7f7444066b20 2022-06-29T15:59:05.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.379+0000 7f744993b700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74440b0e60 con 0x7f744412bde0 2022-06-29T15:59:05.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.379+0000 7f744993b700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f744412cf70 con 0x7f744405d070 2022-06-29T15:59:05.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.379+0000 7f743bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7444066b20 0x7f744412aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:05.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.379+0000 7f743bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7444066b20 0x7f744412aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46416/0 (socket says 172.21.15.60:46416) 2022-06-29T15:59:05.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.379+0000 7f743bfff700 1 -- 172.21.15.60:0/4191285915 learned_addr learned my addr 172.21.15.60:0/4191285915 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:05.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.379+0000 7f7448939700 1 --2- 172.21.15.60:0/4191285915 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f744405d070 0x7f7444142010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:05.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.379+0000 7f744913a700 1 --2- 172.21.15.60:0/4191285915 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f744412bde0 0x7f744412e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:05.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.379+0000 7f743bfff700 1 -- 172.21.15.60:0/4191285915 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f744405d070 msgr2=0x7f7444142010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:05.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.379+0000 7f743bfff700 1 --2- 172.21.15.60:0/4191285915 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f744405d070 0x7f7444142010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:05.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.379+0000 7f743bfff700 1 -- 172.21.15.60:0/4191285915 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f744412bde0 msgr2=0x7f744412e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:05.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.379+0000 7f743bfff700 1 --2- 172.21.15.60:0/4191285915 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f744412bde0 0x7f744412e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:05.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.379+0000 7f743bfff700 1 -- 172.21.15.60:0/4191285915 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f743c009530 con 0x7f7444066b20 2022-06-29T15:59:05.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.379+0000 7f743bfff700 1 --2- 172.21.15.60:0/4191285915 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7444066b20 0x7f744412aa10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f743400c000 tx=0x7f7434006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:05.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.383+0000 7f7439ffb700 1 -- 172.21.15.60:0/4191285915 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7434011750 con 0x7f7444066b20 2022-06-29T15:59:05.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.383+0000 7f7439ffb700 1 -- 172.21.15.60:0/4191285915 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74340118b0 con 0x7f7444066b20 2022-06-29T15:59:05.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.383+0000 7f744993b700 1 -- 172.21.15.60:0/4191285915 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f744412d210 con 0x7f7444066b20 2022-06-29T15:59:05.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.383+0000 7f7439ffb700 1 -- 172.21.15.60:0/4191285915 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7434011c10 con 0x7f7444066b20 2022-06-29T15:59:05.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.383+0000 7f744993b700 1 -- 172.21.15.60:0/4191285915 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7444142500 con 0x7f7444066b20 2022-06-29T15:59:05.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.383+0000 7f744993b700 1 -- 172.21.15.60:0/4191285915 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f744412e710 con 0x7f7444066b20 2022-06-29T15:59:05.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.383+0000 7f7439ffb700 1 -- 172.21.15.60:0/4191285915 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7434011d70 con 0x7f7444066b20 2022-06-29T15:59:05.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.383+0000 7f7439ffb700 1 --2- 172.21.15.60:0/4191285915 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f74240539e0 0x7f7424055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:05.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.383+0000 7f7439ffb700 1 -- 172.21.15.60:0/4191285915 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f743406c330 con 0x7f7444066b20 2022-06-29T15:59:05.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.383+0000 7f7448939700 1 --2- 172.21.15.60:0/4191285915 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f74240539e0 0x7f7424055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:05.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.387+0000 7f7448939700 1 --2- 172.21.15.60:0/4191285915 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f74240539e0 0x7f7424055ea0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f744000d000 tx=0x7f7440000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:05.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.391+0000 7f7439ffb700 1 -- 172.21.15.60:0/4191285915 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7434043140 con 0x7f7444066b20 2022-06-29T15:59:05.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.515+0000 7f744993b700 1 -- 172.21.15.60:0/4191285915 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74440573d0 con 0x7f7444066b20 2022-06-29T15:59:05.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.519+0000 7f7439ffb700 1 -- 172.21.15.60:0/4191285915 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+16051 (secure 0 0 0) 0x7f7434046760 con 0x7f7444066b20 2022-06-29T15:59:05.529 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:59:05.530 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":31,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T15:59:05.256745+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2022-06-29T15:59:03.821123+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2022-06-29T15:59:04.275915+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T15:59:05.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.519+0000 7f744993b700 1 -- 172.21.15.60:0/4191285915 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f74240539e0 msgr2=0x7f7424055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:05.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.519+0000 7f744993b700 1 --2- 172.21.15.60:0/4191285915 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f74240539e0 0x7f7424055ea0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f744000d000 tx=0x7f7440000bc0).stop 2022-06-29T15:59:05.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.519+0000 7f744993b700 1 -- 172.21.15.60:0/4191285915 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7444066b20 msgr2=0x7f744412aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:05.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.519+0000 7f744993b700 1 --2- 172.21.15.60:0/4191285915 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7444066b20 0x7f744412aa10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f743400c000 tx=0x7f7434006df0).stop 2022-06-29T15:59:05.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.523+0000 7f744993b700 1 -- 172.21.15.60:0/4191285915 shutdown_connections 2022-06-29T15:59:05.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.523+0000 7f744993b700 1 --2- 172.21.15.60:0/4191285915 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f74240539e0 0x7f7424055ea0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:05.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.523+0000 7f744993b700 1 --2- 172.21.15.60:0/4191285915 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f744412bde0 0x7f744412e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:05.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.523+0000 7f744993b700 1 --2- 172.21.15.60:0/4191285915 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7444066b20 0x7f744412aa10 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:05.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.523+0000 7f744993b700 1 --2- 172.21.15.60:0/4191285915 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f744405d070 0x7f7444142010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:05.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.523+0000 7f744993b700 1 -- 172.21.15.60:0/4191285915 >> 172.21.15.60:0/4191285915 conn(0x7f74440b5890 msgr2=0x7f7444148800 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:05.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.523+0000 7f744993b700 1 -- 172.21.15.60:0/4191285915 shutdown_connections 2022-06-29T15:59:05.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.523+0000 7f744993b700 1 -- 172.21.15.60:0/4191285915 wait complete. 2022-06-29T15:59:05.543 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T15:59:05.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.639+0000 7f1f40a20700 1 Processor -- start 2022-06-29T15:59:05.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.639+0000 7f1f40a20700 1 -- start start 2022-06-29T15:59:05.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.639+0000 7f1f40a20700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1f3c05d070 0x7f1f3c12b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:05.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.639+0000 7f1f40a20700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f3c0ae9a0 con 0x7f1f3c066b20 2022-06-29T15:59:05.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.639+0000 7f1f40a20700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f3c12c050 con 0x7f1f3c12bbc0 2022-06-29T15:59:05.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.639+0000 7f1f40a20700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f3c143b60 con 0x7f1f3c05d070 2022-06-29T15:59:05.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.639+0000 7f1f3b7fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1f3c05d070 0x7f1f3c12b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:05.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.639+0000 7f1f3affd700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f1f3c066b20 0x7f1f3c142ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54508/0 (socket says 172.21.15.60:54508) 2022-06-29T15:59:05.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.639+0000 7f1f3affd700 1 -- 172.21.15.60:0/3118513334 learned_addr learned my addr 172.21.15.60:0/3118513334 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:59:05.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.639+0000 7f1f3a7fc700 1 -- 172.21.15.60:0/3118513334 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1220552307 0 0) 0x7f1f3c0ae9a0 con 0x7f1f3c066b20 2022-06-29T15:59:05.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.639+0000 7f1f3a7fc700 1 -- 172.21.15.60:0/3118513334 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f1c003260 con 0x7f1f3c066b20 2022-06-29T15:59:05.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.639+0000 7f1f3b7fe700 1 -- 172.21.15.60:0/3118513334 >> v1:172.21.15.92:6789/0 conn(0x7f1f3c12bbc0 legacy=0x7f1f3c1431f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:05.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.639+0000 7f1f3b7fe700 1 -- 172.21.15.60:0/3118513334 >> v1:172.21.15.60:6789/0 conn(0x7f1f3c066b20 legacy=0x7f1f3c142ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:05.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.639+0000 7f1f3b7fe700 1 -- 172.21.15.60:0/3118513334 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f3c05d450 con 0x7f1f3c05d070 2022-06-29T15:59:05.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.639+0000 7f1f3b7fe700 1 --2- 172.21.15.60:0/3118513334 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f3c05d070 0x7f1f3c12b6d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f1f2c002f70 tx=0x7f1f2c006df0).ready entity=mon.0 client_cookie=620d081c02ec58ea server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:05.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.639+0000 7f1f3a7fc700 1 -- 172.21.15.60:0/3118513334 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f2c00d610 con 0x7f1f3c05d070 2022-06-29T15:59:05.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.639+0000 7f1f3a7fc700 1 -- 172.21.15.60:0/3118513334 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2c000e90 con 0x7f1f3c05d070 2022-06-29T15:59:05.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.639+0000 7f1f3a7fc700 1 -- 172.21.15.60:0/3118513334 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f2c00d960 con 0x7f1f3c05d070 2022-06-29T15:59:05.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.643+0000 7f1f40a20700 1 -- 172.21.15.60:0/3118513334 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f3c05d070 msgr2=0x7f1f3c12b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:05.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.643+0000 7f1f40a20700 1 --2- 172.21.15.60:0/3118513334 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f3c05d070 0x7f1f3c12b6d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f1f2c002f70 tx=0x7f1f2c006df0).stop 2022-06-29T15:59:05.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.643+0000 7f1f40a20700 1 -- 172.21.15.60:0/3118513334 shutdown_connections 2022-06-29T15:59:05.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.643+0000 7f1f40a20700 1 --2- 172.21.15.60:0/3118513334 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f3c05d070 0x7f1f3c12b6d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:05.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.643+0000 7f1f40a20700 1 -- 172.21.15.60:0/3118513334 >> 172.21.15.60:0/3118513334 conn(0x7f1f3c0b5890 msgr2=0x7f1f3c0b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:05.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.643+0000 7f1f40a20700 1 -- 172.21.15.60:0/3118513334 shutdown_connections 2022-06-29T15:59:05.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.643+0000 7f1f40a20700 1 -- 172.21.15.60:0/3118513334 wait complete. 2022-06-29T15:59:05.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.643+0000 7f1f40a20700 1 Processor -- start 2022-06-29T15:59:05.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.643+0000 7f1f40a20700 1 -- start start 2022-06-29T15:59:05.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.643+0000 7f1f40a20700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f3c05d070 0x7f1f3c12fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:05.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.643+0000 7f1f40a20700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1f3c066b20 0x7f1f3c1552b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:05.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.643+0000 7f1f40a20700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1f3c12bbc0 0x7f1f3c1557a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:05.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.643+0000 7f1f40a20700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c12cb90 con 0x7f1f3c05d070 2022-06-29T15:59:05.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.643+0000 7f1f40a20700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c12c4c0 con 0x7f1f3c066b20 2022-06-29T15:59:05.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.643+0000 7f1f40a20700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c130790 con 0x7f1f3c12bbc0 2022-06-29T15:59:05.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.643+0000 7f1f3b7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f3c05d070 0x7f1f3c12fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:05.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.643+0000 7f1f3b7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f3c05d070 0x7f1f3c12fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46428/0 (socket says 172.21.15.60:46428) 2022-06-29T15:59:05.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.643+0000 7f1f3b7fe700 1 -- 172.21.15.60:0/370435434 learned_addr learned my addr 172.21.15.60:0/370435434 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:05.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.643+0000 7f1f3affd700 1 --2- 172.21.15.60:0/370435434 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1f3c066b20 0x7f1f3c1552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:05.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.643+0000 7f1f3bfff700 1 --2- 172.21.15.60:0/370435434 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1f3c12bbc0 0x7f1f3c1557a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:05.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.643+0000 7f1f3b7fe700 1 -- 172.21.15.60:0/370435434 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1f3c12bbc0 msgr2=0x7f1f3c1557a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:05.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.643+0000 7f1f3b7fe700 1 --2- 172.21.15.60:0/370435434 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1f3c12bbc0 0x7f1f3c1557a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:05.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.643+0000 7f1f3b7fe700 1 -- 172.21.15.60:0/370435434 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1f3c066b20 msgr2=0x7f1f3c1552b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:05.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.643+0000 7f1f3b7fe700 1 --2- 172.21.15.60:0/370435434 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1f3c066b20 0x7f1f3c1552b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:05.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.643+0000 7f1f3b7fe700 1 -- 172.21.15.60:0/370435434 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f2c00b180 con 0x7f1f3c05d070 2022-06-29T15:59:05.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.647+0000 7f1f3b7fe700 1 --2- 172.21.15.60:0/370435434 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f3c05d070 0x7f1f3c12fd00 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f1f3c130920 tx=0x7f1f2c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:05.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.647+0000 7f1f38ff9700 1 -- 172.21.15.60:0/370435434 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f2c00d290 con 0x7f1f3c05d070 2022-06-29T15:59:05.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.647+0000 7f1f38ff9700 1 -- 172.21.15.60:0/370435434 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2c01e400 con 0x7f1f3c05d070 2022-06-29T15:59:05.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.647+0000 7f1f40a20700 1 -- 172.21.15.60:0/370435434 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f3c12c6f0 con 0x7f1f3c05d070 2022-06-29T15:59:05.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.647+0000 7f1f38ff9700 1 -- 172.21.15.60:0/370435434 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f2c01e760 con 0x7f1f3c05d070 2022-06-29T15:59:05.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.647+0000 7f1f40a20700 1 -- 172.21.15.60:0/370435434 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f3c0b1900 con 0x7f1f3c05d070 2022-06-29T15:59:05.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.647+0000 7f1f40a20700 1 -- 172.21.15.60:0/370435434 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f3c0573d0 con 0x7f1f3c05d070 2022-06-29T15:59:05.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.647+0000 7f1f38ff9700 1 -- 172.21.15.60:0/370435434 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1f2c025030 con 0x7f1f3c05d070 2022-06-29T15:59:05.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.647+0000 7f1f38ff9700 1 --2- 172.21.15.60:0/370435434 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1f20053a70 0x7f1f20055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:05.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.647+0000 7f1f38ff9700 1 -- 172.21.15.60:0/370435434 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1f2c06ce00 con 0x7f1f3c05d070 2022-06-29T15:59:05.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.647+0000 7f1f3affd700 1 --2- 172.21.15.60:0/370435434 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1f20053a70 0x7f1f20055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:05.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.647+0000 7f1f3affd700 1 --2- 172.21.15.60:0/370435434 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1f20053a70 0x7f1f20055f30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f1f2400b000 tx=0x7f1f24009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:05.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.655+0000 7f1f38ff9700 1 -- 172.21.15.60:0/370435434 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1f2c043d20 con 0x7f1f3c05d070 2022-06-29T15:59:05.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.783+0000 7f1f40a20700 1 -- 172.21.15.60:0/370435434 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f3c1527b0 con 0x7f1f3c05d070 2022-06-29T15:59:05.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.787+0000 7f1f38ff9700 1 -- 172.21.15.60:0/370435434 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+16051 (secure 0 0 0) 0x7f1f2c023070 con 0x7f1f3c05d070 2022-06-29T15:59:05.796 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:59:05.796 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":31,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T15:59:05.256745+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2022-06-29T15:59:03.821123+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2022-06-29T15:59:04.275915+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T15:59:05.798 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.787+0000 7f1f40a20700 1 -- 172.21.15.60:0/370435434 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1f20053a70 msgr2=0x7f1f20055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:05.798 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.787+0000 7f1f40a20700 1 --2- 172.21.15.60:0/370435434 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1f20053a70 0x7f1f20055f30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f1f2400b000 tx=0x7f1f24009040).stop 2022-06-29T15:59:05.798 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.787+0000 7f1f40a20700 1 -- 172.21.15.60:0/370435434 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f3c05d070 msgr2=0x7f1f3c12fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:05.799 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.787+0000 7f1f40a20700 1 --2- 172.21.15.60:0/370435434 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f3c05d070 0x7f1f3c12fd00 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f1f3c130920 tx=0x7f1f2c014000).stop 2022-06-29T15:59:05.799 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.787+0000 7f1f40a20700 1 -- 172.21.15.60:0/370435434 shutdown_connections 2022-06-29T15:59:05.799 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.787+0000 7f1f40a20700 1 --2- 172.21.15.60:0/370435434 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1f20053a70 0x7f1f20055f30 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:05.799 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.787+0000 7f1f40a20700 1 --2- 172.21.15.60:0/370435434 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1f3c12bbc0 0x7f1f3c1557a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:05.800 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.787+0000 7f1f40a20700 1 --2- 172.21.15.60:0/370435434 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1f3c066b20 0x7f1f3c1552b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:05.800 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.787+0000 7f1f40a20700 1 --2- 172.21.15.60:0/370435434 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f3c05d070 0x7f1f3c12fd00 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:05.800 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.787+0000 7f1f40a20700 1 -- 172.21.15.60:0/370435434 >> 172.21.15.60:0/370435434 conn(0x7f1f3c0b5890 msgr2=0x7f1f3c1283c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:05.800 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.787+0000 7f1f40a20700 1 -- 172.21.15.60:0/370435434 shutdown_connections 2022-06-29T15:59:05.801 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.787+0000 7f1f40a20700 1 -- 172.21.15.60:0/370435434 wait complete. 2022-06-29T15:59:05.810 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-06-29T15:59:05.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.911+0000 7fcbf5989700 1 Processor -- start 2022-06-29T15:59:05.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.911+0000 7fcbf5989700 1 -- start start 2022-06-29T15:59:05.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.911+0000 7fcbf5989700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcbf00b9bf0 0x7fcbf00b2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:05.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.911+0000 7fcbf5989700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbf00b0890 0x7fcbf00b4540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:05.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.915+0000 7fcbf5989700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbf00b9fd0 con 0x7fcbf00b3b40 2022-06-29T15:59:05.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.915+0000 7fcbf5989700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbf00b1a50 con 0x7fcbf00b9bf0 2022-06-29T15:59:05.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.915+0000 7fcbf5989700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf00b1cf0 con 0x7fcbf00b0890 2022-06-29T15:59:05.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.915+0000 7fcbf4987700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcbf00b9bf0 0x7fcbf00b2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:05.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.915+0000 7fcbf4987700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcbf00b9bf0 0x7fcbf00b2190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46436/0 (socket says 172.21.15.60:46436) 2022-06-29T15:59:05.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.915+0000 7fcbf4987700 1 -- 172.21.15.60:0/3444574183 learned_addr learned my addr 172.21.15.60:0/3444574183 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:05.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.915+0000 7fcbeffff700 1 --2- 172.21.15.60:0/3444574183 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbf00b0890 0x7fcbf00b4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:05.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.915+0000 7fcbf4987700 1 -- 172.21.15.60:0/3444574183 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbf00b0890 msgr2=0x7fcbf00b4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:05.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.915+0000 7fcbf4987700 1 --2- 172.21.15.60:0/3444574183 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbf00b0890 0x7fcbf00b4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:05.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.915+0000 7fcbf4987700 1 -- 172.21.15.60:0/3444574183 >> v1:172.21.15.92:6789/0 conn(0x7fcbf00b3b40 legacy=0x7fcbf005bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:05.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.915+0000 7fcbf4987700 1 -- 172.21.15.60:0/3444574183 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf00b2680 con 0x7fcbf00b9bf0 2022-06-29T15:59:05.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.915+0000 7fcbf4987700 1 --2- 172.21.15.60:0/3444574183 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbf00b9bf0 0x7fcbf00b2190 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7fcbf0061720 tx=0x7fcbe0006eb0).ready entity=mon.0 client_cookie=716b6108bbb36430 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:05.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.915+0000 7fcbef7fe700 1 -- 172.21.15.60:0/3444574183 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbe000d570 con 0x7fcbf00b9bf0 2022-06-29T15:59:05.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.915+0000 7fcbef7fe700 1 -- 172.21.15.60:0/3444574183 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbe0000e90 con 0x7fcbf00b9bf0 2022-06-29T15:59:05.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.915+0000 7fcbef7fe700 1 -- 172.21.15.60:0/3444574183 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbe000d900 con 0x7fcbf00b9bf0 2022-06-29T15:59:05.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.915+0000 7fcbf5989700 1 -- 172.21.15.60:0/3444574183 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbf00b9bf0 msgr2=0x7fcbf00b2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:05.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.915+0000 7fcbf5989700 1 --2- 172.21.15.60:0/3444574183 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbf00b9bf0 0x7fcbf00b2190 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7fcbf0061720 tx=0x7fcbe0006eb0).stop 2022-06-29T15:59:05.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.915+0000 7fcbf5989700 1 -- 172.21.15.60:0/3444574183 shutdown_connections 2022-06-29T15:59:05.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.915+0000 7fcbf5989700 1 --2- 172.21.15.60:0/3444574183 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbf00b0890 0x7fcbf00b4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:05.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.915+0000 7fcbf5989700 1 --2- 172.21.15.60:0/3444574183 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbf00b9bf0 0x7fcbf00b2190 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:05.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.915+0000 7fcbf5989700 1 -- 172.21.15.60:0/3444574183 >> 172.21.15.60:0/3444574183 conn(0x7fcbf0062b60 msgr2=0x7fcbf00b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:05.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.915+0000 7fcbf5989700 1 -- 172.21.15.60:0/3444574183 shutdown_connections 2022-06-29T15:59:05.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.915+0000 7fcbf5989700 1 -- 172.21.15.60:0/3444574183 wait complete. 2022-06-29T15:59:05.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.919+0000 7fcbf5989700 1 Processor -- start 2022-06-29T15:59:05.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.919+0000 7fcbf5989700 1 -- start start 2022-06-29T15:59:05.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.919+0000 7fcbf5989700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbf00b0890 0x7fcbf00af010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:05.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.919+0000 7fcbf5989700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbf00b3b40 0x7fcbf01136c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:05.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.919+0000 7fcbf5989700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcbf00b9bf0 0x7fcbf0113bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:05.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.919+0000 7fcbf5989700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf0146020 con 0x7fcbf00b3b40 2022-06-29T15:59:05.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.919+0000 7fcbf5989700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf0147ed0 con 0x7fcbf00b9bf0 2022-06-29T15:59:05.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.919+0000 7fcbf5989700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf0147990 con 0x7fcbf00b0890 2022-06-29T15:59:05.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.919+0000 7fcbeffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbf00b3b40 0x7fcbf01136c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:05.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.919+0000 7fcbeffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbf00b3b40 0x7fcbf01136c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46444/0 (socket says 172.21.15.60:46444) 2022-06-29T15:59:05.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.919+0000 7fcbeffff700 1 -- 172.21.15.60:0/2225498547 learned_addr learned my addr 172.21.15.60:0/2225498547 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:05.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.919+0000 7fcbf4987700 1 --2- 172.21.15.60:0/2225498547 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbf00b0890 0x7fcbf00af010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:05.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.919+0000 7fcbf5188700 1 --2- 172.21.15.60:0/2225498547 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcbf00b9bf0 0x7fcbf0113bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:05.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.919+0000 7fcbeffff700 1 -- 172.21.15.60:0/2225498547 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbf00b0890 msgr2=0x7fcbf00af010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:05.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.919+0000 7fcbeffff700 1 --2- 172.21.15.60:0/2225498547 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbf00b0890 0x7fcbf00af010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:05.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.919+0000 7fcbeffff700 1 -- 172.21.15.60:0/2225498547 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcbf00b9bf0 msgr2=0x7fcbf0113bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:05.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.919+0000 7fcbeffff700 1 --2- 172.21.15.60:0/2225498547 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcbf00b9bf0 0x7fcbf0113bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:05.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.919+0000 7fcbeffff700 1 -- 172.21.15.60:0/2225498547 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe000b1e0 con 0x7fcbf00b3b40 2022-06-29T15:59:05.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.919+0000 7fcbf4987700 1 --2- 172.21.15.60:0/2225498547 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbf00b0890 0x7fcbf00af010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:59:05.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.919+0000 7fcbeffff700 1 --2- 172.21.15.60:0/2225498547 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbf00b3b40 0x7fcbf01136c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7fcbf0151ab0 tx=0x7fcbe4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:05.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.919+0000 7fcbedffb700 1 -- 172.21.15.60:0/2225498547 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbe4010780 con 0x7fcbf00b3b40 2022-06-29T15:59:05.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.919+0000 7fcbedffb700 1 -- 172.21.15.60:0/2225498547 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbe40108e0 con 0x7fcbf00b3b40 2022-06-29T15:59:05.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.919+0000 7fcbedffb700 1 -- 172.21.15.60:0/2225498547 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbe4010c40 con 0x7fcbf00b3b40 2022-06-29T15:59:05.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.919+0000 7fcbf5989700 1 -- 172.21.15.60:0/2225498547 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbf0147c30 con 0x7fcbf00b3b40 2022-06-29T15:59:05.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.919+0000 7fcbf5989700 1 -- 172.21.15.60:0/2225498547 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbf01140a0 con 0x7fcbf00b3b40 2022-06-29T15:59:05.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.923+0000 7fcbf5989700 1 -- 172.21.15.60:0/2225498547 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbf01475b0 con 0x7fcbf00b3b40 2022-06-29T15:59:05.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.923+0000 7fcbedffb700 1 -- 172.21.15.60:0/2225498547 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcbe401c460 con 0x7fcbf00b3b40 2022-06-29T15:59:05.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.923+0000 7fcbedffb700 1 --2- 172.21.15.60:0/2225498547 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcbd0053b00 0x7fcbd0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:05.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.923+0000 7fcbedffb700 1 -- 172.21.15.60:0/2225498547 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcbe406c390 con 0x7fcbf00b3b40 2022-06-29T15:59:05.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.923+0000 7fcbf4987700 1 --2- 172.21.15.60:0/2225498547 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcbd0053b00 0x7fcbd0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:05.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.923+0000 7fcbf4987700 1 --2- 172.21.15.60:0/2225498547 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcbd0053b00 0x7fcbd0055fc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fcbe001a000 tx=0x7fcbe0015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:05.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:05.927+0000 7fcbedffb700 1 -- 172.21.15.60:0/2225498547 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcbe4043250 con 0x7fcbf00b3b40 2022-06-29T15:59:06.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.091+0000 7fcbf5989700 1 -- 172.21.15.60:0/2225498547 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fcbf00573d0 con 0x7fcbf00b3b40 2022-06-29T15:59:06.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.263+0000 7fcbedffb700 1 -- 172.21.15.60:0/2225498547 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v9) v1 ==== 118+0+0 (secure 0 0 0) 0x7fcbe4046870 con 0x7fcbf00b3b40 2022-06-29T15:59:06.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.267+0000 7fcbf5989700 1 -- 172.21.15.60:0/2225498547 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcbd0053b00 msgr2=0x7fcbd0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:06.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.267+0000 7fcbf5989700 1 --2- 172.21.15.60:0/2225498547 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcbd0053b00 0x7fcbd0055fc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fcbe001a000 tx=0x7fcbe0015000).stop 2022-06-29T15:59:06.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.267+0000 7fcbf5989700 1 -- 172.21.15.60:0/2225498547 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbf00b3b40 msgr2=0x7fcbf01136c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:06.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.267+0000 7fcbf5989700 1 --2- 172.21.15.60:0/2225498547 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbf00b3b40 0x7fcbf01136c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7fcbf0151ab0 tx=0x7fcbe4006df0).stop 2022-06-29T15:59:06.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.267+0000 7fcbf5989700 1 -- 172.21.15.60:0/2225498547 shutdown_connections 2022-06-29T15:59:06.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.267+0000 7fcbf5989700 1 --2- 172.21.15.60:0/2225498547 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcbd0053b00 0x7fcbd0055fc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:06.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.267+0000 7fcbf5989700 1 --2- 172.21.15.60:0/2225498547 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcbf00b9bf0 0x7fcbf0113bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:06.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.267+0000 7fcbf5989700 1 --2- 172.21.15.60:0/2225498547 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbf00b3b40 0x7fcbf01136c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:06.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.267+0000 7fcbf5989700 1 --2- 172.21.15.60:0/2225498547 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbf00b0890 0x7fcbf00af010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:06.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.267+0000 7fcbf5989700 1 -- 172.21.15.60:0/2225498547 >> 172.21.15.60:0/2225498547 conn(0x7fcbf0062b60 msgr2=0x7fcbf00b2870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:06.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.267+0000 7fcbf5989700 1 -- 172.21.15.60:0/2225498547 shutdown_connections 2022-06-29T15:59:06.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.267+0000 7fcbf5989700 1 -- 172.21.15.60:0/2225498547 wait complete. 2022-06-29T15:59:06.291 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T15:59:06.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.395+0000 7fc6034a1700 1 Processor -- start 2022-06-29T15:59:06.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.395+0000 7fc6034a1700 1 -- start start 2022-06-29T15:59:06.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.395+0000 7fc6034a1700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc5fc05fb00 0x7fc5fc13f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:06.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.395+0000 7fc6034a1700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5fc064e20 con 0x7fc5fc1405a0 2022-06-29T15:59:06.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.395+0000 7fc6034a1700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5fc05fee0 con 0x7fc5fc067a90 2022-06-29T15:59:06.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.395+0000 7fc6034a1700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5fc0b10e0 con 0x7fc5fc05fb00 2022-06-29T15:59:06.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.395+0000 7fc602ca0700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc5fc1405a0 0x7fc5fc143fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54540/0 (socket says 172.21.15.60:54540) 2022-06-29T15:59:06.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.395+0000 7fc602ca0700 1 -- 172.21.15.60:0/3642433668 learned_addr learned my addr 172.21.15.60:0/3642433668 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:59:06.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.395+0000 7fc60249f700 1 --2- 172.21.15.60:0/3642433668 >> v2:172.21.15.92:3300/0 conn(0x7fc5fc05fb00 0x7fc5fc13f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:06.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.395+0000 7fc60149d700 1 -- 172.21.15.60:0/3642433668 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1481162542 0 0) 0x7fc5fc064e20 con 0x7fc5fc1405a0 2022-06-29T15:59:06.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.395+0000 7fc60149d700 1 -- 172.21.15.60:0/3642433668 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5e4003260 con 0x7fc5fc1405a0 2022-06-29T15:59:06.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.395+0000 7fc60149d700 1 -- 172.21.15.60:0/3642433668 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2886639489 0 0) 0x7fc5fc05fee0 con 0x7fc5fc067a90 2022-06-29T15:59:06.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.395+0000 7fc60149d700 1 -- 172.21.15.60:0/3642433668 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5fc064e20 con 0x7fc5fc067a90 2022-06-29T15:59:06.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.395+0000 7fc60249f700 1 -- 172.21.15.60:0/3642433668 >> v1:172.21.15.92:6789/0 conn(0x7fc5fc067a90 legacy=0x7fc5fc13fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:06.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.395+0000 7fc60249f700 1 -- 172.21.15.60:0/3642433668 >> v1:172.21.15.60:6789/0 conn(0x7fc5fc1405a0 legacy=0x7fc5fc143fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:06.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.395+0000 7fc60249f700 1 -- 172.21.15.60:0/3642433668 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5fc060b20 con 0x7fc5fc05fb00 2022-06-29T15:59:06.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.395+0000 7fc60249f700 1 --2- 172.21.15.60:0/3642433668 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5fc05fb00 0x7fc5fc13f9a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fc5f8002f70 tx=0x7fc5f8006df0).ready entity=mon.1 client_cookie=b6d5182364f5ed7e server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:06.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.395+0000 7fc60149d700 1 -- 172.21.15.60:0/3642433668 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5f800d540 con 0x7fc5fc05fb00 2022-06-29T15:59:06.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.395+0000 7fc60149d700 1 -- 172.21.15.60:0/3642433668 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5f8000e90 con 0x7fc5fc05fb00 2022-06-29T15:59:06.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.395+0000 7fc60149d700 1 -- 172.21.15.60:0/3642433668 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5f800d890 con 0x7fc5fc05fb00 2022-06-29T15:59:06.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.399+0000 7fc6034a1700 1 -- 172.21.15.60:0/3642433668 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5fc05fb00 msgr2=0x7fc5fc13f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:06.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.399+0000 7fc6034a1700 1 --2- 172.21.15.60:0/3642433668 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5fc05fb00 0x7fc5fc13f9a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fc5f8002f70 tx=0x7fc5f8006df0).stop 2022-06-29T15:59:06.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.399+0000 7fc6034a1700 1 -- 172.21.15.60:0/3642433668 shutdown_connections 2022-06-29T15:59:06.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.399+0000 7fc6034a1700 1 --2- 172.21.15.60:0/3642433668 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5fc05fb00 0x7fc5fc13f9a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:06.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.399+0000 7fc6034a1700 1 -- 172.21.15.60:0/3642433668 >> 172.21.15.60:0/3642433668 conn(0x7fc5fc063a60 msgr2=0x7fc5fc05f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:06.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.399+0000 7fc6034a1700 1 -- 172.21.15.60:0/3642433668 shutdown_connections 2022-06-29T15:59:06.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.399+0000 7fc6034a1700 1 -- 172.21.15.60:0/3642433668 wait complete. 2022-06-29T15:59:06.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.399+0000 7fc6034a1700 1 Processor -- start 2022-06-29T15:59:06.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.399+0000 7fc6034a1700 1 -- start start 2022-06-29T15:59:06.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.399+0000 7fc6034a1700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5fc05fb00 0x7fc5fc153b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:06.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.399+0000 7fc6034a1700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc5fc067a90 0x7fc5fc153370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:06.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.399+0000 7fc6034a1700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5fc1405a0 0x7fc5fc1447c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:06.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.399+0000 7fc6034a1700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5fc0668b0 con 0x7fc5fc1405a0 2022-06-29T15:59:06.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.399+0000 7fc6034a1700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5fc13c250 con 0x7fc5fc05fb00 2022-06-29T15:59:06.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.399+0000 7fc6034a1700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5fc13c4f0 con 0x7fc5fc067a90 2022-06-29T15:59:06.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.399+0000 7fc602ca0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5fc1405a0 0x7fc5fc1447c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:06.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.399+0000 7fc60249f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5fc05fb00 0x7fc5fc153b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:06.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.399+0000 7fc602ca0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5fc1405a0 0x7fc5fc1447c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46460/0 (socket says 172.21.15.60:46460) 2022-06-29T15:59:06.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.399+0000 7fc601c9e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc5fc067a90 0x7fc5fc153370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:06.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.399+0000 7fc602ca0700 1 -- 172.21.15.60:0/1926281561 learned_addr learned my addr 172.21.15.60:0/1926281561 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:06.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.403+0000 7fc602ca0700 1 -- 172.21.15.60:0/1926281561 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc5fc067a90 msgr2=0x7fc5fc153370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:06.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.403+0000 7fc602ca0700 1 --2- 172.21.15.60:0/1926281561 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc5fc067a90 0x7fc5fc153370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:06.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.403+0000 7fc602ca0700 1 -- 172.21.15.60:0/1926281561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5fc05fb00 msgr2=0x7fc5fc153b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:06.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.403+0000 7fc602ca0700 1 --2- 172.21.15.60:0/1926281561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5fc05fb00 0x7fc5fc153b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:06.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.403+0000 7fc602ca0700 1 -- 172.21.15.60:0/1926281561 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f800b180 con 0x7fc5fc1405a0 2022-06-29T15:59:06.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.403+0000 7fc602ca0700 1 --2- 172.21.15.60:0/1926281561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5fc1405a0 0x7fc5fc1447c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fc5f4001490 tx=0x7fc5f4007e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:06.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.403+0000 7fc5eb7fe700 1 -- 172.21.15.60:0/1926281561 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5f4005ba0 con 0x7fc5fc1405a0 2022-06-29T15:59:06.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.403+0000 7fc5eb7fe700 1 -- 172.21.15.60:0/1926281561 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5f4005d00 con 0x7fc5fc1405a0 2022-06-29T15:59:06.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.403+0000 7fc6034a1700 1 -- 172.21.15.60:0/1926281561 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5fc0657c0 con 0x7fc5fc1405a0 2022-06-29T15:59:06.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.403+0000 7fc5eb7fe700 1 -- 172.21.15.60:0/1926281561 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5f40072c0 con 0x7fc5fc1405a0 2022-06-29T15:59:06.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.403+0000 7fc6034a1700 1 -- 172.21.15.60:0/1926281561 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5fc067170 con 0x7fc5fc1405a0 2022-06-29T15:59:06.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.403+0000 7fc6034a1700 1 -- 172.21.15.60:0/1926281561 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5fc0666f0 con 0x7fc5fc1405a0 2022-06-29T15:59:06.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.403+0000 7fc5eb7fe700 1 -- 172.21.15.60:0/1926281561 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc5f401a060 con 0x7fc5fc1405a0 2022-06-29T15:59:06.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.407+0000 7fc5eb7fe700 1 --2- 172.21.15.60:0/1926281561 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc5dc053b00 0x7fc5dc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:06.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.407+0000 7fc5eb7fe700 1 -- 172.21.15.60:0/1926281561 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc5f406b8f0 con 0x7fc5fc1405a0 2022-06-29T15:59:06.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.407+0000 7fc60249f700 1 --2- 172.21.15.60:0/1926281561 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc5dc053b00 0x7fc5dc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:06.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.407+0000 7fc60249f700 1 --2- 172.21.15.60:0/1926281561 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc5dc053b00 0x7fc5dc055fc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fc5f801a000 tx=0x7fc5f8014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:06.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.415+0000 7fc5eb7fe700 1 -- 172.21.15.60:0/1926281561 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc5f4042700 con 0x7fc5fc1405a0 2022-06-29T15:59:06.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.575+0000 7fc6034a1700 1 -- 172.21.15.60:0/1926281561 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5fc0573d0 con 0x7fc5fc1405a0 2022-06-29T15:59:06.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.575+0000 7fc5eb7fe700 1 -- 172.21.15.60:0/1926281561 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+2134 (secure 0 0 0) 0x7fc5f4018030 con 0x7fc5fc1405a0 2022-06-29T15:59:06.586 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:59:06.586 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":9,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4884,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2555925680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2555925680},{"type":"v1","addr":"172.21.15.60:6833","nonce":2555925680}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":7}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T15:59:05.257113+0000","modified":"2022-06-29T15:59:06.259879+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4933},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4933":{"gid":4933,"name":"b","rank":0,"incarnation":8,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/1993861928","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1993861928},{"type":"v1","addr":"172.21.15.92:6835","nonce":1993861928}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-06-29T15:59:06.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.575+0000 7fc6034a1700 1 -- 172.21.15.60:0/1926281561 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc5dc053b00 msgr2=0x7fc5dc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:06.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.575+0000 7fc6034a1700 1 --2- 172.21.15.60:0/1926281561 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc5dc053b00 0x7fc5dc055fc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fc5f801a000 tx=0x7fc5f8014000).stop 2022-06-29T15:59:06.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.575+0000 7fc6034a1700 1 -- 172.21.15.60:0/1926281561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5fc1405a0 msgr2=0x7fc5fc1447c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:06.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.575+0000 7fc6034a1700 1 --2- 172.21.15.60:0/1926281561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5fc1405a0 0x7fc5fc1447c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fc5f4001490 tx=0x7fc5f4007e00).stop 2022-06-29T15:59:06.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.579+0000 7fc6034a1700 1 -- 172.21.15.60:0/1926281561 shutdown_connections 2022-06-29T15:59:06.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.579+0000 7fc6034a1700 1 --2- 172.21.15.60:0/1926281561 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc5dc053b00 0x7fc5dc055fc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:06.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.579+0000 7fc6034a1700 1 --2- 172.21.15.60:0/1926281561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5fc1405a0 0x7fc5fc1447c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:06.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.579+0000 7fc6034a1700 1 --2- 172.21.15.60:0/1926281561 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc5fc067a90 0x7fc5fc153370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:06.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.579+0000 7fc6034a1700 1 --2- 172.21.15.60:0/1926281561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5fc05fb00 0x7fc5fc153b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:06.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.579+0000 7fc6034a1700 1 -- 172.21.15.60:0/1926281561 >> 172.21.15.60:0/1926281561 conn(0x7fc5fc063a60 msgr2=0x7fc5fc13f800 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:06.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.579+0000 7fc6034a1700 1 -- 172.21.15.60:0/1926281561 shutdown_connections 2022-06-29T15:59:06.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.579+0000 7fc6034a1700 1 -- 172.21.15.60:0/1926281561 wait complete. 2022-06-29T15:59:06.594 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 9 2022-06-29T15:59:06.599 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T15:59:06.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.691+0000 7f8bbce7e700 1 Processor -- start 2022-06-29T15:59:06.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.691+0000 7f8bbce7e700 1 -- start start 2022-06-29T15:59:06.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.691+0000 7f8bbce7e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8bb80b5950 0x7f8bb8067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:06.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.691+0000 7f8bbce7e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8bb814af10 0x7f8bb805c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:06.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.691+0000 7f8bbce7e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bb80acba0 con 0x7f8bb8063ad0 2022-06-29T15:59:06.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.691+0000 7f8bbce7e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bb814b3a0 con 0x7f8bb814af10 2022-06-29T15:59:06.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.691+0000 7f8bbce7e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bb8067220 con 0x7f8bb80b5950 2022-06-29T15:59:06.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.691+0000 7f8bb6d9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8bb814af10 0x7f8bb805c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:06.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.691+0000 7f8bb6d9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8bb814af10 0x7f8bb805c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46468/0 (socket says 172.21.15.60:46468) 2022-06-29T15:59:06.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.691+0000 7f8bb6d9d700 1 -- 172.21.15.60:0/4022025485 learned_addr learned my addr 172.21.15.60:0/4022025485 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:06.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.691+0000 7f8bb659c700 1 --2- 172.21.15.60:0/4022025485 >> v2:172.21.15.92:3300/0 conn(0x7f8bb80b5950 0x7f8bb8067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:06.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.691+0000 7f8bb6d9d700 1 -- 172.21.15.60:0/4022025485 >> v2:172.21.15.92:3300/0 conn(0x7f8bb80b5950 msgr2=0x7f8bb8067680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:06.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.691+0000 7f8bb6d9d700 1 --2- 172.21.15.60:0/4022025485 >> v2:172.21.15.92:3300/0 conn(0x7f8bb80b5950 0x7f8bb8067680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:06.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.691+0000 7f8bb6d9d700 1 -- 172.21.15.60:0/4022025485 >> v1:172.21.15.92:6789/0 conn(0x7f8bb8063ad0 legacy=0x7f8bb805bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:06.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.691+0000 7f8bb6d9d700 1 -- 172.21.15.60:0/4022025485 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb80b5d30 con 0x7f8bb814af10 2022-06-29T15:59:06.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.691+0000 7f8bb659c700 1 --2- 172.21.15.60:0/4022025485 >> v2:172.21.15.92:3300/0 conn(0x7f8bb80b5950 0x7f8bb8067680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:59:06.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.691+0000 7f8bb6d9d700 1 --2- 172.21.15.60:0/4022025485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bb814af10 0x7f8bb805c430 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f8bb00032f0 tx=0x7f8bb0009290).ready entity=mon.0 client_cookie=ec8d6344baae8414 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:06.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.691+0000 7f8bb559a700 1 -- 172.21.15.60:0/4022025485 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8bb000b870 con 0x7f8bb814af10 2022-06-29T15:59:06.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.691+0000 7f8bb559a700 1 -- 172.21.15.60:0/4022025485 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bb000b9d0 con 0x7f8bb814af10 2022-06-29T15:59:06.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.691+0000 7f8bb559a700 1 -- 172.21.15.60:0/4022025485 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8bb000bb40 con 0x7f8bb814af10 2022-06-29T15:59:06.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.691+0000 7f8bbce7e700 1 -- 172.21.15.60:0/4022025485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bb814af10 msgr2=0x7f8bb805c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:06.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.691+0000 7f8bbce7e700 1 --2- 172.21.15.60:0/4022025485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bb814af10 0x7f8bb805c430 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f8bb00032f0 tx=0x7f8bb0009290).stop 2022-06-29T15:59:06.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.691+0000 7f8bbce7e700 1 -- 172.21.15.60:0/4022025485 shutdown_connections 2022-06-29T15:59:06.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.691+0000 7f8bbce7e700 1 --2- 172.21.15.60:0/4022025485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bb814af10 0x7f8bb805c430 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:06.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.691+0000 7f8bbce7e700 1 --2- 172.21.15.60:0/4022025485 >> v2:172.21.15.92:3300/0 conn(0x7f8bb80b5950 0x7f8bb8067680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:06.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.691+0000 7f8bbce7e700 1 -- 172.21.15.60:0/4022025485 >> 172.21.15.60:0/4022025485 conn(0x7f8bb80632c0 msgr2=0x7f8bb80b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:06.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.691+0000 7f8bbce7e700 1 -- 172.21.15.60:0/4022025485 shutdown_connections 2022-06-29T15:59:06.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.691+0000 7f8bbce7e700 1 -- 172.21.15.60:0/4022025485 wait complete. 2022-06-29T15:59:06.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.695+0000 7f8bbce7e700 1 Processor -- start 2022-06-29T15:59:06.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.695+0000 7f8bbce7e700 1 -- start start 2022-06-29T15:59:06.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.695+0000 7f8bbce7e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bb8063ad0 0x7f8bb80ad0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:06.721 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.695+0000 7f8bbce7e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bb80b5950 0x7f8bb8143220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:06.721 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.695+0000 7f8bbce7e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bb814af10 0x7f8bb81420e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:06.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.695+0000 7f8bbce7e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb8117360 con 0x7f8bb8063ad0 2022-06-29T15:59:06.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.695+0000 7f8bbce7e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb8156000 con 0x7f8bb80b5950 2022-06-29T15:59:06.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.695+0000 7f8bbce7e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb8157100 con 0x7f8bb814af10 2022-06-29T15:59:06.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.695+0000 7f8bb659c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bb8063ad0 0x7f8bb80ad0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:06.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.695+0000 7f8bb659c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bb8063ad0 0x7f8bb80ad0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46470/0 (socket says 172.21.15.60:46470) 2022-06-29T15:59:06.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.695+0000 7f8bb659c700 1 -- 172.21.15.60:0/2535797072 learned_addr learned my addr 172.21.15.60:0/2535797072 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:06.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.695+0000 7f8bb5d9b700 1 --2- 172.21.15.60:0/2535797072 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bb80b5950 0x7f8bb8143220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:06.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.695+0000 7f8bb6d9d700 1 --2- 172.21.15.60:0/2535797072 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bb814af10 0x7f8bb81420e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:06.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.695+0000 7f8bb659c700 1 -- 172.21.15.60:0/2535797072 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bb814af10 msgr2=0x7f8bb81420e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:06.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.695+0000 7f8bb659c700 1 --2- 172.21.15.60:0/2535797072 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bb814af10 0x7f8bb81420e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:06.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.695+0000 7f8bb659c700 1 -- 172.21.15.60:0/2535797072 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bb80b5950 msgr2=0x7f8bb8143220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:06.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.695+0000 7f8bb659c700 1 --2- 172.21.15.60:0/2535797072 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bb80b5950 0x7f8bb8143220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:06.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.695+0000 7f8bb659c700 1 -- 172.21.15.60:0/2535797072 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb0009530 con 0x7f8bb8063ad0 2022-06-29T15:59:06.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.695+0000 7f8bb6d9d700 1 --2- 172.21.15.60:0/2535797072 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bb814af10 0x7f8bb81420e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:59:06.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.695+0000 7f8bb659c700 1 --2- 172.21.15.60:0/2535797072 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bb8063ad0 0x7f8bb80ad0e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f8ba0006940 tx=0x7f8ba0008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:06.728 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.695+0000 7f8baf7fe700 1 -- 172.21.15.60:0/2535797072 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ba0011850 con 0x7f8bb8063ad0 2022-06-29T15:59:06.728 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.695+0000 7f8baf7fe700 1 -- 172.21.15.60:0/2535797072 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ba00119b0 con 0x7f8bb8063ad0 2022-06-29T15:59:06.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.695+0000 7f8bbce7e700 1 -- 172.21.15.60:0/2535797072 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bb80ae2e0 con 0x7f8bb8063ad0 2022-06-29T15:59:06.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.695+0000 7f8baf7fe700 1 -- 172.21.15.60:0/2535797072 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ba0011d10 con 0x7f8bb8063ad0 2022-06-29T15:59:06.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.695+0000 7f8bbce7e700 1 -- 172.21.15.60:0/2535797072 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bb81180e0 con 0x7f8bb8063ad0 2022-06-29T15:59:06.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.699+0000 7f8bbce7e700 1 -- 172.21.15.60:0/2535797072 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bb80aed00 con 0x7f8bb8063ad0 2022-06-29T15:59:06.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.699+0000 7f8baf7fe700 1 -- 172.21.15.60:0/2535797072 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8ba0012420 con 0x7f8bb8063ad0 2022-06-29T15:59:06.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.699+0000 7f8baf7fe700 1 --2- 172.21.15.60:0/2535797072 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8b9c053a70 0x7f8b9c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:06.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.699+0000 7f8baf7fe700 1 -- 172.21.15.60:0/2535797072 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8ba006bfa0 con 0x7f8bb8063ad0 2022-06-29T15:59:06.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.699+0000 7f8bb5d9b700 1 --2- 172.21.15.60:0/2535797072 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8b9c053a70 0x7f8b9c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:06.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.699+0000 7f8bb5d9b700 1 --2- 172.21.15.60:0/2535797072 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8b9c053a70 0x7f8b9c055f30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f8ba800d000 tx=0x7f8ba8000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:06.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.707+0000 7f8baf7fe700 1 -- 172.21.15.60:0/2535797072 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8ba0042d50 con 0x7f8bb8063ad0 2022-06-29T15:59:06.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.831+0000 7f8bbce7e700 1 -- 172.21.15.60:0/2535797072 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8bb80573d0 con 0x7f8bb8063ad0 2022-06-29T15:59:06.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.835+0000 7f8baf7fe700 1 -- 172.21.15.60:0/2535797072 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16038 (secure 0 0 0) 0x7f8ba0046370 con 0x7f8bb8063ad0 2022-06-29T15:59:06.845 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:59:06.845 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":32,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T15:59:06.260590+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2022-06-29T15:59:03.821123+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2022-06-29T15:59:04.275915+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T15:59:06.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.835+0000 7f8bbce7e700 1 -- 172.21.15.60:0/2535797072 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8b9c053a70 msgr2=0x7f8b9c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:06.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.835+0000 7f8bbce7e700 1 --2- 172.21.15.60:0/2535797072 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8b9c053a70 0x7f8b9c055f30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f8ba800d000 tx=0x7f8ba8000bc0).stop 2022-06-29T15:59:06.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.835+0000 7f8bbce7e700 1 -- 172.21.15.60:0/2535797072 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bb8063ad0 msgr2=0x7f8bb80ad0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:06.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.835+0000 7f8bbce7e700 1 --2- 172.21.15.60:0/2535797072 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bb8063ad0 0x7f8bb80ad0e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f8ba0006940 tx=0x7f8ba0008040).stop 2022-06-29T15:59:06.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.835+0000 7f8bbce7e700 1 -- 172.21.15.60:0/2535797072 shutdown_connections 2022-06-29T15:59:06.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.835+0000 7f8bbce7e700 1 --2- 172.21.15.60:0/2535797072 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8b9c053a70 0x7f8b9c055f30 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:06.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.835+0000 7f8bbce7e700 1 --2- 172.21.15.60:0/2535797072 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8bb814af10 0x7f8bb81420e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:06.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.835+0000 7f8bbce7e700 1 --2- 172.21.15.60:0/2535797072 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bb80b5950 0x7f8bb8143220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:06.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.835+0000 7f8bbce7e700 1 --2- 172.21.15.60:0/2535797072 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8bb8063ad0 0x7f8bb80ad0e0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:06.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.835+0000 7f8bbce7e700 1 -- 172.21.15.60:0/2535797072 >> 172.21.15.60:0/2535797072 conn(0x7f8bb80632c0 msgr2=0x7f8bb805c470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:06.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.835+0000 7f8bbce7e700 1 -- 172.21.15.60:0/2535797072 shutdown_connections 2022-06-29T15:59:06.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.835+0000 7f8bbce7e700 1 -- 172.21.15.60:0/2535797072 wait complete. 2022-06-29T15:59:06.858 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T15:59:06.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.963+0000 7f0e9e712700 1 Processor -- start 2022-06-29T15:59:06.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.963+0000 7f0e9e712700 1 -- start start 2022-06-29T15:59:06.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.963+0000 7f0e9e712700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e980b9c20 0x7f0e980b0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:06.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.963+0000 7f0e9e712700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0e98146480 0x7f0e9805bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:06.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.963+0000 7f0e9e712700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e980b1de0 con 0x7f0e98146040 2022-06-29T15:59:06.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.963+0000 7f0e9e712700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e980b1330 con 0x7f0e98146480 2022-06-29T15:59:06.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.963+0000 7f0e9e712700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e980b2bc0 con 0x7f0e980b9c20 2022-06-29T15:59:06.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.963+0000 7f0e9df11700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0e98146480 0x7f0e9805bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:06.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.963+0000 7f0e9df11700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0e98146480 0x7f0e9805bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46482/0 (socket says 172.21.15.60:46482) 2022-06-29T15:59:06.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.963+0000 7f0e9cf0f700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f0e98146040 0x7f0e980b24b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54566/0 (socket says 172.21.15.60:54566) 2022-06-29T15:59:06.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.963+0000 7f0e9df11700 1 -- 172.21.15.60:0/1162539762 learned_addr learned my addr 172.21.15.60:0/1162539762 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:06.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.963+0000 7f0e9d710700 1 --2- 172.21.15.60:0/1162539762 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e980b9c20 0x7f0e980b0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:06.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.963+0000 7f0e9df11700 1 -- 172.21.15.60:0/1162539762 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e980b9c20 msgr2=0x7f0e980b0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:06.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.963+0000 7f0e9df11700 1 --2- 172.21.15.60:0/1162539762 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e980b9c20 0x7f0e980b0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:06.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.963+0000 7f0e9df11700 1 -- 172.21.15.60:0/1162539762 >> v1:172.21.15.60:6789/0 conn(0x7f0e98146040 legacy=0x7f0e980b24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:06.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.963+0000 7f0e8ffff700 1 -- 172.21.15.60:0/1162539762 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1917377097 0 0) 0x7f0e980b1de0 con 0x7f0e98146040 2022-06-29T15:59:06.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.963+0000 7f0e9df11700 1 -- 172.21.15.60:0/1162539762 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e980ba000 con 0x7f0e98146480 2022-06-29T15:59:06.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.967+0000 7f0e9df11700 1 --2- 172.21.15.60:0/1162539762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e98146480 0x7f0e9805bd20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f0e900032f0 tx=0x7f0e90009290).ready entity=mon.0 client_cookie=c838e6a67e7d3d7f server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:06.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.967+0000 7f0e8ffff700 1 -- 172.21.15.60:0/1162539762 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e9000b8a0 con 0x7f0e98146480 2022-06-29T15:59:06.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.967+0000 7f0e8ffff700 1 -- 172.21.15.60:0/1162539762 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e9000ba00 con 0x7f0e98146480 2022-06-29T15:59:06.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.967+0000 7f0e8ffff700 1 -- 172.21.15.60:0/1162539762 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e9000bb70 con 0x7f0e98146480 2022-06-29T15:59:06.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.967+0000 7f0e9e712700 1 -- 172.21.15.60:0/1162539762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e98146480 msgr2=0x7f0e9805bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:06.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.967+0000 7f0e9e712700 1 --2- 172.21.15.60:0/1162539762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e98146480 0x7f0e9805bd20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f0e900032f0 tx=0x7f0e90009290).stop 2022-06-29T15:59:06.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.967+0000 7f0e9e712700 1 -- 172.21.15.60:0/1162539762 shutdown_connections 2022-06-29T15:59:06.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.967+0000 7f0e9e712700 1 --2- 172.21.15.60:0/1162539762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e98146480 0x7f0e9805bd20 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:06.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.967+0000 7f0e9e712700 1 --2- 172.21.15.60:0/1162539762 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e980b9c20 0x7f0e980b0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:06.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.967+0000 7f0e9e712700 1 -- 172.21.15.60:0/1162539762 >> 172.21.15.60:0/1162539762 conn(0x7f0e98062cb0 msgr2=0x7f0e980b91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:06.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.967+0000 7f0e9e712700 1 -- 172.21.15.60:0/1162539762 shutdown_connections 2022-06-29T15:59:06.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.967+0000 7f0e9e712700 1 -- 172.21.15.60:0/1162539762 wait complete. 2022-06-29T15:59:06.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.971+0000 7f0e9e712700 1 Processor -- start 2022-06-29T15:59:06.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.971+0000 7f0e9e712700 1 -- start start 2022-06-29T15:59:06.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.971+0000 7f0e9e712700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e980b9c20 0x7f0e980acbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:06.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.971+0000 7f0e9e712700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e98146040 0x7f0e9813fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:06.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.971+0000 7f0e9e712700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e98146480 0x7f0e98140130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:06.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.971+0000 7f0e9e712700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e98140620 con 0x7f0e98146040 2022-06-29T15:59:06.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.971+0000 7f0e9e712700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e980aea50 con 0x7f0e980b9c20 2022-06-29T15:59:06.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.971+0000 7f0e9e712700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e98145210 con 0x7f0e98146480 2022-06-29T15:59:06.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.971+0000 7f0e9cf0f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e98146040 0x7f0e9813fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:06.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.971+0000 7f0e9cf0f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e98146040 0x7f0e9813fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46486/0 (socket says 172.21.15.60:46486) 2022-06-29T15:59:06.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.971+0000 7f0e9cf0f700 1 -- 172.21.15.60:0/1824874619 learned_addr learned my addr 172.21.15.60:0/1824874619 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:06.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.971+0000 7f0e9d710700 1 --2- 172.21.15.60:0/1824874619 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e980b9c20 0x7f0e980acbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:06.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.971+0000 7f0e9df11700 1 --2- 172.21.15.60:0/1824874619 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e98146480 0x7f0e98140130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:06.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.971+0000 7f0e9cf0f700 1 -- 172.21.15.60:0/1824874619 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e98146480 msgr2=0x7f0e98140130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:06.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.971+0000 7f0e9cf0f700 1 --2- 172.21.15.60:0/1824874619 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e98146480 0x7f0e98140130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:06.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.971+0000 7f0e9cf0f700 1 -- 172.21.15.60:0/1824874619 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e980b9c20 msgr2=0x7f0e980acbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:06.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.971+0000 7f0e9cf0f700 1 --2- 172.21.15.60:0/1824874619 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e980b9c20 0x7f0e980acbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:06.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.971+0000 7f0e9cf0f700 1 -- 172.21.15.60:0/1824874619 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e90009530 con 0x7f0e98146040 2022-06-29T15:59:06.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.971+0000 7f0e9df11700 1 --2- 172.21.15.60:0/1824874619 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e98146480 0x7f0e98140130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:59:06.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.971+0000 7f0e9cf0f700 1 --2- 172.21.15.60:0/1824874619 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e98146040 0x7f0e9813fc40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f0e94000ed0 tx=0x7f0e94007ac0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:06.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.971+0000 7f0e8e7fc700 1 -- 172.21.15.60:0/1824874619 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e940072e0 con 0x7f0e98146040 2022-06-29T15:59:06.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.971+0000 7f0e8e7fc700 1 -- 172.21.15.60:0/1824874619 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e94005550 con 0x7f0e98146040 2022-06-29T15:59:06.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.971+0000 7f0e8e7fc700 1 -- 172.21.15.60:0/1824874619 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e940058b0 con 0x7f0e98146040 2022-06-29T15:59:06.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.971+0000 7f0e9e712700 1 -- 172.21.15.60:0/1824874619 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e981454b0 con 0x7f0e98146040 2022-06-29T15:59:06.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.971+0000 7f0e9e712700 1 -- 172.21.15.60:0/1824874619 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e98154ef0 con 0x7f0e98146040 2022-06-29T15:59:06.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.975+0000 7f0e9e712700 1 -- 172.21.15.60:0/1824874619 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e98145950 con 0x7f0e98146040 2022-06-29T15:59:06.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.975+0000 7f0e8e7fc700 1 -- 172.21.15.60:0/1824874619 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0e94010070 con 0x7f0e98146040 2022-06-29T15:59:06.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.975+0000 7f0e8e7fc700 1 --2- 172.21.15.60:0/1824874619 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0e80053950 0x7f0e80055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:06.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.975+0000 7f0e8e7fc700 1 -- 172.21.15.60:0/1824874619 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0e9406b6e0 con 0x7f0e98146040 2022-06-29T15:59:06.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.975+0000 7f0e9d710700 1 --2- 172.21.15.60:0/1824874619 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0e80053950 0x7f0e80055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:06.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.975+0000 7f0e9d710700 1 --2- 172.21.15.60:0/1824874619 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0e80053950 0x7f0e80055e10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f0e8800b000 tx=0x7f0e88006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:06.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:06.983+0000 7f0e8e7fc700 1 -- 172.21.15.60:0/1824874619 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0e98145950 con 0x7f0e98146040 2022-06-29T15:59:07.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.147+0000 7f0e9e712700 1 -- 172.21.15.60:0/1824874619 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f0e98145950 con 0x7f0e98146040 2022-06-29T15:59:07.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.155+0000 7f0e8e7fc700 1 -- 172.21.15.60:0/1824874619 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v13) v1 ==== 184+0+0 (secure 0 0 0) 0x7f0e98145950 con 0x7f0e98146040 2022-06-29T15:59:07.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.155+0000 7f0e9e712700 1 -- 172.21.15.60:0/1824874619 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0e80053950 msgr2=0x7f0e80055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:07.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.155+0000 7f0e9e712700 1 --2- 172.21.15.60:0/1824874619 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0e80053950 0x7f0e80055e10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f0e8800b000 tx=0x7f0e88006df0).stop 2022-06-29T15:59:07.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.155+0000 7f0e9e712700 1 -- 172.21.15.60:0/1824874619 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e98146040 msgr2=0x7f0e9813fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:07.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.155+0000 7f0e9e712700 1 --2- 172.21.15.60:0/1824874619 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e98146040 0x7f0e9813fc40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f0e94000ed0 tx=0x7f0e94007ac0).stop 2022-06-29T15:59:07.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.155+0000 7f0e9e712700 1 -- 172.21.15.60:0/1824874619 shutdown_connections 2022-06-29T15:59:07.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.155+0000 7f0e9e712700 1 --2- 172.21.15.60:0/1824874619 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0e80053950 0x7f0e80055e10 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:07.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.155+0000 7f0e9e712700 1 --2- 172.21.15.60:0/1824874619 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e98146480 0x7f0e98140130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:07.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.155+0000 7f0e9e712700 1 --2- 172.21.15.60:0/1824874619 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e98146040 0x7f0e9813fc40 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:07.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.155+0000 7f0e9e712700 1 --2- 172.21.15.60:0/1824874619 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e980b9c20 0x7f0e980acbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:07.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.155+0000 7f0e9e712700 1 -- 172.21.15.60:0/1824874619 >> 172.21.15.60:0/1824874619 conn(0x7f0e98062cb0 msgr2=0x7f0e98140e00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:07.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.155+0000 7f0e9e712700 1 -- 172.21.15.60:0/1824874619 shutdown_connections 2022-06-29T15:59:07.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.155+0000 7f0e9e712700 1 -- 172.21.15.60:0/1824874619 wait complete. 2022-06-29T15:59:07.169 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.0 2022-06-29T15:59:07.178 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T15:59:07.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.279+0000 7f29e93bf700 1 Processor -- start 2022-06-29T15:59:07.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.279+0000 7f29e93bf700 1 -- start start 2022-06-29T15:59:07.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.279+0000 7f29e93bf700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f29e40b5eb0 0x7f29e4142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:07.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.279+0000 7f29e93bf700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f29e4146d60 0x7f29e40b2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:07.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.279+0000 7f29e93bf700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e40b6290 con 0x7f29e40b0280 2022-06-29T15:59:07.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.279+0000 7f29e93bf700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29e40b0ab0 con 0x7f29e4146d60 2022-06-29T15:59:07.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.279+0000 7f29e93bf700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29e41461b0 con 0x7f29e40b5eb0 2022-06-29T15:59:07.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.279+0000 7f29e8bbe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f29e4146d60 0x7f29e40b2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:07.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.279+0000 7f29e8bbe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f29e4146d60 0x7f29e40b2bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46494/0 (socket says 172.21.15.60:46494) 2022-06-29T15:59:07.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.279+0000 7f29e8bbe700 1 -- 172.21.15.60:0/2484565086 learned_addr learned my addr 172.21.15.60:0/2484565086 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:07.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.279+0000 7f29e3fff700 1 --2- 172.21.15.60:0/2484565086 >> v2:172.21.15.92:3300/0 conn(0x7f29e40b5eb0 0x7f29e4142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:07.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.279+0000 7f29e8bbe700 1 -- 172.21.15.60:0/2484565086 >> v2:172.21.15.92:3300/0 conn(0x7f29e40b5eb0 msgr2=0x7f29e4142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:07.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.279+0000 7f29e8bbe700 1 --2- 172.21.15.60:0/2484565086 >> v2:172.21.15.92:3300/0 conn(0x7f29e40b5eb0 0x7f29e4142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:07.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.279+0000 7f29e8bbe700 1 -- 172.21.15.60:0/2484565086 >> v1:172.21.15.92:6789/0 conn(0x7f29e40b0280 legacy=0x7f29e4146650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:07.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.279+0000 7f29e8bbe700 1 -- 172.21.15.60:0/2484565086 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e4142f10 con 0x7f29e4146d60 2022-06-29T15:59:07.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.279+0000 7f29e8bbe700 1 --2- 172.21.15.60:0/2484565086 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29e4146d60 0x7f29e40b2bb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f29d40032f0 tx=0x7f29d4009290).ready entity=mon.0 client_cookie=149054db4d2530b2 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:07.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.283+0000 7f29e2ffd700 1 -- 172.21.15.60:0/2484565086 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29d400b870 con 0x7f29e4146d60 2022-06-29T15:59:07.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.283+0000 7f29e2ffd700 1 -- 172.21.15.60:0/2484565086 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d400b9d0 con 0x7f29e4146d60 2022-06-29T15:59:07.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.283+0000 7f29e2ffd700 1 -- 172.21.15.60:0/2484565086 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29d400bb40 con 0x7f29e4146d60 2022-06-29T15:59:07.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.283+0000 7f29e93bf700 1 -- 172.21.15.60:0/2484565086 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29e4146d60 msgr2=0x7f29e40b2bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:07.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.283+0000 7f29e93bf700 1 --2- 172.21.15.60:0/2484565086 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29e4146d60 0x7f29e40b2bb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f29d40032f0 tx=0x7f29d4009290).stop 2022-06-29T15:59:07.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.283+0000 7f29e93bf700 1 -- 172.21.15.60:0/2484565086 shutdown_connections 2022-06-29T15:59:07.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.283+0000 7f29e93bf700 1 --2- 172.21.15.60:0/2484565086 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29e4146d60 0x7f29e40b2bb0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:07.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.283+0000 7f29e93bf700 1 --2- 172.21.15.60:0/2484565086 >> v2:172.21.15.92:3300/0 conn(0x7f29e40b5eb0 0x7f29e4142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:07.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.283+0000 7f29e93bf700 1 -- 172.21.15.60:0/2484565086 >> 172.21.15.60:0/2484565086 conn(0x7f29e4062090 msgr2=0x7f29e4062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:07.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.283+0000 7f29e93bf700 1 -- 172.21.15.60:0/2484565086 shutdown_connections 2022-06-29T15:59:07.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.283+0000 7f29e93bf700 1 -- 172.21.15.60:0/2484565086 wait complete. 2022-06-29T15:59:07.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.283+0000 7f29e93bf700 1 Processor -- start 2022-06-29T15:59:07.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.283+0000 7f29e93bf700 1 -- start start 2022-06-29T15:59:07.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.283+0000 7f29e93bf700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f29e40b0280 0x7f29e4155660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:07.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.283+0000 7f29e93bf700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f29e40b5eb0 0x7f29e40aef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:07.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.283+0000 7f29e93bf700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29e4146d60 0x7f29e40b1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:07.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.283+0000 7f29e93bf700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e40b2df0 con 0x7f29e4146d60 2022-06-29T15:59:07.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.283+0000 7f29e93bf700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e41562d0 con 0x7f29e40b0280 2022-06-29T15:59:07.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.283+0000 7f29e93bf700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e4155b50 con 0x7f29e40b5eb0 2022-06-29T15:59:07.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.283+0000 7f29e8bbe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29e4146d60 0x7f29e40b1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:07.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.283+0000 7f29e8bbe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29e4146d60 0x7f29e40b1670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46502/0 (socket says 172.21.15.60:46502) 2022-06-29T15:59:07.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.283+0000 7f29e8bbe700 1 -- 172.21.15.60:0/497948386 learned_addr learned my addr 172.21.15.60:0/497948386 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:07.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.283+0000 7f29e3fff700 1 --2- 172.21.15.60:0/497948386 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f29e40b0280 0x7f29e4155660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:07.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.283+0000 7f29e37fe700 1 --2- 172.21.15.60:0/497948386 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f29e40b5eb0 0x7f29e40aef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:07.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.283+0000 7f29e8bbe700 1 -- 172.21.15.60:0/497948386 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f29e40b5eb0 msgr2=0x7f29e40aef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:07.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.283+0000 7f29e8bbe700 1 --2- 172.21.15.60:0/497948386 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f29e40b5eb0 0x7f29e40aef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.283+0000 7f29e8bbe700 1 -- 172.21.15.60:0/497948386 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f29e40b0280 msgr2=0x7f29e4155660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.283+0000 7f29e8bbe700 1 --2- 172.21.15.60:0/497948386 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f29e40b0280 0x7f29e4155660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.283+0000 7f29e8bbe700 1 -- 172.21.15.60:0/497948386 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d4009530 con 0x7f29e4146d60 2022-06-29T15:59:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.283+0000 7f29e3fff700 1 --2- 172.21.15.60:0/497948386 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f29e40b0280 0x7f29e4155660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:59:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.283+0000 7f29e37fe700 1 --2- 172.21.15.60:0/497948386 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f29e40b5eb0 0x7f29e40aef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:59:07.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.287+0000 7f29e8bbe700 1 --2- 172.21.15.60:0/497948386 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29e4146d60 0x7f29e40b1670 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f29d40032f0 tx=0x7f29d4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:07.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.287+0000 7f29e17fa700 1 -- 172.21.15.60:0/497948386 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29d4021630 con 0x7f29e4146d60 2022-06-29T15:59:07.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.287+0000 7f29e93bf700 1 -- 172.21.15.60:0/497948386 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e4155d90 con 0x7f29e4146d60 2022-06-29T15:59:07.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.287+0000 7f29e17fa700 1 -- 172.21.15.60:0/497948386 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d4021790 con 0x7f29e4146d60 2022-06-29T15:59:07.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.287+0000 7f29e17fa700 1 -- 172.21.15.60:0/497948386 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29d4021af0 con 0x7f29e4146d60 2022-06-29T15:59:07.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.287+0000 7f29e93bf700 1 -- 172.21.15.60:0/497948386 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e4152e80 con 0x7f29e4146d60 2022-06-29T15:59:07.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.287+0000 7f29e93bf700 1 -- 172.21.15.60:0/497948386 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29e40aec00 con 0x7f29e4146d60 2022-06-29T15:59:07.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.287+0000 7f29e17fa700 1 -- 172.21.15.60:0/497948386 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f29d400f070 con 0x7f29e4146d60 2022-06-29T15:59:07.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.287+0000 7f29e17fa700 1 --2- 172.21.15.60:0/497948386 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f29c8053a10 0x7f29c8055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:07.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.287+0000 7f29e17fa700 1 -- 172.21.15.60:0/497948386 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f29d406d670 con 0x7f29e4146d60 2022-06-29T15:59:07.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.287+0000 7f29e3fff700 1 --2- 172.21.15.60:0/497948386 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f29c8053a10 0x7f29c8055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:07.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.291+0000 7f29e3fff700 1 --2- 172.21.15.60:0/497948386 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f29c8053a10 0x7f29c8055ed0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f29dc00c000 tx=0x7f29dc008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:07.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.295+0000 7f29e17fa700 1 -- 172.21.15.60:0/497948386 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f29d40444f0 con 0x7f29e4146d60 2022-06-29T15:59:07.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.463+0000 7f29e93bf700 1 -- 172.21.15.60:0/497948386 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f29e40573d0 con 0x7f29e4146d60 2022-06-29T15:59:07.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.467+0000 7f29e17fa700 1 -- 172.21.15.60:0/497948386 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v14) v1 ==== 184+0+0 (secure 0 0 0) 0x7f29d4015070 con 0x7f29e4146d60 2022-06-29T15:59:07.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.471+0000 7f29e93bf700 1 -- 172.21.15.60:0/497948386 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f29c8053a10 msgr2=0x7f29c8055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:07.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.471+0000 7f29e93bf700 1 --2- 172.21.15.60:0/497948386 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f29c8053a10 0x7f29c8055ed0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f29dc00c000 tx=0x7f29dc008040).stop 2022-06-29T15:59:07.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.471+0000 7f29e93bf700 1 -- 172.21.15.60:0/497948386 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29e4146d60 msgr2=0x7f29e40b1670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:07.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.471+0000 7f29e93bf700 1 --2- 172.21.15.60:0/497948386 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29e4146d60 0x7f29e40b1670 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f29d40032f0 tx=0x7f29d4016000).stop 2022-06-29T15:59:07.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.471+0000 7f29e93bf700 1 -- 172.21.15.60:0/497948386 shutdown_connections 2022-06-29T15:59:07.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.471+0000 7f29e93bf700 1 --2- 172.21.15.60:0/497948386 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f29c8053a10 0x7f29c8055ed0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:07.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.471+0000 7f29e93bf700 1 --2- 172.21.15.60:0/497948386 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29e4146d60 0x7f29e40b1670 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:07.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.471+0000 7f29e93bf700 1 --2- 172.21.15.60:0/497948386 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f29e40b5eb0 0x7f29e40aef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:07.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.471+0000 7f29e93bf700 1 --2- 172.21.15.60:0/497948386 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f29e40b0280 0x7f29e4155660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:07.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.471+0000 7f29e93bf700 1 -- 172.21.15.60:0/497948386 >> 172.21.15.60:0/497948386 conn(0x7f29e4062090 msgr2=0x7f29e41427f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:07.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.471+0000 7f29e93bf700 1 -- 172.21.15.60:0/497948386 shutdown_connections 2022-06-29T15:59:07.486 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.471+0000 7f29e93bf700 1 -- 172.21.15.60:0/497948386 wait complete. 2022-06-29T15:59:07.486 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.1 2022-06-29T15:59:07.495 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T15:59:07.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.607+0000 7f430ad9c700 1 Processor -- start 2022-06-29T15:59:07.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.607+0000 7f430ad9c700 1 -- start start 2022-06-29T15:59:07.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.607+0000 7f430ad9c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f43041446e0 0x7f43040b4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:07.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.607+0000 7f430ad9c700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f43040b4920 0x7f43040b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:07.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.607+0000 7f430ad9c700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43040657f0 con 0x7f4304065410 2022-06-29T15:59:07.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.607+0000 7f430ad9c700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4304147da0 con 0x7f43041446e0 2022-06-29T15:59:07.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.607+0000 7f430ad9c700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4304063b40 con 0x7f43040b4920 2022-06-29T15:59:07.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.607+0000 7f4303fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f43041446e0 0x7f43040b4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:07.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.607+0000 7f4303fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f43041446e0 0x7f43040b4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46508/0 (socket says 172.21.15.60:46508) 2022-06-29T15:59:07.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.607+0000 7f4303fff700 1 -- 172.21.15.60:0/795161260 learned_addr learned my addr 172.21.15.60:0/795161260 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:07.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.607+0000 7f4309339700 1 --2- 172.21.15.60:0/795161260 >> v2:172.21.15.92:3300/0 conn(0x7f43040b4920 0x7f43040b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:07.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.607+0000 7f4303fff700 1 -- 172.21.15.60:0/795161260 >> v2:172.21.15.92:3300/0 conn(0x7f43040b4920 msgr2=0x7f43040b4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:07.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.607+0000 7f4303fff700 1 --2- 172.21.15.60:0/795161260 >> v2:172.21.15.92:3300/0 conn(0x7f43040b4920 0x7f43040b4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:07.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.607+0000 7f4303fff700 1 -- 172.21.15.60:0/795161260 >> v1:172.21.15.92:6789/0 conn(0x7f4304065410 legacy=0x7f4304143fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:07.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.607+0000 7f4303fff700 1 -- 172.21.15.60:0/795161260 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4304147980 con 0x7f43041446e0 2022-06-29T15:59:07.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.607+0000 7f4303fff700 1 --2- 172.21.15.60:0/795161260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43041446e0 0x7f43040b4010 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f42f8002f70 tx=0x7f42f8006eb0).ready entity=mon.0 client_cookie=f6fa4a2f78538fb4 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:07.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.607+0000 7f43037fe700 1 -- 172.21.15.60:0/795161260 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f42f800d5c0 con 0x7f43041446e0 2022-06-29T15:59:07.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.607+0000 7f43037fe700 1 -- 172.21.15.60:0/795161260 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42f8000e90 con 0x7f43041446e0 2022-06-29T15:59:07.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.607+0000 7f43037fe700 1 -- 172.21.15.60:0/795161260 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f42f800d950 con 0x7f43041446e0 2022-06-29T15:59:07.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.611+0000 7f430ad9c700 1 -- 172.21.15.60:0/795161260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43041446e0 msgr2=0x7f43040b4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:07.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.611+0000 7f430ad9c700 1 --2- 172.21.15.60:0/795161260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43041446e0 0x7f43040b4010 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f42f8002f70 tx=0x7f42f8006eb0).stop 2022-06-29T15:59:07.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.611+0000 7f430ad9c700 1 -- 172.21.15.60:0/795161260 shutdown_connections 2022-06-29T15:59:07.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.611+0000 7f430ad9c700 1 --2- 172.21.15.60:0/795161260 >> v2:172.21.15.92:3300/0 conn(0x7f43040b4920 0x7f43040b4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:07.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.611+0000 7f430ad9c700 1 --2- 172.21.15.60:0/795161260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43041446e0 0x7f43040b4010 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:07.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.611+0000 7f430ad9c700 1 -- 172.21.15.60:0/795161260 >> 172.21.15.60:0/795161260 conn(0x7f43040b9340 msgr2=0x7f43040b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:07.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.611+0000 7f430ad9c700 1 -- 172.21.15.60:0/795161260 shutdown_connections 2022-06-29T15:59:07.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.611+0000 7f430ad9c700 1 -- 172.21.15.60:0/795161260 wait complete. 2022-06-29T15:59:07.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.611+0000 7f430ad9c700 1 Processor -- start 2022-06-29T15:59:07.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.611+0000 7f430ad9c700 1 -- start start 2022-06-29T15:59:07.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.615+0000 7f430ad9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4304065410 0x7f430414d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:07.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.615+0000 7f430ad9c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43040b4920 0x7f4304156560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:07.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.615+0000 7f430ad9c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43041446e0 0x7f4304156a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:07.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.615+0000 7f430ad9c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4304159b10 con 0x7f43041446e0 2022-06-29T15:59:07.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.615+0000 7f430ad9c700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4304159db0 con 0x7f4304065410 2022-06-29T15:59:07.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.615+0000 7f430ad9c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f430415a050 con 0x7f43040b4920 2022-06-29T15:59:07.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.615+0000 7f4309339700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43041446e0 0x7f4304156a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:07.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.615+0000 7f4303fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43040b4920 0x7f4304156560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:07.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.615+0000 7f4308b38700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4304065410 0x7f430414d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:07.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.615+0000 7f4309339700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43041446e0 0x7f4304156a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46516/0 (socket says 172.21.15.60:46516) 2022-06-29T15:59:07.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.615+0000 7f4309339700 1 -- 172.21.15.60:0/987213795 learned_addr learned my addr 172.21.15.60:0/987213795 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:07.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.615+0000 7f4309339700 1 -- 172.21.15.60:0/987213795 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43040b4920 msgr2=0x7f4304156560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:07.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.615+0000 7f4309339700 1 --2- 172.21.15.60:0/987213795 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43040b4920 0x7f4304156560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:07.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.615+0000 7f4309339700 1 -- 172.21.15.60:0/987213795 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4304065410 msgr2=0x7f430414d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:07.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.615+0000 7f4309339700 1 --2- 172.21.15.60:0/987213795 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4304065410 0x7f430414d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:07.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.615+0000 7f4309339700 1 -- 172.21.15.60:0/987213795 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f800b1e0 con 0x7f43041446e0 2022-06-29T15:59:07.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.615+0000 7f4309339700 1 --2- 172.21.15.60:0/987213795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43041446e0 0x7f4304156a50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f42fc006e00 tx=0x7f42fc0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:07.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.615+0000 7f4301ffb700 1 -- 172.21.15.60:0/987213795 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f42fc012600 con 0x7f43041446e0 2022-06-29T15:59:07.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.615+0000 7f4301ffb700 1 -- 172.21.15.60:0/987213795 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42fc012760 con 0x7f43041446e0 2022-06-29T15:59:07.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.615+0000 7f430ad9c700 1 -- 172.21.15.60:0/987213795 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f430415a2f0 con 0x7f43041446e0 2022-06-29T15:59:07.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.615+0000 7f4301ffb700 1 -- 172.21.15.60:0/987213795 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f42fc012ac0 con 0x7f43041446e0 2022-06-29T15:59:07.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.615+0000 7f430ad9c700 1 -- 172.21.15.60:0/987213795 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f430414cc40 con 0x7f43041446e0 2022-06-29T15:59:07.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.619+0000 7f430ad9c700 1 -- 172.21.15.60:0/987213795 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43040573d0 con 0x7f43041446e0 2022-06-29T15:59:07.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.619+0000 7f4301ffb700 1 -- 172.21.15.60:0/987213795 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f42fc022070 con 0x7f43041446e0 2022-06-29T15:59:07.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.619+0000 7f4301ffb700 1 --2- 172.21.15.60:0/987213795 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f42e4053b00 0x7f42e4055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:07.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.619+0000 7f4301ffb700 1 -- 172.21.15.60:0/987213795 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f42fc06c580 con 0x7f43041446e0 2022-06-29T15:59:07.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.619+0000 7f4308b38700 1 --2- 172.21.15.60:0/987213795 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f42e4053b00 0x7f42e4055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:07.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.619+0000 7f4308b38700 1 --2- 172.21.15.60:0/987213795 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f42e4053b00 0x7f42e4055fc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f42f400d000 tx=0x7f42f4007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:07.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.627+0000 7f4301ffb700 1 -- 172.21.15.60:0/987213795 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f42fc043310 con 0x7f43041446e0 2022-06-29T15:59:07.798 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.787+0000 7f430ad9c700 1 -- 172.21.15.60:0/987213795 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f430405fdb0 con 0x7f43041446e0 2022-06-29T15:59:07.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.795+0000 7f4301ffb700 1 -- 172.21.15.60:0/987213795 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v15) v1 ==== 184+0+0 (secure 0 0 0) 0x7f42fc0160e0 con 0x7f43041446e0 2022-06-29T15:59:07.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.799+0000 7f430ad9c700 1 -- 172.21.15.60:0/987213795 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f42e4053b00 msgr2=0x7f42e4055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:07.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.799+0000 7f430ad9c700 1 --2- 172.21.15.60:0/987213795 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f42e4053b00 0x7f42e4055fc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f42f400d000 tx=0x7f42f4007b90).stop 2022-06-29T15:59:07.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.799+0000 7f430ad9c700 1 -- 172.21.15.60:0/987213795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43041446e0 msgr2=0x7f4304156a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:07.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.799+0000 7f430ad9c700 1 --2- 172.21.15.60:0/987213795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43041446e0 0x7f4304156a50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f42fc006e00 tx=0x7f42fc0091d0).stop 2022-06-29T15:59:07.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.799+0000 7f430ad9c700 1 -- 172.21.15.60:0/987213795 shutdown_connections 2022-06-29T15:59:07.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.799+0000 7f430ad9c700 1 --2- 172.21.15.60:0/987213795 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f42e4053b00 0x7f42e4055fc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:07.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.799+0000 7f430ad9c700 1 --2- 172.21.15.60:0/987213795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43041446e0 0x7f4304156a50 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:07.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.799+0000 7f430ad9c700 1 --2- 172.21.15.60:0/987213795 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43040b4920 0x7f4304156560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:07.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.799+0000 7f430ad9c700 1 --2- 172.21.15.60:0/987213795 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4304065410 0x7f430414d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:07.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.799+0000 7f430ad9c700 1 -- 172.21.15.60:0/987213795 >> 172.21.15.60:0/987213795 conn(0x7f43040b9340 msgr2=0x7f43040b8040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:07.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.799+0000 7f430ad9c700 1 -- 172.21.15.60:0/987213795 shutdown_connections 2022-06-29T15:59:07.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.799+0000 7f430ad9c700 1 -- 172.21.15.60:0/987213795 wait complete. 2022-06-29T15:59:07.811 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.2 2022-06-29T15:59:07.820 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T15:59:07.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.911+0000 7f384599e700 1 Processor -- start 2022-06-29T15:59:07.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.911+0000 7f384599e700 1 -- start start 2022-06-29T15:59:07.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.911+0000 7f384599e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38400b61d0 0x7f38400b0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:07.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.911+0000 7f384599e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f384013f790 0x7f384013bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:07.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.911+0000 7f384599e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3840142820 con 0x7f384013fbd0 2022-06-29T15:59:07.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.911+0000 7f384599e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f384005c430 con 0x7f384013f790 2022-06-29T15:59:07.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.911+0000 7f384599e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3840062bb0 con 0x7f38400b61d0 2022-06-29T15:59:07.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.911+0000 7f3837fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f384013f790 0x7f384013bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:07.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.911+0000 7f383effd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38400b61d0 0x7f38400b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:07.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.911+0000 7f3837fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f384013f790 0x7f384013bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:52550/0 (socket says 172.21.15.60:52550) 2022-06-29T15:59:07.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.911+0000 7f3837fff700 1 -- 172.21.15.60:0/2241440147 learned_addr learned my addr 172.21.15.60:0/2241440147 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:07.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.911+0000 7f383effd700 1 -- 172.21.15.60:0/2241440147 >> v2:172.21.15.92:3300/0 conn(0x7f384013f790 msgr2=0x7f384013bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:07.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.911+0000 7f383effd700 1 --2- 172.21.15.60:0/2241440147 >> v2:172.21.15.92:3300/0 conn(0x7f384013f790 0x7f384013bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:07.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.911+0000 7f383effd700 1 -- 172.21.15.60:0/2241440147 >> v1:172.21.15.92:6789/0 conn(0x7f384013fbd0 legacy=0x7f384005bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:07.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.911+0000 7f383effd700 1 -- 172.21.15.60:0/2241440147 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38400b65b0 con 0x7f38400b61d0 2022-06-29T15:59:07.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.911+0000 7f383e7fc700 1 -- 172.21.15.60:0/2241440147 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2858113928 0 0) 0x7f3840142820 con 0x7f384013fbd0 2022-06-29T15:59:07.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.911+0000 7f3837fff700 1 --2- 172.21.15.60:0/2241440147 >> v2:172.21.15.92:3300/0 conn(0x7f384013f790 0x7f384013bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:59:07.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.911+0000 7f383effd700 1 --2- 172.21.15.60:0/2241440147 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38400b61d0 0x7f38400b0540 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f3830002f70 tx=0x7f3830006df0).ready entity=mon.2 client_cookie=5f0434e29e48daae server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:07.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.911+0000 7f383e7fc700 1 -- 172.21.15.60:0/2241440147 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f383000d570 con 0x7f38400b61d0 2022-06-29T15:59:07.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.911+0000 7f383e7fc700 1 -- 172.21.15.60:0/2241440147 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3830000e90 con 0x7f38400b61d0 2022-06-29T15:59:07.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.911+0000 7f383e7fc700 1 -- 172.21.15.60:0/2241440147 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f383000d8c0 con 0x7f38400b61d0 2022-06-29T15:59:07.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.915+0000 7f384599e700 1 -- 172.21.15.60:0/2241440147 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38400b61d0 msgr2=0x7f38400b0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:07.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.915+0000 7f384599e700 1 --2- 172.21.15.60:0/2241440147 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38400b61d0 0x7f38400b0540 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f3830002f70 tx=0x7f3830006df0).stop 2022-06-29T15:59:07.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.915+0000 7f384599e700 1 -- 172.21.15.60:0/2241440147 shutdown_connections 2022-06-29T15:59:07.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.915+0000 7f384599e700 1 --2- 172.21.15.60:0/2241440147 >> v2:172.21.15.92:3300/0 conn(0x7f384013f790 0x7f384013bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:07.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.915+0000 7f384599e700 1 --2- 172.21.15.60:0/2241440147 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38400b61d0 0x7f38400b0540 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:07.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.915+0000 7f384599e700 1 -- 172.21.15.60:0/2241440147 >> 172.21.15.60:0/2241440147 conn(0x7f3840063b50 msgr2=0x7f38400b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:07.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.915+0000 7f384599e700 1 -- 172.21.15.60:0/2241440147 shutdown_connections 2022-06-29T15:59:07.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.915+0000 7f384599e700 1 -- 172.21.15.60:0/2241440147 wait complete. 2022-06-29T15:59:07.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.915+0000 7f384599e700 1 Processor -- start 2022-06-29T15:59:07.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.915+0000 7f384599e700 1 -- start start 2022-06-29T15:59:07.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.915+0000 7f384599e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38400b61d0 0x7f38401319a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:07.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.915+0000 7f384599e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f384013f790 0x7f3840132550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:07.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.915+0000 7f384599e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f384013fbd0 0x7f38401345f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:07.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.915+0000 7f384599e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3840135020 con 0x7f384013fbd0 2022-06-29T15:59:07.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.915+0000 7f384599e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3840135250 con 0x7f384013f790 2022-06-29T15:59:07.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.915+0000 7f384599e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3840133c30 con 0x7f38400b61d0 2022-06-29T15:59:07.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.919+0000 7f383effd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38400b61d0 0x7f38401319a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:07.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.919+0000 7f383effd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38400b61d0 0x7f38401319a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:59620/0 (socket says 172.21.15.60:59620) 2022-06-29T15:59:07.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.919+0000 7f383f7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f384013fbd0 0x7f38401345f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:07.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.919+0000 7f383effd700 1 -- 172.21.15.60:0/3918581581 learned_addr learned my addr 172.21.15.60:0/3918581581 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:07.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.919+0000 7f3837fff700 1 --2- 172.21.15.60:0/3918581581 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f384013f790 0x7f3840132550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:07.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.919+0000 7f383f7fe700 1 -- 172.21.15.60:0/3918581581 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38400b61d0 msgr2=0x7f38401319a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:07.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.919+0000 7f383f7fe700 1 --2- 172.21.15.60:0/3918581581 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38400b61d0 0x7f38401319a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:07.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.919+0000 7f383f7fe700 1 -- 172.21.15.60:0/3918581581 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f384013f790 msgr2=0x7f3840132550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:07.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.919+0000 7f383f7fe700 1 --2- 172.21.15.60:0/3918581581 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f384013f790 0x7f3840132550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:07.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.919+0000 7f383f7fe700 1 -- 172.21.15.60:0/3918581581 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f382800b1e0 con 0x7f384013fbd0 2022-06-29T15:59:07.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.919+0000 7f383effd700 1 --2- 172.21.15.60:0/3918581581 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38400b61d0 0x7f38401319a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:59:07.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.919+0000 7f383f7fe700 1 --2- 172.21.15.60:0/3918581581 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f384013fbd0 0x7f38401345f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f3838000f40 tx=0x7f3838007ed0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:07.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.919+0000 7f383cff9700 1 -- 172.21.15.60:0/3918581581 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3838006aa0 con 0x7f384013fbd0 2022-06-29T15:59:07.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.919+0000 7f383cff9700 1 -- 172.21.15.60:0/3918581581 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3838006c00 con 0x7f384013fbd0 2022-06-29T15:59:07.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.919+0000 7f383cff9700 1 -- 172.21.15.60:0/3918581581 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3838007030 con 0x7f384013fbd0 2022-06-29T15:59:07.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.919+0000 7f384599e700 1 -- 172.21.15.60:0/3918581581 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f383000b180 con 0x7f384013fbd0 2022-06-29T15:59:07.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.919+0000 7f384599e700 1 -- 172.21.15.60:0/3918581581 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f384013c840 con 0x7f384013fbd0 2022-06-29T15:59:07.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.919+0000 7f384599e700 1 -- 172.21.15.60:0/3918581581 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f384013f050 con 0x7f384013fbd0 2022-06-29T15:59:07.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.923+0000 7f383cff9700 1 -- 172.21.15.60:0/3918581581 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3838010030 con 0x7f384013fbd0 2022-06-29T15:59:07.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.923+0000 7f383cff9700 1 --2- 172.21.15.60:0/3918581581 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3818053b00 0x7f3818055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:07.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.923+0000 7f383cff9700 1 -- 172.21.15.60:0/3918581581 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f383806bea0 con 0x7f384013fbd0 2022-06-29T15:59:07.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.923+0000 7f383effd700 1 --2- 172.21.15.60:0/3918581581 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3818053b00 0x7f3818055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:07.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.923+0000 7f383effd700 1 --2- 172.21.15.60:0/3918581581 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3818053b00 0x7f3818055fc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f383001a000 tx=0x7f3830014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:07.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:07.927+0000 7f383cff9700 1 -- 172.21.15.60:0/3918581581 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3838042cb0 con 0x7f384013fbd0 2022-06-29T15:59:08.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.091+0000 7f384599e700 1 -- 172.21.15.60:0/3918581581 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3840133df0 con 0x7f384013fbd0 2022-06-29T15:59:08.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.091+0000 7f383cff9700 1 -- 172.21.15.60:0/3918581581 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+2134 (secure 0 0 0) 0x7f38380462d0 con 0x7f384013fbd0 2022-06-29T15:59:08.102 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:59:08.103 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":9,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4884,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2555925680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2555925680},{"type":"v1","addr":"172.21.15.60:6833","nonce":2555925680}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":7}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T15:59:05.257113+0000","modified":"2022-06-29T15:59:06.259879+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4933},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4933":{"gid":4933,"name":"b","rank":0,"incarnation":8,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/1993861928","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1993861928},{"type":"v1","addr":"172.21.15.92:6835","nonce":1993861928}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-06-29T15:59:08.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.095+0000 7f384599e700 1 -- 172.21.15.60:0/3918581581 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3818053b00 msgr2=0x7f3818055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:08.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.095+0000 7f384599e700 1 --2- 172.21.15.60:0/3918581581 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3818053b00 0x7f3818055fc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f383001a000 tx=0x7f3830014000).stop 2022-06-29T15:59:08.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.095+0000 7f384599e700 1 -- 172.21.15.60:0/3918581581 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f384013fbd0 msgr2=0x7f38401345f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:08.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.095+0000 7f384599e700 1 --2- 172.21.15.60:0/3918581581 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f384013fbd0 0x7f38401345f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f3838000f40 tx=0x7f3838007ed0).stop 2022-06-29T15:59:08.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.095+0000 7f384599e700 1 -- 172.21.15.60:0/3918581581 shutdown_connections 2022-06-29T15:59:08.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.095+0000 7f384599e700 1 --2- 172.21.15.60:0/3918581581 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3818053b00 0x7f3818055fc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:08.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.095+0000 7f384599e700 1 --2- 172.21.15.60:0/3918581581 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f384013fbd0 0x7f38401345f0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:08.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.095+0000 7f384599e700 1 --2- 172.21.15.60:0/3918581581 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f384013f790 0x7f3840132550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:08.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.095+0000 7f384599e700 1 --2- 172.21.15.60:0/3918581581 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38400b61d0 0x7f38401319a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:08.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.095+0000 7f384599e700 1 -- 172.21.15.60:0/3918581581 >> 172.21.15.60:0/3918581581 conn(0x7f3840063b50 msgr2=0x7f38400b4930 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:08.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.095+0000 7f384599e700 1 -- 172.21.15.60:0/3918581581 shutdown_connections 2022-06-29T15:59:08.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.095+0000 7f384599e700 1 -- 172.21.15.60:0/3918581581 wait complete. 2022-06-29T15:59:08.108 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 9 2022-06-29T15:59:08.317 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T15:59:05.257113+0000', 'modified': '2022-06-29T15:59:06.259879+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4933}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4933': {'gid': 4933, 'name': 'b', 'rank': 0, 'incarnation': 8, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.92:6835/1993861928', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 1993861928}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 1993861928}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T15:59:08.318 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T15:59:08.319 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-06-29T15:59:08.436 INFO:teuthology.orchestra.run.smithi092.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":4933,"want_state":"up:active","state":"up:active","rank_uptime":3.1592838900000002,"mdsmap_epoch":9,"osdmap_epoch":32,"osdmap_epoch_barrier":31,"uptime":9.3553215630000004} 2022-06-29T15:59:08.437 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 4933, "mdsmap_epoch": 9, "osdmap_epoch": 32, "osdmap_epoch_barrier": 31, "rank_uptime": 3.15928389, "state": "up:active", "uptime": 9.355321563, "want_state": "up:active", "whoami": 0 } 2022-06-29T15:59:08.438 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T15:59:08.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.543+0000 7f3eff59e700 1 Processor -- start 2022-06-29T15:59:08.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.543+0000 7f3eff59e700 1 -- start start 2022-06-29T15:59:08.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3eff59e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3f000b2130 0x7f3f00146100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:08.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3eff59e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f0005d450 con 0x7f3f000aed00 2022-06-29T15:59:08.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3eff59e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f000b25d0 con 0x7f3f0005d070 2022-06-29T15:59:08.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3eff59e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f000adf90 con 0x7f3f000b2130 2022-06-29T15:59:08.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3efed9d700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f3f000aed00 0x7f3f001465f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54624/0 (socket says 172.21.15.60:54624) 2022-06-29T15:59:08.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3efed9d700 1 -- 172.21.15.60:0/1584865421 learned_addr learned my addr 172.21.15.60:0/1584865421 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:59:08.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3efdd9b700 1 --2- 172.21.15.60:0/1584865421 >> v2:172.21.15.92:3300/0 conn(0x7f3f000b2130 0x7f3f00146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:08.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3efd59a700 1 -- 172.21.15.60:0/1584865421 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2524766312 0 0) 0x7f3f0005d450 con 0x7f3f000aed00 2022-06-29T15:59:08.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3efd59a700 1 -- 172.21.15.60:0/1584865421 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ee0003260 con 0x7f3f000aed00 2022-06-29T15:59:08.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3efd59a700 1 -- 172.21.15.60:0/1584865421 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1827567573 0 0) 0x7f3ee0003260 con 0x7f3f000aed00 2022-06-29T15:59:08.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3efd59a700 1 -- 172.21.15.60:0/1584865421 >> v2:172.21.15.92:3300/0 conn(0x7f3f000b2130 msgr2=0x7f3f00146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:08.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3efd59a700 1 --2- 172.21.15.60:0/1584865421 >> v2:172.21.15.92:3300/0 conn(0x7f3f000b2130 0x7f3f00146100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:08.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3efd59a700 1 -- 172.21.15.60:0/1584865421 >> v1:172.21.15.92:6789/0 conn(0x7f3f0005d070 legacy=0x7f3f0014aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:08.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3efd59a700 1 -- 172.21.15.60:0/1584865421 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f0014a350 con 0x7f3f000aed00 2022-06-29T15:59:08.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3efd59a700 1 -- 172.21.15.60:0/1584865421 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f3ef80019e0 con 0x7f3f000aed00 2022-06-29T15:59:08.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3efdd9b700 1 --2- 172.21.15.60:0/1584865421 >> v2:172.21.15.92:3300/0 conn(0x7f3f000b2130 0x7f3f00146100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:59:08.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3efd59a700 1 -- 172.21.15.60:0/1584865421 >> v1:172.21.15.60:6789/0 conn(0x7f3f000aed00 legacy=0x7f3f001465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:08.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3efd59a700 1 --2- 172.21.15.60:0/1584865421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee0003750 0x7f3ee0003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:08.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3efd59a700 1 --2- 172.21.15.60:0/1584865421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ee0004500 0x7f3ee0004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:08.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3efd59a700 1 --2- 172.21.15.60:0/1584865421 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ee0004e70 0x7f3ee000b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:08.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3efd59a700 1 -- 172.21.15.60:0/1584865421 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f000adf90 con 0x7f3ee0003750 2022-06-29T15:59:08.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3efd59a700 1 -- 172.21.15.60:0/1584865421 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee000bd20 con 0x7f3ee0004500 2022-06-29T15:59:08.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3efd59a700 1 -- 172.21.15.60:0/1584865421 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee000bee0 con 0x7f3ee0004e70 2022-06-29T15:59:08.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3efdd9b700 1 --2- 172.21.15.60:0/1584865421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ee0004500 0x7f3ee0004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:08.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3efe59c700 1 --2- 172.21.15.60:0/1584865421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee0003750 0x7f3ee0003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:08.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3efed9d700 1 --2- 172.21.15.60:0/1584865421 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ee0004e70 0x7f3ee000b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:08.586 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3efdd9b700 1 -- 172.21.15.60:0/1584865421 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ee0004e70 msgr2=0x7f3ee000b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:08.586 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3efdd9b700 1 --2- 172.21.15.60:0/1584865421 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ee0004e70 0x7f3ee000b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:08.587 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3efdd9b700 1 -- 172.21.15.60:0/1584865421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee0003750 msgr2=0x7f3ee0003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:08.587 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3efdd9b700 1 --2- 172.21.15.60:0/1584865421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee0003750 0x7f3ee0003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:08.587 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3efdd9b700 1 -- 172.21.15.60:0/1584865421 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee0003260 con 0x7f3ee0004500 2022-06-29T15:59:08.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3efed9d700 1 -- 172.21.15.60:0/1584865421 reap_dead start 2022-06-29T15:59:08.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3efdd9b700 1 --2- 172.21.15.60:0/1584865421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ee0004500 0x7f3ee0004980 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f3ef000c000 tx=0x7f3ef0006df0).ready entity=mon.1 client_cookie=6b076581b91651f6 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:08.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3efd59a700 1 -- 172.21.15.60:0/1584865421 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ef000e830 con 0x7f3ee0004500 2022-06-29T15:59:08.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3efd59a700 1 -- 172.21.15.60:0/1584865421 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ef000e990 con 0x7f3ee0004500 2022-06-29T15:59:08.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3efd59a700 1 -- 172.21.15.60:0/1584865421 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ef000ecf0 con 0x7f3ee0004500 2022-06-29T15:59:08.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3eff59e700 1 -- 172.21.15.60:0/1584865421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ee0004500 msgr2=0x7f3ee0004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:08.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3eff59e700 1 --2- 172.21.15.60:0/1584865421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ee0004500 0x7f3ee0004980 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f3ef000c000 tx=0x7f3ef0006df0).stop 2022-06-29T15:59:08.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3eff59e700 1 -- 172.21.15.60:0/1584865421 shutdown_connections 2022-06-29T15:59:08.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3eff59e700 1 --2- 172.21.15.60:0/1584865421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ee0004500 0x7f3ee0004980 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:08.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3eff59e700 1 -- 172.21.15.60:0/1584865421 >> 172.21.15.60:0/1584865421 conn(0x7f3f000b5890 msgr2=0x7f3f000b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:08.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3eff59e700 1 -- 172.21.15.60:0/1584865421 shutdown_connections 2022-06-29T15:59:08.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.547+0000 7f3eff59e700 1 -- 172.21.15.60:0/1584865421 wait complete. 2022-06-29T15:59:08.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.551+0000 7f3eff59e700 1 Processor -- start 2022-06-29T15:59:08.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.551+0000 7f3eff59e700 1 -- start start 2022-06-29T15:59:08.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.551+0000 7f3eff59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee0004500 0x7f3f0014cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:08.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.551+0000 7f3eff59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f0014bca0 0x7f3f00064570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:08.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.551+0000 7f3eff59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f00064a60 0x7f3f00159590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:08.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.551+0000 7f3eff59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0014e510 con 0x7f3ee0004500 2022-06-29T15:59:08.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.551+0000 7f3eff59e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f00063340 con 0x7f3f0014bca0 2022-06-29T15:59:08.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.551+0000 7f3eff59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f000630a0 con 0x7f3f00064a60 2022-06-29T15:59:08.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.551+0000 7f3efe59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee0004500 0x7f3f0014cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:08.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.551+0000 7f3efe59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee0004500 0x7f3f0014cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46546/0 (socket says 172.21.15.60:46546) 2022-06-29T15:59:08.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.551+0000 7f3efe59c700 1 -- 172.21.15.60:0/1056556526 learned_addr learned my addr 172.21.15.60:0/1056556526 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:08.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.551+0000 7f3efed9d700 1 --2- 172.21.15.60:0/1056556526 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f00064a60 0x7f3f00159590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:08.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.551+0000 7f3efdd9b700 1 --2- 172.21.15.60:0/1056556526 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f0014bca0 0x7f3f00064570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:08.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.551+0000 7f3efed9d700 1 -- 172.21.15.60:0/1056556526 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f0014bca0 msgr2=0x7f3f00064570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:08.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.551+0000 7f3efed9d700 1 --2- 172.21.15.60:0/1056556526 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f0014bca0 0x7f3f00064570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:08.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.551+0000 7f3efed9d700 1 -- 172.21.15.60:0/1056556526 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee0004500 msgr2=0x7f3f0014cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:08.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.551+0000 7f3efed9d700 1 --2- 172.21.15.60:0/1056556526 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee0004500 0x7f3f0014cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:08.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.551+0000 7f3efed9d700 1 -- 172.21.15.60:0/1056556526 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f00063f60 con 0x7f3f00064a60 2022-06-29T15:59:08.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.551+0000 7f3efe59c700 1 --2- 172.21.15.60:0/1056556526 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee0004500 0x7f3f0014cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:59:08.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.551+0000 7f3efed9d700 1 --2- 172.21.15.60:0/1056556526 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f00064a60 0x7f3f00159590 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f3f00065730 tx=0x7f3ef80021b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:08.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.551+0000 7f3ef77fe700 1 -- 172.21.15.60:0/1056556526 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ef8001f10 con 0x7f3f00064a60 2022-06-29T15:59:08.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.551+0000 7f3ef77fe700 1 -- 172.21.15.60:0/1056556526 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ef8011780 con 0x7f3f00064a60 2022-06-29T15:59:08.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.551+0000 7f3eff59e700 1 -- 172.21.15.60:0/1056556526 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f00064f10 con 0x7f3f00064a60 2022-06-29T15:59:08.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.555+0000 7f3ef77fe700 1 -- 172.21.15.60:0/1056556526 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ef8011b00 con 0x7f3f00064a60 2022-06-29T15:59:08.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.555+0000 7f3eff59e700 1 -- 172.21.15.60:0/1056556526 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f000635b0 con 0x7f3f00064a60 2022-06-29T15:59:08.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.555+0000 7f3eff59e700 1 -- 172.21.15.60:0/1056556526 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f0014a5f0 con 0x7f3f00064a60 2022-06-29T15:59:08.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.555+0000 7f3ef77fe700 1 -- 172.21.15.60:0/1056556526 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3ef8011c60 con 0x7f3f00064a60 2022-06-29T15:59:08.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.555+0000 7f3ef77fe700 1 --2- 172.21.15.60:0/1056556526 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3ee4053980 0x7f3ee4055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:08.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.555+0000 7f3ef77fe700 1 -- 172.21.15.60:0/1056556526 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3ef806b920 con 0x7f3f00064a60 2022-06-29T15:59:08.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.555+0000 7f3efe59c700 1 --2- 172.21.15.60:0/1056556526 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3ee4053980 0x7f3ee4055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:08.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.559+0000 7f3efe59c700 1 --2- 172.21.15.60:0/1056556526 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3ee4053980 0x7f3ee4055e40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f3ee8003000 tx=0x7f3ee80079e0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:08.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.567+0000 7f3ef77fe700 1 -- 172.21.15.60:0/1056556526 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3ef80427a0 con 0x7f3f00064a60 2022-06-29T15:59:08.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.695+0000 7f3eff59e700 1 -- 172.21.15.60:0/1056556526 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f0014c080 con 0x7f3f00064a60 2022-06-29T15:59:08.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.699+0000 7f3ef77fe700 1 -- 172.21.15.60:0/1056556526 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16038 (secure 0 0 0) 0x7f3ef8019030 con 0x7f3f00064a60 2022-06-29T15:59:08.709 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:59:08.709 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":32,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T15:59:06.260590+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2022-06-29T15:59:03.821123+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2022-06-29T15:59:04.275915+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T15:59:08.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.703+0000 7f3eff59e700 1 -- 172.21.15.60:0/1056556526 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3ee4053980 msgr2=0x7f3ee4055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:08.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.703+0000 7f3eff59e700 1 --2- 172.21.15.60:0/1056556526 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3ee4053980 0x7f3ee4055e40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f3ee8003000 tx=0x7f3ee80079e0).stop 2022-06-29T15:59:08.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.703+0000 7f3eff59e700 1 -- 172.21.15.60:0/1056556526 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f00064a60 msgr2=0x7f3f00159590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:08.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.703+0000 7f3eff59e700 1 --2- 172.21.15.60:0/1056556526 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f00064a60 0x7f3f00159590 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f3f00065730 tx=0x7f3ef80021b0).stop 2022-06-29T15:59:08.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.703+0000 7f3eff59e700 1 -- 172.21.15.60:0/1056556526 shutdown_connections 2022-06-29T15:59:08.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.703+0000 7f3eff59e700 1 --2- 172.21.15.60:0/1056556526 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3ee4053980 0x7f3ee4055e40 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:08.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.703+0000 7f3eff59e700 1 --2- 172.21.15.60:0/1056556526 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f00064a60 0x7f3f00159590 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:08.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.703+0000 7f3eff59e700 1 --2- 172.21.15.60:0/1056556526 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f0014bca0 0x7f3f00064570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:08.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.703+0000 7f3eff59e700 1 --2- 172.21.15.60:0/1056556526 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ee0004500 0x7f3f0014cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:08.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.703+0000 7f3eff59e700 1 -- 172.21.15.60:0/1056556526 >> 172.21.15.60:0/1056556526 conn(0x7f3f000b5890 msgr2=0x7f3f000af720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:08.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.703+0000 7f3eff59e700 1 -- 172.21.15.60:0/1056556526 shutdown_connections 2022-06-29T15:59:08.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.703+0000 7f3eff59e700 1 -- 172.21.15.60:0/1056556526 wait complete. 2022-06-29T15:59:08.726 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-29T15:59:03.821123+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '31', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2022-06-29T15:59:04.275915+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-29T15:59:08.727 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T15:59:08.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.827+0000 7f26c9159700 1 Processor -- start 2022-06-29T15:59:08.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.827+0000 7f26c9159700 1 -- start start 2022-06-29T15:59:08.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.831+0000 7f26c9159700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f26c40b2800 0x7f26c40b2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:08.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.831+0000 7f26c9159700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f26c413f570 0x7f26c40afff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:08.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.831+0000 7f26c9159700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c40af430 con 0x7f26c40b1450 2022-06-29T15:59:08.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.831+0000 7f26c9159700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26c40afb00 con 0x7f26c413f570 2022-06-29T15:59:08.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.831+0000 7f26c9159700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26c413fca0 con 0x7f26c40b2800 2022-06-29T15:59:08.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.831+0000 7f26bb7fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f26c40b1450 0x7f26c413ee60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54642/0 (socket says 172.21.15.60:54642) 2022-06-29T15:59:08.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.831+0000 7f26bb7fe700 1 -- 172.21.15.60:0/1544576917 learned_addr learned my addr 172.21.15.60:0/1544576917 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:59:08.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.831+0000 7f26c8958700 1 --2- 172.21.15.60:0/1544576917 >> v2:172.21.15.60:3300/0 conn(0x7f26c413f570 0x7f26c40afff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:08.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.831+0000 7f26c3fff700 1 --2- 172.21.15.60:0/1544576917 >> v2:172.21.15.92:3300/0 conn(0x7f26c40b2800 0x7f26c40b2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:08.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.831+0000 7f26c37fe700 1 -- 172.21.15.60:0/1544576917 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 731078384 0 0) 0x7f26c40af430 con 0x7f26c40b1450 2022-06-29T15:59:08.843 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.831+0000 7f26c37fe700 1 -- 172.21.15.60:0/1544576917 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26a4003260 con 0x7f26c40b1450 2022-06-29T15:59:08.843 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.831+0000 7f26c3fff700 1 -- 172.21.15.60:0/1544576917 >> v2:172.21.15.60:3300/0 conn(0x7f26c413f570 msgr2=0x7f26c40afff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:08.843 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.831+0000 7f26c3fff700 1 --2- 172.21.15.60:0/1544576917 >> v2:172.21.15.60:3300/0 conn(0x7f26c413f570 0x7f26c40afff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:08.843 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.831+0000 7f26c37fe700 1 -- 172.21.15.60:0/1544576917 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2072759750 0 0) 0x7f26a4003260 con 0x7f26c40b1450 2022-06-29T15:59:08.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.831+0000 7f26c3fff700 1 -- 172.21.15.60:0/1544576917 >> v1:172.21.15.60:6789/0 conn(0x7f26c40b1450 legacy=0x7f26c413ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:08.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.831+0000 7f26c3fff700 1 -- 172.21.15.60:0/1544576917 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c40af660 con 0x7f26c40b2800 2022-06-29T15:59:08.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.831+0000 7f26c3fff700 1 --2- 172.21.15.60:0/1544576917 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f26c40b2800 0x7f26c40b2be0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f26ac001800 tx=0x7f26b4006df0).ready entity=mon.1 client_cookie=17479fa8e167fb62 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:08.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.831+0000 7f26c37fe700 1 -- 172.21.15.60:0/1544576917 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f26ac001660 con 0x7f26c40b2800 2022-06-29T15:59:08.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.831+0000 7f26c37fe700 1 -- 172.21.15.60:0/1544576917 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26b4000e90 con 0x7f26c40b2800 2022-06-29T15:59:08.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.831+0000 7f26c37fe700 1 -- 172.21.15.60:0/1544576917 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f26b4012460 con 0x7f26c40b2800 2022-06-29T15:59:08.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.831+0000 7f26c9159700 1 -- 172.21.15.60:0/1544576917 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f26c40b2800 msgr2=0x7f26c40b2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:08.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.831+0000 7f26c9159700 1 --2- 172.21.15.60:0/1544576917 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f26c40b2800 0x7f26c40b2be0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f26ac001800 tx=0x7f26b4006df0).stop 2022-06-29T15:59:08.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.831+0000 7f26c9159700 1 -- 172.21.15.60:0/1544576917 shutdown_connections 2022-06-29T15:59:08.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.831+0000 7f26c9159700 1 --2- 172.21.15.60:0/1544576917 >> v2:172.21.15.60:3300/0 conn(0x7f26c413f570 0x7f26c40afff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:08.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.831+0000 7f26c9159700 1 --2- 172.21.15.60:0/1544576917 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f26c40b2800 0x7f26c40b2be0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:08.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.831+0000 7f26c9159700 1 -- 172.21.15.60:0/1544576917 >> 172.21.15.60:0/1544576917 conn(0x7f26c40b6120 msgr2=0x7f26c40b20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:08.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.831+0000 7f26c9159700 1 -- 172.21.15.60:0/1544576917 shutdown_connections 2022-06-29T15:59:08.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.831+0000 7f26c9159700 1 -- 172.21.15.60:0/1544576917 wait complete. 2022-06-29T15:59:08.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.835+0000 7f26c9159700 1 Processor -- start 2022-06-29T15:59:08.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.835+0000 7f26c9159700 1 -- start start 2022-06-29T15:59:08.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.839+0000 7f26c9159700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26c40b1450 0x7f26c40667c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:08.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.839+0000 7f26c9159700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f26c40b2800 0x7f26c4142300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:08.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.839+0000 7f26c9159700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f26c413f570 0x7f26c41427f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:08.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.839+0000 7f26c9159700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c409dfc0 con 0x7f26c40b1450 2022-06-29T15:59:08.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.839+0000 7f26c9159700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c409e490 con 0x7f26c413f570 2022-06-29T15:59:08.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.839+0000 7f26c9159700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26c409e1f0 con 0x7f26c40b2800 2022-06-29T15:59:08.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.839+0000 7f26c3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26c40b1450 0x7f26c40667c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:08.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.839+0000 7f26c3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26c40b1450 0x7f26c40667c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46560/0 (socket says 172.21.15.60:46560) 2022-06-29T15:59:08.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.839+0000 7f26c3fff700 1 -- 172.21.15.60:0/1006617668 learned_addr learned my addr 172.21.15.60:0/1006617668 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:08.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.839+0000 7f26c8958700 1 --2- 172.21.15.60:0/1006617668 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f26c413f570 0x7f26c41427f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:08.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.839+0000 7f26bb7fe700 1 --2- 172.21.15.60:0/1006617668 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f26c40b2800 0x7f26c4142300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:08.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.839+0000 7f26c3fff700 1 -- 172.21.15.60:0/1006617668 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f26c40b2800 msgr2=0x7f26c4142300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:08.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.839+0000 7f26c3fff700 1 --2- 172.21.15.60:0/1006617668 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f26c40b2800 0x7f26c4142300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:08.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.839+0000 7f26c3fff700 1 -- 172.21.15.60:0/1006617668 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f26c413f570 msgr2=0x7f26c41427f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:08.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.839+0000 7f26c3fff700 1 --2- 172.21.15.60:0/1006617668 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f26c413f570 0x7f26c41427f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:08.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.839+0000 7f26c3fff700 1 -- 172.21.15.60:0/1006617668 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b400b180 con 0x7f26c40b1450 2022-06-29T15:59:08.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.839+0000 7f26c3fff700 1 --2- 172.21.15.60:0/1006617668 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26c40b1450 0x7f26c40667c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f26ac001800 tx=0x7f26b4014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:08.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.839+0000 7f26c1ffb700 1 -- 172.21.15.60:0/1006617668 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f26b4016030 con 0x7f26c40b1450 2022-06-29T15:59:08.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.839+0000 7f26c1ffb700 1 -- 172.21.15.60:0/1006617668 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26b400dd40 con 0x7f26c40b1450 2022-06-29T15:59:08.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.839+0000 7f26c9159700 1 -- 172.21.15.60:0/1006617668 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26c4142ce0 con 0x7f26c40b1450 2022-06-29T15:59:08.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.839+0000 7f26c1ffb700 1 -- 172.21.15.60:0/1006617668 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f26b4012460 con 0x7f26c40b1450 2022-06-29T15:59:08.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.839+0000 7f26c9159700 1 -- 172.21.15.60:0/1006617668 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26c40abda0 con 0x7f26c40b1450 2022-06-29T15:59:08.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.843+0000 7f26c9159700 1 -- 172.21.15.60:0/1006617668 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26c40ac0b0 con 0x7f26c40b1450 2022-06-29T15:59:08.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.843+0000 7f26c1ffb700 1 -- 172.21.15.60:0/1006617668 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f26b4024030 con 0x7f26c40b1450 2022-06-29T15:59:08.859 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.843+0000 7f26c1ffb700 1 --2- 172.21.15.60:0/1006617668 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f26a8053950 0x7f26a8055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:08.859 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.843+0000 7f26c1ffb700 1 -- 172.21.15.60:0/1006617668 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f26b4017070 con 0x7f26c40b1450 2022-06-29T15:59:08.859 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.843+0000 7f26bb7fe700 1 --2- 172.21.15.60:0/1006617668 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f26a8053950 0x7f26a8055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:08.859 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.843+0000 7f26bb7fe700 1 --2- 172.21.15.60:0/1006617668 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f26a8053950 0x7f26a8055e10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f26ac00d000 tx=0x7f26ac001de0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:08.860 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.847+0000 7f26c1ffb700 1 -- 172.21.15.60:0/1006617668 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f26b4044da0 con 0x7f26c40b1450 2022-06-29T15:59:08.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.971+0000 7f26c9159700 1 -- 172.21.15.60:0/1006617668 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f26c40b2ce0 con 0x7f26c40b1450 2022-06-29T15:59:08.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.971+0000 7f26c1ffb700 1 -- 172.21.15.60:0/1006617668 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v32) v1 ==== 110+0+10 (secure 0 0 0) 0x7f26b4022020 con 0x7f26c40b1450 2022-06-29T15:59:08.981 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T15:59:08.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.971+0000 7f26c9159700 1 -- 172.21.15.60:0/1006617668 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f26a8053950 msgr2=0x7f26a8055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:08.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.971+0000 7f26c9159700 1 --2- 172.21.15.60:0/1006617668 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f26a8053950 0x7f26a8055e10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f26ac00d000 tx=0x7f26ac001de0).stop 2022-06-29T15:59:08.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.971+0000 7f26c9159700 1 -- 172.21.15.60:0/1006617668 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26c40b1450 msgr2=0x7f26c40667c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:08.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.971+0000 7f26c9159700 1 --2- 172.21.15.60:0/1006617668 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26c40b1450 0x7f26c40667c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f26ac001800 tx=0x7f26b4014000).stop 2022-06-29T15:59:08.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.971+0000 7f26c9159700 1 -- 172.21.15.60:0/1006617668 shutdown_connections 2022-06-29T15:59:08.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.971+0000 7f26c9159700 1 --2- 172.21.15.60:0/1006617668 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f26a8053950 0x7f26a8055e10 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:08.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.971+0000 7f26c9159700 1 --2- 172.21.15.60:0/1006617668 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f26c413f570 0x7f26c41427f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:08.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.971+0000 7f26c9159700 1 --2- 172.21.15.60:0/1006617668 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f26c40b2800 0x7f26c4142300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:08.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.971+0000 7f26c9159700 1 --2- 172.21.15.60:0/1006617668 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26c40b1450 0x7f26c40667c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:08.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.971+0000 7f26c9159700 1 -- 172.21.15.60:0/1006617668 >> 172.21.15.60:0/1006617668 conn(0x7f26c40b6120 msgr2=0x7f26c40afff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:08.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.971+0000 7f26c9159700 1 -- 172.21.15.60:0/1006617668 shutdown_connections 2022-06-29T15:59:08.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:08.971+0000 7f26c9159700 1 -- 172.21.15.60:0/1006617668 wait complete. 2022-06-29T15:59:08.993 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T15:59:09.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.091+0000 7fbb349b2700 1 Processor -- start 2022-06-29T15:59:09.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.091+0000 7fbb349b2700 1 -- start start 2022-06-29T15:59:09.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.091+0000 7fbb349b2700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbb3005d030 0x7fbb300b3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:09.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.091+0000 7fbb349b2700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbb3014a2b0 0x7fbb301461d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:09.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.091+0000 7fbb349b2700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb30065820 con 0x7fbb3005bf30 2022-06-29T15:59:09.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.091+0000 7fbb349b2700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb300b3df0 con 0x7fbb3005d030 2022-06-29T15:59:09.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.091+0000 7fbb349b2700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb30066600 con 0x7fbb3014a2b0 2022-06-29T15:59:09.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.091+0000 7fbb2f59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbb3014a2b0 0x7fbb301461d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:09.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.091+0000 7fbb2ed9d700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbb3005d030 0x7fbb300b3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:09.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.091+0000 7fbb27fff700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fbb3005bf30 0x7fbb30065ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:58904/0 (socket says 172.21.15.60:58904) 2022-06-29T15:59:09.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.091+0000 7fbb27fff700 1 -- 172.21.15.60:0/118212075 learned_addr learned my addr 172.21.15.60:0/118212075 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:59:09.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.091+0000 7fbb2ed9d700 1 -- 172.21.15.60:0/118212075 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbb3014a2b0 msgr2=0x7fbb301461d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:09.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.091+0000 7fbb2ed9d700 1 --2- 172.21.15.60:0/118212075 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbb3014a2b0 0x7fbb301461d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:09.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.091+0000 7fbb2ed9d700 1 -- 172.21.15.60:0/118212075 >> v1:172.21.15.92:6789/0 conn(0x7fbb3005bf30 legacy=0x7fbb30065ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:09.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.091+0000 7fbb2ed9d700 1 -- 172.21.15.60:0/118212075 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb3005d410 con 0x7fbb3005d030 2022-06-29T15:59:09.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.095+0000 7fbb2ed9d700 1 --2- 172.21.15.60:0/118212075 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb3005d030 0x7fbb300b3900 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fbb20002f70 tx=0x7fbb20006df0).ready entity=mon.1 client_cookie=ec53df0ebb66aa3d server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:09.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.095+0000 7fbb2e59c700 1 -- 172.21.15.60:0/118212075 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb2000d570 con 0x7fbb3005d030 2022-06-29T15:59:09.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.095+0000 7fbb2e59c700 1 -- 172.21.15.60:0/118212075 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb20000e90 con 0x7fbb3005d030 2022-06-29T15:59:09.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.095+0000 7fbb2e59c700 1 -- 172.21.15.60:0/118212075 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb2000d8c0 con 0x7fbb3005d030 2022-06-29T15:59:09.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.095+0000 7fbb349b2700 1 -- 172.21.15.60:0/118212075 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb3005d030 msgr2=0x7fbb300b3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:09.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.095+0000 7fbb349b2700 1 --2- 172.21.15.60:0/118212075 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb3005d030 0x7fbb300b3900 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fbb20002f70 tx=0x7fbb20006df0).stop 2022-06-29T15:59:09.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.095+0000 7fbb349b2700 1 -- 172.21.15.60:0/118212075 shutdown_connections 2022-06-29T15:59:09.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.095+0000 7fbb349b2700 1 --2- 172.21.15.60:0/118212075 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbb3014a2b0 0x7fbb301461d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:09.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.095+0000 7fbb349b2700 1 --2- 172.21.15.60:0/118212075 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb3005d030 0x7fbb300b3900 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:09.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.095+0000 7fbb349b2700 1 -- 172.21.15.60:0/118212075 >> 172.21.15.60:0/118212075 conn(0x7fbb30064810 msgr2=0x7fbb30064bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:09.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.095+0000 7fbb349b2700 1 -- 172.21.15.60:0/118212075 shutdown_connections 2022-06-29T15:59:09.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.095+0000 7fbb349b2700 1 -- 172.21.15.60:0/118212075 wait complete. 2022-06-29T15:59:09.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.099+0000 7fbb349b2700 1 Processor -- start 2022-06-29T15:59:09.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.099+0000 7fbb349b2700 1 -- start start 2022-06-29T15:59:09.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.099+0000 7fbb349b2700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb3005bf30 0x7fbb30112770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:09.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.099+0000 7fbb349b2700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbb3005d030 0x7fbb30155d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:09.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.099+0000 7fbb349b2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb3014a2b0 0x7fbb30156270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:09.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.099+0000 7fbb349b2700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb30112e30 con 0x7fbb3014a2b0 2022-06-29T15:59:09.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.099+0000 7fbb349b2700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb301130d0 con 0x7fbb3005bf30 2022-06-29T15:59:09.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.099+0000 7fbb349b2700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb301119b0 con 0x7fbb3005d030 2022-06-29T15:59:09.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.099+0000 7fbb2f59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb3014a2b0 0x7fbb30156270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:09.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.099+0000 7fbb2f59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb3014a2b0 0x7fbb30156270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46578/0 (socket says 172.21.15.60:46578) 2022-06-29T15:59:09.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.099+0000 7fbb2f59e700 1 -- 172.21.15.60:0/2614068569 learned_addr learned my addr 172.21.15.60:0/2614068569 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:09.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.099+0000 7fbb27fff700 1 --2- 172.21.15.60:0/2614068569 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbb3005d030 0x7fbb30155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:09.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.099+0000 7fbb2ed9d700 1 --2- 172.21.15.60:0/2614068569 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb3005bf30 0x7fbb30112770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:09.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.099+0000 7fbb2f59e700 1 -- 172.21.15.60:0/2614068569 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbb3005d030 msgr2=0x7fbb30155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:09.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.099+0000 7fbb2f59e700 1 --2- 172.21.15.60:0/2614068569 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbb3005d030 0x7fbb30155d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:09.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.099+0000 7fbb2f59e700 1 -- 172.21.15.60:0/2614068569 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb3005bf30 msgr2=0x7fbb30112770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:09.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.099+0000 7fbb2f59e700 1 --2- 172.21.15.60:0/2614068569 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb3005bf30 0x7fbb30112770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:09.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.099+0000 7fbb2f59e700 1 -- 172.21.15.60:0/2614068569 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb2000b180 con 0x7fbb3014a2b0 2022-06-29T15:59:09.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.099+0000 7fbb2f59e700 1 --2- 172.21.15.60:0/2614068569 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb3014a2b0 0x7fbb30156270 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fbb28006e00 tx=0x7fbb280091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:09.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.099+0000 7fbb2cd99700 1 -- 172.21.15.60:0/2614068569 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb2800c670 con 0x7fbb3014a2b0 2022-06-29T15:59:09.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.099+0000 7fbb2cd99700 1 -- 172.21.15.60:0/2614068569 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb2800c7d0 con 0x7fbb3014a2b0 2022-06-29T15:59:09.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.099+0000 7fbb349b2700 1 -- 172.21.15.60:0/2614068569 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb30111e50 con 0x7fbb3014a2b0 2022-06-29T15:59:09.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.099+0000 7fbb2cd99700 1 -- 172.21.15.60:0/2614068569 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb2800cb30 con 0x7fbb3014a2b0 2022-06-29T15:59:09.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.099+0000 7fbb349b2700 1 -- 172.21.15.60:0/2614068569 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb30156980 con 0x7fbb3014a2b0 2022-06-29T15:59:09.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.103+0000 7fbb349b2700 1 -- 172.21.15.60:0/2614068569 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb301112d0 con 0x7fbb3014a2b0 2022-06-29T15:59:09.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.103+0000 7fbb2cd99700 1 -- 172.21.15.60:0/2614068569 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fbb2800cc90 con 0x7fbb3014a2b0 2022-06-29T15:59:09.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.103+0000 7fbb2cd99700 1 --2- 172.21.15.60:0/2614068569 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbb14053b00 0x7fbb14055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:09.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.103+0000 7fbb2cd99700 1 -- 172.21.15.60:0/2614068569 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbb2806c2d0 con 0x7fbb3014a2b0 2022-06-29T15:59:09.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.103+0000 7fbb2ed9d700 1 --2- 172.21.15.60:0/2614068569 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbb14053b00 0x7fbb14055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:09.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.103+0000 7fbb2ed9d700 1 --2- 172.21.15.60:0/2614068569 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbb14053b00 0x7fbb14055fc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fbb20018000 tx=0x7fbb20014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:09.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.111+0000 7fbb2cd99700 1 -- 172.21.15.60:0/2614068569 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbb301112d0 con 0x7fbb3014a2b0 2022-06-29T15:59:09.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.235+0000 7fbb349b2700 1 -- 172.21.15.60:0/2614068569 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fbb301112d0 con 0x7fbb3014a2b0 2022-06-29T15:59:09.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.239+0000 7fbb2cd99700 1 -- 172.21.15.60:0/2614068569 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v32) v1 ==== 92+0+10 (secure 0 0 0) 0x7fbb301112d0 con 0x7fbb3014a2b0 2022-06-29T15:59:09.247 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T15:59:09.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.239+0000 7fbb349b2700 1 -- 172.21.15.60:0/2614068569 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbb14053b00 msgr2=0x7fbb14055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:09.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.239+0000 7fbb349b2700 1 --2- 172.21.15.60:0/2614068569 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbb14053b00 0x7fbb14055fc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fbb20018000 tx=0x7fbb20014000).stop 2022-06-29T15:59:09.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.239+0000 7fbb349b2700 1 -- 172.21.15.60:0/2614068569 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb3014a2b0 msgr2=0x7fbb30156270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:09.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.239+0000 7fbb349b2700 1 --2- 172.21.15.60:0/2614068569 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb3014a2b0 0x7fbb30156270 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fbb28006e00 tx=0x7fbb280091d0).stop 2022-06-29T15:59:09.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.239+0000 7fbb349b2700 1 -- 172.21.15.60:0/2614068569 shutdown_connections 2022-06-29T15:59:09.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.239+0000 7fbb349b2700 1 --2- 172.21.15.60:0/2614068569 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbb14053b00 0x7fbb14055fc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:09.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.239+0000 7fbb349b2700 1 --2- 172.21.15.60:0/2614068569 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb3014a2b0 0x7fbb30156270 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:09.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.239+0000 7fbb349b2700 1 --2- 172.21.15.60:0/2614068569 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbb3005d030 0x7fbb30155d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:09.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.239+0000 7fbb349b2700 1 --2- 172.21.15.60:0/2614068569 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbb3005bf30 0x7fbb30112770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:09.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.239+0000 7fbb349b2700 1 -- 172.21.15.60:0/2614068569 >> 172.21.15.60:0/2614068569 conn(0x7fbb30064810 msgr2=0x7fbb3014bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:09.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.239+0000 7fbb349b2700 1 -- 172.21.15.60:0/2614068569 shutdown_connections 2022-06-29T15:59:09.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.239+0000 7fbb349b2700 1 -- 172.21.15.60:0/2614068569 wait complete. 2022-06-29T15:59:09.260 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-29T15:59:09.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.359+0000 7fc332923700 1 Processor -- start 2022-06-29T15:59:09.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.359+0000 7fc332923700 1 -- start start 2022-06-29T15:59:09.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.359+0000 7fc332923700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc32c0aed00 0x7fc32c146810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:09.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.359+0000 7fc332923700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc32c05d450 con 0x7fc32c05d070 2022-06-29T15:59:09.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.359+0000 7fc332923700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc32c0b25d0 con 0x7fc32c0b2130 2022-06-29T15:59:09.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.359+0000 7fc332923700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc32c0adf90 con 0x7fc32c0aed00 2022-06-29T15:59:09.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.359+0000 7fc332122700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc32c0aed00 0x7fc32c146810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:09.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.359+0000 7fc331921700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc32c05d070 0x7fc32c14aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54670/0 (socket says 172.21.15.60:54670) 2022-06-29T15:59:09.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.359+0000 7fc331921700 1 -- 172.21.15.60:0/2684371836 learned_addr learned my addr 172.21.15.60:0/2684371836 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:59:09.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.359+0000 7fc332122700 1 -- 172.21.15.60:0/2684371836 >> v1:172.21.15.92:6789/0 conn(0x7fc32c0b2130 legacy=0x7fc32c146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:09.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.359+0000 7fc332122700 1 -- 172.21.15.60:0/2684371836 >> v1:172.21.15.60:6789/0 conn(0x7fc32c05d070 legacy=0x7fc32c14aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:09.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.359+0000 7fc332122700 1 -- 172.21.15.60:0/2684371836 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc32c14a350 con 0x7fc32c0aed00 2022-06-29T15:59:09.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.359+0000 7fc332122700 1 --2- 172.21.15.60:0/2684371836 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc32c0aed00 0x7fc32c146810 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7fc3240032f0 tx=0x7fc3240091d0).ready entity=mon.0 client_cookie=b169684c38d9cd23 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:09.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.359+0000 7fc33091f700 1 -- 172.21.15.60:0/2684371836 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc324009ea0 con 0x7fc32c0aed00 2022-06-29T15:59:09.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.363+0000 7fc33091f700 1 -- 172.21.15.60:0/2684371836 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc32400b8a0 con 0x7fc32c0aed00 2022-06-29T15:59:09.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.363+0000 7fc33091f700 1 -- 172.21.15.60:0/2684371836 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc32400ba10 con 0x7fc32c0aed00 2022-06-29T15:59:09.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.363+0000 7fc332923700 1 -- 172.21.15.60:0/2684371836 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc32c0aed00 msgr2=0x7fc32c146810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:09.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.363+0000 7fc332923700 1 --2- 172.21.15.60:0/2684371836 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc32c0aed00 0x7fc32c146810 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7fc3240032f0 tx=0x7fc3240091d0).stop 2022-06-29T15:59:09.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.363+0000 7fc332923700 1 -- 172.21.15.60:0/2684371836 shutdown_connections 2022-06-29T15:59:09.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.363+0000 7fc332923700 1 --2- 172.21.15.60:0/2684371836 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc32c0aed00 0x7fc32c146810 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:09.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.363+0000 7fc332923700 1 -- 172.21.15.60:0/2684371836 >> 172.21.15.60:0/2684371836 conn(0x7fc32c0b5890 msgr2=0x7fc32c0b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:09.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.363+0000 7fc332923700 1 -- 172.21.15.60:0/2684371836 shutdown_connections 2022-06-29T15:59:09.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.363+0000 7fc332923700 1 -- 172.21.15.60:0/2684371836 wait complete. 2022-06-29T15:59:09.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.363+0000 7fc332923700 1 Processor -- start 2022-06-29T15:59:09.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.363+0000 7fc332923700 1 -- start start 2022-06-29T15:59:09.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.363+0000 7fc332923700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc32c05d070 0x7fc32c116920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:09.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.363+0000 7fc332923700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc32c0aed00 0x7fc32c0641a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:09.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.367+0000 7fc332923700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc32c0b2130 0x7fc32c064690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:09.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.367+0000 7fc332923700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc32c157440 con 0x7fc32c0aed00 2022-06-29T15:59:09.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.367+0000 7fc332923700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc32c064b80 con 0x7fc32c0b2130 2022-06-29T15:59:09.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.367+0000 7fc332923700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc32c118fc0 con 0x7fc32c05d070 2022-06-29T15:59:09.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.367+0000 7fc331120700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc32c0aed00 0x7fc32c0641a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:09.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.367+0000 7fc331921700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc32c05d070 0x7fc32c116920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:09.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.367+0000 7fc332122700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc32c0b2130 0x7fc32c064690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:09.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.367+0000 7fc331120700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc32c0aed00 0x7fc32c0641a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46590/0 (socket says 172.21.15.60:46590) 2022-06-29T15:59:09.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.367+0000 7fc331921700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc32c05d070 0x7fc32c116920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:59682/0 (socket says 172.21.15.60:59682) 2022-06-29T15:59:09.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.367+0000 7fc331120700 1 -- 172.21.15.60:0/3217834520 learned_addr learned my addr 172.21.15.60:0/3217834520 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:09.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.367+0000 7fc331120700 1 -- 172.21.15.60:0/3217834520 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc32c05d070 msgr2=0x7fc32c116920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:09.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.367+0000 7fc331120700 1 --2- 172.21.15.60:0/3217834520 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc32c05d070 0x7fc32c116920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:09.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.367+0000 7fc331120700 1 -- 172.21.15.60:0/3217834520 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc32c0b2130 msgr2=0x7fc32c064690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:09.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.367+0000 7fc331120700 1 --2- 172.21.15.60:0/3217834520 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc32c0b2130 0x7fc32c064690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:09.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.367+0000 7fc331120700 1 -- 172.21.15.60:0/3217834520 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc324009470 con 0x7fc32c0aed00 2022-06-29T15:59:09.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.367+0000 7fc331120700 1 --2- 172.21.15.60:0/3217834520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc32c0aed00 0x7fc32c0641a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fc32c0b5cd0 tx=0x7fc328007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:09.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.367+0000 7fc31affd700 1 -- 172.21.15.60:0/3217834520 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc328012400 con 0x7fc32c0aed00 2022-06-29T15:59:09.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.367+0000 7fc31affd700 1 -- 172.21.15.60:0/3217834520 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc328012560 con 0x7fc32c0aed00 2022-06-29T15:59:09.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.367+0000 7fc332923700 1 -- 172.21.15.60:0/3217834520 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc32c119260 con 0x7fc32c0aed00 2022-06-29T15:59:09.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.367+0000 7fc31affd700 1 -- 172.21.15.60:0/3217834520 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc3280128c0 con 0x7fc32c0aed00 2022-06-29T15:59:09.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.367+0000 7fc332923700 1 -- 172.21.15.60:0/3217834520 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc32c116e10 con 0x7fc32c0aed00 2022-06-29T15:59:09.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.367+0000 7fc332923700 1 -- 172.21.15.60:0/3217834520 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc32c14a5f0 con 0x7fc32c0aed00 2022-06-29T15:59:09.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.371+0000 7fc31affd700 1 -- 172.21.15.60:0/3217834520 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc328019030 con 0x7fc32c0aed00 2022-06-29T15:59:09.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.371+0000 7fc31affd700 1 --2- 172.21.15.60:0/3217834520 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc30c0539e0 0x7fc30c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:09.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.371+0000 7fc31affd700 1 -- 172.21.15.60:0/3217834520 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc32806b160 con 0x7fc32c0aed00 2022-06-29T15:59:09.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.371+0000 7fc331921700 1 --2- 172.21.15.60:0/3217834520 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc30c0539e0 0x7fc30c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:09.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.371+0000 7fc331921700 1 --2- 172.21.15.60:0/3217834520 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc30c0539e0 0x7fc30c055ea0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fc31c00d000 tx=0x7fc31c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:09.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.375+0000 7fc31affd700 1 -- 172.21.15.60:0/3217834520 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc328042080 con 0x7fc32c0aed00 2022-06-29T15:59:09.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.499+0000 7fc332923700 1 -- 172.21.15.60:0/3217834520 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc32c0573d0 con 0x7fc32c0aed00 2022-06-29T15:59:09.510 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.499+0000 7fc31affd700 1 -- 172.21.15.60:0/3217834520 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v32) v1 ==== 104+0+10 (secure 0 0 0) 0x7fc3280456a0 con 0x7fc32c0aed00 2022-06-29T15:59:09.511 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T15:59:09.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.503+0000 7fc332923700 1 -- 172.21.15.60:0/3217834520 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc30c0539e0 msgr2=0x7fc30c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:09.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.503+0000 7fc332923700 1 --2- 172.21.15.60:0/3217834520 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc30c0539e0 0x7fc30c055ea0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fc31c00d000 tx=0x7fc31c009040).stop 2022-06-29T15:59:09.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.503+0000 7fc332923700 1 -- 172.21.15.60:0/3217834520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc32c0aed00 msgr2=0x7fc32c0641a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:09.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.503+0000 7fc332923700 1 --2- 172.21.15.60:0/3217834520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc32c0aed00 0x7fc32c0641a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fc32c0b5cd0 tx=0x7fc328007c50).stop 2022-06-29T15:59:09.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.503+0000 7fc332923700 1 -- 172.21.15.60:0/3217834520 shutdown_connections 2022-06-29T15:59:09.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.503+0000 7fc332923700 1 --2- 172.21.15.60:0/3217834520 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc30c0539e0 0x7fc30c055ea0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:09.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.503+0000 7fc332923700 1 --2- 172.21.15.60:0/3217834520 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc32c0b2130 0x7fc32c064690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:09.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.503+0000 7fc332923700 1 --2- 172.21.15.60:0/3217834520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc32c0aed00 0x7fc32c0641a0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:09.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.503+0000 7fc332923700 1 --2- 172.21.15.60:0/3217834520 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc32c05d070 0x7fc32c116920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:09.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.503+0000 7fc332923700 1 -- 172.21.15.60:0/3217834520 >> 172.21.15.60:0/3217834520 conn(0x7fc32c0b5890 msgr2=0x7fc32c0aca20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:09.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.503+0000 7fc332923700 1 -- 172.21.15.60:0/3217834520 shutdown_connections 2022-06-29T15:59:09.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.503+0000 7fc332923700 1 -- 172.21.15.60:0/3217834520 wait complete. 2022-06-29T15:59:09.524 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-29T15:59:09.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.631+0000 7f45a33da700 1 Processor -- start 2022-06-29T15:59:09.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.631+0000 7f45a33da700 1 -- start start 2022-06-29T15:59:09.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.631+0000 7f45a33da700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f459c0b3270 0x7f459c0ace00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:09.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.631+0000 7f45a33da700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f459c062cd0 0x7f459c1461c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:09.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.631+0000 7f45a33da700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459c0adc80 con 0x7f459c0aca20 2022-06-29T15:59:09.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.631+0000 7f45a33da700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f459c0ad740 con 0x7f459c0b3270 2022-06-29T15:59:09.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.631+0000 7f45a33da700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f459c0ad9e0 con 0x7f459c062cd0 2022-06-29T15:59:09.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.631+0000 7f45a23d8700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f459c0aca20 0x7f459c143030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54682/0 (socket says 172.21.15.60:54682) 2022-06-29T15:59:09.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.631+0000 7f45a23d8700 1 -- 172.21.15.60:0/835844366 learned_addr learned my addr 172.21.15.60:0/835844366 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:59:09.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.631+0000 7f45a2bd9700 1 --2- 172.21.15.60:0/835844366 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f459c062cd0 0x7f459c1461c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:09.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.631+0000 7f45a1bd7700 1 --2- 172.21.15.60:0/835844366 >> v2:172.21.15.92:3300/0 conn(0x7f459c0b3270 0x7f459c0ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:09.646 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.635+0000 7f45a2bd9700 1 -- 172.21.15.60:0/835844366 >> v2:172.21.15.92:3300/0 conn(0x7f459c0b3270 msgr2=0x7f459c0ace00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:09.646 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.635+0000 7f45a2bd9700 1 --2- 172.21.15.60:0/835844366 >> v2:172.21.15.92:3300/0 conn(0x7f459c0b3270 0x7f459c0ace00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:09.647 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.635+0000 7f45a2bd9700 1 -- 172.21.15.60:0/835844366 >> v1:172.21.15.60:6789/0 conn(0x7f459c0aca20 legacy=0x7f459c143030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:09.647 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.635+0000 7f45a2bd9700 1 -- 172.21.15.60:0/835844366 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459c142920 con 0x7f459c062cd0 2022-06-29T15:59:09.647 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.635+0000 7f45a13d6700 1 -- 172.21.15.60:0/835844366 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1672118299 0 0) 0x7f459c0adc80 con 0x7f459c0aca20 2022-06-29T15:59:09.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.635+0000 7f45a2bd9700 1 --2- 172.21.15.60:0/835844366 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f459c062cd0 0x7f459c1461c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f45940032f0 tx=0x7f45940091d0).ready entity=mon.2 client_cookie=ab201930642dd63e server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:09.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.635+0000 7f45a13d6700 1 -- 172.21.15.60:0/835844366 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f459400b6c0 con 0x7f459c062cd0 2022-06-29T15:59:09.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.635+0000 7f45a13d6700 1 -- 172.21.15.60:0/835844366 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f459400b820 con 0x7f459c062cd0 2022-06-29T15:59:09.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.635+0000 7f45a13d6700 1 -- 172.21.15.60:0/835844366 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f459400b990 con 0x7f459c062cd0 2022-06-29T15:59:09.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.635+0000 7f45a33da700 1 -- 172.21.15.60:0/835844366 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f459c062cd0 msgr2=0x7f459c1461c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:09.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.635+0000 7f45a33da700 1 --2- 172.21.15.60:0/835844366 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f459c062cd0 0x7f459c1461c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f45940032f0 tx=0x7f45940091d0).stop 2022-06-29T15:59:09.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.635+0000 7f45a33da700 1 -- 172.21.15.60:0/835844366 shutdown_connections 2022-06-29T15:59:09.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.635+0000 7f45a33da700 1 --2- 172.21.15.60:0/835844366 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f459c062cd0 0x7f459c1461c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:09.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.635+0000 7f45a33da700 1 --2- 172.21.15.60:0/835844366 >> v2:172.21.15.92:3300/0 conn(0x7f459c0b3270 0x7f459c0ace00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:09.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.635+0000 7f45a33da700 1 -- 172.21.15.60:0/835844366 >> 172.21.15.60:0/835844366 conn(0x7f459c0b6f80 msgr2=0x7f459c0b7390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:09.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.635+0000 7f45a33da700 1 -- 172.21.15.60:0/835844366 shutdown_connections 2022-06-29T15:59:09.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.635+0000 7f45a33da700 1 -- 172.21.15.60:0/835844366 wait complete. 2022-06-29T15:59:09.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.635+0000 7f45a33da700 1 Processor -- start 2022-06-29T15:59:09.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.635+0000 7f45a33da700 1 -- start start 2022-06-29T15:59:09.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.639+0000 7f45a33da700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f459c062cd0 0x7f459c1556e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:09.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.639+0000 7f45a33da700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f459c0aca20 0x7f459c064840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:09.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.639+0000 7f45a33da700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f459c0b3270 0x7f459c0af100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:09.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.639+0000 7f45a33da700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459c155bd0 con 0x7f459c0b3270 2022-06-29T15:59:09.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.639+0000 7f45a33da700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459c155e10 con 0x7f459c062cd0 2022-06-29T15:59:09.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.639+0000 7f45a33da700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f459c064d30 con 0x7f459c0aca20 2022-06-29T15:59:09.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.639+0000 7f45a2bd9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f459c0b3270 0x7f459c0af100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:09.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.639+0000 7f45a2bd9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f459c0b3270 0x7f459c0af100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46606/0 (socket says 172.21.15.60:46606) 2022-06-29T15:59:09.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.639+0000 7f45a2bd9700 1 -- 172.21.15.60:0/4123770029 learned_addr learned my addr 172.21.15.60:0/4123770029 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:09.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.639+0000 7f45a1bd7700 1 --2- 172.21.15.60:0/4123770029 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f459c0aca20 0x7f459c064840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:09.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.639+0000 7f45a23d8700 1 --2- 172.21.15.60:0/4123770029 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f459c062cd0 0x7f459c1556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:09.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.639+0000 7f45a1bd7700 1 -- 172.21.15.60:0/4123770029 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f459c062cd0 msgr2=0x7f459c1556e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:09.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.639+0000 7f45a1bd7700 1 --2- 172.21.15.60:0/4123770029 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f459c062cd0 0x7f459c1556e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:09.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.639+0000 7f45a1bd7700 1 -- 172.21.15.60:0/4123770029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f459c0b3270 msgr2=0x7f459c0af100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:09.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.639+0000 7f45a1bd7700 1 --2- 172.21.15.60:0/4123770029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f459c0b3270 0x7f459c0af100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:09.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.639+0000 7f45a1bd7700 1 -- 172.21.15.60:0/4123770029 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4594009470 con 0x7f459c0aca20 2022-06-29T15:59:09.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.639+0000 7f45a1bd7700 1 --2- 172.21.15.60:0/4123770029 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f459c0aca20 0x7f459c064840 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f459800c000 tx=0x7f4598006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:09.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.639+0000 7f458f7fe700 1 -- 172.21.15.60:0/4123770029 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4598011750 con 0x7f459c0aca20 2022-06-29T15:59:09.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.639+0000 7f458f7fe700 1 -- 172.21.15.60:0/4123770029 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45980118b0 con 0x7f459c0aca20 2022-06-29T15:59:09.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.639+0000 7f45a33da700 1 -- 172.21.15.60:0/4123770029 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f459c0ae040 con 0x7f459c0aca20 2022-06-29T15:59:09.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.639+0000 7f458f7fe700 1 -- 172.21.15.60:0/4123770029 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4598011c10 con 0x7f459c0aca20 2022-06-29T15:59:09.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.639+0000 7f45a33da700 1 -- 172.21.15.60:0/4123770029 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f459c0b0f30 con 0x7f459c0aca20 2022-06-29T15:59:09.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.639+0000 7f45a33da700 1 -- 172.21.15.60:0/4123770029 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f459c0573d0 con 0x7f459c0aca20 2022-06-29T15:59:09.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.643+0000 7f458f7fe700 1 -- 172.21.15.60:0/4123770029 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4598011d70 con 0x7f459c0aca20 2022-06-29T15:59:09.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.643+0000 7f458f7fe700 1 --2- 172.21.15.60:0/4123770029 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f457c053b30 0x7f457c055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:09.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.643+0000 7f458f7fe700 1 -- 172.21.15.60:0/4123770029 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f459806c560 con 0x7f459c0aca20 2022-06-29T15:59:09.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.643+0000 7f45a23d8700 1 --2- 172.21.15.60:0/4123770029 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f457c053b30 0x7f457c055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:09.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.643+0000 7f45a23d8700 1 --2- 172.21.15.60:0/4123770029 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f457c053b30 0x7f457c055ff0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f459000d000 tx=0x7f4590007c20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:09.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.651+0000 7f458f7fe700 1 -- 172.21.15.60:0/4123770029 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4598043480 con 0x7f459c0aca20 2022-06-29T15:59:09.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.779+0000 7f45a33da700 1 -- 172.21.15.60:0/4123770029 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f459c143a30 con 0x7f459c0aca20 2022-06-29T15:59:09.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.779+0000 7f458f7fe700 1 -- 172.21.15.60:0/4123770029 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v32) v1 ==== 100+0+10 (secure 0 0 0) 0x7f4598046aa0 con 0x7f459c0aca20 2022-06-29T15:59:09.790 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T15:59:09.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.783+0000 7f45a33da700 1 -- 172.21.15.60:0/4123770029 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f457c053b30 msgr2=0x7f457c055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:09.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.783+0000 7f45a33da700 1 --2- 172.21.15.60:0/4123770029 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f457c053b30 0x7f457c055ff0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f459000d000 tx=0x7f4590007c20).stop 2022-06-29T15:59:09.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.783+0000 7f45a33da700 1 -- 172.21.15.60:0/4123770029 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f459c0aca20 msgr2=0x7f459c064840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:09.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.783+0000 7f45a33da700 1 --2- 172.21.15.60:0/4123770029 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f459c0aca20 0x7f459c064840 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f459800c000 tx=0x7f4598006df0).stop 2022-06-29T15:59:09.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.783+0000 7f45a33da700 1 -- 172.21.15.60:0/4123770029 shutdown_connections 2022-06-29T15:59:09.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.783+0000 7f45a33da700 1 --2- 172.21.15.60:0/4123770029 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f457c053b30 0x7f457c055ff0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:09.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.783+0000 7f45a33da700 1 --2- 172.21.15.60:0/4123770029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f459c0b3270 0x7f459c0af100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:09.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.783+0000 7f45a33da700 1 --2- 172.21.15.60:0/4123770029 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f459c0aca20 0x7f459c064840 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:09.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.783+0000 7f45a33da700 1 --2- 172.21.15.60:0/4123770029 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f459c062cd0 0x7f459c1556e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:09.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.783+0000 7f45a33da700 1 -- 172.21.15.60:0/4123770029 >> 172.21.15.60:0/4123770029 conn(0x7f459c0b6f80 msgr2=0x7f459c0b23e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:09.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.783+0000 7f45a33da700 1 -- 172.21.15.60:0/4123770029 shutdown_connections 2022-06-29T15:59:09.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.783+0000 7f45a33da700 1 -- 172.21.15.60:0/4123770029 wait complete. 2022-06-29T15:59:09.808 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-06-29T15:59:09.809 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T15:59:09.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.899+0000 7f1fc2ae5700 1 Processor -- start 2022-06-29T15:59:09.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.899+0000 7f1fc2ae5700 1 -- start start 2022-06-29T15:59:09.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.903+0000 7f1fc2ae5700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1fbc0b61d0 0x7f1fbc0b0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:09.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.903+0000 7f1fc2ae5700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1fbc13fc00 0x7f1fbc13c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:09.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.903+0000 7f1fc2ae5700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fbc142820 con 0x7f1fbc13f790 2022-06-29T15:59:09.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.903+0000 7f1fc2ae5700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fbc062bb0 con 0x7f1fbc0b61d0 2022-06-29T15:59:09.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.903+0000 7f1fc2ae5700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fbc0b65b0 con 0x7f1fbc13fc00 2022-06-29T15:59:09.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.903+0000 7f1fc0881700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1fbc0b61d0 0x7f1fbc0b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:09.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.903+0000 7f1fc0881700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1fbc0b61d0 0x7f1fbc0b0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46612/0 (socket says 172.21.15.60:46612) 2022-06-29T15:59:09.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.903+0000 7f1fc0881700 1 -- 172.21.15.60:0/1545977010 learned_addr learned my addr 172.21.15.60:0/1545977010 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:09.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.903+0000 7f1fc1082700 1 --2- 172.21.15.60:0/1545977010 >> v2:172.21.15.92:3300/0 conn(0x7f1fbc13fc00 0x7f1fbc13c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:09.915 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.903+0000 7f1fc0881700 1 -- 172.21.15.60:0/1545977010 >> v2:172.21.15.92:3300/0 conn(0x7f1fbc13fc00 msgr2=0x7f1fbc13c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:09.915 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.903+0000 7f1fc0881700 1 --2- 172.21.15.60:0/1545977010 >> v2:172.21.15.92:3300/0 conn(0x7f1fbc13fc00 0x7f1fbc13c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:09.915 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.903+0000 7f1fc0881700 1 -- 172.21.15.60:0/1545977010 >> v1:172.21.15.92:6789/0 conn(0x7f1fbc13f790 legacy=0x7f1fbc13bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:09.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.903+0000 7f1fc0881700 1 -- 172.21.15.60:0/1545977010 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fbc13f050 con 0x7f1fbc0b61d0 2022-06-29T15:59:09.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.903+0000 7f1fc1082700 1 --2- 172.21.15.60:0/1545977010 >> v2:172.21.15.92:3300/0 conn(0x7f1fbc13fc00 0x7f1fbc13c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:59:09.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.903+0000 7f1fc0881700 1 --2- 172.21.15.60:0/1545977010 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fbc0b61d0 0x7f1fbc0b0540 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f1fbc0acf50 tx=0x7f1fb0006eb0).ready entity=mon.0 client_cookie=488ee2de17667f32 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:09.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.903+0000 7f1fbb7fe700 1 -- 172.21.15.60:0/1545977010 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1fb000d610 con 0x7f1fbc0b61d0 2022-06-29T15:59:09.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.903+0000 7f1fbb7fe700 1 -- 172.21.15.60:0/1545977010 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fb0000e90 con 0x7f1fbc0b61d0 2022-06-29T15:59:09.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.903+0000 7f1fbb7fe700 1 -- 172.21.15.60:0/1545977010 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1fb000d960 con 0x7f1fbc0b61d0 2022-06-29T15:59:09.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.903+0000 7f1fc2ae5700 1 -- 172.21.15.60:0/1545977010 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fbc0b61d0 msgr2=0x7f1fbc0b0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:09.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.903+0000 7f1fc2ae5700 1 --2- 172.21.15.60:0/1545977010 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fbc0b61d0 0x7f1fbc0b0540 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f1fbc0acf50 tx=0x7f1fb0006eb0).stop 2022-06-29T15:59:09.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.903+0000 7f1fc2ae5700 1 -- 172.21.15.60:0/1545977010 shutdown_connections 2022-06-29T15:59:09.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.903+0000 7f1fc2ae5700 1 --2- 172.21.15.60:0/1545977010 >> v2:172.21.15.92:3300/0 conn(0x7f1fbc13fc00 0x7f1fbc13c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:09.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.903+0000 7f1fc2ae5700 1 --2- 172.21.15.60:0/1545977010 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fbc0b61d0 0x7f1fbc0b0540 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:09.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.903+0000 7f1fc2ae5700 1 -- 172.21.15.60:0/1545977010 >> 172.21.15.60:0/1545977010 conn(0x7f1fbc063b50 msgr2=0x7f1fbc0b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:09.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.903+0000 7f1fc2ae5700 1 -- 172.21.15.60:0/1545977010 shutdown_connections 2022-06-29T15:59:09.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.903+0000 7f1fc2ae5700 1 -- 172.21.15.60:0/1545977010 wait complete. 2022-06-29T15:59:09.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.907+0000 7f1fc2ae5700 1 Processor -- start 2022-06-29T15:59:09.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.907+0000 7f1fc2ae5700 1 -- start start 2022-06-29T15:59:09.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.907+0000 7f1fc2ae5700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fbc0b61d0 0x7f1fbc0ae7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:09.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.907+0000 7f1fc2ae5700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1fbc13f790 0x7f1fbc14ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:09.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.907+0000 7f1fc2ae5700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fbc13fc00 0x7f1fbc155350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:09.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.907+0000 7f1fc2ae5700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc14b3f0 con 0x7f1fbc0b61d0 2022-06-29T15:59:09.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.907+0000 7f1fc2ae5700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc0665d0 con 0x7f1fbc13f790 2022-06-29T15:59:09.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.907+0000 7f1fc2ae5700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc066870 con 0x7f1fbc13fc00 2022-06-29T15:59:09.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.907+0000 7f1fc0881700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fbc0b61d0 0x7f1fbc0ae7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:09.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.907+0000 7f1fc0881700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fbc0b61d0 0x7f1fbc0ae7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46616/0 (socket says 172.21.15.60:46616) 2022-06-29T15:59:09.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.907+0000 7f1fc0881700 1 -- 172.21.15.60:0/1031539784 learned_addr learned my addr 172.21.15.60:0/1031539784 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:09.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.907+0000 7f1fc1082700 1 --2- 172.21.15.60:0/1031539784 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fbc13fc00 0x7f1fbc155350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:09.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.907+0000 7f1fbbfff700 1 --2- 172.21.15.60:0/1031539784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1fbc13f790 0x7f1fbc14ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:09.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.907+0000 7f1fc0881700 1 -- 172.21.15.60:0/1031539784 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fbc13fc00 msgr2=0x7f1fbc155350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:09.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.907+0000 7f1fc0881700 1 --2- 172.21.15.60:0/1031539784 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fbc13fc00 0x7f1fbc155350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:09.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.907+0000 7f1fc0881700 1 -- 172.21.15.60:0/1031539784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1fbc13f790 msgr2=0x7f1fbc14ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:09.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.907+0000 7f1fc0881700 1 --2- 172.21.15.60:0/1031539784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1fbc13f790 0x7f1fbc14ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:09.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.907+0000 7f1fc0881700 1 -- 172.21.15.60:0/1031539784 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb4009470 con 0x7f1fbc0b61d0 2022-06-29T15:59:09.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.907+0000 7f1fc1082700 1 --2- 172.21.15.60:0/1031539784 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fbc13fc00 0x7f1fbc155350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:59:09.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.907+0000 7f1fc0881700 1 --2- 172.21.15.60:0/1031539784 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fbc0b61d0 0x7f1fbc0ae7f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f1fbc0acf50 tx=0x7f1fb0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:09.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.907+0000 7f1fb9ffb700 1 -- 172.21.15.60:0/1031539784 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1fb0002f50 con 0x7f1fbc0b61d0 2022-06-29T15:59:09.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.907+0000 7f1fb9ffb700 1 -- 172.21.15.60:0/1031539784 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fb000d990 con 0x7f1fbc0b61d0 2022-06-29T15:59:09.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.907+0000 7f1fc2ae5700 1 -- 172.21.15.60:0/1031539784 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fb000b1e0 con 0x7f1fbc0b61d0 2022-06-29T15:59:09.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.907+0000 7f1fb9ffb700 1 -- 172.21.15.60:0/1031539784 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1fb0011e90 con 0x7f1fbc0b61d0 2022-06-29T15:59:09.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.907+0000 7f1fc2ae5700 1 -- 172.21.15.60:0/1031539784 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fbc0af640 con 0x7f1fbc0b61d0 2022-06-29T15:59:09.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.911+0000 7f1fc2ae5700 1 -- 172.21.15.60:0/1031539784 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fbc13f2f0 con 0x7f1fbc0b61d0 2022-06-29T15:59:09.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.911+0000 7f1fb9ffb700 1 -- 172.21.15.60:0/1031539784 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1fb0025030 con 0x7f1fbc0b61d0 2022-06-29T15:59:09.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.911+0000 7f1fb9ffb700 1 --2- 172.21.15.60:0/1031539784 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1f9c053a70 0x7f1f9c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:09.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.911+0000 7f1fb9ffb700 1 -- 172.21.15.60:0/1031539784 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1fb006d120 con 0x7f1fbc0b61d0 2022-06-29T15:59:09.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.911+0000 7f1fbbfff700 1 --2- 172.21.15.60:0/1031539784 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1f9c053a70 0x7f1f9c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:09.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.911+0000 7f1fbbfff700 1 --2- 172.21.15.60:0/1031539784 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1f9c053a70 0x7f1f9c055f30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f1fa800b000 tx=0x7f1fa8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:09.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:09.919+0000 7f1fb9ffb700 1 -- 172.21.15.60:0/1031539784 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1fb0044080 con 0x7f1fbc0b61d0 2022-06-29T15:59:10.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:10.083+0000 7f1fc2ae5700 1 -- 172.21.15.60:0/1031539784 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fbc066a30 con 0x7f1fbc0b61d0 2022-06-29T15:59:10.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:10.083+0000 7f1fb9ffb700 1 -- 172.21.15.60:0/1031539784 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+2134 (secure 0 0 0) 0x7f1fb0023020 con 0x7f1fbc0b61d0 2022-06-29T15:59:10.092 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:59:10.092 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":9,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4884,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2555925680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2555925680},{"type":"v1","addr":"172.21.15.60:6833","nonce":2555925680}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":7}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T15:59:05.257113+0000","modified":"2022-06-29T15:59:06.259879+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4933},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4933":{"gid":4933,"name":"b","rank":0,"incarnation":8,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/1993861928","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1993861928},{"type":"v1","addr":"172.21.15.92:6835","nonce":1993861928}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-06-29T15:59:10.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:10.083+0000 7f1fc2ae5700 1 -- 172.21.15.60:0/1031539784 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1f9c053a70 msgr2=0x7f1f9c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:10.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:10.083+0000 7f1fc2ae5700 1 --2- 172.21.15.60:0/1031539784 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1f9c053a70 0x7f1f9c055f30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f1fa800b000 tx=0x7f1fa8009040).stop 2022-06-29T15:59:10.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:10.083+0000 7f1fc2ae5700 1 -- 172.21.15.60:0/1031539784 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fbc0b61d0 msgr2=0x7f1fbc0ae7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:10.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:10.083+0000 7f1fc2ae5700 1 --2- 172.21.15.60:0/1031539784 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fbc0b61d0 0x7f1fbc0ae7f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f1fbc0acf50 tx=0x7f1fb0015000).stop 2022-06-29T15:59:10.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:10.083+0000 7f1fc2ae5700 1 -- 172.21.15.60:0/1031539784 shutdown_connections 2022-06-29T15:59:10.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:10.083+0000 7f1fc2ae5700 1 --2- 172.21.15.60:0/1031539784 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1f9c053a70 0x7f1f9c055f30 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:10.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:10.083+0000 7f1fc2ae5700 1 --2- 172.21.15.60:0/1031539784 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1fbc13fc00 0x7f1fbc155350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:10.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:10.083+0000 7f1fc2ae5700 1 --2- 172.21.15.60:0/1031539784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1fbc13f790 0x7f1fbc14ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:10.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:10.083+0000 7f1fc2ae5700 1 --2- 172.21.15.60:0/1031539784 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fbc0b61d0 0x7f1fbc0ae7f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:10.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:10.083+0000 7f1fc2ae5700 1 -- 172.21.15.60:0/1031539784 >> 172.21.15.60:0/1031539784 conn(0x7f1fbc063b50 msgr2=0x7f1fbc0b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:10.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:10.083+0000 7f1fc2ae5700 1 -- 172.21.15.60:0/1031539784 shutdown_connections 2022-06-29T15:59:10.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:10.083+0000 7f1fc2ae5700 1 -- 172.21.15.60:0/1031539784 wait complete. 2022-06-29T15:59:10.098 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 9 2022-06-29T15:59:10.306 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T15:59:05.257113+0000', 'modified': '2022-06-29T15:59:06.259879+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4933}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4933': {'gid': 4933, 'name': 'b', 'rank': 0, 'incarnation': 8, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.92:6835/1993861928', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 1993861928}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 1993861928}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T15:59:10.308 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T15:59:10.309 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-06-29T15:59:10.423 INFO:teuthology.orchestra.run.smithi092.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":4933,"want_state":"up:active","state":"up:active","rank_uptime":5.1475701120000004,"mdsmap_epoch":9,"osdmap_epoch":32,"osdmap_epoch_barrier":31,"uptime":11.343616388999999} 2022-06-29T15:59:10.424 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 4933, "mdsmap_epoch": 9, "osdmap_epoch": 32, "osdmap_epoch_barrier": 31, "rank_uptime": 5.147570112, "state": "up:active", "uptime": 11.343616389, "want_state": "up:active", "whoami": 0 } 2022-06-29T15:59:10.425 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-06-29T15:59:10.425 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-06-29T15:59:10.425 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi121.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-06-29T15:59:10.426 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:59:10.426 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-06-29T15:59:10.435 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T15:59:10.454 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T15:59:10.455 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T15:59:10.514 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T15:59:10.515 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T15:59:10.516 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:59:10.517 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T15:59:10.523 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-06-29T15:59:10.524 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-06-29T15:59:10.527 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T15:59:10.542 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T15:59:10.542 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T15:59:10.556 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T15:59:10.557 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T15:59:10.558 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:59:10.559 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T15:59:10.599 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[12335]: starting ceph client2022-06-29T15:59:10.595+0000 7f3b8beb3080 -1 init, newargv = 0x55a8c39a3ff0 newargc=9 2022-06-29T15:59:10.600 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr: 2022-06-29T15:59:10.607 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[12335]: starting fuse 2022-06-29T15:59:11.708 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T15:59:11.727 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T15:59:11.728 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T15:59:11.786 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T15:59:11.787 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T15:59:11.788 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:59:11.789 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T15:59:11.794 INFO:teuthology.orchestra.run.smithi121.stdout:52 2022-06-29T15:59:11.794 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-06-29T15:59:11.795 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:59:11.795 DEBUG:teuthology.orchestra.run.smithi121:> sudo python3 -c ' 2022-06-29T15:59:11.796 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2022-06-29T15:59:11.796 DEBUG:teuthology.orchestra.run.smithi121:> import re 2022-06-29T15:59:11.797 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-06-29T15:59:11.797 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2022-06-29T15:59:11.797 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T15:59:11.798 DEBUG:teuthology.orchestra.run.smithi121:> def find_socket(client_name): 2022-06-29T15:59:11.798 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-06-29T15:59:11.799 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2022-06-29T15:59:11.799 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T15:59:11.799 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2022-06-29T15:59:11.800 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2022-06-29T15:59:11.801 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2022-06-29T15:59:11.801 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2022-06-29T15:59:11.801 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T15:59:11.802 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2022-06-29T15:59:11.802 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-06-29T15:59:11.803 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2022-06-29T15:59:11.803 DEBUG:teuthology.orchestra.run.smithi121:> return f 2022-06-29T15:59:11.804 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-06-29T15:59:11.804 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T15:59:11.804 DEBUG:teuthology.orchestra.run.smithi121:> print(find_socket("client.0")) 2022-06-29T15:59:11.805 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-06-29T15:59:11.850 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.0.12335.asok 2022-06-29T15:59:11.855 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.12335.asok 2022-06-29T15:59:11.856 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:59:11.857 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.12335.asok status 2022-06-29T15:59:11.992 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2022-06-29T15:59:11.992 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2022-06-29T15:59:11.993 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "f25ffac72e5a9e63860ff7deb6767a11f0559a16", 2022-06-29T15:59:11.993 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 15.2.16-304-gf25ffac7 (f25ffac72e5a9e63860ff7deb6767a11f0559a16) octopus (stable)", 2022-06-29T15:59:11.993 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "0", 2022-06-29T15:59:11.994 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2022-06-29T15:59:11.994 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-06-29T15:59:11.994 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "12335", 2022-06-29T15:59:11.995 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2022-06-29T15:59:11.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T15:59:11.997 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2022-06-29T15:59:11.997 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2022-06-29T15:59:11.998 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 4835, 2022-06-29T15:59:11.998 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2022-06-29T15:59:11.999 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2022-06-29T15:59:11.999 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2022-06-29T15:59:11.999 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4835 2022-06-29T15:59:12.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T15:59:12.000 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-06-29T15:59:12.001 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-06-29T15:59:12.001 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-06-29T15:59:12.002 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1872315104 2022-06-29T15:59:12.002 INFO:teuthology.orchestra.run.smithi121.stdout: } 2022-06-29T15:59:12.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T15:59:12.003 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-06-29T15:59:12.003 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-06-29T15:59:12.004 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-06-29T15:59:12.004 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1872315104 2022-06-29T15:59:12.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T15:59:12.005 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.4835 172.21.15.121:0/1872315104", 2022-06-29T15:59:12.005 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "172.21.15.121:0/1872315104", 2022-06-29T15:59:12.006 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2022-06-29T15:59:12.007 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 9, 2022-06-29T15:59:12.007 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 32, 2022-06-29T15:59:12.007 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2022-06-29T15:59:12.007 INFO:teuthology.orchestra.run.smithi121.stdout: "blacklisted": false, 2022-06-29T15:59:12.008 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2022-06-29T15:59:12.008 INFO:teuthology.orchestra.run.smithi121.stdout:} 2022-06-29T15:59:12.010 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:59:12.010 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T15:59:12.011 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T15:59:12.018 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T15:59:12.020 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T15:59:12.020 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:59:12.021 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-06-29T15:59:12.038 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok --format=json config get mds_min_caps_per_client 2022-06-29T15:59:12.147 INFO:teuthology.orchestra.run.smithi060.stdout:{"mds_min_caps_per_client":"100"} 2022-06-29T15:59:12.148 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_min_caps_per_client": "100" } 2022-06-29T15:59:12.149 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T15:59:12.149 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T15:59:12.149 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T15:59:12.158 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T15:59:12.159 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T15:59:12.160 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-06-29T15:59:12.160 DEBUG:teuthology.orchestra.run.smithi121:> import sys 2022-06-29T15:59:12.162 DEBUG:teuthology.orchestra.run.smithi121:> import time 2022-06-29T15:59:12.163 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-06-29T15:59:12.163 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T15:59:12.164 DEBUG:teuthology.orchestra.run.smithi121:> n = 1000 2022-06-29T15:59:12.164 DEBUG:teuthology.orchestra.run.smithi121:> abs_path = "/home/ubuntu/cephtest/mnt.0/dc-dir/dc-file" 2022-06-29T15:59:12.165 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T15:59:12.165 DEBUG:teuthology.orchestra.run.smithi121:> if not os.path.exists(os.path.dirname(abs_path)): 2022-06-29T15:59:12.165 DEBUG:teuthology.orchestra.run.smithi121:> os.makedirs(os.path.dirname(abs_path)) 2022-06-29T15:59:12.166 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T15:59:12.166 DEBUG:teuthology.orchestra.run.smithi121:> for i in range(0, n): 2022-06-29T15:59:12.167 DEBUG:teuthology.orchestra.run.smithi121:> fname = "{0}_{1}".format(abs_path, i) 2022-06-29T15:59:12.167 DEBUG:teuthology.orchestra.run.smithi121:> with open(fname, '"'"'w'"'"') as f: 2022-06-29T15:59:12.168 DEBUG:teuthology.orchestra.run.smithi121:> f.write('"'"'content'"'"') 2022-06-29T15:59:12.168 DEBUG:teuthology.orchestra.run.smithi121:> if True: 2022-06-29T15:59:12.168 DEBUG:teuthology.orchestra.run.smithi121:> f.flush() 2022-06-29T15:59:12.169 DEBUG:teuthology.orchestra.run.smithi121:> os.fsync(f.fileno()) 2022-06-29T15:59:12.169 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-06-29T15:59:21.454 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T15:59:21.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1aa0ad8700 1 Processor -- start 2022-06-29T15:59:21.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1aa0ad8700 1 -- start start 2022-06-29T15:59:21.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1aa0ad8700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a9c0b5950 0x7f1a9c067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:21.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1aa0ad8700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1a9c063ad0 0x7f1a9c14af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:21.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1aa0ad8700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a9c0acba0 con 0x7f1a9c05bd20 2022-06-29T15:59:21.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1aa0ad8700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a9c067220 con 0x7f1a9c063ad0 2022-06-29T15:59:21.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1aa0ad8700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a9c0b5d30 con 0x7f1a9c0b5950 2022-06-29T15:59:21.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1a9ad9d700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f1a9c05bd20 0x7f1a9c05c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54710/0 (socket says 172.21.15.60:54710) 2022-06-29T15:59:21.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1a9ad9d700 1 -- 172.21.15.60:0/4189472397 learned_addr learned my addr 172.21.15.60:0/4189472397 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:59:21.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1a99d9b700 1 --2- 172.21.15.60:0/4189472397 >> v2:172.21.15.92:3300/0 conn(0x7f1a9c063ad0 0x7f1a9c14af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:21.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1a9a59c700 1 --2- 172.21.15.60:0/4189472397 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a9c0b5950 0x7f1a9c067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:21.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1a9959a700 1 -- 172.21.15.60:0/4189472397 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1762326116 0 0) 0x7f1a9c0acba0 con 0x7f1a9c05bd20 2022-06-29T15:59:21.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1a9959a700 1 -- 172.21.15.60:0/4189472397 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a7c003260 con 0x7f1a9c05bd20 2022-06-29T15:59:21.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1a9959a700 1 -- 172.21.15.60:0/4189472397 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 278122281 0 0) 0x7f1a7c003260 con 0x7f1a9c05bd20 2022-06-29T15:59:21.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1a9959a700 1 -- 172.21.15.60:0/4189472397 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a9c0b5950 msgr2=0x7f1a9c067680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:21.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1a9959a700 1 --2- 172.21.15.60:0/4189472397 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a9c0b5950 0x7f1a9c067680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:21.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1a9959a700 1 -- 172.21.15.60:0/4189472397 >> v2:172.21.15.92:3300/0 conn(0x7f1a9c063ad0 msgr2=0x7f1a9c14af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:21.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1a9959a700 1 --2- 172.21.15.60:0/4189472397 >> v2:172.21.15.92:3300/0 conn(0x7f1a9c063ad0 0x7f1a9c14af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:21.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1a9959a700 1 -- 172.21.15.60:0/4189472397 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a9c0ace00 con 0x7f1a9c05bd20 2022-06-29T15:59:21.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1a9959a700 1 -- 172.21.15.60:0/4189472397 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f1a940019e0 con 0x7f1a9c05bd20 2022-06-29T15:59:21.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1a9959a700 1 -- 172.21.15.60:0/4189472397 >> v1:172.21.15.60:6789/0 conn(0x7f1a9c05bd20 legacy=0x7f1a9c05c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:21.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1a9959a700 1 --2- 172.21.15.60:0/4189472397 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a9c0b5950 0x7f1a7c003750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:21.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1a9959a700 1 --2- 172.21.15.60:0/4189472397 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a7c003c80 0x7f1a7c004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:21.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1a9959a700 1 --2- 172.21.15.60:0/4189472397 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a7c0049f0 0x7f1a7c004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:21.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1a9959a700 1 -- 172.21.15.60:0/4189472397 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a9c067220 con 0x7f1a7c0049f0 2022-06-29T15:59:21.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1a9959a700 1 -- 172.21.15.60:0/4189472397 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a9c0b5d30 con 0x7f1a7c003c80 2022-06-29T15:59:21.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1a9959a700 1 -- 172.21.15.60:0/4189472397 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7c00b750 con 0x7f1a9c0b5950 2022-06-29T15:59:21.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1a9ad9d700 1 --2- 172.21.15.60:0/4189472397 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a7c0049f0 0x7f1a7c004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:21.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1a9a59c700 1 --2- 172.21.15.60:0/4189472397 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a9c0b5950 0x7f1a7c003750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:21.586 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1a99d9b700 1 --2- 172.21.15.60:0/4189472397 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a7c003c80 0x7f1a7c004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:21.586 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1a9ad9d700 1 -- 172.21.15.60:0/4189472397 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a9c0b5950 msgr2=0x7f1a7c003750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:21.587 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1a9ad9d700 1 --2- 172.21.15.60:0/4189472397 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a9c0b5950 0x7f1a7c003750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:21.587 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1a9ad9d700 1 -- 172.21.15.60:0/4189472397 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a7c003c80 msgr2=0x7f1a7c004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:21.587 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1a9ad9d700 1 --2- 172.21.15.60:0/4189472397 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a7c003c80 0x7f1a7c004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:21.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1a9ad9d700 1 -- 172.21.15.60:0/4189472397 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a7c003260 con 0x7f1a7c0049f0 2022-06-29T15:59:21.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1a9ad9d700 1 --2- 172.21.15.60:0/4189472397 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a7c0049f0 0x7f1a7c004e70 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f1a7c0054d0 tx=0x7f1a9c0663b0).ready entity=mon.0 client_cookie=4453412fda6d72e9 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:21.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.559+0000 7f1a9959a700 1 -- 172.21.15.60:0/4189472397 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a9400cc40 con 0x7f1a7c0049f0 2022-06-29T15:59:21.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.563+0000 7f1a9959a700 1 -- 172.21.15.60:0/4189472397 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a9400cda0 con 0x7f1a7c0049f0 2022-06-29T15:59:21.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.563+0000 7f1a9959a700 1 -- 172.21.15.60:0/4189472397 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a94005b80 con 0x7f1a7c0049f0 2022-06-29T15:59:21.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.563+0000 7f1aa0ad8700 1 -- 172.21.15.60:0/4189472397 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a7c0049f0 msgr2=0x7f1a7c004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:21.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.563+0000 7f1aa0ad8700 1 --2- 172.21.15.60:0/4189472397 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a7c0049f0 0x7f1a7c004e70 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f1a7c0054d0 tx=0x7f1a9c0663b0).stop 2022-06-29T15:59:21.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.563+0000 7f1a9ad9d700 1 -- 172.21.15.60:0/4189472397 reap_dead start 2022-06-29T15:59:21.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.563+0000 7f1aa0ad8700 1 -- 172.21.15.60:0/4189472397 shutdown_connections 2022-06-29T15:59:21.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.563+0000 7f1aa0ad8700 1 -- 172.21.15.60:0/4189472397 >> 172.21.15.60:0/4189472397 conn(0x7f1a9c0632c0 msgr2=0x7f1a9c0b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:21.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.563+0000 7f1aa0ad8700 1 -- 172.21.15.60:0/4189472397 shutdown_connections 2022-06-29T15:59:21.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.563+0000 7f1aa0ad8700 1 -- 172.21.15.60:0/4189472397 wait complete. 2022-06-29T15:59:21.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.563+0000 7f1aa0ad8700 1 Processor -- start 2022-06-29T15:59:21.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.563+0000 7f1aa0ad8700 1 -- start start 2022-06-29T15:59:21.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.563+0000 7f1aa0ad8700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a9c15a010 0x7f1a9c116900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:21.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.563+0000 7f1aa0ad8700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a9c15a3f0 0x7f1a9c0af880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:21.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.563+0000 7f1aa0ad8700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a9c0b0ad0 0x7f1a9c0b1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:21.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.563+0000 7f1aa0ad8700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a9c142d80 con 0x7f1a9c15a010 2022-06-29T15:59:21.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.563+0000 7f1aa0ad8700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a9c1190d0 con 0x7f1a9c15a3f0 2022-06-29T15:59:21.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.563+0000 7f1aa0ad8700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a9c116fc0 con 0x7f1a9c0b0ad0 2022-06-29T15:59:21.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.563+0000 7f1a9a59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a9c15a010 0x7f1a9c116900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:21.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.563+0000 7f1a9a59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a9c15a010 0x7f1a9c116900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46636/0 (socket says 172.21.15.60:46636) 2022-06-29T15:59:21.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.563+0000 7f1a9a59c700 1 -- 172.21.15.60:0/205228985 learned_addr learned my addr 172.21.15.60:0/205228985 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:21.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.563+0000 7f1a9ad9d700 1 --2- 172.21.15.60:0/205228985 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a9c0b0ad0 0x7f1a9c0b1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:21.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.563+0000 7f1a99d9b700 1 --2- 172.21.15.60:0/205228985 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a9c15a3f0 0x7f1a9c0af880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:21.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.567+0000 7f1a9a59c700 1 -- 172.21.15.60:0/205228985 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a9c0b0ad0 msgr2=0x7f1a9c0b1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:21.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.567+0000 7f1a9a59c700 1 --2- 172.21.15.60:0/205228985 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a9c0b0ad0 0x7f1a9c0b1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:21.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.567+0000 7f1a9a59c700 1 -- 172.21.15.60:0/205228985 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a9c15a3f0 msgr2=0x7f1a9c0af880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:21.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.567+0000 7f1a9a59c700 1 --2- 172.21.15.60:0/205228985 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a9c15a3f0 0x7f1a9c0af880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:21.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.567+0000 7f1a9a59c700 1 -- 172.21.15.60:0/205228985 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a9c118dc0 con 0x7f1a9c15a010 2022-06-29T15:59:21.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.567+0000 7f1a9a59c700 1 --2- 172.21.15.60:0/205228985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a9c15a010 0x7f1a9c116900 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f1a8c00b000 tx=0x7f1a8c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:21.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.567+0000 7f1a937fe700 1 -- 172.21.15.60:0/205228985 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a8c00f760 con 0x7f1a9c15a010 2022-06-29T15:59:21.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.567+0000 7f1a937fe700 1 -- 172.21.15.60:0/205228985 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a8c00f8c0 con 0x7f1a9c15a010 2022-06-29T15:59:21.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.567+0000 7f1aa0ad8700 1 -- 172.21.15.60:0/205228985 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a9c159810 con 0x7f1a9c15a010 2022-06-29T15:59:21.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.567+0000 7f1a937fe700 1 -- 172.21.15.60:0/205228985 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a8c00fc20 con 0x7f1a9c15a010 2022-06-29T15:59:21.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.567+0000 7f1aa0ad8700 1 -- 172.21.15.60:0/205228985 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a9c064b30 con 0x7f1a9c15a010 2022-06-29T15:59:21.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.567+0000 7f1aa0ad8700 1 -- 172.21.15.60:0/205228985 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a9c117180 con 0x7f1a9c15a010 2022-06-29T15:59:21.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.567+0000 7f1a937fe700 1 -- 172.21.15.60:0/205228985 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1a8c014440 con 0x7f1a9c15a010 2022-06-29T15:59:21.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.571+0000 7f1a937fe700 1 --2- 172.21.15.60:0/205228985 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1a80053aa0 0x7f1a80055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:21.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.571+0000 7f1a937fe700 1 -- 172.21.15.60:0/205228985 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1a8c06c7b0 con 0x7f1a9c15a010 2022-06-29T15:59:21.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.571+0000 7f1a99d9b700 1 --2- 172.21.15.60:0/205228985 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1a80053aa0 0x7f1a80055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:21.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.571+0000 7f1a99d9b700 1 --2- 172.21.15.60:0/205228985 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1a80053aa0 0x7f1a80055f60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f1a8400b000 tx=0x7f1a84006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:21.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.575+0000 7f1a937fe700 1 -- 172.21.15.60:0/205228985 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1a8c043670 con 0x7f1a9c15a010 2022-06-29T15:59:21.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.743+0000 7f1aa0ad8700 1 -- 172.21.15.60:0/205228985 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a9c116df0 con 0x7f1a9c15a010 2022-06-29T15:59:21.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.747+0000 7f1a937fe700 1 -- 172.21.15.60:0/205228985 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+2134 (secure 0 0 0) 0x7f1a8c018020 con 0x7f1a9c15a010 2022-06-29T15:59:21.756 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:59:21.757 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":9,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4884,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2555925680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2555925680},{"type":"v1","addr":"172.21.15.60:6833","nonce":2555925680}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":7}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T15:59:05.257113+0000","modified":"2022-06-29T15:59:06.259879+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4933},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4933":{"gid":4933,"name":"b","rank":0,"incarnation":8,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/1993861928","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1993861928},{"type":"v1","addr":"172.21.15.92:6835","nonce":1993861928}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-06-29T15:59:21.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.747+0000 7f1aa0ad8700 1 -- 172.21.15.60:0/205228985 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1a80053aa0 msgr2=0x7f1a80055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:21.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.747+0000 7f1aa0ad8700 1 --2- 172.21.15.60:0/205228985 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1a80053aa0 0x7f1a80055f60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f1a8400b000 tx=0x7f1a84006df0).stop 2022-06-29T15:59:21.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.747+0000 7f1aa0ad8700 1 -- 172.21.15.60:0/205228985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a9c15a010 msgr2=0x7f1a9c116900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:21.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.747+0000 7f1aa0ad8700 1 --2- 172.21.15.60:0/205228985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a9c15a010 0x7f1a9c116900 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f1a8c00b000 tx=0x7f1a8c006df0).stop 2022-06-29T15:59:21.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.747+0000 7f1aa0ad8700 1 -- 172.21.15.60:0/205228985 shutdown_connections 2022-06-29T15:59:21.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.747+0000 7f1aa0ad8700 1 --2- 172.21.15.60:0/205228985 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1a80053aa0 0x7f1a80055f60 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:21.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.747+0000 7f1aa0ad8700 1 --2- 172.21.15.60:0/205228985 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a9c0b0ad0 0x7f1a9c0b1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:21.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.747+0000 7f1aa0ad8700 1 --2- 172.21.15.60:0/205228985 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a9c15a3f0 0x7f1a9c0af880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:21.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.747+0000 7f1aa0ad8700 1 --2- 172.21.15.60:0/205228985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a9c15a010 0x7f1a9c116900 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:21.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.747+0000 7f1aa0ad8700 1 -- 172.21.15.60:0/205228985 >> 172.21.15.60:0/205228985 conn(0x7f1a9c0632c0 msgr2=0x7f1a9c063f60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:21.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.747+0000 7f1aa0ad8700 1 -- 172.21.15.60:0/205228985 shutdown_connections 2022-06-29T15:59:21.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.747+0000 7f1aa0ad8700 1 -- 172.21.15.60:0/205228985 wait complete. 2022-06-29T15:59:21.766 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 9 2022-06-29T15:59:21.769 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json config set mds_recall_max_caps 20 2022-06-29T15:59:21.881 INFO:teuthology.orchestra.run.smithi092.stdout:{"success":"mds_recall_max_caps = '20' (not observed, change may require restart) "} 2022-06-29T15:59:21.882 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_caps = '20' (not observed, change may require restart) " } 2022-06-29T15:59:21.883 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T15:59:21.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.983+0000 7fde01333700 1 Processor -- start 2022-06-29T15:59:21.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.983+0000 7fde01333700 1 -- start start 2022-06-29T15:59:21.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.983+0000 7fde01333700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fddfc05d070 0x7fddfc14aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:21.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.983+0000 7fde01333700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fddfc0b2130 0x7fddfc0aca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:21.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.983+0000 7fde01333700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddfc05d450 con 0x7fddfc14af80 2022-06-29T15:59:21.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.983+0000 7fde01333700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddfc0adf90 con 0x7fddfc0b2130 2022-06-29T15:59:21.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.983+0000 7fde01333700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddfc0b25d0 con 0x7fddfc05d070 2022-06-29T15:59:21.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.983+0000 7fddfb7fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fddfc0b2130 0x7fddfc0aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:21.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.983+0000 7fddfb7fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fddfc0b2130 0x7fddfc0aca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46644/0 (socket says 172.21.15.60:46644) 2022-06-29T15:59:21.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.983+0000 7fddfb7fe700 1 -- 172.21.15.60:0/756981587 learned_addr learned my addr 172.21.15.60:0/756981587 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:21.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.983+0000 7fddfbfff700 1 --2- 172.21.15.60:0/756981587 >> v2:172.21.15.92:3300/0 conn(0x7fddfc05d070 0x7fddfc14aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:21.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.983+0000 7fddfb7fe700 1 -- 172.21.15.60:0/756981587 >> v2:172.21.15.92:3300/0 conn(0x7fddfc05d070 msgr2=0x7fddfc14aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:21.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.983+0000 7fddfb7fe700 1 --2- 172.21.15.60:0/756981587 >> v2:172.21.15.92:3300/0 conn(0x7fddfc05d070 0x7fddfc14aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.983+0000 7fddfb7fe700 1 -- 172.21.15.60:0/756981587 >> v1:172.21.15.92:6789/0 conn(0x7fddfc14af80 legacy=0x7fddfc0acf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:22.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.983+0000 7fddfb7fe700 1 -- 172.21.15.60:0/756981587 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddfc14a350 con 0x7fddfc0b2130 2022-06-29T15:59:22.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.983+0000 7fddfb7fe700 1 --2- 172.21.15.60:0/756981587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddfc0b2130 0x7fddfc0aca20 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7fdde40032f0 tx=0x7fdde4009290).ready entity=mon.0 client_cookie=99bed58a987202c8 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:22.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.987+0000 7fddfaffd700 1 -- 172.21.15.60:0/756981587 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdde400b870 con 0x7fddfc0b2130 2022-06-29T15:59:22.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.987+0000 7fddfaffd700 1 -- 172.21.15.60:0/756981587 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde400b9d0 con 0x7fddfc0b2130 2022-06-29T15:59:22.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.987+0000 7fddfaffd700 1 -- 172.21.15.60:0/756981587 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdde400bb40 con 0x7fddfc0b2130 2022-06-29T15:59:22.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.987+0000 7fde01333700 1 -- 172.21.15.60:0/756981587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddfc0b2130 msgr2=0x7fddfc0aca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:22.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.987+0000 7fde01333700 1 --2- 172.21.15.60:0/756981587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddfc0b2130 0x7fddfc0aca20 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7fdde40032f0 tx=0x7fdde4009290).stop 2022-06-29T15:59:22.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.987+0000 7fde01333700 1 -- 172.21.15.60:0/756981587 shutdown_connections 2022-06-29T15:59:22.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.987+0000 7fde01333700 1 --2- 172.21.15.60:0/756981587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddfc0b2130 0x7fddfc0aca20 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.987+0000 7fde01333700 1 --2- 172.21.15.60:0/756981587 >> v2:172.21.15.92:3300/0 conn(0x7fddfc05d070 0x7fddfc14aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.987+0000 7fde01333700 1 -- 172.21.15.60:0/756981587 >> 172.21.15.60:0/756981587 conn(0x7fddfc0b5890 msgr2=0x7fddfc0b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:22.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.987+0000 7fde01333700 1 -- 172.21.15.60:0/756981587 shutdown_connections 2022-06-29T15:59:22.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.987+0000 7fde01333700 1 -- 172.21.15.60:0/756981587 wait complete. 2022-06-29T15:59:22.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.987+0000 7fde01333700 1 Processor -- start 2022-06-29T15:59:22.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.987+0000 7fde01333700 1 -- start start 2022-06-29T15:59:22.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.987+0000 7fde01333700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fddfc05d070 0x7fddfc156df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:22.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.987+0000 7fde01333700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddfc0b2130 0x7fddfc116780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:22.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.987+0000 7fde01333700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fddfc14af80 0x7fddfc066630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:22.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.987+0000 7fde01333700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddfc1561f0 con 0x7fddfc0b2130 2022-06-29T15:59:22.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.987+0000 7fde01333700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddfc156410 con 0x7fddfc14af80 2022-06-29T15:59:22.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.987+0000 7fde01333700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddfc1566b0 con 0x7fddfc05d070 2022-06-29T15:59:22.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.987+0000 7fddfb7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddfc0b2130 0x7fddfc116780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:22.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.987+0000 7fddfb7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddfc0b2130 0x7fddfc116780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46650/0 (socket says 172.21.15.60:46650) 2022-06-29T15:59:22.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.987+0000 7fddfb7fe700 1 -- 172.21.15.60:0/2701139976 learned_addr learned my addr 172.21.15.60:0/2701139976 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:22.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.991+0000 7fddfbfff700 1 --2- 172.21.15.60:0/2701139976 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fddfc05d070 0x7fddfc156df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:22.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.991+0000 7fde00b32700 1 --2- 172.21.15.60:0/2701139976 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fddfc14af80 0x7fddfc066630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:22.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.991+0000 7fddfb7fe700 1 -- 172.21.15.60:0/2701139976 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fddfc05d070 msgr2=0x7fddfc156df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:22.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.991+0000 7fddfb7fe700 1 --2- 172.21.15.60:0/2701139976 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fddfc05d070 0x7fddfc156df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.991+0000 7fddfb7fe700 1 -- 172.21.15.60:0/2701139976 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fddfc14af80 msgr2=0x7fddfc066630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:22.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.991+0000 7fddfb7fe700 1 --2- 172.21.15.60:0/2701139976 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fddfc14af80 0x7fddfc066630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.991+0000 7fddfb7fe700 1 -- 172.21.15.60:0/2701139976 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde4009530 con 0x7fddfc0b2130 2022-06-29T15:59:22.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.991+0000 7fde00b32700 1 --2- 172.21.15.60:0/2701139976 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fddfc14af80 0x7fddfc066630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:59:22.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.991+0000 7fddfb7fe700 1 --2- 172.21.15.60:0/2701139976 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddfc0b2130 0x7fddfc116780 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fdde40032f0 tx=0x7fdde4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:22.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.991+0000 7fddf97fa700 1 -- 172.21.15.60:0/2701139976 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdde4021630 con 0x7fddfc0b2130 2022-06-29T15:59:22.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.991+0000 7fddf97fa700 1 -- 172.21.15.60:0/2701139976 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde4021790 con 0x7fddfc0b2130 2022-06-29T15:59:22.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.991+0000 7fde01333700 1 -- 172.21.15.60:0/2701139976 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddfc156950 con 0x7fddfc0b2130 2022-06-29T15:59:22.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.991+0000 7fddf97fa700 1 -- 172.21.15.60:0/2701139976 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdde4021af0 con 0x7fddfc0b2130 2022-06-29T15:59:22.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.991+0000 7fde01333700 1 -- 172.21.15.60:0/2701139976 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddfc066b20 con 0x7fddfc0b2130 2022-06-29T15:59:22.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.991+0000 7fde01333700 1 -- 172.21.15.60:0/2701139976 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddfc14a5f0 con 0x7fddfc0b2130 2022-06-29T15:59:22.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.991+0000 7fddf97fa700 1 -- 172.21.15.60:0/2701139976 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdde400f070 con 0x7fddfc0b2130 2022-06-29T15:59:22.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.995+0000 7fddf97fa700 1 --2- 172.21.15.60:0/2701139976 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdde0053a70 0x7fdde0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:22.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.995+0000 7fddf97fa700 1 -- 172.21.15.60:0/2701139976 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fdde406d650 con 0x7fddfc0b2130 2022-06-29T15:59:22.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.995+0000 7fddfbfff700 1 --2- 172.21.15.60:0/2701139976 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdde0053a70 0x7fdde0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:22.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.995+0000 7fddfbfff700 1 --2- 172.21.15.60:0/2701139976 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdde0053a70 0x7fdde0055f30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fddec00b000 tx=0x7fddec006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:22.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:21.999+0000 7fddf97fa700 1 -- 172.21.15.60:0/2701139976 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdde4044570 con 0x7fddfc0b2130 2022-06-29T15:59:22.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.167+0000 7fde01333700 1 -- 172.21.15.60:0/2701139976 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fddfc0573d0 con 0x7fddfc0b2130 2022-06-29T15:59:22.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.167+0000 7fddf97fa700 1 -- 172.21.15.60:0/2701139976 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+2134 (secure 0 0 0) 0x7fdde4015070 con 0x7fddfc0b2130 2022-06-29T15:59:22.176 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:59:22.177 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":9,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4884,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2555925680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2555925680},{"type":"v1","addr":"172.21.15.60:6833","nonce":2555925680}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":7}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T15:59:05.257113+0000","modified":"2022-06-29T15:59:06.259879+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4933},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4933":{"gid":4933,"name":"b","rank":0,"incarnation":8,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/1993861928","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1993861928},{"type":"v1","addr":"172.21.15.92:6835","nonce":1993861928}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-06-29T15:59:22.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.167+0000 7fde01333700 1 -- 172.21.15.60:0/2701139976 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdde0053a70 msgr2=0x7fdde0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:22.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.167+0000 7fde01333700 1 --2- 172.21.15.60:0/2701139976 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdde0053a70 0x7fdde0055f30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fddec00b000 tx=0x7fddec006df0).stop 2022-06-29T15:59:22.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.167+0000 7fde01333700 1 -- 172.21.15.60:0/2701139976 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddfc0b2130 msgr2=0x7fddfc116780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:22.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.167+0000 7fde01333700 1 --2- 172.21.15.60:0/2701139976 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddfc0b2130 0x7fddfc116780 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fdde40032f0 tx=0x7fdde4016000).stop 2022-06-29T15:59:22.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.167+0000 7fde01333700 1 -- 172.21.15.60:0/2701139976 shutdown_connections 2022-06-29T15:59:22.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.167+0000 7fde01333700 1 --2- 172.21.15.60:0/2701139976 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdde0053a70 0x7fdde0055f30 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.167+0000 7fde01333700 1 --2- 172.21.15.60:0/2701139976 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fddfc14af80 0x7fddfc066630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.167+0000 7fde01333700 1 --2- 172.21.15.60:0/2701139976 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fddfc0b2130 0x7fddfc116780 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.167+0000 7fde01333700 1 --2- 172.21.15.60:0/2701139976 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fddfc05d070 0x7fddfc156df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.167+0000 7fde01333700 1 -- 172.21.15.60:0/2701139976 >> 172.21.15.60:0/2701139976 conn(0x7fddfc0b5890 msgr2=0x7fddfc1470a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:22.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.171+0000 7fde01333700 1 -- 172.21.15.60:0/2701139976 shutdown_connections 2022-06-29T15:59:22.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.171+0000 7fde01333700 1 -- 172.21.15.60:0/2701139976 wait complete. 2022-06-29T15:59:22.182 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 9 2022-06-29T15:59:22.190 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json config set mds_recall_max_decay_threshold 400 2022-06-29T15:59:22.300 INFO:teuthology.orchestra.run.smithi092.stdout:{"success":"mds_recall_max_decay_threshold = '400' (not observed, change may require restart) "} 2022-06-29T15:59:22.301 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_decay_threshold = '400' (not observed, change may require restart) " } 2022-06-29T15:59:22.302 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T15:59:22.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.403+0000 7f5229df8700 1 Processor -- start 2022-06-29T15:59:22.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.403+0000 7f5229df8700 1 -- start start 2022-06-29T15:59:22.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.403+0000 7f5229df8700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5224066d20 0x7f5224140810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:22.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.403+0000 7f5229df8700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5224144ad0 0x7f522413d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:22.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.403+0000 7f5229df8700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52240b23a0 con 0x7f52240b3110 2022-06-29T15:59:22.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.403+0000 7f5229df8700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5224067100 con 0x7f5224144ad0 2022-06-29T15:59:22.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.403+0000 7f5229df8700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5224140d00 con 0x7f5224066d20 2022-06-29T15:59:22.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.403+0000 7f52295f7700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5224144ad0 0x7f522413d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:22.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.403+0000 7f52295f7700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5224144ad0 0x7f522413d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46660/0 (socket says 172.21.15.60:46660) 2022-06-29T15:59:22.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.403+0000 7f52295f7700 1 -- 172.21.15.60:0/1091466299 learned_addr learned my addr 172.21.15.60:0/1091466299 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:22.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.403+0000 7f5228df6700 1 --2- 172.21.15.60:0/1091466299 >> v2:172.21.15.92:3300/0 conn(0x7f5224066d20 0x7f5224140810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:22.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.407+0000 7f52295f7700 1 -- 172.21.15.60:0/1091466299 >> v2:172.21.15.92:3300/0 conn(0x7f5224066d20 msgr2=0x7f5224140810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:22.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.407+0000 7f52295f7700 1 --2- 172.21.15.60:0/1091466299 >> v2:172.21.15.92:3300/0 conn(0x7f5224066d20 0x7f5224140810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.407+0000 7f52295f7700 1 -- 172.21.15.60:0/1091466299 >> v1:172.21.15.92:6789/0 conn(0x7f52240b3110 legacy=0x7f52241443c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:22.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.407+0000 7f52295f7700 1 -- 172.21.15.60:0/1091466299 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52240b2100 con 0x7f5224144ad0 2022-06-29T15:59:22.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.407+0000 7f52295f7700 1 --2- 172.21.15.60:0/1091466299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5224144ad0 0x7f522413d8b0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f521c0032f0 tx=0x7f521c009290).ready entity=mon.0 client_cookie=50d3c153b9263e7 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:22.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.407+0000 7f52237fe700 1 -- 172.21.15.60:0/1091466299 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f521c00b870 con 0x7f5224144ad0 2022-06-29T15:59:22.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.407+0000 7f52237fe700 1 -- 172.21.15.60:0/1091466299 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521c00b9d0 con 0x7f5224144ad0 2022-06-29T15:59:22.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.407+0000 7f52237fe700 1 -- 172.21.15.60:0/1091466299 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f521c00bb40 con 0x7f5224144ad0 2022-06-29T15:59:22.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.407+0000 7f5229df8700 1 -- 172.21.15.60:0/1091466299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5224144ad0 msgr2=0x7f522413d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:22.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.407+0000 7f5229df8700 1 --2- 172.21.15.60:0/1091466299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5224144ad0 0x7f522413d8b0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f521c0032f0 tx=0x7f521c009290).stop 2022-06-29T15:59:22.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.407+0000 7f5229df8700 1 -- 172.21.15.60:0/1091466299 shutdown_connections 2022-06-29T15:59:22.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.407+0000 7f5229df8700 1 --2- 172.21.15.60:0/1091466299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5224144ad0 0x7f522413d8b0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.407+0000 7f5229df8700 1 --2- 172.21.15.60:0/1091466299 >> v2:172.21.15.92:3300/0 conn(0x7f5224066d20 0x7f5224140810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.407+0000 7f5229df8700 1 -- 172.21.15.60:0/1091466299 >> 172.21.15.60:0/1091466299 conn(0x7f5224061c80 msgr2=0x7f52240637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:22.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.407+0000 7f5229df8700 1 -- 172.21.15.60:0/1091466299 shutdown_connections 2022-06-29T15:59:22.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.407+0000 7f5229df8700 1 -- 172.21.15.60:0/1091466299 wait complete. 2022-06-29T15:59:22.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.407+0000 7f5229df8700 1 Processor -- start 2022-06-29T15:59:22.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.407+0000 7f5229df8700 1 -- start start 2022-06-29T15:59:22.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.411+0000 7f5229df8700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5224066d20 0x7f5224147670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:22.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.411+0000 7f5229df8700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52240b3110 0x7f522414faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:22.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.411+0000 7f5229df8700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5224144ad0 0x7f522413ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:22.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.411+0000 7f5229df8700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522413e9f0 con 0x7f5224066d20 2022-06-29T15:59:22.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.411+0000 7f5229df8700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522414ff90 con 0x7f52240b3110 2022-06-29T15:59:22.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.411+0000 7f5229df8700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52240ae0b0 con 0x7f5224144ad0 2022-06-29T15:59:22.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.411+0000 7f5228df6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5224066d20 0x7f5224147670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:22.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.411+0000 7f5228df6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5224066d20 0x7f5224147670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46662/0 (socket says 172.21.15.60:46662) 2022-06-29T15:59:22.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.411+0000 7f5228df6700 1 -- 172.21.15.60:0/2120175263 learned_addr learned my addr 172.21.15.60:0/2120175263 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:22.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.411+0000 7f5223fff700 1 --2- 172.21.15.60:0/2120175263 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52240b3110 0x7f522414faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:22.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.411+0000 7f52295f7700 1 --2- 172.21.15.60:0/2120175263 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5224144ad0 0x7f522413ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:22.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.411+0000 7f5228df6700 1 -- 172.21.15.60:0/2120175263 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5224144ad0 msgr2=0x7f522413ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:22.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.411+0000 7f5228df6700 1 --2- 172.21.15.60:0/2120175263 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5224144ad0 0x7f522413ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.411+0000 7f5228df6700 1 -- 172.21.15.60:0/2120175263 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52240b3110 msgr2=0x7f522414faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:22.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.411+0000 7f5228df6700 1 --2- 172.21.15.60:0/2120175263 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52240b3110 0x7f522414faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.411+0000 7f5228df6700 1 -- 172.21.15.60:0/2120175263 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521c009530 con 0x7f5224066d20 2022-06-29T15:59:22.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.411+0000 7f5228df6700 1 --2- 172.21.15.60:0/2120175263 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5224066d20 0x7f5224147670 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f521800c000 tx=0x7f5218006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:22.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.411+0000 7f5221ffb700 1 -- 172.21.15.60:0/2120175263 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5218010780 con 0x7f5224066d20 2022-06-29T15:59:22.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.411+0000 7f5229df8700 1 -- 172.21.15.60:0/2120175263 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52240ae350 con 0x7f5224066d20 2022-06-29T15:59:22.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.411+0000 7f5221ffb700 1 -- 172.21.15.60:0/2120175263 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52180108e0 con 0x7f5224066d20 2022-06-29T15:59:22.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.411+0000 7f5221ffb700 1 -- 172.21.15.60:0/2120175263 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5218010c40 con 0x7f5224066d20 2022-06-29T15:59:22.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.411+0000 7f5229df8700 1 -- 172.21.15.60:0/2120175263 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52240add30 con 0x7f5224066d20 2022-06-29T15:59:22.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.411+0000 7f5229df8700 1 -- 172.21.15.60:0/2120175263 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52240573d0 con 0x7f5224066d20 2022-06-29T15:59:22.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.411+0000 7f5221ffb700 1 -- 172.21.15.60:0/2120175263 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f521801c460 con 0x7f5224066d20 2022-06-29T15:59:22.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.415+0000 7f5221ffb700 1 --2- 172.21.15.60:0/2120175263 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5204053a70 0x7f5204055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:22.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.415+0000 7f5221ffb700 1 -- 172.21.15.60:0/2120175263 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f521806c430 con 0x7f5224066d20 2022-06-29T15:59:22.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.415+0000 7f5223fff700 1 --2- 172.21.15.60:0/2120175263 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5204053a70 0x7f5204055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:22.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.415+0000 7f5223fff700 1 --2- 172.21.15.60:0/2120175263 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5204053a70 0x7f5204055f30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f521000d000 tx=0x7f5210000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:22.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.419+0000 7f5221ffb700 1 -- 172.21.15.60:0/2120175263 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5218043160 con 0x7f5224066d20 2022-06-29T15:59:22.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.579+0000 7f5229df8700 1 -- 172.21.15.60:0/2120175263 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5224060a00 con 0x7f5224066d20 2022-06-29T15:59:22.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.583+0000 7f5221ffb700 1 -- 172.21.15.60:0/2120175263 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+2134 (secure 0 0 0) 0x7f5218046780 con 0x7f5224066d20 2022-06-29T15:59:22.593 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T15:59:22.593 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":9,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4884,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2555925680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2555925680},{"type":"v1","addr":"172.21.15.60:6833","nonce":2555925680}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":7}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T15:59:05.257113+0000","modified":"2022-06-29T15:59:06.259879+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4933},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4933":{"gid":4933,"name":"b","rank":0,"incarnation":8,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/1993861928","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1993861928},{"type":"v1","addr":"172.21.15.92:6835","nonce":1993861928}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-06-29T15:59:22.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.583+0000 7f5229df8700 1 -- 172.21.15.60:0/2120175263 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5204053a70 msgr2=0x7f5204055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:22.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.583+0000 7f5229df8700 1 --2- 172.21.15.60:0/2120175263 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5204053a70 0x7f5204055f30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f521000d000 tx=0x7f5210000bc0).stop 2022-06-29T15:59:22.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.583+0000 7f5229df8700 1 -- 172.21.15.60:0/2120175263 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5224066d20 msgr2=0x7f5224147670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:22.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.583+0000 7f5229df8700 1 --2- 172.21.15.60:0/2120175263 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5224066d20 0x7f5224147670 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f521800c000 tx=0x7f5218006df0).stop 2022-06-29T15:59:22.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.583+0000 7f5229df8700 1 -- 172.21.15.60:0/2120175263 shutdown_connections 2022-06-29T15:59:22.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.583+0000 7f5229df8700 1 --2- 172.21.15.60:0/2120175263 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5204053a70 0x7f5204055f30 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.583+0000 7f5229df8700 1 --2- 172.21.15.60:0/2120175263 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5224144ad0 0x7f522413ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.583+0000 7f5229df8700 1 --2- 172.21.15.60:0/2120175263 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f52240b3110 0x7f522414faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.583+0000 7f5229df8700 1 --2- 172.21.15.60:0/2120175263 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5224066d20 0x7f5224147670 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.583+0000 7f5229df8700 1 -- 172.21.15.60:0/2120175263 >> 172.21.15.60:0/2120175263 conn(0x7f5224061c80 msgr2=0x7f52240b1490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:22.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.583+0000 7f5229df8700 1 -- 172.21.15.60:0/2120175263 shutdown_connections 2022-06-29T15:59:22.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.583+0000 7f5229df8700 1 -- 172.21.15.60:0/2120175263 wait complete. 2022-06-29T15:59:22.600 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 9 2022-06-29T15:59:22.604 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b cache drop 2022-06-29T15:59:22.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.703+0000 7f488ab4a700 1 Processor -- start 2022-06-29T15:59:22.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.703+0000 7f488ab4a700 1 -- start start 2022-06-29T15:59:22.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.707+0000 7f488ab4a700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f48840b4010 0x7f48841426b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:22.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.707+0000 7f488ab4a700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4884143a40 con 0x7f48840b46c0 2022-06-29T15:59:22.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.707+0000 7f488ab4a700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48840aee60 con 0x7f48840ae7c0 2022-06-29T15:59:22.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.707+0000 7f488ab4a700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48840af100 con 0x7f48840b4010 2022-06-29T15:59:22.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.707+0000 7f4883fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f48840b4010 0x7f48841426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:22.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.707+0000 7f48890e7700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f48840b46c0 0x7f48840b4af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54760/0 (socket says 172.21.15.60:54760) 2022-06-29T15:59:22.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.707+0000 7f4883fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f48840b4010 0x7f48841426b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46672/0 (socket says 172.21.15.60:46672) 2022-06-29T15:59:22.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.707+0000 7f48890e7700 1 -- 172.21.15.60:0/3753396436 learned_addr learned my addr 172.21.15.60:0/3753396436 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T15:59:22.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.707+0000 7f4883fff700 1 -- 172.21.15.60:0/3753396436 >> v1:172.21.15.92:6789/0 conn(0x7f48840ae7c0 legacy=0x7f4884141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:22.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.707+0000 7f4883fff700 1 -- 172.21.15.60:0/3753396436 >> v1:172.21.15.60:6789/0 conn(0x7f48840b46c0 legacy=0x7f48840b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:22.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.707+0000 7f4883fff700 1 -- 172.21.15.60:0/3753396436 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48840aeba0 con 0x7f48840b4010 2022-06-29T15:59:22.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.707+0000 7f4883fff700 1 --2- 172.21.15.60:0/3753396436 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48840b4010 0x7f48841426b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f487c001490 tx=0x7f486c006df0).ready entity=mon.0 client_cookie=e08148420d5c6180 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:22.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.707+0000 7f48837fe700 1 -- 172.21.15.60:0/3753396436 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f486c00d5c0 con 0x7f48840b4010 2022-06-29T15:59:22.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.707+0000 7f48837fe700 1 -- 172.21.15.60:0/3753396436 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f486c000e90 con 0x7f48840b4010 2022-06-29T15:59:22.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.707+0000 7f48837fe700 1 -- 172.21.15.60:0/3753396436 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f486c00d910 con 0x7f48840b4010 2022-06-29T15:59:22.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.707+0000 7f488ab4a700 1 -- 172.21.15.60:0/3753396436 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48840b4010 msgr2=0x7f48841426b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:22.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.707+0000 7f488ab4a700 1 --2- 172.21.15.60:0/3753396436 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48840b4010 0x7f48841426b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f487c001490 tx=0x7f486c006df0).stop 2022-06-29T15:59:22.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.707+0000 7f488ab4a700 1 -- 172.21.15.60:0/3753396436 shutdown_connections 2022-06-29T15:59:22.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.707+0000 7f488ab4a700 1 --2- 172.21.15.60:0/3753396436 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48840b4010 0x7f48841426b0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.707+0000 7f488ab4a700 1 -- 172.21.15.60:0/3753396436 >> 172.21.15.60:0/3753396436 conn(0x7f48840b9340 msgr2=0x7f48840b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:22.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.707+0000 7f488ab4a700 1 -- 172.21.15.60:0/3753396436 shutdown_connections 2022-06-29T15:59:22.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.707+0000 7f488ab4a700 1 -- 172.21.15.60:0/3753396436 wait complete. 2022-06-29T15:59:22.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.711+0000 7f488ab4a700 1 Processor -- start 2022-06-29T15:59:22.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.711+0000 7f488ab4a700 1 -- start start 2022-06-29T15:59:22.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.711+0000 7f488ab4a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f48840ae7c0 0x7f4884147910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:22.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.711+0000 7f488ab4a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48840b4010 0x7f4884113480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:22.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.711+0000 7f488ab4a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48840b46c0 0x7f4884155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:22.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.711+0000 7f488ab4a700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48840b0c10 con 0x7f48840b46c0 2022-06-29T15:59:22.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.711+0000 7f488ab4a700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48840b06d0 con 0x7f48840ae7c0 2022-06-29T15:59:22.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.711+0000 7f488ab4a700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48840b0970 con 0x7f48840b4010 2022-06-29T15:59:22.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.711+0000 7f48890e7700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48840b46c0 0x7f4884155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:22.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.711+0000 7f48888e6700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f48840ae7c0 0x7f4884147910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:22.728 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.711+0000 7f4883fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48840b4010 0x7f4884113480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:22.728 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.711+0000 7f48888e6700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f48840ae7c0 0x7f4884147910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:52706/0 (socket says 172.21.15.60:52706) 2022-06-29T15:59:22.728 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.711+0000 7f48888e6700 1 -- 172.21.15.60:0/2180532531 learned_addr learned my addr 172.21.15.60:0/2180532531 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:22.728 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.711+0000 7f48890e7700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48840b46c0 0x7f4884155850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46680/0 (socket says 172.21.15.60:46680) 2022-06-29T15:59:22.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.711+0000 7f48890e7700 1 -- 172.21.15.60:0/2180532531 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48840b4010 msgr2=0x7f4884113480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:22.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.711+0000 7f48890e7700 1 --2- 172.21.15.60:0/2180532531 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48840b4010 0x7f4884113480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.711+0000 7f48890e7700 1 -- 172.21.15.60:0/2180532531 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f48840ae7c0 msgr2=0x7f4884147910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:22.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.711+0000 7f48890e7700 1 --2- 172.21.15.60:0/2180532531 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f48840ae7c0 0x7f4884147910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.711+0000 7f48890e7700 1 -- 172.21.15.60:0/2180532531 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f486c00b180 con 0x7f48840b46c0 2022-06-29T15:59:22.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.711+0000 7f4883fff700 1 --2- 172.21.15.60:0/2180532531 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48840b4010 0x7f4884113480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:59:22.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.711+0000 7f48888e6700 1 --2- 172.21.15.60:0/2180532531 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f48840ae7c0 0x7f4884147910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T15:59:22.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.711+0000 7f48890e7700 1 --2- 172.21.15.60:0/2180532531 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48840b46c0 0x7f4884155850 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f487c001360 tx=0x7f487c007e90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:22.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.711+0000 7f4881ffb700 1 -- 172.21.15.60:0/2180532531 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f487c005ba0 con 0x7f48840b46c0 2022-06-29T15:59:22.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.711+0000 7f4881ffb700 1 -- 172.21.15.60:0/2180532531 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f487c005d00 con 0x7f48840b46c0 2022-06-29T15:59:22.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.711+0000 7f4881ffb700 1 -- 172.21.15.60:0/2180532531 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f487c0072c0 con 0x7f48840b46c0 2022-06-29T15:59:22.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.711+0000 7f488ab4a700 1 -- 172.21.15.60:0/2180532531 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48840b0190 con 0x7f48840b46c0 2022-06-29T15:59:22.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.711+0000 7f488ab4a700 1 -- 172.21.15.60:0/2180532531 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4884152130 con 0x7f48840b46c0 2022-06-29T15:59:22.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.715+0000 7f4881ffb700 1 -- 172.21.15.60:0/2180532531 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f487c010070 con 0x7f48840b46c0 2022-06-29T15:59:22.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.715+0000 7f4881ffb700 1 --2- 172.21.15.60:0/2180532531 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4868053aa0 0x7f4868055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:22.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.715+0000 7f4881ffb700 1 -- 172.21.15.60:0/2180532531 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f487c06b930 con 0x7f48840b46c0 2022-06-29T15:59:22.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.715+0000 7f48888e6700 1 --2- 172.21.15.60:0/2180532531 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4868053aa0 0x7f4868055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:22.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.719+0000 7f48888e6700 1 --2- 172.21.15.60:0/2180532531 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4868053aa0 0x7f4868055f60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f487400d000 tx=0x7f4874009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:22.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.719+0000 7f4894d9f740 1 Processor -- start 2022-06-29T15:59:22.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.719+0000 7f4894d9f740 1 -- start start 2022-06-29T15:59:22.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.719+0000 7f4894d9f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0xf0f4b0 0xed4d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:22.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.719+0000 7f4894d9f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0xf0a6b0 0xed7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:22.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.719+0000 7f4894d9f740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0xf0b060 0xed0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:22.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.719+0000 7f4894d9f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0xe45bb0 con 0xf0f4b0 2022-06-29T15:59:22.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.719+0000 7f4894d9f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0xe26150 con 0xf0a6b0 2022-06-29T15:59:22.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.719+0000 7f4894d9f740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0xed20c0 con 0xf0b060 2022-06-29T15:59:22.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.719+0000 7f48888e6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0xf0f4b0 0xed4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:22.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.719+0000 7f48888e6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0xf0f4b0 0xed4d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46684/0 (socket says 172.21.15.60:46684) 2022-06-29T15:59:22.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.719+0000 7f48888e6700 1 -- 172.21.15.60:0/4221482955 learned_addr learned my addr 172.21.15.60:0/4221482955 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:22.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.719+0000 7f4883fff700 1 --2- 172.21.15.60:0/4221482955 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0xf0a6b0 0xed7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:22.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.719+0000 7f48890e7700 1 --2- 172.21.15.60:0/4221482955 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0xf0b060 0xed0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:22.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.719+0000 7f48888e6700 1 -- 172.21.15.60:0/4221482955 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0xf0b060 msgr2=0xed0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:22.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.719+0000 7f48888e6700 1 --2- 172.21.15.60:0/4221482955 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0xf0b060 0xed0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.719+0000 7f48888e6700 1 -- 172.21.15.60:0/4221482955 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0xf0a6b0 msgr2=0xed7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:22.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.719+0000 7f48888e6700 1 --2- 172.21.15.60:0/4221482955 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0xf0a6b0 0xed7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.719+0000 7f48888e6700 1 -- 172.21.15.60:0/4221482955 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0xe7d400 con 0xf0f4b0 2022-06-29T15:59:22.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.719+0000 7f48890e7700 1 --2- 172.21.15.60:0/4221482955 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0xf0b060 0xed0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:59:22.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.719+0000 7f48888e6700 1 --2- 172.21.15.60:0/4221482955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0xf0f4b0 0xed4d10 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0xe63f40 tx=0x7f4874013000).ready entity=mon.0 client_cookie=27ce8b8433198c49 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:22.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.719+0000 7f488ab4a700 1 -- 172.21.15.60:0/4221482955 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4874014710 con 0xf0f4b0 2022-06-29T15:59:22.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.719+0000 7f488ab4a700 1 -- 172.21.15.60:0/4221482955 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4874014870 con 0xf0f4b0 2022-06-29T15:59:22.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.719+0000 7f488ab4a700 1 -- 172.21.15.60:0/4221482955 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4874014bd0 con 0xf0f4b0 2022-06-29T15:59:22.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.719+0000 7f4894d9f740 1 -- 172.21.15.60:0/4221482955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0xf0f4b0 msgr2=0xed4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:22.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.719+0000 7f4894d9f740 1 --2- 172.21.15.60:0/4221482955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0xf0f4b0 0xed4d10 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0xe63f40 tx=0x7f4874013000).stop 2022-06-29T15:59:22.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.723+0000 7f4894d9f740 1 -- 172.21.15.60:0/4221482955 shutdown_connections 2022-06-29T15:59:22.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.723+0000 7f4894d9f740 1 --2- 172.21.15.60:0/4221482955 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0xf0b060 0xed0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.723+0000 7f4894d9f740 1 --2- 172.21.15.60:0/4221482955 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0xf0a6b0 0xed7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.723+0000 7f4894d9f740 1 --2- 172.21.15.60:0/4221482955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0xf0f4b0 0xed4d10 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.723+0000 7f4894d9f740 1 -- 172.21.15.60:0/4221482955 >> 172.21.15.60:0/4221482955 conn(0xe5d590 msgr2=0xf0ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:22.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.723+0000 7f4894d9f740 1 -- 172.21.15.60:0/4221482955 shutdown_connections 2022-06-29T15:59:22.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.723+0000 7f4894d9f740 1 -- 172.21.15.60:0/4221482955 wait complete. 2022-06-29T15:59:22.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.723+0000 7f4894d9f740 1 -- start start 2022-06-29T15:59:22.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.723+0000 7f4894d9f740 1 Processor -- start 2022-06-29T15:59:22.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.723+0000 7f4894d9f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0xf0a6b0 0xede360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:22.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.723+0000 7f4894d9f740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0xf0b060 0xf19ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:22.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.723+0000 7f4894d9f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0xf0f4b0 0xf1a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:22.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.723+0000 7f4894d9f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0xeda820 con 0xf0a6b0 2022-06-29T15:59:22.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.723+0000 7f4894d9f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0xf08da0 con 0xf0f4b0 2022-06-29T15:59:22.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.723+0000 7f4894d9f740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0xeda2e0 con 0xf0b060 2022-06-29T15:59:22.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.723+0000 7f48888e6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0xf0a6b0 0xede360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:22.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.723+0000 7f4894d9f740 10 client.0 ll_register_callbacks cb 0xe8c9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-06-29T15:59:22.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.723+0000 7f48888e6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0xf0a6b0 0xede360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46690/0 (socket says 172.21.15.60:46690) 2022-06-29T15:59:22.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.723+0000 7f48888e6700 1 -- 172.21.15.60:0/614936432 learned_addr learned my addr 172.21.15.60:0/614936432 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:22.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.723+0000 7f4883fff700 1 --2- 172.21.15.60:0/614936432 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0xf0b060 0xf19ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:22.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.723+0000 7f48890e7700 1 --2- 172.21.15.60:0/614936432 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0xf0f4b0 0xf1a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:22.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.723+0000 7f48888e6700 1 -- 172.21.15.60:0/614936432 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0xf0b060 msgr2=0xf19ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:22.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.723+0000 7f48888e6700 1 --2- 172.21.15.60:0/614936432 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0xf0b060 0xf19ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.723+0000 7f48888e6700 1 -- 172.21.15.60:0/614936432 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0xf0f4b0 msgr2=0xf1a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:22.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.723+0000 7f48888e6700 1 --2- 172.21.15.60:0/614936432 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0xf0f4b0 0xf1a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.723+0000 7f48888e6700 1 -- 172.21.15.60:0/614936432 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48840aee60 con 0xf0a6b0 2022-06-29T15:59:22.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.723+0000 7f4883fff700 1 --2- 172.21.15.60:0/614936432 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0xf0b060 0xf19ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T15:59:22.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.723+0000 7f48888e6700 1 --2- 172.21.15.60:0/614936432 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0xf0a6b0 0xede360 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0xe63f40 tx=0x7f4874017000).ready entity=mon.0 client_cookie=18d51b53ba963d68 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:22.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.723+0000 7f48637fe700 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2022-06-29T15:59:22.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.723+0000 7f48637fe700 1 -- 172.21.15.60:0/614936432 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4874022770 con 0xf0a6b0 2022-06-29T15:59:22.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.727+0000 7f48637fe700 1 -- 172.21.15.60:0/614936432 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f487402ac30 con 0xf0a6b0 2022-06-29T15:59:22.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.727+0000 7f4894d9f740 1 -- 172.21.15.60:0/614936432 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0xe3baa0 con 0xf0a6b0 2022-06-29T15:59:22.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.727+0000 7f48637fe700 1 -- 172.21.15.60:0/614936432 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7f487402adf0 con 0xf0a6b0 2022-06-29T15:59:22.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.727+0000 7f4894d9f740 10 client.5115 fetch_fsmap learned FSMap version 9 2022-06-29T15:59:22.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.727+0000 7f4894d9f740 1 -- 172.21.15.60:0/614936432 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0xe35380 con 0xf0a6b0 2022-06-29T15:59:22.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.727+0000 7f48637fe700 1 -- 172.21.15.60:0/614936432 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0xe3baa0 con 0xf0a6b0 2022-06-29T15:59:22.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.727+0000 7f48637fe700 1 -- 172.21.15.60:0/614936432 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 9) v1 ==== 1247+0+0 (secure 0 0 0) 0x7f4874020410 con 0xf0a6b0 2022-06-29T15:59:22.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.727+0000 7f4894d9f740 10 client.5115 fetch_fsmap finished waiting for FSMap version 9 2022-06-29T15:59:22.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.727+0000 7f4894d9f740 10 client.5115 resolve_mds: resolved ID 'b' to GID 4933 2022-06-29T15:59:22.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.727+0000 7f4894d9f740 20 client.5115 populate_metadata read hostname 'smithi060' 2022-06-29T15:59:22.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.727+0000 7f4894d9f740 1 --2- 172.21.15.60:0/614936432 >> [v2:172.21.15.92:6834/1993861928,v1:172.21.15.92:6835/1993861928] conn(0xf19740 0xf1d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:22.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.727+0000 7f4894d9f740 4 client.5115 mds_command: new command op to 4933 tid=0[{"prefix": "get_command_descriptions"}] 2022-06-29T15:59:22.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.727+0000 7f4894d9f740 1 -- 172.21.15.60:0/614936432 --> [v2:172.21.15.92:6834/1993861928,v1:172.21.15.92:6835/1993861928] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0xe33e10 con 0xf19740 2022-06-29T15:59:22.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.727+0000 7f4883fff700 1 --2- 172.21.15.60:0/614936432 >> [v2:172.21.15.92:6834/1993861928,v1:172.21.15.92:6835/1993861928] conn(0xf19740 0xf1d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:22.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.727+0000 7f48637fe700 10 client.5115 ms_handle_connect on v2:172.21.15.92:6834/1993861928 2022-06-29T15:59:22.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.727+0000 7f4883fff700 1 --2- 172.21.15.60:0/614936432 >> [v2:172.21.15.92:6834/1993861928,v1:172.21.15.92:6835/1993861928] conn(0xf19740 0xf1d4f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=cb6b7df64e8aac41 server_cookie=aab2952539f3c7fa in_seq=0 out_seq=0 2022-06-29T15:59:22.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.727+0000 7f48637fe700 1 -- 172.21.15.60:0/614936432 <== mds.0 v2:172.21.15.92:6834/1993861928 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17002 (crc 0 0 0) 0xe33e10 con 0xf19740 2022-06-29T15:59:22.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.727+0000 7f48637fe700 10 client.5115 handle_command_reply: tid=0 2022-06-29T15:59:22.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.727+0000 7f4894d9f740 1 client.5115 shutdown 2022-06-29T15:59:22.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.727+0000 7f4894d9f740 1 -- 172.21.15.60:0/614936432 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0xf0a6b0 msgr2=0xede360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:22.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.727+0000 7f4894d9f740 1 --2- 172.21.15.60:0/614936432 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0xf0a6b0 0xede360 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0xe63f40 tx=0x7f4874017000).stop 2022-06-29T15:59:22.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.727+0000 7f4894d9f740 1 -- 172.21.15.60:0/614936432 shutdown_connections 2022-06-29T15:59:22.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.727+0000 7f4894d9f740 1 --2- 172.21.15.60:0/614936432 >> [v2:172.21.15.92:6834/1993861928,v1:172.21.15.92:6835/1993861928] conn(0xf19740 0xf1d4f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.727+0000 7f4894d9f740 1 --2- 172.21.15.60:0/614936432 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0xf0f4b0 0xf1a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.727+0000 7f4894d9f740 1 --2- 172.21.15.60:0/614936432 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0xf0b060 0xf19ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.727+0000 7f48637fe700 0 client.5115 ms_handle_reset on v2:172.21.15.92:6834/1993861928 2022-06-29T15:59:22.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.727+0000 7f4894d9f740 1 --2- 172.21.15.60:0/614936432 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0xf0a6b0 0xede360 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.727+0000 7f4894d9f740 1 -- 172.21.15.60:0/614936432 >> 172.21.15.60:0/614936432 conn(0xe5d590 msgr2=0xf091c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:22.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.727+0000 7f4894d9f740 1 -- 172.21.15.60:0/614936432 shutdown_connections 2022-06-29T15:59:22.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.727+0000 7f4894d9f740 1 -- 172.21.15.60:0/614936432 wait complete. 2022-06-29T15:59:22.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.727+0000 7f4894d9f740 20 client.5115 trim_cache size 0 max 16384 2022-06-29T15:59:22.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.739+0000 7f4894d9f740 1 Processor -- start 2022-06-29T15:59:22.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.739+0000 7f4894d9f740 1 -- start start 2022-06-29T15:59:22.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.739+0000 7f4894d9f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0xf0a6b0 0xf091c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:22.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.739+0000 7f4894d9f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0xf0b060 0xf0b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:22.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.739+0000 7f4894d9f740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0xf0f4b0 0xf0b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:22.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.739+0000 7f4894d9f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0xed3400 con 0xf0a6b0 2022-06-29T15:59:22.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.739+0000 7f4894d9f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0xe3baa0 con 0xf0b060 2022-06-29T15:59:22.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.739+0000 7f4894d9f740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0xed0bb0 con 0xf0f4b0 2022-06-29T15:59:22.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.739+0000 7f48888e6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0xf0a6b0 0xf091c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:22.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.739+0000 7f48888e6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0xf0a6b0 0xf091c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46698/0 (socket says 172.21.15.60:46698) 2022-06-29T15:59:22.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.739+0000 7f48888e6700 1 -- 172.21.15.60:0/2785699063 learned_addr learned my addr 172.21.15.60:0/2785699063 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:22.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.739+0000 7f4883fff700 1 --2- 172.21.15.60:0/2785699063 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0xf0b060 0xf0b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:22.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.739+0000 7f48890e7700 1 --2- 172.21.15.60:0/2785699063 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0xf0f4b0 0xf0b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:22.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.739+0000 7f48888e6700 1 -- 172.21.15.60:0/2785699063 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0xf0f4b0 msgr2=0xf0b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:22.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.739+0000 7f48888e6700 1 --2- 172.21.15.60:0/2785699063 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0xf0f4b0 0xf0b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.739+0000 7f48888e6700 1 -- 172.21.15.60:0/2785699063 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0xf0b060 msgr2=0xf0b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:22.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.739+0000 7f48888e6700 1 --2- 172.21.15.60:0/2785699063 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0xf0b060 0xf0b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.739+0000 7f48888e6700 1 -- 172.21.15.60:0/2785699063 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0xe3dc80 con 0xf0a6b0 2022-06-29T15:59:22.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.739+0000 7f48888e6700 1 --2- 172.21.15.60:0/2785699063 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0xf0a6b0 0xf091c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0xe47130 tx=0x7f4874028000).ready entity=mon.0 client_cookie=c72e41673ed9f992 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:22.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.739+0000 7f4879ffb700 1 -- 172.21.15.60:0/2785699063 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4874014710 con 0xf0a6b0 2022-06-29T15:59:22.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.739+0000 7f4879ffb700 1 -- 172.21.15.60:0/2785699063 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4874022320 con 0xf0a6b0 2022-06-29T15:59:22.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.739+0000 7f4879ffb700 1 -- 172.21.15.60:0/2785699063 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4874007560 con 0xf0a6b0 2022-06-29T15:59:22.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4894d9f740 1 -- 172.21.15.60:0/2785699063 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0xf0a6b0 msgr2=0xf091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T15:59:22.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4894d9f740 1 --2- 172.21.15.60:0/2785699063 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0xf0a6b0 0xf091c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0xe47130 tx=0x7f4874028000).stop 2022-06-29T15:59:22.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4894d9f740 1 -- 172.21.15.60:0/2785699063 shutdown_connections 2022-06-29T15:59:22.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4894d9f740 1 --2- 172.21.15.60:0/2785699063 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0xf0f4b0 0xf0b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4894d9f740 1 --2- 172.21.15.60:0/2785699063 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0xf0b060 0xf0b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4894d9f740 1 --2- 172.21.15.60:0/2785699063 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0xf0a6b0 0xf091c0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4894d9f740 1 -- 172.21.15.60:0/2785699063 >> 172.21.15.60:0/2785699063 conn(0xe5d590 msgr2=0xf19be0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T15:59:22.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4894d9f740 1 -- 172.21.15.60:0/2785699063 shutdown_connections 2022-06-29T15:59:22.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4894d9f740 1 -- 172.21.15.60:0/2785699063 wait complete. 2022-06-29T15:59:22.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4894d9f740 1 -- start start 2022-06-29T15:59:22.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4894d9f740 1 Processor -- start 2022-06-29T15:59:22.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4894d9f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0xf0a6b0 0xf6bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:22.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4894d9f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0xf0b060 0xf6c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:22.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4894d9f740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0xf0f4b0 0xedb940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:22.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4894d9f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0xf094f0 con 0xf0b060 2022-06-29T15:59:22.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4894d9f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0xed6120 con 0xf0a6b0 2022-06-29T15:59:22.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4894d9f740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0xed52a0 con 0xf0f4b0 2022-06-29T15:59:22.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4894d9f740 10 client.0 ll_register_callbacks cb 0x7f48840ada50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-06-29T15:59:22.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4883fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0xf0b060 0xf6c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:22.777 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4883fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0xf0b060 0xf6c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46708/0 (socket says 172.21.15.60:46708) 2022-06-29T15:59:22.777 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4883fff700 1 -- 172.21.15.60:0/1031788470 learned_addr learned my addr 172.21.15.60:0/1031788470 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T15:59:22.777 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f48890e7700 1 --2- 172.21.15.60:0/1031788470 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0xf0f4b0 0xedb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:22.777 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f48888e6700 1 --2- 172.21.15.60:0/1031788470 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0xf0a6b0 0xf6bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:22.778 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4883fff700 1 -- 172.21.15.60:0/1031788470 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0xf0f4b0 msgr2=0xedb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:22.778 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4883fff700 1 --2- 172.21.15.60:0/1031788470 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0xf0f4b0 0xedb940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.778 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4883fff700 1 -- 172.21.15.60:0/1031788470 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0xf0a6b0 msgr2=0xf6bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T15:59:22.778 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4883fff700 1 --2- 172.21.15.60:0/1031788470 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0xf0a6b0 0xf6bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T15:59:22.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4883fff700 1 -- 172.21.15.60:0/1031788470 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0xe35380 con 0xf0b060 2022-06-29T15:59:22.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4878ff9700 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2022-06-29T15:59:22.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4883fff700 1 --2- 172.21.15.60:0/1031788470 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0xf0b060 0xf6c2a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f486c01f000 tx=0x7f486c028280).ready entity=mon.0 client_cookie=a8dc792721bcb4ac server_cookie=0 in_seq=0 out_seq=0 2022-06-29T15:59:22.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4878ff9700 1 -- 172.21.15.60:0/1031788470 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f486c026070 con 0xf0b060 2022-06-29T15:59:22.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4878ff9700 1 -- 172.21.15.60:0/1031788470 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f486c02cc70 con 0xf0b060 2022-06-29T15:59:22.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4894d9f740 1 -- 172.21.15.60:0/1031788470 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4874014870 con 0xf0b060 2022-06-29T15:59:22.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4878ff9700 1 -- 172.21.15.60:0/1031788470 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7f486c016040 con 0xf0b060 2022-06-29T15:59:22.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4894d9f740 10 client.5121 fetch_fsmap learned FSMap version 9 2022-06-29T15:59:22.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.743+0000 7f4894d9f740 1 -- 172.21.15.60:0/1031788470 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0xe3a670 con 0xf0b060 2022-06-29T15:59:22.781 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.747+0000 7f4878ff9700 1 -- 172.21.15.60:0/1031788470 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4874014870 con 0xf0b060 2022-06-29T15:59:22.781 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.747+0000 7f4878ff9700 1 -- 172.21.15.60:0/1031788470 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 9) v1 ==== 1247+0+0 (secure 0 0 0) 0x7f486c012bc0 con 0xf0b060 2022-06-29T15:59:22.781 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.747+0000 7f4894d9f740 10 client.5121 fetch_fsmap finished waiting for FSMap version 9 2022-06-29T15:59:22.781 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.747+0000 7f4894d9f740 10 client.5121 resolve_mds: resolved ID 'b' to GID 4933 2022-06-29T15:59:22.781 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.747+0000 7f4894d9f740 20 client.5121 populate_metadata read hostname 'smithi060' 2022-06-29T15:59:22.782 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.747+0000 7f4894d9f740 1 --2- 172.21.15.60:0/1031788470 >> [v2:172.21.15.92:6834/1993861928,v1:172.21.15.92:6835/1993861928] conn(0xf17b60 0xedbe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T15:59:22.782 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.747+0000 7f4894d9f740 4 client.5121 mds_command: new command op to 4933 tid=0[{"prefix": "cache drop"}] 2022-06-29T15:59:22.782 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.747+0000 7f4894d9f740 1 -- 172.21.15.60:0/1031788470 --> [v2:172.21.15.92:6834/1993861928,v1:172.21.15.92:6835/1993861928] -- command(tid 0: {"prefix": "cache drop"}) v1 -- 0xd29590 con 0xf17b60 2022-06-29T15:59:22.782 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.747+0000 7f48888e6700 1 --2- 172.21.15.60:0/1031788470 >> [v2:172.21.15.92:6834/1993861928,v1:172.21.15.92:6835/1993861928] conn(0xf17b60 0xedbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T15:59:22.783 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.747+0000 7f48888e6700 1 --2- 172.21.15.60:0/1031788470 >> [v2:172.21.15.92:6834/1993861928,v1:172.21.15.92:6835/1993861928] conn(0xf17b60 0xedbe30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=43fddd4a8fa06360 server_cookie=e31a3a1f1156650c in_seq=0 out_seq=0 2022-06-29T15:59:22.783 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T15:59:22.747+0000 7f4878ff9700 10 client.5121 ms_handle_connect on v2:172.21.15.92:6834/1993861928 2022-06-29T15:59:28.104 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T15:59:28.107 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T15:59:28.110 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T15:59:28.113 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T15:59:28.115 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T15:59:58.166 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T15:59:58.170 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T15:59:58.173 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T15:59:58.176 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T15:59:58.179 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:00:07.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.103+0000 7f4878ff9700 1 -- 172.21.15.60:0/1031788470 <== mds.0 v2:172.21.15.92:6834/1993861928 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+386 (crc 0 0 0) 0xd29590 con 0xf17b60 2022-06-29T16:00:07.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.103+0000 7f4878ff9700 10 client.5121 handle_command_reply: tid=0 2022-06-29T16:00:07.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.103+0000 7f4894d9f740 1 client.5121 shutdown 2022-06-29T16:00:07.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.103+0000 7f4894d9f740 1 -- 172.21.15.60:0/1031788470 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0xf0b060 msgr2=0xf6c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:07.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.103+0000 7f4894d9f740 1 --2- 172.21.15.60:0/1031788470 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0xf0b060 0xf6c2a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f486c01f000 tx=0x7f486c028280).stop 2022-06-29T16:00:07.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.103+0000 7f4894d9f740 1 -- 172.21.15.60:0/1031788470 shutdown_connections 2022-06-29T16:00:07.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.103+0000 7f4894d9f740 1 --2- 172.21.15.60:0/1031788470 >> [v2:172.21.15.92:6834/1993861928,v1:172.21.15.92:6835/1993861928] conn(0xf17b60 0xedbe30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:07.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.107+0000 7f4894d9f740 1 --2- 172.21.15.60:0/1031788470 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0xf0f4b0 0xedb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:07.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.107+0000 7f4894d9f740 1 --2- 172.21.15.60:0/1031788470 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0xf0b060 0xf6c2a0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:07.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.107+0000 7f4878ff9700 0 client.5121 ms_handle_reset on v2:172.21.15.92:6834/1993861928 2022-06-29T16:00:07.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.107+0000 7f4894d9f740 1 --2- 172.21.15.60:0/1031788470 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0xf0a6b0 0xf6bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:07.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.107+0000 7f4894d9f740 1 -- 172.21.15.60:0/1031788470 >> 172.21.15.60:0/1031788470 conn(0xe5d590 msgr2=0xf18f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:07.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.107+0000 7f4894d9f740 1 -- 172.21.15.60:0/1031788470 shutdown_connections 2022-06-29T16:00:07.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.107+0000 7f4894d9f740 1 -- 172.21.15.60:0/1031788470 wait complete. 2022-06-29T16:00:07.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.107+0000 7f4894d9f740 20 client.5121 trim_cache size 0 max 16384 2022-06-29T16:00:07.118 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:00:07.119 INFO:teuthology.orchestra.run.smithi060.stdout: "client_recall": { 2022-06-29T16:00:07.119 INFO:teuthology.orchestra.run.smithi060.stdout: "return_code": 0, 2022-06-29T16:00:07.120 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "(0) Success", 2022-06-29T16:00:07.120 INFO:teuthology.orchestra.run.smithi060.stdout: "recalled": 920 2022-06-29T16:00:07.121 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.121 INFO:teuthology.orchestra.run.smithi060.stdout: "flush_journal": { 2022-06-29T16:00:07.122 INFO:teuthology.orchestra.run.smithi060.stdout: "return_code": 0, 2022-06-29T16:00:07.122 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "" 2022-06-29T16:00:07.122 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.122 INFO:teuthology.orchestra.run.smithi060.stdout: "trim_cache": { 2022-06-29T16:00:07.123 INFO:teuthology.orchestra.run.smithi060.stdout: "trimmed": 900 2022-06-29T16:00:07.123 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.123 INFO:teuthology.orchestra.run.smithi060.stdout: "cache": { 2022-06-29T16:00:07.123 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": { 2022-06-29T16:00:07.124 INFO:teuthology.orchestra.run.smithi060.stdout: "items": 2517, 2022-06-29T16:00:07.124 INFO:teuthology.orchestra.run.smithi060.stdout: "bytes": 431856 2022-06-29T16:00:07.124 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:07.125 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.125 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 44.358246850999997 2022-06-29T16:00:07.125 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:00:07.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.107+0000 7f488ab4a700 1 -- 172.21.15.60:0/2180532531 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4868053aa0 msgr2=0x7f4868055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:07.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.107+0000 7f488ab4a700 1 --2- 172.21.15.60:0/2180532531 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4868053aa0 0x7f4868055f60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f487400d000 tx=0x7f4874009040).stop 2022-06-29T16:00:07.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.107+0000 7f488ab4a700 1 -- 172.21.15.60:0/2180532531 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48840b46c0 msgr2=0x7f4884155850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:07.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.107+0000 7f488ab4a700 1 --2- 172.21.15.60:0/2180532531 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48840b46c0 0x7f4884155850 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f487c001360 tx=0x7f487c007e90).stop 2022-06-29T16:00:07.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.107+0000 7f488ab4a700 1 -- 172.21.15.60:0/2180532531 shutdown_connections 2022-06-29T16:00:07.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.107+0000 7f488ab4a700 1 --2- 172.21.15.60:0/2180532531 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4868053aa0 0x7f4868055f60 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:07.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.107+0000 7f488ab4a700 1 --2- 172.21.15.60:0/2180532531 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48840b46c0 0x7f4884155850 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:07.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.107+0000 7f488ab4a700 1 --2- 172.21.15.60:0/2180532531 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48840b4010 0x7f4884113480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:07.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.107+0000 7f488ab4a700 1 --2- 172.21.15.60:0/2180532531 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f48840ae7c0 0x7f4884147910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:07.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.107+0000 7f488ab4a700 1 -- 172.21.15.60:0/2180532531 >> 172.21.15.60:0/2180532531 conn(0x7f48840b9340 msgr2=0x7f48840b4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:07.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.107+0000 7f488ab4a700 1 -- 172.21.15.60:0/2180532531 shutdown_connections 2022-06-29T16:00:07.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.107+0000 7f488ab4a700 1 -- 172.21.15.60:0/2180532531 wait complete. 2022-06-29T16:00:07.133 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-06-29T16:00:07.134 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:00:07.155 DEBUG:teuthology.orchestra.run.smithi092:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:00:07.176 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:00:07.199 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:00:07.221 DEBUG:teuthology.orchestra.run.smithi197:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:00:07.243 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-06-29T16:00:07.244 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:00:07.244 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:00:07.245 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:00:07.254 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T16:00:07.256 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T16:00:07.257 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2022-06-29T16:00:07.257 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-06-29T16:00:07.258 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:00:07.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[12335]: fuse finished with error 0 and tester_r 0 2022-06-29T16:00:07.315 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:00:07.316 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:00:07.316 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:00:07.323 INFO:teuthology.orchestra.run.smithi121.stdout:ext2/ext3 2022-06-29T16:00:07.324 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-06-29T16:00:07.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:00:07.442 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:00:07.493 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:00:07.494 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:00:07.494 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:00:07.501 INFO:teuthology.orchestra.run.smithi156.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-06-29T16:00:07.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:00:07.503 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-06-29T16:00:07.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:00:07.505 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:00:07.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:00:07.513 INFO:teuthology.orchestra.run.smithi156.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-06-29T16:00:07.513 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:00:07.514 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:00:07.515 DEBUG:teuthology.orchestra.run.smithi197:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T16:00:07.522 INFO:teuthology.orchestra.run.smithi197.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-06-29T16:00:07.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:00:07.524 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-06-29T16:00:07.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:00:07.525 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T16:00:07.531 INFO:teuthology.orchestra.run.smithi197.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-06-29T16:00:07.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:00:07.533 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-06-29T16:00:07.646 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.635+0000 7fd188b4c700 1 Processor -- start 2022-06-29T16:00:07.646 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.635+0000 7fd188b4c700 1 -- start start 2022-06-29T16:00:07.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.639+0000 7fd188b4c700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd1840b9d70 0x7fd1840b3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:07.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.639+0000 7fd188b4c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd1840b3cc0 0x7fd1841420b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:07.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.639+0000 7fd188b4c700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1840b2ce0 con 0x7fd18405d1f0 2022-06-29T16:00:07.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.639+0000 7fd188b4c700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1840b4d10 con 0x7fd1840b3cc0 2022-06-29T16:00:07.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.639+0000 7fd188b4c700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1840ba150 con 0x7fd1840b9d70 2022-06-29T16:00:07.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.639+0000 7fd182d9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd1840b3cc0 0x7fd1841420b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:07.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.639+0000 7fd182d9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd1840b3cc0 0x7fd1841420b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46716/0 (socket says 172.21.15.60:46716) 2022-06-29T16:00:07.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.639+0000 7fd182d9d700 1 -- 172.21.15.60:0/2899707695 learned_addr learned my addr 172.21.15.60:0/2899707695 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:07.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.639+0000 7fd18259c700 1 --2- 172.21.15.60:0/2899707695 >> v2:172.21.15.92:3300/0 conn(0x7fd1840b9d70 0x7fd1840b3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:07.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.639+0000 7fd182d9d700 1 -- 172.21.15.60:0/2899707695 >> v2:172.21.15.92:3300/0 conn(0x7fd1840b9d70 msgr2=0x7fd1840b3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:07.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.639+0000 7fd182d9d700 1 --2- 172.21.15.60:0/2899707695 >> v2:172.21.15.92:3300/0 conn(0x7fd1840b9d70 0x7fd1840b3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:07.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.639+0000 7fd182d9d700 1 -- 172.21.15.60:0/2899707695 >> v1:172.21.15.92:6789/0 conn(0x7fd18405d1f0 legacy=0x7fd1840b1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:07.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.639+0000 7fd182d9d700 1 -- 172.21.15.60:0/2899707695 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1840b4a70 con 0x7fd1840b3cc0 2022-06-29T16:00:07.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.639+0000 7fd182d9d700 1 --2- 172.21.15.60:0/2899707695 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1840b3cc0 0x7fd1841420b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fd174002f70 tx=0x7fd174006eb0).ready entity=mon.0 client_cookie=836163104fda24ee server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:07.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.639+0000 7fd18159a700 1 -- 172.21.15.60:0/2899707695 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd17400d610 con 0x7fd1840b3cc0 2022-06-29T16:00:07.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.639+0000 7fd18159a700 1 -- 172.21.15.60:0/2899707695 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd174000e90 con 0x7fd1840b3cc0 2022-06-29T16:00:07.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.639+0000 7fd18159a700 1 -- 172.21.15.60:0/2899707695 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd17400d9a0 con 0x7fd1840b3cc0 2022-06-29T16:00:07.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.639+0000 7fd188b4c700 1 -- 172.21.15.60:0/2899707695 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1840b3cc0 msgr2=0x7fd1841420b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:07.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.639+0000 7fd188b4c700 1 --2- 172.21.15.60:0/2899707695 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1840b3cc0 0x7fd1841420b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fd174002f70 tx=0x7fd174006eb0).stop 2022-06-29T16:00:07.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.639+0000 7fd188b4c700 1 -- 172.21.15.60:0/2899707695 shutdown_connections 2022-06-29T16:00:07.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.639+0000 7fd188b4c700 1 --2- 172.21.15.60:0/2899707695 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1840b3cc0 0x7fd1841420b0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:07.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.639+0000 7fd188b4c700 1 --2- 172.21.15.60:0/2899707695 >> v2:172.21.15.92:3300/0 conn(0x7fd1840b9d70 0x7fd1840b3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:07.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.639+0000 7fd188b4c700 1 -- 172.21.15.60:0/2899707695 >> 172.21.15.60:0/2899707695 conn(0x7fd184062ce0 msgr2=0x7fd1840b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:07.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.639+0000 7fd188b4c700 1 -- 172.21.15.60:0/2899707695 shutdown_connections 2022-06-29T16:00:07.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.639+0000 7fd188b4c700 1 -- 172.21.15.60:0/2899707695 wait complete. 2022-06-29T16:00:07.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.643+0000 7fd188b4c700 1 Processor -- start 2022-06-29T16:00:07.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.643+0000 7fd188b4c700 1 -- start start 2022-06-29T16:00:07.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.643+0000 7fd188b4c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd18405d1f0 0x7fd1841369e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:07.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.643+0000 7fd188b4c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1840b3cc0 0x7fd1840acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:07.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.643+0000 7fd188b4c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd1840b9d70 0x7fd18413ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:07.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.643+0000 7fd188b4c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1840adaf0 con 0x7fd1840b3cc0 2022-06-29T16:00:07.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.643+0000 7fd188b4c700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1840add90 con 0x7fd18405d1f0 2022-06-29T16:00:07.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.643+0000 7fd188b4c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1840ae030 con 0x7fd1840b9d70 2022-06-29T16:00:07.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.643+0000 7fd181d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1840b3cc0 0x7fd1840acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:07.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.643+0000 7fd18259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd18405d1f0 0x7fd1841369e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:07.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.643+0000 7fd181d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1840b3cc0 0x7fd1840acdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46720/0 (socket says 172.21.15.60:46720) 2022-06-29T16:00:07.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.643+0000 7fd181d9b700 1 -- 172.21.15.60:0/3903722206 learned_addr learned my addr 172.21.15.60:0/3903722206 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:07.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.643+0000 7fd182d9d700 1 --2- 172.21.15.60:0/3903722206 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd1840b9d70 0x7fd18413ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:07.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.643+0000 7fd181d9b700 1 -- 172.21.15.60:0/3903722206 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd1840b9d70 msgr2=0x7fd18413ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:07.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.643+0000 7fd181d9b700 1 --2- 172.21.15.60:0/3903722206 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd1840b9d70 0x7fd18413ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:07.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.643+0000 7fd181d9b700 1 -- 172.21.15.60:0/3903722206 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd18405d1f0 msgr2=0x7fd1841369e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:07.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.643+0000 7fd181d9b700 1 --2- 172.21.15.60:0/3903722206 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd18405d1f0 0x7fd1841369e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:07.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.643+0000 7fd181d9b700 1 -- 172.21.15.60:0/3903722206 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17400b1e0 con 0x7fd1840b3cc0 2022-06-29T16:00:07.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.647+0000 7fd181d9b700 1 --2- 172.21.15.60:0/3903722206 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1840b3cc0 0x7fd1840acdc0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fd184060000 tx=0x7fd16c007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:07.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.647+0000 7fd17b7fe700 1 -- 172.21.15.60:0/3903722206 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd16c007e70 con 0x7fd1840b3cc0 2022-06-29T16:00:07.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.647+0000 7fd17b7fe700 1 -- 172.21.15.60:0/3903722206 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd16c007030 con 0x7fd1840b3cc0 2022-06-29T16:00:07.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.647+0000 7fd17b7fe700 1 -- 172.21.15.60:0/3903722206 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd16c007390 con 0x7fd1840b3cc0 2022-06-29T16:00:07.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.647+0000 7fd188b4c700 1 -- 172.21.15.60:0/3903722206 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd184146be0 con 0x7fd1840b3cc0 2022-06-29T16:00:07.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.647+0000 7fd188b4c700 1 -- 172.21.15.60:0/3903722206 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1840ae6f0 con 0x7fd1840b3cc0 2022-06-29T16:00:07.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.647+0000 7fd188b4c700 1 -- 172.21.15.60:0/3903722206 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1840ad920 con 0x7fd1840b3cc0 2022-06-29T16:00:07.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.647+0000 7fd17b7fe700 1 -- 172.21.15.60:0/3903722206 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd16c010030 con 0x7fd1840b3cc0 2022-06-29T16:00:07.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.647+0000 7fd17b7fe700 1 --2- 172.21.15.60:0/3903722206 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd168053b30 0x7fd168055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:07.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.647+0000 7fd17b7fe700 1 -- 172.21.15.60:0/3903722206 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd16c06b8e0 con 0x7fd1840b3cc0 2022-06-29T16:00:07.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.647+0000 7fd18259c700 1 --2- 172.21.15.60:0/3903722206 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd168053b30 0x7fd168055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:07.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.651+0000 7fd18259c700 1 --2- 172.21.15.60:0/3903722206 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd168053b30 0x7fd168055ff0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fd17c00d000 tx=0x7fd17c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:07.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.655+0000 7fd17b7fe700 1 -- 172.21.15.60:0/3903722206 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd16c042800 con 0x7fd1840b3cc0 2022-06-29T16:00:07.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.783+0000 7fd188b4c700 1 -- 172.21.15.60:0/3903722206 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fd1840573d0 con 0x7fd1840b3cc0 2022-06-29T16:00:07.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.787+0000 7fd17b7fe700 1 -- 172.21.15.60:0/3903722206 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v32) v1 ==== 81+0+33645 (secure 0 0 0) 0x7fd16c045e20 con 0x7fd1840b3cc0 2022-06-29T16:00:07.796 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:00:07.797 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:00:07.797 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 32, 2022-06-29T16:00:07.798 INFO:teuthology.orchestra.run.smithi060.stdout: "fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", 2022-06-29T16:00:07.798 INFO:teuthology.orchestra.run.smithi060.stdout: "created": "2022-06-29T15:58:07.965719+0000", 2022-06-29T16:00:07.798 INFO:teuthology.orchestra.run.smithi060.stdout: "modified": "2022-06-29T15:59:06.260590+0000", 2022-06-29T16:00:07.799 INFO:teuthology.orchestra.run.smithi060.stdout: "last_up_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:00:07.799 INFO:teuthology.orchestra.run.smithi060.stdout: "last_in_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:00:07.799 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-06-29T16:00:07.800 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_num": 5799936, 2022-06-29T16:00:07.800 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_set": [ 2022-06-29T16:00:07.801 INFO:teuthology.orchestra.run.smithi060.stdout: "pglog_hardlimit", 2022-06-29T16:00:07.801 INFO:teuthology.orchestra.run.smithi060.stdout: "purged_snapdirs", 2022-06-29T16:00:07.801 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_deletes", 2022-06-29T16:00:07.802 INFO:teuthology.orchestra.run.smithi060.stdout: "sortbitwise" 2022-06-29T16:00:07.802 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:00:07.802 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_version": 4, 2022-06-29T16:00:07.802 INFO:teuthology.orchestra.run.smithi060.stdout: "full_ratio": 0.94999998807907104, 2022-06-29T16:00:07.803 INFO:teuthology.orchestra.run.smithi060.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-06-29T16:00:07.803 INFO:teuthology.orchestra.run.smithi060.stdout: "nearfull_ratio": 0.85000002384185791, 2022-06-29T16:00:07.803 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_snapshot": "", 2022-06-29T16:00:07.803 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_max": 6, 2022-06-29T16:00:07.804 INFO:teuthology.orchestra.run.smithi060.stdout: "max_osd": 8, 2022-06-29T16:00:07.804 INFO:teuthology.orchestra.run.smithi060.stdout: "require_min_compat_client": "jewel", 2022-06-29T16:00:07.804 INFO:teuthology.orchestra.run.smithi060.stdout: "min_compat_client": "jewel", 2022-06-29T16:00:07.804 INFO:teuthology.orchestra.run.smithi060.stdout: "require_osd_release": "octopus", 2022-06-29T16:00:07.805 INFO:teuthology.orchestra.run.smithi060.stdout: "pools": [ 2022-06-29T16:00:07.805 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.805 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2022-06-29T16:00:07.806 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "device_health_metrics", 2022-06-29T16:00:07.806 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:10.961127+0000", 2022-06-29T16:00:07.806 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:00:07.806 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:00:07.807 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:00:07.807 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:00:07.807 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:00:07.807 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:00:07.808 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:00:07.808 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:00:07.808 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 1, 2022-06-29T16:00:07.808 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 1, 2022-06-29T16:00:07.809 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 1, 2022-06-29T16:00:07.809 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 1, 2022-06-29T16:00:07.809 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 1, 2022-06-29T16:00:07.809 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:00:07.809 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:00:07.810 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:00:07.810 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:00:07.810 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:00:07.810 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:00:07.811 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:00:07.811 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.811 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "15", 2022-06-29T16:00:07.811 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:00:07.812 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:00:07.812 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:00:07.812 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:00:07.812 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:00:07.813 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:00:07.813 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:00:07.813 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:00:07.813 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:00:07.814 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:00:07.814 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:00:07.814 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:00:07.814 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:00:07.815 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:00:07.815 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:00:07.815 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:00:07.815 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:00:07.815 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:00:07.816 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:00:07.816 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:00:07.816 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:00:07.816 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:00:07.817 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:00:07.817 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:00:07.817 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:00:07.817 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:00:07.818 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.818 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:00:07.818 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:00:07.818 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:00:07.819 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:00:07.819 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:00:07.819 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:00:07.819 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:00:07.820 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:00:07.820 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:00:07.820 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:00:07.820 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:00:07.820 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:00:07.821 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 1 2022-06-29T16:00:07.821 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.821 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:00:07.821 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_devicehealth": {} 2022-06-29T16:00:07.822 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:07.822 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.822 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.823 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 2, 2022-06-29T16:00:07.823 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "rbd", 2022-06-29T16:00:07.823 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:19.184276+0000", 2022-06-29T16:00:07.823 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:00:07.824 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:00:07.824 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:00:07.824 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:00:07.824 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:00:07.825 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:00:07.825 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:00:07.825 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:00:07.825 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:00:07.826 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:00:07.826 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:00:07.826 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:00:07.826 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:00:07.827 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:00:07.827 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:00:07.827 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:00:07.827 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:00:07.828 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:00:07.828 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:00:07.828 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:00:07.828 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.829 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "18", 2022-06-29T16:00:07.829 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:00:07.829 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:00:07.829 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:00:07.829 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:00:07.830 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:00:07.830 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:00:07.830 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:00:07.831 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:00:07.831 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:00:07.831 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:00:07.831 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:00:07.831 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:00:07.832 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:00:07.832 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:00:07.832 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:00:07.832 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:00:07.833 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:00:07.833 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:00:07.833 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:00:07.833 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:00:07.834 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:00:07.834 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:00:07.834 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:00:07.834 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:00:07.835 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:00:07.835 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:00:07.835 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.835 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:00:07.836 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:00:07.836 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:00:07.836 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:00:07.836 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:00:07.837 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:00:07.837 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:00:07.837 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:00:07.837 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:00:07.838 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:00:07.838 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:00:07.838 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:00:07.839 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:00:07.839 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd": {} 2022-06-29T16:00:07.839 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:07.840 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.840 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.840 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 5, 2022-06-29T16:00:07.841 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "cephfs_metadata", 2022-06-29T16:00:07.841 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:59:03.821123+0000", 2022-06-29T16:00:07.842 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:00:07.842 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:00:07.842 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:00:07.843 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:00:07.843 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:00:07.843 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:00:07.844 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:00:07.844 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:00:07.844 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:00:07.845 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:00:07.845 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:00:07.846 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:00:07.846 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:00:07.846 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:00:07.847 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:00:07.847 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:00:07.847 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:00:07.848 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:00:07.848 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:00:07.849 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:00:07.849 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.849 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "31", 2022-06-29T16:00:07.850 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:00:07.850 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:00:07.851 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:00:07.851 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:00:07.851 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:00:07.852 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:00:07.852 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:00:07.852 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:00:07.853 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:00:07.853 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:00:07.854 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:00:07.854 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:00:07.854 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:00:07.855 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:00:07.855 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:00:07.855 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:00:07.856 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:00:07.856 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:00:07.856 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:00:07.856 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:00:07.856 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:00:07.857 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:00:07.857 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:00:07.857 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:00:07.857 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:00:07.858 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:00:07.858 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.858 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:00:07.858 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:00:07.859 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:00:07.859 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:00:07.859 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:00:07.859 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:00:07.860 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:00:07.860 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:00:07.860 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:00:07.860 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:00:07.860 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:00:07.861 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:00:07.861 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_bias": 4, 2022-06-29T16:00:07.861 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 16, 2022-06-29T16:00:07.861 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_priority": 5 2022-06-29T16:00:07.862 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.862 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:00:07.862 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs": { 2022-06-29T16:00:07.862 INFO:teuthology.orchestra.run.smithi060.stdout: "metadata": "cephfs" 2022-06-29T16:00:07.863 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:07.863 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:07.863 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.863 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.863 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 6, 2022-06-29T16:00:07.864 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "cephfs_data", 2022-06-29T16:00:07.864 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:59:04.275915+0000", 2022-06-29T16:00:07.864 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:00:07.864 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:00:07.865 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:00:07.865 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:00:07.865 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:00:07.865 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:00:07.866 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:00:07.866 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:00:07.866 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:00:07.866 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:00:07.866 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:00:07.867 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:00:07.867 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:00:07.867 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:00:07.867 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:00:07.868 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:00:07.868 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:00:07.868 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:00:07.868 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:00:07.868 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:00:07.869 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.869 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "32", 2022-06-29T16:00:07.869 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:00:07.869 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:00:07.870 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:00:07.870 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:00:07.870 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:00:07.870 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:00:07.871 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:00:07.871 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:00:07.871 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:00:07.871 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:00:07.872 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:00:07.872 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:00:07.872 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:00:07.872 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:00:07.872 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:00:07.873 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:00:07.873 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:00:07.873 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:00:07.873 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:00:07.874 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:00:07.874 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:00:07.874 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:00:07.874 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:00:07.875 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:00:07.875 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:00:07.875 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:00:07.875 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.875 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:00:07.876 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:00:07.876 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:00:07.876 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:00:07.876 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:00:07.877 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:00:07.877 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:00:07.877 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:00:07.877 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:00:07.878 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:00:07.878 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:00:07.878 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:00:07.878 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:00:07.879 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs": { 2022-06-29T16:00:07.879 INFO:teuthology.orchestra.run.smithi060.stdout: "data": "cephfs" 2022-06-29T16:00:07.879 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:07.879 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:07.880 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:07.880 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:00:07.880 INFO:teuthology.orchestra.run.smithi060.stdout: "osds": [ 2022-06-29T16:00:07.880 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.881 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:00:07.881 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "11e7ab9e-fce2-470f-8195-ea4c3c52df22", 2022-06-29T16:00:07.881 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:00:07.881 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:00:07.881 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:00:07.882 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:00:07.882 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:00:07.882 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:00:07.883 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:00:07.883 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 30, 2022-06-29T16:00:07.883 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:00:07.883 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:00:07.883 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:00:07.884 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:07.884 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.884 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:07.885 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6816", 2022-06-29T16:00:07.885 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:00:07.885 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.885 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.886 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:07.886 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6825", 2022-06-29T16:00:07.886 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:00:07.886 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:07.887 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:07.887 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.887 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:00:07.887 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:07.888 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.888 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:07.888 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6826", 2022-06-29T16:00:07.888 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:00:07.888 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.889 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.889 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:07.889 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6827", 2022-06-29T16:00:07.889 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:00:07.890 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:07.890 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:07.890 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.890 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:00:07.891 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:07.891 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.891 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:07.891 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6830", 2022-06-29T16:00:07.891 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:00:07.892 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.892 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.892 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:07.892 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6831", 2022-06-29T16:00:07.893 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:00:07.893 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:07.893 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:07.893 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.894 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:00:07.894 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:07.894 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.894 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:07.895 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6828", 2022-06-29T16:00:07.895 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:00:07.895 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.895 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.896 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:07.896 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6829", 2022-06-29T16:00:07.896 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:00:07.896 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:07.897 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:07.897 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.897 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6825/12890", 2022-06-29T16:00:07.897 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6827/12890", 2022-06-29T16:00:07.897 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6831/12890", 2022-06-29T16:00:07.898 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6829/12890", 2022-06-29T16:00:07.898 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:00:07.898 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:00:07.898 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:00:07.899 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:07.899 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.899 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.899 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:00:07.900 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "ca43f136-6f69-4431-a157-fbe073fb585c", 2022-06-29T16:00:07.900 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:00:07.900 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:00:07.900 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:00:07.901 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:00:07.901 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:00:07.901 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:00:07.901 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:00:07.901 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 30, 2022-06-29T16:00:07.902 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:00:07.902 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:00:07.902 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:00:07.902 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:07.903 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.903 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:07.903 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6815", 2022-06-29T16:00:07.903 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:00:07.904 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.904 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.904 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:07.904 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6818", 2022-06-29T16:00:07.904 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:00:07.905 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:07.905 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:07.905 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.905 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:00:07.906 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:07.906 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.906 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:07.906 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6819", 2022-06-29T16:00:07.907 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:00:07.907 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.907 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.907 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:07.907 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6820", 2022-06-29T16:00:07.908 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:00:07.908 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:07.908 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:07.908 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.909 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:00:07.909 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:07.909 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.909 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:07.909 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6823", 2022-06-29T16:00:07.910 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:00:07.910 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.910 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.910 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:07.911 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6824", 2022-06-29T16:00:07.911 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:00:07.911 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:07.911 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:07.912 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.912 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:00:07.912 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:07.912 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.913 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:07.913 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6821", 2022-06-29T16:00:07.913 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:00:07.913 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.913 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.914 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:07.914 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6822", 2022-06-29T16:00:07.914 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:00:07.914 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:07.915 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:07.915 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.915 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6818/12892", 2022-06-29T16:00:07.915 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6820/12892", 2022-06-29T16:00:07.916 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6824/12892", 2022-06-29T16:00:07.916 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6822/12892", 2022-06-29T16:00:07.916 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:00:07.916 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:00:07.916 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:00:07.917 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:07.917 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.917 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.917 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:00:07.918 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "74fac45f-5f07-44ec-a9cb-2644c123fd4a", 2022-06-29T16:00:07.918 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:00:07.918 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:00:07.919 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:00:07.919 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:00:07.919 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:00:07.919 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:00:07.920 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:00:07.920 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 29, 2022-06-29T16:00:07.920 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:00:07.920 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:00:07.920 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:00:07.921 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:07.921 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.921 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:07.921 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6808", 2022-06-29T16:00:07.922 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:00:07.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.922 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:07.922 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6809", 2022-06-29T16:00:07.923 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:00:07.923 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:07.923 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:07.923 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.924 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:00:07.924 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:07.924 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.924 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:07.925 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6810", 2022-06-29T16:00:07.925 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:00:07.925 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.925 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.925 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:07.926 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6811", 2022-06-29T16:00:07.926 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:00:07.926 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:07.927 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:07.927 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.927 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:00:07.928 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:07.928 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.928 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:07.929 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6814", 2022-06-29T16:00:07.929 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:00:07.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.929 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.930 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:07.930 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6817", 2022-06-29T16:00:07.930 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:00:07.930 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:07.931 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:07.931 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.931 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:00:07.931 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:07.932 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.932 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:07.932 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6812", 2022-06-29T16:00:07.932 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:00:07.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.933 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.933 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:07.933 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6813", 2022-06-29T16:00:07.933 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:00:07.934 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:07.934 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:07.934 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.934 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6809/12891", 2022-06-29T16:00:07.935 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6811/12891", 2022-06-29T16:00:07.935 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6817/12891", 2022-06-29T16:00:07.935 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6813/12891", 2022-06-29T16:00:07.935 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:00:07.936 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:00:07.936 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:00:07.936 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:07.936 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.937 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.937 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:00:07.937 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "2581d351-973a-4b4e-85ca-3658f0068765", 2022-06-29T16:00:07.937 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:00:07.937 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:00:07.938 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:00:07.938 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:00:07.938 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:00:07.939 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:00:07.939 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:00:07.939 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 30, 2022-06-29T16:00:07.939 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:00:07.940 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:00:07.940 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:00:07.940 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:07.940 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.940 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:07.941 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6800", 2022-06-29T16:00:07.941 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:00:07.941 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.941 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.942 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:07.942 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6801", 2022-06-29T16:00:07.942 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:00:07.942 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:07.943 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:07.943 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.943 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:00:07.943 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:07.943 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.944 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:07.944 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6802", 2022-06-29T16:00:07.944 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:00:07.944 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.945 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.945 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:07.945 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6803", 2022-06-29T16:00:07.946 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:00:07.946 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:07.946 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:07.946 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.947 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:00:07.947 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:07.947 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.948 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:07.948 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6806", 2022-06-29T16:00:07.948 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:00:07.948 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.948 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.949 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:07.949 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6807", 2022-06-29T16:00:07.949 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:00:07.949 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:07.950 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:07.950 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.950 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:00:07.950 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:07.951 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.951 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:07.951 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6804", 2022-06-29T16:00:07.951 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:00:07.952 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.952 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.952 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:07.952 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6805", 2022-06-29T16:00:07.953 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:00:07.953 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:07.953 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:07.953 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.954 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6801/12889", 2022-06-29T16:00:07.954 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6803/12889", 2022-06-29T16:00:07.954 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6807/12889", 2022-06-29T16:00:07.955 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6805/12889", 2022-06-29T16:00:07.955 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:00:07.955 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:00:07.955 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:00:07.956 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:07.956 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.956 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.956 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:00:07.957 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "3d234db9-87cd-4fe9-814f-0240a287d8b4", 2022-06-29T16:00:07.957 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:00:07.957 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:00:07.957 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:00:07.957 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:00:07.958 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:00:07.958 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:00:07.958 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:00:07.959 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 30, 2022-06-29T16:00:07.959 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:00:07.959 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:00:07.959 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:00:07.960 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:07.960 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.960 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:07.960 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6808", 2022-06-29T16:00:07.960 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:00:07.961 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.961 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.961 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:07.961 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6810", 2022-06-29T16:00:07.962 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:00:07.962 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:07.962 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:07.962 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.963 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:00:07.963 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:07.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.963 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:07.964 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6812", 2022-06-29T16:00:07.964 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:00:07.964 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.964 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:07.965 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6814", 2022-06-29T16:00:07.965 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:00:07.965 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:07.965 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:07.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.966 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:00:07.966 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:07.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.967 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:07.967 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6823", 2022-06-29T16:00:07.967 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:00:07.967 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.968 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:07.968 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6825", 2022-06-29T16:00:07.968 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:00:07.968 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:07.969 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:07.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.969 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:00:07.969 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:07.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.970 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:07.970 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6817", 2022-06-29T16:00:07.970 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:00:07.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.971 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.971 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:07.971 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6820", 2022-06-29T16:00:07.971 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:00:07.972 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:07.972 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:07.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.972 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6810/13249", 2022-06-29T16:00:07.973 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6814/13249", 2022-06-29T16:00:07.973 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6825/13249", 2022-06-29T16:00:07.973 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6820/13249", 2022-06-29T16:00:07.973 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:00:07.973 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:00:07.974 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:00:07.974 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:07.974 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.974 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.975 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:00:07.975 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "e4bf1104-ff5c-438b-b608-8cee31c163f6", 2022-06-29T16:00:07.975 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:00:07.975 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:00:07.975 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:00:07.976 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:00:07.976 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:00:07.976 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:00:07.976 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:00:07.977 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 30, 2022-06-29T16:00:07.977 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:00:07.977 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:00:07.977 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:00:07.977 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:07.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.978 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:07.978 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6809", 2022-06-29T16:00:07.978 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:00:07.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.979 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.979 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:07.979 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6811", 2022-06-29T16:00:07.980 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:00:07.980 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:07.980 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:07.980 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.981 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:00:07.981 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:07.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.981 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:07.982 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6813", 2022-06-29T16:00:07.982 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:00:07.982 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.982 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.983 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:07.983 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6815", 2022-06-29T16:00:07.983 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:00:07.983 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:07.984 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:07.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.984 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:00:07.984 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:07.984 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.985 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:07.985 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6824", 2022-06-29T16:00:07.985 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:00:07.986 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.986 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.986 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:07.986 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6827", 2022-06-29T16:00:07.987 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:00:07.987 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:07.987 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:07.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.988 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:00:07.988 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:07.988 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.989 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:07.989 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6818", 2022-06-29T16:00:07.989 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:00:07.989 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.990 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.990 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:07.990 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6821", 2022-06-29T16:00:07.990 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:00:07.991 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:07.991 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:07.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.991 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6811/13248", 2022-06-29T16:00:07.991 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6815/13248", 2022-06-29T16:00:07.992 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6827/13248", 2022-06-29T16:00:07.992 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6821/13248", 2022-06-29T16:00:07.992 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:00:07.993 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:00:07.993 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:00:07.993 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:07.993 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.994 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:00:07.994 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "8564dd25-27ab-4203-b54e-af6d3aa05857", 2022-06-29T16:00:07.994 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:00:07.994 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:00:07.995 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:00:07.995 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:00:07.995 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:00:07.995 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:00:07.996 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:00:07.996 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 29, 2022-06-29T16:00:07.996 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:00:07.996 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:00:07.997 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:00:07.997 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:07.997 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.998 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:07.998 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6816", 2022-06-29T16:00:07.998 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:00:07.998 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:07.999 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:07.999 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:07.999 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6819", 2022-06-29T16:00:07.999 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:00:08.000 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:08.000 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:08.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:08.000 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:00:08.000 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:08.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:08.001 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:08.001 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6822", 2022-06-29T16:00:08.001 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:00:08.002 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:08.002 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:08.002 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:08.002 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6826", 2022-06-29T16:00:08.003 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:00:08.003 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:08.003 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:08.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:08.003 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:00:08.004 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:08.004 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:08.004 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:08.004 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6830", 2022-06-29T16:00:08.005 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:00:08.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:08.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:08.005 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:08.006 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6831", 2022-06-29T16:00:08.006 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:00:08.006 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:08.006 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:08.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:08.007 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:00:08.007 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:08.007 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:08.007 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:08.008 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6828", 2022-06-29T16:00:08.008 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:00:08.008 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:08.008 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:08.009 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:08.009 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6829", 2022-06-29T16:00:08.009 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:00:08.009 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:08.009 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:08.010 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:08.010 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6819/13250", 2022-06-29T16:00:08.010 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6826/13250", 2022-06-29T16:00:08.010 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6831/13250", 2022-06-29T16:00:08.011 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6829/13250", 2022-06-29T16:00:08.011 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:00:08.011 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:00:08.012 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:00:08.012 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:08.012 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:08.012 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:08.013 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:00:08.013 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "bd513a9c-3b19-4314-9897-d300b5c897bb", 2022-06-29T16:00:08.013 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:00:08.013 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:00:08.014 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:00:08.014 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:00:08.014 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:00:08.015 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:00:08.015 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:00:08.015 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 30, 2022-06-29T16:00:08.015 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:00:08.015 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:00:08.016 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:00:08.016 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:08.016 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:08.016 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:08.017 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6800", 2022-06-29T16:00:08.017 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:00:08.017 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:08.018 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:08.018 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:08.018 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6801", 2022-06-29T16:00:08.018 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:00:08.019 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:08.019 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:08.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:08.019 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:00:08.020 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:08.020 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:08.020 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:08.020 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6802", 2022-06-29T16:00:08.021 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:00:08.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:08.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:08.021 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:08.022 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6803", 2022-06-29T16:00:08.022 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:00:08.022 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:08.023 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:08.023 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:08.023 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:00:08.023 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:08.024 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:08.024 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:08.024 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6806", 2022-06-29T16:00:08.024 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:00:08.025 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:08.025 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:08.025 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:08.025 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6807", 2022-06-29T16:00:08.026 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:00:08.026 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:08.026 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:08.026 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:08.027 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:00:08.027 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:08.027 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:08.027 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:08.028 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6804", 2022-06-29T16:00:08.028 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:00:08.028 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:08.028 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:08.028 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:08.029 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6805", 2022-06-29T16:00:08.029 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:00:08.029 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:08.029 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:08.030 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:08.030 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6801/13247", 2022-06-29T16:00:08.030 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6803/13247", 2022-06-29T16:00:08.030 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6807/13247", 2022-06-29T16:00:08.031 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6805/13247", 2022-06-29T16:00:08.031 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:00:08.031 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:00:08.032 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:00:08.032 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:08.032 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:08.032 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:00:08.033 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_xinfo": [ 2022-06-29T16:00:08.033 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:08.033 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:00:08.034 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:00:08.034 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:00:08.034 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:00:08.035 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:00:08.035 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:00:08.036 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.891342+0000", 2022-06-29T16:00:08.036 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:00:08.037 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:08.037 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:08.037 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:00:08.038 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:00:08.039 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:00:08.039 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:00:08.040 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:00:08.040 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:00:08.040 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.900866+0000", 2022-06-29T16:00:08.041 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:00:08.041 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:08.042 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:08.042 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:00:08.043 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:00:08.043 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:00:08.043 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:00:08.044 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:00:08.044 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:00:08.044 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.912755+0000", 2022-06-29T16:00:08.045 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:00:08.045 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:08.046 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:08.046 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:00:08.046 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:00:08.047 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:00:08.047 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:00:08.047 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:00:08.047 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:00:08.048 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.864332+0000", 2022-06-29T16:00:08.048 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:00:08.048 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:08.048 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:08.049 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:00:08.049 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:00:08.049 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:00:08.049 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:00:08.049 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:00:08.050 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:00:08.050 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.863756+0000", 2022-06-29T16:00:08.050 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:00:08.050 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:08.051 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:08.051 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:00:08.051 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:00:08.051 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:00:08.052 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:00:08.052 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:00:08.052 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:00:08.053 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.869041+0000", 2022-06-29T16:00:08.053 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:00:08.053 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:08.053 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:08.054 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:00:08.054 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:00:08.054 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:00:08.054 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:00:08.055 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:00:08.055 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:00:08.055 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.895043+0000", 2022-06-29T16:00:08.055 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:00:08.056 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:08.056 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:08.056 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:00:08.056 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:00:08.057 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:00:08.057 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:00:08.057 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:00:08.057 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:00:08.058 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.889491+0000", 2022-06-29T16:00:08.058 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:00:08.058 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:08.058 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:00:08.059 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap": [], 2022-06-29T16:00:08.059 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap_items": [], 2022-06-29T16:00:08.059 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_temp": [], 2022-06-29T16:00:08.059 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_temp": [], 2022-06-29T16:00:08.060 INFO:teuthology.orchestra.run.smithi060.stdout: "blacklist": {}, 2022-06-29T16:00:08.060 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profiles": { 2022-06-29T16:00:08.060 INFO:teuthology.orchestra.run.smithi060.stdout: "default": { 2022-06-29T16:00:08.060 INFO:teuthology.orchestra.run.smithi060.stdout: "crush-failure-domain": "osd", 2022-06-29T16:00:08.060 INFO:teuthology.orchestra.run.smithi060.stdout: "k": "2", 2022-06-29T16:00:08.061 INFO:teuthology.orchestra.run.smithi060.stdout: "m": "1", 2022-06-29T16:00:08.061 INFO:teuthology.orchestra.run.smithi060.stdout: "plugin": "jerasure", 2022-06-29T16:00:08.061 INFO:teuthology.orchestra.run.smithi060.stdout: "ruleset-failure-domain": "osd", 2022-06-29T16:00:08.061 INFO:teuthology.orchestra.run.smithi060.stdout: "technique": "reed_sol_van" 2022-06-29T16:00:08.062 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:08.062 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:08.062 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps_queue": [], 2022-06-29T16:00:08.062 INFO:teuthology.orchestra.run.smithi060.stdout: "new_removed_snaps": [], 2022-06-29T16:00:08.063 INFO:teuthology.orchestra.run.smithi060.stdout: "new_purged_snaps": [], 2022-06-29T16:00:08.063 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_node_flags": {}, 2022-06-29T16:00:08.063 INFO:teuthology.orchestra.run.smithi060.stdout: "device_class_flags": {} 2022-06-29T16:00:08.063 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:00:08.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.787+0000 7fd188b4c700 1 -- 172.21.15.60:0/3903722206 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd168053b30 msgr2=0x7fd168055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:08.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.787+0000 7fd188b4c700 1 --2- 172.21.15.60:0/3903722206 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd168053b30 0x7fd168055ff0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fd17c00d000 tx=0x7fd17c008040).stop 2022-06-29T16:00:08.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.787+0000 7fd188b4c700 1 -- 172.21.15.60:0/3903722206 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1840b3cc0 msgr2=0x7fd1840acdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:08.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.787+0000 7fd188b4c700 1 --2- 172.21.15.60:0/3903722206 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1840b3cc0 0x7fd1840acdc0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fd184060000 tx=0x7fd16c007c00).stop 2022-06-29T16:00:08.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.787+0000 7fd188b4c700 1 -- 172.21.15.60:0/3903722206 shutdown_connections 2022-06-29T16:00:08.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.787+0000 7fd188b4c700 1 --2- 172.21.15.60:0/3903722206 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd168053b30 0x7fd168055ff0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:08.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.787+0000 7fd188b4c700 1 --2- 172.21.15.60:0/3903722206 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd1840b9d70 0x7fd18413ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:08.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.787+0000 7fd188b4c700 1 --2- 172.21.15.60:0/3903722206 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1840b3cc0 0x7fd1840acdc0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:08.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.787+0000 7fd188b4c700 1 --2- 172.21.15.60:0/3903722206 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd18405d1f0 0x7fd1841369e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:08.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.787+0000 7fd188b4c700 1 -- 172.21.15.60:0/3903722206 >> 172.21.15.60:0/3903722206 conn(0x7fd184062ce0 msgr2=0x7fd1840b6610 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:08.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.787+0000 7fd188b4c700 1 -- 172.21.15.60:0/3903722206 shutdown_connections 2022-06-29T16:00:08.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:07.787+0000 7fd188b4c700 1 -- 172.21.15.60:0/3903722206 wait complete. 2022-06-29T16:00:08.069 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:00:08.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.175+0000 7f4420c7e700 1 Processor -- start 2022-06-29T16:00:08.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.175+0000 7f4420c7e700 1 -- start start 2022-06-29T16:00:08.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.175+0000 7f4420c7e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f441c063ad0 0x7f441c067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:08.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.175+0000 7f4420c7e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f441c14af10 0x7f441c05c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:08.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.175+0000 7f4420c7e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f441c0acba0 con 0x7f441c0b5950 2022-06-29T16:00:08.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.175+0000 7f4420c7e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f441c14b3a0 con 0x7f441c14af10 2022-06-29T16:00:08.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.175+0000 7f4420c7e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f441c067220 con 0x7f441c063ad0 2022-06-29T16:00:08.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.175+0000 7f4419d9b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f441c063ad0 0x7f441c067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:08.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.175+0000 7f441ad9d700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f441c14af10 0x7f441c05c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:08.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.175+0000 7f441a59c700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f441c0b5950 0x7f441c05bd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:59060/0 (socket says 172.21.15.60:59060) 2022-06-29T16:00:08.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.175+0000 7f441a59c700 1 -- 172.21.15.60:0/210144431 learned_addr learned my addr 172.21.15.60:0/210144431 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:00:08.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.175+0000 7f4419d9b700 1 -- 172.21.15.60:0/210144431 >> v2:172.21.15.92:3300/0 conn(0x7f441c14af10 msgr2=0x7f441c05c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:08.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.175+0000 7f4419d9b700 1 --2- 172.21.15.60:0/210144431 >> v2:172.21.15.92:3300/0 conn(0x7f441c14af10 0x7f441c05c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:08.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.175+0000 7f4419d9b700 1 -- 172.21.15.60:0/210144431 >> v1:172.21.15.92:6789/0 conn(0x7f441c0b5950 legacy=0x7f441c05bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:08.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.175+0000 7f4419d9b700 1 -- 172.21.15.60:0/210144431 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f441c0b5d30 con 0x7f441c063ad0 2022-06-29T16:00:08.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.175+0000 7f4419d9b700 1 --2- 172.21.15.60:0/210144431 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f441c063ad0 0x7f441c067680 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f4404002f70 tx=0x7f4404006df0).ready entity=mon.2 client_cookie=9d89ae9ffe2ac763 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:08.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.175+0000 7f441959a700 1 -- 172.21.15.60:0/210144431 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f440400d570 con 0x7f441c063ad0 2022-06-29T16:00:08.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.175+0000 7f441959a700 1 -- 172.21.15.60:0/210144431 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4404000e90 con 0x7f441c063ad0 2022-06-29T16:00:08.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.175+0000 7f441959a700 1 -- 172.21.15.60:0/210144431 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f440400d8c0 con 0x7f441c063ad0 2022-06-29T16:00:08.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.175+0000 7f4420c7e700 1 -- 172.21.15.60:0/210144431 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f441c063ad0 msgr2=0x7f441c067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:08.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.175+0000 7f4420c7e700 1 --2- 172.21.15.60:0/210144431 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f441c063ad0 0x7f441c067680 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f4404002f70 tx=0x7f4404006df0).stop 2022-06-29T16:00:08.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.175+0000 7f4420c7e700 1 -- 172.21.15.60:0/210144431 shutdown_connections 2022-06-29T16:00:08.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.175+0000 7f4420c7e700 1 --2- 172.21.15.60:0/210144431 >> v2:172.21.15.92:3300/0 conn(0x7f441c14af10 0x7f441c05c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:08.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.175+0000 7f4420c7e700 1 --2- 172.21.15.60:0/210144431 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f441c063ad0 0x7f441c067680 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:08.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.175+0000 7f4420c7e700 1 -- 172.21.15.60:0/210144431 >> 172.21.15.60:0/210144431 conn(0x7f441c0632c0 msgr2=0x7f441c0b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:08.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.175+0000 7f4420c7e700 1 -- 172.21.15.60:0/210144431 shutdown_connections 2022-06-29T16:00:08.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.175+0000 7f4420c7e700 1 -- 172.21.15.60:0/210144431 wait complete. 2022-06-29T16:00:08.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.179+0000 7f4420c7e700 1 Processor -- start 2022-06-29T16:00:08.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.179+0000 7f4420c7e700 1 -- start start 2022-06-29T16:00:08.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.179+0000 7f4420c7e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f441c063ad0 0x7f441c065060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:08.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.179+0000 7f4420c7e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f441c0b5950 0x7f441c142220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:08.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.179+0000 7f4420c7e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f441c14af10 0x7f441c142710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:08.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.179+0000 7f4420c7e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441c0b0430 con 0x7f441c063ad0 2022-06-29T16:00:08.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.179+0000 7f4420c7e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441c0b06d0 con 0x7f441c0b5950 2022-06-29T16:00:08.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.179+0000 7f4420c7e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f441c0b0970 con 0x7f441c14af10 2022-06-29T16:00:08.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.179+0000 7f441a59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f441c063ad0 0x7f441c065060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:08.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.179+0000 7f4419d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f441c0b5950 0x7f441c142220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:08.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.179+0000 7f441ad9d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f441c14af10 0x7f441c142710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:08.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.179+0000 7f441ad9d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f441c14af10 0x7f441c142710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:59830/0 (socket says 172.21.15.60:59830) 2022-06-29T16:00:08.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.179+0000 7f441ad9d700 1 -- 172.21.15.60:0/3587108883 learned_addr learned my addr 172.21.15.60:0/3587108883 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:08.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.179+0000 7f4419d9b700 1 -- 172.21.15.60:0/3587108883 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f441c14af10 msgr2=0x7f441c142710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:08.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.179+0000 7f4419d9b700 1 --2- 172.21.15.60:0/3587108883 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f441c14af10 0x7f441c142710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:08.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.179+0000 7f4419d9b700 1 -- 172.21.15.60:0/3587108883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f441c063ad0 msgr2=0x7f441c065060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:08.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.179+0000 7f4419d9b700 1 --2- 172.21.15.60:0/3587108883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f441c063ad0 0x7f441c065060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:08.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.179+0000 7f4419d9b700 1 -- 172.21.15.60:0/3587108883 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f440400b180 con 0x7f441c0b5950 2022-06-29T16:00:08.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.179+0000 7f441ad9d700 1 --2- 172.21.15.60:0/3587108883 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f441c14af10 0x7f441c142710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:00:08.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.179+0000 7f441a59c700 1 --2- 172.21.15.60:0/3587108883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f441c063ad0 0x7f441c065060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:00:08.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.179+0000 7f4419d9b700 1 --2- 172.21.15.60:0/3587108883 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f441c0b5950 0x7f441c142220 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f4404002f70 tx=0x7f4404014000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:08.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.179+0000 7f44137fe700 1 -- 172.21.15.60:0/3587108883 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f440400dea0 con 0x7f441c0b5950 2022-06-29T16:00:08.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.179+0000 7f44137fe700 1 -- 172.21.15.60:0/3587108883 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4404013de0 con 0x7f441c0b5950 2022-06-29T16:00:08.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.179+0000 7f4420c7e700 1 -- 172.21.15.60:0/3587108883 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f441c0b0e10 con 0x7f441c0b5950 2022-06-29T16:00:08.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.179+0000 7f44137fe700 1 -- 172.21.15.60:0/3587108883 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f440401e5f0 con 0x7f441c0b5950 2022-06-29T16:00:08.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.179+0000 7f4420c7e700 1 -- 172.21.15.60:0/3587108883 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f441c0b42b0 con 0x7f441c0b5950 2022-06-29T16:00:08.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.183+0000 7f4420c7e700 1 -- 172.21.15.60:0/3587108883 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f441c0573d0 con 0x7f441c0b5950 2022-06-29T16:00:08.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.183+0000 7f44137fe700 1 -- 172.21.15.60:0/3587108883 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4404025030 con 0x7f441c0b5950 2022-06-29T16:00:08.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.183+0000 7f44137fe700 1 --2- 172.21.15.60:0/3587108883 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4400053a70 0x7f4400055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:08.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.183+0000 7f44137fe700 1 -- 172.21.15.60:0/3587108883 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f440406cb50 con 0x7f441c0b5950 2022-06-29T16:00:08.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.183+0000 7f441a59c700 1 --2- 172.21.15.60:0/3587108883 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4400053a70 0x7f4400055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:08.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.183+0000 7f441a59c700 1 --2- 172.21.15.60:0/3587108883 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4400053a70 0x7f4400055f30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f440c00d000 tx=0x7f440c007b80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:08.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.195+0000 7f44137fe700 1 -- 172.21.15.60:0/3587108883 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4404043a70 con 0x7f441c0b5950 2022-06-29T16:00:08.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.359+0000 7f4420c7e700 1 -- 172.21.15.60:0/3587108883 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f441c14a9d0 con 0x7f441c0b5950 2022-06-29T16:00:08.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.359+0000 7f44137fe700 1 -- 172.21.15.60:0/3587108883 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+2134 (secure 0 0 0) 0x7f4404023070 con 0x7f441c0b5950 2022-06-29T16:00:08.369 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:00:08.369 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":9,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4884,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2555925680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2555925680},{"type":"v1","addr":"172.21.15.60:6833","nonce":2555925680}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":7}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T15:59:05.257113+0000","modified":"2022-06-29T15:59:06.259879+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4933},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4933":{"gid":4933,"name":"b","rank":0,"incarnation":8,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/1993861928","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1993861928},{"type":"v1","addr":"172.21.15.92:6835","nonce":1993861928}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-06-29T16:00:08.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.363+0000 7f4420c7e700 1 -- 172.21.15.60:0/3587108883 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4400053a70 msgr2=0x7f4400055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:08.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.363+0000 7f4420c7e700 1 --2- 172.21.15.60:0/3587108883 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4400053a70 0x7f4400055f30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f440c00d000 tx=0x7f440c007b80).stop 2022-06-29T16:00:08.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.363+0000 7f4420c7e700 1 -- 172.21.15.60:0/3587108883 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f441c0b5950 msgr2=0x7f441c142220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:08.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.363+0000 7f4420c7e700 1 --2- 172.21.15.60:0/3587108883 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f441c0b5950 0x7f441c142220 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f4404002f70 tx=0x7f4404014000).stop 2022-06-29T16:00:08.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.363+0000 7f4420c7e700 1 -- 172.21.15.60:0/3587108883 shutdown_connections 2022-06-29T16:00:08.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.363+0000 7f4420c7e700 1 --2- 172.21.15.60:0/3587108883 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4400053a70 0x7f4400055f30 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:08.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.363+0000 7f4420c7e700 1 --2- 172.21.15.60:0/3587108883 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f441c14af10 0x7f441c142710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:08.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.363+0000 7f4420c7e700 1 --2- 172.21.15.60:0/3587108883 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f441c0b5950 0x7f441c142220 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:08.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.363+0000 7f4420c7e700 1 --2- 172.21.15.60:0/3587108883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f441c063ad0 0x7f441c065060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:08.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.363+0000 7f4420c7e700 1 -- 172.21.15.60:0/3587108883 >> 172.21.15.60:0/3587108883 conn(0x7f441c0632c0 msgr2=0x7f441c147110 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:08.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.363+0000 7f4420c7e700 1 -- 172.21.15.60:0/3587108883 shutdown_connections 2022-06-29T16:00:08.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.363+0000 7f4420c7e700 1 -- 172.21.15.60:0/3587108883 wait complete. 2022-06-29T16:00:08.375 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 9 2022-06-29T16:00:08.382 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-06-29T16:00:08.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.479+0000 7f90e92b5700 1 Processor -- start 2022-06-29T16:00:08.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.479+0000 7f90e92b5700 1 -- start start 2022-06-29T16:00:08.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.479+0000 7f90e92b5700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f90e40b5950 0x7f90e4067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:08.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.479+0000 7f90e92b5700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f90e4063ad0 0x7f90e414af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:08.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.479+0000 7f90e92b5700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90e405bd20 0x7f90e405c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:08.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.479+0000 7f90e92b5700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90e40acba0 con 0x7f90e40b5950 2022-06-29T16:00:08.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.479+0000 7f90e92b5700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90e4067220 con 0x7f90e4063ad0 2022-06-29T16:00:08.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.479+0000 7f90e92b5700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90e40b5d30 con 0x7f90e405bd20 2022-06-29T16:00:08.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.479+0000 7f90e2ffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f90e40b5950 0x7f90e4067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:08.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.479+0000 7f90e37fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90e405bd20 0x7f90e405c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:08.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.483+0000 7f90e37fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90e405bd20 0x7f90e405c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:59838/0 (socket says 172.21.15.60:59838) 2022-06-29T16:00:08.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.483+0000 7f90e27fc700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f90e4063ad0 0x7f90e414af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:08.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.483+0000 7f90e37fe700 1 -- 172.21.15.60:0/1161953939 learned_addr learned my addr 172.21.15.60:0/1161953939 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:08.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.483+0000 7f90e37fe700 1 -- 172.21.15.60:0/1161953939 >> v2:172.21.15.92:3300/0 conn(0x7f90e4063ad0 msgr2=0x7f90e414af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:08.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.483+0000 7f90e37fe700 1 --2- 172.21.15.60:0/1161953939 >> v2:172.21.15.92:3300/0 conn(0x7f90e4063ad0 0x7f90e414af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:08.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.483+0000 7f90e37fe700 1 -- 172.21.15.60:0/1161953939 >> v2:172.21.15.60:3300/0 conn(0x7f90e40b5950 msgr2=0x7f90e4067680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:08.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.483+0000 7f90e37fe700 1 --2- 172.21.15.60:0/1161953939 >> v2:172.21.15.60:3300/0 conn(0x7f90e40b5950 0x7f90e4067680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:08.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.483+0000 7f90e37fe700 1 -- 172.21.15.60:0/1161953939 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e40ace00 con 0x7f90e405bd20 2022-06-29T16:00:08.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.483+0000 7f90e37fe700 1 --2- 172.21.15.60:0/1161953939 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90e405bd20 0x7f90e405c1b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f90dc0032f0 tx=0x7f90dc009290).ready entity=mon.2 client_cookie=b0dadb541d8224e1 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:08.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.483+0000 7f90e1ffb700 1 -- 172.21.15.60:0/1161953939 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90dc00b680 con 0x7f90e405bd20 2022-06-29T16:00:08.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.483+0000 7f90e1ffb700 1 -- 172.21.15.60:0/1161953939 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90dc00b7e0 con 0x7f90e405bd20 2022-06-29T16:00:08.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.483+0000 7f90e1ffb700 1 -- 172.21.15.60:0/1161953939 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90dc00b950 con 0x7f90e405bd20 2022-06-29T16:00:08.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.483+0000 7f90e92b5700 1 -- 172.21.15.60:0/1161953939 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90e405bd20 msgr2=0x7f90e405c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:08.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.483+0000 7f90e92b5700 1 --2- 172.21.15.60:0/1161953939 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90e405bd20 0x7f90e405c1b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f90dc0032f0 tx=0x7f90dc009290).stop 2022-06-29T16:00:08.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.483+0000 7f90e92b5700 1 -- 172.21.15.60:0/1161953939 shutdown_connections 2022-06-29T16:00:08.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.483+0000 7f90e92b5700 1 --2- 172.21.15.60:0/1161953939 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90e405bd20 0x7f90e405c1b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:08.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.483+0000 7f90e92b5700 1 --2- 172.21.15.60:0/1161953939 >> v2:172.21.15.92:3300/0 conn(0x7f90e4063ad0 0x7f90e414af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:08.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.483+0000 7f90e92b5700 1 --2- 172.21.15.60:0/1161953939 >> v2:172.21.15.60:3300/0 conn(0x7f90e40b5950 0x7f90e4067680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:08.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.483+0000 7f90e92b5700 1 -- 172.21.15.60:0/1161953939 >> 172.21.15.60:0/1161953939 conn(0x7f90e40632c0 msgr2=0x7f90e40b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:08.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.483+0000 7f90e92b5700 1 -- 172.21.15.60:0/1161953939 shutdown_connections 2022-06-29T16:00:08.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.483+0000 7f90e92b5700 1 -- 172.21.15.60:0/1161953939 wait complete. 2022-06-29T16:00:08.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.487+0000 7f90e92b5700 1 Processor -- start 2022-06-29T16:00:08.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.487+0000 7f90e92b5700 1 -- start start 2022-06-29T16:00:08.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.487+0000 7f90e92b5700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90e405bd20 0x7f90e40b0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:08.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.487+0000 7f90e92b5700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90e4063ad0 0x7f90e40aedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:08.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.487+0000 7f90e92b5700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90e40b5950 0x7f90e4117c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:08.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.487+0000 7f90e92b5700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e41498c0 con 0x7f90e4063ad0 2022-06-29T16:00:08.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.487+0000 7f90e92b5700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e4149b60 con 0x7f90e405bd20 2022-06-29T16:00:08.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.487+0000 7f90e92b5700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90e41495b0 con 0x7f90e40b5950 2022-06-29T16:00:08.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.487+0000 7f90e27fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90e4063ad0 0x7f90e40aedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:08.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.487+0000 7f90e27fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90e4063ad0 0x7f90e40aedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46746/0 (socket says 172.21.15.60:46746) 2022-06-29T16:00:08.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.487+0000 7f90e27fc700 1 -- 172.21.15.60:0/614852612 learned_addr learned my addr 172.21.15.60:0/614852612 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:08.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.487+0000 7f90e37fe700 1 --2- 172.21.15.60:0/614852612 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90e40b5950 0x7f90e4117c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:08.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.487+0000 7f90e2ffd700 1 --2- 172.21.15.60:0/614852612 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90e405bd20 0x7f90e40b0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:08.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.487+0000 7f90e27fc700 1 -- 172.21.15.60:0/614852612 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90e40b5950 msgr2=0x7f90e4117c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:08.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.487+0000 7f90e27fc700 1 --2- 172.21.15.60:0/614852612 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90e40b5950 0x7f90e4117c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:08.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.487+0000 7f90e27fc700 1 -- 172.21.15.60:0/614852612 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90e405bd20 msgr2=0x7f90e40b0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:08.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.487+0000 7f90e27fc700 1 --2- 172.21.15.60:0/614852612 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90e405bd20 0x7f90e40b0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:08.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.487+0000 7f90e27fc700 1 -- 172.21.15.60:0/614852612 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90dc009530 con 0x7f90e4063ad0 2022-06-29T16:00:08.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.491+0000 7f90e27fc700 1 --2- 172.21.15.60:0/614852612 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90e4063ad0 0x7f90e40aedb0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f90cc00c000 tx=0x7f90cc006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:08.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.491+0000 7f90dbfff700 1 -- 172.21.15.60:0/614852612 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90cc010780 con 0x7f90e4063ad0 2022-06-29T16:00:08.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.491+0000 7f90dbfff700 1 -- 172.21.15.60:0/614852612 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90cc0108e0 con 0x7f90e4063ad0 2022-06-29T16:00:08.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.491+0000 7f90e92b5700 1 -- 172.21.15.60:0/614852612 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90e40ae980 con 0x7f90e4063ad0 2022-06-29T16:00:08.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.491+0000 7f90dbfff700 1 -- 172.21.15.60:0/614852612 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90cc010c40 con 0x7f90e4063ad0 2022-06-29T16:00:08.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.491+0000 7f90e92b5700 1 -- 172.21.15.60:0/614852612 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90e41595d0 con 0x7f90e4063ad0 2022-06-29T16:00:08.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.491+0000 7f90e92b5700 1 -- 172.21.15.60:0/614852612 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90e41179b0 con 0x7f90e4063ad0 2022-06-29T16:00:08.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.491+0000 7f90dbfff700 1 -- 172.21.15.60:0/614852612 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f90cc01c460 con 0x7f90e4063ad0 2022-06-29T16:00:08.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.491+0000 7f90dbfff700 1 --2- 172.21.15.60:0/614852612 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f90c8053b00 0x7f90c8055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:08.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.491+0000 7f90dbfff700 1 -- 172.21.15.60:0/614852612 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f90cc06c3d0 con 0x7f90e4063ad0 2022-06-29T16:00:08.510 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.491+0000 7f90e2ffd700 1 --2- 172.21.15.60:0/614852612 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f90c8053b00 0x7f90c8055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:08.510 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.495+0000 7f90e2ffd700 1 --2- 172.21.15.60:0/614852612 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f90c8053b00 0x7f90c8055fc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f90d400b000 tx=0x7f90d4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:08.510 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.499+0000 7f90dbfff700 1 -- 172.21.15.60:0/614852612 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f90e41179b0 con 0x7f90e4063ad0 2022-06-29T16:00:08.672 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:08.663+0000 7f90e92b5700 1 -- 172.21.15.60:0/614852612 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f90e41179b0 con 0x7f90e4063ad0 2022-06-29T16:00:09.016 INFO:tasks.ceph.mon.a.smithi060.stderr:2022-06-29T16:00:09.007+0000 7fcc1a376700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-06-29T16:00:09.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.019+0000 7f90dbfff700 1 -- 172.21.15.60:0/614852612 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v10) v1 ==== 161+0+0 (secure 0 0 0) 0x7f90e41179b0 con 0x7f90e4063ad0 2022-06-29T16:00:09.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.023+0000 7f90e92b5700 1 -- 172.21.15.60:0/614852612 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f90c8053b00 msgr2=0x7f90c8055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:09.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.023+0000 7f90e92b5700 1 --2- 172.21.15.60:0/614852612 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f90c8053b00 0x7f90c8055fc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f90d400b000 tx=0x7f90d4006df0).stop 2022-06-29T16:00:09.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.023+0000 7f90e92b5700 1 -- 172.21.15.60:0/614852612 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90e4063ad0 msgr2=0x7f90e40aedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:09.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.023+0000 7f90e92b5700 1 --2- 172.21.15.60:0/614852612 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90e4063ad0 0x7f90e40aedb0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f90cc00c000 tx=0x7f90cc006df0).stop 2022-06-29T16:00:09.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.023+0000 7f90e92b5700 1 -- 172.21.15.60:0/614852612 shutdown_connections 2022-06-29T16:00:09.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.023+0000 7f90e92b5700 1 --2- 172.21.15.60:0/614852612 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f90c8053b00 0x7f90c8055fc0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:09.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.023+0000 7f90e92b5700 1 --2- 172.21.15.60:0/614852612 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90e40b5950 0x7f90e4117c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:09.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.023+0000 7f90e92b5700 1 --2- 172.21.15.60:0/614852612 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90e4063ad0 0x7f90e40aedb0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:09.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.023+0000 7f90e92b5700 1 --2- 172.21.15.60:0/614852612 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90e405bd20 0x7f90e40b0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:09.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.023+0000 7f90e92b5700 1 -- 172.21.15.60:0/614852612 >> 172.21.15.60:0/614852612 conn(0x7f90e40632c0 msgr2=0x7f90e40b30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:09.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.023+0000 7f90e92b5700 1 -- 172.21.15.60:0/614852612 shutdown_connections 2022-06-29T16:00:09.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.023+0000 7f90e92b5700 1 -- 172.21.15.60:0/614852612 wait complete. 2022-06-29T16:00:09.036 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-06-29T16:00:09.049 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:00:09.068 INFO:tasks.ceph.mds.b.smithi092.stdout:starting mds.b at 2022-06-29T16:00:09.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.151+0000 7f483e343700 1 Processor -- start 2022-06-29T16:00:09.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.151+0000 7f483e343700 1 -- start start 2022-06-29T16:00:09.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.155+0000 7f483e343700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f483805fb00 0x7f48380ae890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:09.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.155+0000 7f483e343700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48380ad240 con 0x7f48380b1c00 2022-06-29T16:00:09.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.155+0000 7f483e343700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f483805fee0 con 0x7f48380b0fa0 2022-06-29T16:00:09.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.155+0000 7f483e343700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f483813ce60 con 0x7f483805fb00 2022-06-29T16:00:09.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.155+0000 7f483db42700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f48380b1c00 0x7f4838067250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54844/0 (socket says 172.21.15.60:54844) 2022-06-29T16:00:09.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.155+0000 7f483db42700 1 -- 172.21.15.60:0/671139564 learned_addr learned my addr 172.21.15.60:0/671139564 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:00:09.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.155+0000 7f483d341700 1 --2- 172.21.15.60:0/671139564 >> v2:172.21.15.92:3300/0 conn(0x7f483805fb00 0x7f48380ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:09.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.155+0000 7f4827fff700 1 -- 172.21.15.60:0/671139564 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3451351966 0 0) 0x7f48380ad240 con 0x7f48380b1c00 2022-06-29T16:00:09.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.155+0000 7f4827fff700 1 -- 172.21.15.60:0/671139564 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4820003260 con 0x7f48380b1c00 2022-06-29T16:00:09.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.155+0000 7f483d341700 1 -- 172.21.15.60:0/671139564 >> v1:172.21.15.92:6789/0 conn(0x7f48380b0fa0 legacy=0x7f48380b14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:09.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.155+0000 7f4827fff700 1 -- 172.21.15.60:0/671139564 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4033318828 0 0) 0x7f483805fee0 con 0x7f48380b0fa0 2022-06-29T16:00:09.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.155+0000 7f483d341700 1 -- 172.21.15.60:0/671139564 >> v1:172.21.15.60:6789/0 conn(0x7f48380b1c00 legacy=0x7f4838067250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:09.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.155+0000 7f483d341700 1 -- 172.21.15.60:0/671139564 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4838143180 con 0x7f483805fb00 2022-06-29T16:00:09.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.155+0000 7f483d341700 1 --2- 172.21.15.60:0/671139564 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f483805fb00 0x7f48380ae890 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f4834002f70 tx=0x7f4834006e80).ready entity=mon.1 client_cookie=2a6d7d2ccab4a32e server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:09.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.155+0000 7f4827fff700 1 -- 172.21.15.60:0/671139564 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4834014070 con 0x7f483805fb00 2022-06-29T16:00:09.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.155+0000 7f4827fff700 1 -- 172.21.15.60:0/671139564 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4834000e90 con 0x7f483805fb00 2022-06-29T16:00:09.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.155+0000 7f4827fff700 1 -- 172.21.15.60:0/671139564 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48340107f0 con 0x7f483805fb00 2022-06-29T16:00:09.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.159+0000 7f483e343700 1 -- 172.21.15.60:0/671139564 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f483805fb00 msgr2=0x7f48380ae890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:09.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.159+0000 7f483e343700 1 --2- 172.21.15.60:0/671139564 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f483805fb00 0x7f48380ae890 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f4834002f70 tx=0x7f4834006e80).stop 2022-06-29T16:00:09.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.159+0000 7f483e343700 1 -- 172.21.15.60:0/671139564 shutdown_connections 2022-06-29T16:00:09.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.159+0000 7f483e343700 1 --2- 172.21.15.60:0/671139564 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f483805fb00 0x7f48380ae890 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:09.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.159+0000 7f483e343700 1 -- 172.21.15.60:0/671139564 >> 172.21.15.60:0/671139564 conn(0x7f4838063a60 msgr2=0x7f483805f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:09.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.159+0000 7f483e343700 1 -- 172.21.15.60:0/671139564 shutdown_connections 2022-06-29T16:00:09.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.159+0000 7f483e343700 1 -- 172.21.15.60:0/671139564 wait complete. 2022-06-29T16:00:09.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.159+0000 7f483e343700 1 Processor -- start 2022-06-29T16:00:09.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.159+0000 7f483e343700 1 -- start start 2022-06-29T16:00:09.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.159+0000 7f483e343700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f483805fb00 0x7f4838147f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:09.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.159+0000 7f483e343700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48380b0fa0 0x7f4838145510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:09.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.159+0000 7f483e343700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48380b1c00 0x7f483814fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:09.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.159+0000 7f483e343700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4838145bb0 con 0x7f48380b0fa0 2022-06-29T16:00:09.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.159+0000 7f483e343700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4838145e50 con 0x7f483805fb00 2022-06-29T16:00:09.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.159+0000 7f483e343700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48381460f0 con 0x7f48380b1c00 2022-06-29T16:00:09.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.159+0000 7f483cb40700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48380b0fa0 0x7f4838145510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:09.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.163+0000 7f483d341700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f483805fb00 0x7f4838147f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:09.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.163+0000 7f483d341700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f483805fb00 0x7f4838147f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:52790/0 (socket says 172.21.15.60:52790) 2022-06-29T16:00:09.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.163+0000 7f483d341700 1 -- 172.21.15.60:0/2110640301 learned_addr learned my addr 172.21.15.60:0/2110640301 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:09.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.163+0000 7f483cb40700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48380b0fa0 0x7f4838145510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46762/0 (socket says 172.21.15.60:46762) 2022-06-29T16:00:09.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.163+0000 7f483db42700 1 --2- 172.21.15.60:0/2110640301 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48380b1c00 0x7f483814fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:09.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.163+0000 7f483cb40700 1 -- 172.21.15.60:0/2110640301 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48380b1c00 msgr2=0x7f483814fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:09.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.163+0000 7f483cb40700 1 --2- 172.21.15.60:0/2110640301 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48380b1c00 0x7f483814fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:09.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.163+0000 7f483cb40700 1 -- 172.21.15.60:0/2110640301 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f483805fb00 msgr2=0x7f4838147f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:09.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.163+0000 7f483cb40700 1 --2- 172.21.15.60:0/2110640301 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f483805fb00 0x7f4838147f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:09.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.163+0000 7f483cb40700 1 -- 172.21.15.60:0/2110640301 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483400c310 con 0x7f48380b0fa0 2022-06-29T16:00:09.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.163+0000 7f483d341700 1 --2- 172.21.15.60:0/2110640301 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f483805fb00 0x7f4838147f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:00:09.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.163+0000 7f483cb40700 1 --2- 172.21.15.60:0/2110640301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48380b0fa0 0x7f4838145510 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f4828000bc0 tx=0x7f48280079b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:09.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.163+0000 7f48267fc700 1 -- 172.21.15.60:0/2110640301 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f482800cdd0 con 0x7f48380b0fa0 2022-06-29T16:00:09.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.163+0000 7f48267fc700 1 -- 172.21.15.60:0/2110640301 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4828012400 con 0x7f48380b0fa0 2022-06-29T16:00:09.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.163+0000 7f483e343700 1 -- 172.21.15.60:0/2110640301 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4838146590 con 0x7f48380b0fa0 2022-06-29T16:00:09.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.163+0000 7f48267fc700 1 -- 172.21.15.60:0/2110640301 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4828012760 con 0x7f48380b0fa0 2022-06-29T16:00:09.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.163+0000 7f483e343700 1 -- 172.21.15.60:0/2110640301 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f483810ea40 con 0x7f48380b0fa0 2022-06-29T16:00:09.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.163+0000 7f483e343700 1 -- 172.21.15.60:0/2110640301 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4838145a00 con 0x7f48380b0fa0 2022-06-29T16:00:09.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.163+0000 7f48267fc700 1 -- 172.21.15.60:0/2110640301 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4828019030 con 0x7f48380b0fa0 2022-06-29T16:00:09.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.163+0000 7f48267fc700 1 --2- 172.21.15.60:0/2110640301 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f48180539e0 0x7f4818055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:09.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.163+0000 7f48267fc700 1 -- 172.21.15.60:0/2110640301 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f482806b260 con 0x7f48380b0fa0 2022-06-29T16:00:09.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.167+0000 7f483d341700 1 --2- 172.21.15.60:0/2110640301 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f48180539e0 0x7f4818055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:09.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.167+0000 7f483d341700 1 --2- 172.21.15.60:0/2110640301 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f48180539e0 0x7f4818055ea0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f4834018000 tx=0x7f4838143310).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:09.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.171+0000 7f48267fc700 1 -- 172.21.15.60:0/2110640301 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4828042090 con 0x7f48380b0fa0 2022-06-29T16:00:09.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.335+0000 7f483e343700 1 -- 172.21.15.60:0/2110640301 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48381462b0 con 0x7f48380b0fa0 2022-06-29T16:00:09.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.335+0000 7f48267fc700 1 -- 172.21.15.60:0/2110640301 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+1776 (secure 0 0 0) 0x7f4828007af0 con 0x7f48380b0fa0 2022-06-29T16:00:09.346 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:00:09.346 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":10,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4884,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2555925680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2555925680},{"type":"v1","addr":"172.21.15.60:6833","nonce":2555925680}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":7}],"filesystems":[{"mdsmap":{"epoch":10,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T15:59:05.257113+0000","modified":"2022-06-29T16:00:09.011817+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-06-29T16:00:09.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.339+0000 7f483e343700 1 -- 172.21.15.60:0/2110640301 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f48180539e0 msgr2=0x7f4818055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:09.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.339+0000 7f483e343700 1 --2- 172.21.15.60:0/2110640301 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f48180539e0 0x7f4818055ea0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f4834018000 tx=0x7f4838143310).stop 2022-06-29T16:00:09.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.339+0000 7f483e343700 1 -- 172.21.15.60:0/2110640301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48380b0fa0 msgr2=0x7f4838145510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:09.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.339+0000 7f483e343700 1 --2- 172.21.15.60:0/2110640301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48380b0fa0 0x7f4838145510 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f4828000bc0 tx=0x7f48280079b0).stop 2022-06-29T16:00:09.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.339+0000 7f483e343700 1 -- 172.21.15.60:0/2110640301 shutdown_connections 2022-06-29T16:00:09.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.339+0000 7f483e343700 1 --2- 172.21.15.60:0/2110640301 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f48180539e0 0x7f4818055ea0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:09.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.339+0000 7f483e343700 1 --2- 172.21.15.60:0/2110640301 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f48380b1c00 0x7f483814fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:09.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.339+0000 7f483e343700 1 --2- 172.21.15.60:0/2110640301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f48380b0fa0 0x7f4838145510 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:09.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.339+0000 7f483e343700 1 --2- 172.21.15.60:0/2110640301 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f483805fb00 0x7f4838147f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:09.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.339+0000 7f483e343700 1 -- 172.21.15.60:0/2110640301 >> 172.21.15.60:0/2110640301 conn(0x7f4838063a60 msgr2=0x7f48380b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:09.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.339+0000 7f483e343700 1 -- 172.21.15.60:0/2110640301 shutdown_connections 2022-06-29T16:00:09.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.339+0000 7f483e343700 1 -- 172.21.15.60:0/2110640301 wait complete. 2022-06-29T16:00:09.351 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 10 2022-06-29T16:00:09.359 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-06-29T16:00:09.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.459+0000 7fcda88e9700 1 Processor -- start 2022-06-29T16:00:09.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.459+0000 7fcda88e9700 1 -- start start 2022-06-29T16:00:09.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.459+0000 7fcda88e9700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcda40b4010 0x7fcda41426b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:09.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.459+0000 7fcda88e9700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcda40b46c0 0x7fcda40b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:09.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.459+0000 7fcda88e9700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcda4143a40 con 0x7fcda40ae7c0 2022-06-29T16:00:09.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.459+0000 7fcda88e9700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcda40aee60 con 0x7fcda40b46c0 2022-06-29T16:00:09.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.459+0000 7fcda88e9700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcda40af100 con 0x7fcda40b4010 2022-06-29T16:00:09.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.463+0000 7fcda2d9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcda40b46c0 0x7fcda40b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:09.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.463+0000 7fcda2d9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcda40b46c0 0x7fcda40b4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46772/0 (socket says 172.21.15.60:46772) 2022-06-29T16:00:09.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.463+0000 7fcda2d9d700 1 -- 172.21.15.60:0/51209133 learned_addr learned my addr 172.21.15.60:0/51209133 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:09.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.463+0000 7fcda1d9b700 1 --2- 172.21.15.60:0/51209133 >> v2:172.21.15.92:3300/0 conn(0x7fcda40b4010 0x7fcda41426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:09.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.463+0000 7fcda2d9d700 1 -- 172.21.15.60:0/51209133 >> v2:172.21.15.92:3300/0 conn(0x7fcda40b4010 msgr2=0x7fcda41426b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:09.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.463+0000 7fcda2d9d700 1 --2- 172.21.15.60:0/51209133 >> v2:172.21.15.92:3300/0 conn(0x7fcda40b4010 0x7fcda41426b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:09.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.463+0000 7fcda2d9d700 1 -- 172.21.15.60:0/51209133 >> v1:172.21.15.92:6789/0 conn(0x7fcda40ae7c0 legacy=0x7fcda4141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:09.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.463+0000 7fcda2d9d700 1 -- 172.21.15.60:0/51209133 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda40aeba0 con 0x7fcda40b46c0 2022-06-29T16:00:09.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.463+0000 7fcda1d9b700 1 --2- 172.21.15.60:0/51209133 >> v2:172.21.15.92:3300/0 conn(0x7fcda40b4010 0x7fcda41426b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:00:09.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.463+0000 7fcda2d9d700 1 --2- 172.21.15.60:0/51209133 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcda40b46c0 0x7fcda40b4af0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fcd9c0032f0 tx=0x7fcd9c009290).ready entity=mon.0 client_cookie=24781535fd916c51 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:09.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.463+0000 7fcda159a700 1 -- 172.21.15.60:0/51209133 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd9c00b870 con 0x7fcda40b46c0 2022-06-29T16:00:09.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.463+0000 7fcda159a700 1 -- 172.21.15.60:0/51209133 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd9c00b9d0 con 0x7fcda40b46c0 2022-06-29T16:00:09.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.463+0000 7fcda159a700 1 -- 172.21.15.60:0/51209133 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd9c00bb40 con 0x7fcda40b46c0 2022-06-29T16:00:09.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.463+0000 7fcda88e9700 1 -- 172.21.15.60:0/51209133 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcda40b46c0 msgr2=0x7fcda40b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:09.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.463+0000 7fcda88e9700 1 --2- 172.21.15.60:0/51209133 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcda40b46c0 0x7fcda40b4af0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fcd9c0032f0 tx=0x7fcd9c009290).stop 2022-06-29T16:00:09.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.463+0000 7fcda88e9700 1 -- 172.21.15.60:0/51209133 shutdown_connections 2022-06-29T16:00:09.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.463+0000 7fcda88e9700 1 --2- 172.21.15.60:0/51209133 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcda40b46c0 0x7fcda40b4af0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:09.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.463+0000 7fcda88e9700 1 --2- 172.21.15.60:0/51209133 >> v2:172.21.15.92:3300/0 conn(0x7fcda40b4010 0x7fcda41426b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:09.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.463+0000 7fcda88e9700 1 -- 172.21.15.60:0/51209133 >> 172.21.15.60:0/51209133 conn(0x7fcda40b9340 msgr2=0x7fcda40b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:09.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.463+0000 7fcda88e9700 1 -- 172.21.15.60:0/51209133 shutdown_connections 2022-06-29T16:00:09.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.463+0000 7fcda88e9700 1 -- 172.21.15.60:0/51209133 wait complete. 2022-06-29T16:00:09.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.463+0000 7fcda88e9700 1 Processor -- start 2022-06-29T16:00:09.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.463+0000 7fcda88e9700 1 -- start start 2022-06-29T16:00:09.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.467+0000 7fcda88e9700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcda40ae7c0 0x7fcda4144850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:09.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.467+0000 7fcda88e9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcda40b4010 0x7fcda413f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:09.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.467+0000 7fcda88e9700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcda40b46c0 0x7fcda4152a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:09.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.467+0000 7fcda88e9700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda4146130 con 0x7fcda40b4010 2022-06-29T16:00:09.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.467+0000 7fcda88e9700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda41520b0 con 0x7fcda40ae7c0 2022-06-29T16:00:09.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.467+0000 7fcda88e9700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcda4152350 con 0x7fcda40b46c0 2022-06-29T16:00:09.486 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.467+0000 7fcda1d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcda40b4010 0x7fcda413f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:09.486 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.467+0000 7fcda1d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcda40b4010 0x7fcda413f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46776/0 (socket says 172.21.15.60:46776) 2022-06-29T16:00:09.486 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.467+0000 7fcda1d9b700 1 -- 172.21.15.60:0/3616766174 learned_addr learned my addr 172.21.15.60:0/3616766174 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:09.486 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.467+0000 7fcda259c700 1 --2- 172.21.15.60:0/3616766174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcda40ae7c0 0x7fcda4144850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:09.487 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.467+0000 7fcda2d9d700 1 --2- 172.21.15.60:0/3616766174 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcda40b46c0 0x7fcda4152a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:09.487 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.467+0000 7fcda1d9b700 1 -- 172.21.15.60:0/3616766174 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcda40b46c0 msgr2=0x7fcda4152a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:09.487 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.467+0000 7fcda1d9b700 1 --2- 172.21.15.60:0/3616766174 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcda40b46c0 0x7fcda4152a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:09.487 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.467+0000 7fcda1d9b700 1 -- 172.21.15.60:0/3616766174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcda40ae7c0 msgr2=0x7fcda4144850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:09.487 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.467+0000 7fcda1d9b700 1 --2- 172.21.15.60:0/3616766174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcda40ae7c0 0x7fcda4144850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:09.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.467+0000 7fcda1d9b700 1 -- 172.21.15.60:0/3616766174 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd9c009530 con 0x7fcda40b4010 2022-06-29T16:00:09.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.467+0000 7fcda1d9b700 1 --2- 172.21.15.60:0/3616766174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcda40b4010 0x7fcda413f680 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fcd8c006940 tx=0x7fcd8c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:09.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.467+0000 7fcd9b7fe700 1 -- 172.21.15.60:0/3616766174 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd8c011780 con 0x7fcda40b4010 2022-06-29T16:00:09.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.467+0000 7fcd9b7fe700 1 -- 172.21.15.60:0/3616766174 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd8c0118e0 con 0x7fcda40b4010 2022-06-29T16:00:09.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.467+0000 7fcd9b7fe700 1 -- 172.21.15.60:0/3616766174 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd8c011c40 con 0x7fcda40b4010 2022-06-29T16:00:09.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.467+0000 7fcda88e9700 1 -- 172.21.15.60:0/3616766174 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcda41525f0 con 0x7fcda40b4010 2022-06-29T16:00:09.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.467+0000 7fcda88e9700 1 -- 172.21.15.60:0/3616766174 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcda413fd10 con 0x7fcda40b4010 2022-06-29T16:00:09.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.467+0000 7fcda88e9700 1 -- 172.21.15.60:0/3616766174 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcda4152780 con 0x7fcda40b4010 2022-06-29T16:00:09.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.471+0000 7fcd9b7fe700 1 -- 172.21.15.60:0/3616766174 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcd8c012420 con 0x7fcda40b4010 2022-06-29T16:00:09.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.471+0000 7fcd9b7fe700 1 --2- 172.21.15.60:0/3616766174 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcd88053a70 0x7fcd88055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:09.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.471+0000 7fcd9b7fe700 1 -- 172.21.15.60:0/3616766174 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fcd8c06bef0 con 0x7fcda40b4010 2022-06-29T16:00:09.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.471+0000 7fcda259c700 1 --2- 172.21.15.60:0/3616766174 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcd88053a70 0x7fcd88055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:09.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.471+0000 7fcda259c700 1 --2- 172.21.15.60:0/3616766174 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcd88053a70 0x7fcd88055f30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fcd9400d000 tx=0x7fcd94000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:09.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.475+0000 7fcd9b7fe700 1 -- 172.21.15.60:0/3616766174 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcd8c042bd0 con 0x7fcda40b4010 2022-06-29T16:00:09.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:09.635+0000 7fcda88e9700 1 -- 172.21.15.60:0/3616766174 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fcda40573d0 con 0x7fcda40b4010 2022-06-29T16:00:10.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.027+0000 7fcd9b7fe700 1 -- 172.21.15.60:0/3616766174 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7fcd8c0461f0 con 0x7fcda40b4010 2022-06-29T16:00:10.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.027+0000 7fcda88e9700 1 -- 172.21.15.60:0/3616766174 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcd88053a70 msgr2=0x7fcd88055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:10.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.027+0000 7fcda88e9700 1 --2- 172.21.15.60:0/3616766174 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcd88053a70 0x7fcd88055f30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fcd9400d000 tx=0x7fcd94000bc0).stop 2022-06-29T16:00:10.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.031+0000 7fcda88e9700 1 -- 172.21.15.60:0/3616766174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcda40b4010 msgr2=0x7fcda413f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:10.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.031+0000 7fcda88e9700 1 --2- 172.21.15.60:0/3616766174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcda40b4010 0x7fcda413f680 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fcd8c006940 tx=0x7fcd8c008040).stop 2022-06-29T16:00:10.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.031+0000 7fcda88e9700 1 -- 172.21.15.60:0/3616766174 shutdown_connections 2022-06-29T16:00:10.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.031+0000 7fcda88e9700 1 --2- 172.21.15.60:0/3616766174 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcd88053a70 0x7fcd88055f30 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:10.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.031+0000 7fcda88e9700 1 --2- 172.21.15.60:0/3616766174 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcda40b46c0 0x7fcda4152a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:10.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.031+0000 7fcda88e9700 1 --2- 172.21.15.60:0/3616766174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcda40b4010 0x7fcda413f680 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:10.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.031+0000 7fcda88e9700 1 --2- 172.21.15.60:0/3616766174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcda40ae7c0 0x7fcda4144850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:10.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.031+0000 7fcda88e9700 1 -- 172.21.15.60:0/3616766174 >> 172.21.15.60:0/3616766174 conn(0x7fcda40b9340 msgr2=0x7fcda405cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:10.042 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.031+0000 7fcda88e9700 1 -- 172.21.15.60:0/3616766174 shutdown_connections 2022-06-29T16:00:10.042 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.031+0000 7fcda88e9700 1 -- 172.21.15.60:0/3616766174 wait complete. 2022-06-29T16:00:10.055 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-06-29T16:00:10.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.167+0000 7fdc2c889700 1 Processor -- start 2022-06-29T16:00:10.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.167+0000 7fdc2c889700 1 -- start start 2022-06-29T16:00:10.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.167+0000 7fdc2c889700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc2805d1f0 0x7fdc280b1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:10.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.167+0000 7fdc2c889700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdc280b3cc0 0x7fdc280b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:10.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.167+0000 7fdc2c889700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc280b2ce0 con 0x7fdc280b9d70 2022-06-29T16:00:10.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.167+0000 7fdc2c889700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc280b4d10 con 0x7fdc280b3cc0 2022-06-29T16:00:10.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.167+0000 7fdc2c889700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc280ba150 con 0x7fdc2805d1f0 2022-06-29T16:00:10.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.167+0000 7fdc26d9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdc280b3cc0 0x7fdc280b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:10.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.167+0000 7fdc26d9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdc280b3cc0 0x7fdc280b2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46784/0 (socket says 172.21.15.60:46784) 2022-06-29T16:00:10.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.167+0000 7fdc26d9d700 1 -- 172.21.15.60:0/2102978344 learned_addr learned my addr 172.21.15.60:0/2102978344 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:10.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.167+0000 7fdc25d9b700 1 --2- 172.21.15.60:0/2102978344 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc2805d1f0 0x7fdc280b1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:10.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.167+0000 7fdc26d9d700 1 -- 172.21.15.60:0/2102978344 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc2805d1f0 msgr2=0x7fdc280b1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:10.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.167+0000 7fdc26d9d700 1 --2- 172.21.15.60:0/2102978344 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc2805d1f0 0x7fdc280b1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:10.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.167+0000 7fdc26d9d700 1 -- 172.21.15.60:0/2102978344 >> v1:172.21.15.92:6789/0 conn(0x7fdc280b9d70 legacy=0x7fdc280b3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:10.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.167+0000 7fdc26d9d700 1 -- 172.21.15.60:0/2102978344 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc280b4a70 con 0x7fdc280b3cc0 2022-06-29T16:00:10.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.171+0000 7fdc26d9d700 1 --2- 172.21.15.60:0/2102978344 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc280b3cc0 0x7fdc280b2040 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7fdc20002f70 tx=0x7fdc20006eb0).ready entity=mon.0 client_cookie=f70c169e19153337 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:10.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.171+0000 7fdc2559a700 1 -- 172.21.15.60:0/2102978344 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc2000d570 con 0x7fdc280b3cc0 2022-06-29T16:00:10.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.171+0000 7fdc2559a700 1 -- 172.21.15.60:0/2102978344 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc20000e90 con 0x7fdc280b3cc0 2022-06-29T16:00:10.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.171+0000 7fdc2559a700 1 -- 172.21.15.60:0/2102978344 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc2000d900 con 0x7fdc280b3cc0 2022-06-29T16:00:10.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.171+0000 7fdc2c889700 1 -- 172.21.15.60:0/2102978344 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc280b3cc0 msgr2=0x7fdc280b2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:10.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.171+0000 7fdc2c889700 1 --2- 172.21.15.60:0/2102978344 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc280b3cc0 0x7fdc280b2040 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7fdc20002f70 tx=0x7fdc20006eb0).stop 2022-06-29T16:00:10.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.171+0000 7fdc2c889700 1 -- 172.21.15.60:0/2102978344 shutdown_connections 2022-06-29T16:00:10.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.171+0000 7fdc2c889700 1 --2- 172.21.15.60:0/2102978344 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc280b3cc0 0x7fdc280b2040 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:10.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.171+0000 7fdc2c889700 1 --2- 172.21.15.60:0/2102978344 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc2805d1f0 0x7fdc280b1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:10.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.171+0000 7fdc2c889700 1 -- 172.21.15.60:0/2102978344 >> 172.21.15.60:0/2102978344 conn(0x7fdc28062ce0 msgr2=0x7fdc280b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:10.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.171+0000 7fdc2c889700 1 -- 172.21.15.60:0/2102978344 shutdown_connections 2022-06-29T16:00:10.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.171+0000 7fdc2c889700 1 -- 172.21.15.60:0/2102978344 wait complete. 2022-06-29T16:00:10.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.171+0000 7fdc2c889700 1 Processor -- start 2022-06-29T16:00:10.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.171+0000 7fdc2c889700 1 -- start start 2022-06-29T16:00:10.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.171+0000 7fdc2c889700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc2805d1f0 0x7fdc28145a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:10.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.171+0000 7fdc2c889700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc280b3cc0 0x7fdc28152470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:10.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.171+0000 7fdc2c889700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc280b9d70 0x7fdc280b7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:10.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.171+0000 7fdc2c889700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc28144b20 con 0x7fdc280b3cc0 2022-06-29T16:00:10.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.171+0000 7fdc2c889700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc28155500 con 0x7fdc280b9d70 2022-06-29T16:00:10.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.171+0000 7fdc2c889700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc28151d30 con 0x7fdc2805d1f0 2022-06-29T16:00:10.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.171+0000 7fdc25d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc280b3cc0 0x7fdc28152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:10.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.171+0000 7fdc2659c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc2805d1f0 0x7fdc28145a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:10.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.171+0000 7fdc26d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc280b9d70 0x7fdc280b7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:10.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.171+0000 7fdc2659c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc2805d1f0 0x7fdc28145a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:59882/0 (socket says 172.21.15.60:59882) 2022-06-29T16:00:10.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.171+0000 7fdc2659c700 1 -- 172.21.15.60:0/3599368946 learned_addr learned my addr 172.21.15.60:0/3599368946 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:10.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.175+0000 7fdc2659c700 1 -- 172.21.15.60:0/3599368946 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc280b9d70 msgr2=0x7fdc280b7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:10.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.175+0000 7fdc2659c700 1 --2- 172.21.15.60:0/3599368946 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc280b9d70 0x7fdc280b7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:10.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.175+0000 7fdc2659c700 1 -- 172.21.15.60:0/3599368946 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc280b3cc0 msgr2=0x7fdc28152470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:10.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.175+0000 7fdc2659c700 1 --2- 172.21.15.60:0/3599368946 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc280b3cc0 0x7fdc28152470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:10.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.175+0000 7fdc2659c700 1 -- 172.21.15.60:0/3599368946 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc2000b1e0 con 0x7fdc2805d1f0 2022-06-29T16:00:10.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.175+0000 7fdc2659c700 1 --2- 172.21.15.60:0/3599368946 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc2805d1f0 0x7fdc28145a00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fdc281441b0 tx=0x7fdc18001400).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:10.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.175+0000 7fdc1f7fe700 1 -- 172.21.15.60:0/3599368946 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc1800e4a0 con 0x7fdc2805d1f0 2022-06-29T16:00:10.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.175+0000 7fdc1f7fe700 1 -- 172.21.15.60:0/3599368946 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc1800e600 con 0x7fdc2805d1f0 2022-06-29T16:00:10.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.175+0000 7fdc1f7fe700 1 -- 172.21.15.60:0/3599368946 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc1800e960 con 0x7fdc2805d1f0 2022-06-29T16:00:10.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.175+0000 7fdc2c889700 1 -- 172.21.15.60:0/3599368946 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc28151fd0 con 0x7fdc2805d1f0 2022-06-29T16:00:10.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.175+0000 7fdc2c889700 1 -- 172.21.15.60:0/3599368946 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc28144cc0 con 0x7fdc2805d1f0 2022-06-29T16:00:10.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.175+0000 7fdc2c889700 1 -- 172.21.15.60:0/3599368946 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc280573d0 con 0x7fdc2805d1f0 2022-06-29T16:00:10.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.175+0000 7fdc1f7fe700 1 -- 172.21.15.60:0/3599368946 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdc18019030 con 0x7fdc2805d1f0 2022-06-29T16:00:10.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.179+0000 7fdc1f7fe700 1 --2- 172.21.15.60:0/3599368946 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdc0c053a70 0x7fdc0c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:10.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.179+0000 7fdc1f7fe700 1 -- 172.21.15.60:0/3599368946 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fdc1806b760 con 0x7fdc2805d1f0 2022-06-29T16:00:10.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.179+0000 7fdc25d9b700 1 --2- 172.21.15.60:0/3599368946 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdc0c053a70 0x7fdc0c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:10.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.179+0000 7fdc25d9b700 1 --2- 172.21.15.60:0/3599368946 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdc0c053a70 0x7fdc0c055f30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fdc1000b000 tx=0x7fdc10006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:10.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.187+0000 7fdc1f7fe700 1 -- 172.21.15.60:0/3599368946 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdc180424a0 con 0x7fdc2805d1f0 2022-06-29T16:00:10.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:10.315+0000 7fdc2c889700 1 -- 172.21.15.60:0/3599368946 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdc28067bc0 con 0x7fdc2805d1f0 2022-06-29T16:00:11.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.039+0000 7fdc1f7fe700 1 -- 172.21.15.60:0/3599368946 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v34) v1 ==== 185+0+0 (secure 0 0 0) 0x7fdc18045ac0 con 0x7fdc2805d1f0 2022-06-29T16:00:11.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.043+0000 7fdc2c889700 1 -- 172.21.15.60:0/3599368946 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdc0c053a70 msgr2=0x7fdc0c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:11.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.043+0000 7fdc2c889700 1 --2- 172.21.15.60:0/3599368946 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdc0c053a70 0x7fdc0c055f30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fdc1000b000 tx=0x7fdc10006df0).stop 2022-06-29T16:00:11.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.043+0000 7fdc2c889700 1 -- 172.21.15.60:0/3599368946 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc2805d1f0 msgr2=0x7fdc28145a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:11.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.043+0000 7fdc2c889700 1 --2- 172.21.15.60:0/3599368946 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc2805d1f0 0x7fdc28145a00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fdc281441b0 tx=0x7fdc18001400).stop 2022-06-29T16:00:11.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.043+0000 7fdc2c889700 1 -- 172.21.15.60:0/3599368946 shutdown_connections 2022-06-29T16:00:11.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.043+0000 7fdc2c889700 1 --2- 172.21.15.60:0/3599368946 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdc0c053a70 0x7fdc0c055f30 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:11.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.043+0000 7fdc2c889700 1 --2- 172.21.15.60:0/3599368946 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc280b9d70 0x7fdc280b7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:11.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.043+0000 7fdc2c889700 1 --2- 172.21.15.60:0/3599368946 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc280b3cc0 0x7fdc28152470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:11.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.043+0000 7fdc2c889700 1 --2- 172.21.15.60:0/3599368946 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc2805d1f0 0x7fdc28145a00 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:11.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.043+0000 7fdc2c889700 1 -- 172.21.15.60:0/3599368946 >> 172.21.15.60:0/3599368946 conn(0x7fdc28062ce0 msgr2=0x7fdc280b8380 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:11.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.043+0000 7fdc2c889700 1 -- 172.21.15.60:0/3599368946 shutdown_connections 2022-06-29T16:00:11.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.043+0000 7fdc2c889700 1 -- 172.21.15.60:0/3599368946 wait complete. 2022-06-29T16:00:11.056 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2022-06-29T16:00:11.073 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-06-29T16:00:11.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.183+0000 7fe627fff700 1 Processor -- start 2022-06-29T16:00:11.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.183+0000 7fe627fff700 1 -- start start 2022-06-29T16:00:11.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.183+0000 7fe627fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe6280b9be0 0x7fe62805bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:11.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.183+0000 7fe627fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe6280b28e0 0x7fe628141e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:11.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.183+0000 7fe627fff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6280b1d70 con 0x7fe6280afee0 2022-06-29T16:00:11.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.183+0000 7fe627fff700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6280b9fc0 con 0x7fe6280b28e0 2022-06-29T16:00:11.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.183+0000 7fe627fff700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6280b3c40 con 0x7fe6280b9be0 2022-06-29T16:00:11.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.187+0000 7fe6277fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe6280b28e0 0x7fe628141e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:11.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.187+0000 7fe6277fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe6280b28e0 0x7fe628141e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46798/0 (socket says 172.21.15.60:46798) 2022-06-29T16:00:11.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.187+0000 7fe6277fe700 1 -- 172.21.15.60:0/3999614522 learned_addr learned my addr 172.21.15.60:0/3999614522 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:11.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.187+0000 7fe626ffd700 1 --2- 172.21.15.60:0/3999614522 >> v2:172.21.15.92:3300/0 conn(0x7fe6280b9be0 0x7fe62805bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:11.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.187+0000 7fe6277fe700 1 -- 172.21.15.60:0/3999614522 >> v2:172.21.15.92:3300/0 conn(0x7fe6280b9be0 msgr2=0x7fe62805bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:11.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.187+0000 7fe6277fe700 1 --2- 172.21.15.60:0/3999614522 >> v2:172.21.15.92:3300/0 conn(0x7fe6280b9be0 0x7fe62805bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:11.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.187+0000 7fe6277fe700 1 -- 172.21.15.60:0/3999614522 >> v1:172.21.15.92:6789/0 conn(0x7fe6280afee0 legacy=0x7fe6280b21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:11.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.187+0000 7fe6277fe700 1 -- 172.21.15.60:0/3999614522 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6280b3ee0 con 0x7fe6280b28e0 2022-06-29T16:00:11.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.187+0000 7fe6277fe700 1 --2- 172.21.15.60:0/3999614522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6280b28e0 0x7fe628141e70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7fe6200032f0 tx=0x7fe620009290).ready entity=mon.0 client_cookie=5f229091aff9e863 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:11.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.187+0000 7fe625ffb700 1 -- 172.21.15.60:0/3999614522 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe62000b870 con 0x7fe6280b28e0 2022-06-29T16:00:11.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.187+0000 7fe625ffb700 1 -- 172.21.15.60:0/3999614522 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe62000b9d0 con 0x7fe6280b28e0 2022-06-29T16:00:11.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.187+0000 7fe625ffb700 1 -- 172.21.15.60:0/3999614522 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe62000bb40 con 0x7fe6280b28e0 2022-06-29T16:00:11.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.187+0000 7fe627fff700 1 -- 172.21.15.60:0/3999614522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6280b28e0 msgr2=0x7fe628141e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:11.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.187+0000 7fe627fff700 1 --2- 172.21.15.60:0/3999614522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6280b28e0 0x7fe628141e70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7fe6200032f0 tx=0x7fe620009290).stop 2022-06-29T16:00:11.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.187+0000 7fe627fff700 1 -- 172.21.15.60:0/3999614522 shutdown_connections 2022-06-29T16:00:11.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.187+0000 7fe627fff700 1 --2- 172.21.15.60:0/3999614522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6280b28e0 0x7fe628141e70 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:11.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.187+0000 7fe627fff700 1 --2- 172.21.15.60:0/3999614522 >> v2:172.21.15.92:3300/0 conn(0x7fe6280b9be0 0x7fe62805bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:11.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.187+0000 7fe627fff700 1 -- 172.21.15.60:0/3999614522 >> 172.21.15.60:0/3999614522 conn(0x7fe628062ce0 msgr2=0x7fe6280b9190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:11.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.187+0000 7fe627fff700 1 -- 172.21.15.60:0/3999614522 shutdown_connections 2022-06-29T16:00:11.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.187+0000 7fe627fff700 1 -- 172.21.15.60:0/3999614522 wait complete. 2022-06-29T16:00:11.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.187+0000 7fe627fff700 1 Processor -- start 2022-06-29T16:00:11.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.191+0000 7fe627fff700 1 -- start start 2022-06-29T16:00:11.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.191+0000 7fe627fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6280afee0 0x7fe6280b8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:11.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.191+0000 7fe627fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6280b28e0 0x7fe6280af210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:11.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.191+0000 7fe627fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6280b9be0 0x7fe628151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:11.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.191+0000 7fe627fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe62813dbc0 con 0x7fe6280b28e0 2022-06-29T16:00:11.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.191+0000 7fe627fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6280aebe0 con 0x7fe6280afee0 2022-06-29T16:00:11.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.191+0000 7fe627fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6281518b0 con 0x7fe6280b9be0 2022-06-29T16:00:11.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.191+0000 7fe6267fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6280b28e0 0x7fe6280af210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:11.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.191+0000 7fe6267fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6280b28e0 0x7fe6280af210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46804/0 (socket says 172.21.15.60:46804) 2022-06-29T16:00:11.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.191+0000 7fe6267fc700 1 -- 172.21.15.60:0/2614913489 learned_addr learned my addr 172.21.15.60:0/2614913489 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:11.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.191+0000 7fe626ffd700 1 --2- 172.21.15.60:0/2614913489 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6280afee0 0x7fe6280b8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:11.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.191+0000 7fe6277fe700 1 --2- 172.21.15.60:0/2614913489 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6280b9be0 0x7fe628151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:11.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.191+0000 7fe6267fc700 1 -- 172.21.15.60:0/2614913489 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6280b9be0 msgr2=0x7fe628151ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:11.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.191+0000 7fe6267fc700 1 --2- 172.21.15.60:0/2614913489 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6280b9be0 0x7fe628151ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:11.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.191+0000 7fe6267fc700 1 -- 172.21.15.60:0/2614913489 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6280afee0 msgr2=0x7fe6280b8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:11.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.191+0000 7fe6267fc700 1 --2- 172.21.15.60:0/2614913489 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6280afee0 0x7fe6280b8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:11.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.191+0000 7fe6267fc700 1 -- 172.21.15.60:0/2614913489 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe620009530 con 0x7fe6280b28e0 2022-06-29T16:00:11.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.191+0000 7fe6267fc700 1 --2- 172.21.15.60:0/2614913489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6280b28e0 0x7fe6280af210 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fe628060400 tx=0x7fe618007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:11.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.191+0000 7fe617fff700 1 -- 172.21.15.60:0/2614913489 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe61800ce30 con 0x7fe6280b28e0 2022-06-29T16:00:11.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.191+0000 7fe617fff700 1 -- 172.21.15.60:0/2614913489 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe61800f470 con 0x7fe6280b28e0 2022-06-29T16:00:11.216 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.191+0000 7fe627fff700 1 -- 172.21.15.60:0/2614913489 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6280b9360 con 0x7fe6280b28e0 2022-06-29T16:00:11.216 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.191+0000 7fe617fff700 1 -- 172.21.15.60:0/2614913489 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe61800f7d0 con 0x7fe6280b28e0 2022-06-29T16:00:11.216 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.191+0000 7fe627fff700 1 -- 172.21.15.60:0/2614913489 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6280ae6f0 con 0x7fe6280b28e0 2022-06-29T16:00:11.216 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.195+0000 7fe617fff700 1 -- 172.21.15.60:0/2614913489 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe618019030 con 0x7fe6280b28e0 2022-06-29T16:00:11.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.195+0000 7fe627fff700 1 -- 172.21.15.60:0/2614913489 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6280573d0 con 0x7fe6280b28e0 2022-06-29T16:00:11.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.195+0000 7fe617fff700 1 --2- 172.21.15.60:0/2614913489 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe604053a70 0x7fe604055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:11.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.195+0000 7fe617fff700 1 -- 172.21.15.60:0/2614913489 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5721+0+0 (secure 0 0 0) 0x7fe61806b1f0 con 0x7fe6280b28e0 2022-06-29T16:00:11.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.195+0000 7fe626ffd700 1 --2- 172.21.15.60:0/2614913489 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe604053a70 0x7fe604055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:11.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.195+0000 7fe626ffd700 1 --2- 172.21.15.60:0/2614913489 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe604053a70 0x7fe604055f30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fe61c00c000 tx=0x7fe61c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:11.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.199+0000 7fe617fff700 1 -- 172.21.15.60:0/2614913489 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe618042270 con 0x7fe6280b28e0 2022-06-29T16:00:11.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:11.331+0000 7fe627fff700 1 -- 172.21.15.60:0/2614913489 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe6280aeda0 con 0x7fe6280b28e0 2022-06-29T16:00:12.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.043+0000 7fe617fff700 1 -- 172.21.15.60:0/2614913489 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v35) v1 ==== 173+0+0 (secure 0 0 0) 0x7fe618045890 con 0x7fe6280b28e0 2022-06-29T16:00:12.058 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.047+0000 7fe627fff700 1 -- 172.21.15.60:0/2614913489 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe604053a70 msgr2=0x7fe604055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:12.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.047+0000 7fe627fff700 1 --2- 172.21.15.60:0/2614913489 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe604053a70 0x7fe604055f30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fe61c00c000 tx=0x7fe61c008040).stop 2022-06-29T16:00:12.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.047+0000 7fe627fff700 1 -- 172.21.15.60:0/2614913489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6280b28e0 msgr2=0x7fe6280af210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:12.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.047+0000 7fe627fff700 1 --2- 172.21.15.60:0/2614913489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6280b28e0 0x7fe6280af210 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fe628060400 tx=0x7fe618007c50).stop 2022-06-29T16:00:12.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.047+0000 7fe627fff700 1 -- 172.21.15.60:0/2614913489 shutdown_connections 2022-06-29T16:00:12.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.047+0000 7fe627fff700 1 --2- 172.21.15.60:0/2614913489 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe604053a70 0x7fe604055f30 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:12.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.047+0000 7fe627fff700 1 --2- 172.21.15.60:0/2614913489 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6280b9be0 0x7fe628151ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:12.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.047+0000 7fe627fff700 1 --2- 172.21.15.60:0/2614913489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6280b28e0 0x7fe6280af210 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:12.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.047+0000 7fe627fff700 1 --2- 172.21.15.60:0/2614913489 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6280afee0 0x7fe6280b8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:12.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.051+0000 7fe627fff700 1 -- 172.21.15.60:0/2614913489 >> 172.21.15.60:0/2614913489 conn(0x7fe628062ce0 msgr2=0x7fe6280b61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:12.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.051+0000 7fe627fff700 1 -- 172.21.15.60:0/2614913489 shutdown_connections 2022-06-29T16:00:12.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.051+0000 7fe627fff700 1 -- 172.21.15.60:0/2614913489 wait complete. 2022-06-29T16:00:12.062 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2022-06-29T16:00:12.073 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-06-29T16:00:12.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.179+0000 7f430ad73700 1 Processor -- start 2022-06-29T16:00:12.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.179+0000 7f430ad73700 1 -- start start 2022-06-29T16:00:12.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.179+0000 7f430ad73700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f43040b2270 0x7f43040b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:12.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.179+0000 7f430ad73700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f43040634d0 0x7f430413e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:12.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.179+0000 7f430ad73700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43040b2650 con 0x7f43040aca20 2022-06-29T16:00:12.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.179+0000 7f430ad73700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4304067770 con 0x7f43040b2270 2022-06-29T16:00:12.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.179+0000 7f430ad73700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4304062af0 con 0x7f43040634d0 2022-06-29T16:00:12.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.179+0000 7f4309d71700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f43040b2270 0x7f43040b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:12.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.179+0000 7f4309d71700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f43040b2270 0x7f43040b0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46810/0 (socket says 172.21.15.60:46810) 2022-06-29T16:00:12.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.179+0000 7f4309d71700 1 -- 172.21.15.60:0/3481981863 learned_addr learned my addr 172.21.15.60:0/3481981863 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:12.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.179+0000 7f4309570700 1 --2- 172.21.15.60:0/3481981863 >> v2:172.21.15.92:3300/0 conn(0x7f43040634d0 0x7f430413e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:12.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.179+0000 7f4309d71700 1 -- 172.21.15.60:0/3481981863 >> v2:172.21.15.92:3300/0 conn(0x7f43040634d0 msgr2=0x7f430413e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:12.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.179+0000 7f4309d71700 1 --2- 172.21.15.60:0/3481981863 >> v2:172.21.15.92:3300/0 conn(0x7f43040634d0 0x7f430413e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:12.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.179+0000 7f4309d71700 1 -- 172.21.15.60:0/3481981863 >> v1:172.21.15.92:6789/0 conn(0x7f43040aca20 legacy=0x7f430413e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:12.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.179+0000 7f4309d71700 1 -- 172.21.15.60:0/3481981863 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4304062d90 con 0x7f43040b2270 2022-06-29T16:00:12.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.179+0000 7f4309d71700 1 --2- 172.21.15.60:0/3481981863 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43040b2270 0x7f43040b0b40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f4300002f70 tx=0x7f4300006eb0).ready entity=mon.0 client_cookie=9e563c8565e83d0c server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:12.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.179+0000 7f4308d6f700 1 -- 172.21.15.60:0/3481981863 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f430000d5c0 con 0x7f43040b2270 2022-06-29T16:00:12.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.179+0000 7f4308d6f700 1 -- 172.21.15.60:0/3481981863 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4300000e90 con 0x7f43040b2270 2022-06-29T16:00:12.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.179+0000 7f4308d6f700 1 -- 172.21.15.60:0/3481981863 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f430000d950 con 0x7f43040b2270 2022-06-29T16:00:12.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.179+0000 7f430ad73700 1 -- 172.21.15.60:0/3481981863 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43040b2270 msgr2=0x7f43040b0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:12.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.179+0000 7f430ad73700 1 --2- 172.21.15.60:0/3481981863 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43040b2270 0x7f43040b0b40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f4300002f70 tx=0x7f4300006eb0).stop 2022-06-29T16:00:12.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.179+0000 7f430ad73700 1 -- 172.21.15.60:0/3481981863 shutdown_connections 2022-06-29T16:00:12.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.183+0000 7f430ad73700 1 --2- 172.21.15.60:0/3481981863 >> v2:172.21.15.92:3300/0 conn(0x7f43040634d0 0x7f430413e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:12.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.183+0000 7f430ad73700 1 --2- 172.21.15.60:0/3481981863 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43040b2270 0x7f43040b0b40 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:12.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.183+0000 7f430ad73700 1 -- 172.21.15.60:0/3481981863 >> 172.21.15.60:0/3481981863 conn(0x7f43040b5890 msgr2=0x7f43040b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:12.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.183+0000 7f430ad73700 1 -- 172.21.15.60:0/3481981863 shutdown_connections 2022-06-29T16:00:12.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.183+0000 7f430ad73700 1 -- 172.21.15.60:0/3481981863 wait complete. 2022-06-29T16:00:12.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.183+0000 7f430ad73700 1 Processor -- start 2022-06-29T16:00:12.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.183+0000 7f430ad73700 1 -- start start 2022-06-29T16:00:12.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.183+0000 7f430ad73700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43040634d0 0x7f43040b45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:12.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.183+0000 7f430ad73700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43040aca20 0x7f43041410d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:12.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.183+0000 7f430ad73700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43040b2270 0x7f4304064180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:12.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.183+0000 7f430ad73700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43040656b0 con 0x7f43040aca20 2022-06-29T16:00:12.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.183+0000 7f430ad73700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4304065bd0 con 0x7f43040634d0 2022-06-29T16:00:12.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.183+0000 7f430ad73700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4304134dd0 con 0x7f43040b2270 2022-06-29T16:00:12.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.183+0000 7f4309570700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43040aca20 0x7f43041410d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:12.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.183+0000 7f4309570700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43040aca20 0x7f43041410d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46818/0 (socket says 172.21.15.60:46818) 2022-06-29T16:00:12.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.187+0000 7f4309570700 1 -- 172.21.15.60:0/863531483 learned_addr learned my addr 172.21.15.60:0/863531483 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:12.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.187+0000 7f4309d71700 1 --2- 172.21.15.60:0/863531483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43040634d0 0x7f43040b45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:12.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.187+0000 7f430a572700 1 --2- 172.21.15.60:0/863531483 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43040b2270 0x7f4304064180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:12.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.187+0000 7f4309570700 1 -- 172.21.15.60:0/863531483 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43040b2270 msgr2=0x7f4304064180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:12.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.187+0000 7f4309570700 1 --2- 172.21.15.60:0/863531483 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43040b2270 0x7f4304064180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:12.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.187+0000 7f4309570700 1 -- 172.21.15.60:0/863531483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43040634d0 msgr2=0x7f43040b45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:12.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.187+0000 7f4309570700 1 --2- 172.21.15.60:0/863531483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43040634d0 0x7f43040b45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:12.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.187+0000 7f4309570700 1 -- 172.21.15.60:0/863531483 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f430000b1e0 con 0x7f43040aca20 2022-06-29T16:00:12.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.187+0000 7f430a572700 1 --2- 172.21.15.60:0/863531483 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43040b2270 0x7f4304064180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:00:12.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.187+0000 7f4309570700 1 --2- 172.21.15.60:0/863531483 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43040aca20 0x7f43041410d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f4304134f60 tx=0x7f42f4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:12.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.187+0000 7f42f2ffd700 1 -- 172.21.15.60:0/863531483 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f42f4010800 con 0x7f43040aca20 2022-06-29T16:00:12.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.187+0000 7f42f2ffd700 1 -- 172.21.15.60:0/863531483 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42f4010960 con 0x7f43040aca20 2022-06-29T16:00:12.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.187+0000 7f430ad73700 1 -- 172.21.15.60:0/863531483 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4304135040 con 0x7f43040aca20 2022-06-29T16:00:12.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.187+0000 7f42f2ffd700 1 -- 172.21.15.60:0/863531483 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f42f4010cc0 con 0x7f43040aca20 2022-06-29T16:00:12.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.187+0000 7f430ad73700 1 -- 172.21.15.60:0/863531483 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43041415c0 con 0x7f43040aca20 2022-06-29T16:00:12.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.187+0000 7f430ad73700 1 -- 172.21.15.60:0/863531483 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f430413ec90 con 0x7f43040aca20 2022-06-29T16:00:12.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.187+0000 7f42f2ffd700 1 -- 172.21.15.60:0/863531483 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f42f4012040 con 0x7f43040aca20 2022-06-29T16:00:12.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.191+0000 7f42f2ffd700 1 --2- 172.21.15.60:0/863531483 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f42e40539e0 0x7f42e4055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:12.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.191+0000 7f42f2ffd700 1 -- 172.21.15.60:0/863531483 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f42f406e300 con 0x7f43040aca20 2022-06-29T16:00:12.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.191+0000 7f4309d71700 1 --2- 172.21.15.60:0/863531483 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f42e40539e0 0x7f42e4055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:12.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.191+0000 7f4309d71700 1 --2- 172.21.15.60:0/863531483 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f42e40539e0 0x7f42e4055ea0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f430001c000 tx=0x7f4300015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:12.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.195+0000 7f42f2ffd700 1 -- 172.21.15.60:0/863531483 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f42f4045500 con 0x7f43040aca20 2022-06-29T16:00:12.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.331+0000 7f430ad73700 1 -- 172.21.15.60:0/863531483 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f4304138010 con 0x7f43040aca20 2022-06-29T16:00:12.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.339+0000 7f42f2ffd700 1 -- 172.21.15.60:0/863531483 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v3) v1 ==== 104+0+0 (secure 0 0 0) 0x7f42f4048b20 con 0x7f43040aca20 2022-06-29T16:00:12.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.339+0000 7f430ad73700 1 -- 172.21.15.60:0/863531483 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f42e40539e0 msgr2=0x7f42e4055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:12.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.339+0000 7f430ad73700 1 --2- 172.21.15.60:0/863531483 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f42e40539e0 0x7f42e4055ea0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f430001c000 tx=0x7f4300015000).stop 2022-06-29T16:00:12.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.339+0000 7f430ad73700 1 -- 172.21.15.60:0/863531483 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43040aca20 msgr2=0x7f43041410d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:12.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.339+0000 7f430ad73700 1 --2- 172.21.15.60:0/863531483 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43040aca20 0x7f43041410d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f4304134f60 tx=0x7f42f4006df0).stop 2022-06-29T16:00:12.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.339+0000 7f430ad73700 1 -- 172.21.15.60:0/863531483 shutdown_connections 2022-06-29T16:00:12.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.339+0000 7f430ad73700 1 --2- 172.21.15.60:0/863531483 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f42e40539e0 0x7f42e4055ea0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:12.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.339+0000 7f430ad73700 1 --2- 172.21.15.60:0/863531483 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43040b2270 0x7f4304064180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:12.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.339+0000 7f430ad73700 1 --2- 172.21.15.60:0/863531483 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43040aca20 0x7f43041410d0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:12.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.339+0000 7f430ad73700 1 --2- 172.21.15.60:0/863531483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43040634d0 0x7f43040b45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:12.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.339+0000 7f430ad73700 1 -- 172.21.15.60:0/863531483 >> 172.21.15.60:0/863531483 conn(0x7f43040b5890 msgr2=0x7f43040aff00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:12.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.339+0000 7f430ad73700 1 -- 172.21.15.60:0/863531483 shutdown_connections 2022-06-29T16:00:12.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.339+0000 7f430ad73700 1 -- 172.21.15.60:0/863531483 wait complete. 2022-06-29T16:00:12.362 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command' 2022-06-29T16:00:12.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.459+0000 7f3b513b3700 1 Processor -- start 2022-06-29T16:00:12.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b513b3700 1 -- start start 2022-06-29T16:00:12.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b513b3700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b4c0b5eb0 0x7f3b4c142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:12.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b513b3700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b4c0b0cf0 con 0x7f3b4c0b48b0 2022-06-29T16:00:12.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b513b3700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b4c0b4d10 con 0x7f3b4c0b2c50 2022-06-29T16:00:12.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b513b3700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4c0b54e0 con 0x7f3b4c0b5eb0 2022-06-29T16:00:12.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b437fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f3b4c0b48b0 0x7f3b4c0b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54910/0 (socket says 172.21.15.60:54910) 2022-06-29T16:00:12.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b437fe700 1 -- 172.21.15.60:0/2200937818 learned_addr learned my addr 172.21.15.60:0/2200937818 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:00:12.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b4bfff700 1 --2- 172.21.15.60:0/2200937818 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b4c0b5eb0 0x7f3b4c142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:12.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b4b7fe700 1 -- 172.21.15.60:0/2200937818 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4253005284 0 0) 0x7f3b34001110 con 0x7f3b4c0b48b0 2022-06-29T16:00:12.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b4b7fe700 1 -- 172.21.15.60:0/2200937818 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b2c003260 con 0x7f3b4c0b48b0 2022-06-29T16:00:12.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b4b7fe700 1 -- 172.21.15.60:0/2200937818 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 97043437 0 0) 0x7f3b2c003260 con 0x7f3b4c0b48b0 2022-06-29T16:00:12.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b4b7fe700 1 -- 172.21.15.60:0/2200937818 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b4c0b5eb0 msgr2=0x7f3b4c142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:12.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b4b7fe700 1 --2- 172.21.15.60:0/2200937818 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b4c0b5eb0 0x7f3b4c142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:12.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b4b7fe700 1 -- 172.21.15.60:0/2200937818 >> v1:172.21.15.92:6789/0 conn(0x7f3b4c0b2c50 legacy=0x7f3b4c0b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:12.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b4b7fe700 1 -- 172.21.15.60:0/2200937818 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b4c0b6290 con 0x7f3b4c0b48b0 2022-06-29T16:00:12.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b4b7fe700 1 -- 172.21.15.60:0/2200937818 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f3b340017f0 con 0x7f3b4c0b48b0 2022-06-29T16:00:12.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b4b7fe700 1 -- 172.21.15.60:0/2200937818 >> v1:172.21.15.60:6789/0 conn(0x7f3b4c0b48b0 legacy=0x7f3b4c0b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:12.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b4b7fe700 1 --2- 172.21.15.60:0/2200937818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b2c003780 0x7f3b2c003b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:12.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b4b7fe700 1 --2- 172.21.15.60:0/2200937818 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b4c0b5eb0 0x7f3b2c004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:12.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b4b7fe700 1 --2- 172.21.15.60:0/2200937818 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b2c0049f0 0x7f3b2c004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:12.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b4b7fe700 1 -- 172.21.15.60:0/2200937818 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4c0b54e0 con 0x7f3b2c0049f0 2022-06-29T16:00:12.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b4b7fe700 1 -- 172.21.15.60:0/2200937818 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c00b750 con 0x7f3b2c003780 2022-06-29T16:00:12.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b4b7fe700 1 -- 172.21.15.60:0/2200937818 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c00b8c0 con 0x7f3b4c0b5eb0 2022-06-29T16:00:12.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b437fe700 1 --2- 172.21.15.60:0/2200937818 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b4c0b5eb0 0x7f3b2c004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:12.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b4bfff700 1 --2- 172.21.15.60:0/2200937818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b2c003780 0x7f3b2c003b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:12.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b50bb2700 1 --2- 172.21.15.60:0/2200937818 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b2c0049f0 0x7f3b2c004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:12.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b50bb2700 1 -- 172.21.15.60:0/2200937818 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b4c0b5eb0 msgr2=0x7f3b2c004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:12.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b50bb2700 1 --2- 172.21.15.60:0/2200937818 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b4c0b5eb0 0x7f3b2c004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:12.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b50bb2700 1 -- 172.21.15.60:0/2200937818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b2c003780 msgr2=0x7f3b2c003b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:12.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b50bb2700 1 --2- 172.21.15.60:0/2200937818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b2c003780 0x7f3b2c003b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:12.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b50bb2700 1 -- 172.21.15.60:0/2200937818 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2c003260 con 0x7f3b2c0049f0 2022-06-29T16:00:12.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b50bb2700 1 --2- 172.21.15.60:0/2200937818 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b2c0049f0 0x7f3b2c004e70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f3b440013a0 tx=0x7f3b44007d60).ready entity=mon.0 client_cookie=7e1e7cef0ec07558 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:12.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b4b7fe700 1 -- 172.21.15.60:0/2200937818 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b440016c0 con 0x7f3b2c0049f0 2022-06-29T16:00:12.486 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b4b7fe700 1 -- 172.21.15.60:0/2200937818 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b44006890 con 0x7f3b2c0049f0 2022-06-29T16:00:12.487 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.463+0000 7f3b4b7fe700 1 -- 172.21.15.60:0/2200937818 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b44006bf0 con 0x7f3b2c0049f0 2022-06-29T16:00:12.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.467+0000 7f3b513b3700 1 -- 172.21.15.60:0/2200937818 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b2c0049f0 msgr2=0x7f3b2c004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:12.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.467+0000 7f3b513b3700 1 --2- 172.21.15.60:0/2200937818 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b2c0049f0 0x7f3b2c004e70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f3b440013a0 tx=0x7f3b44007d60).stop 2022-06-29T16:00:12.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.467+0000 7f3b50bb2700 1 -- 172.21.15.60:0/2200937818 reap_dead start 2022-06-29T16:00:12.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.467+0000 7f3b513b3700 1 -- 172.21.15.60:0/2200937818 shutdown_connections 2022-06-29T16:00:12.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.467+0000 7f3b513b3700 1 -- 172.21.15.60:0/2200937818 >> 172.21.15.60:0/2200937818 conn(0x7f3b4c062090 msgr2=0x7f3b4c062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:12.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.467+0000 7f3b513b3700 1 -- 172.21.15.60:0/2200937818 shutdown_connections 2022-06-29T16:00:12.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.467+0000 7f3b513b3700 1 -- 172.21.15.60:0/2200937818 wait complete. 2022-06-29T16:00:12.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.467+0000 7f3b513b3700 1 Processor -- start 2022-06-29T16:00:12.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.467+0000 7f3b513b3700 1 -- start start 2022-06-29T16:00:12.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.467+0000 7f3b513b3700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b4c144150 0x7f3b4c0af520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:12.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.467+0000 7f3b513b3700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b4c148f30 0x7f3b4c147890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:12.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.467+0000 7f3b513b3700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b4c147d80 0x7f3b4c155500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:12.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.467+0000 7f3b513b3700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4c13fa30 con 0x7f3b4c144150 2022-06-29T16:00:12.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.467+0000 7f3b513b3700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4c1471c0 con 0x7f3b4c148f30 2022-06-29T16:00:12.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.467+0000 7f3b513b3700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4c144530 con 0x7f3b4c147d80 2022-06-29T16:00:12.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.467+0000 7f3b4bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b4c144150 0x7f3b4c0af520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:12.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.467+0000 7f3b4bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b4c144150 0x7f3b4c0af520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46834/0 (socket says 172.21.15.60:46834) 2022-06-29T16:00:12.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.467+0000 7f3b4bfff700 1 -- 172.21.15.60:0/4006494289 learned_addr learned my addr 172.21.15.60:0/4006494289 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:12.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.467+0000 7f3b50bb2700 1 --2- 172.21.15.60:0/4006494289 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b4c147d80 0x7f3b4c155500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:12.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.467+0000 7f3b437fe700 1 --2- 172.21.15.60:0/4006494289 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b4c148f30 0x7f3b4c147890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:12.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.467+0000 7f3b4bfff700 1 -- 172.21.15.60:0/4006494289 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b4c147d80 msgr2=0x7f3b4c155500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:12.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.467+0000 7f3b4bfff700 1 --2- 172.21.15.60:0/4006494289 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b4c147d80 0x7f3b4c155500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:12.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.467+0000 7f3b4bfff700 1 -- 172.21.15.60:0/4006494289 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b4c148f30 msgr2=0x7f3b4c147890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:12.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.467+0000 7f3b4bfff700 1 --2- 172.21.15.60:0/4006494289 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b4c148f30 0x7f3b4c147890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:12.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.467+0000 7f3b4bfff700 1 -- 172.21.15.60:0/4006494289 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b4c148410 con 0x7f3b4c144150 2022-06-29T16:00:12.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.467+0000 7f3b4bfff700 1 --2- 172.21.15.60:0/4006494289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b4c144150 0x7f3b4c0af520 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f3b3c00b000 tx=0x7f3b3c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:12.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.467+0000 7f3b49ffb700 1 -- 172.21.15.60:0/4006494289 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b3c011e90 con 0x7f3b4c144150 2022-06-29T16:00:12.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.467+0000 7f3b49ffb700 1 -- 172.21.15.60:0/4006494289 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b3c00f940 con 0x7f3b4c144150 2022-06-29T16:00:12.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.467+0000 7f3b513b3700 1 -- 172.21.15.60:0/4006494289 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b4c1473f0 con 0x7f3b4c144150 2022-06-29T16:00:12.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.467+0000 7f3b49ffb700 1 -- 172.21.15.60:0/4006494289 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b3c00fce0 con 0x7f3b4c144150 2022-06-29T16:00:12.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.467+0000 7f3b513b3700 1 -- 172.21.15.60:0/4006494289 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b4c1559f0 con 0x7f3b4c144150 2022-06-29T16:00:12.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.471+0000 7f3b513b3700 1 -- 172.21.15.60:0/4006494289 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b4c148190 con 0x7f3b4c144150 2022-06-29T16:00:12.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.471+0000 7f3b49ffb700 1 -- 172.21.15.60:0/4006494289 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3b3c014440 con 0x7f3b4c144150 2022-06-29T16:00:12.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.471+0000 7f3b49ffb700 1 --2- 172.21.15.60:0/4006494289 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3b30053a70 0x7f3b30055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:12.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.471+0000 7f3b49ffb700 1 -- 172.21.15.60:0/4006494289 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f3b3c06c600 con 0x7f3b4c144150 2022-06-29T16:00:12.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.471+0000 7f3b437fe700 1 --2- 172.21.15.60:0/4006494289 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3b30053a70 0x7f3b30055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:12.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.471+0000 7f3b437fe700 1 --2- 172.21.15.60:0/4006494289 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3b30053a70 0x7f3b30055f30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f3b34005000 tx=0x7f3b4c0b1780).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:12.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.475+0000 7f3b49ffb700 1 -- 172.21.15.60:0/4006494289 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3b3c0437a0 con 0x7f3b4c144150 2022-06-29T16:00:12.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:12.651+0000 7f3b513b3700 1 -- 172.21.15.60:0/4006494289 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command"]} v 0) v1 -- 0x7f3b4c147580 con 0x7f3b4c144150 2022-06-29T16:00:13.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.043+0000 7f3b49ffb700 1 -- 172.21.15.60:0/4006494289 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command"]}]=0 v96) v1 ==== 139+0+0 (secure 0 0 0) 0x7f3b3c018020 con 0x7f3b4c144150 2022-06-29T16:00:13.058 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.047+0000 7f3b513b3700 1 -- 172.21.15.60:0/4006494289 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3b30053a70 msgr2=0x7f3b30055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:13.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.047+0000 7f3b513b3700 1 --2- 172.21.15.60:0/4006494289 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3b30053a70 0x7f3b30055f30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f3b34005000 tx=0x7f3b4c0b1780).stop 2022-06-29T16:00:13.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.047+0000 7f3b513b3700 1 -- 172.21.15.60:0/4006494289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b4c144150 msgr2=0x7f3b4c0af520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:13.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.047+0000 7f3b513b3700 1 --2- 172.21.15.60:0/4006494289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b4c144150 0x7f3b4c0af520 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f3b3c00b000 tx=0x7f3b3c006df0).stop 2022-06-29T16:00:13.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.051+0000 7f3b513b3700 1 -- 172.21.15.60:0/4006494289 shutdown_connections 2022-06-29T16:00:13.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.051+0000 7f3b513b3700 1 --2- 172.21.15.60:0/4006494289 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3b30053a70 0x7f3b30055f30 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:13.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.051+0000 7f3b513b3700 1 --2- 172.21.15.60:0/4006494289 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b4c147d80 0x7f3b4c155500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:13.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.051+0000 7f3b513b3700 1 --2- 172.21.15.60:0/4006494289 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b4c148f30 0x7f3b4c147890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:13.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.051+0000 7f3b513b3700 1 --2- 172.21.15.60:0/4006494289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b4c144150 0x7f3b4c0af520 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:13.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.051+0000 7f3b513b3700 1 -- 172.21.15.60:0/4006494289 >> 172.21.15.60:0/4006494289 conn(0x7f3b4c062090 msgr2=0x7f3b4c141af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:13.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.051+0000 7f3b513b3700 1 -- 172.21.15.60:0/4006494289 shutdown_connections 2022-06-29T16:00:13.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.051+0000 7f3b513b3700 1 -- 172.21.15.60:0/4006494289 wait complete. 2022-06-29T16:00:13.072 INFO:tasks.cephfs_test_runner:test_drop_cache_command (tasks.cephfs.test_misc.TestCacheDrop) ... ok 2022-06-29T16:00:13.073 INFO:tasks.cephfs_test_runner:Starting test: test_drop_cache_command_dead (tasks.cephfs.test_misc.TestCacheDrop) 2022-06-29T16:00:13.074 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead' 2022-06-29T16:00:13.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc3a4d59700 1 Processor -- start 2022-06-29T16:00:13.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc3a4d59700 1 -- start start 2022-06-29T16:00:13.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc3a4d59700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3a00b2130 0x7fc3a0146100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:13.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc3a4d59700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc3a00aed00 0x7fc3a01465f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:13.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc3a4d59700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a005d450 con 0x7fc3a005d070 2022-06-29T16:00:13.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc3a4d59700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3a00adf90 con 0x7fc3a00aed00 2022-06-29T16:00:13.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc3a4d59700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a00b25d0 con 0x7fc3a00b2130 2022-06-29T16:00:13.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc39f7fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc3a005d070 0x7fc3a014aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54928/0 (socket says 172.21.15.60:54928) 2022-06-29T16:00:13.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc39f7fe700 1 -- 172.21.15.60:0/1140992002 learned_addr learned my addr 172.21.15.60:0/1140992002 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:00:13.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc396ffd700 1 --2- 172.21.15.60:0/1140992002 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3a00b2130 0x7fc3a0146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:13.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc39ffff700 1 --2- 172.21.15.60:0/1140992002 >> v2:172.21.15.92:3300/0 conn(0x7fc3a00aed00 0x7fc3a01465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:13.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc39effd700 1 -- 172.21.15.60:0/1140992002 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1047916574 0 0) 0x7fc3a005d450 con 0x7fc3a005d070 2022-06-29T16:00:13.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc39effd700 1 -- 172.21.15.60:0/1140992002 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc380003260 con 0x7fc3a005d070 2022-06-29T16:00:13.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc39effd700 1 -- 172.21.15.60:0/1140992002 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 581630128 0 0) 0x7fc380003260 con 0x7fc3a005d070 2022-06-29T16:00:13.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc39effd700 1 -- 172.21.15.60:0/1140992002 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3a00b2130 msgr2=0x7fc3a0146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:13.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc39effd700 1 --2- 172.21.15.60:0/1140992002 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3a00b2130 0x7fc3a0146100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:13.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc39effd700 1 -- 172.21.15.60:0/1140992002 >> v2:172.21.15.92:3300/0 conn(0x7fc3a00aed00 msgr2=0x7fc3a01465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:13.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc39effd700 1 --2- 172.21.15.60:0/1140992002 >> v2:172.21.15.92:3300/0 conn(0x7fc3a00aed00 0x7fc3a01465f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:13.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc39effd700 1 -- 172.21.15.60:0/1140992002 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a014a350 con 0x7fc3a005d070 2022-06-29T16:00:13.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc39effd700 1 -- 172.21.15.60:0/1140992002 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7fc390001680 con 0x7fc3a005d070 2022-06-29T16:00:13.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc39effd700 1 -- 172.21.15.60:0/1140992002 >> v1:172.21.15.60:6789/0 conn(0x7fc3a005d070 legacy=0x7fc3a014aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:13.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc39effd700 1 --2- 172.21.15.60:0/1140992002 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3a00b2130 0x7fc380003750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:13.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc39effd700 1 --2- 172.21.15.60:0/1140992002 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc380003c80 0x7fc380004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:13.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc39effd700 1 --2- 172.21.15.60:0/1140992002 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3800049f0 0x7fc380004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:13.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc39effd700 1 -- 172.21.15.60:0/1140992002 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a00adf90 con 0x7fc3800049f0 2022-06-29T16:00:13.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc39effd700 1 -- 172.21.15.60:0/1140992002 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a00b25d0 con 0x7fc380003c80 2022-06-29T16:00:13.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc39effd700 1 -- 172.21.15.60:0/1140992002 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc38000b750 con 0x7fc3a00b2130 2022-06-29T16:00:13.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc39ffff700 1 --2- 172.21.15.60:0/1140992002 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3800049f0 0x7fc380004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:13.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc39f7fe700 1 --2- 172.21.15.60:0/1140992002 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3a00b2130 0x7fc380003750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:13.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc396ffd700 1 --2- 172.21.15.60:0/1140992002 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc380003c80 0x7fc380004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:13.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc39f7fe700 1 -- 172.21.15.60:0/1140992002 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc380003c80 msgr2=0x7fc380004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:13.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc39f7fe700 1 --2- 172.21.15.60:0/1140992002 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc380003c80 0x7fc380004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:13.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc39f7fe700 1 -- 172.21.15.60:0/1140992002 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3800049f0 msgr2=0x7fc380004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:13.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc39f7fe700 1 --2- 172.21.15.60:0/1140992002 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3800049f0 0x7fc380004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:13.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc39f7fe700 1 -- 172.21.15.60:0/1140992002 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc380003260 con 0x7fc3a00b2130 2022-06-29T16:00:13.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc39ffff700 1 --2- 172.21.15.60:0/1140992002 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3800049f0 0x7fc380004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:00:13.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.175+0000 7fc39f7fe700 1 --2- 172.21.15.60:0/1140992002 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3a00b2130 0x7fc380003750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fc390001390 tx=0x7fc390001e70).ready entity=mon.2 client_cookie=d17be0f1f1ee4352 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:13.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.179+0000 7fc39effd700 1 -- 172.21.15.60:0/1140992002 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc390007170 con 0x7fc3a00b2130 2022-06-29T16:00:13.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.179+0000 7fc39effd700 1 -- 172.21.15.60:0/1140992002 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3900072d0 con 0x7fc3a00b2130 2022-06-29T16:00:13.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.179+0000 7fc39effd700 1 -- 172.21.15.60:0/1140992002 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc39000c8f0 con 0x7fc3a00b2130 2022-06-29T16:00:13.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.179+0000 7fc3a4d59700 1 -- 172.21.15.60:0/1140992002 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3a00b2130 msgr2=0x7fc380003750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:13.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.179+0000 7fc3a4d59700 1 --2- 172.21.15.60:0/1140992002 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3a00b2130 0x7fc380003750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fc390001390 tx=0x7fc390001e70).stop 2022-06-29T16:00:13.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.179+0000 7fc39ffff700 1 -- 172.21.15.60:0/1140992002 reap_dead start 2022-06-29T16:00:13.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.179+0000 7fc3a4d59700 1 -- 172.21.15.60:0/1140992002 shutdown_connections 2022-06-29T16:00:13.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.179+0000 7fc3a4d59700 1 -- 172.21.15.60:0/1140992002 >> 172.21.15.60:0/1140992002 conn(0x7fc3a00b5890 msgr2=0x7fc3a00b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:13.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.179+0000 7fc3a4d59700 1 -- 172.21.15.60:0/1140992002 shutdown_connections 2022-06-29T16:00:13.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.179+0000 7fc3a4d59700 1 -- 172.21.15.60:0/1140992002 wait complete. 2022-06-29T16:00:13.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.179+0000 7fc3a4d59700 1 Processor -- start 2022-06-29T16:00:13.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.183+0000 7fc3a4d59700 1 -- start start 2022-06-29T16:00:13.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.183+0000 7fc3a4d59700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3a00b2130 0x7fc3a0147490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:13.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.183+0000 7fc3a4d59700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3a014a9b0 0x7fc3a0156220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:13.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.183+0000 7fc3a4d59700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3a0149ae0 0x7fc3a0148650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:13.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.183+0000 7fc3a4d59700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a014d1d0 con 0x7fc3a014a9b0 2022-06-29T16:00:13.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.183+0000 7fc3a4d59700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a014c8e0 con 0x7fc3a0149ae0 2022-06-29T16:00:13.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.183+0000 7fc3a4d59700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a014b820 con 0x7fc3a00b2130 2022-06-29T16:00:13.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.183+0000 7fc396ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3a014a9b0 0x7fc3a0156220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:13.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.183+0000 7fc396ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3a014a9b0 0x7fc3a0156220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46856/0 (socket says 172.21.15.60:46856) 2022-06-29T16:00:13.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.183+0000 7fc396ffd700 1 -- 172.21.15.60:0/4255254862 learned_addr learned my addr 172.21.15.60:0/4255254862 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:13.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.183+0000 7fc39ffff700 1 --2- 172.21.15.60:0/4255254862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3a0149ae0 0x7fc3a0148650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:13.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.183+0000 7fc39f7fe700 1 --2- 172.21.15.60:0/4255254862 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3a00b2130 0x7fc3a0147490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:13.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.183+0000 7fc396ffd700 1 -- 172.21.15.60:0/4255254862 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3a00b2130 msgr2=0x7fc3a0147490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:13.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.183+0000 7fc396ffd700 1 --2- 172.21.15.60:0/4255254862 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3a00b2130 0x7fc3a0147490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:13.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.183+0000 7fc396ffd700 1 -- 172.21.15.60:0/4255254862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3a0149ae0 msgr2=0x7fc3a0148650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:13.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.183+0000 7fc396ffd700 1 --2- 172.21.15.60:0/4255254862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3a0149ae0 0x7fc3a0148650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:13.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.183+0000 7fc396ffd700 1 -- 172.21.15.60:0/4255254862 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a014bac0 con 0x7fc3a014a9b0 2022-06-29T16:00:13.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.183+0000 7fc39f7fe700 1 --2- 172.21.15.60:0/4255254862 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3a00b2130 0x7fc3a0147490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:00:13.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.183+0000 7fc396ffd700 1 --2- 172.21.15.60:0/4255254862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3a014a9b0 0x7fc3a0156220 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7fc3a0148530 tx=0x7fc388006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:13.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.183+0000 7fc39d7fa700 1 -- 172.21.15.60:0/4255254862 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc38800f940 con 0x7fc3a014a9b0 2022-06-29T16:00:13.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.183+0000 7fc39d7fa700 1 -- 172.21.15.60:0/4255254862 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc388000e90 con 0x7fc3a014a9b0 2022-06-29T16:00:13.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.183+0000 7fc3a4d59700 1 -- 172.21.15.60:0/4255254862 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a01481b0 con 0x7fc3a014a9b0 2022-06-29T16:00:13.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.183+0000 7fc39d7fa700 1 -- 172.21.15.60:0/4255254862 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc38800fc90 con 0x7fc3a014a9b0 2022-06-29T16:00:13.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.183+0000 7fc3a4d59700 1 -- 172.21.15.60:0/4255254862 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a0148ce0 con 0x7fc3a014a9b0 2022-06-29T16:00:13.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.183+0000 7fc3a4d59700 1 -- 172.21.15.60:0/4255254862 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a014a5f0 con 0x7fc3a014a9b0 2022-06-29T16:00:13.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.187+0000 7fc39d7fa700 1 -- 172.21.15.60:0/4255254862 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc388014440 con 0x7fc3a014a9b0 2022-06-29T16:00:13.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.187+0000 7fc39d7fa700 1 --2- 172.21.15.60:0/4255254862 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc384053aa0 0x7fc384055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:13.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.187+0000 7fc39d7fa700 1 -- 172.21.15.60:0/4255254862 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fc38806c5c0 con 0x7fc3a014a9b0 2022-06-29T16:00:13.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.187+0000 7fc39f7fe700 1 --2- 172.21.15.60:0/4255254862 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc384053aa0 0x7fc384055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:13.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.187+0000 7fc39f7fe700 1 --2- 172.21.15.60:0/4255254862 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc384053aa0 0x7fc384055f60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fc39001a000 tx=0x7fc390015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:13.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.195+0000 7fc39d7fa700 1 -- 172.21.15.60:0/4255254862 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc388043650 con 0x7fc3a014a9b0 2022-06-29T16:00:13.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:13.371+0000 7fc3a4d59700 1 -- 172.21.15.60:0/4255254862 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead"]} v 0) v1 -- 0x7fc3a0148340 con 0x7fc3a014a9b0 2022-06-29T16:00:14.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.047+0000 7fc39d7fa700 1 -- 172.21.15.60:0/4255254862 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead"]}]=0 v97) v1 ==== 147+0+0 (secure 0 0 0) 0x7fc388018020 con 0x7fc3a014a9b0 2022-06-29T16:00:14.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.051+0000 7fc3a4d59700 1 -- 172.21.15.60:0/4255254862 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc384053aa0 msgr2=0x7fc384055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:14.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.051+0000 7fc3a4d59700 1 --2- 172.21.15.60:0/4255254862 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc384053aa0 0x7fc384055f60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fc39001a000 tx=0x7fc390015000).stop 2022-06-29T16:00:14.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.051+0000 7fc3a4d59700 1 -- 172.21.15.60:0/4255254862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3a014a9b0 msgr2=0x7fc3a0156220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:14.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.051+0000 7fc3a4d59700 1 --2- 172.21.15.60:0/4255254862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3a014a9b0 0x7fc3a0156220 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7fc3a0148530 tx=0x7fc388006df0).stop 2022-06-29T16:00:14.063 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.051+0000 7fc3a4d59700 1 -- 172.21.15.60:0/4255254862 shutdown_connections 2022-06-29T16:00:14.063 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.051+0000 7fc3a4d59700 1 --2- 172.21.15.60:0/4255254862 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc384053aa0 0x7fc384055f60 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:14.064 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.051+0000 7fc3a4d59700 1 --2- 172.21.15.60:0/4255254862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3a0149ae0 0x7fc3a0148650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:14.064 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.051+0000 7fc3a4d59700 1 --2- 172.21.15.60:0/4255254862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3a014a9b0 0x7fc3a0156220 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:14.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.051+0000 7fc3a4d59700 1 --2- 172.21.15.60:0/4255254862 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3a00b2130 0x7fc3a0147490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:14.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.051+0000 7fc3a4d59700 1 -- 172.21.15.60:0/4255254862 >> 172.21.15.60:0/4255254862 conn(0x7fc3a00b5890 msgr2=0x7fc3a01445c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:14.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.051+0000 7fc3a4d59700 1 -- 172.21.15.60:0/4255254862 shutdown_connections 2022-06-29T16:00:14.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.051+0000 7fc3a4d59700 1 -- 172.21.15.60:0/4255254862 wait complete. 2022-06-29T16:00:14.079 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-06-29T16:00:14.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.183+0000 7fe50359e700 1 Processor -- start 2022-06-29T16:00:14.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.183+0000 7fe50359e700 1 -- start start 2022-06-29T16:00:14.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.187+0000 7fe50359e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe504060610 0x7fe50405c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:14.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.187+0000 7fe50359e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe50405fee0 0x7fe504142a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:14.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.187+0000 7fe50359e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5040b0a40 con 0x7fe50405c5b0 2022-06-29T16:00:14.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.187+0000 7fe50359e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5040b0850 con 0x7fe504060610 2022-06-29T16:00:14.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.187+0000 7fe50359e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5040b0ce0 con 0x7fe50405fee0 2022-06-29T16:00:14.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.187+0000 7fe501d9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe504060610 0x7fe50405c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:14.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.187+0000 7fe50259c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe50405c5b0 0x7fe50413c740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54948/0 (socket says 172.21.15.60:54948) 2022-06-29T16:00:14.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.187+0000 7fe50259c700 1 -- 172.21.15.60:0/1391856308 learned_addr learned my addr 172.21.15.60:0/1391856308 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:00:14.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.187+0000 7fe502d9d700 1 --2- 172.21.15.60:0/1391856308 >> v2:172.21.15.92:3300/0 conn(0x7fe50405fee0 0x7fe504142a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:14.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.187+0000 7fe501d9b700 1 -- 172.21.15.60:0/1391856308 >> v2:172.21.15.92:3300/0 conn(0x7fe50405fee0 msgr2=0x7fe504142a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:14.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.187+0000 7fe501d9b700 1 --2- 172.21.15.60:0/1391856308 >> v2:172.21.15.92:3300/0 conn(0x7fe50405fee0 0x7fe504142a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:14.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.187+0000 7fe501d9b700 1 -- 172.21.15.60:0/1391856308 >> v1:172.21.15.60:6789/0 conn(0x7fe50405c5b0 legacy=0x7fe50413c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:14.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.187+0000 7fe50159a700 1 -- 172.21.15.60:0/1391856308 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588047353 0 0) 0x7fe5040b0a40 con 0x7fe50405c5b0 2022-06-29T16:00:14.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.187+0000 7fe501d9b700 1 -- 172.21.15.60:0/1391856308 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe50413ce50 con 0x7fe504060610 2022-06-29T16:00:14.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.187+0000 7fe501d9b700 1 --2- 172.21.15.60:0/1391856308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe504060610 0x7fe50405c990 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7fe4ec002f70 tx=0x7fe4ec006df0).ready entity=mon.0 client_cookie=817ea92e3f31c1ab server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:14.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.187+0000 7fe50159a700 1 -- 172.21.15.60:0/1391856308 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4ec00d570 con 0x7fe504060610 2022-06-29T16:00:14.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.187+0000 7fe50159a700 1 -- 172.21.15.60:0/1391856308 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4ec000e90 con 0x7fe504060610 2022-06-29T16:00:14.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.187+0000 7fe50159a700 1 -- 172.21.15.60:0/1391856308 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4ec00d8c0 con 0x7fe504060610 2022-06-29T16:00:14.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.187+0000 7fe50359e700 1 -- 172.21.15.60:0/1391856308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe504060610 msgr2=0x7fe50405c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:14.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.187+0000 7fe50359e700 1 --2- 172.21.15.60:0/1391856308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe504060610 0x7fe50405c990 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7fe4ec002f70 tx=0x7fe4ec006df0).stop 2022-06-29T16:00:14.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.187+0000 7fe50359e700 1 -- 172.21.15.60:0/1391856308 shutdown_connections 2022-06-29T16:00:14.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.187+0000 7fe50359e700 1 --2- 172.21.15.60:0/1391856308 >> v2:172.21.15.92:3300/0 conn(0x7fe50405fee0 0x7fe504142a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:14.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.187+0000 7fe50359e700 1 --2- 172.21.15.60:0/1391856308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe504060610 0x7fe50405c990 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:14.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.187+0000 7fe50359e700 1 -- 172.21.15.60:0/1391856308 >> 172.21.15.60:0/1391856308 conn(0x7fe504061c80 msgr2=0x7fe5040637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:14.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.187+0000 7fe50359e700 1 -- 172.21.15.60:0/1391856308 shutdown_connections 2022-06-29T16:00:14.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.187+0000 7fe50359e700 1 -- 172.21.15.60:0/1391856308 wait complete. 2022-06-29T16:00:14.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.191+0000 7fe50359e700 1 Processor -- start 2022-06-29T16:00:14.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.191+0000 7fe50359e700 1 -- start start 2022-06-29T16:00:14.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.191+0000 7fe50359e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe50405c5b0 0x7fe5040b5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:14.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.191+0000 7fe50359e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe50405fee0 0x7fe504151a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:14.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.191+0000 7fe50359e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe504060610 0x7fe504145f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:14.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.191+0000 7fe50359e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe504144620 con 0x7fe50405c5b0 2022-06-29T16:00:14.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.191+0000 7fe50359e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5040b45e0 con 0x7fe50405fee0 2022-06-29T16:00:14.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.191+0000 7fe50359e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5040b5c60 con 0x7fe504060610 2022-06-29T16:00:14.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.191+0000 7fe50259c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe50405c5b0 0x7fe5040b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:14.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.191+0000 7fe50259c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe50405c5b0 0x7fe5040b5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46868/0 (socket says 172.21.15.60:46868) 2022-06-29T16:00:14.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.191+0000 7fe50259c700 1 -- 172.21.15.60:0/3696784805 learned_addr learned my addr 172.21.15.60:0/3696784805 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:14.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.191+0000 7fe501d9b700 1 --2- 172.21.15.60:0/3696784805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe50405fee0 0x7fe504151a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:14.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.191+0000 7fe502d9d700 1 --2- 172.21.15.60:0/3696784805 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe504060610 0x7fe504145f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:14.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.191+0000 7fe50259c700 1 -- 172.21.15.60:0/3696784805 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe504060610 msgr2=0x7fe504145f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:14.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.191+0000 7fe50259c700 1 --2- 172.21.15.60:0/3696784805 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe504060610 0x7fe504145f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:14.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.191+0000 7fe50259c700 1 -- 172.21.15.60:0/3696784805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe50405fee0 msgr2=0x7fe504151a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:14.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.191+0000 7fe50259c700 1 --2- 172.21.15.60:0/3696784805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe50405fee0 0x7fe504151a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:14.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.191+0000 7fe50259c700 1 -- 172.21.15.60:0/3696784805 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ec00b180 con 0x7fe50405c5b0 2022-06-29T16:00:14.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.191+0000 7fe502d9d700 1 --2- 172.21.15.60:0/3696784805 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe504060610 0x7fe504145f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:00:14.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.191+0000 7fe50259c700 1 --2- 172.21.15.60:0/3696784805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe50405c5b0 0x7fe5040b5030 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fe504144340 tx=0x7fe4f4007ac0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:14.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.195+0000 7fe4fb7fe700 1 -- 172.21.15.60:0/3696784805 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4f40070d0 con 0x7fe50405c5b0 2022-06-29T16:00:14.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.195+0000 7fe4fb7fe700 1 -- 172.21.15.60:0/3696784805 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4f4007230 con 0x7fe50405c5b0 2022-06-29T16:00:14.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.195+0000 7fe50359e700 1 -- 172.21.15.60:0/3696784805 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe504136d80 con 0x7fe50405c5b0 2022-06-29T16:00:14.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.195+0000 7fe4fb7fe700 1 -- 172.21.15.60:0/3696784805 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4f4005610 con 0x7fe50405c5b0 2022-06-29T16:00:14.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.195+0000 7fe50359e700 1 -- 172.21.15.60:0/3696784805 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5040b59c0 con 0x7fe50405c5b0 2022-06-29T16:00:14.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.195+0000 7fe50359e700 1 -- 172.21.15.60:0/3696784805 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5040573d0 con 0x7fe50405c5b0 2022-06-29T16:00:14.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.199+0000 7fe4fb7fe700 1 -- 172.21.15.60:0/3696784805 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe4f401a060 con 0x7fe50405c5b0 2022-06-29T16:00:14.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.199+0000 7fe4fb7fe700 1 --2- 172.21.15.60:0/3696784805 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe4dc053b30 0x7fe4dc055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:14.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.199+0000 7fe4fb7fe700 1 -- 172.21.15.60:0/3696784805 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fe4f406b560 con 0x7fe50405c5b0 2022-06-29T16:00:14.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.199+0000 7fe501d9b700 1 --2- 172.21.15.60:0/3696784805 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe4dc053b30 0x7fe4dc055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:14.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.199+0000 7fe501d9b700 1 --2- 172.21.15.60:0/3696784805 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe4dc053b30 0x7fe4dc055ff0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fe4ec01a000 tx=0x7fe4ec014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:14.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.199+0000 7fe4fb7fe700 1 -- 172.21.15.60:0/3696784805 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe4f4042650 con 0x7fe50405c5b0 2022-06-29T16:00:14.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.331+0000 7fe50359e700 1 -- 172.21.15.60:0/3696784805 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fe504068480 con 0x7fe50405c5b0 2022-06-29T16:00:14.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.335+0000 7fe4fb7fe700 1 -- 172.21.15.60:0/3696784805 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v4)=0 v4) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe4f4018020 con 0x7fe50405c5b0 2022-06-29T16:00:14.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.335+0000 7fe50359e700 1 -- 172.21.15.60:0/3696784805 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe4dc053b30 msgr2=0x7fe4dc055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:14.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.335+0000 7fe50359e700 1 --2- 172.21.15.60:0/3696784805 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe4dc053b30 0x7fe4dc055ff0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fe4ec01a000 tx=0x7fe4ec014000).stop 2022-06-29T16:00:14.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.335+0000 7fe50359e700 1 -- 172.21.15.60:0/3696784805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe50405c5b0 msgr2=0x7fe5040b5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:14.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.335+0000 7fe50359e700 1 --2- 172.21.15.60:0/3696784805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe50405c5b0 0x7fe5040b5030 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fe504144340 tx=0x7fe4f4007ac0).stop 2022-06-29T16:00:14.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.335+0000 7fe50359e700 1 -- 172.21.15.60:0/3696784805 shutdown_connections 2022-06-29T16:00:14.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.335+0000 7fe50359e700 1 --2- 172.21.15.60:0/3696784805 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe4dc053b30 0x7fe4dc055ff0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:14.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.335+0000 7fe50359e700 1 --2- 172.21.15.60:0/3696784805 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe504060610 0x7fe504145f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:14.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.335+0000 7fe50359e700 1 --2- 172.21.15.60:0/3696784805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe50405fee0 0x7fe504151a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:14.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.335+0000 7fe50359e700 1 --2- 172.21.15.60:0/3696784805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe50405c5b0 0x7fe5040b5030 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:14.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.335+0000 7fe50359e700 1 -- 172.21.15.60:0/3696784805 >> 172.21.15.60:0/3696784805 conn(0x7fe504061c80 msgr2=0x7fe5040652a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:14.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.339+0000 7fe50359e700 1 -- 172.21.15.60:0/3696784805 shutdown_connections 2022-06-29T16:00:14.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.339+0000 7fe50359e700 1 -- 172.21.15.60:0/3696784805 wait complete. 2022-06-29T16:00:14.358 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-06-29T16:00:14.358 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:00:14.378 DEBUG:teuthology.orchestra.run.smithi092:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:00:14.401 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:00:14.423 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:00:14.444 DEBUG:teuthology.orchestra.run.smithi197:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:00:14.466 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-06-29T16:00:14.467 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:00:14.469 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:00:14.470 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:00:14.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:00:14.480 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-06-29T16:00:14.481 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-06-29T16:00:14.482 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:00:14.483 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:00:14.485 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:00:14.494 INFO:teuthology.orchestra.run.smithi156.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-06-29T16:00:14.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:00:14.498 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-06-29T16:00:14.499 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:00:14.499 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:00:14.499 DEBUG:teuthology.orchestra.run.smithi197:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T16:00:14.507 INFO:teuthology.orchestra.run.smithi197.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-06-29T16:00:14.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:00:14.509 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-06-29T16:00:14.509 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-06-29T16:00:14.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6effff700 1 Processor -- start 2022-06-29T16:00:14.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6effff700 1 -- start start 2022-06-29T16:00:14.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6effff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa6f00b2130 0x7fa6f0146100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:14.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6effff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa6f00aed00 0x7fa6f01465f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:14.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6effff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6f005d450 con 0x7fa6f005d070 2022-06-29T16:00:14.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6effff700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6f00adf90 con 0x7fa6f00aed00 2022-06-29T16:00:14.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6effff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f00b25d0 con 0x7fa6f00b2130 2022-06-29T16:00:14.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6eeffd700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa6f005d070 0x7fa6f014aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54962/0 (socket says 172.21.15.60:54962) 2022-06-29T16:00:14.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6eeffd700 1 -- 172.21.15.60:0/2809780676 learned_addr learned my addr 172.21.15.60:0/2809780676 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:00:14.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6ef7fe700 1 --2- 172.21.15.60:0/2809780676 >> v2:172.21.15.92:3300/0 conn(0x7fa6f00aed00 0x7fa6f01465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:14.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6ee7fc700 1 --2- 172.21.15.60:0/2809780676 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa6f00b2130 0x7fa6f0146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:14.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6edffb700 1 -- 172.21.15.60:0/2809780676 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1696339000 0 0) 0x7fa6f005d450 con 0x7fa6f005d070 2022-06-29T16:00:14.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6edffb700 1 -- 172.21.15.60:0/2809780676 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6d0003260 con 0x7fa6f005d070 2022-06-29T16:00:14.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6edffb700 1 -- 172.21.15.60:0/2809780676 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4037709842 0 0) 0x7fa6d0003260 con 0x7fa6f005d070 2022-06-29T16:00:14.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6edffb700 1 -- 172.21.15.60:0/2809780676 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa6f00b2130 msgr2=0x7fa6f0146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:14.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6edffb700 1 --2- 172.21.15.60:0/2809780676 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa6f00b2130 0x7fa6f0146100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:14.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6edffb700 1 -- 172.21.15.60:0/2809780676 >> v2:172.21.15.92:3300/0 conn(0x7fa6f00aed00 msgr2=0x7fa6f01465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:14.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6edffb700 1 --2- 172.21.15.60:0/2809780676 >> v2:172.21.15.92:3300/0 conn(0x7fa6f00aed00 0x7fa6f01465f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:14.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6edffb700 1 -- 172.21.15.60:0/2809780676 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f014a350 con 0x7fa6f005d070 2022-06-29T16:00:14.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6edffb700 1 -- 172.21.15.60:0/2809780676 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7fa6e0001a00 con 0x7fa6f005d070 2022-06-29T16:00:14.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6edffb700 1 -- 172.21.15.60:0/2809780676 >> v1:172.21.15.60:6789/0 conn(0x7fa6f005d070 legacy=0x7fa6f014aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:14.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6edffb700 1 --2- 172.21.15.60:0/2809780676 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa6f00b2130 0x7fa6d0003750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:14.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6edffb700 1 --2- 172.21.15.60:0/2809780676 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6d0003c80 0x7fa6d0004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:14.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6edffb700 1 --2- 172.21.15.60:0/2809780676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6d00049f0 0x7fa6d0004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:14.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6edffb700 1 -- 172.21.15.60:0/2809780676 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f00adf90 con 0x7fa6d00049f0 2022-06-29T16:00:14.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6edffb700 1 -- 172.21.15.60:0/2809780676 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f00b25d0 con 0x7fa6d0003c80 2022-06-29T16:00:14.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6edffb700 1 -- 172.21.15.60:0/2809780676 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d000b750 con 0x7fa6f00b2130 2022-06-29T16:00:14.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6ef7fe700 1 --2- 172.21.15.60:0/2809780676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6d00049f0 0x7fa6d0004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:14.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6eeffd700 1 --2- 172.21.15.60:0/2809780676 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa6f00b2130 0x7fa6d0003750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:14.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6ee7fc700 1 --2- 172.21.15.60:0/2809780676 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6d0003c80 0x7fa6d0004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:14.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6ef7fe700 1 -- 172.21.15.60:0/2809780676 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa6f00b2130 msgr2=0x7fa6d0003750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:14.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6ef7fe700 1 --2- 172.21.15.60:0/2809780676 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa6f00b2130 0x7fa6d0003750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:14.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6ef7fe700 1 -- 172.21.15.60:0/2809780676 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6d0003c80 msgr2=0x7fa6d0004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:14.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6ef7fe700 1 --2- 172.21.15.60:0/2809780676 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6d0003c80 0x7fa6d0004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:14.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.607+0000 7fa6ef7fe700 1 -- 172.21.15.60:0/2809780676 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d0003260 con 0x7fa6d00049f0 2022-06-29T16:00:14.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.611+0000 7fa6ee7fc700 1 --2- 172.21.15.60:0/2809780676 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6d0003c80 0x7fa6d0004500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:00:14.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.611+0000 7fa6ef7fe700 1 --2- 172.21.15.60:0/2809780676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6d00049f0 0x7fa6d0004e70 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fa6d00054d0 tx=0x7fa6e8006df0).ready entity=mon.0 client_cookie=e93c4d9f472a8e56 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:14.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.611+0000 7fa6edffb700 1 -- 172.21.15.60:0/2809780676 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6e800e830 con 0x7fa6d00049f0 2022-06-29T16:00:14.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.611+0000 7fa6edffb700 1 -- 172.21.15.60:0/2809780676 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6e800be90 con 0x7fa6d00049f0 2022-06-29T16:00:14.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.611+0000 7fa6edffb700 1 -- 172.21.15.60:0/2809780676 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6e800eb80 con 0x7fa6d00049f0 2022-06-29T16:00:14.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.611+0000 7fa6effff700 1 -- 172.21.15.60:0/2809780676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6d00049f0 msgr2=0x7fa6d0004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:14.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.611+0000 7fa6effff700 1 --2- 172.21.15.60:0/2809780676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6d00049f0 0x7fa6d0004e70 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fa6d00054d0 tx=0x7fa6e8006df0).stop 2022-06-29T16:00:14.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.611+0000 7fa6ef7fe700 1 -- 172.21.15.60:0/2809780676 reap_dead start 2022-06-29T16:00:14.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.611+0000 7fa6effff700 1 -- 172.21.15.60:0/2809780676 shutdown_connections 2022-06-29T16:00:14.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.611+0000 7fa6effff700 1 -- 172.21.15.60:0/2809780676 >> 172.21.15.60:0/2809780676 conn(0x7fa6f00b5890 msgr2=0x7fa6f00b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:14.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.611+0000 7fa6effff700 1 -- 172.21.15.60:0/2809780676 shutdown_connections 2022-06-29T16:00:14.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.611+0000 7fa6effff700 1 -- 172.21.15.60:0/2809780676 wait complete. 2022-06-29T16:00:14.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.611+0000 7fa6effff700 1 Processor -- start 2022-06-29T16:00:14.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.611+0000 7fa6effff700 1 -- start start 2022-06-29T16:00:14.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.615+0000 7fa6effff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6f0147370 0x7fa6f0159890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:14.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.615+0000 7fa6effff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa6f014a9b0 0x7fa6f0159d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:14.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.615+0000 7fa6effff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6f015a270 0x7fa6f013fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:14.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.615+0000 7fa6effff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f015a6f0 con 0x7fa6f0147370 2022-06-29T16:00:14.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.615+0000 7fa6effff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f0147750 con 0x7fa6f015a270 2022-06-29T16:00:14.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.615+0000 7fa6effff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f0156f40 con 0x7fa6f014a9b0 2022-06-29T16:00:14.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.615+0000 7fa6eeffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6f0147370 0x7fa6f0159890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:14.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.615+0000 7fa6ee7fc700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa6f014a9b0 0x7fa6f0159d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:14.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.615+0000 7fa6eeffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6f0147370 0x7fa6f0159890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46888/0 (socket says 172.21.15.60:46888) 2022-06-29T16:00:14.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.615+0000 7fa6eeffd700 1 -- 172.21.15.60:0/1448906628 learned_addr learned my addr 172.21.15.60:0/1448906628 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:14.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.615+0000 7fa6ef7fe700 1 --2- 172.21.15.60:0/1448906628 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6f015a270 0x7fa6f013fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:14.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.615+0000 7fa6eeffd700 1 -- 172.21.15.60:0/1448906628 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa6f014a9b0 msgr2=0x7fa6f0159d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:14.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.615+0000 7fa6eeffd700 1 --2- 172.21.15.60:0/1448906628 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa6f014a9b0 0x7fa6f0159d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:14.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.615+0000 7fa6eeffd700 1 -- 172.21.15.60:0/1448906628 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6f015a270 msgr2=0x7fa6f013fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:14.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.615+0000 7fa6eeffd700 1 --2- 172.21.15.60:0/1448906628 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6f015a270 0x7fa6f013fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:14.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.615+0000 7fa6eeffd700 1 -- 172.21.15.60:0/1448906628 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f01571e0 con 0x7fa6f0147370 2022-06-29T16:00:14.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.615+0000 7fa6eeffd700 1 --2- 172.21.15.60:0/1448906628 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6f0147370 0x7fa6f0159890 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7fa6f0149ad0 tx=0x7fa6e0002210).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:14.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.615+0000 7fa6e7fff700 1 -- 172.21.15.60:0/1448906628 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6e000f9b0 con 0x7fa6f0147370 2022-06-29T16:00:14.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.615+0000 7fa6e7fff700 1 -- 172.21.15.60:0/1448906628 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6e000fb10 con 0x7fa6f0147370 2022-06-29T16:00:14.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.615+0000 7fa6e7fff700 1 -- 172.21.15.60:0/1448906628 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6e000fe70 con 0x7fa6f0147370 2022-06-29T16:00:14.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.615+0000 7fa6effff700 1 -- 172.21.15.60:0/1448906628 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6f0066d70 con 0x7fa6f0147370 2022-06-29T16:00:14.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.615+0000 7fa6effff700 1 -- 172.21.15.60:0/1448906628 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6f0149eb0 con 0x7fa6f0147370 2022-06-29T16:00:14.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.619+0000 7fa6effff700 1 -- 172.21.15.60:0/1448906628 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6f0156d90 con 0x7fa6f0147370 2022-06-29T16:00:14.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.619+0000 7fa6e7fff700 1 -- 172.21.15.60:0/1448906628 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa6e001b030 con 0x7fa6f0147370 2022-06-29T16:00:14.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.619+0000 7fa6e7fff700 1 --2- 172.21.15.60:0/1448906628 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa6d4053aa0 0x7fa6d4055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:14.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.619+0000 7fa6e7fff700 1 -- 172.21.15.60:0/1448906628 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fa6e006ba50 con 0x7fa6f0147370 2022-06-29T16:00:14.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.619+0000 7fa6ee7fc700 1 --2- 172.21.15.60:0/1448906628 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa6d4053aa0 0x7fa6d4055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:14.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.619+0000 7fa6ee7fc700 1 --2- 172.21.15.60:0/1448906628 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa6d4053aa0 0x7fa6d4055f60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fa6d800b000 tx=0x7fa6d8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:14.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.623+0000 7fa6e7fff700 1 -- 172.21.15.60:0/1448906628 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa6e0042bf0 con 0x7fa6f0147370 2022-06-29T16:00:14.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.755+0000 7fa6effff700 1 -- 172.21.15.60:0/1448906628 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fa6f014a5f0 con 0x7fa6f0147370 2022-06-29T16:00:14.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.755+0000 7fa6e7fff700 1 -- 172.21.15.60:0/1448906628 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v35) v1 ==== 81+0+28827 (secure 0 0 0) 0x7fa6e002a0b0 con 0x7fa6f0147370 2022-06-29T16:00:14.764 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:00:14.765 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:00:14.765 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 35, 2022-06-29T16:00:14.765 INFO:teuthology.orchestra.run.smithi060.stdout: "fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", 2022-06-29T16:00:14.766 INFO:teuthology.orchestra.run.smithi060.stdout: "created": "2022-06-29T15:58:07.965719+0000", 2022-06-29T16:00:14.766 INFO:teuthology.orchestra.run.smithi060.stdout: "modified": "2022-06-29T16:00:12.038778+0000", 2022-06-29T16:00:14.766 INFO:teuthology.orchestra.run.smithi060.stdout: "last_up_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:00:14.766 INFO:teuthology.orchestra.run.smithi060.stdout: "last_in_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:00:14.767 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-06-29T16:00:14.767 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_num": 5799936, 2022-06-29T16:00:14.767 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_set": [ 2022-06-29T16:00:14.767 INFO:teuthology.orchestra.run.smithi060.stdout: "pglog_hardlimit", 2022-06-29T16:00:14.767 INFO:teuthology.orchestra.run.smithi060.stdout: "purged_snapdirs", 2022-06-29T16:00:14.768 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_deletes", 2022-06-29T16:00:14.768 INFO:teuthology.orchestra.run.smithi060.stdout: "sortbitwise" 2022-06-29T16:00:14.768 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:00:14.768 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_version": 4, 2022-06-29T16:00:14.769 INFO:teuthology.orchestra.run.smithi060.stdout: "full_ratio": 0.94999998807907104, 2022-06-29T16:00:14.769 INFO:teuthology.orchestra.run.smithi060.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-06-29T16:00:14.769 INFO:teuthology.orchestra.run.smithi060.stdout: "nearfull_ratio": 0.85000002384185791, 2022-06-29T16:00:14.769 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_snapshot": "", 2022-06-29T16:00:14.769 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_max": 6, 2022-06-29T16:00:14.770 INFO:teuthology.orchestra.run.smithi060.stdout: "max_osd": 8, 2022-06-29T16:00:14.770 INFO:teuthology.orchestra.run.smithi060.stdout: "require_min_compat_client": "jewel", 2022-06-29T16:00:14.770 INFO:teuthology.orchestra.run.smithi060.stdout: "min_compat_client": "jewel", 2022-06-29T16:00:14.770 INFO:teuthology.orchestra.run.smithi060.stdout: "require_osd_release": "octopus", 2022-06-29T16:00:14.771 INFO:teuthology.orchestra.run.smithi060.stdout: "pools": [ 2022-06-29T16:00:14.771 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.771 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2022-06-29T16:00:14.772 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "device_health_metrics", 2022-06-29T16:00:14.772 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:10.961127+0000", 2022-06-29T16:00:14.772 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:00:14.772 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:00:14.772 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:00:14.773 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:00:14.773 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:00:14.773 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:00:14.773 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:00:14.774 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:00:14.774 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 1, 2022-06-29T16:00:14.774 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 1, 2022-06-29T16:00:14.774 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 1, 2022-06-29T16:00:14.774 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 1, 2022-06-29T16:00:14.775 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 1, 2022-06-29T16:00:14.775 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:00:14.775 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:00:14.775 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:00:14.776 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:00:14.776 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:00:14.776 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:00:14.776 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:00:14.776 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.777 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "15", 2022-06-29T16:00:14.777 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:00:14.777 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:00:14.777 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:00:14.778 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:00:14.778 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:00:14.778 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:00:14.778 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:00:14.779 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:00:14.779 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:00:14.779 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:00:14.779 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:00:14.780 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:00:14.780 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:00:14.780 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:00:14.780 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:00:14.781 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:00:14.781 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:00:14.781 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:00:14.781 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:00:14.781 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:00:14.782 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:00:14.782 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:00:14.782 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:00:14.782 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:00:14.783 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:00:14.783 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:00:14.783 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.783 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:00:14.783 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:00:14.784 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:00:14.784 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:00:14.784 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:00:14.784 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:00:14.784 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:00:14.785 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:00:14.785 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:00:14.785 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:00:14.785 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:00:14.786 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:00:14.786 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 1 2022-06-29T16:00:14.786 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.786 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:00:14.787 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_devicehealth": {} 2022-06-29T16:00:14.787 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.787 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.787 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.787 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 2, 2022-06-29T16:00:14.788 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "rbd", 2022-06-29T16:00:14.788 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:19.184276+0000", 2022-06-29T16:00:14.788 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:00:14.789 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:00:14.789 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:00:14.789 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:00:14.789 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:00:14.790 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:00:14.790 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:00:14.790 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:00:14.790 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:00:14.791 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:00:14.791 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:00:14.791 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:00:14.791 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:00:14.791 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:00:14.792 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:00:14.792 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:00:14.792 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:00:14.792 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:00:14.793 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:00:14.793 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:00:14.793 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.793 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "18", 2022-06-29T16:00:14.794 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:00:14.794 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:00:14.794 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:00:14.794 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:00:14.794 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:00:14.795 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:00:14.795 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:00:14.796 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:00:14.797 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:00:14.797 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:00:14.797 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:00:14.797 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:00:14.798 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:00:14.798 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:00:14.798 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:00:14.798 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:00:14.799 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:00:14.799 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:00:14.799 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:00:14.799 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:00:14.800 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:00:14.800 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:00:14.800 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:00:14.800 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:00:14.800 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:00:14.801 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:00:14.801 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.801 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:00:14.801 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:00:14.802 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:00:14.802 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:00:14.802 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:00:14.803 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:00:14.803 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:00:14.803 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:00:14.803 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:00:14.803 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:00:14.804 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:00:14.804 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:00:14.804 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:00:14.804 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd": {} 2022-06-29T16:00:14.805 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.805 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.805 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:00:14.805 INFO:teuthology.orchestra.run.smithi060.stdout: "osds": [ 2022-06-29T16:00:14.806 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.806 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:00:14.807 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "11e7ab9e-fce2-470f-8195-ea4c3c52df22", 2022-06-29T16:00:14.807 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:00:14.807 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:00:14.808 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:00:14.808 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:00:14.808 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:00:14.808 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:00:14.809 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:00:14.809 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 30, 2022-06-29T16:00:14.809 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:00:14.809 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:00:14.809 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:00:14.810 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:14.810 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.810 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:14.810 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6816", 2022-06-29T16:00:14.811 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:00:14.811 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.811 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.811 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:14.811 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6825", 2022-06-29T16:00:14.812 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:00:14.812 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.812 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.812 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.813 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:00:14.813 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:14.813 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.813 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:14.814 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6826", 2022-06-29T16:00:14.814 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:00:14.814 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.815 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.815 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:14.815 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6827", 2022-06-29T16:00:14.815 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:00:14.816 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.816 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.816 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.817 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:00:14.817 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:14.817 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.818 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:14.818 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6830", 2022-06-29T16:00:14.818 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:00:14.818 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.818 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.819 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:14.819 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6831", 2022-06-29T16:00:14.819 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:00:14.819 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.820 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.820 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.820 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:00:14.820 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:14.820 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.821 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:14.821 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6828", 2022-06-29T16:00:14.821 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:00:14.821 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.821 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.822 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:14.822 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6829", 2022-06-29T16:00:14.822 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:00:14.823 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.823 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.824 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.824 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6825/12890", 2022-06-29T16:00:14.824 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6827/12890", 2022-06-29T16:00:14.825 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6831/12890", 2022-06-29T16:00:14.825 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6829/12890", 2022-06-29T16:00:14.825 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:00:14.825 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:00:14.825 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:00:14.826 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.826 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.826 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.826 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:00:14.827 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "ca43f136-6f69-4431-a157-fbe073fb585c", 2022-06-29T16:00:14.827 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:00:14.828 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:00:14.828 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:00:14.828 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:00:14.828 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:00:14.829 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:00:14.829 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:00:14.829 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 30, 2022-06-29T16:00:14.829 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:00:14.830 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:00:14.830 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:00:14.831 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:14.831 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.831 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:14.831 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6815", 2022-06-29T16:00:14.831 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:00:14.832 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.832 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.832 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:14.833 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6818", 2022-06-29T16:00:14.833 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:00:14.833 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.833 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.834 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.834 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:00:14.834 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:14.835 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.835 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:14.835 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6819", 2022-06-29T16:00:14.835 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:00:14.835 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.836 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.836 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:14.836 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6820", 2022-06-29T16:00:14.836 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:00:14.837 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.837 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.837 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.837 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:00:14.838 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:14.838 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.839 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:14.839 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6823", 2022-06-29T16:00:14.839 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:00:14.839 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.840 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.840 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:14.840 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6824", 2022-06-29T16:00:14.840 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:00:14.841 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.841 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.841 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.842 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:00:14.842 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:14.842 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.842 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:14.843 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6821", 2022-06-29T16:00:14.843 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:00:14.843 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.843 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.844 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:14.844 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6822", 2022-06-29T16:00:14.844 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:00:14.844 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.844 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.845 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.845 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6818/12892", 2022-06-29T16:00:14.845 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6820/12892", 2022-06-29T16:00:14.845 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6824/12892", 2022-06-29T16:00:14.846 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6822/12892", 2022-06-29T16:00:14.846 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:00:14.846 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:00:14.846 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:00:14.846 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.847 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.847 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.847 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:00:14.847 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "74fac45f-5f07-44ec-a9cb-2644c123fd4a", 2022-06-29T16:00:14.848 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:00:14.848 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:00:14.848 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:00:14.848 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:00:14.848 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:00:14.849 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:00:14.849 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:00:14.849 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 29, 2022-06-29T16:00:14.850 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:00:14.850 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:00:14.850 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:00:14.850 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:14.851 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.851 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:14.851 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6808", 2022-06-29T16:00:14.851 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:00:14.852 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.852 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.852 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:14.852 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6809", 2022-06-29T16:00:14.853 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:00:14.853 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.853 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.853 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.854 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:00:14.854 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:14.854 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.854 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:14.855 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6810", 2022-06-29T16:00:14.855 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:00:14.855 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.855 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.856 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:14.856 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6811", 2022-06-29T16:00:14.856 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:00:14.856 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.856 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.857 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.857 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:00:14.857 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:14.857 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.857 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:14.858 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6814", 2022-06-29T16:00:14.858 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:00:14.858 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.859 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.859 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:14.859 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6817", 2022-06-29T16:00:14.859 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:00:14.860 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.860 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.860 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.860 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:00:14.860 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:14.861 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.861 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:14.861 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6812", 2022-06-29T16:00:14.861 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:00:14.862 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.862 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.862 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:14.862 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6813", 2022-06-29T16:00:14.862 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:00:14.863 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.863 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.863 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.863 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6809/12891", 2022-06-29T16:00:14.863 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6811/12891", 2022-06-29T16:00:14.864 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6817/12891", 2022-06-29T16:00:14.864 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6813/12891", 2022-06-29T16:00:14.864 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:00:14.864 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:00:14.865 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:00:14.865 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.865 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.865 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.865 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:00:14.866 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "2581d351-973a-4b4e-85ca-3658f0068765", 2022-06-29T16:00:14.866 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:00:14.866 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:00:14.866 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:00:14.867 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:00:14.867 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:00:14.867 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:00:14.867 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:00:14.868 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 30, 2022-06-29T16:00:14.868 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:00:14.868 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:00:14.868 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:00:14.869 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:14.869 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.869 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:14.869 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6800", 2022-06-29T16:00:14.869 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:00:14.870 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.870 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.870 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:14.870 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6801", 2022-06-29T16:00:14.871 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:00:14.871 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.871 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.871 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.871 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:00:14.872 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:14.872 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.872 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:14.872 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6802", 2022-06-29T16:00:14.872 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:00:14.873 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.873 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.873 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:14.873 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6803", 2022-06-29T16:00:14.874 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:00:14.874 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.874 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.874 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.874 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:00:14.875 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:14.875 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.875 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:14.875 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6806", 2022-06-29T16:00:14.876 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:00:14.876 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.876 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.876 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:14.877 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6807", 2022-06-29T16:00:14.877 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:00:14.877 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.877 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.878 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.878 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:00:14.878 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:14.878 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.878 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:14.879 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6804", 2022-06-29T16:00:14.879 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:00:14.879 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.879 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.880 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:14.880 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6805", 2022-06-29T16:00:14.880 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:00:14.880 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.880 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.881 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.881 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6801/12889", 2022-06-29T16:00:14.881 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6803/12889", 2022-06-29T16:00:14.881 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6807/12889", 2022-06-29T16:00:14.881 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6805/12889", 2022-06-29T16:00:14.882 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:00:14.882 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:00:14.882 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:00:14.882 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.882 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.883 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.883 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:00:14.883 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "3d234db9-87cd-4fe9-814f-0240a287d8b4", 2022-06-29T16:00:14.883 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:00:14.884 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:00:14.884 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:00:14.884 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:00:14.884 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:00:14.884 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:00:14.885 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:00:14.885 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 30, 2022-06-29T16:00:14.885 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:00:14.886 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:00:14.886 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:00:14.886 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:14.886 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.886 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:14.887 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6808", 2022-06-29T16:00:14.887 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:00:14.887 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.887 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.887 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:14.888 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6810", 2022-06-29T16:00:14.888 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:00:14.888 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.888 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.889 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.889 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:00:14.889 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:14.889 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.889 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:14.890 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6812", 2022-06-29T16:00:14.890 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:00:14.890 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.890 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.891 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:14.891 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6814", 2022-06-29T16:00:14.891 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:00:14.891 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.891 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.892 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.892 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:00:14.892 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:14.892 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.893 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:14.893 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6823", 2022-06-29T16:00:14.893 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:00:14.893 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.894 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.894 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:14.894 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6825", 2022-06-29T16:00:14.895 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:00:14.895 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.895 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.895 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.895 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:00:14.896 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:14.896 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.896 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:14.896 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6817", 2022-06-29T16:00:14.896 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:00:14.897 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.897 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.897 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:14.897 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6820", 2022-06-29T16:00:14.898 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:00:14.898 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.898 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.898 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.898 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6810/13249", 2022-06-29T16:00:14.899 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6814/13249", 2022-06-29T16:00:14.899 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6825/13249", 2022-06-29T16:00:14.899 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6820/13249", 2022-06-29T16:00:14.899 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:00:14.900 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:00:14.900 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:00:14.900 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.900 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.900 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.901 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:00:14.901 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "e4bf1104-ff5c-438b-b608-8cee31c163f6", 2022-06-29T16:00:14.901 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:00:14.901 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:00:14.901 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:00:14.902 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:00:14.902 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:00:14.902 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:00:14.902 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:00:14.903 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 30, 2022-06-29T16:00:14.903 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:00:14.903 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:00:14.904 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:00:14.904 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:14.904 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.904 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:14.904 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6809", 2022-06-29T16:00:14.905 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:00:14.905 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.905 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.905 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:14.905 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6811", 2022-06-29T16:00:14.906 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:00:14.906 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.906 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.906 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.907 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:00:14.907 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:14.907 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.907 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:14.907 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6813", 2022-06-29T16:00:14.908 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:00:14.908 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.908 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.908 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:14.908 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6815", 2022-06-29T16:00:14.909 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:00:14.909 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.909 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.909 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.910 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:00:14.910 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:14.910 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.910 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:14.911 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6824", 2022-06-29T16:00:14.911 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:00:14.911 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.911 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.912 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:14.912 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6827", 2022-06-29T16:00:14.912 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:00:14.912 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.913 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.913 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.913 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:00:14.913 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:14.914 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.914 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:14.914 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6818", 2022-06-29T16:00:14.914 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:00:14.914 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.915 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.915 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:14.915 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6821", 2022-06-29T16:00:14.915 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:00:14.916 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.916 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.916 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.916 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6811/13248", 2022-06-29T16:00:14.917 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6815/13248", 2022-06-29T16:00:14.917 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6827/13248", 2022-06-29T16:00:14.917 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6821/13248", 2022-06-29T16:00:14.917 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:00:14.917 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:00:14.918 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:00:14.918 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.918 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.918 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.919 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:00:14.919 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "8564dd25-27ab-4203-b54e-af6d3aa05857", 2022-06-29T16:00:14.919 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:00:14.919 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:00:14.920 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:00:14.920 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:00:14.920 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:00:14.920 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:00:14.921 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:00:14.921 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 29, 2022-06-29T16:00:14.921 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:00:14.921 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:00:14.922 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:00:14.922 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:14.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.922 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:14.922 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6816", 2022-06-29T16:00:14.923 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:00:14.923 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.923 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.923 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:14.924 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6819", 2022-06-29T16:00:14.924 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:00:14.924 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.924 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.924 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.925 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:00:14.925 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:14.925 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.925 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:14.926 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6822", 2022-06-29T16:00:14.926 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:00:14.926 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.926 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.927 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:14.927 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6826", 2022-06-29T16:00:14.927 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:00:14.927 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.927 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.928 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.928 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:00:14.928 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:14.928 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.929 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:14.929 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6830", 2022-06-29T16:00:14.929 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:00:14.930 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.930 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.930 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:14.930 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6831", 2022-06-29T16:00:14.931 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:00:14.931 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.931 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.931 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.931 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:00:14.932 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:14.932 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.932 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:14.932 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6828", 2022-06-29T16:00:14.933 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:00:14.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.933 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.933 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:14.933 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6829", 2022-06-29T16:00:14.934 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:00:14.934 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.934 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.934 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.934 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6819/13250", 2022-06-29T16:00:14.935 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6826/13250", 2022-06-29T16:00:14.935 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6831/13250", 2022-06-29T16:00:14.935 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6829/13250", 2022-06-29T16:00:14.935 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:00:14.936 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:00:14.936 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:00:14.936 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.936 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.936 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.937 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:00:14.937 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "bd513a9c-3b19-4314-9897-d300b5c897bb", 2022-06-29T16:00:14.937 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:00:14.937 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:00:14.937 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:00:14.938 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:00:14.938 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:00:14.938 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:00:14.939 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:00:14.939 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 30, 2022-06-29T16:00:14.939 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:00:14.939 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:00:14.940 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:00:14.940 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:14.940 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.940 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:14.940 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6800", 2022-06-29T16:00:14.941 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:00:14.941 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.941 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.941 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:14.942 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6801", 2022-06-29T16:00:14.942 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:00:14.942 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.942 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.942 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.943 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:00:14.943 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:14.943 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.943 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:14.943 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6802", 2022-06-29T16:00:14.944 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:00:14.944 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.944 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.944 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:14.944 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6803", 2022-06-29T16:00:14.945 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:00:14.945 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.945 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.945 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.946 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:00:14.946 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:14.946 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.946 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:14.946 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6806", 2022-06-29T16:00:14.947 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:00:14.947 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.947 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.948 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:14.948 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6807", 2022-06-29T16:00:14.948 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:00:14.948 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.949 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.949 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.949 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:00:14.949 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:00:14.949 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.950 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:00:14.950 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6804", 2022-06-29T16:00:14.950 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:00:14.950 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.951 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.951 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:00:14.951 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6805", 2022-06-29T16:00:14.951 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:00:14.952 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.952 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.952 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.952 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6801/13247", 2022-06-29T16:00:14.953 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6803/13247", 2022-06-29T16:00:14.953 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6807/13247", 2022-06-29T16:00:14.953 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6805/13247", 2022-06-29T16:00:14.953 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:00:14.953 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:00:14.954 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:00:14.954 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:14.954 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.954 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:00:14.955 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_xinfo": [ 2022-06-29T16:00:14.955 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.955 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:00:14.955 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:00:14.955 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:00:14.956 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:00:14.956 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:00:14.956 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:00:14.956 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.891342+0000", 2022-06-29T16:00:14.957 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:00:14.957 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.957 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.957 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:00:14.958 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:00:14.958 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:00:14.958 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:00:14.958 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:00:14.958 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:00:14.959 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.900866+0000", 2022-06-29T16:00:14.959 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:00:14.959 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.959 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.960 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:00:14.960 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:00:14.960 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:00:14.960 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:00:14.960 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:00:14.961 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:00:14.961 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.912755+0000", 2022-06-29T16:00:14.961 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:00:14.961 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.962 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.962 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:00:14.962 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:00:14.962 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:00:14.962 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:00:14.963 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:00:14.963 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:00:14.963 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.864332+0000", 2022-06-29T16:00:14.964 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:00:14.964 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.964 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:00:14.965 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:00:14.965 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:00:14.965 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:00:14.965 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:00:14.966 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:00:14.966 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.863756+0000", 2022-06-29T16:00:14.966 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:00:14.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.967 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:00:14.967 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:00:14.967 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:00:14.967 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:00:14.967 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:00:14.968 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:00:14.968 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.869041+0000", 2022-06-29T16:00:14.968 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:00:14.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.969 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:00:14.969 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:00:14.969 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:00:14.969 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:00:14.970 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:00:14.970 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:00:14.970 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.895043+0000", 2022-06-29T16:00:14.970 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:00:14.971 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.971 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:14.971 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:00:14.971 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:00:14.971 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:00:14.972 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:00:14.972 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:00:14.972 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:00:14.972 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.889491+0000", 2022-06-29T16:00:14.973 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:00:14.973 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.973 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:00:14.973 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap": [], 2022-06-29T16:00:14.974 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap_items": [], 2022-06-29T16:00:14.974 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_temp": [], 2022-06-29T16:00:14.974 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_temp": [], 2022-06-29T16:00:14.974 INFO:teuthology.orchestra.run.smithi060.stdout: "blacklist": { 2022-06-29T16:00:14.975 INFO:teuthology.orchestra.run.smithi060.stdout: "172.21.15.92:6835/1993861928": "2022-06-30T16:00:08.669295+0000", 2022-06-29T16:00:14.975 INFO:teuthology.orchestra.run.smithi060.stdout: "172.21.15.92:6834/1993861928": "2022-06-30T16:00:08.669295+0000" 2022-06-29T16:00:14.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.975 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profiles": { 2022-06-29T16:00:14.975 INFO:teuthology.orchestra.run.smithi060.stdout: "default": { 2022-06-29T16:00:14.976 INFO:teuthology.orchestra.run.smithi060.stdout: "crush-failure-domain": "osd", 2022-06-29T16:00:14.976 INFO:teuthology.orchestra.run.smithi060.stdout: "k": "2", 2022-06-29T16:00:14.976 INFO:teuthology.orchestra.run.smithi060.stdout: "m": "1", 2022-06-29T16:00:14.976 INFO:teuthology.orchestra.run.smithi060.stdout: "plugin": "jerasure", 2022-06-29T16:00:14.977 INFO:teuthology.orchestra.run.smithi060.stdout: "ruleset-failure-domain": "osd", 2022-06-29T16:00:14.977 INFO:teuthology.orchestra.run.smithi060.stdout: "technique": "reed_sol_van" 2022-06-29T16:00:14.977 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:14.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:14.977 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps_queue": [], 2022-06-29T16:00:14.978 INFO:teuthology.orchestra.run.smithi060.stdout: "new_removed_snaps": [], 2022-06-29T16:00:14.978 INFO:teuthology.orchestra.run.smithi060.stdout: "new_purged_snaps": [], 2022-06-29T16:00:14.978 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_node_flags": {}, 2022-06-29T16:00:14.978 INFO:teuthology.orchestra.run.smithi060.stdout: "device_class_flags": {} 2022-06-29T16:00:14.978 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:00:14.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.759+0000 7fa6effff700 1 -- 172.21.15.60:0/1448906628 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa6d4053aa0 msgr2=0x7fa6d4055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:14.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.759+0000 7fa6effff700 1 --2- 172.21.15.60:0/1448906628 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa6d4053aa0 0x7fa6d4055f60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fa6d800b000 tx=0x7fa6d8008040).stop 2022-06-29T16:00:14.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.759+0000 7fa6effff700 1 -- 172.21.15.60:0/1448906628 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6f0147370 msgr2=0x7fa6f0159890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:14.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.759+0000 7fa6effff700 1 --2- 172.21.15.60:0/1448906628 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6f0147370 0x7fa6f0159890 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7fa6f0149ad0 tx=0x7fa6e0002210).stop 2022-06-29T16:00:14.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.759+0000 7fa6effff700 1 -- 172.21.15.60:0/1448906628 shutdown_connections 2022-06-29T16:00:14.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.759+0000 7fa6effff700 1 --2- 172.21.15.60:0/1448906628 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa6d4053aa0 0x7fa6d4055f60 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:14.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.759+0000 7fa6effff700 1 --2- 172.21.15.60:0/1448906628 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6f015a270 0x7fa6f013fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:14.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.759+0000 7fa6effff700 1 --2- 172.21.15.60:0/1448906628 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa6f014a9b0 0x7fa6f0159d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:14.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.759+0000 7fa6effff700 1 --2- 172.21.15.60:0/1448906628 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6f0147370 0x7fa6f0159890 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:14.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.759+0000 7fa6effff700 1 -- 172.21.15.60:0/1448906628 >> 172.21.15.60:0/1448906628 conn(0x7fa6f00b5890 msgr2=0x7fa6f00af720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:14.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.759+0000 7fa6effff700 1 -- 172.21.15.60:0/1448906628 shutdown_connections 2022-06-29T16:00:14.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:14.759+0000 7fa6effff700 1 -- 172.21.15.60:0/1448906628 wait complete. 2022-06-29T16:00:14.984 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:00:15.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.087+0000 7f907498c700 1 Processor -- start 2022-06-29T16:00:15.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.087+0000 7f907498c700 1 -- start start 2022-06-29T16:00:15.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.091+0000 7f907498c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f907005c5b0 0x7f907005c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:15.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.091+0000 7f907498c700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9070060610 0x7f907013c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:15.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.091+0000 7f907498c700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90700b0a40 con 0x7f907013cc30 2022-06-29T16:00:15.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.091+0000 7f907498c700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90700b0850 con 0x7f9070060610 2022-06-29T16:00:15.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.091+0000 7f907498c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90700b0ce0 con 0x7f907005c5b0 2022-06-29T16:00:15.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.091+0000 7f906ed9d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f907005c5b0 0x7f907005c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:15.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.091+0000 7f9067fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9070060610 0x7f907013c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:15.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.091+0000 7f906f59e700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f907013cc30 0x7f9070142a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:59230/0 (socket says 172.21.15.60:59230) 2022-06-29T16:00:15.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.091+0000 7f906f59e700 1 -- 172.21.15.60:0/2820557998 learned_addr learned my addr 172.21.15.60:0/2820557998 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:00:15.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.091+0000 7f906e59c700 1 -- 172.21.15.60:0/2820557998 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 300023952 0 0) 0x7f90700b0a40 con 0x7f907013cc30 2022-06-29T16:00:15.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.091+0000 7f906e59c700 1 -- 172.21.15.60:0/2820557998 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9054003260 con 0x7f907013cc30 2022-06-29T16:00:15.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.091+0000 7f9067fff700 1 -- 172.21.15.60:0/2820557998 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f907005c5b0 msgr2=0x7f907005c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:15.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.091+0000 7f9067fff700 1 --2- 172.21.15.60:0/2820557998 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f907005c5b0 0x7f907005c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:15.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.091+0000 7f9067fff700 1 -- 172.21.15.60:0/2820557998 >> v1:172.21.15.92:6789/0 conn(0x7f907013cc30 legacy=0x7f9070142a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:15.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.091+0000 7f9067fff700 1 -- 172.21.15.60:0/2820557998 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f907005f350 con 0x7f9070060610 2022-06-29T16:00:15.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.091+0000 7f906ed9d700 1 --2- 172.21.15.60:0/2820557998 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f907005c5b0 0x7f907005c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:00:15.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.091+0000 7f9067fff700 1 --2- 172.21.15.60:0/2820557998 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9070060610 0x7f907013c740 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f9058002f70 tx=0x7f9058006df0).ready entity=mon.1 client_cookie=1343db594adc89c9 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:15.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.091+0000 7f906e59c700 1 -- 172.21.15.60:0/2820557998 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f905800d4a0 con 0x7f9070060610 2022-06-29T16:00:15.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.091+0000 7f906e59c700 1 -- 172.21.15.60:0/2820557998 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9058000e90 con 0x7f9070060610 2022-06-29T16:00:15.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.091+0000 7f906e59c700 1 -- 172.21.15.60:0/2820557998 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f905800d7f0 con 0x7f9070060610 2022-06-29T16:00:15.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.091+0000 7f907498c700 1 -- 172.21.15.60:0/2820557998 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9070060610 msgr2=0x7f907013c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:15.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.091+0000 7f907498c700 1 --2- 172.21.15.60:0/2820557998 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9070060610 0x7f907013c740 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f9058002f70 tx=0x7f9058006df0).stop 2022-06-29T16:00:15.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.091+0000 7f907498c700 1 -- 172.21.15.60:0/2820557998 shutdown_connections 2022-06-29T16:00:15.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.091+0000 7f907498c700 1 --2- 172.21.15.60:0/2820557998 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9070060610 0x7f907013c740 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:15.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.091+0000 7f907498c700 1 --2- 172.21.15.60:0/2820557998 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f907005c5b0 0x7f907005c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:15.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.091+0000 7f907498c700 1 -- 172.21.15.60:0/2820557998 >> 172.21.15.60:0/2820557998 conn(0x7f9070061c80 msgr2=0x7f90700637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:15.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.091+0000 7f907498c700 1 -- 172.21.15.60:0/2820557998 shutdown_connections 2022-06-29T16:00:15.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.091+0000 7f907498c700 1 -- 172.21.15.60:0/2820557998 wait complete. 2022-06-29T16:00:15.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.095+0000 7f907498c700 1 Processor -- start 2022-06-29T16:00:15.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.095+0000 7f907498c700 1 -- start start 2022-06-29T16:00:15.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.095+0000 7f907498c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f907005c5b0 0x7f9070146220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:15.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.095+0000 7f907498c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9070060610 0x7f907013efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:15.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.095+0000 7f907498c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f907013cc30 0x7f907013d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:15.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.095+0000 7f907498c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907013d050 con 0x7f907013cc30 2022-06-29T16:00:15.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.095+0000 7f907498c700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9070143810 con 0x7f907005c5b0 2022-06-29T16:00:15.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.095+0000 7f907498c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9070143ab0 con 0x7f9070060610 2022-06-29T16:00:15.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.095+0000 7f906f59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f907013cc30 0x7f907013d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:15.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.095+0000 7f906f59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f907013cc30 0x7f907013d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46906/0 (socket says 172.21.15.60:46906) 2022-06-29T16:00:15.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.095+0000 7f906f59e700 1 -- 172.21.15.60:0/3396786385 learned_addr learned my addr 172.21.15.60:0/3396786385 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:15.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.095+0000 7f9067fff700 1 --2- 172.21.15.60:0/3396786385 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9070060610 0x7f907013efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:15.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.095+0000 7f906ed9d700 1 --2- 172.21.15.60:0/3396786385 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f907005c5b0 0x7f9070146220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:15.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.095+0000 7f906f59e700 1 -- 172.21.15.60:0/3396786385 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9070060610 msgr2=0x7f907013efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:15.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.099+0000 7f906f59e700 1 --2- 172.21.15.60:0/3396786385 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9070060610 0x7f907013efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:15.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.099+0000 7f906f59e700 1 -- 172.21.15.60:0/3396786385 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f907005c5b0 msgr2=0x7f9070146220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:15.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.099+0000 7f906f59e700 1 --2- 172.21.15.60:0/3396786385 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f907005c5b0 0x7f9070146220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:15.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.099+0000 7f906f59e700 1 -- 172.21.15.60:0/3396786385 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f906000b180 con 0x7f907013cc30 2022-06-29T16:00:15.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.099+0000 7f906f59e700 1 --2- 172.21.15.60:0/3396786385 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f907013cc30 0x7f907013d480 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f9068001490 tx=0x7f9068007e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:15.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.099+0000 7f906cd99700 1 -- 172.21.15.60:0/3396786385 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9068005ba0 con 0x7f907013cc30 2022-06-29T16:00:15.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.099+0000 7f906cd99700 1 -- 172.21.15.60:0/3396786385 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9068005d00 con 0x7f907013cc30 2022-06-29T16:00:15.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.099+0000 7f907498c700 1 -- 172.21.15.60:0/3396786385 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f905800b180 con 0x7f907013cc30 2022-06-29T16:00:15.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.099+0000 7f906cd99700 1 -- 172.21.15.60:0/3396786385 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90680072c0 con 0x7f907013cc30 2022-06-29T16:00:15.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.099+0000 7f907498c700 1 -- 172.21.15.60:0/3396786385 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f907013f4d0 con 0x7f907013cc30 2022-06-29T16:00:15.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.099+0000 7f907498c700 1 -- 172.21.15.60:0/3396786385 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f907013d1c0 con 0x7f907013cc30 2022-06-29T16:00:15.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.099+0000 7f906cd99700 1 -- 172.21.15.60:0/3396786385 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9068010070 con 0x7f907013cc30 2022-06-29T16:00:15.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.099+0000 7f906cd99700 1 --2- 172.21.15.60:0/3396786385 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9048053950 0x7f9048055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:15.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.099+0000 7f906cd99700 1 -- 172.21.15.60:0/3396786385 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f906806b4f0 con 0x7f907013cc30 2022-06-29T16:00:15.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.103+0000 7f906ed9d700 1 --2- 172.21.15.60:0/3396786385 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9048053950 0x7f9048055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:15.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.103+0000 7f906ed9d700 1 --2- 172.21.15.60:0/3396786385 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9048053950 0x7f9048055e10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f906000b000 tx=0x7f9060006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:15.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.107+0000 7f906cd99700 1 -- 172.21.15.60:0/3396786385 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f90680426f0 con 0x7f907013cc30 2022-06-29T16:00:15.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.279+0000 7f907498c700 1 -- 172.21.15.60:0/3396786385 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90700573d0 con 0x7f907013cc30 2022-06-29T16:00:15.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.283+0000 7f906cd99700 1 -- 172.21.15.60:0/3396786385 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+1208 (secure 0 0 0) 0x7f9068069e60 con 0x7f907013cc30 2022-06-29T16:00:15.291 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:00:15.291 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4883,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.92:6835/3646976730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3646976730},{"type":"v1","addr":"172.21.15.92:6835","nonce":3646976730}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":12},{"gid":4884,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2555925680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2555925680},{"type":"v1","addr":"172.21.15.60:6833","nonce":2555925680}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":7}],"filesystems":[]} 2022-06-29T16:00:15.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.283+0000 7f907498c700 1 -- 172.21.15.60:0/3396786385 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9048053950 msgr2=0x7f9048055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:15.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.283+0000 7f907498c700 1 --2- 172.21.15.60:0/3396786385 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9048053950 0x7f9048055e10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f906000b000 tx=0x7f9060006df0).stop 2022-06-29T16:00:15.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.283+0000 7f907498c700 1 -- 172.21.15.60:0/3396786385 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f907013cc30 msgr2=0x7f907013d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:15.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.283+0000 7f907498c700 1 --2- 172.21.15.60:0/3396786385 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f907013cc30 0x7f907013d480 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f9068001490 tx=0x7f9068007e00).stop 2022-06-29T16:00:15.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.283+0000 7f907498c700 1 -- 172.21.15.60:0/3396786385 shutdown_connections 2022-06-29T16:00:15.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.283+0000 7f907498c700 1 --2- 172.21.15.60:0/3396786385 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9048053950 0x7f9048055e10 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:15.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.283+0000 7f907498c700 1 --2- 172.21.15.60:0/3396786385 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f907013cc30 0x7f907013d480 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:15.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.283+0000 7f907498c700 1 --2- 172.21.15.60:0/3396786385 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9070060610 0x7f907013efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:15.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.283+0000 7f907498c700 1 --2- 172.21.15.60:0/3396786385 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f907005c5b0 0x7f9070146220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:15.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.283+0000 7f907498c700 1 -- 172.21.15.60:0/3396786385 >> 172.21.15.60:0/3396786385 conn(0x7f9070061c80 msgr2=0x7f9070148190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:15.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.283+0000 7f907498c700 1 -- 172.21.15.60:0/3396786385 shutdown_connections 2022-06-29T16:00:15.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.283+0000 7f907498c700 1 -- 172.21.15.60:0/3396786385 wait complete. 2022-06-29T16:00:15.296 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 12 2022-06-29T16:00:15.304 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:00:15.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.399+0000 7fb87cf58700 1 Processor -- start 2022-06-29T16:00:15.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.399+0000 7fb87cf58700 1 -- start start 2022-06-29T16:00:15.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.399+0000 7fb87cf58700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb8780b2130 0x7fb878146100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:15.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.399+0000 7fb87cf58700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb87805d450 con 0x7fb87805d070 2022-06-29T16:00:15.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.399+0000 7fb87cf58700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8780b25d0 con 0x7fb8780aed00 2022-06-29T16:00:15.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.399+0000 7fb87cf58700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8780adf90 con 0x7fb8780b2130 2022-06-29T16:00:15.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.399+0000 7fb8777fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb87805d070 0x7fb87814aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54996/0 (socket says 172.21.15.60:54996) 2022-06-29T16:00:15.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.399+0000 7fb8777fe700 1 -- 172.21.15.60:0/4178924980 learned_addr learned my addr 172.21.15.60:0/4178924980 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:00:15.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.399+0000 7fb876ffd700 1 --2- 172.21.15.60:0/4178924980 >> v2:172.21.15.92:3300/0 conn(0x7fb8780b2130 0x7fb878146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:15.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.399+0000 7fb8767fc700 1 -- 172.21.15.60:0/4178924980 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2426220534 0 0) 0x7fb87805d450 con 0x7fb87805d070 2022-06-29T16:00:15.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.399+0000 7fb8767fc700 1 -- 172.21.15.60:0/4178924980 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb858003260 con 0x7fb87805d070 2022-06-29T16:00:15.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.399+0000 7fb876ffd700 1 -- 172.21.15.60:0/4178924980 >> v1:172.21.15.92:6789/0 conn(0x7fb8780aed00 legacy=0x7fb8781465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:15.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.399+0000 7fb876ffd700 1 -- 172.21.15.60:0/4178924980 >> v1:172.21.15.60:6789/0 conn(0x7fb87805d070 legacy=0x7fb87814aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:15.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.399+0000 7fb876ffd700 1 -- 172.21.15.60:0/4178924980 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87814a350 con 0x7fb8780b2130 2022-06-29T16:00:15.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.399+0000 7fb876ffd700 1 --2- 172.21.15.60:0/4178924980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8780b2130 0x7fb878146100 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fb8600032f0 tx=0x7fb8600091d0).ready entity=mon.1 client_cookie=3b4fc69ff7b40a63 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:15.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.403+0000 7fb8767fc700 1 -- 172.21.15.60:0/4178924980 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb86000b6c0 con 0x7fb8780b2130 2022-06-29T16:00:15.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.403+0000 7fb8767fc700 1 -- 172.21.15.60:0/4178924980 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb86000b820 con 0x7fb8780b2130 2022-06-29T16:00:15.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.403+0000 7fb8767fc700 1 -- 172.21.15.60:0/4178924980 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb86000b990 con 0x7fb8780b2130 2022-06-29T16:00:15.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.403+0000 7fb87cf58700 1 -- 172.21.15.60:0/4178924980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8780b2130 msgr2=0x7fb878146100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:15.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.403+0000 7fb87cf58700 1 --2- 172.21.15.60:0/4178924980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8780b2130 0x7fb878146100 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fb8600032f0 tx=0x7fb8600091d0).stop 2022-06-29T16:00:15.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.403+0000 7fb87cf58700 1 -- 172.21.15.60:0/4178924980 shutdown_connections 2022-06-29T16:00:15.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.403+0000 7fb87cf58700 1 --2- 172.21.15.60:0/4178924980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8780b2130 0x7fb878146100 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:15.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.403+0000 7fb87cf58700 1 -- 172.21.15.60:0/4178924980 >> 172.21.15.60:0/4178924980 conn(0x7fb8780b5890 msgr2=0x7fb8780b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:15.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.403+0000 7fb87cf58700 1 -- 172.21.15.60:0/4178924980 shutdown_connections 2022-06-29T16:00:15.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.403+0000 7fb87cf58700 1 -- 172.21.15.60:0/4178924980 wait complete. 2022-06-29T16:00:15.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.403+0000 7fb87cf58700 1 Processor -- start 2022-06-29T16:00:15.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.407+0000 7fb87cf58700 1 -- start start 2022-06-29T16:00:15.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.407+0000 7fb87cf58700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb87805d070 0x7fb8780b4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:15.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.407+0000 7fb87cf58700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8780aed00 0x7fb8780664d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:15.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.407+0000 7fb87cf58700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8780b2130 0x7fb8780669c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:15.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.407+0000 7fb87cf58700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8781429d0 con 0x7fb87805d070 2022-06-29T16:00:15.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.407+0000 7fb87cf58700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb878065390 con 0x7fb8780aed00 2022-06-29T16:00:15.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.407+0000 7fb87cf58700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb878116f90 con 0x7fb8780b2130 2022-06-29T16:00:15.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.407+0000 7fb8777fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb87805d070 0x7fb8780b4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:15.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.407+0000 7fb8777fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb87805d070 0x7fb8780b4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46916/0 (socket says 172.21.15.60:46916) 2022-06-29T16:00:15.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.407+0000 7fb8777fe700 1 -- 172.21.15.60:0/908447250 learned_addr learned my addr 172.21.15.60:0/908447250 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:15.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.407+0000 7fb876ffd700 1 --2- 172.21.15.60:0/908447250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8780aed00 0x7fb8780664d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:15.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.407+0000 7fb877fff700 1 --2- 172.21.15.60:0/908447250 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8780b2130 0x7fb8780669c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:15.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.407+0000 7fb8777fe700 1 -- 172.21.15.60:0/908447250 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8780b2130 msgr2=0x7fb8780669c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:15.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.407+0000 7fb8777fe700 1 --2- 172.21.15.60:0/908447250 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8780b2130 0x7fb8780669c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:15.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.407+0000 7fb8777fe700 1 -- 172.21.15.60:0/908447250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8780aed00 msgr2=0x7fb8780664d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:15.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.407+0000 7fb8777fe700 1 --2- 172.21.15.60:0/908447250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8780aed00 0x7fb8780664d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:15.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.407+0000 7fb8777fe700 1 -- 172.21.15.60:0/908447250 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb860009470 con 0x7fb87805d070 2022-06-29T16:00:15.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.407+0000 7fb876ffd700 1 --2- 172.21.15.60:0/908447250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8780aed00 0x7fb8780664d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:00:15.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.407+0000 7fb8777fe700 1 --2- 172.21.15.60:0/908447250 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb87805d070 0x7fb8780b4440 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fb868001110 tx=0x7fb868007ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:15.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.407+0000 7fb874ff9700 1 -- 172.21.15.60:0/908447250 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8680072e0 con 0x7fb87805d070 2022-06-29T16:00:15.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.407+0000 7fb874ff9700 1 -- 172.21.15.60:0/908447250 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb868007440 con 0x7fb87805d070 2022-06-29T16:00:15.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.407+0000 7fb87cf58700 1 -- 172.21.15.60:0/908447250 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb87815a600 con 0x7fb87805d070 2022-06-29T16:00:15.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.407+0000 7fb874ff9700 1 -- 172.21.15.60:0/908447250 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb868005800 con 0x7fb87805d070 2022-06-29T16:00:15.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.407+0000 7fb87cf58700 1 -- 172.21.15.60:0/908447250 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8780b4930 con 0x7fb87805d070 2022-06-29T16:00:15.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.411+0000 7fb87cf58700 1 -- 172.21.15.60:0/908447250 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb87814a5f0 con 0x7fb87805d070 2022-06-29T16:00:15.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.411+0000 7fb874ff9700 1 -- 172.21.15.60:0/908447250 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb868010070 con 0x7fb87805d070 2022-06-29T16:00:15.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.411+0000 7fb874ff9700 1 --2- 172.21.15.60:0/908447250 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb85c053a60 0x7fb85c055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:15.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.411+0000 7fb874ff9700 1 -- 172.21.15.60:0/908447250 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fb86806b3c0 con 0x7fb87805d070 2022-06-29T16:00:15.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.411+0000 7fb876ffd700 1 --2- 172.21.15.60:0/908447250 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb85c053a60 0x7fb85c055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:15.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.411+0000 7fb876ffd700 1 --2- 172.21.15.60:0/908447250 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb85c053a60 0x7fb85c055f20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fb860019000 tx=0x7fb860016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:15.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.415+0000 7fb874ff9700 1 -- 172.21.15.60:0/908447250 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb8680425c0 con 0x7fb87805d070 2022-06-29T16:00:15.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.583+0000 7fb87cf58700 1 -- 172.21.15.60:0/908447250 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb878116db0 con 0x7fb87805d070 2022-06-29T16:00:15.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.587+0000 7fb874ff9700 1 -- 172.21.15.60:0/908447250 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+1208 (secure 0 0 0) 0x7fb868045be0 con 0x7fb87805d070 2022-06-29T16:00:15.595 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:00:15.595 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4883,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.92:6835/3646976730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3646976730},{"type":"v1","addr":"172.21.15.92:6835","nonce":3646976730}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":12},{"gid":4884,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/2555925680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2555925680},{"type":"v1","addr":"172.21.15.60:6833","nonce":2555925680}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":7}],"filesystems":[]} 2022-06-29T16:00:15.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.587+0000 7fb87cf58700 1 -- 172.21.15.60:0/908447250 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb85c053a60 msgr2=0x7fb85c055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:15.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.587+0000 7fb87cf58700 1 --2- 172.21.15.60:0/908447250 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb85c053a60 0x7fb85c055f20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fb860019000 tx=0x7fb860016000).stop 2022-06-29T16:00:15.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.587+0000 7fb87cf58700 1 -- 172.21.15.60:0/908447250 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb87805d070 msgr2=0x7fb8780b4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:15.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.587+0000 7fb87cf58700 1 --2- 172.21.15.60:0/908447250 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb87805d070 0x7fb8780b4440 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fb868001110 tx=0x7fb868007ad0).stop 2022-06-29T16:00:15.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.587+0000 7fb87cf58700 1 -- 172.21.15.60:0/908447250 shutdown_connections 2022-06-29T16:00:15.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.587+0000 7fb87cf58700 1 --2- 172.21.15.60:0/908447250 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb85c053a60 0x7fb85c055f20 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:15.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.587+0000 7fb87cf58700 1 --2- 172.21.15.60:0/908447250 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8780b2130 0x7fb8780669c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:15.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.587+0000 7fb87cf58700 1 --2- 172.21.15.60:0/908447250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8780aed00 0x7fb8780664d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:15.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.587+0000 7fb87cf58700 1 --2- 172.21.15.60:0/908447250 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb87805d070 0x7fb8780b4440 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:15.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.587+0000 7fb87cf58700 1 -- 172.21.15.60:0/908447250 >> 172.21.15.60:0/908447250 conn(0x7fb8780b5890 msgr2=0x7fb8781452e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:15.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.587+0000 7fb87cf58700 1 -- 172.21.15.60:0/908447250 shutdown_connections 2022-06-29T16:00:15.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.587+0000 7fb87cf58700 1 -- 172.21.15.60:0/908447250 wait complete. 2022-06-29T16:00:15.600 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 12 2022-06-29T16:00:15.608 INFO:tasks.ceph.mds.a:Restarting daemon 2022-06-29T16:00:15.609 INFO:tasks.ceph.mds.a:Stopping old one... 2022-06-29T16:00:15.609 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-06-29T16:00:15.610 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T16:00:15.610 INFO:tasks.ceph.mds.b:Restarting daemon 2022-06-29T16:00:15.611 INFO:tasks.ceph.mds.b:Stopping old one... 2022-06-29T16:00:15.612 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-06-29T16:00:15.612 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T16:00:15.622 INFO:tasks.ceph.mds.a:Stopped 2022-06-29T16:00:15.622 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-06-29T16:00:15.628 INFO:tasks.ceph.mds.b:Stopped 2022-06-29T16:00:15.628 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-06-29T16:00:15.630 INFO:tasks.ceph.mds.b:Started 2022-06-29T16:00:15.631 DEBUG:teuthology.parallel:result is None 2022-06-29T16:00:15.667 INFO:tasks.ceph.mds.a:Started 2022-06-29T16:00:15.668 DEBUG:teuthology.parallel:result is None 2022-06-29T16:00:15.668 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-06-29T16:00:15.698 INFO:tasks.ceph.mds.b.smithi092.stdout:starting mds.b at 2022-06-29T16:00:15.740 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2022-06-29T16:00:15.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.763+0000 7f955759e700 1 Processor -- start 2022-06-29T16:00:15.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.763+0000 7f955759e700 1 -- start start 2022-06-29T16:00:15.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.763+0000 7f955759e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f95580b48b0 0x7f9558142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:15.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.763+0000 7f955759e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95580b2c50 0x7f95580b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:15.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.763+0000 7f955759e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95580b0cf0 con 0x7f95580b5eb0 2022-06-29T16:00:15.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.763+0000 7f955759e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95580b54e0 con 0x7f95580b48b0 2022-06-29T16:00:15.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.763+0000 7f955759e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95580b4d10 con 0x7f95580b2c50 2022-06-29T16:00:15.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.763+0000 7f955659c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f95580b5eb0 0x7f95580b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55022/0 (socket says 172.21.15.60:55022) 2022-06-29T16:00:15.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.763+0000 7f955659c700 1 -- 172.21.15.60:0/3771265390 learned_addr learned my addr 172.21.15.60:0/3771265390 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:00:15.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.763+0000 7f9555d9b700 1 --2- 172.21.15.60:0/3771265390 >> v2:172.21.15.60:3300/0 conn(0x7f95580b48b0 0x7f9558142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:15.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.763+0000 7f9556d9d700 1 --2- 172.21.15.60:0/3771265390 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95580b2c50 0x7f95580b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:15.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.763+0000 7f9555d9b700 1 -- 172.21.15.60:0/3771265390 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95580b2c50 msgr2=0x7f95580b30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:15.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.763+0000 7f9555d9b700 1 --2- 172.21.15.60:0/3771265390 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95580b2c50 0x7f95580b30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:15.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.763+0000 7f9555d9b700 1 -- 172.21.15.60:0/3771265390 >> v1:172.21.15.60:6789/0 conn(0x7f95580b5eb0 legacy=0x7f95580b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:15.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.763+0000 7f9555d9b700 1 -- 172.21.15.60:0/3771265390 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95580b6290 con 0x7f95580b48b0 2022-06-29T16:00:15.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.763+0000 7f9555d9b700 1 --2- 172.21.15.60:0/3771265390 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95580b48b0 0x7f9558142a20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f9540002f70 tx=0x7f9540006df0).ready entity=mon.0 client_cookie=f4faa9d89562af81 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:15.778 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.767+0000 7f955559a700 1 -- 172.21.15.60:0/3771265390 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f954000d570 con 0x7f95580b48b0 2022-06-29T16:00:15.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.767+0000 7f955559a700 1 -- 172.21.15.60:0/3771265390 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9540000e90 con 0x7f95580b48b0 2022-06-29T16:00:15.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.767+0000 7f955559a700 1 -- 172.21.15.60:0/3771265390 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f954000d8c0 con 0x7f95580b48b0 2022-06-29T16:00:15.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.767+0000 7f955759e700 1 -- 172.21.15.60:0/3771265390 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95580b48b0 msgr2=0x7f9558142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:15.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.767+0000 7f955759e700 1 --2- 172.21.15.60:0/3771265390 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95580b48b0 0x7f9558142a20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f9540002f70 tx=0x7f9540006df0).stop 2022-06-29T16:00:15.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.767+0000 7f955759e700 1 -- 172.21.15.60:0/3771265390 shutdown_connections 2022-06-29T16:00:15.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.767+0000 7f955759e700 1 --2- 172.21.15.60:0/3771265390 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95580b2c50 0x7f95580b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:15.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.767+0000 7f955759e700 1 --2- 172.21.15.60:0/3771265390 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95580b48b0 0x7f9558142a20 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:15.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.767+0000 7f955759e700 1 -- 172.21.15.60:0/3771265390 >> 172.21.15.60:0/3771265390 conn(0x7f9558062090 msgr2=0x7f9558062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:15.781 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.767+0000 7f955759e700 1 -- 172.21.15.60:0/3771265390 shutdown_connections 2022-06-29T16:00:15.781 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.767+0000 7f955759e700 1 -- 172.21.15.60:0/3771265390 wait complete. 2022-06-29T16:00:15.782 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.771+0000 7f955759e700 1 Processor -- start 2022-06-29T16:00:15.782 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.771+0000 7f955759e700 1 -- start start 2022-06-29T16:00:15.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.771+0000 7f955759e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95580b2c50 0x7f955813f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:15.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.771+0000 7f955759e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95580b48b0 0x7f95580af220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:15.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.771+0000 7f955759e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95580b5eb0 0x7f9558140720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:15.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.771+0000 7f955759e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9558144060 con 0x7f95580b2c50 2022-06-29T16:00:15.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.771+0000 7f955759e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95580adb10 con 0x7f95580b5eb0 2022-06-29T16:00:15.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.771+0000 7f955759e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f955813e1c0 con 0x7f95580b48b0 2022-06-29T16:00:15.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.771+0000 7f955659c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95580b2c50 0x7f955813f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:15.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.771+0000 7f955659c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95580b2c50 0x7f955813f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46942/0 (socket says 172.21.15.60:46942) 2022-06-29T16:00:15.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.771+0000 7f955659c700 1 -- 172.21.15.60:0/1679910093 learned_addr learned my addr 172.21.15.60:0/1679910093 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:15.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.771+0000 7f9556d9d700 1 --2- 172.21.15.60:0/1679910093 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95580b5eb0 0x7f9558140720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:15.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.771+0000 7f9555d9b700 1 --2- 172.21.15.60:0/1679910093 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95580b48b0 0x7f95580af220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:15.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.771+0000 7f955659c700 1 -- 172.21.15.60:0/1679910093 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95580b48b0 msgr2=0x7f95580af220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:15.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.771+0000 7f955659c700 1 --2- 172.21.15.60:0/1679910093 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95580b48b0 0x7f95580af220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:15.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.771+0000 7f955659c700 1 -- 172.21.15.60:0/1679910093 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95580b5eb0 msgr2=0x7f9558140720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:15.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.771+0000 7f955659c700 1 --2- 172.21.15.60:0/1679910093 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95580b5eb0 0x7f9558140720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:15.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.771+0000 7f955659c700 1 -- 172.21.15.60:0/1679910093 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f954000b180 con 0x7f95580b2c50 2022-06-29T16:00:15.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.771+0000 7f9555d9b700 1 --2- 172.21.15.60:0/1679910093 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95580b48b0 0x7f95580af220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:00:15.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.771+0000 7f955659c700 1 --2- 172.21.15.60:0/1679910093 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95580b2c50 0x7f955813f880 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f9558062f00 tx=0x7f9550007f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:15.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.771+0000 7f954f7fe700 1 -- 172.21.15.60:0/1679910093 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9550005ba0 con 0x7f95580b2c50 2022-06-29T16:00:15.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.771+0000 7f954f7fe700 1 -- 172.21.15.60:0/1679910093 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9550005d00 con 0x7f95580b2c50 2022-06-29T16:00:15.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.771+0000 7f954f7fe700 1 -- 172.21.15.60:0/1679910093 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f95500072c0 con 0x7f95580b2c50 2022-06-29T16:00:15.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.771+0000 7f955759e700 1 -- 172.21.15.60:0/1679910093 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95580af710 con 0x7f95580b2c50 2022-06-29T16:00:15.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.771+0000 7f955759e700 1 -- 172.21.15.60:0/1679910093 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f955813edd0 con 0x7f95580b2c50 2022-06-29T16:00:15.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.775+0000 7f955759e700 1 -- 172.21.15.60:0/1679910093 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95581441d0 con 0x7f95580b2c50 2022-06-29T16:00:15.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.775+0000 7f954f7fe700 1 -- 172.21.15.60:0/1679910093 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9550010070 con 0x7f95580b2c50 2022-06-29T16:00:15.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.775+0000 7f954f7fe700 1 --2- 172.21.15.60:0/1679910093 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f953c053a70 0x7f953c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:15.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.775+0000 7f954f7fe700 1 -- 172.21.15.60:0/1679910093 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f955006b3f0 con 0x7f95580b2c50 2022-06-29T16:00:15.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.775+0000 7f9555d9b700 1 --2- 172.21.15.60:0/1679910093 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f953c053a70 0x7f953c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:15.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.775+0000 7f9555d9b700 1 --2- 172.21.15.60:0/1679910093 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f953c053a70 0x7f953c055f30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f954001a000 tx=0x7f9540015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:15.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.779+0000 7f954f7fe700 1 -- 172.21.15.60:0/1679910093 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f95581441d0 con 0x7f95580b2c50 2022-06-29T16:00:15.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:15.911+0000 7f955759e700 1 -- 172.21.15.60:0/1679910093 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f95580b0cf0 con 0x7f95580b2c50 2022-06-29T16:00:16.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.775+0000 7f954f7fe700 1 -- 172.21.15.60:0/1679910093 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v38) v1 ==== 97+0+0 (secure 0 0 0) 0x7f95580b0cf0 con 0x7f95580b2c50 2022-06-29T16:00:16.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.779+0000 7f955759e700 1 -- 172.21.15.60:0/1679910093 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f953c053a70 msgr2=0x7f953c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:16.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.779+0000 7f955759e700 1 --2- 172.21.15.60:0/1679910093 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f953c053a70 0x7f953c055f30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f954001a000 tx=0x7f9540015000).stop 2022-06-29T16:00:16.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.779+0000 7f955759e700 1 -- 172.21.15.60:0/1679910093 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95580b2c50 msgr2=0x7f955813f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:16.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.779+0000 7f955759e700 1 --2- 172.21.15.60:0/1679910093 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95580b2c50 0x7f955813f880 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f9558062f00 tx=0x7f9550007f90).stop 2022-06-29T16:00:16.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.779+0000 7f955759e700 1 -- 172.21.15.60:0/1679910093 shutdown_connections 2022-06-29T16:00:16.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.779+0000 7f955759e700 1 --2- 172.21.15.60:0/1679910093 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f953c053a70 0x7f953c055f30 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:16.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.779+0000 7f955759e700 1 --2- 172.21.15.60:0/1679910093 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95580b5eb0 0x7f9558140720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:16.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.779+0000 7f955759e700 1 --2- 172.21.15.60:0/1679910093 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95580b48b0 0x7f95580af220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:16.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.779+0000 7f955759e700 1 --2- 172.21.15.60:0/1679910093 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95580b2c50 0x7f955813f880 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:16.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.779+0000 7f955759e700 1 -- 172.21.15.60:0/1679910093 >> 172.21.15.60:0/1679910093 conn(0x7f9558062090 msgr2=0x7f955805bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:16.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.779+0000 7f955759e700 1 -- 172.21.15.60:0/1679910093 shutdown_connections 2022-06-29T16:00:16.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.779+0000 7f955759e700 1 -- 172.21.15.60:0/1679910093 wait complete. 2022-06-29T16:00:16.794 INFO:teuthology.orchestra.run.smithi060.stderr: removed all blacklist entries 2022-06-29T16:00:16.804 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-06-29T16:00:16.804 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-06-29T16:00:16.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.903+0000 7f5b0124c700 1 Processor -- start 2022-06-29T16:00:16.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.903+0000 7f5b0124c700 1 -- start start 2022-06-29T16:00:16.915 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.907+0000 7f5b0124c700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5afc0b9d70 0x7f5afc0b3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:16.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.907+0000 7f5b0124c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5afc0b3cc0 0x7f5afc1420b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:16.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.907+0000 7f5b0124c700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5afc0b2ce0 con 0x7f5afc05d1f0 2022-06-29T16:00:16.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.907+0000 7f5b0124c700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5afc0b4d10 con 0x7f5afc0b9d70 2022-06-29T16:00:16.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.907+0000 7f5b0124c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc0ba150 con 0x7f5afc0b3cc0 2022-06-29T16:00:16.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.907+0000 7f5afad9d700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5afc0b9d70 0x7f5afc0b3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:16.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.907+0000 7f5afb59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5afc0b3cc0 0x7f5afc1420b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:16.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.907+0000 7f5afa59c700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f5afc05d1f0 0x7f5afc0b1b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:59286/0 (socket says 172.21.15.60:59286) 2022-06-29T16:00:16.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.907+0000 7f5afa59c700 1 -- 172.21.15.60:0/2325096685 learned_addr learned my addr 172.21.15.60:0/2325096685 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:00:16.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.907+0000 7f5afb59e700 1 -- 172.21.15.60:0/2325096685 >> v2:172.21.15.92:3300/0 conn(0x7f5afc0b9d70 msgr2=0x7f5afc0b3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:16.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.907+0000 7f5afb59e700 1 --2- 172.21.15.60:0/2325096685 >> v2:172.21.15.92:3300/0 conn(0x7f5afc0b9d70 0x7f5afc0b3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:16.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.907+0000 7f5afb59e700 1 -- 172.21.15.60:0/2325096685 >> v1:172.21.15.92:6789/0 conn(0x7f5afc05d1f0 legacy=0x7f5afc0b1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:16.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.907+0000 7f5afb59e700 1 -- 172.21.15.60:0/2325096685 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5afc0b4a70 con 0x7f5afc0b3cc0 2022-06-29T16:00:16.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.907+0000 7f5afb59e700 1 --2- 172.21.15.60:0/2325096685 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5afc0b3cc0 0x7f5afc1420b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f5af40032f0 tx=0x7f5af40091d0).ready entity=mon.2 client_cookie=ac6396734ac91442 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:16.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.907+0000 7f5af9d9b700 1 -- 172.21.15.60:0/2325096685 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5af400b6c0 con 0x7f5afc0b3cc0 2022-06-29T16:00:16.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.907+0000 7f5af9d9b700 1 -- 172.21.15.60:0/2325096685 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af400b820 con 0x7f5afc0b3cc0 2022-06-29T16:00:16.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.907+0000 7f5af9d9b700 1 -- 172.21.15.60:0/2325096685 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5af400b990 con 0x7f5afc0b3cc0 2022-06-29T16:00:16.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.907+0000 7f5b0124c700 1 -- 172.21.15.60:0/2325096685 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5afc0b3cc0 msgr2=0x7f5afc1420b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:16.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.907+0000 7f5b0124c700 1 --2- 172.21.15.60:0/2325096685 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5afc0b3cc0 0x7f5afc1420b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f5af40032f0 tx=0x7f5af40091d0).stop 2022-06-29T16:00:16.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.907+0000 7f5b0124c700 1 -- 172.21.15.60:0/2325096685 shutdown_connections 2022-06-29T16:00:16.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.907+0000 7f5b0124c700 1 --2- 172.21.15.60:0/2325096685 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5afc0b3cc0 0x7f5afc1420b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:16.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.907+0000 7f5b0124c700 1 --2- 172.21.15.60:0/2325096685 >> v2:172.21.15.92:3300/0 conn(0x7f5afc0b9d70 0x7f5afc0b3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:16.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.907+0000 7f5b0124c700 1 -- 172.21.15.60:0/2325096685 >> 172.21.15.60:0/2325096685 conn(0x7f5afc062ce0 msgr2=0x7f5afc0b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:16.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.907+0000 7f5b0124c700 1 -- 172.21.15.60:0/2325096685 shutdown_connections 2022-06-29T16:00:16.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.907+0000 7f5b0124c700 1 -- 172.21.15.60:0/2325096685 wait complete. 2022-06-29T16:00:16.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.911+0000 7f5b0124c700 1 Processor -- start 2022-06-29T16:00:16.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.911+0000 7f5b0124c700 1 -- start start 2022-06-29T16:00:16.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.911+0000 7f5b0124c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5afc05d1f0 0x7f5afc0ad500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:16.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.911+0000 7f5b0124c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5afc0b3cc0 0x7f5afc0ae290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:16.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.911+0000 7f5b0124c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5afc0b9d70 0x7f5afc145140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:16.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.911+0000 7f5b0124c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc141580 con 0x7f5afc0b3cc0 2022-06-29T16:00:16.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.911+0000 7f5b0124c700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc0af3a0 con 0x7f5afc05d1f0 2022-06-29T16:00:16.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.911+0000 7f5b0124c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc0af5d0 con 0x7f5afc0b9d70 2022-06-29T16:00:16.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.911+0000 7f5afa59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5afc0b3cc0 0x7f5afc0ae290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:16.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.911+0000 7f5afa59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5afc0b3cc0 0x7f5afc0ae290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46960/0 (socket says 172.21.15.60:46960) 2022-06-29T16:00:16.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.911+0000 7f5afa59c700 1 -- 172.21.15.60:0/2697175607 learned_addr learned my addr 172.21.15.60:0/2697175607 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:16.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.911+0000 7f5afad9d700 1 --2- 172.21.15.60:0/2697175607 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5afc05d1f0 0x7f5afc0ad500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:16.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.911+0000 7f5afb59e700 1 --2- 172.21.15.60:0/2697175607 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5afc0b9d70 0x7f5afc145140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:16.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.911+0000 7f5afa59c700 1 -- 172.21.15.60:0/2697175607 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5afc0b9d70 msgr2=0x7f5afc145140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:16.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.911+0000 7f5afa59c700 1 --2- 172.21.15.60:0/2697175607 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5afc0b9d70 0x7f5afc145140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:16.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.911+0000 7f5afa59c700 1 -- 172.21.15.60:0/2697175607 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5afc05d1f0 msgr2=0x7f5afc0ad500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:16.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.911+0000 7f5afa59c700 1 --2- 172.21.15.60:0/2697175607 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5afc05d1f0 0x7f5afc0ad500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:16.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.911+0000 7f5afa59c700 1 -- 172.21.15.60:0/2697175607 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af4009470 con 0x7f5afc0b3cc0 2022-06-29T16:00:16.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.911+0000 7f5afad9d700 1 --2- 172.21.15.60:0/2697175607 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5afc05d1f0 0x7f5afc0ad500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:00:16.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.915+0000 7f5afa59c700 1 --2- 172.21.15.60:0/2697175607 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5afc0b3cc0 0x7f5afc0ae290 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f5afc060000 tx=0x7f5aec007c30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:16.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.915+0000 7f5af3fff700 1 -- 172.21.15.60:0/2697175607 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5aec0070d0 con 0x7f5afc0b3cc0 2022-06-29T16:00:16.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.915+0000 7f5af3fff700 1 -- 172.21.15.60:0/2697175607 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aec007230 con 0x7f5afc0b3cc0 2022-06-29T16:00:16.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.915+0000 7f5b0124c700 1 -- 172.21.15.60:0/2697175607 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5afc145b80 con 0x7f5afc0b3cc0 2022-06-29T16:00:16.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.915+0000 7f5af3fff700 1 -- 172.21.15.60:0/2697175607 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5aec0055d0 con 0x7f5afc0b3cc0 2022-06-29T16:00:16.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.915+0000 7f5b0124c700 1 -- 172.21.15.60:0/2697175607 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5afc145630 con 0x7f5afc0b3cc0 2022-06-29T16:00:16.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.915+0000 7f5b0124c700 1 -- 172.21.15.60:0/2697175607 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5afc0573d0 con 0x7f5afc0b3cc0 2022-06-29T16:00:16.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.915+0000 7f5af3fff700 1 -- 172.21.15.60:0/2697175607 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5aec010070 con 0x7f5afc0b3cc0 2022-06-29T16:00:16.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.915+0000 7f5af3fff700 1 --2- 172.21.15.60:0/2697175607 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5ae0053a70 0x7f5ae0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:16.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.919+0000 7f5af3fff700 1 -- 172.21.15.60:0/2697175607 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f5aec06b0c0 con 0x7f5afc0b3cc0 2022-06-29T16:00:16.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.919+0000 7f5afad9d700 1 --2- 172.21.15.60:0/2697175607 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5ae0053a70 0x7f5ae0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:16.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.919+0000 7f5afad9d700 1 --2- 172.21.15.60:0/2697175607 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5ae0053a70 0x7f5ae0055f30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f5ae400b000 tx=0x7f5ae4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:16.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:16.923+0000 7f5af3fff700 1 -- 172.21.15.60:0/2697175607 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5aec042310 con 0x7f5afc0b3cc0 2022-06-29T16:00:17.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.095+0000 7f5b0124c700 1 -- 172.21.15.60:0/2697175607 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f5afc05d990 con 0x7f5afc0b3cc0 2022-06-29T16:00:17.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.095+0000 7f5af3fff700 1 -- 172.21.15.60:0/2697175607 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+4242 (secure 0 0 0) 0x7f5aec018030 con 0x7f5afc0b3cc0 2022-06-29T16:00:17.105 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:00:17.105 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:00:17.105 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_dump": [ 2022-06-29T16:00:17.106 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:17.106 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.a", 2022-06-29T16:00:17.106 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxivv7JNhAAnA291UxlB/SfgITOipP00A==", 2022-06-29T16:00:17.107 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:00:17.107 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:00:17.108 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:00:17.108 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:00:17.109 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:00:17.109 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:17.110 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:17.110 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:17.111 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.b", 2022-06-29T16:00:17.111 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxiRSr/OhAAfFQTF2YHvzCv5wXBVCA5WQ==", 2022-06-29T16:00:17.112 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:00:17.112 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:00:17.113 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:00:17.113 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:00:17.114 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:00:17.120 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:17.120 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:17.121 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:17.121 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.0", 2022-06-29T16:00:17.122 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiZYPxOBAAIkweWJlTJIXgxnlagkGdXA==", 2022-06-29T16:00:17.122 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:00:17.123 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:00:17.123 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:00:17.124 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:00:17.124 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:17.125 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:17.125 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:17.126 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.1", 2022-06-29T16:00:17.126 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxiS2JGBhAAJn0UfAxZI3IVFyD5QBdb/Q==", 2022-06-29T16:00:17.127 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:00:17.127 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:00:17.128 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:00:17.128 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:00:17.128 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:17.129 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:17.129 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:17.130 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.2", 2022-06-29T16:00:17.130 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxiPhkfEhAApH7r8OUJzW7fZNM6XVv1dg==", 2022-06-29T16:00:17.131 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:00:17.131 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:00:17.132 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:00:17.132 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:00:17.133 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:17.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:17.133 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:17.134 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.3", 2022-06-29T16:00:17.135 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxit/47HhAA1vyU6h7hjPG+op9qP/rVYQ==", 2022-06-29T16:00:17.135 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:00:17.136 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:00:17.136 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:00:17.137 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:00:17.137 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:17.138 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:17.138 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:17.139 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.4", 2022-06-29T16:00:17.139 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxiaKzKBxAAbIwCF9ag7ezsNy11hi0BEw==", 2022-06-29T16:00:17.139 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:00:17.140 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:00:17.140 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:00:17.141 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:00:17.141 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:17.142 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:17.142 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:17.143 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.5", 2022-06-29T16:00:17.143 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxiJQV/IBAASBBDb6UFgEtXILe4fcT/xA==", 2022-06-29T16:00:17.144 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:00:17.144 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:00:17.145 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:00:17.145 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:00:17.146 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:17.146 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:17.147 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:17.147 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.6", 2022-06-29T16:00:17.148 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxi6D07OBAA2FWiZ+mNK+jFMp/kPZnbQQ==", 2022-06-29T16:00:17.148 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:00:17.149 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:00:17.149 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:00:17.150 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:00:17.150 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:17.150 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:17.151 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:17.151 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.7", 2022-06-29T16:00:17.151 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCMdrxinEOcEhAA562ybpzGHNQfWIq8q7zw9w==", 2022-06-29T16:00:17.151 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:00:17.151 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:00:17.152 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:00:17.152 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:00:17.152 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:17.152 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:17.153 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:17.153 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.0", 2022-06-29T16:00:17.153 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiHXPwBRAAkOOwlYJvIW7dqVFdlXXoqw==", 2022-06-29T16:00:17.153 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:00:17.153 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:00:17.154 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2022-06-29T16:00:17.154 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw pool=cephfs_data" 2022-06-29T16:00:17.154 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:17.154 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:17.155 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:17.155 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.1", 2022-06-29T16:00:17.155 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxibe5JCRAAGBJbi9t09w0836JqrAT7+w==", 2022-06-29T16:00:17.155 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:00:17.155 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:00:17.156 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2022-06-29T16:00:17.156 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw pool=cephfs_data" 2022-06-29T16:00:17.156 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:17.157 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:17.157 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:17.157 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.2", 2022-06-29T16:00:17.157 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiK3dBDRAA7XINPSOPQDfOrUYEpHC+gw==", 2022-06-29T16:00:17.158 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:00:17.158 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:00:17.158 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2022-06-29T16:00:17.159 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw pool=cephfs_data" 2022-06-29T16:00:17.159 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:17.159 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:17.159 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:17.159 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.admin", 2022-06-29T16:00:17.160 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxieKVcGBAAnjH80a0qWIArz2qwjYHWdQ==", 2022-06-29T16:00:17.160 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:00:17.160 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T16:00:17.160 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:00:17.160 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:00:17.161 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:00:17.161 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:17.161 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:17.161 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:17.162 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.x", 2022-06-29T16:00:17.162 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxi2LZJNBAA3MKb/pXYbQCsoRx8EQfGbw==", 2022-06-29T16:00:17.162 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:00:17.162 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T16:00:17.163 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2022-06-29T16:00:17.163 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:00:17.163 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:17.163 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:17.163 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:17.164 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.y", 2022-06-29T16:00:17.164 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxi3LmaMBAAN+0mZrdKXqSXOipVTA1drw==", 2022-06-29T16:00:17.165 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:00:17.165 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T16:00:17.165 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2022-06-29T16:00:17.165 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:00:17.166 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:17.166 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:17.166 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:00:17.167 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:00:17.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.095+0000 7f5b0124c700 1 -- 172.21.15.60:0/2697175607 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5ae0053a70 msgr2=0x7f5ae0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:17.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.095+0000 7f5b0124c700 1 --2- 172.21.15.60:0/2697175607 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5ae0053a70 0x7f5ae0055f30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f5ae400b000 tx=0x7f5ae4006df0).stop 2022-06-29T16:00:17.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.095+0000 7f5b0124c700 1 -- 172.21.15.60:0/2697175607 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5afc0b3cc0 msgr2=0x7f5afc0ae290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:17.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.095+0000 7f5b0124c700 1 --2- 172.21.15.60:0/2697175607 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5afc0b3cc0 0x7f5afc0ae290 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f5afc060000 tx=0x7f5aec007c30).stop 2022-06-29T16:00:17.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.095+0000 7f5b0124c700 1 -- 172.21.15.60:0/2697175607 shutdown_connections 2022-06-29T16:00:17.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.095+0000 7f5b0124c700 1 --2- 172.21.15.60:0/2697175607 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5ae0053a70 0x7f5ae0055f30 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:17.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.095+0000 7f5b0124c700 1 --2- 172.21.15.60:0/2697175607 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5afc0b9d70 0x7f5afc145140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:17.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.095+0000 7f5b0124c700 1 --2- 172.21.15.60:0/2697175607 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5afc0b3cc0 0x7f5afc0ae290 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:17.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.095+0000 7f5b0124c700 1 --2- 172.21.15.60:0/2697175607 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5afc05d1f0 0x7f5afc0ad500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:17.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.099+0000 7f5b0124c700 1 -- 172.21.15.60:0/2697175607 >> 172.21.15.60:0/2697175607 conn(0x7f5afc062ce0 msgr2=0x7f5afc13e750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:17.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.099+0000 7f5b0124c700 1 -- 172.21.15.60:0/2697175607 shutdown_connections 2022-06-29T16:00:17.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.099+0000 7f5b0124c700 1 -- 172.21.15.60:0/2697175607 wait complete. 2022-06-29T16:00:17.173 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:00:17.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.275+0000 7f786daa6700 1 Processor -- start 2022-06-29T16:00:17.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.275+0000 7f786daa6700 1 -- start start 2022-06-29T16:00:17.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.275+0000 7f786daa6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78680aed00 0x7f7868146810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:17.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.275+0000 7f786daa6700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f786805d450 con 0x7f78680b2130 2022-06-29T16:00:17.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.275+0000 7f786daa6700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78680b25d0 con 0x7f786805d070 2022-06-29T16:00:17.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.275+0000 7f786daa6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78680adf90 con 0x7f78680aed00 2022-06-29T16:00:17.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.275+0000 7f7867fff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f78680b2130 0x7f7868146100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55052/0 (socket says 172.21.15.60:55052) 2022-06-29T16:00:17.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.275+0000 7f7867fff700 1 -- 172.21.15.60:0/2233574757 learned_addr learned my addr 172.21.15.60:0/2233574757 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:00:17.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.275+0000 7f786d2a5700 1 --2- 172.21.15.60:0/2233574757 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78680aed00 0x7f7868146810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:17.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.275+0000 7f78677fe700 1 -- 172.21.15.60:0/2233574757 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 268497214 0 0) 0x7f786805d450 con 0x7f78680b2130 2022-06-29T16:00:17.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f78677fe700 1 -- 172.21.15.60:0/2233574757 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f784c003260 con 0x7f78680b2130 2022-06-29T16:00:17.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f78677fe700 1 -- 172.21.15.60:0/2233574757 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 129294317 0 0) 0x7f78680b25d0 con 0x7f786805d070 2022-06-29T16:00:17.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f78677fe700 1 -- 172.21.15.60:0/2233574757 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f786805d450 con 0x7f786805d070 2022-06-29T16:00:17.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f78677fe700 1 -- 172.21.15.60:0/2233574757 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1468329764 0 0) 0x7f784c003260 con 0x7f78680b2130 2022-06-29T16:00:17.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f78677fe700 1 -- 172.21.15.60:0/2233574757 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78680aed00 msgr2=0x7f7868146810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:17.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f78677fe700 1 --2- 172.21.15.60:0/2233574757 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78680aed00 0x7f7868146810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:17.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f78677fe700 1 -- 172.21.15.60:0/2233574757 >> v1:172.21.15.92:6789/0 conn(0x7f786805d070 legacy=0x7f786814aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:17.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f78677fe700 1 -- 172.21.15.60:0/2233574757 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f786814a350 con 0x7f78680b2130 2022-06-29T16:00:17.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f78677fe700 1 -- 172.21.15.60:0/2233574757 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f785c001680 con 0x7f78680b2130 2022-06-29T16:00:17.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f78677fe700 1 -- 172.21.15.60:0/2233574757 >> v1:172.21.15.60:6789/0 conn(0x7f78680b2130 legacy=0x7f7868146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:17.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f78677fe700 1 --2- 172.21.15.60:0/2233574757 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78680aed00 0x7f784c003910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:17.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f78677fe700 1 --2- 172.21.15.60:0/2233574757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f784c003e20 0x7f784c004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:17.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f78677fe700 1 --2- 172.21.15.60:0/2233574757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f784c0049f0 0x7f784c004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:17.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f78677fe700 1 -- 172.21.15.60:0/2233574757 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78680adf90 con 0x7f784c0049f0 2022-06-29T16:00:17.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f78677fe700 1 -- 172.21.15.60:0/2233574757 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f784c00b800 con 0x7f784c003e20 2022-06-29T16:00:17.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f78677fe700 1 -- 172.21.15.60:0/2233574757 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f784c00b9c0 con 0x7f78680aed00 2022-06-29T16:00:17.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f786d2a5700 1 --2- 172.21.15.60:0/2233574757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f784c0049f0 0x7f784c004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:17.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f786caa4700 1 --2- 172.21.15.60:0/2233574757 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78680aed00 0x7f784c003910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:17.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f7867fff700 1 --2- 172.21.15.60:0/2233574757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f784c003e20 0x7f784c004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:17.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f786d2a5700 1 -- 172.21.15.60:0/2233574757 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78680aed00 msgr2=0x7f784c003910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:17.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f786d2a5700 1 --2- 172.21.15.60:0/2233574757 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78680aed00 0x7f784c003910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:17.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f786d2a5700 1 -- 172.21.15.60:0/2233574757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f784c003e20 msgr2=0x7f784c004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:17.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f786d2a5700 1 --2- 172.21.15.60:0/2233574757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f784c003e20 0x7f784c004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:17.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f786d2a5700 1 -- 172.21.15.60:0/2233574757 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f784c003260 con 0x7f784c0049f0 2022-06-29T16:00:17.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f786d2a5700 1 --2- 172.21.15.60:0/2233574757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f784c0049f0 0x7f784c004e70 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f786000c000 tx=0x7f78600091d0).ready entity=mon.0 client_cookie=f3f8463211b3acd7 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:17.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f78677fe700 1 -- 172.21.15.60:0/2233574757 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f786000e7e0 con 0x7f784c0049f0 2022-06-29T16:00:17.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f78677fe700 1 -- 172.21.15.60:0/2233574757 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f786000e940 con 0x7f784c0049f0 2022-06-29T16:00:17.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f78677fe700 1 -- 172.21.15.60:0/2233574757 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f786000eca0 con 0x7f784c0049f0 2022-06-29T16:00:17.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f786daa6700 1 -- 172.21.15.60:0/2233574757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f784c0049f0 msgr2=0x7f784c004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:17.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f786daa6700 1 --2- 172.21.15.60:0/2233574757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f784c0049f0 0x7f784c004e70 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f786000c000 tx=0x7f78600091d0).stop 2022-06-29T16:00:17.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f786d2a5700 1 -- 172.21.15.60:0/2233574757 reap_dead start 2022-06-29T16:00:17.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f786daa6700 1 -- 172.21.15.60:0/2233574757 shutdown_connections 2022-06-29T16:00:17.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f786daa6700 1 -- 172.21.15.60:0/2233574757 >> 172.21.15.60:0/2233574757 conn(0x7f78680b5890 msgr2=0x7f78680b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:17.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f786daa6700 1 -- 172.21.15.60:0/2233574757 shutdown_connections 2022-06-29T16:00:17.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f786daa6700 1 -- 172.21.15.60:0/2233574757 wait complete. 2022-06-29T16:00:17.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f786daa6700 1 Processor -- start 2022-06-29T16:00:17.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.279+0000 7f786daa6700 1 -- start start 2022-06-29T16:00:17.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.283+0000 7f786daa6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f784c0049f0 0x7f786814b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:17.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.283+0000 7f786daa6700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f786814a9b0 0x7f7868141420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:17.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.283+0000 7f786daa6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78681598a0 0x7f7868066470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:17.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.283+0000 7f786daa6700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786814c7f0 con 0x7f784c0049f0 2022-06-29T16:00:17.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.283+0000 7f786daa6700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7868149b80 con 0x7f786814a9b0 2022-06-29T16:00:17.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.283+0000 7f786daa6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f786814b480 con 0x7f78681598a0 2022-06-29T16:00:17.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.283+0000 7f786caa4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f784c0049f0 0x7f786814b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:17.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.283+0000 7f786caa4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f784c0049f0 0x7f786814b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46978/0 (socket says 172.21.15.60:46978) 2022-06-29T16:00:17.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.283+0000 7f786caa4700 1 -- 172.21.15.60:0/855702244 learned_addr learned my addr 172.21.15.60:0/855702244 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:17.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.283+0000 7f786d2a5700 1 --2- 172.21.15.60:0/855702244 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78681598a0 0x7f7868066470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:17.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.283+0000 7f7867fff700 1 --2- 172.21.15.60:0/855702244 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f786814a9b0 0x7f7868141420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:17.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.283+0000 7f786caa4700 1 -- 172.21.15.60:0/855702244 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78681598a0 msgr2=0x7f7868066470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:17.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.283+0000 7f786caa4700 1 --2- 172.21.15.60:0/855702244 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78681598a0 0x7f7868066470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:17.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.283+0000 7f786caa4700 1 -- 172.21.15.60:0/855702244 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f786814a9b0 msgr2=0x7f7868141420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:17.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.283+0000 7f786caa4700 1 --2- 172.21.15.60:0/855702244 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f786814a9b0 0x7f7868141420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:17.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.283+0000 7f786caa4700 1 -- 172.21.15.60:0/855702244 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7868066960 con 0x7f784c0049f0 2022-06-29T16:00:17.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.283+0000 7f786caa4700 1 --2- 172.21.15.60:0/855702244 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f784c0049f0 0x7f786814b790 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f785400b000 tx=0x7f7854009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:17.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.283+0000 7f7865ffb700 1 -- 172.21.15.60:0/855702244 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f78540155f0 con 0x7f784c0049f0 2022-06-29T16:00:17.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.283+0000 7f7865ffb700 1 -- 172.21.15.60:0/855702244 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7854010040 con 0x7f784c0049f0 2022-06-29T16:00:17.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.283+0000 7f786daa6700 1 -- 172.21.15.60:0/855702244 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7868066af0 con 0x7f784c0049f0 2022-06-29T16:00:17.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.283+0000 7f7865ffb700 1 -- 172.21.15.60:0/855702244 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7854015940 con 0x7f784c0049f0 2022-06-29T16:00:17.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.283+0000 7f786daa6700 1 -- 172.21.15.60:0/855702244 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7868063520 con 0x7f784c0049f0 2022-06-29T16:00:17.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.283+0000 7f786daa6700 1 -- 172.21.15.60:0/855702244 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f786814a5f0 con 0x7f784c0049f0 2022-06-29T16:00:17.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.283+0000 7f7865ffb700 1 -- 172.21.15.60:0/855702244 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7854015aa0 con 0x7f784c0049f0 2022-06-29T16:00:17.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.287+0000 7f7865ffb700 1 --2- 172.21.15.60:0/855702244 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7848053a10 0x7f7848055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:17.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.287+0000 7f7865ffb700 1 -- 172.21.15.60:0/855702244 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f785406bd90 con 0x7f784c0049f0 2022-06-29T16:00:17.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.287+0000 7f7867fff700 1 --2- 172.21.15.60:0/855702244 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7848053a10 0x7f7848055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:17.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.287+0000 7f7867fff700 1 --2- 172.21.15.60:0/855702244 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7848053a10 0x7f7848055ed0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f785c00d000 tx=0x7f785c007040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:17.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.291+0000 7f7865ffb700 1 -- 172.21.15.60:0/855702244 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7854042fe0 con 0x7f784c0049f0 2022-06-29T16:00:17.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.419+0000 7f786daa6700 1 -- 172.21.15.60:0/855702244 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f78680573d0 con 0x7f784c0049f0 2022-06-29T16:00:17.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.423+0000 7f7865ffb700 1 -- 172.21.15.60:0/855702244 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+13181 (secure 0 0 0) 0x7f7854046600 con 0x7f784c0049f0 2022-06-29T16:00:17.431 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:00:17.432 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":38,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:00:16.773130+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:00:17.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.423+0000 7f786daa6700 1 -- 172.21.15.60:0/855702244 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7848053a10 msgr2=0x7f7848055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:17.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.423+0000 7f786daa6700 1 --2- 172.21.15.60:0/855702244 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7848053a10 0x7f7848055ed0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f785c00d000 tx=0x7f785c007040).stop 2022-06-29T16:00:17.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.423+0000 7f786daa6700 1 -- 172.21.15.60:0/855702244 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f784c0049f0 msgr2=0x7f786814b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:17.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.423+0000 7f786daa6700 1 --2- 172.21.15.60:0/855702244 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f784c0049f0 0x7f786814b790 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f785400b000 tx=0x7f7854009040).stop 2022-06-29T16:00:17.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.427+0000 7f786daa6700 1 -- 172.21.15.60:0/855702244 shutdown_connections 2022-06-29T16:00:17.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.427+0000 7f786daa6700 1 --2- 172.21.15.60:0/855702244 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7848053a10 0x7f7848055ed0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:17.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.427+0000 7f786daa6700 1 --2- 172.21.15.60:0/855702244 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78681598a0 0x7f7868066470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:17.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.427+0000 7f786daa6700 1 --2- 172.21.15.60:0/855702244 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f786814a9b0 0x7f7868141420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:17.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.427+0000 7f786daa6700 1 --2- 172.21.15.60:0/855702244 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f784c0049f0 0x7f786814b790 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:17.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.427+0000 7f786daa6700 1 -- 172.21.15.60:0/855702244 >> 172.21.15.60:0/855702244 conn(0x7f78680b5890 msgr2=0x7f7868144f90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:17.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.427+0000 7f786daa6700 1 -- 172.21.15.60:0/855702244 shutdown_connections 2022-06-29T16:00:17.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.427+0000 7f786daa6700 1 -- 172.21.15.60:0/855702244 wait complete. 2022-06-29T16:00:17.451 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-06-29T16:00:17.452 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:00:17.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.551+0000 7f12ce322700 1 Processor -- start 2022-06-29T16:00:17.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.551+0000 7f12ce322700 1 -- start start 2022-06-29T16:00:17.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.551+0000 7f12ce322700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f12c81471a0 0x7f12c8147fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:17.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.551+0000 7f12ce322700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f12c80afb10 0x7f12c80b0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:17.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.551+0000 7f12ce322700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c80ac8f0 con 0x7f12c80aed60 2022-06-29T16:00:17.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.551+0000 7f12ce322700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12c80acb90 con 0x7f12c80afb10 2022-06-29T16:00:17.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.551+0000 7f12ce322700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12c80b02d0 con 0x7f12c81471a0 2022-06-29T16:00:17.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.551+0000 7f12c7fff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f12c80aed60 0x7f12c8147890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55072/0 (socket says 172.21.15.60:55072) 2022-06-29T16:00:17.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.551+0000 7f12c7fff700 1 -- 172.21.15.60:0/1992023171 learned_addr learned my addr 172.21.15.60:0/1992023171 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:00:17.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.551+0000 7f12cc8bf700 1 --2- 172.21.15.60:0/1992023171 >> v2:172.21.15.60:3300/0 conn(0x7f12c80afb10 0x7f12c80b0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:17.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.551+0000 7f12c77fe700 1 --2- 172.21.15.60:0/1992023171 >> v2:172.21.15.92:3300/0 conn(0x7f12c81471a0 0x7f12c8147fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:17.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.551+0000 7f12c6ffd700 1 -- 172.21.15.60:0/1992023171 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3655176794 0 0) 0x7f12c80ac8f0 con 0x7f12c80aed60 2022-06-29T16:00:17.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.551+0000 7f12c6ffd700 1 -- 172.21.15.60:0/1992023171 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12b4003260 con 0x7f12c80aed60 2022-06-29T16:00:17.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.551+0000 7f12cc8bf700 1 -- 172.21.15.60:0/1992023171 >> v2:172.21.15.92:3300/0 conn(0x7f12c81471a0 msgr2=0x7f12c8147fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:17.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.551+0000 7f12cc8bf700 1 --2- 172.21.15.60:0/1992023171 >> v2:172.21.15.92:3300/0 conn(0x7f12c81471a0 0x7f12c8147fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:17.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.555+0000 7f12cc8bf700 1 -- 172.21.15.60:0/1992023171 >> v1:172.21.15.60:6789/0 conn(0x7f12c80aed60 legacy=0x7f12c8147890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:17.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.555+0000 7f12cc8bf700 1 -- 172.21.15.60:0/1992023171 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c80ac5b0 con 0x7f12c80afb10 2022-06-29T16:00:17.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.555+0000 7f12cc8bf700 1 --2- 172.21.15.60:0/1992023171 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12c80afb10 0x7f12c80b0770 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f12c0002f70 tx=0x7f12c0006df0).ready entity=mon.0 client_cookie=f79137b1df7dff88 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:17.586 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.555+0000 7f12c6ffd700 1 -- 172.21.15.60:0/1992023171 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12c000d4f0 con 0x7f12c80afb10 2022-06-29T16:00:17.586 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.555+0000 7f12c6ffd700 1 -- 172.21.15.60:0/1992023171 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12c0000e90 con 0x7f12c80afb10 2022-06-29T16:00:17.587 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.555+0000 7f12c6ffd700 1 -- 172.21.15.60:0/1992023171 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12c000d840 con 0x7f12c80afb10 2022-06-29T16:00:17.587 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.555+0000 7f12ce322700 1 -- 172.21.15.60:0/1992023171 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12c80afb10 msgr2=0x7f12c80b0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:17.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.555+0000 7f12ce322700 1 --2- 172.21.15.60:0/1992023171 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12c80afb10 0x7f12c80b0770 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f12c0002f70 tx=0x7f12c0006df0).stop 2022-06-29T16:00:17.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.555+0000 7f12ce322700 1 -- 172.21.15.60:0/1992023171 shutdown_connections 2022-06-29T16:00:17.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.555+0000 7f12ce322700 1 --2- 172.21.15.60:0/1992023171 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12c80afb10 0x7f12c80b0770 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:17.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.555+0000 7f12ce322700 1 --2- 172.21.15.60:0/1992023171 >> v2:172.21.15.92:3300/0 conn(0x7f12c81471a0 0x7f12c8147fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:17.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.555+0000 7f12ce322700 1 -- 172.21.15.60:0/1992023171 >> 172.21.15.60:0/1992023171 conn(0x7f12c80b4f60 msgr2=0x7f12c80b5370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:17.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.555+0000 7f12ce322700 1 -- 172.21.15.60:0/1992023171 shutdown_connections 2022-06-29T16:00:17.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.555+0000 7f12ce322700 1 -- 172.21.15.60:0/1992023171 wait complete. 2022-06-29T16:00:17.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.555+0000 7f12ce322700 1 Processor -- start 2022-06-29T16:00:17.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.555+0000 7f12ce322700 1 -- start start 2022-06-29T16:00:17.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.559+0000 7f12ce322700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12c80aed60 0x7f12c80637f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:17.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.559+0000 7f12ce322700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12c80afb10 0x7f12c81598e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:17.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.559+0000 7f12ce322700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12c814c780 0x7f12c8142e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:17.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.559+0000 7f12ce322700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c8063e20 con 0x7f12c80aed60 2022-06-29T16:00:17.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.559+0000 7f12ce322700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c80640c0 con 0x7f12c80afb10 2022-06-29T16:00:17.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.559+0000 7f12ce322700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c80655f0 con 0x7f12c814c780 2022-06-29T16:00:17.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.559+0000 7f12c7fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12c80aed60 0x7f12c80637f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:17.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.559+0000 7f12c7fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12c80aed60 0x7f12c80637f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46992/0 (socket says 172.21.15.60:46992) 2022-06-29T16:00:17.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.559+0000 7f12c7fff700 1 -- 172.21.15.60:0/1356711938 learned_addr learned my addr 172.21.15.60:0/1356711938 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:17.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.559+0000 7f12c77fe700 1 --2- 172.21.15.60:0/1356711938 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12c80afb10 0x7f12c81598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:17.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.559+0000 7f12cc8bf700 1 --2- 172.21.15.60:0/1356711938 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12c814c780 0x7f12c8142e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:17.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.559+0000 7f12c7fff700 1 -- 172.21.15.60:0/1356711938 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12c814c780 msgr2=0x7f12c8142e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:17.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.559+0000 7f12c7fff700 1 --2- 172.21.15.60:0/1356711938 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12c814c780 0x7f12c8142e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:17.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.559+0000 7f12c7fff700 1 -- 172.21.15.60:0/1356711938 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12c80afb10 msgr2=0x7f12c81598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:17.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.559+0000 7f12c7fff700 1 --2- 172.21.15.60:0/1356711938 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12c80afb10 0x7f12c81598e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:17.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.559+0000 7f12c7fff700 1 -- 172.21.15.60:0/1356711938 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c000b180 con 0x7f12c80aed60 2022-06-29T16:00:17.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.559+0000 7f12c7fff700 1 --2- 172.21.15.60:0/1356711938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12c80aed60 0x7f12c80637f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f12bc001110 tx=0x7f12bc007aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:17.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.559+0000 7f12c57fa700 1 -- 172.21.15.60:0/1356711938 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12bc00ce80 con 0x7f12c80aed60 2022-06-29T16:00:17.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.559+0000 7f12c57fa700 1 -- 172.21.15.60:0/1356711938 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12bc0070d0 con 0x7f12c80aed60 2022-06-29T16:00:17.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.559+0000 7f12c57fa700 1 -- 172.21.15.60:0/1356711938 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12bc007430 con 0x7f12c80aed60 2022-06-29T16:00:17.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.559+0000 7f12ce322700 1 -- 172.21.15.60:0/1356711938 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c814a130 con 0x7f12c80aed60 2022-06-29T16:00:17.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.559+0000 7f12ce322700 1 -- 172.21.15.60:0/1356711938 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c814b510 con 0x7f12c80aed60 2022-06-29T16:00:17.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.559+0000 7f12ce322700 1 -- 172.21.15.60:0/1356711938 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c80657b0 con 0x7f12c80aed60 2022-06-29T16:00:17.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.563+0000 7f12c57fa700 1 -- 172.21.15.60:0/1356711938 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f12bc010070 con 0x7f12c80aed60 2022-06-29T16:00:17.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.563+0000 7f12c57fa700 1 --2- 172.21.15.60:0/1356711938 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f12a8053b30 0x7f12a8055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:17.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.563+0000 7f12c57fa700 1 -- 172.21.15.60:0/1356711938 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f12bc06b160 con 0x7f12c80aed60 2022-06-29T16:00:17.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.563+0000 7f12c77fe700 1 --2- 172.21.15.60:0/1356711938 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f12a8053b30 0x7f12a8055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:17.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.563+0000 7f12c77fe700 1 --2- 172.21.15.60:0/1356711938 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f12a8053b30 0x7f12a8055ff0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f12b800b000 tx=0x7f12b80091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:17.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.567+0000 7f12c57fa700 1 -- 172.21.15.60:0/1356711938 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f12bc0423b0 con 0x7f12c80aed60 2022-06-29T16:00:17.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.695+0000 7f12ce322700 1 -- 172.21.15.60:0/1356711938 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f12c80573d0 con 0x7f12c80aed60 2022-06-29T16:00:17.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.695+0000 7f12c57fa700 1 -- 172.21.15.60:0/1356711938 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v38) v1 ==== 110+0+10 (secure 0 0 0) 0x7f12bc018070 con 0x7f12c80aed60 2022-06-29T16:00:17.707 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:00:17.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.699+0000 7f12ce322700 1 -- 172.21.15.60:0/1356711938 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f12a8053b30 msgr2=0x7f12a8055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:17.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.699+0000 7f12ce322700 1 --2- 172.21.15.60:0/1356711938 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f12a8053b30 0x7f12a8055ff0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f12b800b000 tx=0x7f12b80091d0).stop 2022-06-29T16:00:17.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.699+0000 7f12ce322700 1 -- 172.21.15.60:0/1356711938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12c80aed60 msgr2=0x7f12c80637f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:17.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.699+0000 7f12ce322700 1 --2- 172.21.15.60:0/1356711938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12c80aed60 0x7f12c80637f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f12bc001110 tx=0x7f12bc007aa0).stop 2022-06-29T16:00:17.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.699+0000 7f12ce322700 1 -- 172.21.15.60:0/1356711938 shutdown_connections 2022-06-29T16:00:17.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.699+0000 7f12ce322700 1 --2- 172.21.15.60:0/1356711938 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f12a8053b30 0x7f12a8055ff0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:17.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.699+0000 7f12ce322700 1 --2- 172.21.15.60:0/1356711938 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f12c814c780 0x7f12c8142e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:17.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.699+0000 7f12ce322700 1 --2- 172.21.15.60:0/1356711938 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12c80afb10 0x7f12c81598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:17.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.699+0000 7f12ce322700 1 --2- 172.21.15.60:0/1356711938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12c80aed60 0x7f12c80637f0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:17.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.699+0000 7f12ce322700 1 -- 172.21.15.60:0/1356711938 >> 172.21.15.60:0/1356711938 conn(0x7f12c80b4f60 msgr2=0x7f12c80b2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:17.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.699+0000 7f12ce322700 1 -- 172.21.15.60:0/1356711938 shutdown_connections 2022-06-29T16:00:17.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.699+0000 7f12ce322700 1 -- 172.21.15.60:0/1356711938 wait complete. 2022-06-29T16:00:17.718 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:00:17.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.819+0000 7fb8364d8700 1 Processor -- start 2022-06-29T16:00:17.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.819+0000 7fb8364d8700 1 -- start start 2022-06-29T16:00:17.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.819+0000 7fb8364d8700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb83005fb00 0x7fb830146d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:17.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.819+0000 7fb8364d8700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8300affd0 0x7fb8300b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:17.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.819+0000 7fb8364d8700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8300aca20 con 0x7fb8300b0df0 2022-06-29T16:00:17.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.819+0000 7fb8364d8700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb83005fee0 con 0x7fb83005fb00 2022-06-29T16:00:17.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.819+0000 7fb8364d8700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8300b1470 con 0x7fb8300affd0 2022-06-29T16:00:17.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.819+0000 7fb8354d6700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb83005fb00 0x7fb830146d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:17.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.819+0000 7fb8354d6700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb83005fb00 0x7fb830146d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47000/0 (socket says 172.21.15.60:47000) 2022-06-29T16:00:17.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.819+0000 7fb8354d6700 1 -- 172.21.15.60:0/4231568582 learned_addr learned my addr 172.21.15.60:0/4231568582 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:17.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.819+0000 7fb835cd7700 1 --2- 172.21.15.60:0/4231568582 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8300affd0 0x7fb8300b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:17.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.819+0000 7fb8354d6700 1 -- 172.21.15.60:0/4231568582 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8300affd0 msgr2=0x7fb8300b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:17.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.819+0000 7fb8354d6700 1 --2- 172.21.15.60:0/4231568582 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8300affd0 0x7fb8300b2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:17.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.819+0000 7fb8354d6700 1 -- 172.21.15.60:0/4231568582 >> v1:172.21.15.92:6789/0 conn(0x7fb8300b0df0 legacy=0x7fb8300ae040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:17.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.819+0000 7fb8354d6700 1 -- 172.21.15.60:0/4231568582 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8300b1710 con 0x7fb83005fb00 2022-06-29T16:00:17.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.819+0000 7fb8354d6700 1 --2- 172.21.15.60:0/4231568582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb83005fb00 0x7fb830146d10 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fb82c002f70 tx=0x7fb82c006eb0).ready entity=mon.0 client_cookie=16fa11621b2e858c server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:17.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.819+0000 7fb81ffff700 1 -- 172.21.15.60:0/4231568582 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb82c00d5c0 con 0x7fb83005fb00 2022-06-29T16:00:17.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.819+0000 7fb81ffff700 1 -- 172.21.15.60:0/4231568582 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb82c000e90 con 0x7fb83005fb00 2022-06-29T16:00:17.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.819+0000 7fb81ffff700 1 -- 172.21.15.60:0/4231568582 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb82c00d950 con 0x7fb83005fb00 2022-06-29T16:00:17.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.819+0000 7fb8364d8700 1 -- 172.21.15.60:0/4231568582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb83005fb00 msgr2=0x7fb830146d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:17.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.819+0000 7fb8364d8700 1 --2- 172.21.15.60:0/4231568582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb83005fb00 0x7fb830146d10 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fb82c002f70 tx=0x7fb82c006eb0).stop 2022-06-29T16:00:17.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.819+0000 7fb8364d8700 1 -- 172.21.15.60:0/4231568582 shutdown_connections 2022-06-29T16:00:17.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.819+0000 7fb8364d8700 1 --2- 172.21.15.60:0/4231568582 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8300affd0 0x7fb8300b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:17.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.819+0000 7fb8364d8700 1 --2- 172.21.15.60:0/4231568582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb83005fb00 0x7fb830146d10 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:17.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.819+0000 7fb8364d8700 1 -- 172.21.15.60:0/4231568582 >> 172.21.15.60:0/4231568582 conn(0x7fb830063a60 msgr2=0x7fb83005f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:17.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.819+0000 7fb8364d8700 1 -- 172.21.15.60:0/4231568582 shutdown_connections 2022-06-29T16:00:17.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.819+0000 7fb8364d8700 1 -- 172.21.15.60:0/4231568582 wait complete. 2022-06-29T16:00:17.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.823+0000 7fb8364d8700 1 Processor -- start 2022-06-29T16:00:17.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.823+0000 7fb8364d8700 1 -- start start 2022-06-29T16:00:17.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.823+0000 7fb8364d8700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb83005fb00 0x7fb8300aeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:17.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.823+0000 7fb8364d8700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8300affd0 0x7fb830151f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:17.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.823+0000 7fb8364d8700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8300b0df0 0x7fb830144220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:17.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.823+0000 7fb8364d8700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8301480a0 con 0x7fb83005fb00 2022-06-29T16:00:17.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.823+0000 7fb8364d8700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb830147980 con 0x7fb8300b0df0 2022-06-29T16:00:17.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.823+0000 7fb8364d8700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8301562f0 con 0x7fb8300affd0 2022-06-29T16:00:17.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.823+0000 7fb8354d6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb83005fb00 0x7fb8300aeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:17.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.823+0000 7fb8354d6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb83005fb00 0x7fb8300aeaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47006/0 (socket says 172.21.15.60:47006) 2022-06-29T16:00:17.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.823+0000 7fb8354d6700 1 -- 172.21.15.60:0/1080726786 learned_addr learned my addr 172.21.15.60:0/1080726786 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:17.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.823+0000 7fb834cd5700 1 --2- 172.21.15.60:0/1080726786 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8300affd0 0x7fb830151f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:17.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.823+0000 7fb835cd7700 1 --2- 172.21.15.60:0/1080726786 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8300b0df0 0x7fb830144220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:17.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.823+0000 7fb8354d6700 1 -- 172.21.15.60:0/1080726786 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8300affd0 msgr2=0x7fb830151f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:17.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.823+0000 7fb8354d6700 1 --2- 172.21.15.60:0/1080726786 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8300affd0 0x7fb830151f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:17.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.823+0000 7fb8354d6700 1 -- 172.21.15.60:0/1080726786 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8300b0df0 msgr2=0x7fb830144220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:17.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.823+0000 7fb8354d6700 1 --2- 172.21.15.60:0/1080726786 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8300b0df0 0x7fb830144220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:17.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.823+0000 7fb8354d6700 1 -- 172.21.15.60:0/1080726786 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb82c00b1e0 con 0x7fb83005fb00 2022-06-29T16:00:17.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.823+0000 7fb835cd7700 1 --2- 172.21.15.60:0/1080726786 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8300b0df0 0x7fb830144220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:00:17.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.827+0000 7fb8354d6700 1 --2- 172.21.15.60:0/1080726786 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb83005fb00 0x7fb8300aeaf0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fb82c002f70 tx=0x7fb82c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:17.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.827+0000 7fb81e7fc700 1 -- 172.21.15.60:0/1080726786 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb82c011860 con 0x7fb83005fb00 2022-06-29T16:00:17.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.827+0000 7fb81e7fc700 1 -- 172.21.15.60:0/1080726786 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb82c0119c0 con 0x7fb83005fb00 2022-06-29T16:00:17.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.827+0000 7fb8364d8700 1 -- 172.21.15.60:0/1080726786 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb830147c00 con 0x7fb83005fb00 2022-06-29T16:00:17.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.827+0000 7fb81e7fc700 1 -- 172.21.15.60:0/1080726786 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb82c011d20 con 0x7fb83005fb00 2022-06-29T16:00:17.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.827+0000 7fb8364d8700 1 -- 172.21.15.60:0/1080726786 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb830067610 con 0x7fb83005fb00 2022-06-29T16:00:17.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.827+0000 7fb8364d8700 1 -- 172.21.15.60:0/1080726786 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb830147d90 con 0x7fb83005fb00 2022-06-29T16:00:17.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.827+0000 7fb81e7fc700 1 -- 172.21.15.60:0/1080726786 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb82c025030 con 0x7fb83005fb00 2022-06-29T16:00:17.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.827+0000 7fb81e7fc700 1 --2- 172.21.15.60:0/1080726786 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb810053b30 0x7fb810055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:17.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.827+0000 7fb81e7fc700 1 -- 172.21.15.60:0/1080726786 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fb82c06cf00 con 0x7fb83005fb00 2022-06-29T16:00:17.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.827+0000 7fb834cd5700 1 --2- 172.21.15.60:0/1080726786 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb810053b30 0x7fb810055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:17.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.831+0000 7fb834cd5700 1 --2- 172.21.15.60:0/1080726786 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb810053b30 0x7fb810055ff0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fb82000d000 tx=0x7fb820007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:17.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.835+0000 7fb81e7fc700 1 -- 172.21.15.60:0/1080726786 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb82c044150 con 0x7fb83005fb00 2022-06-29T16:00:17.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.955+0000 7fb8364d8700 1 -- 172.21.15.60:0/1080726786 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fb8301560e0 con 0x7fb83005fb00 2022-06-29T16:00:17.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.959+0000 7fb81e7fc700 1 -- 172.21.15.60:0/1080726786 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v38) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb82c0230a0 con 0x7fb83005fb00 2022-06-29T16:00:17.967 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:00:17.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.959+0000 7fb8364d8700 1 -- 172.21.15.60:0/1080726786 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb810053b30 msgr2=0x7fb810055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:17.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.959+0000 7fb8364d8700 1 --2- 172.21.15.60:0/1080726786 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb810053b30 0x7fb810055ff0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fb82000d000 tx=0x7fb820007b90).stop 2022-06-29T16:00:17.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.959+0000 7fb8364d8700 1 -- 172.21.15.60:0/1080726786 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb83005fb00 msgr2=0x7fb8300aeaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:17.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.959+0000 7fb8364d8700 1 --2- 172.21.15.60:0/1080726786 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb83005fb00 0x7fb8300aeaf0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fb82c002f70 tx=0x7fb82c015000).stop 2022-06-29T16:00:17.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.959+0000 7fb8364d8700 1 -- 172.21.15.60:0/1080726786 shutdown_connections 2022-06-29T16:00:17.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.959+0000 7fb8364d8700 1 --2- 172.21.15.60:0/1080726786 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb810053b30 0x7fb810055ff0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:17.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.959+0000 7fb8364d8700 1 --2- 172.21.15.60:0/1080726786 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8300b0df0 0x7fb830144220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:17.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.959+0000 7fb8364d8700 1 --2- 172.21.15.60:0/1080726786 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8300affd0 0x7fb830151f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:17.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.959+0000 7fb8364d8700 1 --2- 172.21.15.60:0/1080726786 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb83005fb00 0x7fb8300aeaf0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:17.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.959+0000 7fb8364d8700 1 -- 172.21.15.60:0/1080726786 >> 172.21.15.60:0/1080726786 conn(0x7fb830063a60 msgr2=0x7fb8300b4560 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:17.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.959+0000 7fb8364d8700 1 -- 172.21.15.60:0/1080726786 shutdown_connections 2022-06-29T16:00:17.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:17.959+0000 7fb8364d8700 1 -- 172.21.15.60:0/1080726786 wait complete. 2022-06-29T16:00:17.981 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-06-29T16:00:18.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.079+0000 7f9748834700 1 Processor -- start 2022-06-29T16:00:18.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.079+0000 7f9748834700 1 -- start start 2022-06-29T16:00:18.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.079+0000 7f9748834700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f974405d070 0x7f974414aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:18.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.079+0000 7f9748834700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f974414af80 0x7f97440ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:18.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.079+0000 7f9748834700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f974405d450 con 0x7f97440b2130 2022-06-29T16:00:18.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.079+0000 7f9748834700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97440adf90 con 0x7f974405d070 2022-06-29T16:00:18.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.079+0000 7f9748834700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97440b25d0 con 0x7f974414af80 2022-06-29T16:00:18.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.079+0000 7f974259c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f97440b2130 0x7f97440aca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55100/0 (socket says 172.21.15.60:55100) 2022-06-29T16:00:18.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.079+0000 7f974259c700 1 -- 172.21.15.60:0/2150822477 learned_addr learned my addr 172.21.15.60:0/2150822477 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:00:18.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.079+0000 7f974359e700 1 --2- 172.21.15.60:0/2150822477 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f974414af80 0x7f97440ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:18.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.079+0000 7f9742d9d700 1 --2- 172.21.15.60:0/2150822477 >> v2:172.21.15.92:3300/0 conn(0x7f974405d070 0x7f974414aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:18.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.079+0000 7f9741d9b700 1 -- 172.21.15.60:0/2150822477 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2062262553 0 0) 0x7f974405d450 con 0x7f97440b2130 2022-06-29T16:00:18.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.079+0000 7f9741d9b700 1 -- 172.21.15.60:0/2150822477 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9728003260 con 0x7f97440b2130 2022-06-29T16:00:18.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.079+0000 7f9741d9b700 1 -- 172.21.15.60:0/2150822477 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 700586211 0 0) 0x7f9728003260 con 0x7f97440b2130 2022-06-29T16:00:18.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.079+0000 7f974359e700 1 -- 172.21.15.60:0/2150822477 >> v2:172.21.15.92:3300/0 conn(0x7f974405d070 msgr2=0x7f974414aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:18.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.079+0000 7f974359e700 1 --2- 172.21.15.60:0/2150822477 >> v2:172.21.15.92:3300/0 conn(0x7f974405d070 0x7f974414aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:18.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.079+0000 7f974359e700 1 -- 172.21.15.60:0/2150822477 >> v1:172.21.15.60:6789/0 conn(0x7f97440b2130 legacy=0x7f97440aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:18.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.079+0000 7f974359e700 1 -- 172.21.15.60:0/2150822477 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f974414a350 con 0x7f974414af80 2022-06-29T16:00:18.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.079+0000 7f974359e700 1 --2- 172.21.15.60:0/2150822477 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f974414af80 0x7f97440ad130 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f9734001b80 tx=0x7f973c006df0).ready entity=mon.2 client_cookie=896490912d0e5563 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:18.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.079+0000 7f9741d9b700 1 -- 172.21.15.60:0/2150822477 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f97340019e0 con 0x7f974414af80 2022-06-29T16:00:18.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.079+0000 7f9741d9b700 1 -- 172.21.15.60:0/2150822477 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f973c000e90 con 0x7f974414af80 2022-06-29T16:00:18.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.079+0000 7f9741d9b700 1 -- 172.21.15.60:0/2150822477 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f973c012460 con 0x7f974414af80 2022-06-29T16:00:18.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.083+0000 7f9748834700 1 -- 172.21.15.60:0/2150822477 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f974414af80 msgr2=0x7f97440ad130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:18.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.083+0000 7f9748834700 1 --2- 172.21.15.60:0/2150822477 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f974414af80 0x7f97440ad130 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f9734001b80 tx=0x7f973c006df0).stop 2022-06-29T16:00:18.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.083+0000 7f9748834700 1 -- 172.21.15.60:0/2150822477 shutdown_connections 2022-06-29T16:00:18.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.083+0000 7f9748834700 1 --2- 172.21.15.60:0/2150822477 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f974414af80 0x7f97440ad130 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:18.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.083+0000 7f9748834700 1 --2- 172.21.15.60:0/2150822477 >> v2:172.21.15.92:3300/0 conn(0x7f974405d070 0x7f974414aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:18.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.083+0000 7f9748834700 1 -- 172.21.15.60:0/2150822477 >> 172.21.15.60:0/2150822477 conn(0x7f97440b5890 msgr2=0x7f97440b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:18.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.083+0000 7f9748834700 1 -- 172.21.15.60:0/2150822477 shutdown_connections 2022-06-29T16:00:18.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.083+0000 7f9748834700 1 -- 172.21.15.60:0/2150822477 wait complete. 2022-06-29T16:00:18.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.083+0000 7f9748834700 1 Processor -- start 2022-06-29T16:00:18.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.083+0000 7f9748834700 1 -- start start 2022-06-29T16:00:18.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.087+0000 7f9748834700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f974405d070 0x7f97440b3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:18.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.087+0000 7f9748834700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97440b2130 0x7f9744063130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:18.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.087+0000 7f9748834700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f974414af80 0x7f9744148240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:18.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.087+0000 7f9748834700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9744063c50 con 0x7f97440b2130 2022-06-29T16:00:18.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.087+0000 7f9748834700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9744063ef0 con 0x7f974405d070 2022-06-29T16:00:18.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.087+0000 7f9748834700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9744148730 con 0x7f974414af80 2022-06-29T16:00:18.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.087+0000 7f974259c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97440b2130 0x7f9744063130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:18.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.087+0000 7f974259c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97440b2130 0x7f9744063130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47022/0 (socket says 172.21.15.60:47022) 2022-06-29T16:00:18.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.087+0000 7f974259c700 1 -- 172.21.15.60:0/443871558 learned_addr learned my addr 172.21.15.60:0/443871558 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:18.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.087+0000 7f9742d9d700 1 --2- 172.21.15.60:0/443871558 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f974405d070 0x7f97440b3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:18.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.087+0000 7f974359e700 1 --2- 172.21.15.60:0/443871558 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f974414af80 0x7f9744148240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:18.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.087+0000 7f974259c700 1 -- 172.21.15.60:0/443871558 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f974414af80 msgr2=0x7f9744148240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:18.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.087+0000 7f974259c700 1 --2- 172.21.15.60:0/443871558 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f974414af80 0x7f9744148240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:18.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.087+0000 7f974259c700 1 -- 172.21.15.60:0/443871558 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f974405d070 msgr2=0x7f97440b3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:18.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.087+0000 7f974259c700 1 --2- 172.21.15.60:0/443871558 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f974405d070 0x7f97440b3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:18.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.087+0000 7f974259c700 1 -- 172.21.15.60:0/443871558 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f973c00b180 con 0x7f97440b2130 2022-06-29T16:00:18.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.087+0000 7f9742d9d700 1 --2- 172.21.15.60:0/443871558 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f974405d070 0x7f97440b3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:00:18.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.087+0000 7f974259c700 1 --2- 172.21.15.60:0/443871558 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97440b2130 0x7f9744063130 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f9734001c40 tx=0x7f97340015b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:18.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.087+0000 7f9733fff700 1 -- 172.21.15.60:0/443871558 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9734004ab0 con 0x7f97440b2130 2022-06-29T16:00:18.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.087+0000 7f9733fff700 1 -- 172.21.15.60:0/443871558 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9734004c10 con 0x7f97440b2130 2022-06-29T16:00:18.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.087+0000 7f9733fff700 1 -- 172.21.15.60:0/443871558 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9734007170 con 0x7f97440b2130 2022-06-29T16:00:18.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.087+0000 7f9748834700 1 -- 172.21.15.60:0/443871558 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9744149a50 con 0x7f97440b2130 2022-06-29T16:00:18.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.087+0000 7f9748834700 1 -- 172.21.15.60:0/443871558 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9744066660 con 0x7f97440b2130 2022-06-29T16:00:18.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.087+0000 7f9748834700 1 -- 172.21.15.60:0/443871558 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f974414a5f0 con 0x7f97440b2130 2022-06-29T16:00:18.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.087+0000 7f9733fff700 1 -- 172.21.15.60:0/443871558 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9734010070 con 0x7f97440b2130 2022-06-29T16:00:18.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.091+0000 7f9733fff700 1 --2- 172.21.15.60:0/443871558 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9720053a10 0x7f9720055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:18.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.091+0000 7f9733fff700 1 -- 172.21.15.60:0/443871558 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f973406b7c0 con 0x7f97440b2130 2022-06-29T16:00:18.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.091+0000 7f9742d9d700 1 --2- 172.21.15.60:0/443871558 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9720053a10 0x7f9720055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:18.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.091+0000 7f9742d9d700 1 --2- 172.21.15.60:0/443871558 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9720053a10 0x7f9720055ed0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f973800b000 tx=0x7f9738008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:18.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.095+0000 7f9733fff700 1 -- 172.21.15.60:0/443871558 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9734042a10 con 0x7f97440b2130 2022-06-29T16:00:18.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.219+0000 7f9748834700 1 -- 172.21.15.60:0/443871558 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f97440640b0 con 0x7f97440b2130 2022-06-29T16:00:18.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.219+0000 7f9733fff700 1 -- 172.21.15.60:0/443871558 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v38) v1 ==== 84+0+143 (secure 0 0 0) 0x7f9734046030 con 0x7f97440b2130 2022-06-29T16:00:18.230 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:00:18.230 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2022-06-29T16:00:18.231 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:18.231 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2022-06-29T16:00:18.231 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": "device_health_metrics" 2022-06-29T16:00:18.232 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:18.232 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:00:18.233 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 2, 2022-06-29T16:00:18.233 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": "rbd" 2022-06-29T16:00:18.234 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:18.234 INFO:teuthology.orchestra.run.smithi060.stdout:] 2022-06-29T16:00:18.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.223+0000 7f9748834700 1 -- 172.21.15.60:0/443871558 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9720053a10 msgr2=0x7f9720055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:18.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.223+0000 7f9748834700 1 --2- 172.21.15.60:0/443871558 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9720053a10 0x7f9720055ed0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f973800b000 tx=0x7f9738008040).stop 2022-06-29T16:00:18.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.223+0000 7f9748834700 1 -- 172.21.15.60:0/443871558 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97440b2130 msgr2=0x7f9744063130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:18.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.223+0000 7f9748834700 1 --2- 172.21.15.60:0/443871558 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97440b2130 0x7f9744063130 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f9734001c40 tx=0x7f97340015b0).stop 2022-06-29T16:00:18.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.223+0000 7f9748834700 1 -- 172.21.15.60:0/443871558 shutdown_connections 2022-06-29T16:00:18.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.223+0000 7f9748834700 1 --2- 172.21.15.60:0/443871558 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9720053a10 0x7f9720055ed0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:18.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.223+0000 7f9748834700 1 --2- 172.21.15.60:0/443871558 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f974414af80 0x7f9744148240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:18.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.223+0000 7f9748834700 1 --2- 172.21.15.60:0/443871558 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97440b2130 0x7f9744063130 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:18.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.223+0000 7f9748834700 1 --2- 172.21.15.60:0/443871558 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f974405d070 0x7f97440b3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:18.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.223+0000 7f9748834700 1 -- 172.21.15.60:0/443871558 >> 172.21.15.60:0/443871558 conn(0x7f97440b5890 msgr2=0x7f97441471c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:18.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.223+0000 7f9748834700 1 -- 172.21.15.60:0/443871558 shutdown_connections 2022-06-29T16:00:18.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.223+0000 7f9748834700 1 -- 172.21.15.60:0/443871558 wait complete. 2022-06-29T16:00:18.242 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-06-29T16:00:18.242 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-06-29T16:00:18.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.343+0000 7f0e97fff700 1 Processor -- start 2022-06-29T16:00:18.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.343+0000 7f0e97fff700 1 -- start start 2022-06-29T16:00:18.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.347+0000 7f0e97fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0e98060610 0x7f0e9805c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:18.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.347+0000 7f0e97fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e9805fee0 0x7f0e98142a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:18.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.347+0000 7f0e97fff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e980b0a40 con 0x7f0e9805c5b0 2022-06-29T16:00:18.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.347+0000 7f0e97fff700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e980b0850 con 0x7f0e98060610 2022-06-29T16:00:18.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.347+0000 7f0e97fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e980b0ce0 con 0x7f0e9805fee0 2022-06-29T16:00:18.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.347+0000 7f0e967fc700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0e98060610 0x7f0e9805c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:18.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.347+0000 7f0e96ffd700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f0e9805c5b0 0x7f0e9813c740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55114/0 (socket says 172.21.15.60:55114) 2022-06-29T16:00:18.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.347+0000 7f0e96ffd700 1 -- 172.21.15.60:0/630758079 learned_addr learned my addr 172.21.15.60:0/630758079 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:00:18.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.347+0000 7f0e977fe700 1 --2- 172.21.15.60:0/630758079 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e9805fee0 0x7f0e98142a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:18.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.347+0000 7f0e967fc700 1 -- 172.21.15.60:0/630758079 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e9805fee0 msgr2=0x7f0e98142a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:18.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.347+0000 7f0e967fc700 1 --2- 172.21.15.60:0/630758079 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e9805fee0 0x7f0e98142a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:18.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.347+0000 7f0e967fc700 1 -- 172.21.15.60:0/630758079 >> v1:172.21.15.60:6789/0 conn(0x7f0e9805c5b0 legacy=0x7f0e9813c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:18.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.347+0000 7f0e967fc700 1 -- 172.21.15.60:0/630758079 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e9813ce50 con 0x7f0e98060610 2022-06-29T16:00:18.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.347+0000 7f0e967fc700 1 --2- 172.21.15.60:0/630758079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e98060610 0x7f0e9805c990 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f0e8c002f70 tx=0x7f0e8c006df0).ready entity=mon.0 client_cookie=aa8c1e7d001d8343 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:18.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.347+0000 7f0e95ffb700 1 -- 172.21.15.60:0/630758079 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e8c00d570 con 0x7f0e98060610 2022-06-29T16:00:18.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.347+0000 7f0e95ffb700 1 -- 172.21.15.60:0/630758079 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e8c000e90 con 0x7f0e98060610 2022-06-29T16:00:18.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.347+0000 7f0e95ffb700 1 -- 172.21.15.60:0/630758079 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e8c00d8c0 con 0x7f0e98060610 2022-06-29T16:00:18.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.347+0000 7f0e97fff700 1 -- 172.21.15.60:0/630758079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e98060610 msgr2=0x7f0e9805c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:18.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.347+0000 7f0e97fff700 1 --2- 172.21.15.60:0/630758079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e98060610 0x7f0e9805c990 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f0e8c002f70 tx=0x7f0e8c006df0).stop 2022-06-29T16:00:18.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.347+0000 7f0e97fff700 1 -- 172.21.15.60:0/630758079 shutdown_connections 2022-06-29T16:00:18.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.347+0000 7f0e97fff700 1 --2- 172.21.15.60:0/630758079 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e9805fee0 0x7f0e98142a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:18.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.347+0000 7f0e97fff700 1 --2- 172.21.15.60:0/630758079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e98060610 0x7f0e9805c990 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:18.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.347+0000 7f0e97fff700 1 -- 172.21.15.60:0/630758079 >> 172.21.15.60:0/630758079 conn(0x7f0e98061c80 msgr2=0x7f0e980637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:18.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.347+0000 7f0e97fff700 1 -- 172.21.15.60:0/630758079 shutdown_connections 2022-06-29T16:00:18.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.347+0000 7f0e97fff700 1 -- 172.21.15.60:0/630758079 wait complete. 2022-06-29T16:00:18.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.351+0000 7f0e97fff700 1 Processor -- start 2022-06-29T16:00:18.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.351+0000 7f0e97fff700 1 -- start start 2022-06-29T16:00:18.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.351+0000 7f0e97fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e9805c5b0 0x7f0e98152a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:18.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.351+0000 7f0e97fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e9805fee0 0x7f0e98144640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:18.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.351+0000 7f0e97fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e98142300 0x7f0e98146510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:18.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.351+0000 7f0e97fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e98152650 con 0x7f0e9805c5b0 2022-06-29T16:00:18.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.351+0000 7f0e97fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e98144f60 con 0x7f0e98142300 2022-06-29T16:00:18.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.351+0000 7f0e97fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e981405d0 con 0x7f0e9805fee0 2022-06-29T16:00:18.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.351+0000 7f0e96ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e9805c5b0 0x7f0e98152a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:18.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.351+0000 7f0e96ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e9805c5b0 0x7f0e98152a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47034/0 (socket says 172.21.15.60:47034) 2022-06-29T16:00:18.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.351+0000 7f0e96ffd700 1 -- 172.21.15.60:0/4072841082 learned_addr learned my addr 172.21.15.60:0/4072841082 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:18.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.351+0000 7f0e977fe700 1 --2- 172.21.15.60:0/4072841082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e98142300 0x7f0e98146510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:18.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.351+0000 7f0e967fc700 1 --2- 172.21.15.60:0/4072841082 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e9805fee0 0x7f0e98144640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:18.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.351+0000 7f0e96ffd700 1 -- 172.21.15.60:0/4072841082 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e9805fee0 msgr2=0x7f0e98144640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:18.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.351+0000 7f0e96ffd700 1 --2- 172.21.15.60:0/4072841082 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e9805fee0 0x7f0e98144640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:18.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.351+0000 7f0e96ffd700 1 -- 172.21.15.60:0/4072841082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e98142300 msgr2=0x7f0e98146510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:18.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.351+0000 7f0e96ffd700 1 --2- 172.21.15.60:0/4072841082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e98142300 0x7f0e98146510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:18.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.351+0000 7f0e96ffd700 1 -- 172.21.15.60:0/4072841082 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e8c00b180 con 0x7f0e9805c5b0 2022-06-29T16:00:18.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.351+0000 7f0e96ffd700 1 --2- 172.21.15.60:0/4072841082 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e9805c5b0 0x7f0e98152a40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f0e980b6fb0 tx=0x7f0e88007c40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:18.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.351+0000 7f0e87fff700 1 -- 172.21.15.60:0/4072841082 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e880070d0 con 0x7f0e9805c5b0 2022-06-29T16:00:18.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.351+0000 7f0e87fff700 1 -- 172.21.15.60:0/4072841082 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e88007230 con 0x7f0e9805c5b0 2022-06-29T16:00:18.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.351+0000 7f0e97fff700 1 -- 172.21.15.60:0/4072841082 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e98144b30 con 0x7f0e9805c5b0 2022-06-29T16:00:18.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.351+0000 7f0e87fff700 1 -- 172.21.15.60:0/4072841082 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e88005610 con 0x7f0e9805c5b0 2022-06-29T16:00:18.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.351+0000 7f0e97fff700 1 -- 172.21.15.60:0/4072841082 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e980af430 con 0x7f0e9805c5b0 2022-06-29T16:00:18.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.355+0000 7f0e97fff700 1 -- 172.21.15.60:0/4072841082 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e981450d0 con 0x7f0e9805c5b0 2022-06-29T16:00:18.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.355+0000 7f0e87fff700 1 -- 172.21.15.60:0/4072841082 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0e88010070 con 0x7f0e9805c5b0 2022-06-29T16:00:18.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.355+0000 7f0e87fff700 1 --2- 172.21.15.60:0/4072841082 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0e74053b00 0x7f0e74055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:18.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.355+0000 7f0e87fff700 1 -- 172.21.15.60:0/4072841082 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f0e8806b0a0 con 0x7f0e9805c5b0 2022-06-29T16:00:18.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.355+0000 7f0e967fc700 1 --2- 172.21.15.60:0/4072841082 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0e74053b00 0x7f0e74055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:18.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.355+0000 7f0e967fc700 1 --2- 172.21.15.60:0/4072841082 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0e74053b00 0x7f0e74055fc0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f0e8c018000 tx=0x7f0e8c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:18.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.363+0000 7f0e87fff700 1 -- 172.21.15.60:0/4072841082 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0e981450d0 con 0x7f0e9805c5b0 2022-06-29T16:00:18.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:18.495+0000 7f0e97fff700 1 -- 172.21.15.60:0/4072841082 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f0e981450d0 con 0x7f0e9805c5b0 2022-06-29T16:00:19.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.359+0000 7f0e87fff700 1 -- 172.21.15.60:0/4072841082 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v39) v1 ==== 133+0+0 (secure 0 0 0) 0x7f0e981450d0 con 0x7f0e9805c5b0 2022-06-29T16:00:19.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.363+0000 7f0e97fff700 1 -- 172.21.15.60:0/4072841082 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0e74053b00 msgr2=0x7f0e74055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:19.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.363+0000 7f0e97fff700 1 --2- 172.21.15.60:0/4072841082 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0e74053b00 0x7f0e74055fc0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f0e8c018000 tx=0x7f0e8c014000).stop 2022-06-29T16:00:19.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.363+0000 7f0e97fff700 1 -- 172.21.15.60:0/4072841082 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e9805c5b0 msgr2=0x7f0e98152a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:19.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.363+0000 7f0e97fff700 1 --2- 172.21.15.60:0/4072841082 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e9805c5b0 0x7f0e98152a40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f0e980b6fb0 tx=0x7f0e88007c40).stop 2022-06-29T16:00:19.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.363+0000 7f0e97fff700 1 -- 172.21.15.60:0/4072841082 shutdown_connections 2022-06-29T16:00:19.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.363+0000 7f0e97fff700 1 --2- 172.21.15.60:0/4072841082 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0e74053b00 0x7f0e74055fc0 secure :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0x7f0e8c018000 tx=0x7f0e8c014000).stop 2022-06-29T16:00:19.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.363+0000 7f0e97fff700 1 --2- 172.21.15.60:0/4072841082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e98142300 0x7f0e98146510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:19.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.363+0000 7f0e97fff700 1 --2- 172.21.15.60:0/4072841082 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e9805fee0 0x7f0e98144640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:19.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.363+0000 7f0e97fff700 1 --2- 172.21.15.60:0/4072841082 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e9805c5b0 0x7f0e98152a40 secure :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0x7f0e980b6fb0 tx=0x7f0e88007c40).stop 2022-06-29T16:00:19.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.363+0000 7f0e97fff700 1 -- 172.21.15.60:0/4072841082 >> 172.21.15.60:0/4072841082 conn(0x7f0e98061c80 msgr2=0x7f0e980af9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:19.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.367+0000 7f0e97fff700 1 -- 172.21.15.60:0/4072841082 shutdown_connections 2022-06-29T16:00:19.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.367+0000 7f0e97fff700 1 -- 172.21.15.60:0/4072841082 wait complete. 2022-06-29T16:00:19.378 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2022-06-29T16:00:19.388 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-06-29T16:00:19.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.487+0000 7f0bf9796700 1 Processor -- start 2022-06-29T16:00:19.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.487+0000 7f0bf9796700 1 -- start start 2022-06-29T16:00:19.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.487+0000 7f0bf9796700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0bf405d070 0x7f0bf414aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:19.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.487+0000 7f0bf9796700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bf414af80 0x7f0bf40ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:19.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.487+0000 7f0bf9796700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bf405d450 con 0x7f0bf40b2130 2022-06-29T16:00:19.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.487+0000 7f0bf9796700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bf40adf90 con 0x7f0bf405d070 2022-06-29T16:00:19.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.487+0000 7f0bf9796700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf40b25d0 con 0x7f0bf414af80 2022-06-29T16:00:19.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.487+0000 7f0bf3fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0bf405d070 0x7f0bf414aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:19.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.487+0000 7f0bf8f95700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bf414af80 0x7f0bf40ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:19.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.487+0000 7f0bf8f95700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bf414af80 0x7f0bf40ad130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:60140/0 (socket says 172.21.15.60:60140) 2022-06-29T16:00:19.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.487+0000 7f0bf3fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0bf405d070 0x7f0bf414aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:53072/0 (socket says 172.21.15.60:53072) 2022-06-29T16:00:19.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.487+0000 7f0bf37fe700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f0bf40b2130 0x7f0bf40aca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:59378/0 (socket says 172.21.15.60:59378) 2022-06-29T16:00:19.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.487+0000 7f0bf8f95700 1 -- 172.21.15.60:0/2547405114 learned_addr learned my addr 172.21.15.60:0/2547405114 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:19.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.491+0000 7f0bf8f95700 1 -- 172.21.15.60:0/2547405114 >> v2:172.21.15.92:3300/0 conn(0x7f0bf405d070 msgr2=0x7f0bf414aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:19.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.491+0000 7f0bf8f95700 1 --2- 172.21.15.60:0/2547405114 >> v2:172.21.15.92:3300/0 conn(0x7f0bf405d070 0x7f0bf414aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:19.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.491+0000 7f0bf8f95700 1 -- 172.21.15.60:0/2547405114 >> v1:172.21.15.92:6789/0 conn(0x7f0bf40b2130 legacy=0x7f0bf40aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:19.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.491+0000 7f0bf8f95700 1 -- 172.21.15.60:0/2547405114 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf414a350 con 0x7f0bf414af80 2022-06-29T16:00:19.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.491+0000 7f0bf8f95700 1 --2- 172.21.15.60:0/2547405114 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bf414af80 0x7f0bf40ad130 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f0be4002f70 tx=0x7f0be4006eb0).ready entity=mon.2 client_cookie=4985d94dbee4caea server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:19.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.491+0000 7f0bf2ffd700 1 -- 172.21.15.60:0/2547405114 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0be400d570 con 0x7f0bf414af80 2022-06-29T16:00:19.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.491+0000 7f0bf2ffd700 1 -- 172.21.15.60:0/2547405114 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be4000e90 con 0x7f0bf414af80 2022-06-29T16:00:19.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.491+0000 7f0bf2ffd700 1 -- 172.21.15.60:0/2547405114 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0be400d900 con 0x7f0bf414af80 2022-06-29T16:00:19.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.491+0000 7f0bf9796700 1 -- 172.21.15.60:0/2547405114 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bf414af80 msgr2=0x7f0bf40ad130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:19.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.491+0000 7f0bf9796700 1 --2- 172.21.15.60:0/2547405114 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bf414af80 0x7f0bf40ad130 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f0be4002f70 tx=0x7f0be4006eb0).stop 2022-06-29T16:00:19.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.491+0000 7f0bf9796700 1 -- 172.21.15.60:0/2547405114 shutdown_connections 2022-06-29T16:00:19.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.491+0000 7f0bf9796700 1 --2- 172.21.15.60:0/2547405114 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bf414af80 0x7f0bf40ad130 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:19.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.491+0000 7f0bf9796700 1 --2- 172.21.15.60:0/2547405114 >> v2:172.21.15.92:3300/0 conn(0x7f0bf405d070 0x7f0bf414aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:19.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.491+0000 7f0bf9796700 1 -- 172.21.15.60:0/2547405114 >> 172.21.15.60:0/2547405114 conn(0x7f0bf40b5890 msgr2=0x7f0bf40b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:19.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.491+0000 7f0bf9796700 1 -- 172.21.15.60:0/2547405114 shutdown_connections 2022-06-29T16:00:19.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.491+0000 7f0bf9796700 1 -- 172.21.15.60:0/2547405114 wait complete. 2022-06-29T16:00:19.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.491+0000 7f0bf9796700 1 Processor -- start 2022-06-29T16:00:19.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.491+0000 7f0bf9796700 1 -- start start 2022-06-29T16:00:19.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.495+0000 7f0bf9796700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bf405d070 0x7f0bf40662e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:19.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.495+0000 7f0bf9796700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bf40b2130 0x7f0bf4156650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:19.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.495+0000 7f0bf9796700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bf414af80 0x7f0bf4156b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:19.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.495+0000 7f0bf9796700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf41174b0 con 0x7f0bf405d070 2022-06-29T16:00:19.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.495+0000 7f0bf9796700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf40651e0 con 0x7f0bf414af80 2022-06-29T16:00:19.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.495+0000 7f0bf9796700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf4065480 con 0x7f0bf40b2130 2022-06-29T16:00:19.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.495+0000 7f0bf3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bf405d070 0x7f0bf40662e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:19.510 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.495+0000 7f0bf3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bf405d070 0x7f0bf40662e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47048/0 (socket says 172.21.15.60:47048) 2022-06-29T16:00:19.510 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.495+0000 7f0bf3fff700 1 -- 172.21.15.60:0/914916805 learned_addr learned my addr 172.21.15.60:0/914916805 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:19.510 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.495+0000 7f0bf37fe700 1 --2- 172.21.15.60:0/914916805 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bf40b2130 0x7f0bf4156650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:19.511 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.495+0000 7f0bf8f95700 1 --2- 172.21.15.60:0/914916805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bf414af80 0x7f0bf4156b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:19.511 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.495+0000 7f0bf3fff700 1 -- 172.21.15.60:0/914916805 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bf40b2130 msgr2=0x7f0bf4156650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:19.511 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.495+0000 7f0bf3fff700 1 --2- 172.21.15.60:0/914916805 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bf40b2130 0x7f0bf4156650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:19.511 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.495+0000 7f0bf3fff700 1 -- 172.21.15.60:0/914916805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bf414af80 msgr2=0x7f0bf4156b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:19.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.495+0000 7f0bf3fff700 1 --2- 172.21.15.60:0/914916805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bf414af80 0x7f0bf4156b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:19.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.495+0000 7f0bf3fff700 1 -- 172.21.15.60:0/914916805 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be400b1e0 con 0x7f0bf405d070 2022-06-29T16:00:19.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.495+0000 7f0bf37fe700 1 --2- 172.21.15.60:0/914916805 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bf40b2130 0x7f0bf4156650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:00:19.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.495+0000 7f0bf3fff700 1 --2- 172.21.15.60:0/914916805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bf405d070 0x7f0bf40662e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f0bec00c000 tx=0x7f0bec0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:19.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.495+0000 7f0bf17fa700 1 -- 172.21.15.60:0/914916805 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0bec00c6a0 con 0x7f0bf405d070 2022-06-29T16:00:19.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.495+0000 7f0bf17fa700 1 -- 172.21.15.60:0/914916805 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bec00c800 con 0x7f0bf405d070 2022-06-29T16:00:19.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.495+0000 7f0bf9796700 1 -- 172.21.15.60:0/914916805 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bf4065720 con 0x7f0bf405d070 2022-06-29T16:00:19.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.495+0000 7f0bf17fa700 1 -- 172.21.15.60:0/914916805 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0bec00cb60 con 0x7f0bf405d070 2022-06-29T16:00:19.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.495+0000 7f0bf9796700 1 -- 172.21.15.60:0/914916805 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bf4064e60 con 0x7f0bf405d070 2022-06-29T16:00:19.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.499+0000 7f0bf9796700 1 -- 172.21.15.60:0/914916805 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bf414a5f0 con 0x7f0bf405d070 2022-06-29T16:00:19.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.499+0000 7f0bf17fa700 1 -- 172.21.15.60:0/914916805 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0bec00ccc0 con 0x7f0bf405d070 2022-06-29T16:00:19.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.499+0000 7f0bf17fa700 1 --2- 172.21.15.60:0/914916805 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0bd4053b00 0x7f0bd4055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:19.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.499+0000 7f0bf17fa700 1 -- 172.21.15.60:0/914916805 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f0bec06bf20 con 0x7f0bf405d070 2022-06-29T16:00:19.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.499+0000 7f0bf37fe700 1 --2- 172.21.15.60:0/914916805 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0bd4053b00 0x7f0bd4055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:19.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.499+0000 7f0bf37fe700 1 --2- 172.21.15.60:0/914916805 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0bd4053b00 0x7f0bd4055fc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f0be800d000 tx=0x7f0be8007b80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:19.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.507+0000 7f0bf17fa700 1 -- 172.21.15.60:0/914916805 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0bec043010 con 0x7f0bf405d070 2022-06-29T16:00:19.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:19.631+0000 7f0bf9796700 1 -- 172.21.15.60:0/914916805 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f0bf4157030 con 0x7f0bf405d070 2022-06-29T16:00:20.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.371+0000 7f0bf17fa700 1 -- 172.21.15.60:0/914916805 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v40) v1 ==== 125+0+0 (secure 0 0 0) 0x7f0bec0102b0 con 0x7f0bf405d070 2022-06-29T16:00:20.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.375+0000 7f0bf9796700 1 -- 172.21.15.60:0/914916805 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0bd4053b00 msgr2=0x7f0bd4055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:20.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.375+0000 7f0bf9796700 1 --2- 172.21.15.60:0/914916805 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0bd4053b00 0x7f0bd4055fc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f0be800d000 tx=0x7f0be8007b80).stop 2022-06-29T16:00:20.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.375+0000 7f0bf9796700 1 -- 172.21.15.60:0/914916805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bf405d070 msgr2=0x7f0bf40662e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:20.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.375+0000 7f0bf9796700 1 --2- 172.21.15.60:0/914916805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bf405d070 0x7f0bf40662e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f0bec00c000 tx=0x7f0bec0091d0).stop 2022-06-29T16:00:20.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.375+0000 7f0bf9796700 1 -- 172.21.15.60:0/914916805 shutdown_connections 2022-06-29T16:00:20.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.375+0000 7f0bf9796700 1 --2- 172.21.15.60:0/914916805 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0bd4053b00 0x7f0bd4055fc0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:20.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.375+0000 7f0bf9796700 1 --2- 172.21.15.60:0/914916805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bf414af80 0x7f0bf4156b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:20.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.375+0000 7f0bf9796700 1 --2- 172.21.15.60:0/914916805 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bf40b2130 0x7f0bf4156650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:20.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.375+0000 7f0bf9796700 1 --2- 172.21.15.60:0/914916805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bf405d070 0x7f0bf40662e0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:20.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.375+0000 7f0bf9796700 1 -- 172.21.15.60:0/914916805 >> 172.21.15.60:0/914916805 conn(0x7f0bf40b5890 msgr2=0x7f0bf4143320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:20.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.379+0000 7f0bf9796700 1 -- 172.21.15.60:0/914916805 shutdown_connections 2022-06-29T16:00:20.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.379+0000 7f0bf9796700 1 -- 172.21.15.60:0/914916805 wait complete. 2022-06-29T16:00:20.395 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2022-06-29T16:00:20.399 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-06-29T16:00:20.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.499+0000 7feca303a700 1 Processor -- start 2022-06-29T16:00:20.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.499+0000 7feca303a700 1 -- start start 2022-06-29T16:00:20.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.499+0000 7feca303a700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fec9c0b28e0 0x7fec9c05bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:20.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.499+0000 7feca303a700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec9c0b1d70 con 0x7fec9c0b9be0 2022-06-29T16:00:20.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.499+0000 7feca303a700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec9c0b9fc0 con 0x7fec9c0afee0 2022-06-29T16:00:20.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.499+0000 7feca303a700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec9c0b3c40 con 0x7fec9c0b28e0 2022-06-29T16:00:20.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.499+0000 7feca2038700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fec9c0b9be0 0x7fec9c0b21d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55142/0 (socket says 172.21.15.60:55142) 2022-06-29T16:00:20.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.499+0000 7feca2038700 1 -- 172.21.15.60:0/127519809 learned_addr learned my addr 172.21.15.60:0/127519809 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:00:20.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.499+0000 7feca1837700 1 --2- 172.21.15.60:0/127519809 >> v2:172.21.15.92:3300/0 conn(0x7fec9c0b28e0 0x7fec9c05bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:20.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.503+0000 7feca1036700 1 -- 172.21.15.60:0/127519809 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2003837190 0 0) 0x7fec9c0b1d70 con 0x7fec9c0b9be0 2022-06-29T16:00:20.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.503+0000 7feca1036700 1 -- 172.21.15.60:0/127519809 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec84003260 con 0x7fec9c0b9be0 2022-06-29T16:00:20.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.503+0000 7feca1036700 1 -- 172.21.15.60:0/127519809 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3075788128 0 0) 0x7fec9c0b9fc0 con 0x7fec9c0afee0 2022-06-29T16:00:20.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.503+0000 7feca1036700 1 -- 172.21.15.60:0/127519809 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec9c0b1d70 con 0x7fec9c0afee0 2022-06-29T16:00:20.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.503+0000 7feca1837700 1 -- 172.21.15.60:0/127519809 >> v1:172.21.15.92:6789/0 conn(0x7fec9c0afee0 legacy=0x7fec9c141e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:20.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.503+0000 7feca1837700 1 -- 172.21.15.60:0/127519809 >> v1:172.21.15.60:6789/0 conn(0x7fec9c0b9be0 legacy=0x7fec9c0b21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:20.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.503+0000 7feca1837700 1 -- 172.21.15.60:0/127519809 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec9c0b3ee0 con 0x7fec9c0b28e0 2022-06-29T16:00:20.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.503+0000 7feca1837700 1 --2- 172.21.15.60:0/127519809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec9c0b28e0 0x7fec9c05bd20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fec98002f70 tx=0x7fec98006df0).ready entity=mon.1 client_cookie=a7d13c9c1482126c server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:20.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.503+0000 7feca1036700 1 -- 172.21.15.60:0/127519809 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec9800d540 con 0x7fec9c0b28e0 2022-06-29T16:00:20.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.503+0000 7feca1036700 1 -- 172.21.15.60:0/127519809 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec98000e90 con 0x7fec9c0b28e0 2022-06-29T16:00:20.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.503+0000 7feca1036700 1 -- 172.21.15.60:0/127519809 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec9800d890 con 0x7fec9c0b28e0 2022-06-29T16:00:20.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.503+0000 7feca303a700 1 -- 172.21.15.60:0/127519809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec9c0b28e0 msgr2=0x7fec9c05bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:20.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.503+0000 7feca303a700 1 --2- 172.21.15.60:0/127519809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec9c0b28e0 0x7fec9c05bd20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fec98002f70 tx=0x7fec98006df0).stop 2022-06-29T16:00:20.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.503+0000 7feca303a700 1 -- 172.21.15.60:0/127519809 shutdown_connections 2022-06-29T16:00:20.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.503+0000 7feca303a700 1 --2- 172.21.15.60:0/127519809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec9c0b28e0 0x7fec9c05bd20 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:20.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.503+0000 7feca303a700 1 -- 172.21.15.60:0/127519809 >> 172.21.15.60:0/127519809 conn(0x7fec9c062ce0 msgr2=0x7fec9c0b9190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:20.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.503+0000 7feca303a700 1 -- 172.21.15.60:0/127519809 shutdown_connections 2022-06-29T16:00:20.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.503+0000 7feca303a700 1 -- 172.21.15.60:0/127519809 wait complete. 2022-06-29T16:00:20.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.503+0000 7feca303a700 1 Processor -- start 2022-06-29T16:00:20.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.503+0000 7feca303a700 1 -- start start 2022-06-29T16:00:20.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.503+0000 7feca303a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec9c0afee0 0x7fec9c1473c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:20.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.503+0000 7feca303a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec9c0b28e0 0x7fec9c0aeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:20.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.503+0000 7feca303a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec9c0b9be0 0x7fec9c144d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:20.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.503+0000 7feca303a700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec9c145e60 con 0x7fec9c0b28e0 2022-06-29T16:00:20.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.503+0000 7feca303a700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec9c13dd60 con 0x7fec9c0b9be0 2022-06-29T16:00:20.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.503+0000 7feca303a700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec9c0b8e40 con 0x7fec9c0afee0 2022-06-29T16:00:20.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.503+0000 7feca1837700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec9c0b28e0 0x7fec9c0aeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:20.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.503+0000 7feca2839700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec9c0b9be0 0x7fec9c144d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:20.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.503+0000 7feca2038700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec9c0afee0 0x7fec9c1473c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:20.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.503+0000 7feca1837700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec9c0b28e0 0x7fec9c0aeaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47064/0 (socket says 172.21.15.60:47064) 2022-06-29T16:00:20.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.503+0000 7feca1837700 1 -- 172.21.15.60:0/1786839905 learned_addr learned my addr 172.21.15.60:0/1786839905 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:20.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.507+0000 7feca1837700 1 -- 172.21.15.60:0/1786839905 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec9c0afee0 msgr2=0x7fec9c1473c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:20.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.507+0000 7feca1837700 1 --2- 172.21.15.60:0/1786839905 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec9c0afee0 0x7fec9c1473c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:20.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.507+0000 7feca1837700 1 -- 172.21.15.60:0/1786839905 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec9c0b9be0 msgr2=0x7fec9c144d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:20.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.507+0000 7feca1837700 1 --2- 172.21.15.60:0/1786839905 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec9c0b9be0 0x7fec9c144d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:20.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.507+0000 7feca1837700 1 -- 172.21.15.60:0/1786839905 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec9800b180 con 0x7fec9c0b28e0 2022-06-29T16:00:20.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.507+0000 7feca2839700 1 --2- 172.21.15.60:0/1786839905 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec9c0b9be0 0x7fec9c144d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:00:20.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.507+0000 7feca2038700 1 --2- 172.21.15.60:0/1786839905 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec9c0afee0 0x7fec9c1473c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:00:20.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.507+0000 7feca1837700 1 --2- 172.21.15.60:0/1786839905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec9c0b28e0 0x7fec9c0aeaf0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fec98002f70 tx=0x7fec98014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:20.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.507+0000 7fec8f7fe700 1 -- 172.21.15.60:0/1786839905 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec98016070 con 0x7fec9c0b28e0 2022-06-29T16:00:20.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.507+0000 7fec8f7fe700 1 -- 172.21.15.60:0/1786839905 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec98013850 con 0x7fec9c0b28e0 2022-06-29T16:00:20.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.507+0000 7feca303a700 1 -- 172.21.15.60:0/1786839905 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec9c0b9070 con 0x7fec9c0b28e0 2022-06-29T16:00:20.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.507+0000 7fec8f7fe700 1 -- 172.21.15.60:0/1786839905 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec98013bb0 con 0x7fec9c0b28e0 2022-06-29T16:00:20.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.507+0000 7feca303a700 1 -- 172.21.15.60:0/1786839905 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec9c0af200 con 0x7fec9c0b28e0 2022-06-29T16:00:20.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.507+0000 7feca303a700 1 -- 172.21.15.60:0/1786839905 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec9c0573d0 con 0x7fec9c0b28e0 2022-06-29T16:00:20.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.507+0000 7fec8f7fe700 1 -- 172.21.15.60:0/1786839905 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fec98025030 con 0x7fec9c0b28e0 2022-06-29T16:00:20.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.511+0000 7fec8f7fe700 1 --2- 172.21.15.60:0/1786839905 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fec7c0539e0 0x7fec7c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:20.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.511+0000 7fec8f7fe700 1 -- 172.21.15.60:0/1786839905 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5957+0+0 (secure 0 0 0) 0x7fec9806cd40 con 0x7fec9c0b28e0 2022-06-29T16:00:20.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.511+0000 7feca2038700 1 --2- 172.21.15.60:0/1786839905 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fec7c0539e0 0x7fec7c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:20.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.511+0000 7feca2038700 1 --2- 172.21.15.60:0/1786839905 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fec7c0539e0 0x7fec7c055ea0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fec9000d000 tx=0x7fec900079e0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:20.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.515+0000 7fec8f7fe700 1 -- 172.21.15.60:0/1786839905 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fec98043c30 con 0x7fec9c0b28e0 2022-06-29T16:00:20.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.679+0000 7feca303a700 1 -- 172.21.15.60:0/1786839905 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7fec9c05d990 con 0x7fec9c0b28e0 2022-06-29T16:00:20.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.687+0000 7fec8f7fe700 1 -- 172.21.15.60:0/1786839905 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 7 and data pool 8 v14) v1 ==== 187+0+0 (secure 0 0 0) 0x7fec98023070 con 0x7fec9c0b28e0 2022-06-29T16:00:20.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.691+0000 7feca303a700 1 -- 172.21.15.60:0/1786839905 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fec7c0539e0 msgr2=0x7fec7c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:20.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.691+0000 7feca303a700 1 --2- 172.21.15.60:0/1786839905 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fec7c0539e0 0x7fec7c055ea0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fec9000d000 tx=0x7fec900079e0).stop 2022-06-29T16:00:20.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.691+0000 7feca303a700 1 -- 172.21.15.60:0/1786839905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec9c0b28e0 msgr2=0x7fec9c0aeaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:20.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.691+0000 7feca303a700 1 --2- 172.21.15.60:0/1786839905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec9c0b28e0 0x7fec9c0aeaf0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fec98002f70 tx=0x7fec98014000).stop 2022-06-29T16:00:20.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.691+0000 7feca303a700 1 -- 172.21.15.60:0/1786839905 shutdown_connections 2022-06-29T16:00:20.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.691+0000 7feca303a700 1 --2- 172.21.15.60:0/1786839905 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fec7c0539e0 0x7fec7c055ea0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:20.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.691+0000 7feca303a700 1 --2- 172.21.15.60:0/1786839905 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec9c0b9be0 0x7fec9c144d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:20.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.691+0000 7feca303a700 1 --2- 172.21.15.60:0/1786839905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec9c0b28e0 0x7fec9c0aeaf0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:20.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.691+0000 7feca303a700 1 --2- 172.21.15.60:0/1786839905 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec9c0afee0 0x7fec9c1473c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:20.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.691+0000 7feca303a700 1 -- 172.21.15.60:0/1786839905 >> 172.21.15.60:0/1786839905 conn(0x7fec9c062ce0 msgr2=0x7fec9c0b60d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:20.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.691+0000 7feca303a700 1 -- 172.21.15.60:0/1786839905 shutdown_connections 2022-06-29T16:00:20.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.691+0000 7feca303a700 1 -- 172.21.15.60:0/1786839905 wait complete. 2022-06-29T16:00:20.705 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 7 and data pool 8 2022-06-29T16:00:20.717 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:00:20.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.803+0000 7f953e02b700 1 Processor -- start 2022-06-29T16:00:20.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.803+0000 7f953e02b700 1 -- start start 2022-06-29T16:00:20.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.803+0000 7f953e02b700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f95380b5eb0 0x7f9538142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:20.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.803+0000 7f953e02b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95380b48b0 0x7f95380b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:20.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.803+0000 7f953e02b700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95380b4d10 con 0x7f95380b2c50 2022-06-29T16:00:20.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.803+0000 7f953e02b700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95380b54e0 con 0x7f95380b5eb0 2022-06-29T16:00:20.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.803+0000 7f953e02b700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95380b6290 con 0x7f95380b48b0 2022-06-29T16:00:20.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.803+0000 7f953d82a700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f95380b2c50 0x7f95380b3090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55168/0 (socket says 172.21.15.60:55168) 2022-06-29T16:00:20.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.803+0000 7f953d82a700 1 -- 172.21.15.60:0/2186150139 learned_addr learned my addr 172.21.15.60:0/2186150139 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:00:20.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.803+0000 7f953d029700 1 --2- 172.21.15.60:0/2186150139 >> v2:172.21.15.92:3300/0 conn(0x7f95380b5eb0 0x7f9538142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:20.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.803+0000 7f953c828700 1 --2- 172.21.15.60:0/2186150139 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95380b48b0 0x7f95380b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:20.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.803+0000 7f952ffff700 1 -- 172.21.15.60:0/2186150139 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3368784353 0 0) 0x7f95380b4d10 con 0x7f95380b2c50 2022-06-29T16:00:20.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.803+0000 7f952ffff700 1 -- 172.21.15.60:0/2186150139 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9524003260 con 0x7f95380b2c50 2022-06-29T16:00:20.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.803+0000 7f953d029700 1 -- 172.21.15.60:0/2186150139 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95380b48b0 msgr2=0x7f95380b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:20.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.803+0000 7f953d029700 1 --2- 172.21.15.60:0/2186150139 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95380b48b0 0x7f95380b05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:20.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.803+0000 7f953d029700 1 -- 172.21.15.60:0/2186150139 >> v1:172.21.15.60:6789/0 conn(0x7f95380b2c50 legacy=0x7f95380b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:20.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.803+0000 7f953d029700 1 -- 172.21.15.60:0/2186150139 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9538142f10 con 0x7f95380b5eb0 2022-06-29T16:00:20.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.803+0000 7f953c828700 1 --2- 172.21.15.60:0/2186150139 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95380b48b0 0x7f95380b05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:00:20.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.803+0000 7f953d029700 1 --2- 172.21.15.60:0/2186150139 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95380b5eb0 0x7f9538142a20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f9534002f70 tx=0x7f9534006df0).ready entity=mon.1 client_cookie=218363e96c1bc206 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:20.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.803+0000 7f952ffff700 1 -- 172.21.15.60:0/2186150139 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f953400d4a0 con 0x7f95380b5eb0 2022-06-29T16:00:20.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.803+0000 7f952ffff700 1 -- 172.21.15.60:0/2186150139 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9534000e90 con 0x7f95380b5eb0 2022-06-29T16:00:20.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.803+0000 7f952ffff700 1 -- 172.21.15.60:0/2186150139 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f953400d7f0 con 0x7f95380b5eb0 2022-06-29T16:00:20.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.807+0000 7f953e02b700 1 -- 172.21.15.60:0/2186150139 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95380b5eb0 msgr2=0x7f9538142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:20.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.807+0000 7f953e02b700 1 --2- 172.21.15.60:0/2186150139 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95380b5eb0 0x7f9538142a20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f9534002f70 tx=0x7f9534006df0).stop 2022-06-29T16:00:20.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.807+0000 7f953e02b700 1 -- 172.21.15.60:0/2186150139 shutdown_connections 2022-06-29T16:00:20.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.807+0000 7f953e02b700 1 --2- 172.21.15.60:0/2186150139 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95380b48b0 0x7f95380b05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:20.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.807+0000 7f953e02b700 1 --2- 172.21.15.60:0/2186150139 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95380b5eb0 0x7f9538142a20 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:20.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.807+0000 7f953e02b700 1 -- 172.21.15.60:0/2186150139 >> 172.21.15.60:0/2186150139 conn(0x7f9538062090 msgr2=0x7f9538062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:20.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.807+0000 7f953e02b700 1 -- 172.21.15.60:0/2186150139 shutdown_connections 2022-06-29T16:00:20.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.807+0000 7f953e02b700 1 -- 172.21.15.60:0/2186150139 wait complete. 2022-06-29T16:00:20.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.807+0000 7f953e02b700 1 Processor -- start 2022-06-29T16:00:20.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.807+0000 7f953e02b700 1 -- start start 2022-06-29T16:00:20.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.811+0000 7f953e02b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95380b2c50 0x7f9538147c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:20.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.811+0000 7f953e02b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95380b48b0 0x7f953813aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:20.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.811+0000 7f953e02b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95380b5eb0 0x7f9538151f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:20.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.811+0000 7f953e02b700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f953813c400 con 0x7f95380b2c50 2022-06-29T16:00:20.829 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.811+0000 7f953e02b700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f953813c630 con 0x7f95380b48b0 2022-06-29T16:00:20.829 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.811+0000 7f953e02b700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f953813c8d0 con 0x7f95380b5eb0 2022-06-29T16:00:20.829 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.811+0000 7f953d029700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95380b2c50 0x7f9538147c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:20.829 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.811+0000 7f953d029700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95380b2c50 0x7f9538147c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47086/0 (socket says 172.21.15.60:47086) 2022-06-29T16:00:20.829 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.811+0000 7f953d029700 1 -- 172.21.15.60:0/282933508 learned_addr learned my addr 172.21.15.60:0/282933508 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:20.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.811+0000 7f953c828700 1 --2- 172.21.15.60:0/282933508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95380b48b0 0x7f953813aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:20.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.811+0000 7f953d82a700 1 --2- 172.21.15.60:0/282933508 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95380b5eb0 0x7f9538151f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:20.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.811+0000 7f953d029700 1 -- 172.21.15.60:0/282933508 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95380b5eb0 msgr2=0x7f9538151f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:20.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.811+0000 7f953d029700 1 --2- 172.21.15.60:0/282933508 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95380b5eb0 0x7f9538151f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:20.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.811+0000 7f953d029700 1 -- 172.21.15.60:0/282933508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95380b48b0 msgr2=0x7f953813aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:20.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.811+0000 7f953d029700 1 --2- 172.21.15.60:0/282933508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95380b48b0 0x7f953813aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:20.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.811+0000 7f953d029700 1 -- 172.21.15.60:0/282933508 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9530009470 con 0x7f95380b2c50 2022-06-29T16:00:20.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.811+0000 7f953d82a700 1 --2- 172.21.15.60:0/282933508 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95380b5eb0 0x7f9538151f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:00:20.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.811+0000 7f953c828700 1 --2- 172.21.15.60:0/282933508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95380b48b0 0x7f953813aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:00:20.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.811+0000 7f953d029700 1 --2- 172.21.15.60:0/282933508 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95380b2c50 0x7f9538147c10 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f9534002f70 tx=0x7f9534014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:20.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.811+0000 7f952e7fc700 1 -- 172.21.15.60:0/282933508 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9534016070 con 0x7f95380b2c50 2022-06-29T16:00:20.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.811+0000 7f952e7fc700 1 -- 172.21.15.60:0/282933508 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9534013a40 con 0x7f95380b2c50 2022-06-29T16:00:20.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.811+0000 7f953e02b700 1 -- 172.21.15.60:0/282933508 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f953400b180 con 0x7f95380b2c50 2022-06-29T16:00:20.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.811+0000 7f952e7fc700 1 -- 172.21.15.60:0/282933508 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9534013da0 con 0x7f95380b2c50 2022-06-29T16:00:20.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.811+0000 7f953e02b700 1 -- 172.21.15.60:0/282933508 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95380b8dc0 con 0x7f95380b2c50 2022-06-29T16:00:20.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.811+0000 7f953e02b700 1 -- 172.21.15.60:0/282933508 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f953813d860 con 0x7f95380b2c50 2022-06-29T16:00:20.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.811+0000 7f952e7fc700 1 -- 172.21.15.60:0/282933508 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9534025030 con 0x7f95380b2c50 2022-06-29T16:00:20.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.815+0000 7f952e7fc700 1 --2- 172.21.15.60:0/282933508 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f95180539e0 0x7f9518055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:20.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.815+0000 7f952e7fc700 1 -- 172.21.15.60:0/282933508 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f953406cdb0 con 0x7f95380b2c50 2022-06-29T16:00:20.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.815+0000 7f953c828700 1 --2- 172.21.15.60:0/282933508 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f95180539e0 0x7f9518055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:20.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.815+0000 7f953c828700 1 --2- 172.21.15.60:0/282933508 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f95180539e0 0x7f9518055ea0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f953000b000 tx=0x7f9530008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:20.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.819+0000 7f952e7fc700 1 -- 172.21.15.60:0/282933508 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9534043cd0 con 0x7f95380b2c50 2022-06-29T16:00:20.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.947+0000 7f953e02b700 1 -- 172.21.15.60:0/282933508 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f953813ca90 con 0x7f95380b2c50 2022-06-29T16:00:20.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.947+0000 7f952e7fc700 1 -- 172.21.15.60:0/282933508 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+16064 (secure 0 0 0) 0x7f9534023090 con 0x7f95380b2c50 2022-06-29T16:00:20.958 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:00:20.958 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":41,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:00:20.685335+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:00:18.552380+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2022-06-29T16:00:19.688558+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:00:20.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.951+0000 7f953e02b700 1 -- 172.21.15.60:0/282933508 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f95180539e0 msgr2=0x7f9518055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:20.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.951+0000 7f953e02b700 1 --2- 172.21.15.60:0/282933508 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f95180539e0 0x7f9518055ea0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f953000b000 tx=0x7f9530008040).stop 2022-06-29T16:00:20.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.951+0000 7f953e02b700 1 -- 172.21.15.60:0/282933508 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95380b2c50 msgr2=0x7f9538147c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:20.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.951+0000 7f953e02b700 1 --2- 172.21.15.60:0/282933508 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95380b2c50 0x7f9538147c10 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f9534002f70 tx=0x7f9534014000).stop 2022-06-29T16:00:20.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.951+0000 7f953e02b700 1 -- 172.21.15.60:0/282933508 shutdown_connections 2022-06-29T16:00:20.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.951+0000 7f953e02b700 1 --2- 172.21.15.60:0/282933508 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f95180539e0 0x7f9518055ea0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:20.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.951+0000 7f953e02b700 1 --2- 172.21.15.60:0/282933508 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95380b5eb0 0x7f9538151f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:20.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.951+0000 7f953e02b700 1 --2- 172.21.15.60:0/282933508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95380b48b0 0x7f953813aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:20.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.951+0000 7f953e02b700 1 --2- 172.21.15.60:0/282933508 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95380b2c50 0x7f9538147c10 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:20.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.951+0000 7f953e02b700 1 -- 172.21.15.60:0/282933508 >> 172.21.15.60:0/282933508 conn(0x7f9538062090 msgr2=0x7f953805bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:20.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.951+0000 7f953e02b700 1 -- 172.21.15.60:0/282933508 shutdown_connections 2022-06-29T16:00:20.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:20.951+0000 7f953e02b700 1 -- 172.21.15.60:0/282933508 wait complete. 2022-06-29T16:00:20.972 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:00:21.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.067+0000 7f1e14614700 1 Processor -- start 2022-06-29T16:00:21.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e14614700 1 -- start start 2022-06-29T16:00:21.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e14614700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1e0c060460 0x7f1e0c0af290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:21.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e14614700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0c0b65b0 con 0x7f1e0c0b12e0 2022-06-29T16:00:21.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e14614700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0c0adf20 con 0x7f1e0c0b61d0 2022-06-29T16:00:21.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e14614700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e0c147110 con 0x7f1e0c060460 2022-06-29T16:00:21.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e11baf700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f1e0c0b12e0 0x7f1e0c1475b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55180/0 (socket says 172.21.15.60:55180) 2022-06-29T16:00:21.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e11baf700 1 -- 172.21.15.60:0/3385076139 learned_addr learned my addr 172.21.15.60:0/3385076139 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:00:21.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e12bb1700 1 --2- 172.21.15.60:0/3385076139 >> v2:172.21.15.92:3300/0 conn(0x7f1e0c060460 0x7f1e0c0af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:21.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e113ae700 1 -- 172.21.15.60:0/3385076139 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4145239763 0 0) 0x7f1e0c0b65b0 con 0x7f1e0c0b12e0 2022-06-29T16:00:21.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e113ae700 1 -- 172.21.15.60:0/3385076139 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1df4003260 con 0x7f1e0c0b12e0 2022-06-29T16:00:21.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e113ae700 1 -- 172.21.15.60:0/3385076139 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2196996009 0 0) 0x7f1df4003260 con 0x7f1e0c0b12e0 2022-06-29T16:00:21.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e113ae700 1 -- 172.21.15.60:0/3385076139 >> v2:172.21.15.92:3300/0 conn(0x7f1e0c060460 msgr2=0x7f1e0c0af290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:21.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e113ae700 1 --2- 172.21.15.60:0/3385076139 >> v2:172.21.15.92:3300/0 conn(0x7f1e0c060460 0x7f1e0c0af290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:21.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e113ae700 1 -- 172.21.15.60:0/3385076139 >> v1:172.21.15.92:6789/0 conn(0x7f1e0c0b61d0 legacy=0x7f1e0c05f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:21.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e113ae700 1 -- 172.21.15.60:0/3385076139 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e0c146e30 con 0x7f1e0c0b12e0 2022-06-29T16:00:21.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e113ae700 1 -- 172.21.15.60:0/3385076139 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f1e00001660 con 0x7f1e0c0b12e0 2022-06-29T16:00:21.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e113ae700 1 -- 172.21.15.60:0/3385076139 >> v1:172.21.15.60:6789/0 conn(0x7f1e0c0b12e0 legacy=0x7f1e0c1475b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:21.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e113ae700 1 --2- 172.21.15.60:0/3385076139 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1df4003750 0x7f1df4003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:21.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e113ae700 1 --2- 172.21.15.60:0/3385076139 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1df4004500 0x7f1df4004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:21.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e113ae700 1 --2- 172.21.15.60:0/3385076139 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1df4004e70 0x7f1df400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:21.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e113ae700 1 -- 172.21.15.60:0/3385076139 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0c147110 con 0x7f1df4003750 2022-06-29T16:00:21.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e113ae700 1 -- 172.21.15.60:0/3385076139 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df400bd50 con 0x7f1df4004500 2022-06-29T16:00:21.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e113ae700 1 -- 172.21.15.60:0/3385076139 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1df400bf10 con 0x7f1df4004e70 2022-06-29T16:00:21.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e123b0700 1 --2- 172.21.15.60:0/3385076139 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1df4003750 0x7f1df4003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:21.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e11baf700 1 --2- 172.21.15.60:0/3385076139 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1df4004500 0x7f1df4004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:21.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e12bb1700 1 --2- 172.21.15.60:0/3385076139 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1df4004e70 0x7f1df400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:21.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e123b0700 1 -- 172.21.15.60:0/3385076139 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1df4004e70 msgr2=0x7f1df400b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:21.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e123b0700 1 --2- 172.21.15.60:0/3385076139 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1df4004e70 0x7f1df400b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:21.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e123b0700 1 -- 172.21.15.60:0/3385076139 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1df4004500 msgr2=0x7f1df4004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:21.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e123b0700 1 --2- 172.21.15.60:0/3385076139 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1df4004500 0x7f1df4004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:21.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e123b0700 1 -- 172.21.15.60:0/3385076139 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df4003260 con 0x7f1df4003750 2022-06-29T16:00:21.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e12bb1700 1 -- 172.21.15.60:0/3385076139 reap_dead start 2022-06-29T16:00:21.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e123b0700 1 --2- 172.21.15.60:0/3385076139 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1df4003750 0x7f1df4003b30 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f1e08001020 tx=0x7f1e080079e0).ready entity=mon.0 client_cookie=b0656c6be3cf5d59 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:21.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e113ae700 1 -- 172.21.15.60:0/3385076139 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e08007cf0 con 0x7f1df4003750 2022-06-29T16:00:21.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e113ae700 1 -- 172.21.15.60:0/3385076139 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e08007030 con 0x7f1df4003750 2022-06-29T16:00:21.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.071+0000 7f1e113ae700 1 -- 172.21.15.60:0/3385076139 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e08007390 con 0x7f1df4003750 2022-06-29T16:00:21.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.075+0000 7f1e14614700 1 -- 172.21.15.60:0/3385076139 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1df4003750 msgr2=0x7f1df4003b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:21.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.075+0000 7f1e14614700 1 --2- 172.21.15.60:0/3385076139 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1df4003750 0x7f1df4003b30 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f1e08001020 tx=0x7f1e080079e0).stop 2022-06-29T16:00:21.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.075+0000 7f1e14614700 1 -- 172.21.15.60:0/3385076139 shutdown_connections 2022-06-29T16:00:21.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.075+0000 7f1e14614700 1 --2- 172.21.15.60:0/3385076139 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1df4003750 0x7f1df4003b30 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:21.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.075+0000 7f1e14614700 1 -- 172.21.15.60:0/3385076139 >> 172.21.15.60:0/3385076139 conn(0x7f1e0c063b50 msgr2=0x7f1e0c0b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:21.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.075+0000 7f1e14614700 1 -- 172.21.15.60:0/3385076139 shutdown_connections 2022-06-29T16:00:21.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.075+0000 7f1e14614700 1 -- 172.21.15.60:0/3385076139 wait complete. 2022-06-29T16:00:21.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.075+0000 7f1e14614700 1 Processor -- start 2022-06-29T16:00:21.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.075+0000 7f1e14614700 1 -- start start 2022-06-29T16:00:21.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.075+0000 7f1e14614700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1df4003750 0x7f1e0c14cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:21.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.075+0000 7f1e14614700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e0c064130 0x7f1e0c0650b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:21.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.075+0000 7f1e14614700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e0c159a30 0x7f1e0c159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:21.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.075+0000 7f1e14614700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0c1562e0 con 0x7f1e0c159a30 2022-06-29T16:00:21.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.075+0000 7f1e14614700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0c066d20 con 0x7f1e0c064130 2022-06-29T16:00:21.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.075+0000 7f1e14614700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0c1595d0 con 0x7f1df4003750 2022-06-29T16:00:21.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.075+0000 7f1e12bb1700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e0c159a30 0x7f1e0c159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:21.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.075+0000 7f1e11baf700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e0c064130 0x7f1e0c0650b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:21.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.075+0000 7f1e123b0700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1df4003750 0x7f1e0c14cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:21.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.075+0000 7f1e123b0700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1df4003750 0x7f1e0c14cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:60200/0 (socket says 172.21.15.60:60200) 2022-06-29T16:00:21.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.075+0000 7f1e11baf700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e0c064130 0x7f1e0c0650b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:53138/0 (socket says 172.21.15.60:53138) 2022-06-29T16:00:21.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.075+0000 7f1e123b0700 1 -- 172.21.15.60:0/2475197040 learned_addr learned my addr 172.21.15.60:0/2475197040 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:21.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.079+0000 7f1e123b0700 1 -- 172.21.15.60:0/2475197040 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e0c064130 msgr2=0x7f1e0c0650b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:21.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.079+0000 7f1e123b0700 1 --2- 172.21.15.60:0/2475197040 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e0c064130 0x7f1e0c0650b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:21.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.079+0000 7f1e123b0700 1 -- 172.21.15.60:0/2475197040 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e0c159a30 msgr2=0x7f1e0c159eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:21.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.079+0000 7f1e123b0700 1 --2- 172.21.15.60:0/2475197040 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e0c159a30 0x7f1e0c159eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:21.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.079+0000 7f1e123b0700 1 -- 172.21.15.60:0/2475197040 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e0c15a3a0 con 0x7f1df4003750 2022-06-29T16:00:21.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.079+0000 7f1e11baf700 1 --2- 172.21.15.60:0/2475197040 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e0c064130 0x7f1e0c0650b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:00:21.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.079+0000 7f1e123b0700 1 --2- 172.21.15.60:0/2475197040 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1df4003750 0x7f1e0c14cd60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f1e0c159910 tx=0x7f1e08015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:21.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.079+0000 7f1dff7fe700 1 -- 172.21.15.60:0/2475197040 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e08007cf0 con 0x7f1df4003750 2022-06-29T16:00:21.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.079+0000 7f1dff7fe700 1 -- 172.21.15.60:0/2475197040 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e08020640 con 0x7f1df4003750 2022-06-29T16:00:21.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.079+0000 7f1dff7fe700 1 -- 172.21.15.60:0/2475197040 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e080209a0 con 0x7f1df4003750 2022-06-29T16:00:21.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.079+0000 7f1e14614700 1 -- 172.21.15.60:0/2475197040 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e0c156040 con 0x7f1df4003750 2022-06-29T16:00:21.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.079+0000 7f1e14614700 1 -- 172.21.15.60:0/2475197040 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e0c064640 con 0x7f1df4003750 2022-06-29T16:00:21.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.079+0000 7f1e14614700 1 -- 172.21.15.60:0/2475197040 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e0c0573d0 con 0x7f1df4003750 2022-06-29T16:00:21.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.079+0000 7f1dff7fe700 1 -- 172.21.15.60:0/2475197040 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1e08027070 con 0x7f1df4003750 2022-06-29T16:00:21.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.079+0000 7f1dff7fe700 1 --2- 172.21.15.60:0/2475197040 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1dec053aa0 0x7f1dec055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:21.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.079+0000 7f1dff7fe700 1 -- 172.21.15.60:0/2475197040 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1e0806ca70 con 0x7f1df4003750 2022-06-29T16:00:21.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.083+0000 7f1e11baf700 1 --2- 172.21.15.60:0/2475197040 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1dec053aa0 0x7f1dec055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:21.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.083+0000 7f1e11baf700 1 --2- 172.21.15.60:0/2475197040 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1dec053aa0 0x7f1dec055f60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f1e00005000 tx=0x7f1e00009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:21.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.087+0000 7f1dff7fe700 1 -- 172.21.15.60:0/2475197040 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1e080438f0 con 0x7f1df4003750 2022-06-29T16:00:21.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.219+0000 7f1e14614700 1 -- 172.21.15.60:0/2475197040 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e0c159710 con 0x7f1df4003750 2022-06-29T16:00:21.230 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.223+0000 7f1dff7fe700 1 -- 172.21.15.60:0/2475197040 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+16064 (secure 0 0 0) 0x7f1e0806ce60 con 0x7f1df4003750 2022-06-29T16:00:21.230 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:00:21.231 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":41,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:00:20.685335+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:00:18.552380+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2022-06-29T16:00:19.688558+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:00:21.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.223+0000 7f1e14614700 1 -- 172.21.15.60:0/2475197040 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1dec053aa0 msgr2=0x7f1dec055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:21.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.223+0000 7f1e14614700 1 --2- 172.21.15.60:0/2475197040 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1dec053aa0 0x7f1dec055f60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f1e00005000 tx=0x7f1e00009040).stop 2022-06-29T16:00:21.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.223+0000 7f1e14614700 1 -- 172.21.15.60:0/2475197040 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1df4003750 msgr2=0x7f1e0c14cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:21.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.223+0000 7f1e14614700 1 --2- 172.21.15.60:0/2475197040 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1df4003750 0x7f1e0c14cd60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f1e0c159910 tx=0x7f1e08015000).stop 2022-06-29T16:00:21.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.223+0000 7f1e14614700 1 -- 172.21.15.60:0/2475197040 shutdown_connections 2022-06-29T16:00:21.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.223+0000 7f1e14614700 1 --2- 172.21.15.60:0/2475197040 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1dec053aa0 0x7f1dec055f60 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:21.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.223+0000 7f1e14614700 1 --2- 172.21.15.60:0/2475197040 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e0c159a30 0x7f1e0c159eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:21.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.223+0000 7f1e14614700 1 --2- 172.21.15.60:0/2475197040 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e0c064130 0x7f1e0c0650b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:21.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.223+0000 7f1e14614700 1 --2- 172.21.15.60:0/2475197040 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1df4003750 0x7f1e0c14cd60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:21.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.223+0000 7f1e14614700 1 -- 172.21.15.60:0/2475197040 >> 172.21.15.60:0/2475197040 conn(0x7f1e0c063b50 msgr2=0x7f1e0c05f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:21.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.223+0000 7f1e14614700 1 -- 172.21.15.60:0/2475197040 shutdown_connections 2022-06-29T16:00:21.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.223+0000 7f1e14614700 1 -- 172.21.15.60:0/2475197040 wait complete. 2022-06-29T16:00:21.250 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-06-29T16:00:21.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.351+0000 7f3aca06f700 1 Processor -- start 2022-06-29T16:00:21.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.351+0000 7f3aca06f700 1 -- start start 2022-06-29T16:00:21.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.351+0000 7f3aca06f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ac40b0890 0x7f3ac40b4540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:21.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.351+0000 7f3aca06f700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac40b9fd0 con 0x7f3ac40b9bf0 2022-06-29T16:00:21.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.351+0000 7f3aca06f700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac40b1a50 con 0x7f3ac40b3b40 2022-06-29T16:00:21.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.351+0000 7f3aca06f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac40b1cf0 con 0x7f3ac40b0890 2022-06-29T16:00:21.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.351+0000 7f3ac906d700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f3ac40b9bf0 0x7f3ac40b2190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55200/0 (socket says 172.21.15.60:55200) 2022-06-29T16:00:21.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.351+0000 7f3ac906d700 1 -- 172.21.15.60:0/876270072 learned_addr learned my addr 172.21.15.60:0/876270072 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:00:21.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.351+0000 7f3ac886c700 1 --2- 172.21.15.60:0/876270072 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ac40b0890 0x7f3ac40b4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:21.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.351+0000 7f3abbfff700 1 -- 172.21.15.60:0/876270072 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3749738331 0 0) 0x7f3ac40b9fd0 con 0x7f3ac40b9bf0 2022-06-29T16:00:21.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.351+0000 7f3abbfff700 1 -- 172.21.15.60:0/876270072 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3aa8003260 con 0x7f3ac40b9bf0 2022-06-29T16:00:21.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.351+0000 7f3abbfff700 1 -- 172.21.15.60:0/876270072 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 265554038 0 0) 0x7f3ac40b1a50 con 0x7f3ac40b3b40 2022-06-29T16:00:21.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.351+0000 7f3abbfff700 1 -- 172.21.15.60:0/876270072 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ac40b9fd0 con 0x7f3ac40b3b40 2022-06-29T16:00:21.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.351+0000 7f3abbfff700 1 -- 172.21.15.60:0/876270072 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2180122037 0 0) 0x7f3aa8003260 con 0x7f3ac40b9bf0 2022-06-29T16:00:21.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.351+0000 7f3abbfff700 1 -- 172.21.15.60:0/876270072 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ac40b0890 msgr2=0x7f3ac40b4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:21.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.351+0000 7f3abbfff700 1 --2- 172.21.15.60:0/876270072 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ac40b0890 0x7f3ac40b4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:21.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.351+0000 7f3abbfff700 1 -- 172.21.15.60:0/876270072 >> v1:172.21.15.92:6789/0 conn(0x7f3ac40b3b40 legacy=0x7f3ac4141b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:21.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.351+0000 7f3abbfff700 1 -- 172.21.15.60:0/876270072 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac40b3470 con 0x7f3ac40b9bf0 2022-06-29T16:00:21.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.351+0000 7f3abbfff700 1 -- 172.21.15.60:0/876270072 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f3ac0001660 con 0x7f3ac40b9bf0 2022-06-29T16:00:21.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.351+0000 7f3abbfff700 1 -- 172.21.15.60:0/876270072 >> v1:172.21.15.60:6789/0 conn(0x7f3ac40b9bf0 legacy=0x7f3ac40b2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:21.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.351+0000 7f3abbfff700 1 --2- 172.21.15.60:0/876270072 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ac40b0890 0x7f3aa8003910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:21.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.351+0000 7f3abbfff700 1 --2- 172.21.15.60:0/876270072 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3aa8003e20 0x7f3aa8004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:21.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.351+0000 7f3abbfff700 1 --2- 172.21.15.60:0/876270072 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3aa80049f0 0x7f3aa8004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:21.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.351+0000 7f3abbfff700 1 -- 172.21.15.60:0/876270072 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac40b1cf0 con 0x7f3aa80049f0 2022-06-29T16:00:21.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.351+0000 7f3abbfff700 1 -- 172.21.15.60:0/876270072 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa800b800 con 0x7f3aa8003e20 2022-06-29T16:00:21.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.351+0000 7f3abbfff700 1 -- 172.21.15.60:0/876270072 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa800b9c0 con 0x7f3ac40b0890 2022-06-29T16:00:21.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.351+0000 7f3ac986e700 1 --2- 172.21.15.60:0/876270072 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3aa80049f0 0x7f3aa8004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:21.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.355+0000 7f3ac886c700 1 --2- 172.21.15.60:0/876270072 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3aa8003e20 0x7f3aa8004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:21.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.355+0000 7f3ac906d700 1 --2- 172.21.15.60:0/876270072 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ac40b0890 0x7f3aa8003910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:21.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.355+0000 7f3ac986e700 1 -- 172.21.15.60:0/876270072 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ac40b0890 msgr2=0x7f3aa8003910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:21.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.355+0000 7f3ac986e700 1 --2- 172.21.15.60:0/876270072 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ac40b0890 0x7f3aa8003910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:21.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.355+0000 7f3ac986e700 1 -- 172.21.15.60:0/876270072 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3aa8003e20 msgr2=0x7f3aa8004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:21.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.355+0000 7f3ac986e700 1 --2- 172.21.15.60:0/876270072 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3aa8003e20 0x7f3aa8004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:21.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.355+0000 7f3ac986e700 1 -- 172.21.15.60:0/876270072 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa8003260 con 0x7f3aa80049f0 2022-06-29T16:00:21.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.355+0000 7f3ac986e700 1 --2- 172.21.15.60:0/876270072 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3aa80049f0 0x7f3aa8004e70 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f3abc001490 tx=0x7f3abc007d80).ready entity=mon.0 client_cookie=5f14ae12910f729f server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:21.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.355+0000 7f3abbfff700 1 -- 172.21.15.60:0/876270072 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3abc001780 con 0x7f3aa80049f0 2022-06-29T16:00:21.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.355+0000 7f3abbfff700 1 -- 172.21.15.60:0/876270072 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3abc006890 con 0x7f3aa80049f0 2022-06-29T16:00:21.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.355+0000 7f3abbfff700 1 -- 172.21.15.60:0/876270072 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3abc006bf0 con 0x7f3aa80049f0 2022-06-29T16:00:21.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.355+0000 7f3aca06f700 1 -- 172.21.15.60:0/876270072 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3aa80049f0 msgr2=0x7f3aa8004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:21.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.355+0000 7f3aca06f700 1 --2- 172.21.15.60:0/876270072 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3aa80049f0 0x7f3aa8004e70 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f3abc001490 tx=0x7f3abc007d80).stop 2022-06-29T16:00:21.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.355+0000 7f3ac986e700 1 -- 172.21.15.60:0/876270072 reap_dead start 2022-06-29T16:00:21.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.355+0000 7f3aca06f700 1 -- 172.21.15.60:0/876270072 shutdown_connections 2022-06-29T16:00:21.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.355+0000 7f3aca06f700 1 -- 172.21.15.60:0/876270072 >> 172.21.15.60:0/876270072 conn(0x7f3ac4062b60 msgr2=0x7f3ac40b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:21.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.355+0000 7f3aca06f700 1 -- 172.21.15.60:0/876270072 shutdown_connections 2022-06-29T16:00:21.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.355+0000 7f3aca06f700 1 -- 172.21.15.60:0/876270072 wait complete. 2022-06-29T16:00:21.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.355+0000 7f3aca06f700 1 Processor -- start 2022-06-29T16:00:21.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.355+0000 7f3aca06f700 1 -- start start 2022-06-29T16:00:21.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.359+0000 7f3aca06f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3aa80049f0 0x7f3ac4151fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:21.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.359+0000 7f3aca06f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ac40af100 0x7f3ac413d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:21.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.359+0000 7f3aca06f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ac413db30 0x7f3ac40aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:21.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.359+0000 7f3aca06f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac40af7c0 con 0x7f3ac40af100 2022-06-29T16:00:21.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.359+0000 7f3aca06f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac40ae5c0 con 0x7f3aa80049f0 2022-06-29T16:00:21.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.359+0000 7f3aca06f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac4145510 con 0x7f3ac413db30 2022-06-29T16:00:21.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.359+0000 7f3ac886c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ac40af100 0x7f3ac413d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:21.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.359+0000 7f3ac886c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ac40af100 0x7f3ac413d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47128/0 (socket says 172.21.15.60:47128) 2022-06-29T16:00:21.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.359+0000 7f3ac886c700 1 -- 172.21.15.60:0/903807466 learned_addr learned my addr 172.21.15.60:0/903807466 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:21.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.359+0000 7f3ac986e700 1 --2- 172.21.15.60:0/903807466 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ac413db30 0x7f3ac40aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:21.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.359+0000 7f3ac906d700 1 --2- 172.21.15.60:0/903807466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3aa80049f0 0x7f3ac4151fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:21.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.359+0000 7f3ac886c700 1 -- 172.21.15.60:0/903807466 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ac413db30 msgr2=0x7f3ac40aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:21.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.359+0000 7f3ac886c700 1 --2- 172.21.15.60:0/903807466 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ac413db30 0x7f3ac40aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:21.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.359+0000 7f3ac886c700 1 -- 172.21.15.60:0/903807466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3aa80049f0 msgr2=0x7f3ac4151fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:21.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.359+0000 7f3ac886c700 1 --2- 172.21.15.60:0/903807466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3aa80049f0 0x7f3ac4151fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:21.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.359+0000 7f3ac886c700 1 -- 172.21.15.60:0/903807466 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac41457b0 con 0x7f3ac40af100 2022-06-29T16:00:21.382 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.359+0000 7f3ac886c700 1 --2- 172.21.15.60:0/903807466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ac40af100 0x7f3ac413d640 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f3ab400b000 tx=0x7f3ab4008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:21.382 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.359+0000 7f3aba7fc700 1 -- 172.21.15.60:0/903807466 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ab4012610 con 0x7f3ac40af100 2022-06-29T16:00:21.382 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.359+0000 7f3aba7fc700 1 -- 172.21.15.60:0/903807466 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ab4012770 con 0x7f3ac40af100 2022-06-29T16:00:21.382 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.359+0000 7f3aca06f700 1 -- 172.21.15.60:0/903807466 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ac4145240 con 0x7f3ac40af100 2022-06-29T16:00:21.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.359+0000 7f3aba7fc700 1 -- 172.21.15.60:0/903807466 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ab4012ad0 con 0x7f3ac40af100 2022-06-29T16:00:21.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.359+0000 7f3aca06f700 1 -- 172.21.15.60:0/903807466 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ac41524d0 con 0x7f3ac40af100 2022-06-29T16:00:21.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.359+0000 7f3aca06f700 1 -- 172.21.15.60:0/903807466 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ac40573d0 con 0x7f3ac40af100 2022-06-29T16:00:21.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.363+0000 7f3aba7fc700 1 -- 172.21.15.60:0/903807466 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3ab4012c30 con 0x7f3ac40af100 2022-06-29T16:00:21.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.363+0000 7f3aba7fc700 1 --2- 172.21.15.60:0/903807466 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3aac053b00 0x7f3aac055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:21.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.363+0000 7f3aba7fc700 1 -- 172.21.15.60:0/903807466 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3ab406c760 con 0x7f3ac40af100 2022-06-29T16:00:21.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.363+0000 7f3ac906d700 1 --2- 172.21.15.60:0/903807466 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3aac053b00 0x7f3aac055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:21.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.363+0000 7f3ac906d700 1 --2- 172.21.15.60:0/903807466 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3aac053b00 0x7f3aac055fc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f3ac000d000 tx=0x7f3ac0007040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:21.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.367+0000 7f3aba7fc700 1 -- 172.21.15.60:0/903807466 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3ab4043680 con 0x7f3ac40af100 2022-06-29T16:00:21.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.535+0000 7f3aca06f700 1 -- 172.21.15.60:0/903807466 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f3ac40afa50 con 0x7f3ac40af100 2022-06-29T16:00:21.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.699+0000 7f3aba7fc700 1 -- 172.21.15.60:0/903807466 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) v1 ==== 118+0+0 (secure 0 0 0) 0x7f3ab4046ca0 con 0x7f3ac40af100 2022-06-29T16:00:21.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.703+0000 7f3aca06f700 1 -- 172.21.15.60:0/903807466 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3aac053b00 msgr2=0x7f3aac055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:21.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.703+0000 7f3aca06f700 1 --2- 172.21.15.60:0/903807466 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3aac053b00 0x7f3aac055fc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f3ac000d000 tx=0x7f3ac0007040).stop 2022-06-29T16:00:21.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.703+0000 7f3aca06f700 1 -- 172.21.15.60:0/903807466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ac40af100 msgr2=0x7f3ac413d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:21.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.703+0000 7f3aca06f700 1 --2- 172.21.15.60:0/903807466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ac40af100 0x7f3ac413d640 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f3ab400b000 tx=0x7f3ab4008040).stop 2022-06-29T16:00:21.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.703+0000 7f3aca06f700 1 -- 172.21.15.60:0/903807466 shutdown_connections 2022-06-29T16:00:21.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.703+0000 7f3aca06f700 1 --2- 172.21.15.60:0/903807466 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3aac053b00 0x7f3aac055fc0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:21.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.703+0000 7f3aca06f700 1 --2- 172.21.15.60:0/903807466 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ac413db30 0x7f3ac40aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:21.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.703+0000 7f3aca06f700 1 --2- 172.21.15.60:0/903807466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ac40af100 0x7f3ac413d640 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:21.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.703+0000 7f3aca06f700 1 --2- 172.21.15.60:0/903807466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3aa80049f0 0x7f3ac4151fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:21.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.703+0000 7f3aca06f700 1 -- 172.21.15.60:0/903807466 >> 172.21.15.60:0/903807466 conn(0x7f3ac4062b60 msgr2=0x7f3ac40b4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:21.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.703+0000 7f3aca06f700 1 -- 172.21.15.60:0/903807466 shutdown_connections 2022-06-29T16:00:21.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.703+0000 7f3aca06f700 1 -- 172.21.15.60:0/903807466 wait complete. 2022-06-29T16:00:21.724 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:00:21.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.823+0000 7f3a3f6d2700 1 Processor -- start 2022-06-29T16:00:21.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.823+0000 7f3a3f6d2700 1 -- start start 2022-06-29T16:00:21.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.827+0000 7f3a3f6d2700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3a380b61d0 0x7f3a380b0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:21.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.827+0000 7f3a3f6d2700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3a3813f790 0x7f3a3813bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:21.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.827+0000 7f3a3f6d2700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a38142820 con 0x7f3a3813fbd0 2022-06-29T16:00:21.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.827+0000 7f3a3f6d2700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a38062bb0 con 0x7f3a3813f790 2022-06-29T16:00:21.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.827+0000 7f3a3f6d2700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a380b65b0 con 0x7f3a380b61d0 2022-06-29T16:00:21.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.827+0000 7f3a3cc6d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3a3813f790 0x7f3a3813bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:21.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.827+0000 7f3a3cc6d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3a3813f790 0x7f3a3813bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47136/0 (socket says 172.21.15.60:47136) 2022-06-29T16:00:21.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.827+0000 7f3a3cc6d700 1 -- 172.21.15.60:0/3129596397 learned_addr learned my addr 172.21.15.60:0/3129596397 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:21.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.827+0000 7f3a3d46e700 1 --2- 172.21.15.60:0/3129596397 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3a380b61d0 0x7f3a380b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:21.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.827+0000 7f3a3cc6d700 1 -- 172.21.15.60:0/3129596397 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3a380b61d0 msgr2=0x7f3a380b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:21.843 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.827+0000 7f3a3cc6d700 1 --2- 172.21.15.60:0/3129596397 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3a380b61d0 0x7f3a380b0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:21.843 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.827+0000 7f3a3cc6d700 1 -- 172.21.15.60:0/3129596397 >> v1:172.21.15.92:6789/0 conn(0x7f3a3813fbd0 legacy=0x7f3a3813c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:21.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.827+0000 7f3a3cc6d700 1 -- 172.21.15.60:0/3129596397 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a3813f050 con 0x7f3a3813f790 2022-06-29T16:00:21.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.827+0000 7f3a3cc6d700 1 --2- 172.21.15.60:0/3129596397 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3a3813f790 0x7f3a3813bc40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f3a28002f70 tx=0x7f3a28006e80).ready entity=mon.0 client_cookie=bcf48e6bc3953e3b server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:21.845 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.827+0000 7f3a27fff700 1 -- 172.21.15.60:0/3129596397 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a2800f5f0 con 0x7f3a3813f790 2022-06-29T16:00:21.845 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.827+0000 7f3a27fff700 1 -- 172.21.15.60:0/3129596397 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a28000e90 con 0x7f3a3813f790 2022-06-29T16:00:21.845 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.827+0000 7f3a27fff700 1 -- 172.21.15.60:0/3129596397 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a2800f940 con 0x7f3a3813f790 2022-06-29T16:00:21.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.827+0000 7f3a3f6d2700 1 -- 172.21.15.60:0/3129596397 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3a3813f790 msgr2=0x7f3a3813bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:21.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.827+0000 7f3a3f6d2700 1 --2- 172.21.15.60:0/3129596397 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3a3813f790 0x7f3a3813bc40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f3a28002f70 tx=0x7f3a28006e80).stop 2022-06-29T16:00:21.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.827+0000 7f3a3f6d2700 1 -- 172.21.15.60:0/3129596397 shutdown_connections 2022-06-29T16:00:21.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.827+0000 7f3a3f6d2700 1 --2- 172.21.15.60:0/3129596397 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3a3813f790 0x7f3a3813bc40 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:21.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.827+0000 7f3a3f6d2700 1 --2- 172.21.15.60:0/3129596397 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3a380b61d0 0x7f3a380b0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:21.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.827+0000 7f3a3f6d2700 1 -- 172.21.15.60:0/3129596397 >> 172.21.15.60:0/3129596397 conn(0x7f3a38063b50 msgr2=0x7f3a380b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:21.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.827+0000 7f3a3f6d2700 1 -- 172.21.15.60:0/3129596397 shutdown_connections 2022-06-29T16:00:21.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.827+0000 7f3a3f6d2700 1 -- 172.21.15.60:0/3129596397 wait complete. 2022-06-29T16:00:21.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.831+0000 7f3a3f6d2700 1 Processor -- start 2022-06-29T16:00:21.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.831+0000 7f3a3f6d2700 1 -- start start 2022-06-29T16:00:21.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.831+0000 7f3a3f6d2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3a380b61d0 0x7f3a38151c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:21.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.831+0000 7f3a3f6d2700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3a3813f790 0x7f3a38141550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:21.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.831+0000 7f3a3f6d2700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3a3813fbd0 0x7f3a3813b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:21.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.831+0000 7f3a3f6d2700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a38113220 con 0x7f3a380b61d0 2022-06-29T16:00:21.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.831+0000 7f3a3f6d2700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a381134c0 con 0x7f3a3813fbd0 2022-06-29T16:00:21.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.831+0000 7f3a3f6d2700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a38113760 con 0x7f3a3813f790 2022-06-29T16:00:21.859 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.831+0000 7f3a3d46e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3a380b61d0 0x7f3a38151c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:21.859 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.831+0000 7f3a3d46e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3a380b61d0 0x7f3a38151c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47140/0 (socket says 172.21.15.60:47140) 2022-06-29T16:00:21.859 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.831+0000 7f3a3d46e700 1 -- 172.21.15.60:0/962426952 learned_addr learned my addr 172.21.15.60:0/962426952 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:21.860 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.831+0000 7f3a3cc6d700 1 --2- 172.21.15.60:0/962426952 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3a3813f790 0x7f3a38141550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:21.860 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.831+0000 7f3a3dc6f700 1 --2- 172.21.15.60:0/962426952 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3a3813fbd0 0x7f3a3813b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:21.861 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.831+0000 7f3a3d46e700 1 -- 172.21.15.60:0/962426952 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3a3813f790 msgr2=0x7f3a38141550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:21.861 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.831+0000 7f3a3d46e700 1 --2- 172.21.15.60:0/962426952 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3a3813f790 0x7f3a38141550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:21.862 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.831+0000 7f3a3d46e700 1 -- 172.21.15.60:0/962426952 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3a3813fbd0 msgr2=0x7f3a3813b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:21.862 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.831+0000 7f3a3d46e700 1 --2- 172.21.15.60:0/962426952 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3a3813fbd0 0x7f3a3813b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:21.862 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.831+0000 7f3a3d46e700 1 -- 172.21.15.60:0/962426952 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a2800b2c0 con 0x7f3a380b61d0 2022-06-29T16:00:21.863 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.831+0000 7f3a3d46e700 1 --2- 172.21.15.60:0/962426952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3a380b61d0 0x7f3a38151c10 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f3a3400e000 tx=0x7f3a3400a040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:21.863 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.831+0000 7f3a267fc700 1 -- 172.21.15.60:0/962426952 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a340072c0 con 0x7f3a380b61d0 2022-06-29T16:00:21.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.831+0000 7f3a267fc700 1 -- 172.21.15.60:0/962426952 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a34007420 con 0x7f3a380b61d0 2022-06-29T16:00:21.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.831+0000 7f3a3f6d2700 1 -- 172.21.15.60:0/962426952 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a380af390 con 0x7f3a380b61d0 2022-06-29T16:00:21.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.831+0000 7f3a267fc700 1 -- 172.21.15.60:0/962426952 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a34005af0 con 0x7f3a380b61d0 2022-06-29T16:00:21.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.831+0000 7f3a3f6d2700 1 -- 172.21.15.60:0/962426952 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a38112e00 con 0x7f3a380b61d0 2022-06-29T16:00:21.866 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.835+0000 7f3a3f6d2700 1 -- 172.21.15.60:0/962426952 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a38065c10 con 0x7f3a380b61d0 2022-06-29T16:00:21.866 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.835+0000 7f3a267fc700 1 -- 172.21.15.60:0/962426952 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3a34005c50 con 0x7f3a380b61d0 2022-06-29T16:00:21.866 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.835+0000 7f3a267fc700 1 --2- 172.21.15.60:0/962426952 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3a18053aa0 0x7f3a18055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:21.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.835+0000 7f3a267fc700 1 -- 172.21.15.60:0/962426952 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3a3406c380 con 0x7f3a380b61d0 2022-06-29T16:00:21.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.835+0000 7f3a3cc6d700 1 --2- 172.21.15.60:0/962426952 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3a18053aa0 0x7f3a18055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:21.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.835+0000 7f3a3cc6d700 1 --2- 172.21.15.60:0/962426952 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3a18053aa0 0x7f3a18055f60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f3a2801b000 tx=0x7f3a3813f1e0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:21.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:21.839+0000 7f3a267fc700 1 -- 172.21.15.60:0/962426952 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3a38065c10 con 0x7f3a380b61d0 2022-06-29T16:00:22.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.011+0000 7f3a3f6d2700 1 -- 172.21.15.60:0/962426952 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a38065c10 con 0x7f3a380b61d0 2022-06-29T16:00:22.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.015+0000 7f3a267fc700 1 -- 172.21.15.60:0/962426952 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+2138 (secure 0 0 0) 0x7f3a38065c10 con 0x7f3a380b61d0 2022-06-29T16:00:22.022 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:00:22.023 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":15,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4937,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/1673184114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1673184114},{"type":"v1","addr":"172.21.15.92:6835","nonce":1673184114}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":13}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:00:20.685676+0000","modified":"2022-06-29T16:00:21.689299+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5308":{"gid":5308,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/1964426267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1964426267},{"type":"v1","addr":"172.21.15.60:6833","nonce":1964426267}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-06-29T16:00:22.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.015+0000 7f3a3f6d2700 1 -- 172.21.15.60:0/962426952 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3a18053aa0 msgr2=0x7f3a18055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:22.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.015+0000 7f3a3f6d2700 1 --2- 172.21.15.60:0/962426952 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3a18053aa0 0x7f3a18055f60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f3a2801b000 tx=0x7f3a3813f1e0).stop 2022-06-29T16:00:22.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.015+0000 7f3a3f6d2700 1 -- 172.21.15.60:0/962426952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3a380b61d0 msgr2=0x7f3a38151c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:22.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.015+0000 7f3a3f6d2700 1 --2- 172.21.15.60:0/962426952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3a380b61d0 0x7f3a38151c10 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f3a3400e000 tx=0x7f3a3400a040).stop 2022-06-29T16:00:22.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.015+0000 7f3a3f6d2700 1 -- 172.21.15.60:0/962426952 shutdown_connections 2022-06-29T16:00:22.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.015+0000 7f3a3f6d2700 1 --2- 172.21.15.60:0/962426952 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3a18053aa0 0x7f3a18055f60 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:22.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.015+0000 7f3a3f6d2700 1 --2- 172.21.15.60:0/962426952 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3a3813fbd0 0x7f3a3813b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:22.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.015+0000 7f3a3f6d2700 1 --2- 172.21.15.60:0/962426952 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3a3813f790 0x7f3a38141550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:22.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.015+0000 7f3a3f6d2700 1 --2- 172.21.15.60:0/962426952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3a380b61d0 0x7f3a38151c10 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:22.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.015+0000 7f3a3f6d2700 1 -- 172.21.15.60:0/962426952 >> 172.21.15.60:0/962426952 conn(0x7f3a38063b50 msgr2=0x7f3a380b5690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:22.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.015+0000 7f3a3f6d2700 1 -- 172.21.15.60:0/962426952 shutdown_connections 2022-06-29T16:00:22.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.015+0000 7f3a3f6d2700 1 -- 172.21.15.60:0/962426952 wait complete. 2022-06-29T16:00:22.027 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 15 2022-06-29T16:00:22.036 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:00:22.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.131+0000 7fb4f3fff700 1 Processor -- start 2022-06-29T16:00:22.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.131+0000 7fb4f3fff700 1 -- start start 2022-06-29T16:00:22.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.131+0000 7fb4f3fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb4f40aed00 0x7fb4f4146810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:22.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.131+0000 7fb4f3fff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4f405d450 con 0x7fb4f405d070 2022-06-29T16:00:22.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.131+0000 7fb4f3fff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4f40b25d0 con 0x7fb4f40b2130 2022-06-29T16:00:22.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.131+0000 7fb4f3fff700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4f40adf90 con 0x7fb4f40aed00 2022-06-29T16:00:22.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.131+0000 7fb4f37fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb4f40aed00 0x7fb4f4146810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:22.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.131+0000 7fb4f2ffd700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb4f405d070 0x7fb4f414aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55234/0 (socket says 172.21.15.60:55234) 2022-06-29T16:00:22.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.131+0000 7fb4f2ffd700 1 -- 172.21.15.60:0/2831117456 learned_addr learned my addr 172.21.15.60:0/2831117456 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:00:22.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.131+0000 7fb4f37fe700 1 -- 172.21.15.60:0/2831117456 >> v1:172.21.15.92:6789/0 conn(0x7fb4f40b2130 legacy=0x7fb4f4146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:22.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.131+0000 7fb4f37fe700 1 -- 172.21.15.60:0/2831117456 >> v1:172.21.15.60:6789/0 conn(0x7fb4f405d070 legacy=0x7fb4f414aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:22.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.131+0000 7fb4f37fe700 1 -- 172.21.15.60:0/2831117456 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f414a350 con 0x7fb4f40aed00 2022-06-29T16:00:22.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.131+0000 7fb4f37fe700 1 --2- 172.21.15.60:0/2831117456 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4f40aed00 0x7fb4f4146810 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fb4ec002f70 tx=0x7fb4ec006df0).ready entity=mon.0 client_cookie=9388c638a2fa98b5 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:22.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.131+0000 7fb4f1ffb700 1 -- 172.21.15.60:0/2831117456 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4ec00d610 con 0x7fb4f40aed00 2022-06-29T16:00:22.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.131+0000 7fb4f1ffb700 1 -- 172.21.15.60:0/2831117456 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4ec000e90 con 0x7fb4f40aed00 2022-06-29T16:00:22.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.131+0000 7fb4f1ffb700 1 -- 172.21.15.60:0/2831117456 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4ec00d960 con 0x7fb4f40aed00 2022-06-29T16:00:22.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.131+0000 7fb4f3fff700 1 -- 172.21.15.60:0/2831117456 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4f40aed00 msgr2=0x7fb4f4146810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:22.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.131+0000 7fb4f3fff700 1 --2- 172.21.15.60:0/2831117456 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4f40aed00 0x7fb4f4146810 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fb4ec002f70 tx=0x7fb4ec006df0).stop 2022-06-29T16:00:22.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.131+0000 7fb4f3fff700 1 -- 172.21.15.60:0/2831117456 shutdown_connections 2022-06-29T16:00:22.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.131+0000 7fb4f3fff700 1 --2- 172.21.15.60:0/2831117456 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4f40aed00 0x7fb4f4146810 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:22.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.131+0000 7fb4f3fff700 1 -- 172.21.15.60:0/2831117456 >> 172.21.15.60:0/2831117456 conn(0x7fb4f40b5890 msgr2=0x7fb4f40b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:22.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.135+0000 7fb4f3fff700 1 -- 172.21.15.60:0/2831117456 shutdown_connections 2022-06-29T16:00:22.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.135+0000 7fb4f3fff700 1 -- 172.21.15.60:0/2831117456 wait complete. 2022-06-29T16:00:22.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.135+0000 7fb4f3fff700 1 Processor -- start 2022-06-29T16:00:22.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.135+0000 7fb4f3fff700 1 -- start start 2022-06-29T16:00:22.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.135+0000 7fb4f3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4f405d070 0x7fb4f4116b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:22.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.135+0000 7fb4f3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4f40aed00 0x7fb4f4156650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:22.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.135+0000 7fb4f3fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb4f40b2130 0x7fb4f4156b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:22.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.135+0000 7fb4f3fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f4066cd0 con 0x7fb4f405d070 2022-06-29T16:00:22.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.135+0000 7fb4f3fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f4065960 con 0x7fb4f40aed00 2022-06-29T16:00:22.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.135+0000 7fb4f3fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f4065b90 con 0x7fb4f40b2130 2022-06-29T16:00:22.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.139+0000 7fb4f2ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4f405d070 0x7fb4f4116b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:22.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.139+0000 7fb4f2ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4f405d070 0x7fb4f4116b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47154/0 (socket says 172.21.15.60:47154) 2022-06-29T16:00:22.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.139+0000 7fb4f2ffd700 1 -- 172.21.15.60:0/3632295933 learned_addr learned my addr 172.21.15.60:0/3632295933 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:22.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.139+0000 7fb4f27fc700 1 --2- 172.21.15.60:0/3632295933 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4f40aed00 0x7fb4f4156650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:22.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.139+0000 7fb4f37fe700 1 --2- 172.21.15.60:0/3632295933 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb4f40b2130 0x7fb4f4156b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:22.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.139+0000 7fb4f2ffd700 1 -- 172.21.15.60:0/3632295933 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb4f40b2130 msgr2=0x7fb4f4156b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:22.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.139+0000 7fb4f2ffd700 1 --2- 172.21.15.60:0/3632295933 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb4f40b2130 0x7fb4f4156b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:22.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.139+0000 7fb4f2ffd700 1 -- 172.21.15.60:0/3632295933 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4f40aed00 msgr2=0x7fb4f4156650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:22.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.139+0000 7fb4f2ffd700 1 --2- 172.21.15.60:0/3632295933 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4f40aed00 0x7fb4f4156650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:22.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.139+0000 7fb4f2ffd700 1 -- 172.21.15.60:0/3632295933 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ec00b180 con 0x7fb4f405d070 2022-06-29T16:00:22.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.139+0000 7fb4f2ffd700 1 --2- 172.21.15.60:0/3632295933 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4f405d070 0x7fb4f4116b50 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fb4f4118570 tx=0x7fb4dc007c40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:22.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.139+0000 7fb4ebfff700 1 -- 172.21.15.60:0/3632295933 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4dc0070d0 con 0x7fb4f405d070 2022-06-29T16:00:22.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.139+0000 7fb4ebfff700 1 -- 172.21.15.60:0/3632295933 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4dc007230 con 0x7fb4f405d070 2022-06-29T16:00:22.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.139+0000 7fb4f3fff700 1 -- 172.21.15.60:0/3632295933 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4f4066950 con 0x7fb4f405d070 2022-06-29T16:00:22.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.139+0000 7fb4ebfff700 1 -- 172.21.15.60:0/3632295933 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4dc005610 con 0x7fb4f405d070 2022-06-29T16:00:22.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.139+0000 7fb4f3fff700 1 -- 172.21.15.60:0/3632295933 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4f4064a50 con 0x7fb4f405d070 2022-06-29T16:00:22.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.139+0000 7fb4f3fff700 1 -- 172.21.15.60:0/3632295933 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4f414a5f0 con 0x7fb4f405d070 2022-06-29T16:00:22.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.139+0000 7fb4ebfff700 1 -- 172.21.15.60:0/3632295933 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb4dc010070 con 0x7fb4f405d070 2022-06-29T16:00:22.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.143+0000 7fb4ebfff700 1 --2- 172.21.15.60:0/3632295933 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb4d8053b00 0x7fb4d8055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:22.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.143+0000 7fb4ebfff700 1 -- 172.21.15.60:0/3632295933 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb4dc06b480 con 0x7fb4f405d070 2022-06-29T16:00:22.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.143+0000 7fb4f27fc700 1 --2- 172.21.15.60:0/3632295933 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb4d8053b00 0x7fb4d8055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:22.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.143+0000 7fb4f27fc700 1 --2- 172.21.15.60:0/3632295933 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb4d8053b00 0x7fb4d8055fc0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fb4e400d000 tx=0x7fb4e4007f10).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:22.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.147+0000 7fb4ebfff700 1 -- 172.21.15.60:0/3632295933 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb4dc042300 con 0x7fb4f405d070 2022-06-29T16:00:22.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.271+0000 7fb4f3fff700 1 -- 172.21.15.60:0/3632295933 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4f40573d0 con 0x7fb4f405d070 2022-06-29T16:00:22.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.275+0000 7fb4ebfff700 1 -- 172.21.15.60:0/3632295933 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+16051 (secure 0 0 0) 0x7fb4dc018070 con 0x7fb4f405d070 2022-06-29T16:00:22.284 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:00:22.284 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":42,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:00:21.688149+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:00:18.552380+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2022-06-29T16:00:19.688558+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:00:22.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.275+0000 7fb4f3fff700 1 -- 172.21.15.60:0/3632295933 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb4d8053b00 msgr2=0x7fb4d8055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:22.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.275+0000 7fb4f3fff700 1 --2- 172.21.15.60:0/3632295933 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb4d8053b00 0x7fb4d8055fc0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fb4e400d000 tx=0x7fb4e4007f10).stop 2022-06-29T16:00:22.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.275+0000 7fb4f3fff700 1 -- 172.21.15.60:0/3632295933 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4f405d070 msgr2=0x7fb4f4116b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:22.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.275+0000 7fb4f3fff700 1 --2- 172.21.15.60:0/3632295933 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4f405d070 0x7fb4f4116b50 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fb4f4118570 tx=0x7fb4dc007c40).stop 2022-06-29T16:00:22.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.275+0000 7fb4f3fff700 1 -- 172.21.15.60:0/3632295933 shutdown_connections 2022-06-29T16:00:22.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.279+0000 7fb4f3fff700 1 --2- 172.21.15.60:0/3632295933 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb4d8053b00 0x7fb4d8055fc0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:22.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.279+0000 7fb4f3fff700 1 --2- 172.21.15.60:0/3632295933 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb4f40b2130 0x7fb4f4156b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:22.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.279+0000 7fb4f3fff700 1 --2- 172.21.15.60:0/3632295933 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4f40aed00 0x7fb4f4156650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:22.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.279+0000 7fb4f3fff700 1 --2- 172.21.15.60:0/3632295933 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4f405d070 0x7fb4f4116b50 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:22.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.279+0000 7fb4f3fff700 1 -- 172.21.15.60:0/3632295933 >> 172.21.15.60:0/3632295933 conn(0x7fb4f40b5890 msgr2=0x7fb4f4145480 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:22.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.279+0000 7fb4f3fff700 1 -- 172.21.15.60:0/3632295933 shutdown_connections 2022-06-29T16:00:22.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.279+0000 7fb4f3fff700 1 -- 172.21.15.60:0/3632295933 wait complete. 2022-06-29T16:00:22.298 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T16:00:22.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.403+0000 7f341045f700 1 Processor -- start 2022-06-29T16:00:22.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.403+0000 7f341045f700 1 -- start start 2022-06-29T16:00:22.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.403+0000 7f341045f700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3408060710 0x7f34080b6270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:22.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.403+0000 7f341045f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f340813ec70 0x7f340813f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:22.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.403+0000 7f341045f700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3408060c60 con 0x7f34080b5e90 2022-06-29T16:00:22.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.403+0000 7f341045f700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34080af0c0 con 0x7f3408060710 2022-06-29T16:00:22.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.403+0000 7f341045f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34080b1d30 con 0x7f340813ec70 2022-06-29T16:00:22.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.403+0000 7f340e1fb700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f34080b5e90 0x7f34080ae9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55248/0 (socket says 172.21.15.60:55248) 2022-06-29T16:00:22.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.403+0000 7f340e1fb700 1 -- 172.21.15.60:0/3803789441 learned_addr learned my addr 172.21.15.60:0/3803789441 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:00:22.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.403+0000 7f340d9fa700 1 --2- 172.21.15.60:0/3803789441 >> v2:172.21.15.92:3300/0 conn(0x7f3408060710 0x7f34080b6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:22.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.403+0000 7f340e9fc700 1 --2- 172.21.15.60:0/3803789441 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f340813ec70 0x7f340813f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:22.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.403+0000 7f340d1f9700 1 -- 172.21.15.60:0/3803789441 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2530966109 0 0) 0x7f3408060c60 con 0x7f34080b5e90 2022-06-29T16:00:22.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.403+0000 7f340d1f9700 1 -- 172.21.15.60:0/3803789441 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33f0003260 con 0x7f34080b5e90 2022-06-29T16:00:22.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.403+0000 7f340d1f9700 1 -- 172.21.15.60:0/3803789441 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3189495774 0 0) 0x7f33f0003260 con 0x7f34080b5e90 2022-06-29T16:00:22.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.403+0000 7f340d1f9700 1 -- 172.21.15.60:0/3803789441 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f340813ec70 msgr2=0x7f340813f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:22.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.403+0000 7f340d1f9700 1 --2- 172.21.15.60:0/3803789441 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f340813ec70 0x7f340813f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:22.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.403+0000 7f340d1f9700 1 -- 172.21.15.60:0/3803789441 >> v2:172.21.15.92:3300/0 conn(0x7f3408060710 msgr2=0x7f34080b6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:22.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.403+0000 7f340d1f9700 1 --2- 172.21.15.60:0/3803789441 >> v2:172.21.15.92:3300/0 conn(0x7f3408060710 0x7f34080b6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:22.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.407+0000 7f340d1f9700 1 -- 172.21.15.60:0/3803789441 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34080ae2e0 con 0x7f34080b5e90 2022-06-29T16:00:22.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.407+0000 7f340d1f9700 1 -- 172.21.15.60:0/3803789441 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f34040019e0 con 0x7f34080b5e90 2022-06-29T16:00:22.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.407+0000 7f340d1f9700 1 -- 172.21.15.60:0/3803789441 >> v1:172.21.15.60:6789/0 conn(0x7f34080b5e90 legacy=0x7f34080ae9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:22.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.407+0000 7f340d1f9700 1 --2- 172.21.15.60:0/3803789441 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f340813ec70 0x7f33f0003750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:22.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.407+0000 7f340d1f9700 1 --2- 172.21.15.60:0/3803789441 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33f0003c80 0x7f33f0004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:22.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.407+0000 7f340d1f9700 1 --2- 172.21.15.60:0/3803789441 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f33f00049f0 0x7f33f0004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:22.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.407+0000 7f340d1f9700 1 -- 172.21.15.60:0/3803789441 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34080af0c0 con 0x7f33f0003c80 2022-06-29T16:00:22.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.407+0000 7f340d1f9700 1 -- 172.21.15.60:0/3803789441 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34080b1d30 con 0x7f33f00049f0 2022-06-29T16:00:22.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.407+0000 7f340d1f9700 1 -- 172.21.15.60:0/3803789441 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33f000b750 con 0x7f340813ec70 2022-06-29T16:00:22.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.407+0000 7f340d9fa700 1 --2- 172.21.15.60:0/3803789441 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33f0003c80 0x7f33f0004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:22.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.407+0000 7f340e1fb700 1 --2- 172.21.15.60:0/3803789441 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f340813ec70 0x7f33f0003750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:22.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.407+0000 7f340e9fc700 1 --2- 172.21.15.60:0/3803789441 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f33f00049f0 0x7f33f0004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:22.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.407+0000 7f340d9fa700 1 -- 172.21.15.60:0/3803789441 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f340813ec70 msgr2=0x7f33f0003750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:22.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.407+0000 7f340d9fa700 1 --2- 172.21.15.60:0/3803789441 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f340813ec70 0x7f33f0003750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:22.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.407+0000 7f340d9fa700 1 -- 172.21.15.60:0/3803789441 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f33f00049f0 msgr2=0x7f33f0004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:22.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.407+0000 7f340d9fa700 1 --2- 172.21.15.60:0/3803789441 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f33f00049f0 0x7f33f0004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:22.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.407+0000 7f340d9fa700 1 -- 172.21.15.60:0/3803789441 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f0003260 con 0x7f33f0003c80 2022-06-29T16:00:22.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.407+0000 7f340d9fa700 1 --2- 172.21.15.60:0/3803789441 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33f0003c80 0x7f33f0004500 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f33f00054d0 tx=0x7f33f8006df0).ready entity=mon.0 client_cookie=e7f4bffa4b002c54 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:22.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.407+0000 7f340d1f9700 1 -- 172.21.15.60:0/3803789441 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f33f800e830 con 0x7f33f0003c80 2022-06-29T16:00:22.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.407+0000 7f340d1f9700 1 -- 172.21.15.60:0/3803789441 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33f800be90 con 0x7f33f0003c80 2022-06-29T16:00:22.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.407+0000 7f340d1f9700 1 -- 172.21.15.60:0/3803789441 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f33f800eb80 con 0x7f33f0003c80 2022-06-29T16:00:22.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.407+0000 7f341045f700 1 -- 172.21.15.60:0/3803789441 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33f0003c80 msgr2=0x7f33f0004500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:22.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.407+0000 7f341045f700 1 --2- 172.21.15.60:0/3803789441 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f33f0003c80 0x7f33f0004500 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f33f00054d0 tx=0x7f33f8006df0).stop 2022-06-29T16:00:22.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.407+0000 7f340e9fc700 1 -- 172.21.15.60:0/3803789441 reap_dead start 2022-06-29T16:00:22.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.407+0000 7f341045f700 1 -- 172.21.15.60:0/3803789441 shutdown_connections 2022-06-29T16:00:22.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.407+0000 7f341045f700 1 -- 172.21.15.60:0/3803789441 >> 172.21.15.60:0/3803789441 conn(0x7f3408063b50 msgr2=0x7f34080b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:22.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.407+0000 7f341045f700 1 -- 172.21.15.60:0/3803789441 shutdown_connections 2022-06-29T16:00:22.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.407+0000 7f341045f700 1 -- 172.21.15.60:0/3803789441 wait complete. 2022-06-29T16:00:22.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.411+0000 7f341045f700 1 Processor -- start 2022-06-29T16:00:22.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.411+0000 7f341045f700 1 -- start start 2022-06-29T16:00:22.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.411+0000 7f341045f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f340805f3f0 0x7f340814fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:22.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.411+0000 7f341045f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3408143860 0x7f3408153520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:22.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.411+0000 7f341045f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3408142bf0 0x7f3408153a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:22.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.411+0000 7f341045f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34081530f0 con 0x7f340805f3f0 2022-06-29T16:00:22.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.411+0000 7f341045f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340805f7d0 con 0x7f3408142bf0 2022-06-29T16:00:22.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.411+0000 7f341045f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34081425c0 con 0x7f3408143860 2022-06-29T16:00:22.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.411+0000 7f340e1fb700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f340805f3f0 0x7f340814fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:22.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.411+0000 7f340e1fb700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f340805f3f0 0x7f340814fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47174/0 (socket says 172.21.15.60:47174) 2022-06-29T16:00:22.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.411+0000 7f340e1fb700 1 -- 172.21.15.60:0/3425748390 learned_addr learned my addr 172.21.15.60:0/3425748390 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:22.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.411+0000 7f340e9fc700 1 --2- 172.21.15.60:0/3425748390 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3408142bf0 0x7f3408153a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:22.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.411+0000 7f340d9fa700 1 --2- 172.21.15.60:0/3425748390 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3408143860 0x7f3408153520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:22.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.411+0000 7f340e1fb700 1 -- 172.21.15.60:0/3425748390 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3408143860 msgr2=0x7f3408153520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:22.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.411+0000 7f340e1fb700 1 --2- 172.21.15.60:0/3425748390 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3408143860 0x7f3408153520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:22.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.411+0000 7f340e1fb700 1 -- 172.21.15.60:0/3425748390 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3408142bf0 msgr2=0x7f3408153a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:22.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.411+0000 7f340e1fb700 1 --2- 172.21.15.60:0/3425748390 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3408142bf0 0x7f3408153a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:22.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.411+0000 7f340e1fb700 1 -- 172.21.15.60:0/3425748390 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3408153f00 con 0x7f340805f3f0 2022-06-29T16:00:22.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.411+0000 7f340e1fb700 1 --2- 172.21.15.60:0/3425748390 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f340805f3f0 0x7f340814fab0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f340400d000 tx=0x7f34080af200).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:22.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.411+0000 7f33f77fe700 1 -- 172.21.15.60:0/3425748390 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f340400f8e0 con 0x7f340805f3f0 2022-06-29T16:00:22.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.411+0000 7f33f77fe700 1 -- 172.21.15.60:0/3425748390 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f340400fa40 con 0x7f340805f3f0 2022-06-29T16:00:22.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.411+0000 7f341045f700 1 -- 172.21.15.60:0/3425748390 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3408142860 con 0x7f340805f3f0 2022-06-29T16:00:22.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.411+0000 7f33f77fe700 1 -- 172.21.15.60:0/3425748390 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f340400fda0 con 0x7f340805f3f0 2022-06-29T16:00:22.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.415+0000 7f341045f700 1 -- 172.21.15.60:0/3425748390 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3408066280 con 0x7f340805f3f0 2022-06-29T16:00:22.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.415+0000 7f33f77fe700 1 -- 172.21.15.60:0/3425748390 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f340401b030 con 0x7f340805f3f0 2022-06-29T16:00:22.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.415+0000 7f33f77fe700 1 --2- 172.21.15.60:0/3425748390 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f33e8053a70 0x7f33e8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:22.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.415+0000 7f33f77fe700 1 -- 172.21.15.60:0/3425748390 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f340406bd70 con 0x7f340805f3f0 2022-06-29T16:00:22.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.415+0000 7f340d9fa700 1 --2- 172.21.15.60:0/3425748390 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f33e8053a70 0x7f33e8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:22.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.415+0000 7f341045f700 1 -- 172.21.15.60:0/3425748390 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3408153280 con 0x7f340805f3f0 2022-06-29T16:00:22.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.415+0000 7f340d9fa700 1 --2- 172.21.15.60:0/3425748390 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f33e8053a70 0x7f33e8055f30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f33f801d000 tx=0x7f33f8019000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:22.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.423+0000 7f33f77fe700 1 -- 172.21.15.60:0/3425748390 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3408153280 con 0x7f340805f3f0 2022-06-29T16:00:22.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.587+0000 7f341045f700 1 -- 172.21.15.60:0/3425748390 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f3408153280 con 0x7f340805f3f0 2022-06-29T16:00:22.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.595+0000 7f33f77fe700 1 -- 172.21.15.60:0/3425748390 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v16) v1 ==== 184+0+0 (secure 0 0 0) 0x7f3408153280 con 0x7f340805f3f0 2022-06-29T16:00:22.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.599+0000 7f341045f700 1 -- 172.21.15.60:0/3425748390 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f33e8053a70 msgr2=0x7f33e8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:22.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.599+0000 7f341045f700 1 --2- 172.21.15.60:0/3425748390 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f33e8053a70 0x7f33e8055f30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f33f801d000 tx=0x7f33f8019000).stop 2022-06-29T16:00:22.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.599+0000 7f341045f700 1 -- 172.21.15.60:0/3425748390 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f340805f3f0 msgr2=0x7f340814fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:22.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.599+0000 7f341045f700 1 --2- 172.21.15.60:0/3425748390 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f340805f3f0 0x7f340814fab0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f340400d000 tx=0x7f34080af200).stop 2022-06-29T16:00:22.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.599+0000 7f341045f700 1 -- 172.21.15.60:0/3425748390 shutdown_connections 2022-06-29T16:00:22.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.599+0000 7f341045f700 1 --2- 172.21.15.60:0/3425748390 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f33e8053a70 0x7f33e8055f30 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:22.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.599+0000 7f341045f700 1 --2- 172.21.15.60:0/3425748390 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3408142bf0 0x7f3408153a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:22.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.599+0000 7f341045f700 1 --2- 172.21.15.60:0/3425748390 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3408143860 0x7f3408153520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:22.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.599+0000 7f341045f700 1 --2- 172.21.15.60:0/3425748390 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f340805f3f0 0x7f340814fab0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:22.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.599+0000 7f341045f700 1 -- 172.21.15.60:0/3425748390 >> 172.21.15.60:0/3425748390 conn(0x7f3408063b50 msgr2=0x7f34080aca20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:22.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.599+0000 7f341045f700 1 -- 172.21.15.60:0/3425748390 shutdown_connections 2022-06-29T16:00:22.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.599+0000 7f341045f700 1 -- 172.21.15.60:0/3425748390 wait complete. 2022-06-29T16:00:22.610 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.0 2022-06-29T16:00:22.621 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T16:00:22.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.723+0000 7f77805ab700 1 Processor -- start 2022-06-29T16:00:22.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.723+0000 7f77805ab700 1 -- start start 2022-06-29T16:00:22.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.723+0000 7f77805ab700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f77780b61d0 0x7f77780af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:22.734 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.723+0000 7f77805ab700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77780b65b0 con 0x7f77780adc80 2022-06-29T16:00:22.734 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.723+0000 7f77805ab700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77780605e0 con 0x7f7778060b60 2022-06-29T16:00:22.734 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.723+0000 7f77805ab700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7778060880 con 0x7f77780b61d0 2022-06-29T16:00:22.734 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.723+0000 7f777db46700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f77780adc80 0x7f7778142a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55270/0 (socket says 172.21.15.60:55270) 2022-06-29T16:00:22.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.723+0000 7f777db46700 1 -- 172.21.15.60:0/1452161743 learned_addr learned my addr 172.21.15.60:0/1452161743 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:00:22.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.723+0000 7f777e347700 1 --2- 172.21.15.60:0/1452161743 >> v2:172.21.15.92:3300/0 conn(0x7f77780b61d0 0x7f77780af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:22.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.723+0000 7f777d345700 1 -- 172.21.15.60:0/1452161743 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3909623567 0 0) 0x7f77780b65b0 con 0x7f77780adc80 2022-06-29T16:00:22.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.723+0000 7f777d345700 1 -- 172.21.15.60:0/1452161743 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7760003260 con 0x7f77780adc80 2022-06-29T16:00:22.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.723+0000 7f777e347700 1 -- 172.21.15.60:0/1452161743 >> v1:172.21.15.92:6789/0 conn(0x7f7778060b60 legacy=0x7f777813c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:22.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.723+0000 7f777d345700 1 -- 172.21.15.60:0/1452161743 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 397300843 0 0) 0x7f7760003260 con 0x7f77780adc80 2022-06-29T16:00:22.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.723+0000 7f777e347700 1 -- 172.21.15.60:0/1452161743 >> v1:172.21.15.60:6789/0 conn(0x7f77780adc80 legacy=0x7f7778142a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:22.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.723+0000 7f777e347700 1 -- 172.21.15.60:0/1452161743 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7778067340 con 0x7f77780b61d0 2022-06-29T16:00:22.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.723+0000 7f777e347700 1 --2- 172.21.15.60:0/1452161743 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77780b61d0 0x7f77780af8d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f77780ad110 tx=0x7f7774006df0).ready entity=mon.1 client_cookie=53fec86aa835cb0a server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:22.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.723+0000 7f777d345700 1 -- 172.21.15.60:0/1452161743 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7768001660 con 0x7f77780b61d0 2022-06-29T16:00:22.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.727+0000 7f777d345700 1 -- 172.21.15.60:0/1452161743 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7774000e90 con 0x7f77780b61d0 2022-06-29T16:00:22.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.727+0000 7f777d345700 1 -- 172.21.15.60:0/1452161743 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f777400d4f0 con 0x7f77780b61d0 2022-06-29T16:00:22.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.727+0000 7f77805ab700 1 -- 172.21.15.60:0/1452161743 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77780b61d0 msgr2=0x7f77780af8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:22.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.727+0000 7f77805ab700 1 --2- 172.21.15.60:0/1452161743 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77780b61d0 0x7f77780af8d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f77780ad110 tx=0x7f7774006df0).stop 2022-06-29T16:00:22.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.727+0000 7f77805ab700 1 -- 172.21.15.60:0/1452161743 shutdown_connections 2022-06-29T16:00:22.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.727+0000 7f77805ab700 1 --2- 172.21.15.60:0/1452161743 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77780b61d0 0x7f77780af8d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:22.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.727+0000 7f77805ab700 1 -- 172.21.15.60:0/1452161743 >> 172.21.15.60:0/1452161743 conn(0x7f7778063b50 msgr2=0x7f77780b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:22.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.727+0000 7f77805ab700 1 -- 172.21.15.60:0/1452161743 shutdown_connections 2022-06-29T16:00:22.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.727+0000 7f77805ab700 1 -- 172.21.15.60:0/1452161743 wait complete. 2022-06-29T16:00:22.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.727+0000 7f77805ab700 1 Processor -- start 2022-06-29T16:00:22.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.727+0000 7f77805ab700 1 -- start start 2022-06-29T16:00:22.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.731+0000 7f77805ab700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7778060b60 0x7f77781464a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:22.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.731+0000 7f77805ab700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77780adc80 0x7f7778146990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:22.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.731+0000 7f77805ab700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77780b61d0 0x7f7778144590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:22.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.731+0000 7f77805ab700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7778065660 con 0x7f77780adc80 2022-06-29T16:00:22.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.731+0000 7f77805ab700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7778065880 con 0x7f77780b61d0 2022-06-29T16:00:22.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.731+0000 7f77805ab700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7778065b20 con 0x7f7778060b60 2022-06-29T16:00:22.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.731+0000 7f777db46700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77780adc80 0x7f7778146990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:22.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.731+0000 7f777db46700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77780adc80 0x7f7778146990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47190/0 (socket says 172.21.15.60:47190) 2022-06-29T16:00:22.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.731+0000 7f777e347700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7778060b60 0x7f77781464a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:22.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.731+0000 7f777db46700 1 -- 172.21.15.60:0/219562825 learned_addr learned my addr 172.21.15.60:0/219562825 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:22.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.731+0000 7f777eb48700 1 --2- 172.21.15.60:0/219562825 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77780b61d0 0x7f7778144590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:22.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.731+0000 7f777db46700 1 -- 172.21.15.60:0/219562825 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7778060b60 msgr2=0x7f77781464a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:22.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.731+0000 7f777db46700 1 --2- 172.21.15.60:0/219562825 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7778060b60 0x7f77781464a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:22.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.731+0000 7f777db46700 1 -- 172.21.15.60:0/219562825 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77780b61d0 msgr2=0x7f7778144590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:22.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.731+0000 7f777db46700 1 --2- 172.21.15.60:0/219562825 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77780b61d0 0x7f7778144590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:22.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.731+0000 7f777db46700 1 -- 172.21.15.60:0/219562825 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f777400b180 con 0x7f77780adc80 2022-06-29T16:00:22.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.731+0000 7f777db46700 1 --2- 172.21.15.60:0/219562825 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77780adc80 0x7f7778146990 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f77680018c0 tx=0x7f7768001e60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:22.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.731+0000 7f77677fe700 1 -- 172.21.15.60:0/219562825 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7768007170 con 0x7f77780adc80 2022-06-29T16:00:22.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.731+0000 7f77677fe700 1 -- 172.21.15.60:0/219562825 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77680072d0 con 0x7f77780adc80 2022-06-29T16:00:22.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.731+0000 7f77805ab700 1 -- 172.21.15.60:0/219562825 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7778146e80 con 0x7f77780adc80 2022-06-29T16:00:22.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.731+0000 7f77677fe700 1 -- 172.21.15.60:0/219562825 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7768004710 con 0x7f77780adc80 2022-06-29T16:00:22.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.731+0000 7f77805ab700 1 -- 172.21.15.60:0/219562825 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77780645a0 con 0x7f77780adc80 2022-06-29T16:00:22.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.731+0000 7f77805ab700 1 -- 172.21.15.60:0/219562825 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7778142530 con 0x7f77780adc80 2022-06-29T16:00:22.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.731+0000 7f77677fe700 1 -- 172.21.15.60:0/219562825 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f776801b070 con 0x7f77780adc80 2022-06-29T16:00:22.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.735+0000 7f77677fe700 1 --2- 172.21.15.60:0/219562825 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f77580539e0 0x7f7758055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:22.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.735+0000 7f77677fe700 1 -- 172.21.15.60:0/219562825 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f776806b440 con 0x7f77780adc80 2022-06-29T16:00:22.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.735+0000 7f777e347700 1 --2- 172.21.15.60:0/219562825 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f77580539e0 0x7f7758055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:22.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.735+0000 7f777e347700 1 --2- 172.21.15.60:0/219562825 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f77580539e0 0x7f7758055ea0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f777401a000 tx=0x7f7774014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:22.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.739+0000 7f77677fe700 1 -- 172.21.15.60:0/219562825 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7778142530 con 0x7f77780adc80 2022-06-29T16:00:22.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.907+0000 7f77805ab700 1 -- 172.21.15.60:0/219562825 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f77780640c0 con 0x7f77780adc80 2022-06-29T16:00:22.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.915+0000 7f77677fe700 1 -- 172.21.15.60:0/219562825 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v17) v1 ==== 184+0+0 (secure 0 0 0) 0x7f77780640c0 con 0x7f77780adc80 2022-06-29T16:00:22.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.919+0000 7f77805ab700 1 -- 172.21.15.60:0/219562825 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f77580539e0 msgr2=0x7f7758055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:22.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.919+0000 7f77805ab700 1 --2- 172.21.15.60:0/219562825 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f77580539e0 0x7f7758055ea0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f777401a000 tx=0x7f7774014000).stop 2022-06-29T16:00:22.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.919+0000 7f77805ab700 1 -- 172.21.15.60:0/219562825 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77780adc80 msgr2=0x7f7778146990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:22.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.919+0000 7f77805ab700 1 --2- 172.21.15.60:0/219562825 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77780adc80 0x7f7778146990 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f77680018c0 tx=0x7f7768001e60).stop 2022-06-29T16:00:22.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.919+0000 7f77805ab700 1 -- 172.21.15.60:0/219562825 shutdown_connections 2022-06-29T16:00:22.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.919+0000 7f77805ab700 1 --2- 172.21.15.60:0/219562825 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f77580539e0 0x7f7758055ea0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:22.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.919+0000 7f77805ab700 1 --2- 172.21.15.60:0/219562825 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77780b61d0 0x7f7778144590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:22.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.919+0000 7f77805ab700 1 --2- 172.21.15.60:0/219562825 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77780adc80 0x7f7778146990 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:22.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.919+0000 7f77805ab700 1 --2- 172.21.15.60:0/219562825 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7778060b60 0x7f77781464a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:22.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.919+0000 7f77805ab700 1 -- 172.21.15.60:0/219562825 >> 172.21.15.60:0/219562825 conn(0x7f7778063b50 msgr2=0x7f777813fb70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:22.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.919+0000 7f77805ab700 1 -- 172.21.15.60:0/219562825 shutdown_connections 2022-06-29T16:00:22.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:22.919+0000 7f77805ab700 1 -- 172.21.15.60:0/219562825 wait complete. 2022-06-29T16:00:22.930 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.1 2022-06-29T16:00:22.940 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T16:00:23.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.043+0000 7f6292ead700 1 Processor -- start 2022-06-29T16:00:23.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.043+0000 7f6292ead700 1 -- start start 2022-06-29T16:00:23.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.043+0000 7f6292ead700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f628c0b5eb0 0x7f628c142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:23.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.043+0000 7f6292ead700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f628c0b0280 0x7f628c146650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:23.058 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.043+0000 7f6292ead700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628c146fd0 con 0x7f628c146b40 2022-06-29T16:00:23.058 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.043+0000 7f6292ead700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f628c0b6290 con 0x7f628c0b5eb0 2022-06-29T16:00:23.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.043+0000 7f6292ead700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f628c0b0ab0 con 0x7f628c0b0280 2022-06-29T16:00:23.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.047+0000 7f6291eab700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f628c0b5eb0 0x7f628c142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:23.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.047+0000 7f62916aa700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f628c0b0280 0x7f628c146650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:23.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.047+0000 7f62926ac700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f628c146b40 0x7f628c0b2bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:59534/0 (socket says 172.21.15.60:59534) 2022-06-29T16:00:23.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.047+0000 7f62926ac700 1 -- 172.21.15.60:0/1686587374 learned_addr learned my addr 172.21.15.60:0/1686587374 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:00:23.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.047+0000 7f6291eab700 1 --2- 172.21.15.60:0/1686587374 >> v2:172.21.15.92:3300/0 conn(0x7f628c0b5eb0 0x7f628c142a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:53228/0 (socket says 172.21.15.60:53228) 2022-06-29T16:00:23.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.047+0000 7f62916aa700 1 -- 172.21.15.60:0/1686587374 >> v2:172.21.15.92:3300/0 conn(0x7f628c0b5eb0 msgr2=0x7f628c142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:23.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.047+0000 7f62916aa700 1 --2- 172.21.15.60:0/1686587374 >> v2:172.21.15.92:3300/0 conn(0x7f628c0b5eb0 0x7f628c142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:23.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.047+0000 7f62916aa700 1 -- 172.21.15.60:0/1686587374 >> v1:172.21.15.92:6789/0 conn(0x7f628c146b40 legacy=0x7f628c0b2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:23.063 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.047+0000 7f62916aa700 1 -- 172.21.15.60:0/1686587374 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f628c1461b0 con 0x7f628c0b0280 2022-06-29T16:00:23.063 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.047+0000 7f6291eab700 1 --2- 172.21.15.60:0/1686587374 >> v2:172.21.15.92:3300/0 conn(0x7f628c0b5eb0 0x7f628c142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:00:23.064 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.047+0000 7f62916aa700 1 --2- 172.21.15.60:0/1686587374 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f628c0b0280 0x7f628c146650 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f6288002f70 tx=0x7f6288006df0).ready entity=mon.2 client_cookie=8aa7c2fa6d46e5ae server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:23.064 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.047+0000 7f6290ea9700 1 -- 172.21.15.60:0/1686587374 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f628800d570 con 0x7f628c0b0280 2022-06-29T16:00:23.064 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.047+0000 7f6290ea9700 1 -- 172.21.15.60:0/1686587374 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6288000e90 con 0x7f628c0b0280 2022-06-29T16:00:23.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.047+0000 7f6290ea9700 1 -- 172.21.15.60:0/1686587374 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f628800d8c0 con 0x7f628c0b0280 2022-06-29T16:00:23.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.047+0000 7f6292ead700 1 -- 172.21.15.60:0/1686587374 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f628c0b0280 msgr2=0x7f628c146650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:23.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.047+0000 7f6292ead700 1 --2- 172.21.15.60:0/1686587374 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f628c0b0280 0x7f628c146650 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f6288002f70 tx=0x7f6288006df0).stop 2022-06-29T16:00:23.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.047+0000 7f6292ead700 1 -- 172.21.15.60:0/1686587374 shutdown_connections 2022-06-29T16:00:23.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.047+0000 7f6292ead700 1 --2- 172.21.15.60:0/1686587374 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f628c0b0280 0x7f628c146650 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:23.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.047+0000 7f6292ead700 1 --2- 172.21.15.60:0/1686587374 >> v2:172.21.15.92:3300/0 conn(0x7f628c0b5eb0 0x7f628c142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:23.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.047+0000 7f6292ead700 1 -- 172.21.15.60:0/1686587374 >> 172.21.15.60:0/1686587374 conn(0x7f628c062090 msgr2=0x7f628c062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:23.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.047+0000 7f6292ead700 1 -- 172.21.15.60:0/1686587374 shutdown_connections 2022-06-29T16:00:23.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.047+0000 7f6292ead700 1 -- 172.21.15.60:0/1686587374 wait complete. 2022-06-29T16:00:23.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.047+0000 7f6292ead700 1 Processor -- start 2022-06-29T16:00:23.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.047+0000 7f6292ead700 1 -- start start 2022-06-29T16:00:23.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.047+0000 7f6292ead700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f628c0b0280 0x7f628c0ae890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:23.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.047+0000 7f6292ead700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f628c0b5eb0 0x7f628c13f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:23.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.047+0000 7f6292ead700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f628c146b40 0x7f628c13f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:23.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.047+0000 7f6292ead700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628c0b3070 con 0x7f628c146b40 2022-06-29T16:00:23.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.047+0000 7f6292ead700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628c0ae070 con 0x7f628c0b0280 2022-06-29T16:00:23.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.047+0000 7f6292ead700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f628c13fb30 con 0x7f628c0b5eb0 2022-06-29T16:00:23.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.051+0000 7f62926ac700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f628c146b40 0x7f628c13f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:23.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.051+0000 7f62916aa700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f628c0b5eb0 0x7f628c13f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:23.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.051+0000 7f6291eab700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f628c0b0280 0x7f628c0ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:23.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.051+0000 7f62926ac700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f628c146b40 0x7f628c13f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47206/0 (socket says 172.21.15.60:47206) 2022-06-29T16:00:23.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.051+0000 7f62926ac700 1 -- 172.21.15.60:0/1878036819 learned_addr learned my addr 172.21.15.60:0/1878036819 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:23.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.051+0000 7f62926ac700 1 -- 172.21.15.60:0/1878036819 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f628c0b5eb0 msgr2=0x7f628c13f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:23.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.051+0000 7f62926ac700 1 --2- 172.21.15.60:0/1878036819 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f628c0b5eb0 0x7f628c13f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:23.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.051+0000 7f62926ac700 1 -- 172.21.15.60:0/1878036819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f628c0b0280 msgr2=0x7f628c0ae890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:23.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.051+0000 7f62926ac700 1 --2- 172.21.15.60:0/1878036819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f628c0b0280 0x7f628c0ae890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:23.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.051+0000 7f62926ac700 1 -- 172.21.15.60:0/1878036819 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f627c00b180 con 0x7f628c146b40 2022-06-29T16:00:23.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.051+0000 7f62916aa700 1 --2- 172.21.15.60:0/1878036819 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f628c0b5eb0 0x7f628c13f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:00:23.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.051+0000 7f6291eab700 1 --2- 172.21.15.60:0/1878036819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f628c0b0280 0x7f628c0ae890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:00:23.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.051+0000 7f62926ac700 1 --2- 172.21.15.60:0/1878036819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f628c146b40 0x7f628c13f640 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f628c05d080 tx=0x7f62840013a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:23.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.051+0000 7f627affd700 1 -- 172.21.15.60:0/1878036819 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6284005ba0 con 0x7f628c146b40 2022-06-29T16:00:23.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.051+0000 7f6292ead700 1 -- 172.21.15.60:0/1878036819 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f628800b180 con 0x7f628c146b40 2022-06-29T16:00:23.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.051+0000 7f6292ead700 1 -- 172.21.15.60:0/1878036819 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f628c0aef20 con 0x7f628c146b40 2022-06-29T16:00:23.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.051+0000 7f627affd700 1 -- 172.21.15.60:0/1878036819 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6284005d00 con 0x7f628c146b40 2022-06-29T16:00:23.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.051+0000 7f627affd700 1 -- 172.21.15.60:0/1878036819 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62840072c0 con 0x7f628c146b40 2022-06-29T16:00:23.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.051+0000 7f6292ead700 1 -- 172.21.15.60:0/1878036819 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6268003410 con 0x7f628c146b40 2022-06-29T16:00:23.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.055+0000 7f627affd700 1 -- 172.21.15.60:0/1878036819 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6284010070 con 0x7f628c146b40 2022-06-29T16:00:23.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.055+0000 7f627affd700 1 --2- 172.21.15.60:0/1878036819 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f626c0539e0 0x7f626c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:23.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.055+0000 7f627affd700 1 -- 172.21.15.60:0/1878036819 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f628406b740 con 0x7f628c146b40 2022-06-29T16:00:23.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.055+0000 7f6291eab700 1 --2- 172.21.15.60:0/1878036819 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f626c0539e0 0x7f626c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:23.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.055+0000 7f6291eab700 1 --2- 172.21.15.60:0/1878036819 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f626c0539e0 0x7f626c055ea0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f627c00b000 tx=0x7f627c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:23.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.059+0000 7f627affd700 1 -- 172.21.15.60:0/1878036819 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f62840424d0 con 0x7f628c146b40 2022-06-29T16:00:23.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.231+0000 7f6292ead700 1 -- 172.21.15.60:0/1878036819 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f62680036c0 con 0x7f628c146b40 2022-06-29T16:00:23.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.239+0000 7f627affd700 1 -- 172.21.15.60:0/1878036819 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v18) v1 ==== 184+0+0 (secure 0 0 0) 0x7f6284045af0 con 0x7f628c146b40 2022-06-29T16:00:23.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.239+0000 7f6292ead700 1 -- 172.21.15.60:0/1878036819 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f626c0539e0 msgr2=0x7f626c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:23.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.239+0000 7f6292ead700 1 --2- 172.21.15.60:0/1878036819 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f626c0539e0 0x7f626c055ea0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f627c00b000 tx=0x7f627c006df0).stop 2022-06-29T16:00:23.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.239+0000 7f6292ead700 1 -- 172.21.15.60:0/1878036819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f628c146b40 msgr2=0x7f628c13f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:23.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.239+0000 7f6292ead700 1 --2- 172.21.15.60:0/1878036819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f628c146b40 0x7f628c13f640 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f628c05d080 tx=0x7f62840013a0).stop 2022-06-29T16:00:23.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.239+0000 7f6292ead700 1 -- 172.21.15.60:0/1878036819 shutdown_connections 2022-06-29T16:00:23.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.239+0000 7f6292ead700 1 --2- 172.21.15.60:0/1878036819 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f626c0539e0 0x7f626c055ea0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:23.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.239+0000 7f6292ead700 1 --2- 172.21.15.60:0/1878036819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f628c146b40 0x7f628c13f640 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:23.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.239+0000 7f6292ead700 1 --2- 172.21.15.60:0/1878036819 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f628c0b5eb0 0x7f628c13f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:23.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.239+0000 7f6292ead700 1 --2- 172.21.15.60:0/1878036819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f628c0b0280 0x7f628c0ae890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:23.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.239+0000 7f6292ead700 1 -- 172.21.15.60:0/1878036819 >> 172.21.15.60:0/1878036819 conn(0x7f628c062090 msgr2=0x7f628c1427f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:23.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.239+0000 7f6292ead700 1 -- 172.21.15.60:0/1878036819 shutdown_connections 2022-06-29T16:00:23.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.239+0000 7f6292ead700 1 -- 172.21.15.60:0/1878036819 wait complete. 2022-06-29T16:00:23.254 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.2 2022-06-29T16:00:23.263 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:00:23.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.359+0000 7f1b1359e700 1 Processor -- start 2022-06-29T16:00:23.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.359+0000 7f1b1359e700 1 -- start start 2022-06-29T16:00:23.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.359+0000 7f1b1359e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1b14145eb0 0x7f1b14066130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:23.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.359+0000 7f1b1359e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b14141860 0x7f1b14141cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:23.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.359+0000 7f1b1359e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b1405c2b0 con 0x7f1b140b3c70 2022-06-29T16:00:23.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.359+0000 7f1b1359e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b14066620 con 0x7f1b14145eb0 2022-06-29T16:00:23.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.359+0000 7f1b1359e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b140b4050 con 0x7f1b14141860 2022-06-29T16:00:23.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.359+0000 7f1b11d9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1b14145eb0 0x7f1b14066130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:23.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.359+0000 7f1b11d9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1b14145eb0 0x7f1b14066130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47212/0 (socket says 172.21.15.60:47212) 2022-06-29T16:00:23.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.359+0000 7f1b11d9b700 1 -- 172.21.15.60:0/1773841416 learned_addr learned my addr 172.21.15.60:0/1773841416 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:23.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.359+0000 7f1b12d9d700 1 --2- 172.21.15.60:0/1773841416 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b14141860 0x7f1b14141cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:23.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.359+0000 7f1b11d9b700 1 -- 172.21.15.60:0/1773841416 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b14141860 msgr2=0x7f1b14141cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:23.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.359+0000 7f1b11d9b700 1 --2- 172.21.15.60:0/1773841416 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b14141860 0x7f1b14141cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:23.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.359+0000 7f1b11d9b700 1 -- 172.21.15.60:0/1773841416 >> v1:172.21.15.92:6789/0 conn(0x7f1b140b3c70 legacy=0x7f1b14065a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:23.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.359+0000 7f1b11d9b700 1 -- 172.21.15.60:0/1773841416 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b1405d120 con 0x7f1b14145eb0 2022-06-29T16:00:23.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.359+0000 7f1b11d9b700 1 --2- 172.21.15.60:0/1773841416 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b14145eb0 0x7f1b14066130 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f1b08002f70 tx=0x7f1b08006eb0).ready entity=mon.0 client_cookie=64606730d8ad54b9 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:23.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.359+0000 7f1b1159a700 1 -- 172.21.15.60:0/1773841416 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b0800d5c0 con 0x7f1b14145eb0 2022-06-29T16:00:23.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.359+0000 7f1b1159a700 1 -- 172.21.15.60:0/1773841416 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b08000e90 con 0x7f1b14145eb0 2022-06-29T16:00:23.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.359+0000 7f1b1159a700 1 -- 172.21.15.60:0/1773841416 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b0800d950 con 0x7f1b14145eb0 2022-06-29T16:00:23.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.359+0000 7f1b1359e700 1 -- 172.21.15.60:0/1773841416 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b14145eb0 msgr2=0x7f1b14066130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:23.382 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.359+0000 7f1b1359e700 1 --2- 172.21.15.60:0/1773841416 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b14145eb0 0x7f1b14066130 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f1b08002f70 tx=0x7f1b08006eb0).stop 2022-06-29T16:00:23.382 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.359+0000 7f1b1359e700 1 -- 172.21.15.60:0/1773841416 shutdown_connections 2022-06-29T16:00:23.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.359+0000 7f1b1359e700 1 --2- 172.21.15.60:0/1773841416 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b14141860 0x7f1b14141cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:23.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.359+0000 7f1b1359e700 1 --2- 172.21.15.60:0/1773841416 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b14145eb0 0x7f1b14066130 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:23.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.359+0000 7f1b1359e700 1 -- 172.21.15.60:0/1773841416 >> 172.21.15.60:0/1773841416 conn(0x7f1b14064810 msgr2=0x7f1b14064bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:23.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.363+0000 7f1b1359e700 1 -- 172.21.15.60:0/1773841416 shutdown_connections 2022-06-29T16:00:23.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.363+0000 7f1b1359e700 1 -- 172.21.15.60:0/1773841416 wait complete. 2022-06-29T16:00:23.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.363+0000 7f1b1359e700 1 Processor -- start 2022-06-29T16:00:23.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.363+0000 7f1b1359e700 1 -- start start 2022-06-29T16:00:23.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.363+0000 7f1b1359e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b140b3c70 0x7f1b140acd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:23.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.367+0000 7f1b1359e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b14141860 0x7f1b1413d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:23.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.367+0000 7f1b1359e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b14145eb0 0x7f1b1413d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:23.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.367+0000 7f1b1359e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b14147bc0 con 0x7f1b14145eb0 2022-06-29T16:00:23.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.367+0000 7f1b1359e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b14147e60 con 0x7f1b140b3c70 2022-06-29T16:00:23.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.367+0000 7f1b1359e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b14148100 con 0x7f1b14141860 2022-06-29T16:00:23.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.367+0000 7f1b11d9b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b14141860 0x7f1b1413d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:23.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.367+0000 7f1b1259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b140b3c70 0x7f1b140acd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:23.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.367+0000 7f1b12d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b14145eb0 0x7f1b1413d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:23.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.367+0000 7f1b1259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b140b3c70 0x7f1b140acd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:53246/0 (socket says 172.21.15.60:53246) 2022-06-29T16:00:23.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.367+0000 7f1b1259c700 1 -- 172.21.15.60:0/747142696 learned_addr learned my addr 172.21.15.60:0/747142696 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:23.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.367+0000 7f1b11d9b700 1 --2- 172.21.15.60:0/747142696 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b14141860 0x7f1b1413d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:60312/0 (socket says 172.21.15.60:60312) 2022-06-29T16:00:23.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.367+0000 7f1b1259c700 1 -- 172.21.15.60:0/747142696 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b14141860 msgr2=0x7f1b1413d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:23.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.367+0000 7f1b1259c700 1 --2- 172.21.15.60:0/747142696 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b14141860 0x7f1b1413d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:23.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.367+0000 7f1b1259c700 1 -- 172.21.15.60:0/747142696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b14145eb0 msgr2=0x7f1b1413d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:23.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.367+0000 7f1b1259c700 1 --2- 172.21.15.60:0/747142696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b14145eb0 0x7f1b1413d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:23.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.367+0000 7f1b1259c700 1 -- 172.21.15.60:0/747142696 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b0800b1e0 con 0x7f1b140b3c70 2022-06-29T16:00:23.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.367+0000 7f1b12d9d700 1 --2- 172.21.15.60:0/747142696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b14145eb0 0x7f1b1413d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:00:23.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.367+0000 7f1b1259c700 1 --2- 172.21.15.60:0/747142696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b140b3c70 0x7f1b140acd60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f1b14064270 tx=0x7f1b04007c50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:23.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.367+0000 7f1b037fe700 1 -- 172.21.15.60:0/747142696 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b04012400 con 0x7f1b140b3c70 2022-06-29T16:00:23.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.367+0000 7f1b037fe700 1 -- 172.21.15.60:0/747142696 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b04012560 con 0x7f1b140b3c70 2022-06-29T16:00:23.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.367+0000 7f1b1359e700 1 -- 172.21.15.60:0/747142696 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b14145420 con 0x7f1b140b3c70 2022-06-29T16:00:23.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.367+0000 7f1b037fe700 1 -- 172.21.15.60:0/747142696 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b040128c0 con 0x7f1b140b3c70 2022-06-29T16:00:23.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.367+0000 7f1b1359e700 1 -- 172.21.15.60:0/747142696 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b14062e80 con 0x7f1b140b3c70 2022-06-29T16:00:23.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.367+0000 7f1b1359e700 1 -- 172.21.15.60:0/747142696 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b140630d0 con 0x7f1b140b3c70 2022-06-29T16:00:23.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.371+0000 7f1b037fe700 1 -- 172.21.15.60:0/747142696 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1b04019030 con 0x7f1b140b3c70 2022-06-29T16:00:23.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.371+0000 7f1b037fe700 1 --2- 172.21.15.60:0/747142696 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1af0053950 0x7f1af0055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:23.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.371+0000 7f1b037fe700 1 -- 172.21.15.60:0/747142696 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1b0406b220 con 0x7f1b140b3c70 2022-06-29T16:00:23.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.371+0000 7f1b11d9b700 1 --2- 172.21.15.60:0/747142696 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1af0053950 0x7f1af0055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:23.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.371+0000 7f1b11d9b700 1 --2- 172.21.15.60:0/747142696 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1af0053950 0x7f1af0055e10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f1b0801c000 tx=0x7f1b08015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:23.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.379+0000 7f1b037fe700 1 -- 172.21.15.60:0/747142696 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1b040420e0 con 0x7f1b140b3c70 2022-06-29T16:00:23.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.543+0000 7f1b1359e700 1 -- 172.21.15.60:0/747142696 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b141451b0 con 0x7f1b140b3c70 2022-06-29T16:00:23.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.543+0000 7f1b037fe700 1 -- 172.21.15.60:0/747142696 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+2138 (secure 0 0 0) 0x7f1b141451b0 con 0x7f1b140b3c70 2022-06-29T16:00:23.553 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:00:23.553 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":15,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4937,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/1673184114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1673184114},{"type":"v1","addr":"172.21.15.92:6835","nonce":1673184114}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":13}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:00:20.685676+0000","modified":"2022-06-29T16:00:21.689299+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5308":{"gid":5308,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/1964426267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1964426267},{"type":"v1","addr":"172.21.15.60:6833","nonce":1964426267}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-06-29T16:00:23.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.547+0000 7f1b1359e700 1 -- 172.21.15.60:0/747142696 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1af0053950 msgr2=0x7f1af0055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:23.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.547+0000 7f1b1359e700 1 --2- 172.21.15.60:0/747142696 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1af0053950 0x7f1af0055e10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f1b0801c000 tx=0x7f1b08015000).stop 2022-06-29T16:00:23.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.547+0000 7f1b1359e700 1 -- 172.21.15.60:0/747142696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b140b3c70 msgr2=0x7f1b140acd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:23.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.547+0000 7f1b1359e700 1 --2- 172.21.15.60:0/747142696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b140b3c70 0x7f1b140acd60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f1b14064270 tx=0x7f1b04007c50).stop 2022-06-29T16:00:23.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.547+0000 7f1b1359e700 1 -- 172.21.15.60:0/747142696 shutdown_connections 2022-06-29T16:00:23.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.547+0000 7f1b1359e700 1 --2- 172.21.15.60:0/747142696 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1af0053950 0x7f1af0055e10 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:23.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.547+0000 7f1b1359e700 1 --2- 172.21.15.60:0/747142696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b14145eb0 0x7f1b1413d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:23.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.547+0000 7f1b1359e700 1 --2- 172.21.15.60:0/747142696 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1b14141860 0x7f1b1413d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:23.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.547+0000 7f1b1359e700 1 --2- 172.21.15.60:0/747142696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b140b3c70 0x7f1b140acd60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:23.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.547+0000 7f1b1359e700 1 -- 172.21.15.60:0/747142696 >> 172.21.15.60:0/747142696 conn(0x7f1b14064810 msgr2=0x7f1b14065920 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:23.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.547+0000 7f1b1359e700 1 -- 172.21.15.60:0/747142696 shutdown_connections 2022-06-29T16:00:23.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.547+0000 7f1b1359e700 1 -- 172.21.15.60:0/747142696 wait complete. 2022-06-29T16:00:23.560 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 15 2022-06-29T16:00:23.772 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:00:20.685676+0000', 'modified': '2022-06-29T16:00:21.689299+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5308}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5308': {'gid': 5308, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.60:6833/1964426267', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6832', 'nonce': 1964426267}, {'type': 'v1', 'addr': '172.21.15.60:6833', 'nonce': 1964426267}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:00:23.773 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:00:23.774 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-06-29T16:00:23.894 INFO:teuthology.orchestra.run.smithi060.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":5308,"want_state":"up:active","state":"up:active","rank_uptime":3.1896070139999999,"mdsmap_epoch":15,"osdmap_epoch":43,"osdmap_epoch_barrier":42,"uptime":8.1418384719999999} 2022-06-29T16:00:23.895 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 5308, "mdsmap_epoch": 15, "osdmap_epoch": 43, "osdmap_epoch_barrier": 42, "rank_uptime": 3.189607014, "state": "up:active", "uptime": 8.141838472, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:00:23.895 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:00:24.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.995+0000 7efedc88c700 1 Processor -- start 2022-06-29T16:00:24.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.995+0000 7efedc88c700 1 -- start start 2022-06-29T16:00:24.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.999+0000 7efedc88c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efed80aed00 0x7efed8146810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:24.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.999+0000 7efedc88c700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed805d450 con 0x7efed80b2130 2022-06-29T16:00:24.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.999+0000 7efedc88c700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed80b25d0 con 0x7efed805d070 2022-06-29T16:00:24.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.999+0000 7efedc88c700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efed80adf90 con 0x7efed80aed00 2022-06-29T16:00:24.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.999+0000 7efed659c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7efed80b2130 0x7efed8146100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55310/0 (socket says 172.21.15.60:55310) 2022-06-29T16:00:24.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.999+0000 7efed659c700 1 -- 172.21.15.60:0/2132364099 learned_addr learned my addr 172.21.15.60:0/2132364099 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:00:24.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.999+0000 7efed759e700 1 --2- 172.21.15.60:0/2132364099 >> v2:172.21.15.60:3300/0 conn(0x7efed80aed00 0x7efed8146810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:24.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.999+0000 7efed5d9b700 1 -- 172.21.15.60:0/2132364099 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 958174517 0 0) 0x7efed805d450 con 0x7efed80b2130 2022-06-29T16:00:24.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.999+0000 7efed5d9b700 1 -- 172.21.15.60:0/2132364099 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeb8003260 con 0x7efed80b2130 2022-06-29T16:00:24.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.999+0000 7efed5d9b700 1 -- 172.21.15.60:0/2132364099 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1166687147 0 0) 0x7efed80b25d0 con 0x7efed805d070 2022-06-29T16:00:24.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.999+0000 7efed5d9b700 1 -- 172.21.15.60:0/2132364099 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efed805d450 con 0x7efed805d070 2022-06-29T16:00:24.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.999+0000 7efed759e700 1 -- 172.21.15.60:0/2132364099 >> v1:172.21.15.92:6789/0 conn(0x7efed805d070 legacy=0x7efed814aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:24.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.999+0000 7efed759e700 1 -- 172.21.15.60:0/2132364099 >> v1:172.21.15.60:6789/0 conn(0x7efed80b2130 legacy=0x7efed8146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:24.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.999+0000 7efed5d9b700 1 -- 172.21.15.60:0/2132364099 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1117912946 0 0) 0x7efeb8003260 con 0x7efed80b2130 2022-06-29T16:00:24.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.999+0000 7efed759e700 1 -- 172.21.15.60:0/2132364099 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed814a350 con 0x7efed80aed00 2022-06-29T16:00:24.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.999+0000 7efed759e700 1 --2- 172.21.15.60:0/2132364099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efed80aed00 0x7efed8146810 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7efec0001800 tx=0x7efed0006df0).ready entity=mon.0 client_cookie=55e06808a81bcaa4 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:24.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.999+0000 7efed5d9b700 1 -- 172.21.15.60:0/2132364099 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efec0001660 con 0x7efed80aed00 2022-06-29T16:00:24.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.999+0000 7efed5d9b700 1 -- 172.21.15.60:0/2132364099 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efed0000e90 con 0x7efed80aed00 2022-06-29T16:00:24.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.999+0000 7efed5d9b700 1 -- 172.21.15.60:0/2132364099 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efed0012460 con 0x7efed80aed00 2022-06-29T16:00:24.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.999+0000 7efedc88c700 1 -- 172.21.15.60:0/2132364099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efed80aed00 msgr2=0x7efed8146810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:24.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.999+0000 7efedc88c700 1 --2- 172.21.15.60:0/2132364099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efed80aed00 0x7efed8146810 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7efec0001800 tx=0x7efed0006df0).stop 2022-06-29T16:00:24.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.999+0000 7efedc88c700 1 -- 172.21.15.60:0/2132364099 shutdown_connections 2022-06-29T16:00:24.016 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.999+0000 7efedc88c700 1 --2- 172.21.15.60:0/2132364099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efed80aed00 0x7efed8146810 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:24.016 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.999+0000 7efedc88c700 1 -- 172.21.15.60:0/2132364099 >> 172.21.15.60:0/2132364099 conn(0x7efed80b5890 msgr2=0x7efed80b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:24.016 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.999+0000 7efedc88c700 1 -- 172.21.15.60:0/2132364099 shutdown_connections 2022-06-29T16:00:24.016 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:23.999+0000 7efedc88c700 1 -- 172.21.15.60:0/2132364099 wait complete. 2022-06-29T16:00:24.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.003+0000 7efedc88c700 1 Processor -- start 2022-06-29T16:00:24.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.003+0000 7efedc88c700 1 -- start start 2022-06-29T16:00:24.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.003+0000 7efedc88c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efed805d070 0x7efed8116da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:24.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.003+0000 7efedc88c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efed80aed00 0x7efed8066360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:24.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.003+0000 7efedc88c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efed80b2130 0x7efed80641a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:24.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.003+0000 7efedc88c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed8065e20 con 0x7efed805d070 2022-06-29T16:00:24.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.003+0000 7efedc88c700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed8118f80 con 0x7efed80aed00 2022-06-29T16:00:24.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.003+0000 7efedc88c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed81191b0 con 0x7efed80b2130 2022-06-29T16:00:24.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.003+0000 7efed6d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efed805d070 0x7efed8116da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:24.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.003+0000 7efed6d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efed805d070 0x7efed8116da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47230/0 (socket says 172.21.15.60:47230) 2022-06-29T16:00:24.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.003+0000 7efed6d9d700 1 -- 172.21.15.60:0/3758865592 learned_addr learned my addr 172.21.15.60:0/3758865592 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:24.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.003+0000 7efed659c700 1 --2- 172.21.15.60:0/3758865592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efed80aed00 0x7efed8066360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:24.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.003+0000 7efed759e700 1 --2- 172.21.15.60:0/3758865592 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efed80b2130 0x7efed80641a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:24.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.007+0000 7efed6d9d700 1 -- 172.21.15.60:0/3758865592 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efed80b2130 msgr2=0x7efed80641a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:24.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.007+0000 7efed6d9d700 1 --2- 172.21.15.60:0/3758865592 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efed80b2130 0x7efed80641a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:24.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.007+0000 7efed6d9d700 1 -- 172.21.15.60:0/3758865592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efed80aed00 msgr2=0x7efed8066360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:24.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.007+0000 7efed6d9d700 1 --2- 172.21.15.60:0/3758865592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efed80aed00 0x7efed8066360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:24.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.007+0000 7efed6d9d700 1 -- 172.21.15.60:0/3758865592 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed000b180 con 0x7efed805d070 2022-06-29T16:00:24.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.007+0000 7efed6d9d700 1 --2- 172.21.15.60:0/3758865592 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efed805d070 0x7efed8116da0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7efec8001490 tx=0x7efec8007e60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:24.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.007+0000 7efecffff700 1 -- 172.21.15.60:0/3758865592 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efec8002240 con 0x7efed805d070 2022-06-29T16:00:24.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.007+0000 7efecffff700 1 -- 172.21.15.60:0/3758865592 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec80023a0 con 0x7efed805d070 2022-06-29T16:00:24.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.007+0000 7efedc88c700 1 -- 172.21.15.60:0/3758865592 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efed8066d50 con 0x7efed805d070 2022-06-29T16:00:24.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.007+0000 7efecffff700 1 -- 172.21.15.60:0/3758865592 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efec8001650 con 0x7efed805d070 2022-06-29T16:00:24.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.007+0000 7efedc88c700 1 -- 172.21.15.60:0/3758865592 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efed8066850 con 0x7efed805d070 2022-06-29T16:00:24.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.007+0000 7efedc88c700 1 -- 172.21.15.60:0/3758865592 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efed814a5f0 con 0x7efed805d070 2022-06-29T16:00:24.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.007+0000 7efecffff700 1 -- 172.21.15.60:0/3758865592 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7efec8010030 con 0x7efed805d070 2022-06-29T16:00:24.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.007+0000 7efecffff700 1 --2- 172.21.15.60:0/3758865592 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efebc053a10 0x7efebc055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:24.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.007+0000 7efecffff700 1 -- 172.21.15.60:0/3758865592 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6073+0+0 (secure 0 0 0) 0x7efec806be10 con 0x7efed805d070 2022-06-29T16:00:24.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.007+0000 7efed659c700 1 --2- 172.21.15.60:0/3758865592 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efebc053a10 0x7efebc055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:24.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.007+0000 7efed659c700 1 --2- 172.21.15.60:0/3758865592 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efebc053a10 0x7efebc055ed0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7efec000d000 tx=0x7efec0001740).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:24.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.015+0000 7efecffff700 1 -- 172.21.15.60:0/3758865592 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efec8042c90 con 0x7efed805d070 2022-06-29T16:00:24.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.139+0000 7efedc88c700 1 -- 172.21.15.60:0/3758865592 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efed8119320 con 0x7efed805d070 2022-06-29T16:00:24.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.143+0000 7efecffff700 1 -- 172.21.15.60:0/3758865592 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+16038 (secure 0 0 0) 0x7efed8119320 con 0x7efed805d070 2022-06-29T16:00:24.152 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:00:24.152 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":43,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:00:22.691182+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:00:18.552380+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2022-06-29T16:00:19.688558+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:00:24.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.143+0000 7efedc88c700 1 -- 172.21.15.60:0/3758865592 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efebc053a10 msgr2=0x7efebc055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:24.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.143+0000 7efedc88c700 1 --2- 172.21.15.60:0/3758865592 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efebc053a10 0x7efebc055ed0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7efec000d000 tx=0x7efec0001740).stop 2022-06-29T16:00:24.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.143+0000 7efedc88c700 1 -- 172.21.15.60:0/3758865592 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efed805d070 msgr2=0x7efed8116da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:24.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.143+0000 7efedc88c700 1 --2- 172.21.15.60:0/3758865592 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efed805d070 0x7efed8116da0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7efec8001490 tx=0x7efec8007e60).stop 2022-06-29T16:00:24.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.143+0000 7efedc88c700 1 -- 172.21.15.60:0/3758865592 shutdown_connections 2022-06-29T16:00:24.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.143+0000 7efedc88c700 1 --2- 172.21.15.60:0/3758865592 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efebc053a10 0x7efebc055ed0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:24.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.143+0000 7efedc88c700 1 --2- 172.21.15.60:0/3758865592 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efed80b2130 0x7efed80641a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:24.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.143+0000 7efedc88c700 1 --2- 172.21.15.60:0/3758865592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efed80aed00 0x7efed8066360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:24.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.143+0000 7efedc88c700 1 --2- 172.21.15.60:0/3758865592 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efed805d070 0x7efed8116da0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:24.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.143+0000 7efedc88c700 1 -- 172.21.15.60:0/3758865592 >> 172.21.15.60:0/3758865592 conn(0x7efed80b5890 msgr2=0x7efed8145480 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:24.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.143+0000 7efedc88c700 1 -- 172.21.15.60:0/3758865592 shutdown_connections 2022-06-29T16:00:24.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.143+0000 7efedc88c700 1 -- 172.21.15.60:0/3758865592 wait complete. 2022-06-29T16:00:24.165 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-29T16:00:18.552380+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2022-06-29T16:00:19.688558+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-29T16:00:24.166 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:00:24.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.259+0000 7fb875453700 1 Processor -- start 2022-06-29T16:00:24.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.259+0000 7fb875453700 1 -- start start 2022-06-29T16:00:24.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.263+0000 7fb875453700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb8700b22c0 0x7fb87005bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:24.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.263+0000 7fb875453700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb8700b02d0 0x7fb87005c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:24.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.263+0000 7fb875453700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8700b1540 con 0x7fb87005c700 2022-06-29T16:00:24.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.263+0000 7fb875453700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb87005cb70 con 0x7fb8700b22c0 2022-06-29T16:00:24.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.263+0000 7fb875453700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8700b26a0 con 0x7fb8700b02d0 2022-06-29T16:00:24.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.263+0000 7fb86ffff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb8700b22c0 0x7fb87005bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:24.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.263+0000 7fb874c52700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb87005c700 0x7fb870140d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55328/0 (socket says 172.21.15.60:55328) 2022-06-29T16:00:24.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.263+0000 7fb874c52700 1 -- 172.21.15.60:0/1421037467 learned_addr learned my addr 172.21.15.60:0/1421037467 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:00:24.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.263+0000 7fb86f7fe700 1 --2- 172.21.15.60:0/1421037467 >> v2:172.21.15.92:3300/0 conn(0x7fb8700b02d0 0x7fb87005c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:24.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.263+0000 7fb86effd700 1 -- 172.21.15.60:0/1421037467 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 280540072 0 0) 0x7fb8700b1540 con 0x7fb87005c700 2022-06-29T16:00:24.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.263+0000 7fb86effd700 1 -- 172.21.15.60:0/1421037467 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb85c003260 con 0x7fb87005c700 2022-06-29T16:00:24.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.263+0000 7fb86ffff700 1 -- 172.21.15.60:0/1421037467 >> v2:172.21.15.92:3300/0 conn(0x7fb8700b02d0 msgr2=0x7fb87005c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:24.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.263+0000 7fb86ffff700 1 --2- 172.21.15.60:0/1421037467 >> v2:172.21.15.92:3300/0 conn(0x7fb8700b02d0 0x7fb87005c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:24.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.263+0000 7fb86ffff700 1 -- 172.21.15.60:0/1421037467 >> v1:172.21.15.60:6789/0 conn(0x7fb87005c700 legacy=0x7fb870140d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:24.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.263+0000 7fb86ffff700 1 -- 172.21.15.60:0/1421037467 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87009a8b0 con 0x7fb8700b22c0 2022-06-29T16:00:24.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.263+0000 7fb86ffff700 1 --2- 172.21.15.60:0/1421037467 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8700b22c0 0x7fb87005bd20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fb864002f70 tx=0x7fb864006df0).ready entity=mon.0 client_cookie=873a41b26677ab5a server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:24.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.263+0000 7fb86effd700 1 -- 172.21.15.60:0/1421037467 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb86400d4f0 con 0x7fb8700b22c0 2022-06-29T16:00:24.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.263+0000 7fb86effd700 1 -- 172.21.15.60:0/1421037467 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb864000e90 con 0x7fb8700b22c0 2022-06-29T16:00:24.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.263+0000 7fb86effd700 1 -- 172.21.15.60:0/1421037467 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb86400d840 con 0x7fb8700b22c0 2022-06-29T16:00:24.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.263+0000 7fb875453700 1 -- 172.21.15.60:0/1421037467 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8700b22c0 msgr2=0x7fb87005bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:24.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.263+0000 7fb875453700 1 --2- 172.21.15.60:0/1421037467 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8700b22c0 0x7fb87005bd20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fb864002f70 tx=0x7fb864006df0).stop 2022-06-29T16:00:24.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.263+0000 7fb875453700 1 -- 172.21.15.60:0/1421037467 shutdown_connections 2022-06-29T16:00:24.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.263+0000 7fb875453700 1 --2- 172.21.15.60:0/1421037467 >> v2:172.21.15.92:3300/0 conn(0x7fb8700b02d0 0x7fb87005c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:24.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.263+0000 7fb875453700 1 --2- 172.21.15.60:0/1421037467 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8700b22c0 0x7fb87005bd20 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:24.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.263+0000 7fb875453700 1 -- 172.21.15.60:0/1421037467 >> 172.21.15.60:0/1421037467 conn(0x7fb8700b58a0 msgr2=0x7fb8700b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:24.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.263+0000 7fb875453700 1 -- 172.21.15.60:0/1421037467 shutdown_connections 2022-06-29T16:00:24.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.263+0000 7fb875453700 1 -- 172.21.15.60:0/1421037467 wait complete. 2022-06-29T16:00:24.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.267+0000 7fb875453700 1 Processor -- start 2022-06-29T16:00:24.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.267+0000 7fb875453700 1 -- start start 2022-06-29T16:00:24.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.267+0000 7fb875453700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb87005c700 0x7fb870144910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:24.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.267+0000 7fb875453700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8700b02d0 0x7fb8700ae2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:24.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.267+0000 7fb875453700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8700b22c0 0x7fb8700ae7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:24.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.267+0000 7fb875453700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb870099b50 con 0x7fb8700b02d0 2022-06-29T16:00:24.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.267+0000 7fb875453700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb870099640 con 0x7fb8700b22c0 2022-06-29T16:00:24.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.267+0000 7fb875453700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8700ade20 con 0x7fb87005c700 2022-06-29T16:00:24.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.267+0000 7fb86f7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8700b02d0 0x7fb8700ae2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:24.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.267+0000 7fb86f7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8700b02d0 0x7fb8700ae2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47246/0 (socket says 172.21.15.60:47246) 2022-06-29T16:00:24.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.267+0000 7fb86f7fe700 1 -- 172.21.15.60:0/3634265985 learned_addr learned my addr 172.21.15.60:0/3634265985 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:24.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.267+0000 7fb86ffff700 1 --2- 172.21.15.60:0/3634265985 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb87005c700 0x7fb870144910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:24.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.267+0000 7fb874c52700 1 --2- 172.21.15.60:0/3634265985 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8700b22c0 0x7fb8700ae7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:24.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.267+0000 7fb86f7fe700 1 -- 172.21.15.60:0/3634265985 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb87005c700 msgr2=0x7fb870144910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:24.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.267+0000 7fb86f7fe700 1 --2- 172.21.15.60:0/3634265985 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb87005c700 0x7fb870144910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:24.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.267+0000 7fb86f7fe700 1 -- 172.21.15.60:0/3634265985 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8700b22c0 msgr2=0x7fb8700ae7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:24.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.267+0000 7fb86f7fe700 1 --2- 172.21.15.60:0/3634265985 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8700b22c0 0x7fb8700ae7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:24.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.267+0000 7fb86f7fe700 1 -- 172.21.15.60:0/3634265985 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86400b180 con 0x7fb8700b02d0 2022-06-29T16:00:24.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.271+0000 7fb86f7fe700 1 --2- 172.21.15.60:0/3634265985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8700b02d0 0x7fb8700ae2c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fb86000c000 tx=0x7fb860006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:24.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.271+0000 7fb86d7fa700 1 -- 172.21.15.60:0/3634265985 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb860010780 con 0x7fb8700b02d0 2022-06-29T16:00:24.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.271+0000 7fb86d7fa700 1 -- 172.21.15.60:0/3634265985 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8600108e0 con 0x7fb8700b02d0 2022-06-29T16:00:24.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.271+0000 7fb875453700 1 -- 172.21.15.60:0/3634265985 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8700adb30 con 0x7fb8700b02d0 2022-06-29T16:00:24.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.271+0000 7fb86d7fa700 1 -- 172.21.15.60:0/3634265985 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb860010c40 con 0x7fb8700b02d0 2022-06-29T16:00:24.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.271+0000 7fb875453700 1 -- 172.21.15.60:0/3634265985 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8700aeca0 con 0x7fb8700b02d0 2022-06-29T16:00:24.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.271+0000 7fb875453700 1 -- 172.21.15.60:0/3634265985 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8700adfe0 con 0x7fb8700b02d0 2022-06-29T16:00:24.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.271+0000 7fb86d7fa700 1 -- 172.21.15.60:0/3634265985 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb86001c460 con 0x7fb8700b02d0 2022-06-29T16:00:24.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.271+0000 7fb86d7fa700 1 --2- 172.21.15.60:0/3634265985 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb850053a70 0x7fb850055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:24.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.271+0000 7fb86d7fa700 1 -- 172.21.15.60:0/3634265985 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb86006c2a0 con 0x7fb8700b02d0 2022-06-29T16:00:24.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.271+0000 7fb86ffff700 1 --2- 172.21.15.60:0/3634265985 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb850053a70 0x7fb850055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:24.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.275+0000 7fb86ffff700 1 --2- 172.21.15.60:0/3634265985 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb850053a70 0x7fb850055f30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fb86401a000 tx=0x7fb864014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:24.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.279+0000 7fb86d7fa700 1 -- 172.21.15.60:0/3634265985 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb860043160 con 0x7fb8700b02d0 2022-06-29T16:00:24.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.403+0000 7fb875453700 1 -- 172.21.15.60:0/3634265985 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fb8700573d0 con 0x7fb8700b02d0 2022-06-29T16:00:24.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.403+0000 7fb86d7fa700 1 -- 172.21.15.60:0/3634265985 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v43) v1 ==== 110+0+10 (secure 0 0 0) 0x7fb860046780 con 0x7fb8700b02d0 2022-06-29T16:00:24.423 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:00:24.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.407+0000 7fb875453700 1 -- 172.21.15.60:0/3634265985 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb850053a70 msgr2=0x7fb850055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:24.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.407+0000 7fb875453700 1 --2- 172.21.15.60:0/3634265985 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb850053a70 0x7fb850055f30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fb86401a000 tx=0x7fb864014000).stop 2022-06-29T16:00:24.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.407+0000 7fb875453700 1 -- 172.21.15.60:0/3634265985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8700b02d0 msgr2=0x7fb8700ae2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:24.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.407+0000 7fb875453700 1 --2- 172.21.15.60:0/3634265985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8700b02d0 0x7fb8700ae2c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fb86000c000 tx=0x7fb860006df0).stop 2022-06-29T16:00:24.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.407+0000 7fb875453700 1 -- 172.21.15.60:0/3634265985 shutdown_connections 2022-06-29T16:00:24.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.407+0000 7fb875453700 1 --2- 172.21.15.60:0/3634265985 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb850053a70 0x7fb850055f30 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:24.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.407+0000 7fb875453700 1 --2- 172.21.15.60:0/3634265985 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8700b22c0 0x7fb8700ae7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:24.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.407+0000 7fb875453700 1 --2- 172.21.15.60:0/3634265985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8700b02d0 0x7fb8700ae2c0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:24.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.407+0000 7fb875453700 1 --2- 172.21.15.60:0/3634265985 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb87005c700 0x7fb870144910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:24.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.407+0000 7fb875453700 1 -- 172.21.15.60:0/3634265985 >> 172.21.15.60:0/3634265985 conn(0x7fb8700b58a0 msgr2=0x7fb8700b3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:24.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.407+0000 7fb875453700 1 -- 172.21.15.60:0/3634265985 shutdown_connections 2022-06-29T16:00:24.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.407+0000 7fb875453700 1 -- 172.21.15.60:0/3634265985 wait complete. 2022-06-29T16:00:24.429 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:00:24.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.539+0000 7efd27235700 1 Processor -- start 2022-06-29T16:00:24.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.539+0000 7efd27235700 1 -- start start 2022-06-29T16:00:24.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.539+0000 7efd27235700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd200b5eb0 0x7efd20142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:24.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.539+0000 7efd27235700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd200b0cf0 con 0x7efd200b2c50 2022-06-29T16:00:24.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.539+0000 7efd27235700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd200b4d10 con 0x7efd200b48b0 2022-06-29T16:00:24.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.539+0000 7efd27235700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd200b54e0 con 0x7efd200b5eb0 2022-06-29T16:00:24.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.543+0000 7efd26a34700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7efd200b2c50 0x7efd200b3090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55342/0 (socket says 172.21.15.60:55342) 2022-06-29T16:00:24.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.543+0000 7efd26a34700 1 -- 172.21.15.60:0/1477724794 learned_addr learned my addr 172.21.15.60:0/1477724794 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:00:24.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.543+0000 7efd26233700 1 --2- 172.21.15.60:0/1477724794 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd200b5eb0 0x7efd20142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:24.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.543+0000 7efd25231700 1 -- 172.21.15.60:0/1477724794 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 466767200 0 0) 0x7efd18001490 con 0x7efd200b2c50 2022-06-29T16:00:24.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.543+0000 7efd25231700 1 -- 172.21.15.60:0/1477724794 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd08003260 con 0x7efd200b2c50 2022-06-29T16:00:24.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.543+0000 7efd25231700 1 -- 172.21.15.60:0/1477724794 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2760008857 0 0) 0x7efd200b4d10 con 0x7efd200b48b0 2022-06-29T16:00:24.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.543+0000 7efd25231700 1 -- 172.21.15.60:0/1477724794 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd18001490 con 0x7efd200b48b0 2022-06-29T16:00:24.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.543+0000 7efd26233700 1 -- 172.21.15.60:0/1477724794 >> v1:172.21.15.92:6789/0 conn(0x7efd200b48b0 legacy=0x7efd200b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:24.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.543+0000 7efd26233700 1 -- 172.21.15.60:0/1477724794 >> v1:172.21.15.60:6789/0 conn(0x7efd200b2c50 legacy=0x7efd200b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:24.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.543+0000 7efd26233700 1 -- 172.21.15.60:0/1477724794 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd200b6290 con 0x7efd200b5eb0 2022-06-29T16:00:24.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.543+0000 7efd26233700 1 --2- 172.21.15.60:0/1477724794 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd200b5eb0 0x7efd20142a20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7efd1c002f70 tx=0x7efd1c006df0).ready entity=mon.2 client_cookie=e31dfa89fde2c4fe server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:24.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.543+0000 7efd25231700 1 -- 172.21.15.60:0/1477724794 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd1c00d5c0 con 0x7efd200b5eb0 2022-06-29T16:00:24.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.543+0000 7efd25231700 1 -- 172.21.15.60:0/1477724794 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd1c000e90 con 0x7efd200b5eb0 2022-06-29T16:00:24.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.543+0000 7efd25231700 1 -- 172.21.15.60:0/1477724794 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd1c00d910 con 0x7efd200b5eb0 2022-06-29T16:00:24.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.543+0000 7efd27235700 1 -- 172.21.15.60:0/1477724794 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd200b5eb0 msgr2=0x7efd20142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:24.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.543+0000 7efd27235700 1 --2- 172.21.15.60:0/1477724794 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd200b5eb0 0x7efd20142a20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7efd1c002f70 tx=0x7efd1c006df0).stop 2022-06-29T16:00:24.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.543+0000 7efd27235700 1 -- 172.21.15.60:0/1477724794 shutdown_connections 2022-06-29T16:00:24.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.543+0000 7efd27235700 1 --2- 172.21.15.60:0/1477724794 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd200b5eb0 0x7efd20142a20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:24.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.543+0000 7efd27235700 1 -- 172.21.15.60:0/1477724794 >> 172.21.15.60:0/1477724794 conn(0x7efd20062090 msgr2=0x7efd20062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:24.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.543+0000 7efd27235700 1 -- 172.21.15.60:0/1477724794 shutdown_connections 2022-06-29T16:00:24.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.543+0000 7efd27235700 1 -- 172.21.15.60:0/1477724794 wait complete. 2022-06-29T16:00:24.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.547+0000 7efd27235700 1 Processor -- start 2022-06-29T16:00:24.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.547+0000 7efd27235700 1 -- start start 2022-06-29T16:00:24.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.547+0000 7efd27235700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd200b2c50 0x7efd200ada60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:24.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.547+0000 7efd27235700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd200b48b0 0x7efd20154690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:24.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.547+0000 7efd27235700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd200b5eb0 0x7efd20154b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:24.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.547+0000 7efd27235700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd200ae650 con 0x7efd200b48b0 2022-06-29T16:00:24.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.547+0000 7efd27235700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd200ae8f0 con 0x7efd200b2c50 2022-06-29T16:00:24.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.547+0000 7efd27235700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd200aeb90 con 0x7efd200b5eb0 2022-06-29T16:00:24.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.547+0000 7efd25a32700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd200b48b0 0x7efd20154690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:24.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.547+0000 7efd25a32700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd200b48b0 0x7efd20154690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47258/0 (socket says 172.21.15.60:47258) 2022-06-29T16:00:24.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.547+0000 7efd25a32700 1 -- 172.21.15.60:0/1759096494 learned_addr learned my addr 172.21.15.60:0/1759096494 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:24.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.547+0000 7efd26a34700 1 --2- 172.21.15.60:0/1759096494 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd200b5eb0 0x7efd20154b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:24.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.547+0000 7efd26233700 1 --2- 172.21.15.60:0/1759096494 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd200b2c50 0x7efd200ada60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:24.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.547+0000 7efd25a32700 1 -- 172.21.15.60:0/1759096494 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd200b5eb0 msgr2=0x7efd20154b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:24.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.547+0000 7efd25a32700 1 --2- 172.21.15.60:0/1759096494 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd200b5eb0 0x7efd20154b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:24.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.547+0000 7efd25a32700 1 -- 172.21.15.60:0/1759096494 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd200b2c50 msgr2=0x7efd200ada60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:24.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.547+0000 7efd25a32700 1 --2- 172.21.15.60:0/1759096494 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd200b2c50 0x7efd200ada60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:24.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.547+0000 7efd25a32700 1 -- 172.21.15.60:0/1759096494 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd1c00b180 con 0x7efd200b48b0 2022-06-29T16:00:24.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.547+0000 7efd26a34700 1 --2- 172.21.15.60:0/1759096494 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd200b5eb0 0x7efd20154b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:00:24.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.547+0000 7efd25a32700 1 --2- 172.21.15.60:0/1759096494 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd200b48b0 0x7efd20154690 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7efd14001110 tx=0x7efd14007ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:24.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.547+0000 7efd137fe700 1 -- 172.21.15.60:0/1759096494 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd14012400 con 0x7efd200b48b0 2022-06-29T16:00:24.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.547+0000 7efd137fe700 1 -- 172.21.15.60:0/1759096494 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd14012560 con 0x7efd200b48b0 2022-06-29T16:00:24.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.547+0000 7efd137fe700 1 -- 172.21.15.60:0/1759096494 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd140128c0 con 0x7efd200b48b0 2022-06-29T16:00:24.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.547+0000 7efd27235700 1 -- 172.21.15.60:0/1759096494 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd200adf50 con 0x7efd200b48b0 2022-06-29T16:00:24.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.547+0000 7efd27235700 1 -- 172.21.15.60:0/1759096494 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd200aca20 con 0x7efd200b48b0 2022-06-29T16:00:24.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.551+0000 7efd137fe700 1 -- 172.21.15.60:0/1759096494 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7efd14019030 con 0x7efd200b48b0 2022-06-29T16:00:24.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.551+0000 7efd27235700 1 -- 172.21.15.60:0/1759096494 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd200ad160 con 0x7efd200b48b0 2022-06-29T16:00:24.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.551+0000 7efd137fe700 1 --2- 172.21.15.60:0/1759096494 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd00053950 0x7efd00055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:24.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.551+0000 7efd137fe700 1 -- 172.21.15.60:0/1759096494 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6073+0+0 (secure 0 0 0) 0x7efd1406b3e0 con 0x7efd200b48b0 2022-06-29T16:00:24.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.551+0000 7efd26233700 1 --2- 172.21.15.60:0/1759096494 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd00053950 0x7efd00055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:24.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.551+0000 7efd26233700 1 --2- 172.21.15.60:0/1759096494 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd00053950 0x7efd00055e10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7efd1c01a000 tx=0x7efd1c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:24.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.559+0000 7efd137fe700 1 -- 172.21.15.60:0/1759096494 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efd200ad160 con 0x7efd200b48b0 2022-06-29T16:00:24.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.683+0000 7efd27235700 1 -- 172.21.15.60:0/1759096494 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7efd200ad160 con 0x7efd200b48b0 2022-06-29T16:00:24.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.687+0000 7efd137fe700 1 -- 172.21.15.60:0/1759096494 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v43) v1 ==== 92+0+10 (secure 0 0 0) 0x7efd200ad160 con 0x7efd200b48b0 2022-06-29T16:00:24.695 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:00:24.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.687+0000 7efd27235700 1 -- 172.21.15.60:0/1759096494 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd00053950 msgr2=0x7efd00055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:24.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.687+0000 7efd27235700 1 --2- 172.21.15.60:0/1759096494 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd00053950 0x7efd00055e10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7efd1c01a000 tx=0x7efd1c014000).stop 2022-06-29T16:00:24.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.687+0000 7efd27235700 1 -- 172.21.15.60:0/1759096494 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd200b48b0 msgr2=0x7efd20154690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:24.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.687+0000 7efd27235700 1 --2- 172.21.15.60:0/1759096494 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd200b48b0 0x7efd20154690 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7efd14001110 tx=0x7efd14007ae0).stop 2022-06-29T16:00:24.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.687+0000 7efd27235700 1 -- 172.21.15.60:0/1759096494 shutdown_connections 2022-06-29T16:00:24.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.687+0000 7efd27235700 1 --2- 172.21.15.60:0/1759096494 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd00053950 0x7efd00055e10 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:24.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.687+0000 7efd27235700 1 --2- 172.21.15.60:0/1759096494 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd200b5eb0 0x7efd20154b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:24.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.687+0000 7efd27235700 1 --2- 172.21.15.60:0/1759096494 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd200b48b0 0x7efd20154690 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:24.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.687+0000 7efd27235700 1 --2- 172.21.15.60:0/1759096494 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd200b2c50 0x7efd200ada60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:24.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.687+0000 7efd27235700 1 -- 172.21.15.60:0/1759096494 >> 172.21.15.60:0/1759096494 conn(0x7efd20062090 msgr2=0x7efd200b1650 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:24.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.691+0000 7efd27235700 1 -- 172.21.15.60:0/1759096494 shutdown_connections 2022-06-29T16:00:24.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.691+0000 7efd27235700 1 -- 172.21.15.60:0/1759096494 wait complete. 2022-06-29T16:00:24.713 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-29T16:00:24.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.819+0000 7fa5f88a6700 1 Processor -- start 2022-06-29T16:00:24.829 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.819+0000 7fa5f88a6700 1 -- start start 2022-06-29T16:00:24.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.819+0000 7fa5f88a6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5f40b0830 0x7fa5f413ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:24.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.819+0000 7fa5f88a6700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5f405d450 con 0x7fa5f405d070 2022-06-29T16:00:24.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.819+0000 7fa5f88a6700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5f4063ba0 con 0x7fa5f40b2130 2022-06-29T16:00:24.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.819+0000 7fa5f88a6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f4063e40 con 0x7fa5f40b0830 2022-06-29T16:00:24.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.819+0000 7fa5f2d9d700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa5f405d070 0x7fa5f413c5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55352/0 (socket says 172.21.15.60:55352) 2022-06-29T16:00:24.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.819+0000 7fa5f2d9d700 1 -- 172.21.15.60:0/449202991 learned_addr learned my addr 172.21.15.60:0/449202991 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:00:24.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.819+0000 7fa5f359e700 1 --2- 172.21.15.60:0/449202991 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5f40b0830 0x7fa5f413ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:24.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5f259c700 1 -- 172.21.15.60:0/449202991 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2595692877 0 0) 0x7fa5f4063ba0 con 0x7fa5f40b2130 2022-06-29T16:00:24.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5f259c700 1 -- 172.21.15.60:0/449202991 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5d4003260 con 0x7fa5f40b2130 2022-06-29T16:00:24.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5f259c700 1 -- 172.21.15.60:0/449202991 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2385971911 0 0) 0x7fa5f405d450 con 0x7fa5f405d070 2022-06-29T16:00:24.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5f359e700 1 -- 172.21.15.60:0/449202991 >> v1:172.21.15.92:6789/0 conn(0x7fa5f40b2130 legacy=0x7fa5f413fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:24.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5f359e700 1 -- 172.21.15.60:0/449202991 >> v1:172.21.15.60:6789/0 conn(0x7fa5f405d070 legacy=0x7fa5f413c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:24.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5f359e700 1 -- 172.21.15.60:0/449202991 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f4067650 con 0x7fa5f40b0830 2022-06-29T16:00:24.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5f359e700 1 --2- 172.21.15.60:0/449202991 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5f40b0830 0x7fa5f413ccc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fa5ec0032f0 tx=0x7fa5ec0091d0).ready entity=mon.2 client_cookie=80267daebd72eeef server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:24.845 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5f259c700 1 -- 172.21.15.60:0/449202991 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5ec00b6c0 con 0x7fa5f40b0830 2022-06-29T16:00:24.845 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5f259c700 1 -- 172.21.15.60:0/449202991 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5ec00b820 con 0x7fa5f40b0830 2022-06-29T16:00:24.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5f259c700 1 -- 172.21.15.60:0/449202991 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5ec00b990 con 0x7fa5f40b0830 2022-06-29T16:00:24.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5f88a6700 1 -- 172.21.15.60:0/449202991 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5f40b0830 msgr2=0x7fa5f413ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:24.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5f88a6700 1 --2- 172.21.15.60:0/449202991 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5f40b0830 0x7fa5f413ccc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fa5ec0032f0 tx=0x7fa5ec0091d0).stop 2022-06-29T16:00:24.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5f88a6700 1 -- 172.21.15.60:0/449202991 shutdown_connections 2022-06-29T16:00:24.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5f88a6700 1 --2- 172.21.15.60:0/449202991 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5f40b0830 0x7fa5f413ccc0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:24.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5f88a6700 1 -- 172.21.15.60:0/449202991 >> 172.21.15.60:0/449202991 conn(0x7fa5f40b5890 msgr2=0x7fa5f40b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:24.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5f88a6700 1 -- 172.21.15.60:0/449202991 shutdown_connections 2022-06-29T16:00:24.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5f88a6700 1 -- 172.21.15.60:0/449202991 wait complete. 2022-06-29T16:00:24.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5f88a6700 1 Processor -- start 2022-06-29T16:00:24.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5f88a6700 1 -- start start 2022-06-29T16:00:24.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5f88a6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5f405d070 0x7fa5f4143110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:24.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5f88a6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5f40b0830 0x7fa5f41348a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:24.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5f88a6700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa5f40b2130 0x7fa5f4134d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:24.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5f88a6700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f413a870 con 0x7fa5f40b0830 2022-06-29T16:00:24.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5f88a6700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f4139c10 con 0x7fa5f40b2130 2022-06-29T16:00:24.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5f88a6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f40646b0 con 0x7fa5f405d070 2022-06-29T16:00:24.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5ea59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5f40b0830 0x7fa5f41348a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:24.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5ea59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5f40b0830 0x7fa5f41348a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47276/0 (socket says 172.21.15.60:47276) 2022-06-29T16:00:24.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5ea59c700 1 -- 172.21.15.60:0/2554376802 learned_addr learned my addr 172.21.15.60:0/2554376802 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:24.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5f359e700 1 --2- 172.21.15.60:0/2554376802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa5f40b2130 0x7fa5f4134d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:24.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5f2d9d700 1 --2- 172.21.15.60:0/2554376802 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5f405d070 0x7fa5f4143110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:24.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5ea59c700 1 -- 172.21.15.60:0/2554376802 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5f405d070 msgr2=0x7fa5f4143110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:24.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5ea59c700 1 --2- 172.21.15.60:0/2554376802 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5f405d070 0x7fa5f4143110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:24.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5ea59c700 1 -- 172.21.15.60:0/2554376802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa5f40b2130 msgr2=0x7fa5f4134d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:24.859 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5ea59c700 1 --2- 172.21.15.60:0/2554376802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa5f40b2130 0x7fa5f4134d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:24.859 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.823+0000 7fa5ea59c700 1 -- 172.21.15.60:0/2554376802 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ec009470 con 0x7fa5f40b0830 2022-06-29T16:00:24.859 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.827+0000 7fa5ea59c700 1 --2- 172.21.15.60:0/2554376802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5f40b0830 0x7fa5f41348a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7fa5dc001110 tx=0x7fa5dc007ab0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:24.860 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.827+0000 7fa5f0d99700 1 -- 172.21.15.60:0/2554376802 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5dc00ce90 con 0x7fa5f40b0830 2022-06-29T16:00:24.860 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.827+0000 7fa5f0d99700 1 -- 172.21.15.60:0/2554376802 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5dc012400 con 0x7fa5f40b0830 2022-06-29T16:00:24.861 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.827+0000 7fa5f88a6700 1 -- 172.21.15.60:0/2554376802 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5f40641a0 con 0x7fa5f40b0830 2022-06-29T16:00:24.861 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.827+0000 7fa5f0d99700 1 -- 172.21.15.60:0/2554376802 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5dc012760 con 0x7fa5f40b0830 2022-06-29T16:00:24.862 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.827+0000 7fa5f88a6700 1 -- 172.21.15.60:0/2554376802 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5f4064480 con 0x7fa5f40b0830 2022-06-29T16:00:24.862 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.827+0000 7fa5f88a6700 1 -- 172.21.15.60:0/2554376802 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5f41466f0 con 0x7fa5f40b0830 2022-06-29T16:00:24.863 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.827+0000 7fa5f0d99700 1 -- 172.21.15.60:0/2554376802 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa5dc019030 con 0x7fa5f40b0830 2022-06-29T16:00:24.863 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.827+0000 7fa5f0d99700 1 --2- 172.21.15.60:0/2554376802 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa5d8053980 0x7fa5d8055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:24.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.827+0000 7fa5f0d99700 1 -- 172.21.15.60:0/2554376802 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa5dc06b270 con 0x7fa5f40b0830 2022-06-29T16:00:24.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.827+0000 7fa5f2d9d700 1 --2- 172.21.15.60:0/2554376802 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa5d8053980 0x7fa5d8055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:24.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.827+0000 7fa5f2d9d700 1 --2- 172.21.15.60:0/2554376802 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa5d8053980 0x7fa5d8055e40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fa5e400b000 tx=0x7fa5e4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:24.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.835+0000 7fa5f0d99700 1 -- 172.21.15.60:0/2554376802 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa5dc042190 con 0x7fa5f40b0830 2022-06-29T16:00:24.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.959+0000 7fa5f88a6700 1 -- 172.21.15.60:0/2554376802 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa5f40573d0 con 0x7fa5f40b0830 2022-06-29T16:00:24.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.959+0000 7fa5f0d99700 1 -- 172.21.15.60:0/2554376802 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v43) v1 ==== 104+0+10 (secure 0 0 0) 0x7fa5dc0457b0 con 0x7fa5f40b0830 2022-06-29T16:00:24.970 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:00:24.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.963+0000 7fa5f88a6700 1 -- 172.21.15.60:0/2554376802 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa5d8053980 msgr2=0x7fa5d8055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:24.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.963+0000 7fa5f88a6700 1 --2- 172.21.15.60:0/2554376802 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa5d8053980 0x7fa5d8055e40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fa5e400b000 tx=0x7fa5e4009040).stop 2022-06-29T16:00:24.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.963+0000 7fa5f88a6700 1 -- 172.21.15.60:0/2554376802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5f40b0830 msgr2=0x7fa5f41348a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:24.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.963+0000 7fa5f88a6700 1 --2- 172.21.15.60:0/2554376802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5f40b0830 0x7fa5f41348a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7fa5dc001110 tx=0x7fa5dc007ab0).stop 2022-06-29T16:00:24.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.963+0000 7fa5f88a6700 1 -- 172.21.15.60:0/2554376802 shutdown_connections 2022-06-29T16:00:24.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.963+0000 7fa5f88a6700 1 --2- 172.21.15.60:0/2554376802 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa5d8053980 0x7fa5d8055e40 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:24.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.963+0000 7fa5f88a6700 1 --2- 172.21.15.60:0/2554376802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa5f40b2130 0x7fa5f4134d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:24.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.963+0000 7fa5f88a6700 1 --2- 172.21.15.60:0/2554376802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5f40b0830 0x7fa5f41348a0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:24.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.963+0000 7fa5f88a6700 1 --2- 172.21.15.60:0/2554376802 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5f405d070 0x7fa5f4143110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:24.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.963+0000 7fa5f88a6700 1 -- 172.21.15.60:0/2554376802 >> 172.21.15.60:0/2554376802 conn(0x7fa5f40b5890 msgr2=0x7fa5f40b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:24.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.963+0000 7fa5f88a6700 1 -- 172.21.15.60:0/2554376802 shutdown_connections 2022-06-29T16:00:24.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:24.963+0000 7fa5f88a6700 1 -- 172.21.15.60:0/2554376802 wait complete. 2022-06-29T16:00:24.988 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-29T16:00:25.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.091+0000 7ff31759e700 1 Processor -- start 2022-06-29T16:00:25.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.091+0000 7ff31759e700 1 -- start start 2022-06-29T16:00:25.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.091+0000 7ff31759e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff3180b5eb0 0x7ff318142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:25.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.091+0000 7ff31759e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3180b0cf0 con 0x7ff3180b2c50 2022-06-29T16:00:25.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.091+0000 7ff31759e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3180b4d10 con 0x7ff3180b48b0 2022-06-29T16:00:25.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.091+0000 7ff31759e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3180b54e0 con 0x7ff3180b5eb0 2022-06-29T16:00:25.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.091+0000 7ff31659c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff3180b5eb0 0x7ff318142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:25.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.091+0000 7ff316d9d700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff3180b2c50 0x7ff31814d380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55370/0 (socket says 172.21.15.60:55370) 2022-06-29T16:00:25.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.091+0000 7ff316d9d700 1 -- 172.21.15.60:0/580911375 learned_addr learned my addr 172.21.15.60:0/580911375 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:00:25.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.095+0000 7ff31659c700 1 -- 172.21.15.60:0/580911375 >> v1:172.21.15.92:6789/0 conn(0x7ff3180b48b0 legacy=0x7ff3180b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:25.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.095+0000 7ff31659c700 1 -- 172.21.15.60:0/580911375 >> v1:172.21.15.60:6789/0 conn(0x7ff3180b2c50 legacy=0x7ff31814d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:25.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.095+0000 7ff31659c700 1 -- 172.21.15.60:0/580911375 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3180b6290 con 0x7ff3180b5eb0 2022-06-29T16:00:25.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.095+0000 7ff315d9b700 1 -- 172.21.15.60:0/580911375 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4251055716 0 0) 0x7ff310001490 con 0x7ff3180b2c50 2022-06-29T16:00:25.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.095+0000 7ff31659c700 1 --2- 172.21.15.60:0/580911375 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3180b5eb0 0x7ff318142a20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7ff308002f70 tx=0x7ff308006df0).ready entity=mon.0 client_cookie=cea995c004a9edf9 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:25.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.095+0000 7ff315d9b700 1 -- 172.21.15.60:0/580911375 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff30800d610 con 0x7ff3180b5eb0 2022-06-29T16:00:25.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.095+0000 7ff315d9b700 1 -- 172.21.15.60:0/580911375 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff308000e90 con 0x7ff3180b5eb0 2022-06-29T16:00:25.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.095+0000 7ff315d9b700 1 -- 172.21.15.60:0/580911375 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff30800d960 con 0x7ff3180b5eb0 2022-06-29T16:00:25.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.095+0000 7ff31759e700 1 -- 172.21.15.60:0/580911375 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3180b5eb0 msgr2=0x7ff318142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:25.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.095+0000 7ff31759e700 1 --2- 172.21.15.60:0/580911375 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3180b5eb0 0x7ff318142a20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7ff308002f70 tx=0x7ff308006df0).stop 2022-06-29T16:00:25.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.095+0000 7ff31759e700 1 -- 172.21.15.60:0/580911375 shutdown_connections 2022-06-29T16:00:25.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.095+0000 7ff31759e700 1 --2- 172.21.15.60:0/580911375 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3180b5eb0 0x7ff318142a20 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:25.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.095+0000 7ff31759e700 1 -- 172.21.15.60:0/580911375 >> 172.21.15.60:0/580911375 conn(0x7ff318062090 msgr2=0x7ff318062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:25.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.095+0000 7ff31759e700 1 -- 172.21.15.60:0/580911375 shutdown_connections 2022-06-29T16:00:25.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.095+0000 7ff31759e700 1 -- 172.21.15.60:0/580911375 wait complete. 2022-06-29T16:00:25.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.099+0000 7ff31759e700 1 Processor -- start 2022-06-29T16:00:25.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.099+0000 7ff31759e700 1 -- start start 2022-06-29T16:00:25.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.099+0000 7ff31759e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3180b2c50 0x7ff318143ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:25.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.099+0000 7ff31759e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3180b48b0 0x7ff318151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:25.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.099+0000 7ff31759e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3180b5eb0 0x7ff318146250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:25.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.099+0000 7ff31759e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3180ae250 con 0x7ff3180b5eb0 2022-06-29T16:00:25.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.099+0000 7ff31759e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3180ade50 con 0x7ff3180b48b0 2022-06-29T16:00:25.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.099+0000 7ff31759e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff318145b10 con 0x7ff3180b2c50 2022-06-29T16:00:25.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.099+0000 7ff30dd9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3180b48b0 0x7ff318151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:25.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.099+0000 7ff316d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3180b5eb0 0x7ff318146250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:25.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.099+0000 7ff30dd9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3180b48b0 0x7ff318151d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:53318/0 (socket says 172.21.15.60:53318) 2022-06-29T16:00:25.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.099+0000 7ff30dd9b700 1 -- 172.21.15.60:0/4070787530 learned_addr learned my addr 172.21.15.60:0/4070787530 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:25.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.099+0000 7ff31659c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3180b2c50 0x7ff318143ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:25.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.099+0000 7ff30dd9b700 1 -- 172.21.15.60:0/4070787530 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3180b2c50 msgr2=0x7ff318143ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:25.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.099+0000 7ff30dd9b700 1 --2- 172.21.15.60:0/4070787530 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3180b2c50 0x7ff318143ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:25.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.099+0000 7ff30dd9b700 1 -- 172.21.15.60:0/4070787530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3180b5eb0 msgr2=0x7ff318146250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:25.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.099+0000 7ff30dd9b700 1 --2- 172.21.15.60:0/4070787530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3180b5eb0 0x7ff318146250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:25.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.099+0000 7ff30dd9b700 1 -- 172.21.15.60:0/4070787530 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff30800b180 con 0x7ff3180b48b0 2022-06-29T16:00:25.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.099+0000 7ff316d9d700 1 --2- 172.21.15.60:0/4070787530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3180b5eb0 0x7ff318146250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:00:25.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.103+0000 7ff30dd9b700 1 --2- 172.21.15.60:0/4070787530 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3180b48b0 0x7ff318151d70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7ff318062fa0 tx=0x7ff300007c20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:25.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.103+0000 7ff30ffff700 1 -- 172.21.15.60:0/4070787530 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff300012400 con 0x7ff3180b48b0 2022-06-29T16:00:25.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.103+0000 7ff30ffff700 1 -- 172.21.15.60:0/4070787530 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff300012560 con 0x7ff3180b48b0 2022-06-29T16:00:25.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.103+0000 7ff31759e700 1 -- 172.21.15.60:0/4070787530 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff318145db0 con 0x7ff3180b48b0 2022-06-29T16:00:25.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.103+0000 7ff30ffff700 1 -- 172.21.15.60:0/4070787530 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3000128c0 con 0x7ff3180b48b0 2022-06-29T16:00:25.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.103+0000 7ff31759e700 1 -- 172.21.15.60:0/4070787530 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3181445f0 con 0x7ff3180b48b0 2022-06-29T16:00:25.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.103+0000 7ff31759e700 1 -- 172.21.15.60:0/4070787530 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3181525a0 con 0x7ff3180b48b0 2022-06-29T16:00:25.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.103+0000 7ff30ffff700 1 -- 172.21.15.60:0/4070787530 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff300019030 con 0x7ff3180b48b0 2022-06-29T16:00:25.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.103+0000 7ff30ffff700 1 --2- 172.21.15.60:0/4070787530 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff2f00539e0 0x7ff2f0055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:25.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.103+0000 7ff30ffff700 1 -- 172.21.15.60:0/4070787530 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff30006b230 con 0x7ff3180b48b0 2022-06-29T16:00:25.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.103+0000 7ff31659c700 1 --2- 172.21.15.60:0/4070787530 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff2f00539e0 0x7ff2f0055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:25.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.103+0000 7ff31659c700 1 --2- 172.21.15.60:0/4070787530 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff2f00539e0 0x7ff2f0055ea0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7ff30801c000 tx=0x7ff308015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:25.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.115+0000 7ff30ffff700 1 -- 172.21.15.60:0/4070787530 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff3181525a0 con 0x7ff3180b48b0 2022-06-29T16:00:25.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.243+0000 7ff31759e700 1 -- 172.21.15.60:0/4070787530 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff3181525a0 con 0x7ff3180b48b0 2022-06-29T16:00:25.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.243+0000 7ff30ffff700 1 -- 172.21.15.60:0/4070787530 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v43) v1 ==== 100+0+10 (secure 0 0 0) 0x7ff3181525a0 con 0x7ff3180b48b0 2022-06-29T16:00:25.254 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:00:25.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.247+0000 7ff31759e700 1 -- 172.21.15.60:0/4070787530 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff2f00539e0 msgr2=0x7ff2f0055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:25.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.247+0000 7ff31759e700 1 --2- 172.21.15.60:0/4070787530 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff2f00539e0 0x7ff2f0055ea0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7ff30801c000 tx=0x7ff308015000).stop 2022-06-29T16:00:25.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.247+0000 7ff31759e700 1 -- 172.21.15.60:0/4070787530 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3180b48b0 msgr2=0x7ff318151d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:25.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.247+0000 7ff31759e700 1 --2- 172.21.15.60:0/4070787530 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3180b48b0 0x7ff318151d70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7ff318062fa0 tx=0x7ff300007c20).stop 2022-06-29T16:00:25.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.247+0000 7ff31759e700 1 -- 172.21.15.60:0/4070787530 shutdown_connections 2022-06-29T16:00:25.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.247+0000 7ff31759e700 1 --2- 172.21.15.60:0/4070787530 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff2f00539e0 0x7ff2f0055ea0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:25.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.247+0000 7ff31759e700 1 --2- 172.21.15.60:0/4070787530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3180b5eb0 0x7ff318146250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:25.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.247+0000 7ff31759e700 1 --2- 172.21.15.60:0/4070787530 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3180b48b0 0x7ff318151d70 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:25.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.247+0000 7ff31759e700 1 --2- 172.21.15.60:0/4070787530 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3180b2c50 0x7ff318143ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:25.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.247+0000 7ff31759e700 1 -- 172.21.15.60:0/4070787530 >> 172.21.15.60:0/4070787530 conn(0x7ff318062090 msgr2=0x7ff31805bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:25.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.247+0000 7ff31759e700 1 -- 172.21.15.60:0/4070787530 shutdown_connections 2022-06-29T16:00:25.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.247+0000 7ff31759e700 1 -- 172.21.15.60:0/4070787530 wait complete. 2022-06-29T16:00:25.273 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-06-29T16:00:25.273 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:00:25.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.367+0000 7fbdeae25700 1 Processor -- start 2022-06-29T16:00:25.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.367+0000 7fbdeae25700 1 -- start start 2022-06-29T16:00:25.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.367+0000 7fbdeae25700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbde405d070 0x7fbde414aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:25.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.367+0000 7fbdeae25700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbde40b2130 0x7fbde40aca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:25.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.367+0000 7fbdeae25700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde405d450 con 0x7fbde414af80 2022-06-29T16:00:25.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.367+0000 7fbdeae25700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbde40adf90 con 0x7fbde40b2130 2022-06-29T16:00:25.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.367+0000 7fbdeae25700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde40b25d0 con 0x7fbde405d070 2022-06-29T16:00:25.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.371+0000 7fbde9e23700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbde405d070 0x7fbde414aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:25.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.371+0000 7fbde9622700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbde40b2130 0x7fbde40aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:25.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.371+0000 7fbdea624700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fbde414af80 0x7fbde40acf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:59630/0 (socket says 172.21.15.60:59630) 2022-06-29T16:00:25.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.371+0000 7fbdea624700 1 -- 172.21.15.60:0/4213365386 learned_addr learned my addr 172.21.15.60:0/4213365386 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:00:25.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.371+0000 7fbde8e21700 1 -- 172.21.15.60:0/4213365386 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 263426226 0 0) 0x7fbde405d450 con 0x7fbde414af80 2022-06-29T16:00:25.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.371+0000 7fbde9e23700 1 -- 172.21.15.60:0/4213365386 >> v2:172.21.15.92:3300/0 conn(0x7fbde40b2130 msgr2=0x7fbde40aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:25.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.371+0000 7fbde9e23700 1 --2- 172.21.15.60:0/4213365386 >> v2:172.21.15.92:3300/0 conn(0x7fbde40b2130 0x7fbde40aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:25.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.371+0000 7fbde9e23700 1 -- 172.21.15.60:0/4213365386 >> v1:172.21.15.92:6789/0 conn(0x7fbde414af80 legacy=0x7fbde40acf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:25.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.371+0000 7fbde9e23700 1 -- 172.21.15.60:0/4213365386 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde414a350 con 0x7fbde405d070 2022-06-29T16:00:25.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.371+0000 7fbde9622700 1 --2- 172.21.15.60:0/4213365386 >> v2:172.21.15.92:3300/0 conn(0x7fbde40b2130 0x7fbde40aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:00:25.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.371+0000 7fbde9e23700 1 --2- 172.21.15.60:0/4213365386 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbde405d070 0x7fbde414aa90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fbddc0032f0 tx=0x7fbddc0091d0).ready entity=mon.2 client_cookie=7359804f24894266 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:25.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.371+0000 7fbde8e21700 1 -- 172.21.15.60:0/4213365386 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbddc00b6c0 con 0x7fbde405d070 2022-06-29T16:00:25.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.371+0000 7fbde8e21700 1 -- 172.21.15.60:0/4213365386 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbddc00b820 con 0x7fbde405d070 2022-06-29T16:00:25.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.371+0000 7fbde8e21700 1 -- 172.21.15.60:0/4213365386 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbddc00b990 con 0x7fbde405d070 2022-06-29T16:00:25.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.371+0000 7fbdeae25700 1 -- 172.21.15.60:0/4213365386 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbde405d070 msgr2=0x7fbde414aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:25.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.371+0000 7fbdeae25700 1 --2- 172.21.15.60:0/4213365386 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbde405d070 0x7fbde414aa90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fbddc0032f0 tx=0x7fbddc0091d0).stop 2022-06-29T16:00:25.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.371+0000 7fbdeae25700 1 -- 172.21.15.60:0/4213365386 shutdown_connections 2022-06-29T16:00:25.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.371+0000 7fbdeae25700 1 --2- 172.21.15.60:0/4213365386 >> v2:172.21.15.92:3300/0 conn(0x7fbde40b2130 0x7fbde40aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:25.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.371+0000 7fbdeae25700 1 --2- 172.21.15.60:0/4213365386 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbde405d070 0x7fbde414aa90 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:25.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.371+0000 7fbdeae25700 1 -- 172.21.15.60:0/4213365386 >> 172.21.15.60:0/4213365386 conn(0x7fbde40b5890 msgr2=0x7fbde40b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:25.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.371+0000 7fbdeae25700 1 -- 172.21.15.60:0/4213365386 shutdown_connections 2022-06-29T16:00:25.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.371+0000 7fbdeae25700 1 -- 172.21.15.60:0/4213365386 wait complete. 2022-06-29T16:00:25.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.371+0000 7fbdeae25700 1 Processor -- start 2022-06-29T16:00:25.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.371+0000 7fbdeae25700 1 -- start start 2022-06-29T16:00:25.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.375+0000 7fbdeae25700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbde405d070 0x7fbde4143e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:25.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.375+0000 7fbdeae25700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbde40b2130 0x7fbde4156090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:25.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.375+0000 7fbdeae25700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbde414af80 0x7fbde4159c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:25.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.375+0000 7fbdeae25700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde414d670 con 0x7fbde40b2130 2022-06-29T16:00:25.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.375+0000 7fbdeae25700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde414d3d0 con 0x7fbde414af80 2022-06-29T16:00:25.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.375+0000 7fbdeae25700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde4159770 con 0x7fbde405d070 2022-06-29T16:00:25.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.375+0000 7fbdea624700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbde414af80 0x7fbde4159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:25.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.375+0000 7fbde9622700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbde40b2130 0x7fbde4156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:25.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.375+0000 7fbde9e23700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbde405d070 0x7fbde4143e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:25.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.375+0000 7fbdea624700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbde414af80 0x7fbde4159c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:53334/0 (socket says 172.21.15.60:53334) 2022-06-29T16:00:25.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.375+0000 7fbdea624700 1 -- 172.21.15.60:0/1556266982 learned_addr learned my addr 172.21.15.60:0/1556266982 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:25.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.375+0000 7fbdea624700 1 -- 172.21.15.60:0/1556266982 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbde405d070 msgr2=0x7fbde4143e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:25.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.375+0000 7fbdea624700 1 --2- 172.21.15.60:0/1556266982 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbde405d070 0x7fbde4143e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:25.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.375+0000 7fbdea624700 1 -- 172.21.15.60:0/1556266982 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbde40b2130 msgr2=0x7fbde4156090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:25.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.375+0000 7fbdea624700 1 --2- 172.21.15.60:0/1556266982 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbde40b2130 0x7fbde4156090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:25.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.375+0000 7fbdea624700 1 -- 172.21.15.60:0/1556266982 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd400b180 con 0x7fbde414af80 2022-06-29T16:00:25.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.375+0000 7fbdea624700 1 --2- 172.21.15.60:0/1556266982 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbde414af80 0x7fbde4159c10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fbde0000bc0 tx=0x7fbde0007980).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:25.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.375+0000 7fbddaffd700 1 -- 172.21.15.60:0/1556266982 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbde000cda0 con 0x7fbde414af80 2022-06-29T16:00:25.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.375+0000 7fbddaffd700 1 -- 172.21.15.60:0/1556266982 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde00070d0 con 0x7fbde414af80 2022-06-29T16:00:25.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.375+0000 7fbdeae25700 1 -- 172.21.15.60:0/1556266982 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbddc009470 con 0x7fbde414af80 2022-06-29T16:00:25.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.375+0000 7fbddaffd700 1 -- 172.21.15.60:0/1556266982 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbde0007430 con 0x7fbde414af80 2022-06-29T16:00:25.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.375+0000 7fbdeae25700 1 -- 172.21.15.60:0/1556266982 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbde4066c60 con 0x7fbde414af80 2022-06-29T16:00:25.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.375+0000 7fbdeae25700 1 -- 172.21.15.60:0/1556266982 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbde414a5f0 con 0x7fbde414af80 2022-06-29T16:00:25.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.379+0000 7fbddaffd700 1 -- 172.21.15.60:0/1556266982 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fbde0010070 con 0x7fbde414af80 2022-06-29T16:00:25.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.379+0000 7fbddaffd700 1 --2- 172.21.15.60:0/1556266982 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbdcc053b00 0x7fbdcc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:25.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.379+0000 7fbddaffd700 1 -- 172.21.15.60:0/1556266982 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbde006bf40 con 0x7fbde414af80 2022-06-29T16:00:25.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.379+0000 7fbde9e23700 1 --2- 172.21.15.60:0/1556266982 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbdcc053b00 0x7fbdcc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:25.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.379+0000 7fbde9e23700 1 --2- 172.21.15.60:0/1556266982 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbdcc053b00 0x7fbdcc055fc0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fbddc01b000 tx=0x7fbddc016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:25.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.387+0000 7fbddaffd700 1 -- 172.21.15.60:0/1556266982 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbde0042d50 con 0x7fbde414af80 2022-06-29T16:00:25.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.551+0000 7fbdeae25700 1 -- 172.21.15.60:0/1556266982 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbde4159930 con 0x7fbde414af80 2022-06-29T16:00:25.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.551+0000 7fbddaffd700 1 -- 172.21.15.60:0/1556266982 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+2138 (secure 0 0 0) 0x7fbde0046370 con 0x7fbde414af80 2022-06-29T16:00:25.562 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:00:25.563 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":15,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4937,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/1673184114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1673184114},{"type":"v1","addr":"172.21.15.92:6835","nonce":1673184114}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":13}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:00:20.685676+0000","modified":"2022-06-29T16:00:21.689299+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5308":{"gid":5308,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/1964426267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1964426267},{"type":"v1","addr":"172.21.15.60:6833","nonce":1964426267}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-06-29T16:00:25.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.555+0000 7fbdeae25700 1 -- 172.21.15.60:0/1556266982 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbdcc053b00 msgr2=0x7fbdcc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:25.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.555+0000 7fbdeae25700 1 --2- 172.21.15.60:0/1556266982 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbdcc053b00 0x7fbdcc055fc0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fbddc01b000 tx=0x7fbddc016000).stop 2022-06-29T16:00:25.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.555+0000 7fbdeae25700 1 -- 172.21.15.60:0/1556266982 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbde414af80 msgr2=0x7fbde4159c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:25.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.555+0000 7fbdeae25700 1 --2- 172.21.15.60:0/1556266982 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbde414af80 0x7fbde4159c10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fbde0000bc0 tx=0x7fbde0007980).stop 2022-06-29T16:00:25.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.555+0000 7fbdeae25700 1 -- 172.21.15.60:0/1556266982 shutdown_connections 2022-06-29T16:00:25.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.555+0000 7fbdeae25700 1 --2- 172.21.15.60:0/1556266982 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbdcc053b00 0x7fbdcc055fc0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:25.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.555+0000 7fbdeae25700 1 --2- 172.21.15.60:0/1556266982 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbde414af80 0x7fbde4159c10 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:25.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.555+0000 7fbdeae25700 1 --2- 172.21.15.60:0/1556266982 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbde40b2130 0x7fbde4156090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:25.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.555+0000 7fbdeae25700 1 --2- 172.21.15.60:0/1556266982 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbde405d070 0x7fbde4143e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:25.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.555+0000 7fbdeae25700 1 -- 172.21.15.60:0/1556266982 >> 172.21.15.60:0/1556266982 conn(0x7fbde40b5890 msgr2=0x7fbde40b4370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:25.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.555+0000 7fbdeae25700 1 -- 172.21.15.60:0/1556266982 shutdown_connections 2022-06-29T16:00:25.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:25.555+0000 7fbdeae25700 1 -- 172.21.15.60:0/1556266982 wait complete. 2022-06-29T16:00:25.570 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 15 2022-06-29T16:00:25.781 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:00:20.685676+0000', 'modified': '2022-06-29T16:00:21.689299+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5308}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5308': {'gid': 5308, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.60:6833/1964426267', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6832', 'nonce': 1964426267}, {'type': 'v1', 'addr': '172.21.15.60:6833', 'nonce': 1964426267}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:00:25.782 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:00:25.783 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-06-29T16:00:25.901 INFO:teuthology.orchestra.run.smithi060.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":5308,"want_state":"up:active","state":"up:active","rank_uptime":5.1964488260000001,"mdsmap_epoch":15,"osdmap_epoch":43,"osdmap_epoch_barrier":42,"uptime":10.148679339999999} 2022-06-29T16:00:25.901 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 5308, "mdsmap_epoch": 15, "osdmap_epoch": 43, "osdmap_epoch_barrier": 42, "rank_uptime": 5.196448826, "state": "up:active", "uptime": 10.14867934, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:00:25.902 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-06-29T16:00:25.903 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-06-29T16:00:25.903 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi121.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-06-29T16:00:25.904 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:00:25.904 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:00:25.914 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:00:25.933 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:00:25.934 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:00:25.945 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:00:25.946 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:00:25.947 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:00:25.948 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:00:25.954 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-06-29T16:00:25.955 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:00:25.958 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:00:25.975 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:00:25.976 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:00:26.032 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:00:26.033 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:00:26.034 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:00:26.034 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:00:26.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:2022-06-29T16:00:26.031+0000 7f97cc767080 -1 init, newargv = 0x55f89c477000 newargc=9ceph-fuse[12432]: starting ceph client 2022-06-29T16:00:26.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr: 2022-06-29T16:00:26.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[12432]: starting fuse 2022-06-29T16:00:27.142 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:00:27.161 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:00:27.162 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:00:27.221 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:00:27.223 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:00:27.224 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:00:27.224 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:00:27.231 INFO:teuthology.orchestra.run.smithi121.stdout:52 2022-06-29T16:00:27.232 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-06-29T16:00:27.232 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:00:27.233 DEBUG:teuthology.orchestra.run.smithi121:> sudo python3 -c ' 2022-06-29T16:00:27.233 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2022-06-29T16:00:27.234 DEBUG:teuthology.orchestra.run.smithi121:> import re 2022-06-29T16:00:27.234 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-06-29T16:00:27.234 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2022-06-29T16:00:27.235 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:00:27.235 DEBUG:teuthology.orchestra.run.smithi121:> def find_socket(client_name): 2022-06-29T16:00:27.236 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-06-29T16:00:27.236 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2022-06-29T16:00:27.237 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:00:27.237 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2022-06-29T16:00:27.237 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2022-06-29T16:00:27.238 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2022-06-29T16:00:27.238 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2022-06-29T16:00:27.239 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:00:27.239 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2022-06-29T16:00:27.240 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-06-29T16:00:27.240 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2022-06-29T16:00:27.240 DEBUG:teuthology.orchestra.run.smithi121:> return f 2022-06-29T16:00:27.241 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-06-29T16:00:27.241 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:00:27.242 DEBUG:teuthology.orchestra.run.smithi121:> print(find_socket("client.0")) 2022-06-29T16:00:27.242 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-06-29T16:00:27.286 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.0.12432.asok 2022-06-29T16:00:27.292 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.12432.asok 2022-06-29T16:00:27.293 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:00:27.294 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.12432.asok status 2022-06-29T16:00:27.428 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2022-06-29T16:00:27.429 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2022-06-29T16:00:27.429 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "f25ffac72e5a9e63860ff7deb6767a11f0559a16", 2022-06-29T16:00:27.429 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 15.2.16-304-gf25ffac7 (f25ffac72e5a9e63860ff7deb6767a11f0559a16) octopus (stable)", 2022-06-29T16:00:27.431 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "0", 2022-06-29T16:00:27.431 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2022-06-29T16:00:27.431 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-06-29T16:00:27.432 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "12432", 2022-06-29T16:00:27.432 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2022-06-29T16:00:27.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:00:27.433 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2022-06-29T16:00:27.434 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2022-06-29T16:00:27.434 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 5382, 2022-06-29T16:00:27.434 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2022-06-29T16:00:27.435 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2022-06-29T16:00:27.435 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2022-06-29T16:00:27.436 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 5382 2022-06-29T16:00:27.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:00:27.437 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-06-29T16:00:27.437 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-06-29T16:00:27.438 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-06-29T16:00:27.438 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2198288093 2022-06-29T16:00:27.439 INFO:teuthology.orchestra.run.smithi121.stdout: } 2022-06-29T16:00:27.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:00:27.439 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-06-29T16:00:27.440 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-06-29T16:00:27.440 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-06-29T16:00:27.440 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2198288093 2022-06-29T16:00:27.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:00:27.441 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.5382 172.21.15.121:0/2198288093", 2022-06-29T16:00:27.441 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "172.21.15.121:0/2198288093", 2022-06-29T16:00:27.441 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2022-06-29T16:00:27.442 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 15, 2022-06-29T16:00:27.442 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 43, 2022-06-29T16:00:27.442 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2022-06-29T16:00:27.442 INFO:teuthology.orchestra.run.smithi121.stdout: "blacklisted": false, 2022-06-29T16:00:27.443 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2022-06-29T16:00:27.443 INFO:teuthology.orchestra.run.smithi121.stdout:} 2022-06-29T16:00:27.445 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:00:27.446 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:00:27.446 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:00:27.454 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T16:00:27.457 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T16:00:27.457 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:00:27.458 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:00:27.516 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:00:27.517 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:00:27.517 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:00:27.525 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T16:00:27.527 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T16:00:27.528 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-06-29T16:00:27.528 DEBUG:teuthology.orchestra.run.smithi121:> import sys 2022-06-29T16:00:27.528 DEBUG:teuthology.orchestra.run.smithi121:> import time 2022-06-29T16:00:27.529 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-06-29T16:00:27.529 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:00:27.530 DEBUG:teuthology.orchestra.run.smithi121:> n = 1000 2022-06-29T16:00:27.530 DEBUG:teuthology.orchestra.run.smithi121:> abs_path = "/home/ubuntu/cephtest/mnt.0/dc-dir/dc-file" 2022-06-29T16:00:27.530 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:00:27.531 DEBUG:teuthology.orchestra.run.smithi121:> if not os.path.exists(os.path.dirname(abs_path)): 2022-06-29T16:00:27.531 DEBUG:teuthology.orchestra.run.smithi121:> os.makedirs(os.path.dirname(abs_path)) 2022-06-29T16:00:27.532 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:00:27.532 DEBUG:teuthology.orchestra.run.smithi121:> for i in range(0, n): 2022-06-29T16:00:27.532 DEBUG:teuthology.orchestra.run.smithi121:> fname = "{0}_{1}".format(abs_path, i) 2022-06-29T16:00:27.533 DEBUG:teuthology.orchestra.run.smithi121:> with open(fname, '"'"'w'"'"') as f: 2022-06-29T16:00:27.533 DEBUG:teuthology.orchestra.run.smithi121:> f.write('"'"'content'"'"') 2022-06-29T16:00:27.534 DEBUG:teuthology.orchestra.run.smithi121:> if True: 2022-06-29T16:00:27.534 DEBUG:teuthology.orchestra.run.smithi121:> f.flush() 2022-06-29T16:00:27.534 DEBUG:teuthology.orchestra.run.smithi121:> os.fsync(f.fileno()) 2022-06-29T16:00:27.535 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-06-29T16:00:28.230 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:00:28.233 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:00:28.235 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:00:28.238 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:00:28.240 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:00:36.850 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:00:36.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.955+0000 7fdb97556700 1 Processor -- start 2022-06-29T16:00:36.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.955+0000 7fdb97556700 1 -- start start 2022-06-29T16:00:36.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.955+0000 7fdb97556700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb901390c0 0x7fdb9013f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:36.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.955+0000 7fdb97556700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdb900642d0 0x7fdb90065660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:36.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.955+0000 7fdb97556700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb9013fe40 con 0x7fdb900b13a0 2022-06-29T16:00:36.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.955+0000 7fdb97556700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb900b1780 con 0x7fdb900642d0 2022-06-29T16:00:36.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.955+0000 7fdb97556700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb90064c80 con 0x7fdb901390c0 2022-06-29T16:00:36.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.955+0000 7fdb952f2700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fdb900b13a0 0x7fdb901389b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55396/0 (socket says 172.21.15.60:55396) 2022-06-29T16:00:36.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.955+0000 7fdb952f2700 1 -- 172.21.15.60:0/706435959 learned_addr learned my addr 172.21.15.60:0/706435959 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:00:36.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.955+0000 7fdb94af1700 1 --2- 172.21.15.60:0/706435959 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb901390c0 0x7fdb9013f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:36.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.955+0000 7fdb95af3700 1 --2- 172.21.15.60:0/706435959 >> v2:172.21.15.92:3300/0 conn(0x7fdb900642d0 0x7fdb90065660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:36.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.955+0000 7fdb7ffff700 1 -- 172.21.15.60:0/706435959 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3644234525 0 0) 0x7fdb9013fe40 con 0x7fdb900b13a0 2022-06-29T16:00:36.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.955+0000 7fdb7ffff700 1 -- 172.21.15.60:0/706435959 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb78003260 con 0x7fdb900b13a0 2022-06-29T16:00:36.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.955+0000 7fdb7ffff700 1 -- 172.21.15.60:0/706435959 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 554068928 0 0) 0x7fdb78003260 con 0x7fdb900b13a0 2022-06-29T16:00:36.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.955+0000 7fdb7ffff700 1 -- 172.21.15.60:0/706435959 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb901390c0 msgr2=0x7fdb9013f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:36.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.955+0000 7fdb7ffff700 1 --2- 172.21.15.60:0/706435959 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb901390c0 0x7fdb9013f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:36.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.955+0000 7fdb7ffff700 1 -- 172.21.15.60:0/706435959 >> v2:172.21.15.92:3300/0 conn(0x7fdb900642d0 msgr2=0x7fdb90065660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:36.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.955+0000 7fdb7ffff700 1 --2- 172.21.15.60:0/706435959 >> v2:172.21.15.92:3300/0 conn(0x7fdb900642d0 0x7fdb90065660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:36.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.955+0000 7fdb7ffff700 1 -- 172.21.15.60:0/706435959 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb90064f20 con 0x7fdb900b13a0 2022-06-29T16:00:36.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.955+0000 7fdb7ffff700 1 -- 172.21.15.60:0/706435959 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7fdb8c001680 con 0x7fdb900b13a0 2022-06-29T16:00:36.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.955+0000 7fdb95af3700 1 --2- 172.21.15.60:0/706435959 >> v2:172.21.15.92:3300/0 conn(0x7fdb900642d0 0x7fdb90065660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:00:36.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.955+0000 7fdb7ffff700 1 -- 172.21.15.60:0/706435959 >> v1:172.21.15.60:6789/0 conn(0x7fdb900b13a0 legacy=0x7fdb901389b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:36.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.955+0000 7fdb7ffff700 1 --2- 172.21.15.60:0/706435959 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb78003750 0x7fdb78003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:36.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.959+0000 7fdb7ffff700 1 --2- 172.21.15.60:0/706435959 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb901390c0 0x7fdb78004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:36.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.959+0000 7fdb7ffff700 1 --2- 172.21.15.60:0/706435959 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb780049f0 0x7fdb78004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:36.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.959+0000 7fdb7ffff700 1 -- 172.21.15.60:0/706435959 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb900b1780 con 0x7fdb780049f0 2022-06-29T16:00:36.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.959+0000 7fdb7ffff700 1 -- 172.21.15.60:0/706435959 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb90064c80 con 0x7fdb78003750 2022-06-29T16:00:36.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.959+0000 7fdb7ffff700 1 -- 172.21.15.60:0/706435959 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb7800b750 con 0x7fdb901390c0 2022-06-29T16:00:36.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.959+0000 7fdb95af3700 1 --2- 172.21.15.60:0/706435959 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb780049f0 0x7fdb78004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:36.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.959+0000 7fdb952f2700 1 --2- 172.21.15.60:0/706435959 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb78003750 0x7fdb78003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:36.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.959+0000 7fdb94af1700 1 --2- 172.21.15.60:0/706435959 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb901390c0 0x7fdb78004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:36.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.959+0000 7fdb95af3700 1 -- 172.21.15.60:0/706435959 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb901390c0 msgr2=0x7fdb78004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:36.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.959+0000 7fdb95af3700 1 --2- 172.21.15.60:0/706435959 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb901390c0 0x7fdb78004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:36.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.959+0000 7fdb95af3700 1 -- 172.21.15.60:0/706435959 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb78003750 msgr2=0x7fdb78003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:36.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.959+0000 7fdb95af3700 1 --2- 172.21.15.60:0/706435959 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb78003750 0x7fdb78003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:36.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.959+0000 7fdb95af3700 1 -- 172.21.15.60:0/706435959 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb78003260 con 0x7fdb780049f0 2022-06-29T16:00:36.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.959+0000 7fdb95af3700 1 --2- 172.21.15.60:0/706435959 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb780049f0 0x7fdb78004e70 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fdb8800c000 tx=0x7fdb880091d0).ready entity=mon.0 client_cookie=fff2b1742cf9c1da server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:36.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.959+0000 7fdb7ffff700 1 -- 172.21.15.60:0/706435959 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb8800e920 con 0x7fdb780049f0 2022-06-29T16:00:36.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.959+0000 7fdb7ffff700 1 -- 172.21.15.60:0/706435959 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb8800ea80 con 0x7fdb780049f0 2022-06-29T16:00:36.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.959+0000 7fdb7ffff700 1 -- 172.21.15.60:0/706435959 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb8800ede0 con 0x7fdb780049f0 2022-06-29T16:00:36.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.959+0000 7fdb97556700 1 -- 172.21.15.60:0/706435959 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb780049f0 msgr2=0x7fdb78004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:36.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.959+0000 7fdb97556700 1 --2- 172.21.15.60:0/706435959 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb780049f0 0x7fdb78004e70 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fdb8800c000 tx=0x7fdb880091d0).stop 2022-06-29T16:00:36.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.959+0000 7fdb95af3700 1 -- 172.21.15.60:0/706435959 reap_dead start 2022-06-29T16:00:36.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.959+0000 7fdb97556700 1 -- 172.21.15.60:0/706435959 shutdown_connections 2022-06-29T16:00:36.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.959+0000 7fdb97556700 1 -- 172.21.15.60:0/706435959 >> 172.21.15.60:0/706435959 conn(0x7fdb900632c0 msgr2=0x7fdb900b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:36.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.959+0000 7fdb97556700 1 -- 172.21.15.60:0/706435959 shutdown_connections 2022-06-29T16:00:36.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.959+0000 7fdb97556700 1 -- 172.21.15.60:0/706435959 wait complete. 2022-06-29T16:00:36.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.959+0000 7fdb97556700 1 Processor -- start 2022-06-29T16:00:36.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.963+0000 7fdb97556700 1 -- start start 2022-06-29T16:00:36.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.963+0000 7fdb97556700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb780049f0 0x7fdb900b3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:36.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.963+0000 7fdb97556700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb900ae830 0x7fdb900af7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:36.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.963+0000 7fdb97556700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb9013dac0 0x7fdb9013df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:36.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.963+0000 7fdb97556700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9013b580 con 0x7fdb900ae830 2022-06-29T16:00:36.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.963+0000 7fdb97556700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb900aca20 con 0x7fdb9013dac0 2022-06-29T16:00:36.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.963+0000 7fdb97556700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb900accc0 con 0x7fdb780049f0 2022-06-29T16:00:36.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.963+0000 7fdb952f2700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb780049f0 0x7fdb900b3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:36.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.963+0000 7fdb952f2700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb780049f0 0x7fdb900b3ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:60416/0 (socket says 172.21.15.60:60416) 2022-06-29T16:00:36.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.963+0000 7fdb952f2700 1 -- 172.21.15.60:0/1959885554 learned_addr learned my addr 172.21.15.60:0/1959885554 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:36.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.963+0000 7fdb95af3700 1 --2- 172.21.15.60:0/1959885554 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb9013dac0 0x7fdb9013df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:36.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.963+0000 7fdb94af1700 1 --2- 172.21.15.60:0/1959885554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb900ae830 0x7fdb900af7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:36.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.963+0000 7fdb952f2700 1 -- 172.21.15.60:0/1959885554 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb9013dac0 msgr2=0x7fdb9013df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:36.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.963+0000 7fdb952f2700 1 --2- 172.21.15.60:0/1959885554 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb9013dac0 0x7fdb9013df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:36.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.963+0000 7fdb952f2700 1 -- 172.21.15.60:0/1959885554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb900ae830 msgr2=0x7fdb900af7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:36.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.963+0000 7fdb952f2700 1 --2- 172.21.15.60:0/1959885554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb900ae830 0x7fdb900af7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:36.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.963+0000 7fdb952f2700 1 -- 172.21.15.60:0/1959885554 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb900afcb0 con 0x7fdb780049f0 2022-06-29T16:00:36.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.963+0000 7fdb952f2700 1 --2- 172.21.15.60:0/1959885554 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb780049f0 0x7fdb900b3ea0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fdb8c00d000 tx=0x7fdb90067950).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:36.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.963+0000 7fdb7e7fc700 1 -- 172.21.15.60:0/1959885554 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb8c0116b0 con 0x7fdb780049f0 2022-06-29T16:00:36.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.963+0000 7fdb7e7fc700 1 -- 172.21.15.60:0/1959885554 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb8c011810 con 0x7fdb780049f0 2022-06-29T16:00:36.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.963+0000 7fdb97556700 1 -- 172.21.15.60:0/1959885554 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb9013b270 con 0x7fdb780049f0 2022-06-29T16:00:36.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.963+0000 7fdb7e7fc700 1 -- 172.21.15.60:0/1959885554 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb8c011b70 con 0x7fdb780049f0 2022-06-29T16:00:36.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.963+0000 7fdb97556700 1 -- 172.21.15.60:0/1959885554 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb900ae5d0 con 0x7fdb780049f0 2022-06-29T16:00:36.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.967+0000 7fdb97556700 1 -- 172.21.15.60:0/1959885554 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb900651c0 con 0x7fdb780049f0 2022-06-29T16:00:36.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.967+0000 7fdb7e7fc700 1 -- 172.21.15.60:0/1959885554 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdb8c01b030 con 0x7fdb780049f0 2022-06-29T16:00:36.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.967+0000 7fdb7e7fc700 1 --2- 172.21.15.60:0/1959885554 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdb700539e0 0x7fdb70055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:36.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.967+0000 7fdb7e7fc700 1 -- 172.21.15.60:0/1959885554 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fdb8c06b910 con 0x7fdb780049f0 2022-06-29T16:00:36.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.967+0000 7fdb94af1700 1 --2- 172.21.15.60:0/1959885554 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdb700539e0 0x7fdb70055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:36.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.967+0000 7fdb94af1700 1 --2- 172.21.15.60:0/1959885554 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdb700539e0 0x7fdb70055ea0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fdb8000b000 tx=0x7fdb80006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:36.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:36.979+0000 7fdb7e7fc700 1 -- 172.21.15.60:0/1959885554 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdb8c042830 con 0x7fdb780049f0 2022-06-29T16:00:37.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.147+0000 7fdb97556700 1 -- 172.21.15.60:0/1959885554 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb900573d0 con 0x7fdb780049f0 2022-06-29T16:00:37.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.147+0000 7fdb7e7fc700 1 -- 172.21.15.60:0/1959885554 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+2138 (secure 0 0 0) 0x7fdb8c019030 con 0x7fdb780049f0 2022-06-29T16:00:37.157 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:00:37.157 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":15,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4937,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/1673184114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1673184114},{"type":"v1","addr":"172.21.15.92:6835","nonce":1673184114}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":13}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:00:20.685676+0000","modified":"2022-06-29T16:00:21.689299+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5308":{"gid":5308,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/1964426267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1964426267},{"type":"v1","addr":"172.21.15.60:6833","nonce":1964426267}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-06-29T16:00:37.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.151+0000 7fdb97556700 1 -- 172.21.15.60:0/1959885554 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdb700539e0 msgr2=0x7fdb70055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:37.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.151+0000 7fdb97556700 1 --2- 172.21.15.60:0/1959885554 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdb700539e0 0x7fdb70055ea0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fdb8000b000 tx=0x7fdb80006df0).stop 2022-06-29T16:00:37.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.151+0000 7fdb97556700 1 -- 172.21.15.60:0/1959885554 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb780049f0 msgr2=0x7fdb900b3ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:37.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.151+0000 7fdb97556700 1 --2- 172.21.15.60:0/1959885554 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb780049f0 0x7fdb900b3ea0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fdb8c00d000 tx=0x7fdb90067950).stop 2022-06-29T16:00:37.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.151+0000 7fdb97556700 1 -- 172.21.15.60:0/1959885554 shutdown_connections 2022-06-29T16:00:37.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.151+0000 7fdb97556700 1 --2- 172.21.15.60:0/1959885554 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdb700539e0 0x7fdb70055ea0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:37.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.151+0000 7fdb97556700 1 --2- 172.21.15.60:0/1959885554 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb9013dac0 0x7fdb9013df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:37.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.151+0000 7fdb97556700 1 --2- 172.21.15.60:0/1959885554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb900ae830 0x7fdb900af7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:37.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.151+0000 7fdb97556700 1 --2- 172.21.15.60:0/1959885554 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb780049f0 0x7fdb900b3ea0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:37.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.151+0000 7fdb97556700 1 -- 172.21.15.60:0/1959885554 >> 172.21.15.60:0/1959885554 conn(0x7fdb900632c0 msgr2=0x7fdb9013e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:37.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.151+0000 7fdb97556700 1 -- 172.21.15.60:0/1959885554 shutdown_connections 2022-06-29T16:00:37.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.151+0000 7fdb97556700 1 -- 172.21.15.60:0/1959885554 wait complete. 2022-06-29T16:00:37.162 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 15 2022-06-29T16:00:37.170 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config set mds_recall_max_caps 20 2022-06-29T16:00:37.269 INFO:teuthology.orchestra.run.smithi060.stdout:{"success":"mds_recall_max_caps = '20' (not observed, change may require restart) "} 2022-06-29T16:00:37.270 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_caps = '20' (not observed, change may require restart) " } 2022-06-29T16:00:37.270 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:00:37.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.363+0000 7fb4dd5ed700 1 Processor -- start 2022-06-29T16:00:37.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.363+0000 7fb4dd5ed700 1 -- start start 2022-06-29T16:00:37.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.363+0000 7fb4dd5ed700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb4d805d070 0x7fb4d814aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:37.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.363+0000 7fb4dd5ed700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb4d80b2130 0x7fb4d80aca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:37.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.363+0000 7fb4dd5ed700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4d805d450 con 0x7fb4d814af80 2022-06-29T16:00:37.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.363+0000 7fb4dd5ed700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4d80adf90 con 0x7fb4d80b2130 2022-06-29T16:00:37.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.363+0000 7fb4dd5ed700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d80b25d0 con 0x7fb4d805d070 2022-06-29T16:00:37.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.363+0000 7fb4d77fe700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb4d80b2130 0x7fb4d80aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:37.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.363+0000 7fb4d77fe700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb4d80b2130 0x7fb4d80aca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:53360/0 (socket says 172.21.15.60:53360) 2022-06-29T16:00:37.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.363+0000 7fb4d77fe700 1 -- 172.21.15.60:0/1199007615 learned_addr learned my addr 172.21.15.60:0/1199007615 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:37.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.363+0000 7fb4d7fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb4d805d070 0x7fb4d814aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:37.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.363+0000 7fb4d77fe700 1 -- 172.21.15.60:0/1199007615 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb4d805d070 msgr2=0x7fb4d814aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:37.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.363+0000 7fb4d77fe700 1 --2- 172.21.15.60:0/1199007615 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb4d805d070 0x7fb4d814aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:37.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.363+0000 7fb4d77fe700 1 -- 172.21.15.60:0/1199007615 >> v1:172.21.15.92:6789/0 conn(0x7fb4d814af80 legacy=0x7fb4d80acf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:37.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.363+0000 7fb4d77fe700 1 -- 172.21.15.60:0/1199007615 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d814a350 con 0x7fb4d80b2130 2022-06-29T16:00:37.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.367+0000 7fb4d77fe700 1 --2- 172.21.15.60:0/1199007615 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4d80b2130 0x7fb4d80aca20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fb4c00032f0 tx=0x7fb4c0009290).ready entity=mon.1 client_cookie=41fc52b2314247bc server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:37.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.367+0000 7fb4d6ffd700 1 -- 172.21.15.60:0/1199007615 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4c000b8a0 con 0x7fb4d80b2130 2022-06-29T16:00:37.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.367+0000 7fb4d6ffd700 1 -- 172.21.15.60:0/1199007615 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c000ba00 con 0x7fb4d80b2130 2022-06-29T16:00:37.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.367+0000 7fb4d6ffd700 1 -- 172.21.15.60:0/1199007615 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4c000bb70 con 0x7fb4d80b2130 2022-06-29T16:00:37.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.367+0000 7fb4dd5ed700 1 -- 172.21.15.60:0/1199007615 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4d80b2130 msgr2=0x7fb4d80aca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:37.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.367+0000 7fb4dd5ed700 1 --2- 172.21.15.60:0/1199007615 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4d80b2130 0x7fb4d80aca20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fb4c00032f0 tx=0x7fb4c0009290).stop 2022-06-29T16:00:37.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.367+0000 7fb4dd5ed700 1 -- 172.21.15.60:0/1199007615 shutdown_connections 2022-06-29T16:00:37.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.367+0000 7fb4dd5ed700 1 --2- 172.21.15.60:0/1199007615 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4d80b2130 0x7fb4d80aca20 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:37.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.367+0000 7fb4dd5ed700 1 --2- 172.21.15.60:0/1199007615 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb4d805d070 0x7fb4d814aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:37.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.367+0000 7fb4dd5ed700 1 -- 172.21.15.60:0/1199007615 >> 172.21.15.60:0/1199007615 conn(0x7fb4d80b5890 msgr2=0x7fb4d80b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:37.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.367+0000 7fb4dd5ed700 1 -- 172.21.15.60:0/1199007615 shutdown_connections 2022-06-29T16:00:37.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.367+0000 7fb4dd5ed700 1 -- 172.21.15.60:0/1199007615 wait complete. 2022-06-29T16:00:37.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.371+0000 7fb4dd5ed700 1 Processor -- start 2022-06-29T16:00:37.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.371+0000 7fb4dd5ed700 1 -- start start 2022-06-29T16:00:37.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.371+0000 7fb4dd5ed700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4d805d070 0x7fb4d80664e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:37.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.371+0000 7fb4dd5ed700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb4d80b2130 0x7fb4d8146450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:37.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.371+0000 7fb4dd5ed700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4d814af80 0x7fb4d8146940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:37.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.371+0000 7fb4dd5ed700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d8118fb0 con 0x7fb4d814af80 2022-06-29T16:00:37.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.371+0000 7fb4dd5ed700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d8066e50 con 0x7fb4d805d070 2022-06-29T16:00:37.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.371+0000 7fb4dd5ed700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d8118d10 con 0x7fb4d80b2130 2022-06-29T16:00:37.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.371+0000 7fb4dcdec700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4d814af80 0x7fb4d8146940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:37.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.371+0000 7fb4dcdec700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4d814af80 0x7fb4d8146940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47340/0 (socket says 172.21.15.60:47340) 2022-06-29T16:00:37.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.371+0000 7fb4dcdec700 1 -- 172.21.15.60:0/3056951920 learned_addr learned my addr 172.21.15.60:0/3056951920 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:37.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.371+0000 7fb4d7fff700 1 --2- 172.21.15.60:0/3056951920 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4d805d070 0x7fb4d80664e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:37.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.371+0000 7fb4d77fe700 1 --2- 172.21.15.60:0/3056951920 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb4d80b2130 0x7fb4d8146450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:37.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.371+0000 7fb4dcdec700 1 -- 172.21.15.60:0/3056951920 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb4d80b2130 msgr2=0x7fb4d8146450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:37.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.371+0000 7fb4dcdec700 1 --2- 172.21.15.60:0/3056951920 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb4d80b2130 0x7fb4d8146450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:37.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.371+0000 7fb4dcdec700 1 -- 172.21.15.60:0/3056951920 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4d805d070 msgr2=0x7fb4d80664e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:37.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.371+0000 7fb4dcdec700 1 --2- 172.21.15.60:0/3056951920 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4d805d070 0x7fb4d80664e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:37.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.371+0000 7fb4dcdec700 1 -- 172.21.15.60:0/3056951920 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c0009530 con 0x7fb4d814af80 2022-06-29T16:00:37.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.371+0000 7fb4dcdec700 1 --2- 172.21.15.60:0/3056951920 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4d814af80 0x7fb4d8146940 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7fb4d814a970 tx=0x7fb4d0007b60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:37.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.371+0000 7fb4d57fa700 1 -- 172.21.15.60:0/3056951920 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4d0013450 con 0x7fb4d814af80 2022-06-29T16:00:37.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.371+0000 7fb4d57fa700 1 -- 172.21.15.60:0/3056951920 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4d00135b0 con 0x7fb4d814af80 2022-06-29T16:00:37.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.371+0000 7fb4dd5ed700 1 -- 172.21.15.60:0/3056951920 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4d8116890 con 0x7fb4d814af80 2022-06-29T16:00:37.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.371+0000 7fb4d57fa700 1 -- 172.21.15.60:0/3056951920 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4d0013910 con 0x7fb4d814af80 2022-06-29T16:00:37.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.371+0000 7fb4dd5ed700 1 -- 172.21.15.60:0/3056951920 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4d8159860 con 0x7fb4d814af80 2022-06-29T16:00:37.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.375+0000 7fb4dd5ed700 1 -- 172.21.15.60:0/3056951920 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4d8118b60 con 0x7fb4d814af80 2022-06-29T16:00:37.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.375+0000 7fb4d57fa700 1 -- 172.21.15.60:0/3056951920 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb4d0010070 con 0x7fb4d814af80 2022-06-29T16:00:37.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.375+0000 7fb4d57fa700 1 --2- 172.21.15.60:0/3056951920 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb4bc053b00 0x7fb4bc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:37.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.375+0000 7fb4d57fa700 1 -- 172.21.15.60:0/3056951920 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb4d006b5b0 con 0x7fb4d814af80 2022-06-29T16:00:37.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.375+0000 7fb4d7fff700 1 --2- 172.21.15.60:0/3056951920 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb4bc053b00 0x7fb4bc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:37.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.375+0000 7fb4d7fff700 1 --2- 172.21.15.60:0/3056951920 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb4bc053b00 0x7fb4bc055fc0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fb4c800b000 tx=0x7fb4c8006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:37.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.379+0000 7fb4d57fa700 1 -- 172.21.15.60:0/3056951920 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb4d00423c0 con 0x7fb4d814af80 2022-06-29T16:00:37.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.543+0000 7fb4dd5ed700 1 -- 172.21.15.60:0/3056951920 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4d814a5f0 con 0x7fb4d814af80 2022-06-29T16:00:37.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.543+0000 7fb4d57fa700 1 -- 172.21.15.60:0/3056951920 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+2138 (secure 0 0 0) 0x7fb4d00459e0 con 0x7fb4d814af80 2022-06-29T16:00:37.553 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:00:37.554 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":15,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4937,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/1673184114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1673184114},{"type":"v1","addr":"172.21.15.92:6835","nonce":1673184114}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":13}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:00:20.685676+0000","modified":"2022-06-29T16:00:21.689299+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5308":{"gid":5308,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/1964426267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1964426267},{"type":"v1","addr":"172.21.15.60:6833","nonce":1964426267}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-06-29T16:00:37.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.547+0000 7fb4dd5ed700 1 -- 172.21.15.60:0/3056951920 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb4bc053b00 msgr2=0x7fb4bc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:37.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.547+0000 7fb4dd5ed700 1 --2- 172.21.15.60:0/3056951920 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb4bc053b00 0x7fb4bc055fc0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fb4c800b000 tx=0x7fb4c8006df0).stop 2022-06-29T16:00:37.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.547+0000 7fb4dd5ed700 1 -- 172.21.15.60:0/3056951920 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4d814af80 msgr2=0x7fb4d8146940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:37.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.547+0000 7fb4dd5ed700 1 --2- 172.21.15.60:0/3056951920 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4d814af80 0x7fb4d8146940 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7fb4d814a970 tx=0x7fb4d0007b60).stop 2022-06-29T16:00:37.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.547+0000 7fb4dd5ed700 1 -- 172.21.15.60:0/3056951920 shutdown_connections 2022-06-29T16:00:37.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.547+0000 7fb4dd5ed700 1 --2- 172.21.15.60:0/3056951920 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb4bc053b00 0x7fb4bc055fc0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:37.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.547+0000 7fb4dd5ed700 1 --2- 172.21.15.60:0/3056951920 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb4d814af80 0x7fb4d8146940 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:37.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.547+0000 7fb4dd5ed700 1 --2- 172.21.15.60:0/3056951920 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb4d80b2130 0x7fb4d8146450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:37.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.547+0000 7fb4dd5ed700 1 --2- 172.21.15.60:0/3056951920 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4d805d070 0x7fb4d80664e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:37.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.547+0000 7fb4dd5ed700 1 -- 172.21.15.60:0/3056951920 >> 172.21.15.60:0/3056951920 conn(0x7fb4d80b5890 msgr2=0x7fb4d8143340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:37.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.547+0000 7fb4dd5ed700 1 -- 172.21.15.60:0/3056951920 shutdown_connections 2022-06-29T16:00:37.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.547+0000 7fb4dd5ed700 1 -- 172.21.15.60:0/3056951920 wait complete. 2022-06-29T16:00:37.560 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 15 2022-06-29T16:00:37.566 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config set mds_recall_max_decay_threshold 400 2022-06-29T16:00:37.670 INFO:teuthology.orchestra.run.smithi060.stdout:{"success":"mds_recall_max_decay_threshold = '400' (not observed, change may require restart) "} 2022-06-29T16:00:37.671 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_decay_threshold = '400' (not observed, change may require restart) " } 2022-06-29T16:00:37.671 INFO:tasks.cephfs.fuse_mount:Killing ceph-fuse connection on ubuntu@smithi121.front.sepia.ceph.com... 2022-06-29T16:00:37.690 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:00:37.800 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.787+0000 7f8510a0b700 1 Processor -- start 2022-06-29T16:00:37.800 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.787+0000 7f8510a0b700 1 -- start start 2022-06-29T16:00:37.801 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.791+0000 7f8510a0b700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f850c0b5eb0 0x7f850c142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:37.801 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.791+0000 7f8510a0b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f850c0b48b0 0x7f850c0b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:37.801 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.791+0000 7f8510a0b700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f850c0b4d10 con 0x7f850c0b2c50 2022-06-29T16:00:37.802 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.791+0000 7f8510a0b700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f850c0b54e0 con 0x7f850c0b48b0 2022-06-29T16:00:37.802 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.791+0000 7f8510a0b700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f850c0b6290 con 0x7f850c0b5eb0 2022-06-29T16:00:37.802 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.791+0000 7f850bfff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f850c0b2c50 0x7f850c145fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55432/0 (socket says 172.21.15.60:55432) 2022-06-29T16:00:37.803 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.791+0000 7f850bfff700 1 -- 172.21.15.60:0/1321438099 learned_addr learned my addr 172.21.15.60:0/1321438099 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:00:37.803 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.791+0000 7f850affd700 1 --2- 172.21.15.60:0/1321438099 >> v2:172.21.15.60:3300/0 conn(0x7f850c0b48b0 0x7f850c0b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:37.803 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.791+0000 7f850b7fe700 1 --2- 172.21.15.60:0/1321438099 >> v2:172.21.15.92:3300/0 conn(0x7f850c0b5eb0 0x7f850c142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:37.803 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.791+0000 7f850a7fc700 1 -- 172.21.15.60:0/1321438099 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3569881041 0 0) 0x7f850c0b4d10 con 0x7f850c0b2c50 2022-06-29T16:00:37.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.791+0000 7f850a7fc700 1 -- 172.21.15.60:0/1321438099 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84f8003260 con 0x7f850c0b2c50 2022-06-29T16:00:37.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.791+0000 7f850a7fc700 1 -- 172.21.15.60:0/1321438099 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1620615429 0 0) 0x7f84f8003260 con 0x7f850c0b2c50 2022-06-29T16:00:37.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.791+0000 7f850affd700 1 -- 172.21.15.60:0/1321438099 >> v2:172.21.15.92:3300/0 conn(0x7f850c0b5eb0 msgr2=0x7f850c142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:37.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.791+0000 7f850affd700 1 --2- 172.21.15.60:0/1321438099 >> v2:172.21.15.92:3300/0 conn(0x7f850c0b5eb0 0x7f850c142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:37.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.791+0000 7f850affd700 1 -- 172.21.15.60:0/1321438099 >> v1:172.21.15.60:6789/0 conn(0x7f850c0b2c50 legacy=0x7f850c145fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:37.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.791+0000 7f850affd700 1 -- 172.21.15.60:0/1321438099 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850c142f10 con 0x7f850c0b48b0 2022-06-29T16:00:37.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.791+0000 7f850b7fe700 1 --2- 172.21.15.60:0/1321438099 >> v2:172.21.15.92:3300/0 conn(0x7f850c0b5eb0 0x7f850c142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:00:37.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.791+0000 7f850affd700 1 --2- 172.21.15.60:0/1321438099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f850c0b48b0 0x7f850c0b05e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f8504001b80 tx=0x7f8500006df0).ready entity=mon.0 client_cookie=3fce3211f53a73f3 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:37.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.791+0000 7f850a7fc700 1 -- 172.21.15.60:0/1321438099 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85040019e0 con 0x7f850c0b48b0 2022-06-29T16:00:37.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.791+0000 7f850a7fc700 1 -- 172.21.15.60:0/1321438099 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8500000e90 con 0x7f850c0b48b0 2022-06-29T16:00:37.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.791+0000 7f850a7fc700 1 -- 172.21.15.60:0/1321438099 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8500012460 con 0x7f850c0b48b0 2022-06-29T16:00:37.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.791+0000 7f8510a0b700 1 -- 172.21.15.60:0/1321438099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f850c0b48b0 msgr2=0x7f850c0b05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:37.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.791+0000 7f8510a0b700 1 --2- 172.21.15.60:0/1321438099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f850c0b48b0 0x7f850c0b05e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f8504001b80 tx=0x7f8500006df0).stop 2022-06-29T16:00:37.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.791+0000 7f8510a0b700 1 -- 172.21.15.60:0/1321438099 shutdown_connections 2022-06-29T16:00:37.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.791+0000 7f8510a0b700 1 --2- 172.21.15.60:0/1321438099 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f850c0b48b0 0x7f850c0b05e0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:37.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.791+0000 7f8510a0b700 1 --2- 172.21.15.60:0/1321438099 >> v2:172.21.15.92:3300/0 conn(0x7f850c0b5eb0 0x7f850c142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:37.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.791+0000 7f8510a0b700 1 -- 172.21.15.60:0/1321438099 >> 172.21.15.60:0/1321438099 conn(0x7f850c062090 msgr2=0x7f850c062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:37.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.791+0000 7f8510a0b700 1 -- 172.21.15.60:0/1321438099 shutdown_connections 2022-06-29T16:00:37.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.791+0000 7f8510a0b700 1 -- 172.21.15.60:0/1321438099 wait complete. 2022-06-29T16:00:37.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.795+0000 7f8510a0b700 1 Processor -- start 2022-06-29T16:00:37.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.795+0000 7f8510a0b700 1 -- start start 2022-06-29T16:00:37.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.795+0000 7f8510a0b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f850c0b2c50 0x7f850c0ace50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:37.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.795+0000 7f8510a0b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f850c0b48b0 0x7f850c140410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:37.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.795+0000 7f8510a0b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f850c0b5eb0 0x7f850c152700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:37.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.795+0000 7f8510a0b700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c13d680 con 0x7f850c0b2c50 2022-06-29T16:00:37.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.795+0000 7f8510a0b700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c152bf0 con 0x7f850c0b5eb0 2022-06-29T16:00:37.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.795+0000 7f8510a0b700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f850c140a60 con 0x7f850c0b48b0 2022-06-29T16:00:37.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.795+0000 7f850b7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f850c0b2c50 0x7f850c0ace50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:37.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.795+0000 7f850b7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f850c0b2c50 0x7f850c0ace50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47350/0 (socket says 172.21.15.60:47350) 2022-06-29T16:00:37.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.795+0000 7f850b7fe700 1 -- 172.21.15.60:0/3480026115 learned_addr learned my addr 172.21.15.60:0/3480026115 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:37.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.795+0000 7f850bfff700 1 --2- 172.21.15.60:0/3480026115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f850c0b5eb0 0x7f850c152700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:37.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.795+0000 7f850affd700 1 --2- 172.21.15.60:0/3480026115 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f850c0b48b0 0x7f850c140410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:37.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.799+0000 7f850b7fe700 1 -- 172.21.15.60:0/3480026115 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f850c0b48b0 msgr2=0x7f850c140410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:37.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.799+0000 7f850b7fe700 1 --2- 172.21.15.60:0/3480026115 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f850c0b48b0 0x7f850c140410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:37.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.799+0000 7f850b7fe700 1 -- 172.21.15.60:0/3480026115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f850c0b5eb0 msgr2=0x7f850c152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:37.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.799+0000 7f850b7fe700 1 --2- 172.21.15.60:0/3480026115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f850c0b5eb0 0x7f850c152700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:37.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.799+0000 7f850b7fe700 1 -- 172.21.15.60:0/3480026115 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84fc00b180 con 0x7f850c0b2c50 2022-06-29T16:00:37.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.799+0000 7f850b7fe700 1 --2- 172.21.15.60:0/3480026115 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f850c0b2c50 0x7f850c0ace50 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f84fc00c000 tx=0x7f84fc006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:37.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.799+0000 7f8508ff9700 1 -- 172.21.15.60:0/3480026115 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84fc00c860 con 0x7f850c0b2c50 2022-06-29T16:00:37.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.799+0000 7f8508ff9700 1 -- 172.21.15.60:0/3480026115 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84fc00c9c0 con 0x7f850c0b2c50 2022-06-29T16:00:37.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.799+0000 7f8510a0b700 1 -- 172.21.15.60:0/3480026115 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f850000b180 con 0x7f850c0b2c50 2022-06-29T16:00:37.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.799+0000 7f8508ff9700 1 -- 172.21.15.60:0/3480026115 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84fc00cd20 con 0x7f850c0b2c50 2022-06-29T16:00:37.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.799+0000 7f8510a0b700 1 -- 172.21.15.60:0/3480026115 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f850c0aece0 con 0x7f850c0b2c50 2022-06-29T16:00:37.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.799+0000 7f8508ff9700 1 -- 172.21.15.60:0/3480026115 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f84fc01c460 con 0x7f850c0b2c50 2022-06-29T16:00:37.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.799+0000 7f8510a0b700 1 -- 172.21.15.60:0/3480026115 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f850c0acb90 con 0x7f850c0b2c50 2022-06-29T16:00:37.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.799+0000 7f8508ff9700 1 --2- 172.21.15.60:0/3480026115 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f84ec053b30 0x7f84ec055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:37.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.799+0000 7f8508ff9700 1 -- 172.21.15.60:0/3480026115 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f84fc06c530 con 0x7f850c0b2c50 2022-06-29T16:00:37.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.799+0000 7f850affd700 1 --2- 172.21.15.60:0/3480026115 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f84ec053b30 0x7f84ec055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:37.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.799+0000 7f850affd700 1 --2- 172.21.15.60:0/3480026115 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f84ec053b30 0x7f84ec055ff0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f8500018000 tx=0x7f8500014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:37.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.807+0000 7f8508ff9700 1 -- 172.21.15.60:0/3480026115 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f850c0acb90 con 0x7f850c0b2c50 2022-06-29T16:00:37.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.971+0000 7f8510a0b700 1 -- 172.21.15.60:0/3480026115 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f850c0acb90 con 0x7f850c0b2c50 2022-06-29T16:00:37.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.971+0000 7f8508ff9700 1 -- 172.21.15.60:0/3480026115 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+2138 (secure 0 0 0) 0x7f850c0acb90 con 0x7f850c0b2c50 2022-06-29T16:00:37.981 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:00:37.981 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":15,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4937,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/1673184114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1673184114},{"type":"v1","addr":"172.21.15.92:6835","nonce":1673184114}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":13}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:00:20.685676+0000","modified":"2022-06-29T16:00:21.689299+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5308":{"gid":5308,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/1964426267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1964426267},{"type":"v1","addr":"172.21.15.60:6833","nonce":1964426267}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-06-29T16:00:37.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.975+0000 7f8510a0b700 1 -- 172.21.15.60:0/3480026115 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f84ec053b30 msgr2=0x7f84ec055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:37.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.975+0000 7f8510a0b700 1 --2- 172.21.15.60:0/3480026115 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f84ec053b30 0x7f84ec055ff0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f8500018000 tx=0x7f8500014000).stop 2022-06-29T16:00:37.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.975+0000 7f8510a0b700 1 -- 172.21.15.60:0/3480026115 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f850c0b2c50 msgr2=0x7f850c0ace50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:37.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.975+0000 7f8510a0b700 1 --2- 172.21.15.60:0/3480026115 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f850c0b2c50 0x7f850c0ace50 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f84fc00c000 tx=0x7f84fc006df0).stop 2022-06-29T16:00:37.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.975+0000 7f8510a0b700 1 -- 172.21.15.60:0/3480026115 shutdown_connections 2022-06-29T16:00:37.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.975+0000 7f8510a0b700 1 --2- 172.21.15.60:0/3480026115 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f84ec053b30 0x7f84ec055ff0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:37.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.975+0000 7f8510a0b700 1 --2- 172.21.15.60:0/3480026115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f850c0b5eb0 0x7f850c152700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:37.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.975+0000 7f8510a0b700 1 --2- 172.21.15.60:0/3480026115 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f850c0b48b0 0x7f850c140410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:37.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.975+0000 7f8510a0b700 1 --2- 172.21.15.60:0/3480026115 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f850c0b2c50 0x7f850c0ace50 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:37.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.975+0000 7f8510a0b700 1 -- 172.21.15.60:0/3480026115 >> 172.21.15.60:0/3480026115 conn(0x7f850c062090 msgr2=0x7f850c1427f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:37.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.975+0000 7f8510a0b700 1 -- 172.21.15.60:0/3480026115 shutdown_connections 2022-06-29T16:00:37.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:37.975+0000 7f8510a0b700 1 -- 172.21.15.60:0/3480026115 wait complete. 2022-06-29T16:00:37.987 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 15 2022-06-29T16:00:37.994 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a cache drop 2022-06-29T16:00:38.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.095+0000 7f495097e700 1 Processor -- start 2022-06-29T16:00:38.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.095+0000 7f495097e700 1 -- start start 2022-06-29T16:00:38.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.095+0000 7f495097e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f494c0b5eb0 0x7f494c142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:38.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.095+0000 7f495097e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f494c0b48b0 0x7f494c0b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:38.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.095+0000 7f495097e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f494c0b4d10 con 0x7f494c0b2c50 2022-06-29T16:00:38.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.095+0000 7f495097e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f494c0b54e0 con 0x7f494c0b5eb0 2022-06-29T16:00:38.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.095+0000 7f495097e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f494c0b6290 con 0x7f494c0b48b0 2022-06-29T16:00:38.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.095+0000 7f494b59e700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f494c0b2c50 0x7f494c145fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55448/0 (socket says 172.21.15.60:55448) 2022-06-29T16:00:38.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.095+0000 7f494b59e700 1 -- 172.21.15.60:0/2044292658 learned_addr learned my addr 172.21.15.60:0/2044292658 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:00:38.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.095+0000 7f494ad9d700 1 --2- 172.21.15.60:0/2044292658 >> v2:172.21.15.60:3300/0 conn(0x7f494c0b5eb0 0x7f494c142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:38.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.095+0000 7f494a59c700 1 --2- 172.21.15.60:0/2044292658 >> v2:172.21.15.92:3300/0 conn(0x7f494c0b48b0 0x7f494c0b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:38.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.095+0000 7f4949d9b700 1 -- 172.21.15.60:0/2044292658 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3296961753 0 0) 0x7f494c0b4d10 con 0x7f494c0b2c50 2022-06-29T16:00:38.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.095+0000 7f4949d9b700 1 -- 172.21.15.60:0/2044292658 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f492c003260 con 0x7f494c0b2c50 2022-06-29T16:00:38.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.099+0000 7f494a59c700 1 -- 172.21.15.60:0/2044292658 >> v2:172.21.15.60:3300/0 conn(0x7f494c0b5eb0 msgr2=0x7f494c142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:38.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.099+0000 7f494a59c700 1 --2- 172.21.15.60:0/2044292658 >> v2:172.21.15.60:3300/0 conn(0x7f494c0b5eb0 0x7f494c142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:38.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.099+0000 7f4949d9b700 1 -- 172.21.15.60:0/2044292658 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3789466980 0 0) 0x7f492c003260 con 0x7f494c0b2c50 2022-06-29T16:00:38.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.099+0000 7f494a59c700 1 -- 172.21.15.60:0/2044292658 >> v1:172.21.15.60:6789/0 conn(0x7f494c0b2c50 legacy=0x7f494c145fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:38.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.099+0000 7f494a59c700 1 -- 172.21.15.60:0/2044292658 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f494c142f10 con 0x7f494c0b48b0 2022-06-29T16:00:38.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.099+0000 7f494a59c700 1 --2- 172.21.15.60:0/2044292658 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f494c0b48b0 0x7f494c0b05e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f4944001b80 tx=0x7f4934006df0).ready entity=mon.1 client_cookie=40dc658b87ad05f4 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:38.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.099+0000 7f4949d9b700 1 -- 172.21.15.60:0/2044292658 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f49440019e0 con 0x7f494c0b48b0 2022-06-29T16:00:38.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.099+0000 7f4949d9b700 1 -- 172.21.15.60:0/2044292658 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4934000e90 con 0x7f494c0b48b0 2022-06-29T16:00:38.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.099+0000 7f4949d9b700 1 -- 172.21.15.60:0/2044292658 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4934012460 con 0x7f494c0b48b0 2022-06-29T16:00:38.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.099+0000 7f495097e700 1 -- 172.21.15.60:0/2044292658 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f494c0b48b0 msgr2=0x7f494c0b05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:38.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.099+0000 7f495097e700 1 --2- 172.21.15.60:0/2044292658 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f494c0b48b0 0x7f494c0b05e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f4944001b80 tx=0x7f4934006df0).stop 2022-06-29T16:00:38.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.099+0000 7f495097e700 1 -- 172.21.15.60:0/2044292658 shutdown_connections 2022-06-29T16:00:38.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.099+0000 7f495097e700 1 --2- 172.21.15.60:0/2044292658 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f494c0b48b0 0x7f494c0b05e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:38.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.099+0000 7f495097e700 1 --2- 172.21.15.60:0/2044292658 >> v2:172.21.15.60:3300/0 conn(0x7f494c0b5eb0 0x7f494c142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:38.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.099+0000 7f495097e700 1 -- 172.21.15.60:0/2044292658 >> 172.21.15.60:0/2044292658 conn(0x7f494c062090 msgr2=0x7f494c062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:38.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.099+0000 7f495097e700 1 -- 172.21.15.60:0/2044292658 shutdown_connections 2022-06-29T16:00:38.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.099+0000 7f495097e700 1 -- 172.21.15.60:0/2044292658 wait complete. 2022-06-29T16:00:38.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.099+0000 7f495097e700 1 Processor -- start 2022-06-29T16:00:38.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.099+0000 7f495097e700 1 -- start start 2022-06-29T16:00:38.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.103+0000 7f495097e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f494c0b2c50 0x7f494c149ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:38.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.103+0000 7f495097e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f494c0b48b0 0x7f494c13cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:38.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.103+0000 7f495097e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f494c0b5eb0 0x7f494c13d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:38.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.103+0000 7f495097e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f494c151ce0 con 0x7f494c0b5eb0 2022-06-29T16:00:38.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.103+0000 7f495097e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f494c151f80 con 0x7f494c0b48b0 2022-06-29T16:00:38.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.103+0000 7f495097e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f494c152220 con 0x7f494c0b2c50 2022-06-29T16:00:38.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.103+0000 7f494b59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f494c0b5eb0 0x7f494c13d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:38.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.103+0000 7f494ad9d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f494c0b2c50 0x7f494c149ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:38.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.103+0000 7f494b59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f494c0b5eb0 0x7f494c13d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47368/0 (socket says 172.21.15.60:47368) 2022-06-29T16:00:38.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.103+0000 7f494b59e700 1 -- 172.21.15.60:0/1998354576 learned_addr learned my addr 172.21.15.60:0/1998354576 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:38.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.103+0000 7f494a59c700 1 --2- 172.21.15.60:0/1998354576 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f494c0b48b0 0x7f494c13cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:38.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.103+0000 7f494b59e700 1 -- 172.21.15.60:0/1998354576 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f494c0b2c50 msgr2=0x7f494c149ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:38.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.103+0000 7f494b59e700 1 --2- 172.21.15.60:0/1998354576 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f494c0b2c50 0x7f494c149ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:38.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.103+0000 7f494b59e700 1 -- 172.21.15.60:0/1998354576 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f494c0b48b0 msgr2=0x7f494c13cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:38.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.103+0000 7f494b59e700 1 --2- 172.21.15.60:0/1998354576 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f494c0b48b0 0x7f494c13cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:38.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.103+0000 7f494b59e700 1 -- 172.21.15.60:0/1998354576 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f493400b180 con 0x7f494c0b5eb0 2022-06-29T16:00:38.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.103+0000 7f494b59e700 1 --2- 172.21.15.60:0/1998354576 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f494c0b5eb0 0x7f494c13d190 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f4944001c40 tx=0x7f49440021a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:38.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.103+0000 7f4943fff700 1 -- 172.21.15.60:0/1998354576 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4944004ab0 con 0x7f494c0b5eb0 2022-06-29T16:00:38.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.103+0000 7f4943fff700 1 -- 172.21.15.60:0/1998354576 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4944004c10 con 0x7f494c0b5eb0 2022-06-29T16:00:38.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.103+0000 7f495097e700 1 -- 172.21.15.60:0/1998354576 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f494c1519a0 con 0x7f494c0b5eb0 2022-06-29T16:00:38.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.103+0000 7f4943fff700 1 -- 172.21.15.60:0/1998354576 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4944007170 con 0x7f494c0b5eb0 2022-06-29T16:00:38.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.103+0000 7f495097e700 1 -- 172.21.15.60:0/1998354576 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f494c144f80 con 0x7f494c0b5eb0 2022-06-29T16:00:38.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.107+0000 7f4943fff700 1 -- 172.21.15.60:0/1998354576 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f494401b070 con 0x7f494c0b5eb0 2022-06-29T16:00:38.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.107+0000 7f4943fff700 1 --2- 172.21.15.60:0/1998354576 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4930053a70 0x7f4930055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:38.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.107+0000 7f4943fff700 1 -- 172.21.15.60:0/1998354576 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f494406b920 con 0x7f494c0b5eb0 2022-06-29T16:00:38.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.107+0000 7f494ad9d700 1 --2- 172.21.15.60:0/1998354576 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4930053a70 0x7f4930055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:38.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.107+0000 7f494ad9d700 1 --2- 172.21.15.60:0/1998354576 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4930053a70 0x7f4930055f30 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f493c00b000 tx=0x7f493c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:38.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.107+0000 7f495b3d4740 1 Processor -- start 2022-06-29T16:00:38.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.107+0000 7f495b3d4740 1 -- start start 2022-06-29T16:00:38.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.107+0000 7f495b3d4740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x24765d0 0x2437650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:38.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.111+0000 7f495b3d4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x2471cd0 0x24384b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:38.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.111+0000 7f495b3d4740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x24390c0 0x243d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:38.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.111+0000 7f495b3d4740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x23acbb0 con 0x24390c0 2022-06-29T16:00:38.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.111+0000 7f495b3d4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x238d150 con 0x2471cd0 2022-06-29T16:00:38.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.111+0000 7f495b3d4740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x2437b40 con 0x24765d0 2022-06-29T16:00:38.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.111+0000 7f494b59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x24390c0 0x243d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:38.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.111+0000 7f494ad9d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x24765d0 0x2437650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:38.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.111+0000 7f494b59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x24390c0 0x243d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47376/0 (socket says 172.21.15.60:47376) 2022-06-29T16:00:38.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.111+0000 7f494b59e700 1 -- 172.21.15.60:0/3069760490 learned_addr learned my addr 172.21.15.60:0/3069760490 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:38.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.111+0000 7f494a59c700 1 --2- 172.21.15.60:0/3069760490 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x2471cd0 0x24384b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:38.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.111+0000 7f494b59e700 1 -- 172.21.15.60:0/3069760490 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x24765d0 msgr2=0x2437650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:38.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.111+0000 7f494b59e700 1 --2- 172.21.15.60:0/3069760490 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x24765d0 0x2437650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:38.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.111+0000 7f494b59e700 1 -- 172.21.15.60:0/3069760490 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x2471cd0 msgr2=0x24384b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:38.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.111+0000 7f494b59e700 1 --2- 172.21.15.60:0/3069760490 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x2471cd0 0x24384b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:38.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.111+0000 7f494b59e700 1 -- 172.21.15.60:0/3069760490 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x23e4400 con 0x24390c0 2022-06-29T16:00:38.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.111+0000 7f494b59e700 1 --2- 172.21.15.60:0/3069760490 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x24390c0 0x243d010 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x2439f30 tx=0x7f4944005000).ready entity=mon.0 client_cookie=af77b5a5fa913529 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:38.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.111+0000 7f495097e700 1 -- 172.21.15.60:0/3069760490 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f494404c070 con 0x24390c0 2022-06-29T16:00:38.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.111+0000 7f495097e700 1 -- 172.21.15.60:0/3069760490 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49440421e0 con 0x24390c0 2022-06-29T16:00:38.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.111+0000 7f495097e700 1 -- 172.21.15.60:0/3069760490 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4944042540 con 0x24390c0 2022-06-29T16:00:38.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.111+0000 7f495b3d4740 1 -- 172.21.15.60:0/3069760490 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x24390c0 msgr2=0x243d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:38.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.111+0000 7f495b3d4740 1 --2- 172.21.15.60:0/3069760490 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x24390c0 0x243d010 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x2439f30 tx=0x7f4944005000).stop 2022-06-29T16:00:38.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.111+0000 7f495b3d4740 1 -- 172.21.15.60:0/3069760490 shutdown_connections 2022-06-29T16:00:38.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.111+0000 7f495b3d4740 1 --2- 172.21.15.60:0/3069760490 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x24390c0 0x243d010 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:38.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.111+0000 7f495b3d4740 1 --2- 172.21.15.60:0/3069760490 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x2471cd0 0x24384b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:38.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.111+0000 7f495b3d4740 1 --2- 172.21.15.60:0/3069760490 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x24765d0 0x2437650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:38.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.111+0000 7f495b3d4740 1 -- 172.21.15.60:0/3069760490 >> 172.21.15.60:0/3069760490 conn(0x23c4590 msgr2=0x2475d10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:38.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.111+0000 7f495b3d4740 1 -- 172.21.15.60:0/3069760490 shutdown_connections 2022-06-29T16:00:38.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.111+0000 7f495b3d4740 1 -- 172.21.15.60:0/3069760490 wait complete. 2022-06-29T16:00:38.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.111+0000 7f495b3d4740 1 -- start start 2022-06-29T16:00:38.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.111+0000 7f495b3d4740 1 Processor -- start 2022-06-29T16:00:38.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f495b3d4740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x24765d0 0x2443760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:38.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f495b3d4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x2440b10 0x243c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:38.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f495b3d4740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x243cee0 0x2445ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:38.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f495b3d4740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x2443c50 con 0x24765d0 2022-06-29T16:00:38.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f495b3d4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x2443e70 con 0x2440b10 2022-06-29T16:00:38.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f495b3d4740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x2444110 con 0x243cee0 2022-06-29T16:00:38.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f495b3d4740 10 client.0 ll_register_callbacks cb 0x23f39e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-06-29T16:00:38.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f494ad9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x24765d0 0x2443760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:38.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f494ad9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x24765d0 0x2443760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47378/0 (socket says 172.21.15.60:47378) 2022-06-29T16:00:38.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f494ad9d700 1 -- 172.21.15.60:0/677714918 learned_addr learned my addr 172.21.15.60:0/677714918 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:38.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f494b59e700 1 --2- 172.21.15.60:0/677714918 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x243cee0 0x2445ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:38.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f494a59c700 1 --2- 172.21.15.60:0/677714918 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x2440b10 0x243c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:38.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f494ad9d700 1 -- 172.21.15.60:0/677714918 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x243cee0 msgr2=0x2445ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:38.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f494ad9d700 1 --2- 172.21.15.60:0/677714918 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x243cee0 0x2445ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:38.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f494ad9d700 1 -- 172.21.15.60:0/677714918 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x2440b10 msgr2=0x243c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:38.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f494ad9d700 1 --2- 172.21.15.60:0/677714918 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x2440b10 0x243c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:38.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f494ad9d700 1 -- 172.21.15.60:0/677714918 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f494c144f80 con 0x24765d0 2022-06-29T16:00:38.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f494b59e700 1 --2- 172.21.15.60:0/677714918 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x243cee0 0x2445ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:00:38.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f494ad9d700 1 --2- 172.21.15.60:0/677714918 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x24765d0 0x2443760 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f493c01a000 tx=0x7f493c016000).ready entity=mon.0 client_cookie=69beded33d355f0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:38.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f4929ffb700 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2022-06-29T16:00:38.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f4929ffb700 1 -- 172.21.15.60:0/677714918 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f493c011e50 con 0x24765d0 2022-06-29T16:00:38.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f4929ffb700 1 -- 172.21.15.60:0/677714918 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(43..43 src has 1..43) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f493c013a40 con 0x24765d0 2022-06-29T16:00:38.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f495b3d4740 1 -- 172.21.15.60:0/677714918 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x23a2aa0 con 0x24765d0 2022-06-29T16:00:38.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f4929ffb700 1 -- 172.21.15.60:0/677714918 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f493c013c00 con 0x24765d0 2022-06-29T16:00:38.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f495b3d4740 10 client.5415 fetch_fsmap learned FSMap version 15 2022-06-29T16:00:38.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f495b3d4740 1 -- 172.21.15.60:0/677714918 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x239c380 con 0x24765d0 2022-06-29T16:00:38.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f4929ffb700 1 -- 172.21.15.60:0/677714918 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x23a2aa0 con 0x24765d0 2022-06-29T16:00:38.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f4929ffb700 1 -- 172.21.15.60:0/677714918 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 15) v1 ==== 1247+0+0 (secure 0 0 0) 0x7f493c029950 con 0x24765d0 2022-06-29T16:00:38.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f495b3d4740 10 client.5415 fetch_fsmap finished waiting for FSMap version 15 2022-06-29T16:00:38.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f495b3d4740 10 client.5415 resolve_mds: resolved ID 'a' to GID 5308 2022-06-29T16:00:38.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f495b3d4740 20 client.5415 populate_metadata read hostname 'smithi060' 2022-06-29T16:00:38.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f495b3d4740 1 --2- 172.21.15.60:0/677714918 >> [v2:172.21.15.60:6832/1964426267,v1:172.21.15.60:6833/1964426267] conn(0x24460e0 0x2483f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:38.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f495b3d4740 4 client.5415 mds_command: new command op to 5308 tid=0[{"prefix": "get_command_descriptions"}] 2022-06-29T16:00:38.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f495b3d4740 1 -- 172.21.15.60:0/677714918 --> [v2:172.21.15.60:6832/1964426267,v1:172.21.15.60:6833/1964426267] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x239ae10 con 0x24460e0 2022-06-29T16:00:38.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f494a59c700 1 --2- 172.21.15.60:0/677714918 >> [v2:172.21.15.60:6832/1964426267,v1:172.21.15.60:6833/1964426267] conn(0x24460e0 0x2483f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:38.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f4929ffb700 10 client.5415 ms_handle_connect on v2:172.21.15.60:6832/1964426267 2022-06-29T16:00:38.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.115+0000 7f494a59c700 1 --2- 172.21.15.60:0/677714918 >> [v2:172.21.15.60:6832/1964426267,v1:172.21.15.60:6833/1964426267] conn(0x24460e0 0x2483f10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=cab35d18e08014a3 server_cookie=644a42e17db268cd in_seq=0 out_seq=0 2022-06-29T16:00:38.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.119+0000 7f4929ffb700 1 -- 172.21.15.60:0/677714918 <== mds.0 v2:172.21.15.60:6832/1964426267 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17002 (crc 0 0 0) 0x239ae10 con 0x24460e0 2022-06-29T16:00:38.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.119+0000 7f4929ffb700 10 client.5415 handle_command_reply: tid=0 2022-06-29T16:00:38.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.119+0000 7f495b3d4740 1 client.5415 shutdown 2022-06-29T16:00:38.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.119+0000 7f495b3d4740 1 -- 172.21.15.60:0/677714918 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x24765d0 msgr2=0x2443760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:38.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.119+0000 7f495b3d4740 1 --2- 172.21.15.60:0/677714918 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x24765d0 0x2443760 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f493c01a000 tx=0x7f493c016000).stop 2022-06-29T16:00:38.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.119+0000 7f495b3d4740 1 -- 172.21.15.60:0/677714918 shutdown_connections 2022-06-29T16:00:38.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.119+0000 7f495b3d4740 1 --2- 172.21.15.60:0/677714918 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x243cee0 0x2445ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:38.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.119+0000 7f495b3d4740 1 --2- 172.21.15.60:0/677714918 >> [v2:172.21.15.60:6832/1964426267,v1:172.21.15.60:6833/1964426267] conn(0x24460e0 0x2483f10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:38.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.119+0000 7f495b3d4740 1 --2- 172.21.15.60:0/677714918 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x2440b10 0x243c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:38.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.119+0000 7f495b3d4740 1 --2- 172.21.15.60:0/677714918 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x24765d0 0x2443760 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:38.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.119+0000 7f4929ffb700 0 client.5415 ms_handle_reset on v2:172.21.15.60:6832/1964426267 2022-06-29T16:00:38.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.119+0000 7f495b3d4740 1 -- 172.21.15.60:0/677714918 >> 172.21.15.60:0/677714918 conn(0x23c4590 msgr2=0x2475c20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:38.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.119+0000 7f495b3d4740 1 -- 172.21.15.60:0/677714918 shutdown_connections 2022-06-29T16:00:38.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.119+0000 7f495b3d4740 1 -- 172.21.15.60:0/677714918 wait complete. 2022-06-29T16:00:38.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.119+0000 7f495b3d4740 20 client.5415 trim_cache size 0 max 16384 2022-06-29T16:00:38.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.135+0000 7f495b3d4740 1 Processor -- start 2022-06-29T16:00:38.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.135+0000 7f495b3d4740 1 -- start start 2022-06-29T16:00:38.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.135+0000 7f495b3d4740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x243cee0 0x243c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:38.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.135+0000 7f495b3d4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x2440b10 0x24383e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:38.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.135+0000 7f495b3d4740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x24765d0 0x24388d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:38.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.135+0000 7f495b3d4740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x24456f0 con 0x24765d0 2022-06-29T16:00:38.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.135+0000 7f495b3d4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x2437f10 con 0x2440b10 2022-06-29T16:00:38.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.135+0000 7f495b3d4740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x2446510 con 0x243cee0 2022-06-29T16:00:38.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.135+0000 7f494b59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x24765d0 0x24388d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:38.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.135+0000 7f494b59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x24765d0 0x24388d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47390/0 (socket says 172.21.15.60:47390) 2022-06-29T16:00:38.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.135+0000 7f494b59e700 1 -- 172.21.15.60:0/2992080406 learned_addr learned my addr 172.21.15.60:0/2992080406 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:38.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.135+0000 7f494ad9d700 1 --2- 172.21.15.60:0/2992080406 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x243cee0 0x243c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:38.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.135+0000 7f494a59c700 1 --2- 172.21.15.60:0/2992080406 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x2440b10 0x24383e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:38.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.135+0000 7f494b59e700 1 -- 172.21.15.60:0/2992080406 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x243cee0 msgr2=0x243c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:38.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.135+0000 7f494b59e700 1 --2- 172.21.15.60:0/2992080406 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x243cee0 0x243c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:38.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.135+0000 7f494b59e700 1 -- 172.21.15.60:0/2992080406 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x2440b10 msgr2=0x24383e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:38.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.135+0000 7f494b59e700 1 --2- 172.21.15.60:0/2992080406 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x2440b10 0x24383e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:38.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.135+0000 7f494b59e700 1 -- 172.21.15.60:0/2992080406 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x23a4c80 con 0x24765d0 2022-06-29T16:00:38.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.135+0000 7f494a59c700 1 --2- 172.21.15.60:0/2992080406 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x2440b10 0x24383e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:00:38.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.135+0000 7f494b59e700 1 --2- 172.21.15.60:0/2992080406 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x24765d0 0x24388d0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f4944054000 tx=0x7f4944015000).ready entity=mon.0 client_cookie=2259a097cb1698d3 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:38.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.135+0000 7f492bfff700 1 -- 172.21.15.60:0/2992080406 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4944059070 con 0x24765d0 2022-06-29T16:00:38.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.135+0000 7f492bfff700 1 -- 172.21.15.60:0/2992080406 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4944045d50 con 0x24765d0 2022-06-29T16:00:38.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.135+0000 7f492bfff700 1 -- 172.21.15.60:0/2992080406 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4944042540 con 0x24765d0 2022-06-29T16:00:38.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.135+0000 7f495b3d4740 1 -- 172.21.15.60:0/2992080406 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x24765d0 msgr2=0x24388d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:38.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.135+0000 7f495b3d4740 1 --2- 172.21.15.60:0/2992080406 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x24765d0 0x24388d0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f4944054000 tx=0x7f4944015000).stop 2022-06-29T16:00:38.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.135+0000 7f495b3d4740 1 -- 172.21.15.60:0/2992080406 shutdown_connections 2022-06-29T16:00:38.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.135+0000 7f495b3d4740 1 --2- 172.21.15.60:0/2992080406 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x24765d0 0x24388d0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:38.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.135+0000 7f495b3d4740 1 --2- 172.21.15.60:0/2992080406 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x2440b10 0x24383e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:38.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.135+0000 7f495b3d4740 1 --2- 172.21.15.60:0/2992080406 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x243cee0 0x243c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:38.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.135+0000 7f495b3d4740 1 -- 172.21.15.60:0/2992080406 >> 172.21.15.60:0/2992080406 conn(0x23c4590 msgr2=0x2443450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:38.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.139+0000 7f495b3d4740 1 -- 172.21.15.60:0/2992080406 shutdown_connections 2022-06-29T16:00:38.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.139+0000 7f495b3d4740 1 -- 172.21.15.60:0/2992080406 wait complete. 2022-06-29T16:00:38.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.139+0000 7f495b3d4740 1 -- start start 2022-06-29T16:00:38.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.139+0000 7f495b3d4740 1 Processor -- start 2022-06-29T16:00:38.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.139+0000 7f495b3d4740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x243cee0 0x24d7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:38.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.139+0000 7f495b3d4740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x2440b10 0x24d8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:38.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.139+0000 7f495b3d4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x24765d0 0x24d8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:38.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.139+0000 7f495b3d4740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x24d8af0 con 0x243cee0 2022-06-29T16:00:38.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.139+0000 7f495b3d4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x247a650 con 0x24765d0 2022-06-29T16:00:38.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.139+0000 7f495b3d4740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x2438e30 con 0x2440b10 2022-06-29T16:00:38.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.139+0000 7f495b3d4740 10 client.0 ll_register_callbacks cb 0x7f493400b310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-06-29T16:00:38.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.139+0000 7f494ad9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x243cee0 0x24d7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:38.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.139+0000 7f494ad9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x243cee0 0x24d7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47392/0 (socket says 172.21.15.60:47392) 2022-06-29T16:00:38.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.139+0000 7f494ad9d700 1 -- 172.21.15.60:0/4170971304 learned_addr learned my addr 172.21.15.60:0/4170971304 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:38.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.139+0000 7f494a59c700 1 --2- 172.21.15.60:0/4170971304 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x2440b10 0x24d8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:38.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.139+0000 7f494b59e700 1 --2- 172.21.15.60:0/4170971304 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x24765d0 0x24d8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:38.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.139+0000 7f494ad9d700 1 -- 172.21.15.60:0/4170971304 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x2440b10 msgr2=0x24d8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:38.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.139+0000 7f494ad9d700 1 --2- 172.21.15.60:0/4170971304 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x2440b10 0x24d8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:38.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.139+0000 7f494ad9d700 1 -- 172.21.15.60:0/4170971304 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x24765d0 msgr2=0x24d8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:38.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.139+0000 7f494ad9d700 1 --2- 172.21.15.60:0/4170971304 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x24765d0 0x24d8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:38.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.139+0000 7f494ad9d700 1 -- 172.21.15.60:0/4170971304 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x23e4400 con 0x243cee0 2022-06-29T16:00:38.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.139+0000 7f494b59e700 1 --2- 172.21.15.60:0/4170971304 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x24765d0 0x24d8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:00:38.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.139+0000 7f494ad9d700 1 --2- 172.21.15.60:0/4170971304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x243cee0 0x24d7c20 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f493c033000 tx=0x7f493c025000).ready entity=mon.0 client_cookie=2c4c24c5c4d46253 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:38.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.139+0000 7f492affd700 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2022-06-29T16:00:38.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.143+0000 7f492affd700 1 -- 172.21.15.60:0/4170971304 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f493c038070 con 0x243cee0 2022-06-29T16:00:38.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.143+0000 7f492affd700 1 -- 172.21.15.60:0/4170971304 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(43..43 src has 1..43) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f493c013c30 con 0x243cee0 2022-06-29T16:00:38.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.143+0000 7f495b3d4740 1 -- 172.21.15.60:0/4170971304 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f49440421e0 con 0x243cee0 2022-06-29T16:00:38.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.143+0000 7f492affd700 1 -- 172.21.15.60:0/4170971304 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f493c013680 con 0x243cee0 2022-06-29T16:00:38.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.143+0000 7f495b3d4740 10 client.5421 fetch_fsmap learned FSMap version 15 2022-06-29T16:00:38.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.143+0000 7f495b3d4740 1 -- 172.21.15.60:0/4170971304 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x23a1670 con 0x243cee0 2022-06-29T16:00:38.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.143+0000 7f492affd700 1 -- 172.21.15.60:0/4170971304 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49440421e0 con 0x243cee0 2022-06-29T16:00:38.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.143+0000 7f492affd700 1 -- 172.21.15.60:0/4170971304 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 15) v1 ==== 1247+0+0 (secure 0 0 0) 0x7f493c029950 con 0x243cee0 2022-06-29T16:00:38.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.143+0000 7f495b3d4740 10 client.5421 fetch_fsmap finished waiting for FSMap version 15 2022-06-29T16:00:38.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.143+0000 7f495b3d4740 10 client.5421 resolve_mds: resolved ID 'a' to GID 5308 2022-06-29T16:00:38.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.143+0000 7f495b3d4740 20 client.5421 populate_metadata read hostname 'smithi060' 2022-06-29T16:00:38.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.143+0000 7f495b3d4740 1 --2- 172.21.15.60:0/4170971304 >> [v2:172.21.15.60:6832/1964426267,v1:172.21.15.60:6833/1964426267] conn(0x24388f0 0x24d3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:38.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.143+0000 7f495b3d4740 4 client.5421 mds_command: new command op to 5308 tid=0[{"prefix": "cache drop"}] 2022-06-29T16:00:38.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.143+0000 7f495b3d4740 1 -- 172.21.15.60:0/4170971304 --> [v2:172.21.15.60:6832/1964426267,v1:172.21.15.60:6833/1964426267] -- command(tid 0: {"prefix": "cache drop"}) v1 -- 0x2290590 con 0x24388f0 2022-06-29T16:00:38.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.143+0000 7f494a59c700 1 --2- 172.21.15.60:0/4170971304 >> [v2:172.21.15.60:6832/1964426267,v1:172.21.15.60:6833/1964426267] conn(0x24388f0 0x24d3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:38.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.143+0000 7f492affd700 10 client.5421 ms_handle_connect on v2:172.21.15.60:6832/1964426267 2022-06-29T16:00:38.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:38.143+0000 7f494a59c700 1 --2- 172.21.15.60:0/4170971304 >> [v2:172.21.15.60:6832/1964426267,v1:172.21.15.60:6833/1964426267] conn(0x24388f0 0x24d3d90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=3e357a6bd91b6a54 server_cookie=56995b84822aa13a in_seq=0 out_seq=0 2022-06-29T16:00:49.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.331+0000 7f492affd700 1 -- 172.21.15.60:0/4170971304 <== mds.0 v2:172.21.15.60:6832/1964426267 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+380 (crc 0 0 0) 0x2290590 con 0x24388f0 2022-06-29T16:00:49.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.331+0000 7f492affd700 10 client.5421 handle_command_reply: tid=0 2022-06-29T16:00:49.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.331+0000 7f495b3d4740 1 client.5421 shutdown 2022-06-29T16:00:49.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.331+0000 7f495b3d4740 1 -- 172.21.15.60:0/4170971304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x243cee0 msgr2=0x24d7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:49.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.331+0000 7f495b3d4740 1 --2- 172.21.15.60:0/4170971304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x243cee0 0x24d7c20 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f493c033000 tx=0x7f493c025000).stop 2022-06-29T16:00:49.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.331+0000 7f495b3d4740 1 -- 172.21.15.60:0/4170971304 shutdown_connections 2022-06-29T16:00:49.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.331+0000 7f495b3d4740 1 --2- 172.21.15.60:0/4170971304 >> [v2:172.21.15.60:6832/1964426267,v1:172.21.15.60:6833/1964426267] conn(0x24388f0 0x24d3d90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:49.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.331+0000 7f495b3d4740 1 --2- 172.21.15.60:0/4170971304 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x24765d0 0x24d8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:49.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.331+0000 7f495b3d4740 1 --2- 172.21.15.60:0/4170971304 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x2440b10 0x24d8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:49.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.331+0000 7f492affd700 0 client.5421 ms_handle_reset on v2:172.21.15.60:6832/1964426267 2022-06-29T16:00:49.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.331+0000 7f495b3d4740 1 --2- 172.21.15.60:0/4170971304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x243cee0 0x24d7c20 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:49.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.331+0000 7f495b3d4740 1 -- 172.21.15.60:0/4170971304 >> 172.21.15.60:0/4170971304 conn(0x23c4590 msgr2=0x2443450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:49.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.331+0000 7f495b3d4740 1 -- 172.21.15.60:0/4170971304 shutdown_connections 2022-06-29T16:00:49.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.331+0000 7f495b3d4740 1 -- 172.21.15.60:0/4170971304 wait complete. 2022-06-29T16:00:49.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.331+0000 7f495b3d4740 20 client.5421 trim_cache size 0 max 16384 2022-06-29T16:00:49.352 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:00:49.353 INFO:teuthology.orchestra.run.smithi060.stdout: "client_recall": { 2022-06-29T16:00:49.353 INFO:teuthology.orchestra.run.smithi060.stdout: "return_code": 0, 2022-06-29T16:00:49.353 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "(0) Success", 2022-06-29T16:00:49.354 INFO:teuthology.orchestra.run.smithi060.stdout: "recalled": 220 2022-06-29T16:00:49.354 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:49.355 INFO:teuthology.orchestra.run.smithi060.stdout: "flush_journal": { 2022-06-29T16:00:49.355 INFO:teuthology.orchestra.run.smithi060.stdout: "return_code": 0, 2022-06-29T16:00:49.355 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "" 2022-06-29T16:00:49.356 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:49.356 INFO:teuthology.orchestra.run.smithi060.stdout: "trim_cache": { 2022-06-29T16:00:49.356 INFO:teuthology.orchestra.run.smithi060.stdout: "trimmed": 0 2022-06-29T16:00:49.357 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:49.360 INFO:teuthology.orchestra.run.smithi060.stdout: "cache": { 2022-06-29T16:00:49.360 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": { 2022-06-29T16:00:49.361 INFO:teuthology.orchestra.run.smithi060.stdout: "items": 20077, 2022-06-29T16:00:49.361 INFO:teuthology.orchestra.run.smithi060.stdout: "bytes": 3105248 2022-06-29T16:00:49.362 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:00:49.362 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:00:49.363 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 11.189239288 2022-06-29T16:00:49.363 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:00:49.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.335+0000 7f495097e700 1 -- 172.21.15.60:0/1998354576 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4930053a70 msgr2=0x7f4930055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:49.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.335+0000 7f495097e700 1 --2- 172.21.15.60:0/1998354576 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4930053a70 0x7f4930055f30 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f493c00b000 tx=0x7f493c006df0).stop 2022-06-29T16:00:49.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.335+0000 7f495097e700 1 -- 172.21.15.60:0/1998354576 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f494c0b5eb0 msgr2=0x7f494c13d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:49.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.335+0000 7f495097e700 1 --2- 172.21.15.60:0/1998354576 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f494c0b5eb0 0x7f494c13d190 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f4944001c40 tx=0x7f49440021a0).stop 2022-06-29T16:00:49.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.335+0000 7f495097e700 1 -- 172.21.15.60:0/1998354576 shutdown_connections 2022-06-29T16:00:49.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.335+0000 7f495097e700 1 --2- 172.21.15.60:0/1998354576 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4930053a70 0x7f4930055f30 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:49.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.335+0000 7f495097e700 1 --2- 172.21.15.60:0/1998354576 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f494c0b5eb0 0x7f494c13d190 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:49.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.335+0000 7f495097e700 1 --2- 172.21.15.60:0/1998354576 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f494c0b48b0 0x7f494c13cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:49.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.335+0000 7f495097e700 1 --2- 172.21.15.60:0/1998354576 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f494c0b2c50 0x7f494c149ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:49.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.335+0000 7f495097e700 1 -- 172.21.15.60:0/1998354576 >> 172.21.15.60:0/1998354576 conn(0x7f494c062090 msgr2=0x7f494c141410 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:49.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.335+0000 7f495097e700 1 -- 172.21.15.60:0/1998354576 shutdown_connections 2022-06-29T16:00:49.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.335+0000 7f495097e700 1 -- 172.21.15.60:0/1998354576 wait complete. 2022-06-29T16:00:49.373 INFO:tasks.cephfs.fuse_mount:Cleaning up killed ceph-fuse connection 2022-06-29T16:00:49.373 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:00:49.384 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:00:49.384 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:00:49.393 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': Transport endpoint is not connected 2022-06-29T16:00:49.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:00:49.394 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.0 2022-06-29T16:00:49.395 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2022-06-29T16:00:49.396 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-06-29T16:00:49.397 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:00:49.419 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:00:49.420 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:00:49.420 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:00:49.468 INFO:teuthology.orchestra.run.smithi121.stdout:ext2/ext3 2022-06-29T16:00:49.469 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-06-29T16:00:49.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:00:49.470 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:00:49.481 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:00:49.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.579+0000 7f350f36f700 1 Processor -- start 2022-06-29T16:00:49.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.579+0000 7f350f36f700 1 -- start start 2022-06-29T16:00:49.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.579+0000 7f350f36f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f350805d070 0x7f35080b1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:49.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.579+0000 7f350f36f700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f35080aee30 0x7f3508141b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:49.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.579+0000 7f350f36f700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f350805d450 con 0x7f350813a460 2022-06-29T16:00:49.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.579+0000 7f350f36f700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3508143a90 con 0x7f35080aee30 2022-06-29T16:00:49.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.579+0000 7f350f36f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3508062200 con 0x7f350805d070 2022-06-29T16:00:49.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.579+0000 7f350eb6e700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f350813a460 0x7f3508140bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55492/0 (socket says 172.21.15.60:55492) 2022-06-29T16:00:49.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.579+0000 7f350eb6e700 1 -- 172.21.15.60:0/839760203 learned_addr learned my addr 172.21.15.60:0/839760203 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:00:49.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.583+0000 7f350e36d700 1 --2- 172.21.15.60:0/839760203 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f350805d070 0x7f35080b1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:49.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.583+0000 7f350db6c700 1 --2- 172.21.15.60:0/839760203 >> v2:172.21.15.92:3300/0 conn(0x7f35080aee30 0x7f3508141b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:49.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.583+0000 7f350d36b700 1 -- 172.21.15.60:0/839760203 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060049037 0 0) 0x7f350805d450 con 0x7f350813a460 2022-06-29T16:00:49.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.583+0000 7f350d36b700 1 -- 172.21.15.60:0/839760203 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34f0003260 con 0x7f350813a460 2022-06-29T16:00:49.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.583+0000 7f350e36d700 1 -- 172.21.15.60:0/839760203 >> v2:172.21.15.92:3300/0 conn(0x7f35080aee30 msgr2=0x7f3508141b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:49.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.583+0000 7f350e36d700 1 --2- 172.21.15.60:0/839760203 >> v2:172.21.15.92:3300/0 conn(0x7f35080aee30 0x7f3508141b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:49.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.583+0000 7f350e36d700 1 -- 172.21.15.60:0/839760203 >> v1:172.21.15.60:6789/0 conn(0x7f350813a460 legacy=0x7f3508140bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:49.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.583+0000 7f350e36d700 1 -- 172.21.15.60:0/839760203 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3508064dc0 con 0x7f350805d070 2022-06-29T16:00:49.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.583+0000 7f350e36d700 1 --2- 172.21.15.60:0/839760203 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f350805d070 0x7f35080b1f30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f3504002f70 tx=0x7f3504006df0).ready entity=mon.2 client_cookie=937bfdf314cde91 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:49.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.583+0000 7f350d36b700 1 -- 172.21.15.60:0/839760203 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f350400d4a0 con 0x7f350805d070 2022-06-29T16:00:49.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.583+0000 7f350d36b700 1 -- 172.21.15.60:0/839760203 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3504000e90 con 0x7f350805d070 2022-06-29T16:00:49.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.583+0000 7f350d36b700 1 -- 172.21.15.60:0/839760203 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f350400d7f0 con 0x7f350805d070 2022-06-29T16:00:49.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.583+0000 7f350f36f700 1 -- 172.21.15.60:0/839760203 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f350805d070 msgr2=0x7f35080b1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:49.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.583+0000 7f350f36f700 1 --2- 172.21.15.60:0/839760203 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f350805d070 0x7f35080b1f30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f3504002f70 tx=0x7f3504006df0).stop 2022-06-29T16:00:49.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.583+0000 7f350f36f700 1 -- 172.21.15.60:0/839760203 shutdown_connections 2022-06-29T16:00:49.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.583+0000 7f350f36f700 1 --2- 172.21.15.60:0/839760203 >> v2:172.21.15.92:3300/0 conn(0x7f35080aee30 0x7f3508141b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:49.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.583+0000 7f350f36f700 1 --2- 172.21.15.60:0/839760203 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f350805d070 0x7f35080b1f30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:49.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.583+0000 7f350f36f700 1 -- 172.21.15.60:0/839760203 >> 172.21.15.60:0/839760203 conn(0x7f35080b5890 msgr2=0x7f35080b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:49.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.583+0000 7f350f36f700 1 -- 172.21.15.60:0/839760203 shutdown_connections 2022-06-29T16:00:49.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.583+0000 7f350f36f700 1 -- 172.21.15.60:0/839760203 wait complete. 2022-06-29T16:00:49.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.583+0000 7f350f36f700 1 Processor -- start 2022-06-29T16:00:49.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.583+0000 7f350f36f700 1 -- start start 2022-06-29T16:00:49.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.587+0000 7f350f36f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f350805d070 0x7f350813c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:49.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.587+0000 7f350f36f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35080aee30 0x7f35081474f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:49.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.587+0000 7f350f36f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f350813a460 0x7f350813f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:49.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.587+0000 7f350f36f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350813d480 con 0x7f350813a460 2022-06-29T16:00:49.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.587+0000 7f350f36f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350813d6b0 con 0x7f350805d070 2022-06-29T16:00:49.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.587+0000 7f350f36f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f350813d950 con 0x7f35080aee30 2022-06-29T16:00:49.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.587+0000 7f350eb6e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f350813a460 0x7f350813f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:49.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.587+0000 7f350eb6e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f350813a460 0x7f350813f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47412/0 (socket says 172.21.15.60:47412) 2022-06-29T16:00:49.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.587+0000 7f350eb6e700 1 -- 172.21.15.60:0/3194522681 learned_addr learned my addr 172.21.15.60:0/3194522681 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:49.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.587+0000 7f350db6c700 1 --2- 172.21.15.60:0/3194522681 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35080aee30 0x7f35081474f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:49.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.587+0000 7f350e36d700 1 --2- 172.21.15.60:0/3194522681 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f350805d070 0x7f350813c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:49.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.587+0000 7f350eb6e700 1 -- 172.21.15.60:0/3194522681 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35080aee30 msgr2=0x7f35081474f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:49.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.587+0000 7f350eb6e700 1 --2- 172.21.15.60:0/3194522681 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35080aee30 0x7f35081474f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:49.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.587+0000 7f350eb6e700 1 -- 172.21.15.60:0/3194522681 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f350805d070 msgr2=0x7f350813c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:49.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.587+0000 7f350eb6e700 1 --2- 172.21.15.60:0/3194522681 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f350805d070 0x7f350813c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:49.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.587+0000 7f350eb6e700 1 -- 172.21.15.60:0/3194522681 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350400b180 con 0x7f350813a460 2022-06-29T16:00:49.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.587+0000 7f350eb6e700 1 --2- 172.21.15.60:0/3194522681 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f350813a460 0x7f350813f150 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f3500001490 tx=0x7f3500007e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:49.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.587+0000 7f34f77fe700 1 -- 172.21.15.60:0/3194522681 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3500005ba0 con 0x7f350813a460 2022-06-29T16:00:49.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.587+0000 7f34f77fe700 1 -- 172.21.15.60:0/3194522681 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3500005d00 con 0x7f350813a460 2022-06-29T16:00:49.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.587+0000 7f350f36f700 1 -- 172.21.15.60:0/3194522681 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f350813df00 con 0x7f350813a460 2022-06-29T16:00:49.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.587+0000 7f34f77fe700 1 -- 172.21.15.60:0/3194522681 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35000072c0 con 0x7f350813a460 2022-06-29T16:00:49.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.587+0000 7f350f36f700 1 -- 172.21.15.60:0/3194522681 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f350813e2f0 con 0x7f350813a460 2022-06-29T16:00:49.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.587+0000 7f350f36f700 1 -- 172.21.15.60:0/3194522681 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f350813db10 con 0x7f350813a460 2022-06-29T16:00:49.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.587+0000 7f34f77fe700 1 -- 172.21.15.60:0/3194522681 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f350001a060 con 0x7f350813a460 2022-06-29T16:00:49.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.591+0000 7f34f77fe700 1 --2- 172.21.15.60:0/3194522681 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f34e8053a70 0x7f34e8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:49.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.591+0000 7f34f77fe700 1 -- 172.21.15.60:0/3194522681 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f350006b960 con 0x7f350813a460 2022-06-29T16:00:49.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.591+0000 7f350e36d700 1 --2- 172.21.15.60:0/3194522681 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f34e8053a70 0x7f34e8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:49.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.591+0000 7f350e36d700 1 --2- 172.21.15.60:0/3194522681 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f34e8053a70 0x7f34e8055f30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f350401a000 tx=0x7f3504014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:49.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.595+0000 7f34f77fe700 1 -- 172.21.15.60:0/3194522681 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3500042880 con 0x7f350813a460 2022-06-29T16:00:49.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.715+0000 7f350f36f700 1 -- 172.21.15.60:0/3194522681 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35080573d0 con 0x7f350813a460 2022-06-29T16:00:49.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.719+0000 7f34f77fe700 1 -- 172.21.15.60:0/3194522681 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+16038 (secure 0 0 0) 0x7f3500018030 con 0x7f350813a460 2022-06-29T16:00:49.727 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:00:49.727 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":43,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:00:22.691182+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:00:18.552380+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2022-06-29T16:00:19.688558+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:00:49.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.719+0000 7f350f36f700 1 -- 172.21.15.60:0/3194522681 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f34e8053a70 msgr2=0x7f34e8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:49.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.719+0000 7f350f36f700 1 --2- 172.21.15.60:0/3194522681 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f34e8053a70 0x7f34e8055f30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f350401a000 tx=0x7f3504014000).stop 2022-06-29T16:00:49.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.719+0000 7f350f36f700 1 -- 172.21.15.60:0/3194522681 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f350813a460 msgr2=0x7f350813f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:49.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.719+0000 7f350f36f700 1 --2- 172.21.15.60:0/3194522681 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f350813a460 0x7f350813f150 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f3500001490 tx=0x7f3500007e00).stop 2022-06-29T16:00:49.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.719+0000 7f350f36f700 1 -- 172.21.15.60:0/3194522681 shutdown_connections 2022-06-29T16:00:49.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.719+0000 7f350f36f700 1 --2- 172.21.15.60:0/3194522681 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f34e8053a70 0x7f34e8055f30 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:49.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.719+0000 7f350f36f700 1 --2- 172.21.15.60:0/3194522681 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f350813a460 0x7f350813f150 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:49.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.719+0000 7f350f36f700 1 --2- 172.21.15.60:0/3194522681 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35080aee30 0x7f35081474f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:49.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.719+0000 7f350f36f700 1 --2- 172.21.15.60:0/3194522681 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f350805d070 0x7f350813c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:49.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.719+0000 7f350f36f700 1 -- 172.21.15.60:0/3194522681 >> 172.21.15.60:0/3194522681 conn(0x7f35080b5890 msgr2=0x7f35080b1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:49.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.719+0000 7f350f36f700 1 -- 172.21.15.60:0/3194522681 shutdown_connections 2022-06-29T16:00:49.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.719+0000 7f350f36f700 1 -- 172.21.15.60:0/3194522681 wait complete. 2022-06-29T16:00:49.740 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-29T16:00:18.552380+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2022-06-29T16:00:19.688558+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-29T16:00:49.741 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:00:49.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.839+0000 7f4cc759e700 1 Processor -- start 2022-06-29T16:00:49.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.839+0000 7f4cc759e700 1 -- start start 2022-06-29T16:00:49.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.839+0000 7f4cc759e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4cc80b07f0 0x7f4cc8142ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:49.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.839+0000 7f4cc759e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4cc81433d0 0x7f4cc813ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:49.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.839+0000 7f4cc759e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cc80b2fa0 con 0x7f4cc80b2bc0 2022-06-29T16:00:49.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.839+0000 7f4cc759e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cc80ae250 con 0x7f4cc80b07f0 2022-06-29T16:00:49.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.839+0000 7f4cc759e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc80ae4f0 con 0x7f4cc81433d0 2022-06-29T16:00:49.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.839+0000 7f4cc659c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4cc80b2bc0 0x7f4cc80ae990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55502/0 (socket says 172.21.15.60:55502) 2022-06-29T16:00:49.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.839+0000 7f4cc659c700 1 -- 172.21.15.60:0/1007220547 learned_addr learned my addr 172.21.15.60:0/1007220547 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:00:49.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.839+0000 7f4cc6d9d700 1 --2- 172.21.15.60:0/1007220547 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4cc81433d0 0x7f4cc813ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:49.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.839+0000 7f4cc5d9b700 1 --2- 172.21.15.60:0/1007220547 >> v2:172.21.15.92:3300/0 conn(0x7f4cc80b07f0 0x7f4cc8142ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:49.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.839+0000 7f4cc559a700 1 -- 172.21.15.60:0/1007220547 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2877836378 0 0) 0x7f4cc80b2fa0 con 0x7f4cc80b2bc0 2022-06-29T16:00:49.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.839+0000 7f4cc559a700 1 -- 172.21.15.60:0/1007220547 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cac003260 con 0x7f4cc80b2bc0 2022-06-29T16:00:49.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.839+0000 7f4cc6d9d700 1 -- 172.21.15.60:0/1007220547 >> v2:172.21.15.92:3300/0 conn(0x7f4cc80b07f0 msgr2=0x7f4cc8142ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:49.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.839+0000 7f4cc6d9d700 1 --2- 172.21.15.60:0/1007220547 >> v2:172.21.15.92:3300/0 conn(0x7f4cc80b07f0 0x7f4cc8142ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:49.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.839+0000 7f4cc6d9d700 1 -- 172.21.15.60:0/1007220547 >> v1:172.21.15.60:6789/0 conn(0x7f4cc80b2bc0 legacy=0x7f4cc80ae990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:49.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.839+0000 7f4cc6d9d700 1 -- 172.21.15.60:0/1007220547 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc80b1f00 con 0x7f4cc81433d0 2022-06-29T16:00:49.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.839+0000 7f4cc6d9d700 1 --2- 172.21.15.60:0/1007220547 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4cc81433d0 0x7f4cc813ca00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f4cc00032f0 tx=0x7f4cc00091d0).ready entity=mon.2 client_cookie=a714ee2101ea711d server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:49.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.839+0000 7f4cc559a700 1 -- 172.21.15.60:0/1007220547 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4cc000b600 con 0x7f4cc81433d0 2022-06-29T16:00:49.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.839+0000 7f4cc559a700 1 -- 172.21.15.60:0/1007220547 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cc000b760 con 0x7f4cc81433d0 2022-06-29T16:00:49.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.839+0000 7f4cc559a700 1 -- 172.21.15.60:0/1007220547 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4cc000b8d0 con 0x7f4cc81433d0 2022-06-29T16:00:49.859 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.839+0000 7f4cc759e700 1 -- 172.21.15.60:0/1007220547 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4cc81433d0 msgr2=0x7f4cc813ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:49.859 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.839+0000 7f4cc759e700 1 --2- 172.21.15.60:0/1007220547 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4cc81433d0 0x7f4cc813ca00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f4cc00032f0 tx=0x7f4cc00091d0).stop 2022-06-29T16:00:49.859 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.839+0000 7f4cc759e700 1 -- 172.21.15.60:0/1007220547 shutdown_connections 2022-06-29T16:00:49.860 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.839+0000 7f4cc759e700 1 --2- 172.21.15.60:0/1007220547 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4cc81433d0 0x7f4cc813ca00 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:49.860 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.839+0000 7f4cc759e700 1 --2- 172.21.15.60:0/1007220547 >> v2:172.21.15.92:3300/0 conn(0x7f4cc80b07f0 0x7f4cc8142ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:49.861 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.843+0000 7f4cc759e700 1 -- 172.21.15.60:0/1007220547 >> 172.21.15.60:0/1007220547 conn(0x7f4cc80637c0 msgr2=0x7f4cc8063ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:49.861 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.843+0000 7f4cc759e700 1 -- 172.21.15.60:0/1007220547 shutdown_connections 2022-06-29T16:00:49.861 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.843+0000 7f4cc759e700 1 -- 172.21.15.60:0/1007220547 wait complete. 2022-06-29T16:00:49.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.843+0000 7f4cc759e700 1 Processor -- start 2022-06-29T16:00:49.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.843+0000 7f4cc759e700 1 -- start start 2022-06-29T16:00:49.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.843+0000 7f4cc759e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4cc80b07f0 0x7f4cc813df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:49.866 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.843+0000 7f4cc759e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cc80b2bc0 0x7f4cc8142580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:49.866 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.843+0000 7f4cc759e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4cc81433d0 0x7f4cc8142a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:49.866 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.843+0000 7f4cc759e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc813e3f0 con 0x7f4cc80b2bc0 2022-06-29T16:00:49.866 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.843+0000 7f4cc759e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc8140de0 con 0x7f4cc80b07f0 2022-06-29T16:00:49.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.843+0000 7f4cc759e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc813e630 con 0x7f4cc81433d0 2022-06-29T16:00:49.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.843+0000 7f4cc5d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cc80b2bc0 0x7f4cc8142580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:49.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.843+0000 7f4cc5d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cc80b2bc0 0x7f4cc8142580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47424/0 (socket says 172.21.15.60:47424) 2022-06-29T16:00:49.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.843+0000 7f4cc5d9b700 1 -- 172.21.15.60:0/3539386669 learned_addr learned my addr 172.21.15.60:0/3539386669 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:49.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.843+0000 7f4cc659c700 1 --2- 172.21.15.60:0/3539386669 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4cc80b07f0 0x7f4cc813df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:49.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.843+0000 7f4cc6d9d700 1 --2- 172.21.15.60:0/3539386669 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4cc81433d0 0x7f4cc8142a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:49.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.843+0000 7f4cc5d9b700 1 -- 172.21.15.60:0/3539386669 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4cc81433d0 msgr2=0x7f4cc8142a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:49.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.843+0000 7f4cc5d9b700 1 --2- 172.21.15.60:0/3539386669 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4cc81433d0 0x7f4cc8142a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:49.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.843+0000 7f4cc5d9b700 1 -- 172.21.15.60:0/3539386669 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4cc80b07f0 msgr2=0x7f4cc813df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:49.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.843+0000 7f4cc5d9b700 1 --2- 172.21.15.60:0/3539386669 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4cc80b07f0 0x7f4cc813df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:49.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.843+0000 7f4cc5d9b700 1 -- 172.21.15.60:0/3539386669 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc0009470 con 0x7f4cc80b2bc0 2022-06-29T16:00:49.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.843+0000 7f4cc5d9b700 1 --2- 172.21.15.60:0/3539386669 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cc80b2bc0 0x7f4cc8142580 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f4cbc00c000 tx=0x7f4cbc006d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:49.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.843+0000 7f4cb77fe700 1 -- 172.21.15.60:0/3539386669 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4cbc010590 con 0x7f4cc80b2bc0 2022-06-29T16:00:49.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.843+0000 7f4cb77fe700 1 -- 172.21.15.60:0/3539386669 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cbc0106f0 con 0x7f4cc80b2bc0 2022-06-29T16:00:49.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.843+0000 7f4cc759e700 1 -- 172.21.15.60:0/3539386669 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cc813e8d0 con 0x7f4cc80b2bc0 2022-06-29T16:00:49.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.843+0000 7f4cb77fe700 1 -- 172.21.15.60:0/3539386669 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4cbc010a50 con 0x7f4cc80b2bc0 2022-06-29T16:00:49.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.843+0000 7f4cc759e700 1 -- 172.21.15.60:0/3539386669 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cc8141240 con 0x7f4cc80b2bc0 2022-06-29T16:00:49.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.847+0000 7f4cb77fe700 1 -- 172.21.15.60:0/3539386669 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4cbc01a030 con 0x7f4cc80b2bc0 2022-06-29T16:00:49.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.847+0000 7f4cb77fe700 1 --2- 172.21.15.60:0/3539386669 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4ca40539e0 0x7f4ca4055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:49.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.847+0000 7f4cb77fe700 1 -- 172.21.15.60:0/3539386669 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4cbc06bd00 con 0x7f4cc80b2bc0 2022-06-29T16:00:49.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.847+0000 7f4cc659c700 1 --2- 172.21.15.60:0/3539386669 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4ca40539e0 0x7f4ca4055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:49.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.847+0000 7f4cc759e700 1 -- 172.21.15.60:0/3539386669 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cc80573d0 con 0x7f4cc80b2bc0 2022-06-29T16:00:49.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.847+0000 7f4cc659c700 1 --2- 172.21.15.60:0/3539386669 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4ca40539e0 0x7f4ca4055ea0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f4cb800b000 tx=0x7f4cb8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:49.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.851+0000 7f4cb77fe700 1 -- 172.21.15.60:0/3539386669 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4cbc042b40 con 0x7f4cc80b2bc0 2022-06-29T16:00:49.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.975+0000 7f4cc759e700 1 -- 172.21.15.60:0/3539386669 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f4cc8067a60 con 0x7f4cc80b2bc0 2022-06-29T16:00:49.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.975+0000 7f4cb77fe700 1 -- 172.21.15.60:0/3539386669 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v43) v1 ==== 110+0+10 (secure 0 0 0) 0x7f4cbc022020 con 0x7f4cc80b2bc0 2022-06-29T16:00:49.984 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:00:49.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.975+0000 7f4cc759e700 1 -- 172.21.15.60:0/3539386669 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4ca40539e0 msgr2=0x7f4ca4055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:49.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.975+0000 7f4cc759e700 1 --2- 172.21.15.60:0/3539386669 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4ca40539e0 0x7f4ca4055ea0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f4cb800b000 tx=0x7f4cb8009040).stop 2022-06-29T16:00:49.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.975+0000 7f4cc759e700 1 -- 172.21.15.60:0/3539386669 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cc80b2bc0 msgr2=0x7f4cc8142580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:49.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.975+0000 7f4cc759e700 1 --2- 172.21.15.60:0/3539386669 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cc80b2bc0 0x7f4cc8142580 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f4cbc00c000 tx=0x7f4cbc006d90).stop 2022-06-29T16:00:49.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.975+0000 7f4cc759e700 1 -- 172.21.15.60:0/3539386669 shutdown_connections 2022-06-29T16:00:49.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.975+0000 7f4cc759e700 1 --2- 172.21.15.60:0/3539386669 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4ca40539e0 0x7f4ca4055ea0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:49.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.975+0000 7f4cc759e700 1 --2- 172.21.15.60:0/3539386669 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4cc81433d0 0x7f4cc8142a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:49.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.975+0000 7f4cc759e700 1 --2- 172.21.15.60:0/3539386669 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cc80b2bc0 0x7f4cc8142580 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:49.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.975+0000 7f4cc759e700 1 --2- 172.21.15.60:0/3539386669 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4cc80b07f0 0x7f4cc813df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:49.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.975+0000 7f4cc759e700 1 -- 172.21.15.60:0/3539386669 >> 172.21.15.60:0/3539386669 conn(0x7f4cc80637c0 msgr2=0x7f4cc80afae0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:49.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.979+0000 7f4cc759e700 1 -- 172.21.15.60:0/3539386669 shutdown_connections 2022-06-29T16:00:49.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:49.979+0000 7f4cc759e700 1 -- 172.21.15.60:0/3539386669 wait complete. 2022-06-29T16:00:49.996 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:00:50.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.091+0000 7ff07d482700 1 Processor -- start 2022-06-29T16:00:50.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.091+0000 7ff07d482700 1 -- start start 2022-06-29T16:00:50.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.091+0000 7ff07d482700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff0780b1f60 0x7ff07813fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:50.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.091+0000 7ff07d482700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff07805c470 0x7ff07813fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:50.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.091+0000 7ff07d482700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0780b40e0 con 0x7ff0780b3d00 2022-06-29T16:00:50.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.091+0000 7ff07d482700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff078144c90 con 0x7ff0780b1f60 2022-06-29T16:00:50.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.091+0000 7ff07d482700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff07805c280 con 0x7ff07805c470 2022-06-29T16:00:50.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.091+0000 7ff0777fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff0780b1f60 0x7ff07813fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:50.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.091+0000 7ff0777fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff0780b1f60 0x7ff07813fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47432/0 (socket says 172.21.15.60:47432) 2022-06-29T16:00:50.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.091+0000 7ff0777fe700 1 -- 172.21.15.60:0/3478274224 learned_addr learned my addr 172.21.15.60:0/3478274224 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:50.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.091+0000 7ff07cc81700 1 --2- 172.21.15.60:0/3478274224 >> v2:172.21.15.92:3300/0 conn(0x7ff07805c470 0x7ff07813fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:50.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.091+0000 7ff0777fe700 1 -- 172.21.15.60:0/3478274224 >> v2:172.21.15.92:3300/0 conn(0x7ff07805c470 msgr2=0x7ff07813fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:50.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.095+0000 7ff0777fe700 1 --2- 172.21.15.60:0/3478274224 >> v2:172.21.15.92:3300/0 conn(0x7ff07805c470 0x7ff07813fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:50.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.095+0000 7ff0777fe700 1 -- 172.21.15.60:0/3478274224 >> v1:172.21.15.92:6789/0 conn(0x7ff0780b3d00 legacy=0x7ff078144580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:50.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.095+0000 7ff0777fe700 1 -- 172.21.15.60:0/3478274224 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff078143e40 con 0x7ff0780b1f60 2022-06-29T16:00:50.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.095+0000 7ff0777fe700 1 --2- 172.21.15.60:0/3478274224 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0780b1f60 0x7ff07813fa00 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7ff0780b4270 tx=0x7ff068006eb0).ready entity=mon.0 client_cookie=fb662011edf24f36 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:50.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.095+0000 7ff076ffd700 1 -- 172.21.15.60:0/3478274224 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff06800d610 con 0x7ff0780b1f60 2022-06-29T16:00:50.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.095+0000 7ff076ffd700 1 -- 172.21.15.60:0/3478274224 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff068000e90 con 0x7ff0780b1f60 2022-06-29T16:00:50.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.095+0000 7ff076ffd700 1 -- 172.21.15.60:0/3478274224 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff06800d960 con 0x7ff0780b1f60 2022-06-29T16:00:50.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.095+0000 7ff07d482700 1 -- 172.21.15.60:0/3478274224 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0780b1f60 msgr2=0x7ff07813fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:50.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.095+0000 7ff07d482700 1 --2- 172.21.15.60:0/3478274224 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0780b1f60 0x7ff07813fa00 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7ff0780b4270 tx=0x7ff068006eb0).stop 2022-06-29T16:00:50.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.095+0000 7ff07d482700 1 -- 172.21.15.60:0/3478274224 shutdown_connections 2022-06-29T16:00:50.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.095+0000 7ff07d482700 1 --2- 172.21.15.60:0/3478274224 >> v2:172.21.15.92:3300/0 conn(0x7ff07805c470 0x7ff07813fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:50.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.095+0000 7ff07d482700 1 --2- 172.21.15.60:0/3478274224 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0780b1f60 0x7ff07813fa00 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:50.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.095+0000 7ff07d482700 1 -- 172.21.15.60:0/3478274224 >> 172.21.15.60:0/3478274224 conn(0x7ff078064810 msgr2=0x7ff078064bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:50.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.095+0000 7ff07d482700 1 -- 172.21.15.60:0/3478274224 shutdown_connections 2022-06-29T16:00:50.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.095+0000 7ff07d482700 1 -- 172.21.15.60:0/3478274224 wait complete. 2022-06-29T16:00:50.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.095+0000 7ff07d482700 1 Processor -- start 2022-06-29T16:00:50.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.095+0000 7ff07d482700 1 -- start start 2022-06-29T16:00:50.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.099+0000 7ff07d482700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff07805c470 0x7ff0780ae350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:50.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.099+0000 7ff07d482700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0780b1f60 0x7ff0780ae840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:50.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.099+0000 7ff07d482700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0780b3d00 0x7ff07813aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:50.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.099+0000 7ff07d482700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff078146b30 con 0x7ff0780b1f60 2022-06-29T16:00:50.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.099+0000 7ff07d482700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff078145730 con 0x7ff07805c470 2022-06-29T16:00:50.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.099+0000 7ff07d482700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff078145bd0 con 0x7ff0780b3d00 2022-06-29T16:00:50.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.099+0000 7ff077fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff07805c470 0x7ff0780ae350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:50.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.099+0000 7ff0777fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0780b1f60 0x7ff0780ae840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:50.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.099+0000 7ff077fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff07805c470 0x7ff0780ae350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:53466/0 (socket says 172.21.15.60:53466) 2022-06-29T16:00:50.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.099+0000 7ff077fff700 1 -- 172.21.15.60:0/3523881745 learned_addr learned my addr 172.21.15.60:0/3523881745 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:50.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.099+0000 7ff07cc81700 1 --2- 172.21.15.60:0/3523881745 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0780b3d00 0x7ff07813aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:50.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.099+0000 7ff077fff700 1 -- 172.21.15.60:0/3523881745 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0780b3d00 msgr2=0x7ff07813aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:50.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.099+0000 7ff077fff700 1 --2- 172.21.15.60:0/3523881745 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0780b3d00 0x7ff07813aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:50.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.099+0000 7ff077fff700 1 -- 172.21.15.60:0/3523881745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0780b1f60 msgr2=0x7ff0780ae840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:50.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.099+0000 7ff077fff700 1 --2- 172.21.15.60:0/3523881745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0780b1f60 0x7ff0780ae840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:50.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.099+0000 7ff077fff700 1 -- 172.21.15.60:0/3523881745 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff06800b1e0 con 0x7ff07805c470 2022-06-29T16:00:50.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.099+0000 7ff077fff700 1 --2- 172.21.15.60:0/3523881745 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff07805c470 0x7ff0780ae350 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7ff07805f760 tx=0x7ff06c007c50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:50.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.099+0000 7ff0757fa700 1 -- 172.21.15.60:0/3523881745 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff06c012400 con 0x7ff07805c470 2022-06-29T16:00:50.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.099+0000 7ff0757fa700 1 -- 172.21.15.60:0/3523881745 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff06c012560 con 0x7ff07805c470 2022-06-29T16:00:50.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.099+0000 7ff07d482700 1 -- 172.21.15.60:0/3523881745 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff078145f30 con 0x7ff07805c470 2022-06-29T16:00:50.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.099+0000 7ff0757fa700 1 -- 172.21.15.60:0/3523881745 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff06c0128c0 con 0x7ff07805c470 2022-06-29T16:00:50.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.099+0000 7ff07d482700 1 -- 172.21.15.60:0/3523881745 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff078139920 con 0x7ff07805c470 2022-06-29T16:00:50.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.099+0000 7ff07d482700 1 -- 172.21.15.60:0/3523881745 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0780573d0 con 0x7ff07805c470 2022-06-29T16:00:50.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.103+0000 7ff0757fa700 1 -- 172.21.15.60:0/3523881745 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff06c019030 con 0x7ff07805c470 2022-06-29T16:00:50.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.103+0000 7ff0757fa700 1 --2- 172.21.15.60:0/3523881745 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff058053aa0 0x7ff058055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:50.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.103+0000 7ff0757fa700 1 -- 172.21.15.60:0/3523881745 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff06c06b1d0 con 0x7ff07805c470 2022-06-29T16:00:50.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.103+0000 7ff0777fe700 1 --2- 172.21.15.60:0/3523881745 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff058053aa0 0x7ff058055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:50.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.103+0000 7ff0777fe700 1 --2- 172.21.15.60:0/3523881745 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff058053aa0 0x7ff058055f60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7ff06801b000 tx=0x7ff068015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:50.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.111+0000 7ff0757fa700 1 -- 172.21.15.60:0/3523881745 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff06c042050 con 0x7ff07805c470 2022-06-29T16:00:50.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.235+0000 7ff07d482700 1 -- 172.21.15.60:0/3523881745 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7ff0780b48b0 con 0x7ff07805c470 2022-06-29T16:00:50.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.235+0000 7ff0757fa700 1 -- 172.21.15.60:0/3523881745 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v43) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff06c045670 con 0x7ff07805c470 2022-06-29T16:00:50.244 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:00:50.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.239+0000 7ff07d482700 1 -- 172.21.15.60:0/3523881745 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff058053aa0 msgr2=0x7ff058055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:50.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.239+0000 7ff07d482700 1 --2- 172.21.15.60:0/3523881745 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff058053aa0 0x7ff058055f60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7ff06801b000 tx=0x7ff068015000).stop 2022-06-29T16:00:50.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.239+0000 7ff07d482700 1 -- 172.21.15.60:0/3523881745 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff07805c470 msgr2=0x7ff0780ae350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:50.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.239+0000 7ff07d482700 1 --2- 172.21.15.60:0/3523881745 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff07805c470 0x7ff0780ae350 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7ff07805f760 tx=0x7ff06c007c50).stop 2022-06-29T16:00:50.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.239+0000 7ff07d482700 1 -- 172.21.15.60:0/3523881745 shutdown_connections 2022-06-29T16:00:50.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.239+0000 7ff07d482700 1 --2- 172.21.15.60:0/3523881745 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff058053aa0 0x7ff058055f60 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:50.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.239+0000 7ff07d482700 1 --2- 172.21.15.60:0/3523881745 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0780b3d00 0x7ff07813aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:50.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.239+0000 7ff07d482700 1 --2- 172.21.15.60:0/3523881745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0780b1f60 0x7ff0780ae840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:50.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.239+0000 7ff07d482700 1 --2- 172.21.15.60:0/3523881745 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff07805c470 0x7ff0780ae350 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:50.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.239+0000 7ff07d482700 1 -- 172.21.15.60:0/3523881745 >> 172.21.15.60:0/3523881745 conn(0x7ff078064810 msgr2=0x7ff07813ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:50.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.239+0000 7ff07d482700 1 -- 172.21.15.60:0/3523881745 shutdown_connections 2022-06-29T16:00:50.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.239+0000 7ff07d482700 1 -- 172.21.15.60:0/3523881745 wait complete. 2022-06-29T16:00:50.261 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-29T16:00:50.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.355+0000 7f5a053d4700 1 Processor -- start 2022-06-29T16:00:50.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.355+0000 7f5a053d4700 1 -- start start 2022-06-29T16:00:50.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.359+0000 7f5a053d4700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5a00060710 0x7f5a000b6270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:50.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.359+0000 7f5a053d4700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5a0013ec70 0x7f5a0013f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:50.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.359+0000 7f5a053d4700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a00060c60 con 0x7f5a000b5e90 2022-06-29T16:00:50.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.359+0000 7f5a053d4700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a000af0c0 con 0x7f5a0013ec70 2022-06-29T16:00:50.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.359+0000 7f5a053d4700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a000b1d30 con 0x7f5a00060710 2022-06-29T16:00:50.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.359+0000 7f59ff7fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5a0013ec70 0x7f5a0013f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:50.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.359+0000 7f59ff7fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5a0013ec70 0x7f5a0013f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47446/0 (socket says 172.21.15.60:47446) 2022-06-29T16:00:50.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.359+0000 7f59ff7fe700 1 -- 172.21.15.60:0/1141665432 learned_addr learned my addr 172.21.15.60:0/1141665432 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:50.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.359+0000 7f59fe7fc700 1 --2- 172.21.15.60:0/1141665432 >> v2:172.21.15.92:3300/0 conn(0x7f5a00060710 0x7f5a000b6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:50.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.359+0000 7f59ff7fe700 1 -- 172.21.15.60:0/1141665432 >> v2:172.21.15.92:3300/0 conn(0x7f5a00060710 msgr2=0x7f5a000b6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:50.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.359+0000 7f59ff7fe700 1 --2- 172.21.15.60:0/1141665432 >> v2:172.21.15.92:3300/0 conn(0x7f5a00060710 0x7f5a000b6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:50.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.359+0000 7f59ff7fe700 1 -- 172.21.15.60:0/1141665432 >> v1:172.21.15.92:6789/0 conn(0x7f5a000b5e90 legacy=0x7f5a000ae9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:50.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.359+0000 7f59ff7fe700 1 -- 172.21.15.60:0/1141665432 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a000ae2e0 con 0x7f5a0013ec70 2022-06-29T16:00:50.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.359+0000 7f59ff7fe700 1 --2- 172.21.15.60:0/1141665432 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a0013ec70 0x7f5a0013f100 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f5a000b20f0 tx=0x7f59f8009290).ready entity=mon.0 client_cookie=e0360f70f98ac023 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:50.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.359+0000 7f59fdffb700 1 -- 172.21.15.60:0/1141665432 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59f800b860 con 0x7f5a0013ec70 2022-06-29T16:00:50.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.359+0000 7f59fdffb700 1 -- 172.21.15.60:0/1141665432 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f800b9c0 con 0x7f5a0013ec70 2022-06-29T16:00:50.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.359+0000 7f59fdffb700 1 -- 172.21.15.60:0/1141665432 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59f800bb30 con 0x7f5a0013ec70 2022-06-29T16:00:50.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.359+0000 7f5a053d4700 1 -- 172.21.15.60:0/1141665432 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a0013ec70 msgr2=0x7f5a0013f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:50.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.359+0000 7f5a053d4700 1 --2- 172.21.15.60:0/1141665432 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a0013ec70 0x7f5a0013f100 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f5a000b20f0 tx=0x7f59f8009290).stop 2022-06-29T16:00:50.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.359+0000 7f5a053d4700 1 -- 172.21.15.60:0/1141665432 shutdown_connections 2022-06-29T16:00:50.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.359+0000 7f5a053d4700 1 --2- 172.21.15.60:0/1141665432 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a0013ec70 0x7f5a0013f100 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:50.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.359+0000 7f5a053d4700 1 --2- 172.21.15.60:0/1141665432 >> v2:172.21.15.92:3300/0 conn(0x7f5a00060710 0x7f5a000b6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:50.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.359+0000 7f5a053d4700 1 -- 172.21.15.60:0/1141665432 >> 172.21.15.60:0/1141665432 conn(0x7f5a00063b50 msgr2=0x7f5a000b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:50.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.359+0000 7f5a053d4700 1 -- 172.21.15.60:0/1141665432 shutdown_connections 2022-06-29T16:00:50.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.359+0000 7f5a053d4700 1 -- 172.21.15.60:0/1141665432 wait complete. 2022-06-29T16:00:50.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.363+0000 7f5a053d4700 1 Processor -- start 2022-06-29T16:00:50.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.363+0000 7f5a053d4700 1 -- start start 2022-06-29T16:00:50.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.363+0000 7f5a053d4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a00060710 0x7f5a0005f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:50.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.363+0000 7f5a053d4700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a000b5e90 0x7f5a0013f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:50.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.363+0000 7f5a053d4700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5a0013ec70 0x7f5a0013fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:50.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.363+0000 7f5a053d4700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a00140240 con 0x7f5a00060710 2022-06-29T16:00:50.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.363+0000 7f5a053d4700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a00145840 con 0x7f5a000b5e90 2022-06-29T16:00:50.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.363+0000 7f5a053d4700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a00145470 con 0x7f5a0013ec70 2022-06-29T16:00:50.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.363+0000 7f59feffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a00060710 0x7f5a0005f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:50.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.363+0000 7f59feffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a00060710 0x7f5a0005f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47450/0 (socket says 172.21.15.60:47450) 2022-06-29T16:00:50.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.363+0000 7f59feffd700 1 -- 172.21.15.60:0/1733588789 learned_addr learned my addr 172.21.15.60:0/1733588789 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:50.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.363+0000 7f59fe7fc700 1 --2- 172.21.15.60:0/1733588789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a000b5e90 0x7f5a0013f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:50.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.363+0000 7f59ff7fe700 1 --2- 172.21.15.60:0/1733588789 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5a0013ec70 0x7f5a0013fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:50.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.363+0000 7f59feffd700 1 -- 172.21.15.60:0/1733588789 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5a0013ec70 msgr2=0x7f5a0013fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:50.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.363+0000 7f59feffd700 1 --2- 172.21.15.60:0/1733588789 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5a0013ec70 0x7f5a0013fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:50.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.363+0000 7f59feffd700 1 -- 172.21.15.60:0/1733588789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a000b5e90 msgr2=0x7f5a0013f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:50.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.363+0000 7f59feffd700 1 --2- 172.21.15.60:0/1733588789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a000b5e90 0x7f5a0013f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:50.382 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.363+0000 7f59feffd700 1 -- 172.21.15.60:0/1733588789 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f8009530 con 0x7f5a00060710 2022-06-29T16:00:50.382 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.363+0000 7f59fe7fc700 1 --2- 172.21.15.60:0/1733588789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a000b5e90 0x7f5a0013f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:00:50.382 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.363+0000 7f59feffd700 1 --2- 172.21.15.60:0/1733588789 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a00060710 0x7f5a0005f250 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f5a000ae180 tx=0x7f59e8007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:50.382 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.363+0000 7f59f7fff700 1 -- 172.21.15.60:0/1733588789 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59e800ce30 con 0x7f5a00060710 2022-06-29T16:00:50.382 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.363+0000 7f59f7fff700 1 -- 172.21.15.60:0/1733588789 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59e8007240 con 0x7f5a00060710 2022-06-29T16:00:50.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.363+0000 7f5a053d4700 1 -- 172.21.15.60:0/1733588789 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a0013f160 con 0x7f5a00060710 2022-06-29T16:00:50.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.363+0000 7f59f7fff700 1 -- 172.21.15.60:0/1733588789 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59e8006700 con 0x7f5a00060710 2022-06-29T16:00:50.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.363+0000 7f5a053d4700 1 -- 172.21.15.60:0/1733588789 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a0005f740 con 0x7f5a00060710 2022-06-29T16:00:50.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.367+0000 7f5a053d4700 1 -- 172.21.15.60:0/1733588789 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a000ae510 con 0x7f5a00060710 2022-06-29T16:00:50.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.367+0000 7f59f7fff700 1 -- 172.21.15.60:0/1733588789 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f59e8010030 con 0x7f5a00060710 2022-06-29T16:00:50.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.367+0000 7f59f7fff700 1 --2- 172.21.15.60:0/1733588789 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f59e4053b00 0x7f59e4055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:50.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.367+0000 7f59f7fff700 1 -- 172.21.15.60:0/1733588789 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f59e806bd60 con 0x7f5a00060710 2022-06-29T16:00:50.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.367+0000 7f59fe7fc700 1 --2- 172.21.15.60:0/1733588789 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f59e4053b00 0x7f59e4055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:50.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.367+0000 7f59fe7fc700 1 --2- 172.21.15.60:0/1733588789 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f59e4053b00 0x7f59e4055fc0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f59f000b000 tx=0x7f59f0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:50.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.379+0000 7f59f7fff700 1 -- 172.21.15.60:0/1733588789 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f59e8042af0 con 0x7f5a00060710 2022-06-29T16:00:50.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.507+0000 7f5a053d4700 1 -- 172.21.15.60:0/1733588789 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5a001459d0 con 0x7f5a00060710 2022-06-29T16:00:50.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.507+0000 7f59f7fff700 1 -- 172.21.15.60:0/1733588789 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v43) v1 ==== 104+0+10 (secure 0 0 0) 0x7f59e8046110 con 0x7f5a00060710 2022-06-29T16:00:50.516 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:00:50.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.507+0000 7f5a053d4700 1 -- 172.21.15.60:0/1733588789 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f59e4053b00 msgr2=0x7f59e4055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:50.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.507+0000 7f5a053d4700 1 --2- 172.21.15.60:0/1733588789 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f59e4053b00 0x7f59e4055fc0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f59f000b000 tx=0x7f59f0008040).stop 2022-06-29T16:00:50.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.507+0000 7f5a053d4700 1 -- 172.21.15.60:0/1733588789 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a00060710 msgr2=0x7f5a0005f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:50.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.507+0000 7f5a053d4700 1 --2- 172.21.15.60:0/1733588789 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a00060710 0x7f5a0005f250 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f5a000ae180 tx=0x7f59e8007c10).stop 2022-06-29T16:00:50.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.507+0000 7f5a053d4700 1 -- 172.21.15.60:0/1733588789 shutdown_connections 2022-06-29T16:00:50.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.507+0000 7f5a053d4700 1 --2- 172.21.15.60:0/1733588789 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f59e4053b00 0x7f59e4055fc0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:50.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.507+0000 7f5a053d4700 1 --2- 172.21.15.60:0/1733588789 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5a0013ec70 0x7f5a0013fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:50.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.507+0000 7f5a053d4700 1 --2- 172.21.15.60:0/1733588789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a000b5e90 0x7f5a0013f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:50.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.507+0000 7f5a053d4700 1 --2- 172.21.15.60:0/1733588789 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a00060710 0x7f5a0005f250 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:50.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.507+0000 7f5a053d4700 1 -- 172.21.15.60:0/1733588789 >> 172.21.15.60:0/1733588789 conn(0x7f5a00063b50 msgr2=0x7f5a000b3680 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:50.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.507+0000 7f5a053d4700 1 -- 172.21.15.60:0/1733588789 shutdown_connections 2022-06-29T16:00:50.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.507+0000 7f5a053d4700 1 -- 172.21.15.60:0/1733588789 wait complete. 2022-06-29T16:00:50.531 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-29T16:00:50.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.631+0000 7f596dfc5700 1 Processor -- start 2022-06-29T16:00:50.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.631+0000 7f596dfc5700 1 -- start start 2022-06-29T16:00:50.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.635+0000 7f596dfc5700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f59680b28e0 0x7f59680b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:50.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.635+0000 7f596dfc5700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f596813fa40 con 0x7f5968065450 2022-06-29T16:00:50.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.635+0000 7f596dfc5700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f596813fce0 con 0x7f5968060760 2022-06-29T16:00:50.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.635+0000 7f596dfc5700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5968140860 con 0x7f59680b28e0 2022-06-29T16:00:50.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.635+0000 7f596cfc3700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f59680b28e0 0x7f59680b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:50.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.635+0000 7f596cfc3700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f59680b28e0 0x7f59680b2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47458/0 (socket says 172.21.15.60:47458) 2022-06-29T16:00:50.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.635+0000 7f596cfc3700 1 -- 172.21.15.60:0/2470172302 learned_addr learned my addr 172.21.15.60:0/2470172302 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:50.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.635+0000 7f5967fff700 1 --1- 172.21.15.60:0/2470172302 >> v1:172.21.15.60:6789/0 conn(0x7f5968065450 0x7f5968139340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55546/0 (socket says 172.21.15.60:55546) 2022-06-29T16:00:50.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.635+0000 7f596cfc3700 1 -- 172.21.15.60:0/2470172302 >> v1:172.21.15.92:6789/0 conn(0x7f5968060760 legacy=0x7f5968066390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:50.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.635+0000 7f596cfc3700 1 -- 172.21.15.60:0/2470172302 >> v1:172.21.15.60:6789/0 conn(0x7f5968065450 legacy=0x7f5968139340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:50.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.635+0000 7f596cfc3700 1 -- 172.21.15.60:0/2470172302 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f596813f6f0 con 0x7f59680b28e0 2022-06-29T16:00:50.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.635+0000 7f596cfc3700 1 --2- 172.21.15.60:0/2470172302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59680b28e0 0x7f59680b2cc0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f59600032f0 tx=0x7f5960009290).ready entity=mon.0 client_cookie=ba544da9001eba2b server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:50.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.635+0000 7f59677fe700 1 -- 172.21.15.60:0/2470172302 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f596000b8a0 con 0x7f59680b28e0 2022-06-29T16:00:50.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.635+0000 7f59677fe700 1 -- 172.21.15.60:0/2470172302 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f596000ba00 con 0x7f59680b28e0 2022-06-29T16:00:50.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.635+0000 7f59677fe700 1 -- 172.21.15.60:0/2470172302 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f596000bb70 con 0x7f59680b28e0 2022-06-29T16:00:50.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.635+0000 7f596dfc5700 1 -- 172.21.15.60:0/2470172302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59680b28e0 msgr2=0x7f59680b2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:50.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.635+0000 7f596dfc5700 1 --2- 172.21.15.60:0/2470172302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59680b28e0 0x7f59680b2cc0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f59600032f0 tx=0x7f5960009290).stop 2022-06-29T16:00:50.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.635+0000 7f596dfc5700 1 -- 172.21.15.60:0/2470172302 shutdown_connections 2022-06-29T16:00:50.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.635+0000 7f596dfc5700 1 --2- 172.21.15.60:0/2470172302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59680b28e0 0x7f59680b2cc0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:50.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.635+0000 7f596dfc5700 1 -- 172.21.15.60:0/2470172302 >> 172.21.15.60:0/2470172302 conn(0x7f5968061c80 msgr2=0x7f59680637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:50.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.635+0000 7f596dfc5700 1 -- 172.21.15.60:0/2470172302 shutdown_connections 2022-06-29T16:00:50.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.635+0000 7f596dfc5700 1 -- 172.21.15.60:0/2470172302 wait complete. 2022-06-29T16:00:50.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.639+0000 7f596dfc5700 1 Processor -- start 2022-06-29T16:00:50.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.639+0000 7f596dfc5700 1 -- start start 2022-06-29T16:00:50.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.639+0000 7f596dfc5700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5968060760 0x7f596813dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:50.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.639+0000 7f596dfc5700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5968065450 0x7f596812c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:50.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.639+0000 7f596dfc5700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59680b28e0 0x7f5968140f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:50.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.639+0000 7f596dfc5700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59680ad9c0 con 0x7f59680b28e0 2022-06-29T16:00:50.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.639+0000 7f596dfc5700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596812cea0 con 0x7f5968060760 2022-06-29T16:00:50.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.639+0000 7f596dfc5700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f596812d010 con 0x7f5968065450 2022-06-29T16:00:50.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.639+0000 7f596d7c4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59680b28e0 0x7f5968140f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:50.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.639+0000 7f596d7c4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59680b28e0 0x7f5968140f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47468/0 (socket says 172.21.15.60:47468) 2022-06-29T16:00:50.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.639+0000 7f596d7c4700 1 -- 172.21.15.60:0/1709133035 learned_addr learned my addr 172.21.15.60:0/1709133035 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:50.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.639+0000 7f5967fff700 1 --2- 172.21.15.60:0/1709133035 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5968065450 0x7f596812c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:50.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.639+0000 7f596cfc3700 1 --2- 172.21.15.60:0/1709133035 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5968060760 0x7f596813dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:50.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.639+0000 7f596d7c4700 1 -- 172.21.15.60:0/1709133035 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5968065450 msgr2=0x7f596812c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:50.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.639+0000 7f596d7c4700 1 --2- 172.21.15.60:0/1709133035 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5968065450 0x7f596812c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:50.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.639+0000 7f596d7c4700 1 -- 172.21.15.60:0/1709133035 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5968060760 msgr2=0x7f596813dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:50.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.639+0000 7f596d7c4700 1 --2- 172.21.15.60:0/1709133035 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5968060760 0x7f596813dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:50.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.639+0000 7f596d7c4700 1 -- 172.21.15.60:0/1709133035 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5960009530 con 0x7f59680b28e0 2022-06-29T16:00:50.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.639+0000 7f596d7c4700 1 --2- 172.21.15.60:0/1709133035 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59680b28e0 0x7f5968140f50 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f59680adb80 tx=0x7f5958007bc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:50.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.639+0000 7f5965ffb700 1 -- 172.21.15.60:0/1709133035 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5958012400 con 0x7f59680b28e0 2022-06-29T16:00:50.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.639+0000 7f5965ffb700 1 -- 172.21.15.60:0/1709133035 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5958012560 con 0x7f59680b28e0 2022-06-29T16:00:50.672 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.639+0000 7f596dfc5700 1 -- 172.21.15.60:0/1709133035 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f596813f310 con 0x7f59680b28e0 2022-06-29T16:00:50.672 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.639+0000 7f5965ffb700 1 -- 172.21.15.60:0/1709133035 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59580128c0 con 0x7f59680b28e0 2022-06-29T16:00:50.672 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.639+0000 7f596dfc5700 1 -- 172.21.15.60:0/1709133035 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5968141d90 con 0x7f59680b28e0 2022-06-29T16:00:50.673 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.643+0000 7f596dfc5700 1 -- 172.21.15.60:0/1709133035 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59680adbc0 con 0x7f59680b28e0 2022-06-29T16:00:50.673 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.643+0000 7f5965ffb700 1 -- 172.21.15.60:0/1709133035 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5958019030 con 0x7f59680b28e0 2022-06-29T16:00:50.674 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.643+0000 7f5965ffb700 1 --2- 172.21.15.60:0/1709133035 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5948053aa0 0x7f5948055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:50.674 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.643+0000 7f5965ffb700 1 -- 172.21.15.60:0/1709133035 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f595806b240 con 0x7f59680b28e0 2022-06-29T16:00:50.675 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.643+0000 7f596cfc3700 1 --2- 172.21.15.60:0/1709133035 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5948053aa0 0x7f5948055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:50.675 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.647+0000 7f596cfc3700 1 --2- 172.21.15.60:0/1709133035 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5948053aa0 0x7f5948055f60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f5960019000 tx=0x7f5960016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:50.676 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.651+0000 7f5965ffb700 1 -- 172.21.15.60:0/1709133035 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5958042100 con 0x7f59680b28e0 2022-06-29T16:00:50.783 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.775+0000 7f596dfc5700 1 -- 172.21.15.60:0/1709133035 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f59680573d0 con 0x7f59680b28e0 2022-06-29T16:00:50.784 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.775+0000 7f5965ffb700 1 -- 172.21.15.60:0/1709133035 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v43) v1 ==== 100+0+10 (secure 0 0 0) 0x7f5958045720 con 0x7f59680b28e0 2022-06-29T16:00:50.785 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:00:50.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.775+0000 7f596dfc5700 1 -- 172.21.15.60:0/1709133035 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5948053aa0 msgr2=0x7f5948055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:50.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.775+0000 7f596dfc5700 1 --2- 172.21.15.60:0/1709133035 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5948053aa0 0x7f5948055f60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f5960019000 tx=0x7f5960016000).stop 2022-06-29T16:00:50.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.775+0000 7f596dfc5700 1 -- 172.21.15.60:0/1709133035 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59680b28e0 msgr2=0x7f5968140f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:50.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.775+0000 7f596dfc5700 1 --2- 172.21.15.60:0/1709133035 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59680b28e0 0x7f5968140f50 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f59680adb80 tx=0x7f5958007bc0).stop 2022-06-29T16:00:50.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.775+0000 7f596dfc5700 1 -- 172.21.15.60:0/1709133035 shutdown_connections 2022-06-29T16:00:50.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.775+0000 7f596dfc5700 1 --2- 172.21.15.60:0/1709133035 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5948053aa0 0x7f5948055f60 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:50.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.775+0000 7f596dfc5700 1 --2- 172.21.15.60:0/1709133035 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59680b28e0 0x7f5968140f50 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:50.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.775+0000 7f596dfc5700 1 --2- 172.21.15.60:0/1709133035 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5968065450 0x7f596812c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:50.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.775+0000 7f596dfc5700 1 --2- 172.21.15.60:0/1709133035 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5968060760 0x7f596813dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:50.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.775+0000 7f596dfc5700 1 -- 172.21.15.60:0/1709133035 >> 172.21.15.60:0/1709133035 conn(0x7f5968061c80 msgr2=0x7f596812d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:50.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.779+0000 7f596dfc5700 1 -- 172.21.15.60:0/1709133035 shutdown_connections 2022-06-29T16:00:50.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.779+0000 7f596dfc5700 1 -- 172.21.15.60:0/1709133035 wait complete. 2022-06-29T16:00:50.797 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-06-29T16:00:50.799 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:00:50.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.895+0000 7f56c9f50700 1 Processor -- start 2022-06-29T16:00:50.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.895+0000 7f56c9f50700 1 -- start start 2022-06-29T16:00:50.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.895+0000 7f56c9f50700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56c413f3d0 0x7f56c4060260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:50.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.895+0000 7f56c9f50700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f56c4143820 0x7f56c40b6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:50.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.895+0000 7f56c9f50700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56c41430e0 con 0x7f56c40b6c00 2022-06-29T16:00:50.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.895+0000 7f56c9f50700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56c4142ef0 con 0x7f56c4143820 2022-06-29T16:00:50.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.895+0000 7f56c9f50700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c40acb90 con 0x7f56c413f3d0 2022-06-29T16:00:50.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.895+0000 7f56c2ffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f56c4143820 0x7f56c40b6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:50.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.895+0000 7f56c2ffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f56c4143820 0x7f56c40b6710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47474/0 (socket says 172.21.15.60:47474) 2022-06-29T16:00:50.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.895+0000 7f56c2ffd700 1 -- 172.21.15.60:0/1598262105 learned_addr learned my addr 172.21.15.60:0/1598262105 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:50.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.895+0000 7f56c37fe700 1 --2- 172.21.15.60:0/1598262105 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56c413f3d0 0x7f56c4060260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:50.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.895+0000 7f56c2ffd700 1 -- 172.21.15.60:0/1598262105 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56c413f3d0 msgr2=0x7f56c4060260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:50.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.895+0000 7f56c2ffd700 1 --2- 172.21.15.60:0/1598262105 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56c413f3d0 0x7f56c4060260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:50.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.895+0000 7f56c2ffd700 1 -- 172.21.15.60:0/1598262105 >> v1:172.21.15.60:6789/0 conn(0x7f56c40b6c00 legacy=0x7f56c413f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:00:50.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.895+0000 7f56c2ffd700 1 -- 172.21.15.60:0/1598262105 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c4143380 con 0x7f56c4143820 2022-06-29T16:00:50.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.895+0000 7f56c2ffd700 1 --2- 172.21.15.60:0/1598262105 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56c4143820 0x7f56c40b6710 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f56bc0032f0 tx=0x7f56bc009290).ready entity=mon.0 client_cookie=64e2221d60d4631b server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:50.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.895+0000 7f56c27fc700 1 -- 172.21.15.60:0/1598262105 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f56bc00b8a0 con 0x7f56c4143820 2022-06-29T16:00:50.915 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.895+0000 7f56c27fc700 1 -- 172.21.15.60:0/1598262105 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56bc00ba00 con 0x7f56c4143820 2022-06-29T16:00:50.915 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.895+0000 7f56c27fc700 1 -- 172.21.15.60:0/1598262105 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f56bc00bb70 con 0x7f56c4143820 2022-06-29T16:00:50.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.895+0000 7f56c9f50700 1 -- 172.21.15.60:0/1598262105 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56c4143820 msgr2=0x7f56c40b6710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:50.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.895+0000 7f56c9f50700 1 --2- 172.21.15.60:0/1598262105 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56c4143820 0x7f56c40b6710 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f56bc0032f0 tx=0x7f56bc009290).stop 2022-06-29T16:00:50.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.895+0000 7f56c9f50700 1 -- 172.21.15.60:0/1598262105 shutdown_connections 2022-06-29T16:00:50.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.895+0000 7f56c9f50700 1 --2- 172.21.15.60:0/1598262105 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56c4143820 0x7f56c40b6710 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:50.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.895+0000 7f56c9f50700 1 --2- 172.21.15.60:0/1598262105 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56c413f3d0 0x7f56c4060260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:50.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.895+0000 7f56c9f50700 1 -- 172.21.15.60:0/1598262105 >> 172.21.15.60:0/1598262105 conn(0x7f56c40643e0 msgr2=0x7f56c40b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:50.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.895+0000 7f56c9f50700 1 -- 172.21.15.60:0/1598262105 shutdown_connections 2022-06-29T16:00:50.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.895+0000 7f56c9f50700 1 -- 172.21.15.60:0/1598262105 wait complete. 2022-06-29T16:00:50.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.899+0000 7f56c9f50700 1 Processor -- start 2022-06-29T16:00:50.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.899+0000 7f56c9f50700 1 -- start start 2022-06-29T16:00:50.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.899+0000 7f56c9f50700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56c40b6c00 0x7f56c40b5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:50.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.899+0000 7f56c9f50700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56c413f3d0 0x7f56c413c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:50.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.899+0000 7f56c9f50700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56c4143820 0x7f56c413c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:50.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.899+0000 7f56c9f50700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c40677d0 con 0x7f56c40b6c00 2022-06-29T16:00:50.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.899+0000 7f56c9f50700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c4065f30 con 0x7f56c4143820 2022-06-29T16:00:50.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.899+0000 7f56c9f50700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c4067530 con 0x7f56c413f3d0 2022-06-29T16:00:50.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.899+0000 7f56c37fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56c40b6c00 0x7f56c40b5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:50.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.899+0000 7f56c37fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56c40b6c00 0x7f56c40b5100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47480/0 (socket says 172.21.15.60:47480) 2022-06-29T16:00:50.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.899+0000 7f56c37fe700 1 -- 172.21.15.60:0/2739918253 learned_addr learned my addr 172.21.15.60:0/2739918253 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:00:50.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.899+0000 7f56c2ffd700 1 --2- 172.21.15.60:0/2739918253 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56c413f3d0 0x7f56c413c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:50.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.899+0000 7f56c3fff700 1 --2- 172.21.15.60:0/2739918253 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56c4143820 0x7f56c413c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:50.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.899+0000 7f56c37fe700 1 -- 172.21.15.60:0/2739918253 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56c413f3d0 msgr2=0x7f56c413c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:50.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.899+0000 7f56c37fe700 1 --2- 172.21.15.60:0/2739918253 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56c413f3d0 0x7f56c413c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:50.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.899+0000 7f56c37fe700 1 -- 172.21.15.60:0/2739918253 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56c4143820 msgr2=0x7f56c413c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:50.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.899+0000 7f56c37fe700 1 --2- 172.21.15.60:0/2739918253 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56c4143820 0x7f56c413c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:50.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.899+0000 7f56c37fe700 1 -- 172.21.15.60:0/2739918253 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56bc009530 con 0x7f56c40b6c00 2022-06-29T16:00:50.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.899+0000 7f56c37fe700 1 --2- 172.21.15.60:0/2739918253 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56c40b6c00 0x7f56c40b5100 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f56ac0030b0 tx=0x7f56ac008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:50.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.899+0000 7f56c0ff9700 1 -- 172.21.15.60:0/2739918253 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f56ac011780 con 0x7f56c40b6c00 2022-06-29T16:00:50.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.899+0000 7f56c0ff9700 1 -- 172.21.15.60:0/2739918253 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56ac0118e0 con 0x7f56c40b6c00 2022-06-29T16:00:50.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.899+0000 7f56c9f50700 1 -- 172.21.15.60:0/2739918253 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56c4099390 con 0x7f56c40b6c00 2022-06-29T16:00:50.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.899+0000 7f56c0ff9700 1 -- 172.21.15.60:0/2739918253 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f56ac011c40 con 0x7f56c40b6c00 2022-06-29T16:00:50.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.903+0000 7f56c9f50700 1 -- 172.21.15.60:0/2739918253 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56c4099830 con 0x7f56c40b6c00 2022-06-29T16:00:50.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.903+0000 7f56c9f50700 1 -- 172.21.15.60:0/2739918253 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56c40573d0 con 0x7f56c40b6c00 2022-06-29T16:00:50.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.903+0000 7f56c0ff9700 1 -- 172.21.15.60:0/2739918253 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f56ac01c460 con 0x7f56c40b6c00 2022-06-29T16:00:50.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.903+0000 7f56c0ff9700 1 --2- 172.21.15.60:0/2739918253 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f56a8053aa0 0x7f56a8055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:00:50.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.903+0000 7f56c0ff9700 1 -- 172.21.15.60:0/2739918253 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f56ac06c080 con 0x7f56c40b6c00 2022-06-29T16:00:50.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.903+0000 7f56c2ffd700 1 --2- 172.21.15.60:0/2739918253 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f56a8053aa0 0x7f56a8055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:00:50.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.903+0000 7f56c2ffd700 1 --2- 172.21.15.60:0/2739918253 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f56a8053aa0 0x7f56a8055f60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f56bc01b000 tx=0x7f56bc016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:00:50.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:50.911+0000 7f56c0ff9700 1 -- 172.21.15.60:0/2739918253 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f56ac042f40 con 0x7f56c40b6c00 2022-06-29T16:00:51.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:51.075+0000 7f56c9f50700 1 -- 172.21.15.60:0/2739918253 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56c4067990 con 0x7f56c40b6c00 2022-06-29T16:00:51.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:51.075+0000 7f56c0ff9700 1 -- 172.21.15.60:0/2739918253 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+2138 (secure 0 0 0) 0x7f56ac046560 con 0x7f56c40b6c00 2022-06-29T16:00:51.085 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:00:51.086 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":15,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4937,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/1673184114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1673184114},{"type":"v1","addr":"172.21.15.92:6835","nonce":1673184114}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":13}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:00:20.685676+0000","modified":"2022-06-29T16:00:21.689299+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5308":{"gid":5308,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/1964426267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1964426267},{"type":"v1","addr":"172.21.15.60:6833","nonce":1964426267}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-06-29T16:00:51.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:51.079+0000 7f56c9f50700 1 -- 172.21.15.60:0/2739918253 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f56a8053aa0 msgr2=0x7f56a8055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:51.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:51.079+0000 7f56c9f50700 1 --2- 172.21.15.60:0/2739918253 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f56a8053aa0 0x7f56a8055f60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f56bc01b000 tx=0x7f56bc016000).stop 2022-06-29T16:00:51.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:51.079+0000 7f56c9f50700 1 -- 172.21.15.60:0/2739918253 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56c40b6c00 msgr2=0x7f56c40b5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:00:51.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:51.079+0000 7f56c9f50700 1 --2- 172.21.15.60:0/2739918253 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56c40b6c00 0x7f56c40b5100 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f56ac0030b0 tx=0x7f56ac008040).stop 2022-06-29T16:00:51.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:51.079+0000 7f56c9f50700 1 -- 172.21.15.60:0/2739918253 shutdown_connections 2022-06-29T16:00:51.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:51.079+0000 7f56c9f50700 1 --2- 172.21.15.60:0/2739918253 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f56a8053aa0 0x7f56a8055f60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:51.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:51.079+0000 7f56c9f50700 1 --2- 172.21.15.60:0/2739918253 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56c4143820 0x7f56c413c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:51.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:51.079+0000 7f56c9f50700 1 --2- 172.21.15.60:0/2739918253 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56c413f3d0 0x7f56c413c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:51.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:51.079+0000 7f56c9f50700 1 --2- 172.21.15.60:0/2739918253 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56c40b6c00 0x7f56c40b5100 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:00:51.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:51.079+0000 7f56c9f50700 1 -- 172.21.15.60:0/2739918253 >> 172.21.15.60:0/2739918253 conn(0x7f56c40643e0 msgr2=0x7f56c40b2990 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:00:51.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:51.079+0000 7f56c9f50700 1 -- 172.21.15.60:0/2739918253 shutdown_connections 2022-06-29T16:00:51.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:00:51.079+0000 7f56c9f50700 1 -- 172.21.15.60:0/2739918253 wait complete. 2022-06-29T16:00:51.091 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 15 2022-06-29T16:00:51.301 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:00:20.685676+0000', 'modified': '2022-06-29T16:00:21.689299+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5308}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5308': {'gid': 5308, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.60:6833/1964426267', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6832', 'nonce': 1964426267}, {'type': 'v1', 'addr': '172.21.15.60:6833', 'nonce': 1964426267}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:00:51.302 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:00:51.302 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-06-29T16:00:51.420 INFO:teuthology.orchestra.run.smithi060.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":5308,"want_state":"up:active","state":"up:active","rank_uptime":30.716231020999999,"mdsmap_epoch":15,"osdmap_epoch":43,"osdmap_epoch_barrier":42,"uptime":35.668464595000003} 2022-06-29T16:00:51.421 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 5308, "mdsmap_epoch": 15, "osdmap_epoch": 43, "osdmap_epoch_barrier": 42, "rank_uptime": 30.716231021, "state": "up:active", "uptime": 35.668464595, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:00:51.422 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-06-29T16:00:51.423 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-06-29T16:00:51.424 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi121.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-06-29T16:00:51.424 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:00:51.425 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:00:51.433 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:00:51.452 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:00:51.452 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:00:51.465 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:00:51.466 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:00:51.467 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:00:51.467 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:00:51.514 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-06-29T16:00:51.515 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:00:51.519 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:00:51.535 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:00:51.536 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:00:51.549 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:00:51.550 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:00:51.551 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:00:51.551 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:00:51.593 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:2022-06-29T16:00:51.587+0000 7fd90990d080 -1 init, newargv = 0x562c91d852a0 newargc=9ceph-fuse[12515 2022-06-29T16:00:51.594 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:]: starting ceph client 2022-06-29T16:00:51.602 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[12515]: starting fuse 2022-06-29T16:00:52.700 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:00:52.720 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:00:52.721 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:00:52.776 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:00:52.778 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:00:52.779 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:00:52.779 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:00:52.786 INFO:teuthology.orchestra.run.smithi121.stdout:52 2022-06-29T16:00:52.787 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-06-29T16:00:52.788 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:00:52.788 DEBUG:teuthology.orchestra.run.smithi121:> sudo python3 -c ' 2022-06-29T16:00:52.789 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2022-06-29T16:00:52.789 DEBUG:teuthology.orchestra.run.smithi121:> import re 2022-06-29T16:00:52.790 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-06-29T16:00:52.790 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2022-06-29T16:00:52.791 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:00:52.791 DEBUG:teuthology.orchestra.run.smithi121:> def find_socket(client_name): 2022-06-29T16:00:52.792 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-06-29T16:00:52.793 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2022-06-29T16:00:52.793 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:00:52.794 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2022-06-29T16:00:52.795 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2022-06-29T16:00:52.795 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2022-06-29T16:00:52.795 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2022-06-29T16:00:52.796 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:00:52.797 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2022-06-29T16:00:52.798 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-06-29T16:00:52.798 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2022-06-29T16:00:52.799 DEBUG:teuthology.orchestra.run.smithi121:> return f 2022-06-29T16:00:52.800 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-06-29T16:00:52.800 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:00:52.801 DEBUG:teuthology.orchestra.run.smithi121:> print(find_socket("client.0")) 2022-06-29T16:00:52.802 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-06-29T16:00:52.846 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.0.12515.asok 2022-06-29T16:00:52.850 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.12515.asok 2022-06-29T16:00:52.851 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:00:52.851 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.12515.asok status 2022-06-29T16:00:52.988 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2022-06-29T16:00:52.988 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2022-06-29T16:00:52.988 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "f25ffac72e5a9e63860ff7deb6767a11f0559a16", 2022-06-29T16:00:52.988 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 15.2.16-304-gf25ffac7 (f25ffac72e5a9e63860ff7deb6767a11f0559a16) octopus (stable)", 2022-06-29T16:00:52.989 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "0", 2022-06-29T16:00:52.989 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2022-06-29T16:00:52.989 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-06-29T16:00:52.989 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "12515", 2022-06-29T16:00:52.990 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2022-06-29T16:00:52.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:00:52.990 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2022-06-29T16:00:52.990 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2022-06-29T16:00:52.991 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 5469, 2022-06-29T16:00:52.991 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2022-06-29T16:00:52.991 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2022-06-29T16:00:52.991 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2022-06-29T16:00:52.992 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 5469 2022-06-29T16:00:52.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:00:52.992 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-06-29T16:00:52.992 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-06-29T16:00:52.992 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-06-29T16:00:52.993 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 486765976 2022-06-29T16:00:52.993 INFO:teuthology.orchestra.run.smithi121.stdout: } 2022-06-29T16:00:52.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:00:52.994 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-06-29T16:00:52.994 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-06-29T16:00:52.994 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-06-29T16:00:52.994 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 486765976 2022-06-29T16:00:52.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:00:52.995 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.5469 172.21.15.121:0/486765976", 2022-06-29T16:00:52.995 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "172.21.15.121:0/486765976", 2022-06-29T16:00:52.995 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2022-06-29T16:00:52.996 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 15, 2022-06-29T16:00:52.996 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 43, 2022-06-29T16:00:52.996 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2022-06-29T16:00:52.996 INFO:teuthology.orchestra.run.smithi121.stdout: "blacklisted": false, 2022-06-29T16:00:52.997 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2022-06-29T16:00:52.997 INFO:teuthology.orchestra.run.smithi121.stdout:} 2022-06-29T16:00:52.998 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:00:52.999 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:00:52.999 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:00:53.009 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T16:00:53.009 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T16:00:53.010 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:00:53.010 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:00:58.293 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:00:58.296 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:00:58.299 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:00:58.302 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:00:58.304 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:01:28.375 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:01:28.379 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:01:28.382 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:01:28.385 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:01:28.387 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:01:40.749 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-06-29T16:01:40.750 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:01:40.769 DEBUG:teuthology.orchestra.run.smithi092:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:01:40.792 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:01:40.813 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:01:40.836 DEBUG:teuthology.orchestra.run.smithi197:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:01:40.857 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-06-29T16:01:40.858 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:01:40.859 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:01:40.859 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:01:40.868 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T16:01:40.869 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T16:01:40.870 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2022-06-29T16:01:40.870 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-06-29T16:01:40.870 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:01:40.885 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[12515]: fuse finished with error 0 and tester_r 0 2022-06-29T16:01:40.887 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:01:40.888 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:01:40.888 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:01:40.935 INFO:teuthology.orchestra.run.smithi121.stdout:ext2/ext3 2022-06-29T16:01:40.936 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-06-29T16:01:41.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:01:41.054 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:01:41.062 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:01:41.062 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:01:41.063 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:01:41.070 INFO:teuthology.orchestra.run.smithi156.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-06-29T16:01:41.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:01:41.072 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-06-29T16:01:41.073 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:01:41.073 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:01:41.121 INFO:teuthology.orchestra.run.smithi156.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-06-29T16:01:41.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:01:41.123 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:01:41.123 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:01:41.124 DEBUG:teuthology.orchestra.run.smithi197:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T16:01:41.131 INFO:teuthology.orchestra.run.smithi197.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-06-29T16:01:41.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:01:41.133 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-06-29T16:01:41.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:01:41.134 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T16:01:41.183 INFO:teuthology.orchestra.run.smithi197.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-06-29T16:01:41.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:01:41.185 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-06-29T16:01:41.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.287+0000 7f78e1d0c700 1 Processor -- start 2022-06-29T16:01:41.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.287+0000 7f78e1d0c700 1 -- start start 2022-06-29T16:01:41.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.291+0000 7f78e1d0c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f78dc0aebc0 0x7f78dc0b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:41.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.291+0000 7f78e1d0c700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f78dc13fa90 0x7f78dc13e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:41.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.291+0000 7f78e1d0c700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78dc13ffd0 con 0x7f78dc0b1080 2022-06-29T16:01:41.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.291+0000 7f78e1d0c700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78dc0b1460 con 0x7f78dc0aebc0 2022-06-29T16:01:41.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.291+0000 7f78e1d0c700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78dc0acb60 con 0x7f78dc13fa90 2022-06-29T16:01:41.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.291+0000 7f78d3fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f78dc0aebc0 0x7f78dc0b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:41.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.291+0000 7f78db7fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f78dc0b1080 0x7f78dc0ae4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55572/0 (socket says 172.21.15.60:55572) 2022-06-29T16:01:41.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.291+0000 7f78d3fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f78dc0aebc0 0x7f78dc0b17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47488/0 (socket says 172.21.15.60:47488) 2022-06-29T16:01:41.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.291+0000 7f78db7fe700 1 -- 172.21.15.60:0/1410835863 learned_addr learned my addr 172.21.15.60:0/1410835863 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:01:41.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.291+0000 7f78dbfff700 1 --2- 172.21.15.60:0/1410835863 >> v2:172.21.15.92:3300/0 conn(0x7f78dc13fa90 0x7f78dc13e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:41.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.291+0000 7f78d3fff700 1 -- 172.21.15.60:0/1410835863 >> v2:172.21.15.92:3300/0 conn(0x7f78dc13fa90 msgr2=0x7f78dc13e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:01:41.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.291+0000 7f78d3fff700 1 --2- 172.21.15.60:0/1410835863 >> v2:172.21.15.92:3300/0 conn(0x7f78dc13fa90 0x7f78dc13e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:41.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.291+0000 7f78d3fff700 1 -- 172.21.15.60:0/1410835863 >> v1:172.21.15.60:6789/0 conn(0x7f78dc0b1080 legacy=0x7f78dc0ae4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:41.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.291+0000 7f78daffd700 1 -- 172.21.15.60:0/1410835863 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3377251180 0 0) 0x7f78dc13ffd0 con 0x7f78dc0b1080 2022-06-29T16:01:41.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.291+0000 7f78d3fff700 1 -- 172.21.15.60:0/1410835863 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78dc0ace00 con 0x7f78dc0aebc0 2022-06-29T16:01:41.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.291+0000 7f78d3fff700 1 --2- 172.21.15.60:0/1410835863 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78dc0aebc0 0x7f78dc0b17c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f78c4002f70 tx=0x7f78c4006df0).ready entity=mon.0 client_cookie=59ca30196b105692 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:41.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.291+0000 7f78daffd700 1 -- 172.21.15.60:0/1410835863 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f78c400fa60 con 0x7f78dc0aebc0 2022-06-29T16:01:41.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.291+0000 7f78daffd700 1 -- 172.21.15.60:0/1410835863 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c4000e90 con 0x7f78dc0aebc0 2022-06-29T16:01:41.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.291+0000 7f78daffd700 1 -- 172.21.15.60:0/1410835863 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f78c400fdb0 con 0x7f78dc0aebc0 2022-06-29T16:01:41.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.291+0000 7f78e1d0c700 1 -- 172.21.15.60:0/1410835863 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78dc0aebc0 msgr2=0x7f78dc0b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:41.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.291+0000 7f78e1d0c700 1 --2- 172.21.15.60:0/1410835863 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78dc0aebc0 0x7f78dc0b17c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f78c4002f70 tx=0x7f78c4006df0).stop 2022-06-29T16:01:41.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.291+0000 7f78e1d0c700 1 -- 172.21.15.60:0/1410835863 shutdown_connections 2022-06-29T16:01:41.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.291+0000 7f78e1d0c700 1 --2- 172.21.15.60:0/1410835863 >> v2:172.21.15.92:3300/0 conn(0x7f78dc13fa90 0x7f78dc13e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:41.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.291+0000 7f78e1d0c700 1 --2- 172.21.15.60:0/1410835863 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78dc0aebc0 0x7f78dc0b17c0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:41.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.291+0000 7f78e1d0c700 1 -- 172.21.15.60:0/1410835863 >> 172.21.15.60:0/1410835863 conn(0x7f78dc0636f0 msgr2=0x7f78dc0b5360 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:41.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.291+0000 7f78e1d0c700 1 -- 172.21.15.60:0/1410835863 shutdown_connections 2022-06-29T16:01:41.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.291+0000 7f78e1d0c700 1 -- 172.21.15.60:0/1410835863 wait complete. 2022-06-29T16:01:41.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.295+0000 7f78e1d0c700 1 Processor -- start 2022-06-29T16:01:41.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.295+0000 7f78e1d0c700 1 -- start start 2022-06-29T16:01:41.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.295+0000 7f78e1d0c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78dc0aebc0 0x7f78dc139210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:41.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.295+0000 7f78e1d0c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78dc0b1080 0x7f78dc0b1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:41.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.295+0000 7f78e1d0c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78dc13fa90 0x7f78dc1384c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:41.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.295+0000 7f78e1d0c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78dc0b26b0 con 0x7f78dc0aebc0 2022-06-29T16:01:41.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.295+0000 7f78e1d0c700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78dc067a50 con 0x7f78dc13fa90 2022-06-29T16:01:41.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.295+0000 7f78e1d0c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78dc067310 con 0x7f78dc0b1080 2022-06-29T16:01:41.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.295+0000 7f78db7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78dc0aebc0 0x7f78dc139210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:41.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.295+0000 7f78db7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78dc0aebc0 0x7f78dc139210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47492/0 (socket says 172.21.15.60:47492) 2022-06-29T16:01:41.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.295+0000 7f78db7fe700 1 -- 172.21.15.60:0/3349230093 learned_addr learned my addr 172.21.15.60:0/3349230093 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:41.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.295+0000 7f78dbfff700 1 --2- 172.21.15.60:0/3349230093 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78dc13fa90 0x7f78dc1384c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:41.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.295+0000 7f78d3fff700 1 --2- 172.21.15.60:0/3349230093 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78dc0b1080 0x7f78dc0b1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:41.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.295+0000 7f78db7fe700 1 -- 172.21.15.60:0/3349230093 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78dc0b1080 msgr2=0x7f78dc0b1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:41.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.295+0000 7f78db7fe700 1 --2- 172.21.15.60:0/3349230093 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78dc0b1080 0x7f78dc0b1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:41.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.295+0000 7f78db7fe700 1 -- 172.21.15.60:0/3349230093 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78dc13fa90 msgr2=0x7f78dc1384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:41.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.295+0000 7f78db7fe700 1 --2- 172.21.15.60:0/3349230093 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78dc13fa90 0x7f78dc1384c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:41.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.295+0000 7f78db7fe700 1 -- 172.21.15.60:0/3349230093 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c400b180 con 0x7f78dc0aebc0 2022-06-29T16:01:41.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.295+0000 7f78db7fe700 1 --2- 172.21.15.60:0/3349230093 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78dc0aebc0 0x7f78dc139210 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f78cc000ed0 tx=0x7f78cc007ac0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:41.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.295+0000 7f78d97fa700 1 -- 172.21.15.60:0/3349230093 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f78cc0070d0 con 0x7f78dc0aebc0 2022-06-29T16:01:41.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.295+0000 7f78d97fa700 1 -- 172.21.15.60:0/3349230093 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78cc007230 con 0x7f78dc0aebc0 2022-06-29T16:01:41.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.295+0000 7f78e1d0c700 1 -- 172.21.15.60:0/3349230093 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78dc0675b0 con 0x7f78dc0aebc0 2022-06-29T16:01:41.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.295+0000 7f78d97fa700 1 -- 172.21.15.60:0/3349230093 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f78cc005610 con 0x7f78dc0aebc0 2022-06-29T16:01:41.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.295+0000 7f78e1d0c700 1 -- 172.21.15.60:0/3349230093 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78dc063e30 con 0x7f78dc0aebc0 2022-06-29T16:01:41.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.299+0000 7f78e1d0c700 1 -- 172.21.15.60:0/3349230093 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78dc067160 con 0x7f78dc0aebc0 2022-06-29T16:01:41.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.299+0000 7f78d97fa700 1 -- 172.21.15.60:0/3349230093 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f78cc01a060 con 0x7f78dc0aebc0 2022-06-29T16:01:41.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.299+0000 7f78d97fa700 1 --2- 172.21.15.60:0/3349230093 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f78c0053a10 0x7f78c0055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:41.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.299+0000 7f78d97fa700 1 -- 172.21.15.60:0/3349230093 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f78cc06b3a0 con 0x7f78dc0aebc0 2022-06-29T16:01:41.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.299+0000 7f78d3fff700 1 --2- 172.21.15.60:0/3349230093 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f78c0053a10 0x7f78c0055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:41.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.299+0000 7f78d3fff700 1 --2- 172.21.15.60:0/3349230093 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f78c0053a10 0x7f78c0055ed0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f78c401b000 tx=0x7f78c4016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:41.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.303+0000 7f78d97fa700 1 -- 172.21.15.60:0/3349230093 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f78cc0421b0 con 0x7f78dc0aebc0 2022-06-29T16:01:41.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.431+0000 7f78e1d0c700 1 -- 172.21.15.60:0/3349230093 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f78dc0573d0 con 0x7f78dc0aebc0 2022-06-29T16:01:41.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.431+0000 7f78d97fa700 1 -- 172.21.15.60:0/3349230093 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v43) v1 ==== 81+0+33645 (secure 0 0 0) 0x7f78cc018020 con 0x7f78dc0aebc0 2022-06-29T16:01:41.441 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:01:41.441 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:01:41.441 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 43, 2022-06-29T16:01:41.442 INFO:teuthology.orchestra.run.smithi060.stdout: "fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", 2022-06-29T16:01:41.442 INFO:teuthology.orchestra.run.smithi060.stdout: "created": "2022-06-29T15:58:07.965719+0000", 2022-06-29T16:01:41.442 INFO:teuthology.orchestra.run.smithi060.stdout: "modified": "2022-06-29T16:00:22.691182+0000", 2022-06-29T16:01:41.442 INFO:teuthology.orchestra.run.smithi060.stdout: "last_up_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:01:41.443 INFO:teuthology.orchestra.run.smithi060.stdout: "last_in_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:01:41.443 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-06-29T16:01:41.443 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_num": 5799936, 2022-06-29T16:01:41.443 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_set": [ 2022-06-29T16:01:41.444 INFO:teuthology.orchestra.run.smithi060.stdout: "pglog_hardlimit", 2022-06-29T16:01:41.444 INFO:teuthology.orchestra.run.smithi060.stdout: "purged_snapdirs", 2022-06-29T16:01:41.444 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_deletes", 2022-06-29T16:01:41.444 INFO:teuthology.orchestra.run.smithi060.stdout: "sortbitwise" 2022-06-29T16:01:41.444 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:01:41.445 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_version": 4, 2022-06-29T16:01:41.445 INFO:teuthology.orchestra.run.smithi060.stdout: "full_ratio": 0.94999998807907104, 2022-06-29T16:01:41.445 INFO:teuthology.orchestra.run.smithi060.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-06-29T16:01:41.445 INFO:teuthology.orchestra.run.smithi060.stdout: "nearfull_ratio": 0.85000002384185791, 2022-06-29T16:01:41.446 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_snapshot": "", 2022-06-29T16:01:41.446 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_max": 8, 2022-06-29T16:01:41.446 INFO:teuthology.orchestra.run.smithi060.stdout: "max_osd": 8, 2022-06-29T16:01:41.446 INFO:teuthology.orchestra.run.smithi060.stdout: "require_min_compat_client": "jewel", 2022-06-29T16:01:41.447 INFO:teuthology.orchestra.run.smithi060.stdout: "min_compat_client": "jewel", 2022-06-29T16:01:41.447 INFO:teuthology.orchestra.run.smithi060.stdout: "require_osd_release": "octopus", 2022-06-29T16:01:41.447 INFO:teuthology.orchestra.run.smithi060.stdout: "pools": [ 2022-06-29T16:01:41.447 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.447 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2022-06-29T16:01:41.448 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "device_health_metrics", 2022-06-29T16:01:41.448 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:10.961127+0000", 2022-06-29T16:01:41.448 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:01:41.448 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:01:41.449 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:01:41.449 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:01:41.449 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:01:41.449 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:01:41.449 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:01:41.450 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:01:41.450 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 1, 2022-06-29T16:01:41.450 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 1, 2022-06-29T16:01:41.450 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 1, 2022-06-29T16:01:41.450 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 1, 2022-06-29T16:01:41.451 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 1, 2022-06-29T16:01:41.451 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:01:41.451 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:01:41.451 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:01:41.452 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:01:41.452 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:01:41.452 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:01:41.453 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:01:41.453 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.453 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "15", 2022-06-29T16:01:41.454 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:01:41.454 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:01:41.455 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:01:41.455 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:01:41.456 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:01:41.456 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:01:41.456 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:01:41.457 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:01:41.457 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:01:41.458 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:01:41.458 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:01:41.458 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:01:41.459 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:01:41.459 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:01:41.459 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:01:41.459 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:01:41.460 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:01:41.460 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:01:41.460 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:01:41.460 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:01:41.460 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:01:41.461 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:01:41.461 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:01:41.461 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:01:41.461 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:01:41.462 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:01:41.462 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.462 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:01:41.462 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:01:41.462 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:01:41.463 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:01:41.463 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:01:41.463 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:01:41.463 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:01:41.464 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:01:41.464 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:01:41.464 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:01:41.464 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:01:41.464 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:01:41.465 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 1 2022-06-29T16:01:41.465 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.465 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:01:41.465 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_devicehealth": {} 2022-06-29T16:01:41.466 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.466 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.466 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.466 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 2, 2022-06-29T16:01:41.466 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "rbd", 2022-06-29T16:01:41.467 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:19.184276+0000", 2022-06-29T16:01:41.467 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:01:41.467 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:01:41.467 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:01:41.468 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:01:41.468 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:01:41.468 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:01:41.468 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:01:41.468 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:01:41.469 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:01:41.469 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:01:41.469 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:01:41.469 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:01:41.469 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:01:41.470 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:01:41.470 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:01:41.470 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:01:41.470 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:01:41.471 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:01:41.471 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:01:41.471 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:01:41.471 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.471 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "18", 2022-06-29T16:01:41.472 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:01:41.472 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:01:41.472 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:01:41.472 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:01:41.472 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:01:41.473 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:01:41.473 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:01:41.473 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:01:41.473 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:01:41.474 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:01:41.474 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:01:41.474 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:01:41.474 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:01:41.474 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:01:41.475 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:01:41.475 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:01:41.475 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:01:41.475 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:01:41.475 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:01:41.476 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:01:41.476 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:01:41.476 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:01:41.476 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:01:41.477 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:01:41.477 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:01:41.477 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:01:41.477 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.477 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:01:41.478 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:01:41.478 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:01:41.478 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:01:41.478 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:01:41.478 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:01:41.479 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:01:41.479 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:01:41.479 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:01:41.479 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:01:41.480 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:01:41.480 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:01:41.480 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:01:41.480 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd": {} 2022-06-29T16:01:41.480 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.481 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.481 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.481 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 7, 2022-06-29T16:01:41.481 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "cephfs_metadata", 2022-06-29T16:01:41.481 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T16:00:18.552380+0000", 2022-06-29T16:01:41.482 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:01:41.482 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:01:41.482 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:01:41.482 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:01:41.483 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:01:41.483 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:01:41.483 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:01:41.483 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:01:41.483 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:01:41.484 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:01:41.484 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:01:41.484 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:01:41.484 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:01:41.484 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:01:41.485 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:01:41.485 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:01:41.485 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:01:41.485 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:01:41.486 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:01:41.486 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:01:41.486 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.486 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "42", 2022-06-29T16:01:41.486 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:01:41.487 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:01:41.487 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:01:41.487 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:01:41.487 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:01:41.488 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:01:41.488 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:01:41.488 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:01:41.488 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:01:41.488 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:01:41.489 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:01:41.489 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:01:41.489 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:01:41.489 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:01:41.490 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:01:41.490 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:01:41.490 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:01:41.490 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:01:41.491 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:01:41.491 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:01:41.491 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:01:41.491 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:01:41.492 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:01:41.492 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:01:41.492 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:01:41.492 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:01:41.492 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.493 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:01:41.493 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:01:41.493 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:01:41.493 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:01:41.494 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:01:41.494 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:01:41.494 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:01:41.494 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:01:41.495 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:01:41.495 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:01:41.495 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:01:41.495 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:01:41.496 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_bias": 4, 2022-06-29T16:01:41.496 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 16, 2022-06-29T16:01:41.496 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_priority": 5 2022-06-29T16:01:41.496 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.497 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:01:41.497 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs": { 2022-06-29T16:01:41.497 INFO:teuthology.orchestra.run.smithi060.stdout: "metadata": "cephfs" 2022-06-29T16:01:41.497 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.498 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.498 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.498 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.498 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 8, 2022-06-29T16:01:41.499 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "cephfs_data", 2022-06-29T16:01:41.499 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T16:00:19.688558+0000", 2022-06-29T16:01:41.499 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:01:41.499 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:01:41.500 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:01:41.500 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:01:41.500 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:01:41.500 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:01:41.501 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:01:41.501 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:01:41.501 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:01:41.501 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:01:41.502 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:01:41.502 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:01:41.502 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:01:41.502 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:01:41.503 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:01:41.503 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:01:41.503 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:01:41.503 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:01:41.504 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:01:41.504 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:01:41.504 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.504 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "43", 2022-06-29T16:01:41.505 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:01:41.505 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:01:41.505 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:01:41.505 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:01:41.505 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:01:41.506 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:01:41.506 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:01:41.506 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:01:41.507 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:01:41.507 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:01:41.507 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:01:41.507 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:01:41.508 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:01:41.508 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:01:41.508 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:01:41.509 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:01:41.509 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:01:41.509 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:01:41.510 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:01:41.510 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:01:41.510 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:01:41.510 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:01:41.511 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:01:41.511 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:01:41.511 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:01:41.511 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:01:41.511 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.512 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:01:41.512 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:01:41.512 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:01:41.512 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:01:41.513 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:01:41.513 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:01:41.513 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:01:41.513 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:01:41.513 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:01:41.514 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:01:41.514 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:01:41.514 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:01:41.515 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:01:41.515 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs": { 2022-06-29T16:01:41.515 INFO:teuthology.orchestra.run.smithi060.stdout: "data": "cephfs" 2022-06-29T16:01:41.515 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.516 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.516 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.516 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:01:41.516 INFO:teuthology.orchestra.run.smithi060.stdout: "osds": [ 2022-06-29T16:01:41.517 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.517 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:01:41.517 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "11e7ab9e-fce2-470f-8195-ea4c3c52df22", 2022-06-29T16:01:41.517 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:01:41.518 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:01:41.518 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:01:41.518 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:01:41.518 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:01:41.519 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:01:41.519 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:01:41.519 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 40, 2022-06-29T16:01:41.519 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:01:41.519 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:01:41.520 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:01:41.520 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:41.520 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.520 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:41.521 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6816", 2022-06-29T16:01:41.521 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:01:41.521 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.521 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.521 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:41.522 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6825", 2022-06-29T16:01:41.522 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:01:41.522 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.522 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.523 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.523 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:01:41.523 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:41.523 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.523 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:41.524 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6826", 2022-06-29T16:01:41.524 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:01:41.524 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.524 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.525 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:41.525 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6827", 2022-06-29T16:01:41.525 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:01:41.525 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.525 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.526 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.526 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:01:41.526 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:41.526 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.526 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:41.527 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6830", 2022-06-29T16:01:41.527 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:01:41.527 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.527 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.528 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:41.528 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6831", 2022-06-29T16:01:41.528 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:01:41.528 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.528 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.529 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.529 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:01:41.529 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:41.529 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.529 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:41.530 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6828", 2022-06-29T16:01:41.530 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:01:41.530 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.530 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.531 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:41.531 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6829", 2022-06-29T16:01:41.531 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:01:41.531 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.531 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.532 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.532 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6825/12890", 2022-06-29T16:01:41.532 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6827/12890", 2022-06-29T16:01:41.532 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6831/12890", 2022-06-29T16:01:41.533 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6829/12890", 2022-06-29T16:01:41.533 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:01:41.533 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:01:41.533 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:01:41.533 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.534 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.534 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.534 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:01:41.534 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "ca43f136-6f69-4431-a157-fbe073fb585c", 2022-06-29T16:01:41.535 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:01:41.535 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:01:41.535 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:01:41.535 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:01:41.535 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:01:41.536 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:01:41.536 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:01:41.536 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 41, 2022-06-29T16:01:41.536 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:01:41.537 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:01:41.537 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:01:41.537 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:41.537 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.538 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:41.538 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6815", 2022-06-29T16:01:41.538 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:01:41.538 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.539 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.539 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:41.539 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6818", 2022-06-29T16:01:41.539 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:01:41.539 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.540 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.540 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.540 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:01:41.540 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:41.541 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.541 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:41.541 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6819", 2022-06-29T16:01:41.541 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:01:41.541 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.542 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.542 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:41.542 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6820", 2022-06-29T16:01:41.542 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:01:41.542 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.543 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.543 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.543 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:01:41.543 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:41.544 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.544 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:41.544 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6823", 2022-06-29T16:01:41.544 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:01:41.544 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.545 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.545 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:41.545 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6824", 2022-06-29T16:01:41.545 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:01:41.545 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.546 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.546 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.546 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:01:41.546 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:41.547 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.547 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:41.547 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6821", 2022-06-29T16:01:41.547 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:01:41.547 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.548 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.548 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:41.548 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6822", 2022-06-29T16:01:41.548 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:01:41.549 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.549 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.549 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.549 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6818/12892", 2022-06-29T16:01:41.549 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6820/12892", 2022-06-29T16:01:41.550 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6824/12892", 2022-06-29T16:01:41.550 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6822/12892", 2022-06-29T16:01:41.550 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:01:41.550 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:01:41.550 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:01:41.551 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.551 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.551 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.551 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:01:41.552 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "74fac45f-5f07-44ec-a9cb-2644c123fd4a", 2022-06-29T16:01:41.552 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:01:41.552 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:01:41.552 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:01:41.552 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:01:41.553 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:01:41.553 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:01:41.553 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:01:41.553 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 29, 2022-06-29T16:01:41.554 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:01:41.554 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:01:41.554 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:01:41.554 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:41.554 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.555 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:41.555 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6808", 2022-06-29T16:01:41.555 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:01:41.555 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.556 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.556 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:41.556 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6809", 2022-06-29T16:01:41.556 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:01:41.557 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.557 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.557 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.557 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:01:41.558 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:41.558 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.558 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:41.558 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6810", 2022-06-29T16:01:41.559 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:01:41.559 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.559 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.559 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:41.560 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6811", 2022-06-29T16:01:41.560 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:01:41.560 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.561 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.561 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.561 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:01:41.561 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:41.562 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.562 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:41.562 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6814", 2022-06-29T16:01:41.562 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:01:41.563 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.563 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.563 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:41.563 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6817", 2022-06-29T16:01:41.564 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:01:41.564 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.564 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.564 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.565 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:01:41.565 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:41.565 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.565 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:41.566 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6812", 2022-06-29T16:01:41.566 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:01:41.566 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.566 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.566 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:41.567 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6813", 2022-06-29T16:01:41.567 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:01:41.567 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.567 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.567 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.568 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6809/12891", 2022-06-29T16:01:41.568 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6811/12891", 2022-06-29T16:01:41.568 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6817/12891", 2022-06-29T16:01:41.568 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6813/12891", 2022-06-29T16:01:41.569 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:01:41.569 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:01:41.569 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:01:41.569 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.569 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.570 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.570 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:01:41.570 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "2581d351-973a-4b4e-85ca-3658f0068765", 2022-06-29T16:01:41.570 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:01:41.571 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:01:41.571 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:01:41.571 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:01:41.571 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:01:41.571 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:01:41.572 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:01:41.572 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 40, 2022-06-29T16:01:41.572 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:01:41.572 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:01:41.572 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:01:41.573 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:41.573 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.573 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:41.573 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6800", 2022-06-29T16:01:41.574 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:01:41.574 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.574 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.575 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:41.575 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6801", 2022-06-29T16:01:41.575 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:01:41.575 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.576 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.576 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.576 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:01:41.576 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:41.576 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.577 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:41.577 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6802", 2022-06-29T16:01:41.577 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:01:41.577 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.578 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.578 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:41.578 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6803", 2022-06-29T16:01:41.578 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:01:41.578 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.579 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.579 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.579 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:01:41.579 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:41.580 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.580 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:41.580 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6806", 2022-06-29T16:01:41.580 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:01:41.580 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.581 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.581 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:41.581 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6807", 2022-06-29T16:01:41.581 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:01:41.581 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.582 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.582 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.582 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:01:41.582 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:41.583 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.583 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:41.583 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6804", 2022-06-29T16:01:41.583 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:01:41.583 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.584 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.584 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:41.584 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6805", 2022-06-29T16:01:41.584 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:01:41.584 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.585 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.585 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.585 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6801/12889", 2022-06-29T16:01:41.585 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6803/12889", 2022-06-29T16:01:41.586 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6807/12889", 2022-06-29T16:01:41.586 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6805/12889", 2022-06-29T16:01:41.586 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:01:41.586 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:01:41.586 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:01:41.587 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.587 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.587 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.587 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:01:41.587 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "3d234db9-87cd-4fe9-814f-0240a287d8b4", 2022-06-29T16:01:41.588 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:01:41.588 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:01:41.588 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:01:41.588 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:01:41.589 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:01:41.589 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:01:41.589 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:01:41.589 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 40, 2022-06-29T16:01:41.589 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:01:41.590 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:01:41.590 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:01:41.590 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:41.590 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.590 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:41.591 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6808", 2022-06-29T16:01:41.591 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:01:41.591 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.591 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.592 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:41.592 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6810", 2022-06-29T16:01:41.592 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:01:41.592 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.592 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.593 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.593 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:01:41.593 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:41.593 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.593 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:41.594 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6812", 2022-06-29T16:01:41.594 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:01:41.594 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.594 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.595 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:41.595 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6814", 2022-06-29T16:01:41.595 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:01:41.595 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.595 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.596 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.596 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:01:41.596 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:41.596 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.597 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:41.597 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6823", 2022-06-29T16:01:41.597 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:01:41.597 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.597 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.598 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:41.598 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6825", 2022-06-29T16:01:41.598 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:01:41.598 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.598 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.599 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.599 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:01:41.599 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:41.599 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.600 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:41.600 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6817", 2022-06-29T16:01:41.600 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:01:41.600 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.600 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.601 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:41.601 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6820", 2022-06-29T16:01:41.601 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:01:41.601 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.601 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.602 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.602 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6810/13249", 2022-06-29T16:01:41.602 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6814/13249", 2022-06-29T16:01:41.602 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6825/13249", 2022-06-29T16:01:41.603 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6820/13249", 2022-06-29T16:01:41.603 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:01:41.603 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:01:41.603 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:01:41.603 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.604 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.604 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.604 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:01:41.604 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "e4bf1104-ff5c-438b-b608-8cee31c163f6", 2022-06-29T16:01:41.604 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:01:41.605 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:01:41.605 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:01:41.605 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:01:41.605 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:01:41.606 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:01:41.606 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:01:41.606 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 40, 2022-06-29T16:01:41.606 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:01:41.606 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:01:41.607 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:01:41.607 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:41.607 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.607 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:41.607 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6809", 2022-06-29T16:01:41.608 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:01:41.608 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.608 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.608 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:41.609 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6811", 2022-06-29T16:01:41.609 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:01:41.609 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.609 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.609 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.610 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:01:41.610 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:41.610 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.610 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:41.610 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6813", 2022-06-29T16:01:41.611 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:01:41.611 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.611 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.611 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:41.611 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6815", 2022-06-29T16:01:41.612 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:01:41.612 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.612 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.612 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.613 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:01:41.613 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:41.613 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.613 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:41.613 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6824", 2022-06-29T16:01:41.614 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:01:41.614 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.614 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.614 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:41.614 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6827", 2022-06-29T16:01:41.615 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:01:41.615 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.615 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.615 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.616 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:01:41.616 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:41.616 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.616 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:41.616 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6818", 2022-06-29T16:01:41.617 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:01:41.617 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.617 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.617 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:41.617 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6821", 2022-06-29T16:01:41.618 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:01:41.618 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.618 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.618 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.619 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6811/13248", 2022-06-29T16:01:41.619 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6815/13248", 2022-06-29T16:01:41.619 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6827/13248", 2022-06-29T16:01:41.619 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6821/13248", 2022-06-29T16:01:41.619 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:01:41.620 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:01:41.620 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:01:41.620 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.620 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.620 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.621 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:01:41.621 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "8564dd25-27ab-4203-b54e-af6d3aa05857", 2022-06-29T16:01:41.621 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:01:41.621 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:01:41.622 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:01:41.622 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:01:41.622 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:01:41.622 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:01:41.622 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:01:41.623 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 40, 2022-06-29T16:01:41.623 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:01:41.623 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:01:41.623 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:01:41.623 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:41.624 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.624 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:41.624 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6816", 2022-06-29T16:01:41.624 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:01:41.624 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.625 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.625 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:41.625 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6819", 2022-06-29T16:01:41.625 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:01:41.626 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.626 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.626 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.626 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:01:41.626 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:41.627 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.627 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:41.627 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6822", 2022-06-29T16:01:41.627 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:01:41.627 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.628 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.628 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:41.628 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6826", 2022-06-29T16:01:41.628 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:01:41.629 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.629 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.629 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.629 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:01:41.629 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:41.630 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.630 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:41.630 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6830", 2022-06-29T16:01:41.630 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:01:41.631 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.631 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.631 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:41.631 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6831", 2022-06-29T16:01:41.631 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:01:41.632 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.632 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.632 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.632 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:01:41.632 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:41.633 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.633 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:41.633 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6828", 2022-06-29T16:01:41.633 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:01:41.634 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.634 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.634 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:41.634 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6829", 2022-06-29T16:01:41.634 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:01:41.635 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.635 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.635 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.635 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6819/13250", 2022-06-29T16:01:41.635 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6826/13250", 2022-06-29T16:01:41.636 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6831/13250", 2022-06-29T16:01:41.636 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6829/13250", 2022-06-29T16:01:41.636 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:01:41.636 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:01:41.637 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:01:41.637 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.637 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.637 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.637 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:01:41.638 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "bd513a9c-3b19-4314-9897-d300b5c897bb", 2022-06-29T16:01:41.638 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:01:41.638 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:01:41.638 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:01:41.638 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:01:41.639 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:01:41.639 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:01:41.639 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:01:41.639 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 39, 2022-06-29T16:01:41.640 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:01:41.640 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:01:41.640 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:01:41.640 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:41.640 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.641 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:41.641 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6800", 2022-06-29T16:01:41.641 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:01:41.641 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.641 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.642 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:41.642 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6801", 2022-06-29T16:01:41.642 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:01:41.642 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.643 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.643 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.643 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:01:41.643 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:41.643 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.644 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:41.644 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6802", 2022-06-29T16:01:41.644 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:01:41.644 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.644 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.645 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:41.645 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6803", 2022-06-29T16:01:41.645 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:01:41.645 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.646 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.646 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.646 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:01:41.646 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:41.646 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.647 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:41.647 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6806", 2022-06-29T16:01:41.647 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:01:41.647 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.648 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.648 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:41.648 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6807", 2022-06-29T16:01:41.648 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:01:41.648 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.649 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.649 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.649 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:01:41.649 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:41.649 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.650 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:41.650 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6804", 2022-06-29T16:01:41.650 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:01:41.650 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.651 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.651 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:41.651 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6805", 2022-06-29T16:01:41.651 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:01:41.651 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.652 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.652 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.652 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6801/13247", 2022-06-29T16:01:41.652 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6803/13247", 2022-06-29T16:01:41.652 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6807/13247", 2022-06-29T16:01:41.653 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6805/13247", 2022-06-29T16:01:41.653 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:01:41.653 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:01:41.653 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:01:41.653 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:41.654 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.654 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:01:41.654 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_xinfo": [ 2022-06-29T16:01:41.654 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.655 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:01:41.655 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:01:41.655 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:01:41.655 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:01:41.655 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:01:41.656 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:01:41.656 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.891342+0000", 2022-06-29T16:01:41.656 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:01:41.656 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.656 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.657 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:01:41.657 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:01:41.657 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:01:41.657 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:01:41.658 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:01:41.658 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:01:41.658 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.900866+0000", 2022-06-29T16:01:41.658 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:01:41.658 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.659 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.659 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:01:41.659 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:01:41.659 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:01:41.660 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:01:41.660 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:01:41.660 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:01:41.660 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.912755+0000", 2022-06-29T16:01:41.660 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:01:41.661 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.661 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.661 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:01:41.661 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:01:41.661 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:01:41.662 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:01:41.662 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:01:41.662 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:01:41.662 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.864332+0000", 2022-06-29T16:01:41.662 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:01:41.663 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.663 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.663 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:01:41.663 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:01:41.664 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:01:41.664 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:01:41.664 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:01:41.664 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:01:41.664 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.863756+0000", 2022-06-29T16:01:41.665 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:01:41.665 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.665 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.665 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:01:41.666 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:01:41.666 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:01:41.666 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:01:41.666 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:01:41.666 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:01:41.667 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.869041+0000", 2022-06-29T16:01:41.667 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:01:41.667 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.667 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.668 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:01:41.668 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:01:41.668 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:01:41.668 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:01:41.668 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:01:41.669 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:01:41.669 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.895043+0000", 2022-06-29T16:01:41.669 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:01:41.669 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.670 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:41.670 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:01:41.670 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:01:41.670 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:01:41.670 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:01:41.671 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:01:41.671 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:01:41.671 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.889491+0000", 2022-06-29T16:01:41.671 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:01:41.671 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.672 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:01:41.672 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap": [], 2022-06-29T16:01:41.672 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap_items": [], 2022-06-29T16:01:41.672 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_temp": [], 2022-06-29T16:01:41.672 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_temp": [], 2022-06-29T16:01:41.673 INFO:teuthology.orchestra.run.smithi060.stdout: "blacklist": {}, 2022-06-29T16:01:41.673 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profiles": { 2022-06-29T16:01:41.673 INFO:teuthology.orchestra.run.smithi060.stdout: "default": { 2022-06-29T16:01:41.673 INFO:teuthology.orchestra.run.smithi060.stdout: "crush-failure-domain": "osd", 2022-06-29T16:01:41.674 INFO:teuthology.orchestra.run.smithi060.stdout: "k": "2", 2022-06-29T16:01:41.674 INFO:teuthology.orchestra.run.smithi060.stdout: "m": "1", 2022-06-29T16:01:41.674 INFO:teuthology.orchestra.run.smithi060.stdout: "plugin": "jerasure", 2022-06-29T16:01:41.674 INFO:teuthology.orchestra.run.smithi060.stdout: "ruleset-failure-domain": "osd", 2022-06-29T16:01:41.674 INFO:teuthology.orchestra.run.smithi060.stdout: "technique": "reed_sol_van" 2022-06-29T16:01:41.675 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:41.675 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:41.675 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps_queue": [], 2022-06-29T16:01:41.675 INFO:teuthology.orchestra.run.smithi060.stdout: "new_removed_snaps": [], 2022-06-29T16:01:41.675 INFO:teuthology.orchestra.run.smithi060.stdout: "new_purged_snaps": [], 2022-06-29T16:01:41.676 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_node_flags": {}, 2022-06-29T16:01:41.676 INFO:teuthology.orchestra.run.smithi060.stdout: "device_class_flags": {} 2022-06-29T16:01:41.676 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:01:41.678 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.435+0000 7f78e1d0c700 1 -- 172.21.15.60:0/3349230093 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f78c0053a10 msgr2=0x7f78c0055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:41.678 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.435+0000 7f78e1d0c700 1 --2- 172.21.15.60:0/3349230093 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f78c0053a10 0x7f78c0055ed0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f78c401b000 tx=0x7f78c4016000).stop 2022-06-29T16:01:41.679 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.435+0000 7f78e1d0c700 1 -- 172.21.15.60:0/3349230093 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78dc0aebc0 msgr2=0x7f78dc139210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:41.679 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.435+0000 7f78e1d0c700 1 --2- 172.21.15.60:0/3349230093 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78dc0aebc0 0x7f78dc139210 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f78cc000ed0 tx=0x7f78cc007ac0).stop 2022-06-29T16:01:41.679 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.435+0000 7f78e1d0c700 1 -- 172.21.15.60:0/3349230093 shutdown_connections 2022-06-29T16:01:41.679 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.435+0000 7f78e1d0c700 1 --2- 172.21.15.60:0/3349230093 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f78c0053a10 0x7f78c0055ed0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:41.680 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.435+0000 7f78e1d0c700 1 --2- 172.21.15.60:0/3349230093 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78dc13fa90 0x7f78dc1384c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:41.680 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.435+0000 7f78e1d0c700 1 --2- 172.21.15.60:0/3349230093 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78dc0b1080 0x7f78dc0b1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:41.680 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.435+0000 7f78e1d0c700 1 --2- 172.21.15.60:0/3349230093 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78dc0aebc0 0x7f78dc139210 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:41.680 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.435+0000 7f78e1d0c700 1 -- 172.21.15.60:0/3349230093 >> 172.21.15.60:0/3349230093 conn(0x7f78dc0636f0 msgr2=0x7f78dc05bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:41.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.435+0000 7f78e1d0c700 1 -- 172.21.15.60:0/3349230093 shutdown_connections 2022-06-29T16:01:41.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.435+0000 7f78e1d0c700 1 -- 172.21.15.60:0/3349230093 wait complete. 2022-06-29T16:01:41.682 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:01:41.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.783+0000 7fc2408c7700 1 Processor -- start 2022-06-29T16:01:41.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.783+0000 7fc2408c7700 1 -- start start 2022-06-29T16:01:41.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.787+0000 7fc2408c7700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc23c05d1f0 0x7fc23c0b1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:41.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.787+0000 7fc2408c7700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc23c0b3cc0 0x7fc23c0b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:41.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.787+0000 7fc2408c7700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc23c0b2ce0 con 0x7fc23c0b9d70 2022-06-29T16:01:41.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.787+0000 7fc2408c7700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc23c0b4d10 con 0x7fc23c0b3cc0 2022-06-29T16:01:41.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.787+0000 7fc2408c7700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c0ba150 con 0x7fc23c05d1f0 2022-06-29T16:01:41.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.787+0000 7fc239d9b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc23c05d1f0 0x7fc23c0b1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:41.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.787+0000 7fc23ad9d700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc23c0b3cc0 0x7fc23c0b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:41.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.787+0000 7fc239d9b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc23c05d1f0 0x7fc23c0b1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:60596/0 (socket says 172.21.15.60:60596) 2022-06-29T16:01:41.797 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.787+0000 7fc239d9b700 1 -- 172.21.15.60:0/3588217511 learned_addr learned my addr 172.21.15.60:0/3588217511 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:41.797 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.787+0000 7fc239d9b700 1 -- 172.21.15.60:0/3588217511 >> v2:172.21.15.92:3300/0 conn(0x7fc23c0b3cc0 msgr2=0x7fc23c0b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:01:41.797 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.787+0000 7fc239d9b700 1 --2- 172.21.15.60:0/3588217511 >> v2:172.21.15.92:3300/0 conn(0x7fc23c0b3cc0 0x7fc23c0b2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:41.797 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.787+0000 7fc239d9b700 1 -- 172.21.15.60:0/3588217511 >> v1:172.21.15.92:6789/0 conn(0x7fc23c0b9d70 legacy=0x7fc23c0b3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:41.798 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.787+0000 7fc239d9b700 1 -- 172.21.15.60:0/3588217511 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc23c0b4a70 con 0x7fc23c05d1f0 2022-06-29T16:01:41.798 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.787+0000 7fc239d9b700 1 --2- 172.21.15.60:0/3588217511 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc23c05d1f0 0x7fc23c0b1b50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fc22c002f70 tx=0x7fc22c006eb0).ready entity=mon.2 client_cookie=5f61d6c2a330cfc server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:41.798 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.787+0000 7fc23959a700 1 -- 172.21.15.60:0/3588217511 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc22c00d570 con 0x7fc23c05d1f0 2022-06-29T16:01:41.798 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.787+0000 7fc23959a700 1 -- 172.21.15.60:0/3588217511 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc22c000e90 con 0x7fc23c05d1f0 2022-06-29T16:01:41.799 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.787+0000 7fc23959a700 1 -- 172.21.15.60:0/3588217511 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc22c00d900 con 0x7fc23c05d1f0 2022-06-29T16:01:41.799 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.787+0000 7fc2408c7700 1 -- 172.21.15.60:0/3588217511 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc23c05d1f0 msgr2=0x7fc23c0b1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:41.799 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.787+0000 7fc2408c7700 1 --2- 172.21.15.60:0/3588217511 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc23c05d1f0 0x7fc23c0b1b50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fc22c002f70 tx=0x7fc22c006eb0).stop 2022-06-29T16:01:41.800 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.787+0000 7fc2408c7700 1 -- 172.21.15.60:0/3588217511 shutdown_connections 2022-06-29T16:01:41.800 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.787+0000 7fc2408c7700 1 --2- 172.21.15.60:0/3588217511 >> v2:172.21.15.92:3300/0 conn(0x7fc23c0b3cc0 0x7fc23c0b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:41.800 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.787+0000 7fc2408c7700 1 --2- 172.21.15.60:0/3588217511 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc23c05d1f0 0x7fc23c0b1b50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:41.800 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.787+0000 7fc2408c7700 1 -- 172.21.15.60:0/3588217511 >> 172.21.15.60:0/3588217511 conn(0x7fc23c062ce0 msgr2=0x7fc23c0b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:41.800 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.787+0000 7fc2408c7700 1 -- 172.21.15.60:0/3588217511 shutdown_connections 2022-06-29T16:01:41.801 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.787+0000 7fc2408c7700 1 -- 172.21.15.60:0/3588217511 wait complete. 2022-06-29T16:01:41.803 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.791+0000 7fc2408c7700 1 Processor -- start 2022-06-29T16:01:41.803 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.791+0000 7fc2408c7700 1 -- start start 2022-06-29T16:01:41.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.791+0000 7fc2408c7700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc23c05d1f0 0x7fc23c0b1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:41.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.791+0000 7fc2408c7700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc23c0b3cc0 0x7fc23c13c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:41.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.791+0000 7fc2408c7700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc23c0b9d70 0x7fc23c13cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:41.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.791+0000 7fc2408c7700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c0aced0 con 0x7fc23c0b3cc0 2022-06-29T16:01:41.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.791+0000 7fc2408c7700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c13d190 con 0x7fc23c05d1f0 2022-06-29T16:01:41.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.791+0000 7fc2408c7700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c0ad100 con 0x7fc23c0b9d70 2022-06-29T16:01:41.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.791+0000 7fc239d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc23c0b3cc0 0x7fc23c13c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:41.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.791+0000 7fc23ad9d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc23c0b9d70 0x7fc23c13cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:41.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.791+0000 7fc23a59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc23c05d1f0 0x7fc23c0b1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:41.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.791+0000 7fc23ad9d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc23c0b9d70 0x7fc23c13cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:60602/0 (socket says 172.21.15.60:60602) 2022-06-29T16:01:41.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.791+0000 7fc23ad9d700 1 -- 172.21.15.60:0/463379584 learned_addr learned my addr 172.21.15.60:0/463379584 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:41.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.791+0000 7fc23a59c700 1 -- 172.21.15.60:0/463379584 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc23c0b9d70 msgr2=0x7fc23c13cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:41.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.791+0000 7fc23a59c700 1 --2- 172.21.15.60:0/463379584 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc23c0b9d70 0x7fc23c13cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:41.807 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.791+0000 7fc23a59c700 1 -- 172.21.15.60:0/463379584 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc23c0b3cc0 msgr2=0x7fc23c13c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:41.807 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.791+0000 7fc23a59c700 1 --2- 172.21.15.60:0/463379584 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc23c0b3cc0 0x7fc23c13c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:41.807 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.791+0000 7fc23a59c700 1 -- 172.21.15.60:0/463379584 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc22c00b1e0 con 0x7fc23c05d1f0 2022-06-29T16:01:41.807 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.791+0000 7fc23a59c700 1 --2- 172.21.15.60:0/463379584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc23c05d1f0 0x7fc23c0b1ba0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fc23c060000 tx=0x7fc224007b90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:41.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.791+0000 7fc2337fe700 1 -- 172.21.15.60:0/463379584 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc22400ccd0 con 0x7fc23c05d1f0 2022-06-29T16:01:41.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.791+0000 7fc2337fe700 1 -- 172.21.15.60:0/463379584 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc22400ce30 con 0x7fc23c05d1f0 2022-06-29T16:01:41.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.791+0000 7fc2408c7700 1 -- 172.21.15.60:0/463379584 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc23c0ad3a0 con 0x7fc23c05d1f0 2022-06-29T16:01:41.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.791+0000 7fc2337fe700 1 -- 172.21.15.60:0/463379584 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2240125f0 con 0x7fc23c05d1f0 2022-06-29T16:01:41.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.791+0000 7fc2408c7700 1 -- 172.21.15.60:0/463379584 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc23c0b24e0 con 0x7fc23c05d1f0 2022-06-29T16:01:41.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.795+0000 7fc2408c7700 1 -- 172.21.15.60:0/463379584 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc23c0ad530 con 0x7fc23c05d1f0 2022-06-29T16:01:41.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.795+0000 7fc2337fe700 1 -- 172.21.15.60:0/463379584 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc224019030 con 0x7fc23c05d1f0 2022-06-29T16:01:41.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.795+0000 7fc2337fe700 1 --2- 172.21.15.60:0/463379584 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc220053aa0 0x7fc220055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:41.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.795+0000 7fc2337fe700 1 -- 172.21.15.60:0/463379584 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc22406b170 con 0x7fc23c05d1f0 2022-06-29T16:01:41.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.795+0000 7fc239d9b700 1 --2- 172.21.15.60:0/463379584 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc220053aa0 0x7fc220055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:41.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.799+0000 7fc239d9b700 1 --2- 172.21.15.60:0/463379584 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc220053aa0 0x7fc220055f60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fc22c01c000 tx=0x7fc22c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:41.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.803+0000 7fc2337fe700 1 -- 172.21.15.60:0/463379584 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc224041f80 con 0x7fc23c05d1f0 2022-06-29T16:01:41.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.971+0000 7fc2408c7700 1 -- 172.21.15.60:0/463379584 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc23c05d990 con 0x7fc23c05d1f0 2022-06-29T16:01:41.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.971+0000 7fc2337fe700 1 -- 172.21.15.60:0/463379584 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+2138 (secure 0 0 0) 0x7fc2240455a0 con 0x7fc23c05d1f0 2022-06-29T16:01:41.980 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:01:41.980 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":15,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4937,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/1673184114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1673184114},{"type":"v1","addr":"172.21.15.92:6835","nonce":1673184114}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":13}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:00:20.685676+0000","modified":"2022-06-29T16:00:21.689299+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5308":{"gid":5308,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/1964426267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1964426267},{"type":"v1","addr":"172.21.15.60:6833","nonce":1964426267}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-06-29T16:01:41.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.975+0000 7fc2408c7700 1 -- 172.21.15.60:0/463379584 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc220053aa0 msgr2=0x7fc220055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:41.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.975+0000 7fc2408c7700 1 --2- 172.21.15.60:0/463379584 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc220053aa0 0x7fc220055f60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fc22c01c000 tx=0x7fc22c015000).stop 2022-06-29T16:01:41.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.975+0000 7fc2408c7700 1 -- 172.21.15.60:0/463379584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc23c05d1f0 msgr2=0x7fc23c0b1ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:41.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.975+0000 7fc2408c7700 1 --2- 172.21.15.60:0/463379584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc23c05d1f0 0x7fc23c0b1ba0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fc23c060000 tx=0x7fc224007b90).stop 2022-06-29T16:01:41.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.975+0000 7fc2408c7700 1 -- 172.21.15.60:0/463379584 shutdown_connections 2022-06-29T16:01:41.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.975+0000 7fc2408c7700 1 --2- 172.21.15.60:0/463379584 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc220053aa0 0x7fc220055f60 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:41.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.975+0000 7fc2408c7700 1 --2- 172.21.15.60:0/463379584 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc23c0b9d70 0x7fc23c13cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:41.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.975+0000 7fc2408c7700 1 --2- 172.21.15.60:0/463379584 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc23c0b3cc0 0x7fc23c13c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:41.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.975+0000 7fc2408c7700 1 --2- 172.21.15.60:0/463379584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc23c05d1f0 0x7fc23c0b1ba0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:41.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.975+0000 7fc2408c7700 1 -- 172.21.15.60:0/463379584 >> 172.21.15.60:0/463379584 conn(0x7fc23c062ce0 msgr2=0x7fc23c0b8780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:41.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.975+0000 7fc2408c7700 1 -- 172.21.15.60:0/463379584 shutdown_connections 2022-06-29T16:01:41.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:41.975+0000 7fc2408c7700 1 -- 172.21.15.60:0/463379584 wait complete. 2022-06-29T16:01:41.985 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 15 2022-06-29T16:01:41.993 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-06-29T16:01:42.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.091+0000 7f914843b700 1 Processor -- start 2022-06-29T16:01:42.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.091+0000 7f914843b700 1 -- start start 2022-06-29T16:01:42.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.095+0000 7f914843b700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f91400b9d70 0x7f91400b3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:42.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.095+0000 7f914843b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f91400b3cc0 0x7f91401420b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:42.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.095+0000 7f914843b700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91400b2ce0 con 0x7f914005d1f0 2022-06-29T16:01:42.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.095+0000 7f914843b700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91400b4d10 con 0x7f91400b3cc0 2022-06-29T16:01:42.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.095+0000 7f914843b700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91400ba150 con 0x7f91400b9d70 2022-06-29T16:01:42.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.095+0000 7f91459d6700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f914005d1f0 0x7f91400b1b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55600/0 (socket says 172.21.15.60:55600) 2022-06-29T16:01:42.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.095+0000 7f91459d6700 1 -- 172.21.15.60:0/683256870 learned_addr learned my addr 172.21.15.60:0/683256870 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:01:42.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.095+0000 7f91469d8700 1 --2- 172.21.15.60:0/683256870 >> v2:172.21.15.60:3300/0 conn(0x7f91400b3cc0 0x7f91401420b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:42.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.095+0000 7f91461d7700 1 --2- 172.21.15.60:0/683256870 >> v2:172.21.15.92:3300/0 conn(0x7f91400b9d70 0x7f91400b3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:42.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.095+0000 7f91451d5700 1 -- 172.21.15.60:0/683256870 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3444544751 0 0) 0x7f91400b2ce0 con 0x7f914005d1f0 2022-06-29T16:01:42.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.095+0000 7f91451d5700 1 -- 172.21.15.60:0/683256870 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9128003260 con 0x7f914005d1f0 2022-06-29T16:01:42.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.095+0000 7f91469d8700 1 -- 172.21.15.60:0/683256870 >> v2:172.21.15.92:3300/0 conn(0x7f91400b9d70 msgr2=0x7f91400b3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:01:42.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.095+0000 7f91469d8700 1 --2- 172.21.15.60:0/683256870 >> v2:172.21.15.92:3300/0 conn(0x7f91400b9d70 0x7f91400b3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:42.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.095+0000 7f91469d8700 1 -- 172.21.15.60:0/683256870 >> v1:172.21.15.60:6789/0 conn(0x7f914005d1f0 legacy=0x7f91400b1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:42.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.095+0000 7f91451d5700 1 -- 172.21.15.60:0/683256870 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3076124941 0 0) 0x7f9128003260 con 0x7f914005d1f0 2022-06-29T16:01:42.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.095+0000 7f91469d8700 1 -- 172.21.15.60:0/683256870 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91400b4a70 con 0x7f91400b3cc0 2022-06-29T16:01:42.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.095+0000 7f91469d8700 1 --2- 172.21.15.60:0/683256870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91400b3cc0 0x7f91401420b0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f9130001800 tx=0x7f91380091d0).ready entity=mon.0 client_cookie=bc9d9f5eae207d20 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:42.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.095+0000 7f91451d5700 1 -- 172.21.15.60:0/683256870 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9130001660 con 0x7f91400b3cc0 2022-06-29T16:01:42.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.095+0000 7f91451d5700 1 -- 172.21.15.60:0/683256870 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f913800f510 con 0x7f91400b3cc0 2022-06-29T16:01:42.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.095+0000 7f91451d5700 1 -- 172.21.15.60:0/683256870 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f913800f680 con 0x7f91400b3cc0 2022-06-29T16:01:42.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.095+0000 7f914843b700 1 -- 172.21.15.60:0/683256870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91400b3cc0 msgr2=0x7f91401420b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:42.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.095+0000 7f914843b700 1 --2- 172.21.15.60:0/683256870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91400b3cc0 0x7f91401420b0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f9130001800 tx=0x7f91380091d0).stop 2022-06-29T16:01:42.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.095+0000 7f914843b700 1 -- 172.21.15.60:0/683256870 shutdown_connections 2022-06-29T16:01:42.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.095+0000 7f914843b700 1 --2- 172.21.15.60:0/683256870 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91400b3cc0 0x7f91401420b0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:42.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.095+0000 7f914843b700 1 --2- 172.21.15.60:0/683256870 >> v2:172.21.15.92:3300/0 conn(0x7f91400b9d70 0x7f91400b3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:42.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.095+0000 7f914843b700 1 -- 172.21.15.60:0/683256870 >> 172.21.15.60:0/683256870 conn(0x7f9140062ce0 msgr2=0x7f91400b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:42.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.095+0000 7f914843b700 1 -- 172.21.15.60:0/683256870 shutdown_connections 2022-06-29T16:01:42.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.095+0000 7f914843b700 1 -- 172.21.15.60:0/683256870 wait complete. 2022-06-29T16:01:42.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.099+0000 7f914843b700 1 Processor -- start 2022-06-29T16:01:42.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.099+0000 7f914843b700 1 -- start start 2022-06-29T16:01:42.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.099+0000 7f914843b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f914005d1f0 0x7f91400aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:42.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.099+0000 7f914843b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91400b3cc0 0x7f914013c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:42.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.099+0000 7f914843b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91400b9d70 0x7f91401399b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:42.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.099+0000 7f914843b700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91401417a0 con 0x7f91400b3cc0 2022-06-29T16:01:42.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.099+0000 7f914843b700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9140155d50 con 0x7f91400b9d70 2022-06-29T16:01:42.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.099+0000 7f914843b700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91400af650 con 0x7f914005d1f0 2022-06-29T16:01:42.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.099+0000 7f91459d6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91400b3cc0 0x7f914013c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:42.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.099+0000 7f91459d6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91400b3cc0 0x7f914013c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47522/0 (socket says 172.21.15.60:47522) 2022-06-29T16:01:42.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.099+0000 7f91459d6700 1 -- 172.21.15.60:0/3956237649 learned_addr learned my addr 172.21.15.60:0/3956237649 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:42.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.099+0000 7f91461d7700 1 --2- 172.21.15.60:0/3956237649 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f914005d1f0 0x7f91400aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:42.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.099+0000 7f91469d8700 1 --2- 172.21.15.60:0/3956237649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91400b9d70 0x7f91401399b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:42.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.099+0000 7f91459d6700 1 -- 172.21.15.60:0/3956237649 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f914005d1f0 msgr2=0x7f91400aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:42.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.099+0000 7f91459d6700 1 --2- 172.21.15.60:0/3956237649 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f914005d1f0 0x7f91400aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:42.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.099+0000 7f91459d6700 1 -- 172.21.15.60:0/3956237649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91400b9d70 msgr2=0x7f91401399b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:42.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.099+0000 7f91459d6700 1 --2- 172.21.15.60:0/3956237649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91400b9d70 0x7f91401399b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:42.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.099+0000 7f91459d6700 1 -- 172.21.15.60:0/3956237649 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9138009470 con 0x7f91400b3cc0 2022-06-29T16:01:42.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.099+0000 7f91469d8700 1 --2- 172.21.15.60:0/3956237649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91400b9d70 0x7f91401399b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:01:42.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.099+0000 7f91459d6700 1 --2- 172.21.15.60:0/3956237649 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91400b3cc0 0x7f914013c7e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f91300018c0 tx=0x7f9130001b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:42.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.099+0000 7f912f7fe700 1 -- 172.21.15.60:0/3956237649 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9130007170 con 0x7f91400b3cc0 2022-06-29T16:01:42.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.099+0000 7f912f7fe700 1 -- 172.21.15.60:0/3956237649 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91300072d0 con 0x7f91400b3cc0 2022-06-29T16:01:42.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.099+0000 7f914843b700 1 -- 172.21.15.60:0/3956237649 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9140155f80 con 0x7f91400b3cc0 2022-06-29T16:01:42.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.099+0000 7f912f7fe700 1 -- 172.21.15.60:0/3956237649 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9130004920 con 0x7f91400b3cc0 2022-06-29T16:01:42.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.099+0000 7f914843b700 1 -- 172.21.15.60:0/3956237649 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f914013f980 con 0x7f91400b3cc0 2022-06-29T16:01:42.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.099+0000 7f914843b700 1 -- 172.21.15.60:0/3956237649 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9140156110 con 0x7f91400b3cc0 2022-06-29T16:01:42.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.103+0000 7f912f7fe700 1 -- 172.21.15.60:0/3956237649 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9130004a80 con 0x7f91400b3cc0 2022-06-29T16:01:42.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.103+0000 7f912f7fe700 1 --2- 172.21.15.60:0/3956237649 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9120053b00 0x7f9120055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:42.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.103+0000 7f912f7fe700 1 -- 172.21.15.60:0/3956237649 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f913006bc90 con 0x7f91400b3cc0 2022-06-29T16:01:42.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.103+0000 7f91461d7700 1 --2- 172.21.15.60:0/3956237649 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9120053b00 0x7f9120055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:42.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.103+0000 7f91461d7700 1 --2- 172.21.15.60:0/3956237649 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9120053b00 0x7f9120055fc0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f913c00b000 tx=0x7f913c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:42.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.107+0000 7f912f7fe700 1 -- 172.21.15.60:0/3956237649 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9130042b50 con 0x7f91400b3cc0 2022-06-29T16:01:42.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:42.275+0000 7f914843b700 1 -- 172.21.15.60:0/3956237649 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f91400573d0 con 0x7f91400b3cc0 2022-06-29T16:01:43.058 INFO:tasks.ceph.mon.a.smithi060.stderr:2022-06-29T16:01:43.051+0000 7fcc1a376700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-06-29T16:01:43.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.063+0000 7f912f7fe700 1 -- 172.21.15.60:0/3956237649 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v16) v1 ==== 161+0+0 (secure 0 0 0) 0x7f9130001c50 con 0x7f91400b3cc0 2022-06-29T16:01:43.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.067+0000 7f914843b700 1 -- 172.21.15.60:0/3956237649 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9120053b00 msgr2=0x7f9120055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:43.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.067+0000 7f914843b700 1 --2- 172.21.15.60:0/3956237649 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9120053b00 0x7f9120055fc0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f913c00b000 tx=0x7f913c006df0).stop 2022-06-29T16:01:43.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.067+0000 7f914843b700 1 -- 172.21.15.60:0/3956237649 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91400b3cc0 msgr2=0x7f914013c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:43.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.067+0000 7f914843b700 1 --2- 172.21.15.60:0/3956237649 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91400b3cc0 0x7f914013c7e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f91300018c0 tx=0x7f9130001b30).stop 2022-06-29T16:01:43.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.067+0000 7f914843b700 1 -- 172.21.15.60:0/3956237649 shutdown_connections 2022-06-29T16:01:43.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.067+0000 7f914843b700 1 --2- 172.21.15.60:0/3956237649 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9120053b00 0x7f9120055fc0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:43.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.067+0000 7f914843b700 1 --2- 172.21.15.60:0/3956237649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91400b9d70 0x7f91401399b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:43.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.067+0000 7f914843b700 1 --2- 172.21.15.60:0/3956237649 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91400b3cc0 0x7f914013c7e0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:43.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.067+0000 7f914843b700 1 --2- 172.21.15.60:0/3956237649 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f914005d1f0 0x7f91400aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:43.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.067+0000 7f914843b700 1 -- 172.21.15.60:0/3956237649 >> 172.21.15.60:0/3956237649 conn(0x7f9140062ce0 msgr2=0x7f91400b50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:43.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.067+0000 7f914843b700 1 -- 172.21.15.60:0/3956237649 shutdown_connections 2022-06-29T16:01:43.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.067+0000 7f914843b700 1 -- 172.21.15.60:0/3956237649 wait complete. 2022-06-29T16:01:43.078 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-06-29T16:01:43.091 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:01:43.111 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2022-06-29T16:01:43.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.187+0000 7ff7388a0700 1 Processor -- start 2022-06-29T16:01:43.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.187+0000 7ff7388a0700 1 -- start start 2022-06-29T16:01:43.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.187+0000 7ff7388a0700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff73405c5b0 0x7ff73405c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:43.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.187+0000 7ff7388a0700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff73405f620 0x7ff734146990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:43.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.187+0000 7ff7388a0700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff734146e80 con 0x7ff7340b0c80 2022-06-29T16:01:43.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.187+0000 7ff7388a0700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff73405fab0 con 0x7ff73405f620 2022-06-29T16:01:43.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.187+0000 7ff7388a0700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff734145fb0 con 0x7ff73405c5b0 2022-06-29T16:01:43.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.191+0000 7ff7337fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff73405c5b0 0x7ff73405c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:43.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.191+0000 7ff7337fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff73405c5b0 0x7ff73405c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:60634/0 (socket says 172.21.15.60:60634) 2022-06-29T16:01:43.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.191+0000 7ff7337fe700 1 -- 172.21.15.60:0/617002654 learned_addr learned my addr 172.21.15.60:0/617002654 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:43.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.191+0000 7ff733fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff73405f620 0x7ff734146990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:43.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.191+0000 7ff7337fe700 1 -- 172.21.15.60:0/617002654 >> v2:172.21.15.92:3300/0 conn(0x7ff73405f620 msgr2=0x7ff734146990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:01:43.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.191+0000 7ff7337fe700 1 --2- 172.21.15.60:0/617002654 >> v2:172.21.15.92:3300/0 conn(0x7ff73405f620 0x7ff734146990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:43.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.191+0000 7ff7337fe700 1 -- 172.21.15.60:0/617002654 >> v1:172.21.15.92:6789/0 conn(0x7ff7340b0c80 legacy=0x7ff73413c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:43.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.191+0000 7ff7337fe700 1 -- 172.21.15.60:0/617002654 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff734146250 con 0x7ff73405c5b0 2022-06-29T16:01:43.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.191+0000 7ff7337fe700 1 --2- 172.21.15.60:0/617002654 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff73405c5b0 0x7ff73405c990 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7ff72c0032f0 tx=0x7ff72c009290).ready entity=mon.2 client_cookie=fed441ad46fa9c2e server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:43.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.191+0000 7ff7327fc700 1 -- 172.21.15.60:0/617002654 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff72c00b8a0 con 0x7ff73405c5b0 2022-06-29T16:01:43.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.191+0000 7ff7327fc700 1 -- 172.21.15.60:0/617002654 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff72c00ba00 con 0x7ff73405c5b0 2022-06-29T16:01:43.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.191+0000 7ff7327fc700 1 -- 172.21.15.60:0/617002654 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff72c00bb70 con 0x7ff73405c5b0 2022-06-29T16:01:43.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.191+0000 7ff7388a0700 1 -- 172.21.15.60:0/617002654 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff73405c5b0 msgr2=0x7ff73405c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:43.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.191+0000 7ff7388a0700 1 --2- 172.21.15.60:0/617002654 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff73405c5b0 0x7ff73405c990 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7ff72c0032f0 tx=0x7ff72c009290).stop 2022-06-29T16:01:43.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.191+0000 7ff7388a0700 1 -- 172.21.15.60:0/617002654 shutdown_connections 2022-06-29T16:01:43.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.191+0000 7ff7388a0700 1 --2- 172.21.15.60:0/617002654 >> v2:172.21.15.92:3300/0 conn(0x7ff73405f620 0x7ff734146990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:43.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.191+0000 7ff7388a0700 1 --2- 172.21.15.60:0/617002654 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff73405c5b0 0x7ff73405c990 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:43.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.191+0000 7ff7388a0700 1 -- 172.21.15.60:0/617002654 >> 172.21.15.60:0/617002654 conn(0x7ff7340637c0 msgr2=0x7ff734063ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:43.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.191+0000 7ff7388a0700 1 -- 172.21.15.60:0/617002654 shutdown_connections 2022-06-29T16:01:43.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.191+0000 7ff7388a0700 1 -- 172.21.15.60:0/617002654 wait complete. 2022-06-29T16:01:43.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.191+0000 7ff7388a0700 1 Processor -- start 2022-06-29T16:01:43.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.191+0000 7ff7388a0700 1 -- start start 2022-06-29T16:01:43.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.195+0000 7ff7388a0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff73405c5b0 0x7ff7341412c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:43.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.195+0000 7ff7388a0700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff73405f620 0x7ff734113f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:43.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.195+0000 7ff7388a0700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7340b0c80 0x7ff734114410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:43.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.195+0000 7ff7388a0700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff734140b40 con 0x7ff73405c5b0 2022-06-29T16:01:43.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.195+0000 7ff7388a0700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73413e740 con 0x7ff7340b0c80 2022-06-29T16:01:43.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.195+0000 7ff7388a0700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff734142050 con 0x7ff73405f620 2022-06-29T16:01:43.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.195+0000 7ff7337fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff73405c5b0 0x7ff7341412c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:43.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.195+0000 7ff7337fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff73405c5b0 0x7ff7341412c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47546/0 (socket says 172.21.15.60:47546) 2022-06-29T16:01:43.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.195+0000 7ff7337fe700 1 -- 172.21.15.60:0/2991088240 learned_addr learned my addr 172.21.15.60:0/2991088240 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:43.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.195+0000 7ff733fff700 1 --2- 172.21.15.60:0/2991088240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7340b0c80 0x7ff734114410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:43.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.195+0000 7ff732ffd700 1 --2- 172.21.15.60:0/2991088240 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff73405f620 0x7ff734113f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:43.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.195+0000 7ff7337fe700 1 -- 172.21.15.60:0/2991088240 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff73405f620 msgr2=0x7ff734113f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:43.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.195+0000 7ff7337fe700 1 --2- 172.21.15.60:0/2991088240 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff73405f620 0x7ff734113f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:43.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.195+0000 7ff7337fe700 1 -- 172.21.15.60:0/2991088240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7340b0c80 msgr2=0x7ff734114410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:43.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.195+0000 7ff7337fe700 1 --2- 172.21.15.60:0/2991088240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7340b0c80 0x7ff734114410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:43.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.195+0000 7ff7337fe700 1 -- 172.21.15.60:0/2991088240 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff72c009530 con 0x7ff73405c5b0 2022-06-29T16:01:43.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.195+0000 7ff7337fe700 1 --2- 172.21.15.60:0/2991088240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff73405c5b0 0x7ff7341412c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7ff72c0032f0 tx=0x7ff72c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:43.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.195+0000 7ff730ff9700 1 -- 172.21.15.60:0/2991088240 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff72c021620 con 0x7ff73405c5b0 2022-06-29T16:01:43.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.195+0000 7ff730ff9700 1 -- 172.21.15.60:0/2991088240 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff72c021780 con 0x7ff73405c5b0 2022-06-29T16:01:43.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.195+0000 7ff7388a0700 1 -- 172.21.15.60:0/2991088240 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff734114ac0 con 0x7ff73405c5b0 2022-06-29T16:01:43.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.195+0000 7ff730ff9700 1 -- 172.21.15.60:0/2991088240 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff72c021ae0 con 0x7ff73405c5b0 2022-06-29T16:01:43.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.195+0000 7ff7388a0700 1 -- 172.21.15.60:0/2991088240 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff734140f60 con 0x7ff73405c5b0 2022-06-29T16:01:43.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.195+0000 7ff7388a0700 1 -- 172.21.15.60:0/2991088240 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff734114900 con 0x7ff73405c5b0 2022-06-29T16:01:43.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.199+0000 7ff730ff9700 1 -- 172.21.15.60:0/2991088240 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff72c00f070 con 0x7ff73405c5b0 2022-06-29T16:01:43.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.199+0000 7ff730ff9700 1 --2- 172.21.15.60:0/2991088240 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff714053a70 0x7ff714055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:43.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.199+0000 7ff730ff9700 1 -- 172.21.15.60:0/2991088240 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6159+0+0 (secure 0 0 0) 0x7ff72c06d560 con 0x7ff73405c5b0 2022-06-29T16:01:43.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.199+0000 7ff732ffd700 1 --2- 172.21.15.60:0/2991088240 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff714053a70 0x7ff714055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:43.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.199+0000 7ff732ffd700 1 --2- 172.21.15.60:0/2991088240 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff714053a70 0x7ff714055f30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7ff72800b000 tx=0x7ff728009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:43.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.203+0000 7ff730ff9700 1 -- 172.21.15.60:0/2991088240 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff72c0443d0 con 0x7ff73405c5b0 2022-06-29T16:01:43.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.371+0000 7ff7388a0700 1 -- 172.21.15.60:0/2991088240 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff734142210 con 0x7ff73405c5b0 2022-06-29T16:01:43.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.371+0000 7ff730ff9700 1 -- 172.21.15.60:0/2991088240 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+1777 (secure 0 0 0) 0x7ff72c015070 con 0x7ff73405c5b0 2022-06-29T16:01:43.381 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:01:43.381 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":16,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4937,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/1673184114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1673184114},{"type":"v1","addr":"172.21.15.92:6835","nonce":1673184114}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":13}],"filesystems":[{"mdsmap":{"epoch":16,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:00:20.685676+0000","modified":"2022-06-29T16:01:43.054329+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-06-29T16:01:43.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.375+0000 7ff7388a0700 1 -- 172.21.15.60:0/2991088240 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff714053a70 msgr2=0x7ff714055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:43.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.375+0000 7ff7388a0700 1 --2- 172.21.15.60:0/2991088240 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff714053a70 0x7ff714055f30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7ff72800b000 tx=0x7ff728009040).stop 2022-06-29T16:01:43.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.375+0000 7ff7388a0700 1 -- 172.21.15.60:0/2991088240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff73405c5b0 msgr2=0x7ff7341412c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:43.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.375+0000 7ff7388a0700 1 --2- 172.21.15.60:0/2991088240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff73405c5b0 0x7ff7341412c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7ff72c0032f0 tx=0x7ff72c016000).stop 2022-06-29T16:01:43.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.375+0000 7ff7388a0700 1 -- 172.21.15.60:0/2991088240 shutdown_connections 2022-06-29T16:01:43.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.375+0000 7ff7388a0700 1 --2- 172.21.15.60:0/2991088240 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff714053a70 0x7ff714055f30 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:43.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.375+0000 7ff7388a0700 1 --2- 172.21.15.60:0/2991088240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7340b0c80 0x7ff734114410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:43.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.375+0000 7ff7388a0700 1 --2- 172.21.15.60:0/2991088240 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff73405f620 0x7ff734113f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:43.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.375+0000 7ff7388a0700 1 --2- 172.21.15.60:0/2991088240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff73405c5b0 0x7ff7341412c0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:43.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.375+0000 7ff7388a0700 1 -- 172.21.15.60:0/2991088240 >> 172.21.15.60:0/2991088240 conn(0x7ff7340637c0 msgr2=0x7ff7341450c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:43.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.375+0000 7ff7388a0700 1 -- 172.21.15.60:0/2991088240 shutdown_connections 2022-06-29T16:01:43.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.375+0000 7ff7388a0700 1 -- 172.21.15.60:0/2991088240 wait complete. 2022-06-29T16:01:43.388 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 16 2022-06-29T16:01:43.399 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-06-29T16:01:43.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.503+0000 7fb31359e700 1 Processor -- start 2022-06-29T16:01:43.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.503+0000 7fb31359e700 1 -- start start 2022-06-29T16:01:43.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.507+0000 7fb31359e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb3140b5eb0 0x7fb314142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:43.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.507+0000 7fb31359e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3140b2c50 0x7fb3140b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:43.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.507+0000 7fb31359e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3140b0cf0 con 0x7fb3140b48b0 2022-06-29T16:01:43.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.507+0000 7fb31359e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3140b54e0 con 0x7fb3140b5eb0 2022-06-29T16:01:43.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.507+0000 7fb31359e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3140b4d10 con 0x7fb3140b2c50 2022-06-29T16:01:43.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.507+0000 7fb311d9b700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb3140b48b0 0x7fb3140b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55644/0 (socket says 172.21.15.60:55644) 2022-06-29T16:01:43.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.507+0000 7fb311d9b700 1 -- 172.21.15.60:0/1282153605 learned_addr learned my addr 172.21.15.60:0/1282153605 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:01:43.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.507+0000 7fb312d9d700 1 --2- 172.21.15.60:0/1282153605 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3140b2c50 0x7fb3140b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:43.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.507+0000 7fb31259c700 1 --2- 172.21.15.60:0/1282153605 >> v2:172.21.15.92:3300/0 conn(0x7fb3140b5eb0 0x7fb314142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:43.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.507+0000 7fb31159a700 1 -- 172.21.15.60:0/1282153605 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368466123 0 0) 0x7fb2fc001490 con 0x7fb3140b48b0 2022-06-29T16:01:43.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.507+0000 7fb31159a700 1 -- 172.21.15.60:0/1282153605 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2f8003260 con 0x7fb3140b48b0 2022-06-29T16:01:43.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.507+0000 7fb31159a700 1 -- 172.21.15.60:0/1282153605 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1774243083 0 0) 0x7fb2f8003260 con 0x7fb3140b48b0 2022-06-29T16:01:43.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.507+0000 7fb312d9d700 1 -- 172.21.15.60:0/1282153605 >> v2:172.21.15.92:3300/0 conn(0x7fb3140b5eb0 msgr2=0x7fb314142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:01:43.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.507+0000 7fb312d9d700 1 --2- 172.21.15.60:0/1282153605 >> v2:172.21.15.92:3300/0 conn(0x7fb3140b5eb0 0x7fb314142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:43.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.507+0000 7fb312d9d700 1 -- 172.21.15.60:0/1282153605 >> v1:172.21.15.60:6789/0 conn(0x7fb3140b48b0 legacy=0x7fb3140b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:43.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.507+0000 7fb312d9d700 1 -- 172.21.15.60:0/1282153605 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3140b6290 con 0x7fb3140b2c50 2022-06-29T16:01:43.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.507+0000 7fb31259c700 1 --2- 172.21.15.60:0/1282153605 >> v2:172.21.15.92:3300/0 conn(0x7fb3140b5eb0 0x7fb314142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:01:43.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.507+0000 7fb312d9d700 1 --2- 172.21.15.60:0/1282153605 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3140b2c50 0x7fb3140b30c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fb2fc001d10 tx=0x7fb304006df0).ready entity=mon.2 client_cookie=29e7390d6f0914af server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:43.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.507+0000 7fb31159a700 1 -- 172.21.15.60:0/1282153605 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2fc001b70 con 0x7fb3140b2c50 2022-06-29T16:01:43.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.507+0000 7fb31159a700 1 -- 172.21.15.60:0/1282153605 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb304000e90 con 0x7fb3140b2c50 2022-06-29T16:01:43.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.507+0000 7fb31159a700 1 -- 172.21.15.60:0/1282153605 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb304012460 con 0x7fb3140b2c50 2022-06-29T16:01:43.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.507+0000 7fb31359e700 1 -- 172.21.15.60:0/1282153605 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3140b2c50 msgr2=0x7fb3140b30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:43.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.507+0000 7fb31359e700 1 --2- 172.21.15.60:0/1282153605 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3140b2c50 0x7fb3140b30c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fb2fc001d10 tx=0x7fb304006df0).stop 2022-06-29T16:01:43.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.507+0000 7fb31359e700 1 -- 172.21.15.60:0/1282153605 shutdown_connections 2022-06-29T16:01:43.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.507+0000 7fb31359e700 1 --2- 172.21.15.60:0/1282153605 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3140b2c50 0x7fb3140b30c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:43.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.507+0000 7fb31359e700 1 --2- 172.21.15.60:0/1282153605 >> v2:172.21.15.92:3300/0 conn(0x7fb3140b5eb0 0x7fb314142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:43.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.507+0000 7fb31359e700 1 -- 172.21.15.60:0/1282153605 >> 172.21.15.60:0/1282153605 conn(0x7fb314062090 msgr2=0x7fb314062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:43.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.507+0000 7fb31359e700 1 -- 172.21.15.60:0/1282153605 shutdown_connections 2022-06-29T16:01:43.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.507+0000 7fb31359e700 1 -- 172.21.15.60:0/1282153605 wait complete. 2022-06-29T16:01:43.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.511+0000 7fb31359e700 1 Processor -- start 2022-06-29T16:01:43.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.511+0000 7fb31359e700 1 -- start start 2022-06-29T16:01:43.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.511+0000 7fb31359e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3140b2c50 0x7fb314151c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:43.539 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.511+0000 7fb31359e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3140b48b0 0x7fb3140b7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:43.539 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.511+0000 7fb31359e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3140b5eb0 0x7fb3140aef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:43.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.511+0000 7fb31359e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31414a230 con 0x7fb3140b5eb0 2022-06-29T16:01:43.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.511+0000 7fb31359e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb314140140 con 0x7fb3140b2c50 2022-06-29T16:01:43.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.511+0000 7fb31359e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3140aca20 con 0x7fb3140b48b0 2022-06-29T16:01:43.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.511+0000 7fb312d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3140b5eb0 0x7fb3140aef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:43.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.511+0000 7fb31259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3140b2c50 0x7fb314151c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:43.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.511+0000 7fb312d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3140b5eb0 0x7fb3140aef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47564/0 (socket says 172.21.15.60:47564) 2022-06-29T16:01:43.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.511+0000 7fb311d9b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3140b48b0 0x7fb3140b7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:43.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.511+0000 7fb312d9d700 1 -- 172.21.15.60:0/2788159039 learned_addr learned my addr 172.21.15.60:0/2788159039 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:43.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.511+0000 7fb312d9d700 1 -- 172.21.15.60:0/2788159039 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3140b48b0 msgr2=0x7fb3140b7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:43.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.511+0000 7fb312d9d700 1 --2- 172.21.15.60:0/2788159039 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3140b48b0 0x7fb3140b7ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:43.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.511+0000 7fb312d9d700 1 -- 172.21.15.60:0/2788159039 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3140b2c50 msgr2=0x7fb314151c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:43.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.511+0000 7fb312d9d700 1 --2- 172.21.15.60:0/2788159039 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3140b2c50 0x7fb314151c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:43.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.511+0000 7fb312d9d700 1 -- 172.21.15.60:0/2788159039 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb30c00b180 con 0x7fb3140b5eb0 2022-06-29T16:01:43.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.511+0000 7fb312d9d700 1 --2- 172.21.15.60:0/2788159039 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3140b5eb0 0x7fb3140aef90 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fb2fc001d10 tx=0x7fb304014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:43.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.511+0000 7fb30b7fe700 1 -- 172.21.15.60:0/2788159039 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb304016070 con 0x7fb3140b5eb0 2022-06-29T16:01:43.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.511+0000 7fb30b7fe700 1 -- 172.21.15.60:0/2788159039 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb30400d5e0 con 0x7fb3140b5eb0 2022-06-29T16:01:43.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.511+0000 7fb31359e700 1 -- 172.21.15.60:0/2788159039 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb30400b180 con 0x7fb3140b5eb0 2022-06-29T16:01:43.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.511+0000 7fb30b7fe700 1 -- 172.21.15.60:0/2788159039 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb30400d940 con 0x7fb3140b5eb0 2022-06-29T16:01:43.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.511+0000 7fb31359e700 1 -- 172.21.15.60:0/2788159039 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3141520f0 con 0x7fb3140b5eb0 2022-06-29T16:01:43.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.515+0000 7fb31359e700 1 -- 172.21.15.60:0/2788159039 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb314142f10 con 0x7fb3140b5eb0 2022-06-29T16:01:43.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.515+0000 7fb30b7fe700 1 -- 172.21.15.60:0/2788159039 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb304025030 con 0x7fb3140b5eb0 2022-06-29T16:01:43.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.515+0000 7fb30b7fe700 1 --2- 172.21.15.60:0/2788159039 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb2ec053aa0 0x7fb2ec055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:43.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.515+0000 7fb30b7fe700 1 -- 172.21.15.60:0/2788159039 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fb30406ccc0 con 0x7fb3140b5eb0 2022-06-29T16:01:43.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.515+0000 7fb31259c700 1 --2- 172.21.15.60:0/2788159039 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb2ec053aa0 0x7fb2ec055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:43.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.515+0000 7fb31259c700 1 --2- 172.21.15.60:0/2788159039 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb2ec053aa0 0x7fb2ec055f60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fb30c00b000 tx=0x7fb30c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:43.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.519+0000 7fb30b7fe700 1 -- 172.21.15.60:0/2788159039 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb304043b90 con 0x7fb3140b5eb0 2022-06-29T16:01:43.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:43.679+0000 7fb31359e700 1 -- 172.21.15.60:0/2788159039 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb3140573d0 con 0x7fb3140b5eb0 2022-06-29T16:01:44.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.067+0000 7fb30b7fe700 1 -- 172.21.15.60:0/2788159039 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v17) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb304023020 con 0x7fb3140b5eb0 2022-06-29T16:01:44.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.071+0000 7fb31359e700 1 -- 172.21.15.60:0/2788159039 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb2ec053aa0 msgr2=0x7fb2ec055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:44.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.071+0000 7fb31359e700 1 --2- 172.21.15.60:0/2788159039 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb2ec053aa0 0x7fb2ec055f60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fb30c00b000 tx=0x7fb30c006df0).stop 2022-06-29T16:01:44.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.071+0000 7fb31359e700 1 -- 172.21.15.60:0/2788159039 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3140b5eb0 msgr2=0x7fb3140aef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:44.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.071+0000 7fb31359e700 1 --2- 172.21.15.60:0/2788159039 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3140b5eb0 0x7fb3140aef90 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fb2fc001d10 tx=0x7fb304014000).stop 2022-06-29T16:01:44.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.071+0000 7fb31359e700 1 -- 172.21.15.60:0/2788159039 shutdown_connections 2022-06-29T16:01:44.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.071+0000 7fb31359e700 1 --2- 172.21.15.60:0/2788159039 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb2ec053aa0 0x7fb2ec055f60 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:44.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.071+0000 7fb31359e700 1 --2- 172.21.15.60:0/2788159039 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3140b5eb0 0x7fb3140aef90 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:44.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.071+0000 7fb31359e700 1 --2- 172.21.15.60:0/2788159039 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3140b48b0 0x7fb3140b7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:44.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.071+0000 7fb31359e700 1 --2- 172.21.15.60:0/2788159039 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3140b2c50 0x7fb314151c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:44.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.071+0000 7fb31359e700 1 -- 172.21.15.60:0/2788159039 >> 172.21.15.60:0/2788159039 conn(0x7fb314062090 msgr2=0x7fb31405bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:44.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.071+0000 7fb31359e700 1 -- 172.21.15.60:0/2788159039 shutdown_connections 2022-06-29T16:01:44.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.071+0000 7fb31359e700 1 -- 172.21.15.60:0/2788159039 wait complete. 2022-06-29T16:01:44.093 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-06-29T16:01:44.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.199+0000 7f3cd7899700 1 Processor -- start 2022-06-29T16:01:44.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.199+0000 7f3cd7899700 1 -- start start 2022-06-29T16:01:44.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.199+0000 7f3cd7899700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3cd00b61d0 0x7f3cd00af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:44.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.199+0000 7f3cd7899700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3cd00adc80 0x7f3cd0142a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:44.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.199+0000 7f3cd7899700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd00b65b0 con 0x7f3cd0060b60 2022-06-29T16:01:44.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.199+0000 7f3cd7899700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cd00605e0 con 0x7f3cd00b61d0 2022-06-29T16:01:44.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.199+0000 7f3cd7899700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd0060880 con 0x7f3cd00adc80 2022-06-29T16:01:44.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.199+0000 7f3cd5635700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3cd00b61d0 0x7f3cd00af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:44.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.199+0000 7f3cd5635700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3cd00b61d0 0x7f3cd00af8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47572/0 (socket says 172.21.15.60:47572) 2022-06-29T16:01:44.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.199+0000 7f3cd5635700 1 -- 172.21.15.60:0/1345911162 learned_addr learned my addr 172.21.15.60:0/1345911162 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:44.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.199+0000 7f3cd4e34700 1 --2- 172.21.15.60:0/1345911162 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3cd00adc80 0x7f3cd0142a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:44.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.203+0000 7f3cd5635700 1 -- 172.21.15.60:0/1345911162 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3cd00adc80 msgr2=0x7f3cd0142a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:01:44.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.203+0000 7f3cd5635700 1 --2- 172.21.15.60:0/1345911162 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3cd00adc80 0x7f3cd0142a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:44.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.203+0000 7f3cd5635700 1 -- 172.21.15.60:0/1345911162 >> v1:172.21.15.92:6789/0 conn(0x7f3cd0060b60 legacy=0x7f3cd0142f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:44.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.203+0000 7f3cd5635700 1 -- 172.21.15.60:0/1345911162 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd0067340 con 0x7f3cd00b61d0 2022-06-29T16:01:44.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.203+0000 7f3cd5635700 1 --2- 172.21.15.60:0/1345911162 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cd00b61d0 0x7f3cd00af8d0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f3cc4002f70 tx=0x7f3cc4006eb0).ready entity=mon.0 client_cookie=d094b8dbe5996a09 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:44.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.203+0000 7f3cc3fff700 1 -- 172.21.15.60:0/1345911162 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cc400d570 con 0x7f3cd00b61d0 2022-06-29T16:01:44.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.203+0000 7f3cc3fff700 1 -- 172.21.15.60:0/1345911162 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc4000e90 con 0x7f3cd00b61d0 2022-06-29T16:01:44.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.203+0000 7f3cc3fff700 1 -- 172.21.15.60:0/1345911162 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cc400d900 con 0x7f3cd00b61d0 2022-06-29T16:01:44.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.203+0000 7f3cd7899700 1 -- 172.21.15.60:0/1345911162 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cd00b61d0 msgr2=0x7f3cd00af8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:44.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.203+0000 7f3cd7899700 1 --2- 172.21.15.60:0/1345911162 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cd00b61d0 0x7f3cd00af8d0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f3cc4002f70 tx=0x7f3cc4006eb0).stop 2022-06-29T16:01:44.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.203+0000 7f3cd7899700 1 -- 172.21.15.60:0/1345911162 shutdown_connections 2022-06-29T16:01:44.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.203+0000 7f3cd7899700 1 --2- 172.21.15.60:0/1345911162 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3cd00adc80 0x7f3cd0142a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:44.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.203+0000 7f3cd7899700 1 --2- 172.21.15.60:0/1345911162 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cd00b61d0 0x7f3cd00af8d0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:44.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.203+0000 7f3cd7899700 1 -- 172.21.15.60:0/1345911162 >> 172.21.15.60:0/1345911162 conn(0x7f3cd0063b50 msgr2=0x7f3cd00b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:44.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.203+0000 7f3cd7899700 1 -- 172.21.15.60:0/1345911162 shutdown_connections 2022-06-29T16:01:44.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.203+0000 7f3cd7899700 1 -- 172.21.15.60:0/1345911162 wait complete. 2022-06-29T16:01:44.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.203+0000 7f3cd7899700 1 Processor -- start 2022-06-29T16:01:44.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.203+0000 7f3cd7899700 1 -- start start 2022-06-29T16:01:44.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.207+0000 7f3cd7899700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cd0060b60 0x7f3cd0146ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:44.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.207+0000 7f3cd7899700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3cd00adc80 0x7f3cd00b4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:44.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.207+0000 7f3cd7899700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cd00b61d0 0x7f3cd0141730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:44.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.207+0000 7f3cd7899700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd0064070 con 0x7f3cd00b61d0 2022-06-29T16:01:44.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.207+0000 7f3cd7899700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd0064260 con 0x7f3cd0060b60 2022-06-29T16:01:44.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.207+0000 7f3cd7899700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd0064500 con 0x7f3cd00adc80 2022-06-29T16:01:44.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.207+0000 7f3cd5e36700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cd00b61d0 0x7f3cd0141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:44.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.207+0000 7f3cd5635700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cd0060b60 0x7f3cd0146ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:44.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.207+0000 7f3cd5e36700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cd00b61d0 0x7f3cd0141730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47580/0 (socket says 172.21.15.60:47580) 2022-06-29T16:01:44.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.207+0000 7f3cd5e36700 1 -- 172.21.15.60:0/2023113412 learned_addr learned my addr 172.21.15.60:0/2023113412 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:44.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.207+0000 7f3cd4e34700 1 --2- 172.21.15.60:0/2023113412 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3cd00adc80 0x7f3cd00b4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:44.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.207+0000 7f3cd5e36700 1 -- 172.21.15.60:0/2023113412 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3cd00adc80 msgr2=0x7f3cd00b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:44.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.207+0000 7f3cd5e36700 1 --2- 172.21.15.60:0/2023113412 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3cd00adc80 0x7f3cd00b4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:44.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.207+0000 7f3cd5e36700 1 -- 172.21.15.60:0/2023113412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cd0060b60 msgr2=0x7f3cd0146ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:44.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.207+0000 7f3cd5e36700 1 --2- 172.21.15.60:0/2023113412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cd0060b60 0x7f3cd0146ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:44.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.207+0000 7f3cd5e36700 1 -- 172.21.15.60:0/2023113412 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc400b1e0 con 0x7f3cd00b61d0 2022-06-29T16:01:44.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.207+0000 7f3cd5e36700 1 --2- 172.21.15.60:0/2023113412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cd00b61d0 0x7f3cd0141730 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f3cd00af7b0 tx=0x7f3cc8007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:44.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.207+0000 7f3cc27fc700 1 -- 172.21.15.60:0/2023113412 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cc800e4a0 con 0x7f3cd00b61d0 2022-06-29T16:01:44.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.207+0000 7f3cc27fc700 1 -- 172.21.15.60:0/2023113412 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc800e600 con 0x7f3cd00b61d0 2022-06-29T16:01:44.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.207+0000 7f3cd7899700 1 -- 172.21.15.60:0/2023113412 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cd0151b30 con 0x7f3cd00b61d0 2022-06-29T16:01:44.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.207+0000 7f3cc27fc700 1 -- 172.21.15.60:0/2023113412 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cc800e960 con 0x7f3cd00b61d0 2022-06-29T16:01:44.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.207+0000 7f3cd7899700 1 -- 172.21.15.60:0/2023113412 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cd00662f0 con 0x7f3cd00b61d0 2022-06-29T16:01:44.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.211+0000 7f3cd7899700 1 -- 172.21.15.60:0/2023113412 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cd00653f0 con 0x7f3cd00b61d0 2022-06-29T16:01:44.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.211+0000 7f3cc27fc700 1 -- 172.21.15.60:0/2023113412 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3cc8019030 con 0x7f3cd00b61d0 2022-06-29T16:01:44.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.211+0000 7f3cc27fc700 1 --2- 172.21.15.60:0/2023113412 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3cb80539e0 0x7f3cb8055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:44.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.211+0000 7f3cc27fc700 1 -- 172.21.15.60:0/2023113412 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f3cc806b680 con 0x7f3cd00b61d0 2022-06-29T16:01:44.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.211+0000 7f3cd5635700 1 --2- 172.21.15.60:0/2023113412 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3cb80539e0 0x7f3cb8055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:44.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.211+0000 7f3cd5635700 1 --2- 172.21.15.60:0/2023113412 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3cb80539e0 0x7f3cb8055ea0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f3cc401c000 tx=0x7f3cc4015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:44.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.219+0000 7f3cc27fc700 1 -- 172.21.15.60:0/2023113412 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3cc80424b0 con 0x7f3cd00b61d0 2022-06-29T16:01:44.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:44.351+0000 7f3cd7899700 1 -- 172.21.15.60:0/2023113412 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3cd00573d0 con 0x7f3cd00b61d0 2022-06-29T16:01:45.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.075+0000 7f3cc27fc700 1 -- 172.21.15.60:0/2023113412 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v45) v1 ==== 185+0+0 (secure 0 0 0) 0x7f3cc8045ad0 con 0x7f3cd00b61d0 2022-06-29T16:01:45.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.079+0000 7f3cd7899700 1 -- 172.21.15.60:0/2023113412 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3cb80539e0 msgr2=0x7f3cb8055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:45.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.079+0000 7f3cd7899700 1 --2- 172.21.15.60:0/2023113412 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3cb80539e0 0x7f3cb8055ea0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f3cc401c000 tx=0x7f3cc4015000).stop 2022-06-29T16:01:45.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.079+0000 7f3cd7899700 1 -- 172.21.15.60:0/2023113412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cd00b61d0 msgr2=0x7f3cd0141730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:45.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.079+0000 7f3cd7899700 1 --2- 172.21.15.60:0/2023113412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cd00b61d0 0x7f3cd0141730 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f3cd00af7b0 tx=0x7f3cc8007f40).stop 2022-06-29T16:01:45.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.079+0000 7f3cd7899700 1 -- 172.21.15.60:0/2023113412 shutdown_connections 2022-06-29T16:01:45.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.079+0000 7f3cd7899700 1 --2- 172.21.15.60:0/2023113412 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3cb80539e0 0x7f3cb8055ea0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:45.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.079+0000 7f3cd7899700 1 --2- 172.21.15.60:0/2023113412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cd00b61d0 0x7f3cd0141730 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:45.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.079+0000 7f3cd7899700 1 --2- 172.21.15.60:0/2023113412 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3cd00adc80 0x7f3cd00b4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:45.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.079+0000 7f3cd7899700 1 --2- 172.21.15.60:0/2023113412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cd0060b60 0x7f3cd0146ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:45.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.079+0000 7f3cd7899700 1 -- 172.21.15.60:0/2023113412 >> 172.21.15.60:0/2023113412 conn(0x7f3cd0063b50 msgr2=0x7f3cd00b0320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:45.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.079+0000 7f3cd7899700 1 -- 172.21.15.60:0/2023113412 shutdown_connections 2022-06-29T16:01:45.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.079+0000 7f3cd7899700 1 -- 172.21.15.60:0/2023113412 wait complete. 2022-06-29T16:01:45.093 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2022-06-29T16:01:45.103 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-06-29T16:01:45.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.207+0000 7f1181431700 1 Processor -- start 2022-06-29T16:01:45.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.207+0000 7f1181431700 1 -- start start 2022-06-29T16:01:45.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.207+0000 7f1181431700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f117c0b28e0 0x7f117c05bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:45.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.207+0000 7f1181431700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f117c0b1d70 con 0x7f117c0afee0 2022-06-29T16:01:45.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.207+0000 7f1181431700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f117c0b9fc0 con 0x7f117c0b9be0 2022-06-29T16:01:45.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.207+0000 7f1181431700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f117c0b3c40 con 0x7f117c0b28e0 2022-06-29T16:01:45.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.207+0000 7f117b7fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f117c0b28e0 0x7f117c05bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:45.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.207+0000 7f1180c30700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f117c0afee0 0x7f117c141e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55672/0 (socket says 172.21.15.60:55672) 2022-06-29T16:01:45.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.207+0000 7f1180c30700 1 -- 172.21.15.60:0/2313159651 learned_addr learned my addr 172.21.15.60:0/2313159651 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:01:45.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.211+0000 7f117b7fe700 1 -- 172.21.15.60:0/2313159651 >> v1:172.21.15.92:6789/0 conn(0x7f117c0b9be0 legacy=0x7f117c0b21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:45.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.211+0000 7f117b7fe700 1 -- 172.21.15.60:0/2313159651 >> v1:172.21.15.60:6789/0 conn(0x7f117c0afee0 legacy=0x7f117c141e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:45.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.211+0000 7f117affd700 1 -- 172.21.15.60:0/2313159651 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 136432315 0 0) 0x7f117c0b1d70 con 0x7f117c0afee0 2022-06-29T16:01:45.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.211+0000 7f117b7fe700 1 -- 172.21.15.60:0/2313159651 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117c0b3ee0 con 0x7f117c0b28e0 2022-06-29T16:01:45.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.211+0000 7f117b7fe700 1 --2- 172.21.15.60:0/2313159651 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f117c0b28e0 0x7f117c05bd20 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f1164002f70 tx=0x7f1164006df0).ready entity=mon.0 client_cookie=5808c27b6e0ecb19 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:45.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.211+0000 7f117affd700 1 -- 172.21.15.60:0/2313159651 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f116400d610 con 0x7f117c0b28e0 2022-06-29T16:01:45.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.211+0000 7f117affd700 1 -- 172.21.15.60:0/2313159651 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1164000e90 con 0x7f117c0b28e0 2022-06-29T16:01:45.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.211+0000 7f117affd700 1 -- 172.21.15.60:0/2313159651 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f116400d960 con 0x7f117c0b28e0 2022-06-29T16:01:45.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.211+0000 7f1181431700 1 -- 172.21.15.60:0/2313159651 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f117c0b28e0 msgr2=0x7f117c05bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:45.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.211+0000 7f1181431700 1 --2- 172.21.15.60:0/2313159651 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f117c0b28e0 0x7f117c05bd20 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f1164002f70 tx=0x7f1164006df0).stop 2022-06-29T16:01:45.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.211+0000 7f1181431700 1 -- 172.21.15.60:0/2313159651 shutdown_connections 2022-06-29T16:01:45.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.211+0000 7f1181431700 1 --2- 172.21.15.60:0/2313159651 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f117c0b28e0 0x7f117c05bd20 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:45.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.211+0000 7f1181431700 1 -- 172.21.15.60:0/2313159651 >> 172.21.15.60:0/2313159651 conn(0x7f117c062ce0 msgr2=0x7f117c0b9190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:45.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.211+0000 7f1181431700 1 -- 172.21.15.60:0/2313159651 shutdown_connections 2022-06-29T16:01:45.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.211+0000 7f1181431700 1 -- 172.21.15.60:0/2313159651 wait complete. 2022-06-29T16:01:45.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.211+0000 7f1181431700 1 Processor -- start 2022-06-29T16:01:45.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.211+0000 7f1181431700 1 -- start start 2022-06-29T16:01:45.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.215+0000 7f1181431700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f117c0afee0 0x7f117c1519a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:45.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.215+0000 7f1181431700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f117c0b28e0 0x7f117c143f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:45.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.215+0000 7f1181431700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f117c0b9be0 0x7f117c13d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:45.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.215+0000 7f1181431700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117c1453c0 con 0x7f117c0b28e0 2022-06-29T16:01:45.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.215+0000 7f1181431700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117c1470f0 con 0x7f117c0afee0 2022-06-29T16:01:45.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.215+0000 7f1181431700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f117c0af550 con 0x7f117c0b9be0 2022-06-29T16:01:45.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.215+0000 7f117b7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f117c0b28e0 0x7f117c143f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:45.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.215+0000 7f117b7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f117c0b28e0 0x7f117c143f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47590/0 (socket says 172.21.15.60:47590) 2022-06-29T16:01:45.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.215+0000 7f117b7fe700 1 -- 172.21.15.60:0/2824245412 learned_addr learned my addr 172.21.15.60:0/2824245412 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:45.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.215+0000 7f1180c30700 1 --2- 172.21.15.60:0/2824245412 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f117c0b9be0 0x7f117c13d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:45.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.215+0000 7f117bfff700 1 --2- 172.21.15.60:0/2824245412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f117c0afee0 0x7f117c1519a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:45.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.215+0000 7f117b7fe700 1 -- 172.21.15.60:0/2824245412 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f117c0b9be0 msgr2=0x7f117c13d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:45.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.215+0000 7f117b7fe700 1 --2- 172.21.15.60:0/2824245412 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f117c0b9be0 0x7f117c13d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:45.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.215+0000 7f117b7fe700 1 -- 172.21.15.60:0/2824245412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f117c0afee0 msgr2=0x7f117c1519a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:45.230 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.215+0000 7f117b7fe700 1 --2- 172.21.15.60:0/2824245412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f117c0afee0 0x7f117c1519a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:45.230 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.215+0000 7f117b7fe700 1 -- 172.21.15.60:0/2824245412 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f116400b180 con 0x7f117c0b28e0 2022-06-29T16:01:45.230 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.215+0000 7f117b7fe700 1 --2- 172.21.15.60:0/2824245412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f117c0b28e0 0x7f117c143f00 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f117c0b9480 tx=0x7f1164015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:45.230 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.215+0000 7f11797fa700 1 -- 172.21.15.60:0/2824245412 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f116400de70 con 0x7f117c0b28e0 2022-06-29T16:01:45.230 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.215+0000 7f11797fa700 1 -- 172.21.15.60:0/2824245412 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1164011ac0 con 0x7f117c0b28e0 2022-06-29T16:01:45.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.215+0000 7f1181431700 1 -- 172.21.15.60:0/2824245412 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f117c147530 con 0x7f117c0b28e0 2022-06-29T16:01:45.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.215+0000 7f11797fa700 1 -- 172.21.15.60:0/2824245412 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1164011e20 con 0x7f117c0b28e0 2022-06-29T16:01:45.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.215+0000 7f1181431700 1 -- 172.21.15.60:0/2824245412 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f117c1443f0 con 0x7f117c0b28e0 2022-06-29T16:01:45.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.219+0000 7f1181431700 1 -- 172.21.15.60:0/2824245412 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f117c147230 con 0x7f117c0b28e0 2022-06-29T16:01:45.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.219+0000 7f11797fa700 1 -- 172.21.15.60:0/2824245412 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1164025030 con 0x7f117c0b28e0 2022-06-29T16:01:45.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.219+0000 7f11797fa700 1 --2- 172.21.15.60:0/2824245412 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1160053aa0 0x7f1160055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:45.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.219+0000 7f11797fa700 1 -- 172.21.15.60:0/2824245412 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f116406d0b0 con 0x7f117c0b28e0 2022-06-29T16:01:45.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.219+0000 7f117bfff700 1 --2- 172.21.15.60:0/2824245412 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1160053aa0 0x7f1160055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:45.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.219+0000 7f117bfff700 1 --2- 172.21.15.60:0/2824245412 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1160053aa0 0x7f1160055f60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f116c00d000 tx=0x7f116c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:45.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.227+0000 7f11797fa700 1 -- 172.21.15.60:0/2824245412 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1164044020 con 0x7f117c0b28e0 2022-06-29T16:01:45.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:45.355+0000 7f1181431700 1 -- 172.21.15.60:0/2824245412 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f117c0573d0 con 0x7f117c0b28e0 2022-06-29T16:01:46.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.091+0000 7f11797fa700 1 -- 172.21.15.60:0/2824245412 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v46) v1 ==== 173+0+0 (secure 0 0 0) 0x7f1164047640 con 0x7f117c0b28e0 2022-06-29T16:01:46.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.095+0000 7f1181431700 1 -- 172.21.15.60:0/2824245412 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1160053aa0 msgr2=0x7f1160055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:46.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.095+0000 7f1181431700 1 --2- 172.21.15.60:0/2824245412 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1160053aa0 0x7f1160055f60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f116c00d000 tx=0x7f116c009040).stop 2022-06-29T16:01:46.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.095+0000 7f1181431700 1 -- 172.21.15.60:0/2824245412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f117c0b28e0 msgr2=0x7f117c143f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:46.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.095+0000 7f1181431700 1 --2- 172.21.15.60:0/2824245412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f117c0b28e0 0x7f117c143f00 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f117c0b9480 tx=0x7f1164015000).stop 2022-06-29T16:01:46.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.095+0000 7f1181431700 1 -- 172.21.15.60:0/2824245412 shutdown_connections 2022-06-29T16:01:46.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.095+0000 7f1181431700 1 --2- 172.21.15.60:0/2824245412 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1160053aa0 0x7f1160055f60 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:46.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.095+0000 7f1181431700 1 --2- 172.21.15.60:0/2824245412 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f117c0b9be0 0x7f117c13d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:46.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.095+0000 7f1181431700 1 --2- 172.21.15.60:0/2824245412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f117c0b28e0 0x7f117c143f00 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:46.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.095+0000 7f1181431700 1 --2- 172.21.15.60:0/2824245412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f117c0afee0 0x7f117c1519a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:46.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.095+0000 7f1181431700 1 -- 172.21.15.60:0/2824245412 >> 172.21.15.60:0/2824245412 conn(0x7f117c062ce0 msgr2=0x7f117c05bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:46.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.095+0000 7f1181431700 1 -- 172.21.15.60:0/2824245412 shutdown_connections 2022-06-29T16:01:46.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.095+0000 7f1181431700 1 -- 172.21.15.60:0/2824245412 wait complete. 2022-06-29T16:01:46.107 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2022-06-29T16:01:46.115 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-06-29T16:01:46.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.211+0000 7fc5c5f2a700 1 Processor -- start 2022-06-29T16:01:46.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.211+0000 7fc5c5f2a700 1 -- start start 2022-06-29T16:01:46.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.211+0000 7fc5c5f2a700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc5c00b28e0 0x7fc5c00b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:46.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.211+0000 7fc5c5f2a700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc5c0065450 0x7fc5c0140180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:46.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.211+0000 7fc5c5f2a700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c0140670 con 0x7fc5c0060760 2022-06-29T16:01:46.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.211+0000 7fc5c5f2a700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5c013fa40 con 0x7fc5c00b28e0 2022-06-29T16:01:46.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.211+0000 7fc5c5f2a700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5c013fce0 con 0x7fc5c0065450 2022-06-29T16:01:46.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.211+0000 7fc5c4f28700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc5c00b28e0 0x7fc5c00b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:46.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.211+0000 7fc5c5729700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc5c0060760 0x7fc5c0139340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55688/0 (socket says 172.21.15.60:55688) 2022-06-29T16:01:46.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.211+0000 7fc5c5729700 1 -- 172.21.15.60:0/3969136555 learned_addr learned my addr 172.21.15.60:0/3969136555 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:01:46.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.211+0000 7fc5bffff700 1 --2- 172.21.15.60:0/3969136555 >> v2:172.21.15.92:3300/0 conn(0x7fc5c0065450 0x7fc5c0140180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:46.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.211+0000 7fc5bf7fe700 1 -- 172.21.15.60:0/3969136555 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2030811447 0 0) 0x7fc5c0140670 con 0x7fc5c0060760 2022-06-29T16:01:46.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.211+0000 7fc5bf7fe700 1 -- 172.21.15.60:0/3969136555 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5a4003260 con 0x7fc5c0060760 2022-06-29T16:01:46.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.211+0000 7fc5c4f28700 1 -- 172.21.15.60:0/3969136555 >> v2:172.21.15.92:3300/0 conn(0x7fc5c0065450 msgr2=0x7fc5c0140180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:01:46.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.211+0000 7fc5c4f28700 1 --2- 172.21.15.60:0/3969136555 >> v2:172.21.15.92:3300/0 conn(0x7fc5c0065450 0x7fc5c0140180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:46.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.211+0000 7fc5c4f28700 1 -- 172.21.15.60:0/3969136555 >> v1:172.21.15.60:6789/0 conn(0x7fc5c0060760 legacy=0x7fc5c0139340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:46.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.211+0000 7fc5c4f28700 1 -- 172.21.15.60:0/3969136555 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c0140860 con 0x7fc5c00b28e0 2022-06-29T16:01:46.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.211+0000 7fc5c4f28700 1 --2- 172.21.15.60:0/3969136555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5c00b28e0 0x7fc5c00b2cc0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fc5a8002f70 tx=0x7fc5a8006df0).ready entity=mon.0 client_cookie=8952d4c394d3e36a server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:46.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.211+0000 7fc5bf7fe700 1 -- 172.21.15.60:0/3969136555 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5a800d4f0 con 0x7fc5c00b28e0 2022-06-29T16:01:46.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.211+0000 7fc5bf7fe700 1 -- 172.21.15.60:0/3969136555 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5a8000e90 con 0x7fc5c00b28e0 2022-06-29T16:01:46.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.211+0000 7fc5bf7fe700 1 -- 172.21.15.60:0/3969136555 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5a800d840 con 0x7fc5c00b28e0 2022-06-29T16:01:46.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.211+0000 7fc5c5f2a700 1 -- 172.21.15.60:0/3969136555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5c00b28e0 msgr2=0x7fc5c00b2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:46.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.215+0000 7fc5c5f2a700 1 --2- 172.21.15.60:0/3969136555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5c00b28e0 0x7fc5c00b2cc0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fc5a8002f70 tx=0x7fc5a8006df0).stop 2022-06-29T16:01:46.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.215+0000 7fc5c5f2a700 1 -- 172.21.15.60:0/3969136555 shutdown_connections 2022-06-29T16:01:46.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.215+0000 7fc5c5f2a700 1 --2- 172.21.15.60:0/3969136555 >> v2:172.21.15.92:3300/0 conn(0x7fc5c0065450 0x7fc5c0140180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:46.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.215+0000 7fc5c5f2a700 1 --2- 172.21.15.60:0/3969136555 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5c00b28e0 0x7fc5c00b2cc0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:46.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.215+0000 7fc5c5f2a700 1 -- 172.21.15.60:0/3969136555 >> 172.21.15.60:0/3969136555 conn(0x7fc5c0061c80 msgr2=0x7fc5c00637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:46.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.215+0000 7fc5c5f2a700 1 -- 172.21.15.60:0/3969136555 shutdown_connections 2022-06-29T16:01:46.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.215+0000 7fc5c5f2a700 1 -- 172.21.15.60:0/3969136555 wait complete. 2022-06-29T16:01:46.230 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.215+0000 7fc5c5f2a700 1 Processor -- start 2022-06-29T16:01:46.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.215+0000 7fc5c5f2a700 1 -- start start 2022-06-29T16:01:46.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.215+0000 7fc5c5f2a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5c0060760 0x7fc5c0144100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:46.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.215+0000 7fc5c5f2a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5c0065450 0x7fc5c0151e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:46.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.215+0000 7fc5c5f2a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc5c00b28e0 0x7fc5c0152360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:46.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.215+0000 7fc5c5f2a700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c0149e30 con 0x7fc5c0065450 2022-06-29T16:01:46.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.215+0000 7fc5c5f2a700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c0147fc0 con 0x7fc5c0060760 2022-06-29T16:01:46.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.215+0000 7fc5c5f2a700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c0148260 con 0x7fc5c00b28e0 2022-06-29T16:01:46.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.215+0000 7fc5bffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5c0065450 0x7fc5c0151e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:46.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.215+0000 7fc5bffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5c0065450 0x7fc5c0151e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47604/0 (socket says 172.21.15.60:47604) 2022-06-29T16:01:46.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.215+0000 7fc5bffff700 1 -- 172.21.15.60:0/703292080 learned_addr learned my addr 172.21.15.60:0/703292080 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:46.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.215+0000 7fc5c5729700 1 --2- 172.21.15.60:0/703292080 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc5c00b28e0 0x7fc5c0152360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:46.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.215+0000 7fc5bffff700 1 -- 172.21.15.60:0/703292080 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc5c00b28e0 msgr2=0x7fc5c0152360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:46.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.215+0000 7fc5bffff700 1 --2- 172.21.15.60:0/703292080 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc5c00b28e0 0x7fc5c0152360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:46.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.215+0000 7fc5bffff700 1 -- 172.21.15.60:0/703292080 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5c0060760 msgr2=0x7fc5c0144100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:46.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.215+0000 7fc5c4f28700 1 --2- 172.21.15.60:0/703292080 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5c0060760 0x7fc5c0144100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:46.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.215+0000 7fc5bffff700 1 --2- 172.21.15.60:0/703292080 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5c0060760 0x7fc5c0144100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:46.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.215+0000 7fc5bffff700 1 -- 172.21.15.60:0/703292080 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a800b180 con 0x7fc5c0065450 2022-06-29T16:01:46.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.215+0000 7fc5c4f28700 1 --2- 172.21.15.60:0/703292080 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5c0060760 0x7fc5c0144100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:01:46.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.215+0000 7fc5bffff700 1 --2- 172.21.15.60:0/703292080 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5c0065450 0x7fc5c0151e70 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fc5b800c000 tx=0x7fc5b80091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:46.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.215+0000 7fc5bdffb700 1 -- 172.21.15.60:0/703292080 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5b800c850 con 0x7fc5c0065450 2022-06-29T16:01:46.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.215+0000 7fc5bdffb700 1 -- 172.21.15.60:0/703292080 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b800c9b0 con 0x7fc5c0065450 2022-06-29T16:01:46.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.215+0000 7fc5c5f2a700 1 -- 172.21.15.60:0/703292080 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5c0149ab0 con 0x7fc5c0065450 2022-06-29T16:01:46.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.215+0000 7fc5bdffb700 1 -- 172.21.15.60:0/703292080 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5b800cd10 con 0x7fc5c0065450 2022-06-29T16:01:46.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.215+0000 7fc5c5f2a700 1 -- 172.21.15.60:0/703292080 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c0152850 con 0x7fc5c0065450 2022-06-29T16:01:46.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.219+0000 7fc5c5f2a700 1 -- 172.21.15.60:0/703292080 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5c00573d0 con 0x7fc5c0065450 2022-06-29T16:01:46.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.219+0000 7fc5bdffb700 1 -- 172.21.15.60:0/703292080 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc5b8022070 con 0x7fc5c0065450 2022-06-29T16:01:46.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.219+0000 7fc5bdffb700 1 --2- 172.21.15.60:0/703292080 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc598053980 0x7fc598055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:46.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.219+0000 7fc5bdffb700 1 -- 172.21.15.60:0/703292080 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fc5b806c3b0 con 0x7fc5c0065450 2022-06-29T16:01:46.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.219+0000 7fc5c4f28700 1 --2- 172.21.15.60:0/703292080 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc598053980 0x7fc598055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:46.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.219+0000 7fc5c4f28700 1 --2- 172.21.15.60:0/703292080 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc598053980 0x7fc598055e40 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fc5a8015000 tx=0x7fc5a8014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:46.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.227+0000 7fc5bdffb700 1 -- 172.21.15.60:0/703292080 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc5b80435b0 con 0x7fc5c0065450 2022-06-29T16:01:46.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.359+0000 7fc5c5f2a700 1 -- 172.21.15.60:0/703292080 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fc5c0068480 con 0x7fc5c0065450 2022-06-29T16:01:46.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.367+0000 7fc5bdffb700 1 -- 172.21.15.60:0/703292080 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v5) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc5b8046bd0 con 0x7fc5c0065450 2022-06-29T16:01:46.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.371+0000 7fc5c5f2a700 1 -- 172.21.15.60:0/703292080 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc598053980 msgr2=0x7fc598055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:46.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.371+0000 7fc5c5f2a700 1 --2- 172.21.15.60:0/703292080 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc598053980 0x7fc598055e40 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fc5a8015000 tx=0x7fc5a8014000).stop 2022-06-29T16:01:46.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.371+0000 7fc5c5f2a700 1 -- 172.21.15.60:0/703292080 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5c0065450 msgr2=0x7fc5c0151e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:46.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.371+0000 7fc5c5f2a700 1 --2- 172.21.15.60:0/703292080 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5c0065450 0x7fc5c0151e70 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fc5b800c000 tx=0x7fc5b80091d0).stop 2022-06-29T16:01:46.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.371+0000 7fc5c5f2a700 1 -- 172.21.15.60:0/703292080 shutdown_connections 2022-06-29T16:01:46.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.371+0000 7fc5c5f2a700 1 --2- 172.21.15.60:0/703292080 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc598053980 0x7fc598055e40 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:46.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.371+0000 7fc5c5f2a700 1 --2- 172.21.15.60:0/703292080 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc5c00b28e0 0x7fc5c0152360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:46.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.371+0000 7fc5c5f2a700 1 --2- 172.21.15.60:0/703292080 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5c0065450 0x7fc5c0151e70 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:46.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.371+0000 7fc5c5f2a700 1 --2- 172.21.15.60:0/703292080 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5c0060760 0x7fc5c0144100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:46.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.371+0000 7fc5c5f2a700 1 -- 172.21.15.60:0/703292080 >> 172.21.15.60:0/703292080 conn(0x7fc5c0061c80 msgr2=0x7fc5c005f240 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:46.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.371+0000 7fc5c5f2a700 1 -- 172.21.15.60:0/703292080 shutdown_connections 2022-06-29T16:01:46.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.371+0000 7fc5c5f2a700 1 -- 172.21.15.60:0/703292080 wait complete. 2022-06-29T16:01:46.394 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead' 2022-06-29T16:01:46.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.491+0000 7fade759e700 1 Processor -- start 2022-06-29T16:01:46.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.491+0000 7fade759e700 1 -- start start 2022-06-29T16:01:46.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.491+0000 7fade759e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fade805c5b0 0x7fade805c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:46.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.491+0000 7fade759e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fade8060610 0x7fade813c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:46.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.491+0000 7fade759e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade80b0a40 con 0x7fade813cc30 2022-06-29T16:01:46.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.491+0000 7fade759e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fade80b0850 con 0x7fade805c5b0 2022-06-29T16:01:46.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.491+0000 7fade759e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fade80b0ce0 con 0x7fade8060610 2022-06-29T16:01:46.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.491+0000 7fadddd9b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fade8060610 0x7fade813c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:46.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.491+0000 7fade659c700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fade805c5b0 0x7fade805c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:46.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.491+0000 7fade6d9d700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fade813cc30 0x7fade8142a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:59950/0 (socket says 172.21.15.60:59950) 2022-06-29T16:01:46.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.491+0000 7fade6d9d700 1 -- 172.21.15.60:0/2262522037 learned_addr learned my addr 172.21.15.60:0/2262522037 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:01:46.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.495+0000 7fade5d9b700 1 -- 172.21.15.60:0/2262522037 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2131898601 0 0) 0x7fade80b0a40 con 0x7fade813cc30 2022-06-29T16:01:46.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.495+0000 7fade5d9b700 1 -- 172.21.15.60:0/2262522037 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadc8003260 con 0x7fade813cc30 2022-06-29T16:01:46.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.495+0000 7fadddd9b700 1 -- 172.21.15.60:0/2262522037 >> v2:172.21.15.92:3300/0 conn(0x7fade805c5b0 msgr2=0x7fade805c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:01:46.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.495+0000 7fadddd9b700 1 --2- 172.21.15.60:0/2262522037 >> v2:172.21.15.92:3300/0 conn(0x7fade805c5b0 0x7fade805c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:46.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.495+0000 7fadddd9b700 1 -- 172.21.15.60:0/2262522037 >> v1:172.21.15.92:6789/0 conn(0x7fade813cc30 legacy=0x7fade8142a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:46.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.495+0000 7fadddd9b700 1 -- 172.21.15.60:0/2262522037 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade805f350 con 0x7fade8060610 2022-06-29T16:01:46.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.495+0000 7fadddd9b700 1 --2- 172.21.15.60:0/2262522037 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fade8060610 0x7fade813c740 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fadd0002f70 tx=0x7fadd0006df0).ready entity=mon.2 client_cookie=58ce72d45071509e server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:46.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.495+0000 7fade5d9b700 1 -- 172.21.15.60:0/2262522037 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fadd000d4a0 con 0x7fade8060610 2022-06-29T16:01:46.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.495+0000 7fade5d9b700 1 -- 172.21.15.60:0/2262522037 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd0000e90 con 0x7fade8060610 2022-06-29T16:01:46.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.495+0000 7fade5d9b700 1 -- 172.21.15.60:0/2262522037 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fadd000d7f0 con 0x7fade8060610 2022-06-29T16:01:46.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.495+0000 7fade759e700 1 -- 172.21.15.60:0/2262522037 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fade8060610 msgr2=0x7fade813c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:46.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.495+0000 7fade759e700 1 --2- 172.21.15.60:0/2262522037 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fade8060610 0x7fade813c740 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fadd0002f70 tx=0x7fadd0006df0).stop 2022-06-29T16:01:46.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.495+0000 7fade759e700 1 -- 172.21.15.60:0/2262522037 shutdown_connections 2022-06-29T16:01:46.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.495+0000 7fade759e700 1 --2- 172.21.15.60:0/2262522037 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fade8060610 0x7fade813c740 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:46.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.495+0000 7fade759e700 1 --2- 172.21.15.60:0/2262522037 >> v2:172.21.15.92:3300/0 conn(0x7fade805c5b0 0x7fade805c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:46.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.495+0000 7fade759e700 1 -- 172.21.15.60:0/2262522037 >> 172.21.15.60:0/2262522037 conn(0x7fade8061c80 msgr2=0x7fade80637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:46.510 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.495+0000 7fade759e700 1 -- 172.21.15.60:0/2262522037 shutdown_connections 2022-06-29T16:01:46.510 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.495+0000 7fade759e700 1 -- 172.21.15.60:0/2262522037 wait complete. 2022-06-29T16:01:46.510 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.495+0000 7fade759e700 1 Processor -- start 2022-06-29T16:01:46.510 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.495+0000 7fade759e700 1 -- start start 2022-06-29T16:01:46.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.499+0000 7fade759e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fade805c5b0 0x7fade8144c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:46.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.499+0000 7fade759e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fade8060610 0x7fade8145170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:46.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.499+0000 7fade759e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fade813cc30 0x7fade8142570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:46.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.499+0000 7fade759e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade8145660 con 0x7fade805c5b0 2022-06-29T16:01:46.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.499+0000 7fade759e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade8145870 con 0x7fade813cc30 2022-06-29T16:01:46.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.499+0000 7fade759e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fade8152610 con 0x7fade8060610 2022-06-29T16:01:46.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.499+0000 7fade659c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fade805c5b0 0x7fade8144c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:46.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.499+0000 7fade659c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fade805c5b0 0x7fade8144c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47618/0 (socket says 172.21.15.60:47618) 2022-06-29T16:01:46.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.499+0000 7fade659c700 1 -- 172.21.15.60:0/492927479 learned_addr learned my addr 172.21.15.60:0/492927479 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:46.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.499+0000 7fadddd9b700 1 --2- 172.21.15.60:0/492927479 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fade8060610 0x7fade8145170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:46.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.499+0000 7fade6d9d700 1 --2- 172.21.15.60:0/492927479 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fade813cc30 0x7fade8142570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:46.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.499+0000 7fade659c700 1 -- 172.21.15.60:0/492927479 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fade8060610 msgr2=0x7fade8145170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:46.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.499+0000 7fade659c700 1 --2- 172.21.15.60:0/492927479 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fade8060610 0x7fade8145170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:46.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.499+0000 7fade659c700 1 -- 172.21.15.60:0/492927479 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fade813cc30 msgr2=0x7fade8142570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:46.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.499+0000 7fade659c700 1 --2- 172.21.15.60:0/492927479 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fade813cc30 0x7fade8142570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:46.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.499+0000 7fade659c700 1 -- 172.21.15.60:0/492927479 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd000b180 con 0x7fade805c5b0 2022-06-29T16:01:46.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.499+0000 7fade659c700 1 --2- 172.21.15.60:0/492927479 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fade805c5b0 0x7fade8144c80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7fadd800c000 tx=0x7fadd8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:46.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.499+0000 7faddffff700 1 -- 172.21.15.60:0/492927479 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fadd8010780 con 0x7fade805c5b0 2022-06-29T16:01:46.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.499+0000 7faddffff700 1 -- 172.21.15.60:0/492927479 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd80108e0 con 0x7fade805c5b0 2022-06-29T16:01:46.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.499+0000 7fade759e700 1 -- 172.21.15.60:0/492927479 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fade81528b0 con 0x7fade805c5b0 2022-06-29T16:01:46.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.499+0000 7faddffff700 1 -- 172.21.15.60:0/492927479 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fadd8010c40 con 0x7fade805c5b0 2022-06-29T16:01:46.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.499+0000 7fade759e700 1 -- 172.21.15.60:0/492927479 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fade80ad950 con 0x7fade805c5b0 2022-06-29T16:01:46.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.499+0000 7fade759e700 1 -- 172.21.15.60:0/492927479 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fade8152a40 con 0x7fade805c5b0 2022-06-29T16:01:46.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.499+0000 7faddffff700 1 -- 172.21.15.60:0/492927479 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fadd801c460 con 0x7fade805c5b0 2022-06-29T16:01:46.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.499+0000 7faddffff700 1 --2- 172.21.15.60:0/492927479 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fadcc053b00 0x7fadcc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:46.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.499+0000 7faddffff700 1 -- 172.21.15.60:0/492927479 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fadd806c010 con 0x7fade805c5b0 2022-06-29T16:01:46.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.503+0000 7fadddd9b700 1 --2- 172.21.15.60:0/492927479 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fadcc053b00 0x7fadcc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:46.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.503+0000 7fadddd9b700 1 --2- 172.21.15.60:0/492927479 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fadcc053b00 0x7fadcc055fc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fadd001a000 tx=0x7fadd0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:46.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.507+0000 7faddffff700 1 -- 172.21.15.60:0/492927479 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fadd80431b0 con 0x7fade805c5b0 2022-06-29T16:01:46.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:46.679+0000 7fade759e700 1 -- 172.21.15.60:0/492927479 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead"]} v 0) v1 -- 0x7fade80573d0 con 0x7fade805c5b0 2022-06-29T16:01:47.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.091+0000 7faddffff700 1 -- 172.21.15.60:0/492927479 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead"]}]=0 v155) v1 ==== 144+0+0 (secure 0 0 0) 0x7fadd80467d0 con 0x7fade805c5b0 2022-06-29T16:01:47.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.095+0000 7fade759e700 1 -- 172.21.15.60:0/492927479 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fadcc053b00 msgr2=0x7fadcc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:47.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.095+0000 7fade759e700 1 --2- 172.21.15.60:0/492927479 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fadcc053b00 0x7fadcc055fc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fadd001a000 tx=0x7fadd0014000).stop 2022-06-29T16:01:47.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.095+0000 7fade759e700 1 -- 172.21.15.60:0/492927479 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fade805c5b0 msgr2=0x7fade8144c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:47.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.095+0000 7fade759e700 1 --2- 172.21.15.60:0/492927479 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fade805c5b0 0x7fade8144c80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7fadd800c000 tx=0x7fadd8006df0).stop 2022-06-29T16:01:47.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.095+0000 7fade759e700 1 -- 172.21.15.60:0/492927479 shutdown_connections 2022-06-29T16:01:47.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.095+0000 7fade759e700 1 --2- 172.21.15.60:0/492927479 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fadcc053b00 0x7fadcc055fc0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:47.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.095+0000 7fade759e700 1 --2- 172.21.15.60:0/492927479 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fade813cc30 0x7fade8142570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:47.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.095+0000 7fade759e700 1 --2- 172.21.15.60:0/492927479 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fade8060610 0x7fade8145170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:47.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.095+0000 7fade759e700 1 --2- 172.21.15.60:0/492927479 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fade805c5b0 0x7fade8144c80 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:47.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.095+0000 7fade759e700 1 -- 172.21.15.60:0/492927479 >> 172.21.15.60:0/492927479 conn(0x7fade8061c80 msgr2=0x7fade8065330 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:47.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.095+0000 7fade759e700 1 -- 172.21.15.60:0/492927479 shutdown_connections 2022-06-29T16:01:47.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.095+0000 7fade759e700 1 -- 172.21.15.60:0/492927479 wait complete. 2022-06-29T16:01:47.121 INFO:tasks.cephfs_test_runner:test_drop_cache_command_dead (tasks.cephfs.test_misc.TestCacheDrop) ... ok 2022-06-29T16:01:47.121 INFO:tasks.cephfs_test_runner:Starting test: test_drop_cache_command_dead_timeout (tasks.cephfs.test_misc.TestCacheDrop) 2022-06-29T16:01:47.122 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead_timeout' 2022-06-29T16:01:47.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.219+0000 7f40523b1700 1 Processor -- start 2022-06-29T16:01:47.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.219+0000 7f40523b1700 1 -- start start 2022-06-29T16:01:47.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.223+0000 7f40523b1700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f404c0b1650 0x7f404c13c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:47.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.223+0000 7f40523b1700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f404c13ef20 0x7f404c13f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:47.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.223+0000 7f40523b1700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f404c0afde0 con 0x7f404c0afa00 2022-06-29T16:01:47.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.223+0000 7f40523b1700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f404c0ae6a0 con 0x7f404c13ef20 2022-06-29T16:01:47.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.223+0000 7f40523b1700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f404c142900 con 0x7f404c0b1650 2022-06-29T16:01:47.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.223+0000 7f405094e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f404c13ef20 0x7f404c13f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:47.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.223+0000 7f405094e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f404c13ef20 0x7f404c13f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47630/0 (socket says 172.21.15.60:47630) 2022-06-29T16:01:47.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.223+0000 7f405094e700 1 -- 172.21.15.60:0/191080362 learned_addr learned my addr 172.21.15.60:0/191080362 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:47.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.223+0000 7f4043fff700 1 --2- 172.21.15.60:0/191080362 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f404c0b1650 0x7f404c13c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:47.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.223+0000 7f404bfff700 1 --1- 172.21.15.60:0/191080362 >> v1:172.21.15.92:6789/0 conn(0x7f404c0afa00 0x7f404c13bc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:59960/0 (socket says 172.21.15.60:59960) 2022-06-29T16:01:47.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.223+0000 7f405094e700 1 -- 172.21.15.60:0/191080362 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f404c0b1650 msgr2=0x7f404c13c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:01:47.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.223+0000 7f405094e700 1 --2- 172.21.15.60:0/191080362 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f404c0b1650 0x7f404c13c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:47.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.223+0000 7f405094e700 1 -- 172.21.15.60:0/191080362 >> v1:172.21.15.92:6789/0 conn(0x7f404c0afa00 legacy=0x7f404c13bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:01:47.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.223+0000 7f405094e700 1 -- 172.21.15.60:0/191080362 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f404c142ba0 con 0x7f404c13ef20 2022-06-29T16:01:47.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.223+0000 7f405094e700 1 --2- 172.21.15.60:0/191080362 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f404c13ef20 0x7f404c13f380 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f404c1431c0 tx=0x7f4044009290).ready entity=mon.0 client_cookie=4c7c36fbac0bec40 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:47.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.223+0000 7f404b7fe700 1 -- 172.21.15.60:0/191080362 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f404400b860 con 0x7f404c13ef20 2022-06-29T16:01:47.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.223+0000 7f404b7fe700 1 -- 172.21.15.60:0/191080362 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f404400b9c0 con 0x7f404c13ef20 2022-06-29T16:01:47.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.223+0000 7f404b7fe700 1 -- 172.21.15.60:0/191080362 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f404400bb70 con 0x7f404c13ef20 2022-06-29T16:01:47.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.227+0000 7f40523b1700 1 -- 172.21.15.60:0/191080362 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f404c13ef20 msgr2=0x7f404c13f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:47.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.227+0000 7f40523b1700 1 --2- 172.21.15.60:0/191080362 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f404c13ef20 0x7f404c13f380 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f404c1431c0 tx=0x7f4044009290).stop 2022-06-29T16:01:47.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.227+0000 7f40523b1700 1 -- 172.21.15.60:0/191080362 shutdown_connections 2022-06-29T16:01:47.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.227+0000 7f40523b1700 1 --2- 172.21.15.60:0/191080362 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f404c13ef20 0x7f404c13f380 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:47.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.227+0000 7f40523b1700 1 --2- 172.21.15.60:0/191080362 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f404c0b1650 0x7f404c13c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:47.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.227+0000 7f40523b1700 1 -- 172.21.15.60:0/191080362 >> 172.21.15.60:0/191080362 conn(0x7f404c05f0b0 msgr2=0x7f404c05f490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:47.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.227+0000 7f40523b1700 1 -- 172.21.15.60:0/191080362 shutdown_connections 2022-06-29T16:01:47.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.227+0000 7f40523b1700 1 -- 172.21.15.60:0/191080362 wait complete. 2022-06-29T16:01:47.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.227+0000 7f40523b1700 1 Processor -- start 2022-06-29T16:01:47.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.227+0000 7f40523b1700 1 -- start start 2022-06-29T16:01:47.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.231+0000 7f40523b1700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f404c0afa00 0x7f404c0acd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:47.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.231+0000 7f40523b1700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f404c0b1650 0x7f404c1395f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:47.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.231+0000 7f40523b1700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f404c13ef20 0x7f404c139ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:47.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.231+0000 7f40523b1700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f404c112570 con 0x7f404c0b1650 2022-06-29T16:01:47.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.231+0000 7f40523b1700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f404c1520f0 con 0x7f404c0afa00 2022-06-29T16:01:47.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.231+0000 7f40523b1700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f404c152390 con 0x7f404c13ef20 2022-06-29T16:01:47.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.231+0000 7f4043fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f404c0b1650 0x7f404c1395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:47.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.231+0000 7f4043fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f404c0b1650 0x7f404c1395f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47634/0 (socket says 172.21.15.60:47634) 2022-06-29T16:01:47.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.231+0000 7f4043fff700 1 -- 172.21.15.60:0/2110853891 learned_addr learned my addr 172.21.15.60:0/2110853891 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:47.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.231+0000 7f405094e700 1 --2- 172.21.15.60:0/2110853891 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f404c13ef20 0x7f404c139ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:47.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.231+0000 7f404bfff700 1 --2- 172.21.15.60:0/2110853891 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f404c0afa00 0x7f404c0acd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:47.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.231+0000 7f4043fff700 1 -- 172.21.15.60:0/2110853891 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f404c13ef20 msgr2=0x7f404c139ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:47.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.231+0000 7f4043fff700 1 --2- 172.21.15.60:0/2110853891 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f404c13ef20 0x7f404c139ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:47.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.231+0000 7f4043fff700 1 -- 172.21.15.60:0/2110853891 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f404c0afa00 msgr2=0x7f404c0acd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:47.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.231+0000 7f4043fff700 1 --2- 172.21.15.60:0/2110853891 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f404c0afa00 0x7f404c0acd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:47.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.231+0000 7f4043fff700 1 -- 172.21.15.60:0/2110853891 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4044009530 con 0x7f404c0b1650 2022-06-29T16:01:47.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.231+0000 7f4043fff700 1 --2- 172.21.15.60:0/2110853891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f404c0b1650 0x7f404c1395f0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f403400e000 tx=0x7f403400f040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:47.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.231+0000 7f4049ffb700 1 -- 172.21.15.60:0/2110853891 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4034009690 con 0x7f404c0b1650 2022-06-29T16:01:47.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.231+0000 7f4049ffb700 1 -- 172.21.15.60:0/2110853891 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f403400a040 con 0x7f404c0b1650 2022-06-29T16:01:47.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.231+0000 7f40523b1700 1 -- 172.21.15.60:0/2110853891 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f404c152630 con 0x7f404c0b1650 2022-06-29T16:01:47.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.231+0000 7f4049ffb700 1 -- 172.21.15.60:0/2110853891 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40340099e0 con 0x7f404c0b1650 2022-06-29T16:01:47.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.231+0000 7f40523b1700 1 -- 172.21.15.60:0/2110853891 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f404c0add10 con 0x7f404c0b1650 2022-06-29T16:01:47.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.231+0000 7f40523b1700 1 -- 172.21.15.60:0/2110853891 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f404c142e40 con 0x7f404c0b1650 2022-06-29T16:01:47.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.231+0000 7f4049ffb700 1 -- 172.21.15.60:0/2110853891 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f403401d030 con 0x7f404c0b1650 2022-06-29T16:01:47.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.235+0000 7f4049ffb700 1 --2- 172.21.15.60:0/2110853891 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f40300539e0 0x7f4030055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:47.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.235+0000 7f4049ffb700 1 -- 172.21.15.60:0/2110853891 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f403406b950 con 0x7f404c0b1650 2022-06-29T16:01:47.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.235+0000 7f404bfff700 1 --2- 172.21.15.60:0/2110853891 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f40300539e0 0x7f4030055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:47.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.235+0000 7f404bfff700 1 --2- 172.21.15.60:0/2110853891 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f40300539e0 0x7f4030055ea0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f403c00d000 tx=0x7f403c007ba0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:47.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.239+0000 7f4049ffb700 1 -- 172.21.15.60:0/2110853891 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4034042b50 con 0x7f404c0b1650 2022-06-29T16:01:47.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:47.407+0000 7f40523b1700 1 -- 172.21.15.60:0/2110853891 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead_timeout"]} v 0) v1 -- 0x7f404c1527c0 con 0x7f404c0b1650 2022-06-29T16:01:48.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.095+0000 7f4049ffb700 1 -- 172.21.15.60:0/2110853891 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead_timeout"]}]=0 v156) v1 ==== 155+0+0 (secure 0 0 0) 0x7f403401b020 con 0x7f404c0b1650 2022-06-29T16:01:48.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.099+0000 7f40523b1700 1 -- 172.21.15.60:0/2110853891 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f40300539e0 msgr2=0x7f4030055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:48.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.099+0000 7f40523b1700 1 --2- 172.21.15.60:0/2110853891 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f40300539e0 0x7f4030055ea0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f403c00d000 tx=0x7f403c007ba0).stop 2022-06-29T16:01:48.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.099+0000 7f40523b1700 1 -- 172.21.15.60:0/2110853891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f404c0b1650 msgr2=0x7f404c1395f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:48.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.099+0000 7f40523b1700 1 --2- 172.21.15.60:0/2110853891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f404c0b1650 0x7f404c1395f0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f403400e000 tx=0x7f403400f040).stop 2022-06-29T16:01:48.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.099+0000 7f40523b1700 1 -- 172.21.15.60:0/2110853891 shutdown_connections 2022-06-29T16:01:48.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.099+0000 7f40523b1700 1 --2- 172.21.15.60:0/2110853891 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f40300539e0 0x7f4030055ea0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:48.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.099+0000 7f40523b1700 1 --2- 172.21.15.60:0/2110853891 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f404c13ef20 0x7f404c139ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:48.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.099+0000 7f40523b1700 1 --2- 172.21.15.60:0/2110853891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f404c0b1650 0x7f404c1395f0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:48.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.099+0000 7f40523b1700 1 --2- 172.21.15.60:0/2110853891 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f404c0afa00 0x7f404c0acd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:48.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.099+0000 7f40523b1700 1 -- 172.21.15.60:0/2110853891 >> 172.21.15.60:0/2110853891 conn(0x7f404c05f0b0 msgr2=0x7f404c13d3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:48.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.099+0000 7f40523b1700 1 -- 172.21.15.60:0/2110853891 shutdown_connections 2022-06-29T16:01:48.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.099+0000 7f40523b1700 1 -- 172.21.15.60:0/2110853891 wait complete. 2022-06-29T16:01:48.122 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-06-29T16:01:48.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.219+0000 7fc46e80d700 1 Processor -- start 2022-06-29T16:01:48.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.219+0000 7fc46e80d700 1 -- start start 2022-06-29T16:01:48.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.223+0000 7fc46e80d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4680b61d0 0x7fc4680af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:48.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.223+0000 7fc46e80d700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4680b65b0 con 0x7fc468060b60 2022-06-29T16:01:48.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.223+0000 7fc46e80d700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4680605e0 con 0x7fc4680adc80 2022-06-29T16:01:48.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.223+0000 7fc46e80d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc468060880 con 0x7fc4680b61d0 2022-06-29T16:01:48.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.223+0000 7fc46cdaa700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc468060b60 0x7fc46813c010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55730/0 (socket says 172.21.15.60:55730) 2022-06-29T16:01:48.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.223+0000 7fc46cdaa700 1 -- 172.21.15.60:0/3588485642 learned_addr learned my addr 172.21.15.60:0/3588485642 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:01:48.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.223+0000 7fc467fff700 1 --2- 172.21.15.60:0/3588485642 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4680b61d0 0x7fc4680af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:48.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.223+0000 7fc466ffd700 1 -- 172.21.15.60:0/3588485642 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 801882092 0 0) 0x7fc4680b65b0 con 0x7fc468060b60 2022-06-29T16:01:48.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.223+0000 7fc466ffd700 1 -- 172.21.15.60:0/3588485642 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc454003260 con 0x7fc468060b60 2022-06-29T16:01:48.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.223+0000 7fc467fff700 1 -- 172.21.15.60:0/3588485642 >> v1:172.21.15.92:6789/0 conn(0x7fc4680adc80 legacy=0x7fc468142a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:48.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.223+0000 7fc467fff700 1 -- 172.21.15.60:0/3588485642 >> v1:172.21.15.60:6789/0 conn(0x7fc468060b60 legacy=0x7fc46813c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:48.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.223+0000 7fc467fff700 1 -- 172.21.15.60:0/3588485642 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc468067340 con 0x7fc4680b61d0 2022-06-29T16:01:48.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.223+0000 7fc467fff700 1 --2- 172.21.15.60:0/3588485642 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4680b61d0 0x7fc4680af8d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fc458002f70 tx=0x7fc458006df0).ready entity=mon.2 client_cookie=f98200a2a5eb6a30 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:48.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.223+0000 7fc466ffd700 1 -- 172.21.15.60:0/3588485642 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc45800d570 con 0x7fc4680b61d0 2022-06-29T16:01:48.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.223+0000 7fc466ffd700 1 -- 172.21.15.60:0/3588485642 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc458000e90 con 0x7fc4680b61d0 2022-06-29T16:01:48.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.223+0000 7fc466ffd700 1 -- 172.21.15.60:0/3588485642 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc45800d8c0 con 0x7fc4680b61d0 2022-06-29T16:01:48.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.227+0000 7fc46e80d700 1 -- 172.21.15.60:0/3588485642 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4680b61d0 msgr2=0x7fc4680af8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:48.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.227+0000 7fc46e80d700 1 --2- 172.21.15.60:0/3588485642 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4680b61d0 0x7fc4680af8d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fc458002f70 tx=0x7fc458006df0).stop 2022-06-29T16:01:48.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.227+0000 7fc46e80d700 1 -- 172.21.15.60:0/3588485642 shutdown_connections 2022-06-29T16:01:48.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.227+0000 7fc46e80d700 1 --2- 172.21.15.60:0/3588485642 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4680b61d0 0x7fc4680af8d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:48.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.227+0000 7fc46e80d700 1 -- 172.21.15.60:0/3588485642 >> 172.21.15.60:0/3588485642 conn(0x7fc468063b50 msgr2=0x7fc4680b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:48.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.227+0000 7fc46e80d700 1 -- 172.21.15.60:0/3588485642 shutdown_connections 2022-06-29T16:01:48.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.227+0000 7fc46e80d700 1 -- 172.21.15.60:0/3588485642 wait complete. 2022-06-29T16:01:48.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.227+0000 7fc46e80d700 1 Processor -- start 2022-06-29T16:01:48.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.227+0000 7fc46e80d700 1 -- start start 2022-06-29T16:01:48.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.227+0000 7fc46e80d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc468060b60 0x7fc468155410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:48.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.227+0000 7fc46e80d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4680adc80 0x7fc468065bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:48.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.227+0000 7fc46e80d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4680b61d0 0x7fc4680b3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:48.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.227+0000 7fc46e80d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc468155900 con 0x7fc4680b61d0 2022-06-29T16:01:48.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.227+0000 7fc46e80d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4680660e0 con 0x7fc468060b60 2022-06-29T16:01:48.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.227+0000 7fc46e80d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc468155b40 con 0x7fc4680adc80 2022-06-29T16:01:48.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.227+0000 7fc46cdaa700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4680b61d0 0x7fc4680b3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:48.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.227+0000 7fc46cdaa700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4680b61d0 0x7fc4680b3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47650/0 (socket says 172.21.15.60:47650) 2022-06-29T16:01:48.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.227+0000 7fc46cdaa700 1 -- 172.21.15.60:0/1544978304 learned_addr learned my addr 172.21.15.60:0/1544978304 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:48.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.227+0000 7fc467fff700 1 --2- 172.21.15.60:0/1544978304 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc468060b60 0x7fc468155410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:48.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.227+0000 7fc4677fe700 1 --2- 172.21.15.60:0/1544978304 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4680adc80 0x7fc468065bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:48.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.231+0000 7fc46cdaa700 1 -- 172.21.15.60:0/1544978304 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4680adc80 msgr2=0x7fc468065bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:48.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.231+0000 7fc46cdaa700 1 --2- 172.21.15.60:0/1544978304 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4680adc80 0x7fc468065bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:48.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.231+0000 7fc46cdaa700 1 -- 172.21.15.60:0/1544978304 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc468060b60 msgr2=0x7fc468155410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:48.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.231+0000 7fc46cdaa700 1 --2- 172.21.15.60:0/1544978304 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc468060b60 0x7fc468155410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:48.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.231+0000 7fc46cdaa700 1 -- 172.21.15.60:0/1544978304 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc45800b180 con 0x7fc4680b61d0 2022-06-29T16:01:48.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.231+0000 7fc46cdaa700 1 --2- 172.21.15.60:0/1544978304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4680b61d0 0x7fc4680b3680 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fc460001490 tx=0x7fc460007e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:48.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.231+0000 7fc4657fa700 1 -- 172.21.15.60:0/1544978304 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc460005ba0 con 0x7fc4680b61d0 2022-06-29T16:01:48.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.231+0000 7fc4657fa700 1 -- 172.21.15.60:0/1544978304 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc460005d00 con 0x7fc4680b61d0 2022-06-29T16:01:48.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.231+0000 7fc46e80d700 1 -- 172.21.15.60:0/1544978304 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc468152c30 con 0x7fc4680b61d0 2022-06-29T16:01:48.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.231+0000 7fc4657fa700 1 -- 172.21.15.60:0/1544978304 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc4600072c0 con 0x7fc4680b61d0 2022-06-29T16:01:48.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.231+0000 7fc46e80d700 1 -- 172.21.15.60:0/1544978304 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc468064580 con 0x7fc4680b61d0 2022-06-29T16:01:48.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.231+0000 7fc46e80d700 1 -- 172.21.15.60:0/1544978304 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4680b5a40 con 0x7fc4680b61d0 2022-06-29T16:01:48.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.231+0000 7fc4657fa700 1 -- 172.21.15.60:0/1544978304 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc460010070 con 0x7fc4680b61d0 2022-06-29T16:01:48.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.231+0000 7fc4657fa700 1 --2- 172.21.15.60:0/1544978304 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc448053a70 0x7fc448055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:48.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.231+0000 7fc4657fa700 1 -- 172.21.15.60:0/1544978304 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fc46006b590 con 0x7fc4680b61d0 2022-06-29T16:01:48.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.231+0000 7fc467fff700 1 --2- 172.21.15.60:0/1544978304 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc448053a70 0x7fc448055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:48.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.235+0000 7fc467fff700 1 --2- 172.21.15.60:0/1544978304 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc448053a70 0x7fc448055f30 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fc45801a000 tx=0x7fc458014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:48.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.239+0000 7fc4657fa700 1 -- 172.21.15.60:0/1544978304 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc460042680 con 0x7fc4680b61d0 2022-06-29T16:01:48.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.367+0000 7fc46e80d700 1 -- 172.21.15.60:0/1544978304 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc468064210 con 0x7fc4680b61d0 2022-06-29T16:01:48.382 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.375+0000 7fc4657fa700 1 -- 172.21.15.60:0/1544978304 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v6)=0 v6) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc460045ca0 con 0x7fc4680b61d0 2022-06-29T16:01:48.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.375+0000 7fc46e80d700 1 -- 172.21.15.60:0/1544978304 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc448053a70 msgr2=0x7fc448055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:48.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.375+0000 7fc46e80d700 1 --2- 172.21.15.60:0/1544978304 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc448053a70 0x7fc448055f30 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fc45801a000 tx=0x7fc458014000).stop 2022-06-29T16:01:48.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.375+0000 7fc46e80d700 1 -- 172.21.15.60:0/1544978304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4680b61d0 msgr2=0x7fc4680b3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:48.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.375+0000 7fc46e80d700 1 --2- 172.21.15.60:0/1544978304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4680b61d0 0x7fc4680b3680 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fc460001490 tx=0x7fc460007e00).stop 2022-06-29T16:01:48.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.375+0000 7fc46e80d700 1 -- 172.21.15.60:0/1544978304 shutdown_connections 2022-06-29T16:01:48.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.375+0000 7fc46e80d700 1 --2- 172.21.15.60:0/1544978304 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc448053a70 0x7fc448055f30 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:48.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.375+0000 7fc46e80d700 1 --2- 172.21.15.60:0/1544978304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc4680b61d0 0x7fc4680b3680 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:48.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.375+0000 7fc46e80d700 1 --2- 172.21.15.60:0/1544978304 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4680adc80 0x7fc468065bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:48.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.375+0000 7fc46e80d700 1 --2- 172.21.15.60:0/1544978304 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc468060b60 0x7fc468155410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:48.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.375+0000 7fc46e80d700 1 -- 172.21.15.60:0/1544978304 >> 172.21.15.60:0/1544978304 conn(0x7fc468063b50 msgr2=0x7fc4680b2950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:48.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.375+0000 7fc46e80d700 1 -- 172.21.15.60:0/1544978304 shutdown_connections 2022-06-29T16:01:48.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.375+0000 7fc46e80d700 1 -- 172.21.15.60:0/1544978304 wait complete. 2022-06-29T16:01:48.398 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-06-29T16:01:48.399 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:01:48.413 DEBUG:teuthology.orchestra.run.smithi092:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:01:48.435 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:01:48.456 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:01:48.477 DEBUG:teuthology.orchestra.run.smithi197:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:01:48.498 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-06-29T16:01:48.499 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:01:48.499 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:01:48.500 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:01:48.507 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-06-29T16:01:48.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:01:48.509 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-06-29T16:01:48.509 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:01:48.510 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:01:48.510 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:01:48.525 INFO:teuthology.orchestra.run.smithi156.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-06-29T16:01:48.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:01:48.526 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-06-29T16:01:48.527 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:01:48.527 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:01:48.528 DEBUG:teuthology.orchestra.run.smithi197:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T16:01:48.535 INFO:teuthology.orchestra.run.smithi197.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-06-29T16:01:48.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:01:48.536 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-06-29T16:01:48.537 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-06-29T16:01:48.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.643+0000 7fd98359e700 1 Processor -- start 2022-06-29T16:01:48.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.643+0000 7fd98359e700 1 -- start start 2022-06-29T16:01:48.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.643+0000 7fd98359e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd98405d070 0x7fd9840b1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:48.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.643+0000 7fd98359e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd9840aee30 0x7fd984141b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:48.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.643+0000 7fd98359e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd98405d450 con 0x7fd98413a460 2022-06-29T16:01:48.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.643+0000 7fd98359e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd984143a90 con 0x7fd9840aee30 2022-06-29T16:01:48.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.643+0000 7fd98359e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd984062200 con 0x7fd98405d070 2022-06-29T16:01:48.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.643+0000 7fd981d9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd9840aee30 0x7fd984141b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:48.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.643+0000 7fd98259c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd98405d070 0x7fd9840b1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:48.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.643+0000 7fd981d9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd9840aee30 0x7fd984141b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47658/0 (socket says 172.21.15.60:47658) 2022-06-29T16:01:48.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.643+0000 7fd981d9b700 1 -- 172.21.15.60:0/1287717771 learned_addr learned my addr 172.21.15.60:0/1287717771 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:48.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.643+0000 7fd981d9b700 1 -- 172.21.15.60:0/1287717771 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd98405d070 msgr2=0x7fd9840b1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:01:48.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.643+0000 7fd981d9b700 1 --2- 172.21.15.60:0/1287717771 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd98405d070 0x7fd9840b1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:48.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.643+0000 7fd981d9b700 1 -- 172.21.15.60:0/1287717771 >> v1:172.21.15.92:6789/0 conn(0x7fd98413a460 legacy=0x7fd984140bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:48.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.643+0000 7fd981d9b700 1 -- 172.21.15.60:0/1287717771 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd984064dc0 con 0x7fd9840aee30 2022-06-29T16:01:48.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.643+0000 7fd981d9b700 1 --2- 172.21.15.60:0/1287717771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9840aee30 0x7fd984141b70 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fd9740032f0 tx=0x7fd974009290).ready entity=mon.0 client_cookie=cbfc387b07d2a648 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:48.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.643+0000 7fd98159a700 1 -- 172.21.15.60:0/1287717771 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd97400b8a0 con 0x7fd9840aee30 2022-06-29T16:01:48.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.643+0000 7fd98159a700 1 -- 172.21.15.60:0/1287717771 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd97400ba00 con 0x7fd9840aee30 2022-06-29T16:01:48.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.643+0000 7fd98159a700 1 -- 172.21.15.60:0/1287717771 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd97400bb70 con 0x7fd9840aee30 2022-06-29T16:01:48.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.647+0000 7fd98359e700 1 -- 172.21.15.60:0/1287717771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9840aee30 msgr2=0x7fd984141b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:48.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.647+0000 7fd98359e700 1 --2- 172.21.15.60:0/1287717771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9840aee30 0x7fd984141b70 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fd9740032f0 tx=0x7fd974009290).stop 2022-06-29T16:01:48.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.647+0000 7fd98359e700 1 -- 172.21.15.60:0/1287717771 shutdown_connections 2022-06-29T16:01:48.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.647+0000 7fd98359e700 1 --2- 172.21.15.60:0/1287717771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9840aee30 0x7fd984141b70 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:48.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.647+0000 7fd98359e700 1 --2- 172.21.15.60:0/1287717771 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd98405d070 0x7fd9840b1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:48.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.647+0000 7fd98359e700 1 -- 172.21.15.60:0/1287717771 >> 172.21.15.60:0/1287717771 conn(0x7fd9840b5890 msgr2=0x7fd9840b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:48.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.647+0000 7fd98359e700 1 -- 172.21.15.60:0/1287717771 shutdown_connections 2022-06-29T16:01:48.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.647+0000 7fd98359e700 1 -- 172.21.15.60:0/1287717771 wait complete. 2022-06-29T16:01:48.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.647+0000 7fd98359e700 1 Processor -- start 2022-06-29T16:01:48.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.647+0000 7fd98359e700 1 -- start start 2022-06-29T16:01:48.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.651+0000 7fd98359e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd98405d070 0x7fd98413abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:48.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.651+0000 7fd98359e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9840aee30 0x7fd98413fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:48.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.651+0000 7fd98359e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd98413a460 0x7fd9840af720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:48.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.651+0000 7fd98359e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98413e790 con 0x7fd98413a460 2022-06-29T16:01:48.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.651+0000 7fd98359e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98413ea30 con 0x7fd9840aee30 2022-06-29T16:01:48.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.651+0000 7fd98359e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd984066b30 con 0x7fd98405d070 2022-06-29T16:01:48.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.651+0000 7fd982d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd98413a460 0x7fd9840af720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:48.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.651+0000 7fd982d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd98413a460 0x7fd9840af720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47662/0 (socket says 172.21.15.60:47662) 2022-06-29T16:01:48.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.651+0000 7fd982d9d700 1 -- 172.21.15.60:0/3118741370 learned_addr learned my addr 172.21.15.60:0/3118741370 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:48.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.651+0000 7fd98259c700 1 --2- 172.21.15.60:0/3118741370 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd98405d070 0x7fd98413abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:48.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.651+0000 7fd981d9b700 1 --2- 172.21.15.60:0/3118741370 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9840aee30 0x7fd98413fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:48.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.651+0000 7fd982d9d700 1 -- 172.21.15.60:0/3118741370 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd98405d070 msgr2=0x7fd98413abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:48.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.651+0000 7fd982d9d700 1 --2- 172.21.15.60:0/3118741370 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd98405d070 0x7fd98413abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:48.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.651+0000 7fd982d9d700 1 -- 172.21.15.60:0/3118741370 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9840aee30 msgr2=0x7fd98413fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:48.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.651+0000 7fd982d9d700 1 --2- 172.21.15.60:0/3118741370 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9840aee30 0x7fd98413fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:48.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.651+0000 7fd982d9d700 1 -- 172.21.15.60:0/3118741370 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd974009530 con 0x7fd98413a460 2022-06-29T16:01:48.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.651+0000 7fd982d9d700 1 --2- 172.21.15.60:0/3118741370 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd98413a460 0x7fd9840af720 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fd984061990 tx=0x7fd978007c20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:48.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.651+0000 7fd9737fe700 1 -- 172.21.15.60:0/3118741370 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd97800ce30 con 0x7fd98413a460 2022-06-29T16:01:48.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.651+0000 7fd9737fe700 1 -- 172.21.15.60:0/3118741370 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd978007240 con 0x7fd98413a460 2022-06-29T16:01:48.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.651+0000 7fd98359e700 1 -- 172.21.15.60:0/3118741370 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd984140310 con 0x7fd98413a460 2022-06-29T16:01:48.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.651+0000 7fd9737fe700 1 -- 172.21.15.60:0/3118741370 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd978006510 con 0x7fd98413a460 2022-06-29T16:01:48.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.651+0000 7fd98359e700 1 -- 172.21.15.60:0/3118741370 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd984153ba0 con 0x7fd98413a460 2022-06-29T16:01:48.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.651+0000 7fd98359e700 1 -- 172.21.15.60:0/3118741370 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9840b0ef0 con 0x7fd98413a460 2022-06-29T16:01:48.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.651+0000 7fd9737fe700 1 -- 172.21.15.60:0/3118741370 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd978010030 con 0x7fd98413a460 2022-06-29T16:01:48.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.655+0000 7fd9737fe700 1 --2- 172.21.15.60:0/3118741370 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd9600539e0 0x7fd960055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:48.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.655+0000 7fd9737fe700 1 -- 172.21.15.60:0/3118741370 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fd97806b910 con 0x7fd98413a460 2022-06-29T16:01:48.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.655+0000 7fd98259c700 1 --2- 172.21.15.60:0/3118741370 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd9600539e0 0x7fd960055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:48.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.655+0000 7fd98259c700 1 --2- 172.21.15.60:0/3118741370 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd9600539e0 0x7fd960055ea0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fd97c00b000 tx=0x7fd97c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:48.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.659+0000 7fd9737fe700 1 -- 172.21.15.60:0/3118741370 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd978042a70 con 0x7fd98413a460 2022-06-29T16:01:48.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.787+0000 7fd98359e700 1 -- 172.21.15.60:0/3118741370 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fd98413fb10 con 0x7fd98413a460 2022-06-29T16:01:48.797 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.787+0000 7fd9737fe700 1 -- 172.21.15.60:0/3118741370 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v46) v1 ==== 81+0+28827 (secure 0 0 0) 0x7fd98413fb10 con 0x7fd98413a460 2022-06-29T16:01:48.797 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:01:48.798 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:01:48.798 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 46, 2022-06-29T16:01:48.798 INFO:teuthology.orchestra.run.smithi060.stdout: "fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", 2022-06-29T16:01:48.799 INFO:teuthology.orchestra.run.smithi060.stdout: "created": "2022-06-29T15:58:07.965719+0000", 2022-06-29T16:01:48.799 INFO:teuthology.orchestra.run.smithi060.stdout: "modified": "2022-06-29T16:01:46.077815+0000", 2022-06-29T16:01:48.799 INFO:teuthology.orchestra.run.smithi060.stdout: "last_up_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:01:48.799 INFO:teuthology.orchestra.run.smithi060.stdout: "last_in_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:01:48.799 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-06-29T16:01:48.800 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_num": 5799936, 2022-06-29T16:01:48.800 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_set": [ 2022-06-29T16:01:48.800 INFO:teuthology.orchestra.run.smithi060.stdout: "pglog_hardlimit", 2022-06-29T16:01:48.800 INFO:teuthology.orchestra.run.smithi060.stdout: "purged_snapdirs", 2022-06-29T16:01:48.801 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_deletes", 2022-06-29T16:01:48.801 INFO:teuthology.orchestra.run.smithi060.stdout: "sortbitwise" 2022-06-29T16:01:48.801 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:01:48.801 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_version": 4, 2022-06-29T16:01:48.802 INFO:teuthology.orchestra.run.smithi060.stdout: "full_ratio": 0.94999998807907104, 2022-06-29T16:01:48.802 INFO:teuthology.orchestra.run.smithi060.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-06-29T16:01:48.802 INFO:teuthology.orchestra.run.smithi060.stdout: "nearfull_ratio": 0.85000002384185791, 2022-06-29T16:01:48.802 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_snapshot": "", 2022-06-29T16:01:48.803 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_max": 8, 2022-06-29T16:01:48.803 INFO:teuthology.orchestra.run.smithi060.stdout: "max_osd": 8, 2022-06-29T16:01:48.803 INFO:teuthology.orchestra.run.smithi060.stdout: "require_min_compat_client": "jewel", 2022-06-29T16:01:48.804 INFO:teuthology.orchestra.run.smithi060.stdout: "min_compat_client": "jewel", 2022-06-29T16:01:48.804 INFO:teuthology.orchestra.run.smithi060.stdout: "require_osd_release": "octopus", 2022-06-29T16:01:48.804 INFO:teuthology.orchestra.run.smithi060.stdout: "pools": [ 2022-06-29T16:01:48.804 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.805 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2022-06-29T16:01:48.805 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "device_health_metrics", 2022-06-29T16:01:48.805 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:10.961127+0000", 2022-06-29T16:01:48.805 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:01:48.805 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:01:48.806 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:01:48.806 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:01:48.806 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:01:48.806 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:01:48.807 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:01:48.807 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:01:48.807 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 1, 2022-06-29T16:01:48.807 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 1, 2022-06-29T16:01:48.808 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 1, 2022-06-29T16:01:48.808 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 1, 2022-06-29T16:01:48.808 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 1, 2022-06-29T16:01:48.808 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:01:48.808 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:01:48.809 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:01:48.809 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:01:48.809 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:01:48.809 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:01:48.809 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:01:48.810 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.810 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "15", 2022-06-29T16:01:48.810 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:01:48.810 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:01:48.811 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:01:48.811 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:01:48.811 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:01:48.811 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:01:48.812 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:01:48.812 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:01:48.812 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:01:48.812 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:01:48.813 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:01:48.813 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:01:48.813 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:01:48.814 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:01:48.814 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:01:48.814 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:01:48.814 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:01:48.815 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:01:48.815 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:01:48.815 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:01:48.815 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:01:48.816 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:01:48.816 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:01:48.816 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:01:48.816 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:01:48.817 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:01:48.817 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.817 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:01:48.817 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:01:48.817 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:01:48.818 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:01:48.818 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:01:48.818 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:01:48.818 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:01:48.819 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:01:48.819 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:01:48.819 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:01:48.819 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:01:48.820 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:01:48.820 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 1 2022-06-29T16:01:48.820 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.820 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:01:48.820 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_devicehealth": {} 2022-06-29T16:01:48.821 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:48.821 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.821 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.821 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 2, 2022-06-29T16:01:48.822 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "rbd", 2022-06-29T16:01:48.822 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:19.184276+0000", 2022-06-29T16:01:48.822 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:01:48.822 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:01:48.823 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:01:48.823 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:01:48.823 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:01:48.823 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:01:48.824 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:01:48.824 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:01:48.824 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:01:48.824 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:01:48.825 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:01:48.825 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:01:48.825 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:01:48.825 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:01:48.826 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:01:48.826 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:01:48.826 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:01:48.826 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:01:48.826 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:01:48.827 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:01:48.827 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.827 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "18", 2022-06-29T16:01:48.827 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:01:48.828 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:01:48.828 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:01:48.828 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:01:48.828 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:01:48.829 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:01:48.829 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:01:48.829 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:01:48.829 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:01:48.829 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:01:48.830 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:01:48.830 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:01:48.830 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:01:48.830 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:01:48.831 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:01:48.831 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:01:48.831 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:01:48.831 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:01:48.832 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:01:48.832 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:01:48.832 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:01:48.832 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:01:48.832 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:01:48.833 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:01:48.833 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:01:48.833 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:01:48.834 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.834 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:01:48.834 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:01:48.835 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:01:48.835 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:01:48.835 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:01:48.835 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:01:48.835 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:01:48.836 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:01:48.836 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:01:48.836 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:01:48.836 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:01:48.837 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:01:48.837 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:01:48.837 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd": {} 2022-06-29T16:01:48.837 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:48.838 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:48.838 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:01:48.838 INFO:teuthology.orchestra.run.smithi060.stdout: "osds": [ 2022-06-29T16:01:48.838 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.839 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:01:48.839 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "11e7ab9e-fce2-470f-8195-ea4c3c52df22", 2022-06-29T16:01:48.839 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:01:48.839 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:01:48.839 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:01:48.840 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:01:48.840 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:01:48.840 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:01:48.840 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:01:48.841 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 40, 2022-06-29T16:01:48.841 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:01:48.841 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:01:48.841 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:01:48.841 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:48.842 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.842 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:48.842 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6816", 2022-06-29T16:01:48.842 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:01:48.843 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.843 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.843 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:48.843 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6825", 2022-06-29T16:01:48.843 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:01:48.844 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:48.844 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.844 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.844 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:01:48.845 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:48.845 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.845 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:48.846 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6826", 2022-06-29T16:01:48.846 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:01:48.846 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.846 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.847 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:48.847 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6827", 2022-06-29T16:01:48.847 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:01:48.848 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:48.848 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.848 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.848 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:01:48.849 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:48.849 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.849 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:48.849 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6830", 2022-06-29T16:01:48.850 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:01:48.850 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.850 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.850 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:48.851 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6831", 2022-06-29T16:01:48.851 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:01:48.851 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:48.851 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.851 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.852 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:01:48.852 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:48.852 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.852 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:48.853 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6828", 2022-06-29T16:01:48.853 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:01:48.853 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.853 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.853 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:48.854 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6829", 2022-06-29T16:01:48.854 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:01:48.854 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:48.854 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.855 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.855 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6825/12890", 2022-06-29T16:01:48.855 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6827/12890", 2022-06-29T16:01:48.856 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6831/12890", 2022-06-29T16:01:48.856 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6829/12890", 2022-06-29T16:01:48.856 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:01:48.856 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:01:48.856 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:01:48.857 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.857 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.857 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.857 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:01:48.858 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "ca43f136-6f69-4431-a157-fbe073fb585c", 2022-06-29T16:01:48.858 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:01:48.858 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:01:48.858 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:01:48.859 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:01:48.859 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:01:48.859 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:01:48.859 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:01:48.859 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 41, 2022-06-29T16:01:48.860 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:01:48.860 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:01:48.860 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:01:48.860 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:48.861 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.861 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:48.861 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6815", 2022-06-29T16:01:48.861 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:01:48.862 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.862 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.862 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:48.862 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6818", 2022-06-29T16:01:48.862 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:01:48.863 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:48.863 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.863 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.863 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:01:48.864 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:48.864 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.864 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:48.864 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6819", 2022-06-29T16:01:48.864 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:01:48.865 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.865 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.865 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:48.865 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6820", 2022-06-29T16:01:48.866 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:01:48.866 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:48.866 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.867 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.867 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:01:48.867 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:48.867 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.868 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:48.868 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6823", 2022-06-29T16:01:48.868 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:01:48.868 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.869 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.869 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:48.869 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6824", 2022-06-29T16:01:48.869 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:01:48.869 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:48.870 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.870 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.870 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:01:48.870 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:48.871 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.871 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:48.871 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6821", 2022-06-29T16:01:48.871 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:01:48.872 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.872 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.872 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:48.872 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6822", 2022-06-29T16:01:48.873 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:01:48.873 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:48.873 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.873 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.873 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6818/12892", 2022-06-29T16:01:48.874 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6820/12892", 2022-06-29T16:01:48.874 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6824/12892", 2022-06-29T16:01:48.874 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6822/12892", 2022-06-29T16:01:48.874 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:01:48.875 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:01:48.875 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:01:48.875 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.875 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.876 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.876 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:01:48.876 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "74fac45f-5f07-44ec-a9cb-2644c123fd4a", 2022-06-29T16:01:48.876 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:01:48.877 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:01:48.877 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:01:48.877 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:01:48.877 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:01:48.878 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:01:48.878 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:01:48.878 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 29, 2022-06-29T16:01:48.879 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:01:48.879 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:01:48.879 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:01:48.879 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:48.880 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.880 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:48.880 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6808", 2022-06-29T16:01:48.881 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:01:48.881 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.881 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.881 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:48.881 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6809", 2022-06-29T16:01:48.882 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:01:48.882 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:48.882 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.882 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.883 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:01:48.883 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:48.883 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.883 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:48.883 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6810", 2022-06-29T16:01:48.884 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:01:48.884 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.884 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.884 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:48.885 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6811", 2022-06-29T16:01:48.885 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:01:48.885 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:48.886 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.886 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.886 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:01:48.886 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:48.887 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.887 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:48.887 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6814", 2022-06-29T16:01:48.888 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:01:48.888 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.888 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.888 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:48.889 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6817", 2022-06-29T16:01:48.889 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:01:48.889 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:48.889 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.890 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.890 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:01:48.890 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:48.891 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.891 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:48.891 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6812", 2022-06-29T16:01:48.892 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:01:48.892 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.892 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.892 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:48.892 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6813", 2022-06-29T16:01:48.893 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:01:48.893 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:48.893 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.893 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.894 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6809/12891", 2022-06-29T16:01:48.894 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6811/12891", 2022-06-29T16:01:48.895 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6817/12891", 2022-06-29T16:01:48.895 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6813/12891", 2022-06-29T16:01:48.896 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:01:48.896 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:01:48.897 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:01:48.897 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.898 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.898 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.898 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:01:48.899 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "2581d351-973a-4b4e-85ca-3658f0068765", 2022-06-29T16:01:48.899 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:01:48.900 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:01:48.900 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:01:48.901 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:01:48.901 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:01:48.902 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:01:48.902 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:01:48.903 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 40, 2022-06-29T16:01:48.903 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:01:48.904 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:01:48.904 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:01:48.904 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:48.905 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.905 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:48.906 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6800", 2022-06-29T16:01:48.906 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:01:48.907 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.907 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.907 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:48.908 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6801", 2022-06-29T16:01:48.908 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:01:48.909 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:48.909 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.909 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.910 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:01:48.910 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:48.910 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.911 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:48.911 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6802", 2022-06-29T16:01:48.912 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:01:48.912 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.912 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.913 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:48.913 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6803", 2022-06-29T16:01:48.913 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:01:48.913 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:48.914 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.914 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.914 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:01:48.914 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:48.915 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.915 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:48.915 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6806", 2022-06-29T16:01:48.915 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:01:48.916 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.916 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.916 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:48.916 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6807", 2022-06-29T16:01:48.916 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:01:48.917 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:48.917 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.917 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.917 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:01:48.917 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:48.918 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.918 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:48.918 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6804", 2022-06-29T16:01:48.918 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:01:48.919 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.919 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.919 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:48.919 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6805", 2022-06-29T16:01:48.920 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:01:48.920 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:48.920 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.920 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.920 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6801/12889", 2022-06-29T16:01:48.921 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6803/12889", 2022-06-29T16:01:48.921 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6807/12889", 2022-06-29T16:01:48.921 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6805/12889", 2022-06-29T16:01:48.921 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:01:48.922 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:01:48.922 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:01:48.922 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.923 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.923 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:01:48.923 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "3d234db9-87cd-4fe9-814f-0240a287d8b4", 2022-06-29T16:01:48.923 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:01:48.923 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:01:48.924 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:01:48.924 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:01:48.924 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:01:48.924 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:01:48.925 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:01:48.925 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 40, 2022-06-29T16:01:48.925 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:01:48.925 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:01:48.925 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:01:48.926 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:48.926 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.926 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:48.926 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6808", 2022-06-29T16:01:48.927 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:01:48.927 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.927 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.927 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:48.928 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6810", 2022-06-29T16:01:48.928 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:01:48.928 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:48.928 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.929 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:01:48.929 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:48.929 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.929 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:48.930 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6812", 2022-06-29T16:01:48.930 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:01:48.930 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.930 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.931 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:48.931 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6814", 2022-06-29T16:01:48.931 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:01:48.931 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:48.932 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.932 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.932 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:01:48.932 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:48.933 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.933 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:48.933 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6823", 2022-06-29T16:01:48.933 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:01:48.934 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.934 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.934 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:48.934 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6825", 2022-06-29T16:01:48.935 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:01:48.935 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:48.935 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.936 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:01:48.936 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:48.936 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.936 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:48.937 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6817", 2022-06-29T16:01:48.937 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:01:48.937 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.937 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.938 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:48.938 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6820", 2022-06-29T16:01:48.938 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:01:48.938 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:48.939 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.939 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.939 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6810/13249", 2022-06-29T16:01:48.940 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6814/13249", 2022-06-29T16:01:48.940 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6825/13249", 2022-06-29T16:01:48.940 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6820/13249", 2022-06-29T16:01:48.940 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:01:48.941 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:01:48.941 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:01:48.941 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.941 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.941 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.942 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:01:48.942 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "e4bf1104-ff5c-438b-b608-8cee31c163f6", 2022-06-29T16:01:48.942 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:01:48.942 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:01:48.943 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:01:48.943 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:01:48.943 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:01:48.943 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:01:48.944 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:01:48.944 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 40, 2022-06-29T16:01:48.944 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:01:48.944 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:01:48.945 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:01:48.945 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:48.945 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.945 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:48.946 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6809", 2022-06-29T16:01:48.946 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:01:48.946 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.947 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.947 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:48.947 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6811", 2022-06-29T16:01:48.947 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:01:48.948 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:48.948 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.948 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.948 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:01:48.949 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:48.949 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.949 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:48.949 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6813", 2022-06-29T16:01:48.950 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:01:48.950 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.950 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.950 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:48.951 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6815", 2022-06-29T16:01:48.951 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:01:48.951 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:48.951 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.952 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.952 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:01:48.952 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:48.952 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.953 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:48.953 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6824", 2022-06-29T16:01:48.953 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:01:48.953 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.954 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.954 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:48.954 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6827", 2022-06-29T16:01:48.954 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:01:48.955 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:48.955 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.955 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.955 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:01:48.955 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:48.956 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.956 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:48.956 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6818", 2022-06-29T16:01:48.956 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:01:48.957 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.957 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.957 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:48.957 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6821", 2022-06-29T16:01:48.957 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:01:48.958 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:48.958 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.958 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.958 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6811/13248", 2022-06-29T16:01:48.959 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6815/13248", 2022-06-29T16:01:48.959 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6827/13248", 2022-06-29T16:01:48.959 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6821/13248", 2022-06-29T16:01:48.959 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:01:48.960 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:01:48.960 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:01:48.960 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.961 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.961 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:01:48.961 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "8564dd25-27ab-4203-b54e-af6d3aa05857", 2022-06-29T16:01:48.961 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:01:48.962 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:01:48.962 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:01:48.962 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:01:48.962 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:01:48.963 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:01:48.963 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:01:48.964 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 40, 2022-06-29T16:01:48.964 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:01:48.964 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:01:48.964 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:01:48.965 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:48.965 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.965 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:48.965 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6816", 2022-06-29T16:01:48.966 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:01:48.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.966 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:48.967 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6819", 2022-06-29T16:01:48.967 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:01:48.967 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:48.967 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.968 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:01:48.968 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:48.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.968 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:48.969 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6822", 2022-06-29T16:01:48.969 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:01:48.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.970 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:48.970 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6826", 2022-06-29T16:01:48.970 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:01:48.971 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:48.971 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.971 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.971 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:01:48.972 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:48.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.978 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:48.979 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6830", 2022-06-29T16:01:48.979 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:01:48.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.980 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:48.980 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6831", 2022-06-29T16:01:48.980 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:01:48.980 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:48.981 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.981 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:01:48.982 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:48.982 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.982 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:48.982 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6828", 2022-06-29T16:01:48.983 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:01:48.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.983 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:48.984 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6829", 2022-06-29T16:01:48.984 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:01:48.984 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:48.984 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.985 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6819/13250", 2022-06-29T16:01:48.985 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6826/13250", 2022-06-29T16:01:48.985 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6831/13250", 2022-06-29T16:01:48.985 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6829/13250", 2022-06-29T16:01:48.986 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:01:48.986 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:01:48.986 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:01:48.986 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.987 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.987 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:01:48.987 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "bd513a9c-3b19-4314-9897-d300b5c897bb", 2022-06-29T16:01:48.987 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:01:48.988 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:01:48.988 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:01:48.988 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:01:48.988 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:01:48.989 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:01:48.989 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:01:48.989 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 39, 2022-06-29T16:01:48.989 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:01:48.990 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:01:48.990 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:01:48.990 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:48.990 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.991 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:48.991 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6800", 2022-06-29T16:01:48.991 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:01:48.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.992 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.992 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:48.992 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6801", 2022-06-29T16:01:48.993 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:01:48.993 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:48.993 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.993 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.994 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:01:48.994 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:48.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.994 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:48.995 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6802", 2022-06-29T16:01:48.995 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:01:48.995 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.995 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.996 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:48.996 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6803", 2022-06-29T16:01:48.996 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:01:48.996 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:48.997 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:48.997 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.997 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:01:48.997 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:48.997 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.998 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:48.998 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6806", 2022-06-29T16:01:48.998 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:01:48.998 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:48.999 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:48.999 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:48.999 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6807", 2022-06-29T16:01:48.999 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:01:49.000 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:49.000 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:49.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:49.000 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:01:49.000 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:01:49.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:49.001 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:01:49.001 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6804", 2022-06-29T16:01:49.001 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:01:49.002 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:49.002 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:49.002 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:01:49.003 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6805", 2022-06-29T16:01:49.003 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:01:49.003 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:49.003 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:49.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:49.004 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6801/13247", 2022-06-29T16:01:49.004 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6803/13247", 2022-06-29T16:01:49.004 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6807/13247", 2022-06-29T16:01:49.005 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6805/13247", 2022-06-29T16:01:49.005 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:01:49.005 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:01:49.005 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:01:49.006 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:49.006 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:49.006 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:01:49.006 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_xinfo": [ 2022-06-29T16:01:49.006 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:49.007 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:01:49.007 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:01:49.007 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:01:49.007 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:01:49.008 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:01:49.008 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:01:49.008 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.891342+0000", 2022-06-29T16:01:49.008 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:01:49.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:49.009 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:49.009 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:01:49.009 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:01:49.010 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:01:49.010 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:01:49.010 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:01:49.010 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:01:49.010 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.900866+0000", 2022-06-29T16:01:49.011 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:01:49.011 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:49.011 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:49.011 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:01:49.012 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:01:49.012 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:01:49.012 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:01:49.012 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:01:49.012 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:01:49.013 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.912755+0000", 2022-06-29T16:01:49.013 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:01:49.013 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:49.014 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:49.014 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:01:49.014 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:01:49.014 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:01:49.015 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:01:49.015 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:01:49.015 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:01:49.015 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.864332+0000", 2022-06-29T16:01:49.016 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:01:49.016 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:49.016 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:49.016 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:01:49.016 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:01:49.017 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:01:49.017 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:01:49.017 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:01:49.017 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:01:49.018 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.863756+0000", 2022-06-29T16:01:49.018 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:01:49.018 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:49.018 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:49.019 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:01:49.019 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:01:49.019 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:01:49.019 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:01:49.019 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:01:49.020 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:01:49.020 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.869041+0000", 2022-06-29T16:01:49.020 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:01:49.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:49.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:49.021 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:01:49.021 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:01:49.021 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:01:49.022 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:01:49.022 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:01:49.022 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:01:49.022 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.895043+0000", 2022-06-29T16:01:49.022 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:01:49.023 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:49.023 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:49.023 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:01:49.023 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:01:49.024 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:01:49.024 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:01:49.024 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:01:49.024 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:01:49.025 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.889491+0000", 2022-06-29T16:01:49.025 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:01:49.025 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:49.025 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:01:49.026 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap": [], 2022-06-29T16:01:49.026 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap_items": [], 2022-06-29T16:01:49.026 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_temp": [], 2022-06-29T16:01:49.026 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_temp": [], 2022-06-29T16:01:49.026 INFO:teuthology.orchestra.run.smithi060.stdout: "blacklist": { 2022-06-29T16:01:49.027 INFO:teuthology.orchestra.run.smithi060.stdout: "172.21.15.60:6833/1964426267": "2022-06-30T16:01:42.280366+0000", 2022-06-29T16:01:49.027 INFO:teuthology.orchestra.run.smithi060.stdout: "172.21.15.60:6832/1964426267": "2022-06-30T16:01:42.280366+0000" 2022-06-29T16:01:49.027 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:49.027 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profiles": { 2022-06-29T16:01:49.028 INFO:teuthology.orchestra.run.smithi060.stdout: "default": { 2022-06-29T16:01:49.028 INFO:teuthology.orchestra.run.smithi060.stdout: "crush-failure-domain": "osd", 2022-06-29T16:01:49.028 INFO:teuthology.orchestra.run.smithi060.stdout: "k": "2", 2022-06-29T16:01:49.028 INFO:teuthology.orchestra.run.smithi060.stdout: "m": "1", 2022-06-29T16:01:49.029 INFO:teuthology.orchestra.run.smithi060.stdout: "plugin": "jerasure", 2022-06-29T16:01:49.029 INFO:teuthology.orchestra.run.smithi060.stdout: "ruleset-failure-domain": "osd", 2022-06-29T16:01:49.029 INFO:teuthology.orchestra.run.smithi060.stdout: "technique": "reed_sol_van" 2022-06-29T16:01:49.029 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:49.030 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:49.030 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps_queue": [], 2022-06-29T16:01:49.030 INFO:teuthology.orchestra.run.smithi060.stdout: "new_removed_snaps": [], 2022-06-29T16:01:49.030 INFO:teuthology.orchestra.run.smithi060.stdout: "new_purged_snaps": [], 2022-06-29T16:01:49.030 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_node_flags": {}, 2022-06-29T16:01:49.031 INFO:teuthology.orchestra.run.smithi060.stdout: "device_class_flags": {} 2022-06-29T16:01:49.031 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:01:49.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.791+0000 7fd98359e700 1 -- 172.21.15.60:0/3118741370 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd9600539e0 msgr2=0x7fd960055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:49.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.791+0000 7fd98359e700 1 --2- 172.21.15.60:0/3118741370 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd9600539e0 0x7fd960055ea0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fd97c00b000 tx=0x7fd97c006df0).stop 2022-06-29T16:01:49.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.791+0000 7fd98359e700 1 -- 172.21.15.60:0/3118741370 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd98413a460 msgr2=0x7fd9840af720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:49.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.791+0000 7fd98359e700 1 --2- 172.21.15.60:0/3118741370 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd98413a460 0x7fd9840af720 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fd984061990 tx=0x7fd978007c20).stop 2022-06-29T16:01:49.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.791+0000 7fd98359e700 1 -- 172.21.15.60:0/3118741370 shutdown_connections 2022-06-29T16:01:49.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.791+0000 7fd98359e700 1 --2- 172.21.15.60:0/3118741370 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd9600539e0 0x7fd960055ea0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:49.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.791+0000 7fd98359e700 1 --2- 172.21.15.60:0/3118741370 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd98413a460 0x7fd9840af720 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:49.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.791+0000 7fd98359e700 1 --2- 172.21.15.60:0/3118741370 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd9840aee30 0x7fd98413fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:49.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.791+0000 7fd98359e700 1 --2- 172.21.15.60:0/3118741370 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd98405d070 0x7fd98413abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:49.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.791+0000 7fd98359e700 1 -- 172.21.15.60:0/3118741370 >> 172.21.15.60:0/3118741370 conn(0x7fd9840b5890 msgr2=0x7fd984143c00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:49.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.791+0000 7fd98359e700 1 -- 172.21.15.60:0/3118741370 shutdown_connections 2022-06-29T16:01:49.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:48.791+0000 7fd98359e700 1 -- 172.21.15.60:0/3118741370 wait complete. 2022-06-29T16:01:49.037 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:01:49.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.139+0000 7f572af41700 1 Processor -- start 2022-06-29T16:01:49.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.139+0000 7f572af41700 1 -- start start 2022-06-29T16:01:49.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.139+0000 7f572af41700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f57240b5eb0 0x7f5724142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:49.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.139+0000 7f572af41700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f57240b2c50 0x7f57240b3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:49.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.143+0000 7f572af41700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57240b0cf0 con 0x7f57240b48b0 2022-06-29T16:01:49.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.143+0000 7f572af41700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57240b54e0 con 0x7f57240b2c50 2022-06-29T16:01:49.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.143+0000 7f572af41700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57240b4d10 con 0x7f57240b5eb0 2022-06-29T16:01:49.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.143+0000 7f5729f3f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f57240b5eb0 0x7f5724142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:49.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.143+0000 7f572a740700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f57240b2c50 0x7f57240b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:49.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.143+0000 7f5729f3f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f57240b5eb0 0x7f5724142a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:60762/0 (socket says 172.21.15.60:60762) 2022-06-29T16:01:49.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.143+0000 7f5729f3f700 1 -- 172.21.15.60:0/2887193544 learned_addr learned my addr 172.21.15.60:0/2887193544 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:49.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.143+0000 7f572973e700 1 --1- 172.21.15.60:0/2887193544 >> v1:172.21.15.92:6789/0 conn(0x7f57240b48b0 0x7f57240b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:60004/0 (socket says 172.21.15.60:60004) 2022-06-29T16:01:49.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.143+0000 7f572a740700 1 -- 172.21.15.60:0/2887193544 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f57240b5eb0 msgr2=0x7f5724142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:01:49.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.143+0000 7f572a740700 1 --2- 172.21.15.60:0/2887193544 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f57240b5eb0 0x7f5724142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:49.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.143+0000 7f572a740700 1 -- 172.21.15.60:0/2887193544 >> v1:172.21.15.92:6789/0 conn(0x7f57240b48b0 legacy=0x7f57240b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:49.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.143+0000 7f572a740700 1 -- 172.21.15.60:0/2887193544 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57240b6290 con 0x7f57240b2c50 2022-06-29T16:01:49.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.143+0000 7f572a740700 1 --2- 172.21.15.60:0/2887193544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57240b2c50 0x7f57240b3090 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f571c0032f0 tx=0x7f571c0091d0).ready entity=mon.0 client_cookie=71b8e9027eb2e91 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:49.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.143+0000 7f5728f3d700 1 -- 172.21.15.60:0/2887193544 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f571c00b6c0 con 0x7f57240b2c50 2022-06-29T16:01:49.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.143+0000 7f5728f3d700 1 -- 172.21.15.60:0/2887193544 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f571c00b820 con 0x7f57240b2c50 2022-06-29T16:01:49.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.143+0000 7f5728f3d700 1 -- 172.21.15.60:0/2887193544 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f571c00b990 con 0x7f57240b2c50 2022-06-29T16:01:49.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.143+0000 7f572af41700 1 -- 172.21.15.60:0/2887193544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57240b2c50 msgr2=0x7f57240b3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:49.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.143+0000 7f572af41700 1 --2- 172.21.15.60:0/2887193544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57240b2c50 0x7f57240b3090 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f571c0032f0 tx=0x7f571c0091d0).stop 2022-06-29T16:01:49.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.143+0000 7f572af41700 1 -- 172.21.15.60:0/2887193544 shutdown_connections 2022-06-29T16:01:49.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.143+0000 7f572af41700 1 --2- 172.21.15.60:0/2887193544 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57240b2c50 0x7f57240b3090 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:49.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.143+0000 7f572af41700 1 --2- 172.21.15.60:0/2887193544 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f57240b5eb0 0x7f5724142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:49.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.143+0000 7f572af41700 1 -- 172.21.15.60:0/2887193544 >> 172.21.15.60:0/2887193544 conn(0x7f5724062090 msgr2=0x7f5724062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:49.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.143+0000 7f572af41700 1 -- 172.21.15.60:0/2887193544 shutdown_connections 2022-06-29T16:01:49.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.143+0000 7f572af41700 1 -- 172.21.15.60:0/2887193544 wait complete. 2022-06-29T16:01:49.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.147+0000 7f572af41700 1 Processor -- start 2022-06-29T16:01:49.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.147+0000 7f572af41700 1 -- start start 2022-06-29T16:01:49.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.147+0000 7f572af41700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f57240b2c50 0x7f5724113e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:49.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.147+0000 7f572af41700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57240b48b0 0x7f5724113710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:49.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.147+0000 7f572af41700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f57240b5eb0 0x7f5724151c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:49.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.147+0000 7f572af41700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5724155ac0 con 0x7f57240b48b0 2022-06-29T16:01:49.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.147+0000 7f572af41700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5724145d20 con 0x7f57240b2c50 2022-06-29T16:01:49.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.147+0000 7f572af41700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5724155cc0 con 0x7f57240b5eb0 2022-06-29T16:01:49.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.147+0000 7f572973e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57240b48b0 0x7f5724113710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:49.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.147+0000 7f572a740700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f57240b5eb0 0x7f5724151c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:49.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.147+0000 7f5729f3f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f57240b2c50 0x7f5724113e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:49.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.147+0000 7f572a740700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f57240b5eb0 0x7f5724151c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:60772/0 (socket says 172.21.15.60:60772) 2022-06-29T16:01:49.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.147+0000 7f572973e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57240b48b0 0x7f5724113710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47676/0 (socket says 172.21.15.60:47676) 2022-06-29T16:01:49.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.147+0000 7f572a740700 1 -- 172.21.15.60:0/1545028611 learned_addr learned my addr 172.21.15.60:0/1545028611 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:49.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.147+0000 7f572973e700 1 -- 172.21.15.60:0/1545028611 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f57240b5eb0 msgr2=0x7f5724151c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:49.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.147+0000 7f572973e700 1 --2- 172.21.15.60:0/1545028611 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f57240b5eb0 0x7f5724151c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:49.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.147+0000 7f572973e700 1 -- 172.21.15.60:0/1545028611 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f57240b2c50 msgr2=0x7f5724113e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:49.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.147+0000 7f572973e700 1 --2- 172.21.15.60:0/1545028611 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f57240b2c50 0x7f5724113e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:49.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.147+0000 7f572973e700 1 -- 172.21.15.60:0/1545028611 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f571c009470 con 0x7f57240b48b0 2022-06-29T16:01:49.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.147+0000 7f5729f3f700 1 --2- 172.21.15.60:0/1545028611 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f57240b2c50 0x7f5724113e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:01:49.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.147+0000 7f572973e700 1 --2- 172.21.15.60:0/1545028611 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57240b48b0 0x7f5724113710 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f5724062f00 tx=0x7f5714007b40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:49.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.147+0000 7f5712ffd700 1 -- 172.21.15.60:0/1545028611 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57140070d0 con 0x7f57240b48b0 2022-06-29T16:01:49.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.147+0000 7f5712ffd700 1 -- 172.21.15.60:0/1545028611 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5714007230 con 0x7f57240b48b0 2022-06-29T16:01:49.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.147+0000 7f5712ffd700 1 -- 172.21.15.60:0/1545028611 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5714005740 con 0x7f57240b48b0 2022-06-29T16:01:49.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.147+0000 7f572af41700 1 -- 172.21.15.60:0/1545028611 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5724155f60 con 0x7f57240b48b0 2022-06-29T16:01:49.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.147+0000 7f572af41700 1 -- 172.21.15.60:0/1545028611 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5724145a10 con 0x7f57240b48b0 2022-06-29T16:01:49.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.151+0000 7f572af41700 1 -- 172.21.15.60:0/1545028611 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5724144c10 con 0x7f57240b48b0 2022-06-29T16:01:49.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.151+0000 7f5712ffd700 1 -- 172.21.15.60:0/1545028611 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5714010070 con 0x7f57240b48b0 2022-06-29T16:01:49.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.151+0000 7f5712ffd700 1 --2- 172.21.15.60:0/1545028611 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5704053aa0 0x7f5704055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:49.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.151+0000 7f5712ffd700 1 -- 172.21.15.60:0/1545028611 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f571406afc0 con 0x7f57240b48b0 2022-06-29T16:01:49.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.151+0000 7f5729f3f700 1 --2- 172.21.15.60:0/1545028611 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5704053aa0 0x7f5704055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:49.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.151+0000 7f5729f3f700 1 --2- 172.21.15.60:0/1545028611 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5704053aa0 0x7f5704055f60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f5720008000 tx=0x7f5720008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:49.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.155+0000 7f5712ffd700 1 -- 172.21.15.60:0/1545028611 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f57140420b0 con 0x7f57240b48b0 2022-06-29T16:01:49.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.323+0000 7f572af41700 1 -- 172.21.15.60:0/1545028611 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57240b0cf0 con 0x7f57240b48b0 2022-06-29T16:01:49.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.323+0000 7f5712ffd700 1 -- 172.21.15.60:0/1545028611 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+1209 (secure 0 0 0) 0x7f57240b0cf0 con 0x7f57240b48b0 2022-06-29T16:01:49.331 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:01:49.331 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":17,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4937,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/1673184114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1673184114},{"type":"v1","addr":"172.21.15.92:6835","nonce":1673184114}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":13},{"gid":5496,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1034264882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1034264882},{"type":"v1","addr":"172.21.15.60:6833","nonce":1034264882}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":17}],"filesystems":[]} 2022-06-29T16:01:49.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.323+0000 7f572af41700 1 -- 172.21.15.60:0/1545028611 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5704053aa0 msgr2=0x7f5704055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:49.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.323+0000 7f572af41700 1 --2- 172.21.15.60:0/1545028611 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5704053aa0 0x7f5704055f60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f5720008000 tx=0x7f5720008040).stop 2022-06-29T16:01:49.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.323+0000 7f572af41700 1 -- 172.21.15.60:0/1545028611 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57240b48b0 msgr2=0x7f5724113710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:49.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.323+0000 7f572af41700 1 --2- 172.21.15.60:0/1545028611 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57240b48b0 0x7f5724113710 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f5724062f00 tx=0x7f5714007b40).stop 2022-06-29T16:01:49.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.323+0000 7f572af41700 1 -- 172.21.15.60:0/1545028611 shutdown_connections 2022-06-29T16:01:49.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.323+0000 7f572af41700 1 --2- 172.21.15.60:0/1545028611 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5704053aa0 0x7f5704055f60 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:49.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.323+0000 7f572af41700 1 --2- 172.21.15.60:0/1545028611 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f57240b5eb0 0x7f5724151c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:49.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.323+0000 7f572af41700 1 --2- 172.21.15.60:0/1545028611 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f57240b48b0 0x7f5724113710 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:49.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.323+0000 7f572af41700 1 --2- 172.21.15.60:0/1545028611 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f57240b2c50 0x7f5724113e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:49.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.323+0000 7f572af41700 1 -- 172.21.15.60:0/1545028611 >> 172.21.15.60:0/1545028611 conn(0x7f5724062090 msgr2=0x7f57241430d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:49.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.323+0000 7f572af41700 1 -- 172.21.15.60:0/1545028611 shutdown_connections 2022-06-29T16:01:49.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.323+0000 7f572af41700 1 -- 172.21.15.60:0/1545028611 wait complete. 2022-06-29T16:01:49.336 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 17 2022-06-29T16:01:49.347 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:01:49.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.439+0000 7f4ad0db6700 1 Processor -- start 2022-06-29T16:01:49.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.439+0000 7f4ad0db6700 1 -- start start 2022-06-29T16:01:49.448 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.439+0000 7f4ad0db6700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4acc0b6a50 0x7f4acc05fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:49.449 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.439+0000 7f4ad0db6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4acc146cf0 0x7f4acc0aaf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:49.449 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.439+0000 7f4ad0db6700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4acc0b6e30 con 0x7f4acc062230 2022-06-29T16:01:49.449 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.439+0000 7f4ad0db6700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4acc098970 con 0x7f4acc0b6a50 2022-06-29T16:01:49.449 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.439+0000 7f4ad0db6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4acc05fa40 con 0x7f4acc146cf0 2022-06-29T16:01:49.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.439+0000 7f4aca59c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4acc0b6a50 0x7f4acc05fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:49.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.439+0000 7f4aca59c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4acc0b6a50 0x7f4acc05fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47684/0 (socket says 172.21.15.60:47684) 2022-06-29T16:01:49.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.439+0000 7f4aca59c700 1 -- 172.21.15.60:0/3998511248 learned_addr learned my addr 172.21.15.60:0/3998511248 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:49.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.439+0000 7f4acad9d700 1 --2- 172.21.15.60:0/3998511248 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4acc146cf0 0x7f4acc0aaf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:49.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.439+0000 7f4aca59c700 1 -- 172.21.15.60:0/3998511248 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4acc146cf0 msgr2=0x7f4acc0aaf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:01:49.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.439+0000 7f4aca59c700 1 --2- 172.21.15.60:0/3998511248 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4acc146cf0 0x7f4acc0aaf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:49.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.439+0000 7f4aca59c700 1 -- 172.21.15.60:0/3998511248 >> v1:172.21.15.92:6789/0 conn(0x7f4acc062230 legacy=0x7f4acc1465e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:49.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.439+0000 7f4aca59c700 1 -- 172.21.15.60:0/3998511248 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4acc05f0b0 con 0x7f4acc0b6a50 2022-06-29T16:01:49.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.439+0000 7f4aca59c700 1 --2- 172.21.15.60:0/3998511248 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4acc0b6a50 0x7f4acc05fee0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f4ac0002f70 tx=0x7f4ac0006eb0).ready entity=mon.0 client_cookie=cd376c122e55d631 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:49.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.439+0000 7f4ac959a700 1 -- 172.21.15.60:0/3998511248 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ac000d570 con 0x7f4acc0b6a50 2022-06-29T16:01:49.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.439+0000 7f4ac959a700 1 -- 172.21.15.60:0/3998511248 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ac0000e90 con 0x7f4acc0b6a50 2022-06-29T16:01:49.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.439+0000 7f4ac959a700 1 -- 172.21.15.60:0/3998511248 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ac000d900 con 0x7f4acc0b6a50 2022-06-29T16:01:49.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.439+0000 7f4ad0db6700 1 -- 172.21.15.60:0/3998511248 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4acc0b6a50 msgr2=0x7f4acc05fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:49.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.439+0000 7f4ad0db6700 1 --2- 172.21.15.60:0/3998511248 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4acc0b6a50 0x7f4acc05fee0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f4ac0002f70 tx=0x7f4ac0006eb0).stop 2022-06-29T16:01:49.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.439+0000 7f4ad0db6700 1 -- 172.21.15.60:0/3998511248 shutdown_connections 2022-06-29T16:01:49.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.439+0000 7f4ad0db6700 1 --2- 172.21.15.60:0/3998511248 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4acc146cf0 0x7f4acc0aaf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:49.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.439+0000 7f4ad0db6700 1 --2- 172.21.15.60:0/3998511248 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4acc0b6a50 0x7f4acc05fee0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:49.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.439+0000 7f4ad0db6700 1 -- 172.21.15.60:0/3998511248 >> 172.21.15.60:0/3998511248 conn(0x7f4acc0643e0 msgr2=0x7f4acc0b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:49.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.439+0000 7f4ad0db6700 1 -- 172.21.15.60:0/3998511248 shutdown_connections 2022-06-29T16:01:49.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.439+0000 7f4ad0db6700 1 -- 172.21.15.60:0/3998511248 wait complete. 2022-06-29T16:01:49.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.443+0000 7f4ad0db6700 1 Processor -- start 2022-06-29T16:01:49.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.443+0000 7f4ad0db6700 1 -- start start 2022-06-29T16:01:49.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.443+0000 7f4ad0db6700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4acc062230 0x7f4acc14a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:49.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.443+0000 7f4ad0db6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4acc0b6a50 0x7f4acc098fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:49.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.443+0000 7f4ad0db6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4acc146cf0 0x7f4acc0994c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:49.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.443+0000 7f4ad0db6700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4acc0999b0 con 0x7f4acc0b6a50 2022-06-29T16:01:49.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.443+0000 7f4ad0db6700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4acc14af90 con 0x7f4acc062230 2022-06-29T16:01:49.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.443+0000 7f4ad0db6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4acc14b230 con 0x7f4acc146cf0 2022-06-29T16:01:49.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.443+0000 7f4ac9d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4acc0b6a50 0x7f4acc098fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:49.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.443+0000 7f4ac9d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4acc0b6a50 0x7f4acc098fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47688/0 (socket says 172.21.15.60:47688) 2022-06-29T16:01:49.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.443+0000 7f4ac9d9b700 1 -- 172.21.15.60:0/1767557551 learned_addr learned my addr 172.21.15.60:0/1767557551 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:49.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.443+0000 7f4acad9d700 1 --2- 172.21.15.60:0/1767557551 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4acc146cf0 0x7f4acc0994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:49.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.443+0000 7f4aca59c700 1 --2- 172.21.15.60:0/1767557551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4acc062230 0x7f4acc14a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:49.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.443+0000 7f4ac9d9b700 1 -- 172.21.15.60:0/1767557551 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4acc146cf0 msgr2=0x7f4acc0994c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:49.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.443+0000 7f4ac9d9b700 1 --2- 172.21.15.60:0/1767557551 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4acc146cf0 0x7f4acc0994c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:49.461 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.443+0000 7f4ac9d9b700 1 -- 172.21.15.60:0/1767557551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4acc062230 msgr2=0x7f4acc14a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:49.461 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.443+0000 7f4ac9d9b700 1 --2- 172.21.15.60:0/1767557551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4acc062230 0x7f4acc14a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:49.461 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.443+0000 7f4ac9d9b700 1 -- 172.21.15.60:0/1767557551 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac000b1e0 con 0x7f4acc0b6a50 2022-06-29T16:01:49.461 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.443+0000 7f4ac9d9b700 1 --2- 172.21.15.60:0/1767557551 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4acc0b6a50 0x7f4acc098fd0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f4acc0b3d60 tx=0x7f4ac40014a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:49.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.447+0000 7f4abb7fe700 1 -- 172.21.15.60:0/1767557551 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ac4010510 con 0x7f4acc0b6a50 2022-06-29T16:01:49.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.447+0000 7f4abb7fe700 1 -- 172.21.15.60:0/1767557551 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ac4010670 con 0x7f4acc0b6a50 2022-06-29T16:01:49.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.447+0000 7f4ad0db6700 1 -- 172.21.15.60:0/1767557551 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4acc14a0e0 con 0x7f4acc0b6a50 2022-06-29T16:01:49.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.447+0000 7f4abb7fe700 1 -- 172.21.15.60:0/1767557551 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ac40107e0 con 0x7f4acc0b6a50 2022-06-29T16:01:49.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.447+0000 7f4ad0db6700 1 -- 172.21.15.60:0/1767557551 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4acc0ad520 con 0x7f4acc0b6a50 2022-06-29T16:01:49.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.447+0000 7f4ad0db6700 1 -- 172.21.15.60:0/1767557551 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4acc0573d0 con 0x7f4acc0b6a50 2022-06-29T16:01:49.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.447+0000 7f4abb7fe700 1 -- 172.21.15.60:0/1767557551 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4ac4010070 con 0x7f4acc0b6a50 2022-06-29T16:01:49.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.447+0000 7f4abb7fe700 1 --2- 172.21.15.60:0/1767557551 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4aa8053950 0x7f4aa8055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:49.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.447+0000 7f4abb7fe700 1 -- 172.21.15.60:0/1767557551 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f4ac406b810 con 0x7f4acc0b6a50 2022-06-29T16:01:49.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.447+0000 7f4aca59c700 1 --2- 172.21.15.60:0/1767557551 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4aa8053950 0x7f4aa8055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:49.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.447+0000 7f4aca59c700 1 --2- 172.21.15.60:0/1767557551 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4aa8053950 0x7f4aa8055e10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f4ac001a000 tx=0x7f4ac0015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:49.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.455+0000 7f4abb7fe700 1 -- 172.21.15.60:0/1767557551 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4ac4042a10 con 0x7f4acc0b6a50 2022-06-29T16:01:49.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.619+0000 7f4ad0db6700 1 -- 172.21.15.60:0/1767557551 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4acc09beb0 con 0x7f4acc0b6a50 2022-06-29T16:01:49.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.623+0000 7f4abb7fe700 1 -- 172.21.15.60:0/1767557551 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+1209 (secure 0 0 0) 0x7f4ac4046030 con 0x7f4acc0b6a50 2022-06-29T16:01:49.630 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:01:49.630 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":17,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4937,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/1673184114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1673184114},{"type":"v1","addr":"172.21.15.92:6835","nonce":1673184114}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":13},{"gid":5496,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1034264882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1034264882},{"type":"v1","addr":"172.21.15.60:6833","nonce":1034264882}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":17}],"filesystems":[]} 2022-06-29T16:01:49.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.623+0000 7f4ad0db6700 1 -- 172.21.15.60:0/1767557551 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4aa8053950 msgr2=0x7f4aa8055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:49.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.623+0000 7f4ad0db6700 1 --2- 172.21.15.60:0/1767557551 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4aa8053950 0x7f4aa8055e10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f4ac001a000 tx=0x7f4ac0015000).stop 2022-06-29T16:01:49.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.623+0000 7f4ad0db6700 1 -- 172.21.15.60:0/1767557551 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4acc0b6a50 msgr2=0x7f4acc098fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:49.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.623+0000 7f4ad0db6700 1 --2- 172.21.15.60:0/1767557551 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4acc0b6a50 0x7f4acc098fd0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f4acc0b3d60 tx=0x7f4ac40014a0).stop 2022-06-29T16:01:49.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.623+0000 7f4ad0db6700 1 -- 172.21.15.60:0/1767557551 shutdown_connections 2022-06-29T16:01:49.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.623+0000 7f4ad0db6700 1 --2- 172.21.15.60:0/1767557551 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4aa8053950 0x7f4aa8055e10 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:49.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.623+0000 7f4ad0db6700 1 --2- 172.21.15.60:0/1767557551 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4acc146cf0 0x7f4acc0994c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:49.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.623+0000 7f4ad0db6700 1 --2- 172.21.15.60:0/1767557551 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4acc0b6a50 0x7f4acc098fd0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:49.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.623+0000 7f4ad0db6700 1 --2- 172.21.15.60:0/1767557551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4acc062230 0x7f4acc14a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:49.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.623+0000 7f4ad0db6700 1 -- 172.21.15.60:0/1767557551 >> 172.21.15.60:0/1767557551 conn(0x7f4acc0643e0 msgr2=0x7f4acc0aafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:49.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.623+0000 7f4ad0db6700 1 -- 172.21.15.60:0/1767557551 shutdown_connections 2022-06-29T16:01:49.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.623+0000 7f4ad0db6700 1 -- 172.21.15.60:0/1767557551 wait complete. 2022-06-29T16:01:49.636 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 17 2022-06-29T16:01:49.651 INFO:tasks.ceph.mds.a:Restarting daemon 2022-06-29T16:01:49.651 INFO:tasks.ceph.mds.a:Stopping old one... 2022-06-29T16:01:49.652 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-06-29T16:01:49.652 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T16:01:49.653 INFO:tasks.ceph.mds.b:Restarting daemon 2022-06-29T16:01:49.653 INFO:tasks.ceph.mds.b:Stopping old one... 2022-06-29T16:01:49.653 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-06-29T16:01:49.653 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T16:01:49.663 INFO:tasks.ceph.mds.a:Stopped 2022-06-29T16:01:49.663 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-06-29T16:01:49.666 INFO:tasks.ceph.mds.a:Started 2022-06-29T16:01:49.667 DEBUG:teuthology.parallel:result is None 2022-06-29T16:01:49.668 INFO:tasks.ceph.mds.b:Stopped 2022-06-29T16:01:49.668 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-06-29T16:01:49.713 INFO:tasks.ceph.mds.b:Started 2022-06-29T16:01:49.713 DEBUG:teuthology.parallel:result is None 2022-06-29T16:01:49.714 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-06-29T16:01:49.736 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2022-06-29T16:01:49.788 INFO:tasks.ceph.mds.b.smithi092.stdout:starting mds.b at 2022-06-29T16:01:49.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.807+0000 7f2a5a3f3700 1 Processor -- start 2022-06-29T16:01:49.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.807+0000 7f2a5a3f3700 1 -- start start 2022-06-29T16:01:49.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.807+0000 7f2a5a3f3700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2a54145eb0 0x7f2a54066130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:49.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.807+0000 7f2a5a3f3700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a5405c2b0 con 0x7f2a540b3c70 2022-06-29T16:01:49.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.807+0000 7f2a5a3f3700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a540b4050 con 0x7f2a54141860 2022-06-29T16:01:49.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.807+0000 7f2a5a3f3700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a54066620 con 0x7f2a54145eb0 2022-06-29T16:01:49.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.807+0000 7f2a593f1700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2a540b3c70 0x7f2a54065a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55796/0 (socket says 172.21.15.60:55796) 2022-06-29T16:01:49.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.807+0000 7f2a593f1700 1 -- 172.21.15.60:0/4068567904 learned_addr learned my addr 172.21.15.60:0/4068567904 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:01:49.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.807+0000 7f2a58bf0700 1 --2- 172.21.15.60:0/4068567904 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2a54145eb0 0x7f2a54066130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:49.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.807+0000 7f2a4bfff700 1 -- 172.21.15.60:0/4068567904 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3987050927 0 0) 0x7f2a5405c2b0 con 0x7f2a540b3c70 2022-06-29T16:01:49.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.807+0000 7f2a4bfff700 1 -- 172.21.15.60:0/4068567904 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a38003260 con 0x7f2a540b3c70 2022-06-29T16:01:49.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.807+0000 7f2a4bfff700 1 -- 172.21.15.60:0/4068567904 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 124687113 0 0) 0x7f2a540b4050 con 0x7f2a54141860 2022-06-29T16:01:49.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.807+0000 7f2a58bf0700 1 -- 172.21.15.60:0/4068567904 >> v1:172.21.15.92:6789/0 conn(0x7f2a54141860 legacy=0x7f2a54141cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:49.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.807+0000 7f2a58bf0700 1 -- 172.21.15.60:0/4068567904 >> v1:172.21.15.60:6789/0 conn(0x7f2a540b3c70 legacy=0x7f2a54065a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:49.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.807+0000 7f2a58bf0700 1 -- 172.21.15.60:0/4068567904 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a5405d120 con 0x7f2a54145eb0 2022-06-29T16:01:49.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.807+0000 7f2a58bf0700 1 --2- 172.21.15.60:0/4068567904 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2a54145eb0 0x7f2a54066130 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f2a44002f70 tx=0x7f2a44006df0).ready entity=mon.2 client_cookie=b4ad39175e4c4dd0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:49.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.807+0000 7f2a4bfff700 1 -- 172.21.15.60:0/4068567904 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a4400d570 con 0x7f2a54145eb0 2022-06-29T16:01:49.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.807+0000 7f2a4bfff700 1 -- 172.21.15.60:0/4068567904 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a44000e90 con 0x7f2a54145eb0 2022-06-29T16:01:49.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.807+0000 7f2a4bfff700 1 -- 172.21.15.60:0/4068567904 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a4400d8c0 con 0x7f2a54145eb0 2022-06-29T16:01:49.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.807+0000 7f2a5a3f3700 1 -- 172.21.15.60:0/4068567904 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2a54145eb0 msgr2=0x7f2a54066130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:49.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.807+0000 7f2a5a3f3700 1 --2- 172.21.15.60:0/4068567904 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2a54145eb0 0x7f2a54066130 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f2a44002f70 tx=0x7f2a44006df0).stop 2022-06-29T16:01:49.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.807+0000 7f2a5a3f3700 1 -- 172.21.15.60:0/4068567904 shutdown_connections 2022-06-29T16:01:49.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.807+0000 7f2a5a3f3700 1 --2- 172.21.15.60:0/4068567904 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2a54145eb0 0x7f2a54066130 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:49.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.807+0000 7f2a5a3f3700 1 -- 172.21.15.60:0/4068567904 >> 172.21.15.60:0/4068567904 conn(0x7f2a54064810 msgr2=0x7f2a54064bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:49.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.807+0000 7f2a5a3f3700 1 -- 172.21.15.60:0/4068567904 shutdown_connections 2022-06-29T16:01:49.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.807+0000 7f2a5a3f3700 1 -- 172.21.15.60:0/4068567904 wait complete. 2022-06-29T16:01:49.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.811+0000 7f2a5a3f3700 1 Processor -- start 2022-06-29T16:01:49.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.811+0000 7f2a5a3f3700 1 -- start start 2022-06-29T16:01:49.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.811+0000 7f2a5a3f3700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a540b3c70 0x7f2a541526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:49.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.811+0000 7f2a5a3f3700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2a54141860 0x7f2a540ad1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:49.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.811+0000 7f2a5a3f3700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a54145eb0 0x7f2a541441d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:49.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.811+0000 7f2a5a3f3700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a54151cf0 con 0x7f2a540b3c70 2022-06-29T16:01:49.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.811+0000 7f2a5a3f3700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a54151f90 con 0x7f2a54145eb0 2022-06-29T16:01:49.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.811+0000 7f2a5a3f3700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a54152230 con 0x7f2a54141860 2022-06-29T16:01:49.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.811+0000 7f2a593f1700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a540b3c70 0x7f2a541526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:49.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.811+0000 7f2a58bf0700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2a54141860 0x7f2a540ad1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:49.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.811+0000 7f2a58bf0700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2a54141860 0x7f2a540ad1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:60808/0 (socket says 172.21.15.60:60808) 2022-06-29T16:01:49.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.811+0000 7f2a58bf0700 1 -- 172.21.15.60:0/4191608328 learned_addr learned my addr 172.21.15.60:0/4191608328 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:49.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.811+0000 7f2a59bf2700 1 --2- 172.21.15.60:0/4191608328 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a54145eb0 0x7f2a541441d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:49.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.811+0000 7f2a58bf0700 1 -- 172.21.15.60:0/4191608328 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a54145eb0 msgr2=0x7f2a541441d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:49.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.811+0000 7f2a58bf0700 1 --2- 172.21.15.60:0/4191608328 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a54145eb0 0x7f2a541441d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:49.829 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.811+0000 7f2a58bf0700 1 -- 172.21.15.60:0/4191608328 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a540b3c70 msgr2=0x7f2a541526d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:49.829 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.811+0000 7f2a58bf0700 1 --2- 172.21.15.60:0/4191608328 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a540b3c70 0x7f2a541526d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:49.829 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.811+0000 7f2a58bf0700 1 -- 172.21.15.60:0/4191608328 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a4400b180 con 0x7f2a54141860 2022-06-29T16:01:49.829 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.811+0000 7f2a58bf0700 1 --2- 172.21.15.60:0/4191608328 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2a54141860 0x7f2a540ad1a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f2a44002f70 tx=0x7f2a44014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:49.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.811+0000 7f2a4a7fc700 1 -- 172.21.15.60:0/4191608328 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a44013da0 con 0x7f2a54141860 2022-06-29T16:01:49.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.811+0000 7f2a4a7fc700 1 -- 172.21.15.60:0/4191608328 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a4401e400 con 0x7f2a54141860 2022-06-29T16:01:49.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.811+0000 7f2a5a3f3700 1 -- 172.21.15.60:0/4191608328 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a54143ed0 con 0x7f2a54141860 2022-06-29T16:01:49.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.811+0000 7f2a4a7fc700 1 -- 172.21.15.60:0/4191608328 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a4401e760 con 0x7f2a54141860 2022-06-29T16:01:49.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.811+0000 7f2a5a3f3700 1 -- 172.21.15.60:0/4191608328 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a54151a40 con 0x7f2a54141860 2022-06-29T16:01:49.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.811+0000 7f2a5a3f3700 1 -- 172.21.15.60:0/4191608328 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a54145b10 con 0x7f2a54141860 2022-06-29T16:01:49.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.811+0000 7f2a4a7fc700 1 -- 172.21.15.60:0/4191608328 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2a44025030 con 0x7f2a54141860 2022-06-29T16:01:49.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.815+0000 7f2a4a7fc700 1 --2- 172.21.15.60:0/4191608328 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2a3c053a70 0x7f2a3c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:49.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.815+0000 7f2a4a7fc700 1 -- 172.21.15.60:0/4191608328 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f2a4406ca00 con 0x7f2a54141860 2022-06-29T16:01:49.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.815+0000 7f2a593f1700 1 --2- 172.21.15.60:0/4191608328 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2a3c053a70 0x7f2a3c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:49.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.815+0000 7f2a593f1700 1 --2- 172.21.15.60:0/4191608328 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2a3c053a70 0x7f2a3c055f30 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f2a5000d000 tx=0x7f2a50009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:49.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.819+0000 7f2a4a7fc700 1 -- 172.21.15.60:0/4191608328 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2a54145b10 con 0x7f2a54141860 2022-06-29T16:01:49.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:49.955+0000 7f2a5a3f3700 1 -- 172.21.15.60:0/4191608328 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f2a54145b10 con 0x7f2a54141860 2022-06-29T16:01:50.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.807+0000 7f2a4a7fc700 1 -- 172.21.15.60:0/4191608328 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v49) v1 ==== 97+0+0 (secure 0 0 0) 0x7f2a54145b10 con 0x7f2a54141860 2022-06-29T16:01:50.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.811+0000 7f2a5a3f3700 1 -- 172.21.15.60:0/4191608328 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2a3c053a70 msgr2=0x7f2a3c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:50.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.811+0000 7f2a5a3f3700 1 --2- 172.21.15.60:0/4191608328 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2a3c053a70 0x7f2a3c055f30 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f2a5000d000 tx=0x7f2a50009040).stop 2022-06-29T16:01:50.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.811+0000 7f2a5a3f3700 1 -- 172.21.15.60:0/4191608328 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2a54141860 msgr2=0x7f2a540ad1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:50.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.811+0000 7f2a5a3f3700 1 --2- 172.21.15.60:0/4191608328 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2a54141860 0x7f2a540ad1a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f2a44002f70 tx=0x7f2a44014000).stop 2022-06-29T16:01:50.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.811+0000 7f2a5a3f3700 1 -- 172.21.15.60:0/4191608328 shutdown_connections 2022-06-29T16:01:50.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.811+0000 7f2a5a3f3700 1 --2- 172.21.15.60:0/4191608328 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2a3c053a70 0x7f2a3c055f30 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:50.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.811+0000 7f2a5a3f3700 1 --2- 172.21.15.60:0/4191608328 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a54145eb0 0x7f2a541441d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:50.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.811+0000 7f2a5a3f3700 1 --2- 172.21.15.60:0/4191608328 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2a54141860 0x7f2a540ad1a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:50.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.811+0000 7f2a5a3f3700 1 --2- 172.21.15.60:0/4191608328 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a540b3c70 0x7f2a541526d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:50.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.811+0000 7f2a5a3f3700 1 -- 172.21.15.60:0/4191608328 >> 172.21.15.60:0/4191608328 conn(0x7f2a54064810 msgr2=0x7f2a54065940 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:50.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.811+0000 7f2a5a3f3700 1 -- 172.21.15.60:0/4191608328 shutdown_connections 2022-06-29T16:01:50.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.811+0000 7f2a5a3f3700 1 -- 172.21.15.60:0/4191608328 wait complete. 2022-06-29T16:01:50.825 INFO:teuthology.orchestra.run.smithi060.stderr: removed all blacklist entries 2022-06-29T16:01:50.834 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-06-29T16:01:50.834 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-06-29T16:01:50.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.979+0000 7f8a39551700 1 Processor -- start 2022-06-29T16:01:50.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.979+0000 7f8a39551700 1 -- start start 2022-06-29T16:01:50.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.979+0000 7f8a39551700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8a340aec30 0x7f8a3414b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:50.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.979+0000 7f8a39551700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a34064e20 con 0x7f8a34067a90 2022-06-29T16:01:50.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.979+0000 7f8a39551700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a3405fee0 con 0x7f8a3405fb00 2022-06-29T16:01:50.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.979+0000 7f8a39551700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a340b10e0 con 0x7f8a340aec30 2022-06-29T16:01:50.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.979+0000 7f8a2b7fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f8a34067a90 0x7f8a341400b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55812/0 (socket says 172.21.15.60:55812) 2022-06-29T16:01:50.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.979+0000 7f8a2b7fe700 1 -- 172.21.15.60:0/3702207839 learned_addr learned my addr 172.21.15.60:0/3702207839 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:01:50.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.979+0000 7f8a38d50700 1 --2- 172.21.15.60:0/3702207839 >> v2:172.21.15.92:3300/0 conn(0x7f8a340aec30 0x7f8a3414b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:50.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.979+0000 7f8a38d50700 1 -- 172.21.15.60:0/3702207839 >> v1:172.21.15.92:6789/0 conn(0x7f8a3405fb00 legacy=0x7f8a3413f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:50.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.979+0000 7f8a38d50700 1 -- 172.21.15.60:0/3702207839 >> v1:172.21.15.60:6789/0 conn(0x7f8a34067a90 legacy=0x7f8a341400b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:50.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.979+0000 7f8a337fe700 1 -- 172.21.15.60:0/3702207839 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4059462144 0 0) 0x7f8a34064e20 con 0x7f8a34067a90 2022-06-29T16:01:50.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.979+0000 7f8a38d50700 1 -- 172.21.15.60:0/3702207839 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a341407c0 con 0x7f8a340aec30 2022-06-29T16:01:50.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.979+0000 7f8a38d50700 1 --2- 172.21.15.60:0/3702207839 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a340aec30 0x7f8a3414b5d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f8a2c0032f0 tx=0x7f8a2c0091d0).ready entity=mon.1 client_cookie=e1b79c543415e5f2 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:50.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.979+0000 7f8a337fe700 1 -- 172.21.15.60:0/3702207839 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a2c009ea0 con 0x7f8a340aec30 2022-06-29T16:01:50.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.979+0000 7f8a337fe700 1 -- 172.21.15.60:0/3702207839 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a2c00b8a0 con 0x7f8a340aec30 2022-06-29T16:01:50.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.979+0000 7f8a337fe700 1 -- 172.21.15.60:0/3702207839 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a2c00ba10 con 0x7f8a340aec30 2022-06-29T16:01:50.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.983+0000 7f8a39551700 1 -- 172.21.15.60:0/3702207839 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a340aec30 msgr2=0x7f8a3414b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:50.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.983+0000 7f8a39551700 1 --2- 172.21.15.60:0/3702207839 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a340aec30 0x7f8a3414b5d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f8a2c0032f0 tx=0x7f8a2c0091d0).stop 2022-06-29T16:01:50.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.983+0000 7f8a39551700 1 -- 172.21.15.60:0/3702207839 shutdown_connections 2022-06-29T16:01:50.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.983+0000 7f8a39551700 1 --2- 172.21.15.60:0/3702207839 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a340aec30 0x7f8a3414b5d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:50.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.983+0000 7f8a39551700 1 -- 172.21.15.60:0/3702207839 >> 172.21.15.60:0/3702207839 conn(0x7f8a34063a60 msgr2=0x7f8a3405f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:50.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.983+0000 7f8a39551700 1 -- 172.21.15.60:0/3702207839 shutdown_connections 2022-06-29T16:01:50.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.983+0000 7f8a39551700 1 -- 172.21.15.60:0/3702207839 wait complete. 2022-06-29T16:01:50.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.983+0000 7f8a39551700 1 Processor -- start 2022-06-29T16:01:50.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.983+0000 7f8a39551700 1 -- start start 2022-06-29T16:01:50.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.983+0000 7f8a39551700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a3405fb00 0x7f8a3414fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:50.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.983+0000 7f8a39551700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a34067a90 0x7f8a340b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:50.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.983+0000 7f8a39551700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8a340aec30 0x7f8a34153330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:50.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.983+0000 7f8a39551700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a340672d0 con 0x7f8a34067a90 2022-06-29T16:01:50.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.983+0000 7f8a39551700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a340674c0 con 0x7f8a3405fb00 2022-06-29T16:01:50.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.983+0000 7f8a39551700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3413dea0 con 0x7f8a340aec30 2022-06-29T16:01:50.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.983+0000 7f8a2b7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a34067a90 0x7f8a340b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:50.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.983+0000 7f8a33fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a3405fb00 0x7f8a3414fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:50.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.983+0000 7f8a33fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a3405fb00 0x7f8a3414fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:53760/0 (socket says 172.21.15.60:53760) 2022-06-29T16:01:50.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.983+0000 7f8a33fff700 1 -- 172.21.15.60:0/1522381572 learned_addr learned my addr 172.21.15.60:0/1522381572 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:50.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.983+0000 7f8a38d50700 1 --2- 172.21.15.60:0/1522381572 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8a340aec30 0x7f8a34153330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:50.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.987+0000 7f8a33fff700 1 -- 172.21.15.60:0/1522381572 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8a340aec30 msgr2=0x7f8a34153330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:50.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.987+0000 7f8a33fff700 1 --2- 172.21.15.60:0/1522381572 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8a340aec30 0x7f8a34153330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:51.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.987+0000 7f8a33fff700 1 -- 172.21.15.60:0/1522381572 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a34067a90 msgr2=0x7f8a340b21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:51.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.987+0000 7f8a33fff700 1 --2- 172.21.15.60:0/1522381572 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a34067a90 0x7f8a340b21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:51.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.987+0000 7f8a33fff700 1 -- 172.21.15.60:0/1522381572 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a2c009470 con 0x7f8a3405fb00 2022-06-29T16:01:51.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.987+0000 7f8a33fff700 1 --2- 172.21.15.60:0/1522381572 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a3405fb00 0x7f8a3414fa20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f8a340627a0 tx=0x7f8a24007c50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:51.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.987+0000 7f8a31ffb700 1 -- 172.21.15.60:0/1522381572 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a24012400 con 0x7f8a3405fb00 2022-06-29T16:01:51.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.987+0000 7f8a31ffb700 1 -- 172.21.15.60:0/1522381572 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a24012560 con 0x7f8a3405fb00 2022-06-29T16:01:51.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.987+0000 7f8a39551700 1 -- 172.21.15.60:0/1522381572 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a3413c320 con 0x7f8a3405fb00 2022-06-29T16:01:51.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.987+0000 7f8a31ffb700 1 -- 172.21.15.60:0/1522381572 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a240128c0 con 0x7f8a3405fb00 2022-06-29T16:01:51.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.987+0000 7f8a39551700 1 -- 172.21.15.60:0/1522381572 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a3413d080 con 0x7f8a3405fb00 2022-06-29T16:01:51.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.987+0000 7f8a39551700 1 -- 172.21.15.60:0/1522381572 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a3413c5f0 con 0x7f8a3405fb00 2022-06-29T16:01:51.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.987+0000 7f8a31ffb700 1 -- 172.21.15.60:0/1522381572 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8a24010070 con 0x7f8a3405fb00 2022-06-29T16:01:51.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.991+0000 7f8a31ffb700 1 --2- 172.21.15.60:0/1522381572 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8a180539e0 0x7f8a18055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:51.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.991+0000 7f8a31ffb700 1 -- 172.21.15.60:0/1522381572 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f8a2406abe0 con 0x7f8a3405fb00 2022-06-29T16:01:51.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.991+0000 7f8a2b7fe700 1 --2- 172.21.15.60:0/1522381572 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8a180539e0 0x7f8a18055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:51.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.991+0000 7f8a2b7fe700 1 --2- 172.21.15.60:0/1522381572 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8a180539e0 0x7f8a18055ea0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f8a1c00d000 tx=0x7f8a1c007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:51.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:50.999+0000 7f8a31ffb700 1 -- 172.21.15.60:0/1522381572 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8a24041d20 con 0x7f8a3405fb00 2022-06-29T16:01:51.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.163+0000 7f8a39551700 1 -- 172.21.15.60:0/1522381572 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f8a34060b20 con 0x7f8a3405fb00 2022-06-29T16:01:51.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.167+0000 7f8a31ffb700 1 -- 172.21.15.60:0/1522381572 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+4242 (secure 0 0 0) 0x7f8a24045340 con 0x7f8a3405fb00 2022-06-29T16:01:51.175 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:01:51.176 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:01:51.176 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_dump": [ 2022-06-29T16:01:51.176 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:51.177 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.a", 2022-06-29T16:01:51.177 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxivv7JNhAAnA291UxlB/SfgITOipP00A==", 2022-06-29T16:01:51.178 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:01:51.178 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:01:51.179 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:01:51.179 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:01:51.180 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:01:51.180 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:51.181 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:51.181 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:51.181 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.b", 2022-06-29T16:01:51.182 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxiRSr/OhAAfFQTF2YHvzCv5wXBVCA5WQ==", 2022-06-29T16:01:51.182 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:01:51.183 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:01:51.183 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:01:51.184 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:01:51.184 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:01:51.184 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:51.185 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:51.185 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:51.186 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.0", 2022-06-29T16:01:51.186 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiZYPxOBAAIkweWJlTJIXgxnlagkGdXA==", 2022-06-29T16:01:51.187 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:01:51.187 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:01:51.188 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:01:51.188 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:01:51.189 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:51.189 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:51.189 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:51.190 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.1", 2022-06-29T16:01:51.190 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxiS2JGBhAAJn0UfAxZI3IVFyD5QBdb/Q==", 2022-06-29T16:01:51.190 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:01:51.191 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:01:51.191 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:01:51.192 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:01:51.192 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:51.192 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:51.192 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:51.192 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.2", 2022-06-29T16:01:51.193 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxiPhkfEhAApH7r8OUJzW7fZNM6XVv1dg==", 2022-06-29T16:01:51.193 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:01:51.193 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:01:51.193 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:01:51.194 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:01:51.194 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:51.194 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:51.194 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:51.195 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.3", 2022-06-29T16:01:51.195 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxit/47HhAA1vyU6h7hjPG+op9qP/rVYQ==", 2022-06-29T16:01:51.195 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:01:51.195 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:01:51.196 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:01:51.196 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:01:51.197 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:51.197 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:51.198 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:51.198 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.4", 2022-06-29T16:01:51.198 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxiaKzKBxAAbIwCF9ag7ezsNy11hi0BEw==", 2022-06-29T16:01:51.199 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:01:51.199 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:01:51.200 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:01:51.200 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:01:51.201 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:51.201 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:51.201 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:51.201 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.5", 2022-06-29T16:01:51.202 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxiJQV/IBAASBBDb6UFgEtXILe4fcT/xA==", 2022-06-29T16:01:51.202 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:01:51.202 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:01:51.202 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:01:51.202 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:01:51.203 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:51.203 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:51.203 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:51.203 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.6", 2022-06-29T16:01:51.204 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxi6D07OBAA2FWiZ+mNK+jFMp/kPZnbQQ==", 2022-06-29T16:01:51.204 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:01:51.204 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:01:51.204 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:01:51.205 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:01:51.205 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:51.205 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:51.205 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:51.206 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.7", 2022-06-29T16:01:51.206 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCMdrxinEOcEhAA562ybpzGHNQfWIq8q7zw9w==", 2022-06-29T16:01:51.206 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:01:51.206 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:01:51.207 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:01:51.207 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:01:51.207 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:51.207 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:51.208 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:51.208 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.0", 2022-06-29T16:01:51.208 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiHXPwBRAAkOOwlYJvIW7dqVFdlXXoqw==", 2022-06-29T16:01:51.208 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:01:51.208 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:01:51.209 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2022-06-29T16:01:51.209 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw pool=cephfs_data" 2022-06-29T16:01:51.209 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:51.209 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:51.210 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:51.210 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.1", 2022-06-29T16:01:51.210 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxibe5JCRAAGBJbi9t09w0836JqrAT7+w==", 2022-06-29T16:01:51.210 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:01:51.211 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:01:51.211 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2022-06-29T16:01:51.211 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw pool=cephfs_data" 2022-06-29T16:01:51.211 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:51.212 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:51.212 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:51.212 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.2", 2022-06-29T16:01:51.212 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiK3dBDRAA7XINPSOPQDfOrUYEpHC+gw==", 2022-06-29T16:01:51.212 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:01:51.213 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:01:51.213 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2022-06-29T16:01:51.213 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw pool=cephfs_data" 2022-06-29T16:01:51.213 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:51.213 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:51.214 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:51.214 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.admin", 2022-06-29T16:01:51.214 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxieKVcGBAAnjH80a0qWIArz2qwjYHWdQ==", 2022-06-29T16:01:51.214 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:01:51.215 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T16:01:51.215 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:01:51.215 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:01:51.215 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:01:51.215 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:51.216 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:51.216 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:51.216 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.x", 2022-06-29T16:01:51.216 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxi2LZJNBAA3MKb/pXYbQCsoRx8EQfGbw==", 2022-06-29T16:01:51.217 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:01:51.217 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T16:01:51.218 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2022-06-29T16:01:51.218 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:01:51.218 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:51.218 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:51.218 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:51.219 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.y", 2022-06-29T16:01:51.219 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxi3LmaMBAAN+0mZrdKXqSXOipVTA1drw==", 2022-06-29T16:01:51.219 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:01:51.219 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T16:01:51.220 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2022-06-29T16:01:51.220 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:01:51.220 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:51.220 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:51.220 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:01:51.221 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:01:51.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.167+0000 7f8a39551700 1 -- 172.21.15.60:0/1522381572 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8a180539e0 msgr2=0x7f8a18055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:51.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.167+0000 7f8a39551700 1 --2- 172.21.15.60:0/1522381572 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8a180539e0 0x7f8a18055ea0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f8a1c00d000 tx=0x7f8a1c007a00).stop 2022-06-29T16:01:51.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.167+0000 7f8a39551700 1 -- 172.21.15.60:0/1522381572 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a3405fb00 msgr2=0x7f8a3414fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:51.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.167+0000 7f8a39551700 1 --2- 172.21.15.60:0/1522381572 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a3405fb00 0x7f8a3414fa20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f8a340627a0 tx=0x7f8a24007c50).stop 2022-06-29T16:01:51.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.167+0000 7f8a39551700 1 -- 172.21.15.60:0/1522381572 shutdown_connections 2022-06-29T16:01:51.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.167+0000 7f8a39551700 1 --2- 172.21.15.60:0/1522381572 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8a180539e0 0x7f8a18055ea0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:51.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.167+0000 7f8a39551700 1 --2- 172.21.15.60:0/1522381572 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8a340aec30 0x7f8a34153330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:51.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.167+0000 7f8a39551700 1 --2- 172.21.15.60:0/1522381572 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8a34067a90 0x7f8a340b21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:51.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.167+0000 7f8a39551700 1 --2- 172.21.15.60:0/1522381572 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a3405fb00 0x7f8a3414fa20 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:51.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.167+0000 7f8a39551700 1 -- 172.21.15.60:0/1522381572 >> 172.21.15.60:0/1522381572 conn(0x7f8a34063a60 msgr2=0x7f8a341409d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:51.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.167+0000 7f8a39551700 1 -- 172.21.15.60:0/1522381572 shutdown_connections 2022-06-29T16:01:51.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.167+0000 7f8a39551700 1 -- 172.21.15.60:0/1522381572 wait complete. 2022-06-29T16:01:51.226 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:01:51.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.323+0000 7f1a8ad26700 1 Processor -- start 2022-06-29T16:01:51.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.327+0000 7f1a8ad26700 1 -- start start 2022-06-29T16:01:51.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.327+0000 7f1a8ad26700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a840b5950 0x7f1a84067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:51.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.327+0000 7f1a8ad26700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1a8414af10 0x7f1a8405c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:51.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.327+0000 7f1a8ad26700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a840acba0 con 0x7f1a84063ad0 2022-06-29T16:01:51.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.327+0000 7f1a8ad26700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a8414b3a0 con 0x7f1a8414af10 2022-06-29T16:01:51.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.327+0000 7f1a8ad26700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a84067220 con 0x7f1a840b5950 2022-06-29T16:01:51.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.327+0000 7f1a892c3700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1a8414af10 0x7f1a8405c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:51.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.327+0000 7f1a83fff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f1a84063ad0 0x7f1a8405bd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55826/0 (socket says 172.21.15.60:55826) 2022-06-29T16:01:51.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.327+0000 7f1a892c3700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1a8414af10 0x7f1a8405c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47740/0 (socket says 172.21.15.60:47740) 2022-06-29T16:01:51.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.327+0000 7f1a83fff700 1 -- 172.21.15.60:0/248920912 learned_addr learned my addr 172.21.15.60:0/248920912 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:01:51.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.327+0000 7f1a88ac2700 1 --2- 172.21.15.60:0/248920912 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a840b5950 0x7f1a84067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:51.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.327+0000 7f1a892c3700 1 -- 172.21.15.60:0/248920912 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a840b5950 msgr2=0x7f1a84067680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:01:51.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.327+0000 7f1a892c3700 1 --2- 172.21.15.60:0/248920912 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a840b5950 0x7f1a84067680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:51.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.327+0000 7f1a892c3700 1 -- 172.21.15.60:0/248920912 >> v1:172.21.15.60:6789/0 conn(0x7f1a84063ad0 legacy=0x7f1a8405bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:51.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.327+0000 7f1a837fe700 1 -- 172.21.15.60:0/248920912 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2524531537 0 0) 0x7f1a840acba0 con 0x7f1a84063ad0 2022-06-29T16:01:51.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.327+0000 7f1a892c3700 1 -- 172.21.15.60:0/248920912 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a840b5d30 con 0x7f1a8414af10 2022-06-29T16:01:51.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.327+0000 7f1a892c3700 1 --2- 172.21.15.60:0/248920912 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a8414af10 0x7f1a8405c430 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f1a7c002f70 tx=0x7f1a7c006df0).ready entity=mon.0 client_cookie=35a08ced2886bdd0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:51.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.327+0000 7f1a837fe700 1 -- 172.21.15.60:0/248920912 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a7c00d570 con 0x7f1a8414af10 2022-06-29T16:01:51.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.327+0000 7f1a837fe700 1 -- 172.21.15.60:0/248920912 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a7c000e90 con 0x7f1a8414af10 2022-06-29T16:01:51.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.327+0000 7f1a837fe700 1 -- 172.21.15.60:0/248920912 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a7c00d8c0 con 0x7f1a8414af10 2022-06-29T16:01:51.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.327+0000 7f1a8ad26700 1 -- 172.21.15.60:0/248920912 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a8414af10 msgr2=0x7f1a8405c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:51.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.327+0000 7f1a8ad26700 1 --2- 172.21.15.60:0/248920912 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a8414af10 0x7f1a8405c430 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f1a7c002f70 tx=0x7f1a7c006df0).stop 2022-06-29T16:01:51.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.327+0000 7f1a8ad26700 1 -- 172.21.15.60:0/248920912 shutdown_connections 2022-06-29T16:01:51.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.327+0000 7f1a8ad26700 1 --2- 172.21.15.60:0/248920912 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a8414af10 0x7f1a8405c430 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:51.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.327+0000 7f1a8ad26700 1 --2- 172.21.15.60:0/248920912 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a840b5950 0x7f1a84067680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:51.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.327+0000 7f1a8ad26700 1 -- 172.21.15.60:0/248920912 >> 172.21.15.60:0/248920912 conn(0x7f1a840632c0 msgr2=0x7f1a840b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:51.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.327+0000 7f1a8ad26700 1 -- 172.21.15.60:0/248920912 shutdown_connections 2022-06-29T16:01:51.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.327+0000 7f1a8ad26700 1 -- 172.21.15.60:0/248920912 wait complete. 2022-06-29T16:01:51.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.331+0000 7f1a8ad26700 1 Processor -- start 2022-06-29T16:01:51.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.331+0000 7f1a8ad26700 1 -- start start 2022-06-29T16:01:51.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.331+0000 7f1a8ad26700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a84063ad0 0x7f1a840afb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:51.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.331+0000 7f1a8ad26700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a840b5950 0x7f1a840af0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:51.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.331+0000 7f1a8ad26700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a8414af10 0x7f1a84142c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:51.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.331+0000 7f1a8ad26700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a840adcb0 con 0x7f1a84063ad0 2022-06-29T16:01:51.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.331+0000 7f1a8ad26700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a841173b0 con 0x7f1a840b5950 2022-06-29T16:01:51.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.331+0000 7f1a8ad26700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8414a220 con 0x7f1a8414af10 2022-06-29T16:01:51.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.331+0000 7f1a88ac2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a84063ad0 0x7f1a840afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:51.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.331+0000 7f1a88ac2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a84063ad0 0x7f1a840afb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47744/0 (socket says 172.21.15.60:47744) 2022-06-29T16:01:51.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.331+0000 7f1a88ac2700 1 -- 172.21.15.60:0/1772645857 learned_addr learned my addr 172.21.15.60:0/1772645857 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:51.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.331+0000 7f1a892c3700 1 --2- 172.21.15.60:0/1772645857 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a8414af10 0x7f1a84142c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:51.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.331+0000 7f1a83fff700 1 --2- 172.21.15.60:0/1772645857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a840b5950 0x7f1a840af0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:51.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.331+0000 7f1a88ac2700 1 -- 172.21.15.60:0/1772645857 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a8414af10 msgr2=0x7f1a84142c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:51.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.331+0000 7f1a88ac2700 1 --2- 172.21.15.60:0/1772645857 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a8414af10 0x7f1a84142c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:51.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.331+0000 7f1a88ac2700 1 -- 172.21.15.60:0/1772645857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a840b5950 msgr2=0x7f1a840af0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:51.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.331+0000 7f1a88ac2700 1 --2- 172.21.15.60:0/1772645857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a840b5950 0x7f1a840af0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:51.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.331+0000 7f1a88ac2700 1 -- 172.21.15.60:0/1772645857 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a7c00b180 con 0x7f1a84063ad0 2022-06-29T16:01:51.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.331+0000 7f1a88ac2700 1 --2- 172.21.15.60:0/1772645857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a84063ad0 0x7f1a840afb40 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f1a74006e00 tx=0x7f1a740091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:51.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.331+0000 7f1a81ffb700 1 -- 172.21.15.60:0/1772645857 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a7400ee60 con 0x7f1a84063ad0 2022-06-29T16:01:51.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.331+0000 7f1a81ffb700 1 -- 172.21.15.60:0/1772645857 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a7400c850 con 0x7f1a84063ad0 2022-06-29T16:01:51.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.331+0000 7f1a8ad26700 1 -- 172.21.15.60:0/1772645857 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a840af5c0 con 0x7f1a84063ad0 2022-06-29T16:01:51.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.331+0000 7f1a81ffb700 1 -- 172.21.15.60:0/1772645857 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a7400cbb0 con 0x7f1a84063ad0 2022-06-29T16:01:51.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.331+0000 7f1a8ad26700 1 -- 172.21.15.60:0/1772645857 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a840ada50 con 0x7f1a84063ad0 2022-06-29T16:01:51.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.335+0000 7f1a8ad26700 1 -- 172.21.15.60:0/1772645857 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a84117220 con 0x7f1a84063ad0 2022-06-29T16:01:51.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.335+0000 7f1a81ffb700 1 -- 172.21.15.60:0/1772645857 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1a74022070 con 0x7f1a84063ad0 2022-06-29T16:01:51.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.335+0000 7f1a81ffb700 1 --2- 172.21.15.60:0/1772645857 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1a68053aa0 0x7f1a68055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:51.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.335+0000 7f1a81ffb700 1 -- 172.21.15.60:0/1772645857 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f1a7406c270 con 0x7f1a84063ad0 2022-06-29T16:01:51.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.335+0000 7f1a83fff700 1 --2- 172.21.15.60:0/1772645857 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1a68053aa0 0x7f1a68055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:51.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.335+0000 7f1a83fff700 1 --2- 172.21.15.60:0/1772645857 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1a68053aa0 0x7f1a68055f60 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f1a6c00d000 tx=0x7f1a6c007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:51.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.339+0000 7f1a81ffb700 1 -- 172.21.15.60:0/1772645857 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1a74043460 con 0x7f1a84063ad0 2022-06-29T16:01:51.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.471+0000 7f1a8ad26700 1 -- 172.21.15.60:0/1772645857 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a840adf80 con 0x7f1a84063ad0 2022-06-29T16:01:51.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.475+0000 7f1a81ffb700 1 -- 172.21.15.60:0/1772645857 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) v1 ==== 74+0+13181 (secure 0 0 0) 0x7f1a74016020 con 0x7f1a84063ad0 2022-06-29T16:01:51.482 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:01:51.483 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":49,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:01:50.800372+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:01:51.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.475+0000 7f1a8ad26700 1 -- 172.21.15.60:0/1772645857 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1a68053aa0 msgr2=0x7f1a68055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:51.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.475+0000 7f1a8ad26700 1 --2- 172.21.15.60:0/1772645857 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1a68053aa0 0x7f1a68055f60 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f1a6c00d000 tx=0x7f1a6c007a00).stop 2022-06-29T16:01:51.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.475+0000 7f1a8ad26700 1 -- 172.21.15.60:0/1772645857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a84063ad0 msgr2=0x7f1a840afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:51.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.475+0000 7f1a8ad26700 1 --2- 172.21.15.60:0/1772645857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a84063ad0 0x7f1a840afb40 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f1a74006e00 tx=0x7f1a740091d0).stop 2022-06-29T16:01:51.486 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.475+0000 7f1a8ad26700 1 -- 172.21.15.60:0/1772645857 shutdown_connections 2022-06-29T16:01:51.486 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.475+0000 7f1a8ad26700 1 --2- 172.21.15.60:0/1772645857 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1a68053aa0 0x7f1a68055f60 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:51.486 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.475+0000 7f1a8ad26700 1 --2- 172.21.15.60:0/1772645857 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1a8414af10 0x7f1a84142c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:51.486 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.475+0000 7f1a8ad26700 1 --2- 172.21.15.60:0/1772645857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1a840b5950 0x7f1a840af0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:51.487 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.475+0000 7f1a8ad26700 1 --2- 172.21.15.60:0/1772645857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1a84063ad0 0x7f1a840afb40 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:51.487 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.475+0000 7f1a8ad26700 1 -- 172.21.15.60:0/1772645857 >> 172.21.15.60:0/1772645857 conn(0x7f1a840632c0 msgr2=0x7f1a840b4ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:51.487 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.475+0000 7f1a8ad26700 1 -- 172.21.15.60:0/1772645857 shutdown_connections 2022-06-29T16:01:51.487 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.475+0000 7f1a8ad26700 1 -- 172.21.15.60:0/1772645857 wait complete. 2022-06-29T16:01:51.496 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-06-29T16:01:51.497 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:01:51.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.599+0000 7fb3b370c700 1 Processor -- start 2022-06-29T16:01:51.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.599+0000 7fb3b370c700 1 -- start start 2022-06-29T16:01:51.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb3b370c700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb3ac13ce10 0x7fb3ac0aca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:51.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb3b370c700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3ac067be0 con 0x7fb3ac1460e0 2022-06-29T16:01:51.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb3b370c700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3ac0af140 con 0x7fb3ac067800 2022-06-29T16:01:51.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb3b370c700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3ac0af8a0 con 0x7fb3ac13ce10 2022-06-29T16:01:51.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb3b1ca9700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb3ac1460e0 0x7fb3ac146570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55842/0 (socket says 172.21.15.60:55842) 2022-06-29T16:01:51.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb3b1ca9700 1 -- 172.21.15.60:0/3809618293 learned_addr learned my addr 172.21.15.60:0/3809618293 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:01:51.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb3b0ca7700 1 --2- 172.21.15.60:0/3809618293 >> v2:172.21.15.92:3300/0 conn(0x7fb3ac13ce10 0x7fb3ac0aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:51.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb39bfff700 1 -- 172.21.15.60:0/3809618293 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 259484274 0 0) 0x7fb3ac067be0 con 0x7fb3ac1460e0 2022-06-29T16:01:51.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb39bfff700 1 -- 172.21.15.60:0/3809618293 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb394003260 con 0x7fb3ac1460e0 2022-06-29T16:01:51.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb39bfff700 1 -- 172.21.15.60:0/3809618293 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 761892737 0 0) 0x7fb3ac0af140 con 0x7fb3ac067800 2022-06-29T16:01:51.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb39bfff700 1 -- 172.21.15.60:0/3809618293 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3ac067be0 con 0x7fb3ac067800 2022-06-29T16:01:51.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb39bfff700 1 -- 172.21.15.60:0/3809618293 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2554048405 0 0) 0x7fb394003260 con 0x7fb3ac1460e0 2022-06-29T16:01:51.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb39bfff700 1 -- 172.21.15.60:0/3809618293 >> v2:172.21.15.92:3300/0 conn(0x7fb3ac13ce10 msgr2=0x7fb3ac0aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:01:51.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb39bfff700 1 --2- 172.21.15.60:0/3809618293 >> v2:172.21.15.92:3300/0 conn(0x7fb3ac13ce10 0x7fb3ac0aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:51.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb39bfff700 1 -- 172.21.15.60:0/3809618293 >> v1:172.21.15.92:6789/0 conn(0x7fb3ac067800 legacy=0x7fb3ac13c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:51.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb39bfff700 1 -- 172.21.15.60:0/3809618293 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ac0afb40 con 0x7fb3ac1460e0 2022-06-29T16:01:51.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb39bfff700 1 -- 172.21.15.60:0/3809618293 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7fb3a40019e0 con 0x7fb3ac1460e0 2022-06-29T16:01:51.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb3b0ca7700 1 --2- 172.21.15.60:0/3809618293 >> v2:172.21.15.92:3300/0 conn(0x7fb3ac13ce10 0x7fb3ac0aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:01:51.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb39bfff700 1 -- 172.21.15.60:0/3809618293 >> v1:172.21.15.60:6789/0 conn(0x7fb3ac1460e0 legacy=0x7fb3ac146570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:51.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb39bfff700 1 --2- 172.21.15.60:0/3809618293 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb394003930 0x7fb394003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:51.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb39bfff700 1 --2- 172.21.15.60:0/3809618293 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb394004500 0x7fb394004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:51.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb39bfff700 1 --2- 172.21.15.60:0/3809618293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb394004e70 0x7fb39400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:51.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb39bfff700 1 -- 172.21.15.60:0/3809618293 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ac0af8a0 con 0x7fb394004e70 2022-06-29T16:01:51.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb39bfff700 1 -- 172.21.15.60:0/3809618293 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39400bd20 con 0x7fb394003930 2022-06-29T16:01:51.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb39bfff700 1 -- 172.21.15.60:0/3809618293 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb39400bee0 con 0x7fb394004500 2022-06-29T16:01:51.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb3b1ca9700 1 --2- 172.21.15.60:0/3809618293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb394004e70 0x7fb39400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:51.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb3b14a8700 1 --2- 172.21.15.60:0/3809618293 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb394003930 0x7fb394003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:51.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb3b0ca7700 1 --2- 172.21.15.60:0/3809618293 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb394004500 0x7fb394004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:51.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb3b1ca9700 1 -- 172.21.15.60:0/3809618293 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb394004500 msgr2=0x7fb394004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:01:51.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb3b1ca9700 1 --2- 172.21.15.60:0/3809618293 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb394004500 0x7fb394004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:51.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb3b1ca9700 1 -- 172.21.15.60:0/3809618293 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb394003930 msgr2=0x7fb394003d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:01:51.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb3b1ca9700 1 --2- 172.21.15.60:0/3809618293 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb394003930 0x7fb394003d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:51.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb3b1ca9700 1 -- 172.21.15.60:0/3809618293 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb394003260 con 0x7fb394004e70 2022-06-29T16:01:51.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb3b0ca7700 1 --2- 172.21.15.60:0/3809618293 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb394004500 0x7fb394004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:01:51.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb3b1ca9700 1 -- 172.21.15.60:0/3809618293 reap_dead start 2022-06-29T16:01:51.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb3b1ca9700 1 --2- 172.21.15.60:0/3809618293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb394004e70 0x7fb39400b750 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7fb3a40017b0 tx=0x7fb3ac05bdd0).ready entity=mon.0 client_cookie=c387b9c2f94dd46c server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:51.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb39bfff700 1 -- 172.21.15.60:0/3809618293 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3a400cb90 con 0x7fb394004e70 2022-06-29T16:01:51.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb39bfff700 1 -- 172.21.15.60:0/3809618293 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3a400ccf0 con 0x7fb394004e70 2022-06-29T16:01:51.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb39bfff700 1 -- 172.21.15.60:0/3809618293 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3a4005b80 con 0x7fb394004e70 2022-06-29T16:01:51.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb3b370c700 1 -- 172.21.15.60:0/3809618293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb394004e70 msgr2=0x7fb39400b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:51.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb3b370c700 1 --2- 172.21.15.60:0/3809618293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb394004e70 0x7fb39400b750 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7fb3a40017b0 tx=0x7fb3ac05bdd0).stop 2022-06-29T16:01:51.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb3b370c700 1 -- 172.21.15.60:0/3809618293 shutdown_connections 2022-06-29T16:01:51.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb3b370c700 1 --2- 172.21.15.60:0/3809618293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb394004e70 0x7fb39400b750 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:51.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb3b370c700 1 -- 172.21.15.60:0/3809618293 >> 172.21.15.60:0/3809618293 conn(0x7fb3ac05f0b0 msgr2=0x7fb3ac05f490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:51.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb3b370c700 1 -- 172.21.15.60:0/3809618293 shutdown_connections 2022-06-29T16:01:51.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.603+0000 7fb3b370c700 1 -- 172.21.15.60:0/3809618293 wait complete. 2022-06-29T16:01:51.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.607+0000 7fb3b370c700 1 Processor -- start 2022-06-29T16:01:51.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.607+0000 7fb3b370c700 1 -- start start 2022-06-29T16:01:51.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.607+0000 7fb3b370c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb394004e70 0x7fb3ac142410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:51.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.607+0000 7fb3b370c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3ac13e2c0 0x7fb3ac13fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:51.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.607+0000 7fb3b370c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3ac151e40 0x7fb3ac1522f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:51.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.607+0000 7fb3b370c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ac140570 con 0x7fb3ac13e2c0 2022-06-29T16:01:51.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.607+0000 7fb3b370c700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ac1402d0 con 0x7fb3ac151e40 2022-06-29T16:01:51.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.607+0000 7fb3b370c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ac13f380 con 0x7fb394004e70 2022-06-29T16:01:51.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.607+0000 7fb3b0ca7700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3ac13e2c0 0x7fb3ac13fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:51.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.607+0000 7fb3b14a8700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb394004e70 0x7fb3ac142410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:51.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.607+0000 7fb3b0ca7700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3ac13e2c0 0x7fb3ac13fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47764/0 (socket says 172.21.15.60:47764) 2022-06-29T16:01:51.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.607+0000 7fb3b0ca7700 1 -- 172.21.15.60:0/881969298 learned_addr learned my addr 172.21.15.60:0/881969298 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:51.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.607+0000 7fb3b1ca9700 1 --2- 172.21.15.60:0/881969298 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3ac151e40 0x7fb3ac1522f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:51.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.607+0000 7fb3b0ca7700 1 -- 172.21.15.60:0/881969298 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb394004e70 msgr2=0x7fb3ac142410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:51.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.607+0000 7fb3b0ca7700 1 --2- 172.21.15.60:0/881969298 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb394004e70 0x7fb3ac142410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:51.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.607+0000 7fb3b0ca7700 1 -- 172.21.15.60:0/881969298 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3ac151e40 msgr2=0x7fb3ac1522f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:51.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.607+0000 7fb3b0ca7700 1 --2- 172.21.15.60:0/881969298 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3ac151e40 0x7fb3ac1522f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:51.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.607+0000 7fb3b0ca7700 1 -- 172.21.15.60:0/881969298 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ac13d370 con 0x7fb3ac13e2c0 2022-06-29T16:01:51.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.607+0000 7fb3b0ca7700 1 --2- 172.21.15.60:0/881969298 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3ac13e2c0 0x7fb3ac13fc80 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fb39c00b000 tx=0x7fb39c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:51.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.607+0000 7fb39a7fc700 1 -- 172.21.15.60:0/881969298 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb39c014750 con 0x7fb3ac13e2c0 2022-06-29T16:01:51.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.607+0000 7fb39a7fc700 1 -- 172.21.15.60:0/881969298 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb39c0148b0 con 0x7fb3ac13e2c0 2022-06-29T16:01:51.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.607+0000 7fb3b370c700 1 -- 172.21.15.60:0/881969298 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3ac1527e0 con 0x7fb3ac13e2c0 2022-06-29T16:01:51.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.607+0000 7fb39a7fc700 1 -- 172.21.15.60:0/881969298 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb39c014c10 con 0x7fb3ac13e2c0 2022-06-29T16:01:51.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.607+0000 7fb3b370c700 1 -- 172.21.15.60:0/881969298 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3ac155390 con 0x7fb3ac13e2c0 2022-06-29T16:01:51.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.611+0000 7fb3b370c700 1 -- 172.21.15.60:0/881969298 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3ac13f540 con 0x7fb3ac13e2c0 2022-06-29T16:01:51.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.611+0000 7fb39a7fc700 1 -- 172.21.15.60:0/881969298 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb39c014d70 con 0x7fb3ac13e2c0 2022-06-29T16:01:51.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.611+0000 7fb39a7fc700 1 --2- 172.21.15.60:0/881969298 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb38c053b30 0x7fb38c055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:51.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.611+0000 7fb39a7fc700 1 -- 172.21.15.60:0/881969298 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fb39c06beb0 con 0x7fb3ac13e2c0 2022-06-29T16:01:51.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.611+0000 7fb3b14a8700 1 --2- 172.21.15.60:0/881969298 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb38c053b30 0x7fb38c055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:51.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.611+0000 7fb3b14a8700 1 --2- 172.21.15.60:0/881969298 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb38c053b30 0x7fb38c055ff0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7fb3a8003000 tx=0x7fb3a8007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:51.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.619+0000 7fb39a7fc700 1 -- 172.21.15.60:0/881969298 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb39c043100 con 0x7fb3ac13e2c0 2022-06-29T16:01:51.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.743+0000 7fb3b370c700 1 -- 172.21.15.60:0/881969298 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fb3ac0573d0 con 0x7fb3ac13e2c0 2022-06-29T16:01:51.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.743+0000 7fb39a7fc700 1 -- 172.21.15.60:0/881969298 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v49) v1 ==== 110+0+10 (secure 0 0 0) 0x7fb39c046720 con 0x7fb3ac13e2c0 2022-06-29T16:01:51.753 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:01:51.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.747+0000 7fb3b370c700 1 -- 172.21.15.60:0/881969298 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb38c053b30 msgr2=0x7fb38c055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:51.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.747+0000 7fb3b370c700 1 --2- 172.21.15.60:0/881969298 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb38c053b30 0x7fb38c055ff0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7fb3a8003000 tx=0x7fb3a8007a00).stop 2022-06-29T16:01:51.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.747+0000 7fb3b370c700 1 -- 172.21.15.60:0/881969298 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3ac13e2c0 msgr2=0x7fb3ac13fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:51.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.747+0000 7fb3b370c700 1 --2- 172.21.15.60:0/881969298 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3ac13e2c0 0x7fb3ac13fc80 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fb39c00b000 tx=0x7fb39c008040).stop 2022-06-29T16:01:51.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.747+0000 7fb3b370c700 1 -- 172.21.15.60:0/881969298 shutdown_connections 2022-06-29T16:01:51.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.747+0000 7fb3b370c700 1 --2- 172.21.15.60:0/881969298 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb38c053b30 0x7fb38c055ff0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:51.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.747+0000 7fb3b370c700 1 --2- 172.21.15.60:0/881969298 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3ac151e40 0x7fb3ac1522f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:51.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.747+0000 7fb3b370c700 1 --2- 172.21.15.60:0/881969298 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3ac13e2c0 0x7fb3ac13fc80 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:51.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.747+0000 7fb3b370c700 1 --2- 172.21.15.60:0/881969298 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb394004e70 0x7fb3ac142410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:51.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.747+0000 7fb3b370c700 1 -- 172.21.15.60:0/881969298 >> 172.21.15.60:0/881969298 conn(0x7fb3ac05f0b0 msgr2=0x7fb3ac144540 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:51.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.747+0000 7fb3b370c700 1 -- 172.21.15.60:0/881969298 shutdown_connections 2022-06-29T16:01:51.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.747+0000 7fb3b370c700 1 -- 172.21.15.60:0/881969298 wait complete. 2022-06-29T16:01:51.770 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:01:51.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.867+0000 7f6264fb5700 1 Processor -- start 2022-06-29T16:01:51.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.867+0000 7f6264fb5700 1 -- start start 2022-06-29T16:01:51.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.867+0000 7f6264fb5700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f62600b5eb0 0x7f6260142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:51.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.867+0000 7f6264fb5700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62600b2c50 0x7f62600b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:51.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.867+0000 7f6264fb5700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62600b0cf0 con 0x7f62600b48b0 2022-06-29T16:01:51.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.867+0000 7f6264fb5700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62600b54e0 con 0x7f62600b5eb0 2022-06-29T16:01:51.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.867+0000 7f6264fb5700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62600b4d10 con 0x7f62600b2c50 2022-06-29T16:01:51.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.867+0000 7f625effd700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f62600b48b0 0x7f62600b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55860/0 (socket says 172.21.15.60:55860) 2022-06-29T16:01:51.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.867+0000 7f625effd700 1 -- 172.21.15.60:0/1289775364 learned_addr learned my addr 172.21.15.60:0/1289775364 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:01:51.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.867+0000 7f625ffff700 1 --2- 172.21.15.60:0/1289775364 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62600b2c50 0x7f62600b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:51.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.867+0000 7f625f7fe700 1 --2- 172.21.15.60:0/1289775364 >> v2:172.21.15.92:3300/0 conn(0x7f62600b5eb0 0x7f6260142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:51.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.867+0000 7f625e7fc700 1 -- 172.21.15.60:0/1289775364 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1376121858 0 0) 0x7f6254001110 con 0x7f62600b48b0 2022-06-29T16:01:51.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.867+0000 7f625e7fc700 1 -- 172.21.15.60:0/1289775364 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f624c003260 con 0x7f62600b48b0 2022-06-29T16:01:51.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.867+0000 7f625f7fe700 1 -- 172.21.15.60:0/1289775364 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62600b2c50 msgr2=0x7f62600b30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:01:51.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.867+0000 7f625f7fe700 1 --2- 172.21.15.60:0/1289775364 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62600b2c50 0x7f62600b30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:51.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.867+0000 7f625f7fe700 1 -- 172.21.15.60:0/1289775364 >> v1:172.21.15.60:6789/0 conn(0x7f62600b48b0 legacy=0x7f62600b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:51.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.867+0000 7f625f7fe700 1 -- 172.21.15.60:0/1289775364 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62600b6290 con 0x7f62600b5eb0 2022-06-29T16:01:51.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.867+0000 7f625ffff700 1 --2- 172.21.15.60:0/1289775364 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62600b2c50 0x7f62600b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:01:51.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.867+0000 7f625f7fe700 1 --2- 172.21.15.60:0/1289775364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62600b5eb0 0x7f6260142a20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f6250002f70 tx=0x7f6250006df0).ready entity=mon.1 client_cookie=a4edea1c6b94b675 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:51.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.871+0000 7f625e7fc700 1 -- 172.21.15.60:0/1289775364 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f625000d4a0 con 0x7f62600b5eb0 2022-06-29T16:01:51.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.871+0000 7f625e7fc700 1 -- 172.21.15.60:0/1289775364 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6250000e90 con 0x7f62600b5eb0 2022-06-29T16:01:51.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.871+0000 7f625e7fc700 1 -- 172.21.15.60:0/1289775364 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f625000d7f0 con 0x7f62600b5eb0 2022-06-29T16:01:51.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.871+0000 7f6264fb5700 1 -- 172.21.15.60:0/1289775364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62600b5eb0 msgr2=0x7f6260142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:51.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.871+0000 7f6264fb5700 1 --2- 172.21.15.60:0/1289775364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62600b5eb0 0x7f6260142a20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f6250002f70 tx=0x7f6250006df0).stop 2022-06-29T16:01:51.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.871+0000 7f6264fb5700 1 -- 172.21.15.60:0/1289775364 shutdown_connections 2022-06-29T16:01:51.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.871+0000 7f6264fb5700 1 --2- 172.21.15.60:0/1289775364 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62600b2c50 0x7f62600b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:51.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.871+0000 7f6264fb5700 1 --2- 172.21.15.60:0/1289775364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62600b5eb0 0x7f6260142a20 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:51.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.871+0000 7f6264fb5700 1 -- 172.21.15.60:0/1289775364 >> 172.21.15.60:0/1289775364 conn(0x7f6260062090 msgr2=0x7f6260062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:51.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.871+0000 7f6264fb5700 1 -- 172.21.15.60:0/1289775364 shutdown_connections 2022-06-29T16:01:51.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.871+0000 7f6264fb5700 1 -- 172.21.15.60:0/1289775364 wait complete. 2022-06-29T16:01:51.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.871+0000 7f6264fb5700 1 Processor -- start 2022-06-29T16:01:51.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.871+0000 7f6264fb5700 1 -- start start 2022-06-29T16:01:51.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.875+0000 7f6264fb5700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62600b2c50 0x7f6260140410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:51.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.875+0000 7f6264fb5700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62600b48b0 0x7f62601484c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:51.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.875+0000 7f6264fb5700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62600b5eb0 0x7f6260151f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:51.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.875+0000 7f6264fb5700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62600ae370 con 0x7f62600b48b0 2022-06-29T16:01:51.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.875+0000 7f6264fb5700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62600adf40 con 0x7f62600b2c50 2022-06-29T16:01:51.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.875+0000 7f6264fb5700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62600af4f0 con 0x7f62600b5eb0 2022-06-29T16:01:51.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.875+0000 7f625effd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62600b48b0 0x7f62601484c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:51.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.875+0000 7f625ffff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62600b5eb0 0x7f6260151f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:51.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.875+0000 7f625effd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62600b48b0 0x7f62601484c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47780/0 (socket says 172.21.15.60:47780) 2022-06-29T16:01:51.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.875+0000 7f625effd700 1 -- 172.21.15.60:0/1543677690 learned_addr learned my addr 172.21.15.60:0/1543677690 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:51.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.875+0000 7f625f7fe700 1 --2- 172.21.15.60:0/1543677690 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62600b2c50 0x7f6260140410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:51.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.875+0000 7f625effd700 1 -- 172.21.15.60:0/1543677690 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62600b5eb0 msgr2=0x7f6260151f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:51.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.875+0000 7f625effd700 1 --2- 172.21.15.60:0/1543677690 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62600b5eb0 0x7f6260151f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:51.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.875+0000 7f625effd700 1 -- 172.21.15.60:0/1543677690 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62600b2c50 msgr2=0x7f6260140410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:51.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.875+0000 7f625effd700 1 --2- 172.21.15.60:0/1543677690 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62600b2c50 0x7f6260140410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:51.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.875+0000 7f625effd700 1 -- 172.21.15.60:0/1543677690 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6258009470 con 0x7f62600b48b0 2022-06-29T16:01:51.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.875+0000 7f625ffff700 1 --2- 172.21.15.60:0/1543677690 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62600b5eb0 0x7f6260151f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:01:51.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.875+0000 7f625effd700 1 --2- 172.21.15.60:0/1543677690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62600b48b0 0x7f62601484c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f62600ae080 tx=0x7f62540078e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:51.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.875+0000 7f625cff9700 1 -- 172.21.15.60:0/1543677690 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62540070d0 con 0x7f62600b48b0 2022-06-29T16:01:51.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.875+0000 7f625cff9700 1 -- 172.21.15.60:0/1543677690 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6254007230 con 0x7f62600b48b0 2022-06-29T16:01:51.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.875+0000 7f6264fb5700 1 -- 172.21.15.60:0/1543677690 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f625000b180 con 0x7f62600b48b0 2022-06-29T16:01:51.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.875+0000 7f625cff9700 1 -- 172.21.15.60:0/1543677690 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6254007590 con 0x7f62600b48b0 2022-06-29T16:01:51.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.875+0000 7f6264fb5700 1 -- 172.21.15.60:0/1543677690 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62600acc50 con 0x7f62600b48b0 2022-06-29T16:01:51.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.879+0000 7f6264fb5700 1 -- 172.21.15.60:0/1543677690 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62600af6f0 con 0x7f62600b48b0 2022-06-29T16:01:51.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.879+0000 7f625cff9700 1 -- 172.21.15.60:0/1543677690 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6254010070 con 0x7f62600b48b0 2022-06-29T16:01:51.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.879+0000 7f625cff9700 1 --2- 172.21.15.60:0/1543677690 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6240053a70 0x7f6240055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:51.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.879+0000 7f625cff9700 1 -- 172.21.15.60:0/1543677690 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f625406b080 con 0x7f62600b48b0 2022-06-29T16:01:51.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.879+0000 7f625f7fe700 1 --2- 172.21.15.60:0/1543677690 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6240053a70 0x7f6240055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:51.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.879+0000 7f625f7fe700 1 --2- 172.21.15.60:0/1543677690 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6240053a70 0x7f6240055f30 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f625001a000 tx=0x7f6250014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:51.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:51.887+0000 7f625cff9700 1 -- 172.21.15.60:0/1543677690 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f62540421c0 con 0x7f62600b48b0 2022-06-29T16:01:52.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.011+0000 7f6264fb5700 1 -- 172.21.15.60:0/1543677690 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f62600b0cf0 con 0x7f62600b48b0 2022-06-29T16:01:52.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.011+0000 7f625cff9700 1 -- 172.21.15.60:0/1543677690 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v49) v1 ==== 92+0+10 (secure 0 0 0) 0x7f62600b0cf0 con 0x7f62600b48b0 2022-06-29T16:01:52.021 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:01:52.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.015+0000 7f6264fb5700 1 -- 172.21.15.60:0/1543677690 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6240053a70 msgr2=0x7f6240055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:52.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.015+0000 7f6264fb5700 1 --2- 172.21.15.60:0/1543677690 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6240053a70 0x7f6240055f30 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f625001a000 tx=0x7f6250014000).stop 2022-06-29T16:01:52.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.015+0000 7f6264fb5700 1 -- 172.21.15.60:0/1543677690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62600b48b0 msgr2=0x7f62601484c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:52.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.015+0000 7f6264fb5700 1 --2- 172.21.15.60:0/1543677690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62600b48b0 0x7f62601484c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f62600ae080 tx=0x7f62540078e0).stop 2022-06-29T16:01:52.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.015+0000 7f6264fb5700 1 -- 172.21.15.60:0/1543677690 shutdown_connections 2022-06-29T16:01:52.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.015+0000 7f6264fb5700 1 --2- 172.21.15.60:0/1543677690 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6240053a70 0x7f6240055f30 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:52.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.015+0000 7f6264fb5700 1 --2- 172.21.15.60:0/1543677690 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f62600b5eb0 0x7f6260151f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:52.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.015+0000 7f6264fb5700 1 --2- 172.21.15.60:0/1543677690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62600b48b0 0x7f62601484c0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:52.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.015+0000 7f6264fb5700 1 --2- 172.21.15.60:0/1543677690 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62600b2c50 0x7f6260140410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:52.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.015+0000 7f6264fb5700 1 -- 172.21.15.60:0/1543677690 >> 172.21.15.60:0/1543677690 conn(0x7f6260062090 msgr2=0x7f62600b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:52.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.015+0000 7f6264fb5700 1 -- 172.21.15.60:0/1543677690 shutdown_connections 2022-06-29T16:01:52.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.015+0000 7f6264fb5700 1 -- 172.21.15.60:0/1543677690 wait complete. 2022-06-29T16:01:52.034 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-06-29T16:01:52.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.123+0000 7fa8af03e700 1 Processor -- start 2022-06-29T16:01:52.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.123+0000 7fa8af03e700 1 -- start start 2022-06-29T16:01:52.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.127+0000 7fa8af03e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8a805d070 0x7fa8a814aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:52.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.127+0000 7fa8af03e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa8a80b2130 0x7fa8a80aca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:52.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.127+0000 7fa8af03e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a805d450 con 0x7fa8a814af80 2022-06-29T16:01:52.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.127+0000 7fa8af03e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8a80adf90 con 0x7fa8a80b2130 2022-06-29T16:01:52.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.127+0000 7fa8af03e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a80b25d0 con 0x7fa8a805d070 2022-06-29T16:01:52.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.127+0000 7fa8ad83b700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa8a80b2130 0x7fa8a80aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:52.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.127+0000 7fa8ae83d700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fa8a814af80 0x7fa8a80acf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:60124/0 (socket says 172.21.15.60:60124) 2022-06-29T16:01:52.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.127+0000 7fa8ae03c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8a805d070 0x7fa8a814aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:52.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.127+0000 7fa8ae83d700 1 -- 172.21.15.60:0/1615669172 learned_addr learned my addr 172.21.15.60:0/1615669172 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:01:52.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.127+0000 7fa8ad83b700 1 -- 172.21.15.60:0/1615669172 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8a805d070 msgr2=0x7fa8a814aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:01:52.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.127+0000 7fa8ad83b700 1 --2- 172.21.15.60:0/1615669172 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8a805d070 0x7fa8a814aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:52.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.127+0000 7fa8ad83b700 1 -- 172.21.15.60:0/1615669172 >> v1:172.21.15.92:6789/0 conn(0x7fa8a814af80 legacy=0x7fa8a80acf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:52.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.127+0000 7fa8ad83b700 1 -- 172.21.15.60:0/1615669172 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a814a350 con 0x7fa8a80b2130 2022-06-29T16:01:52.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.127+0000 7fa8ad83b700 1 --2- 172.21.15.60:0/1615669172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8a80b2130 0x7fa8a80aca20 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fa89c002f70 tx=0x7fa89c006df0).ready entity=mon.1 client_cookie=6992932685acf12a server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:52.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.127+0000 7fa8ad03a700 1 -- 172.21.15.60:0/1615669172 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa89c00d570 con 0x7fa8a80b2130 2022-06-29T16:01:52.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.127+0000 7fa8ad03a700 1 -- 172.21.15.60:0/1615669172 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa89c000e90 con 0x7fa8a80b2130 2022-06-29T16:01:52.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.127+0000 7fa8ad03a700 1 -- 172.21.15.60:0/1615669172 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa89c00d8c0 con 0x7fa8a80b2130 2022-06-29T16:01:52.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.127+0000 7fa8af03e700 1 -- 172.21.15.60:0/1615669172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8a80b2130 msgr2=0x7fa8a80aca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:52.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.127+0000 7fa8af03e700 1 --2- 172.21.15.60:0/1615669172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8a80b2130 0x7fa8a80aca20 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fa89c002f70 tx=0x7fa89c006df0).stop 2022-06-29T16:01:52.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.127+0000 7fa8af03e700 1 -- 172.21.15.60:0/1615669172 shutdown_connections 2022-06-29T16:01:52.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.127+0000 7fa8af03e700 1 --2- 172.21.15.60:0/1615669172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8a80b2130 0x7fa8a80aca20 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:52.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.127+0000 7fa8af03e700 1 --2- 172.21.15.60:0/1615669172 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8a805d070 0x7fa8a814aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:52.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.127+0000 7fa8af03e700 1 -- 172.21.15.60:0/1615669172 >> 172.21.15.60:0/1615669172 conn(0x7fa8a80b5890 msgr2=0x7fa8a80b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:52.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.127+0000 7fa8af03e700 1 -- 172.21.15.60:0/1615669172 shutdown_connections 2022-06-29T16:01:52.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.127+0000 7fa8af03e700 1 -- 172.21.15.60:0/1615669172 wait complete. 2022-06-29T16:01:52.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.131+0000 7fa8af03e700 1 Processor -- start 2022-06-29T16:01:52.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.131+0000 7fa8af03e700 1 -- start start 2022-06-29T16:01:52.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.131+0000 7fa8af03e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8a805d070 0x7fa8a8116920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:52.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.131+0000 7fa8af03e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8a80b2130 0x7fa8a80640d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:52.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.131+0000 7fa8af03e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8a814af80 0x7fa8a80645c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:52.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.131+0000 7fa8af03e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a8119180 con 0x7fa8a80b2130 2022-06-29T16:01:52.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.131+0000 7fa8af03e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a8119420 con 0x7fa8a814af80 2022-06-29T16:01:52.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.131+0000 7fa8af03e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a8066940 con 0x7fa8a805d070 2022-06-29T16:01:52.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.131+0000 7fa8ad83b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8a80b2130 0x7fa8a80640d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:52.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.131+0000 7fa8ad83b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8a80b2130 0x7fa8a80640d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47794/0 (socket says 172.21.15.60:47794) 2022-06-29T16:01:52.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.131+0000 7fa8ad83b700 1 -- 172.21.15.60:0/6070066 learned_addr learned my addr 172.21.15.60:0/6070066 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:52.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.131+0000 7fa8ae83d700 1 --2- 172.21.15.60:0/6070066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8a814af80 0x7fa8a80645c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:52.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.131+0000 7fa8ae03c700 1 --2- 172.21.15.60:0/6070066 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8a805d070 0x7fa8a8116920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:52.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.135+0000 7fa8ad83b700 1 -- 172.21.15.60:0/6070066 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8a805d070 msgr2=0x7fa8a8116920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:52.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.135+0000 7fa8ad83b700 1 --2- 172.21.15.60:0/6070066 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8a805d070 0x7fa8a8116920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:52.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.135+0000 7fa8ad83b700 1 -- 172.21.15.60:0/6070066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8a814af80 msgr2=0x7fa8a80645c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:52.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.135+0000 7fa8ad83b700 1 --2- 172.21.15.60:0/6070066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8a814af80 0x7fa8a80645c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:52.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.135+0000 7fa8ad83b700 1 -- 172.21.15.60:0/6070066 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa89c00b180 con 0x7fa8a80b2130 2022-06-29T16:01:52.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.135+0000 7fa8ad83b700 1 --2- 172.21.15.60:0/6070066 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8a80b2130 0x7fa8a80640d0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7fa89c002f70 tx=0x7fa89c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:52.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.135+0000 7fa89b7fe700 1 -- 172.21.15.60:0/6070066 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa89c016070 con 0x7fa8a80b2130 2022-06-29T16:01:52.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.135+0000 7fa89b7fe700 1 -- 172.21.15.60:0/6070066 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa89c00d8f0 con 0x7fa8a80b2130 2022-06-29T16:01:52.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.135+0000 7fa8af03e700 1 -- 172.21.15.60:0/6070066 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8a8116e10 con 0x7fa8a80b2130 2022-06-29T16:01:52.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.135+0000 7fa89b7fe700 1 -- 172.21.15.60:0/6070066 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa89c013de0 con 0x7fa8a80b2130 2022-06-29T16:01:52.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.135+0000 7fa8af03e700 1 -- 172.21.15.60:0/6070066 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8a8066de0 con 0x7fa8a80b2130 2022-06-29T16:01:52.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.135+0000 7fa8af03e700 1 -- 172.21.15.60:0/6070066 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8a814a5f0 con 0x7fa8a80b2130 2022-06-29T16:01:52.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.135+0000 7fa89b7fe700 1 -- 172.21.15.60:0/6070066 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa89c00f030 con 0x7fa8a80b2130 2022-06-29T16:01:52.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.135+0000 7fa89b7fe700 1 --2- 172.21.15.60:0/6070066 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa888053b00 0x7fa888055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:52.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.139+0000 7fa89b7fe700 1 -- 172.21.15.60:0/6070066 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fa89c06c870 con 0x7fa8a80b2130 2022-06-29T16:01:52.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.139+0000 7fa8ae03c700 1 --2- 172.21.15.60:0/6070066 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa888053b00 0x7fa888055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:52.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.139+0000 7fa8ae03c700 1 --2- 172.21.15.60:0/6070066 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa888053b00 0x7fa888055fc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fa8a400b000 tx=0x7fa8a4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:52.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.143+0000 7fa89b7fe700 1 -- 172.21.15.60:0/6070066 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa89c043a20 con 0x7fa8a80b2130 2022-06-29T16:01:52.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.271+0000 7fa8af03e700 1 -- 172.21.15.60:0/6070066 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa8a8066b00 con 0x7fa8a80b2130 2022-06-29T16:01:52.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.275+0000 7fa89b7fe700 1 -- 172.21.15.60:0/6070066 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v49) v1 ==== 84+0+143 (secure 0 0 0) 0x7fa89c023070 con 0x7fa8a80b2130 2022-06-29T16:01:52.282 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:01:52.282 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2022-06-29T16:01:52.283 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:52.283 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2022-06-29T16:01:52.283 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": "device_health_metrics" 2022-06-29T16:01:52.283 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:01:52.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:01:52.284 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 2, 2022-06-29T16:01:52.284 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": "rbd" 2022-06-29T16:01:52.284 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:01:52.285 INFO:teuthology.orchestra.run.smithi060.stdout:] 2022-06-29T16:01:52.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.275+0000 7fa8af03e700 1 -- 172.21.15.60:0/6070066 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa888053b00 msgr2=0x7fa888055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:52.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.275+0000 7fa8af03e700 1 --2- 172.21.15.60:0/6070066 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa888053b00 0x7fa888055fc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fa8a400b000 tx=0x7fa8a4006df0).stop 2022-06-29T16:01:52.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.275+0000 7fa8af03e700 1 -- 172.21.15.60:0/6070066 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8a80b2130 msgr2=0x7fa8a80640d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:52.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.275+0000 7fa8af03e700 1 --2- 172.21.15.60:0/6070066 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8a80b2130 0x7fa8a80640d0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7fa89c002f70 tx=0x7fa89c014000).stop 2022-06-29T16:01:52.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.275+0000 7fa8af03e700 1 -- 172.21.15.60:0/6070066 shutdown_connections 2022-06-29T16:01:52.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.275+0000 7fa8af03e700 1 --2- 172.21.15.60:0/6070066 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa888053b00 0x7fa888055fc0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:52.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.275+0000 7fa8af03e700 1 --2- 172.21.15.60:0/6070066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8a814af80 0x7fa8a80645c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:52.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.275+0000 7fa8af03e700 1 --2- 172.21.15.60:0/6070066 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8a80b2130 0x7fa8a80640d0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:52.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.275+0000 7fa8af03e700 1 --2- 172.21.15.60:0/6070066 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8a805d070 0x7fa8a8116920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:52.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.275+0000 7fa8af03e700 1 -- 172.21.15.60:0/6070066 >> 172.21.15.60:0/6070066 conn(0x7fa8a80b5890 msgr2=0x7fa8a8143620 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:52.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.275+0000 7fa8af03e700 1 -- 172.21.15.60:0/6070066 shutdown_connections 2022-06-29T16:01:52.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.275+0000 7fa8af03e700 1 -- 172.21.15.60:0/6070066 wait complete. 2022-06-29T16:01:52.294 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-06-29T16:01:52.294 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-06-29T16:01:52.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.391+0000 7f879bfff700 1 Processor -- start 2022-06-29T16:01:52.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.391+0000 7f879bfff700 1 -- start start 2022-06-29T16:01:52.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.391+0000 7f879bfff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f879c065450 0x7f879c0b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:52.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.391+0000 7f879bfff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f879c060760 0x7f879c140180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:52.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.391+0000 7f879bfff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f879c140670 con 0x7f879c0b28e0 2022-06-29T16:01:52.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.391+0000 7f879bfff700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f879c13fa40 con 0x7f879c065450 2022-06-29T16:01:52.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.391+0000 7f879bfff700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f879c13fce0 con 0x7f879c060760 2022-06-29T16:01:52.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.391+0000 7f879a7fc700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f879c065450 0x7f879c0b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:52.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.391+0000 7f879b7fe700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f879c060760 0x7f879c140180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:52.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.391+0000 7f879affd700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f879c0b28e0 0x7f879c139340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:60134/0 (socket says 172.21.15.60:60134) 2022-06-29T16:01:52.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.391+0000 7f879affd700 1 -- 172.21.15.60:0/4044897559 learned_addr learned my addr 172.21.15.60:0/4044897559 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:01:52.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.391+0000 7f879b7fe700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f879c060760 0x7f879c140180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:53832/0 (socket says 172.21.15.60:53832) 2022-06-29T16:01:52.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.391+0000 7f8799ffb700 1 -- 172.21.15.60:0/4044897559 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 504675256 0 0) 0x7f879c140670 con 0x7f879c0b28e0 2022-06-29T16:01:52.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.391+0000 7f8799ffb700 1 -- 172.21.15.60:0/4044897559 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f877c003260 con 0x7f879c0b28e0 2022-06-29T16:01:52.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.391+0000 7f879a7fc700 1 -- 172.21.15.60:0/4044897559 >> v2:172.21.15.92:3300/0 conn(0x7f879c060760 msgr2=0x7f879c140180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:01:52.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.391+0000 7f879a7fc700 1 --2- 172.21.15.60:0/4044897559 >> v2:172.21.15.92:3300/0 conn(0x7f879c060760 0x7f879c140180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:52.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.391+0000 7f879a7fc700 1 -- 172.21.15.60:0/4044897559 >> v1:172.21.15.92:6789/0 conn(0x7f879c0b28e0 legacy=0x7f879c139340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:52.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.391+0000 7f879a7fc700 1 -- 172.21.15.60:0/4044897559 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f879c140860 con 0x7f879c065450 2022-06-29T16:01:52.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.391+0000 7f879a7fc700 1 --2- 172.21.15.60:0/4044897559 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f879c065450 0x7f879c0b2cc0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f8784002f70 tx=0x7f8784006df0).ready entity=mon.0 client_cookie=7348e3131f461afd server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:52.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.391+0000 7f8799ffb700 1 -- 172.21.15.60:0/4044897559 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f878400d4f0 con 0x7f879c065450 2022-06-29T16:01:52.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.391+0000 7f8799ffb700 1 -- 172.21.15.60:0/4044897559 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8784000e90 con 0x7f879c065450 2022-06-29T16:01:52.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.391+0000 7f8799ffb700 1 -- 172.21.15.60:0/4044897559 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f878400d840 con 0x7f879c065450 2022-06-29T16:01:52.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.395+0000 7f879bfff700 1 -- 172.21.15.60:0/4044897559 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f879c065450 msgr2=0x7f879c0b2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:52.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.395+0000 7f879bfff700 1 --2- 172.21.15.60:0/4044897559 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f879c065450 0x7f879c0b2cc0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f8784002f70 tx=0x7f8784006df0).stop 2022-06-29T16:01:52.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.395+0000 7f879bfff700 1 -- 172.21.15.60:0/4044897559 shutdown_connections 2022-06-29T16:01:52.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.395+0000 7f879bfff700 1 --2- 172.21.15.60:0/4044897559 >> v2:172.21.15.92:3300/0 conn(0x7f879c060760 0x7f879c140180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:52.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.395+0000 7f879bfff700 1 --2- 172.21.15.60:0/4044897559 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f879c065450 0x7f879c0b2cc0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:52.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.395+0000 7f879bfff700 1 -- 172.21.15.60:0/4044897559 >> 172.21.15.60:0/4044897559 conn(0x7f879c061c80 msgr2=0x7f879c0637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:52.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.395+0000 7f879bfff700 1 -- 172.21.15.60:0/4044897559 shutdown_connections 2022-06-29T16:01:52.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.395+0000 7f879bfff700 1 -- 172.21.15.60:0/4044897559 wait complete. 2022-06-29T16:01:52.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.395+0000 7f879bfff700 1 Processor -- start 2022-06-29T16:01:52.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.395+0000 7f879bfff700 1 -- start start 2022-06-29T16:01:52.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.395+0000 7f879bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f879c060760 0x7f879c13c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:52.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.395+0000 7f879bfff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f879c065450 0x7f879c138720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:52.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.395+0000 7f879bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f879c0b28e0 0x7f879c138c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:52.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.395+0000 7f879bfff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f879c145670 con 0x7f879c060760 2022-06-29T16:01:52.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.395+0000 7f879bfff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f879c145910 con 0x7f879c0b28e0 2022-06-29T16:01:52.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.395+0000 7f879bfff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f879c145bb0 con 0x7f879c065450 2022-06-29T16:01:52.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.399+0000 7f879affd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f879c060760 0x7f879c13c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:52.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.399+0000 7f879affd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f879c060760 0x7f879c13c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47806/0 (socket says 172.21.15.60:47806) 2022-06-29T16:01:52.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.399+0000 7f879affd700 1 -- 172.21.15.60:0/578222533 learned_addr learned my addr 172.21.15.60:0/578222533 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:52.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.399+0000 7f879b7fe700 1 --2- 172.21.15.60:0/578222533 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f879c0b28e0 0x7f879c138c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:52.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.399+0000 7f879a7fc700 1 --2- 172.21.15.60:0/578222533 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f879c065450 0x7f879c138720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:52.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.399+0000 7f879affd700 1 -- 172.21.15.60:0/578222533 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f879c065450 msgr2=0x7f879c138720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:52.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.399+0000 7f879affd700 1 --2- 172.21.15.60:0/578222533 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f879c065450 0x7f879c138720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:52.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.399+0000 7f879affd700 1 -- 172.21.15.60:0/578222533 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f879c0b28e0 msgr2=0x7f879c138c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:52.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.399+0000 7f879affd700 1 --2- 172.21.15.60:0/578222533 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f879c0b28e0 0x7f879c138c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:52.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.399+0000 7f879affd700 1 -- 172.21.15.60:0/578222533 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f878400b180 con 0x7f879c060760 2022-06-29T16:01:52.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.399+0000 7f879b7fe700 1 --2- 172.21.15.60:0/578222533 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f879c0b28e0 0x7f879c138c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:01:52.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.399+0000 7f879affd700 1 --2- 172.21.15.60:0/578222533 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f879c060760 0x7f879c13c3d0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f8794001490 tx=0x7f8794007e50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:52.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.399+0000 7f8793fff700 1 -- 172.21.15.60:0/578222533 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8794005ba0 con 0x7f879c060760 2022-06-29T16:01:52.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.399+0000 7f8793fff700 1 -- 172.21.15.60:0/578222533 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8794005d00 con 0x7f879c060760 2022-06-29T16:01:52.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.399+0000 7f879bfff700 1 -- 172.21.15.60:0/578222533 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f879c143ad0 con 0x7f879c060760 2022-06-29T16:01:52.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.399+0000 7f8793fff700 1 -- 172.21.15.60:0/578222533 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f87940072c0 con 0x7f879c060760 2022-06-29T16:01:52.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.399+0000 7f879bfff700 1 -- 172.21.15.60:0/578222533 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f879c146920 con 0x7f879c060760 2022-06-29T16:01:52.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.399+0000 7f8793fff700 1 -- 172.21.15.60:0/578222533 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f879401a070 con 0x7f879c060760 2022-06-29T16:01:52.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.399+0000 7f879bfff700 1 -- 172.21.15.60:0/578222533 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f879c146c00 con 0x7f879c060760 2022-06-29T16:01:52.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.403+0000 7f8793fff700 1 --2- 172.21.15.60:0/578222533 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8780053a70 0x7f8780055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:52.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.403+0000 7f8793fff700 1 -- 172.21.15.60:0/578222533 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f879406b1f0 con 0x7f879c060760 2022-06-29T16:01:52.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.403+0000 7f879a7fc700 1 --2- 172.21.15.60:0/578222533 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8780053a70 0x7f8780055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:52.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.403+0000 7f879a7fc700 1 --2- 172.21.15.60:0/578222533 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8780053a70 0x7f8780055f30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f878401a000 tx=0x7f8784014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:52.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.407+0000 7f8793fff700 1 -- 172.21.15.60:0/578222533 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8794042440 con 0x7f879c060760 2022-06-29T16:01:52.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:52.531+0000 7f879bfff700 1 -- 172.21.15.60:0/578222533 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f879c145d70 con 0x7f879c060760 2022-06-29T16:01:53.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.395+0000 7f8793fff700 1 -- 172.21.15.60:0/578222533 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v50) v1 ==== 133+0+0 (secure 0 0 0) 0x7f8794045a60 con 0x7f879c060760 2022-06-29T16:01:53.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.399+0000 7f879bfff700 1 -- 172.21.15.60:0/578222533 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8780053a70 msgr2=0x7f8780055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:53.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.399+0000 7f879bfff700 1 --2- 172.21.15.60:0/578222533 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8780053a70 0x7f8780055f30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f878401a000 tx=0x7f8784014000).stop 2022-06-29T16:01:53.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.399+0000 7f879bfff700 1 -- 172.21.15.60:0/578222533 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f879c060760 msgr2=0x7f879c13c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:53.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.399+0000 7f879bfff700 1 --2- 172.21.15.60:0/578222533 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f879c060760 0x7f879c13c3d0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f8794001490 tx=0x7f8794007e50).stop 2022-06-29T16:01:53.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.399+0000 7f879bfff700 1 -- 172.21.15.60:0/578222533 shutdown_connections 2022-06-29T16:01:53.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.399+0000 7f879bfff700 1 --2- 172.21.15.60:0/578222533 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8780053a70 0x7f8780055f30 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:53.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.399+0000 7f879bfff700 1 --2- 172.21.15.60:0/578222533 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f879c0b28e0 0x7f879c138c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:53.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.399+0000 7f879bfff700 1 --2- 172.21.15.60:0/578222533 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f879c065450 0x7f879c138720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:53.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.399+0000 7f879bfff700 1 --2- 172.21.15.60:0/578222533 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f879c060760 0x7f879c13c3d0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:53.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.399+0000 7f879bfff700 1 -- 172.21.15.60:0/578222533 >> 172.21.15.60:0/578222533 conn(0x7f879c061c80 msgr2=0x7f879c0b6630 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:53.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.403+0000 7f879bfff700 1 -- 172.21.15.60:0/578222533 shutdown_connections 2022-06-29T16:01:53.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.403+0000 7f879bfff700 1 -- 172.21.15.60:0/578222533 wait complete. 2022-06-29T16:01:53.411 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2022-06-29T16:01:53.422 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-06-29T16:01:53.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.531+0000 7fefe7fff700 1 Processor -- start 2022-06-29T16:01:53.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.531+0000 7fefe7fff700 1 -- start start 2022-06-29T16:01:53.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.531+0000 7fefe7fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fefe80b2130 0x7fefe8146100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:53.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.531+0000 7fefe7fff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe805d450 con 0x7fefe805d070 2022-06-29T16:01:53.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.531+0000 7fefe7fff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe80b25d0 con 0x7fefe80aed00 2022-06-29T16:01:53.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.531+0000 7fefe7fff700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefe80adf90 con 0x7fefe80b2130 2022-06-29T16:01:53.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.531+0000 7fefe6ffd700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fefe805d070 0x7fefe814aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55900/0 (socket says 172.21.15.60:55900) 2022-06-29T16:01:53.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.531+0000 7fefe6ffd700 1 -- 172.21.15.60:0/862796234 learned_addr learned my addr 172.21.15.60:0/862796234 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:01:53.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.531+0000 7fefe67fc700 1 --2- 172.21.15.60:0/862796234 >> v2:172.21.15.92:3300/0 conn(0x7fefe80b2130 0x7fefe8146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:53.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.531+0000 7fefe5ffb700 1 -- 172.21.15.60:0/862796234 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 848814521 0 0) 0x7fefe805d450 con 0x7fefe805d070 2022-06-29T16:01:53.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.531+0000 7fefe5ffb700 1 -- 172.21.15.60:0/862796234 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefc8003260 con 0x7fefe805d070 2022-06-29T16:01:53.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.531+0000 7fefe67fc700 1 -- 172.21.15.60:0/862796234 >> v1:172.21.15.92:6789/0 conn(0x7fefe80aed00 legacy=0x7fefe81465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:53.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.531+0000 7fefe67fc700 1 -- 172.21.15.60:0/862796234 >> v1:172.21.15.60:6789/0 conn(0x7fefe805d070 legacy=0x7fefe814aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:53.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.531+0000 7fefe67fc700 1 -- 172.21.15.60:0/862796234 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe814a350 con 0x7fefe80b2130 2022-06-29T16:01:53.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.531+0000 7fefe5ffb700 1 -- 172.21.15.60:0/862796234 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3405031023 0 0) 0x7fefc8003260 con 0x7fefe805d070 2022-06-29T16:01:53.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.531+0000 7fefe67fc700 1 --2- 172.21.15.60:0/862796234 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fefe80b2130 0x7fefe8146100 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fefd8001800 tx=0x7fefd0006df0).ready entity=mon.1 client_cookie=b9fc5e0267114030 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:53.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.531+0000 7fefe5ffb700 1 -- 172.21.15.60:0/862796234 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fefd8001660 con 0x7fefe80b2130 2022-06-29T16:01:53.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.531+0000 7fefe5ffb700 1 -- 172.21.15.60:0/862796234 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefd0000e90 con 0x7fefe80b2130 2022-06-29T16:01:53.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.531+0000 7fefe5ffb700 1 -- 172.21.15.60:0/862796234 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fefd000d4f0 con 0x7fefe80b2130 2022-06-29T16:01:53.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.531+0000 7fefe7fff700 1 -- 172.21.15.60:0/862796234 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fefe80b2130 msgr2=0x7fefe8146100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:53.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.531+0000 7fefe7fff700 1 --2- 172.21.15.60:0/862796234 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fefe80b2130 0x7fefe8146100 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fefd8001800 tx=0x7fefd0006df0).stop 2022-06-29T16:01:53.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.535+0000 7fefe7fff700 1 -- 172.21.15.60:0/862796234 shutdown_connections 2022-06-29T16:01:53.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.535+0000 7fefe7fff700 1 --2- 172.21.15.60:0/862796234 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fefe80b2130 0x7fefe8146100 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:53.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.535+0000 7fefe7fff700 1 -- 172.21.15.60:0/862796234 >> 172.21.15.60:0/862796234 conn(0x7fefe80b5890 msgr2=0x7fefe80b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:53.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.535+0000 7fefe7fff700 1 -- 172.21.15.60:0/862796234 shutdown_connections 2022-06-29T16:01:53.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.535+0000 7fefe7fff700 1 -- 172.21.15.60:0/862796234 wait complete. 2022-06-29T16:01:53.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.535+0000 7fefe7fff700 1 Processor -- start 2022-06-29T16:01:53.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.535+0000 7fefe7fff700 1 -- start start 2022-06-29T16:01:53.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.535+0000 7fefe7fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fefe805d070 0x7fefe8116b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:53.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.535+0000 7fefe7fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fefe80aed00 0x7fefe8142130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:53.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.535+0000 7fefe7fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe80b2130 0x7fefe8142620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:53.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.535+0000 7fefe7fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe81190f0 con 0x7fefe80b2130 2022-06-29T16:01:53.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.535+0000 7fefe7fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe8066da0 con 0x7fefe80aed00 2022-06-29T16:01:53.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.535+0000 7fefe7fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefe8066150 con 0x7fefe805d070 2022-06-29T16:01:53.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.535+0000 7fefe77fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe80b2130 0x7fefe8142620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:53.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.535+0000 7fefe77fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe80b2130 0x7fefe8142620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47824/0 (socket says 172.21.15.60:47824) 2022-06-29T16:01:53.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.535+0000 7fefe77fe700 1 -- 172.21.15.60:0/1866207614 learned_addr learned my addr 172.21.15.60:0/1866207614 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:53.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.535+0000 7fefe6ffd700 1 --2- 172.21.15.60:0/1866207614 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fefe805d070 0x7fefe8116b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:53.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.535+0000 7fefe67fc700 1 --2- 172.21.15.60:0/1866207614 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fefe80aed00 0x7fefe8142130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:53.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.535+0000 7fefe77fe700 1 -- 172.21.15.60:0/1866207614 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fefe805d070 msgr2=0x7fefe8116b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:53.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.535+0000 7fefe77fe700 1 --2- 172.21.15.60:0/1866207614 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fefe805d070 0x7fefe8116b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:53.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.535+0000 7fefe77fe700 1 -- 172.21.15.60:0/1866207614 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fefe80aed00 msgr2=0x7fefe8142130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:53.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.535+0000 7fefe77fe700 1 --2- 172.21.15.60:0/1866207614 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fefe80aed00 0x7fefe8142130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:53.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.535+0000 7fefe77fe700 1 -- 172.21.15.60:0/1866207614 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd000b180 con 0x7fefe80b2130 2022-06-29T16:01:53.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.535+0000 7fefe77fe700 1 --2- 172.21.15.60:0/1866207614 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe80b2130 0x7fefe8142620 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7fefe81185e0 tx=0x7fefe0007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:53.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.535+0000 7fefdffff700 1 -- 172.21.15.60:0/1866207614 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fefe000f470 con 0x7fefe80b2130 2022-06-29T16:01:53.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.535+0000 7fefdffff700 1 -- 172.21.15.60:0/1866207614 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefe000f5d0 con 0x7fefe80b2130 2022-06-29T16:01:53.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.535+0000 7fefe7fff700 1 -- 172.21.15.60:0/1866207614 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefe8066b00 con 0x7fefe80b2130 2022-06-29T16:01:53.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.535+0000 7fefdffff700 1 -- 172.21.15.60:0/1866207614 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fefe000f930 con 0x7fefe80b2130 2022-06-29T16:01:53.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.535+0000 7fefe7fff700 1 -- 172.21.15.60:0/1866207614 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefe8065a60 con 0x7fefe80b2130 2022-06-29T16:01:53.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.539+0000 7fefe7fff700 1 -- 172.21.15.60:0/1866207614 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefe814a5f0 con 0x7fefe80b2130 2022-06-29T16:01:53.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.539+0000 7fefdffff700 1 -- 172.21.15.60:0/1866207614 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fefe0019030 con 0x7fefe80b2130 2022-06-29T16:01:53.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.539+0000 7fefdffff700 1 --2- 172.21.15.60:0/1866207614 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fefcc053a70 0x7fefcc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:53.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.539+0000 7fefdffff700 1 -- 172.21.15.60:0/1866207614 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5607+0+0 (secure 0 0 0) 0x7fefe006b440 con 0x7fefe80b2130 2022-06-29T16:01:53.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.539+0000 7fefe6ffd700 1 --2- 172.21.15.60:0/1866207614 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fefcc053a70 0x7fefcc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:53.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.543+0000 7fefe6ffd700 1 --2- 172.21.15.60:0/1866207614 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fefcc053a70 0x7fefcc055f30 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fefd800d000 tx=0x7fefd8000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:53.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.547+0000 7fefdffff700 1 -- 172.21.15.60:0/1866207614 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fefe00423c0 con 0x7fefe80b2130 2022-06-29T16:01:53.682 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:53.675+0000 7fefe7fff700 1 -- 172.21.15.60:0/1866207614 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7fefe8065cb0 con 0x7fefe80b2130 2022-06-29T16:01:54.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.407+0000 7fefdffff700 1 -- 172.21.15.60:0/1866207614 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v51) v1 ==== 125+0+0 (secure 0 0 0) 0x7fefe00459e0 con 0x7fefe80b2130 2022-06-29T16:01:54.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.415+0000 7fefe7fff700 1 -- 172.21.15.60:0/1866207614 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fefcc053a70 msgr2=0x7fefcc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:54.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.415+0000 7fefe7fff700 1 --2- 172.21.15.60:0/1866207614 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fefcc053a70 0x7fefcc055f30 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fefd800d000 tx=0x7fefd8000bc0).stop 2022-06-29T16:01:54.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.415+0000 7fefe7fff700 1 -- 172.21.15.60:0/1866207614 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe80b2130 msgr2=0x7fefe8142620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:54.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.415+0000 7fefe7fff700 1 --2- 172.21.15.60:0/1866207614 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe80b2130 0x7fefe8142620 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7fefe81185e0 tx=0x7fefe0007f40).stop 2022-06-29T16:01:54.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.415+0000 7fefe7fff700 1 -- 172.21.15.60:0/1866207614 shutdown_connections 2022-06-29T16:01:54.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.415+0000 7fefe7fff700 1 --2- 172.21.15.60:0/1866207614 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fefcc053a70 0x7fefcc055f30 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:54.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.415+0000 7fefe7fff700 1 --2- 172.21.15.60:0/1866207614 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefe80b2130 0x7fefe8142620 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:54.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.415+0000 7fefe7fff700 1 --2- 172.21.15.60:0/1866207614 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fefe80aed00 0x7fefe8142130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:54.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.415+0000 7fefe7fff700 1 --2- 172.21.15.60:0/1866207614 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fefe805d070 0x7fefe8116b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:54.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.415+0000 7fefe7fff700 1 -- 172.21.15.60:0/1866207614 >> 172.21.15.60:0/1866207614 conn(0x7fefe80b5890 msgr2=0x7fefe8145420 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:54.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.415+0000 7fefe7fff700 1 -- 172.21.15.60:0/1866207614 shutdown_connections 2022-06-29T16:01:54.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.415+0000 7fefe7fff700 1 -- 172.21.15.60:0/1866207614 wait complete. 2022-06-29T16:01:54.428 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2022-06-29T16:01:54.437 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-06-29T16:01:54.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.539+0000 7f78eb669700 1 Processor -- start 2022-06-29T16:01:54.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.539+0000 7f78eb669700 1 -- start start 2022-06-29T16:01:54.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.543+0000 7f78eb669700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f78e405c2f0 0x7f78e4146240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:54.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.543+0000 7f78eb669700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78e405cf60 0x7f78e4062f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:54.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.543+0000 7f78eb669700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78e4142bd0 con 0x7f78e4061870 2022-06-29T16:01:54.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.543+0000 7f78eb669700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78e4062050 con 0x7f78e405c2f0 2022-06-29T16:01:54.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.543+0000 7f78eb669700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78e40622f0 con 0x7f78e405cf60 2022-06-29T16:01:54.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.543+0000 7f78e8c04700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f78e405c2f0 0x7f78e4146240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:54.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.543+0000 7f78e8c04700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f78e405c2f0 0x7f78e4146240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47828/0 (socket says 172.21.15.60:47828) 2022-06-29T16:01:54.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.543+0000 7f78e8c04700 1 -- 172.21.15.60:0/4065817522 learned_addr learned my addr 172.21.15.60:0/4065817522 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:54.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.543+0000 7f78e9c06700 1 --2- 172.21.15.60:0/4065817522 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78e405cf60 0x7f78e4062f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:54.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.543+0000 7f78e8c04700 1 -- 172.21.15.60:0/4065817522 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78e405cf60 msgr2=0x7f78e4062f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:01:54.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.543+0000 7f78e8c04700 1 --2- 172.21.15.60:0/4065817522 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78e405cf60 0x7f78e4062f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:54.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.543+0000 7f78e8c04700 1 -- 172.21.15.60:0/4065817522 >> v1:172.21.15.92:6789/0 conn(0x7f78e4061870 legacy=0x7f78e4066240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:54.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.543+0000 7f78e8c04700 1 -- 172.21.15.60:0/4065817522 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78e4061c50 con 0x7f78e405c2f0 2022-06-29T16:01:54.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.543+0000 7f78e9c06700 1 --2- 172.21.15.60:0/4065817522 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78e405cf60 0x7f78e4062f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:01:54.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.543+0000 7f78e8c04700 1 --2- 172.21.15.60:0/4065817522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78e405c2f0 0x7f78e4146240 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f78e0002f70 tx=0x7f78e0006eb0).ready entity=mon.0 client_cookie=9b5fcfbe7102f37c server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:54.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.543+0000 7f78dbfff700 1 -- 172.21.15.60:0/4065817522 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f78e000d5c0 con 0x7f78e405c2f0 2022-06-29T16:01:54.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.543+0000 7f78dbfff700 1 -- 172.21.15.60:0/4065817522 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78e0000e90 con 0x7f78e405c2f0 2022-06-29T16:01:54.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.543+0000 7f78dbfff700 1 -- 172.21.15.60:0/4065817522 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f78e000d950 con 0x7f78e405c2f0 2022-06-29T16:01:54.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.543+0000 7f78eb669700 1 -- 172.21.15.60:0/4065817522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78e405c2f0 msgr2=0x7f78e4146240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:54.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.543+0000 7f78eb669700 1 --2- 172.21.15.60:0/4065817522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78e405c2f0 0x7f78e4146240 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f78e0002f70 tx=0x7f78e0006eb0).stop 2022-06-29T16:01:54.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.543+0000 7f78eb669700 1 -- 172.21.15.60:0/4065817522 shutdown_connections 2022-06-29T16:01:54.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.543+0000 7f78eb669700 1 --2- 172.21.15.60:0/4065817522 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78e405cf60 0x7f78e4062f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:54.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.543+0000 7f78eb669700 1 --2- 172.21.15.60:0/4065817522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78e405c2f0 0x7f78e4146240 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:54.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.543+0000 7f78eb669700 1 -- 172.21.15.60:0/4065817522 >> 172.21.15.60:0/4065817522 conn(0x7f78e4064a10 msgr2=0x7f78e4064df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:54.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.543+0000 7f78eb669700 1 -- 172.21.15.60:0/4065817522 shutdown_connections 2022-06-29T16:01:54.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.543+0000 7f78eb669700 1 -- 172.21.15.60:0/4065817522 wait complete. 2022-06-29T16:01:54.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.543+0000 7f78eb669700 1 Processor -- start 2022-06-29T16:01:54.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.543+0000 7f78eb669700 1 -- start start 2022-06-29T16:01:54.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.547+0000 7f78eb669700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78e405c2f0 0x7f78e4114c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:54.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.547+0000 7f78eb669700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78e405cf60 0x7f78e4115160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:54.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.547+0000 7f78eb669700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78e4061870 0x7f78e4141e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:54.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.547+0000 7f78eb669700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e40b0040 con 0x7f78e4061870 2022-06-29T16:01:54.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.547+0000 7f78eb669700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e40b5030 con 0x7f78e405c2f0 2022-06-29T16:01:54.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.547+0000 7f78eb669700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78e4152a20 con 0x7f78e405cf60 2022-06-29T16:01:54.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.547+0000 7f78e9c06700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78e4061870 0x7f78e4141e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:54.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.547+0000 7f78e8c04700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78e405cf60 0x7f78e4115160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:54.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.547+0000 7f78e8c04700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78e405cf60 0x7f78e4115160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:60930/0 (socket says 172.21.15.60:60930) 2022-06-29T16:01:54.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.547+0000 7f78e9c06700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78e4061870 0x7f78e4141e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47838/0 (socket says 172.21.15.60:47838) 2022-06-29T16:01:54.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.547+0000 7f78e8c04700 1 -- 172.21.15.60:0/338211372 learned_addr learned my addr 172.21.15.60:0/338211372 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:54.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.547+0000 7f78e9405700 1 --2- 172.21.15.60:0/338211372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78e405c2f0 0x7f78e4114c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:54.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.547+0000 7f78e8c04700 1 -- 172.21.15.60:0/338211372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78e405c2f0 msgr2=0x7f78e4114c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:54.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.547+0000 7f78e8c04700 1 --2- 172.21.15.60:0/338211372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78e405c2f0 0x7f78e4114c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:54.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.547+0000 7f78e8c04700 1 -- 172.21.15.60:0/338211372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78e4061870 msgr2=0x7f78e4141e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:54.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.547+0000 7f78e8c04700 1 --2- 172.21.15.60:0/338211372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78e4061870 0x7f78e4141e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:54.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.547+0000 7f78e8c04700 1 -- 172.21.15.60:0/338211372 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78e000b1e0 con 0x7f78e405cf60 2022-06-29T16:01:54.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.547+0000 7f78e9c06700 1 --2- 172.21.15.60:0/338211372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78e4061870 0x7f78e4141e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:01:54.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.547+0000 7f78e8c04700 1 --2- 172.21.15.60:0/338211372 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78e405cf60 0x7f78e4115160 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f78e40b0180 tx=0x7f78e0015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:54.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.547+0000 7f78da7fc700 1 -- 172.21.15.60:0/338211372 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f78e00116e0 con 0x7f78e405cf60 2022-06-29T16:01:54.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.547+0000 7f78da7fc700 1 -- 172.21.15.60:0/338211372 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78e0011840 con 0x7f78e405cf60 2022-06-29T16:01:54.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.547+0000 7f78eb669700 1 -- 172.21.15.60:0/338211372 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78e40b5260 con 0x7f78e405cf60 2022-06-29T16:01:54.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.547+0000 7f78da7fc700 1 -- 172.21.15.60:0/338211372 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f78e0011ba0 con 0x7f78e405cf60 2022-06-29T16:01:54.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.547+0000 7f78eb669700 1 -- 172.21.15.60:0/338211372 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78e40b5500 con 0x7f78e405cf60 2022-06-29T16:01:54.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.551+0000 7f78eb669700 1 -- 172.21.15.60:0/338211372 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78e40573d0 con 0x7f78e405cf60 2022-06-29T16:01:54.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.551+0000 7f78da7fc700 1 -- 172.21.15.60:0/338211372 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f78e0025030 con 0x7f78e405cf60 2022-06-29T16:01:54.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.551+0000 7f78da7fc700 1 --2- 172.21.15.60:0/338211372 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f78cc053a70 0x7f78cc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:54.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.551+0000 7f78da7fc700 1 -- 172.21.15.60:0/338211372 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f78e006d100 con 0x7f78e405cf60 2022-06-29T16:01:54.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.551+0000 7f78e9405700 1 --2- 172.21.15.60:0/338211372 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f78cc053a70 0x7f78cc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:54.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.551+0000 7f78e9405700 1 --2- 172.21.15.60:0/338211372 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f78cc053a70 0x7f78cc055f30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f78d400d000 tx=0x7f78d4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:54.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.559+0000 7f78da7fc700 1 -- 172.21.15.60:0/338211372 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f78e0043f00 con 0x7f78e405cf60 2022-06-29T16:01:54.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.727+0000 7f78eb669700 1 -- 172.21.15.60:0/338211372 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f78e4152b60 con 0x7f78e405cf60 2022-06-29T16:01:54.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.739+0000 7f78da7fc700 1 -- 172.21.15.60:0/338211372 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 9 and data pool 10 v19) v1 ==== 188+0+0 (secure 0 0 0) 0x7f78e00230d0 con 0x7f78e405cf60 2022-06-29T16:01:54.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.743+0000 7f78eb669700 1 -- 172.21.15.60:0/338211372 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f78cc053a70 msgr2=0x7f78cc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:54.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.743+0000 7f78eb669700 1 --2- 172.21.15.60:0/338211372 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f78cc053a70 0x7f78cc055f30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f78d400d000 tx=0x7f78d4009040).stop 2022-06-29T16:01:54.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.743+0000 7f78eb669700 1 -- 172.21.15.60:0/338211372 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78e405cf60 msgr2=0x7f78e4115160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:54.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.743+0000 7f78eb669700 1 --2- 172.21.15.60:0/338211372 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78e405cf60 0x7f78e4115160 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f78e40b0180 tx=0x7f78e0015000).stop 2022-06-29T16:01:54.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.743+0000 7f78eb669700 1 -- 172.21.15.60:0/338211372 shutdown_connections 2022-06-29T16:01:54.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.743+0000 7f78eb669700 1 --2- 172.21.15.60:0/338211372 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f78cc053a70 0x7f78cc055f30 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:54.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.743+0000 7f78eb669700 1 --2- 172.21.15.60:0/338211372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78e4061870 0x7f78e4141e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:54.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.743+0000 7f78eb669700 1 --2- 172.21.15.60:0/338211372 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78e405cf60 0x7f78e4115160 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:54.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.743+0000 7f78eb669700 1 --2- 172.21.15.60:0/338211372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78e405c2f0 0x7f78e4114c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:54.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.743+0000 7f78eb669700 1 -- 172.21.15.60:0/338211372 >> 172.21.15.60:0/338211372 conn(0x7f78e4064a10 msgr2=0x7f78e4145190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:54.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.743+0000 7f78eb669700 1 -- 172.21.15.60:0/338211372 shutdown_connections 2022-06-29T16:01:54.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.743+0000 7f78eb669700 1 -- 172.21.15.60:0/338211372 wait complete. 2022-06-29T16:01:54.757 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 9 and data pool 10 2022-06-29T16:01:54.766 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:01:54.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.851+0000 7f680605d700 1 Processor -- start 2022-06-29T16:01:54.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.851+0000 7f680605d700 1 -- start start 2022-06-29T16:01:54.862 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.851+0000 7f680605d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f680005d070 0x7f680013c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:54.862 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.851+0000 7f680605d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f68000b0830 0x7f680013fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:54.862 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.851+0000 7f680605d700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f680005d450 con 0x7f68000b2130 2022-06-29T16:01:54.863 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.851+0000 7f680605d700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6800063ba0 con 0x7f68000b0830 2022-06-29T16:01:54.863 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.851+0000 7f680605d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6800063e40 con 0x7f680005d070 2022-06-29T16:01:54.863 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.851+0000 7f680485a700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f68000b2130 0x7f680013caa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55930/0 (socket says 172.21.15.60:55930) 2022-06-29T16:01:54.863 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.851+0000 7f680585c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f68000b0830 0x7f680013fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:54.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.851+0000 7f680485a700 1 -- 172.21.15.60:0/955212273 learned_addr learned my addr 172.21.15.60:0/955212273 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:01:54.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.851+0000 7f680505b700 1 --2- 172.21.15.60:0/955212273 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f680005d070 0x7f680013c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:54.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.851+0000 7f680585c700 1 -- 172.21.15.60:0/955212273 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f680005d070 msgr2=0x7f680013c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:01:54.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.851+0000 7f680585c700 1 --2- 172.21.15.60:0/955212273 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f680005d070 0x7f680013c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:54.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.851+0000 7f680585c700 1 -- 172.21.15.60:0/955212273 >> v1:172.21.15.60:6789/0 conn(0x7f68000b2130 legacy=0x7f680013caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:54.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.851+0000 7f680585c700 1 -- 172.21.15.60:0/955212273 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6800067650 con 0x7f68000b0830 2022-06-29T16:01:54.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.851+0000 7f680585c700 1 --2- 172.21.15.60:0/955212273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68000b0830 0x7f680013fa70 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f67f80032f0 tx=0x7f67f80091d0).ready entity=mon.0 client_cookie=6662e366b2f4e065 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:54.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.851+0000 7f67effff700 1 -- 172.21.15.60:0/955212273 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67f8009ea0 con 0x7f68000b0830 2022-06-29T16:01:54.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.851+0000 7f67effff700 1 -- 172.21.15.60:0/955212273 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67f800ba90 con 0x7f68000b0830 2022-06-29T16:01:54.866 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.851+0000 7f67effff700 1 -- 172.21.15.60:0/955212273 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67f800bdf0 con 0x7f68000b0830 2022-06-29T16:01:54.866 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.851+0000 7f680605d700 1 -- 172.21.15.60:0/955212273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68000b0830 msgr2=0x7f680013fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:54.866 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.851+0000 7f680605d700 1 --2- 172.21.15.60:0/955212273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68000b0830 0x7f680013fa70 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f67f80032f0 tx=0x7f67f80091d0).stop 2022-06-29T16:01:54.866 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.851+0000 7f680605d700 1 -- 172.21.15.60:0/955212273 shutdown_connections 2022-06-29T16:01:54.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.851+0000 7f680605d700 1 --2- 172.21.15.60:0/955212273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68000b0830 0x7f680013fa70 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:54.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.851+0000 7f680605d700 1 --2- 172.21.15.60:0/955212273 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f680005d070 0x7f680013c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:54.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.851+0000 7f680605d700 1 -- 172.21.15.60:0/955212273 >> 172.21.15.60:0/955212273 conn(0x7f68000b5890 msgr2=0x7f68000b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:54.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.851+0000 7f680605d700 1 -- 172.21.15.60:0/955212273 shutdown_connections 2022-06-29T16:01:54.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.851+0000 7f680605d700 1 -- 172.21.15.60:0/955212273 wait complete. 2022-06-29T16:01:54.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.855+0000 7f680605d700 1 Processor -- start 2022-06-29T16:01:54.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.855+0000 7f680605d700 1 -- start start 2022-06-29T16:01:54.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.855+0000 7f680605d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f680005d070 0x7f680013ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:54.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.855+0000 7f680605d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68000b0830 0x7f6800138210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:54.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.855+0000 7f680605d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68000b2130 0x7f6800138700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:54.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.855+0000 7f680605d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6800140d00 con 0x7f68000b2130 2022-06-29T16:01:54.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.855+0000 7f680605d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6800140f30 con 0x7f680005d070 2022-06-29T16:01:54.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.855+0000 7f680605d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68001411d0 con 0x7f68000b0830 2022-06-29T16:01:54.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.859+0000 7f680585c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68000b2130 0x7f6800138700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:54.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.859+0000 7f680485a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68000b0830 0x7f6800138210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:54.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.859+0000 7f680585c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68000b2130 0x7f6800138700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47852/0 (socket says 172.21.15.60:47852) 2022-06-29T16:01:54.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.859+0000 7f680585c700 1 -- 172.21.15.60:0/4232121638 learned_addr learned my addr 172.21.15.60:0/4232121638 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:54.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.859+0000 7f680505b700 1 --2- 172.21.15.60:0/4232121638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f680005d070 0x7f680013ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:54.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.859+0000 7f680585c700 1 -- 172.21.15.60:0/4232121638 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68000b0830 msgr2=0x7f6800138210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:54.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.859+0000 7f680585c700 1 --2- 172.21.15.60:0/4232121638 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68000b0830 0x7f6800138210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:54.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.859+0000 7f680585c700 1 -- 172.21.15.60:0/4232121638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f680005d070 msgr2=0x7f680013ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:54.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.859+0000 7f680585c700 1 --2- 172.21.15.60:0/4232121638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f680005d070 0x7f680013ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:54.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.859+0000 7f680585c700 1 -- 172.21.15.60:0/4232121638 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f8009470 con 0x7f68000b2130 2022-06-29T16:01:54.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.859+0000 7f680585c700 1 --2- 172.21.15.60:0/4232121638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68000b2130 0x7f6800138700 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f67f80032f0 tx=0x7f67f8018000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:54.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.859+0000 7f67ee7fc700 1 -- 172.21.15.60:0/4232121638 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67f8009ea0 con 0x7f68000b2130 2022-06-29T16:01:54.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.859+0000 7f67ee7fc700 1 -- 172.21.15.60:0/4232121638 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67f801e040 con 0x7f68000b2130 2022-06-29T16:01:54.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.859+0000 7f680605d700 1 -- 172.21.15.60:0/4232121638 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f680013f0e0 con 0x7f68000b2130 2022-06-29T16:01:54.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.859+0000 7f67ee7fc700 1 -- 172.21.15.60:0/4232121638 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67f8023630 con 0x7f68000b2130 2022-06-29T16:01:54.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.859+0000 7f680605d700 1 -- 172.21.15.60:0/4232121638 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f680013e600 con 0x7f68000b2130 2022-06-29T16:01:54.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.859+0000 7f680605d700 1 -- 172.21.15.60:0/4232121638 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f680013cc40 con 0x7f68000b2130 2022-06-29T16:01:54.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.859+0000 7f67ee7fc700 1 -- 172.21.15.60:0/4232121638 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f67f8029020 con 0x7f68000b2130 2022-06-29T16:01:54.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.863+0000 7f67ee7fc700 1 --2- 172.21.15.60:0/4232121638 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f67e0053aa0 0x7f67e0055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:54.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.863+0000 7f67ee7fc700 1 -- 172.21.15.60:0/4232121638 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f67f806f7c0 con 0x7f68000b2130 2022-06-29T16:01:54.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.863+0000 7f680505b700 1 --2- 172.21.15.60:0/4232121638 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f67e0053aa0 0x7f67e0055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:54.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.863+0000 7f680505b700 1 --2- 172.21.15.60:0/4232121638 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f67e0053aa0 0x7f67e0055f60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f67fc00c000 tx=0x7f67fc008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:54.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.867+0000 7f67ee7fc700 1 -- 172.21.15.60:0/4232121638 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f680013cc40 con 0x7f68000b2130 2022-06-29T16:01:55.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.995+0000 7f680605d700 1 -- 172.21.15.60:0/4232121638 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f680013cc10 con 0x7f68000b2130 2022-06-29T16:01:55.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.995+0000 7f67ee7fc700 1 -- 172.21.15.60:0/4232121638 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+16053 (secure 0 0 0) 0x7f67f80467e0 con 0x7f68000b2130 2022-06-29T16:01:55.004 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:01:55.004 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":52,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:01:54.732158+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:01:52.588508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2022-06-29T16:01:53.732456+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:01:55.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.999+0000 7f680605d700 1 -- 172.21.15.60:0/4232121638 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f67e0053aa0 msgr2=0x7f67e0055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:55.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.999+0000 7f680605d700 1 --2- 172.21.15.60:0/4232121638 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f67e0053aa0 0x7f67e0055f60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f67fc00c000 tx=0x7f67fc008040).stop 2022-06-29T16:01:55.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.999+0000 7f680605d700 1 -- 172.21.15.60:0/4232121638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68000b2130 msgr2=0x7f6800138700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:55.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.999+0000 7f680605d700 1 --2- 172.21.15.60:0/4232121638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68000b2130 0x7f6800138700 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f67f80032f0 tx=0x7f67f8018000).stop 2022-06-29T16:01:55.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.999+0000 7f680605d700 1 -- 172.21.15.60:0/4232121638 shutdown_connections 2022-06-29T16:01:55.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.999+0000 7f680605d700 1 --2- 172.21.15.60:0/4232121638 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f67e0053aa0 0x7f67e0055f60 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:55.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.999+0000 7f680605d700 1 --2- 172.21.15.60:0/4232121638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68000b2130 0x7f6800138700 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:55.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.999+0000 7f680605d700 1 --2- 172.21.15.60:0/4232121638 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f68000b0830 0x7f6800138210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:55.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.999+0000 7f680605d700 1 --2- 172.21.15.60:0/4232121638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f680005d070 0x7f680013ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:55.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.999+0000 7f680605d700 1 -- 172.21.15.60:0/4232121638 >> 172.21.15.60:0/4232121638 conn(0x7f68000b5890 msgr2=0x7f6800144550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:55.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.999+0000 7f680605d700 1 -- 172.21.15.60:0/4232121638 shutdown_connections 2022-06-29T16:01:55.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:54.999+0000 7f680605d700 1 -- 172.21.15.60:0/4232121638 wait complete. 2022-06-29T16:01:55.017 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:01:55.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.107+0000 7fdb5b11a700 1 Processor -- start 2022-06-29T16:01:55.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.111+0000 7fdb5b11a700 1 -- start start 2022-06-29T16:01:55.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.111+0000 7fdb5b11a700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdb540b48b0 0x7fdb54142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:55.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.111+0000 7fdb5b11a700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb540b0cf0 con 0x7fdb540b2c50 2022-06-29T16:01:55.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.111+0000 7fdb5b11a700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb540b4d10 con 0x7fdb540b5eb0 2022-06-29T16:01:55.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.111+0000 7fdb5b11a700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb540b54e0 con 0x7fdb540b48b0 2022-06-29T16:01:55.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.111+0000 7fdb59917700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdb540b48b0 0x7fdb54142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:55.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.111+0000 7fdb59917700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdb540b48b0 0x7fdb54142a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47858/0 (socket says 172.21.15.60:47858) 2022-06-29T16:01:55.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.111+0000 7fdb59917700 1 -- 172.21.15.60:0/2806318550 learned_addr learned my addr 172.21.15.60:0/2806318550 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:55.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.111+0000 7fdb5a919700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fdb540b2c50 0x7fdb5414d380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55946/0 (socket says 172.21.15.60:55946) 2022-06-29T16:01:55.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.111+0000 7fdb59917700 1 -- 172.21.15.60:0/2806318550 >> v1:172.21.15.92:6789/0 conn(0x7fdb540b5eb0 legacy=0x7fdb540b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:55.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.111+0000 7fdb59917700 1 -- 172.21.15.60:0/2806318550 >> v1:172.21.15.60:6789/0 conn(0x7fdb540b2c50 legacy=0x7fdb5414d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:55.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.111+0000 7fdb59917700 1 -- 172.21.15.60:0/2806318550 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb540b6290 con 0x7fdb540b48b0 2022-06-29T16:01:55.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.111+0000 7fdb59917700 1 --2- 172.21.15.60:0/2806318550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb540b48b0 0x7fdb54142a20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7fdb48002f70 tx=0x7fdb48006eb0).ready entity=mon.0 client_cookie=a4eb00cbed59556 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:55.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.111+0000 7fdb59116700 1 -- 172.21.15.60:0/2806318550 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb4800d660 con 0x7fdb540b48b0 2022-06-29T16:01:55.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.111+0000 7fdb59116700 1 -- 172.21.15.60:0/2806318550 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb48000e90 con 0x7fdb540b48b0 2022-06-29T16:01:55.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.111+0000 7fdb59116700 1 -- 172.21.15.60:0/2806318550 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb4800d9f0 con 0x7fdb540b48b0 2022-06-29T16:01:55.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.111+0000 7fdb5b11a700 1 -- 172.21.15.60:0/2806318550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb540b48b0 msgr2=0x7fdb54142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:55.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.111+0000 7fdb5b11a700 1 --2- 172.21.15.60:0/2806318550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb540b48b0 0x7fdb54142a20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7fdb48002f70 tx=0x7fdb48006eb0).stop 2022-06-29T16:01:55.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.111+0000 7fdb5b11a700 1 -- 172.21.15.60:0/2806318550 shutdown_connections 2022-06-29T16:01:55.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.111+0000 7fdb5b11a700 1 --2- 172.21.15.60:0/2806318550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb540b48b0 0x7fdb54142a20 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:55.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.111+0000 7fdb5b11a700 1 -- 172.21.15.60:0/2806318550 >> 172.21.15.60:0/2806318550 conn(0x7fdb54062090 msgr2=0x7fdb54062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:55.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.111+0000 7fdb5b11a700 1 -- 172.21.15.60:0/2806318550 shutdown_connections 2022-06-29T16:01:55.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.111+0000 7fdb5b11a700 1 -- 172.21.15.60:0/2806318550 wait complete. 2022-06-29T16:01:55.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.115+0000 7fdb5b11a700 1 Processor -- start 2022-06-29T16:01:55.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.115+0000 7fdb5b11a700 1 -- start start 2022-06-29T16:01:55.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.115+0000 7fdb5b11a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb540b2c50 0x7fdb54145b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:55.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.115+0000 7fdb5b11a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb540b48b0 0x7fdb54146630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:55.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.115+0000 7fdb5b11a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb540b5eb0 0x7fdb540adba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:55.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.115+0000 7fdb5b11a700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb54146e10 con 0x7fdb540b2c50 2022-06-29T16:01:55.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.115+0000 7fdb5b11a700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb541529a0 con 0x7fdb540b5eb0 2022-06-29T16:01:55.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.115+0000 7fdb5b11a700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb54152c40 con 0x7fdb540b48b0 2022-06-29T16:01:55.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.115+0000 7fdb5a118700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb540b2c50 0x7fdb54145b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:55.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.115+0000 7fdb5a118700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb540b2c50 0x7fdb54145b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47862/0 (socket says 172.21.15.60:47862) 2022-06-29T16:01:55.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.115+0000 7fdb5a118700 1 -- 172.21.15.60:0/1285473502 learned_addr learned my addr 172.21.15.60:0/1285473502 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:55.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.115+0000 7fdb59917700 1 --2- 172.21.15.60:0/1285473502 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb540b48b0 0x7fdb54146630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:55.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.115+0000 7fdb5a919700 1 --2- 172.21.15.60:0/1285473502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb540b5eb0 0x7fdb540adba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:55.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.115+0000 7fdb5a118700 1 -- 172.21.15.60:0/1285473502 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb540b48b0 msgr2=0x7fdb54146630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:55.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.115+0000 7fdb5a118700 1 --2- 172.21.15.60:0/1285473502 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb540b48b0 0x7fdb54146630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:55.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.115+0000 7fdb5a118700 1 -- 172.21.15.60:0/1285473502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb540b5eb0 msgr2=0x7fdb540adba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:55.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.115+0000 7fdb5a118700 1 --2- 172.21.15.60:0/1285473502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb540b5eb0 0x7fdb540adba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:55.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.115+0000 7fdb5a118700 1 -- 172.21.15.60:0/1285473502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb4800b1e0 con 0x7fdb540b2c50 2022-06-29T16:01:55.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.115+0000 7fdb59917700 1 --2- 172.21.15.60:0/1285473502 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb540b48b0 0x7fdb54146630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:01:55.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.115+0000 7fdb5a118700 1 --2- 172.21.15.60:0/1285473502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb540b2c50 0x7fdb54145b50 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fdb54062fa0 tx=0x7fdb50007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:55.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.119+0000 7fdb477fe700 1 -- 172.21.15.60:0/1285473502 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb50007e70 con 0x7fdb540b2c50 2022-06-29T16:01:55.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.119+0000 7fdb477fe700 1 -- 172.21.15.60:0/1285473502 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb50007030 con 0x7fdb540b2c50 2022-06-29T16:01:55.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.119+0000 7fdb5b11a700 1 -- 172.21.15.60:0/1285473502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb5405c850 con 0x7fdb540b2c50 2022-06-29T16:01:55.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.119+0000 7fdb477fe700 1 -- 172.21.15.60:0/1285473502 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb50007390 con 0x7fdb540b2c50 2022-06-29T16:01:55.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.119+0000 7fdb5b11a700 1 -- 172.21.15.60:0/1285473502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb54146b20 con 0x7fdb540b2c50 2022-06-29T16:01:55.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.119+0000 7fdb5b11a700 1 -- 172.21.15.60:0/1285473502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb54152e00 con 0x7fdb540b2c50 2022-06-29T16:01:55.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.119+0000 7fdb477fe700 1 -- 172.21.15.60:0/1285473502 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdb50010030 con 0x7fdb540b2c50 2022-06-29T16:01:55.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.119+0000 7fdb477fe700 1 --2- 172.21.15.60:0/1285473502 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdb34053b00 0x7fdb34055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:55.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.119+0000 7fdb477fe700 1 -- 172.21.15.60:0/1285473502 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fdb5006b9e0 con 0x7fdb540b2c50 2022-06-29T16:01:55.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.119+0000 7fdb59917700 1 --2- 172.21.15.60:0/1285473502 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdb34053b00 0x7fdb34055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:55.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.123+0000 7fdb59917700 1 --2- 172.21.15.60:0/1285473502 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdb34053b00 0x7fdb34055fc0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fdb4801c000 tx=0x7fdb48015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:55.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.127+0000 7fdb477fe700 1 -- 172.21.15.60:0/1285473502 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdb50042860 con 0x7fdb540b2c50 2022-06-29T16:01:55.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.259+0000 7fdb5b11a700 1 -- 172.21.15.60:0/1285473502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb540573d0 con 0x7fdb540b2c50 2022-06-29T16:01:55.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.263+0000 7fdb477fe700 1 -- 172.21.15.60:0/1285473502 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+16053 (secure 0 0 0) 0x7fdb50045e80 con 0x7fdb540b2c50 2022-06-29T16:01:55.269 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:01:55.270 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":52,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:01:54.732158+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:01:52.588508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2022-06-29T16:01:53.732456+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:01:55.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.263+0000 7fdb5b11a700 1 -- 172.21.15.60:0/1285473502 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdb34053b00 msgr2=0x7fdb34055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:55.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.263+0000 7fdb5b11a700 1 --2- 172.21.15.60:0/1285473502 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdb34053b00 0x7fdb34055fc0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fdb4801c000 tx=0x7fdb48015000).stop 2022-06-29T16:01:55.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.263+0000 7fdb5b11a700 1 -- 172.21.15.60:0/1285473502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb540b2c50 msgr2=0x7fdb54145b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:55.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.263+0000 7fdb5b11a700 1 --2- 172.21.15.60:0/1285473502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb540b2c50 0x7fdb54145b50 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fdb54062fa0 tx=0x7fdb50007c00).stop 2022-06-29T16:01:55.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.263+0000 7fdb5b11a700 1 -- 172.21.15.60:0/1285473502 shutdown_connections 2022-06-29T16:01:55.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.263+0000 7fdb5b11a700 1 --2- 172.21.15.60:0/1285473502 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdb34053b00 0x7fdb34055fc0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:55.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.263+0000 7fdb5b11a700 1 --2- 172.21.15.60:0/1285473502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb540b5eb0 0x7fdb540adba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:55.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.263+0000 7fdb5b11a700 1 --2- 172.21.15.60:0/1285473502 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdb540b48b0 0x7fdb54146630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:55.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.263+0000 7fdb5b11a700 1 --2- 172.21.15.60:0/1285473502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb540b2c50 0x7fdb54145b50 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:55.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.263+0000 7fdb5b11a700 1 -- 172.21.15.60:0/1285473502 >> 172.21.15.60:0/1285473502 conn(0x7fdb54062090 msgr2=0x7fdb540b7040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:55.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.263+0000 7fdb5b11a700 1 -- 172.21.15.60:0/1285473502 shutdown_connections 2022-06-29T16:01:55.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.263+0000 7fdb5b11a700 1 -- 172.21.15.60:0/1285473502 wait complete. 2022-06-29T16:01:55.283 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-06-29T16:01:55.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.387+0000 7fa79359e700 1 Processor -- start 2022-06-29T16:01:55.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.387+0000 7fa79359e700 1 -- start start 2022-06-29T16:01:55.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.387+0000 7fa79359e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa7940b22c0 0x7fa794142f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:55.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.387+0000 7fa79359e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7940ab960 con 0x7fa794143b70 2022-06-29T16:01:55.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.387+0000 7fa79359e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7940b0c90 con 0x7fa7940aa920 2022-06-29T16:01:55.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.387+0000 7fa79359e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7940b26a0 con 0x7fa7940b22c0 2022-06-29T16:01:55.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.387+0000 7fa792d9d700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa794143b70 0x7fa7940af930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55960/0 (socket says 172.21.15.60:55960) 2022-06-29T16:01:55.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.387+0000 7fa792d9d700 1 -- 172.21.15.60:0/100221857 learned_addr learned my addr 172.21.15.60:0/100221857 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:01:55.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.387+0000 7fa79259c700 1 --2- 172.21.15.60:0/100221857 >> v2:172.21.15.60:3300/0 conn(0x7fa7940b22c0 0x7fa794142f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:55.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.391+0000 7fa791d9b700 1 -- 172.21.15.60:0/100221857 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2232653900 0 0) 0x7fa7940ab960 con 0x7fa794143b70 2022-06-29T16:01:55.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.391+0000 7fa791d9b700 1 -- 172.21.15.60:0/100221857 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa774003260 con 0x7fa794143b70 2022-06-29T16:01:55.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.391+0000 7fa791d9b700 1 -- 172.21.15.60:0/100221857 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2621546936 0 0) 0x7fa7940b0c90 con 0x7fa7940aa920 2022-06-29T16:01:55.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.391+0000 7fa79259c700 1 -- 172.21.15.60:0/100221857 >> v1:172.21.15.92:6789/0 conn(0x7fa7940aa920 legacy=0x7fa794143460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:55.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.391+0000 7fa79259c700 1 -- 172.21.15.60:0/100221857 >> v1:172.21.15.60:6789/0 conn(0x7fa794143b70 legacy=0x7fa7940af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:55.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.391+0000 7fa79259c700 1 -- 172.21.15.60:0/100221857 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7940abb90 con 0x7fa7940b22c0 2022-06-29T16:01:55.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.391+0000 7fa79259c700 1 --2- 172.21.15.60:0/100221857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7940b22c0 0x7fa794142f70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fa78c001b80 tx=0x7fa784006df0).ready entity=mon.0 client_cookie=ba12c41863c804a3 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:55.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.391+0000 7fa791d9b700 1 -- 172.21.15.60:0/100221857 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa78c0019e0 con 0x7fa7940b22c0 2022-06-29T16:01:55.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.391+0000 7fa791d9b700 1 -- 172.21.15.60:0/100221857 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa784000e90 con 0x7fa7940b22c0 2022-06-29T16:01:55.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.391+0000 7fa791d9b700 1 -- 172.21.15.60:0/100221857 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa784011400 con 0x7fa7940b22c0 2022-06-29T16:01:55.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.391+0000 7fa79359e700 1 -- 172.21.15.60:0/100221857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7940b22c0 msgr2=0x7fa794142f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:55.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.391+0000 7fa79359e700 1 --2- 172.21.15.60:0/100221857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7940b22c0 0x7fa794142f70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fa78c001b80 tx=0x7fa784006df0).stop 2022-06-29T16:01:55.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.391+0000 7fa79359e700 1 -- 172.21.15.60:0/100221857 shutdown_connections 2022-06-29T16:01:55.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.391+0000 7fa79359e700 1 --2- 172.21.15.60:0/100221857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7940b22c0 0x7fa794142f70 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:55.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.391+0000 7fa79359e700 1 -- 172.21.15.60:0/100221857 >> 172.21.15.60:0/100221857 conn(0x7fa7940b58a0 msgr2=0x7fa7940b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:55.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.391+0000 7fa79359e700 1 -- 172.21.15.60:0/100221857 shutdown_connections 2022-06-29T16:01:55.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.391+0000 7fa79359e700 1 -- 172.21.15.60:0/100221857 wait complete. 2022-06-29T16:01:55.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.395+0000 7fa79359e700 1 Processor -- start 2022-06-29T16:01:55.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.395+0000 7fa79359e700 1 -- start start 2022-06-29T16:01:55.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.395+0000 7fa79359e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa7940aa920 0x7fa79408fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:55.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.395+0000 7fa79359e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7940b22c0 0x7fa794065330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:55.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.395+0000 7fa79359e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa794143b70 0x7fa794144120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:55.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.395+0000 7fa79359e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa794144b10 con 0x7fa7940b22c0 2022-06-29T16:01:55.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.395+0000 7fa79359e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa794065960 con 0x7fa794143b70 2022-06-29T16:01:55.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.395+0000 7fa79359e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa794065c00 con 0x7fa7940aa920 2022-06-29T16:01:55.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.395+0000 7fa78bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7940b22c0 0x7fa794065330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:55.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.395+0000 7fa78bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7940b22c0 0x7fa794065330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47878/0 (socket says 172.21.15.60:47878) 2022-06-29T16:01:55.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.395+0000 7fa78bfff700 1 -- 172.21.15.60:0/2754477598 learned_addr learned my addr 172.21.15.60:0/2754477598 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:55.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.395+0000 7fa79259c700 1 --2- 172.21.15.60:0/2754477598 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa7940aa920 0x7fa79408fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:55.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.395+0000 7fa792d9d700 1 --2- 172.21.15.60:0/2754477598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa794143b70 0x7fa794144120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:55.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.395+0000 7fa78bfff700 1 -- 172.21.15.60:0/2754477598 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa7940aa920 msgr2=0x7fa79408fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:55.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.395+0000 7fa78bfff700 1 --2- 172.21.15.60:0/2754477598 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa7940aa920 0x7fa79408fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:55.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.395+0000 7fa78bfff700 1 -- 172.21.15.60:0/2754477598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa794143b70 msgr2=0x7fa794144120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:55.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.395+0000 7fa78bfff700 1 --2- 172.21.15.60:0/2754477598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa794143b70 0x7fa794144120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:55.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.395+0000 7fa78bfff700 1 -- 172.21.15.60:0/2754477598 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa78400b180 con 0x7fa7940b22c0 2022-06-29T16:01:55.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.395+0000 7fa78bfff700 1 --2- 172.21.15.60:0/2754477598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7940b22c0 0x7fa794065330 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7fa77c000ed0 tx=0x7fa77c007ac0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:55.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.395+0000 7fa78b7fe700 1 -- 172.21.15.60:0/2754477598 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa77c012400 con 0x7fa7940b22c0 2022-06-29T16:01:55.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.395+0000 7fa78b7fe700 1 -- 172.21.15.60:0/2754477598 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa77c012560 con 0x7fa7940b22c0 2022-06-29T16:01:55.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.395+0000 7fa79359e700 1 -- 172.21.15.60:0/2754477598 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa794065ea0 con 0x7fa7940b22c0 2022-06-29T16:01:55.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.395+0000 7fa78b7fe700 1 -- 172.21.15.60:0/2754477598 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa77c0128c0 con 0x7fa7940b22c0 2022-06-29T16:01:55.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.395+0000 7fa79359e700 1 -- 172.21.15.60:0/2754477598 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa794097040 con 0x7fa7940b22c0 2022-06-29T16:01:55.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.399+0000 7fa79359e700 1 -- 172.21.15.60:0/2754477598 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa794066030 con 0x7fa7940b22c0 2022-06-29T16:01:55.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.399+0000 7fa78b7fe700 1 -- 172.21.15.60:0/2754477598 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa77c019030 con 0x7fa7940b22c0 2022-06-29T16:01:55.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.399+0000 7fa78b7fe700 1 --2- 172.21.15.60:0/2754477598 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa7780539e0 0x7fa778055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:55.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.399+0000 7fa78b7fe700 1 -- 172.21.15.60:0/2754477598 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa77c06b180 con 0x7fa7940b22c0 2022-06-29T16:01:55.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.399+0000 7fa79259c700 1 --2- 172.21.15.60:0/2754477598 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa7780539e0 0x7fa778055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:55.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.399+0000 7fa79259c700 1 --2- 172.21.15.60:0/2754477598 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa7780539e0 0x7fa778055ea0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fa784019000 tx=0x7fa7940abd20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:55.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.407+0000 7fa78b7fe700 1 -- 172.21.15.60:0/2754477598 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa77c042150 con 0x7fa7940b22c0 2022-06-29T16:01:55.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.571+0000 7fa79359e700 1 -- 172.21.15.60:0/2754477598 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fa7940573d0 con 0x7fa7940b22c0 2022-06-29T16:01:55.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.735+0000 7fa78b7fe700 1 -- 172.21.15.60:0/2754477598 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v20) v1 ==== 118+0+0 (secure 0 0 0) 0x7fa77c045770 con 0x7fa7940b22c0 2022-06-29T16:01:55.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.739+0000 7fa79359e700 1 -- 172.21.15.60:0/2754477598 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa7780539e0 msgr2=0x7fa778055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:55.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.739+0000 7fa79359e700 1 --2- 172.21.15.60:0/2754477598 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa7780539e0 0x7fa778055ea0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fa784019000 tx=0x7fa7940abd20).stop 2022-06-29T16:01:55.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.739+0000 7fa79359e700 1 -- 172.21.15.60:0/2754477598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7940b22c0 msgr2=0x7fa794065330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:55.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.739+0000 7fa79359e700 1 --2- 172.21.15.60:0/2754477598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7940b22c0 0x7fa794065330 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7fa77c000ed0 tx=0x7fa77c007ac0).stop 2022-06-29T16:01:55.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.739+0000 7fa79359e700 1 -- 172.21.15.60:0/2754477598 shutdown_connections 2022-06-29T16:01:55.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.739+0000 7fa79359e700 1 --2- 172.21.15.60:0/2754477598 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa7780539e0 0x7fa778055ea0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:55.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.739+0000 7fa79359e700 1 --2- 172.21.15.60:0/2754477598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa794143b70 0x7fa794144120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:55.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.739+0000 7fa79359e700 1 --2- 172.21.15.60:0/2754477598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa7940b22c0 0x7fa794065330 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:55.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.739+0000 7fa79359e700 1 --2- 172.21.15.60:0/2754477598 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa7940aa920 0x7fa79408fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:55.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.739+0000 7fa79359e700 1 -- 172.21.15.60:0/2754477598 >> 172.21.15.60:0/2754477598 conn(0x7fa7940b58a0 msgr2=0x7fa7940b1850 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:55.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.739+0000 7fa79359e700 1 -- 172.21.15.60:0/2754477598 shutdown_connections 2022-06-29T16:01:55.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.739+0000 7fa79359e700 1 -- 172.21.15.60:0/2754477598 wait complete. 2022-06-29T16:01:55.761 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:01:55.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.855+0000 7fc7c0bcf700 1 Processor -- start 2022-06-29T16:01:55.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.855+0000 7fc7c0bcf700 1 -- start start 2022-06-29T16:01:55.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.855+0000 7fc7c0bcf700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7bc05d070 0x7fc7bc14aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:55.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.855+0000 7fc7c0bcf700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc7bc0b2130 0x7fc7bc0aca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:55.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.855+0000 7fc7c0bcf700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7bc05d450 con 0x7fc7bc14af80 2022-06-29T16:01:55.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.855+0000 7fc7c0bcf700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7bc0adf90 con 0x7fc7bc0b2130 2022-06-29T16:01:55.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.855+0000 7fc7c0bcf700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc0b25d0 con 0x7fc7bc05d070 2022-06-29T16:01:55.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.859+0000 7fc7baffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc7bc0b2130 0x7fc7bc0aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:55.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.859+0000 7fc7bbfff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc7bc14af80 0x7fc7bc0acf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55974/0 (socket says 172.21.15.60:55974) 2022-06-29T16:01:55.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.859+0000 7fc7bbfff700 1 -- 172.21.15.60:0/2345265577 learned_addr learned my addr 172.21.15.60:0/2345265577 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:01:55.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.859+0000 7fc7baffd700 1 --2- 172.21.15.60:0/2345265577 >> v2:172.21.15.60:3300/0 conn(0x7fc7bc0b2130 0x7fc7bc0aca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47886/0 (socket says 172.21.15.60:47886) 2022-06-29T16:01:55.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.859+0000 7fc7bb7fe700 1 --2- 172.21.15.60:0/2345265577 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7bc05d070 0x7fc7bc14aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:55.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.859+0000 7fc7baffd700 1 -- 172.21.15.60:0/2345265577 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7bc05d070 msgr2=0x7fc7bc14aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:01:55.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.859+0000 7fc7baffd700 1 --2- 172.21.15.60:0/2345265577 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7bc05d070 0x7fc7bc14aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:55.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.859+0000 7fc7baffd700 1 -- 172.21.15.60:0/2345265577 >> v1:172.21.15.60:6789/0 conn(0x7fc7bc14af80 legacy=0x7fc7bc0acf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:55.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.859+0000 7fc7ba7fc700 1 -- 172.21.15.60:0/2345265577 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3693249435 0 0) 0x7fc7bc05d450 con 0x7fc7bc14af80 2022-06-29T16:01:55.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.859+0000 7fc7baffd700 1 -- 172.21.15.60:0/2345265577 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7bc14a350 con 0x7fc7bc0b2130 2022-06-29T16:01:55.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.859+0000 7fc7bb7fe700 1 --2- 172.21.15.60:0/2345265577 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7bc05d070 0x7fc7bc14aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:01:55.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.859+0000 7fc7baffd700 1 --2- 172.21.15.60:0/2345265577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7bc0b2130 0x7fc7bc0aca20 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7fc7b0002f70 tx=0x7fc7b0006df0).ready entity=mon.0 client_cookie=43bdb2f9276c730f server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:55.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.859+0000 7fc7ba7fc700 1 -- 172.21.15.60:0/2345265577 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc7b000d570 con 0x7fc7bc0b2130 2022-06-29T16:01:55.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.859+0000 7fc7ba7fc700 1 -- 172.21.15.60:0/2345265577 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b0000e90 con 0x7fc7bc0b2130 2022-06-29T16:01:55.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.859+0000 7fc7ba7fc700 1 -- 172.21.15.60:0/2345265577 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc7b000d8c0 con 0x7fc7bc0b2130 2022-06-29T16:01:55.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.859+0000 7fc7c0bcf700 1 -- 172.21.15.60:0/2345265577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7bc0b2130 msgr2=0x7fc7bc0aca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:55.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.859+0000 7fc7c0bcf700 1 --2- 172.21.15.60:0/2345265577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7bc0b2130 0x7fc7bc0aca20 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7fc7b0002f70 tx=0x7fc7b0006df0).stop 2022-06-29T16:01:55.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.859+0000 7fc7c0bcf700 1 -- 172.21.15.60:0/2345265577 shutdown_connections 2022-06-29T16:01:55.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.859+0000 7fc7c0bcf700 1 --2- 172.21.15.60:0/2345265577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7bc0b2130 0x7fc7bc0aca20 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:55.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.859+0000 7fc7c0bcf700 1 --2- 172.21.15.60:0/2345265577 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7bc05d070 0x7fc7bc14aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:55.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.859+0000 7fc7c0bcf700 1 -- 172.21.15.60:0/2345265577 >> 172.21.15.60:0/2345265577 conn(0x7fc7bc0b5890 msgr2=0x7fc7bc0b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:55.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.859+0000 7fc7c0bcf700 1 -- 172.21.15.60:0/2345265577 shutdown_connections 2022-06-29T16:01:55.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.859+0000 7fc7c0bcf700 1 -- 172.21.15.60:0/2345265577 wait complete. 2022-06-29T16:01:55.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.859+0000 7fc7c0bcf700 1 Processor -- start 2022-06-29T16:01:55.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.863+0000 7fc7c0bcf700 1 -- start start 2022-06-29T16:01:55.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.863+0000 7fc7c0bcf700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7bc05d070 0x7fc7bc063da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:55.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.863+0000 7fc7c0bcf700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7bc0b2130 0x7fc7bc0b4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:55.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.863+0000 7fc7c0bcf700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7bc14af80 0x7fc7bc159590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:55.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.863+0000 7fc7c0bcf700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc063360 con 0x7fc7bc05d070 2022-06-29T16:01:55.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.863+0000 7fc7bb7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7bc05d070 0x7fc7bc063da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:55.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.863+0000 7fc7c0bcf700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc149a70 con 0x7fc7bc0b2130 2022-06-29T16:01:55.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.863+0000 7fc7c0bcf700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc159a80 con 0x7fc7bc14af80 2022-06-29T16:01:55.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.863+0000 7fc7bb7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7bc05d070 0x7fc7bc063da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47890/0 (socket says 172.21.15.60:47890) 2022-06-29T16:01:55.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.863+0000 7fc7bb7fe700 1 -- 172.21.15.60:0/3145287172 learned_addr learned my addr 172.21.15.60:0/3145287172 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:55.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.863+0000 7fc7bbfff700 1 --2- 172.21.15.60:0/3145287172 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7bc14af80 0x7fc7bc159590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:55.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.863+0000 7fc7baffd700 1 --2- 172.21.15.60:0/3145287172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7bc0b2130 0x7fc7bc0b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:55.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.863+0000 7fc7bb7fe700 1 -- 172.21.15.60:0/3145287172 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7bc14af80 msgr2=0x7fc7bc159590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:55.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.863+0000 7fc7bb7fe700 1 --2- 172.21.15.60:0/3145287172 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7bc14af80 0x7fc7bc159590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:55.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.863+0000 7fc7bb7fe700 1 -- 172.21.15.60:0/3145287172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7bc0b2130 msgr2=0x7fc7bc0b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:55.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.863+0000 7fc7bb7fe700 1 --2- 172.21.15.60:0/3145287172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7bc0b2130 0x7fc7bc0b4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:55.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.863+0000 7fc7bb7fe700 1 -- 172.21.15.60:0/3145287172 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b000b180 con 0x7fc7bc05d070 2022-06-29T16:01:55.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.863+0000 7fc7baffd700 1 --2- 172.21.15.60:0/3145287172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7bc0b2130 0x7fc7bc0b4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:01:55.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.863+0000 7fc7bb7fe700 1 --2- 172.21.15.60:0/3145287172 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7bc05d070 0x7fc7bc063da0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7fc7b4006cc0 tx=0x7fc7b4008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:55.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.863+0000 7fc7b8ff9700 1 -- 172.21.15.60:0/3145287172 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc7b4011780 con 0x7fc7bc05d070 2022-06-29T16:01:55.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.863+0000 7fc7b8ff9700 1 -- 172.21.15.60:0/3145287172 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b40118e0 con 0x7fc7bc05d070 2022-06-29T16:01:55.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.863+0000 7fc7c0bcf700 1 -- 172.21.15.60:0/3145287172 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7bc149cd0 con 0x7fc7bc05d070 2022-06-29T16:01:55.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.863+0000 7fc7b8ff9700 1 -- 172.21.15.60:0/3145287172 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc7b4011c40 con 0x7fc7bc05d070 2022-06-29T16:01:55.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.863+0000 7fc7c0bcf700 1 -- 172.21.15.60:0/3145287172 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7bc064cb0 con 0x7fc7bc05d070 2022-06-29T16:01:55.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.867+0000 7fc7c0bcf700 1 -- 172.21.15.60:0/3145287172 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7bc14a5f0 con 0x7fc7bc05d070 2022-06-29T16:01:55.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.867+0000 7fc7b8ff9700 1 -- 172.21.15.60:0/3145287172 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc7b4012420 con 0x7fc7bc05d070 2022-06-29T16:01:55.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.867+0000 7fc7b8ff9700 1 --2- 172.21.15.60:0/3145287172 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc79c053a70 0x7fc79c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:55.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.867+0000 7fc7b8ff9700 1 -- 172.21.15.60:0/3145287172 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc7b406c110 con 0x7fc7bc05d070 2022-06-29T16:01:55.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.867+0000 7fc7baffd700 1 --2- 172.21.15.60:0/3145287172 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc79c053a70 0x7fc79c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:55.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.867+0000 7fc7baffd700 1 --2- 172.21.15.60:0/3145287172 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc79c053a70 0x7fc79c055f30 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fc7b001a000 tx=0x7fc7b0015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:55.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:55.871+0000 7fc7b8ff9700 1 -- 172.21.15.60:0/3145287172 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc7b4042ec0 con 0x7fc7bc05d070 2022-06-29T16:01:56.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.039+0000 7fc7c0bcf700 1 -- 172.21.15.60:0/3145287172 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7bc0573d0 con 0x7fc7bc05d070 2022-06-29T16:01:56.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.039+0000 7fc7b8ff9700 1 -- 172.21.15.60:0/3145287172 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+2139 (secure 0 0 0) 0x7fc7b40464e0 con 0x7fc7bc05d070 2022-06-29T16:01:56.048 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:01:56.048 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":20,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5568,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/4229695218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":4229695218},{"type":"v1","addr":"172.21.15.60:6833","nonce":4229695218}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:01:54.732520+0000","modified":"2022-06-29T16:01:55.735497+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5653":{"gid":5653,"name":"b","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/1602880694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1602880694},{"type":"v1","addr":"172.21.15.92:6835","nonce":1602880694}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-06-29T16:01:56.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.043+0000 7fc7c0bcf700 1 -- 172.21.15.60:0/3145287172 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc79c053a70 msgr2=0x7fc79c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:56.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.043+0000 7fc7c0bcf700 1 --2- 172.21.15.60:0/3145287172 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc79c053a70 0x7fc79c055f30 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fc7b001a000 tx=0x7fc7b0015000).stop 2022-06-29T16:01:56.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.043+0000 7fc7c0bcf700 1 -- 172.21.15.60:0/3145287172 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7bc05d070 msgr2=0x7fc7bc063da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:56.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.043+0000 7fc7c0bcf700 1 --2- 172.21.15.60:0/3145287172 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7bc05d070 0x7fc7bc063da0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7fc7b4006cc0 tx=0x7fc7b4008040).stop 2022-06-29T16:01:56.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.043+0000 7fc7c0bcf700 1 -- 172.21.15.60:0/3145287172 shutdown_connections 2022-06-29T16:01:56.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.043+0000 7fc7c0bcf700 1 --2- 172.21.15.60:0/3145287172 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc79c053a70 0x7fc79c055f30 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:56.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.043+0000 7fc7c0bcf700 1 --2- 172.21.15.60:0/3145287172 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7bc14af80 0x7fc7bc159590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:56.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.043+0000 7fc7c0bcf700 1 --2- 172.21.15.60:0/3145287172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7bc0b2130 0x7fc7bc0b4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:56.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.043+0000 7fc7c0bcf700 1 --2- 172.21.15.60:0/3145287172 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7bc05d070 0x7fc7bc063da0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:56.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.043+0000 7fc7c0bcf700 1 -- 172.21.15.60:0/3145287172 >> 172.21.15.60:0/3145287172 conn(0x7fc7bc0b5890 msgr2=0x7fc7bc146e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:56.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.043+0000 7fc7c0bcf700 1 -- 172.21.15.60:0/3145287172 shutdown_connections 2022-06-29T16:01:56.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.043+0000 7fc7c0bcf700 1 -- 172.21.15.60:0/3145287172 wait complete. 2022-06-29T16:01:56.053 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 20 2022-06-29T16:01:56.062 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:01:56.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.151+0000 7fd08c9e1700 1 Processor -- start 2022-06-29T16:01:56.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.155+0000 7fd08c9e1700 1 -- start start 2022-06-29T16:01:56.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.155+0000 7fd08c9e1700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0880b4010 0x7fd0881426b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:56.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.155+0000 7fd08c9e1700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd0880b46c0 0x7fd0880b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:56.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.155+0000 7fd08c9e1700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd088143a40 con 0x7fd0880ae7c0 2022-06-29T16:01:56.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.155+0000 7fd08c9e1700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0880aee60 con 0x7fd0880b46c0 2022-06-29T16:01:56.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.155+0000 7fd08c9e1700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0880af100 con 0x7fd0880b4010 2022-06-29T16:01:56.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.155+0000 7fd086d9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd0880b46c0 0x7fd0880b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:56.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.155+0000 7fd07dd9b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0880b4010 0x7fd0881426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:56.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.155+0000 7fd086d9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd0880b46c0 0x7fd0880b4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47902/0 (socket says 172.21.15.60:47902) 2022-06-29T16:01:56.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.155+0000 7fd086d9d700 1 -- 172.21.15.60:0/1779078093 learned_addr learned my addr 172.21.15.60:0/1779078093 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:56.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.155+0000 7fd07dd9b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0880b4010 0x7fd0881426b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:60994/0 (socket says 172.21.15.60:60994) 2022-06-29T16:01:56.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.155+0000 7fd08659c700 1 --1- 172.21.15.60:0/1779078093 >> v1:172.21.15.92:6789/0 conn(0x7fd0880ae7c0 0x7fd088141fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:60232/0 (socket says 172.21.15.60:60232) 2022-06-29T16:01:56.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.155+0000 7fd07dd9b700 1 -- 172.21.15.60:0/1779078093 >> v2:172.21.15.60:3300/0 conn(0x7fd0880b46c0 msgr2=0x7fd0880b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:01:56.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.155+0000 7fd07dd9b700 1 --2- 172.21.15.60:0/1779078093 >> v2:172.21.15.60:3300/0 conn(0x7fd0880b46c0 0x7fd0880b4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:56.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.155+0000 7fd07dd9b700 1 -- 172.21.15.60:0/1779078093 >> v1:172.21.15.92:6789/0 conn(0x7fd0880ae7c0 legacy=0x7fd088141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:56.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.155+0000 7fd07dd9b700 1 -- 172.21.15.60:0/1779078093 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0880aeba0 con 0x7fd0880b4010 2022-06-29T16:01:56.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.155+0000 7fd07dd9b700 1 --2- 172.21.15.60:0/1779078093 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0880b4010 0x7fd0881426b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fd070002f70 tx=0x7fd070006df0).ready entity=mon.2 client_cookie=c19ab9f9a3e7c5b3 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:56.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.155+0000 7fd085d9b700 1 -- 172.21.15.60:0/1779078093 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd07000d570 con 0x7fd0880b4010 2022-06-29T16:01:56.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.155+0000 7fd085d9b700 1 -- 172.21.15.60:0/1779078093 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd070000e90 con 0x7fd0880b4010 2022-06-29T16:01:56.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.155+0000 7fd085d9b700 1 -- 172.21.15.60:0/1779078093 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd07000d8c0 con 0x7fd0880b4010 2022-06-29T16:01:56.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.155+0000 7fd08c9e1700 1 -- 172.21.15.60:0/1779078093 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0880b4010 msgr2=0x7fd0881426b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:56.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.155+0000 7fd08c9e1700 1 --2- 172.21.15.60:0/1779078093 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0880b4010 0x7fd0881426b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fd070002f70 tx=0x7fd070006df0).stop 2022-06-29T16:01:56.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.159+0000 7fd08c9e1700 1 -- 172.21.15.60:0/1779078093 shutdown_connections 2022-06-29T16:01:56.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.159+0000 7fd08c9e1700 1 --2- 172.21.15.60:0/1779078093 >> v2:172.21.15.60:3300/0 conn(0x7fd0880b46c0 0x7fd0880b4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:56.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.159+0000 7fd08c9e1700 1 --2- 172.21.15.60:0/1779078093 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0880b4010 0x7fd0881426b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:56.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.159+0000 7fd08c9e1700 1 -- 172.21.15.60:0/1779078093 >> 172.21.15.60:0/1779078093 conn(0x7fd0880b9340 msgr2=0x7fd0880b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:56.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.159+0000 7fd08c9e1700 1 -- 172.21.15.60:0/1779078093 shutdown_connections 2022-06-29T16:01:56.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.159+0000 7fd08c9e1700 1 -- 172.21.15.60:0/1779078093 wait complete. 2022-06-29T16:01:56.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.159+0000 7fd08c9e1700 1 Processor -- start 2022-06-29T16:01:56.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.159+0000 7fd08c9e1700 1 -- start start 2022-06-29T16:01:56.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.159+0000 7fd08c9e1700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0880ae7c0 0x7fd088144940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:56.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.159+0000 7fd08c9e1700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0880b4010 0x7fd088140c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:56.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.159+0000 7fd08c9e1700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0880b46c0 0x7fd088141180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:56.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.159+0000 7fd08c9e1700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0880b2c40 con 0x7fd0880b46c0 2022-06-29T16:01:56.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.159+0000 7fd08c9e1700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0880b1e70 con 0x7fd0880b4010 2022-06-29T16:01:56.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.159+0000 7fd08c9e1700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0880b2110 con 0x7fd0880ae7c0 2022-06-29T16:01:56.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.159+0000 7fd086d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0880b46c0 0x7fd088141180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:56.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.159+0000 7fd08659c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0880ae7c0 0x7fd088144940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:56.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.159+0000 7fd07dd9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0880b4010 0x7fd088140c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:56.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.159+0000 7fd086d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0880b46c0 0x7fd088141180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47908/0 (socket says 172.21.15.60:47908) 2022-06-29T16:01:56.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.159+0000 7fd08659c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0880ae7c0 0x7fd088144940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:60998/0 (socket says 172.21.15.60:60998) 2022-06-29T16:01:56.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.159+0000 7fd086d9d700 1 -- 172.21.15.60:0/1465841851 learned_addr learned my addr 172.21.15.60:0/1465841851 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:56.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.163+0000 7fd086d9d700 1 -- 172.21.15.60:0/1465841851 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0880ae7c0 msgr2=0x7fd088144940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:56.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.163+0000 7fd086d9d700 1 --2- 172.21.15.60:0/1465841851 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0880ae7c0 0x7fd088144940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:56.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.163+0000 7fd086d9d700 1 -- 172.21.15.60:0/1465841851 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0880b4010 msgr2=0x7fd088140c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:56.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.163+0000 7fd086d9d700 1 --2- 172.21.15.60:0/1465841851 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0880b4010 0x7fd088140c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:56.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.163+0000 7fd086d9d700 1 -- 172.21.15.60:0/1465841851 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd07000b180 con 0x7fd0880b46c0 2022-06-29T16:01:56.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.163+0000 7fd086d9d700 1 --2- 172.21.15.60:0/1465841851 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0880b46c0 0x7fd088141180 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7fd08000c000 tx=0x7fd080009290).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:56.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.163+0000 7fd07ffff700 1 -- 172.21.15.60:0/1465841851 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd080012600 con 0x7fd0880b46c0 2022-06-29T16:01:56.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.163+0000 7fd07ffff700 1 -- 172.21.15.60:0/1465841851 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd080012760 con 0x7fd0880b46c0 2022-06-29T16:01:56.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.163+0000 7fd08c9e1700 1 -- 172.21.15.60:0/1465841851 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0880b23b0 con 0x7fd0880b46c0 2022-06-29T16:01:56.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.163+0000 7fd07ffff700 1 -- 172.21.15.60:0/1465841851 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd080012ac0 con 0x7fd0880b46c0 2022-06-29T16:01:56.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.163+0000 7fd08c9e1700 1 -- 172.21.15.60:0/1465841851 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd088144e30 con 0x7fd0880b46c0 2022-06-29T16:01:56.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.163+0000 7fd08c9e1700 1 -- 172.21.15.60:0/1465841851 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0880573d0 con 0x7fd0880b46c0 2022-06-29T16:01:56.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.163+0000 7fd07ffff700 1 -- 172.21.15.60:0/1465841851 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd080022070 con 0x7fd0880b46c0 2022-06-29T16:01:56.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.163+0000 7fd07ffff700 1 --2- 172.21.15.60:0/1465841851 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd060053aa0 0x7fd060055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:56.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.163+0000 7fd07ffff700 1 -- 172.21.15.60:0/1465841851 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd08006c470 con 0x7fd0880b46c0 2022-06-29T16:01:56.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.167+0000 7fd08659c700 1 --2- 172.21.15.60:0/1465841851 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd060053aa0 0x7fd060055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:56.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.167+0000 7fd08659c700 1 --2- 172.21.15.60:0/1465841851 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd060053aa0 0x7fd060055f60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7fd07800d000 tx=0x7fd078007b80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:56.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.171+0000 7fd07ffff700 1 -- 172.21.15.60:0/1465841851 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd080043390 con 0x7fd0880b46c0 2022-06-29T16:01:56.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.299+0000 7fd08c9e1700 1 -- 172.21.15.60:0/1465841851 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd08805fdb0 con 0x7fd0880b46c0 2022-06-29T16:01:56.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.299+0000 7fd07ffff700 1 -- 172.21.15.60:0/1465841851 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+16040 (secure 0 0 0) 0x7fd080016020 con 0x7fd0880b46c0 2022-06-29T16:01:56.308 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:01:56.308 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":53,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:01:55.736061+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:01:52.588508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2022-06-29T16:01:53.732456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:01:56.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.303+0000 7fd08c9e1700 1 -- 172.21.15.60:0/1465841851 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd060053aa0 msgr2=0x7fd060055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:56.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.303+0000 7fd08c9e1700 1 --2- 172.21.15.60:0/1465841851 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd060053aa0 0x7fd060055f60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7fd07800d000 tx=0x7fd078007b80).stop 2022-06-29T16:01:56.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.303+0000 7fd08c9e1700 1 -- 172.21.15.60:0/1465841851 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0880b46c0 msgr2=0x7fd088141180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:56.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.303+0000 7fd08c9e1700 1 --2- 172.21.15.60:0/1465841851 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0880b46c0 0x7fd088141180 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7fd08000c000 tx=0x7fd080009290).stop 2022-06-29T16:01:56.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.303+0000 7fd08c9e1700 1 -- 172.21.15.60:0/1465841851 shutdown_connections 2022-06-29T16:01:56.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.303+0000 7fd08c9e1700 1 --2- 172.21.15.60:0/1465841851 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd060053aa0 0x7fd060055f60 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:56.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.303+0000 7fd08c9e1700 1 --2- 172.21.15.60:0/1465841851 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0880b46c0 0x7fd088141180 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:56.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.303+0000 7fd08c9e1700 1 --2- 172.21.15.60:0/1465841851 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd0880b4010 0x7fd088140c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:56.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.303+0000 7fd08c9e1700 1 --2- 172.21.15.60:0/1465841851 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd0880ae7c0 0x7fd088144940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:56.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.303+0000 7fd08c9e1700 1 -- 172.21.15.60:0/1465841851 >> 172.21.15.60:0/1465841851 conn(0x7fd0880b9340 msgr2=0x7fd0880b4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:56.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.303+0000 7fd08c9e1700 1 -- 172.21.15.60:0/1465841851 shutdown_connections 2022-06-29T16:01:56.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.303+0000 7fd08c9e1700 1 -- 172.21.15.60:0/1465841851 wait complete. 2022-06-29T16:01:56.322 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T16:01:56.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.431+0000 7fc57759e700 1 Processor -- start 2022-06-29T16:01:56.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.431+0000 7fc57759e700 1 -- start start 2022-06-29T16:01:56.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.431+0000 7fc57759e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc5780b5eb0 0x7fc578142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:56.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.431+0000 7fc57759e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc5780b0280 0x7fc578146650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:56.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.431+0000 7fc57759e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5780b6290 con 0x7fc578146b40 2022-06-29T16:01:56.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.431+0000 7fc57759e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5780b0ab0 con 0x7fc5780b5eb0 2022-06-29T16:01:56.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.431+0000 7fc57759e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5781461b0 con 0x7fc5780b0280 2022-06-29T16:01:56.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.431+0000 7fc57659c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc5780b5eb0 0x7fc578142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:56.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.431+0000 7fc576d9d700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc578146b40 0x7fc5780b2bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56000/0 (socket says 172.21.15.60:56000) 2022-06-29T16:01:56.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.431+0000 7fc576d9d700 1 -- 172.21.15.60:0/587440015 learned_addr learned my addr 172.21.15.60:0/587440015 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:01:56.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.431+0000 7fc575d9b700 1 --2- 172.21.15.60:0/587440015 >> v2:172.21.15.92:3300/0 conn(0x7fc5780b0280 0x7fc578146650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:56.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.435+0000 7fc57659c700 1 -- 172.21.15.60:0/587440015 >> v2:172.21.15.92:3300/0 conn(0x7fc5780b0280 msgr2=0x7fc578146650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:01:56.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.435+0000 7fc57659c700 1 --2- 172.21.15.60:0/587440015 >> v2:172.21.15.92:3300/0 conn(0x7fc5780b0280 0x7fc578146650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:56.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.435+0000 7fc57659c700 1 -- 172.21.15.60:0/587440015 >> v1:172.21.15.60:6789/0 conn(0x7fc578146b40 legacy=0x7fc5780b2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:56.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.435+0000 7fc57659c700 1 -- 172.21.15.60:0/587440015 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc578146f70 con 0x7fc5780b5eb0 2022-06-29T16:01:56.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.435+0000 7fc57559a700 1 -- 172.21.15.60:0/587440015 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373826055 0 0) 0x7fc5780b6290 con 0x7fc578146b40 2022-06-29T16:01:56.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.435+0000 7fc57659c700 1 --2- 172.21.15.60:0/587440015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5780b5eb0 0x7fc578142a20 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7fc560002f70 tx=0x7fc560006df0).ready entity=mon.0 client_cookie=8ef371dde5a5e259 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:56.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.435+0000 7fc57559a700 1 -- 172.21.15.60:0/587440015 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc56000d5c0 con 0x7fc5780b5eb0 2022-06-29T16:01:56.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.435+0000 7fc57559a700 1 -- 172.21.15.60:0/587440015 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc560000e90 con 0x7fc5780b5eb0 2022-06-29T16:01:56.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.435+0000 7fc57559a700 1 -- 172.21.15.60:0/587440015 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc56000d910 con 0x7fc5780b5eb0 2022-06-29T16:01:56.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.435+0000 7fc57759e700 1 -- 172.21.15.60:0/587440015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5780b5eb0 msgr2=0x7fc578142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:56.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.435+0000 7fc57759e700 1 --2- 172.21.15.60:0/587440015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5780b5eb0 0x7fc578142a20 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7fc560002f70 tx=0x7fc560006df0).stop 2022-06-29T16:01:56.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.435+0000 7fc57759e700 1 -- 172.21.15.60:0/587440015 shutdown_connections 2022-06-29T16:01:56.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.435+0000 7fc57759e700 1 --2- 172.21.15.60:0/587440015 >> v2:172.21.15.92:3300/0 conn(0x7fc5780b0280 0x7fc578146650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:56.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.435+0000 7fc57759e700 1 --2- 172.21.15.60:0/587440015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5780b5eb0 0x7fc578142a20 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:56.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.435+0000 7fc57759e700 1 -- 172.21.15.60:0/587440015 >> 172.21.15.60:0/587440015 conn(0x7fc578062090 msgr2=0x7fc578062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:56.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.435+0000 7fc57759e700 1 -- 172.21.15.60:0/587440015 shutdown_connections 2022-06-29T16:01:56.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.435+0000 7fc57759e700 1 -- 172.21.15.60:0/587440015 wait complete. 2022-06-29T16:01:56.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.435+0000 7fc57759e700 1 Processor -- start 2022-06-29T16:01:56.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.435+0000 7fc57759e700 1 -- start start 2022-06-29T16:01:56.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.439+0000 7fc57759e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5780b0280 0x7fc5781440d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:56.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.439+0000 7fc57759e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc5780b5eb0 0x7fc57813f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:56.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.439+0000 7fc57759e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc578146b40 0x7fc57813f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:56.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.439+0000 7fc57759e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5780af130 con 0x7fc578146b40 2022-06-29T16:01:56.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.439+0000 7fc57759e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5780acfa0 con 0x7fc5780b0280 2022-06-29T16:01:56.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.439+0000 7fc57759e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5780ad240 con 0x7fc5780b5eb0 2022-06-29T16:01:56.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.439+0000 7fc576d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc578146b40 0x7fc57813f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:56.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.439+0000 7fc576d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc578146b40 0x7fc57813f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47922/0 (socket says 172.21.15.60:47922) 2022-06-29T16:01:56.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.439+0000 7fc576d9d700 1 -- 172.21.15.60:0/629469350 learned_addr learned my addr 172.21.15.60:0/629469350 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:56.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.439+0000 7fc57659c700 1 --2- 172.21.15.60:0/629469350 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5780b0280 0x7fc5781440d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:56.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.439+0000 7fc575d9b700 1 --2- 172.21.15.60:0/629469350 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc5780b5eb0 0x7fc57813f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:56.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.439+0000 7fc576d9d700 1 -- 172.21.15.60:0/629469350 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc5780b5eb0 msgr2=0x7fc57813f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:56.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.439+0000 7fc576d9d700 1 --2- 172.21.15.60:0/629469350 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc5780b5eb0 0x7fc57813f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:56.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.439+0000 7fc576d9d700 1 -- 172.21.15.60:0/629469350 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5780b0280 msgr2=0x7fc5781440d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:56.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.439+0000 7fc576d9d700 1 --2- 172.21.15.60:0/629469350 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5780b0280 0x7fc5781440d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:56.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.439+0000 7fc576d9d700 1 -- 172.21.15.60:0/629469350 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc56000b180 con 0x7fc578146b40 2022-06-29T16:01:56.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.439+0000 7fc575d9b700 1 --2- 172.21.15.60:0/629469350 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc5780b5eb0 0x7fc57813f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:01:56.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.439+0000 7fc576d9d700 1 --2- 172.21.15.60:0/629469350 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc578146b40 0x7fc57813f7e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7fc570001250 tx=0x7fc570001360).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:56.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.439+0000 7fc56f7fe700 1 -- 172.21.15.60:0/629469350 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc570005c70 con 0x7fc578146b40 2022-06-29T16:01:56.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.439+0000 7fc56f7fe700 1 -- 172.21.15.60:0/629469350 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc570005dd0 con 0x7fc578146b40 2022-06-29T16:01:56.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.439+0000 7fc57759e700 1 -- 172.21.15.60:0/629469350 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc578151a20 con 0x7fc578146b40 2022-06-29T16:01:56.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.439+0000 7fc56f7fe700 1 -- 172.21.15.60:0/629469350 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc570004a90 con 0x7fc578146b40 2022-06-29T16:01:56.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.439+0000 7fc57759e700 1 -- 172.21.15.60:0/629469350 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5780aca90 con 0x7fc578146b40 2022-06-29T16:01:56.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.439+0000 7fc57759e700 1 -- 172.21.15.60:0/629469350 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5780573d0 con 0x7fc578146b40 2022-06-29T16:01:56.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.439+0000 7fc56f7fe700 1 -- 172.21.15.60:0/629469350 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc570004bf0 con 0x7fc578146b40 2022-06-29T16:01:56.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.443+0000 7fc56f7fe700 1 --2- 172.21.15.60:0/629469350 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc55c053a70 0x7fc55c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:56.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.443+0000 7fc56f7fe700 1 -- 172.21.15.60:0/629469350 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc57006bca0 con 0x7fc578146b40 2022-06-29T16:01:56.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.443+0000 7fc57659c700 1 --2- 172.21.15.60:0/629469350 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc55c053a70 0x7fc55c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:56.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.443+0000 7fc57659c700 1 --2- 172.21.15.60:0/629469350 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc55c053a70 0x7fc55c055f30 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7fc56001a000 tx=0x7fc560014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:56.461 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.447+0000 7fc56f7fe700 1 -- 172.21.15.60:0/629469350 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc570042ab0 con 0x7fc578146b40 2022-06-29T16:01:56.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.611+0000 7fc57759e700 1 -- 172.21.15.60:0/629469350 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fc5780008d0 con 0x7fc578146b40 2022-06-29T16:01:56.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.619+0000 7fc56f7fe700 1 -- 172.21.15.60:0/629469350 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v19) v1 ==== 184+0+0 (secure 0 0 0) 0x7fc5700460d0 con 0x7fc578146b40 2022-06-29T16:01:56.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.623+0000 7fc57759e700 1 -- 172.21.15.60:0/629469350 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc55c053a70 msgr2=0x7fc55c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:56.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.623+0000 7fc57759e700 1 --2- 172.21.15.60:0/629469350 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc55c053a70 0x7fc55c055f30 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7fc56001a000 tx=0x7fc560014000).stop 2022-06-29T16:01:56.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.623+0000 7fc57759e700 1 -- 172.21.15.60:0/629469350 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc578146b40 msgr2=0x7fc57813f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:56.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.623+0000 7fc57759e700 1 --2- 172.21.15.60:0/629469350 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc578146b40 0x7fc57813f7e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7fc570001250 tx=0x7fc570001360).stop 2022-06-29T16:01:56.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.623+0000 7fc57759e700 1 -- 172.21.15.60:0/629469350 shutdown_connections 2022-06-29T16:01:56.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.623+0000 7fc57759e700 1 --2- 172.21.15.60:0/629469350 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc55c053a70 0x7fc55c055f30 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:56.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.623+0000 7fc57759e700 1 --2- 172.21.15.60:0/629469350 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc578146b40 0x7fc57813f7e0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:56.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.623+0000 7fc57759e700 1 --2- 172.21.15.60:0/629469350 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc5780b5eb0 0x7fc57813f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:56.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.623+0000 7fc57759e700 1 --2- 172.21.15.60:0/629469350 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5780b0280 0x7fc5781440d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:56.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.623+0000 7fc57759e700 1 -- 172.21.15.60:0/629469350 >> 172.21.15.60:0/629469350 conn(0x7fc578062090 msgr2=0x7fc578140060 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:56.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.623+0000 7fc57759e700 1 -- 172.21.15.60:0/629469350 shutdown_connections 2022-06-29T16:01:56.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.623+0000 7fc57759e700 1 -- 172.21.15.60:0/629469350 wait complete. 2022-06-29T16:01:56.633 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.0 2022-06-29T16:01:56.642 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T16:01:56.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.747+0000 7f23c759e700 1 Processor -- start 2022-06-29T16:01:56.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.747+0000 7f23c759e700 1 -- start start 2022-06-29T16:01:56.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.751+0000 7f23c759e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f23c805fb00 0x7f23c813ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:56.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.751+0000 7f23c759e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23c81403f0 0x7f23c8143090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:56.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.751+0000 7f23c759e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c80b15a0 con 0x7f23c8143580 2022-06-29T16:01:56.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.751+0000 7f23c759e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23c805fee0 con 0x7f23c805fb00 2022-06-29T16:01:56.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.751+0000 7f23c759e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23c8067c10 con 0x7f23c81403f0 2022-06-29T16:01:56.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.751+0000 7f23c6d9d700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f23c8143580 0x7f23c80aca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56014/0 (socket says 172.21.15.60:56014) 2022-06-29T16:01:56.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.751+0000 7f23c6d9d700 1 -- 172.21.15.60:0/4236300376 learned_addr learned my addr 172.21.15.60:0/4236300376 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:01:56.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.751+0000 7f23bffff700 1 --2- 172.21.15.60:0/4236300376 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23c81403f0 0x7f23c8143090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:56.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.751+0000 7f23c659c700 1 --2- 172.21.15.60:0/4236300376 >> v2:172.21.15.92:3300/0 conn(0x7f23c805fb00 0x7f23c813ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:56.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.751+0000 7f23c5d9b700 1 -- 172.21.15.60:0/4236300376 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4021555150 0 0) 0x7f23c80b15a0 con 0x7f23c8143580 2022-06-29T16:01:56.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.751+0000 7f23c5d9b700 1 -- 172.21.15.60:0/4236300376 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23a8003260 con 0x7f23c8143580 2022-06-29T16:01:56.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.751+0000 7f23bffff700 1 -- 172.21.15.60:0/4236300376 >> v2:172.21.15.92:3300/0 conn(0x7f23c805fb00 msgr2=0x7f23c813ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:01:56.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.751+0000 7f23bffff700 1 --2- 172.21.15.60:0/4236300376 >> v2:172.21.15.92:3300/0 conn(0x7f23c805fb00 0x7f23c813ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:56.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.751+0000 7f23bffff700 1 -- 172.21.15.60:0/4236300376 >> v1:172.21.15.60:6789/0 conn(0x7f23c8143580 legacy=0x7f23c80aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:56.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.751+0000 7f23bffff700 1 -- 172.21.15.60:0/4236300376 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c80b17d0 con 0x7f23c81403f0 2022-06-29T16:01:56.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.751+0000 7f23bffff700 1 --2- 172.21.15.60:0/4236300376 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23c81403f0 0x7f23c8143090 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f23b0002f70 tx=0x7f23b0006df0).ready entity=mon.2 client_cookie=2c878303373718d7 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:56.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.751+0000 7f23c5d9b700 1 -- 172.21.15.60:0/4236300376 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f23b000d4a0 con 0x7f23c81403f0 2022-06-29T16:01:56.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.751+0000 7f23c5d9b700 1 -- 172.21.15.60:0/4236300376 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23b0000e90 con 0x7f23c81403f0 2022-06-29T16:01:56.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.751+0000 7f23c5d9b700 1 -- 172.21.15.60:0/4236300376 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f23b000d7f0 con 0x7f23c81403f0 2022-06-29T16:01:56.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.751+0000 7f23c759e700 1 -- 172.21.15.60:0/4236300376 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23c81403f0 msgr2=0x7f23c8143090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:56.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.751+0000 7f23c759e700 1 --2- 172.21.15.60:0/4236300376 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23c81403f0 0x7f23c8143090 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f23b0002f70 tx=0x7f23b0006df0).stop 2022-06-29T16:01:56.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.751+0000 7f23c759e700 1 -- 172.21.15.60:0/4236300376 shutdown_connections 2022-06-29T16:01:56.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.751+0000 7f23c759e700 1 --2- 172.21.15.60:0/4236300376 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23c81403f0 0x7f23c8143090 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:56.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.751+0000 7f23c759e700 1 --2- 172.21.15.60:0/4236300376 >> v2:172.21.15.92:3300/0 conn(0x7f23c805fb00 0x7f23c813ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:56.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.751+0000 7f23c759e700 1 -- 172.21.15.60:0/4236300376 >> 172.21.15.60:0/4236300376 conn(0x7f23c8063ab0 msgr2=0x7f23c805f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:56.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.751+0000 7f23c759e700 1 -- 172.21.15.60:0/4236300376 shutdown_connections 2022-06-29T16:01:56.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.751+0000 7f23c759e700 1 -- 172.21.15.60:0/4236300376 wait complete. 2022-06-29T16:01:56.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.755+0000 7f23c759e700 1 Processor -- start 2022-06-29T16:01:56.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.755+0000 7f23c759e700 1 -- start start 2022-06-29T16:01:56.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.755+0000 7f23c759e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23c805fb00 0x7f23c8066d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:56.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.755+0000 7f23c759e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23c81403f0 0x7f23c8064480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:56.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.755+0000 7f23c759e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23c8143580 0x7f23c8155390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:56.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.755+0000 7f23c759e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c8155880 con 0x7f23c8143580 2022-06-29T16:01:56.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.755+0000 7f23c759e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c8142bc0 con 0x7f23c805fb00 2022-06-29T16:01:56.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.755+0000 7f23c759e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23c8142680 con 0x7f23c81403f0 2022-06-29T16:01:56.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.755+0000 7f23c6d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23c8143580 0x7f23c8155390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:56.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.755+0000 7f23c6d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23c8143580 0x7f23c8155390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47936/0 (socket says 172.21.15.60:47936) 2022-06-29T16:01:56.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.755+0000 7f23c6d9d700 1 -- 172.21.15.60:0/1025590578 learned_addr learned my addr 172.21.15.60:0/1025590578 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:56.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.755+0000 7f23c659c700 1 --2- 172.21.15.60:0/1025590578 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23c805fb00 0x7f23c8066d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:56.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.755+0000 7f23bffff700 1 --2- 172.21.15.60:0/1025590578 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23c81403f0 0x7f23c8064480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:56.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.755+0000 7f23c6d9d700 1 -- 172.21.15.60:0/1025590578 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23c81403f0 msgr2=0x7f23c8064480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:56.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.755+0000 7f23c6d9d700 1 --2- 172.21.15.60:0/1025590578 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23c81403f0 0x7f23c8064480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:56.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.755+0000 7f23c6d9d700 1 -- 172.21.15.60:0/1025590578 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23c805fb00 msgr2=0x7f23c8066d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:56.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.755+0000 7f23c6d9d700 1 --2- 172.21.15.60:0/1025590578 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23c805fb00 0x7f23c8066d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:56.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.755+0000 7f23c6d9d700 1 -- 172.21.15.60:0/1025590578 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b000b180 con 0x7f23c8143580 2022-06-29T16:01:56.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.755+0000 7f23c6d9d700 1 --2- 172.21.15.60:0/1025590578 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23c8143580 0x7f23c8155390 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7f23c0001490 tx=0x7f23c0007e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:56.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.759+0000 7f23bf7fe700 1 -- 172.21.15.60:0/1025590578 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f23c0005ba0 con 0x7f23c8143580 2022-06-29T16:01:56.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.759+0000 7f23bf7fe700 1 -- 172.21.15.60:0/1025590578 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c0005d00 con 0x7f23c8143580 2022-06-29T16:01:56.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.759+0000 7f23bf7fe700 1 -- 172.21.15.60:0/1025590578 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f23c00072c0 con 0x7f23c8143580 2022-06-29T16:01:56.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.759+0000 7f23c759e700 1 -- 172.21.15.60:0/1025590578 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23c8142920 con 0x7f23c8143580 2022-06-29T16:01:56.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.759+0000 7f23c759e700 1 -- 172.21.15.60:0/1025590578 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23c80b2490 con 0x7f23c8143580 2022-06-29T16:01:56.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.759+0000 7f23c759e700 1 -- 172.21.15.60:0/1025590578 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23c80573d0 con 0x7f23c8143580 2022-06-29T16:01:56.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.759+0000 7f23bf7fe700 1 -- 172.21.15.60:0/1025590578 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f23c0010070 con 0x7f23c8143580 2022-06-29T16:01:56.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.759+0000 7f23bf7fe700 1 --2- 172.21.15.60:0/1025590578 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f23ac053a10 0x7f23ac055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:56.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.759+0000 7f23bf7fe700 1 -- 172.21.15.60:0/1025590578 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f23c006b860 con 0x7f23c8143580 2022-06-29T16:01:56.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.759+0000 7f23c659c700 1 --2- 172.21.15.60:0/1025590578 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f23ac053a10 0x7f23ac055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:56.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.759+0000 7f23c659c700 1 --2- 172.21.15.60:0/1025590578 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f23ac053a10 0x7f23ac055ed0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f23b800b000 tx=0x7f23b8006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:56.777 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.767+0000 7f23bf7fe700 1 -- 172.21.15.60:0/1025590578 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f23c0042780 con 0x7f23c8143580 2022-06-29T16:01:56.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.927+0000 7f23c759e700 1 -- 172.21.15.60:0/1025590578 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f23c80674a0 con 0x7f23c8143580 2022-06-29T16:01:56.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.935+0000 7f23bf7fe700 1 -- 172.21.15.60:0/1025590578 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v20) v1 ==== 184+0+0 (secure 0 0 0) 0x7f23c0045da0 con 0x7f23c8143580 2022-06-29T16:01:56.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.939+0000 7f23c759e700 1 -- 172.21.15.60:0/1025590578 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f23ac053a10 msgr2=0x7f23ac055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:56.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.939+0000 7f23c759e700 1 --2- 172.21.15.60:0/1025590578 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f23ac053a10 0x7f23ac055ed0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f23b800b000 tx=0x7f23b8006df0).stop 2022-06-29T16:01:56.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.939+0000 7f23c759e700 1 -- 172.21.15.60:0/1025590578 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23c8143580 msgr2=0x7f23c8155390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:56.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.939+0000 7f23c759e700 1 --2- 172.21.15.60:0/1025590578 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23c8143580 0x7f23c8155390 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7f23c0001490 tx=0x7f23c0007e00).stop 2022-06-29T16:01:56.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.939+0000 7f23c759e700 1 -- 172.21.15.60:0/1025590578 shutdown_connections 2022-06-29T16:01:56.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.939+0000 7f23c759e700 1 --2- 172.21.15.60:0/1025590578 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f23ac053a10 0x7f23ac055ed0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:56.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.939+0000 7f23c759e700 1 --2- 172.21.15.60:0/1025590578 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23c8143580 0x7f23c8155390 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:56.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.939+0000 7f23c759e700 1 --2- 172.21.15.60:0/1025590578 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23c81403f0 0x7f23c8064480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:56.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.939+0000 7f23c759e700 1 --2- 172.21.15.60:0/1025590578 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23c805fb00 0x7f23c8066d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:56.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.939+0000 7f23c759e700 1 -- 172.21.15.60:0/1025590578 >> 172.21.15.60:0/1025590578 conn(0x7f23c8063ab0 msgr2=0x7f23c80b5610 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:56.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.939+0000 7f23c759e700 1 -- 172.21.15.60:0/1025590578 shutdown_connections 2022-06-29T16:01:56.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:56.939+0000 7f23c759e700 1 -- 172.21.15.60:0/1025590578 wait complete. 2022-06-29T16:01:56.950 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.1 2022-06-29T16:01:56.960 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T16:01:57.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.067+0000 7f7e5f2f8700 1 Processor -- start 2022-06-29T16:01:57.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.067+0000 7f7e5f2f8700 1 -- start start 2022-06-29T16:01:57.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.067+0000 7f7e5f2f8700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7e580acbc0 0x7f7e58143f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:57.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.067+0000 7f7e5f2f8700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7e580acfa0 0x7f7e580b1f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:57.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.067+0000 7f7e5f2f8700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e580b1ae0 con 0x7f7e580b2400 2022-06-29T16:01:57.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.067+0000 7f7e5f2f8700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e580b2e00 con 0x7f7e580acfa0 2022-06-29T16:01:57.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.067+0000 7f7e5f2f8700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e580666d0 con 0x7f7e580acbc0 2022-06-29T16:01:57.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.067+0000 7f7e5c893700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7e580acfa0 0x7f7e580b1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:57.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.067+0000 7f7e5c893700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7e580acfa0 0x7f7e580b1f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47942/0 (socket says 172.21.15.60:47942) 2022-06-29T16:01:57.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.067+0000 7f7e5c893700 1 -- 172.21.15.60:0/1833566845 learned_addr learned my addr 172.21.15.60:0/1833566845 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:57.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.067+0000 7f7e5d094700 1 --2- 172.21.15.60:0/1833566845 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7e580acbc0 0x7f7e58143f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:57.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.067+0000 7f7e5c893700 1 -- 172.21.15.60:0/1833566845 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7e580acbc0 msgr2=0x7f7e58143f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:01:57.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.067+0000 7f7e5c893700 1 --2- 172.21.15.60:0/1833566845 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7e580acbc0 0x7f7e58143f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:57.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.067+0000 7f7e5c893700 1 -- 172.21.15.60:0/1833566845 >> v1:172.21.15.92:6789/0 conn(0x7f7e580b2400 legacy=0x7f7e5813a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:57.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.067+0000 7f7e5c893700 1 -- 172.21.15.60:0/1833566845 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e580663c0 con 0x7f7e580acfa0 2022-06-29T16:01:57.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.067+0000 7f7e5c893700 1 --2- 172.21.15.60:0/1833566845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e580acfa0 0x7f7e580b1f10 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7f7e48002f70 tx=0x7f7e48006eb0).ready entity=mon.0 client_cookie=747002c979cfaa6d server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:57.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.067+0000 7f7e4ffff700 1 -- 172.21.15.60:0/1833566845 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e4800d5c0 con 0x7f7e580acfa0 2022-06-29T16:01:57.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.067+0000 7f7e4ffff700 1 -- 172.21.15.60:0/1833566845 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e48000e90 con 0x7f7e580acfa0 2022-06-29T16:01:57.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.067+0000 7f7e4ffff700 1 -- 172.21.15.60:0/1833566845 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e4800d950 con 0x7f7e580acfa0 2022-06-29T16:01:57.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.071+0000 7f7e5f2f8700 1 -- 172.21.15.60:0/1833566845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e580acfa0 msgr2=0x7f7e580b1f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:57.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.071+0000 7f7e5f2f8700 1 --2- 172.21.15.60:0/1833566845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e580acfa0 0x7f7e580b1f10 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7f7e48002f70 tx=0x7f7e48006eb0).stop 2022-06-29T16:01:57.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.071+0000 7f7e5f2f8700 1 -- 172.21.15.60:0/1833566845 shutdown_connections 2022-06-29T16:01:57.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.071+0000 7f7e5f2f8700 1 --2- 172.21.15.60:0/1833566845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e580acfa0 0x7f7e580b1f10 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:57.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.071+0000 7f7e5f2f8700 1 --2- 172.21.15.60:0/1833566845 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7e580acbc0 0x7f7e58143f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:57.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.071+0000 7f7e5f2f8700 1 -- 172.21.15.60:0/1833566845 >> 172.21.15.60:0/1833566845 conn(0x7f7e580b7140 msgr2=0x7f7e580b7520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:57.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.071+0000 7f7e5f2f8700 1 -- 172.21.15.60:0/1833566845 shutdown_connections 2022-06-29T16:01:57.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.071+0000 7f7e5f2f8700 1 -- 172.21.15.60:0/1833566845 wait complete. 2022-06-29T16:01:57.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.071+0000 7f7e5f2f8700 1 Processor -- start 2022-06-29T16:01:57.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.071+0000 7f7e5f2f8700 1 -- start start 2022-06-29T16:01:57.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.071+0000 7f7e5f2f8700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e580acbc0 0x7f7e5814fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:57.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.071+0000 7f7e5f2f8700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e580acfa0 0x7f7e581477d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:57.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.071+0000 7f7e5f2f8700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7e580b2400 0x7f7e581535f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:57.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.071+0000 7f7e5f2f8700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e58153190 con 0x7f7e580acbc0 2022-06-29T16:01:57.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.071+0000 7f7e5f2f8700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5813f8c0 con 0x7f7e580acfa0 2022-06-29T16:01:57.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.071+0000 7f7e5f2f8700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e58140600 con 0x7f7e580b2400 2022-06-29T16:01:57.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.071+0000 7f7e5d094700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e580acbc0 0x7f7e5814fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:57.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.071+0000 7f7e5d094700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e580acbc0 0x7f7e5814fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47946/0 (socket says 172.21.15.60:47946) 2022-06-29T16:01:57.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.071+0000 7f7e5d094700 1 -- 172.21.15.60:0/2515014537 learned_addr learned my addr 172.21.15.60:0/2515014537 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:57.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.071+0000 7f7e5d895700 1 --2- 172.21.15.60:0/2515014537 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7e580b2400 0x7f7e581535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:57.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.071+0000 7f7e5c893700 1 --2- 172.21.15.60:0/2515014537 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e580acfa0 0x7f7e581477d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:57.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.075+0000 7f7e5d094700 1 -- 172.21.15.60:0/2515014537 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7e580b2400 msgr2=0x7f7e581535f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:57.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.075+0000 7f7e5d094700 1 --2- 172.21.15.60:0/2515014537 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7e580b2400 0x7f7e581535f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:57.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.075+0000 7f7e5d094700 1 -- 172.21.15.60:0/2515014537 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e580acfa0 msgr2=0x7f7e581477d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:57.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.075+0000 7f7e5d094700 1 --2- 172.21.15.60:0/2515014537 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e580acfa0 0x7f7e581477d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:57.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.075+0000 7f7e5d094700 1 -- 172.21.15.60:0/2515014537 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e4800b1e0 con 0x7f7e580acbc0 2022-06-29T16:01:57.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.075+0000 7f7e5c893700 1 --2- 172.21.15.60:0/2515014537 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e580acfa0 0x7f7e581477d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:01:57.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.075+0000 7f7e5d094700 1 --2- 172.21.15.60:0/2515014537 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e580acbc0 0x7f7e5814fbc0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7f7e540030b0 tx=0x7f7e54008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:57.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.075+0000 7f7e4e7fc700 1 -- 172.21.15.60:0/2515014537 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e54011780 con 0x7f7e580acbc0 2022-06-29T16:01:57.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.075+0000 7f7e4e7fc700 1 -- 172.21.15.60:0/2515014537 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e540118e0 con 0x7f7e580acbc0 2022-06-29T16:01:57.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.075+0000 7f7e5f2f8700 1 -- 172.21.15.60:0/2515014537 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e581408a0 con 0x7f7e580acbc0 2022-06-29T16:01:57.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.075+0000 7f7e4e7fc700 1 -- 172.21.15.60:0/2515014537 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e54011c40 con 0x7f7e580acbc0 2022-06-29T16:01:57.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.075+0000 7f7e5f2f8700 1 -- 172.21.15.60:0/2515014537 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e580b1400 con 0x7f7e580acbc0 2022-06-29T16:01:57.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.075+0000 7f7e5f2f8700 1 -- 172.21.15.60:0/2515014537 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e58153350 con 0x7f7e580acbc0 2022-06-29T16:01:57.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.075+0000 7f7e4e7fc700 1 -- 172.21.15.60:0/2515014537 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7e54012420 con 0x7f7e580acbc0 2022-06-29T16:01:57.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.075+0000 7f7e4e7fc700 1 --2- 172.21.15.60:0/2515014537 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7e40053a70 0x7f7e40055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:57.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.075+0000 7f7e4e7fc700 1 -- 172.21.15.60:0/2515014537 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7e5406bef0 con 0x7f7e580acbc0 2022-06-29T16:01:57.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.075+0000 7f7e5c893700 1 --2- 172.21.15.60:0/2515014537 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7e40053a70 0x7f7e40055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:57.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.079+0000 7f7e5c893700 1 --2- 172.21.15.60:0/2515014537 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7e40053a70 0x7f7e40055f30 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f7e48017000 tx=0x7f7e48015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:57.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.083+0000 7f7e4e7fc700 1 -- 172.21.15.60:0/2515014537 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7e58153350 con 0x7f7e580acbc0 2022-06-29T16:01:57.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.251+0000 7f7e5f2f8700 1 -- 172.21.15.60:0/2515014537 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f7e580573d0 con 0x7f7e580acbc0 2022-06-29T16:01:57.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.259+0000 7f7e4e7fc700 1 -- 172.21.15.60:0/2515014537 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v21) v1 ==== 184+0+0 (secure 0 0 0) 0x7f7e540466d0 con 0x7f7e580acbc0 2022-06-29T16:01:57.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.263+0000 7f7e5f2f8700 1 -- 172.21.15.60:0/2515014537 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7e40053a70 msgr2=0x7f7e40055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:57.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.263+0000 7f7e5f2f8700 1 --2- 172.21.15.60:0/2515014537 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7e40053a70 0x7f7e40055f30 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f7e48017000 tx=0x7f7e48015000).stop 2022-06-29T16:01:57.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.263+0000 7f7e5f2f8700 1 -- 172.21.15.60:0/2515014537 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e580acbc0 msgr2=0x7f7e5814fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:57.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.263+0000 7f7e5f2f8700 1 --2- 172.21.15.60:0/2515014537 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e580acbc0 0x7f7e5814fbc0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7f7e540030b0 tx=0x7f7e54008040).stop 2022-06-29T16:01:57.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.263+0000 7f7e5f2f8700 1 -- 172.21.15.60:0/2515014537 shutdown_connections 2022-06-29T16:01:57.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.263+0000 7f7e5f2f8700 1 --2- 172.21.15.60:0/2515014537 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7e40053a70 0x7f7e40055f30 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:57.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.263+0000 7f7e5f2f8700 1 --2- 172.21.15.60:0/2515014537 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7e580b2400 0x7f7e581535f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:57.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.263+0000 7f7e5f2f8700 1 --2- 172.21.15.60:0/2515014537 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7e580acfa0 0x7f7e581477d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:57.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.263+0000 7f7e5f2f8700 1 --2- 172.21.15.60:0/2515014537 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e580acbc0 0x7f7e5814fbc0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:57.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.263+0000 7f7e5f2f8700 1 -- 172.21.15.60:0/2515014537 >> 172.21.15.60:0/2515014537 conn(0x7f7e580b7140 msgr2=0x7f7e580b7520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:57.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.263+0000 7f7e5f2f8700 1 -- 172.21.15.60:0/2515014537 shutdown_connections 2022-06-29T16:01:57.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.263+0000 7f7e5f2f8700 1 -- 172.21.15.60:0/2515014537 wait complete. 2022-06-29T16:01:57.274 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.2 2022-06-29T16:01:57.287 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:01:57.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.383+0000 7f4eaf59e700 1 Processor -- start 2022-06-29T16:01:57.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.383+0000 7f4eaf59e700 1 -- start start 2022-06-29T16:01:57.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.383+0000 7f4eaf59e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4eb00b3270 0x7f4eb00ace00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:57.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.383+0000 7f4eaf59e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eb00adc80 con 0x7f4eb00aca20 2022-06-29T16:01:57.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.383+0000 7f4eaf59e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eb00ad740 con 0x7f4eb0062cd0 2022-06-29T16:01:57.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.383+0000 7f4eaf59e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4eb00ad9e0 con 0x7f4eb00b3270 2022-06-29T16:01:57.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.383+0000 7f4eadd9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4eb00b3270 0x7f4eb00ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:57.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.383+0000 7f4eae59c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4eb00aca20 0x7f4eb0143030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56040/0 (socket says 172.21.15.60:56040) 2022-06-29T16:01:57.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.383+0000 7f4eae59c700 1 -- 172.21.15.60:0/1523523566 learned_addr learned my addr 172.21.15.60:0/1523523566 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:01:57.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.387+0000 7f4eadd9b700 1 -- 172.21.15.60:0/1523523566 >> v1:172.21.15.92:6789/0 conn(0x7f4eb0062cd0 legacy=0x7f4eb01461c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:57.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.387+0000 7f4eadd9b700 1 -- 172.21.15.60:0/1523523566 >> v1:172.21.15.60:6789/0 conn(0x7f4eb00aca20 legacy=0x7f4eb0143030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:57.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.387+0000 7f4eadd9b700 1 -- 172.21.15.60:0/1523523566 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb0142920 con 0x7f4eb00b3270 2022-06-29T16:01:57.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.387+0000 7f4eadd9b700 1 --2- 172.21.15.60:0/1523523566 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4eb00b3270 0x7f4eb00ace00 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7f4e98002f70 tx=0x7f4e98006df0).ready entity=mon.0 client_cookie=4dd185e9b618886a server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:57.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.387+0000 7f4ead59a700 1 -- 172.21.15.60:0/1523523566 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e9800d610 con 0x7f4eb00b3270 2022-06-29T16:01:57.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.387+0000 7f4ead59a700 1 -- 172.21.15.60:0/1523523566 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e98000e90 con 0x7f4eb00b3270 2022-06-29T16:01:57.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.387+0000 7f4ead59a700 1 -- 172.21.15.60:0/1523523566 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e9800d960 con 0x7f4eb00b3270 2022-06-29T16:01:57.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.387+0000 7f4eaf59e700 1 -- 172.21.15.60:0/1523523566 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4eb00b3270 msgr2=0x7f4eb00ace00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:57.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.387+0000 7f4eaf59e700 1 --2- 172.21.15.60:0/1523523566 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4eb00b3270 0x7f4eb00ace00 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7f4e98002f70 tx=0x7f4e98006df0).stop 2022-06-29T16:01:57.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.387+0000 7f4eaf59e700 1 -- 172.21.15.60:0/1523523566 shutdown_connections 2022-06-29T16:01:57.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.387+0000 7f4eaf59e700 1 --2- 172.21.15.60:0/1523523566 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4eb00b3270 0x7f4eb00ace00 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:57.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.387+0000 7f4eaf59e700 1 -- 172.21.15.60:0/1523523566 >> 172.21.15.60:0/1523523566 conn(0x7f4eb00b6f80 msgr2=0x7f4eb00b7390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:57.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.387+0000 7f4eaf59e700 1 -- 172.21.15.60:0/1523523566 shutdown_connections 2022-06-29T16:01:57.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.387+0000 7f4eaf59e700 1 -- 172.21.15.60:0/1523523566 wait complete. 2022-06-29T16:01:57.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.391+0000 7f4eaf59e700 1 Processor -- start 2022-06-29T16:01:57.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.391+0000 7f4eaf59e700 1 -- start start 2022-06-29T16:01:57.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.391+0000 7f4eaf59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4eb0062cd0 0x7f4eb01479b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:57.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.391+0000 7f4eaf59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4eb00aca20 0x7f4eb01552d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:57.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.391+0000 7f4eaf59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4eb00b3270 0x7f4eb00665e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:57.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.391+0000 7f4eaf59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb013d650 con 0x7f4eb00b3270 2022-06-29T16:01:57.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.391+0000 7f4eaf59e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb00b0b60 con 0x7f4eb0062cd0 2022-06-29T16:01:57.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.391+0000 7f4eaf59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb013d1b0 con 0x7f4eb00aca20 2022-06-29T16:01:57.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.391+0000 7f4eaed9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4eb00b3270 0x7f4eb00665e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:57.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.391+0000 7f4eae59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4eb0062cd0 0x7f4eb01479b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:57.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.391+0000 7f4eaed9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4eb00b3270 0x7f4eb00665e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47964/0 (socket says 172.21.15.60:47964) 2022-06-29T16:01:57.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.391+0000 7f4eae59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4eb0062cd0 0x7f4eb01479b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:53990/0 (socket says 172.21.15.60:53990) 2022-06-29T16:01:57.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.391+0000 7f4eaed9d700 1 -- 172.21.15.60:0/1419027138 learned_addr learned my addr 172.21.15.60:0/1419027138 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:57.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.391+0000 7f4eadd9b700 1 --2- 172.21.15.60:0/1419027138 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4eb00aca20 0x7f4eb01552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:57.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.391+0000 7f4eaed9d700 1 -- 172.21.15.60:0/1419027138 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4eb00aca20 msgr2=0x7f4eb01552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:57.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.391+0000 7f4eaed9d700 1 --2- 172.21.15.60:0/1419027138 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4eb00aca20 0x7f4eb01552d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:57.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.391+0000 7f4eaed9d700 1 -- 172.21.15.60:0/1419027138 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4eb0062cd0 msgr2=0x7f4eb01479b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:57.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.391+0000 7f4eaed9d700 1 --2- 172.21.15.60:0/1419027138 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4eb0062cd0 0x7f4eb01479b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:57.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.391+0000 7f4eaed9d700 1 -- 172.21.15.60:0/1419027138 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e9800b180 con 0x7f4eb00b3270 2022-06-29T16:01:57.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.391+0000 7f4eaed9d700 1 --2- 172.21.15.60:0/1419027138 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4eb00b3270 0x7f4eb00665e0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f4eb00b7c60 tx=0x7f4ea8007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:57.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.391+0000 7f4ea77fe700 1 -- 172.21.15.60:0/1419027138 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ea800f470 con 0x7f4eb00b3270 2022-06-29T16:01:57.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.391+0000 7f4ea77fe700 1 -- 172.21.15.60:0/1419027138 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea800f5d0 con 0x7f4eb00b3270 2022-06-29T16:01:57.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.391+0000 7f4eaf59e700 1 -- 172.21.15.60:0/1419027138 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eb00af220 con 0x7f4eb00b3270 2022-06-29T16:01:57.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.391+0000 7f4ea77fe700 1 -- 172.21.15.60:0/1419027138 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ea800f930 con 0x7f4eb00b3270 2022-06-29T16:01:57.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.391+0000 7f4eaf59e700 1 -- 172.21.15.60:0/1419027138 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eb0066ad0 con 0x7f4eb00b3270 2022-06-29T16:01:57.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.395+0000 7f4eaf59e700 1 -- 172.21.15.60:0/1419027138 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eb013d370 con 0x7f4eb00b3270 2022-06-29T16:01:57.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.395+0000 7f4ea77fe700 1 -- 172.21.15.60:0/1419027138 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4ea8019030 con 0x7f4eb00b3270 2022-06-29T16:01:57.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.395+0000 7f4ea77fe700 1 --2- 172.21.15.60:0/1419027138 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4e94053b30 0x7f4e94055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:57.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.395+0000 7f4ea77fe700 1 -- 172.21.15.60:0/1419027138 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4ea806b4f0 con 0x7f4eb00b3270 2022-06-29T16:01:57.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.395+0000 7f4eae59c700 1 --2- 172.21.15.60:0/1419027138 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4e94053b30 0x7f4e94055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:57.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.395+0000 7f4eae59c700 1 --2- 172.21.15.60:0/1419027138 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4e94053b30 0x7f4e94055ff0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f4ea000d000 tx=0x7f4ea0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:57.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.403+0000 7f4ea77fe700 1 -- 172.21.15.60:0/1419027138 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4ea8042410 con 0x7f4eb00b3270 2022-06-29T16:01:57.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.571+0000 7f4eaf59e700 1 -- 172.21.15.60:0/1419027138 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4eb00b0d20 con 0x7f4eb00b3270 2022-06-29T16:01:57.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.571+0000 7f4ea77fe700 1 -- 172.21.15.60:0/1419027138 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+2139 (secure 0 0 0) 0x7f4ea8045a30 con 0x7f4eb00b3270 2022-06-29T16:01:57.581 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:01:57.581 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":20,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5568,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/4229695218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":4229695218},{"type":"v1","addr":"172.21.15.60:6833","nonce":4229695218}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:01:54.732520+0000","modified":"2022-06-29T16:01:55.735497+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5653":{"gid":5653,"name":"b","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/1602880694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1602880694},{"type":"v1","addr":"172.21.15.92:6835","nonce":1602880694}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-06-29T16:01:57.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.575+0000 7f4eaf59e700 1 -- 172.21.15.60:0/1419027138 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4e94053b30 msgr2=0x7f4e94055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:57.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.575+0000 7f4eaf59e700 1 --2- 172.21.15.60:0/1419027138 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4e94053b30 0x7f4e94055ff0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f4ea000d000 tx=0x7f4ea0009040).stop 2022-06-29T16:01:57.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.575+0000 7f4eaf59e700 1 -- 172.21.15.60:0/1419027138 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4eb00b3270 msgr2=0x7f4eb00665e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:57.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.575+0000 7f4eaf59e700 1 --2- 172.21.15.60:0/1419027138 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4eb00b3270 0x7f4eb00665e0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f4eb00b7c60 tx=0x7f4ea8007f40).stop 2022-06-29T16:01:57.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.575+0000 7f4eaf59e700 1 -- 172.21.15.60:0/1419027138 shutdown_connections 2022-06-29T16:01:57.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.575+0000 7f4eaf59e700 1 --2- 172.21.15.60:0/1419027138 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4e94053b30 0x7f4e94055ff0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:57.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.575+0000 7f4eaf59e700 1 --2- 172.21.15.60:0/1419027138 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4eb00b3270 0x7f4eb00665e0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:57.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.575+0000 7f4eaf59e700 1 --2- 172.21.15.60:0/1419027138 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4eb00aca20 0x7f4eb01552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:57.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.575+0000 7f4eaf59e700 1 --2- 172.21.15.60:0/1419027138 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4eb0062cd0 0x7f4eb01479b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:57.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.575+0000 7f4eaf59e700 1 -- 172.21.15.60:0/1419027138 >> 172.21.15.60:0/1419027138 conn(0x7f4eb00b6f80 msgr2=0x7f4eb0148c70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:57.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.575+0000 7f4eaf59e700 1 -- 172.21.15.60:0/1419027138 shutdown_connections 2022-06-29T16:01:57.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:57.575+0000 7f4eaf59e700 1 -- 172.21.15.60:0/1419027138 wait complete. 2022-06-29T16:01:57.586 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 20 2022-06-29T16:01:57.794 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:01:54.732520+0000', 'modified': '2022-06-29T16:01:55.735497+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5653}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5653': {'gid': 5653, 'name': 'b', 'rank': 0, 'incarnation': 19, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.92:6835/1602880694', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 1602880694}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 1602880694}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:01:57.795 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:01:57.796 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-06-29T16:01:57.910 INFO:teuthology.orchestra.run.smithi092.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":5653,"want_state":"up:active","state":"up:active","rank_uptime":3.1604668469999999,"mdsmap_epoch":20,"osdmap_epoch":53,"osdmap_epoch_barrier":52,"uptime":8.1109540500000001} 2022-06-29T16:01:57.910 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 5653, "mdsmap_epoch": 20, "osdmap_epoch": 53, "osdmap_epoch_barrier": 52, "rank_uptime": 3.160466847, "state": "up:active", "uptime": 8.11095405, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:01:57.911 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:01:58.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.015+0000 7fa07b22d700 1 Processor -- start 2022-06-29T16:01:58.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.015+0000 7fa07b22d700 1 -- start start 2022-06-29T16:01:58.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.015+0000 7fa07b22d700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa07413a920 0x7fa074143f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:58.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.015+0000 7fa07b22d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa074140ab0 0x7fa074140f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:58.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.015+0000 7fa07b22d700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa07413ad00 con 0x7fa0740ae270 2022-06-29T16:01:58.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.015+0000 7fa07b22d700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa074144420 con 0x7fa07413a920 2022-06-29T16:01:58.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.015+0000 7fa07b22d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0740b5d20 con 0x7fa074140ab0 2022-06-29T16:01:58.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.015+0000 7fa073fff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa0740ae270 0x7fa0740b5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56054/0 (socket says 172.21.15.60:56054) 2022-06-29T16:01:58.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.015+0000 7fa073fff700 1 -- 172.21.15.60:0/3874214356 learned_addr learned my addr 172.21.15.60:0/3874214356 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:01:58.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.015+0000 7fa0797ca700 1 --2- 172.21.15.60:0/3874214356 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa074140ab0 0x7fa074140f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:58.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.015+0000 7fa078fc9700 1 --2- 172.21.15.60:0/3874214356 >> v2:172.21.15.92:3300/0 conn(0x7fa07413a920 0x7fa074143f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:58.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.015+0000 7fa0737fe700 1 -- 172.21.15.60:0/3874214356 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 315436851 0 0) 0x7fa07413ad00 con 0x7fa0740ae270 2022-06-29T16:01:58.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.015+0000 7fa0737fe700 1 -- 172.21.15.60:0/3874214356 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa054003260 con 0x7fa0740ae270 2022-06-29T16:01:58.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.015+0000 7fa0797ca700 1 -- 172.21.15.60:0/3874214356 >> v2:172.21.15.92:3300/0 conn(0x7fa07413a920 msgr2=0x7fa074143f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:01:58.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.015+0000 7fa0797ca700 1 --2- 172.21.15.60:0/3874214356 >> v2:172.21.15.92:3300/0 conn(0x7fa07413a920 0x7fa074143f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:58.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.015+0000 7fa0797ca700 1 -- 172.21.15.60:0/3874214356 >> v1:172.21.15.60:6789/0 conn(0x7fa0740ae270 legacy=0x7fa0740b5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:58.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.015+0000 7fa0797ca700 1 -- 172.21.15.60:0/3874214356 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa074144670 con 0x7fa074140ab0 2022-06-29T16:01:58.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.019+0000 7fa0797ca700 1 --2- 172.21.15.60:0/3874214356 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa074140ab0 0x7fa074140f40 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fa06c0032f0 tx=0x7fa06c0091d0).ready entity=mon.2 client_cookie=6491ca43aca7b81 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:58.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.019+0000 7fa0737fe700 1 -- 172.21.15.60:0/3874214356 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa06c00b600 con 0x7fa074140ab0 2022-06-29T16:01:58.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.019+0000 7fa0737fe700 1 -- 172.21.15.60:0/3874214356 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa06c00b760 con 0x7fa074140ab0 2022-06-29T16:01:58.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.019+0000 7fa0737fe700 1 -- 172.21.15.60:0/3874214356 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa06c00b8d0 con 0x7fa074140ab0 2022-06-29T16:01:58.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.019+0000 7fa07b22d700 1 -- 172.21.15.60:0/3874214356 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa074140ab0 msgr2=0x7fa074140f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:58.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.019+0000 7fa07b22d700 1 --2- 172.21.15.60:0/3874214356 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa074140ab0 0x7fa074140f40 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fa06c0032f0 tx=0x7fa06c0091d0).stop 2022-06-29T16:01:58.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.019+0000 7fa07b22d700 1 -- 172.21.15.60:0/3874214356 shutdown_connections 2022-06-29T16:01:58.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.019+0000 7fa07b22d700 1 --2- 172.21.15.60:0/3874214356 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa074140ab0 0x7fa074140f40 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:58.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.019+0000 7fa07b22d700 1 --2- 172.21.15.60:0/3874214356 >> v2:172.21.15.92:3300/0 conn(0x7fa07413a920 0x7fa074143f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:58.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.019+0000 7fa07b22d700 1 -- 172.21.15.60:0/3874214356 >> 172.21.15.60:0/3874214356 conn(0x7fa0740632c0 msgr2=0x7fa0740b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:58.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.019+0000 7fa07b22d700 1 -- 172.21.15.60:0/3874214356 shutdown_connections 2022-06-29T16:01:58.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.019+0000 7fa07b22d700 1 -- 172.21.15.60:0/3874214356 wait complete. 2022-06-29T16:01:58.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.023+0000 7fa07b22d700 1 Processor -- start 2022-06-29T16:01:58.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.023+0000 7fa07b22d700 1 -- start start 2022-06-29T16:01:58.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.023+0000 7fa07b22d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0740ae270 0x7fa0740b5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:58.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.023+0000 7fa07b22d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa07413a920 0x7fa07413b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:58.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.023+0000 7fa07b22d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa074140ab0 0x7fa07413c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:58.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.023+0000 7fa07b22d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa074142580 con 0x7fa07413a920 2022-06-29T16:01:58.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.023+0000 7fa07b22d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa07413ad00 con 0x7fa0740ae270 2022-06-29T16:01:58.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.023+0000 7fa07b22d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa074144110 con 0x7fa074140ab0 2022-06-29T16:01:58.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.023+0000 7fa073fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa07413a920 0x7fa07413b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:58.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.023+0000 7fa073fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa07413a920 0x7fa07413b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47976/0 (socket says 172.21.15.60:47976) 2022-06-29T16:01:58.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.023+0000 7fa073fff700 1 -- 172.21.15.60:0/3020102077 learned_addr learned my addr 172.21.15.60:0/3020102077 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:58.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.023+0000 7fa0797ca700 1 --2- 172.21.15.60:0/3020102077 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa074140ab0 0x7fa07413c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:58.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.023+0000 7fa078fc9700 1 --2- 172.21.15.60:0/3020102077 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0740ae270 0x7fa0740b5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:58.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.023+0000 7fa073fff700 1 -- 172.21.15.60:0/3020102077 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa074140ab0 msgr2=0x7fa07413c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:58.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.023+0000 7fa073fff700 1 --2- 172.21.15.60:0/3020102077 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa074140ab0 0x7fa07413c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:58.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.023+0000 7fa073fff700 1 -- 172.21.15.60:0/3020102077 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0740ae270 msgr2=0x7fa0740b5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:58.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.023+0000 7fa073fff700 1 --2- 172.21.15.60:0/3020102077 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0740ae270 0x7fa0740b5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:58.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.023+0000 7fa073fff700 1 -- 172.21.15.60:0/3020102077 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa06c009470 con 0x7fa07413a920 2022-06-29T16:01:58.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.023+0000 7fa078fc9700 1 --2- 172.21.15.60:0/3020102077 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0740ae270 0x7fa0740b5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:01:58.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.023+0000 7fa073fff700 1 --2- 172.21.15.60:0/3020102077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa07413a920 0x7fa07413b130 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7fa064001110 tx=0x7fa064007aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:58.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.027+0000 7fa071ffb700 1 -- 172.21.15.60:0/3020102077 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0640070d0 con 0x7fa07413a920 2022-06-29T16:01:58.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.027+0000 7fa071ffb700 1 -- 172.21.15.60:0/3020102077 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa064007230 con 0x7fa07413a920 2022-06-29T16:01:58.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.027+0000 7fa07b22d700 1 -- 172.21.15.60:0/3020102077 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa07413b620 con 0x7fa07413a920 2022-06-29T16:01:58.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.027+0000 7fa071ffb700 1 -- 172.21.15.60:0/3020102077 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa064005610 con 0x7fa07413a920 2022-06-29T16:01:58.042 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.027+0000 7fa07b22d700 1 -- 172.21.15.60:0/3020102077 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa074143b80 con 0x7fa07413a920 2022-06-29T16:01:58.044 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.027+0000 7fa07b22d700 1 -- 172.21.15.60:0/3020102077 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa074144910 con 0x7fa07413a920 2022-06-29T16:01:58.044 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.027+0000 7fa071ffb700 1 -- 172.21.15.60:0/3020102077 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa064010070 con 0x7fa07413a920 2022-06-29T16:01:58.044 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.027+0000 7fa071ffb700 1 --2- 172.21.15.60:0/3020102077 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa058053a70 0x7fa058055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:58.045 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.027+0000 7fa071ffb700 1 -- 172.21.15.60:0/3020102077 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa06406b5c0 con 0x7fa07413a920 2022-06-29T16:01:58.045 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.027+0000 7fa078fc9700 1 --2- 172.21.15.60:0/3020102077 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa058053a70 0x7fa058055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:58.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.031+0000 7fa078fc9700 1 --2- 172.21.15.60:0/3020102077 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa058053a70 0x7fa058055f30 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fa05c00b000 tx=0x7fa05c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:58.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.035+0000 7fa071ffb700 1 -- 172.21.15.60:0/3020102077 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa0640423d0 con 0x7fa07413a920 2022-06-29T16:01:58.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.159+0000 7fa07b22d700 1 -- 172.21.15.60:0/3020102077 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0740573d0 con 0x7fa07413a920 2022-06-29T16:01:58.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.163+0000 7fa071ffb700 1 -- 172.21.15.60:0/3020102077 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+16040 (secure 0 0 0) 0x7fa0640459f0 con 0x7fa07413a920 2022-06-29T16:01:58.171 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:01:58.171 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":53,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:01:55.736061+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:01:52.588508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2022-06-29T16:01:53.732456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:01:58.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.163+0000 7fa07b22d700 1 -- 172.21.15.60:0/3020102077 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa058053a70 msgr2=0x7fa058055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:58.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.163+0000 7fa07b22d700 1 --2- 172.21.15.60:0/3020102077 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa058053a70 0x7fa058055f30 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fa05c00b000 tx=0x7fa05c008040).stop 2022-06-29T16:01:58.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.163+0000 7fa07b22d700 1 -- 172.21.15.60:0/3020102077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa07413a920 msgr2=0x7fa07413b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:58.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.163+0000 7fa07b22d700 1 --2- 172.21.15.60:0/3020102077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa07413a920 0x7fa07413b130 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7fa064001110 tx=0x7fa064007aa0).stop 2022-06-29T16:01:58.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.163+0000 7fa07b22d700 1 -- 172.21.15.60:0/3020102077 shutdown_connections 2022-06-29T16:01:58.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.163+0000 7fa07b22d700 1 --2- 172.21.15.60:0/3020102077 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa058053a70 0x7fa058055f30 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:58.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.163+0000 7fa07b22d700 1 --2- 172.21.15.60:0/3020102077 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa074140ab0 0x7fa07413c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:58.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.163+0000 7fa07b22d700 1 --2- 172.21.15.60:0/3020102077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa07413a920 0x7fa07413b130 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:58.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.163+0000 7fa07b22d700 1 --2- 172.21.15.60:0/3020102077 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0740ae270 0x7fa0740b5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:58.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.163+0000 7fa07b22d700 1 -- 172.21.15.60:0/3020102077 >> 172.21.15.60:0/3020102077 conn(0x7fa0740632c0 msgr2=0x7fa074137c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:58.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.163+0000 7fa07b22d700 1 -- 172.21.15.60:0/3020102077 shutdown_connections 2022-06-29T16:01:58.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.163+0000 7fa07b22d700 1 -- 172.21.15.60:0/3020102077 wait complete. 2022-06-29T16:01:58.190 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-29T16:01:52.588508+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2022-06-29T16:01:53.732456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-29T16:01:58.191 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:01:58.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.291+0000 7f56f0830700 1 Processor -- start 2022-06-29T16:01:58.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.291+0000 7f56f0830700 1 -- start start 2022-06-29T16:01:58.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.291+0000 7f56f0830700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56ec0b3c70 0x7f56ec05c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:58.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.291+0000 7f56f0830700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f56ec0aca20 0x7f56ec0b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:58.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.291+0000 7f56f0830700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56ec0b4050 con 0x7f56ec0b2150 2022-06-29T16:01:58.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.291+0000 7f56f0830700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56ec05bea0 con 0x7f56ec0aca20 2022-06-29T16:01:58.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.291+0000 7f56f0830700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56ec05c140 con 0x7f56ec0b3c70 2022-06-29T16:01:58.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.291+0000 7f56eaffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f56ec0aca20 0x7f56ec0b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:58.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.291+0000 7f56ebfff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f56ec0b2150 0x7f56ec0b25e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56070/0 (socket says 172.21.15.60:56070) 2022-06-29T16:01:58.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.291+0000 7f56ebfff700 1 -- 172.21.15.60:0/2945307644 learned_addr learned my addr 172.21.15.60:0/2945307644 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:01:58.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.291+0000 7f56eb7fe700 1 --2- 172.21.15.60:0/2945307644 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56ec0b3c70 0x7f56ec05c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:58.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.291+0000 7f56ea7fc700 1 -- 172.21.15.60:0/2945307644 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 192635825 0 0) 0x7f56ec0b4050 con 0x7f56ec0b2150 2022-06-29T16:01:58.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.291+0000 7f56ea7fc700 1 -- 172.21.15.60:0/2945307644 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56d0003260 con 0x7f56ec0b2150 2022-06-29T16:01:58.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.291+0000 7f56eaffd700 1 -- 172.21.15.60:0/2945307644 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56ec0b3c70 msgr2=0x7f56ec05c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:01:58.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.291+0000 7f56eaffd700 1 --2- 172.21.15.60:0/2945307644 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56ec0b3c70 0x7f56ec05c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:58.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.291+0000 7f56eaffd700 1 -- 172.21.15.60:0/2945307644 >> v1:172.21.15.60:6789/0 conn(0x7f56ec0b2150 legacy=0x7f56ec0b25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:58.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.291+0000 7f56eaffd700 1 -- 172.21.15.60:0/2945307644 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56ec05c3e0 con 0x7f56ec0aca20 2022-06-29T16:01:58.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.291+0000 7f56eb7fe700 1 --2- 172.21.15.60:0/2945307644 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56ec0b3c70 0x7f56ec05c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:01:58.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.291+0000 7f56eaffd700 1 --2- 172.21.15.60:0/2945307644 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56ec0aca20 0x7f56ec0b19f0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f56d4002f70 tx=0x7f56d4006df0).ready entity=mon.0 client_cookie=a79cd5c734bd59a4 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:58.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.291+0000 7f56ea7fc700 1 -- 172.21.15.60:0/2945307644 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f56d400d570 con 0x7f56ec0aca20 2022-06-29T16:01:58.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.291+0000 7f56ea7fc700 1 -- 172.21.15.60:0/2945307644 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d4000e90 con 0x7f56ec0aca20 2022-06-29T16:01:58.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.291+0000 7f56ea7fc700 1 -- 172.21.15.60:0/2945307644 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f56d400d8c0 con 0x7f56ec0aca20 2022-06-29T16:01:58.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.295+0000 7f56f0830700 1 -- 172.21.15.60:0/2945307644 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56ec0aca20 msgr2=0x7f56ec0b19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:58.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.295+0000 7f56f0830700 1 --2- 172.21.15.60:0/2945307644 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56ec0aca20 0x7f56ec0b19f0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f56d4002f70 tx=0x7f56d4006df0).stop 2022-06-29T16:01:58.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.295+0000 7f56f0830700 1 -- 172.21.15.60:0/2945307644 shutdown_connections 2022-06-29T16:01:58.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.295+0000 7f56f0830700 1 --2- 172.21.15.60:0/2945307644 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56ec0aca20 0x7f56ec0b19f0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:58.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.295+0000 7f56f0830700 1 --2- 172.21.15.60:0/2945307644 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56ec0b3c70 0x7f56ec05c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:58.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.295+0000 7f56f0830700 1 -- 172.21.15.60:0/2945307644 >> 172.21.15.60:0/2945307644 conn(0x7f56ec064810 msgr2=0x7f56ec064bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:58.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.295+0000 7f56f0830700 1 -- 172.21.15.60:0/2945307644 shutdown_connections 2022-06-29T16:01:58.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.295+0000 7f56f0830700 1 -- 172.21.15.60:0/2945307644 wait complete. 2022-06-29T16:01:58.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.295+0000 7f56f0830700 1 Processor -- start 2022-06-29T16:01:58.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.295+0000 7f56f0830700 1 -- start start 2022-06-29T16:01:58.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.299+0000 7f56f0830700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56ec0aca20 0x7f56ec0ad220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:58.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.299+0000 7f56f0830700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56ec0b2150 0x7f56ec1395b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:58.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.299+0000 7f56f0830700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56ec0b3c70 0x7f56ec139aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:58.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.299+0000 7f56f0830700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56ec0ad710 con 0x7f56ec0b2150 2022-06-29T16:01:58.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.299+0000 7f56f0830700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56ec13b730 con 0x7f56ec0b3c70 2022-06-29T16:01:58.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.299+0000 7f56f0830700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56ec0ad950 con 0x7f56ec0aca20 2022-06-29T16:01:58.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.299+0000 7f56eaffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56ec0b2150 0x7f56ec1395b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:58.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.299+0000 7f56eaffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56ec0b2150 0x7f56ec1395b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47990/0 (socket says 172.21.15.60:47990) 2022-06-29T16:01:58.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.299+0000 7f56eaffd700 1 -- 172.21.15.60:0/2066706435 learned_addr learned my addr 172.21.15.60:0/2066706435 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:58.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.299+0000 7f56eb7fe700 1 --2- 172.21.15.60:0/2066706435 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56ec0aca20 0x7f56ec0ad220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:58.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.299+0000 7f56ebfff700 1 --2- 172.21.15.60:0/2066706435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56ec0b3c70 0x7f56ec139aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:58.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.299+0000 7f56eb7fe700 1 -- 172.21.15.60:0/2066706435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56ec0b3c70 msgr2=0x7f56ec139aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:58.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.299+0000 7f56eb7fe700 1 --2- 172.21.15.60:0/2066706435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56ec0b3c70 0x7f56ec139aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:58.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.299+0000 7f56eb7fe700 1 -- 172.21.15.60:0/2066706435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56ec0b2150 msgr2=0x7f56ec1395b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:58.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.299+0000 7f56eb7fe700 1 --2- 172.21.15.60:0/2066706435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56ec0b2150 0x7f56ec1395b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:58.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.299+0000 7f56eb7fe700 1 -- 172.21.15.60:0/2066706435 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d400b180 con 0x7f56ec0aca20 2022-06-29T16:01:58.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.299+0000 7f56eb7fe700 1 --2- 172.21.15.60:0/2066706435 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56ec0aca20 0x7f56ec0ad220 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f56dc00a000 tx=0x7f56dc008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:58.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.299+0000 7f56e8ff9700 1 -- 172.21.15.60:0/2066706435 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f56dc013670 con 0x7f56ec0aca20 2022-06-29T16:01:58.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.299+0000 7f56e8ff9700 1 -- 172.21.15.60:0/2066706435 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56dc0137d0 con 0x7f56ec0aca20 2022-06-29T16:01:58.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.299+0000 7f56f0830700 1 -- 172.21.15.60:0/2066706435 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56ec13b9a0 con 0x7f56ec0aca20 2022-06-29T16:01:58.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.299+0000 7f56e8ff9700 1 -- 172.21.15.60:0/2066706435 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f56dc013b30 con 0x7f56ec0aca20 2022-06-29T16:01:58.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.299+0000 7f56f0830700 1 -- 172.21.15.60:0/2066706435 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56ec139f90 con 0x7f56ec0aca20 2022-06-29T16:01:58.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.303+0000 7f56f0830700 1 -- 172.21.15.60:0/2066706435 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56ec13bb30 con 0x7f56ec0aca20 2022-06-29T16:01:58.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.303+0000 7f56e8ff9700 1 -- 172.21.15.60:0/2066706435 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f56dc01a030 con 0x7f56ec0aca20 2022-06-29T16:01:58.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.303+0000 7f56e8ff9700 1 --2- 172.21.15.60:0/2066706435 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f56c4053aa0 0x7f56c4055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:58.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.303+0000 7f56e8ff9700 1 -- 172.21.15.60:0/2066706435 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f56dc06bd50 con 0x7f56ec0aca20 2022-06-29T16:01:58.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.303+0000 7f56eaffd700 1 --2- 172.21.15.60:0/2066706435 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f56c4053aa0 0x7f56c4055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:58.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.303+0000 7f56eaffd700 1 --2- 172.21.15.60:0/2066706435 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f56c4053aa0 0x7f56c4055f60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f56d401a000 tx=0x7f56d4014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:58.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.311+0000 7f56e8ff9700 1 -- 172.21.15.60:0/2066706435 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f56dc042c10 con 0x7f56ec0aca20 2022-06-29T16:01:58.435 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:01:58.437 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:01:58.440 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:01:58.443 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:01:58.445 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:01:58.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.443+0000 7f56f0830700 1 -- 172.21.15.60:0/2066706435 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f56ec0573d0 con 0x7f56ec0aca20 2022-06-29T16:01:58.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.443+0000 7f56e8ff9700 1 -- 172.21.15.60:0/2066706435 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v53) v1 ==== 110+0+10 (secure 0 0 0) 0x7f56dc046230 con 0x7f56ec0aca20 2022-06-29T16:01:58.452 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:01:58.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.447+0000 7f56f0830700 1 -- 172.21.15.60:0/2066706435 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f56c4053aa0 msgr2=0x7f56c4055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:58.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.447+0000 7f56f0830700 1 --2- 172.21.15.60:0/2066706435 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f56c4053aa0 0x7f56c4055f60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f56d401a000 tx=0x7f56d4014000).stop 2022-06-29T16:01:58.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.447+0000 7f56f0830700 1 -- 172.21.15.60:0/2066706435 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56ec0aca20 msgr2=0x7f56ec0ad220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:58.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.447+0000 7f56f0830700 1 --2- 172.21.15.60:0/2066706435 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56ec0aca20 0x7f56ec0ad220 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f56dc00a000 tx=0x7f56dc008040).stop 2022-06-29T16:01:58.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.447+0000 7f56f0830700 1 -- 172.21.15.60:0/2066706435 shutdown_connections 2022-06-29T16:01:58.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.447+0000 7f56f0830700 1 --2- 172.21.15.60:0/2066706435 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f56c4053aa0 0x7f56c4055f60 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:58.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.447+0000 7f56f0830700 1 --2- 172.21.15.60:0/2066706435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56ec0b3c70 0x7f56ec139aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:58.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.447+0000 7f56f0830700 1 --2- 172.21.15.60:0/2066706435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56ec0b2150 0x7f56ec1395b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:58.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.447+0000 7f56f0830700 1 --2- 172.21.15.60:0/2066706435 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f56ec0aca20 0x7f56ec0ad220 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:58.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.447+0000 7f56f0830700 1 -- 172.21.15.60:0/2066706435 >> 172.21.15.60:0/2066706435 conn(0x7f56ec064810 msgr2=0x7f56ec0b7400 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:58.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.447+0000 7f56f0830700 1 -- 172.21.15.60:0/2066706435 shutdown_connections 2022-06-29T16:01:58.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.447+0000 7f56f0830700 1 -- 172.21.15.60:0/2066706435 wait complete. 2022-06-29T16:01:58.465 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:01:58.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.623+0000 7f16ab59e700 1 Processor -- start 2022-06-29T16:01:58.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.623+0000 7f16ab59e700 1 -- start start 2022-06-29T16:01:58.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.623+0000 7f16ab59e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f16ac0b3110 0x7f16ac140810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:58.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.623+0000 7f16ab59e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ac0b23a0 con 0x7f16ac066d20 2022-06-29T16:01:58.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.623+0000 7f16ab59e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ac067100 con 0x7f16ac144ad0 2022-06-29T16:01:58.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.623+0000 7f16ab59e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16ac140d00 con 0x7f16ac0b3110 2022-06-29T16:01:58.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.623+0000 7f16aa59c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f16ac066d20 0x7f16ac1443c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56082/0 (socket says 172.21.15.60:56082) 2022-06-29T16:01:58.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.623+0000 7f16aa59c700 1 -- 172.21.15.60:0/467706856 learned_addr learned my addr 172.21.15.60:0/467706856 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:01:58.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.623+0000 7f16aad9d700 1 --2- 172.21.15.60:0/467706856 >> v2:172.21.15.60:3300/0 conn(0x7f16ac0b3110 0x7f16ac140810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:58.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.623+0000 7f16a959a700 1 -- 172.21.15.60:0/467706856 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1428691119 0 0) 0x7f16ac0b23a0 con 0x7f16ac066d20 2022-06-29T16:01:58.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.623+0000 7f16aad9d700 1 -- 172.21.15.60:0/467706856 >> v1:172.21.15.92:6789/0 conn(0x7f16ac144ad0 legacy=0x7f16ac13d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:58.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.623+0000 7f16aad9d700 1 -- 172.21.15.60:0/467706856 >> v1:172.21.15.60:6789/0 conn(0x7f16ac066d20 legacy=0x7f16ac1443c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:58.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.623+0000 7f16aad9d700 1 -- 172.21.15.60:0/467706856 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ac0b2100 con 0x7f16ac0b3110 2022-06-29T16:01:58.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.623+0000 7f16aad9d700 1 --2- 172.21.15.60:0/467706856 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16ac0b3110 0x7f16ac140810 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7f16a40032f0 tx=0x7f16a40091d0).ready entity=mon.0 client_cookie=d1a7a59864364f20 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:58.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.623+0000 7f16a959a700 1 -- 172.21.15.60:0/467706856 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16a4009ea0 con 0x7f16ac0b3110 2022-06-29T16:01:58.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.623+0000 7f16a959a700 1 -- 172.21.15.60:0/467706856 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a400b8a0 con 0x7f16ac0b3110 2022-06-29T16:01:58.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.623+0000 7f16a959a700 1 -- 172.21.15.60:0/467706856 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16a400ba10 con 0x7f16ac0b3110 2022-06-29T16:01:58.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.627+0000 7f16ab59e700 1 -- 172.21.15.60:0/467706856 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16ac0b3110 msgr2=0x7f16ac140810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:58.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.627+0000 7f16ab59e700 1 --2- 172.21.15.60:0/467706856 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16ac0b3110 0x7f16ac140810 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7f16a40032f0 tx=0x7f16a40091d0).stop 2022-06-29T16:01:58.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.627+0000 7f16ab59e700 1 -- 172.21.15.60:0/467706856 shutdown_connections 2022-06-29T16:01:58.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.627+0000 7f16ab59e700 1 --2- 172.21.15.60:0/467706856 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16ac0b3110 0x7f16ac140810 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:58.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.627+0000 7f16ab59e700 1 -- 172.21.15.60:0/467706856 >> 172.21.15.60:0/467706856 conn(0x7f16ac061c80 msgr2=0x7f16ac0637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:58.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.627+0000 7f16ab59e700 1 -- 172.21.15.60:0/467706856 shutdown_connections 2022-06-29T16:01:58.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.627+0000 7f16ab59e700 1 -- 172.21.15.60:0/467706856 wait complete. 2022-06-29T16:01:58.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.627+0000 7f16ab59e700 1 Processor -- start 2022-06-29T16:01:58.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.627+0000 7f16ab59e700 1 -- start start 2022-06-29T16:01:58.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.627+0000 7f16ab59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16ac066d20 0x7f16ac13c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:58.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.627+0000 7f16ab59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16ac0b3110 0x7f16ac13cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:58.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.627+0000 7f16ab59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16ac144ad0 0x7f16ac1534b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:58.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.627+0000 7f16ab59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac13e200 con 0x7f16ac144ad0 2022-06-29T16:01:58.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.627+0000 7f16ab59e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac1100c0 con 0x7f16ac066d20 2022-06-29T16:01:58.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.627+0000 7f16ab59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac110360 con 0x7f16ac0b3110 2022-06-29T16:01:58.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.627+0000 7f16aad9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16ac144ad0 0x7f16ac1534b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:58.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.627+0000 7f16aad9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16ac144ad0 0x7f16ac1534b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48006/0 (socket says 172.21.15.60:48006) 2022-06-29T16:01:58.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.627+0000 7f16aad9d700 1 -- 172.21.15.60:0/186990160 learned_addr learned my addr 172.21.15.60:0/186990160 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:58.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.627+0000 7f16aa59c700 1 --2- 172.21.15.60:0/186990160 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16ac066d20 0x7f16ac13c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:58.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.627+0000 7f16a9d9b700 1 --2- 172.21.15.60:0/186990160 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16ac0b3110 0x7f16ac13cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:58.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.631+0000 7f16aad9d700 1 -- 172.21.15.60:0/186990160 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16ac0b3110 msgr2=0x7f16ac13cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:58.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.631+0000 7f16aad9d700 1 --2- 172.21.15.60:0/186990160 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16ac0b3110 0x7f16ac13cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:58.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.631+0000 7f16aad9d700 1 -- 172.21.15.60:0/186990160 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16ac066d20 msgr2=0x7f16ac13c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:58.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.631+0000 7f16aad9d700 1 --2- 172.21.15.60:0/186990160 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16ac066d20 0x7f16ac13c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:58.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.631+0000 7f16aad9d700 1 -- 172.21.15.60:0/186990160 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a4009470 con 0x7f16ac144ad0 2022-06-29T16:01:58.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.631+0000 7f16aad9d700 1 --2- 172.21.15.60:0/186990160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16ac144ad0 0x7f16ac1534b0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7f16a40032f0 tx=0x7f16a4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:58.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.631+0000 7f169b7fe700 1 -- 172.21.15.60:0/186990160 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16a4009ea0 con 0x7f16ac144ad0 2022-06-29T16:01:58.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.631+0000 7f169b7fe700 1 -- 172.21.15.60:0/186990160 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a401f3f0 con 0x7f16ac144ad0 2022-06-29T16:01:58.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.631+0000 7f16ab59e700 1 -- 172.21.15.60:0/186990160 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16ac13d810 con 0x7f16ac144ad0 2022-06-29T16:01:58.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.631+0000 7f169b7fe700 1 -- 172.21.15.60:0/186990160 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16a401f750 con 0x7f16ac144ad0 2022-06-29T16:01:58.646 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.631+0000 7f16ab59e700 1 -- 172.21.15.60:0/186990160 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16ac13ded0 con 0x7f16ac144ad0 2022-06-29T16:01:58.647 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.631+0000 7f16ab59e700 1 -- 172.21.15.60:0/186990160 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16ac0573d0 con 0x7f16ac144ad0 2022-06-29T16:01:58.647 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.631+0000 7f169b7fe700 1 -- 172.21.15.60:0/186990160 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f16a4026030 con 0x7f16ac144ad0 2022-06-29T16:01:58.647 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.635+0000 7f169b7fe700 1 --2- 172.21.15.60:0/186990160 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1688053aa0 0x7f1688055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:58.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.635+0000 7f169b7fe700 1 -- 172.21.15.60:0/186990160 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f16a406ca70 con 0x7f16ac144ad0 2022-06-29T16:01:58.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.635+0000 7f16aa59c700 1 --2- 172.21.15.60:0/186990160 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1688053aa0 0x7f1688055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:58.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.635+0000 7f16aa59c700 1 --2- 172.21.15.60:0/186990160 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1688053aa0 0x7f1688055f60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f169c00d000 tx=0x7f169c0011a0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:58.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.643+0000 7f169b7fe700 1 -- 172.21.15.60:0/186990160 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f16a4043990 con 0x7f16ac144ad0 2022-06-29T16:01:58.778 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.771+0000 7f16ab59e700 1 -- 172.21.15.60:0/186990160 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f16ac0b1ac0 con 0x7f16ac144ad0 2022-06-29T16:01:58.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.771+0000 7f169b7fe700 1 -- 172.21.15.60:0/186990160 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v53) v1 ==== 92+0+10 (secure 0 0 0) 0x7f16a406ce60 con 0x7f16ac144ad0 2022-06-29T16:01:58.779 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:01:58.783 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.775+0000 7f16ab59e700 1 -- 172.21.15.60:0/186990160 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1688053aa0 msgr2=0x7f1688055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:58.784 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.775+0000 7f16ab59e700 1 --2- 172.21.15.60:0/186990160 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1688053aa0 0x7f1688055f60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f169c00d000 tx=0x7f169c0011a0).stop 2022-06-29T16:01:58.784 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.775+0000 7f16ab59e700 1 -- 172.21.15.60:0/186990160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16ac144ad0 msgr2=0x7f16ac1534b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:58.784 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.775+0000 7f16ab59e700 1 --2- 172.21.15.60:0/186990160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16ac144ad0 0x7f16ac1534b0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7f16a40032f0 tx=0x7f16a4016000).stop 2022-06-29T16:01:58.784 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.775+0000 7f16ab59e700 1 -- 172.21.15.60:0/186990160 shutdown_connections 2022-06-29T16:01:58.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.775+0000 7f16ab59e700 1 --2- 172.21.15.60:0/186990160 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1688053aa0 0x7f1688055f60 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:58.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.775+0000 7f16ab59e700 1 --2- 172.21.15.60:0/186990160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16ac144ad0 0x7f16ac1534b0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:58.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.775+0000 7f16ab59e700 1 --2- 172.21.15.60:0/186990160 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16ac0b3110 0x7f16ac13cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:58.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.775+0000 7f16ab59e700 1 --2- 172.21.15.60:0/186990160 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16ac066d20 0x7f16ac13c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:58.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.775+0000 7f16ab59e700 1 -- 172.21.15.60:0/186990160 >> 172.21.15.60:0/186990160 conn(0x7f16ac061c80 msgr2=0x7f16ac0b5520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:58.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.775+0000 7f16ab59e700 1 -- 172.21.15.60:0/186990160 shutdown_connections 2022-06-29T16:01:58.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.775+0000 7f16ab59e700 1 -- 172.21.15.60:0/186990160 wait complete. 2022-06-29T16:01:58.798 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-29T16:01:58.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.899+0000 7fa0773d5700 1 Processor -- start 2022-06-29T16:01:58.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.899+0000 7fa0773d5700 1 -- start start 2022-06-29T16:01:58.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.899+0000 7fa0773d5700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa07005d070 0x7fa07012b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:58.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.899+0000 7fa0773d5700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa070066b20 0x7fa07012bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:58.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.899+0000 7fa0773d5700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0700ae9a0 con 0x7fa070142ae0 2022-06-29T16:01:58.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.899+0000 7fa0773d5700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa070143b60 con 0x7fa070066b20 2022-06-29T16:01:58.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.899+0000 7fa0773d5700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa07012c0b0 con 0x7fa07005d070 2022-06-29T16:01:58.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.899+0000 7fa075bd2700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa070066b20 0x7fa07012bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:58.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.899+0000 7fa075bd2700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa070066b20 0x7fa07012bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48010/0 (socket says 172.21.15.60:48010) 2022-06-29T16:01:58.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.899+0000 7fa075bd2700 1 -- 172.21.15.60:0/514029105 learned_addr learned my addr 172.21.15.60:0/514029105 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:58.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.899+0000 7fa0763d3700 1 --2- 172.21.15.60:0/514029105 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa07005d070 0x7fa07012b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:58.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.899+0000 7fa075bd2700 1 -- 172.21.15.60:0/514029105 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa07005d070 msgr2=0x7fa07012b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:01:58.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.899+0000 7fa075bd2700 1 --2- 172.21.15.60:0/514029105 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa07005d070 0x7fa07012b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:58.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.899+0000 7fa075bd2700 1 -- 172.21.15.60:0/514029105 >> v1:172.21.15.92:6789/0 conn(0x7fa070142ae0 legacy=0x7fa070142f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:58.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.899+0000 7fa0753d1700 1 -- 172.21.15.60:0/514029105 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2762315598 0 0) 0x7fa0700ae9a0 con 0x7fa070142ae0 2022-06-29T16:01:58.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.899+0000 7fa075bd2700 1 -- 172.21.15.60:0/514029105 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa07005d450 con 0x7fa070066b20 2022-06-29T16:01:58.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.903+0000 7fa075bd2700 1 --2- 172.21.15.60:0/514029105 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa070066b20 0x7fa07012bbc0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7fa060002f70 tx=0x7fa060006eb0).ready entity=mon.0 client_cookie=a494846212af2056 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:58.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.903+0000 7fa0753d1700 1 -- 172.21.15.60:0/514029105 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa06000d570 con 0x7fa070066b20 2022-06-29T16:01:58.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.903+0000 7fa0753d1700 1 -- 172.21.15.60:0/514029105 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa060000e90 con 0x7fa070066b20 2022-06-29T16:01:58.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.903+0000 7fa0753d1700 1 -- 172.21.15.60:0/514029105 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa06000d8c0 con 0x7fa070066b20 2022-06-29T16:01:58.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.903+0000 7fa0773d5700 1 -- 172.21.15.60:0/514029105 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa070066b20 msgr2=0x7fa07012bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:58.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.903+0000 7fa0773d5700 1 --2- 172.21.15.60:0/514029105 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa070066b20 0x7fa07012bbc0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7fa060002f70 tx=0x7fa060006eb0).stop 2022-06-29T16:01:58.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.903+0000 7fa0773d5700 1 -- 172.21.15.60:0/514029105 shutdown_connections 2022-06-29T16:01:58.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.903+0000 7fa0773d5700 1 --2- 172.21.15.60:0/514029105 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa070066b20 0x7fa07012bbc0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:58.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.903+0000 7fa0773d5700 1 --2- 172.21.15.60:0/514029105 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa07005d070 0x7fa07012b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:58.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.903+0000 7fa0773d5700 1 -- 172.21.15.60:0/514029105 >> 172.21.15.60:0/514029105 conn(0x7fa0700b5890 msgr2=0x7fa0700b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:58.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.903+0000 7fa0773d5700 1 -- 172.21.15.60:0/514029105 shutdown_connections 2022-06-29T16:01:58.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.903+0000 7fa0773d5700 1 -- 172.21.15.60:0/514029105 wait complete. 2022-06-29T16:01:58.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.903+0000 7fa0773d5700 1 Processor -- start 2022-06-29T16:01:58.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.903+0000 7fa0773d5700 1 -- start start 2022-06-29T16:01:58.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.903+0000 7fa0773d5700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa07005d070 0x7fa070143f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:58.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.903+0000 7fa0773d5700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa070066b20 0x7fa070155620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:58.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.903+0000 7fa0773d5700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa070142ae0 0x7fa070151870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:58.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.903+0000 7fa0773d5700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa070130f50 con 0x7fa070142ae0 2022-06-29T16:01:58.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.903+0000 7fa0773d5700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa070154ee0 con 0x7fa070066b20 2022-06-29T16:01:58.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.903+0000 7fa0773d5700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa070155180 con 0x7fa07005d070 2022-06-29T16:01:58.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.903+0000 7fa076bd4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa070142ae0 0x7fa070151870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:58.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.903+0000 7fa076bd4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa070142ae0 0x7fa070151870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48018/0 (socket says 172.21.15.60:48018) 2022-06-29T16:01:58.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.903+0000 7fa076bd4700 1 -- 172.21.15.60:0/4128913242 learned_addr learned my addr 172.21.15.60:0/4128913242 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:58.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.903+0000 7fa0763d3700 1 --2- 172.21.15.60:0/4128913242 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa07005d070 0x7fa070143f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:58.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.903+0000 7fa075bd2700 1 --2- 172.21.15.60:0/4128913242 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa070066b20 0x7fa070155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:58.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.907+0000 7fa076bd4700 1 -- 172.21.15.60:0/4128913242 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa07005d070 msgr2=0x7fa070143f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:58.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.907+0000 7fa076bd4700 1 --2- 172.21.15.60:0/4128913242 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa07005d070 0x7fa070143f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:58.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.907+0000 7fa076bd4700 1 -- 172.21.15.60:0/4128913242 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa070066b20 msgr2=0x7fa070155620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:58.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.907+0000 7fa076bd4700 1 --2- 172.21.15.60:0/4128913242 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa070066b20 0x7fa070155620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:58.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.907+0000 7fa076bd4700 1 -- 172.21.15.60:0/4128913242 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa06000b1e0 con 0x7fa070142ae0 2022-06-29T16:01:58.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.907+0000 7fa076bd4700 1 --2- 172.21.15.60:0/4128913242 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa070142ae0 0x7fa070151870 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7fa068000f40 tx=0x7fa068007f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:58.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.907+0000 7fa05f7fe700 1 -- 172.21.15.60:0/4128913242 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa068006aa0 con 0x7fa070142ae0 2022-06-29T16:01:58.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.907+0000 7fa05f7fe700 1 -- 172.21.15.60:0/4128913242 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa068006c00 con 0x7fa070142ae0 2022-06-29T16:01:58.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.907+0000 7fa0773d5700 1 -- 172.21.15.60:0/4128913242 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa070144440 con 0x7fa070142ae0 2022-06-29T16:01:58.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.907+0000 7fa05f7fe700 1 -- 172.21.15.60:0/4128913242 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa068007030 con 0x7fa070142ae0 2022-06-29T16:01:58.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.907+0000 7fa0773d5700 1 -- 172.21.15.60:0/4128913242 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa07012ccb0 con 0x7fa070142ae0 2022-06-29T16:01:58.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.907+0000 7fa0773d5700 1 -- 172.21.15.60:0/4128913242 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa070155340 con 0x7fa070142ae0 2022-06-29T16:01:58.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.907+0000 7fa05f7fe700 1 -- 172.21.15.60:0/4128913242 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa068010030 con 0x7fa070142ae0 2022-06-29T16:01:58.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.907+0000 7fa05f7fe700 1 --2- 172.21.15.60:0/4128913242 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa050053a70 0x7fa050055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:58.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.907+0000 7fa05f7fe700 1 -- 172.21.15.60:0/4128913242 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa06806bf70 con 0x7fa070142ae0 2022-06-29T16:01:58.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.907+0000 7fa0763d3700 1 --2- 172.21.15.60:0/4128913242 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa050053a70 0x7fa050055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:58.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.911+0000 7fa0763d3700 1 --2- 172.21.15.60:0/4128913242 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa050053a70 0x7fa050055f30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fa06c00b000 tx=0x7fa06c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:58.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:58.915+0000 7fa05f7fe700 1 -- 172.21.15.60:0/4128913242 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa068042d80 con 0x7fa070142ae0 2022-06-29T16:01:59.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.043+0000 7fa0773d5700 1 -- 172.21.15.60:0/4128913242 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa070130d50 con 0x7fa070142ae0 2022-06-29T16:01:59.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.043+0000 7fa05f7fe700 1 -- 172.21.15.60:0/4128913242 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v53) v1 ==== 104+0+10 (secure 0 0 0) 0x7fa0680463a0 con 0x7fa070142ae0 2022-06-29T16:01:59.050 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:01:59.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.043+0000 7fa0773d5700 1 -- 172.21.15.60:0/4128913242 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa050053a70 msgr2=0x7fa050055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:59.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.043+0000 7fa0773d5700 1 --2- 172.21.15.60:0/4128913242 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa050053a70 0x7fa050055f30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fa06c00b000 tx=0x7fa06c008040).stop 2022-06-29T16:01:59.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.043+0000 7fa0773d5700 1 -- 172.21.15.60:0/4128913242 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa070142ae0 msgr2=0x7fa070151870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:59.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.043+0000 7fa0773d5700 1 --2- 172.21.15.60:0/4128913242 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa070142ae0 0x7fa070151870 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7fa068000f40 tx=0x7fa068007f90).stop 2022-06-29T16:01:59.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.043+0000 7fa0773d5700 1 -- 172.21.15.60:0/4128913242 shutdown_connections 2022-06-29T16:01:59.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.043+0000 7fa0773d5700 1 --2- 172.21.15.60:0/4128913242 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa050053a70 0x7fa050055f30 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:59.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.043+0000 7fa0773d5700 1 --2- 172.21.15.60:0/4128913242 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa070142ae0 0x7fa070151870 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:59.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.043+0000 7fa0773d5700 1 --2- 172.21.15.60:0/4128913242 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa070066b20 0x7fa070155620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:59.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.043+0000 7fa0773d5700 1 --2- 172.21.15.60:0/4128913242 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa07005d070 0x7fa070143f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:59.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.043+0000 7fa0773d5700 1 -- 172.21.15.60:0/4128913242 >> 172.21.15.60:0/4128913242 conn(0x7fa0700b5890 msgr2=0x7fa070128cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:59.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.043+0000 7fa0773d5700 1 -- 172.21.15.60:0/4128913242 shutdown_connections 2022-06-29T16:01:59.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.043+0000 7fa0773d5700 1 -- 172.21.15.60:0/4128913242 wait complete. 2022-06-29T16:01:59.063 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-29T16:01:59.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.167+0000 7fa19359e700 1 Processor -- start 2022-06-29T16:01:59.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.167+0000 7fa19359e700 1 -- start start 2022-06-29T16:01:59.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.167+0000 7fa19359e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa1940b3d00 0x7fa194144580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:59.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.167+0000 7fa19359e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1940b40e0 con 0x7fa19405c470 2022-06-29T16:01:59.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.167+0000 7fa19359e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1940ad840 con 0x7fa1940b1f60 2022-06-29T16:01:59.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.167+0000 7fa19359e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa19405c280 con 0x7fa1940b3d00 2022-06-29T16:01:59.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.167+0000 7fa192d9d700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa19405c470 0x7fa1940ad130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56110/0 (socket says 172.21.15.60:56110) 2022-06-29T16:01:59.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.167+0000 7fa192d9d700 1 -- 172.21.15.60:0/2722310531 learned_addr learned my addr 172.21.15.60:0/2722310531 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:01:59.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.167+0000 7fa19259c700 1 --2- 172.21.15.60:0/2722310531 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa1940b3d00 0x7fa194144580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:59.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.167+0000 7fa19159a700 1 -- 172.21.15.60:0/2722310531 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3163828400 0 0) 0x7fa1940b40e0 con 0x7fa19405c470 2022-06-29T16:01:59.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.167+0000 7fa19159a700 1 -- 172.21.15.60:0/2722310531 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa178003260 con 0x7fa19405c470 2022-06-29T16:01:59.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.167+0000 7fa19159a700 1 -- 172.21.15.60:0/2722310531 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1160368423 0 0) 0x7fa1940ad840 con 0x7fa1940b1f60 2022-06-29T16:01:59.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.167+0000 7fa19159a700 1 -- 172.21.15.60:0/2722310531 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1940b40e0 con 0x7fa1940b1f60 2022-06-29T16:01:59.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa19159a700 1 -- 172.21.15.60:0/2722310531 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1471929489 0 0) 0x7fa178003260 con 0x7fa19405c470 2022-06-29T16:01:59.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa19159a700 1 -- 172.21.15.60:0/2722310531 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa1940b3d00 msgr2=0x7fa194144580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:01:59.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa19159a700 1 --2- 172.21.15.60:0/2722310531 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa1940b3d00 0x7fa194144580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:59.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa19159a700 1 -- 172.21.15.60:0/2722310531 >> v1:172.21.15.92:6789/0 conn(0x7fa1940b1f60 legacy=0x7fa1940aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:59.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa19159a700 1 -- 172.21.15.60:0/2722310531 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa194143e40 con 0x7fa19405c470 2022-06-29T16:01:59.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa19259c700 1 --2- 172.21.15.60:0/2722310531 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa1940b3d00 0x7fa194144580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:01:59.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa19159a700 1 -- 172.21.15.60:0/2722310531 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7fa18c0019e0 con 0x7fa19405c470 2022-06-29T16:01:59.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa19159a700 1 -- 172.21.15.60:0/2722310531 >> v1:172.21.15.60:6789/0 conn(0x7fa19405c470 legacy=0x7fa1940ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:59.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa19159a700 1 --2- 172.21.15.60:0/2722310531 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa1940b3d00 0x7fa178003910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:59.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa19159a700 1 --2- 172.21.15.60:0/2722310531 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa178003e50 0x7fa178004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:59.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa19159a700 1 --2- 172.21.15.60:0/2722310531 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1780049f0 0x7fa178004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:59.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa19159a700 1 -- 172.21.15.60:0/2722310531 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17800b800 con 0x7fa1780049f0 2022-06-29T16:01:59.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa19159a700 1 -- 172.21.15.60:0/2722310531 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17800b940 con 0x7fa178003e50 2022-06-29T16:01:59.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa19159a700 1 -- 172.21.15.60:0/2722310531 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa17800bb00 con 0x7fa1940b3d00 2022-06-29T16:01:59.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa192d9d700 1 --2- 172.21.15.60:0/2722310531 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1780049f0 0x7fa178004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:59.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa19259c700 1 --2- 172.21.15.60:0/2722310531 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa1940b3d00 0x7fa178003910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:59.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa191d9b700 1 --2- 172.21.15.60:0/2722310531 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa178003e50 0x7fa178004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:59.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa192d9d700 1 -- 172.21.15.60:0/2722310531 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa1940b3d00 msgr2=0x7fa178003910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:01:59.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa192d9d700 1 --2- 172.21.15.60:0/2722310531 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa1940b3d00 0x7fa178003910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:59.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa192d9d700 1 -- 172.21.15.60:0/2722310531 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa178003e50 msgr2=0x7fa178004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:01:59.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa192d9d700 1 --2- 172.21.15.60:0/2722310531 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa178003e50 0x7fa178004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:59.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa192d9d700 1 -- 172.21.15.60:0/2722310531 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa18400b180 con 0x7fa1780049f0 2022-06-29T16:01:59.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa191d9b700 1 --2- 172.21.15.60:0/2722310531 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa178003e50 0x7fa178004500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:01:59.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa192d9d700 1 --2- 172.21.15.60:0/2722310531 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1780049f0 0x7fa178004e70 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7fa18c0017b0 tx=0x7fa1940b2a10).ready entity=mon.0 client_cookie=e0a46a2d11df7092 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:59.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa19159a700 1 -- 172.21.15.60:0/2722310531 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa18c00cb20 con 0x7fa1780049f0 2022-06-29T16:01:59.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa19159a700 1 -- 172.21.15.60:0/2722310531 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa18c00cc80 con 0x7fa1780049f0 2022-06-29T16:01:59.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa19159a700 1 -- 172.21.15.60:0/2722310531 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa18c005990 con 0x7fa1780049f0 2022-06-29T16:01:59.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa19359e700 1 -- 172.21.15.60:0/2722310531 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1780049f0 msgr2=0x7fa178004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:59.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa19359e700 1 --2- 172.21.15.60:0/2722310531 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1780049f0 0x7fa178004e70 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7fa18c0017b0 tx=0x7fa1940b2a10).stop 2022-06-29T16:01:59.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa192d9d700 1 -- 172.21.15.60:0/2722310531 reap_dead start 2022-06-29T16:01:59.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa19359e700 1 -- 172.21.15.60:0/2722310531 shutdown_connections 2022-06-29T16:01:59.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa19359e700 1 -- 172.21.15.60:0/2722310531 >> 172.21.15.60:0/2722310531 conn(0x7fa194064810 msgr2=0x7fa194064bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:59.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa19359e700 1 -- 172.21.15.60:0/2722310531 shutdown_connections 2022-06-29T16:01:59.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.171+0000 7fa19359e700 1 -- 172.21.15.60:0/2722310531 wait complete. 2022-06-29T16:01:59.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.175+0000 7fa19359e700 1 Processor -- start 2022-06-29T16:01:59.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.175+0000 7fa19359e700 1 -- start start 2022-06-29T16:01:59.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.175+0000 7fa19359e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1940af5f0 0x7fa194152db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:59.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.175+0000 7fa19359e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa194114fa0 0x7fa194115420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:59.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.175+0000 7fa19359e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa19413de00 0x7fa19413e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:59.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.175+0000 7fa19359e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1940af9d0 con 0x7fa1940af5f0 2022-06-29T16:01:59.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.175+0000 7fa19359e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa194112420 con 0x7fa194114fa0 2022-06-29T16:01:59.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.175+0000 7fa19359e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa194114b00 con 0x7fa19413de00 2022-06-29T16:01:59.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.175+0000 7fa19259c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1940af5f0 0x7fa194152db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:59.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.175+0000 7fa19259c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1940af5f0 0x7fa194152db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48034/0 (socket says 172.21.15.60:48034) 2022-06-29T16:01:59.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.175+0000 7fa19259c700 1 -- 172.21.15.60:0/1817510792 learned_addr learned my addr 172.21.15.60:0/1817510792 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:59.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.175+0000 7fa192d9d700 1 --2- 172.21.15.60:0/1817510792 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa19413de00 0x7fa19413e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:59.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.175+0000 7fa191d9b700 1 --2- 172.21.15.60:0/1817510792 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa194114fa0 0x7fa194115420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:59.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.175+0000 7fa19259c700 1 -- 172.21.15.60:0/1817510792 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa19413de00 msgr2=0x7fa19413e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:59.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.175+0000 7fa19259c700 1 --2- 172.21.15.60:0/1817510792 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa19413de00 0x7fa19413e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:59.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.175+0000 7fa19259c700 1 -- 172.21.15.60:0/1817510792 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa194114fa0 msgr2=0x7fa194115420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:59.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.175+0000 7fa19259c700 1 --2- 172.21.15.60:0/1817510792 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa194114fa0 0x7fa194115420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:59.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.175+0000 7fa19259c700 1 -- 172.21.15.60:0/1817510792 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1940b0300 con 0x7fa1940af5f0 2022-06-29T16:01:59.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.175+0000 7fa19259c700 1 --2- 172.21.15.60:0/1817510792 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1940af5f0 0x7fa194152db0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7fa18400b000 tx=0x7fa184006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:59.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.175+0000 7fa1837fe700 1 -- 172.21.15.60:0/1817510792 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa184011ed0 con 0x7fa1940af5f0 2022-06-29T16:01:59.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.175+0000 7fa1837fe700 1 -- 172.21.15.60:0/1817510792 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa18400f940 con 0x7fa1940af5f0 2022-06-29T16:01:59.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.175+0000 7fa1837fe700 1 -- 172.21.15.60:0/1817510792 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa18400fab0 con 0x7fa1940af5f0 2022-06-29T16:01:59.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.175+0000 7fa19359e700 1 -- 172.21.15.60:0/1817510792 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa194111830 con 0x7fa1940af5f0 2022-06-29T16:01:59.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.175+0000 7fa19359e700 1 -- 172.21.15.60:0/1817510792 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1941532a0 con 0x7fa1940af5f0 2022-06-29T16:01:59.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.179+0000 7fa19359e700 1 -- 172.21.15.60:0/1817510792 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1940573d0 con 0x7fa1940af5f0 2022-06-29T16:01:59.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.179+0000 7fa1837fe700 1 -- 172.21.15.60:0/1817510792 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa184024070 con 0x7fa1940af5f0 2022-06-29T16:01:59.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.179+0000 7fa1837fe700 1 --2- 172.21.15.60:0/1817510792 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa170053a70 0x7fa170055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:59.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.179+0000 7fa1837fe700 1 -- 172.21.15.60:0/1817510792 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa18406c970 con 0x7fa1940af5f0 2022-06-29T16:01:59.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.179+0000 7fa191d9b700 1 --2- 172.21.15.60:0/1817510792 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa170053a70 0x7fa170055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:59.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.179+0000 7fa191d9b700 1 --2- 172.21.15.60:0/1817510792 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa170053a70 0x7fa170055f30 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fa18800d000 tx=0x7fa18800c040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:59.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.187+0000 7fa1837fe700 1 -- 172.21.15.60:0/1817510792 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa184043780 con 0x7fa1940af5f0 2022-06-29T16:01:59.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.311+0000 7fa19359e700 1 -- 172.21.15.60:0/1817510792 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa1940b48b0 con 0x7fa1940af5f0 2022-06-29T16:01:59.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.311+0000 7fa1837fe700 1 -- 172.21.15.60:0/1817510792 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v53) v1 ==== 100+0+10 (secure 0 0 0) 0x7fa184046da0 con 0x7fa1940af5f0 2022-06-29T16:01:59.321 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:01:59.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.315+0000 7fa19359e700 1 -- 172.21.15.60:0/1817510792 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa170053a70 msgr2=0x7fa170055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:59.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.315+0000 7fa19359e700 1 --2- 172.21.15.60:0/1817510792 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa170053a70 0x7fa170055f30 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fa18800d000 tx=0x7fa18800c040).stop 2022-06-29T16:01:59.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.315+0000 7fa19359e700 1 -- 172.21.15.60:0/1817510792 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1940af5f0 msgr2=0x7fa194152db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:59.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.315+0000 7fa19359e700 1 --2- 172.21.15.60:0/1817510792 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1940af5f0 0x7fa194152db0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7fa18400b000 tx=0x7fa184006df0).stop 2022-06-29T16:01:59.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.315+0000 7fa19359e700 1 -- 172.21.15.60:0/1817510792 shutdown_connections 2022-06-29T16:01:59.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.315+0000 7fa19359e700 1 --2- 172.21.15.60:0/1817510792 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa170053a70 0x7fa170055f30 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:59.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.315+0000 7fa19359e700 1 --2- 172.21.15.60:0/1817510792 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa19413de00 0x7fa19413e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:59.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.315+0000 7fa19359e700 1 --2- 172.21.15.60:0/1817510792 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa194114fa0 0x7fa194115420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:59.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.315+0000 7fa19359e700 1 --2- 172.21.15.60:0/1817510792 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1940af5f0 0x7fa194152db0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:59.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.315+0000 7fa19359e700 1 -- 172.21.15.60:0/1817510792 >> 172.21.15.60:0/1817510792 conn(0x7fa194064810 msgr2=0x7fa194142a30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:59.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.315+0000 7fa19359e700 1 -- 172.21.15.60:0/1817510792 shutdown_connections 2022-06-29T16:01:59.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.315+0000 7fa19359e700 1 -- 172.21.15.60:0/1817510792 wait complete. 2022-06-29T16:01:59.334 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-06-29T16:01:59.334 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:01:59.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.431+0000 7f20e0fba700 1 Processor -- start 2022-06-29T16:01:59.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.431+0000 7f20e0fba700 1 -- start start 2022-06-29T16:01:59.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.435+0000 7f20e0fba700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f20dc0aed00 0x7f20dc146810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:59.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.435+0000 7f20e0fba700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20dc05d450 con 0x7f20dc0b2130 2022-06-29T16:01:59.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.435+0000 7f20e0fba700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20dc0b25d0 con 0x7f20dc05d070 2022-06-29T16:01:59.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.435+0000 7f20e0fba700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20dc0adf90 con 0x7f20dc0aed00 2022-06-29T16:01:59.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.435+0000 7f20daffd700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f20dc0b2130 0x7f20dc146100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56128/0 (socket says 172.21.15.60:56128) 2022-06-29T16:01:59.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.435+0000 7f20daffd700 1 -- 172.21.15.60:0/625636848 learned_addr learned my addr 172.21.15.60:0/625636848 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:01:59.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.435+0000 7f20dbfff700 1 --2- 172.21.15.60:0/625636848 >> v2:172.21.15.92:3300/0 conn(0x7f20dc0aed00 0x7f20dc146810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:59.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.435+0000 7f20da7fc700 1 -- 172.21.15.60:0/625636848 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1644253494 0 0) 0x7f20dc05d450 con 0x7f20dc0b2130 2022-06-29T16:01:59.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.435+0000 7f20da7fc700 1 -- 172.21.15.60:0/625636848 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20bc003260 con 0x7f20dc0b2130 2022-06-29T16:01:59.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.435+0000 7f20dbfff700 1 -- 172.21.15.60:0/625636848 >> v1:172.21.15.92:6789/0 conn(0x7f20dc05d070 legacy=0x7f20dc14aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:59.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.435+0000 7f20da7fc700 1 -- 172.21.15.60:0/625636848 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2376102522 0 0) 0x7f20bc003260 con 0x7f20dc0b2130 2022-06-29T16:01:59.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.435+0000 7f20dbfff700 1 -- 172.21.15.60:0/625636848 >> v1:172.21.15.60:6789/0 conn(0x7f20dc0b2130 legacy=0x7f20dc146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:59.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.435+0000 7f20dbfff700 1 -- 172.21.15.60:0/625636848 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20dc14a350 con 0x7f20dc0aed00 2022-06-29T16:01:59.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.435+0000 7f20dbfff700 1 --2- 172.21.15.60:0/625636848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20dc0aed00 0x7f20dc146810 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f20c4001800 tx=0x7f20d40091d0).ready entity=mon.1 client_cookie=e95a260495870c4c server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:59.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.435+0000 7f20da7fc700 1 -- 172.21.15.60:0/625636848 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f20c4001660 con 0x7f20dc0aed00 2022-06-29T16:01:59.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.435+0000 7f20da7fc700 1 -- 172.21.15.60:0/625636848 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20d400b580 con 0x7f20dc0aed00 2022-06-29T16:01:59.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.435+0000 7f20da7fc700 1 -- 172.21.15.60:0/625636848 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f20d400b6f0 con 0x7f20dc0aed00 2022-06-29T16:01:59.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.435+0000 7f20e0fba700 1 -- 172.21.15.60:0/625636848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20dc0aed00 msgr2=0x7f20dc146810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:59.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.435+0000 7f20e0fba700 1 --2- 172.21.15.60:0/625636848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20dc0aed00 0x7f20dc146810 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f20c4001800 tx=0x7f20d40091d0).stop 2022-06-29T16:01:59.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.435+0000 7f20e0fba700 1 -- 172.21.15.60:0/625636848 shutdown_connections 2022-06-29T16:01:59.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.435+0000 7f20e0fba700 1 --2- 172.21.15.60:0/625636848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20dc0aed00 0x7f20dc146810 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:59.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.435+0000 7f20e0fba700 1 -- 172.21.15.60:0/625636848 >> 172.21.15.60:0/625636848 conn(0x7f20dc0b5890 msgr2=0x7f20dc0b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:59.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.435+0000 7f20e0fba700 1 -- 172.21.15.60:0/625636848 shutdown_connections 2022-06-29T16:01:59.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.435+0000 7f20e0fba700 1 -- 172.21.15.60:0/625636848 wait complete. 2022-06-29T16:01:59.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.435+0000 7f20e0fba700 1 Processor -- start 2022-06-29T16:01:59.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.439+0000 7f20e0fba700 1 -- start start 2022-06-29T16:01:59.461 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.439+0000 7f20e0fba700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20dc05d070 0x7f20dc1191c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:59.461 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.439+0000 7f20e0fba700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20dc0aed00 0x7f20dc1189c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:59.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.439+0000 7f20e0fba700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20dc0b2130 0x7f20dc0655a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:59.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.439+0000 7f20e0fba700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20dc116c20 con 0x7f20dc0aed00 2022-06-29T16:01:59.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.439+0000 7f20e0fba700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20dc116f10 con 0x7f20dc0b2130 2022-06-29T16:01:59.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.439+0000 7f20e0fba700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20dc1171b0 con 0x7f20dc05d070 2022-06-29T16:01:59.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.439+0000 7f20daffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20dc0aed00 0x7f20dc1189c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:59.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.439+0000 7f20daffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20dc0aed00 0x7f20dc1189c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48048/0 (socket says 172.21.15.60:48048) 2022-06-29T16:01:59.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.439+0000 7f20daffd700 1 -- 172.21.15.60:0/2860659573 learned_addr learned my addr 172.21.15.60:0/2860659573 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:01:59.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.439+0000 7f20dbfff700 1 --2- 172.21.15.60:0/2860659573 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20dc0b2130 0x7f20dc0655a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:59.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.439+0000 7f20db7fe700 1 --2- 172.21.15.60:0/2860659573 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20dc05d070 0x7f20dc1191c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:59.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.439+0000 7f20daffd700 1 -- 172.21.15.60:0/2860659573 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20dc05d070 msgr2=0x7f20dc1191c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:59.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.439+0000 7f20daffd700 1 --2- 172.21.15.60:0/2860659573 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20dc05d070 0x7f20dc1191c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:59.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.439+0000 7f20daffd700 1 -- 172.21.15.60:0/2860659573 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20dc0b2130 msgr2=0x7f20dc0655a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:59.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.439+0000 7f20daffd700 1 --2- 172.21.15.60:0/2860659573 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20dc0b2130 0x7f20dc0655a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:59.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.439+0000 7f20daffd700 1 -- 172.21.15.60:0/2860659573 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d4009470 con 0x7f20dc0aed00 2022-06-29T16:01:59.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.439+0000 7f20db7fe700 1 --2- 172.21.15.60:0/2860659573 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20dc05d070 0x7f20dc1191c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:01:59.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.439+0000 7f20daffd700 1 --2- 172.21.15.60:0/2860659573 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20dc0aed00 0x7f20dc1189c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f20c40018c0 tx=0x7f20c40019e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:59.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.439+0000 7f20d8ff9700 1 -- 172.21.15.60:0/2860659573 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f20c4007170 con 0x7f20dc0aed00 2022-06-29T16:01:59.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.439+0000 7f20d8ff9700 1 -- 172.21.15.60:0/2860659573 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c40072d0 con 0x7f20dc0aed00 2022-06-29T16:01:59.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.439+0000 7f20e0fba700 1 -- 172.21.15.60:0/2860659573 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20dc064b30 con 0x7f20dc0aed00 2022-06-29T16:01:59.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.439+0000 7f20d8ff9700 1 -- 172.21.15.60:0/2860659573 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f20c4004920 con 0x7f20dc0aed00 2022-06-29T16:01:59.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.439+0000 7f20e0fba700 1 -- 172.21.15.60:0/2860659573 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20dc156f40 con 0x7f20dc0aed00 2022-06-29T16:01:59.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.439+0000 7f20e0fba700 1 -- 172.21.15.60:0/2860659573 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20dc14a5f0 con 0x7f20dc0aed00 2022-06-29T16:01:59.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.443+0000 7f20d8ff9700 1 -- 172.21.15.60:0/2860659573 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f20c401b070 con 0x7f20dc0aed00 2022-06-29T16:01:59.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.443+0000 7f20d8ff9700 1 --2- 172.21.15.60:0/2860659573 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f20c0053950 0x7f20c0055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:01:59.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.443+0000 7f20d8ff9700 1 -- 172.21.15.60:0/2860659573 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f20c406b6c0 con 0x7f20dc0aed00 2022-06-29T16:01:59.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.443+0000 7f20db7fe700 1 --2- 172.21.15.60:0/2860659573 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f20c0053950 0x7f20c0055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:01:59.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.443+0000 7f20db7fe700 1 --2- 172.21.15.60:0/2860659573 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f20c0053950 0x7f20c0055e10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f20cc00b000 tx=0x7f20cc009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:01:59.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.447+0000 7f20d8ff9700 1 -- 172.21.15.60:0/2860659573 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f20c40425e0 con 0x7f20dc0aed00 2022-06-29T16:01:59.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.607+0000 7f20e0fba700 1 -- 172.21.15.60:0/2860659573 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20dc118eb0 con 0x7f20dc0aed00 2022-06-29T16:01:59.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.607+0000 7f20d8ff9700 1 -- 172.21.15.60:0/2860659573 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+2139 (secure 0 0 0) 0x7f20c4045c00 con 0x7f20dc0aed00 2022-06-29T16:01:59.617 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:01:59.618 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":20,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5568,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/4229695218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":4229695218},{"type":"v1","addr":"172.21.15.60:6833","nonce":4229695218}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:01:54.732520+0000","modified":"2022-06-29T16:01:55.735497+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5653":{"gid":5653,"name":"b","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/1602880694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1602880694},{"type":"v1","addr":"172.21.15.92:6835","nonce":1602880694}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-06-29T16:01:59.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.611+0000 7f20e0fba700 1 -- 172.21.15.60:0/2860659573 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f20c0053950 msgr2=0x7f20c0055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:59.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.611+0000 7f20e0fba700 1 --2- 172.21.15.60:0/2860659573 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f20c0053950 0x7f20c0055e10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f20cc00b000 tx=0x7f20cc009040).stop 2022-06-29T16:01:59.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.611+0000 7f20e0fba700 1 -- 172.21.15.60:0/2860659573 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20dc0aed00 msgr2=0x7f20dc1189c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:01:59.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.611+0000 7f20e0fba700 1 --2- 172.21.15.60:0/2860659573 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20dc0aed00 0x7f20dc1189c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f20c40018c0 tx=0x7f20c40019e0).stop 2022-06-29T16:01:59.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.611+0000 7f20e0fba700 1 -- 172.21.15.60:0/2860659573 shutdown_connections 2022-06-29T16:01:59.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.611+0000 7f20e0fba700 1 --2- 172.21.15.60:0/2860659573 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f20c0053950 0x7f20c0055e10 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:59.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.611+0000 7f20e0fba700 1 --2- 172.21.15.60:0/2860659573 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20dc0b2130 0x7f20dc0655a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:59.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.611+0000 7f20e0fba700 1 --2- 172.21.15.60:0/2860659573 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20dc0aed00 0x7f20dc1189c0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:59.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.611+0000 7f20e0fba700 1 --2- 172.21.15.60:0/2860659573 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20dc05d070 0x7f20dc1191c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:01:59.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.611+0000 7f20e0fba700 1 -- 172.21.15.60:0/2860659573 >> 172.21.15.60:0/2860659573 conn(0x7f20dc0b5890 msgr2=0x7f20dc1471d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:01:59.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.611+0000 7f20e0fba700 1 -- 172.21.15.60:0/2860659573 shutdown_connections 2022-06-29T16:01:59.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:01:59.611+0000 7f20e0fba700 1 -- 172.21.15.60:0/2860659573 wait complete. 2022-06-29T16:01:59.623 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 20 2022-06-29T16:01:59.830 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:01:54.732520+0000', 'modified': '2022-06-29T16:01:55.735497+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5653}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5653': {'gid': 5653, 'name': 'b', 'rank': 0, 'incarnation': 19, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.92:6835/1602880694', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 1602880694}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 1602880694}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:01:59.831 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:01:59.832 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-06-29T16:01:59.944 INFO:teuthology.orchestra.run.smithi092.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":5653,"want_state":"up:active","state":"up:active","rank_uptime":5.1951033300000002,"mdsmap_epoch":20,"osdmap_epoch":53,"osdmap_epoch_barrier":52,"uptime":10.145598881} 2022-06-29T16:01:59.945 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 5653, "mdsmap_epoch": 20, "osdmap_epoch": 53, "osdmap_epoch_barrier": 52, "rank_uptime": 5.19510333, "state": "up:active", "uptime": 10.145598881, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:01:59.945 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-06-29T16:01:59.946 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-06-29T16:01:59.946 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi121.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-06-29T16:01:59.947 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:01:59.947 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:01:59.956 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:02:00.013 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:02:00.014 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:02:00.028 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:02:00.029 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:02:00.030 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:02:00.030 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:02:00.037 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-06-29T16:02:00.038 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:02:00.040 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:02:00.057 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:02:00.057 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:02:00.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:2022-06-29T16:02:00.107+0000 7f13b4c22080 -1 init, newargv = 0x55b51f25b060 newargc=9ceph-fuse[ 2022-06-29T16:02:00.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:12604]: starting ceph client 2022-06-29T16:02:00.117 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:02:00.118 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:02:00.119 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:02:00.119 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:02:00.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[12604]: starting fuse 2022-06-29T16:02:00.125 INFO:teuthology.orchestra.run.smithi121.stdout:52 2022-06-29T16:02:00.125 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-06-29T16:02:00.126 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:02:00.126 DEBUG:teuthology.orchestra.run.smithi121:> sudo python3 -c ' 2022-06-29T16:02:00.127 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2022-06-29T16:02:00.127 DEBUG:teuthology.orchestra.run.smithi121:> import re 2022-06-29T16:02:00.127 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-06-29T16:02:00.128 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2022-06-29T16:02:00.128 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:02:00.129 DEBUG:teuthology.orchestra.run.smithi121:> def find_socket(client_name): 2022-06-29T16:02:00.129 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-06-29T16:02:00.129 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2022-06-29T16:02:00.130 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:02:00.130 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2022-06-29T16:02:00.131 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2022-06-29T16:02:00.131 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2022-06-29T16:02:00.132 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2022-06-29T16:02:00.132 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:02:00.132 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2022-06-29T16:02:00.133 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-06-29T16:02:00.133 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2022-06-29T16:02:00.134 DEBUG:teuthology.orchestra.run.smithi121:> return f 2022-06-29T16:02:00.134 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-06-29T16:02:00.135 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:02:00.135 DEBUG:teuthology.orchestra.run.smithi121:> print(find_socket("client.0")) 2022-06-29T16:02:00.135 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-06-29T16:02:00.176 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.0.12604.asok 2022-06-29T16:02:00.180 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.12604.asok 2022-06-29T16:02:00.181 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:02:00.182 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.12604.asok status 2022-06-29T16:02:00.314 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2022-06-29T16:02:00.315 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2022-06-29T16:02:00.315 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "f25ffac72e5a9e63860ff7deb6767a11f0559a16", 2022-06-29T16:02:00.315 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 15.2.16-304-gf25ffac7 (f25ffac72e5a9e63860ff7deb6767a11f0559a16) octopus (stable)", 2022-06-29T16:02:00.316 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "0", 2022-06-29T16:02:00.316 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2022-06-29T16:02:00.316 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-06-29T16:02:00.316 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "12604", 2022-06-29T16:02:00.317 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2022-06-29T16:02:00.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:02:00.318 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2022-06-29T16:02:00.318 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2022-06-29T16:02:00.318 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 5751, 2022-06-29T16:02:00.319 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2022-06-29T16:02:00.319 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2022-06-29T16:02:00.319 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2022-06-29T16:02:00.319 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 5751 2022-06-29T16:02:00.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:02:00.320 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-06-29T16:02:00.320 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-06-29T16:02:00.321 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-06-29T16:02:00.321 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1627057151 2022-06-29T16:02:00.321 INFO:teuthology.orchestra.run.smithi121.stdout: } 2022-06-29T16:02:00.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:02:00.322 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-06-29T16:02:00.322 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-06-29T16:02:00.322 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-06-29T16:02:00.322 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1627057151 2022-06-29T16:02:00.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:02:00.323 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.5751 172.21.15.121:0/1627057151", 2022-06-29T16:02:00.323 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "172.21.15.121:0/1627057151", 2022-06-29T16:02:00.323 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2022-06-29T16:02:00.323 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 20, 2022-06-29T16:02:00.324 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 53, 2022-06-29T16:02:00.324 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2022-06-29T16:02:00.324 INFO:teuthology.orchestra.run.smithi121.stdout: "blacklisted": false, 2022-06-29T16:02:00.324 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2022-06-29T16:02:00.325 INFO:teuthology.orchestra.run.smithi121.stdout:} 2022-06-29T16:02:00.326 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:02:00.326 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:02:00.327 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:02:00.333 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T16:02:00.334 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T16:02:00.335 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:02:00.335 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:02:00.353 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:02:00.354 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:02:00.354 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:02:00.403 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T16:02:00.405 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T16:02:00.405 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-06-29T16:02:00.406 DEBUG:teuthology.orchestra.run.smithi121:> import sys 2022-06-29T16:02:00.406 DEBUG:teuthology.orchestra.run.smithi121:> import time 2022-06-29T16:02:00.407 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-06-29T16:02:00.407 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:02:00.408 DEBUG:teuthology.orchestra.run.smithi121:> n = 1000 2022-06-29T16:02:00.408 DEBUG:teuthology.orchestra.run.smithi121:> abs_path = "/home/ubuntu/cephtest/mnt.0/dc-dir/dc-file" 2022-06-29T16:02:00.408 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:02:00.409 DEBUG:teuthology.orchestra.run.smithi121:> if not os.path.exists(os.path.dirname(abs_path)): 2022-06-29T16:02:00.409 DEBUG:teuthology.orchestra.run.smithi121:> os.makedirs(os.path.dirname(abs_path)) 2022-06-29T16:02:00.410 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:02:00.410 DEBUG:teuthology.orchestra.run.smithi121:> for i in range(0, n): 2022-06-29T16:02:00.410 DEBUG:teuthology.orchestra.run.smithi121:> fname = "{0}_{1}".format(abs_path, i) 2022-06-29T16:02:00.411 DEBUG:teuthology.orchestra.run.smithi121:> with open(fname, '"'"'w'"'"') as f: 2022-06-29T16:02:00.411 DEBUG:teuthology.orchestra.run.smithi121:> f.write('"'"'content'"'"') 2022-06-29T16:02:00.412 DEBUG:teuthology.orchestra.run.smithi121:> if True: 2022-06-29T16:02:00.412 DEBUG:teuthology.orchestra.run.smithi121:> f.flush() 2022-06-29T16:02:00.413 DEBUG:teuthology.orchestra.run.smithi121:> os.fsync(f.fileno()) 2022-06-29T16:02:00.413 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-06-29T16:02:09.895 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:02:10.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:09.999+0000 7fa541dbb700 1 Processor -- start 2022-06-29T16:02:10.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:09.999+0000 7fa541dbb700 1 -- start start 2022-06-29T16:02:10.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.003+0000 7fa541dbb700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa53c0b1f60 0x7fa53c13fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:10.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.003+0000 7fa541dbb700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa53c05c470 0x7fa53c13fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:10.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.003+0000 7fa541dbb700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa53c0b40e0 con 0x7fa53c0b3d00 2022-06-29T16:02:10.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.003+0000 7fa541dbb700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa53c144c90 con 0x7fa53c0b1f60 2022-06-29T16:02:10.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.003+0000 7fa541dbb700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa53c05c280 con 0x7fa53c05c470 2022-06-29T16:02:10.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.003+0000 7fa540db9700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa53c0b3d00 0x7fa53c144580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56142/0 (socket says 172.21.15.60:56142) 2022-06-29T16:02:10.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.003+0000 7fa540db9700 1 -- 172.21.15.60:0/3019264583 learned_addr learned my addr 172.21.15.60:0/3019264583 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:02:10.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.003+0000 7fa53bfff700 1 --2- 172.21.15.60:0/3019264583 >> v2:172.21.15.60:3300/0 conn(0x7fa53c0b1f60 0x7fa53c13fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:10.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.003+0000 7fa5415ba700 1 --2- 172.21.15.60:0/3019264583 >> v2:172.21.15.92:3300/0 conn(0x7fa53c05c470 0x7fa53c13fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:10.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.003+0000 7fa53b7fe700 1 -- 172.21.15.60:0/3019264583 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1457966416 0 0) 0x7fa53c0b40e0 con 0x7fa53c0b3d00 2022-06-29T16:02:10.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.003+0000 7fa53b7fe700 1 -- 172.21.15.60:0/3019264583 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa520003260 con 0x7fa53c0b3d00 2022-06-29T16:02:10.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.003+0000 7fa53bfff700 1 -- 172.21.15.60:0/3019264583 >> v2:172.21.15.92:3300/0 conn(0x7fa53c05c470 msgr2=0x7fa53c13fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:02:10.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.003+0000 7fa53bfff700 1 --2- 172.21.15.60:0/3019264583 >> v2:172.21.15.92:3300/0 conn(0x7fa53c05c470 0x7fa53c13fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:10.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.003+0000 7fa53bfff700 1 -- 172.21.15.60:0/3019264583 >> v1:172.21.15.60:6789/0 conn(0x7fa53c0b3d00 legacy=0x7fa53c144580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:10.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.003+0000 7fa53bfff700 1 -- 172.21.15.60:0/3019264583 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa53c143e40 con 0x7fa53c0b1f60 2022-06-29T16:02:10.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.003+0000 7fa5415ba700 1 --2- 172.21.15.60:0/3019264583 >> v2:172.21.15.92:3300/0 conn(0x7fa53c05c470 0x7fa53c13fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:02:10.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.003+0000 7fa53bfff700 1 --2- 172.21.15.60:0/3019264583 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa53c0b1f60 0x7fa53c13fa00 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7fa53c0b4270 tx=0x7fa528006df0).ready entity=mon.0 client_cookie=8ab6d36ca70bfcb1 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:10.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.003+0000 7fa53b7fe700 1 -- 172.21.15.60:0/3019264583 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa52800d5c0 con 0x7fa53c0b1f60 2022-06-29T16:02:10.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.003+0000 7fa53b7fe700 1 -- 172.21.15.60:0/3019264583 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa528000e90 con 0x7fa53c0b1f60 2022-06-29T16:02:10.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.003+0000 7fa53b7fe700 1 -- 172.21.15.60:0/3019264583 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa52800d910 con 0x7fa53c0b1f60 2022-06-29T16:02:10.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.003+0000 7fa541dbb700 1 -- 172.21.15.60:0/3019264583 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa53c0b1f60 msgr2=0x7fa53c13fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:10.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.003+0000 7fa541dbb700 1 --2- 172.21.15.60:0/3019264583 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa53c0b1f60 0x7fa53c13fa00 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7fa53c0b4270 tx=0x7fa528006df0).stop 2022-06-29T16:02:10.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.003+0000 7fa541dbb700 1 -- 172.21.15.60:0/3019264583 shutdown_connections 2022-06-29T16:02:10.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.003+0000 7fa541dbb700 1 --2- 172.21.15.60:0/3019264583 >> v2:172.21.15.92:3300/0 conn(0x7fa53c05c470 0x7fa53c13fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:10.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.003+0000 7fa541dbb700 1 --2- 172.21.15.60:0/3019264583 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa53c0b1f60 0x7fa53c13fa00 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:10.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.003+0000 7fa541dbb700 1 -- 172.21.15.60:0/3019264583 >> 172.21.15.60:0/3019264583 conn(0x7fa53c064810 msgr2=0x7fa53c064bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:02:10.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.003+0000 7fa541dbb700 1 -- 172.21.15.60:0/3019264583 shutdown_connections 2022-06-29T16:02:10.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.003+0000 7fa541dbb700 1 -- 172.21.15.60:0/3019264583 wait complete. 2022-06-29T16:02:10.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.007+0000 7fa541dbb700 1 Processor -- start 2022-06-29T16:02:10.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.007+0000 7fa541dbb700 1 -- start start 2022-06-29T16:02:10.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.007+0000 7fa541dbb700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa53c05c470 0x7fa53c152d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:10.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.007+0000 7fa541dbb700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa53c0b1f60 0x7fa53c13e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:10.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.007+0000 7fa541dbb700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa53c0b3d00 0x7fa53c0aef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:10.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.007+0000 7fa541dbb700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa53c141430 con 0x7fa53c0b3d00 2022-06-29T16:02:10.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.007+0000 7fa541dbb700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa53c146740 con 0x7fa53c0b1f60 2022-06-29T16:02:10.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.007+0000 7fa541dbb700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa53c1464a0 con 0x7fa53c05c470 2022-06-29T16:02:10.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.007+0000 7fa5415ba700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa53c0b3d00 0x7fa53c0aef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:10.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.007+0000 7fa5415ba700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa53c0b3d00 0x7fa53c0aef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48066/0 (socket says 172.21.15.60:48066) 2022-06-29T16:02:10.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.007+0000 7fa5415ba700 1 -- 172.21.15.60:0/969144620 learned_addr learned my addr 172.21.15.60:0/969144620 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:02:10.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.007+0000 7fa540db9700 1 --2- 172.21.15.60:0/969144620 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa53c05c470 0x7fa53c152d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:10.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.007+0000 7fa53bfff700 1 --2- 172.21.15.60:0/969144620 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa53c0b1f60 0x7fa53c13e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:10.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.007+0000 7fa5415ba700 1 -- 172.21.15.60:0/969144620 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa53c05c470 msgr2=0x7fa53c152d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:10.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.007+0000 7fa5415ba700 1 --2- 172.21.15.60:0/969144620 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa53c05c470 0x7fa53c152d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:10.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.007+0000 7fa5415ba700 1 -- 172.21.15.60:0/969144620 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa53c0b1f60 msgr2=0x7fa53c13e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:10.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.007+0000 7fa5415ba700 1 --2- 172.21.15.60:0/969144620 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa53c0b1f60 0x7fa53c13e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:10.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.007+0000 7fa5415ba700 1 -- 172.21.15.60:0/969144620 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa52800b180 con 0x7fa53c0b3d00 2022-06-29T16:02:10.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.011+0000 7fa5415ba700 1 --2- 172.21.15.60:0/969144620 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa53c0b3d00 0x7fa53c0aef60 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7fa534006cc0 tx=0x7fa534008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:10.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.011+0000 7fa539ffb700 1 -- 172.21.15.60:0/969144620 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa534013620 con 0x7fa53c0b3d00 2022-06-29T16:02:10.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.011+0000 7fa539ffb700 1 -- 172.21.15.60:0/969144620 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa534013780 con 0x7fa53c0b3d00 2022-06-29T16:02:10.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.011+0000 7fa541dbb700 1 -- 172.21.15.60:0/969144620 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa53c153250 con 0x7fa53c0b3d00 2022-06-29T16:02:10.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.011+0000 7fa539ffb700 1 -- 172.21.15.60:0/969144620 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa534013ae0 con 0x7fa53c0b3d00 2022-06-29T16:02:10.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.011+0000 7fa541dbb700 1 -- 172.21.15.60:0/969144620 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa53c146070 con 0x7fa53c0b3d00 2022-06-29T16:02:10.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.011+0000 7fa539ffb700 1 -- 172.21.15.60:0/969144620 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa534013c40 con 0x7fa53c0b3d00 2022-06-29T16:02:10.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.011+0000 7fa541dbb700 1 -- 172.21.15.60:0/969144620 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa53c146290 con 0x7fa53c0b3d00 2022-06-29T16:02:10.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.011+0000 7fa539ffb700 1 --2- 172.21.15.60:0/969144620 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa51c0539e0 0x7fa51c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:10.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.011+0000 7fa539ffb700 1 -- 172.21.15.60:0/969144620 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa53406bcf0 con 0x7fa53c0b3d00 2022-06-29T16:02:10.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.011+0000 7fa540db9700 1 --2- 172.21.15.60:0/969144620 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa51c0539e0 0x7fa51c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:10.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.011+0000 7fa540db9700 1 --2- 172.21.15.60:0/969144620 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa51c0539e0 0x7fa51c055ea0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fa53000d000 tx=0x7fa530009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:10.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.015+0000 7fa539ffb700 1 -- 172.21.15.60:0/969144620 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa534042c10 con 0x7fa53c0b3d00 2022-06-29T16:02:10.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.179+0000 7fa541dbb700 1 -- 172.21.15.60:0/969144620 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa53c0573d0 con 0x7fa53c0b3d00 2022-06-29T16:02:10.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.179+0000 7fa539ffb700 1 -- 172.21.15.60:0/969144620 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+2139 (secure 0 0 0) 0x7fa53400f040 con 0x7fa53c0b3d00 2022-06-29T16:02:10.187 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:02:10.188 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":20,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5568,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/4229695218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":4229695218},{"type":"v1","addr":"172.21.15.60:6833","nonce":4229695218}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:01:54.732520+0000","modified":"2022-06-29T16:01:55.735497+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5653":{"gid":5653,"name":"b","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/1602880694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1602880694},{"type":"v1","addr":"172.21.15.92:6835","nonce":1602880694}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-06-29T16:02:10.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.183+0000 7fa541dbb700 1 -- 172.21.15.60:0/969144620 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa51c0539e0 msgr2=0x7fa51c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:10.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.183+0000 7fa541dbb700 1 --2- 172.21.15.60:0/969144620 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa51c0539e0 0x7fa51c055ea0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fa53000d000 tx=0x7fa530009040).stop 2022-06-29T16:02:10.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.183+0000 7fa541dbb700 1 -- 172.21.15.60:0/969144620 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa53c0b3d00 msgr2=0x7fa53c0aef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:10.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.183+0000 7fa541dbb700 1 --2- 172.21.15.60:0/969144620 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa53c0b3d00 0x7fa53c0aef60 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7fa534006cc0 tx=0x7fa534008040).stop 2022-06-29T16:02:10.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.183+0000 7fa541dbb700 1 -- 172.21.15.60:0/969144620 shutdown_connections 2022-06-29T16:02:10.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.183+0000 7fa541dbb700 1 --2- 172.21.15.60:0/969144620 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa51c0539e0 0x7fa51c055ea0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:10.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.183+0000 7fa541dbb700 1 --2- 172.21.15.60:0/969144620 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa53c0b3d00 0x7fa53c0aef60 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:10.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.183+0000 7fa541dbb700 1 --2- 172.21.15.60:0/969144620 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa53c0b1f60 0x7fa53c13e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:10.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.183+0000 7fa541dbb700 1 --2- 172.21.15.60:0/969144620 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa53c05c470 0x7fa53c152d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:10.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.183+0000 7fa541dbb700 1 -- 172.21.15.60:0/969144620 >> 172.21.15.60:0/969144620 conn(0x7fa53c064810 msgr2=0x7fa53c0b6f60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:02:10.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.183+0000 7fa541dbb700 1 -- 172.21.15.60:0/969144620 shutdown_connections 2022-06-29T16:02:10.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.183+0000 7fa541dbb700 1 -- 172.21.15.60:0/969144620 wait complete. 2022-06-29T16:02:10.192 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 20 2022-06-29T16:02:10.201 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json config set mds_recall_max_caps 20 2022-06-29T16:02:10.311 INFO:teuthology.orchestra.run.smithi092.stdout:{"success":"mds_recall_max_caps = '20' (not observed, change may require restart) "} 2022-06-29T16:02:10.312 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_caps = '20' (not observed, change may require restart) " } 2022-06-29T16:02:10.312 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:02:10.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.415+0000 7fab4d635700 1 Processor -- start 2022-06-29T16:02:10.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.415+0000 7fab4d635700 1 -- start start 2022-06-29T16:02:10.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.415+0000 7fab4d635700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fab480b5eb0 0x7fab480b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:10.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.415+0000 7fab4d635700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab4805c730 con 0x7fab480b4340 2022-06-29T16:02:10.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.415+0000 7fab4d635700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab480b6290 con 0x7fab480b4960 2022-06-29T16:02:10.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.415+0000 7fab4d635700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab480b4780 con 0x7fab480b5eb0 2022-06-29T16:02:10.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.415+0000 7fab47fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fab480b5eb0 0x7fab480b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:10.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.415+0000 7fab47fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fab480b5eb0 0x7fab480b50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48070/0 (socket says 172.21.15.60:48070) 2022-06-29T16:02:10.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.415+0000 7fab47fff700 1 -- 172.21.15.60:0/3095811566 learned_addr learned my addr 172.21.15.60:0/3095811566 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:02:10.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.415+0000 7fab47fff700 1 -- 172.21.15.60:0/3095811566 >> v1:172.21.15.92:6789/0 conn(0x7fab480b4960 legacy=0x7fab480b0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:02:10.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.415+0000 7fab47fff700 1 -- 172.21.15.60:0/3095811566 >> v1:172.21.15.60:6789/0 conn(0x7fab480b4340 legacy=0x7fab48146250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:10.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.415+0000 7fab47fff700 1 -- 172.21.15.60:0/3095811566 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4805c960 con 0x7fab480b5eb0 2022-06-29T16:02:10.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.415+0000 7fab47fff700 1 --2- 172.21.15.60:0/3095811566 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab480b5eb0 0x7fab480b50a0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7fab48062fa0 tx=0x7fab38006eb0).ready entity=mon.0 client_cookie=604c8d950295d718 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:10.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.415+0000 7fab477fe700 1 -- 172.21.15.60:0/3095811566 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab3800d610 con 0x7fab480b5eb0 2022-06-29T16:02:10.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.415+0000 7fab477fe700 1 -- 172.21.15.60:0/3095811566 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab38000e90 con 0x7fab480b5eb0 2022-06-29T16:02:10.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.415+0000 7fab477fe700 1 -- 172.21.15.60:0/3095811566 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab3800d960 con 0x7fab480b5eb0 2022-06-29T16:02:10.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.419+0000 7fab4d635700 1 -- 172.21.15.60:0/3095811566 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab480b5eb0 msgr2=0x7fab480b50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:10.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.419+0000 7fab4d635700 1 --2- 172.21.15.60:0/3095811566 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab480b5eb0 0x7fab480b50a0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7fab48062fa0 tx=0x7fab38006eb0).stop 2022-06-29T16:02:10.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.419+0000 7fab4d635700 1 -- 172.21.15.60:0/3095811566 shutdown_connections 2022-06-29T16:02:10.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.419+0000 7fab4d635700 1 --2- 172.21.15.60:0/3095811566 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab480b5eb0 0x7fab480b50a0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:10.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.419+0000 7fab4d635700 1 -- 172.21.15.60:0/3095811566 >> 172.21.15.60:0/3095811566 conn(0x7fab48062090 msgr2=0x7fab48062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:02:10.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.419+0000 7fab4d635700 1 -- 172.21.15.60:0/3095811566 shutdown_connections 2022-06-29T16:02:10.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.419+0000 7fab4d635700 1 -- 172.21.15.60:0/3095811566 wait complete. 2022-06-29T16:02:10.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.419+0000 7fab4d635700 1 Processor -- start 2022-06-29T16:02:10.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.419+0000 7fab4d635700 1 -- start start 2022-06-29T16:02:10.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.419+0000 7fab4d635700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fab480b4340 0x7fab480b72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:10.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.419+0000 7fab4d635700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab480b4960 0x7fab480aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:10.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.419+0000 7fab4d635700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab480b5eb0 0x7fab480aee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:10.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.419+0000 7fab4d635700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab480ae740 con 0x7fab480b5eb0 2022-06-29T16:02:10.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.419+0000 7fab4d635700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab48156600 con 0x7fab480b4960 2022-06-29T16:02:10.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.419+0000 7fab4d635700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab480ae9a0 con 0x7fab480b4340 2022-06-29T16:02:10.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.419+0000 7fab4ce34700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab480b5eb0 0x7fab480aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:10.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.419+0000 7fab47fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fab480b4340 0x7fab480b72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:10.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.419+0000 7fab4ce34700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab480b5eb0 0x7fab480aee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48080/0 (socket says 172.21.15.60:48080) 2022-06-29T16:02:10.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.419+0000 7fab4ce34700 1 -- 172.21.15.60:0/3158694689 learned_addr learned my addr 172.21.15.60:0/3158694689 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:02:10.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.419+0000 7fab3f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab480b4960 0x7fab480aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:10.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.419+0000 7fab4ce34700 1 -- 172.21.15.60:0/3158694689 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fab480b4340 msgr2=0x7fab480b72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:10.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.419+0000 7fab4ce34700 1 --2- 172.21.15.60:0/3158694689 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fab480b4340 0x7fab480b72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:10.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.419+0000 7fab4ce34700 1 -- 172.21.15.60:0/3158694689 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab480b4960 msgr2=0x7fab480aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:10.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.419+0000 7fab4ce34700 1 --2- 172.21.15.60:0/3158694689 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab480b4960 0x7fab480aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:10.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.419+0000 7fab4ce34700 1 -- 172.21.15.60:0/3158694689 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab3800b1e0 con 0x7fab480b5eb0 2022-06-29T16:02:10.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.419+0000 7fab4ce34700 1 --2- 172.21.15.60:0/3158694689 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab480b5eb0 0x7fab480aee40 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7fab48062f00 tx=0x7fab40007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:10.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.419+0000 7fab45ffb700 1 -- 172.21.15.60:0/3158694689 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab4000f470 con 0x7fab480b5eb0 2022-06-29T16:02:10.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.419+0000 7fab45ffb700 1 -- 172.21.15.60:0/3158694689 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab4000f5d0 con 0x7fab480b5eb0 2022-06-29T16:02:10.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.419+0000 7fab4d635700 1 -- 172.21.15.60:0/3158694689 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab48156330 con 0x7fab480b5eb0 2022-06-29T16:02:10.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.419+0000 7fab45ffb700 1 -- 172.21.15.60:0/3158694689 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab4000f930 con 0x7fab480b5eb0 2022-06-29T16:02:10.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.419+0000 7fab4d635700 1 -- 172.21.15.60:0/3158694689 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab480b2540 con 0x7fab480b5eb0 2022-06-29T16:02:10.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.423+0000 7fab4d635700 1 -- 172.21.15.60:0/3158694689 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab4805cc00 con 0x7fab480b5eb0 2022-06-29T16:02:10.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.423+0000 7fab45ffb700 1 -- 172.21.15.60:0/3158694689 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fab40019030 con 0x7fab480b5eb0 2022-06-29T16:02:10.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.423+0000 7fab45ffb700 1 --2- 172.21.15.60:0/3158694689 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fab2c053a70 0x7fab2c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:10.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.423+0000 7fab45ffb700 1 -- 172.21.15.60:0/3158694689 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fab4006b530 con 0x7fab480b5eb0 2022-06-29T16:02:10.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.423+0000 7fab47fff700 1 --2- 172.21.15.60:0/3158694689 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fab2c053a70 0x7fab2c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:10.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.423+0000 7fab47fff700 1 --2- 172.21.15.60:0/3158694689 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fab2c053a70 0x7fab2c055f30 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7fab3801c000 tx=0x7fab38015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:10.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.427+0000 7fab45ffb700 1 -- 172.21.15.60:0/3158694689 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fab400423f0 con 0x7fab480b5eb0 2022-06-29T16:02:10.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.595+0000 7fab4d635700 1 -- 172.21.15.60:0/3158694689 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab480aeb60 con 0x7fab480b5eb0 2022-06-29T16:02:10.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.595+0000 7fab45ffb700 1 -- 172.21.15.60:0/3158694689 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+2139 (secure 0 0 0) 0x7fab40045a10 con 0x7fab480b5eb0 2022-06-29T16:02:10.603 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:02:10.603 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":20,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5568,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/4229695218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":4229695218},{"type":"v1","addr":"172.21.15.60:6833","nonce":4229695218}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:01:54.732520+0000","modified":"2022-06-29T16:01:55.735497+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5653":{"gid":5653,"name":"b","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/1602880694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1602880694},{"type":"v1","addr":"172.21.15.92:6835","nonce":1602880694}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-06-29T16:02:10.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.599+0000 7fab4d635700 1 -- 172.21.15.60:0/3158694689 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fab2c053a70 msgr2=0x7fab2c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:10.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.599+0000 7fab4d635700 1 --2- 172.21.15.60:0/3158694689 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fab2c053a70 0x7fab2c055f30 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7fab3801c000 tx=0x7fab38015000).stop 2022-06-29T16:02:10.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.599+0000 7fab4d635700 1 -- 172.21.15.60:0/3158694689 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab480b5eb0 msgr2=0x7fab480aee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:10.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.599+0000 7fab4d635700 1 --2- 172.21.15.60:0/3158694689 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab480b5eb0 0x7fab480aee40 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7fab48062f00 tx=0x7fab40007f40).stop 2022-06-29T16:02:10.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.599+0000 7fab4d635700 1 -- 172.21.15.60:0/3158694689 shutdown_connections 2022-06-29T16:02:10.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.599+0000 7fab4d635700 1 --2- 172.21.15.60:0/3158694689 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fab2c053a70 0x7fab2c055f30 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:10.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.599+0000 7fab4d635700 1 --2- 172.21.15.60:0/3158694689 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fab480b5eb0 0x7fab480aee40 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:10.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.599+0000 7fab4d635700 1 --2- 172.21.15.60:0/3158694689 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab480b4960 0x7fab480aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:10.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.599+0000 7fab4d635700 1 --2- 172.21.15.60:0/3158694689 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fab480b4340 0x7fab480b72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:10.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.599+0000 7fab4d635700 1 -- 172.21.15.60:0/3158694689 >> 172.21.15.60:0/3158694689 conn(0x7fab48062090 msgr2=0x7fab48149410 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:02:10.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.599+0000 7fab4d635700 1 -- 172.21.15.60:0/3158694689 shutdown_connections 2022-06-29T16:02:10.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.599+0000 7fab4d635700 1 -- 172.21.15.60:0/3158694689 wait complete. 2022-06-29T16:02:10.609 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 20 2022-06-29T16:02:10.621 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json config set mds_recall_max_decay_threshold 400 2022-06-29T16:02:10.732 INFO:teuthology.orchestra.run.smithi092.stdout:{"success":"mds_recall_max_decay_threshold = '400' (not observed, change may require restart) "} 2022-06-29T16:02:10.732 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_decay_threshold = '400' (not observed, change may require restart) " } 2022-06-29T16:02:10.733 INFO:tasks.cephfs.fuse_mount:Killing ceph-fuse connection on ubuntu@smithi121.front.sepia.ceph.com... 2022-06-29T16:02:10.751 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:02:10.860 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.851+0000 7fb58e2ec700 1 Processor -- start 2022-06-29T16:02:10.861 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.851+0000 7fb58e2ec700 1 -- start start 2022-06-29T16:02:10.863 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.855+0000 7fb58e2ec700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb5880b6a50 0x7fb5881461e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:10.863 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.855+0000 7fb58e2ec700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb58805ca50 0x7fb58805ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:10.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.855+0000 7fb58e2ec700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5880b6e30 con 0x7fb58805f0b0 2022-06-29T16:02:10.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.855+0000 7fb58e2ec700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5880b0dd0 con 0x7fb58805ca50 2022-06-29T16:02:10.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.855+0000 7fb58e2ec700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5880b1070 con 0x7fb5880b6a50 2022-06-29T16:02:10.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.855+0000 7fb58c889700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb58805ca50 0x7fb58805ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:10.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.855+0000 7fb57f7fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb58805f0b0 0x7fb588142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56170/0 (socket says 172.21.15.60:56170) 2022-06-29T16:02:10.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.855+0000 7fb58c889700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb58805ca50 0x7fb58805ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48088/0 (socket says 172.21.15.60:48088) 2022-06-29T16:02:10.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.855+0000 7fb57f7fe700 1 -- 172.21.15.60:0/1928547940 learned_addr learned my addr 172.21.15.60:0/1928547940 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:02:10.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.855+0000 7fb587fff700 1 --2- 172.21.15.60:0/1928547940 >> v2:172.21.15.92:3300/0 conn(0x7fb5880b6a50 0x7fb5881461e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:10.866 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.855+0000 7fb5877fe700 1 -- 172.21.15.60:0/1928547940 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4012531838 0 0) 0x7fb5880b6e30 con 0x7fb58805f0b0 2022-06-29T16:02:10.866 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.855+0000 7fb5877fe700 1 -- 172.21.15.60:0/1928547940 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb568003260 con 0x7fb58805f0b0 2022-06-29T16:02:10.866 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.855+0000 7fb58c889700 1 -- 172.21.15.60:0/1928547940 >> v2:172.21.15.92:3300/0 conn(0x7fb5880b6a50 msgr2=0x7fb5881461e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:02:10.866 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.855+0000 7fb58c889700 1 --2- 172.21.15.60:0/1928547940 >> v2:172.21.15.92:3300/0 conn(0x7fb5880b6a50 0x7fb5881461e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:10.866 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.855+0000 7fb58c889700 1 -- 172.21.15.60:0/1928547940 >> v1:172.21.15.60:6789/0 conn(0x7fb58805f0b0 legacy=0x7fb588142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:10.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.855+0000 7fb58c889700 1 -- 172.21.15.60:0/1928547940 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb58805c5b0 con 0x7fb58805ca50 2022-06-29T16:02:10.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.855+0000 7fb58c889700 1 --2- 172.21.15.60:0/1928547940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb58805ca50 0x7fb58805ce50 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7fb5800032f0 tx=0x7fb5800091d0).ready entity=mon.0 client_cookie=81ef24f66f0ce1ee server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:10.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.855+0000 7fb5877fe700 1 -- 172.21.15.60:0/1928547940 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb58000b600 con 0x7fb58805ca50 2022-06-29T16:02:10.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.855+0000 7fb5877fe700 1 -- 172.21.15.60:0/1928547940 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb58000b760 con 0x7fb58805ca50 2022-06-29T16:02:10.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.855+0000 7fb5877fe700 1 -- 172.21.15.60:0/1928547940 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb58000b8d0 con 0x7fb58805ca50 2022-06-29T16:02:10.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.855+0000 7fb58e2ec700 1 -- 172.21.15.60:0/1928547940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb58805ca50 msgr2=0x7fb58805ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:10.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.855+0000 7fb58e2ec700 1 --2- 172.21.15.60:0/1928547940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb58805ca50 0x7fb58805ce50 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7fb5800032f0 tx=0x7fb5800091d0).stop 2022-06-29T16:02:10.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.855+0000 7fb58e2ec700 1 -- 172.21.15.60:0/1928547940 shutdown_connections 2022-06-29T16:02:10.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.855+0000 7fb58e2ec700 1 --2- 172.21.15.60:0/1928547940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb58805ca50 0x7fb58805ce50 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:10.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.855+0000 7fb58e2ec700 1 --2- 172.21.15.60:0/1928547940 >> v2:172.21.15.92:3300/0 conn(0x7fb5880b6a50 0x7fb5881461e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:10.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.855+0000 7fb58e2ec700 1 -- 172.21.15.60:0/1928547940 >> 172.21.15.60:0/1928547940 conn(0x7fb5880643e0 msgr2=0x7fb5880b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:02:10.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.859+0000 7fb58e2ec700 1 -- 172.21.15.60:0/1928547940 shutdown_connections 2022-06-29T16:02:10.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.859+0000 7fb58e2ec700 1 -- 172.21.15.60:0/1928547940 wait complete. 2022-06-29T16:02:10.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.859+0000 7fb58e2ec700 1 Processor -- start 2022-06-29T16:02:10.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.859+0000 7fb58e2ec700 1 -- start start 2022-06-29T16:02:10.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.863+0000 7fb58e2ec700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb58805ca50 0x7fb5880af4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:10.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.863+0000 7fb58e2ec700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb58805f0b0 0x7fb588064960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:10.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.863+0000 7fb58e2ec700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5880b6a50 0x7fb588140fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:10.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.863+0000 7fb58e2ec700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5880af9d0 con 0x7fb5880b6a50 2022-06-29T16:02:10.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.863+0000 7fb58e2ec700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5880afc40 con 0x7fb58805f0b0 2022-06-29T16:02:10.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.863+0000 7fb58e2ec700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb588155e90 con 0x7fb58805ca50 2022-06-29T16:02:10.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.863+0000 7fb58c889700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5880b6a50 0x7fb588140fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:10.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.863+0000 7fb58c889700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5880b6a50 0x7fb588140fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48094/0 (socket says 172.21.15.60:48094) 2022-06-29T16:02:10.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.863+0000 7fb58c889700 1 -- 172.21.15.60:0/4281638077 learned_addr learned my addr 172.21.15.60:0/4281638077 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:02:10.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.863+0000 7fb57f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb58805f0b0 0x7fb588064960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:10.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.863+0000 7fb587fff700 1 --2- 172.21.15.60:0/4281638077 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb58805ca50 0x7fb5880af4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:10.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.863+0000 7fb58c889700 1 -- 172.21.15.60:0/4281638077 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb58805ca50 msgr2=0x7fb5880af4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:10.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.863+0000 7fb58c889700 1 --2- 172.21.15.60:0/4281638077 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb58805ca50 0x7fb5880af4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:10.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.863+0000 7fb58c889700 1 -- 172.21.15.60:0/4281638077 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb58805f0b0 msgr2=0x7fb588064960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:10.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.863+0000 7fb58c889700 1 --2- 172.21.15.60:0/4281638077 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb58805f0b0 0x7fb588064960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:10.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.863+0000 7fb58c889700 1 -- 172.21.15.60:0/4281638077 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb580009470 con 0x7fb5880b6a50 2022-06-29T16:02:10.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.863+0000 7fb58c889700 1 --2- 172.21.15.60:0/4281638077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5880b6a50 0x7fb588140fc0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7fb5800032f0 tx=0x7fb580016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:10.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.863+0000 7fb585ffb700 1 -- 172.21.15.60:0/4281638077 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb580011b60 con 0x7fb5880b6a50 2022-06-29T16:02:10.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.863+0000 7fb585ffb700 1 -- 172.21.15.60:0/4281638077 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb580011cc0 con 0x7fb5880b6a50 2022-06-29T16:02:10.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.863+0000 7fb585ffb700 1 -- 172.21.15.60:0/4281638077 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb58001f5e0 con 0x7fb5880b6a50 2022-06-29T16:02:10.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.863+0000 7fb58e2ec700 1 -- 172.21.15.60:0/4281638077 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb588064e50 con 0x7fb5880b6a50 2022-06-29T16:02:10.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.863+0000 7fb58e2ec700 1 -- 172.21.15.60:0/4281638077 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5881414b0 con 0x7fb5880b6a50 2022-06-29T16:02:10.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.867+0000 7fb58e2ec700 1 -- 172.21.15.60:0/4281638077 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb588156050 con 0x7fb5880b6a50 2022-06-29T16:02:10.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.867+0000 7fb585ffb700 1 -- 172.21.15.60:0/4281638077 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb580026030 con 0x7fb5880b6a50 2022-06-29T16:02:10.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.867+0000 7fb585ffb700 1 --2- 172.21.15.60:0/4281638077 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb56c053b30 0x7fb56c055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:10.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.867+0000 7fb585ffb700 1 -- 172.21.15.60:0/4281638077 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb58006c7b0 con 0x7fb5880b6a50 2022-06-29T16:02:10.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.867+0000 7fb587fff700 1 --2- 172.21.15.60:0/4281638077 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb56c053b30 0x7fb56c055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:10.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.867+0000 7fb587fff700 1 --2- 172.21.15.60:0/4281638077 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb56c053b30 0x7fb56c055ff0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fb57800b000 tx=0x7fb578006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:10.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:10.875+0000 7fb585ffb700 1 -- 172.21.15.60:0/4281638077 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb5800436d0 con 0x7fb5880b6a50 2022-06-29T16:02:11.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.043+0000 7fb58e2ec700 1 -- 172.21.15.60:0/4281638077 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5880573d0 con 0x7fb5880b6a50 2022-06-29T16:02:11.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.043+0000 7fb585ffb700 1 -- 172.21.15.60:0/4281638077 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+2139 (secure 0 0 0) 0x7fb580024870 con 0x7fb5880b6a50 2022-06-29T16:02:11.052 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:02:11.053 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":20,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5568,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/4229695218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":4229695218},{"type":"v1","addr":"172.21.15.60:6833","nonce":4229695218}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:01:54.732520+0000","modified":"2022-06-29T16:01:55.735497+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5653":{"gid":5653,"name":"b","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/1602880694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1602880694},{"type":"v1","addr":"172.21.15.92:6835","nonce":1602880694}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-06-29T16:02:11.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.047+0000 7fb58e2ec700 1 -- 172.21.15.60:0/4281638077 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb56c053b30 msgr2=0x7fb56c055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:11.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.047+0000 7fb58e2ec700 1 --2- 172.21.15.60:0/4281638077 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb56c053b30 0x7fb56c055ff0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fb57800b000 tx=0x7fb578006df0).stop 2022-06-29T16:02:11.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.047+0000 7fb58e2ec700 1 -- 172.21.15.60:0/4281638077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5880b6a50 msgr2=0x7fb588140fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:11.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.047+0000 7fb58e2ec700 1 --2- 172.21.15.60:0/4281638077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5880b6a50 0x7fb588140fc0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7fb5800032f0 tx=0x7fb580016000).stop 2022-06-29T16:02:11.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.047+0000 7fb58e2ec700 1 -- 172.21.15.60:0/4281638077 shutdown_connections 2022-06-29T16:02:11.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.047+0000 7fb58e2ec700 1 --2- 172.21.15.60:0/4281638077 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb56c053b30 0x7fb56c055ff0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:11.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.047+0000 7fb58e2ec700 1 --2- 172.21.15.60:0/4281638077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5880b6a50 0x7fb588140fc0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:11.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.047+0000 7fb58e2ec700 1 --2- 172.21.15.60:0/4281638077 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb58805f0b0 0x7fb588064960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:11.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.047+0000 7fb58e2ec700 1 --2- 172.21.15.60:0/4281638077 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb58805ca50 0x7fb5880af4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:11.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.047+0000 7fb58e2ec700 1 -- 172.21.15.60:0/4281638077 >> 172.21.15.60:0/4281638077 conn(0x7fb5880643e0 msgr2=0x7fb58805ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:02:11.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.047+0000 7fb58e2ec700 1 -- 172.21.15.60:0/4281638077 shutdown_connections 2022-06-29T16:02:11.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.047+0000 7fb58e2ec700 1 -- 172.21.15.60:0/4281638077 wait complete. 2022-06-29T16:02:11.057 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 20 2022-06-29T16:02:11.066 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b cache drop 5 2022-06-29T16:02:11.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.163+0000 7f16c21a3700 1 Processor -- start 2022-06-29T16:02:11.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.163+0000 7f16c21a3700 1 -- start start 2022-06-29T16:02:11.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.167+0000 7f16c21a3700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f16bc0b2120 0x7f16bc0ab880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:11.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.167+0000 7f16c21a3700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f16bc05cf40 0x7f16bc14af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:11.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.167+0000 7f16c21a3700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16bc14a530 con 0x7f16bc05bd20 2022-06-29T16:02:11.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.167+0000 7f16c21a3700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16bc0b1720 con 0x7f16bc0b2120 2022-06-29T16:02:11.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.167+0000 7f16c21a3700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16bc14a7d0 con 0x7f16bc05cf40 2022-06-29T16:02:11.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.167+0000 7f16c11a1700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f16bc0b2120 0x7f16bc0ab880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:11.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.167+0000 7f16c11a1700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f16bc0b2120 0x7f16bc0ab880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48098/0 (socket says 172.21.15.60:48098) 2022-06-29T16:02:11.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.167+0000 7f16c11a1700 1 -- 172.21.15.60:0/578035690 learned_addr learned my addr 172.21.15.60:0/578035690 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:02:11.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.167+0000 7f16c09a0700 1 --2- 172.21.15.60:0/578035690 >> v2:172.21.15.92:3300/0 conn(0x7f16bc05cf40 0x7f16bc14af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:11.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.167+0000 7f16c11a1700 1 -- 172.21.15.60:0/578035690 >> v2:172.21.15.92:3300/0 conn(0x7f16bc05cf40 msgr2=0x7f16bc14af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:02:11.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.167+0000 7f16c11a1700 1 --2- 172.21.15.60:0/578035690 >> v2:172.21.15.92:3300/0 conn(0x7f16bc05cf40 0x7f16bc14af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:11.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.167+0000 7f16c11a1700 1 -- 172.21.15.60:0/578035690 >> v1:172.21.15.92:6789/0 conn(0x7f16bc05bd20 legacy=0x7f16bc05c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:11.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.167+0000 7f16b3fff700 1 -- 172.21.15.60:0/578035690 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1767142616 0 0) 0x7f16bc14a530 con 0x7f16bc05bd20 2022-06-29T16:02:11.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.167+0000 7f16c11a1700 1 -- 172.21.15.60:0/578035690 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16bc14aa70 con 0x7f16bc0b2120 2022-06-29T16:02:11.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.167+0000 7f16c11a1700 1 --2- 172.21.15.60:0/578035690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16bc0b2120 0x7f16bc0ab880 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7f16ac002f70 tx=0x7f16ac006eb0).ready entity=mon.0 client_cookie=f24809d465de5f95 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:11.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.167+0000 7f16b3fff700 1 -- 172.21.15.60:0/578035690 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16ac00d5c0 con 0x7f16bc0b2120 2022-06-29T16:02:11.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.167+0000 7f16b3fff700 1 -- 172.21.15.60:0/578035690 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16ac000e90 con 0x7f16bc0b2120 2022-06-29T16:02:11.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.167+0000 7f16b3fff700 1 -- 172.21.15.60:0/578035690 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16ac00d910 con 0x7f16bc0b2120 2022-06-29T16:02:11.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.167+0000 7f16c21a3700 1 -- 172.21.15.60:0/578035690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16bc0b2120 msgr2=0x7f16bc0ab880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:11.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.167+0000 7f16c21a3700 1 --2- 172.21.15.60:0/578035690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16bc0b2120 0x7f16bc0ab880 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7f16ac002f70 tx=0x7f16ac006eb0).stop 2022-06-29T16:02:11.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.167+0000 7f16c21a3700 1 -- 172.21.15.60:0/578035690 shutdown_connections 2022-06-29T16:02:11.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.167+0000 7f16c21a3700 1 --2- 172.21.15.60:0/578035690 >> v2:172.21.15.92:3300/0 conn(0x7f16bc05cf40 0x7f16bc14af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:11.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.167+0000 7f16c21a3700 1 --2- 172.21.15.60:0/578035690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16bc0b2120 0x7f16bc0ab880 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:11.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.167+0000 7f16c21a3700 1 -- 172.21.15.60:0/578035690 >> 172.21.15.60:0/578035690 conn(0x7f16bc0b58a0 msgr2=0x7f16bc0b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:02:11.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.167+0000 7f16c21a3700 1 -- 172.21.15.60:0/578035690 shutdown_connections 2022-06-29T16:02:11.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.167+0000 7f16c21a3700 1 -- 172.21.15.60:0/578035690 wait complete. 2022-06-29T16:02:11.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.171+0000 7f16c21a3700 1 Processor -- start 2022-06-29T16:02:11.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.171+0000 7f16c21a3700 1 -- start start 2022-06-29T16:02:11.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.171+0000 7f16c21a3700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16bc05bd20 0x7f16bc0b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:11.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.171+0000 7f16c21a3700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16bc05cf40 0x7f16bc0abf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:11.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.171+0000 7f16c21a3700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16bc0b2120 0x7f16bc098500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:11.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.171+0000 7f16c21a3700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc099550 con 0x7f16bc05cf40 2022-06-29T16:02:11.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.171+0000 7f16c21a3700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc0ad0c0 con 0x7f16bc0b2120 2022-06-29T16:02:11.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.171+0000 7f16c21a3700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc0ade80 con 0x7f16bc05bd20 2022-06-29T16:02:11.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.171+0000 7f16c09a0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16bc05cf40 0x7f16bc0abf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:11.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.171+0000 7f16c09a0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16bc05cf40 0x7f16bc0abf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48106/0 (socket says 172.21.15.60:48106) 2022-06-29T16:02:11.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.171+0000 7f16c09a0700 1 -- 172.21.15.60:0/3158990975 learned_addr learned my addr 172.21.15.60:0/3158990975 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:02:11.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.171+0000 7f16c11a1700 1 --2- 172.21.15.60:0/3158990975 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16bc05bd20 0x7f16bc0b3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:11.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.171+0000 7f16c19a2700 1 --2- 172.21.15.60:0/3158990975 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16bc0b2120 0x7f16bc098500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:11.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.171+0000 7f16c09a0700 1 -- 172.21.15.60:0/3158990975 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16bc05bd20 msgr2=0x7f16bc0b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:11.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.171+0000 7f16c09a0700 1 --2- 172.21.15.60:0/3158990975 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16bc05bd20 0x7f16bc0b3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:11.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.171+0000 7f16c09a0700 1 -- 172.21.15.60:0/3158990975 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16bc0b2120 msgr2=0x7f16bc098500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:11.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.171+0000 7f16c09a0700 1 --2- 172.21.15.60:0/3158990975 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16bc0b2120 0x7f16bc098500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:11.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.171+0000 7f16c09a0700 1 -- 172.21.15.60:0/3158990975 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ac00b1e0 con 0x7f16bc05cf40 2022-06-29T16:02:11.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.171+0000 7f16c09a0700 1 --2- 172.21.15.60:0/3158990975 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16bc05cf40 0x7f16bc0abf10 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7f16b800c000 tx=0x7f16b8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:11.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.171+0000 7f16b27fc700 1 -- 172.21.15.60:0/3158990975 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16b8010780 con 0x7f16bc05cf40 2022-06-29T16:02:11.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.171+0000 7f16b27fc700 1 -- 172.21.15.60:0/3158990975 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b80108e0 con 0x7f16bc05cf40 2022-06-29T16:02:11.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.171+0000 7f16c21a3700 1 -- 172.21.15.60:0/3158990975 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16bc098060 con 0x7f16bc05cf40 2022-06-29T16:02:11.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.175+0000 7f16b27fc700 1 -- 172.21.15.60:0/3158990975 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16b8010c40 con 0x7f16bc05cf40 2022-06-29T16:02:11.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.175+0000 7f16c21a3700 1 -- 172.21.15.60:0/3158990975 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16bc09ac80 con 0x7f16bc05cf40 2022-06-29T16:02:11.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.175+0000 7f16b27fc700 1 -- 172.21.15.60:0/3158990975 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f16b801c460 con 0x7f16bc05cf40 2022-06-29T16:02:11.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.175+0000 7f16b27fc700 1 --2- 172.21.15.60:0/3158990975 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f16a4053a70 0x7f16a4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:11.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.175+0000 7f16b27fc700 1 -- 172.21.15.60:0/3158990975 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f16b806c2a0 con 0x7f16bc05cf40 2022-06-29T16:02:11.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.175+0000 7f16c11a1700 1 --2- 172.21.15.60:0/3158990975 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f16a4053a70 0x7f16a4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:11.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.179+0000 7f16c11a1700 1 --2- 172.21.15.60:0/3158990975 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f16a4053a70 0x7f16a4055f30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f16ac01c000 tx=0x7f16ac015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:11.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.179+0000 7f16cd65a740 1 Processor -- start 2022-06-29T16:02:11.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.179+0000 7f16cd65a740 1 -- start start 2022-06-29T16:02:11.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.179+0000 7f16cd65a740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1d28f10 0x1ced940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:11.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.179+0000 7f16cd65a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1cefc00 0x1cf0080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:11.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.179+0000 7f16cd65a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1d23460 0x1d238e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:11.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.179+0000 7f16cd65a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x1c4c2d0 con 0x1cefc00 2022-06-29T16:02:11.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.179+0000 7f16cd65a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x1c5fef0 con 0x1d23460 2022-06-29T16:02:11.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.179+0000 7f16cd65a740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x1c401a0 con 0x1d28f10 2022-06-29T16:02:11.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.179+0000 7f16c09a0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1cefc00 0x1cf0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:11.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.179+0000 7f16c09a0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1cefc00 0x1cf0080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48112/0 (socket says 172.21.15.60:48112) 2022-06-29T16:02:11.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.179+0000 7f16c09a0700 1 -- 172.21.15.60:0/1593253553 learned_addr learned my addr 172.21.15.60:0/1593253553 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:02:11.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.179+0000 7f16c11a1700 1 --2- 172.21.15.60:0/1593253553 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1d28f10 0x1ced940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:11.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.179+0000 7f16c19a2700 1 --2- 172.21.15.60:0/1593253553 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1d23460 0x1d238e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:11.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.179+0000 7f16c09a0700 1 -- 172.21.15.60:0/1593253553 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1d28f10 msgr2=0x1ced940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:02:11.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.179+0000 7f16c09a0700 1 --2- 172.21.15.60:0/1593253553 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1d28f10 0x1ced940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:11.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.179+0000 7f16c09a0700 1 -- 172.21.15.60:0/1593253553 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1d23460 msgr2=0x1d238e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:02:11.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.179+0000 7f16c09a0700 1 --2- 172.21.15.60:0/1593253553 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1d23460 0x1d238e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:11.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.179+0000 7f16c09a0700 1 -- 172.21.15.60:0/1593253553 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1c97df0 con 0x1cefc00 2022-06-29T16:02:11.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.179+0000 7f16c09a0700 1 --2- 172.21.15.60:0/1593253553 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1cefc00 0x1cf0080 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x1cee950 tx=0x7f16b8012000).ready entity=mon.0 client_cookie=d4a5da235694a360 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:11.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.179+0000 7f16c21a3700 1 -- 172.21.15.60:0/1593253553 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16b806c0e0 con 0x1cefc00 2022-06-29T16:02:11.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.179+0000 7f16c21a3700 1 -- 172.21.15.60:0/1593253553 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b80451c0 con 0x1cefc00 2022-06-29T16:02:11.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.179+0000 7f16c21a3700 1 -- 172.21.15.60:0/1593253553 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16b8022cc0 con 0x1cefc00 2022-06-29T16:02:11.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.183+0000 7f16cd65a740 1 -- 172.21.15.60:0/1593253553 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1cefc00 msgr2=0x1cf0080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:11.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.183+0000 7f16cd65a740 1 --2- 172.21.15.60:0/1593253553 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1cefc00 0x1cf0080 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x1cee950 tx=0x7f16b8012000).stop 2022-06-29T16:02:11.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.183+0000 7f16cd65a740 1 -- 172.21.15.60:0/1593253553 shutdown_connections 2022-06-29T16:02:11.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.183+0000 7f16cd65a740 1 --2- 172.21.15.60:0/1593253553 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1d23460 0x1d238e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:11.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.183+0000 7f16cd65a740 1 --2- 172.21.15.60:0/1593253553 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1cefc00 0x1cf0080 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:11.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.183+0000 7f16cd65a740 1 --2- 172.21.15.60:0/1593253553 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1d28f10 0x1ced940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:11.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.183+0000 7f16cd65a740 1 -- 172.21.15.60:0/1593253553 >> 172.21.15.60:0/1593253553 conn(0x1cebb20 msgr2=0x1d28650 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:02:11.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.183+0000 7f16cd65a740 1 -- 172.21.15.60:0/1593253553 shutdown_connections 2022-06-29T16:02:11.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.183+0000 7f16cd65a740 1 -- 172.21.15.60:0/1593253553 wait complete. 2022-06-29T16:02:11.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.183+0000 7f16cd65a740 1 -- start start 2022-06-29T16:02:11.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.183+0000 7f16cd65a740 1 Processor -- start 2022-06-29T16:02:11.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.183+0000 7f16cd65a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1d23460 0x1d229d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:11.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.183+0000 7f16cd65a740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1d28f10 0x1d33d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:11.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.183+0000 7f16cd65a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1cf2e60 0x1d341f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:11.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.183+0000 7f16cd65a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x1cf2a30 con 0x1cf2e60 2022-06-29T16:02:11.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.183+0000 7f16cd65a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x1d21860 con 0x1d23460 2022-06-29T16:02:11.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.183+0000 7f16cd65a740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x1d37590 con 0x1d28f10 2022-06-29T16:02:11.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.183+0000 7f16cd65a740 10 client.0 ll_register_callbacks cb 0x1ca73d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-06-29T16:02:11.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.187+0000 7f16c19a2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1cf2e60 0x1d341f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:11.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.187+0000 7f16c19a2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1cf2e60 0x1d341f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48122/0 (socket says 172.21.15.60:48122) 2022-06-29T16:02:11.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.187+0000 7f16c19a2700 1 -- 172.21.15.60:0/1973251032 learned_addr learned my addr 172.21.15.60:0/1973251032 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:02:11.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.187+0000 7f16c09a0700 1 --2- 172.21.15.60:0/1973251032 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1d28f10 0x1d33d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:11.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.187+0000 7f16c11a1700 1 --2- 172.21.15.60:0/1973251032 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1d23460 0x1d229d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:11.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.187+0000 7f16c19a2700 1 -- 172.21.15.60:0/1973251032 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1d28f10 msgr2=0x1d33d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:02:11.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.187+0000 7f16c19a2700 1 --2- 172.21.15.60:0/1973251032 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1d28f10 0x1d33d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:11.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.187+0000 7f16c19a2700 1 -- 172.21.15.60:0/1973251032 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1d23460 msgr2=0x1d229d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:02:11.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.187+0000 7f16c19a2700 1 --2- 172.21.15.60:0/1973251032 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1d23460 0x1d229d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:11.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.187+0000 7f16c19a2700 1 -- 172.21.15.60:0/1973251032 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16bc09ac80 con 0x1cf2e60 2022-06-29T16:02:11.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.187+0000 7f16c11a1700 1 --2- 172.21.15.60:0/1973251032 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1d23460 0x1d229d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:02:11.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.187+0000 7f16c19a2700 1 --2- 172.21.15.60:0/1973251032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1cf2e60 0x1d341f0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7f16b4009000 tx=0x7f16b4009040).ready entity=mon.0 client_cookie=ed3a37e25d70d472 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:11.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.187+0000 7f1697fff700 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2022-06-29T16:02:11.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.187+0000 7f1697fff700 1 -- 172.21.15.60:0/1973251032 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16b4012530 con 0x1cf2e60 2022-06-29T16:02:11.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.187+0000 7f1697fff700 1 -- 172.21.15.60:0/1973251032 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f16b4012690 con 0x1cf2e60 2022-06-29T16:02:11.216 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.187+0000 7f16cd65a740 1 -- 172.21.15.60:0/1973251032 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x1c356c0 con 0x1cf2e60 2022-06-29T16:02:11.216 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.187+0000 7f1697fff700 1 -- 172.21.15.60:0/1973251032 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16b4012850 con 0x1cf2e60 2022-06-29T16:02:11.216 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.187+0000 7f16cd65a740 10 client.5790 fetch_fsmap learned FSMap version 20 2022-06-29T16:02:11.216 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.187+0000 7f16cd65a740 1 -- 172.21.15.60:0/1973251032 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=2+}) v3 -- 0x1c4f3d0 con 0x1cf2e60 2022-06-29T16:02:11.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.187+0000 7f1697fff700 1 -- 172.21.15.60:0/1973251032 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x1c356c0 con 0x1cf2e60 2022-06-29T16:02:11.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.187+0000 7f1697fff700 1 -- 172.21.15.60:0/1973251032 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 20) v1 ==== 1247+0+0 (secure 0 0 0) 0x7f16b4014420 con 0x1cf2e60 2022-06-29T16:02:11.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.187+0000 7f16cd65a740 10 client.5790 fetch_fsmap finished waiting for FSMap version 20 2022-06-29T16:02:11.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.187+0000 7f16cd65a740 10 client.5790 resolve_mds: resolved ID 'b' to GID 5653 2022-06-29T16:02:11.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.187+0000 7f16cd65a740 20 client.5790 populate_metadata read hostname 'smithi060' 2022-06-29T16:02:11.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.187+0000 7f16cd65a740 1 --2- 172.21.15.60:0/1973251032 >> [v2:172.21.15.92:6834/1602880694,v1:172.21.15.92:6835/1602880694] conn(0x1d224f0 0x1d1fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:11.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.187+0000 7f16cd65a740 4 client.5790 mds_command: new command op to 5653 tid=0[{"prefix": "get_command_descriptions"}] 2022-06-29T16:02:11.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.187+0000 7f16cd65a740 1 -- 172.21.15.60:0/1973251032 --> [v2:172.21.15.92:6834/1602880694,v1:172.21.15.92:6835/1602880694] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x1c55de0 con 0x1d224f0 2022-06-29T16:02:11.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.187+0000 7f16c11a1700 1 --2- 172.21.15.60:0/1973251032 >> [v2:172.21.15.92:6834/1602880694,v1:172.21.15.92:6835/1602880694] conn(0x1d224f0 0x1d1fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:11.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.187+0000 7f16c11a1700 1 --2- 172.21.15.60:0/1973251032 >> [v2:172.21.15.92:6834/1602880694,v1:172.21.15.92:6835/1602880694] conn(0x1d224f0 0x1d1fe50 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=9d2c4ee09b2e5cce server_cookie=2276f86c5783b9de in_seq=0 out_seq=0 2022-06-29T16:02:11.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.187+0000 7f1697fff700 10 client.5790 ms_handle_connect on v2:172.21.15.92:6834/1602880694 2022-06-29T16:02:11.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.191+0000 7f1697fff700 1 -- 172.21.15.60:0/1973251032 <== mds.0 v2:172.21.15.92:6834/1602880694 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17002 (crc 0 0 0) 0x1c55de0 con 0x1d224f0 2022-06-29T16:02:11.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.191+0000 7f1697fff700 10 client.5790 handle_command_reply: tid=0 2022-06-29T16:02:11.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.191+0000 7f16cd65a740 1 client.5790 shutdown 2022-06-29T16:02:11.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.191+0000 7f16cd65a740 1 -- 172.21.15.60:0/1973251032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1cf2e60 msgr2=0x1d341f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:11.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.191+0000 7f16cd65a740 1 --2- 172.21.15.60:0/1973251032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1cf2e60 0x1d341f0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7f16b4009000 tx=0x7f16b4009040).stop 2022-06-29T16:02:11.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.191+0000 7f16cd65a740 1 -- 172.21.15.60:0/1973251032 shutdown_connections 2022-06-29T16:02:11.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.191+0000 7f16cd65a740 1 --2- 172.21.15.60:0/1973251032 >> [v2:172.21.15.92:6834/1602880694,v1:172.21.15.92:6835/1602880694] conn(0x1d224f0 0x1d1fe50 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:11.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.191+0000 7f16cd65a740 1 --2- 172.21.15.60:0/1973251032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1cf2e60 0x1d341f0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:11.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.191+0000 7f16cd65a740 1 --2- 172.21.15.60:0/1973251032 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1d28f10 0x1d33d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:11.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.191+0000 7f16cd65a740 1 --2- 172.21.15.60:0/1973251032 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1d23460 0x1d229d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:11.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.191+0000 7f16cd65a740 1 -- 172.21.15.60:0/1973251032 >> 172.21.15.60:0/1973251032 conn(0x1cefc00 msgr2=0x1d285e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:02:11.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.191+0000 7f1697fff700 0 client.5790 ms_handle_reset on v2:172.21.15.92:6834/1602880694 2022-06-29T16:02:11.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.191+0000 7f16cd65a740 1 -- 172.21.15.60:0/1973251032 shutdown_connections 2022-06-29T16:02:11.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.191+0000 7f16cd65a740 1 -- 172.21.15.60:0/1973251032 wait complete. 2022-06-29T16:02:11.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.191+0000 7f16cd65a740 20 client.5790 trim_cache size 0 max 16384 2022-06-29T16:02:11.230 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.207+0000 7f16cd65a740 1 Processor -- start 2022-06-29T16:02:11.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.207+0000 7f16cd65a740 1 -- start start 2022-06-29T16:02:11.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.207+0000 7f16cd65a740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1cf2e60 0x1cf0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:11.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.207+0000 7f16cd65a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1d23460 0x1d33950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:11.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.207+0000 7f16cd65a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1d28f10 0x1d33e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:11.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.207+0000 7f16cd65a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x1c55de0 con 0x1d23460 2022-06-29T16:02:11.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.207+0000 7f16cd65a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x1cf3e00 con 0x1d28f10 2022-06-29T16:02:11.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.207+0000 7f16cd65a740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x1ceedd0 con 0x1cf2e60 2022-06-29T16:02:11.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.207+0000 7f16c09a0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1d23460 0x1d33950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:11.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.207+0000 7f16c09a0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1d23460 0x1d33950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48126/0 (socket says 172.21.15.60:48126) 2022-06-29T16:02:11.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.207+0000 7f16c09a0700 1 -- 172.21.15.60:0/633145133 learned_addr learned my addr 172.21.15.60:0/633145133 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:02:11.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.207+0000 7f16c11a1700 1 --2- 172.21.15.60:0/633145133 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1cf2e60 0x1cf0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:11.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.207+0000 7f16c19a2700 1 --2- 172.21.15.60:0/633145133 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1d28f10 0x1d33e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:11.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.207+0000 7f16c09a0700 1 -- 172.21.15.60:0/633145133 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1cf2e60 msgr2=0x1cf0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:02:11.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.207+0000 7f16c09a0700 1 --2- 172.21.15.60:0/633145133 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1cf2e60 0x1cf0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:11.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.207+0000 7f16c09a0700 1 -- 172.21.15.60:0/633145133 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1d28f10 msgr2=0x1d33e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:02:11.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.207+0000 7f16c09a0700 1 --2- 172.21.15.60:0/633145133 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1d28f10 0x1d33e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:11.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.207+0000 7f16c09a0700 1 -- 172.21.15.60:0/633145133 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1c57fc0 con 0x1d23460 2022-06-29T16:02:11.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.207+0000 7f16c11a1700 1 --2- 172.21.15.60:0/633145133 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1cf2e60 0x1cf0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:02:11.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.207+0000 7f16c09a0700 1 --2- 172.21.15.60:0/633145133 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1d23460 0x1d33950 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x1c483a0 tx=0x7f16b8027000).ready entity=mon.0 client_cookie=9a5a78096995f91e server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:11.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.207+0000 7f169e7fc700 1 -- 172.21.15.60:0/633145133 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16b8033070 con 0x1d23460 2022-06-29T16:02:11.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.207+0000 7f169e7fc700 1 -- 172.21.15.60:0/633145133 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b806c0e0 con 0x1d23460 2022-06-29T16:02:11.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.207+0000 7f169e7fc700 1 -- 172.21.15.60:0/633145133 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16b8022cc0 con 0x1d23460 2022-06-29T16:02:11.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.207+0000 7f16cd65a740 1 -- 172.21.15.60:0/633145133 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1d23460 msgr2=0x1d33950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:11.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.207+0000 7f16cd65a740 1 --2- 172.21.15.60:0/633145133 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1d23460 0x1d33950 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x1c483a0 tx=0x7f16b8027000).stop 2022-06-29T16:02:11.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.207+0000 7f16cd65a740 1 -- 172.21.15.60:0/633145133 shutdown_connections 2022-06-29T16:02:11.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.207+0000 7f16cd65a740 1 --2- 172.21.15.60:0/633145133 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1d28f10 0x1d33e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:11.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.207+0000 7f16cd65a740 1 --2- 172.21.15.60:0/633145133 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1d23460 0x1d33950 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:11.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.207+0000 7f16cd65a740 1 --2- 172.21.15.60:0/633145133 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1cf2e60 0x1cf0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:11.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.207+0000 7f16cd65a740 1 -- 172.21.15.60:0/633145133 >> 172.21.15.60:0/633145133 conn(0x1cefc00 msgr2=0x1d28560 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:02:11.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.207+0000 7f16cd65a740 1 -- 172.21.15.60:0/633145133 shutdown_connections 2022-06-29T16:02:11.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.207+0000 7f16cd65a740 1 -- 172.21.15.60:0/633145133 wait complete. 2022-06-29T16:02:11.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.211+0000 7f16cd65a740 1 -- start start 2022-06-29T16:02:11.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.211+0000 7f16cd65a740 1 Processor -- start 2022-06-29T16:02:11.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.211+0000 7f16cd65a740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1cf2e60 0x1d33cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:11.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.211+0000 7f16cd65a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1d23460 0x1cf0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:11.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.211+0000 7f16cd65a740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1d28f10 0x1cf1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:11.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.211+0000 7f16cd65a740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x1d87b10 con 0x1d28f10 2022-06-29T16:02:11.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.211+0000 7f16cd65a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x1cf0660 con 0x1d23460 2022-06-29T16:02:11.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.211+0000 7f16cd65a740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x1cf0890 con 0x1cf2e60 2022-06-29T16:02:11.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.211+0000 7f16cd65a740 10 client.0 ll_register_callbacks cb 0x1cf42a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-06-29T16:02:11.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.211+0000 7f16c19a2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1d28f10 0x1cf1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:11.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.211+0000 7f16c11a1700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1cf2e60 0x1d33cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:11.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.211+0000 7f16c09a0700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1d23460 0x1cf0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:11.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.211+0000 7f16c11a1700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1cf2e60 0x1d33cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:32994/0 (socket says 172.21.15.60:32994) 2022-06-29T16:02:11.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.211+0000 7f16c11a1700 1 -- 172.21.15.60:0/4187305789 learned_addr learned my addr 172.21.15.60:0/4187305789 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:02:11.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.211+0000 7f16c19a2700 1 --2- 172.21.15.60:0/4187305789 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1d28f10 0x1cf1220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48136/0 (socket says 172.21.15.60:48136) 2022-06-29T16:02:11.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.211+0000 7f16c11a1700 1 -- 172.21.15.60:0/4187305789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1d23460 msgr2=0x1cf0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:02:11.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.211+0000 7f16c11a1700 1 --2- 172.21.15.60:0/4187305789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1d23460 0x1cf0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:11.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.211+0000 7f16c11a1700 1 -- 172.21.15.60:0/4187305789 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1d28f10 msgr2=0x1cf1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:02:11.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.211+0000 7f16c11a1700 1 --2- 172.21.15.60:0/4187305789 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1d28f10 0x1cf1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:11.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.211+0000 7f16c11a1700 1 -- 172.21.15.60:0/4187305789 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x1c97df0 con 0x1cf2e60 2022-06-29T16:02:11.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.215+0000 7f16c11a1700 1 --2- 172.21.15.60:0/4187305789 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1cf2e60 0x1d33cb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f16ac02a000 tx=0x7f16ac033280).ready entity=mon.2 client_cookie=bf9153793dbc1f50 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:11.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.215+0000 7f169d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.92:3301/0 2022-06-29T16:02:11.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.215+0000 7f169d7fa700 1 -- 172.21.15.60:0/4187305789 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16ac032070 con 0x1cf2e60 2022-06-29T16:02:11.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.215+0000 7f16cd65a740 1 -- 172.21.15.60:0/4187305789 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x1c4de60 con 0x1cf2e60 2022-06-29T16:02:11.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.215+0000 7f169d7fa700 1 -- 172.21.15.60:0/4187305789 <== mon.2 v2:172.21.15.92:3301/0 2 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f16ac01fd90 con 0x1cf2e60 2022-06-29T16:02:11.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.215+0000 7f169d7fa700 1 -- 172.21.15.60:0/4187305789 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16ac00de70 con 0x1cf2e60 2022-06-29T16:02:11.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.215+0000 7f16cd65a740 10 client.5357 fetch_fsmap learned FSMap version 20 2022-06-29T16:02:11.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.215+0000 7f16cd65a740 1 -- 172.21.15.60:0/4187305789 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,fsmap=20,monmap=2+}) v3 -- 0x1c549b0 con 0x1cf2e60 2022-06-29T16:02:11.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.215+0000 7f169d7fa700 1 -- 172.21.15.60:0/4187305789 <== mon.2 v2:172.21.15.92:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x1c4de60 con 0x1cf2e60 2022-06-29T16:02:11.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.215+0000 7f169d7fa700 1 -- 172.21.15.60:0/4187305789 <== mon.2 v2:172.21.15.92:3301/0 5 ==== fsmap(e 20) v1 ==== 1247+0+0 (secure 0 0 0) 0x7f16ac01b030 con 0x1cf2e60 2022-06-29T16:02:11.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.215+0000 7f16cd65a740 10 client.5357 fetch_fsmap finished waiting for FSMap version 20 2022-06-29T16:02:11.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.215+0000 7f16cd65a740 10 client.5357 resolve_mds: resolved ID 'b' to GID 5653 2022-06-29T16:02:11.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.215+0000 7f16cd65a740 20 client.5357 populate_metadata read hostname 'smithi060' 2022-06-29T16:02:11.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.215+0000 7f16cd65a740 1 --2- 172.21.15.60:0/4187305789 >> [v2:172.21.15.92:6834/1602880694,v1:172.21.15.92:6835/1602880694] conn(0x1d806d0 0x1d80ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:11.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.215+0000 7f16cd65a740 4 client.5357 mds_command: new command op to 5653 tid=0[{"prefix": "cache drop", "timeout": 5}] 2022-06-29T16:02:11.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.215+0000 7f16cd65a740 1 -- 172.21.15.60:0/4187305789 --> [v2:172.21.15.92:6834/1602880694,v1:172.21.15.92:6835/1602880694] -- command(tid 0: {"prefix": "cache drop", "timeout": 5}) v1 -- 0x1b435e0 con 0x1d806d0 2022-06-29T16:02:11.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.215+0000 7f16c09a0700 1 --2- 172.21.15.60:0/4187305789 >> [v2:172.21.15.92:6834/1602880694,v1:172.21.15.92:6835/1602880694] conn(0x1d806d0 0x1d80ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:11.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.215+0000 7f16c09a0700 1 --2- 172.21.15.60:0/4187305789 >> [v2:172.21.15.92:6834/1602880694,v1:172.21.15.92:6835/1602880694] conn(0x1d806d0 0x1d80ab0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=7f02ea7834d53adc server_cookie=b5b77f52ed350e94 in_seq=0 out_seq=0 2022-06-29T16:02:11.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:11.215+0000 7f169d7fa700 10 client.5357 ms_handle_connect on v2:172.21.15.92:6834/1602880694 2022-06-29T16:02:16.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.383+0000 7f169d7fa700 1 -- 172.21.15.60:0/4187305789 <== mds.0 v2:172.21.15.92:6834/1602880694 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+397 (crc 0 0 0) 0x1b435e0 con 0x1d806d0 2022-06-29T16:02:16.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.383+0000 7f169d7fa700 10 client.5357 handle_command_reply: tid=0 2022-06-29T16:02:16.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.383+0000 7f16cd65a740 1 client.5357 shutdown 2022-06-29T16:02:16.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.383+0000 7f16cd65a740 1 -- 172.21.15.60:0/4187305789 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1cf2e60 msgr2=0x1d33cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:16.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.383+0000 7f16cd65a740 1 --2- 172.21.15.60:0/4187305789 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1cf2e60 0x1d33cb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f16ac02a000 tx=0x7f16ac033280).stop 2022-06-29T16:02:16.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.383+0000 7f16cd65a740 1 -- 172.21.15.60:0/4187305789 shutdown_connections 2022-06-29T16:02:16.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.383+0000 7f16cd65a740 1 --2- 172.21.15.60:0/4187305789 >> [v2:172.21.15.92:6834/1602880694,v1:172.21.15.92:6835/1602880694] conn(0x1d806d0 0x1d80ab0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:16.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.383+0000 7f16cd65a740 1 --2- 172.21.15.60:0/4187305789 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1d28f10 0x1cf1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:16.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.383+0000 7f169d7fa700 0 client.5357 ms_handle_reset on v2:172.21.15.92:6834/1602880694 2022-06-29T16:02:16.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.383+0000 7f16cd65a740 1 --2- 172.21.15.60:0/4187305789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1d23460 0x1cf0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:16.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.383+0000 7f16cd65a740 1 --2- 172.21.15.60:0/4187305789 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1cf2e60 0x1d33cb0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:16.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.383+0000 7f16cd65a740 1 -- 172.21.15.60:0/4187305789 >> 172.21.15.60:0/4187305789 conn(0x1cefc00 msgr2=0x1ced810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:02:16.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.383+0000 7f16cd65a740 1 -- 172.21.15.60:0/4187305789 shutdown_connections 2022-06-29T16:02:16.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.383+0000 7f16cd65a740 1 -- 172.21.15.60:0/4187305789 wait complete. 2022-06-29T16:02:16.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.387+0000 7f16cd65a740 20 client.5357 trim_cache size 0 max 16384 2022-06-29T16:02:16.397 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:02:16.397 INFO:teuthology.orchestra.run.smithi060.stdout: "client_recall": { 2022-06-29T16:02:16.398 INFO:teuthology.orchestra.run.smithi060.stdout: "return_code": -110, 2022-06-29T16:02:16.398 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "(110) Connection timed out", 2022-06-29T16:02:16.398 INFO:teuthology.orchestra.run.smithi060.stdout: "recalled": 120 2022-06-29T16:02:16.398 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:02:16.399 INFO:teuthology.orchestra.run.smithi060.stdout: "flush_journal": { 2022-06-29T16:02:16.399 INFO:teuthology.orchestra.run.smithi060.stdout: "return_code": 0, 2022-06-29T16:02:16.399 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "" 2022-06-29T16:02:16.399 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:02:16.399 INFO:teuthology.orchestra.run.smithi060.stdout: "trim_cache": { 2022-06-29T16:02:16.400 INFO:teuthology.orchestra.run.smithi060.stdout: "trimmed": 0 2022-06-29T16:02:16.400 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:02:16.400 INFO:teuthology.orchestra.run.smithi060.stdout: "cache": { 2022-06-29T16:02:16.400 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": { 2022-06-29T16:02:16.401 INFO:teuthology.orchestra.run.smithi060.stdout: "items": 20067, 2022-06-29T16:02:16.401 INFO:teuthology.orchestra.run.smithi060.stdout: "bytes": 3104608 2022-06-29T16:02:16.401 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:02:16.401 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:02:16.402 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.168509212 2022-06-29T16:02:16.402 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:02:16.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.387+0000 7f16c21a3700 1 -- 172.21.15.60:0/3158990975 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f16a4053a70 msgr2=0x7f16a4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:16.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.387+0000 7f16c21a3700 1 --2- 172.21.15.60:0/3158990975 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f16a4053a70 0x7f16a4055f30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f16ac01c000 tx=0x7f16ac015000).stop 2022-06-29T16:02:16.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.387+0000 7f16c21a3700 1 -- 172.21.15.60:0/3158990975 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16bc05cf40 msgr2=0x7f16bc0abf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:16.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.387+0000 7f16c21a3700 1 --2- 172.21.15.60:0/3158990975 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16bc05cf40 0x7f16bc0abf10 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7f16b800c000 tx=0x7f16b8006df0).stop 2022-06-29T16:02:16.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.387+0000 7f16c21a3700 1 -- 172.21.15.60:0/3158990975 shutdown_connections 2022-06-29T16:02:16.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.387+0000 7f16c21a3700 1 --2- 172.21.15.60:0/3158990975 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f16a4053a70 0x7f16a4055f30 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:16.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.387+0000 7f16c21a3700 1 --2- 172.21.15.60:0/3158990975 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16bc0b2120 0x7f16bc098500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:16.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.387+0000 7f16c21a3700 1 --2- 172.21.15.60:0/3158990975 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16bc05cf40 0x7f16bc0abf10 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:16.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.387+0000 7f16c21a3700 1 --2- 172.21.15.60:0/3158990975 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16bc05bd20 0x7f16bc0b3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:16.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.387+0000 7f16c21a3700 1 -- 172.21.15.60:0/3158990975 >> 172.21.15.60:0/3158990975 conn(0x7f16bc0b58a0 msgr2=0x7f16bc0b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:02:16.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.387+0000 7f16c21a3700 1 -- 172.21.15.60:0/3158990975 shutdown_connections 2022-06-29T16:02:16.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.387+0000 7f16c21a3700 1 -- 172.21.15.60:0/3158990975 wait complete. 2022-06-29T16:02:16.407 INFO:tasks.cephfs.fuse_mount:Cleaning up killed ceph-fuse connection 2022-06-29T16:02:16.408 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:02:16.409 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:02:16.409 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:02:16.416 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': Transport endpoint is not connected 2022-06-29T16:02:16.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:02:16.418 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.0 2022-06-29T16:02:16.419 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2022-06-29T16:02:16.419 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-06-29T16:02:16.419 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:02:16.438 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:02:16.439 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:02:16.439 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:02:16.486 INFO:teuthology.orchestra.run.smithi121.stdout:ext2/ext3 2022-06-29T16:02:16.488 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-06-29T16:02:16.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:02:16.490 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:02:16.498 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:02:16.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.603+0000 7f201479d700 1 Processor -- start 2022-06-29T16:02:16.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.603+0000 7f201479d700 1 -- start start 2022-06-29T16:02:16.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.603+0000 7f201479d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f200c05cb00 0x7f200c061c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:16.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.603+0000 7f201479d700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f200c147270 0x7f200c147720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:16.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.603+0000 7f201479d700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f200c05c560 con 0x7f200c05c180 2022-06-29T16:02:16.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.603+0000 7f201479d700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f200c148320 con 0x7f200c05cb00 2022-06-29T16:02:16.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.603+0000 7f201479d700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f200c062130 con 0x7f200c147270 2022-06-29T16:02:16.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.603+0000 7f2011d38700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f200c05cb00 0x7f200c061c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:16.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.603+0000 7f2012539700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f200c05c180 0x7f200c061530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56226/0 (socket says 172.21.15.60:56226) 2022-06-29T16:02:16.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.603+0000 7f2012539700 1 -- 172.21.15.60:0/3503069805 learned_addr learned my addr 172.21.15.60:0/3503069805 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:02:16.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.603+0000 7f2012d3a700 1 --2- 172.21.15.60:0/3503069805 >> v2:172.21.15.92:3300/0 conn(0x7f200c147270 0x7f200c147720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:16.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.603+0000 7f2011d38700 1 -- 172.21.15.60:0/3503069805 >> v2:172.21.15.92:3300/0 conn(0x7f200c147270 msgr2=0x7f200c147720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:02:16.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.603+0000 7f2011d38700 1 --2- 172.21.15.60:0/3503069805 >> v2:172.21.15.92:3300/0 conn(0x7f200c147270 0x7f200c147720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:16.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.603+0000 7f2011d38700 1 -- 172.21.15.60:0/3503069805 >> v1:172.21.15.60:6789/0 conn(0x7f200c05c180 legacy=0x7f200c061530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:16.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.603+0000 7f2011537700 1 -- 172.21.15.60:0/3503069805 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 725091953 0 0) 0x7f200c05c560 con 0x7f200c05c180 2022-06-29T16:02:16.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.603+0000 7f2011d38700 1 -- 172.21.15.60:0/3503069805 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f200c0626c0 con 0x7f200c05cb00 2022-06-29T16:02:16.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.603+0000 7f2011d38700 1 --2- 172.21.15.60:0/3503069805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f200c05cb00 0x7f200c061c40 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7f200c0b4b30 tx=0x7f20040091d0).ready entity=mon.0 client_cookie=81adcd076c2a2995 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:16.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.603+0000 7f2011537700 1 -- 172.21.15.60:0/3503069805 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2004009ea0 con 0x7f200c05cb00 2022-06-29T16:02:16.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.603+0000 7f2011537700 1 -- 172.21.15.60:0/3503069805 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f200400b860 con 0x7f200c05cb00 2022-06-29T16:02:16.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.603+0000 7f2011537700 1 -- 172.21.15.60:0/3503069805 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f200400b9d0 con 0x7f200c05cb00 2022-06-29T16:02:16.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.607+0000 7f201479d700 1 -- 172.21.15.60:0/3503069805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f200c05cb00 msgr2=0x7f200c061c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:16.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.607+0000 7f201479d700 1 --2- 172.21.15.60:0/3503069805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f200c05cb00 0x7f200c061c40 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7f200c0b4b30 tx=0x7f20040091d0).stop 2022-06-29T16:02:16.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.607+0000 7f201479d700 1 -- 172.21.15.60:0/3503069805 shutdown_connections 2022-06-29T16:02:16.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.607+0000 7f201479d700 1 --2- 172.21.15.60:0/3503069805 >> v2:172.21.15.92:3300/0 conn(0x7f200c147270 0x7f200c147720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:16.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.607+0000 7f201479d700 1 --2- 172.21.15.60:0/3503069805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f200c05cb00 0x7f200c061c40 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:16.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.607+0000 7f201479d700 1 -- 172.21.15.60:0/3503069805 >> 172.21.15.60:0/3503069805 conn(0x7f200c064a10 msgr2=0x7f200c064df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:02:16.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.607+0000 7f201479d700 1 -- 172.21.15.60:0/3503069805 shutdown_connections 2022-06-29T16:02:16.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.607+0000 7f201479d700 1 -- 172.21.15.60:0/3503069805 wait complete. 2022-06-29T16:02:16.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.607+0000 7f201479d700 1 Processor -- start 2022-06-29T16:02:16.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.607+0000 7f201479d700 1 -- start start 2022-06-29T16:02:16.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.611+0000 7f201479d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f200c05c180 0x7f200c0b5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:16.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.611+0000 7f201479d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f200c05cb00 0x7f200c157260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:16.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.611+0000 7f201479d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f200c147270 0x7f200c0ae030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:16.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.611+0000 7f201479d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f200c0b6750 con 0x7f200c05cb00 2022-06-29T16:02:16.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.611+0000 7f201479d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f200c0b7860 con 0x7f200c05c180 2022-06-29T16:02:16.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.611+0000 7f201479d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f200c0b5030 con 0x7f200c147270 2022-06-29T16:02:16.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.611+0000 7f2011d38700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f200c05cb00 0x7f200c157260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:16.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.611+0000 7f2011d38700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f200c05cb00 0x7f200c157260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48148/0 (socket says 172.21.15.60:48148) 2022-06-29T16:02:16.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.611+0000 7f2011d38700 1 -- 172.21.15.60:0/3412420379 learned_addr learned my addr 172.21.15.60:0/3412420379 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:02:16.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.611+0000 7f2012539700 1 --2- 172.21.15.60:0/3412420379 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f200c05c180 0x7f200c0b5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:16.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.611+0000 7f2012d3a700 1 --2- 172.21.15.60:0/3412420379 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f200c147270 0x7f200c0ae030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:16.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.611+0000 7f2011d38700 1 -- 172.21.15.60:0/3412420379 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f200c147270 msgr2=0x7f200c0ae030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:16.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.611+0000 7f2011d38700 1 --2- 172.21.15.60:0/3412420379 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f200c147270 0x7f200c0ae030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:16.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.611+0000 7f2011d38700 1 -- 172.21.15.60:0/3412420379 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f200c05c180 msgr2=0x7f200c0b5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:16.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.611+0000 7f2011d38700 1 --2- 172.21.15.60:0/3412420379 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f200c05c180 0x7f200c0b5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:16.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.611+0000 7f2011d38700 1 -- 172.21.15.60:0/3412420379 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2004009470 con 0x7f200c05cb00 2022-06-29T16:02:16.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.611+0000 7f2011d38700 1 --2- 172.21.15.60:0/3412420379 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f200c05cb00 0x7f200c157260 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7f200c0b4b30 tx=0x7f2004016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:16.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.611+0000 7f1fff7fe700 1 -- 172.21.15.60:0/3412420379 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2004009ea0 con 0x7f200c05cb00 2022-06-29T16:02:16.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.611+0000 7f1fff7fe700 1 -- 172.21.15.60:0/3412420379 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20040032f0 con 0x7f200c05cb00 2022-06-29T16:02:16.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.611+0000 7f1fff7fe700 1 -- 172.21.15.60:0/3412420379 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f200401f5e0 con 0x7f200c05cb00 2022-06-29T16:02:16.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.611+0000 7f201479d700 1 -- 172.21.15.60:0/3412420379 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f200c0b6d40 con 0x7f200c05cb00 2022-06-29T16:02:16.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.611+0000 7f201479d700 1 -- 172.21.15.60:0/3412420379 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f200c0b6aa0 con 0x7f200c05cb00 2022-06-29T16:02:16.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.615+0000 7f201479d700 1 -- 172.21.15.60:0/3412420379 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f200c157000 con 0x7f200c05cb00 2022-06-29T16:02:16.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.615+0000 7f1fff7fe700 1 -- 172.21.15.60:0/3412420379 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2004026030 con 0x7f200c05cb00 2022-06-29T16:02:16.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.615+0000 7f1fff7fe700 1 --2- 172.21.15.60:0/3412420379 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1fec053b00 0x7f1fec055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:16.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.615+0000 7f1fff7fe700 1 -- 172.21.15.60:0/3412420379 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f200406cbb0 con 0x7f200c05cb00 2022-06-29T16:02:16.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.615+0000 7f2012539700 1 --2- 172.21.15.60:0/3412420379 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1fec053b00 0x7f1fec055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:16.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.615+0000 7f2012539700 1 --2- 172.21.15.60:0/3412420379 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1fec053b00 0x7f1fec055fc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f200000d000 tx=0x7f20000011a0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:16.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.623+0000 7f1fff7fe700 1 -- 172.21.15.60:0/3412420379 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2004043940 con 0x7f200c05cb00 2022-06-29T16:02:16.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.751+0000 7f201479d700 1 -- 172.21.15.60:0/3412420379 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f200c0573d0 con 0x7f200c05cb00 2022-06-29T16:02:16.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.751+0000 7f1fff7fe700 1 -- 172.21.15.60:0/3412420379 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+16040 (secure 0 0 0) 0x7f2004024090 con 0x7f200c05cb00 2022-06-29T16:02:16.760 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:02:16.760 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":53,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:01:55.736061+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:01:52.588508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2022-06-29T16:01:53.732456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:02:16.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.755+0000 7f201479d700 1 -- 172.21.15.60:0/3412420379 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1fec053b00 msgr2=0x7f1fec055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:16.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.755+0000 7f201479d700 1 --2- 172.21.15.60:0/3412420379 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1fec053b00 0x7f1fec055fc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f200000d000 tx=0x7f20000011a0).stop 2022-06-29T16:02:16.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.755+0000 7f201479d700 1 -- 172.21.15.60:0/3412420379 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f200c05cb00 msgr2=0x7f200c157260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:16.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.755+0000 7f201479d700 1 --2- 172.21.15.60:0/3412420379 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f200c05cb00 0x7f200c157260 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7f200c0b4b30 tx=0x7f2004016000).stop 2022-06-29T16:02:16.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.755+0000 7f201479d700 1 -- 172.21.15.60:0/3412420379 shutdown_connections 2022-06-29T16:02:16.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.755+0000 7f201479d700 1 --2- 172.21.15.60:0/3412420379 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1fec053b00 0x7f1fec055fc0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:16.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.755+0000 7f201479d700 1 --2- 172.21.15.60:0/3412420379 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f200c147270 0x7f200c0ae030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:16.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.755+0000 7f201479d700 1 --2- 172.21.15.60:0/3412420379 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f200c05cb00 0x7f200c157260 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:16.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.755+0000 7f201479d700 1 --2- 172.21.15.60:0/3412420379 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f200c05c180 0x7f200c0b5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:16.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.755+0000 7f201479d700 1 -- 172.21.15.60:0/3412420379 >> 172.21.15.60:0/3412420379 conn(0x7f200c064a10 msgr2=0x7f200c0b16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:02:16.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.755+0000 7f201479d700 1 -- 172.21.15.60:0/3412420379 shutdown_connections 2022-06-29T16:02:16.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.755+0000 7f201479d700 1 -- 172.21.15.60:0/3412420379 wait complete. 2022-06-29T16:02:16.773 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-29T16:01:52.588508+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2022-06-29T16:01:53.732456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-29T16:02:16.774 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:02:16.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.875+0000 7f9aa603e700 1 Processor -- start 2022-06-29T16:02:16.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.875+0000 7f9aa603e700 1 -- start start 2022-06-29T16:02:16.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9aa603e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9aa00b3c70 0x7f9aa005c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:16.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9aa603e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa00b4050 con 0x7f9aa00aca20 2022-06-29T16:02:16.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9aa603e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa005bea0 con 0x7f9aa00b2860 2022-06-29T16:02:16.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9aa603e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9aa005c140 con 0x7f9aa00b3c70 2022-06-29T16:02:16.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9aa483b700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9aa00aca20 0x7f9aa00b2150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56240/0 (socket says 172.21.15.60:56240) 2022-06-29T16:02:16.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9aa483b700 1 -- 172.21.15.60:0/3805102205 learned_addr learned my addr 172.21.15.60:0/3805102205 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:02:16.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9aa503c700 1 --2- 172.21.15.60:0/3805102205 >> v2:172.21.15.92:3300/0 conn(0x7f9aa00b3c70 0x7f9aa005c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:16.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9a93fff700 1 -- 172.21.15.60:0/3805102205 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 635755341 0 0) 0x7f9aa00b4050 con 0x7f9aa00aca20 2022-06-29T16:02:16.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9a93fff700 1 -- 172.21.15.60:0/3805102205 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a88003260 con 0x7f9aa00aca20 2022-06-29T16:02:16.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9a93fff700 1 -- 172.21.15.60:0/3805102205 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1519242067 0 0) 0x7f9aa005bea0 con 0x7f9aa00b2860 2022-06-29T16:02:16.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9a93fff700 1 -- 172.21.15.60:0/3805102205 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aa00b4050 con 0x7f9aa00b2860 2022-06-29T16:02:16.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9a93fff700 1 -- 172.21.15.60:0/3805102205 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2221195231 0 0) 0x7f9a88003260 con 0x7f9aa00aca20 2022-06-29T16:02:16.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9a93fff700 1 -- 172.21.15.60:0/3805102205 >> v2:172.21.15.92:3300/0 conn(0x7f9aa00b3c70 msgr2=0x7f9aa005c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:02:16.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9a93fff700 1 --2- 172.21.15.60:0/3805102205 >> v2:172.21.15.92:3300/0 conn(0x7f9aa00b3c70 0x7f9aa005c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:16.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9a93fff700 1 -- 172.21.15.60:0/3805102205 >> v1:172.21.15.92:6789/0 conn(0x7f9aa00b2860 legacy=0x7f9aa013f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:16.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9a93fff700 1 -- 172.21.15.60:0/3805102205 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa005c3e0 con 0x7f9aa00aca20 2022-06-29T16:02:16.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9a93fff700 1 -- 172.21.15.60:0/3805102205 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f9a94001660 con 0x7f9aa00aca20 2022-06-29T16:02:16.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9aa503c700 1 --2- 172.21.15.60:0/3805102205 >> v2:172.21.15.92:3300/0 conn(0x7f9aa00b3c70 0x7f9aa005c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:02:16.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9a93fff700 1 -- 172.21.15.60:0/3805102205 >> v1:172.21.15.60:6789/0 conn(0x7f9aa00aca20 legacy=0x7f9aa00b2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:16.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9a93fff700 1 --2- 172.21.15.60:0/3805102205 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a880038e0 0x7f9a88003cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:16.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9a93fff700 1 --2- 172.21.15.60:0/3805102205 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a88004500 0x7f9a88004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:16.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9a93fff700 1 --2- 172.21.15.60:0/3805102205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a88004e70 0x7f9a8800b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:16.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9a93fff700 1 -- 172.21.15.60:0/3805102205 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa005c140 con 0x7f9a88004e70 2022-06-29T16:02:16.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9a93fff700 1 -- 172.21.15.60:0/3805102205 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8800bca0 con 0x7f9a880038e0 2022-06-29T16:02:16.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9a93fff700 1 -- 172.21.15.60:0/3805102205 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8800be60 con 0x7f9a88004500 2022-06-29T16:02:16.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9aa503c700 1 --2- 172.21.15.60:0/3805102205 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a880038e0 0x7f9a88003cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:16.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9aa583d700 1 --2- 172.21.15.60:0/3805102205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a88004e70 0x7f9a8800b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:16.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9aa483b700 1 --2- 172.21.15.60:0/3805102205 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a88004500 0x7f9a88004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:16.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9aa583d700 1 -- 172.21.15.60:0/3805102205 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a88004500 msgr2=0x7f9a88004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:02:16.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9aa583d700 1 --2- 172.21.15.60:0/3805102205 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a88004500 0x7f9a88004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:16.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9aa583d700 1 -- 172.21.15.60:0/3805102205 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a880038e0 msgr2=0x7f9a88003cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:02:16.903 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9aa583d700 1 --2- 172.21.15.60:0/3805102205 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a880038e0 0x7f9a88003cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:16.903 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9aa583d700 1 -- 172.21.15.60:0/3805102205 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a88003260 con 0x7f9a88004e70 2022-06-29T16:02:16.904 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9aa503c700 1 --2- 172.21.15.60:0/3805102205 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a880038e0 0x7f9a88003cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:02:16.904 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9aa583d700 1 -- 172.21.15.60:0/3805102205 reap_dead start 2022-06-29T16:02:16.904 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9aa583d700 1 --2- 172.21.15.60:0/3805102205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a88004e70 0x7f9a8800b750 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7f9a98001490 tx=0x7f9a98007d80).ready entity=mon.0 client_cookie=209a184946f904d7 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:16.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9a93fff700 1 -- 172.21.15.60:0/3805102205 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a98001780 con 0x7f9a88004e70 2022-06-29T16:02:16.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9a93fff700 1 -- 172.21.15.60:0/3805102205 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a98006890 con 0x7f9a88004e70 2022-06-29T16:02:16.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.879+0000 7f9a93fff700 1 -- 172.21.15.60:0/3805102205 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a98006bf0 con 0x7f9a88004e70 2022-06-29T16:02:16.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.883+0000 7f9aa603e700 1 -- 172.21.15.60:0/3805102205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a88004e70 msgr2=0x7f9a8800b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:16.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.883+0000 7f9aa603e700 1 --2- 172.21.15.60:0/3805102205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a88004e70 0x7f9a8800b750 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7f9a98001490 tx=0x7f9a98007d80).stop 2022-06-29T16:02:16.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.883+0000 7f9aa603e700 1 -- 172.21.15.60:0/3805102205 shutdown_connections 2022-06-29T16:02:16.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.883+0000 7f9aa603e700 1 --2- 172.21.15.60:0/3805102205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a88004e70 0x7f9a8800b750 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:16.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.883+0000 7f9aa603e700 1 -- 172.21.15.60:0/3805102205 >> 172.21.15.60:0/3805102205 conn(0x7f9aa0064810 msgr2=0x7f9aa0064bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:02:16.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.883+0000 7f9aa603e700 1 -- 172.21.15.60:0/3805102205 shutdown_connections 2022-06-29T16:02:16.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.883+0000 7f9aa603e700 1 -- 172.21.15.60:0/3805102205 wait complete. 2022-06-29T16:02:16.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.883+0000 7f9aa603e700 1 Processor -- start 2022-06-29T16:02:16.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.883+0000 7f9aa603e700 1 -- start start 2022-06-29T16:02:16.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.883+0000 7f9aa603e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a88004e70 0x7f9aa013d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:16.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.883+0000 7f9aa603e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9aa0062930 0x7f9aa00ace90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:16.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.883+0000 7f9aa603e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9aa00ad6c0 0x7f9aa013c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:16.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.883+0000 7f9aa603e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa00b6700 con 0x7f9a88004e70 2022-06-29T16:02:16.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.883+0000 7f9aa603e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa00ad380 con 0x7f9aa0062930 2022-06-29T16:02:16.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.883+0000 7f9aa603e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa00631a0 con 0x7f9aa00ad6c0 2022-06-29T16:02:16.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.883+0000 7f9aa503c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a88004e70 0x7f9aa013d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:16.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.883+0000 7f9aa503c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a88004e70 0x7f9aa013d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48168/0 (socket says 172.21.15.60:48168) 2022-06-29T16:02:16.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.883+0000 7f9aa503c700 1 -- 172.21.15.60:0/2850227071 learned_addr learned my addr 172.21.15.60:0/2850227071 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:02:16.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.883+0000 7f9aa583d700 1 --2- 172.21.15.60:0/2850227071 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9aa00ad6c0 0x7f9aa013c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:16.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.883+0000 7f9aa483b700 1 --2- 172.21.15.60:0/2850227071 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9aa0062930 0x7f9aa00ace90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:16.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.883+0000 7f9aa503c700 1 -- 172.21.15.60:0/2850227071 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9aa00ad6c0 msgr2=0x7f9aa013c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:16.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.883+0000 7f9aa503c700 1 --2- 172.21.15.60:0/2850227071 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9aa00ad6c0 0x7f9aa013c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:16.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.883+0000 7f9aa503c700 1 -- 172.21.15.60:0/2850227071 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9aa0062930 msgr2=0x7f9aa00ace90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:16.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.883+0000 7f9aa503c700 1 --2- 172.21.15.60:0/2850227071 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9aa0062930 0x7f9aa00ace90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:16.915 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.883+0000 7f9aa503c700 1 -- 172.21.15.60:0/2850227071 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa00afff0 con 0x7f9a88004e70 2022-06-29T16:02:16.915 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.883+0000 7f9aa503c700 1 --2- 172.21.15.60:0/2850227071 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a88004e70 0x7f9aa013d9b0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7f9aa00b6840 tx=0x7f9a9c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:16.915 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.883+0000 7f9a927fc700 1 -- 172.21.15.60:0/2850227071 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a9c00f8e0 con 0x7f9a88004e70 2022-06-29T16:02:16.915 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.883+0000 7f9a927fc700 1 -- 172.21.15.60:0/2850227071 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a9c00fa40 con 0x7f9a88004e70 2022-06-29T16:02:16.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.883+0000 7f9aa603e700 1 -- 172.21.15.60:0/2850227071 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aa00afd90 con 0x7f9a88004e70 2022-06-29T16:02:16.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.883+0000 7f9a927fc700 1 -- 172.21.15.60:0/2850227071 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a9c00fda0 con 0x7f9a88004e70 2022-06-29T16:02:16.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.883+0000 7f9aa603e700 1 -- 172.21.15.60:0/2850227071 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aa00b63f0 con 0x7f9a88004e70 2022-06-29T16:02:16.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.887+0000 7f9aa603e700 1 -- 172.21.15.60:0/2850227071 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aa00573d0 con 0x7f9a88004e70 2022-06-29T16:02:16.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.887+0000 7f9a927fc700 1 -- 172.21.15.60:0/2850227071 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9a9c01e460 con 0x7f9a88004e70 2022-06-29T16:02:16.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.887+0000 7f9a927fc700 1 --2- 172.21.15.60:0/2850227071 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9a80053a70 0x7f9a80055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:16.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.887+0000 7f9a927fc700 1 -- 172.21.15.60:0/2850227071 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9a9c06cb60 con 0x7f9a88004e70 2022-06-29T16:02:16.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.887+0000 7f9aa483b700 1 --2- 172.21.15.60:0/2850227071 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9a80053a70 0x7f9a80055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:16.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.887+0000 7f9aa483b700 1 --2- 172.21.15.60:0/2850227071 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9a80053a70 0x7f9a80055f30 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f9a9400b000 tx=0x7f9a94007740).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:16.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:16.891+0000 7f9a927fc700 1 -- 172.21.15.60:0/2850227071 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9a9c043a20 con 0x7f9a88004e70 2022-06-29T16:02:17.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.019+0000 7f9aa603e700 1 -- 172.21.15.60:0/2850227071 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f9aa00afa10 con 0x7f9a88004e70 2022-06-29T16:02:17.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.023+0000 7f9a927fc700 1 -- 172.21.15.60:0/2850227071 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v53) v1 ==== 110+0+10 (secure 0 0 0) 0x7f9a9c047040 con 0x7f9a88004e70 2022-06-29T16:02:17.030 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:02:17.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.023+0000 7f9aa603e700 1 -- 172.21.15.60:0/2850227071 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9a80053a70 msgr2=0x7f9a80055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:17.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.023+0000 7f9aa603e700 1 --2- 172.21.15.60:0/2850227071 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9a80053a70 0x7f9a80055f30 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f9a9400b000 tx=0x7f9a94007740).stop 2022-06-29T16:02:17.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.023+0000 7f9aa603e700 1 -- 172.21.15.60:0/2850227071 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a88004e70 msgr2=0x7f9aa013d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:17.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.023+0000 7f9aa603e700 1 --2- 172.21.15.60:0/2850227071 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a88004e70 0x7f9aa013d9b0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7f9aa00b6840 tx=0x7f9a9c006df0).stop 2022-06-29T16:02:17.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.023+0000 7f9aa603e700 1 -- 172.21.15.60:0/2850227071 shutdown_connections 2022-06-29T16:02:17.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.023+0000 7f9aa603e700 1 --2- 172.21.15.60:0/2850227071 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9a80053a70 0x7f9a80055f30 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:17.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.023+0000 7f9aa603e700 1 --2- 172.21.15.60:0/2850227071 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9aa00ad6c0 0x7f9aa013c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:17.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.023+0000 7f9aa603e700 1 --2- 172.21.15.60:0/2850227071 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9aa0062930 0x7f9aa00ace90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:17.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.023+0000 7f9aa603e700 1 --2- 172.21.15.60:0/2850227071 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a88004e70 0x7f9aa013d9b0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:17.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.023+0000 7f9aa603e700 1 -- 172.21.15.60:0/2850227071 >> 172.21.15.60:0/2850227071 conn(0x7f9aa0064810 msgr2=0x7f9aa00b58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:02:17.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.023+0000 7f9aa603e700 1 -- 172.21.15.60:0/2850227071 shutdown_connections 2022-06-29T16:02:17.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.023+0000 7f9aa603e700 1 -- 172.21.15.60:0/2850227071 wait complete. 2022-06-29T16:02:17.043 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:02:17.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.147+0000 7f4bad686700 1 Processor -- start 2022-06-29T16:02:17.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.147+0000 7f4bad686700 1 -- start start 2022-06-29T16:02:17.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.147+0000 7f4bad686700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4ba8060760 0x7f4ba80b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:17.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.147+0000 7f4bad686700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba813fa40 con 0x7f4ba8065450 2022-06-29T16:02:17.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.147+0000 7f4bad686700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba813fce0 con 0x7f4ba80b28e0 2022-06-29T16:02:17.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.147+0000 7f4bad686700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ba8140860 con 0x7f4ba8060760 2022-06-29T16:02:17.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.147+0000 7f4ba77fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4ba8065450 0x7f4ba8066390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56262/0 (socket says 172.21.15.60:56262) 2022-06-29T16:02:17.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.147+0000 7f4ba77fe700 1 -- 172.21.15.60:0/2375672776 learned_addr learned my addr 172.21.15.60:0/2375672776 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:02:17.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.147+0000 7f4bace85700 1 --2- 172.21.15.60:0/2375672776 >> v2:172.21.15.60:3300/0 conn(0x7f4ba8060760 0x7f4ba80b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:17.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.147+0000 7f4ba6ffd700 1 -- 172.21.15.60:0/2375672776 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2129400324 0 0) 0x7f4ba813fa40 con 0x7f4ba8065450 2022-06-29T16:02:17.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.147+0000 7f4ba6ffd700 1 -- 172.21.15.60:0/2375672776 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b8c003260 con 0x7f4ba8065450 2022-06-29T16:02:17.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.147+0000 7f4ba6ffd700 1 -- 172.21.15.60:0/2375672776 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3466991478 0 0) 0x7f4ba813fce0 con 0x7f4ba80b28e0 2022-06-29T16:02:17.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.147+0000 7f4ba6ffd700 1 -- 172.21.15.60:0/2375672776 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ba813fa40 con 0x7f4ba80b28e0 2022-06-29T16:02:17.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.147+0000 7f4bace85700 1 -- 172.21.15.60:0/2375672776 >> v1:172.21.15.92:6789/0 conn(0x7f4ba80b28e0 legacy=0x7f4ba8139340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:17.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.147+0000 7f4bace85700 1 -- 172.21.15.60:0/2375672776 >> v1:172.21.15.60:6789/0 conn(0x7f4ba8065450 legacy=0x7f4ba8066390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:17.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.147+0000 7f4bace85700 1 -- 172.21.15.60:0/2375672776 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba813f6f0 con 0x7f4ba8060760 2022-06-29T16:02:17.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.147+0000 7f4bace85700 1 --2- 172.21.15.60:0/2375672776 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ba8060760 0x7f4ba80b2cc0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7f4b98002f70 tx=0x7f4b98006df0).ready entity=mon.0 client_cookie=41dec7f70aba7c90 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:17.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.151+0000 7f4ba6ffd700 1 -- 172.21.15.60:0/2375672776 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b9800d540 con 0x7f4ba8060760 2022-06-29T16:02:17.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.151+0000 7f4ba6ffd700 1 -- 172.21.15.60:0/2375672776 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b98000e90 con 0x7f4ba8060760 2022-06-29T16:02:17.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.151+0000 7f4ba6ffd700 1 -- 172.21.15.60:0/2375672776 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b9800d890 con 0x7f4ba8060760 2022-06-29T16:02:17.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.151+0000 7f4bad686700 1 -- 172.21.15.60:0/2375672776 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ba8060760 msgr2=0x7f4ba80b2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:17.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.151+0000 7f4bad686700 1 --2- 172.21.15.60:0/2375672776 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ba8060760 0x7f4ba80b2cc0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7f4b98002f70 tx=0x7f4b98006df0).stop 2022-06-29T16:02:17.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.151+0000 7f4bad686700 1 -- 172.21.15.60:0/2375672776 shutdown_connections 2022-06-29T16:02:17.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.151+0000 7f4bad686700 1 --2- 172.21.15.60:0/2375672776 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ba8060760 0x7f4ba80b2cc0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:17.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.151+0000 7f4bad686700 1 -- 172.21.15.60:0/2375672776 >> 172.21.15.60:0/2375672776 conn(0x7f4ba8061c80 msgr2=0x7f4ba80637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:02:17.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.151+0000 7f4bad686700 1 -- 172.21.15.60:0/2375672776 shutdown_connections 2022-06-29T16:02:17.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.151+0000 7f4bad686700 1 -- 172.21.15.60:0/2375672776 wait complete. 2022-06-29T16:02:17.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.151+0000 7f4bad686700 1 Processor -- start 2022-06-29T16:02:17.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.151+0000 7f4bad686700 1 -- start start 2022-06-29T16:02:17.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.155+0000 7f4bad686700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ba8060760 0x7f4ba8138540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:17.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.155+0000 7f4bad686700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ba8065450 0x7f4ba81528a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:17.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.155+0000 7f4bad686700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4ba80b28e0 0x7f4ba80b4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:17.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.155+0000 7f4bad686700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba80b4e00 con 0x7f4ba8065450 2022-06-29T16:02:17.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.155+0000 7f4bad686700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba8139140 con 0x7f4ba8060760 2022-06-29T16:02:17.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.155+0000 7f4bad686700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba81393e0 con 0x7f4ba80b28e0 2022-06-29T16:02:17.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.155+0000 7f4ba77fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ba8065450 0x7f4ba81528a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:17.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.155+0000 7f4ba77fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ba8065450 0x7f4ba81528a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48182/0 (socket says 172.21.15.60:48182) 2022-06-29T16:02:17.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.155+0000 7f4ba77fe700 1 -- 172.21.15.60:0/872045405 learned_addr learned my addr 172.21.15.60:0/872045405 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:02:17.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.155+0000 7f4bace85700 1 --2- 172.21.15.60:0/872045405 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4ba80b28e0 0x7f4ba80b4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:17.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.155+0000 7f4ba7fff700 1 --2- 172.21.15.60:0/872045405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ba8060760 0x7f4ba8138540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:17.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.155+0000 7f4ba77fe700 1 -- 172.21.15.60:0/872045405 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4ba80b28e0 msgr2=0x7f4ba80b4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:17.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.155+0000 7f4ba77fe700 1 --2- 172.21.15.60:0/872045405 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4ba80b28e0 0x7f4ba80b4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:17.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.155+0000 7f4ba77fe700 1 -- 172.21.15.60:0/872045405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ba8060760 msgr2=0x7f4ba8138540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:17.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.155+0000 7f4ba77fe700 1 --2- 172.21.15.60:0/872045405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ba8060760 0x7f4ba8138540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:17.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.155+0000 7f4ba77fe700 1 -- 172.21.15.60:0/872045405 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b9800b180 con 0x7f4ba8065450 2022-06-29T16:02:17.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.155+0000 7f4ba7fff700 1 --2- 172.21.15.60:0/872045405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ba8060760 0x7f4ba8138540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:02:17.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.155+0000 7f4ba77fe700 1 --2- 172.21.15.60:0/872045405 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ba8065450 0x7f4ba81528a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7f4b90001550 tx=0x7f4b90007ed0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:17.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.155+0000 7f4ba57fa700 1 -- 172.21.15.60:0/872045405 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b90005ba0 con 0x7f4ba8065450 2022-06-29T16:02:17.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.155+0000 7f4ba57fa700 1 -- 172.21.15.60:0/872045405 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b90005d00 con 0x7f4ba8065450 2022-06-29T16:02:17.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.155+0000 7f4bad686700 1 -- 172.21.15.60:0/872045405 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba8138270 con 0x7f4ba8065450 2022-06-29T16:02:17.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.155+0000 7f4ba57fa700 1 -- 172.21.15.60:0/872045405 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b90007300 con 0x7f4ba8065450 2022-06-29T16:02:17.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.155+0000 7f4bad686700 1 -- 172.21.15.60:0/872045405 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba80b4ab0 con 0x7f4ba8065450 2022-06-29T16:02:17.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.155+0000 7f4bad686700 1 -- 172.21.15.60:0/872045405 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba80573d0 con 0x7f4ba8065450 2022-06-29T16:02:17.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.155+0000 7f4ba57fa700 1 -- 172.21.15.60:0/872045405 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4b90010030 con 0x7f4ba8065450 2022-06-29T16:02:17.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.155+0000 7f4ba57fa700 1 --2- 172.21.15.60:0/872045405 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4b80053b00 0x7f4b80055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:17.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.159+0000 7f4ba57fa700 1 -- 172.21.15.60:0/872045405 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4b9006be90 con 0x7f4ba8065450 2022-06-29T16:02:17.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.159+0000 7f4ba7fff700 1 --2- 172.21.15.60:0/872045405 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4b80053b00 0x7f4b80055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:17.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.159+0000 7f4ba7fff700 1 --2- 172.21.15.60:0/872045405 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4b80053b00 0x7f4b80055fc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f4ba000b000 tx=0x7f4ba0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:17.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.163+0000 7f4ba57fa700 1 -- 172.21.15.60:0/872045405 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4b90042d30 con 0x7f4ba8065450 2022-06-29T16:02:17.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.291+0000 7f4bad686700 1 -- 172.21.15.60:0/872045405 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f4ba8139b60 con 0x7f4ba8065450 2022-06-29T16:02:17.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.291+0000 7f4ba57fa700 1 -- 172.21.15.60:0/872045405 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v53) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4b90046350 con 0x7f4ba8065450 2022-06-29T16:02:17.298 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:02:17.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.291+0000 7f4bad686700 1 -- 172.21.15.60:0/872045405 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4b80053b00 msgr2=0x7f4b80055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:17.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.291+0000 7f4bad686700 1 --2- 172.21.15.60:0/872045405 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4b80053b00 0x7f4b80055fc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f4ba000b000 tx=0x7f4ba0009040).stop 2022-06-29T16:02:17.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.291+0000 7f4bad686700 1 -- 172.21.15.60:0/872045405 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ba8065450 msgr2=0x7f4ba81528a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:17.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.291+0000 7f4bad686700 1 --2- 172.21.15.60:0/872045405 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ba8065450 0x7f4ba81528a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7f4b90001550 tx=0x7f4b90007ed0).stop 2022-06-29T16:02:17.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.291+0000 7f4bad686700 1 -- 172.21.15.60:0/872045405 shutdown_connections 2022-06-29T16:02:17.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.291+0000 7f4bad686700 1 --2- 172.21.15.60:0/872045405 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4b80053b00 0x7f4b80055fc0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:17.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.291+0000 7f4bad686700 1 --2- 172.21.15.60:0/872045405 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4ba80b28e0 0x7f4ba80b4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:17.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.291+0000 7f4bad686700 1 --2- 172.21.15.60:0/872045405 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ba8065450 0x7f4ba81528a0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:17.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.291+0000 7f4bad686700 1 --2- 172.21.15.60:0/872045405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ba8060760 0x7f4ba8138540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:17.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.291+0000 7f4bad686700 1 -- 172.21.15.60:0/872045405 >> 172.21.15.60:0/872045405 conn(0x7f4ba8061c80 msgr2=0x7f4ba80b6630 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:02:17.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.291+0000 7f4bad686700 1 -- 172.21.15.60:0/872045405 shutdown_connections 2022-06-29T16:02:17.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.291+0000 7f4bad686700 1 -- 172.21.15.60:0/872045405 wait complete. 2022-06-29T16:02:17.311 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-29T16:02:17.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.415+0000 7f8693155700 1 Processor -- start 2022-06-29T16:02:17.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.415+0000 7f8693155700 1 -- start start 2022-06-29T16:02:17.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.419+0000 7f8693155700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f868c0b2270 0x7f868c0b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:17.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.419+0000 7f8693155700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f868c0b2650 con 0x7f868c0aca20 2022-06-29T16:02:17.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.419+0000 7f8693155700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f868c062af0 con 0x7f868c0634d0 2022-06-29T16:02:17.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.419+0000 7f8693155700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f868c067770 con 0x7f868c0b2270 2022-06-29T16:02:17.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.419+0000 7f8692954700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f868c0aca20 0x7f868c0af720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56278/0 (socket says 172.21.15.60:56278) 2022-06-29T16:02:17.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.419+0000 7f8692954700 1 -- 172.21.15.60:0/1493904778 learned_addr learned my addr 172.21.15.60:0/1493904778 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:02:17.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.419+0000 7f8692153700 1 --2- 172.21.15.60:0/1493904778 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f868c0b2270 0x7f868c0b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:17.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.419+0000 7f8691151700 1 -- 172.21.15.60:0/1493904778 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 541041635 0 0) 0x7f868c0b2650 con 0x7f868c0aca20 2022-06-29T16:02:17.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.419+0000 7f8691151700 1 -- 172.21.15.60:0/1493904778 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8674003260 con 0x7f868c0aca20 2022-06-29T16:02:17.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.419+0000 7f8692153700 1 -- 172.21.15.60:0/1493904778 >> v1:172.21.15.92:6789/0 conn(0x7f868c0634d0 legacy=0x7f868c13e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:17.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.419+0000 7f8691151700 1 -- 172.21.15.60:0/1493904778 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2590590715 0 0) 0x7f868c062af0 con 0x7f868c0634d0 2022-06-29T16:02:17.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.419+0000 7f8692153700 1 -- 172.21.15.60:0/1493904778 >> v1:172.21.15.60:6789/0 conn(0x7f868c0aca20 legacy=0x7f868c0af720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:17.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.419+0000 7f8692153700 1 -- 172.21.15.60:0/1493904778 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f868c062d90 con 0x7f868c0b2270 2022-06-29T16:02:17.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.419+0000 7f8692153700 1 --2- 172.21.15.60:0/1493904778 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f868c0b2270 0x7f868c0b0b40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f8688002f70 tx=0x7f8688006df0).ready entity=mon.2 client_cookie=342a7b04513b959c server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:17.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.419+0000 7f8691151700 1 -- 172.21.15.60:0/1493904778 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8688014070 con 0x7f868c0b2270 2022-06-29T16:02:17.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.419+0000 7f8691151700 1 -- 172.21.15.60:0/1493904778 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8688000e90 con 0x7f868c0b2270 2022-06-29T16:02:17.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.419+0000 7f8691151700 1 -- 172.21.15.60:0/1493904778 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f868800fb60 con 0x7f868c0b2270 2022-06-29T16:02:17.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.419+0000 7f8693155700 1 -- 172.21.15.60:0/1493904778 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f868c0b2270 msgr2=0x7f868c0b0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:17.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.419+0000 7f8693155700 1 --2- 172.21.15.60:0/1493904778 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f868c0b2270 0x7f868c0b0b40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f8688002f70 tx=0x7f8688006df0).stop 2022-06-29T16:02:17.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.419+0000 7f8693155700 1 -- 172.21.15.60:0/1493904778 shutdown_connections 2022-06-29T16:02:17.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.419+0000 7f8693155700 1 --2- 172.21.15.60:0/1493904778 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f868c0b2270 0x7f868c0b0b40 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:17.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.419+0000 7f8693155700 1 -- 172.21.15.60:0/1493904778 >> 172.21.15.60:0/1493904778 conn(0x7f868c0b5890 msgr2=0x7f868c0b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:02:17.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.419+0000 7f8693155700 1 -- 172.21.15.60:0/1493904778 shutdown_connections 2022-06-29T16:02:17.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.419+0000 7f8693155700 1 -- 172.21.15.60:0/1493904778 wait complete. 2022-06-29T16:02:17.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.423+0000 7f8693155700 1 Processor -- start 2022-06-29T16:02:17.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.423+0000 7f8693155700 1 -- start start 2022-06-29T16:02:17.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.423+0000 7f8693155700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f868c0634d0 0x7f868c13ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:17.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.423+0000 7f8693155700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f868c0aca20 0x7f868c151950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:17.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.423+0000 7f8693155700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f868c0b2270 0x7f868c155490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:17.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.423+0000 7f8693155700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868c1155f0 con 0x7f868c0634d0 2022-06-29T16:02:17.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.423+0000 7f8693155700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868c116ef0 con 0x7f868c0b2270 2022-06-29T16:02:17.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.423+0000 7f8693155700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f868c116c50 con 0x7f868c0aca20 2022-06-29T16:02:17.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.423+0000 7f8692153700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f868c0634d0 0x7f868c13ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:17.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.423+0000 7f8692153700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f868c0634d0 0x7f868c13ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48194/0 (socket says 172.21.15.60:48194) 2022-06-29T16:02:17.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.423+0000 7f8692153700 1 -- 172.21.15.60:0/3589433760 learned_addr learned my addr 172.21.15.60:0/3589433760 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:02:17.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.423+0000 7f8691952700 1 --2- 172.21.15.60:0/3589433760 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f868c0aca20 0x7f868c151950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:17.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.423+0000 7f8692954700 1 --2- 172.21.15.60:0/3589433760 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f868c0b2270 0x7f868c155490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:17.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.423+0000 7f8692153700 1 -- 172.21.15.60:0/3589433760 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f868c0aca20 msgr2=0x7f868c151950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:17.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.423+0000 7f8692153700 1 --2- 172.21.15.60:0/3589433760 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f868c0aca20 0x7f868c151950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:17.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.423+0000 7f8692153700 1 -- 172.21.15.60:0/3589433760 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f868c0b2270 msgr2=0x7f868c155490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:17.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.423+0000 7f8692153700 1 --2- 172.21.15.60:0/3589433760 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f868c0b2270 0x7f868c155490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:17.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.423+0000 7f8692153700 1 -- 172.21.15.60:0/3589433760 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f868800b180 con 0x7f868c0634d0 2022-06-29T16:02:17.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.423+0000 7f8692153700 1 --2- 172.21.15.60:0/3589433760 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f868c0634d0 0x7f868c13ebb0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7f8688002f70 tx=0x7f8688016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:17.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.423+0000 7f867b7fe700 1 -- 172.21.15.60:0/3589433760 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8688006a80 con 0x7f868c0634d0 2022-06-29T16:02:17.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.423+0000 7f867b7fe700 1 -- 172.21.15.60:0/3589433760 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f868800e1d0 con 0x7f868c0634d0 2022-06-29T16:02:17.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.423+0000 7f8693155700 1 -- 172.21.15.60:0/3589433760 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f868c154ff0 con 0x7f868c0634d0 2022-06-29T16:02:17.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.423+0000 7f867b7fe700 1 -- 172.21.15.60:0/3589433760 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8688010df0 con 0x7f868c0634d0 2022-06-29T16:02:17.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.423+0000 7f8693155700 1 -- 172.21.15.60:0/3589433760 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f868c151e40 con 0x7f868c0634d0 2022-06-29T16:02:17.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.427+0000 7f8693155700 1 -- 172.21.15.60:0/3589433760 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f868c0573d0 con 0x7f868c0634d0 2022-06-29T16:02:17.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.427+0000 7f867b7fe700 1 -- 172.21.15.60:0/3589433760 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8688022020 con 0x7f868c0634d0 2022-06-29T16:02:17.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.427+0000 7f867b7fe700 1 --2- 172.21.15.60:0/3589433760 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f866c053a70 0x7f866c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:17.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.427+0000 7f867b7fe700 1 -- 172.21.15.60:0/3589433760 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f868806d960 con 0x7f868c0634d0 2022-06-29T16:02:17.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.427+0000 7f8691952700 1 --2- 172.21.15.60:0/3589433760 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f866c053a70 0x7f866c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:17.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.427+0000 7f8691952700 1 --2- 172.21.15.60:0/3589433760 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f866c053a70 0x7f866c055f30 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f867c00d000 tx=0x7f867c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:17.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.435+0000 7f867b7fe700 1 -- 172.21.15.60:0/3589433760 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8688044820 con 0x7f868c0634d0 2022-06-29T16:02:17.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.551+0000 7f8693155700 1 -- 172.21.15.60:0/3589433760 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f868c0683d0 con 0x7f868c0634d0 2022-06-29T16:02:17.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.555+0000 7f867b7fe700 1 -- 172.21.15.60:0/3589433760 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v53) v1 ==== 104+0+10 (secure 0 0 0) 0x7f8688047e40 con 0x7f868c0634d0 2022-06-29T16:02:17.562 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:02:17.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.555+0000 7f8693155700 1 -- 172.21.15.60:0/3589433760 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f866c053a70 msgr2=0x7f866c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:17.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.555+0000 7f8693155700 1 --2- 172.21.15.60:0/3589433760 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f866c053a70 0x7f866c055f30 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f867c00d000 tx=0x7f867c009040).stop 2022-06-29T16:02:17.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.555+0000 7f8693155700 1 -- 172.21.15.60:0/3589433760 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f868c0634d0 msgr2=0x7f868c13ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:17.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.555+0000 7f8693155700 1 --2- 172.21.15.60:0/3589433760 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f868c0634d0 0x7f868c13ebb0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7f8688002f70 tx=0x7f8688016000).stop 2022-06-29T16:02:17.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.555+0000 7f8693155700 1 -- 172.21.15.60:0/3589433760 shutdown_connections 2022-06-29T16:02:17.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.555+0000 7f8693155700 1 --2- 172.21.15.60:0/3589433760 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f866c053a70 0x7f866c055f30 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:17.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.555+0000 7f8693155700 1 --2- 172.21.15.60:0/3589433760 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f868c0b2270 0x7f868c155490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:17.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.555+0000 7f8693155700 1 --2- 172.21.15.60:0/3589433760 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f868c0aca20 0x7f868c151950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:17.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.555+0000 7f8693155700 1 --2- 172.21.15.60:0/3589433760 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f868c0634d0 0x7f868c13ebb0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:17.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.555+0000 7f8693155700 1 -- 172.21.15.60:0/3589433760 >> 172.21.15.60:0/3589433760 conn(0x7f868c0b5890 msgr2=0x7f868c141bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:02:17.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.555+0000 7f8693155700 1 -- 172.21.15.60:0/3589433760 shutdown_connections 2022-06-29T16:02:17.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.555+0000 7f8693155700 1 -- 172.21.15.60:0/3589433760 wait complete. 2022-06-29T16:02:17.575 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-29T16:02:17.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.675+0000 7fa3a6829700 1 Processor -- start 2022-06-29T16:02:17.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.679+0000 7fa3a6829700 1 -- start start 2022-06-29T16:02:17.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.679+0000 7fa3a6829700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa3a00ae7c0 0x7fa3a0141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:17.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.679+0000 7fa3a6829700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa3a00b4010 0x7fa3a0142490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:17.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.679+0000 7fa3a6829700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3a0143a40 con 0x7fa3a0146160 2022-06-29T16:02:17.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.679+0000 7fa3a6829700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3a00b56d0 con 0x7fa3a00b4010 2022-06-29T16:02:17.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.679+0000 7fa3a6829700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3a00aee60 con 0x7fa3a00ae7c0 2022-06-29T16:02:17.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.679+0000 7fa39f7fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa3a00b4010 0x7fa3a0142490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:17.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.679+0000 7fa3a4dc6700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa3a0146160 0x7fa3a0146590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56292/0 (socket says 172.21.15.60:56292) 2022-06-29T16:02:17.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.679+0000 7fa3a4dc6700 1 -- 172.21.15.60:0/2845619189 learned_addr learned my addr 172.21.15.60:0/2845619189 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:02:17.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.679+0000 7fa39ffff700 1 --2- 172.21.15.60:0/2845619189 >> v2:172.21.15.92:3300/0 conn(0x7fa3a00ae7c0 0x7fa3a0141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:17.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.679+0000 7fa39effd700 1 -- 172.21.15.60:0/2845619189 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3105708376 0 0) 0x7fa3a0143a40 con 0x7fa3a0146160 2022-06-29T16:02:17.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.679+0000 7fa39effd700 1 -- 172.21.15.60:0/2845619189 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa380003260 con 0x7fa3a0146160 2022-06-29T16:02:17.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.679+0000 7fa39f7fe700 1 -- 172.21.15.60:0/2845619189 >> v2:172.21.15.92:3300/0 conn(0x7fa3a00ae7c0 msgr2=0x7fa3a0141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:02:17.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.679+0000 7fa39f7fe700 1 --2- 172.21.15.60:0/2845619189 >> v2:172.21.15.92:3300/0 conn(0x7fa3a00ae7c0 0x7fa3a0141fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:17.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.679+0000 7fa39f7fe700 1 -- 172.21.15.60:0/2845619189 >> v1:172.21.15.60:6789/0 conn(0x7fa3a0146160 legacy=0x7fa3a0146590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:17.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.679+0000 7fa39effd700 1 -- 172.21.15.60:0/2845619189 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1913100257 0 0) 0x7fa380003260 con 0x7fa3a0146160 2022-06-29T16:02:17.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.679+0000 7fa39f7fe700 1 -- 172.21.15.60:0/2845619189 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a00af100 con 0x7fa3a00b4010 2022-06-29T16:02:17.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.679+0000 7fa39ffff700 1 --2- 172.21.15.60:0/2845619189 >> v2:172.21.15.92:3300/0 conn(0x7fa3a00ae7c0 0x7fa3a0141fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:02:17.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.679+0000 7fa39f7fe700 1 --2- 172.21.15.60:0/2845619189 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3a00b4010 0x7fa3a0142490 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7fa398001b80 tx=0x7fa388006df0).ready entity=mon.0 client_cookie=e9ab6d45447ba6c4 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:17.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.679+0000 7fa39effd700 1 -- 172.21.15.60:0/2845619189 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3980019e0 con 0x7fa3a00b4010 2022-06-29T16:02:17.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.679+0000 7fa39effd700 1 -- 172.21.15.60:0/2845619189 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa388000e90 con 0x7fa3a00b4010 2022-06-29T16:02:17.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.679+0000 7fa39effd700 1 -- 172.21.15.60:0/2845619189 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa38800d4a0 con 0x7fa3a00b4010 2022-06-29T16:02:17.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.679+0000 7fa3a6829700 1 -- 172.21.15.60:0/2845619189 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3a00b4010 msgr2=0x7fa3a0142490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:17.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.679+0000 7fa3a6829700 1 --2- 172.21.15.60:0/2845619189 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3a00b4010 0x7fa3a0142490 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7fa398001b80 tx=0x7fa388006df0).stop 2022-06-29T16:02:17.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.679+0000 7fa3a6829700 1 -- 172.21.15.60:0/2845619189 shutdown_connections 2022-06-29T16:02:17.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.679+0000 7fa3a6829700 1 --2- 172.21.15.60:0/2845619189 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3a00b4010 0x7fa3a0142490 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:17.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.679+0000 7fa3a6829700 1 --2- 172.21.15.60:0/2845619189 >> v2:172.21.15.92:3300/0 conn(0x7fa3a00ae7c0 0x7fa3a0141fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:17.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.679+0000 7fa3a6829700 1 -- 172.21.15.60:0/2845619189 >> 172.21.15.60:0/2845619189 conn(0x7fa3a00b9340 msgr2=0x7fa3a00b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:02:17.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.679+0000 7fa3a6829700 1 -- 172.21.15.60:0/2845619189 shutdown_connections 2022-06-29T16:02:17.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.679+0000 7fa3a6829700 1 -- 172.21.15.60:0/2845619189 wait complete. 2022-06-29T16:02:17.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.683+0000 7fa3a6829700 1 Processor -- start 2022-06-29T16:02:17.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.683+0000 7fa3a6829700 1 -- start start 2022-06-29T16:02:17.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.683+0000 7fa3a6829700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3a00ae7c0 0x7fa3a01486b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:17.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.683+0000 7fa3a6829700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3a00b4010 0x7fa3a01553c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:17.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.683+0000 7fa3a6829700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3a0146160 0x7fa3a0112c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:17.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.683+0000 7fa3a6829700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a01558b0 con 0x7fa3a00ae7c0 2022-06-29T16:02:17.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.683+0000 7fa3a6829700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a00b22c0 con 0x7fa3a00b4010 2022-06-29T16:02:17.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.683+0000 7fa3a6829700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a01127d0 con 0x7fa3a0146160 2022-06-29T16:02:17.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.683+0000 7fa39ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3a00ae7c0 0x7fa3a01486b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:17.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.683+0000 7fa39ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3a00ae7c0 0x7fa3a01486b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48208/0 (socket says 172.21.15.60:48208) 2022-06-29T16:02:17.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.683+0000 7fa39ffff700 1 -- 172.21.15.60:0/3516512015 learned_addr learned my addr 172.21.15.60:0/3516512015 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:02:17.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.683+0000 7fa3a4dc6700 1 --2- 172.21.15.60:0/3516512015 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3a0146160 0x7fa3a0112c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:17.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.683+0000 7fa39f7fe700 1 --2- 172.21.15.60:0/3516512015 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3a00b4010 0x7fa3a01553c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:17.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.683+0000 7fa39ffff700 1 -- 172.21.15.60:0/3516512015 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3a0146160 msgr2=0x7fa3a0112c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:17.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.683+0000 7fa39ffff700 1 --2- 172.21.15.60:0/3516512015 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3a0146160 0x7fa3a0112c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:17.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.683+0000 7fa39ffff700 1 -- 172.21.15.60:0/3516512015 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3a00b4010 msgr2=0x7fa3a01553c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:17.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.683+0000 7fa39ffff700 1 --2- 172.21.15.60:0/3516512015 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3a00b4010 0x7fa3a01553c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:17.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.683+0000 7fa39ffff700 1 -- 172.21.15.60:0/3516512015 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa38800b180 con 0x7fa3a00ae7c0 2022-06-29T16:02:17.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.683+0000 7fa39f7fe700 1 --2- 172.21.15.60:0/3516512015 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3a00b4010 0x7fa3a01553c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:02:17.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.683+0000 7fa39ffff700 1 --2- 172.21.15.60:0/3516512015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3a00ae7c0 0x7fa3a01486b0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7fa390006940 tx=0x7fa390008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:17.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.687+0000 7fa39d7fa700 1 -- 172.21.15.60:0/3516512015 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa390011850 con 0x7fa3a00ae7c0 2022-06-29T16:02:17.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.687+0000 7fa39d7fa700 1 -- 172.21.15.60:0/3516512015 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3900119b0 con 0x7fa3a00ae7c0 2022-06-29T16:02:17.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.687+0000 7fa3a6829700 1 -- 172.21.15.60:0/3516512015 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3a00b2560 con 0x7fa3a00ae7c0 2022-06-29T16:02:17.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.687+0000 7fa39d7fa700 1 -- 172.21.15.60:0/3516512015 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa390011d10 con 0x7fa3a00ae7c0 2022-06-29T16:02:17.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.687+0000 7fa3a6829700 1 -- 172.21.15.60:0/3516512015 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3a00b1e90 con 0x7fa3a00ae7c0 2022-06-29T16:02:17.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.687+0000 7fa3a6829700 1 -- 172.21.15.60:0/3516512015 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3a00b0290 con 0x7fa3a00ae7c0 2022-06-29T16:02:17.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.687+0000 7fa39d7fa700 1 -- 172.21.15.60:0/3516512015 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa390012420 con 0x7fa3a00ae7c0 2022-06-29T16:02:17.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.687+0000 7fa39d7fa700 1 --2- 172.21.15.60:0/3516512015 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa3840539e0 0x7fa384055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:17.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.687+0000 7fa39d7fa700 1 -- 172.21.15.60:0/3516512015 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa39006cb10 con 0x7fa3a00ae7c0 2022-06-29T16:02:17.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.687+0000 7fa39f7fe700 1 --2- 172.21.15.60:0/3516512015 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa3840539e0 0x7fa384055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:17.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.687+0000 7fa39f7fe700 1 --2- 172.21.15.60:0/3516512015 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa3840539e0 0x7fa384055ea0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fa388018000 tx=0x7fa388014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:17.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.695+0000 7fa39d7fa700 1 -- 172.21.15.60:0/3516512015 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa3900439d0 con 0x7fa3a00ae7c0 2022-06-29T16:02:17.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.823+0000 7fa3a6829700 1 -- 172.21.15.60:0/3516512015 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa3a00b0290 con 0x7fa3a00ae7c0 2022-06-29T16:02:17.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.823+0000 7fa39d7fa700 1 -- 172.21.15.60:0/3516512015 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v53) v1 ==== 100+0+10 (secure 0 0 0) 0x7fa3a00b0290 con 0x7fa3a00ae7c0 2022-06-29T16:02:17.832 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:02:17.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.827+0000 7fa3a6829700 1 -- 172.21.15.60:0/3516512015 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa3840539e0 msgr2=0x7fa384055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:17.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.827+0000 7fa3a6829700 1 --2- 172.21.15.60:0/3516512015 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa3840539e0 0x7fa384055ea0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fa388018000 tx=0x7fa388014000).stop 2022-06-29T16:02:17.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.827+0000 7fa3a6829700 1 -- 172.21.15.60:0/3516512015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3a00ae7c0 msgr2=0x7fa3a01486b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:17.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.827+0000 7fa3a6829700 1 --2- 172.21.15.60:0/3516512015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3a00ae7c0 0x7fa3a01486b0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7fa390006940 tx=0x7fa390008040).stop 2022-06-29T16:02:17.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.827+0000 7fa3a6829700 1 -- 172.21.15.60:0/3516512015 shutdown_connections 2022-06-29T16:02:17.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.827+0000 7fa3a6829700 1 --2- 172.21.15.60:0/3516512015 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa3840539e0 0x7fa384055ea0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:17.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.827+0000 7fa3a6829700 1 --2- 172.21.15.60:0/3516512015 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3a0146160 0x7fa3a0112c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:17.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.827+0000 7fa3a6829700 1 --2- 172.21.15.60:0/3516512015 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3a00b4010 0x7fa3a01553c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:17.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.827+0000 7fa3a6829700 1 --2- 172.21.15.60:0/3516512015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3a00ae7c0 0x7fa3a01486b0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:17.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.827+0000 7fa3a6829700 1 -- 172.21.15.60:0/3516512015 >> 172.21.15.60:0/3516512015 conn(0x7fa3a00b9340 msgr2=0x7fa3a005bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:02:17.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.827+0000 7fa3a6829700 1 -- 172.21.15.60:0/3516512015 shutdown_connections 2022-06-29T16:02:17.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.827+0000 7fa3a6829700 1 -- 172.21.15.60:0/3516512015 wait complete. 2022-06-29T16:02:17.845 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-06-29T16:02:17.845 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:02:17.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53da42d700 1 Processor -- start 2022-06-29T16:02:17.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53da42d700 1 -- start start 2022-06-29T16:02:17.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53da42d700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f53d40aed00 0x7f53d4146810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:17.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53da42d700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53d405d450 con 0x7f53d40b2130 2022-06-29T16:02:17.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53da42d700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53d40b25d0 con 0x7f53d405d070 2022-06-29T16:02:17.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53da42d700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53d40adf90 con 0x7f53d40aed00 2022-06-29T16:02:17.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53d8c2a700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f53d40b2130 0x7f53d4146100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56302/0 (socket says 172.21.15.60:56302) 2022-06-29T16:02:17.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53d8c2a700 1 -- 172.21.15.60:0/3064512520 learned_addr learned my addr 172.21.15.60:0/3064512520 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:02:17.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53d9c2c700 1 --2- 172.21.15.60:0/3064512520 >> v2:172.21.15.92:3300/0 conn(0x7f53d40aed00 0x7f53d4146810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:17.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53cbfff700 1 -- 172.21.15.60:0/3064512520 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 909517949 0 0) 0x7f53d405d450 con 0x7f53d40b2130 2022-06-29T16:02:17.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53cbfff700 1 -- 172.21.15.60:0/3064512520 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53b8003260 con 0x7f53d40b2130 2022-06-29T16:02:17.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53cbfff700 1 -- 172.21.15.60:0/3064512520 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2369884611 0 0) 0x7f53d40b25d0 con 0x7f53d405d070 2022-06-29T16:02:17.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53cbfff700 1 -- 172.21.15.60:0/3064512520 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53d405d450 con 0x7f53d405d070 2022-06-29T16:02:17.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53cbfff700 1 -- 172.21.15.60:0/3064512520 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2888276459 0 0) 0x7f53b8003260 con 0x7f53d40b2130 2022-06-29T16:02:17.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53cbfff700 1 -- 172.21.15.60:0/3064512520 >> v2:172.21.15.92:3300/0 conn(0x7f53d40aed00 msgr2=0x7f53d4146810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:02:17.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53cbfff700 1 --2- 172.21.15.60:0/3064512520 >> v2:172.21.15.92:3300/0 conn(0x7f53d40aed00 0x7f53d4146810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:17.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53cbfff700 1 -- 172.21.15.60:0/3064512520 >> v1:172.21.15.92:6789/0 conn(0x7f53d405d070 legacy=0x7f53d414aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:17.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53cbfff700 1 -- 172.21.15.60:0/3064512520 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d414a350 con 0x7f53d40b2130 2022-06-29T16:02:17.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53cbfff700 1 -- 172.21.15.60:0/3064512520 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f53c4001660 con 0x7f53d40b2130 2022-06-29T16:02:17.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53cbfff700 1 -- 172.21.15.60:0/3064512520 >> v1:172.21.15.60:6789/0 conn(0x7f53d40b2130 legacy=0x7f53d4146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:17.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53cbfff700 1 --2- 172.21.15.60:0/3064512520 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f53b80038e0 0x7f53b8003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:17.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53cbfff700 1 --2- 172.21.15.60:0/3064512520 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53b8004500 0x7f53b8004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:17.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53cbfff700 1 --2- 172.21.15.60:0/3064512520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53b8004e70 0x7f53b800b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:17.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53cbfff700 1 -- 172.21.15.60:0/3064512520 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d40adf90 con 0x7f53b8004e70 2022-06-29T16:02:17.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53cbfff700 1 -- 172.21.15.60:0/3064512520 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b800bdd0 con 0x7f53b8004500 2022-06-29T16:02:17.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53cbfff700 1 -- 172.21.15.60:0/3064512520 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b800bf40 con 0x7f53b80038e0 2022-06-29T16:02:17.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53d9c2c700 1 --2- 172.21.15.60:0/3064512520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53b8004e70 0x7f53b800b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:17.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53d8c2a700 1 --2- 172.21.15.60:0/3064512520 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53b8004500 0x7f53b8004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:17.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53d942b700 1 --2- 172.21.15.60:0/3064512520 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f53b80038e0 0x7f53b8003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:17.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53d8c2a700 1 -- 172.21.15.60:0/3064512520 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f53b80038e0 msgr2=0x7f53b8003ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:02:17.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53d8c2a700 1 --2- 172.21.15.60:0/3064512520 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f53b80038e0 0x7f53b8003ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:17.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53d8c2a700 1 -- 172.21.15.60:0/3064512520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53b8004e70 msgr2=0x7f53b800b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:02:17.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53d8c2a700 1 --2- 172.21.15.60:0/3064512520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53b8004e70 0x7f53b800b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:17.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53d8c2a700 1 -- 172.21.15.60:0/3064512520 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b8003260 con 0x7f53b8004500 2022-06-29T16:02:17.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53d9c2c700 1 --2- 172.21.15.60:0/3064512520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53b8004e70 0x7f53b800b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:02:17.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53d9c2c700 1 -- 172.21.15.60:0/3064512520 reap_dead start 2022-06-29T16:02:17.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53d8c2a700 1 --2- 172.21.15.60:0/3064512520 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53b8004500 0x7f53b8004980 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f53c4001430 tx=0x7f53c4001dd0).ready entity=mon.1 client_cookie=b45c21c4c7ec5824 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:17.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53cbfff700 1 -- 172.21.15.60:0/3064512520 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f53c40070d0 con 0x7f53b8004500 2022-06-29T16:02:17.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53cbfff700 1 -- 172.21.15.60:0/3064512520 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53c4007230 con 0x7f53b8004500 2022-06-29T16:02:17.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.943+0000 7f53cbfff700 1 -- 172.21.15.60:0/3064512520 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f53c400c8b0 con 0x7f53b8004500 2022-06-29T16:02:17.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.947+0000 7f53da42d700 1 -- 172.21.15.60:0/3064512520 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53b8004500 msgr2=0x7f53b8004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:17.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.947+0000 7f53da42d700 1 --2- 172.21.15.60:0/3064512520 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53b8004500 0x7f53b8004980 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f53c4001430 tx=0x7f53c4001dd0).stop 2022-06-29T16:02:17.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.947+0000 7f53da42d700 1 -- 172.21.15.60:0/3064512520 shutdown_connections 2022-06-29T16:02:17.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.947+0000 7f53da42d700 1 --2- 172.21.15.60:0/3064512520 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53b8004500 0x7f53b8004980 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:17.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.947+0000 7f53da42d700 1 -- 172.21.15.60:0/3064512520 >> 172.21.15.60:0/3064512520 conn(0x7f53d40b5890 msgr2=0x7f53d40b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:02:17.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.947+0000 7f53da42d700 1 -- 172.21.15.60:0/3064512520 shutdown_connections 2022-06-29T16:02:17.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.947+0000 7f53da42d700 1 -- 172.21.15.60:0/3064512520 wait complete. 2022-06-29T16:02:17.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.947+0000 7f53da42d700 1 Processor -- start 2022-06-29T16:02:17.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.947+0000 7f53da42d700 1 -- start start 2022-06-29T16:02:17.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.951+0000 7f53da42d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53b8004500 0x7f53d41598d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:17.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.951+0000 7f53da42d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f53d4116c90 0x7f53d41181e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:17.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.951+0000 7f53da42d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53d4064560 0x7f53d4064a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:17.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.951+0000 7f53da42d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d4116780 con 0x7f53d4064560 2022-06-29T16:02:17.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.951+0000 7f53da42d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d415a1c0 con 0x7f53b8004500 2022-06-29T16:02:17.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.951+0000 7f53da42d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53d41169b0 con 0x7f53d4116c90 2022-06-29T16:02:17.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.951+0000 7f53d9c2c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53d4064560 0x7f53d4064a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:17.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.951+0000 7f53d9c2c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53d4064560 0x7f53d4064a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48230/0 (socket says 172.21.15.60:48230) 2022-06-29T16:02:17.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.951+0000 7f53d9c2c700 1 -- 172.21.15.60:0/1623660756 learned_addr learned my addr 172.21.15.60:0/1623660756 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:02:17.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.951+0000 7f53d8c2a700 1 --2- 172.21.15.60:0/1623660756 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f53d4116c90 0x7f53d41181e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:17.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.951+0000 7f53d942b700 1 --2- 172.21.15.60:0/1623660756 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53b8004500 0x7f53d41598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:17.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.951+0000 7f53d9c2c700 1 -- 172.21.15.60:0/1623660756 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f53d4116c90 msgr2=0x7f53d41181e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:17.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.951+0000 7f53d9c2c700 1 --2- 172.21.15.60:0/1623660756 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f53d4116c90 0x7f53d41181e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:17.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.951+0000 7f53d9c2c700 1 -- 172.21.15.60:0/1623660756 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53b8004500 msgr2=0x7f53d41598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:17.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.951+0000 7f53d9c2c700 1 --2- 172.21.15.60:0/1623660756 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53b8004500 0x7f53d41598d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:17.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.951+0000 7f53d9c2c700 1 -- 172.21.15.60:0/1623660756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d4159f20 con 0x7f53d4064560 2022-06-29T16:02:17.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.951+0000 7f53d9c2c700 1 --2- 172.21.15.60:0/1623660756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53d4064560 0x7f53d4064a10 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7f53d415a350 tx=0x7f53cc0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:17.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.951+0000 7f53ca7fc700 1 -- 172.21.15.60:0/1623660756 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f53cc019070 con 0x7f53d4064560 2022-06-29T16:02:17.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.951+0000 7f53ca7fc700 1 -- 172.21.15.60:0/1623660756 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53cc00f910 con 0x7f53d4064560 2022-06-29T16:02:17.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.951+0000 7f53ca7fc700 1 -- 172.21.15.60:0/1623660756 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f53cc00fc70 con 0x7f53d4064560 2022-06-29T16:02:17.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.951+0000 7f53da42d700 1 -- 172.21.15.60:0/1623660756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53d4065fc0 con 0x7f53d4064560 2022-06-29T16:02:17.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.951+0000 7f53da42d700 1 -- 172.21.15.60:0/1623660756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53d41186d0 con 0x7f53d4064560 2022-06-29T16:02:17.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.951+0000 7f53da42d700 1 -- 172.21.15.60:0/1623660756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53d414a5f0 con 0x7f53d4064560 2022-06-29T16:02:17.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.955+0000 7f53ca7fc700 1 -- 172.21.15.60:0/1623660756 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f53cc014440 con 0x7f53d4064560 2022-06-29T16:02:17.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.955+0000 7f53ca7fc700 1 --2- 172.21.15.60:0/1623660756 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f53bc053a10 0x7f53bc055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:02:17.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.955+0000 7f53ca7fc700 1 -- 172.21.15.60:0/1623660756 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f53cc06c8f0 con 0x7f53d4064560 2022-06-29T16:02:17.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.955+0000 7f53d942b700 1 --2- 172.21.15.60:0/1623660756 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f53bc053a10 0x7f53bc055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:02:17.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.955+0000 7f53d942b700 1 --2- 172.21.15.60:0/1623660756 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f53bc053a10 0x7f53bc055ed0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f53d0003000 tx=0x7f53d0007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:02:17.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:17.959+0000 7f53ca7fc700 1 -- 172.21.15.60:0/1623660756 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f53cc0437b0 con 0x7f53d4064560 2022-06-29T16:02:18.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:18.127+0000 7f53da42d700 1 -- 172.21.15.60:0/1623660756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53d40573d0 con 0x7f53d4064560 2022-06-29T16:02:18.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:18.127+0000 7f53ca7fc700 1 -- 172.21.15.60:0/1623660756 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+2139 (secure 0 0 0) 0x7f53cc046dd0 con 0x7f53d4064560 2022-06-29T16:02:18.135 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:02:18.135 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":20,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5568,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/4229695218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":4229695218},{"type":"v1","addr":"172.21.15.60:6833","nonce":4229695218}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:01:54.732520+0000","modified":"2022-06-29T16:01:55.735497+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5653":{"gid":5653,"name":"b","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/1602880694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1602880694},{"type":"v1","addr":"172.21.15.92:6835","nonce":1602880694}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-06-29T16:02:18.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:18.127+0000 7f53da42d700 1 -- 172.21.15.60:0/1623660756 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f53bc053a10 msgr2=0x7f53bc055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:18.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:18.127+0000 7f53da42d700 1 --2- 172.21.15.60:0/1623660756 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f53bc053a10 0x7f53bc055ed0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f53d0003000 tx=0x7f53d0007a00).stop 2022-06-29T16:02:18.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:18.127+0000 7f53da42d700 1 -- 172.21.15.60:0/1623660756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53d4064560 msgr2=0x7f53d4064a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:02:18.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:18.127+0000 7f53da42d700 1 --2- 172.21.15.60:0/1623660756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53d4064560 0x7f53d4064a10 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7f53d415a350 tx=0x7f53cc0091d0).stop 2022-06-29T16:02:18.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:18.127+0000 7f53da42d700 1 -- 172.21.15.60:0/1623660756 shutdown_connections 2022-06-29T16:02:18.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:18.127+0000 7f53da42d700 1 --2- 172.21.15.60:0/1623660756 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f53bc053a10 0x7f53bc055ed0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:18.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:18.127+0000 7f53da42d700 1 --2- 172.21.15.60:0/1623660756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53d4064560 0x7f53d4064a10 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:18.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:18.127+0000 7f53da42d700 1 --2- 172.21.15.60:0/1623660756 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f53d4116c90 0x7f53d41181e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:18.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:18.127+0000 7f53da42d700 1 --2- 172.21.15.60:0/1623660756 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53b8004500 0x7f53d41598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:02:18.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:18.127+0000 7f53da42d700 1 -- 172.21.15.60:0/1623660756 >> 172.21.15.60:0/1623660756 conn(0x7f53d40b5890 msgr2=0x7f53d40af720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:02:18.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:18.127+0000 7f53da42d700 1 -- 172.21.15.60:0/1623660756 shutdown_connections 2022-06-29T16:02:18.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:02:18.127+0000 7f53da42d700 1 -- 172.21.15.60:0/1623660756 wait complete. 2022-06-29T16:02:18.140 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 20 2022-06-29T16:02:18.350 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:01:54.732520+0000', 'modified': '2022-06-29T16:01:55.735497+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5653}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5653': {'gid': 5653, 'name': 'b', 'rank': 0, 'incarnation': 19, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.92:6835/1602880694', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 1602880694}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 1602880694}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:02:18.351 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:02:18.351 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-06-29T16:02:18.469 INFO:teuthology.orchestra.run.smithi092.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":5653,"want_state":"up:active","state":"up:active","rank_uptime":23.720533813999999,"mdsmap_epoch":20,"osdmap_epoch":53,"osdmap_epoch_barrier":52,"uptime":28.671031492000001} 2022-06-29T16:02:18.470 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 5653, "mdsmap_epoch": 20, "osdmap_epoch": 53, "osdmap_epoch_barrier": 52, "rank_uptime": 23.720533814, "state": "up:active", "uptime": 28.671031492, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:02:18.470 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-06-29T16:02:18.471 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-06-29T16:02:18.471 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi121.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-06-29T16:02:18.471 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:02:18.472 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:02:18.480 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:02:18.495 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:02:18.496 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:02:18.508 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:02:18.508 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:02:18.509 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:02:18.510 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:02:18.558 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-06-29T16:02:18.559 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:02:18.561 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:02:18.579 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:02:18.580 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:02:18.590 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:02:18.590 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:02:18.591 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:02:18.592 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:02:18.639 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:2022-06-29T16:02:18.635+0000 7fbe02512080 -1 init, newargv = 0x5562de661280 newargc=9ceph-fuse[ 2022-06-29T16:02:18.640 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:12680]: starting ceph client 2022-06-29T16:02:18.648 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[12680]: starting fuse 2022-06-29T16:02:19.749 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:02:19.767 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:02:19.768 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:02:19.779 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:02:19.780 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:02:19.781 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:02:19.781 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:02:19.830 INFO:teuthology.orchestra.run.smithi121.stdout:52 2022-06-29T16:02:19.831 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-06-29T16:02:19.831 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:02:19.832 DEBUG:teuthology.orchestra.run.smithi121:> sudo python3 -c ' 2022-06-29T16:02:19.832 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2022-06-29T16:02:19.833 DEBUG:teuthology.orchestra.run.smithi121:> import re 2022-06-29T16:02:19.833 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-06-29T16:02:19.834 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2022-06-29T16:02:19.834 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:02:19.835 DEBUG:teuthology.orchestra.run.smithi121:> def find_socket(client_name): 2022-06-29T16:02:19.835 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-06-29T16:02:19.835 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2022-06-29T16:02:19.836 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:02:19.836 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2022-06-29T16:02:19.837 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2022-06-29T16:02:19.838 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2022-06-29T16:02:19.838 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2022-06-29T16:02:19.839 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:02:19.839 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2022-06-29T16:02:19.839 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-06-29T16:02:19.840 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2022-06-29T16:02:19.840 DEBUG:teuthology.orchestra.run.smithi121:> return f 2022-06-29T16:02:19.841 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-06-29T16:02:19.841 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:02:19.842 DEBUG:teuthology.orchestra.run.smithi121:> print(find_socket("client.0")) 2022-06-29T16:02:19.842 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-06-29T16:02:19.887 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.0.12680.asok 2022-06-29T16:02:19.892 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.12680.asok 2022-06-29T16:02:19.892 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:02:19.893 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.12680.asok status 2022-06-29T16:02:20.027 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2022-06-29T16:02:20.028 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2022-06-29T16:02:20.028 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "f25ffac72e5a9e63860ff7deb6767a11f0559a16", 2022-06-29T16:02:20.028 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 15.2.16-304-gf25ffac7 (f25ffac72e5a9e63860ff7deb6767a11f0559a16) octopus (stable)", 2022-06-29T16:02:20.028 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "0", 2022-06-29T16:02:20.029 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2022-06-29T16:02:20.029 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-06-29T16:02:20.029 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "12680", 2022-06-29T16:02:20.030 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2022-06-29T16:02:20.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:02:20.030 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2022-06-29T16:02:20.031 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2022-06-29T16:02:20.031 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 5896, 2022-06-29T16:02:20.031 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2022-06-29T16:02:20.031 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2022-06-29T16:02:20.032 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2022-06-29T16:02:20.032 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 5896 2022-06-29T16:02:20.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:02:20.033 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-06-29T16:02:20.033 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-06-29T16:02:20.033 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-06-29T16:02:20.033 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 563048322 2022-06-29T16:02:20.034 INFO:teuthology.orchestra.run.smithi121.stdout: } 2022-06-29T16:02:20.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:02:20.034 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-06-29T16:02:20.034 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-06-29T16:02:20.035 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-06-29T16:02:20.035 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 563048322 2022-06-29T16:02:20.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:02:20.035 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.5896 172.21.15.121:0/563048322", 2022-06-29T16:02:20.036 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "172.21.15.121:0/563048322", 2022-06-29T16:02:20.036 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2022-06-29T16:02:20.036 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 20, 2022-06-29T16:02:20.036 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 53, 2022-06-29T16:02:20.037 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2022-06-29T16:02:20.037 INFO:teuthology.orchestra.run.smithi121.stdout: "blacklisted": false, 2022-06-29T16:02:20.037 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2022-06-29T16:02:20.037 INFO:teuthology.orchestra.run.smithi121.stdout:} 2022-06-29T16:02:20.039 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:02:20.039 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:02:20.040 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:02:20.047 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T16:02:20.049 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T16:02:20.050 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:02:20.050 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:02:28.515 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:02:28.518 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:02:28.522 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:02:28.524 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:02:28.527 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:02:58.624 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:02:58.628 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:02:58.631 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:02:58.634 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:02:58.636 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:03:14.796 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-06-29T16:03:14.797 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:03:14.819 DEBUG:teuthology.orchestra.run.smithi092:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:03:14.840 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:03:14.860 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:03:14.881 DEBUG:teuthology.orchestra.run.smithi197:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:03:14.903 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-06-29T16:03:14.903 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:03:14.904 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:03:14.904 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:03:14.912 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T16:03:14.913 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T16:03:14.914 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2022-06-29T16:03:14.914 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-06-29T16:03:14.915 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:03:14.929 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[12680]: fuse finished with error 0 and tester_r 0 2022-06-29T16:03:14.931 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:03:14.932 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:03:14.932 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:03:14.982 INFO:teuthology.orchestra.run.smithi121.stdout:ext2/ext3 2022-06-29T16:03:14.983 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-06-29T16:03:15.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:03:15.098 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:03:15.106 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:03:15.107 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:03:15.107 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:03:15.114 INFO:teuthology.orchestra.run.smithi156.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-06-29T16:03:15.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:03:15.116 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-06-29T16:03:15.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:03:15.117 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:03:15.163 INFO:teuthology.orchestra.run.smithi156.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-06-29T16:03:15.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:03:15.165 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:03:15.166 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:03:15.167 DEBUG:teuthology.orchestra.run.smithi197:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T16:03:15.174 INFO:teuthology.orchestra.run.smithi197.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-06-29T16:03:15.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:03:15.176 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-06-29T16:03:15.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:03:15.177 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T16:03:15.225 INFO:teuthology.orchestra.run.smithi197.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-06-29T16:03:15.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:03:15.228 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-06-29T16:03:15.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.335+0000 7f0e5bfff700 1 Processor -- start 2022-06-29T16:03:15.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.335+0000 7f0e5bfff700 1 -- start start 2022-06-29T16:03:15.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.335+0000 7f0e5bfff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e5c0b0610 0x7f0e5c0b24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:15.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.335+0000 7f0e5bfff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0e5c05bd20 0x7f0e5c05c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:15.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.335+0000 7f0e5bfff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e5c0b1de0 con 0x7f0e5c0b9c20 2022-06-29T16:03:15.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.335+0000 7f0e5bfff700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e5c0b1330 con 0x7f0e5c05bd20 2022-06-29T16:03:15.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.335+0000 7f0e5bfff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c146750 con 0x7f0e5c0b0610 2022-06-29T16:03:15.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.335+0000 7f0e5b7fe700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0e5c05bd20 0x7f0e5c05c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:15.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.335+0000 7f0e5a7fc700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e5c0b0610 0x7f0e5c0b24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:15.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.335+0000 7f0e5affd700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f0e5c0b9c20 0x7f0e5c146040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:60570/0 (socket says 172.21.15.60:60570) 2022-06-29T16:03:15.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.335+0000 7f0e5affd700 1 -- 172.21.15.60:0/3962729922 learned_addr learned my addr 172.21.15.60:0/3962729922 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:03:15.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.335+0000 7f0e59ffb700 1 -- 172.21.15.60:0/3962729922 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1624119790 0 0) 0x7f0e5c0b1de0 con 0x7f0e5c0b9c20 2022-06-29T16:03:15.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.335+0000 7f0e59ffb700 1 -- 172.21.15.60:0/3962729922 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e40003260 con 0x7f0e5c0b9c20 2022-06-29T16:03:15.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.335+0000 7f0e5a7fc700 1 -- 172.21.15.60:0/3962729922 >> v2:172.21.15.92:3300/0 conn(0x7f0e5c05bd20 msgr2=0x7f0e5c05c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:15.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.335+0000 7f0e5a7fc700 1 --2- 172.21.15.60:0/3962729922 >> v2:172.21.15.92:3300/0 conn(0x7f0e5c05bd20 0x7f0e5c05c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:15.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.335+0000 7f0e5a7fc700 1 -- 172.21.15.60:0/3962729922 >> v1:172.21.15.92:6789/0 conn(0x7f0e5c0b9c20 legacy=0x7f0e5c146040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:15.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.335+0000 7f0e5a7fc700 1 -- 172.21.15.60:0/3962729922 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e5c0ba000 con 0x7f0e5c0b0610 2022-06-29T16:03:15.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.335+0000 7f0e5b7fe700 1 --2- 172.21.15.60:0/3962729922 >> v2:172.21.15.92:3300/0 conn(0x7f0e5c05bd20 0x7f0e5c05c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:03:15.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.335+0000 7f0e5a7fc700 1 --2- 172.21.15.60:0/3962729922 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e5c0b0610 0x7f0e5c0b24b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f0e44002f70 tx=0x7f0e44006df0).ready entity=mon.2 client_cookie=2012c2395ecff3ac server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:15.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.335+0000 7f0e59ffb700 1 -- 172.21.15.60:0/3962729922 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e4400d4a0 con 0x7f0e5c0b0610 2022-06-29T16:03:15.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.335+0000 7f0e59ffb700 1 -- 172.21.15.60:0/3962729922 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e44000e90 con 0x7f0e5c0b0610 2022-06-29T16:03:15.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.335+0000 7f0e59ffb700 1 -- 172.21.15.60:0/3962729922 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e4400d7f0 con 0x7f0e5c0b0610 2022-06-29T16:03:15.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.339+0000 7f0e5bfff700 1 -- 172.21.15.60:0/3962729922 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e5c0b0610 msgr2=0x7f0e5c0b24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:15.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.339+0000 7f0e5bfff700 1 --2- 172.21.15.60:0/3962729922 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e5c0b0610 0x7f0e5c0b24b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f0e44002f70 tx=0x7f0e44006df0).stop 2022-06-29T16:03:15.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.339+0000 7f0e5bfff700 1 -- 172.21.15.60:0/3962729922 shutdown_connections 2022-06-29T16:03:15.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.339+0000 7f0e5bfff700 1 --2- 172.21.15.60:0/3962729922 >> v2:172.21.15.92:3300/0 conn(0x7f0e5c05bd20 0x7f0e5c05c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:15.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.339+0000 7f0e5bfff700 1 --2- 172.21.15.60:0/3962729922 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e5c0b0610 0x7f0e5c0b24b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:15.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.339+0000 7f0e5bfff700 1 -- 172.21.15.60:0/3962729922 >> 172.21.15.60:0/3962729922 conn(0x7f0e5c062cb0 msgr2=0x7f0e5c0b91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:15.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.339+0000 7f0e5bfff700 1 -- 172.21.15.60:0/3962729922 shutdown_connections 2022-06-29T16:03:15.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.339+0000 7f0e5bfff700 1 -- 172.21.15.60:0/3962729922 wait complete. 2022-06-29T16:03:15.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.339+0000 7f0e5bfff700 1 Processor -- start 2022-06-29T16:03:15.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.339+0000 7f0e5bfff700 1 -- start start 2022-06-29T16:03:15.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.339+0000 7f0e5bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e5c05bd20 0x7f0e5c0adc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:15.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.339+0000 7f0e5bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e5c0b0610 0x7f0e5c140960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:15.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.339+0000 7f0e5bfff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e5c0b9c20 0x7f0e5c0ae9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:15.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.339+0000 7f0e5bfff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c0b6740 con 0x7f0e5c05bd20 2022-06-29T16:03:15.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.339+0000 7f0e5bfff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c0b69e0 con 0x7f0e5c0b0610 2022-06-29T16:03:15.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.339+0000 7f0e5bfff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c146080 con 0x7f0e5c0b9c20 2022-06-29T16:03:15.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.339+0000 7f0e5affd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e5c05bd20 0x7f0e5c0adc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:15.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.339+0000 7f0e5a7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e5c0b0610 0x7f0e5c140960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:15.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.339+0000 7f0e5a7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e5c0b0610 0x7f0e5c140960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:54270/0 (socket says 172.21.15.60:54270) 2022-06-29T16:03:15.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.339+0000 7f0e5a7fc700 1 -- 172.21.15.60:0/467921501 learned_addr learned my addr 172.21.15.60:0/467921501 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:15.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.339+0000 7f0e5b7fe700 1 --2- 172.21.15.60:0/467921501 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e5c0b9c20 0x7f0e5c0ae9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:15.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.339+0000 7f0e5a7fc700 1 -- 172.21.15.60:0/467921501 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e5c0b9c20 msgr2=0x7f0e5c0ae9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:15.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.339+0000 7f0e5a7fc700 1 --2- 172.21.15.60:0/467921501 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e5c0b9c20 0x7f0e5c0ae9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:15.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.339+0000 7f0e5a7fc700 1 -- 172.21.15.60:0/467921501 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e5c05bd20 msgr2=0x7f0e5c0adc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:15.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.339+0000 7f0e5a7fc700 1 --2- 172.21.15.60:0/467921501 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e5c05bd20 0x7f0e5c0adc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:15.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.339+0000 7f0e5a7fc700 1 -- 172.21.15.60:0/467921501 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e54009470 con 0x7f0e5c0b0610 2022-06-29T16:03:15.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.339+0000 7f0e5affd700 1 --2- 172.21.15.60:0/467921501 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e5c05bd20 0x7f0e5c0adc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:03:15.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.339+0000 7f0e5a7fc700 1 --2- 172.21.15.60:0/467921501 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e5c0b0610 0x7f0e5c140960 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f0e44002f70 tx=0x7f0e44014000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:15.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.343+0000 7f0e53fff700 1 -- 172.21.15.60:0/467921501 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e44016070 con 0x7f0e5c0b0610 2022-06-29T16:03:15.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.343+0000 7f0e53fff700 1 -- 172.21.15.60:0/467921501 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e440138d0 con 0x7f0e5c0b0610 2022-06-29T16:03:15.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.343+0000 7f0e5bfff700 1 -- 172.21.15.60:0/467921501 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e4400b180 con 0x7f0e5c0b0610 2022-06-29T16:03:15.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.343+0000 7f0e53fff700 1 -- 172.21.15.60:0/467921501 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e44013c30 con 0x7f0e5c0b0610 2022-06-29T16:03:15.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.343+0000 7f0e5bfff700 1 -- 172.21.15.60:0/467921501 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e5c0b63c0 con 0x7f0e5c0b0610 2022-06-29T16:03:15.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.343+0000 7f0e5bfff700 1 -- 172.21.15.60:0/467921501 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e5c0573d0 con 0x7f0e5c0b0610 2022-06-29T16:03:15.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.343+0000 7f0e53fff700 1 -- 172.21.15.60:0/467921501 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0e44025030 con 0x7f0e5c0b0610 2022-06-29T16:03:15.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.343+0000 7f0e53fff700 1 --2- 172.21.15.60:0/467921501 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0e34053aa0 0x7f0e34055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:15.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.343+0000 7f0e53fff700 1 -- 172.21.15.60:0/467921501 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0e4406ce30 con 0x7f0e5c0b0610 2022-06-29T16:03:15.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.343+0000 7f0e5affd700 1 --2- 172.21.15.60:0/467921501 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0e34053aa0 0x7f0e34055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:15.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.347+0000 7f0e5affd700 1 --2- 172.21.15.60:0/467921501 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0e34053aa0 0x7f0e34055f60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f0e4c00d000 tx=0x7f0e4c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:15.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.355+0000 7f0e53fff700 1 -- 172.21.15.60:0/467921501 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0e44043d50 con 0x7f0e5c0b0610 2022-06-29T16:03:15.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.483+0000 7f0e5bfff700 1 -- 172.21.15.60:0/467921501 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f0e5c0b6b20 con 0x7f0e5c0b0610 2022-06-29T16:03:15.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.483+0000 7f0e53fff700 1 -- 172.21.15.60:0/467921501 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v53) v1 ==== 81+0+33647 (secure 0 0 0) 0x7f0e44023090 con 0x7f0e5c0b0610 2022-06-29T16:03:15.493 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:03:15.493 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:03:15.493 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 53, 2022-06-29T16:03:15.494 INFO:teuthology.orchestra.run.smithi060.stdout: "fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", 2022-06-29T16:03:15.494 INFO:teuthology.orchestra.run.smithi060.stdout: "created": "2022-06-29T15:58:07.965719+0000", 2022-06-29T16:03:15.494 INFO:teuthology.orchestra.run.smithi060.stdout: "modified": "2022-06-29T16:01:55.736061+0000", 2022-06-29T16:03:15.494 INFO:teuthology.orchestra.run.smithi060.stdout: "last_up_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:03:15.495 INFO:teuthology.orchestra.run.smithi060.stdout: "last_in_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:03:15.495 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-06-29T16:03:15.495 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_num": 5799936, 2022-06-29T16:03:15.495 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_set": [ 2022-06-29T16:03:15.496 INFO:teuthology.orchestra.run.smithi060.stdout: "pglog_hardlimit", 2022-06-29T16:03:15.496 INFO:teuthology.orchestra.run.smithi060.stdout: "purged_snapdirs", 2022-06-29T16:03:15.496 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_deletes", 2022-06-29T16:03:15.496 INFO:teuthology.orchestra.run.smithi060.stdout: "sortbitwise" 2022-06-29T16:03:15.497 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:03:15.497 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_version": 4, 2022-06-29T16:03:15.497 INFO:teuthology.orchestra.run.smithi060.stdout: "full_ratio": 0.94999998807907104, 2022-06-29T16:03:15.497 INFO:teuthology.orchestra.run.smithi060.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-06-29T16:03:15.497 INFO:teuthology.orchestra.run.smithi060.stdout: "nearfull_ratio": 0.85000002384185791, 2022-06-29T16:03:15.498 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_snapshot": "", 2022-06-29T16:03:15.498 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_max": 10, 2022-06-29T16:03:15.498 INFO:teuthology.orchestra.run.smithi060.stdout: "max_osd": 8, 2022-06-29T16:03:15.498 INFO:teuthology.orchestra.run.smithi060.stdout: "require_min_compat_client": "jewel", 2022-06-29T16:03:15.499 INFO:teuthology.orchestra.run.smithi060.stdout: "min_compat_client": "jewel", 2022-06-29T16:03:15.499 INFO:teuthology.orchestra.run.smithi060.stdout: "require_osd_release": "octopus", 2022-06-29T16:03:15.499 INFO:teuthology.orchestra.run.smithi060.stdout: "pools": [ 2022-06-29T16:03:15.499 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.500 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2022-06-29T16:03:15.500 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "device_health_metrics", 2022-06-29T16:03:15.500 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:10.961127+0000", 2022-06-29T16:03:15.500 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:03:15.500 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:03:15.501 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:03:15.501 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:03:15.501 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:03:15.501 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:03:15.502 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:03:15.502 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:03:15.502 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 1, 2022-06-29T16:03:15.503 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 1, 2022-06-29T16:03:15.503 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 1, 2022-06-29T16:03:15.503 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 1, 2022-06-29T16:03:15.503 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 1, 2022-06-29T16:03:15.504 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:03:15.504 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:03:15.504 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:03:15.504 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:03:15.504 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:03:15.505 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:03:15.505 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:03:15.505 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.505 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "15", 2022-06-29T16:03:15.506 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:03:15.506 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:03:15.506 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:03:15.506 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:03:15.506 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:03:15.507 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:03:15.507 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:03:15.508 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:03:15.508 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:03:15.508 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:03:15.508 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:03:15.509 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:03:15.509 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:03:15.509 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:03:15.509 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:03:15.510 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:03:15.510 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:03:15.510 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:03:15.510 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:03:15.510 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:03:15.511 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:03:15.511 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:03:15.511 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:03:15.511 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:03:15.512 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:03:15.512 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:03:15.512 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.512 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:03:15.512 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:03:15.513 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:03:15.513 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:03:15.513 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:03:15.513 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:03:15.514 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:03:15.514 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:03:15.514 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:03:15.514 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:03:15.515 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:03:15.515 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:03:15.515 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 1 2022-06-29T16:03:15.515 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.515 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:03:15.516 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_devicehealth": {} 2022-06-29T16:03:15.516 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.516 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.516 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.517 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 2, 2022-06-29T16:03:15.517 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "rbd", 2022-06-29T16:03:15.517 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:19.184276+0000", 2022-06-29T16:03:15.517 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:03:15.517 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:03:15.518 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:03:15.518 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:03:15.518 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:03:15.518 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:03:15.519 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:03:15.519 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:03:15.519 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:03:15.519 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:03:15.519 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:03:15.520 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:03:15.520 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:03:15.520 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:03:15.520 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:03:15.521 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:03:15.521 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:03:15.521 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:03:15.521 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:03:15.522 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:03:15.522 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.522 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "18", 2022-06-29T16:03:15.522 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:03:15.522 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:03:15.523 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:03:15.523 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:03:15.523 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:03:15.523 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:03:15.524 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:03:15.524 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:03:15.524 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:03:15.524 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:03:15.525 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:03:15.525 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:03:15.525 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:03:15.525 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:03:15.526 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:03:15.526 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:03:15.526 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:03:15.526 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:03:15.527 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:03:15.527 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:03:15.527 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:03:15.527 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:03:15.528 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:03:15.528 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:03:15.528 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:03:15.528 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:03:15.529 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.529 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:03:15.529 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:03:15.529 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:03:15.529 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:03:15.530 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:03:15.530 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:03:15.530 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:03:15.530 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:03:15.531 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:03:15.531 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:03:15.531 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:03:15.531 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:03:15.531 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:03:15.532 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd": {} 2022-06-29T16:03:15.532 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.532 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.532 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.533 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 9, 2022-06-29T16:03:15.533 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "cephfs_metadata", 2022-06-29T16:03:15.533 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T16:01:52.588508+0000", 2022-06-29T16:03:15.533 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:03:15.533 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:03:15.534 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:03:15.534 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:03:15.534 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:03:15.534 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:03:15.535 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:03:15.535 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:03:15.535 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:03:15.535 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:03:15.535 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:03:15.536 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:03:15.536 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:03:15.536 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:03:15.536 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:03:15.537 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:03:15.537 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:03:15.537 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:03:15.537 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:03:15.537 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:03:15.538 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.538 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "52", 2022-06-29T16:03:15.538 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:03:15.538 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:03:15.539 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:03:15.539 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:03:15.539 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:03:15.540 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:03:15.540 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:03:15.540 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:03:15.540 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:03:15.541 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:03:15.541 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:03:15.541 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:03:15.541 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:03:15.542 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:03:15.542 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:03:15.542 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:03:15.543 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:03:15.543 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:03:15.543 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:03:15.543 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:03:15.543 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:03:15.544 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:03:15.544 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:03:15.544 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:03:15.544 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:03:15.545 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:03:15.545 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.545 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:03:15.545 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:03:15.546 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:03:15.546 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:03:15.546 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:03:15.546 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:03:15.547 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:03:15.547 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:03:15.547 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:03:15.547 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:03:15.547 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:03:15.548 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:03:15.548 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_bias": 4, 2022-06-29T16:03:15.548 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 16, 2022-06-29T16:03:15.548 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_priority": 5 2022-06-29T16:03:15.549 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.549 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:03:15.549 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs": { 2022-06-29T16:03:15.549 INFO:teuthology.orchestra.run.smithi060.stdout: "metadata": "cephfs" 2022-06-29T16:03:15.550 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.550 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.550 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.550 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.551 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 10, 2022-06-29T16:03:15.551 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "cephfs_data", 2022-06-29T16:03:15.551 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T16:01:53.732456+0000", 2022-06-29T16:03:15.551 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:03:15.552 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:03:15.552 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:03:15.552 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:03:15.552 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:03:15.552 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:03:15.553 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:03:15.553 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:03:15.553 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:03:15.553 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:03:15.554 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:03:15.554 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:03:15.554 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:03:15.554 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:03:15.555 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:03:15.555 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:03:15.555 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:03:15.555 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:03:15.555 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:03:15.556 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:03:15.556 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.556 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "53", 2022-06-29T16:03:15.556 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:03:15.557 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:03:15.557 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:03:15.557 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:03:15.557 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:03:15.557 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:03:15.558 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:03:15.558 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:03:15.558 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:03:15.558 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:03:15.559 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:03:15.559 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:03:15.559 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:03:15.559 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:03:15.559 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:03:15.560 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:03:15.560 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:03:15.560 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:03:15.560 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:03:15.561 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:03:15.561 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:03:15.561 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:03:15.561 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:03:15.562 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:03:15.562 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:03:15.562 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:03:15.562 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.562 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:03:15.563 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:03:15.563 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:03:15.563 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:03:15.563 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:03:15.564 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:03:15.564 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:03:15.564 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:03:15.564 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:03:15.564 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:03:15.565 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:03:15.565 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:03:15.565 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:03:15.565 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs": { 2022-06-29T16:03:15.566 INFO:teuthology.orchestra.run.smithi060.stdout: "data": "cephfs" 2022-06-29T16:03:15.566 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.566 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.566 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.567 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:03:15.567 INFO:teuthology.orchestra.run.smithi060.stdout: "osds": [ 2022-06-29T16:03:15.567 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.567 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:03:15.567 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "11e7ab9e-fce2-470f-8195-ea4c3c52df22", 2022-06-29T16:03:15.568 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:03:15.568 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:03:15.568 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:03:15.568 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:03:15.569 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:03:15.569 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:03:15.569 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:03:15.569 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 51, 2022-06-29T16:03:15.569 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:03:15.570 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:03:15.570 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:03:15.570 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:15.570 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.571 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:15.571 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6816", 2022-06-29T16:03:15.571 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:03:15.571 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.571 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.572 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:15.572 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6825", 2022-06-29T16:03:15.572 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:03:15.572 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.573 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.573 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.573 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:03:15.573 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:15.573 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.574 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:15.574 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6826", 2022-06-29T16:03:15.574 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:03:15.574 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.575 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.575 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:15.575 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6827", 2022-06-29T16:03:15.575 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:03:15.576 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.576 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.576 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.576 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:03:15.576 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:15.577 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.577 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:15.577 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6830", 2022-06-29T16:03:15.577 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:03:15.578 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.578 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.578 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:15.578 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6831", 2022-06-29T16:03:15.579 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:03:15.579 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.579 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.579 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.579 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:03:15.580 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:15.580 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.580 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:15.580 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6828", 2022-06-29T16:03:15.580 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:03:15.581 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.581 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.581 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:15.581 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6829", 2022-06-29T16:03:15.582 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:03:15.582 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.582 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.582 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.582 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6825/12890", 2022-06-29T16:03:15.583 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6827/12890", 2022-06-29T16:03:15.583 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6831/12890", 2022-06-29T16:03:15.583 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6829/12890", 2022-06-29T16:03:15.583 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:03:15.584 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:03:15.584 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:03:15.584 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.584 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.584 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.585 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:03:15.585 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "ca43f136-6f69-4431-a157-fbe073fb585c", 2022-06-29T16:03:15.585 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:03:15.585 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:03:15.586 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:03:15.586 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:03:15.586 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:03:15.586 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:03:15.587 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:03:15.587 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 51, 2022-06-29T16:03:15.587 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:03:15.587 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:03:15.587 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:03:15.588 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:15.588 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.588 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:15.588 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6815", 2022-06-29T16:03:15.589 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:03:15.589 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.589 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.589 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:15.589 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6818", 2022-06-29T16:03:15.590 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:03:15.590 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.590 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.590 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.591 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:03:15.591 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:15.591 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.591 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:15.592 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6819", 2022-06-29T16:03:15.592 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:03:15.592 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.592 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.592 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:15.593 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6820", 2022-06-29T16:03:15.593 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:03:15.593 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.593 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.594 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.594 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:03:15.594 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:15.594 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.594 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:15.595 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6823", 2022-06-29T16:03:15.595 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:03:15.595 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.595 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.596 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:15.596 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6824", 2022-06-29T16:03:15.596 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:03:15.596 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.596 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.597 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.597 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:03:15.597 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:15.597 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.597 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:15.598 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6821", 2022-06-29T16:03:15.598 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:03:15.598 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.598 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.599 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:15.599 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6822", 2022-06-29T16:03:15.599 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:03:15.599 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.600 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.600 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.600 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6818/12892", 2022-06-29T16:03:15.600 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6820/12892", 2022-06-29T16:03:15.600 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6824/12892", 2022-06-29T16:03:15.601 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6822/12892", 2022-06-29T16:03:15.601 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:03:15.601 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:03:15.601 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:03:15.602 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.602 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.602 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.602 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:03:15.603 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "74fac45f-5f07-44ec-a9cb-2644c123fd4a", 2022-06-29T16:03:15.603 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:03:15.603 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:03:15.603 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:03:15.603 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:03:15.604 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:03:15.604 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:03:15.604 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:03:15.604 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 51, 2022-06-29T16:03:15.605 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:03:15.605 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:03:15.605 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:03:15.605 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:15.605 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.606 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:15.606 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6808", 2022-06-29T16:03:15.606 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:03:15.606 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.607 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.607 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:15.607 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6809", 2022-06-29T16:03:15.607 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:03:15.608 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.608 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.608 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.608 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:03:15.609 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:15.609 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.609 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:15.609 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6810", 2022-06-29T16:03:15.609 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:03:15.610 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.610 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.610 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:15.610 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6811", 2022-06-29T16:03:15.611 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:03:15.611 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.611 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.611 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.611 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:03:15.612 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:15.612 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.612 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:15.612 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6814", 2022-06-29T16:03:15.613 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:03:15.613 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.613 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.613 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:15.613 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6817", 2022-06-29T16:03:15.614 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:03:15.614 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.614 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.614 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.615 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:03:15.615 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:15.615 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.615 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:15.615 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6812", 2022-06-29T16:03:15.616 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:03:15.616 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.616 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.616 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:15.617 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6813", 2022-06-29T16:03:15.617 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:03:15.617 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.617 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.618 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.618 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6809/12891", 2022-06-29T16:03:15.618 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6811/12891", 2022-06-29T16:03:15.618 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6817/12891", 2022-06-29T16:03:15.619 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6813/12891", 2022-06-29T16:03:15.619 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:03:15.619 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:03:15.619 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:03:15.619 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.620 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.620 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.620 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:03:15.620 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "2581d351-973a-4b4e-85ca-3658f0068765", 2022-06-29T16:03:15.621 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:03:15.621 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:03:15.621 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:03:15.621 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:03:15.621 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:03:15.622 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:03:15.622 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:03:15.622 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 50, 2022-06-29T16:03:15.622 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:03:15.623 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:03:15.623 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:03:15.623 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:15.623 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.624 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:15.624 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6800", 2022-06-29T16:03:15.624 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:03:15.624 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.624 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.625 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:15.625 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6801", 2022-06-29T16:03:15.625 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:03:15.625 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.626 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.626 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.626 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:03:15.626 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:15.626 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.627 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:15.627 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6802", 2022-06-29T16:03:15.627 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:03:15.627 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.628 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.628 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:15.628 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6803", 2022-06-29T16:03:15.628 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:03:15.628 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.629 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.629 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.629 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:03:15.629 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:15.630 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.630 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:15.630 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6806", 2022-06-29T16:03:15.630 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:03:15.630 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.631 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.631 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:15.631 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6807", 2022-06-29T16:03:15.631 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:03:15.632 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.632 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.632 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.632 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:03:15.632 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:15.633 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.633 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:15.633 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6804", 2022-06-29T16:03:15.633 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:03:15.634 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.634 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.634 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:15.634 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6805", 2022-06-29T16:03:15.635 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:03:15.635 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.635 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.635 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.635 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6801/12889", 2022-06-29T16:03:15.636 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6803/12889", 2022-06-29T16:03:15.636 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6807/12889", 2022-06-29T16:03:15.636 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6805/12889", 2022-06-29T16:03:15.636 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:03:15.637 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:03:15.637 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:03:15.637 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.637 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.637 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.638 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:03:15.638 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "3d234db9-87cd-4fe9-814f-0240a287d8b4", 2022-06-29T16:03:15.638 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:03:15.638 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:03:15.639 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:03:15.639 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:03:15.639 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:03:15.639 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:03:15.640 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:03:15.640 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 51, 2022-06-29T16:03:15.640 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:03:15.640 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:03:15.640 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:03:15.641 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:15.641 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.641 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:15.641 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6808", 2022-06-29T16:03:15.642 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:03:15.642 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.642 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.642 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:15.643 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6810", 2022-06-29T16:03:15.643 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:03:15.643 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.643 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.644 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.644 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:03:15.644 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:15.644 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.645 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:15.645 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6812", 2022-06-29T16:03:15.645 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:03:15.645 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.646 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.646 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:15.646 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6814", 2022-06-29T16:03:15.646 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:03:15.647 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.647 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.647 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.647 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:03:15.647 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:15.648 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.648 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:15.648 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6823", 2022-06-29T16:03:15.648 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:03:15.649 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.649 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.649 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:15.649 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6825", 2022-06-29T16:03:15.649 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:03:15.650 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.650 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.650 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.650 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:03:15.651 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:15.651 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.651 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:15.651 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6817", 2022-06-29T16:03:15.651 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:03:15.652 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.652 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.652 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:15.652 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6820", 2022-06-29T16:03:15.653 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:03:15.653 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.653 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.653 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.653 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6810/13249", 2022-06-29T16:03:15.654 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6814/13249", 2022-06-29T16:03:15.654 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6825/13249", 2022-06-29T16:03:15.654 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6820/13249", 2022-06-29T16:03:15.654 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:03:15.655 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:03:15.655 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:03:15.655 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.655 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.655 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.656 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:03:15.656 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "e4bf1104-ff5c-438b-b608-8cee31c163f6", 2022-06-29T16:03:15.656 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:03:15.656 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:03:15.657 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:03:15.657 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:03:15.657 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:03:15.657 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:03:15.657 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:03:15.658 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 51, 2022-06-29T16:03:15.658 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:03:15.658 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:03:15.658 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:03:15.659 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:15.659 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.659 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:15.659 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6809", 2022-06-29T16:03:15.660 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:03:15.660 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.660 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.660 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:15.660 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6811", 2022-06-29T16:03:15.661 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:03:15.661 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.661 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.661 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.662 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:03:15.662 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:15.662 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.662 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:15.663 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6813", 2022-06-29T16:03:15.663 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:03:15.663 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.663 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.663 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:15.664 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6815", 2022-06-29T16:03:15.664 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:03:15.664 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.664 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.665 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.665 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:03:15.665 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:15.665 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.665 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:15.666 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6824", 2022-06-29T16:03:15.666 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:03:15.666 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.666 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.667 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:15.667 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6827", 2022-06-29T16:03:15.667 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:03:15.667 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.668 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.668 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.668 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:03:15.668 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:15.669 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.669 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:15.669 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6818", 2022-06-29T16:03:15.669 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:03:15.669 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.670 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.670 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:15.670 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6821", 2022-06-29T16:03:15.670 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:03:15.671 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.671 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.671 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.671 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6811/13248", 2022-06-29T16:03:15.672 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6815/13248", 2022-06-29T16:03:15.672 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6827/13248", 2022-06-29T16:03:15.672 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6821/13248", 2022-06-29T16:03:15.672 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:03:15.672 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:03:15.673 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:03:15.673 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.673 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.673 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.674 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:03:15.674 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "8564dd25-27ab-4203-b54e-af6d3aa05857", 2022-06-29T16:03:15.674 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:03:15.674 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:03:15.675 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:03:15.675 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:03:15.675 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:03:15.675 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:03:15.675 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:03:15.676 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 51, 2022-06-29T16:03:15.676 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:03:15.676 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:03:15.676 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:03:15.677 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:15.677 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.677 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:15.677 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6816", 2022-06-29T16:03:15.677 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:03:15.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.678 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.678 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:15.678 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6819", 2022-06-29T16:03:15.679 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:03:15.679 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.679 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.679 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.680 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:03:15.680 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:15.680 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.680 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:15.680 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6822", 2022-06-29T16:03:15.681 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:03:15.681 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.681 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.681 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:15.682 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6826", 2022-06-29T16:03:15.682 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:03:15.682 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.682 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.682 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.683 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:03:15.683 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:15.683 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.683 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:15.684 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6830", 2022-06-29T16:03:15.684 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:03:15.684 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.684 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.685 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:15.685 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6831", 2022-06-29T16:03:15.685 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:03:15.685 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.686 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.686 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.686 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:03:15.686 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:15.687 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.687 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:15.687 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6828", 2022-06-29T16:03:15.687 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:03:15.688 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.688 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.688 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:15.688 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6829", 2022-06-29T16:03:15.689 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:03:15.689 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.689 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.689 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.689 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6819/13250", 2022-06-29T16:03:15.690 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6826/13250", 2022-06-29T16:03:15.690 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6831/13250", 2022-06-29T16:03:15.690 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6829/13250", 2022-06-29T16:03:15.690 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:03:15.691 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:03:15.691 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:03:15.691 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.691 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.692 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.692 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:03:15.692 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "bd513a9c-3b19-4314-9897-d300b5c897bb", 2022-06-29T16:03:15.692 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:03:15.692 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:03:15.693 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:03:15.693 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:03:15.693 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:03:15.693 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:03:15.694 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:03:15.694 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 50, 2022-06-29T16:03:15.694 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:03:15.694 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:03:15.695 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:03:15.695 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:15.695 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.695 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:15.696 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6800", 2022-06-29T16:03:15.696 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:03:15.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.697 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:15.697 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6801", 2022-06-29T16:03:15.697 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:03:15.697 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.698 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.698 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.698 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:03:15.698 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:15.699 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.699 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:15.699 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6802", 2022-06-29T16:03:15.699 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:03:15.699 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.700 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.700 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:15.700 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6803", 2022-06-29T16:03:15.700 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:03:15.701 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.701 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.701 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.701 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:03:15.702 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:15.702 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.702 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:15.702 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6806", 2022-06-29T16:03:15.703 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:03:15.703 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.703 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.703 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:15.704 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6807", 2022-06-29T16:03:15.704 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:03:15.704 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.704 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.705 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.705 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:03:15.705 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:15.705 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.706 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:15.706 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6804", 2022-06-29T16:03:15.706 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:03:15.706 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.706 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.707 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:15.707 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6805", 2022-06-29T16:03:15.707 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:03:15.707 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.708 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.708 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.708 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6801/13247", 2022-06-29T16:03:15.708 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6803/13247", 2022-06-29T16:03:15.709 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6807/13247", 2022-06-29T16:03:15.709 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6805/13247", 2022-06-29T16:03:15.709 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:03:15.709 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:03:15.710 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:03:15.710 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:15.710 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.710 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:03:15.710 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_xinfo": [ 2022-06-29T16:03:15.711 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.711 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:03:15.711 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:03:15.711 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:03:15.712 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:03:15.712 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:03:15.712 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:03:15.712 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.891342+0000", 2022-06-29T16:03:15.713 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:03:15.713 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.713 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.713 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:03:15.713 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:03:15.714 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:03:15.714 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:03:15.714 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:03:15.714 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:03:15.715 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.900866+0000", 2022-06-29T16:03:15.715 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:03:15.715 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.715 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.716 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:03:15.716 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:03:15.716 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:03:15.716 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:03:15.717 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:03:15.717 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:03:15.717 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.912755+0000", 2022-06-29T16:03:15.717 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:03:15.717 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.718 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.718 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:03:15.718 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:03:15.718 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:03:15.719 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:03:15.719 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:03:15.719 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:03:15.719 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.864332+0000", 2022-06-29T16:03:15.720 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:03:15.720 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.720 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.720 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:03:15.720 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:03:15.721 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:03:15.721 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:03:15.721 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:03:15.721 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:03:15.722 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.863756+0000", 2022-06-29T16:03:15.722 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:03:15.722 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.722 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.723 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:03:15.723 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:03:15.723 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:03:15.723 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:03:15.724 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:03:15.724 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:03:15.724 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.869041+0000", 2022-06-29T16:03:15.724 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:03:15.724 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.725 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.725 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:03:15.725 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:03:15.725 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:03:15.726 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:03:15.726 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:03:15.726 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:03:15.726 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.895043+0000", 2022-06-29T16:03:15.727 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:03:15.727 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.727 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:15.727 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:03:15.728 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:03:15.728 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:03:15.728 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:03:15.728 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:03:15.728 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:03:15.729 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.889491+0000", 2022-06-29T16:03:15.729 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:03:15.729 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.729 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:03:15.730 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap": [], 2022-06-29T16:03:15.730 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap_items": [], 2022-06-29T16:03:15.730 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_temp": [], 2022-06-29T16:03:15.730 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_temp": [], 2022-06-29T16:03:15.731 INFO:teuthology.orchestra.run.smithi060.stdout: "blacklist": {}, 2022-06-29T16:03:15.731 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profiles": { 2022-06-29T16:03:15.731 INFO:teuthology.orchestra.run.smithi060.stdout: "default": { 2022-06-29T16:03:15.731 INFO:teuthology.orchestra.run.smithi060.stdout: "crush-failure-domain": "osd", 2022-06-29T16:03:15.731 INFO:teuthology.orchestra.run.smithi060.stdout: "k": "2", 2022-06-29T16:03:15.732 INFO:teuthology.orchestra.run.smithi060.stdout: "m": "1", 2022-06-29T16:03:15.732 INFO:teuthology.orchestra.run.smithi060.stdout: "plugin": "jerasure", 2022-06-29T16:03:15.732 INFO:teuthology.orchestra.run.smithi060.stdout: "ruleset-failure-domain": "osd", 2022-06-29T16:03:15.732 INFO:teuthology.orchestra.run.smithi060.stdout: "technique": "reed_sol_van" 2022-06-29T16:03:15.733 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:15.733 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:15.733 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps_queue": [], 2022-06-29T16:03:15.733 INFO:teuthology.orchestra.run.smithi060.stdout: "new_removed_snaps": [], 2022-06-29T16:03:15.733 INFO:teuthology.orchestra.run.smithi060.stdout: "new_purged_snaps": [], 2022-06-29T16:03:15.734 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_node_flags": {}, 2022-06-29T16:03:15.734 INFO:teuthology.orchestra.run.smithi060.stdout: "device_class_flags": {} 2022-06-29T16:03:15.734 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:03:15.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.487+0000 7f0e5bfff700 1 -- 172.21.15.60:0/467921501 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0e34053aa0 msgr2=0x7f0e34055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:15.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.487+0000 7f0e5bfff700 1 --2- 172.21.15.60:0/467921501 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0e34053aa0 0x7f0e34055f60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f0e4c00d000 tx=0x7f0e4c009040).stop 2022-06-29T16:03:15.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.487+0000 7f0e5bfff700 1 -- 172.21.15.60:0/467921501 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e5c0b0610 msgr2=0x7f0e5c140960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:15.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.487+0000 7f0e5bfff700 1 --2- 172.21.15.60:0/467921501 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e5c0b0610 0x7f0e5c140960 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f0e44002f70 tx=0x7f0e44014000).stop 2022-06-29T16:03:15.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.487+0000 7f0e5bfff700 1 -- 172.21.15.60:0/467921501 shutdown_connections 2022-06-29T16:03:15.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.487+0000 7f0e5bfff700 1 --2- 172.21.15.60:0/467921501 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0e34053aa0 0x7f0e34055f60 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:15.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.487+0000 7f0e5bfff700 1 --2- 172.21.15.60:0/467921501 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e5c0b9c20 0x7f0e5c0ae9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:15.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.487+0000 7f0e5bfff700 1 --2- 172.21.15.60:0/467921501 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e5c0b0610 0x7f0e5c140960 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:15.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.487+0000 7f0e5bfff700 1 --2- 172.21.15.60:0/467921501 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e5c05bd20 0x7f0e5c0adc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:15.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.487+0000 7f0e5bfff700 1 -- 172.21.15.60:0/467921501 >> 172.21.15.60:0/467921501 conn(0x7f0e5c062cb0 msgr2=0x7f0e5c1421e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:15.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.487+0000 7f0e5bfff700 1 -- 172.21.15.60:0/467921501 shutdown_connections 2022-06-29T16:03:15.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.487+0000 7f0e5bfff700 1 -- 172.21.15.60:0/467921501 wait complete. 2022-06-29T16:03:15.740 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:03:15.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.843+0000 7f8f3ffff700 1 Processor -- start 2022-06-29T16:03:15.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.843+0000 7f8f3ffff700 1 -- start start 2022-06-29T16:03:15.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.843+0000 7f8f3ffff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8f400b48b0 0x7f8f40142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:15.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.843+0000 7f8f3ffff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8f400b2c50 0x7f8f400b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:15.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.843+0000 7f8f3ffff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f400b0cf0 con 0x7f8f400b5eb0 2022-06-29T16:03:15.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.843+0000 7f8f3ffff700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f400b54e0 con 0x7f8f400b48b0 2022-06-29T16:03:15.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.843+0000 7f8f3ffff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f400b4d10 con 0x7f8f400b2c50 2022-06-29T16:03:15.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.847+0000 7f8f3e7fc700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8f400b48b0 0x7f8f40142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:15.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.847+0000 7f8f3f7fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8f400b2c50 0x7f8f400b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:15.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.847+0000 7f8f3effd700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f8f400b5eb0 0x7f8f400b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:60582/0 (socket says 172.21.15.60:60582) 2022-06-29T16:03:15.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.847+0000 7f8f3effd700 1 -- 172.21.15.60:0/2694166172 learned_addr learned my addr 172.21.15.60:0/2694166172 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:03:15.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.847+0000 7f8f3e7fc700 1 -- 172.21.15.60:0/2694166172 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8f400b2c50 msgr2=0x7f8f400b30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:15.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.847+0000 7f8f3e7fc700 1 --2- 172.21.15.60:0/2694166172 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8f400b2c50 0x7f8f400b30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:15.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.847+0000 7f8f3e7fc700 1 -- 172.21.15.60:0/2694166172 >> v1:172.21.15.92:6789/0 conn(0x7f8f400b5eb0 legacy=0x7f8f400b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:15.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.847+0000 7f8f3e7fc700 1 -- 172.21.15.60:0/2694166172 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f400b6290 con 0x7f8f400b48b0 2022-06-29T16:03:15.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.847+0000 7f8f3e7fc700 1 --2- 172.21.15.60:0/2694166172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f400b48b0 0x7f8f40142a20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f8f280032f0 tx=0x7f8f280091d0).ready entity=mon.1 client_cookie=2246dea42c6c77d server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:15.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.847+0000 7f8f3dffb700 1 -- 172.21.15.60:0/2694166172 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f2800b6c0 con 0x7f8f400b48b0 2022-06-29T16:03:15.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.847+0000 7f8f3dffb700 1 -- 172.21.15.60:0/2694166172 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f2800b820 con 0x7f8f400b48b0 2022-06-29T16:03:15.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.847+0000 7f8f3dffb700 1 -- 172.21.15.60:0/2694166172 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f2800b990 con 0x7f8f400b48b0 2022-06-29T16:03:15.861 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.847+0000 7f8f3ffff700 1 -- 172.21.15.60:0/2694166172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f400b48b0 msgr2=0x7f8f40142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:15.861 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.847+0000 7f8f3ffff700 1 --2- 172.21.15.60:0/2694166172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f400b48b0 0x7f8f40142a20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f8f280032f0 tx=0x7f8f280091d0).stop 2022-06-29T16:03:15.861 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.847+0000 7f8f3ffff700 1 -- 172.21.15.60:0/2694166172 shutdown_connections 2022-06-29T16:03:15.861 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.847+0000 7f8f3ffff700 1 --2- 172.21.15.60:0/2694166172 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8f400b2c50 0x7f8f400b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:15.862 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.847+0000 7f8f3ffff700 1 --2- 172.21.15.60:0/2694166172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f400b48b0 0x7f8f40142a20 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:15.862 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.847+0000 7f8f3ffff700 1 -- 172.21.15.60:0/2694166172 >> 172.21.15.60:0/2694166172 conn(0x7f8f40062090 msgr2=0x7f8f40062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:15.862 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.847+0000 7f8f3ffff700 1 -- 172.21.15.60:0/2694166172 shutdown_connections 2022-06-29T16:03:15.863 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.847+0000 7f8f3ffff700 1 -- 172.21.15.60:0/2694166172 wait complete. 2022-06-29T16:03:15.863 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.851+0000 7f8f3ffff700 1 Processor -- start 2022-06-29T16:03:15.863 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.851+0000 7f8f3ffff700 1 -- start start 2022-06-29T16:03:15.863 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.851+0000 7f8f3ffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f400b2c50 0x7f8f40140550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:15.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.851+0000 7f8f3ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8f400b48b0 0x7f8f4013e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:15.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.851+0000 7f8f3ffff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8f400b5eb0 0x7f8f4013e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:15.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.851+0000 7f8f3ffff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f400aeb70 con 0x7f8f400b48b0 2022-06-29T16:03:15.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.851+0000 7f8f3ffff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f400aef60 con 0x7f8f400b2c50 2022-06-29T16:03:15.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.851+0000 7f8f3ffff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f400af3c0 con 0x7f8f400b5eb0 2022-06-29T16:03:15.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.851+0000 7f8f3e7fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8f400b48b0 0x7f8f4013e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:15.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.851+0000 7f8f3effd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f400b2c50 0x7f8f40140550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:15.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.851+0000 7f8f3effd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f400b2c50 0x7f8f40140550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:54284/0 (socket says 172.21.15.60:54284) 2022-06-29T16:03:15.866 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.851+0000 7f8f3e7fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8f400b48b0 0x7f8f4013e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48258/0 (socket says 172.21.15.60:48258) 2022-06-29T16:03:15.866 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.851+0000 7f8f3effd700 1 -- 172.21.15.60:0/374098835 learned_addr learned my addr 172.21.15.60:0/374098835 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:15.866 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.851+0000 7f8f3f7fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8f400b5eb0 0x7f8f4013e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:15.866 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.851+0000 7f8f3effd700 1 -- 172.21.15.60:0/374098835 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8f400b5eb0 msgr2=0x7f8f4013e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:15.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.851+0000 7f8f3effd700 1 --2- 172.21.15.60:0/374098835 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8f400b5eb0 0x7f8f4013e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:15.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.851+0000 7f8f3effd700 1 -- 172.21.15.60:0/374098835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8f400b48b0 msgr2=0x7f8f4013e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:15.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.851+0000 7f8f3effd700 1 --2- 172.21.15.60:0/374098835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8f400b48b0 0x7f8f4013e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:15.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.851+0000 7f8f3effd700 1 -- 172.21.15.60:0/374098835 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f28009470 con 0x7f8f400b2c50 2022-06-29T16:03:15.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.851+0000 7f8f3e7fc700 1 --2- 172.21.15.60:0/374098835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8f400b48b0 0x7f8f4013e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:03:15.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.851+0000 7f8f3effd700 1 --2- 172.21.15.60:0/374098835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f400b2c50 0x7f8f40140550 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f8f40062f00 tx=0x7f8f38007c60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:15.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.851+0000 7f8f37fff700 1 -- 172.21.15.60:0/374098835 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f38007110 con 0x7f8f400b2c50 2022-06-29T16:03:15.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.851+0000 7f8f37fff700 1 -- 172.21.15.60:0/374098835 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f38007270 con 0x7f8f400b2c50 2022-06-29T16:03:15.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.851+0000 7f8f3ffff700 1 -- 172.21.15.60:0/374098835 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f400ae8d0 con 0x7f8f400b2c50 2022-06-29T16:03:15.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.851+0000 7f8f37fff700 1 -- 172.21.15.60:0/374098835 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f38005030 con 0x7f8f400b2c50 2022-06-29T16:03:15.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.851+0000 7f8f3ffff700 1 -- 172.21.15.60:0/374098835 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f400ade70 con 0x7f8f400b2c50 2022-06-29T16:03:15.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.855+0000 7f8f3ffff700 1 -- 172.21.15.60:0/374098835 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f400573d0 con 0x7f8f400b2c50 2022-06-29T16:03:15.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.855+0000 7f8f37fff700 1 -- 172.21.15.60:0/374098835 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8f3801b070 con 0x7f8f400b2c50 2022-06-29T16:03:15.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.855+0000 7f8f37fff700 1 --2- 172.21.15.60:0/374098835 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8f24053aa0 0x7f8f24055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:15.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.855+0000 7f8f37fff700 1 -- 172.21.15.60:0/374098835 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8f3806b2c0 con 0x7f8f400b2c50 2022-06-29T16:03:15.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.855+0000 7f8f3e7fc700 1 --2- 172.21.15.60:0/374098835 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8f24053aa0 0x7f8f24055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:15.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.859+0000 7f8f3e7fc700 1 --2- 172.21.15.60:0/374098835 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8f24053aa0 0x7f8f24055f60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f8f2801b000 tx=0x7f8f28016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:15.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:15.867+0000 7f8f37fff700 1 -- 172.21.15.60:0/374098835 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8f38042050 con 0x7f8f400b2c50 2022-06-29T16:03:16.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.031+0000 7f8f3ffff700 1 -- 172.21.15.60:0/374098835 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f400b0cf0 con 0x7f8f400b2c50 2022-06-29T16:03:16.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.031+0000 7f8f37fff700 1 -- 172.21.15.60:0/374098835 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+2139 (secure 0 0 0) 0x7f8f400b0cf0 con 0x7f8f400b2c50 2022-06-29T16:03:16.039 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:03:16.039 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":20,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5568,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/4229695218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":4229695218},{"type":"v1","addr":"172.21.15.60:6833","nonce":4229695218}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:01:54.732520+0000","modified":"2022-06-29T16:01:55.735497+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5653":{"gid":5653,"name":"b","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/1602880694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1602880694},{"type":"v1","addr":"172.21.15.92:6835","nonce":1602880694}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-06-29T16:03:16.042 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.035+0000 7f8f3ffff700 1 -- 172.21.15.60:0/374098835 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8f24053aa0 msgr2=0x7f8f24055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:16.042 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.035+0000 7f8f3ffff700 1 --2- 172.21.15.60:0/374098835 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8f24053aa0 0x7f8f24055f60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f8f2801b000 tx=0x7f8f28016000).stop 2022-06-29T16:03:16.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.035+0000 7f8f3ffff700 1 -- 172.21.15.60:0/374098835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f400b2c50 msgr2=0x7f8f40140550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:16.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.035+0000 7f8f3ffff700 1 --2- 172.21.15.60:0/374098835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f400b2c50 0x7f8f40140550 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f8f40062f00 tx=0x7f8f38007c60).stop 2022-06-29T16:03:16.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.035+0000 7f8f3ffff700 1 -- 172.21.15.60:0/374098835 shutdown_connections 2022-06-29T16:03:16.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.035+0000 7f8f3ffff700 1 --2- 172.21.15.60:0/374098835 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8f24053aa0 0x7f8f24055f60 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:16.044 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.035+0000 7f8f3ffff700 1 --2- 172.21.15.60:0/374098835 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8f400b5eb0 0x7f8f4013e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:16.044 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.035+0000 7f8f3ffff700 1 --2- 172.21.15.60:0/374098835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8f400b48b0 0x7f8f4013e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:16.044 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.035+0000 7f8f3ffff700 1 --2- 172.21.15.60:0/374098835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f400b2c50 0x7f8f40140550 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:16.044 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.035+0000 7f8f3ffff700 1 -- 172.21.15.60:0/374098835 >> 172.21.15.60:0/374098835 conn(0x7f8f40062090 msgr2=0x7f8f400b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:16.044 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.035+0000 7f8f3ffff700 1 -- 172.21.15.60:0/374098835 shutdown_connections 2022-06-29T16:03:16.045 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.035+0000 7f8f3ffff700 1 -- 172.21.15.60:0/374098835 wait complete. 2022-06-29T16:03:16.046 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 20 2022-06-29T16:03:16.058 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-06-29T16:03:16.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.151+0000 7f5978aa5700 1 Processor -- start 2022-06-29T16:03:16.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.151+0000 7f5978aa5700 1 -- start start 2022-06-29T16:03:16.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.155+0000 7f5978aa5700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f59740ae7c0 0x7f5974141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:16.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.155+0000 7f5978aa5700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f59740b46c0 0x7f59740b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:16.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.155+0000 7f5978aa5700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5974143a40 con 0x7f59740b4010 2022-06-29T16:03:16.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.155+0000 7f5978aa5700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59740aee60 con 0x7f59740ae7c0 2022-06-29T16:03:16.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.155+0000 7f5978aa5700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59740af100 con 0x7f59740b46c0 2022-06-29T16:03:16.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.155+0000 7f597259c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f59740ae7c0 0x7f5974141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:16.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.155+0000 7f597259c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f59740ae7c0 0x7f5974141fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48264/0 (socket says 172.21.15.60:48264) 2022-06-29T16:03:16.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.155+0000 7f597259c700 1 -- 172.21.15.60:0/464908937 learned_addr learned my addr 172.21.15.60:0/464908937 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:16.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.155+0000 7f5972d9d700 1 --2- 172.21.15.60:0/464908937 >> v2:172.21.15.92:3300/0 conn(0x7f59740b46c0 0x7f59740b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:16.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.155+0000 7f597259c700 1 -- 172.21.15.60:0/464908937 >> v2:172.21.15.92:3300/0 conn(0x7f59740b46c0 msgr2=0x7f59740b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:16.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.155+0000 7f597259c700 1 --2- 172.21.15.60:0/464908937 >> v2:172.21.15.92:3300/0 conn(0x7f59740b46c0 0x7f59740b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:16.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.155+0000 7f597259c700 1 -- 172.21.15.60:0/464908937 >> v1:172.21.15.92:6789/0 conn(0x7f59740b4010 legacy=0x7f5974142490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:16.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.155+0000 7f597259c700 1 -- 172.21.15.60:0/464908937 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59740aeba0 con 0x7f59740ae7c0 2022-06-29T16:03:16.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.155+0000 7f597259c700 1 --2- 172.21.15.60:0/464908937 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59740ae7c0 0x7f5974141fa0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7f5964002f70 tx=0x7f5964006eb0).ready entity=mon.0 client_cookie=3bbd6e193419a073 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:16.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.155+0000 7f597159a700 1 -- 172.21.15.60:0/464908937 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f596400d5c0 con 0x7f59740ae7c0 2022-06-29T16:03:16.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.155+0000 7f597159a700 1 -- 172.21.15.60:0/464908937 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5964000e90 con 0x7f59740ae7c0 2022-06-29T16:03:16.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.155+0000 7f597159a700 1 -- 172.21.15.60:0/464908937 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f596400d950 con 0x7f59740ae7c0 2022-06-29T16:03:16.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.155+0000 7f5978aa5700 1 -- 172.21.15.60:0/464908937 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59740ae7c0 msgr2=0x7f5974141fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:16.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.155+0000 7f5978aa5700 1 --2- 172.21.15.60:0/464908937 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59740ae7c0 0x7f5974141fa0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7f5964002f70 tx=0x7f5964006eb0).stop 2022-06-29T16:03:16.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.155+0000 7f5978aa5700 1 -- 172.21.15.60:0/464908937 shutdown_connections 2022-06-29T16:03:16.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.155+0000 7f5978aa5700 1 --2- 172.21.15.60:0/464908937 >> v2:172.21.15.92:3300/0 conn(0x7f59740b46c0 0x7f59740b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:16.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.155+0000 7f5978aa5700 1 --2- 172.21.15.60:0/464908937 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59740ae7c0 0x7f5974141fa0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:16.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.155+0000 7f5978aa5700 1 -- 172.21.15.60:0/464908937 >> 172.21.15.60:0/464908937 conn(0x7f59740b9340 msgr2=0x7f59740b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:16.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.155+0000 7f5978aa5700 1 -- 172.21.15.60:0/464908937 shutdown_connections 2022-06-29T16:03:16.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.155+0000 7f5978aa5700 1 -- 172.21.15.60:0/464908937 wait complete. 2022-06-29T16:03:16.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.159+0000 7f5978aa5700 1 Processor -- start 2022-06-29T16:03:16.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.159+0000 7f5978aa5700 1 -- start start 2022-06-29T16:03:16.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.159+0000 7f5978aa5700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59740ae7c0 0x7f59740b7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:16.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.159+0000 7f5978aa5700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59740b4010 0x7f5974152690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:16.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.159+0000 7f5978aa5700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59740b46c0 0x7f5974152b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:16.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.159+0000 7f5978aa5700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59741129a0 con 0x7f59740b4010 2022-06-29T16:03:16.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.159+0000 7f5978aa5700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5974153070 con 0x7f59740ae7c0 2022-06-29T16:03:16.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.159+0000 7f5978aa5700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59740b1e40 con 0x7f59740b46c0 2022-06-29T16:03:16.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.159+0000 7f5971d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59740b4010 0x7f5974152690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:16.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.159+0000 7f597259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59740ae7c0 0x7f59740b7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:16.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.159+0000 7f5971d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59740b4010 0x7f5974152690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48270/0 (socket says 172.21.15.60:48270) 2022-06-29T16:03:16.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.159+0000 7f5971d9b700 1 -- 172.21.15.60:0/1191339949 learned_addr learned my addr 172.21.15.60:0/1191339949 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:16.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.159+0000 7f5972d9d700 1 --2- 172.21.15.60:0/1191339949 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59740b46c0 0x7f5974152b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:16.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.163+0000 7f5971d9b700 1 -- 172.21.15.60:0/1191339949 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59740b46c0 msgr2=0x7f5974152b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:16.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.163+0000 7f5971d9b700 1 --2- 172.21.15.60:0/1191339949 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59740b46c0 0x7f5974152b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:16.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.163+0000 7f5971d9b700 1 -- 172.21.15.60:0/1191339949 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59740ae7c0 msgr2=0x7f59740b7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:16.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.163+0000 7f5971d9b700 1 --2- 172.21.15.60:0/1191339949 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59740ae7c0 0x7f59740b7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:16.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.163+0000 7f5971d9b700 1 -- 172.21.15.60:0/1191339949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f596400b1e0 con 0x7f59740b4010 2022-06-29T16:03:16.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.163+0000 7f5971d9b700 1 --2- 172.21.15.60:0/1191339949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59740b4010 0x7f5974152690 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0x7f5974153200 tx=0x7f595c007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:16.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.163+0000 7f596b7fe700 1 -- 172.21.15.60:0/1191339949 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f595c007030 con 0x7f59740b4010 2022-06-29T16:03:16.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.163+0000 7f596b7fe700 1 -- 172.21.15.60:0/1191339949 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f595c007190 con 0x7f59740b4010 2022-06-29T16:03:16.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.163+0000 7f5978aa5700 1 -- 172.21.15.60:0/1191339949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5974112bd0 con 0x7f59740b4010 2022-06-29T16:03:16.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.163+0000 7f596b7fe700 1 -- 172.21.15.60:0/1191339949 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f595c006510 con 0x7f59740b4010 2022-06-29T16:03:16.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.163+0000 7f5978aa5700 1 -- 172.21.15.60:0/1191339949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5974152110 con 0x7f59740b4010 2022-06-29T16:03:16.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.163+0000 7f5978aa5700 1 -- 172.21.15.60:0/1191339949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59740b1a60 con 0x7f59740b4010 2022-06-29T16:03:16.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.163+0000 7f596b7fe700 1 -- 172.21.15.60:0/1191339949 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f595c010030 con 0x7f59740b4010 2022-06-29T16:03:16.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.163+0000 7f596b7fe700 1 --2- 172.21.15.60:0/1191339949 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5958053b00 0x7f5958055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:16.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.167+0000 7f596b7fe700 1 -- 172.21.15.60:0/1191339949 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f595c06ba40 con 0x7f59740b4010 2022-06-29T16:03:16.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.167+0000 7f597259c700 1 --2- 172.21.15.60:0/1191339949 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5958053b00 0x7f5958055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:16.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.167+0000 7f597259c700 1 --2- 172.21.15.60:0/1191339949 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5958053b00 0x7f5958055fc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f596401c000 tx=0x7f5964015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:16.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.171+0000 7f596b7fe700 1 -- 172.21.15.60:0/1191339949 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f595c042960 con 0x7f59740b4010 2022-06-29T16:03:16.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:16.331+0000 7f5978aa5700 1 -- 172.21.15.60:0/1191339949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f59740573d0 con 0x7f59740b4010 2022-06-29T16:03:17.060 INFO:tasks.ceph.mon.a.smithi060.stderr:2022-06-29T16:03:17.055+0000 7fcc1a376700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-06-29T16:03:17.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.067+0000 7f596b7fe700 1 -- 172.21.15.60:0/1191339949 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v21) v1 ==== 161+0+0 (secure 0 0 0) 0x7f595c045f80 con 0x7f59740b4010 2022-06-29T16:03:17.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.071+0000 7f5978aa5700 1 -- 172.21.15.60:0/1191339949 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5958053b00 msgr2=0x7f5958055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:17.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.071+0000 7f5978aa5700 1 --2- 172.21.15.60:0/1191339949 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5958053b00 0x7f5958055fc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f596401c000 tx=0x7f5964015000).stop 2022-06-29T16:03:17.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.071+0000 7f5978aa5700 1 -- 172.21.15.60:0/1191339949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59740b4010 msgr2=0x7f5974152690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:17.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.071+0000 7f5978aa5700 1 --2- 172.21.15.60:0/1191339949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59740b4010 0x7f5974152690 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0x7f5974153200 tx=0x7f595c007c10).stop 2022-06-29T16:03:17.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.071+0000 7f5978aa5700 1 -- 172.21.15.60:0/1191339949 shutdown_connections 2022-06-29T16:03:17.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.071+0000 7f5978aa5700 1 --2- 172.21.15.60:0/1191339949 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5958053b00 0x7f5958055fc0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:17.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.071+0000 7f5978aa5700 1 --2- 172.21.15.60:0/1191339949 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59740b46c0 0x7f5974152b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:17.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.071+0000 7f5978aa5700 1 --2- 172.21.15.60:0/1191339949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59740b4010 0x7f5974152690 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:17.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.071+0000 7f5978aa5700 1 --2- 172.21.15.60:0/1191339949 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59740ae7c0 0x7f59740b7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:17.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.071+0000 7f5978aa5700 1 -- 172.21.15.60:0/1191339949 >> 172.21.15.60:0/1191339949 conn(0x7f59740b9340 msgr2=0x7f597405cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:17.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.071+0000 7f5978aa5700 1 -- 172.21.15.60:0/1191339949 shutdown_connections 2022-06-29T16:03:17.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.071+0000 7f5978aa5700 1 -- 172.21.15.60:0/1191339949 wait complete. 2022-06-29T16:03:17.081 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-06-29T16:03:17.095 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:03:17.108 INFO:tasks.ceph.mds.b.smithi092.stdout:starting mds.b at 2022-06-29T16:03:17.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.199+0000 7f6985b83700 1 Processor -- start 2022-06-29T16:03:17.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.199+0000 7f6985b83700 1 -- start start 2022-06-29T16:03:17.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.199+0000 7f6985b83700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f69800b5950 0x7f6980067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:17.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.199+0000 7f6985b83700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6980063ad0 0x7f698014af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:17.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.199+0000 7f6985b83700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f698005bd20 0x7f698005c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:17.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.199+0000 7f6985b83700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69800acba0 con 0x7f698005bd20 2022-06-29T16:03:17.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.199+0000 7f6985b83700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6980067220 con 0x7f69800b5950 2022-06-29T16:03:17.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.199+0000 7f6985b83700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69800b5d30 con 0x7f6980063ad0 2022-06-29T16:03:17.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.199+0000 7f697ffff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f698005bd20 0x7f698005c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:17.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.199+0000 7f697ffff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f698005bd20 0x7f698005c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48280/0 (socket says 172.21.15.60:48280) 2022-06-29T16:03:17.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.199+0000 7f697ffff700 1 -- 172.21.15.60:0/1307927388 learned_addr learned my addr 172.21.15.60:0/1307927388 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:17.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.199+0000 7f697f7fe700 1 --2- 172.21.15.60:0/1307927388 >> v2:172.21.15.92:3300/0 conn(0x7f69800b5950 0x7f6980067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:17.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.199+0000 7f697effd700 1 --2- 172.21.15.60:0/1307927388 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6980063ad0 0x7f698014af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:17.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.199+0000 7f697ffff700 1 -- 172.21.15.60:0/1307927388 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6980063ad0 msgr2=0x7f698014af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:17.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.199+0000 7f697ffff700 1 --2- 172.21.15.60:0/1307927388 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6980063ad0 0x7f698014af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:17.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.199+0000 7f697ffff700 1 -- 172.21.15.60:0/1307927388 >> v2:172.21.15.92:3300/0 conn(0x7f69800b5950 msgr2=0x7f6980067680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:17.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.199+0000 7f697ffff700 1 --2- 172.21.15.60:0/1307927388 >> v2:172.21.15.92:3300/0 conn(0x7f69800b5950 0x7f6980067680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:17.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.199+0000 7f697ffff700 1 -- 172.21.15.60:0/1307927388 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69800ace00 con 0x7f698005bd20 2022-06-29T16:03:17.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.199+0000 7f697effd700 1 --2- 172.21.15.60:0/1307927388 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6980063ad0 0x7f698014af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:03:17.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.199+0000 7f697ffff700 1 --2- 172.21.15.60:0/1307927388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f698005bd20 0x7f698005c1b0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7f6970002f70 tx=0x7f6970006eb0).ready entity=mon.0 client_cookie=9b32b9de51c9008e server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:17.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.199+0000 7f697e7fc700 1 -- 172.21.15.60:0/1307927388 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f697000d570 con 0x7f698005bd20 2022-06-29T16:03:17.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.199+0000 7f697e7fc700 1 -- 172.21.15.60:0/1307927388 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6970000e90 con 0x7f698005bd20 2022-06-29T16:03:17.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.199+0000 7f697e7fc700 1 -- 172.21.15.60:0/1307927388 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f697000d900 con 0x7f698005bd20 2022-06-29T16:03:17.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.199+0000 7f6985b83700 1 -- 172.21.15.60:0/1307927388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f698005bd20 msgr2=0x7f698005c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:17.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.199+0000 7f6985b83700 1 --2- 172.21.15.60:0/1307927388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f698005bd20 0x7f698005c1b0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7f6970002f70 tx=0x7f6970006eb0).stop 2022-06-29T16:03:17.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.203+0000 7f6985b83700 1 -- 172.21.15.60:0/1307927388 shutdown_connections 2022-06-29T16:03:17.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.203+0000 7f6985b83700 1 --2- 172.21.15.60:0/1307927388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f698005bd20 0x7f698005c1b0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:17.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.203+0000 7f6985b83700 1 --2- 172.21.15.60:0/1307927388 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6980063ad0 0x7f698014af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:17.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.203+0000 7f6985b83700 1 --2- 172.21.15.60:0/1307927388 >> v2:172.21.15.92:3300/0 conn(0x7f69800b5950 0x7f6980067680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:17.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.203+0000 7f6985b83700 1 -- 172.21.15.60:0/1307927388 >> 172.21.15.60:0/1307927388 conn(0x7f69800632c0 msgr2=0x7f69800b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:17.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.203+0000 7f6985b83700 1 -- 172.21.15.60:0/1307927388 shutdown_connections 2022-06-29T16:03:17.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.203+0000 7f6985b83700 1 -- 172.21.15.60:0/1307927388 wait complete. 2022-06-29T16:03:17.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.203+0000 7f6985b83700 1 Processor -- start 2022-06-29T16:03:17.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.203+0000 7f6985b83700 1 -- start start 2022-06-29T16:03:17.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.203+0000 7f6985b83700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f698005bd20 0x7f69800af2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:17.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.203+0000 7f6985b83700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6980063ad0 0x7f6980159660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:17.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.203+0000 7f6985b83700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69800b5950 0x7f6980148580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:17.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.203+0000 7f6985b83700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6980159f90 con 0x7f69800b5950 2022-06-29T16:03:17.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.203+0000 7f6985b83700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69800ad630 con 0x7f6980063ad0 2022-06-29T16:03:17.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.203+0000 7f6985b83700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69800ad8d0 con 0x7f698005bd20 2022-06-29T16:03:17.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.203+0000 7f697ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69800b5950 0x7f6980148580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:17.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.203+0000 7f697ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69800b5950 0x7f6980148580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48286/0 (socket says 172.21.15.60:48286) 2022-06-29T16:03:17.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.203+0000 7f697ffff700 1 -- 172.21.15.60:0/94212143 learned_addr learned my addr 172.21.15.60:0/94212143 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:17.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.203+0000 7f697effd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6980063ad0 0x7f6980159660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:17.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.203+0000 7f697f7fe700 1 --2- 172.21.15.60:0/94212143 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f698005bd20 0x7f69800af2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:17.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.203+0000 7f697ffff700 1 -- 172.21.15.60:0/94212143 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f698005bd20 msgr2=0x7f69800af2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:17.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.203+0000 7f697ffff700 1 --2- 172.21.15.60:0/94212143 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f698005bd20 0x7f69800af2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:17.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.203+0000 7f697ffff700 1 -- 172.21.15.60:0/94212143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6980063ad0 msgr2=0x7f6980159660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:17.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.203+0000 7f697ffff700 1 --2- 172.21.15.60:0/94212143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6980063ad0 0x7f6980159660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:17.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.203+0000 7f697ffff700 1 -- 172.21.15.60:0/94212143 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f697000b1e0 con 0x7f69800b5950 2022-06-29T16:03:17.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.203+0000 7f697ffff700 1 --2- 172.21.15.60:0/94212143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69800b5950 0x7f6980148580 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0x7f6970002f70 tx=0x7f6970015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:17.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.207+0000 7f697cff9700 1 -- 172.21.15.60:0/94212143 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f697000de20 con 0x7f69800b5950 2022-06-29T16:03:17.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.207+0000 7f697cff9700 1 -- 172.21.15.60:0/94212143 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6970006b40 con 0x7f69800b5950 2022-06-29T16:03:17.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.207+0000 7f697cff9700 1 -- 172.21.15.60:0/94212143 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6970011d70 con 0x7f69800b5950 2022-06-29T16:03:17.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.207+0000 7f6985b83700 1 -- 172.21.15.60:0/94212143 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6980156ef0 con 0x7f69800b5950 2022-06-29T16:03:17.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.207+0000 7f6985b83700 1 -- 172.21.15.60:0/94212143 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69800ade30 con 0x7f69800b5950 2022-06-29T16:03:17.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.207+0000 7f697cff9700 1 -- 172.21.15.60:0/94212143 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6970025030 con 0x7f69800b5950 2022-06-29T16:03:17.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.207+0000 7f6985b83700 1 -- 172.21.15.60:0/94212143 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69800ada90 con 0x7f69800b5950 2022-06-29T16:03:17.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.207+0000 7f697cff9700 1 --2- 172.21.15.60:0/94212143 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6964053a70 0x7f6964055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:17.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.207+0000 7f697cff9700 1 -- 172.21.15.60:0/94212143 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f697006cf20 con 0x7f69800b5950 2022-06-29T16:03:17.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.207+0000 7f697f7fe700 1 --2- 172.21.15.60:0/94212143 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6964053a70 0x7f6964055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:17.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.207+0000 7f697f7fe700 1 --2- 172.21.15.60:0/94212143 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6964053a70 0x7f6964055f30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f697800b000 tx=0x7f69780091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:17.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.215+0000 7f697cff9700 1 -- 172.21.15.60:0/94212143 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6970043e30 con 0x7f69800b5950 2022-06-29T16:03:17.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.379+0000 7f6985b83700 1 -- 172.21.15.60:0/94212143 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69800573d0 con 0x7f69800b5950 2022-06-29T16:03:17.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.379+0000 7f697cff9700 1 -- 172.21.15.60:0/94212143 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+1778 (secure 0 0 0) 0x7f6970047410 con 0x7f69800b5950 2022-06-29T16:03:17.387 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:03:17.388 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":21,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5568,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/4229695218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":4229695218},{"type":"v1","addr":"172.21.15.60:6833","nonce":4229695218}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":21,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:01:54.732520+0000","modified":"2022-06-29T16:03:17.057935+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-06-29T16:03:17.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.383+0000 7f6985b83700 1 -- 172.21.15.60:0/94212143 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6964053a70 msgr2=0x7f6964055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:17.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.383+0000 7f6985b83700 1 --2- 172.21.15.60:0/94212143 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6964053a70 0x7f6964055f30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f697800b000 tx=0x7f69780091d0).stop 2022-06-29T16:03:17.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.383+0000 7f6985b83700 1 -- 172.21.15.60:0/94212143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69800b5950 msgr2=0x7f6980148580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:17.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.383+0000 7f6985b83700 1 --2- 172.21.15.60:0/94212143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69800b5950 0x7f6980148580 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0x7f6970002f70 tx=0x7f6970015000).stop 2022-06-29T16:03:17.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.383+0000 7f6985b83700 1 -- 172.21.15.60:0/94212143 shutdown_connections 2022-06-29T16:03:17.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.383+0000 7f6985b83700 1 --2- 172.21.15.60:0/94212143 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6964053a70 0x7f6964055f30 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:17.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.383+0000 7f6985b83700 1 --2- 172.21.15.60:0/94212143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69800b5950 0x7f6980148580 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:17.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.383+0000 7f6985b83700 1 --2- 172.21.15.60:0/94212143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6980063ad0 0x7f6980159660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:17.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.383+0000 7f6985b83700 1 --2- 172.21.15.60:0/94212143 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f698005bd20 0x7f69800af2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:17.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.383+0000 7f6985b83700 1 -- 172.21.15.60:0/94212143 >> 172.21.15.60:0/94212143 conn(0x7f69800632c0 msgr2=0x7f6980145f50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:17.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.383+0000 7f6985b83700 1 -- 172.21.15.60:0/94212143 shutdown_connections 2022-06-29T16:03:17.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.383+0000 7f6985b83700 1 -- 172.21.15.60:0/94212143 wait complete. 2022-06-29T16:03:17.392 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 21 2022-06-29T16:03:17.401 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-06-29T16:03:17.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.499+0000 7ff1b7fff700 1 Processor -- start 2022-06-29T16:03:17.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.499+0000 7ff1b7fff700 1 -- start start 2022-06-29T16:03:17.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.503+0000 7ff1b7fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1b80b48b0 0x7ff1b8142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:17.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.503+0000 7ff1b7fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff1b80b2c50 0x7ff1b80b3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:17.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.503+0000 7ff1b7fff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1b80b0cf0 con 0x7ff1b80b5eb0 2022-06-29T16:03:17.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.503+0000 7ff1b7fff700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1b80b54e0 con 0x7ff1b80b2c50 2022-06-29T16:03:17.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.503+0000 7ff1b7fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b80b4d10 con 0x7ff1b80b48b0 2022-06-29T16:03:17.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.503+0000 7ff1b77fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff1b80b2c50 0x7ff1b80b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:17.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.503+0000 7ff1b6ffd700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff1b80b5eb0 0x7ff1b80b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56376/0 (socket says 172.21.15.60:56376) 2022-06-29T16:03:17.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.503+0000 7ff1b77fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff1b80b2c50 0x7ff1b80b3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48292/0 (socket says 172.21.15.60:48292) 2022-06-29T16:03:17.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.503+0000 7ff1b6ffd700 1 -- 172.21.15.60:0/406983108 learned_addr learned my addr 172.21.15.60:0/406983108 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:03:17.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.503+0000 7ff1affff700 1 --2- 172.21.15.60:0/406983108 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1b80b48b0 0x7ff1b8142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:17.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.503+0000 7ff1b77fe700 1 -- 172.21.15.60:0/406983108 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1b80b48b0 msgr2=0x7ff1b8142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:17.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.503+0000 7ff1b77fe700 1 --2- 172.21.15.60:0/406983108 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1b80b48b0 0x7ff1b8142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:17.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.503+0000 7ff1b77fe700 1 -- 172.21.15.60:0/406983108 >> v1:172.21.15.60:6789/0 conn(0x7ff1b80b5eb0 legacy=0x7ff1b80b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:17.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.503+0000 7ff1b77fe700 1 -- 172.21.15.60:0/406983108 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1b80b6290 con 0x7ff1b80b2c50 2022-06-29T16:03:17.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.503+0000 7ff1b77fe700 1 --2- 172.21.15.60:0/406983108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1b80b2c50 0x7ff1b80b3090 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7ff1b00032f0 tx=0x7ff1b00091d0).ready entity=mon.0 client_cookie=ecde9f3f63db4bf6 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:17.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.503+0000 7ff1b67fc700 1 -- 172.21.15.60:0/406983108 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff1b000b6c0 con 0x7ff1b80b2c50 2022-06-29T16:03:17.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.503+0000 7ff1b67fc700 1 -- 172.21.15.60:0/406983108 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1b000b820 con 0x7ff1b80b2c50 2022-06-29T16:03:17.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.503+0000 7ff1b67fc700 1 -- 172.21.15.60:0/406983108 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff1b000b990 con 0x7ff1b80b2c50 2022-06-29T16:03:17.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.503+0000 7ff1b7fff700 1 -- 172.21.15.60:0/406983108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1b80b2c50 msgr2=0x7ff1b80b3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:17.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.503+0000 7ff1b7fff700 1 --2- 172.21.15.60:0/406983108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1b80b2c50 0x7ff1b80b3090 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7ff1b00032f0 tx=0x7ff1b00091d0).stop 2022-06-29T16:03:17.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.503+0000 7ff1b7fff700 1 -- 172.21.15.60:0/406983108 shutdown_connections 2022-06-29T16:03:17.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.503+0000 7ff1b7fff700 1 --2- 172.21.15.60:0/406983108 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1b80b2c50 0x7ff1b80b3090 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:17.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.503+0000 7ff1b7fff700 1 --2- 172.21.15.60:0/406983108 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1b80b48b0 0x7ff1b8142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:17.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.503+0000 7ff1b7fff700 1 -- 172.21.15.60:0/406983108 >> 172.21.15.60:0/406983108 conn(0x7ff1b8062090 msgr2=0x7ff1b8062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:17.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.503+0000 7ff1b7fff700 1 -- 172.21.15.60:0/406983108 shutdown_connections 2022-06-29T16:03:17.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.503+0000 7ff1b7fff700 1 -- 172.21.15.60:0/406983108 wait complete. 2022-06-29T16:03:17.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.507+0000 7ff1b7fff700 1 Processor -- start 2022-06-29T16:03:17.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.507+0000 7ff1b7fff700 1 -- start start 2022-06-29T16:03:17.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.507+0000 7ff1b7fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1b80b2c50 0x7ff1b8144220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:17.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.507+0000 7ff1b7fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1b80b48b0 0x7ff1b8152620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:17.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.507+0000 7ff1b7fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1b80b5eb0 0x7ff1b8152b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:17.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.507+0000 7ff1b7fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b8153000 con 0x7ff1b80b5eb0 2022-06-29T16:03:17.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.507+0000 7ff1b7fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b8151fc0 con 0x7ff1b80b2c50 2022-06-29T16:03:17.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.507+0000 7ff1b7fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b81407c0 con 0x7ff1b80b48b0 2022-06-29T16:03:17.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.507+0000 7ff1b77fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1b80b5eb0 0x7ff1b8152b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:17.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.507+0000 7ff1b77fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1b80b5eb0 0x7ff1b8152b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48300/0 (socket says 172.21.15.60:48300) 2022-06-29T16:03:17.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.507+0000 7ff1b77fe700 1 -- 172.21.15.60:0/2452471325 learned_addr learned my addr 172.21.15.60:0/2452471325 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:17.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.507+0000 7ff1affff700 1 --2- 172.21.15.60:0/2452471325 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1b80b48b0 0x7ff1b8152620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:17.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.507+0000 7ff1b6ffd700 1 --2- 172.21.15.60:0/2452471325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1b80b2c50 0x7ff1b8144220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:17.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.507+0000 7ff1b77fe700 1 -- 172.21.15.60:0/2452471325 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1b80b48b0 msgr2=0x7ff1b8152620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:17.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.507+0000 7ff1b77fe700 1 --2- 172.21.15.60:0/2452471325 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1b80b48b0 0x7ff1b8152620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:17.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.507+0000 7ff1b77fe700 1 -- 172.21.15.60:0/2452471325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1b80b2c50 msgr2=0x7ff1b8144220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:17.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.507+0000 7ff1b77fe700 1 --2- 172.21.15.60:0/2452471325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1b80b2c50 0x7ff1b8144220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:17.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.507+0000 7ff1b77fe700 1 -- 172.21.15.60:0/2452471325 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1b0009470 con 0x7ff1b80b5eb0 2022-06-29T16:03:17.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.507+0000 7ff1b77fe700 1 --2- 172.21.15.60:0/2452471325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1b80b5eb0 0x7ff1b8152b10 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x7ff1b00032f0 tx=0x7ff1b0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:17.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.507+0000 7ff1b4ff9700 1 -- 172.21.15.60:0/2452471325 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff1b0011e20 con 0x7ff1b80b5eb0 2022-06-29T16:03:17.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.507+0000 7ff1b4ff9700 1 -- 172.21.15.60:0/2452471325 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1b0021440 con 0x7ff1b80b5eb0 2022-06-29T16:03:17.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.507+0000 7ff1b7fff700 1 -- 172.21.15.60:0/2452471325 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1b80af450 con 0x7ff1b80b5eb0 2022-06-29T16:03:17.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.507+0000 7ff1b4ff9700 1 -- 172.21.15.60:0/2452471325 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff1b00217a0 con 0x7ff1b80b5eb0 2022-06-29T16:03:17.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.507+0000 7ff1b7fff700 1 -- 172.21.15.60:0/2452471325 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1b8140320 con 0x7ff1b80b5eb0 2022-06-29T16:03:17.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.511+0000 7ff1b7fff700 1 -- 172.21.15.60:0/2452471325 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1b8142f10 con 0x7ff1b80b5eb0 2022-06-29T16:03:17.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.511+0000 7ff1b4ff9700 1 -- 172.21.15.60:0/2452471325 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff1b0027020 con 0x7ff1b80b5eb0 2022-06-29T16:03:17.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.511+0000 7ff1b4ff9700 1 --2- 172.21.15.60:0/2452471325 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff19c053b00 0x7ff19c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:17.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.511+0000 7ff1b4ff9700 1 -- 172.21.15.60:0/2452471325 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6159+0+0 (secure 0 0 0) 0x7ff1b006d4e0 con 0x7ff1b80b5eb0 2022-06-29T16:03:17.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.511+0000 7ff1b6ffd700 1 --2- 172.21.15.60:0/2452471325 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff19c053b00 0x7ff19c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:17.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.511+0000 7ff1b6ffd700 1 --2- 172.21.15.60:0/2452471325 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff19c053b00 0x7ff19c055fc0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7ff1a800d000 tx=0x7ff1a8007be0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:17.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.519+0000 7ff1b4ff9700 1 -- 172.21.15.60:0/2452471325 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff1b00442a0 con 0x7ff1b80b5eb0 2022-06-29T16:03:17.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:17.679+0000 7ff1b7fff700 1 -- 172.21.15.60:0/2452471325 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff1b8140980 con 0x7ff1b80b5eb0 2022-06-29T16:03:18.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.071+0000 7ff1b4ff9700 1 -- 172.21.15.60:0/2452471325 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v22) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff1b00478c0 con 0x7ff1b80b5eb0 2022-06-29T16:03:18.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.075+0000 7ff1b7fff700 1 -- 172.21.15.60:0/2452471325 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff19c053b00 msgr2=0x7ff19c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:18.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.075+0000 7ff1b7fff700 1 --2- 172.21.15.60:0/2452471325 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff19c053b00 0x7ff19c055fc0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7ff1a800d000 tx=0x7ff1a8007be0).stop 2022-06-29T16:03:18.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.075+0000 7ff1b7fff700 1 -- 172.21.15.60:0/2452471325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1b80b5eb0 msgr2=0x7ff1b8152b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:18.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.075+0000 7ff1b7fff700 1 --2- 172.21.15.60:0/2452471325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1b80b5eb0 0x7ff1b8152b10 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x7ff1b00032f0 tx=0x7ff1b0016000).stop 2022-06-29T16:03:18.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.075+0000 7ff1b7fff700 1 -- 172.21.15.60:0/2452471325 shutdown_connections 2022-06-29T16:03:18.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.075+0000 7ff1b7fff700 1 --2- 172.21.15.60:0/2452471325 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff19c053b00 0x7ff19c055fc0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:18.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.075+0000 7ff1b7fff700 1 --2- 172.21.15.60:0/2452471325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1b80b5eb0 0x7ff1b8152b10 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:18.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.075+0000 7ff1b7fff700 1 --2- 172.21.15.60:0/2452471325 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1b80b48b0 0x7ff1b8152620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:18.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.075+0000 7ff1b7fff700 1 --2- 172.21.15.60:0/2452471325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1b80b2c50 0x7ff1b8144220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:18.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.075+0000 7ff1b7fff700 1 -- 172.21.15.60:0/2452471325 >> 172.21.15.60:0/2452471325 conn(0x7ff1b8062090 msgr2=0x7ff1b81430d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:18.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.075+0000 7ff1b7fff700 1 -- 172.21.15.60:0/2452471325 shutdown_connections 2022-06-29T16:03:18.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.075+0000 7ff1b7fff700 1 -- 172.21.15.60:0/2452471325 wait complete. 2022-06-29T16:03:18.139 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-06-29T16:03:18.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.251+0000 7f6e7c9fa700 1 Processor -- start 2022-06-29T16:03:18.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.251+0000 7f6e7c9fa700 1 -- start start 2022-06-29T16:03:18.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.255+0000 7f6e7c9fa700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6e781426b0 0x7f6e780b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:18.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.255+0000 7f6e7c9fa700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e78143a40 con 0x7f6e780b4010 2022-06-29T16:03:18.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.255+0000 7f6e7c9fa700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e780aee60 con 0x7f6e780ae7c0 2022-06-29T16:03:18.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.255+0000 7f6e7c9fa700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e780af100 con 0x7f6e781426b0 2022-06-29T16:03:18.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.255+0000 7f6e76d9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6e781426b0 0x7f6e780b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:18.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.255+0000 7f6e75d9b700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6e780b4010 0x7f6e780b46c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56390/0 (socket says 172.21.15.60:56390) 2022-06-29T16:03:18.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.255+0000 7f6e75d9b700 1 -- 172.21.15.60:0/3046301284 learned_addr learned my addr 172.21.15.60:0/3046301284 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:03:18.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.255+0000 7f6e76d9d700 1 -- 172.21.15.60:0/3046301284 >> v1:172.21.15.92:6789/0 conn(0x7f6e780ae7c0 legacy=0x7f6e78141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:18.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.255+0000 7f6e76d9d700 1 -- 172.21.15.60:0/3046301284 >> v1:172.21.15.60:6789/0 conn(0x7f6e780b4010 legacy=0x7f6e780b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:18.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.255+0000 7f6e76d9d700 1 -- 172.21.15.60:0/3046301284 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e780aeba0 con 0x7f6e781426b0 2022-06-29T16:03:18.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.255+0000 7f6e7559a700 1 -- 172.21.15.60:0/3046301284 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2307969008 0 0) 0x7f6e78143a40 con 0x7f6e780b4010 2022-06-29T16:03:18.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.255+0000 7f6e76d9d700 1 --2- 172.21.15.60:0/3046301284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e781426b0 0x7f6e780b4dd0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7f6e700032f0 tx=0x7f6e700091d0).ready entity=mon.0 client_cookie=19670f8d14c00177 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:18.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.255+0000 7f6e7559a700 1 -- 172.21.15.60:0/3046301284 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e70009ea0 con 0x7f6e781426b0 2022-06-29T16:03:18.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.255+0000 7f6e7559a700 1 -- 172.21.15.60:0/3046301284 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e7000b8a0 con 0x7f6e781426b0 2022-06-29T16:03:18.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.255+0000 7f6e7559a700 1 -- 172.21.15.60:0/3046301284 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e7000ba10 con 0x7f6e781426b0 2022-06-29T16:03:18.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.255+0000 7f6e7c9fa700 1 -- 172.21.15.60:0/3046301284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e781426b0 msgr2=0x7f6e780b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:18.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.255+0000 7f6e7c9fa700 1 --2- 172.21.15.60:0/3046301284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e781426b0 0x7f6e780b4dd0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7f6e700032f0 tx=0x7f6e700091d0).stop 2022-06-29T16:03:18.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.255+0000 7f6e7c9fa700 1 -- 172.21.15.60:0/3046301284 shutdown_connections 2022-06-29T16:03:18.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.255+0000 7f6e7c9fa700 1 --2- 172.21.15.60:0/3046301284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e781426b0 0x7f6e780b4dd0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:18.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.255+0000 7f6e7c9fa700 1 -- 172.21.15.60:0/3046301284 >> 172.21.15.60:0/3046301284 conn(0x7f6e780b9340 msgr2=0x7f6e780b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:18.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.255+0000 7f6e7c9fa700 1 -- 172.21.15.60:0/3046301284 shutdown_connections 2022-06-29T16:03:18.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.255+0000 7f6e7c9fa700 1 -- 172.21.15.60:0/3046301284 wait complete. 2022-06-29T16:03:18.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.259+0000 7f6e7c9fa700 1 Processor -- start 2022-06-29T16:03:18.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.259+0000 7f6e7c9fa700 1 -- start start 2022-06-29T16:03:18.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.259+0000 7f6e7c9fa700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e780ae7c0 0x7f6e780b89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:18.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.259+0000 7f6e7c9fa700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e780b4010 0x7f6e78146360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:18.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.259+0000 7f6e7c9fa700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e781426b0 0x7f6e7813d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:18.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.259+0000 7f6e7c9fa700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e78147870 con 0x7f6e780ae7c0 2022-06-29T16:03:18.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.259+0000 7f6e7c9fa700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e78147b10 con 0x7f6e781426b0 2022-06-29T16:03:18.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.259+0000 7f6e7c9fa700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e78147db0 con 0x7f6e780b4010 2022-06-29T16:03:18.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.259+0000 7f6e7659c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e780ae7c0 0x7f6e780b89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:18.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.263+0000 7f6e7659c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e780ae7c0 0x7f6e780b89d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48308/0 (socket says 172.21.15.60:48308) 2022-06-29T16:03:18.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.263+0000 7f6e7659c700 1 -- 172.21.15.60:0/1884520272 learned_addr learned my addr 172.21.15.60:0/1884520272 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:18.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.263+0000 7f6e75d9b700 1 --2- 172.21.15.60:0/1884520272 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e780b4010 0x7f6e78146360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:18.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.263+0000 7f6e76d9d700 1 --2- 172.21.15.60:0/1884520272 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e781426b0 0x7f6e7813d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:18.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.263+0000 7f6e7659c700 1 -- 172.21.15.60:0/1884520272 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e780b4010 msgr2=0x7f6e78146360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:18.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.263+0000 7f6e7659c700 1 --2- 172.21.15.60:0/1884520272 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e780b4010 0x7f6e78146360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:18.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.263+0000 7f6e7659c700 1 -- 172.21.15.60:0/1884520272 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e781426b0 msgr2=0x7f6e7813d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:18.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.263+0000 7f6e7659c700 1 --2- 172.21.15.60:0/1884520272 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e781426b0 0x7f6e7813d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:18.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.263+0000 7f6e7659c700 1 -- 172.21.15.60:0/1884520272 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e70009470 con 0x7f6e780ae7c0 2022-06-29T16:03:18.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.263+0000 7f6e7659c700 1 --2- 172.21.15.60:0/1884520272 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e780ae7c0 0x7f6e780b89d0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0x7f6e78060a90 tx=0x7f6e60007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:18.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.263+0000 7f6e6f7fe700 1 -- 172.21.15.60:0/1884520272 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e60012400 con 0x7f6e780ae7c0 2022-06-29T16:03:18.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.263+0000 7f6e6f7fe700 1 -- 172.21.15.60:0/1884520272 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e60012560 con 0x7f6e780ae7c0 2022-06-29T16:03:18.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.263+0000 7f6e7c9fa700 1 -- 172.21.15.60:0/1884520272 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e78146850 con 0x7f6e780ae7c0 2022-06-29T16:03:18.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.263+0000 7f6e6f7fe700 1 -- 172.21.15.60:0/1884520272 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e600128c0 con 0x7f6e780ae7c0 2022-06-29T16:03:18.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.263+0000 7f6e7c9fa700 1 -- 172.21.15.60:0/1884520272 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e78146fa0 con 0x7f6e780ae7c0 2022-06-29T16:03:18.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.263+0000 7f6e7c9fa700 1 -- 172.21.15.60:0/1884520272 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e780b10e0 con 0x7f6e780ae7c0 2022-06-29T16:03:18.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.263+0000 7f6e6f7fe700 1 -- 172.21.15.60:0/1884520272 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6e60019030 con 0x7f6e780ae7c0 2022-06-29T16:03:18.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.267+0000 7f6e6f7fe700 1 --2- 172.21.15.60:0/1884520272 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6e5c053b30 0x7f6e5c055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:18.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.267+0000 7f6e6f7fe700 1 -- 172.21.15.60:0/1884520272 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f6e6006b260 con 0x7f6e780ae7c0 2022-06-29T16:03:18.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.267+0000 7f6e75d9b700 1 --2- 172.21.15.60:0/1884520272 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6e5c053b30 0x7f6e5c055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:18.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.267+0000 7f6e75d9b700 1 --2- 172.21.15.60:0/1884520272 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6e5c053b30 0x7f6e5c055ff0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f6e6800d000 tx=0x7f6e680011a0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:18.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.271+0000 7f6e6f7fe700 1 -- 172.21.15.60:0/1884520272 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6e60042130 con 0x7f6e780ae7c0 2022-06-29T16:03:18.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:18.399+0000 7f6e7c9fa700 1 -- 172.21.15.60:0/1884520272 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6e780573d0 con 0x7f6e780ae7c0 2022-06-29T16:03:19.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.079+0000 7f6e6f7fe700 1 -- 172.21.15.60:0/1884520272 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v55) v1 ==== 185+0+0 (secure 0 0 0) 0x7f6e60045750 con 0x7f6e780ae7c0 2022-06-29T16:03:19.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.083+0000 7f6e7c9fa700 1 -- 172.21.15.60:0/1884520272 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6e5c053b30 msgr2=0x7f6e5c055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:19.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.083+0000 7f6e7c9fa700 1 --2- 172.21.15.60:0/1884520272 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6e5c053b30 0x7f6e5c055ff0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f6e6800d000 tx=0x7f6e680011a0).stop 2022-06-29T16:03:19.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.083+0000 7f6e7c9fa700 1 -- 172.21.15.60:0/1884520272 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e780ae7c0 msgr2=0x7f6e780b89d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:19.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.083+0000 7f6e7c9fa700 1 --2- 172.21.15.60:0/1884520272 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e780ae7c0 0x7f6e780b89d0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0x7f6e78060a90 tx=0x7f6e60007c50).stop 2022-06-29T16:03:19.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.087+0000 7f6e7c9fa700 1 -- 172.21.15.60:0/1884520272 shutdown_connections 2022-06-29T16:03:19.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.087+0000 7f6e7c9fa700 1 --2- 172.21.15.60:0/1884520272 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6e5c053b30 0x7f6e5c055ff0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:19.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.087+0000 7f6e7c9fa700 1 --2- 172.21.15.60:0/1884520272 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e781426b0 0x7f6e7813d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:19.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.087+0000 7f6e7c9fa700 1 --2- 172.21.15.60:0/1884520272 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e780b4010 0x7f6e78146360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:19.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.087+0000 7f6e7c9fa700 1 --2- 172.21.15.60:0/1884520272 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e780ae7c0 0x7f6e780b89d0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:19.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.087+0000 7f6e7c9fa700 1 -- 172.21.15.60:0/1884520272 >> 172.21.15.60:0/1884520272 conn(0x7f6e780b9340 msgr2=0x7f6e780b45e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:19.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.087+0000 7f6e7c9fa700 1 -- 172.21.15.60:0/1884520272 shutdown_connections 2022-06-29T16:03:19.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.087+0000 7f6e7c9fa700 1 -- 172.21.15.60:0/1884520272 wait complete. 2022-06-29T16:03:19.101 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2022-06-29T16:03:19.105 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-06-29T16:03:19.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.211+0000 7f6e6b7e1700 1 Processor -- start 2022-06-29T16:03:19.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.211+0000 7f6e6b7e1700 1 -- start start 2022-06-29T16:03:19.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.211+0000 7f6e6b7e1700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6e640b9d70 0x7f6e640b3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:19.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.211+0000 7f6e6b7e1700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6e6405d1f0 0x7f6e640b1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:19.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.211+0000 7f6e6b7e1700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e640b2ce0 con 0x7f6e640b3cc0 2022-06-29T16:03:19.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.211+0000 7f6e6b7e1700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e640b4d10 con 0x7f6e640b9d70 2022-06-29T16:03:19.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.211+0000 7f6e6b7e1700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e640ba150 con 0x7f6e6405d1f0 2022-06-29T16:03:19.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.211+0000 7f6e6957d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6e640b9d70 0x7f6e640b3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:19.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.211+0000 7f6e6957d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6e640b9d70 0x7f6e640b3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48316/0 (socket says 172.21.15.60:48316) 2022-06-29T16:03:19.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.211+0000 7f6e6957d700 1 -- 172.21.15.60:0/3490733320 learned_addr learned my addr 172.21.15.60:0/3490733320 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:19.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.211+0000 7f6e68d7c700 1 --2- 172.21.15.60:0/3490733320 >> v2:172.21.15.92:3300/0 conn(0x7f6e6405d1f0 0x7f6e640b1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:19.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.215+0000 7f6e6957d700 1 -- 172.21.15.60:0/3490733320 >> v2:172.21.15.92:3300/0 conn(0x7f6e6405d1f0 msgr2=0x7f6e640b1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:19.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.215+0000 7f6e6957d700 1 --2- 172.21.15.60:0/3490733320 >> v2:172.21.15.92:3300/0 conn(0x7f6e6405d1f0 0x7f6e640b1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:19.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.215+0000 7f6e6957d700 1 -- 172.21.15.60:0/3490733320 >> v1:172.21.15.92:6789/0 conn(0x7f6e640b3cc0 legacy=0x7f6e641420b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:19.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.215+0000 7f6e6957d700 1 -- 172.21.15.60:0/3490733320 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e640b4a70 con 0x7f6e640b9d70 2022-06-29T16:03:19.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.215+0000 7f6e6957d700 1 --2- 172.21.15.60:0/3490733320 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e640b9d70 0x7f6e640b3140 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7f6e54002f70 tx=0x7f6e54006eb0).ready entity=mon.0 client_cookie=19d1e16f5d0794de server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:19.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.215+0000 7f6e5bfff700 1 -- 172.21.15.60:0/3490733320 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e5400d5c0 con 0x7f6e640b9d70 2022-06-29T16:03:19.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.215+0000 7f6e5bfff700 1 -- 172.21.15.60:0/3490733320 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e54000e90 con 0x7f6e640b9d70 2022-06-29T16:03:19.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.215+0000 7f6e5bfff700 1 -- 172.21.15.60:0/3490733320 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e5400d950 con 0x7f6e640b9d70 2022-06-29T16:03:19.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.215+0000 7f6e6b7e1700 1 -- 172.21.15.60:0/3490733320 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e640b9d70 msgr2=0x7f6e640b3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:19.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.215+0000 7f6e6b7e1700 1 --2- 172.21.15.60:0/3490733320 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e640b9d70 0x7f6e640b3140 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7f6e54002f70 tx=0x7f6e54006eb0).stop 2022-06-29T16:03:19.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.215+0000 7f6e6b7e1700 1 -- 172.21.15.60:0/3490733320 shutdown_connections 2022-06-29T16:03:19.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.215+0000 7f6e6b7e1700 1 --2- 172.21.15.60:0/3490733320 >> v2:172.21.15.92:3300/0 conn(0x7f6e6405d1f0 0x7f6e640b1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:19.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.215+0000 7f6e6b7e1700 1 --2- 172.21.15.60:0/3490733320 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e640b9d70 0x7f6e640b3140 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:19.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.215+0000 7f6e6b7e1700 1 -- 172.21.15.60:0/3490733320 >> 172.21.15.60:0/3490733320 conn(0x7f6e64062ce0 msgr2=0x7f6e640b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:19.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.215+0000 7f6e6b7e1700 1 -- 172.21.15.60:0/3490733320 shutdown_connections 2022-06-29T16:03:19.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.215+0000 7f6e6b7e1700 1 -- 172.21.15.60:0/3490733320 wait complete. 2022-06-29T16:03:19.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.219+0000 7f6e6b7e1700 1 Processor -- start 2022-06-29T16:03:19.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.219+0000 7f6e6b7e1700 1 -- start start 2022-06-29T16:03:19.230 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.219+0000 7f6e6b7e1700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e6405d1f0 0x7f6e64155200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:19.230 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.219+0000 7f6e6b7e1700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e640b3cc0 0x7f6e641464f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:19.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.219+0000 7f6e6b7e1700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e640b9d70 0x7f6e64149a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:19.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.219+0000 7f6e6b7e1700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e641495f0 con 0x7f6e640b3cc0 2022-06-29T16:03:19.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.219+0000 7f6e6b7e1700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e640af5a0 con 0x7f6e640b9d70 2022-06-29T16:03:19.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.219+0000 7f6e6b7e1700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e640af840 con 0x7f6e6405d1f0 2022-06-29T16:03:19.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.219+0000 7f6e68d7c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e640b3cc0 0x7f6e641464f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:19.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.219+0000 7f6e68d7c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e640b3cc0 0x7f6e641464f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48324/0 (socket says 172.21.15.60:48324) 2022-06-29T16:03:19.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.219+0000 7f6e68d7c700 1 -- 172.21.15.60:0/809737739 learned_addr learned my addr 172.21.15.60:0/809737739 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:19.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.219+0000 7f6e6957d700 1 --2- 172.21.15.60:0/809737739 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e6405d1f0 0x7f6e64155200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:19.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.219+0000 7f6e69d7e700 1 --2- 172.21.15.60:0/809737739 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e640b9d70 0x7f6e64149a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:19.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.219+0000 7f6e68d7c700 1 -- 172.21.15.60:0/809737739 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e6405d1f0 msgr2=0x7f6e64155200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:19.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.219+0000 7f6e68d7c700 1 --2- 172.21.15.60:0/809737739 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e6405d1f0 0x7f6e64155200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:19.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.219+0000 7f6e68d7c700 1 -- 172.21.15.60:0/809737739 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e640b9d70 msgr2=0x7f6e64149a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:19.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.219+0000 7f6e68d7c700 1 --2- 172.21.15.60:0/809737739 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e640b9d70 0x7f6e64149a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:19.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.219+0000 7f6e68d7c700 1 -- 172.21.15.60:0/809737739 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5400b1e0 con 0x7f6e640b3cc0 2022-06-29T16:03:19.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.219+0000 7f6e68d7c700 1 --2- 172.21.15.60:0/809737739 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e640b3cc0 0x7f6e641464f0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7f6e6000c000 tx=0x7f6e60006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:19.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.223+0000 7f6e5a7fc700 1 -- 172.21.15.60:0/809737739 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e60011750 con 0x7f6e640b3cc0 2022-06-29T16:03:19.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.223+0000 7f6e5a7fc700 1 -- 172.21.15.60:0/809737739 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e600118b0 con 0x7f6e640b3cc0 2022-06-29T16:03:19.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.223+0000 7f6e6b7e1700 1 -- 172.21.15.60:0/809737739 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e640ae2e0 con 0x7f6e640b3cc0 2022-06-29T16:03:19.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.223+0000 7f6e5a7fc700 1 -- 172.21.15.60:0/809737739 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e60011c10 con 0x7f6e640b3cc0 2022-06-29T16:03:19.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.223+0000 7f6e6b7e1700 1 -- 172.21.15.60:0/809737739 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e640ae880 con 0x7f6e640b3cc0 2022-06-29T16:03:19.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.223+0000 7f6e6b7e1700 1 -- 172.21.15.60:0/809737739 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e640573d0 con 0x7f6e640b3cc0 2022-06-29T16:03:19.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.223+0000 7f6e5a7fc700 1 -- 172.21.15.60:0/809737739 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6e60011d70 con 0x7f6e640b3cc0 2022-06-29T16:03:19.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.223+0000 7f6e5a7fc700 1 --2- 172.21.15.60:0/809737739 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6e4c053b00 0x7f6e4c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:19.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.223+0000 7f6e5a7fc700 1 -- 172.21.15.60:0/809737739 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f6e6006c170 con 0x7f6e640b3cc0 2022-06-29T16:03:19.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.227+0000 7f6e6957d700 1 --2- 172.21.15.60:0/809737739 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6e4c053b00 0x7f6e4c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:19.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.227+0000 7f6e6957d700 1 --2- 172.21.15.60:0/809737739 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6e4c053b00 0x7f6e4c055fc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f6e5401c000 tx=0x7f6e54015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:19.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.231+0000 7f6e5a7fc700 1 -- 172.21.15.60:0/809737739 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6e60043190 con 0x7f6e640b3cc0 2022-06-29T16:03:19.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:19.359+0000 7f6e6b7e1700 1 -- 172.21.15.60:0/809737739 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6e64149780 con 0x7f6e640b3cc0 2022-06-29T16:03:20.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.095+0000 7f6e5a7fc700 1 -- 172.21.15.60:0/809737739 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v56) v1 ==== 173+0+0 (secure 0 0 0) 0x7f6e600467b0 con 0x7f6e640b3cc0 2022-06-29T16:03:20.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.095+0000 7f6e6b7e1700 1 -- 172.21.15.60:0/809737739 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6e4c053b00 msgr2=0x7f6e4c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:20.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.095+0000 7f6e6b7e1700 1 --2- 172.21.15.60:0/809737739 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6e4c053b00 0x7f6e4c055fc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f6e5401c000 tx=0x7f6e54015000).stop 2022-06-29T16:03:20.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.095+0000 7f6e6b7e1700 1 -- 172.21.15.60:0/809737739 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e640b3cc0 msgr2=0x7f6e641464f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:20.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.095+0000 7f6e6b7e1700 1 --2- 172.21.15.60:0/809737739 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e640b3cc0 0x7f6e641464f0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7f6e6000c000 tx=0x7f6e60006df0).stop 2022-06-29T16:03:20.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.095+0000 7f6e6b7e1700 1 -- 172.21.15.60:0/809737739 shutdown_connections 2022-06-29T16:03:20.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.095+0000 7f6e6b7e1700 1 --2- 172.21.15.60:0/809737739 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6e4c053b00 0x7f6e4c055fc0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:20.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.095+0000 7f6e6b7e1700 1 --2- 172.21.15.60:0/809737739 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e640b9d70 0x7f6e64149a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:20.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.095+0000 7f6e6b7e1700 1 --2- 172.21.15.60:0/809737739 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e640b3cc0 0x7f6e641464f0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:20.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.095+0000 7f6e6b7e1700 1 --2- 172.21.15.60:0/809737739 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e6405d1f0 0x7f6e64155200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:20.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.095+0000 7f6e6b7e1700 1 -- 172.21.15.60:0/809737739 >> 172.21.15.60:0/809737739 conn(0x7f6e64062ce0 msgr2=0x7f6e640b6410 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:20.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.095+0000 7f6e6b7e1700 1 -- 172.21.15.60:0/809737739 shutdown_connections 2022-06-29T16:03:20.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.095+0000 7f6e6b7e1700 1 -- 172.21.15.60:0/809737739 wait complete. 2022-06-29T16:03:20.109 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2022-06-29T16:03:20.117 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-06-29T16:03:20.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.211+0000 7f15c9310700 1 Processor -- start 2022-06-29T16:03:20.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.211+0000 7f15c9310700 1 -- start start 2022-06-29T16:03:20.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.211+0000 7f15c9310700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f15c405fb00 0x7f15c4143430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:20.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.211+0000 7f15c9310700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f15c4062aa0 0x7f15c4143920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:20.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.211+0000 7f15c9310700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15c40b1640 0x7f15c40aca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:20.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.211+0000 7f15c9310700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15c4142fd0 con 0x7f15c405fb00 2022-06-29T16:03:20.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.211+0000 7f15c9310700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15c4067d70 con 0x7f15c4062aa0 2022-06-29T16:03:20.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.211+0000 7f15c9310700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15c405fee0 con 0x7f15c40b1640 2022-06-29T16:03:20.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.211+0000 7f15c3fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f15c405fb00 0x7f15c4143430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:20.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.211+0000 7f15c3fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f15c405fb00 0x7f15c4143430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48330/0 (socket says 172.21.15.60:48330) 2022-06-29T16:03:20.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.211+0000 7f15c3fff700 1 -- 172.21.15.60:0/1310271429 learned_addr learned my addr 172.21.15.60:0/1310271429 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:20.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.211+0000 7f15c37fe700 1 --2- 172.21.15.60:0/1310271429 >> v2:172.21.15.92:3300/0 conn(0x7f15c4062aa0 0x7f15c4143920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:20.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.211+0000 7f15c8b0f700 1 --2- 172.21.15.60:0/1310271429 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15c40b1640 0x7f15c40aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:20.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.211+0000 7f15c3fff700 1 -- 172.21.15.60:0/1310271429 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15c40b1640 msgr2=0x7f15c40aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:20.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.211+0000 7f15c3fff700 1 --2- 172.21.15.60:0/1310271429 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15c40b1640 0x7f15c40aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:20.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.211+0000 7f15c3fff700 1 -- 172.21.15.60:0/1310271429 >> v2:172.21.15.92:3300/0 conn(0x7f15c4062aa0 msgr2=0x7f15c4143920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:20.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.211+0000 7f15c3fff700 1 --2- 172.21.15.60:0/1310271429 >> v2:172.21.15.92:3300/0 conn(0x7f15c4062aa0 0x7f15c4143920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:20.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.211+0000 7f15c3fff700 1 -- 172.21.15.60:0/1310271429 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c4143e10 con 0x7f15c405fb00 2022-06-29T16:03:20.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.211+0000 7f15c37fe700 1 --2- 172.21.15.60:0/1310271429 >> v2:172.21.15.92:3300/0 conn(0x7f15c4062aa0 0x7f15c4143920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:03:20.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.211+0000 7f15c3fff700 1 --2- 172.21.15.60:0/1310271429 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15c405fb00 0x7f15c4143430 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0x7f15b4002f70 tx=0x7f15b4006eb0).ready entity=mon.0 client_cookie=7d83c89537be44b5 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:20.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.211+0000 7f15c2ffd700 1 -- 172.21.15.60:0/1310271429 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15b400d570 con 0x7f15c405fb00 2022-06-29T16:03:20.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.211+0000 7f15c2ffd700 1 -- 172.21.15.60:0/1310271429 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15b4000e90 con 0x7f15c405fb00 2022-06-29T16:03:20.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.211+0000 7f15c2ffd700 1 -- 172.21.15.60:0/1310271429 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15b400d900 con 0x7f15c405fb00 2022-06-29T16:03:20.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.211+0000 7f15c9310700 1 -- 172.21.15.60:0/1310271429 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15c405fb00 msgr2=0x7f15c4143430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:20.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.211+0000 7f15c9310700 1 --2- 172.21.15.60:0/1310271429 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15c405fb00 0x7f15c4143430 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0x7f15b4002f70 tx=0x7f15b4006eb0).stop 2022-06-29T16:03:20.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.211+0000 7f15c9310700 1 -- 172.21.15.60:0/1310271429 shutdown_connections 2022-06-29T16:03:20.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.211+0000 7f15c9310700 1 --2- 172.21.15.60:0/1310271429 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15c40b1640 0x7f15c40aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:20.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.211+0000 7f15c9310700 1 --2- 172.21.15.60:0/1310271429 >> v2:172.21.15.92:3300/0 conn(0x7f15c4062aa0 0x7f15c4143920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:20.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.211+0000 7f15c9310700 1 --2- 172.21.15.60:0/1310271429 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15c405fb00 0x7f15c4143430 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:20.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.211+0000 7f15c9310700 1 -- 172.21.15.60:0/1310271429 >> 172.21.15.60:0/1310271429 conn(0x7f15c4063a60 msgr2=0x7f15c405f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:20.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.211+0000 7f15c9310700 1 -- 172.21.15.60:0/1310271429 shutdown_connections 2022-06-29T16:03:20.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.211+0000 7f15c9310700 1 -- 172.21.15.60:0/1310271429 wait complete. 2022-06-29T16:03:20.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.215+0000 7f15c9310700 1 Processor -- start 2022-06-29T16:03:20.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.215+0000 7f15c9310700 1 -- start start 2022-06-29T16:03:20.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.215+0000 7f15c9310700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15c405fb00 0x7f15c40b2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:20.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.215+0000 7f15c9310700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15c4062aa0 0x7f15c413cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:20.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.215+0000 7f15c9310700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15c40b1640 0x7f15c413d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:20.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.215+0000 7f15c9310700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c4064f60 con 0x7f15c405fb00 2022-06-29T16:03:20.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.215+0000 7f15c9310700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c4065200 con 0x7f15c40b1640 2022-06-29T16:03:20.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.215+0000 7f15c9310700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15c40654a0 con 0x7f15c4062aa0 2022-06-29T16:03:20.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.215+0000 7f15c3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15c405fb00 0x7f15c40b2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:20.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.215+0000 7f15c3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15c405fb00 0x7f15c40b2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48336/0 (socket says 172.21.15.60:48336) 2022-06-29T16:03:20.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.215+0000 7f15c3fff700 1 -- 172.21.15.60:0/4162707590 learned_addr learned my addr 172.21.15.60:0/4162707590 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:20.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.215+0000 7f15c37fe700 1 --2- 172.21.15.60:0/4162707590 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15c4062aa0 0x7f15c413cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:20.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.215+0000 7f15c8b0f700 1 --2- 172.21.15.60:0/4162707590 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15c40b1640 0x7f15c413d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:20.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.215+0000 7f15c3fff700 1 -- 172.21.15.60:0/4162707590 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15c4062aa0 msgr2=0x7f15c413cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:20.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.215+0000 7f15c3fff700 1 --2- 172.21.15.60:0/4162707590 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15c4062aa0 0x7f15c413cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:20.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.215+0000 7f15c3fff700 1 -- 172.21.15.60:0/4162707590 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15c40b1640 msgr2=0x7f15c413d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:20.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.215+0000 7f15c3fff700 1 --2- 172.21.15.60:0/4162707590 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15c40b1640 0x7f15c413d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:20.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.215+0000 7f15c3fff700 1 -- 172.21.15.60:0/4162707590 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b400b1e0 con 0x7f15c405fb00 2022-06-29T16:03:20.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.215+0000 7f15c3fff700 1 --2- 172.21.15.60:0/4162707590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15c405fb00 0x7f15c40b2770 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7f15b4002f70 tx=0x7f15b4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:20.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.215+0000 7f15c17fa700 1 -- 172.21.15.60:0/4162707590 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15b4006b40 con 0x7f15c405fb00 2022-06-29T16:03:20.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.215+0000 7f15c17fa700 1 -- 172.21.15.60:0/4162707590 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15b4011b80 con 0x7f15c405fb00 2022-06-29T16:03:20.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.215+0000 7f15c9310700 1 -- 172.21.15.60:0/4162707590 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15c40b2f70 con 0x7f15c405fb00 2022-06-29T16:03:20.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.215+0000 7f15c17fa700 1 -- 172.21.15.60:0/4162707590 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15b401e460 con 0x7f15c405fb00 2022-06-29T16:03:20.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.215+0000 7f15c9310700 1 -- 172.21.15.60:0/4162707590 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15c4153200 con 0x7f15c405fb00 2022-06-29T16:03:20.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.219+0000 7f15c9310700 1 -- 172.21.15.60:0/4162707590 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15c40573d0 con 0x7f15c405fb00 2022-06-29T16:03:20.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.219+0000 7f15c17fa700 1 -- 172.21.15.60:0/4162707590 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f15b401a040 con 0x7f15c405fb00 2022-06-29T16:03:20.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.219+0000 7f15c17fa700 1 --2- 172.21.15.60:0/4162707590 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f15a8053aa0 0x7f15a8055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:20.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.219+0000 7f15c17fa700 1 -- 172.21.15.60:0/4162707590 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f15b406c9d0 con 0x7f15c405fb00 2022-06-29T16:03:20.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.219+0000 7f15c37fe700 1 --2- 172.21.15.60:0/4162707590 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f15a8053aa0 0x7f15a8055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:20.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.219+0000 7f15c37fe700 1 --2- 172.21.15.60:0/4162707590 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f15a8053aa0 0x7f15a8055f60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f15ac00c000 tx=0x7f15ac008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:20.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.227+0000 7f15c17fa700 1 -- 172.21.15.60:0/4162707590 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f15b4043b30 con 0x7f15c405fb00 2022-06-29T16:03:20.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.355+0000 7f15c9310700 1 -- 172.21.15.60:0/4162707590 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f15c4140dc0 con 0x7f15c405fb00 2022-06-29T16:03:20.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.363+0000 7f15c17fa700 1 -- 172.21.15.60:0/4162707590 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v7) v1 ==== 104+0+0 (secure 0 0 0) 0x7f15b40230a0 con 0x7f15c405fb00 2022-06-29T16:03:20.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.363+0000 7f15c9310700 1 -- 172.21.15.60:0/4162707590 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f15a8053aa0 msgr2=0x7f15a8055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:20.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.363+0000 7f15c9310700 1 --2- 172.21.15.60:0/4162707590 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f15a8053aa0 0x7f15a8055f60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f15ac00c000 tx=0x7f15ac008040).stop 2022-06-29T16:03:20.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.363+0000 7f15c9310700 1 -- 172.21.15.60:0/4162707590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15c405fb00 msgr2=0x7f15c40b2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:20.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.363+0000 7f15c9310700 1 --2- 172.21.15.60:0/4162707590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15c405fb00 0x7f15c40b2770 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7f15b4002f70 tx=0x7f15b4015000).stop 2022-06-29T16:03:20.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.363+0000 7f15c9310700 1 -- 172.21.15.60:0/4162707590 shutdown_connections 2022-06-29T16:03:20.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.363+0000 7f15c9310700 1 --2- 172.21.15.60:0/4162707590 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f15a8053aa0 0x7f15a8055f60 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:20.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.363+0000 7f15c9310700 1 --2- 172.21.15.60:0/4162707590 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15c40b1640 0x7f15c413d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:20.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.363+0000 7f15c9310700 1 --2- 172.21.15.60:0/4162707590 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15c4062aa0 0x7f15c413cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:20.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.363+0000 7f15c9310700 1 --2- 172.21.15.60:0/4162707590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15c405fb00 0x7f15c40b2770 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:20.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.363+0000 7f15c9310700 1 -- 172.21.15.60:0/4162707590 >> 172.21.15.60:0/4162707590 conn(0x7f15c4063a60 msgr2=0x7f15c405f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:20.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.363+0000 7f15c9310700 1 -- 172.21.15.60:0/4162707590 shutdown_connections 2022-06-29T16:03:20.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.363+0000 7f15c9310700 1 -- 172.21.15.60:0/4162707590 wait complete. 2022-06-29T16:03:20.384 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead_timeout' 2022-06-29T16:03:20.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.483+0000 7f899a5c7700 1 Processor -- start 2022-06-29T16:03:20.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.483+0000 7f899a5c7700 1 -- start start 2022-06-29T16:03:20.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.483+0000 7f899a5c7700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f899405d070 0x7f899414aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:20.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.483+0000 7f899a5c7700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f899405d450 con 0x7f89940b2130 2022-06-29T16:03:20.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.483+0000 7f899a5c7700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89940ad840 con 0x7f899414af80 2022-06-29T16:03:20.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.483+0000 7f899a5c7700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89940adf90 con 0x7f899405d070 2022-06-29T16:03:20.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.483+0000 7f8998dc4700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f89940b2130 0x7f89940aca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56432/0 (socket says 172.21.15.60:56432) 2022-06-29T16:03:20.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.483+0000 7f8998dc4700 1 -- 172.21.15.60:0/2970169734 learned_addr learned my addr 172.21.15.60:0/2970169734 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:03:20.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.483+0000 7f89995c5700 1 --2- 172.21.15.60:0/2970169734 >> v2:172.21.15.92:3300/0 conn(0x7f899405d070 0x7f899414aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:20.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.487+0000 7f8993fff700 1 -- 172.21.15.60:0/2970169734 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2965271094 0 0) 0x7f89940ad840 con 0x7f899414af80 2022-06-29T16:03:20.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.487+0000 7f8993fff700 1 -- 172.21.15.60:0/2970169734 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8980003260 con 0x7f899414af80 2022-06-29T16:03:20.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.487+0000 7f8993fff700 1 -- 172.21.15.60:0/2970169734 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3051581331 0 0) 0x7f899405d450 con 0x7f89940b2130 2022-06-29T16:03:20.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.487+0000 7f8993fff700 1 -- 172.21.15.60:0/2970169734 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89940ad840 con 0x7f89940b2130 2022-06-29T16:03:20.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.487+0000 7f89995c5700 1 -- 172.21.15.60:0/2970169734 >> v1:172.21.15.92:6789/0 conn(0x7f899414af80 legacy=0x7f89940ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:20.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.487+0000 7f89995c5700 1 -- 172.21.15.60:0/2970169734 >> v1:172.21.15.60:6789/0 conn(0x7f89940b2130 legacy=0x7f89940aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:20.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.487+0000 7f89995c5700 1 -- 172.21.15.60:0/2970169734 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89940b25d0 con 0x7f899405d070 2022-06-29T16:03:20.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.487+0000 7f89995c5700 1 --2- 172.21.15.60:0/2970169734 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f899405d070 0x7f899414aa90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f898c0032f0 tx=0x7f898c0091d0).ready entity=mon.1 client_cookie=7305e85a459d37e4 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:20.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.487+0000 7f8993fff700 1 -- 172.21.15.60:0/2970169734 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f898c00b600 con 0x7f899405d070 2022-06-29T16:03:20.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.487+0000 7f8993fff700 1 -- 172.21.15.60:0/2970169734 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f898c00b760 con 0x7f899405d070 2022-06-29T16:03:20.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.487+0000 7f8993fff700 1 -- 172.21.15.60:0/2970169734 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f898c00b8d0 con 0x7f899405d070 2022-06-29T16:03:20.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.487+0000 7f899a5c7700 1 -- 172.21.15.60:0/2970169734 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f899405d070 msgr2=0x7f899414aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:20.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.487+0000 7f899a5c7700 1 --2- 172.21.15.60:0/2970169734 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f899405d070 0x7f899414aa90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f898c0032f0 tx=0x7f898c0091d0).stop 2022-06-29T16:03:20.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.487+0000 7f899a5c7700 1 -- 172.21.15.60:0/2970169734 shutdown_connections 2022-06-29T16:03:20.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.487+0000 7f899a5c7700 1 --2- 172.21.15.60:0/2970169734 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f899405d070 0x7f899414aa90 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:20.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.487+0000 7f899a5c7700 1 -- 172.21.15.60:0/2970169734 >> 172.21.15.60:0/2970169734 conn(0x7f89940b5890 msgr2=0x7f89940b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:20.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.487+0000 7f899a5c7700 1 -- 172.21.15.60:0/2970169734 shutdown_connections 2022-06-29T16:03:20.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.487+0000 7f899a5c7700 1 -- 172.21.15.60:0/2970169734 wait complete. 2022-06-29T16:03:20.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.487+0000 7f899a5c7700 1 Processor -- start 2022-06-29T16:03:20.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.487+0000 7f899a5c7700 1 -- start start 2022-06-29T16:03:20.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.487+0000 7f899a5c7700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f899405d070 0x7f8994066170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:20.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.487+0000 7f899a5c7700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f89940b2130 0x7f89940b4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:20.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.487+0000 7f899a5c7700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f899414af80 0x7f8994159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:20.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.487+0000 7f899a5c7700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8994066660 con 0x7f899414af80 2022-06-29T16:03:20.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.487+0000 7f899a5c7700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8994066880 con 0x7f899405d070 2022-06-29T16:03:20.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.487+0000 7f899a5c7700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8994066b20 con 0x7f89940b2130 2022-06-29T16:03:20.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.491+0000 7f8999dc6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f899414af80 0x7f8994159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:20.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.491+0000 7f8999dc6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f899414af80 0x7f8994159ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48354/0 (socket says 172.21.15.60:48354) 2022-06-29T16:03:20.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.491+0000 7f8999dc6700 1 -- 172.21.15.60:0/2838534215 learned_addr learned my addr 172.21.15.60:0/2838534215 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:20.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.491+0000 7f8998dc4700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f89940b2130 0x7f89940b4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:20.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.491+0000 7f89995c5700 1 --2- 172.21.15.60:0/2838534215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f899405d070 0x7f8994066170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:20.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.491+0000 7f8999dc6700 1 -- 172.21.15.60:0/2838534215 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f89940b2130 msgr2=0x7f89940b4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:20.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.491+0000 7f8999dc6700 1 --2- 172.21.15.60:0/2838534215 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f89940b2130 0x7f89940b4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:20.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.491+0000 7f8999dc6700 1 -- 172.21.15.60:0/2838534215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f899405d070 msgr2=0x7f8994066170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:20.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.491+0000 7f8999dc6700 1 --2- 172.21.15.60:0/2838534215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f899405d070 0x7f8994066170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:20.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.491+0000 7f8999dc6700 1 -- 172.21.15.60:0/2838534215 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f898c009470 con 0x7f899414af80 2022-06-29T16:03:20.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.491+0000 7f8999dc6700 1 --2- 172.21.15.60:0/2838534215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f899414af80 0x7f8994159ea0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x7f8984001110 tx=0x7f8984007a40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:20.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.491+0000 7f89927fc700 1 -- 172.21.15.60:0/2838534215 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8984012400 con 0x7f899414af80 2022-06-29T16:03:20.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.491+0000 7f89927fc700 1 -- 172.21.15.60:0/2838534215 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8984012560 con 0x7f899414af80 2022-06-29T16:03:20.510 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.491+0000 7f899a5c7700 1 -- 172.21.15.60:0/2838534215 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8994066dc0 con 0x7f899414af80 2022-06-29T16:03:20.510 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.491+0000 7f89927fc700 1 -- 172.21.15.60:0/2838534215 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89840128c0 con 0x7f899414af80 2022-06-29T16:03:20.511 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.491+0000 7f899a5c7700 1 -- 172.21.15.60:0/2838534215 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8994159710 con 0x7f899414af80 2022-06-29T16:03:20.511 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.491+0000 7f899a5c7700 1 -- 172.21.15.60:0/2838534215 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f899414a350 con 0x7f899414af80 2022-06-29T16:03:20.511 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.491+0000 7f89927fc700 1 -- 172.21.15.60:0/2838534215 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8984019030 con 0x7f899414af80 2022-06-29T16:03:20.511 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.491+0000 7f89927fc700 1 --2- 172.21.15.60:0/2838534215 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8974053a70 0x7f8974055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:20.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.491+0000 7f89927fc700 1 -- 172.21.15.60:0/2838534215 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f898406ae70 con 0x7f899414af80 2022-06-29T16:03:20.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.495+0000 7f89995c5700 1 --2- 172.21.15.60:0/2838534215 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8974053a70 0x7f8974055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:20.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.495+0000 7f89995c5700 1 --2- 172.21.15.60:0/2838534215 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8974053a70 0x7f8974055f30 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f898c019000 tx=0x7f898c016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:20.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.499+0000 7f89927fc700 1 -- 172.21.15.60:0/2838534215 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8984041fd0 con 0x7f899414af80 2022-06-29T16:03:20.675 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:20.667+0000 7f899a5c7700 1 -- 172.21.15.60:0/2838534215 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead_timeout"]} v 0) v1 -- 0x7f8994142910 con 0x7f899414af80 2022-06-29T16:03:21.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.095+0000 7f89927fc700 1 -- 172.21.15.60:0/2838534215 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead_timeout"]}]=0 v213) v1 ==== 152+0+0 (secure 0 0 0) 0x7f8984017020 con 0x7f899414af80 2022-06-29T16:03:21.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.099+0000 7f899a5c7700 1 -- 172.21.15.60:0/2838534215 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8974053a70 msgr2=0x7f8974055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:21.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.099+0000 7f899a5c7700 1 --2- 172.21.15.60:0/2838534215 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8974053a70 0x7f8974055f30 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f898c019000 tx=0x7f898c016000).stop 2022-06-29T16:03:21.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.099+0000 7f899a5c7700 1 -- 172.21.15.60:0/2838534215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f899414af80 msgr2=0x7f8994159ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:21.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.099+0000 7f899a5c7700 1 --2- 172.21.15.60:0/2838534215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f899414af80 0x7f8994159ea0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x7f8984001110 tx=0x7f8984007a40).stop 2022-06-29T16:03:21.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.099+0000 7f899a5c7700 1 -- 172.21.15.60:0/2838534215 shutdown_connections 2022-06-29T16:03:21.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.099+0000 7f899a5c7700 1 --2- 172.21.15.60:0/2838534215 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8974053a70 0x7f8974055f30 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:21.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.099+0000 7f899a5c7700 1 --2- 172.21.15.60:0/2838534215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f899414af80 0x7f8994159ea0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:21.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.099+0000 7f899a5c7700 1 --2- 172.21.15.60:0/2838534215 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f89940b2130 0x7f89940b4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:21.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.099+0000 7f899a5c7700 1 --2- 172.21.15.60:0/2838534215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f899405d070 0x7f8994066170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:21.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.099+0000 7f899a5c7700 1 -- 172.21.15.60:0/2838534215 >> 172.21.15.60:0/2838534215 conn(0x7f89940b5890 msgr2=0x7f89941433b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:21.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.099+0000 7f899a5c7700 1 -- 172.21.15.60:0/2838534215 shutdown_connections 2022-06-29T16:03:21.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.099+0000 7f899a5c7700 1 -- 172.21.15.60:0/2838534215 wait complete. 2022-06-29T16:03:21.123 INFO:tasks.cephfs_test_runner:test_drop_cache_command_dead_timeout (tasks.cephfs.test_misc.TestCacheDrop) ... ok 2022-06-29T16:03:21.124 INFO:tasks.cephfs_test_runner:Starting test: test_drop_cache_command_timeout (tasks.cephfs.test_misc.TestCacheDrop) 2022-06-29T16:03:21.125 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_timeout' 2022-06-29T16:03:21.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.231+0000 7ff13cef4700 1 Processor -- start 2022-06-29T16:03:21.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.231+0000 7ff13cef4700 1 -- start start 2022-06-29T16:03:21.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.235+0000 7ff13cef4700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff13805fb00 0x7ff1380ae890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:21.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.235+0000 7ff13cef4700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff1380b1c00 0x7ff138067250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:21.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.235+0000 7ff13cef4700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1380ad240 con 0x7ff1380b0fa0 2022-06-29T16:03:21.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.235+0000 7ff13cef4700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff13813ce60 con 0x7ff1380b1c00 2022-06-29T16:03:21.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.235+0000 7ff13cef4700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff13805fee0 con 0x7ff13805fb00 2022-06-29T16:03:21.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.235+0000 7ff137fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff1380b1c00 0x7ff138067250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:21.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.235+0000 7ff137fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff1380b1c00 0x7ff138067250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48362/0 (socket says 172.21.15.60:48362) 2022-06-29T16:03:21.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.235+0000 7ff137fff700 1 -- 172.21.15.60:0/4070075277 learned_addr learned my addr 172.21.15.60:0/4070075277 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:21.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.235+0000 7ff1377fe700 1 --2- 172.21.15.60:0/4070075277 >> v2:172.21.15.92:3300/0 conn(0x7ff13805fb00 0x7ff1380ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:21.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.235+0000 7ff137fff700 1 -- 172.21.15.60:0/4070075277 >> v2:172.21.15.92:3300/0 conn(0x7ff13805fb00 msgr2=0x7ff1380ae890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:21.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.235+0000 7ff137fff700 1 --2- 172.21.15.60:0/4070075277 >> v2:172.21.15.92:3300/0 conn(0x7ff13805fb00 0x7ff1380ae890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:21.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.235+0000 7ff137fff700 1 -- 172.21.15.60:0/4070075277 >> v1:172.21.15.92:6789/0 conn(0x7ff1380b0fa0 legacy=0x7ff1380b14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:21.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.235+0000 7ff137fff700 1 -- 172.21.15.60:0/4070075277 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff138143180 con 0x7ff1380b1c00 2022-06-29T16:03:21.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.235+0000 7ff137fff700 1 --2- 172.21.15.60:0/4070075277 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1380b1c00 0x7ff138067250 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7ff1300032f0 tx=0x7ff13000b530).ready entity=mon.0 client_cookie=8524ba55c7f9f154 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:21.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.235+0000 7ff136ffd700 1 -- 172.21.15.60:0/4070075277 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff1300153f0 con 0x7ff1380b1c00 2022-06-29T16:03:21.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.235+0000 7ff136ffd700 1 -- 172.21.15.60:0/4070075277 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff130015550 con 0x7ff1380b1c00 2022-06-29T16:03:21.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.235+0000 7ff136ffd700 1 -- 172.21.15.60:0/4070075277 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff1300158b0 con 0x7ff1380b1c00 2022-06-29T16:03:21.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.235+0000 7ff13cef4700 1 -- 172.21.15.60:0/4070075277 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1380b1c00 msgr2=0x7ff138067250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:21.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.235+0000 7ff13cef4700 1 --2- 172.21.15.60:0/4070075277 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1380b1c00 0x7ff138067250 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7ff1300032f0 tx=0x7ff13000b530).stop 2022-06-29T16:03:21.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.235+0000 7ff13cef4700 1 -- 172.21.15.60:0/4070075277 shutdown_connections 2022-06-29T16:03:21.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.235+0000 7ff13cef4700 1 --2- 172.21.15.60:0/4070075277 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1380b1c00 0x7ff138067250 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:21.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.235+0000 7ff13cef4700 1 --2- 172.21.15.60:0/4070075277 >> v2:172.21.15.92:3300/0 conn(0x7ff13805fb00 0x7ff1380ae890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:21.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.235+0000 7ff13cef4700 1 -- 172.21.15.60:0/4070075277 >> 172.21.15.60:0/4070075277 conn(0x7ff138063a60 msgr2=0x7ff13805f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:21.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.235+0000 7ff13cef4700 1 -- 172.21.15.60:0/4070075277 shutdown_connections 2022-06-29T16:03:21.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.235+0000 7ff13cef4700 1 -- 172.21.15.60:0/4070075277 wait complete. 2022-06-29T16:03:21.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.239+0000 7ff13cef4700 1 Processor -- start 2022-06-29T16:03:21.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.239+0000 7ff13cef4700 1 -- start start 2022-06-29T16:03:21.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.239+0000 7ff13cef4700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff13805fb00 0x7ff1380668f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:21.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.239+0000 7ff13cef4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1380b0fa0 0x7ff138142980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:21.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.239+0000 7ff13cef4700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1380b1c00 0x7ff138151000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:21.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.239+0000 7ff13cef4700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff138112560 con 0x7ff1380b0fa0 2022-06-29T16:03:21.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.239+0000 7ff13cef4700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1381514f0 con 0x7ff13805fb00 2022-06-29T16:03:21.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.239+0000 7ff13cef4700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1381122c0 con 0x7ff1380b1c00 2022-06-29T16:03:21.254 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.239+0000 7ff12ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1380b0fa0 0x7ff138142980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:21.254 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.239+0000 7ff12ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1380b0fa0 0x7ff138142980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48366/0 (socket says 172.21.15.60:48366) 2022-06-29T16:03:21.254 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.239+0000 7ff12ffff700 1 -- 172.21.15.60:0/2175836492 learned_addr learned my addr 172.21.15.60:0/2175836492 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:21.254 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.239+0000 7ff137fff700 1 --2- 172.21.15.60:0/2175836492 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1380b1c00 0x7ff138151000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:21.255 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.239+0000 7ff1377fe700 1 --2- 172.21.15.60:0/2175836492 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff13805fb00 0x7ff1380668f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:21.255 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.239+0000 7ff12ffff700 1 -- 172.21.15.60:0/2175836492 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1380b1c00 msgr2=0x7ff138151000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:21.255 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.239+0000 7ff12ffff700 1 --2- 172.21.15.60:0/2175836492 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1380b1c00 0x7ff138151000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:21.255 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.239+0000 7ff12ffff700 1 -- 172.21.15.60:0/2175836492 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff13805fb00 msgr2=0x7ff1380668f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:21.255 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.239+0000 7ff12ffff700 1 --2- 172.21.15.60:0/2175836492 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff13805fb00 0x7ff1380668f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:21.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.239+0000 7ff12ffff700 1 -- 172.21.15.60:0/2175836492 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff13000b7d0 con 0x7ff1380b0fa0 2022-06-29T16:03:21.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.239+0000 7ff12ffff700 1 --2- 172.21.15.60:0/2175836492 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1380b0fa0 0x7ff138142980 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0x7ff138062600 tx=0x7ff120007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:21.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.243+0000 7ff1357fa700 1 -- 172.21.15.60:0/2175836492 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff120007030 con 0x7ff1380b0fa0 2022-06-29T16:03:21.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.243+0000 7ff1357fa700 1 -- 172.21.15.60:0/2175836492 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff120007190 con 0x7ff1380b0fa0 2022-06-29T16:03:21.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.243+0000 7ff13cef4700 1 -- 172.21.15.60:0/2175836492 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff138065000 con 0x7ff1380b0fa0 2022-06-29T16:03:21.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.243+0000 7ff1357fa700 1 -- 172.21.15.60:0/2175836492 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff120006510 con 0x7ff1380b0fa0 2022-06-29T16:03:21.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.243+0000 7ff13cef4700 1 -- 172.21.15.60:0/2175836492 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1380b4d00 con 0x7ff1380b0fa0 2022-06-29T16:03:21.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.243+0000 7ff13cef4700 1 -- 172.21.15.60:0/2175836492 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff138112110 con 0x7ff1380b0fa0 2022-06-29T16:03:21.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.243+0000 7ff1357fa700 1 -- 172.21.15.60:0/2175836492 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff120006670 con 0x7ff1380b0fa0 2022-06-29T16:03:21.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.243+0000 7ff1357fa700 1 --2- 172.21.15.60:0/2175836492 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff11c053b00 0x7ff11c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:21.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.243+0000 7ff1357fa700 1 -- 172.21.15.60:0/2175836492 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5339+0+0 (secure 0 0 0) 0x7ff12006b510 con 0x7ff1380b0fa0 2022-06-29T16:03:21.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.243+0000 7ff1377fe700 1 --2- 172.21.15.60:0/2175836492 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff11c053b00 0x7ff11c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:21.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.247+0000 7ff1377fe700 1 --2- 172.21.15.60:0/2175836492 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff11c053b00 0x7ff11c055fc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7ff12800b000 tx=0x7ff128006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:21.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.251+0000 7ff1357fa700 1 -- 172.21.15.60:0/2175836492 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff1200426b0 con 0x7ff1380b0fa0 2022-06-29T16:03:21.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:21.419+0000 7ff13cef4700 1 -- 172.21.15.60:0/2175836492 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_timeout"]} v 0) v1 -- 0x7ff138065190 con 0x7ff1380b0fa0 2022-06-29T16:03:22.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.099+0000 7ff1357fa700 1 -- 172.21.15.60:0/2175836492 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_timeout"]}]=0 v214) v1 ==== 150+0+0 (secure 0 0 0) 0x7ff120045cd0 con 0x7ff1380b0fa0 2022-06-29T16:03:22.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.103+0000 7ff13cef4700 1 -- 172.21.15.60:0/2175836492 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff11c053b00 msgr2=0x7ff11c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:22.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.103+0000 7ff13cef4700 1 --2- 172.21.15.60:0/2175836492 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff11c053b00 0x7ff11c055fc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7ff12800b000 tx=0x7ff128006df0).stop 2022-06-29T16:03:22.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.103+0000 7ff13cef4700 1 -- 172.21.15.60:0/2175836492 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1380b0fa0 msgr2=0x7ff138142980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:22.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.103+0000 7ff13cef4700 1 --2- 172.21.15.60:0/2175836492 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1380b0fa0 0x7ff138142980 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0x7ff138062600 tx=0x7ff120007c00).stop 2022-06-29T16:03:22.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.103+0000 7ff13cef4700 1 -- 172.21.15.60:0/2175836492 shutdown_connections 2022-06-29T16:03:22.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.103+0000 7ff13cef4700 1 --2- 172.21.15.60:0/2175836492 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff11c053b00 0x7ff11c055fc0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:22.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.103+0000 7ff13cef4700 1 --2- 172.21.15.60:0/2175836492 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff1380b1c00 0x7ff138151000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:22.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.103+0000 7ff13cef4700 1 --2- 172.21.15.60:0/2175836492 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff1380b0fa0 0x7ff138142980 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:22.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.103+0000 7ff13cef4700 1 --2- 172.21.15.60:0/2175836492 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff13805fb00 0x7ff1380668f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:22.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.103+0000 7ff13cef4700 1 -- 172.21.15.60:0/2175836492 >> 172.21.15.60:0/2175836492 conn(0x7ff138063a60 msgr2=0x7ff1380b1470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:22.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.103+0000 7ff13cef4700 1 -- 172.21.15.60:0/2175836492 shutdown_connections 2022-06-29T16:03:22.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.103+0000 7ff13cef4700 1 -- 172.21.15.60:0/2175836492 wait complete. 2022-06-29T16:03:22.126 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-06-29T16:03:22.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.235+0000 7f9c0b59e700 1 Processor -- start 2022-06-29T16:03:22.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.235+0000 7f9c0b59e700 1 -- start start 2022-06-29T16:03:22.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.235+0000 7f9c0b59e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9c0c0b3360 0x7f9c0c142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:22.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.235+0000 7f9c0b59e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c0c0b0cf0 con 0x7f9c0c0b5eb0 2022-06-29T16:03:22.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.235+0000 7f9c0b59e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c0c0b4d10 con 0x7f9c0c0b48b0 2022-06-29T16:03:22.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.235+0000 7f9c0b59e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c0c0b54e0 con 0x7f9c0c0b3360 2022-06-29T16:03:22.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.235+0000 7f9c0ad9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9c0c0b3360 0x7f9c0c142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:22.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.235+0000 7f9c0a59c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9c0c0b5eb0 0x7f9c0c0b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56458/0 (socket says 172.21.15.60:56458) 2022-06-29T16:03:22.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.235+0000 7f9c0a59c700 1 -- 172.21.15.60:0/3018000376 learned_addr learned my addr 172.21.15.60:0/3018000376 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:03:22.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.235+0000 7f9c0ad9d700 1 -- 172.21.15.60:0/3018000376 >> v1:172.21.15.92:6789/0 conn(0x7f9c0c0b48b0 legacy=0x7f9c0c0b2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:22.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.235+0000 7f9c0ad9d700 1 -- 172.21.15.60:0/3018000376 >> v1:172.21.15.60:6789/0 conn(0x7f9c0c0b5eb0 legacy=0x7f9c0c0b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:22.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.235+0000 7f9c0ad9d700 1 -- 172.21.15.60:0/3018000376 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c0c0b6290 con 0x7f9c0c0b3360 2022-06-29T16:03:22.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.235+0000 7f9c0ad9d700 1 --2- 172.21.15.60:0/3018000376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c0c0b3360 0x7f9c0c142a20 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7f9c040032f0 tx=0x7f9c040091d0).ready entity=mon.0 client_cookie=6bfac5448970ca4e server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:22.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.235+0000 7f9c09d9b700 1 -- 172.21.15.60:0/3018000376 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c04009ea0 con 0x7f9c0c0b3360 2022-06-29T16:03:22.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.235+0000 7f9c09d9b700 1 -- 172.21.15.60:0/3018000376 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c0400b8a0 con 0x7f9c0c0b3360 2022-06-29T16:03:22.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.235+0000 7f9c09d9b700 1 -- 172.21.15.60:0/3018000376 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c0400ba10 con 0x7f9c0c0b3360 2022-06-29T16:03:22.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.235+0000 7f9c0b59e700 1 -- 172.21.15.60:0/3018000376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c0c0b3360 msgr2=0x7f9c0c142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:22.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.235+0000 7f9c0b59e700 1 --2- 172.21.15.60:0/3018000376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c0c0b3360 0x7f9c0c142a20 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7f9c040032f0 tx=0x7f9c040091d0).stop 2022-06-29T16:03:22.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.239+0000 7f9c0b59e700 1 -- 172.21.15.60:0/3018000376 shutdown_connections 2022-06-29T16:03:22.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.239+0000 7f9c0b59e700 1 --2- 172.21.15.60:0/3018000376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c0c0b3360 0x7f9c0c142a20 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:22.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.239+0000 7f9c0b59e700 1 -- 172.21.15.60:0/3018000376 >> 172.21.15.60:0/3018000376 conn(0x7f9c0c062090 msgr2=0x7f9c0c062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:22.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.239+0000 7f9c0b59e700 1 -- 172.21.15.60:0/3018000376 shutdown_connections 2022-06-29T16:03:22.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.239+0000 7f9c0b59e700 1 -- 172.21.15.60:0/3018000376 wait complete. 2022-06-29T16:03:22.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.239+0000 7f9c0b59e700 1 Processor -- start 2022-06-29T16:03:22.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.239+0000 7f9c0b59e700 1 -- start start 2022-06-29T16:03:22.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.239+0000 7f9c0b59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9c0c0b3360 0x7f9c0c113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:22.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.239+0000 7f9c0b59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9c0c0b48b0 0x7f9c0c1143c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:22.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.239+0000 7f9c0b59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c0c0b5eb0 0x7f9c0c1553b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:22.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.239+0000 7f9c0b59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c116890 con 0x7f9c0c0b5eb0 2022-06-29T16:03:22.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.239+0000 7f9c0b59e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c116d10 con 0x7f9c0c0b48b0 2022-06-29T16:03:22.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.239+0000 7f9c0b59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c1148b0 con 0x7f9c0c0b3360 2022-06-29T16:03:22.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.239+0000 7f9c0ad9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c0c0b5eb0 0x7f9c0c1553b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:22.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.239+0000 7f9c0ad9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c0c0b5eb0 0x7f9c0c1553b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48382/0 (socket says 172.21.15.60:48382) 2022-06-29T16:03:22.254 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.239+0000 7f9c0a59c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9c0c0b3360 0x7f9c0c113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:22.254 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.239+0000 7f9c0ad9d700 1 -- 172.21.15.60:0/3108200934 learned_addr learned my addr 172.21.15.60:0/3108200934 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:22.254 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.239+0000 7f9c01d9b700 1 --2- 172.21.15.60:0/3108200934 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9c0c0b48b0 0x7f9c0c1143c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:22.254 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.239+0000 7f9c0ad9d700 1 -- 172.21.15.60:0/3108200934 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9c0c0b3360 msgr2=0x7f9c0c113ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:22.255 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.239+0000 7f9c0ad9d700 1 --2- 172.21.15.60:0/3108200934 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9c0c0b3360 0x7f9c0c113ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:22.255 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.239+0000 7f9c0ad9d700 1 -- 172.21.15.60:0/3108200934 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9c0c0b48b0 msgr2=0x7f9c0c1143c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:22.255 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.239+0000 7f9c0ad9d700 1 --2- 172.21.15.60:0/3108200934 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9c0c0b48b0 0x7f9c0c1143c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:22.255 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.239+0000 7f9c0ad9d700 1 -- 172.21.15.60:0/3108200934 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c04009470 con 0x7f9c0c0b5eb0 2022-06-29T16:03:22.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.239+0000 7f9c0ad9d700 1 --2- 172.21.15.60:0/3108200934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c0c0b5eb0 0x7f9c0c1553b0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7f9c0c146a10 tx=0x7f9c04016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:22.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.239+0000 7f9c03fff700 1 -- 172.21.15.60:0/3108200934 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c04009ea0 con 0x7f9c0c0b5eb0 2022-06-29T16:03:22.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.239+0000 7f9c03fff700 1 -- 172.21.15.60:0/3108200934 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c0401f3f0 con 0x7f9c0c0b5eb0 2022-06-29T16:03:22.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.239+0000 7f9c0b59e700 1 -- 172.21.15.60:0/3108200934 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c0c116a70 con 0x7f9c0c0b5eb0 2022-06-29T16:03:22.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.239+0000 7f9c03fff700 1 -- 172.21.15.60:0/3108200934 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c04011ea0 con 0x7f9c0c0b5eb0 2022-06-29T16:03:22.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.239+0000 7f9c0b59e700 1 -- 172.21.15.60:0/3108200934 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c0c112450 con 0x7f9c0c0b5eb0 2022-06-29T16:03:22.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.243+0000 7f9c0b59e700 1 -- 172.21.15.60:0/3108200934 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c0c112850 con 0x7f9c0c0b5eb0 2022-06-29T16:03:22.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.243+0000 7f9c03fff700 1 -- 172.21.15.60:0/3108200934 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9c04026030 con 0x7f9c0c0b5eb0 2022-06-29T16:03:22.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.243+0000 7f9c03fff700 1 --2- 172.21.15.60:0/3108200934 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9bf0053b00 0x7f9bf0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:22.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.243+0000 7f9c03fff700 1 -- 172.21.15.60:0/3108200934 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f9c0406c7e0 con 0x7f9c0c0b5eb0 2022-06-29T16:03:22.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.243+0000 7f9c0a59c700 1 --2- 172.21.15.60:0/3108200934 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9bf0053b00 0x7f9bf0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:22.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.243+0000 7f9c0a59c700 1 --2- 172.21.15.60:0/3108200934 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9bf0053b00 0x7f9bf0055fc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f9bfc00d000 tx=0x7f9bfc007be0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:22.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.247+0000 7f9c03fff700 1 -- 172.21.15.60:0/3108200934 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9c040439e0 con 0x7f9c0c0b5eb0 2022-06-29T16:03:22.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.375+0000 7f9c0b59e700 1 -- 172.21.15.60:0/3108200934 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f9c0c0573d0 con 0x7f9c0c0b5eb0 2022-06-29T16:03:22.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.383+0000 7f9c03fff700 1 -- 172.21.15.60:0/3108200934 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v8)=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9c04047000 con 0x7f9c0c0b5eb0 2022-06-29T16:03:22.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.383+0000 7f9c0b59e700 1 -- 172.21.15.60:0/3108200934 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9bf0053b00 msgr2=0x7f9bf0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:22.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.383+0000 7f9c0b59e700 1 --2- 172.21.15.60:0/3108200934 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9bf0053b00 0x7f9bf0055fc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f9bfc00d000 tx=0x7f9bfc007be0).stop 2022-06-29T16:03:22.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.383+0000 7f9c0b59e700 1 -- 172.21.15.60:0/3108200934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c0c0b5eb0 msgr2=0x7f9c0c1553b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:22.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.383+0000 7f9c0b59e700 1 --2- 172.21.15.60:0/3108200934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c0c0b5eb0 0x7f9c0c1553b0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7f9c0c146a10 tx=0x7f9c04016000).stop 2022-06-29T16:03:22.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.383+0000 7f9c0b59e700 1 -- 172.21.15.60:0/3108200934 shutdown_connections 2022-06-29T16:03:22.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.383+0000 7f9c0b59e700 1 --2- 172.21.15.60:0/3108200934 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9bf0053b00 0x7f9bf0055fc0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:22.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.383+0000 7f9c0b59e700 1 --2- 172.21.15.60:0/3108200934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9c0c0b5eb0 0x7f9c0c1553b0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:22.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.383+0000 7f9c0b59e700 1 --2- 172.21.15.60:0/3108200934 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9c0c0b48b0 0x7f9c0c1143c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:22.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.383+0000 7f9c0b59e700 1 --2- 172.21.15.60:0/3108200934 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9c0c0b3360 0x7f9c0c113ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:22.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.383+0000 7f9c0b59e700 1 -- 172.21.15.60:0/3108200934 >> 172.21.15.60:0/3108200934 conn(0x7f9c0c062090 msgr2=0x7f9c0c1410d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:22.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.383+0000 7f9c0b59e700 1 -- 172.21.15.60:0/3108200934 shutdown_connections 2022-06-29T16:03:22.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.383+0000 7f9c0b59e700 1 -- 172.21.15.60:0/3108200934 wait complete. 2022-06-29T16:03:22.403 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-06-29T16:03:22.403 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:03:22.423 DEBUG:teuthology.orchestra.run.smithi092:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:03:22.444 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:03:22.465 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:03:22.486 DEBUG:teuthology.orchestra.run.smithi197:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:03:22.507 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-06-29T16:03:22.507 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:03:22.508 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:03:22.508 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:03:22.515 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-06-29T16:03:22.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:03:22.517 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-06-29T16:03:22.517 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:03:22.518 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:03:22.518 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:03:22.525 INFO:teuthology.orchestra.run.smithi156.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-06-29T16:03:22.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:03:22.527 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-06-29T16:03:22.527 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:03:22.528 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:03:22.528 DEBUG:teuthology.orchestra.run.smithi197:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T16:03:22.535 INFO:teuthology.orchestra.run.smithi197.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-06-29T16:03:22.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:03:22.537 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-06-29T16:03:22.538 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-06-29T16:03:22.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.643+0000 7f6879a56700 1 Processor -- start 2022-06-29T16:03:22.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.643+0000 7f6879a56700 1 -- start start 2022-06-29T16:03:22.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.643+0000 7f6879a56700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f687405d070 0x7f687414aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:22.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.643+0000 7f6879a56700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f687414af80 0x7f68740ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:22.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.643+0000 7f6879a56700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f687405d450 con 0x7f68740b2130 2022-06-29T16:03:22.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.643+0000 7f6879a56700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68740adf90 con 0x7f687405d070 2022-06-29T16:03:22.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.643+0000 7f6879a56700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68740b25d0 con 0x7f687414af80 2022-06-29T16:03:22.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.643+0000 7f6878a54700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f687405d070 0x7f687414aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:22.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.643+0000 7f6873fff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f68740b2130 0x7f68740aca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56472/0 (socket says 172.21.15.60:56472) 2022-06-29T16:03:22.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.643+0000 7f6873fff700 1 -- 172.21.15.60:0/903604095 learned_addr learned my addr 172.21.15.60:0/903604095 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:03:22.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.643+0000 7f6879255700 1 --2- 172.21.15.60:0/903604095 >> v2:172.21.15.92:3300/0 conn(0x7f687414af80 0x7f68740ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:22.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.643+0000 7f6878a54700 1 -- 172.21.15.60:0/903604095 >> v2:172.21.15.92:3300/0 conn(0x7f687414af80 msgr2=0x7f68740ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:22.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.643+0000 7f6878a54700 1 --2- 172.21.15.60:0/903604095 >> v2:172.21.15.92:3300/0 conn(0x7f687414af80 0x7f68740ad130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:22.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.643+0000 7f6878a54700 1 -- 172.21.15.60:0/903604095 >> v1:172.21.15.60:6789/0 conn(0x7f68740b2130 legacy=0x7f68740aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:22.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.643+0000 7f68737fe700 1 -- 172.21.15.60:0/903604095 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1172393919 0 0) 0x7f687405d450 con 0x7f68740b2130 2022-06-29T16:03:22.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.643+0000 7f6878a54700 1 -- 172.21.15.60:0/903604095 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f687414a350 con 0x7f687405d070 2022-06-29T16:03:22.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.643+0000 7f6879255700 1 --2- 172.21.15.60:0/903604095 >> v2:172.21.15.92:3300/0 conn(0x7f687414af80 0x7f68740ad130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:03:22.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.647+0000 7f6878a54700 1 --2- 172.21.15.60:0/903604095 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f687405d070 0x7f687414aa90 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7f6868002f70 tx=0x7f6868006df0).ready entity=mon.0 client_cookie=9a349581fbb7274c server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:22.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.647+0000 7f68737fe700 1 -- 172.21.15.60:0/903604095 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f686800d5c0 con 0x7f687405d070 2022-06-29T16:03:22.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.647+0000 7f68737fe700 1 -- 172.21.15.60:0/903604095 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6868000e90 con 0x7f687405d070 2022-06-29T16:03:22.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.647+0000 7f68737fe700 1 -- 172.21.15.60:0/903604095 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f686800d910 con 0x7f687405d070 2022-06-29T16:03:22.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.647+0000 7f6879a56700 1 -- 172.21.15.60:0/903604095 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f687405d070 msgr2=0x7f687414aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:22.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.647+0000 7f6879a56700 1 --2- 172.21.15.60:0/903604095 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f687405d070 0x7f687414aa90 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7f6868002f70 tx=0x7f6868006df0).stop 2022-06-29T16:03:22.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.647+0000 7f6879a56700 1 -- 172.21.15.60:0/903604095 shutdown_connections 2022-06-29T16:03:22.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.647+0000 7f6879a56700 1 --2- 172.21.15.60:0/903604095 >> v2:172.21.15.92:3300/0 conn(0x7f687414af80 0x7f68740ad130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:22.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.647+0000 7f6879a56700 1 --2- 172.21.15.60:0/903604095 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f687405d070 0x7f687414aa90 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:22.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.647+0000 7f6879a56700 1 -- 172.21.15.60:0/903604095 >> 172.21.15.60:0/903604095 conn(0x7f68740b5890 msgr2=0x7f68740b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:22.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.647+0000 7f6879a56700 1 -- 172.21.15.60:0/903604095 shutdown_connections 2022-06-29T16:03:22.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.647+0000 7f6879a56700 1 -- 172.21.15.60:0/903604095 wait complete. 2022-06-29T16:03:22.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.647+0000 7f6879a56700 1 Processor -- start 2022-06-29T16:03:22.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.647+0000 7f6879a56700 1 -- start start 2022-06-29T16:03:22.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.647+0000 7f6879a56700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68740b2130 0x7f6874156dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:22.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.647+0000 7f6879a56700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f687414af80 0x7f6874066a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:22.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.647+0000 7f6879a56700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6874065430 0x7f68740630c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:22.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.647+0000 7f6879a56700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68741561a0 con 0x7f6874065430 2022-06-29T16:03:22.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.647+0000 7f6879a56700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6874157410 con 0x7f68740b2130 2022-06-29T16:03:22.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.647+0000 7f6879a56700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68741563f0 con 0x7f687414af80 2022-06-29T16:03:22.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.647+0000 7f6879255700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6874065430 0x7f68740630c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:22.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.647+0000 7f6879255700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6874065430 0x7f68740630c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48394/0 (socket says 172.21.15.60:48394) 2022-06-29T16:03:22.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.647+0000 7f6879255700 1 -- 172.21.15.60:0/1633702276 learned_addr learned my addr 172.21.15.60:0/1633702276 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:22.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.651+0000 7f6873fff700 1 --2- 172.21.15.60:0/1633702276 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f687414af80 0x7f6874066a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:22.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.651+0000 7f6878a54700 1 --2- 172.21.15.60:0/1633702276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68740b2130 0x7f6874156dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:22.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.651+0000 7f6879255700 1 -- 172.21.15.60:0/1633702276 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f687414af80 msgr2=0x7f6874066a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:22.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.651+0000 7f6879255700 1 --2- 172.21.15.60:0/1633702276 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f687414af80 0x7f6874066a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:22.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.651+0000 7f6879255700 1 -- 172.21.15.60:0/1633702276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68740b2130 msgr2=0x7f6874156dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:22.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.651+0000 7f6879255700 1 --2- 172.21.15.60:0/1633702276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68740b2130 0x7f6874156dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:22.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.651+0000 7f6879255700 1 -- 172.21.15.60:0/1633702276 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f686800b180 con 0x7f6874065430 2022-06-29T16:03:22.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.651+0000 7f6878a54700 1 --2- 172.21.15.60:0/1633702276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68740b2130 0x7f6874156dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:03:22.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.651+0000 7f6879255700 1 --2- 172.21.15.60:0/1633702276 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6874065430 0x7f68740630c0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0x7f686c00a000 tx=0x7f686c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:22.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.651+0000 7f6871ffb700 1 -- 172.21.15.60:0/1633702276 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f686c011780 con 0x7f6874065430 2022-06-29T16:03:22.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.651+0000 7f6871ffb700 1 -- 172.21.15.60:0/1633702276 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f686c0118e0 con 0x7f6874065430 2022-06-29T16:03:22.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.651+0000 7f6879a56700 1 -- 172.21.15.60:0/1633702276 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6874156690 con 0x7f6874065430 2022-06-29T16:03:22.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.651+0000 7f6871ffb700 1 -- 172.21.15.60:0/1633702276 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f686c011c40 con 0x7f6874065430 2022-06-29T16:03:22.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.651+0000 7f6879a56700 1 -- 172.21.15.60:0/1633702276 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68740658c0 con 0x7f6874065430 2022-06-29T16:03:22.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.651+0000 7f6879a56700 1 -- 172.21.15.60:0/1633702276 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6874065b10 con 0x7f6874065430 2022-06-29T16:03:22.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.655+0000 7f6871ffb700 1 -- 172.21.15.60:0/1633702276 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f686c012420 con 0x7f6874065430 2022-06-29T16:03:22.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.655+0000 7f6871ffb700 1 --2- 172.21.15.60:0/1633702276 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6854053b00 0x7f6854055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:22.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.655+0000 7f6871ffb700 1 -- 172.21.15.60:0/1633702276 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f686c06be50 con 0x7f6874065430 2022-06-29T16:03:22.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.655+0000 7f6878a54700 1 --2- 172.21.15.60:0/1633702276 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6854053b00 0x7f6854055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:22.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.655+0000 7f6878a54700 1 --2- 172.21.15.60:0/1633702276 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6854053b00 0x7f6854055fc0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f6868018000 tx=0x7f6868014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:22.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.659+0000 7f6871ffb700 1 -- 172.21.15.60:0/1633702276 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f686c042f50 con 0x7f6874065430 2022-06-29T16:03:22.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.787+0000 7f6879a56700 1 -- 172.21.15.60:0/1633702276 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f687414a5f0 con 0x7f6874065430 2022-06-29T16:03:22.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.787+0000 7f6871ffb700 1 -- 172.21.15.60:0/1633702276 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v56) v1 ==== 81+0+28828 (secure 0 0 0) 0x7f686c046570 con 0x7f6874065430 2022-06-29T16:03:22.796 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:03:22.796 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:03:22.797 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 56, 2022-06-29T16:03:22.797 INFO:teuthology.orchestra.run.smithi060.stdout: "fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", 2022-06-29T16:03:22.797 INFO:teuthology.orchestra.run.smithi060.stdout: "created": "2022-06-29T15:58:07.965719+0000", 2022-06-29T16:03:22.797 INFO:teuthology.orchestra.run.smithi060.stdout: "modified": "2022-06-29T16:03:20.080611+0000", 2022-06-29T16:03:22.798 INFO:teuthology.orchestra.run.smithi060.stdout: "last_up_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:03:22.798 INFO:teuthology.orchestra.run.smithi060.stdout: "last_in_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:03:22.798 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-06-29T16:03:22.798 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_num": 5799936, 2022-06-29T16:03:22.799 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_set": [ 2022-06-29T16:03:22.799 INFO:teuthology.orchestra.run.smithi060.stdout: "pglog_hardlimit", 2022-06-29T16:03:22.799 INFO:teuthology.orchestra.run.smithi060.stdout: "purged_snapdirs", 2022-06-29T16:03:22.799 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_deletes", 2022-06-29T16:03:22.800 INFO:teuthology.orchestra.run.smithi060.stdout: "sortbitwise" 2022-06-29T16:03:22.800 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:03:22.800 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_version": 4, 2022-06-29T16:03:22.800 INFO:teuthology.orchestra.run.smithi060.stdout: "full_ratio": 0.94999998807907104, 2022-06-29T16:03:22.800 INFO:teuthology.orchestra.run.smithi060.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-06-29T16:03:22.801 INFO:teuthology.orchestra.run.smithi060.stdout: "nearfull_ratio": 0.85000002384185791, 2022-06-29T16:03:22.801 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_snapshot": "", 2022-06-29T16:03:22.801 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_max": 10, 2022-06-29T16:03:22.801 INFO:teuthology.orchestra.run.smithi060.stdout: "max_osd": 8, 2022-06-29T16:03:22.802 INFO:teuthology.orchestra.run.smithi060.stdout: "require_min_compat_client": "jewel", 2022-06-29T16:03:22.802 INFO:teuthology.orchestra.run.smithi060.stdout: "min_compat_client": "jewel", 2022-06-29T16:03:22.802 INFO:teuthology.orchestra.run.smithi060.stdout: "require_osd_release": "octopus", 2022-06-29T16:03:22.802 INFO:teuthology.orchestra.run.smithi060.stdout: "pools": [ 2022-06-29T16:03:22.803 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.803 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2022-06-29T16:03:22.803 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "device_health_metrics", 2022-06-29T16:03:22.804 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:10.961127+0000", 2022-06-29T16:03:22.804 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:03:22.804 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:03:22.804 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:03:22.805 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:03:22.805 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:03:22.805 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:03:22.805 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:03:22.805 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:03:22.806 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 1, 2022-06-29T16:03:22.806 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 1, 2022-06-29T16:03:22.806 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 1, 2022-06-29T16:03:22.806 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 1, 2022-06-29T16:03:22.807 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 1, 2022-06-29T16:03:22.807 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:03:22.807 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:03:22.807 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:03:22.808 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:03:22.808 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:03:22.808 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:03:22.809 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:03:22.809 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.809 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "15", 2022-06-29T16:03:22.810 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:03:22.810 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:03:22.810 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:03:22.810 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:03:22.811 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:03:22.811 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:03:22.811 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:03:22.811 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:03:22.812 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:03:22.812 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:03:22.812 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:03:22.812 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:03:22.812 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:03:22.813 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:03:22.813 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:03:22.813 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:03:22.813 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:03:22.814 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:03:22.814 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:03:22.814 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:03:22.815 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:03:22.815 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:03:22.815 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:03:22.815 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:03:22.815 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:03:22.816 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:03:22.816 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.816 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:03:22.816 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:03:22.817 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:03:22.817 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:03:22.817 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:03:22.817 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:03:22.818 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:03:22.818 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:03:22.818 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:03:22.818 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:03:22.818 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:03:22.819 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:03:22.819 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 1 2022-06-29T16:03:22.819 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.819 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:03:22.820 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_devicehealth": {} 2022-06-29T16:03:22.820 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.820 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.820 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.820 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 2, 2022-06-29T16:03:22.821 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "rbd", 2022-06-29T16:03:22.821 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:19.184276+0000", 2022-06-29T16:03:22.821 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:03:22.821 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:03:22.822 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:03:22.822 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:03:22.822 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:03:22.822 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:03:22.823 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:03:22.823 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:03:22.823 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:03:22.824 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:03:22.824 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:03:22.824 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:03:22.825 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:03:22.825 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:03:22.826 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:03:22.826 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:03:22.826 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:03:22.827 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:03:22.827 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:03:22.827 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:03:22.828 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.828 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "18", 2022-06-29T16:03:22.828 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:03:22.828 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:03:22.829 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:03:22.829 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:03:22.829 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:03:22.829 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:03:22.829 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:03:22.830 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:03:22.830 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:03:22.830 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:03:22.830 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:03:22.831 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:03:22.831 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:03:22.831 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:03:22.831 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:03:22.831 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:03:22.832 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:03:22.832 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:03:22.832 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:03:22.832 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:03:22.833 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:03:22.833 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:03:22.833 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:03:22.833 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:03:22.833 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:03:22.834 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:03:22.834 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.834 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:03:22.834 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:03:22.835 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:03:22.835 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:03:22.835 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:03:22.835 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:03:22.835 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:03:22.836 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:03:22.836 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:03:22.836 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:03:22.836 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:03:22.837 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:03:22.837 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:03:22.837 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd": {} 2022-06-29T16:03:22.837 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.838 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.838 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:03:22.838 INFO:teuthology.orchestra.run.smithi060.stdout: "osds": [ 2022-06-29T16:03:22.838 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.838 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:03:22.839 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "11e7ab9e-fce2-470f-8195-ea4c3c52df22", 2022-06-29T16:03:22.839 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:03:22.839 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:03:22.839 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:03:22.840 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:03:22.840 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:03:22.840 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:03:22.840 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:03:22.840 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 51, 2022-06-29T16:03:22.841 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:03:22.841 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:03:22.841 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:03:22.842 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:22.842 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.843 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:22.843 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6816", 2022-06-29T16:03:22.843 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:03:22.843 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.844 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.844 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:22.844 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6825", 2022-06-29T16:03:22.844 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:03:22.845 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.845 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.846 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.846 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:03:22.846 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:22.846 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.847 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:22.847 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6826", 2022-06-29T16:03:22.847 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:03:22.847 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.847 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.848 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:22.848 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6827", 2022-06-29T16:03:22.848 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:03:22.848 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.849 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.849 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.849 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:03:22.849 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:22.849 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.850 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:22.850 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6830", 2022-06-29T16:03:22.850 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:03:22.850 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.851 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.851 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:22.851 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6831", 2022-06-29T16:03:22.851 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:03:22.852 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.852 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.852 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.852 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:03:22.852 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:22.853 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.853 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:22.853 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6828", 2022-06-29T16:03:22.853 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:03:22.853 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.854 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.854 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:22.854 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6829", 2022-06-29T16:03:22.854 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:03:22.855 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.855 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.855 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.855 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6825/12890", 2022-06-29T16:03:22.856 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6827/12890", 2022-06-29T16:03:22.856 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6831/12890", 2022-06-29T16:03:22.856 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6829/12890", 2022-06-29T16:03:22.856 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:03:22.856 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:03:22.857 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:03:22.857 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.857 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.857 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.858 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:03:22.858 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "ca43f136-6f69-4431-a157-fbe073fb585c", 2022-06-29T16:03:22.858 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:03:22.858 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:03:22.859 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:03:22.859 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:03:22.859 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:03:22.859 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:03:22.859 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:03:22.860 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 51, 2022-06-29T16:03:22.860 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:03:22.860 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:03:22.860 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:03:22.860 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:22.861 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.861 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:22.861 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6815", 2022-06-29T16:03:22.861 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:03:22.862 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.862 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.862 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:22.862 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6818", 2022-06-29T16:03:22.863 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:03:22.863 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.863 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.863 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.863 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:03:22.864 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:22.864 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.864 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:22.864 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6819", 2022-06-29T16:03:22.865 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:03:22.865 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.865 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.865 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:22.865 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6820", 2022-06-29T16:03:22.866 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:03:22.866 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.866 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.866 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.867 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:03:22.867 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:22.867 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.867 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:22.867 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6823", 2022-06-29T16:03:22.868 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:03:22.868 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.868 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.868 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:22.869 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6824", 2022-06-29T16:03:22.869 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:03:22.869 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.869 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.869 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.870 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:03:22.870 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:22.870 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.870 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:22.871 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6821", 2022-06-29T16:03:22.871 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:03:22.871 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.871 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.871 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:22.872 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6822", 2022-06-29T16:03:22.872 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:03:22.873 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.873 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.873 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.874 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6818/12892", 2022-06-29T16:03:22.874 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6820/12892", 2022-06-29T16:03:22.874 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6824/12892", 2022-06-29T16:03:22.874 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6822/12892", 2022-06-29T16:03:22.874 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:03:22.875 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:03:22.875 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:03:22.875 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.875 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.876 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.876 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:03:22.876 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "74fac45f-5f07-44ec-a9cb-2644c123fd4a", 2022-06-29T16:03:22.876 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:03:22.877 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:03:22.877 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:03:22.877 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:03:22.877 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:03:22.877 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:03:22.878 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:03:22.878 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 51, 2022-06-29T16:03:22.878 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:03:22.878 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:03:22.879 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:03:22.879 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:22.879 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.879 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:22.879 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6808", 2022-06-29T16:03:22.880 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:03:22.880 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.880 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.880 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:22.881 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6809", 2022-06-29T16:03:22.881 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:03:22.881 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.881 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.881 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.882 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:03:22.882 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:22.882 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.882 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:22.883 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6810", 2022-06-29T16:03:22.883 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:03:22.883 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.883 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.884 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:22.884 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6811", 2022-06-29T16:03:22.884 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:03:22.884 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.884 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.885 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.885 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:03:22.885 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:22.885 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.886 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:22.886 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6814", 2022-06-29T16:03:22.886 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:03:22.886 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.887 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.887 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:22.887 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6817", 2022-06-29T16:03:22.887 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:03:22.887 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.888 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.888 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.888 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:03:22.888 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:22.889 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.889 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:22.889 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6812", 2022-06-29T16:03:22.889 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:03:22.889 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.890 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.890 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:22.890 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6813", 2022-06-29T16:03:22.890 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:03:22.891 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.891 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.891 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.891 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6809/12891", 2022-06-29T16:03:22.892 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6811/12891", 2022-06-29T16:03:22.892 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6817/12891", 2022-06-29T16:03:22.892 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6813/12891", 2022-06-29T16:03:22.892 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:03:22.892 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:03:22.893 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:03:22.893 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.893 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.893 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.894 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:03:22.894 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "2581d351-973a-4b4e-85ca-3658f0068765", 2022-06-29T16:03:22.894 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:03:22.894 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:03:22.895 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:03:22.895 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:03:22.895 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:03:22.895 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:03:22.895 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:03:22.896 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 50, 2022-06-29T16:03:22.896 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:03:22.896 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:03:22.896 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:03:22.897 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:22.897 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.897 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:22.897 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6800", 2022-06-29T16:03:22.897 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:03:22.898 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.898 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.898 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:22.898 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6801", 2022-06-29T16:03:22.899 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:03:22.899 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.899 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.899 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.900 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:03:22.900 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:22.900 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.900 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:22.900 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6802", 2022-06-29T16:03:22.901 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:03:22.901 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.901 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.901 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:22.902 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6803", 2022-06-29T16:03:22.902 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:03:22.902 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.902 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.902 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.903 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:03:22.903 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:22.903 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.903 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:22.904 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6806", 2022-06-29T16:03:22.904 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:03:22.904 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.904 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.904 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:22.905 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6807", 2022-06-29T16:03:22.905 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:03:22.905 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.905 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.905 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.906 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:03:22.906 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:22.906 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.906 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:22.907 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6804", 2022-06-29T16:03:22.907 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:03:22.907 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.907 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.907 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:22.908 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6805", 2022-06-29T16:03:22.908 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:03:22.908 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.908 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.909 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.909 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6801/12889", 2022-06-29T16:03:22.909 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6803/12889", 2022-06-29T16:03:22.909 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6807/12889", 2022-06-29T16:03:22.909 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6805/12889", 2022-06-29T16:03:22.910 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:03:22.910 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:03:22.910 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:03:22.910 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.911 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.911 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.911 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:03:22.911 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "3d234db9-87cd-4fe9-814f-0240a287d8b4", 2022-06-29T16:03:22.911 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:03:22.912 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:03:22.912 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:03:22.912 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:03:22.912 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:03:22.913 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:03:22.913 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:03:22.913 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 51, 2022-06-29T16:03:22.913 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:03:22.913 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:03:22.914 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:03:22.914 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:22.914 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.914 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:22.915 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6808", 2022-06-29T16:03:22.915 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:03:22.915 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.915 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.915 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:22.916 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6810", 2022-06-29T16:03:22.916 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:03:22.916 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.916 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.917 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.917 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:03:22.917 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:22.917 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.918 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:22.918 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6812", 2022-06-29T16:03:22.918 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:03:22.918 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.919 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.919 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:22.919 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6814", 2022-06-29T16:03:22.919 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:03:22.920 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.920 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.920 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.920 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:03:22.920 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:22.921 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.921 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:22.921 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6823", 2022-06-29T16:03:22.921 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:03:22.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.922 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:22.922 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6825", 2022-06-29T16:03:22.923 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:03:22.923 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.923 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.923 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.923 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:03:22.924 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:22.924 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.924 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:22.924 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6817", 2022-06-29T16:03:22.925 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:03:22.925 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.925 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.925 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:22.925 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6820", 2022-06-29T16:03:22.926 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:03:22.926 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.926 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.927 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.927 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6810/13249", 2022-06-29T16:03:22.927 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6814/13249", 2022-06-29T16:03:22.927 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6825/13249", 2022-06-29T16:03:22.927 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6820/13249", 2022-06-29T16:03:22.928 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:03:22.928 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:03:22.928 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:03:22.928 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.929 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.929 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:03:22.929 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "e4bf1104-ff5c-438b-b608-8cee31c163f6", 2022-06-29T16:03:22.930 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:03:22.930 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:03:22.930 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:03:22.931 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:03:22.931 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:03:22.932 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:03:22.932 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:03:22.933 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 51, 2022-06-29T16:03:22.933 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:03:22.933 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:03:22.934 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:03:22.934 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:22.935 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.935 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:22.936 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6809", 2022-06-29T16:03:22.936 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:03:22.937 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.937 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.937 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:22.938 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6811", 2022-06-29T16:03:22.938 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:03:22.939 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.939 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.940 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.940 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:03:22.941 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:22.941 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.942 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:22.942 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6813", 2022-06-29T16:03:22.942 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:03:22.943 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.943 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.944 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:22.944 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6815", 2022-06-29T16:03:22.945 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:03:22.945 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.946 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.946 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.947 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:03:22.947 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:22.948 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.948 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:22.948 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6824", 2022-06-29T16:03:22.949 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:03:22.949 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.950 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.950 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:22.951 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6827", 2022-06-29T16:03:22.951 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:03:22.952 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.952 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.953 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.953 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:03:22.953 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:22.954 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.954 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:22.955 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6818", 2022-06-29T16:03:22.955 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:03:22.956 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.956 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.957 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:22.957 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6821", 2022-06-29T16:03:22.957 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:03:22.958 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.958 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.959 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.959 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6811/13248", 2022-06-29T16:03:22.960 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6815/13248", 2022-06-29T16:03:22.960 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6827/13248", 2022-06-29T16:03:22.961 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6821/13248", 2022-06-29T16:03:22.961 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:03:22.961 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:03:22.961 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:03:22.962 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.962 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.962 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.962 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:03:22.962 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "8564dd25-27ab-4203-b54e-af6d3aa05857", 2022-06-29T16:03:22.963 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:03:22.963 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:03:22.963 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:03:22.963 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:03:22.964 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:03:22.964 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:03:22.964 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:03:22.964 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 51, 2022-06-29T16:03:22.965 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:03:22.965 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:03:22.965 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:03:22.965 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:22.965 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.966 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:22.966 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6816", 2022-06-29T16:03:22.966 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:03:22.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.967 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:22.967 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6819", 2022-06-29T16:03:22.967 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:03:22.967 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.968 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.968 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:03:22.968 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:22.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.969 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:22.969 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6822", 2022-06-29T16:03:22.969 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:03:22.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.970 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:22.970 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6826", 2022-06-29T16:03:22.970 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:03:22.971 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.971 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.971 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.971 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:03:22.971 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:22.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.972 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:22.972 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6830", 2022-06-29T16:03:22.973 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:03:22.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.973 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:22.974 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6831", 2022-06-29T16:03:22.974 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:03:22.974 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.974 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.975 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:03:22.975 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:22.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.976 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:22.976 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6828", 2022-06-29T16:03:22.976 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:03:22.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.977 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.977 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:22.977 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6829", 2022-06-29T16:03:22.978 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:03:22.978 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.978 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.979 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6819/13250", 2022-06-29T16:03:22.979 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6826/13250", 2022-06-29T16:03:22.979 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6831/13250", 2022-06-29T16:03:22.979 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6829/13250", 2022-06-29T16:03:22.980 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:03:22.980 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:03:22.980 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:03:22.980 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.981 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:03:22.981 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "bd513a9c-3b19-4314-9897-d300b5c897bb", 2022-06-29T16:03:22.981 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:03:22.982 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:03:22.982 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:03:22.982 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:03:22.982 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:03:22.983 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:03:22.983 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:03:22.983 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 50, 2022-06-29T16:03:22.983 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:03:22.983 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:03:22.984 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:03:22.984 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:22.984 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.984 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:22.985 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6800", 2022-06-29T16:03:22.985 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:03:22.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.985 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:22.986 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6801", 2022-06-29T16:03:22.986 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:03:22.986 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.986 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.987 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:03:22.987 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:22.987 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.987 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:22.988 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6802", 2022-06-29T16:03:22.988 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:03:22.988 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.988 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.989 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:22.989 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6803", 2022-06-29T16:03:22.989 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:03:22.989 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.990 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.990 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:03:22.990 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:22.990 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.991 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:22.991 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6806", 2022-06-29T16:03:22.991 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:03:22.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.992 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.992 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:22.992 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6807", 2022-06-29T16:03:22.992 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:03:22.992 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.993 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.993 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.993 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:03:22.993 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:22.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.994 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:22.994 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6804", 2022-06-29T16:03:22.994 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:03:22.995 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.995 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.995 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:22.995 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6805", 2022-06-29T16:03:22.995 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:03:22.996 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.996 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:22.996 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6801/13247", 2022-06-29T16:03:22.996 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6803/13247", 2022-06-29T16:03:22.997 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6807/13247", 2022-06-29T16:03:22.997 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6805/13247", 2022-06-29T16:03:22.997 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:03:22.997 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:03:22.998 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:03:22.998 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:22.998 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:22.998 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:03:22.999 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_xinfo": [ 2022-06-29T16:03:22.999 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:22.999 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:03:22.999 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:03:22.999 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:03:23.000 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:03:23.000 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:03:23.000 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:03:23.000 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.891342+0000", 2022-06-29T16:03:23.000 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:03:23.001 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:23.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:23.001 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:03:23.001 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:03:23.002 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:03:23.002 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:03:23.002 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:03:23.002 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:03:23.003 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.900866+0000", 2022-06-29T16:03:23.003 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:03:23.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:23.003 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:23.003 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:03:23.004 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:03:23.004 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:03:23.004 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:03:23.004 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:03:23.004 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:03:23.005 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.912755+0000", 2022-06-29T16:03:23.005 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:03:23.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:23.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:23.006 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:03:23.006 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:03:23.006 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:03:23.006 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:03:23.006 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:03:23.007 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:03:23.007 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.864332+0000", 2022-06-29T16:03:23.007 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:03:23.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:23.008 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:23.008 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:03:23.008 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:03:23.008 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:03:23.008 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:03:23.009 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:03:23.009 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:03:23.009 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.863756+0000", 2022-06-29T16:03:23.009 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:03:23.010 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:23.010 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:23.010 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:03:23.010 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:03:23.010 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:03:23.011 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:03:23.011 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:03:23.011 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:03:23.011 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.869041+0000", 2022-06-29T16:03:23.012 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:03:23.012 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:23.012 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:23.012 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:03:23.012 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:03:23.013 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:03:23.013 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:03:23.013 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:03:23.014 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:03:23.014 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.895043+0000", 2022-06-29T16:03:23.014 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:03:23.014 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:23.015 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:23.015 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:03:23.015 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:03:23.015 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:03:23.015 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:03:23.016 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:03:23.016 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:03:23.016 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.889491+0000", 2022-06-29T16:03:23.016 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:03:23.016 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:23.017 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:03:23.017 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap": [], 2022-06-29T16:03:23.017 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap_items": [], 2022-06-29T16:03:23.017 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_temp": [], 2022-06-29T16:03:23.018 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_temp": [], 2022-06-29T16:03:23.018 INFO:teuthology.orchestra.run.smithi060.stdout: "blacklist": { 2022-06-29T16:03:23.018 INFO:teuthology.orchestra.run.smithi060.stdout: "172.21.15.92:6835/1602880694": "2022-06-30T16:03:16.338493+0000", 2022-06-29T16:03:23.018 INFO:teuthology.orchestra.run.smithi060.stdout: "172.21.15.92:6834/1602880694": "2022-06-30T16:03:16.338493+0000" 2022-06-29T16:03:23.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:23.019 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profiles": { 2022-06-29T16:03:23.019 INFO:teuthology.orchestra.run.smithi060.stdout: "default": { 2022-06-29T16:03:23.019 INFO:teuthology.orchestra.run.smithi060.stdout: "crush-failure-domain": "osd", 2022-06-29T16:03:23.019 INFO:teuthology.orchestra.run.smithi060.stdout: "k": "2", 2022-06-29T16:03:23.020 INFO:teuthology.orchestra.run.smithi060.stdout: "m": "1", 2022-06-29T16:03:23.020 INFO:teuthology.orchestra.run.smithi060.stdout: "plugin": "jerasure", 2022-06-29T16:03:23.020 INFO:teuthology.orchestra.run.smithi060.stdout: "ruleset-failure-domain": "osd", 2022-06-29T16:03:23.020 INFO:teuthology.orchestra.run.smithi060.stdout: "technique": "reed_sol_van" 2022-06-29T16:03:23.021 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:23.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:23.021 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps_queue": [], 2022-06-29T16:03:23.021 INFO:teuthology.orchestra.run.smithi060.stdout: "new_removed_snaps": [], 2022-06-29T16:03:23.021 INFO:teuthology.orchestra.run.smithi060.stdout: "new_purged_snaps": [], 2022-06-29T16:03:23.022 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_node_flags": {}, 2022-06-29T16:03:23.022 INFO:teuthology.orchestra.run.smithi060.stdout: "device_class_flags": {} 2022-06-29T16:03:23.022 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:03:23.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.791+0000 7f6879a56700 1 -- 172.21.15.60:0/1633702276 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6854053b00 msgr2=0x7f6854055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:23.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.791+0000 7f6879a56700 1 --2- 172.21.15.60:0/1633702276 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6854053b00 0x7f6854055fc0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f6868018000 tx=0x7f6868014000).stop 2022-06-29T16:03:23.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.791+0000 7f6879a56700 1 -- 172.21.15.60:0/1633702276 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6874065430 msgr2=0x7f68740630c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:23.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.791+0000 7f6879a56700 1 --2- 172.21.15.60:0/1633702276 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6874065430 0x7f68740630c0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0x7f686c00a000 tx=0x7f686c008040).stop 2022-06-29T16:03:23.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.791+0000 7f6879a56700 1 -- 172.21.15.60:0/1633702276 shutdown_connections 2022-06-29T16:03:23.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.791+0000 7f6879a56700 1 --2- 172.21.15.60:0/1633702276 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6854053b00 0x7f6854055fc0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:23.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.791+0000 7f6879a56700 1 --2- 172.21.15.60:0/1633702276 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6874065430 0x7f68740630c0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:23.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.791+0000 7f6879a56700 1 --2- 172.21.15.60:0/1633702276 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f687414af80 0x7f6874066a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:23.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.791+0000 7f6879a56700 1 --2- 172.21.15.60:0/1633702276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f68740b2130 0x7f6874156dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:23.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.791+0000 7f6879a56700 1 -- 172.21.15.60:0/1633702276 >> 172.21.15.60:0/1633702276 conn(0x7f68740b5890 msgr2=0x7f6874145ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:23.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.791+0000 7f6879a56700 1 -- 172.21.15.60:0/1633702276 shutdown_connections 2022-06-29T16:03:23.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:22.791+0000 7f6879a56700 1 -- 172.21.15.60:0/1633702276 wait complete. 2022-06-29T16:03:23.028 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:03:23.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.139+0000 7ff0f28f4700 1 Processor -- start 2022-06-29T16:03:23.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.139+0000 7ff0f28f4700 1 -- start start 2022-06-29T16:03:23.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.139+0000 7ff0f28f4700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff0ec0b48b0 0x7ff0ec142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:23.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.139+0000 7ff0f28f4700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0ec0b2c50 0x7ff0ec0b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:23.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.139+0000 7ff0f28f4700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0ec0b0cf0 con 0x7ff0ec0b5eb0 2022-06-29T16:03:23.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.139+0000 7ff0f28f4700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0ec0b54e0 con 0x7ff0ec0b48b0 2022-06-29T16:03:23.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.139+0000 7ff0f28f4700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ec0b4d10 con 0x7ff0ec0b2c50 2022-06-29T16:03:23.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.139+0000 7ff0f10f1700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff0ec0b48b0 0x7ff0ec142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:23.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.139+0000 7ff0f20f3700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0ec0b2c50 0x7ff0ec0b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:23.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.139+0000 7ff0f18f2700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7ff0ec0b5eb0 0x7ff0ec0b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:60734/0 (socket says 172.21.15.60:60734) 2022-06-29T16:03:23.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.139+0000 7ff0f18f2700 1 -- 172.21.15.60:0/1020069950 learned_addr learned my addr 172.21.15.60:0/1020069950 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:03:23.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.139+0000 7ff0f10f1700 1 -- 172.21.15.60:0/1020069950 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0ec0b2c50 msgr2=0x7ff0ec0b30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:23.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.139+0000 7ff0f10f1700 1 --2- 172.21.15.60:0/1020069950 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0ec0b2c50 0x7ff0ec0b30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:23.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.139+0000 7ff0f10f1700 1 -- 172.21.15.60:0/1020069950 >> v1:172.21.15.92:6789/0 conn(0x7ff0ec0b5eb0 legacy=0x7ff0ec0b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:23.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.139+0000 7ff0f10f1700 1 -- 172.21.15.60:0/1020069950 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0ec0b6290 con 0x7ff0ec0b48b0 2022-06-29T16:03:23.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.139+0000 7ff0f20f3700 1 --2- 172.21.15.60:0/1020069950 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0ec0b2c50 0x7ff0ec0b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:03:23.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.139+0000 7ff0f10f1700 1 --2- 172.21.15.60:0/1020069950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff0ec0b48b0 0x7ff0ec142a20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7ff0e40032f0 tx=0x7ff0e40091d0).ready entity=mon.1 client_cookie=53fc64f26e9ea258 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:23.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.139+0000 7ff0f08f0700 1 -- 172.21.15.60:0/1020069950 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0e400b6c0 con 0x7ff0ec0b48b0 2022-06-29T16:03:23.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.139+0000 7ff0f08f0700 1 -- 172.21.15.60:0/1020069950 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e400b820 con 0x7ff0ec0b48b0 2022-06-29T16:03:23.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.139+0000 7ff0f08f0700 1 -- 172.21.15.60:0/1020069950 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0e400b990 con 0x7ff0ec0b48b0 2022-06-29T16:03:23.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.139+0000 7ff0f28f4700 1 -- 172.21.15.60:0/1020069950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff0ec0b48b0 msgr2=0x7ff0ec142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:23.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.139+0000 7ff0f28f4700 1 --2- 172.21.15.60:0/1020069950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff0ec0b48b0 0x7ff0ec142a20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7ff0e40032f0 tx=0x7ff0e40091d0).stop 2022-06-29T16:03:23.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.139+0000 7ff0f28f4700 1 -- 172.21.15.60:0/1020069950 shutdown_connections 2022-06-29T16:03:23.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.139+0000 7ff0f28f4700 1 --2- 172.21.15.60:0/1020069950 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0ec0b2c50 0x7ff0ec0b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:23.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.139+0000 7ff0f28f4700 1 --2- 172.21.15.60:0/1020069950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff0ec0b48b0 0x7ff0ec142a20 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:23.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.139+0000 7ff0f28f4700 1 -- 172.21.15.60:0/1020069950 >> 172.21.15.60:0/1020069950 conn(0x7ff0ec062090 msgr2=0x7ff0ec062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:23.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.139+0000 7ff0f28f4700 1 -- 172.21.15.60:0/1020069950 shutdown_connections 2022-06-29T16:03:23.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.143+0000 7ff0f28f4700 1 -- 172.21.15.60:0/1020069950 wait complete. 2022-06-29T16:03:23.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.143+0000 7ff0f28f4700 1 Processor -- start 2022-06-29T16:03:23.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.143+0000 7ff0f28f4700 1 -- start start 2022-06-29T16:03:23.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.143+0000 7ff0f28f4700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff0ec0b2c50 0x7ff0ec151da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:23.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.143+0000 7ff0f28f4700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0ec0b48b0 0x7ff0ec146330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:23.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.143+0000 7ff0f28f4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0ec0b5eb0 0x7ff0ec140210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:23.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.143+0000 7ff0f28f4700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ec146af0 con 0x7ff0ec0b5eb0 2022-06-29T16:03:23.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.143+0000 7ff0f28f4700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ec1443f0 con 0x7ff0ec0b2c50 2022-06-29T16:03:23.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.143+0000 7ff0f28f4700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ec144690 con 0x7ff0ec0b48b0 2022-06-29T16:03:23.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.143+0000 7ff0f20f3700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0ec0b5eb0 0x7ff0ec140210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:23.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.143+0000 7ff0f10f1700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0ec0b48b0 0x7ff0ec146330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:23.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.143+0000 7ff0f18f2700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff0ec0b2c50 0x7ff0ec151da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:23.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.143+0000 7ff0f10f1700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0ec0b48b0 0x7ff0ec146330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:33266/0 (socket says 172.21.15.60:33266) 2022-06-29T16:03:23.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.143+0000 7ff0f10f1700 1 -- 172.21.15.60:0/4082360433 learned_addr learned my addr 172.21.15.60:0/4082360433 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:23.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.143+0000 7ff0f20f3700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0ec0b5eb0 0x7ff0ec140210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48408/0 (socket says 172.21.15.60:48408) 2022-06-29T16:03:23.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.143+0000 7ff0f18f2700 1 -- 172.21.15.60:0/4082360433 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0ec0b48b0 msgr2=0x7ff0ec146330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:23.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.143+0000 7ff0f18f2700 1 --2- 172.21.15.60:0/4082360433 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0ec0b48b0 0x7ff0ec146330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:23.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.143+0000 7ff0f18f2700 1 -- 172.21.15.60:0/4082360433 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0ec0b5eb0 msgr2=0x7ff0ec140210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:23.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.143+0000 7ff0f18f2700 1 --2- 172.21.15.60:0/4082360433 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0ec0b5eb0 0x7ff0ec140210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:23.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.143+0000 7ff0f18f2700 1 -- 172.21.15.60:0/4082360433 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0dc00b180 con 0x7ff0ec0b2c50 2022-06-29T16:03:23.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.143+0000 7ff0f10f1700 1 --2- 172.21.15.60:0/4082360433 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0ec0b48b0 0x7ff0ec146330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:03:23.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.143+0000 7ff0f20f3700 1 --2- 172.21.15.60:0/4082360433 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0ec0b5eb0 0x7ff0ec140210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:03:23.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.143+0000 7ff0f18f2700 1 --2- 172.21.15.60:0/4082360433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff0ec0b2c50 0x7ff0ec151da0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7ff0ec062f00 tx=0x7ff0e8007be0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:23.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.143+0000 7ff0e2ffd700 1 -- 172.21.15.60:0/4082360433 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0e8007e50 con 0x7ff0ec0b2c50 2022-06-29T16:03:23.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.143+0000 7ff0e2ffd700 1 -- 172.21.15.60:0/4082360433 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e80070d0 con 0x7ff0ec0b2c50 2022-06-29T16:03:23.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.143+0000 7ff0f28f4700 1 -- 172.21.15.60:0/4082360433 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0e4009470 con 0x7ff0ec0b2c50 2022-06-29T16:03:23.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.143+0000 7ff0e2ffd700 1 -- 172.21.15.60:0/4082360433 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0e8007430 con 0x7ff0ec0b2c50 2022-06-29T16:03:23.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.143+0000 7ff0f28f4700 1 -- 172.21.15.60:0/4082360433 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0ec140780 con 0x7ff0ec0b2c50 2022-06-29T16:03:23.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.147+0000 7ff0f28f4700 1 -- 172.21.15.60:0/4082360433 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0ec0573d0 con 0x7ff0ec0b2c50 2022-06-29T16:03:23.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.147+0000 7ff0e2ffd700 1 -- 172.21.15.60:0/4082360433 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff0e8010070 con 0x7ff0ec0b2c50 2022-06-29T16:03:23.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.147+0000 7ff0e2ffd700 1 --2- 172.21.15.60:0/4082360433 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff0cc053a70 0x7ff0cc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:23.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.147+0000 7ff0e2ffd700 1 -- 172.21.15.60:0/4082360433 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5339+0+0 (secure 0 0 0) 0x7ff0e806afe0 con 0x7ff0ec0b2c50 2022-06-29T16:03:23.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.147+0000 7ff0f10f1700 1 --2- 172.21.15.60:0/4082360433 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff0cc053a70 0x7ff0cc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:23.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.147+0000 7ff0f10f1700 1 --2- 172.21.15.60:0/4082360433 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff0cc053a70 0x7ff0cc055f30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7ff0e401b000 tx=0x7ff0e4016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:23.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.155+0000 7ff0e2ffd700 1 -- 172.21.15.60:0/4082360433 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff0e80420d0 con 0x7ff0ec0b2c50 2022-06-29T16:03:23.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.323+0000 7ff0f28f4700 1 -- 172.21.15.60:0/4082360433 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0ec0b0cf0 con 0x7ff0ec0b2c50 2022-06-29T16:03:23.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.323+0000 7ff0e2ffd700 1 -- 172.21.15.60:0/4082360433 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+1209 (secure 0 0 0) 0x7ff0ec0b0cf0 con 0x7ff0ec0b2c50 2022-06-29T16:03:23.332 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:03:23.332 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":22,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5568,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/4229695218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":4229695218},{"type":"v1","addr":"172.21.15.60:6833","nonce":4229695218}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":18},{"gid":5932,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/1745041125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1745041125},{"type":"v1","addr":"172.21.15.92:6835","nonce":1745041125}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":22}],"filesystems":[]} 2022-06-29T16:03:23.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.327+0000 7ff0f28f4700 1 -- 172.21.15.60:0/4082360433 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff0cc053a70 msgr2=0x7ff0cc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:23.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.327+0000 7ff0f28f4700 1 --2- 172.21.15.60:0/4082360433 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff0cc053a70 0x7ff0cc055f30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7ff0e401b000 tx=0x7ff0e4016000).stop 2022-06-29T16:03:23.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.327+0000 7ff0f28f4700 1 -- 172.21.15.60:0/4082360433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff0ec0b2c50 msgr2=0x7ff0ec151da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:23.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.327+0000 7ff0f28f4700 1 --2- 172.21.15.60:0/4082360433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff0ec0b2c50 0x7ff0ec151da0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7ff0ec062f00 tx=0x7ff0e8007be0).stop 2022-06-29T16:03:23.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.327+0000 7ff0f28f4700 1 -- 172.21.15.60:0/4082360433 shutdown_connections 2022-06-29T16:03:23.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.327+0000 7ff0f28f4700 1 --2- 172.21.15.60:0/4082360433 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff0cc053a70 0x7ff0cc055f30 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:23.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.327+0000 7ff0f28f4700 1 --2- 172.21.15.60:0/4082360433 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0ec0b5eb0 0x7ff0ec140210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:23.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.327+0000 7ff0f28f4700 1 --2- 172.21.15.60:0/4082360433 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0ec0b48b0 0x7ff0ec146330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:23.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.327+0000 7ff0f28f4700 1 --2- 172.21.15.60:0/4082360433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff0ec0b2c50 0x7ff0ec151da0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:23.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.327+0000 7ff0f28f4700 1 -- 172.21.15.60:0/4082360433 >> 172.21.15.60:0/4082360433 conn(0x7ff0ec062090 msgr2=0x7ff0ec141ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:23.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.327+0000 7ff0f28f4700 1 -- 172.21.15.60:0/4082360433 shutdown_connections 2022-06-29T16:03:23.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.327+0000 7ff0f28f4700 1 -- 172.21.15.60:0/4082360433 wait complete. 2022-06-29T16:03:23.337 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 22 2022-06-29T16:03:23.345 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:03:23.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.443+0000 7fc2991fb700 1 Processor -- start 2022-06-29T16:03:23.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.443+0000 7fc2991fb700 1 -- start start 2022-06-29T16:03:23.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.443+0000 7fc2991fb700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2940b2130 0x7fc294146100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:23.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.443+0000 7fc2991fb700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc2940aed00 0x7fc2941465f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:23.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.443+0000 7fc2991fb700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc29405d450 con 0x7fc29405d070 2022-06-29T16:03:23.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.443+0000 7fc2991fb700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2940adf90 con 0x7fc2940aed00 2022-06-29T16:03:23.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.443+0000 7fc2991fb700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2940b25d0 con 0x7fc2940b2130 2022-06-29T16:03:23.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.447+0000 7fc2989fa700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc2940aed00 0x7fc2941465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:23.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.447+0000 7fc2989fa700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc2940aed00 0x7fc2941465f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48414/0 (socket says 172.21.15.60:48414) 2022-06-29T16:03:23.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.447+0000 7fc2989fa700 1 -- 172.21.15.60:0/1188820505 learned_addr learned my addr 172.21.15.60:0/1188820505 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:23.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.447+0000 7fc293fff700 1 --1- 172.21.15.60:0/1188820505 >> v1:172.21.15.60:6789/0 conn(0x7fc29405d070 0x7fc29414aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56498/0 (socket says 172.21.15.60:56498) 2022-06-29T16:03:23.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.447+0000 7fc28bfff700 1 --2- 172.21.15.60:0/1188820505 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2940b2130 0x7fc294146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:23.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.447+0000 7fc2989fa700 1 -- 172.21.15.60:0/1188820505 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2940b2130 msgr2=0x7fc294146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:23.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.447+0000 7fc2989fa700 1 --2- 172.21.15.60:0/1188820505 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2940b2130 0x7fc294146100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:23.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.447+0000 7fc2989fa700 1 -- 172.21.15.60:0/1188820505 >> v1:172.21.15.60:6789/0 conn(0x7fc29405d070 legacy=0x7fc29414aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:23.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.447+0000 7fc2937fe700 1 -- 172.21.15.60:0/1188820505 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4056906262 0 0) 0x7fc29405d450 con 0x7fc29405d070 2022-06-29T16:03:23.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.447+0000 7fc2989fa700 1 -- 172.21.15.60:0/1188820505 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc29414a350 con 0x7fc2940aed00 2022-06-29T16:03:23.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.447+0000 7fc28bfff700 1 --2- 172.21.15.60:0/1188820505 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2940b2130 0x7fc294146100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:03:23.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.447+0000 7fc2989fa700 1 --2- 172.21.15.60:0/1188820505 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2940aed00 0x7fc2941465f0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x7fc28c0032f0 tx=0x7fc28c009290).ready entity=mon.0 client_cookie=b096703206e8c6c7 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:23.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.447+0000 7fc2937fe700 1 -- 172.21.15.60:0/1188820505 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc28c00b8a0 con 0x7fc2940aed00 2022-06-29T16:03:23.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.447+0000 7fc2937fe700 1 -- 172.21.15.60:0/1188820505 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc28c00ba00 con 0x7fc2940aed00 2022-06-29T16:03:23.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.447+0000 7fc2937fe700 1 -- 172.21.15.60:0/1188820505 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc28c00bb70 con 0x7fc2940aed00 2022-06-29T16:03:23.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.447+0000 7fc2991fb700 1 -- 172.21.15.60:0/1188820505 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2940aed00 msgr2=0x7fc2941465f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:23.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.447+0000 7fc2991fb700 1 --2- 172.21.15.60:0/1188820505 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2940aed00 0x7fc2941465f0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x7fc28c0032f0 tx=0x7fc28c009290).stop 2022-06-29T16:03:23.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.447+0000 7fc2991fb700 1 -- 172.21.15.60:0/1188820505 shutdown_connections 2022-06-29T16:03:23.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.447+0000 7fc2991fb700 1 --2- 172.21.15.60:0/1188820505 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2940aed00 0x7fc2941465f0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:23.461 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.447+0000 7fc2991fb700 1 --2- 172.21.15.60:0/1188820505 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2940b2130 0x7fc294146100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:23.461 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.447+0000 7fc2991fb700 1 -- 172.21.15.60:0/1188820505 >> 172.21.15.60:0/1188820505 conn(0x7fc2940b5890 msgr2=0x7fc2940b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:23.461 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.447+0000 7fc2991fb700 1 -- 172.21.15.60:0/1188820505 shutdown_connections 2022-06-29T16:03:23.461 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.447+0000 7fc2991fb700 1 -- 172.21.15.60:0/1188820505 wait complete. 2022-06-29T16:03:23.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.447+0000 7fc2991fb700 1 Processor -- start 2022-06-29T16:03:23.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.447+0000 7fc2991fb700 1 -- start start 2022-06-29T16:03:23.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.451+0000 7fc2991fb700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc29405d070 0x7fc294118e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:23.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.451+0000 7fc2991fb700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2940aed00 0x7fc294066440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:23.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.451+0000 7fc2991fb700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2940b2130 0x7fc294116920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:23.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.451+0000 7fc2991fb700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc294118490 con 0x7fc2940aed00 2022-06-29T16:03:23.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.451+0000 7fc2991fb700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc294118730 con 0x7fc29405d070 2022-06-29T16:03:23.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.451+0000 7fc2991fb700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2941189d0 con 0x7fc2940b2130 2022-06-29T16:03:23.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.451+0000 7fc28bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2940aed00 0x7fc294066440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:23.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.451+0000 7fc28bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2940aed00 0x7fc294066440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48418/0 (socket says 172.21.15.60:48418) 2022-06-29T16:03:23.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.451+0000 7fc28bfff700 1 -- 172.21.15.60:0/288004618 learned_addr learned my addr 172.21.15.60:0/288004618 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:23.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.451+0000 7fc2989fa700 1 --2- 172.21.15.60:0/288004618 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2940b2130 0x7fc294116920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:23.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.451+0000 7fc293fff700 1 --2- 172.21.15.60:0/288004618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc29405d070 0x7fc294118e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:23.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.451+0000 7fc28bfff700 1 -- 172.21.15.60:0/288004618 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2940b2130 msgr2=0x7fc294116920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:23.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.451+0000 7fc28bfff700 1 --2- 172.21.15.60:0/288004618 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2940b2130 0x7fc294116920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:23.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.451+0000 7fc28bfff700 1 -- 172.21.15.60:0/288004618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc29405d070 msgr2=0x7fc294118e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:23.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.451+0000 7fc28bfff700 1 --2- 172.21.15.60:0/288004618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc29405d070 0x7fc294118e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:23.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.451+0000 7fc28bfff700 1 -- 172.21.15.60:0/288004618 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc27c00b180 con 0x7fc2940aed00 2022-06-29T16:03:23.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.451+0000 7fc28bfff700 1 --2- 172.21.15.60:0/288004618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2940aed00 0x7fc294066440 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7fc27c00c000 tx=0x7fc27c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:23.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.451+0000 7fc291ffb700 1 -- 172.21.15.60:0/288004618 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc27c010780 con 0x7fc2940aed00 2022-06-29T16:03:23.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.451+0000 7fc291ffb700 1 -- 172.21.15.60:0/288004618 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc27c0108e0 con 0x7fc2940aed00 2022-06-29T16:03:23.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.451+0000 7fc2991fb700 1 -- 172.21.15.60:0/288004618 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc28c009530 con 0x7fc2940aed00 2022-06-29T16:03:23.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.451+0000 7fc291ffb700 1 -- 172.21.15.60:0/288004618 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc27c010c40 con 0x7fc2940aed00 2022-06-29T16:03:23.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.451+0000 7fc2991fb700 1 -- 172.21.15.60:0/288004618 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc294066b40 con 0x7fc2940aed00 2022-06-29T16:03:23.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.451+0000 7fc2991fb700 1 -- 172.21.15.60:0/288004618 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc29414a5f0 con 0x7fc2940aed00 2022-06-29T16:03:23.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.455+0000 7fc291ffb700 1 -- 172.21.15.60:0/288004618 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc27c01c460 con 0x7fc2940aed00 2022-06-29T16:03:23.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.455+0000 7fc291ffb700 1 --2- 172.21.15.60:0/288004618 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc2780539e0 0x7fc278055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:23.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.455+0000 7fc291ffb700 1 -- 172.21.15.60:0/288004618 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fc27c06c0c0 con 0x7fc2940aed00 2022-06-29T16:03:23.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.455+0000 7fc293fff700 1 --2- 172.21.15.60:0/288004618 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc2780539e0 0x7fc278055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:23.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.455+0000 7fc293fff700 1 --2- 172.21.15.60:0/288004618 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc2780539e0 0x7fc278055ea0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fc28400d000 tx=0x7fc284009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:23.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.463+0000 7fc291ffb700 1 -- 172.21.15.60:0/288004618 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc27c043260 con 0x7fc2940aed00 2022-06-29T16:03:23.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.627+0000 7fc2991fb700 1 -- 172.21.15.60:0/288004618 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2940573d0 con 0x7fc2940aed00 2022-06-29T16:03:23.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.627+0000 7fc291ffb700 1 -- 172.21.15.60:0/288004618 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+1209 (secure 0 0 0) 0x7fc27c046880 con 0x7fc2940aed00 2022-06-29T16:03:23.635 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:03:23.635 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":22,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5568,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/4229695218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":4229695218},{"type":"v1","addr":"172.21.15.60:6833","nonce":4229695218}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":18},{"gid":5932,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/1745041125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1745041125},{"type":"v1","addr":"172.21.15.92:6835","nonce":1745041125}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":22}],"filesystems":[]} 2022-06-29T16:03:23.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.627+0000 7fc2991fb700 1 -- 172.21.15.60:0/288004618 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc2780539e0 msgr2=0x7fc278055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:23.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.627+0000 7fc2991fb700 1 --2- 172.21.15.60:0/288004618 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc2780539e0 0x7fc278055ea0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fc28400d000 tx=0x7fc284009040).stop 2022-06-29T16:03:23.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.627+0000 7fc2991fb700 1 -- 172.21.15.60:0/288004618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2940aed00 msgr2=0x7fc294066440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:23.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.627+0000 7fc2991fb700 1 --2- 172.21.15.60:0/288004618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2940aed00 0x7fc294066440 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7fc27c00c000 tx=0x7fc27c006df0).stop 2022-06-29T16:03:23.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.627+0000 7fc2991fb700 1 -- 172.21.15.60:0/288004618 shutdown_connections 2022-06-29T16:03:23.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.627+0000 7fc2991fb700 1 --2- 172.21.15.60:0/288004618 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc2780539e0 0x7fc278055ea0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:23.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.627+0000 7fc2991fb700 1 --2- 172.21.15.60:0/288004618 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2940b2130 0x7fc294116920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:23.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.627+0000 7fc2991fb700 1 --2- 172.21.15.60:0/288004618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2940aed00 0x7fc294066440 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:23.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.631+0000 7fc2991fb700 1 --2- 172.21.15.60:0/288004618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc29405d070 0x7fc294118e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:23.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.631+0000 7fc2991fb700 1 -- 172.21.15.60:0/288004618 >> 172.21.15.60:0/288004618 conn(0x7fc2940b5890 msgr2=0x7fc2941453b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:23.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.631+0000 7fc2991fb700 1 -- 172.21.15.60:0/288004618 shutdown_connections 2022-06-29T16:03:23.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.631+0000 7fc2991fb700 1 -- 172.21.15.60:0/288004618 wait complete. 2022-06-29T16:03:23.639 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 22 2022-06-29T16:03:23.647 INFO:tasks.ceph.mds.a:Restarting daemon 2022-06-29T16:03:23.647 INFO:tasks.ceph.mds.a:Stopping old one... 2022-06-29T16:03:23.648 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-06-29T16:03:23.648 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T16:03:23.648 INFO:tasks.ceph.mds.b:Restarting daemon 2022-06-29T16:03:23.649 INFO:tasks.ceph.mds.b:Stopping old one... 2022-06-29T16:03:23.650 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-06-29T16:03:23.650 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T16:03:23.663 INFO:tasks.ceph.mds.a:Stopped 2022-06-29T16:03:23.663 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-06-29T16:03:23.666 INFO:tasks.ceph.mds.b:Stopped 2022-06-29T16:03:23.666 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-06-29T16:03:23.667 INFO:tasks.ceph.mds.a:Started 2022-06-29T16:03:23.668 DEBUG:teuthology.parallel:result is None 2022-06-29T16:03:23.670 INFO:tasks.ceph.mds.b:Started 2022-06-29T16:03:23.670 DEBUG:teuthology.parallel:result is None 2022-06-29T16:03:23.671 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-06-29T16:03:23.744 INFO:tasks.ceph.mds.b.smithi092.stdout:starting mds.b at 2022-06-29T16:03:23.748 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2022-06-29T16:03:23.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.763+0000 7ff231f85700 1 Processor -- start 2022-06-29T16:03:23.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.763+0000 7ff231f85700 1 -- start start 2022-06-29T16:03:23.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.763+0000 7ff231f85700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff22c067a90 0x7ff22c1400b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:23.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.763+0000 7ff231f85700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff22c1405a0 0x7ff22c143fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:23.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.763+0000 7ff231f85700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff22c064e20 con 0x7ff22c05fb00 2022-06-29T16:03:23.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.763+0000 7ff231f85700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff22c0b10e0 con 0x7ff22c1405a0 2022-06-29T16:03:23.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.763+0000 7ff231f85700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff22c05fee0 con 0x7ff22c067a90 2022-06-29T16:03:23.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.763+0000 7ff230f83700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff22c05fb00 0x7ff22c13f9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56522/0 (socket says 172.21.15.60:56522) 2022-06-29T16:03:23.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.763+0000 7ff230f83700 1 -- 172.21.15.60:0/3353931211 learned_addr learned my addr 172.21.15.60:0/3353931211 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:03:23.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.763+0000 7ff231784700 1 --2- 172.21.15.60:0/3353931211 >> v2:172.21.15.60:3300/0 conn(0x7ff22c1405a0 0x7ff22c143fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:23.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.763+0000 7ff22bfff700 1 --2- 172.21.15.60:0/3353931211 >> v2:172.21.15.92:3300/0 conn(0x7ff22c067a90 0x7ff22c1400b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:23.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.763+0000 7ff22b7fe700 1 -- 172.21.15.60:0/3353931211 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4223575457 0 0) 0x7ff22c064e20 con 0x7ff22c05fb00 2022-06-29T16:03:23.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.763+0000 7ff22b7fe700 1 -- 172.21.15.60:0/3353931211 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff210003260 con 0x7ff22c05fb00 2022-06-29T16:03:23.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.763+0000 7ff231784700 1 -- 172.21.15.60:0/3353931211 >> v2:172.21.15.92:3300/0 conn(0x7ff22c067a90 msgr2=0x7ff22c1400b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:23.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.763+0000 7ff231784700 1 --2- 172.21.15.60:0/3353931211 >> v2:172.21.15.92:3300/0 conn(0x7ff22c067a90 0x7ff22c1400b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:23.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.763+0000 7ff231784700 1 -- 172.21.15.60:0/3353931211 >> v1:172.21.15.60:6789/0 conn(0x7ff22c05fb00 legacy=0x7ff22c13f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:23.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.763+0000 7ff231784700 1 -- 172.21.15.60:0/3353931211 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff22c060b20 con 0x7ff22c1405a0 2022-06-29T16:03:23.777 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.763+0000 7ff231784700 1 --2- 172.21.15.60:0/3353931211 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff22c1405a0 0x7ff22c143fa0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7ff2240032f0 tx=0x7ff2240091d0).ready entity=mon.0 client_cookie=985420e01526888b server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:23.777 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.763+0000 7ff22b7fe700 1 -- 172.21.15.60:0/3353931211 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff22400b600 con 0x7ff22c1405a0 2022-06-29T16:03:23.777 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.763+0000 7ff22b7fe700 1 -- 172.21.15.60:0/3353931211 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff22400b760 con 0x7ff22c1405a0 2022-06-29T16:03:23.777 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.763+0000 7ff22b7fe700 1 -- 172.21.15.60:0/3353931211 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff22400b8d0 con 0x7ff22c1405a0 2022-06-29T16:03:23.778 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.767+0000 7ff231f85700 1 -- 172.21.15.60:0/3353931211 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff22c1405a0 msgr2=0x7ff22c143fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:23.778 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.767+0000 7ff231f85700 1 --2- 172.21.15.60:0/3353931211 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff22c1405a0 0x7ff22c143fa0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7ff2240032f0 tx=0x7ff2240091d0).stop 2022-06-29T16:03:23.778 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.767+0000 7ff231f85700 1 -- 172.21.15.60:0/3353931211 shutdown_connections 2022-06-29T16:03:23.778 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.767+0000 7ff231f85700 1 --2- 172.21.15.60:0/3353931211 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff22c1405a0 0x7ff22c143fa0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:23.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.767+0000 7ff231f85700 1 --2- 172.21.15.60:0/3353931211 >> v2:172.21.15.92:3300/0 conn(0x7ff22c067a90 0x7ff22c1400b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:23.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.767+0000 7ff231f85700 1 -- 172.21.15.60:0/3353931211 >> 172.21.15.60:0/3353931211 conn(0x7ff22c063a60 msgr2=0x7ff22c05f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:23.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.767+0000 7ff231f85700 1 -- 172.21.15.60:0/3353931211 shutdown_connections 2022-06-29T16:03:23.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.767+0000 7ff231f85700 1 -- 172.21.15.60:0/3353931211 wait complete. 2022-06-29T16:03:23.783 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.767+0000 7ff231f85700 1 Processor -- start 2022-06-29T16:03:23.783 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.767+0000 7ff231f85700 1 -- start start 2022-06-29T16:03:23.783 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.771+0000 7ff231f85700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff22c05fb00 0x7ff22c0b4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:23.783 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.771+0000 7ff231f85700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff22c067a90 0x7ff22c144030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:23.784 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.771+0000 7ff231f85700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff22c1405a0 0x7ff22c144520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:23.784 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.771+0000 7ff231f85700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22c067460 con 0x7ff22c1405a0 2022-06-29T16:03:23.784 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.771+0000 7ff231f85700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22c0671c0 con 0x7ff22c067a90 2022-06-29T16:03:23.784 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.771+0000 7ff231f85700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff22c142f60 con 0x7ff22c05fb00 2022-06-29T16:03:23.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.771+0000 7ff231784700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff22c1405a0 0x7ff22c144520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:23.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.771+0000 7ff231784700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff22c1405a0 0x7ff22c144520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48446/0 (socket says 172.21.15.60:48446) 2022-06-29T16:03:23.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.771+0000 7ff231784700 1 -- 172.21.15.60:0/1920884630 learned_addr learned my addr 172.21.15.60:0/1920884630 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:23.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.771+0000 7ff230f83700 1 --2- 172.21.15.60:0/1920884630 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff22c05fb00 0x7ff22c0b4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:23.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.771+0000 7ff22bfff700 1 --2- 172.21.15.60:0/1920884630 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff22c067a90 0x7ff22c144030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:23.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.771+0000 7ff231784700 1 -- 172.21.15.60:0/1920884630 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff22c05fb00 msgr2=0x7ff22c0b4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:23.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.771+0000 7ff231784700 1 --2- 172.21.15.60:0/1920884630 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff22c05fb00 0x7ff22c0b4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:23.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.771+0000 7ff231784700 1 -- 172.21.15.60:0/1920884630 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff22c067a90 msgr2=0x7ff22c144030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:23.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.771+0000 7ff231784700 1 --2- 172.21.15.60:0/1920884630 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff22c067a90 0x7ff22c144030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:23.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.771+0000 7ff231784700 1 -- 172.21.15.60:0/1920884630 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff224009470 con 0x7ff22c1405a0 2022-06-29T16:03:23.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.771+0000 7ff22bfff700 1 --2- 172.21.15.60:0/1920884630 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff22c067a90 0x7ff22c144030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:03:23.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.771+0000 7ff230f83700 1 --2- 172.21.15.60:0/1920884630 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff22c05fb00 0x7ff22c0b4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:03:23.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.771+0000 7ff231784700 1 --2- 172.21.15.60:0/1920884630 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff22c1405a0 0x7ff22c144520 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0x7ff2240032f0 tx=0x7ff224016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:23.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.771+0000 7ff229ffb700 1 -- 172.21.15.60:0/1920884630 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff22400b8d0 con 0x7ff22c1405a0 2022-06-29T16:03:23.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.771+0000 7ff229ffb700 1 -- 172.21.15.60:0/1920884630 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff22401f5e0 con 0x7ff22c1405a0 2022-06-29T16:03:23.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.771+0000 7ff231f85700 1 -- 172.21.15.60:0/1920884630 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff22c0aca20 con 0x7ff22c1405a0 2022-06-29T16:03:23.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.771+0000 7ff229ffb700 1 -- 172.21.15.60:0/1920884630 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff22400b600 con 0x7ff22c1405a0 2022-06-29T16:03:23.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.771+0000 7ff231f85700 1 -- 172.21.15.60:0/1920884630 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff22c0ae9a0 con 0x7ff22c1405a0 2022-06-29T16:03:23.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.771+0000 7ff231f85700 1 -- 172.21.15.60:0/1920884630 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff22c0573d0 con 0x7ff22c1405a0 2022-06-29T16:03:23.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.771+0000 7ff229ffb700 1 -- 172.21.15.60:0/1920884630 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff224026030 con 0x7ff22c1405a0 2022-06-29T16:03:23.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.775+0000 7ff229ffb700 1 --2- 172.21.15.60:0/1920884630 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff20c053a70 0x7ff20c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:23.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.775+0000 7ff229ffb700 1 -- 172.21.15.60:0/1920884630 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5339+0+0 (secure 0 0 0) 0x7ff22406c680 con 0x7ff22c1405a0 2022-06-29T16:03:23.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.775+0000 7ff230f83700 1 --2- 172.21.15.60:0/1920884630 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff20c053a70 0x7ff20c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:23.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.775+0000 7ff230f83700 1 --2- 172.21.15.60:0/1920884630 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff20c053a70 0x7ff20c055f30 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7ff22000b000 tx=0x7ff220009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:23.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.779+0000 7ff229ffb700 1 -- 172.21.15.60:0/1920884630 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff224043860 con 0x7ff22c1405a0 2022-06-29T16:03:23.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:23.919+0000 7ff231f85700 1 -- 172.21.15.60:0/1920884630 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7ff22c0b2ec0 con 0x7ff22c1405a0 2022-06-29T16:03:24.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.847+0000 7ff229ffb700 1 -- 172.21.15.60:0/1920884630 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v57) v1 ==== 97+0+0 (secure 0 0 0) 0x7ff224024080 con 0x7ff22c1405a0 2022-06-29T16:03:24.861 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.855+0000 7ff231f85700 1 -- 172.21.15.60:0/1920884630 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff20c053a70 msgr2=0x7ff20c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:24.861 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.855+0000 7ff231f85700 1 --2- 172.21.15.60:0/1920884630 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff20c053a70 0x7ff20c055f30 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7ff22000b000 tx=0x7ff220009040).stop 2022-06-29T16:03:24.861 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.855+0000 7ff231f85700 1 -- 172.21.15.60:0/1920884630 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff22c1405a0 msgr2=0x7ff22c144520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:24.862 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.855+0000 7ff231f85700 1 --2- 172.21.15.60:0/1920884630 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff22c1405a0 0x7ff22c144520 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0x7ff2240032f0 tx=0x7ff224016000).stop 2022-06-29T16:03:24.862 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.855+0000 7ff231f85700 1 -- 172.21.15.60:0/1920884630 shutdown_connections 2022-06-29T16:03:24.862 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.855+0000 7ff231f85700 1 --2- 172.21.15.60:0/1920884630 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff20c053a70 0x7ff20c055f30 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:24.863 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.855+0000 7ff231f85700 1 --2- 172.21.15.60:0/1920884630 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff22c1405a0 0x7ff22c144520 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:24.863 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.855+0000 7ff231f85700 1 --2- 172.21.15.60:0/1920884630 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff22c067a90 0x7ff22c144030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:24.863 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.855+0000 7ff231f85700 1 --2- 172.21.15.60:0/1920884630 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff22c05fb00 0x7ff22c0b4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:24.863 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.855+0000 7ff231f85700 1 -- 172.21.15.60:0/1920884630 >> 172.21.15.60:0/1920884630 conn(0x7ff22c063a60 msgr2=0x7ff22c1410b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:24.863 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.855+0000 7ff231f85700 1 -- 172.21.15.60:0/1920884630 shutdown_connections 2022-06-29T16:03:24.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.855+0000 7ff231f85700 1 -- 172.21.15.60:0/1920884630 wait complete. 2022-06-29T16:03:24.864 INFO:teuthology.orchestra.run.smithi060.stderr: removed all blacklist entries 2022-06-29T16:03:24.877 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-06-29T16:03:24.877 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-06-29T16:03:24.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.975+0000 7f34396c6700 1 Processor -- start 2022-06-29T16:03:24.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.975+0000 7f34396c6700 1 -- start start 2022-06-29T16:03:24.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.975+0000 7f34396c6700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f34340b2130 0x7f3434146100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:24.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.975+0000 7f34396c6700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f34340aed00 0x7f34341465f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:24.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.975+0000 7f34396c6700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f343405d450 con 0x7f343405d070 2022-06-29T16:03:24.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.975+0000 7f34396c6700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34340adf90 con 0x7f34340b2130 2022-06-29T16:03:24.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.975+0000 7f34396c6700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34340b25d0 con 0x7f34340aed00 2022-06-29T16:03:24.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.975+0000 7f34337fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f34340b2130 0x7f3434146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:24.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.979+0000 7f34337fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f34340b2130 0x7f3434146100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48452/0 (socket says 172.21.15.60:48452) 2022-06-29T16:03:24.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.979+0000 7f34337fe700 1 -- 172.21.15.60:0/1522091357 learned_addr learned my addr 172.21.15.60:0/1522091357 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:24.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.979+0000 7f3438ec5700 1 --2- 172.21.15.60:0/1522091357 >> v2:172.21.15.92:3300/0 conn(0x7f34340aed00 0x7f34341465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:24.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.979+0000 7f34337fe700 1 -- 172.21.15.60:0/1522091357 >> v2:172.21.15.92:3300/0 conn(0x7f34340aed00 msgr2=0x7f34341465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:24.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.979+0000 7f34337fe700 1 --2- 172.21.15.60:0/1522091357 >> v2:172.21.15.92:3300/0 conn(0x7f34340aed00 0x7f34341465f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:24.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.979+0000 7f34337fe700 1 -- 172.21.15.60:0/1522091357 >> v1:172.21.15.92:6789/0 conn(0x7f343405d070 legacy=0x7f343414aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:24.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.979+0000 7f34337fe700 1 -- 172.21.15.60:0/1522091357 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f343414a350 con 0x7f34340b2130 2022-06-29T16:03:24.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.979+0000 7f3438ec5700 1 --2- 172.21.15.60:0/1522091357 >> v2:172.21.15.92:3300/0 conn(0x7f34340aed00 0x7f34341465f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:03:24.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.979+0000 7f34337fe700 1 --2- 172.21.15.60:0/1522091357 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34340b2130 0x7f3434146100 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0x7f3424002f70 tx=0x7f3424006eb0).ready entity=mon.0 client_cookie=a96cd6bbab5b016 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:24.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.979+0000 7f3432ffd700 1 -- 172.21.15.60:0/1522091357 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f342400d610 con 0x7f34340b2130 2022-06-29T16:03:24.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.979+0000 7f3432ffd700 1 -- 172.21.15.60:0/1522091357 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3424000e90 con 0x7f34340b2130 2022-06-29T16:03:24.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.979+0000 7f3432ffd700 1 -- 172.21.15.60:0/1522091357 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f342400d9a0 con 0x7f34340b2130 2022-06-29T16:03:24.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.979+0000 7f34396c6700 1 -- 172.21.15.60:0/1522091357 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34340b2130 msgr2=0x7f3434146100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:24.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.979+0000 7f34396c6700 1 --2- 172.21.15.60:0/1522091357 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34340b2130 0x7f3434146100 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0x7f3424002f70 tx=0x7f3424006eb0).stop 2022-06-29T16:03:24.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.979+0000 7f34396c6700 1 -- 172.21.15.60:0/1522091357 shutdown_connections 2022-06-29T16:03:24.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.979+0000 7f34396c6700 1 --2- 172.21.15.60:0/1522091357 >> v2:172.21.15.92:3300/0 conn(0x7f34340aed00 0x7f34341465f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:24.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.979+0000 7f34396c6700 1 --2- 172.21.15.60:0/1522091357 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f34340b2130 0x7f3434146100 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:24.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.979+0000 7f34396c6700 1 -- 172.21.15.60:0/1522091357 >> 172.21.15.60:0/1522091357 conn(0x7f34340b5890 msgr2=0x7f34340b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:24.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.979+0000 7f34396c6700 1 -- 172.21.15.60:0/1522091357 shutdown_connections 2022-06-29T16:03:24.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.979+0000 7f34396c6700 1 -- 172.21.15.60:0/1522091357 wait complete. 2022-06-29T16:03:24.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.979+0000 7f34396c6700 1 Processor -- start 2022-06-29T16:03:24.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.979+0000 7f34396c6700 1 -- start start 2022-06-29T16:03:24.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.983+0000 7f34396c6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f343405d070 0x7f34340642c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:24.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.983+0000 7f34396c6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f34340aed00 0x7f3434063400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:24.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.983+0000 7f34396c6700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f34340b2130 0x7f3434065d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:24.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.983+0000 7f34396c6700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34340661f0 con 0x7f343405d070 2022-06-29T16:03:24.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.983+0000 7f34396c6700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34341196a0 con 0x7f34340b2130 2022-06-29T16:03:24.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.983+0000 7f34396c6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3434119940 con 0x7f34340aed00 2022-06-29T16:03:24.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.983+0000 7f3433fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f343405d070 0x7f34340642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:24.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.983+0000 7f3433fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f343405d070 0x7f34340642c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48456/0 (socket says 172.21.15.60:48456) 2022-06-29T16:03:24.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.983+0000 7f3433fff700 1 -- 172.21.15.60:0/1369634649 learned_addr learned my addr 172.21.15.60:0/1369634649 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:24.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.983+0000 7f34337fe700 1 --2- 172.21.15.60:0/1369634649 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f34340aed00 0x7f3434063400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:24.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.983+0000 7f3438ec5700 1 --2- 172.21.15.60:0/1369634649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f34340b2130 0x7f3434065d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:24.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.983+0000 7f3433fff700 1 -- 172.21.15.60:0/1369634649 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f34340aed00 msgr2=0x7f3434063400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:24.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.983+0000 7f3433fff700 1 --2- 172.21.15.60:0/1369634649 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f34340aed00 0x7f3434063400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:24.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.983+0000 7f3433fff700 1 -- 172.21.15.60:0/1369634649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f34340b2130 msgr2=0x7f3434065d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:24.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.983+0000 7f3433fff700 1 --2- 172.21.15.60:0/1369634649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f34340b2130 0x7f3434065d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:24.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.983+0000 7f3433fff700 1 -- 172.21.15.60:0/1369634649 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f342400b1e0 con 0x7f343405d070 2022-06-29T16:03:24.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.983+0000 7f3433fff700 1 --2- 172.21.15.60:0/1369634649 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f343405d070 0x7f34340642c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7f3434061990 tx=0x7f3428007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:24.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.983+0000 7f34317fa700 1 -- 172.21.15.60:0/1369634649 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3428012400 con 0x7f343405d070 2022-06-29T16:03:24.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.983+0000 7f34317fa700 1 -- 172.21.15.60:0/1369634649 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3428012560 con 0x7f343405d070 2022-06-29T16:03:25.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.983+0000 7f34396c6700 1 -- 172.21.15.60:0/1369634649 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34341598f0 con 0x7f343405d070 2022-06-29T16:03:25.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.983+0000 7f34317fa700 1 -- 172.21.15.60:0/1369634649 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34280128c0 con 0x7f343405d070 2022-06-29T16:03:25.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.983+0000 7f34396c6700 1 -- 172.21.15.60:0/1369634649 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3434066d90 con 0x7f343405d070 2022-06-29T16:03:25.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.983+0000 7f34396c6700 1 -- 172.21.15.60:0/1369634649 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f343414a5f0 con 0x7f343405d070 2022-06-29T16:03:25.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.987+0000 7f34317fa700 1 -- 172.21.15.60:0/1369634649 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3428019030 con 0x7f343405d070 2022-06-29T16:03:25.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.987+0000 7f34317fa700 1 --2- 172.21.15.60:0/1369634649 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3414053b00 0x7f3414055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:25.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.987+0000 7f34317fa700 1 -- 172.21.15.60:0/1369634649 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f342806ae40 con 0x7f343405d070 2022-06-29T16:03:25.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.987+0000 7f34337fe700 1 --2- 172.21.15.60:0/1369634649 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3414053b00 0x7f3414055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:25.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.987+0000 7f34337fe700 1 --2- 172.21.15.60:0/1369634649 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3414053b00 0x7f3414055fc0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f342401c000 tx=0x7f3424015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:25.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:24.991+0000 7f34317fa700 1 -- 172.21.15.60:0/1369634649 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3428042090 con 0x7f343405d070 2022-06-29T16:03:25.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.163+0000 7f34396c6700 1 -- 172.21.15.60:0/1369634649 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f34340573d0 con 0x7f343405d070 2022-06-29T16:03:25.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.163+0000 7f34317fa700 1 -- 172.21.15.60:0/1369634649 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v21) v1 ==== 80+0+4242 (secure 0 0 0) 0x7f34280456b0 con 0x7f343405d070 2022-06-29T16:03:25.172 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:03:25.173 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:03:25.173 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_dump": [ 2022-06-29T16:03:25.174 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:25.174 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.a", 2022-06-29T16:03:25.175 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxivv7JNhAAnA291UxlB/SfgITOipP00A==", 2022-06-29T16:03:25.175 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:03:25.176 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:03:25.176 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:03:25.176 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:03:25.177 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:03:25.177 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:25.178 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:25.178 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:25.179 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.b", 2022-06-29T16:03:25.179 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxiRSr/OhAAfFQTF2YHvzCv5wXBVCA5WQ==", 2022-06-29T16:03:25.180 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:03:25.180 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:03:25.181 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:03:25.181 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:03:25.182 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:03:25.182 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:25.183 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:25.183 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:25.183 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.0", 2022-06-29T16:03:25.184 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiZYPxOBAAIkweWJlTJIXgxnlagkGdXA==", 2022-06-29T16:03:25.184 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:03:25.185 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:03:25.185 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:03:25.186 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:03:25.187 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:25.187 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:25.188 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:25.188 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.1", 2022-06-29T16:03:25.188 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxiS2JGBhAAJn0UfAxZI3IVFyD5QBdb/Q==", 2022-06-29T16:03:25.189 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:03:25.189 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:03:25.190 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:03:25.191 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:03:25.191 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:25.192 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:25.192 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:25.193 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.2", 2022-06-29T16:03:25.193 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxiPhkfEhAApH7r8OUJzW7fZNM6XVv1dg==", 2022-06-29T16:03:25.194 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:03:25.194 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:03:25.195 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:03:25.195 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:03:25.195 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:25.196 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:25.196 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:25.197 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.3", 2022-06-29T16:03:25.197 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxit/47HhAA1vyU6h7hjPG+op9qP/rVYQ==", 2022-06-29T16:03:25.198 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:03:25.198 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:03:25.198 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:03:25.199 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:03:25.199 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:25.199 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:25.199 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:25.200 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.4", 2022-06-29T16:03:25.200 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxiaKzKBxAAbIwCF9ag7ezsNy11hi0BEw==", 2022-06-29T16:03:25.200 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:03:25.200 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:03:25.200 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:03:25.201 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:03:25.201 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:25.201 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:25.201 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:25.201 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.5", 2022-06-29T16:03:25.202 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxiJQV/IBAASBBDb6UFgEtXILe4fcT/xA==", 2022-06-29T16:03:25.202 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:03:25.202 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:03:25.202 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:03:25.203 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:03:25.203 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:25.203 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:25.203 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:25.203 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.6", 2022-06-29T16:03:25.204 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxi6D07OBAA2FWiZ+mNK+jFMp/kPZnbQQ==", 2022-06-29T16:03:25.204 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:03:25.204 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:03:25.204 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:03:25.205 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:03:25.205 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:25.205 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:25.205 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:25.205 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.7", 2022-06-29T16:03:25.206 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCMdrxinEOcEhAA562ybpzGHNQfWIq8q7zw9w==", 2022-06-29T16:03:25.206 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:03:25.206 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:03:25.206 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:03:25.206 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:03:25.207 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:25.207 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:25.207 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:25.208 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.0", 2022-06-29T16:03:25.208 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiHXPwBRAAkOOwlYJvIW7dqVFdlXXoqw==", 2022-06-29T16:03:25.208 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:03:25.208 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:03:25.209 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2022-06-29T16:03:25.209 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw pool=cephfs_data" 2022-06-29T16:03:25.209 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:25.209 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:25.209 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:25.210 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.1", 2022-06-29T16:03:25.210 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxibe5JCRAAGBJbi9t09w0836JqrAT7+w==", 2022-06-29T16:03:25.210 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:03:25.210 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:03:25.210 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2022-06-29T16:03:25.211 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw pool=cephfs_data" 2022-06-29T16:03:25.211 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:25.211 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:25.211 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:25.212 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.2", 2022-06-29T16:03:25.212 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiK3dBDRAA7XINPSOPQDfOrUYEpHC+gw==", 2022-06-29T16:03:25.212 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:03:25.212 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:03:25.212 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2022-06-29T16:03:25.213 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw pool=cephfs_data" 2022-06-29T16:03:25.213 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:25.213 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:25.213 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:25.213 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.admin", 2022-06-29T16:03:25.214 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxieKVcGBAAnjH80a0qWIArz2qwjYHWdQ==", 2022-06-29T16:03:25.214 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:03:25.214 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T16:03:25.214 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:03:25.214 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:03:25.215 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:03:25.215 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:25.215 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:25.215 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:25.216 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.x", 2022-06-29T16:03:25.216 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxi2LZJNBAA3MKb/pXYbQCsoRx8EQfGbw==", 2022-06-29T16:03:25.216 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:03:25.216 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T16:03:25.216 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2022-06-29T16:03:25.217 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:03:25.217 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:25.217 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:25.217 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:25.218 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.y", 2022-06-29T16:03:25.218 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxi3LmaMBAAN+0mZrdKXqSXOipVTA1drw==", 2022-06-29T16:03:25.218 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:03:25.218 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T16:03:25.219 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2022-06-29T16:03:25.219 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:03:25.219 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:25.219 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:25.219 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:25.220 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:03:25.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.167+0000 7f34396c6700 1 -- 172.21.15.60:0/1369634649 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3414053b00 msgr2=0x7f3414055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:25.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.167+0000 7f34396c6700 1 --2- 172.21.15.60:0/1369634649 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3414053b00 0x7f3414055fc0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f342401c000 tx=0x7f3424015000).stop 2022-06-29T16:03:25.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.167+0000 7f34396c6700 1 -- 172.21.15.60:0/1369634649 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f343405d070 msgr2=0x7f34340642c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:25.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.167+0000 7f34396c6700 1 --2- 172.21.15.60:0/1369634649 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f343405d070 0x7f34340642c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7f3434061990 tx=0x7f3428007c50).stop 2022-06-29T16:03:25.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.167+0000 7f34396c6700 1 -- 172.21.15.60:0/1369634649 shutdown_connections 2022-06-29T16:03:25.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.167+0000 7f34396c6700 1 --2- 172.21.15.60:0/1369634649 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3414053b00 0x7f3414055fc0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:25.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.167+0000 7f34396c6700 1 --2- 172.21.15.60:0/1369634649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f34340b2130 0x7f3434065d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:25.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.167+0000 7f34396c6700 1 --2- 172.21.15.60:0/1369634649 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f34340aed00 0x7f3434063400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:25.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.167+0000 7f34396c6700 1 --2- 172.21.15.60:0/1369634649 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f343405d070 0x7f34340642c0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:25.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.167+0000 7f34396c6700 1 -- 172.21.15.60:0/1369634649 >> 172.21.15.60:0/1369634649 conn(0x7f34340b5890 msgr2=0x7f34340aca20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:25.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.167+0000 7f34396c6700 1 -- 172.21.15.60:0/1369634649 shutdown_connections 2022-06-29T16:03:25.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.167+0000 7f34396c6700 1 -- 172.21.15.60:0/1369634649 wait complete. 2022-06-29T16:03:25.225 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:03:25.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.327+0000 7f4b97fff700 1 Processor -- start 2022-06-29T16:03:25.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.327+0000 7f4b97fff700 1 -- start start 2022-06-29T16:03:25.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.327+0000 7f4b97fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4b9805d070 0x7f4b980b1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:25.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.327+0000 7f4b97fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4b980aee30 0x7f4b98141b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:25.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.327+0000 7f4b97fff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b9805d450 con 0x7f4b9813a460 2022-06-29T16:03:25.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.327+0000 7f4b97fff700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b98143a90 con 0x7f4b9805d070 2022-06-29T16:03:25.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.327+0000 7f4b97fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b98062200 con 0x7f4b980aee30 2022-06-29T16:03:25.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.327+0000 7f4b96ffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4b9805d070 0x7f4b980b1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:25.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.327+0000 7f4b977fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4b9813a460 0x7f4b98140bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56554/0 (socket says 172.21.15.60:56554) 2022-06-29T16:03:25.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.331+0000 7f4b977fe700 1 -- 172.21.15.60:0/831267202 learned_addr learned my addr 172.21.15.60:0/831267202 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:03:25.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.331+0000 7f4b8ffff700 1 --2- 172.21.15.60:0/831267202 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4b980aee30 0x7f4b98141b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:25.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.331+0000 7f4b96ffd700 1 -- 172.21.15.60:0/831267202 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4b980aee30 msgr2=0x7f4b98141b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:25.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.331+0000 7f4b96ffd700 1 --2- 172.21.15.60:0/831267202 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4b980aee30 0x7f4b98141b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:25.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.331+0000 7f4b967fc700 1 -- 172.21.15.60:0/831267202 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2102935180 0 0) 0x7f4b9805d450 con 0x7f4b9813a460 2022-06-29T16:03:25.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.331+0000 7f4b96ffd700 1 -- 172.21.15.60:0/831267202 >> v1:172.21.15.60:6789/0 conn(0x7f4b9813a460 legacy=0x7f4b98140bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:25.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.331+0000 7f4b96ffd700 1 -- 172.21.15.60:0/831267202 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b98064dc0 con 0x7f4b9805d070 2022-06-29T16:03:25.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.331+0000 7f4b96ffd700 1 --2- 172.21.15.60:0/831267202 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b9805d070 0x7f4b980b1f30 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7f4b88002f70 tx=0x7f4b88006df0).ready entity=mon.0 client_cookie=2249685a799f2021 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:25.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.331+0000 7f4b967fc700 1 -- 172.21.15.60:0/831267202 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b8800d570 con 0x7f4b9805d070 2022-06-29T16:03:25.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.331+0000 7f4b967fc700 1 -- 172.21.15.60:0/831267202 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b88000e90 con 0x7f4b9805d070 2022-06-29T16:03:25.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.331+0000 7f4b967fc700 1 -- 172.21.15.60:0/831267202 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b8800d8c0 con 0x7f4b9805d070 2022-06-29T16:03:25.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.331+0000 7f4b97fff700 1 -- 172.21.15.60:0/831267202 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b9805d070 msgr2=0x7f4b980b1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:25.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.331+0000 7f4b97fff700 1 --2- 172.21.15.60:0/831267202 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b9805d070 0x7f4b980b1f30 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7f4b88002f70 tx=0x7f4b88006df0).stop 2022-06-29T16:03:25.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.331+0000 7f4b97fff700 1 -- 172.21.15.60:0/831267202 shutdown_connections 2022-06-29T16:03:25.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.331+0000 7f4b97fff700 1 --2- 172.21.15.60:0/831267202 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4b980aee30 0x7f4b98141b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:25.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.331+0000 7f4b97fff700 1 --2- 172.21.15.60:0/831267202 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b9805d070 0x7f4b980b1f30 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:25.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.331+0000 7f4b97fff700 1 -- 172.21.15.60:0/831267202 >> 172.21.15.60:0/831267202 conn(0x7f4b980b5890 msgr2=0x7f4b980b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:25.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.331+0000 7f4b97fff700 1 -- 172.21.15.60:0/831267202 shutdown_connections 2022-06-29T16:03:25.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.331+0000 7f4b97fff700 1 -- 172.21.15.60:0/831267202 wait complete. 2022-06-29T16:03:25.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.331+0000 7f4b97fff700 1 Processor -- start 2022-06-29T16:03:25.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.331+0000 7f4b97fff700 1 -- start start 2022-06-29T16:03:25.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.335+0000 7f4b97fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b980aee30 0x7f4b9813dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:25.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.335+0000 7f4b97fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4b9813a460 0x7f4b98066b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:25.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.335+0000 7f4b97fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b9813b650 0x7f4b9813d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:25.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.335+0000 7f4b97fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9814eb60 con 0x7f4b980aee30 2022-06-29T16:03:25.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.335+0000 7f4b97fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b981393e0 con 0x7f4b9813b650 2022-06-29T16:03:25.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.335+0000 7f4b97fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9813cd30 con 0x7f4b9813a460 2022-06-29T16:03:25.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.335+0000 7f4b96ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b980aee30 0x7f4b9813dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:25.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.335+0000 7f4b96ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b980aee30 0x7f4b9813dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48470/0 (socket says 172.21.15.60:48470) 2022-06-29T16:03:25.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.335+0000 7f4b96ffd700 1 -- 172.21.15.60:0/849738844 learned_addr learned my addr 172.21.15.60:0/849738844 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:25.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.335+0000 7f4b8ffff700 1 --2- 172.21.15.60:0/849738844 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4b9813a460 0x7f4b98066b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:25.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.335+0000 7f4b977fe700 1 --2- 172.21.15.60:0/849738844 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b9813b650 0x7f4b9813d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:25.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.335+0000 7f4b96ffd700 1 -- 172.21.15.60:0/849738844 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4b9813a460 msgr2=0x7f4b98066b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:25.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.335+0000 7f4b96ffd700 1 --2- 172.21.15.60:0/849738844 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4b9813a460 0x7f4b98066b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:25.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.335+0000 7f4b96ffd700 1 -- 172.21.15.60:0/849738844 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b9813b650 msgr2=0x7f4b9813d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:25.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.335+0000 7f4b96ffd700 1 --2- 172.21.15.60:0/849738844 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b9813b650 0x7f4b9813d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:25.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.335+0000 7f4b96ffd700 1 -- 172.21.15.60:0/849738844 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b8800b180 con 0x7f4b980aee30 2022-06-29T16:03:25.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.335+0000 7f4b96ffd700 1 --2- 172.21.15.60:0/849738844 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b980aee30 0x7f4b9813dd20 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7f4b98139520 tx=0x7f4b88014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:25.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.335+0000 7f4b94ff9700 1 -- 172.21.15.60:0/849738844 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b88016070 con 0x7f4b980aee30 2022-06-29T16:03:25.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.335+0000 7f4b94ff9700 1 -- 172.21.15.60:0/849738844 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b88013880 con 0x7f4b980aee30 2022-06-29T16:03:25.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.335+0000 7f4b94ff9700 1 -- 172.21.15.60:0/849738844 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b88013be0 con 0x7f4b980aee30 2022-06-29T16:03:25.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.335+0000 7f4b97fff700 1 -- 172.21.15.60:0/849738844 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b98065d30 con 0x7f4b980aee30 2022-06-29T16:03:25.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.335+0000 7f4b97fff700 1 -- 172.21.15.60:0/849738844 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b980b1260 con 0x7f4b980aee30 2022-06-29T16:03:25.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.339+0000 7f4b97fff700 1 -- 172.21.15.60:0/849738844 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b980573d0 con 0x7f4b980aee30 2022-06-29T16:03:25.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.339+0000 7f4b94ff9700 1 -- 172.21.15.60:0/849738844 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4b8800f030 con 0x7f4b980aee30 2022-06-29T16:03:25.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.339+0000 7f4b94ff9700 1 --2- 172.21.15.60:0/849738844 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4b7c0539e0 0x7f4b7c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:25.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.339+0000 7f4b94ff9700 1 -- 172.21.15.60:0/849738844 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f4b8806c890 con 0x7f4b980aee30 2022-06-29T16:03:25.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.339+0000 7f4b8ffff700 1 --2- 172.21.15.60:0/849738844 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4b7c0539e0 0x7f4b7c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:25.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.339+0000 7f4b8ffff700 1 --2- 172.21.15.60:0/849738844 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4b7c0539e0 0x7f4b7c055ea0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f4b8000f000 tx=0x7f4b8000f040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:25.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.343+0000 7f4b94ff9700 1 -- 172.21.15.60:0/849738844 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4b880439d0 con 0x7f4b980aee30 2022-06-29T16:03:25.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.471+0000 7f4b97fff700 1 -- 172.21.15.60:0/849738844 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b98062f00 con 0x7f4b980aee30 2022-06-29T16:03:25.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.475+0000 7f4b94ff9700 1 -- 172.21.15.60:0/849738844 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+13182 (secure 0 0 0) 0x7f4b88023070 con 0x7f4b980aee30 2022-06-29T16:03:25.481 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:03:25.481 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":57,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:03:24.845966+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:03:25.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.475+0000 7f4b97fff700 1 -- 172.21.15.60:0/849738844 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4b7c0539e0 msgr2=0x7f4b7c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:25.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.475+0000 7f4b97fff700 1 --2- 172.21.15.60:0/849738844 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4b7c0539e0 0x7f4b7c055ea0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f4b8000f000 tx=0x7f4b8000f040).stop 2022-06-29T16:03:25.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.475+0000 7f4b97fff700 1 -- 172.21.15.60:0/849738844 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b980aee30 msgr2=0x7f4b9813dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:25.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.475+0000 7f4b97fff700 1 --2- 172.21.15.60:0/849738844 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b980aee30 0x7f4b9813dd20 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7f4b98139520 tx=0x7f4b88014000).stop 2022-06-29T16:03:25.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.475+0000 7f4b97fff700 1 -- 172.21.15.60:0/849738844 shutdown_connections 2022-06-29T16:03:25.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.475+0000 7f4b97fff700 1 --2- 172.21.15.60:0/849738844 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4b7c0539e0 0x7f4b7c055ea0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:25.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.475+0000 7f4b97fff700 1 --2- 172.21.15.60:0/849738844 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b9813b650 0x7f4b9813d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:25.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.475+0000 7f4b97fff700 1 --2- 172.21.15.60:0/849738844 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4b9813a460 0x7f4b98066b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:25.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.475+0000 7f4b97fff700 1 --2- 172.21.15.60:0/849738844 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b980aee30 0x7f4b9813dd20 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:25.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.475+0000 7f4b97fff700 1 -- 172.21.15.60:0/849738844 >> 172.21.15.60:0/849738844 conn(0x7f4b980b5890 msgr2=0x7f4b980ade00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:25.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.475+0000 7f4b97fff700 1 -- 172.21.15.60:0/849738844 shutdown_connections 2022-06-29T16:03:25.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.475+0000 7f4b97fff700 1 -- 172.21.15.60:0/849738844 wait complete. 2022-06-29T16:03:25.494 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-06-29T16:03:25.495 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:03:25.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.595+0000 7faf5ac2c700 1 Processor -- start 2022-06-29T16:03:25.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.595+0000 7faf5ac2c700 1 -- start start 2022-06-29T16:03:25.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.595+0000 7faf5ac2c700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7faf540ae8f0 0x7faf54067740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:25.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.595+0000 7faf5ac2c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faf540b0b20 0x7faf540af900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:25.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.595+0000 7faf5ac2c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7faf540afdf0 0x7faf540b0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:25.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.595+0000 7faf5ac2c700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf540b0750 con 0x7faf540afdf0 2022-06-29T16:03:25.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.595+0000 7faf5ac2c700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf540af020 con 0x7faf540ae8f0 2022-06-29T16:03:25.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.595+0000 7faf5ac2c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf54144b10 con 0x7faf540b0b20 2022-06-29T16:03:25.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.595+0000 7faf591c9700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7faf540afdf0 0x7faf540b0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:25.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.595+0000 7faf591c9700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7faf540afdf0 0x7faf540b0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48478/0 (socket says 172.21.15.60:48478) 2022-06-29T16:03:25.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.595+0000 7faf591c9700 1 -- 172.21.15.60:0/2593847550 learned_addr learned my addr 172.21.15.60:0/2593847550 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:25.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.595+0000 7faf53fff700 1 --2- 172.21.15.60:0/2593847550 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faf540b0b20 0x7faf540af900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:25.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.595+0000 7faf589c8700 1 --2- 172.21.15.60:0/2593847550 >> v2:172.21.15.92:3300/0 conn(0x7faf540ae8f0 0x7faf54067740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:25.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.595+0000 7faf591c9700 1 -- 172.21.15.60:0/2593847550 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faf540b0b20 msgr2=0x7faf540af900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:25.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.595+0000 7faf591c9700 1 --2- 172.21.15.60:0/2593847550 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faf540b0b20 0x7faf540af900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:25.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.595+0000 7faf591c9700 1 -- 172.21.15.60:0/2593847550 >> v2:172.21.15.92:3300/0 conn(0x7faf540ae8f0 msgr2=0x7faf54067740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:25.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.595+0000 7faf591c9700 1 --2- 172.21.15.60:0/2593847550 >> v2:172.21.15.92:3300/0 conn(0x7faf540ae8f0 0x7faf54067740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:25.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.595+0000 7faf591c9700 1 -- 172.21.15.60:0/2593847550 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf540aecd0 con 0x7faf540afdf0 2022-06-29T16:03:25.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.595+0000 7faf53fff700 1 --2- 172.21.15.60:0/2593847550 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faf540b0b20 0x7faf540af900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:03:25.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.595+0000 7faf591c9700 1 --2- 172.21.15.60:0/2593847550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf540afdf0 0x7faf540b0260 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0x7faf44002f70 tx=0x7faf44006eb0).ready entity=mon.0 client_cookie=d38daafe2576d0e9 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:25.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.595+0000 7faf537fe700 1 -- 172.21.15.60:0/2593847550 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf4400d570 con 0x7faf540afdf0 2022-06-29T16:03:25.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.595+0000 7faf537fe700 1 -- 172.21.15.60:0/2593847550 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf44000e90 con 0x7faf540afdf0 2022-06-29T16:03:25.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.595+0000 7faf537fe700 1 -- 172.21.15.60:0/2593847550 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf4400d900 con 0x7faf540afdf0 2022-06-29T16:03:25.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.599+0000 7faf5ac2c700 1 -- 172.21.15.60:0/2593847550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf540afdf0 msgr2=0x7faf540b0260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:25.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.599+0000 7faf5ac2c700 1 --2- 172.21.15.60:0/2593847550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf540afdf0 0x7faf540b0260 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0x7faf44002f70 tx=0x7faf44006eb0).stop 2022-06-29T16:03:25.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.599+0000 7faf5ac2c700 1 -- 172.21.15.60:0/2593847550 shutdown_connections 2022-06-29T16:03:25.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.599+0000 7faf5ac2c700 1 --2- 172.21.15.60:0/2593847550 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf540afdf0 0x7faf540b0260 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:25.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.599+0000 7faf5ac2c700 1 --2- 172.21.15.60:0/2593847550 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faf540b0b20 0x7faf540af900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:25.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.599+0000 7faf5ac2c700 1 --2- 172.21.15.60:0/2593847550 >> v2:172.21.15.92:3300/0 conn(0x7faf540ae8f0 0x7faf54067740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:25.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.599+0000 7faf5ac2c700 1 -- 172.21.15.60:0/2593847550 >> 172.21.15.60:0/2593847550 conn(0x7faf540b4f60 msgr2=0x7faf540b5370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:25.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.599+0000 7faf5ac2c700 1 -- 172.21.15.60:0/2593847550 shutdown_connections 2022-06-29T16:03:25.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.599+0000 7faf5ac2c700 1 -- 172.21.15.60:0/2593847550 wait complete. 2022-06-29T16:03:25.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.599+0000 7faf5ac2c700 1 Processor -- start 2022-06-29T16:03:25.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.599+0000 7faf5ac2c700 1 -- start start 2022-06-29T16:03:25.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.603+0000 7faf5ac2c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faf540ae8f0 0x7faf5409e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:25.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.603+0000 7faf5ac2c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faf540afdf0 0x7faf540653e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:25.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.603+0000 7faf5ac2c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf540b0b20 0x7faf5413edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:25.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.603+0000 7faf5ac2c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf54153cc0 con 0x7faf540b0b20 2022-06-29T16:03:25.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.603+0000 7faf5ac2c700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf54066420 con 0x7faf540ae8f0 2022-06-29T16:03:25.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.603+0000 7faf5ac2c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf540666c0 con 0x7faf540afdf0 2022-06-29T16:03:25.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.603+0000 7faf591c9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf540b0b20 0x7faf5413edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:25.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.603+0000 7faf591c9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf540b0b20 0x7faf5413edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48486/0 (socket says 172.21.15.60:48486) 2022-06-29T16:03:25.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.603+0000 7faf591c9700 1 -- 172.21.15.60:0/2499827840 learned_addr learned my addr 172.21.15.60:0/2499827840 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:25.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.603+0000 7faf53fff700 1 --2- 172.21.15.60:0/2499827840 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faf540afdf0 0x7faf540653e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:25.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.603+0000 7faf589c8700 1 --2- 172.21.15.60:0/2499827840 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faf540ae8f0 0x7faf5409e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:25.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.603+0000 7faf591c9700 1 -- 172.21.15.60:0/2499827840 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faf540afdf0 msgr2=0x7faf540653e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:25.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.603+0000 7faf591c9700 1 --2- 172.21.15.60:0/2499827840 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faf540afdf0 0x7faf540653e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:25.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.603+0000 7faf591c9700 1 -- 172.21.15.60:0/2499827840 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faf540ae8f0 msgr2=0x7faf5409e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:25.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.603+0000 7faf591c9700 1 --2- 172.21.15.60:0/2499827840 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faf540ae8f0 0x7faf5409e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:25.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.603+0000 7faf591c9700 1 -- 172.21.15.60:0/2499827840 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf4400b1e0 con 0x7faf540b0b20 2022-06-29T16:03:25.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.603+0000 7faf591c9700 1 --2- 172.21.15.60:0/2499827840 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf540b0b20 0x7faf5413edd0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0x7faf44002f70 tx=0x7faf44015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:25.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.603+0000 7faf51ffb700 1 -- 172.21.15.60:0/2499827840 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf440117e0 con 0x7faf540b0b20 2022-06-29T16:03:25.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.603+0000 7faf51ffb700 1 -- 172.21.15.60:0/2499827840 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf44011940 con 0x7faf540b0b20 2022-06-29T16:03:25.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.603+0000 7faf51ffb700 1 -- 172.21.15.60:0/2499827840 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf44011ca0 con 0x7faf540b0b20 2022-06-29T16:03:25.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.603+0000 7faf5ac2c700 1 -- 172.21.15.60:0/2499827840 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf54066960 con 0x7faf540b0b20 2022-06-29T16:03:25.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.603+0000 7faf5ac2c700 1 -- 172.21.15.60:0/2499827840 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf5409e650 con 0x7faf540b0b20 2022-06-29T16:03:25.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.603+0000 7faf5ac2c700 1 -- 172.21.15.60:0/2499827840 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf540573d0 con 0x7faf540b0b20 2022-06-29T16:03:25.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.603+0000 7faf51ffb700 1 -- 172.21.15.60:0/2499827840 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7faf44025030 con 0x7faf540b0b20 2022-06-29T16:03:25.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.607+0000 7faf51ffb700 1 --2- 172.21.15.60:0/2499827840 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7faf380539e0 0x7faf38055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:25.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.607+0000 7faf51ffb700 1 -- 172.21.15.60:0/2499827840 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5253+0+0 (secure 0 0 0) 0x7faf4406cf60 con 0x7faf540b0b20 2022-06-29T16:03:25.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.607+0000 7faf589c8700 1 --2- 172.21.15.60:0/2499827840 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7faf380539e0 0x7faf38055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:25.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.607+0000 7faf589c8700 1 --2- 172.21.15.60:0/2499827840 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7faf380539e0 0x7faf38055ea0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7faf4c00d000 tx=0x7faf4c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:25.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.615+0000 7faf51ffb700 1 -- 172.21.15.60:0/2499827840 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7faf44044020 con 0x7faf540b0b20 2022-06-29T16:03:25.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.743+0000 7faf5ac2c700 1 -- 172.21.15.60:0/2499827840 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7faf54068560 con 0x7faf540b0b20 2022-06-29T16:03:25.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.743+0000 7faf51ffb700 1 -- 172.21.15.60:0/2499827840 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v57) v1 ==== 110+0+10 (secure 0 0 0) 0x7faf44023020 con 0x7faf540b0b20 2022-06-29T16:03:25.751 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:03:25.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.747+0000 7faf5ac2c700 1 -- 172.21.15.60:0/2499827840 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7faf380539e0 msgr2=0x7faf38055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:25.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.747+0000 7faf5ac2c700 1 --2- 172.21.15.60:0/2499827840 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7faf380539e0 0x7faf38055ea0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7faf4c00d000 tx=0x7faf4c008040).stop 2022-06-29T16:03:25.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.747+0000 7faf5ac2c700 1 -- 172.21.15.60:0/2499827840 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf540b0b20 msgr2=0x7faf5413edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:25.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.747+0000 7faf5ac2c700 1 --2- 172.21.15.60:0/2499827840 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf540b0b20 0x7faf5413edd0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0x7faf44002f70 tx=0x7faf44015000).stop 2022-06-29T16:03:25.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.747+0000 7faf5ac2c700 1 -- 172.21.15.60:0/2499827840 shutdown_connections 2022-06-29T16:03:25.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.747+0000 7faf5ac2c700 1 --2- 172.21.15.60:0/2499827840 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7faf380539e0 0x7faf38055ea0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:25.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.747+0000 7faf5ac2c700 1 --2- 172.21.15.60:0/2499827840 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf540b0b20 0x7faf5413edd0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:25.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.747+0000 7faf5ac2c700 1 --2- 172.21.15.60:0/2499827840 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faf540afdf0 0x7faf540653e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:25.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.747+0000 7faf5ac2c700 1 --2- 172.21.15.60:0/2499827840 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faf540ae8f0 0x7faf5409e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:25.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.747+0000 7faf5ac2c700 1 -- 172.21.15.60:0/2499827840 >> 172.21.15.60:0/2499827840 conn(0x7faf540b4f60 msgr2=0x7faf54141000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:25.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.747+0000 7faf5ac2c700 1 -- 172.21.15.60:0/2499827840 shutdown_connections 2022-06-29T16:03:25.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.747+0000 7faf5ac2c700 1 -- 172.21.15.60:0/2499827840 wait complete. 2022-06-29T16:03:25.770 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:03:25.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.875+0000 7f1cf946c700 1 Processor -- start 2022-06-29T16:03:25.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.875+0000 7f1cf946c700 1 -- start start 2022-06-29T16:03:25.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.879+0000 7f1cf946c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1cf4066d20 0x7f1cf40b2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:25.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.879+0000 7f1cf946c700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf4067100 con 0x7f1cf4144b00 2022-06-29T16:03:25.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.879+0000 7f1cf946c700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf40603a0 con 0x7f1cf40b3430 2022-06-29T16:03:25.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.879+0000 7f1cf946c700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cf4060640 con 0x7f1cf4066d20 2022-06-29T16:03:25.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.879+0000 7f1cf8c6b700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f1cf4144b00 0x7f1cf413f7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56580/0 (socket says 172.21.15.60:56580) 2022-06-29T16:03:25.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.879+0000 7f1cf8c6b700 1 -- 172.21.15.60:0/798730205 learned_addr learned my addr 172.21.15.60:0/798730205 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:03:25.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.879+0000 7f1cf3fff700 1 --2- 172.21.15.60:0/798730205 >> v2:172.21.15.60:3300/0 conn(0x7f1cf4066d20 0x7f1cf40b2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:25.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.879+0000 7f1cf37fe700 1 -- 172.21.15.60:0/798730205 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2948193359 0 0) 0x7f1cf4067100 con 0x7f1cf4144b00 2022-06-29T16:03:25.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.879+0000 7f1cf37fe700 1 -- 172.21.15.60:0/798730205 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cd4003260 con 0x7f1cf4144b00 2022-06-29T16:03:25.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.879+0000 7f1cf3fff700 1 -- 172.21.15.60:0/798730205 >> v1:172.21.15.92:6789/0 conn(0x7f1cf40b3430 legacy=0x7f1cf41443f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:25.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.879+0000 7f1cf37fe700 1 -- 172.21.15.60:0/798730205 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2930551577 0 0) 0x7f1cd4003260 con 0x7f1cf4144b00 2022-06-29T16:03:25.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.879+0000 7f1cf3fff700 1 -- 172.21.15.60:0/798730205 >> v1:172.21.15.60:6789/0 conn(0x7f1cf4144b00 legacy=0x7f1cf413f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:25.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.879+0000 7f1cf3fff700 1 -- 172.21.15.60:0/798730205 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf4143f50 con 0x7f1cf4066d20 2022-06-29T16:03:25.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.879+0000 7f1cf3fff700 1 --2- 172.21.15.60:0/798730205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cf4066d20 0x7f1cf40b2ea0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7f1cec001b80 tx=0x7f1ce4006df0).ready entity=mon.0 client_cookie=93e988484240e1b9 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:25.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.879+0000 7f1cf37fe700 1 -- 172.21.15.60:0/798730205 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1cec0019e0 con 0x7f1cf4066d20 2022-06-29T16:03:25.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.879+0000 7f1cf37fe700 1 -- 172.21.15.60:0/798730205 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce4000e90 con 0x7f1cf4066d20 2022-06-29T16:03:25.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.879+0000 7f1cf37fe700 1 -- 172.21.15.60:0/798730205 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ce400d4f0 con 0x7f1cf4066d20 2022-06-29T16:03:25.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.879+0000 7f1cf946c700 1 -- 172.21.15.60:0/798730205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cf4066d20 msgr2=0x7f1cf40b2ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:25.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.879+0000 7f1cf946c700 1 --2- 172.21.15.60:0/798730205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cf4066d20 0x7f1cf40b2ea0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7f1cec001b80 tx=0x7f1ce4006df0).stop 2022-06-29T16:03:25.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.879+0000 7f1cf946c700 1 -- 172.21.15.60:0/798730205 shutdown_connections 2022-06-29T16:03:25.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.879+0000 7f1cf946c700 1 --2- 172.21.15.60:0/798730205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cf4066d20 0x7f1cf40b2ea0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:25.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.879+0000 7f1cf946c700 1 -- 172.21.15.60:0/798730205 >> 172.21.15.60:0/798730205 conn(0x7f1cf40637c0 msgr2=0x7f1cf4063ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:25.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.879+0000 7f1cf946c700 1 -- 172.21.15.60:0/798730205 shutdown_connections 2022-06-29T16:03:25.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.879+0000 7f1cf946c700 1 -- 172.21.15.60:0/798730205 wait complete. 2022-06-29T16:03:25.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.883+0000 7f1cf946c700 1 Processor -- start 2022-06-29T16:03:25.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.883+0000 7f1cf946c700 1 -- start start 2022-06-29T16:03:25.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.883+0000 7f1cf946c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cf4066d20 0x7f1cf41508e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:25.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.883+0000 7f1cf946c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1cf40b3430 0x7f1cf40adce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:25.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.883+0000 7f1cf946c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cf4144b00 0x7f1cf40aebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:25.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.883+0000 7f1cf946c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf40ad810 con 0x7f1cf4144b00 2022-06-29T16:03:25.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.883+0000 7f1cf946c700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf40ae1d0 con 0x7f1cf40b3430 2022-06-29T16:03:25.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.883+0000 7f1cf946c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf40ada40 con 0x7f1cf4066d20 2022-06-29T16:03:25.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.883+0000 7f1cf8c6b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cf4144b00 0x7f1cf40aebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:25.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.883+0000 7f1cf8c6b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cf4144b00 0x7f1cf40aebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48500/0 (socket says 172.21.15.60:48500) 2022-06-29T16:03:25.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.883+0000 7f1cf8c6b700 1 -- 172.21.15.60:0/2973662052 learned_addr learned my addr 172.21.15.60:0/2973662052 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:25.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.883+0000 7f1cf3fff700 1 --2- 172.21.15.60:0/2973662052 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cf4066d20 0x7f1cf41508e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:25.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.883+0000 7f1ceb7fe700 1 --2- 172.21.15.60:0/2973662052 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1cf40b3430 0x7f1cf40adce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:25.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.883+0000 7f1cf8c6b700 1 -- 172.21.15.60:0/2973662052 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cf4066d20 msgr2=0x7f1cf41508e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:25.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.883+0000 7f1cf8c6b700 1 --2- 172.21.15.60:0/2973662052 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cf4066d20 0x7f1cf41508e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:25.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.883+0000 7f1cf8c6b700 1 -- 172.21.15.60:0/2973662052 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1cf40b3430 msgr2=0x7f1cf40adce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:25.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.883+0000 7f1cf8c6b700 1 --2- 172.21.15.60:0/2973662052 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1cf40b3430 0x7f1cf40adce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:25.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.883+0000 7f1cf8c6b700 1 -- 172.21.15.60:0/2973662052 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce400b180 con 0x7f1cf4144b00 2022-06-29T16:03:25.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.883+0000 7f1cf3fff700 1 --2- 172.21.15.60:0/2973662052 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cf4066d20 0x7f1cf41508e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:03:25.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.883+0000 7f1cf8c6b700 1 --2- 172.21.15.60:0/2973662052 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cf4144b00 0x7f1cf40aebc0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0x7f1cec001250 tx=0x7f1cec001ac0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:25.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.883+0000 7f1cf1ffb700 1 -- 172.21.15.60:0/2973662052 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1cec004ab0 con 0x7f1cf4144b00 2022-06-29T16:03:25.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.883+0000 7f1cf1ffb700 1 -- 172.21.15.60:0/2973662052 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cec004c10 con 0x7f1cf4144b00 2022-06-29T16:03:25.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.887+0000 7f1cf946c700 1 -- 172.21.15.60:0/2973662052 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cf40affb0 con 0x7f1cf4144b00 2022-06-29T16:03:25.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.887+0000 7f1cf1ffb700 1 -- 172.21.15.60:0/2973662052 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1cec007170 con 0x7f1cf4144b00 2022-06-29T16:03:25.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.887+0000 7f1cf946c700 1 -- 172.21.15.60:0/2973662052 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cf40af250 con 0x7f1cf4144b00 2022-06-29T16:03:25.903 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.887+0000 7f1cf946c700 1 -- 172.21.15.60:0/2973662052 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cf40573d0 con 0x7f1cf4144b00 2022-06-29T16:03:25.903 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.887+0000 7f1cf1ffb700 1 -- 172.21.15.60:0/2973662052 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1cec0072d0 con 0x7f1cf4144b00 2022-06-29T16:03:25.904 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.887+0000 7f1cf1ffb700 1 --2- 172.21.15.60:0/2973662052 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1cd8053aa0 0x7f1cd8055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:25.904 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.887+0000 7f1cf1ffb700 1 -- 172.21.15.60:0/2973662052 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f1cec06b960 con 0x7f1cf4144b00 2022-06-29T16:03:25.904 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.887+0000 7f1cf3fff700 1 --2- 172.21.15.60:0/2973662052 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1cd8053aa0 0x7f1cd8055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:25.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.891+0000 7f1cf3fff700 1 --2- 172.21.15.60:0/2973662052 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1cd8053aa0 0x7f1cd8055f60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f1ce4018000 tx=0x7f1ce4014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:25.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:25.895+0000 7f1cf1ffb700 1 -- 172.21.15.60:0/2973662052 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1cec042b10 con 0x7f1cf4144b00 2022-06-29T16:03:26.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.023+0000 7f1cf946c700 1 -- 172.21.15.60:0/2973662052 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f1cf4067d20 con 0x7f1cf4144b00 2022-06-29T16:03:26.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.023+0000 7f1cf1ffb700 1 -- 172.21.15.60:0/2973662052 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v57) v1 ==== 92+0+10 (secure 0 0 0) 0x7f1cec046130 con 0x7f1cf4144b00 2022-06-29T16:03:26.030 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:03:26.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.023+0000 7f1cf946c700 1 -- 172.21.15.60:0/2973662052 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1cd8053aa0 msgr2=0x7f1cd8055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:26.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.023+0000 7f1cf946c700 1 --2- 172.21.15.60:0/2973662052 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1cd8053aa0 0x7f1cd8055f60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f1ce4018000 tx=0x7f1ce4014000).stop 2022-06-29T16:03:26.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.027+0000 7f1cf946c700 1 -- 172.21.15.60:0/2973662052 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cf4144b00 msgr2=0x7f1cf40aebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:26.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.027+0000 7f1cf946c700 1 --2- 172.21.15.60:0/2973662052 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cf4144b00 0x7f1cf40aebc0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0x7f1cec001250 tx=0x7f1cec001ac0).stop 2022-06-29T16:03:26.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.027+0000 7f1cf946c700 1 -- 172.21.15.60:0/2973662052 shutdown_connections 2022-06-29T16:03:26.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.027+0000 7f1cf946c700 1 --2- 172.21.15.60:0/2973662052 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1cd8053aa0 0x7f1cd8055f60 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:26.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.027+0000 7f1cf946c700 1 --2- 172.21.15.60:0/2973662052 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1cf4144b00 0x7f1cf40aebc0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:26.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.027+0000 7f1cf946c700 1 --2- 172.21.15.60:0/2973662052 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1cf40b3430 0x7f1cf40adce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:26.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.027+0000 7f1cf946c700 1 --2- 172.21.15.60:0/2973662052 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1cf4066d20 0x7f1cf41508e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:26.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.027+0000 7f1cf946c700 1 -- 172.21.15.60:0/2973662052 >> 172.21.15.60:0/2973662052 conn(0x7f1cf40637c0 msgr2=0x7f1cf41434c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:26.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.027+0000 7f1cf946c700 1 -- 172.21.15.60:0/2973662052 shutdown_connections 2022-06-29T16:03:26.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.027+0000 7f1cf946c700 1 -- 172.21.15.60:0/2973662052 wait complete. 2022-06-29T16:03:26.044 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-06-29T16:03:26.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.139+0000 7f697f59e700 1 Processor -- start 2022-06-29T16:03:26.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.139+0000 7f697f59e700 1 -- start start 2022-06-29T16:03:26.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.139+0000 7f697f59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f69800aed00 0x7f6980146810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:26.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.139+0000 7f697f59e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f698005d450 con 0x7f69800b2130 2022-06-29T16:03:26.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.139+0000 7f697f59e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69800b25d0 con 0x7f698005d070 2022-06-29T16:03:26.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.139+0000 7f697f59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69800adf90 con 0x7f69800aed00 2022-06-29T16:03:26.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.139+0000 7f697dd9b700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f69800b2130 0x7f6980146100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56592/0 (socket says 172.21.15.60:56592) 2022-06-29T16:03:26.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.139+0000 7f697dd9b700 1 -- 172.21.15.60:0/3842242646 learned_addr learned my addr 172.21.15.60:0/3842242646 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:03:26.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.139+0000 7f697ed9d700 1 --2- 172.21.15.60:0/3842242646 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f69800aed00 0x7f6980146810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:26.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.139+0000 7f697d59a700 1 -- 172.21.15.60:0/3842242646 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2737006461 0 0) 0x7f698005d450 con 0x7f69800b2130 2022-06-29T16:03:26.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.139+0000 7f697d59a700 1 -- 172.21.15.60:0/3842242646 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6964003260 con 0x7f69800b2130 2022-06-29T16:03:26.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.139+0000 7f697d59a700 1 -- 172.21.15.60:0/3842242646 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 43262441 0 0) 0x7f69800b25d0 con 0x7f698005d070 2022-06-29T16:03:26.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.139+0000 7f697d59a700 1 -- 172.21.15.60:0/3842242646 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f698005d450 con 0x7f698005d070 2022-06-29T16:03:26.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.139+0000 7f697d59a700 1 -- 172.21.15.60:0/3842242646 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2969688955 0 0) 0x7f6964003260 con 0x7f69800b2130 2022-06-29T16:03:26.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.139+0000 7f697d59a700 1 -- 172.21.15.60:0/3842242646 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f69800aed00 msgr2=0x7f6980146810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:26.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.139+0000 7f697d59a700 1 --2- 172.21.15.60:0/3842242646 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f69800aed00 0x7f6980146810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:26.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.139+0000 7f697d59a700 1 -- 172.21.15.60:0/3842242646 >> v1:172.21.15.92:6789/0 conn(0x7f698005d070 legacy=0x7f698014aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:26.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.139+0000 7f697d59a700 1 -- 172.21.15.60:0/3842242646 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f698014a350 con 0x7f69800b2130 2022-06-29T16:03:26.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.139+0000 7f697d59a700 1 -- 172.21.15.60:0/3842242646 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f6968001600 con 0x7f69800b2130 2022-06-29T16:03:26.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.139+0000 7f697d59a700 1 -- 172.21.15.60:0/3842242646 >> v1:172.21.15.60:6789/0 conn(0x7f69800b2130 legacy=0x7f6980146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:26.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.139+0000 7f697d59a700 1 --2- 172.21.15.60:0/3842242646 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6964003910 0x7f6964003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:26.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.139+0000 7f697d59a700 1 --2- 172.21.15.60:0/3842242646 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f69800aed00 0x7f6964004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:26.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.139+0000 7f697d59a700 1 --2- 172.21.15.60:0/3842242646 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69640049f0 0x7f6964004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:26.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.139+0000 7f697d59a700 1 -- 172.21.15.60:0/3842242646 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69800adf90 con 0x7f69640049f0 2022-06-29T16:03:26.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.139+0000 7f697d59a700 1 -- 172.21.15.60:0/3842242646 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696400b800 con 0x7f6964003910 2022-06-29T16:03:26.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.139+0000 7f697d59a700 1 -- 172.21.15.60:0/3842242646 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f696400b9c0 con 0x7f69800aed00 2022-06-29T16:03:26.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.139+0000 7f697ed9d700 1 --2- 172.21.15.60:0/3842242646 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69640049f0 0x7f6964004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:26.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.139+0000 7f697dd9b700 1 --2- 172.21.15.60:0/3842242646 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f69800aed00 0x7f6964004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:26.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.139+0000 7f697e59c700 1 --2- 172.21.15.60:0/3842242646 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6964003910 0x7f6964003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:26.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.143+0000 7f697ed9d700 1 -- 172.21.15.60:0/3842242646 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f69800aed00 msgr2=0x7f6964004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:26.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.143+0000 7f697ed9d700 1 --2- 172.21.15.60:0/3842242646 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f69800aed00 0x7f6964004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:26.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.143+0000 7f697ed9d700 1 -- 172.21.15.60:0/3842242646 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6964003910 msgr2=0x7f6964003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:26.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.143+0000 7f697ed9d700 1 --2- 172.21.15.60:0/3842242646 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6964003910 0x7f6964003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:26.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.143+0000 7f697ed9d700 1 -- 172.21.15.60:0/3842242646 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6964003260 con 0x7f69640049f0 2022-06-29T16:03:26.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.143+0000 7f697dd9b700 1 --2- 172.21.15.60:0/3842242646 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f69800aed00 0x7f6964004500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:03:26.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.143+0000 7f697ed9d700 1 --2- 172.21.15.60:0/3842242646 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69640049f0 0x7f6964004e70 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7f697800c000 tx=0x7f69780091d0).ready entity=mon.0 client_cookie=387a661975a14d1a server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:26.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.143+0000 7f697d59a700 1 -- 172.21.15.60:0/3842242646 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f697800e7e0 con 0x7f69640049f0 2022-06-29T16:03:26.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.143+0000 7f697d59a700 1 -- 172.21.15.60:0/3842242646 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f697800e940 con 0x7f69640049f0 2022-06-29T16:03:26.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.143+0000 7f697d59a700 1 -- 172.21.15.60:0/3842242646 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f697800eca0 con 0x7f69640049f0 2022-06-29T16:03:26.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.143+0000 7f697f59e700 1 -- 172.21.15.60:0/3842242646 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69640049f0 msgr2=0x7f6964004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:26.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.143+0000 7f697f59e700 1 --2- 172.21.15.60:0/3842242646 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69640049f0 0x7f6964004e70 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7f697800c000 tx=0x7f69780091d0).stop 2022-06-29T16:03:26.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.143+0000 7f697ed9d700 1 -- 172.21.15.60:0/3842242646 reap_dead start 2022-06-29T16:03:26.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.143+0000 7f697f59e700 1 -- 172.21.15.60:0/3842242646 shutdown_connections 2022-06-29T16:03:26.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.143+0000 7f697f59e700 1 -- 172.21.15.60:0/3842242646 >> 172.21.15.60:0/3842242646 conn(0x7f69800b5890 msgr2=0x7f69800b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:26.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.143+0000 7f697f59e700 1 -- 172.21.15.60:0/3842242646 shutdown_connections 2022-06-29T16:03:26.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.143+0000 7f697f59e700 1 -- 172.21.15.60:0/3842242646 wait complete. 2022-06-29T16:03:26.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.147+0000 7f697f59e700 1 Processor -- start 2022-06-29T16:03:26.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.147+0000 7f697f59e700 1 -- start start 2022-06-29T16:03:26.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.147+0000 7f697f59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69640049f0 0x7f698014b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:26.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.147+0000 7f697f59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69801481b0 0x7f6980149d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:26.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.147+0000 7f697f59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6980063610 0x7f6980063a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:26.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.147+0000 7f697f59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6980148610 con 0x7f69640049f0 2022-06-29T16:03:26.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.147+0000 7f697f59e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6980146760 con 0x7f69801481b0 2022-06-29T16:03:26.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.147+0000 7f697f59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6980146a00 con 0x7f6980063610 2022-06-29T16:03:26.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.147+0000 7f697e59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69640049f0 0x7f698014b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:26.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.147+0000 7f697dd9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69801481b0 0x7f6980149d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:26.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.147+0000 7f697ed9d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6980063610 0x7f6980063a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:26.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.147+0000 7f697dd9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69801481b0 0x7f6980149d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:54546/0 (socket says 172.21.15.60:54546) 2022-06-29T16:03:26.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.147+0000 7f697dd9b700 1 -- 172.21.15.60:0/2282279020 learned_addr learned my addr 172.21.15.60:0/2282279020 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:26.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.147+0000 7f697e59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69640049f0 0x7f698014b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48518/0 (socket says 172.21.15.60:48518) 2022-06-29T16:03:26.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.147+0000 7f697dd9b700 1 -- 172.21.15.60:0/2282279020 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6980063610 msgr2=0x7f6980063a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:26.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.147+0000 7f697dd9b700 1 --2- 172.21.15.60:0/2282279020 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6980063610 0x7f6980063a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:26.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.147+0000 7f697dd9b700 1 -- 172.21.15.60:0/2282279020 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69640049f0 msgr2=0x7f698014b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:26.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.147+0000 7f697dd9b700 1 --2- 172.21.15.60:0/2282279020 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69640049f0 0x7f698014b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:26.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.147+0000 7f697dd9b700 1 -- 172.21.15.60:0/2282279020 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6980063f80 con 0x7f69801481b0 2022-06-29T16:03:26.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.147+0000 7f697dd9b700 1 --2- 172.21.15.60:0/2282279020 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69801481b0 0x7f6980149d00 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f696800b000 tx=0x7f6968001d30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:26.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.147+0000 7f69777fe700 1 -- 172.21.15.60:0/2282279020 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6968015630 con 0x7f69801481b0 2022-06-29T16:03:26.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.147+0000 7f69777fe700 1 -- 172.21.15.60:0/2282279020 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6968010040 con 0x7f69801481b0 2022-06-29T16:03:26.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.147+0000 7f697f59e700 1 -- 172.21.15.60:0/2282279020 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f698014c5c0 con 0x7f69801481b0 2022-06-29T16:03:26.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.147+0000 7f69777fe700 1 -- 172.21.15.60:0/2282279020 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6968015980 con 0x7f69801481b0 2022-06-29T16:03:26.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.147+0000 7f697f59e700 1 -- 172.21.15.60:0/2282279020 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69800664b0 con 0x7f69801481b0 2022-06-29T16:03:26.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.147+0000 7f697f59e700 1 -- 172.21.15.60:0/2282279020 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f698014a5f0 con 0x7f69801481b0 2022-06-29T16:03:26.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.151+0000 7f69777fe700 1 -- 172.21.15.60:0/2282279020 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f696801e3f0 con 0x7f69801481b0 2022-06-29T16:03:26.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.151+0000 7f69777fe700 1 --2- 172.21.15.60:0/2282279020 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6958053b00 0x7f6958055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:26.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.151+0000 7f69777fe700 1 -- 172.21.15.60:0/2282279020 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f696806b5b0 con 0x7f69801481b0 2022-06-29T16:03:26.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.151+0000 7f697e59c700 1 --2- 172.21.15.60:0/2282279020 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6958053b00 0x7f6958055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:26.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.151+0000 7f697e59c700 1 --2- 172.21.15.60:0/2282279020 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6958053b00 0x7f6958055fc0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f6970003000 tx=0x7f6970007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:26.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.159+0000 7f69777fe700 1 -- 172.21.15.60:0/2282279020 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6968042720 con 0x7f69801481b0 2022-06-29T16:03:26.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.291+0000 7f697f59e700 1 -- 172.21.15.60:0/2282279020 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f6980146580 con 0x7f69801481b0 2022-06-29T16:03:26.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.291+0000 7f69777fe700 1 -- 172.21.15.60:0/2282279020 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v57) v1 ==== 84+0+143 (secure 0 0 0) 0x7f6968045d40 con 0x7f69801481b0 2022-06-29T16:03:26.299 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:03:26.299 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2022-06-29T16:03:26.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:26.299 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2022-06-29T16:03:26.300 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": "device_health_metrics" 2022-06-29T16:03:26.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:26.300 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:26.300 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 2, 2022-06-29T16:03:26.301 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": "rbd" 2022-06-29T16:03:26.301 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:26.301 INFO:teuthology.orchestra.run.smithi060.stdout:] 2022-06-29T16:03:26.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.295+0000 7f697f59e700 1 -- 172.21.15.60:0/2282279020 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6958053b00 msgr2=0x7f6958055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:26.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.295+0000 7f697f59e700 1 --2- 172.21.15.60:0/2282279020 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6958053b00 0x7f6958055fc0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f6970003000 tx=0x7f6970007a00).stop 2022-06-29T16:03:26.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.295+0000 7f697f59e700 1 -- 172.21.15.60:0/2282279020 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69801481b0 msgr2=0x7f6980149d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:26.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.295+0000 7f697f59e700 1 --2- 172.21.15.60:0/2282279020 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69801481b0 0x7f6980149d00 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f696800b000 tx=0x7f6968001d30).stop 2022-06-29T16:03:26.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.295+0000 7f697f59e700 1 -- 172.21.15.60:0/2282279020 shutdown_connections 2022-06-29T16:03:26.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.295+0000 7f697f59e700 1 --2- 172.21.15.60:0/2282279020 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6958053b00 0x7f6958055fc0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:26.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.295+0000 7f697f59e700 1 --2- 172.21.15.60:0/2282279020 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6980063610 0x7f6980063a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:26.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.295+0000 7f697f59e700 1 --2- 172.21.15.60:0/2282279020 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69801481b0 0x7f6980149d00 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:26.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.295+0000 7f697f59e700 1 --2- 172.21.15.60:0/2282279020 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69640049f0 0x7f698014b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:26.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.295+0000 7f697f59e700 1 -- 172.21.15.60:0/2282279020 >> 172.21.15.60:0/2282279020 conn(0x7f69800b5890 msgr2=0x7f69800b45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:26.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.295+0000 7f697f59e700 1 -- 172.21.15.60:0/2282279020 shutdown_connections 2022-06-29T16:03:26.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.295+0000 7f697f59e700 1 -- 172.21.15.60:0/2282279020 wait complete. 2022-06-29T16:03:26.311 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-06-29T16:03:26.312 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-06-29T16:03:26.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.415+0000 7f3cf03b9700 1 Processor -- start 2022-06-29T16:03:26.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.415+0000 7f3cf03b9700 1 -- start start 2022-06-29T16:03:26.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.415+0000 7f3cf03b9700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ce80b5db0 0x7f3ce8144850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:26.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.415+0000 7f3cf03b9700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3ce805c9c0 0x7f3ce813a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:26.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.415+0000 7f3cf03b9700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ce80b6190 con 0x7f3ce805c2c0 2022-06-29T16:03:26.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.415+0000 7f3cf03b9700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ce8067df0 con 0x7f3ce805c9c0 2022-06-29T16:03:26.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.415+0000 7f3cf03b9700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce8144110 con 0x7f3ce80b5db0 2022-06-29T16:03:26.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.415+0000 7f3ced954700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f3ce805c2c0 0x7f3ce8140970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56612/0 (socket says 172.21.15.60:56612) 2022-06-29T16:03:26.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.415+0000 7f3ced954700 1 -- 172.21.15.60:0/1983075259 learned_addr learned my addr 172.21.15.60:0/1983075259 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:03:26.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.415+0000 7f3cee155700 1 --2- 172.21.15.60:0/1983075259 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ce80b5db0 0x7f3ce8144850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:26.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.415+0000 7f3cee956700 1 --2- 172.21.15.60:0/1983075259 >> v2:172.21.15.92:3300/0 conn(0x7f3ce805c9c0 0x7f3ce813a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:26.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.415+0000 7f3ced153700 1 -- 172.21.15.60:0/1983075259 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 548264838 0 0) 0x7f3ce80b6190 con 0x7f3ce805c2c0 2022-06-29T16:03:26.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.415+0000 7f3ced153700 1 -- 172.21.15.60:0/1983075259 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cd0003260 con 0x7f3ce805c2c0 2022-06-29T16:03:26.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.415+0000 7f3cee155700 1 -- 172.21.15.60:0/1983075259 >> v2:172.21.15.92:3300/0 conn(0x7f3ce805c9c0 msgr2=0x7f3ce813a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:26.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.415+0000 7f3cee155700 1 --2- 172.21.15.60:0/1983075259 >> v2:172.21.15.92:3300/0 conn(0x7f3ce805c9c0 0x7f3ce813a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:26.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.415+0000 7f3cee155700 1 -- 172.21.15.60:0/1983075259 >> v1:172.21.15.60:6789/0 conn(0x7f3ce805c2c0 legacy=0x7f3ce8140970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:26.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.415+0000 7f3cee155700 1 -- 172.21.15.60:0/1983075259 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce81443b0 con 0x7f3ce80b5db0 2022-06-29T16:03:26.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.415+0000 7f3cee956700 1 --2- 172.21.15.60:0/1983075259 >> v2:172.21.15.92:3300/0 conn(0x7f3ce805c9c0 0x7f3ce813a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:03:26.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.415+0000 7f3cee155700 1 --2- 172.21.15.60:0/1983075259 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ce80b5db0 0x7f3ce8144850 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f3ce4002f70 tx=0x7f3ce4006df0).ready entity=mon.2 client_cookie=394e47881f3c450a server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:26.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.415+0000 7f3ced153700 1 -- 172.21.15.60:0/1983075259 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ce400d4a0 con 0x7f3ce80b5db0 2022-06-29T16:03:26.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.415+0000 7f3ced153700 1 -- 172.21.15.60:0/1983075259 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ce4000e90 con 0x7f3ce80b5db0 2022-06-29T16:03:26.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.415+0000 7f3ced153700 1 -- 172.21.15.60:0/1983075259 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ce400d7f0 con 0x7f3ce80b5db0 2022-06-29T16:03:26.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.419+0000 7f3cf03b9700 1 -- 172.21.15.60:0/1983075259 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ce80b5db0 msgr2=0x7f3ce8144850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:26.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.419+0000 7f3cf03b9700 1 --2- 172.21.15.60:0/1983075259 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ce80b5db0 0x7f3ce8144850 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f3ce4002f70 tx=0x7f3ce4006df0).stop 2022-06-29T16:03:26.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.419+0000 7f3cf03b9700 1 -- 172.21.15.60:0/1983075259 shutdown_connections 2022-06-29T16:03:26.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.419+0000 7f3cf03b9700 1 --2- 172.21.15.60:0/1983075259 >> v2:172.21.15.92:3300/0 conn(0x7f3ce805c9c0 0x7f3ce813a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:26.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.419+0000 7f3cf03b9700 1 --2- 172.21.15.60:0/1983075259 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ce80b5db0 0x7f3ce8144850 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:26.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.419+0000 7f3cf03b9700 1 -- 172.21.15.60:0/1983075259 >> 172.21.15.60:0/1983075259 conn(0x7f3ce80636f0 msgr2=0x7f3ce80b5360 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:26.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.419+0000 7f3cf03b9700 1 -- 172.21.15.60:0/1983075259 shutdown_connections 2022-06-29T16:03:26.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.419+0000 7f3cf03b9700 1 -- 172.21.15.60:0/1983075259 wait complete. 2022-06-29T16:03:26.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.419+0000 7f3cf03b9700 1 Processor -- start 2022-06-29T16:03:26.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.419+0000 7f3cf03b9700 1 -- start start 2022-06-29T16:03:26.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.419+0000 7f3cf03b9700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ce805c2c0 0x7f3ce813ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:26.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.419+0000 7f3cf03b9700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ce805c9c0 0x7f3ce814fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:26.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.419+0000 7f3cf03b9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ce80b5db0 0x7f3ce814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:26.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.419+0000 7f3cf03b9700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce81530d0 con 0x7f3ce80b5db0 2022-06-29T16:03:26.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.419+0000 7f3cf03b9700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce8153370 con 0x7f3ce805c9c0 2022-06-29T16:03:26.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.419+0000 7f3cf03b9700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce8153610 con 0x7f3ce805c2c0 2022-06-29T16:03:26.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.419+0000 7f3cee956700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ce80b5db0 0x7f3ce814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:26.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.419+0000 7f3cee956700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ce80b5db0 0x7f3ce814fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48534/0 (socket says 172.21.15.60:48534) 2022-06-29T16:03:26.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.419+0000 7f3cee956700 1 -- 172.21.15.60:0/1079074724 learned_addr learned my addr 172.21.15.60:0/1079074724 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:26.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.419+0000 7f3ced954700 1 --2- 172.21.15.60:0/1079074724 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ce805c9c0 0x7f3ce814fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:26.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.419+0000 7f3cee155700 1 --2- 172.21.15.60:0/1079074724 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ce805c2c0 0x7f3ce813ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:26.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.423+0000 7f3cee155700 1 -- 172.21.15.60:0/1079074724 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ce805c9c0 msgr2=0x7f3ce814fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:26.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.423+0000 7f3cee155700 1 --2- 172.21.15.60:0/1079074724 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ce805c9c0 0x7f3ce814fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:26.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.423+0000 7f3cee155700 1 -- 172.21.15.60:0/1079074724 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ce80b5db0 msgr2=0x7f3ce814fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:26.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.423+0000 7f3cee155700 1 --2- 172.21.15.60:0/1079074724 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ce80b5db0 0x7f3ce814fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:26.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.423+0000 7f3cee155700 1 -- 172.21.15.60:0/1079074724 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce0009470 con 0x7f3ce805c2c0 2022-06-29T16:03:26.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.423+0000 7f3cee155700 1 --2- 172.21.15.60:0/1079074724 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ce805c2c0 0x7f3ce813ca90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f3ce4002f70 tx=0x7f3ce4014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:26.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.423+0000 7f3cd77fe700 1 -- 172.21.15.60:0/1079074724 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ce4006780 con 0x7f3ce805c2c0 2022-06-29T16:03:26.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.423+0000 7f3cd77fe700 1 -- 172.21.15.60:0/1079074724 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ce4013bb0 con 0x7f3ce805c2c0 2022-06-29T16:03:26.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.423+0000 7f3cf03b9700 1 -- 172.21.15.60:0/1079074724 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ce400b180 con 0x7f3ce805c2c0 2022-06-29T16:03:26.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.423+0000 7f3cd77fe700 1 -- 172.21.15.60:0/1079074724 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ce401e400 con 0x7f3ce805c2c0 2022-06-29T16:03:26.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.423+0000 7f3cf03b9700 1 -- 172.21.15.60:0/1079074724 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ce81504e0 con 0x7f3ce805c2c0 2022-06-29T16:03:26.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.423+0000 7f3cf03b9700 1 -- 172.21.15.60:0/1079074724 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ce8153ab0 con 0x7f3ce805c2c0 2022-06-29T16:03:26.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.423+0000 7f3cd77fe700 1 -- 172.21.15.60:0/1079074724 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3ce401c070 con 0x7f3ce805c2c0 2022-06-29T16:03:26.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.427+0000 7f3cd77fe700 1 --2- 172.21.15.60:0/1079074724 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3cc8053aa0 0x7f3cc8055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:26.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.427+0000 7f3cd77fe700 1 -- 172.21.15.60:0/1079074724 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f3ce406c3e0 con 0x7f3ce805c2c0 2022-06-29T16:03:26.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.427+0000 7f3ced954700 1 --2- 172.21.15.60:0/1079074724 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3cc8053aa0 0x7f3cc8055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:26.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.427+0000 7f3ced954700 1 --2- 172.21.15.60:0/1079074724 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3cc8053aa0 0x7f3cc8055f60 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f3cd800d000 tx=0x7f3cd8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:26.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.435+0000 7f3cd77fe700 1 -- 172.21.15.60:0/1079074724 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3ce8153ab0 con 0x7f3ce805c2c0 2022-06-29T16:03:26.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.563+0000 7f3cf03b9700 1 -- 172.21.15.60:0/1079074724 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f3ce8153ab0 con 0x7f3ce805c2c0 2022-06-29T16:03:26.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.871+0000 7f3cd77fe700 1 -- 172.21.15.60:0/1079074724 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v58) v1 ==== 133+0+0 (secure 0 0 0) 0x7f3ce8153ab0 con 0x7f3ce805c2c0 2022-06-29T16:03:26.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.875+0000 7f3cf03b9700 1 -- 172.21.15.60:0/1079074724 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3cc8053aa0 msgr2=0x7f3cc8055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:26.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.875+0000 7f3cf03b9700 1 --2- 172.21.15.60:0/1079074724 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3cc8053aa0 0x7f3cc8055f60 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f3cd800d000 tx=0x7f3cd8009040).stop 2022-06-29T16:03:26.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.875+0000 7f3cf03b9700 1 -- 172.21.15.60:0/1079074724 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ce805c2c0 msgr2=0x7f3ce813ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:26.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.875+0000 7f3cf03b9700 1 --2- 172.21.15.60:0/1079074724 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ce805c2c0 0x7f3ce813ca90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f3ce4002f70 tx=0x7f3ce4014000).stop 2022-06-29T16:03:26.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.875+0000 7f3cf03b9700 1 -- 172.21.15.60:0/1079074724 shutdown_connections 2022-06-29T16:03:26.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.875+0000 7f3cf03b9700 1 --2- 172.21.15.60:0/1079074724 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3cc8053aa0 0x7f3cc8055f60 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:26.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.875+0000 7f3cf03b9700 1 --2- 172.21.15.60:0/1079074724 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ce80b5db0 0x7f3ce814fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:26.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.875+0000 7f3cf03b9700 1 --2- 172.21.15.60:0/1079074724 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ce805c9c0 0x7f3ce814fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:26.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.875+0000 7f3cf03b9700 1 --2- 172.21.15.60:0/1079074724 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ce805c2c0 0x7f3ce813ca90 secure :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0x7f3ce4002f70 tx=0x7f3ce4014000).stop 2022-06-29T16:03:26.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.879+0000 7f3cf03b9700 1 -- 172.21.15.60:0/1079074724 >> 172.21.15.60:0/1079074724 conn(0x7f3ce80636f0 msgr2=0x7f3ce813b790 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:26.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.879+0000 7f3cf03b9700 1 -- 172.21.15.60:0/1079074724 shutdown_connections 2022-06-29T16:03:26.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:26.879+0000 7f3cf03b9700 1 -- 172.21.15.60:0/1079074724 wait complete. 2022-06-29T16:03:26.887 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2022-06-29T16:03:26.899 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-06-29T16:03:27.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.011+0000 7f076d951700 1 Processor -- start 2022-06-29T16:03:27.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.011+0000 7f076d951700 1 -- start start 2022-06-29T16:03:27.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.011+0000 7f076d951700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f07680b28b0 0x7f076813f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:27.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.011+0000 7f076d951700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0768060650 0x7f07680b2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:27.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.011+0000 7f076d951700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0768142d60 con 0x7f0768143190 2022-06-29T16:03:27.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.011+0000 7f076d951700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07680b3180 con 0x7f0768060650 2022-06-29T16:03:27.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.011+0000 7f076d951700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07680654f0 con 0x7f07680b28b0 2022-06-29T16:03:27.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.011+0000 7f0767fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0768060650 0x7f07680b2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:27.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.011+0000 7f0767fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0768060650 0x7f07680b2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48540/0 (socket says 172.21.15.60:48540) 2022-06-29T16:03:27.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.011+0000 7f0767fff700 1 -- 172.21.15.60:0/3904564607 learned_addr learned my addr 172.21.15.60:0/3904564607 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:27.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.011+0000 7f076c94f700 1 --2- 172.21.15.60:0/3904564607 >> v2:172.21.15.92:3300/0 conn(0x7f07680b28b0 0x7f076813f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:27.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.015+0000 7f07677fe700 1 -- 172.21.15.60:0/3904564607 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506617090 0 0) 0x7f0768142d60 con 0x7f0768143190 2022-06-29T16:03:27.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.015+0000 7f07677fe700 1 -- 172.21.15.60:0/3904564607 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0748003260 con 0x7f0768143190 2022-06-29T16:03:27.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.015+0000 7f0767fff700 1 -- 172.21.15.60:0/3904564607 >> v2:172.21.15.92:3300/0 conn(0x7f07680b28b0 msgr2=0x7f076813f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:27.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.015+0000 7f0767fff700 1 --2- 172.21.15.60:0/3904564607 >> v2:172.21.15.92:3300/0 conn(0x7f07680b28b0 0x7f076813f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:27.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.015+0000 7f0767fff700 1 -- 172.21.15.60:0/3904564607 >> v1:172.21.15.92:6789/0 conn(0x7f0768143190 legacy=0x7f0768143620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:27.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.015+0000 7f0767fff700 1 -- 172.21.15.60:0/3904564607 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0768065790 con 0x7f0768060650 2022-06-29T16:03:27.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.015+0000 7f0767fff700 1 --2- 172.21.15.60:0/3904564607 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0768060650 0x7f07680b2c90 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7f0750002f70 tx=0x7f0750006eb0).ready entity=mon.0 client_cookie=e05d236a840f7490 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:27.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.015+0000 7f07677fe700 1 -- 172.21.15.60:0/3904564607 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f075000d610 con 0x7f0768060650 2022-06-29T16:03:27.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.015+0000 7f07677fe700 1 -- 172.21.15.60:0/3904564607 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0750000e90 con 0x7f0768060650 2022-06-29T16:03:27.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.015+0000 7f07677fe700 1 -- 172.21.15.60:0/3904564607 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f075000d9a0 con 0x7f0768060650 2022-06-29T16:03:27.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.015+0000 7f076d951700 1 -- 172.21.15.60:0/3904564607 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0768060650 msgr2=0x7f07680b2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:27.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.015+0000 7f076d951700 1 --2- 172.21.15.60:0/3904564607 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0768060650 0x7f07680b2c90 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7f0750002f70 tx=0x7f0750006eb0).stop 2022-06-29T16:03:27.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.015+0000 7f076d951700 1 -- 172.21.15.60:0/3904564607 shutdown_connections 2022-06-29T16:03:27.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.015+0000 7f076d951700 1 --2- 172.21.15.60:0/3904564607 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0768060650 0x7f07680b2c90 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:27.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.015+0000 7f076d951700 1 --2- 172.21.15.60:0/3904564607 >> v2:172.21.15.92:3300/0 conn(0x7f07680b28b0 0x7f076813f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:27.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.015+0000 7f076d951700 1 -- 172.21.15.60:0/3904564607 >> 172.21.15.60:0/3904564607 conn(0x7f07680637c0 msgr2=0x7f0768063ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:27.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.015+0000 7f076d951700 1 -- 172.21.15.60:0/3904564607 shutdown_connections 2022-06-29T16:03:27.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.015+0000 7f076d951700 1 -- 172.21.15.60:0/3904564607 wait complete. 2022-06-29T16:03:27.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.019+0000 7f076d951700 1 Processor -- start 2022-06-29T16:03:27.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.019+0000 7f076d951700 1 -- start start 2022-06-29T16:03:27.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.019+0000 7f076d951700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0768060650 0x7f07680b5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:27.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.019+0000 7f076d951700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07680b28b0 0x7f0768151d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:27.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.019+0000 7f076d951700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0768143190 0x7f076813c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:27.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.019+0000 7f076d951700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0768152430 con 0x7f0768060650 2022-06-29T16:03:27.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.019+0000 7f076d951700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07681526d0 con 0x7f0768143190 2022-06-29T16:03:27.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.019+0000 7f076d951700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0768152970 con 0x7f07680b28b0 2022-06-29T16:03:27.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.019+0000 7f076c94f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0768060650 0x7f07680b5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:27.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.019+0000 7f076c94f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0768060650 0x7f07680b5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48544/0 (socket says 172.21.15.60:48544) 2022-06-29T16:03:27.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.019+0000 7f076c94f700 1 -- 172.21.15.60:0/3370429336 learned_addr learned my addr 172.21.15.60:0/3370429336 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:27.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.019+0000 7f0767fff700 1 --2- 172.21.15.60:0/3370429336 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07680b28b0 0x7f0768151d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:27.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.019+0000 7f076d150700 1 --2- 172.21.15.60:0/3370429336 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0768143190 0x7f076813c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:27.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.019+0000 7f076c94f700 1 -- 172.21.15.60:0/3370429336 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07680b28b0 msgr2=0x7f0768151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:27.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.019+0000 7f076c94f700 1 --2- 172.21.15.60:0/3370429336 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07680b28b0 0x7f0768151d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:27.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.019+0000 7f076c94f700 1 -- 172.21.15.60:0/3370429336 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0768143190 msgr2=0x7f076813c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:27.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.019+0000 7f076c94f700 1 --2- 172.21.15.60:0/3370429336 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0768143190 0x7f076813c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:27.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.019+0000 7f076c94f700 1 -- 172.21.15.60:0/3370429336 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f075000b1e0 con 0x7f0768060650 2022-06-29T16:03:27.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.019+0000 7f076c94f700 1 --2- 172.21.15.60:0/3370429336 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0768060650 0x7f07680b5400 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0x7f075800c000 tx=0x7f0758006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:27.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.023+0000 7f0765ffb700 1 -- 172.21.15.60:0/3370429336 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f075800c830 con 0x7f0768060650 2022-06-29T16:03:27.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.023+0000 7f0765ffb700 1 -- 172.21.15.60:0/3370429336 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f075800c990 con 0x7f0768060650 2022-06-29T16:03:27.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.023+0000 7f076d951700 1 -- 172.21.15.60:0/3370429336 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f076810f1c0 con 0x7f0768060650 2022-06-29T16:03:27.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.023+0000 7f0765ffb700 1 -- 172.21.15.60:0/3370429336 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f075800ccf0 con 0x7f0768060650 2022-06-29T16:03:27.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.023+0000 7f076d951700 1 -- 172.21.15.60:0/3370429336 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f076813cd40 con 0x7f0768060650 2022-06-29T16:03:27.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.023+0000 7f076d951700 1 -- 172.21.15.60:0/3370429336 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0768152280 con 0x7f0768060650 2022-06-29T16:03:27.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.023+0000 7f0765ffb700 1 -- 172.21.15.60:0/3370429336 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f075801c460 con 0x7f0768060650 2022-06-29T16:03:27.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.023+0000 7f0765ffb700 1 --2- 172.21.15.60:0/3370429336 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f074c053aa0 0x7f074c055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:27.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.023+0000 7f0765ffb700 1 -- 172.21.15.60:0/3370429336 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f075806c180 con 0x7f0768060650 2022-06-29T16:03:27.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.023+0000 7f0767fff700 1 --2- 172.21.15.60:0/3370429336 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f074c053aa0 0x7f074c055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:27.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.027+0000 7f0767fff700 1 --2- 172.21.15.60:0/3370429336 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f074c053aa0 0x7f074c055f60 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f075001a000 tx=0x7f0750015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:27.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.031+0000 7f0765ffb700 1 -- 172.21.15.60:0/3370429336 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0758043210 con 0x7f0768060650 2022-06-29T16:03:27.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.159+0000 7f076d951700 1 -- 172.21.15.60:0/3370429336 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f07680573d0 con 0x7f0768060650 2022-06-29T16:03:27.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.795+0000 7f0765ffb700 1 -- 172.21.15.60:0/3370429336 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v59) v1 ==== 125+0+0 (secure 0 0 0) 0x7f0758046830 con 0x7f0768060650 2022-06-29T16:03:27.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.803+0000 7f076d951700 1 -- 172.21.15.60:0/3370429336 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f074c053aa0 msgr2=0x7f074c055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:27.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.803+0000 7f076d951700 1 --2- 172.21.15.60:0/3370429336 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f074c053aa0 0x7f074c055f60 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f075001a000 tx=0x7f0750015000).stop 2022-06-29T16:03:27.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.803+0000 7f076d951700 1 -- 172.21.15.60:0/3370429336 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0768060650 msgr2=0x7f07680b5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:27.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.803+0000 7f076d951700 1 --2- 172.21.15.60:0/3370429336 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0768060650 0x7f07680b5400 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0x7f075800c000 tx=0x7f0758006df0).stop 2022-06-29T16:03:27.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.807+0000 7f076d951700 1 -- 172.21.15.60:0/3370429336 shutdown_connections 2022-06-29T16:03:27.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.807+0000 7f076d951700 1 --2- 172.21.15.60:0/3370429336 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f074c053aa0 0x7f074c055f60 secure :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0x7f075001a000 tx=0x7f0750015000).stop 2022-06-29T16:03:27.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.807+0000 7f076d951700 1 --2- 172.21.15.60:0/3370429336 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0768143190 0x7f076813c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:27.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.807+0000 7f076d951700 1 --2- 172.21.15.60:0/3370429336 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07680b28b0 0x7f0768151d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:27.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.807+0000 7f076d951700 1 --2- 172.21.15.60:0/3370429336 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0768060650 0x7f07680b5400 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:27.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.807+0000 7f076d951700 1 -- 172.21.15.60:0/3370429336 >> 172.21.15.60:0/3370429336 conn(0x7f07680637c0 msgr2=0x7f076805f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:27.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.807+0000 7f076d951700 1 -- 172.21.15.60:0/3370429336 shutdown_connections 2022-06-29T16:03:27.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.807+0000 7f076d951700 1 -- 172.21.15.60:0/3370429336 wait complete. 2022-06-29T16:03:27.815 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2022-06-29T16:03:27.826 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-06-29T16:03:27.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.927+0000 7f512b59e700 1 Processor -- start 2022-06-29T16:03:27.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.927+0000 7f512b59e700 1 -- start start 2022-06-29T16:03:27.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.927+0000 7f512b59e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f512c066590 0x7f512c0673e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:27.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.927+0000 7f512b59e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f512c0b1480 0x7f512c0afa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:27.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.927+0000 7f512b59e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f512c0b0f50 con 0x7f512c140300 2022-06-29T16:03:27.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.927+0000 7f512b59e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f512c067080 con 0x7f512c0b1480 2022-06-29T16:03:27.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.927+0000 7f512b59e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f512c065670 con 0x7f512c066590 2022-06-29T16:03:27.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.927+0000 7f512ad9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f512c0b1480 0x7f512c0afa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:27.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.927+0000 7f512ad9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f512c0b1480 0x7f512c0afa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48558/0 (socket says 172.21.15.60:48558) 2022-06-29T16:03:27.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.927+0000 7f512ad9d700 1 -- 172.21.15.60:0/2235066008 learned_addr learned my addr 172.21.15.60:0/2235066008 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:27.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.927+0000 7f512a59c700 1 --2- 172.21.15.60:0/2235066008 >> v2:172.21.15.92:3300/0 conn(0x7f512c066590 0x7f512c0673e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:27.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.927+0000 7f512ad9d700 1 -- 172.21.15.60:0/2235066008 >> v2:172.21.15.92:3300/0 conn(0x7f512c066590 msgr2=0x7f512c0673e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:27.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.927+0000 7f512ad9d700 1 --2- 172.21.15.60:0/2235066008 >> v2:172.21.15.92:3300/0 conn(0x7f512c066590 0x7f512c0673e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:27.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.927+0000 7f512ad9d700 1 -- 172.21.15.60:0/2235066008 >> v1:172.21.15.92:6789/0 conn(0x7f512c140300 legacy=0x7f512c066970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:27.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.927+0000 7f512ad9d700 1 -- 172.21.15.60:0/2235066008 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f512c064db0 con 0x7f512c0b1480 2022-06-29T16:03:27.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.927+0000 7f512ad9d700 1 --2- 172.21.15.60:0/2235066008 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f512c0b1480 0x7f512c0afa50 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7f51240032f0 tx=0x7f5124009290).ready entity=mon.0 client_cookie=3330a330af950ad server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:27.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.927+0000 7f512959a700 1 -- 172.21.15.60:0/2235066008 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f512400b870 con 0x7f512c0b1480 2022-06-29T16:03:27.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.927+0000 7f512959a700 1 -- 172.21.15.60:0/2235066008 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f512400b9d0 con 0x7f512c0b1480 2022-06-29T16:03:27.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.927+0000 7f512959a700 1 -- 172.21.15.60:0/2235066008 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f512400bb40 con 0x7f512c0b1480 2022-06-29T16:03:27.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.927+0000 7f512b59e700 1 -- 172.21.15.60:0/2235066008 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f512c0b1480 msgr2=0x7f512c0afa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:27.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.927+0000 7f512b59e700 1 --2- 172.21.15.60:0/2235066008 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f512c0b1480 0x7f512c0afa50 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7f51240032f0 tx=0x7f5124009290).stop 2022-06-29T16:03:27.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.927+0000 7f512b59e700 1 -- 172.21.15.60:0/2235066008 shutdown_connections 2022-06-29T16:03:27.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.927+0000 7f512b59e700 1 --2- 172.21.15.60:0/2235066008 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f512c0b1480 0x7f512c0afa50 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:27.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.927+0000 7f512b59e700 1 --2- 172.21.15.60:0/2235066008 >> v2:172.21.15.92:3300/0 conn(0x7f512c066590 0x7f512c0673e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:27.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.927+0000 7f512b59e700 1 -- 172.21.15.60:0/2235066008 >> 172.21.15.60:0/2235066008 conn(0x7f512c0b5e70 msgr2=0x7f512c0b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:27.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.927+0000 7f512b59e700 1 -- 172.21.15.60:0/2235066008 shutdown_connections 2022-06-29T16:03:27.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.927+0000 7f512b59e700 1 -- 172.21.15.60:0/2235066008 wait complete. 2022-06-29T16:03:27.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.931+0000 7f512b59e700 1 Processor -- start 2022-06-29T16:03:27.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.931+0000 7f512b59e700 1 -- start start 2022-06-29T16:03:27.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.931+0000 7f512b59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f512c066590 0x7f512c1435b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:27.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.931+0000 7f512b59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f512c0b1480 0x7f512c140e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:27.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.931+0000 7f512b59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f512c140300 0x7f512c1412f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:27.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.931+0000 7f512b59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f512c142510 con 0x7f512c140300 2022-06-29T16:03:27.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.931+0000 7f512b59e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f512c13b120 con 0x7f512c066590 2022-06-29T16:03:27.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.931+0000 7f512b59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f512c13b3c0 con 0x7f512c0b1480 2022-06-29T16:03:27.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.931+0000 7f512ad9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f512c140300 0x7f512c1412f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:27.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.931+0000 7f512ad9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f512c140300 0x7f512c1412f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48564/0 (socket says 172.21.15.60:48564) 2022-06-29T16:03:27.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.931+0000 7f512ad9d700 1 -- 172.21.15.60:0/3941042132 learned_addr learned my addr 172.21.15.60:0/3941042132 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:27.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.931+0000 7f512a59c700 1 --2- 172.21.15.60:0/3941042132 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f512c066590 0x7f512c1435b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:27.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.931+0000 7f5129d9b700 1 --2- 172.21.15.60:0/3941042132 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f512c0b1480 0x7f512c140e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:27.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.931+0000 7f512ad9d700 1 -- 172.21.15.60:0/3941042132 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f512c0b1480 msgr2=0x7f512c140e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:27.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.931+0000 7f512ad9d700 1 --2- 172.21.15.60:0/3941042132 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f512c0b1480 0x7f512c140e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:27.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.931+0000 7f512ad9d700 1 -- 172.21.15.60:0/3941042132 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f512c066590 msgr2=0x7f512c1435b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:27.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.931+0000 7f512ad9d700 1 --2- 172.21.15.60:0/3941042132 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f512c066590 0x7f512c1435b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:27.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.931+0000 7f512ad9d700 1 -- 172.21.15.60:0/3941042132 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5124009530 con 0x7f512c140300 2022-06-29T16:03:27.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.931+0000 7f512ad9d700 1 --2- 172.21.15.60:0/3941042132 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f512c140300 0x7f512c1412f0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7f51240032f0 tx=0x7f5124016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:27.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.931+0000 7f51237fe700 1 -- 172.21.15.60:0/3941042132 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f512400b870 con 0x7f512c140300 2022-06-29T16:03:27.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.931+0000 7f51237fe700 1 -- 172.21.15.60:0/3941042132 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5124021630 con 0x7f512c140300 2022-06-29T16:03:27.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.931+0000 7f512b59e700 1 -- 172.21.15.60:0/3941042132 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f512c13b660 con 0x7f512c140300 2022-06-29T16:03:27.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.931+0000 7f51237fe700 1 -- 172.21.15.60:0/3941042132 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5124021990 con 0x7f512c140300 2022-06-29T16:03:27.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.931+0000 7f512b59e700 1 -- 172.21.15.60:0/3941042132 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f512c0af930 con 0x7f512c140300 2022-06-29T16:03:27.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.931+0000 7f512b59e700 1 -- 172.21.15.60:0/3941042132 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f512c0acd60 con 0x7f512c140300 2022-06-29T16:03:27.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.935+0000 7f51237fe700 1 -- 172.21.15.60:0/3941042132 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5124027020 con 0x7f512c140300 2022-06-29T16:03:27.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.935+0000 7f51237fe700 1 --2- 172.21.15.60:0/3941042132 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5110053a70 0x7f5110055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:27.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.935+0000 7f51237fe700 1 -- 172.21.15.60:0/3941042132 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f512406d1f0 con 0x7f512c140300 2022-06-29T16:03:27.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.935+0000 7f512a59c700 1 --2- 172.21.15.60:0/3941042132 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5110053a70 0x7f5110055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:27.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.935+0000 7f512a59c700 1 --2- 172.21.15.60:0/3941042132 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5110053a70 0x7f5110055f30 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f511400b000 tx=0x7f5114006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:27.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:27.943+0000 7f51237fe700 1 -- 172.21.15.60:0/3941042132 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5124044180 con 0x7f512c140300 2022-06-29T16:03:28.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.107+0000 7f512b59e700 1 -- 172.21.15.60:0/3941042132 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f512c0573d0 con 0x7f512c140300 2022-06-29T16:03:28.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.115+0000 7f51237fe700 1 -- 172.21.15.60:0/3941042132 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 11 and data pool 12 v24) v1 ==== 189+0+0 (secure 0 0 0) 0x7f5124015020 con 0x7f512c140300 2022-06-29T16:03:28.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.119+0000 7f512b59e700 1 -- 172.21.15.60:0/3941042132 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5110053a70 msgr2=0x7f5110055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:28.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.119+0000 7f512b59e700 1 --2- 172.21.15.60:0/3941042132 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5110053a70 0x7f5110055f30 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f511400b000 tx=0x7f5114006df0).stop 2022-06-29T16:03:28.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.119+0000 7f512b59e700 1 -- 172.21.15.60:0/3941042132 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f512c140300 msgr2=0x7f512c1412f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:28.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.119+0000 7f512b59e700 1 --2- 172.21.15.60:0/3941042132 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f512c140300 0x7f512c1412f0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7f51240032f0 tx=0x7f5124016000).stop 2022-06-29T16:03:28.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.119+0000 7f512b59e700 1 -- 172.21.15.60:0/3941042132 shutdown_connections 2022-06-29T16:03:28.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.119+0000 7f512b59e700 1 --2- 172.21.15.60:0/3941042132 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5110053a70 0x7f5110055f30 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:28.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.119+0000 7f512b59e700 1 --2- 172.21.15.60:0/3941042132 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f512c140300 0x7f512c1412f0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:28.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.119+0000 7f512b59e700 1 --2- 172.21.15.60:0/3941042132 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f512c0b1480 0x7f512c140e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:28.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.119+0000 7f512b59e700 1 --2- 172.21.15.60:0/3941042132 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f512c066590 0x7f512c1435b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:28.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.119+0000 7f512b59e700 1 -- 172.21.15.60:0/3941042132 >> 172.21.15.60:0/3941042132 conn(0x7f512c0b5e70 msgr2=0x7f512c13f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:28.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.123+0000 7f512b59e700 1 -- 172.21.15.60:0/3941042132 shutdown_connections 2022-06-29T16:03:28.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.123+0000 7f512b59e700 1 -- 172.21.15.60:0/3941042132 wait complete. 2022-06-29T16:03:28.131 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 11 and data pool 12 2022-06-29T16:03:28.143 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:03:28.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.227+0000 7f9392d21700 1 Processor -- start 2022-06-29T16:03:28.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.227+0000 7f9392d21700 1 -- start start 2022-06-29T16:03:28.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.227+0000 7f9392d21700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f938c05d070 0x7f938c14aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:28.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.227+0000 7f9392d21700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f938c05d450 con 0x7f938c14af80 2022-06-29T16:03:28.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.227+0000 7f9392d21700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f938c0ad840 con 0x7f938c0b2130 2022-06-29T16:03:28.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.227+0000 7f9392d21700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f938c0adf90 con 0x7f938c05d070 2022-06-29T16:03:28.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.227+0000 7f9392520700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f938c14af80 0x7f938c0ad130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56658/0 (socket says 172.21.15.60:56658) 2022-06-29T16:03:28.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.227+0000 7f9392520700 1 -- 172.21.15.60:0/2839777581 learned_addr learned my addr 172.21.15.60:0/2839777581 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:03:28.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.227+0000 7f9391d1f700 1 --2- 172.21.15.60:0/2839777581 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f938c05d070 0x7f938c14aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:28.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.227+0000 7f9390d1d700 1 -- 172.21.15.60:0/2839777581 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 293892418 0 0) 0x7f938c05d450 con 0x7f938c14af80 2022-06-29T16:03:28.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.227+0000 7f9390d1d700 1 -- 172.21.15.60:0/2839777581 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9370003260 con 0x7f938c14af80 2022-06-29T16:03:28.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.227+0000 7f9390d1d700 1 -- 172.21.15.60:0/2839777581 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 374293566 0 0) 0x7f938c0ad840 con 0x7f938c0b2130 2022-06-29T16:03:28.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.227+0000 7f9390d1d700 1 -- 172.21.15.60:0/2839777581 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f938c05d450 con 0x7f938c0b2130 2022-06-29T16:03:28.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.227+0000 7f9391d1f700 1 -- 172.21.15.60:0/2839777581 >> v1:172.21.15.92:6789/0 conn(0x7f938c0b2130 legacy=0x7f938c0aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:28.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.227+0000 7f9391d1f700 1 -- 172.21.15.60:0/2839777581 >> v1:172.21.15.60:6789/0 conn(0x7f938c14af80 legacy=0x7f938c0ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:28.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.227+0000 7f9391d1f700 1 -- 172.21.15.60:0/2839777581 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f938c0b25d0 con 0x7f938c05d070 2022-06-29T16:03:28.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.227+0000 7f9391d1f700 1 --2- 172.21.15.60:0/2839777581 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f938c05d070 0x7f938c14aa90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f9380002f70 tx=0x7f9380006df0).ready entity=mon.2 client_cookie=912301455dc74f02 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:28.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.227+0000 7f9390d1d700 1 -- 172.21.15.60:0/2839777581 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f938000d5c0 con 0x7f938c05d070 2022-06-29T16:03:28.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.227+0000 7f9390d1d700 1 -- 172.21.15.60:0/2839777581 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9380000e90 con 0x7f938c05d070 2022-06-29T16:03:28.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.227+0000 7f9390d1d700 1 -- 172.21.15.60:0/2839777581 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f938000d910 con 0x7f938c05d070 2022-06-29T16:03:28.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.231+0000 7f9392d21700 1 -- 172.21.15.60:0/2839777581 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f938c05d070 msgr2=0x7f938c14aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:28.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.231+0000 7f9392d21700 1 --2- 172.21.15.60:0/2839777581 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f938c05d070 0x7f938c14aa90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f9380002f70 tx=0x7f9380006df0).stop 2022-06-29T16:03:28.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.231+0000 7f9392d21700 1 -- 172.21.15.60:0/2839777581 shutdown_connections 2022-06-29T16:03:28.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.231+0000 7f9392d21700 1 --2- 172.21.15.60:0/2839777581 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f938c05d070 0x7f938c14aa90 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:28.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.231+0000 7f9392d21700 1 -- 172.21.15.60:0/2839777581 >> 172.21.15.60:0/2839777581 conn(0x7f938c0b5890 msgr2=0x7f938c0b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:28.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.231+0000 7f9392d21700 1 -- 172.21.15.60:0/2839777581 shutdown_connections 2022-06-29T16:03:28.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.231+0000 7f9392d21700 1 -- 172.21.15.60:0/2839777581 wait complete. 2022-06-29T16:03:28.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.231+0000 7f9392d21700 1 Processor -- start 2022-06-29T16:03:28.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.231+0000 7f9392d21700 1 -- start start 2022-06-29T16:03:28.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.231+0000 7f9392d21700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f938c05d070 0x7f938c0651c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:28.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.231+0000 7f9392d21700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f938c0b2130 0x7f938c0b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:28.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.231+0000 7f9392d21700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f938c14af80 0x7f938c155760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:28.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.231+0000 7f9392d21700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f938c117200 con 0x7f938c0b2130 2022-06-29T16:03:28.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.231+0000 7f9392d21700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f938c066a20 con 0x7f938c05d070 2022-06-29T16:03:28.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.231+0000 7f9392d21700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f938c066cc0 con 0x7f938c14af80 2022-06-29T16:03:28.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.231+0000 7f939151e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f938c0b2130 0x7f938c0b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:28.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.231+0000 7f939151e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f938c0b2130 0x7f938c0b42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48576/0 (socket says 172.21.15.60:48576) 2022-06-29T16:03:28.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.231+0000 7f939151e700 1 -- 172.21.15.60:0/3264400934 learned_addr learned my addr 172.21.15.60:0/3264400934 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:28.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.231+0000 7f9392520700 1 --2- 172.21.15.60:0/3264400934 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f938c14af80 0x7f938c155760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:28.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.231+0000 7f9391d1f700 1 --2- 172.21.15.60:0/3264400934 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f938c05d070 0x7f938c0651c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:28.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.231+0000 7f939151e700 1 -- 172.21.15.60:0/3264400934 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f938c14af80 msgr2=0x7f938c155760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:28.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.231+0000 7f939151e700 1 --2- 172.21.15.60:0/3264400934 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f938c14af80 0x7f938c155760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:28.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.231+0000 7f939151e700 1 -- 172.21.15.60:0/3264400934 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f938c05d070 msgr2=0x7f938c0651c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:28.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.231+0000 7f939151e700 1 --2- 172.21.15.60:0/3264400934 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f938c05d070 0x7f938c0651c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:28.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.231+0000 7f939151e700 1 -- 172.21.15.60:0/3264400934 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f938000b180 con 0x7f938c0b2130 2022-06-29T16:03:28.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.231+0000 7f9391d1f700 1 --2- 172.21.15.60:0/3264400934 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f938c05d070 0x7f938c0651c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:03:28.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.235+0000 7f939151e700 1 --2- 172.21.15.60:0/3264400934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f938c0b2130 0x7f938c0b42f0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7f9388001490 tx=0x7f9388007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:28.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.235+0000 7f937effd700 1 -- 172.21.15.60:0/3264400934 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9388002310 con 0x7f938c0b2130 2022-06-29T16:03:28.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.235+0000 7f937effd700 1 -- 172.21.15.60:0/3264400934 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9388002470 con 0x7f938c0b2130 2022-06-29T16:03:28.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.235+0000 7f9392d21700 1 -- 172.21.15.60:0/3264400934 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f938c0b47e0 con 0x7f938c0b2130 2022-06-29T16:03:28.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.235+0000 7f937effd700 1 -- 172.21.15.60:0/3264400934 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f93880025e0 con 0x7f938c0b2130 2022-06-29T16:03:28.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.235+0000 7f9392d21700 1 -- 172.21.15.60:0/3264400934 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f938c1598c0 con 0x7f938c0b2130 2022-06-29T16:03:28.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.235+0000 7f9392d21700 1 -- 172.21.15.60:0/3264400934 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f938c14a350 con 0x7f938c0b2130 2022-06-29T16:03:28.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.235+0000 7f937effd700 1 -- 172.21.15.60:0/3264400934 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9388010030 con 0x7f938c0b2130 2022-06-29T16:03:28.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.235+0000 7f937effd700 1 --2- 172.21.15.60:0/3264400934 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f93740539e0 0x7f9374055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:28.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.235+0000 7f937effd700 1 -- 172.21.15.60:0/3264400934 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f938806bf50 con 0x7f938c0b2130 2022-06-29T16:03:28.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.235+0000 7f9391d1f700 1 --2- 172.21.15.60:0/3264400934 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f93740539e0 0x7f9374055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:28.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.239+0000 7f9391d1f700 1 --2- 172.21.15.60:0/3264400934 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f93740539e0 0x7f9374055ea0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f9380015000 tx=0x7f9380014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:28.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.239+0000 7f937effd700 1 -- 172.21.15.60:0/3264400934 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9388042e70 con 0x7f938c0b2130 2022-06-29T16:03:28.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.367+0000 7f9392d21700 1 -- 172.21.15.60:0/3264400934 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f938c066e80 con 0x7f938c0b2130 2022-06-29T16:03:28.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.367+0000 7f937effd700 1 -- 172.21.15.60:0/3264400934 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) v1 ==== 74+0+16054 (secure 0 0 0) 0x7f9388046490 con 0x7f938c0b2130 2022-06-29T16:03:28.376 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:03:28.376 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":61,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:03:28.112197+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:03:26.621123+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2022-06-29T16:03:27.220279+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:03:28.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.371+0000 7f9392d21700 1 -- 172.21.15.60:0/3264400934 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f93740539e0 msgr2=0x7f9374055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:28.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.371+0000 7f9392d21700 1 --2- 172.21.15.60:0/3264400934 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f93740539e0 0x7f9374055ea0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f9380015000 tx=0x7f9380014000).stop 2022-06-29T16:03:28.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.371+0000 7f9392d21700 1 -- 172.21.15.60:0/3264400934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f938c0b2130 msgr2=0x7f938c0b42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:28.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.371+0000 7f9392d21700 1 --2- 172.21.15.60:0/3264400934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f938c0b2130 0x7f938c0b42f0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7f9388001490 tx=0x7f9388007e10).stop 2022-06-29T16:03:28.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.371+0000 7f9392d21700 1 -- 172.21.15.60:0/3264400934 shutdown_connections 2022-06-29T16:03:28.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.371+0000 7f9392d21700 1 --2- 172.21.15.60:0/3264400934 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f93740539e0 0x7f9374055ea0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:28.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.371+0000 7f9392d21700 1 --2- 172.21.15.60:0/3264400934 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f938c14af80 0x7f938c155760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:28.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.371+0000 7f9392d21700 1 --2- 172.21.15.60:0/3264400934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f938c0b2130 0x7f938c0b42f0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:28.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.371+0000 7f9392d21700 1 --2- 172.21.15.60:0/3264400934 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f938c05d070 0x7f938c0651c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:28.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.371+0000 7f9392d21700 1 -- 172.21.15.60:0/3264400934 >> 172.21.15.60:0/3264400934 conn(0x7f938c0b5890 msgr2=0x7f938c0b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:28.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.371+0000 7f9392d21700 1 -- 172.21.15.60:0/3264400934 shutdown_connections 2022-06-29T16:03:28.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.371+0000 7f9392d21700 1 -- 172.21.15.60:0/3264400934 wait complete. 2022-06-29T16:03:28.389 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:03:28.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.483+0000 7f4319b6e700 1 Processor -- start 2022-06-29T16:03:28.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.483+0000 7f4319b6e700 1 -- start start 2022-06-29T16:03:28.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.483+0000 7f4319b6e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f431405d070 0x7f431412b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:28.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.483+0000 7f4319b6e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f431412bbc0 0x7f43141431f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:28.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.483+0000 7f4319b6e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43140ae9a0 con 0x7f4314066b20 2022-06-29T16:03:28.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.483+0000 7f4319b6e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4314143b60 con 0x7f431405d070 2022-06-29T16:03:28.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.483+0000 7f4319b6e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43140aced0 con 0x7f431412bbc0 2022-06-29T16:03:28.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.483+0000 7f4318b6c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f431405d070 0x7f431412b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:28.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.483+0000 7f4318b6c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f431405d070 0x7f431412b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48584/0 (socket says 172.21.15.60:48584) 2022-06-29T16:03:28.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.483+0000 7f4318b6c700 1 -- 172.21.15.60:0/1289711938 learned_addr learned my addr 172.21.15.60:0/1289711938 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:28.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.483+0000 7f431936d700 1 --2- 172.21.15.60:0/1289711938 >> v2:172.21.15.92:3300/0 conn(0x7f431412bbc0 0x7f43141431f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:28.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.487+0000 7f43137fe700 1 -- 172.21.15.60:0/1289711938 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4164503217 0 0) 0x7f43140ae9a0 con 0x7f4314066b20 2022-06-29T16:03:28.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.487+0000 7f43137fe700 1 -- 172.21.15.60:0/1289711938 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42f4003260 con 0x7f4314066b20 2022-06-29T16:03:28.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.487+0000 7f4318b6c700 1 -- 172.21.15.60:0/1289711938 >> v2:172.21.15.92:3300/0 conn(0x7f431412bbc0 msgr2=0x7f43141431f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:28.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.487+0000 7f4318b6c700 1 --2- 172.21.15.60:0/1289711938 >> v2:172.21.15.92:3300/0 conn(0x7f431412bbc0 0x7f43141431f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:28.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.487+0000 7f4318b6c700 1 -- 172.21.15.60:0/1289711938 >> v1:172.21.15.92:6789/0 conn(0x7f4314066b20 legacy=0x7f4314142ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:28.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.487+0000 7f4318b6c700 1 -- 172.21.15.60:0/1289711938 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431412c050 con 0x7f431405d070 2022-06-29T16:03:28.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.487+0000 7f4318b6c700 1 --2- 172.21.15.60:0/1289711938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f431405d070 0x7f431412b6d0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7f4304002f70 tx=0x7f4304006eb0).ready entity=mon.0 client_cookie=1b1448d2b521636c server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:28.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.487+0000 7f43137fe700 1 -- 172.21.15.60:0/1289711938 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f430400d610 con 0x7f431405d070 2022-06-29T16:03:28.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.487+0000 7f43137fe700 1 -- 172.21.15.60:0/1289711938 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4304000e90 con 0x7f431405d070 2022-06-29T16:03:28.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.487+0000 7f43137fe700 1 -- 172.21.15.60:0/1289711938 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f430400d9a0 con 0x7f431405d070 2022-06-29T16:03:28.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.487+0000 7f4319b6e700 1 -- 172.21.15.60:0/1289711938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f431405d070 msgr2=0x7f431412b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:28.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.487+0000 7f4319b6e700 1 --2- 172.21.15.60:0/1289711938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f431405d070 0x7f431412b6d0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7f4304002f70 tx=0x7f4304006eb0).stop 2022-06-29T16:03:28.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.487+0000 7f4319b6e700 1 -- 172.21.15.60:0/1289711938 shutdown_connections 2022-06-29T16:03:28.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.487+0000 7f4319b6e700 1 --2- 172.21.15.60:0/1289711938 >> v2:172.21.15.92:3300/0 conn(0x7f431412bbc0 0x7f43141431f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:28.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.487+0000 7f4319b6e700 1 --2- 172.21.15.60:0/1289711938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f431405d070 0x7f431412b6d0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:28.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.487+0000 7f4319b6e700 1 -- 172.21.15.60:0/1289711938 >> 172.21.15.60:0/1289711938 conn(0x7f43140b5890 msgr2=0x7f43140b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:28.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.487+0000 7f4319b6e700 1 -- 172.21.15.60:0/1289711938 shutdown_connections 2022-06-29T16:03:28.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.487+0000 7f4319b6e700 1 -- 172.21.15.60:0/1289711938 wait complete. 2022-06-29T16:03:28.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.487+0000 7f4319b6e700 1 Processor -- start 2022-06-29T16:03:28.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.487+0000 7f4319b6e700 1 -- start start 2022-06-29T16:03:28.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.491+0000 7f4319b6e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4314066b20 0x7f4314145080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:28.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.491+0000 7f4319b6e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f431412bbc0 0x7f4314155470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:28.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.491+0000 7f4319b6e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4314142240 0x7f4314155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:28.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.491+0000 7f4319b6e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431412d6f0 con 0x7f4314066b20 2022-06-29T16:03:28.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.491+0000 7f4319b6e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4314131d00 con 0x7f4314142240 2022-06-29T16:03:28.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.491+0000 7f4319b6e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f431412e4b0 con 0x7f431412bbc0 2022-06-29T16:03:28.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.491+0000 7f4318b6c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4314066b20 0x7f4314145080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:28.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.491+0000 7f431936d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4314142240 0x7f4314155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:28.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.491+0000 7f4313fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f431412bbc0 0x7f4314155470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:28.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.491+0000 7f431936d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4314142240 0x7f4314155960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:54622/0 (socket says 172.21.15.60:54622) 2022-06-29T16:03:28.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.491+0000 7f4318b6c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4314066b20 0x7f4314145080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48588/0 (socket says 172.21.15.60:48588) 2022-06-29T16:03:28.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.491+0000 7f431936d700 1 -- 172.21.15.60:0/1922464947 learned_addr learned my addr 172.21.15.60:0/1922464947 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:28.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.491+0000 7f431936d700 1 -- 172.21.15.60:0/1922464947 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f431412bbc0 msgr2=0x7f4314155470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:28.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.491+0000 7f431936d700 1 --2- 172.21.15.60:0/1922464947 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f431412bbc0 0x7f4314155470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:28.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.491+0000 7f431936d700 1 -- 172.21.15.60:0/1922464947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4314066b20 msgr2=0x7f4314145080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:28.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.491+0000 7f431936d700 1 --2- 172.21.15.60:0/1922464947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4314066b20 0x7f4314145080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:28.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.491+0000 7f431936d700 1 -- 172.21.15.60:0/1922464947 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f430400b1e0 con 0x7f4314142240 2022-06-29T16:03:28.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.491+0000 7f4318b6c700 1 --2- 172.21.15.60:0/1922464947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4314066b20 0x7f4314145080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:03:28.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.491+0000 7f4313fff700 1 --2- 172.21.15.60:0/1922464947 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f431412bbc0 0x7f4314155470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:03:28.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.491+0000 7f431936d700 1 --2- 172.21.15.60:0/1922464947 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4314142240 0x7f4314155960 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f430c006e00 tx=0x7f430c0091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:28.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.491+0000 7f4311ffb700 1 -- 172.21.15.60:0/1922464947 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f430c012600 con 0x7f4314142240 2022-06-29T16:03:28.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.491+0000 7f4311ffb700 1 -- 172.21.15.60:0/1922464947 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f430c012760 con 0x7f4314142240 2022-06-29T16:03:28.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.491+0000 7f4319b6e700 1 -- 172.21.15.60:0/1922464947 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f431412e0b0 con 0x7f4314142240 2022-06-29T16:03:28.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.491+0000 7f4311ffb700 1 -- 172.21.15.60:0/1922464947 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f430c012ac0 con 0x7f4314142240 2022-06-29T16:03:28.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.491+0000 7f4319b6e700 1 -- 172.21.15.60:0/1922464947 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4314155ff0 con 0x7f4314142240 2022-06-29T16:03:28.510 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.491+0000 7f4319b6e700 1 -- 172.21.15.60:0/1922464947 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f431412e730 con 0x7f4314142240 2022-06-29T16:03:28.510 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.495+0000 7f4311ffb700 1 -- 172.21.15.60:0/1922464947 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f430c012c20 con 0x7f4314142240 2022-06-29T16:03:28.510 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.495+0000 7f4311ffb700 1 --2- 172.21.15.60:0/1922464947 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f42f8053aa0 0x7f42f8055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:28.511 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.495+0000 7f4311ffb700 1 -- 172.21.15.60:0/1922464947 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f430c06bfb0 con 0x7f4314142240 2022-06-29T16:03:28.511 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.495+0000 7f4318b6c700 1 --2- 172.21.15.60:0/1922464947 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f42f8053aa0 0x7f42f8055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:28.511 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.495+0000 7f4318b6c700 1 --2- 172.21.15.60:0/1922464947 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f42f8053aa0 0x7f42f8055f60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f430401c000 tx=0x7f4304015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:28.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.503+0000 7f4311ffb700 1 -- 172.21.15.60:0/1922464947 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f430c042e30 con 0x7f4314142240 2022-06-29T16:03:28.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.627+0000 7f4319b6e700 1 -- 172.21.15.60:0/1922464947 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4314155ff0 con 0x7f4314142240 2022-06-29T16:03:28.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.631+0000 7f4311ffb700 1 -- 172.21.15.60:0/1922464947 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) v1 ==== 74+0+16054 (secure 0 0 0) 0x7f4314155ff0 con 0x7f4314142240 2022-06-29T16:03:28.636 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:03:28.637 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":61,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:03:28.112197+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:03:26.621123+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2022-06-29T16:03:27.220279+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:03:28.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.631+0000 7f4319b6e700 1 -- 172.21.15.60:0/1922464947 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f42f8053aa0 msgr2=0x7f42f8055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:28.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.631+0000 7f4319b6e700 1 --2- 172.21.15.60:0/1922464947 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f42f8053aa0 0x7f42f8055f60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f430401c000 tx=0x7f4304015000).stop 2022-06-29T16:03:28.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.631+0000 7f4319b6e700 1 -- 172.21.15.60:0/1922464947 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4314142240 msgr2=0x7f4314155960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:28.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.631+0000 7f4319b6e700 1 --2- 172.21.15.60:0/1922464947 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4314142240 0x7f4314155960 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f430c006e00 tx=0x7f430c0091d0).stop 2022-06-29T16:03:28.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.631+0000 7f4319b6e700 1 -- 172.21.15.60:0/1922464947 shutdown_connections 2022-06-29T16:03:28.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.631+0000 7f4319b6e700 1 --2- 172.21.15.60:0/1922464947 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f42f8053aa0 0x7f42f8055f60 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:28.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.631+0000 7f4319b6e700 1 --2- 172.21.15.60:0/1922464947 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4314142240 0x7f4314155960 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:28.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.631+0000 7f4319b6e700 1 --2- 172.21.15.60:0/1922464947 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f431412bbc0 0x7f4314155470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:28.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.631+0000 7f4319b6e700 1 --2- 172.21.15.60:0/1922464947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4314066b20 0x7f4314145080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:28.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.631+0000 7f4319b6e700 1 -- 172.21.15.60:0/1922464947 >> 172.21.15.60:0/1922464947 conn(0x7f43140b5890 msgr2=0x7f43140b4650 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:28.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.631+0000 7f4319b6e700 1 -- 172.21.15.60:0/1922464947 shutdown_connections 2022-06-29T16:03:28.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.631+0000 7f4319b6e700 1 -- 172.21.15.60:0/1922464947 wait complete. 2022-06-29T16:03:28.656 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-06-29T16:03:28.688 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:03:28.706 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:03:28.709 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:03:28.712 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:03:28.714 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:03:28.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.763+0000 7fe3c7fff700 1 Processor -- start 2022-06-29T16:03:28.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.763+0000 7fe3c7fff700 1 -- start start 2022-06-29T16:03:28.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.763+0000 7fe3c7fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe3c80ad520 0x7fe3c80b2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:28.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.763+0000 7fe3c7fff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c80633e0 con 0x7fe3c8140240 2022-06-29T16:03:28.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.763+0000 7fe3c7fff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c80ad140 con 0x7fe3c80acd60 2022-06-29T16:03:28.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.763+0000 7fe3c7fff700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3c80b35d0 con 0x7fe3c80ad520 2022-06-29T16:03:28.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.763+0000 7fe3be7fc700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe3c8140240 0x7fe3c805bd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56684/0 (socket says 172.21.15.60:56684) 2022-06-29T16:03:28.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.763+0000 7fe3be7fc700 1 -- 172.21.15.60:0/2227056440 learned_addr learned my addr 172.21.15.60:0/2227056440 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:03:28.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.763+0000 7fe3c77fe700 1 --2- 172.21.15.60:0/2227056440 >> v2:172.21.15.92:3300/0 conn(0x7fe3c80ad520 0x7fe3c80b2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:28.777 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.763+0000 7fe3c67fc700 1 -- 172.21.15.60:0/2227056440 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3372019102 0 0) 0x7fe3c80633e0 con 0x7fe3c8140240 2022-06-29T16:03:28.777 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.763+0000 7fe3c67fc700 1 -- 172.21.15.60:0/2227056440 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3ac003260 con 0x7fe3c8140240 2022-06-29T16:03:28.777 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.763+0000 7fe3c67fc700 1 -- 172.21.15.60:0/2227056440 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4166015021 0 0) 0x7fe3c80ad140 con 0x7fe3c80acd60 2022-06-29T16:03:28.778 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.763+0000 7fe3c67fc700 1 -- 172.21.15.60:0/2227056440 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3c80633e0 con 0x7fe3c80acd60 2022-06-29T16:03:28.778 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.763+0000 7fe3c67fc700 1 -- 172.21.15.60:0/2227056440 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 259046344 0 0) 0x7fe3ac003260 con 0x7fe3c8140240 2022-06-29T16:03:28.778 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.763+0000 7fe3c67fc700 1 -- 172.21.15.60:0/2227056440 >> v2:172.21.15.92:3300/0 conn(0x7fe3c80ad520 msgr2=0x7fe3c80b2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:28.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.763+0000 7fe3c67fc700 1 --2- 172.21.15.60:0/2227056440 >> v2:172.21.15.92:3300/0 conn(0x7fe3c80ad520 0x7fe3c80b2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:28.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.763+0000 7fe3c67fc700 1 -- 172.21.15.60:0/2227056440 >> v1:172.21.15.92:6789/0 conn(0x7fe3c80acd60 legacy=0x7fe3c813fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:28.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.763+0000 7fe3c67fc700 1 -- 172.21.15.60:0/2227056440 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c8146030 con 0x7fe3c8140240 2022-06-29T16:03:28.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.763+0000 7fe3c67fc700 1 -- 172.21.15.60:0/2227056440 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7fe3b0001660 con 0x7fe3c8140240 2022-06-29T16:03:28.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.763+0000 7fe3c67fc700 1 -- 172.21.15.60:0/2227056440 >> v1:172.21.15.60:6789/0 conn(0x7fe3c8140240 legacy=0x7fe3c805bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:28.781 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.763+0000 7fe3c67fc700 1 --2- 172.21.15.60:0/2227056440 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3ac0038e0 0x7fe3ac003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:28.781 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.763+0000 7fe3c67fc700 1 --2- 172.21.15.60:0/2227056440 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3ac004500 0x7fe3ac004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:28.781 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.763+0000 7fe3c67fc700 1 --2- 172.21.15.60:0/2227056440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3ac004e70 0x7fe3ac00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:28.782 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.763+0000 7fe3c67fc700 1 -- 172.21.15.60:0/2227056440 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c80b35d0 con 0x7fe3ac004e70 2022-06-29T16:03:28.782 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.763+0000 7fe3c67fc700 1 -- 172.21.15.60:0/2227056440 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ac00bdd0 con 0x7fe3ac004500 2022-06-29T16:03:28.782 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.763+0000 7fe3c67fc700 1 -- 172.21.15.60:0/2227056440 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ac00bf40 con 0x7fe3ac0038e0 2022-06-29T16:03:28.783 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.763+0000 7fe3be7fc700 1 --2- 172.21.15.60:0/2227056440 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3ac004500 0x7fe3ac004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:28.783 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.763+0000 7fe3c6ffd700 1 --2- 172.21.15.60:0/2227056440 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3ac0038e0 0x7fe3ac003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:28.784 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.763+0000 7fe3c77fe700 1 --2- 172.21.15.60:0/2227056440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3ac004e70 0x7fe3ac00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:28.784 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.767+0000 7fe3c6ffd700 1 -- 172.21.15.60:0/2227056440 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3ac004500 msgr2=0x7fe3ac004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:28.784 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.767+0000 7fe3c6ffd700 1 --2- 172.21.15.60:0/2227056440 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3ac004500 0x7fe3ac004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:28.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.767+0000 7fe3c6ffd700 1 -- 172.21.15.60:0/2227056440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3ac004e70 msgr2=0x7fe3ac00b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:28.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.767+0000 7fe3c6ffd700 1 --2- 172.21.15.60:0/2227056440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3ac004e70 0x7fe3ac00b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:28.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.767+0000 7fe3c6ffd700 1 -- 172.21.15.60:0/2227056440 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3ac003260 con 0x7fe3ac0038e0 2022-06-29T16:03:28.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.767+0000 7fe3c77fe700 1 -- 172.21.15.60:0/2227056440 reap_dead start 2022-06-29T16:03:28.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.767+0000 7fe3be7fc700 1 --2- 172.21.15.60:0/2227056440 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3ac004500 0x7fe3ac004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:03:28.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.767+0000 7fe3c6ffd700 1 --2- 172.21.15.60:0/2227056440 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3ac0038e0 0x7fe3ac003ce0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fe3b8001110 tx=0x7fe3b8007a00).ready entity=mon.2 client_cookie=18c68113bd6ee6d9 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:28.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.767+0000 7fe3c67fc700 1 -- 172.21.15.60:0/2227056440 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3b8007d10 con 0x7fe3ac0038e0 2022-06-29T16:03:28.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.767+0000 7fe3c67fc700 1 -- 172.21.15.60:0/2227056440 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b8007030 con 0x7fe3ac0038e0 2022-06-29T16:03:28.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.767+0000 7fe3c67fc700 1 -- 172.21.15.60:0/2227056440 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3b8007390 con 0x7fe3ac0038e0 2022-06-29T16:03:28.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.767+0000 7fe3c7fff700 1 -- 172.21.15.60:0/2227056440 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3ac0038e0 msgr2=0x7fe3ac003ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:28.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.767+0000 7fe3c7fff700 1 --2- 172.21.15.60:0/2227056440 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3ac0038e0 0x7fe3ac003ce0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fe3b8001110 tx=0x7fe3b8007a00).stop 2022-06-29T16:03:28.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.767+0000 7fe3c7fff700 1 -- 172.21.15.60:0/2227056440 shutdown_connections 2022-06-29T16:03:28.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.767+0000 7fe3c7fff700 1 --2- 172.21.15.60:0/2227056440 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3ac0038e0 0x7fe3ac003ce0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:28.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.767+0000 7fe3c7fff700 1 -- 172.21.15.60:0/2227056440 >> 172.21.15.60:0/2227056440 conn(0x7fe3c80b6fb0 msgr2=0x7fe3c80b7390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:28.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.767+0000 7fe3c7fff700 1 -- 172.21.15.60:0/2227056440 shutdown_connections 2022-06-29T16:03:28.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.767+0000 7fe3c7fff700 1 -- 172.21.15.60:0/2227056440 wait complete. 2022-06-29T16:03:28.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.767+0000 7fe3c7fff700 1 Processor -- start 2022-06-29T16:03:28.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.767+0000 7fe3c7fff700 1 -- start start 2022-06-29T16:03:28.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.767+0000 7fe3c7fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3ac0038e0 0x7fe3c81424f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:28.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.767+0000 7fe3c7fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3c81409f0 0x7fe3c813b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:28.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.767+0000 7fe3c7fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3c813bd10 0x7fe3c8155100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:28.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.767+0000 7fe3c7fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c813c190 con 0x7fe3c81409f0 2022-06-29T16:03:28.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.767+0000 7fe3c7fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c80650f0 con 0x7fe3ac0038e0 2022-06-29T16:03:28.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.767+0000 7fe3c7fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c8148260 con 0x7fe3c813bd10 2022-06-29T16:03:28.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.771+0000 7fe3be7fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3c81409f0 0x7fe3c813b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:28.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.771+0000 7fe3be7fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3c81409f0 0x7fe3c813b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48610/0 (socket says 172.21.15.60:48610) 2022-06-29T16:03:28.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.771+0000 7fe3be7fc700 1 -- 172.21.15.60:0/674844964 learned_addr learned my addr 172.21.15.60:0/674844964 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:28.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.771+0000 7fe3c6ffd700 1 --2- 172.21.15.60:0/674844964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3ac0038e0 0x7fe3c81424f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:28.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.771+0000 7fe3c77fe700 1 --2- 172.21.15.60:0/674844964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3c813bd10 0x7fe3c8155100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:28.797 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.771+0000 7fe3be7fc700 1 -- 172.21.15.60:0/674844964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3c813bd10 msgr2=0x7fe3c8155100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:28.797 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.771+0000 7fe3be7fc700 1 --2- 172.21.15.60:0/674844964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3c813bd10 0x7fe3c8155100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:28.797 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.771+0000 7fe3be7fc700 1 -- 172.21.15.60:0/674844964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3ac0038e0 msgr2=0x7fe3c81424f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:28.797 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.771+0000 7fe3be7fc700 1 --2- 172.21.15.60:0/674844964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3ac0038e0 0x7fe3c81424f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:28.798 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.771+0000 7fe3be7fc700 1 -- 172.21.15.60:0/674844964 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c8148500 con 0x7fe3c81409f0 2022-06-29T16:03:28.798 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.771+0000 7fe3c77fe700 1 --2- 172.21.15.60:0/674844964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3c813bd10 0x7fe3c8155100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:03:28.798 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.771+0000 7fe3be7fc700 1 --2- 172.21.15.60:0/674844964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3c81409f0 0x7fe3c813b820 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7fe3b000d000 tx=0x7fe3c8061e90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:28.798 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.771+0000 7fe3c4ff9700 1 -- 172.21.15.60:0/674844964 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3b0011780 con 0x7fe3c81409f0 2022-06-29T16:03:28.799 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.771+0000 7fe3c4ff9700 1 -- 172.21.15.60:0/674844964 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b00118e0 con 0x7fe3c81409f0 2022-06-29T16:03:28.799 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.771+0000 7fe3c4ff9700 1 -- 172.21.15.60:0/674844964 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3b0011c40 con 0x7fe3c81409f0 2022-06-29T16:03:28.799 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.771+0000 7fe3c7fff700 1 -- 172.21.15.60:0/674844964 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3c81487a0 con 0x7fe3c81409f0 2022-06-29T16:03:28.799 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.771+0000 7fe3c7fff700 1 -- 172.21.15.60:0/674844964 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3c8148ee0 con 0x7fe3c81409f0 2022-06-29T16:03:28.800 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.771+0000 7fe3c7fff700 1 -- 172.21.15.60:0/674844964 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3c80573d0 con 0x7fe3c81409f0 2022-06-29T16:03:28.800 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.771+0000 7fe3c4ff9700 1 -- 172.21.15.60:0/674844964 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe3b001b030 con 0x7fe3c81409f0 2022-06-29T16:03:28.800 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.771+0000 7fe3c4ff9700 1 --2- 172.21.15.60:0/674844964 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe3a00539e0 0x7fe3a0055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:28.800 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.771+0000 7fe3c4ff9700 1 -- 172.21.15.60:0/674844964 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe3b006bbc0 con 0x7fe3c81409f0 2022-06-29T16:03:28.800 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.775+0000 7fe3c6ffd700 1 --2- 172.21.15.60:0/674844964 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe3a00539e0 0x7fe3a0055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:28.801 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.775+0000 7fe3c6ffd700 1 --2- 172.21.15.60:0/674844964 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe3a00539e0 0x7fe3a0055ea0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fe3b801a000 tx=0x7fe3b8015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:28.801 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.779+0000 7fe3c4ff9700 1 -- 172.21.15.60:0/674844964 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe3b0042a40 con 0x7fe3c81409f0 2022-06-29T16:03:28.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:28.939+0000 7fe3c7fff700 1 -- 172.21.15.60:0/674844964 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe3c8067b20 con 0x7fe3c81409f0 2022-06-29T16:03:29.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.119+0000 7fe3c4ff9700 1 -- 172.21.15.60:0/674844964 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v25) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe3b0019030 con 0x7fe3c81409f0 2022-06-29T16:03:29.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.119+0000 7fe3c7fff700 1 -- 172.21.15.60:0/674844964 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe3a00539e0 msgr2=0x7fe3a0055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:29.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.119+0000 7fe3c7fff700 1 --2- 172.21.15.60:0/674844964 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe3a00539e0 0x7fe3a0055ea0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fe3b801a000 tx=0x7fe3b8015000).stop 2022-06-29T16:03:29.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.119+0000 7fe3c7fff700 1 -- 172.21.15.60:0/674844964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3c81409f0 msgr2=0x7fe3c813b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:29.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.119+0000 7fe3c7fff700 1 --2- 172.21.15.60:0/674844964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3c81409f0 0x7fe3c813b820 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7fe3b000d000 tx=0x7fe3c8061e90).stop 2022-06-29T16:03:29.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.123+0000 7fe3c7fff700 1 -- 172.21.15.60:0/674844964 shutdown_connections 2022-06-29T16:03:29.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.123+0000 7fe3c7fff700 1 --2- 172.21.15.60:0/674844964 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe3a00539e0 0x7fe3a0055ea0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:29.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.123+0000 7fe3c7fff700 1 --2- 172.21.15.60:0/674844964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3c813bd10 0x7fe3c8155100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:29.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.123+0000 7fe3c7fff700 1 --2- 172.21.15.60:0/674844964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3c81409f0 0x7fe3c813b820 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:29.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.123+0000 7fe3c7fff700 1 --2- 172.21.15.60:0/674844964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3ac0038e0 0x7fe3c81424f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:29.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.123+0000 7fe3c7fff700 1 -- 172.21.15.60:0/674844964 >> 172.21.15.60:0/674844964 conn(0x7fe3c80b6fb0 msgr2=0x7fe3c80b2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:29.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.123+0000 7fe3c7fff700 1 -- 172.21.15.60:0/674844964 shutdown_connections 2022-06-29T16:03:29.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.123+0000 7fe3c7fff700 1 -- 172.21.15.60:0/674844964 wait complete. 2022-06-29T16:03:29.146 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:03:29.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.247+0000 7f13bb59e700 1 Processor -- start 2022-06-29T16:03:29.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.247+0000 7f13bb59e700 1 -- start start 2022-06-29T16:03:29.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.247+0000 7f13bb59e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f13bc0b5eb0 0x7f13bc142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:29.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.247+0000 7f13bb59e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13bc0b0cf0 con 0x7f13bc0b48b0 2022-06-29T16:03:29.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.247+0000 7f13bb59e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13bc0b4d10 con 0x7f13bc0b2c50 2022-06-29T16:03:29.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.247+0000 7f13bb59e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13bc0b54e0 con 0x7f13bc0b5eb0 2022-06-29T16:03:29.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.247+0000 7f13b9d9b700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f13bc0b48b0 0x7f13bc0b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56702/0 (socket says 172.21.15.60:56702) 2022-06-29T16:03:29.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.247+0000 7f13b9d9b700 1 -- 172.21.15.60:0/20886069 learned_addr learned my addr 172.21.15.60:0/20886069 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:03:29.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.247+0000 7f13ba59c700 1 --2- 172.21.15.60:0/20886069 >> v2:172.21.15.92:3300/0 conn(0x7f13bc0b5eb0 0x7f13bc142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:29.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.247+0000 7f13b959a700 1 -- 172.21.15.60:0/20886069 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3445212531 0 0) 0x7f13b4001490 con 0x7f13bc0b48b0 2022-06-29T16:03:29.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.247+0000 7f13b959a700 1 -- 172.21.15.60:0/20886069 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f139c003260 con 0x7f13bc0b48b0 2022-06-29T16:03:29.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.247+0000 7f13b959a700 1 -- 172.21.15.60:0/20886069 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1255614066 0 0) 0x7f13bc0b4d10 con 0x7f13bc0b2c50 2022-06-29T16:03:29.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.247+0000 7f13b959a700 1 -- 172.21.15.60:0/20886069 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13b4001490 con 0x7f13bc0b2c50 2022-06-29T16:03:29.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.247+0000 7f13b959a700 1 -- 172.21.15.60:0/20886069 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1182841049 0 0) 0x7f139c003260 con 0x7f13bc0b48b0 2022-06-29T16:03:29.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.247+0000 7f13b959a700 1 -- 172.21.15.60:0/20886069 >> v2:172.21.15.92:3300/0 conn(0x7f13bc0b5eb0 msgr2=0x7f13bc142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:29.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.247+0000 7f13b959a700 1 --2- 172.21.15.60:0/20886069 >> v2:172.21.15.92:3300/0 conn(0x7f13bc0b5eb0 0x7f13bc142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:29.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.247+0000 7f13b959a700 1 -- 172.21.15.60:0/20886069 >> v1:172.21.15.92:6789/0 conn(0x7f13bc0b2c50 legacy=0x7f13bc14d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:29.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.247+0000 7f13b959a700 1 -- 172.21.15.60:0/20886069 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13bc0b6290 con 0x7f13bc0b48b0 2022-06-29T16:03:29.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.247+0000 7f13b959a700 1 -- 172.21.15.60:0/20886069 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f13b4001b70 con 0x7f13bc0b48b0 2022-06-29T16:03:29.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.247+0000 7f13ba59c700 1 --2- 172.21.15.60:0/20886069 >> v2:172.21.15.92:3300/0 conn(0x7f13bc0b5eb0 0x7f13bc142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:03:29.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.247+0000 7f13b959a700 1 -- 172.21.15.60:0/20886069 >> v1:172.21.15.60:6789/0 conn(0x7f13bc0b48b0 legacy=0x7f13bc0b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:29.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.247+0000 7f13b959a700 1 --2- 172.21.15.60:0/20886069 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f139c003930 0x7f139c003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:29.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.247+0000 7f13b959a700 1 --2- 172.21.15.60:0/20886069 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f139c004500 0x7f139c004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:29.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.247+0000 7f13b959a700 1 --2- 172.21.15.60:0/20886069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f139c004e70 0x7f139c00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:29.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.247+0000 7f13b959a700 1 -- 172.21.15.60:0/20886069 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc0b54e0 con 0x7f139c004e70 2022-06-29T16:03:29.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.247+0000 7f13b959a700 1 -- 172.21.15.60:0/20886069 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139c00bdd0 con 0x7f139c004500 2022-06-29T16:03:29.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.247+0000 7f13b959a700 1 -- 172.21.15.60:0/20886069 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f139c00bf90 con 0x7f139c003930 2022-06-29T16:03:29.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.247+0000 7f13bad9d700 1 --2- 172.21.15.60:0/20886069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f139c004e70 0x7f139c00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:29.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.247+0000 7f13ba59c700 1 --2- 172.21.15.60:0/20886069 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f139c003930 0x7f139c003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:29.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.247+0000 7f13b9d9b700 1 --2- 172.21.15.60:0/20886069 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f139c004500 0x7f139c004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:29.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.251+0000 7f13ba59c700 1 -- 172.21.15.60:0/20886069 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f139c004500 msgr2=0x7f139c004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:29.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.251+0000 7f13ba59c700 1 --2- 172.21.15.60:0/20886069 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f139c004500 0x7f139c004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:29.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.251+0000 7f13ba59c700 1 -- 172.21.15.60:0/20886069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f139c004e70 msgr2=0x7f139c00b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:29.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.251+0000 7f13ba59c700 1 --2- 172.21.15.60:0/20886069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f139c004e70 0x7f139c00b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:29.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.251+0000 7f13ba59c700 1 -- 172.21.15.60:0/20886069 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f139c003260 con 0x7f139c003930 2022-06-29T16:03:29.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.251+0000 7f13bad9d700 1 -- 172.21.15.60:0/20886069 reap_dead start 2022-06-29T16:03:29.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.251+0000 7f13ba59c700 1 --2- 172.21.15.60:0/20886069 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f139c003930 0x7f139c003d30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f139c0054d0 tx=0x7f13a4006df0).ready entity=mon.2 client_cookie=e4ac4cb345118170 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:29.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.251+0000 7f13b959a700 1 -- 172.21.15.60:0/20886069 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13a400e800 con 0x7f139c003930 2022-06-29T16:03:29.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.251+0000 7f13b959a700 1 -- 172.21.15.60:0/20886069 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a400e960 con 0x7f139c003930 2022-06-29T16:03:29.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.251+0000 7f13b959a700 1 -- 172.21.15.60:0/20886069 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13a400ecc0 con 0x7f139c003930 2022-06-29T16:03:29.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.251+0000 7f13bb59e700 1 -- 172.21.15.60:0/20886069 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f139c003930 msgr2=0x7f139c003d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:29.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.251+0000 7f13bb59e700 1 --2- 172.21.15.60:0/20886069 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f139c003930 0x7f139c003d30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f139c0054d0 tx=0x7f13a4006df0).stop 2022-06-29T16:03:29.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.251+0000 7f13bb59e700 1 -- 172.21.15.60:0/20886069 shutdown_connections 2022-06-29T16:03:29.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.251+0000 7f13bb59e700 1 --2- 172.21.15.60:0/20886069 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f139c003930 0x7f139c003d30 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:29.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.251+0000 7f13bb59e700 1 -- 172.21.15.60:0/20886069 >> 172.21.15.60:0/20886069 conn(0x7f13bc062090 msgr2=0x7f13bc062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:29.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.251+0000 7f13bb59e700 1 -- 172.21.15.60:0/20886069 shutdown_connections 2022-06-29T16:03:29.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.251+0000 7f13bb59e700 1 -- 172.21.15.60:0/20886069 wait complete. 2022-06-29T16:03:29.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.251+0000 7f13bb59e700 1 Processor -- start 2022-06-29T16:03:29.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.251+0000 7f13bb59e700 1 -- start start 2022-06-29T16:03:29.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.255+0000 7f13bb59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f139c003930 0x7f13bc151ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:29.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.255+0000 7f13bb59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13bc0aea30 0x7f13bc152560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:29.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.255+0000 7f13bb59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13bc0aeeb0 0x7f13bc145ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:29.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.255+0000 7f13bb59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc0ace60 con 0x7f13bc0aea30 2022-06-29T16:03:29.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.255+0000 7f13bb59e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc0ae0e0 con 0x7f13bc0aeeb0 2022-06-29T16:03:29.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.255+0000 7f13bb59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc0adba0 con 0x7f139c003930 2022-06-29T16:03:29.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.255+0000 7f13b9d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13bc0aea30 0x7f13bc152560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:29.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.255+0000 7f13b9d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13bc0aea30 0x7f13bc152560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48630/0 (socket says 172.21.15.60:48630) 2022-06-29T16:03:29.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.255+0000 7f13ba59c700 1 --2- 172.21.15.60:0/157332193 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f139c003930 0x7f13bc151ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:29.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.255+0000 7f13b9d9b700 1 -- 172.21.15.60:0/157332193 learned_addr learned my addr 172.21.15.60:0/157332193 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:29.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.255+0000 7f13bad9d700 1 --2- 172.21.15.60:0/157332193 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13bc0aeeb0 0x7f13bc145ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:29.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.255+0000 7f13b9d9b700 1 -- 172.21.15.60:0/157332193 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f139c003930 msgr2=0x7f13bc151ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:29.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.255+0000 7f13b9d9b700 1 --2- 172.21.15.60:0/157332193 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f139c003930 0x7f13bc151ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:29.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.255+0000 7f13b9d9b700 1 -- 172.21.15.60:0/157332193 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13bc0aeeb0 msgr2=0x7f13bc145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:29.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.255+0000 7f13b9d9b700 1 --2- 172.21.15.60:0/157332193 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13bc0aeeb0 0x7f13bc145ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:29.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.255+0000 7f13b9d9b700 1 -- 172.21.15.60:0/157332193 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13bc0ade40 con 0x7f13bc0aea30 2022-06-29T16:03:29.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.255+0000 7f13b9d9b700 1 --2- 172.21.15.60:0/157332193 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13bc0aea30 0x7f13bc152560 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 rx=0x7f13bc144f10 tx=0x7f13bc0b6bd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:29.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.255+0000 7f13b37fe700 1 -- 172.21.15.60:0/157332193 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13b4011850 con 0x7f13bc0aea30 2022-06-29T16:03:29.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.255+0000 7f13b37fe700 1 -- 172.21.15.60:0/157332193 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b40119b0 con 0x7f13bc0aea30 2022-06-29T16:03:29.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.255+0000 7f13bb59e700 1 -- 172.21.15.60:0/157332193 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13bc0ae5c0 con 0x7f13bc0aea30 2022-06-29T16:03:29.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.255+0000 7f13b37fe700 1 -- 172.21.15.60:0/157332193 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13b4011d10 con 0x7f13bc0aea30 2022-06-29T16:03:29.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.255+0000 7f13bb59e700 1 -- 172.21.15.60:0/157332193 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13bc146be0 con 0x7f13bc0aea30 2022-06-29T16:03:29.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.259+0000 7f13b37fe700 1 -- 172.21.15.60:0/157332193 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f13b401b030 con 0x7f13bc0aea30 2022-06-29T16:03:29.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.259+0000 7f13bb59e700 1 -- 172.21.15.60:0/157332193 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13bc0ad9c0 con 0x7f13bc0aea30 2022-06-29T16:03:29.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.259+0000 7f13b37fe700 1 --2- 172.21.15.60:0/157332193 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f13a0053a10 0x7f13a0055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:29.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.259+0000 7f13b37fe700 1 -- 172.21.15.60:0/157332193 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f13b406ba60 con 0x7f13bc0aea30 2022-06-29T16:03:29.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.259+0000 7f13ba59c700 1 --2- 172.21.15.60:0/157332193 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f13a0053a10 0x7f13a0055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:29.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.259+0000 7f13ba59c700 1 --2- 172.21.15.60:0/157332193 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f13a0053a10 0x7f13a0055ed0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f13a401d000 tx=0x7f13a4019000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:29.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.263+0000 7f13b37fe700 1 -- 172.21.15.60:0/157332193 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f13b4042920 con 0x7f13bc0aea30 2022-06-29T16:03:29.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.427+0000 7f13bb59e700 1 -- 172.21.15.60:0/157332193 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13bc0b0cf0 con 0x7f13bc0aea30 2022-06-29T16:03:29.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.431+0000 7f13b37fe700 1 -- 172.21.15.60:0/157332193 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f13bc0b0cf0 con 0x7f13bc0aea30 2022-06-29T16:03:29.436 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:03:29.437 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":25,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5955,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.60:6833/66690512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":66690512},{"type":"v1","addr":"172.21.15.60:6833","nonce":66690512}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":23}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:03:28.112667+0000","modified":"2022-06-29T16:03:29.115727+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6013},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6013":{"gid":6013,"name":"b","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/3876875590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3876875590},{"type":"v1","addr":"172.21.15.92:6835","nonce":3876875590}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-06-29T16:03:29.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.431+0000 7f13bb59e700 1 -- 172.21.15.60:0/157332193 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f13a0053a10 msgr2=0x7f13a0055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:29.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.431+0000 7f13bb59e700 1 --2- 172.21.15.60:0/157332193 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f13a0053a10 0x7f13a0055ed0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f13a401d000 tx=0x7f13a4019000).stop 2022-06-29T16:03:29.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.431+0000 7f13bb59e700 1 -- 172.21.15.60:0/157332193 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13bc0aea30 msgr2=0x7f13bc152560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:29.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.431+0000 7f13bb59e700 1 --2- 172.21.15.60:0/157332193 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13bc0aea30 0x7f13bc152560 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 rx=0x7f13bc144f10 tx=0x7f13bc0b6bd0).stop 2022-06-29T16:03:29.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.431+0000 7f13bb59e700 1 -- 172.21.15.60:0/157332193 shutdown_connections 2022-06-29T16:03:29.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.431+0000 7f13bb59e700 1 --2- 172.21.15.60:0/157332193 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f13a0053a10 0x7f13a0055ed0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:29.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.431+0000 7f13bb59e700 1 --2- 172.21.15.60:0/157332193 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13bc0aeeb0 0x7f13bc145ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:29.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.431+0000 7f13bb59e700 1 --2- 172.21.15.60:0/157332193 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13bc0aea30 0x7f13bc152560 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:29.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.431+0000 7f13bb59e700 1 --2- 172.21.15.60:0/157332193 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f139c003930 0x7f13bc151ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:29.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.431+0000 7f13bb59e700 1 -- 172.21.15.60:0/157332193 >> 172.21.15.60:0/157332193 conn(0x7f13bc062090 msgr2=0x7f13bc0b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:29.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.431+0000 7f13bb59e700 1 -- 172.21.15.60:0/157332193 shutdown_connections 2022-06-29T16:03:29.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.431+0000 7f13bb59e700 1 -- 172.21.15.60:0/157332193 wait complete. 2022-06-29T16:03:29.441 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 25 2022-06-29T16:03:29.453 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:03:29.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.547+0000 7f915899d700 1 Processor -- start 2022-06-29T16:03:29.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.551+0000 7f915899d700 1 -- start start 2022-06-29T16:03:29.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.551+0000 7f915899d700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f91540b3d00 0x7f9154144580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:29.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.551+0000 7f915899d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91540b1f60 0x7f91540aca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:29.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.551+0000 7f915899d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f915405c470 0x7f91540acf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:29.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.551+0000 7f915899d700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f915405c280 con 0x7f915405c470 2022-06-29T16:03:29.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.551+0000 7f915899d700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91540b40e0 con 0x7f91540b3d00 2022-06-29T16:03:29.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.551+0000 7f915899d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9154143e40 con 0x7f91540b1f60 2022-06-29T16:03:29.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.551+0000 7f9153fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f915405c470 0x7f91540acf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:29.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.551+0000 7f9153fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f915405c470 0x7f91540acf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48640/0 (socket says 172.21.15.60:48640) 2022-06-29T16:03:29.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.551+0000 7f9153fff700 1 -- 172.21.15.60:0/2756443795 learned_addr learned my addr 172.21.15.60:0/2756443795 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:29.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.551+0000 7f9152ffd700 1 --2- 172.21.15.60:0/2756443795 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91540b1f60 0x7f91540aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:29.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.551+0000 7f91537fe700 1 --2- 172.21.15.60:0/2756443795 >> v2:172.21.15.92:3300/0 conn(0x7f91540b3d00 0x7f9154144580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:29.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.551+0000 7f9153fff700 1 -- 172.21.15.60:0/2756443795 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91540b1f60 msgr2=0x7f91540aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:29.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.551+0000 7f9153fff700 1 --2- 172.21.15.60:0/2756443795 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91540b1f60 0x7f91540aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:29.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.551+0000 7f9153fff700 1 -- 172.21.15.60:0/2756443795 >> v2:172.21.15.92:3300/0 conn(0x7f91540b3d00 msgr2=0x7f9154144580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:29.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.551+0000 7f9153fff700 1 --2- 172.21.15.60:0/2756443795 >> v2:172.21.15.92:3300/0 conn(0x7f91540b3d00 0x7f9154144580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:29.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.551+0000 7f9153fff700 1 -- 172.21.15.60:0/2756443795 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91541440e0 con 0x7f915405c470 2022-06-29T16:03:29.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.551+0000 7f9153fff700 1 --2- 172.21.15.60:0/2756443795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f915405c470 0x7f91540acf10 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7f914c0032f0 tx=0x7f914c009290).ready entity=mon.0 client_cookie=746b49f38ddcba8c server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:29.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.551+0000 7f91527fc700 1 -- 172.21.15.60:0/2756443795 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f914c00b870 con 0x7f915405c470 2022-06-29T16:03:29.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.551+0000 7f91527fc700 1 -- 172.21.15.60:0/2756443795 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f914c00b9d0 con 0x7f915405c470 2022-06-29T16:03:29.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.551+0000 7f91527fc700 1 -- 172.21.15.60:0/2756443795 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f914c00bb40 con 0x7f915405c470 2022-06-29T16:03:29.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.551+0000 7f915899d700 1 -- 172.21.15.60:0/2756443795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f915405c470 msgr2=0x7f91540acf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:29.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.551+0000 7f915899d700 1 --2- 172.21.15.60:0/2756443795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f915405c470 0x7f91540acf10 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7f914c0032f0 tx=0x7f914c009290).stop 2022-06-29T16:03:29.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.551+0000 7f915899d700 1 -- 172.21.15.60:0/2756443795 shutdown_connections 2022-06-29T16:03:29.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.551+0000 7f915899d700 1 --2- 172.21.15.60:0/2756443795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f915405c470 0x7f91540acf10 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:29.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.551+0000 7f915899d700 1 --2- 172.21.15.60:0/2756443795 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91540b1f60 0x7f91540aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:29.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.551+0000 7f915899d700 1 --2- 172.21.15.60:0/2756443795 >> v2:172.21.15.92:3300/0 conn(0x7f91540b3d00 0x7f9154144580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:29.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.551+0000 7f915899d700 1 -- 172.21.15.60:0/2756443795 >> 172.21.15.60:0/2756443795 conn(0x7f9154064810 msgr2=0x7f9154064bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:29.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.551+0000 7f915899d700 1 -- 172.21.15.60:0/2756443795 shutdown_connections 2022-06-29T16:03:29.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.551+0000 7f915899d700 1 -- 172.21.15.60:0/2756443795 wait complete. 2022-06-29T16:03:29.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.555+0000 7f915899d700 1 Processor -- start 2022-06-29T16:03:29.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.555+0000 7f915899d700 1 -- start start 2022-06-29T16:03:29.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.555+0000 7f915899d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f915405c470 0x7f91540b0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:29.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.555+0000 7f915899d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91540b1f60 0x7f9154066cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:29.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.555+0000 7f915899d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91540b3d00 0x7f91540aefa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:29.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.555+0000 7f915899d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915413d760 con 0x7f915405c470 2022-06-29T16:03:29.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.555+0000 7f915899d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9154113e60 con 0x7f91540b1f60 2022-06-29T16:03:29.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.555+0000 7f915899d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f915414fc30 con 0x7f91540b3d00 2022-06-29T16:03:29.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.559+0000 7f91537fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f915405c470 0x7f91540b0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:29.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.559+0000 7f91537fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f915405c470 0x7f91540b0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48642/0 (socket says 172.21.15.60:48642) 2022-06-29T16:03:29.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.559+0000 7f91537fe700 1 -- 172.21.15.60:0/3268076222 learned_addr learned my addr 172.21.15.60:0/3268076222 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:29.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.559+0000 7f9152ffd700 1 --2- 172.21.15.60:0/3268076222 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91540b1f60 0x7f9154066cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:29.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.559+0000 7f9153fff700 1 --2- 172.21.15.60:0/3268076222 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91540b3d00 0x7f91540aefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:29.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.559+0000 7f91537fe700 1 -- 172.21.15.60:0/3268076222 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91540b3d00 msgr2=0x7f91540aefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:29.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.559+0000 7f91537fe700 1 --2- 172.21.15.60:0/3268076222 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91540b3d00 0x7f91540aefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:29.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.559+0000 7f91537fe700 1 -- 172.21.15.60:0/3268076222 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91540b1f60 msgr2=0x7f9154066cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:29.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.559+0000 7f91537fe700 1 --2- 172.21.15.60:0/3268076222 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91540b1f60 0x7f9154066cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:29.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.559+0000 7f91537fe700 1 -- 172.21.15.60:0/3268076222 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f914c009530 con 0x7f915405c470 2022-06-29T16:03:29.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.559+0000 7f91537fe700 1 --2- 172.21.15.60:0/3268076222 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f915405c470 0x7f91540b0cb0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0x7f91440030b0 tx=0x7f9144008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:29.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.559+0000 7f9150ff9700 1 -- 172.21.15.60:0/3268076222 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9144011780 con 0x7f915405c470 2022-06-29T16:03:29.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.559+0000 7f915899d700 1 -- 172.21.15.60:0/3268076222 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f915414fed0 con 0x7f915405c470 2022-06-29T16:03:29.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.559+0000 7f9150ff9700 1 -- 172.21.15.60:0/3268076222 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91440118e0 con 0x7f915405c470 2022-06-29T16:03:29.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.559+0000 7f915899d700 1 -- 172.21.15.60:0/3268076222 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91540af490 con 0x7f915405c470 2022-06-29T16:03:29.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.559+0000 7f9150ff9700 1 -- 172.21.15.60:0/3268076222 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9144011c80 con 0x7f915405c470 2022-06-29T16:03:29.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.559+0000 7f9150ff9700 1 -- 172.21.15.60:0/3268076222 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9144006030 con 0x7f915405c470 2022-06-29T16:03:29.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.559+0000 7f915899d700 1 -- 172.21.15.60:0/3268076222 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91540573d0 con 0x7f915405c470 2022-06-29T16:03:29.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.559+0000 7f9150ff9700 1 --2- 172.21.15.60:0/3268076222 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9134053b00 0x7f9134055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:29.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.559+0000 7f9150ff9700 1 -- 172.21.15.60:0/3268076222 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f914406bf90 con 0x7f915405c470 2022-06-29T16:03:29.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.559+0000 7f9152ffd700 1 --2- 172.21.15.60:0/3268076222 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9134053b00 0x7f9134055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:29.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.563+0000 7f9152ffd700 1 --2- 172.21.15.60:0/3268076222 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9134053b00 0x7f9134055fc0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f914800b000 tx=0x7f9148006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:29.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.567+0000 7f9150ff9700 1 -- 172.21.15.60:0/3268076222 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9144042e10 con 0x7f915405c470 2022-06-29T16:03:29.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.695+0000 7f915899d700 1 -- 172.21.15.60:0/3268076222 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91540677b0 con 0x7f915405c470 2022-06-29T16:03:29.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.695+0000 7f9150ff9700 1 -- 172.21.15.60:0/3268076222 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) v1 ==== 74+0+16041 (secure 0 0 0) 0x7f9144046430 con 0x7f915405c470 2022-06-29T16:03:29.703 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:03:29.704 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":62,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:03:29.116409+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:03:26.621123+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2022-06-29T16:03:27.220279+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:03:29.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.699+0000 7f915899d700 1 -- 172.21.15.60:0/3268076222 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9134053b00 msgr2=0x7f9134055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:29.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.699+0000 7f915899d700 1 --2- 172.21.15.60:0/3268076222 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9134053b00 0x7f9134055fc0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f914800b000 tx=0x7f9148006df0).stop 2022-06-29T16:03:29.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.699+0000 7f915899d700 1 -- 172.21.15.60:0/3268076222 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f915405c470 msgr2=0x7f91540b0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:29.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.699+0000 7f915899d700 1 --2- 172.21.15.60:0/3268076222 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f915405c470 0x7f91540b0cb0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0x7f91440030b0 tx=0x7f9144008040).stop 2022-06-29T16:03:29.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.699+0000 7f915899d700 1 -- 172.21.15.60:0/3268076222 shutdown_connections 2022-06-29T16:03:29.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.699+0000 7f915899d700 1 --2- 172.21.15.60:0/3268076222 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9134053b00 0x7f9134055fc0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:29.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.699+0000 7f915899d700 1 --2- 172.21.15.60:0/3268076222 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91540b3d00 0x7f91540aefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:29.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.699+0000 7f915899d700 1 --2- 172.21.15.60:0/3268076222 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91540b1f60 0x7f9154066cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:29.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.699+0000 7f915899d700 1 --2- 172.21.15.60:0/3268076222 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f915405c470 0x7f91540b0cb0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:29.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.699+0000 7f915899d700 1 -- 172.21.15.60:0/3268076222 >> 172.21.15.60:0/3268076222 conn(0x7f9154064810 msgr2=0x7f915413ecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:29.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.699+0000 7f915899d700 1 -- 172.21.15.60:0/3268076222 shutdown_connections 2022-06-29T16:03:29.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.699+0000 7f915899d700 1 -- 172.21.15.60:0/3268076222 wait complete. 2022-06-29T16:03:29.718 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T16:03:29.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.823+0000 7f967e389700 1 Processor -- start 2022-06-29T16:03:29.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.823+0000 7f967e389700 1 -- start start 2022-06-29T16:03:29.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.823+0000 7f967e389700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96780ae7c0 0x7f9678141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:29.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.823+0000 7f967e389700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f96780b4010 0x7f9678142490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:29.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.823+0000 7f967e389700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9678143a40 con 0x7f9678146160 2022-06-29T16:03:29.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.823+0000 7f967e389700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96780b56d0 con 0x7f96780b4010 2022-06-29T16:03:29.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.823+0000 7f967e389700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96780aee60 con 0x7f96780ae7c0 2022-06-29T16:03:29.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.823+0000 7f96777fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f96780b4010 0x7f9678142490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:29.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.823+0000 7f967c926700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9678146160 0x7f96781465f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56736/0 (socket says 172.21.15.60:56736) 2022-06-29T16:03:29.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.823+0000 7f96777fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f96780b4010 0x7f9678142490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48650/0 (socket says 172.21.15.60:48650) 2022-06-29T16:03:29.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.823+0000 7f967c926700 1 -- 172.21.15.60:0/553469817 learned_addr learned my addr 172.21.15.60:0/553469817 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:03:29.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.823+0000 7f9677fff700 1 --2- 172.21.15.60:0/553469817 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96780ae7c0 0x7f9678141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:29.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.827+0000 7f96777fe700 1 -- 172.21.15.60:0/553469817 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96780ae7c0 msgr2=0x7f9678141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:29.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.827+0000 7f96777fe700 1 --2- 172.21.15.60:0/553469817 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96780ae7c0 0x7f9678141fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:29.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.827+0000 7f96777fe700 1 -- 172.21.15.60:0/553469817 >> v1:172.21.15.60:6789/0 conn(0x7f9678146160 legacy=0x7f96781465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:29.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.827+0000 7f96777fe700 1 -- 172.21.15.60:0/553469817 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96780af100 con 0x7f96780b4010 2022-06-29T16:03:29.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.827+0000 7f96777fe700 1 --2- 172.21.15.60:0/553469817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96780b4010 0x7f9678142490 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7f9668002f70 tx=0x7f9668006df0).ready entity=mon.0 client_cookie=431e35e95c531bcb server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:29.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.827+0000 7f9676ffd700 1 -- 172.21.15.60:0/553469817 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f966800d570 con 0x7f96780b4010 2022-06-29T16:03:29.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.827+0000 7f9676ffd700 1 -- 172.21.15.60:0/553469817 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9668000e90 con 0x7f96780b4010 2022-06-29T16:03:29.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.827+0000 7f9676ffd700 1 -- 172.21.15.60:0/553469817 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f966800d8c0 con 0x7f96780b4010 2022-06-29T16:03:29.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.827+0000 7f967e389700 1 -- 172.21.15.60:0/553469817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96780b4010 msgr2=0x7f9678142490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:29.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.827+0000 7f967e389700 1 --2- 172.21.15.60:0/553469817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96780b4010 0x7f9678142490 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7f9668002f70 tx=0x7f9668006df0).stop 2022-06-29T16:03:29.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.827+0000 7f967e389700 1 -- 172.21.15.60:0/553469817 shutdown_connections 2022-06-29T16:03:29.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.827+0000 7f967e389700 1 --2- 172.21.15.60:0/553469817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96780b4010 0x7f9678142490 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:29.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.827+0000 7f967e389700 1 --2- 172.21.15.60:0/553469817 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96780ae7c0 0x7f9678141fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:29.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.827+0000 7f967e389700 1 -- 172.21.15.60:0/553469817 >> 172.21.15.60:0/553469817 conn(0x7f96780b9340 msgr2=0x7f96780b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:29.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.827+0000 7f967e389700 1 -- 172.21.15.60:0/553469817 shutdown_connections 2022-06-29T16:03:29.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.827+0000 7f967e389700 1 -- 172.21.15.60:0/553469817 wait complete. 2022-06-29T16:03:29.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.827+0000 7f967e389700 1 Processor -- start 2022-06-29T16:03:29.845 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.827+0000 7f967e389700 1 -- start start 2022-06-29T16:03:29.845 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.831+0000 7f967e389700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96780ae7c0 0x7f9678140c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:29.845 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.831+0000 7f967e389700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96780b4010 0x7f96781524d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:29.845 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.831+0000 7f967e389700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9678146160 0x7f96781529c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:29.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.831+0000 7f967e389700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9678152eb0 con 0x7f96780ae7c0 2022-06-29T16:03:29.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.831+0000 7f967e389700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96780afc80 con 0x7f9678146160 2022-06-29T16:03:29.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.831+0000 7f967e389700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9678141150 con 0x7f96780b4010 2022-06-29T16:03:29.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.831+0000 7f9677fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96780ae7c0 0x7f9678140c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:29.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.831+0000 7f9677fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96780ae7c0 0x7f9678140c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48656/0 (socket says 172.21.15.60:48656) 2022-06-29T16:03:29.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.831+0000 7f9677fff700 1 -- 172.21.15.60:0/3997200691 learned_addr learned my addr 172.21.15.60:0/3997200691 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:29.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.831+0000 7f96777fe700 1 --2- 172.21.15.60:0/3997200691 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96780b4010 0x7f96781524d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:29.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.831+0000 7f967c926700 1 --2- 172.21.15.60:0/3997200691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9678146160 0x7f96781529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:29.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.831+0000 7f9677fff700 1 -- 172.21.15.60:0/3997200691 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96780b4010 msgr2=0x7f96781524d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:29.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.831+0000 7f9677fff700 1 --2- 172.21.15.60:0/3997200691 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96780b4010 0x7f96781524d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:29.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.831+0000 7f9677fff700 1 -- 172.21.15.60:0/3997200691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9678146160 msgr2=0x7f96781529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:29.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.831+0000 7f9677fff700 1 --2- 172.21.15.60:0/3997200691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9678146160 0x7f96781529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:29.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.831+0000 7f9677fff700 1 -- 172.21.15.60:0/3997200691 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f966800b180 con 0x7f96780ae7c0 2022-06-29T16:03:29.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.831+0000 7f96777fe700 1 --2- 172.21.15.60:0/3997200691 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96780b4010 0x7f96781524d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:03:29.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.831+0000 7f9677fff700 1 --2- 172.21.15.60:0/3997200691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96780ae7c0 0x7f9678140c60 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7f96600030b0 tx=0x7f9660008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:29.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.831+0000 7f96757fa700 1 -- 172.21.15.60:0/3997200691 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9660011780 con 0x7f96780ae7c0 2022-06-29T16:03:29.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.831+0000 7f96757fa700 1 -- 172.21.15.60:0/3997200691 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96600118e0 con 0x7f96780ae7c0 2022-06-29T16:03:29.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.831+0000 7f967e389700 1 -- 172.21.15.60:0/3997200691 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96780b3150 con 0x7f96780ae7c0 2022-06-29T16:03:29.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.831+0000 7f96757fa700 1 -- 172.21.15.60:0/3997200691 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9660011c40 con 0x7f96780ae7c0 2022-06-29T16:03:29.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.831+0000 7f967e389700 1 -- 172.21.15.60:0/3997200691 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9678145f30 con 0x7f96780ae7c0 2022-06-29T16:03:29.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.831+0000 7f967e389700 1 -- 172.21.15.60:0/3997200691 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96780afad0 con 0x7f96780ae7c0 2022-06-29T16:03:29.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.835+0000 7f96757fa700 1 -- 172.21.15.60:0/3997200691 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f966001c460 con 0x7f96780ae7c0 2022-06-29T16:03:29.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.835+0000 7f96757fa700 1 --2- 172.21.15.60:0/3997200691 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f965c053b30 0x7f965c055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:29.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.835+0000 7f96757fa700 1 -- 172.21.15.60:0/3997200691 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f966006c070 con 0x7f96780ae7c0 2022-06-29T16:03:29.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.835+0000 7f96777fe700 1 --2- 172.21.15.60:0/3997200691 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f965c053b30 0x7f965c055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:29.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.835+0000 7f96777fe700 1 --2- 172.21.15.60:0/3997200691 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f965c053b30 0x7f965c055ff0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f966801a000 tx=0x7f9668015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:29.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:29.843+0000 7f96757fa700 1 -- 172.21.15.60:0/3997200691 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9660042f30 con 0x7f96780ae7c0 2022-06-29T16:03:30.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.007+0000 7f967e389700 1 -- 172.21.15.60:0/3997200691 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f96780573d0 con 0x7f96780ae7c0 2022-06-29T16:03:30.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.011+0000 7f96757fa700 1 -- 172.21.15.60:0/3997200691 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v22) v1 ==== 184+0+0 (secure 0 0 0) 0x7f9660046550 con 0x7f96780ae7c0 2022-06-29T16:03:30.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.015+0000 7f967e389700 1 -- 172.21.15.60:0/3997200691 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f965c053b30 msgr2=0x7f965c055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:30.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.015+0000 7f967e389700 1 --2- 172.21.15.60:0/3997200691 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f965c053b30 0x7f965c055ff0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f966801a000 tx=0x7f9668015000).stop 2022-06-29T16:03:30.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.015+0000 7f967e389700 1 -- 172.21.15.60:0/3997200691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96780ae7c0 msgr2=0x7f9678140c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:30.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.015+0000 7f967e389700 1 --2- 172.21.15.60:0/3997200691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96780ae7c0 0x7f9678140c60 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7f96600030b0 tx=0x7f9660008040).stop 2022-06-29T16:03:30.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.015+0000 7f967e389700 1 -- 172.21.15.60:0/3997200691 shutdown_connections 2022-06-29T16:03:30.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.015+0000 7f967e389700 1 --2- 172.21.15.60:0/3997200691 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f965c053b30 0x7f965c055ff0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:30.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.015+0000 7f967e389700 1 --2- 172.21.15.60:0/3997200691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9678146160 0x7f96781529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:30.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.015+0000 7f967e389700 1 --2- 172.21.15.60:0/3997200691 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96780b4010 0x7f96781524d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:30.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.015+0000 7f967e389700 1 --2- 172.21.15.60:0/3997200691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96780ae7c0 0x7f9678140c60 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:30.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.015+0000 7f967e389700 1 -- 172.21.15.60:0/3997200691 >> 172.21.15.60:0/3997200691 conn(0x7f96780b9340 msgr2=0x7f96780b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:30.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.015+0000 7f967e389700 1 -- 172.21.15.60:0/3997200691 shutdown_connections 2022-06-29T16:03:30.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.015+0000 7f967e389700 1 -- 172.21.15.60:0/3997200691 wait complete. 2022-06-29T16:03:30.026 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.0 2022-06-29T16:03:30.032 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T16:03:30.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.139+0000 7f1e32fa6700 1 Processor -- start 2022-06-29T16:03:30.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.139+0000 7f1e32fa6700 1 -- start start 2022-06-29T16:03:30.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.143+0000 7f1e32fa6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1e2c067830 0x7f1e2c1426b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:30.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.143+0000 7f1e32fa6700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1e2c05f7c0 0x7f1e2c0aeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:30.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.143+0000 7f1e32fa6700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e2c13f0b0 con 0x7f1e2c05fc20 2022-06-29T16:03:30.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.143+0000 7f1e32fa6700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e2c0ad4e0 con 0x7f1e2c05f7c0 2022-06-29T16:03:30.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.143+0000 7f1e32fa6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2c060b20 con 0x7f1e2c067830 2022-06-29T16:03:30.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.143+0000 7f1e317a3700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1e2c05f7c0 0x7f1e2c0aeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:30.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.143+0000 7f1e327a5700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f1e2c05fc20 0x7f1e2c0b2030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:32770/0 (socket says 172.21.15.60:32770) 2022-06-29T16:03:30.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.143+0000 7f1e327a5700 1 -- 172.21.15.60:0/3134917315 learned_addr learned my addr 172.21.15.60:0/3134917315 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:03:30.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.143+0000 7f1e31fa4700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1e2c067830 0x7f1e2c1426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:30.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.143+0000 7f1e30fa2700 1 -- 172.21.15.60:0/3134917315 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483577918 0 0) 0x7f1e24001490 con 0x7f1e2c05fc20 2022-06-29T16:03:30.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.143+0000 7f1e30fa2700 1 -- 172.21.15.60:0/3134917315 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e14003260 con 0x7f1e2c05fc20 2022-06-29T16:03:30.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.143+0000 7f1e317a3700 1 -- 172.21.15.60:0/3134917315 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1e2c067830 msgr2=0x7f1e2c1426b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:30.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.143+0000 7f1e317a3700 1 --2- 172.21.15.60:0/3134917315 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1e2c067830 0x7f1e2c1426b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:30.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.143+0000 7f1e317a3700 1 -- 172.21.15.60:0/3134917315 >> v1:172.21.15.92:6789/0 conn(0x7f1e2c05fc20 legacy=0x7f1e2c0b2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:30.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.143+0000 7f1e317a3700 1 -- 172.21.15.60:0/3134917315 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e2c0b0f70 con 0x7f1e2c05f7c0 2022-06-29T16:03:30.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.143+0000 7f1e31fa4700 1 --2- 172.21.15.60:0/3134917315 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1e2c067830 0x7f1e2c1426b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:03:30.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.143+0000 7f1e317a3700 1 --2- 172.21.15.60:0/3134917315 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e2c05f7c0 0x7f1e2c0aeeb0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f1e1c002f70 tx=0x7f1e1c006df0).ready entity=mon.1 client_cookie=e88d728e1a8587b9 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:30.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.143+0000 7f1e30fa2700 1 -- 172.21.15.60:0/3134917315 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e1c00d4a0 con 0x7f1e2c05f7c0 2022-06-29T16:03:30.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.143+0000 7f1e30fa2700 1 -- 172.21.15.60:0/3134917315 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e1c000e90 con 0x7f1e2c05f7c0 2022-06-29T16:03:30.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.143+0000 7f1e30fa2700 1 -- 172.21.15.60:0/3134917315 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e1c00d7f0 con 0x7f1e2c05f7c0 2022-06-29T16:03:30.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.147+0000 7f1e32fa6700 1 -- 172.21.15.60:0/3134917315 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e2c05f7c0 msgr2=0x7f1e2c0aeeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:30.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.147+0000 7f1e32fa6700 1 --2- 172.21.15.60:0/3134917315 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e2c05f7c0 0x7f1e2c0aeeb0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f1e1c002f70 tx=0x7f1e1c006df0).stop 2022-06-29T16:03:30.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.147+0000 7f1e32fa6700 1 -- 172.21.15.60:0/3134917315 shutdown_connections 2022-06-29T16:03:30.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.147+0000 7f1e32fa6700 1 --2- 172.21.15.60:0/3134917315 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e2c05f7c0 0x7f1e2c0aeeb0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:30.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.147+0000 7f1e32fa6700 1 --2- 172.21.15.60:0/3134917315 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1e2c067830 0x7f1e2c1426b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:30.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.147+0000 7f1e32fa6700 1 -- 172.21.15.60:0/3134917315 >> 172.21.15.60:0/3134917315 conn(0x7f1e2c063ab0 msgr2=0x7f1e2c05f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:30.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.147+0000 7f1e32fa6700 1 -- 172.21.15.60:0/3134917315 shutdown_connections 2022-06-29T16:03:30.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.147+0000 7f1e32fa6700 1 -- 172.21.15.60:0/3134917315 wait complete. 2022-06-29T16:03:30.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.147+0000 7f1e32fa6700 1 Processor -- start 2022-06-29T16:03:30.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.147+0000 7f1e32fa6700 1 -- start start 2022-06-29T16:03:30.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.151+0000 7f1e32fa6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1e2c05f7c0 0x7f1e2c115340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:30.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.151+0000 7f1e32fa6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e2c05fc20 0x7f1e2c115830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:30.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.151+0000 7f1e32fa6700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e2c067830 0x7f1e2c13c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:30.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.151+0000 7f1e32fa6700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2c117c10 con 0x7f1e2c05fc20 2022-06-29T16:03:30.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.151+0000 7f1e32fa6700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2c117e40 con 0x7f1e2c067830 2022-06-29T16:03:30.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.151+0000 7f1e32fa6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2c113cb0 con 0x7f1e2c05f7c0 2022-06-29T16:03:30.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.151+0000 7f1e317a3700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e2c05fc20 0x7f1e2c115830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:30.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.151+0000 7f1e317a3700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e2c05fc20 0x7f1e2c115830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48672/0 (socket says 172.21.15.60:48672) 2022-06-29T16:03:30.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.151+0000 7f1e317a3700 1 -- 172.21.15.60:0/3411379791 learned_addr learned my addr 172.21.15.60:0/3411379791 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:30.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.151+0000 7f1e31fa4700 1 --2- 172.21.15.60:0/3411379791 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1e2c05f7c0 0x7f1e2c115340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:30.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.151+0000 7f1e327a5700 1 --2- 172.21.15.60:0/3411379791 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e2c067830 0x7f1e2c13c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:30.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.151+0000 7f1e317a3700 1 -- 172.21.15.60:0/3411379791 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1e2c05f7c0 msgr2=0x7f1e2c115340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:30.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.151+0000 7f1e317a3700 1 --2- 172.21.15.60:0/3411379791 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1e2c05f7c0 0x7f1e2c115340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:30.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.151+0000 7f1e317a3700 1 -- 172.21.15.60:0/3411379791 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e2c067830 msgr2=0x7f1e2c13c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:30.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.151+0000 7f1e317a3700 1 --2- 172.21.15.60:0/3411379791 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e2c067830 0x7f1e2c13c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:30.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.151+0000 7f1e317a3700 1 -- 172.21.15.60:0/3411379791 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e2800c200 con 0x7f1e2c05fc20 2022-06-29T16:03:30.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.151+0000 7f1e317a3700 1 --2- 172.21.15.60:0/3411379791 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e2c05fc20 0x7f1e2c115830 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0x7f1e1c002f70 tx=0x7f1e1c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:30.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.151+0000 7f1e1affd700 1 -- 172.21.15.60:0/3411379791 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e1c012930 con 0x7f1e2c05fc20 2022-06-29T16:03:30.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.151+0000 7f1e1affd700 1 -- 172.21.15.60:0/3411379791 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e1c012a90 con 0x7f1e2c05fc20 2022-06-29T16:03:30.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.151+0000 7f1e1affd700 1 -- 172.21.15.60:0/3411379791 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e1c012df0 con 0x7f1e2c05fc20 2022-06-29T16:03:30.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.151+0000 7f1e32fa6700 1 -- 172.21.15.60:0/3411379791 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e1c00b180 con 0x7f1e2c05fc20 2022-06-29T16:03:30.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.151+0000 7f1e32fa6700 1 -- 172.21.15.60:0/3411379791 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e2c1139a0 con 0x7f1e2c05fc20 2022-06-29T16:03:30.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.155+0000 7f1e32fa6700 1 -- 172.21.15.60:0/3411379791 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e2c0573d0 con 0x7f1e2c05fc20 2022-06-29T16:03:30.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.155+0000 7f1e1affd700 1 -- 172.21.15.60:0/3411379791 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1e1c01c070 con 0x7f1e2c05fc20 2022-06-29T16:03:30.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.155+0000 7f1e1affd700 1 --2- 172.21.15.60:0/3411379791 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1e0c053950 0x7f1e0c055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:30.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.155+0000 7f1e1affd700 1 -- 172.21.15.60:0/3411379791 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1e1c06cb60 con 0x7f1e2c05fc20 2022-06-29T16:03:30.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.155+0000 7f1e31fa4700 1 --2- 172.21.15.60:0/3411379791 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1e0c053950 0x7f1e0c055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:30.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.155+0000 7f1e31fa4700 1 --2- 172.21.15.60:0/3411379791 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1e0c053950 0x7f1e0c055e10 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f1e28009000 tx=0x7f1e28009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:30.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.159+0000 7f1e1affd700 1 -- 172.21.15.60:0/3411379791 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1e1c043970 con 0x7f1e2c05fc20 2022-06-29T16:03:30.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.327+0000 7f1e32fa6700 1 -- 172.21.15.60:0/3411379791 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f1e2c068560 con 0x7f1e2c05fc20 2022-06-29T16:03:30.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.331+0000 7f1e1affd700 1 -- 172.21.15.60:0/3411379791 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v23) v1 ==== 184+0+0 (secure 0 0 0) 0x7f1e1c019030 con 0x7f1e2c05fc20 2022-06-29T16:03:30.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.335+0000 7f1e32fa6700 1 -- 172.21.15.60:0/3411379791 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1e0c053950 msgr2=0x7f1e0c055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:30.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.335+0000 7f1e32fa6700 1 --2- 172.21.15.60:0/3411379791 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1e0c053950 0x7f1e0c055e10 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f1e28009000 tx=0x7f1e28009040).stop 2022-06-29T16:03:30.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.335+0000 7f1e32fa6700 1 -- 172.21.15.60:0/3411379791 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e2c05fc20 msgr2=0x7f1e2c115830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:30.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.335+0000 7f1e32fa6700 1 --2- 172.21.15.60:0/3411379791 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e2c05fc20 0x7f1e2c115830 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0x7f1e1c002f70 tx=0x7f1e1c014000).stop 2022-06-29T16:03:30.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.335+0000 7f1e32fa6700 1 -- 172.21.15.60:0/3411379791 shutdown_connections 2022-06-29T16:03:30.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.335+0000 7f1e32fa6700 1 --2- 172.21.15.60:0/3411379791 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1e0c053950 0x7f1e0c055e10 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:30.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.335+0000 7f1e32fa6700 1 --2- 172.21.15.60:0/3411379791 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e2c067830 0x7f1e2c13c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:30.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.335+0000 7f1e32fa6700 1 --2- 172.21.15.60:0/3411379791 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e2c05fc20 0x7f1e2c115830 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:30.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.335+0000 7f1e32fa6700 1 --2- 172.21.15.60:0/3411379791 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1e2c05f7c0 0x7f1e2c115340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:30.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.335+0000 7f1e32fa6700 1 -- 172.21.15.60:0/3411379791 >> 172.21.15.60:0/3411379791 conn(0x7f1e2c063ab0 msgr2=0x7f1e2c0acdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:30.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.335+0000 7f1e32fa6700 1 -- 172.21.15.60:0/3411379791 shutdown_connections 2022-06-29T16:03:30.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.335+0000 7f1e32fa6700 1 -- 172.21.15.60:0/3411379791 wait complete. 2022-06-29T16:03:30.347 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.1 2022-06-29T16:03:30.355 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T16:03:30.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.463+0000 7f081e68b700 1 Processor -- start 2022-06-29T16:03:30.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.463+0000 7f081e68b700 1 -- start start 2022-06-29T16:03:30.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.467+0000 7f081e68b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f08180b5eb0 0x7f0818142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:30.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.467+0000 7f081e68b700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0818146d60 0x7f08180b2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:30.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.467+0000 7f081e68b700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08180b6290 con 0x7f08180b0280 2022-06-29T16:03:30.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.467+0000 7f081e68b700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08180b0ab0 con 0x7f08180b5eb0 2022-06-29T16:03:30.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.467+0000 7f081e68b700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08181461b0 con 0x7f0818146d60 2022-06-29T16:03:30.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.467+0000 7f081d689700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f08180b5eb0 0x7f0818142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:30.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.467+0000 7f081d689700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f08180b5eb0 0x7f0818142a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48678/0 (socket says 172.21.15.60:48678) 2022-06-29T16:03:30.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.467+0000 7f081d689700 1 -- 172.21.15.60:0/4178887744 learned_addr learned my addr 172.21.15.60:0/4178887744 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:30.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.467+0000 7f081de8a700 1 --2- 172.21.15.60:0/4178887744 >> v2:172.21.15.92:3300/0 conn(0x7f0818146d60 0x7f08180b2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:30.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.467+0000 7f081ce88700 1 --1- 172.21.15.60:0/4178887744 >> v1:172.21.15.92:6789/0 conn(0x7f08180b0280 0x7f0818146650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:32782/0 (socket says 172.21.15.60:32782) 2022-06-29T16:03:30.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.467+0000 7f081d689700 1 -- 172.21.15.60:0/4178887744 >> v2:172.21.15.92:3300/0 conn(0x7f0818146d60 msgr2=0x7f08180b2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:30.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.467+0000 7f081d689700 1 --2- 172.21.15.60:0/4178887744 >> v2:172.21.15.92:3300/0 conn(0x7f0818146d60 0x7f08180b2bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:30.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.467+0000 7f081d689700 1 -- 172.21.15.60:0/4178887744 >> v1:172.21.15.92:6789/0 conn(0x7f08180b0280 legacy=0x7f0818146650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:30.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.467+0000 7f081d689700 1 -- 172.21.15.60:0/4178887744 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0818142f10 con 0x7f08180b5eb0 2022-06-29T16:03:30.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.467+0000 7f081d689700 1 --2- 172.21.15.60:0/4178887744 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08180b5eb0 0x7f0818142a20 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7f0814002f70 tx=0x7f0814006eb0).ready entity=mon.0 client_cookie=3d489eabe030a3e1 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:30.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.467+0000 7f0807fff700 1 -- 172.21.15.60:0/4178887744 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f081400d5c0 con 0x7f08180b5eb0 2022-06-29T16:03:30.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.467+0000 7f0807fff700 1 -- 172.21.15.60:0/4178887744 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0814000e90 con 0x7f08180b5eb0 2022-06-29T16:03:30.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.467+0000 7f0807fff700 1 -- 172.21.15.60:0/4178887744 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f081400d950 con 0x7f08180b5eb0 2022-06-29T16:03:30.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.467+0000 7f081e68b700 1 -- 172.21.15.60:0/4178887744 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08180b5eb0 msgr2=0x7f0818142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:30.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.467+0000 7f081e68b700 1 --2- 172.21.15.60:0/4178887744 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08180b5eb0 0x7f0818142a20 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7f0814002f70 tx=0x7f0814006eb0).stop 2022-06-29T16:03:30.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.467+0000 7f081e68b700 1 -- 172.21.15.60:0/4178887744 shutdown_connections 2022-06-29T16:03:30.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.467+0000 7f081e68b700 1 --2- 172.21.15.60:0/4178887744 >> v2:172.21.15.92:3300/0 conn(0x7f0818146d60 0x7f08180b2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:30.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.467+0000 7f081e68b700 1 --2- 172.21.15.60:0/4178887744 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08180b5eb0 0x7f0818142a20 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:30.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.467+0000 7f081e68b700 1 -- 172.21.15.60:0/4178887744 >> 172.21.15.60:0/4178887744 conn(0x7f0818062090 msgr2=0x7f0818062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:30.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.467+0000 7f081e68b700 1 -- 172.21.15.60:0/4178887744 shutdown_connections 2022-06-29T16:03:30.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.467+0000 7f081e68b700 1 -- 172.21.15.60:0/4178887744 wait complete. 2022-06-29T16:03:30.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.471+0000 7f081e68b700 1 Processor -- start 2022-06-29T16:03:30.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.471+0000 7f081e68b700 1 -- start start 2022-06-29T16:03:30.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.471+0000 7f081e68b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08180b0280 0x7f0818141e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:30.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.471+0000 7f081e68b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08180b5eb0 0x7f0818155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:30.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.471+0000 7f081e68b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0818146d60 0x7f0818145620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:30.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.471+0000 7f081e68b700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0818144ce0 con 0x7f08180b5eb0 2022-06-29T16:03:30.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.471+0000 7f081e68b700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0818143fd0 con 0x7f0818146d60 2022-06-29T16:03:30.486 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.471+0000 7f081e68b700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0818144f10 con 0x7f08180b0280 2022-06-29T16:03:30.486 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.471+0000 7f081ce88700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08180b5eb0 0x7f0818155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:30.486 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.471+0000 7f081d689700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08180b0280 0x7f0818141e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:30.486 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.471+0000 7f081ce88700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08180b5eb0 0x7f0818155190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48686/0 (socket says 172.21.15.60:48686) 2022-06-29T16:03:30.487 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.471+0000 7f081ce88700 1 -- 172.21.15.60:0/2527916723 learned_addr learned my addr 172.21.15.60:0/2527916723 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:30.487 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.471+0000 7f081de8a700 1 --2- 172.21.15.60:0/2527916723 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0818146d60 0x7f0818145620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:30.487 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.475+0000 7f081ce88700 1 -- 172.21.15.60:0/2527916723 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08180b0280 msgr2=0x7f0818141e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:30.487 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.475+0000 7f081ce88700 1 --2- 172.21.15.60:0/2527916723 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08180b0280 0x7f0818141e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:30.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.475+0000 7f081ce88700 1 -- 172.21.15.60:0/2527916723 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0818146d60 msgr2=0x7f0818145620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:30.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.475+0000 7f081ce88700 1 --2- 172.21.15.60:0/2527916723 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0818146d60 0x7f0818145620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:30.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.475+0000 7f081ce88700 1 -- 172.21.15.60:0/2527916723 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f081400b1e0 con 0x7f08180b5eb0 2022-06-29T16:03:30.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.475+0000 7f081ce88700 1 --2- 172.21.15.60:0/2527916723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08180b5eb0 0x7f0818155190 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7f08180ba240 tx=0x7f0808007c40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:30.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.475+0000 7f08067fc700 1 -- 172.21.15.60:0/2527916723 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08080070d0 con 0x7f08180b5eb0 2022-06-29T16:03:30.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.475+0000 7f08067fc700 1 -- 172.21.15.60:0/2527916723 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0808007230 con 0x7f08180b5eb0 2022-06-29T16:03:30.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.475+0000 7f081e68b700 1 -- 172.21.15.60:0/2527916723 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0818155680 con 0x7f08180b5eb0 2022-06-29T16:03:30.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.475+0000 7f08067fc700 1 -- 172.21.15.60:0/2527916723 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0808005550 con 0x7f08180b5eb0 2022-06-29T16:03:30.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.475+0000 7f081e68b700 1 -- 172.21.15.60:0/2527916723 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f081813d530 con 0x7f08180b5eb0 2022-06-29T16:03:30.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.475+0000 7f081e68b700 1 -- 172.21.15.60:0/2527916723 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08180573d0 con 0x7f08180b5eb0 2022-06-29T16:03:30.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.475+0000 7f08067fc700 1 -- 172.21.15.60:0/2527916723 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f080801a070 con 0x7f08180b5eb0 2022-06-29T16:03:30.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.475+0000 7f08067fc700 1 --2- 172.21.15.60:0/2527916723 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f07f80539e0 0x7f07f8055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:30.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.475+0000 7f08067fc700 1 -- 172.21.15.60:0/2527916723 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f080806ae50 con 0x7f08180b5eb0 2022-06-29T16:03:30.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.475+0000 7f081d689700 1 --2- 172.21.15.60:0/2527916723 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f07f80539e0 0x7f07f8055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:30.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.479+0000 7f081d689700 1 --2- 172.21.15.60:0/2527916723 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f07f80539e0 0x7f07f8055ea0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f081401c000 tx=0x7f0814015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:30.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.483+0000 7f08067fc700 1 -- 172.21.15.60:0/2527916723 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0808041d70 con 0x7f08180b5eb0 2022-06-29T16:03:30.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.647+0000 7f081e68b700 1 -- 172.21.15.60:0/2527916723 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f0818144140 con 0x7f08180b5eb0 2022-06-29T16:03:30.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.655+0000 7f08067fc700 1 -- 172.21.15.60:0/2527916723 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v24) v1 ==== 184+0+0 (secure 0 0 0) 0x7f0808045390 con 0x7f08180b5eb0 2022-06-29T16:03:30.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.659+0000 7f081e68b700 1 -- 172.21.15.60:0/2527916723 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f07f80539e0 msgr2=0x7f07f8055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:30.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.659+0000 7f081e68b700 1 --2- 172.21.15.60:0/2527916723 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f07f80539e0 0x7f07f8055ea0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f081401c000 tx=0x7f0814015000).stop 2022-06-29T16:03:30.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.659+0000 7f081e68b700 1 -- 172.21.15.60:0/2527916723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08180b5eb0 msgr2=0x7f0818155190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:30.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.659+0000 7f081e68b700 1 --2- 172.21.15.60:0/2527916723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08180b5eb0 0x7f0818155190 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7f08180ba240 tx=0x7f0808007c40).stop 2022-06-29T16:03:30.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.659+0000 7f081e68b700 1 -- 172.21.15.60:0/2527916723 shutdown_connections 2022-06-29T16:03:30.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.659+0000 7f081e68b700 1 --2- 172.21.15.60:0/2527916723 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f07f80539e0 0x7f07f8055ea0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:30.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.659+0000 7f081e68b700 1 --2- 172.21.15.60:0/2527916723 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0818146d60 0x7f0818145620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:30.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.659+0000 7f081e68b700 1 --2- 172.21.15.60:0/2527916723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08180b5eb0 0x7f0818155190 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:30.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.659+0000 7f081e68b700 1 --2- 172.21.15.60:0/2527916723 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08180b0280 0x7f0818141e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:30.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.659+0000 7f081e68b700 1 -- 172.21.15.60:0/2527916723 >> 172.21.15.60:0/2527916723 conn(0x7f0818062090 msgr2=0x7f0818143560 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:30.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.659+0000 7f081e68b700 1 -- 172.21.15.60:0/2527916723 shutdown_connections 2022-06-29T16:03:30.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.659+0000 7f081e68b700 1 -- 172.21.15.60:0/2527916723 wait complete. 2022-06-29T16:03:30.670 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.2 2022-06-29T16:03:30.682 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:03:30.798 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.791+0000 7f6a9484f700 1 Processor -- start 2022-06-29T16:03:30.798 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.791+0000 7f6a9484f700 1 -- start start 2022-06-29T16:03:30.801 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.791+0000 7f6a9484f700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6a900b5eb0 0x7f6a90142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:30.802 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.791+0000 7f6a9484f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6a900b2c50 0x7f6a900b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:30.802 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.791+0000 7f6a9484f700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a900b0cf0 con 0x7f6a900b48b0 2022-06-29T16:03:30.802 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.791+0000 7f6a9484f700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a900b54e0 con 0x7f6a900b5eb0 2022-06-29T16:03:30.802 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.791+0000 7f6a9484f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a900b4d10 con 0x7f6a900b2c50 2022-06-29T16:03:30.803 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.791+0000 7f6a8effd700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6a900b48b0 0x7f6a900b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56780/0 (socket says 172.21.15.60:56780) 2022-06-29T16:03:30.803 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.791+0000 7f6a8effd700 1 -- 172.21.15.60:0/1324089281 learned_addr learned my addr 172.21.15.60:0/1324089281 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:03:30.803 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.791+0000 7f6a8f7fe700 1 --2- 172.21.15.60:0/1324089281 >> v2:172.21.15.92:3300/0 conn(0x7f6a900b5eb0 0x7f6a90142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:30.803 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.791+0000 7f6a8ffff700 1 --2- 172.21.15.60:0/1324089281 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6a900b2c50 0x7f6a900b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:30.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.791+0000 7f6a8e7fc700 1 -- 172.21.15.60:0/1324089281 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 563914222 0 0) 0x7f6a84001110 con 0x7f6a900b48b0 2022-06-29T16:03:30.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.791+0000 7f6a8e7fc700 1 -- 172.21.15.60:0/1324089281 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a7c003260 con 0x7f6a900b48b0 2022-06-29T16:03:30.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.791+0000 7f6a8f7fe700 1 -- 172.21.15.60:0/1324089281 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6a900b2c50 msgr2=0x7f6a900b30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:30.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.791+0000 7f6a8f7fe700 1 --2- 172.21.15.60:0/1324089281 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6a900b2c50 0x7f6a900b30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:30.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.791+0000 7f6a8f7fe700 1 -- 172.21.15.60:0/1324089281 >> v1:172.21.15.60:6789/0 conn(0x7f6a900b48b0 legacy=0x7f6a900b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:30.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.791+0000 7f6a8f7fe700 1 -- 172.21.15.60:0/1324089281 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a900b6290 con 0x7f6a900b5eb0 2022-06-29T16:03:30.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.791+0000 7f6a8f7fe700 1 --2- 172.21.15.60:0/1324089281 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6a900b5eb0 0x7f6a90142a20 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f6a880032f0 tx=0x7f6a880091d0).ready entity=mon.1 client_cookie=b9a2e9b086afaa20 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:30.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.791+0000 7f6a8e7fc700 1 -- 172.21.15.60:0/1324089281 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a8800b600 con 0x7f6a900b5eb0 2022-06-29T16:03:30.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.795+0000 7f6a8e7fc700 1 -- 172.21.15.60:0/1324089281 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a8800b760 con 0x7f6a900b5eb0 2022-06-29T16:03:30.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.795+0000 7f6a8e7fc700 1 -- 172.21.15.60:0/1324089281 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a8800b8d0 con 0x7f6a900b5eb0 2022-06-29T16:03:30.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.795+0000 7f6a9484f700 1 -- 172.21.15.60:0/1324089281 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6a900b5eb0 msgr2=0x7f6a90142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:30.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.795+0000 7f6a9484f700 1 --2- 172.21.15.60:0/1324089281 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6a900b5eb0 0x7f6a90142a20 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f6a880032f0 tx=0x7f6a880091d0).stop 2022-06-29T16:03:30.807 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.795+0000 7f6a9484f700 1 -- 172.21.15.60:0/1324089281 shutdown_connections 2022-06-29T16:03:30.807 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.795+0000 7f6a9484f700 1 --2- 172.21.15.60:0/1324089281 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6a900b2c50 0x7f6a900b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:30.807 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.795+0000 7f6a9484f700 1 --2- 172.21.15.60:0/1324089281 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6a900b5eb0 0x7f6a90142a20 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:30.807 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.795+0000 7f6a9484f700 1 -- 172.21.15.60:0/1324089281 >> 172.21.15.60:0/1324089281 conn(0x7f6a90062090 msgr2=0x7f6a90062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:30.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.795+0000 7f6a9484f700 1 -- 172.21.15.60:0/1324089281 shutdown_connections 2022-06-29T16:03:30.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.795+0000 7f6a9484f700 1 -- 172.21.15.60:0/1324089281 wait complete. 2022-06-29T16:03:30.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.795+0000 7f6a9484f700 1 Processor -- start 2022-06-29T16:03:30.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.795+0000 7f6a9484f700 1 -- start start 2022-06-29T16:03:30.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.799+0000 7f6a9484f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6a900b2c50 0x7f6a90148ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:30.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.799+0000 7f6a9484f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a900b48b0 0x7f6a90149db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:30.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.799+0000 7f6a9484f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6a900b5eb0 0x7f6a900af1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:30.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.799+0000 7f6a9484f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a900af6c0 con 0x7f6a900b48b0 2022-06-29T16:03:30.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.799+0000 7f6a9484f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a90146ce0 con 0x7f6a900b5eb0 2022-06-29T16:03:30.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.799+0000 7f6a9484f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a901483c0 con 0x7f6a900b2c50 2022-06-29T16:03:30.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.799+0000 7f6a8effd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a900b48b0 0x7f6a90149db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:30.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.799+0000 7f6a8f7fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6a900b2c50 0x7f6a90148ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:30.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.799+0000 7f6a8effd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a900b48b0 0x7f6a90149db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48700/0 (socket says 172.21.15.60:48700) 2022-06-29T16:03:30.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.799+0000 7f6a8effd700 1 -- 172.21.15.60:0/491648484 learned_addr learned my addr 172.21.15.60:0/491648484 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:30.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.799+0000 7f6a8ffff700 1 --2- 172.21.15.60:0/491648484 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6a900b5eb0 0x7f6a900af1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:30.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.799+0000 7f6a8effd700 1 -- 172.21.15.60:0/491648484 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6a900b2c50 msgr2=0x7f6a90148ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:30.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.799+0000 7f6a8effd700 1 --2- 172.21.15.60:0/491648484 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6a900b2c50 0x7f6a90148ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:30.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.799+0000 7f6a8effd700 1 -- 172.21.15.60:0/491648484 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6a900b5eb0 msgr2=0x7f6a900af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:30.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.799+0000 7f6a8effd700 1 --2- 172.21.15.60:0/491648484 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6a900b5eb0 0x7f6a900af1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:30.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.799+0000 7f6a8effd700 1 -- 172.21.15.60:0/491648484 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a88009470 con 0x7f6a900b48b0 2022-06-29T16:03:30.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.799+0000 7f6a8effd700 1 --2- 172.21.15.60:0/491648484 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a900b48b0 0x7f6a90149db0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7f6a840012a0 tx=0x7f6a840078e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:30.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.799+0000 7f6a8cff9700 1 -- 172.21.15.60:0/491648484 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a840070d0 con 0x7f6a900b48b0 2022-06-29T16:03:30.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.799+0000 7f6a8cff9700 1 -- 172.21.15.60:0/491648484 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a84007230 con 0x7f6a900b48b0 2022-06-29T16:03:30.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.799+0000 7f6a8cff9700 1 -- 172.21.15.60:0/491648484 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a84007590 con 0x7f6a900b48b0 2022-06-29T16:03:30.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.799+0000 7f6a9484f700 1 -- 172.21.15.60:0/491648484 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a900add10 con 0x7f6a900b48b0 2022-06-29T16:03:30.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.799+0000 7f6a9484f700 1 -- 172.21.15.60:0/491648484 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a90144070 con 0x7f6a900b48b0 2022-06-29T16:03:30.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.803+0000 7f6a9484f700 1 -- 172.21.15.60:0/491648484 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a90142f10 con 0x7f6a900b48b0 2022-06-29T16:03:30.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.803+0000 7f6a8cff9700 1 -- 172.21.15.60:0/491648484 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6a84010070 con 0x7f6a900b48b0 2022-06-29T16:03:30.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.803+0000 7f6a8cff9700 1 --2- 172.21.15.60:0/491648484 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6a70053b00 0x7f6a70055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:30.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.803+0000 7f6a8cff9700 1 -- 172.21.15.60:0/491648484 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6a8406b430 con 0x7f6a900b48b0 2022-06-29T16:03:30.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.803+0000 7f6a8f7fe700 1 --2- 172.21.15.60:0/491648484 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6a70053b00 0x7f6a70055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:30.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.803+0000 7f6a8f7fe700 1 --2- 172.21.15.60:0/491648484 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6a70053b00 0x7f6a70055fc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f6a8801b000 tx=0x7f6a88016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:30.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.807+0000 7f6a8cff9700 1 -- 172.21.15.60:0/491648484 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6a84042240 con 0x7f6a900b48b0 2022-06-29T16:03:30.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.971+0000 7f6a9484f700 1 -- 172.21.15.60:0/491648484 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a900b0cf0 con 0x7f6a900b48b0 2022-06-29T16:03:30.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.975+0000 7f6a8cff9700 1 -- 172.21.15.60:0/491648484 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f6a900b0cf0 con 0x7f6a900b48b0 2022-06-29T16:03:30.981 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:03:30.981 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":25,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5955,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.60:6833/66690512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":66690512},{"type":"v1","addr":"172.21.15.60:6833","nonce":66690512}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":23}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:03:28.112667+0000","modified":"2022-06-29T16:03:29.115727+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6013},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6013":{"gid":6013,"name":"b","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/3876875590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3876875590},{"type":"v1","addr":"172.21.15.92:6835","nonce":3876875590}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-06-29T16:03:30.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.975+0000 7f6a9484f700 1 -- 172.21.15.60:0/491648484 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6a70053b00 msgr2=0x7f6a70055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:30.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.975+0000 7f6a9484f700 1 --2- 172.21.15.60:0/491648484 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6a70053b00 0x7f6a70055fc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f6a8801b000 tx=0x7f6a88016000).stop 2022-06-29T16:03:30.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.975+0000 7f6a9484f700 1 -- 172.21.15.60:0/491648484 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a900b48b0 msgr2=0x7f6a90149db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:30.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.975+0000 7f6a9484f700 1 --2- 172.21.15.60:0/491648484 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a900b48b0 0x7f6a90149db0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7f6a840012a0 tx=0x7f6a840078e0).stop 2022-06-29T16:03:30.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.975+0000 7f6a9484f700 1 -- 172.21.15.60:0/491648484 shutdown_connections 2022-06-29T16:03:30.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.975+0000 7f6a9484f700 1 --2- 172.21.15.60:0/491648484 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6a70053b00 0x7f6a70055fc0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:30.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.975+0000 7f6a9484f700 1 --2- 172.21.15.60:0/491648484 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6a900b5eb0 0x7f6a900af1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:30.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.975+0000 7f6a9484f700 1 --2- 172.21.15.60:0/491648484 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a900b48b0 0x7f6a90149db0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:30.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.975+0000 7f6a9484f700 1 --2- 172.21.15.60:0/491648484 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6a900b2c50 0x7f6a90148ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:30.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.975+0000 7f6a9484f700 1 -- 172.21.15.60:0/491648484 >> 172.21.15.60:0/491648484 conn(0x7f6a90062090 msgr2=0x7f6a900b2220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:30.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.975+0000 7f6a9484f700 1 -- 172.21.15.60:0/491648484 shutdown_connections 2022-06-29T16:03:30.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:30.975+0000 7f6a9484f700 1 -- 172.21.15.60:0/491648484 wait complete. 2022-06-29T16:03:30.986 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 25 2022-06-29T16:03:31.196 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:03:28.112667+0000', 'modified': '2022-06-29T16:03:29.115727+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6013}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6013': {'gid': 6013, 'name': 'b', 'rank': 0, 'incarnation': 24, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.92:6835/3876875590', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3876875590}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3876875590}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:03:31.197 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:03:31.197 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-06-29T16:03:31.313 INFO:teuthology.orchestra.run.smithi092.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":6013,"want_state":"up:active","state":"up:active","rank_uptime":3.1843298920000001,"mdsmap_epoch":25,"osdmap_epoch":62,"osdmap_epoch_barrier":61,"uptime":7.5585779679999998} 2022-06-29T16:03:31.314 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 6013, "mdsmap_epoch": 25, "osdmap_epoch": 62, "osdmap_epoch_barrier": 61, "rank_uptime": 3.184329892, "state": "up:active", "uptime": 7.558577968, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:03:31.315 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:03:31.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.419+0000 7fb171c14700 1 Processor -- start 2022-06-29T16:03:31.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.419+0000 7fb171c14700 1 -- start start 2022-06-29T16:03:31.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.419+0000 7fb171c14700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb16c0b59f0 0x7fb16c14d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:31.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.419+0000 7fb171c14700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb16c0b1340 con 0x7fb16c0b5610 2022-06-29T16:03:31.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.419+0000 7fb171c14700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb16c1461b0 con 0x7fb16c0b0e30 2022-06-29T16:03:31.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.419+0000 7fb171c14700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb16c065350 con 0x7fb16c0b59f0 2022-06-29T16:03:31.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.419+0000 7fb16b7fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb16c0b5610 0x7fb16c142a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56792/0 (socket says 172.21.15.60:56792) 2022-06-29T16:03:31.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.419+0000 7fb16b7fe700 1 -- 172.21.15.60:0/1896371396 learned_addr learned my addr 172.21.15.60:0/1896371396 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:03:31.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.419+0000 7fb16bfff700 1 --2- 172.21.15.60:0/1896371396 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb16c0b59f0 0x7fb16c14d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:31.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.419+0000 7fb16affd700 1 -- 172.21.15.60:0/1896371396 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3278646081 0 0) 0x7fb16c0b1340 con 0x7fb16c0b5610 2022-06-29T16:03:31.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.419+0000 7fb16affd700 1 -- 172.21.15.60:0/1896371396 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb14c003260 con 0x7fb16c0b5610 2022-06-29T16:03:31.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.419+0000 7fb16affd700 1 -- 172.21.15.60:0/1896371396 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3201969357 0 0) 0x7fb16c1461b0 con 0x7fb16c0b0e30 2022-06-29T16:03:31.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.419+0000 7fb16affd700 1 -- 172.21.15.60:0/1896371396 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb16c0b1340 con 0x7fb16c0b0e30 2022-06-29T16:03:31.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.419+0000 7fb16bfff700 1 -- 172.21.15.60:0/1896371396 >> v1:172.21.15.92:6789/0 conn(0x7fb16c0b0e30 legacy=0x7fb16c146650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:31.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.419+0000 7fb16affd700 1 -- 172.21.15.60:0/1896371396 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 251889754 0 0) 0x7fb14c003260 con 0x7fb16c0b5610 2022-06-29T16:03:31.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.419+0000 7fb16bfff700 1 -- 172.21.15.60:0/1896371396 >> v1:172.21.15.60:6789/0 conn(0x7fb16c0b5610 legacy=0x7fb16c142a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:31.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.419+0000 7fb16bfff700 1 -- 172.21.15.60:0/1896371396 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb16c146d60 con 0x7fb16c0b59f0 2022-06-29T16:03:31.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.419+0000 7fb16bfff700 1 --2- 172.21.15.60:0/1896371396 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb16c0b59f0 0x7fb16c14d3d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fb15c001800 tx=0x7fb1640091d0).ready entity=mon.2 client_cookie=6b90812189db2ec3 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:31.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.419+0000 7fb16affd700 1 -- 172.21.15.60:0/1896371396 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb15c001660 con 0x7fb16c0b59f0 2022-06-29T16:03:31.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.419+0000 7fb16affd700 1 -- 172.21.15.60:0/1896371396 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb16400f510 con 0x7fb16c0b59f0 2022-06-29T16:03:31.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.419+0000 7fb16affd700 1 -- 172.21.15.60:0/1896371396 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb16400f680 con 0x7fb16c0b59f0 2022-06-29T16:03:31.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.423+0000 7fb171c14700 1 -- 172.21.15.60:0/1896371396 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb16c0b59f0 msgr2=0x7fb16c14d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:31.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.423+0000 7fb171c14700 1 --2- 172.21.15.60:0/1896371396 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb16c0b59f0 0x7fb16c14d3d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fb15c001800 tx=0x7fb1640091d0).stop 2022-06-29T16:03:31.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.423+0000 7fb171c14700 1 -- 172.21.15.60:0/1896371396 shutdown_connections 2022-06-29T16:03:31.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.423+0000 7fb171c14700 1 --2- 172.21.15.60:0/1896371396 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb16c0b59f0 0x7fb16c14d3d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:31.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.423+0000 7fb171c14700 1 -- 172.21.15.60:0/1896371396 >> 172.21.15.60:0/1896371396 conn(0x7fb16c0632c0 msgr2=0x7fb16c0b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:31.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.423+0000 7fb171c14700 1 -- 172.21.15.60:0/1896371396 shutdown_connections 2022-06-29T16:03:31.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.423+0000 7fb171c14700 1 -- 172.21.15.60:0/1896371396 wait complete. 2022-06-29T16:03:31.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.423+0000 7fb171c14700 1 Processor -- start 2022-06-29T16:03:31.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.423+0000 7fb171c14700 1 -- start start 2022-06-29T16:03:31.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.423+0000 7fb171c14700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb16c0b0e30 0x7fb16c13c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:31.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.423+0000 7fb171c14700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb16c0b5610 0x7fb16c0b41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:31.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.423+0000 7fb171c14700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb16c0b59f0 0x7fb16c13d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:31.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.423+0000 7fb171c14700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb16c12f5e0 con 0x7fb16c0b5610 2022-06-29T16:03:31.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.423+0000 7fb171c14700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb16c12f810 con 0x7fb16c0b0e30 2022-06-29T16:03:31.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.423+0000 7fb171c14700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb16c13f8a0 con 0x7fb16c0b59f0 2022-06-29T16:03:31.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.427+0000 7fb162ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb16c0b5610 0x7fb16c0b41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:31.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.427+0000 7fb162ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb16c0b5610 0x7fb16c0b41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48714/0 (socket says 172.21.15.60:48714) 2022-06-29T16:03:31.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.427+0000 7fb162ffd700 1 -- 172.21.15.60:0/420127210 learned_addr learned my addr 172.21.15.60:0/420127210 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:31.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.427+0000 7fb16b7fe700 1 --2- 172.21.15.60:0/420127210 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb16c0b0e30 0x7fb16c13c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:31.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.427+0000 7fb16bfff700 1 --2- 172.21.15.60:0/420127210 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb16c0b59f0 0x7fb16c13d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:31.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.427+0000 7fb162ffd700 1 -- 172.21.15.60:0/420127210 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb16c0b59f0 msgr2=0x7fb16c13d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:31.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.427+0000 7fb162ffd700 1 --2- 172.21.15.60:0/420127210 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb16c0b59f0 0x7fb16c13d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:31.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.427+0000 7fb162ffd700 1 -- 172.21.15.60:0/420127210 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb16c0b0e30 msgr2=0x7fb16c13c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:31.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.427+0000 7fb162ffd700 1 --2- 172.21.15.60:0/420127210 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb16c0b0e30 0x7fb16c13c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:31.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.427+0000 7fb162ffd700 1 -- 172.21.15.60:0/420127210 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb164009470 con 0x7fb16c0b5610 2022-06-29T16:03:31.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.427+0000 7fb162ffd700 1 --2- 172.21.15.60:0/420127210 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb16c0b5610 0x7fb16c0b41e0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0x7fb154001110 tx=0x7fb154007ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:31.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.427+0000 7fb1697fa700 1 -- 172.21.15.60:0/420127210 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb154012400 con 0x7fb16c0b5610 2022-06-29T16:03:31.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.427+0000 7fb1697fa700 1 -- 172.21.15.60:0/420127210 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb154012560 con 0x7fb16c0b5610 2022-06-29T16:03:31.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.427+0000 7fb1697fa700 1 -- 172.21.15.60:0/420127210 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb1540128c0 con 0x7fb16c0b5610 2022-06-29T16:03:31.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.427+0000 7fb171c14700 1 -- 172.21.15.60:0/420127210 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb16c13fdc0 con 0x7fb16c0b5610 2022-06-29T16:03:31.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.427+0000 7fb171c14700 1 -- 172.21.15.60:0/420127210 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb16c13f2a0 con 0x7fb16c0b5610 2022-06-29T16:03:31.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.427+0000 7fb171c14700 1 -- 172.21.15.60:0/420127210 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb16c12f9b0 con 0x7fb16c0b5610 2022-06-29T16:03:31.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.431+0000 7fb1697fa700 1 -- 172.21.15.60:0/420127210 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb154019030 con 0x7fb16c0b5610 2022-06-29T16:03:31.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.431+0000 7fb1697fa700 1 --2- 172.21.15.60:0/420127210 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb150053a70 0x7fb150055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:31.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.431+0000 7fb1697fa700 1 -- 172.21.15.60:0/420127210 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb15406b250 con 0x7fb16c0b5610 2022-06-29T16:03:31.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.431+0000 7fb16b7fe700 1 --2- 172.21.15.60:0/420127210 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb150053a70 0x7fb150055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:31.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.431+0000 7fb16b7fe700 1 --2- 172.21.15.60:0/420127210 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb150053a70 0x7fb150055f30 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7fb15c00b000 tx=0x7fb15c001ae0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:31.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.439+0000 7fb1697fa700 1 -- 172.21.15.60:0/420127210 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb1540420d0 con 0x7fb16c0b5610 2022-06-29T16:03:31.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.567+0000 7fb171c14700 1 -- 172.21.15.60:0/420127210 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb16c1400d0 con 0x7fb16c0b5610 2022-06-29T16:03:31.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.571+0000 7fb1697fa700 1 -- 172.21.15.60:0/420127210 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) v1 ==== 74+0+16041 (secure 0 0 0) 0x7fb1540456f0 con 0x7fb16c0b5610 2022-06-29T16:03:31.578 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:03:31.578 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":62,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:03:29.116409+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:03:26.621123+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2022-06-29T16:03:27.220279+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:03:31.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.571+0000 7fb171c14700 1 -- 172.21.15.60:0/420127210 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb150053a70 msgr2=0x7fb150055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:31.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.571+0000 7fb171c14700 1 --2- 172.21.15.60:0/420127210 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb150053a70 0x7fb150055f30 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7fb15c00b000 tx=0x7fb15c001ae0).stop 2022-06-29T16:03:31.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.571+0000 7fb171c14700 1 -- 172.21.15.60:0/420127210 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb16c0b5610 msgr2=0x7fb16c0b41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:31.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.571+0000 7fb171c14700 1 --2- 172.21.15.60:0/420127210 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb16c0b5610 0x7fb16c0b41e0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0x7fb154001110 tx=0x7fb154007ae0).stop 2022-06-29T16:03:31.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.571+0000 7fb171c14700 1 -- 172.21.15.60:0/420127210 shutdown_connections 2022-06-29T16:03:31.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.571+0000 7fb171c14700 1 --2- 172.21.15.60:0/420127210 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb150053a70 0x7fb150055f30 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:31.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.571+0000 7fb171c14700 1 --2- 172.21.15.60:0/420127210 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb16c0b59f0 0x7fb16c13d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:31.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.571+0000 7fb171c14700 1 --2- 172.21.15.60:0/420127210 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb16c0b5610 0x7fb16c0b41e0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:31.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.571+0000 7fb171c14700 1 --2- 172.21.15.60:0/420127210 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb16c0b0e30 0x7fb16c13c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:31.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.571+0000 7fb171c14700 1 -- 172.21.15.60:0/420127210 >> 172.21.15.60:0/420127210 conn(0x7fb16c0632c0 msgr2=0x7fb16c0b2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:31.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.571+0000 7fb171c14700 1 -- 172.21.15.60:0/420127210 shutdown_connections 2022-06-29T16:03:31.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.571+0000 7fb171c14700 1 -- 172.21.15.60:0/420127210 wait complete. 2022-06-29T16:03:31.591 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-29T16:03:26.621123+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2022-06-29T16:03:27.220279+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-29T16:03:31.592 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:03:31.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.695+0000 7f837759e700 1 Processor -- start 2022-06-29T16:03:31.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.695+0000 7f837759e700 1 -- start start 2022-06-29T16:03:31.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.695+0000 7f837759e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f83780b9bf0 0x7f83780b2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:31.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.695+0000 7f837759e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83780b9fd0 con 0x7f83780b0890 2022-06-29T16:03:31.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.695+0000 7f837759e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83780b1a50 con 0x7f83780b3b40 2022-06-29T16:03:31.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.695+0000 7f837759e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83780b1cf0 con 0x7f83780b9bf0 2022-06-29T16:03:31.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.695+0000 7f837659c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f83780b9bf0 0x7f83780b2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:31.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.695+0000 7f8375d9b700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f83780b0890 0x7f83780b4540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56808/0 (socket says 172.21.15.60:56808) 2022-06-29T16:03:31.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.695+0000 7f837659c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f83780b9bf0 0x7f83780b2190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48720/0 (socket says 172.21.15.60:48720) 2022-06-29T16:03:31.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.695+0000 7f8375d9b700 1 -- 172.21.15.60:0/188326445 learned_addr learned my addr 172.21.15.60:0/188326445 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:03:31.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.695+0000 7f837659c700 1 -- 172.21.15.60:0/188326445 >> v1:172.21.15.92:6789/0 conn(0x7f83780b3b40 legacy=0x7f8378141b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:31.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.695+0000 7f837659c700 1 -- 172.21.15.60:0/188326445 >> v1:172.21.15.60:6789/0 conn(0x7f83780b0890 legacy=0x7f83780b4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:31.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.695+0000 7f837659c700 1 -- 172.21.15.60:0/188326445 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83780b2680 con 0x7f83780b9bf0 2022-06-29T16:03:31.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.695+0000 7f837559a700 1 -- 172.21.15.60:0/188326445 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269272177 0 0) 0x7f83780b9fd0 con 0x7f83780b0890 2022-06-29T16:03:31.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.695+0000 7f837659c700 1 --2- 172.21.15.60:0/188326445 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83780b9bf0 0x7f83780b2190 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0x7f8368002f70 tx=0x7f8368006df0).ready entity=mon.0 client_cookie=3501c0cbb82ad8a4 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:31.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.695+0000 7f837559a700 1 -- 172.21.15.60:0/188326445 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f836800d610 con 0x7f83780b9bf0 2022-06-29T16:03:31.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.695+0000 7f837559a700 1 -- 172.21.15.60:0/188326445 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8368000e90 con 0x7f83780b9bf0 2022-06-29T16:03:31.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.695+0000 7f837559a700 1 -- 172.21.15.60:0/188326445 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f836800d960 con 0x7f83780b9bf0 2022-06-29T16:03:31.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.699+0000 7f837759e700 1 -- 172.21.15.60:0/188326445 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83780b9bf0 msgr2=0x7f83780b2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:31.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.699+0000 7f837759e700 1 --2- 172.21.15.60:0/188326445 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83780b9bf0 0x7f83780b2190 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0x7f8368002f70 tx=0x7f8368006df0).stop 2022-06-29T16:03:31.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.699+0000 7f837759e700 1 -- 172.21.15.60:0/188326445 shutdown_connections 2022-06-29T16:03:31.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.699+0000 7f837759e700 1 --2- 172.21.15.60:0/188326445 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83780b9bf0 0x7f83780b2190 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:31.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.699+0000 7f837759e700 1 -- 172.21.15.60:0/188326445 >> 172.21.15.60:0/188326445 conn(0x7f8378062b60 msgr2=0x7f83780b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:31.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.699+0000 7f837759e700 1 -- 172.21.15.60:0/188326445 shutdown_connections 2022-06-29T16:03:31.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.699+0000 7f837759e700 1 -- 172.21.15.60:0/188326445 wait complete. 2022-06-29T16:03:31.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.699+0000 7f837759e700 1 Processor -- start 2022-06-29T16:03:31.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.699+0000 7f837759e700 1 -- start start 2022-06-29T16:03:31.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.703+0000 7f837759e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83780b0890 0x7f83781473c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:31.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.703+0000 7f837759e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83780b3b40 0x7f83780aefa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:31.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.703+0000 7f837759e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83780b9bf0 0x7f837813d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:31.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.703+0000 7f837759e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83781469e0 con 0x7f83780b3b40 2022-06-29T16:03:31.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.703+0000 7f837759e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8378146c80 con 0x7f83780b0890 2022-06-29T16:03:31.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.703+0000 7f837759e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8378146f20 con 0x7f83780b9bf0 2022-06-29T16:03:31.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.703+0000 7f8375d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83780b3b40 0x7f83780aefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:31.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.703+0000 7f8375d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83780b3b40 0x7f83780aefa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48728/0 (socket says 172.21.15.60:48728) 2022-06-29T16:03:31.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.703+0000 7f8375d9b700 1 -- 172.21.15.60:0/2998885510 learned_addr learned my addr 172.21.15.60:0/2998885510 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:31.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.703+0000 7f837659c700 1 --2- 172.21.15.60:0/2998885510 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83780b0890 0x7f83781473c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:31.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.703+0000 7f8376d9d700 1 --2- 172.21.15.60:0/2998885510 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83780b9bf0 0x7f837813d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:31.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.703+0000 7f837659c700 1 -- 172.21.15.60:0/2998885510 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83780b9bf0 msgr2=0x7f837813d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:31.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.703+0000 7f837659c700 1 --2- 172.21.15.60:0/2998885510 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83780b9bf0 0x7f837813d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:31.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.703+0000 7f837659c700 1 -- 172.21.15.60:0/2998885510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83780b3b40 msgr2=0x7f83780aefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:31.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.703+0000 7f837659c700 1 --2- 172.21.15.60:0/2998885510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83780b3b40 0x7f83780aefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:31.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.703+0000 7f837659c700 1 -- 172.21.15.60:0/2998885510 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f836800b180 con 0x7f83780b0890 2022-06-29T16:03:31.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.703+0000 7f8376d9d700 1 --2- 172.21.15.60:0/2998885510 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83780b9bf0 0x7f837813d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:03:31.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.703+0000 7f8375d9b700 1 --2- 172.21.15.60:0/2998885510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83780b3b40 0x7f83780aefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:03:31.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.703+0000 7f837659c700 1 --2- 172.21.15.60:0/2998885510 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83780b0890 0x7f83781473c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f8368002f70 tx=0x7f8368015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:31.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.703+0000 7f83677fe700 1 -- 172.21.15.60:0/2998885510 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f836800de70 con 0x7f83780b0890 2022-06-29T16:03:31.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.703+0000 7f837759e700 1 -- 172.21.15.60:0/2998885510 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83780ae450 con 0x7f83780b0890 2022-06-29T16:03:31.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.703+0000 7f83677fe700 1 -- 172.21.15.60:0/2998885510 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8368011b00 con 0x7f83780b0890 2022-06-29T16:03:31.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.703+0000 7f83677fe700 1 -- 172.21.15.60:0/2998885510 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8368011e60 con 0x7f83780b0890 2022-06-29T16:03:31.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.703+0000 7f837759e700 1 -- 172.21.15.60:0/2998885510 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83781453a0 con 0x7f83780b0890 2022-06-29T16:03:31.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.703+0000 7f837759e700 1 -- 172.21.15.60:0/2998885510 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83781470e0 con 0x7f83780b0890 2022-06-29T16:03:31.721 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.707+0000 7f83677fe700 1 -- 172.21.15.60:0/2998885510 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f836801a040 con 0x7f83780b0890 2022-06-29T16:03:31.721 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.707+0000 7f83677fe700 1 --2- 172.21.15.60:0/2998885510 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8354053b00 0x7f8354055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:31.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.707+0000 7f83677fe700 1 -- 172.21.15.60:0/2998885510 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f836806d050 con 0x7f83780b0890 2022-06-29T16:03:31.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.707+0000 7f8375d9b700 1 --2- 172.21.15.60:0/2998885510 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8354053b00 0x7f8354055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:31.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.707+0000 7f8375d9b700 1 --2- 172.21.15.60:0/2998885510 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8354053b00 0x7f8354055fc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f836c00d000 tx=0x7f836c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:31.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.715+0000 7f83677fe700 1 -- 172.21.15.60:0/2998885510 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8368043de0 con 0x7f83780b0890 2022-06-29T16:03:31.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.843+0000 7f837759e700 1 -- 172.21.15.60:0/2998885510 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f83780573d0 con 0x7f83780b0890 2022-06-29T16:03:31.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.843+0000 7f83677fe700 1 -- 172.21.15.60:0/2998885510 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v62) v1 ==== 110+0+10 (secure 0 0 0) 0x7f8368047400 con 0x7f83780b0890 2022-06-29T16:03:31.852 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:03:31.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.847+0000 7f837759e700 1 -- 172.21.15.60:0/2998885510 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8354053b00 msgr2=0x7f8354055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:31.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.847+0000 7f837759e700 1 --2- 172.21.15.60:0/2998885510 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8354053b00 0x7f8354055fc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f836c00d000 tx=0x7f836c009040).stop 2022-06-29T16:03:31.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.847+0000 7f837759e700 1 -- 172.21.15.60:0/2998885510 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83780b0890 msgr2=0x7f83781473c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:31.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.847+0000 7f837759e700 1 --2- 172.21.15.60:0/2998885510 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83780b0890 0x7f83781473c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f8368002f70 tx=0x7f8368015000).stop 2022-06-29T16:03:31.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.847+0000 7f837759e700 1 -- 172.21.15.60:0/2998885510 shutdown_connections 2022-06-29T16:03:31.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.847+0000 7f837759e700 1 --2- 172.21.15.60:0/2998885510 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8354053b00 0x7f8354055fc0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:31.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.847+0000 7f837759e700 1 --2- 172.21.15.60:0/2998885510 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83780b9bf0 0x7f837813d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:31.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.847+0000 7f837759e700 1 --2- 172.21.15.60:0/2998885510 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83780b3b40 0x7f83780aefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:31.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.847+0000 7f837759e700 1 --2- 172.21.15.60:0/2998885510 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83780b0890 0x7f83781473c0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:31.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.847+0000 7f837759e700 1 -- 172.21.15.60:0/2998885510 >> 172.21.15.60:0/2998885510 conn(0x7f8378062b60 msgr2=0x7f83780b5bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:31.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.847+0000 7f837759e700 1 -- 172.21.15.60:0/2998885510 shutdown_connections 2022-06-29T16:03:31.859 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.847+0000 7f837759e700 1 -- 172.21.15.60:0/2998885510 wait complete. 2022-06-29T16:03:31.871 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:03:31.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.971+0000 7f350b3fb700 1 Processor -- start 2022-06-29T16:03:31.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.971+0000 7f350b3fb700 1 -- start start 2022-06-29T16:03:31.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.975+0000 7f350b3fb700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f35040b61d0 0x7f35040af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:31.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.975+0000 7f350b3fb700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3504060b60 0x7f3504143180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:31.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.975+0000 7f350b3fb700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35040b65b0 con 0x7f35040adc80 2022-06-29T16:03:31.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.975+0000 7f350b3fb700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35040605e0 con 0x7f35040b61d0 2022-06-29T16:03:31.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.975+0000 7f350b3fb700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3504060880 con 0x7f3504060b60 2022-06-29T16:03:31.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.975+0000 7f3509197700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f35040b61d0 0x7f35040af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:31.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.975+0000 7f3509197700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f35040b61d0 0x7f35040af8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48736/0 (socket says 172.21.15.60:48736) 2022-06-29T16:03:31.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.975+0000 7f3509197700 1 -- 172.21.15.60:0/4065887633 learned_addr learned my addr 172.21.15.60:0/4065887633 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:31.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.975+0000 7f3509998700 1 --2- 172.21.15.60:0/4065887633 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3504060b60 0x7f3504143180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:31.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.975+0000 7f3509197700 1 -- 172.21.15.60:0/4065887633 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3504060b60 msgr2=0x7f3504143180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:31.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.975+0000 7f3509197700 1 --2- 172.21.15.60:0/4065887633 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3504060b60 0x7f3504143180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:31.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.975+0000 7f3509197700 1 -- 172.21.15.60:0/4065887633 >> v1:172.21.15.92:6789/0 conn(0x7f35040adc80 legacy=0x7f3504142a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:31.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.975+0000 7f34f3fff700 1 -- 172.21.15.60:0/4065887633 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806773793 0 0) 0x7f35040b65b0 con 0x7f35040adc80 2022-06-29T16:03:31.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.975+0000 7f3509197700 1 -- 172.21.15.60:0/4065887633 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3504067340 con 0x7f35040b61d0 2022-06-29T16:03:31.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.975+0000 7f3509197700 1 --2- 172.21.15.60:0/4065887633 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35040b61d0 0x7f35040af8d0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0x7f35040ad110 tx=0x7f3500006eb0).ready entity=mon.0 client_cookie=38f181b50b2fdb07 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:31.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.975+0000 7f34f3fff700 1 -- 172.21.15.60:0/4065887633 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f350000d5c0 con 0x7f35040b61d0 2022-06-29T16:03:31.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.975+0000 7f34f3fff700 1 -- 172.21.15.60:0/4065887633 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3500000e90 con 0x7f35040b61d0 2022-06-29T16:03:31.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.975+0000 7f34f3fff700 1 -- 172.21.15.60:0/4065887633 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f350000d910 con 0x7f35040b61d0 2022-06-29T16:03:31.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.975+0000 7f350b3fb700 1 -- 172.21.15.60:0/4065887633 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35040b61d0 msgr2=0x7f35040af8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:31.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.975+0000 7f350b3fb700 1 --2- 172.21.15.60:0/4065887633 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35040b61d0 0x7f35040af8d0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0x7f35040ad110 tx=0x7f3500006eb0).stop 2022-06-29T16:03:31.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.975+0000 7f350b3fb700 1 -- 172.21.15.60:0/4065887633 shutdown_connections 2022-06-29T16:03:31.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.975+0000 7f350b3fb700 1 --2- 172.21.15.60:0/4065887633 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3504060b60 0x7f3504143180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:31.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.975+0000 7f350b3fb700 1 --2- 172.21.15.60:0/4065887633 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35040b61d0 0x7f35040af8d0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:31.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.975+0000 7f350b3fb700 1 -- 172.21.15.60:0/4065887633 >> 172.21.15.60:0/4065887633 conn(0x7f3504063b50 msgr2=0x7f35040b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:31.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.975+0000 7f350b3fb700 1 -- 172.21.15.60:0/4065887633 shutdown_connections 2022-06-29T16:03:31.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.975+0000 7f350b3fb700 1 -- 172.21.15.60:0/4065887633 wait complete. 2022-06-29T16:03:31.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.979+0000 7f350b3fb700 1 Processor -- start 2022-06-29T16:03:31.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.979+0000 7f350b3fb700 1 -- start start 2022-06-29T16:03:31.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.979+0000 7f350b3fb700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3504060b60 0x7f35041447f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:31.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.979+0000 7f350b3fb700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35040adc80 0x7f3504144ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:31.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.979+0000 7f350b3fb700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3504064070 0x7f3504151d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:31.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.979+0000 7f350b3fb700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3504066d10 con 0x7f3504060b60 2022-06-29T16:03:31.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.979+0000 7f350b3fb700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35041452f0 con 0x7f3504064070 2022-06-29T16:03:31.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.979+0000 7f350b3fb700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3504152920 con 0x7f35040adc80 2022-06-29T16:03:31.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.979+0000 7f3509197700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3504060b60 0x7f35041447f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:31.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.979+0000 7f3508996700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35040adc80 0x7f3504144ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:31.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.979+0000 7f3509197700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3504060b60 0x7f35041447f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48742/0 (socket says 172.21.15.60:48742) 2022-06-29T16:03:31.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.979+0000 7f3509998700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3504064070 0x7f3504151d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:31.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.979+0000 7f3509197700 1 -- 172.21.15.60:0/3548778492 learned_addr learned my addr 172.21.15.60:0/3548778492 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:31.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.979+0000 7f3509197700 1 -- 172.21.15.60:0/3548778492 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35040adc80 msgr2=0x7f3504144ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:31.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.979+0000 7f3509197700 1 --2- 172.21.15.60:0/3548778492 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35040adc80 0x7f3504144ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:31.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.979+0000 7f3509197700 1 -- 172.21.15.60:0/3548778492 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3504064070 msgr2=0x7f3504151d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:31.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.979+0000 7f3509197700 1 --2- 172.21.15.60:0/3548778492 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3504064070 0x7f3504151d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:31.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.979+0000 7f3509197700 1 -- 172.21.15.60:0/3548778492 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350000b1e0 con 0x7f3504060b60 2022-06-29T16:03:31.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.979+0000 7f3508996700 1 --2- 172.21.15.60:0/3548778492 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35040adc80 0x7f3504144ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:03:31.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.979+0000 7f3509197700 1 --2- 172.21.15.60:0/3548778492 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3504060b60 0x7f35041447f0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7f35040ad110 tx=0x7f3500015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:31.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.979+0000 7f34f27fc700 1 -- 172.21.15.60:0/3548778492 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3500002f70 con 0x7f3504060b60 2022-06-29T16:03:31.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.979+0000 7f34f27fc700 1 -- 172.21.15.60:0/3548778492 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f350000de40 con 0x7f3504060b60 2022-06-29T16:03:31.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.979+0000 7f350b3fb700 1 -- 172.21.15.60:0/3548778492 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3504152dc0 con 0x7f3504060b60 2022-06-29T16:03:31.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.979+0000 7f34f27fc700 1 -- 172.21.15.60:0/3548778492 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3500011d70 con 0x7f3504060b60 2022-06-29T16:03:31.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.979+0000 7f350b3fb700 1 -- 172.21.15.60:0/3548778492 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3504146ec0 con 0x7f3504060b60 2022-06-29T16:03:31.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.983+0000 7f350b3fb700 1 -- 172.21.15.60:0/3548778492 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35040573d0 con 0x7f3504060b60 2022-06-29T16:03:31.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.983+0000 7f34f27fc700 1 -- 172.21.15.60:0/3548778492 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3500025030 con 0x7f3504060b60 2022-06-29T16:03:31.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.983+0000 7f34f27fc700 1 --2- 172.21.15.60:0/3548778492 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f34e4053a70 0x7f34e4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:31.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.983+0000 7f34f27fc700 1 -- 172.21.15.60:0/3548778492 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f350006cea0 con 0x7f3504060b60 2022-06-29T16:03:32.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.983+0000 7f3508996700 1 --2- 172.21.15.60:0/3548778492 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f34e4053a70 0x7f34e4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:32.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.983+0000 7f3508996700 1 --2- 172.21.15.60:0/3548778492 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f34e4053a70 0x7f34e4055f30 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f34f400d000 tx=0x7f34f4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:32.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:31.991+0000 7f34f27fc700 1 -- 172.21.15.60:0/3548778492 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3500043cb0 con 0x7f3504060b60 2022-06-29T16:03:32.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.115+0000 7f350b3fb700 1 -- 172.21.15.60:0/3548778492 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f3504146ec0 con 0x7f3504060b60 2022-06-29T16:03:32.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.115+0000 7f34f27fc700 1 -- 172.21.15.60:0/3548778492 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v62) v1 ==== 92+0+10 (secure 0 0 0) 0x7f3504146ec0 con 0x7f3504060b60 2022-06-29T16:03:32.123 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:03:32.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.119+0000 7f350b3fb700 1 -- 172.21.15.60:0/3548778492 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f34e4053a70 msgr2=0x7f34e4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:32.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.119+0000 7f350b3fb700 1 --2- 172.21.15.60:0/3548778492 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f34e4053a70 0x7f34e4055f30 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f34f400d000 tx=0x7f34f4009040).stop 2022-06-29T16:03:32.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.119+0000 7f350b3fb700 1 -- 172.21.15.60:0/3548778492 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3504060b60 msgr2=0x7f35041447f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:32.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.119+0000 7f350b3fb700 1 --2- 172.21.15.60:0/3548778492 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3504060b60 0x7f35041447f0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7f35040ad110 tx=0x7f3500015000).stop 2022-06-29T16:03:32.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.119+0000 7f350b3fb700 1 -- 172.21.15.60:0/3548778492 shutdown_connections 2022-06-29T16:03:32.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.119+0000 7f350b3fb700 1 --2- 172.21.15.60:0/3548778492 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f34e4053a70 0x7f34e4055f30 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:32.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.119+0000 7f350b3fb700 1 --2- 172.21.15.60:0/3548778492 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3504064070 0x7f3504151d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:32.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.119+0000 7f350b3fb700 1 --2- 172.21.15.60:0/3548778492 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f35040adc80 0x7f3504144ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:32.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.119+0000 7f350b3fb700 1 --2- 172.21.15.60:0/3548778492 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3504060b60 0x7f35041447f0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:32.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.119+0000 7f350b3fb700 1 -- 172.21.15.60:0/3548778492 >> 172.21.15.60:0/3548778492 conn(0x7f3504063b50 msgr2=0x7f3504142a50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:32.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.119+0000 7f350b3fb700 1 -- 172.21.15.60:0/3548778492 shutdown_connections 2022-06-29T16:03:32.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.119+0000 7f350b3fb700 1 -- 172.21.15.60:0/3548778492 wait complete. 2022-06-29T16:03:32.136 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-29T16:03:32.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.239+0000 7fb31ce5d700 1 Processor -- start 2022-06-29T16:03:32.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.239+0000 7fb31ce5d700 1 -- start start 2022-06-29T16:03:32.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.243+0000 7fb31ce5d700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb3180b5eb0 0x7fb318142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:32.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.243+0000 7fb31ce5d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3180b48b0 0x7fb3180b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:32.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.243+0000 7fb31ce5d700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3180b4d10 con 0x7fb3180b2c50 2022-06-29T16:03:32.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.243+0000 7fb31ce5d700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3180b54e0 con 0x7fb3180b5eb0 2022-06-29T16:03:32.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.243+0000 7fb31ce5d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3180b6290 con 0x7fb3180b48b0 2022-06-29T16:03:32.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.243+0000 7fb317fff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb3180b2c50 0x7fb3180b3090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56836/0 (socket says 172.21.15.60:56836) 2022-06-29T16:03:32.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.243+0000 7fb317fff700 1 -- 172.21.15.60:0/8234155 learned_addr learned my addr 172.21.15.60:0/8234155 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:03:32.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.243+0000 7fb316ffd700 1 --2- 172.21.15.60:0/8234155 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3180b48b0 0x7fb3180b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:32.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.243+0000 7fb3177fe700 1 --2- 172.21.15.60:0/8234155 >> v2:172.21.15.92:3300/0 conn(0x7fb3180b5eb0 0x7fb318142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:32.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.243+0000 7fb3167fc700 1 -- 172.21.15.60:0/8234155 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483529610 0 0) 0x7fb3180b4d10 con 0x7fb3180b2c50 2022-06-29T16:03:32.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.243+0000 7fb3167fc700 1 -- 172.21.15.60:0/8234155 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2f8003260 con 0x7fb3180b2c50 2022-06-29T16:03:32.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.243+0000 7fb3177fe700 1 -- 172.21.15.60:0/8234155 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3180b48b0 msgr2=0x7fb3180b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:32.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.243+0000 7fb3177fe700 1 --2- 172.21.15.60:0/8234155 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3180b48b0 0x7fb3180b05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:32.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.243+0000 7fb3177fe700 1 -- 172.21.15.60:0/8234155 >> v1:172.21.15.60:6789/0 conn(0x7fb3180b2c50 legacy=0x7fb3180b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:32.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.243+0000 7fb3177fe700 1 -- 172.21.15.60:0/8234155 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb318142f10 con 0x7fb3180b5eb0 2022-06-29T16:03:32.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.243+0000 7fb316ffd700 1 --2- 172.21.15.60:0/8234155 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3180b48b0 0x7fb3180b05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:03:32.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.243+0000 7fb3177fe700 1 --2- 172.21.15.60:0/8234155 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3180b5eb0 0x7fb318142a20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fb3080032f0 tx=0x7fb3080091d0).ready entity=mon.1 client_cookie=d89c07df752513c8 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:32.254 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.243+0000 7fb3167fc700 1 -- 172.21.15.60:0/8234155 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb30800b600 con 0x7fb3180b5eb0 2022-06-29T16:03:32.254 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.243+0000 7fb3167fc700 1 -- 172.21.15.60:0/8234155 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb30800b760 con 0x7fb3180b5eb0 2022-06-29T16:03:32.254 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.243+0000 7fb3167fc700 1 -- 172.21.15.60:0/8234155 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb30800b8d0 con 0x7fb3180b5eb0 2022-06-29T16:03:32.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.243+0000 7fb31ce5d700 1 -- 172.21.15.60:0/8234155 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3180b5eb0 msgr2=0x7fb318142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:32.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.243+0000 7fb31ce5d700 1 --2- 172.21.15.60:0/8234155 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3180b5eb0 0x7fb318142a20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fb3080032f0 tx=0x7fb3080091d0).stop 2022-06-29T16:03:32.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.243+0000 7fb31ce5d700 1 -- 172.21.15.60:0/8234155 shutdown_connections 2022-06-29T16:03:32.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.243+0000 7fb31ce5d700 1 --2- 172.21.15.60:0/8234155 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3180b48b0 0x7fb3180b05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:32.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.243+0000 7fb31ce5d700 1 --2- 172.21.15.60:0/8234155 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3180b5eb0 0x7fb318142a20 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:32.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.243+0000 7fb31ce5d700 1 -- 172.21.15.60:0/8234155 >> 172.21.15.60:0/8234155 conn(0x7fb318062090 msgr2=0x7fb318062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:32.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.243+0000 7fb31ce5d700 1 -- 172.21.15.60:0/8234155 shutdown_connections 2022-06-29T16:03:32.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.243+0000 7fb31ce5d700 1 -- 172.21.15.60:0/8234155 wait complete. 2022-06-29T16:03:32.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.247+0000 7fb31ce5d700 1 Processor -- start 2022-06-29T16:03:32.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.247+0000 7fb31ce5d700 1 -- start start 2022-06-29T16:03:32.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.247+0000 7fb31ce5d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3180b2c50 0x7fb318144320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:32.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.247+0000 7fb31ce5d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3180b48b0 0x7fb3180aec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:32.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.247+0000 7fb31ce5d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3180b5eb0 0x7fb318139990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:32.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.247+0000 7fb31ce5d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb318146b20 con 0x7fb3180b2c50 2022-06-29T16:03:32.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.247+0000 7fb31ce5d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb318146dc0 con 0x7fb3180b5eb0 2022-06-29T16:03:32.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.247+0000 7fb31ce5d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb318146810 con 0x7fb3180b48b0 2022-06-29T16:03:32.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.247+0000 7fb3177fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3180b2c50 0x7fb318144320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:32.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.247+0000 7fb3177fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3180b2c50 0x7fb318144320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48758/0 (socket says 172.21.15.60:48758) 2022-06-29T16:03:32.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.247+0000 7fb3177fe700 1 -- 172.21.15.60:0/1162499357 learned_addr learned my addr 172.21.15.60:0/1162499357 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:32.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.247+0000 7fb316ffd700 1 --2- 172.21.15.60:0/1162499357 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3180b48b0 0x7fb3180aec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:32.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.247+0000 7fb317fff700 1 --2- 172.21.15.60:0/1162499357 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3180b5eb0 0x7fb318139990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:32.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.247+0000 7fb316ffd700 1 -- 172.21.15.60:0/1162499357 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3180b5eb0 msgr2=0x7fb318139990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:32.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.247+0000 7fb316ffd700 1 --2- 172.21.15.60:0/1162499357 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3180b5eb0 0x7fb318139990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:32.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.247+0000 7fb316ffd700 1 -- 172.21.15.60:0/1162499357 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3180b2c50 msgr2=0x7fb318144320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:32.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.247+0000 7fb316ffd700 1 --2- 172.21.15.60:0/1162499357 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3180b2c50 0x7fb318144320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:32.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.247+0000 7fb316ffd700 1 -- 172.21.15.60:0/1162499357 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb30000b180 con 0x7fb3180b48b0 2022-06-29T16:03:32.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.247+0000 7fb3177fe700 1 --2- 172.21.15.60:0/1162499357 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3180b2c50 0x7fb318144320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:03:32.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.247+0000 7fb316ffd700 1 --2- 172.21.15.60:0/1162499357 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3180b48b0 0x7fb3180aec30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fb30000c000 tx=0x7fb300006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:32.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.247+0000 7fb314ff9700 1 -- 172.21.15.60:0/1162499357 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb300010780 con 0x7fb3180b48b0 2022-06-29T16:03:32.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.247+0000 7fb314ff9700 1 -- 172.21.15.60:0/1162499357 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3000108e0 con 0x7fb3180b48b0 2022-06-29T16:03:32.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.247+0000 7fb31ce5d700 1 -- 172.21.15.60:0/1162499357 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb308009470 con 0x7fb3180b48b0 2022-06-29T16:03:32.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.247+0000 7fb314ff9700 1 -- 172.21.15.60:0/1162499357 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb300010c40 con 0x7fb3180b48b0 2022-06-29T16:03:32.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.247+0000 7fb31ce5d700 1 -- 172.21.15.60:0/1162499357 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb318149f50 con 0x7fb3180b48b0 2022-06-29T16:03:32.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.251+0000 7fb31ce5d700 1 -- 172.21.15.60:0/1162499357 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3180573d0 con 0x7fb3180b48b0 2022-06-29T16:03:32.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.251+0000 7fb314ff9700 1 -- 172.21.15.60:0/1162499357 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb30001c460 con 0x7fb3180b48b0 2022-06-29T16:03:32.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.251+0000 7fb314ff9700 1 --2- 172.21.15.60:0/1162499357 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb2fc053a70 0x7fb2fc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:32.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.251+0000 7fb314ff9700 1 -- 172.21.15.60:0/1162499357 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb30006c320 con 0x7fb3180b48b0 2022-06-29T16:03:32.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.251+0000 7fb3177fe700 1 --2- 172.21.15.60:0/1162499357 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb2fc053a70 0x7fb2fc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:32.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.255+0000 7fb3177fe700 1 --2- 172.21.15.60:0/1162499357 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb2fc053a70 0x7fb2fc055f30 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fb30801b000 tx=0x7fb308016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:32.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.263+0000 7fb314ff9700 1 -- 172.21.15.60:0/1162499357 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb3000431e0 con 0x7fb3180b48b0 2022-06-29T16:03:32.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.391+0000 7fb31ce5d700 1 -- 172.21.15.60:0/1162499357 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb318067a30 con 0x7fb3180b48b0 2022-06-29T16:03:32.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.391+0000 7fb314ff9700 1 -- 172.21.15.60:0/1162499357 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v62) v1 ==== 104+0+10 (secure 0 0 0) 0x7fb300046800 con 0x7fb3180b48b0 2022-06-29T16:03:32.397 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:03:32.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.395+0000 7fb31ce5d700 1 -- 172.21.15.60:0/1162499357 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb2fc053a70 msgr2=0x7fb2fc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:32.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.395+0000 7fb31ce5d700 1 --2- 172.21.15.60:0/1162499357 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb2fc053a70 0x7fb2fc055f30 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fb30801b000 tx=0x7fb308016000).stop 2022-06-29T16:03:32.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.395+0000 7fb31ce5d700 1 -- 172.21.15.60:0/1162499357 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3180b48b0 msgr2=0x7fb3180aec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:32.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.395+0000 7fb31ce5d700 1 --2- 172.21.15.60:0/1162499357 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3180b48b0 0x7fb3180aec30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fb30000c000 tx=0x7fb300006df0).stop 2022-06-29T16:03:32.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.395+0000 7fb31ce5d700 1 -- 172.21.15.60:0/1162499357 shutdown_connections 2022-06-29T16:03:32.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.395+0000 7fb31ce5d700 1 --2- 172.21.15.60:0/1162499357 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb2fc053a70 0x7fb2fc055f30 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:32.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.395+0000 7fb31ce5d700 1 --2- 172.21.15.60:0/1162499357 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3180b5eb0 0x7fb318139990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:32.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.395+0000 7fb31ce5d700 1 --2- 172.21.15.60:0/1162499357 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3180b48b0 0x7fb3180aec30 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:32.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.395+0000 7fb31ce5d700 1 --2- 172.21.15.60:0/1162499357 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3180b2c50 0x7fb318144320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:32.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.395+0000 7fb31ce5d700 1 -- 172.21.15.60:0/1162499357 >> 172.21.15.60:0/1162499357 conn(0x7fb318062090 msgr2=0x7fb3180b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:32.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.395+0000 7fb31ce5d700 1 -- 172.21.15.60:0/1162499357 shutdown_connections 2022-06-29T16:03:32.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.395+0000 7fb31ce5d700 1 -- 172.21.15.60:0/1162499357 wait complete. 2022-06-29T16:03:32.417 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-29T16:03:32.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.523+0000 7fedecfc9700 1 Processor -- start 2022-06-29T16:03:32.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.523+0000 7fedecfc9700 1 -- start start 2022-06-29T16:03:32.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.523+0000 7fedecfc9700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fede80aca20 0x7fede80ace00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:32.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.523+0000 7fedecfc9700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fede80b24f0 0x7fede81468c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:32.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.523+0000 7fedecfc9700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fede8146db0 0x7fede8065cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:32.539 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.523+0000 7fedecfc9700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fede8143da0 con 0x7fede80aca20 2022-06-29T16:03:32.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.523+0000 7fedecfc9700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fede80adf00 con 0x7fede80b24f0 2022-06-29T16:03:32.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.523+0000 7fedecfc9700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede805cf40 con 0x7fede8146db0 2022-06-29T16:03:32.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.523+0000 7fede659c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fede80aca20 0x7fede80ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:32.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.523+0000 7fede659c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fede80aca20 0x7fede80ace00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48762/0 (socket says 172.21.15.60:48762) 2022-06-29T16:03:32.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.523+0000 7fede659c700 1 -- 172.21.15.60:0/4166694380 learned_addr learned my addr 172.21.15.60:0/4166694380 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:32.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.523+0000 7fede6d9d700 1 --2- 172.21.15.60:0/4166694380 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fede8146db0 0x7fede8065cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:32.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.523+0000 7fede5d9b700 1 --2- 172.21.15.60:0/4166694380 >> v2:172.21.15.92:3300/0 conn(0x7fede80b24f0 0x7fede81468c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:32.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.527+0000 7fede659c700 1 -- 172.21.15.60:0/4166694380 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fede8146db0 msgr2=0x7fede8065cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:32.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.527+0000 7fede659c700 1 --2- 172.21.15.60:0/4166694380 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fede8146db0 0x7fede8065cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:32.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.527+0000 7fede659c700 1 -- 172.21.15.60:0/4166694380 >> v2:172.21.15.92:3300/0 conn(0x7fede80b24f0 msgr2=0x7fede81468c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:32.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.527+0000 7fede659c700 1 --2- 172.21.15.60:0/4166694380 >> v2:172.21.15.92:3300/0 conn(0x7fede80b24f0 0x7fede81468c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:32.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.527+0000 7fede659c700 1 -- 172.21.15.60:0/4166694380 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede805d1e0 con 0x7fede80aca20 2022-06-29T16:03:32.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.527+0000 7fede659c700 1 --2- 172.21.15.60:0/4166694380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fede80aca20 0x7fede80ace00 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7fedd8002f70 tx=0x7fedd8006eb0).ready entity=mon.0 client_cookie=b32b93d0173eef19 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:32.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.527+0000 7fede559a700 1 -- 172.21.15.60:0/4166694380 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fedd800d570 con 0x7fede80aca20 2022-06-29T16:03:32.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.527+0000 7fede559a700 1 -- 172.21.15.60:0/4166694380 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd8000e90 con 0x7fede80aca20 2022-06-29T16:03:32.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.527+0000 7fede559a700 1 -- 172.21.15.60:0/4166694380 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fedd800d900 con 0x7fede80aca20 2022-06-29T16:03:32.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.527+0000 7fedecfc9700 1 -- 172.21.15.60:0/4166694380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fede80aca20 msgr2=0x7fede80ace00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:32.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.527+0000 7fedecfc9700 1 --2- 172.21.15.60:0/4166694380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fede80aca20 0x7fede80ace00 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7fedd8002f70 tx=0x7fedd8006eb0).stop 2022-06-29T16:03:32.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.527+0000 7fedecfc9700 1 -- 172.21.15.60:0/4166694380 shutdown_connections 2022-06-29T16:03:32.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.527+0000 7fedecfc9700 1 --2- 172.21.15.60:0/4166694380 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fede8146db0 0x7fede8065cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:32.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.527+0000 7fedecfc9700 1 --2- 172.21.15.60:0/4166694380 >> v2:172.21.15.92:3300/0 conn(0x7fede80b24f0 0x7fede81468c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:32.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.527+0000 7fedecfc9700 1 --2- 172.21.15.60:0/4166694380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fede80aca20 0x7fede80ace00 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:32.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.527+0000 7fedecfc9700 1 -- 172.21.15.60:0/4166694380 >> 172.21.15.60:0/4166694380 conn(0x7fede80b7140 msgr2=0x7fede80b7520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:32.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.527+0000 7fedecfc9700 1 -- 172.21.15.60:0/4166694380 shutdown_connections 2022-06-29T16:03:32.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.527+0000 7fedecfc9700 1 -- 172.21.15.60:0/4166694380 wait complete. 2022-06-29T16:03:32.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.531+0000 7fedecfc9700 1 Processor -- start 2022-06-29T16:03:32.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.531+0000 7fedecfc9700 1 -- start start 2022-06-29T16:03:32.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.531+0000 7fedecfc9700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fede80aca20 0x7fede8144650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:32.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.531+0000 7fedecfc9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fede80b24f0 0x7fede8152060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:32.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.531+0000 7fedecfc9700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fede8146db0 0x7fede8152550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:32.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.531+0000 7fedecfc9700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede8151c30 con 0x7fede80b24f0 2022-06-29T16:03:32.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.531+0000 7fedecfc9700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede813d420 con 0x7fede8146db0 2022-06-29T16:03:32.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.531+0000 7fedecfc9700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede813d6c0 con 0x7fede80aca20 2022-06-29T16:03:32.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.531+0000 7fede5d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fede80b24f0 0x7fede8152060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:32.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.531+0000 7fede5d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fede80b24f0 0x7fede8152060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48770/0 (socket says 172.21.15.60:48770) 2022-06-29T16:03:32.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.531+0000 7fede5d9b700 1 -- 172.21.15.60:0/427127962 learned_addr learned my addr 172.21.15.60:0/427127962 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:32.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.531+0000 7fede659c700 1 --2- 172.21.15.60:0/427127962 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fede80aca20 0x7fede8144650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:32.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.531+0000 7fede6d9d700 1 --2- 172.21.15.60:0/427127962 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fede8146db0 0x7fede8152550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:32.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.531+0000 7fede5d9b700 1 -- 172.21.15.60:0/427127962 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fede80aca20 msgr2=0x7fede8144650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:32.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.531+0000 7fede5d9b700 1 --2- 172.21.15.60:0/427127962 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fede80aca20 0x7fede8144650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:32.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.531+0000 7fede5d9b700 1 -- 172.21.15.60:0/427127962 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fede8146db0 msgr2=0x7fede8152550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:32.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.531+0000 7fede5d9b700 1 --2- 172.21.15.60:0/427127962 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fede8146db0 0x7fede8152550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:32.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.531+0000 7fede5d9b700 1 -- 172.21.15.60:0/427127962 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd800b1e0 con 0x7fede80b24f0 2022-06-29T16:03:32.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.531+0000 7fede5d9b700 1 --2- 172.21.15.60:0/427127962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fede80b24f0 0x7fede8152060 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7feddc00c000 tx=0x7feddc006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:32.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.531+0000 7fedd77fe700 1 -- 172.21.15.60:0/427127962 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feddc010780 con 0x7fede80b24f0 2022-06-29T16:03:32.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.531+0000 7fedd77fe700 1 -- 172.21.15.60:0/427127962 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feddc0108e0 con 0x7fede80b24f0 2022-06-29T16:03:32.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.531+0000 7fedecfc9700 1 -- 172.21.15.60:0/427127962 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fede813d960 con 0x7fede80b24f0 2022-06-29T16:03:32.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.531+0000 7fedd77fe700 1 -- 172.21.15.60:0/427127962 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feddc010c40 con 0x7fede80b24f0 2022-06-29T16:03:32.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.531+0000 7fedecfc9700 1 -- 172.21.15.60:0/427127962 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fede8155df0 con 0x7fede80b24f0 2022-06-29T16:03:32.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.535+0000 7fedecfc9700 1 -- 172.21.15.60:0/427127962 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fede813daf0 con 0x7fede80b24f0 2022-06-29T16:03:32.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.535+0000 7fedd77fe700 1 -- 172.21.15.60:0/427127962 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7feddc01c460 con 0x7fede80b24f0 2022-06-29T16:03:32.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.535+0000 7fedd77fe700 1 --2- 172.21.15.60:0/427127962 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fedc40539e0 0x7fedc4055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:32.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.535+0000 7fedd77fe700 1 -- 172.21.15.60:0/427127962 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7feddc06c4a0 con 0x7fede80b24f0 2022-06-29T16:03:32.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.535+0000 7fede659c700 1 --2- 172.21.15.60:0/427127962 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fedc40539e0 0x7fedc4055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:32.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.535+0000 7fede659c700 1 --2- 172.21.15.60:0/427127962 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fedc40539e0 0x7fedc4055ea0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7fedd801c000 tx=0x7fedd8015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:32.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.543+0000 7fedd77fe700 1 -- 172.21.15.60:0/427127962 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7feddc0431d0 con 0x7fede80b24f0 2022-06-29T16:03:32.677 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.671+0000 7fedecfc9700 1 -- 172.21.15.60:0/427127962 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fede80573d0 con 0x7fede80b24f0 2022-06-29T16:03:32.678 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.671+0000 7fedd77fe700 1 -- 172.21.15.60:0/427127962 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v62) v1 ==== 100+0+10 (secure 0 0 0) 0x7feddc0467f0 con 0x7fede80b24f0 2022-06-29T16:03:32.679 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:03:32.680 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.675+0000 7fedecfc9700 1 -- 172.21.15.60:0/427127962 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fedc40539e0 msgr2=0x7fedc4055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:32.680 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.675+0000 7fedecfc9700 1 --2- 172.21.15.60:0/427127962 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fedc40539e0 0x7fedc4055ea0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7fedd801c000 tx=0x7fedd8015000).stop 2022-06-29T16:03:32.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.675+0000 7fedecfc9700 1 -- 172.21.15.60:0/427127962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fede80b24f0 msgr2=0x7fede8152060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:32.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.675+0000 7fedecfc9700 1 --2- 172.21.15.60:0/427127962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fede80b24f0 0x7fede8152060 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7feddc00c000 tx=0x7feddc006df0).stop 2022-06-29T16:03:32.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.675+0000 7fedecfc9700 1 -- 172.21.15.60:0/427127962 shutdown_connections 2022-06-29T16:03:32.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.675+0000 7fedecfc9700 1 --2- 172.21.15.60:0/427127962 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fedc40539e0 0x7fedc4055ea0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:32.682 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.675+0000 7fedecfc9700 1 --2- 172.21.15.60:0/427127962 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fede8146db0 0x7fede8152550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:32.682 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.675+0000 7fedecfc9700 1 --2- 172.21.15.60:0/427127962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fede80b24f0 0x7fede8152060 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:32.682 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.675+0000 7fedecfc9700 1 --2- 172.21.15.60:0/427127962 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fede80aca20 0x7fede8144650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:32.682 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.675+0000 7fedecfc9700 1 -- 172.21.15.60:0/427127962 >> 172.21.15.60:0/427127962 conn(0x7fede80b7140 msgr2=0x7fede80b5040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:32.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.675+0000 7fedecfc9700 1 -- 172.21.15.60:0/427127962 shutdown_connections 2022-06-29T16:03:32.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.675+0000 7fedecfc9700 1 -- 172.21.15.60:0/427127962 wait complete. 2022-06-29T16:03:32.691 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-06-29T16:03:32.692 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:03:32.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.791+0000 7f9fd331f700 1 Processor -- start 2022-06-29T16:03:32.797 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.791+0000 7f9fd331f700 1 -- start start 2022-06-29T16:03:32.800 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.791+0000 7f9fd331f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9fcc0b2130 0x7f9fcc146100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:32.801 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.791+0000 7f9fd331f700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9fcc0aed00 0x7f9fcc1465f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:32.801 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.791+0000 7f9fd331f700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fcc05d450 con 0x7f9fcc05d070 2022-06-29T16:03:32.801 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.791+0000 7f9fd331f700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fcc0adf90 con 0x7f9fcc0aed00 2022-06-29T16:03:32.802 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.791+0000 7f9fd331f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fcc0b25d0 con 0x7f9fcc0b2130 2022-06-29T16:03:32.802 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.791+0000 7f9fd231d700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9fcc05d070 0x7f9fcc14aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56862/0 (socket says 172.21.15.60:56862) 2022-06-29T16:03:32.802 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.791+0000 7f9fd231d700 1 -- 172.21.15.60:0/3410025648 learned_addr learned my addr 172.21.15.60:0/3410025648 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:03:32.802 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.791+0000 7f9fd2b1e700 1 --2- 172.21.15.60:0/3410025648 >> v2:172.21.15.60:3300/0 conn(0x7f9fcc0aed00 0x7f9fcc1465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:32.803 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.791+0000 7f9fd1b1c700 1 --2- 172.21.15.60:0/3410025648 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9fcc0b2130 0x7f9fcc146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:32.803 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.791+0000 7f9fd131b700 1 -- 172.21.15.60:0/3410025648 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 932968890 0 0) 0x7f9fcc05d450 con 0x7f9fcc05d070 2022-06-29T16:03:32.803 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.791+0000 7f9fd131b700 1 -- 172.21.15.60:0/3410025648 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fb8003260 con 0x7f9fcc05d070 2022-06-29T16:03:32.803 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.791+0000 7f9fd2b1e700 1 -- 172.21.15.60:0/3410025648 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9fcc0b2130 msgr2=0x7f9fcc146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:32.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.791+0000 7f9fd2b1e700 1 --2- 172.21.15.60:0/3410025648 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9fcc0b2130 0x7f9fcc146100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:32.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.791+0000 7f9fd131b700 1 -- 172.21.15.60:0/3410025648 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4034823534 0 0) 0x7f9fb8003260 con 0x7f9fcc05d070 2022-06-29T16:03:32.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.791+0000 7f9fd2b1e700 1 -- 172.21.15.60:0/3410025648 >> v1:172.21.15.60:6789/0 conn(0x7f9fcc05d070 legacy=0x7f9fcc14aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:32.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.791+0000 7f9fd2b1e700 1 -- 172.21.15.60:0/3410025648 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fcc14a350 con 0x7f9fcc0aed00 2022-06-29T16:03:32.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.791+0000 7f9fd2b1e700 1 --2- 172.21.15.60:0/3410025648 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fcc0aed00 0x7f9fcc1465f0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x7f9fc8001820 tx=0x7f9fbc006df0).ready entity=mon.0 client_cookie=aa6e98d7dc317273 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:32.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.791+0000 7f9fd131b700 1 -- 172.21.15.60:0/3410025648 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9fc8001680 con 0x7f9fcc0aed00 2022-06-29T16:03:32.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.791+0000 7f9fd131b700 1 -- 172.21.15.60:0/3410025648 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fbc000e90 con 0x7f9fcc0aed00 2022-06-29T16:03:32.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.791+0000 7f9fd131b700 1 -- 172.21.15.60:0/3410025648 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9fbc012460 con 0x7f9fcc0aed00 2022-06-29T16:03:32.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.795+0000 7f9fd331f700 1 -- 172.21.15.60:0/3410025648 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fcc0aed00 msgr2=0x7f9fcc1465f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:32.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.795+0000 7f9fd331f700 1 --2- 172.21.15.60:0/3410025648 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fcc0aed00 0x7f9fcc1465f0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x7f9fc8001820 tx=0x7f9fbc006df0).stop 2022-06-29T16:03:32.807 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.795+0000 7f9fd331f700 1 -- 172.21.15.60:0/3410025648 shutdown_connections 2022-06-29T16:03:32.807 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.795+0000 7f9fd331f700 1 --2- 172.21.15.60:0/3410025648 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fcc0aed00 0x7f9fcc1465f0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:32.807 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.795+0000 7f9fd331f700 1 --2- 172.21.15.60:0/3410025648 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9fcc0b2130 0x7f9fcc146100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:32.807 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.795+0000 7f9fd331f700 1 -- 172.21.15.60:0/3410025648 >> 172.21.15.60:0/3410025648 conn(0x7f9fcc0b5890 msgr2=0x7f9fcc0b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:32.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.795+0000 7f9fd331f700 1 -- 172.21.15.60:0/3410025648 shutdown_connections 2022-06-29T16:03:32.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.795+0000 7f9fd331f700 1 -- 172.21.15.60:0/3410025648 wait complete. 2022-06-29T16:03:32.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.795+0000 7f9fd331f700 1 Processor -- start 2022-06-29T16:03:32.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.795+0000 7f9fd331f700 1 -- start start 2022-06-29T16:03:32.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.795+0000 7f9fd331f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9fcc05d070 0x7f9fcc14bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:32.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.795+0000 7f9fd331f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9fcc0aed00 0x7f9fcc14e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:32.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.795+0000 7f9fd331f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fcc0b2130 0x7f9fcc156090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:32.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.795+0000 7f9fd331f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fcc134a90 con 0x7f9fcc0b2130 2022-06-29T16:03:32.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.795+0000 7f9fd331f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fcc14b010 con 0x7f9fcc05d070 2022-06-29T16:03:32.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.795+0000 7f9fd331f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fcc14b2b0 con 0x7f9fcc0aed00 2022-06-29T16:03:32.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.799+0000 7f9fd2b1e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fcc0b2130 0x7f9fcc156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:32.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.799+0000 7f9fd2b1e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fcc0b2130 0x7f9fcc156090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48786/0 (socket says 172.21.15.60:48786) 2022-06-29T16:03:32.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.799+0000 7f9fd2b1e700 1 -- 172.21.15.60:0/2445357722 learned_addr learned my addr 172.21.15.60:0/2445357722 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:32.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.799+0000 7f9fd231d700 1 --2- 172.21.15.60:0/2445357722 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9fcc05d070 0x7f9fcc14bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:32.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.799+0000 7f9fd1b1c700 1 --2- 172.21.15.60:0/2445357722 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9fcc0aed00 0x7f9fcc14e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:32.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.799+0000 7f9fd2b1e700 1 -- 172.21.15.60:0/2445357722 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9fcc0aed00 msgr2=0x7f9fcc14e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:32.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.799+0000 7f9fd2b1e700 1 --2- 172.21.15.60:0/2445357722 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9fcc0aed00 0x7f9fcc14e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:32.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.799+0000 7f9fd2b1e700 1 -- 172.21.15.60:0/2445357722 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9fcc05d070 msgr2=0x7f9fcc14bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:32.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.799+0000 7f9fd2b1e700 1 --2- 172.21.15.60:0/2445357722 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9fcc05d070 0x7f9fcc14bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:32.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.799+0000 7f9fd2b1e700 1 -- 172.21.15.60:0/2445357722 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fbc00b180 con 0x7f9fcc0b2130 2022-06-29T16:03:32.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.799+0000 7f9fd2b1e700 1 --2- 172.21.15.60:0/2445357722 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fcc0b2130 0x7f9fcc156090 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0x7f9fc8001820 tx=0x7f9fbc014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:32.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.799+0000 7f9fc37fe700 1 -- 172.21.15.60:0/2445357722 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9fbc016070 con 0x7f9fcc0b2130 2022-06-29T16:03:32.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.799+0000 7f9fc37fe700 1 -- 172.21.15.60:0/2445357722 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fbc00d5e0 con 0x7f9fcc0b2130 2022-06-29T16:03:32.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.799+0000 7f9fd331f700 1 -- 172.21.15.60:0/2445357722 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fcc134ec0 con 0x7f9fcc0b2130 2022-06-29T16:03:32.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.799+0000 7f9fc37fe700 1 -- 172.21.15.60:0/2445357722 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9fbc00d980 con 0x7f9fcc0b2130 2022-06-29T16:03:32.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.799+0000 7f9fd331f700 1 -- 172.21.15.60:0/2445357722 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fcc14a9b0 con 0x7f9fcc0b2130 2022-06-29T16:03:32.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.799+0000 7f9fd331f700 1 -- 172.21.15.60:0/2445357722 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fcc14a5f0 con 0x7f9fcc0b2130 2022-06-29T16:03:32.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.799+0000 7f9fc37fe700 1 -- 172.21.15.60:0/2445357722 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9fbc025030 con 0x7f9fcc0b2130 2022-06-29T16:03:32.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.803+0000 7f9fc37fe700 1 --2- 172.21.15.60:0/2445357722 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9fac053a10 0x7f9fac055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:32.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.803+0000 7f9fc37fe700 1 -- 172.21.15.60:0/2445357722 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9fbc06cea0 con 0x7f9fcc0b2130 2022-06-29T16:03:32.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.803+0000 7f9fd231d700 1 --2- 172.21.15.60:0/2445357722 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9fac053a10 0x7f9fac055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:32.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.803+0000 7f9fd231d700 1 --2- 172.21.15.60:0/2445357722 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9fac053a10 0x7f9fac055ed0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f9fc800d000 tx=0x7f9fc8001de0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:32.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.807+0000 7f9fc37fe700 1 -- 172.21.15.60:0/2445357722 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9fbc043cb0 con 0x7f9fcc0b2130 2022-06-29T16:03:32.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.967+0000 7f9fd331f700 1 -- 172.21.15.60:0/2445357722 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9fcc0573d0 con 0x7f9fcc0b2130 2022-06-29T16:03:32.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.967+0000 7f9fc37fe700 1 -- 172.21.15.60:0/2445357722 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f9fbc023070 con 0x7f9fcc0b2130 2022-06-29T16:03:32.974 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:03:32.975 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":25,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5955,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.60:6833/66690512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":66690512},{"type":"v1","addr":"172.21.15.60:6833","nonce":66690512}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":23}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:03:28.112667+0000","modified":"2022-06-29T16:03:29.115727+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6013},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6013":{"gid":6013,"name":"b","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/3876875590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3876875590},{"type":"v1","addr":"172.21.15.92:6835","nonce":3876875590}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-06-29T16:03:32.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.967+0000 7f9fd331f700 1 -- 172.21.15.60:0/2445357722 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9fac053a10 msgr2=0x7f9fac055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:32.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.967+0000 7f9fd331f700 1 --2- 172.21.15.60:0/2445357722 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9fac053a10 0x7f9fac055ed0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f9fc800d000 tx=0x7f9fc8001de0).stop 2022-06-29T16:03:32.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.967+0000 7f9fd331f700 1 -- 172.21.15.60:0/2445357722 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fcc0b2130 msgr2=0x7f9fcc156090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:32.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.967+0000 7f9fd331f700 1 --2- 172.21.15.60:0/2445357722 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fcc0b2130 0x7f9fcc156090 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0x7f9fc8001820 tx=0x7f9fbc014000).stop 2022-06-29T16:03:32.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.967+0000 7f9fd331f700 1 -- 172.21.15.60:0/2445357722 shutdown_connections 2022-06-29T16:03:32.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.967+0000 7f9fd331f700 1 --2- 172.21.15.60:0/2445357722 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9fac053a10 0x7f9fac055ed0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:32.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.967+0000 7f9fd331f700 1 --2- 172.21.15.60:0/2445357722 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fcc0b2130 0x7f9fcc156090 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:32.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.967+0000 7f9fd331f700 1 --2- 172.21.15.60:0/2445357722 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9fcc0aed00 0x7f9fcc14e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:32.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.967+0000 7f9fd331f700 1 --2- 172.21.15.60:0/2445357722 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9fcc05d070 0x7f9fcc14bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:32.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.967+0000 7f9fd331f700 1 -- 172.21.15.60:0/2445357722 >> 172.21.15.60:0/2445357722 conn(0x7f9fcc0b5890 msgr2=0x7f9fcc144420 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:32.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.967+0000 7f9fd331f700 1 -- 172.21.15.60:0/2445357722 shutdown_connections 2022-06-29T16:03:32.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:32.967+0000 7f9fd331f700 1 -- 172.21.15.60:0/2445357722 wait complete. 2022-06-29T16:03:32.982 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 25 2022-06-29T16:03:33.189 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:03:28.112667+0000', 'modified': '2022-06-29T16:03:29.115727+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6013}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6013': {'gid': 6013, 'name': 'b', 'rank': 0, 'incarnation': 24, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.92:6835/3876875590', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 3876875590}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 3876875590}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:03:33.189 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:03:33.190 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-06-29T16:03:33.306 INFO:teuthology.orchestra.run.smithi092.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":6013,"want_state":"up:active","state":"up:active","rank_uptime":5.177686316,"mdsmap_epoch":25,"osdmap_epoch":62,"osdmap_epoch_barrier":61,"uptime":9.5519333480000004} 2022-06-29T16:03:33.307 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 6013, "mdsmap_epoch": 25, "osdmap_epoch": 62, "osdmap_epoch_barrier": 61, "rank_uptime": 5.177686316, "state": "up:active", "uptime": 9.551933348, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:03:33.307 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-06-29T16:03:33.308 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-06-29T16:03:33.308 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi121.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-06-29T16:03:33.309 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:03:33.310 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:03:33.318 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:03:33.376 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:03:33.376 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:03:33.387 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:03:33.387 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:03:33.389 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:03:33.389 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:03:33.394 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-06-29T16:03:33.395 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:03:33.397 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:03:33.414 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:03:33.415 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:03:33.471 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:03:33.472 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:03:33.473 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:03:33.474 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:03:33.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:2022-06-29T16:03:33.471+0000 7eff0f585080 -1 init, newargv = 0x55c1a02ba1e0 newargc=9ceph-fuse[12777]: starting ceph client 2022-06-29T16:03:33.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr: 2022-06-29T16:03:33.482 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[12777]: starting fuse 2022-06-29T16:03:34.583 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:03:34.602 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:03:34.603 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:03:34.658 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:03:34.659 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:03:34.660 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:03:34.661 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:03:34.669 INFO:teuthology.orchestra.run.smithi121.stdout:52 2022-06-29T16:03:34.669 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-06-29T16:03:34.670 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:03:34.671 DEBUG:teuthology.orchestra.run.smithi121:> sudo python3 -c ' 2022-06-29T16:03:34.671 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2022-06-29T16:03:34.672 DEBUG:teuthology.orchestra.run.smithi121:> import re 2022-06-29T16:03:34.672 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-06-29T16:03:34.672 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2022-06-29T16:03:34.673 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:03:34.673 DEBUG:teuthology.orchestra.run.smithi121:> def find_socket(client_name): 2022-06-29T16:03:34.674 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-06-29T16:03:34.674 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2022-06-29T16:03:34.675 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:03:34.675 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2022-06-29T16:03:34.676 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2022-06-29T16:03:34.676 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2022-06-29T16:03:34.676 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2022-06-29T16:03:34.677 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:03:34.677 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2022-06-29T16:03:34.678 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-06-29T16:03:34.678 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2022-06-29T16:03:34.679 DEBUG:teuthology.orchestra.run.smithi121:> return f 2022-06-29T16:03:34.679 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-06-29T16:03:34.679 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:03:34.680 DEBUG:teuthology.orchestra.run.smithi121:> print(find_socket("client.0")) 2022-06-29T16:03:34.680 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-06-29T16:03:34.757 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.0.12777.asok 2022-06-29T16:03:34.762 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.12777.asok 2022-06-29T16:03:34.762 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:03:34.763 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.12777.asok status 2022-06-29T16:03:34.854 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2022-06-29T16:03:34.854 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2022-06-29T16:03:34.855 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "f25ffac72e5a9e63860ff7deb6767a11f0559a16", 2022-06-29T16:03:34.855 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 15.2.16-304-gf25ffac7 (f25ffac72e5a9e63860ff7deb6767a11f0559a16) octopus (stable)", 2022-06-29T16:03:34.856 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "0", 2022-06-29T16:03:34.856 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2022-06-29T16:03:34.857 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-06-29T16:03:34.857 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "12777", 2022-06-29T16:03:34.858 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2022-06-29T16:03:34.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:03:34.859 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2022-06-29T16:03:34.859 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2022-06-29T16:03:34.859 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 6175, 2022-06-29T16:03:34.860 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2022-06-29T16:03:34.860 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2022-06-29T16:03:34.861 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2022-06-29T16:03:34.861 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 6175 2022-06-29T16:03:34.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:03:34.862 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-06-29T16:03:34.862 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-06-29T16:03:34.862 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-06-29T16:03:34.863 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 358791107 2022-06-29T16:03:34.863 INFO:teuthology.orchestra.run.smithi121.stdout: } 2022-06-29T16:03:34.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:03:34.863 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-06-29T16:03:34.864 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-06-29T16:03:34.864 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-06-29T16:03:34.864 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 358791107 2022-06-29T16:03:34.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:03:34.865 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.6175 172.21.15.121:0/358791107", 2022-06-29T16:03:34.865 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "172.21.15.121:0/358791107", 2022-06-29T16:03:34.865 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2022-06-29T16:03:34.865 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 25, 2022-06-29T16:03:34.866 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 62, 2022-06-29T16:03:34.866 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2022-06-29T16:03:34.866 INFO:teuthology.orchestra.run.smithi121.stdout: "blacklisted": false, 2022-06-29T16:03:34.866 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2022-06-29T16:03:34.866 INFO:teuthology.orchestra.run.smithi121.stdout:} 2022-06-29T16:03:34.868 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:03:34.868 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:03:34.869 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:03:34.876 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T16:03:34.878 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T16:03:34.878 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:03:34.879 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:03:34.936 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:03:34.937 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:03:34.938 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:03:34.946 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T16:03:34.947 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T16:03:34.948 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-06-29T16:03:34.948 DEBUG:teuthology.orchestra.run.smithi121:> import sys 2022-06-29T16:03:34.949 DEBUG:teuthology.orchestra.run.smithi121:> import time 2022-06-29T16:03:34.949 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-06-29T16:03:34.949 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:03:34.950 DEBUG:teuthology.orchestra.run.smithi121:> n = 1000 2022-06-29T16:03:34.950 DEBUG:teuthology.orchestra.run.smithi121:> abs_path = "/home/ubuntu/cephtest/mnt.0/dc-dir/dc-file" 2022-06-29T16:03:34.951 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:03:34.951 DEBUG:teuthology.orchestra.run.smithi121:> if not os.path.exists(os.path.dirname(abs_path)): 2022-06-29T16:03:34.952 DEBUG:teuthology.orchestra.run.smithi121:> os.makedirs(os.path.dirname(abs_path)) 2022-06-29T16:03:34.952 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:03:34.952 DEBUG:teuthology.orchestra.run.smithi121:> for i in range(0, n): 2022-06-29T16:03:34.953 DEBUG:teuthology.orchestra.run.smithi121:> fname = "{0}_{1}".format(abs_path, i) 2022-06-29T16:03:34.953 DEBUG:teuthology.orchestra.run.smithi121:> with open(fname, '"'"'w'"'"') as f: 2022-06-29T16:03:34.954 DEBUG:teuthology.orchestra.run.smithi121:> f.write('"'"'content'"'"') 2022-06-29T16:03:34.954 DEBUG:teuthology.orchestra.run.smithi121:> if True: 2022-06-29T16:03:34.955 DEBUG:teuthology.orchestra.run.smithi121:> f.flush() 2022-06-29T16:03:34.955 DEBUG:teuthology.orchestra.run.smithi121:> os.fsync(f.fileno()) 2022-06-29T16:03:34.956 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-06-29T16:03:43.634 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:03:43.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.743+0000 7f4af2ba5700 1 Processor -- start 2022-06-29T16:03:43.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.743+0000 7f4af2ba5700 1 -- start start 2022-06-29T16:03:43.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.743+0000 7f4af2ba5700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4aec05c390 0x7f4aec0b2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:43.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.743+0000 7f4af2ba5700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4aec1442b0 0x7f4aec144710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:43.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.743+0000 7f4af2ba5700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aec05c770 con 0x7f4aec140a20 2022-06-29T16:03:43.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.743+0000 7f4af2ba5700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aec140750 con 0x7f4aec1442b0 2022-06-29T16:03:43.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.743+0000 7f4af2ba5700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aec143e80 con 0x7f4aec05c390 2022-06-29T16:03:43.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.743+0000 7f4af1142700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4aec140a20 0x7f4aec140e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56880/0 (socket says 172.21.15.60:56880) 2022-06-29T16:03:43.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.743+0000 7f4af1142700 1 -- 172.21.15.60:0/3235305116 learned_addr learned my addr 172.21.15.60:0/3235305116 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:03:43.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.743+0000 7f4aebfff700 1 --2- 172.21.15.60:0/3235305116 >> v2:172.21.15.92:3300/0 conn(0x7f4aec1442b0 0x7f4aec144710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:43.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.743+0000 7f4af0941700 1 --2- 172.21.15.60:0/3235305116 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4aec05c390 0x7f4aec0b2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:43.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.743+0000 7f4aeb7fe700 1 -- 172.21.15.60:0/3235305116 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 418419134 0 0) 0x7f4aec05c770 con 0x7f4aec140a20 2022-06-29T16:03:43.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.743+0000 7f4aeb7fe700 1 -- 172.21.15.60:0/3235305116 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4acc003260 con 0x7f4aec140a20 2022-06-29T16:03:43.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.743+0000 7f4aebfff700 1 -- 172.21.15.60:0/3235305116 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4aec05c390 msgr2=0x7f4aec0b2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:43.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.743+0000 7f4aebfff700 1 --2- 172.21.15.60:0/3235305116 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4aec05c390 0x7f4aec0b2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:43.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.743+0000 7f4aebfff700 1 -- 172.21.15.60:0/3235305116 >> v1:172.21.15.60:6789/0 conn(0x7f4aec140a20 legacy=0x7f4aec140e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:43.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.743+0000 7f4aebfff700 1 -- 172.21.15.60:0/3235305116 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aec065f40 con 0x7f4aec1442b0 2022-06-29T16:03:43.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.743+0000 7f4aebfff700 1 --2- 172.21.15.60:0/3235305116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4aec1442b0 0x7f4aec144710 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f4ad40032f0 tx=0x7f4ad40091d0).ready entity=mon.1 client_cookie=80bffd8df6410302 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:43.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.743+0000 7f4aeb7fe700 1 -- 172.21.15.60:0/3235305116 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ad400b600 con 0x7f4aec1442b0 2022-06-29T16:03:43.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.743+0000 7f4aeb7fe700 1 -- 172.21.15.60:0/3235305116 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ad400b760 con 0x7f4aec1442b0 2022-06-29T16:03:43.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.743+0000 7f4aeb7fe700 1 -- 172.21.15.60:0/3235305116 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ad400b8d0 con 0x7f4aec1442b0 2022-06-29T16:03:43.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.747+0000 7f4af2ba5700 1 -- 172.21.15.60:0/3235305116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4aec1442b0 msgr2=0x7f4aec144710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:43.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.747+0000 7f4af2ba5700 1 --2- 172.21.15.60:0/3235305116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4aec1442b0 0x7f4aec144710 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f4ad40032f0 tx=0x7f4ad40091d0).stop 2022-06-29T16:03:43.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.747+0000 7f4af2ba5700 1 -- 172.21.15.60:0/3235305116 shutdown_connections 2022-06-29T16:03:43.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.747+0000 7f4af2ba5700 1 --2- 172.21.15.60:0/3235305116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4aec1442b0 0x7f4aec144710 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:43.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.747+0000 7f4af2ba5700 1 --2- 172.21.15.60:0/3235305116 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4aec05c390 0x7f4aec0b2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:43.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.747+0000 7f4af2ba5700 1 -- 172.21.15.60:0/3235305116 >> 172.21.15.60:0/3235305116 conn(0x7f4aec064a10 msgr2=0x7f4aec064df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:43.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.747+0000 7f4af2ba5700 1 -- 172.21.15.60:0/3235305116 shutdown_connections 2022-06-29T16:03:43.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.747+0000 7f4af2ba5700 1 -- 172.21.15.60:0/3235305116 wait complete. 2022-06-29T16:03:43.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.747+0000 7f4af2ba5700 1 Processor -- start 2022-06-29T16:03:43.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.747+0000 7f4af2ba5700 1 -- start start 2022-06-29T16:03:43.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.751+0000 7f4af2ba5700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4aec05c390 0x7f4aec0b6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:43.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.751+0000 7f4af2ba5700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4aec140a20 0x7f4aec0ae360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:43.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.751+0000 7f4af2ba5700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4aec1442b0 0x7f4aec0b2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:43.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.751+0000 7f4af2ba5700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aec0b7ed0 con 0x7f4aec05c390 2022-06-29T16:03:43.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.751+0000 7f4af2ba5700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aec0b7990 con 0x7f4aec140a20 2022-06-29T16:03:43.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.751+0000 7f4af2ba5700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aec0b7c30 con 0x7f4aec1442b0 2022-06-29T16:03:43.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.751+0000 7f4af0941700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4aec05c390 0x7f4aec0b6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:43.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.751+0000 7f4aebfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4aec140a20 0x7f4aec0ae360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:43.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.751+0000 7f4aebfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4aec140a20 0x7f4aec0ae360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:54828/0 (socket says 172.21.15.60:54828) 2022-06-29T16:03:43.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.751+0000 7f4aebfff700 1 -- 172.21.15.60:0/3284741120 learned_addr learned my addr 172.21.15.60:0/3284741120 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:43.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.751+0000 7f4af0941700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4aec05c390 0x7f4aec0b6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48796/0 (socket says 172.21.15.60:48796) 2022-06-29T16:03:43.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.751+0000 7f4af1142700 1 --2- 172.21.15.60:0/3284741120 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4aec1442b0 0x7f4aec0b2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:43.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.751+0000 7f4aebfff700 1 -- 172.21.15.60:0/3284741120 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4aec1442b0 msgr2=0x7f4aec0b2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:43.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.751+0000 7f4aebfff700 1 --2- 172.21.15.60:0/3284741120 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4aec1442b0 0x7f4aec0b2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:43.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.751+0000 7f4aebfff700 1 -- 172.21.15.60:0/3284741120 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4aec05c390 msgr2=0x7f4aec0b6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:43.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.751+0000 7f4aebfff700 1 --2- 172.21.15.60:0/3284741120 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4aec05c390 0x7f4aec0b6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:43.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.751+0000 7f4aebfff700 1 -- 172.21.15.60:0/3284741120 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad4009470 con 0x7f4aec140a20 2022-06-29T16:03:43.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.751+0000 7f4af1142700 1 --2- 172.21.15.60:0/3284741120 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4aec1442b0 0x7f4aec0b2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:03:43.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.751+0000 7f4aebfff700 1 --2- 172.21.15.60:0/3284741120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4aec140a20 0x7f4aec0ae360 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f4ad40032f0 tx=0x7f4ad4016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:43.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.751+0000 7f4ae9ffb700 1 -- 172.21.15.60:0/3284741120 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ad400b8d0 con 0x7f4aec140a20 2022-06-29T16:03:43.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.751+0000 7f4ae9ffb700 1 -- 172.21.15.60:0/3284741120 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ad40205f0 con 0x7f4aec140a20 2022-06-29T16:03:43.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.751+0000 7f4af2ba5700 1 -- 172.21.15.60:0/3284741120 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aec0b2f80 con 0x7f4aec140a20 2022-06-29T16:03:43.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.751+0000 7f4ae9ffb700 1 -- 172.21.15.60:0/3284741120 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ad400b600 con 0x7f4aec140a20 2022-06-29T16:03:43.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.751+0000 7f4af2ba5700 1 -- 172.21.15.60:0/3284741120 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aec146d00 con 0x7f4aec140a20 2022-06-29T16:03:43.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.755+0000 7f4ae9ffb700 1 -- 172.21.15.60:0/3284741120 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4ad4027030 con 0x7f4aec140a20 2022-06-29T16:03:43.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.755+0000 7f4af2ba5700 1 -- 172.21.15.60:0/3284741120 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aec0573d0 con 0x7f4aec140a20 2022-06-29T16:03:43.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.755+0000 7f4ae9ffb700 1 --2- 172.21.15.60:0/3284741120 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4ad00539e0 0x7f4ad0055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:43.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.755+0000 7f4ae9ffb700 1 -- 172.21.15.60:0/3284741120 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4ad406cf30 con 0x7f4aec140a20 2022-06-29T16:03:43.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.755+0000 7f4af0941700 1 --2- 172.21.15.60:0/3284741120 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4ad00539e0 0x7f4ad0055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:43.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.755+0000 7f4af0941700 1 --2- 172.21.15.60:0/3284741120 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4ad00539e0 0x7f4ad0055ea0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f4ae400b000 tx=0x7f4ae4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:43.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.763+0000 7f4ae9ffb700 1 -- 172.21.15.60:0/3284741120 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4ad4043e50 con 0x7f4aec140a20 2022-06-29T16:03:43.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.923+0000 7f4af2ba5700 1 -- 172.21.15.60:0/3284741120 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4aec0b5cd0 con 0x7f4aec140a20 2022-06-29T16:03:43.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.923+0000 7f4ae9ffb700 1 -- 172.21.15.60:0/3284741120 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f4ad4025090 con 0x7f4aec140a20 2022-06-29T16:03:43.931 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:03:43.931 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":25,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5955,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.60:6833/66690512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":66690512},{"type":"v1","addr":"172.21.15.60:6833","nonce":66690512}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":23}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:03:28.112667+0000","modified":"2022-06-29T16:03:29.115727+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6013},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6013":{"gid":6013,"name":"b","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/3876875590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3876875590},{"type":"v1","addr":"172.21.15.92:6835","nonce":3876875590}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-06-29T16:03:43.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.927+0000 7f4af2ba5700 1 -- 172.21.15.60:0/3284741120 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4ad00539e0 msgr2=0x7f4ad0055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:43.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.927+0000 7f4af2ba5700 1 --2- 172.21.15.60:0/3284741120 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4ad00539e0 0x7f4ad0055ea0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f4ae400b000 tx=0x7f4ae4006df0).stop 2022-06-29T16:03:43.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.927+0000 7f4af2ba5700 1 -- 172.21.15.60:0/3284741120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4aec140a20 msgr2=0x7f4aec0ae360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:43.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.927+0000 7f4af2ba5700 1 --2- 172.21.15.60:0/3284741120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4aec140a20 0x7f4aec0ae360 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f4ad40032f0 tx=0x7f4ad4016000).stop 2022-06-29T16:03:43.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.927+0000 7f4af2ba5700 1 -- 172.21.15.60:0/3284741120 shutdown_connections 2022-06-29T16:03:43.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.927+0000 7f4af2ba5700 1 --2- 172.21.15.60:0/3284741120 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4ad00539e0 0x7f4ad0055ea0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:43.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.927+0000 7f4af2ba5700 1 --2- 172.21.15.60:0/3284741120 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4aec1442b0 0x7f4aec0b2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:43.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.927+0000 7f4af2ba5700 1 --2- 172.21.15.60:0/3284741120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4aec140a20 0x7f4aec0ae360 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:43.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.927+0000 7f4af2ba5700 1 --2- 172.21.15.60:0/3284741120 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4aec05c390 0x7f4aec0b6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:43.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.927+0000 7f4af2ba5700 1 -- 172.21.15.60:0/3284741120 >> 172.21.15.60:0/3284741120 conn(0x7f4aec064a10 msgr2=0x7f4aec141e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:43.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.927+0000 7f4af2ba5700 1 -- 172.21.15.60:0/3284741120 shutdown_connections 2022-06-29T16:03:43.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:43.927+0000 7f4af2ba5700 1 -- 172.21.15.60:0/3284741120 wait complete. 2022-06-29T16:03:43.937 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 25 2022-06-29T16:03:43.944 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json config set mds_recall_max_caps 20 2022-06-29T16:03:44.051 INFO:teuthology.orchestra.run.smithi092.stdout:{"success":"mds_recall_max_caps = '20' (not observed, change may require restart) "} 2022-06-29T16:03:44.052 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_caps = '20' (not observed, change may require restart) " } 2022-06-29T16:03:44.052 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:03:44.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.155+0000 7f9e6ad11700 1 Processor -- start 2022-06-29T16:03:44.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.155+0000 7f9e6ad11700 1 -- start start 2022-06-29T16:03:44.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.159+0000 7f9e6ad11700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e640b5eb0 0x7f9e64142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:44.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.159+0000 7f9e6ad11700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e640b0cf0 con 0x7f9e640b48b0 2022-06-29T16:03:44.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.159+0000 7f9e6ad11700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e640b4d10 con 0x7f9e640b2c50 2022-06-29T16:03:44.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.159+0000 7f9e6ad11700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e640b54e0 con 0x7f9e640b5eb0 2022-06-29T16:03:44.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.159+0000 7f9e6950e700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9e640b48b0 0x7f9e640b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56894/0 (socket says 172.21.15.60:56894) 2022-06-29T16:03:44.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.159+0000 7f9e6950e700 1 -- 172.21.15.60:0/1135468496 learned_addr learned my addr 172.21.15.60:0/1135468496 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:03:44.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.159+0000 7f9e69d0f700 1 --2- 172.21.15.60:0/1135468496 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e640b5eb0 0x7f9e64142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:44.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.159+0000 7f9e68d0d700 1 -- 172.21.15.60:0/1135468496 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2121297370 0 0) 0x7f9e54001110 con 0x7f9e640b48b0 2022-06-29T16:03:44.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.159+0000 7f9e68d0d700 1 -- 172.21.15.60:0/1135468496 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e4c003260 con 0x7f9e640b48b0 2022-06-29T16:03:44.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.159+0000 7f9e68d0d700 1 -- 172.21.15.60:0/1135468496 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 566835821 0 0) 0x7f9e640b4d10 con 0x7f9e640b2c50 2022-06-29T16:03:44.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.159+0000 7f9e68d0d700 1 -- 172.21.15.60:0/1135468496 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e54001110 con 0x7f9e640b2c50 2022-06-29T16:03:44.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.159+0000 7f9e69d0f700 1 -- 172.21.15.60:0/1135468496 >> v1:172.21.15.92:6789/0 conn(0x7f9e640b2c50 legacy=0x7f9e640b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:44.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.159+0000 7f9e68d0d700 1 -- 172.21.15.60:0/1135468496 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3240658159 0 0) 0x7f9e4c003260 con 0x7f9e640b48b0 2022-06-29T16:03:44.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.159+0000 7f9e69d0f700 1 -- 172.21.15.60:0/1135468496 >> v1:172.21.15.60:6789/0 conn(0x7f9e640b48b0 legacy=0x7f9e640b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:44.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.159+0000 7f9e69d0f700 1 -- 172.21.15.60:0/1135468496 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e640b6290 con 0x7f9e640b5eb0 2022-06-29T16:03:44.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.159+0000 7f9e69d0f700 1 --2- 172.21.15.60:0/1135468496 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e640b5eb0 0x7f9e64142a20 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f9e540012a0 tx=0x7f9e60006df0).ready entity=mon.2 client_cookie=54b26c1748e4a2c5 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:44.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.159+0000 7f9e68d0d700 1 -- 172.21.15.60:0/1135468496 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e540017f0 con 0x7f9e640b5eb0 2022-06-29T16:03:44.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.159+0000 7f9e68d0d700 1 -- 172.21.15.60:0/1135468496 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e60000e90 con 0x7f9e640b5eb0 2022-06-29T16:03:44.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.159+0000 7f9e68d0d700 1 -- 172.21.15.60:0/1135468496 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e60012460 con 0x7f9e640b5eb0 2022-06-29T16:03:44.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.159+0000 7f9e6ad11700 1 -- 172.21.15.60:0/1135468496 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e640b5eb0 msgr2=0x7f9e64142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:44.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.159+0000 7f9e6ad11700 1 --2- 172.21.15.60:0/1135468496 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e640b5eb0 0x7f9e64142a20 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f9e540012a0 tx=0x7f9e60006df0).stop 2022-06-29T16:03:44.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.163+0000 7f9e6ad11700 1 -- 172.21.15.60:0/1135468496 shutdown_connections 2022-06-29T16:03:44.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.163+0000 7f9e6ad11700 1 --2- 172.21.15.60:0/1135468496 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e640b5eb0 0x7f9e64142a20 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.163+0000 7f9e6ad11700 1 -- 172.21.15.60:0/1135468496 >> 172.21.15.60:0/1135468496 conn(0x7f9e64062090 msgr2=0x7f9e64062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:44.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.163+0000 7f9e6ad11700 1 -- 172.21.15.60:0/1135468496 shutdown_connections 2022-06-29T16:03:44.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.163+0000 7f9e6ad11700 1 -- 172.21.15.60:0/1135468496 wait complete. 2022-06-29T16:03:44.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.163+0000 7f9e6ad11700 1 Processor -- start 2022-06-29T16:03:44.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.163+0000 7f9e6ad11700 1 -- start start 2022-06-29T16:03:44.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.163+0000 7f9e6ad11700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e640b2c50 0x7f9e64154fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:44.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.163+0000 7f9e6ad11700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e640b48b0 0x7f9e641408d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:44.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.163+0000 7f9e6ad11700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e640b5eb0 0x7f9e64146530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:44.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.163+0000 7f9e6ad11700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e64148df0 con 0x7f9e640b2c50 2022-06-29T16:03:44.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.163+0000 7f9e6ad11700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e64147f50 con 0x7f9e640b48b0 2022-06-29T16:03:44.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.163+0000 7f9e6ad11700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e64143ee0 con 0x7f9e640b5eb0 2022-06-29T16:03:44.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.163+0000 7f9e69d0f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e640b2c50 0x7f9e64154fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:44.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.163+0000 7f9e69d0f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e640b2c50 0x7f9e64154fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48810/0 (socket says 172.21.15.60:48810) 2022-06-29T16:03:44.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.163+0000 7f9e69d0f700 1 -- 172.21.15.60:0/2113903856 learned_addr learned my addr 172.21.15.60:0/2113903856 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:44.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.163+0000 7f9e6950e700 1 --2- 172.21.15.60:0/2113903856 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e640b48b0 0x7f9e641408d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:44.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.163+0000 7f9e6a510700 1 --2- 172.21.15.60:0/2113903856 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e640b5eb0 0x7f9e64146530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:44.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.167+0000 7f9e69d0f700 1 -- 172.21.15.60:0/2113903856 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e640b5eb0 msgr2=0x7f9e64146530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:44.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.167+0000 7f9e69d0f700 1 --2- 172.21.15.60:0/2113903856 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e640b5eb0 0x7f9e64146530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.167+0000 7f9e69d0f700 1 -- 172.21.15.60:0/2113903856 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e640b48b0 msgr2=0x7f9e641408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:44.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.167+0000 7f9e69d0f700 1 --2- 172.21.15.60:0/2113903856 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e640b48b0 0x7f9e641408d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.167+0000 7f9e69d0f700 1 -- 172.21.15.60:0/2113903856 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e6000b180 con 0x7f9e640b2c50 2022-06-29T16:03:44.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.167+0000 7f9e69d0f700 1 --2- 172.21.15.60:0/2113903856 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e640b2c50 0x7f9e64154fb0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7f9e540012a0 tx=0x7f9e60014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:44.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.167+0000 7f9e52ffd700 1 -- 172.21.15.60:0/2113903856 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e60016070 con 0x7f9e640b2c50 2022-06-29T16:03:44.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.167+0000 7f9e52ffd700 1 -- 172.21.15.60:0/2113903856 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e60002f50 con 0x7f9e640b2c50 2022-06-29T16:03:44.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.167+0000 7f9e52ffd700 1 -- 172.21.15.60:0/2113903856 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e60012460 con 0x7f9e640b2c50 2022-06-29T16:03:44.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.167+0000 7f9e6ad11700 1 -- 172.21.15.60:0/2113903856 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e64148350 con 0x7f9e640b2c50 2022-06-29T16:03:44.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.167+0000 7f9e6ad11700 1 -- 172.21.15.60:0/2113903856 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e64147a40 con 0x7f9e640b2c50 2022-06-29T16:03:44.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.167+0000 7f9e6ad11700 1 -- 172.21.15.60:0/2113903856 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e64142f10 con 0x7f9e640b2c50 2022-06-29T16:03:44.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.167+0000 7f9e52ffd700 1 -- 172.21.15.60:0/2113903856 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9e6000f030 con 0x7f9e640b2c50 2022-06-29T16:03:44.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.171+0000 7f9e52ffd700 1 --2- 172.21.15.60:0/2113903856 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9e44053b00 0x7f9e44055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:44.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.171+0000 7f9e52ffd700 1 -- 172.21.15.60:0/2113903856 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9e6006c940 con 0x7f9e640b2c50 2022-06-29T16:03:44.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.171+0000 7f9e6950e700 1 --2- 172.21.15.60:0/2113903856 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9e44053b00 0x7f9e44055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:44.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.171+0000 7f9e6950e700 1 --2- 172.21.15.60:0/2113903856 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9e44053b00 0x7f9e44055fc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f9e5400d000 tx=0x7f9e54001450).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:44.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.179+0000 7f9e52ffd700 1 -- 172.21.15.60:0/2113903856 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9e60043870 con 0x7f9e640b2c50 2022-06-29T16:03:44.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.343+0000 7f9e6ad11700 1 -- 172.21.15.60:0/2113903856 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e640573d0 con 0x7f9e640b2c50 2022-06-29T16:03:44.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.343+0000 7f9e52ffd700 1 -- 172.21.15.60:0/2113903856 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f9e60023030 con 0x7f9e640b2c50 2022-06-29T16:03:44.352 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:03:44.352 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":25,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5955,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.60:6833/66690512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":66690512},{"type":"v1","addr":"172.21.15.60:6833","nonce":66690512}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":23}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:03:28.112667+0000","modified":"2022-06-29T16:03:29.115727+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6013},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6013":{"gid":6013,"name":"b","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/3876875590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3876875590},{"type":"v1","addr":"172.21.15.92:6835","nonce":3876875590}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-06-29T16:03:44.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.347+0000 7f9e6ad11700 1 -- 172.21.15.60:0/2113903856 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9e44053b00 msgr2=0x7f9e44055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:44.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.347+0000 7f9e6ad11700 1 --2- 172.21.15.60:0/2113903856 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9e44053b00 0x7f9e44055fc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f9e5400d000 tx=0x7f9e54001450).stop 2022-06-29T16:03:44.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.347+0000 7f9e6ad11700 1 -- 172.21.15.60:0/2113903856 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e640b2c50 msgr2=0x7f9e64154fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:44.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.347+0000 7f9e6ad11700 1 --2- 172.21.15.60:0/2113903856 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e640b2c50 0x7f9e64154fb0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7f9e540012a0 tx=0x7f9e60014000).stop 2022-06-29T16:03:44.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.347+0000 7f9e6ad11700 1 -- 172.21.15.60:0/2113903856 shutdown_connections 2022-06-29T16:03:44.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.347+0000 7f9e6ad11700 1 --2- 172.21.15.60:0/2113903856 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9e44053b00 0x7f9e44055fc0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.347+0000 7f9e6ad11700 1 --2- 172.21.15.60:0/2113903856 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e640b5eb0 0x7f9e64146530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.347+0000 7f9e6ad11700 1 --2- 172.21.15.60:0/2113903856 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e640b48b0 0x7f9e641408d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.347+0000 7f9e6ad11700 1 --2- 172.21.15.60:0/2113903856 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e640b2c50 0x7f9e64154fb0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.347+0000 7f9e6ad11700 1 -- 172.21.15.60:0/2113903856 >> 172.21.15.60:0/2113903856 conn(0x7f9e64062090 msgr2=0x7f9e640b1500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:44.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.347+0000 7f9e6ad11700 1 -- 172.21.15.60:0/2113903856 shutdown_connections 2022-06-29T16:03:44.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.347+0000 7f9e6ad11700 1 -- 172.21.15.60:0/2113903856 wait complete. 2022-06-29T16:03:44.361 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 25 2022-06-29T16:03:44.365 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json config set mds_recall_max_decay_threshold 400 2022-06-29T16:03:44.478 INFO:teuthology.orchestra.run.smithi092.stdout:{"success":"mds_recall_max_decay_threshold = '400' (not observed, change may require restart) "} 2022-06-29T16:03:44.478 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_decay_threshold = '400' (not observed, change may require restart) " } 2022-06-29T16:03:44.479 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:03:44.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.583+0000 7f90b4d6a700 1 Processor -- start 2022-06-29T16:03:44.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.583+0000 7f90b4d6a700 1 -- start start 2022-06-29T16:03:44.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.583+0000 7f90b4d6a700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f90b0063ad0 0x7f90b0067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:44.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.583+0000 7f90b4d6a700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b00acba0 con 0x7f90b014af10 2022-06-29T16:03:44.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.583+0000 7f90b4d6a700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b0067220 con 0x7f90b00b5950 2022-06-29T16:03:44.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.583+0000 7f90b4d6a700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90b014b3a0 con 0x7f90b0063ad0 2022-06-29T16:03:44.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.583+0000 7f90add9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f90b0063ad0 0x7f90b0067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:44.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.583+0000 7f90add9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f90b0063ad0 0x7f90b0067680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48818/0 (socket says 172.21.15.60:48818) 2022-06-29T16:03:44.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.583+0000 7f90add9b700 1 -- 172.21.15.60:0/901096091 learned_addr learned my addr 172.21.15.60:0/901096091 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:44.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.583+0000 7f90aed9d700 1 --1- 172.21.15.60:0/901096091 >> v1:172.21.15.60:6789/0 conn(0x7f90b014af10 0x7f90b005c430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56908/0 (socket says 172.21.15.60:56908) 2022-06-29T16:03:44.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.583+0000 7f90add9b700 1 -- 172.21.15.60:0/901096091 >> v1:172.21.15.92:6789/0 conn(0x7f90b00b5950 legacy=0x7f90b005bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:44.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.583+0000 7f90add9b700 1 -- 172.21.15.60:0/901096091 >> v1:172.21.15.60:6789/0 conn(0x7f90b014af10 legacy=0x7f90b005c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:44.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.583+0000 7f90add9b700 1 -- 172.21.15.60:0/901096091 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b00b5d30 con 0x7f90b0063ad0 2022-06-29T16:03:44.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.583+0000 7f90add9b700 1 --2- 172.21.15.60:0/901096091 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90b0063ad0 0x7f90b0067680 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7f90a4002f70 tx=0x7f90a4006eb0).ready entity=mon.0 client_cookie=a456f26aff3189b1 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:44.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.583+0000 7f90ad59a700 1 -- 172.21.15.60:0/901096091 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90a400d660 con 0x7f90b0063ad0 2022-06-29T16:03:44.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.583+0000 7f90ad59a700 1 -- 172.21.15.60:0/901096091 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90a4000e90 con 0x7f90b0063ad0 2022-06-29T16:03:44.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.583+0000 7f90ad59a700 1 -- 172.21.15.60:0/901096091 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90a400d9f0 con 0x7f90b0063ad0 2022-06-29T16:03:44.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.583+0000 7f90b4d6a700 1 -- 172.21.15.60:0/901096091 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90b0063ad0 msgr2=0x7f90b0067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:44.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.583+0000 7f90b4d6a700 1 --2- 172.21.15.60:0/901096091 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90b0063ad0 0x7f90b0067680 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7f90a4002f70 tx=0x7f90a4006eb0).stop 2022-06-29T16:03:44.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.583+0000 7f90b4d6a700 1 -- 172.21.15.60:0/901096091 shutdown_connections 2022-06-29T16:03:44.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.583+0000 7f90b4d6a700 1 --2- 172.21.15.60:0/901096091 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90b0063ad0 0x7f90b0067680 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.583+0000 7f90b4d6a700 1 -- 172.21.15.60:0/901096091 >> 172.21.15.60:0/901096091 conn(0x7f90b00632c0 msgr2=0x7f90b00b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:44.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.583+0000 7f90b4d6a700 1 -- 172.21.15.60:0/901096091 shutdown_connections 2022-06-29T16:03:44.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.583+0000 7f90b4d6a700 1 -- 172.21.15.60:0/901096091 wait complete. 2022-06-29T16:03:44.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.587+0000 7f90b4d6a700 1 Processor -- start 2022-06-29T16:03:44.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.587+0000 7f90b4d6a700 1 -- start start 2022-06-29T16:03:44.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.587+0000 7f90b4d6a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90b0063ad0 0x7f90b0148660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:44.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.587+0000 7f90b4d6a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90b00b5950 0x7f90b00b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:44.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.587+0000 7f90b4d6a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90b014af10 0x7f90b01561b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:44.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.587+0000 7f90b4d6a700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b00b05b0 con 0x7f90b00b5950 2022-06-29T16:03:44.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.587+0000 7f90b4d6a700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b015a310 con 0x7f90b0063ad0 2022-06-29T16:03:44.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.587+0000 7f90b4d6a700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90b015a5b0 con 0x7f90b014af10 2022-06-29T16:03:44.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.587+0000 7f90add9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90b00b5950 0x7f90b00b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:44.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.587+0000 7f90ae59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90b0063ad0 0x7f90b0148660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:44.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.587+0000 7f90add9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90b00b5950 0x7f90b00b09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48826/0 (socket says 172.21.15.60:48826) 2022-06-29T16:03:44.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.587+0000 7f90add9b700 1 -- 172.21.15.60:0/742751871 learned_addr learned my addr 172.21.15.60:0/742751871 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:44.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.587+0000 7f90aed9d700 1 --2- 172.21.15.60:0/742751871 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90b014af10 0x7f90b01561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:44.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.587+0000 7f90add9b700 1 -- 172.21.15.60:0/742751871 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90b014af10 msgr2=0x7f90b01561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:44.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.587+0000 7f90add9b700 1 --2- 172.21.15.60:0/742751871 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90b014af10 0x7f90b01561b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.587+0000 7f90add9b700 1 -- 172.21.15.60:0/742751871 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90b0063ad0 msgr2=0x7f90b0148660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:44.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.587+0000 7f90add9b700 1 --2- 172.21.15.60:0/742751871 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90b0063ad0 0x7f90b0148660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.587+0000 7f90add9b700 1 -- 172.21.15.60:0/742751871 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a400b1e0 con 0x7f90b00b5950 2022-06-29T16:03:44.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.587+0000 7f90add9b700 1 --2- 172.21.15.60:0/742751871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90b00b5950 0x7f90b00b09e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0x7f90a4002f70 tx=0x7f90a4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:44.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.587+0000 7f909f7fe700 1 -- 172.21.15.60:0/742751871 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90a40068b0 con 0x7f90b00b5950 2022-06-29T16:03:44.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.587+0000 7f909f7fe700 1 -- 172.21.15.60:0/742751871 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90a4011ae0 con 0x7f90b00b5950 2022-06-29T16:03:44.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.587+0000 7f90b4d6a700 1 -- 172.21.15.60:0/742751871 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90b014a6b0 con 0x7f90b00b5950 2022-06-29T16:03:44.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.587+0000 7f909f7fe700 1 -- 172.21.15.60:0/742751871 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90a4011e40 con 0x7f90b00b5950 2022-06-29T16:03:44.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.587+0000 7f90b4d6a700 1 -- 172.21.15.60:0/742751871 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90b00ad5f0 con 0x7f90b00b5950 2022-06-29T16:03:44.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.591+0000 7f90b4d6a700 1 -- 172.21.15.60:0/742751871 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90b015a160 con 0x7f90b00b5950 2022-06-29T16:03:44.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.591+0000 7f909f7fe700 1 -- 172.21.15.60:0/742751871 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f90a4025030 con 0x7f90b00b5950 2022-06-29T16:03:44.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.591+0000 7f909f7fe700 1 --2- 172.21.15.60:0/742751871 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f908c053a70 0x7f908c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:44.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.591+0000 7f909f7fe700 1 -- 172.21.15.60:0/742751871 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f90a406d0e0 con 0x7f90b00b5950 2022-06-29T16:03:44.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.591+0000 7f90ae59c700 1 --2- 172.21.15.60:0/742751871 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f908c053a70 0x7f908c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:44.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.591+0000 7f90ae59c700 1 --2- 172.21.15.60:0/742751871 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f908c053a70 0x7f908c055f30 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f90a000d000 tx=0x7f90a0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:44.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.595+0000 7f909f7fe700 1 -- 172.21.15.60:0/742751871 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f90a4044000 con 0x7f90b00b5950 2022-06-29T16:03:44.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.755+0000 7f90b4d6a700 1 -- 172.21.15.60:0/742751871 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90b00573d0 con 0x7f90b00b5950 2022-06-29T16:03:44.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.755+0000 7f909f7fe700 1 -- 172.21.15.60:0/742751871 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f90a4023090 con 0x7f90b00b5950 2022-06-29T16:03:44.764 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:03:44.764 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":25,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5955,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.60:6833/66690512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":66690512},{"type":"v1","addr":"172.21.15.60:6833","nonce":66690512}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":23}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:03:28.112667+0000","modified":"2022-06-29T16:03:29.115727+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6013},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6013":{"gid":6013,"name":"b","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/3876875590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3876875590},{"type":"v1","addr":"172.21.15.92:6835","nonce":3876875590}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-06-29T16:03:44.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.759+0000 7f90b4d6a700 1 -- 172.21.15.60:0/742751871 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f908c053a70 msgr2=0x7f908c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:44.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.759+0000 7f90b4d6a700 1 --2- 172.21.15.60:0/742751871 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f908c053a70 0x7f908c055f30 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f90a000d000 tx=0x7f90a0009040).stop 2022-06-29T16:03:44.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.759+0000 7f90b4d6a700 1 -- 172.21.15.60:0/742751871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90b00b5950 msgr2=0x7f90b00b09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:44.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.759+0000 7f90b4d6a700 1 --2- 172.21.15.60:0/742751871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90b00b5950 0x7f90b00b09e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0x7f90a4002f70 tx=0x7f90a4015000).stop 2022-06-29T16:03:44.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.759+0000 7f90b4d6a700 1 -- 172.21.15.60:0/742751871 shutdown_connections 2022-06-29T16:03:44.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.759+0000 7f90b4d6a700 1 --2- 172.21.15.60:0/742751871 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f908c053a70 0x7f908c055f30 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.759+0000 7f90b4d6a700 1 --2- 172.21.15.60:0/742751871 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90b014af10 0x7f90b01561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.759+0000 7f90b4d6a700 1 --2- 172.21.15.60:0/742751871 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90b00b5950 0x7f90b00b09e0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.759+0000 7f90b4d6a700 1 --2- 172.21.15.60:0/742751871 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90b0063ad0 0x7f90b0148660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.759+0000 7f90b4d6a700 1 -- 172.21.15.60:0/742751871 >> 172.21.15.60:0/742751871 conn(0x7f90b00632c0 msgr2=0x7f90b0145f50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:44.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.759+0000 7f90b4d6a700 1 -- 172.21.15.60:0/742751871 shutdown_connections 2022-06-29T16:03:44.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.759+0000 7f90b4d6a700 1 -- 172.21.15.60:0/742751871 wait complete. 2022-06-29T16:03:44.771 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 25 2022-06-29T16:03:44.781 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b cache drop 10 2022-06-29T16:03:44.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.883+0000 7f3e7f139700 1 Processor -- start 2022-06-29T16:03:44.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.887+0000 7f3e7f139700 1 -- start start 2022-06-29T16:03:44.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.887+0000 7f3e7f139700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e780b2b40 0x7f3e780af420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:44.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.887+0000 7f3e7f139700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3e78143e10 0x7f3e780b05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:44.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.887+0000 7f3e7f139700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e780b2f20 con 0x7f3e780b0ae0 2022-06-29T16:03:44.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.887+0000 7f3e7f139700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e780aea40 con 0x7f3e78143e10 2022-06-29T16:03:44.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.887+0000 7f3e7f139700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e780aece0 con 0x7f3e780b2b40 2022-06-29T16:03:44.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.887+0000 7f3e7d936700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3e78143e10 0x7f3e780b05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:44.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.887+0000 7f3e7e938700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f3e780b0ae0 0x7f3e7813fdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56920/0 (socket says 172.21.15.60:56920) 2022-06-29T16:03:44.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.887+0000 7f3e7e938700 1 -- 172.21.15.60:0/984938700 learned_addr learned my addr 172.21.15.60:0/984938700 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:03:44.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.887+0000 7f3e7e137700 1 --2- 172.21.15.60:0/984938700 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e780b2b40 0x7f3e780af420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:44.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.887+0000 7f3e7d135700 1 -- 172.21.15.60:0/984938700 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 364184949 0 0) 0x7f3e780b2f20 con 0x7f3e780b0ae0 2022-06-29T16:03:44.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.887+0000 7f3e7d135700 1 -- 172.21.15.60:0/984938700 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e60003260 con 0x7f3e780b0ae0 2022-06-29T16:03:44.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.887+0000 7f3e7d936700 1 -- 172.21.15.60:0/984938700 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e780b2b40 msgr2=0x7f3e780af420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:44.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.887+0000 7f3e7d936700 1 --2- 172.21.15.60:0/984938700 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e780b2b40 0x7f3e780af420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.887+0000 7f3e7d936700 1 -- 172.21.15.60:0/984938700 >> v1:172.21.15.60:6789/0 conn(0x7f3e780b0ae0 legacy=0x7f3e7813fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:44.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.887+0000 7f3e7d936700 1 -- 172.21.15.60:0/984938700 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e780aef80 con 0x7f3e78143e10 2022-06-29T16:03:44.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.887+0000 7f3e7d936700 1 --2- 172.21.15.60:0/984938700 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e78143e10 0x7f3e780b05f0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7f3e68002f70 tx=0x7f3e68006df0).ready entity=mon.0 client_cookie=b3a6532b0aa5d0e5 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:44.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.887+0000 7f3e7d135700 1 -- 172.21.15.60:0/984938700 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e6800d4a0 con 0x7f3e78143e10 2022-06-29T16:03:44.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.887+0000 7f3e7d135700 1 -- 172.21.15.60:0/984938700 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e68000e90 con 0x7f3e78143e10 2022-06-29T16:03:44.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.887+0000 7f3e7d135700 1 -- 172.21.15.60:0/984938700 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e6800d7f0 con 0x7f3e78143e10 2022-06-29T16:03:44.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.891+0000 7f3e7f139700 1 -- 172.21.15.60:0/984938700 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e78143e10 msgr2=0x7f3e780b05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:44.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.891+0000 7f3e7f139700 1 --2- 172.21.15.60:0/984938700 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e78143e10 0x7f3e780b05f0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7f3e68002f70 tx=0x7f3e68006df0).stop 2022-06-29T16:03:44.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.891+0000 7f3e7f139700 1 -- 172.21.15.60:0/984938700 shutdown_connections 2022-06-29T16:03:44.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.891+0000 7f3e7f139700 1 --2- 172.21.15.60:0/984938700 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e78143e10 0x7f3e780b05f0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.891+0000 7f3e7f139700 1 --2- 172.21.15.60:0/984938700 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e780b2b40 0x7f3e780af420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.891+0000 7f3e7f139700 1 -- 172.21.15.60:0/984938700 >> 172.21.15.60:0/984938700 conn(0x7f3e780b6120 msgr2=0x7f3e780b20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:44.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.891+0000 7f3e7f139700 1 -- 172.21.15.60:0/984938700 shutdown_connections 2022-06-29T16:03:44.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.891+0000 7f3e7f139700 1 -- 172.21.15.60:0/984938700 wait complete. 2022-06-29T16:03:44.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.891+0000 7f3e7f139700 1 Processor -- start 2022-06-29T16:03:44.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.891+0000 7f3e7f139700 1 -- start start 2022-06-29T16:03:44.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.891+0000 7f3e7f139700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e780b0ae0 0x7f3e780b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:44.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.891+0000 7f3e7f139700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e780b2b40 0x7f3e78067560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:44.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.891+0000 7f3e7f139700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e78143e10 0x7f3e78139b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:44.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.891+0000 7f3e7f139700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e78063dd0 con 0x7f3e78143e10 2022-06-29T16:03:44.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.891+0000 7f3e7f139700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e78065e90 con 0x7f3e780b2b40 2022-06-29T16:03:44.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.891+0000 7f3e7f139700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e780add20 con 0x7f3e780b0ae0 2022-06-29T16:03:44.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.891+0000 7f3e7e938700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e78143e10 0x7f3e78139b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:44.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.891+0000 7f3e7e137700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e780b0ae0 0x7f3e780b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:44.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.891+0000 7f3e7d936700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e780b2b40 0x7f3e78067560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:44.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.891+0000 7f3e7e137700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e780b0ae0 0x7f3e780b4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:33700/0 (socket says 172.21.15.60:33700) 2022-06-29T16:03:44.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.891+0000 7f3e7d936700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e780b2b40 0x7f3e78067560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:54870/0 (socket says 172.21.15.60:54870) 2022-06-29T16:03:44.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.891+0000 7f3e7e938700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e78143e10 0x7f3e78139b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48842/0 (socket says 172.21.15.60:48842) 2022-06-29T16:03:44.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.891+0000 7f3e7e137700 1 -- 172.21.15.60:0/540847 learned_addr learned my addr 172.21.15.60:0/540847 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:44.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.891+0000 7f3e7e938700 1 -- 172.21.15.60:0/540847 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e780b0ae0 msgr2=0x7f3e780b4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:44.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.895+0000 7f3e7e938700 1 --2- 172.21.15.60:0/540847 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e780b0ae0 0x7f3e780b4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.895+0000 7f3e7e938700 1 -- 172.21.15.60:0/540847 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e780b2b40 msgr2=0x7f3e78067560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:44.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.895+0000 7f3e7e938700 1 --2- 172.21.15.60:0/540847 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e780b2b40 0x7f3e78067560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.895+0000 7f3e7e938700 1 -- 172.21.15.60:0/540847 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e6800b180 con 0x7f3e78143e10 2022-06-29T16:03:44.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.895+0000 7f3e7d936700 1 --2- 172.21.15.60:0/540847 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e780b2b40 0x7f3e78067560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:03:44.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.895+0000 7f3e7e137700 1 --2- 172.21.15.60:0/540847 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e780b0ae0 0x7f3e780b4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:03:44.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.895+0000 7f3e7e938700 1 --2- 172.21.15.60:0/540847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e78143e10 0x7f3e78139b50 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0x7f3e74001110 tx=0x7f3e740079c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:44.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.895+0000 7f3e677fe700 1 -- 172.21.15.60:0/540847 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e7400cdb0 con 0x7f3e78143e10 2022-06-29T16:03:44.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.895+0000 7f3e677fe700 1 -- 172.21.15.60:0/540847 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e740070d0 con 0x7f3e78143e10 2022-06-29T16:03:44.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.895+0000 7f3e7f139700 1 -- 172.21.15.60:0/540847 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e78066130 con 0x7f3e78143e10 2022-06-29T16:03:44.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.895+0000 7f3e677fe700 1 -- 172.21.15.60:0/540847 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e74007430 con 0x7f3e78143e10 2022-06-29T16:03:44.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.895+0000 7f3e7f139700 1 -- 172.21.15.60:0/540847 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e78067a50 con 0x7f3e78143e10 2022-06-29T16:03:44.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.895+0000 7f3e677fe700 1 -- 172.21.15.60:0/540847 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3e74010070 con 0x7f3e78143e10 2022-06-29T16:03:44.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.895+0000 7f3e8a5f0740 1 Processor -- start 2022-06-29T16:03:44.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.895+0000 7f3e8a5f0740 1 -- start start 2022-06-29T16:03:44.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.895+0000 7f3e677fe700 1 --2- 172.21.15.60:0/540847 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3e58053a10 0x7f3e58055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:44.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.899+0000 7f3e677fe700 1 -- 172.21.15.60:0/540847 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3e7406b430 con 0x7f3e78143e10 2022-06-29T16:03:44.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.899+0000 7f3e7d936700 1 --2- 172.21.15.60:0/540847 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3e58053a10 0x7f3e58055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:44.915 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.899+0000 7f3e7d936700 1 --2- 172.21.15.60:0/540847 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3e58053a10 0x7f3e58055ed0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f3e6801a000 tx=0x7f3e68014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:44.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.899+0000 7f3e8a5f0740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1b62f10 0x1b27940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:44.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.899+0000 7f3e8a5f0740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1b29c00 0x1b2a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:44.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.899+0000 7f3e8a5f0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1b5d460 0x1b5d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:44.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.899+0000 7f3e8a5f0740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x1a862d0 con 0x1b62f10 2022-06-29T16:03:44.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.899+0000 7f3e8a5f0740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x1a99ef0 con 0x1b5d460 2022-06-29T16:03:44.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.899+0000 7f3e8a5f0740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x1a7a1a0 con 0x1b29c00 2022-06-29T16:03:44.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.899+0000 7f3e7e137700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1b62f10 0x1b27940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:44.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.899+0000 7f3e7e137700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1b62f10 0x1b27940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48846/0 (socket says 172.21.15.60:48846) 2022-06-29T16:03:44.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.899+0000 7f3e7e137700 1 -- 172.21.15.60:0/216108866 learned_addr learned my addr 172.21.15.60:0/216108866 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:44.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.899+0000 7f3e7d936700 1 --2- 172.21.15.60:0/216108866 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1b29c00 0x1b2a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:44.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.899+0000 7f3e7e938700 1 --2- 172.21.15.60:0/216108866 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1b5d460 0x1b5d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:44.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.899+0000 7f3e7e137700 1 -- 172.21.15.60:0/216108866 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1b29c00 msgr2=0x1b2a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:44.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.899+0000 7f3e7e137700 1 --2- 172.21.15.60:0/216108866 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1b29c00 0x1b2a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.899+0000 7f3e7e137700 1 -- 172.21.15.60:0/216108866 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1b5d460 msgr2=0x1b5d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:44.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.899+0000 7f3e7e137700 1 --2- 172.21.15.60:0/216108866 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1b5d460 0x1b5d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.899+0000 7f3e7e137700 1 -- 172.21.15.60:0/216108866 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1ad1df0 con 0x1b62f10 2022-06-29T16:03:44.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.899+0000 7f3e7e938700 1 --2- 172.21.15.60:0/216108866 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1b5d460 0x1b5d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:03:44.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.899+0000 7f3e7e137700 1 --2- 172.21.15.60:0/216108866 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1b62f10 0x1b27940 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x1b288a0 tx=0x7f3e700091d0).ready entity=mon.0 client_cookie=38d39cfd3681d75e server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:44.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.899+0000 7f3e7f139700 1 -- 172.21.15.60:0/216108866 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e70014650 con 0x1b62f10 2022-06-29T16:03:44.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.899+0000 7f3e7f139700 1 -- 172.21.15.60:0/216108866 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e700147b0 con 0x1b62f10 2022-06-29T16:03:44.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.899+0000 7f3e7f139700 1 -- 172.21.15.60:0/216108866 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e70014b10 con 0x1b62f10 2022-06-29T16:03:44.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.899+0000 7f3e8a5f0740 1 -- 172.21.15.60:0/216108866 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1b62f10 msgr2=0x1b27940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:44.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.899+0000 7f3e8a5f0740 1 --2- 172.21.15.60:0/216108866 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1b62f10 0x1b27940 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x1b288a0 tx=0x7f3e700091d0).stop 2022-06-29T16:03:44.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.899+0000 7f3e8a5f0740 1 -- 172.21.15.60:0/216108866 shutdown_connections 2022-06-29T16:03:44.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.899+0000 7f3e8a5f0740 1 --2- 172.21.15.60:0/216108866 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1b5d460 0x1b5d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.899+0000 7f3e8a5f0740 1 --2- 172.21.15.60:0/216108866 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1b29c00 0x1b2a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.899+0000 7f3e8a5f0740 1 --2- 172.21.15.60:0/216108866 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1b62f10 0x1b27940 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.899+0000 7f3e8a5f0740 1 -- 172.21.15.60:0/216108866 >> 172.21.15.60:0/216108866 conn(0x1b25b20 msgr2=0x1b62650 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:44.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.899+0000 7f3e8a5f0740 1 -- 172.21.15.60:0/216108866 shutdown_connections 2022-06-29T16:03:44.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.903+0000 7f3e8a5f0740 1 -- 172.21.15.60:0/216108866 wait complete. 2022-06-29T16:03:44.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.903+0000 7f3e8a5f0740 1 -- start start 2022-06-29T16:03:44.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.903+0000 7f3e8a5f0740 1 Processor -- start 2022-06-29T16:03:44.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.903+0000 7f3e8a5f0740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1b5d460 0x1b59e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:44.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.903+0000 7f3e8a5f0740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1b62f10 0x1b5a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:44.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.903+0000 7f3e8a5f0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1b5c580 0x1b5a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:44.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.903+0000 7f3e8a5f0740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x1b5ad20 con 0x1b62f10 2022-06-29T16:03:44.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.903+0000 7f3e8a5f0740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x1b6d570 con 0x1b5c580 2022-06-29T16:03:44.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.903+0000 7f3e8a5f0740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x1b5ca20 con 0x1b5d460 2022-06-29T16:03:44.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.903+0000 7f3e7d936700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1b62f10 0x1b5a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:44.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.903+0000 7f3e8a5f0740 10 client.0 ll_register_callbacks cb 0x1ae13d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-06-29T16:03:44.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.903+0000 7f3e7d936700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1b62f10 0x1b5a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48854/0 (socket says 172.21.15.60:48854) 2022-06-29T16:03:44.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.903+0000 7f3e7d936700 1 -- 172.21.15.60:0/2118126322 learned_addr learned my addr 172.21.15.60:0/2118126322 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:44.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.903+0000 7f3e7e137700 1 --2- 172.21.15.60:0/2118126322 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1b5d460 0x1b59e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:44.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.903+0000 7f3e7e938700 1 --2- 172.21.15.60:0/2118126322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1b5c580 0x1b5a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:44.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.903+0000 7f3e7d936700 1 -- 172.21.15.60:0/2118126322 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1b5d460 msgr2=0x1b59e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:44.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.903+0000 7f3e7d936700 1 --2- 172.21.15.60:0/2118126322 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1b5d460 0x1b59e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.903+0000 7f3e7d936700 1 -- 172.21.15.60:0/2118126322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1b5c580 msgr2=0x1b5a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:44.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.903+0000 7f3e7d936700 1 --2- 172.21.15.60:0/2118126322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1b5c580 0x1b5a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.903+0000 7f3e7d936700 1 -- 172.21.15.60:0/2118126322 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e700112c0 con 0x1b62f10 2022-06-29T16:03:44.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.903+0000 7f3e7d936700 1 --2- 172.21.15.60:0/2118126322 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1b62f10 0x1b5a340 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7f3e68023000 tx=0x7f3e6801f000).ready entity=mon.0 client_cookie=67ccc34ac92145c0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:44.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.903+0000 7f3e557fa700 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2022-06-29T16:03:44.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.903+0000 7f3e557fa700 1 -- 172.21.15.60:0/2118126322 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e68013e80 con 0x1b62f10 2022-06-29T16:03:44.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.903+0000 7f3e557fa700 1 -- 172.21.15.60:0/2118126322 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3e6802cc20 con 0x1b62f10 2022-06-29T16:03:44.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.903+0000 7f3e8a5f0740 1 -- 172.21.15.60:0/2118126322 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x1a6f6c0 con 0x1b62f10 2022-06-29T16:03:44.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.903+0000 7f3e557fa700 1 -- 172.21.15.60:0/2118126322 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e6802cde0 con 0x1b62f10 2022-06-29T16:03:44.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.903+0000 7f3e8a5f0740 10 client.6156 fetch_fsmap learned FSMap version 25 2022-06-29T16:03:44.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.903+0000 7f3e8a5f0740 1 -- 172.21.15.60:0/2118126322 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x1a893d0 con 0x1b62f10 2022-06-29T16:03:44.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.903+0000 7f3e557fa700 1 -- 172.21.15.60:0/2118126322 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x1a6f6c0 con 0x1b62f10 2022-06-29T16:03:44.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.907+0000 7f3e557fa700 1 -- 172.21.15.60:0/2118126322 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 25) v1 ==== 1247+0+0 (secure 0 0 0) 0x7f3e6801fb50 con 0x1b62f10 2022-06-29T16:03:44.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.907+0000 7f3e8a5f0740 10 client.6156 fetch_fsmap finished waiting for FSMap version 25 2022-06-29T16:03:44.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.907+0000 7f3e8a5f0740 10 client.6156 resolve_mds: resolved ID 'b' to GID 6013 2022-06-29T16:03:44.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.907+0000 7f3e8a5f0740 20 client.6156 populate_metadata read hostname 'smithi060' 2022-06-29T16:03:44.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.907+0000 7f3e8a5f0740 1 --2- 172.21.15.60:0/2118126322 >> [v2:172.21.15.92:6834/3876875590,v1:172.21.15.92:6835/3876875590] conn(0x1b58df0 0x1b591d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:44.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.907+0000 7f3e8a5f0740 4 client.6156 mds_command: new command op to 6013 tid=0[{"prefix": "get_command_descriptions"}] 2022-06-29T16:03:44.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.907+0000 7f3e8a5f0740 1 -- 172.21.15.60:0/2118126322 --> [v2:172.21.15.92:6834/3876875590,v1:172.21.15.92:6835/3876875590] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x1a8fde0 con 0x1b58df0 2022-06-29T16:03:44.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.907+0000 7f3e7e137700 1 --2- 172.21.15.60:0/2118126322 >> [v2:172.21.15.92:6834/3876875590,v1:172.21.15.92:6835/3876875590] conn(0x1b58df0 0x1b591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:44.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.907+0000 7f3e557fa700 10 client.6156 ms_handle_connect on v2:172.21.15.92:6834/3876875590 2022-06-29T16:03:44.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.907+0000 7f3e7e137700 1 --2- 172.21.15.60:0/2118126322 >> [v2:172.21.15.92:6834/3876875590,v1:172.21.15.92:6835/3876875590] conn(0x1b58df0 0x1b591d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=adb8d2bb4b459c16 server_cookie=bb69c36c4b95d04b in_seq=0 out_seq=0 2022-06-29T16:03:44.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.907+0000 7f3e557fa700 1 -- 172.21.15.60:0/2118126322 <== mds.0 v2:172.21.15.92:6834/3876875590 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17002 (crc 0 0 0) 0x1a8fde0 con 0x1b58df0 2022-06-29T16:03:44.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.907+0000 7f3e557fa700 10 client.6156 handle_command_reply: tid=0 2022-06-29T16:03:44.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.907+0000 7f3e8a5f0740 1 client.6156 shutdown 2022-06-29T16:03:44.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.907+0000 7f3e8a5f0740 1 -- 172.21.15.60:0/2118126322 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1b62f10 msgr2=0x1b5a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:44.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.907+0000 7f3e8a5f0740 1 --2- 172.21.15.60:0/2118126322 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1b62f10 0x1b5a340 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7f3e68023000 tx=0x7f3e6801f000).stop 2022-06-29T16:03:44.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.907+0000 7f3e8a5f0740 1 -- 172.21.15.60:0/2118126322 shutdown_connections 2022-06-29T16:03:44.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.907+0000 7f3e8a5f0740 1 --2- 172.21.15.60:0/2118126322 >> [v2:172.21.15.92:6834/3876875590,v1:172.21.15.92:6835/3876875590] conn(0x1b58df0 0x1b591d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.907+0000 7f3e8a5f0740 1 --2- 172.21.15.60:0/2118126322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1b5c580 0x1b5a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.907+0000 7f3e557fa700 0 client.6156 ms_handle_reset on v2:172.21.15.92:6834/3876875590 2022-06-29T16:03:44.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.907+0000 7f3e8a5f0740 1 --2- 172.21.15.60:0/2118126322 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1b62f10 0x1b5a340 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.907+0000 7f3e8a5f0740 1 --2- 172.21.15.60:0/2118126322 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1b5d460 0x1b59e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.907+0000 7f3e8a5f0740 1 -- 172.21.15.60:0/2118126322 >> 172.21.15.60:0/2118126322 conn(0x1b29c00 msgr2=0x1b625b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:44.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.907+0000 7f3e8a5f0740 1 -- 172.21.15.60:0/2118126322 shutdown_connections 2022-06-29T16:03:44.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.907+0000 7f3e8a5f0740 1 -- 172.21.15.60:0/2118126322 wait complete. 2022-06-29T16:03:44.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.907+0000 7f3e8a5f0740 20 client.6156 trim_cache size 0 max 16384 2022-06-29T16:03:44.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.923+0000 7f3e8a5f0740 1 Processor -- start 2022-06-29T16:03:44.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.923+0000 7f3e8a5f0740 1 -- start start 2022-06-29T16:03:44.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.923+0000 7f3e8a5f0740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1b5c580 0x1b5f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:44.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.923+0000 7f3e8a5f0740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1b5d460 0x1b2aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:44.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.923+0000 7f3e8a5f0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1b62f10 0x1b2b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:44.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.923+0000 7f3e8a5f0740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x1b6d420 con 0x1b5d460 2022-06-29T16:03:44.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.923+0000 7f3e8a5f0740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x1b25ff0 con 0x1b62f10 2022-06-29T16:03:44.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.923+0000 7f3e8a5f0740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x1b2db60 con 0x1b5c580 2022-06-29T16:03:44.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.923+0000 7f3e7d936700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1b5d460 0x1b2aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:44.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.923+0000 7f3e7d936700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1b5d460 0x1b2aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48862/0 (socket says 172.21.15.60:48862) 2022-06-29T16:03:44.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.923+0000 7f3e7d936700 1 -- 172.21.15.60:0/968426571 learned_addr learned my addr 172.21.15.60:0/968426571 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:44.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.923+0000 7f3e7e938700 1 --2- 172.21.15.60:0/968426571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1b62f10 0x1b2b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:44.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.923+0000 7f3e7e137700 1 --2- 172.21.15.60:0/968426571 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1b5c580 0x1b5f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:44.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.927+0000 7f3e7d936700 1 -- 172.21.15.60:0/968426571 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1b5c580 msgr2=0x1b5f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:44.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.927+0000 7f3e7d936700 1 --2- 172.21.15.60:0/968426571 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1b5c580 0x1b5f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.927+0000 7f3e7d936700 1 -- 172.21.15.60:0/968426571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1b62f10 msgr2=0x1b2b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:44.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.927+0000 7f3e7d936700 1 --2- 172.21.15.60:0/968426571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1b62f10 0x1b2b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.927+0000 7f3e7d936700 1 -- 172.21.15.60:0/968426571 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1a91fc0 con 0x1b5d460 2022-06-29T16:03:44.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.927+0000 7f3e7e938700 1 --2- 172.21.15.60:0/968426571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1b62f10 0x1b2b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:03:44.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.927+0000 7f3e7d936700 1 --2- 172.21.15.60:0/968426571 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1b5d460 0x1b2aff0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x1aa0f60 tx=0x7f3e6802a000).ready entity=mon.0 client_cookie=85fcb0d7098e4a5 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:44.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.927+0000 7f3e577fe700 1 -- 172.21.15.60:0/968426571 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e6800db00 con 0x1b5d460 2022-06-29T16:03:44.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.927+0000 7f3e577fe700 1 -- 172.21.15.60:0/968426571 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e68011490 con 0x1b5d460 2022-06-29T16:03:44.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.927+0000 7f3e577fe700 1 -- 172.21.15.60:0/968426571 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e680118d0 con 0x1b5d460 2022-06-29T16:03:44.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.927+0000 7f3e8a5f0740 1 -- 172.21.15.60:0/968426571 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1b5d460 msgr2=0x1b2aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:44.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.927+0000 7f3e8a5f0740 1 --2- 172.21.15.60:0/968426571 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1b5d460 0x1b2aff0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x1aa0f60 tx=0x7f3e6802a000).stop 2022-06-29T16:03:44.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.927+0000 7f3e8a5f0740 1 -- 172.21.15.60:0/968426571 shutdown_connections 2022-06-29T16:03:44.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.927+0000 7f3e8a5f0740 1 --2- 172.21.15.60:0/968426571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1b62f10 0x1b2b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.927+0000 7f3e8a5f0740 1 --2- 172.21.15.60:0/968426571 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1b5d460 0x1b2aff0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.927+0000 7f3e8a5f0740 1 --2- 172.21.15.60:0/968426571 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1b5c580 0x1b5f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.927+0000 7f3e8a5f0740 1 -- 172.21.15.60:0/968426571 >> 172.21.15.60:0/968426571 conn(0x1b29c00 msgr2=0x1b5dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:44.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.927+0000 7f3e8a5f0740 1 -- 172.21.15.60:0/968426571 shutdown_connections 2022-06-29T16:03:44.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.927+0000 7f3e8a5f0740 1 -- 172.21.15.60:0/968426571 wait complete. 2022-06-29T16:03:44.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.927+0000 7f3e8a5f0740 1 -- start start 2022-06-29T16:03:44.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.927+0000 7f3e8a5f0740 1 Processor -- start 2022-06-29T16:03:44.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e8a5f0740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1b5c580 0x1b69db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:44.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e8a5f0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1b5d460 0x1bc45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:44.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e8a5f0740 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1b62f10 0x1bc4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:44.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e8a5f0740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x1b7ce40 con 0x1b5c580 2022-06-29T16:03:44.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e8a5f0740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x1b69980 con 0x1b5d460 2022-06-29T16:03:44.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e8a5f0740 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x1b59d70 con 0x1b62f10 2022-06-29T16:03:44.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e8a5f0740 10 client.0 ll_register_callbacks cb 0x1b5f090 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-06-29T16:03:44.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e7e137700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1b5c580 0x1b69db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:44.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e7e137700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1b5c580 0x1b69db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48866/0 (socket says 172.21.15.60:48866) 2022-06-29T16:03:44.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e7e137700 1 -- 172.21.15.60:0/1723732467 learned_addr learned my addr 172.21.15.60:0/1723732467 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:44.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e7e938700 1 --2- 172.21.15.60:0/1723732467 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1b62f10 0x1bc4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:44.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e7d936700 1 --2- 172.21.15.60:0/1723732467 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1b5d460 0x1bc45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:44.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e7e137700 1 -- 172.21.15.60:0/1723732467 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1b62f10 msgr2=0x1bc4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:44.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e7e137700 1 --2- 172.21.15.60:0/1723732467 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1b62f10 0x1bc4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e7e137700 1 -- 172.21.15.60:0/1723732467 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1b5d460 msgr2=0x1bc45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:44.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e7e137700 1 --2- 172.21.15.60:0/1723732467 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1b5d460 0x1bc45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:44.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e7e137700 1 -- 172.21.15.60:0/1723732467 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e78067a50 con 0x1b5c580 2022-06-29T16:03:44.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e7e938700 1 --2- 172.21.15.60:0/1723732467 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1b62f10 0x1bc4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:03:44.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e7d936700 1 --2- 172.21.15.60:0/1723732467 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1b5d460 0x1bc45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:03:44.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e7e137700 1 --2- 172.21.15.60:0/1723732467 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1b5c580 0x1b69db0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7f3e70021000 tx=0x7f3e7001f040).ready entity=mon.0 client_cookie=fcd90b35277d7376 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:44.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e567fc700 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2022-06-29T16:03:44.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e567fc700 1 -- 172.21.15.60:0/1723732467 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e70006ea0 con 0x1b5c580 2022-06-29T16:03:44.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e567fc700 1 -- 172.21.15.60:0/1723732467 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3e70014d80 con 0x1b5c580 2022-06-29T16:03:44.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e8a5f0740 1 -- 172.21.15.60:0/1723732467 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3e700147b0 con 0x1b5c580 2022-06-29T16:03:44.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e567fc700 1 -- 172.21.15.60:0/1723732467 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e70028460 con 0x1b5c580 2022-06-29T16:03:44.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e8a5f0740 10 client.6162 fetch_fsmap learned FSMap version 25 2022-06-29T16:03:44.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e8a5f0740 1 -- 172.21.15.60:0/1723732467 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x1a893d0 con 0x1b5c580 2022-06-29T16:03:44.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e567fc700 1 -- 172.21.15.60:0/1723732467 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e700147b0 con 0x1b5c580 2022-06-29T16:03:44.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e567fc700 1 -- 172.21.15.60:0/1723732467 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 25) v1 ==== 1247+0+0 (secure 0 0 0) 0x7f3e70028b10 con 0x1b5c580 2022-06-29T16:03:44.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e8a5f0740 10 client.6162 fetch_fsmap finished waiting for FSMap version 25 2022-06-29T16:03:44.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e8a5f0740 10 client.6162 resolve_mds: resolved ID 'b' to GID 6013 2022-06-29T16:03:44.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e8a5f0740 20 client.6162 populate_metadata read hostname 'smithi060' 2022-06-29T16:03:44.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e8a5f0740 1 --2- 172.21.15.60:0/1723732467 >> [v2:172.21.15.92:6834/3876875590,v1:172.21.15.92:6835/3876875590] conn(0x1b2b6a0 0x1bc0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:44.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e8a5f0740 4 client.6162 mds_command: new command op to 6013 tid=0[{"prefix": "cache drop", "timeout": 10}] 2022-06-29T16:03:44.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e8a5f0740 1 -- 172.21.15.60:0/1723732467 --> [v2:172.21.15.92:6834/3876875590,v1:172.21.15.92:6835/3876875590] -- command(tid 0: {"prefix": "cache drop", "timeout": 10}) v1 -- 0x1a87e60 con 0x1b2b6a0 2022-06-29T16:03:44.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e7d936700 1 --2- 172.21.15.60:0/1723732467 >> [v2:172.21.15.92:6834/3876875590,v1:172.21.15.92:6835/3876875590] conn(0x1b2b6a0 0x1bc0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:44.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e567fc700 10 client.6162 ms_handle_connect on v2:172.21.15.92:6834/3876875590 2022-06-29T16:03:44.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:44.931+0000 7f3e7d936700 1 --2- 172.21.15.60:0/1723732467 >> [v2:172.21.15.92:6834/3876875590,v1:172.21.15.92:6835/3876875590] conn(0x1b2b6a0 0x1bc0470 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=94f043a40012ddb9 server_cookie=9efe0136700dabe2 in_seq=0 out_seq=0 2022-06-29T16:03:55.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.159+0000 7f3e567fc700 1 -- 172.21.15.60:0/1723732467 <== mds.0 v2:172.21.15.92:6834/3876875590 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+406 (crc 0 0 0) 0x1a87e60 con 0x1b2b6a0 2022-06-29T16:03:55.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.159+0000 7f3e567fc700 10 client.6162 handle_command_reply: tid=0 2022-06-29T16:03:55.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.159+0000 7f3e8a5f0740 1 client.6162 shutdown 2022-06-29T16:03:55.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.159+0000 7f3e8a5f0740 1 -- 172.21.15.60:0/1723732467 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1b5c580 msgr2=0x1b69db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:55.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.159+0000 7f3e8a5f0740 1 --2- 172.21.15.60:0/1723732467 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1b5c580 0x1b69db0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7f3e70021000 tx=0x7f3e7001f040).stop 2022-06-29T16:03:55.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.159+0000 7f3e8a5f0740 1 -- 172.21.15.60:0/1723732467 shutdown_connections 2022-06-29T16:03:55.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.159+0000 7f3e8a5f0740 1 --2- 172.21.15.60:0/1723732467 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x1b62f10 0x1bc4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:55.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.159+0000 7f3e8a5f0740 1 --2- 172.21.15.60:0/1723732467 >> [v2:172.21.15.92:6834/3876875590,v1:172.21.15.92:6835/3876875590] conn(0x1b2b6a0 0x1bc0470 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:55.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.159+0000 7f3e8a5f0740 1 --2- 172.21.15.60:0/1723732467 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x1b5d460 0x1bc45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:55.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.159+0000 7f3e8a5f0740 1 --2- 172.21.15.60:0/1723732467 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x1b5c580 0x1b69db0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:55.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.159+0000 7f3e567fc700 0 client.6162 ms_handle_reset on v2:172.21.15.92:6834/3876875590 2022-06-29T16:03:55.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.159+0000 7f3e8a5f0740 1 -- 172.21.15.60:0/1723732467 >> 172.21.15.60:0/1723732467 conn(0x1b29c00 msgr2=0x1b5b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:55.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.159+0000 7f3e8a5f0740 1 -- 172.21.15.60:0/1723732467 shutdown_connections 2022-06-29T16:03:55.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.159+0000 7f3e8a5f0740 1 -- 172.21.15.60:0/1723732467 wait complete. 2022-06-29T16:03:55.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.159+0000 7f3e8a5f0740 20 client.6162 trim_cache size 0 max 16384 2022-06-29T16:03:55.171 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:03:55.172 INFO:teuthology.orchestra.run.smithi060.stdout: "client_recall": { 2022-06-29T16:03:55.172 INFO:teuthology.orchestra.run.smithi060.stdout: "return_code": -110, 2022-06-29T16:03:55.172 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "(110) Connection timed out", 2022-06-29T16:03:55.172 INFO:teuthology.orchestra.run.smithi060.stdout: "recalled": 240 2022-06-29T16:03:55.172 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.173 INFO:teuthology.orchestra.run.smithi060.stdout: "flush_journal": { 2022-06-29T16:03:55.173 INFO:teuthology.orchestra.run.smithi060.stdout: "return_code": 0, 2022-06-29T16:03:55.173 INFO:teuthology.orchestra.run.smithi060.stdout: "message": "" 2022-06-29T16:03:55.173 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.174 INFO:teuthology.orchestra.run.smithi060.stdout: "trim_cache": { 2022-06-29T16:03:55.174 INFO:teuthology.orchestra.run.smithi060.stdout: "trimmed": 220 2022-06-29T16:03:55.174 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.174 INFO:teuthology.orchestra.run.smithi060.stdout: "cache": { 2022-06-29T16:03:55.175 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": { 2022-06-29T16:03:55.175 INFO:teuthology.orchestra.run.smithi060.stdout: "items": 15434, 2022-06-29T16:03:55.175 INFO:teuthology.orchestra.run.smithi060.stdout: "bytes": 2428352 2022-06-29T16:03:55.175 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:55.175 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.176 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 10.226731384000001 2022-06-29T16:03:55.176 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:03:55.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.163+0000 7f3e7f139700 1 -- 172.21.15.60:0/540847 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3e58053a10 msgr2=0x7f3e58055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:55.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.163+0000 7f3e7f139700 1 --2- 172.21.15.60:0/540847 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3e58053a10 0x7f3e58055ed0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f3e6801a000 tx=0x7f3e68014000).stop 2022-06-29T16:03:55.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.163+0000 7f3e7f139700 1 -- 172.21.15.60:0/540847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e78143e10 msgr2=0x7f3e78139b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:55.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.163+0000 7f3e7f139700 1 --2- 172.21.15.60:0/540847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e78143e10 0x7f3e78139b50 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0x7f3e74001110 tx=0x7f3e740079c0).stop 2022-06-29T16:03:55.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.163+0000 7f3e7f139700 1 -- 172.21.15.60:0/540847 shutdown_connections 2022-06-29T16:03:55.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.163+0000 7f3e7f139700 1 --2- 172.21.15.60:0/540847 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3e58053a10 0x7f3e58055ed0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:55.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.163+0000 7f3e7f139700 1 --2- 172.21.15.60:0/540847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e78143e10 0x7f3e78139b50 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:55.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.163+0000 7f3e7f139700 1 --2- 172.21.15.60:0/540847 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e780b2b40 0x7f3e78067560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:55.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.163+0000 7f3e7f139700 1 --2- 172.21.15.60:0/540847 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e780b0ae0 0x7f3e780b4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:55.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.163+0000 7f3e7f139700 1 -- 172.21.15.60:0/540847 >> 172.21.15.60:0/540847 conn(0x7f3e780b6120 msgr2=0x7f3e78142220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:55.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.163+0000 7f3e7f139700 1 -- 172.21.15.60:0/540847 shutdown_connections 2022-06-29T16:03:55.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.163+0000 7f3e7f139700 1 -- 172.21.15.60:0/540847 wait complete. 2022-06-29T16:03:55.184 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-06-29T16:03:55.184 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:03:55.202 DEBUG:teuthology.orchestra.run.smithi092:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:03:55.222 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:03:55.244 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:03:55.266 DEBUG:teuthology.orchestra.run.smithi197:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:03:55.288 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-06-29T16:03:55.289 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:03:55.289 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:03:55.290 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:03:55.298 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T16:03:55.299 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T16:03:55.299 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2022-06-29T16:03:55.300 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-06-29T16:03:55.300 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:03:55.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[12777]: fuse finished with error 0 and tester_r 0 2022-06-29T16:03:55.317 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:03:55.318 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:03:55.319 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:03:55.365 INFO:teuthology.orchestra.run.smithi121.stdout:ext2/ext3 2022-06-29T16:03:55.366 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-06-29T16:03:55.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:03:55.484 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:03:55.491 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:03:55.491 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:03:55.492 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:03:55.498 INFO:teuthology.orchestra.run.smithi156.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-06-29T16:03:55.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:03:55.500 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-06-29T16:03:55.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:03:55.501 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:03:55.550 INFO:teuthology.orchestra.run.smithi156.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-06-29T16:03:55.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:03:55.552 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:03:55.553 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:03:55.553 DEBUG:teuthology.orchestra.run.smithi197:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T16:03:55.560 INFO:teuthology.orchestra.run.smithi197.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-06-29T16:03:55.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:03:55.562 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-06-29T16:03:55.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:03:55.564 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T16:03:55.612 INFO:teuthology.orchestra.run.smithi197.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-06-29T16:03:55.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:03:55.614 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-06-29T16:03:55.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.719+0000 7fdd4e015700 1 Processor -- start 2022-06-29T16:03:55.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.719+0000 7fdd4e015700 1 -- start start 2022-06-29T16:03:55.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.719+0000 7fdd4e015700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdd4813f3d0 0x7fdd48060260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:55.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.719+0000 7fdd4e015700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd480b6e20 con 0x7fdd4813f7b0 2022-06-29T16:03:55.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.719+0000 7fdd4e015700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd481430e0 con 0x7fdd48143820 2022-06-29T16:03:55.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.719+0000 7fdd4e015700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd48142ef0 con 0x7fdd4813f3d0 2022-06-29T16:03:55.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.719+0000 7fdd47fff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fdd4813f7b0 0x7fdd4813fbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56962/0 (socket says 172.21.15.60:56962) 2022-06-29T16:03:55.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.719+0000 7fdd47fff700 1 -- 172.21.15.60:0/3003556608 learned_addr learned my addr 172.21.15.60:0/3003556608 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:03:55.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.719+0000 7fdd477fe700 1 --2- 172.21.15.60:0/3003556608 >> v2:172.21.15.92:3300/0 conn(0x7fdd4813f3d0 0x7fdd48060260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:55.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.719+0000 7fdd46ffd700 1 -- 172.21.15.60:0/3003556608 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2593214000 0 0) 0x7fdd480b6e20 con 0x7fdd4813f7b0 2022-06-29T16:03:55.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.719+0000 7fdd46ffd700 1 -- 172.21.15.60:0/3003556608 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd28003260 con 0x7fdd4813f7b0 2022-06-29T16:03:55.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.719+0000 7fdd46ffd700 1 -- 172.21.15.60:0/3003556608 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1002066979 0 0) 0x7fdd481430e0 con 0x7fdd48143820 2022-06-29T16:03:55.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.719+0000 7fdd46ffd700 1 -- 172.21.15.60:0/3003556608 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd480b6e20 con 0x7fdd48143820 2022-06-29T16:03:55.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.719+0000 7fdd46ffd700 1 -- 172.21.15.60:0/3003556608 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2584592866 0 0) 0x7fdd28003260 con 0x7fdd4813f7b0 2022-06-29T16:03:55.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.719+0000 7fdd46ffd700 1 -- 172.21.15.60:0/3003556608 >> v2:172.21.15.92:3300/0 conn(0x7fdd4813f3d0 msgr2=0x7fdd48060260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:55.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.719+0000 7fdd46ffd700 1 --2- 172.21.15.60:0/3003556608 >> v2:172.21.15.92:3300/0 conn(0x7fdd4813f3d0 0x7fdd48060260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:55.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.719+0000 7fdd46ffd700 1 -- 172.21.15.60:0/3003556608 >> v1:172.21.15.92:6789/0 conn(0x7fdd48143820 legacy=0x7fdd480b6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:55.734 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.719+0000 7fdd46ffd700 1 -- 172.21.15.60:0/3003556608 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd48143380 con 0x7fdd4813f7b0 2022-06-29T16:03:55.734 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.719+0000 7fdd46ffd700 1 -- 172.21.15.60:0/3003556608 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7fdd400019e0 con 0x7fdd4813f7b0 2022-06-29T16:03:55.734 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.719+0000 7fdd46ffd700 1 -- 172.21.15.60:0/3003556608 >> v1:172.21.15.60:6789/0 conn(0x7fdd4813f7b0 legacy=0x7fdd4813fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:55.734 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.719+0000 7fdd46ffd700 1 --2- 172.21.15.60:0/3003556608 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd280038e0 0x7fdd28003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:55.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.719+0000 7fdd46ffd700 1 --2- 172.21.15.60:0/3003556608 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd28004500 0x7fdd28004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:55.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.719+0000 7fdd46ffd700 1 --2- 172.21.15.60:0/3003556608 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdd28004e70 0x7fdd2800b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:55.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.719+0000 7fdd46ffd700 1 -- 172.21.15.60:0/3003556608 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2800bdd0 con 0x7fdd280038e0 2022-06-29T16:03:55.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.719+0000 7fdd46ffd700 1 -- 172.21.15.60:0/3003556608 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2800bf10 con 0x7fdd28004500 2022-06-29T16:03:55.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.719+0000 7fdd46ffd700 1 -- 172.21.15.60:0/3003556608 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2800c080 con 0x7fdd28004e70 2022-06-29T16:03:55.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.719+0000 7fdd477fe700 1 --2- 172.21.15.60:0/3003556608 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd280038e0 0x7fdd28003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:55.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.719+0000 7fdd3effd700 1 --2- 172.21.15.60:0/3003556608 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd28004500 0x7fdd28004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:55.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.719+0000 7fdd47fff700 1 --2- 172.21.15.60:0/3003556608 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdd28004e70 0x7fdd2800b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:55.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.719+0000 7fdd477fe700 1 -- 172.21.15.60:0/3003556608 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdd28004e70 msgr2=0x7fdd2800b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:55.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.719+0000 7fdd477fe700 1 --2- 172.21.15.60:0/3003556608 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdd28004e70 0x7fdd2800b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:55.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.719+0000 7fdd477fe700 1 -- 172.21.15.60:0/3003556608 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd28004500 msgr2=0x7fdd28004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:55.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.719+0000 7fdd477fe700 1 --2- 172.21.15.60:0/3003556608 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd28004500 0x7fdd28004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:55.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.723+0000 7fdd47fff700 1 -- 172.21.15.60:0/3003556608 reap_dead start 2022-06-29T16:03:55.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.723+0000 7fdd477fe700 1 -- 172.21.15.60:0/3003556608 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd28003260 con 0x7fdd280038e0 2022-06-29T16:03:55.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.723+0000 7fdd3effd700 1 --2- 172.21.15.60:0/3003556608 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd28004500 0x7fdd28004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:03:55.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.723+0000 7fdd477fe700 1 --2- 172.21.15.60:0/3003556608 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd280038e0 0x7fdd28003ce0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7fdd3800c000 tx=0x7fdd38006df0).ready entity=mon.0 client_cookie=f5ddb6b60699c0f1 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:55.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.723+0000 7fdd46ffd700 1 -- 172.21.15.60:0/3003556608 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd3800e830 con 0x7fdd280038e0 2022-06-29T16:03:55.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.723+0000 7fdd46ffd700 1 -- 172.21.15.60:0/3003556608 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd3800e990 con 0x7fdd280038e0 2022-06-29T16:03:55.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.723+0000 7fdd46ffd700 1 -- 172.21.15.60:0/3003556608 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd3800ecf0 con 0x7fdd280038e0 2022-06-29T16:03:55.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.723+0000 7fdd4e015700 1 -- 172.21.15.60:0/3003556608 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd280038e0 msgr2=0x7fdd28003ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:55.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.723+0000 7fdd4e015700 1 --2- 172.21.15.60:0/3003556608 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd280038e0 0x7fdd28003ce0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7fdd3800c000 tx=0x7fdd38006df0).stop 2022-06-29T16:03:55.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.723+0000 7fdd4e015700 1 -- 172.21.15.60:0/3003556608 shutdown_connections 2022-06-29T16:03:55.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.723+0000 7fdd4e015700 1 --2- 172.21.15.60:0/3003556608 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd280038e0 0x7fdd28003ce0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:55.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.723+0000 7fdd4e015700 1 -- 172.21.15.60:0/3003556608 >> 172.21.15.60:0/3003556608 conn(0x7fdd480643e0 msgr2=0x7fdd480b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:55.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.723+0000 7fdd4e015700 1 -- 172.21.15.60:0/3003556608 shutdown_connections 2022-06-29T16:03:55.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.723+0000 7fdd4e015700 1 -- 172.21.15.60:0/3003556608 wait complete. 2022-06-29T16:03:55.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.723+0000 7fdd4e015700 1 Processor -- start 2022-06-29T16:03:55.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.723+0000 7fdd4e015700 1 -- start start 2022-06-29T16:03:55.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.727+0000 7fdd4e015700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd280038e0 0x7fdd4813d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:55.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.727+0000 7fdd4e015700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd48064f20 0x7fdd48065910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:55.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.727+0000 7fdd4e015700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdd4814e210 0x7fdd480b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:55.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.727+0000 7fdd4e015700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd480ad610 con 0x7fdd48064f20 2022-06-29T16:03:55.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.727+0000 7fdd4e015700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd481512a0 con 0x7fdd280038e0 2022-06-29T16:03:55.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.727+0000 7fdd4e015700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4814dad0 con 0x7fdd4814e210 2022-06-29T16:03:55.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.727+0000 7fdd3effd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd48064f20 0x7fdd48065910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:55.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.727+0000 7fdd3effd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd48064f20 0x7fdd48065910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48888/0 (socket says 172.21.15.60:48888) 2022-06-29T16:03:55.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.727+0000 7fdd3effd700 1 -- 172.21.15.60:0/3424777023 learned_addr learned my addr 172.21.15.60:0/3424777023 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:55.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.727+0000 7fdd477fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd280038e0 0x7fdd4813d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:55.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.727+0000 7fdd47fff700 1 --2- 172.21.15.60:0/3424777023 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdd4814e210 0x7fdd480b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:55.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.727+0000 7fdd3effd700 1 -- 172.21.15.60:0/3424777023 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdd4814e210 msgr2=0x7fdd480b4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:55.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.727+0000 7fdd3effd700 1 --2- 172.21.15.60:0/3424777023 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdd4814e210 0x7fdd480b4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:55.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.727+0000 7fdd3effd700 1 -- 172.21.15.60:0/3424777023 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd280038e0 msgr2=0x7fdd4813d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:55.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.727+0000 7fdd3effd700 1 --2- 172.21.15.60:0/3424777023 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd280038e0 0x7fdd4813d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:55.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.727+0000 7fdd3effd700 1 -- 172.21.15.60:0/3424777023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd4814dd70 con 0x7fdd48064f20 2022-06-29T16:03:55.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.727+0000 7fdd477fe700 1 --2- 172.21.15.60:0/3424777023 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd280038e0 0x7fdd4813d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:03:55.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.727+0000 7fdd3effd700 1 --2- 172.21.15.60:0/3424777023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd48064f20 0x7fdd48065910 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0x7fdd3000d000 tx=0x7fdd30007a00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:55.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.727+0000 7fdd457fa700 1 -- 172.21.15.60:0/3424777023 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd300114c0 con 0x7fdd48064f20 2022-06-29T16:03:55.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.727+0000 7fdd4e015700 1 -- 172.21.15.60:0/3424777023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd48150f90 con 0x7fdd48064f20 2022-06-29T16:03:55.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.727+0000 7fdd457fa700 1 -- 172.21.15.60:0/3424777023 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd30011620 con 0x7fdd48064f20 2022-06-29T16:03:55.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.727+0000 7fdd457fa700 1 -- 172.21.15.60:0/3424777023 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd30011980 con 0x7fdd48064f20 2022-06-29T16:03:55.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.727+0000 7fdd4e015700 1 -- 172.21.15.60:0/3424777023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd48065e00 con 0x7fdd48064f20 2022-06-29T16:03:55.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.731+0000 7fdd4e015700 1 -- 172.21.15.60:0/3424777023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd4814df00 con 0x7fdd48064f20 2022-06-29T16:03:55.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.731+0000 7fdd457fa700 1 -- 172.21.15.60:0/3424777023 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdd3001b030 con 0x7fdd48064f20 2022-06-29T16:03:55.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.731+0000 7fdd457fa700 1 --2- 172.21.15.60:0/3424777023 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdd2c053b30 0x7fdd2c055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:55.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.731+0000 7fdd457fa700 1 -- 172.21.15.60:0/3424777023 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fdd3006bdc0 con 0x7fdd48064f20 2022-06-29T16:03:55.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.731+0000 7fdd477fe700 1 --2- 172.21.15.60:0/3424777023 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdd2c053b30 0x7fdd2c055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:55.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.731+0000 7fdd477fe700 1 --2- 172.21.15.60:0/3424777023 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdd2c053b30 0x7fdd2c055ff0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fdd3801d000 tx=0x7fdd38019000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:55.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.739+0000 7fdd457fa700 1 -- 172.21.15.60:0/3424777023 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdd30042bd0 con 0x7fdd48064f20 2022-06-29T16:03:55.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.867+0000 7fdd4e015700 1 -- 172.21.15.60:0/3424777023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fdd480573d0 con 0x7fdd48064f20 2022-06-29T16:03:55.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.867+0000 7fdd457fa700 1 -- 172.21.15.60:0/3424777023 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v62) v1 ==== 81+0+33648 (secure 0 0 0) 0x7fdd300461f0 con 0x7fdd48064f20 2022-06-29T16:03:55.877 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:03:55.877 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:03:55.878 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 62, 2022-06-29T16:03:55.878 INFO:teuthology.orchestra.run.smithi060.stdout: "fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", 2022-06-29T16:03:55.879 INFO:teuthology.orchestra.run.smithi060.stdout: "created": "2022-06-29T15:58:07.965719+0000", 2022-06-29T16:03:55.879 INFO:teuthology.orchestra.run.smithi060.stdout: "modified": "2022-06-29T16:03:29.116409+0000", 2022-06-29T16:03:55.879 INFO:teuthology.orchestra.run.smithi060.stdout: "last_up_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:03:55.880 INFO:teuthology.orchestra.run.smithi060.stdout: "last_in_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:03:55.880 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-06-29T16:03:55.880 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_num": 5799936, 2022-06-29T16:03:55.880 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_set": [ 2022-06-29T16:03:55.881 INFO:teuthology.orchestra.run.smithi060.stdout: "pglog_hardlimit", 2022-06-29T16:03:55.881 INFO:teuthology.orchestra.run.smithi060.stdout: "purged_snapdirs", 2022-06-29T16:03:55.881 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_deletes", 2022-06-29T16:03:55.882 INFO:teuthology.orchestra.run.smithi060.stdout: "sortbitwise" 2022-06-29T16:03:55.882 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:03:55.882 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_version": 4, 2022-06-29T16:03:55.882 INFO:teuthology.orchestra.run.smithi060.stdout: "full_ratio": 0.94999998807907104, 2022-06-29T16:03:55.883 INFO:teuthology.orchestra.run.smithi060.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-06-29T16:03:55.883 INFO:teuthology.orchestra.run.smithi060.stdout: "nearfull_ratio": 0.85000002384185791, 2022-06-29T16:03:55.883 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_snapshot": "", 2022-06-29T16:03:55.883 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_max": 12, 2022-06-29T16:03:55.884 INFO:teuthology.orchestra.run.smithi060.stdout: "max_osd": 8, 2022-06-29T16:03:55.884 INFO:teuthology.orchestra.run.smithi060.stdout: "require_min_compat_client": "jewel", 2022-06-29T16:03:55.884 INFO:teuthology.orchestra.run.smithi060.stdout: "min_compat_client": "jewel", 2022-06-29T16:03:55.885 INFO:teuthology.orchestra.run.smithi060.stdout: "require_osd_release": "octopus", 2022-06-29T16:03:55.885 INFO:teuthology.orchestra.run.smithi060.stdout: "pools": [ 2022-06-29T16:03:55.885 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:55.885 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2022-06-29T16:03:55.886 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "device_health_metrics", 2022-06-29T16:03:55.886 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:10.961127+0000", 2022-06-29T16:03:55.886 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:03:55.886 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:03:55.887 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:03:55.887 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:03:55.887 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:03:55.887 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:03:55.888 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:03:55.888 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:03:55.888 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 1, 2022-06-29T16:03:55.888 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 1, 2022-06-29T16:03:55.888 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 1, 2022-06-29T16:03:55.889 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 1, 2022-06-29T16:03:55.889 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 1, 2022-06-29T16:03:55.889 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:03:55.889 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:03:55.890 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:03:55.890 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:03:55.890 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:03:55.890 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:03:55.891 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:03:55.891 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.891 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "15", 2022-06-29T16:03:55.891 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:03:55.891 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:03:55.892 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:03:55.892 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:03:55.892 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:03:55.892 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:03:55.893 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:03:55.893 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:03:55.893 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:03:55.893 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:03:55.893 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:03:55.894 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:03:55.894 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:03:55.894 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:03:55.894 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:03:55.895 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:03:55.895 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:03:55.895 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:03:55.895 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:03:55.896 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:03:55.896 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:03:55.896 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:03:55.896 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:03:55.896 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:03:55.897 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:03:55.897 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:03:55.897 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.897 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:03:55.898 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:03:55.898 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:03:55.898 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:03:55.898 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:03:55.898 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:03:55.899 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:03:55.899 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:03:55.899 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:03:55.899 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:03:55.900 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:03:55.900 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:03:55.900 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 1 2022-06-29T16:03:55.900 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.901 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:03:55.901 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_devicehealth": {} 2022-06-29T16:03:55.901 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:55.901 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.901 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:55.902 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 2, 2022-06-29T16:03:55.902 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "rbd", 2022-06-29T16:03:55.903 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:19.184276+0000", 2022-06-29T16:03:55.903 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:03:55.903 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:03:55.904 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:03:55.904 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:03:55.904 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:03:55.905 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:03:55.905 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:03:55.906 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:03:55.906 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:03:55.906 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:03:55.907 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:03:55.907 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:03:55.907 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:03:55.908 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:03:55.908 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:03:55.909 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:03:55.909 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:03:55.909 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:03:55.910 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:03:55.910 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:03:55.911 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.911 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "18", 2022-06-29T16:03:55.911 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:03:55.912 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:03:55.912 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:03:55.912 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:03:55.912 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:03:55.912 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:03:55.913 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:03:55.913 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:03:55.913 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:03:55.913 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:03:55.914 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:03:55.914 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:03:55.914 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:03:55.914 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:03:55.914 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:03:55.915 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:03:55.915 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:03:55.915 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:03:55.915 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:03:55.916 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:03:55.916 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:03:55.916 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:03:55.916 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:03:55.916 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:03:55.917 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:03:55.917 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:03:55.917 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.917 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:03:55.918 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:03:55.918 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:03:55.918 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:03:55.918 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:03:55.919 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:03:55.919 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:03:55.919 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:03:55.919 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:03:55.920 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:03:55.920 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:03:55.920 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:03:55.920 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:03:55.920 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd": {} 2022-06-29T16:03:55.921 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:55.921 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.921 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:55.921 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 11, 2022-06-29T16:03:55.922 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "cephfs_metadata", 2022-06-29T16:03:55.922 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T16:03:26.621123+0000", 2022-06-29T16:03:55.922 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:03:55.922 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:03:55.922 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:03:55.923 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:03:55.923 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:03:55.923 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:03:55.923 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:03:55.924 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:03:55.924 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:03:55.924 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:03:55.924 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:03:55.925 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:03:55.925 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:03:55.925 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:03:55.925 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:03:55.925 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:03:55.926 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:03:55.926 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:03:55.926 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:03:55.926 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:03:55.927 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.927 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "61", 2022-06-29T16:03:55.927 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:03:55.927 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:03:55.928 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:03:55.928 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:03:55.928 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:03:55.928 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:03:55.929 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:03:55.929 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:03:55.929 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:03:55.929 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:03:55.929 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:03:55.930 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:03:55.930 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:03:55.930 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:03:55.930 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:03:55.931 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:03:55.931 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:03:55.931 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:03:55.931 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:03:55.931 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:03:55.932 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:03:55.932 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:03:55.932 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:03:55.932 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:03:55.933 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:03:55.933 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:03:55.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.933 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:03:55.933 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:03:55.934 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:03:55.934 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:03:55.934 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:03:55.934 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:03:55.935 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:03:55.935 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:03:55.935 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:03:55.935 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:03:55.936 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:03:55.936 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:03:55.936 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_bias": 4, 2022-06-29T16:03:55.936 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 16, 2022-06-29T16:03:55.936 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_priority": 5 2022-06-29T16:03:55.937 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.937 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:03:55.937 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs": { 2022-06-29T16:03:55.937 INFO:teuthology.orchestra.run.smithi060.stdout: "metadata": "cephfs" 2022-06-29T16:03:55.937 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:55.938 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:55.938 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.938 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:55.939 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 12, 2022-06-29T16:03:55.939 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "cephfs_data", 2022-06-29T16:03:55.939 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T16:03:27.220279+0000", 2022-06-29T16:03:55.939 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:03:55.939 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:03:55.940 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:03:55.940 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:03:55.940 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:03:55.940 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:03:55.941 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:03:55.941 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:03:55.941 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:03:55.941 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:03:55.941 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:03:55.942 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:03:55.942 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:03:55.942 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:03:55.942 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:03:55.943 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:03:55.943 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:03:55.943 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:03:55.943 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:03:55.943 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:03:55.944 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.944 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "62", 2022-06-29T16:03:55.944 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:03:55.944 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:03:55.945 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:03:55.945 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:03:55.945 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:03:55.945 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:03:55.945 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:03:55.946 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:03:55.946 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:03:55.946 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:03:55.946 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:03:55.947 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:03:55.947 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:03:55.947 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:03:55.947 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:03:55.948 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:03:55.948 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:03:55.948 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:03:55.948 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:03:55.948 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:03:55.949 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:03:55.949 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:03:55.949 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:03:55.949 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:03:55.950 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:03:55.950 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:03:55.950 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.950 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:03:55.951 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:03:55.951 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:03:55.951 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:03:55.951 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:03:55.952 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:03:55.952 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:03:55.952 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:03:55.952 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:03:55.952 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:03:55.953 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:03:55.953 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:03:55.953 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:03:55.953 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs": { 2022-06-29T16:03:55.954 INFO:teuthology.orchestra.run.smithi060.stdout: "data": "cephfs" 2022-06-29T16:03:55.954 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:55.954 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:55.954 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:55.955 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:03:55.955 INFO:teuthology.orchestra.run.smithi060.stdout: "osds": [ 2022-06-29T16:03:55.955 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:55.956 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:03:55.956 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "11e7ab9e-fce2-470f-8195-ea4c3c52df22", 2022-06-29T16:03:55.956 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:03:55.957 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:03:55.957 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:03:55.958 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:03:55.958 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:03:55.958 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:03:55.959 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:03:55.959 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 59, 2022-06-29T16:03:55.959 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:03:55.960 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:03:55.960 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:03:55.961 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:55.961 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:55.961 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:55.962 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6816", 2022-06-29T16:03:55.962 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:03:55.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:55.964 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:55.964 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6825", 2022-06-29T16:03:55.965 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:03:55.965 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:55.965 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:55.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.966 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:03:55.967 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:55.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:55.967 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:55.968 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6826", 2022-06-29T16:03:55.968 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:03:55.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:55.969 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:55.969 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6827", 2022-06-29T16:03:55.970 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:03:55.970 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:55.970 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:55.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.971 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:03:55.971 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:55.971 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:55.971 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:55.971 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6830", 2022-06-29T16:03:55.972 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:03:55.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:55.972 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:55.973 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6831", 2022-06-29T16:03:55.973 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:03:55.973 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:55.973 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:55.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.974 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:03:55.974 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:55.974 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:55.974 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:55.975 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6828", 2022-06-29T16:03:55.975 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:03:55.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:55.976 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:55.976 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6829", 2022-06-29T16:03:55.976 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:03:55.976 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:55.976 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:55.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.977 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6825/12890", 2022-06-29T16:03:55.977 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6827/12890", 2022-06-29T16:03:55.977 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6831/12890", 2022-06-29T16:03:55.978 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6829/12890", 2022-06-29T16:03:55.978 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:03:55.978 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:03:55.978 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:03:55.979 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:55.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.979 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:55.979 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:03:55.979 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "ca43f136-6f69-4431-a157-fbe073fb585c", 2022-06-29T16:03:55.980 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:03:55.980 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:03:55.980 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:03:55.980 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:03:55.981 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:03:55.981 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:03:55.981 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:03:55.981 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 59, 2022-06-29T16:03:55.981 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:03:55.982 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:03:55.982 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:03:55.982 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:55.982 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:55.983 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:55.983 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6815", 2022-06-29T16:03:55.983 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:03:55.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:55.984 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:55.984 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6818", 2022-06-29T16:03:55.984 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:03:55.984 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:55.985 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:55.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.985 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:03:55.985 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:55.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:55.986 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:55.986 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6819", 2022-06-29T16:03:55.986 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:03:55.986 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.987 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:55.987 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:55.987 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6820", 2022-06-29T16:03:55.987 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:03:55.988 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:55.988 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:55.988 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.988 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:03:55.988 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:55.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:55.989 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:55.989 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6823", 2022-06-29T16:03:55.989 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:03:55.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.990 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:55.990 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:55.990 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6824", 2022-06-29T16:03:55.991 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:03:55.991 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:55.991 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:55.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.991 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:03:55.992 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:55.992 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:55.992 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:55.993 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6821", 2022-06-29T16:03:55.993 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:03:55.993 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.993 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:55.994 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:55.994 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6822", 2022-06-29T16:03:55.994 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:03:55.994 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:55.995 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:55.995 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.995 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6818/12892", 2022-06-29T16:03:55.995 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6820/12892", 2022-06-29T16:03:55.996 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6824/12892", 2022-06-29T16:03:55.996 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6822/12892", 2022-06-29T16:03:55.996 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:03:55.996 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:03:55.996 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:03:55.997 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:55.997 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:55.997 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:55.997 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:03:55.998 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "74fac45f-5f07-44ec-a9cb-2644c123fd4a", 2022-06-29T16:03:55.998 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:03:55.998 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:03:55.998 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:03:55.998 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:03:55.999 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:03:55.999 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:03:55.999 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:03:55.999 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 58, 2022-06-29T16:03:56.000 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:03:56.000 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:03:56.000 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:03:56.000 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:56.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.001 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:56.001 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6808", 2022-06-29T16:03:56.001 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:03:56.001 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.002 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.002 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:56.002 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6809", 2022-06-29T16:03:56.002 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:03:56.003 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:56.003 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:56.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.003 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:03:56.004 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:56.004 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.004 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:56.004 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6810", 2022-06-29T16:03:56.004 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:03:56.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.005 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:56.005 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6811", 2022-06-29T16:03:56.006 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:03:56.006 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:56.006 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:56.006 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.006 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:03:56.007 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:56.007 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.007 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:56.007 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6814", 2022-06-29T16:03:56.008 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:03:56.008 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.008 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.008 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:56.009 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6817", 2022-06-29T16:03:56.009 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:03:56.009 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:56.009 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:56.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.010 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:03:56.010 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:56.010 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.010 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:56.011 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6812", 2022-06-29T16:03:56.011 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:03:56.011 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.011 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.012 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:56.012 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6813", 2022-06-29T16:03:56.012 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:03:56.012 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:56.013 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:56.013 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.013 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6809/12891", 2022-06-29T16:03:56.013 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6811/12891", 2022-06-29T16:03:56.013 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6817/12891", 2022-06-29T16:03:56.014 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6813/12891", 2022-06-29T16:03:56.014 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:03:56.014 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:03:56.014 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:03:56.015 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:56.015 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.015 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.015 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:03:56.016 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "2581d351-973a-4b4e-85ca-3658f0068765", 2022-06-29T16:03:56.016 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:03:56.016 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:03:56.016 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:03:56.016 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:03:56.017 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:03:56.017 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:03:56.017 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:03:56.017 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 50, 2022-06-29T16:03:56.018 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:03:56.018 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:03:56.018 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:03:56.018 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:56.019 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.019 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:56.019 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6800", 2022-06-29T16:03:56.019 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:03:56.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.020 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.020 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:56.020 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6801", 2022-06-29T16:03:56.020 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:03:56.021 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:56.021 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:56.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.021 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:03:56.022 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:56.022 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.022 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:56.023 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6802", 2022-06-29T16:03:56.023 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:03:56.023 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.023 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.023 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:56.024 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6803", 2022-06-29T16:03:56.024 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:03:56.024 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:56.024 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:56.025 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.025 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:03:56.025 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:56.025 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.026 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:56.026 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6806", 2022-06-29T16:03:56.026 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:03:56.027 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.027 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.027 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:56.028 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6807", 2022-06-29T16:03:56.028 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:03:56.028 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:56.028 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:56.028 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.029 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:03:56.029 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:56.029 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.029 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:56.030 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6804", 2022-06-29T16:03:56.030 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:03:56.030 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.030 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.031 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:56.031 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6805", 2022-06-29T16:03:56.031 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:03:56.031 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:56.032 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:56.032 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.032 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6801/12889", 2022-06-29T16:03:56.032 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6803/12889", 2022-06-29T16:03:56.033 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6807/12889", 2022-06-29T16:03:56.033 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6805/12889", 2022-06-29T16:03:56.033 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:03:56.034 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:03:56.034 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:03:56.034 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:56.035 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.035 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.035 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:03:56.035 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "3d234db9-87cd-4fe9-814f-0240a287d8b4", 2022-06-29T16:03:56.036 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:03:56.036 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:03:56.036 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:03:56.036 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:03:56.037 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:03:56.037 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:03:56.037 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:03:56.037 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 59, 2022-06-29T16:03:56.038 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:03:56.038 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:03:56.038 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:03:56.038 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:56.039 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.039 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:56.039 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6808", 2022-06-29T16:03:56.039 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:03:56.040 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.040 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.040 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:56.040 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6810", 2022-06-29T16:03:56.041 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:03:56.041 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:56.041 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:56.041 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.041 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:03:56.042 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:56.042 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.042 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:56.042 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6812", 2022-06-29T16:03:56.043 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:03:56.043 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.043 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.043 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:56.044 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6814", 2022-06-29T16:03:56.044 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:03:56.044 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:56.044 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:56.044 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.045 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:03:56.045 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:56.045 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.045 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:56.046 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6823", 2022-06-29T16:03:56.046 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:03:56.046 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.046 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.047 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:56.047 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6825", 2022-06-29T16:03:56.047 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:03:56.047 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:56.047 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:56.048 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.048 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:03:56.048 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:56.048 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.049 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:56.049 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6817", 2022-06-29T16:03:56.049 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:03:56.049 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.049 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.050 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:56.050 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6820", 2022-06-29T16:03:56.050 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:03:56.050 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:56.051 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:56.051 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.051 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6810/13249", 2022-06-29T16:03:56.051 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6814/13249", 2022-06-29T16:03:56.051 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6825/13249", 2022-06-29T16:03:56.052 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6820/13249", 2022-06-29T16:03:56.052 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:03:56.052 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:03:56.052 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:03:56.053 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:56.053 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.053 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.053 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:03:56.053 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "e4bf1104-ff5c-438b-b608-8cee31c163f6", 2022-06-29T16:03:56.054 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:03:56.054 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:03:56.054 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:03:56.054 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:03:56.055 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:03:56.055 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:03:56.055 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:03:56.055 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 58, 2022-06-29T16:03:56.055 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:03:56.056 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:03:56.056 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:03:56.056 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:56.056 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.057 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:56.057 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6809", 2022-06-29T16:03:56.057 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:03:56.057 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.057 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.058 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:56.058 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6811", 2022-06-29T16:03:56.058 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:03:56.058 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:56.059 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:56.059 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.059 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:03:56.059 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:56.059 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.060 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:56.060 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6813", 2022-06-29T16:03:56.060 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:03:56.060 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.061 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.061 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:56.061 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6815", 2022-06-29T16:03:56.061 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:03:56.061 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:56.062 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:56.062 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.062 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:03:56.062 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:56.063 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.063 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:56.063 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6824", 2022-06-29T16:03:56.063 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:03:56.064 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.064 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.064 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:56.064 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6827", 2022-06-29T16:03:56.065 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:03:56.065 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:56.065 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:56.065 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.066 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:03:56.066 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:56.066 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.066 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:56.067 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6818", 2022-06-29T16:03:56.067 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:03:56.067 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.067 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.068 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:56.068 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6821", 2022-06-29T16:03:56.068 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:03:56.068 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:56.069 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:56.069 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.069 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6811/13248", 2022-06-29T16:03:56.069 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6815/13248", 2022-06-29T16:03:56.070 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6827/13248", 2022-06-29T16:03:56.070 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6821/13248", 2022-06-29T16:03:56.070 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:03:56.071 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:03:56.071 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:03:56.071 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:56.071 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.072 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.072 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:03:56.072 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "8564dd25-27ab-4203-b54e-af6d3aa05857", 2022-06-29T16:03:56.072 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:03:56.073 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:03:56.073 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:03:56.073 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:03:56.073 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:03:56.074 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:03:56.074 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:03:56.074 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 59, 2022-06-29T16:03:56.075 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:03:56.075 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:03:56.075 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:03:56.075 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:56.076 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.076 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:56.076 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6816", 2022-06-29T16:03:56.076 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:03:56.077 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.077 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.077 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:56.077 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6819", 2022-06-29T16:03:56.078 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:03:56.078 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:56.078 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:56.079 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.079 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:03:56.079 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:56.079 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.080 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:56.080 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6822", 2022-06-29T16:03:56.080 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:03:56.080 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.081 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.081 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:56.081 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6826", 2022-06-29T16:03:56.081 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:03:56.082 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:56.082 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:56.082 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.082 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:03:56.082 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:56.083 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.083 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:56.083 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6830", 2022-06-29T16:03:56.083 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:03:56.084 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.084 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.084 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:56.084 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6831", 2022-06-29T16:03:56.084 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:03:56.085 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:56.085 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:56.085 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.085 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:03:56.086 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:56.086 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.086 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:56.086 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6828", 2022-06-29T16:03:56.086 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:03:56.087 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.087 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.087 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:56.087 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6829", 2022-06-29T16:03:56.088 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:03:56.088 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:56.088 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:56.088 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.089 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6819/13250", 2022-06-29T16:03:56.089 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6826/13250", 2022-06-29T16:03:56.089 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6831/13250", 2022-06-29T16:03:56.090 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6829/13250", 2022-06-29T16:03:56.090 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:03:56.091 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:03:56.091 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:03:56.092 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:56.092 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.093 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.093 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:03:56.093 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "bd513a9c-3b19-4314-9897-d300b5c897bb", 2022-06-29T16:03:56.094 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:03:56.094 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:03:56.095 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:03:56.095 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:03:56.096 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:03:56.096 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:03:56.096 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:03:56.097 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 58, 2022-06-29T16:03:56.097 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:03:56.098 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:03:56.098 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:03:56.099 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:56.099 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.100 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:56.100 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6800", 2022-06-29T16:03:56.100 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:03:56.101 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.101 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.102 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:56.102 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6801", 2022-06-29T16:03:56.103 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:03:56.103 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:56.104 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:56.104 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.104 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:03:56.105 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:56.105 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.106 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:56.106 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6802", 2022-06-29T16:03:56.107 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:03:56.107 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.107 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.107 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:56.108 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6803", 2022-06-29T16:03:56.108 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:03:56.108 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:56.108 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:56.109 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.109 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:03:56.109 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:56.109 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.109 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:56.110 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6806", 2022-06-29T16:03:56.110 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:03:56.110 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.110 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.111 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:56.111 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6807", 2022-06-29T16:03:56.111 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:03:56.111 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:56.112 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:56.112 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.112 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:03:56.112 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:03:56.112 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.113 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:03:56.113 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6804", 2022-06-29T16:03:56.113 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:03:56.113 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.114 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.114 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:03:56.114 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6805", 2022-06-29T16:03:56.114 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:03:56.114 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:56.115 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:56.115 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.115 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6801/13247", 2022-06-29T16:03:56.115 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6803/13247", 2022-06-29T16:03:56.116 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6807/13247", 2022-06-29T16:03:56.116 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6805/13247", 2022-06-29T16:03:56.116 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:03:56.116 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:03:56.116 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:03:56.117 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:03:56.117 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:56.117 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:03:56.117 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_xinfo": [ 2022-06-29T16:03:56.118 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.118 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:03:56.118 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:03:56.118 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:03:56.118 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:03:56.119 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:03:56.119 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:03:56.119 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.891342+0000", 2022-06-29T16:03:56.119 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:03:56.120 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.120 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.120 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:03:56.120 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:03:56.120 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:03:56.121 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:03:56.121 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:03:56.121 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:03:56.121 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.900866+0000", 2022-06-29T16:03:56.122 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:03:56.122 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.122 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.122 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:03:56.123 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:03:56.123 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:03:56.123 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:03:56.123 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:03:56.123 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:03:56.124 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.912755+0000", 2022-06-29T16:03:56.124 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:03:56.124 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.124 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.125 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:03:56.125 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:03:56.125 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:03:56.125 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:03:56.125 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:03:56.126 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:03:56.126 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.864332+0000", 2022-06-29T16:03:56.126 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:03:56.126 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.127 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.127 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:03:56.127 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:03:56.127 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:03:56.128 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:03:56.128 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:03:56.128 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:03:56.128 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.863756+0000", 2022-06-29T16:03:56.129 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:03:56.129 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.129 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.129 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:03:56.130 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:03:56.130 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:03:56.130 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:03:56.130 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:03:56.131 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:03:56.131 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.869041+0000", 2022-06-29T16:03:56.131 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:03:56.131 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.131 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.132 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:03:56.132 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:03:56.132 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:03:56.132 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:03:56.133 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:03:56.133 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:03:56.133 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.895043+0000", 2022-06-29T16:03:56.133 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:03:56.134 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.134 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:03:56.134 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:03:56.134 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:03:56.135 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:03:56.135 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:03:56.135 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:03:56.135 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:03:56.135 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.889491+0000", 2022-06-29T16:03:56.136 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:03:56.136 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:56.136 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:03:56.136 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap": [], 2022-06-29T16:03:56.137 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap_items": [], 2022-06-29T16:03:56.137 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_temp": [], 2022-06-29T16:03:56.137 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_temp": [], 2022-06-29T16:03:56.137 INFO:teuthology.orchestra.run.smithi060.stdout: "blacklist": {}, 2022-06-29T16:03:56.138 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profiles": { 2022-06-29T16:03:56.138 INFO:teuthology.orchestra.run.smithi060.stdout: "default": { 2022-06-29T16:03:56.138 INFO:teuthology.orchestra.run.smithi060.stdout: "crush-failure-domain": "osd", 2022-06-29T16:03:56.138 INFO:teuthology.orchestra.run.smithi060.stdout: "k": "2", 2022-06-29T16:03:56.138 INFO:teuthology.orchestra.run.smithi060.stdout: "m": "1", 2022-06-29T16:03:56.139 INFO:teuthology.orchestra.run.smithi060.stdout: "plugin": "jerasure", 2022-06-29T16:03:56.139 INFO:teuthology.orchestra.run.smithi060.stdout: "ruleset-failure-domain": "osd", 2022-06-29T16:03:56.139 INFO:teuthology.orchestra.run.smithi060.stdout: "technique": "reed_sol_van" 2022-06-29T16:03:56.139 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:03:56.140 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:03:56.140 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps_queue": [], 2022-06-29T16:03:56.140 INFO:teuthology.orchestra.run.smithi060.stdout: "new_removed_snaps": [], 2022-06-29T16:03:56.140 INFO:teuthology.orchestra.run.smithi060.stdout: "new_purged_snaps": [], 2022-06-29T16:03:56.140 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_node_flags": {}, 2022-06-29T16:03:56.141 INFO:teuthology.orchestra.run.smithi060.stdout: "device_class_flags": {} 2022-06-29T16:03:56.141 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:03:56.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.871+0000 7fdd4e015700 1 -- 172.21.15.60:0/3424777023 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdd2c053b30 msgr2=0x7fdd2c055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:56.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.871+0000 7fdd4e015700 1 --2- 172.21.15.60:0/3424777023 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdd2c053b30 0x7fdd2c055ff0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fdd3801d000 tx=0x7fdd38019000).stop 2022-06-29T16:03:56.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.871+0000 7fdd4e015700 1 -- 172.21.15.60:0/3424777023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd48064f20 msgr2=0x7fdd48065910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:56.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.871+0000 7fdd4e015700 1 --2- 172.21.15.60:0/3424777023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd48064f20 0x7fdd48065910 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0x7fdd3000d000 tx=0x7fdd30007a00).stop 2022-06-29T16:03:56.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.871+0000 7fdd4e015700 1 -- 172.21.15.60:0/3424777023 shutdown_connections 2022-06-29T16:03:56.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.871+0000 7fdd4e015700 1 --2- 172.21.15.60:0/3424777023 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdd2c053b30 0x7fdd2c055ff0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:56.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.871+0000 7fdd4e015700 1 --2- 172.21.15.60:0/3424777023 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdd4814e210 0x7fdd480b4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:56.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.871+0000 7fdd4e015700 1 --2- 172.21.15.60:0/3424777023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd48064f20 0x7fdd48065910 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:56.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.871+0000 7fdd4e015700 1 --2- 172.21.15.60:0/3424777023 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd280038e0 0x7fdd4813d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:56.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.871+0000 7fdd4e015700 1 -- 172.21.15.60:0/3424777023 >> 172.21.15.60:0/3424777023 conn(0x7fdd480643e0 msgr2=0x7fdd4813b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:56.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.871+0000 7fdd4e015700 1 -- 172.21.15.60:0/3424777023 shutdown_connections 2022-06-29T16:03:56.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:55.871+0000 7fdd4e015700 1 -- 172.21.15.60:0/3424777023 wait complete. 2022-06-29T16:03:56.146 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:03:56.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.251+0000 7fe689e07700 1 Processor -- start 2022-06-29T16:03:56.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.251+0000 7fe689e07700 1 -- start start 2022-06-29T16:03:56.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.251+0000 7fe689e07700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe6840ae7c0 0x7fe684141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:56.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.251+0000 7fe689e07700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6840b4010 0x7fe684142490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:56.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.251+0000 7fe689e07700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe684143a40 con 0x7fe684146160 2022-06-29T16:03:56.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.251+0000 7fe689e07700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6840b56d0 con 0x7fe6840ae7c0 2022-06-29T16:03:56.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.251+0000 7fe689e07700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6840aee60 con 0x7fe6840b4010 2022-06-29T16:03:56.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.251+0000 7fe6837fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe6840ae7c0 0x7fe684141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:56.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.251+0000 7fe6837fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe6840ae7c0 0x7fe684141fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48894/0 (socket says 172.21.15.60:48894) 2022-06-29T16:03:56.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.251+0000 7fe6837fe700 1 -- 172.21.15.60:0/3681332727 learned_addr learned my addr 172.21.15.60:0/3681332727 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:56.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.251+0000 7fe682ffd700 1 --2- 172.21.15.60:0/3681332727 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6840b4010 0x7fe684142490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:56.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.251+0000 7fe6827fc700 1 -- 172.21.15.60:0/3681332727 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 582952203 0 0) 0x7fe684143a40 con 0x7fe684146160 2022-06-29T16:03:56.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.251+0000 7fe6827fc700 1 -- 172.21.15.60:0/3681332727 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe664003260 con 0x7fe684146160 2022-06-29T16:03:56.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.251+0000 7fe6837fe700 1 -- 172.21.15.60:0/3681332727 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6840b4010 msgr2=0x7fe684142490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:56.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.251+0000 7fe6837fe700 1 --2- 172.21.15.60:0/3681332727 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6840b4010 0x7fe684142490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:56.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.251+0000 7fe6837fe700 1 -- 172.21.15.60:0/3681332727 >> v1:172.21.15.92:6789/0 conn(0x7fe684146160 legacy=0x7fe6841465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:56.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.251+0000 7fe6837fe700 1 -- 172.21.15.60:0/3681332727 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6840af100 con 0x7fe6840ae7c0 2022-06-29T16:03:56.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.251+0000 7fe6837fe700 1 --2- 172.21.15.60:0/3681332727 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6840ae7c0 0x7fe684141fa0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7fe66c002f70 tx=0x7fe66c006eb0).ready entity=mon.0 client_cookie=ed6277660195ca0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:56.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.251+0000 7fe6827fc700 1 -- 172.21.15.60:0/3681332727 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe66c00d5c0 con 0x7fe6840ae7c0 2022-06-29T16:03:56.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.251+0000 7fe6827fc700 1 -- 172.21.15.60:0/3681332727 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe66c000e90 con 0x7fe6840ae7c0 2022-06-29T16:03:56.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.251+0000 7fe6827fc700 1 -- 172.21.15.60:0/3681332727 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe66c00d950 con 0x7fe6840ae7c0 2022-06-29T16:03:56.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.255+0000 7fe689e07700 1 -- 172.21.15.60:0/3681332727 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6840ae7c0 msgr2=0x7fe684141fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:56.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.255+0000 7fe689e07700 1 --2- 172.21.15.60:0/3681332727 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6840ae7c0 0x7fe684141fa0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7fe66c002f70 tx=0x7fe66c006eb0).stop 2022-06-29T16:03:56.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.255+0000 7fe689e07700 1 -- 172.21.15.60:0/3681332727 shutdown_connections 2022-06-29T16:03:56.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.255+0000 7fe689e07700 1 --2- 172.21.15.60:0/3681332727 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6840b4010 0x7fe684142490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:56.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.255+0000 7fe689e07700 1 --2- 172.21.15.60:0/3681332727 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6840ae7c0 0x7fe684141fa0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:56.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.255+0000 7fe689e07700 1 -- 172.21.15.60:0/3681332727 >> 172.21.15.60:0/3681332727 conn(0x7fe6840b9340 msgr2=0x7fe6840b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:56.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.255+0000 7fe689e07700 1 -- 172.21.15.60:0/3681332727 shutdown_connections 2022-06-29T16:03:56.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.255+0000 7fe689e07700 1 -- 172.21.15.60:0/3681332727 wait complete. 2022-06-29T16:03:56.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.255+0000 7fe689e07700 1 Processor -- start 2022-06-29T16:03:56.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.255+0000 7fe689e07700 1 -- start start 2022-06-29T16:03:56.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.255+0000 7fe689e07700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6840ae7c0 0x7fe684140520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:56.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.255+0000 7fe689e07700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6840b4010 0x7fe6840b7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:56.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.255+0000 7fe689e07700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe684146160 0x7fe684151d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:56.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.255+0000 7fe689e07700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6840af900 con 0x7fe6840b4010 2022-06-29T16:03:56.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.255+0000 7fe689e07700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6840af660 con 0x7fe6840ae7c0 2022-06-29T16:03:56.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.255+0000 7fe689e07700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6841553c0 con 0x7fe684146160 2022-06-29T16:03:56.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.259+0000 7fe682ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6840b4010 0x7fe6840b7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:56.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.259+0000 7fe682ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6840b4010 0x7fe6840b7660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48902/0 (socket says 172.21.15.60:48902) 2022-06-29T16:03:56.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.259+0000 7fe682ffd700 1 -- 172.21.15.60:0/2153320598 learned_addr learned my addr 172.21.15.60:0/2153320598 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:56.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.259+0000 7fe6837fe700 1 --2- 172.21.15.60:0/2153320598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6840ae7c0 0x7fe684140520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:56.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.259+0000 7fe683fff700 1 --2- 172.21.15.60:0/2153320598 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe684146160 0x7fe684151d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:56.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.259+0000 7fe682ffd700 1 -- 172.21.15.60:0/2153320598 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe684146160 msgr2=0x7fe684151d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:56.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.259+0000 7fe682ffd700 1 --2- 172.21.15.60:0/2153320598 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe684146160 0x7fe684151d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:56.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.259+0000 7fe682ffd700 1 -- 172.21.15.60:0/2153320598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6840ae7c0 msgr2=0x7fe684140520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:56.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.259+0000 7fe682ffd700 1 --2- 172.21.15.60:0/2153320598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6840ae7c0 0x7fe684140520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:56.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.259+0000 7fe682ffd700 1 -- 172.21.15.60:0/2153320598 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe66c00b1e0 con 0x7fe6840b4010 2022-06-29T16:03:56.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.259+0000 7fe682ffd700 1 --2- 172.21.15.60:0/2153320598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6840b4010 0x7fe6840b7660 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7fe67400c000 tx=0x7fe674006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:56.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.259+0000 7fe680ff9700 1 -- 172.21.15.60:0/2153320598 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe674010780 con 0x7fe6840b4010 2022-06-29T16:03:56.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.259+0000 7fe680ff9700 1 -- 172.21.15.60:0/2153320598 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6740108e0 con 0x7fe6840b4010 2022-06-29T16:03:56.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.259+0000 7fe689e07700 1 -- 172.21.15.60:0/2153320598 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe684152250 con 0x7fe6840b4010 2022-06-29T16:03:56.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.259+0000 7fe680ff9700 1 -- 172.21.15.60:0/2153320598 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe674010c40 con 0x7fe6840b4010 2022-06-29T16:03:56.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.259+0000 7fe689e07700 1 -- 172.21.15.60:0/2153320598 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6840b31a0 con 0x7fe6840b4010 2022-06-29T16:03:56.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.259+0000 7fe689e07700 1 -- 172.21.15.60:0/2153320598 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6840573d0 con 0x7fe6840b4010 2022-06-29T16:03:56.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.259+0000 7fe680ff9700 1 -- 172.21.15.60:0/2153320598 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe67401c460 con 0x7fe6840b4010 2022-06-29T16:03:56.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.259+0000 7fe680ff9700 1 --2- 172.21.15.60:0/2153320598 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe668053aa0 0x7fe668055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:56.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.263+0000 7fe680ff9700 1 -- 172.21.15.60:0/2153320598 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe67406c270 con 0x7fe6840b4010 2022-06-29T16:03:56.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.263+0000 7fe6837fe700 1 --2- 172.21.15.60:0/2153320598 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe668053aa0 0x7fe668055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:56.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.263+0000 7fe6837fe700 1 --2- 172.21.15.60:0/2153320598 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe668053aa0 0x7fe668055f60 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fe66c01c000 tx=0x7fe66c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:56.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.267+0000 7fe680ff9700 1 -- 172.21.15.60:0/2153320598 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe674043130 con 0x7fe6840b4010 2022-06-29T16:03:56.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.427+0000 7fe689e07700 1 -- 172.21.15.60:0/2153320598 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe68405fdb0 con 0x7fe6840b4010 2022-06-29T16:03:56.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.431+0000 7fe680ff9700 1 -- 172.21.15.60:0/2153320598 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+2134 (secure 0 0 0) 0x7fe674046750 con 0x7fe6840b4010 2022-06-29T16:03:56.437 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:03:56.437 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":25,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5955,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.60:6833/66690512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":66690512},{"type":"v1","addr":"172.21.15.60:6833","nonce":66690512}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":23}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:03:28.112667+0000","modified":"2022-06-29T16:03:29.115727+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6013},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6013":{"gid":6013,"name":"b","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/3876875590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3876875590},{"type":"v1","addr":"172.21.15.92:6835","nonce":3876875590}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-06-29T16:03:56.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.431+0000 7fe689e07700 1 -- 172.21.15.60:0/2153320598 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe668053aa0 msgr2=0x7fe668055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:56.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.431+0000 7fe689e07700 1 --2- 172.21.15.60:0/2153320598 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe668053aa0 0x7fe668055f60 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fe66c01c000 tx=0x7fe66c015000).stop 2022-06-29T16:03:56.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.431+0000 7fe689e07700 1 -- 172.21.15.60:0/2153320598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6840b4010 msgr2=0x7fe6840b7660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:56.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.431+0000 7fe689e07700 1 --2- 172.21.15.60:0/2153320598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6840b4010 0x7fe6840b7660 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7fe67400c000 tx=0x7fe674006df0).stop 2022-06-29T16:03:56.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.431+0000 7fe689e07700 1 -- 172.21.15.60:0/2153320598 shutdown_connections 2022-06-29T16:03:56.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.431+0000 7fe689e07700 1 --2- 172.21.15.60:0/2153320598 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe668053aa0 0x7fe668055f60 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:56.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.431+0000 7fe689e07700 1 --2- 172.21.15.60:0/2153320598 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe684146160 0x7fe684151d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:56.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.431+0000 7fe689e07700 1 --2- 172.21.15.60:0/2153320598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6840b4010 0x7fe6840b7660 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:56.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.431+0000 7fe689e07700 1 --2- 172.21.15.60:0/2153320598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6840ae7c0 0x7fe684140520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:56.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.431+0000 7fe689e07700 1 -- 172.21.15.60:0/2153320598 >> 172.21.15.60:0/2153320598 conn(0x7fe6840b9340 msgr2=0x7fe684141d70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:56.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.431+0000 7fe689e07700 1 -- 172.21.15.60:0/2153320598 shutdown_connections 2022-06-29T16:03:56.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.431+0000 7fe689e07700 1 -- 172.21.15.60:0/2153320598 wait complete. 2022-06-29T16:03:56.443 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 25 2022-06-29T16:03:56.449 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-06-29T16:03:56.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.547+0000 7fc2b359e700 1 Processor -- start 2022-06-29T16:03:56.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.547+0000 7fc2b359e700 1 -- start start 2022-06-29T16:03:56.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.547+0000 7fc2b359e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc2b405d070 0x7fc2b414aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:56.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.547+0000 7fc2b359e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2b40b2130 0x7fc2b40aca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:56.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.547+0000 7fc2b359e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc2b414af80 0x7fc2b40acf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:56.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.547+0000 7fc2b359e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2b40adf90 con 0x7fc2b414af80 2022-06-29T16:03:56.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.547+0000 7fc2b359e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2b405d450 con 0x7fc2b405d070 2022-06-29T16:03:56.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.547+0000 7fc2b359e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b40b25d0 con 0x7fc2b40b2130 2022-06-29T16:03:56.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.547+0000 7fc2b2d9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc2b414af80 0x7fc2b40acf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:56.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.547+0000 7fc2b2d9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc2b414af80 0x7fc2b40acf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48912/0 (socket says 172.21.15.60:48912) 2022-06-29T16:03:56.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.547+0000 7fc2b2d9d700 1 -- 172.21.15.60:0/767407525 learned_addr learned my addr 172.21.15.60:0/767407525 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:56.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.547+0000 7fc2b1d9b700 1 --2- 172.21.15.60:0/767407525 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2b40b2130 0x7fc2b40aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:56.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.547+0000 7fc2b259c700 1 --2- 172.21.15.60:0/767407525 >> v2:172.21.15.92:3300/0 conn(0x7fc2b405d070 0x7fc2b414aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:56.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.547+0000 7fc2b2d9d700 1 -- 172.21.15.60:0/767407525 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2b40b2130 msgr2=0x7fc2b40aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:56.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.547+0000 7fc2b2d9d700 1 --2- 172.21.15.60:0/767407525 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2b40b2130 0x7fc2b40aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:56.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.547+0000 7fc2b2d9d700 1 -- 172.21.15.60:0/767407525 >> v2:172.21.15.92:3300/0 conn(0x7fc2b405d070 msgr2=0x7fc2b414aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:56.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.547+0000 7fc2b2d9d700 1 --2- 172.21.15.60:0/767407525 >> v2:172.21.15.92:3300/0 conn(0x7fc2b405d070 0x7fc2b414aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:56.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.547+0000 7fc2b2d9d700 1 -- 172.21.15.60:0/767407525 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b414a350 con 0x7fc2b414af80 2022-06-29T16:03:56.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.547+0000 7fc2b2d9d700 1 --2- 172.21.15.60:0/767407525 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2b414af80 0x7fc2b40acf10 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7fc2ac0032f0 tx=0x7fc2ac009290).ready entity=mon.0 client_cookie=bfe3d10d8bd76a12 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:56.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.547+0000 7fc2b159a700 1 -- 172.21.15.60:0/767407525 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2ac00b870 con 0x7fc2b414af80 2022-06-29T16:03:56.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.547+0000 7fc2b159a700 1 -- 172.21.15.60:0/767407525 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2ac00b9d0 con 0x7fc2b414af80 2022-06-29T16:03:56.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.547+0000 7fc2b159a700 1 -- 172.21.15.60:0/767407525 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2ac00bb40 con 0x7fc2b414af80 2022-06-29T16:03:56.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.547+0000 7fc2b359e700 1 -- 172.21.15.60:0/767407525 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2b414af80 msgr2=0x7fc2b40acf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:56.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.547+0000 7fc2b359e700 1 --2- 172.21.15.60:0/767407525 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2b414af80 0x7fc2b40acf10 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7fc2ac0032f0 tx=0x7fc2ac009290).stop 2022-06-29T16:03:56.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.547+0000 7fc2b359e700 1 -- 172.21.15.60:0/767407525 shutdown_connections 2022-06-29T16:03:56.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.547+0000 7fc2b359e700 1 --2- 172.21.15.60:0/767407525 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2b414af80 0x7fc2b40acf10 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:56.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.547+0000 7fc2b359e700 1 --2- 172.21.15.60:0/767407525 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2b40b2130 0x7fc2b40aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:56.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.547+0000 7fc2b359e700 1 --2- 172.21.15.60:0/767407525 >> v2:172.21.15.92:3300/0 conn(0x7fc2b405d070 0x7fc2b414aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:56.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.547+0000 7fc2b359e700 1 -- 172.21.15.60:0/767407525 >> 172.21.15.60:0/767407525 conn(0x7fc2b40b5890 msgr2=0x7fc2b40b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:56.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.547+0000 7fc2b359e700 1 -- 172.21.15.60:0/767407525 shutdown_connections 2022-06-29T16:03:56.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.547+0000 7fc2b359e700 1 -- 172.21.15.60:0/767407525 wait complete. 2022-06-29T16:03:56.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.551+0000 7fc2b359e700 1 Processor -- start 2022-06-29T16:03:56.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.551+0000 7fc2b359e700 1 -- start start 2022-06-29T16:03:56.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.555+0000 7fc2b359e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2b405d070 0x7fc2b4156b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:56.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.555+0000 7fc2b359e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2b40b2130 0x7fc2b4116780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:56.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.555+0000 7fc2b359e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2b414af80 0x7fc2b4065250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:56.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.555+0000 7fc2b359e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b4066ee0 con 0x7fc2b414af80 2022-06-29T16:03:56.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.555+0000 7fc2b359e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b4156170 con 0x7fc2b405d070 2022-06-29T16:03:56.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.555+0000 7fc2b359e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b4156410 con 0x7fc2b40b2130 2022-06-29T16:03:56.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.555+0000 7fc2b2d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2b414af80 0x7fc2b4065250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:56.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.555+0000 7fc2b2d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2b414af80 0x7fc2b4065250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48918/0 (socket says 172.21.15.60:48918) 2022-06-29T16:03:56.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.555+0000 7fc2b2d9d700 1 -- 172.21.15.60:0/1345322903 learned_addr learned my addr 172.21.15.60:0/1345322903 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:56.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.555+0000 7fc2b259c700 1 --2- 172.21.15.60:0/1345322903 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2b405d070 0x7fc2b4156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:56.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.555+0000 7fc2b1d9b700 1 --2- 172.21.15.60:0/1345322903 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2b40b2130 0x7fc2b4116780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:56.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.555+0000 7fc2b2d9d700 1 -- 172.21.15.60:0/1345322903 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2b40b2130 msgr2=0x7fc2b4116780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:56.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.555+0000 7fc2b2d9d700 1 --2- 172.21.15.60:0/1345322903 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2b40b2130 0x7fc2b4116780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:56.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.555+0000 7fc2b2d9d700 1 -- 172.21.15.60:0/1345322903 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2b405d070 msgr2=0x7fc2b4156b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:56.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.555+0000 7fc2b2d9d700 1 --2- 172.21.15.60:0/1345322903 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2b405d070 0x7fc2b4156b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:56.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.555+0000 7fc2b2d9d700 1 -- 172.21.15.60:0/1345322903 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2ac009530 con 0x7fc2b414af80 2022-06-29T16:03:56.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.555+0000 7fc2b259c700 1 --2- 172.21.15.60:0/1345322903 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2b405d070 0x7fc2b4156b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:03:56.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.555+0000 7fc2b2d9d700 1 --2- 172.21.15.60:0/1345322903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2b414af80 0x7fc2b4065250 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0x7fc2ac0032f0 tx=0x7fc2ac016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:56.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.555+0000 7fc2a37fe700 1 -- 172.21.15.60:0/1345322903 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2ac00b870 con 0x7fc2b414af80 2022-06-29T16:03:56.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.555+0000 7fc2a37fe700 1 -- 172.21.15.60:0/1345322903 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2ac0105b0 con 0x7fc2b414af80 2022-06-29T16:03:56.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.555+0000 7fc2b359e700 1 -- 172.21.15.60:0/1345322903 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2b41566b0 con 0x7fc2b414af80 2022-06-29T16:03:56.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.555+0000 7fc2a37fe700 1 -- 172.21.15.60:0/1345322903 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2ac010910 con 0x7fc2b414af80 2022-06-29T16:03:56.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.555+0000 7fc2b359e700 1 -- 172.21.15.60:0/1345322903 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2b4116c70 con 0x7fc2b414af80 2022-06-29T16:03:56.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.555+0000 7fc2b359e700 1 -- 172.21.15.60:0/1345322903 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2b414a5f0 con 0x7fc2b414af80 2022-06-29T16:03:56.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.555+0000 7fc2a37fe700 1 -- 172.21.15.60:0/1345322903 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc2ac00f070 con 0x7fc2b414af80 2022-06-29T16:03:56.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.555+0000 7fc2a37fe700 1 --2- 172.21.15.60:0/1345322903 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc2900539e0 0x7fc290055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:56.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.555+0000 7fc2a37fe700 1 -- 172.21.15.60:0/1345322903 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc2ac06d280 con 0x7fc2b414af80 2022-06-29T16:03:56.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.555+0000 7fc2b259c700 1 --2- 172.21.15.60:0/1345322903 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc2900539e0 0x7fc290055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:56.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.559+0000 7fc2b259c700 1 --2- 172.21.15.60:0/1345322903 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc2900539e0 0x7fc290055ea0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7fc2a800c000 tx=0x7fc2a8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:56.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.563+0000 7fc2a37fe700 1 -- 172.21.15.60:0/1345322903 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc2ac0441a0 con 0x7fc2b414af80 2022-06-29T16:03:56.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:56.723+0000 7fc2b359e700 1 -- 172.21.15.60:0/1345322903 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc2b40573d0 con 0x7fc2b414af80 2022-06-29T16:03:57.122 INFO:tasks.ceph.mon.a.smithi060.stderr:2022-06-29T16:03:57.115+0000 7fcc1a376700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-06-29T16:03:57.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.127+0000 7fc2a37fe700 1 -- 172.21.15.60:0/1345322903 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v26) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc2ac015020 con 0x7fc2b414af80 2022-06-29T16:03:57.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.131+0000 7fc2b359e700 1 -- 172.21.15.60:0/1345322903 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc2900539e0 msgr2=0x7fc290055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:57.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.131+0000 7fc2b359e700 1 --2- 172.21.15.60:0/1345322903 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc2900539e0 0x7fc290055ea0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7fc2a800c000 tx=0x7fc2a8008040).stop 2022-06-29T16:03:57.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.131+0000 7fc2b359e700 1 -- 172.21.15.60:0/1345322903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2b414af80 msgr2=0x7fc2b4065250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:57.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.131+0000 7fc2b359e700 1 --2- 172.21.15.60:0/1345322903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2b414af80 0x7fc2b4065250 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0x7fc2ac0032f0 tx=0x7fc2ac016000).stop 2022-06-29T16:03:57.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.131+0000 7fc2b359e700 1 -- 172.21.15.60:0/1345322903 shutdown_connections 2022-06-29T16:03:57.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.131+0000 7fc2b359e700 1 --2- 172.21.15.60:0/1345322903 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc2900539e0 0x7fc290055ea0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:57.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.131+0000 7fc2b359e700 1 --2- 172.21.15.60:0/1345322903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2b414af80 0x7fc2b4065250 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:57.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.131+0000 7fc2b359e700 1 --2- 172.21.15.60:0/1345322903 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2b40b2130 0x7fc2b4116780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:57.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.131+0000 7fc2b359e700 1 --2- 172.21.15.60:0/1345322903 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2b405d070 0x7fc2b4156b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:57.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.131+0000 7fc2b359e700 1 -- 172.21.15.60:0/1345322903 >> 172.21.15.60:0/1345322903 conn(0x7fc2b40b5890 msgr2=0x7fc2b4143220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:57.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.131+0000 7fc2b359e700 1 -- 172.21.15.60:0/1345322903 shutdown_connections 2022-06-29T16:03:57.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.131+0000 7fc2b359e700 1 -- 172.21.15.60:0/1345322903 wait complete. 2022-06-29T16:03:57.141 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-06-29T16:03:57.153 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:03:57.169 INFO:tasks.ceph.mds.b.smithi092.stdout:starting mds.b at 2022-06-29T16:03:57.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.255+0000 7ff2392c2700 1 Processor -- start 2022-06-29T16:03:57.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.255+0000 7ff2392c2700 1 -- start start 2022-06-29T16:03:57.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.255+0000 7ff2392c2700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff2340b5950 0x7ff234067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:57.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.255+0000 7ff2392c2700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff234063ad0 0x7ff23414af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:57.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.255+0000 7ff2392c2700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2340acba0 con 0x7ff23405bd20 2022-06-29T16:03:57.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.255+0000 7ff2392c2700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff234067220 con 0x7ff2340b5950 2022-06-29T16:03:57.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.255+0000 7ff2392c2700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2340b5d30 con 0x7ff234063ad0 2022-06-29T16:03:57.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.255+0000 7ff2337fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff23405bd20 0x7ff23405c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57012/0 (socket says 172.21.15.60:57012) 2022-06-29T16:03:57.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.255+0000 7ff2337fe700 1 -- 172.21.15.60:0/1228156555 learned_addr learned my addr 172.21.15.60:0/1228156555 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:03:57.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.255+0000 7ff232ffd700 1 --2- 172.21.15.60:0/1228156555 >> v2:172.21.15.92:3300/0 conn(0x7ff2340b5950 0x7ff234067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:57.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.255+0000 7ff2327fc700 1 --2- 172.21.15.60:0/1228156555 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff234063ad0 0x7ff23414af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:57.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.255+0000 7ff231ffb700 1 -- 172.21.15.60:0/1228156555 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 330900791 0 0) 0x7ff2340acba0 con 0x7ff23405bd20 2022-06-29T16:03:57.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.255+0000 7ff231ffb700 1 -- 172.21.15.60:0/1228156555 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff214003260 con 0x7ff23405bd20 2022-06-29T16:03:57.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.255+0000 7ff232ffd700 1 -- 172.21.15.60:0/1228156555 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff234063ad0 msgr2=0x7ff23414af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:57.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.255+0000 7ff232ffd700 1 --2- 172.21.15.60:0/1228156555 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff234063ad0 0x7ff23414af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:57.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.255+0000 7ff232ffd700 1 -- 172.21.15.60:0/1228156555 >> v1:172.21.15.60:6789/0 conn(0x7ff23405bd20 legacy=0x7ff23405c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:57.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.255+0000 7ff232ffd700 1 -- 172.21.15.60:0/1228156555 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2340ace00 con 0x7ff2340b5950 2022-06-29T16:03:57.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.255+0000 7ff232ffd700 1 --2- 172.21.15.60:0/1228156555 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2340b5950 0x7ff234067680 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7ff21c002f70 tx=0x7ff21c006df0).ready entity=mon.1 client_cookie=e35e06e556b8e88d server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:57.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.255+0000 7ff231ffb700 1 -- 172.21.15.60:0/1228156555 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff21c00d4a0 con 0x7ff2340b5950 2022-06-29T16:03:57.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.255+0000 7ff231ffb700 1 -- 172.21.15.60:0/1228156555 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff21c000e90 con 0x7ff2340b5950 2022-06-29T16:03:57.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.255+0000 7ff231ffb700 1 -- 172.21.15.60:0/1228156555 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff21c00d7f0 con 0x7ff2340b5950 2022-06-29T16:03:57.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.259+0000 7ff2392c2700 1 -- 172.21.15.60:0/1228156555 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2340b5950 msgr2=0x7ff234067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:57.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.259+0000 7ff2392c2700 1 --2- 172.21.15.60:0/1228156555 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2340b5950 0x7ff234067680 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7ff21c002f70 tx=0x7ff21c006df0).stop 2022-06-29T16:03:57.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.259+0000 7ff2392c2700 1 -- 172.21.15.60:0/1228156555 shutdown_connections 2022-06-29T16:03:57.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.259+0000 7ff2392c2700 1 --2- 172.21.15.60:0/1228156555 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff234063ad0 0x7ff23414af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:57.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.259+0000 7ff2392c2700 1 --2- 172.21.15.60:0/1228156555 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2340b5950 0x7ff234067680 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:57.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.259+0000 7ff2392c2700 1 -- 172.21.15.60:0/1228156555 >> 172.21.15.60:0/1228156555 conn(0x7ff2340632c0 msgr2=0x7ff2340b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:57.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.259+0000 7ff2392c2700 1 -- 172.21.15.60:0/1228156555 shutdown_connections 2022-06-29T16:03:57.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.259+0000 7ff2392c2700 1 -- 172.21.15.60:0/1228156555 wait complete. 2022-06-29T16:03:57.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.259+0000 7ff2392c2700 1 Processor -- start 2022-06-29T16:03:57.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.259+0000 7ff2392c2700 1 -- start start 2022-06-29T16:03:57.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.259+0000 7ff2392c2700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff23405bd20 0x7ff234156940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:57.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.259+0000 7ff2392c2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff234063ad0 0x7ff2340af1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:57.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.259+0000 7ff2392c2700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2340b5950 0x7ff234148750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:57.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.259+0000 7ff2392c2700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2340ae880 con 0x7ff234063ad0 2022-06-29T16:03:57.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.259+0000 7ff2392c2700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2340ada80 con 0x7ff23405bd20 2022-06-29T16:03:57.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.259+0000 7ff2392c2700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2340aeab0 con 0x7ff2340b5950 2022-06-29T16:03:57.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.259+0000 7ff2327fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff234063ad0 0x7ff2340af1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:57.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.259+0000 7ff2327fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff234063ad0 0x7ff2340af1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48928/0 (socket says 172.21.15.60:48928) 2022-06-29T16:03:57.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.259+0000 7ff2327fc700 1 -- 172.21.15.60:0/2389370232 learned_addr learned my addr 172.21.15.60:0/2389370232 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:57.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.259+0000 7ff232ffd700 1 --2- 172.21.15.60:0/2389370232 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff23405bd20 0x7ff234156940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:57.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.259+0000 7ff2337fe700 1 --2- 172.21.15.60:0/2389370232 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2340b5950 0x7ff234148750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:57.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.259+0000 7ff2327fc700 1 -- 172.21.15.60:0/2389370232 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2340b5950 msgr2=0x7ff234148750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:57.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.259+0000 7ff2327fc700 1 --2- 172.21.15.60:0/2389370232 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2340b5950 0x7ff234148750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:57.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.259+0000 7ff2327fc700 1 -- 172.21.15.60:0/2389370232 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff23405bd20 msgr2=0x7ff234156940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:57.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.259+0000 7ff2327fc700 1 --2- 172.21.15.60:0/2389370232 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff23405bd20 0x7ff234156940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:57.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.259+0000 7ff2327fc700 1 -- 172.21.15.60:0/2389370232 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff21c00b180 con 0x7ff234063ad0 2022-06-29T16:03:57.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.259+0000 7ff232ffd700 1 --2- 172.21.15.60:0/2389370232 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff23405bd20 0x7ff234156940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:03:57.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.259+0000 7ff2327fc700 1 --2- 172.21.15.60:0/2389370232 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff234063ad0 0x7ff2340af1c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7ff22400c000 tx=0x7ff2240091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:57.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.263+0000 7ff22bfff700 1 -- 172.21.15.60:0/2389370232 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff22400c850 con 0x7ff234063ad0 2022-06-29T16:03:57.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.263+0000 7ff2392c2700 1 -- 172.21.15.60:0/2389370232 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff234156e30 con 0x7ff234063ad0 2022-06-29T16:03:57.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.263+0000 7ff22bfff700 1 -- 172.21.15.60:0/2389370232 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff22400c9b0 con 0x7ff234063ad0 2022-06-29T16:03:57.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.263+0000 7ff22bfff700 1 -- 172.21.15.60:0/2389370232 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff22400cd10 con 0x7ff234063ad0 2022-06-29T16:03:57.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.263+0000 7ff2392c2700 1 -- 172.21.15.60:0/2389370232 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2340ad500 con 0x7ff234063ad0 2022-06-29T16:03:57.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.263+0000 7ff2392c2700 1 -- 172.21.15.60:0/2389370232 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff234156730 con 0x7ff234063ad0 2022-06-29T16:03:57.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.263+0000 7ff22bfff700 1 -- 172.21.15.60:0/2389370232 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff224022070 con 0x7ff234063ad0 2022-06-29T16:03:57.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.263+0000 7ff22bfff700 1 --2- 172.21.15.60:0/2389370232 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff218053aa0 0x7ff218055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:57.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.263+0000 7ff22bfff700 1 -- 172.21.15.60:0/2389370232 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6159+0+0 (secure 0 0 0) 0x7ff22406c6b0 con 0x7ff234063ad0 2022-06-29T16:03:57.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.263+0000 7ff232ffd700 1 --2- 172.21.15.60:0/2389370232 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff218053aa0 0x7ff218055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:57.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.263+0000 7ff232ffd700 1 --2- 172.21.15.60:0/2389370232 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff218053aa0 0x7ff218055f60 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7ff21c018000 tx=0x7ff21c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:57.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.267+0000 7ff22bfff700 1 -- 172.21.15.60:0/2389370232 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff234156730 con 0x7ff234063ad0 2022-06-29T16:03:57.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.427+0000 7ff2392c2700 1 -- 172.21.15.60:0/2389370232 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff234156730 con 0x7ff234063ad0 2022-06-29T16:03:57.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.427+0000 7ff22bfff700 1 -- 172.21.15.60:0/2389370232 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+1773 (secure 0 0 0) 0x7ff234156730 con 0x7ff234063ad0 2022-06-29T16:03:57.434 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:03:57.434 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":26,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5955,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.60:6833/66690512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":66690512},{"type":"v1","addr":"172.21.15.60:6833","nonce":66690512}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":23}],"filesystems":[{"mdsmap":{"epoch":26,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:03:28.112667+0000","modified":"2022-06-29T16:03:57.119556+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-06-29T16:03:57.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.427+0000 7ff2392c2700 1 -- 172.21.15.60:0/2389370232 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff218053aa0 msgr2=0x7ff218055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:57.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.427+0000 7ff2392c2700 1 --2- 172.21.15.60:0/2389370232 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff218053aa0 0x7ff218055f60 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7ff21c018000 tx=0x7ff21c014000).stop 2022-06-29T16:03:57.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.427+0000 7ff2392c2700 1 -- 172.21.15.60:0/2389370232 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff234063ad0 msgr2=0x7ff2340af1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:57.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.427+0000 7ff2392c2700 1 --2- 172.21.15.60:0/2389370232 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff234063ad0 0x7ff2340af1c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7ff22400c000 tx=0x7ff2240091d0).stop 2022-06-29T16:03:57.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.427+0000 7ff2392c2700 1 -- 172.21.15.60:0/2389370232 shutdown_connections 2022-06-29T16:03:57.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.427+0000 7ff2392c2700 1 --2- 172.21.15.60:0/2389370232 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff218053aa0 0x7ff218055f60 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:57.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.427+0000 7ff2392c2700 1 --2- 172.21.15.60:0/2389370232 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2340b5950 0x7ff234148750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:57.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.427+0000 7ff2392c2700 1 --2- 172.21.15.60:0/2389370232 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff234063ad0 0x7ff2340af1c0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:57.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.427+0000 7ff2392c2700 1 --2- 172.21.15.60:0/2389370232 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff23405bd20 0x7ff234156940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:57.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.427+0000 7ff2392c2700 1 -- 172.21.15.60:0/2389370232 >> 172.21.15.60:0/2389370232 conn(0x7ff2340632c0 msgr2=0x7ff234064e00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:57.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.431+0000 7ff2392c2700 1 -- 172.21.15.60:0/2389370232 shutdown_connections 2022-06-29T16:03:57.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.431+0000 7ff2392c2700 1 -- 172.21.15.60:0/2389370232 wait complete. 2022-06-29T16:03:57.439 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 26 2022-06-29T16:03:57.447 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-06-29T16:03:57.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.547+0000 7efc36738700 1 Processor -- start 2022-06-29T16:03:57.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.547+0000 7efc36738700 1 -- start start 2022-06-29T16:03:57.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.547+0000 7efc36738700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7efc3005d070 0x7efc3014aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:57.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.547+0000 7efc36738700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc3005d450 con 0x7efc3014af80 2022-06-29T16:03:57.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.547+0000 7efc36738700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc300ad840 con 0x7efc300b2130 2022-06-29T16:03:57.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.547+0000 7efc36738700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc300adf90 con 0x7efc3005d070 2022-06-29T16:03:57.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.547+0000 7efc35f37700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7efc3014af80 0x7efc300ad130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57026/0 (socket says 172.21.15.60:57026) 2022-06-29T16:03:57.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.547+0000 7efc35f37700 1 -- 172.21.15.60:0/1232877222 learned_addr learned my addr 172.21.15.60:0/1232877222 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:03:57.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.551+0000 7efc35736700 1 --2- 172.21.15.60:0/1232877222 >> v2:172.21.15.92:3300/0 conn(0x7efc3005d070 0x7efc3014aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:57.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.551+0000 7efc27fff700 1 -- 172.21.15.60:0/1232877222 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2813963265 0 0) 0x7efc3005d450 con 0x7efc3014af80 2022-06-29T16:03:57.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.551+0000 7efc27fff700 1 -- 172.21.15.60:0/1232877222 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc14003260 con 0x7efc3014af80 2022-06-29T16:03:57.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.551+0000 7efc35736700 1 -- 172.21.15.60:0/1232877222 >> v1:172.21.15.92:6789/0 conn(0x7efc300b2130 legacy=0x7efc300aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:57.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.551+0000 7efc35736700 1 -- 172.21.15.60:0/1232877222 >> v1:172.21.15.60:6789/0 conn(0x7efc3014af80 legacy=0x7efc300ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:57.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.551+0000 7efc35736700 1 -- 172.21.15.60:0/1232877222 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc300b25d0 con 0x7efc3005d070 2022-06-29T16:03:57.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.551+0000 7efc35736700 1 --2- 172.21.15.60:0/1232877222 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc3005d070 0x7efc3014aa90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7efc280032f0 tx=0x7efc280091d0).ready entity=mon.1 client_cookie=d40b2646ffd4dcc server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:57.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.551+0000 7efc27fff700 1 -- 172.21.15.60:0/1232877222 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc2800b6c0 con 0x7efc3005d070 2022-06-29T16:03:57.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.551+0000 7efc27fff700 1 -- 172.21.15.60:0/1232877222 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc2800b820 con 0x7efc3005d070 2022-06-29T16:03:57.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.551+0000 7efc27fff700 1 -- 172.21.15.60:0/1232877222 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc2800b990 con 0x7efc3005d070 2022-06-29T16:03:57.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.551+0000 7efc36738700 1 -- 172.21.15.60:0/1232877222 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc3005d070 msgr2=0x7efc3014aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:57.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.551+0000 7efc36738700 1 --2- 172.21.15.60:0/1232877222 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc3005d070 0x7efc3014aa90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7efc280032f0 tx=0x7efc280091d0).stop 2022-06-29T16:03:57.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.551+0000 7efc36738700 1 -- 172.21.15.60:0/1232877222 shutdown_connections 2022-06-29T16:03:57.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.551+0000 7efc36738700 1 --2- 172.21.15.60:0/1232877222 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc3005d070 0x7efc3014aa90 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:57.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.551+0000 7efc36738700 1 -- 172.21.15.60:0/1232877222 >> 172.21.15.60:0/1232877222 conn(0x7efc300b5890 msgr2=0x7efc300b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:57.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.551+0000 7efc36738700 1 -- 172.21.15.60:0/1232877222 shutdown_connections 2022-06-29T16:03:57.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.551+0000 7efc36738700 1 -- 172.21.15.60:0/1232877222 wait complete. 2022-06-29T16:03:57.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.555+0000 7efc36738700 1 Processor -- start 2022-06-29T16:03:57.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.555+0000 7efc36738700 1 -- start start 2022-06-29T16:03:57.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.555+0000 7efc36738700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc3005d070 0x7efc300b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:57.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.555+0000 7efc36738700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc300b2130 0x7efc301557f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:57.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.555+0000 7efc36738700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc3014af80 0x7efc30155ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:57.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.555+0000 7efc36738700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc30066590 con 0x7efc3005d070 2022-06-29T16:03:57.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.555+0000 7efc36738700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc300658d0 con 0x7efc300b2130 2022-06-29T16:03:57.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.555+0000 7efc36738700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc300662f0 con 0x7efc3014af80 2022-06-29T16:03:57.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.555+0000 7efc35736700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc3005d070 0x7efc300b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:57.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.555+0000 7efc35736700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc3005d070 0x7efc300b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48942/0 (socket says 172.21.15.60:48942) 2022-06-29T16:03:57.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.555+0000 7efc35736700 1 -- 172.21.15.60:0/2099708585 learned_addr learned my addr 172.21.15.60:0/2099708585 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:57.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.555+0000 7efc35f37700 1 --2- 172.21.15.60:0/2099708585 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc3014af80 0x7efc30155ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:57.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.555+0000 7efc34f35700 1 --2- 172.21.15.60:0/2099708585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc300b2130 0x7efc301557f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:57.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.555+0000 7efc35736700 1 -- 172.21.15.60:0/2099708585 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc3014af80 msgr2=0x7efc30155ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:57.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.555+0000 7efc35736700 1 --2- 172.21.15.60:0/2099708585 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc3014af80 0x7efc30155ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:57.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.555+0000 7efc35736700 1 -- 172.21.15.60:0/2099708585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc300b2130 msgr2=0x7efc301557f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:57.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.555+0000 7efc35736700 1 --2- 172.21.15.60:0/2099708585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc300b2130 0x7efc301557f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:57.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.555+0000 7efc35736700 1 -- 172.21.15.60:0/2099708585 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc28009470 con 0x7efc3005d070 2022-06-29T16:03:57.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.555+0000 7efc34f35700 1 --2- 172.21.15.60:0/2099708585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc300b2130 0x7efc301557f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:03:57.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.555+0000 7efc35736700 1 --2- 172.21.15.60:0/2099708585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc3005d070 0x7efc300b41f0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7efc280032f0 tx=0x7efc28016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:57.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.559+0000 7efc267fc700 1 -- 172.21.15.60:0/2099708585 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc2800b6c0 con 0x7efc3005d070 2022-06-29T16:03:57.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.559+0000 7efc267fc700 1 -- 172.21.15.60:0/2099708585 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc28021630 con 0x7efc3005d070 2022-06-29T16:03:57.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.559+0000 7efc267fc700 1 -- 172.21.15.60:0/2099708585 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc28021990 con 0x7efc3005d070 2022-06-29T16:03:57.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.559+0000 7efc36738700 1 -- 172.21.15.60:0/2099708585 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc300644b0 con 0x7efc3005d070 2022-06-29T16:03:57.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.559+0000 7efc36738700 1 -- 172.21.15.60:0/2099708585 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc30064f90 con 0x7efc3005d070 2022-06-29T16:03:57.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.559+0000 7efc36738700 1 -- 172.21.15.60:0/2099708585 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc3014a350 con 0x7efc3005d070 2022-06-29T16:03:57.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.559+0000 7efc267fc700 1 -- 172.21.15.60:0/2099708585 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7efc28027020 con 0x7efc3005d070 2022-06-29T16:03:57.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.559+0000 7efc267fc700 1 --2- 172.21.15.60:0/2099708585 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efc18053a10 0x7efc18055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:57.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.559+0000 7efc267fc700 1 -- 172.21.15.60:0/2099708585 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6159+0+0 (secure 0 0 0) 0x7efc2806d510 con 0x7efc3005d070 2022-06-29T16:03:57.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.559+0000 7efc34f35700 1 --2- 172.21.15.60:0/2099708585 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efc18053a10 0x7efc18055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:57.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.563+0000 7efc34f35700 1 --2- 172.21.15.60:0/2099708585 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efc18053a10 0x7efc18055ed0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7efc2000b000 tx=0x7efc20009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:57.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.567+0000 7efc267fc700 1 -- 172.21.15.60:0/2099708585 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efc280442d0 con 0x7efc3005d070 2022-06-29T16:03:57.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:57.735+0000 7efc36738700 1 -- 172.21.15.60:0/2099708585 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7efc300573d0 con 0x7efc3005d070 2022-06-29T16:03:58.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.131+0000 7efc267fc700 1 -- 172.21.15.60:0/2099708585 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7efc28006d00 con 0x7efc3005d070 2022-06-29T16:03:58.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.135+0000 7efc36738700 1 -- 172.21.15.60:0/2099708585 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efc18053a10 msgr2=0x7efc18055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:58.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.135+0000 7efc36738700 1 --2- 172.21.15.60:0/2099708585 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efc18053a10 0x7efc18055ed0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7efc2000b000 tx=0x7efc20009040).stop 2022-06-29T16:03:58.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.135+0000 7efc36738700 1 -- 172.21.15.60:0/2099708585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc3005d070 msgr2=0x7efc300b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:58.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.135+0000 7efc36738700 1 --2- 172.21.15.60:0/2099708585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc3005d070 0x7efc300b41f0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7efc280032f0 tx=0x7efc28016000).stop 2022-06-29T16:03:58.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.135+0000 7efc36738700 1 -- 172.21.15.60:0/2099708585 shutdown_connections 2022-06-29T16:03:58.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.135+0000 7efc36738700 1 --2- 172.21.15.60:0/2099708585 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efc18053a10 0x7efc18055ed0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:58.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.135+0000 7efc36738700 1 --2- 172.21.15.60:0/2099708585 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc3014af80 0x7efc30155ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:58.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.135+0000 7efc36738700 1 --2- 172.21.15.60:0/2099708585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc300b2130 0x7efc301557f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:58.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.135+0000 7efc36738700 1 --2- 172.21.15.60:0/2099708585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc3005d070 0x7efc300b41f0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:58.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.135+0000 7efc36738700 1 -- 172.21.15.60:0/2099708585 >> 172.21.15.60:0/2099708585 conn(0x7efc300b5890 msgr2=0x7efc301433b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:58.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.135+0000 7efc36738700 1 -- 172.21.15.60:0/2099708585 shutdown_connections 2022-06-29T16:03:58.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.135+0000 7efc36738700 1 -- 172.21.15.60:0/2099708585 wait complete. 2022-06-29T16:03:58.156 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-06-29T16:03:58.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.271+0000 7f936f073700 1 Processor -- start 2022-06-29T16:03:58.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.271+0000 7f936f073700 1 -- start start 2022-06-29T16:03:58.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.271+0000 7f936f073700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93680b9be0 0x7f936805bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:58.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.271+0000 7f936f073700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f93680b28e0 0x7f9368141e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:58.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.271+0000 7f936f073700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93680b1d70 con 0x7f93680afee0 2022-06-29T16:03:58.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.271+0000 7f936f073700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93680b9fc0 con 0x7f93680b28e0 2022-06-29T16:03:58.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.271+0000 7f936f073700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93680b3c40 con 0x7f93680b9be0 2022-06-29T16:03:58.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.271+0000 7f936d870700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f93680afee0 0x7f93680b21d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57038/0 (socket says 172.21.15.60:57038) 2022-06-29T16:03:58.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.271+0000 7f936d870700 1 -- 172.21.15.60:0/1765954741 learned_addr learned my addr 172.21.15.60:0/1765954741 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:03:58.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.271+0000 7f936e872700 1 --2- 172.21.15.60:0/1765954741 >> v2:172.21.15.92:3300/0 conn(0x7f93680b28e0 0x7f9368141e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:58.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.271+0000 7f936e071700 1 --2- 172.21.15.60:0/1765954741 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93680b9be0 0x7f936805bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:58.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.275+0000 7f936d06f700 1 -- 172.21.15.60:0/1765954741 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 31323732 0 0) 0x7f93680b1d70 con 0x7f93680afee0 2022-06-29T16:03:58.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.275+0000 7f936d06f700 1 -- 172.21.15.60:0/1765954741 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9350003260 con 0x7f93680afee0 2022-06-29T16:03:58.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.275+0000 7f936e872700 1 -- 172.21.15.60:0/1765954741 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93680b9be0 msgr2=0x7f936805bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:58.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.275+0000 7f936e872700 1 --2- 172.21.15.60:0/1765954741 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93680b9be0 0x7f936805bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:58.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.275+0000 7f936e872700 1 -- 172.21.15.60:0/1765954741 >> v1:172.21.15.60:6789/0 conn(0x7f93680afee0 legacy=0x7f93680b21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:58.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.275+0000 7f936e872700 1 -- 172.21.15.60:0/1765954741 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93680b3ee0 con 0x7f93680b28e0 2022-06-29T16:03:58.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.275+0000 7f936e872700 1 --2- 172.21.15.60:0/1765954741 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f93680b28e0 0x7f9368141e70 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f93600032f0 tx=0x7f93600091d0).ready entity=mon.1 client_cookie=bc66f7492650e631 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:58.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.275+0000 7f936d06f700 1 -- 172.21.15.60:0/1765954741 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f936000b600 con 0x7f93680b28e0 2022-06-29T16:03:58.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.275+0000 7f936d06f700 1 -- 172.21.15.60:0/1765954741 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f936000b760 con 0x7f93680b28e0 2022-06-29T16:03:58.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.275+0000 7f936d06f700 1 -- 172.21.15.60:0/1765954741 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f936000b8d0 con 0x7f93680b28e0 2022-06-29T16:03:58.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.275+0000 7f936f073700 1 -- 172.21.15.60:0/1765954741 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f93680b28e0 msgr2=0x7f9368141e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:58.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.275+0000 7f936f073700 1 --2- 172.21.15.60:0/1765954741 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f93680b28e0 0x7f9368141e70 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f93600032f0 tx=0x7f93600091d0).stop 2022-06-29T16:03:58.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.275+0000 7f936f073700 1 -- 172.21.15.60:0/1765954741 shutdown_connections 2022-06-29T16:03:58.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.275+0000 7f936f073700 1 --2- 172.21.15.60:0/1765954741 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f93680b28e0 0x7f9368141e70 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:58.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.275+0000 7f936f073700 1 --2- 172.21.15.60:0/1765954741 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93680b9be0 0x7f936805bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:58.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.275+0000 7f936f073700 1 -- 172.21.15.60:0/1765954741 >> 172.21.15.60:0/1765954741 conn(0x7f9368062ce0 msgr2=0x7f93680b9190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:58.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.275+0000 7f936f073700 1 -- 172.21.15.60:0/1765954741 shutdown_connections 2022-06-29T16:03:58.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.275+0000 7f936f073700 1 -- 172.21.15.60:0/1765954741 wait complete. 2022-06-29T16:03:58.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.275+0000 7f936f073700 1 Processor -- start 2022-06-29T16:03:58.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.275+0000 7f936f073700 1 -- start start 2022-06-29T16:03:58.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.279+0000 7f936f073700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93680afee0 0x7f9368151850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:58.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.279+0000 7f936f073700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f93680b28e0 0x7f9368149590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:58.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.279+0000 7f936f073700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93680b9be0 0x7f9368147490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:58.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.279+0000 7f936f073700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93681551f0 con 0x7f93680afee0 2022-06-29T16:03:58.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.279+0000 7f936f073700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9368147030 con 0x7f93680b28e0 2022-06-29T16:03:58.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.279+0000 7f936f073700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9368154f50 con 0x7f93680b9be0 2022-06-29T16:03:58.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.279+0000 7f936e071700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93680afee0 0x7f9368151850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:58.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.279+0000 7f936e071700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93680afee0 0x7f9368151850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48956/0 (socket says 172.21.15.60:48956) 2022-06-29T16:03:58.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.279+0000 7f936e071700 1 -- 172.21.15.60:0/3315587644 learned_addr learned my addr 172.21.15.60:0/3315587644 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:58.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.279+0000 7f936d870700 1 --2- 172.21.15.60:0/3315587644 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f93680b28e0 0x7f9368149590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:58.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.279+0000 7f936e872700 1 --2- 172.21.15.60:0/3315587644 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93680b9be0 0x7f9368147490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:58.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.279+0000 7f936e071700 1 -- 172.21.15.60:0/3315587644 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93680b9be0 msgr2=0x7f9368147490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:58.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.279+0000 7f936e071700 1 --2- 172.21.15.60:0/3315587644 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93680b9be0 0x7f9368147490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:58.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.279+0000 7f936e071700 1 -- 172.21.15.60:0/3315587644 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f93680b28e0 msgr2=0x7f9368149590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:58.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.279+0000 7f936e071700 1 --2- 172.21.15.60:0/3315587644 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f93680b28e0 0x7f9368149590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:58.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.279+0000 7f936e071700 1 -- 172.21.15.60:0/3315587644 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9360009470 con 0x7f93680afee0 2022-06-29T16:03:58.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.279+0000 7f936e872700 1 --2- 172.21.15.60:0/3315587644 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93680b9be0 0x7f9368147490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:03:58.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.279+0000 7f936d870700 1 --2- 172.21.15.60:0/3315587644 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f93680b28e0 0x7f9368149590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:03:58.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.279+0000 7f936e071700 1 --2- 172.21.15.60:0/3315587644 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93680afee0 0x7f9368151850 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7f935800c000 tx=0x7f9358006d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:58.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.279+0000 7f93577fe700 1 -- 172.21.15.60:0/3315587644 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9358011750 con 0x7f93680afee0 2022-06-29T16:03:58.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.279+0000 7f93577fe700 1 -- 172.21.15.60:0/3315587644 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93580118b0 con 0x7f93680afee0 2022-06-29T16:03:58.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.279+0000 7f936f073700 1 -- 172.21.15.60:0/3315587644 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9368145080 con 0x7f93680afee0 2022-06-29T16:03:58.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.279+0000 7f93577fe700 1 -- 172.21.15.60:0/3315587644 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9358011c10 con 0x7f93680afee0 2022-06-29T16:03:58.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.279+0000 7f936f073700 1 -- 172.21.15.60:0/3315587644 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9368145f60 con 0x7f93680afee0 2022-06-29T16:03:58.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.279+0000 7f936f073700 1 -- 172.21.15.60:0/3315587644 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93680573d0 con 0x7f93680afee0 2022-06-29T16:03:58.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.279+0000 7f93577fe700 1 -- 172.21.15.60:0/3315587644 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9358011d70 con 0x7f93680afee0 2022-06-29T16:03:58.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.283+0000 7f93577fe700 1 --2- 172.21.15.60:0/3315587644 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9348053b30 0x7f9348055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:58.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.283+0000 7f93577fe700 1 -- 172.21.15.60:0/3315587644 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f935806c2a0 con 0x7f93680afee0 2022-06-29T16:03:58.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.283+0000 7f936d870700 1 --2- 172.21.15.60:0/3315587644 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9348053b30 0x7f9348055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:58.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.283+0000 7f936d870700 1 --2- 172.21.15.60:0/3315587644 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9348053b30 0x7f9348055ff0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f9364008000 tx=0x7f9364009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:58.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.287+0000 7f93577fe700 1 -- 172.21.15.60:0/3315587644 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9358043070 con 0x7f93680afee0 2022-06-29T16:03:58.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:58.415+0000 7f936f073700 1 -- 172.21.15.60:0/3315587644 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f936805d990 con 0x7f93680afee0 2022-06-29T16:03:58.768 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:03:58.771 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:03:58.774 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:03:58.777 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:03:58.779 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:03:59.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.143+0000 7f93577fe700 1 -- 172.21.15.60:0/3315587644 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v64) v1 ==== 185+0+0 (secure 0 0 0) 0x7f9358046690 con 0x7f93680afee0 2022-06-29T16:03:59.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.143+0000 7f936f073700 1 -- 172.21.15.60:0/3315587644 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9348053b30 msgr2=0x7f9348055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:59.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.143+0000 7f936f073700 1 --2- 172.21.15.60:0/3315587644 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9348053b30 0x7f9348055ff0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f9364008000 tx=0x7f9364009040).stop 2022-06-29T16:03:59.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.143+0000 7f936f073700 1 -- 172.21.15.60:0/3315587644 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93680afee0 msgr2=0x7f9368151850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:59.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.143+0000 7f936f073700 1 --2- 172.21.15.60:0/3315587644 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93680afee0 0x7f9368151850 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7f935800c000 tx=0x7f9358006d90).stop 2022-06-29T16:03:59.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.143+0000 7f936f073700 1 -- 172.21.15.60:0/3315587644 shutdown_connections 2022-06-29T16:03:59.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.143+0000 7f936f073700 1 --2- 172.21.15.60:0/3315587644 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9348053b30 0x7f9348055ff0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:59.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.143+0000 7f936f073700 1 --2- 172.21.15.60:0/3315587644 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93680b9be0 0x7f9368147490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:59.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.143+0000 7f936f073700 1 --2- 172.21.15.60:0/3315587644 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f93680b28e0 0x7f9368149590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:59.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.143+0000 7f936f073700 1 --2- 172.21.15.60:0/3315587644 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93680afee0 0x7f9368151850 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:59.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.143+0000 7f936f073700 1 -- 172.21.15.60:0/3315587644 >> 172.21.15.60:0/3315587644 conn(0x7f9368062ce0 msgr2=0x7f93680b5f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:59.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.143+0000 7f936f073700 1 -- 172.21.15.60:0/3315587644 shutdown_connections 2022-06-29T16:03:59.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.143+0000 7f936f073700 1 -- 172.21.15.60:0/3315587644 wait complete. 2022-06-29T16:03:59.157 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2022-06-29T16:03:59.165 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-06-29T16:03:59.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.275+0000 7f9638335700 1 Processor -- start 2022-06-29T16:03:59.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.275+0000 7f9638335700 1 -- start start 2022-06-29T16:03:59.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.279+0000 7f9638335700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f96300b3e70 0x7f96301460b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:59.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.279+0000 7f9638335700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f963013fbe0 0x7f9630140030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:59.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.279+0000 7f9638335700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9630140520 con 0x7f963005bf30 2022-06-29T16:03:59.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.279+0000 7f9638335700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f963005d030 con 0x7f96300b3e70 2022-06-29T16:03:59.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.279+0000 7f9638335700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9630065ba0 con 0x7f963013fbe0 2022-06-29T16:03:59.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.279+0000 7f96358d0700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f963005bf30 0x7f9630066000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57054/0 (socket says 172.21.15.60:57054) 2022-06-29T16:03:59.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.279+0000 7f96358d0700 1 -- 172.21.15.60:0/2398759432 learned_addr learned my addr 172.21.15.60:0/2398759432 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:03:59.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.279+0000 7f96368d2700 1 --2- 172.21.15.60:0/2398759432 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f963013fbe0 0x7f9630140030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:59.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.279+0000 7f96360d1700 1 --2- 172.21.15.60:0/2398759432 >> v2:172.21.15.92:3300/0 conn(0x7f96300b3e70 0x7f96301460b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:59.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.279+0000 7f96350cf700 1 -- 172.21.15.60:0/2398759432 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3271269173 0 0) 0x7f9630140520 con 0x7f963005bf30 2022-06-29T16:03:59.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.279+0000 7f96350cf700 1 -- 172.21.15.60:0/2398759432 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9618003260 con 0x7f963005bf30 2022-06-29T16:03:59.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.279+0000 7f96360d1700 1 -- 172.21.15.60:0/2398759432 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f963013fbe0 msgr2=0x7f9630140030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:03:59.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.279+0000 7f96360d1700 1 --2- 172.21.15.60:0/2398759432 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f963013fbe0 0x7f9630140030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:59.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.279+0000 7f96360d1700 1 -- 172.21.15.60:0/2398759432 >> v1:172.21.15.60:6789/0 conn(0x7f963005bf30 legacy=0x7f9630066000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:59.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.279+0000 7f96360d1700 1 -- 172.21.15.60:0/2398759432 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96300b4250 con 0x7f96300b3e70 2022-06-29T16:03:59.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.279+0000 7f96368d2700 1 --2- 172.21.15.60:0/2398759432 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f963013fbe0 0x7f9630140030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:03:59.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.279+0000 7f96360d1700 1 --2- 172.21.15.60:0/2398759432 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96300b3e70 0x7f96301460b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f962c002f70 tx=0x7f962c006df0).ready entity=mon.1 client_cookie=72d89d258b095b63 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:59.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.279+0000 7f96350cf700 1 -- 172.21.15.60:0/2398759432 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f962c00d4a0 con 0x7f96300b3e70 2022-06-29T16:03:59.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.279+0000 7f96350cf700 1 -- 172.21.15.60:0/2398759432 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f962c000e90 con 0x7f96300b3e70 2022-06-29T16:03:59.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.279+0000 7f96350cf700 1 -- 172.21.15.60:0/2398759432 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f962c00d7f0 con 0x7f96300b3e70 2022-06-29T16:03:59.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.279+0000 7f9638335700 1 -- 172.21.15.60:0/2398759432 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96300b3e70 msgr2=0x7f96301460b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:59.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.279+0000 7f9638335700 1 --2- 172.21.15.60:0/2398759432 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96300b3e70 0x7f96301460b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f962c002f70 tx=0x7f962c006df0).stop 2022-06-29T16:03:59.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.279+0000 7f9638335700 1 -- 172.21.15.60:0/2398759432 shutdown_connections 2022-06-29T16:03:59.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.279+0000 7f9638335700 1 --2- 172.21.15.60:0/2398759432 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f963013fbe0 0x7f9630140030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:59.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.279+0000 7f9638335700 1 --2- 172.21.15.60:0/2398759432 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96300b3e70 0x7f96301460b0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:59.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.279+0000 7f9638335700 1 -- 172.21.15.60:0/2398759432 >> 172.21.15.60:0/2398759432 conn(0x7f9630064a10 msgr2=0x7f9630064df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:03:59.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.279+0000 7f9638335700 1 -- 172.21.15.60:0/2398759432 shutdown_connections 2022-06-29T16:03:59.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.279+0000 7f9638335700 1 -- 172.21.15.60:0/2398759432 wait complete. 2022-06-29T16:03:59.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.283+0000 7f9638335700 1 Processor -- start 2022-06-29T16:03:59.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.283+0000 7f9638335700 1 -- start start 2022-06-29T16:03:59.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.283+0000 7f9638335700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f963005bf30 0x7f96300aff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:59.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.283+0000 7f9638335700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96300b3e70 0x7f96300ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:59.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.283+0000 7f9638335700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f963013fbe0 0x7f963013c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:59.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.283+0000 7f9638335700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96300af470 con 0x7f963013fbe0 2022-06-29T16:03:59.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.283+0000 7f9638335700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9630155ed0 con 0x7f963005bf30 2022-06-29T16:03:59.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.283+0000 7f9638335700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96300b05f0 con 0x7f96300b3e70 2022-06-29T16:03:59.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.283+0000 7f96368d2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f963013fbe0 0x7f963013c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:59.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.283+0000 7f96368d2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f963013fbe0 0x7f963013c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48974/0 (socket says 172.21.15.60:48974) 2022-06-29T16:03:59.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.283+0000 7f96368d2700 1 -- 172.21.15.60:0/3731230756 learned_addr learned my addr 172.21.15.60:0/3731230756 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:03:59.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.283+0000 7f96360d1700 1 --2- 172.21.15.60:0/3731230756 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f963005bf30 0x7f96300aff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:59.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.283+0000 7f96358d0700 1 --2- 172.21.15.60:0/3731230756 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96300b3e70 0x7f96300ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:59.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.283+0000 7f96368d2700 1 -- 172.21.15.60:0/3731230756 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96300b3e70 msgr2=0x7f96300ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:59.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.283+0000 7f96368d2700 1 --2- 172.21.15.60:0/3731230756 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96300b3e70 0x7f96300ad0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:59.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.283+0000 7f96368d2700 1 -- 172.21.15.60:0/3731230756 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f963005bf30 msgr2=0x7f96300aff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:03:59.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.283+0000 7f96368d2700 1 --2- 172.21.15.60:0/3731230756 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f963005bf30 0x7f96300aff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:03:59.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.287+0000 7f96368d2700 1 -- 172.21.15.60:0/3731230756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9628009470 con 0x7f963013fbe0 2022-06-29T16:03:59.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.287+0000 7f96368d2700 1 --2- 172.21.15.60:0/3731230756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f963013fbe0 0x7f963013c980 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0x7f96301406d0 tx=0x7f96280091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:59.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.287+0000 7f961f7fe700 1 -- 172.21.15.60:0/3731230756 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f962800c850 con 0x7f963013fbe0 2022-06-29T16:03:59.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.287+0000 7f961f7fe700 1 -- 172.21.15.60:0/3731230756 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f962800c9b0 con 0x7f963013fbe0 2022-06-29T16:03:59.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.287+0000 7f9638335700 1 -- 172.21.15.60:0/3731230756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f962c00b180 con 0x7f963013fbe0 2022-06-29T16:03:59.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.287+0000 7f961f7fe700 1 -- 172.21.15.60:0/3731230756 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f962800cd10 con 0x7f963013fbe0 2022-06-29T16:03:59.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.287+0000 7f9638335700 1 -- 172.21.15.60:0/3731230756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96300ad5c0 con 0x7f963013fbe0 2022-06-29T16:03:59.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.287+0000 7f9638335700 1 -- 172.21.15.60:0/3731230756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96300573d0 con 0x7f963013fbe0 2022-06-29T16:03:59.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.287+0000 7f961f7fe700 1 -- 172.21.15.60:0/3731230756 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9628022070 con 0x7f963013fbe0 2022-06-29T16:03:59.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.291+0000 7f961f7fe700 1 --2- 172.21.15.60:0/3731230756 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f96100539e0 0x7f9610055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:03:59.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.291+0000 7f961f7fe700 1 -- 172.21.15.60:0/3731230756 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f962806c580 con 0x7f963013fbe0 2022-06-29T16:03:59.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.291+0000 7f96360d1700 1 --2- 172.21.15.60:0/3731230756 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f96100539e0 0x7f9610055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:03:59.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.291+0000 7f96360d1700 1 --2- 172.21.15.60:0/3731230756 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f96100539e0 0x7f9610055ea0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f962c01a000 tx=0x7f962c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:03:59.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.295+0000 7f961f7fe700 1 -- 172.21.15.60:0/3731230756 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9628043560 con 0x7f963013fbe0 2022-06-29T16:03:59.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:03:59.419+0000 7f9638335700 1 -- 172.21.15.60:0/3731230756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f963005d990 con 0x7f963013fbe0 2022-06-29T16:04:00.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.147+0000 7f961f7fe700 1 -- 172.21.15.60:0/3731230756 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v65) v1 ==== 173+0+0 (secure 0 0 0) 0x7f9628016020 con 0x7f963013fbe0 2022-06-29T16:04:00.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.151+0000 7f9638335700 1 -- 172.21.15.60:0/3731230756 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f96100539e0 msgr2=0x7f9610055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:00.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.151+0000 7f9638335700 1 --2- 172.21.15.60:0/3731230756 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f96100539e0 0x7f9610055ea0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f962c01a000 tx=0x7f962c014000).stop 2022-06-29T16:04:00.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.151+0000 7f9638335700 1 -- 172.21.15.60:0/3731230756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f963013fbe0 msgr2=0x7f963013c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:00.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.151+0000 7f9638335700 1 --2- 172.21.15.60:0/3731230756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f963013fbe0 0x7f963013c980 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0x7f96301406d0 tx=0x7f96280091d0).stop 2022-06-29T16:04:00.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.151+0000 7f9638335700 1 -- 172.21.15.60:0/3731230756 shutdown_connections 2022-06-29T16:04:00.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.151+0000 7f9638335700 1 --2- 172.21.15.60:0/3731230756 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f96100539e0 0x7f9610055ea0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:00.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.151+0000 7f9638335700 1 --2- 172.21.15.60:0/3731230756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f963013fbe0 0x7f963013c980 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:00.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.151+0000 7f9638335700 1 --2- 172.21.15.60:0/3731230756 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96300b3e70 0x7f96300ad0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:00.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.151+0000 7f9638335700 1 --2- 172.21.15.60:0/3731230756 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f963005bf30 0x7f96300aff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:00.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.151+0000 7f9638335700 1 -- 172.21.15.60:0/3731230756 >> 172.21.15.60:0/3731230756 conn(0x7f9630064a10 msgr2=0x7f963013e8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:00.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.151+0000 7f9638335700 1 -- 172.21.15.60:0/3731230756 shutdown_connections 2022-06-29T16:04:00.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.151+0000 7f9638335700 1 -- 172.21.15.60:0/3731230756 wait complete. 2022-06-29T16:04:00.165 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2022-06-29T16:04:00.171 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-06-29T16:04:00.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.271+0000 7f4017fff700 1 Processor -- start 2022-06-29T16:04:00.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.271+0000 7f4017fff700 1 -- start start 2022-06-29T16:04:00.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.271+0000 7f4017fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f40180b5eb0 0x7f4018142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:00.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.271+0000 7f4017fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f40180b2c50 0x7f40180b3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:00.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.271+0000 7f4017fff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40180b0cf0 con 0x7f40180b48b0 2022-06-29T16:04:00.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.271+0000 7f4017fff700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40180b54e0 con 0x7f40180b2c50 2022-06-29T16:04:00.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.271+0000 7f4017fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40180b4d10 con 0x7f40180b5eb0 2022-06-29T16:04:00.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.271+0000 7f40177fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f40180b2c50 0x7f40180b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:00.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.271+0000 7f40167fc700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f40180b48b0 0x7f40180b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57066/0 (socket says 172.21.15.60:57066) 2022-06-29T16:04:00.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.271+0000 7f40167fc700 1 -- 172.21.15.60:0/2250008205 learned_addr learned my addr 172.21.15.60:0/2250008205 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:04:00.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.271+0000 7f4016ffd700 1 --2- 172.21.15.60:0/2250008205 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f40180b5eb0 0x7f4018142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:00.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.271+0000 7f40177fe700 1 -- 172.21.15.60:0/2250008205 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f40180b5eb0 msgr2=0x7f4018142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:00.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.271+0000 7f40177fe700 1 --2- 172.21.15.60:0/2250008205 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f40180b5eb0 0x7f4018142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:00.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.271+0000 7f40177fe700 1 -- 172.21.15.60:0/2250008205 >> v1:172.21.15.60:6789/0 conn(0x7f40180b48b0 legacy=0x7f40180b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:00.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.271+0000 7f40177fe700 1 -- 172.21.15.60:0/2250008205 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40180b6290 con 0x7f40180b2c50 2022-06-29T16:04:00.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.271+0000 7f40177fe700 1 --2- 172.21.15.60:0/2250008205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40180b2c50 0x7f40180b3090 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0x7f40100032f0 tx=0x7f40100091d0).ready entity=mon.0 client_cookie=b14f52975834295c server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:00.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.271+0000 7f4015ffb700 1 -- 172.21.15.60:0/2250008205 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f401000b6c0 con 0x7f40180b2c50 2022-06-29T16:04:00.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.271+0000 7f4015ffb700 1 -- 172.21.15.60:0/2250008205 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f401000b820 con 0x7f40180b2c50 2022-06-29T16:04:00.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.271+0000 7f4015ffb700 1 -- 172.21.15.60:0/2250008205 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f401000b990 con 0x7f40180b2c50 2022-06-29T16:04:00.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.271+0000 7f4017fff700 1 -- 172.21.15.60:0/2250008205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40180b2c50 msgr2=0x7f40180b3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:00.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.271+0000 7f4017fff700 1 --2- 172.21.15.60:0/2250008205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40180b2c50 0x7f40180b3090 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0x7f40100032f0 tx=0x7f40100091d0).stop 2022-06-29T16:04:00.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.271+0000 7f4017fff700 1 -- 172.21.15.60:0/2250008205 shutdown_connections 2022-06-29T16:04:00.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.271+0000 7f4017fff700 1 --2- 172.21.15.60:0/2250008205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40180b2c50 0x7f40180b3090 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:00.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.271+0000 7f4017fff700 1 --2- 172.21.15.60:0/2250008205 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f40180b5eb0 0x7f4018142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:00.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.271+0000 7f4017fff700 1 -- 172.21.15.60:0/2250008205 >> 172.21.15.60:0/2250008205 conn(0x7f4018062090 msgr2=0x7f4018062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:00.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.275+0000 7f4017fff700 1 -- 172.21.15.60:0/2250008205 shutdown_connections 2022-06-29T16:04:00.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.275+0000 7f4017fff700 1 -- 172.21.15.60:0/2250008205 wait complete. 2022-06-29T16:04:00.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.275+0000 7f4017fff700 1 Processor -- start 2022-06-29T16:04:00.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.275+0000 7f4017fff700 1 -- start start 2022-06-29T16:04:00.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.275+0000 7f4017fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40180b2c50 0x7f4018146b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:00.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.275+0000 7f4017fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f40180b48b0 0x7f40180af030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:00.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.275+0000 7f4017fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40180b5eb0 0x7f40180b8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:00.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.275+0000 7f4017fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40181401e0 con 0x7f40180b5eb0 2022-06-29T16:04:00.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.275+0000 7f4017fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4018140440 con 0x7f40180b2c50 2022-06-29T16:04:00.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.275+0000 7f4017fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4018156120 con 0x7f40180b48b0 2022-06-29T16:04:00.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.275+0000 7f40177fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40180b5eb0 0x7f40180b8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:00.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.275+0000 7f40177fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40180b5eb0 0x7f40180b8210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48988/0 (socket says 172.21.15.60:48988) 2022-06-29T16:04:00.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.275+0000 7f40177fe700 1 -- 172.21.15.60:0/3309390748 learned_addr learned my addr 172.21.15.60:0/3309390748 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:00.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.275+0000 7f40167fc700 1 --2- 172.21.15.60:0/3309390748 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f40180b48b0 0x7f40180af030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:00.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.275+0000 7f4016ffd700 1 --2- 172.21.15.60:0/3309390748 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40180b2c50 0x7f4018146b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:00.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.275+0000 7f40177fe700 1 -- 172.21.15.60:0/3309390748 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f40180b48b0 msgr2=0x7f40180af030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:00.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.275+0000 7f40177fe700 1 --2- 172.21.15.60:0/3309390748 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f40180b48b0 0x7f40180af030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:00.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.275+0000 7f40177fe700 1 -- 172.21.15.60:0/3309390748 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40180b2c50 msgr2=0x7f4018146b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:00.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.275+0000 7f40177fe700 1 --2- 172.21.15.60:0/3309390748 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40180b2c50 0x7f4018146b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:00.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.275+0000 7f40177fe700 1 -- 172.21.15.60:0/3309390748 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4010009470 con 0x7f40180b5eb0 2022-06-29T16:04:00.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.275+0000 7f40177fe700 1 --2- 172.21.15.60:0/3309390748 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40180b5eb0 0x7f40180b8210 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0x7f40100032f0 tx=0x7f4010016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:00.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.275+0000 7f400ffff700 1 -- 172.21.15.60:0/3309390748 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4010021440 con 0x7f40180b5eb0 2022-06-29T16:04:00.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.275+0000 7f400ffff700 1 -- 172.21.15.60:0/3309390748 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40100215a0 con 0x7f40180b5eb0 2022-06-29T16:04:00.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.275+0000 7f4017fff700 1 -- 172.21.15.60:0/3309390748 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40180af520 con 0x7f40180b5eb0 2022-06-29T16:04:00.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.275+0000 7f400ffff700 1 -- 172.21.15.60:0/3309390748 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4010021900 con 0x7f40180b5eb0 2022-06-29T16:04:00.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.275+0000 7f4017fff700 1 -- 172.21.15.60:0/3309390748 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4018155cf0 con 0x7f40180b5eb0 2022-06-29T16:04:00.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.279+0000 7f4017fff700 1 -- 172.21.15.60:0/3309390748 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40180b8a40 con 0x7f40180b5eb0 2022-06-29T16:04:00.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.279+0000 7f400ffff700 1 -- 172.21.15.60:0/3309390748 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4010027020 con 0x7f40180b5eb0 2022-06-29T16:04:00.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.279+0000 7f400ffff700 1 --2- 172.21.15.60:0/3309390748 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3ff0053a70 0x7f3ff0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:00.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.279+0000 7f400ffff700 1 -- 172.21.15.60:0/3309390748 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f401006d170 con 0x7f40180b5eb0 2022-06-29T16:04:00.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.279+0000 7f4016ffd700 1 --2- 172.21.15.60:0/3309390748 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3ff0053a70 0x7f3ff0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:00.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.279+0000 7f4016ffd700 1 --2- 172.21.15.60:0/3309390748 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3ff0053a70 0x7f3ff0055f30 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f400000b000 tx=0x7f4000008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:00.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.283+0000 7f400ffff700 1 -- 172.21.15.60:0/3309390748 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4010044370 con 0x7f40180b5eb0 2022-06-29T16:04:00.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.415+0000 7f4017fff700 1 -- 172.21.15.60:0/3309390748 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f4018142f10 con 0x7f40180b5eb0 2022-06-29T16:04:00.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.423+0000 7f400ffff700 1 -- 172.21.15.60:0/3309390748 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v9) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4010047990 con 0x7f40180b5eb0 2022-06-29T16:04:00.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.423+0000 7f4017fff700 1 -- 172.21.15.60:0/3309390748 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3ff0053a70 msgr2=0x7f3ff0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:00.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.423+0000 7f4017fff700 1 --2- 172.21.15.60:0/3309390748 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3ff0053a70 0x7f3ff0055f30 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f400000b000 tx=0x7f4000008040).stop 2022-06-29T16:04:00.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.423+0000 7f4017fff700 1 -- 172.21.15.60:0/3309390748 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40180b5eb0 msgr2=0x7f40180b8210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:00.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.423+0000 7f4017fff700 1 --2- 172.21.15.60:0/3309390748 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40180b5eb0 0x7f40180b8210 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0x7f40100032f0 tx=0x7f4010016000).stop 2022-06-29T16:04:00.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.423+0000 7f4017fff700 1 -- 172.21.15.60:0/3309390748 shutdown_connections 2022-06-29T16:04:00.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.423+0000 7f4017fff700 1 --2- 172.21.15.60:0/3309390748 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3ff0053a70 0x7f3ff0055f30 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:00.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.423+0000 7f4017fff700 1 --2- 172.21.15.60:0/3309390748 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40180b5eb0 0x7f40180b8210 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:00.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.423+0000 7f4017fff700 1 --2- 172.21.15.60:0/3309390748 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f40180b48b0 0x7f40180af030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:00.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.423+0000 7f4017fff700 1 --2- 172.21.15.60:0/3309390748 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40180b2c50 0x7f4018146b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:00.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.423+0000 7f4017fff700 1 -- 172.21.15.60:0/3309390748 >> 172.21.15.60:0/3309390748 conn(0x7f4018062090 msgr2=0x7f4018147120 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:00.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.423+0000 7f4017fff700 1 -- 172.21.15.60:0/3309390748 shutdown_connections 2022-06-29T16:04:00.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.423+0000 7f4017fff700 1 -- 172.21.15.60:0/3309390748 wait complete. 2022-06-29T16:04:00.443 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_timeout' 2022-06-29T16:04:00.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.543+0000 7f38d1513700 1 Processor -- start 2022-06-29T16:04:00.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.543+0000 7f38d1513700 1 -- start start 2022-06-29T16:04:00.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.543+0000 7f38d1513700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f38cc0b5950 0x7f38cc067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:00.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.543+0000 7f38d1513700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f38cc063ad0 0x7f38cc14af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:00.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.543+0000 7f38d1513700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38cc0acba0 con 0x7f38cc05bd20 2022-06-29T16:04:00.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.543+0000 7f38d1513700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38cc067220 con 0x7f38cc063ad0 2022-06-29T16:04:00.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.543+0000 7f38d1513700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38cc0b5d30 con 0x7f38cc0b5950 2022-06-29T16:04:00.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.543+0000 7f38c27fc700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f38cc063ad0 0x7f38cc14af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:00.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.543+0000 7f38cb7fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f38cc05bd20 0x7f38cc05c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57082/0 (socket says 172.21.15.60:57082) 2022-06-29T16:04:00.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.543+0000 7f38c27fc700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f38cc063ad0 0x7f38cc14af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48992/0 (socket says 172.21.15.60:48992) 2022-06-29T16:04:00.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.543+0000 7f38cb7fe700 1 -- 172.21.15.60:0/450819740 learned_addr learned my addr 172.21.15.60:0/450819740 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:04:00.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.543+0000 7f38caffd700 1 --2- 172.21.15.60:0/450819740 >> v2:172.21.15.92:3300/0 conn(0x7f38cc0b5950 0x7f38cc067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:00.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.543+0000 7f38c27fc700 1 -- 172.21.15.60:0/450819740 >> v2:172.21.15.92:3300/0 conn(0x7f38cc0b5950 msgr2=0x7f38cc067680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:00.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.543+0000 7f38c27fc700 1 --2- 172.21.15.60:0/450819740 >> v2:172.21.15.92:3300/0 conn(0x7f38cc0b5950 0x7f38cc067680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:00.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.543+0000 7f38c27fc700 1 -- 172.21.15.60:0/450819740 >> v1:172.21.15.60:6789/0 conn(0x7f38cc05bd20 legacy=0x7f38cc05c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:00.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.543+0000 7f38c27fc700 1 -- 172.21.15.60:0/450819740 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38cc0ace00 con 0x7f38cc063ad0 2022-06-29T16:04:00.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.543+0000 7f38c27fc700 1 --2- 172.21.15.60:0/450819740 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38cc063ad0 0x7f38cc14af10 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7f38b4002f70 tx=0x7f38b4006df0).ready entity=mon.0 client_cookie=3d88a323b4fb56d3 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:00.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.543+0000 7f38ca7fc700 1 -- 172.21.15.60:0/450819740 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38b400d570 con 0x7f38cc063ad0 2022-06-29T16:04:00.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.543+0000 7f38ca7fc700 1 -- 172.21.15.60:0/450819740 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38b4000e90 con 0x7f38cc063ad0 2022-06-29T16:04:00.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.543+0000 7f38ca7fc700 1 -- 172.21.15.60:0/450819740 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38b400d8c0 con 0x7f38cc063ad0 2022-06-29T16:04:00.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.543+0000 7f38d1513700 1 -- 172.21.15.60:0/450819740 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38cc063ad0 msgr2=0x7f38cc14af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:00.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.543+0000 7f38d1513700 1 --2- 172.21.15.60:0/450819740 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38cc063ad0 0x7f38cc14af10 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7f38b4002f70 tx=0x7f38b4006df0).stop 2022-06-29T16:04:00.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.543+0000 7f38d1513700 1 -- 172.21.15.60:0/450819740 shutdown_connections 2022-06-29T16:04:00.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.543+0000 7f38d1513700 1 --2- 172.21.15.60:0/450819740 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38cc063ad0 0x7f38cc14af10 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:00.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.543+0000 7f38d1513700 1 --2- 172.21.15.60:0/450819740 >> v2:172.21.15.92:3300/0 conn(0x7f38cc0b5950 0x7f38cc067680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:00.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.543+0000 7f38d1513700 1 -- 172.21.15.60:0/450819740 >> 172.21.15.60:0/450819740 conn(0x7f38cc0632c0 msgr2=0x7f38cc0b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:00.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.543+0000 7f38d1513700 1 -- 172.21.15.60:0/450819740 shutdown_connections 2022-06-29T16:04:00.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.543+0000 7f38d1513700 1 -- 172.21.15.60:0/450819740 wait complete. 2022-06-29T16:04:00.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.547+0000 7f38d1513700 1 Processor -- start 2022-06-29T16:04:00.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.547+0000 7f38d1513700 1 -- start start 2022-06-29T16:04:00.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.547+0000 7f38d1513700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38cc05bd20 0x7f38cc155fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:00.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.547+0000 7f38d1513700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38cc063ad0 0x7f38cc14bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:00.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.547+0000 7f38d1513700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38cc0b5950 0x7f38cc1440f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:00.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.547+0000 7f38d1513700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38cc157270 con 0x7f38cc0b5950 2022-06-29T16:04:00.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.547+0000 7f38d1513700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38cc118610 con 0x7f38cc063ad0 2022-06-29T16:04:00.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.547+0000 7f38d1513700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38cc118840 con 0x7f38cc05bd20 2022-06-29T16:04:00.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.547+0000 7f38cb7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38cc0b5950 0x7f38cc1440f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:00.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.547+0000 7f38cb7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38cc0b5950 0x7f38cc1440f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49002/0 (socket says 172.21.15.60:49002) 2022-06-29T16:04:00.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.547+0000 7f38cb7fe700 1 -- 172.21.15.60:0/2874466763 learned_addr learned my addr 172.21.15.60:0/2874466763 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:00.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.547+0000 7f38c27fc700 1 --2- 172.21.15.60:0/2874466763 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38cc063ad0 0x7f38cc14bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:00.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.547+0000 7f38caffd700 1 --2- 172.21.15.60:0/2874466763 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38cc05bd20 0x7f38cc155fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:00.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.547+0000 7f38cb7fe700 1 -- 172.21.15.60:0/2874466763 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38cc05bd20 msgr2=0x7f38cc155fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:00.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.547+0000 7f38cb7fe700 1 --2- 172.21.15.60:0/2874466763 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38cc05bd20 0x7f38cc155fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:00.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.547+0000 7f38cb7fe700 1 -- 172.21.15.60:0/2874466763 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38cc063ad0 msgr2=0x7f38cc14bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:00.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.547+0000 7f38cb7fe700 1 --2- 172.21.15.60:0/2874466763 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38cc063ad0 0x7f38cc14bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:00.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.547+0000 7f38cb7fe700 1 -- 172.21.15.60:0/2874466763 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b400b180 con 0x7f38cc0b5950 2022-06-29T16:04:00.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.547+0000 7f38cb7fe700 1 --2- 172.21.15.60:0/2874466763 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38cc0b5950 0x7f38cc1440f0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0x7f38cc0ad0a0 tx=0x7f38c4007f30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:00.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.547+0000 7f38c8ff9700 1 -- 172.21.15.60:0/2874466763 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38c4005b80 con 0x7f38cc0b5950 2022-06-29T16:04:00.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.547+0000 7f38c8ff9700 1 -- 172.21.15.60:0/2874466763 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38c4005ce0 con 0x7f38cc0b5950 2022-06-29T16:04:00.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.547+0000 7f38d1513700 1 -- 172.21.15.60:0/2874466763 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38cc0ae3d0 con 0x7f38cc0b5950 2022-06-29T16:04:00.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.547+0000 7f38c8ff9700 1 -- 172.21.15.60:0/2874466763 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38c40072c0 con 0x7f38cc0b5950 2022-06-29T16:04:00.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.547+0000 7f38d1513700 1 -- 172.21.15.60:0/2874466763 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38cc0af320 con 0x7f38cc0b5950 2022-06-29T16:04:00.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.547+0000 7f38d1513700 1 -- 172.21.15.60:0/2874466763 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38cc118a00 con 0x7f38cc0b5950 2022-06-29T16:04:00.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.551+0000 7f38c8ff9700 1 -- 172.21.15.60:0/2874466763 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f38c4010070 con 0x7f38cc0b5950 2022-06-29T16:04:00.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.551+0000 7f38c8ff9700 1 --2- 172.21.15.60:0/2874466763 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f38b00539e0 0x7f38b0055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:00.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.551+0000 7f38c8ff9700 1 -- 172.21.15.60:0/2874466763 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f38c406b5e0 con 0x7f38cc0b5950 2022-06-29T16:04:00.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.551+0000 7f38caffd700 1 --2- 172.21.15.60:0/2874466763 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f38b00539e0 0x7f38b0055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:00.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.551+0000 7f38caffd700 1 --2- 172.21.15.60:0/2874466763 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f38b00539e0 0x7f38b0055ea0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f38bc00b000 tx=0x7f38bc008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:00.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.559+0000 7f38c8ff9700 1 -- 172.21.15.60:0/2874466763 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f38c4042740 con 0x7f38cc0b5950 2022-06-29T16:04:00.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:00.723+0000 7f38d1513700 1 -- 172.21.15.60:0/2874466763 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_timeout"]} v 0) v1 -- 0x7f38cc0573d0 con 0x7f38cc0b5950 2022-06-29T16:04:01.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.159+0000 7f38c8ff9700 1 -- 172.21.15.60:0/2874466763 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_timeout"]}]=0 v243) v1 ==== 147+0+0 (secure 0 0 0) 0x7f38c4045d60 con 0x7f38cc0b5950 2022-06-29T16:04:01.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.163+0000 7f38d1513700 1 -- 172.21.15.60:0/2874466763 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f38b00539e0 msgr2=0x7f38b0055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:01.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.163+0000 7f38d1513700 1 --2- 172.21.15.60:0/2874466763 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f38b00539e0 0x7f38b0055ea0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f38bc00b000 tx=0x7f38bc008040).stop 2022-06-29T16:04:01.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.163+0000 7f38d1513700 1 -- 172.21.15.60:0/2874466763 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38cc0b5950 msgr2=0x7f38cc1440f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:01.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.163+0000 7f38d1513700 1 --2- 172.21.15.60:0/2874466763 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38cc0b5950 0x7f38cc1440f0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0x7f38cc0ad0a0 tx=0x7f38c4007f30).stop 2022-06-29T16:04:01.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.163+0000 7f38d1513700 1 -- 172.21.15.60:0/2874466763 shutdown_connections 2022-06-29T16:04:01.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.163+0000 7f38d1513700 1 --2- 172.21.15.60:0/2874466763 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f38b00539e0 0x7f38b0055ea0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:01.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.163+0000 7f38d1513700 1 --2- 172.21.15.60:0/2874466763 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38cc0b5950 0x7f38cc1440f0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:01.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.163+0000 7f38d1513700 1 --2- 172.21.15.60:0/2874466763 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38cc063ad0 0x7f38cc14bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:01.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.163+0000 7f38d1513700 1 --2- 172.21.15.60:0/2874466763 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38cc05bd20 0x7f38cc155fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:01.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.163+0000 7f38d1513700 1 -- 172.21.15.60:0/2874466763 >> 172.21.15.60:0/2874466763 conn(0x7f38cc0632c0 msgr2=0x7f38cc0b4ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:01.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.163+0000 7f38d1513700 1 -- 172.21.15.60:0/2874466763 shutdown_connections 2022-06-29T16:04:01.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.163+0000 7f38d1513700 1 -- 172.21.15.60:0/2874466763 wait complete. 2022-06-29T16:04:01.185 INFO:tasks.cephfs_test_runner:test_drop_cache_command_timeout (tasks.cephfs.test_misc.TestCacheDrop) ... ok 2022-06-29T16:04:01.186 INFO:tasks.cephfs_test_runner:Starting test: test_cap_revoke_nonresponder (tasks.cephfs.test_misc.TestMisc) 2022-06-29T16:04:01.186 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_cap_revoke_nonresponder' 2022-06-29T16:04:01.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.287+0000 7fc25b59e700 1 Processor -- start 2022-06-29T16:04:01.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.287+0000 7fc25b59e700 1 -- start start 2022-06-29T16:04:01.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.291+0000 7fc25b59e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc25c0acd60 0x7fc25c144900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:01.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.291+0000 7fc25b59e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc25c0ad990 0x7fc25c1296f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:01.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.291+0000 7fc25b59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc25c0657d0 0x7fc25c129be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:01.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.291+0000 7fc25b59e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc25c064a70 con 0x7fc25c0acd60 2022-06-29T16:04:01.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.291+0000 7fc25b59e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc25c143cf0 con 0x7fc25c0ad990 2022-06-29T16:04:01.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.291+0000 7fc25b59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc25c0ad140 con 0x7fc25c0657d0 2022-06-29T16:04:01.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.291+0000 7fc25a59c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc25c0acd60 0x7fc25c144900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:01.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.291+0000 7fc25a59c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc25c0acd60 0x7fc25c144900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49008/0 (socket says 172.21.15.60:49008) 2022-06-29T16:04:01.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.291+0000 7fc25ad9d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc25c0657d0 0x7fc25c129be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:01.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.291+0000 7fc253fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc25c0ad990 0x7fc25c1296f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:01.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.291+0000 7fc25a59c700 1 -- 172.21.15.60:0/779930454 learned_addr learned my addr 172.21.15.60:0/779930454 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:01.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.291+0000 7fc253fff700 1 -- 172.21.15.60:0/779930454 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc25c0657d0 msgr2=0x7fc25c129be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:01.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.291+0000 7fc253fff700 1 --2- 172.21.15.60:0/779930454 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc25c0657d0 0x7fc25c129be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:01.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.291+0000 7fc253fff700 1 -- 172.21.15.60:0/779930454 >> v2:172.21.15.60:3300/0 conn(0x7fc25c0acd60 msgr2=0x7fc25c144900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:01.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.291+0000 7fc253fff700 1 --2- 172.21.15.60:0/779930454 >> v2:172.21.15.60:3300/0 conn(0x7fc25c0acd60 0x7fc25c144900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:01.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.291+0000 7fc253fff700 1 -- 172.21.15.60:0/779930454 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc25c143f20 con 0x7fc25c0ad990 2022-06-29T16:04:01.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.291+0000 7fc25a59c700 1 --2- 172.21.15.60:0/779930454 >> v2:172.21.15.60:3300/0 conn(0x7fc25c0acd60 0x7fc25c144900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:04:01.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.291+0000 7fc253fff700 1 --2- 172.21.15.60:0/779930454 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc25c0ad990 0x7fc25c1296f0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7fc244002f70 tx=0x7fc244006df0).ready entity=mon.1 client_cookie=f3411f8523f7f719 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:01.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.291+0000 7fc259d9b700 1 -- 172.21.15.60:0/779930454 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc24400d4a0 con 0x7fc25c0ad990 2022-06-29T16:04:01.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.291+0000 7fc259d9b700 1 -- 172.21.15.60:0/779930454 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc244000e90 con 0x7fc25c0ad990 2022-06-29T16:04:01.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.291+0000 7fc259d9b700 1 -- 172.21.15.60:0/779930454 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc24400d7f0 con 0x7fc25c0ad990 2022-06-29T16:04:01.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.291+0000 7fc25b59e700 1 -- 172.21.15.60:0/779930454 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc25c0ad990 msgr2=0x7fc25c1296f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:01.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.291+0000 7fc25b59e700 1 --2- 172.21.15.60:0/779930454 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc25c0ad990 0x7fc25c1296f0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7fc244002f70 tx=0x7fc244006df0).stop 2022-06-29T16:04:01.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.291+0000 7fc25b59e700 1 -- 172.21.15.60:0/779930454 shutdown_connections 2022-06-29T16:04:01.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.291+0000 7fc25b59e700 1 --2- 172.21.15.60:0/779930454 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc25c0657d0 0x7fc25c129be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:01.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.291+0000 7fc25b59e700 1 --2- 172.21.15.60:0/779930454 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc25c0ad990 0x7fc25c1296f0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:01.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.291+0000 7fc25b59e700 1 --2- 172.21.15.60:0/779930454 >> v2:172.21.15.60:3300/0 conn(0x7fc25c0acd60 0x7fc25c144900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:01.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.291+0000 7fc25b59e700 1 -- 172.21.15.60:0/779930454 >> 172.21.15.60:0/779930454 conn(0x7fc25c0b6f80 msgr2=0x7fc25c0b7390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:01.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.291+0000 7fc25b59e700 1 -- 172.21.15.60:0/779930454 shutdown_connections 2022-06-29T16:04:01.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.291+0000 7fc25b59e700 1 -- 172.21.15.60:0/779930454 wait complete. 2022-06-29T16:04:01.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.295+0000 7fc25b59e700 1 Processor -- start 2022-06-29T16:04:01.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.295+0000 7fc25b59e700 1 -- start start 2022-06-29T16:04:01.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.295+0000 7fc25b59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc25c0657d0 0x7fc25c0b0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:01.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.295+0000 7fc25b59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc25c0acd60 0x7fc25c126790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:01.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.295+0000 7fc25b59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc25c0ad990 0x7fc25c126c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:01.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.295+0000 7fc25b59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc25c14f880 con 0x7fc25c0acd60 2022-06-29T16:04:01.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.295+0000 7fc25b59e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc25c146010 con 0x7fc25c0657d0 2022-06-29T16:04:01.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.295+0000 7fc25b59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc25c1462b0 con 0x7fc25c0ad990 2022-06-29T16:04:01.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.295+0000 7fc253fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc25c0acd60 0x7fc25c126790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:01.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.295+0000 7fc253fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc25c0acd60 0x7fc25c126790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49012/0 (socket says 172.21.15.60:49012) 2022-06-29T16:04:01.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.295+0000 7fc253fff700 1 -- 172.21.15.60:0/871693754 learned_addr learned my addr 172.21.15.60:0/871693754 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:01.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.295+0000 7fc25ad9d700 1 --2- 172.21.15.60:0/871693754 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc25c0ad990 0x7fc25c126c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:01.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.295+0000 7fc25a59c700 1 --2- 172.21.15.60:0/871693754 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc25c0657d0 0x7fc25c0b0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:01.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.295+0000 7fc25ad9d700 1 -- 172.21.15.60:0/871693754 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc25c0657d0 msgr2=0x7fc25c0b0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:01.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.295+0000 7fc25ad9d700 1 --2- 172.21.15.60:0/871693754 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc25c0657d0 0x7fc25c0b0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:01.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.295+0000 7fc25ad9d700 1 -- 172.21.15.60:0/871693754 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc25c0acd60 msgr2=0x7fc25c126790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:01.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.295+0000 7fc25ad9d700 1 --2- 172.21.15.60:0/871693754 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc25c0acd60 0x7fc25c126790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:01.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.295+0000 7fc25ad9d700 1 -- 172.21.15.60:0/871693754 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc24c00b1e0 con 0x7fc25c0ad990 2022-06-29T16:04:01.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.295+0000 7fc25ad9d700 1 --2- 172.21.15.60:0/871693754 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc25c0ad990 0x7fc25c126c80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fc25400c000 tx=0x7fc2540091d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:01.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.295+0000 7fc2537fe700 1 -- 172.21.15.60:0/871693754 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc25400c670 con 0x7fc25c0ad990 2022-06-29T16:04:01.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.295+0000 7fc2537fe700 1 -- 172.21.15.60:0/871693754 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc25400c7d0 con 0x7fc25c0ad990 2022-06-29T16:04:01.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.295+0000 7fc2537fe700 1 -- 172.21.15.60:0/871693754 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc25400cb30 con 0x7fc25c0ad990 2022-06-29T16:04:01.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.295+0000 7fc25b59e700 1 -- 172.21.15.60:0/871693754 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc24400b180 con 0x7fc25c0ad990 2022-06-29T16:04:01.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.295+0000 7fc25b59e700 1 -- 172.21.15.60:0/871693754 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc25c146810 con 0x7fc25c0ad990 2022-06-29T16:04:01.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.299+0000 7fc25b59e700 1 -- 172.21.15.60:0/871693754 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc25c12a860 con 0x7fc25c0ad990 2022-06-29T16:04:01.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.299+0000 7fc2537fe700 1 -- 172.21.15.60:0/871693754 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc254022070 con 0x7fc25c0ad990 2022-06-29T16:04:01.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.299+0000 7fc2537fe700 1 --2- 172.21.15.60:0/871693754 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc240053950 0x7fc240055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:01.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.299+0000 7fc2537fe700 1 -- 172.21.15.60:0/871693754 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fc25406c200 con 0x7fc25c0ad990 2022-06-29T16:04:01.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.299+0000 7fc25a59c700 1 --2- 172.21.15.60:0/871693754 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc240053950 0x7fc240055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:01.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.299+0000 7fc25a59c700 1 --2- 172.21.15.60:0/871693754 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc240053950 0x7fc240055e10 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fc24c00b000 tx=0x7fc24c006eb0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:01.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.311+0000 7fc2537fe700 1 -- 172.21.15.60:0/871693754 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc254043400 con 0x7fc25c0ad990 2022-06-29T16:04:01.486 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:01.479+0000 7fc25b59e700 1 -- 172.21.15.60:0/871693754 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_cap_revoke_nonresponder"]} v 0) v1 -- 0x7fc25c0573d0 con 0x7fc25c0ad990 2022-06-29T16:04:02.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.163+0000 7fc2537fe700 1 -- 172.21.15.60:0/871693754 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_cap_revoke_nonresponder"]}]=0 v244) v1 ==== 142+0+0 (secure 0 0 0) 0x7fc2540160d0 con 0x7fc25c0ad990 2022-06-29T16:04:02.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.167+0000 7fc25b59e700 1 -- 172.21.15.60:0/871693754 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc240053950 msgr2=0x7fc240055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:02.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.167+0000 7fc25b59e700 1 --2- 172.21.15.60:0/871693754 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc240053950 0x7fc240055e10 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fc24c00b000 tx=0x7fc24c006eb0).stop 2022-06-29T16:04:02.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.167+0000 7fc25b59e700 1 -- 172.21.15.60:0/871693754 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc25c0ad990 msgr2=0x7fc25c126c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:02.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.167+0000 7fc25b59e700 1 --2- 172.21.15.60:0/871693754 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc25c0ad990 0x7fc25c126c80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fc25400c000 tx=0x7fc2540091d0).stop 2022-06-29T16:04:02.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.167+0000 7fc25b59e700 1 -- 172.21.15.60:0/871693754 shutdown_connections 2022-06-29T16:04:02.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.167+0000 7fc25b59e700 1 --2- 172.21.15.60:0/871693754 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc240053950 0x7fc240055e10 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:02.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.167+0000 7fc25b59e700 1 --2- 172.21.15.60:0/871693754 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc25c0ad990 0x7fc25c126c80 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:02.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.167+0000 7fc25b59e700 1 --2- 172.21.15.60:0/871693754 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc25c0acd60 0x7fc25c126790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:02.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.167+0000 7fc25b59e700 1 --2- 172.21.15.60:0/871693754 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc25c0657d0 0x7fc25c0b0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:02.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.167+0000 7fc25b59e700 1 -- 172.21.15.60:0/871693754 >> 172.21.15.60:0/871693754 conn(0x7fc25c0b6f80 msgr2=0x7fc25c140950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:02.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.167+0000 7fc25b59e700 1 -- 172.21.15.60:0/871693754 shutdown_connections 2022-06-29T16:04:02.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.167+0000 7fc25b59e700 1 -- 172.21.15.60:0/871693754 wait complete. 2022-06-29T16:04:02.187 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-06-29T16:04:02.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.295+0000 7f40c17bc700 1 Processor -- start 2022-06-29T16:04:02.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.295+0000 7f40c17bc700 1 -- start start 2022-06-29T16:04:02.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.299+0000 7f40c17bc700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f40bc0af850 0x7f40bc05fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:02.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.299+0000 7f40c17bc700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f40bc067020 0x7f40bc067490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:02.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.299+0000 7f40c17bc700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40bc0ae000 con 0x7f40bc05f7c0 2022-06-29T16:04:02.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.299+0000 7f40c17bc700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40bc146f50 con 0x7f40bc0af850 2022-06-29T16:04:02.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.299+0000 7f40c17bc700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40bc0ae230 con 0x7f40bc067020 2022-06-29T16:04:02.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.299+0000 7f40bbfff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f40bc05f7c0 0x7f40bc0ae6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57106/0 (socket says 172.21.15.60:57106) 2022-06-29T16:04:02.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.299+0000 7f40bbfff700 1 -- 172.21.15.60:0/384971397 learned_addr learned my addr 172.21.15.60:0/384971397 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:04:02.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.299+0000 7f40b3fff700 1 --2- 172.21.15.60:0/384971397 >> v2:172.21.15.60:3300/0 conn(0x7f40bc0af850 0x7f40bc05fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:02.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.299+0000 7f40c0fbb700 1 --2- 172.21.15.60:0/384971397 >> v2:172.21.15.92:3300/0 conn(0x7f40bc067020 0x7f40bc067490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:02.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.299+0000 7f40bb7fe700 1 -- 172.21.15.60:0/384971397 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2161627065 0 0) 0x7f40bc0ae000 con 0x7f40bc05f7c0 2022-06-29T16:04:02.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.299+0000 7f40bb7fe700 1 -- 172.21.15.60:0/384971397 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f409c003260 con 0x7f40bc05f7c0 2022-06-29T16:04:02.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.299+0000 7f40b3fff700 1 -- 172.21.15.60:0/384971397 >> v2:172.21.15.92:3300/0 conn(0x7f40bc067020 msgr2=0x7f40bc067490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:02.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.299+0000 7f40b3fff700 1 --2- 172.21.15.60:0/384971397 >> v2:172.21.15.92:3300/0 conn(0x7f40bc067020 0x7f40bc067490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:02.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.299+0000 7f40b3fff700 1 -- 172.21.15.60:0/384971397 >> v1:172.21.15.60:6789/0 conn(0x7f40bc05f7c0 legacy=0x7f40bc0ae6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:02.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.299+0000 7f40b3fff700 1 -- 172.21.15.60:0/384971397 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40bc146bd0 con 0x7f40bc0af850 2022-06-29T16:04:02.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.299+0000 7f40b3fff700 1 --2- 172.21.15.60:0/384971397 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40bc0af850 0x7f40bc05fba0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0x7f40a4002f70 tx=0x7f40a4006df0).ready entity=mon.0 client_cookie=9071859fd833dff5 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:02.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.299+0000 7f40bb7fe700 1 -- 172.21.15.60:0/384971397 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40a400d4f0 con 0x7f40bc0af850 2022-06-29T16:04:02.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.299+0000 7f40bb7fe700 1 -- 172.21.15.60:0/384971397 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40a4000e90 con 0x7f40bc0af850 2022-06-29T16:04:02.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.299+0000 7f40bb7fe700 1 -- 172.21.15.60:0/384971397 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40a400d840 con 0x7f40bc0af850 2022-06-29T16:04:02.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.299+0000 7f40c17bc700 1 -- 172.21.15.60:0/384971397 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40bc0af850 msgr2=0x7f40bc05fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:02.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.299+0000 7f40c17bc700 1 --2- 172.21.15.60:0/384971397 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40bc0af850 0x7f40bc05fba0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0x7f40a4002f70 tx=0x7f40a4006df0).stop 2022-06-29T16:04:02.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.299+0000 7f40c17bc700 1 -- 172.21.15.60:0/384971397 shutdown_connections 2022-06-29T16:04:02.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.299+0000 7f40c17bc700 1 --2- 172.21.15.60:0/384971397 >> v2:172.21.15.92:3300/0 conn(0x7f40bc067020 0x7f40bc067490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:02.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.299+0000 7f40c17bc700 1 --2- 172.21.15.60:0/384971397 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40bc0af850 0x7f40bc05fba0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:02.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.299+0000 7f40c17bc700 1 -- 172.21.15.60:0/384971397 >> 172.21.15.60:0/384971397 conn(0x7f40bc063a60 msgr2=0x7f40bc05f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:02.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.299+0000 7f40c17bc700 1 -- 172.21.15.60:0/384971397 shutdown_connections 2022-06-29T16:04:02.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.299+0000 7f40c17bc700 1 -- 172.21.15.60:0/384971397 wait complete. 2022-06-29T16:04:02.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.299+0000 7f40c17bc700 1 Processor -- start 2022-06-29T16:04:02.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.299+0000 7f40c17bc700 1 -- start start 2022-06-29T16:04:02.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.303+0000 7f40c17bc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40bc05f7c0 0x7f40bc0b0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:02.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.303+0000 7f40c17bc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40bc067020 0x7f40bc13f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:02.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.303+0000 7f40c17bc700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f40bc063e40 0x7f40bc13f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:02.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.303+0000 7f40c17bc700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc145460 con 0x7f40bc067020 2022-06-29T16:04:02.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.303+0000 7f40c17bc700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc145690 con 0x7f40bc05f7c0 2022-06-29T16:04:02.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.303+0000 7f40c17bc700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc145930 con 0x7f40bc063e40 2022-06-29T16:04:02.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.303+0000 7f40b3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40bc067020 0x7f40bc13f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:02.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.303+0000 7f40bbfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40bc05f7c0 0x7f40bc0b0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:02.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.303+0000 7f40c0fbb700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f40bc063e40 0x7f40bc13f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:02.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.303+0000 7f40b3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40bc067020 0x7f40bc13f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49030/0 (socket says 172.21.15.60:49030) 2022-06-29T16:04:02.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.303+0000 7f40b3fff700 1 -- 172.21.15.60:0/1954218338 learned_addr learned my addr 172.21.15.60:0/1954218338 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:02.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.303+0000 7f40b3fff700 1 -- 172.21.15.60:0/1954218338 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f40bc063e40 msgr2=0x7f40bc13f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:02.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.303+0000 7f40b3fff700 1 --2- 172.21.15.60:0/1954218338 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f40bc063e40 0x7f40bc13f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:02.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.303+0000 7f40b3fff700 1 -- 172.21.15.60:0/1954218338 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40bc05f7c0 msgr2=0x7f40bc0b0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:02.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.303+0000 7f40b3fff700 1 --2- 172.21.15.60:0/1954218338 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40bc05f7c0 0x7f40bc0b0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:02.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.303+0000 7f40b3fff700 1 -- 172.21.15.60:0/1954218338 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a400b180 con 0x7f40bc067020 2022-06-29T16:04:02.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.303+0000 7f40b3fff700 1 --2- 172.21.15.60:0/1954218338 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40bc067020 0x7f40bc13f4f0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7f40a4000b60 tx=0x7f40a4014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:02.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.303+0000 7f40b9ffb700 1 -- 172.21.15.60:0/1954218338 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40a4013d60 con 0x7f40bc067020 2022-06-29T16:04:02.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.303+0000 7f40b9ffb700 1 -- 172.21.15.60:0/1954218338 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40a400de40 con 0x7f40bc067020 2022-06-29T16:04:02.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.303+0000 7f40b9ffb700 1 -- 172.21.15.60:0/1954218338 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40a401e5e0 con 0x7f40bc067020 2022-06-29T16:04:02.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.303+0000 7f40c17bc700 1 -- 172.21.15.60:0/1954218338 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40bc0b0a30 con 0x7f40bc067020 2022-06-29T16:04:02.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.303+0000 7f40c17bc700 1 -- 172.21.15.60:0/1954218338 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40bc1440c0 con 0x7f40bc067020 2022-06-29T16:04:02.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.307+0000 7f40c17bc700 1 -- 172.21.15.60:0/1954218338 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40bc146a30 con 0x7f40bc067020 2022-06-29T16:04:02.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.307+0000 7f40b9ffb700 1 -- 172.21.15.60:0/1954218338 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f40a401c070 con 0x7f40bc067020 2022-06-29T16:04:02.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.307+0000 7f40b9ffb700 1 --2- 172.21.15.60:0/1954218338 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f40a00539e0 0x7f40a0055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:02.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.307+0000 7f40b9ffb700 1 -- 172.21.15.60:0/1954218338 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f40a406c790 con 0x7f40bc067020 2022-06-29T16:04:02.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.307+0000 7f40bbfff700 1 --2- 172.21.15.60:0/1954218338 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f40a00539e0 0x7f40a0055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:02.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.307+0000 7f40bbfff700 1 --2- 172.21.15.60:0/1954218338 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f40a00539e0 0x7f40a0055ea0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f40ac00d000 tx=0x7f40ac009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:02.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.315+0000 7f40b9ffb700 1 -- 172.21.15.60:0/1954218338 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f40bc146a30 con 0x7f40bc067020 2022-06-29T16:04:02.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.439+0000 7f40c17bc700 1 -- 172.21.15.60:0/1954218338 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f40bc146a30 con 0x7f40bc067020 2022-06-29T16:04:02.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.447+0000 7f40b9ffb700 1 -- 172.21.15.60:0/1954218338 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f40bc146a30 con 0x7f40bc067020 2022-06-29T16:04:02.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.451+0000 7f40c17bc700 1 -- 172.21.15.60:0/1954218338 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f40a00539e0 msgr2=0x7f40a0055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:02.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.451+0000 7f40c17bc700 1 --2- 172.21.15.60:0/1954218338 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f40a00539e0 0x7f40a0055ea0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f40ac00d000 tx=0x7f40ac009040).stop 2022-06-29T16:04:02.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.451+0000 7f40c17bc700 1 -- 172.21.15.60:0/1954218338 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40bc067020 msgr2=0x7f40bc13f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:02.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.451+0000 7f40c17bc700 1 --2- 172.21.15.60:0/1954218338 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40bc067020 0x7f40bc13f4f0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7f40a4000b60 tx=0x7f40a4014000).stop 2022-06-29T16:04:02.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.451+0000 7f40c17bc700 1 -- 172.21.15.60:0/1954218338 shutdown_connections 2022-06-29T16:04:02.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.451+0000 7f40c17bc700 1 --2- 172.21.15.60:0/1954218338 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f40a00539e0 0x7f40a0055ea0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:02.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.451+0000 7f40c17bc700 1 --2- 172.21.15.60:0/1954218338 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f40bc063e40 0x7f40bc13f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:02.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.451+0000 7f40c17bc700 1 --2- 172.21.15.60:0/1954218338 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40bc067020 0x7f40bc13f4f0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:02.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.451+0000 7f40c17bc700 1 --2- 172.21.15.60:0/1954218338 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40bc05f7c0 0x7f40bc0b0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:02.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.451+0000 7f40c17bc700 1 -- 172.21.15.60:0/1954218338 >> 172.21.15.60:0/1954218338 conn(0x7f40bc063a60 msgr2=0x7f40bc142c40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:02.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.451+0000 7f40c17bc700 1 -- 172.21.15.60:0/1954218338 shutdown_connections 2022-06-29T16:04:02.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.451+0000 7f40c17bc700 1 -- 172.21.15.60:0/1954218338 wait complete. 2022-06-29T16:04:02.469 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-06-29T16:04:02.470 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:04:02.486 DEBUG:teuthology.orchestra.run.smithi092:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:04:02.509 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:04:02.531 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:04:02.552 DEBUG:teuthology.orchestra.run.smithi197:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:04:02.575 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-06-29T16:04:02.575 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:04:02.576 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:04:02.576 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:04:02.584 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-06-29T16:04:02.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:04:02.585 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-06-29T16:04:02.586 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:04:02.586 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:04:02.587 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:04:02.603 INFO:teuthology.orchestra.run.smithi156.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-06-29T16:04:02.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:04:02.605 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-06-29T16:04:02.605 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:04:02.606 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:04:02.606 DEBUG:teuthology.orchestra.run.smithi197:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T16:04:02.624 INFO:teuthology.orchestra.run.smithi197.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-06-29T16:04:02.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:04:02.626 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-06-29T16:04:02.626 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-06-29T16:04:02.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.731+0000 7f833b59e700 1 Processor -- start 2022-06-29T16:04:02.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.731+0000 7f833b59e700 1 -- start start 2022-06-29T16:04:02.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.735+0000 7f833b59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f833c0b3d00 0x7f833c146790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:02.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.735+0000 7f833b59e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f833c05bf30 0x7f833c142900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:02.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.735+0000 7f833b59e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f833c142df0 0x7f833c13fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:02.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.735+0000 7f833b59e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f833c05d4d0 con 0x7f833c05bf30 2022-06-29T16:04:02.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.735+0000 7f833b59e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f833c0b40e0 con 0x7f833c142df0 2022-06-29T16:04:02.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.735+0000 7f833b59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833c146050 con 0x7f833c0b3d00 2022-06-29T16:04:02.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.735+0000 7f8339d9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f833c05bf30 0x7f833c142900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:02.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.735+0000 7f8339d9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f833c05bf30 0x7f833c142900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49034/0 (socket says 172.21.15.60:49034) 2022-06-29T16:04:02.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.735+0000 7f8339d9b700 1 -- 172.21.15.60:0/2989627747 learned_addr learned my addr 172.21.15.60:0/2989627747 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:02.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.735+0000 7f833a59c700 1 --2- 172.21.15.60:0/2989627747 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f833c0b3d00 0x7f833c146790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:02.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.735+0000 7f833ad9d700 1 --2- 172.21.15.60:0/2989627747 >> v2:172.21.15.92:3300/0 conn(0x7f833c142df0 0x7f833c13fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:02.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.735+0000 7f8339d9b700 1 -- 172.21.15.60:0/2989627747 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f833c0b3d00 msgr2=0x7f833c146790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:02.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.735+0000 7f8339d9b700 1 --2- 172.21.15.60:0/2989627747 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f833c0b3d00 0x7f833c146790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:02.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.735+0000 7f8339d9b700 1 -- 172.21.15.60:0/2989627747 >> v2:172.21.15.92:3300/0 conn(0x7f833c142df0 msgr2=0x7f833c13fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:02.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.735+0000 7f8339d9b700 1 --2- 172.21.15.60:0/2989627747 >> v2:172.21.15.92:3300/0 conn(0x7f833c142df0 0x7f833c13fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:02.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.735+0000 7f8339d9b700 1 -- 172.21.15.60:0/2989627747 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833c1462f0 con 0x7f833c05bf30 2022-06-29T16:04:02.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.735+0000 7f833ad9d700 1 --2- 172.21.15.60:0/2989627747 >> v2:172.21.15.92:3300/0 conn(0x7f833c142df0 0x7f833c13fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:04:02.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.735+0000 7f8339d9b700 1 --2- 172.21.15.60:0/2989627747 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f833c05bf30 0x7f833c142900 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0x7f83240032f0 tx=0x7f8324009290).ready entity=mon.0 client_cookie=80872725501b8f13 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:02.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.735+0000 7f833959a700 1 -- 172.21.15.60:0/2989627747 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f832400b870 con 0x7f833c05bf30 2022-06-29T16:04:02.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.735+0000 7f833959a700 1 -- 172.21.15.60:0/2989627747 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f832400b9d0 con 0x7f833c05bf30 2022-06-29T16:04:02.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.735+0000 7f833959a700 1 -- 172.21.15.60:0/2989627747 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f832400bb40 con 0x7f833c05bf30 2022-06-29T16:04:02.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.735+0000 7f833b59e700 1 -- 172.21.15.60:0/2989627747 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f833c05bf30 msgr2=0x7f833c142900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:02.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.735+0000 7f833b59e700 1 --2- 172.21.15.60:0/2989627747 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f833c05bf30 0x7f833c142900 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0x7f83240032f0 tx=0x7f8324009290).stop 2022-06-29T16:04:02.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.735+0000 7f833b59e700 1 -- 172.21.15.60:0/2989627747 shutdown_connections 2022-06-29T16:04:02.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.735+0000 7f833b59e700 1 --2- 172.21.15.60:0/2989627747 >> v2:172.21.15.92:3300/0 conn(0x7f833c142df0 0x7f833c13fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:02.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.735+0000 7f833b59e700 1 --2- 172.21.15.60:0/2989627747 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f833c05bf30 0x7f833c142900 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:02.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.735+0000 7f833b59e700 1 --2- 172.21.15.60:0/2989627747 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f833c0b3d00 0x7f833c146790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:02.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.735+0000 7f833b59e700 1 -- 172.21.15.60:0/2989627747 >> 172.21.15.60:0/2989627747 conn(0x7f833c064810 msgr2=0x7f833c064bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:02.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.735+0000 7f833b59e700 1 -- 172.21.15.60:0/2989627747 shutdown_connections 2022-06-29T16:04:02.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.735+0000 7f833b59e700 1 -- 172.21.15.60:0/2989627747 wait complete. 2022-06-29T16:04:02.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.735+0000 7f833b59e700 1 Processor -- start 2022-06-29T16:04:02.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.735+0000 7f833b59e700 1 -- start start 2022-06-29T16:04:02.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.739+0000 7f833b59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f833c05bf30 0x7f833c0b07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:02.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.739+0000 7f833b59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f833c0b3d00 0x7f833c151a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:02.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.739+0000 7f833b59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f833c142df0 0x7f833c0669d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:02.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.739+0000 7f833b59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833c115050 con 0x7f833c0b3d00 2022-06-29T16:04:02.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.739+0000 7f833b59e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833c114b10 con 0x7f833c142df0 2022-06-29T16:04:02.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.739+0000 7f833b59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833c114db0 con 0x7f833c05bf30 2022-06-29T16:04:02.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.739+0000 7f8339d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f833c0b3d00 0x7f833c151a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:02.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.739+0000 7f8339d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f833c0b3d00 0x7f833c151a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49040/0 (socket says 172.21.15.60:49040) 2022-06-29T16:04:02.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.739+0000 7f8339d9b700 1 -- 172.21.15.60:0/1510728425 learned_addr learned my addr 172.21.15.60:0/1510728425 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:02.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.739+0000 7f833a59c700 1 --2- 172.21.15.60:0/1510728425 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f833c05bf30 0x7f833c0b07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:02.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.739+0000 7f833ad9d700 1 --2- 172.21.15.60:0/1510728425 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f833c142df0 0x7f833c0669d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:02.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.739+0000 7f8339d9b700 1 -- 172.21.15.60:0/1510728425 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f833c05bf30 msgr2=0x7f833c0b07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:02.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.739+0000 7f8339d9b700 1 --2- 172.21.15.60:0/1510728425 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f833c05bf30 0x7f833c0b07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:02.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.739+0000 7f8339d9b700 1 -- 172.21.15.60:0/1510728425 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f833c142df0 msgr2=0x7f833c0669d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:02.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.739+0000 7f8339d9b700 1 --2- 172.21.15.60:0/1510728425 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f833c142df0 0x7f833c0669d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:02.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.739+0000 7f8339d9b700 1 -- 172.21.15.60:0/1510728425 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8324009530 con 0x7f833c0b3d00 2022-06-29T16:04:02.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.739+0000 7f833ad9d700 1 --2- 172.21.15.60:0/1510728425 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f833c142df0 0x7f833c0669d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:04:02.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.739+0000 7f8339d9b700 1 --2- 172.21.15.60:0/1510728425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f833c0b3d00 0x7f833c151a60 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0x7f83240032f0 tx=0x7f8324016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:02.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.739+0000 7f83337fe700 1 -- 172.21.15.60:0/1510728425 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f832400b870 con 0x7f833c0b3d00 2022-06-29T16:04:02.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.739+0000 7f83337fe700 1 -- 172.21.15.60:0/1510728425 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8324021630 con 0x7f833c0b3d00 2022-06-29T16:04:02.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.739+0000 7f83337fe700 1 -- 172.21.15.60:0/1510728425 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8324021990 con 0x7f833c0b3d00 2022-06-29T16:04:02.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.739+0000 7f833b59e700 1 -- 172.21.15.60:0/1510728425 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f833c114590 con 0x7f833c0b3d00 2022-06-29T16:04:02.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.739+0000 7f833b59e700 1 -- 172.21.15.60:0/1510728425 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f833c13fcc0 con 0x7f833c0b3d00 2022-06-29T16:04:02.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.739+0000 7f833b59e700 1 -- 172.21.15.60:0/1510728425 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f833c1164a0 con 0x7f833c0b3d00 2022-06-29T16:04:02.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.743+0000 7f83337fe700 1 -- 172.21.15.60:0/1510728425 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8324028070 con 0x7f833c0b3d00 2022-06-29T16:04:02.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.743+0000 7f83337fe700 1 --2- 172.21.15.60:0/1510728425 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8320053b00 0x7f8320055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:02.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.743+0000 7f83337fe700 1 -- 172.21.15.60:0/1510728425 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f832406cbe0 con 0x7f833c0b3d00 2022-06-29T16:04:02.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.743+0000 7f833a59c700 1 --2- 172.21.15.60:0/1510728425 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8320053b00 0x7f8320055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:02.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.743+0000 7f833a59c700 1 --2- 172.21.15.60:0/1510728425 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8320053b00 0x7f8320055fc0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f832c00c000 tx=0x7f832c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:02.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.747+0000 7f83337fe700 1 -- 172.21.15.60:0/1510728425 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8324043de0 con 0x7f833c0b3d00 2022-06-29T16:04:02.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.879+0000 7f833b59e700 1 -- 172.21.15.60:0/1510728425 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f833c0573d0 con 0x7f833c0b3d00 2022-06-29T16:04:02.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.879+0000 7f83337fe700 1 -- 172.21.15.60:0/1510728425 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v65) v1 ==== 81+0+28828 (secure 0 0 0) 0x7f8324015030 con 0x7f833c0b3d00 2022-06-29T16:04:02.887 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:04:02.887 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:04:02.887 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 65, 2022-06-29T16:04:02.887 INFO:teuthology.orchestra.run.smithi060.stdout: "fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", 2022-06-29T16:04:02.888 INFO:teuthology.orchestra.run.smithi060.stdout: "created": "2022-06-29T15:58:07.965719+0000", 2022-06-29T16:04:02.888 INFO:teuthology.orchestra.run.smithi060.stdout: "modified": "2022-06-29T16:04:00.142102+0000", 2022-06-29T16:04:02.888 INFO:teuthology.orchestra.run.smithi060.stdout: "last_up_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:04:02.888 INFO:teuthology.orchestra.run.smithi060.stdout: "last_in_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:04:02.889 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-06-29T16:04:02.889 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_num": 5799936, 2022-06-29T16:04:02.889 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_set": [ 2022-06-29T16:04:02.889 INFO:teuthology.orchestra.run.smithi060.stdout: "pglog_hardlimit", 2022-06-29T16:04:02.890 INFO:teuthology.orchestra.run.smithi060.stdout: "purged_snapdirs", 2022-06-29T16:04:02.890 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_deletes", 2022-06-29T16:04:02.890 INFO:teuthology.orchestra.run.smithi060.stdout: "sortbitwise" 2022-06-29T16:04:02.890 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:04:02.891 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_version": 4, 2022-06-29T16:04:02.891 INFO:teuthology.orchestra.run.smithi060.stdout: "full_ratio": 0.94999998807907104, 2022-06-29T16:04:02.891 INFO:teuthology.orchestra.run.smithi060.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-06-29T16:04:02.891 INFO:teuthology.orchestra.run.smithi060.stdout: "nearfull_ratio": 0.85000002384185791, 2022-06-29T16:04:02.891 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_snapshot": "", 2022-06-29T16:04:02.892 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_max": 12, 2022-06-29T16:04:02.892 INFO:teuthology.orchestra.run.smithi060.stdout: "max_osd": 8, 2022-06-29T16:04:02.892 INFO:teuthology.orchestra.run.smithi060.stdout: "require_min_compat_client": "jewel", 2022-06-29T16:04:02.892 INFO:teuthology.orchestra.run.smithi060.stdout: "min_compat_client": "jewel", 2022-06-29T16:04:02.893 INFO:teuthology.orchestra.run.smithi060.stdout: "require_osd_release": "octopus", 2022-06-29T16:04:02.893 INFO:teuthology.orchestra.run.smithi060.stdout: "pools": [ 2022-06-29T16:04:02.893 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:02.893 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2022-06-29T16:04:02.893 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "device_health_metrics", 2022-06-29T16:04:02.894 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:10.961127+0000", 2022-06-29T16:04:02.894 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:04:02.894 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:04:02.894 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:04:02.895 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:04:02.895 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:04:02.895 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:04:02.895 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:04:02.896 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:04:02.896 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 1, 2022-06-29T16:04:02.896 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 1, 2022-06-29T16:04:02.896 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 1, 2022-06-29T16:04:02.896 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 1, 2022-06-29T16:04:02.897 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 1, 2022-06-29T16:04:02.897 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:04:02.897 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:04:02.897 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:04:02.898 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:04:02.898 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:04:02.898 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:04:02.898 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:04:02.899 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:02.899 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "15", 2022-06-29T16:04:02.899 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:04:02.900 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:04:02.900 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:04:02.900 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:04:02.900 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:04:02.900 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:04:02.901 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:04:02.901 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:04:02.901 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:04:02.901 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:04:02.902 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:04:02.902 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:04:02.902 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:04:02.902 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:04:02.903 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:04:02.903 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:04:02.903 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:04:02.903 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:04:02.903 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:04:02.904 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:04:02.904 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:04:02.904 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:04:02.904 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:04:02.905 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:04:02.905 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:04:02.905 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:04:02.905 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:02.906 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:04:02.906 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:04:02.906 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:04:02.906 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:04:02.907 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:04:02.907 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:04:02.907 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:04:02.907 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:04:02.908 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:04:02.908 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:04:02.908 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:04:02.909 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:04:02.909 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 1 2022-06-29T16:04:02.909 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:02.909 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:04:02.910 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_devicehealth": {} 2022-06-29T16:04:02.910 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:02.910 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:02.910 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:02.911 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 2, 2022-06-29T16:04:02.911 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "rbd", 2022-06-29T16:04:02.911 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:19.184276+0000", 2022-06-29T16:04:02.911 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:04:02.912 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:04:02.912 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:04:02.912 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:04:02.913 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:04:02.913 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:04:02.913 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:04:02.913 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:04:02.914 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:04:02.914 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:04:02.915 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:04:02.915 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:04:02.915 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:04:02.916 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:04:02.916 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:04:02.916 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:04:02.916 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:04:02.917 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:04:02.917 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:04:02.917 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:04:02.917 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:02.918 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "18", 2022-06-29T16:04:02.918 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:04:02.918 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:04:02.919 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:04:02.919 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:04:02.919 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:04:02.920 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:04:02.920 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:04:02.921 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:04:02.921 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:04:02.922 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:04:02.922 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:04:02.923 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:04:02.923 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:04:02.924 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:04:02.924 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:04:02.924 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:04:02.925 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:04:02.925 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:04:02.926 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:04:02.926 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:04:02.927 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:04:02.927 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:04:02.928 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:04:02.928 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:04:02.928 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:04:02.929 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:04:02.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:02.930 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:04:02.930 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:04:02.930 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:04:02.931 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:04:02.931 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:04:02.932 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:04:02.932 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:04:02.932 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:04:02.933 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:04:02.933 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:04:02.933 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:04:02.934 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:04:02.934 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:04:02.934 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd": {} 2022-06-29T16:04:02.934 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:02.934 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:02.935 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:04:02.935 INFO:teuthology.orchestra.run.smithi060.stdout: "osds": [ 2022-06-29T16:04:02.935 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:02.935 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:04:02.936 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "11e7ab9e-fce2-470f-8195-ea4c3c52df22", 2022-06-29T16:04:02.936 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:04:02.936 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:04:02.936 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:04:02.937 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:04:02.937 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:04:02.937 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:04:02.937 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:04:02.937 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 59, 2022-06-29T16:04:02.938 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:04:02.938 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:04:02.938 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:04:02.938 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:04:02.944 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:02.944 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:04:02.945 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6816", 2022-06-29T16:04:02.945 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:04:02.945 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:02.946 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:02.946 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:04:02.946 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6825", 2022-06-29T16:04:02.946 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:04:02.947 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:02.947 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:02.947 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:02.947 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:04:02.947 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:04:02.948 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:02.948 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:04:02.948 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6826", 2022-06-29T16:04:02.948 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:04:02.949 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:02.949 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:02.949 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:04:02.949 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6827", 2022-06-29T16:04:02.949 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:04:02.950 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:02.950 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:02.950 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:02.951 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:04:02.951 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:04:02.951 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:02.951 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:04:02.951 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6830", 2022-06-29T16:04:02.952 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:04:02.952 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:02.952 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:02.953 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:04:02.953 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6831", 2022-06-29T16:04:02.953 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:04:02.954 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:02.954 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:02.954 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:02.954 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:04:02.955 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:04:02.955 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:02.955 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:04:02.955 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6828", 2022-06-29T16:04:02.956 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:04:02.956 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:02.956 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:02.956 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:04:02.957 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6829", 2022-06-29T16:04:02.957 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:04:02.957 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:02.957 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:02.958 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:02.958 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6825/12890", 2022-06-29T16:04:02.958 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6827/12890", 2022-06-29T16:04:02.959 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6831/12890", 2022-06-29T16:04:02.959 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6829/12890", 2022-06-29T16:04:02.959 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:04:02.959 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:04:02.960 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:04:02.960 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:02.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:02.960 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:02.960 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:04:02.961 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "ca43f136-6f69-4431-a157-fbe073fb585c", 2022-06-29T16:04:02.961 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:04:02.961 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:04:02.961 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:04:02.962 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:04:02.962 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:04:02.962 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:04:02.962 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:04:02.963 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 59, 2022-06-29T16:04:02.963 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:04:02.963 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:04:02.963 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:04:02.964 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:04:02.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:02.964 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:04:02.964 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6815", 2022-06-29T16:04:02.964 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:04:02.965 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:02.965 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:02.965 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:04:02.965 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6818", 2022-06-29T16:04:02.966 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:04:02.966 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:02.966 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:02.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:02.967 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:04:02.967 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:04:02.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:02.967 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:04:02.967 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6819", 2022-06-29T16:04:02.968 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:04:02.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:02.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:02.969 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:04:02.969 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6820", 2022-06-29T16:04:02.969 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:04:02.969 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:02.970 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:02.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:02.970 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:04:02.970 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:04:02.971 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:02.971 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:04:02.971 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6823", 2022-06-29T16:04:02.971 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:04:02.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:02.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:02.972 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:04:02.972 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6824", 2022-06-29T16:04:02.972 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:04:02.973 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:02.973 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:02.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:02.973 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:04:02.974 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:04:02.974 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:02.974 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:04:02.974 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6821", 2022-06-29T16:04:02.975 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:04:02.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:02.976 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:02.976 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:04:02.976 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6822", 2022-06-29T16:04:02.977 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:04:02.977 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:02.978 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:02.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:02.979 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6818/12892", 2022-06-29T16:04:02.979 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6820/12892", 2022-06-29T16:04:02.979 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6824/12892", 2022-06-29T16:04:02.980 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6822/12892", 2022-06-29T16:04:02.980 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:04:02.980 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:04:02.980 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:04:02.981 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:02.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:02.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:02.981 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:04:02.982 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "74fac45f-5f07-44ec-a9cb-2644c123fd4a", 2022-06-29T16:04:02.982 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:04:02.982 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:04:02.982 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:04:02.982 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:04:02.983 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:04:02.983 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:04:02.983 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:04:02.983 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 58, 2022-06-29T16:04:02.984 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:04:02.984 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:04:02.984 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:04:02.984 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:04:02.984 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:02.985 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:04:02.985 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6808", 2022-06-29T16:04:02.985 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:04:02.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:02.986 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:02.986 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:04:02.986 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6809", 2022-06-29T16:04:02.986 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:04:02.986 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:02.987 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:02.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:02.987 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:04:02.987 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:04:02.987 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:02.988 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:04:02.988 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6810", 2022-06-29T16:04:02.988 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:04:02.988 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:02.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:02.989 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:04:02.989 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6811", 2022-06-29T16:04:02.989 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:04:02.989 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:02.990 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:02.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:02.990 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:04:02.990 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:04:02.990 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:02.991 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:04:02.991 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6814", 2022-06-29T16:04:02.991 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:04:02.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:02.992 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:02.992 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:04:02.992 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6817", 2022-06-29T16:04:02.992 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:04:02.992 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:02.993 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:02.993 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:02.993 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:04:02.993 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:04:02.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:02.994 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:04:02.994 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6812", 2022-06-29T16:04:02.994 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:04:02.995 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:02.995 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:02.995 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:04:02.995 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6813", 2022-06-29T16:04:02.995 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:04:02.996 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:02.996 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:02.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:02.996 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6809/12891", 2022-06-29T16:04:02.997 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6811/12891", 2022-06-29T16:04:02.997 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6817/12891", 2022-06-29T16:04:02.997 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6813/12891", 2022-06-29T16:04:02.997 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:04:02.998 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:04:02.998 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:04:02.998 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:02.998 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:02.999 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:02.999 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:04:02.999 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "2581d351-973a-4b4e-85ca-3658f0068765", 2022-06-29T16:04:02.999 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:04:02.999 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:04:03.000 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:04:03.000 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:04:03.000 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:04:03.000 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:04:03.000 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:04:03.001 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 50, 2022-06-29T16:04:03.001 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:04:03.001 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:04:03.001 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:04:03.002 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:04:03.002 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.002 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:04:03.002 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6800", 2022-06-29T16:04:03.002 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:04:03.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.003 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.003 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:04:03.003 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6801", 2022-06-29T16:04:03.004 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:04:03.004 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:03.004 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:03.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.004 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:04:03.005 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:04:03.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.005 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:04:03.005 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6802", 2022-06-29T16:04:03.006 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:04:03.006 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.006 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.006 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:04:03.006 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6803", 2022-06-29T16:04:03.007 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:04:03.007 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:03.007 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:03.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.008 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:04:03.008 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:04:03.008 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.008 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:04:03.008 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6806", 2022-06-29T16:04:03.009 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:04:03.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.009 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.009 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:04:03.010 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6807", 2022-06-29T16:04:03.010 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:04:03.010 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:03.011 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:03.011 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.011 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:04:03.011 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:04:03.011 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.012 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:04:03.012 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6804", 2022-06-29T16:04:03.012 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:04:03.012 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.013 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.013 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:04:03.013 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6805", 2022-06-29T16:04:03.013 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:04:03.013 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:03.014 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:03.014 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.014 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6801/12889", 2022-06-29T16:04:03.014 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6803/12889", 2022-06-29T16:04:03.015 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6807/12889", 2022-06-29T16:04:03.015 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6805/12889", 2022-06-29T16:04:03.015 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:04:03.015 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:04:03.016 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:04:03.016 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:03.016 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.016 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.016 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:04:03.017 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "3d234db9-87cd-4fe9-814f-0240a287d8b4", 2022-06-29T16:04:03.017 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:04:03.017 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:04:03.017 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:04:03.018 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:04:03.018 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:04:03.018 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:04:03.018 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:04:03.019 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 59, 2022-06-29T16:04:03.019 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:04:03.019 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:04:03.019 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:04:03.019 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:04:03.020 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.020 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:04:03.020 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6808", 2022-06-29T16:04:03.021 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:04:03.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.022 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:04:03.022 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6810", 2022-06-29T16:04:03.022 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:04:03.023 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:03.023 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:03.023 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.023 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:04:03.023 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:04:03.024 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.024 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:04:03.024 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6812", 2022-06-29T16:04:03.024 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:04:03.025 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.025 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.025 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:04:03.026 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6814", 2022-06-29T16:04:03.026 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:04:03.027 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:03.027 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:03.027 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.028 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:04:03.028 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:04:03.029 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.029 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:04:03.029 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6823", 2022-06-29T16:04:03.029 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:04:03.030 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.030 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.030 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:04:03.031 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6825", 2022-06-29T16:04:03.031 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:04:03.032 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:03.032 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:03.032 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.033 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:04:03.033 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:04:03.034 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.034 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:04:03.034 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6817", 2022-06-29T16:04:03.035 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:04:03.035 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.036 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.036 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:04:03.036 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6820", 2022-06-29T16:04:03.037 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:04:03.037 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:03.038 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:03.038 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.038 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6810/13249", 2022-06-29T16:04:03.039 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6814/13249", 2022-06-29T16:04:03.039 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6825/13249", 2022-06-29T16:04:03.040 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6820/13249", 2022-06-29T16:04:03.040 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:04:03.040 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:04:03.040 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:04:03.041 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:03.041 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.041 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.041 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:04:03.041 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "e4bf1104-ff5c-438b-b608-8cee31c163f6", 2022-06-29T16:04:03.042 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:04:03.042 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:04:03.042 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:04:03.042 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:04:03.043 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:04:03.043 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:04:03.043 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:04:03.043 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 58, 2022-06-29T16:04:03.044 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:04:03.044 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:04:03.044 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:04:03.044 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:04:03.044 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.045 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:04:03.045 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6809", 2022-06-29T16:04:03.045 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:04:03.045 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.046 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.046 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:04:03.046 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6811", 2022-06-29T16:04:03.046 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:04:03.047 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:03.047 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:03.047 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.047 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:04:03.047 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:04:03.048 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.048 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:04:03.048 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6813", 2022-06-29T16:04:03.048 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:04:03.049 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.049 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.049 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:04:03.049 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6815", 2022-06-29T16:04:03.049 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:04:03.050 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:03.050 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:03.050 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.051 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:04:03.051 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:04:03.051 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.051 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:04:03.051 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6824", 2022-06-29T16:04:03.052 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:04:03.052 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.052 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.052 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:04:03.053 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6827", 2022-06-29T16:04:03.053 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:04:03.053 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:03.053 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:03.054 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.054 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:04:03.054 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:04:03.054 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.055 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:04:03.055 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6818", 2022-06-29T16:04:03.055 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:04:03.056 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.056 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.056 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:04:03.057 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6821", 2022-06-29T16:04:03.057 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:04:03.057 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:03.057 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:03.058 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.058 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6811/13248", 2022-06-29T16:04:03.058 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6815/13248", 2022-06-29T16:04:03.059 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6827/13248", 2022-06-29T16:04:03.059 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6821/13248", 2022-06-29T16:04:03.059 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:04:03.060 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:04:03.060 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:04:03.060 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:03.060 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.061 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.061 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:04:03.061 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "8564dd25-27ab-4203-b54e-af6d3aa05857", 2022-06-29T16:04:03.062 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:04:03.062 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:04:03.062 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:04:03.062 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:04:03.063 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:04:03.063 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:04:03.063 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:04:03.063 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 59, 2022-06-29T16:04:03.064 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:04:03.064 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:04:03.064 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:04:03.064 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:04:03.064 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.065 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:04:03.065 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6816", 2022-06-29T16:04:03.065 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:04:03.065 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.065 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.066 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:04:03.066 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6819", 2022-06-29T16:04:03.066 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:04:03.066 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:03.067 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:03.067 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.067 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:04:03.067 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:04:03.068 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.068 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:04:03.068 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6822", 2022-06-29T16:04:03.068 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:04:03.068 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.069 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.069 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:04:03.069 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6826", 2022-06-29T16:04:03.069 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:04:03.070 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:03.070 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:03.070 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.070 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:04:03.071 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:04:03.071 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.071 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:04:03.071 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6830", 2022-06-29T16:04:03.071 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:04:03.072 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.072 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.072 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:04:03.073 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6831", 2022-06-29T16:04:03.073 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:04:03.073 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:03.073 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:03.073 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.074 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:04:03.074 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:04:03.074 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.075 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:04:03.075 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6828", 2022-06-29T16:04:03.075 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:04:03.076 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.076 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.076 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:04:03.076 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6829", 2022-06-29T16:04:03.076 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:04:03.077 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:03.077 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:03.077 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.077 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6819/13250", 2022-06-29T16:04:03.078 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6826/13250", 2022-06-29T16:04:03.078 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6831/13250", 2022-06-29T16:04:03.078 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6829/13250", 2022-06-29T16:04:03.078 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:04:03.079 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:04:03.079 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:04:03.079 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:03.079 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.079 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.080 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:04:03.080 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "bd513a9c-3b19-4314-9897-d300b5c897bb", 2022-06-29T16:04:03.080 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:04:03.080 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:04:03.081 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:04:03.081 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:04:03.081 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:04:03.081 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:04:03.081 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:04:03.082 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 58, 2022-06-29T16:04:03.082 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:04:03.082 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:04:03.082 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:04:03.083 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:04:03.083 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.083 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:04:03.083 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6800", 2022-06-29T16:04:03.084 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:04:03.084 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.084 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.084 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:04:03.084 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6801", 2022-06-29T16:04:03.085 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:04:03.085 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:03.085 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:03.085 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.086 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:04:03.086 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:04:03.086 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.086 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:04:03.086 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6802", 2022-06-29T16:04:03.087 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:04:03.087 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.087 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.088 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:04:03.088 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6803", 2022-06-29T16:04:03.088 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:04:03.088 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:03.088 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:03.089 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.089 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:04:03.089 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:04:03.089 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.090 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:04:03.090 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6806", 2022-06-29T16:04:03.090 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:04:03.090 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.091 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.091 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:04:03.091 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6807", 2022-06-29T16:04:03.091 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:04:03.092 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:03.092 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:03.092 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.092 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:04:03.092 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:04:03.093 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.093 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:04:03.093 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6804", 2022-06-29T16:04:03.093 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:04:03.094 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.094 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.094 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:04:03.094 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6805", 2022-06-29T16:04:03.095 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:04:03.095 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:03.095 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:03.095 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.095 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6801/13247", 2022-06-29T16:04:03.096 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6803/13247", 2022-06-29T16:04:03.096 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6807/13247", 2022-06-29T16:04:03.096 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6805/13247", 2022-06-29T16:04:03.096 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:04:03.096 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:04:03.097 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:04:03.097 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:03.097 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:03.097 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:04:03.098 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_xinfo": [ 2022-06-29T16:04:03.098 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.098 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:04:03.098 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:04:03.099 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:04:03.099 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:04:03.099 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:04:03.099 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:04:03.100 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.891342+0000", 2022-06-29T16:04:03.100 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:04:03.100 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.100 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.101 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:04:03.101 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:04:03.101 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:04:03.101 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:04:03.101 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:04:03.102 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:04:03.102 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.900866+0000", 2022-06-29T16:04:03.102 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:04:03.102 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.103 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.103 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:04:03.103 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:04:03.104 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:04:03.104 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:04:03.104 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:04:03.105 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:04:03.105 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.912755+0000", 2022-06-29T16:04:03.106 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:04:03.106 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.107 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.107 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:04:03.107 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:04:03.108 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:04:03.108 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:04:03.109 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:04:03.109 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:04:03.109 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.864332+0000", 2022-06-29T16:04:03.110 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:04:03.110 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.110 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.111 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:04:03.111 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:04:03.111 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:04:03.112 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:04:03.112 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:04:03.112 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:04:03.112 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.863756+0000", 2022-06-29T16:04:03.113 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:04:03.113 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.113 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.113 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:04:03.113 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:04:03.114 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:04:03.114 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:04:03.114 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:04:03.114 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:04:03.115 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.869041+0000", 2022-06-29T16:04:03.115 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:04:03.115 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.115 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.115 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:04:03.116 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:04:03.116 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:04:03.116 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:04:03.116 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:04:03.116 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:04:03.117 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.895043+0000", 2022-06-29T16:04:03.117 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:04:03.117 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.117 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:03.118 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:04:03.118 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:04:03.118 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:04:03.118 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:04:03.118 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:04:03.119 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:04:03.119 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.889491+0000", 2022-06-29T16:04:03.119 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:04:03.119 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:03.120 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:04:03.120 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap": [], 2022-06-29T16:04:03.120 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap_items": [], 2022-06-29T16:04:03.120 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_temp": [], 2022-06-29T16:04:03.120 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_temp": [], 2022-06-29T16:04:03.121 INFO:teuthology.orchestra.run.smithi060.stdout: "blacklist": { 2022-06-29T16:04:03.121 INFO:teuthology.orchestra.run.smithi060.stdout: "172.21.15.92:6835/3876875590": "2022-06-30T16:03:56.730111+0000", 2022-06-29T16:04:03.121 INFO:teuthology.orchestra.run.smithi060.stdout: "172.21.15.92:6834/3876875590": "2022-06-30T16:03:56.730111+0000" 2022-06-29T16:04:03.121 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.122 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profiles": { 2022-06-29T16:04:03.122 INFO:teuthology.orchestra.run.smithi060.stdout: "default": { 2022-06-29T16:04:03.122 INFO:teuthology.orchestra.run.smithi060.stdout: "crush-failure-domain": "osd", 2022-06-29T16:04:03.122 INFO:teuthology.orchestra.run.smithi060.stdout: "k": "2", 2022-06-29T16:04:03.123 INFO:teuthology.orchestra.run.smithi060.stdout: "m": "1", 2022-06-29T16:04:03.123 INFO:teuthology.orchestra.run.smithi060.stdout: "plugin": "jerasure", 2022-06-29T16:04:03.123 INFO:teuthology.orchestra.run.smithi060.stdout: "ruleset-failure-domain": "osd", 2022-06-29T16:04:03.123 INFO:teuthology.orchestra.run.smithi060.stdout: "technique": "reed_sol_van" 2022-06-29T16:04:03.124 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:03.124 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:03.124 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps_queue": [], 2022-06-29T16:04:03.124 INFO:teuthology.orchestra.run.smithi060.stdout: "new_removed_snaps": [], 2022-06-29T16:04:03.124 INFO:teuthology.orchestra.run.smithi060.stdout: "new_purged_snaps": [], 2022-06-29T16:04:03.125 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_node_flags": {}, 2022-06-29T16:04:03.125 INFO:teuthology.orchestra.run.smithi060.stdout: "device_class_flags": {} 2022-06-29T16:04:03.125 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:04:03.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.883+0000 7f833b59e700 1 -- 172.21.15.60:0/1510728425 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8320053b00 msgr2=0x7f8320055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:03.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.883+0000 7f833b59e700 1 --2- 172.21.15.60:0/1510728425 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8320053b00 0x7f8320055fc0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f832c00c000 tx=0x7f832c008040).stop 2022-06-29T16:04:03.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.883+0000 7f833b59e700 1 -- 172.21.15.60:0/1510728425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f833c0b3d00 msgr2=0x7f833c151a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:03.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.883+0000 7f833b59e700 1 --2- 172.21.15.60:0/1510728425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f833c0b3d00 0x7f833c151a60 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0x7f83240032f0 tx=0x7f8324016000).stop 2022-06-29T16:04:03.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.883+0000 7f833b59e700 1 -- 172.21.15.60:0/1510728425 shutdown_connections 2022-06-29T16:04:03.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.883+0000 7f833b59e700 1 --2- 172.21.15.60:0/1510728425 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8320053b00 0x7f8320055fc0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:03.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.883+0000 7f833b59e700 1 --2- 172.21.15.60:0/1510728425 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f833c142df0 0x7f833c0669d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:03.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.883+0000 7f833b59e700 1 --2- 172.21.15.60:0/1510728425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f833c0b3d00 0x7f833c151a60 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:03.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.883+0000 7f833b59e700 1 --2- 172.21.15.60:0/1510728425 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f833c05bf30 0x7f833c0b07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:03.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.883+0000 7f833b59e700 1 -- 172.21.15.60:0/1510728425 >> 172.21.15.60:0/1510728425 conn(0x7f833c064810 msgr2=0x7f833c144190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:03.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.883+0000 7f833b59e700 1 -- 172.21.15.60:0/1510728425 shutdown_connections 2022-06-29T16:04:03.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:02.883+0000 7f833b59e700 1 -- 172.21.15.60:0/1510728425 wait complete. 2022-06-29T16:04:03.132 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:04:03.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.239+0000 7fcdcda80700 1 Processor -- start 2022-06-29T16:04:03.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.239+0000 7fcdcda80700 1 -- start start 2022-06-29T16:04:03.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.239+0000 7fcdcda80700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcdc80b4010 0x7fcdc81426b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:03.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.239+0000 7fcdcda80700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdc8143a40 con 0x7fcdc80b46c0 2022-06-29T16:04:03.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.239+0000 7fcdcda80700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdc80aee60 con 0x7fcdc80ae7c0 2022-06-29T16:04:03.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.239+0000 7fcdcda80700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc80af100 con 0x7fcdc80b4010 2022-06-29T16:04:03.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.239+0000 7fcdc7fff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcdc80b46c0 0x7fcdc80b4b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57138/0 (socket says 172.21.15.60:57138) 2022-06-29T16:04:03.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.239+0000 7fcdc7fff700 1 -- 172.21.15.60:0/889919184 learned_addr learned my addr 172.21.15.60:0/889919184 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:04:03.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.239+0000 7fcdc6ffd700 1 --2- 172.21.15.60:0/889919184 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcdc80b4010 0x7fcdc81426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:03.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.239+0000 7fcdc67fc700 1 -- 172.21.15.60:0/889919184 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1808347291 0 0) 0x7fcdc8143a40 con 0x7fcdc80b46c0 2022-06-29T16:04:03.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.239+0000 7fcdc67fc700 1 -- 172.21.15.60:0/889919184 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcda8003260 con 0x7fcdc80b46c0 2022-06-29T16:04:03.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.239+0000 7fcdc67fc700 1 -- 172.21.15.60:0/889919184 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2890024432 0 0) 0x7fcdc80aee60 con 0x7fcdc80ae7c0 2022-06-29T16:04:03.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.239+0000 7fcdc6ffd700 1 -- 172.21.15.60:0/889919184 >> v1:172.21.15.92:6789/0 conn(0x7fcdc80ae7c0 legacy=0x7fcdc8141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:03.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.239+0000 7fcdc6ffd700 1 -- 172.21.15.60:0/889919184 >> v1:172.21.15.60:6789/0 conn(0x7fcdc80b46c0 legacy=0x7fcdc80b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:03.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.239+0000 7fcdc6ffd700 1 -- 172.21.15.60:0/889919184 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc80aeba0 con 0x7fcdc80b4010 2022-06-29T16:04:03.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.239+0000 7fcdc6ffd700 1 --2- 172.21.15.60:0/889919184 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcdc80b4010 0x7fcdc81426b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fcdc00032f0 tx=0x7fcdc00091d0).ready entity=mon.2 client_cookie=47c597f0dbdf4d3c server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:03.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.239+0000 7fcdc67fc700 1 -- 172.21.15.60:0/889919184 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdc000b6c0 con 0x7fcdc80b4010 2022-06-29T16:04:03.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.239+0000 7fcdc67fc700 1 -- 172.21.15.60:0/889919184 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc000b820 con 0x7fcdc80b4010 2022-06-29T16:04:03.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.239+0000 7fcdc67fc700 1 -- 172.21.15.60:0/889919184 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdc000b990 con 0x7fcdc80b4010 2022-06-29T16:04:03.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.239+0000 7fcdcda80700 1 -- 172.21.15.60:0/889919184 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcdc80b4010 msgr2=0x7fcdc81426b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:03.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.239+0000 7fcdcda80700 1 --2- 172.21.15.60:0/889919184 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcdc80b4010 0x7fcdc81426b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fcdc00032f0 tx=0x7fcdc00091d0).stop 2022-06-29T16:04:03.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.239+0000 7fcdcda80700 1 -- 172.21.15.60:0/889919184 shutdown_connections 2022-06-29T16:04:03.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.239+0000 7fcdcda80700 1 --2- 172.21.15.60:0/889919184 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcdc80b4010 0x7fcdc81426b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:03.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.239+0000 7fcdcda80700 1 -- 172.21.15.60:0/889919184 >> 172.21.15.60:0/889919184 conn(0x7fcdc80b9340 msgr2=0x7fcdc80b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:03.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.239+0000 7fcdcda80700 1 -- 172.21.15.60:0/889919184 shutdown_connections 2022-06-29T16:04:03.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.243+0000 7fcdcda80700 1 -- 172.21.15.60:0/889919184 wait complete. 2022-06-29T16:04:03.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.243+0000 7fcdcda80700 1 Processor -- start 2022-06-29T16:04:03.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.243+0000 7fcdcda80700 1 -- start start 2022-06-29T16:04:03.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.243+0000 7fcdcda80700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcdc80ae7c0 0x7fcdc80b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:03.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.243+0000 7fcdcda80700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdc80b4010 0x7fcdc80b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:03.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.243+0000 7fcdcda80700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcdc80b46c0 0x7fcdc8151e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:03.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.243+0000 7fcdcda80700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc8145c90 con 0x7fcdc80ae7c0 2022-06-29T16:04:03.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.243+0000 7fcdcda80700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc8145f30 con 0x7fcdc80b4010 2022-06-29T16:04:03.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.243+0000 7fcdcda80700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc80af5c0 con 0x7fcdc80b46c0 2022-06-29T16:04:03.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.243+0000 7fcdc77fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcdc80ae7c0 0x7fcdc80b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:03.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.243+0000 7fcdc77fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcdc80ae7c0 0x7fcdc80b0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49054/0 (socket says 172.21.15.60:49054) 2022-06-29T16:04:03.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.243+0000 7fcdc77fe700 1 -- 172.21.15.60:0/949622673 learned_addr learned my addr 172.21.15.60:0/949622673 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:03.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.243+0000 7fcdc7fff700 1 --2- 172.21.15.60:0/949622673 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcdc80b46c0 0x7fcdc8151e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:03.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.243+0000 7fcdc6ffd700 1 --2- 172.21.15.60:0/949622673 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdc80b4010 0x7fcdc80b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:03.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.243+0000 7fcdc7fff700 1 -- 172.21.15.60:0/949622673 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdc80b4010 msgr2=0x7fcdc80b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:03.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.243+0000 7fcdc7fff700 1 --2- 172.21.15.60:0/949622673 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdc80b4010 0x7fcdc80b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:03.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.243+0000 7fcdc7fff700 1 -- 172.21.15.60:0/949622673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcdc80ae7c0 msgr2=0x7fcdc80b0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:03.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.243+0000 7fcdc7fff700 1 --2- 172.21.15.60:0/949622673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcdc80ae7c0 0x7fcdc80b0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:03.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.243+0000 7fcdc7fff700 1 -- 172.21.15.60:0/949622673 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc0009470 con 0x7fcdc80b46c0 2022-06-29T16:04:03.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.243+0000 7fcdc77fe700 1 --2- 172.21.15.60:0/949622673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcdc80ae7c0 0x7fcdc80b0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:04:03.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.247+0000 7fcdc7fff700 1 --2- 172.21.15.60:0/949622673 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcdc80b46c0 0x7fcdc8151e40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fcdb8001110 tx=0x7fcdb80079c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:03.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.247+0000 7fcdc4ff9700 1 -- 172.21.15.60:0/949622673 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdb800cd00 con 0x7fcdc80b46c0 2022-06-29T16:04:03.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.247+0000 7fcdcda80700 1 -- 172.21.15.60:0/949622673 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdc8146a90 con 0x7fcdc80b46c0 2022-06-29T16:04:03.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.247+0000 7fcdc4ff9700 1 -- 172.21.15.60:0/949622673 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdb800ce60 con 0x7fcdc80b46c0 2022-06-29T16:04:03.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.247+0000 7fcdc4ff9700 1 -- 172.21.15.60:0/949622673 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdb80072c0 con 0x7fcdc80b46c0 2022-06-29T16:04:03.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.247+0000 7fcdcda80700 1 -- 172.21.15.60:0/949622673 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdc80b0090 con 0x7fcdc80b46c0 2022-06-29T16:04:03.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.247+0000 7fcdcda80700 1 -- 172.21.15.60:0/949622673 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdc81460a0 con 0x7fcdc80b46c0 2022-06-29T16:04:03.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.247+0000 7fcdc4ff9700 1 -- 172.21.15.60:0/949622673 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcdb8010070 con 0x7fcdc80b46c0 2022-06-29T16:04:03.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.247+0000 7fcdc4ff9700 1 --2- 172.21.15.60:0/949622673 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcdac053980 0x7fcdac055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:03.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.247+0000 7fcdc4ff9700 1 -- 172.21.15.60:0/949622673 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fcdb806b0d0 con 0x7fcdc80b46c0 2022-06-29T16:04:03.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.247+0000 7fcdc77fe700 1 --2- 172.21.15.60:0/949622673 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcdac053980 0x7fcdac055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:03.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.251+0000 7fcdc77fe700 1 --2- 172.21.15.60:0/949622673 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcdac053980 0x7fcdac055e40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fcdb000d000 tx=0x7fcdb0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:03.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.259+0000 7fcdc4ff9700 1 -- 172.21.15.60:0/949622673 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcdb8042230 con 0x7fcdc80b46c0 2022-06-29T16:04:03.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.423+0000 7fcdcda80700 1 -- 172.21.15.60:0/949622673 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcdc81467a0 con 0x7fcdc80b46c0 2022-06-29T16:04:03.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.423+0000 7fcdc4ff9700 1 -- 172.21.15.60:0/949622673 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+1200 (secure 0 0 0) 0x7fcdb8045850 con 0x7fcdc80b46c0 2022-06-29T16:04:03.431 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:04:03.431 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":27,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5955,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.60:6833/66690512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":66690512},{"type":"v1","addr":"172.21.15.60:6833","nonce":66690512}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":23},{"gid":6235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/336230151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":336230151},{"type":"v1","addr":"172.21.15.92:6835","nonce":336230151}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":27}],"filesystems":[]} 2022-06-29T16:04:03.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.427+0000 7fcdcda80700 1 -- 172.21.15.60:0/949622673 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcdac053980 msgr2=0x7fcdac055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:03.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.427+0000 7fcdcda80700 1 --2- 172.21.15.60:0/949622673 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcdac053980 0x7fcdac055e40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fcdb000d000 tx=0x7fcdb0009040).stop 2022-06-29T16:04:03.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.427+0000 7fcdcda80700 1 -- 172.21.15.60:0/949622673 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcdc80b46c0 msgr2=0x7fcdc8151e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:03.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.427+0000 7fcdcda80700 1 --2- 172.21.15.60:0/949622673 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcdc80b46c0 0x7fcdc8151e40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fcdb8001110 tx=0x7fcdb80079c0).stop 2022-06-29T16:04:03.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.427+0000 7fcdcda80700 1 -- 172.21.15.60:0/949622673 shutdown_connections 2022-06-29T16:04:03.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.427+0000 7fcdcda80700 1 --2- 172.21.15.60:0/949622673 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcdac053980 0x7fcdac055e40 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:03.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.427+0000 7fcdcda80700 1 --2- 172.21.15.60:0/949622673 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcdc80b46c0 0x7fcdc8151e40 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:03.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.427+0000 7fcdcda80700 1 --2- 172.21.15.60:0/949622673 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdc80b4010 0x7fcdc80b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:03.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.427+0000 7fcdcda80700 1 --2- 172.21.15.60:0/949622673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcdc80ae7c0 0x7fcdc80b0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:03.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.427+0000 7fcdcda80700 1 -- 172.21.15.60:0/949622673 >> 172.21.15.60:0/949622673 conn(0x7fcdc80b9340 msgr2=0x7fcdc80b82e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:03.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.427+0000 7fcdcda80700 1 -- 172.21.15.60:0/949622673 shutdown_connections 2022-06-29T16:04:03.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.427+0000 7fcdcda80700 1 -- 172.21.15.60:0/949622673 wait complete. 2022-06-29T16:04:03.436 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 27 2022-06-29T16:04:03.444 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:04:03.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.543+0000 7f1eaed7a700 1 Processor -- start 2022-06-29T16:04:03.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.543+0000 7f1eaed7a700 1 -- start start 2022-06-29T16:04:03.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.543+0000 7f1eaed7a700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1ea805d070 0x7f1ea814aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:03.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.543+0000 7f1eaed7a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ea814af80 0x7f1ea80ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:03.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.543+0000 7f1eaed7a700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea805d450 con 0x7f1ea80b2130 2022-06-29T16:04:03.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.543+0000 7f1eaed7a700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ea80adf90 con 0x7f1ea805d070 2022-06-29T16:04:03.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.543+0000 7f1eaed7a700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea80b25d0 con 0x7f1ea814af80 2022-06-29T16:04:03.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.543+0000 7f1ead577700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f1ea80b2130 0x7f1ea80aca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57148/0 (socket says 172.21.15.60:57148) 2022-06-29T16:04:03.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.543+0000 7f1ead577700 1 -- 172.21.15.60:0/1007643302 learned_addr learned my addr 172.21.15.60:0/1007643302 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:04:03.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eae579700 1 --2- 172.21.15.60:0/1007643302 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ea814af80 0x7f1ea80ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:03.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.543+0000 7f1eadd78700 1 --2- 172.21.15.60:0/1007643302 >> v2:172.21.15.92:3300/0 conn(0x7f1ea805d070 0x7f1ea814aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:03.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eacd76700 1 -- 172.21.15.60:0/1007643302 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2715893697 0 0) 0x7f1ea805d450 con 0x7f1ea80b2130 2022-06-29T16:04:03.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eacd76700 1 -- 172.21.15.60:0/1007643302 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e88003260 con 0x7f1ea80b2130 2022-06-29T16:04:03.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eacd76700 1 -- 172.21.15.60:0/1007643302 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1255567951 0 0) 0x7f1e88003260 con 0x7f1ea80b2130 2022-06-29T16:04:03.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eacd76700 1 -- 172.21.15.60:0/1007643302 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ea814af80 msgr2=0x7f1ea80ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:03.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eacd76700 1 --2- 172.21.15.60:0/1007643302 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ea814af80 0x7f1ea80ad130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:03.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eacd76700 1 -- 172.21.15.60:0/1007643302 >> v2:172.21.15.92:3300/0 conn(0x7f1ea805d070 msgr2=0x7f1ea814aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:03.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eacd76700 1 --2- 172.21.15.60:0/1007643302 >> v2:172.21.15.92:3300/0 conn(0x7f1ea805d070 0x7f1ea814aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:03.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eacd76700 1 -- 172.21.15.60:0/1007643302 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea814a350 con 0x7f1ea80b2130 2022-06-29T16:04:03.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eacd76700 1 -- 172.21.15.60:0/1007643302 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f1e980019e0 con 0x7f1ea80b2130 2022-06-29T16:04:03.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eacd76700 1 -- 172.21.15.60:0/1007643302 >> v1:172.21.15.60:6789/0 conn(0x7f1ea80b2130 legacy=0x7f1ea80aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:03.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eacd76700 1 --2- 172.21.15.60:0/1007643302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e88003750 0x7f1e88003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:03.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eacd76700 1 --2- 172.21.15.60:0/1007643302 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e88004500 0x7f1e88004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:03.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eacd76700 1 --2- 172.21.15.60:0/1007643302 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ea814af80 0x7f1e88004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:03.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eacd76700 1 -- 172.21.15.60:0/1007643302 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea80adf90 con 0x7f1e88003750 2022-06-29T16:04:03.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eacd76700 1 -- 172.21.15.60:0/1007643302 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea80b25d0 con 0x7f1e88004500 2022-06-29T16:04:03.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eacd76700 1 -- 172.21.15.60:0/1007643302 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8800b750 con 0x7f1ea814af80 2022-06-29T16:04:03.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eae579700 1 --2- 172.21.15.60:0/1007643302 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ea814af80 0x7f1e88004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:03.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1ead577700 1 --2- 172.21.15.60:0/1007643302 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e88004500 0x7f1e88004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:03.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eadd78700 1 --2- 172.21.15.60:0/1007643302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e88003750 0x7f1e88003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:03.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eae579700 1 -- 172.21.15.60:0/1007643302 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e88004500 msgr2=0x7f1e88004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:03.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eae579700 1 --2- 172.21.15.60:0/1007643302 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e88004500 0x7f1e88004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:03.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eae579700 1 -- 172.21.15.60:0/1007643302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e88003750 msgr2=0x7f1e88003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:03.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eae579700 1 --2- 172.21.15.60:0/1007643302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e88003750 0x7f1e88003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:03.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eae579700 1 -- 172.21.15.60:0/1007643302 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e88003260 con 0x7f1ea814af80 2022-06-29T16:04:03.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eadd78700 1 --2- 172.21.15.60:0/1007643302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1e88003750 0x7f1e88003b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:04:03.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eae579700 1 --2- 172.21.15.60:0/1007643302 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ea814af80 0x7f1e88004e70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f1ea000c000 tx=0x7f1ea0006df0).ready entity=mon.2 client_cookie=d89ce186e4c6c1f server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:03.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eacd76700 1 -- 172.21.15.60:0/1007643302 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ea000e800 con 0x7f1ea814af80 2022-06-29T16:04:03.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eacd76700 1 -- 172.21.15.60:0/1007643302 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea000e960 con 0x7f1ea814af80 2022-06-29T16:04:03.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eacd76700 1 -- 172.21.15.60:0/1007643302 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ea000ecc0 con 0x7f1ea814af80 2022-06-29T16:04:03.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eaed7a700 1 -- 172.21.15.60:0/1007643302 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ea814af80 msgr2=0x7f1e88004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:03.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eaed7a700 1 --2- 172.21.15.60:0/1007643302 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ea814af80 0x7f1e88004e70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f1ea000c000 tx=0x7f1ea0006df0).stop 2022-06-29T16:04:03.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eae579700 1 -- 172.21.15.60:0/1007643302 reap_dead start 2022-06-29T16:04:03.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eaed7a700 1 -- 172.21.15.60:0/1007643302 shutdown_connections 2022-06-29T16:04:03.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eaed7a700 1 -- 172.21.15.60:0/1007643302 >> 172.21.15.60:0/1007643302 conn(0x7f1ea80b5890 msgr2=0x7f1ea80b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:03.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eaed7a700 1 -- 172.21.15.60:0/1007643302 shutdown_connections 2022-06-29T16:04:03.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.547+0000 7f1eaed7a700 1 -- 172.21.15.60:0/1007643302 wait complete. 2022-06-29T16:04:03.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.551+0000 7f1eaed7a700 1 Processor -- start 2022-06-29T16:04:03.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.551+0000 7f1eaed7a700 1 -- start start 2022-06-29T16:04:03.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.551+0000 7f1eaed7a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ea8066a50 0x7f1ea8156d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:03.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.551+0000 7f1eaed7a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ea8065430 0x7f1ea80630c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:03.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.551+0000 7f1eaed7a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ea8065890 0x7f1ea80635b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:03.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.551+0000 7f1eaed7a700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea8117210 con 0x7f1ea8065890 2022-06-29T16:04:03.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.551+0000 7f1eaed7a700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea81177a0 con 0x7f1ea8066a50 2022-06-29T16:04:03.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.551+0000 7f1eaed7a700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea8066e30 con 0x7f1ea8065430 2022-06-29T16:04:03.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.551+0000 7f1eae579700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ea8065890 0x7f1ea80635b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:03.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.551+0000 7f1eadd78700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ea8066a50 0x7f1ea8156d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:03.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.551+0000 7f1eadd78700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ea8066a50 0x7f1ea8156d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:55104/0 (socket says 172.21.15.60:55104) 2022-06-29T16:04:03.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.551+0000 7f1eadd78700 1 -- 172.21.15.60:0/2335067545 learned_addr learned my addr 172.21.15.60:0/2335067545 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:03.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.551+0000 7f1ead577700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ea8065430 0x7f1ea80630c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:03.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.551+0000 7f1eadd78700 1 -- 172.21.15.60:0/2335067545 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ea8065430 msgr2=0x7f1ea80630c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:03.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.551+0000 7f1eadd78700 1 --2- 172.21.15.60:0/2335067545 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ea8065430 0x7f1ea80630c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:03.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.551+0000 7f1eadd78700 1 -- 172.21.15.60:0/2335067545 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ea8065890 msgr2=0x7f1ea80635b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:03.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.551+0000 7f1eadd78700 1 --2- 172.21.15.60:0/2335067545 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ea8065890 0x7f1ea80635b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:03.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.551+0000 7f1eadd78700 1 -- 172.21.15.60:0/2335067545 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea8063aa0 con 0x7f1ea8066a50 2022-06-29T16:04:03.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.551+0000 7f1eae579700 1 --2- 172.21.15.60:0/2335067545 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ea8065890 0x7f1ea80635b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:04:03.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.551+0000 7f1eadd78700 1 --2- 172.21.15.60:0/2335067545 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ea8066a50 0x7f1ea8156d60 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f1ea8066fc0 tx=0x7f1e90006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:03.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.555+0000 7f1ea6ffd700 1 -- 172.21.15.60:0/2335067545 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e90011eb0 con 0x7f1ea8066a50 2022-06-29T16:04:03.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.555+0000 7f1ea6ffd700 1 -- 172.21.15.60:0/2335067545 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e9000f940 con 0x7f1ea8066a50 2022-06-29T16:04:03.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.555+0000 7f1eaed7a700 1 -- 172.21.15.60:0/2335067545 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ea8156380 con 0x7f1ea8066a50 2022-06-29T16:04:03.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.555+0000 7f1ea6ffd700 1 -- 172.21.15.60:0/2335067545 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e9000fab0 con 0x7f1ea8066a50 2022-06-29T16:04:03.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.555+0000 7f1eaed7a700 1 -- 172.21.15.60:0/2335067545 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ea81573a0 con 0x7f1ea8066a50 2022-06-29T16:04:03.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.555+0000 7f1eaed7a700 1 -- 172.21.15.60:0/2335067545 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ea814a5f0 con 0x7f1ea8066a50 2022-06-29T16:04:03.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.555+0000 7f1ea6ffd700 1 -- 172.21.15.60:0/2335067545 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1e90024070 con 0x7f1ea8066a50 2022-06-29T16:04:03.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.555+0000 7f1ea6ffd700 1 --2- 172.21.15.60:0/2335067545 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1e8c053a70 0x7f1e8c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:03.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.555+0000 7f1ea6ffd700 1 -- 172.21.15.60:0/2335067545 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f1e9006c4b0 con 0x7f1ea8066a50 2022-06-29T16:04:03.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.555+0000 7f1ead577700 1 --2- 172.21.15.60:0/2335067545 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1e8c053a70 0x7f1e8c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:03.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.559+0000 7f1ead577700 1 --2- 172.21.15.60:0/2335067545 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1e8c053a70 0x7f1e8c055f30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f1e98005000 tx=0x7f1e980021d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:03.586 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.567+0000 7f1ea6ffd700 1 -- 172.21.15.60:0/2335067545 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1e900436b0 con 0x7f1ea8066a50 2022-06-29T16:04:03.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.731+0000 7f1eaed7a700 1 -- 172.21.15.60:0/2335067545 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ea8156510 con 0x7f1ea8066a50 2022-06-29T16:04:03.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.735+0000 7f1ea6ffd700 1 -- 172.21.15.60:0/2335067545 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+1200 (secure 0 0 0) 0x7f1e90046cd0 con 0x7f1ea8066a50 2022-06-29T16:04:03.740 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:04:03.741 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":27,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5955,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.60:6833/66690512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":66690512},{"type":"v1","addr":"172.21.15.60:6833","nonce":66690512}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":23},{"gid":6235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/336230151","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":336230151},{"type":"v1","addr":"172.21.15.92:6835","nonce":336230151}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":27}],"filesystems":[]} 2022-06-29T16:04:03.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.735+0000 7f1eaed7a700 1 -- 172.21.15.60:0/2335067545 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1e8c053a70 msgr2=0x7f1e8c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:03.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.735+0000 7f1eaed7a700 1 --2- 172.21.15.60:0/2335067545 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1e8c053a70 0x7f1e8c055f30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f1e98005000 tx=0x7f1e980021d0).stop 2022-06-29T16:04:03.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.735+0000 7f1eaed7a700 1 -- 172.21.15.60:0/2335067545 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ea8066a50 msgr2=0x7f1ea8156d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:03.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.735+0000 7f1eaed7a700 1 --2- 172.21.15.60:0/2335067545 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ea8066a50 0x7f1ea8156d60 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f1ea8066fc0 tx=0x7f1e90006df0).stop 2022-06-29T16:04:03.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.735+0000 7f1eaed7a700 1 -- 172.21.15.60:0/2335067545 shutdown_connections 2022-06-29T16:04:03.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.735+0000 7f1eaed7a700 1 --2- 172.21.15.60:0/2335067545 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1e8c053a70 0x7f1e8c055f30 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:03.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.735+0000 7f1eaed7a700 1 --2- 172.21.15.60:0/2335067545 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ea8065890 0x7f1ea80635b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:03.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.735+0000 7f1eaed7a700 1 --2- 172.21.15.60:0/2335067545 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ea8065430 0x7f1ea80630c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:03.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.735+0000 7f1eaed7a700 1 --2- 172.21.15.60:0/2335067545 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ea8066a50 0x7f1ea8156d60 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:03.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.735+0000 7f1eaed7a700 1 -- 172.21.15.60:0/2335067545 >> 172.21.15.60:0/2335067545 conn(0x7f1ea80b5890 msgr2=0x7f1ea80ad210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:03.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.735+0000 7f1eaed7a700 1 -- 172.21.15.60:0/2335067545 shutdown_connections 2022-06-29T16:04:03.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.735+0000 7f1eaed7a700 1 -- 172.21.15.60:0/2335067545 wait complete. 2022-06-29T16:04:03.746 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 27 2022-06-29T16:04:03.759 INFO:tasks.ceph.mds.a:Restarting daemon 2022-06-29T16:04:03.759 INFO:tasks.ceph.mds.a:Stopping old one... 2022-06-29T16:04:03.759 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-06-29T16:04:03.760 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T16:04:03.760 INFO:tasks.ceph.mds.b:Restarting daemon 2022-06-29T16:04:03.761 INFO:tasks.ceph.mds.b:Stopping old one... 2022-06-29T16:04:03.762 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-06-29T16:04:03.762 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T16:04:03.775 INFO:tasks.ceph.mds.a:Stopped 2022-06-29T16:04:03.775 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-06-29T16:04:03.778 INFO:tasks.ceph.mds.b:Stopped 2022-06-29T16:04:03.778 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-06-29T16:04:03.817 INFO:tasks.ceph.mds.a:Started 2022-06-29T16:04:03.818 DEBUG:teuthology.parallel:result is None 2022-06-29T16:04:03.821 INFO:tasks.ceph.mds.b:Started 2022-06-29T16:04:03.821 DEBUG:teuthology.parallel:result is None 2022-06-29T16:04:03.822 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-06-29T16:04:03.900 INFO:tasks.ceph.mds.b.smithi092.stdout:starting mds.b at 2022-06-29T16:04:03.906 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2022-06-29T16:04:03.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.923+0000 7f0f73fff700 1 Processor -- start 2022-06-29T16:04:03.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.923+0000 7f0f73fff700 1 -- start start 2022-06-29T16:04:03.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.927+0000 7f0f73fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0f7405fb00 0x7f0f7413f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:03.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.927+0000 7f0f73fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0f74067a90 0x7f0f7413fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:03.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.927+0000 7f0f73fff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f74064e20 con 0x7f0f740aec30 2022-06-29T16:04:03.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.927+0000 7f0f73fff700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f740b10e0 con 0x7f0f7405fb00 2022-06-29T16:04:03.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.927+0000 7f0f73fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7405fee0 con 0x7f0f74067a90 2022-06-29T16:04:03.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.927+0000 7f0f727fc700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0f74067a90 0x7f0f7413fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:03.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.927+0000 7f0f727fc700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0f74067a90 0x7f0f7413fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:33960/0 (socket says 172.21.15.60:33960) 2022-06-29T16:04:03.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.927+0000 7f0f727fc700 1 -- 172.21.15.60:0/3951147622 learned_addr learned my addr 172.21.15.60:0/3951147622 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:03.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.927+0000 7f0f72ffd700 1 --2- 172.21.15.60:0/3951147622 >> v2:172.21.15.92:3300/0 conn(0x7f0f7405fb00 0x7f0f7413f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:03.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.927+0000 7f0f727fc700 1 -- 172.21.15.60:0/3951147622 >> v2:172.21.15.92:3300/0 conn(0x7f0f7405fb00 msgr2=0x7f0f7413f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:03.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.927+0000 7f0f727fc700 1 --2- 172.21.15.60:0/3951147622 >> v2:172.21.15.92:3300/0 conn(0x7f0f7405fb00 0x7f0f7413f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:03.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.927+0000 7f0f727fc700 1 -- 172.21.15.60:0/3951147622 >> v1:172.21.15.92:6789/0 conn(0x7f0f740aec30 legacy=0x7f0f740b2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:03.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.927+0000 7f0f727fc700 1 -- 172.21.15.60:0/3951147622 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f74060b20 con 0x7f0f74067a90 2022-06-29T16:04:03.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.927+0000 7f0f71ffb700 1 -- 172.21.15.60:0/3951147622 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 368345530 0 0) 0x7f0f74064e20 con 0x7f0f740aec30 2022-06-29T16:04:03.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.927+0000 7f0f727fc700 1 --2- 172.21.15.60:0/3951147622 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0f74067a90 0x7f0f7413fe90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f0f640032f0 tx=0x7f0f64009290).ready entity=mon.2 client_cookie=b81a30cd9e8e0def server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:03.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.927+0000 7f0f71ffb700 1 -- 172.21.15.60:0/3951147622 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f6400b8a0 con 0x7f0f74067a90 2022-06-29T16:04:03.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.927+0000 7f0f71ffb700 1 -- 172.21.15.60:0/3951147622 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f6400ba00 con 0x7f0f74067a90 2022-06-29T16:04:03.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.927+0000 7f0f71ffb700 1 -- 172.21.15.60:0/3951147622 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f6400bb70 con 0x7f0f74067a90 2022-06-29T16:04:03.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.927+0000 7f0f73fff700 1 -- 172.21.15.60:0/3951147622 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0f74067a90 msgr2=0x7f0f7413fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:03.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.927+0000 7f0f73fff700 1 --2- 172.21.15.60:0/3951147622 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0f74067a90 0x7f0f7413fe90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f0f640032f0 tx=0x7f0f64009290).stop 2022-06-29T16:04:03.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.927+0000 7f0f73fff700 1 -- 172.21.15.60:0/3951147622 shutdown_connections 2022-06-29T16:04:03.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.927+0000 7f0f73fff700 1 --2- 172.21.15.60:0/3951147622 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0f74067a90 0x7f0f7413fe90 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:03.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.927+0000 7f0f73fff700 1 --2- 172.21.15.60:0/3951147622 >> v2:172.21.15.92:3300/0 conn(0x7f0f7405fb00 0x7f0f7413f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:03.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.927+0000 7f0f73fff700 1 -- 172.21.15.60:0/3951147622 >> 172.21.15.60:0/3951147622 conn(0x7f0f74063a60 msgr2=0x7f0f7405f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:03.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.927+0000 7f0f73fff700 1 -- 172.21.15.60:0/3951147622 shutdown_connections 2022-06-29T16:04:03.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.927+0000 7f0f73fff700 1 -- 172.21.15.60:0/3951147622 wait complete. 2022-06-29T16:04:03.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.931+0000 7f0f73fff700 1 Processor -- start 2022-06-29T16:04:03.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.931+0000 7f0f73fff700 1 -- start start 2022-06-29T16:04:03.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.931+0000 7f0f73fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f7405fb00 0x7f0f7413a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:03.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.931+0000 7f0f73fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f74067a90 0x7f0f74153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:03.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.931+0000 7f0f73fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0f740aec30 0x7f0f74135a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:03.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.931+0000 7f0f73fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f74066eb0 con 0x7f0f74067a90 2022-06-29T16:04:03.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.931+0000 7f0f73fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f74065ed0 con 0x7f0f7405fb00 2022-06-29T16:04:03.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.931+0000 7f0f73fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f74142e20 con 0x7f0f740aec30 2022-06-29T16:04:03.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.931+0000 7f0f727fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f74067a90 0x7f0f74153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:03.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.931+0000 7f0f72ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f7405fb00 0x7f0f7413a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:03.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.931+0000 7f0f727fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f74067a90 0x7f0f74153090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49102/0 (socket says 172.21.15.60:49102) 2022-06-29T16:04:03.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.931+0000 7f0f727fc700 1 -- 172.21.15.60:0/406074759 learned_addr learned my addr 172.21.15.60:0/406074759 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:03.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.931+0000 7f0f737fe700 1 --2- 172.21.15.60:0/406074759 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0f740aec30 0x7f0f74135a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:03.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.931+0000 7f0f727fc700 1 -- 172.21.15.60:0/406074759 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0f740aec30 msgr2=0x7f0f74135a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:03.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.931+0000 7f0f727fc700 1 --2- 172.21.15.60:0/406074759 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0f740aec30 0x7f0f74135a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:03.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.931+0000 7f0f727fc700 1 -- 172.21.15.60:0/406074759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f7405fb00 msgr2=0x7f0f7413a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:03.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.931+0000 7f0f727fc700 1 --2- 172.21.15.60:0/406074759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f7405fb00 0x7f0f7413a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:03.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.931+0000 7f0f727fc700 1 -- 172.21.15.60:0/406074759 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f64009530 con 0x7f0f74067a90 2022-06-29T16:04:03.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.931+0000 7f0f727fc700 1 --2- 172.21.15.60:0/406074759 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f74067a90 0x7f0f74153090 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0x7f0f640032f0 tx=0x7f0f64016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:03.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.935+0000 7f0f6bfff700 1 -- 172.21.15.60:0/406074759 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f6400bb70 con 0x7f0f74067a90 2022-06-29T16:04:03.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.935+0000 7f0f6bfff700 1 -- 172.21.15.60:0/406074759 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f64021630 con 0x7f0f74067a90 2022-06-29T16:04:03.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.935+0000 7f0f6bfff700 1 -- 172.21.15.60:0/406074759 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f6400b8a0 con 0x7f0f74067a90 2022-06-29T16:04:03.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.935+0000 7f0f73fff700 1 -- 172.21.15.60:0/406074759 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f7413ae10 con 0x7f0f74067a90 2022-06-29T16:04:03.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.935+0000 7f0f73fff700 1 -- 172.21.15.60:0/406074759 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f74065890 con 0x7f0f74067a90 2022-06-29T16:04:03.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.935+0000 7f0f73fff700 1 -- 172.21.15.60:0/406074759 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f74142c70 con 0x7f0f74067a90 2022-06-29T16:04:03.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.935+0000 7f0f6bfff700 1 -- 172.21.15.60:0/406074759 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0f64027020 con 0x7f0f74067a90 2022-06-29T16:04:03.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.939+0000 7f0f6bfff700 1 --2- 172.21.15.60:0/406074759 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0f580539e0 0x7f0f58055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:03.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.939+0000 7f0f6bfff700 1 -- 172.21.15.60:0/406074759 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f0f6406d240 con 0x7f0f74067a90 2022-06-29T16:04:03.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.939+0000 7f0f72ffd700 1 --2- 172.21.15.60:0/406074759 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0f580539e0 0x7f0f58055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:03.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.939+0000 7f0f72ffd700 1 --2- 172.21.15.60:0/406074759 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0f580539e0 0x7f0f58055ea0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f0f5c00b000 tx=0x7f0f5c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:03.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:03.943+0000 7f0f6bfff700 1 -- 172.21.15.60:0/406074759 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0f640443a0 con 0x7f0f74067a90 2022-06-29T16:04:04.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:04.083+0000 7f0f73fff700 1 -- 172.21.15.60:0/406074759 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f0f74142fe0 con 0x7f0f74067a90 2022-06-29T16:04:04.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:04.943+0000 7f0f6bfff700 1 -- 172.21.15.60:0/406074759 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v67) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0f64015020 con 0x7f0f74067a90 2022-06-29T16:04:04.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:04.947+0000 7f0f73fff700 1 -- 172.21.15.60:0/406074759 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0f580539e0 msgr2=0x7f0f58055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:04.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:04.947+0000 7f0f73fff700 1 --2- 172.21.15.60:0/406074759 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0f580539e0 0x7f0f58055ea0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f0f5c00b000 tx=0x7f0f5c006df0).stop 2022-06-29T16:04:04.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:04.947+0000 7f0f73fff700 1 -- 172.21.15.60:0/406074759 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f74067a90 msgr2=0x7f0f74153090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:04.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:04.947+0000 7f0f73fff700 1 --2- 172.21.15.60:0/406074759 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f74067a90 0x7f0f74153090 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0x7f0f640032f0 tx=0x7f0f64016000).stop 2022-06-29T16:04:04.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:04.947+0000 7f0f73fff700 1 -- 172.21.15.60:0/406074759 shutdown_connections 2022-06-29T16:04:04.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:04.947+0000 7f0f73fff700 1 --2- 172.21.15.60:0/406074759 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0f580539e0 0x7f0f58055ea0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:04.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:04.947+0000 7f0f73fff700 1 --2- 172.21.15.60:0/406074759 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0f740aec30 0x7f0f74135a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:04.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:04.947+0000 7f0f73fff700 1 --2- 172.21.15.60:0/406074759 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f74067a90 0x7f0f74153090 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:04.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:04.947+0000 7f0f73fff700 1 --2- 172.21.15.60:0/406074759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f7405fb00 0x7f0f7413a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:04.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:04.947+0000 7f0f73fff700 1 -- 172.21.15.60:0/406074759 >> 172.21.15.60:0/406074759 conn(0x7f0f74063a60 msgr2=0x7f0f740affd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:04.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:04.947+0000 7f0f73fff700 1 -- 172.21.15.60:0/406074759 shutdown_connections 2022-06-29T16:04:04.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:04.947+0000 7f0f73fff700 1 -- 172.21.15.60:0/406074759 wait complete. 2022-06-29T16:04:04.973 INFO:teuthology.orchestra.run.smithi060.stderr: removed all blacklist entries 2022-06-29T16:04:04.975 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-06-29T16:04:04.976 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-06-29T16:04:05.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.079+0000 7f007f59e700 1 Processor -- start 2022-06-29T16:04:05.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.079+0000 7f007f59e700 1 -- start start 2022-06-29T16:04:05.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.083+0000 7f007f59e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f00800b3270 0x7f00800ace00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:05.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.083+0000 7f007f59e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00800adc80 con 0x7f00800aca20 2022-06-29T16:04:05.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.083+0000 7f007f59e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00800ad740 con 0x7f0080062cd0 2022-06-29T16:04:05.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.083+0000 7f007f59e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00800ad9e0 con 0x7f00800b3270 2022-06-29T16:04:05.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.083+0000 7f0077fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f00800b3270 0x7f00800ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:05.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.083+0000 7f0077fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f00800b3270 0x7f00800ace00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49112/0 (socket says 172.21.15.60:49112) 2022-06-29T16:04:05.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.083+0000 7f0077fff700 1 -- 172.21.15.60:0/1643276719 learned_addr learned my addr 172.21.15.60:0/1643276719 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:05.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.083+0000 7f007e59c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f00800aca20 0x7f0080143030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57196/0 (socket says 172.21.15.60:57196) 2022-06-29T16:04:05.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.083+0000 7f0077fff700 1 -- 172.21.15.60:0/1643276719 >> v1:172.21.15.92:6789/0 conn(0x7f0080062cd0 legacy=0x7f00801461c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:05.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.083+0000 7f0077fff700 1 -- 172.21.15.60:0/1643276719 >> v1:172.21.15.60:6789/0 conn(0x7f00800aca20 legacy=0x7f0080143030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:05.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.083+0000 7f0077fff700 1 -- 172.21.15.60:0/1643276719 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0080142920 con 0x7f00800b3270 2022-06-29T16:04:05.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.083+0000 7f0077fff700 1 --2- 172.21.15.60:0/1643276719 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00800b3270 0x7f00800ace00 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7f00800b7c60 tx=0x7f0068006eb0).ready entity=mon.0 client_cookie=995db8dfcab4799c server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:05.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.083+0000 7f007dd9b700 1 -- 172.21.15.60:0/1643276719 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f006800d610 con 0x7f00800b3270 2022-06-29T16:04:05.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.083+0000 7f007dd9b700 1 -- 172.21.15.60:0/1643276719 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0068000e90 con 0x7f00800b3270 2022-06-29T16:04:05.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.083+0000 7f007dd9b700 1 -- 172.21.15.60:0/1643276719 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f006800d960 con 0x7f00800b3270 2022-06-29T16:04:05.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.083+0000 7f007f59e700 1 -- 172.21.15.60:0/1643276719 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00800b3270 msgr2=0x7f00800ace00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:05.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.083+0000 7f007f59e700 1 --2- 172.21.15.60:0/1643276719 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00800b3270 0x7f00800ace00 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7f00800b7c60 tx=0x7f0068006eb0).stop 2022-06-29T16:04:05.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.083+0000 7f007f59e700 1 -- 172.21.15.60:0/1643276719 shutdown_connections 2022-06-29T16:04:05.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.083+0000 7f007f59e700 1 --2- 172.21.15.60:0/1643276719 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00800b3270 0x7f00800ace00 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:05.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.083+0000 7f007f59e700 1 -- 172.21.15.60:0/1643276719 >> 172.21.15.60:0/1643276719 conn(0x7f00800b6f80 msgr2=0x7f00800b7390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:05.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.083+0000 7f007f59e700 1 -- 172.21.15.60:0/1643276719 shutdown_connections 2022-06-29T16:04:05.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.083+0000 7f007f59e700 1 -- 172.21.15.60:0/1643276719 wait complete. 2022-06-29T16:04:05.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.087+0000 7f007f59e700 1 Processor -- start 2022-06-29T16:04:05.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.087+0000 7f007f59e700 1 -- start start 2022-06-29T16:04:05.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.087+0000 7f007f59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0080062cd0 0x7f0080151f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:05.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.087+0000 7f007f59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00800aca20 0x7f0080148e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:05.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.087+0000 7f007f59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00800b3270 0x7f0080141d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:05.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.087+0000 7f007f59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00800b0390 con 0x7f00800aca20 2022-06-29T16:04:05.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.087+0000 7f007f59e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00800b0600 con 0x7f00800b3270 2022-06-29T16:04:05.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.087+0000 7f007f59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0080155480 con 0x7f0080062cd0 2022-06-29T16:04:05.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.087+0000 7f0077fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00800aca20 0x7f0080148e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:05.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.087+0000 7f0077fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00800aca20 0x7f0080148e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49118/0 (socket says 172.21.15.60:49118) 2022-06-29T16:04:05.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.087+0000 7f0077fff700 1 -- 172.21.15.60:0/3111223022 learned_addr learned my addr 172.21.15.60:0/3111223022 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:05.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.087+0000 7f007e59c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0080062cd0 0x7f0080151f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:05.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.087+0000 7f007ed9d700 1 --2- 172.21.15.60:0/3111223022 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00800b3270 0x7f0080141d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:05.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.091+0000 7f0077fff700 1 -- 172.21.15.60:0/3111223022 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0080062cd0 msgr2=0x7f0080151f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:05.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.091+0000 7f0077fff700 1 --2- 172.21.15.60:0/3111223022 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0080062cd0 0x7f0080151f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:05.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.091+0000 7f0077fff700 1 -- 172.21.15.60:0/3111223022 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00800b3270 msgr2=0x7f0080141d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:05.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.091+0000 7f0077fff700 1 --2- 172.21.15.60:0/3111223022 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00800b3270 0x7f0080141d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:05.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.091+0000 7f0077fff700 1 -- 172.21.15.60:0/3111223022 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006800b1e0 con 0x7f00800aca20 2022-06-29T16:04:05.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.091+0000 7f007ed9d700 1 --2- 172.21.15.60:0/3111223022 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00800b3270 0x7f0080141d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:04:05.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.091+0000 7f0077fff700 1 --2- 172.21.15.60:0/3111223022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00800aca20 0x7f0080148e70 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0x7f00800b7c60 tx=0x7f0068015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:05.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.091+0000 7f00777fe700 1 -- 172.21.15.60:0/3111223022 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f006800bec0 con 0x7f00800aca20 2022-06-29T16:04:05.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.091+0000 7f00777fe700 1 -- 172.21.15.60:0/3111223022 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f006800d970 con 0x7f00800aca20 2022-06-29T16:04:05.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.091+0000 7f007f59e700 1 -- 172.21.15.60:0/3111223022 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00800ae750 con 0x7f00800aca20 2022-06-29T16:04:05.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.091+0000 7f00777fe700 1 -- 172.21.15.60:0/3111223022 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0068011d70 con 0x7f00800aca20 2022-06-29T16:04:05.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.091+0000 7f007f59e700 1 -- 172.21.15.60:0/3111223022 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00800ae310 con 0x7f00800aca20 2022-06-29T16:04:05.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.091+0000 7f007f59e700 1 -- 172.21.15.60:0/3111223022 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00800af7e0 con 0x7f00800aca20 2022-06-29T16:04:05.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.091+0000 7f00777fe700 1 -- 172.21.15.60:0/3111223022 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0068025030 con 0x7f00800aca20 2022-06-29T16:04:05.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.091+0000 7f00777fe700 1 --2- 172.21.15.60:0/3111223022 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0064053950 0x7f0064055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:05.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.091+0000 7f00777fe700 1 -- 172.21.15.60:0/3111223022 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f006806cce0 con 0x7f00800aca20 2022-06-29T16:04:05.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.091+0000 7f007e59c700 1 --2- 172.21.15.60:0/3111223022 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0064053950 0x7f0064055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:05.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.095+0000 7f007e59c700 1 --2- 172.21.15.60:0/3111223022 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0064053950 0x7f0064055e10 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f007000d000 tx=0x7f0070007b40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:05.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.099+0000 7f00777fe700 1 -- 172.21.15.60:0/3111223022 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f00800af7e0 con 0x7f00800aca20 2022-06-29T16:04:05.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.271+0000 7f007f59e700 1 -- 172.21.15.60:0/3111223022 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f00800af7e0 con 0x7f00800aca20 2022-06-29T16:04:05.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.271+0000 7f00777fe700 1 -- 172.21.15.60:0/3111223022 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v24) v1 ==== 80+0+4242 (secure 0 0 0) 0x7f00800af7e0 con 0x7f00800aca20 2022-06-29T16:04:05.277 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:04:05.278 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:04:05.278 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_dump": [ 2022-06-29T16:04:05.278 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:05.279 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.a", 2022-06-29T16:04:05.279 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxivv7JNhAAnA291UxlB/SfgITOipP00A==", 2022-06-29T16:04:05.279 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:04:05.279 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:04:05.279 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:04:05.280 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:04:05.280 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:04:05.280 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:05.280 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:05.281 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:05.281 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.b", 2022-06-29T16:04:05.281 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxiRSr/OhAAfFQTF2YHvzCv5wXBVCA5WQ==", 2022-06-29T16:04:05.281 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:04:05.282 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:04:05.282 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:04:05.282 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:04:05.283 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:04:05.283 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:05.283 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:05.283 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:05.283 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.0", 2022-06-29T16:04:05.284 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiZYPxOBAAIkweWJlTJIXgxnlagkGdXA==", 2022-06-29T16:04:05.284 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:04:05.284 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:04:05.285 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:04:05.285 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:04:05.285 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:05.285 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:05.285 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:05.286 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.1", 2022-06-29T16:04:05.286 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxiS2JGBhAAJn0UfAxZI3IVFyD5QBdb/Q==", 2022-06-29T16:04:05.286 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:04:05.286 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:04:05.287 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:04:05.287 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:04:05.287 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:05.287 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:05.288 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:05.288 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.2", 2022-06-29T16:04:05.288 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxiPhkfEhAApH7r8OUJzW7fZNM6XVv1dg==", 2022-06-29T16:04:05.288 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:04:05.288 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:04:05.289 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:04:05.289 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:04:05.289 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:05.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:05.290 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:05.290 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.3", 2022-06-29T16:04:05.290 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxit/47HhAA1vyU6h7hjPG+op9qP/rVYQ==", 2022-06-29T16:04:05.290 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:04:05.290 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:04:05.291 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:04:05.291 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:04:05.291 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:05.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:05.292 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:05.292 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.4", 2022-06-29T16:04:05.292 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxiaKzKBxAAbIwCF9ag7ezsNy11hi0BEw==", 2022-06-29T16:04:05.292 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:04:05.292 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:04:05.293 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:04:05.293 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:04:05.293 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:05.294 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:05.294 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:05.294 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.5", 2022-06-29T16:04:05.294 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxiJQV/IBAASBBDb6UFgEtXILe4fcT/xA==", 2022-06-29T16:04:05.295 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:04:05.295 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:04:05.295 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:04:05.295 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:04:05.295 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:05.296 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:05.296 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:05.296 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.6", 2022-06-29T16:04:05.296 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxi6D07OBAA2FWiZ+mNK+jFMp/kPZnbQQ==", 2022-06-29T16:04:05.297 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:04:05.297 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:04:05.297 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:04:05.297 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:04:05.298 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:05.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:05.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:05.299 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.7", 2022-06-29T16:04:05.299 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCMdrxinEOcEhAA562ybpzGHNQfWIq8q7zw9w==", 2022-06-29T16:04:05.299 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:04:05.300 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:04:05.300 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:04:05.301 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:04:05.301 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:05.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:05.302 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:05.302 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.0", 2022-06-29T16:04:05.303 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiHXPwBRAAkOOwlYJvIW7dqVFdlXXoqw==", 2022-06-29T16:04:05.303 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:04:05.303 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:04:05.303 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2022-06-29T16:04:05.304 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw pool=cephfs_data" 2022-06-29T16:04:05.304 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:05.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:05.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:05.305 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.1", 2022-06-29T16:04:05.305 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxibe5JCRAAGBJbi9t09w0836JqrAT7+w==", 2022-06-29T16:04:05.305 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:04:05.306 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:04:05.306 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2022-06-29T16:04:05.307 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw pool=cephfs_data" 2022-06-29T16:04:05.307 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:05.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:05.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:05.308 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.2", 2022-06-29T16:04:05.309 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiK3dBDRAA7XINPSOPQDfOrUYEpHC+gw==", 2022-06-29T16:04:05.309 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:04:05.310 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:04:05.310 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2022-06-29T16:04:05.311 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw pool=cephfs_data" 2022-06-29T16:04:05.311 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:05.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:05.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:05.313 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.admin", 2022-06-29T16:04:05.313 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxieKVcGBAAnjH80a0qWIArz2qwjYHWdQ==", 2022-06-29T16:04:05.314 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:04:05.314 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T16:04:05.314 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:04:05.315 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:04:05.315 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:04:05.315 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:05.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:05.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:05.317 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.x", 2022-06-29T16:04:05.317 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxi2LZJNBAA3MKb/pXYbQCsoRx8EQfGbw==", 2022-06-29T16:04:05.317 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:04:05.318 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T16:04:05.318 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2022-06-29T16:04:05.318 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:04:05.319 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:05.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:05.320 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:05.320 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.y", 2022-06-29T16:04:05.321 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxi3LmaMBAAN+0mZrdKXqSXOipVTA1drw==", 2022-06-29T16:04:05.321 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:04:05.322 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T16:04:05.322 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2022-06-29T16:04:05.323 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:04:05.323 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:05.324 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:05.324 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:04:05.324 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:04:05.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.271+0000 7f007f59e700 1 -- 172.21.15.60:0/3111223022 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0064053950 msgr2=0x7f0064055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:05.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.271+0000 7f007f59e700 1 --2- 172.21.15.60:0/3111223022 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0064053950 0x7f0064055e10 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f007000d000 tx=0x7f0070007b40).stop 2022-06-29T16:04:05.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.271+0000 7f007f59e700 1 -- 172.21.15.60:0/3111223022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00800aca20 msgr2=0x7f0080148e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:05.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.271+0000 7f007f59e700 1 --2- 172.21.15.60:0/3111223022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00800aca20 0x7f0080148e70 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0x7f00800b7c60 tx=0x7f0068015000).stop 2022-06-29T16:04:05.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.271+0000 7f007f59e700 1 -- 172.21.15.60:0/3111223022 shutdown_connections 2022-06-29T16:04:05.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.271+0000 7f007f59e700 1 --2- 172.21.15.60:0/3111223022 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0064053950 0x7f0064055e10 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:05.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.271+0000 7f007f59e700 1 --2- 172.21.15.60:0/3111223022 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00800b3270 0x7f0080141d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:05.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.271+0000 7f007f59e700 1 --2- 172.21.15.60:0/3111223022 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00800aca20 0x7f0080148e70 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:05.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.271+0000 7f007f59e700 1 --2- 172.21.15.60:0/3111223022 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0080062cd0 0x7f0080151f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:05.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.271+0000 7f007f59e700 1 -- 172.21.15.60:0/3111223022 >> 172.21.15.60:0/3111223022 conn(0x7f00800b6f80 msgr2=0x7f00800b21a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:05.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.271+0000 7f007f59e700 1 -- 172.21.15.60:0/3111223022 shutdown_connections 2022-06-29T16:04:05.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.271+0000 7f007f59e700 1 -- 172.21.15.60:0/3111223022 wait complete. 2022-06-29T16:04:05.331 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:04:05.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.435+0000 7ffa2e27d700 1 Processor -- start 2022-06-29T16:04:05.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.435+0000 7ffa2e27d700 1 -- start start 2022-06-29T16:04:05.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.435+0000 7ffa2e27d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffa280b2130 0x7ffa28146100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:05.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.435+0000 7ffa2e27d700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa2805d450 con 0x7ffa2805d070 2022-06-29T16:04:05.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.435+0000 7ffa2e27d700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa280b25d0 con 0x7ffa280aed00 2022-06-29T16:04:05.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.435+0000 7ffa2e27d700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa280adf90 con 0x7ffa280b2130 2022-06-29T16:04:05.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.435+0000 7ffa2ca7a700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffa280b2130 0x7ffa28146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:05.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.435+0000 7ffa2d27b700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ffa2805d070 0x7ffa2814aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57210/0 (socket says 172.21.15.60:57210) 2022-06-29T16:04:05.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.435+0000 7ffa2ca7a700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ffa280b2130 0x7ffa28146100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49124/0 (socket says 172.21.15.60:49124) 2022-06-29T16:04:05.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.435+0000 7ffa2d27b700 1 -- 172.21.15.60:0/2175491502 learned_addr learned my addr 172.21.15.60:0/2175491502 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:04:05.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.435+0000 7ffa2ca7a700 1 -- 172.21.15.60:0/2175491502 >> v1:172.21.15.92:6789/0 conn(0x7ffa280aed00 legacy=0x7ffa281465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:05.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.435+0000 7ffa2ca7a700 1 -- 172.21.15.60:0/2175491502 >> v1:172.21.15.60:6789/0 conn(0x7ffa2805d070 legacy=0x7ffa2814aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:05.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.435+0000 7ffa2ca7a700 1 -- 172.21.15.60:0/2175491502 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2814a350 con 0x7ffa280b2130 2022-06-29T16:04:05.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.435+0000 7ffa2ca7a700 1 --2- 172.21.15.60:0/2175491502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa280b2130 0x7ffa28146100 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0x7ffa280b5cd0 tx=0x7ffa200091d0).ready entity=mon.0 client_cookie=da358f5b7bdab344 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:05.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.435+0000 7ffa1ffff700 1 -- 172.21.15.60:0/2175491502 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa20009ea0 con 0x7ffa280b2130 2022-06-29T16:04:05.448 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.435+0000 7ffa1ffff700 1 -- 172.21.15.60:0/2175491502 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa2000b860 con 0x7ffa280b2130 2022-06-29T16:04:05.448 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.435+0000 7ffa1ffff700 1 -- 172.21.15.60:0/2175491502 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa2000b9d0 con 0x7ffa280b2130 2022-06-29T16:04:05.448 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.435+0000 7ffa2e27d700 1 -- 172.21.15.60:0/2175491502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa280b2130 msgr2=0x7ffa28146100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:05.449 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.435+0000 7ffa2e27d700 1 --2- 172.21.15.60:0/2175491502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa280b2130 0x7ffa28146100 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0x7ffa280b5cd0 tx=0x7ffa200091d0).stop 2022-06-29T16:04:05.449 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.435+0000 7ffa2e27d700 1 -- 172.21.15.60:0/2175491502 shutdown_connections 2022-06-29T16:04:05.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.435+0000 7ffa2e27d700 1 --2- 172.21.15.60:0/2175491502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa280b2130 0x7ffa28146100 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:05.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.435+0000 7ffa2e27d700 1 -- 172.21.15.60:0/2175491502 >> 172.21.15.60:0/2175491502 conn(0x7ffa280b5890 msgr2=0x7ffa280b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:05.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.435+0000 7ffa2e27d700 1 -- 172.21.15.60:0/2175491502 shutdown_connections 2022-06-29T16:04:05.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.435+0000 7ffa2e27d700 1 -- 172.21.15.60:0/2175491502 wait complete. 2022-06-29T16:04:05.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.439+0000 7ffa2e27d700 1 Processor -- start 2022-06-29T16:04:05.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.439+0000 7ffa2e27d700 1 -- start start 2022-06-29T16:04:05.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.439+0000 7ffa2e27d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffa2805d070 0x7ffa28144850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:05.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.439+0000 7ffa2e27d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa280aed00 0x7ffa28159a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:05.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.439+0000 7ffa2e27d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffa280b2130 0x7ffa280657f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:05.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.439+0000 7ffa2e27d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa28066090 con 0x7ffa280aed00 2022-06-29T16:04:05.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.439+0000 7ffa2e27d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa28066330 con 0x7ffa2805d070 2022-06-29T16:04:05.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.439+0000 7ffa2e27d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa281188c0 con 0x7ffa280b2130 2022-06-29T16:04:05.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.439+0000 7ffa2ca7a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa280aed00 0x7ffa28159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:05.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.439+0000 7ffa2ca7a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa280aed00 0x7ffa28159a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49132/0 (socket says 172.21.15.60:49132) 2022-06-29T16:04:05.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.439+0000 7ffa2ca7a700 1 -- 172.21.15.60:0/3699668213 learned_addr learned my addr 172.21.15.60:0/3699668213 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:05.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.443+0000 7ffa2d27b700 1 --2- 172.21.15.60:0/3699668213 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffa2805d070 0x7ffa28144850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:05.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.443+0000 7ffa2da7c700 1 --2- 172.21.15.60:0/3699668213 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffa280b2130 0x7ffa280657f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:05.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.443+0000 7ffa2ca7a700 1 -- 172.21.15.60:0/3699668213 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffa280b2130 msgr2=0x7ffa280657f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:05.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.443+0000 7ffa2ca7a700 1 --2- 172.21.15.60:0/3699668213 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffa280b2130 0x7ffa280657f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:05.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.443+0000 7ffa2ca7a700 1 -- 172.21.15.60:0/3699668213 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffa2805d070 msgr2=0x7ffa28144850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:05.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.443+0000 7ffa2ca7a700 1 --2- 172.21.15.60:0/3699668213 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffa2805d070 0x7ffa28144850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:05.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.443+0000 7ffa2ca7a700 1 -- 172.21.15.60:0/3699668213 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa20009470 con 0x7ffa280aed00 2022-06-29T16:04:05.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.443+0000 7ffa2da7c700 1 --2- 172.21.15.60:0/3699668213 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffa280b2130 0x7ffa280657f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:04:05.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.443+0000 7ffa2ca7a700 1 --2- 172.21.15.60:0/3699668213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa280aed00 0x7ffa28159a00 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7ffa280b5cd0 tx=0x7ffa20016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:05.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.443+0000 7ffa1e7fc700 1 -- 172.21.15.60:0/3699668213 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa200032f0 con 0x7ffa280aed00 2022-06-29T16:04:05.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.443+0000 7ffa1e7fc700 1 -- 172.21.15.60:0/3699668213 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa2001f3f0 con 0x7ffa280aed00 2022-06-29T16:04:05.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.443+0000 7ffa2e27d700 1 -- 172.21.15.60:0/3699668213 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa28065ce0 con 0x7ffa280aed00 2022-06-29T16:04:05.461 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.443+0000 7ffa1e7fc700 1 -- 172.21.15.60:0/3699668213 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa2001f750 con 0x7ffa280aed00 2022-06-29T16:04:05.461 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.443+0000 7ffa2e27d700 1 -- 172.21.15.60:0/3699668213 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa28118d60 con 0x7ffa280aed00 2022-06-29T16:04:05.461 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.443+0000 7ffa2e27d700 1 -- 172.21.15.60:0/3699668213 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa2814a5f0 con 0x7ffa280aed00 2022-06-29T16:04:05.461 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.443+0000 7ffa1e7fc700 1 -- 172.21.15.60:0/3699668213 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ffa2000f070 con 0x7ffa280aed00 2022-06-29T16:04:05.461 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.443+0000 7ffa1e7fc700 1 --2- 172.21.15.60:0/3699668213 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ffa08053aa0 0x7ffa08055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:05.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.443+0000 7ffa1e7fc700 1 -- 172.21.15.60:0/3699668213 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5253+0+0 (secure 0 0 0) 0x7ffa2006c590 con 0x7ffa280aed00 2022-06-29T16:04:05.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.443+0000 7ffa2d27b700 1 --2- 172.21.15.60:0/3699668213 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ffa08053aa0 0x7ffa08055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:05.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.447+0000 7ffa2d27b700 1 --2- 172.21.15.60:0/3699668213 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ffa08053aa0 0x7ffa08055f60 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7ffa2400d000 tx=0x7ffa24009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:05.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.451+0000 7ffa1e7fc700 1 -- 172.21.15.60:0/3699668213 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ffa20043740 con 0x7ffa280aed00 2022-06-29T16:04:05.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.579+0000 7ffa2e27d700 1 -- 172.21.15.60:0/3699668213 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa280573d0 con 0x7ffa280aed00 2022-06-29T16:04:05.587 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.579+0000 7ffa1e7fc700 1 -- 172.21.15.60:0/3699668213 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+13182 (secure 0 0 0) 0x7ffa20024090 con 0x7ffa280aed00 2022-06-29T16:04:05.588 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:04:05.588 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":67,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:04:04.941545+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:04:05.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.583+0000 7ffa2e27d700 1 -- 172.21.15.60:0/3699668213 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ffa08053aa0 msgr2=0x7ffa08055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:05.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.583+0000 7ffa2e27d700 1 --2- 172.21.15.60:0/3699668213 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ffa08053aa0 0x7ffa08055f60 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7ffa2400d000 tx=0x7ffa24009040).stop 2022-06-29T16:04:05.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.583+0000 7ffa2e27d700 1 -- 172.21.15.60:0/3699668213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa280aed00 msgr2=0x7ffa28159a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:05.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.583+0000 7ffa2e27d700 1 --2- 172.21.15.60:0/3699668213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa280aed00 0x7ffa28159a00 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7ffa280b5cd0 tx=0x7ffa20016000).stop 2022-06-29T16:04:05.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.583+0000 7ffa2e27d700 1 -- 172.21.15.60:0/3699668213 shutdown_connections 2022-06-29T16:04:05.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.583+0000 7ffa2e27d700 1 --2- 172.21.15.60:0/3699668213 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ffa08053aa0 0x7ffa08055f60 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:05.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.583+0000 7ffa2e27d700 1 --2- 172.21.15.60:0/3699668213 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffa280b2130 0x7ffa280657f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:05.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.583+0000 7ffa2e27d700 1 --2- 172.21.15.60:0/3699668213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa280aed00 0x7ffa28159a00 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:05.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.583+0000 7ffa2e27d700 1 --2- 172.21.15.60:0/3699668213 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffa2805d070 0x7ffa28144850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:05.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.583+0000 7ffa2e27d700 1 -- 172.21.15.60:0/3699668213 >> 172.21.15.60:0/3699668213 conn(0x7ffa280b5890 msgr2=0x7ffa280af720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:05.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.583+0000 7ffa2e27d700 1 -- 172.21.15.60:0/3699668213 shutdown_connections 2022-06-29T16:04:05.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.583+0000 7ffa2e27d700 1 -- 172.21.15.60:0/3699668213 wait complete. 2022-06-29T16:04:05.602 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-06-29T16:04:05.603 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:04:05.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.699+0000 7fc309252700 1 Processor -- start 2022-06-29T16:04:05.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.699+0000 7fc309252700 1 -- start start 2022-06-29T16:04:05.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.699+0000 7fc309252700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc3040b9d70 0x7fc3041465d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:05.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.699+0000 7fc309252700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc3040b3d30 0x7fc304146ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:05.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.699+0000 7fc309252700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc304061d90 0x7fc30405bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:05.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.699+0000 7fc309252700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3040b4f50 con 0x7fc3040b9d70 2022-06-29T16:04:05.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.699+0000 7fc309252700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3041461a0 con 0x7fc3040b3d30 2022-06-29T16:04:05.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.699+0000 7fc309252700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3040ba150 con 0x7fc304061d90 2022-06-29T16:04:05.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.699+0000 7fc302d9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc3040b9d70 0x7fc3041465d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:05.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.699+0000 7fc302d9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc3040b9d70 0x7fc3041465d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49140/0 (socket says 172.21.15.60:49140) 2022-06-29T16:04:05.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.699+0000 7fc302d9d700 1 -- 172.21.15.60:0/681027301 learned_addr learned my addr 172.21.15.60:0/681027301 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:05.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.699+0000 7fc30259c700 1 --2- 172.21.15.60:0/681027301 >> v2:172.21.15.92:3300/0 conn(0x7fc3040b3d30 0x7fc304146ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:05.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.699+0000 7fc30359e700 1 --2- 172.21.15.60:0/681027301 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc304061d90 0x7fc30405bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:05.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.699+0000 7fc302d9d700 1 -- 172.21.15.60:0/681027301 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc304061d90 msgr2=0x7fc30405bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:05.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.699+0000 7fc302d9d700 1 --2- 172.21.15.60:0/681027301 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc304061d90 0x7fc30405bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:05.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.699+0000 7fc302d9d700 1 -- 172.21.15.60:0/681027301 >> v2:172.21.15.92:3300/0 conn(0x7fc3040b3d30 msgr2=0x7fc304146ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:05.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.699+0000 7fc302d9d700 1 --2- 172.21.15.60:0/681027301 >> v2:172.21.15.92:3300/0 conn(0x7fc3040b3d30 0x7fc304146ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:05.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.699+0000 7fc302d9d700 1 -- 172.21.15.60:0/681027301 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3040b5180 con 0x7fc3040b9d70 2022-06-29T16:04:05.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.699+0000 7fc30259c700 1 --2- 172.21.15.60:0/681027301 >> v2:172.21.15.92:3300/0 conn(0x7fc3040b3d30 0x7fc304146ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:04:05.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.699+0000 7fc302d9d700 1 --2- 172.21.15.60:0/681027301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3040b9d70 0x7fc3041465d0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7fc3040b1c90 tx=0x7fc2fc009290).ready entity=mon.0 client_cookie=c32aec686164e6be server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:05.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.699+0000 7fc301d9b700 1 -- 172.21.15.60:0/681027301 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2fc00b830 con 0x7fc3040b9d70 2022-06-29T16:04:05.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.699+0000 7fc301d9b700 1 -- 172.21.15.60:0/681027301 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2fc00b990 con 0x7fc3040b9d70 2022-06-29T16:04:05.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.699+0000 7fc301d9b700 1 -- 172.21.15.60:0/681027301 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2fc00bb00 con 0x7fc3040b9d70 2022-06-29T16:04:05.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.703+0000 7fc309252700 1 -- 172.21.15.60:0/681027301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3040b9d70 msgr2=0x7fc3041465d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:05.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.703+0000 7fc309252700 1 --2- 172.21.15.60:0/681027301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3040b9d70 0x7fc3041465d0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7fc3040b1c90 tx=0x7fc2fc009290).stop 2022-06-29T16:04:05.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.703+0000 7fc309252700 1 -- 172.21.15.60:0/681027301 shutdown_connections 2022-06-29T16:04:05.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.703+0000 7fc309252700 1 --2- 172.21.15.60:0/681027301 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc304061d90 0x7fc30405bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:05.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.703+0000 7fc309252700 1 --2- 172.21.15.60:0/681027301 >> v2:172.21.15.92:3300/0 conn(0x7fc3040b3d30 0x7fc304146ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:05.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.703+0000 7fc309252700 1 --2- 172.21.15.60:0/681027301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3040b9d70 0x7fc3041465d0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:05.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.703+0000 7fc309252700 1 -- 172.21.15.60:0/681027301 >> 172.21.15.60:0/681027301 conn(0x7fc304062e70 msgr2=0x7fc3040b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:05.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.703+0000 7fc309252700 1 -- 172.21.15.60:0/681027301 shutdown_connections 2022-06-29T16:04:05.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.703+0000 7fc309252700 1 -- 172.21.15.60:0/681027301 wait complete. 2022-06-29T16:04:05.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.703+0000 7fc309252700 1 Processor -- start 2022-06-29T16:04:05.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.703+0000 7fc309252700 1 -- start start 2022-06-29T16:04:05.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.703+0000 7fc309252700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc304061d90 0x7fc3040ae000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:05.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.703+0000 7fc309252700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3040b3d30 0x7fc30413fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:05.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.703+0000 7fc309252700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3040b9d70 0x7fc304140190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:05.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.703+0000 7fc309252700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc304140680 con 0x7fc3040b9d70 2022-06-29T16:04:05.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.703+0000 7fc309252700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc304113150 con 0x7fc304061d90 2022-06-29T16:04:05.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.703+0000 7fc309252700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3041133f0 con 0x7fc3040b3d30 2022-06-29T16:04:05.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.703+0000 7fc30359e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3040b9d70 0x7fc304140190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:05.721 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.703+0000 7fc30359e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3040b9d70 0x7fc304140190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49148/0 (socket says 172.21.15.60:49148) 2022-06-29T16:04:05.721 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.703+0000 7fc30359e700 1 -- 172.21.15.60:0/3044470967 learned_addr learned my addr 172.21.15.60:0/3044470967 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:05.721 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.703+0000 7fc302d9d700 1 --2- 172.21.15.60:0/3044470967 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc304061d90 0x7fc3040ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:05.721 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.703+0000 7fc30259c700 1 --2- 172.21.15.60:0/3044470967 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3040b3d30 0x7fc30413fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:05.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.703+0000 7fc30359e700 1 -- 172.21.15.60:0/3044470967 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3040b3d30 msgr2=0x7fc30413fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:05.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.703+0000 7fc30359e700 1 --2- 172.21.15.60:0/3044470967 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3040b3d30 0x7fc30413fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:05.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.703+0000 7fc30359e700 1 -- 172.21.15.60:0/3044470967 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc304061d90 msgr2=0x7fc3040ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:05.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.703+0000 7fc30359e700 1 --2- 172.21.15.60:0/3044470967 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc304061d90 0x7fc3040ae000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:05.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.703+0000 7fc30359e700 1 -- 172.21.15.60:0/3044470967 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2fc009530 con 0x7fc3040b9d70 2022-06-29T16:04:05.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.703+0000 7fc30359e700 1 --2- 172.21.15.60:0/3044470967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3040b9d70 0x7fc304140190 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0x7fc2ec00a000 tx=0x7fc2ec008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:05.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.703+0000 7fc2fbfff700 1 -- 172.21.15.60:0/3044470967 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2ec011780 con 0x7fc3040b9d70 2022-06-29T16:04:05.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.703+0000 7fc2fbfff700 1 -- 172.21.15.60:0/3044470967 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2ec0118e0 con 0x7fc3040b9d70 2022-06-29T16:04:05.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.703+0000 7fc309252700 1 -- 172.21.15.60:0/3044470967 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3040ae4f0 con 0x7fc3040b9d70 2022-06-29T16:04:05.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.703+0000 7fc2fbfff700 1 -- 172.21.15.60:0/3044470967 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2ec011c40 con 0x7fc3040b9d70 2022-06-29T16:04:05.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.703+0000 7fc309252700 1 -- 172.21.15.60:0/3044470967 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3040aed20 con 0x7fc3040b9d70 2022-06-29T16:04:05.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.707+0000 7fc309252700 1 -- 172.21.15.60:0/3044470967 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3040573d0 con 0x7fc3040b9d70 2022-06-29T16:04:05.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.707+0000 7fc2fbfff700 1 -- 172.21.15.60:0/3044470967 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc2ec01c460 con 0x7fc3040b9d70 2022-06-29T16:04:05.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.707+0000 7fc2fbfff700 1 --2- 172.21.15.60:0/3044470967 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc2e8053aa0 0x7fc2e8055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:05.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.707+0000 7fc2fbfff700 1 -- 172.21.15.60:0/3044470967 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fc2ec06bba0 con 0x7fc3040b9d70 2022-06-29T16:04:05.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.707+0000 7fc302d9d700 1 --2- 172.21.15.60:0/3044470967 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc2e8053aa0 0x7fc2e8055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:05.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.707+0000 7fc302d9d700 1 --2- 172.21.15.60:0/3044470967 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc2e8053aa0 0x7fc2e8055f60 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fc2fc01a000 tx=0x7fc2fc016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:05.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.711+0000 7fc2fbfff700 1 -- 172.21.15.60:0/3044470967 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc2ec042d90 con 0x7fc3040b9d70 2022-06-29T16:04:05.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.835+0000 7fc309252700 1 -- 172.21.15.60:0/3044470967 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fc30405da40 con 0x7fc3040b9d70 2022-06-29T16:04:05.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.835+0000 7fc2fbfff700 1 -- 172.21.15.60:0/3044470967 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v67) v1 ==== 110+0+10 (secure 0 0 0) 0x7fc2ec0463b0 con 0x7fc3040b9d70 2022-06-29T16:04:05.842 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:04:05.845 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.839+0000 7fc309252700 1 -- 172.21.15.60:0/3044470967 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc2e8053aa0 msgr2=0x7fc2e8055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:05.845 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.839+0000 7fc309252700 1 --2- 172.21.15.60:0/3044470967 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc2e8053aa0 0x7fc2e8055f60 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fc2fc01a000 tx=0x7fc2fc016000).stop 2022-06-29T16:04:05.845 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.839+0000 7fc309252700 1 -- 172.21.15.60:0/3044470967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3040b9d70 msgr2=0x7fc304140190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:05.845 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.839+0000 7fc309252700 1 --2- 172.21.15.60:0/3044470967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3040b9d70 0x7fc304140190 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0x7fc2ec00a000 tx=0x7fc2ec008040).stop 2022-06-29T16:04:05.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.839+0000 7fc309252700 1 -- 172.21.15.60:0/3044470967 shutdown_connections 2022-06-29T16:04:05.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.839+0000 7fc309252700 1 --2- 172.21.15.60:0/3044470967 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc2e8053aa0 0x7fc2e8055f60 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:05.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.839+0000 7fc309252700 1 --2- 172.21.15.60:0/3044470967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc3040b9d70 0x7fc304140190 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:05.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.839+0000 7fc309252700 1 --2- 172.21.15.60:0/3044470967 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc3040b3d30 0x7fc30413fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:05.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.839+0000 7fc309252700 1 --2- 172.21.15.60:0/3044470967 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc304061d90 0x7fc3040ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:05.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.839+0000 7fc309252700 1 -- 172.21.15.60:0/3044470967 >> 172.21.15.60:0/3044470967 conn(0x7fc304062e70 msgr2=0x7fc304142940 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:05.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.839+0000 7fc309252700 1 -- 172.21.15.60:0/3044470967 shutdown_connections 2022-06-29T16:04:05.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.839+0000 7fc309252700 1 -- 172.21.15.60:0/3044470967 wait complete. 2022-06-29T16:04:05.859 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:04:05.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.955+0000 7f486d89c700 1 Processor -- start 2022-06-29T16:04:05.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.955+0000 7f486d89c700 1 -- start start 2022-06-29T16:04:05.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.959+0000 7f486d89c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f486813f7b0 0x7f486813fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:05.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.959+0000 7f486d89c700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48680b6e20 con 0x7f486813f3d0 2022-06-29T16:04:05.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.959+0000 7f486d89c700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48681430e0 con 0x7f4868060260 2022-06-29T16:04:05.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.959+0000 7f486d89c700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4868142ef0 con 0x7f486813f7b0 2022-06-29T16:04:05.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.959+0000 7f4866ffd700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f486813f3d0 0x7f4868143820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57238/0 (socket says 172.21.15.60:57238) 2022-06-29T16:04:05.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.959+0000 7f4866ffd700 1 -- 172.21.15.60:0/815776857 learned_addr learned my addr 172.21.15.60:0/815776857 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:04:05.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.959+0000 7f48677fe700 1 --2- 172.21.15.60:0/815776857 >> v2:172.21.15.60:3300/0 conn(0x7f486813f7b0 0x7f486813fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:05.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.959+0000 7f4865ffb700 1 -- 172.21.15.60:0/815776857 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 8479782 0 0) 0x7f48680b6e20 con 0x7f486813f3d0 2022-06-29T16:04:05.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.959+0000 7f4865ffb700 1 -- 172.21.15.60:0/815776857 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f484c003260 con 0x7f486813f3d0 2022-06-29T16:04:05.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.959+0000 7f4865ffb700 1 -- 172.21.15.60:0/815776857 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1049408451 0 0) 0x7f48681430e0 con 0x7f4868060260 2022-06-29T16:04:05.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.959+0000 7f4865ffb700 1 -- 172.21.15.60:0/815776857 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48680b6e20 con 0x7f4868060260 2022-06-29T16:04:05.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.959+0000 7f48677fe700 1 -- 172.21.15.60:0/815776857 >> v1:172.21.15.92:6789/0 conn(0x7f4868060260 legacy=0x7f48680b6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:05.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.959+0000 7f48677fe700 1 -- 172.21.15.60:0/815776857 >> v1:172.21.15.60:6789/0 conn(0x7f486813f3d0 legacy=0x7f4868143820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:05.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.959+0000 7f48677fe700 1 -- 172.21.15.60:0/815776857 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4868143380 con 0x7f486813f7b0 2022-06-29T16:04:05.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.959+0000 7f48677fe700 1 --2- 172.21.15.60:0/815776857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f486813f7b0 0x7f486813fbe0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0x7f48600032f0 tx=0x7f48600091d0).ready entity=mon.0 client_cookie=811452613609c2fc server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:05.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.959+0000 7f4865ffb700 1 -- 172.21.15.60:0/815776857 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4860010a90 con 0x7f486813f7b0 2022-06-29T16:04:05.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.959+0000 7f4865ffb700 1 -- 172.21.15.60:0/815776857 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4860010bf0 con 0x7f486813f7b0 2022-06-29T16:04:05.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.959+0000 7f4865ffb700 1 -- 172.21.15.60:0/815776857 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4860018420 con 0x7f486813f7b0 2022-06-29T16:04:05.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.959+0000 7f486d89c700 1 -- 172.21.15.60:0/815776857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f486813f7b0 msgr2=0x7f486813fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:05.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.959+0000 7f486d89c700 1 --2- 172.21.15.60:0/815776857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f486813f7b0 0x7f486813fbe0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0x7f48600032f0 tx=0x7f48600091d0).stop 2022-06-29T16:04:05.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.959+0000 7f486d89c700 1 -- 172.21.15.60:0/815776857 shutdown_connections 2022-06-29T16:04:05.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.959+0000 7f486d89c700 1 --2- 172.21.15.60:0/815776857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f486813f7b0 0x7f486813fbe0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:05.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.959+0000 7f486d89c700 1 -- 172.21.15.60:0/815776857 >> 172.21.15.60:0/815776857 conn(0x7f48680643e0 msgr2=0x7f48680b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:05.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.959+0000 7f486d89c700 1 -- 172.21.15.60:0/815776857 shutdown_connections 2022-06-29T16:04:05.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.959+0000 7f486d89c700 1 -- 172.21.15.60:0/815776857 wait complete. 2022-06-29T16:04:05.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.963+0000 7f486d89c700 1 Processor -- start 2022-06-29T16:04:05.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.963+0000 7f486d89c700 1 -- start start 2022-06-29T16:04:05.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.963+0000 7f486d89c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4868060260 0x7f4868150d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:05.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.963+0000 7f486d89c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f486813f3d0 0x7f486805c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:05.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.963+0000 7f486d89c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f486813f7b0 0x7f48680ad7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:05.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.963+0000 7f486d89c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4868066690 con 0x7f486813f3d0 2022-06-29T16:04:05.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.963+0000 7f486d89c700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48680674c0 con 0x7f486813f7b0 2022-06-29T16:04:05.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.963+0000 7f486d89c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4868066170 con 0x7f4868060260 2022-06-29T16:04:05.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.963+0000 7f48667fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f486813f3d0 0x7f486805c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:05.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.967+0000 7f48667fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f486813f3d0 0x7f486805c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49160/0 (socket says 172.21.15.60:49160) 2022-06-29T16:04:05.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.967+0000 7f48667fc700 1 -- 172.21.15.60:0/3692171993 learned_addr learned my addr 172.21.15.60:0/3692171993 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:05.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.967+0000 7f4866ffd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4868060260 0x7f4868150d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:05.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.967+0000 7f48677fe700 1 --2- 172.21.15.60:0/3692171993 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f486813f7b0 0x7f48680ad7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:05.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.967+0000 7f48667fc700 1 -- 172.21.15.60:0/3692171993 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4868060260 msgr2=0x7f4868150d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:05.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.967+0000 7f48667fc700 1 --2- 172.21.15.60:0/3692171993 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4868060260 0x7f4868150d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:05.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.967+0000 7f48667fc700 1 -- 172.21.15.60:0/3692171993 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f486813f7b0 msgr2=0x7f48680ad7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:05.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.967+0000 7f48667fc700 1 --2- 172.21.15.60:0/3692171993 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f486813f7b0 0x7f48680ad7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:05.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.967+0000 7f48667fc700 1 -- 172.21.15.60:0/3692171993 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4860009470 con 0x7f486813f3d0 2022-06-29T16:04:05.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.967+0000 7f48677fe700 1 --2- 172.21.15.60:0/3692171993 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f486813f7b0 0x7f48680ad7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:04:05.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.967+0000 7f4866ffd700 1 --2- 172.21.15.60:0/3692171993 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4868060260 0x7f4868150d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:04:05.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.967+0000 7f48667fc700 1 --2- 172.21.15.60:0/3692171993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f486813f3d0 0x7f486805c990 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7f485c001110 tx=0x7f485c007a90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:05.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.967+0000 7f4857fff700 1 -- 172.21.15.60:0/3692171993 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f485c012400 con 0x7f486813f3d0 2022-06-29T16:04:05.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.967+0000 7f4857fff700 1 -- 172.21.15.60:0/3692171993 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f485c012560 con 0x7f486813f3d0 2022-06-29T16:04:05.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.967+0000 7f4857fff700 1 -- 172.21.15.60:0/3692171993 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f485c0128c0 con 0x7f486813f3d0 2022-06-29T16:04:05.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.967+0000 7f486d89c700 1 -- 172.21.15.60:0/3692171993 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48680b00a0 con 0x7f486813f3d0 2022-06-29T16:04:05.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.967+0000 7f486d89c700 1 -- 172.21.15.60:0/3692171993 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48680b03b0 con 0x7f486813f3d0 2022-06-29T16:04:05.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.967+0000 7f486d89c700 1 -- 172.21.15.60:0/3692171993 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4868066800 con 0x7f486813f3d0 2022-06-29T16:04:05.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.967+0000 7f4857fff700 1 -- 172.21.15.60:0/3692171993 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f485c019030 con 0x7f486813f3d0 2022-06-29T16:04:05.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.971+0000 7f4857fff700 1 --2- 172.21.15.60:0/3692171993 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4844053950 0x7f4844055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:05.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.971+0000 7f4857fff700 1 -- 172.21.15.60:0/3692171993 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f485c06af30 con 0x7f486813f3d0 2022-06-29T16:04:05.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.971+0000 7f4866ffd700 1 --2- 172.21.15.60:0/3692171993 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4844053950 0x7f4844055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:05.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.971+0000 7f4866ffd700 1 --2- 172.21.15.60:0/3692171993 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4844053950 0x7f4844055e10 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f485800d000 tx=0x7f4858009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:05.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:05.979+0000 7f4857fff700 1 -- 172.21.15.60:0/3692171993 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4868066800 con 0x7f486813f3d0 2022-06-29T16:04:06.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.103+0000 7f486d89c700 1 -- 172.21.15.60:0/3692171993 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f4868066330 con 0x7f486813f3d0 2022-06-29T16:04:06.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.103+0000 7f4857fff700 1 -- 172.21.15.60:0/3692171993 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v67) v1 ==== 92+0+10 (secure 0 0 0) 0x7f485c042180 con 0x7f486813f3d0 2022-06-29T16:04:06.112 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:04:06.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.107+0000 7f486d89c700 1 -- 172.21.15.60:0/3692171993 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4844053950 msgr2=0x7f4844055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:06.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.107+0000 7f486d89c700 1 --2- 172.21.15.60:0/3692171993 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4844053950 0x7f4844055e10 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f485800d000 tx=0x7f4858009040).stop 2022-06-29T16:04:06.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.107+0000 7f486d89c700 1 -- 172.21.15.60:0/3692171993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f486813f3d0 msgr2=0x7f486805c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:06.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.107+0000 7f486d89c700 1 --2- 172.21.15.60:0/3692171993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f486813f3d0 0x7f486805c990 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7f485c001110 tx=0x7f485c007a90).stop 2022-06-29T16:04:06.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.107+0000 7f486d89c700 1 -- 172.21.15.60:0/3692171993 shutdown_connections 2022-06-29T16:04:06.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.107+0000 7f486d89c700 1 --2- 172.21.15.60:0/3692171993 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4844053950 0x7f4844055e10 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:06.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.107+0000 7f486d89c700 1 --2- 172.21.15.60:0/3692171993 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f486813f7b0 0x7f48680ad7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:06.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.107+0000 7f486d89c700 1 --2- 172.21.15.60:0/3692171993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f486813f3d0 0x7f486805c990 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:06.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.107+0000 7f486d89c700 1 --2- 172.21.15.60:0/3692171993 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4868060260 0x7f4868150d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:06.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.107+0000 7f486d89c700 1 -- 172.21.15.60:0/3692171993 >> 172.21.15.60:0/3692171993 conn(0x7f48680643e0 msgr2=0x7f48680b1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:06.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.107+0000 7f486d89c700 1 -- 172.21.15.60:0/3692171993 shutdown_connections 2022-06-29T16:04:06.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.107+0000 7f486d89c700 1 -- 172.21.15.60:0/3692171993 wait complete. 2022-06-29T16:04:06.125 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-06-29T16:04:06.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.227+0000 7f0ed0f25700 1 Processor -- start 2022-06-29T16:04:06.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.227+0000 7f0ed0f25700 1 -- start start 2022-06-29T16:04:06.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.227+0000 7f0ed0f25700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0ecc05bf30 0x7f0ecc0b3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:06.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.227+0000 7f0ed0f25700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ecc065820 con 0x7f0ecc05d030 2022-06-29T16:04:06.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.227+0000 7f0ed0f25700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ecc05d410 con 0x7f0ecc14a2b0 2022-06-29T16:04:06.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.227+0000 7f0ed0f25700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ecc066600 con 0x7f0ecc05bf30 2022-06-29T16:04:06.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.227+0000 7f0ecb7fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f0ecc05d030 0x7f0ecc065ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57252/0 (socket says 172.21.15.60:57252) 2022-06-29T16:04:06.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.227+0000 7f0ecb7fe700 1 -- 172.21.15.60:0/953682157 learned_addr learned my addr 172.21.15.60:0/953682157 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:04:06.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.227+0000 7f0ec2ffd700 1 --2- 172.21.15.60:0/953682157 >> v2:172.21.15.92:3300/0 conn(0x7f0ecc05bf30 0x7f0ecc0b3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:06.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.227+0000 7f0ecaffd700 1 -- 172.21.15.60:0/953682157 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 351279169 0 0) 0x7f0ecc065820 con 0x7f0ecc05d030 2022-06-29T16:04:06.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.227+0000 7f0ecaffd700 1 -- 172.21.15.60:0/953682157 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0eac003260 con 0x7f0ecc05d030 2022-06-29T16:04:06.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.227+0000 7f0ec2ffd700 1 -- 172.21.15.60:0/953682157 >> v1:172.21.15.92:6789/0 conn(0x7f0ecc14a2b0 legacy=0x7f0ecc1461d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:06.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.227+0000 7f0ec2ffd700 1 -- 172.21.15.60:0/953682157 >> v1:172.21.15.60:6789/0 conn(0x7f0ecc05d030 legacy=0x7f0ecc065ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:06.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.227+0000 7f0ecaffd700 1 -- 172.21.15.60:0/953682157 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4211377917 0 0) 0x7f0eac003260 con 0x7f0ecc05d030 2022-06-29T16:04:06.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.227+0000 7f0ec2ffd700 1 -- 172.21.15.60:0/953682157 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ecc0b2230 con 0x7f0ecc05bf30 2022-06-29T16:04:06.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.227+0000 7f0ec2ffd700 1 --2- 172.21.15.60:0/953682157 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ecc05bf30 0x7f0ecc0b3900 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f0ebc001800 tx=0x7f0eb4006df0).ready entity=mon.1 client_cookie=3cfee772921f821c server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:06.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.227+0000 7f0ecaffd700 1 -- 172.21.15.60:0/953682157 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ebc001660 con 0x7f0ecc05bf30 2022-06-29T16:04:06.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.227+0000 7f0ecaffd700 1 -- 172.21.15.60:0/953682157 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eb4000e90 con 0x7f0ecc05bf30 2022-06-29T16:04:06.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.227+0000 7f0ecaffd700 1 -- 172.21.15.60:0/953682157 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0eb400d4f0 con 0x7f0ecc05bf30 2022-06-29T16:04:06.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.227+0000 7f0ed0f25700 1 -- 172.21.15.60:0/953682157 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ecc05bf30 msgr2=0x7f0ecc0b3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:06.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.227+0000 7f0ed0f25700 1 --2- 172.21.15.60:0/953682157 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ecc05bf30 0x7f0ecc0b3900 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f0ebc001800 tx=0x7f0eb4006df0).stop 2022-06-29T16:04:06.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.227+0000 7f0ed0f25700 1 -- 172.21.15.60:0/953682157 shutdown_connections 2022-06-29T16:04:06.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.227+0000 7f0ed0f25700 1 --2- 172.21.15.60:0/953682157 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ecc05bf30 0x7f0ecc0b3900 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:06.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.227+0000 7f0ed0f25700 1 -- 172.21.15.60:0/953682157 >> 172.21.15.60:0/953682157 conn(0x7f0ecc064810 msgr2=0x7f0ecc064bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:06.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.227+0000 7f0ed0f25700 1 -- 172.21.15.60:0/953682157 shutdown_connections 2022-06-29T16:04:06.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.227+0000 7f0ed0f25700 1 -- 172.21.15.60:0/953682157 wait complete. 2022-06-29T16:04:06.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.231+0000 7f0ed0f25700 1 Processor -- start 2022-06-29T16:04:06.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.231+0000 7f0ed0f25700 1 -- start start 2022-06-29T16:04:06.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.231+0000 7f0ed0f25700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ecc05bf30 0x7f0ecc14dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:06.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.231+0000 7f0ed0f25700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0ecc05d030 0x7f0ecc156300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:06.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.231+0000 7f0ed0f25700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ecc14a2b0 0x7f0ecc1567f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:06.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.231+0000 7f0ed0f25700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ecc0b3df0 con 0x7f0ecc05bf30 2022-06-29T16:04:06.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.231+0000 7f0ed0f25700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ecc159850 con 0x7f0ecc14a2b0 2022-06-29T16:04:06.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.231+0000 7f0ed0f25700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ecc0aece0 con 0x7f0ecc05d030 2022-06-29T16:04:06.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.231+0000 7f0ecb7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ecc05bf30 0x7f0ecc14dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:06.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.231+0000 7f0ec2ffd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0ecc05d030 0x7f0ecc156300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:06.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.231+0000 7f0ecb7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ecc05bf30 0x7f0ecc14dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49174/0 (socket says 172.21.15.60:49174) 2022-06-29T16:04:06.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.231+0000 7f0ecb7fe700 1 -- 172.21.15.60:0/876267128 learned_addr learned my addr 172.21.15.60:0/876267128 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:06.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.231+0000 7f0ecbfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ecc14a2b0 0x7f0ecc1567f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:06.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.231+0000 7f0ec2ffd700 1 -- 172.21.15.60:0/876267128 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ecc14a2b0 msgr2=0x7f0ecc1567f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:06.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.231+0000 7f0ec2ffd700 1 --2- 172.21.15.60:0/876267128 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ecc14a2b0 0x7f0ecc1567f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:06.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.231+0000 7f0ec2ffd700 1 -- 172.21.15.60:0/876267128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ecc05bf30 msgr2=0x7f0ecc14dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:06.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.231+0000 7f0ec2ffd700 1 --2- 172.21.15.60:0/876267128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ecc05bf30 0x7f0ecc14dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:06.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.231+0000 7f0ec2ffd700 1 -- 172.21.15.60:0/876267128 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb400b180 con 0x7f0ecc05d030 2022-06-29T16:04:06.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.231+0000 7f0ecb7fe700 1 --2- 172.21.15.60:0/876267128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ecc05bf30 0x7f0ecc14dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:04:06.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.231+0000 7f0ec2ffd700 1 --2- 172.21.15.60:0/876267128 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0ecc05d030 0x7f0ecc156300 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f0ecc14e1b0 tx=0x7f0eb4014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:06.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.231+0000 7f0ec97fa700 1 -- 172.21.15.60:0/876267128 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0eb4016070 con 0x7f0ecc05d030 2022-06-29T16:04:06.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.231+0000 7f0ec97fa700 1 -- 172.21.15.60:0/876267128 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eb4013af0 con 0x7f0ecc05d030 2022-06-29T16:04:06.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.231+0000 7f0ec97fa700 1 -- 172.21.15.60:0/876267128 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0eb400d4f0 con 0x7f0ecc05d030 2022-06-29T16:04:06.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.231+0000 7f0ed0f25700 1 -- 172.21.15.60:0/876267128 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ecc155f10 con 0x7f0ecc05d030 2022-06-29T16:04:06.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.231+0000 7f0ed0f25700 1 -- 172.21.15.60:0/876267128 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ecc0ae9d0 con 0x7f0ecc05d030 2022-06-29T16:04:06.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.235+0000 7f0ed0f25700 1 -- 172.21.15.60:0/876267128 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ecc1560a0 con 0x7f0ecc05d030 2022-06-29T16:04:06.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.235+0000 7f0ec97fa700 1 -- 172.21.15.60:0/876267128 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0eb4024070 con 0x7f0ecc05d030 2022-06-29T16:04:06.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.235+0000 7f0ec97fa700 1 --2- 172.21.15.60:0/876267128 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0eb0053980 0x7f0eb0055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:06.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.235+0000 7f0ec97fa700 1 -- 172.21.15.60:0/876267128 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f0eb406c200 con 0x7f0ecc05d030 2022-06-29T16:04:06.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.235+0000 7f0ecb7fe700 1 --2- 172.21.15.60:0/876267128 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0eb0053980 0x7f0eb0055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:06.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.235+0000 7f0ecb7fe700 1 --2- 172.21.15.60:0/876267128 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0eb0053980 0x7f0eb0055e40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f0ebc00d000 tx=0x7f0ebc000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:06.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.243+0000 7f0ec97fa700 1 -- 172.21.15.60:0/876267128 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0eb40433b0 con 0x7f0ecc05d030 2022-06-29T16:04:06.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.371+0000 7f0ed0f25700 1 -- 172.21.15.60:0/876267128 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f0ecc0573d0 con 0x7f0ecc05d030 2022-06-29T16:04:06.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.375+0000 7f0ec97fa700 1 -- 172.21.15.60:0/876267128 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v67) v1 ==== 84+0+143 (secure 0 0 0) 0x7f0eb40469d0 con 0x7f0ecc05d030 2022-06-29T16:04:06.380 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:04:06.381 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2022-06-29T16:04:06.381 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:06.381 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2022-06-29T16:04:06.381 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": "device_health_metrics" 2022-06-29T16:04:06.382 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:04:06.382 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:04:06.382 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 2, 2022-06-29T16:04:06.383 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": "rbd" 2022-06-29T16:04:06.383 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:04:06.383 INFO:teuthology.orchestra.run.smithi060.stdout:] 2022-06-29T16:04:06.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.379+0000 7f0ed0f25700 1 -- 172.21.15.60:0/876267128 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0eb0053980 msgr2=0x7f0eb0055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:06.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.379+0000 7f0ed0f25700 1 --2- 172.21.15.60:0/876267128 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0eb0053980 0x7f0eb0055e40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f0ebc00d000 tx=0x7f0ebc000bc0).stop 2022-06-29T16:04:06.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.379+0000 7f0ed0f25700 1 -- 172.21.15.60:0/876267128 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0ecc05d030 msgr2=0x7f0ecc156300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:06.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.379+0000 7f0ed0f25700 1 --2- 172.21.15.60:0/876267128 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0ecc05d030 0x7f0ecc156300 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f0ecc14e1b0 tx=0x7f0eb4014000).stop 2022-06-29T16:04:06.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.379+0000 7f0ed0f25700 1 -- 172.21.15.60:0/876267128 shutdown_connections 2022-06-29T16:04:06.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.379+0000 7f0ed0f25700 1 --2- 172.21.15.60:0/876267128 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0eb0053980 0x7f0eb0055e40 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:06.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.379+0000 7f0ed0f25700 1 --2- 172.21.15.60:0/876267128 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ecc14a2b0 0x7f0ecc1567f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:06.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.379+0000 7f0ed0f25700 1 --2- 172.21.15.60:0/876267128 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0ecc05d030 0x7f0ecc156300 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:06.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.379+0000 7f0ed0f25700 1 --2- 172.21.15.60:0/876267128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ecc05bf30 0x7f0ecc14dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:06.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.379+0000 7f0ed0f25700 1 -- 172.21.15.60:0/876267128 >> 172.21.15.60:0/876267128 conn(0x7f0ecc064810 msgr2=0x7f0ecc149640 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:06.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.379+0000 7f0ed0f25700 1 -- 172.21.15.60:0/876267128 shutdown_connections 2022-06-29T16:04:06.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.379+0000 7f0ed0f25700 1 -- 172.21.15.60:0/876267128 wait complete. 2022-06-29T16:04:06.401 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-06-29T16:04:06.402 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-06-29T16:04:06.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.507+0000 7f2690c24700 1 Processor -- start 2022-06-29T16:04:06.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.507+0000 7f2690c24700 1 -- start start 2022-06-29T16:04:06.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.507+0000 7f2690c24700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f268c0b4010 0x7f268c1426b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:06.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.507+0000 7f2690c24700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f268c143a40 con 0x7f268c0ae7c0 2022-06-29T16:04:06.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.507+0000 7f2690c24700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f268c0aee60 con 0x7f268c0b46c0 2022-06-29T16:04:06.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.507+0000 7f2690c24700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f268c0af100 con 0x7f268c0b4010 2022-06-29T16:04:06.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.507+0000 7f268a59c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f268c0ae7c0 0x7f268c141fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57266/0 (socket says 172.21.15.60:57266) 2022-06-29T16:04:06.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.507+0000 7f268a59c700 1 -- 172.21.15.60:0/2907938804 learned_addr learned my addr 172.21.15.60:0/2907938804 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:04:06.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.507+0000 7f2683fff700 1 --2- 172.21.15.60:0/2907938804 >> v2:172.21.15.92:3300/0 conn(0x7f268c0b4010 0x7f268c1426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:06.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.507+0000 7f2689d9b700 1 -- 172.21.15.60:0/2907938804 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 286564511 0 0) 0x7f268c143a40 con 0x7f268c0ae7c0 2022-06-29T16:04:06.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.507+0000 7f2689d9b700 1 -- 172.21.15.60:0/2907938804 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f266c003260 con 0x7f268c0ae7c0 2022-06-29T16:04:06.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.507+0000 7f2689d9b700 1 -- 172.21.15.60:0/2907938804 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3766823191 0 0) 0x7f266c003260 con 0x7f268c0ae7c0 2022-06-29T16:04:06.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.507+0000 7f2689d9b700 1 -- 172.21.15.60:0/2907938804 >> v2:172.21.15.92:3300/0 conn(0x7f268c0b4010 msgr2=0x7f268c1426b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:06.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.507+0000 7f2689d9b700 1 --2- 172.21.15.60:0/2907938804 >> v2:172.21.15.92:3300/0 conn(0x7f268c0b4010 0x7f268c1426b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:06.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.507+0000 7f2689d9b700 1 -- 172.21.15.60:0/2907938804 >> v1:172.21.15.92:6789/0 conn(0x7f268c0b46c0 legacy=0x7f268c0b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:06.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.507+0000 7f2689d9b700 1 -- 172.21.15.60:0/2907938804 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f268c0aeba0 con 0x7f268c0ae7c0 2022-06-29T16:04:06.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.507+0000 7f2689d9b700 1 -- 172.21.15.60:0/2907938804 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f267c001660 con 0x7f268c0ae7c0 2022-06-29T16:04:06.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.507+0000 7f2689d9b700 1 -- 172.21.15.60:0/2907938804 >> v1:172.21.15.60:6789/0 conn(0x7f268c0ae7c0 legacy=0x7f268c141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:06.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.507+0000 7f2689d9b700 1 --2- 172.21.15.60:0/2907938804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f266c003750 0x7f266c003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:06.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.511+0000 7f2689d9b700 1 --2- 172.21.15.60:0/2907938804 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f266c004500 0x7f266c004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:06.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.511+0000 7f2689d9b700 1 --2- 172.21.15.60:0/2907938804 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f266c004e70 0x7f266c00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:06.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.511+0000 7f2689d9b700 1 -- 172.21.15.60:0/2907938804 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268c0af100 con 0x7f266c003750 2022-06-29T16:04:06.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.511+0000 7f2689d9b700 1 -- 172.21.15.60:0/2907938804 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266c00bd20 con 0x7f266c004500 2022-06-29T16:04:06.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.511+0000 7f2689d9b700 1 -- 172.21.15.60:0/2907938804 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f266c00bee0 con 0x7f266c004e70 2022-06-29T16:04:06.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.511+0000 7f268a59c700 1 --2- 172.21.15.60:0/2907938804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f266c003750 0x7f266c003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:06.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.511+0000 7f268ad9d700 1 --2- 172.21.15.60:0/2907938804 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f266c004e70 0x7f266c00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:06.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.511+0000 7f2683fff700 1 --2- 172.21.15.60:0/2907938804 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f266c004500 0x7f266c004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:06.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.511+0000 7f268a59c700 1 -- 172.21.15.60:0/2907938804 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f266c004e70 msgr2=0x7f266c00b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:06.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.511+0000 7f268a59c700 1 --2- 172.21.15.60:0/2907938804 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f266c004e70 0x7f266c00b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:06.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.511+0000 7f268a59c700 1 -- 172.21.15.60:0/2907938804 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f266c004500 msgr2=0x7f266c004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:06.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.511+0000 7f268a59c700 1 --2- 172.21.15.60:0/2907938804 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f266c004500 0x7f266c004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:06.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.511+0000 7f268ad9d700 1 -- 172.21.15.60:0/2907938804 reap_dead start 2022-06-29T16:04:06.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.511+0000 7f268a59c700 1 -- 172.21.15.60:0/2907938804 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f266c003260 con 0x7f266c003750 2022-06-29T16:04:06.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.511+0000 7f268a59c700 1 --2- 172.21.15.60:0/2907938804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f266c003750 0x7f266c003b70 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0x7f267c001370 tx=0x7f267c001e10).ready entity=mon.0 client_cookie=fb93bf64d3a9d663 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:06.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.511+0000 7f2689d9b700 1 -- 172.21.15.60:0/2907938804 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f267c0071a0 con 0x7f266c003750 2022-06-29T16:04:06.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.511+0000 7f2689d9b700 1 -- 172.21.15.60:0/2907938804 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f267c007300 con 0x7f266c003750 2022-06-29T16:04:06.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.511+0000 7f2689d9b700 1 -- 172.21.15.60:0/2907938804 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f267c00c8b0 con 0x7f266c003750 2022-06-29T16:04:06.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.511+0000 7f2690c24700 1 -- 172.21.15.60:0/2907938804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f266c003750 msgr2=0x7f266c003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:06.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.511+0000 7f2690c24700 1 --2- 172.21.15.60:0/2907938804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f266c003750 0x7f266c003b70 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0x7f267c001370 tx=0x7f267c001e10).stop 2022-06-29T16:04:06.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.511+0000 7f2690c24700 1 -- 172.21.15.60:0/2907938804 shutdown_connections 2022-06-29T16:04:06.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.511+0000 7f2690c24700 1 --2- 172.21.15.60:0/2907938804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f266c003750 0x7f266c003b70 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:06.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.511+0000 7f2690c24700 1 -- 172.21.15.60:0/2907938804 >> 172.21.15.60:0/2907938804 conn(0x7f268c0b9340 msgr2=0x7f268c0b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:06.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.511+0000 7f2690c24700 1 -- 172.21.15.60:0/2907938804 shutdown_connections 2022-06-29T16:04:06.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.511+0000 7f2690c24700 1 -- 172.21.15.60:0/2907938804 wait complete. 2022-06-29T16:04:06.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.511+0000 7f2690c24700 1 Processor -- start 2022-06-29T16:04:06.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.511+0000 7f2690c24700 1 -- start start 2022-06-29T16:04:06.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.515+0000 7f2690c24700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f266c003750 0x7f268c145290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:06.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.515+0000 7f2690c24700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f268c112530 0x7f268c1443a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:06.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.515+0000 7f2690c24700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f268c0b28d0 0x7f268c0b2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:06.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.515+0000 7f2690c24700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268c0b11e0 con 0x7f268c0b28d0 2022-06-29T16:04:06.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.515+0000 7f2690c24700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268c146320 con 0x7f268c112530 2022-06-29T16:04:06.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.515+0000 7f2690c24700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f268c0b0230 con 0x7f266c003750 2022-06-29T16:04:06.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.515+0000 7f268ad9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f268c0b28d0 0x7f268c0b2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:06.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.515+0000 7f268ad9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f268c0b28d0 0x7f268c0b2d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49196/0 (socket says 172.21.15.60:49196) 2022-06-29T16:04:06.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.515+0000 7f268ad9d700 1 -- 172.21.15.60:0/3405436085 learned_addr learned my addr 172.21.15.60:0/3405436085 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:06.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.515+0000 7f268a59c700 1 --2- 172.21.15.60:0/3405436085 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f266c003750 0x7f268c145290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:06.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.515+0000 7f2683fff700 1 --2- 172.21.15.60:0/3405436085 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f268c112530 0x7f268c1443a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:06.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.515+0000 7f268ad9d700 1 -- 172.21.15.60:0/3405436085 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f266c003750 msgr2=0x7f268c145290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:06.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.515+0000 7f268ad9d700 1 --2- 172.21.15.60:0/3405436085 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f266c003750 0x7f268c145290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:06.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.515+0000 7f268ad9d700 1 -- 172.21.15.60:0/3405436085 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f268c112530 msgr2=0x7f268c1443a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:06.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.515+0000 7f268ad9d700 1 --2- 172.21.15.60:0/3405436085 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f268c112530 0x7f268c1443a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:06.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.515+0000 7f268ad9d700 1 -- 172.21.15.60:0/3405436085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f268c0b04d0 con 0x7f268c0b28d0 2022-06-29T16:04:06.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.515+0000 7f268ad9d700 1 --2- 172.21.15.60:0/3405436085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f268c0b28d0 0x7f268c0b2d80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0x7f2684000b60 tx=0x7f2684007d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:06.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.515+0000 7f26837fe700 1 -- 172.21.15.60:0/3405436085 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2684006c90 con 0x7f268c0b28d0 2022-06-29T16:04:06.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.515+0000 7f26837fe700 1 -- 172.21.15.60:0/3405436085 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2684006df0 con 0x7f268c0b28d0 2022-06-29T16:04:06.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.515+0000 7f2690c24700 1 -- 172.21.15.60:0/3405436085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f268c0b3270 con 0x7f268c0b28d0 2022-06-29T16:04:06.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.515+0000 7f26837fe700 1 -- 172.21.15.60:0/3405436085 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2684007220 con 0x7f268c0b28d0 2022-06-29T16:04:06.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.515+0000 7f2690c24700 1 -- 172.21.15.60:0/3405436085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f268c146d60 con 0x7f268c0b28d0 2022-06-29T16:04:06.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.515+0000 7f2690c24700 1 -- 172.21.15.60:0/3405436085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f268c0573d0 con 0x7f268c0b28d0 2022-06-29T16:04:06.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.515+0000 7f26837fe700 1 -- 172.21.15.60:0/3405436085 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2684007380 con 0x7f268c0b28d0 2022-06-29T16:04:06.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.515+0000 7f26837fe700 1 --2- 172.21.15.60:0/3405436085 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f26700539e0 0x7f2670055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:06.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.515+0000 7f26837fe700 1 -- 172.21.15.60:0/3405436085 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f268406ba40 con 0x7f268c0b28d0 2022-06-29T16:04:06.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.515+0000 7f268a59c700 1 --2- 172.21.15.60:0/3405436085 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f26700539e0 0x7f2670055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:06.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.519+0000 7f268a59c700 1 --2- 172.21.15.60:0/3405436085 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f26700539e0 0x7f2670055ea0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f267c017000 tx=0x7f267c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:06.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.523+0000 7f26837fe700 1 -- 172.21.15.60:0/3405436085 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2684042c90 con 0x7f268c0b28d0 2022-06-29T16:04:06.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.647+0000 7f2690c24700 1 -- 172.21.15.60:0/3405436085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f268c05fdb0 con 0x7f268c0b28d0 2022-06-29T16:04:06.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.951+0000 7f26837fe700 1 -- 172.21.15.60:0/3405436085 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v68) v1 ==== 133+0+0 (secure 0 0 0) 0x7f26840462b0 con 0x7f268c0b28d0 2022-06-29T16:04:06.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.955+0000 7f2690c24700 1 -- 172.21.15.60:0/3405436085 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f26700539e0 msgr2=0x7f2670055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:06.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.955+0000 7f2690c24700 1 --2- 172.21.15.60:0/3405436085 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f26700539e0 0x7f2670055ea0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f267c017000 tx=0x7f267c015000).stop 2022-06-29T16:04:06.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.955+0000 7f2690c24700 1 -- 172.21.15.60:0/3405436085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f268c0b28d0 msgr2=0x7f268c0b2d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:06.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.955+0000 7f2690c24700 1 --2- 172.21.15.60:0/3405436085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f268c0b28d0 0x7f268c0b2d80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0x7f2684000b60 tx=0x7f2684007d80).stop 2022-06-29T16:04:06.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.955+0000 7f2690c24700 1 -- 172.21.15.60:0/3405436085 shutdown_connections 2022-06-29T16:04:06.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.955+0000 7f2690c24700 1 --2- 172.21.15.60:0/3405436085 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f26700539e0 0x7f2670055ea0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:06.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.955+0000 7f2690c24700 1 --2- 172.21.15.60:0/3405436085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f268c0b28d0 0x7f268c0b2d80 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:06.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.955+0000 7f2690c24700 1 --2- 172.21.15.60:0/3405436085 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f268c112530 0x7f268c1443a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:06.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.955+0000 7f2690c24700 1 --2- 172.21.15.60:0/3405436085 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f266c003750 0x7f268c145290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:06.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.955+0000 7f2690c24700 1 -- 172.21.15.60:0/3405436085 >> 172.21.15.60:0/3405436085 conn(0x7f268c0b9340 msgr2=0x7f268c142380 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:06.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.959+0000 7f2690c24700 1 -- 172.21.15.60:0/3405436085 shutdown_connections 2022-06-29T16:04:06.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:06.959+0000 7f2690c24700 1 -- 172.21.15.60:0/3405436085 wait complete. 2022-06-29T16:04:06.967 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2022-06-29T16:04:06.979 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-06-29T16:04:07.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.091+0000 7f2a534a3700 1 Processor -- start 2022-06-29T16:04:07.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.095+0000 7f2a534a3700 1 -- start start 2022-06-29T16:04:07.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.095+0000 7f2a534a3700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2a4c0b2130 0x7f2a4c146100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:07.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.095+0000 7f2a534a3700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2a4c0aed00 0x7f2a4c1465f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:07.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.095+0000 7f2a534a3700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a4c05d450 con 0x7f2a4c05d070 2022-06-29T16:04:07.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.095+0000 7f2a534a3700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a4c0adf90 con 0x7f2a4c0aed00 2022-06-29T16:04:07.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.095+0000 7f2a534a3700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a4c0b25d0 con 0x7f2a4c0b2130 2022-06-29T16:04:07.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.095+0000 7f2a52ca2700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2a4c0aed00 0x7f2a4c1465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:07.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.095+0000 7f2a52ca2700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2a4c0aed00 0x7f2a4c1465f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49204/0 (socket says 172.21.15.60:49204) 2022-06-29T16:04:07.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.095+0000 7f2a52ca2700 1 -- 172.21.15.60:0/2696078592 learned_addr learned my addr 172.21.15.60:0/2696078592 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:07.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.095+0000 7f2a51ca0700 1 --2- 172.21.15.60:0/2696078592 >> v2:172.21.15.92:3300/0 conn(0x7f2a4c0b2130 0x7f2a4c146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:07.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.095+0000 7f2a52ca2700 1 -- 172.21.15.60:0/2696078592 >> v2:172.21.15.92:3300/0 conn(0x7f2a4c0b2130 msgr2=0x7f2a4c146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:07.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.095+0000 7f2a52ca2700 1 --2- 172.21.15.60:0/2696078592 >> v2:172.21.15.92:3300/0 conn(0x7f2a4c0b2130 0x7f2a4c146100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:07.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.095+0000 7f2a52ca2700 1 -- 172.21.15.60:0/2696078592 >> v1:172.21.15.92:6789/0 conn(0x7f2a4c05d070 legacy=0x7f2a4c14aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:07.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.095+0000 7f2a52ca2700 1 -- 172.21.15.60:0/2696078592 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a4c14a350 con 0x7f2a4c0aed00 2022-06-29T16:04:07.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.095+0000 7f2a52ca2700 1 --2- 172.21.15.60:0/2696078592 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a4c0aed00 0x7f2a4c1465f0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7f2a440032f0 tx=0x7f2a44009290).ready entity=mon.0 client_cookie=c4905fffb2ded878 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:07.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.095+0000 7f2a5149f700 1 -- 172.21.15.60:0/2696078592 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a4400b870 con 0x7f2a4c0aed00 2022-06-29T16:04:07.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.095+0000 7f2a5149f700 1 -- 172.21.15.60:0/2696078592 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a4400b9d0 con 0x7f2a4c0aed00 2022-06-29T16:04:07.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.095+0000 7f2a5149f700 1 -- 172.21.15.60:0/2696078592 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a4400bb40 con 0x7f2a4c0aed00 2022-06-29T16:04:07.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.095+0000 7f2a534a3700 1 -- 172.21.15.60:0/2696078592 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a4c0aed00 msgr2=0x7f2a4c1465f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:07.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.095+0000 7f2a534a3700 1 --2- 172.21.15.60:0/2696078592 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a4c0aed00 0x7f2a4c1465f0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7f2a440032f0 tx=0x7f2a44009290).stop 2022-06-29T16:04:07.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.095+0000 7f2a534a3700 1 -- 172.21.15.60:0/2696078592 shutdown_connections 2022-06-29T16:04:07.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.095+0000 7f2a534a3700 1 --2- 172.21.15.60:0/2696078592 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a4c0aed00 0x7f2a4c1465f0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:07.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.095+0000 7f2a534a3700 1 --2- 172.21.15.60:0/2696078592 >> v2:172.21.15.92:3300/0 conn(0x7f2a4c0b2130 0x7f2a4c146100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.095+0000 7f2a534a3700 1 -- 172.21.15.60:0/2696078592 >> 172.21.15.60:0/2696078592 conn(0x7f2a4c0b5890 msgr2=0x7f2a4c0b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.095+0000 7f2a534a3700 1 -- 172.21.15.60:0/2696078592 shutdown_connections 2022-06-29T16:04:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.095+0000 7f2a534a3700 1 -- 172.21.15.60:0/2696078592 wait complete. 2022-06-29T16:04:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.099+0000 7f2a534a3700 1 Processor -- start 2022-06-29T16:04:07.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.099+0000 7f2a534a3700 1 -- start start 2022-06-29T16:04:07.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.099+0000 7f2a534a3700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a4c05d070 0x7f2a4c116f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:07.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.099+0000 7f2a534a3700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a4c0aed00 0x7f2a4c1561d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:07.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.099+0000 7f2a534a3700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2a4c0b2130 0x7f2a4c1566c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:07.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.099+0000 7f2a534a3700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4c142f30 con 0x7f2a4c05d070 2022-06-29T16:04:07.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.099+0000 7f2a534a3700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4c156bb0 con 0x7f2a4c0aed00 2022-06-29T16:04:07.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.099+0000 7f2a534a3700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4c1195b0 con 0x7f2a4c0b2130 2022-06-29T16:04:07.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.099+0000 7f2a524a1700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a4c05d070 0x7f2a4c116f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:07.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.099+0000 7f2a524a1700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a4c05d070 0x7f2a4c116f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49206/0 (socket says 172.21.15.60:49206) 2022-06-29T16:04:07.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.099+0000 7f2a524a1700 1 -- 172.21.15.60:0/2117649728 learned_addr learned my addr 172.21.15.60:0/2117649728 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:07.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.099+0000 7f2a52ca2700 1 --2- 172.21.15.60:0/2117649728 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2a4c0b2130 0x7f2a4c1566c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:07.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.099+0000 7f2a51ca0700 1 --2- 172.21.15.60:0/2117649728 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a4c0aed00 0x7f2a4c1561d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:07.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.099+0000 7f2a524a1700 1 -- 172.21.15.60:0/2117649728 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2a4c0b2130 msgr2=0x7f2a4c1566c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:07.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.099+0000 7f2a524a1700 1 --2- 172.21.15.60:0/2117649728 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2a4c0b2130 0x7f2a4c1566c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:07.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.099+0000 7f2a524a1700 1 -- 172.21.15.60:0/2117649728 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a4c0aed00 msgr2=0x7f2a4c1561d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:07.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.099+0000 7f2a524a1700 1 --2- 172.21.15.60:0/2117649728 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a4c0aed00 0x7f2a4c1561d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:07.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.099+0000 7f2a524a1700 1 -- 172.21.15.60:0/2117649728 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a44009530 con 0x7f2a4c05d070 2022-06-29T16:04:07.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.099+0000 7f2a52ca2700 1 --2- 172.21.15.60:0/2117649728 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2a4c0b2130 0x7f2a4c1566c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:04:07.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.099+0000 7f2a51ca0700 1 --2- 172.21.15.60:0/2117649728 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a4c0aed00 0x7f2a4c1561d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:04:07.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.099+0000 7f2a524a1700 1 --2- 172.21.15.60:0/2117649728 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a4c05d070 0x7f2a4c116f60 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0x7f2a4c1186c0 tx=0x7f2a3c007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:07.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.099+0000 7f2a3b7fe700 1 -- 172.21.15.60:0/2117649728 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a3c00ce20 con 0x7f2a4c05d070 2022-06-29T16:04:07.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.099+0000 7f2a3b7fe700 1 -- 172.21.15.60:0/2117649728 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a3c007030 con 0x7f2a4c05d070 2022-06-29T16:04:07.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.099+0000 7f2a534a3700 1 -- 172.21.15.60:0/2117649728 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a4c116820 con 0x7f2a4c05d070 2022-06-29T16:04:07.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.103+0000 7f2a3b7fe700 1 -- 172.21.15.60:0/2117649728 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a3c007390 con 0x7f2a4c05d070 2022-06-29T16:04:07.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.103+0000 7f2a534a3700 1 -- 172.21.15.60:0/2117649728 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a4c066d20 con 0x7f2a4c05d070 2022-06-29T16:04:07.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.103+0000 7f2a3b7fe700 1 -- 172.21.15.60:0/2117649728 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2a3c010030 con 0x7f2a4c05d070 2022-06-29T16:04:07.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.103+0000 7f2a534a3700 1 -- 172.21.15.60:0/2117649728 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a4c14a5f0 con 0x7f2a4c05d070 2022-06-29T16:04:07.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.103+0000 7f2a3b7fe700 1 --2- 172.21.15.60:0/2117649728 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2a2c053a70 0x7f2a2c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:07.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.103+0000 7f2a3b7fe700 1 -- 172.21.15.60:0/2117649728 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f2a3c06b890 con 0x7f2a4c05d070 2022-06-29T16:04:07.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.103+0000 7f2a51ca0700 1 --2- 172.21.15.60:0/2117649728 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2a2c053a70 0x7f2a2c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:07.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.103+0000 7f2a51ca0700 1 --2- 172.21.15.60:0/2117649728 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2a2c053a70 0x7f2a2c055f30 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f2a4800c000 tx=0x7f2a48008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:07.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.115+0000 7f2a3b7fe700 1 -- 172.21.15.60:0/2117649728 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2a3c042920 con 0x7f2a4c05d070 2022-06-29T16:04:07.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.247+0000 7f2a534a3700 1 -- 172.21.15.60:0/2117649728 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f2a4c1169b0 con 0x7f2a4c05d070 2022-06-29T16:04:07.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.947+0000 7f2a3b7fe700 1 -- 172.21.15.60:0/2117649728 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v69) v1 ==== 125+0+0 (secure 0 0 0) 0x7f2a3c045f40 con 0x7f2a4c05d070 2022-06-29T16:04:07.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.955+0000 7f2a534a3700 1 -- 172.21.15.60:0/2117649728 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2a2c053a70 msgr2=0x7f2a2c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:07.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.955+0000 7f2a534a3700 1 --2- 172.21.15.60:0/2117649728 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2a2c053a70 0x7f2a2c055f30 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f2a4800c000 tx=0x7f2a48008040).stop 2022-06-29T16:04:07.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.955+0000 7f2a534a3700 1 -- 172.21.15.60:0/2117649728 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a4c05d070 msgr2=0x7f2a4c116f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:07.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.955+0000 7f2a534a3700 1 --2- 172.21.15.60:0/2117649728 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a4c05d070 0x7f2a4c116f60 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0x7f2a4c1186c0 tx=0x7f2a3c007c00).stop 2022-06-29T16:04:07.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.959+0000 7f2a534a3700 1 -- 172.21.15.60:0/2117649728 shutdown_connections 2022-06-29T16:04:07.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.959+0000 7f2a534a3700 1 --2- 172.21.15.60:0/2117649728 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2a2c053a70 0x7f2a2c055f30 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:07.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.959+0000 7f2a534a3700 1 --2- 172.21.15.60:0/2117649728 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2a4c0b2130 0x7f2a4c1566c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:07.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.959+0000 7f2a534a3700 1 --2- 172.21.15.60:0/2117649728 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a4c0aed00 0x7f2a4c1561d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:07.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.959+0000 7f2a534a3700 1 --2- 172.21.15.60:0/2117649728 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2a4c05d070 0x7f2a4c116f60 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:07.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.959+0000 7f2a534a3700 1 -- 172.21.15.60:0/2117649728 >> 172.21.15.60:0/2117649728 conn(0x7f2a4c0b5890 msgr2=0x7f2a4c0aca20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:07.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.959+0000 7f2a534a3700 1 -- 172.21.15.60:0/2117649728 shutdown_connections 2022-06-29T16:04:07.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:07.959+0000 7f2a534a3700 1 -- 172.21.15.60:0/2117649728 wait complete. 2022-06-29T16:04:07.967 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2022-06-29T16:04:07.977 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-06-29T16:04:08.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.079+0000 7ff0b97de700 1 Processor -- start 2022-06-29T16:04:08.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.079+0000 7ff0b97de700 1 -- start start 2022-06-29T16:04:08.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.079+0000 7ff0b97de700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0b405c730 0x7ff0b4143f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:08.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.079+0000 7ff0b97de700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff0b40605d0 0x7ff0b41443f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:08.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.079+0000 7ff0b97de700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff0b414a2e0 0x7ff0b414a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:08.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.079+0000 7ff0b97de700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0b4067d60 con 0x7ff0b40605d0 2022-06-29T16:04:08.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.079+0000 7ff0b97de700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0b40b2780 con 0x7ff0b414a2e0 2022-06-29T16:04:08.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.079+0000 7ff0b97de700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b4065cc0 con 0x7ff0b405c730 2022-06-29T16:04:08.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.079+0000 7ff0b37fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff0b40605d0 0x7ff0b41443f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:08.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.079+0000 7ff0b37fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff0b40605d0 0x7ff0b41443f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49212/0 (socket says 172.21.15.60:49212) 2022-06-29T16:04:08.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.079+0000 7ff0b37fe700 1 -- 172.21.15.60:0/3456529141 learned_addr learned my addr 172.21.15.60:0/3456529141 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:08.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.079+0000 7ff0b3fff700 1 --2- 172.21.15.60:0/3456529141 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0b405c730 0x7ff0b4143f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:08.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.079+0000 7ff0b8fdd700 1 --2- 172.21.15.60:0/3456529141 >> v2:172.21.15.92:3300/0 conn(0x7ff0b414a2e0 0x7ff0b414a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:08.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.079+0000 7ff0b37fe700 1 -- 172.21.15.60:0/3456529141 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0b405c730 msgr2=0x7ff0b4143f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:08.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.079+0000 7ff0b37fe700 1 --2- 172.21.15.60:0/3456529141 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0b405c730 0x7ff0b4143f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:08.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.079+0000 7ff0b37fe700 1 -- 172.21.15.60:0/3456529141 >> v2:172.21.15.92:3300/0 conn(0x7ff0b414a2e0 msgr2=0x7ff0b414a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:08.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.079+0000 7ff0b37fe700 1 --2- 172.21.15.60:0/3456529141 >> v2:172.21.15.92:3300/0 conn(0x7ff0b414a2e0 0x7ff0b414a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:08.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.079+0000 7ff0b37fe700 1 -- 172.21.15.60:0/3456529141 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b4065f20 con 0x7ff0b40605d0 2022-06-29T16:04:08.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.079+0000 7ff0b8fdd700 1 --2- 172.21.15.60:0/3456529141 >> v2:172.21.15.92:3300/0 conn(0x7ff0b414a2e0 0x7ff0b414a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:04:08.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.079+0000 7ff0b37fe700 1 --2- 172.21.15.60:0/3456529141 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0b40605d0 0x7ff0b41443f0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0x7ff0b4067ea0 tx=0x7ff0a8006eb0).ready entity=mon.0 client_cookie=db176de1d203da8d server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:08.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.079+0000 7ff0b2ffd700 1 -- 172.21.15.60:0/3456529141 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0a800d570 con 0x7ff0b40605d0 2022-06-29T16:04:08.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.079+0000 7ff0b2ffd700 1 -- 172.21.15.60:0/3456529141 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a8000e90 con 0x7ff0b40605d0 2022-06-29T16:04:08.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.079+0000 7ff0b2ffd700 1 -- 172.21.15.60:0/3456529141 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0a800d900 con 0x7ff0b40605d0 2022-06-29T16:04:08.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.079+0000 7ff0b97de700 1 -- 172.21.15.60:0/3456529141 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0b40605d0 msgr2=0x7ff0b41443f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:08.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.079+0000 7ff0b97de700 1 --2- 172.21.15.60:0/3456529141 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0b40605d0 0x7ff0b41443f0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0x7ff0b4067ea0 tx=0x7ff0a8006eb0).stop 2022-06-29T16:04:08.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.083+0000 7ff0b97de700 1 -- 172.21.15.60:0/3456529141 shutdown_connections 2022-06-29T16:04:08.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.083+0000 7ff0b97de700 1 --2- 172.21.15.60:0/3456529141 >> v2:172.21.15.92:3300/0 conn(0x7ff0b414a2e0 0x7ff0b414a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:08.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.083+0000 7ff0b97de700 1 --2- 172.21.15.60:0/3456529141 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0b40605d0 0x7ff0b41443f0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:08.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.083+0000 7ff0b97de700 1 --2- 172.21.15.60:0/3456529141 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0b405c730 0x7ff0b4143f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:08.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.083+0000 7ff0b97de700 1 -- 172.21.15.60:0/3456529141 >> 172.21.15.60:0/3456529141 conn(0x7ff0b40b5e70 msgr2=0x7ff0b40b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:08.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.083+0000 7ff0b97de700 1 -- 172.21.15.60:0/3456529141 shutdown_connections 2022-06-29T16:04:08.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.083+0000 7ff0b97de700 1 -- 172.21.15.60:0/3456529141 wait complete. 2022-06-29T16:04:08.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.083+0000 7ff0b97de700 1 Processor -- start 2022-06-29T16:04:08.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.083+0000 7ff0b97de700 1 -- start start 2022-06-29T16:04:08.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.083+0000 7ff0b97de700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0b405c730 0x7ff0b4146360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:08.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.083+0000 7ff0b97de700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff0b40605d0 0x7ff0b40aeec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:08.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.083+0000 7ff0b97de700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0b414a2e0 0x7ff0b40af3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:08.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.083+0000 7ff0b97de700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b4145600 con 0x7ff0b405c730 2022-06-29T16:04:08.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.083+0000 7ff0b97de700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b40af8a0 con 0x7ff0b40605d0 2022-06-29T16:04:08.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.083+0000 7ff0b97de700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b41438f0 con 0x7ff0b414a2e0 2022-06-29T16:04:08.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.083+0000 7ff0b3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0b405c730 0x7ff0b4146360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:08.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.083+0000 7ff0b3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0b405c730 0x7ff0b4146360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49218/0 (socket says 172.21.15.60:49218) 2022-06-29T16:04:08.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.083+0000 7ff0b3fff700 1 -- 172.21.15.60:0/2136768716 learned_addr learned my addr 172.21.15.60:0/2136768716 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:08.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.083+0000 7ff0b37fe700 1 --2- 172.21.15.60:0/2136768716 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff0b40605d0 0x7ff0b40aeec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:08.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.083+0000 7ff0b8fdd700 1 --2- 172.21.15.60:0/2136768716 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0b414a2e0 0x7ff0b40af3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:08.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.087+0000 7ff0b8fdd700 1 -- 172.21.15.60:0/2136768716 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff0b40605d0 msgr2=0x7ff0b40aeec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:08.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.087+0000 7ff0b8fdd700 1 --2- 172.21.15.60:0/2136768716 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff0b40605d0 0x7ff0b40aeec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:08.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.087+0000 7ff0b8fdd700 1 -- 172.21.15.60:0/2136768716 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0b405c730 msgr2=0x7ff0b4146360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:08.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.087+0000 7ff0b8fdd700 1 --2- 172.21.15.60:0/2136768716 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0b405c730 0x7ff0b4146360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:08.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.087+0000 7ff0b8fdd700 1 -- 172.21.15.60:0/2136768716 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a800b1e0 con 0x7ff0b414a2e0 2022-06-29T16:04:08.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.087+0000 7ff0b3fff700 1 --2- 172.21.15.60:0/2136768716 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0b405c730 0x7ff0b4146360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:04:08.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.087+0000 7ff0b8fdd700 1 --2- 172.21.15.60:0/2136768716 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0b414a2e0 0x7ff0b40af3b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7ff0ac00a000 tx=0x7ff0ac008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:08.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.087+0000 7ff0b17fa700 1 -- 172.21.15.60:0/2136768716 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0ac00f710 con 0x7ff0b414a2e0 2022-06-29T16:04:08.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.087+0000 7ff0b17fa700 1 -- 172.21.15.60:0/2136768716 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0ac00f870 con 0x7ff0b414a2e0 2022-06-29T16:04:08.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.087+0000 7ff0b97de700 1 -- 172.21.15.60:0/2136768716 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b4143650 con 0x7ff0b414a2e0 2022-06-29T16:04:08.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.087+0000 7ff0b17fa700 1 -- 172.21.15.60:0/2136768716 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0ac00fbd0 con 0x7ff0b414a2e0 2022-06-29T16:04:08.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.087+0000 7ff0b97de700 1 -- 172.21.15.60:0/2136768716 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b40b0980 con 0x7ff0b414a2e0 2022-06-29T16:04:08.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.087+0000 7ff0b97de700 1 -- 172.21.15.60:0/2136768716 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0b40573d0 con 0x7ff0b414a2e0 2022-06-29T16:04:08.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.087+0000 7ff0b17fa700 1 -- 172.21.15.60:0/2136768716 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff0ac01a030 con 0x7ff0b414a2e0 2022-06-29T16:04:08.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.087+0000 7ff0b17fa700 1 --2- 172.21.15.60:0/2136768716 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff094053aa0 0x7ff094055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:08.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.087+0000 7ff0b17fa700 1 -- 172.21.15.60:0/2136768716 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5957+0+0 (secure 0 0 0) 0x7ff0ac06bdb0 con 0x7ff0b414a2e0 2022-06-29T16:04:08.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.091+0000 7ff0b3fff700 1 --2- 172.21.15.60:0/2136768716 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff094053aa0 0x7ff094055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:08.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.091+0000 7ff0b3fff700 1 --2- 172.21.15.60:0/2136768716 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff094053aa0 0x7ff094055f60 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7ff0a400b000 tx=0x7ff0a4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:08.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.099+0000 7ff0b17fa700 1 -- 172.21.15.60:0/2136768716 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff0ac070070 con 0x7ff0b414a2e0 2022-06-29T16:04:08.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.267+0000 7ff0b97de700 1 -- 172.21.15.60:0/2136768716 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7ff0b405d990 con 0x7ff0b414a2e0 2022-06-29T16:04:08.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.283+0000 7ff0b17fa700 1 -- 172.21.15.60:0/2136768716 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 13 and data pool 14 v29) v1 ==== 189+0+0 (secure 0 0 0) 0x7ff0ac042bd0 con 0x7ff0b414a2e0 2022-06-29T16:04:08.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.287+0000 7ff0b97de700 1 -- 172.21.15.60:0/2136768716 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff094053aa0 msgr2=0x7ff094055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:08.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.287+0000 7ff0b97de700 1 --2- 172.21.15.60:0/2136768716 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff094053aa0 0x7ff094055f60 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7ff0a400b000 tx=0x7ff0a4006df0).stop 2022-06-29T16:04:08.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.287+0000 7ff0b97de700 1 -- 172.21.15.60:0/2136768716 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0b414a2e0 msgr2=0x7ff0b40af3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:08.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.287+0000 7ff0b97de700 1 --2- 172.21.15.60:0/2136768716 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0b414a2e0 0x7ff0b40af3b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7ff0ac00a000 tx=0x7ff0ac008040).stop 2022-06-29T16:04:08.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.287+0000 7ff0b97de700 1 -- 172.21.15.60:0/2136768716 shutdown_connections 2022-06-29T16:04:08.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.287+0000 7ff0b97de700 1 --2- 172.21.15.60:0/2136768716 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff094053aa0 0x7ff094055f60 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:08.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.287+0000 7ff0b97de700 1 --2- 172.21.15.60:0/2136768716 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff0b414a2e0 0x7ff0b40af3b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:08.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.287+0000 7ff0b97de700 1 --2- 172.21.15.60:0/2136768716 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff0b40605d0 0x7ff0b40aeec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:08.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.287+0000 7ff0b97de700 1 --2- 172.21.15.60:0/2136768716 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff0b405c730 0x7ff0b4146360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:08.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.287+0000 7ff0b97de700 1 -- 172.21.15.60:0/2136768716 >> 172.21.15.60:0/2136768716 conn(0x7ff0b40b5e70 msgr2=0x7ff0b4064f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:08.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.287+0000 7ff0b97de700 1 -- 172.21.15.60:0/2136768716 shutdown_connections 2022-06-29T16:04:08.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.287+0000 7ff0b97de700 1 -- 172.21.15.60:0/2136768716 wait complete. 2022-06-29T16:04:08.297 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 13 and data pool 14 2022-06-29T16:04:08.306 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:04:08.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.395+0000 7f3f7723f700 1 Processor -- start 2022-06-29T16:04:08.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.395+0000 7f3f7723f700 1 -- start start 2022-06-29T16:04:08.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.395+0000 7f3f7723f700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3f700af970 0x7f3f70140390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:08.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.395+0000 7f3f7723f700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f70060780 con 0x7f3f700af4e0 2022-06-29T16:04:08.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.395+0000 7f3f7723f700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f700604e0 con 0x7f3f700b6030 2022-06-29T16:04:08.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.395+0000 7f3f7723f700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f70062db0 con 0x7f3f700af970 2022-06-29T16:04:08.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.395+0000 7f3f6ffff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f3f700af4e0 0x7f3f7013fc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57312/0 (socket says 172.21.15.60:57312) 2022-06-29T16:04:08.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.395+0000 7f3f6ffff700 1 -- 172.21.15.60:0/1559672303 learned_addr learned my addr 172.21.15.60:0/1559672303 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:04:08.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.395+0000 7f3f757dc700 1 --2- 172.21.15.60:0/1559672303 >> v2:172.21.15.92:3300/0 conn(0x7f3f700af970 0x7f3f70140390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:08.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.395+0000 7f3f6f7fe700 1 -- 172.21.15.60:0/1559672303 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 288727382 0 0) 0x7f3f70060780 con 0x7f3f700af4e0 2022-06-29T16:04:08.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.395+0000 7f3f6f7fe700 1 -- 172.21.15.60:0/1559672303 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f50003260 con 0x7f3f700af4e0 2022-06-29T16:04:08.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.395+0000 7f3f6f7fe700 1 -- 172.21.15.60:0/1559672303 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4038590259 0 0) 0x7f3f700604e0 con 0x7f3f700b6030 2022-06-29T16:04:08.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.395+0000 7f3f6f7fe700 1 -- 172.21.15.60:0/1559672303 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f70060780 con 0x7f3f700b6030 2022-06-29T16:04:08.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.395+0000 7f3f6f7fe700 1 -- 172.21.15.60:0/1559672303 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3533834863 0 0) 0x7f3f50003260 con 0x7f3f700af4e0 2022-06-29T16:04:08.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.395+0000 7f3f6f7fe700 1 -- 172.21.15.60:0/1559672303 >> v2:172.21.15.92:3300/0 conn(0x7f3f700af970 msgr2=0x7f3f70140390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:08.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.395+0000 7f3f6f7fe700 1 --2- 172.21.15.60:0/1559672303 >> v2:172.21.15.92:3300/0 conn(0x7f3f700af970 0x7f3f70140390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:08.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.395+0000 7f3f6f7fe700 1 -- 172.21.15.60:0/1559672303 >> v1:172.21.15.92:6789/0 conn(0x7f3f700b6030 legacy=0x7f3f70143e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:08.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.395+0000 7f3f6f7fe700 1 -- 172.21.15.60:0/1559672303 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f70064130 con 0x7f3f700af4e0 2022-06-29T16:04:08.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.395+0000 7f3f6f7fe700 1 -- 172.21.15.60:0/1559672303 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f3f58001660 con 0x7f3f700af4e0 2022-06-29T16:04:08.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.395+0000 7f3f6f7fe700 1 -- 172.21.15.60:0/1559672303 >> v1:172.21.15.60:6789/0 conn(0x7f3f700af4e0 legacy=0x7f3f7013fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:08.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.395+0000 7f3f6f7fe700 1 --2- 172.21.15.60:0/1559672303 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f500038e0 0x7f3f50003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:08.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.395+0000 7f3f6f7fe700 1 --2- 172.21.15.60:0/1559672303 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f50004500 0x7f3f50004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:08.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.395+0000 7f3f6f7fe700 1 --2- 172.21.15.60:0/1559672303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f50004e70 0x7f3f5000b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:08.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.395+0000 7f3f6f7fe700 1 -- 172.21.15.60:0/1559672303 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5000bcf0 con 0x7f3f50004e70 2022-06-29T16:04:08.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.395+0000 7f3f6f7fe700 1 -- 172.21.15.60:0/1559672303 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5000be30 con 0x7f3f50004500 2022-06-29T16:04:08.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.395+0000 7f3f6f7fe700 1 -- 172.21.15.60:0/1559672303 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5000bff0 con 0x7f3f500038e0 2022-06-29T16:04:08.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.395+0000 7f3f74fdb700 1 --2- 172.21.15.60:0/1559672303 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f500038e0 0x7f3f50003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:08.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.395+0000 7f3f757dc700 1 --2- 172.21.15.60:0/1559672303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f50004e70 0x7f3f5000b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:08.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.395+0000 7f3f6ffff700 1 --2- 172.21.15.60:0/1559672303 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f50004500 0x7f3f50004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:08.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.399+0000 7f3f757dc700 1 -- 172.21.15.60:0/1559672303 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f500038e0 msgr2=0x7f3f50003ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:08.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.399+0000 7f3f757dc700 1 --2- 172.21.15.60:0/1559672303 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f500038e0 0x7f3f50003ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:08.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.399+0000 7f3f757dc700 1 -- 172.21.15.60:0/1559672303 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f50004500 msgr2=0x7f3f50004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:08.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.399+0000 7f3f757dc700 1 --2- 172.21.15.60:0/1559672303 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f50004500 0x7f3f50004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:08.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.399+0000 7f3f757dc700 1 -- 172.21.15.60:0/1559672303 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f50003260 con 0x7f3f50004e70 2022-06-29T16:04:08.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.399+0000 7f3f74fdb700 1 --2- 172.21.15.60:0/1559672303 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f500038e0 0x7f3f50003ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:04:08.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.399+0000 7f3f757dc700 1 -- 172.21.15.60:0/1559672303 reap_dead start 2022-06-29T16:04:08.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.399+0000 7f3f757dc700 1 --2- 172.21.15.60:0/1559672303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f50004e70 0x7f3f5000b750 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7f3f6800c000 tx=0x7f3f680091d0).ready entity=mon.0 client_cookie=e6821307315b9f3 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:08.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.399+0000 7f3f6f7fe700 1 -- 172.21.15.60:0/1559672303 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f6800e6e0 con 0x7f3f50004e70 2022-06-29T16:04:08.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.399+0000 7f3f6f7fe700 1 -- 172.21.15.60:0/1559672303 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f6800e840 con 0x7f3f50004e70 2022-06-29T16:04:08.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.399+0000 7f3f6f7fe700 1 -- 172.21.15.60:0/1559672303 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f6800eba0 con 0x7f3f50004e70 2022-06-29T16:04:08.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.399+0000 7f3f7723f700 1 -- 172.21.15.60:0/1559672303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f50004e70 msgr2=0x7f3f5000b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:08.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.399+0000 7f3f7723f700 1 --2- 172.21.15.60:0/1559672303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f50004e70 0x7f3f5000b750 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7f3f6800c000 tx=0x7f3f680091d0).stop 2022-06-29T16:04:08.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.399+0000 7f3f7723f700 1 -- 172.21.15.60:0/1559672303 shutdown_connections 2022-06-29T16:04:08.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.399+0000 7f3f7723f700 1 --2- 172.21.15.60:0/1559672303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f50004e70 0x7f3f5000b750 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:08.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.399+0000 7f3f7723f700 1 -- 172.21.15.60:0/1559672303 >> 172.21.15.60:0/1559672303 conn(0x7f3f70063b50 msgr2=0x7f3f700b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:08.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.399+0000 7f3f7723f700 1 -- 172.21.15.60:0/1559672303 shutdown_connections 2022-06-29T16:04:08.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.399+0000 7f3f7723f700 1 -- 172.21.15.60:0/1559672303 wait complete. 2022-06-29T16:04:08.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.399+0000 7f3f7723f700 1 Processor -- start 2022-06-29T16:04:08.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.399+0000 7f3f7723f700 1 -- start start 2022-06-29T16:04:08.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.403+0000 7f3f7723f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f50004e70 0x7f3f70144600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:08.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.403+0000 7f3f7723f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f70153020 0x7f3f7013c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:08.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.403+0000 7f3f7723f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f7013cbd0 0x7f3f700b3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:08.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.403+0000 7f3f7723f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f70142e10 con 0x7f3f70153020 2022-06-29T16:04:08.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.403+0000 7f3f7723f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f700b5260 con 0x7f3f50004e70 2022-06-29T16:04:08.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.403+0000 7f3f7723f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f700b4cc0 con 0x7f3f7013cbd0 2022-06-29T16:04:08.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.403+0000 7f3f6ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f70153020 0x7f3f7013c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:08.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.403+0000 7f3f6ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f70153020 0x7f3f7013c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49242/0 (socket says 172.21.15.60:49242) 2022-06-29T16:04:08.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.403+0000 7f3f74fdb700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f50004e70 0x7f3f70144600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:08.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.403+0000 7f3f757dc700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f7013cbd0 0x7f3f700b3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:08.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.403+0000 7f3f6ffff700 1 -- 172.21.15.60:0/534632980 learned_addr learned my addr 172.21.15.60:0/534632980 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:08.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.403+0000 7f3f6ffff700 1 -- 172.21.15.60:0/534632980 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f7013cbd0 msgr2=0x7f3f700b3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:08.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.403+0000 7f3f6ffff700 1 --2- 172.21.15.60:0/534632980 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f7013cbd0 0x7f3f700b3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:08.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.403+0000 7f3f6ffff700 1 -- 172.21.15.60:0/534632980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f50004e70 msgr2=0x7f3f70144600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:08.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.403+0000 7f3f6ffff700 1 --2- 172.21.15.60:0/534632980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f50004e70 0x7f3f70144600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:08.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.403+0000 7f3f6ffff700 1 -- 172.21.15.60:0/534632980 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f701420e0 con 0x7f3f70153020 2022-06-29T16:04:08.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.403+0000 7f3f6ffff700 1 --2- 172.21.15.60:0/534632980 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f70153020 0x7f3f7013c6e0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0x7f3f58001840 tx=0x7f3f700642c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:08.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.403+0000 7f3f6dffb700 1 -- 172.21.15.60:0/534632980 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f580116b0 con 0x7f3f70153020 2022-06-29T16:04:08.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.403+0000 7f3f6dffb700 1 -- 172.21.15.60:0/534632980 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f58011810 con 0x7f3f70153020 2022-06-29T16:04:08.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.403+0000 7f3f7723f700 1 -- 172.21.15.60:0/534632980 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f700b4740 con 0x7f3f70153020 2022-06-29T16:04:08.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.403+0000 7f3f6dffb700 1 -- 172.21.15.60:0/534632980 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f58011b70 con 0x7f3f70153020 2022-06-29T16:04:08.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.403+0000 7f3f7723f700 1 -- 172.21.15.60:0/534632980 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f700b5a00 con 0x7f3f70153020 2022-06-29T16:04:08.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.403+0000 7f3f7723f700 1 -- 172.21.15.60:0/534632980 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f700b5460 con 0x7f3f70153020 2022-06-29T16:04:08.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.407+0000 7f3f6dffb700 1 -- 172.21.15.60:0/534632980 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3f5801b030 con 0x7f3f70153020 2022-06-29T16:04:08.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.407+0000 7f3f6dffb700 1 --2- 172.21.15.60:0/534632980 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3f54053b00 0x7f3f54055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:08.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.407+0000 7f3f6dffb700 1 -- 172.21.15.60:0/534632980 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3f5806b710 con 0x7f3f70153020 2022-06-29T16:04:08.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.407+0000 7f3f74fdb700 1 --2- 172.21.15.60:0/534632980 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3f54053b00 0x7f3f54055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:08.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.407+0000 7f3f74fdb700 1 --2- 172.21.15.60:0/534632980 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3f54053b00 0x7f3f54055fc0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f3f60003000 tx=0x7f3f60007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:08.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.415+0000 7f3f6dffb700 1 -- 172.21.15.60:0/534632980 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3f58042630 con 0x7f3f70153020 2022-06-29T16:04:08.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.539+0000 7f3f7723f700 1 -- 172.21.15.60:0/534632980 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f700b4920 con 0x7f3f70153020 2022-06-29T16:04:08.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.543+0000 7f3f6dffb700 1 -- 172.21.15.60:0/534632980 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16054 (secure 0 0 0) 0x7f3f700b4920 con 0x7f3f70153020 2022-06-29T16:04:08.549 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:04:08.549 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":70,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:04:08.274701+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:04:06.708236+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2022-06-29T16:04:07.305396+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:04:08.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.543+0000 7f3f7723f700 1 -- 172.21.15.60:0/534632980 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3f54053b00 msgr2=0x7f3f54055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:08.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.543+0000 7f3f7723f700 1 --2- 172.21.15.60:0/534632980 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3f54053b00 0x7f3f54055fc0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f3f60003000 tx=0x7f3f60007a00).stop 2022-06-29T16:04:08.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.543+0000 7f3f7723f700 1 -- 172.21.15.60:0/534632980 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f70153020 msgr2=0x7f3f7013c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:08.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.543+0000 7f3f7723f700 1 --2- 172.21.15.60:0/534632980 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f70153020 0x7f3f7013c6e0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0x7f3f58001840 tx=0x7f3f700642c0).stop 2022-06-29T16:04:08.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.543+0000 7f3f7723f700 1 -- 172.21.15.60:0/534632980 shutdown_connections 2022-06-29T16:04:08.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.543+0000 7f3f7723f700 1 --2- 172.21.15.60:0/534632980 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3f54053b00 0x7f3f54055fc0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:08.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.543+0000 7f3f7723f700 1 --2- 172.21.15.60:0/534632980 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f7013cbd0 0x7f3f700b3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:08.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.543+0000 7f3f7723f700 1 --2- 172.21.15.60:0/534632980 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f70153020 0x7f3f7013c6e0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:08.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.543+0000 7f3f7723f700 1 --2- 172.21.15.60:0/534632980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f50004e70 0x7f3f70144600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:08.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.543+0000 7f3f7723f700 1 -- 172.21.15.60:0/534632980 >> 172.21.15.60:0/534632980 conn(0x7f3f70063b50 msgr2=0x7f3f7013f5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:08.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.543+0000 7f3f7723f700 1 -- 172.21.15.60:0/534632980 shutdown_connections 2022-06-29T16:04:08.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.543+0000 7f3f7723f700 1 -- 172.21.15.60:0/534632980 wait complete. 2022-06-29T16:04:08.563 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:04:08.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.659+0000 7f5a5b3fe700 1 Processor -- start 2022-06-29T16:04:08.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.659+0000 7f5a5b3fe700 1 -- start start 2022-06-29T16:04:08.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.659+0000 7f5a5b3fe700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5a540b2130 0x7f5a54146100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:08.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.659+0000 7f5a5b3fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5a540aed00 0x7f5a541465f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:08.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.659+0000 7f5a5b3fe700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a5405d450 con 0x7f5a5405d070 2022-06-29T16:04:08.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.659+0000 7f5a5b3fe700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a540adf90 con 0x7f5a540aed00 2022-06-29T16:04:08.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.659+0000 7f5a5b3fe700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a540b25d0 con 0x7f5a540b2130 2022-06-29T16:04:08.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.659+0000 7f5a5abfd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5a540aed00 0x7f5a541465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:08.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.659+0000 7f5a5abfd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5a540aed00 0x7f5a541465f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49250/0 (socket says 172.21.15.60:49250) 2022-06-29T16:04:08.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.659+0000 7f5a5abfd700 1 -- 172.21.15.60:0/688279727 learned_addr learned my addr 172.21.15.60:0/688279727 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:08.672 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.659+0000 7f5a59bfb700 1 --2- 172.21.15.60:0/688279727 >> v2:172.21.15.92:3300/0 conn(0x7f5a540b2130 0x7f5a54146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:08.672 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.663+0000 7f5a59bfb700 1 -- 172.21.15.60:0/688279727 >> v2:172.21.15.60:3300/0 conn(0x7f5a540aed00 msgr2=0x7f5a541465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:08.672 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.663+0000 7f5a59bfb700 1 --2- 172.21.15.60:0/688279727 >> v2:172.21.15.60:3300/0 conn(0x7f5a540aed00 0x7f5a541465f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:08.673 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.663+0000 7f5a59bfb700 1 -- 172.21.15.60:0/688279727 >> v1:172.21.15.92:6789/0 conn(0x7f5a5405d070 legacy=0x7f5a5414aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:08.673 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.663+0000 7f5a59bfb700 1 -- 172.21.15.60:0/688279727 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a5414a350 con 0x7f5a540b2130 2022-06-29T16:04:08.673 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.663+0000 7f5a5abfd700 1 --2- 172.21.15.60:0/688279727 >> v2:172.21.15.60:3300/0 conn(0x7f5a540aed00 0x7f5a541465f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:04:08.673 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.663+0000 7f5a59bfb700 1 --2- 172.21.15.60:0/688279727 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a540b2130 0x7f5a54146100 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f5a48002f70 tx=0x7f5a48006df0).ready entity=mon.1 client_cookie=8bbe3be93da7e73c server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:08.673 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.663+0000 7f5a593fa700 1 -- 172.21.15.60:0/688279727 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a4800d570 con 0x7f5a540b2130 2022-06-29T16:04:08.674 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.663+0000 7f5a593fa700 1 -- 172.21.15.60:0/688279727 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a48000e90 con 0x7f5a540b2130 2022-06-29T16:04:08.674 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.663+0000 7f5a593fa700 1 -- 172.21.15.60:0/688279727 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a4800d8c0 con 0x7f5a540b2130 2022-06-29T16:04:08.674 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.663+0000 7f5a5b3fe700 1 -- 172.21.15.60:0/688279727 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a540b2130 msgr2=0x7f5a54146100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:08.674 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.663+0000 7f5a5b3fe700 1 --2- 172.21.15.60:0/688279727 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a540b2130 0x7f5a54146100 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f5a48002f70 tx=0x7f5a48006df0).stop 2022-06-29T16:04:08.675 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.663+0000 7f5a5b3fe700 1 -- 172.21.15.60:0/688279727 shutdown_connections 2022-06-29T16:04:08.675 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.663+0000 7f5a5b3fe700 1 --2- 172.21.15.60:0/688279727 >> v2:172.21.15.60:3300/0 conn(0x7f5a540aed00 0x7f5a541465f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:08.675 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.663+0000 7f5a5b3fe700 1 --2- 172.21.15.60:0/688279727 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a540b2130 0x7f5a54146100 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:08.675 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.663+0000 7f5a5b3fe700 1 -- 172.21.15.60:0/688279727 >> 172.21.15.60:0/688279727 conn(0x7f5a540b5890 msgr2=0x7f5a540b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:08.676 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.663+0000 7f5a5b3fe700 1 -- 172.21.15.60:0/688279727 shutdown_connections 2022-06-29T16:04:08.676 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.663+0000 7f5a5b3fe700 1 -- 172.21.15.60:0/688279727 wait complete. 2022-06-29T16:04:08.679 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.663+0000 7f5a5b3fe700 1 Processor -- start 2022-06-29T16:04:08.680 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.663+0000 7f5a5b3fe700 1 -- start start 2022-06-29T16:04:08.680 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.667+0000 7f5a5b3fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5a5405d070 0x7f5a54065380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:08.680 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.667+0000 7f5a5b3fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a540aed00 0x7f5a541563a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:08.680 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.667+0000 7f5a5b3fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a540b2130 0x7f5a54156890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:08.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.667+0000 7f5a5b3fe700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a54063610 con 0x7f5a540b2130 2022-06-29T16:04:08.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.667+0000 7f5a5b3fe700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a54064ad0 con 0x7f5a540aed00 2022-06-29T16:04:08.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.667+0000 7f5a5b3fe700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a54064d70 con 0x7f5a5405d070 2022-06-29T16:04:08.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.667+0000 7f5a5abfd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a540b2130 0x7f5a54156890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:08.682 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.667+0000 7f5a5abfd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a540b2130 0x7f5a54156890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49254/0 (socket says 172.21.15.60:49254) 2022-06-29T16:04:08.682 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.667+0000 7f5a5abfd700 1 -- 172.21.15.60:0/792279457 learned_addr learned my addr 172.21.15.60:0/792279457 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:08.682 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.667+0000 7f5a5a3fc700 1 --2- 172.21.15.60:0/792279457 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5a5405d070 0x7f5a54065380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:08.682 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.667+0000 7f5a59bfb700 1 --2- 172.21.15.60:0/792279457 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a540aed00 0x7f5a541563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:08.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.667+0000 7f5a5abfd700 1 -- 172.21.15.60:0/792279457 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5a5405d070 msgr2=0x7f5a54065380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:08.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.667+0000 7f5a5abfd700 1 --2- 172.21.15.60:0/792279457 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5a5405d070 0x7f5a54065380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:08.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.667+0000 7f5a5abfd700 1 -- 172.21.15.60:0/792279457 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a540aed00 msgr2=0x7f5a541563a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:08.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.667+0000 7f5a5abfd700 1 --2- 172.21.15.60:0/792279457 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a540aed00 0x7f5a541563a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:08.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.667+0000 7f5a5abfd700 1 -- 172.21.15.60:0/792279457 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a4400b1e0 con 0x7f5a540b2130 2022-06-29T16:04:08.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.667+0000 7f5a5abfd700 1 --2- 172.21.15.60:0/792279457 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a540b2130 0x7f5a54156890 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7f5a4400c000 tx=0x7f5a44006eb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:08.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.667+0000 7f5a537fe700 1 -- 172.21.15.60:0/792279457 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a44012600 con 0x7f5a540b2130 2022-06-29T16:04:08.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.667+0000 7f5a537fe700 1 -- 172.21.15.60:0/792279457 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a44012760 con 0x7f5a540b2130 2022-06-29T16:04:08.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.667+0000 7f5a5b3fe700 1 -- 172.21.15.60:0/792279457 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a4800b180 con 0x7f5a540b2130 2022-06-29T16:04:08.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.667+0000 7f5a537fe700 1 -- 172.21.15.60:0/792279457 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a44012ac0 con 0x7f5a540b2130 2022-06-29T16:04:08.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.667+0000 7f5a5b3fe700 1 -- 172.21.15.60:0/792279457 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a54063a40 con 0x7f5a540b2130 2022-06-29T16:04:08.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.671+0000 7f5a537fe700 1 -- 172.21.15.60:0/792279457 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5a44022070 con 0x7f5a540b2130 2022-06-29T16:04:08.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.671+0000 7f5a5b3fe700 1 -- 172.21.15.60:0/792279457 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a54063780 con 0x7f5a540b2130 2022-06-29T16:04:08.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.671+0000 7f5a537fe700 1 --2- 172.21.15.60:0/792279457 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5a34053b00 0x7f5a34055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:08.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.671+0000 7f5a537fe700 1 -- 172.21.15.60:0/792279457 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5a4406c360 con 0x7f5a540b2130 2022-06-29T16:04:08.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.671+0000 7f5a5a3fc700 1 --2- 172.21.15.60:0/792279457 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5a34053b00 0x7f5a34055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:08.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.671+0000 7f5a5a3fc700 1 --2- 172.21.15.60:0/792279457 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5a34053b00 0x7f5a34055fc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f5a4c00d000 tx=0x7f5a4c007f10).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:08.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.675+0000 7f5a537fe700 1 -- 172.21.15.60:0/792279457 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5a54063780 con 0x7f5a540b2130 2022-06-29T16:04:08.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.799+0000 7f5a5b3fe700 1 -- 172.21.15.60:0/792279457 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a5414a5f0 con 0x7f5a540b2130 2022-06-29T16:04:08.807 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.799+0000 7f5a537fe700 1 -- 172.21.15.60:0/792279457 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16054 (secure 0 0 0) 0x7f5a44016090 con 0x7f5a540b2130 2022-06-29T16:04:08.808 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:04:08.808 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":70,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:04:08.274701+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:04:06.708236+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2022-06-29T16:04:07.305396+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:04:08.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.803+0000 7f5a5b3fe700 1 -- 172.21.15.60:0/792279457 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5a34053b00 msgr2=0x7f5a34055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:08.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.803+0000 7f5a5b3fe700 1 --2- 172.21.15.60:0/792279457 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5a34053b00 0x7f5a34055fc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f5a4c00d000 tx=0x7f5a4c007f10).stop 2022-06-29T16:04:08.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.803+0000 7f5a5b3fe700 1 -- 172.21.15.60:0/792279457 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a540b2130 msgr2=0x7f5a54156890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:08.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.803+0000 7f5a5b3fe700 1 --2- 172.21.15.60:0/792279457 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a540b2130 0x7f5a54156890 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7f5a4400c000 tx=0x7f5a44006eb0).stop 2022-06-29T16:04:08.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.803+0000 7f5a5b3fe700 1 -- 172.21.15.60:0/792279457 shutdown_connections 2022-06-29T16:04:08.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.803+0000 7f5a5b3fe700 1 --2- 172.21.15.60:0/792279457 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5a34053b00 0x7f5a34055fc0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:08.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.803+0000 7f5a5b3fe700 1 --2- 172.21.15.60:0/792279457 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a540b2130 0x7f5a54156890 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:08.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.803+0000 7f5a5b3fe700 1 --2- 172.21.15.60:0/792279457 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a540aed00 0x7f5a541563a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:08.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.803+0000 7f5a5b3fe700 1 --2- 172.21.15.60:0/792279457 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5a5405d070 0x7f5a54065380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:08.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.803+0000 7f5a5b3fe700 1 -- 172.21.15.60:0/792279457 >> 172.21.15.60:0/792279457 conn(0x7f5a540b5890 msgr2=0x7f5a540b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:08.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.803+0000 7f5a5b3fe700 1 -- 172.21.15.60:0/792279457 shutdown_connections 2022-06-29T16:04:08.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.803+0000 7f5a5b3fe700 1 -- 172.21.15.60:0/792279457 wait complete. 2022-06-29T16:04:08.820 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-06-29T16:04:08.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.923+0000 7fbbf15b2700 1 Processor -- start 2022-06-29T16:04:08.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.923+0000 7fbbf15b2700 1 -- start start 2022-06-29T16:04:08.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.923+0000 7fbbf15b2700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbbec0b0890 0x7fbbec0b4540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:08.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.923+0000 7fbbf15b2700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbec0b9fd0 con 0x7fbbec0b3b40 2022-06-29T16:04:08.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.923+0000 7fbbf15b2700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbec0b1a50 con 0x7fbbec0b9bf0 2022-06-29T16:04:08.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.923+0000 7fbbf15b2700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbec0b1cf0 con 0x7fbbec0b0890 2022-06-29T16:04:08.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.923+0000 7fbbf0db1700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fbbec0b3b40 0x7fbbec141b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57350/0 (socket says 172.21.15.60:57350) 2022-06-29T16:04:08.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.923+0000 7fbbf0db1700 1 -- 172.21.15.60:0/198421383 learned_addr learned my addr 172.21.15.60:0/198421383 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:04:08.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.923+0000 7fbbeb7fe700 1 --2- 172.21.15.60:0/198421383 >> v2:172.21.15.92:3300/0 conn(0x7fbbec0b0890 0x7fbbec0b4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:08.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.923+0000 7fbbeaffd700 1 -- 172.21.15.60:0/198421383 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1004902311 0 0) 0x7fbbec0b9fd0 con 0x7fbbec0b3b40 2022-06-29T16:04:08.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.927+0000 7fbbeaffd700 1 -- 172.21.15.60:0/198421383 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbd8003260 con 0x7fbbec0b3b40 2022-06-29T16:04:08.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.927+0000 7fbbeb7fe700 1 -- 172.21.15.60:0/198421383 >> v1:172.21.15.92:6789/0 conn(0x7fbbec0b9bf0 legacy=0x7fbbec0b2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:08.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.927+0000 7fbbeb7fe700 1 -- 172.21.15.60:0/198421383 >> v1:172.21.15.60:6789/0 conn(0x7fbbec0b3b40 legacy=0x7fbbec141b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:08.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.927+0000 7fbbeb7fe700 1 -- 172.21.15.60:0/198421383 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbec0b3470 con 0x7fbbec0b0890 2022-06-29T16:04:08.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.927+0000 7fbbeb7fe700 1 --2- 172.21.15.60:0/198421383 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbec0b0890 0x7fbbec0b4540 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7fbbe0002f70 tx=0x7fbbe0006df0).ready entity=mon.1 client_cookie=a5e66d945eb12024 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:08.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.927+0000 7fbbeaffd700 1 -- 172.21.15.60:0/198421383 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbbe000d610 con 0x7fbbec0b0890 2022-06-29T16:04:08.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.927+0000 7fbbeaffd700 1 -- 172.21.15.60:0/198421383 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbe0000e90 con 0x7fbbec0b0890 2022-06-29T16:04:08.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.927+0000 7fbbeaffd700 1 -- 172.21.15.60:0/198421383 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbbe000d960 con 0x7fbbec0b0890 2022-06-29T16:04:08.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.927+0000 7fbbf15b2700 1 -- 172.21.15.60:0/198421383 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbec0b0890 msgr2=0x7fbbec0b4540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:08.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.927+0000 7fbbf15b2700 1 --2- 172.21.15.60:0/198421383 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbec0b0890 0x7fbbec0b4540 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7fbbe0002f70 tx=0x7fbbe0006df0).stop 2022-06-29T16:04:08.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.927+0000 7fbbf15b2700 1 -- 172.21.15.60:0/198421383 shutdown_connections 2022-06-29T16:04:08.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.927+0000 7fbbf15b2700 1 --2- 172.21.15.60:0/198421383 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbec0b0890 0x7fbbec0b4540 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:08.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.927+0000 7fbbf15b2700 1 -- 172.21.15.60:0/198421383 >> 172.21.15.60:0/198421383 conn(0x7fbbec062b60 msgr2=0x7fbbec0b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:08.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.927+0000 7fbbf15b2700 1 -- 172.21.15.60:0/198421383 shutdown_connections 2022-06-29T16:04:08.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.927+0000 7fbbf15b2700 1 -- 172.21.15.60:0/198421383 wait complete. 2022-06-29T16:04:08.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.927+0000 7fbbf15b2700 1 Processor -- start 2022-06-29T16:04:08.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.927+0000 7fbbf15b2700 1 -- start start 2022-06-29T16:04:08.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.931+0000 7fbbf15b2700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbec0b0890 0x7fbbec144060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:08.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.931+0000 7fbbf15b2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbbec0b3b40 0x7fbbec0aef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:08.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.931+0000 7fbbf15b2700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbbec0b9bf0 0x7fbbec1555b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:08.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.931+0000 7fbbf15b2700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbec1453e0 con 0x7fbbec0b3b40 2022-06-29T16:04:08.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.931+0000 7fbbf15b2700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbec147ec0 con 0x7fbbec0b0890 2022-06-29T16:04:08.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.931+0000 7fbbf15b2700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbec113440 con 0x7fbbec0b9bf0 2022-06-29T16:04:08.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.931+0000 7fbbeb7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbbec0b3b40 0x7fbbec0aef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:08.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.931+0000 7fbbeb7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbbec0b3b40 0x7fbbec0aef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49268/0 (socket says 172.21.15.60:49268) 2022-06-29T16:04:08.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.931+0000 7fbbeb7fe700 1 -- 172.21.15.60:0/2596801451 learned_addr learned my addr 172.21.15.60:0/2596801451 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:08.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.931+0000 7fbbf0db1700 1 --2- 172.21.15.60:0/2596801451 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbbec0b9bf0 0x7fbbec1555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:08.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.931+0000 7fbbebfff700 1 --2- 172.21.15.60:0/2596801451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbec0b0890 0x7fbbec144060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:08.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.931+0000 7fbbeb7fe700 1 -- 172.21.15.60:0/2596801451 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbbec0b9bf0 msgr2=0x7fbbec1555b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:08.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.931+0000 7fbbeb7fe700 1 --2- 172.21.15.60:0/2596801451 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbbec0b9bf0 0x7fbbec1555b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:08.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.931+0000 7fbbeb7fe700 1 -- 172.21.15.60:0/2596801451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbec0b0890 msgr2=0x7fbbec144060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:08.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.931+0000 7fbbeb7fe700 1 --2- 172.21.15.60:0/2596801451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbec0b0890 0x7fbbec144060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:08.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.931+0000 7fbbeb7fe700 1 -- 172.21.15.60:0/2596801451 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe000b180 con 0x7fbbec0b3b40 2022-06-29T16:04:08.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.931+0000 7fbbeb7fe700 1 --2- 172.21.15.60:0/2596801451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbbec0b3b40 0x7fbbec0aef90 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0x7fbbe0002f70 tx=0x7fbbe0014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:08.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.931+0000 7fbbe97fa700 1 -- 172.21.15.60:0/2596801451 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbbe0013de0 con 0x7fbbec0b3b40 2022-06-29T16:04:08.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.931+0000 7fbbe97fa700 1 -- 172.21.15.60:0/2596801451 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbe000d290 con 0x7fbbec0b3b40 2022-06-29T16:04:08.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.931+0000 7fbbf15b2700 1 -- 172.21.15.60:0/2596801451 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbec1136d0 con 0x7fbbec0b3b40 2022-06-29T16:04:08.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.931+0000 7fbbe97fa700 1 -- 172.21.15.60:0/2596801451 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbbe001e5f0 con 0x7fbbec0b3b40 2022-06-29T16:04:08.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.931+0000 7fbbf15b2700 1 -- 172.21.15.60:0/2596801451 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbec146030 con 0x7fbbec0b3b40 2022-06-29T16:04:08.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.931+0000 7fbbf15b2700 1 -- 172.21.15.60:0/2596801451 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbec1458c0 con 0x7fbbec0b3b40 2022-06-29T16:04:08.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.935+0000 7fbbe97fa700 1 -- 172.21.15.60:0/2596801451 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fbbe001c070 con 0x7fbbec0b3b40 2022-06-29T16:04:08.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.935+0000 7fbbe97fa700 1 --2- 172.21.15.60:0/2596801451 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbbcc053b00 0x7fbbcc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:08.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.935+0000 7fbbe97fa700 1 -- 172.21.15.60:0/2596801451 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbbe006c970 con 0x7fbbec0b3b40 2022-06-29T16:04:08.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.935+0000 7fbbebfff700 1 --2- 172.21.15.60:0/2596801451 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbbcc053b00 0x7fbbcc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:08.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.935+0000 7fbbebfff700 1 --2- 172.21.15.60:0/2596801451 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbbcc053b00 0x7fbbcc055fc0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7fbbdc00d000 tx=0x7fbbdc007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:08.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:08.939+0000 7fbbe97fa700 1 -- 172.21.15.60:0/2596801451 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbbe0043780 con 0x7fbbec0b3b40 2022-06-29T16:04:09.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.107+0000 7fbbf15b2700 1 -- 172.21.15.60:0/2596801451 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fbbec0573d0 con 0x7fbbec0b3b40 2022-06-29T16:04:09.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.287+0000 7fbbe97fa700 1 -- 172.21.15.60:0/2596801451 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v30) v1 ==== 118+0+0 (secure 0 0 0) 0x7fbbe0023070 con 0x7fbbec0b3b40 2022-06-29T16:04:09.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.287+0000 7fbbf15b2700 1 -- 172.21.15.60:0/2596801451 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbbcc053b00 msgr2=0x7fbbcc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:09.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.287+0000 7fbbf15b2700 1 --2- 172.21.15.60:0/2596801451 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbbcc053b00 0x7fbbcc055fc0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7fbbdc00d000 tx=0x7fbbdc007b90).stop 2022-06-29T16:04:09.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.291+0000 7fbbf15b2700 1 -- 172.21.15.60:0/2596801451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbbec0b3b40 msgr2=0x7fbbec0aef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:09.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.291+0000 7fbbf15b2700 1 --2- 172.21.15.60:0/2596801451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbbec0b3b40 0x7fbbec0aef90 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0x7fbbe0002f70 tx=0x7fbbe0014000).stop 2022-06-29T16:04:09.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.291+0000 7fbbf15b2700 1 -- 172.21.15.60:0/2596801451 shutdown_connections 2022-06-29T16:04:09.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.291+0000 7fbbf15b2700 1 --2- 172.21.15.60:0/2596801451 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbbcc053b00 0x7fbbcc055fc0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:09.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.291+0000 7fbbf15b2700 1 --2- 172.21.15.60:0/2596801451 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbbec0b9bf0 0x7fbbec1555b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:09.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.291+0000 7fbbf15b2700 1 --2- 172.21.15.60:0/2596801451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbbec0b3b40 0x7fbbec0aef90 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:09.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.291+0000 7fbbf15b2700 1 --2- 172.21.15.60:0/2596801451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbec0b0890 0x7fbbec144060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:09.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.291+0000 7fbbf15b2700 1 -- 172.21.15.60:0/2596801451 >> 172.21.15.60:0/2596801451 conn(0x7fbbec062b60 msgr2=0x7fbbec0b66c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:09.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.291+0000 7fbbf15b2700 1 -- 172.21.15.60:0/2596801451 shutdown_connections 2022-06-29T16:04:09.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.291+0000 7fbbf15b2700 1 -- 172.21.15.60:0/2596801451 wait complete. 2022-06-29T16:04:09.310 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:04:09.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.411+0000 7f5279cbe700 1 Processor -- start 2022-06-29T16:04:09.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.411+0000 7f5279cbe700 1 -- start start 2022-06-29T16:04:09.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.411+0000 7f5279cbe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f527405d070 0x7f527414aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:09.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.411+0000 7f5279cbe700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527405d450 con 0x7f527414af80 2022-06-29T16:04:09.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.411+0000 7f5279cbe700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52740ad840 con 0x7f52740b2130 2022-06-29T16:04:09.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.411+0000 7f5279cbe700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52740adf90 con 0x7f527405d070 2022-06-29T16:04:09.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.411+0000 7f5278cbc700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f527405d070 0x7f527414aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:09.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.411+0000 7f5278cbc700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f527405d070 0x7f527414aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49274/0 (socket says 172.21.15.60:49274) 2022-06-29T16:04:09.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.411+0000 7f5278cbc700 1 -- 172.21.15.60:0/1679564007 learned_addr learned my addr 172.21.15.60:0/1679564007 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:09.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.411+0000 7f52794bd700 1 --1- 172.21.15.60:0/1679564007 >> v1:172.21.15.60:6789/0 conn(0x7f527414af80 0x7f52740ad130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57364/0 (socket says 172.21.15.60:57364) 2022-06-29T16:04:09.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.411+0000 7f5278cbc700 1 -- 172.21.15.60:0/1679564007 >> v1:172.21.15.92:6789/0 conn(0x7f52740b2130 legacy=0x7f52740aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:09.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.411+0000 7f5278cbc700 1 -- 172.21.15.60:0/1679564007 >> v1:172.21.15.60:6789/0 conn(0x7f527414af80 legacy=0x7f52740ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:09.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.411+0000 7f5278cbc700 1 -- 172.21.15.60:0/1679564007 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52740b25d0 con 0x7f527405d070 2022-06-29T16:04:09.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.411+0000 7f5278cbc700 1 --2- 172.21.15.60:0/1679564007 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f527405d070 0x7f527414aa90 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0x7f52740b5cd0 tx=0x7f526c009290).ready entity=mon.0 client_cookie=da4d02c482b005d server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:09.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.411+0000 7f52737fe700 1 -- 172.21.15.60:0/1679564007 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f526c00b860 con 0x7f527405d070 2022-06-29T16:04:09.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.411+0000 7f52737fe700 1 -- 172.21.15.60:0/1679564007 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f526c00b9c0 con 0x7f527405d070 2022-06-29T16:04:09.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.411+0000 7f52737fe700 1 -- 172.21.15.60:0/1679564007 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f526c00bb30 con 0x7f527405d070 2022-06-29T16:04:09.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.411+0000 7f5279cbe700 1 -- 172.21.15.60:0/1679564007 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f527405d070 msgr2=0x7f527414aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:09.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.411+0000 7f5279cbe700 1 --2- 172.21.15.60:0/1679564007 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f527405d070 0x7f527414aa90 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0x7f52740b5cd0 tx=0x7f526c009290).stop 2022-06-29T16:04:09.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.411+0000 7f5279cbe700 1 -- 172.21.15.60:0/1679564007 shutdown_connections 2022-06-29T16:04:09.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.411+0000 7f5279cbe700 1 --2- 172.21.15.60:0/1679564007 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f527405d070 0x7f527414aa90 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:09.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.411+0000 7f5279cbe700 1 -- 172.21.15.60:0/1679564007 >> 172.21.15.60:0/1679564007 conn(0x7f52740b5890 msgr2=0x7f52740b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:09.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.415+0000 7f5279cbe700 1 -- 172.21.15.60:0/1679564007 shutdown_connections 2022-06-29T16:04:09.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.415+0000 7f5279cbe700 1 -- 172.21.15.60:0/1679564007 wait complete. 2022-06-29T16:04:09.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.415+0000 7f5279cbe700 1 Processor -- start 2022-06-29T16:04:09.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.415+0000 7f5279cbe700 1 -- start start 2022-06-29T16:04:09.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.415+0000 7f5279cbe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f527405d070 0x7f5274118f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:09.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.415+0000 7f5279cbe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f52740b2130 0x7f52740654f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:09.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.415+0000 7f5279cbe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f527414af80 0x7f5274116780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:09.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.415+0000 7f5279cbe700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52741179e0 con 0x7f527405d070 2022-06-29T16:04:09.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.415+0000 7f5279cbe700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52741185a0 con 0x7f527414af80 2022-06-29T16:04:09.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.415+0000 7f5279cbe700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5274118840 con 0x7f52740b2130 2022-06-29T16:04:09.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.415+0000 7f5278cbc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f527405d070 0x7f5274118f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:09.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.415+0000 7f5278cbc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f527405d070 0x7f5274118f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49280/0 (socket says 172.21.15.60:49280) 2022-06-29T16:04:09.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.415+0000 7f5278cbc700 1 -- 172.21.15.60:0/836659798 learned_addr learned my addr 172.21.15.60:0/836659798 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:09.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.415+0000 7f5273fff700 1 --2- 172.21.15.60:0/836659798 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f52740b2130 0x7f52740654f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:09.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.415+0000 7f52794bd700 1 --2- 172.21.15.60:0/836659798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f527414af80 0x7f5274116780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:09.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.415+0000 7f5278cbc700 1 -- 172.21.15.60:0/836659798 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f52740b2130 msgr2=0x7f52740654f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:09.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.415+0000 7f5278cbc700 1 --2- 172.21.15.60:0/836659798 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f52740b2130 0x7f52740654f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:09.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.415+0000 7f5278cbc700 1 -- 172.21.15.60:0/836659798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f527414af80 msgr2=0x7f5274116780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:09.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.415+0000 7f5278cbc700 1 --2- 172.21.15.60:0/836659798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f527414af80 0x7f5274116780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:09.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.415+0000 7f5278cbc700 1 -- 172.21.15.60:0/836659798 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f526c009530 con 0x7f527405d070 2022-06-29T16:04:09.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.415+0000 7f5278cbc700 1 --2- 172.21.15.60:0/836659798 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f527405d070 0x7f5274118f80 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0x7f52740b5cd0 tx=0x7f526c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:09.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.415+0000 7f5271ffb700 1 -- 172.21.15.60:0/836659798 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f526c00bda0 con 0x7f527405d070 2022-06-29T16:04:09.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.415+0000 7f5271ffb700 1 -- 172.21.15.60:0/836659798 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f526c01f3f0 con 0x7f527405d070 2022-06-29T16:04:09.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.415+0000 7f5279cbe700 1 -- 172.21.15.60:0/836659798 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5274118ae0 con 0x7f527405d070 2022-06-29T16:04:09.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.415+0000 7f5271ffb700 1 -- 172.21.15.60:0/836659798 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f526c01f750 con 0x7f527405d070 2022-06-29T16:04:09.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.415+0000 7f5279cbe700 1 -- 172.21.15.60:0/836659798 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5274116c70 con 0x7f527405d070 2022-06-29T16:04:09.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.419+0000 7f5279cbe700 1 -- 172.21.15.60:0/836659798 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f527414a350 con 0x7f527405d070 2022-06-29T16:04:09.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.419+0000 7f5271ffb700 1 -- 172.21.15.60:0/836659798 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f526c026030 con 0x7f527405d070 2022-06-29T16:04:09.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.419+0000 7f5271ffb700 1 --2- 172.21.15.60:0/836659798 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5254053a70 0x7f5254055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:09.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.419+0000 7f5271ffb700 1 -- 172.21.15.60:0/836659798 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f526c06cc30 con 0x7f527405d070 2022-06-29T16:04:09.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.419+0000 7f5273fff700 1 --2- 172.21.15.60:0/836659798 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5254053a70 0x7f5254055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:09.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.419+0000 7f5273fff700 1 --2- 172.21.15.60:0/836659798 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5254053a70 0x7f5254055f30 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f526800d000 tx=0x7f5268007b70).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:09.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.427+0000 7f5271ffb700 1 -- 172.21.15.60:0/836659798 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f526c047580 con 0x7f527405d070 2022-06-29T16:04:09.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.595+0000 7f5279cbe700 1 -- 172.21.15.60:0/836659798 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5274118c70 con 0x7f527405d070 2022-06-29T16:04:09.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.595+0000 7f5271ffb700 1 -- 172.21.15.60:0/836659798 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f526c015070 con 0x7f527405d070 2022-06-29T16:04:09.602 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:04:09.603 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":30,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5690,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1158969084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1158969084},{"type":"v1","addr":"172.21.15.60:6833","nonce":1158969084}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":28}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:04:08.275039+0000","modified":"2022-06-29T16:04:09.279046+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6304":{"gid":6304,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/953938586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":953938586},{"type":"v1","addr":"172.21.15.92:6835","nonce":953938586}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-06-29T16:04:09.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.595+0000 7f5279cbe700 1 -- 172.21.15.60:0/836659798 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5254053a70 msgr2=0x7f5254055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:09.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.595+0000 7f5279cbe700 1 --2- 172.21.15.60:0/836659798 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5254053a70 0x7f5254055f30 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f526800d000 tx=0x7f5268007b70).stop 2022-06-29T16:04:09.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.595+0000 7f5279cbe700 1 -- 172.21.15.60:0/836659798 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f527405d070 msgr2=0x7f5274118f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:09.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.595+0000 7f5279cbe700 1 --2- 172.21.15.60:0/836659798 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f527405d070 0x7f5274118f80 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0x7f52740b5cd0 tx=0x7f526c016000).stop 2022-06-29T16:04:09.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.595+0000 7f5279cbe700 1 -- 172.21.15.60:0/836659798 shutdown_connections 2022-06-29T16:04:09.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.595+0000 7f5279cbe700 1 --2- 172.21.15.60:0/836659798 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5254053a70 0x7f5254055f30 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:09.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.595+0000 7f5279cbe700 1 --2- 172.21.15.60:0/836659798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f527414af80 0x7f5274116780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:09.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.595+0000 7f5279cbe700 1 --2- 172.21.15.60:0/836659798 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f52740b2130 0x7f52740654f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:09.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.595+0000 7f5279cbe700 1 --2- 172.21.15.60:0/836659798 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f527405d070 0x7f5274118f80 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:09.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.599+0000 7f5279cbe700 1 -- 172.21.15.60:0/836659798 >> 172.21.15.60:0/836659798 conn(0x7f52740b5890 msgr2=0x7f5274062450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:09.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.599+0000 7f5279cbe700 1 -- 172.21.15.60:0/836659798 shutdown_connections 2022-06-29T16:04:09.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.599+0000 7f5279cbe700 1 -- 172.21.15.60:0/836659798 wait complete. 2022-06-29T16:04:09.608 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 30 2022-06-29T16:04:09.615 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:04:09.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.711+0000 7f11467fa700 1 Processor -- start 2022-06-29T16:04:09.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.711+0000 7f11467fa700 1 -- start start 2022-06-29T16:04:09.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.711+0000 7f11467fa700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f114005cf60 0x7f1140146240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:09.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.711+0000 7f11467fa700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1140142bd0 con 0x7f114005c2f0 2022-06-29T16:04:09.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.711+0000 7f11467fa700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1140062050 con 0x7f1140061870 2022-06-29T16:04:09.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.711+0000 7f11467fa700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11400622f0 con 0x7f114005cf60 2022-06-29T16:04:09.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.711+0000 7f1137fff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f114005c2f0 0x7f11400b3750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57374/0 (socket says 172.21.15.60:57374) 2022-06-29T16:04:09.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.711+0000 7f1137fff700 1 -- 172.21.15.60:0/3923279245 learned_addr learned my addr 172.21.15.60:0/3923279245 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:04:09.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f1144d97700 1 --2- 172.21.15.60:0/3923279245 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f114005cf60 0x7f1140146240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:09.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f113f7fe700 1 -- 172.21.15.60:0/3923279245 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2815954292 0 0) 0x7f1140142bd0 con 0x7f114005c2f0 2022-06-29T16:04:09.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f113f7fe700 1 -- 172.21.15.60:0/3923279245 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1120003260 con 0x7f114005c2f0 2022-06-29T16:04:09.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f113f7fe700 1 -- 172.21.15.60:0/3923279245 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 196816776 0 0) 0x7f1120003260 con 0x7f114005c2f0 2022-06-29T16:04:09.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f113f7fe700 1 -- 172.21.15.60:0/3923279245 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f114005cf60 msgr2=0x7f1140146240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:09.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f113f7fe700 1 --2- 172.21.15.60:0/3923279245 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f114005cf60 0x7f1140146240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:09.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f113f7fe700 1 -- 172.21.15.60:0/3923279245 >> v1:172.21.15.92:6789/0 conn(0x7f1140061870 legacy=0x7f1140066240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:09.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f113f7fe700 1 -- 172.21.15.60:0/3923279245 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1140061c50 con 0x7f114005c2f0 2022-06-29T16:04:09.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f113f7fe700 1 -- 172.21.15.60:0/3923279245 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f1128001660 con 0x7f114005c2f0 2022-06-29T16:04:09.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f113f7fe700 1 -- 172.21.15.60:0/3923279245 >> v1:172.21.15.60:6789/0 conn(0x7f114005c2f0 legacy=0x7f11400b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:09.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f113f7fe700 1 --2- 172.21.15.60:0/3923279245 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1120003780 0x7f1120003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:09.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f113f7fe700 1 --2- 172.21.15.60:0/3923279245 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f114005cf60 0x7f1120004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:09.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f113f7fe700 1 --2- 172.21.15.60:0/3923279245 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11200049f0 0x7f1120004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:09.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f113f7fe700 1 -- 172.21.15.60:0/3923279245 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11400622f0 con 0x7f1120003780 2022-06-29T16:04:09.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f113f7fe700 1 -- 172.21.15.60:0/3923279245 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f112000b780 con 0x7f11200049f0 2022-06-29T16:04:09.728 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f113f7fe700 1 -- 172.21.15.60:0/3923279245 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f112000b940 con 0x7f114005cf60 2022-06-29T16:04:09.728 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f113ffff700 1 --2- 172.21.15.60:0/3923279245 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1120003780 0x7f1120003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:09.728 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f1144d97700 1 --2- 172.21.15.60:0/3923279245 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11200049f0 0x7f1120004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:09.728 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f1137fff700 1 --2- 172.21.15.60:0/3923279245 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f114005cf60 0x7f1120004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:09.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f113ffff700 1 -- 172.21.15.60:0/3923279245 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f114005cf60 msgr2=0x7f1120004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:09.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f113ffff700 1 --2- 172.21.15.60:0/3923279245 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f114005cf60 0x7f1120004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:09.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f113ffff700 1 -- 172.21.15.60:0/3923279245 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11200049f0 msgr2=0x7f1120004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:09.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f113ffff700 1 --2- 172.21.15.60:0/3923279245 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11200049f0 0x7f1120004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:09.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f113ffff700 1 -- 172.21.15.60:0/3923279245 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1120003260 con 0x7f1120003780 2022-06-29T16:04:09.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f113ffff700 1 --2- 172.21.15.60:0/3923279245 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1120003780 0x7f1120003ba0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7f11400ada90 tx=0x7f1130007a00).ready entity=mon.0 client_cookie=f20f3d1f44b29409 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:09.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f113f7fe700 1 -- 172.21.15.60:0/3923279245 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1130007d10 con 0x7f1120003780 2022-06-29T16:04:09.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f113f7fe700 1 -- 172.21.15.60:0/3923279245 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1130007030 con 0x7f1120003780 2022-06-29T16:04:09.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f113f7fe700 1 -- 172.21.15.60:0/3923279245 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1130007390 con 0x7f1120003780 2022-06-29T16:04:09.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f11467fa700 1 -- 172.21.15.60:0/3923279245 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1120003780 msgr2=0x7f1120003ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:09.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f11467fa700 1 --2- 172.21.15.60:0/3923279245 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1120003780 0x7f1120003ba0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7f11400ada90 tx=0x7f1130007a00).stop 2022-06-29T16:04:09.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f1144d97700 1 -- 172.21.15.60:0/3923279245 reap_dead start 2022-06-29T16:04:09.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f11467fa700 1 -- 172.21.15.60:0/3923279245 shutdown_connections 2022-06-29T16:04:09.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f11467fa700 1 -- 172.21.15.60:0/3923279245 >> 172.21.15.60:0/3923279245 conn(0x7f1140064a10 msgr2=0x7f1140064df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:09.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f11467fa700 1 -- 172.21.15.60:0/3923279245 shutdown_connections 2022-06-29T16:04:09.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.715+0000 7f11467fa700 1 -- 172.21.15.60:0/3923279245 wait complete. 2022-06-29T16:04:09.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.719+0000 7f11467fa700 1 Processor -- start 2022-06-29T16:04:09.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.719+0000 7f11467fa700 1 -- start start 2022-06-29T16:04:09.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.719+0000 7f11467fa700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1120003780 0x7f1140140b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:09.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.719+0000 7f11467fa700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11400b5f00 0x7f11400b5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:09.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.719+0000 7f11467fa700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11400b6af0 0x7f11400ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:09.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.719+0000 7f11467fa700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11400b7b00 con 0x7f11400b5f00 2022-06-29T16:04:09.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.719+0000 7f11467fa700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11400b5520 con 0x7f1120003780 2022-06-29T16:04:09.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.719+0000 7f11467fa700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11400b6420 con 0x7f11400b6af0 2022-06-29T16:04:09.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.719+0000 7f1137fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11400b5f00 0x7f11400b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:09.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.719+0000 7f1137fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11400b5f00 0x7f11400b5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49304/0 (socket says 172.21.15.60:49304) 2022-06-29T16:04:09.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.719+0000 7f1137fff700 1 -- 172.21.15.60:0/2069406423 learned_addr learned my addr 172.21.15.60:0/2069406423 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:09.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.719+0000 7f1144d97700 1 --2- 172.21.15.60:0/2069406423 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11400b6af0 0x7f11400ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:09.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.719+0000 7f113ffff700 1 --2- 172.21.15.60:0/2069406423 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1120003780 0x7f1140140b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:09.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.719+0000 7f1137fff700 1 -- 172.21.15.60:0/2069406423 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11400b6af0 msgr2=0x7f11400ae5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:09.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.719+0000 7f1137fff700 1 --2- 172.21.15.60:0/2069406423 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11400b6af0 0x7f11400ae5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:09.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.719+0000 7f1137fff700 1 -- 172.21.15.60:0/2069406423 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1120003780 msgr2=0x7f1140140b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:09.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.719+0000 7f1137fff700 1 --2- 172.21.15.60:0/2069406423 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1120003780 0x7f1140140b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:09.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.719+0000 7f1137fff700 1 -- 172.21.15.60:0/2069406423 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11400b5740 con 0x7f11400b5f00 2022-06-29T16:04:09.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.719+0000 7f1137fff700 1 --2- 172.21.15.60:0/2069406423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11400b5f00 0x7f11400b5030 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0x7f1140153000 tx=0x7f1128001e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:09.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.719+0000 7f113dffb700 1 -- 172.21.15.60:0/2069406423 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11280116b0 con 0x7f11400b5f00 2022-06-29T16:04:09.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.719+0000 7f113dffb700 1 -- 172.21.15.60:0/2069406423 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1128011810 con 0x7f11400b5f00 2022-06-29T16:04:09.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.719+0000 7f11467fa700 1 -- 172.21.15.60:0/2069406423 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11400b6650 con 0x7f11400b5f00 2022-06-29T16:04:09.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.719+0000 7f113dffb700 1 -- 172.21.15.60:0/2069406423 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1128011b70 con 0x7f11400b5f00 2022-06-29T16:04:09.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.719+0000 7f11467fa700 1 -- 172.21.15.60:0/2069406423 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11400aea90 con 0x7f11400b5f00 2022-06-29T16:04:09.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.723+0000 7f113dffb700 1 -- 172.21.15.60:0/2069406423 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f112801b030 con 0x7f11400b5f00 2022-06-29T16:04:09.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.723+0000 7f11467fa700 1 -- 172.21.15.60:0/2069406423 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11400573d0 con 0x7f11400b5f00 2022-06-29T16:04:09.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.723+0000 7f113dffb700 1 --2- 172.21.15.60:0/2069406423 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1124053a70 0x7f1124055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:09.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.723+0000 7f113dffb700 1 -- 172.21.15.60:0/2069406423 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f112806b970 con 0x7f11400b5f00 2022-06-29T16:04:09.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.723+0000 7f113ffff700 1 --2- 172.21.15.60:0/2069406423 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1124053a70 0x7f1124055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:09.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.723+0000 7f113ffff700 1 --2- 172.21.15.60:0/2069406423 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1124053a70 0x7f1124055f30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f113001b000 tx=0x7f1130015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:09.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.731+0000 7f113dffb700 1 -- 172.21.15.60:0/2069406423 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1128042890 con 0x7f11400b5f00 2022-06-29T16:04:09.866 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.859+0000 7f11467fa700 1 -- 172.21.15.60:0/2069406423 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11400684b0 con 0x7f11400b5f00 2022-06-29T16:04:09.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.863+0000 7f113dffb700 1 -- 172.21.15.60:0/2069406423 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16041 (secure 0 0 0) 0x7f1128019030 con 0x7f11400b5f00 2022-06-29T16:04:09.868 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:04:09.869 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":71,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:04:09.277985+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:04:06.708236+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2022-06-29T16:04:07.305396+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:04:09.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.863+0000 7f11467fa700 1 -- 172.21.15.60:0/2069406423 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1124053a70 msgr2=0x7f1124055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:09.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.863+0000 7f11467fa700 1 --2- 172.21.15.60:0/2069406423 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1124053a70 0x7f1124055f30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f113001b000 tx=0x7f1130015000).stop 2022-06-29T16:04:09.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.863+0000 7f11467fa700 1 -- 172.21.15.60:0/2069406423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11400b5f00 msgr2=0x7f11400b5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:09.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.863+0000 7f11467fa700 1 --2- 172.21.15.60:0/2069406423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11400b5f00 0x7f11400b5030 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0x7f1140153000 tx=0x7f1128001e10).stop 2022-06-29T16:04:09.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.863+0000 7f11467fa700 1 -- 172.21.15.60:0/2069406423 shutdown_connections 2022-06-29T16:04:09.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.863+0000 7f11467fa700 1 --2- 172.21.15.60:0/2069406423 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1124053a70 0x7f1124055f30 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:09.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.863+0000 7f11467fa700 1 --2- 172.21.15.60:0/2069406423 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f11400b6af0 0x7f11400ae5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:09.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.863+0000 7f11467fa700 1 --2- 172.21.15.60:0/2069406423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f11400b5f00 0x7f11400b5030 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:09.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.863+0000 7f11467fa700 1 --2- 172.21.15.60:0/2069406423 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1120003780 0x7f1140140b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:09.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.863+0000 7f11467fa700 1 -- 172.21.15.60:0/2069406423 >> 172.21.15.60:0/2069406423 conn(0x7f1140064a10 msgr2=0x7f11400ad210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:09.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.863+0000 7f11467fa700 1 -- 172.21.15.60:0/2069406423 shutdown_connections 2022-06-29T16:04:09.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.863+0000 7f11467fa700 1 -- 172.21.15.60:0/2069406423 wait complete. 2022-06-29T16:04:09.883 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T16:04:10.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.995+0000 7f8f4ecbe700 1 Processor -- start 2022-06-29T16:04:10.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.995+0000 7f8f4ecbe700 1 -- start start 2022-06-29T16:04:10.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.995+0000 7f8f4ecbe700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8f480b1500 0x7f8f480b2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:10.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.995+0000 7f8f4ecbe700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f480b2f40 con 0x7f8f480ae750 2022-06-29T16:04:10.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.995+0000 7f8f4ecbe700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f4805fee0 con 0x7f8f4805fb00 2022-06-29T16:04:10.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.995+0000 7f8f4ecbe700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f480b0150 con 0x7f8f480b1500 2022-06-29T16:04:10.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.995+0000 7f8f4d4bb700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f8f480ae750 0x7f8f48142fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57396/0 (socket says 172.21.15.60:57396) 2022-06-29T16:04:10.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.995+0000 7f8f4d4bb700 1 -- 172.21.15.60:0/981634310 learned_addr learned my addr 172.21.15.60:0/981634310 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:04:10.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.995+0000 7f8f4e4bd700 1 --2- 172.21.15.60:0/981634310 >> v2:172.21.15.92:3300/0 conn(0x7f8f480b1500 0x7f8f480b2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:10.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.995+0000 7f8f4ccba700 1 -- 172.21.15.60:0/981634310 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 627575986 0 0) 0x7f8f3c001110 con 0x7f8f480ae750 2022-06-29T16:04:10.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.999+0000 7f8f4ccba700 1 -- 172.21.15.60:0/981634310 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f34003260 con 0x7f8f480ae750 2022-06-29T16:04:10.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.999+0000 7f8f4ccba700 1 -- 172.21.15.60:0/981634310 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483103222 0 0) 0x7f8f4805fee0 con 0x7f8f4805fb00 2022-06-29T16:04:10.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.999+0000 7f8f4ccba700 1 -- 172.21.15.60:0/981634310 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f3c001110 con 0x7f8f4805fb00 2022-06-29T16:04:10.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.999+0000 7f8f4e4bd700 1 -- 172.21.15.60:0/981634310 >> v1:172.21.15.92:6789/0 conn(0x7f8f4805fb00 legacy=0x7f8f480ae040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:10.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.999+0000 7f8f4e4bd700 1 -- 172.21.15.60:0/981634310 >> v1:172.21.15.60:6789/0 conn(0x7f8f480ae750 legacy=0x7f8f48142fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:10.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.999+0000 7f8f4e4bd700 1 -- 172.21.15.60:0/981634310 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f480acba0 con 0x7f8f480b1500 2022-06-29T16:04:10.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.999+0000 7f8f4e4bd700 1 --2- 172.21.15.60:0/981634310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f480b1500 0x7f8f480b2a50 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f8f400032f0 tx=0x7f8f400091d0).ready entity=mon.1 client_cookie=9cc979b66b87fe9f server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:10.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.999+0000 7f8f4ccba700 1 -- 172.21.15.60:0/981634310 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f4000b650 con 0x7f8f480b1500 2022-06-29T16:04:10.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.999+0000 7f8f4ccba700 1 -- 172.21.15.60:0/981634310 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f4000b7b0 con 0x7f8f480b1500 2022-06-29T16:04:10.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.999+0000 7f8f4ccba700 1 -- 172.21.15.60:0/981634310 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f4000b920 con 0x7f8f480b1500 2022-06-29T16:04:10.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.999+0000 7f8f4ecbe700 1 -- 172.21.15.60:0/981634310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f480b1500 msgr2=0x7f8f480b2a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:10.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.999+0000 7f8f4ecbe700 1 --2- 172.21.15.60:0/981634310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f480b1500 0x7f8f480b2a50 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f8f400032f0 tx=0x7f8f400091d0).stop 2022-06-29T16:04:10.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.999+0000 7f8f4ecbe700 1 -- 172.21.15.60:0/981634310 shutdown_connections 2022-06-29T16:04:10.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.999+0000 7f8f4ecbe700 1 --2- 172.21.15.60:0/981634310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f480b1500 0x7f8f480b2a50 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:10.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.999+0000 7f8f4ecbe700 1 -- 172.21.15.60:0/981634310 >> 172.21.15.60:0/981634310 conn(0x7f8f48063ab0 msgr2=0x7f8f4805f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:10.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.999+0000 7f8f4ecbe700 1 -- 172.21.15.60:0/981634310 shutdown_connections 2022-06-29T16:04:10.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.999+0000 7f8f4ecbe700 1 -- 172.21.15.60:0/981634310 wait complete. 2022-06-29T16:04:10.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.999+0000 7f8f4ecbe700 1 Processor -- start 2022-06-29T16:04:10.016 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:09.999+0000 7f8f4ecbe700 1 -- start start 2022-06-29T16:04:10.016 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.003+0000 7f8f4ecbe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8f4805fb00 0x7f8f481554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:10.016 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.003+0000 7f8f4ecbe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f480ae750 0x7f8f48065040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:10.016 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.003+0000 7f8f4ecbe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8f480b1500 0x7f8f48147840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:10.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.003+0000 7f8f4ecbe700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f48144810 con 0x7f8f480b1500 2022-06-29T16:04:10.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.003+0000 7f8f4ecbe700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f48112db0 con 0x7f8f480ae750 2022-06-29T16:04:10.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.003+0000 7f8f4ecbe700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f48145380 con 0x7f8f4805fb00 2022-06-29T16:04:10.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.003+0000 7f8f4e4bd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8f480b1500 0x7f8f48147840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:10.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.003+0000 7f8f4e4bd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8f480b1500 0x7f8f48147840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49318/0 (socket says 172.21.15.60:49318) 2022-06-29T16:04:10.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.003+0000 7f8f4e4bd700 1 -- 172.21.15.60:0/3856906137 learned_addr learned my addr 172.21.15.60:0/3856906137 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:10.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.003+0000 7f8f4d4bb700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f480ae750 0x7f8f48065040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:10.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.003+0000 7f8f4dcbc700 1 --2- 172.21.15.60:0/3856906137 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8f4805fb00 0x7f8f481554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:10.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.003+0000 7f8f4e4bd700 1 -- 172.21.15.60:0/3856906137 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8f4805fb00 msgr2=0x7f8f481554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:10.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.003+0000 7f8f4e4bd700 1 --2- 172.21.15.60:0/3856906137 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8f4805fb00 0x7f8f481554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:10.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.003+0000 7f8f4e4bd700 1 -- 172.21.15.60:0/3856906137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f480ae750 msgr2=0x7f8f48065040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:10.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.003+0000 7f8f4e4bd700 1 --2- 172.21.15.60:0/3856906137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f480ae750 0x7f8f48065040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:10.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.003+0000 7f8f4e4bd700 1 -- 172.21.15.60:0/3856906137 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f40009470 con 0x7f8f480b1500 2022-06-29T16:04:10.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.003+0000 7f8f4e4bd700 1 --2- 172.21.15.60:0/3856906137 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8f480b1500 0x7f8f48147840 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7f8f400032f0 tx=0x7f8f40016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:10.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.003+0000 7f8f46ffd700 1 -- 172.21.15.60:0/3856906137 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f4000b920 con 0x7f8f480b1500 2022-06-29T16:04:10.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.003+0000 7f8f46ffd700 1 -- 172.21.15.60:0/3856906137 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f40011e20 con 0x7f8f480b1500 2022-06-29T16:04:10.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.003+0000 7f8f46ffd700 1 -- 172.21.15.60:0/3856906137 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f4000b650 con 0x7f8f480b1500 2022-06-29T16:04:10.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.003+0000 7f8f4ecbe700 1 -- 172.21.15.60:0/3856906137 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f48145820 con 0x7f8f480b1500 2022-06-29T16:04:10.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.003+0000 7f8f4ecbe700 1 -- 172.21.15.60:0/3856906137 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f48067a60 con 0x7f8f480b1500 2022-06-29T16:04:10.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.003+0000 7f8f4ecbe700 1 -- 172.21.15.60:0/3856906137 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f48145540 con 0x7f8f480b1500 2022-06-29T16:04:10.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.003+0000 7f8f46ffd700 1 -- 172.21.15.60:0/3856906137 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8f40026030 con 0x7f8f480b1500 2022-06-29T16:04:10.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.007+0000 7f8f46ffd700 1 --2- 172.21.15.60:0/3856906137 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8f28053aa0 0x7f8f28055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:10.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.007+0000 7f8f46ffd700 1 -- 172.21.15.60:0/3856906137 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8f4006ca50 con 0x7f8f480b1500 2022-06-29T16:04:10.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.007+0000 7f8f4dcbc700 1 --2- 172.21.15.60:0/3856906137 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8f28053aa0 0x7f8f28055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:10.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.007+0000 7f8f4dcbc700 1 --2- 172.21.15.60:0/3856906137 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8f28053aa0 0x7f8f28055f60 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f8f3800d000 tx=0x7f8f38007a20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:10.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.015+0000 7f8f46ffd700 1 -- 172.21.15.60:0/3856906137 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8f40043860 con 0x7f8f480b1500 2022-06-29T16:04:10.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.183+0000 7f8f4ecbe700 1 -- 172.21.15.60:0/3856906137 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f8f480573d0 con 0x7f8f480b1500 2022-06-29T16:04:10.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.191+0000 7f8f46ffd700 1 -- 172.21.15.60:0/3856906137 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v25) v1 ==== 184+0+0 (secure 0 0 0) 0x7f8f400150e0 con 0x7f8f480b1500 2022-06-29T16:04:10.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.191+0000 7f8f4ecbe700 1 -- 172.21.15.60:0/3856906137 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8f28053aa0 msgr2=0x7f8f28055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:10.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.191+0000 7f8f4ecbe700 1 --2- 172.21.15.60:0/3856906137 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8f28053aa0 0x7f8f28055f60 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f8f3800d000 tx=0x7f8f38007a20).stop 2022-06-29T16:04:10.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.191+0000 7f8f4ecbe700 1 -- 172.21.15.60:0/3856906137 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8f480b1500 msgr2=0x7f8f48147840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:10.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.191+0000 7f8f4ecbe700 1 --2- 172.21.15.60:0/3856906137 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8f480b1500 0x7f8f48147840 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7f8f400032f0 tx=0x7f8f40016000).stop 2022-06-29T16:04:10.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.191+0000 7f8f4ecbe700 1 -- 172.21.15.60:0/3856906137 shutdown_connections 2022-06-29T16:04:10.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.191+0000 7f8f4ecbe700 1 --2- 172.21.15.60:0/3856906137 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8f28053aa0 0x7f8f28055f60 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:10.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.191+0000 7f8f4ecbe700 1 --2- 172.21.15.60:0/3856906137 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8f480b1500 0x7f8f48147840 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:10.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.191+0000 7f8f4ecbe700 1 --2- 172.21.15.60:0/3856906137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f480ae750 0x7f8f48065040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:10.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.191+0000 7f8f4ecbe700 1 --2- 172.21.15.60:0/3856906137 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8f4805fb00 0x7f8f481554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:10.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.191+0000 7f8f4ecbe700 1 -- 172.21.15.60:0/3856906137 >> 172.21.15.60:0/3856906137 conn(0x7f8f48063ab0 msgr2=0x7f8f481482d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:10.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.191+0000 7f8f4ecbe700 1 -- 172.21.15.60:0/3856906137 shutdown_connections 2022-06-29T16:04:10.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.191+0000 7f8f4ecbe700 1 -- 172.21.15.60:0/3856906137 wait complete. 2022-06-29T16:04:10.202 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.0 2022-06-29T16:04:10.212 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T16:04:10.326 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.319+0000 7f9dda43b700 1 Processor -- start 2022-06-29T16:04:10.326 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.319+0000 7f9dda43b700 1 -- start start 2022-06-29T16:04:10.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.319+0000 7f9dda43b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9dd40ae7c0 0x7f9dd4141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:10.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.319+0000 7f9dda43b700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9dd40b4010 0x7f9dd4142490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:10.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.319+0000 7f9dda43b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9dd4146160 0x7f9dd41465f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:10.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.319+0000 7f9dda43b700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dd4143a40 con 0x7f9dd4146160 2022-06-29T16:04:10.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.319+0000 7f9dda43b700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dd40aee60 con 0x7f9dd40b4010 2022-06-29T16:04:10.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.319+0000 7f9dda43b700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd40af100 con 0x7f9dd40ae7c0 2022-06-29T16:04:10.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.319+0000 7f9dd89d8700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9dd4146160 0x7f9dd41465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:10.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.323+0000 7f9dd89d8700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9dd4146160 0x7f9dd41465f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49326/0 (socket says 172.21.15.60:49326) 2022-06-29T16:04:10.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.323+0000 7f9dd89d8700 1 -- 172.21.15.60:0/3111338109 learned_addr learned my addr 172.21.15.60:0/3111338109 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:10.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.323+0000 7f9dd3fff700 1 --2- 172.21.15.60:0/3111338109 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9dd40ae7c0 0x7f9dd4141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:10.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.323+0000 7f9dd37fe700 1 --2- 172.21.15.60:0/3111338109 >> v2:172.21.15.92:3300/0 conn(0x7f9dd40b4010 0x7f9dd4142490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:10.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.323+0000 7f9dd89d8700 1 -- 172.21.15.60:0/3111338109 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9dd40ae7c0 msgr2=0x7f9dd4141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:10.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.323+0000 7f9dd89d8700 1 --2- 172.21.15.60:0/3111338109 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9dd40ae7c0 0x7f9dd4141fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:10.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.323+0000 7f9dd89d8700 1 -- 172.21.15.60:0/3111338109 >> v2:172.21.15.92:3300/0 conn(0x7f9dd40b4010 msgr2=0x7f9dd4142490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:10.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.323+0000 7f9dd89d8700 1 --2- 172.21.15.60:0/3111338109 >> v2:172.21.15.92:3300/0 conn(0x7f9dd40b4010 0x7f9dd4142490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:10.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.323+0000 7f9dd89d8700 1 -- 172.21.15.60:0/3111338109 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd40aeba0 con 0x7f9dd4146160 2022-06-29T16:04:10.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.323+0000 7f9dd89d8700 1 --2- 172.21.15.60:0/3111338109 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dd4146160 0x7f9dd41465f0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7f9dcc0032f0 tx=0x7f9dcc009290).ready entity=mon.0 client_cookie=764d8487dd3ba8e8 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:10.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.323+0000 7f9dd2ffd700 1 -- 172.21.15.60:0/3111338109 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dcc00b870 con 0x7f9dd4146160 2022-06-29T16:04:10.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.323+0000 7f9dd2ffd700 1 -- 172.21.15.60:0/3111338109 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dcc00b9d0 con 0x7f9dd4146160 2022-06-29T16:04:10.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.323+0000 7f9dd2ffd700 1 -- 172.21.15.60:0/3111338109 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dcc00bb40 con 0x7f9dd4146160 2022-06-29T16:04:10.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.323+0000 7f9dda43b700 1 -- 172.21.15.60:0/3111338109 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dd4146160 msgr2=0x7f9dd41465f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:10.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.323+0000 7f9dda43b700 1 --2- 172.21.15.60:0/3111338109 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dd4146160 0x7f9dd41465f0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7f9dcc0032f0 tx=0x7f9dcc009290).stop 2022-06-29T16:04:10.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.323+0000 7f9dda43b700 1 -- 172.21.15.60:0/3111338109 shutdown_connections 2022-06-29T16:04:10.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.323+0000 7f9dda43b700 1 --2- 172.21.15.60:0/3111338109 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dd4146160 0x7f9dd41465f0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:10.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.323+0000 7f9dda43b700 1 --2- 172.21.15.60:0/3111338109 >> v2:172.21.15.92:3300/0 conn(0x7f9dd40b4010 0x7f9dd4142490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:10.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.323+0000 7f9dda43b700 1 --2- 172.21.15.60:0/3111338109 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9dd40ae7c0 0x7f9dd4141fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:10.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.323+0000 7f9dda43b700 1 -- 172.21.15.60:0/3111338109 >> 172.21.15.60:0/3111338109 conn(0x7f9dd40b9340 msgr2=0x7f9dd40b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:10.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.323+0000 7f9dda43b700 1 -- 172.21.15.60:0/3111338109 shutdown_connections 2022-06-29T16:04:10.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.323+0000 7f9dda43b700 1 -- 172.21.15.60:0/3111338109 wait complete. 2022-06-29T16:04:10.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.327+0000 7f9dda43b700 1 Processor -- start 2022-06-29T16:04:10.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.327+0000 7f9dda43b700 1 -- start start 2022-06-29T16:04:10.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.327+0000 7f9dda43b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dd40ae7c0 0x7f9dd4147fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:10.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.327+0000 7f9dda43b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9dd40b4010 0x7f9dd40af4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:10.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.327+0000 7f9dda43b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dd4146160 0x7f9dd40af9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:10.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.327+0000 7f9dda43b700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd4148720 con 0x7f9dd4146160 2022-06-29T16:04:10.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.327+0000 7f9dda43b700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd4156050 con 0x7f9dd40ae7c0 2022-06-29T16:04:10.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.327+0000 7f9dda43b700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd40b0ae0 con 0x7f9dd40b4010 2022-06-29T16:04:10.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.327+0000 7f9dd89d8700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dd4146160 0x7f9dd40af9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:10.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.327+0000 7f9dd89d8700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dd4146160 0x7f9dd40af9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49332/0 (socket says 172.21.15.60:49332) 2022-06-29T16:04:10.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.327+0000 7f9dd89d8700 1 -- 172.21.15.60:0/2713963714 learned_addr learned my addr 172.21.15.60:0/2713963714 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:10.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.327+0000 7f9dd3fff700 1 --2- 172.21.15.60:0/2713963714 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dd40ae7c0 0x7f9dd4147fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:10.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.327+0000 7f9dd37fe700 1 --2- 172.21.15.60:0/2713963714 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9dd40b4010 0x7f9dd40af4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:10.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.327+0000 7f9dd89d8700 1 -- 172.21.15.60:0/2713963714 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9dd40b4010 msgr2=0x7f9dd40af4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:10.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.327+0000 7f9dd89d8700 1 --2- 172.21.15.60:0/2713963714 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9dd40b4010 0x7f9dd40af4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:10.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.327+0000 7f9dd89d8700 1 -- 172.21.15.60:0/2713963714 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dd40ae7c0 msgr2=0x7f9dd4147fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:10.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.327+0000 7f9dd89d8700 1 --2- 172.21.15.60:0/2713963714 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dd40ae7c0 0x7f9dd4147fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:10.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.327+0000 7f9dd89d8700 1 -- 172.21.15.60:0/2713963714 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dcc009530 con 0x7f9dd4146160 2022-06-29T16:04:10.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.327+0000 7f9dd89d8700 1 --2- 172.21.15.60:0/2713963714 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dd4146160 0x7f9dd40af9d0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 rx=0x7f9dd40b3520 tx=0x7f9dcc016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:10.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.327+0000 7f9dd17fa700 1 -- 172.21.15.60:0/2713963714 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dcc00b870 con 0x7f9dd4146160 2022-06-29T16:04:10.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.327+0000 7f9dd17fa700 1 -- 172.21.15.60:0/2713963714 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dcc021630 con 0x7f9dd4146160 2022-06-29T16:04:10.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.327+0000 7f9dda43b700 1 -- 172.21.15.60:0/2713963714 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd40b0840 con 0x7f9dd4146160 2022-06-29T16:04:10.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.327+0000 7f9dd17fa700 1 -- 172.21.15.60:0/2713963714 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dcc021990 con 0x7f9dd4146160 2022-06-29T16:04:10.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.331+0000 7f9dda43b700 1 -- 172.21.15.60:0/2713963714 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd40b1250 con 0x7f9dd4146160 2022-06-29T16:04:10.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.331+0000 7f9dd17fa700 1 -- 172.21.15.60:0/2713963714 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9dcc027020 con 0x7f9dd4146160 2022-06-29T16:04:10.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.331+0000 7f9dd17fa700 1 --2- 172.21.15.60:0/2713963714 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9db40539e0 0x7f9db4055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:10.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.331+0000 7f9dda43b700 1 -- 172.21.15.60:0/2713963714 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dd41401e0 con 0x7f9dd4146160 2022-06-29T16:04:10.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.331+0000 7f9dd17fa700 1 -- 172.21.15.60:0/2713963714 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9dcc06d120 con 0x7f9dd4146160 2022-06-29T16:04:10.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.331+0000 7f9dd3fff700 1 --2- 172.21.15.60:0/2713963714 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9db40539e0 0x7f9db4055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:10.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.331+0000 7f9dd3fff700 1 --2- 172.21.15.60:0/2713963714 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9db40539e0 0x7f9db4055ea0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f9dc400c000 tx=0x7f9dc4008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:10.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.339+0000 7f9dd17fa700 1 -- 172.21.15.60:0/2713963714 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9dd41401e0 con 0x7f9dd4146160 2022-06-29T16:04:10.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.499+0000 7f9dda43b700 1 -- 172.21.15.60:0/2713963714 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f9dd41401e0 con 0x7f9dd4146160 2022-06-29T16:04:10.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.507+0000 7f9dd17fa700 1 -- 172.21.15.60:0/2713963714 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v26) v1 ==== 184+0+0 (secure 0 0 0) 0x7f9dd41401e0 con 0x7f9dd4146160 2022-06-29T16:04:10.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.511+0000 7f9dda43b700 1 -- 172.21.15.60:0/2713963714 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9db40539e0 msgr2=0x7f9db4055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:10.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.511+0000 7f9dda43b700 1 --2- 172.21.15.60:0/2713963714 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9db40539e0 0x7f9db4055ea0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f9dc400c000 tx=0x7f9dc4008040).stop 2022-06-29T16:04:10.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.511+0000 7f9dda43b700 1 -- 172.21.15.60:0/2713963714 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dd4146160 msgr2=0x7f9dd40af9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:10.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.511+0000 7f9dda43b700 1 --2- 172.21.15.60:0/2713963714 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dd4146160 0x7f9dd40af9d0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 rx=0x7f9dd40b3520 tx=0x7f9dcc016000).stop 2022-06-29T16:04:10.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.511+0000 7f9dda43b700 1 -- 172.21.15.60:0/2713963714 shutdown_connections 2022-06-29T16:04:10.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.511+0000 7f9dda43b700 1 --2- 172.21.15.60:0/2713963714 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9db40539e0 0x7f9db4055ea0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:10.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.511+0000 7f9dda43b700 1 --2- 172.21.15.60:0/2713963714 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9dd4146160 0x7f9dd40af9d0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:10.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.511+0000 7f9dda43b700 1 --2- 172.21.15.60:0/2713963714 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9dd40b4010 0x7f9dd40af4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:10.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.511+0000 7f9dda43b700 1 --2- 172.21.15.60:0/2713963714 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dd40ae7c0 0x7f9dd4147fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:10.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.511+0000 7f9dda43b700 1 -- 172.21.15.60:0/2713963714 >> 172.21.15.60:0/2713963714 conn(0x7f9dd40b9340 msgr2=0x7f9dd40b65f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:10.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.511+0000 7f9dda43b700 1 -- 172.21.15.60:0/2713963714 shutdown_connections 2022-06-29T16:04:10.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.511+0000 7f9dda43b700 1 -- 172.21.15.60:0/2713963714 wait complete. 2022-06-29T16:04:10.521 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.1 2022-06-29T16:04:10.531 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T16:04:10.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.639+0000 7f04d3fff700 1 Processor -- start 2022-06-29T16:04:10.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.639+0000 7f04d3fff700 1 -- start start 2022-06-29T16:04:10.647 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.639+0000 7f04d3fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f04d405c5b0 0x7f04d405c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:10.647 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.639+0000 7f04d3fff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d405fab0 con 0x7f04d40b0c80 2022-06-29T16:04:10.647 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.639+0000 7f04d3fff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d4145fb0 con 0x7f04d405f620 2022-06-29T16:04:10.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.639+0000 7f04d3fff700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04d4146250 con 0x7f04d405c5b0 2022-06-29T16:04:10.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.639+0000 7f04d27fc700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f04d40b0c80 0x7f04d413c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57424/0 (socket says 172.21.15.60:57424) 2022-06-29T16:04:10.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.639+0000 7f04d27fc700 1 -- 172.21.15.60:0/741673176 learned_addr learned my addr 172.21.15.60:0/741673176 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:04:10.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.639+0000 7f04d2ffd700 1 --2- 172.21.15.60:0/741673176 >> v2:172.21.15.92:3300/0 conn(0x7f04d405c5b0 0x7f04d405c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:10.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.643+0000 7f04d1ffb700 1 -- 172.21.15.60:0/741673176 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 445448281 0 0) 0x7f04d405fab0 con 0x7f04d40b0c80 2022-06-29T16:04:10.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.643+0000 7f04d1ffb700 1 -- 172.21.15.60:0/741673176 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04b8003260 con 0x7f04d40b0c80 2022-06-29T16:04:10.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.643+0000 7f04d1ffb700 1 -- 172.21.15.60:0/741673176 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 622245668 0 0) 0x7f04d4145fb0 con 0x7f04d405f620 2022-06-29T16:04:10.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.643+0000 7f04d1ffb700 1 -- 172.21.15.60:0/741673176 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04d405fab0 con 0x7f04d405f620 2022-06-29T16:04:10.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.643+0000 7f04d2ffd700 1 -- 172.21.15.60:0/741673176 >> v1:172.21.15.92:6789/0 conn(0x7f04d405f620 legacy=0x7f04d41429e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:10.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.643+0000 7f04d2ffd700 1 -- 172.21.15.60:0/741673176 >> v1:172.21.15.60:6789/0 conn(0x7f04d40b0c80 legacy=0x7f04d413c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:10.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.643+0000 7f04d2ffd700 1 -- 172.21.15.60:0/741673176 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d41464f0 con 0x7f04d405c5b0 2022-06-29T16:04:10.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.643+0000 7f04d2ffd700 1 --2- 172.21.15.60:0/741673176 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04d405c5b0 0x7f04d405c990 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f04c4002f70 tx=0x7f04c4006df0).ready entity=mon.1 client_cookie=25dcf64f83ea8b9e server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:10.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.643+0000 7f04d1ffb700 1 -- 172.21.15.60:0/741673176 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04c400d540 con 0x7f04d405c5b0 2022-06-29T16:04:10.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.643+0000 7f04d1ffb700 1 -- 172.21.15.60:0/741673176 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04c4000e90 con 0x7f04d405c5b0 2022-06-29T16:04:10.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.643+0000 7f04d1ffb700 1 -- 172.21.15.60:0/741673176 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04c400d890 con 0x7f04d405c5b0 2022-06-29T16:04:10.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.643+0000 7f04d3fff700 1 -- 172.21.15.60:0/741673176 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04d405c5b0 msgr2=0x7f04d405c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:10.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.643+0000 7f04d3fff700 1 --2- 172.21.15.60:0/741673176 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04d405c5b0 0x7f04d405c990 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f04c4002f70 tx=0x7f04c4006df0).stop 2022-06-29T16:04:10.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.643+0000 7f04d3fff700 1 -- 172.21.15.60:0/741673176 shutdown_connections 2022-06-29T16:04:10.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.643+0000 7f04d3fff700 1 --2- 172.21.15.60:0/741673176 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04d405c5b0 0x7f04d405c990 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:10.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.643+0000 7f04d3fff700 1 -- 172.21.15.60:0/741673176 >> 172.21.15.60:0/741673176 conn(0x7f04d40637c0 msgr2=0x7f04d4063ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:10.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.643+0000 7f04d3fff700 1 -- 172.21.15.60:0/741673176 shutdown_connections 2022-06-29T16:04:10.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.643+0000 7f04d3fff700 1 -- 172.21.15.60:0/741673176 wait complete. 2022-06-29T16:04:10.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.643+0000 7f04d3fff700 1 Processor -- start 2022-06-29T16:04:10.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.643+0000 7f04d3fff700 1 -- start start 2022-06-29T16:04:10.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.647+0000 7f04d3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04d405c5b0 0x7f04d41473b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:10.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.647+0000 7f04d3fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04d405f620 0x7f04d4155160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:10.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.647+0000 7f04d3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04d40b0c80 0x7f04d4155650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:10.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.647+0000 7f04d3fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d40b0080 con 0x7f04d40b0c80 2022-06-29T16:04:10.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.647+0000 7f04d3fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d4065600 con 0x7f04d405c5b0 2022-06-29T16:04:10.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.647+0000 7f04d3fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04d40afde0 con 0x7f04d405f620 2022-06-29T16:04:10.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.647+0000 7f04d37fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04d40b0c80 0x7f04d4155650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:10.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.647+0000 7f04d37fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04d40b0c80 0x7f04d4155650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49346/0 (socket says 172.21.15.60:49346) 2022-06-29T16:04:10.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.647+0000 7f04d37fe700 1 -- 172.21.15.60:0/2774601278 learned_addr learned my addr 172.21.15.60:0/2774601278 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:10.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.647+0000 7f04d27fc700 1 --2- 172.21.15.60:0/2774601278 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04d405f620 0x7f04d4155160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:10.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.647+0000 7f04d2ffd700 1 --2- 172.21.15.60:0/2774601278 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04d405c5b0 0x7f04d41473b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:10.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.647+0000 7f04d37fe700 1 -- 172.21.15.60:0/2774601278 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04d405f620 msgr2=0x7f04d4155160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:10.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.647+0000 7f04d37fe700 1 --2- 172.21.15.60:0/2774601278 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04d405f620 0x7f04d4155160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:10.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.647+0000 7f04d37fe700 1 -- 172.21.15.60:0/2774601278 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04d405c5b0 msgr2=0x7f04d41473b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:10.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.647+0000 7f04d37fe700 1 --2- 172.21.15.60:0/2774601278 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04d405c5b0 0x7f04d41473b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:10.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.647+0000 7f04d37fe700 1 -- 172.21.15.60:0/2774601278 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c400b180 con 0x7f04d40b0c80 2022-06-29T16:04:10.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.647+0000 7f04d37fe700 1 --2- 172.21.15.60:0/2774601278 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04d40b0c80 0x7f04d4155650 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0x7f04cc001490 tx=0x7f04cc007dd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:10.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.647+0000 7f04c3fff700 1 -- 172.21.15.60:0/2774601278 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04cc002240 con 0x7f04d40b0c80 2022-06-29T16:04:10.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.647+0000 7f04c3fff700 1 -- 172.21.15.60:0/2774601278 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04cc0023a0 con 0x7f04d40b0c80 2022-06-29T16:04:10.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.647+0000 7f04d3fff700 1 -- 172.21.15.60:0/2774601278 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04d40ad390 con 0x7f04d40b0c80 2022-06-29T16:04:10.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.647+0000 7f04c3fff700 1 -- 172.21.15.60:0/2774601278 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04cc001650 con 0x7f04d40b0c80 2022-06-29T16:04:10.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.647+0000 7f04d3fff700 1 -- 172.21.15.60:0/2774601278 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04d40ad830 con 0x7f04d40b0c80 2022-06-29T16:04:10.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.651+0000 7f04d3fff700 1 -- 172.21.15.60:0/2774601278 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04d40adae0 con 0x7f04d40b0c80 2022-06-29T16:04:10.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.651+0000 7f04c3fff700 1 -- 172.21.15.60:0/2774601278 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f04cc010030 con 0x7f04d40b0c80 2022-06-29T16:04:10.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.651+0000 7f04c3fff700 1 --2- 172.21.15.60:0/2774601278 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f04b0053aa0 0x7f04b0055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:10.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.651+0000 7f04c3fff700 1 -- 172.21.15.60:0/2774601278 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f04cc06beb0 con 0x7f04d40b0c80 2022-06-29T16:04:10.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.651+0000 7f04d2ffd700 1 --2- 172.21.15.60:0/2774601278 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f04b0053aa0 0x7f04b0055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:10.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.651+0000 7f04d2ffd700 1 --2- 172.21.15.60:0/2774601278 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f04b0053aa0 0x7f04b0055f60 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f04c401a000 tx=0x7f04c4014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:10.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.655+0000 7f04c3fff700 1 -- 172.21.15.60:0/2774601278 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f04cc042c40 con 0x7f04d40b0c80 2022-06-29T16:04:10.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.823+0000 7f04d3fff700 1 -- 172.21.15.60:0/2774601278 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f04d40ad520 con 0x7f04d40b0c80 2022-06-29T16:04:10.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.831+0000 7f04c3fff700 1 -- 172.21.15.60:0/2774601278 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v27) v1 ==== 184+0+0 (secure 0 0 0) 0x7f04cc046260 con 0x7f04d40b0c80 2022-06-29T16:04:10.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.831+0000 7f04d3fff700 1 -- 172.21.15.60:0/2774601278 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f04b0053aa0 msgr2=0x7f04b0055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:10.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.831+0000 7f04d3fff700 1 --2- 172.21.15.60:0/2774601278 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f04b0053aa0 0x7f04b0055f60 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f04c401a000 tx=0x7f04c4014000).stop 2022-06-29T16:04:10.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.831+0000 7f04d3fff700 1 -- 172.21.15.60:0/2774601278 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04d40b0c80 msgr2=0x7f04d4155650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:10.843 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.831+0000 7f04d3fff700 1 --2- 172.21.15.60:0/2774601278 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04d40b0c80 0x7f04d4155650 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0x7f04cc001490 tx=0x7f04cc007dd0).stop 2022-06-29T16:04:10.843 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.831+0000 7f04d3fff700 1 -- 172.21.15.60:0/2774601278 shutdown_connections 2022-06-29T16:04:10.843 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.831+0000 7f04d3fff700 1 --2- 172.21.15.60:0/2774601278 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f04b0053aa0 0x7f04b0055f60 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:10.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.831+0000 7f04d3fff700 1 --2- 172.21.15.60:0/2774601278 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04d40b0c80 0x7f04d4155650 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:10.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.831+0000 7f04d3fff700 1 --2- 172.21.15.60:0/2774601278 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04d405f620 0x7f04d4155160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:10.845 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.831+0000 7f04d3fff700 1 --2- 172.21.15.60:0/2774601278 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04d405c5b0 0x7f04d41473b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:10.845 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.831+0000 7f04d3fff700 1 -- 172.21.15.60:0/2774601278 >> 172.21.15.60:0/2774601278 conn(0x7f04d40637c0 msgr2=0x7f04d4145610 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:10.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.831+0000 7f04d3fff700 1 -- 172.21.15.60:0/2774601278 shutdown_connections 2022-06-29T16:04:10.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.835+0000 7f04d3fff700 1 -- 172.21.15.60:0/2774601278 wait complete. 2022-06-29T16:04:10.846 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.2 2022-06-29T16:04:10.853 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:04:10.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.951+0000 7efdd4c35700 1 Processor -- start 2022-06-29T16:04:10.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.951+0000 7efdd4c35700 1 -- start start 2022-06-29T16:04:10.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.951+0000 7efdd4c35700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7efdd00b2220 0x7efdd0141130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:10.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.951+0000 7efdd4c35700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd00b2630 con 0x7efdd01449c0 2022-06-29T16:04:10.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.951+0000 7efdd4c35700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd005c770 con 0x7efdd005c390 2022-06-29T16:04:10.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.951+0000 7efdd4c35700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdd0140750 con 0x7efdd00b2220 2022-06-29T16:04:10.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.951+0000 7efdcdd9b700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7efdd01449c0 0x7efdd0140a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57440/0 (socket says 172.21.15.60:57440) 2022-06-29T16:04:10.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.951+0000 7efdcdd9b700 1 -- 172.21.15.60:0/3228889617 learned_addr learned my addr 172.21.15.60:0/3228889617 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:04:10.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.951+0000 7efdced9d700 1 --2- 172.21.15.60:0/3228889617 >> v2:172.21.15.92:3300/0 conn(0x7efdd00b2220 0x7efdd0141130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:10.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.951+0000 7efdcd59a700 1 -- 172.21.15.60:0/3228889617 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4219208925 0 0) 0x7efdd00b2630 con 0x7efdd01449c0 2022-06-29T16:04:10.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.951+0000 7efdcd59a700 1 -- 172.21.15.60:0/3228889617 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdb4003260 con 0x7efdd01449c0 2022-06-29T16:04:10.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.951+0000 7efdcd59a700 1 -- 172.21.15.60:0/3228889617 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 838536930 0 0) 0x7efdd005c770 con 0x7efdd005c390 2022-06-29T16:04:10.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.951+0000 7efdcd59a700 1 -- 172.21.15.60:0/3228889617 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdd00b2630 con 0x7efdd005c390 2022-06-29T16:04:10.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.951+0000 7efdced9d700 1 -- 172.21.15.60:0/3228889617 >> v1:172.21.15.92:6789/0 conn(0x7efdd005c390 legacy=0x7efdd01442b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:10.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.951+0000 7efdced9d700 1 -- 172.21.15.60:0/3228889617 >> v1:172.21.15.60:6789/0 conn(0x7efdd01449c0 legacy=0x7efdd0140a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:10.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.951+0000 7efdced9d700 1 -- 172.21.15.60:0/3228889617 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd0143e80 con 0x7efdd00b2220 2022-06-29T16:04:10.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.951+0000 7efdced9d700 1 --2- 172.21.15.60:0/3228889617 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efdd00b2220 0x7efdd0141130 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7efdc80032f0 tx=0x7efdc80091d0).ready entity=mon.1 client_cookie=11e476d394d8e7a4 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:10.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.951+0000 7efdcd59a700 1 -- 172.21.15.60:0/3228889617 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efdc800b600 con 0x7efdd00b2220 2022-06-29T16:04:10.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.951+0000 7efdcd59a700 1 -- 172.21.15.60:0/3228889617 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdc800b760 con 0x7efdd00b2220 2022-06-29T16:04:10.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.951+0000 7efdcd59a700 1 -- 172.21.15.60:0/3228889617 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efdc800b8d0 con 0x7efdd00b2220 2022-06-29T16:04:10.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.955+0000 7efdd4c35700 1 -- 172.21.15.60:0/3228889617 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efdd00b2220 msgr2=0x7efdd0141130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:10.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.955+0000 7efdd4c35700 1 --2- 172.21.15.60:0/3228889617 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efdd00b2220 0x7efdd0141130 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7efdc80032f0 tx=0x7efdc80091d0).stop 2022-06-29T16:04:10.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.955+0000 7efdd4c35700 1 -- 172.21.15.60:0/3228889617 shutdown_connections 2022-06-29T16:04:10.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.955+0000 7efdd4c35700 1 --2- 172.21.15.60:0/3228889617 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efdd00b2220 0x7efdd0141130 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:10.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.955+0000 7efdd4c35700 1 -- 172.21.15.60:0/3228889617 >> 172.21.15.60:0/3228889617 conn(0x7efdd0064a10 msgr2=0x7efdd0064df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:10.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.955+0000 7efdd4c35700 1 -- 172.21.15.60:0/3228889617 shutdown_connections 2022-06-29T16:04:10.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.955+0000 7efdd4c35700 1 -- 172.21.15.60:0/3228889617 wait complete. 2022-06-29T16:04:10.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.955+0000 7efdd4c35700 1 Processor -- start 2022-06-29T16:04:10.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.955+0000 7efdd4c35700 1 -- start start 2022-06-29T16:04:10.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.955+0000 7efdd4c35700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efdd005c390 0x7efdd0142380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:10.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.955+0000 7efdd4c35700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdd00b2220 0x7efdd0063010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:10.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.955+0000 7efdd4c35700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdd01449c0 0x7efdd0147ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:10.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.955+0000 7efdd4c35700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd0142b60 con 0x7efdd00b2220 2022-06-29T16:04:10.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.955+0000 7efdd4c35700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd00ae1b0 con 0x7efdd005c390 2022-06-29T16:04:10.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.955+0000 7efdd4c35700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdd00ae450 con 0x7efdd01449c0 2022-06-29T16:04:10.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.955+0000 7efdcdd9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdd00b2220 0x7efdd0063010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:10.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.955+0000 7efdcdd9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdd00b2220 0x7efdd0063010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49356/0 (socket says 172.21.15.60:49356) 2022-06-29T16:04:10.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.955+0000 7efdcdd9b700 1 -- 172.21.15.60:0/2879002188 learned_addr learned my addr 172.21.15.60:0/2879002188 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:10.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.955+0000 7efdced9d700 1 --2- 172.21.15.60:0/2879002188 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdd01449c0 0x7efdd0147ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:10.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.955+0000 7efdce59c700 1 --2- 172.21.15.60:0/2879002188 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efdd005c390 0x7efdd0142380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:10.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.959+0000 7efdcdd9b700 1 -- 172.21.15.60:0/2879002188 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdd01449c0 msgr2=0x7efdd0147ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:10.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.959+0000 7efdcdd9b700 1 --2- 172.21.15.60:0/2879002188 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdd01449c0 0x7efdd0147ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:10.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.959+0000 7efdcdd9b700 1 -- 172.21.15.60:0/2879002188 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efdd005c390 msgr2=0x7efdd0142380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:10.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.959+0000 7efdcdd9b700 1 --2- 172.21.15.60:0/2879002188 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efdd005c390 0x7efdd0142380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:10.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.959+0000 7efdcdd9b700 1 -- 172.21.15.60:0/2879002188 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc8009470 con 0x7efdd00b2220 2022-06-29T16:04:10.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.959+0000 7efdced9d700 1 --2- 172.21.15.60:0/2879002188 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdd01449c0 0x7efdd0147ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:04:10.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.959+0000 7efdcdd9b700 1 --2- 172.21.15.60:0/2879002188 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdd00b2220 0x7efdd0063010 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7efdc40011d0 tx=0x7efdc4007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:10.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.959+0000 7efdbf7fe700 1 -- 172.21.15.60:0/2879002188 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efdc40070d0 con 0x7efdd00b2220 2022-06-29T16:04:10.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.959+0000 7efdbf7fe700 1 -- 172.21.15.60:0/2879002188 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdc4007230 con 0x7efdd00b2220 2022-06-29T16:04:10.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.959+0000 7efdd4c35700 1 -- 172.21.15.60:0/2879002188 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdd00ae6f0 con 0x7efdd00b2220 2022-06-29T16:04:10.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.959+0000 7efdbf7fe700 1 -- 172.21.15.60:0/2879002188 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efdc4005610 con 0x7efdd00b2220 2022-06-29T16:04:10.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.959+0000 7efdd4c35700 1 -- 172.21.15.60:0/2879002188 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdd0155c80 con 0x7efdd00b2220 2022-06-29T16:04:10.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.959+0000 7efdd4c35700 1 -- 172.21.15.60:0/2879002188 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdd0155f90 con 0x7efdd00b2220 2022-06-29T16:04:10.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.959+0000 7efdbf7fe700 1 -- 172.21.15.60:0/2879002188 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7efdc4010070 con 0x7efdd00b2220 2022-06-29T16:04:10.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.959+0000 7efdbf7fe700 1 --2- 172.21.15.60:0/2879002188 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efdac053b00 0x7efdac055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:10.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.959+0000 7efdbf7fe700 1 -- 172.21.15.60:0/2879002188 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7efdc406b630 con 0x7efdd00b2220 2022-06-29T16:04:10.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.963+0000 7efdce59c700 1 --2- 172.21.15.60:0/2879002188 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efdac053b00 0x7efdac055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:10.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.963+0000 7efdce59c700 1 --2- 172.21.15.60:0/2879002188 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efdac053b00 0x7efdac055fc0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7efdc000d000 tx=0x7efdc0007a20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:10.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:10.967+0000 7efdbf7fe700 1 -- 172.21.15.60:0/2879002188 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efdc40424b0 con 0x7efdd00b2220 2022-06-29T16:04:11.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.135+0000 7efdd4c35700 1 -- 172.21.15.60:0/2879002188 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efdd00573d0 con 0x7efdd00b2220 2022-06-29T16:04:11.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.135+0000 7efdbf7fe700 1 -- 172.21.15.60:0/2879002188 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2137 (secure 0 0 0) 0x7efdc4045ad0 con 0x7efdd00b2220 2022-06-29T16:04:11.142 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:04:11.142 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":30,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5690,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1158969084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1158969084},{"type":"v1","addr":"172.21.15.60:6833","nonce":1158969084}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":28}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:04:08.275039+0000","modified":"2022-06-29T16:04:09.279046+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6304":{"gid":6304,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/953938586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":953938586},{"type":"v1","addr":"172.21.15.92:6835","nonce":953938586}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-06-29T16:04:11.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.135+0000 7efdd4c35700 1 -- 172.21.15.60:0/2879002188 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efdac053b00 msgr2=0x7efdac055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:11.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.135+0000 7efdd4c35700 1 --2- 172.21.15.60:0/2879002188 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efdac053b00 0x7efdac055fc0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7efdc000d000 tx=0x7efdc0007a20).stop 2022-06-29T16:04:11.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.135+0000 7efdd4c35700 1 -- 172.21.15.60:0/2879002188 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdd00b2220 msgr2=0x7efdd0063010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:11.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.135+0000 7efdd4c35700 1 --2- 172.21.15.60:0/2879002188 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdd00b2220 0x7efdd0063010 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7efdc40011d0 tx=0x7efdc4007b50).stop 2022-06-29T16:04:11.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.139+0000 7efdd4c35700 1 -- 172.21.15.60:0/2879002188 shutdown_connections 2022-06-29T16:04:11.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.139+0000 7efdd4c35700 1 --2- 172.21.15.60:0/2879002188 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efdac053b00 0x7efdac055fc0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:11.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.139+0000 7efdd4c35700 1 --2- 172.21.15.60:0/2879002188 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdd01449c0 0x7efdd0147ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:11.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.139+0000 7efdd4c35700 1 --2- 172.21.15.60:0/2879002188 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdd00b2220 0x7efdd0063010 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:11.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.139+0000 7efdd4c35700 1 --2- 172.21.15.60:0/2879002188 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efdd005c390 0x7efdd0142380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:11.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.139+0000 7efdd4c35700 1 -- 172.21.15.60:0/2879002188 >> 172.21.15.60:0/2879002188 conn(0x7efdd0064a10 msgr2=0x7efdd00b17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:11.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.139+0000 7efdd4c35700 1 -- 172.21.15.60:0/2879002188 shutdown_connections 2022-06-29T16:04:11.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.139+0000 7efdd4c35700 1 -- 172.21.15.60:0/2879002188 wait complete. 2022-06-29T16:04:11.148 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 30 2022-06-29T16:04:11.357 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:04:08.275039+0000', 'modified': '2022-06-29T16:04:09.279046+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6304}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6304': {'gid': 6304, 'name': 'b', 'rank': 0, 'incarnation': 29, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.92:6835/953938586', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 953938586}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 953938586}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:04:11.358 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:04:11.359 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-06-29T16:04:11.476 INFO:teuthology.orchestra.run.smithi092.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":6304,"want_state":"up:active","state":"up:active","rank_uptime":3.1854535639999999,"mdsmap_epoch":30,"osdmap_epoch":71,"osdmap_epoch_barrier":71,"uptime":7.5665804359999997} 2022-06-29T16:04:11.477 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 6304, "mdsmap_epoch": 30, "osdmap_epoch": 71, "osdmap_epoch_barrier": 71, "rank_uptime": 3.185453564, "state": "up:active", "uptime": 7.566580436, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:04:11.477 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:04:11.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.583+0000 7f075885e700 1 Processor -- start 2022-06-29T16:04:11.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.583+0000 7f075885e700 1 -- start start 2022-06-29T16:04:11.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.583+0000 7f075885e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f075405d070 0x7f075414aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:11.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.583+0000 7f075885e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f075405d450 con 0x7f07540b2130 2022-06-29T16:04:11.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.583+0000 7f075885e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07540ad840 con 0x7f075414af80 2022-06-29T16:04:11.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.583+0000 7f075885e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07540adf90 con 0x7f075405d070 2022-06-29T16:04:11.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.583+0000 7f074a59c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f07540b2130 0x7f07540aca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57452/0 (socket says 172.21.15.60:57452) 2022-06-29T16:04:11.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.583+0000 7f074a59c700 1 -- 172.21.15.60:0/862783899 learned_addr learned my addr 172.21.15.60:0/862783899 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:04:11.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.583+0000 7f0752d9d700 1 --2- 172.21.15.60:0/862783899 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f075405d070 0x7f075414aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:11.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.583+0000 7f075259c700 1 -- 172.21.15.60:0/862783899 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2592029378 0 0) 0x7f075405d450 con 0x7f07540b2130 2022-06-29T16:04:11.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.583+0000 7f075259c700 1 -- 172.21.15.60:0/862783899 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0734003260 con 0x7f07540b2130 2022-06-29T16:04:11.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.583+0000 7f0752d9d700 1 -- 172.21.15.60:0/862783899 >> v1:172.21.15.92:6789/0 conn(0x7f075414af80 legacy=0x7f07540ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:11.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.583+0000 7f0752d9d700 1 -- 172.21.15.60:0/862783899 >> v1:172.21.15.60:6789/0 conn(0x7f07540b2130 legacy=0x7f07540aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:11.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.583+0000 7f0752d9d700 1 -- 172.21.15.60:0/862783899 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07540b25d0 con 0x7f075405d070 2022-06-29T16:04:11.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.587+0000 7f0752d9d700 1 --2- 172.21.15.60:0/862783899 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f075405d070 0x7f075414aa90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f0744002f70 tx=0x7f0744006df0).ready entity=mon.2 client_cookie=d8c53af91633e5d1 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:11.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.587+0000 7f075259c700 1 -- 172.21.15.60:0/862783899 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f074400d570 con 0x7f075405d070 2022-06-29T16:04:11.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.587+0000 7f075259c700 1 -- 172.21.15.60:0/862783899 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0744000e90 con 0x7f075405d070 2022-06-29T16:04:11.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.587+0000 7f075259c700 1 -- 172.21.15.60:0/862783899 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f074400d8c0 con 0x7f075405d070 2022-06-29T16:04:11.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.587+0000 7f075885e700 1 -- 172.21.15.60:0/862783899 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f075405d070 msgr2=0x7f075414aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:11.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.587+0000 7f075885e700 1 --2- 172.21.15.60:0/862783899 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f075405d070 0x7f075414aa90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f0744002f70 tx=0x7f0744006df0).stop 2022-06-29T16:04:11.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.587+0000 7f075885e700 1 -- 172.21.15.60:0/862783899 shutdown_connections 2022-06-29T16:04:11.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.587+0000 7f075885e700 1 --2- 172.21.15.60:0/862783899 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f075405d070 0x7f075414aa90 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:11.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.587+0000 7f075885e700 1 -- 172.21.15.60:0/862783899 >> 172.21.15.60:0/862783899 conn(0x7f07540b5890 msgr2=0x7f07540b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:11.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.587+0000 7f075885e700 1 -- 172.21.15.60:0/862783899 shutdown_connections 2022-06-29T16:04:11.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.587+0000 7f075885e700 1 -- 172.21.15.60:0/862783899 wait complete. 2022-06-29T16:04:11.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.587+0000 7f075885e700 1 Processor -- start 2022-06-29T16:04:11.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.587+0000 7f075885e700 1 -- start start 2022-06-29T16:04:11.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.587+0000 7f075885e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f075405d070 0x7f07540b4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:11.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.587+0000 7f075885e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07540b2130 0x7f0754065590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:11.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.587+0000 7f075885e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f075414af80 0x7f0754066640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:11.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.587+0000 7f075885e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0754116f90 con 0x7f075414af80 2022-06-29T16:04:11.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.587+0000 7f075885e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075415a410 con 0x7f075405d070 2022-06-29T16:04:11.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.587+0000 7f075885e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f075415a6b0 con 0x7f07540b2130 2022-06-29T16:04:11.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.587+0000 7f075359e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f075414af80 0x7f0754066640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:11.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.591+0000 7f074a59c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07540b2130 0x7f0754065590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:11.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.591+0000 7f075359e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f075414af80 0x7f0754066640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49374/0 (socket says 172.21.15.60:49374) 2022-06-29T16:04:11.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.591+0000 7f074a59c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07540b2130 0x7f0754065590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:34232/0 (socket says 172.21.15.60:34232) 2022-06-29T16:04:11.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.591+0000 7f0752d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f075405d070 0x7f07540b4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:11.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.591+0000 7f075359e700 1 -- 172.21.15.60:0/452164279 learned_addr learned my addr 172.21.15.60:0/452164279 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:11.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.591+0000 7f075359e700 1 -- 172.21.15.60:0/452164279 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07540b2130 msgr2=0x7f0754065590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:11.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.591+0000 7f075359e700 1 --2- 172.21.15.60:0/452164279 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07540b2130 0x7f0754065590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:11.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.591+0000 7f075359e700 1 -- 172.21.15.60:0/452164279 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f075405d070 msgr2=0x7f07540b4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:11.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.591+0000 7f075359e700 1 --2- 172.21.15.60:0/452164279 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f075405d070 0x7f07540b4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:11.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.591+0000 7f075359e700 1 -- 172.21.15.60:0/452164279 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f074400b180 con 0x7f075414af80 2022-06-29T16:04:11.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.591+0000 7f075359e700 1 --2- 172.21.15.60:0/452164279 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f075414af80 0x7f0754066640 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0x7f0754061990 tx=0x7f074c007ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:11.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.591+0000 7f0750d99700 1 -- 172.21.15.60:0/452164279 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f074c00ce20 con 0x7f075414af80 2022-06-29T16:04:11.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.591+0000 7f0750d99700 1 -- 172.21.15.60:0/452164279 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f074c00f470 con 0x7f075414af80 2022-06-29T16:04:11.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.591+0000 7f075885e700 1 -- 172.21.15.60:0/452164279 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07540b4850 con 0x7f075414af80 2022-06-29T16:04:11.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.591+0000 7f0750d99700 1 -- 172.21.15.60:0/452164279 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f074c00f7d0 con 0x7f075414af80 2022-06-29T16:04:11.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.591+0000 7f075885e700 1 -- 172.21.15.60:0/452164279 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0754116c10 con 0x7f075414af80 2022-06-29T16:04:11.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.591+0000 7f075885e700 1 -- 172.21.15.60:0/452164279 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f075414a350 con 0x7f075414af80 2022-06-29T16:04:11.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.591+0000 7f0750d99700 1 -- 172.21.15.60:0/452164279 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f074c019030 con 0x7f075414af80 2022-06-29T16:04:11.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.591+0000 7f0750d99700 1 --2- 172.21.15.60:0/452164279 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f07380539e0 0x7f0738055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:11.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.591+0000 7f0750d99700 1 -- 172.21.15.60:0/452164279 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f074c06b370 con 0x7f075414af80 2022-06-29T16:04:11.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.595+0000 7f0752d9d700 1 --2- 172.21.15.60:0/452164279 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f07380539e0 0x7f0738055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:11.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.595+0000 7f0752d9d700 1 --2- 172.21.15.60:0/452164279 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f07380539e0 0x7f0738055ea0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f074401a000 tx=0x7f0744014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:11.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.599+0000 7f0750d99700 1 -- 172.21.15.60:0/452164279 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f074c042290 con 0x7f075414af80 2022-06-29T16:04:11.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.723+0000 7f075885e700 1 -- 172.21.15.60:0/452164279 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07541562e0 con 0x7f075414af80 2022-06-29T16:04:11.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.727+0000 7f0750d99700 1 -- 172.21.15.60:0/452164279 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16041 (secure 0 0 0) 0x7f074c0458b0 con 0x7f075414af80 2022-06-29T16:04:11.733 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:04:11.734 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":71,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:04:09.277985+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:04:06.708236+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2022-06-29T16:04:07.305396+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:04:11.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.731+0000 7f075885e700 1 -- 172.21.15.60:0/452164279 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f07380539e0 msgr2=0x7f0738055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:11.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.731+0000 7f075885e700 1 --2- 172.21.15.60:0/452164279 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f07380539e0 0x7f0738055ea0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f074401a000 tx=0x7f0744014000).stop 2022-06-29T16:04:11.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.731+0000 7f075885e700 1 -- 172.21.15.60:0/452164279 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f075414af80 msgr2=0x7f0754066640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:11.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.731+0000 7f075885e700 1 --2- 172.21.15.60:0/452164279 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f075414af80 0x7f0754066640 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0x7f0754061990 tx=0x7f074c007ef0).stop 2022-06-29T16:04:11.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.731+0000 7f075885e700 1 -- 172.21.15.60:0/452164279 shutdown_connections 2022-06-29T16:04:11.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.731+0000 7f075885e700 1 --2- 172.21.15.60:0/452164279 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f07380539e0 0x7f0738055ea0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:11.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.731+0000 7f075885e700 1 --2- 172.21.15.60:0/452164279 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f075414af80 0x7f0754066640 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:11.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.731+0000 7f075885e700 1 --2- 172.21.15.60:0/452164279 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07540b2130 0x7f0754065590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:11.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.731+0000 7f075885e700 1 --2- 172.21.15.60:0/452164279 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f075405d070 0x7f07540b4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:11.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.731+0000 7f075885e700 1 -- 172.21.15.60:0/452164279 >> 172.21.15.60:0/452164279 conn(0x7f07540b5890 msgr2=0x7f07540aebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:11.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.731+0000 7f075885e700 1 -- 172.21.15.60:0/452164279 shutdown_connections 2022-06-29T16:04:11.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.731+0000 7f075885e700 1 -- 172.21.15.60:0/452164279 wait complete. 2022-06-29T16:04:11.752 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-29T16:04:06.708236+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2022-06-29T16:04:07.305396+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-29T16:04:11.753 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:04:11.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.863+0000 7fefaee51700 1 Processor -- start 2022-06-29T16:04:11.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.863+0000 7fefaee51700 1 -- start start 2022-06-29T16:04:11.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.863+0000 7fefaee51700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fefa80b9d70 0x7fefa81465d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:11.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.863+0000 7fefaee51700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefa80b4f50 con 0x7fefa80b3d30 2022-06-29T16:04:11.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.863+0000 7fefaee51700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefa81461a0 con 0x7fefa8061d90 2022-06-29T16:04:11.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.863+0000 7fefaee51700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefa80ba150 con 0x7fefa80b9d70 2022-06-29T16:04:11.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.863+0000 7fefa7fff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fefa80b3d30 0x7fefa8146ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57466/0 (socket says 172.21.15.60:57466) 2022-06-29T16:04:11.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.863+0000 7fefa7fff700 1 -- 172.21.15.60:0/2366714859 learned_addr learned my addr 172.21.15.60:0/2366714859 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:04:11.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefacbed700 1 --2- 172.21.15.60:0/2366714859 >> v2:172.21.15.92:3300/0 conn(0x7fefa80b9d70 0x7fefa81465d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:11.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefa77fe700 1 -- 172.21.15.60:0/2366714859 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044351440 0 0) 0x7fefa80b4f50 con 0x7fefa80b3d30 2022-06-29T16:04:11.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefa77fe700 1 -- 172.21.15.60:0/2366714859 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef94003260 con 0x7fefa80b3d30 2022-06-29T16:04:11.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefa77fe700 1 -- 172.21.15.60:0/2366714859 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 895906722 0 0) 0x7fefa81461a0 con 0x7fefa8061d90 2022-06-29T16:04:11.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefa77fe700 1 -- 172.21.15.60:0/2366714859 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefa80b4f50 con 0x7fefa8061d90 2022-06-29T16:04:11.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefa77fe700 1 -- 172.21.15.60:0/2366714859 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2753928380 0 0) 0x7fef94003260 con 0x7fefa80b3d30 2022-06-29T16:04:11.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefa77fe700 1 -- 172.21.15.60:0/2366714859 >> v2:172.21.15.92:3300/0 conn(0x7fefa80b9d70 msgr2=0x7fefa81465d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:11.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefa77fe700 1 --2- 172.21.15.60:0/2366714859 >> v2:172.21.15.92:3300/0 conn(0x7fefa80b9d70 0x7fefa81465d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:11.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefa77fe700 1 -- 172.21.15.60:0/2366714859 >> v1:172.21.15.92:6789/0 conn(0x7fefa8061d90 legacy=0x7fefa8142b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:11.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefa77fe700 1 -- 172.21.15.60:0/2366714859 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa80b5180 con 0x7fefa80b3d30 2022-06-29T16:04:11.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefa77fe700 1 -- 172.21.15.60:0/2366714859 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7fef98001660 con 0x7fefa80b3d30 2022-06-29T16:04:11.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefa77fe700 1 -- 172.21.15.60:0/2366714859 >> v1:172.21.15.60:6789/0 conn(0x7fefa80b3d30 legacy=0x7fefa8146ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:11.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefa77fe700 1 --2- 172.21.15.60:0/2366714859 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef940038e0 0x7fef94003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:11.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefa77fe700 1 --2- 172.21.15.60:0/2366714859 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef94004500 0x7fef94004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:11.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefa77fe700 1 --2- 172.21.15.60:0/2366714859 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef94004e70 0x7fef9400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:11.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefa77fe700 1 -- 172.21.15.60:0/2366714859 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa80ba150 con 0x7fef94004e70 2022-06-29T16:04:11.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefa77fe700 1 -- 172.21.15.60:0/2366714859 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9400bcf0 con 0x7fef940038e0 2022-06-29T16:04:11.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefa77fe700 1 -- 172.21.15.60:0/2366714859 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef9400beb0 con 0x7fef94004500 2022-06-29T16:04:11.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefacbed700 1 --2- 172.21.15.60:0/2366714859 >> v2:172.21.15.92:3300/0 conn(0x7fefa80b9d70 0x7fefa81465d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:04:11.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefad3ee700 1 --2- 172.21.15.60:0/2366714859 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef94004e70 0x7fef9400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:11.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefa7fff700 1 --2- 172.21.15.60:0/2366714859 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef94004500 0x7fef94004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:11.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefacbed700 1 --2- 172.21.15.60:0/2366714859 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef940038e0 0x7fef94003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:11.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefad3ee700 1 -- 172.21.15.60:0/2366714859 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef94004500 msgr2=0x7fef94004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:11.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefad3ee700 1 --2- 172.21.15.60:0/2366714859 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef94004500 0x7fef94004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:11.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefad3ee700 1 -- 172.21.15.60:0/2366714859 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef940038e0 msgr2=0x7fef94003ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:11.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefad3ee700 1 --2- 172.21.15.60:0/2366714859 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef940038e0 0x7fef94003ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:11.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefad3ee700 1 -- 172.21.15.60:0/2366714859 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef94003260 con 0x7fef94004e70 2022-06-29T16:04:11.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefa7fff700 1 --2- 172.21.15.60:0/2366714859 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef94004500 0x7fef94004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:04:11.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefacbed700 1 --2- 172.21.15.60:0/2366714859 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef940038e0 0x7fef94003ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:04:11.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefad3ee700 1 -- 172.21.15.60:0/2366714859 reap_dead start 2022-06-29T16:04:11.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefad3ee700 1 --2- 172.21.15.60:0/2366714859 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef94004e70 0x7fef9400b750 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 rx=0x7fefa0001490 tx=0x7fefa0007d40).ready entity=mon.0 client_cookie=76d262c1815b528f server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:11.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefa77fe700 1 -- 172.21.15.60:0/2366714859 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fefa0006890 con 0x7fef94004e70 2022-06-29T16:04:11.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefa77fe700 1 -- 172.21.15.60:0/2366714859 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefa00069f0 con 0x7fef94004e70 2022-06-29T16:04:11.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefa77fe700 1 -- 172.21.15.60:0/2366714859 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fefa0006d50 con 0x7fef94004e70 2022-06-29T16:04:11.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefaee51700 1 -- 172.21.15.60:0/2366714859 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef94004e70 msgr2=0x7fef9400b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:11.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefaee51700 1 --2- 172.21.15.60:0/2366714859 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef94004e70 0x7fef9400b750 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 rx=0x7fefa0001490 tx=0x7fefa0007d40).stop 2022-06-29T16:04:11.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefaee51700 1 -- 172.21.15.60:0/2366714859 shutdown_connections 2022-06-29T16:04:11.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefaee51700 1 --2- 172.21.15.60:0/2366714859 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef94004e70 0x7fef9400b750 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:11.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefaee51700 1 -- 172.21.15.60:0/2366714859 >> 172.21.15.60:0/2366714859 conn(0x7fefa8062e70 msgr2=0x7fefa80b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:11.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefaee51700 1 -- 172.21.15.60:0/2366714859 shutdown_connections 2022-06-29T16:04:11.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.867+0000 7fefaee51700 1 -- 172.21.15.60:0/2366714859 wait complete. 2022-06-29T16:04:11.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.871+0000 7fefaee51700 1 Processor -- start 2022-06-29T16:04:11.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.871+0000 7fefaee51700 1 -- start start 2022-06-29T16:04:11.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.871+0000 7fefaee51700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef94004e70 0x7fefa8152740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:11.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.871+0000 7fefaee51700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa80aca20 0x7fefa8140470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:11.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.871+0000 7fefaee51700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fefa8140960 0x7fefa8145730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:11.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.871+0000 7fefaee51700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa80ad470 con 0x7fefa80aca20 2022-06-29T16:04:11.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.871+0000 7fefaee51700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa8152c30 con 0x7fefa8140960 2022-06-29T16:04:11.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.871+0000 7fefaee51700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefa80acf30 con 0x7fef94004e70 2022-06-29T16:04:11.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.871+0000 7fefa7fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa80aca20 0x7fefa8140470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:11.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.871+0000 7fefa7fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa80aca20 0x7fefa8140470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49394/0 (socket says 172.21.15.60:49394) 2022-06-29T16:04:11.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.871+0000 7fefa7fff700 1 -- 172.21.15.60:0/669266875 learned_addr learned my addr 172.21.15.60:0/669266875 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:11.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.871+0000 7fefad3ee700 1 --2- 172.21.15.60:0/669266875 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fefa8140960 0x7fefa8145730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:11.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.871+0000 7fefacbed700 1 --2- 172.21.15.60:0/669266875 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef94004e70 0x7fefa8152740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:11.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.871+0000 7fefa7fff700 1 -- 172.21.15.60:0/669266875 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef94004e70 msgr2=0x7fefa8152740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:11.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.871+0000 7fefa7fff700 1 --2- 172.21.15.60:0/669266875 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef94004e70 0x7fefa8152740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:11.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.871+0000 7fefa7fff700 1 -- 172.21.15.60:0/669266875 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fefa8140960 msgr2=0x7fefa8145730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:11.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.871+0000 7fefa7fff700 1 --2- 172.21.15.60:0/669266875 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fefa8140960 0x7fefa8145730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:11.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.871+0000 7fefa7fff700 1 -- 172.21.15.60:0/669266875 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa80ad1d0 con 0x7fefa80aca20 2022-06-29T16:04:11.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.871+0000 7fefad3ee700 1 --2- 172.21.15.60:0/669266875 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fefa8140960 0x7fefa8145730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:04:11.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.871+0000 7fefa7fff700 1 --2- 172.21.15.60:0/669266875 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa80aca20 0x7fefa8140470 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0x7fefa80ad5b0 tx=0x7fef98009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:11.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.871+0000 7fefa5ffb700 1 -- 172.21.15.60:0/669266875 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef98007170 con 0x7fefa80aca20 2022-06-29T16:04:11.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.871+0000 7fefa5ffb700 1 -- 172.21.15.60:0/669266875 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef980072d0 con 0x7fefa80aca20 2022-06-29T16:04:11.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.871+0000 7fefaee51700 1 -- 172.21.15.60:0/669266875 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefa8144e30 con 0x7fefa80aca20 2022-06-29T16:04:11.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.871+0000 7fefa5ffb700 1 -- 172.21.15.60:0/669266875 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef98004920 con 0x7fefa80aca20 2022-06-29T16:04:11.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.871+0000 7fefaee51700 1 -- 172.21.15.60:0/669266875 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefa8145dc0 con 0x7fefa80aca20 2022-06-29T16:04:11.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.875+0000 7fefa5ffb700 1 -- 172.21.15.60:0/669266875 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fef98004a80 con 0x7fefa80aca20 2022-06-29T16:04:11.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.875+0000 7fefa5ffb700 1 --2- 172.21.15.60:0/669266875 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fef88053b00 0x7fef88055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:11.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.875+0000 7fefa5ffb700 1 -- 172.21.15.60:0/669266875 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fef9806bd40 con 0x7fefa80aca20 2022-06-29T16:04:11.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.875+0000 7fefacbed700 1 --2- 172.21.15.60:0/669266875 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fef88053b00 0x7fef88055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:11.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.875+0000 7fefaee51700 1 -- 172.21.15.60:0/669266875 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefa80573d0 con 0x7fefa80aca20 2022-06-29T16:04:11.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.875+0000 7fefacbed700 1 --2- 172.21.15.60:0/669266875 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fef88053b00 0x7fef88055fc0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7fef9c00b000 tx=0x7fef9c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:11.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:11.879+0000 7fefa5ffb700 1 -- 172.21.15.60:0/669266875 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fef98042bc0 con 0x7fefa80aca20 2022-06-29T16:04:12.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.007+0000 7fefaee51700 1 -- 172.21.15.60:0/669266875 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fefa805da40 con 0x7fefa80aca20 2022-06-29T16:04:12.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.007+0000 7fefa5ffb700 1 -- 172.21.15.60:0/669266875 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v71) v1 ==== 110+0+10 (secure 0 0 0) 0x7fef980461e0 con 0x7fefa80aca20 2022-06-29T16:04:12.015 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:04:12.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.007+0000 7fefaee51700 1 -- 172.21.15.60:0/669266875 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fef88053b00 msgr2=0x7fef88055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:12.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.007+0000 7fefaee51700 1 --2- 172.21.15.60:0/669266875 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fef88053b00 0x7fef88055fc0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7fef9c00b000 tx=0x7fef9c008040).stop 2022-06-29T16:04:12.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.007+0000 7fefaee51700 1 -- 172.21.15.60:0/669266875 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa80aca20 msgr2=0x7fefa8140470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:12.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.007+0000 7fefaee51700 1 --2- 172.21.15.60:0/669266875 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa80aca20 0x7fefa8140470 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0x7fefa80ad5b0 tx=0x7fef98009040).stop 2022-06-29T16:04:12.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.007+0000 7fefaee51700 1 -- 172.21.15.60:0/669266875 shutdown_connections 2022-06-29T16:04:12.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.007+0000 7fefaee51700 1 --2- 172.21.15.60:0/669266875 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fef88053b00 0x7fef88055fc0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.007+0000 7fefaee51700 1 --2- 172.21.15.60:0/669266875 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fefa8140960 0x7fefa8145730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.007+0000 7fefaee51700 1 --2- 172.21.15.60:0/669266875 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa80aca20 0x7fefa8140470 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.007+0000 7fefaee51700 1 --2- 172.21.15.60:0/669266875 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef94004e70 0x7fefa8152740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.011+0000 7fefaee51700 1 -- 172.21.15.60:0/669266875 >> 172.21.15.60:0/669266875 conn(0x7fefa8062e70 msgr2=0x7fefa80b1d10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:12.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.011+0000 7fefaee51700 1 -- 172.21.15.60:0/669266875 shutdown_connections 2022-06-29T16:04:12.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.011+0000 7fefaee51700 1 -- 172.21.15.60:0/669266875 wait complete. 2022-06-29T16:04:12.027 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:04:12.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.127+0000 7f7c6d779700 1 Processor -- start 2022-06-29T16:04:12.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.127+0000 7f7c6d779700 1 -- start start 2022-06-29T16:04:12.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.127+0000 7f7c6d779700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7c680b61d0 0x7f7c680b0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:12.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.127+0000 7f7c6d779700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c6813f790 0x7f7c6813bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:12.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.127+0000 7f7c6d779700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7c6813fbd0 0x7f7c6813c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:12.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.127+0000 7f7c6d779700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c68142820 con 0x7f7c6813fbd0 2022-06-29T16:04:12.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.127+0000 7f7c6d779700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c68062bb0 con 0x7f7c680b61d0 2022-06-29T16:04:12.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.127+0000 7f7c6d779700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c680b65b0 con 0x7f7c6813f790 2022-06-29T16:04:12.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.127+0000 7f7c677fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7c6813fbd0 0x7f7c6813c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:12.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.127+0000 7f7c667fc700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c6813f790 0x7f7c6813bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:12.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.127+0000 7f7c66ffd700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7c680b61d0 0x7f7c680b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:12.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.127+0000 7f7c677fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7c6813fbd0 0x7f7c6813c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49402/0 (socket says 172.21.15.60:49402) 2022-06-29T16:04:12.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.127+0000 7f7c677fe700 1 -- 172.21.15.60:0/4287475112 learned_addr learned my addr 172.21.15.60:0/4287475112 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:12.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.127+0000 7f7c677fe700 1 -- 172.21.15.60:0/4287475112 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c6813f790 msgr2=0x7f7c6813bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:12.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.127+0000 7f7c677fe700 1 --2- 172.21.15.60:0/4287475112 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c6813f790 0x7f7c6813bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.127+0000 7f7c677fe700 1 -- 172.21.15.60:0/4287475112 >> v2:172.21.15.92:3300/0 conn(0x7f7c680b61d0 msgr2=0x7f7c680b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:12.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.127+0000 7f7c677fe700 1 --2- 172.21.15.60:0/4287475112 >> v2:172.21.15.92:3300/0 conn(0x7f7c680b61d0 0x7f7c680b0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.127+0000 7f7c677fe700 1 -- 172.21.15.60:0/4287475112 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c6813f050 con 0x7f7c6813fbd0 2022-06-29T16:04:12.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.127+0000 7f7c677fe700 1 --2- 172.21.15.60:0/4287475112 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c6813fbd0 0x7f7c6813c130 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0x7f7c600032f0 tx=0x7f7c60009290).ready entity=mon.0 client_cookie=e1f270e75fde91ac server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:12.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.131+0000 7f7c65ffb700 1 -- 172.21.15.60:0/4287475112 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c6000b680 con 0x7f7c6813fbd0 2022-06-29T16:04:12.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.131+0000 7f7c65ffb700 1 -- 172.21.15.60:0/4287475112 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c6000b7e0 con 0x7f7c6813fbd0 2022-06-29T16:04:12.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.131+0000 7f7c65ffb700 1 -- 172.21.15.60:0/4287475112 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c6000b950 con 0x7f7c6813fbd0 2022-06-29T16:04:12.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.131+0000 7f7c6d779700 1 -- 172.21.15.60:0/4287475112 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c6813fbd0 msgr2=0x7f7c6813c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:12.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.131+0000 7f7c6d779700 1 --2- 172.21.15.60:0/4287475112 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c6813fbd0 0x7f7c6813c130 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0x7f7c600032f0 tx=0x7f7c60009290).stop 2022-06-29T16:04:12.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.131+0000 7f7c6d779700 1 -- 172.21.15.60:0/4287475112 shutdown_connections 2022-06-29T16:04:12.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.131+0000 7f7c6d779700 1 --2- 172.21.15.60:0/4287475112 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c6813fbd0 0x7f7c6813c130 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.131+0000 7f7c6d779700 1 --2- 172.21.15.60:0/4287475112 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c6813f790 0x7f7c6813bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.131+0000 7f7c6d779700 1 --2- 172.21.15.60:0/4287475112 >> v2:172.21.15.92:3300/0 conn(0x7f7c680b61d0 0x7f7c680b0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.131+0000 7f7c6d779700 1 -- 172.21.15.60:0/4287475112 >> 172.21.15.60:0/4287475112 conn(0x7f7c68063b50 msgr2=0x7f7c680b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:12.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.131+0000 7f7c6d779700 1 -- 172.21.15.60:0/4287475112 shutdown_connections 2022-06-29T16:04:12.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.131+0000 7f7c6d779700 1 -- 172.21.15.60:0/4287475112 wait complete. 2022-06-29T16:04:12.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.131+0000 7f7c6d779700 1 Processor -- start 2022-06-29T16:04:12.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.131+0000 7f7c6d779700 1 -- start start 2022-06-29T16:04:12.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.135+0000 7f7c6d779700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c680b61d0 0x7f7c681129b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:12.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.135+0000 7f7c6d779700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c6813f790 0x7f7c6813cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:12.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.135+0000 7f7c6d779700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c6813fbd0 0x7f7c6813d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:12.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.135+0000 7f7c6d779700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c68113160 con 0x7f7c6813fbd0 2022-06-29T16:04:12.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.135+0000 7f7c6d779700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c68112050 con 0x7f7c680b61d0 2022-06-29T16:04:12.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.135+0000 7f7c6d779700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6813d700 con 0x7f7c6813f790 2022-06-29T16:04:12.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.135+0000 7f7c677fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c6813fbd0 0x7f7c6813d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:12.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.135+0000 7f7c677fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c6813fbd0 0x7f7c6813d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49406/0 (socket says 172.21.15.60:49406) 2022-06-29T16:04:12.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.135+0000 7f7c677fe700 1 -- 172.21.15.60:0/3050472742 learned_addr learned my addr 172.21.15.60:0/3050472742 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:12.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.135+0000 7f7c667fc700 1 --2- 172.21.15.60:0/3050472742 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c6813f790 0x7f7c6813cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:12.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.135+0000 7f7c66ffd700 1 --2- 172.21.15.60:0/3050472742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c680b61d0 0x7f7c681129b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:12.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.135+0000 7f7c677fe700 1 -- 172.21.15.60:0/3050472742 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c6813f790 msgr2=0x7f7c6813cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:12.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.135+0000 7f7c677fe700 1 --2- 172.21.15.60:0/3050472742 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c6813f790 0x7f7c6813cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.135+0000 7f7c677fe700 1 -- 172.21.15.60:0/3050472742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c680b61d0 msgr2=0x7f7c681129b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:12.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.135+0000 7f7c677fe700 1 --2- 172.21.15.60:0/3050472742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c680b61d0 0x7f7c681129b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.135+0000 7f7c677fe700 1 -- 172.21.15.60:0/3050472742 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c60009530 con 0x7f7c6813fbd0 2022-06-29T16:04:12.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.135+0000 7f7c667fc700 1 --2- 172.21.15.60:0/3050472742 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c6813f790 0x7f7c6813cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:04:12.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.135+0000 7f7c677fe700 1 --2- 172.21.15.60:0/3050472742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c6813fbd0 0x7f7c6813d210 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7f7c600032f0 tx=0x7f7c60017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:12.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.135+0000 7f7c57fff700 1 -- 172.21.15.60:0/3050472742 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c6001f070 con 0x7f7c6813fbd0 2022-06-29T16:04:12.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.135+0000 7f7c57fff700 1 -- 172.21.15.60:0/3050472742 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c60011ce0 con 0x7f7c6813fbd0 2022-06-29T16:04:12.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.135+0000 7f7c6d779700 1 -- 172.21.15.60:0/3050472742 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c68110eb0 con 0x7f7c6813fbd0 2022-06-29T16:04:12.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.135+0000 7f7c57fff700 1 -- 172.21.15.60:0/3050472742 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c6000b680 con 0x7f7c6813fbd0 2022-06-29T16:04:12.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.135+0000 7f7c6d779700 1 -- 172.21.15.60:0/3050472742 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c68110ab0 con 0x7f7c6813fbd0 2022-06-29T16:04:12.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.135+0000 7f7c6d779700 1 -- 172.21.15.60:0/3050472742 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c6813f230 con 0x7f7c6813fbd0 2022-06-29T16:04:12.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.139+0000 7f7c57fff700 1 -- 172.21.15.60:0/3050472742 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7c6000f070 con 0x7f7c6813fbd0 2022-06-29T16:04:12.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.139+0000 7f7c57fff700 1 --2- 172.21.15.60:0/3050472742 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7c440539e0 0x7f7c44055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:12.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.139+0000 7f7c57fff700 1 -- 172.21.15.60:0/3050472742 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7c6006d350 con 0x7f7c6813fbd0 2022-06-29T16:04:12.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.139+0000 7f7c66ffd700 1 --2- 172.21.15.60:0/3050472742 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7c440539e0 0x7f7c44055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:12.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.139+0000 7f7c66ffd700 1 --2- 172.21.15.60:0/3050472742 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7c440539e0 0x7f7c44055ea0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f7c5800b000 tx=0x7f7c58008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:12.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.143+0000 7f7c57fff700 1 -- 172.21.15.60:0/3050472742 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7c6813f230 con 0x7f7c6813fbd0 2022-06-29T16:04:12.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.271+0000 7f7c6d779700 1 -- 172.21.15.60:0/3050472742 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f7c681121c0 con 0x7f7c6813fbd0 2022-06-29T16:04:12.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.271+0000 7f7c57fff700 1 -- 172.21.15.60:0/3050472742 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v71) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7c681121c0 con 0x7f7c6813fbd0 2022-06-29T16:04:12.278 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:04:12.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.271+0000 7f7c6d779700 1 -- 172.21.15.60:0/3050472742 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7c440539e0 msgr2=0x7f7c44055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:12.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.271+0000 7f7c6d779700 1 --2- 172.21.15.60:0/3050472742 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7c440539e0 0x7f7c44055ea0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f7c5800b000 tx=0x7f7c58008040).stop 2022-06-29T16:04:12.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.271+0000 7f7c6d779700 1 -- 172.21.15.60:0/3050472742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c6813fbd0 msgr2=0x7f7c6813d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:12.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.271+0000 7f7c6d779700 1 --2- 172.21.15.60:0/3050472742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c6813fbd0 0x7f7c6813d210 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7f7c600032f0 tx=0x7f7c60017000).stop 2022-06-29T16:04:12.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.275+0000 7f7c6d779700 1 -- 172.21.15.60:0/3050472742 shutdown_connections 2022-06-29T16:04:12.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.275+0000 7f7c6d779700 1 --2- 172.21.15.60:0/3050472742 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7c440539e0 0x7f7c44055ea0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.275+0000 7f7c6d779700 1 --2- 172.21.15.60:0/3050472742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c6813fbd0 0x7f7c6813d210 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.275+0000 7f7c6d779700 1 --2- 172.21.15.60:0/3050472742 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7c6813f790 0x7f7c6813cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.275+0000 7f7c6d779700 1 --2- 172.21.15.60:0/3050472742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c680b61d0 0x7f7c681129b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.275+0000 7f7c6d779700 1 -- 172.21.15.60:0/3050472742 >> 172.21.15.60:0/3050472742 conn(0x7f7c68063b50 msgr2=0x7f7c680b4370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:12.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.275+0000 7f7c6d779700 1 -- 172.21.15.60:0/3050472742 shutdown_connections 2022-06-29T16:04:12.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.275+0000 7f7c6d779700 1 -- 172.21.15.60:0/3050472742 wait complete. 2022-06-29T16:04:12.291 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-29T16:04:12.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.399+0000 7fa99229a700 1 Processor -- start 2022-06-29T16:04:12.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.399+0000 7fa99229a700 1 -- start start 2022-06-29T16:04:12.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.403+0000 7fa99229a700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa98c066b80 0x7fa98c05f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:12.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.403+0000 7fa99229a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa98c0b2b50 0x7fa98c140e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:12.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.403+0000 7fa99229a700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa98c1407e0 con 0x7fa98c0b3060 2022-06-29T16:04:12.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.403+0000 7fa99229a700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa98c1441b0 con 0x7fa98c066b80 2022-06-29T16:04:12.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.403+0000 7fa99229a700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c140a20 con 0x7fa98c0b2b50 2022-06-29T16:04:12.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.403+0000 7fa990a97700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa98c0b3060 0x7fa98c13a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57498/0 (socket says 172.21.15.60:57498) 2022-06-29T16:04:12.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.403+0000 7fa990a97700 1 -- 172.21.15.60:0/70575692 learned_addr learned my addr 172.21.15.60:0/70575692 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:04:12.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.403+0000 7fa991298700 1 --2- 172.21.15.60:0/70575692 >> v2:172.21.15.60:3300/0 conn(0x7fa98c066b80 0x7fa98c05f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:12.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.403+0000 7fa977fff700 1 -- 172.21.15.60:0/70575692 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3356319953 0 0) 0x7fa980001110 con 0x7fa98c0b3060 2022-06-29T16:04:12.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.403+0000 7fa991a99700 1 --2- 172.21.15.60:0/70575692 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa98c0b2b50 0x7fa98c140e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:12.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.403+0000 7fa977fff700 1 -- 172.21.15.60:0/70575692 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa970003260 con 0x7fa98c0b3060 2022-06-29T16:04:12.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.403+0000 7fa977fff700 1 -- 172.21.15.60:0/70575692 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3135728879 0 0) 0x7fa970003260 con 0x7fa98c0b3060 2022-06-29T16:04:12.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.403+0000 7fa991298700 1 -- 172.21.15.60:0/70575692 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa98c0b2b50 msgr2=0x7fa98c140e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:12.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.403+0000 7fa991298700 1 --2- 172.21.15.60:0/70575692 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa98c0b2b50 0x7fa98c140e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.403+0000 7fa991298700 1 -- 172.21.15.60:0/70575692 >> v1:172.21.15.60:6789/0 conn(0x7fa98c0b3060 legacy=0x7fa98c13a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:12.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.403+0000 7fa991298700 1 -- 172.21.15.60:0/70575692 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa98c05f9e0 con 0x7fa98c066b80 2022-06-29T16:04:12.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.403+0000 7fa991a99700 1 --2- 172.21.15.60:0/70575692 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa98c0b2b50 0x7fa98c140e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:04:12.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.403+0000 7fa991298700 1 --2- 172.21.15.60:0/70575692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa98c066b80 0x7fa98c05f4f0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0x7fa980001990 tx=0x7fa988006df0).ready entity=mon.0 client_cookie=fb5d449763132e44 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:12.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.403+0000 7fa977fff700 1 -- 172.21.15.60:0/70575692 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9800017f0 con 0x7fa98c066b80 2022-06-29T16:04:12.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.403+0000 7fa977fff700 1 -- 172.21.15.60:0/70575692 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa988000e90 con 0x7fa98c066b80 2022-06-29T16:04:12.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.403+0000 7fa977fff700 1 -- 172.21.15.60:0/70575692 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa98800d4a0 con 0x7fa98c066b80 2022-06-29T16:04:12.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.403+0000 7fa99229a700 1 -- 172.21.15.60:0/70575692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa98c066b80 msgr2=0x7fa98c05f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:12.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.403+0000 7fa99229a700 1 --2- 172.21.15.60:0/70575692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa98c066b80 0x7fa98c05f4f0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0x7fa980001990 tx=0x7fa988006df0).stop 2022-06-29T16:04:12.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.403+0000 7fa99229a700 1 -- 172.21.15.60:0/70575692 shutdown_connections 2022-06-29T16:04:12.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.403+0000 7fa99229a700 1 --2- 172.21.15.60:0/70575692 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa98c0b2b50 0x7fa98c140e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.403+0000 7fa99229a700 1 --2- 172.21.15.60:0/70575692 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa98c066b80 0x7fa98c05f4f0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.403+0000 7fa99229a700 1 -- 172.21.15.60:0/70575692 >> 172.21.15.60:0/70575692 conn(0x7fa98c061c80 msgr2=0x7fa98c0637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:12.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.403+0000 7fa99229a700 1 -- 172.21.15.60:0/70575692 shutdown_connections 2022-06-29T16:04:12.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.403+0000 7fa99229a700 1 -- 172.21.15.60:0/70575692 wait complete. 2022-06-29T16:04:12.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.407+0000 7fa99229a700 1 Processor -- start 2022-06-29T16:04:12.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.407+0000 7fa99229a700 1 -- start start 2022-06-29T16:04:12.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.407+0000 7fa99229a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa98c066b80 0x7fa98c1388b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:12.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.407+0000 7fa99229a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa98c0b2b50 0x7fa98c153890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:12.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.407+0000 7fa99229a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa98c0b3060 0x7fa98c153d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:12.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.407+0000 7fa99229a700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c153330 con 0x7fa98c0b3060 2022-06-29T16:04:12.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.407+0000 7fa99229a700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c13fd20 con 0x7fa98c0b2b50 2022-06-29T16:04:12.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.407+0000 7fa99229a700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c13ee40 con 0x7fa98c066b80 2022-06-29T16:04:12.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.407+0000 7fa991a99700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa98c0b3060 0x7fa98c153d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:12.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.407+0000 7fa991298700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa98c066b80 0x7fa98c1388b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:12.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.407+0000 7fa990a97700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa98c0b2b50 0x7fa98c153890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:12.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.407+0000 7fa991a99700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa98c0b3060 0x7fa98c153d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49422/0 (socket says 172.21.15.60:49422) 2022-06-29T16:04:12.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.407+0000 7fa991a99700 1 -- 172.21.15.60:0/3565653931 learned_addr learned my addr 172.21.15.60:0/3565653931 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:12.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.411+0000 7fa991a99700 1 -- 172.21.15.60:0/3565653931 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa98c066b80 msgr2=0x7fa98c1388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:12.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.411+0000 7fa991a99700 1 --2- 172.21.15.60:0/3565653931 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa98c066b80 0x7fa98c1388b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.411+0000 7fa991a99700 1 -- 172.21.15.60:0/3565653931 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa98c0b2b50 msgr2=0x7fa98c153890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:12.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.411+0000 7fa991a99700 1 --2- 172.21.15.60:0/3565653931 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa98c0b2b50 0x7fa98c153890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.411+0000 7fa991a99700 1 -- 172.21.15.60:0/3565653931 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa98800b180 con 0x7fa98c0b3060 2022-06-29T16:04:12.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.411+0000 7fa991298700 1 --2- 172.21.15.60:0/3565653931 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa98c066b80 0x7fa98c1388b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:04:12.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.411+0000 7fa990a97700 1 --2- 172.21.15.60:0/3565653931 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa98c0b2b50 0x7fa98c153890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:04:12.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.411+0000 7fa991a99700 1 --2- 172.21.15.60:0/3565653931 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa98c0b3060 0x7fa98c153d80 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0x7fa98c13efd0 tx=0x7fa984008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:12.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.411+0000 7fa97effd700 1 -- 172.21.15.60:0/3565653931 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa984013620 con 0x7fa98c0b3060 2022-06-29T16:04:12.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.411+0000 7fa97effd700 1 -- 172.21.15.60:0/3565653931 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa984013780 con 0x7fa98c0b3060 2022-06-29T16:04:12.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.411+0000 7fa99229a700 1 -- 172.21.15.60:0/3565653931 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa98c13eba0 con 0x7fa98c0b3060 2022-06-29T16:04:12.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.411+0000 7fa97effd700 1 -- 172.21.15.60:0/3565653931 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa984013ae0 con 0x7fa98c0b3060 2022-06-29T16:04:12.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.411+0000 7fa99229a700 1 -- 172.21.15.60:0/3565653931 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa98c139150 con 0x7fa98c0b3060 2022-06-29T16:04:12.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.411+0000 7fa99229a700 1 -- 172.21.15.60:0/3565653931 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa98c0573d0 con 0x7fa98c0b3060 2022-06-29T16:04:12.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.411+0000 7fa97effd700 1 -- 172.21.15.60:0/3565653931 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa984013c40 con 0x7fa98c0b3060 2022-06-29T16:04:12.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.415+0000 7fa97effd700 1 --2- 172.21.15.60:0/3565653931 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa968053a10 0x7fa968055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:12.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.415+0000 7fa97effd700 1 -- 172.21.15.60:0/3565653931 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa98406c070 con 0x7fa98c0b3060 2022-06-29T16:04:12.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.415+0000 7fa991298700 1 --2- 172.21.15.60:0/3565653931 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa968053a10 0x7fa968055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:12.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.415+0000 7fa991298700 1 --2- 172.21.15.60:0/3565653931 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa968053a10 0x7fa968055ed0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7fa98801a000 tx=0x7fa988014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:12.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.419+0000 7fa97effd700 1 -- 172.21.15.60:0/3565653931 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa984042f30 con 0x7fa98c0b3060 2022-06-29T16:04:12.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.547+0000 7fa99229a700 1 -- 172.21.15.60:0/3565653931 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa98c13f050 con 0x7fa98c0b3060 2022-06-29T16:04:12.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.547+0000 7fa97effd700 1 -- 172.21.15.60:0/3565653931 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v71) v1 ==== 104+0+10 (secure 0 0 0) 0x7fa984046550 con 0x7fa98c0b3060 2022-06-29T16:04:12.554 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:04:12.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.547+0000 7fa99229a700 1 -- 172.21.15.60:0/3565653931 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa968053a10 msgr2=0x7fa968055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:12.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.547+0000 7fa99229a700 1 --2- 172.21.15.60:0/3565653931 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa968053a10 0x7fa968055ed0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7fa98801a000 tx=0x7fa988014000).stop 2022-06-29T16:04:12.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.547+0000 7fa99229a700 1 -- 172.21.15.60:0/3565653931 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa98c0b3060 msgr2=0x7fa98c153d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:12.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.547+0000 7fa99229a700 1 --2- 172.21.15.60:0/3565653931 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa98c0b3060 0x7fa98c153d80 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0x7fa98c13efd0 tx=0x7fa984008040).stop 2022-06-29T16:04:12.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.551+0000 7fa99229a700 1 -- 172.21.15.60:0/3565653931 shutdown_connections 2022-06-29T16:04:12.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.551+0000 7fa99229a700 1 --2- 172.21.15.60:0/3565653931 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa968053a10 0x7fa968055ed0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.551+0000 7fa99229a700 1 --2- 172.21.15.60:0/3565653931 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa98c0b3060 0x7fa98c153d80 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.551+0000 7fa99229a700 1 --2- 172.21.15.60:0/3565653931 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa98c0b2b50 0x7fa98c153890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.551+0000 7fa99229a700 1 --2- 172.21.15.60:0/3565653931 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa98c066b80 0x7fa98c1388b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.551+0000 7fa99229a700 1 -- 172.21.15.60:0/3565653931 >> 172.21.15.60:0/3565653931 conn(0x7fa98c061c80 msgr2=0x7fa98c05fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:12.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.551+0000 7fa99229a700 1 -- 172.21.15.60:0/3565653931 shutdown_connections 2022-06-29T16:04:12.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.551+0000 7fa99229a700 1 -- 172.21.15.60:0/3565653931 wait complete. 2022-06-29T16:04:12.568 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-29T16:04:12.678 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.671+0000 7f7d1f59e700 1 Processor -- start 2022-06-29T16:04:12.678 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.671+0000 7f7d1f59e700 1 -- start start 2022-06-29T16:04:12.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.671+0000 7f7d1f59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7d2005d070 0x7f7d2014aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:12.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.671+0000 7f7d1f59e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7d200b2130 0x7f7d200aca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:12.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.671+0000 7f7d1f59e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7d2014af80 0x7f7d200acf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:12.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.671+0000 7f7d1f59e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d200adf90 con 0x7f7d2014af80 2022-06-29T16:04:12.682 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.671+0000 7f7d1f59e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d2005d450 con 0x7f7d200b2130 2022-06-29T16:04:12.682 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.671+0000 7f7d1f59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d200b25d0 con 0x7f7d2005d070 2022-06-29T16:04:12.682 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.671+0000 7f7d1ed9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7d2014af80 0x7f7d200acf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:12.682 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.671+0000 7f7d1ed9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7d2014af80 0x7f7d200acf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49430/0 (socket says 172.21.15.60:49430) 2022-06-29T16:04:12.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.671+0000 7f7d1ed9d700 1 -- 172.21.15.60:0/2441046850 learned_addr learned my addr 172.21.15.60:0/2441046850 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:12.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.671+0000 7f7d1dd9b700 1 --2- 172.21.15.60:0/2441046850 >> v2:172.21.15.92:3300/0 conn(0x7f7d200b2130 0x7f7d200aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:12.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.671+0000 7f7d1e59c700 1 --2- 172.21.15.60:0/2441046850 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7d2005d070 0x7f7d2014aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:12.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.671+0000 7f7d1ed9d700 1 -- 172.21.15.60:0/2441046850 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7d2005d070 msgr2=0x7f7d2014aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:12.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.671+0000 7f7d1ed9d700 1 --2- 172.21.15.60:0/2441046850 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7d2005d070 0x7f7d2014aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.671+0000 7f7d1ed9d700 1 -- 172.21.15.60:0/2441046850 >> v2:172.21.15.92:3300/0 conn(0x7f7d200b2130 msgr2=0x7f7d200aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:12.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.671+0000 7f7d1ed9d700 1 --2- 172.21.15.60:0/2441046850 >> v2:172.21.15.92:3300/0 conn(0x7f7d200b2130 0x7f7d200aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.675+0000 7f7d1ed9d700 1 -- 172.21.15.60:0/2441046850 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d2014a350 con 0x7f7d2014af80 2022-06-29T16:04:12.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.675+0000 7f7d1dd9b700 1 --2- 172.21.15.60:0/2441046850 >> v2:172.21.15.92:3300/0 conn(0x7f7d200b2130 0x7f7d200aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:04:12.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.675+0000 7f7d1ed9d700 1 --2- 172.21.15.60:0/2441046850 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d2014af80 0x7f7d200acf10 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0x7f7d18002f70 tx=0x7f7d18006eb0).ready entity=mon.0 client_cookie=7e79b8b758b71585 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:12.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.675+0000 7f7d1d59a700 1 -- 172.21.15.60:0/2441046850 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d1800d570 con 0x7f7d2014af80 2022-06-29T16:04:12.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.675+0000 7f7d1d59a700 1 -- 172.21.15.60:0/2441046850 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d18000e90 con 0x7f7d2014af80 2022-06-29T16:04:12.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.675+0000 7f7d1d59a700 1 -- 172.21.15.60:0/2441046850 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d1800d900 con 0x7f7d2014af80 2022-06-29T16:04:12.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.675+0000 7f7d1f59e700 1 -- 172.21.15.60:0/2441046850 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d2014af80 msgr2=0x7f7d200acf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:12.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.675+0000 7f7d1f59e700 1 --2- 172.21.15.60:0/2441046850 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d2014af80 0x7f7d200acf10 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0x7f7d18002f70 tx=0x7f7d18006eb0).stop 2022-06-29T16:04:12.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.675+0000 7f7d1f59e700 1 -- 172.21.15.60:0/2441046850 shutdown_connections 2022-06-29T16:04:12.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.675+0000 7f7d1f59e700 1 --2- 172.21.15.60:0/2441046850 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d2014af80 0x7f7d200acf10 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.675+0000 7f7d1f59e700 1 --2- 172.21.15.60:0/2441046850 >> v2:172.21.15.92:3300/0 conn(0x7f7d200b2130 0x7f7d200aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.675+0000 7f7d1f59e700 1 --2- 172.21.15.60:0/2441046850 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7d2005d070 0x7f7d2014aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.675+0000 7f7d1f59e700 1 -- 172.21.15.60:0/2441046850 >> 172.21.15.60:0/2441046850 conn(0x7f7d200b5890 msgr2=0x7f7d200b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:12.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.675+0000 7f7d1f59e700 1 -- 172.21.15.60:0/2441046850 shutdown_connections 2022-06-29T16:04:12.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.675+0000 7f7d1f59e700 1 -- 172.21.15.60:0/2441046850 wait complete. 2022-06-29T16:04:12.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.679+0000 7f7d1f59e700 1 Processor -- start 2022-06-29T16:04:12.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.679+0000 7f7d1f59e700 1 -- start start 2022-06-29T16:04:12.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.679+0000 7f7d1f59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7d2005d070 0x7f7d20149720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:12.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.679+0000 7f7d1f59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d200b2130 0x7f7d20063dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:12.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.679+0000 7f7d1f59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d2014af80 0x7f7d2014bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:12.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.679+0000 7f7d1f59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d200652a0 con 0x7f7d2014af80 2022-06-29T16:04:12.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.679+0000 7f7d1f59e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2014d200 con 0x7f7d200b2130 2022-06-29T16:04:12.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.679+0000 7f7d1f59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2014b3c0 con 0x7f7d2005d070 2022-06-29T16:04:12.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.679+0000 7f7d1ed9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d2014af80 0x7f7d2014bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:12.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.679+0000 7f7d1ed9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d2014af80 0x7f7d2014bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49434/0 (socket says 172.21.15.60:49434) 2022-06-29T16:04:12.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.679+0000 7f7d1ed9d700 1 -- 172.21.15.60:0/544045897 learned_addr learned my addr 172.21.15.60:0/544045897 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:12.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.679+0000 7f7d1e59c700 1 --2- 172.21.15.60:0/544045897 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7d2005d070 0x7f7d20149720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:12.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.679+0000 7f7d1dd9b700 1 --2- 172.21.15.60:0/544045897 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d200b2130 0x7f7d20063dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:12.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.679+0000 7f7d1ed9d700 1 -- 172.21.15.60:0/544045897 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7d2005d070 msgr2=0x7f7d20149720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:12.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.679+0000 7f7d1ed9d700 1 --2- 172.21.15.60:0/544045897 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7d2005d070 0x7f7d20149720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.679+0000 7f7d1ed9d700 1 -- 172.21.15.60:0/544045897 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d200b2130 msgr2=0x7f7d20063dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:12.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.679+0000 7f7d1ed9d700 1 --2- 172.21.15.60:0/544045897 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d200b2130 0x7f7d20063dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.696 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.679+0000 7f7d1ed9d700 1 -- 172.21.15.60:0/544045897 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d1800b1e0 con 0x7f7d2014af80 2022-06-29T16:04:12.696 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.679+0000 7f7d1ed9d700 1 --2- 172.21.15.60:0/544045897 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d2014af80 0x7f7d2014bb00 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0x7f7d18002f70 tx=0x7f7d18015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:12.696 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.679+0000 7f7d177fe700 1 -- 172.21.15.60:0/544045897 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d18011610 con 0x7f7d2014af80 2022-06-29T16:04:12.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.679+0000 7f7d177fe700 1 -- 172.21.15.60:0/544045897 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d18011770 con 0x7f7d2014af80 2022-06-29T16:04:12.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.679+0000 7f7d1f59e700 1 -- 172.21.15.60:0/544045897 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d2014b660 con 0x7f7d2014af80 2022-06-29T16:04:12.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.679+0000 7f7d177fe700 1 -- 172.21.15.60:0/544045897 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d18011ad0 con 0x7f7d2014af80 2022-06-29T16:04:12.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.679+0000 7f7d1f59e700 1 -- 172.21.15.60:0/544045897 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d20066d80 con 0x7f7d2014af80 2022-06-29T16:04:12.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.683+0000 7f7d1f59e700 1 -- 172.21.15.60:0/544045897 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d2014a5f0 con 0x7f7d2014af80 2022-06-29T16:04:12.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.683+0000 7f7d177fe700 1 -- 172.21.15.60:0/544045897 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7d18025030 con 0x7f7d2014af80 2022-06-29T16:04:12.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.683+0000 7f7d177fe700 1 --2- 172.21.15.60:0/544045897 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7d040539e0 0x7f7d04055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:12.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.683+0000 7f7d177fe700 1 -- 172.21.15.60:0/544045897 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7d1806cf60 con 0x7f7d2014af80 2022-06-29T16:04:12.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.683+0000 7f7d1e59c700 1 --2- 172.21.15.60:0/544045897 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7d040539e0 0x7f7d04055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:12.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.683+0000 7f7d1e59c700 1 --2- 172.21.15.60:0/544045897 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7d040539e0 0x7f7d04055ea0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f7d1000c000 tx=0x7f7d10008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:12.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.691+0000 7f7d177fe700 1 -- 172.21.15.60:0/544045897 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7d18043e00 con 0x7f7d2014af80 2022-06-29T16:04:12.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.815+0000 7f7d1f59e700 1 -- 172.21.15.60:0/544045897 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7d2014b7f0 con 0x7f7d2014af80 2022-06-29T16:04:12.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.815+0000 7f7d177fe700 1 -- 172.21.15.60:0/544045897 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v71) v1 ==== 100+0+10 (secure 0 0 0) 0x7f7d18023020 con 0x7f7d2014af80 2022-06-29T16:04:12.822 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:04:12.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.819+0000 7f7d1f59e700 1 -- 172.21.15.60:0/544045897 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7d040539e0 msgr2=0x7f7d04055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:12.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.819+0000 7f7d1f59e700 1 --2- 172.21.15.60:0/544045897 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7d040539e0 0x7f7d04055ea0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f7d1000c000 tx=0x7f7d10008040).stop 2022-06-29T16:04:12.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.819+0000 7f7d1f59e700 1 -- 172.21.15.60:0/544045897 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d2014af80 msgr2=0x7f7d2014bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:12.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.819+0000 7f7d1f59e700 1 --2- 172.21.15.60:0/544045897 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d2014af80 0x7f7d2014bb00 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0x7f7d18002f70 tx=0x7f7d18015000).stop 2022-06-29T16:04:12.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.819+0000 7f7d1f59e700 1 -- 172.21.15.60:0/544045897 shutdown_connections 2022-06-29T16:04:12.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.819+0000 7f7d1f59e700 1 --2- 172.21.15.60:0/544045897 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7d040539e0 0x7f7d04055ea0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.819+0000 7f7d1f59e700 1 --2- 172.21.15.60:0/544045897 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d2014af80 0x7f7d2014bb00 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.819+0000 7f7d1f59e700 1 --2- 172.21.15.60:0/544045897 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d200b2130 0x7f7d20063dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.819+0000 7f7d1f59e700 1 --2- 172.21.15.60:0/544045897 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7d2005d070 0x7f7d20149720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.819+0000 7f7d1f59e700 1 -- 172.21.15.60:0/544045897 >> 172.21.15.60:0/544045897 conn(0x7f7d200b5890 msgr2=0x7f7d20146e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:12.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.819+0000 7f7d1f59e700 1 -- 172.21.15.60:0/544045897 shutdown_connections 2022-06-29T16:04:12.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.819+0000 7f7d1f59e700 1 -- 172.21.15.60:0/544045897 wait complete. 2022-06-29T16:04:12.836 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-06-29T16:04:12.836 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:04:12.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.935+0000 7f64be0de700 1 Processor -- start 2022-06-29T16:04:12.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.935+0000 7f64be0de700 1 -- start start 2022-06-29T16:04:12.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.939+0000 7f64be0de700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f64b812bde0 0x7f64b8142ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:12.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.939+0000 7f64be0de700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b80ae9a0 con 0x7f64b805d070 2022-06-29T16:04:12.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.939+0000 7f64be0de700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b805d450 con 0x7f64b8066b20 2022-06-29T16:04:12.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.939+0000 7f64be0de700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64b8143b60 con 0x7f64b812bde0 2022-06-29T16:04:12.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.939+0000 7f64bc8db700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f64b812bde0 0x7f64b8142ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:12.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.939+0000 7f64bd0dc700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f64b805d070 0x7f64b812b6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57528/0 (socket says 172.21.15.60:57528) 2022-06-29T16:04:12.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.939+0000 7f64bd0dc700 1 -- 172.21.15.60:0/3480855085 learned_addr learned my addr 172.21.15.60:0/3480855085 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:04:12.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.939+0000 7f64bc8db700 1 -- 172.21.15.60:0/3480855085 >> v1:172.21.15.92:6789/0 conn(0x7f64b8066b20 legacy=0x7f64b8142fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:12.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.939+0000 7f64bc8db700 1 -- 172.21.15.60:0/3480855085 >> v1:172.21.15.60:6789/0 conn(0x7f64b805d070 legacy=0x7f64b812b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:12.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.939+0000 7f64bc8db700 1 -- 172.21.15.60:0/3480855085 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b8067240 con 0x7f64b812bde0 2022-06-29T16:04:12.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.939+0000 7f64bc8db700 1 --2- 172.21.15.60:0/3480855085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64b812bde0 0x7f64b8142ae0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0x7f64b8061990 tx=0x7f64a8006df0).ready entity=mon.0 client_cookie=1179e4606d19b8f9 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:12.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.939+0000 7f64affff700 1 -- 172.21.15.60:0/3480855085 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64a800d5c0 con 0x7f64b812bde0 2022-06-29T16:04:12.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.939+0000 7f64affff700 1 -- 172.21.15.60:0/3480855085 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64a8000e90 con 0x7f64b812bde0 2022-06-29T16:04:12.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.939+0000 7f64affff700 1 -- 172.21.15.60:0/3480855085 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64a800d910 con 0x7f64b812bde0 2022-06-29T16:04:12.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.939+0000 7f64be0de700 1 -- 172.21.15.60:0/3480855085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64b812bde0 msgr2=0x7f64b8142ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:12.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.939+0000 7f64be0de700 1 --2- 172.21.15.60:0/3480855085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64b812bde0 0x7f64b8142ae0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0x7f64b8061990 tx=0x7f64a8006df0).stop 2022-06-29T16:04:12.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.939+0000 7f64be0de700 1 -- 172.21.15.60:0/3480855085 shutdown_connections 2022-06-29T16:04:12.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.939+0000 7f64be0de700 1 --2- 172.21.15.60:0/3480855085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64b812bde0 0x7f64b8142ae0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.939+0000 7f64be0de700 1 -- 172.21.15.60:0/3480855085 >> 172.21.15.60:0/3480855085 conn(0x7f64b80b5890 msgr2=0x7f64b80b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:12.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.939+0000 7f64be0de700 1 -- 172.21.15.60:0/3480855085 shutdown_connections 2022-06-29T16:04:12.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.939+0000 7f64be0de700 1 -- 172.21.15.60:0/3480855085 wait complete. 2022-06-29T16:04:12.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.939+0000 7f64be0de700 1 Processor -- start 2022-06-29T16:04:12.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.939+0000 7f64be0de700 1 -- start start 2022-06-29T16:04:12.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.939+0000 7f64be0de700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64b805d070 0x7f64b8130240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:12.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.943+0000 7f64be0de700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64b8066b20 0x7f64b8131fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:12.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.943+0000 7f64be0de700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64b812bde0 0x7f64b81324a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:12.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.943+0000 7f64be0de700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b8132990 con 0x7f64b8066b20 2022-06-29T16:04:12.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.943+0000 7f64be0de700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b812fad0 con 0x7f64b812bde0 2022-06-29T16:04:12.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.943+0000 7f64be0de700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64b8132bb0 con 0x7f64b805d070 2022-06-29T16:04:12.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.943+0000 7f64bc8db700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64b8066b20 0x7f64b8131fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:12.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.943+0000 7f64bc8db700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64b8066b20 0x7f64b8131fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49448/0 (socket says 172.21.15.60:49448) 2022-06-29T16:04:12.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.943+0000 7f64bc8db700 1 -- 172.21.15.60:0/3659282559 learned_addr learned my addr 172.21.15.60:0/3659282559 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:12.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.943+0000 7f64bd0dc700 1 --2- 172.21.15.60:0/3659282559 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64b805d070 0x7f64b8130240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:12.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.943+0000 7f64bd8dd700 1 --2- 172.21.15.60:0/3659282559 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64b812bde0 0x7f64b81324a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:12.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.943+0000 7f64bc8db700 1 -- 172.21.15.60:0/3659282559 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64b805d070 msgr2=0x7f64b8130240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:12.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.943+0000 7f64bc8db700 1 --2- 172.21.15.60:0/3659282559 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64b805d070 0x7f64b8130240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.943+0000 7f64bc8db700 1 -- 172.21.15.60:0/3659282559 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64b812bde0 msgr2=0x7f64b81324a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:12.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.943+0000 7f64bc8db700 1 --2- 172.21.15.60:0/3659282559 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64b812bde0 0x7f64b81324a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:12.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.943+0000 7f64bc8db700 1 -- 172.21.15.60:0/3659282559 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a800b180 con 0x7f64b8066b20 2022-06-29T16:04:12.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.943+0000 7f64bd0dc700 1 --2- 172.21.15.60:0/3659282559 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64b805d070 0x7f64b8130240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:04:12.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.943+0000 7f64bd8dd700 1 --2- 172.21.15.60:0/3659282559 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64b812bde0 0x7f64b81324a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:04:12.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.943+0000 7f64bc8db700 1 --2- 172.21.15.60:0/3659282559 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64b8066b20 0x7f64b8131fb0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 rx=0x7f64b8061990 tx=0x7f64a8012000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:12.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.943+0000 7f64ae7fc700 1 -- 172.21.15.60:0/3659282559 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64a8002f50 con 0x7f64b8066b20 2022-06-29T16:04:12.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.943+0000 7f64ae7fc700 1 -- 172.21.15.60:0/3659282559 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64a8013d50 con 0x7f64b8066b20 2022-06-29T16:04:12.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.943+0000 7f64be0de700 1 -- 172.21.15.60:0/3659282559 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64b8132e50 con 0x7f64b8066b20 2022-06-29T16:04:12.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.943+0000 7f64ae7fc700 1 -- 172.21.15.60:0/3659282559 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64a801f640 con 0x7f64b8066b20 2022-06-29T16:04:12.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.943+0000 7f64be0de700 1 -- 172.21.15.60:0/3659282559 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64b8130a90 con 0x7f64b8066b20 2022-06-29T16:04:12.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.943+0000 7f64be0de700 1 -- 172.21.15.60:0/3659282559 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64b80573d0 con 0x7f64b8066b20 2022-06-29T16:04:12.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.943+0000 7f64ae7fc700 1 -- 172.21.15.60:0/3659282559 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f64a8026070 con 0x7f64b8066b20 2022-06-29T16:04:12.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.943+0000 7f64ae7fc700 1 --2- 172.21.15.60:0/3659282559 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f64a0053a70 0x7f64a0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:12.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.943+0000 7f64ae7fc700 1 -- 172.21.15.60:0/3659282559 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f64a806c620 con 0x7f64b8066b20 2022-06-29T16:04:12.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.943+0000 7f64bd0dc700 1 --2- 172.21.15.60:0/3659282559 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f64a0053a70 0x7f64a0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:12.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.947+0000 7f64bd0dc700 1 --2- 172.21.15.60:0/3659282559 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f64a0053a70 0x7f64a0055f30 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f64b400b000 tx=0x7f64b4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:12.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:12.951+0000 7f64ae7fc700 1 -- 172.21.15.60:0/3659282559 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f64a8043540 con 0x7f64b8066b20 2022-06-29T16:04:13.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:13.119+0000 7f64be0de700 1 -- 172.21.15.60:0/3659282559 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64b812fce0 con 0x7f64b8066b20 2022-06-29T16:04:13.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:13.119+0000 7f64ae7fc700 1 -- 172.21.15.60:0/3659282559 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f64a8046b60 con 0x7f64b8066b20 2022-06-29T16:04:13.126 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:04:13.126 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":30,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5690,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1158969084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1158969084},{"type":"v1","addr":"172.21.15.60:6833","nonce":1158969084}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":28}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:04:08.275039+0000","modified":"2022-06-29T16:04:09.279046+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6304":{"gid":6304,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/953938586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":953938586},{"type":"v1","addr":"172.21.15.92:6835","nonce":953938586}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-06-29T16:04:13.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:13.119+0000 7f64be0de700 1 -- 172.21.15.60:0/3659282559 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f64a0053a70 msgr2=0x7f64a0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:13.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:13.119+0000 7f64be0de700 1 --2- 172.21.15.60:0/3659282559 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f64a0053a70 0x7f64a0055f30 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f64b400b000 tx=0x7f64b4009040).stop 2022-06-29T16:04:13.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:13.119+0000 7f64be0de700 1 -- 172.21.15.60:0/3659282559 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64b8066b20 msgr2=0x7f64b8131fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:13.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:13.119+0000 7f64be0de700 1 --2- 172.21.15.60:0/3659282559 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64b8066b20 0x7f64b8131fb0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 rx=0x7f64b8061990 tx=0x7f64a8012000).stop 2022-06-29T16:04:13.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:13.119+0000 7f64be0de700 1 -- 172.21.15.60:0/3659282559 shutdown_connections 2022-06-29T16:04:13.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:13.119+0000 7f64be0de700 1 --2- 172.21.15.60:0/3659282559 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f64a0053a70 0x7f64a0055f30 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:13.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:13.119+0000 7f64be0de700 1 --2- 172.21.15.60:0/3659282559 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64b812bde0 0x7f64b81324a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:13.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:13.119+0000 7f64be0de700 1 --2- 172.21.15.60:0/3659282559 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64b8066b20 0x7f64b8131fb0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:13.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:13.119+0000 7f64be0de700 1 --2- 172.21.15.60:0/3659282559 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64b805d070 0x7f64b8130240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:13.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:13.119+0000 7f64be0de700 1 -- 172.21.15.60:0/3659282559 >> 172.21.15.60:0/3659282559 conn(0x7f64b80b5890 msgr2=0x7f64b80b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:13.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:13.119+0000 7f64be0de700 1 -- 172.21.15.60:0/3659282559 shutdown_connections 2022-06-29T16:04:13.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:13.123+0000 7f64be0de700 1 -- 172.21.15.60:0/3659282559 wait complete. 2022-06-29T16:04:13.131 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 30 2022-06-29T16:04:13.340 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:04:08.275039+0000', 'modified': '2022-06-29T16:04:09.279046+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6304}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6304': {'gid': 6304, 'name': 'b', 'rank': 0, 'incarnation': 29, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.92:6835/953938586', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 953938586}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 953938586}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:04:13.341 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:04:13.342 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-06-29T16:04:13.457 INFO:teuthology.orchestra.run.smithi092.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":6304,"want_state":"up:active","state":"up:active","rank_uptime":5.1667288400000002,"mdsmap_epoch":30,"osdmap_epoch":71,"osdmap_epoch_barrier":71,"uptime":9.5478471650000003} 2022-06-29T16:04:13.458 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 6304, "mdsmap_epoch": 30, "osdmap_epoch": 71, "osdmap_epoch_barrier": 71, "rank_uptime": 5.16672884, "state": "up:active", "uptime": 9.547847165, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:04:13.458 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-06-29T16:04:13.459 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-06-29T16:04:13.459 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi121.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-06-29T16:04:13.460 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:04:13.460 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:04:13.469 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:04:13.527 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:04:13.528 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:04:13.540 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:04:13.541 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:04:13.542 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:04:13.543 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:04:13.550 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-06-29T16:04:13.550 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:04:13.553 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:04:13.569 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:04:13.570 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:04:13.625 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:2022-06-29T16:04:13.623+0000 7fbd079e8080 -1 init, newargv = 0x564fb5a98540 newargc=9 2022-06-29T16:04:13.625 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[12874]: starting ceph client 2022-06-29T16:04:13.626 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:04:13.627 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:04:13.627 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:04:13.628 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:04:13.634 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[12874]: starting fuse 2022-06-29T16:04:14.734 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:04:14.753 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:04:14.754 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:04:14.810 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:04:14.811 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:04:14.812 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:04:14.812 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:04:14.817 INFO:teuthology.orchestra.run.smithi121.stdout:52 2022-06-29T16:04:14.818 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-06-29T16:04:14.818 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:04:14.818 DEBUG:teuthology.orchestra.run.smithi121:> sudo python3 -c ' 2022-06-29T16:04:14.819 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2022-06-29T16:04:14.819 DEBUG:teuthology.orchestra.run.smithi121:> import re 2022-06-29T16:04:14.820 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-06-29T16:04:14.820 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2022-06-29T16:04:14.821 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:04:14.821 DEBUG:teuthology.orchestra.run.smithi121:> def find_socket(client_name): 2022-06-29T16:04:14.822 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-06-29T16:04:14.822 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2022-06-29T16:04:14.822 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:04:14.823 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2022-06-29T16:04:14.824 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2022-06-29T16:04:14.824 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2022-06-29T16:04:14.824 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2022-06-29T16:04:14.825 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:04:14.825 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2022-06-29T16:04:14.826 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-06-29T16:04:14.826 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2022-06-29T16:04:14.827 DEBUG:teuthology.orchestra.run.smithi121:> return f 2022-06-29T16:04:14.827 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-06-29T16:04:14.828 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:04:14.828 DEBUG:teuthology.orchestra.run.smithi121:> print(find_socket("client.0")) 2022-06-29T16:04:14.828 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-06-29T16:04:14.869 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.0.12874.asok 2022-06-29T16:04:14.874 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.12874.asok 2022-06-29T16:04:14.874 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:04:14.875 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.12874.asok status 2022-06-29T16:04:15.010 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2022-06-29T16:04:15.010 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2022-06-29T16:04:15.011 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "f25ffac72e5a9e63860ff7deb6767a11f0559a16", 2022-06-29T16:04:15.011 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 15.2.16-304-gf25ffac7 (f25ffac72e5a9e63860ff7deb6767a11f0559a16) octopus (stable)", 2022-06-29T16:04:15.011 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "0", 2022-06-29T16:04:15.011 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2022-06-29T16:04:15.011 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-06-29T16:04:15.012 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "12874", 2022-06-29T16:04:15.012 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2022-06-29T16:04:15.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:04:15.012 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2022-06-29T16:04:15.013 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2022-06-29T16:04:15.013 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 6435, 2022-06-29T16:04:15.013 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2022-06-29T16:04:15.013 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2022-06-29T16:04:15.013 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2022-06-29T16:04:15.014 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 6435 2022-06-29T16:04:15.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:04:15.014 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-06-29T16:04:15.014 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-06-29T16:04:15.015 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-06-29T16:04:15.015 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2990201791 2022-06-29T16:04:15.015 INFO:teuthology.orchestra.run.smithi121.stdout: } 2022-06-29T16:04:15.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:04:15.015 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-06-29T16:04:15.016 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-06-29T16:04:15.016 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-06-29T16:04:15.016 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2990201791 2022-06-29T16:04:15.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:04:15.017 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.6435 172.21.15.121:0/2990201791", 2022-06-29T16:04:15.017 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "172.21.15.121:0/2990201791", 2022-06-29T16:04:15.017 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2022-06-29T16:04:15.017 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 30, 2022-06-29T16:04:15.017 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 71, 2022-06-29T16:04:15.018 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2022-06-29T16:04:15.018 INFO:teuthology.orchestra.run.smithi121.stdout: "blacklisted": false, 2022-06-29T16:04:15.018 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2022-06-29T16:04:15.018 INFO:teuthology.orchestra.run.smithi121.stdout:} 2022-06-29T16:04:15.020 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:04:15.020 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:04:15.021 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:04:15.028 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T16:04:15.029 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T16:04:15.029 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:04:15.030 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:04:15.088 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:04:15.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.195+0000 7f4d628eb700 1 Processor -- start 2022-06-29T16:04:15.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.195+0000 7f4d628eb700 1 -- start start 2022-06-29T16:04:15.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.195+0000 7f4d628eb700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4d5c0b2130 0x7f4d5c146100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:15.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.195+0000 7f4d628eb700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d5c05d450 con 0x7f4d5c05d070 2022-06-29T16:04:15.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.195+0000 7f4d628eb700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d5c0b25d0 con 0x7f4d5c0aed00 2022-06-29T16:04:15.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.195+0000 7f4d628eb700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d5c0adf90 con 0x7f4d5c0b2130 2022-06-29T16:04:15.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.195+0000 7f4d618e9700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4d5c05d070 0x7f4d5c14aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57542/0 (socket says 172.21.15.60:57542) 2022-06-29T16:04:15.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.195+0000 7f4d618e9700 1 -- 172.21.15.60:0/2085785398 learned_addr learned my addr 172.21.15.60:0/2085785398 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:04:15.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.195+0000 7f4d610e8700 1 --2- 172.21.15.60:0/2085785398 >> v2:172.21.15.92:3300/0 conn(0x7f4d5c0b2130 0x7f4d5c146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:15.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.195+0000 7f4d608e7700 1 -- 172.21.15.60:0/2085785398 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910974310 0 0) 0x7f4d5c05d450 con 0x7f4d5c05d070 2022-06-29T16:04:15.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.195+0000 7f4d608e7700 1 -- 172.21.15.60:0/2085785398 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d44003260 con 0x7f4d5c05d070 2022-06-29T16:04:15.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.195+0000 7f4d608e7700 1 -- 172.21.15.60:0/2085785398 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 702934131 0 0) 0x7f4d44003260 con 0x7f4d5c05d070 2022-06-29T16:04:15.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.195+0000 7f4d608e7700 1 -- 172.21.15.60:0/2085785398 >> v2:172.21.15.92:3300/0 conn(0x7f4d5c0b2130 msgr2=0x7f4d5c146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:15.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.195+0000 7f4d608e7700 1 --2- 172.21.15.60:0/2085785398 >> v2:172.21.15.92:3300/0 conn(0x7f4d5c0b2130 0x7f4d5c146100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:15.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.195+0000 7f4d608e7700 1 -- 172.21.15.60:0/2085785398 >> v1:172.21.15.92:6789/0 conn(0x7f4d5c0aed00 legacy=0x7f4d5c1465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:15.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.195+0000 7f4d608e7700 1 -- 172.21.15.60:0/2085785398 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d5c14a350 con 0x7f4d5c05d070 2022-06-29T16:04:15.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.195+0000 7f4d608e7700 1 -- 172.21.15.60:0/2085785398 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f4d50001660 con 0x7f4d5c05d070 2022-06-29T16:04:15.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.195+0000 7f4d608e7700 1 -- 172.21.15.60:0/2085785398 >> v1:172.21.15.60:6789/0 conn(0x7f4d5c05d070 legacy=0x7f4d5c14aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:15.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.195+0000 7f4d608e7700 1 --2- 172.21.15.60:0/2085785398 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4d44003750 0x7f4d44003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:15.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.195+0000 7f4d608e7700 1 --2- 172.21.15.60:0/2085785398 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d44004500 0x7f4d44004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:15.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.195+0000 7f4d608e7700 1 --2- 172.21.15.60:0/2085785398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d44004e70 0x7f4d4400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:15.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.195+0000 7f4d608e7700 1 -- 172.21.15.60:0/2085785398 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5c0adf90 con 0x7f4d44004e70 2022-06-29T16:04:15.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.195+0000 7f4d608e7700 1 -- 172.21.15.60:0/2085785398 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4400bdd0 con 0x7f4d44004500 2022-06-29T16:04:15.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.195+0000 7f4d608e7700 1 -- 172.21.15.60:0/2085785398 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4400bf60 con 0x7f4d44003750 2022-06-29T16:04:15.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.195+0000 7f4d620ea700 1 --2- 172.21.15.60:0/2085785398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d44004e70 0x7f4d4400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:15.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.199+0000 7f4d610e8700 1 --2- 172.21.15.60:0/2085785398 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d44004500 0x7f4d44004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:15.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.199+0000 7f4d618e9700 1 --2- 172.21.15.60:0/2085785398 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4d44003750 0x7f4d44003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:15.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.199+0000 7f4d620ea700 1 -- 172.21.15.60:0/2085785398 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4d44003750 msgr2=0x7f4d44003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:15.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.199+0000 7f4d620ea700 1 --2- 172.21.15.60:0/2085785398 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4d44003750 0x7f4d44003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:15.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.199+0000 7f4d620ea700 1 -- 172.21.15.60:0/2085785398 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d44004500 msgr2=0x7f4d44004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:15.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.199+0000 7f4d620ea700 1 --2- 172.21.15.60:0/2085785398 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d44004500 0x7f4d44004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:15.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.199+0000 7f4d620ea700 1 -- 172.21.15.60:0/2085785398 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d44003260 con 0x7f4d44004e70 2022-06-29T16:04:15.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.199+0000 7f4d620ea700 1 -- 172.21.15.60:0/2085785398 reap_dead start 2022-06-29T16:04:15.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.199+0000 7f4d620ea700 1 --2- 172.21.15.60:0/2085785398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d44004e70 0x7f4d4400b750 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 rx=0x7f4d5c0b5cd0 tx=0x7f4d54007d80).ready entity=mon.0 client_cookie=c0cce2fd13646a63 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:15.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.199+0000 7f4d608e7700 1 -- 172.21.15.60:0/2085785398 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d54001620 con 0x7f4d44004e70 2022-06-29T16:04:15.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.199+0000 7f4d608e7700 1 -- 172.21.15.60:0/2085785398 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d54006890 con 0x7f4d44004e70 2022-06-29T16:04:15.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.199+0000 7f4d608e7700 1 -- 172.21.15.60:0/2085785398 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d54006bf0 con 0x7f4d44004e70 2022-06-29T16:04:15.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.199+0000 7f4d628eb700 1 -- 172.21.15.60:0/2085785398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d44004e70 msgr2=0x7f4d4400b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:15.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.199+0000 7f4d628eb700 1 --2- 172.21.15.60:0/2085785398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d44004e70 0x7f4d4400b750 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 rx=0x7f4d5c0b5cd0 tx=0x7f4d54007d80).stop 2022-06-29T16:04:15.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.199+0000 7f4d628eb700 1 -- 172.21.15.60:0/2085785398 shutdown_connections 2022-06-29T16:04:15.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.199+0000 7f4d628eb700 1 --2- 172.21.15.60:0/2085785398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d44004e70 0x7f4d4400b750 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:15.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.199+0000 7f4d628eb700 1 -- 172.21.15.60:0/2085785398 >> 172.21.15.60:0/2085785398 conn(0x7f4d5c0b5890 msgr2=0x7f4d5c0b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:15.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.199+0000 7f4d628eb700 1 -- 172.21.15.60:0/2085785398 shutdown_connections 2022-06-29T16:04:15.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.199+0000 7f4d628eb700 1 -- 172.21.15.60:0/2085785398 wait complete. 2022-06-29T16:04:15.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.203+0000 7f4d628eb700 1 Processor -- start 2022-06-29T16:04:15.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.203+0000 7f4d628eb700 1 -- start start 2022-06-29T16:04:15.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.203+0000 7f4d628eb700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d44004e70 0x7f4d5c14acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:15.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.203+0000 7f4d628eb700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d5c14bb20 0x7f4d5c1565d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:15.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.203+0000 7f4d628eb700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4d5c156ac0 0x7f4d5c156f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:15.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.203+0000 7f4d628eb700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5c14d6d0 con 0x7f4d5c14bb20 2022-06-29T16:04:15.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.203+0000 7f4d628eb700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5c14cd00 con 0x7f4d44004e70 2022-06-29T16:04:15.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.203+0000 7f4d628eb700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5c14e4a0 con 0x7f4d5c156ac0 2022-06-29T16:04:15.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.203+0000 7f4d610e8700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d5c14bb20 0x7f4d5c1565d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:15.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.203+0000 7f4d610e8700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d5c14bb20 0x7f4d5c1565d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49468/0 (socket says 172.21.15.60:49468) 2022-06-29T16:04:15.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.203+0000 7f4d610e8700 1 -- 172.21.15.60:0/899086229 learned_addr learned my addr 172.21.15.60:0/899086229 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:15.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.203+0000 7f4d618e9700 1 --2- 172.21.15.60:0/899086229 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d44004e70 0x7f4d5c14acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:15.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.203+0000 7f4d620ea700 1 --2- 172.21.15.60:0/899086229 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4d5c156ac0 0x7f4d5c156f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:15.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.203+0000 7f4d610e8700 1 -- 172.21.15.60:0/899086229 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4d5c156ac0 msgr2=0x7f4d5c156f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:15.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.203+0000 7f4d610e8700 1 --2- 172.21.15.60:0/899086229 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4d5c156ac0 0x7f4d5c156f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:15.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.203+0000 7f4d610e8700 1 -- 172.21.15.60:0/899086229 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d44004e70 msgr2=0x7f4d5c14acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:15.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.203+0000 7f4d610e8700 1 --2- 172.21.15.60:0/899086229 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d44004e70 0x7f4d5c14acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:15.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.203+0000 7f4d610e8700 1 -- 172.21.15.60:0/899086229 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d5c14e740 con 0x7f4d5c14bb20 2022-06-29T16:04:15.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.203+0000 7f4d610e8700 1 --2- 172.21.15.60:0/899086229 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d5c14bb20 0x7f4d5c1565d0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0x7f4d5800b000 tx=0x7f4d58006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:15.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.207+0000 7f4d4effd700 1 -- 172.21.15.60:0/899086229 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d58013680 con 0x7f4d5c14bb20 2022-06-29T16:04:15.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.207+0000 7f4d4effd700 1 -- 172.21.15.60:0/899086229 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d580137e0 con 0x7f4d5c14bb20 2022-06-29T16:04:15.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.207+0000 7f4d628eb700 1 -- 172.21.15.60:0/899086229 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d5c156100 con 0x7f4d5c14bb20 2022-06-29T16:04:15.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.207+0000 7f4d4effd700 1 -- 172.21.15.60:0/899086229 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d58013b80 con 0x7f4d5c14bb20 2022-06-29T16:04:15.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.207+0000 7f4d628eb700 1 -- 172.21.15.60:0/899086229 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d5c14e160 con 0x7f4d5c14bb20 2022-06-29T16:04:15.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.207+0000 7f4d628eb700 1 -- 172.21.15.60:0/899086229 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d5c14a5f0 con 0x7f4d5c14bb20 2022-06-29T16:04:15.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.207+0000 7f4d4effd700 1 -- 172.21.15.60:0/899086229 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4d58013ce0 con 0x7f4d5c14bb20 2022-06-29T16:04:15.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.207+0000 7f4d4effd700 1 --2- 172.21.15.60:0/899086229 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4d3c053a10 0x7f4d3c055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:15.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.207+0000 7f4d4effd700 1 -- 172.21.15.60:0/899086229 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4d5806c7d0 con 0x7f4d5c14bb20 2022-06-29T16:04:15.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.207+0000 7f4d618e9700 1 --2- 172.21.15.60:0/899086229 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4d3c053a10 0x7f4d3c055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:15.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.211+0000 7f4d618e9700 1 --2- 172.21.15.60:0/899086229 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4d3c053a10 0x7f4d3c055ed0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f4d5000d000 tx=0x7f4d50007040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:15.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.215+0000 7f4d4effd700 1 -- 172.21.15.60:0/899086229 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4d580435e0 con 0x7f4d5c14bb20 2022-06-29T16:04:15.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.347+0000 7f4d628eb700 1 -- 172.21.15.60:0/899086229 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d5c0573d0 con 0x7f4d5c14bb20 2022-06-29T16:04:15.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.347+0000 7f4d4effd700 1 -- 172.21.15.60:0/899086229 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16041 (secure 0 0 0) 0x7f4d58018020 con 0x7f4d5c14bb20 2022-06-29T16:04:15.355 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:04:15.355 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":71,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:04:09.277985+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:04:06.708236+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2022-06-29T16:04:07.305396+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:04:15.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.351+0000 7f4d628eb700 1 -- 172.21.15.60:0/899086229 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4d3c053a10 msgr2=0x7f4d3c055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:15.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.351+0000 7f4d628eb700 1 --2- 172.21.15.60:0/899086229 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4d3c053a10 0x7f4d3c055ed0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f4d5000d000 tx=0x7f4d50007040).stop 2022-06-29T16:04:15.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.351+0000 7f4d628eb700 1 -- 172.21.15.60:0/899086229 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d5c14bb20 msgr2=0x7f4d5c1565d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:15.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.351+0000 7f4d628eb700 1 --2- 172.21.15.60:0/899086229 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d5c14bb20 0x7f4d5c1565d0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0x7f4d5800b000 tx=0x7f4d58006df0).stop 2022-06-29T16:04:15.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.351+0000 7f4d628eb700 1 -- 172.21.15.60:0/899086229 shutdown_connections 2022-06-29T16:04:15.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.351+0000 7f4d628eb700 1 --2- 172.21.15.60:0/899086229 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4d3c053a10 0x7f4d3c055ed0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:15.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.351+0000 7f4d628eb700 1 --2- 172.21.15.60:0/899086229 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4d5c156ac0 0x7f4d5c156f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:15.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.351+0000 7f4d628eb700 1 --2- 172.21.15.60:0/899086229 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4d5c14bb20 0x7f4d5c1565d0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:15.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.351+0000 7f4d628eb700 1 --2- 172.21.15.60:0/899086229 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d44004e70 0x7f4d5c14acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:15.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.351+0000 7f4d628eb700 1 -- 172.21.15.60:0/899086229 >> 172.21.15.60:0/899086229 conn(0x7f4d5c0b5890 msgr2=0x7f4d5c0af720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:15.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.351+0000 7f4d628eb700 1 -- 172.21.15.60:0/899086229 shutdown_connections 2022-06-29T16:04:15.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.351+0000 7f4d628eb700 1 -- 172.21.15.60:0/899086229 wait complete. 2022-06-29T16:04:15.368 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-29T16:04:06.708236+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2022-06-29T16:04:07.305396+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-29T16:04:15.369 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:04:15.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.475+0000 7f5e84801700 1 Processor -- start 2022-06-29T16:04:15.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.475+0000 7f5e84801700 1 -- start start 2022-06-29T16:04:15.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.475+0000 7f5e84801700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e80064560 0x7f5e8005c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:15.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.475+0000 7f5e84801700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5e80146240 0x7f5e8005fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:15.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.475+0000 7f5e84801700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e8005f0b0 con 0x7f5e8005c460 2022-06-29T16:04:15.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.475+0000 7f5e84801700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e800b11f0 con 0x7f5e80146240 2022-06-29T16:04:15.486 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.475+0000 7f5e84801700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e800b0f50 con 0x7f5e80064560 2022-06-29T16:04:15.486 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.475+0000 7f5e7e59c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5e8005c460 0x7f5e8013c920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57560/0 (socket says 172.21.15.60:57560) 2022-06-29T16:04:15.486 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.475+0000 7f5e7e59c700 1 -- 172.21.15.60:0/2809505164 learned_addr learned my addr 172.21.15.60:0/2809505164 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:04:15.486 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.475+0000 7f5e7ed9d700 1 --2- 172.21.15.60:0/2809505164 >> v2:172.21.15.92:3300/0 conn(0x7f5e80146240 0x7f5e8005fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:15.487 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.475+0000 7f5e77fff700 1 --2- 172.21.15.60:0/2809505164 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e80064560 0x7f5e8005c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:15.487 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.479+0000 7f5e7dd9b700 1 -- 172.21.15.60:0/2809505164 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4141140491 0 0) 0x7f5e8005f0b0 con 0x7f5e8005c460 2022-06-29T16:04:15.487 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.479+0000 7f5e7dd9b700 1 -- 172.21.15.60:0/2809505164 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e60003260 con 0x7f5e8005c460 2022-06-29T16:04:15.487 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.479+0000 7f5e77fff700 1 -- 172.21.15.60:0/2809505164 >> v2:172.21.15.92:3300/0 conn(0x7f5e80146240 msgr2=0x7f5e8005fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:15.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.479+0000 7f5e77fff700 1 --2- 172.21.15.60:0/2809505164 >> v2:172.21.15.92:3300/0 conn(0x7f5e80146240 0x7f5e8005fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:15.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.479+0000 7f5e77fff700 1 -- 172.21.15.60:0/2809505164 >> v1:172.21.15.60:6789/0 conn(0x7f5e8005c460 legacy=0x7f5e8013c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:15.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.479+0000 7f5e77fff700 1 -- 172.21.15.60:0/2809505164 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e8005f2e0 con 0x7f5e80064560 2022-06-29T16:04:15.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.479+0000 7f5e7ed9d700 1 --2- 172.21.15.60:0/2809505164 >> v2:172.21.15.92:3300/0 conn(0x7f5e80146240 0x7f5e8005fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:04:15.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.479+0000 7f5e77fff700 1 --2- 172.21.15.60:0/2809505164 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e80064560 0x7f5e8005c840 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f5e68002f70 tx=0x7f5e68006df0).ready entity=mon.2 client_cookie=9bad484bac183533 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:15.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.479+0000 7f5e7dd9b700 1 -- 172.21.15.60:0/2809505164 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e6800d4a0 con 0x7f5e80064560 2022-06-29T16:04:15.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.479+0000 7f5e7dd9b700 1 -- 172.21.15.60:0/2809505164 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e68000e90 con 0x7f5e80064560 2022-06-29T16:04:15.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.479+0000 7f5e7dd9b700 1 -- 172.21.15.60:0/2809505164 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e6800d7f0 con 0x7f5e80064560 2022-06-29T16:04:15.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.479+0000 7f5e84801700 1 -- 172.21.15.60:0/2809505164 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e80064560 msgr2=0x7f5e8005c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:15.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.479+0000 7f5e84801700 1 --2- 172.21.15.60:0/2809505164 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e80064560 0x7f5e8005c840 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f5e68002f70 tx=0x7f5e68006df0).stop 2022-06-29T16:04:15.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.479+0000 7f5e84801700 1 -- 172.21.15.60:0/2809505164 shutdown_connections 2022-06-29T16:04:15.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.479+0000 7f5e84801700 1 --2- 172.21.15.60:0/2809505164 >> v2:172.21.15.92:3300/0 conn(0x7f5e80146240 0x7f5e8005fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:15.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.479+0000 7f5e84801700 1 --2- 172.21.15.60:0/2809505164 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e80064560 0x7f5e8005c840 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:15.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.479+0000 7f5e84801700 1 -- 172.21.15.60:0/2809505164 >> 172.21.15.60:0/2809505164 conn(0x7f5e80061f30 msgr2=0x7f5e800639e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:15.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.479+0000 7f5e84801700 1 -- 172.21.15.60:0/2809505164 shutdown_connections 2022-06-29T16:04:15.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.479+0000 7f5e84801700 1 -- 172.21.15.60:0/2809505164 wait complete. 2022-06-29T16:04:15.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.479+0000 7f5e84801700 1 Processor -- start 2022-06-29T16:04:15.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.479+0000 7f5e84801700 1 -- start start 2022-06-29T16:04:15.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.483+0000 7f5e84801700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e8005c460 0x7f5e80066600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:15.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.483+0000 7f5e84801700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e80064560 0x7f5e800b4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:15.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.483+0000 7f5e84801700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e80146240 0x7f5e8013ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:15.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.483+0000 7f5e84801700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e80065d50 con 0x7f5e8005c460 2022-06-29T16:04:15.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.483+0000 7f5e84801700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e800659b0 con 0x7f5e80064560 2022-06-29T16:04:15.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.483+0000 7f5e84801700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e80065470 con 0x7f5e80146240 2022-06-29T16:04:15.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.483+0000 7f5e7e59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e8005c460 0x7f5e80066600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:15.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.483+0000 7f5e7e59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e8005c460 0x7f5e80066600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49480/0 (socket says 172.21.15.60:49480) 2022-06-29T16:04:15.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.483+0000 7f5e7e59c700 1 -- 172.21.15.60:0/780808759 learned_addr learned my addr 172.21.15.60:0/780808759 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:15.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.483+0000 7f5e77fff700 1 --2- 172.21.15.60:0/780808759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e80064560 0x7f5e800b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:15.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.483+0000 7f5e7ed9d700 1 --2- 172.21.15.60:0/780808759 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e80146240 0x7f5e8013ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:15.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.483+0000 7f5e7e59c700 1 -- 172.21.15.60:0/780808759 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e80146240 msgr2=0x7f5e8013ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:15.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.483+0000 7f5e7e59c700 1 --2- 172.21.15.60:0/780808759 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e80146240 0x7f5e8013ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:15.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.483+0000 7f5e7e59c700 1 -- 172.21.15.60:0/780808759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e80064560 msgr2=0x7f5e800b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:15.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.483+0000 7f5e7e59c700 1 --2- 172.21.15.60:0/780808759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e80064560 0x7f5e800b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:15.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.483+0000 7f5e7e59c700 1 -- 172.21.15.60:0/780808759 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e78009470 con 0x7f5e8005c460 2022-06-29T16:04:15.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.483+0000 7f5e7e59c700 1 --2- 172.21.15.60:0/780808759 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e8005c460 0x7f5e80066600 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7f5e70001110 tx=0x7f5e70007ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:15.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.483+0000 7f5e777fe700 1 -- 172.21.15.60:0/780808759 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e7000ce20 con 0x7f5e8005c460 2022-06-29T16:04:15.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.483+0000 7f5e777fe700 1 -- 172.21.15.60:0/780808759 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e700072e0 con 0x7f5e8005c460 2022-06-29T16:04:15.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.483+0000 7f5e84801700 1 -- 172.21.15.60:0/780808759 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e6800b180 con 0x7f5e8005c460 2022-06-29T16:04:15.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.483+0000 7f5e777fe700 1 -- 172.21.15.60:0/780808759 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e70005800 con 0x7f5e8005c460 2022-06-29T16:04:15.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.483+0000 7f5e84801700 1 -- 172.21.15.60:0/780808759 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e800675a0 con 0x7f5e8005c460 2022-06-29T16:04:15.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.487+0000 7f5e84801700 1 -- 172.21.15.60:0/780808759 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e800573d0 con 0x7f5e8005c460 2022-06-29T16:04:15.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.487+0000 7f5e777fe700 1 -- 172.21.15.60:0/780808759 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5e70010070 con 0x7f5e8005c460 2022-06-29T16:04:15.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.487+0000 7f5e777fe700 1 --2- 172.21.15.60:0/780808759 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5e64053b00 0x7f5e64055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:15.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.487+0000 7f5e777fe700 1 -- 172.21.15.60:0/780808759 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5e7006b6a0 con 0x7f5e8005c460 2022-06-29T16:04:15.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.487+0000 7f5e77fff700 1 --2- 172.21.15.60:0/780808759 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5e64053b00 0x7f5e64055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:15.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.487+0000 7f5e77fff700 1 --2- 172.21.15.60:0/780808759 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5e64053b00 0x7f5e64055fc0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f5e6801a000 tx=0x7f5e68014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:15.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.491+0000 7f5e777fe700 1 -- 172.21.15.60:0/780808759 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5e700424b0 con 0x7f5e8005c460 2022-06-29T16:04:15.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.623+0000 7f5e84801700 1 -- 172.21.15.60:0/780808759 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f5e80068640 con 0x7f5e8005c460 2022-06-29T16:04:15.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.623+0000 7f5e777fe700 1 -- 172.21.15.60:0/780808759 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v71) v1 ==== 110+0+10 (secure 0 0 0) 0x7f5e70045ad0 con 0x7f5e8005c460 2022-06-29T16:04:15.631 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:04:15.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.627+0000 7f5e84801700 1 -- 172.21.15.60:0/780808759 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5e64053b00 msgr2=0x7f5e64055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:15.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.627+0000 7f5e84801700 1 --2- 172.21.15.60:0/780808759 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5e64053b00 0x7f5e64055fc0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f5e6801a000 tx=0x7f5e68014000).stop 2022-06-29T16:04:15.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.627+0000 7f5e84801700 1 -- 172.21.15.60:0/780808759 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e8005c460 msgr2=0x7f5e80066600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:15.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.627+0000 7f5e84801700 1 --2- 172.21.15.60:0/780808759 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e8005c460 0x7f5e80066600 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7f5e70001110 tx=0x7f5e70007ad0).stop 2022-06-29T16:04:15.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.627+0000 7f5e84801700 1 -- 172.21.15.60:0/780808759 shutdown_connections 2022-06-29T16:04:15.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.627+0000 7f5e84801700 1 --2- 172.21.15.60:0/780808759 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5e64053b00 0x7f5e64055fc0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:15.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.627+0000 7f5e84801700 1 --2- 172.21.15.60:0/780808759 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e80146240 0x7f5e8013ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:15.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.627+0000 7f5e84801700 1 --2- 172.21.15.60:0/780808759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e80064560 0x7f5e800b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:15.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.627+0000 7f5e84801700 1 --2- 172.21.15.60:0/780808759 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e8005c460 0x7f5e80066600 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:15.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.627+0000 7f5e84801700 1 -- 172.21.15.60:0/780808759 >> 172.21.15.60:0/780808759 conn(0x7f5e80061f30 msgr2=0x7f5e800b32d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:15.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.627+0000 7f5e84801700 1 -- 172.21.15.60:0/780808759 shutdown_connections 2022-06-29T16:04:15.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.627+0000 7f5e84801700 1 -- 172.21.15.60:0/780808759 wait complete. 2022-06-29T16:04:15.645 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:04:15.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.743+0000 7fb6cb59e700 1 Processor -- start 2022-06-29T16:04:15.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.743+0000 7fb6cb59e700 1 -- start start 2022-06-29T16:04:15.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.747+0000 7fb6cb59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6cc05d070 0x7fb6cc14aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:15.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.747+0000 7fb6cb59e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb6cc14af80 0x7fb6cc0ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:15.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.747+0000 7fb6cb59e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6cc05d450 con 0x7fb6cc0b2130 2022-06-29T16:04:15.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.747+0000 7fb6cb59e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6cc0adf90 con 0x7fb6cc14af80 2022-06-29T16:04:15.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.747+0000 7fb6cb59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6cc0b25d0 con 0x7fb6cc05d070 2022-06-29T16:04:15.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.747+0000 7fb6cad9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb6cc14af80 0x7fb6cc0ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:15.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.747+0000 7fb6cad9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb6cc14af80 0x7fb6cc0ad130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49492/0 (socket says 172.21.15.60:49492) 2022-06-29T16:04:15.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.747+0000 7fb6cad9d700 1 -- 172.21.15.60:0/936504212 learned_addr learned my addr 172.21.15.60:0/936504212 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:15.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.747+0000 7fb6ca59c700 1 --2- 172.21.15.60:0/936504212 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6cc05d070 0x7fb6cc14aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:15.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.747+0000 7fb6cad9d700 1 -- 172.21.15.60:0/936504212 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6cc05d070 msgr2=0x7fb6cc14aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:15.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.747+0000 7fb6cad9d700 1 --2- 172.21.15.60:0/936504212 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6cc05d070 0x7fb6cc14aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:15.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.747+0000 7fb6cad9d700 1 -- 172.21.15.60:0/936504212 >> v1:172.21.15.92:6789/0 conn(0x7fb6cc0b2130 legacy=0x7fb6cc0aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:15.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.747+0000 7fb6cad9d700 1 -- 172.21.15.60:0/936504212 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6cc14a350 con 0x7fb6cc14af80 2022-06-29T16:04:15.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.747+0000 7fb6cad9d700 1 --2- 172.21.15.60:0/936504212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6cc14af80 0x7fb6cc0ad130 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0x7fb6bc002f70 tx=0x7fb6bc006eb0).ready entity=mon.0 client_cookie=2e2899301e208a85 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:15.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.747+0000 7fb6c959a700 1 -- 172.21.15.60:0/936504212 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6bc00d570 con 0x7fb6cc14af80 2022-06-29T16:04:15.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.747+0000 7fb6c959a700 1 -- 172.21.15.60:0/936504212 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6bc000e90 con 0x7fb6cc14af80 2022-06-29T16:04:15.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.747+0000 7fb6c959a700 1 -- 172.21.15.60:0/936504212 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6bc00d900 con 0x7fb6cc14af80 2022-06-29T16:04:15.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.747+0000 7fb6cb59e700 1 -- 172.21.15.60:0/936504212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6cc14af80 msgr2=0x7fb6cc0ad130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:15.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.747+0000 7fb6cb59e700 1 --2- 172.21.15.60:0/936504212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6cc14af80 0x7fb6cc0ad130 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0x7fb6bc002f70 tx=0x7fb6bc006eb0).stop 2022-06-29T16:04:15.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.747+0000 7fb6cb59e700 1 -- 172.21.15.60:0/936504212 shutdown_connections 2022-06-29T16:04:15.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.747+0000 7fb6cb59e700 1 --2- 172.21.15.60:0/936504212 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6cc14af80 0x7fb6cc0ad130 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:15.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.747+0000 7fb6cb59e700 1 --2- 172.21.15.60:0/936504212 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6cc05d070 0x7fb6cc14aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:15.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.747+0000 7fb6cb59e700 1 -- 172.21.15.60:0/936504212 >> 172.21.15.60:0/936504212 conn(0x7fb6cc0b5890 msgr2=0x7fb6cc0b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:15.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.747+0000 7fb6cb59e700 1 -- 172.21.15.60:0/936504212 shutdown_connections 2022-06-29T16:04:15.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.747+0000 7fb6cb59e700 1 -- 172.21.15.60:0/936504212 wait complete. 2022-06-29T16:04:15.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.751+0000 7fb6cb59e700 1 Processor -- start 2022-06-29T16:04:15.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.751+0000 7fb6cb59e700 1 -- start start 2022-06-29T16:04:15.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.751+0000 7fb6cb59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6cc05d070 0x7fb6cc0b46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:15.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.751+0000 7fb6cb59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6cc0b2130 0x7fb6cc139d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:15.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.751+0000 7fb6cb59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6cc14af80 0x7fb6cc13a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:15.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.751+0000 7fb6cb59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6cc066640 con 0x7fb6cc0b2130 2022-06-29T16:04:15.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.751+0000 7fb6cb59e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6cc0668e0 con 0x7fb6cc14af80 2022-06-29T16:04:15.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.751+0000 7fb6cb59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6cc066b80 con 0x7fb6cc05d070 2022-06-29T16:04:15.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.751+0000 7fb6c9d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6cc0b2130 0x7fb6cc139d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:15.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.751+0000 7fb6c9d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6cc0b2130 0x7fb6cc139d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49496/0 (socket says 172.21.15.60:49496) 2022-06-29T16:04:15.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.751+0000 7fb6c9d9b700 1 -- 172.21.15.60:0/3389695003 learned_addr learned my addr 172.21.15.60:0/3389695003 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:15.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.751+0000 7fb6cad9d700 1 --2- 172.21.15.60:0/3389695003 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6cc14af80 0x7fb6cc13a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:15.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.751+0000 7fb6ca59c700 1 --2- 172.21.15.60:0/3389695003 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6cc05d070 0x7fb6cc0b46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:15.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.751+0000 7fb6c9d9b700 1 -- 172.21.15.60:0/3389695003 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6cc05d070 msgr2=0x7fb6cc0b46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:15.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.751+0000 7fb6c9d9b700 1 --2- 172.21.15.60:0/3389695003 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6cc05d070 0x7fb6cc0b46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:15.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.751+0000 7fb6c9d9b700 1 -- 172.21.15.60:0/3389695003 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6cc14af80 msgr2=0x7fb6cc13a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:15.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.751+0000 7fb6c9d9b700 1 --2- 172.21.15.60:0/3389695003 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6cc14af80 0x7fb6cc13a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:15.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.751+0000 7fb6c9d9b700 1 -- 172.21.15.60:0/3389695003 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6bc00b1e0 con 0x7fb6cc0b2130 2022-06-29T16:04:15.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.751+0000 7fb6cad9d700 1 --2- 172.21.15.60:0/3389695003 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6cc14af80 0x7fb6cc13a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:04:15.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.751+0000 7fb6ca59c700 1 --2- 172.21.15.60:0/3389695003 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6cc05d070 0x7fb6cc0b46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:04:15.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.751+0000 7fb6c9d9b700 1 --2- 172.21.15.60:0/3389695003 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6cc0b2130 0x7fb6cc139d10 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0x7fb6cc14a970 tx=0x7fb6c0007be0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:15.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.751+0000 7fb6bb7fe700 1 -- 172.21.15.60:0/3389695003 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6c000ce00 con 0x7fb6cc0b2130 2022-06-29T16:04:15.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.751+0000 7fb6bb7fe700 1 -- 172.21.15.60:0/3389695003 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6c0007240 con 0x7fb6cc0b2130 2022-06-29T16:04:15.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.751+0000 7fb6cb59e700 1 -- 172.21.15.60:0/3389695003 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6cc0653e0 con 0x7fb6cc0b2130 2022-06-29T16:04:15.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.751+0000 7fb6bb7fe700 1 -- 172.21.15.60:0/3389695003 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6c0006700 con 0x7fb6cc0b2130 2022-06-29T16:04:15.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.751+0000 7fb6cb59e700 1 -- 172.21.15.60:0/3389695003 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6cc0661a0 con 0x7fb6cc0b2130 2022-06-29T16:04:15.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.755+0000 7fb6cb59e700 1 -- 172.21.15.60:0/3389695003 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6cc065a40 con 0x7fb6cc0b2130 2022-06-29T16:04:15.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.755+0000 7fb6bb7fe700 1 -- 172.21.15.60:0/3389695003 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb6c0006860 con 0x7fb6cc0b2130 2022-06-29T16:04:15.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.755+0000 7fb6bb7fe700 1 --2- 172.21.15.60:0/3389695003 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb6a8053a70 0x7fb6a8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:15.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.755+0000 7fb6bb7fe700 1 -- 172.21.15.60:0/3389695003 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb6c006b960 con 0x7fb6cc0b2130 2022-06-29T16:04:15.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.755+0000 7fb6ca59c700 1 --2- 172.21.15.60:0/3389695003 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb6a8053a70 0x7fb6a8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:15.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.755+0000 7fb6ca59c700 1 --2- 172.21.15.60:0/3389695003 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb6a8053a70 0x7fb6a8055f30 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fb6c400b000 tx=0x7fb6c4008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:15.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.759+0000 7fb6bb7fe700 1 -- 172.21.15.60:0/3389695003 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb6cc065a40 con 0x7fb6cc0b2130 2022-06-29T16:04:15.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.887+0000 7fb6cb59e700 1 -- 172.21.15.60:0/3389695003 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fb6cc065a40 con 0x7fb6cc0b2130 2022-06-29T16:04:15.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.887+0000 7fb6bb7fe700 1 -- 172.21.15.60:0/3389695003 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v71) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb6cc065a40 con 0x7fb6cc0b2130 2022-06-29T16:04:15.895 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:04:15.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.891+0000 7fb6cb59e700 1 -- 172.21.15.60:0/3389695003 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb6a8053a70 msgr2=0x7fb6a8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:15.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.891+0000 7fb6cb59e700 1 --2- 172.21.15.60:0/3389695003 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb6a8053a70 0x7fb6a8055f30 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fb6c400b000 tx=0x7fb6c4008040).stop 2022-06-29T16:04:15.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.891+0000 7fb6cb59e700 1 -- 172.21.15.60:0/3389695003 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6cc0b2130 msgr2=0x7fb6cc139d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:15.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.891+0000 7fb6cb59e700 1 --2- 172.21.15.60:0/3389695003 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6cc0b2130 0x7fb6cc139d10 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0x7fb6cc14a970 tx=0x7fb6c0007be0).stop 2022-06-29T16:04:15.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.891+0000 7fb6cb59e700 1 -- 172.21.15.60:0/3389695003 shutdown_connections 2022-06-29T16:04:15.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.891+0000 7fb6cb59e700 1 --2- 172.21.15.60:0/3389695003 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb6a8053a70 0x7fb6a8055f30 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:15.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.891+0000 7fb6cb59e700 1 --2- 172.21.15.60:0/3389695003 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6cc14af80 0x7fb6cc13a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:15.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.891+0000 7fb6cb59e700 1 --2- 172.21.15.60:0/3389695003 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb6cc0b2130 0x7fb6cc139d10 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:15.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.891+0000 7fb6cb59e700 1 --2- 172.21.15.60:0/3389695003 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6cc05d070 0x7fb6cc0b46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:15.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.891+0000 7fb6cb59e700 1 -- 172.21.15.60:0/3389695003 >> 172.21.15.60:0/3389695003 conn(0x7fb6cc0b5890 msgr2=0x7fb6cc143470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:15.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.891+0000 7fb6cb59e700 1 -- 172.21.15.60:0/3389695003 shutdown_connections 2022-06-29T16:04:15.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:15.891+0000 7fb6cb59e700 1 -- 172.21.15.60:0/3389695003 wait complete. 2022-06-29T16:04:15.909 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-29T16:04:16.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.011+0000 7f6ea0984700 1 Processor -- start 2022-06-29T16:04:16.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.011+0000 7f6ea0984700 1 -- start start 2022-06-29T16:04:16.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.011+0000 7f6ea0984700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e9c1426b0 0x7f6e9c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:16.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.011+0000 7f6ea0984700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e9c143a40 con 0x7f6e9c0ae7c0 2022-06-29T16:04:16.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.011+0000 7f6ea0984700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e9c0aee60 con 0x7f6e9c0b4010 2022-06-29T16:04:16.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.011+0000 7f6ea0984700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c0af100 con 0x7f6e9c1426b0 2022-06-29T16:04:16.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.015+0000 7f6e9a59c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6e9c0ae7c0 0x7f6e9c141fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57588/0 (socket says 172.21.15.60:57588) 2022-06-29T16:04:16.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.015+0000 7f6e9a59c700 1 -- 172.21.15.60:0/1597378342 learned_addr learned my addr 172.21.15.60:0/1597378342 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:04:16.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.015+0000 7f6e9ad9d700 1 --2- 172.21.15.60:0/1597378342 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e9c1426b0 0x7f6e9c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:16.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.015+0000 7f6e9959a700 1 -- 172.21.15.60:0/1597378342 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707120515 0 0) 0x7f6e9c143a40 con 0x7f6e9c0ae7c0 2022-06-29T16:04:16.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.015+0000 7f6e9959a700 1 -- 172.21.15.60:0/1597378342 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e80003260 con 0x7f6e9c0ae7c0 2022-06-29T16:04:16.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.015+0000 7f6e9959a700 1 -- 172.21.15.60:0/1597378342 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269850240 0 0) 0x7f6e9c0aee60 con 0x7f6e9c0b4010 2022-06-29T16:04:16.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.015+0000 7f6e9959a700 1 -- 172.21.15.60:0/1597378342 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e9c143a40 con 0x7f6e9c0b4010 2022-06-29T16:04:16.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.015+0000 7f6e9ad9d700 1 -- 172.21.15.60:0/1597378342 >> v1:172.21.15.92:6789/0 conn(0x7f6e9c0b4010 legacy=0x7f6e9c0b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:16.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.015+0000 7f6e9ad9d700 1 -- 172.21.15.60:0/1597378342 >> v1:172.21.15.60:6789/0 conn(0x7f6e9c0ae7c0 legacy=0x7f6e9c141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:16.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.015+0000 7f6e9ad9d700 1 -- 172.21.15.60:0/1597378342 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e9c0aeba0 con 0x7f6e9c1426b0 2022-06-29T16:04:16.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.015+0000 7f6e9ad9d700 1 --2- 172.21.15.60:0/1597378342 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e9c1426b0 0x7f6e9c0b4dd0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f6e940032f0 tx=0x7f6e940091d0).ready entity=mon.2 client_cookie=e1c88e9264f1a5ef server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:16.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.015+0000 7f6e9959a700 1 -- 172.21.15.60:0/1597378342 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e9400b640 con 0x7f6e9c1426b0 2022-06-29T16:04:16.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.015+0000 7f6e9959a700 1 -- 172.21.15.60:0/1597378342 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e9400b7a0 con 0x7f6e9c1426b0 2022-06-29T16:04:16.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.015+0000 7f6e9959a700 1 -- 172.21.15.60:0/1597378342 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e9400b910 con 0x7f6e9c1426b0 2022-06-29T16:04:16.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.015+0000 7f6ea0984700 1 -- 172.21.15.60:0/1597378342 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e9c1426b0 msgr2=0x7f6e9c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:16.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.015+0000 7f6ea0984700 1 --2- 172.21.15.60:0/1597378342 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e9c1426b0 0x7f6e9c0b4dd0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f6e940032f0 tx=0x7f6e940091d0).stop 2022-06-29T16:04:16.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.015+0000 7f6ea0984700 1 -- 172.21.15.60:0/1597378342 shutdown_connections 2022-06-29T16:04:16.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.015+0000 7f6ea0984700 1 --2- 172.21.15.60:0/1597378342 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e9c1426b0 0x7f6e9c0b4dd0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:16.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.015+0000 7f6ea0984700 1 -- 172.21.15.60:0/1597378342 >> 172.21.15.60:0/1597378342 conn(0x7f6e9c0b9340 msgr2=0x7f6e9c0b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:16.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.015+0000 7f6ea0984700 1 -- 172.21.15.60:0/1597378342 shutdown_connections 2022-06-29T16:04:16.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.015+0000 7f6ea0984700 1 -- 172.21.15.60:0/1597378342 wait complete. 2022-06-29T16:04:16.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.019+0000 7f6ea0984700 1 Processor -- start 2022-06-29T16:04:16.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.019+0000 7f6ea0984700 1 -- start start 2022-06-29T16:04:16.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.019+0000 7f6ea0984700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e9c0ae7c0 0x7f6e9c0af680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:16.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.019+0000 7f6ea0984700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e9c0b4010 0x7f6e9c155d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:16.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.019+0000 7f6ea0984700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e9c1426b0 0x7f6e9c156220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:16.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.019+0000 7f6ea0984700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c0b1720 con 0x7f6e9c0ae7c0 2022-06-29T16:04:16.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.019+0000 7f6ea0984700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c0b1950 con 0x7f6e9c1426b0 2022-06-29T16:04:16.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.019+0000 7f6ea0984700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c0b1bf0 con 0x7f6e9c0b4010 2022-06-29T16:04:16.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.019+0000 7f6e9a59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e9c0ae7c0 0x7f6e9c0af680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:16.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.019+0000 7f6e9a59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e9c0ae7c0 0x7f6e9c0af680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49508/0 (socket says 172.21.15.60:49508) 2022-06-29T16:04:16.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.019+0000 7f6e9a59c700 1 -- 172.21.15.60:0/2567968391 learned_addr learned my addr 172.21.15.60:0/2567968391 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:16.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.019+0000 7f6e99d9b700 1 --2- 172.21.15.60:0/2567968391 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e9c0b4010 0x7f6e9c155d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:16.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.019+0000 7f6e9ad9d700 1 --2- 172.21.15.60:0/2567968391 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e9c1426b0 0x7f6e9c156220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:16.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.019+0000 7f6e9a59c700 1 -- 172.21.15.60:0/2567968391 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e9c0b4010 msgr2=0x7f6e9c155d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:16.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.019+0000 7f6e9a59c700 1 --2- 172.21.15.60:0/2567968391 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e9c0b4010 0x7f6e9c155d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:16.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.019+0000 7f6e9a59c700 1 -- 172.21.15.60:0/2567968391 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e9c1426b0 msgr2=0x7f6e9c156220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:16.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.019+0000 7f6e9a59c700 1 --2- 172.21.15.60:0/2567968391 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e9c1426b0 0x7f6e9c156220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:16.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.019+0000 7f6e9a59c700 1 -- 172.21.15.60:0/2567968391 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e94009470 con 0x7f6e9c0ae7c0 2022-06-29T16:04:16.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.019+0000 7f6e9a59c700 1 --2- 172.21.15.60:0/2567968391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e9c0ae7c0 0x7f6e9c0af680 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0x7f6e8c0010d0 tx=0x7f6e8c007b00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:16.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.019+0000 7f6e8b7fe700 1 -- 172.21.15.60:0/2567968391 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e8c00ce70 con 0x7f6e9c0ae7c0 2022-06-29T16:04:16.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.019+0000 7f6ea0984700 1 -- 172.21.15.60:0/2567968391 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e9c0b1e90 con 0x7f6e9c0ae7c0 2022-06-29T16:04:16.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.019+0000 7f6e8b7fe700 1 -- 172.21.15.60:0/2567968391 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e8c0070d0 con 0x7f6e9c0ae7c0 2022-06-29T16:04:16.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.019+0000 7f6e8b7fe700 1 -- 172.21.15.60:0/2567968391 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e8c007430 con 0x7f6e9c0ae7c0 2022-06-29T16:04:16.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.019+0000 7f6ea0984700 1 -- 172.21.15.60:0/2567968391 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e9c155700 con 0x7f6e9c0ae7c0 2022-06-29T16:04:16.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.023+0000 7f6e8b7fe700 1 -- 172.21.15.60:0/2567968391 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6e8c010070 con 0x7f6e9c0ae7c0 2022-06-29T16:04:16.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.023+0000 7f6ea0984700 1 -- 172.21.15.60:0/2567968391 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e9c112870 con 0x7f6e9c0ae7c0 2022-06-29T16:04:16.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.023+0000 7f6e8b7fe700 1 --2- 172.21.15.60:0/2567968391 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6e78053a70 0x7f6e78055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:16.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.023+0000 7f6e8b7fe700 1 -- 172.21.15.60:0/2567968391 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6e8c06b380 con 0x7f6e9c0ae7c0 2022-06-29T16:04:16.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.023+0000 7f6e99d9b700 1 --2- 172.21.15.60:0/2567968391 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6e78053a70 0x7f6e78055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:16.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.023+0000 7f6e99d9b700 1 --2- 172.21.15.60:0/2567968391 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6e78053a70 0x7f6e78055f30 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f6e9000d000 tx=0x7f6e90007a20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:16.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.031+0000 7f6e8b7fe700 1 -- 172.21.15.60:0/2567968391 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6e9c112870 con 0x7f6e9c0ae7c0 2022-06-29T16:04:16.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.159+0000 7f6ea0984700 1 -- 172.21.15.60:0/2567968391 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6e9c112870 con 0x7f6e9c0ae7c0 2022-06-29T16:04:16.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.159+0000 7f6e8b7fe700 1 -- 172.21.15.60:0/2567968391 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v71) v1 ==== 104+0+10 (secure 0 0 0) 0x7f6e9c112870 con 0x7f6e9c0ae7c0 2022-06-29T16:04:16.168 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:04:16.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.163+0000 7f6ea0984700 1 -- 172.21.15.60:0/2567968391 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6e78053a70 msgr2=0x7f6e78055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:16.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.163+0000 7f6ea0984700 1 --2- 172.21.15.60:0/2567968391 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6e78053a70 0x7f6e78055f30 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f6e9000d000 tx=0x7f6e90007a20).stop 2022-06-29T16:04:16.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.163+0000 7f6ea0984700 1 -- 172.21.15.60:0/2567968391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e9c0ae7c0 msgr2=0x7f6e9c0af680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:16.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.163+0000 7f6ea0984700 1 --2- 172.21.15.60:0/2567968391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e9c0ae7c0 0x7f6e9c0af680 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0x7f6e8c0010d0 tx=0x7f6e8c007b00).stop 2022-06-29T16:04:16.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.163+0000 7f6ea0984700 1 -- 172.21.15.60:0/2567968391 shutdown_connections 2022-06-29T16:04:16.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.163+0000 7f6ea0984700 1 --2- 172.21.15.60:0/2567968391 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6e78053a70 0x7f6e78055f30 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:16.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.163+0000 7f6ea0984700 1 --2- 172.21.15.60:0/2567968391 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e9c1426b0 0x7f6e9c156220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:16.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.163+0000 7f6ea0984700 1 --2- 172.21.15.60:0/2567968391 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e9c0b4010 0x7f6e9c155d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:16.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.163+0000 7f6ea0984700 1 --2- 172.21.15.60:0/2567968391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e9c0ae7c0 0x7f6e9c0af680 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:16.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.163+0000 7f6ea0984700 1 -- 172.21.15.60:0/2567968391 >> 172.21.15.60:0/2567968391 conn(0x7f6e9c0b9340 msgr2=0x7f6e9c1402e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:16.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.163+0000 7f6ea0984700 1 -- 172.21.15.60:0/2567968391 shutdown_connections 2022-06-29T16:04:16.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.163+0000 7f6ea0984700 1 -- 172.21.15.60:0/2567968391 wait complete. 2022-06-29T16:04:16.181 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-29T16:04:16.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.283+0000 7f0cc2102700 1 Processor -- start 2022-06-29T16:04:16.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.283+0000 7f0cc2102700 1 -- start start 2022-06-29T16:04:16.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.283+0000 7f0cc2102700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0cbc0b5eb0 0x7f0cbc142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:16.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.283+0000 7f0cc2102700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0cbc0b2c50 0x7f0cbc0b3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:16.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.283+0000 7f0cc2102700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cbc0b0cf0 con 0x7f0cbc0b48b0 2022-06-29T16:04:16.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.283+0000 7f0cc2102700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cbc0b54e0 con 0x7f0cbc0b2c50 2022-06-29T16:04:16.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.283+0000 7f0cc2102700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc0b4d10 con 0x7f0cbc0b5eb0 2022-06-29T16:04:16.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.283+0000 7f0cc1901700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0cbc0b2c50 0x7f0cbc0b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:16.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.283+0000 7f0cc1901700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0cbc0b2c50 0x7f0cbc0b3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49520/0 (socket says 172.21.15.60:49520) 2022-06-29T16:04:16.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.283+0000 7f0cc1901700 1 -- 172.21.15.60:0/1241797291 learned_addr learned my addr 172.21.15.60:0/1241797291 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:16.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.283+0000 7f0cc1100700 1 --2- 172.21.15.60:0/1241797291 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0cbc0b5eb0 0x7f0cbc142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:16.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.283+0000 7f0cc1901700 1 -- 172.21.15.60:0/1241797291 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0cbc0b5eb0 msgr2=0x7f0cbc142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:16.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.283+0000 7f0cabfff700 1 -- 172.21.15.60:0/1241797291 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 184864045 0 0) 0x7f0cac001110 con 0x7f0cbc0b48b0 2022-06-29T16:04:16.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.283+0000 7f0cc1901700 1 --2- 172.21.15.60:0/1241797291 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0cbc0b5eb0 0x7f0cbc142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:16.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.283+0000 7f0cc1901700 1 -- 172.21.15.60:0/1241797291 >> v1:172.21.15.92:6789/0 conn(0x7f0cbc0b48b0 legacy=0x7f0cbc0b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:16.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.283+0000 7f0cc1901700 1 -- 172.21.15.60:0/1241797291 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cbc0b6290 con 0x7f0cbc0b2c50 2022-06-29T16:04:16.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.283+0000 7f0cc1901700 1 --2- 172.21.15.60:0/1241797291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0cbc0b2c50 0x7f0cbc0b3090 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 rx=0x7f0cb8002f70 tx=0x7f0cb8006eb0).ready entity=mon.0 client_cookie=76deb2b851d5d502 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:16.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.283+0000 7f0cabfff700 1 -- 172.21.15.60:0/1241797291 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cb800d570 con 0x7f0cbc0b2c50 2022-06-29T16:04:16.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.283+0000 7f0cabfff700 1 -- 172.21.15.60:0/1241797291 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cb8000e90 con 0x7f0cbc0b2c50 2022-06-29T16:04:16.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.283+0000 7f0cabfff700 1 -- 172.21.15.60:0/1241797291 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cb800d8c0 con 0x7f0cbc0b2c50 2022-06-29T16:04:16.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.287+0000 7f0cc2102700 1 -- 172.21.15.60:0/1241797291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0cbc0b2c50 msgr2=0x7f0cbc0b3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:16.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.287+0000 7f0cc2102700 1 --2- 172.21.15.60:0/1241797291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0cbc0b2c50 0x7f0cbc0b3090 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 rx=0x7f0cb8002f70 tx=0x7f0cb8006eb0).stop 2022-06-29T16:04:16.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.287+0000 7f0cc2102700 1 -- 172.21.15.60:0/1241797291 shutdown_connections 2022-06-29T16:04:16.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.287+0000 7f0cc2102700 1 --2- 172.21.15.60:0/1241797291 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0cbc0b2c50 0x7f0cbc0b3090 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:16.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.287+0000 7f0cc2102700 1 --2- 172.21.15.60:0/1241797291 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0cbc0b5eb0 0x7f0cbc142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:16.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.287+0000 7f0cc2102700 1 -- 172.21.15.60:0/1241797291 >> 172.21.15.60:0/1241797291 conn(0x7f0cbc062090 msgr2=0x7f0cbc062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:16.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.287+0000 7f0cc2102700 1 -- 172.21.15.60:0/1241797291 shutdown_connections 2022-06-29T16:04:16.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.287+0000 7f0cc2102700 1 -- 172.21.15.60:0/1241797291 wait complete. 2022-06-29T16:04:16.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.287+0000 7f0cc2102700 1 Processor -- start 2022-06-29T16:04:16.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.287+0000 7f0cc2102700 1 -- start start 2022-06-29T16:04:16.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.291+0000 7f0cc2102700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0cbc0b2c50 0x7f0cbc0af440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:16.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.291+0000 7f0cc2102700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0cbc0b48b0 0x7f0cbc152800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:16.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.291+0000 7f0cc2102700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0cbc0b5eb0 0x7f0cbc0aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:16.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.291+0000 7f0cc2102700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc148fa0 con 0x7f0cbc0b48b0 2022-06-29T16:04:16.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.291+0000 7f0cc2102700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc0b1870 con 0x7f0cbc0b2c50 2022-06-29T16:04:16.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.291+0000 7f0cc2102700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc140520 con 0x7f0cbc0b5eb0 2022-06-29T16:04:16.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.291+0000 7f0cc08ff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0cbc0b48b0 0x7f0cbc152800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:16.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.291+0000 7f0cc08ff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0cbc0b48b0 0x7f0cbc152800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49522/0 (socket says 172.21.15.60:49522) 2022-06-29T16:04:16.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.291+0000 7f0cc08ff700 1 -- 172.21.15.60:0/3984260548 learned_addr learned my addr 172.21.15.60:0/3984260548 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:16.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.291+0000 7f0cc1100700 1 --2- 172.21.15.60:0/3984260548 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0cbc0b2c50 0x7f0cbc0af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:16.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.291+0000 7f0cc1901700 1 --2- 172.21.15.60:0/3984260548 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0cbc0b5eb0 0x7f0cbc0aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:16.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.291+0000 7f0cc08ff700 1 -- 172.21.15.60:0/3984260548 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0cbc0b5eb0 msgr2=0x7f0cbc0aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:16.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.291+0000 7f0cc08ff700 1 --2- 172.21.15.60:0/3984260548 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0cbc0b5eb0 0x7f0cbc0aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:16.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.291+0000 7f0cc08ff700 1 -- 172.21.15.60:0/3984260548 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0cbc0b2c50 msgr2=0x7f0cbc0af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:16.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.291+0000 7f0cc08ff700 1 --2- 172.21.15.60:0/3984260548 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0cbc0b2c50 0x7f0cbc0af440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:16.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.291+0000 7f0cc08ff700 1 -- 172.21.15.60:0/3984260548 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb800b1e0 con 0x7f0cbc0b48b0 2022-06-29T16:04:16.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.291+0000 7f0cc1901700 1 --2- 172.21.15.60:0/3984260548 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0cbc0b5eb0 0x7f0cbc0aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:04:16.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.291+0000 7f0cc08ff700 1 --2- 172.21.15.60:0/3984260548 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0cbc0b48b0 0x7f0cbc152800 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0x7f0cac000bc0 tx=0x7f0cac0017f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:16.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.291+0000 7f0caa7fc700 1 -- 172.21.15.60:0/3984260548 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cac012400 con 0x7f0cbc0b48b0 2022-06-29T16:04:16.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.291+0000 7f0caa7fc700 1 -- 172.21.15.60:0/3984260548 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cac012560 con 0x7f0cbc0b48b0 2022-06-29T16:04:16.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.291+0000 7f0cc2102700 1 -- 172.21.15.60:0/3984260548 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cbc152cf0 con 0x7f0cbc0b48b0 2022-06-29T16:04:16.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.291+0000 7f0caa7fc700 1 -- 172.21.15.60:0/3984260548 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cac0128c0 con 0x7f0cbc0b48b0 2022-06-29T16:04:16.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.291+0000 7f0cc2102700 1 -- 172.21.15.60:0/3984260548 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cbc1401a0 con 0x7f0cbc0b48b0 2022-06-29T16:04:16.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.291+0000 7f0cc2102700 1 -- 172.21.15.60:0/3984260548 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cbc142f10 con 0x7f0cbc0b48b0 2022-06-29T16:04:16.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.295+0000 7f0caa7fc700 1 -- 172.21.15.60:0/3984260548 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0cac019030 con 0x7f0cbc0b48b0 2022-06-29T16:04:16.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.295+0000 7f0caa7fc700 1 --2- 172.21.15.60:0/3984260548 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0c9c053b30 0x7f0c9c055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:16.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.295+0000 7f0caa7fc700 1 -- 172.21.15.60:0/3984260548 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0cac06b110 con 0x7f0cbc0b48b0 2022-06-29T16:04:16.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.295+0000 7f0cc1100700 1 --2- 172.21.15.60:0/3984260548 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0c9c053b30 0x7f0c9c055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:16.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.295+0000 7f0cc1100700 1 --2- 172.21.15.60:0/3984260548 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0c9c053b30 0x7f0c9c055ff0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f0cb400b000 tx=0x7f0cb40091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:16.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.299+0000 7f0caa7fc700 1 -- 172.21.15.60:0/3984260548 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0cac041f90 con 0x7f0cbc0b48b0 2022-06-29T16:04:16.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.427+0000 7f0cc2102700 1 -- 172.21.15.60:0/3984260548 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0cbc0b0cf0 con 0x7f0cbc0b48b0 2022-06-29T16:04:16.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.427+0000 7f0caa7fc700 1 -- 172.21.15.60:0/3984260548 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v71) v1 ==== 100+0+10 (secure 0 0 0) 0x7f0cbc0b0cf0 con 0x7f0cbc0b48b0 2022-06-29T16:04:16.433 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:04:16.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.427+0000 7f0cc2102700 1 -- 172.21.15.60:0/3984260548 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0c9c053b30 msgr2=0x7f0c9c055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:16.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.427+0000 7f0cc2102700 1 --2- 172.21.15.60:0/3984260548 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0c9c053b30 0x7f0c9c055ff0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f0cb400b000 tx=0x7f0cb40091d0).stop 2022-06-29T16:04:16.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.427+0000 7f0cc2102700 1 -- 172.21.15.60:0/3984260548 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0cbc0b48b0 msgr2=0x7f0cbc152800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:16.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.427+0000 7f0cc2102700 1 --2- 172.21.15.60:0/3984260548 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0cbc0b48b0 0x7f0cbc152800 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0x7f0cac000bc0 tx=0x7f0cac0017f0).stop 2022-06-29T16:04:16.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.427+0000 7f0cc2102700 1 -- 172.21.15.60:0/3984260548 shutdown_connections 2022-06-29T16:04:16.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.427+0000 7f0cc2102700 1 --2- 172.21.15.60:0/3984260548 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0c9c053b30 0x7f0c9c055ff0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:16.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.427+0000 7f0cc2102700 1 --2- 172.21.15.60:0/3984260548 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0cbc0b5eb0 0x7f0cbc0aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:16.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.427+0000 7f0cc2102700 1 --2- 172.21.15.60:0/3984260548 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0cbc0b48b0 0x7f0cbc152800 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:16.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.427+0000 7f0cc2102700 1 --2- 172.21.15.60:0/3984260548 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0cbc0b2c50 0x7f0cbc0af440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:16.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.427+0000 7f0cc2102700 1 -- 172.21.15.60:0/3984260548 >> 172.21.15.60:0/3984260548 conn(0x7f0cbc062090 msgr2=0x7f0cbc0b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:16.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.427+0000 7f0cc2102700 1 -- 172.21.15.60:0/3984260548 shutdown_connections 2022-06-29T16:04:16.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.427+0000 7f0cc2102700 1 -- 172.21.15.60:0/3984260548 wait complete. 2022-06-29T16:04:16.450 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-06-29T16:04:16.450 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:04:16.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.551+0000 7f00a60c3700 1 Processor -- start 2022-06-29T16:04:16.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.551+0000 7f00a60c3700 1 -- start start 2022-06-29T16:04:16.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.551+0000 7f00a60c3700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00a0063ad0 0x7f00a0067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:16.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.551+0000 7f00a60c3700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f00a014af10 0x7f00a005c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:16.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.551+0000 7f00a60c3700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00a00acba0 con 0x7f00a00b5950 2022-06-29T16:04:16.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.551+0000 7f00a60c3700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00a014b3a0 con 0x7f00a014af10 2022-06-29T16:04:16.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.551+0000 7f00a60c3700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00a0067220 con 0x7f00a0063ad0 2022-06-29T16:04:16.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.551+0000 7f009f7fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f00a00b5950 0x7f00a005bd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57616/0 (socket says 172.21.15.60:57616) 2022-06-29T16:04:16.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.551+0000 7f009f7fe700 1 -- 172.21.15.60:0/3833078326 learned_addr learned my addr 172.21.15.60:0/3833078326 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:04:16.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.551+0000 7f009effd700 1 --2- 172.21.15.60:0/3833078326 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00a0063ad0 0x7f00a0067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:16.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.551+0000 7f009ffff700 1 --2- 172.21.15.60:0/3833078326 >> v2:172.21.15.92:3300/0 conn(0x7f00a014af10 0x7f00a005c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:16.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.551+0000 7f009e7fc700 1 -- 172.21.15.60:0/3833078326 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1462646079 0 0) 0x7f00a00acba0 con 0x7f00a00b5950 2022-06-29T16:04:16.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.551+0000 7f009e7fc700 1 -- 172.21.15.60:0/3833078326 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0084003260 con 0x7f00a00b5950 2022-06-29T16:04:16.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.551+0000 7f009effd700 1 -- 172.21.15.60:0/3833078326 >> v2:172.21.15.92:3300/0 conn(0x7f00a014af10 msgr2=0x7f00a005c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:16.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.551+0000 7f009effd700 1 --2- 172.21.15.60:0/3833078326 >> v2:172.21.15.92:3300/0 conn(0x7f00a014af10 0x7f00a005c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:16.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.551+0000 7f009effd700 1 -- 172.21.15.60:0/3833078326 >> v1:172.21.15.60:6789/0 conn(0x7f00a00b5950 legacy=0x7f00a005bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:16.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.551+0000 7f009effd700 1 -- 172.21.15.60:0/3833078326 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a00b5d30 con 0x7f00a0063ad0 2022-06-29T16:04:16.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.551+0000 7f009effd700 1 --2- 172.21.15.60:0/3833078326 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00a0063ad0 0x7f00a0067680 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f0090002f70 tx=0x7f0090006df0).ready entity=mon.2 client_cookie=cf64bc007f55fede server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:16.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.551+0000 7f009e7fc700 1 -- 172.21.15.60:0/3833078326 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f009000d4a0 con 0x7f00a0063ad0 2022-06-29T16:04:16.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.551+0000 7f009e7fc700 1 -- 172.21.15.60:0/3833078326 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0090000e90 con 0x7f00a0063ad0 2022-06-29T16:04:16.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.551+0000 7f009e7fc700 1 -- 172.21.15.60:0/3833078326 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f009000d7f0 con 0x7f00a0063ad0 2022-06-29T16:04:16.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.551+0000 7f00a60c3700 1 -- 172.21.15.60:0/3833078326 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00a0063ad0 msgr2=0x7f00a0067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:16.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.551+0000 7f00a60c3700 1 --2- 172.21.15.60:0/3833078326 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00a0063ad0 0x7f00a0067680 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f0090002f70 tx=0x7f0090006df0).stop 2022-06-29T16:04:16.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.551+0000 7f00a60c3700 1 -- 172.21.15.60:0/3833078326 shutdown_connections 2022-06-29T16:04:16.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.551+0000 7f00a60c3700 1 --2- 172.21.15.60:0/3833078326 >> v2:172.21.15.92:3300/0 conn(0x7f00a014af10 0x7f00a005c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:16.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.551+0000 7f00a60c3700 1 --2- 172.21.15.60:0/3833078326 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00a0063ad0 0x7f00a0067680 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:16.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.551+0000 7f00a60c3700 1 -- 172.21.15.60:0/3833078326 >> 172.21.15.60:0/3833078326 conn(0x7f00a00632c0 msgr2=0x7f00a00b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:16.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.555+0000 7f00a60c3700 1 -- 172.21.15.60:0/3833078326 shutdown_connections 2022-06-29T16:04:16.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.555+0000 7f00a60c3700 1 -- 172.21.15.60:0/3833078326 wait complete. 2022-06-29T16:04:16.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.555+0000 7f00a60c3700 1 Processor -- start 2022-06-29T16:04:16.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.555+0000 7f00a60c3700 1 -- start start 2022-06-29T16:04:16.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.555+0000 7f00a60c3700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00a0063ad0 0x7f00a00aee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:16.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.555+0000 7f00a60c3700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00a00b5950 0x7f00a0156940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:16.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.555+0000 7f00a60c3700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00a014af10 0x7f00a0148320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:16.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.555+0000 7f00a60c3700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a0116cf0 con 0x7f00a014af10 2022-06-29T16:04:16.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.555+0000 7f00a60c3700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a0156200 con 0x7f00a0063ad0 2022-06-29T16:04:16.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.555+0000 7f00a60c3700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00a01564a0 con 0x7f00a00b5950 2022-06-29T16:04:16.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.555+0000 7f009ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00a014af10 0x7f00a0148320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:16.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.555+0000 7f009ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00a014af10 0x7f00a0148320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49538/0 (socket says 172.21.15.60:49538) 2022-06-29T16:04:16.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.555+0000 7f009ffff700 1 -- 172.21.15.60:0/939357940 learned_addr learned my addr 172.21.15.60:0/939357940 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:16.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.555+0000 7f009effd700 1 --2- 172.21.15.60:0/939357940 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00a00b5950 0x7f00a0156940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:16.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.555+0000 7f009f7fe700 1 --2- 172.21.15.60:0/939357940 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00a0063ad0 0x7f00a00aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:16.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.555+0000 7f009ffff700 1 -- 172.21.15.60:0/939357940 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00a00b5950 msgr2=0x7f00a0156940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:16.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.555+0000 7f009ffff700 1 --2- 172.21.15.60:0/939357940 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00a00b5950 0x7f00a0156940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:16.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.555+0000 7f009ffff700 1 -- 172.21.15.60:0/939357940 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00a0063ad0 msgr2=0x7f00a00aee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:16.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.555+0000 7f009ffff700 1 --2- 172.21.15.60:0/939357940 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00a0063ad0 0x7f00a00aee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:16.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.555+0000 7f009ffff700 1 -- 172.21.15.60:0/939357940 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f009000b180 con 0x7f00a014af10 2022-06-29T16:04:16.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.555+0000 7f009ffff700 1 --2- 172.21.15.60:0/939357940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00a014af10 0x7f00a0148320 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0x7f0098006e00 tx=0x7f00980091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:16.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.555+0000 7f009cff9700 1 -- 172.21.15.60:0/939357940 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0098012600 con 0x7f00a014af10 2022-06-29T16:04:16.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.559+0000 7f009cff9700 1 -- 172.21.15.60:0/939357940 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0098012760 con 0x7f00a014af10 2022-06-29T16:04:16.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.559+0000 7f009cff9700 1 -- 172.21.15.60:0/939357940 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0098012ac0 con 0x7f00a014af10 2022-06-29T16:04:16.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.559+0000 7f00a60c3700 1 -- 172.21.15.60:0/939357940 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00a0149f00 con 0x7f00a014af10 2022-06-29T16:04:16.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.559+0000 7f00a60c3700 1 -- 172.21.15.60:0/939357940 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00a0157220 con 0x7f00a014af10 2022-06-29T16:04:16.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.559+0000 7f00a60c3700 1 -- 172.21.15.60:0/939357940 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00a00573d0 con 0x7f00a014af10 2022-06-29T16:04:16.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.559+0000 7f009cff9700 1 -- 172.21.15.60:0/939357940 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0098022070 con 0x7f00a014af10 2022-06-29T16:04:16.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.559+0000 7f009cff9700 1 --2- 172.21.15.60:0/939357940 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0078053aa0 0x7f0078055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:16.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.559+0000 7f009cff9700 1 -- 172.21.15.60:0/939357940 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f009806c370 con 0x7f00a014af10 2022-06-29T16:04:16.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.559+0000 7f009f7fe700 1 --2- 172.21.15.60:0/939357940 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0078053aa0 0x7f0078055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:16.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.563+0000 7f009f7fe700 1 --2- 172.21.15.60:0/939357940 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0078053aa0 0x7f0078055f60 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f008800d000 tx=0x7f0088009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:16.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.567+0000 7f009cff9700 1 -- 172.21.15.60:0/939357940 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f00980432a0 con 0x7f00a014af10 2022-06-29T16:04:16.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.731+0000 7f00a60c3700 1 -- 172.21.15.60:0/939357940 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00a0064980 con 0x7f00a014af10 2022-06-29T16:04:16.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.731+0000 7f009cff9700 1 -- 172.21.15.60:0/939357940 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f0098016870 con 0x7f00a014af10 2022-06-29T16:04:16.739 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:04:16.740 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":30,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5690,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1158969084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1158969084},{"type":"v1","addr":"172.21.15.60:6833","nonce":1158969084}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":28}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:04:08.275039+0000","modified":"2022-06-29T16:04:09.279046+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6304":{"gid":6304,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/953938586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":953938586},{"type":"v1","addr":"172.21.15.92:6835","nonce":953938586}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-06-29T16:04:16.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.735+0000 7f00a60c3700 1 -- 172.21.15.60:0/939357940 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0078053aa0 msgr2=0x7f0078055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:16.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.735+0000 7f00a60c3700 1 --2- 172.21.15.60:0/939357940 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0078053aa0 0x7f0078055f60 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f008800d000 tx=0x7f0088009040).stop 2022-06-29T16:04:16.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.735+0000 7f00a60c3700 1 -- 172.21.15.60:0/939357940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00a014af10 msgr2=0x7f00a0148320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:16.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.735+0000 7f00a60c3700 1 --2- 172.21.15.60:0/939357940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00a014af10 0x7f00a0148320 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0x7f0098006e00 tx=0x7f00980091d0).stop 2022-06-29T16:04:16.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.735+0000 7f00a60c3700 1 -- 172.21.15.60:0/939357940 shutdown_connections 2022-06-29T16:04:16.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.735+0000 7f00a60c3700 1 --2- 172.21.15.60:0/939357940 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0078053aa0 0x7f0078055f60 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:16.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.735+0000 7f00a60c3700 1 --2- 172.21.15.60:0/939357940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00a014af10 0x7f00a0148320 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:16.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.735+0000 7f00a60c3700 1 --2- 172.21.15.60:0/939357940 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00a00b5950 0x7f00a0156940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:16.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.735+0000 7f00a60c3700 1 --2- 172.21.15.60:0/939357940 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00a0063ad0 0x7f00a00aee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:16.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.735+0000 7f00a60c3700 1 -- 172.21.15.60:0/939357940 >> 172.21.15.60:0/939357940 conn(0x7f00a00632c0 msgr2=0x7f00a0144440 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:16.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.735+0000 7f00a60c3700 1 -- 172.21.15.60:0/939357940 shutdown_connections 2022-06-29T16:04:16.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:16.735+0000 7f00a60c3700 1 -- 172.21.15.60:0/939357940 wait complete. 2022-06-29T16:04:16.744 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 30 2022-06-29T16:04:16.954 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:04:08.275039+0000', 'modified': '2022-06-29T16:04:09.279046+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6304}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6304': {'gid': 6304, 'name': 'b', 'rank': 0, 'incarnation': 29, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.92:6835/953938586', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 953938586}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 953938586}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:04:16.955 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:04:16.956 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-06-29T16:04:17.068 INFO:teuthology.orchestra.run.smithi092.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":6304,"want_state":"up:active","state":"up:active","rank_uptime":8.7780139869999996,"mdsmap_epoch":30,"osdmap_epoch":71,"osdmap_epoch_barrier":71,"uptime":13.159149523} 2022-06-29T16:04:17.069 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 6304, "mdsmap_epoch": 30, "osdmap_epoch": 71, "osdmap_epoch_barrier": 71, "rank_uptime": 8.778013987, "state": "up:active", "uptime": 13.159149523, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:04:17.070 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-06-29T16:04:17.070 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-06-29T16:04:17.071 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi156.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-06-29T16:04:17.071 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:04:17.071 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:04:17.080 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2022-06-29T16:04:17.096 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:04:17.097 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:04:17.156 INFO:teuthology.orchestra.run.smithi156.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:04:17.157 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:04:17.158 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:04:17.158 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2022-06-29T16:04:17.163 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-06-29T16:04:17.164 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:04:17.166 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2022-06-29T16:04:17.183 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:04:17.184 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:04:17.239 INFO:teuthology.orchestra.run.smithi156.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:04:17.241 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:04:17.242 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:04:17.242 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2022-06-29T16:04:17.243 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:2022-06-29T16:04:17.239+0000 7fa58bec3080 -1 init, newargv = 0x55ce754120f0 newargc=9 2022-06-29T16:04:17.244 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[12384]: starting ceph client 2022-06-29T16:04:17.252 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[12384]: starting fuse 2022-06-29T16:04:18.351 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2022-06-29T16:04:18.369 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:04:18.370 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:04:18.428 INFO:teuthology.orchestra.run.smithi156.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:04:18.428 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:04:18.429 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:04:18.430 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2022-06-29T16:04:18.435 INFO:teuthology.orchestra.run.smithi156.stdout:52 2022-06-29T16:04:18.435 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-06-29T16:04:18.436 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:04:18.436 DEBUG:teuthology.orchestra.run.smithi156:> sudo python3 -c ' 2022-06-29T16:04:18.436 DEBUG:teuthology.orchestra.run.smithi156:> import glob 2022-06-29T16:04:18.437 DEBUG:teuthology.orchestra.run.smithi156:> import re 2022-06-29T16:04:18.437 DEBUG:teuthology.orchestra.run.smithi156:> import os 2022-06-29T16:04:18.438 DEBUG:teuthology.orchestra.run.smithi156:> import subprocess 2022-06-29T16:04:18.438 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:04:18.438 DEBUG:teuthology.orchestra.run.smithi156:> def find_socket(client_name): 2022-06-29T16:04:18.439 DEBUG:teuthology.orchestra.run.smithi156:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-06-29T16:04:18.439 DEBUG:teuthology.orchestra.run.smithi156:> files = glob.glob(asok_path) 2022-06-29T16:04:18.440 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:04:18.440 DEBUG:teuthology.orchestra.run.smithi156:> # Given a non-glob path, it better be there 2022-06-29T16:04:18.440 DEBUG:teuthology.orchestra.run.smithi156:> if "*" not in asok_path: 2022-06-29T16:04:18.441 DEBUG:teuthology.orchestra.run.smithi156:> assert(len(files) == 1) 2022-06-29T16:04:18.442 DEBUG:teuthology.orchestra.run.smithi156:> return files[0] 2022-06-29T16:04:18.443 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:04:18.443 DEBUG:teuthology.orchestra.run.smithi156:> for f in files: 2022-06-29T16:04:18.444 DEBUG:teuthology.orchestra.run.smithi156:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-06-29T16:04:18.444 DEBUG:teuthology.orchestra.run.smithi156:> if os.path.exists("/proc/{0}".format(pid)): 2022-06-29T16:04:18.445 DEBUG:teuthology.orchestra.run.smithi156:> return f 2022-06-29T16:04:18.445 DEBUG:teuthology.orchestra.run.smithi156:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-06-29T16:04:18.445 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:04:18.446 DEBUG:teuthology.orchestra.run.smithi156:> print(find_socket("client.1")) 2022-06-29T16:04:18.446 DEBUG:teuthology.orchestra.run.smithi156:> ' 2022-06-29T16:04:18.491 INFO:teuthology.orchestra.run.smithi156.stdout:/var/run/ceph/ceph-client.1.12384.asok 2022-06-29T16:04:18.496 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.12384.asok 2022-06-29T16:04:18.496 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:04:18.497 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.12384.asok status 2022-06-29T16:04:18.634 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2022-06-29T16:04:18.634 INFO:teuthology.orchestra.run.smithi156.stdout: "metadata": { 2022-06-29T16:04:18.634 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_sha1": "f25ffac72e5a9e63860ff7deb6767a11f0559a16", 2022-06-29T16:04:18.635 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_version": "ceph version 15.2.16-304-gf25ffac7 (f25ffac72e5a9e63860ff7deb6767a11f0559a16) octopus (stable)", 2022-06-29T16:04:18.635 INFO:teuthology.orchestra.run.smithi156.stdout: "entity_id": "1", 2022-06-29T16:04:18.635 INFO:teuthology.orchestra.run.smithi156.stdout: "hostname": "smithi156", 2022-06-29T16:04:18.635 INFO:teuthology.orchestra.run.smithi156.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-06-29T16:04:18.636 INFO:teuthology.orchestra.run.smithi156.stdout: "pid": "12384", 2022-06-29T16:04:18.636 INFO:teuthology.orchestra.run.smithi156.stdout: "root": "/" 2022-06-29T16:04:18.636 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:04:18.636 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_count": 0, 2022-06-29T16:04:18.637 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_pinned_count": 0, 2022-06-29T16:04:18.637 INFO:teuthology.orchestra.run.smithi156.stdout: "id": 6477, 2022-06-29T16:04:18.637 INFO:teuthology.orchestra.run.smithi156.stdout: "inst": { 2022-06-29T16:04:18.637 INFO:teuthology.orchestra.run.smithi156.stdout: "name": { 2022-06-29T16:04:18.638 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "client", 2022-06-29T16:04:18.638 INFO:teuthology.orchestra.run.smithi156.stdout: "num": 6477 2022-06-29T16:04:18.638 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:04:18.638 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2022-06-29T16:04:18.638 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2022-06-29T16:04:18.639 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "172.21.15.156:0", 2022-06-29T16:04:18.639 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 4073660424 2022-06-29T16:04:18.639 INFO:teuthology.orchestra.run.smithi156.stdout: } 2022-06-29T16:04:18.639 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:04:18.640 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2022-06-29T16:04:18.640 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2022-06-29T16:04:18.640 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "172.21.15.156:0", 2022-06-29T16:04:18.640 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 4073660424 2022-06-29T16:04:18.641 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:04:18.641 INFO:teuthology.orchestra.run.smithi156.stdout: "inst_str": "client.6477 172.21.15.156:0/4073660424", 2022-06-29T16:04:18.641 INFO:teuthology.orchestra.run.smithi156.stdout: "addr_str": "172.21.15.156:0/4073660424", 2022-06-29T16:04:18.641 INFO:teuthology.orchestra.run.smithi156.stdout: "inode_count": 1, 2022-06-29T16:04:18.641 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_epoch": 30, 2022-06-29T16:04:18.642 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch": 71, 2022-06-29T16:04:18.642 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch_barrier": 0, 2022-06-29T16:04:18.642 INFO:teuthology.orchestra.run.smithi156.stdout: "blacklisted": false, 2022-06-29T16:04:18.642 INFO:teuthology.orchestra.run.smithi156.stdout: "fs_name": "cephfs" 2022-06-29T16:04:18.643 INFO:teuthology.orchestra.run.smithi156.stdout:} 2022-06-29T16:04:18.644 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:04:18.644 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:04:18.645 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:04:18.653 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2022-06-29T16:04:18.654 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-06-29T16:04:18.655 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:04:18.655 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:04:18.670 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:04:18.783 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.775+0000 7ff83cd06700 1 Processor -- start 2022-06-29T16:04:18.783 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.775+0000 7ff83cd06700 1 -- start start 2022-06-29T16:04:18.783 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.775+0000 7ff83cd06700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff83805d070 0x7ff83814aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:18.784 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.775+0000 7ff83cd06700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff83805d450 con 0x7ff8380b2130 2022-06-29T16:04:18.784 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.775+0000 7ff83cd06700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8380ad840 con 0x7ff83814af80 2022-06-29T16:04:18.784 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.775+0000 7ff83cd06700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8380adf90 con 0x7ff83805d070 2022-06-29T16:04:18.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.775+0000 7ff836ffd700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff8380b2130 0x7ff8380aca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57632/0 (socket says 172.21.15.60:57632) 2022-06-29T16:04:18.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.775+0000 7ff836ffd700 1 -- 172.21.15.60:0/1260183359 learned_addr learned my addr 172.21.15.60:0/1260183359 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:04:18.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.775+0000 7ff8377fe700 1 --2- 172.21.15.60:0/1260183359 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff83805d070 0x7ff83814aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:18.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.775+0000 7ff8367fc700 1 -- 172.21.15.60:0/1260183359 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840609923 0 0) 0x7ff83805d450 con 0x7ff8380b2130 2022-06-29T16:04:18.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.775+0000 7ff8367fc700 1 -- 172.21.15.60:0/1260183359 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff818003260 con 0x7ff8380b2130 2022-06-29T16:04:18.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.775+0000 7ff8367fc700 1 -- 172.21.15.60:0/1260183359 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2069091482 0 0) 0x7ff8380ad840 con 0x7ff83814af80 2022-06-29T16:04:18.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.775+0000 7ff8367fc700 1 -- 172.21.15.60:0/1260183359 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff83805d450 con 0x7ff83814af80 2022-06-29T16:04:18.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.775+0000 7ff8367fc700 1 -- 172.21.15.60:0/1260183359 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1994709084 0 0) 0x7ff818003260 con 0x7ff8380b2130 2022-06-29T16:04:18.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.775+0000 7ff8377fe700 1 -- 172.21.15.60:0/1260183359 >> v1:172.21.15.92:6789/0 conn(0x7ff83814af80 legacy=0x7ff8380ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:18.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.775+0000 7ff8377fe700 1 -- 172.21.15.60:0/1260183359 >> v1:172.21.15.60:6789/0 conn(0x7ff8380b2130 legacy=0x7ff8380aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:18.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.775+0000 7ff8377fe700 1 -- 172.21.15.60:0/1260183359 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8380b25d0 con 0x7ff83805d070 2022-06-29T16:04:18.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.775+0000 7ff8377fe700 1 --2- 172.21.15.60:0/1260183359 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff83805d070 0x7ff83814aa90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7ff830001490 tx=0x7ff820006df0).ready entity=mon.2 client_cookie=7d166fdba3bbb226 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:18.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.775+0000 7ff8367fc700 1 -- 172.21.15.60:0/1260183359 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff830001a00 con 0x7ff83805d070 2022-06-29T16:04:18.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.775+0000 7ff8367fc700 1 -- 172.21.15.60:0/1260183359 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff820000e90 con 0x7ff83805d070 2022-06-29T16:04:18.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.775+0000 7ff8367fc700 1 -- 172.21.15.60:0/1260183359 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff820012460 con 0x7ff83805d070 2022-06-29T16:04:18.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.779+0000 7ff83cd06700 1 -- 172.21.15.60:0/1260183359 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff83805d070 msgr2=0x7ff83814aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:18.797 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.779+0000 7ff83cd06700 1 --2- 172.21.15.60:0/1260183359 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff83805d070 0x7ff83814aa90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7ff830001490 tx=0x7ff820006df0).stop 2022-06-29T16:04:18.797 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.779+0000 7ff83cd06700 1 -- 172.21.15.60:0/1260183359 shutdown_connections 2022-06-29T16:04:18.798 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.779+0000 7ff83cd06700 1 --2- 172.21.15.60:0/1260183359 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff83805d070 0x7ff83814aa90 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:18.798 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.779+0000 7ff83cd06700 1 -- 172.21.15.60:0/1260183359 >> 172.21.15.60:0/1260183359 conn(0x7ff8380b5890 msgr2=0x7ff8380b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:18.799 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.779+0000 7ff83cd06700 1 -- 172.21.15.60:0/1260183359 shutdown_connections 2022-06-29T16:04:18.799 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.779+0000 7ff83cd06700 1 -- 172.21.15.60:0/1260183359 wait complete. 2022-06-29T16:04:18.800 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.779+0000 7ff83cd06700 1 Processor -- start 2022-06-29T16:04:18.800 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.779+0000 7ff83cd06700 1 -- start start 2022-06-29T16:04:18.801 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.779+0000 7ff83cd06700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff83805d070 0x7ff8380b4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:18.801 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.779+0000 7ff83cd06700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8380b2130 0x7ff838159e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:18.802 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.779+0000 7ff83cd06700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff83814af80 0x7ff83815a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:18.802 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.779+0000 7ff83cd06700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff838064670 con 0x7ff8380b2130 2022-06-29T16:04:18.803 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.779+0000 7ff83cd06700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff838066810 con 0x7ff83805d070 2022-06-29T16:04:18.803 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.779+0000 7ff83cd06700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff838066ab0 con 0x7ff83814af80 2022-06-29T16:04:18.803 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.779+0000 7ff836ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8380b2130 0x7ff838159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:18.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.779+0000 7ff836ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8380b2130 0x7ff838159e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49552/0 (socket says 172.21.15.60:49552) 2022-06-29T16:04:18.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.779+0000 7ff836ffd700 1 -- 172.21.15.60:0/957693972 learned_addr learned my addr 172.21.15.60:0/957693972 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:18.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.779+0000 7ff8377fe700 1 --2- 172.21.15.60:0/957693972 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff83805d070 0x7ff8380b4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:18.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.779+0000 7ff837fff700 1 --2- 172.21.15.60:0/957693972 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff83814af80 0x7ff83815a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:18.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.783+0000 7ff836ffd700 1 -- 172.21.15.60:0/957693972 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff83814af80 msgr2=0x7ff83815a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:18.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.783+0000 7ff836ffd700 1 --2- 172.21.15.60:0/957693972 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff83814af80 0x7ff83815a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:18.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.783+0000 7ff836ffd700 1 -- 172.21.15.60:0/957693972 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff83805d070 msgr2=0x7ff8380b4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:18.807 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.783+0000 7ff836ffd700 1 --2- 172.21.15.60:0/957693972 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff83805d070 0x7ff8380b4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:18.807 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.783+0000 7ff836ffd700 1 -- 172.21.15.60:0/957693972 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff82000b180 con 0x7ff8380b2130 2022-06-29T16:04:18.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.783+0000 7ff836ffd700 1 --2- 172.21.15.60:0/957693972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8380b2130 0x7ff838159e80 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0x7ff830001be0 tx=0x7ff8300021a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:18.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.783+0000 7ff834ff9700 1 -- 172.21.15.60:0/957693972 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff830004ab0 con 0x7ff8380b2130 2022-06-29T16:04:18.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.783+0000 7ff834ff9700 1 -- 172.21.15.60:0/957693972 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff830004c10 con 0x7ff8380b2130 2022-06-29T16:04:18.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.783+0000 7ff83cd06700 1 -- 172.21.15.60:0/957693972 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff838066d50 con 0x7ff8380b2130 2022-06-29T16:04:18.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.783+0000 7ff834ff9700 1 -- 172.21.15.60:0/957693972 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff830007170 con 0x7ff8380b2130 2022-06-29T16:04:18.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.783+0000 7ff83cd06700 1 -- 172.21.15.60:0/957693972 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8380b4780 con 0x7ff8380b2130 2022-06-29T16:04:18.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.783+0000 7ff83cd06700 1 -- 172.21.15.60:0/957693972 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff83814a350 con 0x7ff8380b2130 2022-06-29T16:04:18.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.783+0000 7ff834ff9700 1 -- 172.21.15.60:0/957693972 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff83001b070 con 0x7ff8380b2130 2022-06-29T16:04:18.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.783+0000 7ff834ff9700 1 --2- 172.21.15.60:0/957693972 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff81c0539e0 0x7ff81c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:18.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.783+0000 7ff834ff9700 1 -- 172.21.15.60:0/957693972 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff83006b760 con 0x7ff8380b2130 2022-06-29T16:04:18.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.787+0000 7ff8377fe700 1 --2- 172.21.15.60:0/957693972 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff81c0539e0 0x7ff81c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:18.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.787+0000 7ff8377fe700 1 --2- 172.21.15.60:0/957693972 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff81c0539e0 0x7ff81c055ea0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7ff82001a000 tx=0x7ff820014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:18.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.787+0000 7ff834ff9700 1 -- 172.21.15.60:0/957693972 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff830042680 con 0x7ff8380b2130 2022-06-29T16:04:18.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.955+0000 7ff83cd06700 1 -- 172.21.15.60:0/957693972 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff838066ee0 con 0x7ff8380b2130 2022-06-29T16:04:18.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.955+0000 7ff834ff9700 1 -- 172.21.15.60:0/957693972 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2137 (secure 0 0 0) 0x7ff830045ca0 con 0x7ff8380b2130 2022-06-29T16:04:18.964 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:04:18.964 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":30,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5690,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1158969084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1158969084},{"type":"v1","addr":"172.21.15.60:6833","nonce":1158969084}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":28}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:04:08.275039+0000","modified":"2022-06-29T16:04:09.279046+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6304":{"gid":6304,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/953938586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":953938586},{"type":"v1","addr":"172.21.15.92:6835","nonce":953938586}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-06-29T16:04:18.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.959+0000 7ff83cd06700 1 -- 172.21.15.60:0/957693972 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff81c0539e0 msgr2=0x7ff81c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:18.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.959+0000 7ff83cd06700 1 --2- 172.21.15.60:0/957693972 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff81c0539e0 0x7ff81c055ea0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7ff82001a000 tx=0x7ff820014000).stop 2022-06-29T16:04:18.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.959+0000 7ff83cd06700 1 -- 172.21.15.60:0/957693972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8380b2130 msgr2=0x7ff838159e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:18.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.959+0000 7ff83cd06700 1 --2- 172.21.15.60:0/957693972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8380b2130 0x7ff838159e80 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0x7ff830001be0 tx=0x7ff8300021a0).stop 2022-06-29T16:04:18.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.959+0000 7ff83cd06700 1 -- 172.21.15.60:0/957693972 shutdown_connections 2022-06-29T16:04:18.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.959+0000 7ff83cd06700 1 --2- 172.21.15.60:0/957693972 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff81c0539e0 0x7ff81c055ea0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:18.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.959+0000 7ff83cd06700 1 --2- 172.21.15.60:0/957693972 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff83814af80 0x7ff83815a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:18.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.959+0000 7ff83cd06700 1 --2- 172.21.15.60:0/957693972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8380b2130 0x7ff838159e80 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:18.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.959+0000 7ff83cd06700 1 --2- 172.21.15.60:0/957693972 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff83805d070 0x7ff8380b4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:18.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.959+0000 7ff83cd06700 1 -- 172.21.15.60:0/957693972 >> 172.21.15.60:0/957693972 conn(0x7ff8380b5890 msgr2=0x7ff8381433f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:18.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.959+0000 7ff83cd06700 1 -- 172.21.15.60:0/957693972 shutdown_connections 2022-06-29T16:04:18.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:18.959+0000 7ff83cd06700 1 -- 172.21.15.60:0/957693972 wait complete. 2022-06-29T16:04:18.970 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 30 2022-06-29T16:04:18.982 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:04:19.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.079+0000 7f99fecc6700 1 Processor -- start 2022-06-29T16:04:19.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.079+0000 7f99fecc6700 1 -- start start 2022-06-29T16:04:19.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.079+0000 7f99fecc6700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f99f80aec30 0x7f99f8140520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:19.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.079+0000 7f99fecc6700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99f805fee0 con 0x7f99f8067b30 2022-06-29T16:04:19.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.079+0000 7f99fecc6700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99f80b0df0 con 0x7f99f805fb00 2022-06-29T16:04:19.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.079+0000 7f99fecc6700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99f80af580 con 0x7f99f80aec30 2022-06-29T16:04:19.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.079+0000 7f99fd4c3700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f99f8067b30 0x7f99f8143fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57646/0 (socket says 172.21.15.60:57646) 2022-06-29T16:04:19.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.079+0000 7f99fd4c3700 1 -- 172.21.15.60:0/4199969358 learned_addr learned my addr 172.21.15.60:0/4199969358 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:04:19.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.079+0000 7f99fe4c5700 1 --2- 172.21.15.60:0/4199969358 >> v2:172.21.15.92:3300/0 conn(0x7f99f80aec30 0x7f99f8140520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:19.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fccc2700 1 -- 172.21.15.60:0/4199969358 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2115097052 0 0) 0x7f99f805fee0 con 0x7f99f8067b30 2022-06-29T16:04:19.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fccc2700 1 -- 172.21.15.60:0/4199969358 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99e0003260 con 0x7f99f8067b30 2022-06-29T16:04:19.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fccc2700 1 -- 172.21.15.60:0/4199969358 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 401459123 0 0) 0x7f99f80b0df0 con 0x7f99f805fb00 2022-06-29T16:04:19.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fccc2700 1 -- 172.21.15.60:0/4199969358 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99f805fee0 con 0x7f99f805fb00 2022-06-29T16:04:19.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fccc2700 1 -- 172.21.15.60:0/4199969358 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3553741631 0 0) 0x7f99e0003260 con 0x7f99f8067b30 2022-06-29T16:04:19.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fccc2700 1 -- 172.21.15.60:0/4199969358 >> v2:172.21.15.92:3300/0 conn(0x7f99f80aec30 msgr2=0x7f99f8140520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:19.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fccc2700 1 --2- 172.21.15.60:0/4199969358 >> v2:172.21.15.92:3300/0 conn(0x7f99f80aec30 0x7f99f8140520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:19.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fccc2700 1 -- 172.21.15.60:0/4199969358 >> v1:172.21.15.92:6789/0 conn(0x7f99f805fb00 legacy=0x7f99f813fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:19.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fccc2700 1 -- 172.21.15.60:0/4199969358 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f80af820 con 0x7f99f8067b30 2022-06-29T16:04:19.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fccc2700 1 -- 172.21.15.60:0/4199969358 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f99e8001660 con 0x7f99f8067b30 2022-06-29T16:04:19.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fccc2700 1 -- 172.21.15.60:0/4199969358 >> v1:172.21.15.60:6789/0 conn(0x7f99f8067b30 legacy=0x7f99f8143fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:19.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fccc2700 1 --2- 172.21.15.60:0/4199969358 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99e00038e0 0x7f99e0003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:19.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fccc2700 1 --2- 172.21.15.60:0/4199969358 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99e0004500 0x7f99e0004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:19.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fccc2700 1 --2- 172.21.15.60:0/4199969358 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f99e0004e70 0x7f99e000b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:19.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fccc2700 1 -- 172.21.15.60:0/4199969358 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f80af580 con 0x7f99e00038e0 2022-06-29T16:04:19.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fccc2700 1 -- 172.21.15.60:0/4199969358 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e000bcf0 con 0x7f99e0004500 2022-06-29T16:04:19.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fccc2700 1 -- 172.21.15.60:0/4199969358 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99e000beb0 con 0x7f99e0004e70 2022-06-29T16:04:19.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fe4c5700 1 --2- 172.21.15.60:0/4199969358 >> v2:172.21.15.92:3300/0 conn(0x7f99f80aec30 0x7f99f8140520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:04:19.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fdcc4700 1 --2- 172.21.15.60:0/4199969358 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99e00038e0 0x7f99e0003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:19.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fd4c3700 1 --2- 172.21.15.60:0/4199969358 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99e0004500 0x7f99e0004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:19.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fe4c5700 1 --2- 172.21.15.60:0/4199969358 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f99e0004e70 0x7f99e000b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:19.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fdcc4700 1 -- 172.21.15.60:0/4199969358 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f99e0004e70 msgr2=0x7f99e000b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:19.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fdcc4700 1 --2- 172.21.15.60:0/4199969358 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f99e0004e70 0x7f99e000b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:19.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fdcc4700 1 -- 172.21.15.60:0/4199969358 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99e0004500 msgr2=0x7f99e0004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:04:19.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fdcc4700 1 --2- 172.21.15.60:0/4199969358 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99e0004500 0x7f99e0004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:19.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fdcc4700 1 -- 172.21.15.60:0/4199969358 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e0003260 con 0x7f99e00038e0 2022-06-29T16:04:19.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fe4c5700 1 -- 172.21.15.60:0/4199969358 reap_dead start 2022-06-29T16:04:19.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fdcc4700 1 --2- 172.21.15.60:0/4199969358 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99e00038e0 0x7f99e0003ce0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7f99f4001110 tx=0x7f99f40079e0).ready entity=mon.0 client_cookie=65c892f50f2f8a78 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:19.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fccc2700 1 -- 172.21.15.60:0/4199969358 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99f4007cf0 con 0x7f99e00038e0 2022-06-29T16:04:19.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fccc2700 1 -- 172.21.15.60:0/4199969358 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99f4007030 con 0x7f99e00038e0 2022-06-29T16:04:19.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fccc2700 1 -- 172.21.15.60:0/4199969358 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99f4007390 con 0x7f99e00038e0 2022-06-29T16:04:19.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fecc6700 1 -- 172.21.15.60:0/4199969358 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99e00038e0 msgr2=0x7f99e0003ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:19.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fecc6700 1 --2- 172.21.15.60:0/4199969358 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99e00038e0 0x7f99e0003ce0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7f99f4001110 tx=0x7f99f40079e0).stop 2022-06-29T16:04:19.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fecc6700 1 -- 172.21.15.60:0/4199969358 shutdown_connections 2022-06-29T16:04:19.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fecc6700 1 --2- 172.21.15.60:0/4199969358 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99e00038e0 0x7f99e0003ce0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:19.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fecc6700 1 -- 172.21.15.60:0/4199969358 >> 172.21.15.60:0/4199969358 conn(0x7f99f8063a60 msgr2=0x7f99f805f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:19.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fecc6700 1 -- 172.21.15.60:0/4199969358 shutdown_connections 2022-06-29T16:04:19.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.083+0000 7f99fecc6700 1 -- 172.21.15.60:0/4199969358 wait complete. 2022-06-29T16:04:19.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.087+0000 7f99fecc6700 1 Processor -- start 2022-06-29T16:04:19.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.087+0000 7f99fecc6700 1 -- start start 2022-06-29T16:04:19.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.087+0000 7f99fecc6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f99e00038e0 0x7f99f813a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:19.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.087+0000 7f99fecc6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99f80ad570 0x7f99f80acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:19.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.087+0000 7f99fecc6700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99f8065410 0x7f99f80658a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:19.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.087+0000 7f99fecc6700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f80646c0 con 0x7f99f80ad570 2022-06-29T16:04:19.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.087+0000 7f99fecc6700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f8065d90 con 0x7f99f8065410 2022-06-29T16:04:19.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.087+0000 7f99fecc6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f80adc10 con 0x7f99e00038e0 2022-06-29T16:04:19.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.087+0000 7f99fd4c3700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99f80ad570 0x7f99f80acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:19.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.087+0000 7f99fd4c3700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99f80ad570 0x7f99f80acb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49572/0 (socket says 172.21.15.60:49572) 2022-06-29T16:04:19.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.087+0000 7f99fd4c3700 1 -- 172.21.15.60:0/1870826983 learned_addr learned my addr 172.21.15.60:0/1870826983 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:04:19.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.087+0000 7f99fdcc4700 1 --2- 172.21.15.60:0/1870826983 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f99e00038e0 0x7f99f813a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:19.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.087+0000 7f99fe4c5700 1 --2- 172.21.15.60:0/1870826983 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99f8065410 0x7f99f80658a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:19.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.087+0000 7f99fd4c3700 1 -- 172.21.15.60:0/1870826983 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f99e00038e0 msgr2=0x7f99f813a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:19.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.087+0000 7f99fd4c3700 1 --2- 172.21.15.60:0/1870826983 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f99e00038e0 0x7f99f813a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:19.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.087+0000 7f99fd4c3700 1 -- 172.21.15.60:0/1870826983 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99f8065410 msgr2=0x7f99f80658a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:19.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.087+0000 7f99fd4c3700 1 --2- 172.21.15.60:0/1870826983 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99f8065410 0x7f99f80658a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:19.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.087+0000 7f99fd4c3700 1 -- 172.21.15.60:0/1870826983 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f80adeb0 con 0x7f99f80ad570 2022-06-29T16:04:19.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.087+0000 7f99fd4c3700 1 --2- 172.21.15.60:0/1870826983 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99f80ad570 0x7f99f80acb00 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0x7f99e800b000 tx=0x7f99f80afcc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:19.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.087+0000 7f99e6ffd700 1 -- 172.21.15.60:0/1870826983 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99e8010640 con 0x7f99f80ad570 2022-06-29T16:04:19.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.087+0000 7f99e6ffd700 1 -- 172.21.15.60:0/1870826983 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99e80107a0 con 0x7f99f80ad570 2022-06-29T16:04:19.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.087+0000 7f99fecc6700 1 -- 172.21.15.60:0/1870826983 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99f80666c0 con 0x7f99f80ad570 2022-06-29T16:04:19.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.087+0000 7f99e6ffd700 1 -- 172.21.15.60:0/1870826983 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99e8010b00 con 0x7f99f80ad570 2022-06-29T16:04:19.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.087+0000 7f99fecc6700 1 -- 172.21.15.60:0/1870826983 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f8066c20 con 0x7f99f80ad570 2022-06-29T16:04:19.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.091+0000 7f99fecc6700 1 -- 172.21.15.60:0/1870826983 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99f8066850 con 0x7f99f80ad570 2022-06-29T16:04:19.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.091+0000 7f99e6ffd700 1 -- 172.21.15.60:0/1870826983 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f99e801a020 con 0x7f99f80ad570 2022-06-29T16:04:19.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.091+0000 7f99e6ffd700 1 --2- 172.21.15.60:0/1870826983 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f99d8053a70 0x7f99d8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:04:19.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.091+0000 7f99e6ffd700 1 -- 172.21.15.60:0/1870826983 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f99e806ba20 con 0x7f99f80ad570 2022-06-29T16:04:19.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.091+0000 7f99fdcc4700 1 --2- 172.21.15.60:0/1870826983 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f99d8053a70 0x7f99d8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:04:19.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.091+0000 7f99fdcc4700 1 --2- 172.21.15.60:0/1870826983 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f99d8053a70 0x7f99d8055f30 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f99f401b000 tx=0x7f99f4015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:04:19.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.095+0000 7f99e6ffd700 1 -- 172.21.15.60:0/1870826983 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f99e8042940 con 0x7f99f80ad570 2022-06-29T16:04:19.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.259+0000 7f99fecc6700 1 -- 172.21.15.60:0/1870826983 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99f80573d0 con 0x7f99f80ad570 2022-06-29T16:04:19.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.259+0000 7f99e6ffd700 1 -- 172.21.15.60:0/1870826983 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f99e8045f60 con 0x7f99f80ad570 2022-06-29T16:04:19.265 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:04:19.266 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":30,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5690,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1158969084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1158969084},{"type":"v1","addr":"172.21.15.60:6833","nonce":1158969084}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":28}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:04:08.275039+0000","modified":"2022-06-29T16:04:09.279046+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6304":{"gid":6304,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/953938586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":953938586},{"type":"v1","addr":"172.21.15.92:6835","nonce":953938586}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-06-29T16:04:19.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.259+0000 7f99fecc6700 1 -- 172.21.15.60:0/1870826983 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f99d8053a70 msgr2=0x7f99d8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:19.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.259+0000 7f99fecc6700 1 --2- 172.21.15.60:0/1870826983 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f99d8053a70 0x7f99d8055f30 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f99f401b000 tx=0x7f99f4015000).stop 2022-06-29T16:04:19.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.259+0000 7f99fecc6700 1 -- 172.21.15.60:0/1870826983 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99f80ad570 msgr2=0x7f99f80acb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:04:19.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.259+0000 7f99fecc6700 1 --2- 172.21.15.60:0/1870826983 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99f80ad570 0x7f99f80acb00 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0x7f99e800b000 tx=0x7f99f80afcc0).stop 2022-06-29T16:04:19.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.259+0000 7f99fecc6700 1 -- 172.21.15.60:0/1870826983 shutdown_connections 2022-06-29T16:04:19.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.259+0000 7f99fecc6700 1 --2- 172.21.15.60:0/1870826983 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f99d8053a70 0x7f99d8055f30 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:19.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.259+0000 7f99fecc6700 1 --2- 172.21.15.60:0/1870826983 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99f8065410 0x7f99f80658a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:19.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.259+0000 7f99fecc6700 1 --2- 172.21.15.60:0/1870826983 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99f80ad570 0x7f99f80acb00 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:19.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.259+0000 7f99fecc6700 1 --2- 172.21.15.60:0/1870826983 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f99e00038e0 0x7f99f813a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:04:19.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.259+0000 7f99fecc6700 1 -- 172.21.15.60:0/1870826983 >> 172.21.15.60:0/1870826983 conn(0x7f99f8063a60 msgr2=0x7f99f8141090 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:04:19.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.259+0000 7f99fecc6700 1 -- 172.21.15.60:0/1870826983 shutdown_connections 2022-06-29T16:04:19.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:04:19.259+0000 7f99fecc6700 1 -- 172.21.15.60:0/1870826983 wait complete. 2022-06-29T16:04:19.271 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 30 2022-06-29T16:04:19.279 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json config set mds_cap_revoke_eviction_timeout 30.0 2022-06-29T16:04:19.394 INFO:teuthology.orchestra.run.smithi092.stdout:{"success":"mds_cap_revoke_eviction_timeout = '30.000000' "} 2022-06-29T16:04:19.395 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_cap_revoke_eviction_timeout = '30.000000' " } 2022-06-29T16:04:19.395 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:04:19.396 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:04:19.397 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:04:19.406 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T16:04:19.407 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T16:04:19.408 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-06-29T16:04:19.409 DEBUG:teuthology.orchestra.run.smithi121:> import time 2022-06-29T16:04:19.409 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:04:19.410 DEBUG:teuthology.orchestra.run.smithi121:> with open("/home/ubuntu/cephtest/mnt.0/background_file", '"'"'w'"'"') as f: 2022-06-29T16:04:19.410 DEBUG:teuthology.orchestra.run.smithi121:> f.write('"'"'content'"'"') 2022-06-29T16:04:19.411 DEBUG:teuthology.orchestra.run.smithi121:> f.flush() 2022-06-29T16:04:19.411 DEBUG:teuthology.orchestra.run.smithi121:> f.write('"'"'content2'"'"') 2022-06-29T16:04:19.411 DEBUG:teuthology.orchestra.run.smithi121:> while True: 2022-06-29T16:04:19.412 DEBUG:teuthology.orchestra.run.smithi121:> time.sleep(1) 2022-06-29T16:04:19.412 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-06-29T16:04:19.415 DEBUG:teuthology.orchestra.run.smithi121:> sudo ls /home/ubuntu/cephtest/mnt.0/background_file 2022-06-29T16:04:19.433 INFO:teuthology.orchestra.run.smithi121.stderr:ls: cannot access '/home/ubuntu/cephtest/mnt.0/background_file': No such file or directory 2022-06-29T16:04:19.434 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-06-29T16:04:20.436 DEBUG:teuthology.orchestra.run.smithi121:> sudo ls /home/ubuntu/cephtest/mnt.0/background_file 2022-06-29T16:04:20.453 INFO:teuthology.orchestra.run.smithi121.stdout:/home/ubuntu/cephtest/mnt.0/background_file 2022-06-29T16:04:20.454 DEBUG:tasks.cephfs.mount:File background_file became visible from 0 after 1s 2022-06-29T16:04:20.455 DEBUG:teuthology.orchestra.run.smithi156:> sudo ls /home/ubuntu/cephtest/mnt.1/background_file 2022-06-29T16:04:20.491 INFO:teuthology.orchestra.run.smithi156.stdout:/home/ubuntu/cephtest/mnt.1/background_file 2022-06-29T16:04:20.493 DEBUG:tasks.cephfs.mount:File background_file became visible from 1 after 0s 2022-06-29T16:04:20.493 INFO:tasks.cephfs.fuse_mount:Killing ceph-fuse connection on ubuntu@smithi121.front.sepia.ceph.com... 2022-06-29T16:04:20.508 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:04:20.509 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:04:20.509 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:04:20.518 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2022-06-29T16:04:20.519 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-06-29T16:04:20.519 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-06-29T16:04:20.520 DEBUG:teuthology.orchestra.run.smithi156:> import os 2022-06-29T16:04:20.520 DEBUG:teuthology.orchestra.run.smithi156:> import time 2022-06-29T16:04:20.521 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:04:20.521 DEBUG:teuthology.orchestra.run.smithi156:> fd = os.open("/home/ubuntu/cephtest/mnt.1/background_file", os.O_RDWR | os.O_CREAT, 0o644) 2022-06-29T16:04:20.522 DEBUG:teuthology.orchestra.run.smithi156:> try: 2022-06-29T16:04:20.523 DEBUG:teuthology.orchestra.run.smithi156:> while True: 2022-06-29T16:04:20.524 DEBUG:teuthology.orchestra.run.smithi156:> os.write(fd, b'"'"'content'"'"') 2022-06-29T16:04:20.524 DEBUG:teuthology.orchestra.run.smithi156:> time.sleep(1) 2022-06-29T16:04:20.525 DEBUG:teuthology.orchestra.run.smithi156:> if not False: 2022-06-29T16:04:20.526 DEBUG:teuthology.orchestra.run.smithi156:> break 2022-06-29T16:04:20.527 DEBUG:teuthology.orchestra.run.smithi156:> except IOError as e: 2022-06-29T16:04:20.527 DEBUG:teuthology.orchestra.run.smithi156:> pass 2022-06-29T16:04:20.528 DEBUG:teuthology.orchestra.run.smithi156:> os.close(fd) 2022-06-29T16:04:20.529 DEBUG:teuthology.orchestra.run.smithi156:> ' 2022-06-29T16:04:28.833 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:04:28.837 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:04:28.840 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:04:28.843 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:04:28.846 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:04:58.897 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:04:58.900 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:04:58.903 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:04:58.905 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:04:58.908 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:05:00.088 INFO:tasks.cephfs.test_misc:cap_waiter waited 39.52634620666504s 2022-06-29T16:05:00.090 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist ls --format=json 2022-06-29T16:05:00.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.190+0000 7f4c55425700 1 Processor -- start 2022-06-29T16:05:00.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.190+0000 7f4c55425700 1 -- start start 2022-06-29T16:05:00.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.194+0000 7f4c55425700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4c500b2800 0x7f4c500b2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:00.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.194+0000 7f4c55425700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c50143330 con 0x7f4c50142b90 2022-06-29T16:05:00.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.194+0000 7f4c55425700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c500b1eb0 con 0x7f4c500b14c0 2022-06-29T16:05:00.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.194+0000 7f4c55425700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c500ae430 con 0x7f4c500b2800 2022-06-29T16:05:00.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.194+0000 7f4c54c24700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4c50142b90 0x7f4c5013eeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57668/0 (socket says 172.21.15.60:57668) 2022-06-29T16:05:00.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.194+0000 7f4c54c24700 1 -- 172.21.15.60:0/2083076776 learned_addr learned my addr 172.21.15.60:0/2083076776 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:00.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.194+0000 7f4c4ffff700 1 --2- 172.21.15.60:0/2083076776 >> v2:172.21.15.92:3300/0 conn(0x7f4c500b2800 0x7f4c500b2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:00.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.194+0000 7f4c4effd700 1 -- 172.21.15.60:0/2083076776 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 65546745 0 0) 0x7f4c50143330 con 0x7f4c50142b90 2022-06-29T16:05:00.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.194+0000 7f4c4effd700 1 -- 172.21.15.60:0/2083076776 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c3c003260 con 0x7f4c50142b90 2022-06-29T16:05:00.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.194+0000 7f4c4effd700 1 -- 172.21.15.60:0/2083076776 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3367865788 0 0) 0x7f4c500b1eb0 con 0x7f4c500b14c0 2022-06-29T16:05:00.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.194+0000 7f4c4effd700 1 -- 172.21.15.60:0/2083076776 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c50143330 con 0x7f4c500b14c0 2022-06-29T16:05:00.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.194+0000 7f4c4ffff700 1 -- 172.21.15.60:0/2083076776 >> v1:172.21.15.92:6789/0 conn(0x7f4c500b14c0 legacy=0x7f4c50142480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:00.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.194+0000 7f4c4ffff700 1 -- 172.21.15.60:0/2083076776 >> v1:172.21.15.60:6789/0 conn(0x7f4c50142b90 legacy=0x7f4c5013eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:00.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.194+0000 7f4c4ffff700 1 -- 172.21.15.60:0/2083076776 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c500af540 con 0x7f4c500b2800 2022-06-29T16:05:00.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.194+0000 7f4c4ffff700 1 --2- 172.21.15.60:0/2083076776 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c500b2800 0x7f4c500b2be0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f4c480032f0 tx=0x7f4c480091d0).ready entity=mon.1 client_cookie=1d82e42302943b0a server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:00.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.194+0000 7f4c4effd700 1 -- 172.21.15.60:0/2083076776 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c4800f8a0 con 0x7f4c500b2800 2022-06-29T16:05:00.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.194+0000 7f4c4effd700 1 -- 172.21.15.60:0/2083076776 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4800fa00 con 0x7f4c500b2800 2022-06-29T16:05:00.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.194+0000 7f4c4effd700 1 -- 172.21.15.60:0/2083076776 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c4800fda0 con 0x7f4c500b2800 2022-06-29T16:05:00.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.194+0000 7f4c55425700 1 -- 172.21.15.60:0/2083076776 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c500b2800 msgr2=0x7f4c500b2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:00.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.194+0000 7f4c55425700 1 --2- 172.21.15.60:0/2083076776 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c500b2800 0x7f4c500b2be0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f4c480032f0 tx=0x7f4c480091d0).stop 2022-06-29T16:05:00.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.194+0000 7f4c55425700 1 -- 172.21.15.60:0/2083076776 shutdown_connections 2022-06-29T16:05:00.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.194+0000 7f4c55425700 1 --2- 172.21.15.60:0/2083076776 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c500b2800 0x7f4c500b2be0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:00.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.194+0000 7f4c55425700 1 -- 172.21.15.60:0/2083076776 >> 172.21.15.60:0/2083076776 conn(0x7f4c500b6120 msgr2=0x7f4c500b20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:00.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.194+0000 7f4c55425700 1 -- 172.21.15.60:0/2083076776 shutdown_connections 2022-06-29T16:05:00.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.194+0000 7f4c55425700 1 -- 172.21.15.60:0/2083076776 wait complete. 2022-06-29T16:05:00.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.198+0000 7f4c55425700 1 Processor -- start 2022-06-29T16:05:00.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.198+0000 7f4c55425700 1 -- start start 2022-06-29T16:05:00.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.198+0000 7f4c55425700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c500b14c0 0x7f4c500a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:00.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.198+0000 7f4c55425700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c500b2800 0x7f4c50065b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:00.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.198+0000 7f4c55425700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4c50142b90 0x7f4c501550d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:00.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.198+0000 7f4c55425700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c500a3bf0 con 0x7f4c500b2800 2022-06-29T16:05:00.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.198+0000 7f4c55425700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c50065820 con 0x7f4c500b14c0 2022-06-29T16:05:00.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.198+0000 7f4c55425700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c50152a60 con 0x7f4c50142b90 2022-06-29T16:05:00.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.198+0000 7f4c4f7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c500b2800 0x7f4c50065b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:00.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.198+0000 7f4c4ffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c500b14c0 0x7f4c500a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:00.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.198+0000 7f4c4f7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c500b2800 0x7f4c50065b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49586/0 (socket says 172.21.15.60:49586) 2022-06-29T16:05:00.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.198+0000 7f4c4f7fe700 1 -- 172.21.15.60:0/1747202277 learned_addr learned my addr 172.21.15.60:0/1747202277 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:00.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.198+0000 7f4c54c24700 1 --2- 172.21.15.60:0/1747202277 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4c50142b90 0x7f4c501550d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:00.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.198+0000 7f4c4f7fe700 1 -- 172.21.15.60:0/1747202277 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4c50142b90 msgr2=0x7f4c501550d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:00.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.198+0000 7f4c4f7fe700 1 --2- 172.21.15.60:0/1747202277 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4c50142b90 0x7f4c501550d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:00.216 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.198+0000 7f4c4f7fe700 1 -- 172.21.15.60:0/1747202277 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c500b14c0 msgr2=0x7f4c500a49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:00.216 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.198+0000 7f4c4f7fe700 1 --2- 172.21.15.60:0/1747202277 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c500b14c0 0x7f4c500a49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:00.216 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.198+0000 7f4c4f7fe700 1 -- 172.21.15.60:0/1747202277 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c48009470 con 0x7f4c500b2800 2022-06-29T16:05:00.216 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.198+0000 7f4c4ffff700 1 --2- 172.21.15.60:0/1747202277 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c500b14c0 0x7f4c500a49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:05:00.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.198+0000 7f4c4f7fe700 1 --2- 172.21.15.60:0/1747202277 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c500b2800 0x7f4c50065b60 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0x7f4c44001110 tx=0x7f4c44007a90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:00.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.198+0000 7f4c4d7fa700 1 -- 172.21.15.60:0/1747202277 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c44012400 con 0x7f4c500b2800 2022-06-29T16:05:00.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.198+0000 7f4c4d7fa700 1 -- 172.21.15.60:0/1747202277 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c44012560 con 0x7f4c500b2800 2022-06-29T16:05:00.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.198+0000 7f4c55425700 1 -- 172.21.15.60:0/1747202277 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c500a5040 con 0x7f4c500b2800 2022-06-29T16:05:00.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.198+0000 7f4c4d7fa700 1 -- 172.21.15.60:0/1747202277 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c440128c0 con 0x7f4c500b2800 2022-06-29T16:05:00.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.198+0000 7f4c55425700 1 -- 172.21.15.60:0/1747202277 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c50066050 con 0x7f4c500b2800 2022-06-29T16:05:00.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.202+0000 7f4c55425700 1 -- 172.21.15.60:0/1747202277 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c501528b0 con 0x7f4c500b2800 2022-06-29T16:05:00.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.202+0000 7f4c4d7fa700 1 -- 172.21.15.60:0/1747202277 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4c44019030 con 0x7f4c500b2800 2022-06-29T16:05:00.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.202+0000 7f4c4d7fa700 1 --2- 172.21.15.60:0/1747202277 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4c300539e0 0x7f4c30055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:00.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.202+0000 7f4c4d7fa700 1 -- 172.21.15.60:0/1747202277 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6116+0+0 (secure 0 0 0) 0x7f4c4406b370 con 0x7f4c500b2800 2022-06-29T16:05:00.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.202+0000 7f4c4ffff700 1 --2- 172.21.15.60:0/1747202277 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4c300539e0 0x7f4c30055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:00.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.202+0000 7f4c4ffff700 1 --2- 172.21.15.60:0/1747202277 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4c300539e0 0x7f4c30055ea0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f4c4801b000 tx=0x7f4c48018000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:00.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.210+0000 7f4c4d7fa700 1 -- 172.21.15.60:0/1747202277 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4c44042150 con 0x7f4c500b2800 2022-06-29T16:05:00.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.346+0000 7f4c55425700 1 -- 172.21.15.60:0/1747202277 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blacklist ls", "format": "json"} v 0) v1 -- 0x7f4c500a54e0 con 0x7f4c500b2800 2022-06-29T16:05:00.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.346+0000 7f4c4d7fa700 1 -- 172.21.15.60:0/1747202277 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist ls", "format": "json"}]=0 listed 1 entries v72) v1 ==== 98+0+81 (secure 0 0 0) 0x7f4c44045770 con 0x7f4c500b2800 2022-06-29T16:05:00.356 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:00.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.350+0000 7f4c55425700 1 -- 172.21.15.60:0/1747202277 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4c300539e0 msgr2=0x7f4c30055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:00.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.350+0000 7f4c55425700 1 --2- 172.21.15.60:0/1747202277 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4c300539e0 0x7f4c30055ea0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f4c4801b000 tx=0x7f4c48018000).stop 2022-06-29T16:05:00.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.350+0000 7f4c55425700 1 -- 172.21.15.60:0/1747202277 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c500b2800 msgr2=0x7f4c50065b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:00.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.350+0000 7f4c55425700 1 --2- 172.21.15.60:0/1747202277 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c500b2800 0x7f4c50065b60 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0x7f4c44001110 tx=0x7f4c44007a90).stop 2022-06-29T16:05:00.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.350+0000 7f4c55425700 1 -- 172.21.15.60:0/1747202277 shutdown_connections 2022-06-29T16:05:00.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.350+0000 7f4c55425700 1 --2- 172.21.15.60:0/1747202277 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4c300539e0 0x7f4c30055ea0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:00.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.350+0000 7f4c55425700 1 --2- 172.21.15.60:0/1747202277 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4c50142b90 0x7f4c501550d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:00.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.350+0000 7f4c55425700 1 --2- 172.21.15.60:0/1747202277 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c500b2800 0x7f4c50065b60 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:00.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.350+0000 7f4c55425700 1 --2- 172.21.15.60:0/1747202277 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c500b14c0 0x7f4c500a49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:00.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.350+0000 7f4c55425700 1 -- 172.21.15.60:0/1747202277 >> 172.21.15.60:0/1747202277 conn(0x7f4c500b6120 msgr2=0x7f4c500b20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:00.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.350+0000 7f4c55425700 1 -- 172.21.15.60:0/1747202277 shutdown_connections 2022-06-29T16:05:00.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.350+0000 7f4c55425700 1 -- 172.21.15.60:0/1747202277 wait complete. 2022-06-29T16:05:00.360 INFO:teuthology.orchestra.run.smithi060.stderr:listed 1 entries 2022-06-29T16:05:00.375 INFO:teuthology.orchestra.run.smithi060.stdout:[{"addr":"172.21.15.121:0/2990201791","until":"2022-06-29T17:04:53.908824+0000"}] 2022-06-29T16:05:00.391 INFO:tasks.cephfs.fuse_mount:Cleaning up killed ceph-fuse connection 2022-06-29T16:05:00.391 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:00.392 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:05:00.392 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:05:00.400 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': Transport endpoint is not connected 2022-06-29T16:05:00.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:05:00.401 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.0 2022-06-29T16:05:00.402 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2022-06-29T16:05:00.402 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-06-29T16:05:00.403 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:05:00.463 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:00.464 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:05:00.464 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:05:00.470 INFO:teuthology.orchestra.run.smithi121.stdout:ext2/ext3 2022-06-29T16:05:00.471 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-06-29T16:05:00.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:05:00.472 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:05:00.478 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:05:00.587 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.578+0000 7f0b7a633700 1 Processor -- start 2022-06-29T16:05:00.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.578+0000 7f0b7a633700 1 -- start start 2022-06-29T16:05:00.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.582+0000 7f0b7a633700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b740ae7c0 0x7f0b74141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:00.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.582+0000 7f0b7a633700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0b740b46c0 0x7f0b740b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:00.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.582+0000 7f0b7a633700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b74143a40 con 0x7f0b740b4010 2022-06-29T16:05:00.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.582+0000 7f0b7a633700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b740aee60 con 0x7f0b740b46c0 2022-06-29T16:05:00.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.582+0000 7f0b7a633700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b740af100 con 0x7f0b740ae7c0 2022-06-29T16:05:00.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.582+0000 7f0b78bd0700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0b740b46c0 0x7f0b740b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:00.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.582+0000 7f0b78bd0700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0b740b46c0 0x7f0b740b4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49596/0 (socket says 172.21.15.60:49596) 2022-06-29T16:05:00.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.582+0000 7f0b78bd0700 1 -- 172.21.15.60:0/620632255 learned_addr learned my addr 172.21.15.60:0/620632255 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:00.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.582+0000 7f0b73fff700 1 --2- 172.21.15.60:0/620632255 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b740ae7c0 0x7f0b74141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:00.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.582+0000 7f0b78bd0700 1 -- 172.21.15.60:0/620632255 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b740ae7c0 msgr2=0x7f0b74141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:00.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.582+0000 7f0b78bd0700 1 --2- 172.21.15.60:0/620632255 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b740ae7c0 0x7f0b74141fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:00.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.582+0000 7f0b78bd0700 1 -- 172.21.15.60:0/620632255 >> v1:172.21.15.92:6789/0 conn(0x7f0b740b4010 legacy=0x7f0b74142490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:00.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.582+0000 7f0b78bd0700 1 -- 172.21.15.60:0/620632255 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b740aeba0 con 0x7f0b740b46c0 2022-06-29T16:05:00.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.582+0000 7f0b78bd0700 1 --2- 172.21.15.60:0/620632255 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b740b46c0 0x7f0b740b4b50 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7f0b6c0032f0 tx=0x7f0b6c009290).ready entity=mon.0 client_cookie=69baef69b88de5b1 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:00.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.582+0000 7f0b72ffd700 1 -- 172.21.15.60:0/620632255 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b6c00b8a0 con 0x7f0b740b46c0 2022-06-29T16:05:00.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.582+0000 7f0b72ffd700 1 -- 172.21.15.60:0/620632255 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b6c00ba00 con 0x7f0b740b46c0 2022-06-29T16:05:00.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.582+0000 7f0b72ffd700 1 -- 172.21.15.60:0/620632255 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b6c00bb70 con 0x7f0b740b46c0 2022-06-29T16:05:00.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.582+0000 7f0b7a633700 1 -- 172.21.15.60:0/620632255 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b740b46c0 msgr2=0x7f0b740b4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:00.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.582+0000 7f0b7a633700 1 --2- 172.21.15.60:0/620632255 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b740b46c0 0x7f0b740b4b50 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7f0b6c0032f0 tx=0x7f0b6c009290).stop 2022-06-29T16:05:00.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.582+0000 7f0b7a633700 1 -- 172.21.15.60:0/620632255 shutdown_connections 2022-06-29T16:05:00.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.582+0000 7f0b7a633700 1 --2- 172.21.15.60:0/620632255 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b740b46c0 0x7f0b740b4b50 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:00.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.582+0000 7f0b7a633700 1 --2- 172.21.15.60:0/620632255 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b740ae7c0 0x7f0b74141fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:00.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.582+0000 7f0b7a633700 1 -- 172.21.15.60:0/620632255 >> 172.21.15.60:0/620632255 conn(0x7f0b740b9340 msgr2=0x7f0b740b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:00.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.582+0000 7f0b7a633700 1 -- 172.21.15.60:0/620632255 shutdown_connections 2022-06-29T16:05:00.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.582+0000 7f0b7a633700 1 -- 172.21.15.60:0/620632255 wait complete. 2022-06-29T16:05:00.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.586+0000 7f0b7a633700 1 Processor -- start 2022-06-29T16:05:00.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.586+0000 7f0b7a633700 1 -- start start 2022-06-29T16:05:00.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.586+0000 7f0b7a633700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b740ae7c0 0x7f0b7413fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:00.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.586+0000 7f0b7a633700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b740b4010 0x7f0b74144060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:00.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.586+0000 7f0b7a633700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b740b46c0 0x7f0b74144550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:00.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.586+0000 7f0b7a633700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b74145d20 con 0x7f0b740b4010 2022-06-29T16:05:00.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.586+0000 7f0b7a633700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b740b1650 con 0x7f0b740b46c0 2022-06-29T16:05:00.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.586+0000 7f0b7a633700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b74145f50 con 0x7f0b740ae7c0 2022-06-29T16:05:00.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.590+0000 7f0b737fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b740b4010 0x7f0b74144060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:00.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.590+0000 7f0b73fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b740ae7c0 0x7f0b7413fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:00.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.590+0000 7f0b73fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b740ae7c0 0x7f0b7413fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:34460/0 (socket says 172.21.15.60:34460) 2022-06-29T16:05:00.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.590+0000 7f0b73fff700 1 -- 172.21.15.60:0/1582886698 learned_addr learned my addr 172.21.15.60:0/1582886698 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:00.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.590+0000 7f0b78bd0700 1 --2- 172.21.15.60:0/1582886698 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b740b46c0 0x7f0b74144550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:00.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.590+0000 7f0b73fff700 1 -- 172.21.15.60:0/1582886698 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b740b46c0 msgr2=0x7f0b74144550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:00.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.590+0000 7f0b73fff700 1 --2- 172.21.15.60:0/1582886698 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b740b46c0 0x7f0b74144550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:00.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.590+0000 7f0b73fff700 1 -- 172.21.15.60:0/1582886698 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b740b4010 msgr2=0x7f0b74144060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:00.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.590+0000 7f0b73fff700 1 --2- 172.21.15.60:0/1582886698 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b740b4010 0x7f0b74144060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:00.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.590+0000 7f0b73fff700 1 -- 172.21.15.60:0/1582886698 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b6c009530 con 0x7f0b740ae7c0 2022-06-29T16:05:00.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.590+0000 7f0b737fe700 1 --2- 172.21.15.60:0/1582886698 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b740b4010 0x7f0b74144060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:05:00.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.590+0000 7f0b73fff700 1 --2- 172.21.15.60:0/1582886698 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b740ae7c0 0x7f0b7413fc90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f0b740b14f0 tx=0x7f0b64006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:00.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.590+0000 7f0b717fa700 1 -- 172.21.15.60:0/1582886698 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b64010780 con 0x7f0b740ae7c0 2022-06-29T16:05:00.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.590+0000 7f0b717fa700 1 -- 172.21.15.60:0/1582886698 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b640108e0 con 0x7f0b740ae7c0 2022-06-29T16:05:00.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.590+0000 7f0b7a633700 1 -- 172.21.15.60:0/1582886698 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b74146d40 con 0x7f0b740ae7c0 2022-06-29T16:05:00.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.590+0000 7f0b717fa700 1 -- 172.21.15.60:0/1582886698 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b64010c40 con 0x7f0b740ae7c0 2022-06-29T16:05:00.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.590+0000 7f0b7a633700 1 -- 172.21.15.60:0/1582886698 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b74145920 con 0x7f0b740ae7c0 2022-06-29T16:05:00.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.590+0000 7f0b7a633700 1 -- 172.21.15.60:0/1582886698 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b7413df20 con 0x7f0b740ae7c0 2022-06-29T16:05:00.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.590+0000 7f0b717fa700 1 -- 172.21.15.60:0/1582886698 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0b6401c460 con 0x7f0b740ae7c0 2022-06-29T16:05:00.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.594+0000 7f0b717fa700 1 --2- 172.21.15.60:0/1582886698 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0b58053b00 0x7f0b58055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:00.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.594+0000 7f0b717fa700 1 -- 172.21.15.60:0/1582886698 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6116+0+0 (secure 0 0 0) 0x7f0b6406c3c0 con 0x7f0b740ae7c0 2022-06-29T16:05:00.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.594+0000 7f0b737fe700 1 --2- 172.21.15.60:0/1582886698 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0b58053b00 0x7f0b58055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:00.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.594+0000 7f0b737fe700 1 --2- 172.21.15.60:0/1582886698 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0b58053b00 0x7f0b58055fc0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f0b5c00d000 tx=0x7f0b5c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:00.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.602+0000 7f0b717fa700 1 -- 172.21.15.60:0/1582886698 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0b640431b0 con 0x7f0b740ae7c0 2022-06-29T16:05:00.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.730+0000 7f0b7a633700 1 -- 172.21.15.60:0/1582886698 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b74146ed0 con 0x7f0b740ae7c0 2022-06-29T16:05:00.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.730+0000 7f0b717fa700 1 -- 172.21.15.60:0/1582886698 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+16103 (secure 0 0 0) 0x7f0b640467d0 con 0x7f0b740ae7c0 2022-06-29T16:05:00.740 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:00.740 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":72,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:04:54.882383+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:04:06.708236+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2022-06-29T16:04:07.305396+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{"172.21.15.121:0/2990201791":"2022-06-29T17:04:53.908824+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:05:00.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.734+0000 7f0b7a633700 1 -- 172.21.15.60:0/1582886698 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0b58053b00 msgr2=0x7f0b58055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:00.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.734+0000 7f0b7a633700 1 --2- 172.21.15.60:0/1582886698 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0b58053b00 0x7f0b58055fc0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f0b5c00d000 tx=0x7f0b5c009040).stop 2022-06-29T16:05:00.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.734+0000 7f0b7a633700 1 -- 172.21.15.60:0/1582886698 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b740ae7c0 msgr2=0x7f0b7413fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:00.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.734+0000 7f0b7a633700 1 --2- 172.21.15.60:0/1582886698 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b740ae7c0 0x7f0b7413fc90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f0b740b14f0 tx=0x7f0b64006df0).stop 2022-06-29T16:05:00.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.734+0000 7f0b7a633700 1 -- 172.21.15.60:0/1582886698 shutdown_connections 2022-06-29T16:05:00.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.734+0000 7f0b7a633700 1 --2- 172.21.15.60:0/1582886698 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0b58053b00 0x7f0b58055fc0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:00.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.734+0000 7f0b7a633700 1 --2- 172.21.15.60:0/1582886698 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b740b46c0 0x7f0b74144550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:00.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.734+0000 7f0b7a633700 1 --2- 172.21.15.60:0/1582886698 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b740b4010 0x7f0b74144060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:00.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.734+0000 7f0b7a633700 1 --2- 172.21.15.60:0/1582886698 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b740ae7c0 0x7f0b7413fc90 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:00.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.734+0000 7f0b7a633700 1 -- 172.21.15.60:0/1582886698 >> 172.21.15.60:0/1582886698 conn(0x7f0b740b9340 msgr2=0x7f0b7405cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:00.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.734+0000 7f0b7a633700 1 -- 172.21.15.60:0/1582886698 shutdown_connections 2022-06-29T16:05:00.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.734+0000 7f0b7a633700 1 -- 172.21.15.60:0/1582886698 wait complete. 2022-06-29T16:05:00.760 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-29T16:04:06.708236+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2022-06-29T16:04:07.305396+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-29T16:05:00.761 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:05:00.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.866+0000 7fe6d72e9700 1 Processor -- start 2022-06-29T16:05:00.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.866+0000 7fe6d72e9700 1 -- start start 2022-06-29T16:05:00.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.866+0000 7fe6d72e9700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe6d00b22c0 0x7fe6d00ab920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:00.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.866+0000 7fe6d72e9700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe6d00b0800 0x7fe6d01466e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:00.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.866+0000 7fe6d72e9700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6d00b26a0 con 0x7fe6d00b0f90 2022-06-29T16:05:00.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.866+0000 7fe6d72e9700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6d00afe20 con 0x7fe6d00b22c0 2022-06-29T16:05:00.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.866+0000 7fe6d72e9700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6d00b00c0 con 0x7fe6d00b0800 2022-06-29T16:05:00.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.866+0000 7fe6d62e7700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe6d00b22c0 0x7fe6d00ab920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:00.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.866+0000 7fe6d62e7700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe6d00b22c0 0x7fe6d00ab920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49606/0 (socket says 172.21.15.60:49606) 2022-06-29T16:05:00.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.866+0000 7fe6d62e7700 1 -- 172.21.15.60:0/3345855513 learned_addr learned my addr 172.21.15.60:0/3345855513 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:00.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.866+0000 7fe6d6ae8700 1 --2- 172.21.15.60:0/3345855513 >> v2:172.21.15.92:3300/0 conn(0x7fe6d00b0800 0x7fe6d01466e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:00.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.866+0000 7fe6d62e7700 1 -- 172.21.15.60:0/3345855513 >> v2:172.21.15.92:3300/0 conn(0x7fe6d00b0800 msgr2=0x7fe6d01466e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:00.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.866+0000 7fe6d62e7700 1 --2- 172.21.15.60:0/3345855513 >> v2:172.21.15.92:3300/0 conn(0x7fe6d00b0800 0x7fe6d01466e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:00.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.866+0000 7fe6d62e7700 1 -- 172.21.15.60:0/3345855513 >> v1:172.21.15.92:6789/0 conn(0x7fe6d00b0f90 legacy=0x7fe6d0145fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:00.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.866+0000 7fe6d62e7700 1 -- 172.21.15.60:0/3345855513 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d00b0360 con 0x7fe6d00b22c0 2022-06-29T16:05:00.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.866+0000 7fe6d62e7700 1 --2- 172.21.15.60:0/3345855513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6d00b22c0 0x7fe6d00ab920 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7fe6cc002f70 tx=0x7fe6cc006eb0).ready entity=mon.0 client_cookie=721cea87c861209c server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:00.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.870+0000 7fe6d52e5700 1 -- 172.21.15.60:0/3345855513 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6cc00d5c0 con 0x7fe6d00b22c0 2022-06-29T16:05:00.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.870+0000 7fe6d52e5700 1 -- 172.21.15.60:0/3345855513 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6cc000e90 con 0x7fe6d00b22c0 2022-06-29T16:05:00.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.870+0000 7fe6d52e5700 1 -- 172.21.15.60:0/3345855513 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6cc00d950 con 0x7fe6d00b22c0 2022-06-29T16:05:00.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.870+0000 7fe6d72e9700 1 -- 172.21.15.60:0/3345855513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6d00b22c0 msgr2=0x7fe6d00ab920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:00.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.870+0000 7fe6d72e9700 1 --2- 172.21.15.60:0/3345855513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6d00b22c0 0x7fe6d00ab920 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7fe6cc002f70 tx=0x7fe6cc006eb0).stop 2022-06-29T16:05:00.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.870+0000 7fe6d72e9700 1 -- 172.21.15.60:0/3345855513 shutdown_connections 2022-06-29T16:05:00.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.870+0000 7fe6d72e9700 1 --2- 172.21.15.60:0/3345855513 >> v2:172.21.15.92:3300/0 conn(0x7fe6d00b0800 0x7fe6d01466e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:00.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.870+0000 7fe6d72e9700 1 --2- 172.21.15.60:0/3345855513 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6d00b22c0 0x7fe6d00ab920 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:00.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.870+0000 7fe6d72e9700 1 -- 172.21.15.60:0/3345855513 >> 172.21.15.60:0/3345855513 conn(0x7fe6d00b58a0 msgr2=0x7fe6d00b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:00.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.870+0000 7fe6d72e9700 1 -- 172.21.15.60:0/3345855513 shutdown_connections 2022-06-29T16:05:00.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.870+0000 7fe6d72e9700 1 -- 172.21.15.60:0/3345855513 wait complete. 2022-06-29T16:05:00.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.870+0000 7fe6d72e9700 1 Processor -- start 2022-06-29T16:05:00.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.870+0000 7fe6d72e9700 1 -- start start 2022-06-29T16:05:00.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.874+0000 7fe6d72e9700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6d00b0800 0x7fe6d00ad540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:00.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.874+0000 7fe6d72e9700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6d00b0f90 0x7fe6d009a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:00.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.874+0000 7fe6d72e9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6d00b22c0 0x7fe6d01520e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:00.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.874+0000 7fe6d72e9700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d0151cf0 con 0x7fe6d00b22c0 2022-06-29T16:05:00.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.874+0000 7fe6d72e9700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d0095c30 con 0x7fe6d00b0f90 2022-06-29T16:05:00.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.874+0000 7fe6d72e9700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d0095e60 con 0x7fe6d00b0800 2022-06-29T16:05:00.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.874+0000 7fe6d6ae8700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6d00b22c0 0x7fe6d01520e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:00.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.874+0000 7fe6d6ae8700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6d00b22c0 0x7fe6d01520e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49614/0 (socket says 172.21.15.60:49614) 2022-06-29T16:05:00.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.874+0000 7fe6d6ae8700 1 -- 172.21.15.60:0/2877453454 learned_addr learned my addr 172.21.15.60:0/2877453454 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:00.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.874+0000 7fe6d62e7700 1 --2- 172.21.15.60:0/2877453454 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6d00b0800 0x7fe6d00ad540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:00.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.874+0000 7fe6d5ae6700 1 --2- 172.21.15.60:0/2877453454 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6d00b0f90 0x7fe6d009a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:00.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.874+0000 7fe6d6ae8700 1 -- 172.21.15.60:0/2877453454 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6d00b0800 msgr2=0x7fe6d00ad540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:00.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.874+0000 7fe6d6ae8700 1 --2- 172.21.15.60:0/2877453454 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6d00b0800 0x7fe6d00ad540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:00.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.874+0000 7fe6d6ae8700 1 -- 172.21.15.60:0/2877453454 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6d00b0f90 msgr2=0x7fe6d009a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:00.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.874+0000 7fe6d6ae8700 1 --2- 172.21.15.60:0/2877453454 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6d00b0f90 0x7fe6d009a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:00.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.874+0000 7fe6d6ae8700 1 -- 172.21.15.60:0/2877453454 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6cc00b1e0 con 0x7fe6d00b22c0 2022-06-29T16:05:00.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.874+0000 7fe6d6ae8700 1 --2- 172.21.15.60:0/2877453454 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6d00b22c0 0x7fe6d01520e0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0x7fe6c8006e00 tx=0x7fe6c80091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:00.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.874+0000 7fe6bf7fe700 1 -- 172.21.15.60:0/2877453454 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6c800c640 con 0x7fe6d00b22c0 2022-06-29T16:05:00.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.874+0000 7fe6bf7fe700 1 -- 172.21.15.60:0/2877453454 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6c800c7a0 con 0x7fe6d00b22c0 2022-06-29T16:05:00.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.874+0000 7fe6d72e9700 1 -- 172.21.15.60:0/2877453454 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6d009a6e0 con 0x7fe6d00b22c0 2022-06-29T16:05:00.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.874+0000 7fe6bf7fe700 1 -- 172.21.15.60:0/2877453454 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6c800cb00 con 0x7fe6d00b22c0 2022-06-29T16:05:00.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.874+0000 7fe6d72e9700 1 -- 172.21.15.60:0/2877453454 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d0093b70 con 0x7fe6d00b22c0 2022-06-29T16:05:00.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.874+0000 7fe6d72e9700 1 -- 172.21.15.60:0/2877453454 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6d0093e80 con 0x7fe6d00b22c0 2022-06-29T16:05:00.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.878+0000 7fe6bf7fe700 1 -- 172.21.15.60:0/2877453454 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe6c800cc60 con 0x7fe6d00b22c0 2022-06-29T16:05:00.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.878+0000 7fe6bf7fe700 1 --2- 172.21.15.60:0/2877453454 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe6b0053aa0 0x7fe6b0055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:00.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.878+0000 7fe6bf7fe700 1 -- 172.21.15.60:0/2877453454 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6116+0+0 (secure 0 0 0) 0x7fe6c806c0f0 con 0x7fe6d00b22c0 2022-06-29T16:05:00.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.878+0000 7fe6d62e7700 1 --2- 172.21.15.60:0/2877453454 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe6b0053aa0 0x7fe6b0055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:00.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.878+0000 7fe6d62e7700 1 --2- 172.21.15.60:0/2877453454 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe6b0053aa0 0x7fe6b0055f60 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7fe6cc01c000 tx=0x7fe6cc015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:00.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:00.886+0000 7fe6bf7fe700 1 -- 172.21.15.60:0/2877453454 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe6c8042fe0 con 0x7fe6d00b22c0 2022-06-29T16:05:01.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.014+0000 7fe6d72e9700 1 -- 172.21.15.60:0/2877453454 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fe6d00573d0 con 0x7fe6d00b22c0 2022-06-29T16:05:01.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.014+0000 7fe6bf7fe700 1 -- 172.21.15.60:0/2877453454 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v72) v1 ==== 110+0+10 (secure 0 0 0) 0x7fe6c8046600 con 0x7fe6d00b22c0 2022-06-29T16:05:01.023 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:05:01.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.018+0000 7fe6d72e9700 1 -- 172.21.15.60:0/2877453454 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe6b0053aa0 msgr2=0x7fe6b0055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:01.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.018+0000 7fe6d72e9700 1 --2- 172.21.15.60:0/2877453454 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe6b0053aa0 0x7fe6b0055f60 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7fe6cc01c000 tx=0x7fe6cc015000).stop 2022-06-29T16:05:01.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.018+0000 7fe6d72e9700 1 -- 172.21.15.60:0/2877453454 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6d00b22c0 msgr2=0x7fe6d01520e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:01.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.018+0000 7fe6d72e9700 1 --2- 172.21.15.60:0/2877453454 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6d00b22c0 0x7fe6d01520e0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0x7fe6c8006e00 tx=0x7fe6c80091d0).stop 2022-06-29T16:05:01.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.018+0000 7fe6d72e9700 1 -- 172.21.15.60:0/2877453454 shutdown_connections 2022-06-29T16:05:01.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.018+0000 7fe6d72e9700 1 --2- 172.21.15.60:0/2877453454 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe6b0053aa0 0x7fe6b0055f60 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.018+0000 7fe6d72e9700 1 --2- 172.21.15.60:0/2877453454 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6d00b22c0 0x7fe6d01520e0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.018+0000 7fe6d72e9700 1 --2- 172.21.15.60:0/2877453454 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6d00b0f90 0x7fe6d009a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.018+0000 7fe6d72e9700 1 --2- 172.21.15.60:0/2877453454 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6d00b0800 0x7fe6d00ad540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.018+0000 7fe6d72e9700 1 -- 172.21.15.60:0/2877453454 >> 172.21.15.60:0/2877453454 conn(0x7fe6d00b58a0 msgr2=0x7fe6d00b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:01.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.018+0000 7fe6d72e9700 1 -- 172.21.15.60:0/2877453454 shutdown_connections 2022-06-29T16:05:01.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.018+0000 7fe6d72e9700 1 -- 172.21.15.60:0/2877453454 wait complete. 2022-06-29T16:05:01.036 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:05:01.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.134+0000 7fc83b59e700 1 Processor -- start 2022-06-29T16:05:01.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.134+0000 7fc83b59e700 1 -- start start 2022-06-29T16:05:01.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.134+0000 7fc83b59e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc83c144b00 0x7fc83c0b2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:01.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.134+0000 7fc83b59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc83c0b3430 0x7fc83c0b1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:01.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.134+0000 7fc83b59e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc83c067100 con 0x7fc83c066d20 2022-06-29T16:05:01.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.134+0000 7fc83b59e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc83c0603a0 con 0x7fc83c144b00 2022-06-29T16:05:01.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.134+0000 7fc83b59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc83c060640 con 0x7fc83c0b3430 2022-06-29T16:05:01.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.134+0000 7fc839d9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc83c144b00 0x7fc83c0b2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:01.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.134+0000 7fc839d9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc83c144b00 0x7fc83c0b2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49622/0 (socket says 172.21.15.60:49622) 2022-06-29T16:05:01.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.134+0000 7fc839d9b700 1 -- 172.21.15.60:0/2338645776 learned_addr learned my addr 172.21.15.60:0/2338645776 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:01.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.134+0000 7fc83ad9d700 1 --2- 172.21.15.60:0/2338645776 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc83c0b3430 0x7fc83c0b1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:01.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.134+0000 7fc839d9b700 1 -- 172.21.15.60:0/2338645776 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc83c0b3430 msgr2=0x7fc83c0b1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:01.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.134+0000 7fc839d9b700 1 --2- 172.21.15.60:0/2338645776 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc83c0b3430 0x7fc83c0b1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.134+0000 7fc839d9b700 1 -- 172.21.15.60:0/2338645776 >> v1:172.21.15.92:6789/0 conn(0x7fc83c066d20 legacy=0x7fc83c1443f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:01.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.134+0000 7fc839d9b700 1 -- 172.21.15.60:0/2338645776 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc83c143f50 con 0x7fc83c144b00 2022-06-29T16:05:01.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.134+0000 7fc83ad9d700 1 --2- 172.21.15.60:0/2338645776 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc83c0b3430 0x7fc83c0b1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:01.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.134+0000 7fc839d9b700 1 --2- 172.21.15.60:0/2338645776 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc83c144b00 0x7fc83c0b2ea0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7fc824002f70 tx=0x7fc824006eb0).ready entity=mon.0 client_cookie=915c331f32d2049b server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:01.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.138+0000 7fc83959a700 1 -- 172.21.15.60:0/2338645776 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc82400d5c0 con 0x7fc83c144b00 2022-06-29T16:05:01.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.138+0000 7fc83959a700 1 -- 172.21.15.60:0/2338645776 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc824000e90 con 0x7fc83c144b00 2022-06-29T16:05:01.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.138+0000 7fc83959a700 1 -- 172.21.15.60:0/2338645776 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc82400d950 con 0x7fc83c144b00 2022-06-29T16:05:01.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.138+0000 7fc83b59e700 1 -- 172.21.15.60:0/2338645776 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc83c144b00 msgr2=0x7fc83c0b2ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:01.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.138+0000 7fc83b59e700 1 --2- 172.21.15.60:0/2338645776 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc83c144b00 0x7fc83c0b2ea0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7fc824002f70 tx=0x7fc824006eb0).stop 2022-06-29T16:05:01.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.138+0000 7fc83b59e700 1 -- 172.21.15.60:0/2338645776 shutdown_connections 2022-06-29T16:05:01.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.138+0000 7fc83b59e700 1 --2- 172.21.15.60:0/2338645776 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc83c0b3430 0x7fc83c0b1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.138+0000 7fc83b59e700 1 --2- 172.21.15.60:0/2338645776 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc83c144b00 0x7fc83c0b2ea0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.138+0000 7fc83b59e700 1 -- 172.21.15.60:0/2338645776 >> 172.21.15.60:0/2338645776 conn(0x7fc83c0637c0 msgr2=0x7fc83c063ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:01.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.138+0000 7fc83b59e700 1 -- 172.21.15.60:0/2338645776 shutdown_connections 2022-06-29T16:05:01.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.138+0000 7fc83b59e700 1 -- 172.21.15.60:0/2338645776 wait complete. 2022-06-29T16:05:01.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.138+0000 7fc83b59e700 1 Processor -- start 2022-06-29T16:05:01.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.138+0000 7fc83b59e700 1 -- start start 2022-06-29T16:05:01.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.138+0000 7fc83b59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc83c066d20 0x7fc83c153020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:01.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.138+0000 7fc83b59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc83c0b3430 0x7fc83c0b3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:01.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.138+0000 7fc83b59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc83c144b00 0x7fc83c13bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:01.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.138+0000 7fc83b59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc83c1105d0 con 0x7fc83c144b00 2022-06-29T16:05:01.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.138+0000 7fc83b59e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc83c0af490 con 0x7fc83c0b3430 2022-06-29T16:05:01.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.138+0000 7fc83b59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc83c0af730 con 0x7fc83c066d20 2022-06-29T16:05:01.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.142+0000 7fc83ad9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc83c144b00 0x7fc83c13bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:01.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.142+0000 7fc83ad9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc83c144b00 0x7fc83c13bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49630/0 (socket says 172.21.15.60:49630) 2022-06-29T16:05:01.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.142+0000 7fc839d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc83c0b3430 0x7fc83c0b3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:01.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.142+0000 7fc83ad9d700 1 -- 172.21.15.60:0/3212762229 learned_addr learned my addr 172.21.15.60:0/3212762229 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:01.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.142+0000 7fc83a59c700 1 --2- 172.21.15.60:0/3212762229 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc83c066d20 0x7fc83c153020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:01.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.142+0000 7fc83ad9d700 1 -- 172.21.15.60:0/3212762229 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc83c066d20 msgr2=0x7fc83c153020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:01.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.142+0000 7fc83ad9d700 1 --2- 172.21.15.60:0/3212762229 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc83c066d20 0x7fc83c153020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.142+0000 7fc83ad9d700 1 -- 172.21.15.60:0/3212762229 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc83c0b3430 msgr2=0x7fc83c0b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:01.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.142+0000 7fc83ad9d700 1 --2- 172.21.15.60:0/3212762229 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc83c0b3430 0x7fc83c0b3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.142+0000 7fc83ad9d700 1 -- 172.21.15.60:0/3212762229 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc82400b1e0 con 0x7fc83c144b00 2022-06-29T16:05:01.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.142+0000 7fc83a59c700 1 --2- 172.21.15.60:0/3212762229 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc83c066d20 0x7fc83c153020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:01.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.142+0000 7fc83ad9d700 1 --2- 172.21.15.60:0/3212762229 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc83c144b00 0x7fc83c13bbe0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0x7fc834006cc0 tx=0x7fc834008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:01.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.142+0000 7fc8337fe700 1 -- 172.21.15.60:0/3212762229 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc834013620 con 0x7fc83c144b00 2022-06-29T16:05:01.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.142+0000 7fc8337fe700 1 -- 172.21.15.60:0/3212762229 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc834013780 con 0x7fc83c144b00 2022-06-29T16:05:01.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.142+0000 7fc83b59e700 1 -- 172.21.15.60:0/3212762229 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc83c0af110 con 0x7fc83c144b00 2022-06-29T16:05:01.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.142+0000 7fc8337fe700 1 -- 172.21.15.60:0/3212762229 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc834013ae0 con 0x7fc83c144b00 2022-06-29T16:05:01.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.142+0000 7fc83b59e700 1 -- 172.21.15.60:0/3212762229 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc83c0b4160 con 0x7fc83c144b00 2022-06-29T16:05:01.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.142+0000 7fc8337fe700 1 -- 172.21.15.60:0/3212762229 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc834013c40 con 0x7fc83c144b00 2022-06-29T16:05:01.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.142+0000 7fc83b59e700 1 -- 172.21.15.60:0/3212762229 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc83c0af8f0 con 0x7fc83c144b00 2022-06-29T16:05:01.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.142+0000 7fc8337fe700 1 --2- 172.21.15.60:0/3212762229 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc820053b30 0x7fc820055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:01.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.142+0000 7fc8337fe700 1 -- 172.21.15.60:0/3212762229 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6116+0+0 (secure 0 0 0) 0x7fc83406c170 con 0x7fc83c144b00 2022-06-29T16:05:01.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.142+0000 7fc83a59c700 1 --2- 172.21.15.60:0/3212762229 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc820053b30 0x7fc820055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:01.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.146+0000 7fc83a59c700 1 --2- 172.21.15.60:0/3212762229 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc820053b30 0x7fc820055ff0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7fc82c00b000 tx=0x7fc82c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:01.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.146+0000 7fc8337fe700 1 -- 172.21.15.60:0/3212762229 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc834042fc0 con 0x7fc83c144b00 2022-06-29T16:05:01.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.278+0000 7fc83b59e700 1 -- 172.21.15.60:0/3212762229 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fc83c0573d0 con 0x7fc83c144b00 2022-06-29T16:05:01.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.278+0000 7fc8337fe700 1 -- 172.21.15.60:0/3212762229 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v72) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc8340465e0 con 0x7fc83c144b00 2022-06-29T16:05:01.286 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:05:01.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.278+0000 7fc83b59e700 1 -- 172.21.15.60:0/3212762229 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc820053b30 msgr2=0x7fc820055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:01.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.278+0000 7fc83b59e700 1 --2- 172.21.15.60:0/3212762229 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc820053b30 0x7fc820055ff0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7fc82c00b000 tx=0x7fc82c009040).stop 2022-06-29T16:05:01.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.278+0000 7fc83b59e700 1 -- 172.21.15.60:0/3212762229 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc83c144b00 msgr2=0x7fc83c13bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:01.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.278+0000 7fc83b59e700 1 --2- 172.21.15.60:0/3212762229 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc83c144b00 0x7fc83c13bbe0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0x7fc834006cc0 tx=0x7fc834008040).stop 2022-06-29T16:05:01.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.278+0000 7fc83b59e700 1 -- 172.21.15.60:0/3212762229 shutdown_connections 2022-06-29T16:05:01.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.278+0000 7fc83b59e700 1 --2- 172.21.15.60:0/3212762229 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc820053b30 0x7fc820055ff0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.278+0000 7fc83b59e700 1 --2- 172.21.15.60:0/3212762229 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc83c144b00 0x7fc83c13bbe0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.278+0000 7fc83b59e700 1 --2- 172.21.15.60:0/3212762229 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc83c0b3430 0x7fc83c0b3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.278+0000 7fc83b59e700 1 --2- 172.21.15.60:0/3212762229 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc83c066d20 0x7fc83c153020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.278+0000 7fc83b59e700 1 -- 172.21.15.60:0/3212762229 >> 172.21.15.60:0/3212762229 conn(0x7fc83c0637c0 msgr2=0x7fc83c13c970 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:01.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.282+0000 7fc83b59e700 1 -- 172.21.15.60:0/3212762229 shutdown_connections 2022-06-29T16:05:01.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.282+0000 7fc83b59e700 1 -- 172.21.15.60:0/3212762229 wait complete. 2022-06-29T16:05:01.299 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-29T16:05:01.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.398+0000 7f49789ab700 1 Processor -- start 2022-06-29T16:05:01.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.398+0000 7f49789ab700 1 -- start start 2022-06-29T16:05:01.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.398+0000 7f49789ab700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f49740634d0 0x7f497413e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:01.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.398+0000 7f49789ab700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f49740aca20 0x7f49740b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:01.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.398+0000 7f49789ab700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49740b2650 con 0x7f49740b2270 2022-06-29T16:05:01.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.398+0000 7f49789ab700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4974067770 con 0x7f49740634d0 2022-06-29T16:05:01.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.398+0000 7f49789ab700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4974062af0 con 0x7f49740aca20 2022-06-29T16:05:01.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.398+0000 7f4972ffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f49740634d0 0x7f497413e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:01.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.398+0000 7f49737fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f49740b2270 0x7f497413e270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57720/0 (socket says 172.21.15.60:57720) 2022-06-29T16:05:01.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.398+0000 7f49737fe700 1 -- 172.21.15.60:0/2331494243 learned_addr learned my addr 172.21.15.60:0/2331494243 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:01.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.398+0000 7f4973fff700 1 --2- 172.21.15.60:0/2331494243 >> v2:172.21.15.92:3300/0 conn(0x7f49740aca20 0x7f49740b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:01.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.398+0000 7f4972ffd700 1 -- 172.21.15.60:0/2331494243 >> v2:172.21.15.92:3300/0 conn(0x7f49740aca20 msgr2=0x7f49740b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:01.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.398+0000 7f4972ffd700 1 --2- 172.21.15.60:0/2331494243 >> v2:172.21.15.92:3300/0 conn(0x7f49740aca20 0x7f49740b0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.398+0000 7f4972ffd700 1 -- 172.21.15.60:0/2331494243 >> v1:172.21.15.60:6789/0 conn(0x7f49740b2270 legacy=0x7f497413e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:01.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.398+0000 7f4972ffd700 1 -- 172.21.15.60:0/2331494243 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4974062d90 con 0x7f49740634d0 2022-06-29T16:05:01.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.398+0000 7f4972ffd700 1 --2- 172.21.15.60:0/2331494243 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49740634d0 0x7f497413e980 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0x7f495c002f70 tx=0x7f495c006df0).ready entity=mon.0 client_cookie=4a29d02155985f35 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:01.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.398+0000 7f49727fc700 1 -- 172.21.15.60:0/2331494243 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f495c00d5c0 con 0x7f49740634d0 2022-06-29T16:05:01.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.398+0000 7f49727fc700 1 -- 172.21.15.60:0/2331494243 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f495c000e90 con 0x7f49740634d0 2022-06-29T16:05:01.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.398+0000 7f49727fc700 1 -- 172.21.15.60:0/2331494243 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f495c00d910 con 0x7f49740634d0 2022-06-29T16:05:01.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.402+0000 7f49789ab700 1 -- 172.21.15.60:0/2331494243 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49740634d0 msgr2=0x7f497413e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:01.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.402+0000 7f49789ab700 1 --2- 172.21.15.60:0/2331494243 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49740634d0 0x7f497413e980 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0x7f495c002f70 tx=0x7f495c006df0).stop 2022-06-29T16:05:01.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.402+0000 7f49789ab700 1 -- 172.21.15.60:0/2331494243 shutdown_connections 2022-06-29T16:05:01.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.402+0000 7f49789ab700 1 --2- 172.21.15.60:0/2331494243 >> v2:172.21.15.92:3300/0 conn(0x7f49740aca20 0x7f49740b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.402+0000 7f49789ab700 1 --2- 172.21.15.60:0/2331494243 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49740634d0 0x7f497413e980 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.402+0000 7f49789ab700 1 -- 172.21.15.60:0/2331494243 >> 172.21.15.60:0/2331494243 conn(0x7f49740b5890 msgr2=0x7f49740b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:01.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.402+0000 7f49789ab700 1 -- 172.21.15.60:0/2331494243 shutdown_connections 2022-06-29T16:05:01.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.402+0000 7f49789ab700 1 -- 172.21.15.60:0/2331494243 wait complete. 2022-06-29T16:05:01.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.402+0000 7f49789ab700 1 Processor -- start 2022-06-29T16:05:01.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.406+0000 7f49789ab700 1 -- start start 2022-06-29T16:05:01.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.406+0000 7f49789ab700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49740634d0 0x7f4974063b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:01.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.406+0000 7f49789ab700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49740aca20 0x7f4974145a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:01.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.406+0000 7f49789ab700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f49740b2270 0x7f49740b2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:01.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.406+0000 7f49789ab700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4974064da0 con 0x7f49740aca20 2022-06-29T16:05:01.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.406+0000 7f49789ab700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49740b48b0 con 0x7f49740634d0 2022-06-29T16:05:01.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.406+0000 7f49789ab700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49740b4b10 con 0x7f49740b2270 2022-06-29T16:05:01.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.406+0000 7f4972ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49740aca20 0x7f4974145a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:01.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.406+0000 7f4972ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49740aca20 0x7f4974145a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49642/0 (socket says 172.21.15.60:49642) 2022-06-29T16:05:01.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.406+0000 7f4972ffd700 1 -- 172.21.15.60:0/3345380926 learned_addr learned my addr 172.21.15.60:0/3345380926 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:01.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.406+0000 7f49737fe700 1 --2- 172.21.15.60:0/3345380926 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49740634d0 0x7f4974063b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:01.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.406+0000 7f4973fff700 1 --2- 172.21.15.60:0/3345380926 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f49740b2270 0x7f49740b2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:01.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.406+0000 7f4972ffd700 1 -- 172.21.15.60:0/3345380926 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f49740b2270 msgr2=0x7f49740b2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:01.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.406+0000 7f4972ffd700 1 --2- 172.21.15.60:0/3345380926 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f49740b2270 0x7f49740b2bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.406+0000 7f4972ffd700 1 -- 172.21.15.60:0/3345380926 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49740634d0 msgr2=0x7f4974063b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:01.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.406+0000 7f4972ffd700 1 --2- 172.21.15.60:0/3345380926 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49740634d0 0x7f4974063b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.406+0000 7f4972ffd700 1 -- 172.21.15.60:0/3345380926 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f495c00b180 con 0x7f49740aca20 2022-06-29T16:05:01.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.406+0000 7f4972ffd700 1 --2- 172.21.15.60:0/3345380926 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49740aca20 0x7f4974145a80 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0x7f495c002f70 tx=0x7f495c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:01.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.406+0000 7f4970ff9700 1 -- 172.21.15.60:0/3345380926 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f495c016070 con 0x7f49740aca20 2022-06-29T16:05:01.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.406+0000 7f4970ff9700 1 -- 172.21.15.60:0/3345380926 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f495c00d940 con 0x7f49740aca20 2022-06-29T16:05:01.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.406+0000 7f49789ab700 1 -- 172.21.15.60:0/3345380926 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4974066280 con 0x7f49740aca20 2022-06-29T16:05:01.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.406+0000 7f4970ff9700 1 -- 172.21.15.60:0/3345380926 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f495c013da0 con 0x7f49740aca20 2022-06-29T16:05:01.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.406+0000 7f49789ab700 1 -- 172.21.15.60:0/3345380926 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4974066720 con 0x7f49740aca20 2022-06-29T16:05:01.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.410+0000 7f49789ab700 1 -- 172.21.15.60:0/3345380926 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4974066410 con 0x7f49740aca20 2022-06-29T16:05:01.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.410+0000 7f4970ff9700 1 -- 172.21.15.60:0/3345380926 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f495c025030 con 0x7f49740aca20 2022-06-29T16:05:01.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.410+0000 7f4970ff9700 1 --2- 172.21.15.60:0/3345380926 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4958053aa0 0x7f4958055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:01.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.410+0000 7f4970ff9700 1 -- 172.21.15.60:0/3345380926 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6116+0+0 (secure 0 0 0) 0x7f495c06ce70 con 0x7f49740aca20 2022-06-29T16:05:01.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.410+0000 7f49737fe700 1 --2- 172.21.15.60:0/3345380926 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4958053aa0 0x7f4958055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:01.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.414+0000 7f49737fe700 1 --2- 172.21.15.60:0/3345380926 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4958053aa0 0x7f4958055f60 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f496400d000 tx=0x7f4964000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:01.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.422+0000 7f4970ff9700 1 -- 172.21.15.60:0/3345380926 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f495c043d60 con 0x7f49740aca20 2022-06-29T16:05:01.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.546+0000 7f49789ab700 1 -- 172.21.15.60:0/3345380926 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f49740573d0 con 0x7f49740aca20 2022-06-29T16:05:01.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.546+0000 7f4970ff9700 1 -- 172.21.15.60:0/3345380926 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v72) v1 ==== 104+0+10 (secure 0 0 0) 0x7f495c047380 con 0x7f49740aca20 2022-06-29T16:05:01.554 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:05:01.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.546+0000 7f49789ab700 1 -- 172.21.15.60:0/3345380926 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4958053aa0 msgr2=0x7f4958055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:01.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.546+0000 7f49789ab700 1 --2- 172.21.15.60:0/3345380926 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4958053aa0 0x7f4958055f60 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f496400d000 tx=0x7f4964000bc0).stop 2022-06-29T16:05:01.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.546+0000 7f49789ab700 1 -- 172.21.15.60:0/3345380926 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49740aca20 msgr2=0x7f4974145a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:01.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.546+0000 7f49789ab700 1 --2- 172.21.15.60:0/3345380926 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49740aca20 0x7f4974145a80 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0x7f495c002f70 tx=0x7f495c014000).stop 2022-06-29T16:05:01.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.546+0000 7f49789ab700 1 -- 172.21.15.60:0/3345380926 shutdown_connections 2022-06-29T16:05:01.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.546+0000 7f49789ab700 1 --2- 172.21.15.60:0/3345380926 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4958053aa0 0x7f4958055f60 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.546+0000 7f49789ab700 1 --2- 172.21.15.60:0/3345380926 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f49740b2270 0x7f49740b2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.546+0000 7f49789ab700 1 --2- 172.21.15.60:0/3345380926 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49740aca20 0x7f4974145a80 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.546+0000 7f49789ab700 1 --2- 172.21.15.60:0/3345380926 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49740634d0 0x7f4974063b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.546+0000 7f49789ab700 1 -- 172.21.15.60:0/3345380926 >> 172.21.15.60:0/3345380926 conn(0x7f49740b5890 msgr2=0x7f49740ad810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:01.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.546+0000 7f49789ab700 1 -- 172.21.15.60:0/3345380926 shutdown_connections 2022-06-29T16:05:01.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.546+0000 7f49789ab700 1 -- 172.21.15.60:0/3345380926 wait complete. 2022-06-29T16:05:01.567 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-29T16:05:01.680 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.674+0000 7f7238c33700 1 Processor -- start 2022-06-29T16:05:01.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.674+0000 7f7238c33700 1 -- start start 2022-06-29T16:05:01.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.674+0000 7f7238c33700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f72340b61d0 0x7f72340b0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:01.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.674+0000 7f7238c33700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f723413fbd0 0x7f723413c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:01.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.674+0000 7f7238c33700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7234142820 con 0x7f723413f790 2022-06-29T16:05:01.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.674+0000 7f7238c33700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7234062bb0 con 0x7f723413fbd0 2022-06-29T16:05:01.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.674+0000 7f7238c33700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72340b65b0 con 0x7f72340b61d0 2022-06-29T16:05:01.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.674+0000 7f7232d9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f723413fbd0 0x7f723413c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:01.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.674+0000 7f722bfff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f723413f790 0x7f723413bc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57736/0 (socket says 172.21.15.60:57736) 2022-06-29T16:05:01.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.674+0000 7f722bfff700 1 -- 172.21.15.60:0/3375577922 learned_addr learned my addr 172.21.15.60:0/3375577922 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:01.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.674+0000 7f723259c700 1 --2- 172.21.15.60:0/3375577922 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f72340b61d0 0x7f72340b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:01.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.674+0000 7f7232d9d700 1 -- 172.21.15.60:0/3375577922 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f72340b61d0 msgr2=0x7f72340b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:01.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.674+0000 7f7232d9d700 1 --2- 172.21.15.60:0/3375577922 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f72340b61d0 0x7f72340b0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.674+0000 7f7232d9d700 1 -- 172.21.15.60:0/3375577922 >> v1:172.21.15.60:6789/0 conn(0x7f723413f790 legacy=0x7f723413bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:01.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.674+0000 7f7232d9d700 1 -- 172.21.15.60:0/3375577922 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f723413f050 con 0x7f723413fbd0 2022-06-29T16:05:01.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.674+0000 7f723259c700 1 --2- 172.21.15.60:0/3375577922 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f72340b61d0 0x7f72340b0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:01.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.674+0000 7f7232d9d700 1 --2- 172.21.15.60:0/3375577922 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f723413fbd0 0x7f723413c350 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0x7f722c0032f0 tx=0x7f722c0091d0).ready entity=mon.0 client_cookie=55961da08e8f1048 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:01.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.674+0000 7f7231d9b700 1 -- 172.21.15.60:0/3375577922 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f722c009ea0 con 0x7f723413fbd0 2022-06-29T16:05:01.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.674+0000 7f7231d9b700 1 -- 172.21.15.60:0/3375577922 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f722c00ba90 con 0x7f723413fbd0 2022-06-29T16:05:01.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.674+0000 7f7231d9b700 1 -- 172.21.15.60:0/3375577922 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f722c00bdf0 con 0x7f723413fbd0 2022-06-29T16:05:01.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.674+0000 7f7238c33700 1 -- 172.21.15.60:0/3375577922 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f723413fbd0 msgr2=0x7f723413c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:01.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.674+0000 7f7238c33700 1 --2- 172.21.15.60:0/3375577922 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f723413fbd0 0x7f723413c350 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0x7f722c0032f0 tx=0x7f722c0091d0).stop 2022-06-29T16:05:01.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.674+0000 7f7238c33700 1 -- 172.21.15.60:0/3375577922 shutdown_connections 2022-06-29T16:05:01.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.674+0000 7f7238c33700 1 --2- 172.21.15.60:0/3375577922 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f723413fbd0 0x7f723413c350 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.674+0000 7f7238c33700 1 --2- 172.21.15.60:0/3375577922 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f72340b61d0 0x7f72340b0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.674+0000 7f7238c33700 1 -- 172.21.15.60:0/3375577922 >> 172.21.15.60:0/3375577922 conn(0x7f7234063b50 msgr2=0x7f72340b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:01.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.674+0000 7f7238c33700 1 -- 172.21.15.60:0/3375577922 shutdown_connections 2022-06-29T16:05:01.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.674+0000 7f7238c33700 1 -- 172.21.15.60:0/3375577922 wait complete. 2022-06-29T16:05:01.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.678+0000 7f7238c33700 1 Processor -- start 2022-06-29T16:05:01.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.678+0000 7f7238c33700 1 -- start start 2022-06-29T16:05:01.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.678+0000 7f7238c33700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f72340b61d0 0x7f72340af210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:01.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.678+0000 7f7238c33700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f723413f790 0x7f72341134d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:01.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.678+0000 7f7238c33700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f723413fbd0 0x7f7234112d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:01.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.678+0000 7f7238c33700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72341561e0 con 0x7f723413f790 2022-06-29T16:05:01.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.678+0000 7f7238c33700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7234066ff0 con 0x7f72340b61d0 2022-06-29T16:05:01.696 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.678+0000 7f7238c33700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72341159a0 con 0x7f723413fbd0 2022-06-29T16:05:01.696 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.678+0000 7f722bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f723413f790 0x7f72341134d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:01.696 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.678+0000 7f722bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f723413f790 0x7f72341134d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49656/0 (socket says 172.21.15.60:49656) 2022-06-29T16:05:01.696 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.678+0000 7f722bfff700 1 -- 172.21.15.60:0/124583023 learned_addr learned my addr 172.21.15.60:0/124583023 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:01.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.678+0000 7f723259c700 1 --2- 172.21.15.60:0/124583023 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f72340b61d0 0x7f72340af210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:01.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.678+0000 7f7232d9d700 1 --2- 172.21.15.60:0/124583023 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f723413fbd0 0x7f7234112d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:01.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.678+0000 7f722bfff700 1 -- 172.21.15.60:0/124583023 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f723413fbd0 msgr2=0x7f7234112d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:01.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.678+0000 7f722bfff700 1 --2- 172.21.15.60:0/124583023 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f723413fbd0 0x7f7234112d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.678+0000 7f722bfff700 1 -- 172.21.15.60:0/124583023 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f72340b61d0 msgr2=0x7f72340af210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:01.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.678+0000 7f722bfff700 1 --2- 172.21.15.60:0/124583023 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f72340b61d0 0x7f72340af210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.678+0000 7f722bfff700 1 -- 172.21.15.60:0/124583023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f722c009470 con 0x7f723413f790 2022-06-29T16:05:01.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.678+0000 7f722bfff700 1 --2- 172.21.15.60:0/124583023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f723413f790 0x7f72341134d0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0x7f7234062a10 tx=0x7f721c007c40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:01.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.678+0000 7f722b7fe700 1 -- 172.21.15.60:0/124583023 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f721c0070d0 con 0x7f723413f790 2022-06-29T16:05:01.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.678+0000 7f722b7fe700 1 -- 172.21.15.60:0/124583023 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f721c007230 con 0x7f723413f790 2022-06-29T16:05:01.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.678+0000 7f7238c33700 1 -- 172.21.15.60:0/124583023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72341139c0 con 0x7f723413f790 2022-06-29T16:05:01.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.682+0000 7f722b7fe700 1 -- 172.21.15.60:0/124583023 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f721c005610 con 0x7f723413f790 2022-06-29T16:05:01.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.682+0000 7f7238c33700 1 -- 172.21.15.60:0/124583023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7234113270 con 0x7f723413f790 2022-06-29T16:05:01.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.682+0000 7f7238c33700 1 -- 172.21.15.60:0/124583023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7234065a60 con 0x7f723413f790 2022-06-29T16:05:01.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.682+0000 7f722b7fe700 1 -- 172.21.15.60:0/124583023 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f721c010070 con 0x7f723413f790 2022-06-29T16:05:01.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.682+0000 7f722b7fe700 1 --2- 172.21.15.60:0/124583023 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f72180539e0 0x7f7218055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:01.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.682+0000 7f722b7fe700 1 -- 172.21.15.60:0/124583023 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6116+0+0 (secure 0 0 0) 0x7f721c06b3f0 con 0x7f723413f790 2022-06-29T16:05:01.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.682+0000 7f723259c700 1 --2- 172.21.15.60:0/124583023 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f72180539e0 0x7f7218055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:01.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.682+0000 7f723259c700 1 --2- 172.21.15.60:0/124583023 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f72180539e0 0x7f7218055ea0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f722400c000 tx=0x7f7224008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:01.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.686+0000 7f722b7fe700 1 -- 172.21.15.60:0/124583023 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f721c0422e0 con 0x7f723413f790 2022-06-29T16:05:01.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.814+0000 7f7238c33700 1 -- 172.21.15.60:0/124583023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7234115b10 con 0x7f723413f790 2022-06-29T16:05:01.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.814+0000 7f722b7fe700 1 -- 172.21.15.60:0/124583023 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v72) v1 ==== 100+0+10 (secure 0 0 0) 0x7f721c045900 con 0x7f723413f790 2022-06-29T16:05:01.824 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:05:01.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.818+0000 7f7238c33700 1 -- 172.21.15.60:0/124583023 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f72180539e0 msgr2=0x7f7218055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:01.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.818+0000 7f7238c33700 1 --2- 172.21.15.60:0/124583023 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f72180539e0 0x7f7218055ea0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f722400c000 tx=0x7f7224008040).stop 2022-06-29T16:05:01.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.818+0000 7f7238c33700 1 -- 172.21.15.60:0/124583023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f723413f790 msgr2=0x7f72341134d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:01.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.818+0000 7f7238c33700 1 --2- 172.21.15.60:0/124583023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f723413f790 0x7f72341134d0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0x7f7234062a10 tx=0x7f721c007c40).stop 2022-06-29T16:05:01.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.818+0000 7f7238c33700 1 -- 172.21.15.60:0/124583023 shutdown_connections 2022-06-29T16:05:01.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.818+0000 7f7238c33700 1 --2- 172.21.15.60:0/124583023 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f72180539e0 0x7f7218055ea0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.818+0000 7f7238c33700 1 --2- 172.21.15.60:0/124583023 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f723413fbd0 0x7f7234112d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.818+0000 7f7238c33700 1 --2- 172.21.15.60:0/124583023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f723413f790 0x7f72341134d0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.818+0000 7f7238c33700 1 --2- 172.21.15.60:0/124583023 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f72340b61d0 0x7f72340af210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.818+0000 7f7238c33700 1 -- 172.21.15.60:0/124583023 >> 172.21.15.60:0/124583023 conn(0x7f7234063b50 msgr2=0x7f72340b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:01.829 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.818+0000 7f7238c33700 1 -- 172.21.15.60:0/124583023 shutdown_connections 2022-06-29T16:05:01.829 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.818+0000 7f7238c33700 1 -- 172.21.15.60:0/124583023 wait complete. 2022-06-29T16:05:01.837 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-06-29T16:05:01.838 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:05:01.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.930+0000 7f86ab59e700 1 Processor -- start 2022-06-29T16:05:01.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.930+0000 7f86ab59e700 1 -- start start 2022-06-29T16:05:01.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.930+0000 7f86ab59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f86ac05fb00 0x7f86ac0aea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:01.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.930+0000 7f86ab59e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f86ac0aef40 0x7f86ac0b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:01.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.930+0000 7f86ab59e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86ac05fee0 con 0x7f86ac0b1650 2022-06-29T16:05:01.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.930+0000 7f86ab59e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86ac0b0870 con 0x7f86ac0aef40 2022-06-29T16:05:01.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.930+0000 7f86ab59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac0b0b10 con 0x7f86ac05fb00 2022-06-29T16:05:01.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.930+0000 7f86aad9d700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f86ac0b1650 0x7f86ac14aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57752/0 (socket says 172.21.15.60:57752) 2022-06-29T16:05:01.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.930+0000 7f86aad9d700 1 -- 172.21.15.60:0/3075649424 learned_addr learned my addr 172.21.15.60:0/3075649424 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:01.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.930+0000 7f86aa59c700 1 --2- 172.21.15.60:0/3075649424 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f86ac05fb00 0x7f86ac0aea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:01.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.930+0000 7f86a9d9b700 1 --2- 172.21.15.60:0/3075649424 >> v2:172.21.15.92:3300/0 conn(0x7f86ac0aef40 0x7f86ac0b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:01.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.930+0000 7f86a959a700 1 -- 172.21.15.60:0/3075649424 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4195154553 0 0) 0x7f86ac05fee0 con 0x7f86ac0b1650 2022-06-29T16:05:01.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.930+0000 7f86a959a700 1 -- 172.21.15.60:0/3075649424 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f868c003260 con 0x7f86ac0b1650 2022-06-29T16:05:01.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.930+0000 7f86a959a700 1 -- 172.21.15.60:0/3075649424 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3998381803 0 0) 0x7f868c003260 con 0x7f86ac0b1650 2022-06-29T16:05:01.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.930+0000 7f86aa59c700 1 -- 172.21.15.60:0/3075649424 >> v2:172.21.15.92:3300/0 conn(0x7f86ac0aef40 msgr2=0x7f86ac0b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:01.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.930+0000 7f86aa59c700 1 --2- 172.21.15.60:0/3075649424 >> v2:172.21.15.92:3300/0 conn(0x7f86ac0aef40 0x7f86ac0b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.930+0000 7f86aa59c700 1 -- 172.21.15.60:0/3075649424 >> v1:172.21.15.60:6789/0 conn(0x7f86ac0b1650 legacy=0x7f86ac14aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:01.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.930+0000 7f86aa59c700 1 -- 172.21.15.60:0/3075649424 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86ac060b20 con 0x7f86ac05fb00 2022-06-29T16:05:01.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.930+0000 7f86aa59c700 1 --2- 172.21.15.60:0/3075649424 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f86ac05fb00 0x7f86ac0aea50 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f86a4001800 tx=0x7f869c0091d0).ready entity=mon.2 client_cookie=11e6df76d116d16a server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:01.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.930+0000 7f86a959a700 1 -- 172.21.15.60:0/3075649424 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86a4001660 con 0x7f86ac05fb00 2022-06-29T16:05:01.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.934+0000 7f86a959a700 1 -- 172.21.15.60:0/3075649424 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f869c00f510 con 0x7f86ac05fb00 2022-06-29T16:05:01.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.934+0000 7f86a959a700 1 -- 172.21.15.60:0/3075649424 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f869c00f680 con 0x7f86ac05fb00 2022-06-29T16:05:01.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.934+0000 7f86ab59e700 1 -- 172.21.15.60:0/3075649424 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f86ac05fb00 msgr2=0x7f86ac0aea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:01.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.934+0000 7f86ab59e700 1 --2- 172.21.15.60:0/3075649424 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f86ac05fb00 0x7f86ac0aea50 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f86a4001800 tx=0x7f869c0091d0).stop 2022-06-29T16:05:01.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.934+0000 7f86ab59e700 1 -- 172.21.15.60:0/3075649424 shutdown_connections 2022-06-29T16:05:01.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.934+0000 7f86ab59e700 1 --2- 172.21.15.60:0/3075649424 >> v2:172.21.15.92:3300/0 conn(0x7f86ac0aef40 0x7f86ac0b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.934+0000 7f86ab59e700 1 --2- 172.21.15.60:0/3075649424 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f86ac05fb00 0x7f86ac0aea50 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.934+0000 7f86ab59e700 1 -- 172.21.15.60:0/3075649424 >> 172.21.15.60:0/3075649424 conn(0x7f86ac063a60 msgr2=0x7f86ac05f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:01.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.934+0000 7f86ab59e700 1 -- 172.21.15.60:0/3075649424 shutdown_connections 2022-06-29T16:05:01.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.934+0000 7f86ab59e700 1 -- 172.21.15.60:0/3075649424 wait complete. 2022-06-29T16:05:01.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.934+0000 7f86ab59e700 1 Processor -- start 2022-06-29T16:05:01.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.934+0000 7f86ab59e700 1 -- start start 2022-06-29T16:05:01.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.934+0000 7f86ab59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86ac05fb00 0x7f86ac064cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:01.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.934+0000 7f86ab59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f86ac0aef40 0x7f86ac065560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:01.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.934+0000 7f86ab59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f86ac0b1650 0x7f86ac065a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:01.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.934+0000 7f86ab59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac14d070 con 0x7f86ac05fb00 2022-06-29T16:05:01.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.934+0000 7f86ab59e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac14d310 con 0x7f86ac0b1650 2022-06-29T16:05:01.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.934+0000 7f86ab59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac10f5e0 con 0x7f86ac0aef40 2022-06-29T16:05:01.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.934+0000 7f86aa59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86ac05fb00 0x7f86ac064cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:01.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.934+0000 7f86aa59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86ac05fb00 0x7f86ac064cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49668/0 (socket says 172.21.15.60:49668) 2022-06-29T16:05:01.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.934+0000 7f86aa59c700 1 -- 172.21.15.60:0/3318156743 learned_addr learned my addr 172.21.15.60:0/3318156743 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:01.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.934+0000 7f86a9d9b700 1 --2- 172.21.15.60:0/3318156743 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f86ac0aef40 0x7f86ac065560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:01.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.934+0000 7f86aad9d700 1 --2- 172.21.15.60:0/3318156743 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f86ac0b1650 0x7f86ac065a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:01.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.938+0000 7f86aa59c700 1 -- 172.21.15.60:0/3318156743 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f86ac0aef40 msgr2=0x7f86ac065560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:01.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.938+0000 7f86aa59c700 1 --2- 172.21.15.60:0/3318156743 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f86ac0aef40 0x7f86ac065560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.938+0000 7f86aa59c700 1 -- 172.21.15.60:0/3318156743 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f86ac0b1650 msgr2=0x7f86ac065a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:01.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.938+0000 7f86aa59c700 1 --2- 172.21.15.60:0/3318156743 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f86ac0b1650 0x7f86ac065a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:01.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.938+0000 7f86aa59c700 1 -- 172.21.15.60:0/3318156743 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f869c009470 con 0x7f86ac05fb00 2022-06-29T16:05:01.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.938+0000 7f86aad9d700 1 --2- 172.21.15.60:0/3318156743 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f86ac0b1650 0x7f86ac065a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:01.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.938+0000 7f86aa59c700 1 --2- 172.21.15.60:0/3318156743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86ac05fb00 0x7f86ac064cd0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 rx=0x7f86a4001800 tx=0x7f869c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:01.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.938+0000 7f86a37fe700 1 -- 172.21.15.60:0/3318156743 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f869c00f680 con 0x7f86ac05fb00 2022-06-29T16:05:01.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.938+0000 7f86a37fe700 1 -- 172.21.15.60:0/3318156743 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f869c0032f0 con 0x7f86ac05fb00 2022-06-29T16:05:01.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.938+0000 7f86ab59e700 1 -- 172.21.15.60:0/3318156743 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86ac10f880 con 0x7f86ac05fb00 2022-06-29T16:05:01.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.938+0000 7f86a37fe700 1 -- 172.21.15.60:0/3318156743 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f869c0107a0 con 0x7f86ac05fb00 2022-06-29T16:05:01.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.938+0000 7f86ab59e700 1 -- 172.21.15.60:0/3318156743 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86ac064180 con 0x7f86ac05fb00 2022-06-29T16:05:01.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.938+0000 7f86ab59e700 1 -- 172.21.15.60:0/3318156743 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86ac0573d0 con 0x7f86ac05fb00 2022-06-29T16:05:01.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.938+0000 7f86a37fe700 1 -- 172.21.15.60:0/3318156743 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f869c028070 con 0x7f86ac05fb00 2022-06-29T16:05:01.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.938+0000 7f86a37fe700 1 --2- 172.21.15.60:0/3318156743 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8690053b00 0x7f8690055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:01.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.938+0000 7f86a37fe700 1 -- 172.21.15.60:0/3318156743 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6116+0+0 (secure 0 0 0) 0x7f869c06cd80 con 0x7f86ac05fb00 2022-06-29T16:05:01.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.938+0000 7f86a9d9b700 1 --2- 172.21.15.60:0/3318156743 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8690053b00 0x7f8690055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:01.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.942+0000 7f86a9d9b700 1 --2- 172.21.15.60:0/3318156743 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8690053b00 0x7f8690055fc0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f869400b000 tx=0x7f8694006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:01.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:01.946+0000 7f86a37fe700 1 -- 172.21.15.60:0/3318156743 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f869c043c70 con 0x7f86ac05fb00 2022-06-29T16:05:02.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:02.110+0000 7f86ab59e700 1 -- 172.21.15.60:0/3318156743 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86ac0af5d0 con 0x7f86ac05fb00 2022-06-29T16:05:02.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:02.110+0000 7f86a37fe700 1 -- 172.21.15.60:0/3318156743 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f869c06e020 con 0x7f86ac05fb00 2022-06-29T16:05:02.118 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:02.118 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":30,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5690,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1158969084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1158969084},{"type":"v1","addr":"172.21.15.60:6833","nonce":1158969084}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":28}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:04:08.275039+0000","modified":"2022-06-29T16:04:09.279046+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6304":{"gid":6304,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/953938586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":953938586},{"type":"v1","addr":"172.21.15.92:6835","nonce":953938586}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-06-29T16:05:02.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:02.110+0000 7f86ab59e700 1 -- 172.21.15.60:0/3318156743 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8690053b00 msgr2=0x7f8690055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:02.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:02.110+0000 7f86ab59e700 1 --2- 172.21.15.60:0/3318156743 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8690053b00 0x7f8690055fc0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f869400b000 tx=0x7f8694006df0).stop 2022-06-29T16:05:02.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:02.110+0000 7f86ab59e700 1 -- 172.21.15.60:0/3318156743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86ac05fb00 msgr2=0x7f86ac064cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:02.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:02.110+0000 7f86ab59e700 1 --2- 172.21.15.60:0/3318156743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86ac05fb00 0x7f86ac064cd0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 rx=0x7f86a4001800 tx=0x7f869c016000).stop 2022-06-29T16:05:02.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:02.110+0000 7f86ab59e700 1 -- 172.21.15.60:0/3318156743 shutdown_connections 2022-06-29T16:05:02.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:02.110+0000 7f86ab59e700 1 --2- 172.21.15.60:0/3318156743 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8690053b00 0x7f8690055fc0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:02.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:02.110+0000 7f86ab59e700 1 --2- 172.21.15.60:0/3318156743 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f86ac0b1650 0x7f86ac065a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:02.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:02.110+0000 7f86ab59e700 1 --2- 172.21.15.60:0/3318156743 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f86ac0aef40 0x7f86ac065560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:02.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:02.110+0000 7f86ab59e700 1 --2- 172.21.15.60:0/3318156743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86ac05fb00 0x7f86ac064cd0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:02.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:02.110+0000 7f86ab59e700 1 -- 172.21.15.60:0/3318156743 >> 172.21.15.60:0/3318156743 conn(0x7f86ac063a60 msgr2=0x7f86ac0b5570 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:02.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:02.110+0000 7f86ab59e700 1 -- 172.21.15.60:0/3318156743 shutdown_connections 2022-06-29T16:05:02.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:02.114+0000 7f86ab59e700 1 -- 172.21.15.60:0/3318156743 wait complete. 2022-06-29T16:05:02.123 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 30 2022-06-29T16:05:02.331 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:04:08.275039+0000', 'modified': '2022-06-29T16:04:09.279046+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6304}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6304': {'gid': 6304, 'name': 'b', 'rank': 0, 'incarnation': 29, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.92:6835/953938586', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 953938586}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 953938586}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:05:02.332 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:05:02.333 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-06-29T16:05:02.447 INFO:teuthology.orchestra.run.smithi092.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":6304,"want_state":"up:active","state":"up:active","rank_uptime":54.151551535999999,"mdsmap_epoch":30,"osdmap_epoch":72,"osdmap_epoch_barrier":72,"uptime":58.532674356999998} 2022-06-29T16:05:02.448 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 6304, "mdsmap_epoch": 30, "osdmap_epoch": 72, "osdmap_epoch_barrier": 72, "rank_uptime": 54.151551536, "state": "up:active", "uptime": 58.532674357, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:05:02.448 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-06-29T16:05:02.449 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-06-29T16:05:02.449 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi121.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-06-29T16:05:02.449 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:02.450 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:05:02.458 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:05:02.519 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:05:02.519 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:05:02.534 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:05:02.535 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:05:02.536 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:02.537 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:05:02.543 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-06-29T16:05:02.543 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:05:02.546 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:05:02.561 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:05:02.562 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:05:02.570 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:05:02.571 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:05:02.572 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:02.573 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:05:02.623 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:2022-06-29T16:05:02.619+0000 7fec9d754080 -1 init, newargv = 0x56308a3111d0 newargc=9ceph-fuse[12965]: starting ceph client 2022-06-29T16:05:02.623 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr: 2022-06-29T16:05:02.631 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[12965]: starting fuse 2022-06-29T16:05:03.721 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:05:03.740 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:05:03.741 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:05:03.751 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:05:03.752 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:05:03.753 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:03.753 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:05:03.800 INFO:teuthology.orchestra.run.smithi121.stdout:52 2022-06-29T16:05:03.801 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-06-29T16:05:03.801 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:03.802 DEBUG:teuthology.orchestra.run.smithi121:> sudo python3 -c ' 2022-06-29T16:05:03.802 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2022-06-29T16:05:03.802 DEBUG:teuthology.orchestra.run.smithi121:> import re 2022-06-29T16:05:03.803 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-06-29T16:05:03.803 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2022-06-29T16:05:03.804 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:05:03.804 DEBUG:teuthology.orchestra.run.smithi121:> def find_socket(client_name): 2022-06-29T16:05:03.804 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-06-29T16:05:03.805 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2022-06-29T16:05:03.805 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:05:03.806 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2022-06-29T16:05:03.806 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2022-06-29T16:05:03.807 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2022-06-29T16:05:03.807 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2022-06-29T16:05:03.808 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:05:03.808 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2022-06-29T16:05:03.808 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-06-29T16:05:03.809 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2022-06-29T16:05:03.809 DEBUG:teuthology.orchestra.run.smithi121:> return f 2022-06-29T16:05:03.810 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-06-29T16:05:03.810 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:05:03.810 DEBUG:teuthology.orchestra.run.smithi121:> print(find_socket("client.0")) 2022-06-29T16:05:03.811 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-06-29T16:05:03.857 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.0.12965.asok 2022-06-29T16:05:03.861 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.12965.asok 2022-06-29T16:05:03.862 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:03.862 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.12965.asok status 2022-06-29T16:05:03.952 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2022-06-29T16:05:03.953 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2022-06-29T16:05:03.953 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "f25ffac72e5a9e63860ff7deb6767a11f0559a16", 2022-06-29T16:05:03.953 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 15.2.16-304-gf25ffac7 (f25ffac72e5a9e63860ff7deb6767a11f0559a16) octopus (stable)", 2022-06-29T16:05:03.953 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "0", 2022-06-29T16:05:03.954 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2022-06-29T16:05:03.954 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-06-29T16:05:03.954 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "12965", 2022-06-29T16:05:03.954 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2022-06-29T16:05:03.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:05:03.955 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2022-06-29T16:05:03.955 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2022-06-29T16:05:03.955 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 6592, 2022-06-29T16:05:03.956 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2022-06-29T16:05:03.956 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2022-06-29T16:05:03.956 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2022-06-29T16:05:03.956 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 6592 2022-06-29T16:05:03.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:05:03.957 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-06-29T16:05:03.957 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-06-29T16:05:03.957 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-06-29T16:05:03.957 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 200664724 2022-06-29T16:05:03.958 INFO:teuthology.orchestra.run.smithi121.stdout: } 2022-06-29T16:05:03.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:05:03.958 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-06-29T16:05:03.958 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-06-29T16:05:03.959 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-06-29T16:05:03.959 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 200664724 2022-06-29T16:05:03.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:05:03.959 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.6592 172.21.15.121:0/200664724", 2022-06-29T16:05:03.959 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "172.21.15.121:0/200664724", 2022-06-29T16:05:03.960 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2022-06-29T16:05:03.960 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 30, 2022-06-29T16:05:03.960 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 72, 2022-06-29T16:05:03.960 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2022-06-29T16:05:03.961 INFO:teuthology.orchestra.run.smithi121.stdout: "blacklisted": false, 2022-06-29T16:05:03.961 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2022-06-29T16:05:03.961 INFO:teuthology.orchestra.run.smithi121.stdout:} 2022-06-29T16:05:03.966 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:03.966 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:05:03.967 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:05:03.975 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T16:05:03.976 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T16:05:03.976 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:03.977 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:05:03.996 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-06-29T16:05:03.997 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:05:04.019 DEBUG:teuthology.orchestra.run.smithi092:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:05:04.041 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:05:04.062 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:05:04.083 DEBUG:teuthology.orchestra.run.smithi197:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:05:04.104 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-06-29T16:05:04.105 INFO:tasks.cephfs.mount:Terminating background process 2022-06-29T16:05:04.105 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:04.106 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:05:04.106 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:05:04.114 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T16:05:04.115 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T16:05:04.116 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2022-06-29T16:05:04.116 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-06-29T16:05:04.117 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:05:04.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[12965]: fuse finished with error 0 and tester_r 0 2022-06-29T16:05:04.133 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:04.134 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:05:04.134 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:05:04.180 INFO:teuthology.orchestra.run.smithi121.stdout:ext2/ext3 2022-06-29T16:05:04.181 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-06-29T16:05:04.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:05:04.218 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:05:04.225 INFO:tasks.cephfs.mount:Terminating background process 2022-06-29T16:05:04.226 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:04.226 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:05:04.227 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:05:04.235 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2022-06-29T16:05:04.236 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-06-29T16:05:04.237 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi156.front.sepia.ceph.com... 2022-06-29T16:05:04.237 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-06-29T16:05:04.238 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:05:04.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[12384]: fuse finished with error 0 and tester_r 0 2022-06-29T16:05:04.295 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:04.296 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:05:04.296 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:05:04.302 INFO:teuthology.orchestra.run.smithi156.stdout:ext2/ext3 2022-06-29T16:05:04.303 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-06-29T16:05:04.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:05:04.422 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:05:04.477 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:04.478 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:05:04.478 DEBUG:teuthology.orchestra.run.smithi197:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T16:05:04.486 INFO:teuthology.orchestra.run.smithi197.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-06-29T16:05:04.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:05:04.487 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-06-29T16:05:04.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:05:04.488 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T16:05:04.535 INFO:teuthology.orchestra.run.smithi197.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-06-29T16:05:04.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:05:04.537 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-06-29T16:05:04.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.642+0000 7f062759e700 1 Processor -- start 2022-06-29T16:05:04.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.642+0000 7f062759e700 1 -- start start 2022-06-29T16:05:04.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.642+0000 7f062759e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f06280b48b0 0x7f0628142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:04.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.642+0000 7f062759e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06280b2c50 0x7f06280b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:04.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.642+0000 7f062759e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06280b0cf0 con 0x7f06280b5eb0 2022-06-29T16:05:04.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.642+0000 7f062759e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06280b54e0 con 0x7f06280b48b0 2022-06-29T16:05:04.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.642+0000 7f062759e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06280b4d10 con 0x7f06280b2c50 2022-06-29T16:05:04.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.642+0000 7f0625d9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f06280b48b0 0x7f0628142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:04.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.642+0000 7f062659c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f06280b5eb0 0x7f06280b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57762/0 (socket says 172.21.15.60:57762) 2022-06-29T16:05:04.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.642+0000 7f062659c700 1 -- 172.21.15.60:0/423607320 learned_addr learned my addr 172.21.15.60:0/423607320 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:04.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.646+0000 7f0626d9d700 1 --2- 172.21.15.60:0/423607320 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06280b2c50 0x7f06280b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:04.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.646+0000 7f0625d9b700 1 -- 172.21.15.60:0/423607320 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06280b2c50 msgr2=0x7f06280b30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:04.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.646+0000 7f0625d9b700 1 --2- 172.21.15.60:0/423607320 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06280b2c50 0x7f06280b30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:04.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.646+0000 7f0625d9b700 1 -- 172.21.15.60:0/423607320 >> v1:172.21.15.60:6789/0 conn(0x7f06280b5eb0 legacy=0x7f06280b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:04.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.646+0000 7f062559a700 1 -- 172.21.15.60:0/423607320 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3279720274 0 0) 0x7f061c001110 con 0x7f06280b5eb0 2022-06-29T16:05:04.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.646+0000 7f0625d9b700 1 -- 172.21.15.60:0/423607320 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06280b6290 con 0x7f06280b48b0 2022-06-29T16:05:04.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.646+0000 7f0625d9b700 1 --2- 172.21.15.60:0/423607320 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06280b48b0 0x7f0628142a20 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 rx=0x7f0618002f70 tx=0x7f0618006df0).ready entity=mon.0 client_cookie=e84d5f65be324442 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:04.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.646+0000 7f062559a700 1 -- 172.21.15.60:0/423607320 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f061800d570 con 0x7f06280b48b0 2022-06-29T16:05:04.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.646+0000 7f062559a700 1 -- 172.21.15.60:0/423607320 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0618000e90 con 0x7f06280b48b0 2022-06-29T16:05:04.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.646+0000 7f062559a700 1 -- 172.21.15.60:0/423607320 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f061800d8c0 con 0x7f06280b48b0 2022-06-29T16:05:04.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.646+0000 7f062759e700 1 -- 172.21.15.60:0/423607320 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06280b48b0 msgr2=0x7f0628142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:04.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.646+0000 7f062759e700 1 --2- 172.21.15.60:0/423607320 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06280b48b0 0x7f0628142a20 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 rx=0x7f0618002f70 tx=0x7f0618006df0).stop 2022-06-29T16:05:04.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.646+0000 7f062759e700 1 -- 172.21.15.60:0/423607320 shutdown_connections 2022-06-29T16:05:04.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.646+0000 7f062759e700 1 --2- 172.21.15.60:0/423607320 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06280b2c50 0x7f06280b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:04.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.646+0000 7f062759e700 1 --2- 172.21.15.60:0/423607320 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06280b48b0 0x7f0628142a20 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:04.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.646+0000 7f062759e700 1 -- 172.21.15.60:0/423607320 >> 172.21.15.60:0/423607320 conn(0x7f0628062090 msgr2=0x7f0628062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:04.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.646+0000 7f062759e700 1 -- 172.21.15.60:0/423607320 shutdown_connections 2022-06-29T16:05:04.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.646+0000 7f062759e700 1 -- 172.21.15.60:0/423607320 wait complete. 2022-06-29T16:05:04.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.646+0000 7f062759e700 1 Processor -- start 2022-06-29T16:05:04.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.646+0000 7f062759e700 1 -- start start 2022-06-29T16:05:04.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.650+0000 7f062759e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06280b2c50 0x7f0628145c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:04.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.650+0000 7f062759e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06280b48b0 0x7f06280af300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:04.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.650+0000 7f062759e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06280b5eb0 0x7f0628140e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:04.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.650+0000 7f062759e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06280ae260 con 0x7f06280b48b0 2022-06-29T16:05:04.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.650+0000 7f062759e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06280add50 con 0x7f06280b2c50 2022-06-29T16:05:04.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.650+0000 7f062759e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06280adec0 con 0x7f06280b5eb0 2022-06-29T16:05:04.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.650+0000 7f0625d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06280b48b0 0x7f06280af300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:04.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.650+0000 7f0625d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06280b48b0 0x7f06280af300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49684/0 (socket says 172.21.15.60:49684) 2022-06-29T16:05:04.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.650+0000 7f0625d9b700 1 -- 172.21.15.60:0/3065252276 learned_addr learned my addr 172.21.15.60:0/3065252276 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:04.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.650+0000 7f062659c700 1 --2- 172.21.15.60:0/3065252276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06280b2c50 0x7f0628145c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:04.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.650+0000 7f0626d9d700 1 --2- 172.21.15.60:0/3065252276 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06280b5eb0 0x7f0628140e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:04.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.650+0000 7f0625d9b700 1 -- 172.21.15.60:0/3065252276 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06280b5eb0 msgr2=0x7f0628140e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:04.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.650+0000 7f0625d9b700 1 --2- 172.21.15.60:0/3065252276 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06280b5eb0 0x7f0628140e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:04.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.650+0000 7f0625d9b700 1 -- 172.21.15.60:0/3065252276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06280b2c50 msgr2=0x7f0628145c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:04.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.650+0000 7f0625d9b700 1 --2- 172.21.15.60:0/3065252276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06280b2c50 0x7f0628145c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:04.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.650+0000 7f0625d9b700 1 -- 172.21.15.60:0/3065252276 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f061800b180 con 0x7f06280b48b0 2022-06-29T16:05:04.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.650+0000 7f062659c700 1 --2- 172.21.15.60:0/3065252276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06280b2c50 0x7f0628145c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:04.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.650+0000 7f0625d9b700 1 --2- 172.21.15.60:0/3065252276 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06280b48b0 0x7f06280af300 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0x7f0628141ba0 tx=0x7f0618014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:04.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.650+0000 7f06177fe700 1 -- 172.21.15.60:0/3065252276 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f061800dea0 con 0x7f06280b48b0 2022-06-29T16:05:04.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.650+0000 7f06177fe700 1 -- 172.21.15.60:0/3065252276 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f061801e400 con 0x7f06280b48b0 2022-06-29T16:05:04.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.650+0000 7f06177fe700 1 -- 172.21.15.60:0/3065252276 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f061801e760 con 0x7f06280b48b0 2022-06-29T16:05:04.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.650+0000 7f062759e700 1 -- 172.21.15.60:0/3065252276 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f062813fb70 con 0x7f06280b48b0 2022-06-29T16:05:04.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.650+0000 7f062759e700 1 -- 172.21.15.60:0/3065252276 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0628146de0 con 0x7f06280b48b0 2022-06-29T16:05:04.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.654+0000 7f062759e700 1 -- 172.21.15.60:0/3065252276 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06280ae420 con 0x7f06280b48b0 2022-06-29T16:05:04.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.654+0000 7f06177fe700 1 -- 172.21.15.60:0/3065252276 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f061801c070 con 0x7f06280b48b0 2022-06-29T16:05:04.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.654+0000 7f06177fe700 1 --2- 172.21.15.60:0/3065252276 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0604053a70 0x7f0604055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:04.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.654+0000 7f06177fe700 1 -- 172.21.15.60:0/3065252276 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6116+0+0 (secure 0 0 0) 0x7f061806cb40 con 0x7f06280b48b0 2022-06-29T16:05:04.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.654+0000 7f062659c700 1 --2- 172.21.15.60:0/3065252276 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0604053a70 0x7f0604055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:04.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.654+0000 7f062659c700 1 --2- 172.21.15.60:0/3065252276 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0604053a70 0x7f0604055f30 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f061c00b000 tx=0x7f061c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:04.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.662+0000 7f06177fe700 1 -- 172.21.15.60:0/3065252276 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0618043960 con 0x7f06280b48b0 2022-06-29T16:05:04.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.786+0000 7f062759e700 1 -- 172.21.15.60:0/3065252276 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f06280ae870 con 0x7f06280b48b0 2022-06-29T16:05:04.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.786+0000 7f06177fe700 1 -- 172.21.15.60:0/3065252276 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v72) v1 ==== 81+0+33725 (secure 0 0 0) 0x7f0618023070 con 0x7f06280b48b0 2022-06-29T16:05:04.797 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:04.798 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:05:04.798 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 72, 2022-06-29T16:05:04.799 INFO:teuthology.orchestra.run.smithi060.stdout: "fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", 2022-06-29T16:05:04.799 INFO:teuthology.orchestra.run.smithi060.stdout: "created": "2022-06-29T15:58:07.965719+0000", 2022-06-29T16:05:04.800 INFO:teuthology.orchestra.run.smithi060.stdout: "modified": "2022-06-29T16:04:54.882383+0000", 2022-06-29T16:05:04.800 INFO:teuthology.orchestra.run.smithi060.stdout: "last_up_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:05:04.801 INFO:teuthology.orchestra.run.smithi060.stdout: "last_in_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:05:04.801 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-06-29T16:05:04.801 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_num": 5799936, 2022-06-29T16:05:04.802 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_set": [ 2022-06-29T16:05:04.802 INFO:teuthology.orchestra.run.smithi060.stdout: "pglog_hardlimit", 2022-06-29T16:05:04.803 INFO:teuthology.orchestra.run.smithi060.stdout: "purged_snapdirs", 2022-06-29T16:05:04.803 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_deletes", 2022-06-29T16:05:04.804 INFO:teuthology.orchestra.run.smithi060.stdout: "sortbitwise" 2022-06-29T16:05:04.804 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:05:04.805 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_version": 4, 2022-06-29T16:05:04.805 INFO:teuthology.orchestra.run.smithi060.stdout: "full_ratio": 0.94999998807907104, 2022-06-29T16:05:04.806 INFO:teuthology.orchestra.run.smithi060.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-06-29T16:05:04.806 INFO:teuthology.orchestra.run.smithi060.stdout: "nearfull_ratio": 0.85000002384185791, 2022-06-29T16:05:04.807 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_snapshot": "", 2022-06-29T16:05:04.807 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_max": 14, 2022-06-29T16:05:04.808 INFO:teuthology.orchestra.run.smithi060.stdout: "max_osd": 8, 2022-06-29T16:05:04.808 INFO:teuthology.orchestra.run.smithi060.stdout: "require_min_compat_client": "jewel", 2022-06-29T16:05:04.809 INFO:teuthology.orchestra.run.smithi060.stdout: "min_compat_client": "jewel", 2022-06-29T16:05:04.809 INFO:teuthology.orchestra.run.smithi060.stdout: "require_osd_release": "octopus", 2022-06-29T16:05:04.810 INFO:teuthology.orchestra.run.smithi060.stdout: "pools": [ 2022-06-29T16:05:04.810 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.811 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2022-06-29T16:05:04.811 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "device_health_metrics", 2022-06-29T16:05:04.812 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:10.961127+0000", 2022-06-29T16:05:04.812 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:05:04.813 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:05:04.813 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:05:04.813 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:05:04.814 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:05:04.814 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:05:04.815 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:05:04.815 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:05:04.816 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 1, 2022-06-29T16:05:04.816 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 1, 2022-06-29T16:05:04.817 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 1, 2022-06-29T16:05:04.817 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 1, 2022-06-29T16:05:04.818 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 1, 2022-06-29T16:05:04.818 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:05:04.819 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:05:04.819 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:05:04.820 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:05:04.820 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:05:04.821 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:05:04.821 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:05:04.822 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.822 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "15", 2022-06-29T16:05:04.823 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:05:04.823 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:05:04.823 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:05:04.824 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:05:04.824 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:05:04.825 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:05:04.825 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:05:04.826 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:05:04.826 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:05:04.827 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:05:04.827 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:05:04.828 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:05:04.828 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:05:04.828 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:05:04.828 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:05:04.829 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:05:04.829 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:05:04.829 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:05:04.829 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:05:04.830 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:05:04.830 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:05:04.830 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:05:04.830 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:05:04.831 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:05:04.831 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:05:04.831 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:05:04.831 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.832 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:05:04.832 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:05:04.832 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:05:04.832 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:05:04.833 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:05:04.833 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:05:04.833 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:05:04.833 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:05:04.834 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:05:04.834 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:05:04.834 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:05:04.834 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:05:04.835 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 1 2022-06-29T16:05:04.835 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.835 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:05:04.835 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_devicehealth": {} 2022-06-29T16:05:04.835 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:04.836 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.836 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.836 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 2, 2022-06-29T16:05:04.836 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "rbd", 2022-06-29T16:05:04.837 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:19.184276+0000", 2022-06-29T16:05:04.837 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:05:04.837 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:05:04.837 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:05:04.838 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:05:04.838 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:05:04.838 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:05:04.838 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:05:04.839 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:05:04.839 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:05:04.839 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:05:04.839 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:05:04.840 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:05:04.840 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:05:04.840 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:05:04.840 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:05:04.841 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:05:04.841 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:05:04.841 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:05:04.841 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:05:04.841 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:05:04.842 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.842 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "18", 2022-06-29T16:05:04.842 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:05:04.842 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:05:04.843 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:05:04.843 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:05:04.843 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:05:04.843 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:05:04.844 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:05:04.844 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:05:04.844 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:05:04.844 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:05:04.845 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:05:04.845 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:05:04.845 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:05:04.845 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:05:04.846 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:05:04.846 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:05:04.846 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:05:04.846 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:05:04.847 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:05:04.847 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:05:04.847 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:05:04.847 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:05:04.848 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:05:04.848 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:05:04.848 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:05:04.848 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:05:04.849 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.849 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:05:04.849 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:05:04.849 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:05:04.849 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:05:04.850 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:05:04.850 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:05:04.850 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:05:04.851 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:05:04.851 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:05:04.851 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:05:04.851 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:05:04.852 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:05:04.852 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:05:04.852 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd": {} 2022-06-29T16:05:04.852 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:04.853 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.853 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.853 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 13, 2022-06-29T16:05:04.853 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "cephfs_metadata", 2022-06-29T16:05:04.853 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T16:04:06.708236+0000", 2022-06-29T16:05:04.854 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:05:04.854 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:05:04.854 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:05:04.854 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:05:04.855 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:05:04.855 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:05:04.855 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:05:04.856 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:05:04.856 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:05:04.856 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:05:04.856 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:05:04.857 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:05:04.857 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:05:04.857 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:05:04.857 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:05:04.858 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:05:04.858 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:05:04.858 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:05:04.858 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:05:04.859 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:05:04.859 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.859 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "70", 2022-06-29T16:05:04.859 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:05:04.860 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:05:04.860 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:05:04.860 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:05:04.861 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:05:04.861 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:05:04.861 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:05:04.861 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:05:04.862 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:05:04.862 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:05:04.862 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:05:04.863 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:05:04.863 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:05:04.863 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:05:04.863 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:05:04.864 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:05:04.864 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:05:04.864 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:05:04.864 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:05:04.865 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:05:04.865 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:05:04.865 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:05:04.865 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:05:04.865 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:05:04.866 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:05:04.866 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:05:04.866 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.866 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:05:04.867 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:05:04.867 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:05:04.867 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:05:04.867 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:05:04.868 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:05:04.868 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:05:04.868 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:05:04.868 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:05:04.869 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:05:04.869 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:05:04.869 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:05:04.869 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_bias": 4, 2022-06-29T16:05:04.870 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 16, 2022-06-29T16:05:04.870 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_priority": 5 2022-06-29T16:05:04.870 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.870 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:05:04.871 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs": { 2022-06-29T16:05:04.871 INFO:teuthology.orchestra.run.smithi060.stdout: "metadata": "cephfs" 2022-06-29T16:05:04.871 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:04.871 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:04.872 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.872 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.872 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 14, 2022-06-29T16:05:04.872 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "cephfs_data", 2022-06-29T16:05:04.872 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T16:04:07.305396+0000", 2022-06-29T16:05:04.873 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:05:04.873 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:05:04.873 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:05:04.873 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:05:04.874 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:05:04.874 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:05:04.874 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:05:04.874 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:05:04.875 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:05:04.875 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:05:04.875 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:05:04.875 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:05:04.876 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:05:04.876 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:05:04.876 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:05:04.876 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:05:04.877 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:05:04.877 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:05:04.877 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:05:04.877 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:05:04.877 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.878 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "71", 2022-06-29T16:05:04.878 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:05:04.878 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:05:04.879 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:05:04.879 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:05:04.879 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:05:04.879 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:05:04.880 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:05:04.880 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:05:04.880 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:05:04.880 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:05:04.881 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:05:04.881 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:05:04.881 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:05:04.882 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:05:04.882 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:05:04.882 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:05:04.882 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:05:04.883 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:05:04.883 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:05:04.883 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:05:04.883 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:05:04.883 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:05:04.884 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:05:04.884 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:05:04.884 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:05:04.884 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:05:04.885 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.885 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:05:04.885 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:05:04.885 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:05:04.886 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:05:04.886 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:05:04.886 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:05:04.887 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:05:04.887 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:05:04.887 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:05:04.887 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:05:04.888 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:05:04.888 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:05:04.888 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:05:04.888 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs": { 2022-06-29T16:05:04.889 INFO:teuthology.orchestra.run.smithi060.stdout: "data": "cephfs" 2022-06-29T16:05:04.889 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:04.889 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:04.889 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:04.890 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:05:04.890 INFO:teuthology.orchestra.run.smithi060.stdout: "osds": [ 2022-06-29T16:05:04.890 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.890 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:05:04.891 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "11e7ab9e-fce2-470f-8195-ea4c3c52df22", 2022-06-29T16:05:04.891 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:04.891 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:04.891 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:04.892 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:04.892 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:04.892 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:04.892 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:04.892 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 69, 2022-06-29T16:05:04.893 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:04.893 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:04.893 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:04.894 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:04.894 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.894 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:04.894 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6816", 2022-06-29T16:05:04.895 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:04.895 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.895 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.896 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:04.896 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6825", 2022-06-29T16:05:04.896 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:04.896 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:04.897 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:04.897 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.897 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:04.898 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:04.898 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.898 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:04.898 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6826", 2022-06-29T16:05:04.899 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:04.899 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.899 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.899 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:04.900 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6827", 2022-06-29T16:05:04.900 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:04.900 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:04.900 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:04.901 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.901 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:04.901 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:04.901 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.901 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:04.902 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6830", 2022-06-29T16:05:04.902 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:04.902 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.902 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.903 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:04.903 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6831", 2022-06-29T16:05:04.903 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:04.903 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:04.904 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:04.904 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.904 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:04.904 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:04.904 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.905 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:04.905 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6828", 2022-06-29T16:05:04.905 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:04.905 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.906 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.906 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:04.906 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6829", 2022-06-29T16:05:04.906 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:04.907 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:04.907 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:04.907 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.907 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6825/12890", 2022-06-29T16:05:04.908 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6827/12890", 2022-06-29T16:05:04.908 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6831/12890", 2022-06-29T16:05:04.908 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6829/12890", 2022-06-29T16:05:04.908 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:04.909 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:04.909 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:04.909 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:04.909 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.909 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.910 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:05:04.910 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "ca43f136-6f69-4431-a157-fbe073fb585c", 2022-06-29T16:05:04.910 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:04.910 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:04.911 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:04.911 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:04.911 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:04.912 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:04.912 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:04.912 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 59, 2022-06-29T16:05:04.913 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:04.913 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:04.913 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:04.913 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:04.914 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.914 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:04.914 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6815", 2022-06-29T16:05:04.914 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:04.915 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.915 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.915 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:04.915 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6818", 2022-06-29T16:05:04.916 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:04.916 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:04.916 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:04.916 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.917 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:04.917 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:04.917 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.918 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:04.918 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6819", 2022-06-29T16:05:04.918 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:04.918 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.919 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.919 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:04.920 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6820", 2022-06-29T16:05:04.920 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:04.920 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:04.920 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:04.921 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.921 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:04.921 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:04.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.922 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:04.922 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6823", 2022-06-29T16:05:04.923 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:04.923 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.923 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.923 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:04.924 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6824", 2022-06-29T16:05:04.924 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:04.924 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:04.924 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:04.924 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.925 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:04.925 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:04.925 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.925 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:04.926 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6821", 2022-06-29T16:05:04.926 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:04.926 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.926 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.927 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:04.927 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6822", 2022-06-29T16:05:04.927 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:04.927 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:04.928 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:04.928 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.928 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6818/12892", 2022-06-29T16:05:04.928 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6820/12892", 2022-06-29T16:05:04.929 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6824/12892", 2022-06-29T16:05:04.929 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6822/12892", 2022-06-29T16:05:04.929 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:04.929 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:04.930 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:04.930 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:04.930 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.930 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.931 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:05:04.931 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "74fac45f-5f07-44ec-a9cb-2644c123fd4a", 2022-06-29T16:05:04.931 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:04.931 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:04.932 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:04.932 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:04.932 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:04.932 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:04.932 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:04.933 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 69, 2022-06-29T16:05:04.933 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:04.933 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:04.933 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:04.934 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:04.934 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.934 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:04.934 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6808", 2022-06-29T16:05:04.935 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:04.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.935 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.935 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:04.936 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6809", 2022-06-29T16:05:04.936 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:04.936 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:04.936 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:04.936 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.937 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:04.937 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:04.937 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.937 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:04.938 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6810", 2022-06-29T16:05:04.938 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:04.938 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.938 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.939 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:04.939 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6811", 2022-06-29T16:05:04.939 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:04.939 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:04.940 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:04.940 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.940 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:04.940 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:04.941 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.941 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:04.941 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6814", 2022-06-29T16:05:04.941 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:04.941 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.942 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.942 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:04.942 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6817", 2022-06-29T16:05:04.943 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:04.943 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:04.943 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:04.944 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.944 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:04.944 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:04.944 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.945 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:04.945 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6812", 2022-06-29T16:05:04.945 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:04.946 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.946 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.946 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:04.947 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6813", 2022-06-29T16:05:04.947 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:04.947 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:04.947 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:04.948 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.948 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6809/12891", 2022-06-29T16:05:04.948 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6811/12891", 2022-06-29T16:05:04.948 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6817/12891", 2022-06-29T16:05:04.949 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6813/12891", 2022-06-29T16:05:04.949 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:04.949 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:04.949 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:04.949 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:04.950 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.950 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.950 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:05:04.951 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "2581d351-973a-4b4e-85ca-3658f0068765", 2022-06-29T16:05:04.951 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:04.952 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:04.952 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:04.952 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:04.953 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:04.953 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:04.953 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:04.953 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 69, 2022-06-29T16:05:04.953 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:04.954 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:04.954 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:04.954 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:04.954 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.955 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:04.955 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6800", 2022-06-29T16:05:04.955 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:04.956 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.956 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.956 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:04.956 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6801", 2022-06-29T16:05:04.957 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:04.957 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:04.957 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:04.957 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.958 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:04.958 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:04.958 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.958 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:04.959 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6802", 2022-06-29T16:05:04.959 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:04.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.960 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.960 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:04.961 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6803", 2022-06-29T16:05:04.961 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:04.962 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:04.962 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:04.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.963 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:04.964 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:04.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.964 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:04.965 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6806", 2022-06-29T16:05:04.965 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:04.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.967 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:04.967 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6807", 2022-06-29T16:05:04.968 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:04.968 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:04.968 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:04.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.969 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:04.969 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:04.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.970 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:04.970 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6804", 2022-06-29T16:05:04.970 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:04.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.971 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.971 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:04.971 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6805", 2022-06-29T16:05:04.971 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:04.972 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:04.972 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:04.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.972 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6801/12889", 2022-06-29T16:05:04.972 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6803/12889", 2022-06-29T16:05:04.973 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6807/12889", 2022-06-29T16:05:04.973 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6805/12889", 2022-06-29T16:05:04.973 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:04.973 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:04.974 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:04.974 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:04.974 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.975 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:05:04.975 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "3d234db9-87cd-4fe9-814f-0240a287d8b4", 2022-06-29T16:05:04.976 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:04.976 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:04.976 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:04.976 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:04.977 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:04.977 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:04.977 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:04.977 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 68, 2022-06-29T16:05:04.978 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:04.978 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:04.978 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:04.978 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:04.979 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.979 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:04.979 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6808", 2022-06-29T16:05:04.979 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:04.980 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.980 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:04.980 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6810", 2022-06-29T16:05:04.981 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:04.981 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:04.981 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:04.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.982 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:04.982 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:04.982 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.982 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:04.983 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6812", 2022-06-29T16:05:04.983 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:04.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.983 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:04.984 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6814", 2022-06-29T16:05:04.984 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:04.984 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:04.984 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:04.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.985 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:04.985 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:04.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.986 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:04.986 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6823", 2022-06-29T16:05:04.986 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:04.986 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.987 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.987 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:04.987 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6825", 2022-06-29T16:05:04.987 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:04.988 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:04.988 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:04.988 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.989 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:04.989 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:04.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.989 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:04.990 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6817", 2022-06-29T16:05:04.990 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:04.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.990 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.991 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:04.991 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6820", 2022-06-29T16:05:04.991 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:04.991 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:04.992 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:04.992 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.992 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6810/13249", 2022-06-29T16:05:04.992 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6814/13249", 2022-06-29T16:05:04.993 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6825/13249", 2022-06-29T16:05:04.993 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6820/13249", 2022-06-29T16:05:04.993 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:04.993 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:04.993 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:04.994 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:04.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.994 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:05:04.995 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "e4bf1104-ff5c-438b-b608-8cee31c163f6", 2022-06-29T16:05:04.995 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:04.995 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:04.995 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:04.996 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:04.996 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:04.996 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:04.996 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:04.997 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 69, 2022-06-29T16:05:04.997 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:04.997 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:04.997 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:04.998 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:04.998 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.998 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:04.998 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6809", 2022-06-29T16:05:04.998 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:04.999 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:04.999 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:04.999 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:04.999 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6811", 2022-06-29T16:05:05.000 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:05.000 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:05.000 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:05.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.001 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:05.001 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:05.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:05.001 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:05.001 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6813", 2022-06-29T16:05:05.002 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:05.002 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.002 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:05.002 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:05.003 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6815", 2022-06-29T16:05:05.003 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:05.003 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:05.003 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:05.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.004 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:05.004 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:05.004 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:05.004 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:05.005 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6824", 2022-06-29T16:05:05.005 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:05.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:05.006 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:05.006 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6827", 2022-06-29T16:05:05.006 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:05.006 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:05.007 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:05.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.007 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:05.007 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:05.008 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:05.008 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:05.008 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6818", 2022-06-29T16:05:05.008 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:05.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.009 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:05.009 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:05.009 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6821", 2022-06-29T16:05:05.010 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:05.010 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:05.010 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:05.010 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.011 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6811/13248", 2022-06-29T16:05:05.011 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6815/13248", 2022-06-29T16:05:05.011 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6827/13248", 2022-06-29T16:05:05.011 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6821/13248", 2022-06-29T16:05:05.011 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:05.012 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:05.012 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:05.012 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:05.012 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.013 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:05.013 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:05:05.013 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "8564dd25-27ab-4203-b54e-af6d3aa05857", 2022-06-29T16:05:05.013 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:05.014 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:05.014 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:05.014 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:05.014 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:05.015 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:05.015 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:05.015 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 69, 2022-06-29T16:05:05.015 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:05.016 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:05.016 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:05.016 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:05.016 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:05.017 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:05.017 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6816", 2022-06-29T16:05:05.017 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:05.017 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.018 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:05.018 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:05.018 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6819", 2022-06-29T16:05:05.019 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:05.019 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:05.019 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:05.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.020 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:05.020 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:05.020 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:05.020 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:05.021 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6822", 2022-06-29T16:05:05.021 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:05.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:05.022 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:05.022 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6826", 2022-06-29T16:05:05.022 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:05.022 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:05.023 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:05.023 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.023 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:05.023 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:05.023 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:05.024 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:05.024 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6830", 2022-06-29T16:05:05.024 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:05.024 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.025 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:05.025 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:05.025 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6831", 2022-06-29T16:05:05.025 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:05.026 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:05.026 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:05.026 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.026 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:05.027 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:05.027 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:05.027 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:05.027 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6828", 2022-06-29T16:05:05.028 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:05.028 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.028 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:05.028 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:05.029 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6829", 2022-06-29T16:05:05.029 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:05.029 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:05.029 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:05.030 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.030 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6819/13250", 2022-06-29T16:05:05.030 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6826/13250", 2022-06-29T16:05:05.030 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6831/13250", 2022-06-29T16:05:05.031 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6829/13250", 2022-06-29T16:05:05.031 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:05.031 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:05.032 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:05.032 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:05.032 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.032 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:05.032 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:05:05.033 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "bd513a9c-3b19-4314-9897-d300b5c897bb", 2022-06-29T16:05:05.033 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:05.033 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:05.033 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:05.034 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:05.034 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:05.034 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:05.034 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:05.034 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 68, 2022-06-29T16:05:05.035 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:05.035 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:05.035 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:05.035 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:05.036 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:05.036 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:05.036 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6800", 2022-06-29T16:05:05.036 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:05.036 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.037 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:05.037 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:05.037 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6801", 2022-06-29T16:05:05.037 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:05.038 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:05.038 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:05.038 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.038 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:05.039 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:05.039 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:05.039 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:05.039 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6802", 2022-06-29T16:05:05.040 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:05.040 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.040 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:05.040 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:05.041 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6803", 2022-06-29T16:05:05.041 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:05.041 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:05.041 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:05.041 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.042 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:05.042 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:05.042 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:05.042 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:05.043 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6806", 2022-06-29T16:05:05.043 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:05.043 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.043 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:05.044 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:05.044 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6807", 2022-06-29T16:05:05.044 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:05.044 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:05.044 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:05.045 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.045 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:05.045 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:05.045 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:05.046 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:05.046 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6804", 2022-06-29T16:05:05.046 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:05.046 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.047 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:05.047 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:05.047 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6805", 2022-06-29T16:05:05.047 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:05.048 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:05.048 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:05.048 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.048 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6801/13247", 2022-06-29T16:05:05.049 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6803/13247", 2022-06-29T16:05:05.049 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6807/13247", 2022-06-29T16:05:05.049 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6805/13247", 2022-06-29T16:05:05.049 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:05.050 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:05.050 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:05.050 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:05.050 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:05.051 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:05:05.051 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_xinfo": [ 2022-06-29T16:05:05.051 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:05.051 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:05:05.052 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:05.052 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:05.052 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:05.052 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:05.053 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:05.053 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.891342+0000", 2022-06-29T16:05:05.053 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:05.053 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.053 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:05.054 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:05:05.054 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:05.054 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:05.054 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:05.055 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:05.055 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:05.055 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.900866+0000", 2022-06-29T16:05:05.055 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:05.055 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.056 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:05.056 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:05:05.056 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:05.056 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:05.057 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:05.057 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:05.057 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:05.057 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.912755+0000", 2022-06-29T16:05:05.058 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:05.058 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.058 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:05.058 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:05:05.059 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:05.059 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:05.059 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:05.060 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:05.060 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:05.060 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.864332+0000", 2022-06-29T16:05:05.060 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:05.061 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.061 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:05.061 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:05:05.061 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:05.061 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:05.062 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:05.062 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:05.062 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:05.062 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.863756+0000", 2022-06-29T16:05:05.063 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:05.063 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.063 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:05.063 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:05:05.064 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:05.064 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:05.064 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:05.064 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:05.065 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:05.065 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.869041+0000", 2022-06-29T16:05:05.065 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:05.065 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.066 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:05.066 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:05:05.066 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:05.066 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:05.067 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:05.067 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:05.067 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:05.067 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.895043+0000", 2022-06-29T16:05:05.068 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:05.068 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.068 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:05.068 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:05:05.069 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:05.069 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:05.069 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:05.069 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:05.070 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:05.070 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.889491+0000", 2022-06-29T16:05:05.070 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:05.070 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:05.071 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:05:05.071 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap": [], 2022-06-29T16:05:05.071 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap_items": [], 2022-06-29T16:05:05.071 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_temp": [], 2022-06-29T16:05:05.072 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_temp": [], 2022-06-29T16:05:05.072 INFO:teuthology.orchestra.run.smithi060.stdout: "blacklist": { 2022-06-29T16:05:05.072 INFO:teuthology.orchestra.run.smithi060.stdout: "172.21.15.121:0/2990201791": "2022-06-29T17:04:53.908824+0000" 2022-06-29T16:05:05.072 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.073 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profiles": { 2022-06-29T16:05:05.073 INFO:teuthology.orchestra.run.smithi060.stdout: "default": { 2022-06-29T16:05:05.073 INFO:teuthology.orchestra.run.smithi060.stdout: "crush-failure-domain": "osd", 2022-06-29T16:05:05.073 INFO:teuthology.orchestra.run.smithi060.stdout: "k": "2", 2022-06-29T16:05:05.073 INFO:teuthology.orchestra.run.smithi060.stdout: "m": "1", 2022-06-29T16:05:05.074 INFO:teuthology.orchestra.run.smithi060.stdout: "plugin": "jerasure", 2022-06-29T16:05:05.074 INFO:teuthology.orchestra.run.smithi060.stdout: "ruleset-failure-domain": "osd", 2022-06-29T16:05:05.074 INFO:teuthology.orchestra.run.smithi060.stdout: "technique": "reed_sol_van" 2022-06-29T16:05:05.074 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:05.075 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:05.075 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps_queue": [], 2022-06-29T16:05:05.075 INFO:teuthology.orchestra.run.smithi060.stdout: "new_removed_snaps": [], 2022-06-29T16:05:05.075 INFO:teuthology.orchestra.run.smithi060.stdout: "new_purged_snaps": [], 2022-06-29T16:05:05.076 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_node_flags": {}, 2022-06-29T16:05:05.076 INFO:teuthology.orchestra.run.smithi060.stdout: "device_class_flags": {} 2022-06-29T16:05:05.076 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:05:05.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.790+0000 7f062759e700 1 -- 172.21.15.60:0/3065252276 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0604053a70 msgr2=0x7f0604055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:05.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.790+0000 7f062759e700 1 --2- 172.21.15.60:0/3065252276 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0604053a70 0x7f0604055f30 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f061c00b000 tx=0x7f061c009040).stop 2022-06-29T16:05:05.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.790+0000 7f062759e700 1 -- 172.21.15.60:0/3065252276 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06280b48b0 msgr2=0x7f06280af300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:05.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.790+0000 7f062759e700 1 --2- 172.21.15.60:0/3065252276 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06280b48b0 0x7f06280af300 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0x7f0628141ba0 tx=0x7f0618014000).stop 2022-06-29T16:05:05.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.790+0000 7f062759e700 1 -- 172.21.15.60:0/3065252276 shutdown_connections 2022-06-29T16:05:05.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.790+0000 7f062759e700 1 --2- 172.21.15.60:0/3065252276 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0604053a70 0x7f0604055f30 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:05.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.790+0000 7f062759e700 1 --2- 172.21.15.60:0/3065252276 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06280b5eb0 0x7f0628140e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:05.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.790+0000 7f062759e700 1 --2- 172.21.15.60:0/3065252276 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06280b48b0 0x7f06280af300 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:05.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.790+0000 7f062759e700 1 --2- 172.21.15.60:0/3065252276 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06280b2c50 0x7f0628145c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:05.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.790+0000 7f062759e700 1 -- 172.21.15.60:0/3065252276 >> 172.21.15.60:0/3065252276 conn(0x7f0628062090 msgr2=0x7f0628143290 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:05.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.790+0000 7f062759e700 1 -- 172.21.15.60:0/3065252276 shutdown_connections 2022-06-29T16:05:05.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:04.790+0000 7f062759e700 1 -- 172.21.15.60:0/3065252276 wait complete. 2022-06-29T16:05:05.082 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:05:05.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.186+0000 7fa8c8d45700 1 Processor -- start 2022-06-29T16:05:05.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.186+0000 7fa8c8d45700 1 -- start start 2022-06-29T16:05:05.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.190+0000 7fa8c8d45700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa8c40b2130 0x7fa8c4146100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:05.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.190+0000 7fa8c8d45700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8c40aed00 0x7fa8c41465f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:05.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.190+0000 7fa8c8d45700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c405d450 con 0x7fa8c405d070 2022-06-29T16:05:05.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.190+0000 7fa8c8d45700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8c40adf90 con 0x7fa8c40b2130 2022-06-29T16:05:05.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.190+0000 7fa8c8d45700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c40b25d0 con 0x7fa8c40aed00 2022-06-29T16:05:05.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.190+0000 7fa8c2ffd700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa8c40b2130 0x7fa8c4146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:05.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.190+0000 7fa8c3fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8c40aed00 0x7fa8c41465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:05.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.190+0000 7fa8c37fe700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fa8c405d070 0x7fa8c414aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:33794/0 (socket says 172.21.15.60:33794) 2022-06-29T16:05:05.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.190+0000 7fa8c3fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8c40aed00 0x7fa8c41465f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:34556/0 (socket says 172.21.15.60:34556) 2022-06-29T16:05:05.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.190+0000 7fa8c37fe700 1 -- 172.21.15.60:0/1491736200 learned_addr learned my addr 172.21.15.60:0/1491736200 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:05.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.190+0000 7fa8c3fff700 1 -- 172.21.15.60:0/1491736200 >> v2:172.21.15.92:3300/0 conn(0x7fa8c40b2130 msgr2=0x7fa8c4146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:05.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.190+0000 7fa8c3fff700 1 --2- 172.21.15.60:0/1491736200 >> v2:172.21.15.92:3300/0 conn(0x7fa8c40b2130 0x7fa8c4146100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:05.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.190+0000 7fa8c3fff700 1 -- 172.21.15.60:0/1491736200 >> v1:172.21.15.92:6789/0 conn(0x7fa8c405d070 legacy=0x7fa8c414aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:05.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.190+0000 7fa8c3fff700 1 -- 172.21.15.60:0/1491736200 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c414a350 con 0x7fa8c40aed00 2022-06-29T16:05:05.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.190+0000 7fa8c3fff700 1 --2- 172.21.15.60:0/1491736200 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8c40aed00 0x7fa8c41465f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fa8bc0032f0 tx=0x7fa8bc0091d0).ready entity=mon.2 client_cookie=d277a0f18df382af server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:05.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.190+0000 7fa8c27fc700 1 -- 172.21.15.60:0/1491736200 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8bc00b6c0 con 0x7fa8c40aed00 2022-06-29T16:05:05.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.190+0000 7fa8c27fc700 1 -- 172.21.15.60:0/1491736200 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8bc00b820 con 0x7fa8c40aed00 2022-06-29T16:05:05.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.190+0000 7fa8c27fc700 1 -- 172.21.15.60:0/1491736200 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8bc00b990 con 0x7fa8c40aed00 2022-06-29T16:05:05.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.190+0000 7fa8c8d45700 1 -- 172.21.15.60:0/1491736200 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8c40aed00 msgr2=0x7fa8c41465f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:05.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.190+0000 7fa8c8d45700 1 --2- 172.21.15.60:0/1491736200 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8c40aed00 0x7fa8c41465f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fa8bc0032f0 tx=0x7fa8bc0091d0).stop 2022-06-29T16:05:05.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.190+0000 7fa8c8d45700 1 -- 172.21.15.60:0/1491736200 shutdown_connections 2022-06-29T16:05:05.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.190+0000 7fa8c8d45700 1 --2- 172.21.15.60:0/1491736200 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8c40aed00 0x7fa8c41465f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:05.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.190+0000 7fa8c8d45700 1 --2- 172.21.15.60:0/1491736200 >> v2:172.21.15.92:3300/0 conn(0x7fa8c40b2130 0x7fa8c4146100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:05.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.190+0000 7fa8c8d45700 1 -- 172.21.15.60:0/1491736200 >> 172.21.15.60:0/1491736200 conn(0x7fa8c40b5890 msgr2=0x7fa8c40b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:05.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.190+0000 7fa8c8d45700 1 -- 172.21.15.60:0/1491736200 shutdown_connections 2022-06-29T16:05:05.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.190+0000 7fa8c8d45700 1 -- 172.21.15.60:0/1491736200 wait complete. 2022-06-29T16:05:05.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.194+0000 7fa8c8d45700 1 Processor -- start 2022-06-29T16:05:05.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.194+0000 7fa8c8d45700 1 -- start start 2022-06-29T16:05:05.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.198+0000 7fa8c8d45700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8c405d070 0x7fa8c41471a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:05.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.198+0000 7fa8c8d45700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8c40aed00 0x7fa8c4066530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:05.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.198+0000 7fa8c8d45700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8c40b2130 0x7fa8c4066a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:05.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.198+0000 7fa8c8d45700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c414c760 con 0x7fa8c40aed00 2022-06-29T16:05:05.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.198+0000 7fa8c8d45700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c4149a00 con 0x7fa8c405d070 2022-06-29T16:05:05.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.198+0000 7fa8c8d45700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c4066f10 con 0x7fa8c40b2130 2022-06-29T16:05:05.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.198+0000 7fa8c2ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8c40aed00 0x7fa8c4066530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:05.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.198+0000 7fa8c2ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8c40aed00 0x7fa8c4066530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49698/0 (socket says 172.21.15.60:49698) 2022-06-29T16:05:05.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.198+0000 7fa8c2ffd700 1 -- 172.21.15.60:0/1019928693 learned_addr learned my addr 172.21.15.60:0/1019928693 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:05.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.198+0000 7fa8c37fe700 1 --2- 172.21.15.60:0/1019928693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8c405d070 0x7fa8c41471a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:05.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.198+0000 7fa8c3fff700 1 --2- 172.21.15.60:0/1019928693 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8c40b2130 0x7fa8c4066a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:05.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.198+0000 7fa8c2ffd700 1 -- 172.21.15.60:0/1019928693 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8c40b2130 msgr2=0x7fa8c4066a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:05.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.198+0000 7fa8c2ffd700 1 --2- 172.21.15.60:0/1019928693 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8c40b2130 0x7fa8c4066a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:05.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.198+0000 7fa8c2ffd700 1 -- 172.21.15.60:0/1019928693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8c405d070 msgr2=0x7fa8c41471a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:05.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.198+0000 7fa8c2ffd700 1 --2- 172.21.15.60:0/1019928693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8c405d070 0x7fa8c41471a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:05.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.198+0000 7fa8c2ffd700 1 -- 172.21.15.60:0/1019928693 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8bc009470 con 0x7fa8c40aed00 2022-06-29T16:05:05.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.198+0000 7fa8c2ffd700 1 --2- 172.21.15.60:0/1019928693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8c40aed00 0x7fa8c4066530 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 rx=0x7fa8c414d940 tx=0x7fa8b4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:05.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.198+0000 7fa8c0ff9700 1 -- 172.21.15.60:0/1019928693 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8b4010780 con 0x7fa8c40aed00 2022-06-29T16:05:05.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.198+0000 7fa8c0ff9700 1 -- 172.21.15.60:0/1019928693 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8b40108e0 con 0x7fa8c40aed00 2022-06-29T16:05:05.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.198+0000 7fa8c0ff9700 1 -- 172.21.15.60:0/1019928693 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8b4010c40 con 0x7fa8c40aed00 2022-06-29T16:05:05.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.198+0000 7fa8c8d45700 1 -- 172.21.15.60:0/1019928693 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c4149c30 con 0x7fa8c40aed00 2022-06-29T16:05:05.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.198+0000 7fa8c8d45700 1 -- 172.21.15.60:0/1019928693 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8c414b610 con 0x7fa8c40aed00 2022-06-29T16:05:05.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.198+0000 7fa8c8d45700 1 -- 172.21.15.60:0/1019928693 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8c414a0d0 con 0x7fa8c40aed00 2022-06-29T16:05:05.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.202+0000 7fa8c0ff9700 1 -- 172.21.15.60:0/1019928693 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa8b401c460 con 0x7fa8c40aed00 2022-06-29T16:05:05.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.202+0000 7fa8c0ff9700 1 --2- 172.21.15.60:0/1019928693 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa8a40539e0 0x7fa8a4055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:05.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.202+0000 7fa8c0ff9700 1 -- 172.21.15.60:0/1019928693 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6116+0+0 (secure 0 0 0) 0x7fa8b406c190 con 0x7fa8c40aed00 2022-06-29T16:05:05.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.202+0000 7fa8c37fe700 1 --2- 172.21.15.60:0/1019928693 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa8a40539e0 0x7fa8a4055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:05.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.202+0000 7fa8c37fe700 1 --2- 172.21.15.60:0/1019928693 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa8a40539e0 0x7fa8a4055ea0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fa8b800d000 tx=0x7fa8b8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:05.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.206+0000 7fa8c0ff9700 1 -- 172.21.15.60:0/1019928693 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa8b4043020 con 0x7fa8c40aed00 2022-06-29T16:05:05.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.374+0000 7fa8c8d45700 1 -- 172.21.15.60:0/1019928693 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8c414a5f0 con 0x7fa8c40aed00 2022-06-29T16:05:05.382 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.374+0000 7fa8c0ff9700 1 -- 172.21.15.60:0/1019928693 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2137 (secure 0 0 0) 0x7fa8b4046640 con 0x7fa8c40aed00 2022-06-29T16:05:05.382 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:05.383 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":30,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5690,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1158969084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1158969084},{"type":"v1","addr":"172.21.15.60:6833","nonce":1158969084}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":28}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:04:08.275039+0000","modified":"2022-06-29T16:04:09.279046+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6304},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6304":{"gid":6304,"name":"b","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/953938586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":953938586},{"type":"v1","addr":"172.21.15.92:6835","nonce":953938586}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-06-29T16:05:05.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.378+0000 7fa8c8d45700 1 -- 172.21.15.60:0/1019928693 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa8a40539e0 msgr2=0x7fa8a4055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:05.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.378+0000 7fa8c8d45700 1 --2- 172.21.15.60:0/1019928693 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa8a40539e0 0x7fa8a4055ea0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fa8b800d000 tx=0x7fa8b8009040).stop 2022-06-29T16:05:05.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.378+0000 7fa8c8d45700 1 -- 172.21.15.60:0/1019928693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8c40aed00 msgr2=0x7fa8c4066530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:05.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.378+0000 7fa8c8d45700 1 --2- 172.21.15.60:0/1019928693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8c40aed00 0x7fa8c4066530 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 rx=0x7fa8c414d940 tx=0x7fa8b4006df0).stop 2022-06-29T16:05:05.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.378+0000 7fa8c8d45700 1 -- 172.21.15.60:0/1019928693 shutdown_connections 2022-06-29T16:05:05.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.378+0000 7fa8c8d45700 1 --2- 172.21.15.60:0/1019928693 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa8a40539e0 0x7fa8a4055ea0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:05.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.378+0000 7fa8c8d45700 1 --2- 172.21.15.60:0/1019928693 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa8c40b2130 0x7fa8c4066a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:05.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.378+0000 7fa8c8d45700 1 --2- 172.21.15.60:0/1019928693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa8c40aed00 0x7fa8c4066530 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:05.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.378+0000 7fa8c8d45700 1 --2- 172.21.15.60:0/1019928693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8c405d070 0x7fa8c41471a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:05.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.378+0000 7fa8c8d45700 1 -- 172.21.15.60:0/1019928693 >> 172.21.15.60:0/1019928693 conn(0x7fa8c40b5890 msgr2=0x7fa8c4144510 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:05.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.378+0000 7fa8c8d45700 1 -- 172.21.15.60:0/1019928693 shutdown_connections 2022-06-29T16:05:05.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.378+0000 7fa8c8d45700 1 -- 172.21.15.60:0/1019928693 wait complete. 2022-06-29T16:05:05.388 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 30 2022-06-29T16:05:05.396 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-06-29T16:05:05.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa26489f700 1 Processor -- start 2022-06-29T16:05:05.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa26489f700 1 -- start start 2022-06-29T16:05:05.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa26489f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa2600aed00 0x7fa260146810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:05.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa26489f700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa26005d450 con 0x7fa2600b2130 2022-06-29T16:05:05.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa26489f700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2600b25d0 con 0x7fa26005d070 2022-06-29T16:05:05.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa26489f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2600adf90 con 0x7fa2600aed00 2022-06-29T16:05:05.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa25659c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa2600b2130 0x7fa260146100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57792/0 (socket says 172.21.15.60:57792) 2022-06-29T16:05:05.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa25659c700 1 -- 172.21.15.60:0/1503607254 learned_addr learned my addr 172.21.15.60:0/1503607254 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:05.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa25f59e700 1 --2- 172.21.15.60:0/1503607254 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa2600aed00 0x7fa260146810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:05.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa25e59c700 1 -- 172.21.15.60:0/1503607254 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3586432258 0 0) 0x7fa26005d450 con 0x7fa2600b2130 2022-06-29T16:05:05.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa25e59c700 1 -- 172.21.15.60:0/1503607254 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa240003260 con 0x7fa2600b2130 2022-06-29T16:05:05.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa25e59c700 1 -- 172.21.15.60:0/1503607254 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3634278584 0 0) 0x7fa2600b25d0 con 0x7fa26005d070 2022-06-29T16:05:05.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa25e59c700 1 -- 172.21.15.60:0/1503607254 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa26005d450 con 0x7fa26005d070 2022-06-29T16:05:05.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa25e59c700 1 -- 172.21.15.60:0/1503607254 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1179210199 0 0) 0x7fa240003260 con 0x7fa2600b2130 2022-06-29T16:05:05.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa25e59c700 1 -- 172.21.15.60:0/1503607254 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa2600aed00 msgr2=0x7fa260146810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:05.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa25e59c700 1 --2- 172.21.15.60:0/1503607254 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa2600aed00 0x7fa260146810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:05.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa25e59c700 1 -- 172.21.15.60:0/1503607254 >> v1:172.21.15.92:6789/0 conn(0x7fa26005d070 legacy=0x7fa26014aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:05.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa25e59c700 1 -- 172.21.15.60:0/1503607254 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa26014a350 con 0x7fa2600b2130 2022-06-29T16:05:05.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa25e59c700 1 -- 172.21.15.60:0/1503607254 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7fa248001680 con 0x7fa2600b2130 2022-06-29T16:05:05.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa25e59c700 1 -- 172.21.15.60:0/1503607254 >> v1:172.21.15.60:6789/0 conn(0x7fa2600b2130 legacy=0x7fa260146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:05.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa25e59c700 1 --2- 172.21.15.60:0/1503607254 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa240003910 0x7fa240003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:05.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa25e59c700 1 --2- 172.21.15.60:0/1503607254 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa2600aed00 0x7fa240004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:05.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa25e59c700 1 --2- 172.21.15.60:0/1503607254 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2400049f0 0x7fa240004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:05.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa25e59c700 1 -- 172.21.15.60:0/1503607254 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2600adf90 con 0x7fa2400049f0 2022-06-29T16:05:05.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa25e59c700 1 -- 172.21.15.60:0/1503607254 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24000b7b0 con 0x7fa240003910 2022-06-29T16:05:05.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa25e59c700 1 -- 172.21.15.60:0/1503607254 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa24000b970 con 0x7fa2600aed00 2022-06-29T16:05:05.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa25f59e700 1 --2- 172.21.15.60:0/1503607254 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2400049f0 0x7fa240004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:05.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa25ed9d700 1 --2- 172.21.15.60:0/1503607254 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa240003910 0x7fa240003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:05.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa25659c700 1 --2- 172.21.15.60:0/1503607254 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa2600aed00 0x7fa240004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:05.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa25f59e700 1 -- 172.21.15.60:0/1503607254 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa2600aed00 msgr2=0x7fa240004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:05.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa25f59e700 1 --2- 172.21.15.60:0/1503607254 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa2600aed00 0x7fa240004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:05.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa25f59e700 1 -- 172.21.15.60:0/1503607254 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa240003910 msgr2=0x7fa240003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:05.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa25f59e700 1 --2- 172.21.15.60:0/1503607254 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa240003910 0x7fa240003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:05.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa25f59e700 1 -- 172.21.15.60:0/1503607254 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa240003260 con 0x7fa2400049f0 2022-06-29T16:05:05.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa25f59e700 1 --2- 172.21.15.60:0/1503607254 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2400049f0 0x7fa240004e70 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 rx=0x7fa25800c000 tx=0x7fa2580091d0).ready entity=mon.0 client_cookie=3c6237ffb710886 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:05.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa25e59c700 1 -- 172.21.15.60:0/1503607254 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa25800e7e0 con 0x7fa2400049f0 2022-06-29T16:05:05.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa25e59c700 1 -- 172.21.15.60:0/1503607254 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa25800e940 con 0x7fa2400049f0 2022-06-29T16:05:05.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.494+0000 7fa25e59c700 1 -- 172.21.15.60:0/1503607254 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa25800eca0 con 0x7fa2400049f0 2022-06-29T16:05:05.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.498+0000 7fa26489f700 1 -- 172.21.15.60:0/1503607254 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2400049f0 msgr2=0x7fa240004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:05.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.498+0000 7fa26489f700 1 --2- 172.21.15.60:0/1503607254 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2400049f0 0x7fa240004e70 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 rx=0x7fa25800c000 tx=0x7fa2580091d0).stop 2022-06-29T16:05:05.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.498+0000 7fa25f59e700 1 -- 172.21.15.60:0/1503607254 reap_dead start 2022-06-29T16:05:05.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.498+0000 7fa26489f700 1 -- 172.21.15.60:0/1503607254 shutdown_connections 2022-06-29T16:05:05.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.498+0000 7fa26489f700 1 -- 172.21.15.60:0/1503607254 >> 172.21.15.60:0/1503607254 conn(0x7fa2600b5890 msgr2=0x7fa2600b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:05.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.498+0000 7fa26489f700 1 -- 172.21.15.60:0/1503607254 shutdown_connections 2022-06-29T16:05:05.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.498+0000 7fa26489f700 1 -- 172.21.15.60:0/1503607254 wait complete. 2022-06-29T16:05:05.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.498+0000 7fa26489f700 1 Processor -- start 2022-06-29T16:05:05.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.498+0000 7fa26489f700 1 -- start start 2022-06-29T16:05:05.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.498+0000 7fa26489f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2601462b0 0x7fa260159820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:05.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.498+0000 7fa26489f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa260063e90 0x7fa260159d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:05.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.498+0000 7fa26489f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa26014a9b0 0x7fa26015a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:05.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.498+0000 7fa26489f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26015a6f0 con 0x7fa2601462b0 2022-06-29T16:05:05.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.498+0000 7fa26489f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2600666b0 con 0x7fa26014a9b0 2022-06-29T16:05:05.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.498+0000 7fa26489f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa260146690 con 0x7fa260063e90 2022-06-29T16:05:05.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.498+0000 7fa25ed9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2601462b0 0x7fa260159820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:05.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.498+0000 7fa25659c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa260063e90 0x7fa260159d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:05.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.498+0000 7fa25f59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa26014a9b0 0x7fa26015a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:05.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.498+0000 7fa25659c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa260063e90 0x7fa260159d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:34580/0 (socket says 172.21.15.60:34580) 2022-06-29T16:05:05.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.498+0000 7fa25659c700 1 -- 172.21.15.60:0/3941561398 learned_addr learned my addr 172.21.15.60:0/3941561398 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:05.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.498+0000 7fa25f59e700 1 -- 172.21.15.60:0/3941561398 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa260063e90 msgr2=0x7fa260159d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:05.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.498+0000 7fa25f59e700 1 --2- 172.21.15.60:0/3941561398 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa260063e90 0x7fa260159d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:05.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.498+0000 7fa25f59e700 1 -- 172.21.15.60:0/3941561398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2601462b0 msgr2=0x7fa260159820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:05.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.498+0000 7fa25f59e700 1 --2- 172.21.15.60:0/3941561398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2601462b0 0x7fa260159820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:05.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.498+0000 7fa25f59e700 1 -- 172.21.15.60:0/3941561398 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa260066900 con 0x7fa26014a9b0 2022-06-29T16:05:05.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.498+0000 7fa25659c700 1 --2- 172.21.15.60:0/3941561398 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa260063e90 0x7fa260159d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:05:05.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.502+0000 7fa25f59e700 1 --2- 172.21.15.60:0/3941561398 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa26014a9b0 0x7fa26015a200 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fa260149980 tx=0x7fa25801a000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:05.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.502+0000 7fa25cd99700 1 -- 172.21.15.60:0/3941561398 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa25800e7e0 con 0x7fa26014a9b0 2022-06-29T16:05:05.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.502+0000 7fa25cd99700 1 -- 172.21.15.60:0/3941561398 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa258012600 con 0x7fa26014a9b0 2022-06-29T16:05:05.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.502+0000 7fa26489f700 1 -- 172.21.15.60:0/3941561398 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa260066ba0 con 0x7fa26014a9b0 2022-06-29T16:05:05.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.502+0000 7fa25cd99700 1 -- 172.21.15.60:0/3941561398 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa258009600 con 0x7fa26014a9b0 2022-06-29T16:05:05.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.502+0000 7fa26489f700 1 -- 172.21.15.60:0/3941561398 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa26013fe30 con 0x7fa26014a9b0 2022-06-29T16:05:05.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.502+0000 7fa26489f700 1 -- 172.21.15.60:0/3941561398 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa26014a5f0 con 0x7fa26014a9b0 2022-06-29T16:05:05.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.502+0000 7fa25cd99700 1 -- 172.21.15.60:0/3941561398 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa258012770 con 0x7fa26014a9b0 2022-06-29T16:05:05.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.502+0000 7fa25cd99700 1 --2- 172.21.15.60:0/3941561398 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa2440539e0 0x7fa244055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:05.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.502+0000 7fa25cd99700 1 -- 172.21.15.60:0/3941561398 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6116+0+0 (secure 0 0 0) 0x7fa25806dba0 con 0x7fa26014a9b0 2022-06-29T16:05:05.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.506+0000 7fa25ed9d700 1 --2- 172.21.15.60:0/3941561398 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa2440539e0 0x7fa244055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:05.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.506+0000 7fa25ed9d700 1 --2- 172.21.15.60:0/3941561398 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa2440539e0 0x7fa244055ea0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7fa250003000 tx=0x7fa250007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:05.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.510+0000 7fa25cd99700 1 -- 172.21.15.60:0/3941561398 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa2580449f0 con 0x7fa26014a9b0 2022-06-29T16:05:05.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.678+0000 7fa26489f700 1 -- 172.21.15.60:0/3941561398 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa260066d60 con 0x7fa26014a9b0 2022-06-29T16:05:05.891 INFO:tasks.ceph.mon.a.smithi060.stderr:2022-06-29T16:05:05.886+0000 7fcc1a376700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-06-29T16:05:05.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.898+0000 7fa25cd99700 1 -- 172.21.15.60:0/3941561398 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v31) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa258016020 con 0x7fa26014a9b0 2022-06-29T16:05:05.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.902+0000 7fa26489f700 1 -- 172.21.15.60:0/3941561398 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa2440539e0 msgr2=0x7fa244055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:05.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.902+0000 7fa26489f700 1 --2- 172.21.15.60:0/3941561398 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa2440539e0 0x7fa244055ea0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7fa250003000 tx=0x7fa250007a00).stop 2022-06-29T16:05:05.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.902+0000 7fa26489f700 1 -- 172.21.15.60:0/3941561398 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa26014a9b0 msgr2=0x7fa26015a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:05.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.902+0000 7fa26489f700 1 --2- 172.21.15.60:0/3941561398 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa26014a9b0 0x7fa26015a200 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fa260149980 tx=0x7fa25801a000).stop 2022-06-29T16:05:05.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.902+0000 7fa26489f700 1 -- 172.21.15.60:0/3941561398 shutdown_connections 2022-06-29T16:05:05.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.902+0000 7fa26489f700 1 --2- 172.21.15.60:0/3941561398 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa2440539e0 0x7fa244055ea0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:05.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.902+0000 7fa26489f700 1 --2- 172.21.15.60:0/3941561398 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa26014a9b0 0x7fa26015a200 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:05.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.902+0000 7fa26489f700 1 --2- 172.21.15.60:0/3941561398 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa260063e90 0x7fa260159d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:05.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.902+0000 7fa26489f700 1 --2- 172.21.15.60:0/3941561398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa2601462b0 0x7fa260159820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:05.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.902+0000 7fa26489f700 1 -- 172.21.15.60:0/3941561398 >> 172.21.15.60:0/3941561398 conn(0x7fa2600b5890 msgr2=0x7fa2600af720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:05.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.902+0000 7fa26489f700 1 -- 172.21.15.60:0/3941561398 shutdown_connections 2022-06-29T16:05:05.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:05.902+0000 7fa26489f700 1 -- 172.21.15.60:0/3941561398 wait complete. 2022-06-29T16:05:05.914 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-06-29T16:05:05.927 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:05:05.933 INFO:tasks.ceph.mds.b.smithi092.stdout:starting mds.b at 2022-06-29T16:05:06.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.030+0000 7f8511848700 1 Processor -- start 2022-06-29T16:05:06.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.030+0000 7f8511848700 1 -- start start 2022-06-29T16:05:06.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.030+0000 7f8511848700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f850c0b5950 0x7f850c067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:06.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.030+0000 7f8511848700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f850c063ad0 0x7f850c14af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:06.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.030+0000 7f8511848700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f850c0acba0 con 0x7f850c05bd20 2022-06-29T16:05:06.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.030+0000 7f8511848700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f850c067220 con 0x7f850c063ad0 2022-06-29T16:05:06.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.030+0000 7f8511848700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f850c0b5d30 con 0x7f850c0b5950 2022-06-29T16:05:06.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.030+0000 7f850b7fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f850c05bd20 0x7f850c05c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57814/0 (socket says 172.21.15.60:57814) 2022-06-29T16:05:06.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.030+0000 7f850b7fe700 1 -- 172.21.15.60:0/3891484691 learned_addr learned my addr 172.21.15.60:0/3891484691 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:06.042 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.030+0000 7f850a7fc700 1 --2- 172.21.15.60:0/3891484691 >> v2:172.21.15.60:3300/0 conn(0x7f850c063ad0 0x7f850c14af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:06.042 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.030+0000 7f850affd700 1 --2- 172.21.15.60:0/3891484691 >> v2:172.21.15.92:3300/0 conn(0x7f850c0b5950 0x7f850c067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:06.042 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.030+0000 7f8509ffb700 1 -- 172.21.15.60:0/3891484691 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3139006429 0 0) 0x7f850c0acba0 con 0x7f850c05bd20 2022-06-29T16:05:06.042 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.030+0000 7f8509ffb700 1 -- 172.21.15.60:0/3891484691 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84ec003260 con 0x7f850c05bd20 2022-06-29T16:05:06.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.030+0000 7f850affd700 1 -- 172.21.15.60:0/3891484691 >> v2:172.21.15.60:3300/0 conn(0x7f850c063ad0 msgr2=0x7f850c14af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:06.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.030+0000 7f850affd700 1 --2- 172.21.15.60:0/3891484691 >> v2:172.21.15.60:3300/0 conn(0x7f850c063ad0 0x7f850c14af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:06.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.030+0000 7f850affd700 1 -- 172.21.15.60:0/3891484691 >> v1:172.21.15.60:6789/0 conn(0x7f850c05bd20 legacy=0x7f850c05c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:06.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.030+0000 7f850affd700 1 -- 172.21.15.60:0/3891484691 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850c0ace00 con 0x7f850c0b5950 2022-06-29T16:05:06.044 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.030+0000 7f850affd700 1 --2- 172.21.15.60:0/3891484691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f850c0b5950 0x7f850c067680 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f84fc0032f0 tx=0x7f84fc0091d0).ready entity=mon.1 client_cookie=3d3539f86e0dad6a server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:06.044 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.034+0000 7f8509ffb700 1 -- 172.21.15.60:0/3891484691 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84fc00b600 con 0x7f850c0b5950 2022-06-29T16:05:06.044 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.034+0000 7f8509ffb700 1 -- 172.21.15.60:0/3891484691 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84fc00b760 con 0x7f850c0b5950 2022-06-29T16:05:06.044 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.034+0000 7f8509ffb700 1 -- 172.21.15.60:0/3891484691 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84fc00b8d0 con 0x7f850c0b5950 2022-06-29T16:05:06.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.034+0000 7f8511848700 1 -- 172.21.15.60:0/3891484691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f850c0b5950 msgr2=0x7f850c067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:06.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.034+0000 7f8511848700 1 --2- 172.21.15.60:0/3891484691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f850c0b5950 0x7f850c067680 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f84fc0032f0 tx=0x7f84fc0091d0).stop 2022-06-29T16:05:06.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.034+0000 7f8511848700 1 -- 172.21.15.60:0/3891484691 shutdown_connections 2022-06-29T16:05:06.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.034+0000 7f8511848700 1 --2- 172.21.15.60:0/3891484691 >> v2:172.21.15.60:3300/0 conn(0x7f850c063ad0 0x7f850c14af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:06.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.034+0000 7f8511848700 1 --2- 172.21.15.60:0/3891484691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f850c0b5950 0x7f850c067680 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:06.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.034+0000 7f8511848700 1 -- 172.21.15.60:0/3891484691 >> 172.21.15.60:0/3891484691 conn(0x7f850c0632c0 msgr2=0x7f850c0b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:06.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.034+0000 7f8511848700 1 -- 172.21.15.60:0/3891484691 shutdown_connections 2022-06-29T16:05:06.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.034+0000 7f8511848700 1 -- 172.21.15.60:0/3891484691 wait complete. 2022-06-29T16:05:06.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.038+0000 7f8511848700 1 Processor -- start 2022-06-29T16:05:06.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.038+0000 7f8511848700 1 -- start start 2022-06-29T16:05:06.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.038+0000 7f8511848700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f850c05bd20 0x7f850c14cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:06.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.038+0000 7f8511848700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f850c063ad0 0x7f850c1594e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:06.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.038+0000 7f8511848700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f850c0b5950 0x7f850c1565a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:06.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.038+0000 7f8511848700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c1486d0 con 0x7f850c0b5950 2022-06-29T16:05:06.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.038+0000 7f8511848700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c155f20 con 0x7f850c05bd20 2022-06-29T16:05:06.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.038+0000 7f8511848700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f850c156140 con 0x7f850c063ad0 2022-06-29T16:05:06.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.038+0000 7f850b7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f850c0b5950 0x7f850c1565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:06.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.038+0000 7f850b7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f850c0b5950 0x7f850c1565a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49734/0 (socket says 172.21.15.60:49734) 2022-06-29T16:05:06.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.038+0000 7f850b7fe700 1 -- 172.21.15.60:0/3062342802 learned_addr learned my addr 172.21.15.60:0/3062342802 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:06.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.038+0000 7f850a7fc700 1 --2- 172.21.15.60:0/3062342802 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f850c063ad0 0x7f850c1594e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:06.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.038+0000 7f850affd700 1 --2- 172.21.15.60:0/3062342802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f850c05bd20 0x7f850c14cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:06.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.038+0000 7f850b7fe700 1 -- 172.21.15.60:0/3062342802 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f850c063ad0 msgr2=0x7f850c1594e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:06.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.038+0000 7f850b7fe700 1 --2- 172.21.15.60:0/3062342802 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f850c063ad0 0x7f850c1594e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:06.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.038+0000 7f850b7fe700 1 -- 172.21.15.60:0/3062342802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f850c05bd20 msgr2=0x7f850c14cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:06.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.038+0000 7f850b7fe700 1 --2- 172.21.15.60:0/3062342802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f850c05bd20 0x7f850c14cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:06.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.038+0000 7f850b7fe700 1 -- 172.21.15.60:0/3062342802 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84fc009470 con 0x7f850c0b5950 2022-06-29T16:05:06.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.038+0000 7f850b7fe700 1 --2- 172.21.15.60:0/3062342802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f850c0b5950 0x7f850c1565a0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0x7f850c156060 tx=0x7f85040079e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:06.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.038+0000 7f8503fff700 1 -- 172.21.15.60:0/3062342802 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f850400cea0 con 0x7f850c0b5950 2022-06-29T16:05:06.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.038+0000 7f8503fff700 1 -- 172.21.15.60:0/3062342802 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85040070d0 con 0x7f850c0b5950 2022-06-29T16:05:06.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.038+0000 7f8511848700 1 -- 172.21.15.60:0/3062342802 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f850c0ad6f0 con 0x7f850c0b5950 2022-06-29T16:05:06.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.038+0000 7f8503fff700 1 -- 172.21.15.60:0/3062342802 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8504007430 con 0x7f850c0b5950 2022-06-29T16:05:06.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.038+0000 7f8511848700 1 -- 172.21.15.60:0/3062342802 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f850c0acfe0 con 0x7f850c0b5950 2022-06-29T16:05:06.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.042+0000 7f8511848700 1 -- 172.21.15.60:0/3062342802 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f850c1562e0 con 0x7f850c0b5950 2022-06-29T16:05:06.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.042+0000 7f8503fff700 1 -- 172.21.15.60:0/3062342802 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8504010070 con 0x7f850c0b5950 2022-06-29T16:05:06.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.042+0000 7f8503fff700 1 --2- 172.21.15.60:0/3062342802 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f84f0053a70 0x7f84f0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:06.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.042+0000 7f8503fff700 1 -- 172.21.15.60:0/3062342802 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6202+0+0 (secure 0 0 0) 0x7f850406b570 con 0x7f850c0b5950 2022-06-29T16:05:06.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.042+0000 7f850affd700 1 --2- 172.21.15.60:0/3062342802 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f84f0053a70 0x7f84f0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:06.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.042+0000 7f850affd700 1 --2- 172.21.15.60:0/3062342802 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f84f0053a70 0x7f84f0055f30 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f84fc01b000 tx=0x7f84fc016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:06.058 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.050+0000 7f8503fff700 1 -- 172.21.15.60:0/3062342802 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8504042410 con 0x7f850c0b5950 2022-06-29T16:05:06.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.210+0000 7f8511848700 1 -- 172.21.15.60:0/3062342802 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f850c0573d0 con 0x7f850c0b5950 2022-06-29T16:05:06.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.210+0000 7f8503fff700 1 -- 172.21.15.60:0/3062342802 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+1779 (secure 0 0 0) 0x7f8504045a30 con 0x7f850c0b5950 2022-06-29T16:05:06.220 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:06.220 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":31,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5690,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1158969084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1158969084},{"type":"v1","addr":"172.21.15.60:6833","nonce":1158969084}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":28}],"filesystems":[{"mdsmap":{"epoch":31,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:04:08.275039+0000","modified":"2022-06-29T16:05:05.888732+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-06-29T16:05:06.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.214+0000 7f8511848700 1 -- 172.21.15.60:0/3062342802 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f84f0053a70 msgr2=0x7f84f0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:06.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.214+0000 7f8511848700 1 --2- 172.21.15.60:0/3062342802 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f84f0053a70 0x7f84f0055f30 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f84fc01b000 tx=0x7f84fc016000).stop 2022-06-29T16:05:06.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.214+0000 7f8511848700 1 -- 172.21.15.60:0/3062342802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f850c0b5950 msgr2=0x7f850c1565a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:06.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.214+0000 7f8511848700 1 --2- 172.21.15.60:0/3062342802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f850c0b5950 0x7f850c1565a0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0x7f850c156060 tx=0x7f85040079e0).stop 2022-06-29T16:05:06.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.214+0000 7f8511848700 1 -- 172.21.15.60:0/3062342802 shutdown_connections 2022-06-29T16:05:06.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.214+0000 7f8511848700 1 --2- 172.21.15.60:0/3062342802 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f84f0053a70 0x7f84f0055f30 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:06.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.214+0000 7f8511848700 1 --2- 172.21.15.60:0/3062342802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f850c0b5950 0x7f850c1565a0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:06.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.214+0000 7f8511848700 1 --2- 172.21.15.60:0/3062342802 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f850c063ad0 0x7f850c1594e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:06.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.214+0000 7f8511848700 1 --2- 172.21.15.60:0/3062342802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f850c05bd20 0x7f850c14cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:06.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.214+0000 7f8511848700 1 -- 172.21.15.60:0/3062342802 >> 172.21.15.60:0/3062342802 conn(0x7f850c0632c0 msgr2=0x7f850c0b3ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:06.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.214+0000 7f8511848700 1 -- 172.21.15.60:0/3062342802 shutdown_connections 2022-06-29T16:05:06.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.214+0000 7f8511848700 1 -- 172.21.15.60:0/3062342802 wait complete. 2022-06-29T16:05:06.225 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 31 2022-06-29T16:05:06.232 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-06-29T16:05:06.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.326+0000 7f232359e700 1 Processor -- start 2022-06-29T16:05:06.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.326+0000 7f232359e700 1 -- start start 2022-06-29T16:05:06.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.330+0000 7f232359e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f232405fb00 0x7f232413f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:06.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.330+0000 7f232359e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f23241405a0 0x7f2324143fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:06.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.330+0000 7f232359e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2324064e20 con 0x7f2324067a90 2022-06-29T16:05:06.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.330+0000 7f232359e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23240b10e0 con 0x7f23241405a0 2022-06-29T16:05:06.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.330+0000 7f232359e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f232405fee0 con 0x7f232405fb00 2022-06-29T16:05:06.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.330+0000 7f2321d9b700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2324067a90 0x7f232413fe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57824/0 (socket says 172.21.15.60:57824) 2022-06-29T16:05:06.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.330+0000 7f2321d9b700 1 -- 172.21.15.60:0/3455196323 learned_addr learned my addr 172.21.15.60:0/3455196323 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:06.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.330+0000 7f232259c700 1 --2- 172.21.15.60:0/3455196323 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f232405fb00 0x7f232413f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:06.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.330+0000 7f2322d9d700 1 --2- 172.21.15.60:0/3455196323 >> v2:172.21.15.92:3300/0 conn(0x7f23241405a0 0x7f2324143fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:06.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.330+0000 7f232159a700 1 -- 172.21.15.60:0/3455196323 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1502712698 0 0) 0x7f2324064e20 con 0x7f2324067a90 2022-06-29T16:05:06.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.330+0000 7f232159a700 1 -- 172.21.15.60:0/3455196323 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2308003260 con 0x7f2324067a90 2022-06-29T16:05:06.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.330+0000 7f2322d9d700 1 -- 172.21.15.60:0/3455196323 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f232405fb00 msgr2=0x7f232413f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:06.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.330+0000 7f2322d9d700 1 --2- 172.21.15.60:0/3455196323 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f232405fb00 0x7f232413f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:06.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.330+0000 7f232159a700 1 -- 172.21.15.60:0/3455196323 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 611019018 0 0) 0x7f2308003260 con 0x7f2324067a90 2022-06-29T16:05:06.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.330+0000 7f2322d9d700 1 -- 172.21.15.60:0/3455196323 >> v1:172.21.15.60:6789/0 conn(0x7f2324067a90 legacy=0x7f232413fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:06.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.330+0000 7f2322d9d700 1 -- 172.21.15.60:0/3455196323 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2324060b20 con 0x7f23241405a0 2022-06-29T16:05:06.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.330+0000 7f2322d9d700 1 --2- 172.21.15.60:0/3455196323 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23241405a0 0x7f2324143fa0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f2314001800 tx=0x7f231c0091d0).ready entity=mon.1 client_cookie=8ff12b8604fe85ed server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:06.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.330+0000 7f232159a700 1 -- 172.21.15.60:0/3455196323 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2314001660 con 0x7f23241405a0 2022-06-29T16:05:06.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.330+0000 7f232159a700 1 -- 172.21.15.60:0/3455196323 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231c00f510 con 0x7f23241405a0 2022-06-29T16:05:06.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.330+0000 7f232159a700 1 -- 172.21.15.60:0/3455196323 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f231c00f680 con 0x7f23241405a0 2022-06-29T16:05:06.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.330+0000 7f232359e700 1 -- 172.21.15.60:0/3455196323 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23241405a0 msgr2=0x7f2324143fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:06.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.330+0000 7f232359e700 1 --2- 172.21.15.60:0/3455196323 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23241405a0 0x7f2324143fa0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f2314001800 tx=0x7f231c0091d0).stop 2022-06-29T16:05:06.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.330+0000 7f232359e700 1 -- 172.21.15.60:0/3455196323 shutdown_connections 2022-06-29T16:05:06.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.330+0000 7f232359e700 1 --2- 172.21.15.60:0/3455196323 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23241405a0 0x7f2324143fa0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:06.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.330+0000 7f232359e700 1 --2- 172.21.15.60:0/3455196323 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f232405fb00 0x7f232413f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:06.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.330+0000 7f232359e700 1 -- 172.21.15.60:0/3455196323 >> 172.21.15.60:0/3455196323 conn(0x7f2324063a60 msgr2=0x7f232405f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:06.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.330+0000 7f232359e700 1 -- 172.21.15.60:0/3455196323 shutdown_connections 2022-06-29T16:05:06.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.330+0000 7f232359e700 1 -- 172.21.15.60:0/3455196323 wait complete. 2022-06-29T16:05:06.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.334+0000 7f232359e700 1 Processor -- start 2022-06-29T16:05:06.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.334+0000 7f232359e700 1 -- start start 2022-06-29T16:05:06.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.334+0000 7f232359e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f232405fb00 0x7f232413b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:06.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.334+0000 7f232359e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2324067a90 0x7f23241503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:06.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.334+0000 7f232359e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23241405a0 0x7f2324152ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:06.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.334+0000 7f232359e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2324066570 con 0x7f2324067a90 2022-06-29T16:05:06.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.334+0000 7f232359e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f232414fa00 con 0x7f232405fb00 2022-06-29T16:05:06.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.334+0000 7f232359e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f232414fca0 con 0x7f23241405a0 2022-06-29T16:05:06.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.334+0000 7f2321d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2324067a90 0x7f23241503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:06.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.334+0000 7f2321d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2324067a90 0x7f23241503e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49744/0 (socket says 172.21.15.60:49744) 2022-06-29T16:05:06.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.334+0000 7f2321d9b700 1 -- 172.21.15.60:0/2811130374 learned_addr learned my addr 172.21.15.60:0/2811130374 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:06.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.334+0000 7f2322d9d700 1 --2- 172.21.15.60:0/2811130374 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23241405a0 0x7f2324152ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:06.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.334+0000 7f232259c700 1 --2- 172.21.15.60:0/2811130374 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f232405fb00 0x7f232413b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:06.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.334+0000 7f2321d9b700 1 -- 172.21.15.60:0/2811130374 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23241405a0 msgr2=0x7f2324152ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:06.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.334+0000 7f2321d9b700 1 --2- 172.21.15.60:0/2811130374 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23241405a0 0x7f2324152ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:06.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.334+0000 7f2321d9b700 1 -- 172.21.15.60:0/2811130374 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f232405fb00 msgr2=0x7f232413b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:06.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.334+0000 7f2321d9b700 1 --2- 172.21.15.60:0/2811130374 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f232405fb00 0x7f232413b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:06.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.334+0000 7f2321d9b700 1 -- 172.21.15.60:0/2811130374 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f231c009470 con 0x7f2324067a90 2022-06-29T16:05:06.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.334+0000 7f2322d9d700 1 --2- 172.21.15.60:0/2811130374 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23241405a0 0x7f2324152ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:06.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.334+0000 7f232259c700 1 --2- 172.21.15.60:0/2811130374 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f232405fb00 0x7f232413b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:06.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.334+0000 7f2321d9b700 1 --2- 172.21.15.60:0/2811130374 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2324067a90 0x7f23241503e0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 rx=0x7f23140018c0 tx=0x7f23140019f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:06.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.334+0000 7f23137fe700 1 -- 172.21.15.60:0/2811130374 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2314007320 con 0x7f2324067a90 2022-06-29T16:05:06.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.334+0000 7f232359e700 1 -- 172.21.15.60:0/2811130374 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f232414ff40 con 0x7f2324067a90 2022-06-29T16:05:06.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.334+0000 7f23137fe700 1 -- 172.21.15.60:0/2811130374 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2314007480 con 0x7f2324067a90 2022-06-29T16:05:06.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.334+0000 7f232359e700 1 -- 172.21.15.60:0/2811130374 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23241420c0 con 0x7f2324067a90 2022-06-29T16:05:06.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.334+0000 7f23137fe700 1 -- 172.21.15.60:0/2811130374 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2314005800 con 0x7f2324067a90 2022-06-29T16:05:06.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.334+0000 7f232359e700 1 -- 172.21.15.60:0/2811130374 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f232413c710 con 0x7f2324067a90 2022-06-29T16:05:06.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.334+0000 7f23137fe700 1 -- 172.21.15.60:0/2811130374 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2314010030 con 0x7f2324067a90 2022-06-29T16:05:06.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.338+0000 7f23137fe700 1 --2- 172.21.15.60:0/2811130374 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f23000539e0 0x7f2300055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:06.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.338+0000 7f23137fe700 1 -- 172.21.15.60:0/2811130374 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6202+0+0 (secure 0 0 0) 0x7f231406c010 con 0x7f2324067a90 2022-06-29T16:05:06.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.338+0000 7f232259c700 1 --2- 172.21.15.60:0/2811130374 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f23000539e0 0x7f2300055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:06.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.338+0000 7f232259c700 1 --2- 172.21.15.60:0/2811130374 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f23000539e0 0x7f2300055ea0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f231800b000 tx=0x7f2318008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:06.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.346+0000 7f23137fe700 1 -- 172.21.15.60:0/2811130374 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f232413c710 con 0x7f2324067a90 2022-06-29T16:05:06.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.514+0000 7f232359e700 1 -- 172.21.15.60:0/2811130374 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f232413c710 con 0x7f2324067a90 2022-06-29T16:05:06.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.898+0000 7f23137fe700 1 -- 172.21.15.60:0/2811130374 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v32) v1 ==== 104+0+0 (secure 0 0 0) 0x7f232413c710 con 0x7f2324067a90 2022-06-29T16:05:06.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.902+0000 7f232359e700 1 -- 172.21.15.60:0/2811130374 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f23000539e0 msgr2=0x7f2300055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:06.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.902+0000 7f232359e700 1 --2- 172.21.15.60:0/2811130374 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f23000539e0 0x7f2300055ea0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f231800b000 tx=0x7f2318008040).stop 2022-06-29T16:05:06.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.902+0000 7f232359e700 1 -- 172.21.15.60:0/2811130374 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2324067a90 msgr2=0x7f23241503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:06.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.902+0000 7f232359e700 1 --2- 172.21.15.60:0/2811130374 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2324067a90 0x7f23241503e0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 rx=0x7f23140018c0 tx=0x7f23140019f0).stop 2022-06-29T16:05:06.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.902+0000 7f232359e700 1 -- 172.21.15.60:0/2811130374 shutdown_connections 2022-06-29T16:05:06.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.902+0000 7f232359e700 1 --2- 172.21.15.60:0/2811130374 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f23000539e0 0x7f2300055ea0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:06.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.902+0000 7f232359e700 1 --2- 172.21.15.60:0/2811130374 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23241405a0 0x7f2324152ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:06.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.902+0000 7f232359e700 1 --2- 172.21.15.60:0/2811130374 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2324067a90 0x7f23241503e0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:06.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.902+0000 7f232359e700 1 --2- 172.21.15.60:0/2811130374 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f232405fb00 0x7f232413b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:06.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.902+0000 7f232359e700 1 -- 172.21.15.60:0/2811130374 >> 172.21.15.60:0/2811130374 conn(0x7f2324063a60 msgr2=0x7f23240b1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:06.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.902+0000 7f232359e700 1 -- 172.21.15.60:0/2811130374 shutdown_connections 2022-06-29T16:05:06.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:06.902+0000 7f232359e700 1 -- 172.21.15.60:0/2811130374 wait complete. 2022-06-29T16:05:06.927 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-06-29T16:05:07.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.034+0000 7fbbbc879700 1 Processor -- start 2022-06-29T16:05:07.044 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.034+0000 7fbbbc879700 1 -- start start 2022-06-29T16:05:07.045 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.038+0000 7fbbbc879700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbbb80b28e0 0x7fbbb805bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:07.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.038+0000 7fbbbc879700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbbb80afee0 0x7fbbb8141e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:07.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.038+0000 7fbbbc879700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbb80b1d70 con 0x7fbbb80b9be0 2022-06-29T16:05:07.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.038+0000 7fbbbc879700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbb80b9fc0 con 0x7fbbb80b28e0 2022-06-29T16:05:07.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.038+0000 7fbbbc879700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb80b3c40 con 0x7fbbb80afee0 2022-06-29T16:05:07.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.038+0000 7fbbaffff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbbb80b28e0 0x7fbbb805bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:07.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.038+0000 7fbbaffff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbbb80b28e0 0x7fbbb805bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49756/0 (socket says 172.21.15.60:49756) 2022-06-29T16:05:07.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.038+0000 7fbbaffff700 1 -- 172.21.15.60:0/2364897320 learned_addr learned my addr 172.21.15.60:0/2364897320 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:07.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.038+0000 7fbbb759e700 1 --2- 172.21.15.60:0/2364897320 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbbb80afee0 0x7fbbb8141e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:07.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.038+0000 7fbbaffff700 1 -- 172.21.15.60:0/2364897320 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbbb80afee0 msgr2=0x7fbbb8141e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:07.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.038+0000 7fbbaffff700 1 --2- 172.21.15.60:0/2364897320 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbbb80afee0 0x7fbbb8141e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:07.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.038+0000 7fbbaffff700 1 -- 172.21.15.60:0/2364897320 >> v1:172.21.15.92:6789/0 conn(0x7fbbb80b9be0 legacy=0x7fbbb80b21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:07.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.038+0000 7fbbaffff700 1 -- 172.21.15.60:0/2364897320 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb80b3ee0 con 0x7fbbb80b28e0 2022-06-29T16:05:07.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.038+0000 7fbbaffff700 1 --2- 172.21.15.60:0/2364897320 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbbb80b28e0 0x7fbbb805bd20 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 rx=0x7fbba0002f70 tx=0x7fbba0006eb0).ready entity=mon.0 client_cookie=5fa5bd4a75c73fbe server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:07.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.038+0000 7fbbb659c700 1 -- 172.21.15.60:0/2364897320 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbba000d570 con 0x7fbbb80b28e0 2022-06-29T16:05:07.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.038+0000 7fbbb659c700 1 -- 172.21.15.60:0/2364897320 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbba0000e90 con 0x7fbbb80b28e0 2022-06-29T16:05:07.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.038+0000 7fbbb659c700 1 -- 172.21.15.60:0/2364897320 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbba000d900 con 0x7fbbb80b28e0 2022-06-29T16:05:07.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.038+0000 7fbbbc879700 1 -- 172.21.15.60:0/2364897320 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbbb80b28e0 msgr2=0x7fbbb805bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:07.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.038+0000 7fbbbc879700 1 --2- 172.21.15.60:0/2364897320 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbbb80b28e0 0x7fbbb805bd20 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 rx=0x7fbba0002f70 tx=0x7fbba0006eb0).stop 2022-06-29T16:05:07.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.038+0000 7fbbbc879700 1 -- 172.21.15.60:0/2364897320 shutdown_connections 2022-06-29T16:05:07.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.038+0000 7fbbbc879700 1 --2- 172.21.15.60:0/2364897320 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbbb80afee0 0x7fbbb8141e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:07.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.038+0000 7fbbbc879700 1 --2- 172.21.15.60:0/2364897320 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbbb80b28e0 0x7fbbb805bd20 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:07.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.038+0000 7fbbbc879700 1 -- 172.21.15.60:0/2364897320 >> 172.21.15.60:0/2364897320 conn(0x7fbbb8062ce0 msgr2=0x7fbbb80b9190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:07.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.038+0000 7fbbbc879700 1 -- 172.21.15.60:0/2364897320 shutdown_connections 2022-06-29T16:05:07.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.038+0000 7fbbbc879700 1 -- 172.21.15.60:0/2364897320 wait complete. 2022-06-29T16:05:07.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.042+0000 7fbbbc879700 1 Processor -- start 2022-06-29T16:05:07.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.042+0000 7fbbbc879700 1 -- start start 2022-06-29T16:05:07.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.042+0000 7fbbbc879700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbb80afee0 0x7fbbb8147b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:07.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.042+0000 7fbbbc879700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbbb80b28e0 0x7fbbb8145ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:07.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.042+0000 7fbbbc879700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbbb80b9be0 0x7fbbb80b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:07.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.042+0000 7fbbbc879700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb8140e70 con 0x7fbbb80b9be0 2022-06-29T16:05:07.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.042+0000 7fbbbc879700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb80ad460 con 0x7fbbb80afee0 2022-06-29T16:05:07.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.042+0000 7fbbbc879700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb80ae470 con 0x7fbbb80b28e0 2022-06-29T16:05:07.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.042+0000 7fbbb759e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbbb80b9be0 0x7fbbb80b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:07.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.042+0000 7fbbb759e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbbb80b9be0 0x7fbbb80b7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49760/0 (socket says 172.21.15.60:49760) 2022-06-29T16:05:07.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.042+0000 7fbbb759e700 1 -- 172.21.15.60:0/364974802 learned_addr learned my addr 172.21.15.60:0/364974802 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:07.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.042+0000 7fbbaffff700 1 --2- 172.21.15.60:0/364974802 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbbb80b28e0 0x7fbbb8145ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:07.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.042+0000 7fbbb6d9d700 1 --2- 172.21.15.60:0/364974802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbb80afee0 0x7fbbb8147b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:07.058 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.042+0000 7fbbb759e700 1 -- 172.21.15.60:0/364974802 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbbb80b28e0 msgr2=0x7fbbb8145ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:07.058 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.042+0000 7fbbb759e700 1 --2- 172.21.15.60:0/364974802 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbbb80b28e0 0x7fbbb8145ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:07.058 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.042+0000 7fbbb759e700 1 -- 172.21.15.60:0/364974802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbb80afee0 msgr2=0x7fbbb8147b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:07.058 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.042+0000 7fbbb759e700 1 --2- 172.21.15.60:0/364974802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbb80afee0 0x7fbbb8147b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:07.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.042+0000 7fbbb759e700 1 -- 172.21.15.60:0/364974802 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba000b1e0 con 0x7fbbb80b9be0 2022-06-29T16:05:07.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.042+0000 7fbbb759e700 1 --2- 172.21.15.60:0/364974802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbbb80b9be0 0x7fbbb80b7bd0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 rx=0x7fbbb0006e00 tx=0x7fbbb00091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:07.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.042+0000 7fbbb4d99700 1 -- 172.21.15.60:0/364974802 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbbb000c640 con 0x7fbbb80b9be0 2022-06-29T16:05:07.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.042+0000 7fbbb4d99700 1 -- 172.21.15.60:0/364974802 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbb000c7a0 con 0x7fbbb80b9be0 2022-06-29T16:05:07.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.042+0000 7fbbbc879700 1 -- 172.21.15.60:0/364974802 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbb80b80c0 con 0x7fbbb80b9be0 2022-06-29T16:05:07.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.042+0000 7fbbb4d99700 1 -- 172.21.15.60:0/364974802 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbbb000cb00 con 0x7fbbb80b9be0 2022-06-29T16:05:07.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.042+0000 7fbbbc879700 1 -- 172.21.15.60:0/364974802 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbb8147460 con 0x7fbbb80b9be0 2022-06-29T16:05:07.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.046+0000 7fbbbc879700 1 -- 172.21.15.60:0/364974802 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbb80ae2d0 con 0x7fbbb80b9be0 2022-06-29T16:05:07.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.046+0000 7fbbb4d99700 1 -- 172.21.15.60:0/364974802 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fbbb000cc60 con 0x7fbbb80b9be0 2022-06-29T16:05:07.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.046+0000 7fbbb4d99700 1 --2- 172.21.15.60:0/364974802 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbb9c053b00 0x7fbb9c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:07.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.046+0000 7fbbb4d99700 1 -- 172.21.15.60:0/364974802 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6202+0+0 (secure 0 0 0) 0x7fbbb006c190 con 0x7fbbb80b9be0 2022-06-29T16:05:07.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.046+0000 7fbbb6d9d700 1 --2- 172.21.15.60:0/364974802 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbb9c053b00 0x7fbb9c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:07.063 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.046+0000 7fbbb6d9d700 1 --2- 172.21.15.60:0/364974802 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbb9c053b00 0x7fbb9c055fc0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7fbba800d000 tx=0x7fbba8007be0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:07.063 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.050+0000 7fbbb4d99700 1 -- 172.21.15.60:0/364974802 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbbb80ae2d0 con 0x7fbbb80b9be0 2022-06-29T16:05:07.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.178+0000 7fbbbc879700 1 -- 172.21.15.60:0/364974802 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbbb80ae2d0 con 0x7fbbb80b9be0 2022-06-29T16:05:07.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.918+0000 7fbbb4d99700 1 -- 172.21.15.60:0/364974802 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v74) v1 ==== 185+0+0 (secure 0 0 0) 0x7fbbb80ae2d0 con 0x7fbbb80b9be0 2022-06-29T16:05:07.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.922+0000 7fbbbc879700 1 -- 172.21.15.60:0/364974802 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbb9c053b00 msgr2=0x7fbb9c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:07.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.922+0000 7fbbbc879700 1 --2- 172.21.15.60:0/364974802 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbb9c053b00 0x7fbb9c055fc0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7fbba800d000 tx=0x7fbba8007be0).stop 2022-06-29T16:05:07.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.922+0000 7fbbbc879700 1 -- 172.21.15.60:0/364974802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbbb80b9be0 msgr2=0x7fbbb80b7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:07.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.922+0000 7fbbbc879700 1 --2- 172.21.15.60:0/364974802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbbb80b9be0 0x7fbbb80b7bd0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 rx=0x7fbbb0006e00 tx=0x7fbbb00091d0).stop 2022-06-29T16:05:07.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.922+0000 7fbbbc879700 1 -- 172.21.15.60:0/364974802 shutdown_connections 2022-06-29T16:05:07.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.922+0000 7fbbbc879700 1 --2- 172.21.15.60:0/364974802 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbb9c053b00 0x7fbb9c055fc0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:07.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.922+0000 7fbbbc879700 1 --2- 172.21.15.60:0/364974802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbbb80b9be0 0x7fbbb80b7bd0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:07.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.922+0000 7fbbbc879700 1 --2- 172.21.15.60:0/364974802 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbbb80b28e0 0x7fbbb8145ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:07.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.922+0000 7fbbbc879700 1 --2- 172.21.15.60:0/364974802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbb80afee0 0x7fbbb8147b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:07.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.922+0000 7fbbbc879700 1 -- 172.21.15.60:0/364974802 >> 172.21.15.60:0/364974802 conn(0x7fbbb8062ce0 msgr2=0x7fbbb8142e80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:07.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.922+0000 7fbbbc879700 1 -- 172.21.15.60:0/364974802 shutdown_connections 2022-06-29T16:05:07.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:07.922+0000 7fbbbc879700 1 -- 172.21.15.60:0/364974802 wait complete. 2022-06-29T16:05:07.933 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2022-06-29T16:05:07.944 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-06-29T16:05:08.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.050+0000 7fc239b96700 1 Processor -- start 2022-06-29T16:05:08.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.050+0000 7fc239b96700 1 -- start start 2022-06-29T16:05:08.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.054+0000 7fc239b96700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc23405d070 0x7fc234142c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:08.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.054+0000 7fc239b96700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2340b0820 0x7fc234143110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:08.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.054+0000 7fc239b96700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc234064030 con 0x7fc2340ade00 2022-06-29T16:05:08.063 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.054+0000 7fc239b96700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc23413f570 con 0x7fc23405d070 2022-06-29T16:05:08.063 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.054+0000 7fc239b96700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2340b1660 con 0x7fc2340b0820 2022-06-29T16:05:08.063 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.054+0000 7fc238b94700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc23405d070 0x7fc234142c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:08.063 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.054+0000 7fc238b94700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc23405d070 0x7fc234142c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49766/0 (socket says 172.21.15.60:49766) 2022-06-29T16:05:08.063 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.054+0000 7fc233fff700 1 --2- 172.21.15.60:0/3999680678 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2340b0820 0x7fc234143110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:08.064 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.054+0000 7fc238b94700 1 -- 172.21.15.60:0/3999680678 learned_addr learned my addr 172.21.15.60:0/3999680678 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:08.064 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.054+0000 7fc238b94700 1 -- 172.21.15.60:0/3999680678 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2340b0820 msgr2=0x7fc234143110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:08.064 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.054+0000 7fc2337fe700 1 -- 172.21.15.60:0/3999680678 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2616872454 0 0) 0x7fc234064030 con 0x7fc2340ade00 2022-06-29T16:05:08.064 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.054+0000 7fc238b94700 1 --2- 172.21.15.60:0/3999680678 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2340b0820 0x7fc234143110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:08.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.054+0000 7fc238b94700 1 -- 172.21.15.60:0/3999680678 >> v1:172.21.15.92:6789/0 conn(0x7fc2340ade00 legacy=0x7fc2340aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:08.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.054+0000 7fc238b94700 1 -- 172.21.15.60:0/3999680678 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc23405d450 con 0x7fc23405d070 2022-06-29T16:05:08.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.054+0000 7fc238b94700 1 --2- 172.21.15.60:0/3999680678 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc23405d070 0x7fc234142c20 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 rx=0x7fc224002f70 tx=0x7fc224006eb0).ready entity=mon.0 client_cookie=a73554211f79ed3e server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:08.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.054+0000 7fc2337fe700 1 -- 172.21.15.60:0/3999680678 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc22400d570 con 0x7fc23405d070 2022-06-29T16:05:08.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.054+0000 7fc2337fe700 1 -- 172.21.15.60:0/3999680678 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc224000e90 con 0x7fc23405d070 2022-06-29T16:05:08.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.054+0000 7fc2337fe700 1 -- 172.21.15.60:0/3999680678 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc22400d8c0 con 0x7fc23405d070 2022-06-29T16:05:08.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.054+0000 7fc239b96700 1 -- 172.21.15.60:0/3999680678 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc23405d070 msgr2=0x7fc234142c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:08.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.054+0000 7fc239b96700 1 --2- 172.21.15.60:0/3999680678 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc23405d070 0x7fc234142c20 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 rx=0x7fc224002f70 tx=0x7fc224006eb0).stop 2022-06-29T16:05:08.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.054+0000 7fc239b96700 1 -- 172.21.15.60:0/3999680678 shutdown_connections 2022-06-29T16:05:08.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.054+0000 7fc239b96700 1 --2- 172.21.15.60:0/3999680678 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2340b0820 0x7fc234143110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:08.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.054+0000 7fc239b96700 1 --2- 172.21.15.60:0/3999680678 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc23405d070 0x7fc234142c20 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:08.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.054+0000 7fc239b96700 1 -- 172.21.15.60:0/3999680678 >> 172.21.15.60:0/3999680678 conn(0x7fc2340b5840 msgr2=0x7fc2340b1810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:08.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.054+0000 7fc239b96700 1 -- 172.21.15.60:0/3999680678 shutdown_connections 2022-06-29T16:05:08.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.054+0000 7fc239b96700 1 -- 172.21.15.60:0/3999680678 wait complete. 2022-06-29T16:05:08.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.058+0000 7fc239b96700 1 Processor -- start 2022-06-29T16:05:08.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.058+0000 7fc239b96700 1 -- start start 2022-06-29T16:05:08.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.062+0000 7fc239b96700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc23405d070 0x7fc234141e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:08.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.062+0000 7fc239b96700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2340ade00 0x7fc23413cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:08.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.062+0000 7fc239b96700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2340b0820 0x7fc2341553c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:08.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.062+0000 7fc239b96700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc234135190 con 0x7fc2340b0820 2022-06-29T16:05:08.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.062+0000 7fc239b96700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc234135430 con 0x7fc2340ade00 2022-06-29T16:05:08.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.062+0000 7fc239b96700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2341356d0 con 0x7fc23405d070 2022-06-29T16:05:08.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.062+0000 7fc238b94700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc23405d070 0x7fc234141e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:08.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.062+0000 7fc239395700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2340b0820 0x7fc2341553c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:08.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.062+0000 7fc233fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2340ade00 0x7fc23413cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:08.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.062+0000 7fc238b94700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc23405d070 0x7fc234141e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:34634/0 (socket says 172.21.15.60:34634) 2022-06-29T16:05:08.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.062+0000 7fc239395700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2340b0820 0x7fc2341553c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49776/0 (socket says 172.21.15.60:49776) 2022-06-29T16:05:08.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.062+0000 7fc238b94700 1 -- 172.21.15.60:0/2385044695 learned_addr learned my addr 172.21.15.60:0/2385044695 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:08.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.062+0000 7fc239395700 1 -- 172.21.15.60:0/2385044695 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc23405d070 msgr2=0x7fc234141e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:08.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.062+0000 7fc239395700 1 --2- 172.21.15.60:0/2385044695 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc23405d070 0x7fc234141e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:08.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.062+0000 7fc239395700 1 -- 172.21.15.60:0/2385044695 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2340ade00 msgr2=0x7fc23413cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:08.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.062+0000 7fc239395700 1 --2- 172.21.15.60:0/2385044695 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2340ade00 0x7fc23413cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:08.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.062+0000 7fc239395700 1 -- 172.21.15.60:0/2385044695 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc22400b1e0 con 0x7fc2340b0820 2022-06-29T16:05:08.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.062+0000 7fc238b94700 1 --2- 172.21.15.60:0/2385044695 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc23405d070 0x7fc234141e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:05:08.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.062+0000 7fc239395700 1 --2- 172.21.15.60:0/2385044695 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2340b0820 0x7fc2341553c0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 rx=0x7fc22c001250 tx=0x7fc22c007d30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:08.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.062+0000 7fc231ffb700 1 -- 172.21.15.60:0/2385044695 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc22c00f470 con 0x7fc2340b0820 2022-06-29T16:05:08.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.062+0000 7fc231ffb700 1 -- 172.21.15.60:0/2385044695 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc22c00f5d0 con 0x7fc2340b0820 2022-06-29T16:05:08.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.062+0000 7fc239b96700 1 -- 172.21.15.60:0/2385044695 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc234135b70 con 0x7fc2340b0820 2022-06-29T16:05:08.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.062+0000 7fc231ffb700 1 -- 172.21.15.60:0/2385044695 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc22c00f930 con 0x7fc2340b0820 2022-06-29T16:05:08.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.062+0000 7fc239b96700 1 -- 172.21.15.60:0/2385044695 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc234136090 con 0x7fc2340b0820 2022-06-29T16:05:08.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.066+0000 7fc239b96700 1 -- 172.21.15.60:0/2385044695 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc234063d90 con 0x7fc2340b0820 2022-06-29T16:05:08.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.066+0000 7fc231ffb700 1 -- 172.21.15.60:0/2385044695 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc22c019030 con 0x7fc2340b0820 2022-06-29T16:05:08.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.066+0000 7fc231ffb700 1 --2- 172.21.15.60:0/2385044695 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc2140539e0 0x7fc214055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:08.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.066+0000 7fc231ffb700 1 -- 172.21.15.60:0/2385044695 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5764+0+0 (secure 0 0 0) 0x7fc22c06b2b0 con 0x7fc2340b0820 2022-06-29T16:05:08.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.066+0000 7fc238b94700 1 --2- 172.21.15.60:0/2385044695 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc2140539e0 0x7fc214055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:08.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.066+0000 7fc238b94700 1 --2- 172.21.15.60:0/2385044695 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc2140539e0 0x7fc214055ea0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7fc22401c000 tx=0x7fc224015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:08.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.070+0000 7fc231ffb700 1 -- 172.21.15.60:0/2385044695 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc22c0423b0 con 0x7fc2340b0820 2022-06-29T16:05:08.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.202+0000 7fc239b96700 1 -- 172.21.15.60:0/2385044695 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc234135890 con 0x7fc2340b0820 2022-06-29T16:05:08.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.918+0000 7fc231ffb700 1 -- 172.21.15.60:0/2385044695 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v75) v1 ==== 173+0+0 (secure 0 0 0) 0x7fc22c0459d0 con 0x7fc2340b0820 2022-06-29T16:05:08.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.922+0000 7fc239b96700 1 -- 172.21.15.60:0/2385044695 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc2140539e0 msgr2=0x7fc214055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:08.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.922+0000 7fc239b96700 1 --2- 172.21.15.60:0/2385044695 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc2140539e0 0x7fc214055ea0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7fc22401c000 tx=0x7fc224015000).stop 2022-06-29T16:05:08.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.922+0000 7fc239b96700 1 -- 172.21.15.60:0/2385044695 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2340b0820 msgr2=0x7fc2341553c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:08.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.922+0000 7fc239b96700 1 --2- 172.21.15.60:0/2385044695 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2340b0820 0x7fc2341553c0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 rx=0x7fc22c001250 tx=0x7fc22c007d30).stop 2022-06-29T16:05:08.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.922+0000 7fc239b96700 1 -- 172.21.15.60:0/2385044695 shutdown_connections 2022-06-29T16:05:08.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.922+0000 7fc239b96700 1 --2- 172.21.15.60:0/2385044695 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc2140539e0 0x7fc214055ea0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:08.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.922+0000 7fc239b96700 1 --2- 172.21.15.60:0/2385044695 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2340b0820 0x7fc2341553c0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:08.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.922+0000 7fc239b96700 1 --2- 172.21.15.60:0/2385044695 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2340ade00 0x7fc23413cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:08.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.922+0000 7fc239b96700 1 --2- 172.21.15.60:0/2385044695 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc23405d070 0x7fc234141e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:08.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.922+0000 7fc239b96700 1 -- 172.21.15.60:0/2385044695 >> 172.21.15.60:0/2385044695 conn(0x7fc2340b5840 msgr2=0x7fc2340b0070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:08.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.926+0000 7fc239b96700 1 -- 172.21.15.60:0/2385044695 shutdown_connections 2022-06-29T16:05:08.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:08.926+0000 7fc239b96700 1 -- 172.21.15.60:0/2385044695 wait complete. 2022-06-29T16:05:08.935 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2022-06-29T16:05:08.946 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-06-29T16:05:09.063 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.054+0000 7fd060eb0700 1 Processor -- start 2022-06-29T16:05:09.063 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.054+0000 7fd060eb0700 1 -- start start 2022-06-29T16:05:09.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.058+0000 7fd060eb0700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd05c0b48b0 0x7fd05c142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:09.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.058+0000 7fd060eb0700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd05c0b2c50 0x7fd05c0b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:09.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.058+0000 7fd060eb0700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd05c0b0cf0 con 0x7fd05c0b5eb0 2022-06-29T16:05:09.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.058+0000 7fd060eb0700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd05c0b54e0 con 0x7fd05c0b48b0 2022-06-29T16:05:09.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.058+0000 7fd060eb0700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd05c0b4d10 con 0x7fd05c0b2c50 2022-06-29T16:05:09.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.058+0000 7fd05affd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd05c0b48b0 0x7fd05c142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:09.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.058+0000 7fd05bfff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd05c0b2c50 0x7fd05c0b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:09.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.058+0000 7fd05bfff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd05c0b2c50 0x7fd05c0b30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:55812/0 (socket says 172.21.15.60:55812) 2022-06-29T16:05:09.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.058+0000 7fd05affd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd05c0b48b0 0x7fd05c142a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49784/0 (socket says 172.21.15.60:49784) 2022-06-29T16:05:09.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.058+0000 7fd05bfff700 1 -- 172.21.15.60:0/3362276248 learned_addr learned my addr 172.21.15.60:0/3362276248 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:09.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.058+0000 7fd05bfff700 1 -- 172.21.15.60:0/3362276248 >> v2:172.21.15.60:3300/0 conn(0x7fd05c0b48b0 msgr2=0x7fd05c142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:09.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.058+0000 7fd05bfff700 1 --2- 172.21.15.60:0/3362276248 >> v2:172.21.15.60:3300/0 conn(0x7fd05c0b48b0 0x7fd05c142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:09.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.058+0000 7fd05bfff700 1 -- 172.21.15.60:0/3362276248 >> v1:172.21.15.92:6789/0 conn(0x7fd05c0b5eb0 legacy=0x7fd05c0b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:09.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.058+0000 7fd05bfff700 1 -- 172.21.15.60:0/3362276248 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd05c0b6290 con 0x7fd05c0b2c50 2022-06-29T16:05:09.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.058+0000 7fd05bfff700 1 --2- 172.21.15.60:0/3362276248 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd05c0b2c50 0x7fd05c0b30c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fd054002f70 tx=0x7fd054006eb0).ready entity=mon.1 client_cookie=31c299a076de403c server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:09.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.058+0000 7fd05a7fc700 1 -- 172.21.15.60:0/3362276248 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd05400d5c0 con 0x7fd05c0b2c50 2022-06-29T16:05:09.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.058+0000 7fd05a7fc700 1 -- 172.21.15.60:0/3362276248 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd054000e90 con 0x7fd05c0b2c50 2022-06-29T16:05:09.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.058+0000 7fd05a7fc700 1 -- 172.21.15.60:0/3362276248 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd05400d950 con 0x7fd05c0b2c50 2022-06-29T16:05:09.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.058+0000 7fd060eb0700 1 -- 172.21.15.60:0/3362276248 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd05c0b2c50 msgr2=0x7fd05c0b30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:09.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.058+0000 7fd060eb0700 1 --2- 172.21.15.60:0/3362276248 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd05c0b2c50 0x7fd05c0b30c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fd054002f70 tx=0x7fd054006eb0).stop 2022-06-29T16:05:09.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.058+0000 7fd060eb0700 1 -- 172.21.15.60:0/3362276248 shutdown_connections 2022-06-29T16:05:09.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.058+0000 7fd060eb0700 1 --2- 172.21.15.60:0/3362276248 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd05c0b2c50 0x7fd05c0b30c0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:09.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.058+0000 7fd060eb0700 1 --2- 172.21.15.60:0/3362276248 >> v2:172.21.15.60:3300/0 conn(0x7fd05c0b48b0 0x7fd05c142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:09.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.058+0000 7fd060eb0700 1 -- 172.21.15.60:0/3362276248 >> 172.21.15.60:0/3362276248 conn(0x7fd05c062090 msgr2=0x7fd05c062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:09.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.058+0000 7fd060eb0700 1 -- 172.21.15.60:0/3362276248 shutdown_connections 2022-06-29T16:05:09.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.058+0000 7fd060eb0700 1 -- 172.21.15.60:0/3362276248 wait complete. 2022-06-29T16:05:09.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.062+0000 7fd060eb0700 1 Processor -- start 2022-06-29T16:05:09.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.062+0000 7fd060eb0700 1 -- start start 2022-06-29T16:05:09.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.062+0000 7fd060eb0700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd05c0b2c50 0x7fd05c114280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:09.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.062+0000 7fd060eb0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd05c0b48b0 0x7fd05c0b8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:09.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.062+0000 7fd060eb0700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd05c0b5eb0 0x7fd05c145da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:09.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.062+0000 7fd060eb0700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c144130 con 0x7fd05c0b48b0 2022-06-29T16:05:09.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.062+0000 7fd060eb0700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c144fd0 con 0x7fd05c0b2c50 2022-06-29T16:05:09.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.062+0000 7fd060eb0700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c155430 con 0x7fd05c0b5eb0 2022-06-29T16:05:09.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.062+0000 7fd05affd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd05c0b48b0 0x7fd05c0b8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:09.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.062+0000 7fd05affd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd05c0b48b0 0x7fd05c0b8230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49788/0 (socket says 172.21.15.60:49788) 2022-06-29T16:05:09.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.062+0000 7fd05affd700 1 -- 172.21.15.60:0/2983950416 learned_addr learned my addr 172.21.15.60:0/2983950416 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:09.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.062+0000 7fd05bfff700 1 --2- 172.21.15.60:0/2983950416 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd05c0b5eb0 0x7fd05c145da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:09.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.062+0000 7fd05b7fe700 1 --2- 172.21.15.60:0/2983950416 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd05c0b2c50 0x7fd05c114280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:09.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.062+0000 7fd05bfff700 1 -- 172.21.15.60:0/2983950416 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd05c0b2c50 msgr2=0x7fd05c114280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:09.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.062+0000 7fd05bfff700 1 --2- 172.21.15.60:0/2983950416 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd05c0b2c50 0x7fd05c114280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:09.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.062+0000 7fd05bfff700 1 -- 172.21.15.60:0/2983950416 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd05c0b48b0 msgr2=0x7fd05c0b8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:09.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.062+0000 7fd05bfff700 1 --2- 172.21.15.60:0/2983950416 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd05c0b48b0 0x7fd05c0b8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:09.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.062+0000 7fd05bfff700 1 -- 172.21.15.60:0/2983950416 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd05400b1e0 con 0x7fd05c0b5eb0 2022-06-29T16:05:09.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.062+0000 7fd05affd700 1 --2- 172.21.15.60:0/2983950416 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd05c0b48b0 0x7fd05c0b8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:05:09.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.062+0000 7fd05bfff700 1 --2- 172.21.15.60:0/2983950416 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd05c0b5eb0 0x7fd05c145da0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fd05c113710 tx=0x7fd054015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:09.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.062+0000 7fd058ff9700 1 -- 172.21.15.60:0/2983950416 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0540116e0 con 0x7fd05c0b5eb0 2022-06-29T16:05:09.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.062+0000 7fd060eb0700 1 -- 172.21.15.60:0/2983950416 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd05c146290 con 0x7fd05c0b5eb0 2022-06-29T16:05:09.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.062+0000 7fd058ff9700 1 -- 172.21.15.60:0/2983950416 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd054011840 con 0x7fd05c0b5eb0 2022-06-29T16:05:09.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.062+0000 7fd058ff9700 1 -- 172.21.15.60:0/2983950416 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd054011ba0 con 0x7fd05c0b5eb0 2022-06-29T16:05:09.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.062+0000 7fd060eb0700 1 -- 172.21.15.60:0/2983950416 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd05c144ce0 con 0x7fd05c0b5eb0 2022-06-29T16:05:09.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.066+0000 7fd060eb0700 1 -- 172.21.15.60:0/2983950416 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd05c145190 con 0x7fd05c0b5eb0 2022-06-29T16:05:09.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.066+0000 7fd058ff9700 1 -- 172.21.15.60:0/2983950416 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd05401a040 con 0x7fd05c0b5eb0 2022-06-29T16:05:09.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.066+0000 7fd058ff9700 1 --2- 172.21.15.60:0/2983950416 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd0400539e0 0x7fd040055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:09.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.066+0000 7fd058ff9700 1 -- 172.21.15.60:0/2983950416 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fd05406cc00 con 0x7fd05c0b5eb0 2022-06-29T16:05:09.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.066+0000 7fd05b7fe700 1 --2- 172.21.15.60:0/2983950416 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd0400539e0 0x7fd040055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:09.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.066+0000 7fd05b7fe700 1 --2- 172.21.15.60:0/2983950416 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd0400539e0 0x7fd040055ea0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fd04c00d000 tx=0x7fd04c007ef0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:09.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.074+0000 7fd058ff9700 1 -- 172.21.15.60:0/2983950416 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd054043e10 con 0x7fd05c0b5eb0 2022-06-29T16:05:09.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.202+0000 7fd060eb0700 1 -- 172.21.15.60:0/2983950416 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fd05c1555c0 con 0x7fd05c0b5eb0 2022-06-29T16:05:09.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.210+0000 7fd058ff9700 1 -- 172.21.15.60:0/2983950416 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd054023020 con 0x7fd05c0b5eb0 2022-06-29T16:05:09.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.214+0000 7fd060eb0700 1 -- 172.21.15.60:0/2983950416 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd0400539e0 msgr2=0x7fd040055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:09.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.214+0000 7fd060eb0700 1 --2- 172.21.15.60:0/2983950416 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd0400539e0 0x7fd040055ea0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fd04c00d000 tx=0x7fd04c007ef0).stop 2022-06-29T16:05:09.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.214+0000 7fd060eb0700 1 -- 172.21.15.60:0/2983950416 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd05c0b5eb0 msgr2=0x7fd05c145da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:09.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.214+0000 7fd060eb0700 1 --2- 172.21.15.60:0/2983950416 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd05c0b5eb0 0x7fd05c145da0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fd05c113710 tx=0x7fd054015000).stop 2022-06-29T16:05:09.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.214+0000 7fd060eb0700 1 -- 172.21.15.60:0/2983950416 shutdown_connections 2022-06-29T16:05:09.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.214+0000 7fd060eb0700 1 --2- 172.21.15.60:0/2983950416 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd0400539e0 0x7fd040055ea0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:09.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.214+0000 7fd060eb0700 1 --2- 172.21.15.60:0/2983950416 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd05c0b5eb0 0x7fd05c145da0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:09.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.214+0000 7fd060eb0700 1 --2- 172.21.15.60:0/2983950416 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd05c0b48b0 0x7fd05c0b8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:09.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.214+0000 7fd060eb0700 1 --2- 172.21.15.60:0/2983950416 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd05c0b2c50 0x7fd05c114280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:09.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.214+0000 7fd060eb0700 1 -- 172.21.15.60:0/2983950416 >> 172.21.15.60:0/2983950416 conn(0x7fd05c062090 msgr2=0x7fd05c05bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:09.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.214+0000 7fd060eb0700 1 -- 172.21.15.60:0/2983950416 shutdown_connections 2022-06-29T16:05:09.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.214+0000 7fd060eb0700 1 -- 172.21.15.60:0/2983950416 wait complete. 2022-06-29T16:05:09.235 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_cap_revoke_nonresponder' 2022-06-29T16:05:09.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.330+0000 7fc564c3c700 1 Processor -- start 2022-06-29T16:05:09.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.330+0000 7fc564c3c700 1 -- start start 2022-06-29T16:05:09.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.330+0000 7fc564c3c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc5600b5950 0x7fc560067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:09.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.330+0000 7fc564c3c700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5600acba0 con 0x7fc56014af10 2022-06-29T16:05:09.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.330+0000 7fc564c3c700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc560067220 con 0x7fc560063ad0 2022-06-29T16:05:09.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.330+0000 7fc564c3c700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc56014b3a0 con 0x7fc5600b5950 2022-06-29T16:05:09.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.330+0000 7fc55e59c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc5600b5950 0x7fc560067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:09.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.330+0000 7fc55ed9d700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc56014af10 0x7fc56005c430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57884/0 (socket says 172.21.15.60:57884) 2022-06-29T16:05:09.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.330+0000 7fc55ed9d700 1 -- 172.21.15.60:0/1094600172 learned_addr learned my addr 172.21.15.60:0/1094600172 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:09.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.330+0000 7fc55e59c700 1 -- 172.21.15.60:0/1094600172 >> v1:172.21.15.92:6789/0 conn(0x7fc560063ad0 legacy=0x7fc56005bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:09.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.330+0000 7fc55e59c700 1 -- 172.21.15.60:0/1094600172 >> v1:172.21.15.60:6789/0 conn(0x7fc56014af10 legacy=0x7fc56005c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:09.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.330+0000 7fc55e59c700 1 -- 172.21.15.60:0/1094600172 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5600b5d30 con 0x7fc5600b5950 2022-06-29T16:05:09.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.334+0000 7fc55e59c700 1 --2- 172.21.15.60:0/1094600172 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5600b5950 0x7fc560067680 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 rx=0x7fc554002f70 tx=0x7fc554006df0).ready entity=mon.0 client_cookie=45ac102b3003fb7e server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:09.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.334+0000 7fc55d59a700 1 -- 172.21.15.60:0/1094600172 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc55400d610 con 0x7fc5600b5950 2022-06-29T16:05:09.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.334+0000 7fc55d59a700 1 -- 172.21.15.60:0/1094600172 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc554000e90 con 0x7fc5600b5950 2022-06-29T16:05:09.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.334+0000 7fc55d59a700 1 -- 172.21.15.60:0/1094600172 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc55400d960 con 0x7fc5600b5950 2022-06-29T16:05:09.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.334+0000 7fc564c3c700 1 -- 172.21.15.60:0/1094600172 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5600b5950 msgr2=0x7fc560067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:09.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.334+0000 7fc564c3c700 1 --2- 172.21.15.60:0/1094600172 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5600b5950 0x7fc560067680 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 rx=0x7fc554002f70 tx=0x7fc554006df0).stop 2022-06-29T16:05:09.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.334+0000 7fc564c3c700 1 -- 172.21.15.60:0/1094600172 shutdown_connections 2022-06-29T16:05:09.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.334+0000 7fc564c3c700 1 --2- 172.21.15.60:0/1094600172 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc5600b5950 0x7fc560067680 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:09.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.334+0000 7fc564c3c700 1 -- 172.21.15.60:0/1094600172 >> 172.21.15.60:0/1094600172 conn(0x7fc5600632c0 msgr2=0x7fc5600b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:09.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.334+0000 7fc564c3c700 1 -- 172.21.15.60:0/1094600172 shutdown_connections 2022-06-29T16:05:09.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.334+0000 7fc564c3c700 1 -- 172.21.15.60:0/1094600172 wait complete. 2022-06-29T16:05:09.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.338+0000 7fc564c3c700 1 Processor -- start 2022-06-29T16:05:09.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.338+0000 7fc564c3c700 1 -- start start 2022-06-29T16:05:09.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.338+0000 7fc564c3c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc560063ad0 0x7fc560156b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:09.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.338+0000 7fc564c3c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5600b5950 0x7fc5600b0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:09.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.338+0000 7fc564c3c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc56014af10 0x7fc5600b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:09.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.338+0000 7fc564c3c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5600af790 con 0x7fc56014af10 2022-06-29T16:05:09.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.338+0000 7fc564c3c700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc560156160 con 0x7fc5600b5950 2022-06-29T16:05:09.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.338+0000 7fc564c3c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc560156400 con 0x7fc560063ad0 2022-06-29T16:05:09.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.338+0000 7fc55ed9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc56014af10 0x7fc5600b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:09.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.338+0000 7fc55ed9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc56014af10 0x7fc5600b3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49804/0 (socket says 172.21.15.60:49804) 2022-06-29T16:05:09.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.338+0000 7fc55ed9d700 1 -- 172.21.15.60:0/273993466 learned_addr learned my addr 172.21.15.60:0/273993466 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:09.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.338+0000 7fc55e59c700 1 --2- 172.21.15.60:0/273993466 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc560063ad0 0x7fc560156b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:09.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.338+0000 7fc55dd9b700 1 --2- 172.21.15.60:0/273993466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5600b5950 0x7fc5600b0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:09.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.338+0000 7fc55ed9d700 1 -- 172.21.15.60:0/273993466 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc560063ad0 msgr2=0x7fc560156b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:09.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.338+0000 7fc55ed9d700 1 --2- 172.21.15.60:0/273993466 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc560063ad0 0x7fc560156b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:09.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.338+0000 7fc55ed9d700 1 -- 172.21.15.60:0/273993466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5600b5950 msgr2=0x7fc5600b0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:09.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.338+0000 7fc55ed9d700 1 --2- 172.21.15.60:0/273993466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5600b5950 0x7fc5600b0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:09.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.338+0000 7fc55ed9d700 1 -- 172.21.15.60:0/273993466 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc55400b180 con 0x7fc56014af10 2022-06-29T16:05:09.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.338+0000 7fc55ed9d700 1 --2- 172.21.15.60:0/273993466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc56014af10 0x7fc5600b3f80 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0x7fc5600ad0a0 tx=0x7fc558007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:09.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.338+0000 7fc54f7fe700 1 -- 172.21.15.60:0/273993466 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc558007e50 con 0x7fc56014af10 2022-06-29T16:05:09.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.338+0000 7fc54f7fe700 1 -- 172.21.15.60:0/273993466 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5580070d0 con 0x7fc56014af10 2022-06-29T16:05:09.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.338+0000 7fc564c3c700 1 -- 172.21.15.60:0/273993466 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5601566a0 con 0x7fc56014af10 2022-06-29T16:05:09.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.338+0000 7fc54f7fe700 1 -- 172.21.15.60:0/273993466 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc558007430 con 0x7fc56014af10 2022-06-29T16:05:09.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.342+0000 7fc564c3c700 1 -- 172.21.15.60:0/273993466 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc560157120 con 0x7fc56014af10 2022-06-29T16:05:09.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.342+0000 7fc564c3c700 1 -- 172.21.15.60:0/273993466 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc560156830 con 0x7fc56014af10 2022-06-29T16:05:09.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.342+0000 7fc54f7fe700 1 -- 172.21.15.60:0/273993466 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc55801a060 con 0x7fc56014af10 2022-06-29T16:05:09.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.342+0000 7fc54f7fe700 1 --2- 172.21.15.60:0/273993466 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc53c053aa0 0x7fc53c055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:09.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.342+0000 7fc54f7fe700 1 -- 172.21.15.60:0/273993466 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fc55806ad40 con 0x7fc56014af10 2022-06-29T16:05:09.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.342+0000 7fc55e59c700 1 --2- 172.21.15.60:0/273993466 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc53c053aa0 0x7fc53c055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:09.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.342+0000 7fc55e59c700 1 --2- 172.21.15.60:0/273993466 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc53c053aa0 0x7fc53c055f60 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fc55401c000 tx=0x7fc554015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:09.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.350+0000 7fc54f7fe700 1 -- 172.21.15.60:0/273993466 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc558041f10 con 0x7fc56014af10 2022-06-29T16:05:09.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.514+0000 7fc564c3c700 1 -- 172.21.15.60:0/273993466 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_cap_revoke_nonresponder"]} v 0) v1 -- 0x7fc5600573d0 con 0x7fc56014af10 2022-06-29T16:05:09.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.934+0000 7fc54f7fe700 1 -- 172.21.15.60:0/273993466 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_cap_revoke_nonresponder"]}]=0 v291) v1 ==== 139+0+0 (secure 0 0 0) 0x7fc558018070 con 0x7fc56014af10 2022-06-29T16:05:09.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.938+0000 7fc564c3c700 1 -- 172.21.15.60:0/273993466 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc53c053aa0 msgr2=0x7fc53c055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:09.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.938+0000 7fc564c3c700 1 --2- 172.21.15.60:0/273993466 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc53c053aa0 0x7fc53c055f60 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fc55401c000 tx=0x7fc554015000).stop 2022-06-29T16:05:09.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.938+0000 7fc564c3c700 1 -- 172.21.15.60:0/273993466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc56014af10 msgr2=0x7fc5600b3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:09.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.938+0000 7fc564c3c700 1 --2- 172.21.15.60:0/273993466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc56014af10 0x7fc5600b3f80 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0x7fc5600ad0a0 tx=0x7fc558007c00).stop 2022-06-29T16:05:09.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.938+0000 7fc564c3c700 1 -- 172.21.15.60:0/273993466 shutdown_connections 2022-06-29T16:05:09.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.938+0000 7fc564c3c700 1 --2- 172.21.15.60:0/273993466 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc53c053aa0 0x7fc53c055f60 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:09.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.938+0000 7fc564c3c700 1 --2- 172.21.15.60:0/273993466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc56014af10 0x7fc5600b3f80 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:09.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.938+0000 7fc564c3c700 1 --2- 172.21.15.60:0/273993466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5600b5950 0x7fc5600b0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:09.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.938+0000 7fc564c3c700 1 --2- 172.21.15.60:0/273993466 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc560063ad0 0x7fc560156b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:09.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.938+0000 7fc564c3c700 1 -- 172.21.15.60:0/273993466 >> 172.21.15.60:0/273993466 conn(0x7fc5600632c0 msgr2=0x7fc5600b4ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:09.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.938+0000 7fc564c3c700 1 -- 172.21.15.60:0/273993466 shutdown_connections 2022-06-29T16:05:09.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:09.938+0000 7fc564c3c700 1 -- 172.21.15.60:0/273993466 wait complete. 2022-06-29T16:05:09.961 INFO:tasks.cephfs_test_runner:test_cap_revoke_nonresponder (tasks.cephfs.test_misc.TestMisc) ... ok 2022-06-29T16:05:09.961 INFO:tasks.cephfs_test_runner:Starting test: test_dump_inode (tasks.cephfs.test_misc.TestMisc) 2022-06-29T16:05:09.962 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_dump_inode' 2022-06-29T16:05:10.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.062+0000 7fc12759e700 1 Processor -- start 2022-06-29T16:05:10.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.062+0000 7fc12759e700 1 -- start start 2022-06-29T16:05:10.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.066+0000 7fc12759e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc12805d070 0x7fc12814aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:10.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.066+0000 7fc12759e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc1280b2130 0x7fc1280aca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:10.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.066+0000 7fc12759e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc12805d450 con 0x7fc12814af80 2022-06-29T16:05:10.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.066+0000 7fc12759e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1280adf90 con 0x7fc1280b2130 2022-06-29T16:05:10.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.066+0000 7fc12759e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1280b25d0 con 0x7fc12805d070 2022-06-29T16:05:10.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.066+0000 7fc125d9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc1280b2130 0x7fc1280aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:10.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.066+0000 7fc126d9d700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc12814af80 0x7fc1280acf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57896/0 (socket says 172.21.15.60:57896) 2022-06-29T16:05:10.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.066+0000 7fc126d9d700 1 -- 172.21.15.60:0/2571475411 learned_addr learned my addr 172.21.15.60:0/2571475411 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:10.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.066+0000 7fc125d9b700 1 --2- 172.21.15.60:0/2571475411 >> v2:172.21.15.60:3300/0 conn(0x7fc1280b2130 0x7fc1280aca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49808/0 (socket says 172.21.15.60:49808) 2022-06-29T16:05:10.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.066+0000 7fc12659c700 1 --2- 172.21.15.60:0/2571475411 >> v2:172.21.15.92:3300/0 conn(0x7fc12805d070 0x7fc12814aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:10.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.066+0000 7fc125d9b700 1 -- 172.21.15.60:0/2571475411 >> v2:172.21.15.92:3300/0 conn(0x7fc12805d070 msgr2=0x7fc12814aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:10.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.066+0000 7fc125d9b700 1 --2- 172.21.15.60:0/2571475411 >> v2:172.21.15.92:3300/0 conn(0x7fc12805d070 0x7fc12814aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:10.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.066+0000 7fc125d9b700 1 -- 172.21.15.60:0/2571475411 >> v1:172.21.15.60:6789/0 conn(0x7fc12814af80 legacy=0x7fc1280acf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:10.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.066+0000 7fc125d9b700 1 -- 172.21.15.60:0/2571475411 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12814a350 con 0x7fc1280b2130 2022-06-29T16:05:10.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.066+0000 7fc125d9b700 1 --2- 172.21.15.60:0/2571475411 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1280b2130 0x7fc1280aca20 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 rx=0x7fc118002f70 tx=0x7fc118006df0).ready entity=mon.0 client_cookie=d5a80abac01436b4 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:10.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.066+0000 7fc12559a700 1 -- 172.21.15.60:0/2571475411 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc11800d5c0 con 0x7fc1280b2130 2022-06-29T16:05:10.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.066+0000 7fc12559a700 1 -- 172.21.15.60:0/2571475411 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc118000e90 con 0x7fc1280b2130 2022-06-29T16:05:10.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.066+0000 7fc12559a700 1 -- 172.21.15.60:0/2571475411 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc11800d910 con 0x7fc1280b2130 2022-06-29T16:05:10.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.066+0000 7fc12759e700 1 -- 172.21.15.60:0/2571475411 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1280b2130 msgr2=0x7fc1280aca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:10.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.066+0000 7fc12759e700 1 --2- 172.21.15.60:0/2571475411 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1280b2130 0x7fc1280aca20 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 rx=0x7fc118002f70 tx=0x7fc118006df0).stop 2022-06-29T16:05:10.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.066+0000 7fc12759e700 1 -- 172.21.15.60:0/2571475411 shutdown_connections 2022-06-29T16:05:10.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.066+0000 7fc12759e700 1 --2- 172.21.15.60:0/2571475411 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1280b2130 0x7fc1280aca20 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:10.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.066+0000 7fc12759e700 1 --2- 172.21.15.60:0/2571475411 >> v2:172.21.15.92:3300/0 conn(0x7fc12805d070 0x7fc12814aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:10.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.066+0000 7fc12759e700 1 -- 172.21.15.60:0/2571475411 >> 172.21.15.60:0/2571475411 conn(0x7fc1280b5890 msgr2=0x7fc1280b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:10.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.066+0000 7fc12759e700 1 -- 172.21.15.60:0/2571475411 shutdown_connections 2022-06-29T16:05:10.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.066+0000 7fc12759e700 1 -- 172.21.15.60:0/2571475411 wait complete. 2022-06-29T16:05:10.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.066+0000 7fc12759e700 1 Processor -- start 2022-06-29T16:05:10.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.066+0000 7fc12759e700 1 -- start start 2022-06-29T16:05:10.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.070+0000 7fc12759e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc12805d070 0x7fc1281190e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:10.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.070+0000 7fc12759e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1280b2130 0x7fc128116920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:10.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.070+0000 7fc12759e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc12814af80 0x7fc128116e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:10.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.070+0000 7fc12759e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc128064840 con 0x7fc12805d070 2022-06-29T16:05:10.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.070+0000 7fc12759e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1281173f0 con 0x7fc12814af80 2022-06-29T16:05:10.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.070+0000 7fc12759e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1281189a0 con 0x7fc1280b2130 2022-06-29T16:05:10.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.070+0000 7fc12659c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc12805d070 0x7fc1281190e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:10.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.070+0000 7fc12659c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc12805d070 0x7fc1281190e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49814/0 (socket says 172.21.15.60:49814) 2022-06-29T16:05:10.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.070+0000 7fc12659c700 1 -- 172.21.15.60:0/859890925 learned_addr learned my addr 172.21.15.60:0/859890925 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:10.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.070+0000 7fc125d9b700 1 --2- 172.21.15.60:0/859890925 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1280b2130 0x7fc128116920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:10.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.070+0000 7fc126d9d700 1 --2- 172.21.15.60:0/859890925 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc12814af80 0x7fc128116e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:10.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.070+0000 7fc12659c700 1 -- 172.21.15.60:0/859890925 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1280b2130 msgr2=0x7fc128116920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:10.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.070+0000 7fc12659c700 1 --2- 172.21.15.60:0/859890925 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1280b2130 0x7fc128116920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:10.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.070+0000 7fc12659c700 1 -- 172.21.15.60:0/859890925 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc12814af80 msgr2=0x7fc128116e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:10.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.070+0000 7fc12659c700 1 --2- 172.21.15.60:0/859890925 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc12814af80 0x7fc128116e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:10.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.070+0000 7fc12659c700 1 -- 172.21.15.60:0/859890925 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc11800b180 con 0x7fc12805d070 2022-06-29T16:05:10.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.070+0000 7fc125d9b700 1 --2- 172.21.15.60:0/859890925 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1280b2130 0x7fc128116920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:05:10.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.070+0000 7fc126d9d700 1 --2- 172.21.15.60:0/859890925 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc12814af80 0x7fc128116e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:05:10.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.070+0000 7fc12659c700 1 --2- 172.21.15.60:0/859890925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc12805d070 0x7fc1281190e0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 rx=0x7fc128064980 tx=0x7fc110008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:10.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.070+0000 7fc11f7fe700 1 -- 172.21.15.60:0/859890925 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc110011780 con 0x7fc12805d070 2022-06-29T16:05:10.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.070+0000 7fc11f7fe700 1 -- 172.21.15.60:0/859890925 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1100118e0 con 0x7fc12805d070 2022-06-29T16:05:10.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.070+0000 7fc12759e700 1 -- 172.21.15.60:0/859890925 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc128118c40 con 0x7fc12805d070 2022-06-29T16:05:10.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.070+0000 7fc11f7fe700 1 -- 172.21.15.60:0/859890925 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc110011c40 con 0x7fc12805d070 2022-06-29T16:05:10.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.070+0000 7fc12759e700 1 -- 172.21.15.60:0/859890925 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc128066da0 con 0x7fc12805d070 2022-06-29T16:05:10.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.070+0000 7fc12759e700 1 -- 172.21.15.60:0/859890925 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc12814a5f0 con 0x7fc12805d070 2022-06-29T16:05:10.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.070+0000 7fc11f7fe700 1 -- 172.21.15.60:0/859890925 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc11001c460 con 0x7fc12805d070 2022-06-29T16:05:10.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.070+0000 7fc11f7fe700 1 --2- 172.21.15.60:0/859890925 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc10c053b00 0x7fc10c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:10.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.070+0000 7fc11f7fe700 1 -- 172.21.15.60:0/859890925 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fc11006be60 con 0x7fc12805d070 2022-06-29T16:05:10.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.074+0000 7fc125d9b700 1 --2- 172.21.15.60:0/859890925 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc10c053b00 0x7fc10c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:10.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.074+0000 7fc125d9b700 1 --2- 172.21.15.60:0/859890925 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc10c053b00 0x7fc10c055fc0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7fc118015000 tx=0x7fc118014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:10.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.078+0000 7fc11f7fe700 1 -- 172.21.15.60:0/859890925 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc110042fd0 con 0x7fc12805d070 2022-06-29T16:05:10.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.250+0000 7fc12759e700 1 -- 172.21.15.60:0/859890925 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_dump_inode"]} v 0) v1 -- 0x7fc1280573d0 con 0x7fc12805d070 2022-06-29T16:05:10.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.934+0000 7fc11f7fe700 1 -- 172.21.15.60:0/859890925 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_dump_inode"]}]=0 v292) v1 ==== 129+0+0 (secure 0 0 0) 0x7fc1100465f0 con 0x7fc12805d070 2022-06-29T16:05:10.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.938+0000 7fc12759e700 1 -- 172.21.15.60:0/859890925 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc10c053b00 msgr2=0x7fc10c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:10.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.938+0000 7fc12759e700 1 --2- 172.21.15.60:0/859890925 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc10c053b00 0x7fc10c055fc0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7fc118015000 tx=0x7fc118014000).stop 2022-06-29T16:05:10.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.938+0000 7fc12759e700 1 -- 172.21.15.60:0/859890925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc12805d070 msgr2=0x7fc1281190e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:10.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.938+0000 7fc12759e700 1 --2- 172.21.15.60:0/859890925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc12805d070 0x7fc1281190e0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 rx=0x7fc128064980 tx=0x7fc110008040).stop 2022-06-29T16:05:10.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.938+0000 7fc12759e700 1 -- 172.21.15.60:0/859890925 shutdown_connections 2022-06-29T16:05:10.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.938+0000 7fc12759e700 1 --2- 172.21.15.60:0/859890925 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc10c053b00 0x7fc10c055fc0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:10.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.938+0000 7fc12759e700 1 --2- 172.21.15.60:0/859890925 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc12814af80 0x7fc128116e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:10.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.938+0000 7fc12759e700 1 --2- 172.21.15.60:0/859890925 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1280b2130 0x7fc128116920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:10.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.938+0000 7fc12759e700 1 --2- 172.21.15.60:0/859890925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc12805d070 0x7fc1281190e0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:10.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.938+0000 7fc12759e700 1 -- 172.21.15.60:0/859890925 >> 172.21.15.60:0/859890925 conn(0x7fc1280b5890 msgr2=0x7fc128147130 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:10.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.938+0000 7fc12759e700 1 -- 172.21.15.60:0/859890925 shutdown_connections 2022-06-29T16:05:10.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:10.938+0000 7fc12759e700 1 -- 172.21.15.60:0/859890925 wait complete. 2022-06-29T16:05:10.962 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-06-29T16:05:11.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.074+0000 7f7a6b726700 1 Processor -- start 2022-06-29T16:05:11.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.074+0000 7f7a6b726700 1 -- start start 2022-06-29T16:05:11.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.074+0000 7f7a6b726700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7a640ae7c0 0x7f7a64141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:11.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.074+0000 7f7a6b726700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7a640b46c0 0x7f7a640b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:11.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.074+0000 7f7a6b726700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a64143a40 con 0x7f7a640b4010 2022-06-29T16:05:11.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.074+0000 7f7a6b726700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a640aee60 con 0x7f7a640b46c0 2022-06-29T16:05:11.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.074+0000 7f7a6b726700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a640af100 con 0x7f7a640ae7c0 2022-06-29T16:05:11.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.074+0000 7f7a69cc3700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7a640b46c0 0x7f7a640b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:11.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.074+0000 7f7a68cc1700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f7a640b4010 0x7f7a64142490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57910/0 (socket says 172.21.15.60:57910) 2022-06-29T16:05:11.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.074+0000 7f7a68cc1700 1 -- 172.21.15.60:0/2842803158 learned_addr learned my addr 172.21.15.60:0/2842803158 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:11.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.074+0000 7f7a694c2700 1 --2- 172.21.15.60:0/2842803158 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7a640ae7c0 0x7f7a64141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:11.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.074+0000 7f7a57fff700 1 -- 172.21.15.60:0/2842803158 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3423289863 0 0) 0x7f7a64143a40 con 0x7f7a640b4010 2022-06-29T16:05:11.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.074+0000 7f7a57fff700 1 -- 172.21.15.60:0/2842803158 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a48003260 con 0x7f7a640b4010 2022-06-29T16:05:11.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.074+0000 7f7a69cc3700 1 -- 172.21.15.60:0/2842803158 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7a640ae7c0 msgr2=0x7f7a64141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:11.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.078+0000 7f7a69cc3700 1 --2- 172.21.15.60:0/2842803158 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7a640ae7c0 0x7f7a64141fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:11.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.078+0000 7f7a69cc3700 1 -- 172.21.15.60:0/2842803158 >> v1:172.21.15.60:6789/0 conn(0x7f7a640b4010 legacy=0x7f7a64142490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:11.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.078+0000 7f7a69cc3700 1 -- 172.21.15.60:0/2842803158 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a640aeba0 con 0x7f7a640b46c0 2022-06-29T16:05:11.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.078+0000 7f7a69cc3700 1 --2- 172.21.15.60:0/2842803158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a640b46c0 0x7f7a640b4b50 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 rx=0x7f7a5c0032f0 tx=0x7f7a5c0091d0).ready entity=mon.0 client_cookie=adb41e5e619de22f server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:11.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.078+0000 7f7a57fff700 1 -- 172.21.15.60:0/2842803158 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a5c00b6c0 con 0x7f7a640b46c0 2022-06-29T16:05:11.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.078+0000 7f7a57fff700 1 -- 172.21.15.60:0/2842803158 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a5c00b820 con 0x7f7a640b46c0 2022-06-29T16:05:11.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.078+0000 7f7a57fff700 1 -- 172.21.15.60:0/2842803158 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a5c00b990 con 0x7f7a640b46c0 2022-06-29T16:05:11.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.078+0000 7f7a6b726700 1 -- 172.21.15.60:0/2842803158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a640b46c0 msgr2=0x7f7a640b4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:11.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.078+0000 7f7a6b726700 1 --2- 172.21.15.60:0/2842803158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a640b46c0 0x7f7a640b4b50 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 rx=0x7f7a5c0032f0 tx=0x7f7a5c0091d0).stop 2022-06-29T16:05:11.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.078+0000 7f7a6b726700 1 -- 172.21.15.60:0/2842803158 shutdown_connections 2022-06-29T16:05:11.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.078+0000 7f7a6b726700 1 --2- 172.21.15.60:0/2842803158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a640b46c0 0x7f7a640b4b50 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:11.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.078+0000 7f7a6b726700 1 --2- 172.21.15.60:0/2842803158 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7a640ae7c0 0x7f7a64141fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:11.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.078+0000 7f7a6b726700 1 -- 172.21.15.60:0/2842803158 >> 172.21.15.60:0/2842803158 conn(0x7f7a640b9340 msgr2=0x7f7a640b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:11.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.078+0000 7f7a6b726700 1 -- 172.21.15.60:0/2842803158 shutdown_connections 2022-06-29T16:05:11.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.078+0000 7f7a6b726700 1 -- 172.21.15.60:0/2842803158 wait complete. 2022-06-29T16:05:11.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.078+0000 7f7a6b726700 1 Processor -- start 2022-06-29T16:05:11.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.078+0000 7f7a6b726700 1 -- start start 2022-06-29T16:05:11.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.078+0000 7f7a6b726700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a640ae7c0 0x7f7a64146a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:11.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.078+0000 7f7a6b726700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7a640b4010 0x7f7a640b3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:11.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.078+0000 7f7a6b726700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a640b46c0 0x7f7a640b0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:11.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.078+0000 7f7a6b726700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a640b0380 con 0x7f7a640ae7c0 2022-06-29T16:05:11.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.078+0000 7f7a6b726700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a640afe40 con 0x7f7a640b46c0 2022-06-29T16:05:11.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.078+0000 7f7a6b726700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a640b00e0 con 0x7f7a640b4010 2022-06-29T16:05:11.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.078+0000 7f7a694c2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a640ae7c0 0x7f7a64146a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:11.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.082+0000 7f7a694c2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a640ae7c0 0x7f7a64146a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49828/0 (socket says 172.21.15.60:49828) 2022-06-29T16:05:11.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.082+0000 7f7a694c2700 1 -- 172.21.15.60:0/1870097446 learned_addr learned my addr 172.21.15.60:0/1870097446 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:11.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.082+0000 7f7a69cc3700 1 --2- 172.21.15.60:0/1870097446 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a640b46c0 0x7f7a640b0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:11.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.082+0000 7f7a68cc1700 1 --2- 172.21.15.60:0/1870097446 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7a640b4010 0x7f7a640b3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:11.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.082+0000 7f7a694c2700 1 -- 172.21.15.60:0/1870097446 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7a640b4010 msgr2=0x7f7a640b3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:11.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.082+0000 7f7a694c2700 1 --2- 172.21.15.60:0/1870097446 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7a640b4010 0x7f7a640b3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:11.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.082+0000 7f7a694c2700 1 -- 172.21.15.60:0/1870097446 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a640b46c0 msgr2=0x7f7a640b0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:11.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.082+0000 7f7a694c2700 1 --2- 172.21.15.60:0/1870097446 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a640b46c0 0x7f7a640b0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:11.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.082+0000 7f7a694c2700 1 -- 172.21.15.60:0/1870097446 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a5c009470 con 0x7f7a640ae7c0 2022-06-29T16:05:11.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.082+0000 7f7a68cc1700 1 --2- 172.21.15.60:0/1870097446 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7a640b4010 0x7f7a640b3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:05:11.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.082+0000 7f7a69cc3700 1 --2- 172.21.15.60:0/1870097446 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a640b46c0 0x7f7a640b0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:05:11.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.082+0000 7f7a694c2700 1 --2- 172.21.15.60:0/1870097446 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a640ae7c0 0x7f7a64146a20 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 rx=0x7f7a580030b0 tx=0x7f7a58008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:11.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.082+0000 7f7a567fc700 1 -- 172.21.15.60:0/1870097446 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a58011780 con 0x7f7a640ae7c0 2022-06-29T16:05:11.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.082+0000 7f7a567fc700 1 -- 172.21.15.60:0/1870097446 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a580118e0 con 0x7f7a640ae7c0 2022-06-29T16:05:11.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.082+0000 7f7a6b726700 1 -- 172.21.15.60:0/1870097446 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a640b0d80 con 0x7f7a640ae7c0 2022-06-29T16:05:11.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.082+0000 7f7a567fc700 1 -- 172.21.15.60:0/1870097446 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a58011c40 con 0x7f7a640ae7c0 2022-06-29T16:05:11.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.082+0000 7f7a6b726700 1 -- 172.21.15.60:0/1870097446 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a641451e0 con 0x7f7a640ae7c0 2022-06-29T16:05:11.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.082+0000 7f7a567fc700 1 -- 172.21.15.60:0/1870097446 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7a5801c460 con 0x7f7a640ae7c0 2022-06-29T16:05:11.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.082+0000 7f7a6b726700 1 -- 172.21.15.60:0/1870097446 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a64146710 con 0x7f7a640ae7c0 2022-06-29T16:05:11.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.082+0000 7f7a567fc700 1 --2- 172.21.15.60:0/1870097446 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7a4c0539e0 0x7f7a4c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:11.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.082+0000 7f7a567fc700 1 -- 172.21.15.60:0/1870097446 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f7a5806bea0 con 0x7f7a640ae7c0 2022-06-29T16:05:11.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.082+0000 7f7a68cc1700 1 --2- 172.21.15.60:0/1870097446 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7a4c0539e0 0x7f7a4c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:11.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.086+0000 7f7a68cc1700 1 --2- 172.21.15.60:0/1870097446 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7a4c0539e0 0x7f7a4c055ea0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f7a60008000 tx=0x7f7a60009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:11.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.090+0000 7f7a567fc700 1 -- 172.21.15.60:0/1870097446 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7a58043010 con 0x7f7a640ae7c0 2022-06-29T16:05:11.230 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.222+0000 7f7a6b726700 1 -- 172.21.15.60:0/1870097446 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f7a640573d0 con 0x7f7a640ae7c0 2022-06-29T16:05:11.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.230+0000 7f7a567fc700 1 -- 172.21.15.60:0/1870097446 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7a58046630 con 0x7f7a640ae7c0 2022-06-29T16:05:11.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.234+0000 7f7a6b726700 1 -- 172.21.15.60:0/1870097446 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7a4c0539e0 msgr2=0x7f7a4c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:11.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.234+0000 7f7a6b726700 1 --2- 172.21.15.60:0/1870097446 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7a4c0539e0 0x7f7a4c055ea0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f7a60008000 tx=0x7f7a60009040).stop 2022-06-29T16:05:11.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.234+0000 7f7a6b726700 1 -- 172.21.15.60:0/1870097446 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a640ae7c0 msgr2=0x7f7a64146a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:11.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.234+0000 7f7a6b726700 1 --2- 172.21.15.60:0/1870097446 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a640ae7c0 0x7f7a64146a20 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 rx=0x7f7a580030b0 tx=0x7f7a58008040).stop 2022-06-29T16:05:11.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.234+0000 7f7a6b726700 1 -- 172.21.15.60:0/1870097446 shutdown_connections 2022-06-29T16:05:11.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.234+0000 7f7a6b726700 1 --2- 172.21.15.60:0/1870097446 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7a4c0539e0 0x7f7a4c055ea0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:11.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.234+0000 7f7a6b726700 1 --2- 172.21.15.60:0/1870097446 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a640b46c0 0x7f7a640b0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:11.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.234+0000 7f7a6b726700 1 --2- 172.21.15.60:0/1870097446 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7a640b4010 0x7f7a640b3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:11.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.234+0000 7f7a6b726700 1 --2- 172.21.15.60:0/1870097446 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a640ae7c0 0x7f7a64146a20 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:11.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.234+0000 7f7a6b726700 1 -- 172.21.15.60:0/1870097446 >> 172.21.15.60:0/1870097446 conn(0x7f7a640b9340 msgr2=0x7f7a640b8450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:11.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.234+0000 7f7a6b726700 1 -- 172.21.15.60:0/1870097446 shutdown_connections 2022-06-29T16:05:11.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.234+0000 7f7a6b726700 1 -- 172.21.15.60:0/1870097446 wait complete. 2022-06-29T16:05:11.253 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-06-29T16:05:11.253 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:05:11.273 DEBUG:teuthology.orchestra.run.smithi092:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:05:11.296 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:05:11.317 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:05:11.338 DEBUG:teuthology.orchestra.run.smithi197:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:05:11.360 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-06-29T16:05:11.360 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:11.361 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:05:11.361 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:05:11.368 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-06-29T16:05:11.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:05:11.370 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-06-29T16:05:11.370 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:11.371 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:05:11.371 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:05:11.378 INFO:teuthology.orchestra.run.smithi156.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-06-29T16:05:11.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:05:11.380 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-06-29T16:05:11.381 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:11.381 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:05:11.381 DEBUG:teuthology.orchestra.run.smithi197:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T16:05:11.408 INFO:teuthology.orchestra.run.smithi197.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-06-29T16:05:11.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:05:11.410 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-06-29T16:05:11.410 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-06-29T16:05:11.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.510+0000 7f3eb5907700 1 Processor -- start 2022-06-29T16:05:11.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.510+0000 7f3eb5907700 1 -- start start 2022-06-29T16:05:11.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.514+0000 7f3eb5907700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3eb00b9c20 0x7f3eb00b0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:11.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.514+0000 7f3eb5907700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3eb0146040 0x7f3eb00b24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:11.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.514+0000 7f3eb5907700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb00b1de0 con 0x7f3eb0146480 2022-06-29T16:05:11.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.514+0000 7f3eb5907700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eb00b1330 con 0x7f3eb00b9c20 2022-06-29T16:05:11.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.514+0000 7f3eb5907700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb00ba000 con 0x7f3eb0146040 2022-06-29T16:05:11.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.514+0000 7f3eb4905700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3eb00b9c20 0x7f3eb00b0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:11.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.514+0000 7f3eb5106700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f3eb0146480 0x7f3eb0141a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57924/0 (socket says 172.21.15.60:57924) 2022-06-29T16:05:11.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.514+0000 7f3eb5106700 1 -- 172.21.15.60:0/1126531181 learned_addr learned my addr 172.21.15.60:0/1126531181 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:11.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.514+0000 7f3eaffff700 1 --2- 172.21.15.60:0/1126531181 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3eb0146040 0x7f3eb00b24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:11.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.514+0000 7f3eb4905700 1 -- 172.21.15.60:0/1126531181 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3eb0146040 msgr2=0x7f3eb00b24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:11.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.514+0000 7f3eb4905700 1 --2- 172.21.15.60:0/1126531181 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3eb0146040 0x7f3eb00b24b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:11.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.514+0000 7f3eb4905700 1 -- 172.21.15.60:0/1126531181 >> v1:172.21.15.60:6789/0 conn(0x7f3eb0146480 legacy=0x7f3eb0141a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:11.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.514+0000 7f3eb4905700 1 -- 172.21.15.60:0/1126531181 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb00b00a0 con 0x7f3eb00b9c20 2022-06-29T16:05:11.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.514+0000 7f3eb4905700 1 --2- 172.21.15.60:0/1126531181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3eb00b9c20 0x7f3eb00b0610 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 rx=0x7f3ea0002f70 tx=0x7f3ea0006df0).ready entity=mon.0 client_cookie=d316f934462d4fb1 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:11.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.514+0000 7f3eaf7fe700 1 -- 172.21.15.60:0/1126531181 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ea000d570 con 0x7f3eb00b9c20 2022-06-29T16:05:11.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.514+0000 7f3eaf7fe700 1 -- 172.21.15.60:0/1126531181 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea0000e90 con 0x7f3eb00b9c20 2022-06-29T16:05:11.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.514+0000 7f3eaf7fe700 1 -- 172.21.15.60:0/1126531181 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ea000d8c0 con 0x7f3eb00b9c20 2022-06-29T16:05:11.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.514+0000 7f3eb5907700 1 -- 172.21.15.60:0/1126531181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3eb00b9c20 msgr2=0x7f3eb00b0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:11.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.514+0000 7f3eb5907700 1 --2- 172.21.15.60:0/1126531181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3eb00b9c20 0x7f3eb00b0610 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 rx=0x7f3ea0002f70 tx=0x7f3ea0006df0).stop 2022-06-29T16:05:11.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.514+0000 7f3eb5907700 1 -- 172.21.15.60:0/1126531181 shutdown_connections 2022-06-29T16:05:11.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.514+0000 7f3eb5907700 1 --2- 172.21.15.60:0/1126531181 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3eb0146040 0x7f3eb00b24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:11.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.514+0000 7f3eb5907700 1 --2- 172.21.15.60:0/1126531181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3eb00b9c20 0x7f3eb00b0610 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:11.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.514+0000 7f3eb5907700 1 -- 172.21.15.60:0/1126531181 >> 172.21.15.60:0/1126531181 conn(0x7f3eb0062cb0 msgr2=0x7f3eb00b91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:11.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.514+0000 7f3eb5907700 1 -- 172.21.15.60:0/1126531181 shutdown_connections 2022-06-29T16:05:11.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.514+0000 7f3eb5907700 1 -- 172.21.15.60:0/1126531181 wait complete. 2022-06-29T16:05:11.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.518+0000 7f3eb5907700 1 Processor -- start 2022-06-29T16:05:11.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.518+0000 7f3eb5907700 1 -- start start 2022-06-29T16:05:11.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.518+0000 7f3eb5907700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3eb00b9c20 0x7f3eb00b73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:11.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.518+0000 7f3eb5907700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3eb0146040 0x7f3eb0143f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:11.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.518+0000 7f3eb5907700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3eb0146480 0x7f3eb0152290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:11.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.518+0000 7f3eb5907700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb00b7b70 con 0x7f3eb00b9c20 2022-06-29T16:05:11.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.518+0000 7f3eb5907700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb00ae420 con 0x7f3eb0146040 2022-06-29T16:05:11.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.518+0000 7f3eb5907700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb00ad890 con 0x7f3eb0146480 2022-06-29T16:05:11.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.518+0000 7f3eb4905700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3eb00b9c20 0x7f3eb00b73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:11.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.518+0000 7f3eb4905700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3eb00b9c20 0x7f3eb00b73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49842/0 (socket says 172.21.15.60:49842) 2022-06-29T16:05:11.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.518+0000 7f3eb4905700 1 -- 172.21.15.60:0/915264137 learned_addr learned my addr 172.21.15.60:0/915264137 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:11.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.518+0000 7f3eb5106700 1 --2- 172.21.15.60:0/915264137 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3eb0146480 0x7f3eb0152290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:11.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.518+0000 7f3eaffff700 1 --2- 172.21.15.60:0/915264137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3eb0146040 0x7f3eb0143f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:11.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.518+0000 7f3eb4905700 1 -- 172.21.15.60:0/915264137 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3eb0146480 msgr2=0x7f3eb0152290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:11.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.518+0000 7f3eb4905700 1 --2- 172.21.15.60:0/915264137 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3eb0146480 0x7f3eb0152290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:11.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.518+0000 7f3eb4905700 1 -- 172.21.15.60:0/915264137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3eb0146040 msgr2=0x7f3eb0143f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:11.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.518+0000 7f3eb4905700 1 --2- 172.21.15.60:0/915264137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3eb0146040 0x7f3eb0143f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:11.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.518+0000 7f3eb4905700 1 -- 172.21.15.60:0/915264137 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea000b180 con 0x7f3eb00b9c20 2022-06-29T16:05:11.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.518+0000 7f3eb4905700 1 --2- 172.21.15.60:0/915264137 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3eb00b9c20 0x7f3eb00b73b0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 rx=0x7f3eb00ad420 tx=0x7f3ea0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:11.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.518+0000 7f3eadffb700 1 -- 172.21.15.60:0/915264137 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ea000d8f0 con 0x7f3eb00b9c20 2022-06-29T16:05:11.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.518+0000 7f3eb5907700 1 -- 172.21.15.60:0/915264137 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eb0151b50 con 0x7f3eb00b9c20 2022-06-29T16:05:11.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.518+0000 7f3eadffb700 1 -- 172.21.15.60:0/915264137 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea0011b40 con 0x7f3eb00b9c20 2022-06-29T16:05:11.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.518+0000 7f3eadffb700 1 -- 172.21.15.60:0/915264137 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ea0011ea0 con 0x7f3eb00b9c20 2022-06-29T16:05:11.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.518+0000 7f3eb5907700 1 -- 172.21.15.60:0/915264137 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb00ad530 con 0x7f3eb00b9c20 2022-06-29T16:05:11.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.522+0000 7f3eb5907700 1 -- 172.21.15.60:0/915264137 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb00573d0 con 0x7f3eb00b9c20 2022-06-29T16:05:11.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.522+0000 7f3eadffb700 1 -- 172.21.15.60:0/915264137 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3ea0025030 con 0x7f3eb00b9c20 2022-06-29T16:05:11.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.522+0000 7f3eadffb700 1 --2- 172.21.15.60:0/915264137 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3e94053a70 0x7f3e94055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:11.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.522+0000 7f3eadffb700 1 -- 172.21.15.60:0/915264137 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f3ea006cee0 con 0x7f3eb00b9c20 2022-06-29T16:05:11.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.522+0000 7f3eaffff700 1 --2- 172.21.15.60:0/915264137 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3e94053a70 0x7f3e94055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:11.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.522+0000 7f3eaffff700 1 --2- 172.21.15.60:0/915264137 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3e94053a70 0x7f3e94055f30 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f3e9800c000 tx=0x7f3e98008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:11.539 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.526+0000 7f3eadffb700 1 -- 172.21.15.60:0/915264137 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3ea0044010 con 0x7f3eb00b9c20 2022-06-29T16:05:11.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.654+0000 7f3eb5907700 1 -- 172.21.15.60:0/915264137 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f3eb0067ae0 con 0x7f3eb00b9c20 2022-06-29T16:05:11.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.658+0000 7f3eadffb700 1 -- 172.21.15.60:0/915264137 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v75) v1 ==== 81+0+28899 (secure 0 0 0) 0x7f3ea0023090 con 0x7f3eb00b9c20 2022-06-29T16:05:11.667 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:11.667 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:05:11.667 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 75, 2022-06-29T16:05:11.667 INFO:teuthology.orchestra.run.smithi060.stdout: "fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", 2022-06-29T16:05:11.668 INFO:teuthology.orchestra.run.smithi060.stdout: "created": "2022-06-29T15:58:07.965719+0000", 2022-06-29T16:05:11.668 INFO:teuthology.orchestra.run.smithi060.stdout: "modified": "2022-06-29T16:05:08.916616+0000", 2022-06-29T16:05:11.668 INFO:teuthology.orchestra.run.smithi060.stdout: "last_up_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:05:11.668 INFO:teuthology.orchestra.run.smithi060.stdout: "last_in_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:05:11.669 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-06-29T16:05:11.669 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_num": 5799936, 2022-06-29T16:05:11.669 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_set": [ 2022-06-29T16:05:11.669 INFO:teuthology.orchestra.run.smithi060.stdout: "pglog_hardlimit", 2022-06-29T16:05:11.670 INFO:teuthology.orchestra.run.smithi060.stdout: "purged_snapdirs", 2022-06-29T16:05:11.670 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_deletes", 2022-06-29T16:05:11.670 INFO:teuthology.orchestra.run.smithi060.stdout: "sortbitwise" 2022-06-29T16:05:11.670 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:05:11.670 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_version": 4, 2022-06-29T16:05:11.671 INFO:teuthology.orchestra.run.smithi060.stdout: "full_ratio": 0.94999998807907104, 2022-06-29T16:05:11.671 INFO:teuthology.orchestra.run.smithi060.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-06-29T16:05:11.671 INFO:teuthology.orchestra.run.smithi060.stdout: "nearfull_ratio": 0.85000002384185791, 2022-06-29T16:05:11.671 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_snapshot": "", 2022-06-29T16:05:11.672 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_max": 14, 2022-06-29T16:05:11.672 INFO:teuthology.orchestra.run.smithi060.stdout: "max_osd": 8, 2022-06-29T16:05:11.672 INFO:teuthology.orchestra.run.smithi060.stdout: "require_min_compat_client": "jewel", 2022-06-29T16:05:11.672 INFO:teuthology.orchestra.run.smithi060.stdout: "min_compat_client": "jewel", 2022-06-29T16:05:11.673 INFO:teuthology.orchestra.run.smithi060.stdout: "require_osd_release": "octopus", 2022-06-29T16:05:11.673 INFO:teuthology.orchestra.run.smithi060.stdout: "pools": [ 2022-06-29T16:05:11.673 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.673 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2022-06-29T16:05:11.673 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "device_health_metrics", 2022-06-29T16:05:11.674 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:10.961127+0000", 2022-06-29T16:05:11.674 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:05:11.674 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:05:11.674 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:05:11.675 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:05:11.675 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:05:11.675 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:05:11.675 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:05:11.676 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:05:11.676 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 1, 2022-06-29T16:05:11.676 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 1, 2022-06-29T16:05:11.676 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 1, 2022-06-29T16:05:11.676 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 1, 2022-06-29T16:05:11.677 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 1, 2022-06-29T16:05:11.677 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:05:11.677 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:05:11.677 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:05:11.678 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:05:11.678 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:05:11.678 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:05:11.678 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:05:11.679 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.679 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "15", 2022-06-29T16:05:11.679 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:05:11.679 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:05:11.679 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:05:11.680 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:05:11.680 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:05:11.680 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:05:11.680 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:05:11.681 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:05:11.681 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:05:11.681 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:05:11.681 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:05:11.682 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:05:11.682 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:05:11.682 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:05:11.682 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:05:11.683 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:05:11.683 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:05:11.683 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:05:11.683 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:05:11.683 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:05:11.684 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:05:11.684 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:05:11.684 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:05:11.684 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:05:11.685 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:05:11.685 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:05:11.685 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.685 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:05:11.685 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:05:11.686 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:05:11.686 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:05:11.686 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:05:11.686 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:05:11.687 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:05:11.687 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:05:11.687 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:05:11.687 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:05:11.688 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:05:11.688 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:05:11.688 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 1 2022-06-29T16:05:11.688 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.689 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:05:11.689 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_devicehealth": {} 2022-06-29T16:05:11.689 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.689 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.690 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 2, 2022-06-29T16:05:11.690 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "rbd", 2022-06-29T16:05:11.690 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:19.184276+0000", 2022-06-29T16:05:11.691 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:05:11.691 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:05:11.691 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:05:11.691 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:05:11.691 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:05:11.692 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:05:11.692 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:05:11.692 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:05:11.692 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:05:11.693 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:05:11.693 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:05:11.693 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:05:11.693 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:05:11.693 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:05:11.694 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:05:11.694 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:05:11.694 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:05:11.694 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:05:11.695 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:05:11.695 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:05:11.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.695 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "18", 2022-06-29T16:05:11.696 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:05:11.696 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:05:11.696 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:05:11.696 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:05:11.696 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:05:11.697 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:05:11.697 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:05:11.697 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:05:11.697 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:05:11.698 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:05:11.698 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:05:11.698 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:05:11.698 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:05:11.699 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:05:11.699 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:05:11.699 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:05:11.699 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:05:11.699 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:05:11.700 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:05:11.700 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:05:11.700 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:05:11.700 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:05:11.701 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:05:11.701 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:05:11.701 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:05:11.701 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:05:11.702 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.702 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:05:11.702 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:05:11.702 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:05:11.702 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:05:11.703 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:05:11.703 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:05:11.703 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:05:11.703 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:05:11.704 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:05:11.704 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:05:11.704 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:05:11.704 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:05:11.704 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:05:11.705 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd": {} 2022-06-29T16:05:11.705 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.705 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.705 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:05:11.706 INFO:teuthology.orchestra.run.smithi060.stdout: "osds": [ 2022-06-29T16:05:11.706 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.706 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:05:11.706 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "11e7ab9e-fce2-470f-8195-ea4c3c52df22", 2022-06-29T16:05:11.707 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:11.707 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:11.707 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:11.707 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:11.707 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:11.708 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:11.708 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:11.708 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 69, 2022-06-29T16:05:11.708 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:11.709 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:11.709 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:11.709 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:11.709 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.710 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:11.710 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6816", 2022-06-29T16:05:11.710 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:11.710 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.711 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.711 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:11.711 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6825", 2022-06-29T16:05:11.711 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:11.711 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.712 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.712 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.712 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:11.712 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:11.713 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.713 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:11.713 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6826", 2022-06-29T16:05:11.713 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:11.714 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.714 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.714 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:11.714 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6827", 2022-06-29T16:05:11.715 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:11.715 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.715 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.715 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.715 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:11.716 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:11.716 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.716 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:11.716 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6830", 2022-06-29T16:05:11.717 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:11.717 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.717 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.717 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:11.717 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6831", 2022-06-29T16:05:11.718 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:11.718 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.718 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.718 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.719 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:11.719 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:11.719 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.719 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:11.720 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6828", 2022-06-29T16:05:11.720 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:11.720 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.720 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.720 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:11.721 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6829", 2022-06-29T16:05:11.721 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:11.721 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.721 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.722 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.722 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6825/12890", 2022-06-29T16:05:11.722 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6827/12890", 2022-06-29T16:05:11.722 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6831/12890", 2022-06-29T16:05:11.722 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6829/12890", 2022-06-29T16:05:11.723 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:11.723 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:11.723 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:11.723 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.724 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.724 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.724 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:05:11.724 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "ca43f136-6f69-4431-a157-fbe073fb585c", 2022-06-29T16:05:11.724 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:11.725 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:11.725 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:11.725 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:11.725 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:11.726 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:11.726 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:11.726 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 59, 2022-06-29T16:05:11.726 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:11.727 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:11.727 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:11.727 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:11.727 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.727 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:11.728 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6815", 2022-06-29T16:05:11.728 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:11.728 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.728 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.729 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:11.729 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6818", 2022-06-29T16:05:11.729 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:11.729 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.729 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.730 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.730 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:11.730 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:11.730 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.731 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:11.731 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6819", 2022-06-29T16:05:11.731 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:11.731 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.731 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.732 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:11.732 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6820", 2022-06-29T16:05:11.732 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:11.732 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.733 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.733 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.733 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:11.733 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:11.734 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.734 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:11.734 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6823", 2022-06-29T16:05:11.734 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:11.734 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.735 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.735 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:11.735 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6824", 2022-06-29T16:05:11.735 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:11.736 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.736 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.736 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.736 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:11.736 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:11.737 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.737 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:11.737 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6821", 2022-06-29T16:05:11.737 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:11.738 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.738 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.738 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:11.738 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6822", 2022-06-29T16:05:11.739 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:11.739 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.739 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.739 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.739 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6818/12892", 2022-06-29T16:05:11.740 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6820/12892", 2022-06-29T16:05:11.740 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6824/12892", 2022-06-29T16:05:11.740 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6822/12892", 2022-06-29T16:05:11.740 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:11.741 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:11.741 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:11.741 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.741 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.741 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.742 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:05:11.742 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "74fac45f-5f07-44ec-a9cb-2644c123fd4a", 2022-06-29T16:05:11.742 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:11.742 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:11.743 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:11.743 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:11.743 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:11.743 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:11.744 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:11.744 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 69, 2022-06-29T16:05:11.744 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:11.744 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:11.744 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:11.745 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:11.745 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.745 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:11.745 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6808", 2022-06-29T16:05:11.746 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:11.746 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.746 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.746 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:11.746 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6809", 2022-06-29T16:05:11.747 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:11.747 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.747 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.747 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.748 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:11.748 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:11.748 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.748 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:11.748 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6810", 2022-06-29T16:05:11.749 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:11.749 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.749 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.749 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:11.750 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6811", 2022-06-29T16:05:11.750 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:11.750 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.750 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.751 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.751 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:11.751 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:11.751 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.752 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:11.752 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6814", 2022-06-29T16:05:11.752 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:11.752 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.753 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.753 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:11.753 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6817", 2022-06-29T16:05:11.753 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:11.753 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.754 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.754 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.754 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:11.754 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:11.755 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.755 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:11.755 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6812", 2022-06-29T16:05:11.755 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:11.756 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.756 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.756 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:11.756 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6813", 2022-06-29T16:05:11.756 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:11.757 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.757 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.757 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.757 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6809/12891", 2022-06-29T16:05:11.758 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6811/12891", 2022-06-29T16:05:11.758 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6817/12891", 2022-06-29T16:05:11.758 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6813/12891", 2022-06-29T16:05:11.758 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:11.759 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:11.759 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:11.759 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.759 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.760 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.760 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:05:11.760 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "2581d351-973a-4b4e-85ca-3658f0068765", 2022-06-29T16:05:11.760 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:11.761 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:11.761 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:11.761 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:11.762 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:11.762 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:11.762 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:11.762 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 69, 2022-06-29T16:05:11.763 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:11.763 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:11.763 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:11.763 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:11.763 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.764 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:11.764 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6800", 2022-06-29T16:05:11.764 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:11.764 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.765 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.765 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:11.765 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6801", 2022-06-29T16:05:11.765 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:11.765 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.766 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.766 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.766 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:11.766 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:11.767 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.767 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:11.767 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6802", 2022-06-29T16:05:11.767 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:11.767 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.768 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.768 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:11.768 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6803", 2022-06-29T16:05:11.768 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:11.769 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.769 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.769 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.769 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:11.769 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:11.770 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.770 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:11.770 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6806", 2022-06-29T16:05:11.770 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:11.771 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.771 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.771 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:11.771 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6807", 2022-06-29T16:05:11.771 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:11.772 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.772 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.772 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.772 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:11.773 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:11.773 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.773 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:11.773 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6804", 2022-06-29T16:05:11.773 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:11.774 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.774 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.774 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:11.774 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6805", 2022-06-29T16:05:11.775 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:11.775 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.775 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.775 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.776 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6801/12889", 2022-06-29T16:05:11.776 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6803/12889", 2022-06-29T16:05:11.776 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6807/12889", 2022-06-29T16:05:11.776 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6805/12889", 2022-06-29T16:05:11.776 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:11.777 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:11.777 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:11.777 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.777 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.778 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.778 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:05:11.778 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "3d234db9-87cd-4fe9-814f-0240a287d8b4", 2022-06-29T16:05:11.778 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:11.778 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:11.779 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:11.779 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:11.779 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:11.779 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:11.780 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:11.780 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 68, 2022-06-29T16:05:11.780 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:11.780 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:11.780 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:11.781 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:11.781 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.781 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:11.781 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6808", 2022-06-29T16:05:11.782 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:11.782 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.782 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.782 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:11.782 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6810", 2022-06-29T16:05:11.783 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:11.783 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.783 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.783 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.784 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:11.784 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:11.784 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.784 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:11.785 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6812", 2022-06-29T16:05:11.785 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:11.785 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.785 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.785 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:11.786 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6814", 2022-06-29T16:05:11.786 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:11.786 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.786 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.787 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.787 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:11.787 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:11.787 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.787 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:11.788 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6823", 2022-06-29T16:05:11.788 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:11.788 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.788 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.789 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:11.789 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6825", 2022-06-29T16:05:11.789 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:11.789 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.790 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.790 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.790 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:11.790 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:11.790 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.791 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:11.791 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6817", 2022-06-29T16:05:11.791 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:11.792 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.792 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.792 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:11.792 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6820", 2022-06-29T16:05:11.792 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:11.793 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.793 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.793 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.793 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6810/13249", 2022-06-29T16:05:11.794 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6814/13249", 2022-06-29T16:05:11.794 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6825/13249", 2022-06-29T16:05:11.794 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6820/13249", 2022-06-29T16:05:11.794 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:11.795 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:11.795 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:11.795 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.795 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.795 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.796 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:05:11.796 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "e4bf1104-ff5c-438b-b608-8cee31c163f6", 2022-06-29T16:05:11.796 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:11.796 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:11.797 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:11.797 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:11.797 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:11.797 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:11.797 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:11.798 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 69, 2022-06-29T16:05:11.798 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:11.798 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:11.798 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:11.799 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:11.799 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.799 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:11.799 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6809", 2022-06-29T16:05:11.799 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:11.800 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.800 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.800 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:11.800 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6811", 2022-06-29T16:05:11.801 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:11.801 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.801 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.801 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.801 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:11.802 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:11.802 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.802 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:11.802 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6813", 2022-06-29T16:05:11.803 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:11.803 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.803 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.803 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:11.804 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6815", 2022-06-29T16:05:11.804 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:11.804 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.804 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.804 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.805 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:11.805 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:11.805 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.805 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:11.806 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6824", 2022-06-29T16:05:11.806 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:11.806 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.806 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.807 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:11.807 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6827", 2022-06-29T16:05:11.807 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:11.807 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.807 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.808 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.808 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:11.808 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:11.809 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.809 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:11.809 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6818", 2022-06-29T16:05:11.809 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:11.810 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.810 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.810 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:11.810 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6821", 2022-06-29T16:05:11.811 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:11.811 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.811 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.811 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.811 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6811/13248", 2022-06-29T16:05:11.812 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6815/13248", 2022-06-29T16:05:11.812 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6827/13248", 2022-06-29T16:05:11.812 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6821/13248", 2022-06-29T16:05:11.812 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:11.813 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:11.813 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:11.813 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.813 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.814 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.814 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:05:11.814 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "8564dd25-27ab-4203-b54e-af6d3aa05857", 2022-06-29T16:05:11.814 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:11.815 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:11.815 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:11.815 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:11.815 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:11.816 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:11.816 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:11.816 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 69, 2022-06-29T16:05:11.816 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:11.817 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:11.817 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:11.817 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:11.817 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.818 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:11.818 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6816", 2022-06-29T16:05:11.818 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:11.818 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.819 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.819 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:11.819 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6819", 2022-06-29T16:05:11.819 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:11.820 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.820 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.820 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.820 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:11.820 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:11.821 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.821 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:11.821 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6822", 2022-06-29T16:05:11.821 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:11.822 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.822 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.822 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:11.822 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6826", 2022-06-29T16:05:11.823 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:11.823 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.823 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.823 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.823 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:11.824 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:11.824 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.824 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:11.824 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6830", 2022-06-29T16:05:11.825 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:11.825 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.825 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.825 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:11.825 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6831", 2022-06-29T16:05:11.826 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:11.826 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.826 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.826 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.827 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:11.827 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:11.827 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.827 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:11.827 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6828", 2022-06-29T16:05:11.828 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:11.828 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.828 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.828 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:11.829 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6829", 2022-06-29T16:05:11.829 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:11.829 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.829 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.830 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.830 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6819/13250", 2022-06-29T16:05:11.830 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6826/13250", 2022-06-29T16:05:11.830 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6831/13250", 2022-06-29T16:05:11.830 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6829/13250", 2022-06-29T16:05:11.831 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:11.831 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:11.831 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:11.831 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.832 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.832 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.832 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:05:11.832 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "bd513a9c-3b19-4314-9897-d300b5c897bb", 2022-06-29T16:05:11.833 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:11.833 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:11.833 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:11.833 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:11.833 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:11.834 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:11.834 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:11.834 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 68, 2022-06-29T16:05:11.834 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:11.835 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:11.835 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:11.835 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:11.835 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.835 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:11.836 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6800", 2022-06-29T16:05:11.836 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:11.836 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.836 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.837 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:11.837 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6801", 2022-06-29T16:05:11.837 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:11.837 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.837 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.838 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.838 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:11.838 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:11.838 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.839 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:11.839 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6802", 2022-06-29T16:05:11.839 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:11.839 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.840 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.840 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:11.840 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6803", 2022-06-29T16:05:11.840 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:11.840 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.841 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.841 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.841 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:11.841 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:11.842 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.842 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:11.842 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6806", 2022-06-29T16:05:11.842 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:11.843 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.843 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.843 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:11.843 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6807", 2022-06-29T16:05:11.844 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:11.844 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.844 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.844 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.844 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:11.845 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:11.845 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.845 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:11.845 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6804", 2022-06-29T16:05:11.846 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:11.846 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.846 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.846 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:11.847 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6805", 2022-06-29T16:05:11.847 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:11.847 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.847 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.848 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.848 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6801/13247", 2022-06-29T16:05:11.848 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6803/13247", 2022-06-29T16:05:11.848 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6807/13247", 2022-06-29T16:05:11.849 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6805/13247", 2022-06-29T16:05:11.849 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:11.849 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:11.849 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:11.849 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:11.850 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.850 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:05:11.850 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_xinfo": [ 2022-06-29T16:05:11.850 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.851 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:05:11.851 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:11.851 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:11.851 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:11.852 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:11.852 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:11.852 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.891342+0000", 2022-06-29T16:05:11.852 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:11.852 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.853 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.853 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:05:11.853 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:11.853 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:11.854 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:11.854 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:11.854 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:11.854 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.900866+0000", 2022-06-29T16:05:11.855 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:11.855 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.855 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.855 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:05:11.855 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:11.856 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:11.856 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:11.856 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:11.856 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:11.856 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.912755+0000", 2022-06-29T16:05:11.857 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:11.857 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.857 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.857 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:05:11.858 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:11.858 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:11.858 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:11.858 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:11.859 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:11.859 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.864332+0000", 2022-06-29T16:05:11.859 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:11.859 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.859 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.860 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:05:11.860 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:11.860 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:11.860 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:11.861 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:11.861 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:11.861 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.863756+0000", 2022-06-29T16:05:11.861 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:11.861 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.862 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.862 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:05:11.862 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:11.862 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:11.863 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:11.863 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:11.863 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:11.863 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.869041+0000", 2022-06-29T16:05:11.864 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:11.864 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.864 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.864 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:05:11.864 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:11.865 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:11.865 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:11.865 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:11.865 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:11.866 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.895043+0000", 2022-06-29T16:05:11.866 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:11.866 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.866 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:11.866 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:05:11.867 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:11.867 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:11.867 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:11.867 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:11.868 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:11.868 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.889491+0000", 2022-06-29T16:05:11.868 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:11.868 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.869 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:05:11.869 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap": [], 2022-06-29T16:05:11.869 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap_items": [], 2022-06-29T16:05:11.869 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_temp": [], 2022-06-29T16:05:11.870 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_temp": [], 2022-06-29T16:05:11.870 INFO:teuthology.orchestra.run.smithi060.stdout: "blacklist": { 2022-06-29T16:05:11.870 INFO:teuthology.orchestra.run.smithi060.stdout: "172.21.15.92:6835/953938586": "2022-06-30T16:05:05.686603+0000", 2022-06-29T16:05:11.870 INFO:teuthology.orchestra.run.smithi060.stdout: "172.21.15.92:6834/953938586": "2022-06-30T16:05:05.686603+0000", 2022-06-29T16:05:11.871 INFO:teuthology.orchestra.run.smithi060.stdout: "172.21.15.121:0/2990201791": "2022-06-29T17:04:53.908824+0000" 2022-06-29T16:05:11.871 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.871 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profiles": { 2022-06-29T16:05:11.871 INFO:teuthology.orchestra.run.smithi060.stdout: "default": { 2022-06-29T16:05:11.872 INFO:teuthology.orchestra.run.smithi060.stdout: "crush-failure-domain": "osd", 2022-06-29T16:05:11.872 INFO:teuthology.orchestra.run.smithi060.stdout: "k": "2", 2022-06-29T16:05:11.872 INFO:teuthology.orchestra.run.smithi060.stdout: "m": "1", 2022-06-29T16:05:11.872 INFO:teuthology.orchestra.run.smithi060.stdout: "plugin": "jerasure", 2022-06-29T16:05:11.872 INFO:teuthology.orchestra.run.smithi060.stdout: "ruleset-failure-domain": "osd", 2022-06-29T16:05:11.873 INFO:teuthology.orchestra.run.smithi060.stdout: "technique": "reed_sol_van" 2022-06-29T16:05:11.873 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:11.873 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:11.873 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps_queue": [], 2022-06-29T16:05:11.874 INFO:teuthology.orchestra.run.smithi060.stdout: "new_removed_snaps": [], 2022-06-29T16:05:11.874 INFO:teuthology.orchestra.run.smithi060.stdout: "new_purged_snaps": [], 2022-06-29T16:05:11.874 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_node_flags": {}, 2022-06-29T16:05:11.874 INFO:teuthology.orchestra.run.smithi060.stdout: "device_class_flags": {} 2022-06-29T16:05:11.875 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:05:11.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.662+0000 7f3eb5907700 1 -- 172.21.15.60:0/915264137 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3e94053a70 msgr2=0x7f3e94055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:11.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.662+0000 7f3eb5907700 1 --2- 172.21.15.60:0/915264137 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3e94053a70 0x7f3e94055f30 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f3e9800c000 tx=0x7f3e98008040).stop 2022-06-29T16:05:11.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.662+0000 7f3eb5907700 1 -- 172.21.15.60:0/915264137 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3eb00b9c20 msgr2=0x7f3eb00b73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:11.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.662+0000 7f3eb5907700 1 --2- 172.21.15.60:0/915264137 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3eb00b9c20 0x7f3eb00b73b0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 rx=0x7f3eb00ad420 tx=0x7f3ea0015000).stop 2022-06-29T16:05:11.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.662+0000 7f3eb5907700 1 -- 172.21.15.60:0/915264137 shutdown_connections 2022-06-29T16:05:11.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.662+0000 7f3eb5907700 1 --2- 172.21.15.60:0/915264137 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3e94053a70 0x7f3e94055f30 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:11.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.662+0000 7f3eb5907700 1 --2- 172.21.15.60:0/915264137 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3eb0146480 0x7f3eb0152290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:11.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.662+0000 7f3eb5907700 1 --2- 172.21.15.60:0/915264137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3eb0146040 0x7f3eb0143f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:11.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.662+0000 7f3eb5907700 1 --2- 172.21.15.60:0/915264137 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3eb00b9c20 0x7f3eb00b73b0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:11.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.662+0000 7f3eb5907700 1 -- 172.21.15.60:0/915264137 >> 172.21.15.60:0/915264137 conn(0x7f3eb0062cb0 msgr2=0x7f3eb005c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:11.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.662+0000 7f3eb5907700 1 -- 172.21.15.60:0/915264137 shutdown_connections 2022-06-29T16:05:11.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.662+0000 7f3eb5907700 1 -- 172.21.15.60:0/915264137 wait complete. 2022-06-29T16:05:11.880 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:05:11.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.982+0000 7feec3fff700 1 Processor -- start 2022-06-29T16:05:11.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.982+0000 7feec3fff700 1 -- start start 2022-06-29T16:05:11.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.986+0000 7feec3fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7feec4060760 0x7feec40b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:11.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.986+0000 7feec3fff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feec413fa40 con 0x7feec4065450 2022-06-29T16:05:11.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.986+0000 7feec3fff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feec413fce0 con 0x7feec40b28e0 2022-06-29T16:05:11.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.986+0000 7feec3fff700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feec4140860 con 0x7feec4060760 2022-06-29T16:05:11.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.986+0000 7feeba7fc700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7feec4065450 0x7feec4066390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57938/0 (socket says 172.21.15.60:57938) 2022-06-29T16:05:11.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.986+0000 7feeba7fc700 1 -- 172.21.15.60:0/1140535905 learned_addr learned my addr 172.21.15.60:0/1140535905 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:11.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.986+0000 7feec37fe700 1 --2- 172.21.15.60:0/1140535905 >> v2:172.21.15.92:3300/0 conn(0x7feec4060760 0x7feec40b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:11.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.986+0000 7feec27fc700 1 -- 172.21.15.60:0/1140535905 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3053411726 0 0) 0x7feec413fa40 con 0x7feec4065450 2022-06-29T16:05:11.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.986+0000 7feec27fc700 1 -- 172.21.15.60:0/1140535905 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feea4003260 con 0x7feec4065450 2022-06-29T16:05:11.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.986+0000 7feec37fe700 1 -- 172.21.15.60:0/1140535905 >> v1:172.21.15.92:6789/0 conn(0x7feec40b28e0 legacy=0x7feec4139340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:11.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.986+0000 7feec37fe700 1 -- 172.21.15.60:0/1140535905 >> v1:172.21.15.60:6789/0 conn(0x7feec4065450 legacy=0x7feec4066390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:11.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.986+0000 7feec37fe700 1 -- 172.21.15.60:0/1140535905 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec413f6f0 con 0x7feec4060760 2022-06-29T16:05:11.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.986+0000 7feec37fe700 1 --2- 172.21.15.60:0/1140535905 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feec4060760 0x7feec40b2cc0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7feebc0032f0 tx=0x7feebc0091d0).ready entity=mon.1 client_cookie=70222b1a65f0c06d server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:11.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.986+0000 7feec27fc700 1 -- 172.21.15.60:0/1140535905 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feebc00b6c0 con 0x7feec4060760 2022-06-29T16:05:11.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.986+0000 7feec27fc700 1 -- 172.21.15.60:0/1140535905 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feebc00b820 con 0x7feec4060760 2022-06-29T16:05:11.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.986+0000 7feec27fc700 1 -- 172.21.15.60:0/1140535905 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feebc00b990 con 0x7feec4060760 2022-06-29T16:05:11.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.986+0000 7feec3fff700 1 -- 172.21.15.60:0/1140535905 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feec4060760 msgr2=0x7feec40b2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:11.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.986+0000 7feec3fff700 1 --2- 172.21.15.60:0/1140535905 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feec4060760 0x7feec40b2cc0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7feebc0032f0 tx=0x7feebc0091d0).stop 2022-06-29T16:05:11.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.986+0000 7feec3fff700 1 -- 172.21.15.60:0/1140535905 shutdown_connections 2022-06-29T16:05:11.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.986+0000 7feec3fff700 1 --2- 172.21.15.60:0/1140535905 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feec4060760 0x7feec40b2cc0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:11.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.986+0000 7feec3fff700 1 -- 172.21.15.60:0/1140535905 >> 172.21.15.60:0/1140535905 conn(0x7feec4061c80 msgr2=0x7feec40637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:12.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.986+0000 7feec3fff700 1 -- 172.21.15.60:0/1140535905 shutdown_connections 2022-06-29T16:05:12.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.986+0000 7feec3fff700 1 -- 172.21.15.60:0/1140535905 wait complete. 2022-06-29T16:05:12.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.990+0000 7feec3fff700 1 Processor -- start 2022-06-29T16:05:12.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.990+0000 7feec3fff700 1 -- start start 2022-06-29T16:05:12.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.990+0000 7feec3fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7feec4060760 0x7feec40af880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:12.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.990+0000 7feec3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feec4065450 0x7feec413b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:12.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.990+0000 7feec3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feec40b28e0 0x7feec413cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:12.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.990+0000 7feec3fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec413ac10 con 0x7feec4065450 2022-06-29T16:05:12.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.990+0000 7feec3fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec405d030 con 0x7feec40b28e0 2022-06-29T16:05:12.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.990+0000 7feec3fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feec405cd90 con 0x7feec4060760 2022-06-29T16:05:12.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.990+0000 7feeba7fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feec4065450 0x7feec413b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:12.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.990+0000 7feec37fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feec40b28e0 0x7feec413cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:12.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.990+0000 7feeba7fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feec4065450 0x7feec413b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49860/0 (socket says 172.21.15.60:49860) 2022-06-29T16:05:12.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.990+0000 7feeba7fc700 1 -- 172.21.15.60:0/3041858160 learned_addr learned my addr 172.21.15.60:0/3041858160 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:12.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.990+0000 7feec2ffd700 1 --2- 172.21.15.60:0/3041858160 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7feec4060760 0x7feec40af880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:12.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.994+0000 7feeba7fc700 1 -- 172.21.15.60:0/3041858160 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7feec4060760 msgr2=0x7feec40af880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:12.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.994+0000 7feeba7fc700 1 --2- 172.21.15.60:0/3041858160 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7feec4060760 0x7feec40af880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:12.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.994+0000 7feeba7fc700 1 -- 172.21.15.60:0/3041858160 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feec40b28e0 msgr2=0x7feec413cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:12.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.994+0000 7feeba7fc700 1 --2- 172.21.15.60:0/3041858160 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feec40b28e0 0x7feec413cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:12.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.994+0000 7feeba7fc700 1 -- 172.21.15.60:0/3041858160 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feebc009470 con 0x7feec4065450 2022-06-29T16:05:12.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.994+0000 7feeba7fc700 1 --2- 172.21.15.60:0/3041858160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feec4065450 0x7feec413b070 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 rx=0x7feeac001110 tx=0x7feeac007ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:12.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.994+0000 7feec0ff9700 1 -- 172.21.15.60:0/3041858160 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feeac0070d0 con 0x7feec4065450 2022-06-29T16:05:12.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.994+0000 7feec0ff9700 1 -- 172.21.15.60:0/3041858160 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeac007230 con 0x7feec4065450 2022-06-29T16:05:12.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.994+0000 7feec3fff700 1 -- 172.21.15.60:0/3041858160 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feec4145510 con 0x7feec4065450 2022-06-29T16:05:12.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.994+0000 7feec0ff9700 1 -- 172.21.15.60:0/3041858160 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feeac005610 con 0x7feec4065450 2022-06-29T16:05:12.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.994+0000 7feec3fff700 1 -- 172.21.15.60:0/3041858160 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feec413b560 con 0x7feec4065450 2022-06-29T16:05:12.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.994+0000 7feec3fff700 1 -- 172.21.15.60:0/3041858160 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feec413aa20 con 0x7feec4065450 2022-06-29T16:05:12.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.994+0000 7feec0ff9700 1 -- 172.21.15.60:0/3041858160 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7feeac010070 con 0x7feec4065450 2022-06-29T16:05:12.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.994+0000 7feec0ff9700 1 --2- 172.21.15.60:0/3041858160 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7feea8053b00 0x7feea8055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:12.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.994+0000 7feec0ff9700 1 -- 172.21.15.60:0/3041858160 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5382+0+0 (secure 0 0 0) 0x7feeac06b250 con 0x7feec4065450 2022-06-29T16:05:12.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.994+0000 7feec2ffd700 1 --2- 172.21.15.60:0/3041858160 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7feea8053b00 0x7feea8055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:12.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.998+0000 7feec2ffd700 1 --2- 172.21.15.60:0/3041858160 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7feea8053b00 0x7feea8055fc0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7feeb400d000 tx=0x7feeb4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:12.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:11.998+0000 7feec0ff9700 1 -- 172.21.15.60:0/3041858160 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7feeac042420 con 0x7feec4065450 2022-06-29T16:05:12.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.162+0000 7feec3fff700 1 -- 172.21.15.60:0/3041858160 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feec40573d0 con 0x7feec4065450 2022-06-29T16:05:12.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.162+0000 7feec0ff9700 1 -- 172.21.15.60:0/3041858160 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+1209 (secure 0 0 0) 0x7feeac045a40 con 0x7feec4065450 2022-06-29T16:05:12.172 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:12.172 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":32,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5690,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1158969084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1158969084},{"type":"v1","addr":"172.21.15.60:6833","nonce":1158969084}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":28},{"gid":5894,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/2464413229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2464413229},{"type":"v1","addr":"172.21.15.92:6835","nonce":2464413229}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":32}],"filesystems":[]} 2022-06-29T16:05:12.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.166+0000 7feec3fff700 1 -- 172.21.15.60:0/3041858160 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7feea8053b00 msgr2=0x7feea8055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:12.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.166+0000 7feec3fff700 1 --2- 172.21.15.60:0/3041858160 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7feea8053b00 0x7feea8055fc0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7feeb400d000 tx=0x7feeb4009040).stop 2022-06-29T16:05:12.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.166+0000 7feec3fff700 1 -- 172.21.15.60:0/3041858160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feec4065450 msgr2=0x7feec413b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:12.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.166+0000 7feec3fff700 1 --2- 172.21.15.60:0/3041858160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feec4065450 0x7feec413b070 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 rx=0x7feeac001110 tx=0x7feeac007ad0).stop 2022-06-29T16:05:12.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.166+0000 7feec3fff700 1 -- 172.21.15.60:0/3041858160 shutdown_connections 2022-06-29T16:05:12.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.166+0000 7feec3fff700 1 --2- 172.21.15.60:0/3041858160 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7feea8053b00 0x7feea8055fc0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:12.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.166+0000 7feec3fff700 1 --2- 172.21.15.60:0/3041858160 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feec40b28e0 0x7feec413cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:12.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.166+0000 7feec3fff700 1 --2- 172.21.15.60:0/3041858160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feec4065450 0x7feec413b070 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:12.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.166+0000 7feec3fff700 1 --2- 172.21.15.60:0/3041858160 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7feec4060760 0x7feec40af880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:12.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.166+0000 7feec3fff700 1 -- 172.21.15.60:0/3041858160 >> 172.21.15.60:0/3041858160 conn(0x7feec4061c80 msgr2=0x7feec40b5580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:12.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.166+0000 7feec3fff700 1 -- 172.21.15.60:0/3041858160 shutdown_connections 2022-06-29T16:05:12.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.166+0000 7feec3fff700 1 -- 172.21.15.60:0/3041858160 wait complete. 2022-06-29T16:05:12.178 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 32 2022-06-29T16:05:12.189 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:05:12.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.278+0000 7fc44ffff700 1 Processor -- start 2022-06-29T16:05:12.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.278+0000 7fc44ffff700 1 -- start start 2022-06-29T16:05:12.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.278+0000 7fc44ffff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc45005d070 0x7fc45014aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:12.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.278+0000 7fc44ffff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4500b2130 0x7fc4500aca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:12.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.278+0000 7fc44ffff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc45005d450 con 0x7fc45014af80 2022-06-29T16:05:12.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.278+0000 7fc44ffff700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4500adf90 con 0x7fc45005d070 2022-06-29T16:05:12.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.278+0000 7fc44ffff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4500b25d0 con 0x7fc4500b2130 2022-06-29T16:05:12.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.278+0000 7fc44f7fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc45014af80 0x7fc4500acf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57954/0 (socket says 172.21.15.60:57954) 2022-06-29T16:05:12.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.278+0000 7fc44f7fe700 1 -- 172.21.15.60:0/2432912356 learned_addr learned my addr 172.21.15.60:0/2432912356 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:12.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.278+0000 7fc44effd700 1 --2- 172.21.15.60:0/2432912356 >> v2:172.21.15.92:3300/0 conn(0x7fc45005d070 0x7fc45014aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:12.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.278+0000 7fc44e7fc700 1 --2- 172.21.15.60:0/2432912356 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4500b2130 0x7fc4500aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:12.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.278+0000 7fc44dffb700 1 -- 172.21.15.60:0/2432912356 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 838108349 0 0) 0x7fc45005d450 con 0x7fc45014af80 2022-06-29T16:05:12.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.278+0000 7fc44dffb700 1 -- 172.21.15.60:0/2432912356 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc434003260 con 0x7fc45014af80 2022-06-29T16:05:12.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.278+0000 7fc44dffb700 1 -- 172.21.15.60:0/2432912356 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2516699241 0 0) 0x7fc434003260 con 0x7fc45014af80 2022-06-29T16:05:12.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.278+0000 7fc44effd700 1 -- 172.21.15.60:0/2432912356 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4500b2130 msgr2=0x7fc4500aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:12.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.282+0000 7fc44effd700 1 --2- 172.21.15.60:0/2432912356 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4500b2130 0x7fc4500aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:12.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.282+0000 7fc44effd700 1 -- 172.21.15.60:0/2432912356 >> v1:172.21.15.60:6789/0 conn(0x7fc45014af80 legacy=0x7fc4500acf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:12.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.282+0000 7fc44effd700 1 -- 172.21.15.60:0/2432912356 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc45014a350 con 0x7fc45005d070 2022-06-29T16:05:12.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.282+0000 7fc44effd700 1 --2- 172.21.15.60:0/2432912356 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc45005d070 0x7fc45014aa90 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fc440001820 tx=0x7fc4480091d0).ready entity=mon.1 client_cookie=2d5cc0cb13784d82 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:12.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.282+0000 7fc44dffb700 1 -- 172.21.15.60:0/2432912356 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc440001680 con 0x7fc45005d070 2022-06-29T16:05:12.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.282+0000 7fc44dffb700 1 -- 172.21.15.60:0/2432912356 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc44800f510 con 0x7fc45005d070 2022-06-29T16:05:12.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.282+0000 7fc44dffb700 1 -- 172.21.15.60:0/2432912356 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc44800f680 con 0x7fc45005d070 2022-06-29T16:05:12.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.282+0000 7fc44ffff700 1 -- 172.21.15.60:0/2432912356 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc45005d070 msgr2=0x7fc45014aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:12.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.282+0000 7fc44ffff700 1 --2- 172.21.15.60:0/2432912356 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc45005d070 0x7fc45014aa90 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fc440001820 tx=0x7fc4480091d0).stop 2022-06-29T16:05:12.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.282+0000 7fc44ffff700 1 -- 172.21.15.60:0/2432912356 shutdown_connections 2022-06-29T16:05:12.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.282+0000 7fc44ffff700 1 --2- 172.21.15.60:0/2432912356 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc4500b2130 0x7fc4500aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:12.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.282+0000 7fc44ffff700 1 --2- 172.21.15.60:0/2432912356 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc45005d070 0x7fc45014aa90 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:12.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.282+0000 7fc44ffff700 1 -- 172.21.15.60:0/2432912356 >> 172.21.15.60:0/2432912356 conn(0x7fc4500b5890 msgr2=0x7fc4500b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:12.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.282+0000 7fc44ffff700 1 -- 172.21.15.60:0/2432912356 shutdown_connections 2022-06-29T16:05:12.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.282+0000 7fc44ffff700 1 -- 172.21.15.60:0/2432912356 wait complete. 2022-06-29T16:05:12.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.282+0000 7fc44ffff700 1 Processor -- start 2022-06-29T16:05:12.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.282+0000 7fc44ffff700 1 -- start start 2022-06-29T16:05:12.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.286+0000 7fc44ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc45005d070 0x7fc4500b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:12.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.286+0000 7fc44ffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4500b2130 0x7fc450066590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:12.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.286+0000 7fc44ffff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc45014af80 0x7fc450066a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:12.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.286+0000 7fc44ffff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc450066f70 con 0x7fc45005d070 2022-06-29T16:05:12.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.286+0000 7fc44ffff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc450063850 con 0x7fc4500b2130 2022-06-29T16:05:12.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.286+0000 7fc44ffff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc45015a7b0 con 0x7fc45014af80 2022-06-29T16:05:12.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.286+0000 7fc44effd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc45005d070 0x7fc4500b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:12.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.286+0000 7fc44effd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc45005d070 0x7fc4500b41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49870/0 (socket says 172.21.15.60:49870) 2022-06-29T16:05:12.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.286+0000 7fc44effd700 1 -- 172.21.15.60:0/2777936636 learned_addr learned my addr 172.21.15.60:0/2777936636 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:12.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.286+0000 7fc44e7fc700 1 --2- 172.21.15.60:0/2777936636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4500b2130 0x7fc450066590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:12.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.286+0000 7fc44f7fe700 1 --2- 172.21.15.60:0/2777936636 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc45014af80 0x7fc450066a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:12.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.286+0000 7fc44effd700 1 -- 172.21.15.60:0/2777936636 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc45014af80 msgr2=0x7fc450066a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:12.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.286+0000 7fc44effd700 1 --2- 172.21.15.60:0/2777936636 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc45014af80 0x7fc450066a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:12.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.286+0000 7fc44effd700 1 -- 172.21.15.60:0/2777936636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4500b2130 msgr2=0x7fc450066590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:12.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.286+0000 7fc44effd700 1 --2- 172.21.15.60:0/2777936636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4500b2130 0x7fc450066590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:12.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.286+0000 7fc44effd700 1 -- 172.21.15.60:0/2777936636 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc448009470 con 0x7fc45005d070 2022-06-29T16:05:12.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.286+0000 7fc44effd700 1 --2- 172.21.15.60:0/2777936636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc45005d070 0x7fc4500b41c0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 rx=0x7fc440001820 tx=0x7fc448016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:12.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.286+0000 7fc43ffff700 1 -- 172.21.15.60:0/2777936636 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc44800f680 con 0x7fc45005d070 2022-06-29T16:05:12.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.286+0000 7fc43ffff700 1 -- 172.21.15.60:0/2777936636 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4480032f0 con 0x7fc45005d070 2022-06-29T16:05:12.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.286+0000 7fc44ffff700 1 -- 172.21.15.60:0/2777936636 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4500635b0 con 0x7fc45005d070 2022-06-29T16:05:12.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.286+0000 7fc43ffff700 1 -- 172.21.15.60:0/2777936636 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc448021670 con 0x7fc45005d070 2022-06-29T16:05:12.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.286+0000 7fc44ffff700 1 -- 172.21.15.60:0/2777936636 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc450149ec0 con 0x7fc45005d070 2022-06-29T16:05:12.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.290+0000 7fc43ffff700 1 -- 172.21.15.60:0/2777936636 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc44800f070 con 0x7fc45005d070 2022-06-29T16:05:12.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.290+0000 7fc44ffff700 1 -- 172.21.15.60:0/2777936636 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc45014a5f0 con 0x7fc45005d070 2022-06-29T16:05:12.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.290+0000 7fc43ffff700 1 --2- 172.21.15.60:0/2777936636 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc42c053a70 0x7fc42c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:12.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.290+0000 7fc43ffff700 1 -- 172.21.15.60:0/2777936636 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fc44806d100 con 0x7fc45005d070 2022-06-29T16:05:12.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.290+0000 7fc44e7fc700 1 --2- 172.21.15.60:0/2777936636 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc42c053a70 0x7fc42c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:12.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.290+0000 7fc44e7fc700 1 --2- 172.21.15.60:0/2777936636 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc42c053a70 0x7fc42c055f30 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7fc44400b000 tx=0x7fc444006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:12.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.298+0000 7fc43ffff700 1 -- 172.21.15.60:0/2777936636 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc4480441c0 con 0x7fc45005d070 2022-06-29T16:05:12.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.462+0000 7fc44ffff700 1 -- 172.21.15.60:0/2777936636 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4500573d0 con 0x7fc45005d070 2022-06-29T16:05:12.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.462+0000 7fc43ffff700 1 -- 172.21.15.60:0/2777936636 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+1209 (secure 0 0 0) 0x7fc448015070 con 0x7fc45005d070 2022-06-29T16:05:12.469 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:12.470 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":32,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5690,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1158969084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1158969084},{"type":"v1","addr":"172.21.15.60:6833","nonce":1158969084}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":28},{"gid":5894,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/2464413229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2464413229},{"type":"v1","addr":"172.21.15.92:6835","nonce":2464413229}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":32}],"filesystems":[]} 2022-06-29T16:05:12.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.462+0000 7fc44ffff700 1 -- 172.21.15.60:0/2777936636 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc42c053a70 msgr2=0x7fc42c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:12.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.462+0000 7fc44ffff700 1 --2- 172.21.15.60:0/2777936636 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc42c053a70 0x7fc42c055f30 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7fc44400b000 tx=0x7fc444006df0).stop 2022-06-29T16:05:12.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.462+0000 7fc44ffff700 1 -- 172.21.15.60:0/2777936636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc45005d070 msgr2=0x7fc4500b41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:12.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.462+0000 7fc44ffff700 1 --2- 172.21.15.60:0/2777936636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc45005d070 0x7fc4500b41c0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 rx=0x7fc440001820 tx=0x7fc448016000).stop 2022-06-29T16:05:12.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.462+0000 7fc44ffff700 1 -- 172.21.15.60:0/2777936636 shutdown_connections 2022-06-29T16:05:12.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.462+0000 7fc44ffff700 1 --2- 172.21.15.60:0/2777936636 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc42c053a70 0x7fc42c055f30 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:12.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.462+0000 7fc44ffff700 1 --2- 172.21.15.60:0/2777936636 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc45014af80 0x7fc450066a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:12.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.462+0000 7fc44ffff700 1 --2- 172.21.15.60:0/2777936636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4500b2130 0x7fc450066590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:12.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.462+0000 7fc44ffff700 1 --2- 172.21.15.60:0/2777936636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc45005d070 0x7fc4500b41c0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:12.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.462+0000 7fc44ffff700 1 -- 172.21.15.60:0/2777936636 >> 172.21.15.60:0/2777936636 conn(0x7fc4500b5890 msgr2=0x7fc4501452d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:12.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.462+0000 7fc44ffff700 1 -- 172.21.15.60:0/2777936636 shutdown_connections 2022-06-29T16:05:12.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.462+0000 7fc44ffff700 1 -- 172.21.15.60:0/2777936636 wait complete. 2022-06-29T16:05:12.474 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 32 2022-06-29T16:05:12.482 INFO:tasks.ceph.mds.a:Restarting daemon 2022-06-29T16:05:12.483 INFO:tasks.ceph.mds.a:Stopping old one... 2022-06-29T16:05:12.483 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-06-29T16:05:12.483 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T16:05:12.484 INFO:tasks.ceph.mds.b:Restarting daemon 2022-06-29T16:05:12.484 INFO:tasks.ceph.mds.b:Stopping old one... 2022-06-29T16:05:12.484 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-06-29T16:05:12.485 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T16:05:12.493 INFO:tasks.ceph.mds.a:Stopped 2022-06-29T16:05:12.494 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-06-29T16:05:12.496 INFO:tasks.ceph.mds.a:Started 2022-06-29T16:05:12.496 DEBUG:teuthology.parallel:result is None 2022-06-29T16:05:12.501 INFO:tasks.ceph.mds.b:Stopped 2022-06-29T16:05:12.502 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-06-29T16:05:12.546 INFO:tasks.ceph.mds.b:Started 2022-06-29T16:05:12.546 DEBUG:teuthology.parallel:result is None 2022-06-29T16:05:12.547 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-06-29T16:05:12.568 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2022-06-29T16:05:12.623 INFO:tasks.ceph.mds.b.smithi092.stdout:starting mds.b at 2022-06-29T16:05:12.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.642+0000 7fcd156ef700 1 Processor -- start 2022-06-29T16:05:12.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.642+0000 7fcd156ef700 1 -- start start 2022-06-29T16:05:12.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.642+0000 7fcd156ef700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd1005c5b0 0x7fcd1005c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:12.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.642+0000 7fcd156ef700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd100b0a40 con 0x7fcd1005fee0 2022-06-29T16:05:12.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.642+0000 7fcd156ef700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd100b0ce0 con 0x7fcd10060610 2022-06-29T16:05:12.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.642+0000 7fcd156ef700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd100b0850 con 0x7fcd1005c5b0 2022-06-29T16:05:12.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.646+0000 7fcd14eee700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcd1005fee0 0x7fcd10142a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57980/0 (socket says 172.21.15.60:57980) 2022-06-29T16:05:12.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.646+0000 7fcd14eee700 1 -- 172.21.15.60:0/601534396 learned_addr learned my addr 172.21.15.60:0/601534396 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:12.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.646+0000 7fcd0ffff700 1 --2- 172.21.15.60:0/601534396 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd1005c5b0 0x7fcd1005c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:12.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.646+0000 7fcd0effd700 1 -- 172.21.15.60:0/601534396 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 604572963 0 0) 0x7fcd100b0a40 con 0x7fcd1005fee0 2022-06-29T16:05:12.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.646+0000 7fcd0effd700 1 -- 172.21.15.60:0/601534396 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccfc003260 con 0x7fcd1005fee0 2022-06-29T16:05:12.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.646+0000 7fcd0effd700 1 -- 172.21.15.60:0/601534396 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 492578107 0 0) 0x7fcd100b0ce0 con 0x7fcd10060610 2022-06-29T16:05:12.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.646+0000 7fcd0effd700 1 -- 172.21.15.60:0/601534396 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd100b0a40 con 0x7fcd10060610 2022-06-29T16:05:12.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.646+0000 7fcd0ffff700 1 -- 172.21.15.60:0/601534396 >> v1:172.21.15.92:6789/0 conn(0x7fcd10060610 legacy=0x7fcd1013c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:12.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.646+0000 7fcd0ffff700 1 -- 172.21.15.60:0/601534396 >> v1:172.21.15.60:6789/0 conn(0x7fcd1005fee0 legacy=0x7fcd10142a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:12.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.646+0000 7fcd0ffff700 1 -- 172.21.15.60:0/601534396 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1013ce50 con 0x7fcd1005c5b0 2022-06-29T16:05:12.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.646+0000 7fcd0ffff700 1 --2- 172.21.15.60:0/601534396 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd1005c5b0 0x7fcd1005c990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fcd080032f0 tx=0x7fcd080091d0).ready entity=mon.2 client_cookie=510cefb58c6884b3 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:12.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.646+0000 7fcd0effd700 1 -- 172.21.15.60:0/601534396 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd0800b640 con 0x7fcd1005c5b0 2022-06-29T16:05:12.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.646+0000 7fcd0effd700 1 -- 172.21.15.60:0/601534396 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd0800b7a0 con 0x7fcd1005c5b0 2022-06-29T16:05:12.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.646+0000 7fcd0effd700 1 -- 172.21.15.60:0/601534396 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd0800b910 con 0x7fcd1005c5b0 2022-06-29T16:05:12.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.646+0000 7fcd156ef700 1 -- 172.21.15.60:0/601534396 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd1005c5b0 msgr2=0x7fcd1005c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:12.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.646+0000 7fcd156ef700 1 --2- 172.21.15.60:0/601534396 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd1005c5b0 0x7fcd1005c990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fcd080032f0 tx=0x7fcd080091d0).stop 2022-06-29T16:05:12.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.646+0000 7fcd156ef700 1 -- 172.21.15.60:0/601534396 shutdown_connections 2022-06-29T16:05:12.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.646+0000 7fcd156ef700 1 --2- 172.21.15.60:0/601534396 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd1005c5b0 0x7fcd1005c990 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:12.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.646+0000 7fcd156ef700 1 -- 172.21.15.60:0/601534396 >> 172.21.15.60:0/601534396 conn(0x7fcd10061c80 msgr2=0x7fcd100637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:12.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.646+0000 7fcd156ef700 1 -- 172.21.15.60:0/601534396 shutdown_connections 2022-06-29T16:05:12.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.646+0000 7fcd156ef700 1 -- 172.21.15.60:0/601534396 wait complete. 2022-06-29T16:05:12.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.646+0000 7fcd156ef700 1 Processor -- start 2022-06-29T16:05:12.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.646+0000 7fcd156ef700 1 -- start start 2022-06-29T16:05:12.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.650+0000 7fcd156ef700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd1005c5b0 0x7fcd10146a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:12.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.650+0000 7fcd156ef700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd1005fee0 0x7fcd10144210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:12.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.650+0000 7fcd156ef700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd10060610 0x7fcd100ae340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:12.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.650+0000 7fcd156ef700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd10145d00 con 0x7fcd1005fee0 2022-06-29T16:05:12.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.650+0000 7fcd156ef700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd101438c0 con 0x7fcd10060610 2022-06-29T16:05:12.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.650+0000 7fcd156ef700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd10140db0 con 0x7fcd1005c5b0 2022-06-29T16:05:12.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.650+0000 7fcd0ffff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd1005c5b0 0x7fcd10146a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:12.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.650+0000 7fcd0f7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd1005fee0 0x7fcd10144210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:12.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.650+0000 7fcd0ffff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd1005c5b0 0x7fcd10146a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:34758/0 (socket says 172.21.15.60:34758) 2022-06-29T16:05:12.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.650+0000 7fcd0ffff700 1 -- 172.21.15.60:0/3317693638 learned_addr learned my addr 172.21.15.60:0/3317693638 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:12.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.650+0000 7fcd14eee700 1 --2- 172.21.15.60:0/3317693638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd10060610 0x7fcd100ae340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:12.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.650+0000 7fcd0ffff700 1 -- 172.21.15.60:0/3317693638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd10060610 msgr2=0x7fcd100ae340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:12.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.650+0000 7fcd0ffff700 1 --2- 172.21.15.60:0/3317693638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd10060610 0x7fcd100ae340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:12.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.650+0000 7fcd0ffff700 1 -- 172.21.15.60:0/3317693638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd1005fee0 msgr2=0x7fcd10144210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:12.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.650+0000 7fcd0ffff700 1 --2- 172.21.15.60:0/3317693638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd1005fee0 0x7fcd10144210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:12.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.650+0000 7fcd0ffff700 1 -- 172.21.15.60:0/3317693638 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd08009470 con 0x7fcd1005c5b0 2022-06-29T16:05:12.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.650+0000 7fcd0f7fe700 1 --2- 172.21.15.60:0/3317693638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd1005fee0 0x7fcd10144210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:05:12.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.650+0000 7fcd0ffff700 1 --2- 172.21.15.60:0/3317693638 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd1005c5b0 0x7fcd10146a20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fcd080032f0 tx=0x7fcd08016000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:12.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.650+0000 7fcd0d7fa700 1 -- 172.21.15.60:0/3317693638 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd0800b910 con 0x7fcd1005c5b0 2022-06-29T16:05:12.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.650+0000 7fcd0d7fa700 1 -- 172.21.15.60:0/3317693638 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd08011e00 con 0x7fcd1005c5b0 2022-06-29T16:05:12.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.650+0000 7fcd0d7fa700 1 -- 172.21.15.60:0/3317693638 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd0800b640 con 0x7fcd1005c5b0 2022-06-29T16:05:12.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.650+0000 7fcd156ef700 1 -- 172.21.15.60:0/3317693638 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd101452d0 con 0x7fcd1005c5b0 2022-06-29T16:05:12.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.650+0000 7fcd156ef700 1 -- 172.21.15.60:0/3317693638 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd10140670 con 0x7fcd1005c5b0 2022-06-29T16:05:12.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.650+0000 7fcd156ef700 1 -- 172.21.15.60:0/3317693638 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd100573d0 con 0x7fcd1005c5b0 2022-06-29T16:05:12.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.654+0000 7fcd0d7fa700 1 -- 172.21.15.60:0/3317693638 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcd0800f070 con 0x7fcd1005c5b0 2022-06-29T16:05:12.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.654+0000 7fcd0d7fa700 1 --2- 172.21.15.60:0/3317693638 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fccf0053aa0 0x7fccf0055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:12.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.654+0000 7fcd0d7fa700 1 -- 172.21.15.60:0/3317693638 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fcd0806c8d0 con 0x7fcd1005c5b0 2022-06-29T16:05:12.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.654+0000 7fcd0f7fe700 1 --2- 172.21.15.60:0/3317693638 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fccf0053aa0 0x7fccf0055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:12.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.654+0000 7fcd0f7fe700 1 --2- 172.21.15.60:0/3317693638 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fccf0053aa0 0x7fccf0055f60 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fcd0400d000 tx=0x7fcd04007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:12.672 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.662+0000 7fcd0d7fa700 1 -- 172.21.15.60:0/3317693638 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcd08043af0 con 0x7fcd1005c5b0 2022-06-29T16:05:12.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.798+0000 7fcd156ef700 1 -- 172.21.15.60:0/3317693638 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fcd10068480 con 0x7fcd1005c5b0 2022-06-29T16:05:12.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.954+0000 7fcd0d7fa700 1 -- 172.21.15.60:0/3317693638 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v76) v1 ==== 97+0+0 (secure 0 0 0) 0x7fcd08024030 con 0x7fcd1005c5b0 2022-06-29T16:05:12.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.958+0000 7fcd156ef700 1 -- 172.21.15.60:0/3317693638 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fccf0053aa0 msgr2=0x7fccf0055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:12.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.958+0000 7fcd156ef700 1 --2- 172.21.15.60:0/3317693638 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fccf0053aa0 0x7fccf0055f60 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fcd0400d000 tx=0x7fcd04007a00).stop 2022-06-29T16:05:12.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.958+0000 7fcd156ef700 1 -- 172.21.15.60:0/3317693638 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd1005c5b0 msgr2=0x7fcd10146a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:12.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.958+0000 7fcd156ef700 1 --2- 172.21.15.60:0/3317693638 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd1005c5b0 0x7fcd10146a20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fcd080032f0 tx=0x7fcd08016000).stop 2022-06-29T16:05:12.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.958+0000 7fcd156ef700 1 -- 172.21.15.60:0/3317693638 shutdown_connections 2022-06-29T16:05:12.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.958+0000 7fcd156ef700 1 --2- 172.21.15.60:0/3317693638 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fccf0053aa0 0x7fccf0055f60 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:12.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.958+0000 7fcd156ef700 1 --2- 172.21.15.60:0/3317693638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd10060610 0x7fcd100ae340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:12.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.958+0000 7fcd156ef700 1 --2- 172.21.15.60:0/3317693638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd1005fee0 0x7fcd10144210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:12.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.958+0000 7fcd156ef700 1 --2- 172.21.15.60:0/3317693638 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd1005c5b0 0x7fcd10146a20 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:12.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.958+0000 7fcd156ef700 1 -- 172.21.15.60:0/3317693638 >> 172.21.15.60:0/3317693638 conn(0x7fcd10061c80 msgr2=0x7fcd1013bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:12.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.958+0000 7fcd156ef700 1 -- 172.21.15.60:0/3317693638 shutdown_connections 2022-06-29T16:05:12.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:12.958+0000 7fcd156ef700 1 -- 172.21.15.60:0/3317693638 wait complete. 2022-06-29T16:05:12.968 INFO:teuthology.orchestra.run.smithi060.stderr: removed all blacklist entries 2022-06-29T16:05:12.982 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-06-29T16:05:12.982 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-06-29T16:05:13.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.122+0000 7f16485c6700 1 Processor -- start 2022-06-29T16:05:13.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.122+0000 7f16485c6700 1 -- start start 2022-06-29T16:05:13.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.122+0000 7f16485c6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f164013a920 0x7f1640143f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:13.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.122+0000 7f16485c6700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1640140ab0 0x7f1640140f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:13.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.122+0000 7f16485c6700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f164013ad00 con 0x7f16400ae270 2022-06-29T16:05:13.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.122+0000 7f16485c6700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1640144420 con 0x7f1640140ab0 2022-06-29T16:05:13.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.122+0000 7f16485c6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16400b5d20 con 0x7f164013a920 2022-06-29T16:05:13.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.122+0000 7f1646b63700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1640140ab0 0x7f1640140f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:13.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.122+0000 7f1645b61700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f16400ae270 0x7f16400b5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57990/0 (socket says 172.21.15.60:57990) 2022-06-29T16:05:13.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.122+0000 7f1645b61700 1 -- 172.21.15.60:0/4211845552 learned_addr learned my addr 172.21.15.60:0/4211845552 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:13.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.122+0000 7f1646362700 1 --2- 172.21.15.60:0/4211845552 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f164013a920 0x7f1640143f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:13.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.122+0000 7f1646b63700 1 -- 172.21.15.60:0/4211845552 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f164013a920 msgr2=0x7f1640143f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:13.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.122+0000 7f1646b63700 1 --2- 172.21.15.60:0/4211845552 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f164013a920 0x7f1640143f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:13.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.122+0000 7f1646b63700 1 -- 172.21.15.60:0/4211845552 >> v1:172.21.15.60:6789/0 conn(0x7f16400ae270 legacy=0x7f16400b5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:13.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.122+0000 7f1646b63700 1 -- 172.21.15.60:0/4211845552 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1640144670 con 0x7f1640140ab0 2022-06-29T16:05:13.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.122+0000 7f1645360700 1 -- 172.21.15.60:0/4211845552 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1541427263 0 0) 0x7f164013ad00 con 0x7f16400ae270 2022-06-29T16:05:13.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.122+0000 7f1646b63700 1 --2- 172.21.15.60:0/4211845552 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1640140ab0 0x7f1640140f40 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 rx=0x7f16380032f0 tx=0x7f16380091d0).ready entity=mon.0 client_cookie=fb51646f4befeaa8 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:13.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.126+0000 7f1645360700 1 -- 172.21.15.60:0/4211845552 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f163800b6c0 con 0x7f1640140ab0 2022-06-29T16:05:13.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.126+0000 7f1645360700 1 -- 172.21.15.60:0/4211845552 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f163800b820 con 0x7f1640140ab0 2022-06-29T16:05:13.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.126+0000 7f1645360700 1 -- 172.21.15.60:0/4211845552 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f163800b990 con 0x7f1640140ab0 2022-06-29T16:05:13.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.126+0000 7f16485c6700 1 -- 172.21.15.60:0/4211845552 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1640140ab0 msgr2=0x7f1640140f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:13.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.126+0000 7f16485c6700 1 --2- 172.21.15.60:0/4211845552 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1640140ab0 0x7f1640140f40 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 rx=0x7f16380032f0 tx=0x7f16380091d0).stop 2022-06-29T16:05:13.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.126+0000 7f16485c6700 1 -- 172.21.15.60:0/4211845552 shutdown_connections 2022-06-29T16:05:13.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.126+0000 7f16485c6700 1 --2- 172.21.15.60:0/4211845552 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1640140ab0 0x7f1640140f40 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:13.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.126+0000 7f16485c6700 1 --2- 172.21.15.60:0/4211845552 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f164013a920 0x7f1640143f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:13.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.126+0000 7f16485c6700 1 -- 172.21.15.60:0/4211845552 >> 172.21.15.60:0/4211845552 conn(0x7f16400632c0 msgr2=0x7f16400b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:13.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.126+0000 7f16485c6700 1 -- 172.21.15.60:0/4211845552 shutdown_connections 2022-06-29T16:05:13.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.126+0000 7f16485c6700 1 -- 172.21.15.60:0/4211845552 wait complete. 2022-06-29T16:05:13.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.126+0000 7f16485c6700 1 Processor -- start 2022-06-29T16:05:13.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.126+0000 7f16485c6700 1 -- start start 2022-06-29T16:05:13.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.130+0000 7f16485c6700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16400ae270 0x7f1640143de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:13.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.130+0000 7f16485c6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f164013a920 0x7f16401422c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:13.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.130+0000 7f16485c6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1640140ab0 0x7f1640136aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:13.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.130+0000 7f16485c6700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f164013ad00 con 0x7f164013a920 2022-06-29T16:05:13.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.130+0000 7f16485c6700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16400642b0 con 0x7f16400ae270 2022-06-29T16:05:13.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.130+0000 7f16485c6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f164013af30 con 0x7f1640140ab0 2022-06-29T16:05:13.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.130+0000 7f1645b61700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f164013a920 0x7f16401422c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:13.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.130+0000 7f1645b61700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f164013a920 0x7f16401422c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49912/0 (socket says 172.21.15.60:49912) 2022-06-29T16:05:13.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.130+0000 7f1645b61700 1 -- 172.21.15.60:0/2812849243 learned_addr learned my addr 172.21.15.60:0/2812849243 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:13.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.130+0000 7f1646b63700 1 --2- 172.21.15.60:0/2812849243 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1640140ab0 0x7f1640136aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:13.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.130+0000 7f1646362700 1 --2- 172.21.15.60:0/2812849243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16400ae270 0x7f1640143de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:13.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.130+0000 7f1645b61700 1 -- 172.21.15.60:0/2812849243 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1640140ab0 msgr2=0x7f1640136aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:13.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.130+0000 7f1645b61700 1 --2- 172.21.15.60:0/2812849243 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1640140ab0 0x7f1640136aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:13.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.130+0000 7f1645b61700 1 -- 172.21.15.60:0/2812849243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16400ae270 msgr2=0x7f1640143de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:13.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.130+0000 7f1645b61700 1 --2- 172.21.15.60:0/2812849243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16400ae270 0x7f1640143de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:13.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.130+0000 7f1645b61700 1 -- 172.21.15.60:0/2812849243 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1638009470 con 0x7f164013a920 2022-06-29T16:05:13.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.130+0000 7f1645b61700 1 --2- 172.21.15.60:0/2812849243 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f164013a920 0x7f16401422c0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 rx=0x7f1630000ed0 tx=0x7f1630007ac0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:13.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.130+0000 7f162f7fe700 1 -- 172.21.15.60:0/2812849243 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f163000ce50 con 0x7f164013a920 2022-06-29T16:05:13.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.130+0000 7f162f7fe700 1 -- 172.21.15.60:0/2812849243 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16300072e0 con 0x7f164013a920 2022-06-29T16:05:13.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.130+0000 7f16485c6700 1 -- 172.21.15.60:0/2812849243 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1640064640 con 0x7f164013a920 2022-06-29T16:05:13.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.130+0000 7f162f7fe700 1 -- 172.21.15.60:0/2812849243 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1630005610 con 0x7f164013a920 2022-06-29T16:05:13.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.130+0000 7f16485c6700 1 -- 172.21.15.60:0/2812849243 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1640064b10 con 0x7f164013a920 2022-06-29T16:05:13.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.134+0000 7f16485c6700 1 -- 172.21.15.60:0/2812849243 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1640064d00 con 0x7f164013a920 2022-06-29T16:05:13.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.134+0000 7f162f7fe700 1 -- 172.21.15.60:0/2812849243 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1630010070 con 0x7f164013a920 2022-06-29T16:05:13.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.134+0000 7f162f7fe700 1 --2- 172.21.15.60:0/2812849243 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1620053b00 0x7f1620055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:13.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.134+0000 7f162f7fe700 1 -- 172.21.15.60:0/2812849243 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f163006b450 con 0x7f164013a920 2022-06-29T16:05:13.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.134+0000 7f1646362700 1 --2- 172.21.15.60:0/2812849243 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1620053b00 0x7f1620055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:13.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.134+0000 7f1646362700 1 --2- 172.21.15.60:0/2812849243 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1620053b00 0x7f1620055fc0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f163c00b000 tx=0x7f163c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:13.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.138+0000 7f162f7fe700 1 -- 172.21.15.60:0/2812849243 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1630042600 con 0x7f164013a920 2022-06-29T16:05:13.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.306+0000 7f16485c6700 1 -- 172.21.15.60:0/2812849243 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f1640144910 con 0x7f164013a920 2022-06-29T16:05:13.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.306+0000 7f162f7fe700 1 -- 172.21.15.60:0/2812849243 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v27) v1 ==== 80+0+4242 (secure 0 0 0) 0x7f1630045c20 con 0x7f164013a920 2022-06-29T16:05:13.315 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:13.315 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:05:13.316 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_dump": [ 2022-06-29T16:05:13.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:13.316 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.a", 2022-06-29T16:05:13.316 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxivv7JNhAAnA291UxlB/SfgITOipP00A==", 2022-06-29T16:05:13.317 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:05:13.317 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:05:13.317 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:05:13.317 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:05:13.318 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:05:13.318 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:13.318 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:13.318 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:13.319 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.b", 2022-06-29T16:05:13.319 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxiRSr/OhAAfFQTF2YHvzCv5wXBVCA5WQ==", 2022-06-29T16:05:13.319 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:05:13.319 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:05:13.319 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:05:13.320 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:05:13.320 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:05:13.320 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:13.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:13.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:13.321 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.0", 2022-06-29T16:05:13.321 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiZYPxOBAAIkweWJlTJIXgxnlagkGdXA==", 2022-06-29T16:05:13.321 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:05:13.322 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:05:13.322 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:05:13.322 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:05:13.322 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:13.323 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:13.323 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:13.323 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.1", 2022-06-29T16:05:13.323 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxiS2JGBhAAJn0UfAxZI3IVFyD5QBdb/Q==", 2022-06-29T16:05:13.323 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:05:13.324 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:05:13.324 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:05:13.324 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:05:13.324 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:13.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:13.325 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:13.325 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.2", 2022-06-29T16:05:13.325 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxiPhkfEhAApH7r8OUJzW7fZNM6XVv1dg==", 2022-06-29T16:05:13.325 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:05:13.326 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:05:13.326 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:05:13.326 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:05:13.326 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:13.327 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:13.327 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:13.327 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.3", 2022-06-29T16:05:13.327 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxit/47HhAA1vyU6h7hjPG+op9qP/rVYQ==", 2022-06-29T16:05:13.328 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:05:13.328 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:05:13.328 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:05:13.328 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:05:13.329 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:13.329 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:13.329 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:13.329 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.4", 2022-06-29T16:05:13.330 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxiaKzKBxAAbIwCF9ag7ezsNy11hi0BEw==", 2022-06-29T16:05:13.330 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:05:13.330 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:05:13.330 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:05:13.331 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:05:13.331 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:13.331 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:13.331 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:13.332 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.5", 2022-06-29T16:05:13.332 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxiJQV/IBAASBBDb6UFgEtXILe4fcT/xA==", 2022-06-29T16:05:13.332 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:05:13.332 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:05:13.333 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:05:13.333 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:05:13.333 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:13.333 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:13.333 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:13.334 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.6", 2022-06-29T16:05:13.334 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxi6D07OBAA2FWiZ+mNK+jFMp/kPZnbQQ==", 2022-06-29T16:05:13.334 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:05:13.334 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:05:13.335 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:05:13.335 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:05:13.335 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:13.335 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:13.336 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:13.336 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.7", 2022-06-29T16:05:13.336 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCMdrxinEOcEhAA562ybpzGHNQfWIq8q7zw9w==", 2022-06-29T16:05:13.336 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:05:13.336 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:05:13.337 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:05:13.337 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:05:13.337 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:13.337 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:13.338 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:13.338 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.0", 2022-06-29T16:05:13.338 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiHXPwBRAAkOOwlYJvIW7dqVFdlXXoqw==", 2022-06-29T16:05:13.338 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:05:13.339 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:05:13.339 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2022-06-29T16:05:13.339 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw pool=cephfs_data" 2022-06-29T16:05:13.339 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:13.339 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:13.340 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:13.340 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.1", 2022-06-29T16:05:13.340 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxibe5JCRAAGBJbi9t09w0836JqrAT7+w==", 2022-06-29T16:05:13.340 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:05:13.341 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:05:13.341 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2022-06-29T16:05:13.341 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw pool=cephfs_data" 2022-06-29T16:05:13.341 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:13.342 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:13.342 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:13.342 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.2", 2022-06-29T16:05:13.342 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiK3dBDRAA7XINPSOPQDfOrUYEpHC+gw==", 2022-06-29T16:05:13.343 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:05:13.343 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:05:13.343 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2022-06-29T16:05:13.343 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw pool=cephfs_data" 2022-06-29T16:05:13.343 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:13.344 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:13.344 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:13.344 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.admin", 2022-06-29T16:05:13.344 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxieKVcGBAAnjH80a0qWIArz2qwjYHWdQ==", 2022-06-29T16:05:13.345 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:05:13.345 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T16:05:13.345 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:05:13.345 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:05:13.346 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:05:13.346 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:13.346 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:13.346 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:13.346 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.x", 2022-06-29T16:05:13.347 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxi2LZJNBAA3MKb/pXYbQCsoRx8EQfGbw==", 2022-06-29T16:05:13.347 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:05:13.347 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T16:05:13.347 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2022-06-29T16:05:13.348 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:05:13.348 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:13.348 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:13.348 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:13.349 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.y", 2022-06-29T16:05:13.349 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxi3LmaMBAAN+0mZrdKXqSXOipVTA1drw==", 2022-06-29T16:05:13.349 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:05:13.349 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T16:05:13.350 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2022-06-29T16:05:13.350 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:05:13.350 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:13.350 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:13.350 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:13.351 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:05:13.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.310+0000 7f16485c6700 1 -- 172.21.15.60:0/2812849243 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1620053b00 msgr2=0x7f1620055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:13.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.310+0000 7f16485c6700 1 --2- 172.21.15.60:0/2812849243 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1620053b00 0x7f1620055fc0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f163c00b000 tx=0x7f163c008040).stop 2022-06-29T16:05:13.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.310+0000 7f16485c6700 1 -- 172.21.15.60:0/2812849243 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f164013a920 msgr2=0x7f16401422c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:13.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.310+0000 7f16485c6700 1 --2- 172.21.15.60:0/2812849243 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f164013a920 0x7f16401422c0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 rx=0x7f1630000ed0 tx=0x7f1630007ac0).stop 2022-06-29T16:05:13.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.310+0000 7f16485c6700 1 -- 172.21.15.60:0/2812849243 shutdown_connections 2022-06-29T16:05:13.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.310+0000 7f16485c6700 1 --2- 172.21.15.60:0/2812849243 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1620053b00 0x7f1620055fc0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:13.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.310+0000 7f16485c6700 1 --2- 172.21.15.60:0/2812849243 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1640140ab0 0x7f1640136aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:13.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.310+0000 7f16485c6700 1 --2- 172.21.15.60:0/2812849243 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f164013a920 0x7f16401422c0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:13.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.310+0000 7f16485c6700 1 --2- 172.21.15.60:0/2812849243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16400ae270 0x7f1640143de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:13.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.310+0000 7f16485c6700 1 -- 172.21.15.60:0/2812849243 >> 172.21.15.60:0/2812849243 conn(0x7f16400632c0 msgr2=0x7f1640137d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:13.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.310+0000 7f16485c6700 1 -- 172.21.15.60:0/2812849243 shutdown_connections 2022-06-29T16:05:13.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.310+0000 7f16485c6700 1 -- 172.21.15.60:0/2812849243 wait complete. 2022-06-29T16:05:13.356 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:05:13.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.458+0000 7f163ebf6700 1 Processor -- start 2022-06-29T16:05:13.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.458+0000 7f163ebf6700 1 -- start start 2022-06-29T16:05:13.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.458+0000 7f163ebf6700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f16380b2130 0x7f1638146100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:13.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.458+0000 7f163ebf6700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f16380aed00 0x7f16381465f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:13.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.458+0000 7f163ebf6700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f163805d450 con 0x7f163805d070 2022-06-29T16:05:13.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.458+0000 7f163ebf6700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16380adf90 con 0x7f16380b2130 2022-06-29T16:05:13.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.458+0000 7f163ebf6700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16380b25d0 con 0x7f16380aed00 2022-06-29T16:05:13.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.458+0000 7f163d3f3700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f16380b2130 0x7f1638146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:13.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.458+0000 7f163e3f5700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f16380aed00 0x7f16381465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:13.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.458+0000 7f163d3f3700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f16380b2130 0x7f1638146100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49920/0 (socket says 172.21.15.60:49920) 2022-06-29T16:05:13.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.458+0000 7f163dbf4700 1 --1- 172.21.15.60:0/1231232905 >> v1:172.21.15.92:6789/0 conn(0x7f163805d070 0x7f163814aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:34020/0 (socket says 172.21.15.60:34020) 2022-06-29T16:05:13.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.458+0000 7f163d3f3700 1 -- 172.21.15.60:0/1231232905 learned_addr learned my addr 172.21.15.60:0/1231232905 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:13.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.458+0000 7f163d3f3700 1 -- 172.21.15.60:0/1231232905 >> v2:172.21.15.92:3300/0 conn(0x7f16380aed00 msgr2=0x7f16381465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:13.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.458+0000 7f163d3f3700 1 --2- 172.21.15.60:0/1231232905 >> v2:172.21.15.92:3300/0 conn(0x7f16380aed00 0x7f16381465f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:13.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.458+0000 7f163cbf2700 1 -- 172.21.15.60:0/1231232905 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3708042230 0 0) 0x7f163805d450 con 0x7f163805d070 2022-06-29T16:05:13.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.458+0000 7f163d3f3700 1 -- 172.21.15.60:0/1231232905 >> v1:172.21.15.92:6789/0 conn(0x7f163805d070 legacy=0x7f163814aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:13.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.458+0000 7f163d3f3700 1 -- 172.21.15.60:0/1231232905 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f163814a350 con 0x7f16380b2130 2022-06-29T16:05:13.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.458+0000 7f163e3f5700 1 --2- 172.21.15.60:0/1231232905 >> v2:172.21.15.92:3300/0 conn(0x7f16380aed00 0x7f16381465f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:05:13.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.458+0000 7f163d3f3700 1 --2- 172.21.15.60:0/1231232905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16380b2130 0x7f1638146100 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 rx=0x7f1628002f70 tx=0x7f1628006eb0).ready entity=mon.0 client_cookie=dbd0bd04db13a469 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:13.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.458+0000 7f163cbf2700 1 -- 172.21.15.60:0/1231232905 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f162800d5c0 con 0x7f16380b2130 2022-06-29T16:05:13.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.458+0000 7f163cbf2700 1 -- 172.21.15.60:0/1231232905 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1628000e90 con 0x7f16380b2130 2022-06-29T16:05:13.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.458+0000 7f163cbf2700 1 -- 172.21.15.60:0/1231232905 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f162800d910 con 0x7f16380b2130 2022-06-29T16:05:13.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.458+0000 7f163ebf6700 1 -- 172.21.15.60:0/1231232905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16380b2130 msgr2=0x7f1638146100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:13.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.458+0000 7f163ebf6700 1 --2- 172.21.15.60:0/1231232905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16380b2130 0x7f1638146100 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 rx=0x7f1628002f70 tx=0x7f1628006eb0).stop 2022-06-29T16:05:13.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.458+0000 7f163ebf6700 1 -- 172.21.15.60:0/1231232905 shutdown_connections 2022-06-29T16:05:13.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.458+0000 7f163ebf6700 1 --2- 172.21.15.60:0/1231232905 >> v2:172.21.15.92:3300/0 conn(0x7f16380aed00 0x7f16381465f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:13.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.458+0000 7f163ebf6700 1 --2- 172.21.15.60:0/1231232905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16380b2130 0x7f1638146100 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:13.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.458+0000 7f163ebf6700 1 -- 172.21.15.60:0/1231232905 >> 172.21.15.60:0/1231232905 conn(0x7f16380b5890 msgr2=0x7f16380b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:13.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.458+0000 7f163ebf6700 1 -- 172.21.15.60:0/1231232905 shutdown_connections 2022-06-29T16:05:13.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.458+0000 7f163ebf6700 1 -- 172.21.15.60:0/1231232905 wait complete. 2022-06-29T16:05:13.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.462+0000 7f163ebf6700 1 Processor -- start 2022-06-29T16:05:13.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.462+0000 7f163ebf6700 1 -- start start 2022-06-29T16:05:13.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.462+0000 7f163ebf6700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f163805d070 0x7f16380658e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:13.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.462+0000 7f163ebf6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16380aed00 0x7f1638064210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:13.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.462+0000 7f163ebf6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16380b2130 0x7f1638064700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:13.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.462+0000 7f163ebf6700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1638142cb0 con 0x7f16380aed00 2022-06-29T16:05:13.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.462+0000 7f163ebf6700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1638117b40 con 0x7f163805d070 2022-06-29T16:05:13.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.462+0000 7f163ebf6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16381188c0 con 0x7f16380b2130 2022-06-29T16:05:13.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.462+0000 7f163d3f3700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16380aed00 0x7f1638064210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:13.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.462+0000 7f163d3f3700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16380aed00 0x7f1638064210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49926/0 (socket says 172.21.15.60:49926) 2022-06-29T16:05:13.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.462+0000 7f163d3f3700 1 -- 172.21.15.60:0/218238305 learned_addr learned my addr 172.21.15.60:0/218238305 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:13.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.462+0000 7f163dbf4700 1 --2- 172.21.15.60:0/218238305 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f163805d070 0x7f16380658e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:13.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.462+0000 7f163e3f5700 1 --2- 172.21.15.60:0/218238305 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16380b2130 0x7f1638064700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:13.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.462+0000 7f163d3f3700 1 -- 172.21.15.60:0/218238305 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16380b2130 msgr2=0x7f1638064700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:13.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.462+0000 7f163d3f3700 1 --2- 172.21.15.60:0/218238305 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16380b2130 0x7f1638064700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:13.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.462+0000 7f163d3f3700 1 -- 172.21.15.60:0/218238305 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f163805d070 msgr2=0x7f16380658e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:13.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.462+0000 7f163d3f3700 1 --2- 172.21.15.60:0/218238305 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f163805d070 0x7f16380658e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:13.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.462+0000 7f163d3f3700 1 -- 172.21.15.60:0/218238305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1630009470 con 0x7f16380aed00 2022-06-29T16:05:13.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.462+0000 7f163dbf4700 1 --2- 172.21.15.60:0/218238305 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f163805d070 0x7f16380658e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:13.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.462+0000 7f163d3f3700 1 --2- 172.21.15.60:0/218238305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16380aed00 0x7f1638064210 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 rx=0x7f1628002f70 tx=0x7f1628015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:13.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.462+0000 7f162effd700 1 -- 172.21.15.60:0/218238305 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1628006b40 con 0x7f16380aed00 2022-06-29T16:05:13.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.462+0000 7f162effd700 1 -- 172.21.15.60:0/218238305 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f162800de40 con 0x7f16380aed00 2022-06-29T16:05:13.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.462+0000 7f163ebf6700 1 -- 172.21.15.60:0/218238305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f162800b1e0 con 0x7f16380aed00 2022-06-29T16:05:13.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.462+0000 7f162effd700 1 -- 172.21.15.60:0/218238305 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1628011e90 con 0x7f16380aed00 2022-06-29T16:05:13.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.462+0000 7f163ebf6700 1 -- 172.21.15.60:0/218238305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1638117240 con 0x7f16380aed00 2022-06-29T16:05:13.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.466+0000 7f162effd700 1 -- 172.21.15.60:0/218238305 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f162801a040 con 0x7f16380aed00 2022-06-29T16:05:13.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.466+0000 7f163ebf6700 1 -- 172.21.15.60:0/218238305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1638118a80 con 0x7f16380aed00 2022-06-29T16:05:13.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.466+0000 7f162effd700 1 --2- 172.21.15.60:0/218238305 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1620053b00 0x7f1620055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:13.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.466+0000 7f162effd700 1 -- 172.21.15.60:0/218238305 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f162806cab0 con 0x7f16380aed00 2022-06-29T16:05:13.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.466+0000 7f163dbf4700 1 --2- 172.21.15.60:0/218238305 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1620053b00 0x7f1620055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:13.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.466+0000 7f163dbf4700 1 --2- 172.21.15.60:0/218238305 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1620053b00 0x7f1620055fc0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f163400b000 tx=0x7f1634007100).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:13.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.474+0000 7f162effd700 1 -- 172.21.15.60:0/218238305 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1638118a80 con 0x7f16380aed00 2022-06-29T16:05:13.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.602+0000 7f163ebf6700 1 -- 172.21.15.60:0/218238305 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1638118a80 con 0x7f16380aed00 2022-06-29T16:05:13.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.602+0000 7f162effd700 1 -- 172.21.15.60:0/218238305 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+13182 (secure 0 0 0) 0x7f1638118a80 con 0x7f16380aed00 2022-06-29T16:05:13.612 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:13.612 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":76,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:05:12.948588+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:05:13.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.606+0000 7f163ebf6700 1 -- 172.21.15.60:0/218238305 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1620053b00 msgr2=0x7f1620055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:13.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.606+0000 7f163ebf6700 1 --2- 172.21.15.60:0/218238305 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1620053b00 0x7f1620055fc0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f163400b000 tx=0x7f1634007100).stop 2022-06-29T16:05:13.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.606+0000 7f163ebf6700 1 -- 172.21.15.60:0/218238305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16380aed00 msgr2=0x7f1638064210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:13.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.606+0000 7f163ebf6700 1 --2- 172.21.15.60:0/218238305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16380aed00 0x7f1638064210 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 rx=0x7f1628002f70 tx=0x7f1628015000).stop 2022-06-29T16:05:13.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.606+0000 7f163ebf6700 1 -- 172.21.15.60:0/218238305 shutdown_connections 2022-06-29T16:05:13.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.606+0000 7f163ebf6700 1 --2- 172.21.15.60:0/218238305 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1620053b00 0x7f1620055fc0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:13.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.606+0000 7f163ebf6700 1 --2- 172.21.15.60:0/218238305 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f16380b2130 0x7f1638064700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:13.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.606+0000 7f163ebf6700 1 --2- 172.21.15.60:0/218238305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16380aed00 0x7f1638064210 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:13.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.606+0000 7f163ebf6700 1 --2- 172.21.15.60:0/218238305 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f163805d070 0x7f16380658e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:13.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.606+0000 7f163ebf6700 1 -- 172.21.15.60:0/218238305 >> 172.21.15.60:0/218238305 conn(0x7f16380b5890 msgr2=0x7f16380b1950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:13.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.606+0000 7f163ebf6700 1 -- 172.21.15.60:0/218238305 shutdown_connections 2022-06-29T16:05:13.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.606+0000 7f163ebf6700 1 -- 172.21.15.60:0/218238305 wait complete. 2022-06-29T16:05:13.625 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-06-29T16:05:13.626 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:05:13.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.726+0000 7fc03d5ab700 1 Processor -- start 2022-06-29T16:05:13.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.726+0000 7fc03d5ab700 1 -- start start 2022-06-29T16:05:13.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.726+0000 7fc03d5ab700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc0380b2800 0x7fc0380b2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:13.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.726+0000 7fc03d5ab700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc03805bf60 0x7fc0380af240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:13.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.726+0000 7fc03d5ab700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0380acf00 con 0x7fc0380af730 2022-06-29T16:05:13.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.726+0000 7fc03d5ab700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0381437a0 con 0x7fc0380b2800 2022-06-29T16:05:13.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.726+0000 7fc03d5ab700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc038143a40 con 0x7fc03805bf60 2022-06-29T16:05:13.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.730+0000 7fc03cdaa700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc0380af730 0x7fc0380afff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58022/0 (socket says 172.21.15.60:58022) 2022-06-29T16:05:13.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.730+0000 7fc03cdaa700 1 -- 172.21.15.60:0/1254989671 learned_addr learned my addr 172.21.15.60:0/1254989671 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:13.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.730+0000 7fc0377fe700 1 --2- 172.21.15.60:0/1254989671 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc03805bf60 0x7fc0380af240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:13.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.730+0000 7fc037fff700 1 --2- 172.21.15.60:0/1254989671 >> v2:172.21.15.92:3300/0 conn(0x7fc0380b2800 0x7fc0380b2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:13.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.730+0000 7fc036ffd700 1 -- 172.21.15.60:0/1254989671 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2281341693 0 0) 0x7fc0380acf00 con 0x7fc0380af730 2022-06-29T16:05:13.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.730+0000 7fc0377fe700 1 -- 172.21.15.60:0/1254989671 >> v2:172.21.15.92:3300/0 conn(0x7fc0380b2800 msgr2=0x7fc0380b2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:13.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.730+0000 7fc0377fe700 1 --2- 172.21.15.60:0/1254989671 >> v2:172.21.15.92:3300/0 conn(0x7fc0380b2800 0x7fc0380b2be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:13.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.730+0000 7fc0377fe700 1 -- 172.21.15.60:0/1254989671 >> v1:172.21.15.60:6789/0 conn(0x7fc0380af730 legacy=0x7fc0380afff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:13.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.730+0000 7fc0377fe700 1 -- 172.21.15.60:0/1254989671 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc038067420 con 0x7fc03805bf60 2022-06-29T16:05:13.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.730+0000 7fc0377fe700 1 --2- 172.21.15.60:0/1254989671 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc03805bf60 0x7fc0380af240 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fc02c002f70 tx=0x7fc02c006df0).ready entity=mon.2 client_cookie=4757eb161cb78623 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:13.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.730+0000 7fc036ffd700 1 -- 172.21.15.60:0/1254989671 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc02c00d570 con 0x7fc03805bf60 2022-06-29T16:05:13.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.730+0000 7fc036ffd700 1 -- 172.21.15.60:0/1254989671 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc02c000e90 con 0x7fc03805bf60 2022-06-29T16:05:13.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.730+0000 7fc036ffd700 1 -- 172.21.15.60:0/1254989671 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc02c00d8c0 con 0x7fc03805bf60 2022-06-29T16:05:13.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.730+0000 7fc03d5ab700 1 -- 172.21.15.60:0/1254989671 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc03805bf60 msgr2=0x7fc0380af240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:13.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.730+0000 7fc03d5ab700 1 --2- 172.21.15.60:0/1254989671 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc03805bf60 0x7fc0380af240 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fc02c002f70 tx=0x7fc02c006df0).stop 2022-06-29T16:05:13.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.730+0000 7fc03d5ab700 1 -- 172.21.15.60:0/1254989671 shutdown_connections 2022-06-29T16:05:13.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.730+0000 7fc03d5ab700 1 --2- 172.21.15.60:0/1254989671 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc03805bf60 0x7fc0380af240 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:13.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.730+0000 7fc03d5ab700 1 --2- 172.21.15.60:0/1254989671 >> v2:172.21.15.92:3300/0 conn(0x7fc0380b2800 0x7fc0380b2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:13.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.730+0000 7fc03d5ab700 1 -- 172.21.15.60:0/1254989671 >> 172.21.15.60:0/1254989671 conn(0x7fc0380b6120 msgr2=0x7fc0380b20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:13.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.730+0000 7fc03d5ab700 1 -- 172.21.15.60:0/1254989671 shutdown_connections 2022-06-29T16:05:13.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.730+0000 7fc03d5ab700 1 -- 172.21.15.60:0/1254989671 wait complete. 2022-06-29T16:05:13.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.734+0000 7fc03d5ab700 1 Processor -- start 2022-06-29T16:05:13.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.734+0000 7fc03d5ab700 1 -- start start 2022-06-29T16:05:13.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.734+0000 7fc03d5ab700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc03805bf60 0x7fc03809e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:13.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.734+0000 7fc03d5ab700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0380af730 0x7fc03809e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:13.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.734+0000 7fc03d5ab700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0380b2800 0x7fc038154f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:13.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.734+0000 7fc03d5ab700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0381482d0 con 0x7fc0380af730 2022-06-29T16:05:13.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.734+0000 7fc03d5ab700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc038147c00 con 0x7fc0380b2800 2022-06-29T16:05:13.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.734+0000 7fc03d5ab700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0380a0b00 con 0x7fc03805bf60 2022-06-29T16:05:13.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.734+0000 7fc037fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc03805bf60 0x7fc03809e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:13.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.734+0000 7fc0377fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0380af730 0x7fc03809e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:13.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.734+0000 7fc037fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc03805bf60 0x7fc03809e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:34800/0 (socket says 172.21.15.60:34800) 2022-06-29T16:05:13.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.734+0000 7fc037fff700 1 -- 172.21.15.60:0/276851955 learned_addr learned my addr 172.21.15.60:0/276851955 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:13.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.734+0000 7fc03cdaa700 1 --2- 172.21.15.60:0/276851955 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0380b2800 0x7fc038154f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:13.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.734+0000 7fc0377fe700 1 -- 172.21.15.60:0/276851955 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc03805bf60 msgr2=0x7fc03809e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:13.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.734+0000 7fc0377fe700 1 --2- 172.21.15.60:0/276851955 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc03805bf60 0x7fc03809e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:13.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.734+0000 7fc0377fe700 1 -- 172.21.15.60:0/276851955 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0380b2800 msgr2=0x7fc038154f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:13.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.734+0000 7fc0377fe700 1 --2- 172.21.15.60:0/276851955 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0380b2800 0x7fc038154f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:13.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.734+0000 7fc0377fe700 1 -- 172.21.15.60:0/276851955 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc02c00b180 con 0x7fc0380af730 2022-06-29T16:05:13.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.734+0000 7fc0377fe700 1 --2- 172.21.15.60:0/276851955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0380af730 0x7fc03809e7f0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 rx=0x7fc02c002f70 tx=0x7fc02c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:13.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.738+0000 7fc0357fa700 1 -- 172.21.15.60:0/276851955 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc02c00dea0 con 0x7fc0380af730 2022-06-29T16:05:13.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.738+0000 7fc0357fa700 1 -- 172.21.15.60:0/276851955 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc02c013de0 con 0x7fc0380af730 2022-06-29T16:05:13.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.738+0000 7fc03d5ab700 1 -- 172.21.15.60:0/276851955 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0380a0da0 con 0x7fc0380af730 2022-06-29T16:05:13.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.738+0000 7fc0357fa700 1 -- 172.21.15.60:0/276851955 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc02c01e5f0 con 0x7fc0380af730 2022-06-29T16:05:13.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.738+0000 7fc03d5ab700 1 -- 172.21.15.60:0/276851955 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc038148c60 con 0x7fc0380af730 2022-06-29T16:05:13.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.738+0000 7fc03d5ab700 1 -- 172.21.15.60:0/276851955 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0380573d0 con 0x7fc0380af730 2022-06-29T16:05:13.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.738+0000 7fc0357fa700 1 -- 172.21.15.60:0/276851955 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc02c025030 con 0x7fc0380af730 2022-06-29T16:05:13.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.738+0000 7fc0357fa700 1 --2- 172.21.15.60:0/276851955 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc018053b00 0x7fc018055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:13.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.738+0000 7fc0357fa700 1 -- 172.21.15.60:0/276851955 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fc02c06c810 con 0x7fc0380af730 2022-06-29T16:05:13.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.738+0000 7fc037fff700 1 --2- 172.21.15.60:0/276851955 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc018053b00 0x7fc018055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:13.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.742+0000 7fc037fff700 1 --2- 172.21.15.60:0/276851955 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc018053b00 0x7fc018055fc0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7fc02800b000 tx=0x7fc028006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:13.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.742+0000 7fc0357fa700 1 -- 172.21.15.60:0/276851955 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc02c043a60 con 0x7fc0380af730 2022-06-29T16:05:13.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.866+0000 7fc03d5ab700 1 -- 172.21.15.60:0/276851955 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fc03805d990 con 0x7fc0380af730 2022-06-29T16:05:13.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.870+0000 7fc0357fa700 1 -- 172.21.15.60:0/276851955 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v76) v1 ==== 110+0+10 (secure 0 0 0) 0x7fc02c023070 con 0x7fc0380af730 2022-06-29T16:05:13.877 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:05:13.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.870+0000 7fc03d5ab700 1 -- 172.21.15.60:0/276851955 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc018053b00 msgr2=0x7fc018055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:13.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.870+0000 7fc03d5ab700 1 --2- 172.21.15.60:0/276851955 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc018053b00 0x7fc018055fc0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7fc02800b000 tx=0x7fc028006df0).stop 2022-06-29T16:05:13.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.870+0000 7fc03d5ab700 1 -- 172.21.15.60:0/276851955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0380af730 msgr2=0x7fc03809e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:13.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.870+0000 7fc03d5ab700 1 --2- 172.21.15.60:0/276851955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0380af730 0x7fc03809e7f0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 rx=0x7fc02c002f70 tx=0x7fc02c014000).stop 2022-06-29T16:05:13.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.870+0000 7fc03d5ab700 1 -- 172.21.15.60:0/276851955 shutdown_connections 2022-06-29T16:05:13.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.870+0000 7fc03d5ab700 1 --2- 172.21.15.60:0/276851955 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc018053b00 0x7fc018055fc0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:13.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.870+0000 7fc03d5ab700 1 --2- 172.21.15.60:0/276851955 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0380b2800 0x7fc038154f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:13.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.870+0000 7fc03d5ab700 1 --2- 172.21.15.60:0/276851955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0380af730 0x7fc03809e7f0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:13.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.870+0000 7fc03d5ab700 1 --2- 172.21.15.60:0/276851955 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc03805bf60 0x7fc03809e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:13.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.870+0000 7fc03d5ab700 1 -- 172.21.15.60:0/276851955 >> 172.21.15.60:0/276851955 conn(0x7fc0380b6120 msgr2=0x7fc0380b4510 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:13.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.870+0000 7fc03d5ab700 1 -- 172.21.15.60:0/276851955 shutdown_connections 2022-06-29T16:05:13.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.870+0000 7fc03d5ab700 1 -- 172.21.15.60:0/276851955 wait complete. 2022-06-29T16:05:13.890 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:05:13.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.982+0000 7fb854a01700 1 Processor -- start 2022-06-29T16:05:13.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.982+0000 7fb854a01700 1 -- start start 2022-06-29T16:05:13.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.986+0000 7fb854a01700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb8500b13a0 0x7fb85013f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:13.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.986+0000 7fb854a01700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb8501390c0 0x7fb850065660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:13.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.986+0000 7fb854a01700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85013fe40 con 0x7fb8500642d0 2022-06-29T16:05:13.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.986+0000 7fb854a01700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8500b1780 con 0x7fb8501390c0 2022-06-29T16:05:13.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.986+0000 7fb854a01700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb850064c80 con 0x7fb8500b13a0 2022-06-29T16:05:13.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.986+0000 7fb84ed9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb8501390c0 0x7fb850065660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:13.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.986+0000 7fb84dd9b700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb8500642d0 0x7fb8501389b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58032/0 (socket says 172.21.15.60:58032) 2022-06-29T16:05:13.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.986+0000 7fb84dd9b700 1 -- 172.21.15.60:0/1548381301 learned_addr learned my addr 172.21.15.60:0/1548381301 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:13.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.986+0000 7fb84e59c700 1 --2- 172.21.15.60:0/1548381301 >> v2:172.21.15.92:3300/0 conn(0x7fb8500b13a0 0x7fb85013f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:13.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.986+0000 7fb84ed9d700 1 -- 172.21.15.60:0/1548381301 >> v2:172.21.15.92:3300/0 conn(0x7fb8500b13a0 msgr2=0x7fb85013f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:13.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.986+0000 7fb84ed9d700 1 --2- 172.21.15.60:0/1548381301 >> v2:172.21.15.92:3300/0 conn(0x7fb8500b13a0 0x7fb85013f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:13.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.986+0000 7fb84ed9d700 1 -- 172.21.15.60:0/1548381301 >> v1:172.21.15.60:6789/0 conn(0x7fb8500642d0 legacy=0x7fb8501389b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:13.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.986+0000 7fb84ed9d700 1 -- 172.21.15.60:0/1548381301 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb850064f20 con 0x7fb8501390c0 2022-06-29T16:05:13.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.986+0000 7fb84d59a700 1 -- 172.21.15.60:0/1548381301 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 759921997 0 0) 0x7fb85013fe40 con 0x7fb8500642d0 2022-06-29T16:05:13.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.986+0000 7fb84ed9d700 1 --2- 172.21.15.60:0/1548381301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8501390c0 0x7fb850065660 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 rx=0x7fb8480032f0 tx=0x7fb8480091d0).ready entity=mon.0 client_cookie=b6521db4b8f2fd87 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:13.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.986+0000 7fb84d59a700 1 -- 172.21.15.60:0/1548381301 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb848009ea0 con 0x7fb8501390c0 2022-06-29T16:05:13.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.986+0000 7fb84d59a700 1 -- 172.21.15.60:0/1548381301 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb84800b870 con 0x7fb8501390c0 2022-06-29T16:05:13.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.986+0000 7fb84d59a700 1 -- 172.21.15.60:0/1548381301 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb84800b9e0 con 0x7fb8501390c0 2022-06-29T16:05:14.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.986+0000 7fb854a01700 1 -- 172.21.15.60:0/1548381301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8501390c0 msgr2=0x7fb850065660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:14.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.986+0000 7fb854a01700 1 --2- 172.21.15.60:0/1548381301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8501390c0 0x7fb850065660 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 rx=0x7fb8480032f0 tx=0x7fb8480091d0).stop 2022-06-29T16:05:14.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.986+0000 7fb854a01700 1 -- 172.21.15.60:0/1548381301 shutdown_connections 2022-06-29T16:05:14.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.986+0000 7fb854a01700 1 --2- 172.21.15.60:0/1548381301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8501390c0 0x7fb850065660 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:14.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.986+0000 7fb854a01700 1 --2- 172.21.15.60:0/1548381301 >> v2:172.21.15.92:3300/0 conn(0x7fb8500b13a0 0x7fb85013f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:14.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.986+0000 7fb854a01700 1 -- 172.21.15.60:0/1548381301 >> 172.21.15.60:0/1548381301 conn(0x7fb8500632c0 msgr2=0x7fb8500b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:14.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.986+0000 7fb854a01700 1 -- 172.21.15.60:0/1548381301 shutdown_connections 2022-06-29T16:05:14.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.986+0000 7fb854a01700 1 -- 172.21.15.60:0/1548381301 wait complete. 2022-06-29T16:05:14.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.986+0000 7fb854a01700 1 Processor -- start 2022-06-29T16:05:14.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.990+0000 7fb854a01700 1 -- start start 2022-06-29T16:05:14.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.990+0000 7fb854a01700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8500642d0 0x7fb850147d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:14.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.990+0000 7fb854a01700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8500b13a0 0x7fb850142840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:14.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.990+0000 7fb854a01700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8501390c0 0x7fb850142d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:14.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.990+0000 7fb854a01700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb850153140 con 0x7fb8500642d0 2022-06-29T16:05:14.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.990+0000 7fb854a01700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8500ad090 con 0x7fb8500b13a0 2022-06-29T16:05:14.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.990+0000 7fb854a01700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb850155ea0 con 0x7fb8501390c0 2022-06-29T16:05:14.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.990+0000 7fb84e59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8500642d0 0x7fb850147d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:14.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.990+0000 7fb84e59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8500642d0 0x7fb850147d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49952/0 (socket says 172.21.15.60:49952) 2022-06-29T16:05:14.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.990+0000 7fb84e59c700 1 -- 172.21.15.60:0/2672318378 learned_addr learned my addr 172.21.15.60:0/2672318378 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:14.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.990+0000 7fb84ed9d700 1 --2- 172.21.15.60:0/2672318378 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8501390c0 0x7fb850142d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:14.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.990+0000 7fb84dd9b700 1 --2- 172.21.15.60:0/2672318378 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8500b13a0 0x7fb850142840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:14.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.990+0000 7fb84e59c700 1 -- 172.21.15.60:0/2672318378 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8501390c0 msgr2=0x7fb850142d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:14.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.990+0000 7fb84e59c700 1 --2- 172.21.15.60:0/2672318378 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8501390c0 0x7fb850142d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:14.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.990+0000 7fb84e59c700 1 -- 172.21.15.60:0/2672318378 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8500b13a0 msgr2=0x7fb850142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:14.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.990+0000 7fb84e59c700 1 --2- 172.21.15.60:0/2672318378 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8500b13a0 0x7fb850142840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:14.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.990+0000 7fb84e59c700 1 -- 172.21.15.60:0/2672318378 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb848009470 con 0x7fb8500642d0 2022-06-29T16:05:14.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.990+0000 7fb84e59c700 1 --2- 172.21.15.60:0/2672318378 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8500642d0 0x7fb850147d20 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 rx=0x7fb8380030b0 tx=0x7fb838008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:14.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.990+0000 7fb8477fe700 1 -- 172.21.15.60:0/2672318378 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb838011780 con 0x7fb8500642d0 2022-06-29T16:05:14.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.990+0000 7fb8477fe700 1 -- 172.21.15.60:0/2672318378 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8380118e0 con 0x7fb8500642d0 2022-06-29T16:05:14.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.990+0000 7fb8477fe700 1 -- 172.21.15.60:0/2672318378 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb838011c40 con 0x7fb8500642d0 2022-06-29T16:05:14.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.990+0000 7fb854a01700 1 -- 172.21.15.60:0/2672318378 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb850156370 con 0x7fb8500642d0 2022-06-29T16:05:14.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.990+0000 7fb854a01700 1 -- 172.21.15.60:0/2672318378 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb85013df50 con 0x7fb8500642d0 2022-06-29T16:05:14.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.990+0000 7fb854a01700 1 -- 172.21.15.60:0/2672318378 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8501433c0 con 0x7fb8500642d0 2022-06-29T16:05:14.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.994+0000 7fb8477fe700 1 -- 172.21.15.60:0/2672318378 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb838012420 con 0x7fb8500642d0 2022-06-29T16:05:14.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.994+0000 7fb8477fe700 1 --2- 172.21.15.60:0/2672318378 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb834053a70 0x7fb834055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:14.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.994+0000 7fb8477fe700 1 -- 172.21.15.60:0/2672318378 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fb83806bbc0 con 0x7fb8500642d0 2022-06-29T16:05:14.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.994+0000 7fb84dd9b700 1 --2- 172.21.15.60:0/2672318378 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb834053a70 0x7fb834055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:14.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.994+0000 7fb84dd9b700 1 --2- 172.21.15.60:0/2672318378 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb834053a70 0x7fb834055f30 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fb84000b000 tx=0x7fb840009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:14.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:13.998+0000 7fb8477fe700 1 -- 172.21.15.60:0/2672318378 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb838042ca0 con 0x7fb8500642d0 2022-06-29T16:05:14.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.126+0000 7fb854a01700 1 -- 172.21.15.60:0/2672318378 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fb8500651c0 con 0x7fb8500642d0 2022-06-29T16:05:14.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.130+0000 7fb8477fe700 1 -- 172.21.15.60:0/2672318378 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v76) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb8380462c0 con 0x7fb8500642d0 2022-06-29T16:05:14.137 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:05:14.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.130+0000 7fb854a01700 1 -- 172.21.15.60:0/2672318378 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb834053a70 msgr2=0x7fb834055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:14.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.130+0000 7fb854a01700 1 --2- 172.21.15.60:0/2672318378 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb834053a70 0x7fb834055f30 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fb84000b000 tx=0x7fb840009040).stop 2022-06-29T16:05:14.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.130+0000 7fb854a01700 1 -- 172.21.15.60:0/2672318378 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8500642d0 msgr2=0x7fb850147d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:14.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.130+0000 7fb854a01700 1 --2- 172.21.15.60:0/2672318378 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8500642d0 0x7fb850147d20 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 rx=0x7fb8380030b0 tx=0x7fb838008040).stop 2022-06-29T16:05:14.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.130+0000 7fb854a01700 1 -- 172.21.15.60:0/2672318378 shutdown_connections 2022-06-29T16:05:14.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.130+0000 7fb854a01700 1 --2- 172.21.15.60:0/2672318378 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb834053a70 0x7fb834055f30 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:14.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.130+0000 7fb854a01700 1 --2- 172.21.15.60:0/2672318378 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb8501390c0 0x7fb850142d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:14.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.130+0000 7fb854a01700 1 --2- 172.21.15.60:0/2672318378 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8500b13a0 0x7fb850142840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:14.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.130+0000 7fb854a01700 1 --2- 172.21.15.60:0/2672318378 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb8500642d0 0x7fb850147d20 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:14.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.130+0000 7fb854a01700 1 -- 172.21.15.60:0/2672318378 >> 172.21.15.60:0/2672318378 conn(0x7fb8500632c0 msgr2=0x7fb8500b2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:14.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.130+0000 7fb854a01700 1 -- 172.21.15.60:0/2672318378 shutdown_connections 2022-06-29T16:05:14.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.130+0000 7fb854a01700 1 -- 172.21.15.60:0/2672318378 wait complete. 2022-06-29T16:05:14.150 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-06-29T16:05:14.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.246+0000 7f70fd5a1700 1 Processor -- start 2022-06-29T16:05:14.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.246+0000 7f70fd5a1700 1 -- start start 2022-06-29T16:05:14.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.246+0000 7f70fd5a1700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f70f805d070 0x7f70f814aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:14.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.246+0000 7f70fd5a1700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70f814af80 0x7f70f80ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:14.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.246+0000 7f70fd5a1700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70f805d450 con 0x7f70f80b2130 2022-06-29T16:05:14.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.246+0000 7f70fd5a1700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70f80adf90 con 0x7f70f805d070 2022-06-29T16:05:14.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.246+0000 7f70fd5a1700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70f80b25d0 con 0x7f70f814af80 2022-06-29T16:05:14.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.246+0000 7f70f7fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f70f805d070 0x7f70f814aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:14.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.246+0000 7f70ef7fe700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f70f80b2130 0x7f70f80aca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:34064/0 (socket says 172.21.15.60:34064) 2022-06-29T16:05:14.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.246+0000 7f70ef7fe700 1 -- 172.21.15.60:0/2756401696 learned_addr learned my addr 172.21.15.60:0/2756401696 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:14.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.246+0000 7f70fcda0700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70f814af80 0x7f70f80ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:14.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.246+0000 7f70f7fff700 1 -- 172.21.15.60:0/2756401696 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70f814af80 msgr2=0x7f70f80ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:14.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.246+0000 7f70f7fff700 1 --2- 172.21.15.60:0/2756401696 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70f814af80 0x7f70f80ad130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:14.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.246+0000 7f70f7fff700 1 -- 172.21.15.60:0/2756401696 >> v1:172.21.15.92:6789/0 conn(0x7f70f80b2130 legacy=0x7f70f80aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:14.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.246+0000 7f70f7fff700 1 -- 172.21.15.60:0/2756401696 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f814a350 con 0x7f70f805d070 2022-06-29T16:05:14.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.246+0000 7f70f77fe700 1 -- 172.21.15.60:0/2756401696 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1668519351 0 0) 0x7f70f805d450 con 0x7f70f80b2130 2022-06-29T16:05:14.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.246+0000 7f70f7fff700 1 --2- 172.21.15.60:0/2756401696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70f805d070 0x7f70f814aa90 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f70e8002f70 tx=0x7f70e8006df0).ready entity=mon.1 client_cookie=6d6dbb7248a53d71 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:14.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.246+0000 7f70f77fe700 1 -- 172.21.15.60:0/2756401696 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70e800d570 con 0x7f70f805d070 2022-06-29T16:05:14.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.246+0000 7f70f77fe700 1 -- 172.21.15.60:0/2756401696 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e8000e90 con 0x7f70f805d070 2022-06-29T16:05:14.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.246+0000 7f70f77fe700 1 -- 172.21.15.60:0/2756401696 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70e800d8c0 con 0x7f70f805d070 2022-06-29T16:05:14.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.246+0000 7f70fd5a1700 1 -- 172.21.15.60:0/2756401696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70f805d070 msgr2=0x7f70f814aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:14.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.246+0000 7f70fd5a1700 1 --2- 172.21.15.60:0/2756401696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70f805d070 0x7f70f814aa90 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f70e8002f70 tx=0x7f70e8006df0).stop 2022-06-29T16:05:14.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.246+0000 7f70fd5a1700 1 -- 172.21.15.60:0/2756401696 shutdown_connections 2022-06-29T16:05:14.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.246+0000 7f70fd5a1700 1 --2- 172.21.15.60:0/2756401696 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70f814af80 0x7f70f80ad130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:14.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.246+0000 7f70fd5a1700 1 --2- 172.21.15.60:0/2756401696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70f805d070 0x7f70f814aa90 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:14.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.246+0000 7f70fd5a1700 1 -- 172.21.15.60:0/2756401696 >> 172.21.15.60:0/2756401696 conn(0x7f70f80b5890 msgr2=0x7f70f80b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:14.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.246+0000 7f70fd5a1700 1 -- 172.21.15.60:0/2756401696 shutdown_connections 2022-06-29T16:05:14.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.246+0000 7f70fd5a1700 1 -- 172.21.15.60:0/2756401696 wait complete. 2022-06-29T16:05:14.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.250+0000 7f70fd5a1700 1 Processor -- start 2022-06-29T16:05:14.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.250+0000 7f70fd5a1700 1 -- start start 2022-06-29T16:05:14.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.250+0000 7f70fd5a1700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70f805d070 0x7f70f80b3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:14.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.250+0000 7f70fd5a1700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70f80b2130 0x7f70f8159720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:14.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.250+0000 7f70fd5a1700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70f814af80 0x7f70f8064510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:14.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.250+0000 7f70fd5a1700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f80636f0 con 0x7f70f805d070 2022-06-29T16:05:14.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.250+0000 7f70fd5a1700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f8063f70 con 0x7f70f814af80 2022-06-29T16:05:14.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.250+0000 7f70fd5a1700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70f8149b50 con 0x7f70f80b2130 2022-06-29T16:05:14.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.250+0000 7f70f7fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70f805d070 0x7f70f80b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:14.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.250+0000 7f70f7fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70f805d070 0x7f70f80b3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49966/0 (socket says 172.21.15.60:49966) 2022-06-29T16:05:14.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.250+0000 7f70f7fff700 1 -- 172.21.15.60:0/153296191 learned_addr learned my addr 172.21.15.60:0/153296191 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:14.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.250+0000 7f70ef7fe700 1 --2- 172.21.15.60:0/153296191 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70f80b2130 0x7f70f8159720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:14.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.250+0000 7f70fcda0700 1 --2- 172.21.15.60:0/153296191 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70f814af80 0x7f70f8064510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:14.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.250+0000 7f70f7fff700 1 -- 172.21.15.60:0/153296191 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70f80b2130 msgr2=0x7f70f8159720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:14.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.250+0000 7f70f7fff700 1 --2- 172.21.15.60:0/153296191 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70f80b2130 0x7f70f8159720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:14.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.250+0000 7f70f7fff700 1 -- 172.21.15.60:0/153296191 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70f814af80 msgr2=0x7f70f8064510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:14.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.250+0000 7f70f7fff700 1 --2- 172.21.15.60:0/153296191 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70f814af80 0x7f70f8064510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:14.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.250+0000 7f70f7fff700 1 -- 172.21.15.60:0/153296191 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e800b180 con 0x7f70f805d070 2022-06-29T16:05:14.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.250+0000 7f70fcda0700 1 --2- 172.21.15.60:0/153296191 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70f814af80 0x7f70f8064510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:14.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.250+0000 7f70f7fff700 1 --2- 172.21.15.60:0/153296191 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70f805d070 0x7f70f80b3470 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 rx=0x7f70f8063830 tx=0x7f70e8014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:14.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.250+0000 7f70f5ffb700 1 -- 172.21.15.60:0/153296191 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70e800dea0 con 0x7f70f805d070 2022-06-29T16:05:14.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.250+0000 7f70f5ffb700 1 -- 172.21.15.60:0/153296191 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e8013e50 con 0x7f70f805d070 2022-06-29T16:05:14.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.250+0000 7f70f5ffb700 1 -- 172.21.15.60:0/153296191 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70e801e5f0 con 0x7f70f805d070 2022-06-29T16:05:14.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.250+0000 7f70fd5a1700 1 -- 172.21.15.60:0/153296191 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70f8149df0 con 0x7f70f805d070 2022-06-29T16:05:14.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.250+0000 7f70fd5a1700 1 -- 172.21.15.60:0/153296191 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70f8066d30 con 0x7f70f805d070 2022-06-29T16:05:14.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.254+0000 7f70fd5a1700 1 -- 172.21.15.60:0/153296191 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70f814a5f0 con 0x7f70f805d070 2022-06-29T16:05:14.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.254+0000 7f70f5ffb700 1 -- 172.21.15.60:0/153296191 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f70e8025030 con 0x7f70f805d070 2022-06-29T16:05:14.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.254+0000 7f70f5ffb700 1 --2- 172.21.15.60:0/153296191 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f70dc053a70 0x7f70dc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:14.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.254+0000 7f70f5ffb700 1 -- 172.21.15.60:0/153296191 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f70e806c9d0 con 0x7f70f805d070 2022-06-29T16:05:14.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.254+0000 7f70ef7fe700 1 --2- 172.21.15.60:0/153296191 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f70dc053a70 0x7f70dc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:14.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.254+0000 7f70ef7fe700 1 --2- 172.21.15.60:0/153296191 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f70dc053a70 0x7f70dc055f30 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f70e000d000 tx=0x7f70e0007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:14.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.258+0000 7f70f5ffb700 1 -- 172.21.15.60:0/153296191 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f70e8043bc0 con 0x7f70f805d070 2022-06-29T16:05:14.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.386+0000 7f70fd5a1700 1 -- 172.21.15.60:0/153296191 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f70f80642b0 con 0x7f70f805d070 2022-06-29T16:05:14.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.386+0000 7f70f5ffb700 1 -- 172.21.15.60:0/153296191 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v76) v1 ==== 84+0+143 (secure 0 0 0) 0x7f70e8023070 con 0x7f70f805d070 2022-06-29T16:05:14.394 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:14.394 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2022-06-29T16:05:14.395 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:14.395 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2022-06-29T16:05:14.395 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": "device_health_metrics" 2022-06-29T16:05:14.396 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:14.396 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:14.396 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 2, 2022-06-29T16:05:14.396 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": "rbd" 2022-06-29T16:05:14.397 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:14.397 INFO:teuthology.orchestra.run.smithi060.stdout:] 2022-06-29T16:05:14.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.386+0000 7f70fd5a1700 1 -- 172.21.15.60:0/153296191 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f70dc053a70 msgr2=0x7f70dc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:14.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.386+0000 7f70fd5a1700 1 --2- 172.21.15.60:0/153296191 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f70dc053a70 0x7f70dc055f30 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f70e000d000 tx=0x7f70e0007a00).stop 2022-06-29T16:05:14.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.386+0000 7f70fd5a1700 1 -- 172.21.15.60:0/153296191 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70f805d070 msgr2=0x7f70f80b3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:14.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.386+0000 7f70fd5a1700 1 --2- 172.21.15.60:0/153296191 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70f805d070 0x7f70f80b3470 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 rx=0x7f70f8063830 tx=0x7f70e8014000).stop 2022-06-29T16:05:14.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.386+0000 7f70fd5a1700 1 -- 172.21.15.60:0/153296191 shutdown_connections 2022-06-29T16:05:14.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.386+0000 7f70fd5a1700 1 --2- 172.21.15.60:0/153296191 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f70dc053a70 0x7f70dc055f30 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:14.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.390+0000 7f70fd5a1700 1 --2- 172.21.15.60:0/153296191 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70f814af80 0x7f70f8064510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:14.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.390+0000 7f70fd5a1700 1 --2- 172.21.15.60:0/153296191 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70f80b2130 0x7f70f8159720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:14.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.390+0000 7f70fd5a1700 1 --2- 172.21.15.60:0/153296191 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70f805d070 0x7f70f80b3470 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:14.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.390+0000 7f70fd5a1700 1 -- 172.21.15.60:0/153296191 >> 172.21.15.60:0/153296191 conn(0x7f70f80b5890 msgr2=0x7f70f80b4390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:14.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.390+0000 7f70fd5a1700 1 -- 172.21.15.60:0/153296191 shutdown_connections 2022-06-29T16:05:14.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.390+0000 7f70fd5a1700 1 -- 172.21.15.60:0/153296191 wait complete. 2022-06-29T16:05:14.407 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-06-29T16:05:14.408 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-06-29T16:05:14.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.510+0000 7f4efb066700 1 Processor -- start 2022-06-29T16:05:14.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.510+0000 7f4efb066700 1 -- start start 2022-06-29T16:05:14.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.514+0000 7f4efb066700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4ef41426b0 0x7f4ef40b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:14.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.514+0000 7f4efb066700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef4143a40 con 0x7f4ef40b4010 2022-06-29T16:05:14.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.514+0000 7f4efb066700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef40aee60 con 0x7f4ef40ae7c0 2022-06-29T16:05:14.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.514+0000 7f4efb066700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ef40af100 con 0x7f4ef41426b0 2022-06-29T16:05:14.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.514+0000 7f4ef9603700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4ef41426b0 0x7f4ef40b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:14.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.514+0000 7f4ef3fff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4ef40b4010 0x7f4ef40b46c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58062/0 (socket says 172.21.15.60:58062) 2022-06-29T16:05:14.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.514+0000 7f4ef3fff700 1 -- 172.21.15.60:0/391967664 learned_addr learned my addr 172.21.15.60:0/391967664 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:14.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.514+0000 7f4ef9603700 1 -- 172.21.15.60:0/391967664 >> v1:172.21.15.92:6789/0 conn(0x7f4ef40ae7c0 legacy=0x7f4ef4141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:14.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.514+0000 7f4ef9603700 1 -- 172.21.15.60:0/391967664 >> v1:172.21.15.60:6789/0 conn(0x7f4ef40b4010 legacy=0x7f4ef40b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:14.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.514+0000 7f4ef37fe700 1 -- 172.21.15.60:0/391967664 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3899347997 0 0) 0x7f4ef4143a40 con 0x7f4ef40b4010 2022-06-29T16:05:14.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.514+0000 7f4ef9603700 1 -- 172.21.15.60:0/391967664 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef40aeba0 con 0x7f4ef41426b0 2022-06-29T16:05:14.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.514+0000 7f4ef9603700 1 --2- 172.21.15.60:0/391967664 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ef41426b0 0x7f4ef40b4dd0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 rx=0x7f4eec0032f0 tx=0x7f4eec0091d0).ready entity=mon.0 client_cookie=5084305eeaf7f1a6 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:14.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.514+0000 7f4ef37fe700 1 -- 172.21.15.60:0/391967664 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4eec009ea0 con 0x7f4ef41426b0 2022-06-29T16:05:14.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.514+0000 7f4ef37fe700 1 -- 172.21.15.60:0/391967664 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eec00b8a0 con 0x7f4ef41426b0 2022-06-29T16:05:14.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.514+0000 7f4ef37fe700 1 -- 172.21.15.60:0/391967664 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4eec00ba10 con 0x7f4ef41426b0 2022-06-29T16:05:14.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.514+0000 7f4efb066700 1 -- 172.21.15.60:0/391967664 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ef41426b0 msgr2=0x7f4ef40b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:14.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.514+0000 7f4efb066700 1 --2- 172.21.15.60:0/391967664 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ef41426b0 0x7f4ef40b4dd0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 rx=0x7f4eec0032f0 tx=0x7f4eec0091d0).stop 2022-06-29T16:05:14.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.514+0000 7f4efb066700 1 -- 172.21.15.60:0/391967664 shutdown_connections 2022-06-29T16:05:14.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.514+0000 7f4efb066700 1 --2- 172.21.15.60:0/391967664 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ef41426b0 0x7f4ef40b4dd0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:14.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.514+0000 7f4efb066700 1 -- 172.21.15.60:0/391967664 >> 172.21.15.60:0/391967664 conn(0x7f4ef40b9340 msgr2=0x7f4ef40b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:14.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.514+0000 7f4efb066700 1 -- 172.21.15.60:0/391967664 shutdown_connections 2022-06-29T16:05:14.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.514+0000 7f4efb066700 1 -- 172.21.15.60:0/391967664 wait complete. 2022-06-29T16:05:14.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.518+0000 7f4efb066700 1 Processor -- start 2022-06-29T16:05:14.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.518+0000 7f4efb066700 1 -- start start 2022-06-29T16:05:14.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.518+0000 7f4efb066700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ef40ae7c0 0x7f4ef40b3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:14.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.518+0000 7f4efb066700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ef40b4010 0x7f4ef4144330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:14.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.518+0000 7f4efb066700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4ef41426b0 0x7f4ef413d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:14.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.518+0000 7f4efb066700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef41452b0 con 0x7f4ef40ae7c0 2022-06-29T16:05:14.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.518+0000 7f4efb066700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef41460f0 con 0x7f4ef40b4010 2022-06-29T16:05:14.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.518+0000 7f4efb066700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef4146390 con 0x7f4ef41426b0 2022-06-29T16:05:14.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.518+0000 7f4ef8e02700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ef40ae7c0 0x7f4ef40b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:14.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.518+0000 7f4ef8e02700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ef40ae7c0 0x7f4ef40b3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49980/0 (socket says 172.21.15.60:49980) 2022-06-29T16:05:14.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.518+0000 7f4ef8e02700 1 -- 172.21.15.60:0/577088817 learned_addr learned my addr 172.21.15.60:0/577088817 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:14.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.518+0000 7f4ef9603700 1 --2- 172.21.15.60:0/577088817 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4ef41426b0 0x7f4ef413d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:14.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.518+0000 7f4ef3fff700 1 --2- 172.21.15.60:0/577088817 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ef40b4010 0x7f4ef4144330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:14.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.522+0000 7f4ef8e02700 1 -- 172.21.15.60:0/577088817 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4ef41426b0 msgr2=0x7f4ef413d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:14.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.522+0000 7f4ef8e02700 1 --2- 172.21.15.60:0/577088817 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4ef41426b0 0x7f4ef413d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:14.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.522+0000 7f4ef8e02700 1 -- 172.21.15.60:0/577088817 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ef40b4010 msgr2=0x7f4ef4144330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:14.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.522+0000 7f4ef8e02700 1 --2- 172.21.15.60:0/577088817 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ef40b4010 0x7f4ef4144330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:14.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.522+0000 7f4ef8e02700 1 -- 172.21.15.60:0/577088817 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eec009470 con 0x7f4ef40ae7c0 2022-06-29T16:05:14.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.522+0000 7f4ef8e02700 1 --2- 172.21.15.60:0/577088817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ef40ae7c0 0x7f4ef40b3080 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 rx=0x7f4ef4060a90 tx=0x7f4ee8007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:14.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.522+0000 7f4ef1ffb700 1 -- 172.21.15.60:0/577088817 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ee800ce30 con 0x7f4ef40ae7c0 2022-06-29T16:05:14.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.522+0000 7f4ef1ffb700 1 -- 172.21.15.60:0/577088817 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ee800f470 con 0x7f4ef40ae7c0 2022-06-29T16:05:14.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.522+0000 7f4efb066700 1 -- 172.21.15.60:0/577088817 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef4146630 con 0x7f4ef40ae7c0 2022-06-29T16:05:14.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.522+0000 7f4efb066700 1 -- 172.21.15.60:0/577088817 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef4145d40 con 0x7f4ef40ae7c0 2022-06-29T16:05:14.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.522+0000 7f4ef1ffb700 1 -- 172.21.15.60:0/577088817 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ee800f7d0 con 0x7f4ef40ae7c0 2022-06-29T16:05:14.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.522+0000 7f4efb066700 1 -- 172.21.15.60:0/577088817 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ef41467c0 con 0x7f4ef40ae7c0 2022-06-29T16:05:14.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.522+0000 7f4ef1ffb700 1 -- 172.21.15.60:0/577088817 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4ee8019030 con 0x7f4ef40ae7c0 2022-06-29T16:05:14.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.526+0000 7f4ef1ffb700 1 --2- 172.21.15.60:0/577088817 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4ed4053a70 0x7f4ed4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:14.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.526+0000 7f4ef1ffb700 1 -- 172.21.15.60:0/577088817 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f4ee806b130 con 0x7f4ef40ae7c0 2022-06-29T16:05:14.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.526+0000 7f4ef3fff700 1 --2- 172.21.15.60:0/577088817 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4ed4053a70 0x7f4ed4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:14.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.526+0000 7f4ef3fff700 1 --2- 172.21.15.60:0/577088817 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4ed4053a70 0x7f4ed4055f30 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f4ee400b000 tx=0x7f4ee4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:14.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.534+0000 7f4ef1ffb700 1 -- 172.21.15.60:0/577088817 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4ee80422e0 con 0x7f4ef40ae7c0 2022-06-29T16:05:14.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.658+0000 7f4efb066700 1 -- 172.21.15.60:0/577088817 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f4ef40573d0 con 0x7f4ef40ae7c0 2022-06-29T16:05:14.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.974+0000 7f4ef1ffb700 1 -- 172.21.15.60:0/577088817 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v77) v1 ==== 133+0+0 (secure 0 0 0) 0x7f4ee8045900 con 0x7f4ef40ae7c0 2022-06-29T16:05:14.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.978+0000 7f4efb066700 1 -- 172.21.15.60:0/577088817 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4ed4053a70 msgr2=0x7f4ed4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:14.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.978+0000 7f4efb066700 1 --2- 172.21.15.60:0/577088817 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4ed4053a70 0x7f4ed4055f30 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f4ee400b000 tx=0x7f4ee4009040).stop 2022-06-29T16:05:14.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.978+0000 7f4efb066700 1 -- 172.21.15.60:0/577088817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ef40ae7c0 msgr2=0x7f4ef40b3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:14.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.978+0000 7f4efb066700 1 --2- 172.21.15.60:0/577088817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ef40ae7c0 0x7f4ef40b3080 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 rx=0x7f4ef4060a90 tx=0x7f4ee8007c50).stop 2022-06-29T16:05:14.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.982+0000 7f4efb066700 1 -- 172.21.15.60:0/577088817 shutdown_connections 2022-06-29T16:05:14.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.982+0000 7f4efb066700 1 --2- 172.21.15.60:0/577088817 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4ed4053a70 0x7f4ed4055f30 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:14.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.982+0000 7f4efb066700 1 --2- 172.21.15.60:0/577088817 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4ef41426b0 0x7f4ef413d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:14.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.982+0000 7f4efb066700 1 --2- 172.21.15.60:0/577088817 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ef40b4010 0x7f4ef4144330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:14.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.982+0000 7f4efb066700 1 --2- 172.21.15.60:0/577088817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ef40ae7c0 0x7f4ef40b3080 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:14.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.982+0000 7f4efb066700 1 -- 172.21.15.60:0/577088817 >> 172.21.15.60:0/577088817 conn(0x7f4ef40b9340 msgr2=0x7f4ef40b4da0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:14.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.982+0000 7f4efb066700 1 -- 172.21.15.60:0/577088817 shutdown_connections 2022-06-29T16:05:14.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:14.982+0000 7f4efb066700 1 -- 172.21.15.60:0/577088817 wait complete. 2022-06-29T16:05:14.992 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2022-06-29T16:05:15.002 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-06-29T16:05:15.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.102+0000 7febbe5c9700 1 Processor -- start 2022-06-29T16:05:15.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.102+0000 7febbe5c9700 1 -- start start 2022-06-29T16:05:15.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.106+0000 7febbe5c9700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7febb80b5eb0 0x7febb8142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:15.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.106+0000 7febbe5c9700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7febb80b48b0 0x7febb80b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:15.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.106+0000 7febbe5c9700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb80b4d10 con 0x7febb80b2c50 2022-06-29T16:05:15.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.106+0000 7febbe5c9700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febb80b54e0 con 0x7febb80b48b0 2022-06-29T16:05:15.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.106+0000 7febbe5c9700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febb80b6290 con 0x7febb80b5eb0 2022-06-29T16:05:15.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.106+0000 7febbcdc6700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7febb80b48b0 0x7febb80b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:15.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.106+0000 7febbcdc6700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7febb80b48b0 0x7febb80b05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49988/0 (socket says 172.21.15.60:49988) 2022-06-29T16:05:15.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.106+0000 7febbcdc6700 1 -- 172.21.15.60:0/1398618573 learned_addr learned my addr 172.21.15.60:0/1398618573 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:15.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.106+0000 7febbd5c7700 1 --2- 172.21.15.60:0/1398618573 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7febb80b5eb0 0x7febb8142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:15.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.106+0000 7febbcdc6700 1 -- 172.21.15.60:0/1398618573 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7febb80b5eb0 msgr2=0x7febb8142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:15.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.106+0000 7febbcdc6700 1 --2- 172.21.15.60:0/1398618573 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7febb80b5eb0 0x7febb8142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:15.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.106+0000 7febbcdc6700 1 -- 172.21.15.60:0/1398618573 >> v1:172.21.15.92:6789/0 conn(0x7febb80b2c50 legacy=0x7febb80b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:15.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.106+0000 7febbcdc6700 1 -- 172.21.15.60:0/1398618573 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb8142f10 con 0x7febb80b48b0 2022-06-29T16:05:15.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.106+0000 7febbcdc6700 1 --2- 172.21.15.60:0/1398618573 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febb80b48b0 0x7febb80b05e0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 rx=0x7febb4002f70 tx=0x7febb4006eb0).ready entity=mon.0 client_cookie=77b3622e6519eff8 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:15.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.106+0000 7febaffff700 1 -- 172.21.15.60:0/1398618573 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febb400d5c0 con 0x7febb80b48b0 2022-06-29T16:05:15.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.106+0000 7febaffff700 1 -- 172.21.15.60:0/1398618573 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febb4000e90 con 0x7febb80b48b0 2022-06-29T16:05:15.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.106+0000 7febaffff700 1 -- 172.21.15.60:0/1398618573 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febb400d950 con 0x7febb80b48b0 2022-06-29T16:05:15.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.106+0000 7febbe5c9700 1 -- 172.21.15.60:0/1398618573 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febb80b48b0 msgr2=0x7febb80b05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:15.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.106+0000 7febbe5c9700 1 --2- 172.21.15.60:0/1398618573 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febb80b48b0 0x7febb80b05e0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 rx=0x7febb4002f70 tx=0x7febb4006eb0).stop 2022-06-29T16:05:15.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.106+0000 7febbe5c9700 1 -- 172.21.15.60:0/1398618573 shutdown_connections 2022-06-29T16:05:15.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.106+0000 7febbe5c9700 1 --2- 172.21.15.60:0/1398618573 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febb80b48b0 0x7febb80b05e0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:15.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.106+0000 7febbe5c9700 1 --2- 172.21.15.60:0/1398618573 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7febb80b5eb0 0x7febb8142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:15.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.106+0000 7febbe5c9700 1 -- 172.21.15.60:0/1398618573 >> 172.21.15.60:0/1398618573 conn(0x7febb8062090 msgr2=0x7febb8062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:15.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.106+0000 7febbe5c9700 1 -- 172.21.15.60:0/1398618573 shutdown_connections 2022-06-29T16:05:15.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.106+0000 7febbe5c9700 1 -- 172.21.15.60:0/1398618573 wait complete. 2022-06-29T16:05:15.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.110+0000 7febbe5c9700 1 Processor -- start 2022-06-29T16:05:15.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.110+0000 7febbe5c9700 1 -- start start 2022-06-29T16:05:15.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.110+0000 7febbe5c9700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7febb80b2c50 0x7febb80af3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:15.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.110+0000 7febbe5c9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febb80b48b0 0x7febb813ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:15.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.110+0000 7febbe5c9700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7febb80b5eb0 0x7febb813cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:15.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.110+0000 7febbe5c9700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb80adbc0 con 0x7febb80b48b0 2022-06-29T16:05:15.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.110+0000 7febbe5c9700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb8146c10 con 0x7febb80b5eb0 2022-06-29T16:05:15.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.110+0000 7febbe5c9700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febb80b14e0 con 0x7febb80b2c50 2022-06-29T16:05:15.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.110+0000 7febbcdc6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febb80b48b0 0x7febb813ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:15.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.110+0000 7febbcdc6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febb80b48b0 0x7febb813ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49996/0 (socket says 172.21.15.60:49996) 2022-06-29T16:05:15.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.110+0000 7febbcdc6700 1 -- 172.21.15.60:0/456998779 learned_addr learned my addr 172.21.15.60:0/456998779 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:15.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.110+0000 7febbd5c7700 1 --2- 172.21.15.60:0/456998779 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7febb80b2c50 0x7febb80af3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:15.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.110+0000 7febbddc8700 1 --2- 172.21.15.60:0/456998779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7febb80b5eb0 0x7febb813cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:15.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.110+0000 7febbcdc6700 1 -- 172.21.15.60:0/456998779 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7febb80b2c50 msgr2=0x7febb80af3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:15.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.110+0000 7febbcdc6700 1 --2- 172.21.15.60:0/456998779 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7febb80b2c50 0x7febb80af3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:15.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.110+0000 7febbcdc6700 1 -- 172.21.15.60:0/456998779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7febb80b5eb0 msgr2=0x7febb813cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:15.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.110+0000 7febbcdc6700 1 --2- 172.21.15.60:0/456998779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7febb80b5eb0 0x7febb813cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:15.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.110+0000 7febbcdc6700 1 -- 172.21.15.60:0/456998779 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb400b1e0 con 0x7febb80b48b0 2022-06-29T16:05:15.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.110+0000 7febbddc8700 1 --2- 172.21.15.60:0/456998779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7febb80b5eb0 0x7febb813cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:15.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.110+0000 7febbcdc6700 1 --2- 172.21.15.60:0/456998779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febb80b48b0 0x7febb813ca30 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 rx=0x7febb4002f70 tx=0x7febb4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:15.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.110+0000 7febae7fc700 1 -- 172.21.15.60:0/456998779 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febb40117e0 con 0x7febb80b48b0 2022-06-29T16:05:15.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.110+0000 7febae7fc700 1 -- 172.21.15.60:0/456998779 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febb4011940 con 0x7febb80b48b0 2022-06-29T16:05:15.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.110+0000 7febbe5c9700 1 -- 172.21.15.60:0/456998779 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febb80b1780 con 0x7febb80b48b0 2022-06-29T16:05:15.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.110+0000 7febae7fc700 1 -- 172.21.15.60:0/456998779 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febb4011ca0 con 0x7febb80b48b0 2022-06-29T16:05:15.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.110+0000 7febbe5c9700 1 -- 172.21.15.60:0/456998779 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febb813d410 con 0x7febb80b48b0 2022-06-29T16:05:15.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.114+0000 7febbe5c9700 1 -- 172.21.15.60:0/456998779 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febb80b4f70 con 0x7febb80b48b0 2022-06-29T16:05:15.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.114+0000 7febae7fc700 1 -- 172.21.15.60:0/456998779 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7febb4025030 con 0x7febb80b48b0 2022-06-29T16:05:15.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.114+0000 7febae7fc700 1 --2- 172.21.15.60:0/456998779 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7feb980539e0 0x7feb98055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:15.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.114+0000 7febae7fc700 1 -- 172.21.15.60:0/456998779 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5607+0+0 (secure 0 0 0) 0x7febb406d100 con 0x7febb80b48b0 2022-06-29T16:05:15.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.114+0000 7febbd5c7700 1 --2- 172.21.15.60:0/456998779 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7feb980539e0 0x7feb98055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:15.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.114+0000 7febbd5c7700 1 --2- 172.21.15.60:0/456998779 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7feb980539e0 0x7feb98055ea0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7feba800c000 tx=0x7feba8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:15.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.122+0000 7febae7fc700 1 -- 172.21.15.60:0/456998779 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7febb4044060 con 0x7febb80b48b0 2022-06-29T16:05:15.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.250+0000 7febbe5c9700 1 -- 172.21.15.60:0/456998779 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7febb80573d0 con 0x7febb80b48b0 2022-06-29T16:05:15.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.982+0000 7febae7fc700 1 -- 172.21.15.60:0/456998779 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v78) v1 ==== 125+0+0 (secure 0 0 0) 0x7febb40230a0 con 0x7febb80b48b0 2022-06-29T16:05:15.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.990+0000 7febbe5c9700 1 -- 172.21.15.60:0/456998779 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7feb980539e0 msgr2=0x7feb98055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:15.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.990+0000 7febbe5c9700 1 --2- 172.21.15.60:0/456998779 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7feb980539e0 0x7feb98055ea0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7feba800c000 tx=0x7feba8008040).stop 2022-06-29T16:05:15.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.990+0000 7febbe5c9700 1 -- 172.21.15.60:0/456998779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febb80b48b0 msgr2=0x7febb813ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:16.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.990+0000 7febbe5c9700 1 --2- 172.21.15.60:0/456998779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febb80b48b0 0x7febb813ca30 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 rx=0x7febb4002f70 tx=0x7febb4015000).stop 2022-06-29T16:05:16.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.990+0000 7febbe5c9700 1 -- 172.21.15.60:0/456998779 shutdown_connections 2022-06-29T16:05:16.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.990+0000 7febbe5c9700 1 --2- 172.21.15.60:0/456998779 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7feb980539e0 0x7feb98055ea0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:16.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.990+0000 7febbe5c9700 1 --2- 172.21.15.60:0/456998779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7febb80b5eb0 0x7febb813cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:16.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.990+0000 7febbe5c9700 1 --2- 172.21.15.60:0/456998779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febb80b48b0 0x7febb813ca30 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:16.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.990+0000 7febbe5c9700 1 --2- 172.21.15.60:0/456998779 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7febb80b2c50 0x7febb80af3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:16.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.990+0000 7febbe5c9700 1 -- 172.21.15.60:0/456998779 >> 172.21.15.60:0/456998779 conn(0x7febb8062090 msgr2=0x7febb81431a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:16.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.990+0000 7febbe5c9700 1 -- 172.21.15.60:0/456998779 shutdown_connections 2022-06-29T16:05:16.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:15.990+0000 7febbe5c9700 1 -- 172.21.15.60:0/456998779 wait complete. 2022-06-29T16:05:16.002 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2022-06-29T16:05:16.010 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-06-29T16:05:16.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.106+0000 7ff92907e700 1 Processor -- start 2022-06-29T16:05:16.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.106+0000 7ff92907e700 1 -- start start 2022-06-29T16:05:16.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.110+0000 7ff92907e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9240b9be0 0x7ff92405bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:16.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.110+0000 7ff92907e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9240b1d70 con 0x7ff9240afee0 2022-06-29T16:05:16.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.110+0000 7ff92907e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9240b9fc0 con 0x7ff9240b28e0 2022-06-29T16:05:16.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.110+0000 7ff92907e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9240b3c40 con 0x7ff9240b9be0 2022-06-29T16:05:16.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.110+0000 7ff9237fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff9240afee0 0x7ff9240b21d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58088/0 (socket says 172.21.15.60:58088) 2022-06-29T16:05:16.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.110+0000 7ff9237fe700 1 -- 172.21.15.60:0/2545862221 learned_addr learned my addr 172.21.15.60:0/2545862221 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:16.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.110+0000 7ff923fff700 1 --2- 172.21.15.60:0/2545862221 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9240b9be0 0x7ff92405bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:16.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.110+0000 7ff922ffd700 1 -- 172.21.15.60:0/2545862221 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1112652168 0 0) 0x7ff9240b1d70 con 0x7ff9240afee0 2022-06-29T16:05:16.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.110+0000 7ff922ffd700 1 -- 172.21.15.60:0/2545862221 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff910003260 con 0x7ff9240afee0 2022-06-29T16:05:16.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.110+0000 7ff922ffd700 1 -- 172.21.15.60:0/2545862221 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 542030507 0 0) 0x7ff9240b9fc0 con 0x7ff9240b28e0 2022-06-29T16:05:16.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.110+0000 7ff922ffd700 1 -- 172.21.15.60:0/2545862221 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9240b1d70 con 0x7ff9240b28e0 2022-06-29T16:05:16.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.110+0000 7ff923fff700 1 -- 172.21.15.60:0/2545862221 >> v1:172.21.15.92:6789/0 conn(0x7ff9240b28e0 legacy=0x7ff924141e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:16.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.110+0000 7ff923fff700 1 -- 172.21.15.60:0/2545862221 >> v1:172.21.15.60:6789/0 conn(0x7ff9240afee0 legacy=0x7ff9240b21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:16.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.110+0000 7ff923fff700 1 -- 172.21.15.60:0/2545862221 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9240b3ee0 con 0x7ff9240b9be0 2022-06-29T16:05:16.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.110+0000 7ff923fff700 1 --2- 172.21.15.60:0/2545862221 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9240b9be0 0x7ff92405bd20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7ff914002f70 tx=0x7ff914006d90).ready entity=mon.2 client_cookie=abcf733051ebdaf2 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:16.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.110+0000 7ff922ffd700 1 -- 172.21.15.60:0/2545862221 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff91400d4a0 con 0x7ff9240b9be0 2022-06-29T16:05:16.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.110+0000 7ff922ffd700 1 -- 172.21.15.60:0/2545862221 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff914000e90 con 0x7ff9240b9be0 2022-06-29T16:05:16.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.110+0000 7ff922ffd700 1 -- 172.21.15.60:0/2545862221 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff91400d7f0 con 0x7ff9240b9be0 2022-06-29T16:05:16.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.110+0000 7ff92907e700 1 -- 172.21.15.60:0/2545862221 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9240b9be0 msgr2=0x7ff92405bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:16.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.110+0000 7ff92907e700 1 --2- 172.21.15.60:0/2545862221 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9240b9be0 0x7ff92405bd20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7ff914002f70 tx=0x7ff914006d90).stop 2022-06-29T16:05:16.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.110+0000 7ff92907e700 1 -- 172.21.15.60:0/2545862221 shutdown_connections 2022-06-29T16:05:16.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.110+0000 7ff92907e700 1 --2- 172.21.15.60:0/2545862221 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9240b9be0 0x7ff92405bd20 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:16.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.110+0000 7ff92907e700 1 -- 172.21.15.60:0/2545862221 >> 172.21.15.60:0/2545862221 conn(0x7ff924062ce0 msgr2=0x7ff9240b9190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:16.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.110+0000 7ff92907e700 1 -- 172.21.15.60:0/2545862221 shutdown_connections 2022-06-29T16:05:16.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.110+0000 7ff92907e700 1 -- 172.21.15.60:0/2545862221 wait complete. 2022-06-29T16:05:16.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.114+0000 7ff92907e700 1 Processor -- start 2022-06-29T16:05:16.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.114+0000 7ff92907e700 1 -- start start 2022-06-29T16:05:16.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.114+0000 7ff92907e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9240afee0 0x7ff924144290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:16.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.114+0000 7ff92907e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9240b28e0 0x7ff92413d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:16.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.114+0000 7ff92907e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9240b9be0 0x7ff924151a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:16.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.114+0000 7ff92907e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff924148100 con 0x7ff9240b28e0 2022-06-29T16:05:16.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.114+0000 7ff92907e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9241483a0 con 0x7ff9240b9be0 2022-06-29T16:05:16.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.114+0000 7ff92907e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff924148640 con 0x7ff9240afee0 2022-06-29T16:05:16.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.114+0000 7ff9237fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9240b28e0 0x7ff92413d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:16.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.114+0000 7ff923fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9240afee0 0x7ff924144290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:16.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.114+0000 7ff9237fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9240b28e0 0x7ff92413d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50010/0 (socket says 172.21.15.60:50010) 2022-06-29T16:05:16.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.114+0000 7ff9237fe700 1 -- 172.21.15.60:0/3843788774 learned_addr learned my addr 172.21.15.60:0/3843788774 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:16.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.114+0000 7ff923fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9240afee0 0x7ff924144290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:34870/0 (socket says 172.21.15.60:34870) 2022-06-29T16:05:16.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.114+0000 7ff92887d700 1 --2- 172.21.15.60:0/3843788774 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9240b9be0 0x7ff924151a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:16.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.118+0000 7ff9237fe700 1 -- 172.21.15.60:0/3843788774 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9240afee0 msgr2=0x7ff924144290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:16.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.118+0000 7ff9237fe700 1 --2- 172.21.15.60:0/3843788774 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9240afee0 0x7ff924144290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:16.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.118+0000 7ff9237fe700 1 -- 172.21.15.60:0/3843788774 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9240b9be0 msgr2=0x7ff924151a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:16.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.118+0000 7ff9237fe700 1 --2- 172.21.15.60:0/3843788774 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9240b9be0 0x7ff924151a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:16.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.118+0000 7ff9237fe700 1 -- 172.21.15.60:0/3843788774 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff91400b120 con 0x7ff9240b28e0 2022-06-29T16:05:16.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.118+0000 7ff92887d700 1 --2- 172.21.15.60:0/3843788774 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9240b9be0 0x7ff924151a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:05:16.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.118+0000 7ff9237fe700 1 --2- 172.21.15.60:0/3843788774 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9240b28e0 0x7ff92413d630 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 rx=0x7ff9180011d0 tx=0x7ff918007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:16.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.118+0000 7ff9217fa700 1 -- 172.21.15.60:0/3843788774 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff9180070d0 con 0x7ff9240b28e0 2022-06-29T16:05:16.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.118+0000 7ff9217fa700 1 -- 172.21.15.60:0/3843788774 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff918007230 con 0x7ff9240b28e0 2022-06-29T16:05:16.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.118+0000 7ff9217fa700 1 -- 172.21.15.60:0/3843788774 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff918005610 con 0x7ff9240b28e0 2022-06-29T16:05:16.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.118+0000 7ff92907e700 1 -- 172.21.15.60:0/3843788774 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff924146790 con 0x7ff9240b28e0 2022-06-29T16:05:16.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.118+0000 7ff92907e700 1 -- 172.21.15.60:0/3843788774 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9241449a0 con 0x7ff9240b28e0 2022-06-29T16:05:16.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.118+0000 7ff92907e700 1 -- 172.21.15.60:0/3843788774 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9240573d0 con 0x7ff9240b28e0 2022-06-29T16:05:16.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.118+0000 7ff9217fa700 1 -- 172.21.15.60:0/3843788774 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff918010070 con 0x7ff9240b28e0 2022-06-29T16:05:16.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.118+0000 7ff9217fa700 1 --2- 172.21.15.60:0/3843788774 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff904053b00 0x7ff904055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:16.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.118+0000 7ff9217fa700 1 -- 172.21.15.60:0/3843788774 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5957+0+0 (secure 0 0 0) 0x7ff91806b620 con 0x7ff9240b28e0 2022-06-29T16:05:16.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.122+0000 7ff923fff700 1 --2- 172.21.15.60:0/3843788774 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff904053b00 0x7ff904055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:16.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.122+0000 7ff923fff700 1 --2- 172.21.15.60:0/3843788774 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff904053b00 0x7ff904055fc0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7ff914018000 tx=0x7ff914014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:16.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.126+0000 7ff9217fa700 1 -- 172.21.15.60:0/3843788774 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff9180425b0 con 0x7ff9240b28e0 2022-06-29T16:05:16.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.286+0000 7ff92907e700 1 -- 172.21.15.60:0/3843788774 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7ff92405d990 con 0x7ff9240b28e0 2022-06-29T16:05:16.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.298+0000 7ff9217fa700 1 -- 172.21.15.60:0/3843788774 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 15 and data pool 16 v33) v1 ==== 189+0+0 (secure 0 0 0) 0x7ff9180180d0 con 0x7ff9240b28e0 2022-06-29T16:05:16.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.302+0000 7ff92907e700 1 -- 172.21.15.60:0/3843788774 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff904053b00 msgr2=0x7ff904055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:16.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.302+0000 7ff92907e700 1 --2- 172.21.15.60:0/3843788774 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff904053b00 0x7ff904055fc0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7ff914018000 tx=0x7ff914014000).stop 2022-06-29T16:05:16.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.302+0000 7ff92907e700 1 -- 172.21.15.60:0/3843788774 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9240b28e0 msgr2=0x7ff92413d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:16.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.302+0000 7ff92907e700 1 --2- 172.21.15.60:0/3843788774 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9240b28e0 0x7ff92413d630 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 rx=0x7ff9180011d0 tx=0x7ff918007b50).stop 2022-06-29T16:05:16.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.302+0000 7ff92907e700 1 -- 172.21.15.60:0/3843788774 shutdown_connections 2022-06-29T16:05:16.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.302+0000 7ff92907e700 1 --2- 172.21.15.60:0/3843788774 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff904053b00 0x7ff904055fc0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:16.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.302+0000 7ff92907e700 1 --2- 172.21.15.60:0/3843788774 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff9240b9be0 0x7ff924151a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:16.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.302+0000 7ff92907e700 1 --2- 172.21.15.60:0/3843788774 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9240b28e0 0x7ff92413d630 secure :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 rx=0x7ff9180011d0 tx=0x7ff918007b50).stop 2022-06-29T16:05:16.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.302+0000 7ff92907e700 1 --2- 172.21.15.60:0/3843788774 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9240afee0 0x7ff924144290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:16.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.302+0000 7ff92907e700 1 -- 172.21.15.60:0/3843788774 >> 172.21.15.60:0/3843788774 conn(0x7ff924062ce0 msgr2=0x7ff9240b6140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:16.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.302+0000 7ff92907e700 1 -- 172.21.15.60:0/3843788774 shutdown_connections 2022-06-29T16:05:16.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.306+0000 7ff92907e700 1 -- 172.21.15.60:0/3843788774 wait complete. 2022-06-29T16:05:16.314 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 15 and data pool 16 2022-06-29T16:05:16.323 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:05:16.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.414+0000 7f6942656700 1 Processor -- start 2022-06-29T16:05:16.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.414+0000 7f6942656700 1 -- start start 2022-06-29T16:05:16.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.414+0000 7f6942656700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f693c14b5c0 0x7f693c14d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:16.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.414+0000 7f6942656700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f693c0b1de0 con 0x7f693c0b0610 2022-06-29T16:05:16.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.414+0000 7f6942656700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f693c146750 con 0x7f693c0b9c20 2022-06-29T16:05:16.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.414+0000 7f6942656700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f693c0b1330 con 0x7f693c14b5c0 2022-06-29T16:05:16.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.414+0000 7f6940e53700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f693c0b0610 0x7f693c0b24b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58104/0 (socket says 172.21.15.60:58104) 2022-06-29T16:05:16.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.414+0000 7f6940e53700 1 -- 172.21.15.60:0/3733116179 learned_addr learned my addr 172.21.15.60:0/3733116179 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:16.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.414+0000 7f6941e55700 1 --2- 172.21.15.60:0/3733116179 >> v2:172.21.15.60:3300/0 conn(0x7f693c14b5c0 0x7f693c14d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:16.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.414+0000 7f693bfff700 1 -- 172.21.15.60:0/3733116179 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1012599531 0 0) 0x7f693c0b1de0 con 0x7f693c0b0610 2022-06-29T16:05:16.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.414+0000 7f693bfff700 1 -- 172.21.15.60:0/3733116179 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6928003260 con 0x7f693c0b0610 2022-06-29T16:05:16.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.414+0000 7f693bfff700 1 -- 172.21.15.60:0/3733116179 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 90976794 0 0) 0x7f6928003260 con 0x7f693c0b0610 2022-06-29T16:05:16.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.414+0000 7f6941e55700 1 -- 172.21.15.60:0/3733116179 >> v1:172.21.15.92:6789/0 conn(0x7f693c0b9c20 legacy=0x7f693c146040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:16.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.414+0000 7f6941e55700 1 -- 172.21.15.60:0/3733116179 >> v1:172.21.15.60:6789/0 conn(0x7f693c0b0610 legacy=0x7f693c0b24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:16.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.414+0000 7f6941e55700 1 -- 172.21.15.60:0/3733116179 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f693c0b2bc0 con 0x7f693c14b5c0 2022-06-29T16:05:16.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.414+0000 7f6941e55700 1 --2- 172.21.15.60:0/3733116179 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f693c14b5c0 0x7f693c14d9b0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 rx=0x7f6934001b80 tx=0x7f692c006df0).ready entity=mon.0 client_cookie=33893c7b1f9f24b7 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:16.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.414+0000 7f693bfff700 1 -- 172.21.15.60:0/3733116179 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69340019e0 con 0x7f693c14b5c0 2022-06-29T16:05:16.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.414+0000 7f693bfff700 1 -- 172.21.15.60:0/3733116179 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f692c000e90 con 0x7f693c14b5c0 2022-06-29T16:05:16.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.414+0000 7f693bfff700 1 -- 172.21.15.60:0/3733116179 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f692c011400 con 0x7f693c14b5c0 2022-06-29T16:05:16.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.418+0000 7f6942656700 1 -- 172.21.15.60:0/3733116179 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f693c14b5c0 msgr2=0x7f693c14d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:16.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.418+0000 7f6942656700 1 --2- 172.21.15.60:0/3733116179 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f693c14b5c0 0x7f693c14d9b0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 rx=0x7f6934001b80 tx=0x7f692c006df0).stop 2022-06-29T16:05:16.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.418+0000 7f6942656700 1 -- 172.21.15.60:0/3733116179 shutdown_connections 2022-06-29T16:05:16.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.418+0000 7f6942656700 1 --2- 172.21.15.60:0/3733116179 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f693c14b5c0 0x7f693c14d9b0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:16.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.418+0000 7f6942656700 1 -- 172.21.15.60:0/3733116179 >> 172.21.15.60:0/3733116179 conn(0x7f693c062cb0 msgr2=0x7f693c0b91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:16.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.418+0000 7f6942656700 1 -- 172.21.15.60:0/3733116179 shutdown_connections 2022-06-29T16:05:16.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.418+0000 7f6942656700 1 -- 172.21.15.60:0/3733116179 wait complete. 2022-06-29T16:05:16.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.418+0000 7f6942656700 1 Processor -- start 2022-06-29T16:05:16.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.418+0000 7f6942656700 1 -- start start 2022-06-29T16:05:16.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.422+0000 7f6942656700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f693c0b0610 0x7f693c0b7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:16.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.422+0000 7f6942656700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f693c0b9c20 0x7f693c0aefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:16.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.422+0000 7f6942656700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f693c14b5c0 0x7f693c14a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:16.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.422+0000 7f6942656700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693c0af920 con 0x7f693c0b9c20 2022-06-29T16:05:16.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.422+0000 7f6942656700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693c0ad400 con 0x7f693c0b0610 2022-06-29T16:05:16.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.422+0000 7f6942656700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f693c0ad660 con 0x7f693c14b5c0 2022-06-29T16:05:16.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.422+0000 7f6940e53700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f693c0b9c20 0x7f693c0aefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:16.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.422+0000 7f6940e53700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f693c0b9c20 0x7f693c0aefc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50024/0 (socket says 172.21.15.60:50024) 2022-06-29T16:05:16.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.422+0000 7f6940e53700 1 -- 172.21.15.60:0/4060261308 learned_addr learned my addr 172.21.15.60:0/4060261308 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:16.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.422+0000 7f6941e55700 1 --2- 172.21.15.60:0/4060261308 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f693c14b5c0 0x7f693c14a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:16.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.422+0000 7f6941654700 1 --2- 172.21.15.60:0/4060261308 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f693c0b0610 0x7f693c0b7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:16.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.422+0000 7f6940e53700 1 -- 172.21.15.60:0/4060261308 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f693c14b5c0 msgr2=0x7f693c14a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:16.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.422+0000 7f6940e53700 1 --2- 172.21.15.60:0/4060261308 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f693c14b5c0 0x7f693c14a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:16.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.422+0000 7f6940e53700 1 -- 172.21.15.60:0/4060261308 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f693c0b0610 msgr2=0x7f693c0b7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:16.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.422+0000 7f6940e53700 1 --2- 172.21.15.60:0/4060261308 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f693c0b0610 0x7f693c0b7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:16.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.422+0000 7f6940e53700 1 -- 172.21.15.60:0/4060261308 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f692c00b180 con 0x7f693c0b9c20 2022-06-29T16:05:16.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.422+0000 7f6941e55700 1 --2- 172.21.15.60:0/4060261308 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f693c14b5c0 0x7f693c14a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:16.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.422+0000 7f6941654700 1 --2- 172.21.15.60:0/4060261308 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f693c0b0610 0x7f693c0b7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:16.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.422+0000 7f6940e53700 1 --2- 172.21.15.60:0/4060261308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f693c0b9c20 0x7f693c0aefc0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 rx=0x7f6934001c40 tx=0x7f69340021e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:16.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.422+0000 7f693a7fc700 1 -- 172.21.15.60:0/4060261308 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6934004ab0 con 0x7f693c0b9c20 2022-06-29T16:05:16.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.422+0000 7f693a7fc700 1 -- 172.21.15.60:0/4060261308 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6934004c10 con 0x7f693c0b9c20 2022-06-29T16:05:16.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.422+0000 7f6942656700 1 -- 172.21.15.60:0/4060261308 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f693c0ad900 con 0x7f693c0b9c20 2022-06-29T16:05:16.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.422+0000 7f693a7fc700 1 -- 172.21.15.60:0/4060261308 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6934007170 con 0x7f693c0b9c20 2022-06-29T16:05:16.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.422+0000 7f6942656700 1 -- 172.21.15.60:0/4060261308 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f693c0af4b0 con 0x7f693c0b9c20 2022-06-29T16:05:16.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.422+0000 7f6942656700 1 -- 172.21.15.60:0/4060261308 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f693c0573d0 con 0x7f693c0b9c20 2022-06-29T16:05:16.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.426+0000 7f693a7fc700 1 -- 172.21.15.60:0/4060261308 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6934010070 con 0x7f693c0b9c20 2022-06-29T16:05:16.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.426+0000 7f693a7fc700 1 --2- 172.21.15.60:0/4060261308 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f691c053950 0x7f691c055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:16.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.426+0000 7f693a7fc700 1 -- 172.21.15.60:0/4060261308 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f693406be30 con 0x7f693c0b9c20 2022-06-29T16:05:16.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.426+0000 7f6941654700 1 --2- 172.21.15.60:0/4060261308 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f691c053950 0x7f691c055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:16.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.426+0000 7f6941654700 1 --2- 172.21.15.60:0/4060261308 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f691c053950 0x7f691c055e10 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f693000b000 tx=0x7f6930009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:16.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.434+0000 7f693a7fc700 1 -- 172.21.15.60:0/4060261308 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6934042d50 con 0x7f693c0b9c20 2022-06-29T16:05:16.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.562+0000 7f6942656700 1 -- 172.21.15.60:0/4060261308 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f693c067ae0 con 0x7f693c0b9c20 2022-06-29T16:05:16.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.566+0000 7f693a7fc700 1 -- 172.21.15.60:0/4060261308 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+16054 (secure 0 0 0) 0x7f6934046370 con 0x7f693c0b9c20 2022-06-29T16:05:16.574 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:16.574 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":79,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:05:16.293622+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:05:14.715486+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2022-06-29T16:05:15.306355+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:05:16.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.566+0000 7f6942656700 1 -- 172.21.15.60:0/4060261308 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f691c053950 msgr2=0x7f691c055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:16.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.566+0000 7f6942656700 1 --2- 172.21.15.60:0/4060261308 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f691c053950 0x7f691c055e10 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f693000b000 tx=0x7f6930009040).stop 2022-06-29T16:05:16.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.566+0000 7f6942656700 1 -- 172.21.15.60:0/4060261308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f693c0b9c20 msgr2=0x7f693c0aefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:16.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.566+0000 7f6942656700 1 --2- 172.21.15.60:0/4060261308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f693c0b9c20 0x7f693c0aefc0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 rx=0x7f6934001c40 tx=0x7f69340021e0).stop 2022-06-29T16:05:16.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.566+0000 7f6942656700 1 -- 172.21.15.60:0/4060261308 shutdown_connections 2022-06-29T16:05:16.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.566+0000 7f6942656700 1 --2- 172.21.15.60:0/4060261308 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f691c053950 0x7f691c055e10 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:16.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.566+0000 7f6942656700 1 --2- 172.21.15.60:0/4060261308 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f693c14b5c0 0x7f693c14a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:16.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.566+0000 7f6942656700 1 --2- 172.21.15.60:0/4060261308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f693c0b9c20 0x7f693c0aefc0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:16.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.566+0000 7f6942656700 1 --2- 172.21.15.60:0/4060261308 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f693c0b0610 0x7f693c0b7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:16.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.566+0000 7f6942656700 1 -- 172.21.15.60:0/4060261308 >> 172.21.15.60:0/4060261308 conn(0x7f693c062cb0 msgr2=0x7f693c0b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:16.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.570+0000 7f6942656700 1 -- 172.21.15.60:0/4060261308 shutdown_connections 2022-06-29T16:05:16.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.570+0000 7f6942656700 1 -- 172.21.15.60:0/4060261308 wait complete. 2022-06-29T16:05:16.587 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:05:16.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.682+0000 7efd8ed83700 1 Processor -- start 2022-06-29T16:05:16.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.682+0000 7efd8ed83700 1 -- start start 2022-06-29T16:05:16.696 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.686+0000 7efd8ed83700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd880ae7c0 0x7efd88141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:16.696 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.686+0000 7efd8ed83700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efd880b4010 0x7efd88142490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:16.696 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.686+0000 7efd8ed83700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7efd88146160 0x7efd881465f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:16.696 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.686+0000 7efd8ed83700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd88143a40 con 0x7efd880b4010 2022-06-29T16:05:16.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.686+0000 7efd8ed83700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd880aee60 con 0x7efd88146160 2022-06-29T16:05:16.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.686+0000 7efd8ed83700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd880af100 con 0x7efd880ae7c0 2022-06-29T16:05:16.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.686+0000 7efd87fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efd880b4010 0x7efd88142490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:16.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.686+0000 7efd8cb1f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd880ae7c0 0x7efd88141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:16.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.686+0000 7efd8cb1f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd880ae7c0 0x7efd88141fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:34892/0 (socket says 172.21.15.60:34892) 2022-06-29T16:05:16.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.686+0000 7efd8cb1f700 1 -- 172.21.15.60:0/2944470189 learned_addr learned my addr 172.21.15.60:0/2944470189 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:16.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.686+0000 7efd8d320700 1 --2- 172.21.15.60:0/2944470189 >> v2:172.21.15.92:3300/0 conn(0x7efd88146160 0x7efd881465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:16.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.686+0000 7efd8cb1f700 1 -- 172.21.15.60:0/2944470189 >> v2:172.21.15.92:3300/0 conn(0x7efd88146160 msgr2=0x7efd881465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:16.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.686+0000 7efd8cb1f700 1 --2- 172.21.15.60:0/2944470189 >> v2:172.21.15.92:3300/0 conn(0x7efd88146160 0x7efd881465f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:16.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.686+0000 7efd8cb1f700 1 -- 172.21.15.60:0/2944470189 >> v2:172.21.15.60:3300/0 conn(0x7efd880b4010 msgr2=0x7efd88142490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:16.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.686+0000 7efd8cb1f700 1 --2- 172.21.15.60:0/2944470189 >> v2:172.21.15.60:3300/0 conn(0x7efd880b4010 0x7efd88142490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:16.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.686+0000 7efd8cb1f700 1 -- 172.21.15.60:0/2944470189 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd880aeba0 con 0x7efd880ae7c0 2022-06-29T16:05:16.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.686+0000 7efd87fff700 1 --2- 172.21.15.60:0/2944470189 >> v2:172.21.15.60:3300/0 conn(0x7efd880b4010 0x7efd88142490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:05:16.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.686+0000 7efd8cb1f700 1 --2- 172.21.15.60:0/2944470189 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd880ae7c0 0x7efd88141fa0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7efd78002f70 tx=0x7efd78006eb0).ready entity=mon.2 client_cookie=5a4a69efadb9d297 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:16.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.686+0000 7efd877fe700 1 -- 172.21.15.60:0/2944470189 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd7800d570 con 0x7efd880ae7c0 2022-06-29T16:05:16.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.686+0000 7efd877fe700 1 -- 172.21.15.60:0/2944470189 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd78000e90 con 0x7efd880ae7c0 2022-06-29T16:05:16.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.686+0000 7efd877fe700 1 -- 172.21.15.60:0/2944470189 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd7800d900 con 0x7efd880ae7c0 2022-06-29T16:05:16.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.686+0000 7efd8ed83700 1 -- 172.21.15.60:0/2944470189 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd880ae7c0 msgr2=0x7efd88141fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:16.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.686+0000 7efd8ed83700 1 --2- 172.21.15.60:0/2944470189 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd880ae7c0 0x7efd88141fa0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7efd78002f70 tx=0x7efd78006eb0).stop 2022-06-29T16:05:16.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.686+0000 7efd8ed83700 1 -- 172.21.15.60:0/2944470189 shutdown_connections 2022-06-29T16:05:16.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.686+0000 7efd8ed83700 1 --2- 172.21.15.60:0/2944470189 >> v2:172.21.15.92:3300/0 conn(0x7efd88146160 0x7efd881465f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:16.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.686+0000 7efd8ed83700 1 --2- 172.21.15.60:0/2944470189 >> v2:172.21.15.60:3300/0 conn(0x7efd880b4010 0x7efd88142490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:16.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.686+0000 7efd8ed83700 1 --2- 172.21.15.60:0/2944470189 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd880ae7c0 0x7efd88141fa0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:16.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.686+0000 7efd8ed83700 1 -- 172.21.15.60:0/2944470189 >> 172.21.15.60:0/2944470189 conn(0x7efd880b9340 msgr2=0x7efd880b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:16.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.686+0000 7efd8ed83700 1 -- 172.21.15.60:0/2944470189 shutdown_connections 2022-06-29T16:05:16.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.686+0000 7efd8ed83700 1 -- 172.21.15.60:0/2944470189 wait complete. 2022-06-29T16:05:16.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.686+0000 7efd8ed83700 1 Processor -- start 2022-06-29T16:05:16.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.686+0000 7efd8ed83700 1 -- start start 2022-06-29T16:05:16.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.690+0000 7efd8ed83700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd880ae7c0 0x7efd880b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:16.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.690+0000 7efd8ed83700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd880b4010 0x7efd8813ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:16.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.690+0000 7efd8ed83700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd88146160 0x7efd8813f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:16.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.690+0000 7efd8ed83700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd880b8c40 con 0x7efd880ae7c0 2022-06-29T16:05:16.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.690+0000 7efd8ed83700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd880af5f0 con 0x7efd88146160 2022-06-29T16:05:16.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.690+0000 7efd8ed83700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd880b12a0 con 0x7efd880b4010 2022-06-29T16:05:16.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.690+0000 7efd8cb1f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd880ae7c0 0x7efd880b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:16.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.690+0000 7efd87fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd880b4010 0x7efd8813ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:16.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.690+0000 7efd8cb1f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd880ae7c0 0x7efd880b2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50036/0 (socket says 172.21.15.60:50036) 2022-06-29T16:05:16.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.690+0000 7efd8cb1f700 1 -- 172.21.15.60:0/685870734 learned_addr learned my addr 172.21.15.60:0/685870734 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:16.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.690+0000 7efd87fff700 1 --2- 172.21.15.60:0/685870734 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd880b4010 0x7efd8813ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:34900/0 (socket says 172.21.15.60:34900) 2022-06-29T16:05:16.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.690+0000 7efd8d320700 1 --2- 172.21.15.60:0/685870734 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd88146160 0x7efd8813f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:16.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.690+0000 7efd8cb1f700 1 -- 172.21.15.60:0/685870734 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd880b4010 msgr2=0x7efd8813ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:16.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.690+0000 7efd8cb1f700 1 --2- 172.21.15.60:0/685870734 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd880b4010 0x7efd8813ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:16.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.690+0000 7efd8cb1f700 1 -- 172.21.15.60:0/685870734 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd88146160 msgr2=0x7efd8813f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:16.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.690+0000 7efd8cb1f700 1 --2- 172.21.15.60:0/685870734 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd88146160 0x7efd8813f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:16.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.690+0000 7efd8cb1f700 1 -- 172.21.15.60:0/685870734 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd7000b180 con 0x7efd880ae7c0 2022-06-29T16:05:16.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.690+0000 7efd8cb1f700 1 --2- 172.21.15.60:0/685870734 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd880ae7c0 0x7efd880b2cf0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 rx=0x7efd78002f70 tx=0x7efd78015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:16.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.690+0000 7efd85ffb700 1 -- 172.21.15.60:0/685870734 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd780116e0 con 0x7efd880ae7c0 2022-06-29T16:05:16.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.690+0000 7efd85ffb700 1 -- 172.21.15.60:0/685870734 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd78011840 con 0x7efd880ae7c0 2022-06-29T16:05:16.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.690+0000 7efd85ffb700 1 -- 172.21.15.60:0/685870734 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd78011ba0 con 0x7efd880ae7c0 2022-06-29T16:05:16.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.690+0000 7efd8ed83700 1 -- 172.21.15.60:0/685870734 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd7800b1e0 con 0x7efd880ae7c0 2022-06-29T16:05:16.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.690+0000 7efd8ed83700 1 -- 172.21.15.60:0/685870734 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd880b6800 con 0x7efd880ae7c0 2022-06-29T16:05:16.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.690+0000 7efd8ed83700 1 -- 172.21.15.60:0/685870734 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd880afb90 con 0x7efd880ae7c0 2022-06-29T16:05:16.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.690+0000 7efd85ffb700 1 -- 172.21.15.60:0/685870734 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7efd7801a040 con 0x7efd880ae7c0 2022-06-29T16:05:16.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.694+0000 7efd85ffb700 1 --2- 172.21.15.60:0/685870734 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd6c053aa0 0x7efd6c055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:16.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.694+0000 7efd85ffb700 1 -- 172.21.15.60:0/685870734 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6159+0+0 (secure 0 0 0) 0x7efd7806cbe0 con 0x7efd880ae7c0 2022-06-29T16:05:16.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.694+0000 7efd87fff700 1 --2- 172.21.15.60:0/685870734 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd6c053aa0 0x7efd6c055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:16.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.694+0000 7efd87fff700 1 --2- 172.21.15.60:0/685870734 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd6c053aa0 0x7efd6c055f60 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7efd7000b000 tx=0x7efd70006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:16.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.698+0000 7efd85ffb700 1 -- 172.21.15.60:0/685870734 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efd78043b30 con 0x7efd880ae7c0 2022-06-29T16:05:16.829 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.822+0000 7efd8ed83700 1 -- 172.21.15.60:0/685870734 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd880af7b0 con 0x7efd880ae7c0 2022-06-29T16:05:16.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.822+0000 7efd85ffb700 1 -- 172.21.15.60:0/685870734 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16170 (secure 0 0 0) 0x7efd78023020 con 0x7efd880ae7c0 2022-06-29T16:05:16.831 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:16.831 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":81,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:05:16.667913+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:05:14.715486+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2022-06-29T16:05:15.306355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{"172.21.15.92:6835/2464413229":"2022-06-30T16:05:16.667864+0000","172.21.15.92:6834/2464413229":"2022-06-30T16:05:16.667864+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:05:16.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.826+0000 7efd8ed83700 1 -- 172.21.15.60:0/685870734 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd6c053aa0 msgr2=0x7efd6c055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:16.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.826+0000 7efd8ed83700 1 --2- 172.21.15.60:0/685870734 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd6c053aa0 0x7efd6c055f60 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7efd7000b000 tx=0x7efd70006df0).stop 2022-06-29T16:05:16.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.826+0000 7efd8ed83700 1 -- 172.21.15.60:0/685870734 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd880ae7c0 msgr2=0x7efd880b2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:16.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.826+0000 7efd8ed83700 1 --2- 172.21.15.60:0/685870734 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd880ae7c0 0x7efd880b2cf0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 rx=0x7efd78002f70 tx=0x7efd78015000).stop 2022-06-29T16:05:16.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.826+0000 7efd8ed83700 1 -- 172.21.15.60:0/685870734 shutdown_connections 2022-06-29T16:05:16.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.826+0000 7efd8ed83700 1 --2- 172.21.15.60:0/685870734 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd6c053aa0 0x7efd6c055f60 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:16.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.826+0000 7efd8ed83700 1 --2- 172.21.15.60:0/685870734 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd88146160 0x7efd8813f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:16.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.826+0000 7efd8ed83700 1 --2- 172.21.15.60:0/685870734 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd880b4010 0x7efd8813ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:16.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.826+0000 7efd8ed83700 1 --2- 172.21.15.60:0/685870734 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd880ae7c0 0x7efd880b2cf0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:16.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.826+0000 7efd8ed83700 1 -- 172.21.15.60:0/685870734 >> 172.21.15.60:0/685870734 conn(0x7efd880b9340 msgr2=0x7efd88141110 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:16.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.826+0000 7efd8ed83700 1 -- 172.21.15.60:0/685870734 shutdown_connections 2022-06-29T16:05:16.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.826+0000 7efd8ed83700 1 -- 172.21.15.60:0/685870734 wait complete. 2022-06-29T16:05:16.845 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-06-29T16:05:16.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.946+0000 7f4519dfa700 1 Processor -- start 2022-06-29T16:05:16.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.946+0000 7f4519dfa700 1 -- start start 2022-06-29T16:05:16.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.946+0000 7f4519dfa700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f451405cf40 0x7f45140ab880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:16.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.946+0000 7f4519dfa700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451414b380 con 0x7f451414af10 2022-06-29T16:05:16.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.946+0000 7f4519dfa700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451405cb40 con 0x7f45140b2120 2022-06-29T16:05:16.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.946+0000 7f4519dfa700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45140b1720 con 0x7f451405cf40 2022-06-29T16:05:16.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.950+0000 7f45195f9700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f451414af10 0x7f451405c430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58134/0 (socket says 172.21.15.60:58134) 2022-06-29T16:05:16.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.950+0000 7f45195f9700 1 -- 172.21.15.60:0/508916363 learned_addr learned my addr 172.21.15.60:0/508916363 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:16.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.950+0000 7f4513fff700 1 --2- 172.21.15.60:0/508916363 >> v2:172.21.15.92:3300/0 conn(0x7f451405cf40 0x7f45140ab880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:16.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.950+0000 7f45137fe700 1 -- 172.21.15.60:0/508916363 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1518764222 0 0) 0x7f451414b380 con 0x7f451414af10 2022-06-29T16:05:16.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.950+0000 7f45137fe700 1 -- 172.21.15.60:0/508916363 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4500003260 con 0x7f451414af10 2022-06-29T16:05:16.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.950+0000 7f4513fff700 1 -- 172.21.15.60:0/508916363 >> v1:172.21.15.92:6789/0 conn(0x7f45140b2120 legacy=0x7f451405bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:16.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.950+0000 7f4513fff700 1 -- 172.21.15.60:0/508916363 >> v1:172.21.15.60:6789/0 conn(0x7f451414af10 legacy=0x7f451405c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:16.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.950+0000 7f4513fff700 1 -- 172.21.15.60:0/508916363 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f451414a530 con 0x7f451405cf40 2022-06-29T16:05:16.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.950+0000 7f4513fff700 1 --2- 172.21.15.60:0/508916363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f451405cf40 0x7f45140ab880 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f45040032f0 tx=0x7f45040091d0).ready entity=mon.1 client_cookie=bcf335fd12f122c9 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:16.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.950+0000 7f45137fe700 1 -- 172.21.15.60:0/508916363 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f450400b6c0 con 0x7f451405cf40 2022-06-29T16:05:16.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.950+0000 7f45137fe700 1 -- 172.21.15.60:0/508916363 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f450400b820 con 0x7f451405cf40 2022-06-29T16:05:16.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.950+0000 7f45137fe700 1 -- 172.21.15.60:0/508916363 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f450400b990 con 0x7f451405cf40 2022-06-29T16:05:16.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.950+0000 7f4519dfa700 1 -- 172.21.15.60:0/508916363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f451405cf40 msgr2=0x7f45140ab880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:16.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.950+0000 7f4519dfa700 1 --2- 172.21.15.60:0/508916363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f451405cf40 0x7f45140ab880 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f45040032f0 tx=0x7f45040091d0).stop 2022-06-29T16:05:16.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.950+0000 7f4519dfa700 1 -- 172.21.15.60:0/508916363 shutdown_connections 2022-06-29T16:05:16.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.950+0000 7f4519dfa700 1 --2- 172.21.15.60:0/508916363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f451405cf40 0x7f45140ab880 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:16.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.950+0000 7f4519dfa700 1 -- 172.21.15.60:0/508916363 >> 172.21.15.60:0/508916363 conn(0x7f45140b58a0 msgr2=0x7f45140b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:16.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.950+0000 7f4519dfa700 1 -- 172.21.15.60:0/508916363 shutdown_connections 2022-06-29T16:05:16.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.950+0000 7f4519dfa700 1 -- 172.21.15.60:0/508916363 wait complete. 2022-06-29T16:05:16.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.950+0000 7f4519dfa700 1 Processor -- start 2022-06-29T16:05:16.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.950+0000 7f4519dfa700 1 -- start start 2022-06-29T16:05:16.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.954+0000 7f4519dfa700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f451405cf40 0x7f4514159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:16.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.954+0000 7f4519dfa700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45140b2120 0x7f4514146470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:16.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.954+0000 7f4519dfa700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f451414af10 0x7f4514145430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:16.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.954+0000 7f4519dfa700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4514159530 con 0x7f45140b2120 2022-06-29T16:05:16.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.954+0000 7f4519dfa700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45140b0560 con 0x7f451405cf40 2022-06-29T16:05:16.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.954+0000 7f4519dfa700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4514159760 con 0x7f451414af10 2022-06-29T16:05:16.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.954+0000 7f4513fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45140b2120 0x7f4514146470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:16.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.954+0000 7f4513fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45140b2120 0x7f4514146470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50052/0 (socket says 172.21.15.60:50052) 2022-06-29T16:05:16.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.954+0000 7f4513fff700 1 -- 172.21.15.60:0/4083026882 learned_addr learned my addr 172.21.15.60:0/4083026882 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:16.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.954+0000 7f45195f9700 1 --2- 172.21.15.60:0/4083026882 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f451414af10 0x7f4514145430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:16.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.954+0000 7f4518df8700 1 --2- 172.21.15.60:0/4083026882 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f451405cf40 0x7f4514159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:16.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.954+0000 7f4513fff700 1 -- 172.21.15.60:0/4083026882 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f451414af10 msgr2=0x7f4514145430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:16.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.954+0000 7f4513fff700 1 --2- 172.21.15.60:0/4083026882 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f451414af10 0x7f4514145430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:16.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.954+0000 7f4513fff700 1 -- 172.21.15.60:0/4083026882 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f451405cf40 msgr2=0x7f4514159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:16.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.954+0000 7f4513fff700 1 --2- 172.21.15.60:0/4083026882 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f451405cf40 0x7f4514159ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:16.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.954+0000 7f4513fff700 1 -- 172.21.15.60:0/4083026882 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4504009470 con 0x7f45140b2120 2022-06-29T16:05:16.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.954+0000 7f4513fff700 1 --2- 172.21.15.60:0/4083026882 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45140b2120 0x7f4514146470 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 rx=0x7f45040032f0 tx=0x7f4504016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:16.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.954+0000 7f4511ffb700 1 -- 172.21.15.60:0/4083026882 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f450400b6c0 con 0x7f45140b2120 2022-06-29T16:05:16.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.954+0000 7f4511ffb700 1 -- 172.21.15.60:0/4083026882 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45040107a0 con 0x7f45140b2120 2022-06-29T16:05:16.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.954+0000 7f4519dfa700 1 -- 172.21.15.60:0/4083026882 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4514159a00 con 0x7f45140b2120 2022-06-29T16:05:16.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.954+0000 7f4511ffb700 1 -- 172.21.15.60:0/4083026882 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4504010b00 con 0x7f45140b2120 2022-06-29T16:05:16.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.954+0000 7f4519dfa700 1 -- 172.21.15.60:0/4083026882 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4514064030 con 0x7f45140b2120 2022-06-29T16:05:16.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.958+0000 7f4519dfa700 1 -- 172.21.15.60:0/4083026882 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45140573d0 con 0x7f45140b2120 2022-06-29T16:05:16.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.958+0000 7f4511ffb700 1 -- 172.21.15.60:0/4083026882 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f450400f070 con 0x7f45140b2120 2022-06-29T16:05:16.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.958+0000 7f4511ffb700 1 --2- 172.21.15.60:0/4083026882 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f44f4053a70 0x7f44f4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:16.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.958+0000 7f4511ffb700 1 -- 172.21.15.60:0/4083026882 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f450406d6e0 con 0x7f45140b2120 2022-06-29T16:05:16.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.958+0000 7f4518df8700 1 --2- 172.21.15.60:0/4083026882 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f44f4053a70 0x7f44f4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:16.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.958+0000 7f4518df8700 1 --2- 172.21.15.60:0/4083026882 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f44f4053a70 0x7f44f4055f30 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f450800b000 tx=0x7f4508009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:16.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:16.962+0000 7f4511ffb700 1 -- 172.21.15.60:0/4083026882 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f45040445b0 con 0x7f45140b2120 2022-06-29T16:05:17.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.126+0000 7f4519dfa700 1 -- 172.21.15.60:0/4083026882 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f451406a440 con 0x7f45140b2120 2022-06-29T16:05:17.299 INFO:tasks.ceph.mon.a.smithi060.stderr:2022-06-29T16:05:17.294+0000 7fcc1a376700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-06-29T16:05:17.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.298+0000 7f4511ffb700 1 -- 172.21.15.60:0/4083026882 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v34) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4504015020 con 0x7f45140b2120 2022-06-29T16:05:17.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.302+0000 7f4519dfa700 1 -- 172.21.15.60:0/4083026882 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f44f4053a70 msgr2=0x7f44f4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:17.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.302+0000 7f4519dfa700 1 --2- 172.21.15.60:0/4083026882 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f44f4053a70 0x7f44f4055f30 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f450800b000 tx=0x7f4508009040).stop 2022-06-29T16:05:17.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.302+0000 7f4519dfa700 1 -- 172.21.15.60:0/4083026882 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45140b2120 msgr2=0x7f4514146470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:17.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.302+0000 7f4519dfa700 1 --2- 172.21.15.60:0/4083026882 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45140b2120 0x7f4514146470 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 rx=0x7f45040032f0 tx=0x7f4504016000).stop 2022-06-29T16:05:17.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.302+0000 7f4519dfa700 1 -- 172.21.15.60:0/4083026882 shutdown_connections 2022-06-29T16:05:17.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.302+0000 7f4519dfa700 1 --2- 172.21.15.60:0/4083026882 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f44f4053a70 0x7f44f4055f30 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:17.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.302+0000 7f4519dfa700 1 --2- 172.21.15.60:0/4083026882 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f451414af10 0x7f4514145430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:17.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.302+0000 7f4519dfa700 1 --2- 172.21.15.60:0/4083026882 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45140b2120 0x7f4514146470 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:17.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.302+0000 7f4519dfa700 1 --2- 172.21.15.60:0/4083026882 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f451405cf40 0x7f4514159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:17.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.302+0000 7f4519dfa700 1 -- 172.21.15.60:0/4083026882 >> 172.21.15.60:0/4083026882 conn(0x7f45140b58a0 msgr2=0x7f45140ab180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:17.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.302+0000 7f4519dfa700 1 -- 172.21.15.60:0/4083026882 shutdown_connections 2022-06-29T16:05:17.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.302+0000 7f4519dfa700 1 -- 172.21.15.60:0/4083026882 wait complete. 2022-06-29T16:05:17.326 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:05:17.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.426+0000 7f6e0b4b4700 1 Processor -- start 2022-06-29T16:05:17.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.426+0000 7f6e0b4b4700 1 -- start start 2022-06-29T16:05:17.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.426+0000 7f6e0b4b4700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6e040b2130 0x7f6e04146100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:17.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.426+0000 7f6e0b4b4700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e0405d450 con 0x7f6e040aed00 2022-06-29T16:05:17.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.426+0000 7f6e0b4b4700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e040b25d0 con 0x7f6e0405d070 2022-06-29T16:05:17.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.426+0000 7f6e0b4b4700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e040adf90 con 0x7f6e040b2130 2022-06-29T16:05:17.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.426+0000 7f6e09cb1700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6e040b2130 0x7f6e04146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:17.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.426+0000 7f6e09cb1700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6e040b2130 0x7f6e04146100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50062/0 (socket says 172.21.15.60:50062) 2022-06-29T16:05:17.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.426+0000 7f6e09cb1700 1 -- 172.21.15.60:0/4219582444 learned_addr learned my addr 172.21.15.60:0/4219582444 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:17.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.426+0000 7f6e0acb3700 1 --1- 172.21.15.60:0/4219582444 >> v1:172.21.15.60:6789/0 conn(0x7f6e040aed00 0x7f6e041465f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58150/0 (socket says 172.21.15.60:58150) 2022-06-29T16:05:17.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.426+0000 7f6e09cb1700 1 -- 172.21.15.60:0/4219582444 >> v1:172.21.15.92:6789/0 conn(0x7f6e0405d070 legacy=0x7f6e0414aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:17.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.426+0000 7f6e09cb1700 1 -- 172.21.15.60:0/4219582444 >> v1:172.21.15.60:6789/0 conn(0x7f6e040aed00 legacy=0x7f6e041465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:17.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.426+0000 7f6e09cb1700 1 -- 172.21.15.60:0/4219582444 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e0414a350 con 0x7f6e040b2130 2022-06-29T16:05:17.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.426+0000 7f6e09cb1700 1 --2- 172.21.15.60:0/4219582444 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e040b2130 0x7f6e04146100 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0x7f6df8002f70 tx=0x7f6df8006eb0).ready entity=mon.0 client_cookie=8674e0792ca92370 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:17.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.426+0000 7f6e094b0700 1 -- 172.21.15.60:0/4219582444 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6df800d660 con 0x7f6e040b2130 2022-06-29T16:05:17.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.426+0000 7f6e094b0700 1 -- 172.21.15.60:0/4219582444 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df8000e90 con 0x7f6e040b2130 2022-06-29T16:05:17.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.426+0000 7f6e094b0700 1 -- 172.21.15.60:0/4219582444 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6df800d9f0 con 0x7f6e040b2130 2022-06-29T16:05:17.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.426+0000 7f6e0b4b4700 1 -- 172.21.15.60:0/4219582444 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e040b2130 msgr2=0x7f6e04146100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:17.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.426+0000 7f6e0b4b4700 1 --2- 172.21.15.60:0/4219582444 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e040b2130 0x7f6e04146100 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0x7f6df8002f70 tx=0x7f6df8006eb0).stop 2022-06-29T16:05:17.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.426+0000 7f6e0b4b4700 1 -- 172.21.15.60:0/4219582444 shutdown_connections 2022-06-29T16:05:17.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.426+0000 7f6e0b4b4700 1 --2- 172.21.15.60:0/4219582444 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e040b2130 0x7f6e04146100 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:17.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.426+0000 7f6e0b4b4700 1 -- 172.21.15.60:0/4219582444 >> 172.21.15.60:0/4219582444 conn(0x7f6e040b5890 msgr2=0x7f6e040b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:17.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.430+0000 7f6e0b4b4700 1 -- 172.21.15.60:0/4219582444 shutdown_connections 2022-06-29T16:05:17.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.430+0000 7f6e0b4b4700 1 -- 172.21.15.60:0/4219582444 wait complete. 2022-06-29T16:05:17.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.430+0000 7f6e0b4b4700 1 Processor -- start 2022-06-29T16:05:17.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.430+0000 7f6e0b4b4700 1 -- start start 2022-06-29T16:05:17.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.430+0000 7f6e0b4b4700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e0405d070 0x7f6e040656e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:17.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.430+0000 7f6e0b4b4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e040aed00 0x7f6e04116920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:17.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.430+0000 7f6e0b4b4700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e040b2130 0x7f6e040630c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:17.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.430+0000 7f6e0b4b4700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e04156dd0 con 0x7f6e040aed00 2022-06-29T16:05:17.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.430+0000 7f6e0b4b4700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e04064b90 con 0x7f6e0405d070 2022-06-29T16:05:17.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.430+0000 7f6e0b4b4700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e04065bd0 con 0x7f6e040b2130 2022-06-29T16:05:17.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.430+0000 7f6e09cb1700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e040aed00 0x7f6e04116920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:17.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.430+0000 7f6e0a4b2700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e0405d070 0x7f6e040656e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:17.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.430+0000 7f6e09cb1700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e040aed00 0x7f6e04116920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50068/0 (socket says 172.21.15.60:50068) 2022-06-29T16:05:17.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.430+0000 7f6e09cb1700 1 -- 172.21.15.60:0/3994984848 learned_addr learned my addr 172.21.15.60:0/3994984848 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:17.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.430+0000 7f6e0acb3700 1 --2- 172.21.15.60:0/3994984848 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e040b2130 0x7f6e040630c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:17.448 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.430+0000 7f6e09cb1700 1 -- 172.21.15.60:0/3994984848 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e040b2130 msgr2=0x7f6e040630c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:17.448 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.430+0000 7f6e09cb1700 1 --2- 172.21.15.60:0/3994984848 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e040b2130 0x7f6e040630c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:17.448 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.430+0000 7f6e09cb1700 1 -- 172.21.15.60:0/3994984848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e0405d070 msgr2=0x7f6e040656e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:17.448 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.430+0000 7f6e09cb1700 1 --2- 172.21.15.60:0/3994984848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e0405d070 0x7f6e040656e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:17.449 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.430+0000 7f6e09cb1700 1 -- 172.21.15.60:0/3994984848 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df800b1e0 con 0x7f6e040aed00 2022-06-29T16:05:17.449 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.434+0000 7f6e09cb1700 1 --2- 172.21.15.60:0/3994984848 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e040aed00 0x7f6e04116920 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 rx=0x7f6df8002f70 tx=0x7f6df8015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:17.449 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.434+0000 7f6e037fe700 1 -- 172.21.15.60:0/3994984848 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6df8006b40 con 0x7f6e040aed00 2022-06-29T16:05:17.449 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.434+0000 7f6e037fe700 1 -- 172.21.15.60:0/3994984848 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df8011b20 con 0x7f6e040aed00 2022-06-29T16:05:17.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.434+0000 7f6e0b4b4700 1 -- 172.21.15.60:0/3994984848 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e041172a0 con 0x7f6e040aed00 2022-06-29T16:05:17.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.434+0000 7f6e037fe700 1 -- 172.21.15.60:0/3994984848 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6df8011e80 con 0x7f6e040aed00 2022-06-29T16:05:17.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.434+0000 7f6e0b4b4700 1 -- 172.21.15.60:0/3994984848 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e040660b0 con 0x7f6e040aed00 2022-06-29T16:05:17.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.434+0000 7f6e0b4b4700 1 -- 172.21.15.60:0/3994984848 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e0414a5f0 con 0x7f6e040aed00 2022-06-29T16:05:17.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.434+0000 7f6e037fe700 1 -- 172.21.15.60:0/3994984848 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6df8025030 con 0x7f6e040aed00 2022-06-29T16:05:17.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.434+0000 7f6e037fe700 1 --2- 172.21.15.60:0/3994984848 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6de40539e0 0x7f6de4055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:17.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.434+0000 7f6e037fe700 1 -- 172.21.15.60:0/3994984848 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f6df806d1d0 con 0x7f6e040aed00 2022-06-29T16:05:17.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.434+0000 7f6e0a4b2700 1 --2- 172.21.15.60:0/3994984848 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6de40539e0 0x7f6de4055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:17.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.438+0000 7f6e0a4b2700 1 --2- 172.21.15.60:0/3994984848 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6de40539e0 0x7f6de4055ea0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f6df400d000 tx=0x7f6df4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:17.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.442+0000 7f6e037fe700 1 -- 172.21.15.60:0/3994984848 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6df80440a0 con 0x7f6e040aed00 2022-06-29T16:05:17.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.606+0000 7f6e0b4b4700 1 -- 172.21.15.60:0/3994984848 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e040573d0 con 0x7f6e040aed00 2022-06-29T16:05:17.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.606+0000 7f6e037fe700 1 -- 172.21.15.60:0/3994984848 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+2143 (secure 0 0 0) 0x7f6df8023020 con 0x7f6e040aed00 2022-06-29T16:05:17.615 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:17.616 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":35,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6657,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.60:6833/3121204394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3121204394},{"type":"v1","addr":"172.21.15.60:6833","nonce":3121204394}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":34}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:05:16.293932+0000","modified":"2022-06-29T16:05:17.304396+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6688},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6688":{"gid":6688,"name":"b","rank":0,"incarnation":35,"state":"up:creating","state_seq":2,"addr":"172.21.15.92:6835/1245503283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1245503283},{"type":"v1","addr":"172.21.15.92:6835","nonce":1245503283}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-06-29T16:05:17.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.610+0000 7f6e0b4b4700 1 -- 172.21.15.60:0/3994984848 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6de40539e0 msgr2=0x7f6de4055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:17.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.610+0000 7f6e0b4b4700 1 --2- 172.21.15.60:0/3994984848 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6de40539e0 0x7f6de4055ea0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f6df400d000 tx=0x7f6df4009040).stop 2022-06-29T16:05:17.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.610+0000 7f6e0b4b4700 1 -- 172.21.15.60:0/3994984848 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e040aed00 msgr2=0x7f6e04116920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:17.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.610+0000 7f6e0b4b4700 1 --2- 172.21.15.60:0/3994984848 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e040aed00 0x7f6e04116920 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 rx=0x7f6df8002f70 tx=0x7f6df8015000).stop 2022-06-29T16:05:17.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.610+0000 7f6e0b4b4700 1 -- 172.21.15.60:0/3994984848 shutdown_connections 2022-06-29T16:05:17.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.610+0000 7f6e0b4b4700 1 --2- 172.21.15.60:0/3994984848 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6de40539e0 0x7f6de4055ea0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:17.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.610+0000 7f6e0b4b4700 1 --2- 172.21.15.60:0/3994984848 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e040b2130 0x7f6e040630c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:17.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.610+0000 7f6e0b4b4700 1 --2- 172.21.15.60:0/3994984848 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e040aed00 0x7f6e04116920 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:17.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.610+0000 7f6e0b4b4700 1 --2- 172.21.15.60:0/3994984848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e0405d070 0x7f6e040656e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:17.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.610+0000 7f6e0b4b4700 1 -- 172.21.15.60:0/3994984848 >> 172.21.15.60:0/3994984848 conn(0x7f6e040b5890 msgr2=0x7f6e040b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:17.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.610+0000 7f6e0b4b4700 1 -- 172.21.15.60:0/3994984848 shutdown_connections 2022-06-29T16:05:17.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.610+0000 7f6e0b4b4700 1 -- 172.21.15.60:0/3994984848 wait complete. 2022-06-29T16:05:17.621 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 35 2022-06-29T16:05:17.636 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:05:17.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.730+0000 7f4976f7c700 1 Processor -- start 2022-06-29T16:05:17.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.730+0000 7f4976f7c700 1 -- start start 2022-06-29T16:05:17.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.734+0000 7f4976f7c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f497013a920 0x7f4970143f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:17.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.734+0000 7f4976f7c700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f49700ae270 0x7f49700b5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:17.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.734+0000 7f4976f7c700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f497013ad00 con 0x7f49700b5b00 2022-06-29T16:05:17.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.734+0000 7f4976f7c700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4970144420 con 0x7f497013a920 2022-06-29T16:05:17.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.734+0000 7f4976f7c700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4970144670 con 0x7f49700ae270 2022-06-29T16:05:17.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.734+0000 7f4974d18700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f497013a920 0x7f4970143f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:17.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.734+0000 7f4975519700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f49700b5b00 0x7f4970140ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58162/0 (socket says 172.21.15.60:58162) 2022-06-29T16:05:17.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.734+0000 7f4975519700 1 -- 172.21.15.60:0/3259654437 learned_addr learned my addr 172.21.15.60:0/3259654437 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:17.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.734+0000 7f496ffff700 1 --2- 172.21.15.60:0/3259654437 >> v2:172.21.15.92:3300/0 conn(0x7f49700ae270 0x7f49700b5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:17.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.734+0000 7f4974d18700 1 -- 172.21.15.60:0/3259654437 >> v2:172.21.15.92:3300/0 conn(0x7f49700ae270 msgr2=0x7f49700b5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:17.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.734+0000 7f4974d18700 1 --2- 172.21.15.60:0/3259654437 >> v2:172.21.15.92:3300/0 conn(0x7f49700ae270 0x7f49700b5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:17.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.734+0000 7f4974d18700 1 -- 172.21.15.60:0/3259654437 >> v1:172.21.15.60:6789/0 conn(0x7f49700b5b00 legacy=0x7f4970140ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:17.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.734+0000 7f4974d18700 1 -- 172.21.15.60:0/3259654437 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4970144910 con 0x7f497013a920 2022-06-29T16:05:17.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.734+0000 7f496ffff700 1 --2- 172.21.15.60:0/3259654437 >> v2:172.21.15.92:3300/0 conn(0x7f49700ae270 0x7f49700b5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:17.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.734+0000 7f4974d18700 1 --2- 172.21.15.60:0/3259654437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f497013a920 0x7f4970143f30 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 rx=0x7f4960002f70 tx=0x7f4960006df0).ready entity=mon.0 client_cookie=84a889ee974f3e0f server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:17.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.734+0000 7f496f7fe700 1 -- 172.21.15.60:0/3259654437 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f496000d5c0 con 0x7f497013a920 2022-06-29T16:05:17.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.734+0000 7f496f7fe700 1 -- 172.21.15.60:0/3259654437 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4960000e90 con 0x7f497013a920 2022-06-29T16:05:17.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.734+0000 7f496f7fe700 1 -- 172.21.15.60:0/3259654437 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f496000d910 con 0x7f497013a920 2022-06-29T16:05:17.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.734+0000 7f4976f7c700 1 -- 172.21.15.60:0/3259654437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f497013a920 msgr2=0x7f4970143f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:17.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.734+0000 7f4976f7c700 1 --2- 172.21.15.60:0/3259654437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f497013a920 0x7f4970143f30 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 rx=0x7f4960002f70 tx=0x7f4960006df0).stop 2022-06-29T16:05:17.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.734+0000 7f4976f7c700 1 -- 172.21.15.60:0/3259654437 shutdown_connections 2022-06-29T16:05:17.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.734+0000 7f4976f7c700 1 --2- 172.21.15.60:0/3259654437 >> v2:172.21.15.92:3300/0 conn(0x7f49700ae270 0x7f49700b5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:17.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.734+0000 7f4976f7c700 1 --2- 172.21.15.60:0/3259654437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f497013a920 0x7f4970143f30 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:17.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.734+0000 7f4976f7c700 1 -- 172.21.15.60:0/3259654437 >> 172.21.15.60:0/3259654437 conn(0x7f49700632c0 msgr2=0x7f49700b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:17.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.734+0000 7f4976f7c700 1 -- 172.21.15.60:0/3259654437 shutdown_connections 2022-06-29T16:05:17.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.734+0000 7f4976f7c700 1 -- 172.21.15.60:0/3259654437 wait complete. 2022-06-29T16:05:17.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.738+0000 7f4976f7c700 1 Processor -- start 2022-06-29T16:05:17.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.738+0000 7f4976f7c700 1 -- start start 2022-06-29T16:05:17.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.738+0000 7f4976f7c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49700ae270 0x7f497013b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:17.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.738+0000 7f4976f7c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49700b5b00 0x7f497010e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:17.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.738+0000 7f4976f7c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f497013a920 0x7f4970143350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:17.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.738+0000 7f4976f7c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497013dd40 con 0x7f49700ae270 2022-06-29T16:05:17.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.738+0000 7f4976f7c700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497010e5e0 con 0x7f49700b5b00 2022-06-29T16:05:17.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.738+0000 7f4976f7c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f497010e750 con 0x7f497013a920 2022-06-29T16:05:17.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.738+0000 7f4974d18700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49700ae270 0x7f497013b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:17.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.738+0000 7f4974d18700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49700ae270 0x7f497013b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50080/0 (socket says 172.21.15.60:50080) 2022-06-29T16:05:17.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.738+0000 7f4974d18700 1 -- 172.21.15.60:0/2199387223 learned_addr learned my addr 172.21.15.60:0/2199387223 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:17.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.738+0000 7f4975519700 1 --2- 172.21.15.60:0/2199387223 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f497013a920 0x7f4970143350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:17.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.738+0000 7f496ffff700 1 --2- 172.21.15.60:0/2199387223 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49700b5b00 0x7f497010e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:17.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.738+0000 7f4974d18700 1 -- 172.21.15.60:0/2199387223 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f497013a920 msgr2=0x7f4970143350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:17.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.738+0000 7f4974d18700 1 --2- 172.21.15.60:0/2199387223 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f497013a920 0x7f4970143350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:17.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.738+0000 7f4974d18700 1 -- 172.21.15.60:0/2199387223 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49700b5b00 msgr2=0x7f497010e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:17.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.738+0000 7f4974d18700 1 --2- 172.21.15.60:0/2199387223 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49700b5b00 0x7f497010e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:17.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.738+0000 7f4974d18700 1 -- 172.21.15.60:0/2199387223 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f496000b180 con 0x7f49700ae270 2022-06-29T16:05:17.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.738+0000 7f4974d18700 1 --2- 172.21.15.60:0/2199387223 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49700ae270 0x7f497013b760 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 rx=0x7f4960002f70 tx=0x7f4960014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:17.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.738+0000 7f496dffb700 1 -- 172.21.15.60:0/2199387223 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f496001e400 con 0x7f49700ae270 2022-06-29T16:05:17.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.738+0000 7f496dffb700 1 -- 172.21.15.60:0/2199387223 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f496001e560 con 0x7f49700ae270 2022-06-29T16:05:17.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.738+0000 7f496dffb700 1 -- 172.21.15.60:0/2199387223 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f496001e8c0 con 0x7f49700ae270 2022-06-29T16:05:17.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.738+0000 7f4976f7c700 1 -- 172.21.15.60:0/2199387223 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49700b0cc0 con 0x7f49700ae270 2022-06-29T16:05:17.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.738+0000 7f4976f7c700 1 -- 172.21.15.60:0/2199387223 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49700b1370 con 0x7f49700ae270 2022-06-29T16:05:17.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.742+0000 7f4976f7c700 1 -- 172.21.15.60:0/2199387223 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4970144bb0 con 0x7f49700ae270 2022-06-29T16:05:17.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.742+0000 7f496dffb700 1 -- 172.21.15.60:0/2199387223 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f496001c070 con 0x7f49700ae270 2022-06-29T16:05:17.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.742+0000 7f496dffb700 1 --2- 172.21.15.60:0/2199387223 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4950053b30 0x7f4950055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:17.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.742+0000 7f496dffb700 1 -- 172.21.15.60:0/2199387223 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f496006ca80 con 0x7f49700ae270 2022-06-29T16:05:17.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.742+0000 7f496ffff700 1 --2- 172.21.15.60:0/2199387223 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4950053b30 0x7f4950055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:17.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.742+0000 7f496ffff700 1 --2- 172.21.15.60:0/2199387223 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4950053b30 0x7f4950055ff0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f496400c000 tx=0x7f4964008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:17.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.746+0000 7f496dffb700 1 -- 172.21.15.60:0/2199387223 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4960043950 con 0x7f49700ae270 2022-06-29T16:05:17.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.870+0000 7f4976f7c700 1 -- 172.21.15.60:0/2199387223 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49700573d0 con 0x7f49700ae270 2022-06-29T16:05:17.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.878+0000 7f496dffb700 1 -- 172.21.15.60:0/2199387223 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16170 (secure 0 0 0) 0x7f4960023070 con 0x7f49700ae270 2022-06-29T16:05:17.885 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:17.885 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":81,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:05:16.667913+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:05:14.715486+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2022-06-29T16:05:15.306355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{"172.21.15.92:6835/2464413229":"2022-06-30T16:05:16.667864+0000","172.21.15.92:6834/2464413229":"2022-06-30T16:05:16.667864+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:05:17.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.882+0000 7f4976f7c700 1 -- 172.21.15.60:0/2199387223 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4950053b30 msgr2=0x7f4950055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:17.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.882+0000 7f4976f7c700 1 --2- 172.21.15.60:0/2199387223 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4950053b30 0x7f4950055ff0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f496400c000 tx=0x7f4964008040).stop 2022-06-29T16:05:17.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.882+0000 7f4976f7c700 1 -- 172.21.15.60:0/2199387223 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49700ae270 msgr2=0x7f497013b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:17.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.882+0000 7f4976f7c700 1 --2- 172.21.15.60:0/2199387223 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49700ae270 0x7f497013b760 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 rx=0x7f4960002f70 tx=0x7f4960014000).stop 2022-06-29T16:05:17.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.882+0000 7f4976f7c700 1 -- 172.21.15.60:0/2199387223 shutdown_connections 2022-06-29T16:05:17.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.882+0000 7f4976f7c700 1 --2- 172.21.15.60:0/2199387223 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4950053b30 0x7f4950055ff0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:17.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.882+0000 7f4976f7c700 1 --2- 172.21.15.60:0/2199387223 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f497013a920 0x7f4970143350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:17.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.882+0000 7f4976f7c700 1 --2- 172.21.15.60:0/2199387223 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49700b5b00 0x7f497010e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:17.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.882+0000 7f4976f7c700 1 --2- 172.21.15.60:0/2199387223 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f49700ae270 0x7f497013b760 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:17.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.882+0000 7f4976f7c700 1 -- 172.21.15.60:0/2199387223 >> 172.21.15.60:0/2199387223 conn(0x7f49700632c0 msgr2=0x7f4970067010 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:17.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.882+0000 7f4976f7c700 1 -- 172.21.15.60:0/2199387223 shutdown_connections 2022-06-29T16:05:17.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:17.882+0000 7f4976f7c700 1 -- 172.21.15.60:0/2199387223 wait complete. 2022-06-29T16:05:17.902 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T16:05:18.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.010+0000 7fc72b59e700 1 Processor -- start 2022-06-29T16:05:18.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.010+0000 7fc72b59e700 1 -- start start 2022-06-29T16:05:18.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.010+0000 7fc72b59e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc72c05fb00 0x7fc72c13fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:18.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.010+0000 7fc72b59e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc72c064320 0x7fc72c0647b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:18.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.010+0000 7fc72b59e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc72c143fe0 con 0x7fc72c062aa0 2022-06-29T16:05:18.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.010+0000 7fc72b59e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc72c0b15d0 con 0x7fc72c064320 2022-06-29T16:05:18.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.010+0000 7fc72b59e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc72c05fee0 con 0x7fc72c05fb00 2022-06-29T16:05:18.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.010+0000 7fc729d9b700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc72c062aa0 0x7fc72c140250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58174/0 (socket says 172.21.15.60:58174) 2022-06-29T16:05:18.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.010+0000 7fc729d9b700 1 -- 172.21.15.60:0/3524538106 learned_addr learned my addr 172.21.15.60:0/3524538106 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:18.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.010+0000 7fc72ad9d700 1 --2- 172.21.15.60:0/3524538106 >> v2:172.21.15.60:3300/0 conn(0x7fc72c064320 0x7fc72c0647b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:18.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.010+0000 7fc72a59c700 1 --2- 172.21.15.60:0/3524538106 >> v2:172.21.15.92:3300/0 conn(0x7fc72c05fb00 0x7fc72c13fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:18.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.010+0000 7fc72959a700 1 -- 172.21.15.60:0/3524538106 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4002029921 0 0) 0x7fc72c143fe0 con 0x7fc72c062aa0 2022-06-29T16:05:18.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.010+0000 7fc72959a700 1 -- 172.21.15.60:0/3524538106 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc710003260 con 0x7fc72c062aa0 2022-06-29T16:05:18.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.010+0000 7fc72ad9d700 1 -- 172.21.15.60:0/3524538106 >> v2:172.21.15.92:3300/0 conn(0x7fc72c05fb00 msgr2=0x7fc72c13fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:18.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.010+0000 7fc72ad9d700 1 --2- 172.21.15.60:0/3524538106 >> v2:172.21.15.92:3300/0 conn(0x7fc72c05fb00 0x7fc72c13fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:18.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.010+0000 7fc72ad9d700 1 -- 172.21.15.60:0/3524538106 >> v1:172.21.15.60:6789/0 conn(0x7fc72c062aa0 legacy=0x7fc72c140250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:18.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.010+0000 7fc72ad9d700 1 -- 172.21.15.60:0/3524538106 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72c067740 con 0x7fc72c064320 2022-06-29T16:05:18.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.014+0000 7fc72ad9d700 1 --2- 172.21.15.60:0/3524538106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc72c064320 0x7fc72c0647b0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0x7fc72c0b02b0 tx=0x7fc7240091d0).ready entity=mon.0 client_cookie=4a23ab432ef41d03 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:18.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.014+0000 7fc72959a700 1 -- 172.21.15.60:0/3524538106 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc72400b5a0 con 0x7fc72c064320 2022-06-29T16:05:18.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.014+0000 7fc72959a700 1 -- 172.21.15.60:0/3524538106 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc72400b700 con 0x7fc72c064320 2022-06-29T16:05:18.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.014+0000 7fc72959a700 1 -- 172.21.15.60:0/3524538106 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc72400b870 con 0x7fc72c064320 2022-06-29T16:05:18.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.014+0000 7fc72b59e700 1 -- 172.21.15.60:0/3524538106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc72c064320 msgr2=0x7fc72c0647b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:18.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.014+0000 7fc72b59e700 1 --2- 172.21.15.60:0/3524538106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc72c064320 0x7fc72c0647b0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0x7fc72c0b02b0 tx=0x7fc7240091d0).stop 2022-06-29T16:05:18.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.014+0000 7fc72b59e700 1 -- 172.21.15.60:0/3524538106 shutdown_connections 2022-06-29T16:05:18.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.014+0000 7fc72b59e700 1 --2- 172.21.15.60:0/3524538106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc72c064320 0x7fc72c0647b0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:18.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.014+0000 7fc72b59e700 1 --2- 172.21.15.60:0/3524538106 >> v2:172.21.15.92:3300/0 conn(0x7fc72c05fb00 0x7fc72c13fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:18.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.014+0000 7fc72b59e700 1 -- 172.21.15.60:0/3524538106 >> 172.21.15.60:0/3524538106 conn(0x7fc72c063ab0 msgr2=0x7fc72c05f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:18.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.014+0000 7fc72b59e700 1 -- 172.21.15.60:0/3524538106 shutdown_connections 2022-06-29T16:05:18.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.014+0000 7fc72b59e700 1 -- 172.21.15.60:0/3524538106 wait complete. 2022-06-29T16:05:18.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.014+0000 7fc72b59e700 1 Processor -- start 2022-06-29T16:05:18.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.014+0000 7fc72b59e700 1 -- start start 2022-06-29T16:05:18.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.014+0000 7fc72b59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc72c05fb00 0x7fc72c145460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:18.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.014+0000 7fc72b59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc72c062aa0 0x7fc72c147c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:18.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.014+0000 7fc72b59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc72c064320 0x7fc72c148120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:18.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.014+0000 7fc72b59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72c145d50 con 0x7fc72c064320 2022-06-29T16:05:18.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.014+0000 7fc72b59e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72c145ff0 con 0x7fc72c05fb00 2022-06-29T16:05:18.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.014+0000 7fc72b59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc72c146290 con 0x7fc72c062aa0 2022-06-29T16:05:18.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.014+0000 7fc72ad9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc72c064320 0x7fc72c148120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:18.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.014+0000 7fc729d9b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc72c062aa0 0x7fc72c147c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:18.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.014+0000 7fc729d9b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc72c062aa0 0x7fc72c147c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:34956/0 (socket says 172.21.15.60:34956) 2022-06-29T16:05:18.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.014+0000 7fc72ad9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc72c064320 0x7fc72c148120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50098/0 (socket says 172.21.15.60:50098) 2022-06-29T16:05:18.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.014+0000 7fc729d9b700 1 -- 172.21.15.60:0/3948626169 learned_addr learned my addr 172.21.15.60:0/3948626169 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:18.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.014+0000 7fc72a59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc72c05fb00 0x7fc72c145460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:18.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.014+0000 7fc729d9b700 1 -- 172.21.15.60:0/3948626169 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc72c05fb00 msgr2=0x7fc72c145460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:18.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.014+0000 7fc729d9b700 1 --2- 172.21.15.60:0/3948626169 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc72c05fb00 0x7fc72c145460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:18.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.014+0000 7fc729d9b700 1 -- 172.21.15.60:0/3948626169 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc72c064320 msgr2=0x7fc72c148120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:18.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.014+0000 7fc729d9b700 1 --2- 172.21.15.60:0/3948626169 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc72c064320 0x7fc72c148120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:18.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.014+0000 7fc729d9b700 1 -- 172.21.15.60:0/3948626169 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc724009470 con 0x7fc72c062aa0 2022-06-29T16:05:18.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.014+0000 7fc729d9b700 1 --2- 172.21.15.60:0/3948626169 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc72c062aa0 0x7fc72c147c30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fc71c001110 tx=0x7fc71c007a40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:18.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.018+0000 7fc71b7fe700 1 -- 172.21.15.60:0/3948626169 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc71c0070d0 con 0x7fc72c062aa0 2022-06-29T16:05:18.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.018+0000 7fc71b7fe700 1 -- 172.21.15.60:0/3948626169 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc71c007230 con 0x7fc72c062aa0 2022-06-29T16:05:18.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.018+0000 7fc71b7fe700 1 -- 172.21.15.60:0/3948626169 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc71c005610 con 0x7fc72c062aa0 2022-06-29T16:05:18.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.018+0000 7fc72b59e700 1 -- 172.21.15.60:0/3948626169 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc72c145a40 con 0x7fc72c062aa0 2022-06-29T16:05:18.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.018+0000 7fc72b59e700 1 -- 172.21.15.60:0/3948626169 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc72c143bc0 con 0x7fc72c062aa0 2022-06-29T16:05:18.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.018+0000 7fc71b7fe700 1 -- 172.21.15.60:0/3948626169 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc71c01a070 con 0x7fc72c062aa0 2022-06-29T16:05:18.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.018+0000 7fc72b59e700 1 -- 172.21.15.60:0/3948626169 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc72c143980 con 0x7fc72c062aa0 2022-06-29T16:05:18.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.018+0000 7fc71b7fe700 1 --2- 172.21.15.60:0/3948626169 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc708053a10 0x7fc708055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:18.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.018+0000 7fc71b7fe700 1 -- 172.21.15.60:0/3948626169 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fc71c06b280 con 0x7fc72c062aa0 2022-06-29T16:05:18.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.018+0000 7fc72a59c700 1 --2- 172.21.15.60:0/3948626169 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc708053a10 0x7fc708055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:18.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.018+0000 7fc72a59c700 1 --2- 172.21.15.60:0/3948626169 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc708053a10 0x7fc708055ed0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fc72000b000 tx=0x7fc720006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:18.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.022+0000 7fc71b7fe700 1 -- 172.21.15.60:0/3948626169 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc72c143980 con 0x7fc72c062aa0 2022-06-29T16:05:18.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.190+0000 7fc72b59e700 1 -- 172.21.15.60:0/3948626169 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fc72c143980 con 0x7fc72c062aa0 2022-06-29T16:05:18.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.202+0000 7fc71b7fe700 1 -- 172.21.15.60:0/3948626169 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v28) v1 ==== 184+0+0 (secure 0 0 0) 0x7fc72c143980 con 0x7fc72c062aa0 2022-06-29T16:05:18.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.202+0000 7fc72b59e700 1 -- 172.21.15.60:0/3948626169 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc708053a10 msgr2=0x7fc708055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:18.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.202+0000 7fc72b59e700 1 --2- 172.21.15.60:0/3948626169 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc708053a10 0x7fc708055ed0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fc72000b000 tx=0x7fc720006df0).stop 2022-06-29T16:05:18.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.202+0000 7fc72b59e700 1 -- 172.21.15.60:0/3948626169 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc72c062aa0 msgr2=0x7fc72c147c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:18.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.202+0000 7fc72b59e700 1 --2- 172.21.15.60:0/3948626169 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc72c062aa0 0x7fc72c147c30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fc71c001110 tx=0x7fc71c007a40).stop 2022-06-29T16:05:18.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.206+0000 7fc72b59e700 1 -- 172.21.15.60:0/3948626169 shutdown_connections 2022-06-29T16:05:18.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.206+0000 7fc72b59e700 1 --2- 172.21.15.60:0/3948626169 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc708053a10 0x7fc708055ed0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:18.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.206+0000 7fc72b59e700 1 --2- 172.21.15.60:0/3948626169 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc72c064320 0x7fc72c148120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:18.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.206+0000 7fc72b59e700 1 --2- 172.21.15.60:0/3948626169 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc72c062aa0 0x7fc72c147c30 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:18.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.206+0000 7fc72b59e700 1 --2- 172.21.15.60:0/3948626169 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc72c05fb00 0x7fc72c145460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:18.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.206+0000 7fc72b59e700 1 -- 172.21.15.60:0/3948626169 >> 172.21.15.60:0/3948626169 conn(0x7fc72c063ab0 msgr2=0x7fc72c05f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:18.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.206+0000 7fc72b59e700 1 -- 172.21.15.60:0/3948626169 shutdown_connections 2022-06-29T16:05:18.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.206+0000 7fc72b59e700 1 -- 172.21.15.60:0/3948626169 wait complete. 2022-06-29T16:05:18.216 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.0 2022-06-29T16:05:18.226 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T16:05:18.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.330+0000 7f6e16917700 1 Processor -- start 2022-06-29T16:05:18.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.330+0000 7f6e16917700 1 -- start start 2022-06-29T16:05:18.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.334+0000 7f6e16917700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6e100b0280 0x7f6e10146650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:18.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.334+0000 7f6e16917700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e10146b40 0x7f6e100b2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:18.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.334+0000 7f6e16917700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e10146fd0 con 0x7f6e100b5eb0 2022-06-29T16:05:18.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.334+0000 7f6e16917700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e100b6290 con 0x7f6e100b0280 2022-06-29T16:05:18.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.334+0000 7f6e16917700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e100b0ab0 con 0x7f6e10146b40 2022-06-29T16:05:18.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.334+0000 7f6e15915700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f6e100b5eb0 0x7f6e10142a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:34204/0 (socket says 172.21.15.60:34204) 2022-06-29T16:05:18.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.334+0000 7f6e15915700 1 -- 172.21.15.60:0/649452683 learned_addr learned my addr 172.21.15.60:0/649452683 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:18.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.334+0000 7f6e15114700 1 --2- 172.21.15.60:0/649452683 >> v2:172.21.15.92:3300/0 conn(0x7f6e100b0280 0x7f6e10146650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:18.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.334+0000 7f6e16116700 1 --2- 172.21.15.60:0/649452683 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e10146b40 0x7f6e100b2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:18.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.334+0000 7f6e14913700 1 -- 172.21.15.60:0/649452683 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2727058576 0 0) 0x7f6e00001110 con 0x7f6e100b5eb0 2022-06-29T16:05:18.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.334+0000 7f6e14913700 1 -- 172.21.15.60:0/649452683 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dfc003260 con 0x7f6e100b5eb0 2022-06-29T16:05:18.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.334+0000 7f6e16116700 1 -- 172.21.15.60:0/649452683 >> v2:172.21.15.92:3300/0 conn(0x7f6e100b0280 msgr2=0x7f6e10146650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:18.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.334+0000 7f6e16116700 1 --2- 172.21.15.60:0/649452683 >> v2:172.21.15.92:3300/0 conn(0x7f6e100b0280 0x7f6e10146650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:18.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.334+0000 7f6e16116700 1 -- 172.21.15.60:0/649452683 >> v1:172.21.15.92:6789/0 conn(0x7f6e100b5eb0 legacy=0x7f6e10142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:18.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.334+0000 7f6e16116700 1 -- 172.21.15.60:0/649452683 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e101461b0 con 0x7f6e10146b40 2022-06-29T16:05:18.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.334+0000 7f6e16116700 1 --2- 172.21.15.60:0/649452683 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e10146b40 0x7f6e100b2bb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f6e10143130 tx=0x7f6e080091d0).ready entity=mon.2 client_cookie=e487e8961b51b9e9 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:18.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.334+0000 7f6e14913700 1 -- 172.21.15.60:0/649452683 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e0800b600 con 0x7f6e10146b40 2022-06-29T16:05:18.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.334+0000 7f6e14913700 1 -- 172.21.15.60:0/649452683 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e0800b760 con 0x7f6e10146b40 2022-06-29T16:05:18.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.334+0000 7f6e14913700 1 -- 172.21.15.60:0/649452683 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e0800b8d0 con 0x7f6e10146b40 2022-06-29T16:05:18.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.334+0000 7f6e16917700 1 -- 172.21.15.60:0/649452683 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e10146b40 msgr2=0x7f6e100b2bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:18.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.334+0000 7f6e16917700 1 --2- 172.21.15.60:0/649452683 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e10146b40 0x7f6e100b2bb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f6e10143130 tx=0x7f6e080091d0).stop 2022-06-29T16:05:18.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.334+0000 7f6e16917700 1 -- 172.21.15.60:0/649452683 shutdown_connections 2022-06-29T16:05:18.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.334+0000 7f6e16917700 1 --2- 172.21.15.60:0/649452683 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e10146b40 0x7f6e100b2bb0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:18.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.334+0000 7f6e16917700 1 --2- 172.21.15.60:0/649452683 >> v2:172.21.15.92:3300/0 conn(0x7f6e100b0280 0x7f6e10146650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:18.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.334+0000 7f6e16917700 1 -- 172.21.15.60:0/649452683 >> 172.21.15.60:0/649452683 conn(0x7f6e10062090 msgr2=0x7f6e10062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:18.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.334+0000 7f6e16917700 1 -- 172.21.15.60:0/649452683 shutdown_connections 2022-06-29T16:05:18.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.334+0000 7f6e16917700 1 -- 172.21.15.60:0/649452683 wait complete. 2022-06-29T16:05:18.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.338+0000 7f6e16917700 1 Processor -- start 2022-06-29T16:05:18.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.338+0000 7f6e16917700 1 -- start start 2022-06-29T16:05:18.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.338+0000 7f6e16917700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e100b0280 0x7f6e1013ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:18.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.338+0000 7f6e16917700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e100b5eb0 0x7f6e1013dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:18.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.338+0000 7f6e16917700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e10146b40 0x7f6e10155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:18.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.338+0000 7f6e16917700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e10155370 con 0x7f6e100b5eb0 2022-06-29T16:05:18.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.338+0000 7f6e16917700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e10155610 con 0x7f6e100b0280 2022-06-29T16:05:18.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.338+0000 7f6e16917700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e10148c60 con 0x7f6e10146b40 2022-06-29T16:05:18.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.338+0000 7f6e15114700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e100b5eb0 0x7f6e1013dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:18.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.338+0000 7f6e16116700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e10146b40 0x7f6e10155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:18.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.338+0000 7f6e15114700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e100b5eb0 0x7f6e1013dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50110/0 (socket says 172.21.15.60:50110) 2022-06-29T16:05:18.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.338+0000 7f6e15114700 1 -- 172.21.15.60:0/2122134505 learned_addr learned my addr 172.21.15.60:0/2122134505 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:18.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.338+0000 7f6e16116700 1 --2- 172.21.15.60:0/2122134505 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e10146b40 0x7f6e10155ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:34974/0 (socket says 172.21.15.60:34974) 2022-06-29T16:05:18.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.338+0000 7f6e15915700 1 --2- 172.21.15.60:0/2122134505 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e100b0280 0x7f6e1013ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:18.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.338+0000 7f6e15114700 1 -- 172.21.15.60:0/2122134505 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e10146b40 msgr2=0x7f6e10155ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:18.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.338+0000 7f6e15114700 1 --2- 172.21.15.60:0/2122134505 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e10146b40 0x7f6e10155ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:18.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.338+0000 7f6e15114700 1 -- 172.21.15.60:0/2122134505 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e100b0280 msgr2=0x7f6e1013ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:18.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.338+0000 7f6e15114700 1 --2- 172.21.15.60:0/2122134505 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e100b0280 0x7f6e1013ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:18.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.338+0000 7f6e15114700 1 -- 172.21.15.60:0/2122134505 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e08009470 con 0x7f6e100b5eb0 2022-06-29T16:05:18.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.338+0000 7f6e16116700 1 --2- 172.21.15.60:0/2122134505 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e10146b40 0x7f6e10155ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:05:18.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.338+0000 7f6e15114700 1 --2- 172.21.15.60:0/2122134505 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e100b5eb0 0x7f6e1013dd40 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 rx=0x7f6e0400c000 tx=0x7f6e04006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:18.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.342+0000 7f6e0effd700 1 -- 172.21.15.60:0/2122134505 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e04010780 con 0x7f6e100b5eb0 2022-06-29T16:05:18.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.342+0000 7f6e0effd700 1 -- 172.21.15.60:0/2122134505 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e040108e0 con 0x7f6e100b5eb0 2022-06-29T16:05:18.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.342+0000 7f6e0effd700 1 -- 172.21.15.60:0/2122134505 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e04010c40 con 0x7f6e100b5eb0 2022-06-29T16:05:18.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.342+0000 7f6e16917700 1 -- 172.21.15.60:0/2122134505 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e10152d50 con 0x7f6e100b5eb0 2022-06-29T16:05:18.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.342+0000 7f6e16917700 1 -- 172.21.15.60:0/2122134505 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e10147170 con 0x7f6e100b5eb0 2022-06-29T16:05:18.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.342+0000 7f6e16917700 1 -- 172.21.15.60:0/2122134505 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e100573d0 con 0x7f6e100b5eb0 2022-06-29T16:05:18.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.342+0000 7f6e0effd700 1 -- 172.21.15.60:0/2122134505 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6e0401c460 con 0x7f6e100b5eb0 2022-06-29T16:05:18.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.342+0000 7f6e0effd700 1 --2- 172.21.15.60:0/2122134505 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6df0053b00 0x7f6df0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:18.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.342+0000 7f6e0effd700 1 -- 172.21.15.60:0/2122134505 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f6e0406c300 con 0x7f6e100b5eb0 2022-06-29T16:05:18.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.342+0000 7f6e15915700 1 --2- 172.21.15.60:0/2122134505 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6df0053b00 0x7f6df0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:18.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.346+0000 7f6e15915700 1 --2- 172.21.15.60:0/2122134505 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6df0053b00 0x7f6df0055fc0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f6e0000d000 tx=0x7f6e000078a0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:18.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.346+0000 7f6e0effd700 1 -- 172.21.15.60:0/2122134505 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6e04043170 con 0x7f6e100b5eb0 2022-06-29T16:05:18.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.514+0000 7f6e16917700 1 -- 172.21.15.60:0/2122134505 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f6e10067a30 con 0x7f6e100b5eb0 2022-06-29T16:05:18.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.522+0000 7f6e0effd700 1 -- 172.21.15.60:0/2122134505 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v29) v1 ==== 184+0+0 (secure 0 0 0) 0x7f6e04046790 con 0x7f6e100b5eb0 2022-06-29T16:05:18.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.522+0000 7f6e16917700 1 -- 172.21.15.60:0/2122134505 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6df0053b00 msgr2=0x7f6df0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:18.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.522+0000 7f6e16917700 1 --2- 172.21.15.60:0/2122134505 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6df0053b00 0x7f6df0055fc0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f6e0000d000 tx=0x7f6e000078a0).stop 2022-06-29T16:05:18.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.522+0000 7f6e16917700 1 -- 172.21.15.60:0/2122134505 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e100b5eb0 msgr2=0x7f6e1013dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:18.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.522+0000 7f6e16917700 1 --2- 172.21.15.60:0/2122134505 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e100b5eb0 0x7f6e1013dd40 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 rx=0x7f6e0400c000 tx=0x7f6e04006df0).stop 2022-06-29T16:05:18.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.522+0000 7f6e16917700 1 -- 172.21.15.60:0/2122134505 shutdown_connections 2022-06-29T16:05:18.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.522+0000 7f6e16917700 1 --2- 172.21.15.60:0/2122134505 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6df0053b00 0x7f6df0055fc0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:18.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.522+0000 7f6e16917700 1 --2- 172.21.15.60:0/2122134505 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6e10146b40 0x7f6e10155ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:18.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.522+0000 7f6e16917700 1 --2- 172.21.15.60:0/2122134505 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6e100b5eb0 0x7f6e1013dd40 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:18.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.522+0000 7f6e16917700 1 --2- 172.21.15.60:0/2122134505 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e100b0280 0x7f6e1013ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:18.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.522+0000 7f6e16917700 1 -- 172.21.15.60:0/2122134505 >> 172.21.15.60:0/2122134505 conn(0x7f6e10062090 msgr2=0x7f6e10143170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:18.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.522+0000 7f6e16917700 1 -- 172.21.15.60:0/2122134505 shutdown_connections 2022-06-29T16:05:18.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.522+0000 7f6e16917700 1 -- 172.21.15.60:0/2122134505 wait complete. 2022-06-29T16:05:18.534 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.1 2022-06-29T16:05:18.543 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T16:05:18.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.650+0000 7fa3d7a57700 1 Processor -- start 2022-06-29T16:05:18.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.650+0000 7fa3d7a57700 1 -- start start 2022-06-29T16:05:18.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.650+0000 7fa3d7a57700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa3d00aca20 0x7fa3d00ace00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:18.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.650+0000 7fa3d7a57700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa3d0146160 0x7fa3d0065cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:18.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.650+0000 7fa3d7a57700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3d0143da0 con 0x7fa3d00b24f0 2022-06-29T16:05:18.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.650+0000 7fa3d7a57700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3d00adf00 con 0x7fa3d0146160 2022-06-29T16:05:18.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.650+0000 7fa3d7a57700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3d005cf40 con 0x7fa3d00aca20 2022-06-29T16:05:18.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.650+0000 7fa3d5ff4700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa3d0146160 0x7fa3d0065cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:18.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.650+0000 7fa3d4ff2700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa3d00b24f0 0x7fa3d01468c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58202/0 (socket says 172.21.15.60:58202) 2022-06-29T16:05:18.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.650+0000 7fa3d4ff2700 1 -- 172.21.15.60:0/3448695188 learned_addr learned my addr 172.21.15.60:0/3448695188 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:18.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.650+0000 7fa3d57f3700 1 --2- 172.21.15.60:0/3448695188 >> v2:172.21.15.92:3300/0 conn(0x7fa3d00aca20 0x7fa3d00ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:18.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.654+0000 7fa3bffff700 1 -- 172.21.15.60:0/3448695188 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454677577 0 0) 0x7fa3d0143da0 con 0x7fa3d00b24f0 2022-06-29T16:05:18.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.654+0000 7fa3bffff700 1 -- 172.21.15.60:0/3448695188 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3b8003260 con 0x7fa3d00b24f0 2022-06-29T16:05:18.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.654+0000 7fa3d5ff4700 1 -- 172.21.15.60:0/3448695188 >> v2:172.21.15.92:3300/0 conn(0x7fa3d00aca20 msgr2=0x7fa3d00ace00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:18.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.654+0000 7fa3d5ff4700 1 --2- 172.21.15.60:0/3448695188 >> v2:172.21.15.92:3300/0 conn(0x7fa3d00aca20 0x7fa3d00ace00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:18.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.654+0000 7fa3d5ff4700 1 -- 172.21.15.60:0/3448695188 >> v1:172.21.15.60:6789/0 conn(0x7fa3d00b24f0 legacy=0x7fa3d01468c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:18.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.654+0000 7fa3d5ff4700 1 -- 172.21.15.60:0/3448695188 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d005d1e0 con 0x7fa3d0146160 2022-06-29T16:05:18.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.654+0000 7fa3d5ff4700 1 --2- 172.21.15.60:0/3448695188 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3d0146160 0x7fa3d0065cf0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 rx=0x7fa3c80032f0 tx=0x7fa3c80091d0).ready entity=mon.0 client_cookie=17a3d2f9aa51442b server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:18.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.654+0000 7fa3bffff700 1 -- 172.21.15.60:0/3448695188 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3c800b600 con 0x7fa3d0146160 2022-06-29T16:05:18.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.654+0000 7fa3bffff700 1 -- 172.21.15.60:0/3448695188 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c800b760 con 0x7fa3d0146160 2022-06-29T16:05:18.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.654+0000 7fa3bffff700 1 -- 172.21.15.60:0/3448695188 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3c800b8d0 con 0x7fa3d0146160 2022-06-29T16:05:18.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.654+0000 7fa3d7a57700 1 -- 172.21.15.60:0/3448695188 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3d0146160 msgr2=0x7fa3d0065cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:18.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.654+0000 7fa3d7a57700 1 --2- 172.21.15.60:0/3448695188 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3d0146160 0x7fa3d0065cf0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 rx=0x7fa3c80032f0 tx=0x7fa3c80091d0).stop 2022-06-29T16:05:18.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.654+0000 7fa3d7a57700 1 -- 172.21.15.60:0/3448695188 shutdown_connections 2022-06-29T16:05:18.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.654+0000 7fa3d7a57700 1 --2- 172.21.15.60:0/3448695188 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3d0146160 0x7fa3d0065cf0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:18.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.654+0000 7fa3d7a57700 1 --2- 172.21.15.60:0/3448695188 >> v2:172.21.15.92:3300/0 conn(0x7fa3d00aca20 0x7fa3d00ace00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:18.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.654+0000 7fa3d7a57700 1 -- 172.21.15.60:0/3448695188 >> 172.21.15.60:0/3448695188 conn(0x7fa3d00b7140 msgr2=0x7fa3d00b7520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:18.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.654+0000 7fa3d7a57700 1 -- 172.21.15.60:0/3448695188 shutdown_connections 2022-06-29T16:05:18.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.654+0000 7fa3d7a57700 1 -- 172.21.15.60:0/3448695188 wait complete. 2022-06-29T16:05:18.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.658+0000 7fa3d7a57700 1 Processor -- start 2022-06-29T16:05:18.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.658+0000 7fa3d7a57700 1 -- start start 2022-06-29T16:05:18.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.658+0000 7fa3d7a57700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3d00aca20 0x7fa3d00ae5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:18.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.658+0000 7fa3d7a57700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3d00b24f0 0x7fa3d013be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:18.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.658+0000 7fa3d7a57700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3d0146160 0x7fa3d013c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:18.672 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.658+0000 7fa3d7a57700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d00b0660 con 0x7fa3d0146160 2022-06-29T16:05:18.672 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.658+0000 7fa3d7a57700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d00b08c0 con 0x7fa3d00aca20 2022-06-29T16:05:18.672 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.658+0000 7fa3d7a57700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d00b6620 con 0x7fa3d00b24f0 2022-06-29T16:05:18.672 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.658+0000 7fa3d5ff4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3d0146160 0x7fa3d013c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:18.673 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.658+0000 7fa3d5ff4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3d0146160 0x7fa3d013c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50126/0 (socket says 172.21.15.60:50126) 2022-06-29T16:05:18.673 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.658+0000 7fa3d5ff4700 1 -- 172.21.15.60:0/841735076 learned_addr learned my addr 172.21.15.60:0/841735076 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:18.673 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.658+0000 7fa3d4ff2700 1 --2- 172.21.15.60:0/841735076 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3d00b24f0 0x7fa3d013be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:18.673 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.658+0000 7fa3d57f3700 1 --2- 172.21.15.60:0/841735076 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3d00aca20 0x7fa3d00ae5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:18.673 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.658+0000 7fa3d5ff4700 1 -- 172.21.15.60:0/841735076 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3d00b24f0 msgr2=0x7fa3d013be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:18.674 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.658+0000 7fa3d5ff4700 1 --2- 172.21.15.60:0/841735076 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3d00b24f0 0x7fa3d013be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:18.674 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.658+0000 7fa3d5ff4700 1 -- 172.21.15.60:0/841735076 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3d00aca20 msgr2=0x7fa3d00ae5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:18.674 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.658+0000 7fa3d5ff4700 1 --2- 172.21.15.60:0/841735076 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3d00aca20 0x7fa3d00ae5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:18.674 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.658+0000 7fa3d5ff4700 1 -- 172.21.15.60:0/841735076 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c8009470 con 0x7fa3d0146160 2022-06-29T16:05:18.675 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.658+0000 7fa3d57f3700 1 --2- 172.21.15.60:0/841735076 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3d00aca20 0x7fa3d00ae5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:18.675 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.658+0000 7fa3d5ff4700 1 --2- 172.21.15.60:0/841735076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3d0146160 0x7fa3d013c300 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0x7fa3c80032f0 tx=0x7fa3c8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:18.675 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.658+0000 7fa3be7fc700 1 -- 172.21.15.60:0/841735076 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3c8011c10 con 0x7fa3d0146160 2022-06-29T16:05:18.675 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.658+0000 7fa3be7fc700 1 -- 172.21.15.60:0/841735076 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c8011d70 con 0x7fa3d0146160 2022-06-29T16:05:18.676 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.658+0000 7fa3d7a57700 1 -- 172.21.15.60:0/841735076 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3d013c7f0 con 0x7fa3d0146160 2022-06-29T16:05:18.676 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.658+0000 7fa3be7fc700 1 -- 172.21.15.60:0/841735076 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3c801f5e0 con 0x7fa3d0146160 2022-06-29T16:05:18.676 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.658+0000 7fa3d7a57700 1 -- 172.21.15.60:0/841735076 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3d00614c0 con 0x7fa3d0146160 2022-06-29T16:05:18.676 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.662+0000 7fa3d7a57700 1 -- 172.21.15.60:0/841735076 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3d00573d0 con 0x7fa3d0146160 2022-06-29T16:05:18.677 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.662+0000 7fa3be7fc700 1 -- 172.21.15.60:0/841735076 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa3c8026030 con 0x7fa3d0146160 2022-06-29T16:05:18.678 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.662+0000 7fa3be7fc700 1 --2- 172.21.15.60:0/841735076 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa3b0053b00 0x7fa3b0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:18.678 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.662+0000 7fa3be7fc700 1 -- 172.21.15.60:0/841735076 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fa3c806ca90 con 0x7fa3d0146160 2022-06-29T16:05:18.678 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.662+0000 7fa3d57f3700 1 --2- 172.21.15.60:0/841735076 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa3b0053b00 0x7fa3b0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:18.678 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.662+0000 7fa3d57f3700 1 --2- 172.21.15.60:0/841735076 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa3b0053b00 0x7fa3b0055fc0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fa3c000b000 tx=0x7fa3c0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:18.679 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.670+0000 7fa3be7fc700 1 -- 172.21.15.60:0/841735076 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa3c8043890 con 0x7fa3d0146160 2022-06-29T16:05:18.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.834+0000 7fa3d7a57700 1 -- 172.21.15.60:0/841735076 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fa3d00b67f0 con 0x7fa3d0146160 2022-06-29T16:05:18.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.838+0000 7fa3be7fc700 1 -- 172.21.15.60:0/841735076 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v30) v1 ==== 184+0+0 (secure 0 0 0) 0x7fa3c80240e0 con 0x7fa3d0146160 2022-06-29T16:05:18.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.842+0000 7fa3d7a57700 1 -- 172.21.15.60:0/841735076 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa3b0053b00 msgr2=0x7fa3b0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:18.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.842+0000 7fa3d7a57700 1 --2- 172.21.15.60:0/841735076 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa3b0053b00 0x7fa3b0055fc0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fa3c000b000 tx=0x7fa3c0008040).stop 2022-06-29T16:05:18.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.842+0000 7fa3d7a57700 1 -- 172.21.15.60:0/841735076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3d0146160 msgr2=0x7fa3d013c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:18.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.842+0000 7fa3d7a57700 1 --2- 172.21.15.60:0/841735076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3d0146160 0x7fa3d013c300 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0x7fa3c80032f0 tx=0x7fa3c8016000).stop 2022-06-29T16:05:18.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.842+0000 7fa3d7a57700 1 -- 172.21.15.60:0/841735076 shutdown_connections 2022-06-29T16:05:18.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.842+0000 7fa3d7a57700 1 --2- 172.21.15.60:0/841735076 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa3b0053b00 0x7fa3b0055fc0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:18.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.842+0000 7fa3d7a57700 1 --2- 172.21.15.60:0/841735076 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3d0146160 0x7fa3d013c300 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:18.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.842+0000 7fa3d7a57700 1 --2- 172.21.15.60:0/841735076 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3d00b24f0 0x7fa3d013be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:18.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.842+0000 7fa3d7a57700 1 --2- 172.21.15.60:0/841735076 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3d00aca20 0x7fa3d00ae5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:18.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.842+0000 7fa3d7a57700 1 -- 172.21.15.60:0/841735076 >> 172.21.15.60:0/841735076 conn(0x7fa3d00b7140 msgr2=0x7fa3d0142be0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:18.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.846+0000 7fa3d7a57700 1 -- 172.21.15.60:0/841735076 shutdown_connections 2022-06-29T16:05:18.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.846+0000 7fa3d7a57700 1 -- 172.21.15.60:0/841735076 wait complete. 2022-06-29T16:05:18.855 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.2 2022-06-29T16:05:18.865 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:05:18.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.958+0000 7fdc4ea0f700 1 Processor -- start 2022-06-29T16:05:18.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.958+0000 7fdc4ea0f700 1 -- start start 2022-06-29T16:05:18.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.958+0000 7fdc4ea0f700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdc480ae7c0 0x7fdc48141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:18.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.958+0000 7fdc4ea0f700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdc480b46c0 0x7fdc480b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:18.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.958+0000 7fdc4ea0f700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc48143a40 con 0x7fdc480b4010 2022-06-29T16:05:18.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.958+0000 7fdc4ea0f700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc480aee60 con 0x7fdc480ae7c0 2022-06-29T16:05:18.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.958+0000 7fdc4ea0f700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc480af100 con 0x7fdc480b46c0 2022-06-29T16:05:18.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.958+0000 7fdc477fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fdc480b4010 0x7fdc48142490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58216/0 (socket says 172.21.15.60:58216) 2022-06-29T16:05:18.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.958+0000 7fdc477fe700 1 -- 172.21.15.60:0/2727961686 learned_addr learned my addr 172.21.15.60:0/2727961686 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:18.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.958+0000 7fdc47fff700 1 --2- 172.21.15.60:0/2727961686 >> v2:172.21.15.60:3300/0 conn(0x7fdc480ae7c0 0x7fdc48141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:18.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.958+0000 7fdc4cfac700 1 --2- 172.21.15.60:0/2727961686 >> v2:172.21.15.92:3300/0 conn(0x7fdc480b46c0 0x7fdc480b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:18.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.958+0000 7fdc46ffd700 1 -- 172.21.15.60:0/2727961686 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3305661729 0 0) 0x7fdc48143a40 con 0x7fdc480b4010 2022-06-29T16:05:18.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.958+0000 7fdc46ffd700 1 -- 172.21.15.60:0/2727961686 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc34003260 con 0x7fdc480b4010 2022-06-29T16:05:18.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.958+0000 7fdc47fff700 1 -- 172.21.15.60:0/2727961686 >> v2:172.21.15.92:3300/0 conn(0x7fdc480b46c0 msgr2=0x7fdc480b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:18.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.958+0000 7fdc47fff700 1 --2- 172.21.15.60:0/2727961686 >> v2:172.21.15.92:3300/0 conn(0x7fdc480b46c0 0x7fdc480b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:18.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.958+0000 7fdc47fff700 1 -- 172.21.15.60:0/2727961686 >> v1:172.21.15.60:6789/0 conn(0x7fdc480b4010 legacy=0x7fdc48142490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:18.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.958+0000 7fdc47fff700 1 -- 172.21.15.60:0/2727961686 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc480aeba0 con 0x7fdc480ae7c0 2022-06-29T16:05:18.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.958+0000 7fdc47fff700 1 --2- 172.21.15.60:0/2727961686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc480ae7c0 0x7fdc48141fa0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 rx=0x7fdc3c002f70 tx=0x7fdc3c006df0).ready entity=mon.0 client_cookie=d40d2d4509fccef2 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:18.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.958+0000 7fdc46ffd700 1 -- 172.21.15.60:0/2727961686 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc3c00d4f0 con 0x7fdc480ae7c0 2022-06-29T16:05:18.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.958+0000 7fdc46ffd700 1 -- 172.21.15.60:0/2727961686 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc3c000e90 con 0x7fdc480ae7c0 2022-06-29T16:05:18.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.958+0000 7fdc46ffd700 1 -- 172.21.15.60:0/2727961686 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc3c00d840 con 0x7fdc480ae7c0 2022-06-29T16:05:18.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.962+0000 7fdc4ea0f700 1 -- 172.21.15.60:0/2727961686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc480ae7c0 msgr2=0x7fdc48141fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:18.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.962+0000 7fdc4ea0f700 1 --2- 172.21.15.60:0/2727961686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc480ae7c0 0x7fdc48141fa0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 rx=0x7fdc3c002f70 tx=0x7fdc3c006df0).stop 2022-06-29T16:05:18.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.962+0000 7fdc4ea0f700 1 -- 172.21.15.60:0/2727961686 shutdown_connections 2022-06-29T16:05:18.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.962+0000 7fdc4ea0f700 1 --2- 172.21.15.60:0/2727961686 >> v2:172.21.15.92:3300/0 conn(0x7fdc480b46c0 0x7fdc480b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:18.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.962+0000 7fdc4ea0f700 1 --2- 172.21.15.60:0/2727961686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc480ae7c0 0x7fdc48141fa0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:18.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.962+0000 7fdc4ea0f700 1 -- 172.21.15.60:0/2727961686 >> 172.21.15.60:0/2727961686 conn(0x7fdc480b9340 msgr2=0x7fdc480b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:18.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.962+0000 7fdc4ea0f700 1 -- 172.21.15.60:0/2727961686 shutdown_connections 2022-06-29T16:05:18.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.962+0000 7fdc4ea0f700 1 -- 172.21.15.60:0/2727961686 wait complete. 2022-06-29T16:05:18.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.962+0000 7fdc4ea0f700 1 Processor -- start 2022-06-29T16:05:18.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.962+0000 7fdc4ea0f700 1 -- start start 2022-06-29T16:05:18.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.966+0000 7fdc4ea0f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc480ae7c0 0x7fdc480b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:18.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.966+0000 7fdc4ea0f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc480b4010 0x7fdc4813f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:18.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.966+0000 7fdc4ea0f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc480b46c0 0x7fdc4813fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:18.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.966+0000 7fdc4ea0f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc48146830 con 0x7fdc480b46c0 2022-06-29T16:05:18.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.966+0000 7fdc4ea0f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc480b0e70 con 0x7fdc480ae7c0 2022-06-29T16:05:18.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.966+0000 7fdc4ea0f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc48146c60 con 0x7fdc480b4010 2022-06-29T16:05:18.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.966+0000 7fdc4cfac700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc480b46c0 0x7fdc4813fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:18.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.966+0000 7fdc4cfac700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc480b46c0 0x7fdc4813fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50140/0 (socket says 172.21.15.60:50140) 2022-06-29T16:05:18.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.966+0000 7fdc4cfac700 1 -- 172.21.15.60:0/230342432 learned_addr learned my addr 172.21.15.60:0/230342432 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:18.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.966+0000 7fdc47fff700 1 --2- 172.21.15.60:0/230342432 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc480ae7c0 0x7fdc480b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:18.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.966+0000 7fdc477fe700 1 --2- 172.21.15.60:0/230342432 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc480b4010 0x7fdc4813f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:18.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.966+0000 7fdc4cfac700 1 -- 172.21.15.60:0/230342432 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc480b4010 msgr2=0x7fdc4813f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:18.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.966+0000 7fdc4cfac700 1 --2- 172.21.15.60:0/230342432 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc480b4010 0x7fdc4813f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:18.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.966+0000 7fdc4cfac700 1 -- 172.21.15.60:0/230342432 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc480ae7c0 msgr2=0x7fdc480b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:18.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.966+0000 7fdc4cfac700 1 --2- 172.21.15.60:0/230342432 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc480ae7c0 0x7fdc480b29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:18.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.966+0000 7fdc4cfac700 1 -- 172.21.15.60:0/230342432 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc3c00b180 con 0x7fdc480b46c0 2022-06-29T16:05:18.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.966+0000 7fdc4cfac700 1 --2- 172.21.15.60:0/230342432 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc480b46c0 0x7fdc4813fa10 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0x7fdc40006e00 tx=0x7fdc400091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:18.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.966+0000 7fdc457fa700 1 -- 172.21.15.60:0/230342432 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc40012600 con 0x7fdc480b46c0 2022-06-29T16:05:18.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.966+0000 7fdc457fa700 1 -- 172.21.15.60:0/230342432 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc40012760 con 0x7fdc480b46c0 2022-06-29T16:05:18.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.966+0000 7fdc4ea0f700 1 -- 172.21.15.60:0/230342432 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc480b10d0 con 0x7fdc480b46c0 2022-06-29T16:05:18.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.966+0000 7fdc457fa700 1 -- 172.21.15.60:0/230342432 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc40012ac0 con 0x7fdc480b46c0 2022-06-29T16:05:18.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.966+0000 7fdc4ea0f700 1 -- 172.21.15.60:0/230342432 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc4813ff00 con 0x7fdc480b46c0 2022-06-29T16:05:18.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.966+0000 7fdc4ea0f700 1 -- 172.21.15.60:0/230342432 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc481469a0 con 0x7fdc480b46c0 2022-06-29T16:05:18.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.970+0000 7fdc457fa700 1 -- 172.21.15.60:0/230342432 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdc40022070 con 0x7fdc480b46c0 2022-06-29T16:05:18.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.970+0000 7fdc457fa700 1 --2- 172.21.15.60:0/230342432 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdc28053aa0 0x7fdc28055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:18.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.970+0000 7fdc457fa700 1 -- 172.21.15.60:0/230342432 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fdc4006c5a0 con 0x7fdc480b46c0 2022-06-29T16:05:18.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.970+0000 7fdc47fff700 1 --2- 172.21.15.60:0/230342432 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdc28053aa0 0x7fdc28055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:18.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.970+0000 7fdc47fff700 1 --2- 172.21.15.60:0/230342432 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdc28053aa0 0x7fdc28055f60 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7fdc3c01a000 tx=0x7fdc3c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:18.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:18.974+0000 7fdc457fa700 1 -- 172.21.15.60:0/230342432 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdc481469a0 con 0x7fdc480b46c0 2022-06-29T16:05:19.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.142+0000 7fdc4ea0f700 1 -- 172.21.15.60:0/230342432 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc48146970 con 0x7fdc480b46c0 2022-06-29T16:05:19.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.142+0000 7fdc457fa700 1 -- 172.21.15.60:0/230342432 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+2141 (secure 0 0 0) 0x7fdc40016020 con 0x7fdc480b46c0 2022-06-29T16:05:19.151 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:19.152 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":36,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6657,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.60:6833/3121204394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3121204394},{"type":"v1","addr":"172.21.15.60:6833","nonce":3121204394}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":34}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:05:16.293932+0000","modified":"2022-06-29T16:05:18.324125+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6688},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6688":{"gid":6688,"name":"b","rank":0,"incarnation":35,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/1245503283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1245503283},{"type":"v1","addr":"172.21.15.92:6835","nonce":1245503283}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-06-29T16:05:19.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.146+0000 7fdc4ea0f700 1 -- 172.21.15.60:0/230342432 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdc28053aa0 msgr2=0x7fdc28055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:19.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.146+0000 7fdc4ea0f700 1 --2- 172.21.15.60:0/230342432 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdc28053aa0 0x7fdc28055f60 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7fdc3c01a000 tx=0x7fdc3c014000).stop 2022-06-29T16:05:19.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.146+0000 7fdc4ea0f700 1 -- 172.21.15.60:0/230342432 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc480b46c0 msgr2=0x7fdc4813fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:19.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.146+0000 7fdc4ea0f700 1 --2- 172.21.15.60:0/230342432 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc480b46c0 0x7fdc4813fa10 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0x7fdc40006e00 tx=0x7fdc400091d0).stop 2022-06-29T16:05:19.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.146+0000 7fdc4ea0f700 1 -- 172.21.15.60:0/230342432 shutdown_connections 2022-06-29T16:05:19.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.146+0000 7fdc4ea0f700 1 --2- 172.21.15.60:0/230342432 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdc28053aa0 0x7fdc28055f60 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:19.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.146+0000 7fdc4ea0f700 1 --2- 172.21.15.60:0/230342432 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc480b46c0 0x7fdc4813fa10 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:19.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.146+0000 7fdc4ea0f700 1 --2- 172.21.15.60:0/230342432 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc480b4010 0x7fdc4813f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:19.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.146+0000 7fdc4ea0f700 1 --2- 172.21.15.60:0/230342432 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc480ae7c0 0x7fdc480b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:19.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.146+0000 7fdc4ea0f700 1 -- 172.21.15.60:0/230342432 >> 172.21.15.60:0/230342432 conn(0x7fdc480b9340 msgr2=0x7fdc4805cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:19.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.146+0000 7fdc4ea0f700 1 -- 172.21.15.60:0/230342432 shutdown_connections 2022-06-29T16:05:19.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.146+0000 7fdc4ea0f700 1 -- 172.21.15.60:0/230342432 wait complete. 2022-06-29T16:05:19.158 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 36 2022-06-29T16:05:19.372 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 36, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:05:16.293932+0000', 'modified': '2022-06-29T16:05:18.324125+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 81, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6688}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6688': {'gid': 6688, 'name': 'b', 'rank': 0, 'incarnation': 35, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.92:6835/1245503283', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 1245503283}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 1245503283}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:05:19.373 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:05:19.373 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-06-29T16:05:19.489 INFO:teuthology.orchestra.run.smithi092.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":6688,"want_state":"up:active","state":"up:active","rank_uptime":2.163744978,"mdsmap_epoch":36,"osdmap_epoch":81,"osdmap_epoch_barrier":81,"uptime":6.8529614409999997} 2022-06-29T16:05:19.490 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 6688, "mdsmap_epoch": 36, "osdmap_epoch": 81, "osdmap_epoch_barrier": 81, "rank_uptime": 2.163744978, "state": "up:active", "uptime": 6.852961441, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:05:19.491 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:05:19.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.590+0000 7f2eb7fff700 1 Processor -- start 2022-06-29T16:05:19.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.594+0000 7f2eb7fff700 1 -- start start 2022-06-29T16:05:19.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.594+0000 7f2eb7fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2eb80b28e0 0x7f2eb80b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:19.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.594+0000 7f2eb7fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2eb8065450 0x7f2eb8140180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:19.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.594+0000 7f2eb7fff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb8140670 con 0x7f2eb8060760 2022-06-29T16:05:19.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.594+0000 7f2eb7fff700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2eb813fa40 con 0x7f2eb80b28e0 2022-06-29T16:05:19.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.594+0000 7f2eb7fff700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2eb813fce0 con 0x7f2eb8065450 2022-06-29T16:05:19.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.594+0000 7f2eb6ffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2eb80b28e0 0x7f2eb80b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:19.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.594+0000 7f2eb77fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2eb8060760 0x7f2eb8139340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58234/0 (socket says 172.21.15.60:58234) 2022-06-29T16:05:19.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.594+0000 7f2eb6ffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2eb80b28e0 0x7f2eb80b2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50144/0 (socket says 172.21.15.60:50144) 2022-06-29T16:05:19.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.594+0000 7f2eb77fe700 1 -- 172.21.15.60:0/4200114426 learned_addr learned my addr 172.21.15.60:0/4200114426 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:19.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.594+0000 7f2eaffff700 1 --2- 172.21.15.60:0/4200114426 >> v2:172.21.15.92:3300/0 conn(0x7f2eb8065450 0x7f2eb8140180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:19.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.594+0000 7f2eb6ffd700 1 -- 172.21.15.60:0/4200114426 >> v2:172.21.15.92:3300/0 conn(0x7f2eb8065450 msgr2=0x7f2eb8140180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:19.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.594+0000 7f2eb6ffd700 1 --2- 172.21.15.60:0/4200114426 >> v2:172.21.15.92:3300/0 conn(0x7f2eb8065450 0x7f2eb8140180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:19.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.594+0000 7f2eb6ffd700 1 -- 172.21.15.60:0/4200114426 >> v1:172.21.15.60:6789/0 conn(0x7f2eb8060760 legacy=0x7f2eb8139340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:19.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.594+0000 7f2eb67fc700 1 -- 172.21.15.60:0/4200114426 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3770590062 0 0) 0x7f2eb8140670 con 0x7f2eb8060760 2022-06-29T16:05:19.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.594+0000 7f2eb6ffd700 1 -- 172.21.15.60:0/4200114426 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb8140860 con 0x7f2eb80b28e0 2022-06-29T16:05:19.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.594+0000 7f2eb6ffd700 1 --2- 172.21.15.60:0/4200114426 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2eb80b28e0 0x7f2eb80b2cc0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0x7f2ea8002f70 tx=0x7f2ea8006df0).ready entity=mon.0 client_cookie=85c8530dae6a3f2c server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:19.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.594+0000 7f2eb67fc700 1 -- 172.21.15.60:0/4200114426 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ea800d570 con 0x7f2eb80b28e0 2022-06-29T16:05:19.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.594+0000 7f2eb67fc700 1 -- 172.21.15.60:0/4200114426 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea8000e90 con 0x7f2eb80b28e0 2022-06-29T16:05:19.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.594+0000 7f2eb67fc700 1 -- 172.21.15.60:0/4200114426 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ea800d8c0 con 0x7f2eb80b28e0 2022-06-29T16:05:19.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.598+0000 7f2eb7fff700 1 -- 172.21.15.60:0/4200114426 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2eb80b28e0 msgr2=0x7f2eb80b2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:19.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.598+0000 7f2eb7fff700 1 --2- 172.21.15.60:0/4200114426 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2eb80b28e0 0x7f2eb80b2cc0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0x7f2ea8002f70 tx=0x7f2ea8006df0).stop 2022-06-29T16:05:19.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.598+0000 7f2eb7fff700 1 -- 172.21.15.60:0/4200114426 shutdown_connections 2022-06-29T16:05:19.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.598+0000 7f2eb7fff700 1 --2- 172.21.15.60:0/4200114426 >> v2:172.21.15.92:3300/0 conn(0x7f2eb8065450 0x7f2eb8140180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:19.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.598+0000 7f2eb7fff700 1 --2- 172.21.15.60:0/4200114426 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2eb80b28e0 0x7f2eb80b2cc0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:19.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.598+0000 7f2eb7fff700 1 -- 172.21.15.60:0/4200114426 >> 172.21.15.60:0/4200114426 conn(0x7f2eb8061c80 msgr2=0x7f2eb80637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:19.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.598+0000 7f2eb7fff700 1 -- 172.21.15.60:0/4200114426 shutdown_connections 2022-06-29T16:05:19.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.598+0000 7f2eb7fff700 1 -- 172.21.15.60:0/4200114426 wait complete. 2022-06-29T16:05:19.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.598+0000 7f2eb7fff700 1 Processor -- start 2022-06-29T16:05:19.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.598+0000 7f2eb7fff700 1 -- start start 2022-06-29T16:05:19.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.598+0000 7f2eb7fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2eb8060760 0x7f2eb81467a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:19.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.598+0000 7f2eb7fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2eb8065450 0x7f2eb80b51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:19.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.598+0000 7f2eb7fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2eb80b28e0 0x7f2eb80afa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:19.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.598+0000 7f2eb7fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb80aff30 con 0x7f2eb8065450 2022-06-29T16:05:19.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.598+0000 7f2eb7fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb80b0150 con 0x7f2eb8060760 2022-06-29T16:05:19.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.598+0000 7f2eb7fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb80ad6a0 con 0x7f2eb80b28e0 2022-06-29T16:05:19.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.598+0000 7f2eaffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2eb8065450 0x7f2eb80b51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:19.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.598+0000 7f2eaffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2eb8065450 0x7f2eb80b51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50152/0 (socket says 172.21.15.60:50152) 2022-06-29T16:05:19.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.598+0000 7f2eb6ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2eb8060760 0x7f2eb81467a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:19.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.598+0000 7f2eaffff700 1 -- 172.21.15.60:0/2199134686 learned_addr learned my addr 172.21.15.60:0/2199134686 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:19.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.598+0000 7f2eb77fe700 1 --2- 172.21.15.60:0/2199134686 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2eb80b28e0 0x7f2eb80afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:19.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.602+0000 7f2eaffff700 1 -- 172.21.15.60:0/2199134686 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2eb80b28e0 msgr2=0x7f2eb80afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:19.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.602+0000 7f2eaffff700 1 --2- 172.21.15.60:0/2199134686 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2eb80b28e0 0x7f2eb80afa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:19.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.602+0000 7f2eaffff700 1 -- 172.21.15.60:0/2199134686 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2eb8060760 msgr2=0x7f2eb81467a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:19.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.602+0000 7f2eaffff700 1 --2- 172.21.15.60:0/2199134686 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2eb8060760 0x7f2eb81467a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:19.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.602+0000 7f2eaffff700 1 -- 172.21.15.60:0/2199134686 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea800b180 con 0x7f2eb8065450 2022-06-29T16:05:19.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.602+0000 7f2eaffff700 1 --2- 172.21.15.60:0/2199134686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2eb8065450 0x7f2eb80b51d0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0x7f2ea000c000 tx=0x7f2ea0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:19.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.602+0000 7f2eb4ff9700 1 -- 172.21.15.60:0/2199134686 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ea0010780 con 0x7f2eb8065450 2022-06-29T16:05:19.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.602+0000 7f2eb4ff9700 1 -- 172.21.15.60:0/2199134686 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea00108e0 con 0x7f2eb8065450 2022-06-29T16:05:19.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.602+0000 7f2eb7fff700 1 -- 172.21.15.60:0/2199134686 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eb80ad940 con 0x7f2eb8065450 2022-06-29T16:05:19.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.602+0000 7f2eb4ff9700 1 -- 172.21.15.60:0/2199134686 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ea0010c40 con 0x7f2eb8065450 2022-06-29T16:05:19.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.602+0000 7f2eb7fff700 1 -- 172.21.15.60:0/2199134686 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eb80ae000 con 0x7f2eb8065450 2022-06-29T16:05:19.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.602+0000 7f2eb7fff700 1 -- 172.21.15.60:0/2199134686 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eb80573d0 con 0x7f2eb8065450 2022-06-29T16:05:19.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.602+0000 7f2eb4ff9700 1 -- 172.21.15.60:0/2199134686 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2ea001c460 con 0x7f2eb8065450 2022-06-29T16:05:19.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.606+0000 7f2eb4ff9700 1 --2- 172.21.15.60:0/2199134686 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2e9c053a70 0x7f2e9c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:19.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.606+0000 7f2eb4ff9700 1 -- 172.21.15.60:0/2199134686 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f2ea006c2f0 con 0x7f2eb8065450 2022-06-29T16:05:19.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.606+0000 7f2eb6ffd700 1 --2- 172.21.15.60:0/2199134686 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2e9c053a70 0x7f2e9c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:19.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.606+0000 7f2eb6ffd700 1 --2- 172.21.15.60:0/2199134686 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2e9c053a70 0x7f2e9c055f30 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f2ea801a000 tx=0x7f2ea8014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:19.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.610+0000 7f2eb4ff9700 1 -- 172.21.15.60:0/2199134686 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2ea0043160 con 0x7f2eb8065450 2022-06-29T16:05:19.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.738+0000 7f2eb7fff700 1 -- 172.21.15.60:0/2199134686 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2eb8068480 con 0x7f2eb8065450 2022-06-29T16:05:19.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.738+0000 7f2eb4ff9700 1 -- 172.21.15.60:0/2199134686 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16170 (secure 0 0 0) 0x7f2ea0046780 con 0x7f2eb8065450 2022-06-29T16:05:19.748 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:19.748 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":81,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:05:16.667913+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:05:14.715486+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2022-06-29T16:05:15.306355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{"172.21.15.92:6835/2464413229":"2022-06-30T16:05:16.667864+0000","172.21.15.92:6834/2464413229":"2022-06-30T16:05:16.667864+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:05:19.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.742+0000 7f2eb7fff700 1 -- 172.21.15.60:0/2199134686 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2e9c053a70 msgr2=0x7f2e9c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:19.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.742+0000 7f2eb7fff700 1 --2- 172.21.15.60:0/2199134686 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2e9c053a70 0x7f2e9c055f30 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f2ea801a000 tx=0x7f2ea8014000).stop 2022-06-29T16:05:19.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.742+0000 7f2eb7fff700 1 -- 172.21.15.60:0/2199134686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2eb8065450 msgr2=0x7f2eb80b51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:19.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.742+0000 7f2eb7fff700 1 --2- 172.21.15.60:0/2199134686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2eb8065450 0x7f2eb80b51d0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0x7f2ea000c000 tx=0x7f2ea0006df0).stop 2022-06-29T16:05:19.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.742+0000 7f2eb7fff700 1 -- 172.21.15.60:0/2199134686 shutdown_connections 2022-06-29T16:05:19.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.742+0000 7f2eb7fff700 1 --2- 172.21.15.60:0/2199134686 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2e9c053a70 0x7f2e9c055f30 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:19.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.742+0000 7f2eb7fff700 1 --2- 172.21.15.60:0/2199134686 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2eb80b28e0 0x7f2eb80afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:19.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.742+0000 7f2eb7fff700 1 --2- 172.21.15.60:0/2199134686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2eb8065450 0x7f2eb80b51d0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:19.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.742+0000 7f2eb7fff700 1 --2- 172.21.15.60:0/2199134686 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2eb8060760 0x7f2eb81467a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:19.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.742+0000 7f2eb7fff700 1 -- 172.21.15.60:0/2199134686 >> 172.21.15.60:0/2199134686 conn(0x7f2eb8061c80 msgr2=0x7f2eb80b5e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:19.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.742+0000 7f2eb7fff700 1 -- 172.21.15.60:0/2199134686 shutdown_connections 2022-06-29T16:05:19.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.742+0000 7f2eb7fff700 1 -- 172.21.15.60:0/2199134686 wait complete. 2022-06-29T16:05:19.761 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-29T16:05:14.715486+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2022-06-29T16:05:15.306355+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-29T16:05:19.762 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:05:19.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.866+0000 7f610359e700 1 Processor -- start 2022-06-29T16:05:19.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.866+0000 7f610359e700 1 -- start start 2022-06-29T16:05:19.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.866+0000 7f610359e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6104140240 0x7f61040b2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:19.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.866+0000 7f610359e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61040ad520 0x7f610405bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:19.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.866+0000 7f610359e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61040633e0 con 0x7f61040acd60 2022-06-29T16:05:19.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.866+0000 7f610359e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61040b35d0 con 0x7f6104140240 2022-06-29T16:05:19.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.866+0000 7f610359e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61040ad140 con 0x7f61040ad520 2022-06-29T16:05:19.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.866+0000 7f610259c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f61040acd60 0x7f610413fb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58244/0 (socket says 172.21.15.60:58244) 2022-06-29T16:05:19.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.866+0000 7f610259c700 1 -- 172.21.15.60:0/445546678 learned_addr learned my addr 172.21.15.60:0/445546678 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:19.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.870+0000 7f6102d9d700 1 --2- 172.21.15.60:0/445546678 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61040ad520 0x7f610405bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:19.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.870+0000 7f60fbfff700 1 --2- 172.21.15.60:0/445546678 >> v2:172.21.15.92:3300/0 conn(0x7f6104140240 0x7f61040b2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:19.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.870+0000 7f6101d9b700 1 -- 172.21.15.60:0/445546678 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267414684 0 0) 0x7f61040633e0 con 0x7f61040acd60 2022-06-29T16:05:19.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.870+0000 7f6101d9b700 1 -- 172.21.15.60:0/445546678 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60e4003260 con 0x7f61040acd60 2022-06-29T16:05:19.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.870+0000 7f6102d9d700 1 -- 172.21.15.60:0/445546678 >> v2:172.21.15.92:3300/0 conn(0x7f6104140240 msgr2=0x7f61040b2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:19.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.870+0000 7f6102d9d700 1 --2- 172.21.15.60:0/445546678 >> v2:172.21.15.92:3300/0 conn(0x7f6104140240 0x7f61040b2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:19.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.870+0000 7f6102d9d700 1 -- 172.21.15.60:0/445546678 >> v1:172.21.15.60:6789/0 conn(0x7f61040acd60 legacy=0x7f610413fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:19.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.870+0000 7f6102d9d700 1 -- 172.21.15.60:0/445546678 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6104146030 con 0x7f61040ad520 2022-06-29T16:05:19.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.870+0000 7f6102d9d700 1 --2- 172.21.15.60:0/445546678 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61040ad520 0x7f610405bd20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f60fc0032f0 tx=0x7f60fc0091d0).ready entity=mon.2 client_cookie=f833a05f2a010ad6 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:19.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.870+0000 7f6101d9b700 1 -- 172.21.15.60:0/445546678 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60fc00b600 con 0x7f61040ad520 2022-06-29T16:05:19.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.870+0000 7f6101d9b700 1 -- 172.21.15.60:0/445546678 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60fc00b760 con 0x7f61040ad520 2022-06-29T16:05:19.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.870+0000 7f6101d9b700 1 -- 172.21.15.60:0/445546678 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60fc00b8d0 con 0x7f61040ad520 2022-06-29T16:05:19.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.870+0000 7f610359e700 1 -- 172.21.15.60:0/445546678 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61040ad520 msgr2=0x7f610405bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:19.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.870+0000 7f610359e700 1 --2- 172.21.15.60:0/445546678 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61040ad520 0x7f610405bd20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f60fc0032f0 tx=0x7f60fc0091d0).stop 2022-06-29T16:05:19.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.870+0000 7f610359e700 1 -- 172.21.15.60:0/445546678 shutdown_connections 2022-06-29T16:05:19.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.870+0000 7f610359e700 1 --2- 172.21.15.60:0/445546678 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61040ad520 0x7f610405bd20 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:19.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.870+0000 7f610359e700 1 --2- 172.21.15.60:0/445546678 >> v2:172.21.15.92:3300/0 conn(0x7f6104140240 0x7f61040b2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:19.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.870+0000 7f610359e700 1 -- 172.21.15.60:0/445546678 >> 172.21.15.60:0/445546678 conn(0x7f61040b6fb0 msgr2=0x7f61040b7390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:19.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.870+0000 7f610359e700 1 -- 172.21.15.60:0/445546678 shutdown_connections 2022-06-29T16:05:19.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.870+0000 7f610359e700 1 -- 172.21.15.60:0/445546678 wait complete. 2022-06-29T16:05:19.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.870+0000 7f610359e700 1 Processor -- start 2022-06-29T16:05:19.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.870+0000 7f610359e700 1 -- start start 2022-06-29T16:05:19.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.874+0000 7f610359e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61040acd60 0x7f6104151db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:19.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.874+0000 7f610359e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61040ad520 0x7f6104149d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:19.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.874+0000 7f610359e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6104140240 0x7f6104152520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:19.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.874+0000 7f610359e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6104141ac0 con 0x7f6104140240 2022-06-29T16:05:19.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.874+0000 7f610359e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6104141d30 con 0x7f61040acd60 2022-06-29T16:05:19.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.874+0000 7f610359e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61040ae510 con 0x7f61040ad520 2022-06-29T16:05:19.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.874+0000 7f6102d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6104140240 0x7f6104152520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:19.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.874+0000 7f6102d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6104140240 0x7f6104152520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50168/0 (socket says 172.21.15.60:50168) 2022-06-29T16:05:19.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.874+0000 7f6102d9d700 1 -- 172.21.15.60:0/1346608623 learned_addr learned my addr 172.21.15.60:0/1346608623 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:19.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.874+0000 7f610259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61040acd60 0x7f6104151db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:19.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.874+0000 7f60fbfff700 1 --2- 172.21.15.60:0/1346608623 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61040ad520 0x7f6104149d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:19.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.874+0000 7f6102d9d700 1 -- 172.21.15.60:0/1346608623 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61040ad520 msgr2=0x7f6104149d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:19.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.874+0000 7f6102d9d700 1 --2- 172.21.15.60:0/1346608623 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61040ad520 0x7f6104149d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:19.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.874+0000 7f6102d9d700 1 -- 172.21.15.60:0/1346608623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61040acd60 msgr2=0x7f6104151db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:19.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.874+0000 7f6102d9d700 1 --2- 172.21.15.60:0/1346608623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61040acd60 0x7f6104151db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:19.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.874+0000 7f6102d9d700 1 -- 172.21.15.60:0/1346608623 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60fc009470 con 0x7f6104140240 2022-06-29T16:05:19.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.874+0000 7f6102d9d700 1 --2- 172.21.15.60:0/1346608623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6104140240 0x7f6104152520 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7f60fc0032f0 tx=0x7f60fc016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:19.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.874+0000 7f60fb7fe700 1 -- 172.21.15.60:0/1346608623 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60fc00b600 con 0x7f6104140240 2022-06-29T16:05:19.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.874+0000 7f60fb7fe700 1 -- 172.21.15.60:0/1346608623 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60fc011e00 con 0x7f6104140240 2022-06-29T16:05:19.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.874+0000 7f610359e700 1 -- 172.21.15.60:0/1346608623 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61040ae8a0 con 0x7f6104140240 2022-06-29T16:05:19.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.874+0000 7f60fb7fe700 1 -- 172.21.15.60:0/1346608623 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60fc021630 con 0x7f6104140240 2022-06-29T16:05:19.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.874+0000 7f610359e700 1 -- 172.21.15.60:0/1346608623 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6104143b00 con 0x7f6104140240 2022-06-29T16:05:19.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.878+0000 7f610359e700 1 -- 172.21.15.60:0/1346608623 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61040573d0 con 0x7f6104140240 2022-06-29T16:05:19.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.878+0000 7f60fb7fe700 1 -- 172.21.15.60:0/1346608623 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f60fc027020 con 0x7f6104140240 2022-06-29T16:05:19.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.878+0000 7f60fb7fe700 1 --2- 172.21.15.60:0/1346608623 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f60e8053a70 0x7f60e8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:19.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.878+0000 7f60fb7fe700 1 -- 172.21.15.60:0/1346608623 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f60fc06d500 con 0x7f6104140240 2022-06-29T16:05:19.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.878+0000 7f610259c700 1 --2- 172.21.15.60:0/1346608623 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f60e8053a70 0x7f60e8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:19.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.878+0000 7f610259c700 1 --2- 172.21.15.60:0/1346608623 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f60e8053a70 0x7f60e8055f30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f60f400d000 tx=0x7f60f4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:19.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:19.886+0000 7f60fb7fe700 1 -- 172.21.15.60:0/1346608623 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f60fc044330 con 0x7f6104140240 2022-06-29T16:05:20.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.010+0000 7f610359e700 1 -- 172.21.15.60:0/1346608623 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f610413fdb0 con 0x7f6104140240 2022-06-29T16:05:20.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.014+0000 7f60fb7fe700 1 -- 172.21.15.60:0/1346608623 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v81) v1 ==== 110+0+10 (secure 0 0 0) 0x7f60fc0150b0 con 0x7f6104140240 2022-06-29T16:05:20.022 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:05:20.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.018+0000 7f610359e700 1 -- 172.21.15.60:0/1346608623 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f60e8053a70 msgr2=0x7f60e8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:20.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.018+0000 7f610359e700 1 --2- 172.21.15.60:0/1346608623 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f60e8053a70 0x7f60e8055f30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f60f400d000 tx=0x7f60f4009040).stop 2022-06-29T16:05:20.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.018+0000 7f610359e700 1 -- 172.21.15.60:0/1346608623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6104140240 msgr2=0x7f6104152520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:20.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.018+0000 7f610359e700 1 --2- 172.21.15.60:0/1346608623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6104140240 0x7f6104152520 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7f60fc0032f0 tx=0x7f60fc016000).stop 2022-06-29T16:05:20.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.018+0000 7f610359e700 1 -- 172.21.15.60:0/1346608623 shutdown_connections 2022-06-29T16:05:20.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.018+0000 7f610359e700 1 --2- 172.21.15.60:0/1346608623 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f60e8053a70 0x7f60e8055f30 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.018+0000 7f610359e700 1 --2- 172.21.15.60:0/1346608623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6104140240 0x7f6104152520 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.018+0000 7f610359e700 1 --2- 172.21.15.60:0/1346608623 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61040ad520 0x7f6104149d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.018+0000 7f610359e700 1 --2- 172.21.15.60:0/1346608623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61040acd60 0x7f6104151db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.018+0000 7f610359e700 1 -- 172.21.15.60:0/1346608623 >> 172.21.15.60:0/1346608623 conn(0x7f61040b6fb0 msgr2=0x7f61040b7390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:20.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.018+0000 7f610359e700 1 -- 172.21.15.60:0/1346608623 shutdown_connections 2022-06-29T16:05:20.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.018+0000 7f610359e700 1 -- 172.21.15.60:0/1346608623 wait complete. 2022-06-29T16:05:20.040 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:05:20.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.142+0000 7fb76cc35700 1 Processor -- start 2022-06-29T16:05:20.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.142+0000 7fb76cc35700 1 -- start start 2022-06-29T16:05:20.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.142+0000 7fb76cc35700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb7680acd60 0x7fb768144900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:20.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.142+0000 7fb76cc35700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb7680ad990 0x7fb7681296f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:20.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.142+0000 7fb76cc35700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb768143cf0 con 0x7fb7680657d0 2022-06-29T16:05:20.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.142+0000 7fb76cc35700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb768064a70 con 0x7fb7680ad990 2022-06-29T16:05:20.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.142+0000 7fb76cc35700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7680ad140 con 0x7fb7680acd60 2022-06-29T16:05:20.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.142+0000 7fb766ffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb7680ad990 0x7fb7681296f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:20.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.142+0000 7fb766ffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb7680ad990 0x7fb7681296f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50172/0 (socket says 172.21.15.60:50172) 2022-06-29T16:05:20.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.142+0000 7fb766ffd700 1 -- 172.21.15.60:0/1549619869 learned_addr learned my addr 172.21.15.60:0/1549619869 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:20.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.142+0000 7fb7677fe700 1 --2- 172.21.15.60:0/1549619869 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb7680acd60 0x7fb768144900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:20.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.142+0000 7fb766ffd700 1 -- 172.21.15.60:0/1549619869 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb7680acd60 msgr2=0x7fb768144900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:20.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.142+0000 7fb766ffd700 1 --2- 172.21.15.60:0/1549619869 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb7680acd60 0x7fb768144900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.142+0000 7fb766ffd700 1 -- 172.21.15.60:0/1549619869 >> v1:172.21.15.92:6789/0 conn(0x7fb7680657d0 legacy=0x7fb768129be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:20.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.142+0000 7fb766ffd700 1 -- 172.21.15.60:0/1549619869 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb768143f20 con 0x7fb7680ad990 2022-06-29T16:05:20.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.142+0000 7fb7677fe700 1 --2- 172.21.15.60:0/1549619869 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb7680acd60 0x7fb768144900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:20.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.142+0000 7fb766ffd700 1 --2- 172.21.15.60:0/1549619869 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7680ad990 0x7fb7681296f0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0x7fb75c002f70 tx=0x7fb75c006eb0).ready entity=mon.0 client_cookie=27240812c08c784b server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:20.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.142+0000 7fb7667fc700 1 -- 172.21.15.60:0/1549619869 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb75c00d5c0 con 0x7fb7680ad990 2022-06-29T16:05:20.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.142+0000 7fb7667fc700 1 -- 172.21.15.60:0/1549619869 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb75c000e90 con 0x7fb7680ad990 2022-06-29T16:05:20.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.142+0000 7fb7667fc700 1 -- 172.21.15.60:0/1549619869 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb75c00d950 con 0x7fb7680ad990 2022-06-29T16:05:20.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.146+0000 7fb76cc35700 1 -- 172.21.15.60:0/1549619869 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7680ad990 msgr2=0x7fb7681296f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:20.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.146+0000 7fb76cc35700 1 --2- 172.21.15.60:0/1549619869 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7680ad990 0x7fb7681296f0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0x7fb75c002f70 tx=0x7fb75c006eb0).stop 2022-06-29T16:05:20.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.146+0000 7fb76cc35700 1 -- 172.21.15.60:0/1549619869 shutdown_connections 2022-06-29T16:05:20.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.146+0000 7fb76cc35700 1 --2- 172.21.15.60:0/1549619869 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7680ad990 0x7fb7681296f0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.146+0000 7fb76cc35700 1 --2- 172.21.15.60:0/1549619869 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb7680acd60 0x7fb768144900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.146+0000 7fb76cc35700 1 -- 172.21.15.60:0/1549619869 >> 172.21.15.60:0/1549619869 conn(0x7fb7680b6f80 msgr2=0x7fb7680b7390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:20.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.146+0000 7fb76cc35700 1 -- 172.21.15.60:0/1549619869 shutdown_connections 2022-06-29T16:05:20.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.146+0000 7fb76cc35700 1 -- 172.21.15.60:0/1549619869 wait complete. 2022-06-29T16:05:20.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.146+0000 7fb76cc35700 1 Processor -- start 2022-06-29T16:05:20.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.146+0000 7fb76cc35700 1 -- start start 2022-06-29T16:05:20.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.146+0000 7fb76cc35700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7680657d0 0x7fb7681430b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:20.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.146+0000 7fb76cc35700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7680acd60 0x7fb7680af850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:20.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.146+0000 7fb76cc35700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb7680ad990 0x7fb768061c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:20.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.146+0000 7fb76cc35700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76812fa40 con 0x7fb7680acd60 2022-06-29T16:05:20.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.146+0000 7fb76cc35700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb768061520 con 0x7fb7680657d0 2022-06-29T16:05:20.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.146+0000 7fb76cc35700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7680617c0 con 0x7fb7680ad990 2022-06-29T16:05:20.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.146+0000 7fb766ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7680acd60 0x7fb7680af850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:20.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.146+0000 7fb766ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7680acd60 0x7fb7680af850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50180/0 (socket says 172.21.15.60:50180) 2022-06-29T16:05:20.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.146+0000 7fb766ffd700 1 -- 172.21.15.60:0/1754699599 learned_addr learned my addr 172.21.15.60:0/1754699599 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:20.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.146+0000 7fb7677fe700 1 --2- 172.21.15.60:0/1754699599 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7680657d0 0x7fb7681430b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:20.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.146+0000 7fb767fff700 1 --2- 172.21.15.60:0/1754699599 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb7680ad990 0x7fb768061c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:20.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.150+0000 7fb766ffd700 1 -- 172.21.15.60:0/1754699599 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb7680ad990 msgr2=0x7fb768061c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:20.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.150+0000 7fb766ffd700 1 --2- 172.21.15.60:0/1754699599 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb7680ad990 0x7fb768061c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.150+0000 7fb766ffd700 1 -- 172.21.15.60:0/1754699599 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7680657d0 msgr2=0x7fb7681430b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:20.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.150+0000 7fb766ffd700 1 --2- 172.21.15.60:0/1754699599 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7680657d0 0x7fb7681430b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.150+0000 7fb766ffd700 1 -- 172.21.15.60:0/1754699599 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb75c00b1e0 con 0x7fb7680acd60 2022-06-29T16:05:20.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.150+0000 7fb767fff700 1 --2- 172.21.15.60:0/1754699599 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb7680ad990 0x7fb768061c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:20.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.150+0000 7fb766ffd700 1 --2- 172.21.15.60:0/1754699599 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7680acd60 0x7fb7680af850 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0x7fb75c002f70 tx=0x7fb75c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:20.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.150+0000 7fb764ff9700 1 -- 172.21.15.60:0/1754699599 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb75c00d940 con 0x7fb7680acd60 2022-06-29T16:05:20.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.150+0000 7fb764ff9700 1 -- 172.21.15.60:0/1754699599 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb75c006b40 con 0x7fb7680acd60 2022-06-29T16:05:20.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.150+0000 7fb764ff9700 1 -- 172.21.15.60:0/1754699599 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb75c011de0 con 0x7fb7680acd60 2022-06-29T16:05:20.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.150+0000 7fb76cc35700 1 -- 172.21.15.60:0/1754699599 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb76812f6c0 con 0x7fb7680acd60 2022-06-29T16:05:20.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.150+0000 7fb76cc35700 1 -- 172.21.15.60:0/1754699599 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb768062ea0 con 0x7fb7680acd60 2022-06-29T16:05:20.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.150+0000 7fb76cc35700 1 -- 172.21.15.60:0/1754699599 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7680573d0 con 0x7fb7680acd60 2022-06-29T16:05:20.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.150+0000 7fb764ff9700 1 -- 172.21.15.60:0/1754699599 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb75c025030 con 0x7fb7680acd60 2022-06-29T16:05:20.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.150+0000 7fb764ff9700 1 --2- 172.21.15.60:0/1754699599 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb748053b00 0x7fb748055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:20.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.150+0000 7fb764ff9700 1 -- 172.21.15.60:0/1754699599 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fb75c06d150 con 0x7fb7680acd60 2022-06-29T16:05:20.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.150+0000 7fb7677fe700 1 --2- 172.21.15.60:0/1754699599 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb748053b00 0x7fb748055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:20.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.154+0000 7fb7677fe700 1 --2- 172.21.15.60:0/1754699599 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb748053b00 0x7fb748055fc0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fb75800c000 tx=0x7fb758008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:20.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.158+0000 7fb764ff9700 1 -- 172.21.15.60:0/1754699599 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb75c044020 con 0x7fb7680acd60 2022-06-29T16:05:20.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.286+0000 7fb76cc35700 1 -- 172.21.15.60:0/1754699599 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fb768061980 con 0x7fb7680acd60 2022-06-29T16:05:20.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.286+0000 7fb764ff9700 1 -- 172.21.15.60:0/1754699599 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v81) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb75c023020 con 0x7fb7680acd60 2022-06-29T16:05:20.295 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:05:20.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.290+0000 7fb76cc35700 1 -- 172.21.15.60:0/1754699599 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb748053b00 msgr2=0x7fb748055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:20.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.290+0000 7fb76cc35700 1 --2- 172.21.15.60:0/1754699599 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb748053b00 0x7fb748055fc0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fb75800c000 tx=0x7fb758008040).stop 2022-06-29T16:05:20.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.290+0000 7fb76cc35700 1 -- 172.21.15.60:0/1754699599 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7680acd60 msgr2=0x7fb7680af850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:20.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.290+0000 7fb76cc35700 1 --2- 172.21.15.60:0/1754699599 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7680acd60 0x7fb7680af850 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0x7fb75c002f70 tx=0x7fb75c015000).stop 2022-06-29T16:05:20.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.290+0000 7fb76cc35700 1 -- 172.21.15.60:0/1754699599 shutdown_connections 2022-06-29T16:05:20.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.290+0000 7fb76cc35700 1 --2- 172.21.15.60:0/1754699599 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb748053b00 0x7fb748055fc0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.290+0000 7fb76cc35700 1 --2- 172.21.15.60:0/1754699599 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb7680ad990 0x7fb768061c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.290+0000 7fb76cc35700 1 --2- 172.21.15.60:0/1754699599 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb7680acd60 0x7fb7680af850 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.290+0000 7fb76cc35700 1 --2- 172.21.15.60:0/1754699599 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7680657d0 0x7fb7681430b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.290+0000 7fb76cc35700 1 -- 172.21.15.60:0/1754699599 >> 172.21.15.60:0/1754699599 conn(0x7fb7680b6f80 msgr2=0x7fb7680b7360 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:20.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.290+0000 7fb76cc35700 1 -- 172.21.15.60:0/1754699599 shutdown_connections 2022-06-29T16:05:20.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.290+0000 7fb76cc35700 1 -- 172.21.15.60:0/1754699599 wait complete. 2022-06-29T16:05:20.308 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-29T16:05:20.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.406+0000 7fa1bbfff700 1 Processor -- start 2022-06-29T16:05:20.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.406+0000 7fa1bbfff700 1 -- start start 2022-06-29T16:05:20.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.406+0000 7fa1bbfff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa1bc05d070 0x7fa1bc0b1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:20.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.406+0000 7fa1bbfff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1bc05d450 con 0x7fa1bc0aee30 2022-06-29T16:05:20.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.406+0000 7fa1bbfff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1bc062200 con 0x7fa1bc141b70 2022-06-29T16:05:20.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.406+0000 7fa1bbfff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1bc143a90 con 0x7fa1bc05d070 2022-06-29T16:05:20.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.406+0000 7fa1ba7fc700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa1bc0aee30 0x7fa1bc13a460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58274/0 (socket says 172.21.15.60:58274) 2022-06-29T16:05:20.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.406+0000 7fa1ba7fc700 1 -- 172.21.15.60:0/2671186883 learned_addr learned my addr 172.21.15.60:0/2671186883 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:20.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.406+0000 7fa1baffd700 1 --2- 172.21.15.60:0/2671186883 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa1bc05d070 0x7fa1bc0b1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:20.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.410+0000 7fa1b9ffb700 1 -- 172.21.15.60:0/2671186883 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 71353787 0 0) 0x7fa1bc05d450 con 0x7fa1bc0aee30 2022-06-29T16:05:20.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.410+0000 7fa1b9ffb700 1 -- 172.21.15.60:0/2671186883 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa19c003260 con 0x7fa1bc0aee30 2022-06-29T16:05:20.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.410+0000 7fa1b9ffb700 1 -- 172.21.15.60:0/2671186883 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3303007680 0 0) 0x7fa1bc062200 con 0x7fa1bc141b70 2022-06-29T16:05:20.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.410+0000 7fa1b9ffb700 1 -- 172.21.15.60:0/2671186883 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1bc05d450 con 0x7fa1bc141b70 2022-06-29T16:05:20.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.410+0000 7fa1baffd700 1 -- 172.21.15.60:0/2671186883 >> v1:172.21.15.92:6789/0 conn(0x7fa1bc141b70 legacy=0x7fa1bc140bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:20.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.410+0000 7fa1baffd700 1 -- 172.21.15.60:0/2671186883 >> v1:172.21.15.60:6789/0 conn(0x7fa1bc0aee30 legacy=0x7fa1bc13a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:20.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.410+0000 7fa1baffd700 1 -- 172.21.15.60:0/2671186883 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1bc064dc0 con 0x7fa1bc05d070 2022-06-29T16:05:20.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.410+0000 7fa1baffd700 1 --2- 172.21.15.60:0/2671186883 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa1bc05d070 0x7fa1bc0b1f30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fa1ac002f70 tx=0x7fa1ac006df0).ready entity=mon.2 client_cookie=20e7076abb944955 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:20.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.410+0000 7fa1b9ffb700 1 -- 172.21.15.60:0/2671186883 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa1ac00d5c0 con 0x7fa1bc05d070 2022-06-29T16:05:20.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.410+0000 7fa1b9ffb700 1 -- 172.21.15.60:0/2671186883 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1ac000e90 con 0x7fa1bc05d070 2022-06-29T16:05:20.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.410+0000 7fa1b9ffb700 1 -- 172.21.15.60:0/2671186883 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa1ac00d910 con 0x7fa1bc05d070 2022-06-29T16:05:20.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.410+0000 7fa1bbfff700 1 -- 172.21.15.60:0/2671186883 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa1bc05d070 msgr2=0x7fa1bc0b1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:20.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.410+0000 7fa1bbfff700 1 --2- 172.21.15.60:0/2671186883 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa1bc05d070 0x7fa1bc0b1f30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fa1ac002f70 tx=0x7fa1ac006df0).stop 2022-06-29T16:05:20.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.410+0000 7fa1bbfff700 1 -- 172.21.15.60:0/2671186883 shutdown_connections 2022-06-29T16:05:20.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.410+0000 7fa1bbfff700 1 --2- 172.21.15.60:0/2671186883 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa1bc05d070 0x7fa1bc0b1f30 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.410+0000 7fa1bbfff700 1 -- 172.21.15.60:0/2671186883 >> 172.21.15.60:0/2671186883 conn(0x7fa1bc0b5890 msgr2=0x7fa1bc0b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:20.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.410+0000 7fa1bbfff700 1 -- 172.21.15.60:0/2671186883 shutdown_connections 2022-06-29T16:05:20.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.410+0000 7fa1bbfff700 1 -- 172.21.15.60:0/2671186883 wait complete. 2022-06-29T16:05:20.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.410+0000 7fa1bbfff700 1 Processor -- start 2022-06-29T16:05:20.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.410+0000 7fa1bbfff700 1 -- start start 2022-06-29T16:05:20.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.410+0000 7fa1bbfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1bc05d070 0x7fa1bc151cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:20.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.410+0000 7fa1bbfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa1bc0aee30 0x7fa1bc10e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:20.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.410+0000 7fa1bbfff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa1bc141b70 0x7fa1bc111f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:20.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.410+0000 7fa1bbfff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1bc152cc0 con 0x7fa1bc05d070 2022-06-29T16:05:20.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.410+0000 7fa1bbfff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1bc13a0b0 con 0x7fa1bc0aee30 2022-06-29T16:05:20.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.410+0000 7fa1bbfff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1bc152f00 con 0x7fa1bc141b70 2022-06-29T16:05:20.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.410+0000 7fa1baffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1bc05d070 0x7fa1bc151cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:20.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.410+0000 7fa1baffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1bc05d070 0x7fa1bc151cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50190/0 (socket says 172.21.15.60:50190) 2022-06-29T16:05:20.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.410+0000 7fa1baffd700 1 -- 172.21.15.60:0/1849307442 learned_addr learned my addr 172.21.15.60:0/1849307442 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:20.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.410+0000 7fa1ba7fc700 1 --2- 172.21.15.60:0/1849307442 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa1bc0aee30 0x7fa1bc10e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:20.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.410+0000 7fa1bb7fe700 1 --2- 172.21.15.60:0/1849307442 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa1bc141b70 0x7fa1bc111f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:20.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.414+0000 7fa1baffd700 1 -- 172.21.15.60:0/1849307442 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa1bc141b70 msgr2=0x7fa1bc111f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:20.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.414+0000 7fa1baffd700 1 --2- 172.21.15.60:0/1849307442 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa1bc141b70 0x7fa1bc111f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.414+0000 7fa1baffd700 1 -- 172.21.15.60:0/1849307442 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa1bc0aee30 msgr2=0x7fa1bc10e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:20.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.414+0000 7fa1baffd700 1 --2- 172.21.15.60:0/1849307442 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa1bc0aee30 0x7fa1bc10e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.414+0000 7fa1baffd700 1 -- 172.21.15.60:0/1849307442 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ac00b180 con 0x7fa1bc05d070 2022-06-29T16:05:20.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.414+0000 7fa1baffd700 1 --2- 172.21.15.60:0/1849307442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1bc05d070 0x7fa1bc151cc0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0x7fa1bc13a1f0 tx=0x7fa1ac014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:20.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.414+0000 7fa1b3fff700 1 -- 172.21.15.60:0/1849307442 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa1ac016070 con 0x7fa1bc05d070 2022-06-29T16:05:20.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.414+0000 7fa1b3fff700 1 -- 172.21.15.60:0/1849307442 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1ac0137d0 con 0x7fa1bc05d070 2022-06-29T16:05:20.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.414+0000 7fa1b3fff700 1 -- 172.21.15.60:0/1849307442 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa1ac013b30 con 0x7fa1bc05d070 2022-06-29T16:05:20.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.414+0000 7fa1bbfff700 1 -- 172.21.15.60:0/1849307442 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1bc1522e0 con 0x7fa1bc05d070 2022-06-29T16:05:20.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.414+0000 7fa1bbfff700 1 -- 172.21.15.60:0/1849307442 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1bc152820 con 0x7fa1bc05d070 2022-06-29T16:05:20.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.414+0000 7fa1bbfff700 1 -- 172.21.15.60:0/1849307442 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1bc1529e0 con 0x7fa1bc05d070 2022-06-29T16:05:20.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.414+0000 7fa1b3fff700 1 -- 172.21.15.60:0/1849307442 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa1ac00f030 con 0x7fa1bc05d070 2022-06-29T16:05:20.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.414+0000 7fa1b3fff700 1 --2- 172.21.15.60:0/1849307442 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa1a0053b00 0x7fa1a0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:20.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.414+0000 7fa1b3fff700 1 -- 172.21.15.60:0/1849307442 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fa1ac06cb10 con 0x7fa1bc05d070 2022-06-29T16:05:20.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.414+0000 7fa1ba7fc700 1 --2- 172.21.15.60:0/1849307442 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa1a0053b00 0x7fa1a0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:20.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.418+0000 7fa1ba7fc700 1 --2- 172.21.15.60:0/1849307442 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa1a0053b00 0x7fa1a0055fc0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7fa1a400d000 tx=0x7fa1a4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:20.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.422+0000 7fa1b3fff700 1 -- 172.21.15.60:0/1849307442 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa1ac043940 con 0x7fa1bc05d070 2022-06-29T16:05:20.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.550+0000 7fa1bbfff700 1 -- 172.21.15.60:0/1849307442 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa1bc0573d0 con 0x7fa1bc05d070 2022-06-29T16:05:20.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.550+0000 7fa1b3fff700 1 -- 172.21.15.60:0/1849307442 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) v1 ==== 104+0+10 (secure 0 0 0) 0x7fa1ac0230a0 con 0x7fa1bc05d070 2022-06-29T16:05:20.559 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:05:20.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.554+0000 7fa1bbfff700 1 -- 172.21.15.60:0/1849307442 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa1a0053b00 msgr2=0x7fa1a0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:20.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.554+0000 7fa1bbfff700 1 --2- 172.21.15.60:0/1849307442 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa1a0053b00 0x7fa1a0055fc0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7fa1a400d000 tx=0x7fa1a4009040).stop 2022-06-29T16:05:20.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.554+0000 7fa1bbfff700 1 -- 172.21.15.60:0/1849307442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1bc05d070 msgr2=0x7fa1bc151cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:20.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.554+0000 7fa1bbfff700 1 --2- 172.21.15.60:0/1849307442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1bc05d070 0x7fa1bc151cc0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0x7fa1bc13a1f0 tx=0x7fa1ac014000).stop 2022-06-29T16:05:20.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.554+0000 7fa1bbfff700 1 -- 172.21.15.60:0/1849307442 shutdown_connections 2022-06-29T16:05:20.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.554+0000 7fa1bbfff700 1 --2- 172.21.15.60:0/1849307442 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa1a0053b00 0x7fa1a0055fc0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.554+0000 7fa1bbfff700 1 --2- 172.21.15.60:0/1849307442 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa1bc141b70 0x7fa1bc111f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.554+0000 7fa1bbfff700 1 --2- 172.21.15.60:0/1849307442 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa1bc0aee30 0x7fa1bc10e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.554+0000 7fa1bbfff700 1 --2- 172.21.15.60:0/1849307442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa1bc05d070 0x7fa1bc151cc0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.554+0000 7fa1bbfff700 1 -- 172.21.15.60:0/1849307442 >> 172.21.15.60:0/1849307442 conn(0x7fa1bc0b5890 msgr2=0x7fa1bc13a440 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:20.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.554+0000 7fa1bbfff700 1 -- 172.21.15.60:0/1849307442 shutdown_connections 2022-06-29T16:05:20.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.554+0000 7fa1bbfff700 1 -- 172.21.15.60:0/1849307442 wait complete. 2022-06-29T16:05:20.572 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-29T16:05:20.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.678+0000 7f657615f700 1 Processor -- start 2022-06-29T16:05:20.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.678+0000 7f657615f700 1 -- start start 2022-06-29T16:05:20.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.678+0000 7f657615f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f65700b48b0 0x7f6570142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:20.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.678+0000 7f657615f700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f65700b2c50 0x7f65700b3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:20.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.678+0000 7f657615f700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65700b0cf0 con 0x7f65700b5eb0 2022-06-29T16:05:20.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.678+0000 7f657615f700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65700b54e0 con 0x7f65700b2c50 2022-06-29T16:05:20.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.678+0000 7f657615f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65700b4d10 con 0x7f65700b48b0 2022-06-29T16:05:20.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.678+0000 7f657595e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f65700b2c50 0x7f65700b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:20.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.678+0000 7f657595e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f65700b2c50 0x7f65700b3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50202/0 (socket says 172.21.15.60:50202) 2022-06-29T16:05:20.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.682+0000 7f657595e700 1 -- 172.21.15.60:0/75567995 learned_addr learned my addr 172.21.15.60:0/75567995 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:20.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.678+0000 7f657515d700 1 --1- 172.21.15.60:0/75567995 >> v1:172.21.15.60:6789/0 conn(0x7f65700b5eb0 0x7f65700b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58284/0 (socket says 172.21.15.60:58284) 2022-06-29T16:05:20.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.682+0000 7f657495c700 1 --2- 172.21.15.60:0/75567995 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f65700b48b0 0x7f6570142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:20.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.682+0000 7f657595e700 1 -- 172.21.15.60:0/75567995 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f65700b48b0 msgr2=0x7f6570142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:20.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.682+0000 7f657595e700 1 --2- 172.21.15.60:0/75567995 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f65700b48b0 0x7f6570142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.682+0000 7f6563fff700 1 -- 172.21.15.60:0/75567995 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2096041264 0 0) 0x7f6564001110 con 0x7f65700b5eb0 2022-06-29T16:05:20.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.682+0000 7f657595e700 1 -- 172.21.15.60:0/75567995 >> v1:172.21.15.60:6789/0 conn(0x7f65700b5eb0 legacy=0x7f65700b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:20.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.682+0000 7f657595e700 1 -- 172.21.15.60:0/75567995 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65700b6290 con 0x7f65700b2c50 2022-06-29T16:05:20.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.682+0000 7f657595e700 1 --2- 172.21.15.60:0/75567995 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65700b2c50 0x7f65700b3090 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0x7f65680032f0 tx=0x7f6568009290).ready entity=mon.0 client_cookie=8dc4e84ebdcdaf27 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:20.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.682+0000 7f6563fff700 1 -- 172.21.15.60:0/75567995 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f656800b8a0 con 0x7f65700b2c50 2022-06-29T16:05:20.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.682+0000 7f6563fff700 1 -- 172.21.15.60:0/75567995 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f656800ba00 con 0x7f65700b2c50 2022-06-29T16:05:20.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.682+0000 7f6563fff700 1 -- 172.21.15.60:0/75567995 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f656800bb70 con 0x7f65700b2c50 2022-06-29T16:05:20.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.682+0000 7f657615f700 1 -- 172.21.15.60:0/75567995 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65700b2c50 msgr2=0x7f65700b3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:20.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.682+0000 7f657615f700 1 --2- 172.21.15.60:0/75567995 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65700b2c50 0x7f65700b3090 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0x7f65680032f0 tx=0x7f6568009290).stop 2022-06-29T16:05:20.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.682+0000 7f657615f700 1 -- 172.21.15.60:0/75567995 shutdown_connections 2022-06-29T16:05:20.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.682+0000 7f657615f700 1 --2- 172.21.15.60:0/75567995 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65700b2c50 0x7f65700b3090 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.682+0000 7f657615f700 1 --2- 172.21.15.60:0/75567995 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f65700b48b0 0x7f6570142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.682+0000 7f657615f700 1 -- 172.21.15.60:0/75567995 >> 172.21.15.60:0/75567995 conn(0x7f6570062090 msgr2=0x7f6570062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:20.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.682+0000 7f657615f700 1 -- 172.21.15.60:0/75567995 shutdown_connections 2022-06-29T16:05:20.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.682+0000 7f657615f700 1 -- 172.21.15.60:0/75567995 wait complete. 2022-06-29T16:05:20.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.682+0000 7f657615f700 1 Processor -- start 2022-06-29T16:05:20.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.682+0000 7f657615f700 1 -- start start 2022-06-29T16:05:20.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.686+0000 7f657615f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f65700b2c50 0x7f65701461f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:20.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.686+0000 7f657615f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f65700b48b0 0x7f6570151ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:20.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.686+0000 7f657615f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65700b5eb0 0x7f6570152090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:20.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.686+0000 7f657615f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65701447c0 con 0x7f65700b5eb0 2022-06-29T16:05:20.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.686+0000 7f657615f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6570145a90 con 0x7f65700b48b0 2022-06-29T16:05:20.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.686+0000 7f657615f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65701449f0 con 0x7f65700b2c50 2022-06-29T16:05:20.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.686+0000 7f657515d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f65700b2c50 0x7f65701461f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:20.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.686+0000 7f657595e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65700b5eb0 0x7f6570152090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:20.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.686+0000 7f657495c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f65700b48b0 0x7f6570151ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:20.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.686+0000 7f657515d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f65700b2c50 0x7f65701461f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:35066/0 (socket says 172.21.15.60:35066) 2022-06-29T16:05:20.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.686+0000 7f657515d700 1 -- 172.21.15.60:0/3502831779 learned_addr learned my addr 172.21.15.60:0/3502831779 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:20.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.686+0000 7f657595e700 1 -- 172.21.15.60:0/3502831779 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f65700b2c50 msgr2=0x7f65701461f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:20.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.686+0000 7f657595e700 1 --2- 172.21.15.60:0/3502831779 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f65700b2c50 0x7f65701461f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.686+0000 7f657595e700 1 -- 172.21.15.60:0/3502831779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f65700b48b0 msgr2=0x7f6570151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:20.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.686+0000 7f657595e700 1 --2- 172.21.15.60:0/3502831779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f65700b48b0 0x7f6570151ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.686+0000 7f657595e700 1 -- 172.21.15.60:0/3502831779 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6568009530 con 0x7f65700b5eb0 2022-06-29T16:05:20.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.686+0000 7f657495c700 1 --2- 172.21.15.60:0/3502831779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f65700b48b0 0x7f6570151ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:05:20.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.686+0000 7f657515d700 1 --2- 172.21.15.60:0/3502831779 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f65700b2c50 0x7f65701461f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:05:20.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.686+0000 7f657595e700 1 --2- 172.21.15.60:0/3502831779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65700b5eb0 0x7f6570152090 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0x7f6570145c20 tx=0x7f6568016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:20.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.686+0000 7f65627fc700 1 -- 172.21.15.60:0/3502831779 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f656800bb70 con 0x7f65700b5eb0 2022-06-29T16:05:20.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.686+0000 7f65627fc700 1 -- 172.21.15.60:0/3502831779 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6568021630 con 0x7f65700b5eb0 2022-06-29T16:05:20.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.686+0000 7f657615f700 1 -- 172.21.15.60:0/3502831779 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65701457f0 con 0x7f65700b5eb0 2022-06-29T16:05:20.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.686+0000 7f65627fc700 1 -- 172.21.15.60:0/3502831779 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f656800b8a0 con 0x7f65700b5eb0 2022-06-29T16:05:20.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.686+0000 7f657615f700 1 -- 172.21.15.60:0/3502831779 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6570113e50 con 0x7f65700b5eb0 2022-06-29T16:05:20.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.690+0000 7f657615f700 1 -- 172.21.15.60:0/3502831779 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f654c003410 con 0x7f65700b5eb0 2022-06-29T16:05:20.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.690+0000 7f65627fc700 1 -- 172.21.15.60:0/3502831779 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6568027020 con 0x7f65700b5eb0 2022-06-29T16:05:20.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.690+0000 7f65627fc700 1 --2- 172.21.15.60:0/3502831779 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6558053b00 0x7f6558055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:20.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.690+0000 7f65627fc700 1 -- 172.21.15.60:0/3502831779 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f656806d310 con 0x7f65700b5eb0 2022-06-29T16:05:20.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.690+0000 7f657515d700 1 --2- 172.21.15.60:0/3502831779 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6558053b00 0x7f6558055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:20.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.690+0000 7f657515d700 1 --2- 172.21.15.60:0/3502831779 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6558053b00 0x7f6558055fc0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f656400d000 tx=0x7f6564007c30).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:20.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.694+0000 7f65627fc700 1 -- 172.21.15.60:0/3502831779 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6568044180 con 0x7f65700b5eb0 2022-06-29T16:05:20.829 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.822+0000 7f657615f700 1 -- 172.21.15.60:0/3502831779 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f654c003c40 con 0x7f65700b5eb0 2022-06-29T16:05:20.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.822+0000 7f65627fc700 1 -- 172.21.15.60:0/3502831779 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) v1 ==== 100+0+10 (secure 0 0 0) 0x7f6568015020 con 0x7f65700b5eb0 2022-06-29T16:05:20.831 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:05:20.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.826+0000 7f657615f700 1 -- 172.21.15.60:0/3502831779 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6558053b00 msgr2=0x7f6558055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:20.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.826+0000 7f657615f700 1 --2- 172.21.15.60:0/3502831779 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6558053b00 0x7f6558055fc0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f656400d000 tx=0x7f6564007c30).stop 2022-06-29T16:05:20.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.826+0000 7f657615f700 1 -- 172.21.15.60:0/3502831779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65700b5eb0 msgr2=0x7f6570152090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:20.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.826+0000 7f657615f700 1 --2- 172.21.15.60:0/3502831779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65700b5eb0 0x7f6570152090 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0x7f6570145c20 tx=0x7f6568016000).stop 2022-06-29T16:05:20.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.826+0000 7f657615f700 1 -- 172.21.15.60:0/3502831779 shutdown_connections 2022-06-29T16:05:20.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.826+0000 7f657615f700 1 --2- 172.21.15.60:0/3502831779 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6558053b00 0x7f6558055fc0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.826+0000 7f657615f700 1 --2- 172.21.15.60:0/3502831779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65700b5eb0 0x7f6570152090 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.826+0000 7f657615f700 1 --2- 172.21.15.60:0/3502831779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f65700b48b0 0x7f6570151ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.826+0000 7f657615f700 1 --2- 172.21.15.60:0/3502831779 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f65700b2c50 0x7f65701461f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.826+0000 7f657615f700 1 -- 172.21.15.60:0/3502831779 >> 172.21.15.60:0/3502831779 conn(0x7f6570062090 msgr2=0x7f65701430d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:20.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.826+0000 7f657615f700 1 -- 172.21.15.60:0/3502831779 shutdown_connections 2022-06-29T16:05:20.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.826+0000 7f657615f700 1 -- 172.21.15.60:0/3502831779 wait complete. 2022-06-29T16:05:20.844 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-06-29T16:05:20.844 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:05:20.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.938+0000 7f0d57fff700 1 Processor -- start 2022-06-29T16:05:20.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.938+0000 7f0d57fff700 1 -- start start 2022-06-29T16:05:20.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.938+0000 7f0d57fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0d5805d070 0x7f0d5812b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:20.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.938+0000 7f0d57fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d58066b20 0x7f0d5812bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:20.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.938+0000 7f0d57fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0d58142ae0 0x7f0d58142f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:20.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.938+0000 7f0d57fff700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d58143b60 con 0x7f0d58142ae0 2022-06-29T16:05:20.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.938+0000 7f0d57fff700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d5812c0b0 con 0x7f0d5805d070 2022-06-29T16:05:20.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.938+0000 7f0d57fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d580aced0 con 0x7f0d58066b20 2022-06-29T16:05:20.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.942+0000 7f0d577fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0d58142ae0 0x7f0d58142f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:20.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.942+0000 7f0d577fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0d58142ae0 0x7f0d58142f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50214/0 (socket says 172.21.15.60:50214) 2022-06-29T16:05:20.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.942+0000 7f0d577fe700 1 -- 172.21.15.60:0/2659783563 learned_addr learned my addr 172.21.15.60:0/2659783563 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:20.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.942+0000 7f0d56ffd700 1 --2- 172.21.15.60:0/2659783563 >> v2:172.21.15.92:3300/0 conn(0x7f0d5805d070 0x7f0d5812b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:20.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.942+0000 7f0d567fc700 1 --2- 172.21.15.60:0/2659783563 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d58066b20 0x7f0d5812bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:20.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.942+0000 7f0d577fe700 1 -- 172.21.15.60:0/2659783563 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d58066b20 msgr2=0x7f0d5812bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:20.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.942+0000 7f0d577fe700 1 --2- 172.21.15.60:0/2659783563 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d58066b20 0x7f0d5812bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.942+0000 7f0d577fe700 1 -- 172.21.15.60:0/2659783563 >> v2:172.21.15.92:3300/0 conn(0x7f0d5805d070 msgr2=0x7f0d5812b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:20.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.942+0000 7f0d577fe700 1 --2- 172.21.15.60:0/2659783563 >> v2:172.21.15.92:3300/0 conn(0x7f0d5805d070 0x7f0d5812b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.942+0000 7f0d577fe700 1 -- 172.21.15.60:0/2659783563 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d580ae9a0 con 0x7f0d58142ae0 2022-06-29T16:05:20.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.942+0000 7f0d577fe700 1 --2- 172.21.15.60:0/2659783563 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d58142ae0 0x7f0d58142f20 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0x7f0d500032f0 tx=0x7f0d50009290).ready entity=mon.0 client_cookie=3158c1102e73b48a server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:20.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.942+0000 7f0d55ffb700 1 -- 172.21.15.60:0/2659783563 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d5000b680 con 0x7f0d58142ae0 2022-06-29T16:05:20.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.942+0000 7f0d55ffb700 1 -- 172.21.15.60:0/2659783563 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d5000b7e0 con 0x7f0d58142ae0 2022-06-29T16:05:20.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.942+0000 7f0d55ffb700 1 -- 172.21.15.60:0/2659783563 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d5000b950 con 0x7f0d58142ae0 2022-06-29T16:05:20.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.942+0000 7f0d57fff700 1 -- 172.21.15.60:0/2659783563 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d58142ae0 msgr2=0x7f0d58142f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:20.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.942+0000 7f0d57fff700 1 --2- 172.21.15.60:0/2659783563 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d58142ae0 0x7f0d58142f20 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0x7f0d500032f0 tx=0x7f0d50009290).stop 2022-06-29T16:05:20.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.942+0000 7f0d57fff700 1 -- 172.21.15.60:0/2659783563 shutdown_connections 2022-06-29T16:05:20.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.942+0000 7f0d57fff700 1 --2- 172.21.15.60:0/2659783563 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d58142ae0 0x7f0d58142f20 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.942+0000 7f0d57fff700 1 --2- 172.21.15.60:0/2659783563 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d58066b20 0x7f0d5812bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.942+0000 7f0d57fff700 1 --2- 172.21.15.60:0/2659783563 >> v2:172.21.15.92:3300/0 conn(0x7f0d5805d070 0x7f0d5812b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.942+0000 7f0d57fff700 1 -- 172.21.15.60:0/2659783563 >> 172.21.15.60:0/2659783563 conn(0x7f0d580b5890 msgr2=0x7f0d580b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:20.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.942+0000 7f0d57fff700 1 -- 172.21.15.60:0/2659783563 shutdown_connections 2022-06-29T16:05:20.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.942+0000 7f0d57fff700 1 -- 172.21.15.60:0/2659783563 wait complete. 2022-06-29T16:05:20.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.942+0000 7f0d57fff700 1 Processor -- start 2022-06-29T16:05:20.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.942+0000 7f0d57fff700 1 -- start start 2022-06-29T16:05:20.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.942+0000 7f0d57fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d5805d070 0x7f0d5812dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:20.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.942+0000 7f0d57fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d58066b20 0x7f0d5812c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:20.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.942+0000 7f0d57fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d58142ae0 0x7f0d58128110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:20.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.942+0000 7f0d57fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d58154e10 con 0x7f0d5805d070 2022-06-29T16:05:20.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.942+0000 7f0d57fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5812e290 con 0x7f0d58142ae0 2022-06-29T16:05:20.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.942+0000 7f0d57fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5812e530 con 0x7f0d58066b20 2022-06-29T16:05:20.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.946+0000 7f0d56ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d5805d070 0x7f0d5812dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:20.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.946+0000 7f0d567fc700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d58066b20 0x7f0d5812c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:20.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.946+0000 7f0d56ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d5805d070 0x7f0d5812dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50218/0 (socket says 172.21.15.60:50218) 2022-06-29T16:05:20.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.946+0000 7f0d577fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d58142ae0 0x7f0d58128110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:20.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.946+0000 7f0d56ffd700 1 -- 172.21.15.60:0/1199534019 learned_addr learned my addr 172.21.15.60:0/1199534019 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:20.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.946+0000 7f0d56ffd700 1 -- 172.21.15.60:0/1199534019 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d58066b20 msgr2=0x7f0d5812c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:20.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.946+0000 7f0d56ffd700 1 --2- 172.21.15.60:0/1199534019 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d58066b20 0x7f0d5812c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.946+0000 7f0d56ffd700 1 -- 172.21.15.60:0/1199534019 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d58142ae0 msgr2=0x7f0d58128110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:20.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.946+0000 7f0d56ffd700 1 --2- 172.21.15.60:0/1199534019 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d58142ae0 0x7f0d58128110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:20.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.946+0000 7f0d56ffd700 1 -- 172.21.15.60:0/1199534019 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d50009530 con 0x7f0d5805d070 2022-06-29T16:05:20.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.946+0000 7f0d56ffd700 1 --2- 172.21.15.60:0/1199534019 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d5805d070 0x7f0d5812dbd0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0x7f0d4800c000 tx=0x7f0d48006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:20.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.946+0000 7f0d4ffff700 1 -- 172.21.15.60:0/1199534019 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d48010780 con 0x7f0d5805d070 2022-06-29T16:05:20.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.946+0000 7f0d4ffff700 1 -- 172.21.15.60:0/1199534019 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d480108e0 con 0x7f0d5805d070 2022-06-29T16:05:20.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.946+0000 7f0d57fff700 1 -- 172.21.15.60:0/1199534019 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d5812e7d0 con 0x7f0d5805d070 2022-06-29T16:05:20.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.946+0000 7f0d4ffff700 1 -- 172.21.15.60:0/1199534019 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d48010c40 con 0x7f0d5805d070 2022-06-29T16:05:20.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.946+0000 7f0d57fff700 1 -- 172.21.15.60:0/1199534019 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d5812cbf0 con 0x7f0d5805d070 2022-06-29T16:05:20.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.946+0000 7f0d4ffff700 1 -- 172.21.15.60:0/1199534019 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0d4801c460 con 0x7f0d5805d070 2022-06-29T16:05:20.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.946+0000 7f0d57fff700 1 -- 172.21.15.60:0/1199534019 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d5812e960 con 0x7f0d5805d070 2022-06-29T16:05:20.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.946+0000 7f0d4ffff700 1 --2- 172.21.15.60:0/1199534019 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0d3c053a10 0x7f0d3c055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:20.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.950+0000 7f0d4ffff700 1 -- 172.21.15.60:0/1199534019 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f0d4806c520 con 0x7f0d5805d070 2022-06-29T16:05:20.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.950+0000 7f0d567fc700 1 --2- 172.21.15.60:0/1199534019 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0d3c053a10 0x7f0d3c055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:20.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.950+0000 7f0d567fc700 1 --2- 172.21.15.60:0/1199534019 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0d3c053a10 0x7f0d3c055ed0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f0d4000b000 tx=0x7f0d40006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:20.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:20.954+0000 7f0d4ffff700 1 -- 172.21.15.60:0/1199534019 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0d48043200 con 0x7f0d5805d070 2022-06-29T16:05:21.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:21.114+0000 7f0d57fff700 1 -- 172.21.15.60:0/1199534019 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d580573d0 con 0x7f0d5805d070 2022-06-29T16:05:21.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:21.114+0000 7f0d4ffff700 1 -- 172.21.15.60:0/1199534019 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+2141 (secure 0 0 0) 0x7f0d48046820 con 0x7f0d5805d070 2022-06-29T16:05:21.123 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:21.124 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":36,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6657,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.60:6833/3121204394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3121204394},{"type":"v1","addr":"172.21.15.60:6833","nonce":3121204394}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":34}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:05:16.293932+0000","modified":"2022-06-29T16:05:18.324125+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6688},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6688":{"gid":6688,"name":"b","rank":0,"incarnation":35,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/1245503283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1245503283},{"type":"v1","addr":"172.21.15.92:6835","nonce":1245503283}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-06-29T16:05:21.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:21.118+0000 7f0d57fff700 1 -- 172.21.15.60:0/1199534019 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0d3c053a10 msgr2=0x7f0d3c055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:21.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:21.118+0000 7f0d57fff700 1 --2- 172.21.15.60:0/1199534019 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0d3c053a10 0x7f0d3c055ed0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f0d4000b000 tx=0x7f0d40006df0).stop 2022-06-29T16:05:21.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:21.118+0000 7f0d57fff700 1 -- 172.21.15.60:0/1199534019 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d5805d070 msgr2=0x7f0d5812dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:21.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:21.118+0000 7f0d57fff700 1 --2- 172.21.15.60:0/1199534019 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d5805d070 0x7f0d5812dbd0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0x7f0d4800c000 tx=0x7f0d48006df0).stop 2022-06-29T16:05:21.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:21.118+0000 7f0d57fff700 1 -- 172.21.15.60:0/1199534019 shutdown_connections 2022-06-29T16:05:21.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:21.118+0000 7f0d57fff700 1 --2- 172.21.15.60:0/1199534019 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0d3c053a10 0x7f0d3c055ed0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:21.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:21.118+0000 7f0d57fff700 1 --2- 172.21.15.60:0/1199534019 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0d58142ae0 0x7f0d58128110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:21.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:21.118+0000 7f0d57fff700 1 --2- 172.21.15.60:0/1199534019 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0d58066b20 0x7f0d5812c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:21.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:21.118+0000 7f0d57fff700 1 --2- 172.21.15.60:0/1199534019 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d5805d070 0x7f0d5812dbd0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:21.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:21.118+0000 7f0d57fff700 1 -- 172.21.15.60:0/1199534019 >> 172.21.15.60:0/1199534019 conn(0x7f0d580b5890 msgr2=0x7f0d58128c50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:21.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:21.118+0000 7f0d57fff700 1 -- 172.21.15.60:0/1199534019 shutdown_connections 2022-06-29T16:05:21.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:21.118+0000 7f0d57fff700 1 -- 172.21.15.60:0/1199534019 wait complete. 2022-06-29T16:05:21.129 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 36 2022-06-29T16:05:21.338 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 36, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:05:16.293932+0000', 'modified': '2022-06-29T16:05:18.324125+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 81, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6688}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6688': {'gid': 6688, 'name': 'b', 'rank': 0, 'incarnation': 35, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.92:6835/1245503283', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 1245503283}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 1245503283}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:05:21.339 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:05:21.340 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-06-29T16:05:21.457 INFO:teuthology.orchestra.run.smithi092.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":6688,"want_state":"up:active","state":"up:active","rank_uptime":4.1314462069999998,"mdsmap_epoch":36,"osdmap_epoch":81,"osdmap_epoch_barrier":81,"uptime":8.8206633879999998} 2022-06-29T16:05:21.458 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 6688, "mdsmap_epoch": 36, "osdmap_epoch": 81, "osdmap_epoch_barrier": 81, "rank_uptime": 4.131446207, "state": "up:active", "uptime": 8.820663388, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:05:21.458 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-06-29T16:05:21.459 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-06-29T16:05:21.459 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi121.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-06-29T16:05:21.460 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:21.460 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:05:21.468 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:05:21.486 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:05:21.487 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:05:21.502 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:05:21.503 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:05:21.504 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:21.504 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:05:21.552 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-06-29T16:05:21.553 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:05:21.556 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:05:21.572 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:05:21.573 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:05:21.582 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:05:21.583 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:05:21.584 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:21.585 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:05:21.631 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:2022-06-29T16:05:21.626+0000 7f35f75bb080 -1 init, newargv = 0x55b452e7f320 newargc=9ceph-fuse[13048]: starting ceph client 2022-06-29T16:05:21.632 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr: 2022-06-29T16:05:21.639 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[13048]: starting fuse 2022-06-29T16:05:22.692 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:05:22.710 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:05:22.711 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:05:22.720 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:05:22.721 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:05:22.722 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:22.723 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:05:22.773 INFO:teuthology.orchestra.run.smithi121.stdout:52 2022-06-29T16:05:22.773 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-06-29T16:05:22.774 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:22.775 DEBUG:teuthology.orchestra.run.smithi121:> sudo python3 -c ' 2022-06-29T16:05:22.775 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2022-06-29T16:05:22.776 DEBUG:teuthology.orchestra.run.smithi121:> import re 2022-06-29T16:05:22.776 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-06-29T16:05:22.776 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2022-06-29T16:05:22.777 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:05:22.777 DEBUG:teuthology.orchestra.run.smithi121:> def find_socket(client_name): 2022-06-29T16:05:22.778 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-06-29T16:05:22.778 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2022-06-29T16:05:22.779 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:05:22.779 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2022-06-29T16:05:22.779 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2022-06-29T16:05:22.780 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2022-06-29T16:05:22.780 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2022-06-29T16:05:22.781 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:05:22.781 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2022-06-29T16:05:22.781 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-06-29T16:05:22.782 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2022-06-29T16:05:22.782 DEBUG:teuthology.orchestra.run.smithi121:> return f 2022-06-29T16:05:22.783 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-06-29T16:05:22.783 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:05:22.783 DEBUG:teuthology.orchestra.run.smithi121:> print(find_socket("client.0")) 2022-06-29T16:05:22.784 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-06-29T16:05:22.828 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.0.13048.asok 2022-06-29T16:05:22.833 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.13048.asok 2022-06-29T16:05:22.834 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:22.834 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.13048.asok status 2022-06-29T16:05:22.921 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2022-06-29T16:05:22.921 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2022-06-29T16:05:22.922 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "f25ffac72e5a9e63860ff7deb6767a11f0559a16", 2022-06-29T16:05:22.922 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 15.2.16-304-gf25ffac7 (f25ffac72e5a9e63860ff7deb6767a11f0559a16) octopus (stable)", 2022-06-29T16:05:22.922 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "0", 2022-06-29T16:05:22.922 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2022-06-29T16:05:22.923 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-06-29T16:05:22.923 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "13048", 2022-06-29T16:05:22.923 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2022-06-29T16:05:22.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:05:22.924 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2022-06-29T16:05:22.924 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2022-06-29T16:05:22.924 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 6843, 2022-06-29T16:05:22.924 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2022-06-29T16:05:22.924 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2022-06-29T16:05:22.925 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2022-06-29T16:05:22.925 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 6843 2022-06-29T16:05:22.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:05:22.925 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-06-29T16:05:22.926 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-06-29T16:05:22.926 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-06-29T16:05:22.926 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3447021701 2022-06-29T16:05:22.926 INFO:teuthology.orchestra.run.smithi121.stdout: } 2022-06-29T16:05:22.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:05:22.927 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-06-29T16:05:22.927 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-06-29T16:05:22.927 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-06-29T16:05:22.927 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3447021701 2022-06-29T16:05:22.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:05:22.928 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.6843 172.21.15.121:0/3447021701", 2022-06-29T16:05:22.928 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "172.21.15.121:0/3447021701", 2022-06-29T16:05:22.928 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2022-06-29T16:05:22.929 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 36, 2022-06-29T16:05:22.929 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 81, 2022-06-29T16:05:22.929 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2022-06-29T16:05:22.929 INFO:teuthology.orchestra.run.smithi121.stdout: "blacklisted": false, 2022-06-29T16:05:22.929 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2022-06-29T16:05:22.930 INFO:teuthology.orchestra.run.smithi121.stdout:} 2022-06-29T16:05:22.931 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:22.932 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:05:22.932 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:05:22.939 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T16:05:22.940 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T16:05:22.940 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:22.941 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:05:22.955 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:05:23.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.062+0000 7f611f030700 1 Processor -- start 2022-06-29T16:05:23.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.062+0000 7f611f030700 1 -- start start 2022-06-29T16:05:23.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.062+0000 7f611f030700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61180ae7c0 0x7f6118141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:23.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.062+0000 7f611f030700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f61180b4010 0x7f6118142490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:23.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.062+0000 7f611f030700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6118143a40 con 0x7f6118146160 2022-06-29T16:05:23.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.062+0000 7f611f030700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61180b56d0 con 0x7f61180b4010 2022-06-29T16:05:23.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.062+0000 7f611f030700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61180aee60 con 0x7f61180ae7c0 2022-06-29T16:05:23.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.062+0000 7f6117fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f61180b4010 0x7f6118142490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:23.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.062+0000 7f611d5cd700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6118146160 0x7f61181465f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58316/0 (socket says 172.21.15.60:58316) 2022-06-29T16:05:23.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.062+0000 7f611d5cd700 1 -- 172.21.15.60:0/1222645488 learned_addr learned my addr 172.21.15.60:0/1222645488 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:23.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.062+0000 7f611cdcc700 1 --2- 172.21.15.60:0/1222645488 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61180ae7c0 0x7f6118141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:23.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.062+0000 7f61177fe700 1 -- 172.21.15.60:0/1222645488 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1100946190 0 0) 0x7f6118143a40 con 0x7f6118146160 2022-06-29T16:05:23.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.062+0000 7f61177fe700 1 -- 172.21.15.60:0/1222645488 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60f8003260 con 0x7f6118146160 2022-06-29T16:05:23.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.062+0000 7f6117fff700 1 -- 172.21.15.60:0/1222645488 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61180ae7c0 msgr2=0x7f6118141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:23.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.062+0000 7f6117fff700 1 --2- 172.21.15.60:0/1222645488 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61180ae7c0 0x7f6118141fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:23.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.062+0000 7f6117fff700 1 -- 172.21.15.60:0/1222645488 >> v1:172.21.15.60:6789/0 conn(0x7f6118146160 legacy=0x7f61181465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:23.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.062+0000 7f6117fff700 1 -- 172.21.15.60:0/1222645488 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61180af100 con 0x7f61180b4010 2022-06-29T16:05:23.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.062+0000 7f6117fff700 1 --2- 172.21.15.60:0/1222645488 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61180b4010 0x7f6118142490 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 rx=0x7f6100002f70 tx=0x7f6100006df0).ready entity=mon.0 client_cookie=9c337588fc07805e server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:23.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.062+0000 7f61177fe700 1 -- 172.21.15.60:0/1222645488 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f610000d4a0 con 0x7f61180b4010 2022-06-29T16:05:23.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.062+0000 7f61177fe700 1 -- 172.21.15.60:0/1222645488 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6100000e90 con 0x7f61180b4010 2022-06-29T16:05:23.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.062+0000 7f61177fe700 1 -- 172.21.15.60:0/1222645488 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f610000d7f0 con 0x7f61180b4010 2022-06-29T16:05:23.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.066+0000 7f611f030700 1 -- 172.21.15.60:0/1222645488 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61180b4010 msgr2=0x7f6118142490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:23.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.066+0000 7f611f030700 1 --2- 172.21.15.60:0/1222645488 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61180b4010 0x7f6118142490 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 rx=0x7f6100002f70 tx=0x7f6100006df0).stop 2022-06-29T16:05:23.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.066+0000 7f611f030700 1 -- 172.21.15.60:0/1222645488 shutdown_connections 2022-06-29T16:05:23.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.066+0000 7f611f030700 1 --2- 172.21.15.60:0/1222645488 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61180b4010 0x7f6118142490 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:23.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.066+0000 7f611f030700 1 --2- 172.21.15.60:0/1222645488 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61180ae7c0 0x7f6118141fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:23.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.066+0000 7f611f030700 1 -- 172.21.15.60:0/1222645488 >> 172.21.15.60:0/1222645488 conn(0x7f61180b9340 msgr2=0x7f61180b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:23.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.066+0000 7f611f030700 1 -- 172.21.15.60:0/1222645488 shutdown_connections 2022-06-29T16:05:23.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.066+0000 7f611f030700 1 -- 172.21.15.60:0/1222645488 wait complete. 2022-06-29T16:05:23.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.066+0000 7f611f030700 1 Processor -- start 2022-06-29T16:05:23.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.066+0000 7f611f030700 1 -- start start 2022-06-29T16:05:23.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.066+0000 7f611f030700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61180ae7c0 0x7f6118151c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:23.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.066+0000 7f611f030700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61180b4010 0x7f6118149830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:23.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.066+0000 7f611f030700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6118146160 0x7f61180af4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:23.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.066+0000 7f611f030700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61180b0da0 con 0x7f6118146160 2022-06-29T16:05:23.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.066+0000 7f611f030700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61180b1040 con 0x7f61180b4010 2022-06-29T16:05:23.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.066+0000 7f611f030700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61180b12e0 con 0x7f61180ae7c0 2022-06-29T16:05:23.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.066+0000 7f611d5cd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6118146160 0x7f61180af4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:23.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.066+0000 7f611d5cd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6118146160 0x7f61180af4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50236/0 (socket says 172.21.15.60:50236) 2022-06-29T16:05:23.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.066+0000 7f611d5cd700 1 -- 172.21.15.60:0/1243146273 learned_addr learned my addr 172.21.15.60:0/1243146273 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:23.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.070+0000 7f611cdcc700 1 --2- 172.21.15.60:0/1243146273 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61180ae7c0 0x7f6118151c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:23.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.070+0000 7f6117fff700 1 --2- 172.21.15.60:0/1243146273 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61180b4010 0x7f6118149830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:23.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.070+0000 7f611d5cd700 1 -- 172.21.15.60:0/1243146273 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61180ae7c0 msgr2=0x7f6118151c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:23.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.070+0000 7f611d5cd700 1 --2- 172.21.15.60:0/1243146273 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61180ae7c0 0x7f6118151c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:23.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.070+0000 7f611d5cd700 1 -- 172.21.15.60:0/1243146273 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61180b4010 msgr2=0x7f6118149830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:23.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.070+0000 7f611d5cd700 1 --2- 172.21.15.60:0/1243146273 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61180b4010 0x7f6118149830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:23.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.070+0000 7f611d5cd700 1 -- 172.21.15.60:0/1243146273 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f610000b180 con 0x7f6118146160 2022-06-29T16:05:23.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.070+0000 7f611cdcc700 1 --2- 172.21.15.60:0/1243146273 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61180ae7c0 0x7f6118151c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:23.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.070+0000 7f611d5cd700 1 --2- 172.21.15.60:0/1243146273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6118146160 0x7f61180af4e0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0x7f6110001110 tx=0x7f6110007a80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:23.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.070+0000 7f6115ffb700 1 -- 172.21.15.60:0/1243146273 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61100070d0 con 0x7f6118146160 2022-06-29T16:05:23.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.070+0000 7f6115ffb700 1 -- 172.21.15.60:0/1243146273 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6110007230 con 0x7f6118146160 2022-06-29T16:05:23.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.070+0000 7f611f030700 1 -- 172.21.15.60:0/1243146273 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6118148d50 con 0x7f6118146160 2022-06-29T16:05:23.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.070+0000 7f6115ffb700 1 -- 172.21.15.60:0/1243146273 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6110005610 con 0x7f6118146160 2022-06-29T16:05:23.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.070+0000 7f611f030700 1 -- 172.21.15.60:0/1243146273 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61180b29e0 con 0x7f6118146160 2022-06-29T16:05:23.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.070+0000 7f611f030700 1 -- 172.21.15.60:0/1243146273 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61180af9d0 con 0x7f6118146160 2022-06-29T16:05:23.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.070+0000 7f6115ffb700 1 -- 172.21.15.60:0/1243146273 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f611001a060 con 0x7f6118146160 2022-06-29T16:05:23.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.070+0000 7f6115ffb700 1 --2- 172.21.15.60:0/1243146273 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f60fc053b00 0x7f60fc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:23.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.070+0000 7f6115ffb700 1 -- 172.21.15.60:0/1243146273 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f611006b6d0 con 0x7f6118146160 2022-06-29T16:05:23.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.074+0000 7f611cdcc700 1 --2- 172.21.15.60:0/1243146273 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f60fc053b00 0x7f60fc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:23.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.074+0000 7f611cdcc700 1 --2- 172.21.15.60:0/1243146273 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f60fc053b00 0x7f60fc055fc0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f610800b000 tx=0x7f6108008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:23.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.082+0000 7f6115ffb700 1 -- 172.21.15.60:0/1243146273 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f61100425a0 con 0x7f6118146160 2022-06-29T16:05:23.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.210+0000 7f611f030700 1 -- 172.21.15.60:0/1243146273 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61180573d0 con 0x7f6118146160 2022-06-29T16:05:23.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.214+0000 7f6115ffb700 1 -- 172.21.15.60:0/1243146273 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16170 (secure 0 0 0) 0x7f6110018070 con 0x7f6118146160 2022-06-29T16:05:23.221 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:23.221 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":81,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:05:16.667913+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:05:14.715486+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2022-06-29T16:05:15.306355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{"172.21.15.92:6835/2464413229":"2022-06-30T16:05:16.667864+0000","172.21.15.92:6834/2464413229":"2022-06-30T16:05:16.667864+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:05:23.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.214+0000 7f611f030700 1 -- 172.21.15.60:0/1243146273 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f60fc053b00 msgr2=0x7f60fc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:23.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.214+0000 7f611f030700 1 --2- 172.21.15.60:0/1243146273 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f60fc053b00 0x7f60fc055fc0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f610800b000 tx=0x7f6108008040).stop 2022-06-29T16:05:23.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.214+0000 7f611f030700 1 -- 172.21.15.60:0/1243146273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6118146160 msgr2=0x7f61180af4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:23.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.214+0000 7f611f030700 1 --2- 172.21.15.60:0/1243146273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6118146160 0x7f61180af4e0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0x7f6110001110 tx=0x7f6110007a80).stop 2022-06-29T16:05:23.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.214+0000 7f611f030700 1 -- 172.21.15.60:0/1243146273 shutdown_connections 2022-06-29T16:05:23.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.214+0000 7f611f030700 1 --2- 172.21.15.60:0/1243146273 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f60fc053b00 0x7f60fc055fc0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:23.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.214+0000 7f611f030700 1 --2- 172.21.15.60:0/1243146273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6118146160 0x7f61180af4e0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:23.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.214+0000 7f611f030700 1 --2- 172.21.15.60:0/1243146273 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61180b4010 0x7f6118149830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:23.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.214+0000 7f611f030700 1 --2- 172.21.15.60:0/1243146273 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61180ae7c0 0x7f6118151c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:23.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.214+0000 7f611f030700 1 -- 172.21.15.60:0/1243146273 >> 172.21.15.60:0/1243146273 conn(0x7f61180b9340 msgr2=0x7f6118141d70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:23.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.214+0000 7f611f030700 1 -- 172.21.15.60:0/1243146273 shutdown_connections 2022-06-29T16:05:23.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.214+0000 7f611f030700 1 -- 172.21.15.60:0/1243146273 wait complete. 2022-06-29T16:05:23.235 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-29T16:05:14.715486+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2022-06-29T16:05:15.306355+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-29T16:05:23.235 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:05:23.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.342+0000 7f0721b39700 1 Processor -- start 2022-06-29T16:05:23.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.342+0000 7f0721b39700 1 -- start start 2022-06-29T16:05:23.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.342+0000 7f0721b39700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f071c0b2b40 0x7f071c13c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:23.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.342+0000 7f0721b39700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f071c0b1c90 0x7f071c13cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:23.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.342+0000 7f0721b39700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f071c142ce0 0x7f071c143170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:23.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.342+0000 7f0721b39700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f071c0b1100 con 0x7f071c0b2b40 2022-06-29T16:05:23.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.342+0000 7f0721b39700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f071c0b2f20 con 0x7f071c0b1c90 2022-06-29T16:05:23.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.342+0000 7f0721b39700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f071c13fac0 con 0x7f071c142ce0 2022-06-29T16:05:23.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.342+0000 7f0720b37700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f071c0b2b40 0x7f071c13c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:23.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.342+0000 7f0720b37700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f071c0b2b40 0x7f071c13c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50240/0 (socket says 172.21.15.60:50240) 2022-06-29T16:05:23.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.342+0000 7f0720b37700 1 -- 172.21.15.60:0/4088138762 learned_addr learned my addr 172.21.15.60:0/4088138762 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:23.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.342+0000 7f071bfff700 1 --2- 172.21.15.60:0/4088138762 >> v2:172.21.15.92:3300/0 conn(0x7f071c0b1c90 0x7f071c13cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:23.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.342+0000 7f0721338700 1 --2- 172.21.15.60:0/4088138762 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f071c142ce0 0x7f071c143170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:23.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.342+0000 7f0720b37700 1 -- 172.21.15.60:0/4088138762 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f071c142ce0 msgr2=0x7f071c143170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:23.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.342+0000 7f0720b37700 1 --2- 172.21.15.60:0/4088138762 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f071c142ce0 0x7f071c143170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:23.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.342+0000 7f0720b37700 1 -- 172.21.15.60:0/4088138762 >> v2:172.21.15.92:3300/0 conn(0x7f071c0b1c90 msgr2=0x7f071c13cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:23.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.342+0000 7f0720b37700 1 --2- 172.21.15.60:0/4088138762 >> v2:172.21.15.92:3300/0 conn(0x7f071c0b1c90 0x7f071c13cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:23.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.342+0000 7f0720b37700 1 -- 172.21.15.60:0/4088138762 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f071c13f820 con 0x7f071c0b2b40 2022-06-29T16:05:23.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.342+0000 7f0720b37700 1 --2- 172.21.15.60:0/4088138762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f071c0b2b40 0x7f071c13c6c0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0x7f070c0032f0 tx=0x7f070c009290).ready entity=mon.0 client_cookie=b4d03d52d395f9a7 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:23.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.346+0000 7f071b7fe700 1 -- 172.21.15.60:0/4088138762 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f070c00b870 con 0x7f071c0b2b40 2022-06-29T16:05:23.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.346+0000 7f071b7fe700 1 -- 172.21.15.60:0/4088138762 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f070c00b9d0 con 0x7f071c0b2b40 2022-06-29T16:05:23.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.346+0000 7f071b7fe700 1 -- 172.21.15.60:0/4088138762 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f070c00bb40 con 0x7f071c0b2b40 2022-06-29T16:05:23.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.346+0000 7f0721b39700 1 -- 172.21.15.60:0/4088138762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f071c0b2b40 msgr2=0x7f071c13c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:23.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.346+0000 7f0721b39700 1 --2- 172.21.15.60:0/4088138762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f071c0b2b40 0x7f071c13c6c0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0x7f070c0032f0 tx=0x7f070c009290).stop 2022-06-29T16:05:23.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.346+0000 7f0721b39700 1 -- 172.21.15.60:0/4088138762 shutdown_connections 2022-06-29T16:05:23.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.346+0000 7f0721b39700 1 --2- 172.21.15.60:0/4088138762 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f071c142ce0 0x7f071c143170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:23.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.346+0000 7f0721b39700 1 --2- 172.21.15.60:0/4088138762 >> v2:172.21.15.92:3300/0 conn(0x7f071c0b1c90 0x7f071c13cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:23.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.346+0000 7f0721b39700 1 --2- 172.21.15.60:0/4088138762 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f071c0b2b40 0x7f071c13c6c0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:23.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.346+0000 7f0721b39700 1 -- 172.21.15.60:0/4088138762 >> 172.21.15.60:0/4088138762 conn(0x7f071c0b6120 msgr2=0x7f071c0b20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:23.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.346+0000 7f0721b39700 1 -- 172.21.15.60:0/4088138762 shutdown_connections 2022-06-29T16:05:23.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.346+0000 7f0721b39700 1 -- 172.21.15.60:0/4088138762 wait complete. 2022-06-29T16:05:23.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.350+0000 7f0721b39700 1 Processor -- start 2022-06-29T16:05:23.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.350+0000 7f0721b39700 1 -- start start 2022-06-29T16:05:23.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.350+0000 7f0721b39700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f071c0b1c90 0x7f071c141680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:23.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.350+0000 7f0721b39700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f071c0b2b40 0x7f071c0b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:23.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.350+0000 7f0721b39700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f071c142ce0 0x7f071c0ac990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:23.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.350+0000 7f0721b39700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071c139830 con 0x7f071c142ce0 2022-06-29T16:05:23.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.350+0000 7f0721b39700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071c141f60 con 0x7f071c0b2b40 2022-06-29T16:05:23.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.350+0000 7f0721b39700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f071c142200 con 0x7f071c0b1c90 2022-06-29T16:05:23.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.350+0000 7f0721338700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f071c142ce0 0x7f071c0ac990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:23.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.350+0000 7f0721338700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f071c142ce0 0x7f071c0ac990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50250/0 (socket says 172.21.15.60:50250) 2022-06-29T16:05:23.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.350+0000 7f0721338700 1 -- 172.21.15.60:0/4043628040 learned_addr learned my addr 172.21.15.60:0/4043628040 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:23.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.350+0000 7f0720b37700 1 --2- 172.21.15.60:0/4043628040 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f071c0b1c90 0x7f071c141680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:23.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.350+0000 7f071bfff700 1 --2- 172.21.15.60:0/4043628040 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f071c0b2b40 0x7f071c0b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:23.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.350+0000 7f0721338700 1 -- 172.21.15.60:0/4043628040 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f071c0b1c90 msgr2=0x7f071c141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:23.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.350+0000 7f0721338700 1 --2- 172.21.15.60:0/4043628040 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f071c0b1c90 0x7f071c141680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:23.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.350+0000 7f0721338700 1 -- 172.21.15.60:0/4043628040 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f071c0b2b40 msgr2=0x7f071c0b0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:23.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.350+0000 7f0721338700 1 --2- 172.21.15.60:0/4043628040 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f071c0b2b40 0x7f071c0b0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:23.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.350+0000 7f0721338700 1 -- 172.21.15.60:0/4043628040 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f070c009530 con 0x7f071c142ce0 2022-06-29T16:05:23.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.350+0000 7f0721338700 1 --2- 172.21.15.60:0/4043628040 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f071c142ce0 0x7f071c0ac990 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0x7f071400a000 tx=0x7f0714008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:23.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.350+0000 7f0719ffb700 1 -- 172.21.15.60:0/4043628040 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07140106e0 con 0x7f071c142ce0 2022-06-29T16:05:23.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.350+0000 7f0719ffb700 1 -- 172.21.15.60:0/4043628040 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0714010840 con 0x7f071c142ce0 2022-06-29T16:05:23.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.350+0000 7f0721b39700 1 -- 172.21.15.60:0/4043628040 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f071c141410 con 0x7f071c142ce0 2022-06-29T16:05:23.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.350+0000 7f0719ffb700 1 -- 172.21.15.60:0/4043628040 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0714010ba0 con 0x7f071c142ce0 2022-06-29T16:05:23.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.350+0000 7f0721b39700 1 -- 172.21.15.60:0/4043628040 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f071c0640e0 con 0x7f071c142ce0 2022-06-29T16:05:23.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.354+0000 7f0721b39700 1 -- 172.21.15.60:0/4043628040 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f071c143fb0 con 0x7f071c142ce0 2022-06-29T16:05:23.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.354+0000 7f0719ffb700 1 -- 172.21.15.60:0/4043628040 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f071401a030 con 0x7f071c142ce0 2022-06-29T16:05:23.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.354+0000 7f0719ffb700 1 --2- 172.21.15.60:0/4043628040 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f06fc053a70 0x7f06fc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:23.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.354+0000 7f0719ffb700 1 -- 172.21.15.60:0/4043628040 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f071406ba70 con 0x7f071c142ce0 2022-06-29T16:05:23.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.354+0000 7f0720b37700 1 --2- 172.21.15.60:0/4043628040 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f06fc053a70 0x7f06fc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:23.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.354+0000 7f0720b37700 1 --2- 172.21.15.60:0/4043628040 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f06fc053a70 0x7f06fc055f30 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f070c01b000 tx=0x7f070c016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:23.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.358+0000 7f0719ffb700 1 -- 172.21.15.60:0/4043628040 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f07140428e0 con 0x7f071c142ce0 2022-06-29T16:05:23.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.486+0000 7f0721b39700 1 -- 172.21.15.60:0/4043628040 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f071c0573d0 con 0x7f071c142ce0 2022-06-29T16:05:23.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.490+0000 7f0719ffb700 1 -- 172.21.15.60:0/4043628040 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v81) v1 ==== 110+0+10 (secure 0 0 0) 0x7f071406be60 con 0x7f071c142ce0 2022-06-29T16:05:23.497 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:05:23.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.490+0000 7f0721b39700 1 -- 172.21.15.60:0/4043628040 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f06fc053a70 msgr2=0x7f06fc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:23.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.490+0000 7f0721b39700 1 --2- 172.21.15.60:0/4043628040 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f06fc053a70 0x7f06fc055f30 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f070c01b000 tx=0x7f070c016000).stop 2022-06-29T16:05:23.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.490+0000 7f0721b39700 1 -- 172.21.15.60:0/4043628040 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f071c142ce0 msgr2=0x7f071c0ac990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:23.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.490+0000 7f0721b39700 1 --2- 172.21.15.60:0/4043628040 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f071c142ce0 0x7f071c0ac990 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0x7f071400a000 tx=0x7f0714008040).stop 2022-06-29T16:05:23.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.490+0000 7f0721b39700 1 -- 172.21.15.60:0/4043628040 shutdown_connections 2022-06-29T16:05:23.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.490+0000 7f0721b39700 1 --2- 172.21.15.60:0/4043628040 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f06fc053a70 0x7f06fc055f30 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:23.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.490+0000 7f0721b39700 1 --2- 172.21.15.60:0/4043628040 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f071c142ce0 0x7f071c0ac990 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:23.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.490+0000 7f0721b39700 1 --2- 172.21.15.60:0/4043628040 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f071c0b2b40 0x7f071c0b0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:23.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.490+0000 7f0721b39700 1 --2- 172.21.15.60:0/4043628040 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f071c0b1c90 0x7f071c141680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:23.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.490+0000 7f0721b39700 1 -- 172.21.15.60:0/4043628040 >> 172.21.15.60:0/4043628040 conn(0x7f071c0b6120 msgr2=0x7f071c143480 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:23.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.490+0000 7f0721b39700 1 -- 172.21.15.60:0/4043628040 shutdown_connections 2022-06-29T16:05:23.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.490+0000 7f0721b39700 1 -- 172.21.15.60:0/4043628040 wait complete. 2022-06-29T16:05:23.510 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:05:23.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.614+0000 7f6922c4e700 1 Processor -- start 2022-06-29T16:05:23.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.614+0000 7f6922c4e700 1 -- start start 2022-06-29T16:05:23.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.614+0000 7f6922c4e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f691c0b48b0 0x7f691c142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:23.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.614+0000 7f6922c4e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691c0b0cf0 con 0x7f691c0b5eb0 2022-06-29T16:05:23.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.614+0000 7f6922c4e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691c0b4d10 con 0x7f691c0b2c50 2022-06-29T16:05:23.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.614+0000 7f6922c4e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f691c0b54e0 con 0x7f691c0b48b0 2022-06-29T16:05:23.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.614+0000 7f692144b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f691c0b48b0 0x7f691c142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:23.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.614+0000 7f6921c4c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f691c0b5eb0 0x7f691c0b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58340/0 (socket says 172.21.15.60:58340) 2022-06-29T16:05:23.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.614+0000 7f6921c4c700 1 -- 172.21.15.60:0/2366546122 learned_addr learned my addr 172.21.15.60:0/2366546122 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:23.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.618+0000 7f6920c4a700 1 -- 172.21.15.60:0/2366546122 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1654617321 0 0) 0x7f691c0b4d10 con 0x7f691c0b2c50 2022-06-29T16:05:23.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.618+0000 7f692144b700 1 -- 172.21.15.60:0/2366546122 >> v1:172.21.15.92:6789/0 conn(0x7f691c0b2c50 legacy=0x7f691c14d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:23.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.618+0000 7f692144b700 1 -- 172.21.15.60:0/2366546122 >> v1:172.21.15.60:6789/0 conn(0x7f691c0b5eb0 legacy=0x7f691c0b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:23.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.618+0000 7f692144b700 1 -- 172.21.15.60:0/2366546122 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691c0b6290 con 0x7f691c0b48b0 2022-06-29T16:05:23.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.618+0000 7f692144b700 1 --2- 172.21.15.60:0/2366546122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f691c0b48b0 0x7f691c142a20 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 rx=0x7f69100012a0 tx=0x7f6918006df0).ready entity=mon.0 client_cookie=b67ee7f1b162aa57 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:23.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.618+0000 7f6920c4a700 1 -- 172.21.15.60:0/2366546122 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f691800d5c0 con 0x7f691c0b48b0 2022-06-29T16:05:23.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.618+0000 7f6920c4a700 1 -- 172.21.15.60:0/2366546122 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6918000e90 con 0x7f691c0b48b0 2022-06-29T16:05:23.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.618+0000 7f6920c4a700 1 -- 172.21.15.60:0/2366546122 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f691800d910 con 0x7f691c0b48b0 2022-06-29T16:05:23.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.618+0000 7f6922c4e700 1 -- 172.21.15.60:0/2366546122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f691c0b48b0 msgr2=0x7f691c142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:23.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.618+0000 7f6922c4e700 1 --2- 172.21.15.60:0/2366546122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f691c0b48b0 0x7f691c142a20 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 rx=0x7f69100012a0 tx=0x7f6918006df0).stop 2022-06-29T16:05:23.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.618+0000 7f6922c4e700 1 -- 172.21.15.60:0/2366546122 shutdown_connections 2022-06-29T16:05:23.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.618+0000 7f6922c4e700 1 --2- 172.21.15.60:0/2366546122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f691c0b48b0 0x7f691c142a20 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:23.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.618+0000 7f6922c4e700 1 -- 172.21.15.60:0/2366546122 >> 172.21.15.60:0/2366546122 conn(0x7f691c062090 msgr2=0x7f691c062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:23.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.618+0000 7f6922c4e700 1 -- 172.21.15.60:0/2366546122 shutdown_connections 2022-06-29T16:05:23.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.618+0000 7f6922c4e700 1 -- 172.21.15.60:0/2366546122 wait complete. 2022-06-29T16:05:23.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.622+0000 7f6922c4e700 1 Processor -- start 2022-06-29T16:05:23.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.622+0000 7f6922c4e700 1 -- start start 2022-06-29T16:05:23.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.622+0000 7f6922c4e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f691c0b2c50 0x7f691c1483b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:23.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.622+0000 7f6922c4e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f691c0b48b0 0x7f691c0acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:23.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.622+0000 7f6922c4e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f691c0b5eb0 0x7f691c0af190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:23.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.622+0000 7f6922c4e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691c13c9a0 con 0x7f691c0b5eb0 2022-06-29T16:05:23.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.622+0000 7f6922c4e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691c13cb70 con 0x7f691c0b2c50 2022-06-29T16:05:23.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.622+0000 7f6922c4e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f691c156150 con 0x7f691c0b48b0 2022-06-29T16:05:23.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.622+0000 7f692244d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f691c0b5eb0 0x7f691c0af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:23.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.622+0000 7f692244d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f691c0b5eb0 0x7f691c0af190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50264/0 (socket says 172.21.15.60:50264) 2022-06-29T16:05:23.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.622+0000 7f692244d700 1 -- 172.21.15.60:0/1961793449 learned_addr learned my addr 172.21.15.60:0/1961793449 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:23.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.622+0000 7f692144b700 1 --2- 172.21.15.60:0/1961793449 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f691c0b48b0 0x7f691c0acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:23.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.622+0000 7f6921c4c700 1 --2- 172.21.15.60:0/1961793449 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f691c0b2c50 0x7f691c1483b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:23.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.622+0000 7f692244d700 1 -- 172.21.15.60:0/1961793449 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f691c0b48b0 msgr2=0x7f691c0acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:23.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.622+0000 7f692244d700 1 --2- 172.21.15.60:0/1961793449 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f691c0b48b0 0x7f691c0acdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:23.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.622+0000 7f692244d700 1 -- 172.21.15.60:0/1961793449 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f691c0b2c50 msgr2=0x7f691c1483b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:23.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.622+0000 7f692244d700 1 --2- 172.21.15.60:0/1961793449 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f691c0b2c50 0x7f691c1483b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:23.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.622+0000 7f692244d700 1 -- 172.21.15.60:0/1961793449 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691800b180 con 0x7f691c0b5eb0 2022-06-29T16:05:23.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.622+0000 7f692244d700 1 --2- 172.21.15.60:0/1961793449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f691c0b5eb0 0x7f691c0af190 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 rx=0x7f6914001250 tx=0x7f6914007dc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:23.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.622+0000 7f690effd700 1 -- 172.21.15.60:0/1961793449 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f691400f470 con 0x7f691c0b5eb0 2022-06-29T16:05:23.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.626+0000 7f690effd700 1 -- 172.21.15.60:0/1961793449 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f691400f5d0 con 0x7f691c0b5eb0 2022-06-29T16:05:23.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.626+0000 7f6922c4e700 1 -- 172.21.15.60:0/1961793449 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f691c13cda0 con 0x7f691c0b5eb0 2022-06-29T16:05:23.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.626+0000 7f690effd700 1 -- 172.21.15.60:0/1961793449 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f691400f930 con 0x7f691c0b5eb0 2022-06-29T16:05:23.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.626+0000 7f6922c4e700 1 -- 172.21.15.60:0/1961793449 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f691c0ad2c0 con 0x7f691c0b5eb0 2022-06-29T16:05:23.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.626+0000 7f6922c4e700 1 -- 172.21.15.60:0/1961793449 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f691c13cf30 con 0x7f691c0b5eb0 2022-06-29T16:05:23.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.626+0000 7f690effd700 1 -- 172.21.15.60:0/1961793449 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6914019030 con 0x7f691c0b5eb0 2022-06-29T16:05:23.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.626+0000 7f690effd700 1 --2- 172.21.15.60:0/1961793449 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f68fc053a70 0x7f68fc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:23.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.626+0000 7f690effd700 1 -- 172.21.15.60:0/1961793449 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f691406b480 con 0x7f691c0b5eb0 2022-06-29T16:05:23.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.626+0000 7f6921c4c700 1 --2- 172.21.15.60:0/1961793449 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f68fc053a70 0x7f68fc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:23.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.626+0000 7f6921c4c700 1 --2- 172.21.15.60:0/1961793449 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f68fc053a70 0x7f68fc055f30 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f691000d000 tx=0x7f6910009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:23.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.634+0000 7f690effd700 1 -- 172.21.15.60:0/1961793449 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6914042400 con 0x7f691c0b5eb0 2022-06-29T16:05:23.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.758+0000 7f6922c4e700 1 -- 172.21.15.60:0/1961793449 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f691c0573d0 con 0x7f691c0b5eb0 2022-06-29T16:05:23.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.762+0000 7f690effd700 1 -- 172.21.15.60:0/1961793449 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v81) v1 ==== 92+0+10 (secure 0 0 0) 0x7f6914017070 con 0x7f691c0b5eb0 2022-06-29T16:05:23.769 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:05:23.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.762+0000 7f6922c4e700 1 -- 172.21.15.60:0/1961793449 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f68fc053a70 msgr2=0x7f68fc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:23.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.762+0000 7f6922c4e700 1 --2- 172.21.15.60:0/1961793449 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f68fc053a70 0x7f68fc055f30 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f691000d000 tx=0x7f6910009040).stop 2022-06-29T16:05:23.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.762+0000 7f6922c4e700 1 -- 172.21.15.60:0/1961793449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f691c0b5eb0 msgr2=0x7f691c0af190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:23.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.762+0000 7f6922c4e700 1 --2- 172.21.15.60:0/1961793449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f691c0b5eb0 0x7f691c0af190 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 rx=0x7f6914001250 tx=0x7f6914007dc0).stop 2022-06-29T16:05:23.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.762+0000 7f6922c4e700 1 -- 172.21.15.60:0/1961793449 shutdown_connections 2022-06-29T16:05:23.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.762+0000 7f6922c4e700 1 --2- 172.21.15.60:0/1961793449 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f68fc053a70 0x7f68fc055f30 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:23.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.762+0000 7f6922c4e700 1 --2- 172.21.15.60:0/1961793449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f691c0b5eb0 0x7f691c0af190 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:23.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.762+0000 7f6922c4e700 1 --2- 172.21.15.60:0/1961793449 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f691c0b48b0 0x7f691c0acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:23.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.762+0000 7f6922c4e700 1 --2- 172.21.15.60:0/1961793449 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f691c0b2c50 0x7f691c1483b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:23.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.762+0000 7f6922c4e700 1 -- 172.21.15.60:0/1961793449 >> 172.21.15.60:0/1961793449 conn(0x7f691c062090 msgr2=0x7f691c0b17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:23.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.762+0000 7f6922c4e700 1 -- 172.21.15.60:0/1961793449 shutdown_connections 2022-06-29T16:05:23.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.762+0000 7f6922c4e700 1 -- 172.21.15.60:0/1961793449 wait complete. 2022-06-29T16:05:23.782 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-29T16:05:23.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.882+0000 7f3e45532700 1 Processor -- start 2022-06-29T16:05:23.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.882+0000 7f3e45532700 1 -- start start 2022-06-29T16:05:23.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.882+0000 7f3e45532700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e400b0610 0x7f3e400b24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:23.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.882+0000 7f3e45532700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3e4005bd20 0x7f3e4005c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:23.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.882+0000 7f3e45532700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e400b1de0 con 0x7f3e400b9c20 2022-06-29T16:05:23.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.882+0000 7f3e45532700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e400b1330 con 0x7f3e4005bd20 2022-06-29T16:05:23.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.882+0000 7f3e45532700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e40146750 con 0x7f3e400b0610 2022-06-29T16:05:23.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.886+0000 7f3e44d31700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3e4005bd20 0x7f3e4005c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:23.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.886+0000 7f3e3ffff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f3e400b9c20 0x7f3e40146040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58354/0 (socket says 172.21.15.60:58354) 2022-06-29T16:05:23.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.886+0000 7f3e3ffff700 1 -- 172.21.15.60:0/1185671799 learned_addr learned my addr 172.21.15.60:0/1185671799 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:23.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.886+0000 7f3e3f7fe700 1 --2- 172.21.15.60:0/1185671799 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e400b0610 0x7f3e400b24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:23.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.886+0000 7f3e3f7fe700 1 -- 172.21.15.60:0/1185671799 >> v2:172.21.15.60:3300/0 conn(0x7f3e4005bd20 msgr2=0x7f3e4005c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:23.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.886+0000 7f3e3f7fe700 1 --2- 172.21.15.60:0/1185671799 >> v2:172.21.15.60:3300/0 conn(0x7f3e4005bd20 0x7f3e4005c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:23.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.886+0000 7f3e3f7fe700 1 -- 172.21.15.60:0/1185671799 >> v1:172.21.15.60:6789/0 conn(0x7f3e400b9c20 legacy=0x7f3e40146040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:23.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.886+0000 7f3e3f7fe700 1 -- 172.21.15.60:0/1185671799 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e400ba000 con 0x7f3e400b0610 2022-06-29T16:05:23.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.886+0000 7f3e3f7fe700 1 --2- 172.21.15.60:0/1185671799 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e400b0610 0x7f3e400b24b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f3e28002f70 tx=0x7f3e28006df0).ready entity=mon.2 client_cookie=255df2a66053bf9c server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:23.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.886+0000 7f3e3effd700 1 -- 172.21.15.60:0/1185671799 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e2800d570 con 0x7f3e400b0610 2022-06-29T16:05:23.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.886+0000 7f3e3effd700 1 -- 172.21.15.60:0/1185671799 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e28000e90 con 0x7f3e400b0610 2022-06-29T16:05:23.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.886+0000 7f3e3effd700 1 -- 172.21.15.60:0/1185671799 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e2800d8c0 con 0x7f3e400b0610 2022-06-29T16:05:23.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.886+0000 7f3e45532700 1 -- 172.21.15.60:0/1185671799 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e400b0610 msgr2=0x7f3e400b24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:23.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.886+0000 7f3e45532700 1 --2- 172.21.15.60:0/1185671799 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e400b0610 0x7f3e400b24b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f3e28002f70 tx=0x7f3e28006df0).stop 2022-06-29T16:05:23.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.886+0000 7f3e45532700 1 -- 172.21.15.60:0/1185671799 shutdown_connections 2022-06-29T16:05:23.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.886+0000 7f3e45532700 1 --2- 172.21.15.60:0/1185671799 >> v2:172.21.15.60:3300/0 conn(0x7f3e4005bd20 0x7f3e4005c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:23.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.886+0000 7f3e45532700 1 --2- 172.21.15.60:0/1185671799 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e400b0610 0x7f3e400b24b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:23.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.886+0000 7f3e45532700 1 -- 172.21.15.60:0/1185671799 >> 172.21.15.60:0/1185671799 conn(0x7f3e40062cb0 msgr2=0x7f3e400b91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:23.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.886+0000 7f3e45532700 1 -- 172.21.15.60:0/1185671799 shutdown_connections 2022-06-29T16:05:23.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.886+0000 7f3e45532700 1 -- 172.21.15.60:0/1185671799 wait complete. 2022-06-29T16:05:23.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.890+0000 7f3e45532700 1 Processor -- start 2022-06-29T16:05:23.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.890+0000 7f3e45532700 1 -- start start 2022-06-29T16:05:23.903 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.890+0000 7f3e45532700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e4005bd20 0x7f3e400b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:23.903 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.890+0000 7f3e45532700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e400b0610 0x7f3e401158e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:23.903 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.890+0000 7f3e45532700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e400b9c20 0x7f3e401555a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:23.903 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.890+0000 7f3e45532700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4013f590 con 0x7f3e400b9c20 2022-06-29T16:05:23.903 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.890+0000 7f3e45532700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e40113af0 con 0x7f3e4005bd20 2022-06-29T16:05:23.904 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.890+0000 7f3e45532700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e400af170 con 0x7f3e400b0610 2022-06-29T16:05:23.904 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.890+0000 7f3e44d31700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e400b9c20 0x7f3e401555a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:23.904 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.890+0000 7f3e44d31700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e400b9c20 0x7f3e401555a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50278/0 (socket says 172.21.15.60:50278) 2022-06-29T16:05:23.904 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.890+0000 7f3e44d31700 1 -- 172.21.15.60:0/2554966602 learned_addr learned my addr 172.21.15.60:0/2554966602 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:23.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.890+0000 7f3e3f7fe700 1 --2- 172.21.15.60:0/2554966602 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e400b0610 0x7f3e401158e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:23.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.890+0000 7f3e3ffff700 1 --2- 172.21.15.60:0/2554966602 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e4005bd20 0x7f3e400b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:23.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.890+0000 7f3e44d31700 1 -- 172.21.15.60:0/2554966602 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e400b0610 msgr2=0x7f3e401158e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:23.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.890+0000 7f3e44d31700 1 --2- 172.21.15.60:0/2554966602 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e400b0610 0x7f3e401158e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:23.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.890+0000 7f3e44d31700 1 -- 172.21.15.60:0/2554966602 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e4005bd20 msgr2=0x7f3e400b50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:23.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.890+0000 7f3e44d31700 1 --2- 172.21.15.60:0/2554966602 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e4005bd20 0x7f3e400b50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:23.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.890+0000 7f3e44d31700 1 -- 172.21.15.60:0/2554966602 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e2800b180 con 0x7f3e400b9c20 2022-06-29T16:05:23.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.890+0000 7f3e44d31700 1 --2- 172.21.15.60:0/2554966602 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e400b9c20 0x7f3e401555a0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 rx=0x7f3e38006e00 tx=0x7f3e380091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:23.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.890+0000 7f3e3d7fa700 1 -- 172.21.15.60:0/2554966602 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e3800c640 con 0x7f3e400b9c20 2022-06-29T16:05:23.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.890+0000 7f3e3d7fa700 1 -- 172.21.15.60:0/2554966602 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e3800c7a0 con 0x7f3e400b9c20 2022-06-29T16:05:23.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.890+0000 7f3e45532700 1 -- 172.21.15.60:0/2554966602 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e400af410 con 0x7f3e400b9c20 2022-06-29T16:05:23.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.890+0000 7f3e3d7fa700 1 -- 172.21.15.60:0/2554966602 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e3800cb00 con 0x7f3e400b9c20 2022-06-29T16:05:23.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.890+0000 7f3e45532700 1 -- 172.21.15.60:0/2554966602 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e401551c0 con 0x7f3e400b9c20 2022-06-29T16:05:23.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.894+0000 7f3e45532700 1 -- 172.21.15.60:0/2554966602 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e400ad330 con 0x7f3e400b9c20 2022-06-29T16:05:23.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.894+0000 7f3e3d7fa700 1 -- 172.21.15.60:0/2554966602 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3e38022070 con 0x7f3e400b9c20 2022-06-29T16:05:23.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.894+0000 7f3e3d7fa700 1 --2- 172.21.15.60:0/2554966602 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3e24053a70 0x7f3e24055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:23.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.894+0000 7f3e3d7fa700 1 -- 172.21.15.60:0/2554966602 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f3e3806c500 con 0x7f3e400b9c20 2022-06-29T16:05:23.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.894+0000 7f3e3ffff700 1 --2- 172.21.15.60:0/2554966602 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3e24053a70 0x7f3e24055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:23.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.894+0000 7f3e3ffff700 1 --2- 172.21.15.60:0/2554966602 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3e24053a70 0x7f3e24055f30 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f3e3000d000 tx=0x7f3e30000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:23.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:23.898+0000 7f3e3d7fa700 1 -- 172.21.15.60:0/2554966602 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3e380433d0 con 0x7f3e400b9c20 2022-06-29T16:05:24.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.022+0000 7f3e45532700 1 -- 172.21.15.60:0/2554966602 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3e400573d0 con 0x7f3e400b9c20 2022-06-29T16:05:24.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.022+0000 7f3e3d7fa700 1 -- 172.21.15.60:0/2554966602 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) v1 ==== 104+0+10 (secure 0 0 0) 0x7f3e380469f0 con 0x7f3e400b9c20 2022-06-29T16:05:24.031 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:05:24.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.026+0000 7f3e45532700 1 -- 172.21.15.60:0/2554966602 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3e24053a70 msgr2=0x7f3e24055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:24.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.026+0000 7f3e45532700 1 --2- 172.21.15.60:0/2554966602 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3e24053a70 0x7f3e24055f30 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f3e3000d000 tx=0x7f3e30000bc0).stop 2022-06-29T16:05:24.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.026+0000 7f3e45532700 1 -- 172.21.15.60:0/2554966602 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e400b9c20 msgr2=0x7f3e401555a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:24.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.026+0000 7f3e45532700 1 --2- 172.21.15.60:0/2554966602 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e400b9c20 0x7f3e401555a0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 rx=0x7f3e38006e00 tx=0x7f3e380091d0).stop 2022-06-29T16:05:24.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.026+0000 7f3e45532700 1 -- 172.21.15.60:0/2554966602 shutdown_connections 2022-06-29T16:05:24.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.026+0000 7f3e45532700 1 --2- 172.21.15.60:0/2554966602 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3e24053a70 0x7f3e24055f30 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:24.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.026+0000 7f3e45532700 1 --2- 172.21.15.60:0/2554966602 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e400b9c20 0x7f3e401555a0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:24.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.026+0000 7f3e45532700 1 --2- 172.21.15.60:0/2554966602 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e400b0610 0x7f3e401158e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:24.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.026+0000 7f3e45532700 1 --2- 172.21.15.60:0/2554966602 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e4005bd20 0x7f3e400b50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:24.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.026+0000 7f3e45532700 1 -- 172.21.15.60:0/2554966602 >> 172.21.15.60:0/2554966602 conn(0x7f3e40062cb0 msgr2=0x7f3e40140ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:24.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.026+0000 7f3e45532700 1 -- 172.21.15.60:0/2554966602 shutdown_connections 2022-06-29T16:05:24.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.026+0000 7f3e45532700 1 -- 172.21.15.60:0/2554966602 wait complete. 2022-06-29T16:05:24.044 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-29T16:05:24.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.142+0000 7f461f4f1700 1 Processor -- start 2022-06-29T16:05:24.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.142+0000 7f461f4f1700 1 -- start start 2022-06-29T16:05:24.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.146+0000 7f461f4f1700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f461805f0b0 0x7f46180b0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:24.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.146+0000 7f461f4f1700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f461805f970 0x7f461805fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:24.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.146+0000 7f461f4f1700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4618146f40 con 0x7f461805c750 2022-06-29T16:05:24.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.146+0000 7f461f4f1700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4618147170 con 0x7f461805f970 2022-06-29T16:05:24.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.146+0000 7f461f4f1700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4618147410 con 0x7f461805f0b0 2022-06-29T16:05:24.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.146+0000 7f461dcee700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f461805f0b0 0x7f46180b0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:24.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.146+0000 7f461ecf0700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f461805f970 0x7f461805fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:24.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.146+0000 7f461dcee700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f461805f0b0 0x7f46180b0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:35146/0 (socket says 172.21.15.60:35146) 2022-06-29T16:05:24.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.146+0000 7f461dcee700 1 -- 172.21.15.60:0/2881911002 learned_addr learned my addr 172.21.15.60:0/2881911002 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:24.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.146+0000 7f461dcee700 1 -- 172.21.15.60:0/2881911002 >> v2:172.21.15.92:3300/0 conn(0x7f461805f970 msgr2=0x7f461805fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:24.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.146+0000 7f461dcee700 1 --2- 172.21.15.60:0/2881911002 >> v2:172.21.15.92:3300/0 conn(0x7f461805f970 0x7f461805fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:24.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.146+0000 7f461dcee700 1 -- 172.21.15.60:0/2881911002 >> v1:172.21.15.92:6789/0 conn(0x7f461805c750 legacy=0x7f46181478b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:24.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.146+0000 7f461dcee700 1 -- 172.21.15.60:0/2881911002 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46180b1070 con 0x7f461805f0b0 2022-06-29T16:05:24.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.146+0000 7f461ecf0700 1 --2- 172.21.15.60:0/2881911002 >> v2:172.21.15.92:3300/0 conn(0x7f461805f970 0x7f461805fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:05:24.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.146+0000 7f461dcee700 1 --2- 172.21.15.60:0/2881911002 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f461805f0b0 0x7f46180b0b80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f460c002f70 tx=0x7f460c006eb0).ready entity=mon.2 client_cookie=a91d388456609cd0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:24.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.146+0000 7f461d4ed700 1 -- 172.21.15.60:0/2881911002 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f460c00d570 con 0x7f461805f0b0 2022-06-29T16:05:24.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.146+0000 7f461d4ed700 1 -- 172.21.15.60:0/2881911002 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f460c000e90 con 0x7f461805f0b0 2022-06-29T16:05:24.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.146+0000 7f461d4ed700 1 -- 172.21.15.60:0/2881911002 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f460c00d900 con 0x7f461805f0b0 2022-06-29T16:05:24.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.146+0000 7f461f4f1700 1 -- 172.21.15.60:0/2881911002 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f461805f0b0 msgr2=0x7f46180b0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:24.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.146+0000 7f461f4f1700 1 --2- 172.21.15.60:0/2881911002 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f461805f0b0 0x7f46180b0b80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f460c002f70 tx=0x7f460c006eb0).stop 2022-06-29T16:05:24.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.146+0000 7f461f4f1700 1 -- 172.21.15.60:0/2881911002 shutdown_connections 2022-06-29T16:05:24.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.146+0000 7f461f4f1700 1 --2- 172.21.15.60:0/2881911002 >> v2:172.21.15.92:3300/0 conn(0x7f461805f970 0x7f461805fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:24.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.146+0000 7f461f4f1700 1 --2- 172.21.15.60:0/2881911002 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f461805f0b0 0x7f46180b0b80 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:24.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.146+0000 7f461f4f1700 1 -- 172.21.15.60:0/2881911002 >> 172.21.15.60:0/2881911002 conn(0x7f4618061c80 msgr2=0x7f46180637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:24.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.146+0000 7f461f4f1700 1 -- 172.21.15.60:0/2881911002 shutdown_connections 2022-06-29T16:05:24.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.146+0000 7f461f4f1700 1 -- 172.21.15.60:0/2881911002 wait complete. 2022-06-29T16:05:24.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.150+0000 7f461f4f1700 1 Processor -- start 2022-06-29T16:05:24.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.150+0000 7f461f4f1700 1 -- start start 2022-06-29T16:05:24.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.150+0000 7f461f4f1700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f461805c750 0x7f4618065450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:24.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.150+0000 7f461f4f1700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f461805f0b0 0x7f4618145570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:24.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.150+0000 7f461f4f1700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f461805f970 0x7f4618145a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:24.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.150+0000 7f461f4f1700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4618067130 con 0x7f461805c750 2022-06-29T16:05:24.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.150+0000 7f461f4f1700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46180b3ce0 con 0x7f461805f970 2022-06-29T16:05:24.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.150+0000 7f461f4f1700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4618064670 con 0x7f461805f0b0 2022-06-29T16:05:24.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.150+0000 7f461e4ef700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f461805c750 0x7f4618065450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:24.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.150+0000 7f461dcee700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f461805f0b0 0x7f4618145570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:24.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.150+0000 7f461e4ef700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f461805c750 0x7f4618065450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50288/0 (socket says 172.21.15.60:50288) 2022-06-29T16:05:24.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.150+0000 7f461ecf0700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f461805f970 0x7f4618145a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:24.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.150+0000 7f461e4ef700 1 -- 172.21.15.60:0/4014809197 learned_addr learned my addr 172.21.15.60:0/4014809197 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:24.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.150+0000 7f461e4ef700 1 -- 172.21.15.60:0/4014809197 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f461805f0b0 msgr2=0x7f4618145570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:24.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.150+0000 7f461e4ef700 1 --2- 172.21.15.60:0/4014809197 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f461805f0b0 0x7f4618145570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:24.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.150+0000 7f461e4ef700 1 -- 172.21.15.60:0/4014809197 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f461805f970 msgr2=0x7f4618145a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:24.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.150+0000 7f461e4ef700 1 --2- 172.21.15.60:0/4014809197 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f461805f970 0x7f4618145a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:24.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.150+0000 7f461e4ef700 1 -- 172.21.15.60:0/4014809197 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4610009470 con 0x7f461805c750 2022-06-29T16:05:24.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.150+0000 7f461e4ef700 1 --2- 172.21.15.60:0/4014809197 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f461805c750 0x7f4618065450 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 rx=0x7f46180b3a50 tx=0x7f4614007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:24.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.154+0000 7f460b7fe700 1 -- 172.21.15.60:0/4014809197 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4614012400 con 0x7f461805c750 2022-06-29T16:05:24.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.154+0000 7f460b7fe700 1 -- 172.21.15.60:0/4014809197 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4614012560 con 0x7f461805c750 2022-06-29T16:05:24.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.154+0000 7f460b7fe700 1 -- 172.21.15.60:0/4014809197 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f46140128c0 con 0x7f461805c750 2022-06-29T16:05:24.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.154+0000 7f461f4f1700 1 -- 172.21.15.60:0/4014809197 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f460c00b1e0 con 0x7f461805c750 2022-06-29T16:05:24.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.154+0000 7f461f4f1700 1 -- 172.21.15.60:0/4014809197 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46180648a0 con 0x7f461805c750 2022-06-29T16:05:24.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.154+0000 7f461f4f1700 1 -- 172.21.15.60:0/4014809197 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4618064a30 con 0x7f461805c750 2022-06-29T16:05:24.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.154+0000 7f460b7fe700 1 -- 172.21.15.60:0/4014809197 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4614019030 con 0x7f461805c750 2022-06-29T16:05:24.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.154+0000 7f460b7fe700 1 --2- 172.21.15.60:0/4014809197 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f45f8053a70 0x7f45f8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:24.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.154+0000 7f460b7fe700 1 -- 172.21.15.60:0/4014809197 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f461406b300 con 0x7f461805c750 2022-06-29T16:05:24.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.154+0000 7f461dcee700 1 --2- 172.21.15.60:0/4014809197 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f45f8053a70 0x7f45f8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:24.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.158+0000 7f461dcee700 1 --2- 172.21.15.60:0/4014809197 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f45f8053a70 0x7f45f8055f30 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f460c01c000 tx=0x7f460c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:24.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.162+0000 7f460b7fe700 1 -- 172.21.15.60:0/4014809197 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f46140421d0 con 0x7f461805c750 2022-06-29T16:05:24.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.290+0000 7f461f4f1700 1 -- 172.21.15.60:0/4014809197 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f46180573d0 con 0x7f461805c750 2022-06-29T16:05:24.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.294+0000 7f460b7fe700 1 -- 172.21.15.60:0/4014809197 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) v1 ==== 100+0+10 (secure 0 0 0) 0x7f46140457f0 con 0x7f461805c750 2022-06-29T16:05:24.300 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:05:24.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.294+0000 7f461f4f1700 1 -- 172.21.15.60:0/4014809197 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f45f8053a70 msgr2=0x7f45f8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:24.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.294+0000 7f461f4f1700 1 --2- 172.21.15.60:0/4014809197 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f45f8053a70 0x7f45f8055f30 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f460c01c000 tx=0x7f460c015000).stop 2022-06-29T16:05:24.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.294+0000 7f461f4f1700 1 -- 172.21.15.60:0/4014809197 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f461805c750 msgr2=0x7f4618065450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:24.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.294+0000 7f461f4f1700 1 --2- 172.21.15.60:0/4014809197 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f461805c750 0x7f4618065450 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 rx=0x7f46180b3a50 tx=0x7f4614007c50).stop 2022-06-29T16:05:24.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.294+0000 7f461f4f1700 1 -- 172.21.15.60:0/4014809197 shutdown_connections 2022-06-29T16:05:24.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.294+0000 7f461f4f1700 1 --2- 172.21.15.60:0/4014809197 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f45f8053a70 0x7f45f8055f30 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:24.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.294+0000 7f461f4f1700 1 --2- 172.21.15.60:0/4014809197 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f461805f970 0x7f4618145a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:24.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.294+0000 7f461f4f1700 1 --2- 172.21.15.60:0/4014809197 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f461805f0b0 0x7f4618145570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:24.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.294+0000 7f461f4f1700 1 --2- 172.21.15.60:0/4014809197 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f461805c750 0x7f4618065450 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:24.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.294+0000 7f461f4f1700 1 -- 172.21.15.60:0/4014809197 >> 172.21.15.60:0/4014809197 conn(0x7f4618061c80 msgr2=0x7f461805cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:24.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.294+0000 7f461f4f1700 1 -- 172.21.15.60:0/4014809197 shutdown_connections 2022-06-29T16:05:24.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.294+0000 7f461f4f1700 1 -- 172.21.15.60:0/4014809197 wait complete. 2022-06-29T16:05:24.314 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-06-29T16:05:24.314 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:05:24.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.406+0000 7f1c39fd5700 1 Processor -- start 2022-06-29T16:05:24.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.406+0000 7f1c39fd5700 1 -- start start 2022-06-29T16:05:24.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.406+0000 7f1c39fd5700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1c340acbc0 0x7f1c3414a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:24.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.406+0000 7f1c39fd5700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1c340b24f0 0x7f1c34146410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:24.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.406+0000 7f1c39fd5700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c340ad9d0 con 0x7f1c3405cf40 2022-06-29T16:05:24.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.406+0000 7f1c39fd5700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c340ad460 con 0x7f1c340b24f0 2022-06-29T16:05:24.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.406+0000 7f1c39fd5700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c340adc00 con 0x7f1c340acbc0 2022-06-29T16:05:24.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.406+0000 7f1c32ffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1c340b24f0 0x7f1c34146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:24.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.406+0000 7f1c337fe700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1c340acbc0 0x7f1c3414a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:24.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.406+0000 7f1c32ffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1c340b24f0 0x7f1c34146410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50298/0 (socket says 172.21.15.60:50298) 2022-06-29T16:05:24.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.410+0000 7f1c32ffd700 1 -- 172.21.15.60:0/552377965 learned_addr learned my addr 172.21.15.60:0/552377965 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:24.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.410+0000 7f1c33fff700 1 --1- 172.21.15.60:0/552377965 >> v1:172.21.15.92:6789/0 conn(0x7f1c3405cf40 0x7f1c34065da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:34402/0 (socket says 172.21.15.60:34402) 2022-06-29T16:05:24.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.410+0000 7f1c337fe700 1 -- 172.21.15.60:0/552377965 >> v2:172.21.15.60:3300/0 conn(0x7f1c340b24f0 msgr2=0x7f1c34146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:24.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.410+0000 7f1c337fe700 1 --2- 172.21.15.60:0/552377965 >> v2:172.21.15.60:3300/0 conn(0x7f1c340b24f0 0x7f1c34146410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:24.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.410+0000 7f1c337fe700 1 -- 172.21.15.60:0/552377965 >> v1:172.21.15.92:6789/0 conn(0x7f1c3405cf40 legacy=0x7f1c34065da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:24.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.410+0000 7f1c337fe700 1 -- 172.21.15.60:0/552377965 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c340adea0 con 0x7f1c340acbc0 2022-06-29T16:05:24.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.410+0000 7f1c337fe700 1 --2- 172.21.15.60:0/552377965 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c340acbc0 0x7f1c3414a560 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f1c2c0032f0 tx=0x7f1c2c0091d0).ready entity=mon.1 client_cookie=8fb551ea53570c3d server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:24.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.410+0000 7f1c327fc700 1 -- 172.21.15.60:0/552377965 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c2c00b680 con 0x7f1c340acbc0 2022-06-29T16:05:24.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.410+0000 7f1c327fc700 1 -- 172.21.15.60:0/552377965 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c2c00b7e0 con 0x7f1c340acbc0 2022-06-29T16:05:24.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.410+0000 7f1c327fc700 1 -- 172.21.15.60:0/552377965 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c2c00b950 con 0x7f1c340acbc0 2022-06-29T16:05:24.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.410+0000 7f1c39fd5700 1 -- 172.21.15.60:0/552377965 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c340acbc0 msgr2=0x7f1c3414a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:24.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.410+0000 7f1c39fd5700 1 --2- 172.21.15.60:0/552377965 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c340acbc0 0x7f1c3414a560 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f1c2c0032f0 tx=0x7f1c2c0091d0).stop 2022-06-29T16:05:24.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.410+0000 7f1c39fd5700 1 -- 172.21.15.60:0/552377965 shutdown_connections 2022-06-29T16:05:24.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.410+0000 7f1c39fd5700 1 --2- 172.21.15.60:0/552377965 >> v2:172.21.15.60:3300/0 conn(0x7f1c340b24f0 0x7f1c34146410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:24.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.410+0000 7f1c39fd5700 1 --2- 172.21.15.60:0/552377965 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c340acbc0 0x7f1c3414a560 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:24.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.410+0000 7f1c39fd5700 1 -- 172.21.15.60:0/552377965 >> 172.21.15.60:0/552377965 conn(0x7f1c340b7140 msgr2=0x7f1c340b7520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:24.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.410+0000 7f1c39fd5700 1 -- 172.21.15.60:0/552377965 shutdown_connections 2022-06-29T16:05:24.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.410+0000 7f1c39fd5700 1 -- 172.21.15.60:0/552377965 wait complete. 2022-06-29T16:05:24.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.414+0000 7f1c39fd5700 1 Processor -- start 2022-06-29T16:05:24.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.414+0000 7f1c39fd5700 1 -- start start 2022-06-29T16:05:24.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.414+0000 7f1c39fd5700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c3405cf40 0x7f1c340ae5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:24.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.414+0000 7f1c39fd5700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c340acbc0 0x7f1c341139c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:24.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.414+0000 7f1c39fd5700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c340b24f0 0x7f1c34113eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:24.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.414+0000 7f1c39fd5700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c34159a70 con 0x7f1c340b24f0 2022-06-29T16:05:24.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.414+0000 7f1c39fd5700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c34159d10 con 0x7f1c340acbc0 2022-06-29T16:05:24.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.414+0000 7f1c39fd5700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c34159fb0 con 0x7f1c3405cf40 2022-06-29T16:05:24.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.414+0000 7f1c33fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c340b24f0 0x7f1c34113eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:24.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.414+0000 7f1c337fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c3405cf40 0x7f1c340ae5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:24.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.414+0000 7f1c337fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c3405cf40 0x7f1c340ae5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:35164/0 (socket says 172.21.15.60:35164) 2022-06-29T16:05:24.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.414+0000 7f1c33fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c340b24f0 0x7f1c34113eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50306/0 (socket says 172.21.15.60:50306) 2022-06-29T16:05:24.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.414+0000 7f1c32ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c340acbc0 0x7f1c341139c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:24.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.414+0000 7f1c337fe700 1 -- 172.21.15.60:0/1468083941 learned_addr learned my addr 172.21.15.60:0/1468083941 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:24.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.414+0000 7f1c337fe700 1 -- 172.21.15.60:0/1468083941 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c340acbc0 msgr2=0x7f1c341139c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:24.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.414+0000 7f1c337fe700 1 --2- 172.21.15.60:0/1468083941 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c340acbc0 0x7f1c341139c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:24.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.414+0000 7f1c337fe700 1 -- 172.21.15.60:0/1468083941 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c340b24f0 msgr2=0x7f1c34113eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:24.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.414+0000 7f1c337fe700 1 --2- 172.21.15.60:0/1468083941 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c340b24f0 0x7f1c34113eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:24.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.414+0000 7f1c337fe700 1 -- 172.21.15.60:0/1468083941 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c2c009470 con 0x7f1c3405cf40 2022-06-29T16:05:24.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.414+0000 7f1c33fff700 1 --2- 172.21.15.60:0/1468083941 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c340b24f0 0x7f1c34113eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:05:24.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.414+0000 7f1c337fe700 1 --2- 172.21.15.60:0/1468083941 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c3405cf40 0x7f1c340ae5c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f1c2c0032f0 tx=0x7f1c2c016000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:24.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.414+0000 7f1c30ff9700 1 -- 172.21.15.60:0/1468083941 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c2c011e50 con 0x7f1c3405cf40 2022-06-29T16:05:24.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.414+0000 7f1c30ff9700 1 -- 172.21.15.60:0/1468083941 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c2c021440 con 0x7f1c3405cf40 2022-06-29T16:05:24.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.414+0000 7f1c39fd5700 1 -- 172.21.15.60:0/1468083941 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c341596f0 con 0x7f1c3405cf40 2022-06-29T16:05:24.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.414+0000 7f1c30ff9700 1 -- 172.21.15.60:0/1468083941 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c2c0217a0 con 0x7f1c3405cf40 2022-06-29T16:05:24.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.414+0000 7f1c39fd5700 1 -- 172.21.15.60:0/1468083941 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c3414a480 con 0x7f1c3405cf40 2022-06-29T16:05:24.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.418+0000 7f1c39fd5700 1 -- 172.21.15.60:0/1468083941 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c3414a640 con 0x7f1c3405cf40 2022-06-29T16:05:24.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.418+0000 7f1c30ff9700 1 -- 172.21.15.60:0/1468083941 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1c2c00f070 con 0x7f1c3405cf40 2022-06-29T16:05:24.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.418+0000 7f1c30ff9700 1 --2- 172.21.15.60:0/1468083941 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1c18053b30 0x7f1c18055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:24.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.418+0000 7f1c30ff9700 1 -- 172.21.15.60:0/1468083941 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f1c2c06d490 con 0x7f1c3405cf40 2022-06-29T16:05:24.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.418+0000 7f1c32ffd700 1 --2- 172.21.15.60:0/1468083941 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1c18053b30 0x7f1c18055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:24.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.418+0000 7f1c32ffd700 1 --2- 172.21.15.60:0/1468083941 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1c18053b30 0x7f1c18055ff0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f1c1c008000 tx=0x7f1c1c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:24.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.430+0000 7f1c30ff9700 1 -- 172.21.15.60:0/1468083941 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1c2c044250 con 0x7f1c3405cf40 2022-06-29T16:05:24.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.594+0000 7f1c39fd5700 1 -- 172.21.15.60:0/1468083941 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c3405d990 con 0x7f1c3405cf40 2022-06-29T16:05:24.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.594+0000 7f1c30ff9700 1 -- 172.21.15.60:0/1468083941 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+2141 (secure 0 0 0) 0x7f1c2c015020 con 0x7f1c3405cf40 2022-06-29T16:05:24.604 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:24.604 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":36,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6657,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.60:6833/3121204394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3121204394},{"type":"v1","addr":"172.21.15.60:6833","nonce":3121204394}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":34}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:05:16.293932+0000","modified":"2022-06-29T16:05:18.324125+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6688},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6688":{"gid":6688,"name":"b","rank":0,"incarnation":35,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/1245503283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1245503283},{"type":"v1","addr":"172.21.15.92:6835","nonce":1245503283}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-06-29T16:05:24.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.598+0000 7f1c39fd5700 1 -- 172.21.15.60:0/1468083941 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1c18053b30 msgr2=0x7f1c18055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:24.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.598+0000 7f1c39fd5700 1 --2- 172.21.15.60:0/1468083941 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1c18053b30 0x7f1c18055ff0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f1c1c008000 tx=0x7f1c1c008040).stop 2022-06-29T16:05:24.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.598+0000 7f1c39fd5700 1 -- 172.21.15.60:0/1468083941 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c3405cf40 msgr2=0x7f1c340ae5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:24.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.598+0000 7f1c39fd5700 1 --2- 172.21.15.60:0/1468083941 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c3405cf40 0x7f1c340ae5c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f1c2c0032f0 tx=0x7f1c2c016000).stop 2022-06-29T16:05:24.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.598+0000 7f1c39fd5700 1 -- 172.21.15.60:0/1468083941 shutdown_connections 2022-06-29T16:05:24.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.598+0000 7f1c39fd5700 1 --2- 172.21.15.60:0/1468083941 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1c18053b30 0x7f1c18055ff0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:24.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.598+0000 7f1c39fd5700 1 --2- 172.21.15.60:0/1468083941 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c340b24f0 0x7f1c34113eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:24.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.598+0000 7f1c39fd5700 1 --2- 172.21.15.60:0/1468083941 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c340acbc0 0x7f1c341139c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:24.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.598+0000 7f1c39fd5700 1 --2- 172.21.15.60:0/1468083941 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c3405cf40 0x7f1c340ae5c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:24.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.598+0000 7f1c39fd5700 1 -- 172.21.15.60:0/1468083941 >> 172.21.15.60:0/1468083941 conn(0x7f1c340b7140 msgr2=0x7f1c340b5d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:24.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.598+0000 7f1c39fd5700 1 -- 172.21.15.60:0/1468083941 shutdown_connections 2022-06-29T16:05:24.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:24.598+0000 7f1c39fd5700 1 -- 172.21.15.60:0/1468083941 wait complete. 2022-06-29T16:05:24.609 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 36 2022-06-29T16:05:24.818 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 36, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:05:16.293932+0000', 'modified': '2022-06-29T16:05:18.324125+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 81, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6688}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6688': {'gid': 6688, 'name': 'b', 'rank': 0, 'incarnation': 35, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.92:6835/1245503283', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 1245503283}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 1245503283}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:05:24.819 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:05:24.819 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-06-29T16:05:24.934 INFO:teuthology.orchestra.run.smithi092.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":6688,"want_state":"up:active","state":"up:active","rank_uptime":7.6093674299999998,"mdsmap_epoch":36,"osdmap_epoch":81,"osdmap_epoch_barrier":81,"uptime":12.298586440999999} 2022-06-29T16:05:24.935 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 6688, "mdsmap_epoch": 36, "osdmap_epoch": 81, "osdmap_epoch_barrier": 81, "rank_uptime": 7.60936743, "state": "up:active", "uptime": 12.298586441, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:05:24.936 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-06-29T16:05:24.936 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-06-29T16:05:24.936 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi156.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-06-29T16:05:24.937 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:24.937 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:05:24.945 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2022-06-29T16:05:24.960 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:05:24.961 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:05:24.973 INFO:teuthology.orchestra.run.smithi156.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:05:24.974 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:05:24.975 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:24.975 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2022-06-29T16:05:25.025 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-06-29T16:05:25.025 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:05:25.028 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2022-06-29T16:05:25.046 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:05:25.047 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:05:25.056 INFO:teuthology.orchestra.run.smithi156.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:05:25.056 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:05:25.057 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:25.058 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2022-06-29T16:05:25.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:2022-06-29T16:05:25.099+0000 7f096752d080 -1 init, newargv = 0x564600cfbf10 newargc=9 2022-06-29T16:05:25.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[12487]: starting ceph client 2022-06-29T16:05:25.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[12487]: starting fuse 2022-06-29T16:05:26.205 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2022-06-29T16:05:26.223 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:05:26.224 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:05:26.234 INFO:teuthology.orchestra.run.smithi156.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:05:26.235 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:05:26.236 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:26.237 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2022-06-29T16:05:26.242 INFO:teuthology.orchestra.run.smithi156.stdout:52 2022-06-29T16:05:26.243 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-06-29T16:05:26.243 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:26.244 DEBUG:teuthology.orchestra.run.smithi156:> sudo python3 -c ' 2022-06-29T16:05:26.244 DEBUG:teuthology.orchestra.run.smithi156:> import glob 2022-06-29T16:05:26.245 DEBUG:teuthology.orchestra.run.smithi156:> import re 2022-06-29T16:05:26.245 DEBUG:teuthology.orchestra.run.smithi156:> import os 2022-06-29T16:05:26.246 DEBUG:teuthology.orchestra.run.smithi156:> import subprocess 2022-06-29T16:05:26.246 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:05:26.247 DEBUG:teuthology.orchestra.run.smithi156:> def find_socket(client_name): 2022-06-29T16:05:26.247 DEBUG:teuthology.orchestra.run.smithi156:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-06-29T16:05:26.247 DEBUG:teuthology.orchestra.run.smithi156:> files = glob.glob(asok_path) 2022-06-29T16:05:26.248 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:05:26.248 DEBUG:teuthology.orchestra.run.smithi156:> # Given a non-glob path, it better be there 2022-06-29T16:05:26.249 DEBUG:teuthology.orchestra.run.smithi156:> if "*" not in asok_path: 2022-06-29T16:05:26.249 DEBUG:teuthology.orchestra.run.smithi156:> assert(len(files) == 1) 2022-06-29T16:05:26.250 DEBUG:teuthology.orchestra.run.smithi156:> return files[0] 2022-06-29T16:05:26.250 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:05:26.250 DEBUG:teuthology.orchestra.run.smithi156:> for f in files: 2022-06-29T16:05:26.251 DEBUG:teuthology.orchestra.run.smithi156:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-06-29T16:05:26.251 DEBUG:teuthology.orchestra.run.smithi156:> if os.path.exists("/proc/{0}".format(pid)): 2022-06-29T16:05:26.252 DEBUG:teuthology.orchestra.run.smithi156:> return f 2022-06-29T16:05:26.252 DEBUG:teuthology.orchestra.run.smithi156:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-06-29T16:05:26.253 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:05:26.253 DEBUG:teuthology.orchestra.run.smithi156:> print(find_socket("client.1")) 2022-06-29T16:05:26.254 DEBUG:teuthology.orchestra.run.smithi156:> ' 2022-06-29T16:05:26.329 INFO:teuthology.orchestra.run.smithi156.stdout:/var/run/ceph/ceph-client.1.12487.asok 2022-06-29T16:05:26.334 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.12487.asok 2022-06-29T16:05:26.335 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:26.335 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.12487.asok status 2022-06-29T16:05:26.427 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2022-06-29T16:05:26.428 INFO:teuthology.orchestra.run.smithi156.stdout: "metadata": { 2022-06-29T16:05:26.428 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_sha1": "f25ffac72e5a9e63860ff7deb6767a11f0559a16", 2022-06-29T16:05:26.428 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_version": "ceph version 15.2.16-304-gf25ffac7 (f25ffac72e5a9e63860ff7deb6767a11f0559a16) octopus (stable)", 2022-06-29T16:05:26.428 INFO:teuthology.orchestra.run.smithi156.stdout: "entity_id": "1", 2022-06-29T16:05:26.429 INFO:teuthology.orchestra.run.smithi156.stdout: "hostname": "smithi156", 2022-06-29T16:05:26.429 INFO:teuthology.orchestra.run.smithi156.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-06-29T16:05:26.429 INFO:teuthology.orchestra.run.smithi156.stdout: "pid": "12487", 2022-06-29T16:05:26.429 INFO:teuthology.orchestra.run.smithi156.stdout: "root": "/" 2022-06-29T16:05:26.430 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:05:26.430 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_count": 0, 2022-06-29T16:05:26.430 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_pinned_count": 0, 2022-06-29T16:05:26.430 INFO:teuthology.orchestra.run.smithi156.stdout: "id": 6101, 2022-06-29T16:05:26.431 INFO:teuthology.orchestra.run.smithi156.stdout: "inst": { 2022-06-29T16:05:26.431 INFO:teuthology.orchestra.run.smithi156.stdout: "name": { 2022-06-29T16:05:26.431 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "client", 2022-06-29T16:05:26.431 INFO:teuthology.orchestra.run.smithi156.stdout: "num": 6101 2022-06-29T16:05:26.432 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:05:26.432 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2022-06-29T16:05:26.432 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2022-06-29T16:05:26.432 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "172.21.15.156:0", 2022-06-29T16:05:26.433 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 3631523619 2022-06-29T16:05:26.433 INFO:teuthology.orchestra.run.smithi156.stdout: } 2022-06-29T16:05:26.433 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:05:26.433 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2022-06-29T16:05:26.434 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2022-06-29T16:05:26.434 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "172.21.15.156:0", 2022-06-29T16:05:26.434 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 3631523619 2022-06-29T16:05:26.434 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:05:26.435 INFO:teuthology.orchestra.run.smithi156.stdout: "inst_str": "client.6101 172.21.15.156:0/3631523619", 2022-06-29T16:05:26.435 INFO:teuthology.orchestra.run.smithi156.stdout: "addr_str": "172.21.15.156:0/3631523619", 2022-06-29T16:05:26.435 INFO:teuthology.orchestra.run.smithi156.stdout: "inode_count": 1, 2022-06-29T16:05:26.435 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_epoch": 36, 2022-06-29T16:05:26.436 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch": 81, 2022-06-29T16:05:26.436 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch_barrier": 0, 2022-06-29T16:05:26.436 INFO:teuthology.orchestra.run.smithi156.stdout: "blacklisted": false, 2022-06-29T16:05:26.436 INFO:teuthology.orchestra.run.smithi156.stdout: "fs_name": "cephfs" 2022-06-29T16:05:26.437 INFO:teuthology.orchestra.run.smithi156.stdout:} 2022-06-29T16:05:26.438 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:26.439 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:05:26.439 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:05:26.447 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2022-06-29T16:05:26.448 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-06-29T16:05:26.448 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:26.449 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:05:26.508 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:05:26.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.610+0000 7f2d358e5700 1 Processor -- start 2022-06-29T16:05:26.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.610+0000 7f2d358e5700 1 -- start start 2022-06-29T16:05:26.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.614+0000 7f2d358e5700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d301449c0 0x7f2d300b2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:26.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.614+0000 7f2d358e5700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2d30140a20 0x7f2d30140e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:26.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.614+0000 7f2d358e5700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d3005c770 con 0x7f2d3005c390 2022-06-29T16:05:26.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.614+0000 7f2d358e5700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d30140750 con 0x7f2d30140a20 2022-06-29T16:05:26.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.614+0000 7f2d358e5700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d30143e80 con 0x7f2d301449c0 2022-06-29T16:05:26.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.614+0000 7f2d2effd700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2d3005c390 0x7f2d301442b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58396/0 (socket says 172.21.15.60:58396) 2022-06-29T16:05:26.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.614+0000 7f2d2effd700 1 -- 172.21.15.60:0/306326734 learned_addr learned my addr 172.21.15.60:0/306326734 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:26.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.614+0000 7f2d2e7fc700 1 --2- 172.21.15.60:0/306326734 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d301449c0 0x7f2d300b2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:26.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.614+0000 7f2d2f7fe700 1 --2- 172.21.15.60:0/306326734 >> v2:172.21.15.92:3300/0 conn(0x7f2d30140a20 0x7f2d30140e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:26.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.614+0000 7f2d2dffb700 1 -- 172.21.15.60:0/306326734 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1533712231 0 0) 0x7f2d3005c770 con 0x7f2d3005c390 2022-06-29T16:05:26.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.614+0000 7f2d2dffb700 1 -- 172.21.15.60:0/306326734 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d10003260 con 0x7f2d3005c390 2022-06-29T16:05:26.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.614+0000 7f2d2e7fc700 1 -- 172.21.15.60:0/306326734 >> v2:172.21.15.92:3300/0 conn(0x7f2d30140a20 msgr2=0x7f2d30140e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:26.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.614+0000 7f2d2e7fc700 1 --2- 172.21.15.60:0/306326734 >> v2:172.21.15.92:3300/0 conn(0x7f2d30140a20 0x7f2d30140e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:26.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.614+0000 7f2d2e7fc700 1 -- 172.21.15.60:0/306326734 >> v1:172.21.15.60:6789/0 conn(0x7f2d3005c390 legacy=0x7f2d301442b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:26.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.614+0000 7f2d2e7fc700 1 -- 172.21.15.60:0/306326734 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d30065f40 con 0x7f2d301449c0 2022-06-29T16:05:26.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.614+0000 7f2d2f7fe700 1 --2- 172.21.15.60:0/306326734 >> v2:172.21.15.92:3300/0 conn(0x7f2d30140a20 0x7f2d30140e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:05:26.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.614+0000 7f2d2e7fc700 1 --2- 172.21.15.60:0/306326734 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d301449c0 0x7f2d300b2220 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f2d20002f70 tx=0x7f2d20006df0).ready entity=mon.2 client_cookie=ee02be3ab426b2af server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:26.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.614+0000 7f2d2dffb700 1 -- 172.21.15.60:0/306326734 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d2000d4a0 con 0x7f2d301449c0 2022-06-29T16:05:26.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.614+0000 7f2d2dffb700 1 -- 172.21.15.60:0/306326734 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d20000e90 con 0x7f2d301449c0 2022-06-29T16:05:26.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.614+0000 7f2d2dffb700 1 -- 172.21.15.60:0/306326734 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d2000d7f0 con 0x7f2d301449c0 2022-06-29T16:05:26.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.614+0000 7f2d358e5700 1 -- 172.21.15.60:0/306326734 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d301449c0 msgr2=0x7f2d300b2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:26.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.614+0000 7f2d358e5700 1 --2- 172.21.15.60:0/306326734 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d301449c0 0x7f2d300b2220 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f2d20002f70 tx=0x7f2d20006df0).stop 2022-06-29T16:05:26.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.614+0000 7f2d358e5700 1 -- 172.21.15.60:0/306326734 shutdown_connections 2022-06-29T16:05:26.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.614+0000 7f2d358e5700 1 --2- 172.21.15.60:0/306326734 >> v2:172.21.15.92:3300/0 conn(0x7f2d30140a20 0x7f2d30140e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:26.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.614+0000 7f2d358e5700 1 --2- 172.21.15.60:0/306326734 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d301449c0 0x7f2d300b2220 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:26.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.614+0000 7f2d358e5700 1 -- 172.21.15.60:0/306326734 >> 172.21.15.60:0/306326734 conn(0x7f2d30064a10 msgr2=0x7f2d30064df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:26.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.614+0000 7f2d358e5700 1 -- 172.21.15.60:0/306326734 shutdown_connections 2022-06-29T16:05:26.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.614+0000 7f2d358e5700 1 -- 172.21.15.60:0/306326734 wait complete. 2022-06-29T16:05:26.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.618+0000 7f2d358e5700 1 Processor -- start 2022-06-29T16:05:26.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.618+0000 7f2d358e5700 1 -- start start 2022-06-29T16:05:26.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.618+0000 7f2d358e5700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d3005c390 0x7f2d300b7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:26.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.618+0000 7f2d358e5700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d30140a20 0x7f2d300b5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:26.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.618+0000 7f2d358e5700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d301449c0 0x7f2d30147350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:26.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.618+0000 7f2d358e5700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d30153d70 con 0x7f2d30140a20 2022-06-29T16:05:26.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.618+0000 7f2d358e5700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d300ae4b0 con 0x7f2d3005c390 2022-06-29T16:05:26.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.618+0000 7f2d358e5700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d300ae250 con 0x7f2d301449c0 2022-06-29T16:05:26.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.618+0000 7f2d2e7fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d30140a20 0x7f2d300b5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:26.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.618+0000 7f2d2e7fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d30140a20 0x7f2d300b5240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50318/0 (socket says 172.21.15.60:50318) 2022-06-29T16:05:26.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.618+0000 7f2d2e7fc700 1 -- 172.21.15.60:0/3834153635 learned_addr learned my addr 172.21.15.60:0/3834153635 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:26.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.618+0000 7f2d2f7fe700 1 --2- 172.21.15.60:0/3834153635 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d301449c0 0x7f2d30147350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:26.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.618+0000 7f2d2effd700 1 --2- 172.21.15.60:0/3834153635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d3005c390 0x7f2d300b7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:26.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.618+0000 7f2d2e7fc700 1 -- 172.21.15.60:0/3834153635 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d301449c0 msgr2=0x7f2d30147350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:26.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.618+0000 7f2d2e7fc700 1 --2- 172.21.15.60:0/3834153635 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d301449c0 0x7f2d30147350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:26.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.618+0000 7f2d2e7fc700 1 -- 172.21.15.60:0/3834153635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d3005c390 msgr2=0x7f2d300b7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:26.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.618+0000 7f2d2e7fc700 1 --2- 172.21.15.60:0/3834153635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d3005c390 0x7f2d300b7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:26.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.618+0000 7f2d2e7fc700 1 -- 172.21.15.60:0/3834153635 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d28009470 con 0x7f2d30140a20 2022-06-29T16:05:26.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.618+0000 7f2d2e7fc700 1 --2- 172.21.15.60:0/3834153635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d30140a20 0x7f2d300b5240 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 rx=0x7f2d30148320 tx=0x7f2d20014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:26.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.618+0000 7f2d27fff700 1 -- 172.21.15.60:0/3834153635 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d20016070 con 0x7f2d30140a20 2022-06-29T16:05:26.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.618+0000 7f2d27fff700 1 -- 172.21.15.60:0/3834153635 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d200138d0 con 0x7f2d30140a20 2022-06-29T16:05:26.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.618+0000 7f2d358e5700 1 -- 172.21.15.60:0/3834153635 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d2000b180 con 0x7f2d30140a20 2022-06-29T16:05:26.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.618+0000 7f2d27fff700 1 -- 172.21.15.60:0/3834153635 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d20013c30 con 0x7f2d30140a20 2022-06-29T16:05:26.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.618+0000 7f2d358e5700 1 -- 172.21.15.60:0/3834153635 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d3013e390 con 0x7f2d30140a20 2022-06-29T16:05:26.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.622+0000 7f2d27fff700 1 -- 172.21.15.60:0/3834153635 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2d20025030 con 0x7f2d30140a20 2022-06-29T16:05:26.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.622+0000 7f2d358e5700 1 -- 172.21.15.60:0/3834153635 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d3013e550 con 0x7f2d30140a20 2022-06-29T16:05:26.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.622+0000 7f2d27fff700 1 --2- 172.21.15.60:0/3834153635 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2d14053b00 0x7f2d14055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:26.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.622+0000 7f2d27fff700 1 -- 172.21.15.60:0/3834153635 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f2d2006cfa0 con 0x7f2d30140a20 2022-06-29T16:05:26.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.622+0000 7f2d2effd700 1 --2- 172.21.15.60:0/3834153635 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2d14053b00 0x7f2d14055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:26.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.622+0000 7f2d2effd700 1 --2- 172.21.15.60:0/3834153635 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2d14053b00 0x7f2d14055fc0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f2d1800d000 tx=0x7f2d18009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:26.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.626+0000 7f2d27fff700 1 -- 172.21.15.60:0/3834153635 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2d20043d60 con 0x7f2d30140a20 2022-06-29T16:05:26.800 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.790+0000 7f2d358e5700 1 -- 172.21.15.60:0/3834153635 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d300573d0 con 0x7f2d30140a20 2022-06-29T16:05:26.801 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.794+0000 7f2d27fff700 1 -- 172.21.15.60:0/3834153635 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+2141 (secure 0 0 0) 0x7f2d20023090 con 0x7f2d30140a20 2022-06-29T16:05:26.801 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:26.801 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":36,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6657,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.60:6833/3121204394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3121204394},{"type":"v1","addr":"172.21.15.60:6833","nonce":3121204394}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":34}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:05:16.293932+0000","modified":"2022-06-29T16:05:18.324125+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6688},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6688":{"gid":6688,"name":"b","rank":0,"incarnation":35,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/1245503283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1245503283},{"type":"v1","addr":"172.21.15.92:6835","nonce":1245503283}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-06-29T16:05:26.803 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.794+0000 7f2d358e5700 1 -- 172.21.15.60:0/3834153635 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2d14053b00 msgr2=0x7f2d14055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:26.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.794+0000 7f2d358e5700 1 --2- 172.21.15.60:0/3834153635 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2d14053b00 0x7f2d14055fc0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f2d1800d000 tx=0x7f2d18009040).stop 2022-06-29T16:05:26.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.794+0000 7f2d358e5700 1 -- 172.21.15.60:0/3834153635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d30140a20 msgr2=0x7f2d300b5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:26.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.794+0000 7f2d358e5700 1 --2- 172.21.15.60:0/3834153635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d30140a20 0x7f2d300b5240 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 rx=0x7f2d30148320 tx=0x7f2d20014000).stop 2022-06-29T16:05:26.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.794+0000 7f2d358e5700 1 -- 172.21.15.60:0/3834153635 shutdown_connections 2022-06-29T16:05:26.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.794+0000 7f2d358e5700 1 --2- 172.21.15.60:0/3834153635 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2d14053b00 0x7f2d14055fc0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:26.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.794+0000 7f2d358e5700 1 --2- 172.21.15.60:0/3834153635 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d301449c0 0x7f2d30147350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:26.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.794+0000 7f2d358e5700 1 --2- 172.21.15.60:0/3834153635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d30140a20 0x7f2d300b5240 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:26.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.794+0000 7f2d358e5700 1 --2- 172.21.15.60:0/3834153635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d3005c390 0x7f2d300b7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:26.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.794+0000 7f2d358e5700 1 -- 172.21.15.60:0/3834153635 >> 172.21.15.60:0/3834153635 conn(0x7f2d30064a10 msgr2=0x7f2d300aca20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:26.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.794+0000 7f2d358e5700 1 -- 172.21.15.60:0/3834153635 shutdown_connections 2022-06-29T16:05:26.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:26.794+0000 7f2d358e5700 1 -- 172.21.15.60:0/3834153635 wait complete. 2022-06-29T16:05:26.807 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 36 2022-06-29T16:05:26.815 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json dump inode 1 2022-06-29T16:05:26.931 INFO:teuthology.orchestra.run.smithi092.stdout:{"path":"/","ino":1,"rdev":0,"ctime":"2022-06-29T16:05:22.950605+0000","btime":"2022-06-29T16:05:17.316282+0000","mode":17407,"uid":0,"gid":0,"nlink":1,"dir_layout":{"dir_hash":2,"unused1":0,"unused2":0,"unused3":0},"layout":{"stripe_unit":4194304,"stripe_count":1,"object_size":4194304,"pool_id":16,"pool_ns":""},"old_pools":[],"size":0,"truncate_seq":0,"truncate_size":18446744073709551615,"truncate_from":0,"truncate_pending":0,"mtime":"2022-06-29T16:05:17.316282+0000","atime":"0.000000","time_warp_seq":0,"change_attr":1,"export_pin":-1,"export_ephemeral_random_pin":0,"export_ephemeral_distributed_pin":false,"client_ranges":[],"dirstat":{"version":0,"mtime":"0.000000","num_files":0,"num_subdirs":0,"change_attr":0},"rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":1,"rsnaps":0,"rctime":"2022-06-29T16:05:22.950605+0000"},"accounted_rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":1,"rsnaps":0,"rctime":"2022-06-29T16:05:17.316282+0000"},"version":3,"file_data_version":0,"xattr_version":1,"backtrace_version":3,"stray_prior_path":"","max_size_ever":0,"quota":{"max_bytes":0,"max_files":0},"last_scrub_stamp":"0.000000","last_scrub_version":0,"symlink":"","xattrs":[],"dirfragtree":{"splits":[]},"old_inodes":[],"oldest_snap":18446744073709551614,"damage_flags":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":2,"is_frozen":false,"is_freezing":false,"pins":{"request":1,"lock":3,"dirfrag":1,"caps":1,"dirtyparent":1,"dirty":1,"authpin":1},"nref":9,"versionlock":{"gather_set":[],"state":"lock","is_leased":false,"num_rdlocks":0,"num_wrlocks":1,"num_xlocks":0,"xlock_by":{}},"authlock":{"gather_set":[],"state":"xlockdone","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":1,"xlock_by":{}},"linklock":{},"dirfragtreelock":{},"filelock":{},"xattrlock":{},"snaplock":{"gather_set":[],"state":"sync","is_leased":false,"num_rdlocks":1,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"nestlock":{},"flocklock":{},"policylock":{},"states":["auth","dirty","dirtyparent"],"client_caps":[{"client_id":6101,"pending":"pLsXsFs","issued":"pLsXsFs","wanted":"-","last_sent":2},{"client_id":6843,"pending":"pLsXsFs","issued":"pLsXsFs","wanted":"-","last_sent":2}],"loner":-1,"want_loner":-1,"mds_caps_wanted":[]} 2022-06-29T16:05:26.932 DEBUG:tasks.cephfs.filesystem:_json_asok output { "accounted_rstat": { "rbytes": 0, "rctime": "2022-06-29T16:05:17.316282+0000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "atime": "0.000000", "auth_pins": 2, "auth_state": { "replicas": {} }, "authlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "state": "xlockdone", "xlock_by": {} }, "backtrace_version": 3, "btime": "2022-06-29T16:05:17.316282+0000", "change_attr": 1, "client_caps": [ { "client_id": 6101, "issued": "pLsXsFs", "last_sent": 2, "pending": "pLsXsFs", "wanted": "-" }, { "client_id": 6843, "issued": "pLsXsFs", "last_sent": 2, "pending": "pLsXsFs", "wanted": "-" } ], "client_ranges": [], "ctime": "2022-06-29T16:05:22.950605+0000", "damage_flags": 0, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "dirfragtree": { "splits": [] }, "dirfragtreelock": {}, "dirstat": { "change_attr": 0, "mtime": "0.000000", "num_files": 0, "num_subdirs": 0, "version": 0 }, "export_ephemeral_distributed_pin": false, "export_ephemeral_random_pin": 0, "export_pin": -1, "file_data_version": 0, "filelock": {}, "flocklock": {}, "gid": 0, "ino": 1, "is_auth": true, "is_freezing": false, "is_frozen": false, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "layout": { "object_size": 4194304, "pool_id": 16, "pool_ns": "", "stripe_count": 1, "stripe_unit": 4194304 }, "linklock": {}, "loner": -1, "max_size_ever": 0, "mds_caps_wanted": [], "mode": 17407, "mtime": "2022-06-29T16:05:17.316282+0000", "nestlock": {}, "nlink": 1, "nref": 9, "old_inodes": [], "old_pools": [], "oldest_snap": 18446744073709551614, "path": "/", "pins": { "authpin": 1, "caps": 1, "dirfrag": 1, "dirty": 1, "dirtyparent": 1, "lock": 3, "request": 1 }, "policylock": {}, "quota": { "max_bytes": 0, "max_files": 0 }, "rdev": 0, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "rstat": { "rbytes": 0, "rctime": "2022-06-29T16:05:22.950605+0000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "size": 0, "snaplock": { "gather_set": [], "is_leased": false, "num_rdlocks": 1, "num_wrlocks": 0, "num_xlocks": 0, "state": "sync", "xlock_by": {} }, "states": [ "auth", "dirty", "dirtyparent" ], "stray_prior_path": "", "symlink": "", "time_warp_seq": 0, "truncate_from": 0, "truncate_pending": 0, "truncate_seq": 0, "truncate_size": 18446744073709551615, "uid": 0, "version": 3, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 1, "num_xlocks": 0, "state": "lock", "xlock_by": {} }, "want_loner": -1, "xattr_version": 1, "xattrlock": {}, "xattrs": [] } 2022-06-29T16:05:26.933 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-06-29T16:05:26.934 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:05:26.955 DEBUG:teuthology.orchestra.run.smithi092:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:05:26.975 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:05:26.997 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:05:27.017 DEBUG:teuthology.orchestra.run.smithi197:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:05:27.039 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-06-29T16:05:27.039 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:27.040 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:05:27.040 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:05:27.048 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T16:05:27.049 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T16:05:27.050 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2022-06-29T16:05:27.050 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-06-29T16:05:27.051 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:05:27.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[13048]: fuse finished with error 0 and tester_r 0 2022-06-29T16:05:27.067 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:27.068 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:05:27.068 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:05:27.076 INFO:teuthology.orchestra.run.smithi121.stdout:ext2/ext3 2022-06-29T16:05:27.077 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-06-29T16:05:27.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:05:27.193 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:05:27.200 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:27.201 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:05:27.201 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:05:27.210 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2022-06-29T16:05:27.211 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-06-29T16:05:27.211 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi156.front.sepia.ceph.com... 2022-06-29T16:05:27.212 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-06-29T16:05:27.212 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:05:27.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[12487]: fuse finished with error 0 and tester_r 0 2022-06-29T16:05:27.230 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:27.230 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:05:27.231 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:05:27.236 INFO:teuthology.orchestra.run.smithi156.stdout:ext2/ext3 2022-06-29T16:05:27.238 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-06-29T16:05:27.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:05:27.299 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:05:27.306 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:27.306 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:05:27.307 DEBUG:teuthology.orchestra.run.smithi197:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T16:05:27.313 INFO:teuthology.orchestra.run.smithi197.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-06-29T16:05:27.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:05:27.315 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-06-29T16:05:27.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:05:27.316 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T16:05:27.364 INFO:teuthology.orchestra.run.smithi197.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-06-29T16:05:27.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:05:27.366 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-06-29T16:05:27.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.470+0000 7fc06359e700 1 Processor -- start 2022-06-29T16:05:27.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.470+0000 7fc06359e700 1 -- start start 2022-06-29T16:05:27.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.470+0000 7fc06359e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc0640aed00 0x7fc064146810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:27.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.470+0000 7fc06359e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc06405d450 con 0x7fc06405d070 2022-06-29T16:05:27.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.470+0000 7fc06359e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0640b25d0 con 0x7fc0640b2130 2022-06-29T16:05:27.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.470+0000 7fc06359e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0640adf90 con 0x7fc0640aed00 2022-06-29T16:05:27.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.470+0000 7fc06259c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc06405d070 0x7fc06414aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58410/0 (socket says 172.21.15.60:58410) 2022-06-29T16:05:27.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.470+0000 7fc06259c700 1 -- 172.21.15.60:0/576382654 learned_addr learned my addr 172.21.15.60:0/576382654 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:27.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.470+0000 7fc062d9d700 1 --2- 172.21.15.60:0/576382654 >> v2:172.21.15.60:3300/0 conn(0x7fc0640aed00 0x7fc064146810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:27.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.470+0000 7fc06159a700 1 -- 172.21.15.60:0/576382654 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1157044769 0 0) 0x7fc06405d450 con 0x7fc06405d070 2022-06-29T16:05:27.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.470+0000 7fc06159a700 1 -- 172.21.15.60:0/576382654 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc048003260 con 0x7fc06405d070 2022-06-29T16:05:27.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.470+0000 7fc06159a700 1 -- 172.21.15.60:0/576382654 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4190794932 0 0) 0x7fc0640b25d0 con 0x7fc0640b2130 2022-06-29T16:05:27.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.470+0000 7fc06159a700 1 -- 172.21.15.60:0/576382654 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc06405d450 con 0x7fc0640b2130 2022-06-29T16:05:27.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.470+0000 7fc062d9d700 1 -- 172.21.15.60:0/576382654 >> v1:172.21.15.92:6789/0 conn(0x7fc0640b2130 legacy=0x7fc064146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:27.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.470+0000 7fc062d9d700 1 -- 172.21.15.60:0/576382654 >> v1:172.21.15.60:6789/0 conn(0x7fc06405d070 legacy=0x7fc06414aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:27.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.470+0000 7fc06159a700 1 -- 172.21.15.60:0/576382654 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 422063635 0 0) 0x7fc048003260 con 0x7fc06405d070 2022-06-29T16:05:27.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.470+0000 7fc062d9d700 1 -- 172.21.15.60:0/576382654 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc06414a350 con 0x7fc0640aed00 2022-06-29T16:05:27.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.470+0000 7fc062d9d700 1 --2- 172.21.15.60:0/576382654 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0640aed00 0x7fc064146810 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 rx=0x7fc054001490 tx=0x7fc05c006df0).ready entity=mon.0 client_cookie=8e66e922d30eaabb server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:27.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.470+0000 7fc06159a700 1 -- 172.21.15.60:0/576382654 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc05c00d4f0 con 0x7fc0640aed00 2022-06-29T16:05:27.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.470+0000 7fc06159a700 1 -- 172.21.15.60:0/576382654 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc05c000e90 con 0x7fc0640aed00 2022-06-29T16:05:27.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.470+0000 7fc06159a700 1 -- 172.21.15.60:0/576382654 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc05c00d840 con 0x7fc0640aed00 2022-06-29T16:05:27.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.474+0000 7fc06359e700 1 -- 172.21.15.60:0/576382654 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0640aed00 msgr2=0x7fc064146810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:27.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.474+0000 7fc06359e700 1 --2- 172.21.15.60:0/576382654 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0640aed00 0x7fc064146810 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 rx=0x7fc054001490 tx=0x7fc05c006df0).stop 2022-06-29T16:05:27.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.474+0000 7fc06359e700 1 -- 172.21.15.60:0/576382654 shutdown_connections 2022-06-29T16:05:27.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.474+0000 7fc06359e700 1 --2- 172.21.15.60:0/576382654 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0640aed00 0x7fc064146810 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:27.486 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.474+0000 7fc06359e700 1 -- 172.21.15.60:0/576382654 >> 172.21.15.60:0/576382654 conn(0x7fc0640b5890 msgr2=0x7fc0640b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:27.486 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.474+0000 7fc06359e700 1 -- 172.21.15.60:0/576382654 shutdown_connections 2022-06-29T16:05:27.486 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.474+0000 7fc06359e700 1 -- 172.21.15.60:0/576382654 wait complete. 2022-06-29T16:05:27.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.474+0000 7fc06359e700 1 Processor -- start 2022-06-29T16:05:27.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.474+0000 7fc06359e700 1 -- start start 2022-06-29T16:05:27.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.474+0000 7fc06359e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc06405d070 0x7fc064065950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:27.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.474+0000 7fc06359e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc0640aed00 0x7fc064066330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:27.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.474+0000 7fc06359e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0640b2130 0x7fc0641596e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:27.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.474+0000 7fc06359e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc064116ed0 con 0x7fc0640b2130 2022-06-29T16:05:27.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.474+0000 7fc06359e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc064118f40 con 0x7fc06405d070 2022-06-29T16:05:27.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.474+0000 7fc06359e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0641191e0 con 0x7fc0640aed00 2022-06-29T16:05:27.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.474+0000 7fc062d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0640b2130 0x7fc0641596e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:27.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.474+0000 7fc062d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0640b2130 0x7fc0641596e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50334/0 (socket says 172.21.15.60:50334) 2022-06-29T16:05:27.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.474+0000 7fc062d9d700 1 -- 172.21.15.60:0/344519018 learned_addr learned my addr 172.21.15.60:0/344519018 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:27.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.474+0000 7fc061d9b700 1 --2- 172.21.15.60:0/344519018 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc0640aed00 0x7fc064066330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:27.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.474+0000 7fc06259c700 1 --2- 172.21.15.60:0/344519018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc06405d070 0x7fc064065950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:27.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.474+0000 7fc062d9d700 1 -- 172.21.15.60:0/344519018 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc0640aed00 msgr2=0x7fc064066330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:27.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.474+0000 7fc062d9d700 1 --2- 172.21.15.60:0/344519018 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc0640aed00 0x7fc064066330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:27.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.474+0000 7fc062d9d700 1 -- 172.21.15.60:0/344519018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc06405d070 msgr2=0x7fc064065950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:27.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.474+0000 7fc062d9d700 1 --2- 172.21.15.60:0/344519018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc06405d070 0x7fc064065950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:27.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.474+0000 7fc062d9d700 1 -- 172.21.15.60:0/344519018 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc05c00b180 con 0x7fc0640b2130 2022-06-29T16:05:27.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.474+0000 7fc06259c700 1 --2- 172.21.15.60:0/344519018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc06405d070 0x7fc064065950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:27.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.474+0000 7fc062d9d700 1 --2- 172.21.15.60:0/344519018 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0640b2130 0x7fc0641596e0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 rx=0x7fc054001490 tx=0x7fc05c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:27.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.474+0000 7fc05b7fe700 1 -- 172.21.15.60:0/344519018 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc05c013a40 con 0x7fc0640b2130 2022-06-29T16:05:27.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.474+0000 7fc05b7fe700 1 -- 172.21.15.60:0/344519018 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc05c013ba0 con 0x7fc0640b2130 2022-06-29T16:05:27.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.474+0000 7fc05b7fe700 1 -- 172.21.15.60:0/344519018 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc05c01e400 con 0x7fc0640b2130 2022-06-29T16:05:27.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.474+0000 7fc06359e700 1 -- 172.21.15.60:0/344519018 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc064116bc0 con 0x7fc0640b2130 2022-06-29T16:05:27.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.474+0000 7fc06359e700 1 -- 172.21.15.60:0/344519018 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc064065350 con 0x7fc0640b2130 2022-06-29T16:05:27.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.478+0000 7fc06359e700 1 -- 172.21.15.60:0/344519018 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc064066b60 con 0x7fc0640b2130 2022-06-29T16:05:27.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.478+0000 7fc05b7fe700 1 -- 172.21.15.60:0/344519018 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc05c025030 con 0x7fc0640b2130 2022-06-29T16:05:27.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.478+0000 7fc05b7fe700 1 --2- 172.21.15.60:0/344519018 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc03c053950 0x7fc03c055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:27.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.478+0000 7fc05b7fe700 1 -- 172.21.15.60:0/344519018 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fc05c06cd40 con 0x7fc0640b2130 2022-06-29T16:05:27.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.478+0000 7fc06259c700 1 --2- 172.21.15.60:0/344519018 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc03c053950 0x7fc03c055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:27.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.478+0000 7fc06259c700 1 --2- 172.21.15.60:0/344519018 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc03c053950 0x7fc03c055e10 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fc04c00b000 tx=0x7fc04c001680).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:27.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.486+0000 7fc05b7fe700 1 -- 172.21.15.60:0/344519018 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc05c043bb0 con 0x7fc0640b2130 2022-06-29T16:05:27.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.610+0000 7fc06359e700 1 -- 172.21.15.60:0/344519018 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fc064065750 con 0x7fc0640b2130 2022-06-29T16:05:27.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.614+0000 7fc05b7fe700 1 -- 172.21.15.60:0/344519018 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v81) v1 ==== 81+0+33802 (secure 0 0 0) 0x7fc05c023070 con 0x7fc0640b2130 2022-06-29T16:05:27.626 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:27.627 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:05:27.627 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 81, 2022-06-29T16:05:27.627 INFO:teuthology.orchestra.run.smithi060.stdout: "fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", 2022-06-29T16:05:27.627 INFO:teuthology.orchestra.run.smithi060.stdout: "created": "2022-06-29T15:58:07.965719+0000", 2022-06-29T16:05:27.628 INFO:teuthology.orchestra.run.smithi060.stdout: "modified": "2022-06-29T16:05:16.667913+0000", 2022-06-29T16:05:27.628 INFO:teuthology.orchestra.run.smithi060.stdout: "last_up_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:05:27.628 INFO:teuthology.orchestra.run.smithi060.stdout: "last_in_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:05:27.628 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-06-29T16:05:27.628 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_num": 5799936, 2022-06-29T16:05:27.629 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_set": [ 2022-06-29T16:05:27.629 INFO:teuthology.orchestra.run.smithi060.stdout: "pglog_hardlimit", 2022-06-29T16:05:27.629 INFO:teuthology.orchestra.run.smithi060.stdout: "purged_snapdirs", 2022-06-29T16:05:27.629 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_deletes", 2022-06-29T16:05:27.630 INFO:teuthology.orchestra.run.smithi060.stdout: "sortbitwise" 2022-06-29T16:05:27.630 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:05:27.630 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_version": 4, 2022-06-29T16:05:27.631 INFO:teuthology.orchestra.run.smithi060.stdout: "full_ratio": 0.94999998807907104, 2022-06-29T16:05:27.631 INFO:teuthology.orchestra.run.smithi060.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-06-29T16:05:27.631 INFO:teuthology.orchestra.run.smithi060.stdout: "nearfull_ratio": 0.85000002384185791, 2022-06-29T16:05:27.631 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_snapshot": "", 2022-06-29T16:05:27.631 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_max": 16, 2022-06-29T16:05:27.632 INFO:teuthology.orchestra.run.smithi060.stdout: "max_osd": 8, 2022-06-29T16:05:27.632 INFO:teuthology.orchestra.run.smithi060.stdout: "require_min_compat_client": "jewel", 2022-06-29T16:05:27.632 INFO:teuthology.orchestra.run.smithi060.stdout: "min_compat_client": "jewel", 2022-06-29T16:05:27.632 INFO:teuthology.orchestra.run.smithi060.stdout: "require_osd_release": "octopus", 2022-06-29T16:05:27.633 INFO:teuthology.orchestra.run.smithi060.stdout: "pools": [ 2022-06-29T16:05:27.633 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.634 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2022-06-29T16:05:27.634 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "device_health_metrics", 2022-06-29T16:05:27.634 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:10.961127+0000", 2022-06-29T16:05:27.635 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:05:27.635 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:05:27.635 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:05:27.636 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:05:27.636 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:05:27.636 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:05:27.636 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:05:27.637 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:05:27.637 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 1, 2022-06-29T16:05:27.637 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 1, 2022-06-29T16:05:27.637 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 1, 2022-06-29T16:05:27.637 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 1, 2022-06-29T16:05:27.638 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 1, 2022-06-29T16:05:27.638 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:05:27.638 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:05:27.639 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:05:27.639 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:05:27.639 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:05:27.639 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:05:27.639 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:05:27.640 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.640 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "15", 2022-06-29T16:05:27.640 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:05:27.640 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:05:27.641 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:05:27.641 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:05:27.641 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:05:27.641 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:05:27.642 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:05:27.642 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:05:27.642 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:05:27.642 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:05:27.642 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:05:27.643 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:05:27.643 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:05:27.643 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:05:27.643 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:05:27.644 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:05:27.644 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:05:27.644 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:05:27.644 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:05:27.645 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:05:27.645 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:05:27.645 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:05:27.645 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:05:27.645 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:05:27.646 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:05:27.646 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:05:27.646 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.646 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:05:27.647 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:05:27.647 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:05:27.647 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:05:27.647 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:05:27.648 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:05:27.648 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:05:27.648 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:05:27.648 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:05:27.648 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:05:27.649 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:05:27.649 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:05:27.649 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 1 2022-06-29T16:05:27.649 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.650 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:05:27.650 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_devicehealth": {} 2022-06-29T16:05:27.650 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.650 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.651 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.651 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 2, 2022-06-29T16:05:27.651 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "rbd", 2022-06-29T16:05:27.651 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:19.184276+0000", 2022-06-29T16:05:27.651 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:05:27.652 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:05:27.652 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:05:27.652 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:05:27.652 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:05:27.653 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:05:27.653 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:05:27.653 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:05:27.653 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:05:27.653 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:05:27.654 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:05:27.654 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:05:27.654 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:05:27.655 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:05:27.655 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:05:27.655 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:05:27.656 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:05:27.656 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:05:27.657 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:05:27.657 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:05:27.657 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.658 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "18", 2022-06-29T16:05:27.658 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:05:27.659 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:05:27.659 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:05:27.659 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:05:27.660 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:05:27.660 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:05:27.661 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:05:27.661 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:05:27.661 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:05:27.662 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:05:27.662 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:05:27.662 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:05:27.663 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:05:27.663 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:05:27.663 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:05:27.664 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:05:27.664 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:05:27.664 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:05:27.665 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:05:27.665 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:05:27.665 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:05:27.666 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:05:27.666 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:05:27.666 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:05:27.667 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:05:27.667 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:05:27.667 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.668 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:05:27.668 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:05:27.668 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:05:27.669 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:05:27.669 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:05:27.670 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:05:27.670 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:05:27.670 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:05:27.670 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:05:27.671 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:05:27.671 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:05:27.671 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:05:27.671 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:05:27.672 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd": {} 2022-06-29T16:05:27.672 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.672 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.672 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.672 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 15, 2022-06-29T16:05:27.673 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "cephfs_metadata", 2022-06-29T16:05:27.673 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T16:05:14.715486+0000", 2022-06-29T16:05:27.673 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:05:27.673 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:05:27.674 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:05:27.674 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:05:27.674 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:05:27.674 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:05:27.675 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:05:27.675 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:05:27.675 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:05:27.675 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:05:27.675 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:05:27.676 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:05:27.676 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:05:27.676 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:05:27.676 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:05:27.677 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:05:27.677 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:05:27.677 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:05:27.677 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:05:27.677 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:05:27.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.678 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "79", 2022-06-29T16:05:27.678 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:05:27.678 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:05:27.679 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:05:27.679 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:05:27.679 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:05:27.679 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:05:27.679 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:05:27.680 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:05:27.680 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:05:27.680 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:05:27.680 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:05:27.681 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:05:27.681 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:05:27.681 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:05:27.681 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:05:27.681 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:05:27.682 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:05:27.682 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:05:27.682 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:05:27.682 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:05:27.683 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:05:27.683 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:05:27.683 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:05:27.683 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:05:27.683 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:05:27.684 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:05:27.684 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.684 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:05:27.684 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:05:27.685 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:05:27.685 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:05:27.685 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:05:27.685 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:05:27.685 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:05:27.686 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:05:27.686 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:05:27.686 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:05:27.686 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:05:27.687 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:05:27.687 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_bias": 4, 2022-06-29T16:05:27.687 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 16, 2022-06-29T16:05:27.687 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_priority": 5 2022-06-29T16:05:27.687 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.688 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:05:27.688 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs": { 2022-06-29T16:05:27.688 INFO:teuthology.orchestra.run.smithi060.stdout: "metadata": "cephfs" 2022-06-29T16:05:27.688 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.689 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.689 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.689 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 16, 2022-06-29T16:05:27.689 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "cephfs_data", 2022-06-29T16:05:27.690 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T16:05:15.306355+0000", 2022-06-29T16:05:27.690 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:05:27.690 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:05:27.690 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:05:27.691 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:05:27.691 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:05:27.691 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:05:27.691 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:05:27.692 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:05:27.692 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:05:27.692 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:05:27.692 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:05:27.692 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:05:27.693 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:05:27.693 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:05:27.693 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:05:27.693 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:05:27.694 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:05:27.694 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:05:27.694 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:05:27.694 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:05:27.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.695 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "80", 2022-06-29T16:05:27.695 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:05:27.695 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:05:27.696 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:05:27.696 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:05:27.696 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:05:27.696 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:05:27.696 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:05:27.697 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:05:27.697 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:05:27.697 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:05:27.697 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:05:27.698 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:05:27.698 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:05:27.698 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:05:27.698 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:05:27.699 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:05:27.699 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:05:27.699 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:05:27.699 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:05:27.699 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:05:27.700 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:05:27.700 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:05:27.700 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:05:27.700 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:05:27.701 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:05:27.701 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:05:27.701 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.701 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:05:27.701 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:05:27.702 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:05:27.702 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:05:27.702 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:05:27.702 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:05:27.703 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:05:27.703 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:05:27.703 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:05:27.703 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:05:27.704 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:05:27.704 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:05:27.704 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:05:27.704 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs": { 2022-06-29T16:05:27.704 INFO:teuthology.orchestra.run.smithi060.stdout: "data": "cephfs" 2022-06-29T16:05:27.705 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.705 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.705 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.705 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:05:27.706 INFO:teuthology.orchestra.run.smithi060.stdout: "osds": [ 2022-06-29T16:05:27.706 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.706 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:05:27.706 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "11e7ab9e-fce2-470f-8195-ea4c3c52df22", 2022-06-29T16:05:27.707 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:27.707 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:27.707 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:27.707 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:27.708 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:27.708 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:27.708 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:27.708 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 78, 2022-06-29T16:05:27.708 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:27.709 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:27.709 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:27.709 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:27.709 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.709 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:27.710 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6816", 2022-06-29T16:05:27.710 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:27.710 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.710 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.711 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:27.711 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6825", 2022-06-29T16:05:27.711 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:27.711 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.712 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.712 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.712 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:27.712 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:27.712 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.713 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:27.713 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6826", 2022-06-29T16:05:27.713 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:27.713 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.713 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.714 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:27.714 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6827", 2022-06-29T16:05:27.714 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:27.714 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.715 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.715 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.715 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:27.715 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:27.715 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.716 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:27.716 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6830", 2022-06-29T16:05:27.716 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:27.716 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.717 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.717 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:27.717 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6831", 2022-06-29T16:05:27.717 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:27.717 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.718 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.718 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.718 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:27.718 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:27.719 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.719 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:27.719 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6828", 2022-06-29T16:05:27.719 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:27.719 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.720 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.720 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:27.720 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6829", 2022-06-29T16:05:27.720 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:27.721 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.721 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.721 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.721 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6825/12890", 2022-06-29T16:05:27.722 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6827/12890", 2022-06-29T16:05:27.722 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6831/12890", 2022-06-29T16:05:27.722 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6829/12890", 2022-06-29T16:05:27.722 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:27.723 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:27.723 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:27.723 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.723 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.723 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.724 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:05:27.724 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "ca43f136-6f69-4431-a157-fbe073fb585c", 2022-06-29T16:05:27.724 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:27.724 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:27.725 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:27.725 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:27.725 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:27.725 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:27.725 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:27.726 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 77, 2022-06-29T16:05:27.726 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:27.726 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:27.726 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:27.727 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:27.727 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.727 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:27.727 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6815", 2022-06-29T16:05:27.728 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:27.728 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.728 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.728 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:27.728 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6818", 2022-06-29T16:05:27.729 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:27.729 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.729 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.729 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.730 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:27.730 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:27.730 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.730 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:27.730 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6819", 2022-06-29T16:05:27.731 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:27.731 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.731 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.731 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:27.732 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6820", 2022-06-29T16:05:27.732 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:27.732 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.732 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.733 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.733 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:27.733 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:27.733 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.733 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:27.734 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6823", 2022-06-29T16:05:27.734 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:27.734 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.734 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.735 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:27.735 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6824", 2022-06-29T16:05:27.735 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:27.735 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.735 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.736 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.736 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:27.736 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:27.736 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.737 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:27.737 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6821", 2022-06-29T16:05:27.737 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:27.737 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.737 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.738 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:27.738 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6822", 2022-06-29T16:05:27.738 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:27.738 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.739 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.739 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.739 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6818/12892", 2022-06-29T16:05:27.739 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6820/12892", 2022-06-29T16:05:27.739 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6824/12892", 2022-06-29T16:05:27.740 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6822/12892", 2022-06-29T16:05:27.740 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:27.740 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:27.740 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:27.741 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.741 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.741 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.741 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:05:27.741 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "74fac45f-5f07-44ec-a9cb-2644c123fd4a", 2022-06-29T16:05:27.742 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:27.742 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:27.742 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:27.742 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:27.743 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:27.743 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:27.743 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:27.743 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 78, 2022-06-29T16:05:27.743 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:27.744 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:27.744 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:27.744 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:27.744 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.744 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:27.745 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6808", 2022-06-29T16:05:27.745 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:27.745 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.745 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.746 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:27.746 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6809", 2022-06-29T16:05:27.746 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:27.746 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.746 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.747 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.747 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:27.747 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:27.747 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.748 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:27.748 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6810", 2022-06-29T16:05:27.748 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:27.748 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.748 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.749 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:27.749 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6811", 2022-06-29T16:05:27.749 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:27.749 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.750 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.750 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.750 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:27.750 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:27.750 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.751 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:27.751 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6814", 2022-06-29T16:05:27.751 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:27.751 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.752 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.752 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:27.752 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6817", 2022-06-29T16:05:27.752 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:27.752 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.753 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.753 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.753 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:27.753 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:27.754 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.754 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:27.754 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6812", 2022-06-29T16:05:27.754 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:27.755 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.755 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.755 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:27.755 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6813", 2022-06-29T16:05:27.755 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:27.756 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.756 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.756 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.756 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6809/12891", 2022-06-29T16:05:27.757 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6811/12891", 2022-06-29T16:05:27.757 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6817/12891", 2022-06-29T16:05:27.757 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6813/12891", 2022-06-29T16:05:27.757 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:27.757 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:27.758 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:27.758 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.758 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.758 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.759 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:05:27.759 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "2581d351-973a-4b4e-85ca-3658f0068765", 2022-06-29T16:05:27.759 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:27.759 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:27.759 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:27.760 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:27.760 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:27.760 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:27.760 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:27.761 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 78, 2022-06-29T16:05:27.761 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:27.761 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:27.761 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:27.761 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:27.762 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.762 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:27.762 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6800", 2022-06-29T16:05:27.762 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:27.763 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.763 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.763 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:27.763 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6801", 2022-06-29T16:05:27.764 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:27.764 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.764 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.764 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.765 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:27.765 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:27.765 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.765 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:27.766 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6802", 2022-06-29T16:05:27.766 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:27.766 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.766 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.766 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:27.767 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6803", 2022-06-29T16:05:27.767 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:27.767 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.767 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.768 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.768 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:27.768 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:27.768 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.768 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:27.769 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6806", 2022-06-29T16:05:27.769 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:27.769 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.769 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.770 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:27.770 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6807", 2022-06-29T16:05:27.770 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:27.770 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.770 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.771 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.771 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:27.771 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:27.771 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.772 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:27.772 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6804", 2022-06-29T16:05:27.772 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:27.772 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.773 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.773 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:27.773 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6805", 2022-06-29T16:05:27.773 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:27.773 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.774 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.774 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.774 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6801/12889", 2022-06-29T16:05:27.774 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6803/12889", 2022-06-29T16:05:27.775 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6807/12889", 2022-06-29T16:05:27.775 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6805/12889", 2022-06-29T16:05:27.775 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:27.775 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:27.775 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:27.776 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.776 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.776 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.776 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:05:27.776 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "3d234db9-87cd-4fe9-814f-0240a287d8b4", 2022-06-29T16:05:27.777 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:27.777 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:27.777 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:27.777 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:27.778 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:27.778 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:27.778 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:27.778 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 78, 2022-06-29T16:05:27.778 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:27.779 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:27.779 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:27.779 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:27.779 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.780 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:27.780 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6808", 2022-06-29T16:05:27.780 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:27.780 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.780 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.781 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:27.781 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6810", 2022-06-29T16:05:27.781 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:27.781 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.782 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.782 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.782 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:27.782 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:27.782 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.783 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:27.783 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6812", 2022-06-29T16:05:27.783 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:27.783 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.784 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.784 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:27.784 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6814", 2022-06-29T16:05:27.784 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:27.785 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.785 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.785 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.785 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:27.785 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:27.786 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.786 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:27.786 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6823", 2022-06-29T16:05:27.786 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:27.787 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.787 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.787 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:27.787 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6825", 2022-06-29T16:05:27.788 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:27.788 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.788 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.788 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.788 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:27.789 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:27.789 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.789 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:27.789 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6817", 2022-06-29T16:05:27.790 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:27.790 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.790 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.790 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:27.791 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6820", 2022-06-29T16:05:27.791 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:27.791 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.791 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.792 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.792 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6810/13249", 2022-06-29T16:05:27.792 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6814/13249", 2022-06-29T16:05:27.792 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6825/13249", 2022-06-29T16:05:27.792 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6820/13249", 2022-06-29T16:05:27.793 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:27.793 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:27.793 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:27.793 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.794 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.794 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.794 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:05:27.794 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "e4bf1104-ff5c-438b-b608-8cee31c163f6", 2022-06-29T16:05:27.794 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:27.795 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:27.795 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:27.795 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:27.795 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:27.796 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:27.796 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:27.796 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 78, 2022-06-29T16:05:27.796 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:27.797 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:27.797 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:27.797 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:27.797 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.797 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:27.798 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6809", 2022-06-29T16:05:27.798 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:27.798 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.798 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.798 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:27.799 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6811", 2022-06-29T16:05:27.799 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:27.799 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.799 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.800 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.800 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:27.800 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:27.800 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.800 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:27.801 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6813", 2022-06-29T16:05:27.801 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:27.801 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.801 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.802 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:27.802 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6815", 2022-06-29T16:05:27.802 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:27.802 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.802 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.803 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.803 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:27.803 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:27.803 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.803 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:27.804 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6824", 2022-06-29T16:05:27.804 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:27.804 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.804 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.805 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:27.805 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6827", 2022-06-29T16:05:27.805 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:27.805 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.805 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.806 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.806 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:27.806 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:27.806 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.807 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:27.807 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6818", 2022-06-29T16:05:27.807 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:27.807 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.808 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.808 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:27.808 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6821", 2022-06-29T16:05:27.808 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:27.809 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.809 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.809 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.809 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6811/13248", 2022-06-29T16:05:27.809 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6815/13248", 2022-06-29T16:05:27.810 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6827/13248", 2022-06-29T16:05:27.810 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6821/13248", 2022-06-29T16:05:27.810 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:27.810 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:27.811 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:27.811 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.811 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.811 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.811 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:05:27.812 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "8564dd25-27ab-4203-b54e-af6d3aa05857", 2022-06-29T16:05:27.812 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:27.812 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:27.812 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:27.813 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:27.813 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:27.813 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:27.813 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:27.813 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 78, 2022-06-29T16:05:27.814 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:27.814 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:27.814 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:27.814 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:27.814 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.815 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:27.815 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6816", 2022-06-29T16:05:27.815 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:27.815 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.816 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.816 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:27.816 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6819", 2022-06-29T16:05:27.816 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:27.816 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.817 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.817 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.817 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:27.817 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:27.818 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.818 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:27.818 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6822", 2022-06-29T16:05:27.818 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:27.818 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.819 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.819 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:27.819 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6826", 2022-06-29T16:05:27.819 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:27.820 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.820 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.820 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.820 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:27.820 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:27.821 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.821 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:27.821 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6830", 2022-06-29T16:05:27.821 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:27.822 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.822 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.822 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:27.822 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6831", 2022-06-29T16:05:27.822 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:27.823 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.823 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.823 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.823 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:27.824 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:27.824 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.824 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:27.824 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6828", 2022-06-29T16:05:27.824 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:27.825 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.825 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.825 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:27.825 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6829", 2022-06-29T16:05:27.826 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:27.826 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.826 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.826 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.826 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6819/13250", 2022-06-29T16:05:27.827 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6826/13250", 2022-06-29T16:05:27.827 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6831/13250", 2022-06-29T16:05:27.827 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6829/13250", 2022-06-29T16:05:27.827 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:27.828 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:27.828 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:27.828 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.828 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.829 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.829 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:05:27.829 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "bd513a9c-3b19-4314-9897-d300b5c897bb", 2022-06-29T16:05:27.829 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:27.829 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:27.830 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:27.830 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:27.830 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:27.830 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:27.831 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:27.831 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 68, 2022-06-29T16:05:27.831 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:27.831 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:27.831 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:27.832 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:27.832 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.832 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:27.832 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6800", 2022-06-29T16:05:27.833 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:27.833 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.833 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.833 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:27.834 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6801", 2022-06-29T16:05:27.834 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:27.834 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.834 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.834 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.835 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:27.835 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:27.835 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.835 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:27.836 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6802", 2022-06-29T16:05:27.836 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:27.836 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.836 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.837 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:27.837 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6803", 2022-06-29T16:05:27.837 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:27.837 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.837 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.838 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.838 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:27.838 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:27.838 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.839 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:27.839 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6806", 2022-06-29T16:05:27.839 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:27.839 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.839 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.840 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:27.840 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6807", 2022-06-29T16:05:27.840 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:27.840 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.841 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.841 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.841 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:27.841 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:27.841 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.842 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:27.842 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6804", 2022-06-29T16:05:27.842 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:27.842 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.843 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.843 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:27.843 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6805", 2022-06-29T16:05:27.843 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:27.844 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.844 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.844 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.844 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6801/13247", 2022-06-29T16:05:27.844 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6803/13247", 2022-06-29T16:05:27.845 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6807/13247", 2022-06-29T16:05:27.845 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6805/13247", 2022-06-29T16:05:27.845 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:27.845 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:27.846 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:27.846 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:27.846 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.846 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:05:27.847 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_xinfo": [ 2022-06-29T16:05:27.847 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.847 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:05:27.847 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:27.847 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:27.848 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:27.848 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:27.848 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:27.848 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.891342+0000", 2022-06-29T16:05:27.849 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:27.849 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.849 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.849 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:05:27.850 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:27.850 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:27.850 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:27.850 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:27.851 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:27.851 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.900866+0000", 2022-06-29T16:05:27.851 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:27.851 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.851 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.852 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:05:27.852 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:27.852 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:27.852 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:27.853 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:27.853 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:27.853 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.912755+0000", 2022-06-29T16:05:27.853 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:27.853 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.854 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.854 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:05:27.854 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:27.854 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:27.855 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:27.855 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:27.855 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:27.855 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.864332+0000", 2022-06-29T16:05:27.856 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:27.856 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.856 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.856 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:05:27.856 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:27.857 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:27.857 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:27.857 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:27.857 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:27.857 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.863756+0000", 2022-06-29T16:05:27.858 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:27.858 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.858 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.858 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:05:27.859 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:27.859 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:27.859 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:27.859 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:27.860 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:27.860 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.869041+0000", 2022-06-29T16:05:27.860 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:27.860 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.860 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.861 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:05:27.861 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:27.861 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:27.861 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:27.862 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:27.862 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:27.862 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.895043+0000", 2022-06-29T16:05:27.862 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:27.863 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.863 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:27.863 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:05:27.863 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:27.864 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:27.864 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:27.864 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:27.864 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:27.864 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.889491+0000", 2022-06-29T16:05:27.865 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:27.865 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.865 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:05:27.865 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap": [], 2022-06-29T16:05:27.866 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap_items": [], 2022-06-29T16:05:27.866 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_temp": [], 2022-06-29T16:05:27.866 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_temp": [], 2022-06-29T16:05:27.866 INFO:teuthology.orchestra.run.smithi060.stdout: "blacklist": { 2022-06-29T16:05:27.867 INFO:teuthology.orchestra.run.smithi060.stdout: "172.21.15.92:6835/2464413229": "2022-06-30T16:05:16.667864+0000", 2022-06-29T16:05:27.867 INFO:teuthology.orchestra.run.smithi060.stdout: "172.21.15.92:6834/2464413229": "2022-06-30T16:05:16.667864+0000" 2022-06-29T16:05:27.867 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.867 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profiles": { 2022-06-29T16:05:27.868 INFO:teuthology.orchestra.run.smithi060.stdout: "default": { 2022-06-29T16:05:27.868 INFO:teuthology.orchestra.run.smithi060.stdout: "crush-failure-domain": "osd", 2022-06-29T16:05:27.868 INFO:teuthology.orchestra.run.smithi060.stdout: "k": "2", 2022-06-29T16:05:27.868 INFO:teuthology.orchestra.run.smithi060.stdout: "m": "1", 2022-06-29T16:05:27.868 INFO:teuthology.orchestra.run.smithi060.stdout: "plugin": "jerasure", 2022-06-29T16:05:27.869 INFO:teuthology.orchestra.run.smithi060.stdout: "ruleset-failure-domain": "osd", 2022-06-29T16:05:27.869 INFO:teuthology.orchestra.run.smithi060.stdout: "technique": "reed_sol_van" 2022-06-29T16:05:27.869 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:27.869 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:27.870 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps_queue": [], 2022-06-29T16:05:27.870 INFO:teuthology.orchestra.run.smithi060.stdout: "new_removed_snaps": [], 2022-06-29T16:05:27.870 INFO:teuthology.orchestra.run.smithi060.stdout: "new_purged_snaps": [], 2022-06-29T16:05:27.870 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_node_flags": {}, 2022-06-29T16:05:27.870 INFO:teuthology.orchestra.run.smithi060.stdout: "device_class_flags": {} 2022-06-29T16:05:27.871 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:05:27.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.614+0000 7fc06359e700 1 -- 172.21.15.60:0/344519018 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc03c053950 msgr2=0x7fc03c055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:27.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.614+0000 7fc06359e700 1 --2- 172.21.15.60:0/344519018 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc03c053950 0x7fc03c055e10 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fc04c00b000 tx=0x7fc04c001680).stop 2022-06-29T16:05:27.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.614+0000 7fc06359e700 1 -- 172.21.15.60:0/344519018 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0640b2130 msgr2=0x7fc0641596e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:27.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.614+0000 7fc06359e700 1 --2- 172.21.15.60:0/344519018 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0640b2130 0x7fc0641596e0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 rx=0x7fc054001490 tx=0x7fc05c014000).stop 2022-06-29T16:05:27.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.614+0000 7fc06359e700 1 -- 172.21.15.60:0/344519018 shutdown_connections 2022-06-29T16:05:27.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.614+0000 7fc06359e700 1 --2- 172.21.15.60:0/344519018 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc03c053950 0x7fc03c055e10 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:27.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.614+0000 7fc06359e700 1 --2- 172.21.15.60:0/344519018 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0640b2130 0x7fc0641596e0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:27.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.614+0000 7fc06359e700 1 --2- 172.21.15.60:0/344519018 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc0640aed00 0x7fc064066330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:27.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.614+0000 7fc06359e700 1 --2- 172.21.15.60:0/344519018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc06405d070 0x7fc064065950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:27.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.614+0000 7fc06359e700 1 -- 172.21.15.60:0/344519018 >> 172.21.15.60:0/344519018 conn(0x7fc0640b5890 msgr2=0x7fc0640b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:27.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.614+0000 7fc06359e700 1 -- 172.21.15.60:0/344519018 shutdown_connections 2022-06-29T16:05:27.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.614+0000 7fc06359e700 1 -- 172.21.15.60:0/344519018 wait complete. 2022-06-29T16:05:27.876 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:05:27.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.982+0000 7f13f1e47700 1 Processor -- start 2022-06-29T16:05:27.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.982+0000 7f13f1e47700 1 -- start start 2022-06-29T16:05:27.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.982+0000 7f13f1e47700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13ec13f3d0 0x7f13ec060260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:27.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.982+0000 7f13f1e47700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f13ec143820 0x7f13ec0b6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:27.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.982+0000 7f13f1e47700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13ec1430e0 con 0x7f13ec0b6c00 2022-06-29T16:05:27.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.982+0000 7f13f1e47700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13ec142ef0 con 0x7f13ec143820 2022-06-29T16:05:27.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.982+0000 7f13f1e47700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec0acb90 con 0x7f13ec13f3d0 2022-06-29T16:05:27.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.982+0000 7f13eb7fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13ec13f3d0 0x7f13ec060260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:27.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.982+0000 7f13eaffd700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f13ec143820 0x7f13ec0b6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:27.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.982+0000 7f13ebfff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f13ec0b6c00 0x7f13ec13f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58428/0 (socket says 172.21.15.60:58428) 2022-06-29T16:05:27.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.982+0000 7f13ebfff700 1 -- 172.21.15.60:0/362665709 learned_addr learned my addr 172.21.15.60:0/362665709 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:27.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.986+0000 7f13eb7fe700 1 -- 172.21.15.60:0/362665709 >> v2:172.21.15.92:3300/0 conn(0x7f13ec143820 msgr2=0x7f13ec0b6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:27.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.986+0000 7f13eb7fe700 1 --2- 172.21.15.60:0/362665709 >> v2:172.21.15.92:3300/0 conn(0x7f13ec143820 0x7f13ec0b6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:27.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.986+0000 7f13eb7fe700 1 -- 172.21.15.60:0/362665709 >> v1:172.21.15.60:6789/0 conn(0x7f13ec0b6c00 legacy=0x7f13ec13f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:27.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.986+0000 7f13eb7fe700 1 -- 172.21.15.60:0/362665709 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ec143380 con 0x7f13ec13f3d0 2022-06-29T16:05:27.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.986+0000 7f13eaffd700 1 --2- 172.21.15.60:0/362665709 >> v2:172.21.15.92:3300/0 conn(0x7f13ec143820 0x7f13ec0b6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:05:27.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.986+0000 7f13eb7fe700 1 --2- 172.21.15.60:0/362665709 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13ec13f3d0 0x7f13ec060260 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f13dc002f70 tx=0x7f13dc006df0).ready entity=mon.2 client_cookie=a9429bdb0304d91d server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:27.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.986+0000 7f13ea7fc700 1 -- 172.21.15.60:0/362665709 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13dc00d570 con 0x7f13ec13f3d0 2022-06-29T16:05:27.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.986+0000 7f13ea7fc700 1 -- 172.21.15.60:0/362665709 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13dc000e90 con 0x7f13ec13f3d0 2022-06-29T16:05:27.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.986+0000 7f13ea7fc700 1 -- 172.21.15.60:0/362665709 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13dc00d8c0 con 0x7f13ec13f3d0 2022-06-29T16:05:27.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.986+0000 7f13f1e47700 1 -- 172.21.15.60:0/362665709 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13ec13f3d0 msgr2=0x7f13ec060260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:27.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.986+0000 7f13f1e47700 1 --2- 172.21.15.60:0/362665709 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13ec13f3d0 0x7f13ec060260 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f13dc002f70 tx=0x7f13dc006df0).stop 2022-06-29T16:05:27.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.986+0000 7f13f1e47700 1 -- 172.21.15.60:0/362665709 shutdown_connections 2022-06-29T16:05:27.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.986+0000 7f13f1e47700 1 --2- 172.21.15.60:0/362665709 >> v2:172.21.15.92:3300/0 conn(0x7f13ec143820 0x7f13ec0b6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:27.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.986+0000 7f13f1e47700 1 --2- 172.21.15.60:0/362665709 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13ec13f3d0 0x7f13ec060260 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:27.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.986+0000 7f13f1e47700 1 -- 172.21.15.60:0/362665709 >> 172.21.15.60:0/362665709 conn(0x7f13ec0643e0 msgr2=0x7f13ec0b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:27.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.986+0000 7f13f1e47700 1 -- 172.21.15.60:0/362665709 shutdown_connections 2022-06-29T16:05:27.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.986+0000 7f13f1e47700 1 -- 172.21.15.60:0/362665709 wait complete. 2022-06-29T16:05:28.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.990+0000 7f13f1e47700 1 Processor -- start 2022-06-29T16:05:28.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.990+0000 7f13f1e47700 1 -- start start 2022-06-29T16:05:28.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.990+0000 7f13f1e47700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13ec0b6c00 0x7f13ec0b55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:28.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.990+0000 7f13f1e47700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13ec13f3d0 0x7f13ec0af940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:28.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.990+0000 7f13f1e47700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13ec143820 0x7f13ec0afe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:28.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.990+0000 7f13f1e47700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec065da0 con 0x7f13ec143820 2022-06-29T16:05:28.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.990+0000 7f13f1e47700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec09ba20 con 0x7f13ec13f3d0 2022-06-29T16:05:28.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.990+0000 7f13f1e47700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec09acf0 con 0x7f13ec0b6c00 2022-06-29T16:05:28.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.990+0000 7f13ebfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13ec143820 0x7f13ec0afe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:28.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.990+0000 7f13ebfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13ec143820 0x7f13ec0afe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50348/0 (socket says 172.21.15.60:50348) 2022-06-29T16:05:28.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.990+0000 7f13ebfff700 1 -- 172.21.15.60:0/3358568503 learned_addr learned my addr 172.21.15.60:0/3358568503 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:28.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.990+0000 7f13eb7fe700 1 --2- 172.21.15.60:0/3358568503 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13ec0b6c00 0x7f13ec0b55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:28.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.990+0000 7f13eaffd700 1 --2- 172.21.15.60:0/3358568503 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13ec13f3d0 0x7f13ec0af940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:28.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.990+0000 7f13ebfff700 1 -- 172.21.15.60:0/3358568503 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13ec0b6c00 msgr2=0x7f13ec0b55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:28.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.990+0000 7f13ebfff700 1 --2- 172.21.15.60:0/3358568503 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13ec0b6c00 0x7f13ec0b55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:28.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.990+0000 7f13ebfff700 1 -- 172.21.15.60:0/3358568503 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13ec13f3d0 msgr2=0x7f13ec0af940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:28.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.990+0000 7f13ebfff700 1 --2- 172.21.15.60:0/3358568503 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13ec13f3d0 0x7f13ec0af940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:28.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.990+0000 7f13ebfff700 1 -- 172.21.15.60:0/3358568503 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e000b180 con 0x7f13ec143820 2022-06-29T16:05:28.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.994+0000 7f13ebfff700 1 --2- 172.21.15.60:0/3358568503 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13ec143820 0x7f13ec0afe30 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 rx=0x7f13ec066990 tx=0x7f13e4007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:28.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.994+0000 7f13e8ff9700 1 -- 172.21.15.60:0/3358568503 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13e4005ba0 con 0x7f13ec143820 2022-06-29T16:05:28.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.994+0000 7f13e8ff9700 1 -- 172.21.15.60:0/3358568503 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13e4005d00 con 0x7f13ec143820 2022-06-29T16:05:28.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.994+0000 7f13f1e47700 1 -- 172.21.15.60:0/3358568503 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13dc00b180 con 0x7f13ec143820 2022-06-29T16:05:28.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.994+0000 7f13e8ff9700 1 -- 172.21.15.60:0/3358568503 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13e40072c0 con 0x7f13ec143820 2022-06-29T16:05:28.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.994+0000 7f13f1e47700 1 -- 172.21.15.60:0/3358568503 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13ec0b0320 con 0x7f13ec143820 2022-06-29T16:05:28.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.994+0000 7f13e8ff9700 1 -- 172.21.15.60:0/3358568503 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f13e4010070 con 0x7f13ec143820 2022-06-29T16:05:28.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.994+0000 7f13e8ff9700 1 --2- 172.21.15.60:0/3358568503 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f13cc0539e0 0x7f13cc055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:28.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.994+0000 7f13e8ff9700 1 -- 172.21.15.60:0/3358568503 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f13e406b870 con 0x7f13ec143820 2022-06-29T16:05:28.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.994+0000 7f13f1e47700 1 -- 172.21.15.60:0/3358568503 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13ec09a910 con 0x7f13ec143820 2022-06-29T16:05:28.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.994+0000 7f13eb7fe700 1 --2- 172.21.15.60:0/3358568503 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f13cc0539e0 0x7f13cc055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:28.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:27.994+0000 7f13eb7fe700 1 --2- 172.21.15.60:0/3358568503 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f13cc0539e0 0x7f13cc055ea0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f13dc01a000 tx=0x7f13dc014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:28.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.002+0000 7f13e8ff9700 1 -- 172.21.15.60:0/3358568503 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f13e40426a0 con 0x7f13ec143820 2022-06-29T16:05:28.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.170+0000 7f13f1e47700 1 -- 172.21.15.60:0/3358568503 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13ec0573d0 con 0x7f13ec143820 2022-06-29T16:05:28.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.170+0000 7f13e8ff9700 1 -- 172.21.15.60:0/3358568503 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+2141 (secure 0 0 0) 0x7f13e404a020 con 0x7f13ec143820 2022-06-29T16:05:28.178 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:28.178 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":36,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6657,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.60:6833/3121204394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3121204394},{"type":"v1","addr":"172.21.15.60:6833","nonce":3121204394}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":34}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:05:16.293932+0000","modified":"2022-06-29T16:05:18.324125+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6688},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6688":{"gid":6688,"name":"b","rank":0,"incarnation":35,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/1245503283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1245503283},{"type":"v1","addr":"172.21.15.92:6835","nonce":1245503283}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-06-29T16:05:28.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.174+0000 7f13f1e47700 1 -- 172.21.15.60:0/3358568503 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f13cc0539e0 msgr2=0x7f13cc055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:28.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.174+0000 7f13f1e47700 1 --2- 172.21.15.60:0/3358568503 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f13cc0539e0 0x7f13cc055ea0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f13dc01a000 tx=0x7f13dc014000).stop 2022-06-29T16:05:28.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.174+0000 7f13f1e47700 1 -- 172.21.15.60:0/3358568503 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13ec143820 msgr2=0x7f13ec0afe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:28.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.174+0000 7f13f1e47700 1 --2- 172.21.15.60:0/3358568503 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13ec143820 0x7f13ec0afe30 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 rx=0x7f13ec066990 tx=0x7f13e4007f40).stop 2022-06-29T16:05:28.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.174+0000 7f13f1e47700 1 -- 172.21.15.60:0/3358568503 shutdown_connections 2022-06-29T16:05:28.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.174+0000 7f13f1e47700 1 --2- 172.21.15.60:0/3358568503 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f13cc0539e0 0x7f13cc055ea0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:28.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.174+0000 7f13f1e47700 1 --2- 172.21.15.60:0/3358568503 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13ec143820 0x7f13ec0afe30 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:28.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.174+0000 7f13f1e47700 1 --2- 172.21.15.60:0/3358568503 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13ec13f3d0 0x7f13ec0af940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:28.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.174+0000 7f13f1e47700 1 --2- 172.21.15.60:0/3358568503 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13ec0b6c00 0x7f13ec0b55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:28.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.174+0000 7f13f1e47700 1 -- 172.21.15.60:0/3358568503 >> 172.21.15.60:0/3358568503 conn(0x7f13ec0643e0 msgr2=0x7f13ec13fb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:28.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.174+0000 7f13f1e47700 1 -- 172.21.15.60:0/3358568503 shutdown_connections 2022-06-29T16:05:28.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.174+0000 7f13f1e47700 1 -- 172.21.15.60:0/3358568503 wait complete. 2022-06-29T16:05:28.185 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 36 2022-06-29T16:05:28.197 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-06-29T16:05:28.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.298+0000 7f69d60fa700 1 Processor -- start 2022-06-29T16:05:28.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.298+0000 7f69d60fa700 1 -- start start 2022-06-29T16:05:28.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.298+0000 7f69d60fa700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f69d005d070 0x7f69d014aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:28.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.298+0000 7f69d60fa700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f69d00b2130 0x7f69d00aca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:28.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.298+0000 7f69d60fa700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69d005d450 con 0x7f69d014af80 2022-06-29T16:05:28.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.298+0000 7f69d60fa700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69d00adf90 con 0x7f69d005d070 2022-06-29T16:05:28.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.298+0000 7f69d60fa700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69d00b25d0 con 0x7f69d00b2130 2022-06-29T16:05:28.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.298+0000 7f69d50f8700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f69d005d070 0x7f69d014aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:28.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.298+0000 7f69d50f8700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f69d005d070 0x7f69d014aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50354/0 (socket says 172.21.15.60:50354) 2022-06-29T16:05:28.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.298+0000 7f69d50f8700 1 -- 172.21.15.60:0/495875666 learned_addr learned my addr 172.21.15.60:0/495875666 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:28.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.298+0000 7f69d48f7700 1 --2- 172.21.15.60:0/495875666 >> v2:172.21.15.92:3300/0 conn(0x7f69d00b2130 0x7f69d00aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:28.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.298+0000 7f69d50f8700 1 -- 172.21.15.60:0/495875666 >> v2:172.21.15.92:3300/0 conn(0x7f69d00b2130 msgr2=0x7f69d00aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:28.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.298+0000 7f69d50f8700 1 --2- 172.21.15.60:0/495875666 >> v2:172.21.15.92:3300/0 conn(0x7f69d00b2130 0x7f69d00aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:28.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.298+0000 7f69d50f8700 1 -- 172.21.15.60:0/495875666 >> v1:172.21.15.92:6789/0 conn(0x7f69d014af80 legacy=0x7f69d00acf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:28.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.298+0000 7f69d50f8700 1 -- 172.21.15.60:0/495875666 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d014a350 con 0x7f69d005d070 2022-06-29T16:05:28.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.298+0000 7f69d50f8700 1 --2- 172.21.15.60:0/495875666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69d005d070 0x7f69d014aa90 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 rx=0x7f69c4002f70 tx=0x7f69c4006eb0).ready entity=mon.0 client_cookie=548ffab89eec402b server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:28.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.298+0000 7f69cffff700 1 -- 172.21.15.60:0/495875666 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69c400d610 con 0x7f69d005d070 2022-06-29T16:05:28.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.298+0000 7f69cffff700 1 -- 172.21.15.60:0/495875666 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69c4000e90 con 0x7f69d005d070 2022-06-29T16:05:28.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.298+0000 7f69cffff700 1 -- 172.21.15.60:0/495875666 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69c400d9a0 con 0x7f69d005d070 2022-06-29T16:05:28.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.298+0000 7f69d60fa700 1 -- 172.21.15.60:0/495875666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69d005d070 msgr2=0x7f69d014aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:28.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.298+0000 7f69d60fa700 1 --2- 172.21.15.60:0/495875666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69d005d070 0x7f69d014aa90 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 rx=0x7f69c4002f70 tx=0x7f69c4006eb0).stop 2022-06-29T16:05:28.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.298+0000 7f69d60fa700 1 -- 172.21.15.60:0/495875666 shutdown_connections 2022-06-29T16:05:28.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.298+0000 7f69d60fa700 1 --2- 172.21.15.60:0/495875666 >> v2:172.21.15.92:3300/0 conn(0x7f69d00b2130 0x7f69d00aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:28.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.298+0000 7f69d60fa700 1 --2- 172.21.15.60:0/495875666 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69d005d070 0x7f69d014aa90 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:28.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.298+0000 7f69d60fa700 1 -- 172.21.15.60:0/495875666 >> 172.21.15.60:0/495875666 conn(0x7f69d00b5890 msgr2=0x7f69d00b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:28.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.298+0000 7f69d60fa700 1 -- 172.21.15.60:0/495875666 shutdown_connections 2022-06-29T16:05:28.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.298+0000 7f69d60fa700 1 -- 172.21.15.60:0/495875666 wait complete. 2022-06-29T16:05:28.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.302+0000 7f69d60fa700 1 Processor -- start 2022-06-29T16:05:28.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.302+0000 7f69d60fa700 1 -- start start 2022-06-29T16:05:28.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.302+0000 7f69d60fa700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69d005d070 0x7f69d00b4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:28.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.302+0000 7f69d60fa700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69d00b2130 0x7f69d0142d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:28.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.302+0000 7f69d60fa700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f69d014af80 0x7f69d01431f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:28.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.302+0000 7f69d60fa700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d00636e0 con 0x7f69d00b2130 2022-06-29T16:05:28.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.302+0000 7f69d60fa700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d0064880 con 0x7f69d005d070 2022-06-29T16:05:28.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.302+0000 7f69d60fa700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69d0064db0 con 0x7f69d014af80 2022-06-29T16:05:28.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.302+0000 7f69d48f7700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69d00b2130 0x7f69d0142d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:28.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.302+0000 7f69d48f7700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69d00b2130 0x7f69d0142d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50360/0 (socket says 172.21.15.60:50360) 2022-06-29T16:05:28.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.302+0000 7f69d48f7700 1 -- 172.21.15.60:0/1713975791 learned_addr learned my addr 172.21.15.60:0/1713975791 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:28.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.302+0000 7f69d58f9700 1 --2- 172.21.15.60:0/1713975791 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f69d014af80 0x7f69d01431f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:28.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.302+0000 7f69d50f8700 1 --2- 172.21.15.60:0/1713975791 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69d005d070 0x7f69d00b4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:28.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.302+0000 7f69d48f7700 1 -- 172.21.15.60:0/1713975791 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f69d014af80 msgr2=0x7f69d01431f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:28.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.302+0000 7f69d48f7700 1 --2- 172.21.15.60:0/1713975791 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f69d014af80 0x7f69d01431f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:28.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.302+0000 7f69d48f7700 1 -- 172.21.15.60:0/1713975791 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69d005d070 msgr2=0x7f69d00b4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:28.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.302+0000 7f69d48f7700 1 --2- 172.21.15.60:0/1713975791 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69d005d070 0x7f69d00b4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:28.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.302+0000 7f69d48f7700 1 -- 172.21.15.60:0/1713975791 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c400b1e0 con 0x7f69d00b2130 2022-06-29T16:05:28.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.306+0000 7f69d48f7700 1 --2- 172.21.15.60:0/1713975791 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69d00b2130 0x7f69d0142d00 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 rx=0x7f69c000a000 tx=0x7f69c0008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:28.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.306+0000 7f69ce7fc700 1 -- 172.21.15.60:0/1713975791 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69c0013620 con 0x7f69d00b2130 2022-06-29T16:05:28.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.306+0000 7f69ce7fc700 1 -- 172.21.15.60:0/1713975791 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69c0013780 con 0x7f69d00b2130 2022-06-29T16:05:28.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.306+0000 7f69ce7fc700 1 -- 172.21.15.60:0/1713975791 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69c0013ae0 con 0x7f69d00b2130 2022-06-29T16:05:28.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.306+0000 7f69d60fa700 1 -- 172.21.15.60:0/1713975791 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69d0063240 con 0x7f69d00b2130 2022-06-29T16:05:28.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.306+0000 7f69d60fa700 1 -- 172.21.15.60:0/1713975791 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69d0157320 con 0x7f69d00b2130 2022-06-29T16:05:28.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.306+0000 7f69d60fa700 1 -- 172.21.15.60:0/1713975791 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69d014a5f0 con 0x7f69d00b2130 2022-06-29T16:05:28.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.306+0000 7f69ce7fc700 1 -- 172.21.15.60:0/1713975791 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f69c0013c40 con 0x7f69d00b2130 2022-06-29T16:05:28.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.306+0000 7f69ce7fc700 1 --2- 172.21.15.60:0/1713975791 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f69b0053a70 0x7f69b0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:28.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.306+0000 7f69ce7fc700 1 -- 172.21.15.60:0/1713975791 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f69c006bd80 con 0x7f69d00b2130 2022-06-29T16:05:28.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.306+0000 7f69d50f8700 1 --2- 172.21.15.60:0/1713975791 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f69b0053a70 0x7f69b0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:28.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.306+0000 7f69d50f8700 1 --2- 172.21.15.60:0/1713975791 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f69b0053a70 0x7f69b0055f30 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f69c401c000 tx=0x7f69c4015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:28.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.314+0000 7f69ce7fc700 1 -- 172.21.15.60:0/1713975791 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f69c006f030 con 0x7f69d00b2130 2022-06-29T16:05:28.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.474+0000 7f69d60fa700 1 -- 172.21.15.60:0/1713975791 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f69d0064bd0 con 0x7f69d00b2130 2022-06-29T16:05:28.920 INFO:tasks.ceph.mon.a.smithi060.stderr:2022-06-29T16:05:28.914+0000 7fcc1a376700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-06-29T16:05:28.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.918+0000 7f69ce7fc700 1 -- 172.21.15.60:0/1713975791 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v37) v1 ==== 161+0+0 (secure 0 0 0) 0x7f69c0042b70 con 0x7f69d00b2130 2022-06-29T16:05:28.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.922+0000 7f69d60fa700 1 -- 172.21.15.60:0/1713975791 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f69b0053a70 msgr2=0x7f69b0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:28.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.922+0000 7f69d60fa700 1 --2- 172.21.15.60:0/1713975791 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f69b0053a70 0x7f69b0055f30 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f69c401c000 tx=0x7f69c4015000).stop 2022-06-29T16:05:28.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.922+0000 7f69d60fa700 1 -- 172.21.15.60:0/1713975791 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69d00b2130 msgr2=0x7f69d0142d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:28.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.922+0000 7f69d60fa700 1 --2- 172.21.15.60:0/1713975791 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69d00b2130 0x7f69d0142d00 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 rx=0x7f69c000a000 tx=0x7f69c0008040).stop 2022-06-29T16:05:28.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.922+0000 7f69d60fa700 1 -- 172.21.15.60:0/1713975791 shutdown_connections 2022-06-29T16:05:28.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.922+0000 7f69d60fa700 1 --2- 172.21.15.60:0/1713975791 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f69b0053a70 0x7f69b0055f30 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:28.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.922+0000 7f69d60fa700 1 --2- 172.21.15.60:0/1713975791 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f69d014af80 0x7f69d01431f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:28.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.922+0000 7f69d60fa700 1 --2- 172.21.15.60:0/1713975791 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f69d00b2130 0x7f69d0142d00 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:28.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.922+0000 7f69d60fa700 1 --2- 172.21.15.60:0/1713975791 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69d005d070 0x7f69d00b4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:28.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.922+0000 7f69d60fa700 1 -- 172.21.15.60:0/1713975791 >> 172.21.15.60:0/1713975791 conn(0x7f69d00b5890 msgr2=0x7f69d01460e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:28.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.922+0000 7f69d60fa700 1 -- 172.21.15.60:0/1713975791 shutdown_connections 2022-06-29T16:05:28.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:28.922+0000 7f69d60fa700 1 -- 172.21.15.60:0/1713975791 wait complete. 2022-06-29T16:05:28.934 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-06-29T16:05:28.948 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:05:28.959 INFO:tasks.ceph.mds.b.smithi092.stdout:starting mds.b at 2022-06-29T16:05:28.983 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:05:28.985 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:05:28.988 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:05:28.990 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:05:28.993 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:05:29.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.050+0000 7f1f46dde700 1 Processor -- start 2022-06-29T16:05:29.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.050+0000 7f1f46dde700 1 -- start start 2022-06-29T16:05:29.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.050+0000 7f1f46dde700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1f400b5950 0x7f1f401537f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:29.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.050+0000 7f1f46dde700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1f40063a20 0x7f1f400800a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:29.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.050+0000 7f1f46dde700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f400b5d30 con 0x7f1f40080590 2022-06-29T16:05:29.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.050+0000 7f1f46dde700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f40150860 con 0x7f1f40063a20 2022-06-29T16:05:29.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.050+0000 7f1f46dde700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f40067370 con 0x7f1f400b5950 2022-06-29T16:05:29.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.050+0000 7f1f3ffff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1f40063a20 0x7f1f400800a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:29.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.050+0000 7f1f4537b700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f1f40080590 0x7f1f4005bd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58454/0 (socket says 172.21.15.60:58454) 2022-06-29T16:05:29.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.050+0000 7f1f4537b700 1 -- 172.21.15.60:0/968055642 learned_addr learned my addr 172.21.15.60:0/968055642 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:29.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.054+0000 7f1f3f7fe700 1 -- 172.21.15.60:0/968055642 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 301633064 0 0) 0x7f1f400b5d30 con 0x7f1f40080590 2022-06-29T16:05:29.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.050+0000 7f1f44b7a700 1 --2- 172.21.15.60:0/968055642 >> v2:172.21.15.92:3300/0 conn(0x7f1f400b5950 0x7f1f401537f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:29.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.138+0000 7f1f3f7fe700 1 -- 172.21.15.60:0/968055642 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f28003240 con 0x7f1f40080590 2022-06-29T16:05:29.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.138+0000 7f1f3ffff700 1 -- 172.21.15.60:0/968055642 >> v2:172.21.15.92:3300/0 conn(0x7f1f400b5950 msgr2=0x7f1f401537f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:29.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.138+0000 7f1f3ffff700 1 --2- 172.21.15.60:0/968055642 >> v2:172.21.15.92:3300/0 conn(0x7f1f400b5950 0x7f1f401537f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:29.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.138+0000 7f1f3ffff700 1 -- 172.21.15.60:0/968055642 >> v1:172.21.15.60:6789/0 conn(0x7f1f40080590 legacy=0x7f1f4005bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:29.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.138+0000 7f1f3ffff700 1 -- 172.21.15.60:0/968055642 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f40067610 con 0x7f1f40063a20 2022-06-29T16:05:29.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.138+0000 7f1f44b7a700 1 --2- 172.21.15.60:0/968055642 >> v2:172.21.15.92:3300/0 conn(0x7f1f400b5950 0x7f1f401537f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:05:29.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.138+0000 7f1f3ffff700 1 --2- 172.21.15.60:0/968055642 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f40063a20 0x7f1f400800a0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 rx=0x7f1f30002f70 tx=0x7f1f30006df0).ready entity=mon.0 client_cookie=74e0a7ebbee00e5a server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:29.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.138+0000 7f1f3f7fe700 1 -- 172.21.15.60:0/968055642 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f3000d6b0 con 0x7f1f40063a20 2022-06-29T16:05:29.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.138+0000 7f1f3f7fe700 1 -- 172.21.15.60:0/968055642 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f30000e90 con 0x7f1f40063a20 2022-06-29T16:05:29.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.138+0000 7f1f3f7fe700 1 -- 172.21.15.60:0/968055642 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f3000da00 con 0x7f1f40063a20 2022-06-29T16:05:29.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.138+0000 7f1f46dde700 1 -- 172.21.15.60:0/968055642 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f40063a20 msgr2=0x7f1f400800a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:29.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.138+0000 7f1f46dde700 1 --2- 172.21.15.60:0/968055642 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f40063a20 0x7f1f400800a0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 rx=0x7f1f30002f70 tx=0x7f1f30006df0).stop 2022-06-29T16:05:29.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.138+0000 7f1f46dde700 1 -- 172.21.15.60:0/968055642 shutdown_connections 2022-06-29T16:05:29.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.138+0000 7f1f46dde700 1 --2- 172.21.15.60:0/968055642 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f40063a20 0x7f1f400800a0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:29.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.138+0000 7f1f46dde700 1 --2- 172.21.15.60:0/968055642 >> v2:172.21.15.92:3300/0 conn(0x7f1f400b5950 0x7f1f401537f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:29.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.138+0000 7f1f46dde700 1 -- 172.21.15.60:0/968055642 >> 172.21.15.60:0/968055642 conn(0x7f1f400632d0 msgr2=0x7f1f400b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:29.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.138+0000 7f1f46dde700 1 -- 172.21.15.60:0/968055642 shutdown_connections 2022-06-29T16:05:29.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.138+0000 7f1f46dde700 1 -- 172.21.15.60:0/968055642 wait complete. 2022-06-29T16:05:29.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.142+0000 7f1f46dde700 1 Processor -- start 2022-06-29T16:05:29.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.142+0000 7f1f46dde700 1 -- start start 2022-06-29T16:05:29.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.142+0000 7f1f46dde700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1f40063a20 0x7f1f40070e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:29.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.142+0000 7f1f46dde700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f40080590 0x7f1f40071300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:29.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.142+0000 7f1f46dde700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1f400b5950 0x7f1f40073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:29.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.142+0000 7f1f46dde700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f40070900 con 0x7f1f40080590 2022-06-29T16:05:29.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.142+0000 7f1f46dde700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f40070660 con 0x7f1f40063a20 2022-06-29T16:05:29.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.142+0000 7f1f46dde700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f40072bf0 con 0x7f1f400b5950 2022-06-29T16:05:29.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.142+0000 7f1f3ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f40080590 0x7f1f40071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:29.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.146+0000 7f1f3ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f40080590 0x7f1f40071300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50374/0 (socket says 172.21.15.60:50374) 2022-06-29T16:05:29.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.146+0000 7f1f3ffff700 1 -- 172.21.15.60:0/1069668326 learned_addr learned my addr 172.21.15.60:0/1069668326 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:29.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.146+0000 7f1f44b7a700 1 --2- 172.21.15.60:0/1069668326 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1f40063a20 0x7f1f40070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:29.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.146+0000 7f1f4537b700 1 --2- 172.21.15.60:0/1069668326 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1f400b5950 0x7f1f40073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:29.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.146+0000 7f1f3ffff700 1 -- 172.21.15.60:0/1069668326 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1f400b5950 msgr2=0x7f1f40073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:29.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.146+0000 7f1f3ffff700 1 --2- 172.21.15.60:0/1069668326 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1f400b5950 0x7f1f40073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:29.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.146+0000 7f1f3ffff700 1 -- 172.21.15.60:0/1069668326 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1f40063a20 msgr2=0x7f1f40070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:29.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.146+0000 7f1f3ffff700 1 --2- 172.21.15.60:0/1069668326 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1f40063a20 0x7f1f40070e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:29.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.146+0000 7f1f3ffff700 1 -- 172.21.15.60:0/1069668326 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f3000c290 con 0x7f1f40080590 2022-06-29T16:05:29.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.146+0000 7f1f3ffff700 1 --2- 172.21.15.60:0/1069668326 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f40080590 0x7f1f40071300 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 rx=0x7f1f30002f70 tx=0x7f1f30015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:29.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.146+0000 7f1f3dffb700 1 -- 172.21.15.60:0/1069668326 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f30010d70 con 0x7f1f40080590 2022-06-29T16:05:29.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.146+0000 7f1f3dffb700 1 -- 172.21.15.60:0/1069668326 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f3000d330 con 0x7f1f40080590 2022-06-29T16:05:29.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.146+0000 7f1f46dde700 1 -- 172.21.15.60:0/1069668326 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f400b07a0 con 0x7f1f40080590 2022-06-29T16:05:29.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.146+0000 7f1f3dffb700 1 -- 172.21.15.60:0/1069668326 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f300205f0 con 0x7f1f40080590 2022-06-29T16:05:29.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.146+0000 7f1f46dde700 1 -- 172.21.15.60:0/1069668326 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f40073580 con 0x7f1f40080590 2022-06-29T16:05:29.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.150+0000 7f1f3dffb700 1 -- 172.21.15.60:0/1069668326 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1f3000e070 con 0x7f1f40080590 2022-06-29T16:05:29.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.150+0000 7f1f46dde700 1 -- 172.21.15.60:0/1069668326 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f400704b0 con 0x7f1f40080590 2022-06-29T16:05:29.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.150+0000 7f1f3dffb700 1 --2- 172.21.15.60:0/1069668326 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1f200539e0 0x7f1f20055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:29.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.150+0000 7f1f3dffb700 1 -- 172.21.15.60:0/1069668326 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f1f30019030 con 0x7f1f40080590 2022-06-29T16:05:29.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.150+0000 7f1f44b7a700 1 --2- 172.21.15.60:0/1069668326 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1f200539e0 0x7f1f20055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:29.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.150+0000 7f1f44b7a700 1 --2- 172.21.15.60:0/1069668326 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1f200539e0 0x7f1f20055ea0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f1f3400c000 tx=0x7f1f34006dd0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:29.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.158+0000 7f1f3dffb700 1 -- 172.21.15.60:0/1069668326 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1f30043ab0 con 0x7f1f40080590 2022-06-29T16:05:29.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.326+0000 7f1f46dde700 1 -- 172.21.15.60:0/1069668326 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f40072db0 con 0x7f1f40080590 2022-06-29T16:05:29.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.326+0000 7f1f3dffb700 1 -- 172.21.15.60:0/1069668326 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+1779 (secure 0 0 0) 0x7f1f30014070 con 0x7f1f40080590 2022-06-29T16:05:29.335 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:29.335 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":37,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6657,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.60:6833/3121204394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3121204394},{"type":"v1","addr":"172.21.15.60:6833","nonce":3121204394}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":34}],"filesystems":[{"mdsmap":{"epoch":37,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:05:16.293932+0000","modified":"2022-06-29T16:05:28.917913+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-06-29T16:05:29.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.330+0000 7f1f46dde700 1 -- 172.21.15.60:0/1069668326 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1f200539e0 msgr2=0x7f1f20055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:29.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.330+0000 7f1f46dde700 1 --2- 172.21.15.60:0/1069668326 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1f200539e0 0x7f1f20055ea0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f1f3400c000 tx=0x7f1f34006dd0).stop 2022-06-29T16:05:29.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.330+0000 7f1f46dde700 1 -- 172.21.15.60:0/1069668326 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f40080590 msgr2=0x7f1f40071300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:29.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.330+0000 7f1f46dde700 1 --2- 172.21.15.60:0/1069668326 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f40080590 0x7f1f40071300 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 rx=0x7f1f30002f70 tx=0x7f1f30015000).stop 2022-06-29T16:05:29.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.330+0000 7f1f46dde700 1 -- 172.21.15.60:0/1069668326 shutdown_connections 2022-06-29T16:05:29.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.330+0000 7f1f46dde700 1 --2- 172.21.15.60:0/1069668326 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1f200539e0 0x7f1f20055ea0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:29.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.330+0000 7f1f46dde700 1 --2- 172.21.15.60:0/1069668326 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1f400b5950 0x7f1f40073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:29.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.330+0000 7f1f46dde700 1 --2- 172.21.15.60:0/1069668326 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f40080590 0x7f1f40071300 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:29.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.330+0000 7f1f46dde700 1 --2- 172.21.15.60:0/1069668326 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1f40063a20 0x7f1f40070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:29.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.330+0000 7f1f46dde700 1 -- 172.21.15.60:0/1069668326 >> 172.21.15.60:0/1069668326 conn(0x7f1f400632d0 msgr2=0x7f1f400b4530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:29.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.330+0000 7f1f46dde700 1 -- 172.21.15.60:0/1069668326 shutdown_connections 2022-06-29T16:05:29.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.330+0000 7f1f46dde700 1 -- 172.21.15.60:0/1069668326 wait complete. 2022-06-29T16:05:29.341 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 37 2022-06-29T16:05:29.348 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-06-29T16:05:29.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.450+0000 7fcd0ebde700 1 Processor -- start 2022-06-29T16:05:29.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.450+0000 7fcd0ebde700 1 -- start start 2022-06-29T16:05:29.461 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.450+0000 7fcd0ebde700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcd080b5950 0x7fcd08067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:29.461 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.450+0000 7fcd0ebde700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd0814af10 0x7fcd0805c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:29.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.450+0000 7fcd0ebde700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd080acba0 con 0x7fcd08063ad0 2022-06-29T16:05:29.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.450+0000 7fcd0ebde700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd0814b3a0 con 0x7fcd080b5950 2022-06-29T16:05:29.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.450+0000 7fcd0ebde700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd08067220 con 0x7fcd0814af10 2022-06-29T16:05:29.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.450+0000 7fcd0c97a700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcd080b5950 0x7fcd08067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:29.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.450+0000 7fcd0d17b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd0814af10 0x7fcd0805c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:29.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.450+0000 7fcd07fff700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fcd08063ad0 0x7fcd0805bd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:34484/0 (socket says 172.21.15.60:34484) 2022-06-29T16:05:29.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.450+0000 7fcd07fff700 1 -- 172.21.15.60:0/3264170462 learned_addr learned my addr 172.21.15.60:0/3264170462 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:29.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.450+0000 7fcd0c97a700 1 -- 172.21.15.60:0/3264170462 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd0814af10 msgr2=0x7fcd0805c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:29.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.450+0000 7fcd0c97a700 1 --2- 172.21.15.60:0/3264170462 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd0814af10 0x7fcd0805c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:29.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.450+0000 7fcd0c97a700 1 -- 172.21.15.60:0/3264170462 >> v1:172.21.15.92:6789/0 conn(0x7fcd08063ad0 legacy=0x7fcd0805bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:29.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.450+0000 7fcd0c97a700 1 -- 172.21.15.60:0/3264170462 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd080b5d30 con 0x7fcd080b5950 2022-06-29T16:05:29.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.450+0000 7fcd0c97a700 1 --2- 172.21.15.60:0/3264170462 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd080b5950 0x7fcd08067680 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fccfc002f70 tx=0x7fccfc006df0).ready entity=mon.1 client_cookie=e02fdd2ecc087309 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:29.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.450+0000 7fcd077fe700 1 -- 172.21.15.60:0/3264170462 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fccfc00d570 con 0x7fcd080b5950 2022-06-29T16:05:29.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.450+0000 7fcd077fe700 1 -- 172.21.15.60:0/3264170462 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccfc000e90 con 0x7fcd080b5950 2022-06-29T16:05:29.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.450+0000 7fcd077fe700 1 -- 172.21.15.60:0/3264170462 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fccfc00d8c0 con 0x7fcd080b5950 2022-06-29T16:05:29.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.450+0000 7fcd0ebde700 1 -- 172.21.15.60:0/3264170462 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd080b5950 msgr2=0x7fcd08067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:29.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.450+0000 7fcd0ebde700 1 --2- 172.21.15.60:0/3264170462 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd080b5950 0x7fcd08067680 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fccfc002f70 tx=0x7fccfc006df0).stop 2022-06-29T16:05:29.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.450+0000 7fcd0ebde700 1 -- 172.21.15.60:0/3264170462 shutdown_connections 2022-06-29T16:05:29.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.450+0000 7fcd0ebde700 1 --2- 172.21.15.60:0/3264170462 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd0814af10 0x7fcd0805c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:29.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.450+0000 7fcd0ebde700 1 --2- 172.21.15.60:0/3264170462 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd080b5950 0x7fcd08067680 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:29.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.450+0000 7fcd0ebde700 1 -- 172.21.15.60:0/3264170462 >> 172.21.15.60:0/3264170462 conn(0x7fcd080632c0 msgr2=0x7fcd080b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:29.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.450+0000 7fcd0ebde700 1 -- 172.21.15.60:0/3264170462 shutdown_connections 2022-06-29T16:05:29.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.454+0000 7fcd0ebde700 1 -- 172.21.15.60:0/3264170462 wait complete. 2022-06-29T16:05:29.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.454+0000 7fcd0ebde700 1 Processor -- start 2022-06-29T16:05:29.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.454+0000 7fcd0ebde700 1 -- start start 2022-06-29T16:05:29.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.454+0000 7fcd0ebde700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd08063ad0 0x7fcd08064f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:29.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.454+0000 7fcd0ebde700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd080b5950 0x7fcd08142420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:29.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.454+0000 7fcd0ebde700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd0814af10 0x7fcd081181f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:29.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.454+0000 7fcd0ebde700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd081176b0 con 0x7fcd0814af10 2022-06-29T16:05:29.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.454+0000 7fcd0ebde700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd08117110 con 0x7fcd080b5950 2022-06-29T16:05:29.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.454+0000 7fcd0ebde700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd080b0a10 con 0x7fcd08063ad0 2022-06-29T16:05:29.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.454+0000 7fcd0c97a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd08063ad0 0x7fcd08064f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:29.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.454+0000 7fcd0d17b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd0814af10 0x7fcd081181f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:29.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.454+0000 7fcd0c97a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd08063ad0 0x7fcd08064f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:35248/0 (socket says 172.21.15.60:35248) 2022-06-29T16:05:29.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.454+0000 7fcd0c97a700 1 -- 172.21.15.60:0/2421953157 learned_addr learned my addr 172.21.15.60:0/2421953157 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:29.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.454+0000 7fcd07fff700 1 --2- 172.21.15.60:0/2421953157 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd080b5950 0x7fcd08142420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:29.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.454+0000 7fcd0c97a700 1 -- 172.21.15.60:0/2421953157 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd080b5950 msgr2=0x7fcd08142420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:29.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.454+0000 7fcd0c97a700 1 --2- 172.21.15.60:0/2421953157 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd080b5950 0x7fcd08142420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:29.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.454+0000 7fcd0c97a700 1 -- 172.21.15.60:0/2421953157 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd0814af10 msgr2=0x7fcd081181f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:29.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.454+0000 7fcd0c97a700 1 --2- 172.21.15.60:0/2421953157 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd0814af10 0x7fcd081181f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:29.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.454+0000 7fcd0c97a700 1 -- 172.21.15.60:0/2421953157 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccfc00b180 con 0x7fcd08063ad0 2022-06-29T16:05:29.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.454+0000 7fcd07fff700 1 --2- 172.21.15.60:0/2421953157 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd080b5950 0x7fcd08142420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:29.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.454+0000 7fcd0c97a700 1 --2- 172.21.15.60:0/2421953157 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd08063ad0 0x7fcd08064f40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fccfc002f70 tx=0x7fccfc014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:29.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.458+0000 7fcd05ffb700 1 -- 172.21.15.60:0/2421953157 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fccfc00dea0 con 0x7fcd08063ad0 2022-06-29T16:05:29.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.458+0000 7fcd05ffb700 1 -- 172.21.15.60:0/2421953157 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccfc013e90 con 0x7fcd08063ad0 2022-06-29T16:05:29.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.458+0000 7fcd0ebde700 1 -- 172.21.15.60:0/2421953157 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd08159640 con 0x7fcd08063ad0 2022-06-29T16:05:29.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.458+0000 7fcd05ffb700 1 -- 172.21.15.60:0/2421953157 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fccfc01e5f0 con 0x7fcd08063ad0 2022-06-29T16:05:29.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.458+0000 7fcd0ebde700 1 -- 172.21.15.60:0/2421953157 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd08116d90 con 0x7fcd08063ad0 2022-06-29T16:05:29.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.458+0000 7fcd0ebde700 1 -- 172.21.15.60:0/2421953157 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd08116f50 con 0x7fcd08063ad0 2022-06-29T16:05:29.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.458+0000 7fcd05ffb700 1 -- 172.21.15.60:0/2421953157 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fccfc025030 con 0x7fcd08063ad0 2022-06-29T16:05:29.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.458+0000 7fcd05ffb700 1 --2- 172.21.15.60:0/2421953157 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcce8053b00 0x7fcce8055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:29.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.458+0000 7fcd05ffb700 1 -- 172.21.15.60:0/2421953157 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fccfc06d250 con 0x7fcd08063ad0 2022-06-29T16:05:29.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.458+0000 7fcd07fff700 1 --2- 172.21.15.60:0/2421953157 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcce8053b00 0x7fcce8055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:29.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.462+0000 7fcd07fff700 1 --2- 172.21.15.60:0/2421953157 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcce8053b00 0x7fcce8055fc0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fccf400b000 tx=0x7fccf4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:29.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.466+0000 7fcd05ffb700 1 -- 172.21.15.60:0/2421953157 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fccfc0440c0 con 0x7fcd08063ad0 2022-06-29T16:05:29.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.630+0000 7fcd0ebde700 1 -- 172.21.15.60:0/2421953157 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fcd080573d0 con 0x7fcd08063ad0 2022-06-29T16:05:29.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.938+0000 7fcd05ffb700 1 -- 172.21.15.60:0/2421953157 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v38) v1 ==== 104+0+0 (secure 0 0 0) 0x7fccfc0230a0 con 0x7fcd08063ad0 2022-06-29T16:05:29.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.942+0000 7fcd0ebde700 1 -- 172.21.15.60:0/2421953157 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcce8053b00 msgr2=0x7fcce8055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:29.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.942+0000 7fcd0ebde700 1 --2- 172.21.15.60:0/2421953157 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcce8053b00 0x7fcce8055fc0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fccf400b000 tx=0x7fccf4009040).stop 2022-06-29T16:05:29.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.942+0000 7fcd0ebde700 1 -- 172.21.15.60:0/2421953157 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd08063ad0 msgr2=0x7fcd08064f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:29.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.942+0000 7fcd0ebde700 1 --2- 172.21.15.60:0/2421953157 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd08063ad0 0x7fcd08064f40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fccfc002f70 tx=0x7fccfc014000).stop 2022-06-29T16:05:29.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.942+0000 7fcd0ebde700 1 -- 172.21.15.60:0/2421953157 shutdown_connections 2022-06-29T16:05:29.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.942+0000 7fcd0ebde700 1 --2- 172.21.15.60:0/2421953157 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcce8053b00 0x7fcce8055fc0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:29.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.942+0000 7fcd0ebde700 1 --2- 172.21.15.60:0/2421953157 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd0814af10 0x7fcd081181f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:29.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.942+0000 7fcd0ebde700 1 --2- 172.21.15.60:0/2421953157 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd080b5950 0x7fcd08142420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:29.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.942+0000 7fcd0ebde700 1 --2- 172.21.15.60:0/2421953157 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd08063ad0 0x7fcd08064f40 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:29.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.942+0000 7fcd0ebde700 1 -- 172.21.15.60:0/2421953157 >> 172.21.15.60:0/2421953157 conn(0x7fcd080632c0 msgr2=0x7fcd08143290 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:29.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.942+0000 7fcd0ebde700 1 -- 172.21.15.60:0/2421953157 shutdown_connections 2022-06-29T16:05:29.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:29.942+0000 7fcd0ebde700 1 -- 172.21.15.60:0/2421953157 wait complete. 2022-06-29T16:05:29.966 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-06-29T16:05:30.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.074+0000 7fa9ee052700 1 Processor -- start 2022-06-29T16:05:30.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.074+0000 7fa9ee052700 1 -- start start 2022-06-29T16:05:30.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.078+0000 7fa9ee052700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa9e80b28e0 0x7fa9e805bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:30.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.078+0000 7fa9ee052700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e80b1d70 con 0x7fa9e80b9be0 2022-06-29T16:05:30.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.078+0000 7fa9ee052700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e80b9fc0 con 0x7fa9e80afee0 2022-06-29T16:05:30.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.078+0000 7fa9ee052700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9e80b3c40 con 0x7fa9e80b28e0 2022-06-29T16:05:30.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.078+0000 7fa9ec84f700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa9e80b28e0 0x7fa9e805bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:30.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.078+0000 7fa9ec84f700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa9e80b28e0 0x7fa9e805bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50396/0 (socket says 172.21.15.60:50396) 2022-06-29T16:05:30.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.078+0000 7fa9ec84f700 1 -- 172.21.15.60:0/3874631106 learned_addr learned my addr 172.21.15.60:0/3874631106 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:30.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.078+0000 7fa9ed050700 1 --1- 172.21.15.60:0/3874631106 >> v1:172.21.15.60:6789/0 conn(0x7fa9e80b9be0 0x7fa9e80b21d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58480/0 (socket says 172.21.15.60:58480) 2022-06-29T16:05:30.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.078+0000 7fa9ec84f700 1 -- 172.21.15.60:0/3874631106 >> v1:172.21.15.92:6789/0 conn(0x7fa9e80afee0 legacy=0x7fa9e8141e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:30.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.078+0000 7fa9ec84f700 1 -- 172.21.15.60:0/3874631106 >> v1:172.21.15.60:6789/0 conn(0x7fa9e80b9be0 legacy=0x7fa9e80b21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:30.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.078+0000 7fa9ec84f700 1 -- 172.21.15.60:0/3874631106 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e80b3ee0 con 0x7fa9e80b28e0 2022-06-29T16:05:30.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.078+0000 7fa9d3fff700 1 -- 172.21.15.60:0/3874631106 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 599608713 0 0) 0x7fa9e80b1d70 con 0x7fa9e80b9be0 2022-06-29T16:05:30.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.078+0000 7fa9ec84f700 1 --2- 172.21.15.60:0/3874631106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9e80b28e0 0x7fa9e805bd20 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 rx=0x7fa9e0001490 tx=0x7fa9dc006eb0).ready entity=mon.0 client_cookie=35f0f10ae2ff08cf server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:30.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.078+0000 7fa9d3fff700 1 -- 172.21.15.60:0/3874631106 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9dc00d610 con 0x7fa9e80b28e0 2022-06-29T16:05:30.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.078+0000 7fa9d3fff700 1 -- 172.21.15.60:0/3874631106 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9dc000e90 con 0x7fa9e80b28e0 2022-06-29T16:05:30.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.078+0000 7fa9d3fff700 1 -- 172.21.15.60:0/3874631106 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9dc00d960 con 0x7fa9e80b28e0 2022-06-29T16:05:30.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.082+0000 7fa9ee052700 1 -- 172.21.15.60:0/3874631106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9e80b28e0 msgr2=0x7fa9e805bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:30.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.082+0000 7fa9ee052700 1 --2- 172.21.15.60:0/3874631106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9e80b28e0 0x7fa9e805bd20 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 rx=0x7fa9e0001490 tx=0x7fa9dc006eb0).stop 2022-06-29T16:05:30.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.082+0000 7fa9ee052700 1 -- 172.21.15.60:0/3874631106 shutdown_connections 2022-06-29T16:05:30.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.082+0000 7fa9ee052700 1 --2- 172.21.15.60:0/3874631106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9e80b28e0 0x7fa9e805bd20 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:30.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.082+0000 7fa9ee052700 1 -- 172.21.15.60:0/3874631106 >> 172.21.15.60:0/3874631106 conn(0x7fa9e8062ce0 msgr2=0x7fa9e80b9190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:30.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.082+0000 7fa9ee052700 1 -- 172.21.15.60:0/3874631106 shutdown_connections 2022-06-29T16:05:30.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.082+0000 7fa9ee052700 1 -- 172.21.15.60:0/3874631106 wait complete. 2022-06-29T16:05:30.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.082+0000 7fa9ee052700 1 Processor -- start 2022-06-29T16:05:30.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.082+0000 7fa9ee052700 1 -- start start 2022-06-29T16:05:30.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.082+0000 7fa9ee052700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa9e80afee0 0x7fa9e80b83a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:30.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.082+0000 7fa9ee052700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa9e80b28e0 0x7fa9e81550a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:30.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.082+0000 7fa9ee052700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9e80b9be0 0x7fa9e80add20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:30.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.082+0000 7fa9ee052700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e8155590 con 0x7fa9e80b9be0 2022-06-29T16:05:30.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.082+0000 7fa9ee052700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e8147e30 con 0x7fa9e80b28e0 2022-06-29T16:05:30.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.082+0000 7fa9ee052700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e80ad8c0 con 0x7fa9e80afee0 2022-06-29T16:05:30.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.086+0000 7fa9ed851700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9e80b9be0 0x7fa9e80add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:30.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.086+0000 7fa9ec84f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa9e80b28e0 0x7fa9e81550a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:30.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.086+0000 7fa9ed050700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa9e80afee0 0x7fa9e80b83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:30.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.086+0000 7fa9ed851700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9e80b9be0 0x7fa9e80add20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50404/0 (socket says 172.21.15.60:50404) 2022-06-29T16:05:30.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.086+0000 7fa9ed851700 1 -- 172.21.15.60:0/458716803 learned_addr learned my addr 172.21.15.60:0/458716803 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:30.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.086+0000 7fa9ed851700 1 -- 172.21.15.60:0/458716803 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa9e80afee0 msgr2=0x7fa9e80b83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:30.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.086+0000 7fa9ed851700 1 --2- 172.21.15.60:0/458716803 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa9e80afee0 0x7fa9e80b83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:30.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.086+0000 7fa9ed851700 1 -- 172.21.15.60:0/458716803 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa9e80b28e0 msgr2=0x7fa9e81550a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:30.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.086+0000 7fa9ed851700 1 --2- 172.21.15.60:0/458716803 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa9e80b28e0 0x7fa9e81550a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:30.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.086+0000 7fa9ed851700 1 -- 172.21.15.60:0/458716803 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9dc00b1e0 con 0x7fa9e80b9be0 2022-06-29T16:05:30.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.086+0000 7fa9ed851700 1 --2- 172.21.15.60:0/458716803 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9e80b9be0 0x7fa9e80add20 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 rx=0x7fa9e0001360 tx=0x7fa9e0007e90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:30.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.086+0000 7fa9daffd700 1 -- 172.21.15.60:0/458716803 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9e000e4a0 con 0x7fa9e80b9be0 2022-06-29T16:05:30.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.086+0000 7fa9daffd700 1 -- 172.21.15.60:0/458716803 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e000e600 con 0x7fa9e80b9be0 2022-06-29T16:05:30.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.086+0000 7fa9ee052700 1 -- 172.21.15.60:0/458716803 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9e8140570 con 0x7fa9e80b9be0 2022-06-29T16:05:30.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.086+0000 7fa9daffd700 1 -- 172.21.15.60:0/458716803 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9e000e960 con 0x7fa9e80b9be0 2022-06-29T16:05:30.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.086+0000 7fa9ee052700 1 -- 172.21.15.60:0/458716803 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e8141380 con 0x7fa9e80b9be0 2022-06-29T16:05:30.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.086+0000 7fa9ee052700 1 -- 172.21.15.60:0/458716803 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e80573d0 con 0x7fa9e80b9be0 2022-06-29T16:05:30.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.090+0000 7fa9daffd700 1 -- 172.21.15.60:0/458716803 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa9e0010070 con 0x7fa9e80b9be0 2022-06-29T16:05:30.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.090+0000 7fa9daffd700 1 --2- 172.21.15.60:0/458716803 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa9c4053b00 0x7fa9c4055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:30.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.090+0000 7fa9daffd700 1 -- 172.21.15.60:0/458716803 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fa9e006b450 con 0x7fa9e80b9be0 2022-06-29T16:05:30.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.090+0000 7fa9ed050700 1 --2- 172.21.15.60:0/458716803 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa9c4053b00 0x7fa9c4055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:30.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.090+0000 7fa9ed050700 1 --2- 172.21.15.60:0/458716803 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa9c4053b00 0x7fa9c4055fc0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fa9e400d000 tx=0x7fa9e4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:30.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.098+0000 7fa9daffd700 1 -- 172.21.15.60:0/458716803 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa9e0042220 con 0x7fa9e80b9be0 2022-06-29T16:05:30.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.230+0000 7fa9ee052700 1 -- 172.21.15.60:0/458716803 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa9e805d990 con 0x7fa9e80b9be0 2022-06-29T16:05:30.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.990+0000 7fa9daffd700 1 -- 172.21.15.60:0/458716803 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v83) v1 ==== 185+0+0 (secure 0 0 0) 0x7fa9e0017020 con 0x7fa9e80b9be0 2022-06-29T16:05:31.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.994+0000 7fa9ee052700 1 -- 172.21.15.60:0/458716803 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa9c4053b00 msgr2=0x7fa9c4055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:31.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.994+0000 7fa9ee052700 1 --2- 172.21.15.60:0/458716803 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa9c4053b00 0x7fa9c4055fc0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fa9e400d000 tx=0x7fa9e4009040).stop 2022-06-29T16:05:31.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.994+0000 7fa9ee052700 1 -- 172.21.15.60:0/458716803 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9e80b9be0 msgr2=0x7fa9e80add20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:31.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.994+0000 7fa9ee052700 1 --2- 172.21.15.60:0/458716803 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9e80b9be0 0x7fa9e80add20 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 rx=0x7fa9e0001360 tx=0x7fa9e0007e90).stop 2022-06-29T16:05:31.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.994+0000 7fa9ee052700 1 -- 172.21.15.60:0/458716803 shutdown_connections 2022-06-29T16:05:31.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.994+0000 7fa9ee052700 1 --2- 172.21.15.60:0/458716803 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa9c4053b00 0x7fa9c4055fc0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:31.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.994+0000 7fa9ee052700 1 --2- 172.21.15.60:0/458716803 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa9e80b9be0 0x7fa9e80add20 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:31.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.994+0000 7fa9ee052700 1 --2- 172.21.15.60:0/458716803 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa9e80b28e0 0x7fa9e81550a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:31.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.994+0000 7fa9ee052700 1 --2- 172.21.15.60:0/458716803 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa9e80afee0 0x7fa9e80b83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:31.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.994+0000 7fa9ee052700 1 -- 172.21.15.60:0/458716803 >> 172.21.15.60:0/458716803 conn(0x7fa9e8062ce0 msgr2=0x7fa9e80b67b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:31.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.994+0000 7fa9ee052700 1 -- 172.21.15.60:0/458716803 shutdown_connections 2022-06-29T16:05:31.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:30.994+0000 7fa9ee052700 1 -- 172.21.15.60:0/458716803 wait complete. 2022-06-29T16:05:31.007 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2022-06-29T16:05:31.018 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-06-29T16:05:31.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.130+0000 7fd75a1f1700 1 Processor -- start 2022-06-29T16:05:31.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.130+0000 7fd75a1f1700 1 -- start start 2022-06-29T16:05:31.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.130+0000 7fd75a1f1700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd7540b9be0 0x7fd75405bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:31.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.130+0000 7fd75a1f1700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7540b1d70 con 0x7fd7540b28e0 2022-06-29T16:05:31.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.130+0000 7fd75a1f1700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7540b9fc0 con 0x7fd7540afee0 2022-06-29T16:05:31.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.130+0000 7fd75a1f1700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7540b3c40 con 0x7fd7540b9be0 2022-06-29T16:05:31.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd7599f0700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd7540b28e0 0x7fd754141e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58498/0 (socket says 172.21.15.60:58498) 2022-06-29T16:05:31.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd7599f0700 1 -- 172.21.15.60:0/3693205029 learned_addr learned my addr 172.21.15.60:0/3693205029 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:31.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd7591ef700 1 --2- 172.21.15.60:0/3693205029 >> v2:172.21.15.92:3300/0 conn(0x7fd7540b9be0 0x7fd75405bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:31.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd73ffff700 1 -- 172.21.15.60:0/3693205029 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 851890573 0 0) 0x7fd7540b1d70 con 0x7fd7540b28e0 2022-06-29T16:05:31.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd73ffff700 1 -- 172.21.15.60:0/3693205029 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd738003260 con 0x7fd7540b28e0 2022-06-29T16:05:31.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd73ffff700 1 -- 172.21.15.60:0/3693205029 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4155011264 0 0) 0x7fd7540b9fc0 con 0x7fd7540afee0 2022-06-29T16:05:31.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd73ffff700 1 -- 172.21.15.60:0/3693205029 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7540b1d70 con 0x7fd7540afee0 2022-06-29T16:05:31.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd73ffff700 1 -- 172.21.15.60:0/3693205029 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1688183683 0 0) 0x7fd738003260 con 0x7fd7540b28e0 2022-06-29T16:05:31.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd73ffff700 1 -- 172.21.15.60:0/3693205029 >> v2:172.21.15.92:3300/0 conn(0x7fd7540b9be0 msgr2=0x7fd75405bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:31.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd73ffff700 1 --2- 172.21.15.60:0/3693205029 >> v2:172.21.15.92:3300/0 conn(0x7fd7540b9be0 0x7fd75405bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:31.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd73ffff700 1 -- 172.21.15.60:0/3693205029 >> v1:172.21.15.92:6789/0 conn(0x7fd7540afee0 legacy=0x7fd7540b21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:31.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd73ffff700 1 -- 172.21.15.60:0/3693205029 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7540b3ee0 con 0x7fd7540b28e0 2022-06-29T16:05:31.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd73ffff700 1 -- 172.21.15.60:0/3693205029 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7fd74c0019e0 con 0x7fd7540b28e0 2022-06-29T16:05:31.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd73ffff700 1 -- 172.21.15.60:0/3693205029 >> v1:172.21.15.60:6789/0 conn(0x7fd7540b28e0 legacy=0x7fd754141e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:31.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd73ffff700 1 --2- 172.21.15.60:0/3693205029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7380038e0 0x7fd738003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:31.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd73ffff700 1 --2- 172.21.15.60:0/3693205029 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd738004500 0x7fd738004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:31.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd73ffff700 1 --2- 172.21.15.60:0/3693205029 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd738004e70 0x7fd73800b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:31.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd73ffff700 1 -- 172.21.15.60:0/3693205029 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7540b3c40 con 0x7fd7380038e0 2022-06-29T16:05:31.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd73ffff700 1 -- 172.21.15.60:0/3693205029 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd73800bdd0 con 0x7fd738004e70 2022-06-29T16:05:31.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd73ffff700 1 -- 172.21.15.60:0/3693205029 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd73800bf40 con 0x7fd738004500 2022-06-29T16:05:31.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd7591ef700 1 --2- 172.21.15.60:0/3693205029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7380038e0 0x7fd738003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:31.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd7589ee700 1 --2- 172.21.15.60:0/3693205029 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd738004500 0x7fd738004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:31.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd7591ef700 1 -- 172.21.15.60:0/3693205029 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd738004500 msgr2=0x7fd738004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:31.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd7591ef700 1 --2- 172.21.15.60:0/3693205029 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd738004500 0x7fd738004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:31.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd7591ef700 1 -- 172.21.15.60:0/3693205029 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd738004e70 msgr2=0x7fd73800b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:31.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd7591ef700 1 --2- 172.21.15.60:0/3693205029 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd738004e70 0x7fd73800b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-06-29T16:05:31.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd7591ef700 1 -- 172.21.15.60:0/3693205029 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd738003260 con 0x7fd7380038e0 2022-06-29T16:05:31.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd7599f0700 1 -- 172.21.15.60:0/3693205029 reap_dead start 2022-06-29T16:05:31.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd7589ee700 1 --2- 172.21.15.60:0/3693205029 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd738004500 0x7fd738004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:05:31.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd7591ef700 1 --2- 172.21.15.60:0/3693205029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7380038e0 0x7fd738003ce0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 rx=0x7fd75000c000 tx=0x7fd750006d90).ready entity=mon.0 client_cookie=571d6ca43d1af502 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:31.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd73ffff700 1 -- 172.21.15.60:0/3693205029 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd750013450 con 0x7fd7380038e0 2022-06-29T16:05:31.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd73ffff700 1 -- 172.21.15.60:0/3693205029 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7500135b0 con 0x7fd7380038e0 2022-06-29T16:05:31.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd73ffff700 1 -- 172.21.15.60:0/3693205029 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd750013910 con 0x7fd7380038e0 2022-06-29T16:05:31.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd75a1f1700 1 -- 172.21.15.60:0/3693205029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7380038e0 msgr2=0x7fd738003ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:31.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd75a1f1700 1 --2- 172.21.15.60:0/3693205029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7380038e0 0x7fd738003ce0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 rx=0x7fd75000c000 tx=0x7fd750006d90).stop 2022-06-29T16:05:31.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd75a1f1700 1 -- 172.21.15.60:0/3693205029 shutdown_connections 2022-06-29T16:05:31.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd75a1f1700 1 --2- 172.21.15.60:0/3693205029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd7380038e0 0x7fd738003ce0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:31.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd75a1f1700 1 -- 172.21.15.60:0/3693205029 >> 172.21.15.60:0/3693205029 conn(0x7fd754062ce0 msgr2=0x7fd7540b9190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:31.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd75a1f1700 1 -- 172.21.15.60:0/3693205029 shutdown_connections 2022-06-29T16:05:31.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.134+0000 7fd75a1f1700 1 -- 172.21.15.60:0/3693205029 wait complete. 2022-06-29T16:05:31.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.138+0000 7fd75a1f1700 1 Processor -- start 2022-06-29T16:05:31.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.138+0000 7fd75a1f1700 1 -- start start 2022-06-29T16:05:31.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.138+0000 7fd75a1f1700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd7380038e0 0x7fd754147620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:31.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.138+0000 7fd75a1f1700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd754155410 0x7fd754145ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:31.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.138+0000 7fd75a1f1700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd75413dec0 0x7fd75413e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:31.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.138+0000 7fd75a1f1700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7541464e0 con 0x7fd754155410 2022-06-29T16:05:31.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.138+0000 7fd75a1f1700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd754146700 con 0x7fd7380038e0 2022-06-29T16:05:31.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.138+0000 7fd75a1f1700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7541469a0 con 0x7fd75413dec0 2022-06-29T16:05:31.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.138+0000 7fd7589ee700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd754155410 0x7fd754145ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:31.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.138+0000 7fd7589ee700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd754155410 0x7fd754145ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50420/0 (socket says 172.21.15.60:50420) 2022-06-29T16:05:31.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.138+0000 7fd7589ee700 1 -- 172.21.15.60:0/2869532296 learned_addr learned my addr 172.21.15.60:0/2869532296 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:31.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.138+0000 7fd7591ef700 1 --2- 172.21.15.60:0/2869532296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd7380038e0 0x7fd754147620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:31.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.138+0000 7fd7599f0700 1 --2- 172.21.15.60:0/2869532296 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd75413dec0 0x7fd75413e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:31.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.138+0000 7fd7589ee700 1 -- 172.21.15.60:0/2869532296 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd75413dec0 msgr2=0x7fd75413e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:31.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.138+0000 7fd7589ee700 1 --2- 172.21.15.60:0/2869532296 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd75413dec0 0x7fd75413e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:31.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.138+0000 7fd7589ee700 1 -- 172.21.15.60:0/2869532296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd7380038e0 msgr2=0x7fd754147620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:31.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.138+0000 7fd7589ee700 1 --2- 172.21.15.60:0/2869532296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd7380038e0 0x7fd754147620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:31.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.138+0000 7fd7589ee700 1 -- 172.21.15.60:0/2869532296 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd754146c40 con 0x7fd754155410 2022-06-29T16:05:31.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.138+0000 7fd7589ee700 1 --2- 172.21.15.60:0/2869532296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd754155410 0x7fd754145ff0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 rx=0x7fd748008000 tx=0x7fd748009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:31.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.138+0000 7fd746ffd700 1 -- 172.21.15.60:0/2869532296 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd748013470 con 0x7fd754155410 2022-06-29T16:05:31.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.138+0000 7fd746ffd700 1 -- 172.21.15.60:0/2869532296 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7480135d0 con 0x7fd754155410 2022-06-29T16:05:31.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.138+0000 7fd75a1f1700 1 -- 172.21.15.60:0/2869532296 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd75413e830 con 0x7fd754155410 2022-06-29T16:05:31.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.138+0000 7fd746ffd700 1 -- 172.21.15.60:0/2869532296 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd748013930 con 0x7fd754155410 2022-06-29T16:05:31.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.138+0000 7fd75a1f1700 1 -- 172.21.15.60:0/2869532296 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd75413fb20 con 0x7fd754155410 2022-06-29T16:05:31.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.142+0000 7fd75a1f1700 1 -- 172.21.15.60:0/2869532296 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd754146dd0 con 0x7fd754155410 2022-06-29T16:05:31.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.142+0000 7fd746ffd700 1 -- 172.21.15.60:0/2869532296 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd748013a90 con 0x7fd754155410 2022-06-29T16:05:31.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.142+0000 7fd746ffd700 1 --2- 172.21.15.60:0/2869532296 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd730053aa0 0x7fd730055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:31.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.142+0000 7fd746ffd700 1 -- 172.21.15.60:0/2869532296 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5807+0+0 (secure 0 0 0) 0x7fd74806b750 con 0x7fd754155410 2022-06-29T16:05:31.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.142+0000 7fd7591ef700 1 --2- 172.21.15.60:0/2869532296 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd730053aa0 0x7fd730055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:31.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.142+0000 7fd7591ef700 1 --2- 172.21.15.60:0/2869532296 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd730053aa0 0x7fd730055f60 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fd75001a000 tx=0x7fd750018000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:31.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.150+0000 7fd746ffd700 1 -- 172.21.15.60:0/2869532296 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd748042780 con 0x7fd754155410 2022-06-29T16:05:31.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.274+0000 7fd75a1f1700 1 -- 172.21.15.60:0/2869532296 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd75405cf60 con 0x7fd754155410 2022-06-29T16:05:32.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:31.998+0000 7fd746ffd700 1 -- 172.21.15.60:0/2869532296 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v84) v1 ==== 173+0+0 (secure 0 0 0) 0x7fd748045da0 con 0x7fd754155410 2022-06-29T16:05:32.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.002+0000 7fd75a1f1700 1 -- 172.21.15.60:0/2869532296 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd730053aa0 msgr2=0x7fd730055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:32.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.002+0000 7fd75a1f1700 1 --2- 172.21.15.60:0/2869532296 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd730053aa0 0x7fd730055f60 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fd75001a000 tx=0x7fd750018000).stop 2022-06-29T16:05:32.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.002+0000 7fd75a1f1700 1 -- 172.21.15.60:0/2869532296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd754155410 msgr2=0x7fd754145ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:32.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.002+0000 7fd75a1f1700 1 --2- 172.21.15.60:0/2869532296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd754155410 0x7fd754145ff0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 rx=0x7fd748008000 tx=0x7fd748009040).stop 2022-06-29T16:05:32.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.002+0000 7fd75a1f1700 1 -- 172.21.15.60:0/2869532296 shutdown_connections 2022-06-29T16:05:32.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.002+0000 7fd75a1f1700 1 --2- 172.21.15.60:0/2869532296 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd730053aa0 0x7fd730055f60 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:32.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.002+0000 7fd75a1f1700 1 --2- 172.21.15.60:0/2869532296 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd75413dec0 0x7fd75413e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:32.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.002+0000 7fd75a1f1700 1 --2- 172.21.15.60:0/2869532296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd754155410 0x7fd754145ff0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:32.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.002+0000 7fd75a1f1700 1 --2- 172.21.15.60:0/2869532296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd7380038e0 0x7fd754147620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:32.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.002+0000 7fd75a1f1700 1 -- 172.21.15.60:0/2869532296 >> 172.21.15.60:0/2869532296 conn(0x7fd754062ce0 msgr2=0x7fd7540b4450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:32.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.002+0000 7fd75a1f1700 1 -- 172.21.15.60:0/2869532296 shutdown_connections 2022-06-29T16:05:32.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.002+0000 7fd75a1f1700 1 -- 172.21.15.60:0/2869532296 wait complete. 2022-06-29T16:05:32.014 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2022-06-29T16:05:32.027 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-06-29T16:05:32.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.134+0000 7f144196a700 1 Processor -- start 2022-06-29T16:05:32.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.134+0000 7f144196a700 1 -- start start 2022-06-29T16:05:32.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.134+0000 7f144196a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f143c05c990 0x7f143c0b0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:32.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.134+0000 7f144196a700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f143c05f620 0x7f143c146990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:32.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.134+0000 7f144196a700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f143c05fad0 con 0x7f143c05c5b0 2022-06-29T16:05:32.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.134+0000 7f144196a700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f143c146e80 con 0x7f143c05f620 2022-06-29T16:05:32.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.134+0000 7f144196a700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f143c145fb0 con 0x7f143c05c990 2022-06-29T16:05:32.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.134+0000 7f1440968700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f143c05c5b0 0x7f143c13c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58514/0 (socket says 172.21.15.60:58514) 2022-06-29T16:05:32.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.134+0000 7f1440968700 1 -- 172.21.15.60:0/767172166 learned_addr learned my addr 172.21.15.60:0/767172166 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:32.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.134+0000 7f1433fff700 1 --2- 172.21.15.60:0/767172166 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f143c05c990 0x7f143c0b0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:32.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.134+0000 7f1441169700 1 --2- 172.21.15.60:0/767172166 >> v2:172.21.15.92:3300/0 conn(0x7f143c05f620 0x7f143c146990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:32.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.138+0000 7f14337fe700 1 -- 172.21.15.60:0/767172166 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3393639207 0 0) 0x7f143c05fad0 con 0x7f143c05c5b0 2022-06-29T16:05:32.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.138+0000 7f14337fe700 1 -- 172.21.15.60:0/767172166 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1428003260 con 0x7f143c05c5b0 2022-06-29T16:05:32.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.138+0000 7f1433fff700 1 -- 172.21.15.60:0/767172166 >> v2:172.21.15.92:3300/0 conn(0x7f143c05f620 msgr2=0x7f143c146990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:32.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.138+0000 7f1433fff700 1 --2- 172.21.15.60:0/767172166 >> v2:172.21.15.92:3300/0 conn(0x7f143c05f620 0x7f143c146990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:32.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.138+0000 7f1433fff700 1 -- 172.21.15.60:0/767172166 >> v1:172.21.15.60:6789/0 conn(0x7f143c05c5b0 legacy=0x7f143c13c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:32.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.138+0000 7f1433fff700 1 -- 172.21.15.60:0/767172166 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f143c146250 con 0x7f143c05c990 2022-06-29T16:05:32.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.138+0000 7f1433fff700 1 --2- 172.21.15.60:0/767172166 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f143c05c990 0x7f143c0b0c80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f143c0b18b0 tx=0x7f142c006df0).ready entity=mon.2 client_cookie=607d5b2d89e64edf server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:32.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.138+0000 7f14337fe700 1 -- 172.21.15.60:0/767172166 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f142c00d570 con 0x7f143c05c990 2022-06-29T16:05:32.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.138+0000 7f14337fe700 1 -- 172.21.15.60:0/767172166 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f142c000e90 con 0x7f143c05c990 2022-06-29T16:05:32.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.138+0000 7f14337fe700 1 -- 172.21.15.60:0/767172166 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f142c00d8c0 con 0x7f143c05c990 2022-06-29T16:05:32.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.138+0000 7f144196a700 1 -- 172.21.15.60:0/767172166 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f143c05c990 msgr2=0x7f143c0b0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:32.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.138+0000 7f144196a700 1 --2- 172.21.15.60:0/767172166 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f143c05c990 0x7f143c0b0c80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f143c0b18b0 tx=0x7f142c006df0).stop 2022-06-29T16:05:32.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.138+0000 7f144196a700 1 -- 172.21.15.60:0/767172166 shutdown_connections 2022-06-29T16:05:32.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.138+0000 7f144196a700 1 --2- 172.21.15.60:0/767172166 >> v2:172.21.15.92:3300/0 conn(0x7f143c05f620 0x7f143c146990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:32.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.138+0000 7f144196a700 1 --2- 172.21.15.60:0/767172166 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f143c05c990 0x7f143c0b0c80 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:32.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.138+0000 7f144196a700 1 -- 172.21.15.60:0/767172166 >> 172.21.15.60:0/767172166 conn(0x7f143c0637c0 msgr2=0x7f143c063ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:32.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.138+0000 7f144196a700 1 -- 172.21.15.60:0/767172166 shutdown_connections 2022-06-29T16:05:32.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.138+0000 7f144196a700 1 -- 172.21.15.60:0/767172166 wait complete. 2022-06-29T16:05:32.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.138+0000 7f144196a700 1 Processor -- start 2022-06-29T16:05:32.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.138+0000 7f144196a700 1 -- start start 2022-06-29T16:05:32.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.142+0000 7f144196a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f143c05c5b0 0x7f143c0af7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:32.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.142+0000 7f144196a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f143c05c990 0x7f143c151320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:32.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.142+0000 7f144196a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f143c05f620 0x7f143c140e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:32.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.142+0000 7f144196a700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f143c152f00 con 0x7f143c05f620 2022-06-29T16:05:32.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.142+0000 7f144196a700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f143c13d5c0 con 0x7f143c05c990 2022-06-29T16:05:32.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.142+0000 7f144196a700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f143c143b50 con 0x7f143c05c5b0 2022-06-29T16:05:32.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.142+0000 7f1441169700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f143c05f620 0x7f143c140e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:32.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.142+0000 7f1441169700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f143c05f620 0x7f143c140e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50438/0 (socket says 172.21.15.60:50438) 2022-06-29T16:05:32.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.142+0000 7f1441169700 1 -- 172.21.15.60:0/1914406319 learned_addr learned my addr 172.21.15.60:0/1914406319 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:32.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.142+0000 7f1440968700 1 --2- 172.21.15.60:0/1914406319 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f143c05c5b0 0x7f143c0af7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:32.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.142+0000 7f1433fff700 1 --2- 172.21.15.60:0/1914406319 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f143c05c990 0x7f143c151320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:32.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.142+0000 7f1441169700 1 -- 172.21.15.60:0/1914406319 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f143c05c5b0 msgr2=0x7f143c0af7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:32.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.142+0000 7f1441169700 1 --2- 172.21.15.60:0/1914406319 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f143c05c5b0 0x7f143c0af7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:32.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.142+0000 7f1441169700 1 -- 172.21.15.60:0/1914406319 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f143c05c990 msgr2=0x7f143c151320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:32.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.142+0000 7f1441169700 1 --2- 172.21.15.60:0/1914406319 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f143c05c990 0x7f143c151320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:32.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.142+0000 7f1441169700 1 -- 172.21.15.60:0/1914406319 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f142c00b180 con 0x7f143c05f620 2022-06-29T16:05:32.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.142+0000 7f1441169700 1 --2- 172.21.15.60:0/1914406319 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f143c05f620 0x7f143c140e90 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 rx=0x7f1434006e00 tx=0x7f14340091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:32.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.142+0000 7f1431ffb700 1 -- 172.21.15.60:0/1914406319 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f143400c7f0 con 0x7f143c05f620 2022-06-29T16:05:32.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.142+0000 7f1431ffb700 1 -- 172.21.15.60:0/1914406319 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f143400c950 con 0x7f143c05f620 2022-06-29T16:05:32.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.142+0000 7f144196a700 1 -- 172.21.15.60:0/1914406319 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f143c1529a0 con 0x7f143c05f620 2022-06-29T16:05:32.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.142+0000 7f1431ffb700 1 -- 172.21.15.60:0/1914406319 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f143400ccb0 con 0x7f143c05f620 2022-06-29T16:05:32.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.142+0000 7f144196a700 1 -- 172.21.15.60:0/1914406319 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f143c1526c0 con 0x7f143c05f620 2022-06-29T16:05:32.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.142+0000 7f1431ffb700 1 -- 172.21.15.60:0/1914406319 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1434022070 con 0x7f143c05f620 2022-06-29T16:05:32.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.142+0000 7f144196a700 1 -- 172.21.15.60:0/1914406319 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f143c0573d0 con 0x7f143c05f620 2022-06-29T16:05:32.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.146+0000 7f1431ffb700 1 --2- 172.21.15.60:0/1914406319 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f141c053aa0 0x7f141c055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:32.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.146+0000 7f1431ffb700 1 -- 172.21.15.60:0/1914406319 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5425+0+0 (secure 0 0 0) 0x7f143406c2c0 con 0x7f143c05f620 2022-06-29T16:05:32.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.146+0000 7f1440968700 1 --2- 172.21.15.60:0/1914406319 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f141c053aa0 0x7f141c055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:32.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.146+0000 7f1440968700 1 --2- 172.21.15.60:0/1914406319 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f141c053aa0 0x7f141c055f60 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f143800b000 tx=0x7f1438009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:32.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.150+0000 7f1431ffb700 1 -- 172.21.15.60:0/1914406319 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1434043460 con 0x7f143c05f620 2022-06-29T16:05:32.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.282+0000 7f144196a700 1 -- 172.21.15.60:0/1914406319 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f143c067d20 con 0x7f143c05f620 2022-06-29T16:05:32.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.290+0000 7f1431ffb700 1 -- 172.21.15.60:0/1914406319 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1434046a80 con 0x7f143c05f620 2022-06-29T16:05:32.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.290+0000 7f144196a700 1 -- 172.21.15.60:0/1914406319 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f141c053aa0 msgr2=0x7f141c055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:32.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.290+0000 7f144196a700 1 --2- 172.21.15.60:0/1914406319 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f141c053aa0 0x7f141c055f60 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f143800b000 tx=0x7f1438009040).stop 2022-06-29T16:05:32.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.290+0000 7f144196a700 1 -- 172.21.15.60:0/1914406319 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f143c05f620 msgr2=0x7f143c140e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:32.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.290+0000 7f144196a700 1 --2- 172.21.15.60:0/1914406319 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f143c05f620 0x7f143c140e90 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 rx=0x7f1434006e00 tx=0x7f14340091d0).stop 2022-06-29T16:05:32.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.290+0000 7f144196a700 1 -- 172.21.15.60:0/1914406319 shutdown_connections 2022-06-29T16:05:32.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.290+0000 7f144196a700 1 --2- 172.21.15.60:0/1914406319 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f141c053aa0 0x7f141c055f60 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:32.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.290+0000 7f144196a700 1 --2- 172.21.15.60:0/1914406319 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f143c05f620 0x7f143c140e90 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:32.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.290+0000 7f144196a700 1 --2- 172.21.15.60:0/1914406319 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f143c05c990 0x7f143c151320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:32.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.290+0000 7f144196a700 1 --2- 172.21.15.60:0/1914406319 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f143c05c5b0 0x7f143c0af7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:32.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.290+0000 7f144196a700 1 -- 172.21.15.60:0/1914406319 >> 172.21.15.60:0/1914406319 conn(0x7f143c0637c0 msgr2=0x7f143c145760 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:32.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.290+0000 7f144196a700 1 -- 172.21.15.60:0/1914406319 shutdown_connections 2022-06-29T16:05:32.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.290+0000 7f144196a700 1 -- 172.21.15.60:0/1914406319 wait complete. 2022-06-29T16:05:32.313 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_dump_inode' 2022-06-29T16:05:32.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.410+0000 7fef24349700 1 Processor -- start 2022-06-29T16:05:32.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.410+0000 7fef24349700 1 -- start start 2022-06-29T16:05:32.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.410+0000 7fef24349700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef1c063ad0 0x7fef1c067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:32.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.410+0000 7fef24349700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fef1c14af10 0x7fef1c05c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:32.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.410+0000 7fef24349700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef1c0acba0 con 0x7fef1c0b5950 2022-06-29T16:05:32.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.410+0000 7fef24349700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef1c14b3a0 con 0x7fef1c14af10 2022-06-29T16:05:32.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.410+0000 7fef24349700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef1c067220 con 0x7fef1c063ad0 2022-06-29T16:05:32.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.410+0000 7fef220e5700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fef1c0b5950 0x7fef1c05bd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:34544/0 (socket says 172.21.15.60:34544) 2022-06-29T16:05:32.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.410+0000 7fef228e6700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fef1c14af10 0x7fef1c05c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:32.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.410+0000 7fef220e5700 1 -- 172.21.15.60:0/1007412240 learned_addr learned my addr 172.21.15.60:0/1007412240 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:32.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.410+0000 7fef218e4700 1 --2- 172.21.15.60:0/1007412240 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef1c063ad0 0x7fef1c067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:32.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.410+0000 7fef228e6700 1 -- 172.21.15.60:0/1007412240 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef1c063ad0 msgr2=0x7fef1c067680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:32.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.410+0000 7fef228e6700 1 --2- 172.21.15.60:0/1007412240 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef1c063ad0 0x7fef1c067680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:32.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.410+0000 7fef228e6700 1 -- 172.21.15.60:0/1007412240 >> v1:172.21.15.92:6789/0 conn(0x7fef1c0b5950 legacy=0x7fef1c05bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:32.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.410+0000 7fef228e6700 1 -- 172.21.15.60:0/1007412240 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef1c0b5d30 con 0x7fef1c14af10 2022-06-29T16:05:32.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.410+0000 7fef210e3700 1 -- 172.21.15.60:0/1007412240 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3809236384 0 0) 0x7fef1c0acba0 con 0x7fef1c0b5950 2022-06-29T16:05:32.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.410+0000 7fef228e6700 1 --2- 172.21.15.60:0/1007412240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef1c14af10 0x7fef1c05c430 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fef0c002f70 tx=0x7fef0c006df0).ready entity=mon.1 client_cookie=ab2027856b8779d0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:32.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.414+0000 7fef210e3700 1 -- 172.21.15.60:0/1007412240 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef0c00d570 con 0x7fef1c14af10 2022-06-29T16:05:32.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.414+0000 7fef210e3700 1 -- 172.21.15.60:0/1007412240 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef0c000e90 con 0x7fef1c14af10 2022-06-29T16:05:32.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.414+0000 7fef210e3700 1 -- 172.21.15.60:0/1007412240 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef0c00d8c0 con 0x7fef1c14af10 2022-06-29T16:05:32.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.414+0000 7fef24349700 1 -- 172.21.15.60:0/1007412240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef1c14af10 msgr2=0x7fef1c05c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:32.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.414+0000 7fef24349700 1 --2- 172.21.15.60:0/1007412240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef1c14af10 0x7fef1c05c430 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fef0c002f70 tx=0x7fef0c006df0).stop 2022-06-29T16:05:32.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.414+0000 7fef24349700 1 -- 172.21.15.60:0/1007412240 shutdown_connections 2022-06-29T16:05:32.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.414+0000 7fef24349700 1 --2- 172.21.15.60:0/1007412240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef1c14af10 0x7fef1c05c430 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:32.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.414+0000 7fef24349700 1 --2- 172.21.15.60:0/1007412240 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef1c063ad0 0x7fef1c067680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:32.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.414+0000 7fef24349700 1 -- 172.21.15.60:0/1007412240 >> 172.21.15.60:0/1007412240 conn(0x7fef1c0632c0 msgr2=0x7fef1c0b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:32.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.414+0000 7fef24349700 1 -- 172.21.15.60:0/1007412240 shutdown_connections 2022-06-29T16:05:32.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.414+0000 7fef24349700 1 -- 172.21.15.60:0/1007412240 wait complete. 2022-06-29T16:05:32.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.414+0000 7fef24349700 1 Processor -- start 2022-06-29T16:05:32.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.414+0000 7fef24349700 1 -- start start 2022-06-29T16:05:32.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.414+0000 7fef24349700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef1c063ad0 0x7fef1c159370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:32.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.414+0000 7fef24349700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef1c0b5950 0x7fef1c0b4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:32.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.414+0000 7fef24349700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef1c14af10 0x7fef1c0aec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:32.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.414+0000 7fef24349700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1c0b1370 con 0x7fef1c0b5950 2022-06-29T16:05:32.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.414+0000 7fef24349700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1c0b4a10 con 0x7fef1c063ad0 2022-06-29T16:05:32.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.414+0000 7fef24349700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef1c0ad790 con 0x7fef1c14af10 2022-06-29T16:05:32.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.418+0000 7fef218e4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef1c0b5950 0x7fef1c0b4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:32.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.418+0000 7fef218e4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef1c0b5950 0x7fef1c0b4360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50452/0 (socket says 172.21.15.60:50452) 2022-06-29T16:05:32.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.418+0000 7fef220e5700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef1c063ad0 0x7fef1c159370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:32.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.418+0000 7fef218e4700 1 -- 172.21.15.60:0/1309848271 learned_addr learned my addr 172.21.15.60:0/1309848271 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:32.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.418+0000 7fef228e6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef1c14af10 0x7fef1c0aec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:32.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.418+0000 7fef218e4700 1 -- 172.21.15.60:0/1309848271 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef1c14af10 msgr2=0x7fef1c0aec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:32.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.418+0000 7fef218e4700 1 --2- 172.21.15.60:0/1309848271 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef1c14af10 0x7fef1c0aec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:32.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.418+0000 7fef218e4700 1 -- 172.21.15.60:0/1309848271 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef1c063ad0 msgr2=0x7fef1c159370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:32.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.418+0000 7fef218e4700 1 --2- 172.21.15.60:0/1309848271 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef1c063ad0 0x7fef1c159370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:32.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.418+0000 7fef218e4700 1 -- 172.21.15.60:0/1309848271 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef0c00b180 con 0x7fef1c0b5950 2022-06-29T16:05:32.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.418+0000 7fef218e4700 1 --2- 172.21.15.60:0/1309848271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef1c0b5950 0x7fef1c0b4360 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 rx=0x7fef1000c000 tx=0x7fef10006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:32.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.418+0000 7fef1b7fe700 1 -- 172.21.15.60:0/1309848271 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef10010780 con 0x7fef1c0b5950 2022-06-29T16:05:32.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.418+0000 7fef1b7fe700 1 -- 172.21.15.60:0/1309848271 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef100108e0 con 0x7fef1c0b5950 2022-06-29T16:05:32.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.418+0000 7fef24349700 1 -- 172.21.15.60:0/1309848271 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef1c0adb90 con 0x7fef1c0b5950 2022-06-29T16:05:32.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.418+0000 7fef1b7fe700 1 -- 172.21.15.60:0/1309848271 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef10010c40 con 0x7fef1c0b5950 2022-06-29T16:05:32.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.418+0000 7fef24349700 1 -- 172.21.15.60:0/1309848271 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef1c0af120 con 0x7fef1c0b5950 2022-06-29T16:05:32.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.418+0000 7fef24349700 1 -- 172.21.15.60:0/1309848271 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef1c0ad9b0 con 0x7fef1c0b5950 2022-06-29T16:05:32.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.418+0000 7fef1b7fe700 1 -- 172.21.15.60:0/1309848271 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fef1001c460 con 0x7fef1c0b5950 2022-06-29T16:05:32.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.422+0000 7fef1b7fe700 1 --2- 172.21.15.60:0/1309848271 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7feefc053b00 0x7feefc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:32.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.422+0000 7fef1b7fe700 1 -- 172.21.15.60:0/1309848271 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5425+0+0 (secure 0 0 0) 0x7fef1006c050 con 0x7fef1c0b5950 2022-06-29T16:05:32.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.422+0000 7fef220e5700 1 --2- 172.21.15.60:0/1309848271 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7feefc053b00 0x7feefc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:32.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.422+0000 7fef220e5700 1 --2- 172.21.15.60:0/1309848271 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7feefc053b00 0x7feefc055fc0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fef1400d000 tx=0x7fef14001520).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:32.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.426+0000 7fef1b7fe700 1 -- 172.21.15.60:0/1309848271 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fef1c0ad9b0 con 0x7fef1c0b5950 2022-06-29T16:05:32.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:32.598+0000 7fef24349700 1 -- 172.21.15.60:0/1309848271 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_dump_inode"]} v 0) v1 -- 0x7fef1c0ad9b0 con 0x7fef1c0b5950 2022-06-29T16:05:33.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.002+0000 7fef1b7fe700 1 -- 172.21.15.60:0/1309848271 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_dump_inode"]}]=0 v313) v1 ==== 126+0+0 (secure 0 0 0) 0x7fef1c0ad9b0 con 0x7fef1c0b5950 2022-06-29T16:05:33.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.006+0000 7fef24349700 1 -- 172.21.15.60:0/1309848271 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7feefc053b00 msgr2=0x7feefc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:33.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.006+0000 7fef24349700 1 --2- 172.21.15.60:0/1309848271 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7feefc053b00 0x7feefc055fc0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fef1400d000 tx=0x7fef14001520).stop 2022-06-29T16:05:33.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.006+0000 7fef24349700 1 -- 172.21.15.60:0/1309848271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef1c0b5950 msgr2=0x7fef1c0b4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:33.016 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.006+0000 7fef24349700 1 --2- 172.21.15.60:0/1309848271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef1c0b5950 0x7fef1c0b4360 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 rx=0x7fef1000c000 tx=0x7fef10006df0).stop 2022-06-29T16:05:33.016 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.006+0000 7fef24349700 1 -- 172.21.15.60:0/1309848271 shutdown_connections 2022-06-29T16:05:33.016 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.006+0000 7fef24349700 1 --2- 172.21.15.60:0/1309848271 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7feefc053b00 0x7feefc055fc0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:33.016 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.006+0000 7fef24349700 1 --2- 172.21.15.60:0/1309848271 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef1c14af10 0x7fef1c0aec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:33.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.006+0000 7fef24349700 1 --2- 172.21.15.60:0/1309848271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef1c0b5950 0x7fef1c0b4360 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:33.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.006+0000 7fef24349700 1 --2- 172.21.15.60:0/1309848271 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef1c063ad0 0x7fef1c159370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:33.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.006+0000 7fef24349700 1 -- 172.21.15.60:0/1309848271 >> 172.21.15.60:0/1309848271 conn(0x7fef1c0632c0 msgr2=0x7fef1c143840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:33.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.006+0000 7fef24349700 1 -- 172.21.15.60:0/1309848271 shutdown_connections 2022-06-29T16:05:33.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.006+0000 7fef24349700 1 -- 172.21.15.60:0/1309848271 wait complete. 2022-06-29T16:05:33.034 INFO:tasks.cephfs_test_runner:test_dump_inode (tasks.cephfs.test_misc.TestMisc) ... ok 2022-06-29T16:05:33.034 INFO:tasks.cephfs_test_runner:Starting test: test_dump_inode_hexademical (tasks.cephfs.test_misc.TestMisc) 2022-06-29T16:05:33.035 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_dump_inode_hexademical' 2022-06-29T16:05:33.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.134+0000 7fa39f59e700 1 Processor -- start 2022-06-29T16:05:33.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.134+0000 7fa39f59e700 1 -- start start 2022-06-29T16:05:33.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.134+0000 7fa39f59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3a00b1c00 0x7fa3a00ae890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:33.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.134+0000 7fa39f59e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa3a00b0fa0 0x7fa3a0067250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:33.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.134+0000 7fa39f59e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3a00ad240 con 0x7fa3a005fb00 2022-06-29T16:05:33.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.134+0000 7fa39f59e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3a013ce60 con 0x7fa3a00b0fa0 2022-06-29T16:05:33.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.134+0000 7fa39f59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a005fee0 con 0x7fa3a00b1c00 2022-06-29T16:05:33.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.134+0000 7fa39ed9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa3a00b0fa0 0x7fa3a0067250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:33.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.134+0000 7fa395d9b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3a00b1c00 0x7fa3a00ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:33.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.134+0000 7fa39ed9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa3a00b0fa0 0x7fa3a0067250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50460/0 (socket says 172.21.15.60:50460) 2022-06-29T16:05:33.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.134+0000 7fa39ed9d700 1 -- 172.21.15.60:0/91949341 learned_addr learned my addr 172.21.15.60:0/91949341 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:33.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.134+0000 7fa39ed9d700 1 -- 172.21.15.60:0/91949341 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3a00b1c00 msgr2=0x7fa3a00ae890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:33.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.134+0000 7fa39ed9d700 1 --2- 172.21.15.60:0/91949341 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3a00b1c00 0x7fa3a00ae890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:33.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.134+0000 7fa39ed9d700 1 -- 172.21.15.60:0/91949341 >> v1:172.21.15.92:6789/0 conn(0x7fa3a005fb00 legacy=0x7fa3a00b14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:33.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.134+0000 7fa39dd9b700 1 -- 172.21.15.60:0/91949341 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3989346504 0 0) 0x7fa3a00ad240 con 0x7fa3a005fb00 2022-06-29T16:05:33.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.134+0000 7fa39ed9d700 1 -- 172.21.15.60:0/91949341 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a0143180 con 0x7fa3a00b0fa0 2022-06-29T16:05:33.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.134+0000 7fa39ed9d700 1 --2- 172.21.15.60:0/91949341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3a00b0fa0 0x7fa3a0067250 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 rx=0x7fa3980032f0 tx=0x7fa398009290).ready entity=mon.0 client_cookie=13079ba9e4db5250 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:33.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.134+0000 7fa39dd9b700 1 -- 172.21.15.60:0/91949341 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa39800b8a0 con 0x7fa3a00b0fa0 2022-06-29T16:05:33.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.134+0000 7fa39dd9b700 1 -- 172.21.15.60:0/91949341 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa39800ba00 con 0x7fa3a00b0fa0 2022-06-29T16:05:33.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.134+0000 7fa39dd9b700 1 -- 172.21.15.60:0/91949341 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa39800bb70 con 0x7fa3a00b0fa0 2022-06-29T16:05:33.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.134+0000 7fa39f59e700 1 -- 172.21.15.60:0/91949341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3a00b0fa0 msgr2=0x7fa3a0067250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:33.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.134+0000 7fa39f59e700 1 --2- 172.21.15.60:0/91949341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3a00b0fa0 0x7fa3a0067250 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 rx=0x7fa3980032f0 tx=0x7fa398009290).stop 2022-06-29T16:05:33.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.134+0000 7fa39f59e700 1 -- 172.21.15.60:0/91949341 shutdown_connections 2022-06-29T16:05:33.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.134+0000 7fa39f59e700 1 --2- 172.21.15.60:0/91949341 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3a00b0fa0 0x7fa3a0067250 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:33.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.134+0000 7fa39f59e700 1 --2- 172.21.15.60:0/91949341 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3a00b1c00 0x7fa3a00ae890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:33.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.134+0000 7fa39f59e700 1 -- 172.21.15.60:0/91949341 >> 172.21.15.60:0/91949341 conn(0x7fa3a0063a60 msgr2=0x7fa3a005f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:33.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.134+0000 7fa39f59e700 1 -- 172.21.15.60:0/91949341 shutdown_connections 2022-06-29T16:05:33.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.134+0000 7fa39f59e700 1 -- 172.21.15.60:0/91949341 wait complete. 2022-06-29T16:05:33.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.138+0000 7fa39f59e700 1 Processor -- start 2022-06-29T16:05:33.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.138+0000 7fa39f59e700 1 -- start start 2022-06-29T16:05:33.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.138+0000 7fa39f59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3a005fb00 0x7fa3a0142220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:33.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.138+0000 7fa39f59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3a00b0fa0 0x7fa3a00663b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:33.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.138+0000 7fa39f59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3a00b1c00 0x7fa3a0115550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:33.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.138+0000 7fa39f59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a0114e80 con 0x7fa3a00b1c00 2022-06-29T16:05:33.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.138+0000 7fa39f59e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a0112c30 con 0x7fa3a00b0fa0 2022-06-29T16:05:33.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.138+0000 7fa39f59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a01150b0 con 0x7fa3a005fb00 2022-06-29T16:05:33.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.138+0000 7fa39ed9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3a00b1c00 0x7fa3a0115550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:33.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.138+0000 7fa39ed9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3a00b1c00 0x7fa3a0115550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50466/0 (socket says 172.21.15.60:50466) 2022-06-29T16:05:33.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.138+0000 7fa39ed9d700 1 -- 172.21.15.60:0/1977221130 learned_addr learned my addr 172.21.15.60:0/1977221130 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:33.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.138+0000 7fa395d9b700 1 --2- 172.21.15.60:0/1977221130 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3a00b0fa0 0x7fa3a00663b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:33.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.138+0000 7fa39e59c700 1 --2- 172.21.15.60:0/1977221130 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3a005fb00 0x7fa3a0142220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:33.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.138+0000 7fa39ed9d700 1 -- 172.21.15.60:0/1977221130 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3a005fb00 msgr2=0x7fa3a0142220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:33.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.138+0000 7fa39ed9d700 1 --2- 172.21.15.60:0/1977221130 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3a005fb00 0x7fa3a0142220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:33.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.138+0000 7fa39ed9d700 1 -- 172.21.15.60:0/1977221130 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3a00b0fa0 msgr2=0x7fa3a00663b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:33.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.138+0000 7fa39ed9d700 1 --2- 172.21.15.60:0/1977221130 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3a00b0fa0 0x7fa3a00663b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:33.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.138+0000 7fa39ed9d700 1 -- 172.21.15.60:0/1977221130 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa398009530 con 0x7fa3a00b1c00 2022-06-29T16:05:33.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.138+0000 7fa39ed9d700 1 --2- 172.21.15.60:0/1977221130 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3a00b1c00 0x7fa3a0115550 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 rx=0x7fa3980032f0 tx=0x7fa398016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:33.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.138+0000 7fa397fff700 1 -- 172.21.15.60:0/1977221130 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa39800b8a0 con 0x7fa3a00b1c00 2022-06-29T16:05:33.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.138+0000 7fa397fff700 1 -- 172.21.15.60:0/1977221130 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa398021630 con 0x7fa3a00b1c00 2022-06-29T16:05:33.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.138+0000 7fa39f59e700 1 -- 172.21.15.60:0/1977221130 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3a0112ed0 con 0x7fa3a00b1c00 2022-06-29T16:05:33.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.142+0000 7fa397fff700 1 -- 172.21.15.60:0/1977221130 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa398021990 con 0x7fa3a00b1c00 2022-06-29T16:05:33.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.142+0000 7fa39f59e700 1 -- 172.21.15.60:0/1977221130 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3a0115be0 con 0x7fa3a00b1c00 2022-06-29T16:05:33.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.142+0000 7fa39f59e700 1 -- 172.21.15.60:0/1977221130 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3a0115270 con 0x7fa3a00b1c00 2022-06-29T16:05:33.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.142+0000 7fa397fff700 1 -- 172.21.15.60:0/1977221130 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa398027020 con 0x7fa3a00b1c00 2022-06-29T16:05:33.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.142+0000 7fa397fff700 1 --2- 172.21.15.60:0/1977221130 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa384053b00 0x7fa384055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:33.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.142+0000 7fa397fff700 1 -- 172.21.15.60:0/1977221130 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5425+0+0 (secure 0 0 0) 0x7fa39806d060 con 0x7fa3a00b1c00 2022-06-29T16:05:33.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.142+0000 7fa39e59c700 1 --2- 172.21.15.60:0/1977221130 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa384053b00 0x7fa384055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:33.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.142+0000 7fa39e59c700 1 --2- 172.21.15.60:0/1977221130 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa384053b00 0x7fa384055fc0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fa39000d000 tx=0x7fa390007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:33.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.150+0000 7fa397fff700 1 -- 172.21.15.60:0/1977221130 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa398044160 con 0x7fa3a00b1c00 2022-06-29T16:05:33.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:33.318+0000 7fa39f59e700 1 -- 172.21.15.60:0/1977221130 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_dump_inode_hexademical"]} v 0) v1 -- 0x7fa3a00573d0 con 0x7fa3a00b1c00 2022-06-29T16:05:34.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.010+0000 7fa397fff700 1 -- 172.21.15.60:0/1977221130 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_dump_inode_hexademical"]}]=0 v314) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa398015020 con 0x7fa3a00b1c00 2022-06-29T16:05:34.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.014+0000 7fa39f59e700 1 -- 172.21.15.60:0/1977221130 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa384053b00 msgr2=0x7fa384055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:34.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.014+0000 7fa39f59e700 1 --2- 172.21.15.60:0/1977221130 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa384053b00 0x7fa384055fc0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fa39000d000 tx=0x7fa390007a00).stop 2022-06-29T16:05:34.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.014+0000 7fa39f59e700 1 -- 172.21.15.60:0/1977221130 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3a00b1c00 msgr2=0x7fa3a0115550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:34.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.014+0000 7fa39f59e700 1 --2- 172.21.15.60:0/1977221130 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3a00b1c00 0x7fa3a0115550 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 rx=0x7fa3980032f0 tx=0x7fa398016000).stop 2022-06-29T16:05:34.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.014+0000 7fa39f59e700 1 -- 172.21.15.60:0/1977221130 shutdown_connections 2022-06-29T16:05:34.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.014+0000 7fa39f59e700 1 --2- 172.21.15.60:0/1977221130 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa384053b00 0x7fa384055fc0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:34.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.014+0000 7fa39f59e700 1 --2- 172.21.15.60:0/1977221130 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa3a00b1c00 0x7fa3a0115550 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:34.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.014+0000 7fa39f59e700 1 --2- 172.21.15.60:0/1977221130 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3a00b0fa0 0x7fa3a00663b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:34.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.014+0000 7fa39f59e700 1 --2- 172.21.15.60:0/1977221130 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3a005fb00 0x7fa3a0142220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:34.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.014+0000 7fa39f59e700 1 -- 172.21.15.60:0/1977221130 >> 172.21.15.60:0/1977221130 conn(0x7fa3a0063a60 msgr2=0x7fa3a00b1470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:34.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.014+0000 7fa39f59e700 1 -- 172.21.15.60:0/1977221130 shutdown_connections 2022-06-29T16:05:34.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.014+0000 7fa39f59e700 1 -- 172.21.15.60:0/1977221130 wait complete. 2022-06-29T16:05:34.040 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-06-29T16:05:34.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.150+0000 7f371b118700 1 Processor -- start 2022-06-29T16:05:34.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.150+0000 7f371b118700 1 -- start start 2022-06-29T16:05:34.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.154+0000 7f371b118700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f37140b3c70 0x7f3714065ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:34.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.154+0000 7f371b118700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f371405bf30 0x7f3714143cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:34.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.154+0000 7f371b118700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3714065820 con 0x7f37141441b0 2022-06-29T16:05:34.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.154+0000 7f371b118700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37140b4050 con 0x7f37140b3c70 2022-06-29T16:05:34.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.154+0000 7f371b118700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f371405c4e0 con 0x7f371405bf30 2022-06-29T16:05:34.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.154+0000 7f371a917700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f37141441b0 0x7f371413fa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58560/0 (socket says 172.21.15.60:58560) 2022-06-29T16:05:34.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.154+0000 7f371a917700 1 -- 172.21.15.60:0/2692457714 learned_addr learned my addr 172.21.15.60:0/2692457714 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:34.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.154+0000 7f3719915700 1 --2- 172.21.15.60:0/2692457714 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f371405bf30 0x7f3714143cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:34.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.154+0000 7f371a116700 1 --2- 172.21.15.60:0/2692457714 >> v2:172.21.15.92:3300/0 conn(0x7f37140b3c70 0x7f3714065ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:34.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.154+0000 7f3719114700 1 -- 172.21.15.60:0/2692457714 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3737103621 0 0) 0x7f3714065820 con 0x7f37141441b0 2022-06-29T16:05:34.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.154+0000 7f3719114700 1 -- 172.21.15.60:0/2692457714 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36fc003260 con 0x7f37141441b0 2022-06-29T16:05:34.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.154+0000 7f3719915700 1 -- 172.21.15.60:0/2692457714 >> v2:172.21.15.92:3300/0 conn(0x7f37140b3c70 msgr2=0x7f3714065ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:34.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.154+0000 7f3719915700 1 --2- 172.21.15.60:0/2692457714 >> v2:172.21.15.92:3300/0 conn(0x7f37140b3c70 0x7f3714065ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:34.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.154+0000 7f3719915700 1 -- 172.21.15.60:0/2692457714 >> v1:172.21.15.60:6789/0 conn(0x7f37141441b0 legacy=0x7f371413fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:34.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.154+0000 7f3719915700 1 -- 172.21.15.60:0/2692457714 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f371405c780 con 0x7f371405bf30 2022-06-29T16:05:34.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.154+0000 7f3719915700 1 --2- 172.21.15.60:0/2692457714 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f371405bf30 0x7f3714143cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f3710002f70 tx=0x7f3710006df0).ready entity=mon.2 client_cookie=a53b7d793469e6a7 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:34.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.154+0000 7f3719114700 1 -- 172.21.15.60:0/2692457714 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f371000d4a0 con 0x7f371405bf30 2022-06-29T16:05:34.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.154+0000 7f3719114700 1 -- 172.21.15.60:0/2692457714 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3710000e90 con 0x7f371405bf30 2022-06-29T16:05:34.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.154+0000 7f3719114700 1 -- 172.21.15.60:0/2692457714 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f371000d7f0 con 0x7f371405bf30 2022-06-29T16:05:34.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.154+0000 7f371b118700 1 -- 172.21.15.60:0/2692457714 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f371405bf30 msgr2=0x7f3714143cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:34.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.154+0000 7f371b118700 1 --2- 172.21.15.60:0/2692457714 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f371405bf30 0x7f3714143cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f3710002f70 tx=0x7f3710006df0).stop 2022-06-29T16:05:34.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.154+0000 7f371b118700 1 -- 172.21.15.60:0/2692457714 shutdown_connections 2022-06-29T16:05:34.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.154+0000 7f371b118700 1 --2- 172.21.15.60:0/2692457714 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f371405bf30 0x7f3714143cc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:34.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.154+0000 7f371b118700 1 --2- 172.21.15.60:0/2692457714 >> v2:172.21.15.92:3300/0 conn(0x7f37140b3c70 0x7f3714065ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:34.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.154+0000 7f371b118700 1 -- 172.21.15.60:0/2692457714 >> 172.21.15.60:0/2692457714 conn(0x7f3714064810 msgr2=0x7f3714064bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:34.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.154+0000 7f371b118700 1 -- 172.21.15.60:0/2692457714 shutdown_connections 2022-06-29T16:05:34.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.154+0000 7f371b118700 1 -- 172.21.15.60:0/2692457714 wait complete. 2022-06-29T16:05:34.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.158+0000 7f371b118700 1 Processor -- start 2022-06-29T16:05:34.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.158+0000 7f371b118700 1 -- start start 2022-06-29T16:05:34.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.158+0000 7f371b118700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f371405bf30 0x7f3714114570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:34.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.158+0000 7f371b118700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37140b3c70 0x7f3714113960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:34.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.158+0000 7f371b118700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37141441b0 0x7f37141516e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:34.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.158+0000 7f371b118700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3714113030 con 0x7f37140b3c70 2022-06-29T16:05:34.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.158+0000 7f371b118700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3714116350 con 0x7f37141441b0 2022-06-29T16:05:34.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.158+0000 7f371b118700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37141165f0 con 0x7f371405bf30 2022-06-29T16:05:34.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.158+0000 7f371a116700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f371405bf30 0x7f3714114570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:34.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.158+0000 7f3719915700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37140b3c70 0x7f3714113960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:34.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.158+0000 7f371a116700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f371405bf30 0x7f3714114570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:35338/0 (socket says 172.21.15.60:35338) 2022-06-29T16:05:34.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.158+0000 7f371a116700 1 -- 172.21.15.60:0/4090952312 learned_addr learned my addr 172.21.15.60:0/4090952312 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:34.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.158+0000 7f371a917700 1 --2- 172.21.15.60:0/4090952312 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37141441b0 0x7f37141516e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:34.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.158+0000 7f371a116700 1 -- 172.21.15.60:0/4090952312 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37141441b0 msgr2=0x7f37141516e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:34.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.158+0000 7f371a116700 1 --2- 172.21.15.60:0/4090952312 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37141441b0 0x7f37141516e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:34.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.158+0000 7f371a116700 1 -- 172.21.15.60:0/4090952312 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37140b3c70 msgr2=0x7f3714113960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:34.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.158+0000 7f371a116700 1 --2- 172.21.15.60:0/4090952312 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37140b3c70 0x7f3714113960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:34.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.158+0000 7f371a116700 1 -- 172.21.15.60:0/4090952312 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f371000b180 con 0x7f371405bf30 2022-06-29T16:05:34.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.158+0000 7f371a116700 1 --2- 172.21.15.60:0/4090952312 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f371405bf30 0x7f3714114570 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f370800c000 tx=0x7f3708006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:34.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.158+0000 7f37077fe700 1 -- 172.21.15.60:0/4090952312 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3708010780 con 0x7f371405bf30 2022-06-29T16:05:34.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.158+0000 7f37077fe700 1 -- 172.21.15.60:0/4090952312 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37080108e0 con 0x7f371405bf30 2022-06-29T16:05:34.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.158+0000 7f371b118700 1 -- 172.21.15.60:0/4090952312 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3714116890 con 0x7f371405bf30 2022-06-29T16:05:34.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.158+0000 7f37077fe700 1 -- 172.21.15.60:0/4090952312 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3708010c40 con 0x7f371405bf30 2022-06-29T16:05:34.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.162+0000 7f371b118700 1 -- 172.21.15.60:0/4090952312 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3714113ed0 con 0x7f371405bf30 2022-06-29T16:05:34.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.162+0000 7f371b118700 1 -- 172.21.15.60:0/4090952312 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37140573d0 con 0x7f371405bf30 2022-06-29T16:05:34.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.162+0000 7f37077fe700 1 -- 172.21.15.60:0/4090952312 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f370801c460 con 0x7f371405bf30 2022-06-29T16:05:34.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.162+0000 7f37077fe700 1 --2- 172.21.15.60:0/4090952312 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f36f4053a10 0x7f36f4055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:34.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.162+0000 7f37077fe700 1 -- 172.21.15.60:0/4090952312 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5425+0+0 (secure 0 0 0) 0x7f370806c100 con 0x7f371405bf30 2022-06-29T16:05:34.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.162+0000 7f3719915700 1 --2- 172.21.15.60:0/4090952312 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f36f4053a10 0x7f36f4055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:34.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.166+0000 7f3719915700 1 --2- 172.21.15.60:0/4090952312 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f36f4053a10 0x7f36f4055ed0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f371001a000 tx=0x7f3710014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:34.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.170+0000 7f37077fe700 1 -- 172.21.15.60:0/4090952312 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3708043130 con 0x7f371405bf30 2022-06-29T16:05:34.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.302+0000 7f371b118700 1 -- 172.21.15.60:0/4090952312 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f3714068320 con 0x7f371405bf30 2022-06-29T16:05:34.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.310+0000 7f37077fe700 1 -- 172.21.15.60:0/4090952312 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3708046750 con 0x7f371405bf30 2022-06-29T16:05:34.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.314+0000 7f371b118700 1 -- 172.21.15.60:0/4090952312 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f36f4053a10 msgr2=0x7f36f4055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:34.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.314+0000 7f371b118700 1 --2- 172.21.15.60:0/4090952312 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f36f4053a10 0x7f36f4055ed0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f371001a000 tx=0x7f3710014000).stop 2022-06-29T16:05:34.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.314+0000 7f371b118700 1 -- 172.21.15.60:0/4090952312 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f371405bf30 msgr2=0x7f3714114570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:34.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.314+0000 7f371b118700 1 --2- 172.21.15.60:0/4090952312 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f371405bf30 0x7f3714114570 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f370800c000 tx=0x7f3708006df0).stop 2022-06-29T16:05:34.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.314+0000 7f371b118700 1 -- 172.21.15.60:0/4090952312 shutdown_connections 2022-06-29T16:05:34.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.314+0000 7f371b118700 1 --2- 172.21.15.60:0/4090952312 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f36f4053a10 0x7f36f4055ed0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:34.326 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.314+0000 7f371b118700 1 --2- 172.21.15.60:0/4090952312 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37141441b0 0x7f37141516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:34.326 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.314+0000 7f371b118700 1 --2- 172.21.15.60:0/4090952312 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f37140b3c70 0x7f3714113960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:34.326 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.314+0000 7f371b118700 1 --2- 172.21.15.60:0/4090952312 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f371405bf30 0x7f3714114570 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:34.326 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.314+0000 7f371b118700 1 -- 172.21.15.60:0/4090952312 >> 172.21.15.60:0/4090952312 conn(0x7f3714064810 msgr2=0x7f3714140030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:34.326 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.314+0000 7f371b118700 1 -- 172.21.15.60:0/4090952312 shutdown_connections 2022-06-29T16:05:34.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.314+0000 7f371b118700 1 -- 172.21.15.60:0/4090952312 wait complete. 2022-06-29T16:05:34.341 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-06-29T16:05:34.341 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:05:34.355 DEBUG:teuthology.orchestra.run.smithi092:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:05:34.376 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:05:34.398 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:05:34.417 DEBUG:teuthology.orchestra.run.smithi197:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:05:34.438 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-06-29T16:05:34.439 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:34.439 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:05:34.440 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:05:34.447 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-06-29T16:05:34.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:05:34.449 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-06-29T16:05:34.449 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:34.450 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:05:34.450 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:05:34.465 INFO:teuthology.orchestra.run.smithi156.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-06-29T16:05:34.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:05:34.466 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-06-29T16:05:34.467 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:34.467 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:05:34.468 DEBUG:teuthology.orchestra.run.smithi197:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T16:05:34.475 INFO:teuthology.orchestra.run.smithi197.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-06-29T16:05:34.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:05:34.477 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-06-29T16:05:34.477 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-06-29T16:05:34.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.582+0000 7fb043644700 1 Processor -- start 2022-06-29T16:05:34.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.582+0000 7fb043644700 1 -- start start 2022-06-29T16:05:34.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.582+0000 7fb043644700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb03c0b4010 0x7fb03c1426b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:34.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.582+0000 7fb043644700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb03c0b46c0 0x7fb03c0b4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:34.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.582+0000 7fb043644700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb03c143a40 con 0x7fb03c0ae7c0 2022-06-29T16:05:34.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.582+0000 7fb043644700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb03c0aee60 con 0x7fb03c0b4010 2022-06-29T16:05:34.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.582+0000 7fb043644700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb03c0af100 con 0x7fb03c0b46c0 2022-06-29T16:05:34.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.582+0000 7fb040bdf700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb03c0b4010 0x7fb03c1426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:34.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.582+0000 7fb0413e0700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb03c0ae7c0 0x7fb03c141fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58570/0 (socket says 172.21.15.60:58570) 2022-06-29T16:05:34.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.582+0000 7fb0413e0700 1 -- 172.21.15.60:0/3396604509 learned_addr learned my addr 172.21.15.60:0/3396604509 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:34.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.582+0000 7fb041be1700 1 --2- 172.21.15.60:0/3396604509 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb03c0b46c0 0x7fb03c0b4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:34.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.582+0000 7fb040bdf700 1 -- 172.21.15.60:0/3396604509 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb03c0b46c0 msgr2=0x7fb03c0b4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:34.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.582+0000 7fb040bdf700 1 --2- 172.21.15.60:0/3396604509 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb03c0b46c0 0x7fb03c0b4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:34.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.582+0000 7fb040bdf700 1 -- 172.21.15.60:0/3396604509 >> v1:172.21.15.60:6789/0 conn(0x7fb03c0ae7c0 legacy=0x7fb03c141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:34.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.582+0000 7fb040bdf700 1 -- 172.21.15.60:0/3396604509 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb03c0aeba0 con 0x7fb03c0b4010 2022-06-29T16:05:34.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.582+0000 7fb041be1700 1 --2- 172.21.15.60:0/3396604509 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb03c0b46c0 0x7fb03c0b4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:05:34.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.582+0000 7fb040bdf700 1 --2- 172.21.15.60:0/3396604509 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb03c0b4010 0x7fb03c1426b0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 rx=0x7fb030002f70 tx=0x7fb030006df0).ready entity=mon.0 client_cookie=5d6061444fa391ae server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:34.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.582+0000 7fb03bfff700 1 -- 172.21.15.60:0/3396604509 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb03000d570 con 0x7fb03c0b4010 2022-06-29T16:05:34.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.582+0000 7fb03bfff700 1 -- 172.21.15.60:0/3396604509 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb030000e90 con 0x7fb03c0b4010 2022-06-29T16:05:34.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.582+0000 7fb03bfff700 1 -- 172.21.15.60:0/3396604509 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb03000d8c0 con 0x7fb03c0b4010 2022-06-29T16:05:34.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.582+0000 7fb043644700 1 -- 172.21.15.60:0/3396604509 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb03c0b4010 msgr2=0x7fb03c1426b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:34.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.582+0000 7fb043644700 1 --2- 172.21.15.60:0/3396604509 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb03c0b4010 0x7fb03c1426b0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 rx=0x7fb030002f70 tx=0x7fb030006df0).stop 2022-06-29T16:05:34.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.582+0000 7fb043644700 1 -- 172.21.15.60:0/3396604509 shutdown_connections 2022-06-29T16:05:34.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.582+0000 7fb043644700 1 --2- 172.21.15.60:0/3396604509 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb03c0b46c0 0x7fb03c0b4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:34.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.582+0000 7fb043644700 1 --2- 172.21.15.60:0/3396604509 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb03c0b4010 0x7fb03c1426b0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:34.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.582+0000 7fb043644700 1 -- 172.21.15.60:0/3396604509 >> 172.21.15.60:0/3396604509 conn(0x7fb03c0b9340 msgr2=0x7fb03c0b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:34.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.582+0000 7fb043644700 1 -- 172.21.15.60:0/3396604509 shutdown_connections 2022-06-29T16:05:34.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.582+0000 7fb043644700 1 -- 172.21.15.60:0/3396604509 wait complete. 2022-06-29T16:05:34.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.586+0000 7fb043644700 1 Processor -- start 2022-06-29T16:05:34.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.586+0000 7fb043644700 1 -- start start 2022-06-29T16:05:34.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.586+0000 7fb043644700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb03c0ae7c0 0x7fb03c0b7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:34.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.586+0000 7fb043644700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb03c0b4010 0x7fb03c0b0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:34.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.586+0000 7fb043644700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb03c0b46c0 0x7fb03c13ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:34.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.586+0000 7fb043644700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb03c0af4e0 con 0x7fb03c0b46c0 2022-06-29T16:05:34.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.586+0000 7fb043644700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb03c0af710 con 0x7fb03c0ae7c0 2022-06-29T16:05:34.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.586+0000 7fb043644700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb03c0af9b0 con 0x7fb03c0b4010 2022-06-29T16:05:34.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.586+0000 7fb041be1700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb03c0b46c0 0x7fb03c13ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:34.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.586+0000 7fb041be1700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb03c0b46c0 0x7fb03c13ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50494/0 (socket says 172.21.15.60:50494) 2022-06-29T16:05:34.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.586+0000 7fb040bdf700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb03c0b4010 0x7fb03c0b0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:34.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.586+0000 7fb041be1700 1 -- 172.21.15.60:0/2796802139 learned_addr learned my addr 172.21.15.60:0/2796802139 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:34.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.586+0000 7fb0413e0700 1 --2- 172.21.15.60:0/2796802139 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb03c0ae7c0 0x7fb03c0b7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:34.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.586+0000 7fb041be1700 1 -- 172.21.15.60:0/2796802139 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb03c0b4010 msgr2=0x7fb03c0b0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:34.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.586+0000 7fb041be1700 1 --2- 172.21.15.60:0/2796802139 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb03c0b4010 0x7fb03c0b0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:34.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.586+0000 7fb041be1700 1 -- 172.21.15.60:0/2796802139 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb03c0ae7c0 msgr2=0x7fb03c0b7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:34.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.586+0000 7fb041be1700 1 --2- 172.21.15.60:0/2796802139 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb03c0ae7c0 0x7fb03c0b7820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:34.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.586+0000 7fb041be1700 1 -- 172.21.15.60:0/2796802139 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb03000b180 con 0x7fb03c0b46c0 2022-06-29T16:05:34.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.586+0000 7fb041be1700 1 --2- 172.21.15.60:0/2796802139 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb03c0b46c0 0x7fb03c13ea20 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 rx=0x7fb02c007000 tx=0x7fb02c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:34.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.586+0000 7fb03a7fc700 1 -- 172.21.15.60:0/2796802139 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb02c00ce20 con 0x7fb03c0b46c0 2022-06-29T16:05:34.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.586+0000 7fb03a7fc700 1 -- 172.21.15.60:0/2796802139 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb02c00f4e0 con 0x7fb03c0b46c0 2022-06-29T16:05:34.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.586+0000 7fb043644700 1 -- 172.21.15.60:0/2796802139 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb03c155490 con 0x7fb03c0b46c0 2022-06-29T16:05:34.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.586+0000 7fb03a7fc700 1 -- 172.21.15.60:0/2796802139 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb02c00f840 con 0x7fb03c0b46c0 2022-06-29T16:05:34.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.586+0000 7fb043644700 1 -- 172.21.15.60:0/2796802139 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb03c145e30 con 0x7fb03c0b46c0 2022-06-29T16:05:34.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.590+0000 7fb043644700 1 -- 172.21.15.60:0/2796802139 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb03c155620 con 0x7fb03c0b46c0 2022-06-29T16:05:34.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.590+0000 7fb03a7fc700 1 -- 172.21.15.60:0/2796802139 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb02c019030 con 0x7fb03c0b46c0 2022-06-29T16:05:34.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.590+0000 7fb03a7fc700 1 --2- 172.21.15.60:0/2796802139 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb01c053a70 0x7fb01c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:34.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.590+0000 7fb03a7fc700 1 -- 172.21.15.60:0/2796802139 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5425+0+0 (secure 0 0 0) 0x7fb02c06b680 con 0x7fb03c0b46c0 2022-06-29T16:05:34.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.590+0000 7fb0413e0700 1 --2- 172.21.15.60:0/2796802139 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb01c053a70 0x7fb01c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:34.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.590+0000 7fb0413e0700 1 --2- 172.21.15.60:0/2796802139 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb01c053a70 0x7fb01c055f30 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7fb03400d000 tx=0x7fb034009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:34.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.598+0000 7fb03a7fc700 1 -- 172.21.15.60:0/2796802139 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb02c042780 con 0x7fb03c0b46c0 2022-06-29T16:05:34.734 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.726+0000 7fb043644700 1 -- 172.21.15.60:0/2796802139 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fb03c0573d0 con 0x7fb03c0b46c0 2022-06-29T16:05:34.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.726+0000 7fb03a7fc700 1 -- 172.21.15.60:0/2796802139 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v84) v1 ==== 81+0+28978 (secure 0 0 0) 0x7fb02c045da0 con 0x7fb03c0b46c0 2022-06-29T16:05:34.736 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:34.736 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:05:34.737 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 84, 2022-06-29T16:05:34.737 INFO:teuthology.orchestra.run.smithi060.stdout: "fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", 2022-06-29T16:05:34.737 INFO:teuthology.orchestra.run.smithi060.stdout: "created": "2022-06-29T15:58:07.965719+0000", 2022-06-29T16:05:34.737 INFO:teuthology.orchestra.run.smithi060.stdout: "modified": "2022-06-29T16:05:31.992063+0000", 2022-06-29T16:05:34.738 INFO:teuthology.orchestra.run.smithi060.stdout: "last_up_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:05:34.738 INFO:teuthology.orchestra.run.smithi060.stdout: "last_in_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:05:34.738 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-06-29T16:05:34.738 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_num": 5799936, 2022-06-29T16:05:34.739 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_set": [ 2022-06-29T16:05:34.739 INFO:teuthology.orchestra.run.smithi060.stdout: "pglog_hardlimit", 2022-06-29T16:05:34.739 INFO:teuthology.orchestra.run.smithi060.stdout: "purged_snapdirs", 2022-06-29T16:05:34.739 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_deletes", 2022-06-29T16:05:34.740 INFO:teuthology.orchestra.run.smithi060.stdout: "sortbitwise" 2022-06-29T16:05:34.740 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:05:34.740 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_version": 4, 2022-06-29T16:05:34.740 INFO:teuthology.orchestra.run.smithi060.stdout: "full_ratio": 0.94999998807907104, 2022-06-29T16:05:34.740 INFO:teuthology.orchestra.run.smithi060.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-06-29T16:05:34.741 INFO:teuthology.orchestra.run.smithi060.stdout: "nearfull_ratio": 0.85000002384185791, 2022-06-29T16:05:34.741 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_snapshot": "", 2022-06-29T16:05:34.741 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_max": 16, 2022-06-29T16:05:34.741 INFO:teuthology.orchestra.run.smithi060.stdout: "max_osd": 8, 2022-06-29T16:05:34.742 INFO:teuthology.orchestra.run.smithi060.stdout: "require_min_compat_client": "jewel", 2022-06-29T16:05:34.742 INFO:teuthology.orchestra.run.smithi060.stdout: "min_compat_client": "jewel", 2022-06-29T16:05:34.742 INFO:teuthology.orchestra.run.smithi060.stdout: "require_osd_release": "octopus", 2022-06-29T16:05:34.742 INFO:teuthology.orchestra.run.smithi060.stdout: "pools": [ 2022-06-29T16:05:34.742 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.743 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2022-06-29T16:05:34.743 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "device_health_metrics", 2022-06-29T16:05:34.743 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:10.961127+0000", 2022-06-29T16:05:34.743 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:05:34.744 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:05:34.744 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:05:34.744 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:05:34.744 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:05:34.744 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:05:34.745 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:05:34.745 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:05:34.745 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 1, 2022-06-29T16:05:34.745 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 1, 2022-06-29T16:05:34.745 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 1, 2022-06-29T16:05:34.746 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 1, 2022-06-29T16:05:34.746 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 1, 2022-06-29T16:05:34.746 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:05:34.746 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:05:34.747 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:05:34.747 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:05:34.747 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:05:34.747 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:05:34.747 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:05:34.748 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.748 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "15", 2022-06-29T16:05:34.748 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:05:34.748 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:05:34.749 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:05:34.749 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:05:34.749 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:05:34.749 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:05:34.749 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:05:34.750 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:05:34.750 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:05:34.750 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:05:34.750 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:05:34.751 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:05:34.751 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:05:34.751 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:05:34.751 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:05:34.752 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:05:34.752 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:05:34.752 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:05:34.752 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:05:34.752 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:05:34.753 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:05:34.753 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:05:34.753 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:05:34.753 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:05:34.754 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:05:34.754 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:05:34.754 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.754 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:05:34.755 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:05:34.755 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:05:34.755 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:05:34.755 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:05:34.756 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:05:34.756 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:05:34.756 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:05:34.756 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:05:34.756 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:05:34.757 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:05:34.757 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:05:34.757 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 1 2022-06-29T16:05:34.757 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.758 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:05:34.758 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_devicehealth": {} 2022-06-29T16:05:34.758 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.758 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.758 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.759 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 2, 2022-06-29T16:05:34.759 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "rbd", 2022-06-29T16:05:34.759 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:19.184276+0000", 2022-06-29T16:05:34.760 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:05:34.760 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:05:34.760 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:05:34.760 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:05:34.760 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:05:34.761 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:05:34.761 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:05:34.761 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:05:34.761 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:05:34.762 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:05:34.762 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:05:34.762 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:05:34.762 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:05:34.763 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:05:34.763 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:05:34.763 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:05:34.763 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:05:34.764 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:05:34.764 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:05:34.764 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:05:34.764 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.764 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "18", 2022-06-29T16:05:34.765 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:05:34.765 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:05:34.765 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:05:34.765 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:05:34.766 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:05:34.766 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:05:34.766 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:05:34.766 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:05:34.766 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:05:34.767 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:05:34.767 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:05:34.767 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:05:34.767 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:05:34.768 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:05:34.768 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:05:34.768 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:05:34.768 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:05:34.768 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:05:34.769 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:05:34.769 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:05:34.769 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:05:34.769 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:05:34.770 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:05:34.770 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:05:34.770 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:05:34.770 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:05:34.770 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.771 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:05:34.771 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:05:34.771 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:05:34.771 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:05:34.772 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:05:34.772 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:05:34.772 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:05:34.772 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:05:34.773 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:05:34.773 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:05:34.773 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:05:34.773 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:05:34.773 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:05:34.774 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd": {} 2022-06-29T16:05:34.774 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.774 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.774 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:05:34.775 INFO:teuthology.orchestra.run.smithi060.stdout: "osds": [ 2022-06-29T16:05:34.775 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.775 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:05:34.775 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "11e7ab9e-fce2-470f-8195-ea4c3c52df22", 2022-06-29T16:05:34.775 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:34.776 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:34.776 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:34.776 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:34.776 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:34.777 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:34.777 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:34.777 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 78, 2022-06-29T16:05:34.777 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:34.777 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:34.778 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:34.778 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:34.778 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.779 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:34.779 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6816", 2022-06-29T16:05:34.779 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:34.779 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.780 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.780 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:34.780 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6825", 2022-06-29T16:05:34.780 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:34.781 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.781 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.781 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.781 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:34.781 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:34.782 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.782 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:34.782 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6826", 2022-06-29T16:05:34.782 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:34.782 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.783 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.783 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:34.783 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6827", 2022-06-29T16:05:34.783 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:34.784 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.784 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.784 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.784 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:34.784 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:34.785 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.785 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:34.785 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6830", 2022-06-29T16:05:34.786 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:34.786 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.786 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.786 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:34.786 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6831", 2022-06-29T16:05:34.787 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:34.787 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.787 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.787 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.788 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:34.788 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:34.788 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.788 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:34.788 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6828", 2022-06-29T16:05:34.789 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:34.789 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.789 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.789 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:34.790 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6829", 2022-06-29T16:05:34.790 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:34.790 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.790 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.790 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.791 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6825/12890", 2022-06-29T16:05:34.791 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6827/12890", 2022-06-29T16:05:34.791 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6831/12890", 2022-06-29T16:05:34.791 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6829/12890", 2022-06-29T16:05:34.792 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:34.792 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:34.792 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:34.792 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.793 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.793 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.793 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:05:34.793 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "ca43f136-6f69-4431-a157-fbe073fb585c", 2022-06-29T16:05:34.794 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:34.794 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:34.794 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:34.794 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:34.795 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:34.795 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:34.795 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:34.795 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 77, 2022-06-29T16:05:34.795 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:34.796 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:34.796 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:34.796 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:34.796 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.796 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:34.797 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6815", 2022-06-29T16:05:34.797 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:34.797 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.797 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.798 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:34.798 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6818", 2022-06-29T16:05:34.798 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:34.798 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.798 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.799 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.799 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:34.799 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:34.799 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.800 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:34.800 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6819", 2022-06-29T16:05:34.800 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:34.801 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.801 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.801 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:34.801 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6820", 2022-06-29T16:05:34.802 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:34.802 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.802 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.802 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.803 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:34.803 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:34.803 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.803 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:34.803 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6823", 2022-06-29T16:05:34.804 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:34.804 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.804 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.804 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:34.805 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6824", 2022-06-29T16:05:34.805 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:34.805 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.805 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.805 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.806 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:34.806 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:34.806 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.806 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:34.807 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6821", 2022-06-29T16:05:34.807 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:34.807 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.807 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.808 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:34.808 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6822", 2022-06-29T16:05:34.808 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:34.808 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.808 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.809 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.809 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6818/12892", 2022-06-29T16:05:34.809 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6820/12892", 2022-06-29T16:05:34.809 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6824/12892", 2022-06-29T16:05:34.809 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6822/12892", 2022-06-29T16:05:34.810 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:34.810 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:34.810 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:34.810 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.811 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.811 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.811 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:05:34.811 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "74fac45f-5f07-44ec-a9cb-2644c123fd4a", 2022-06-29T16:05:34.811 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:34.812 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:34.812 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:34.812 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:34.812 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:34.813 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:34.813 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:34.813 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 78, 2022-06-29T16:05:34.813 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:34.814 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:34.814 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:34.814 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:34.814 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.814 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:34.815 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6808", 2022-06-29T16:05:34.815 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:34.815 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.815 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.816 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:34.816 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6809", 2022-06-29T16:05:34.816 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:34.816 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.816 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.817 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.817 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:34.817 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:34.817 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.817 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:34.818 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6810", 2022-06-29T16:05:34.818 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:34.818 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.818 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.819 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:34.819 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6811", 2022-06-29T16:05:34.819 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:34.819 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.819 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.820 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.820 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:34.820 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:34.820 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.821 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:34.821 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6814", 2022-06-29T16:05:34.821 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:34.821 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.821 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.822 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:34.822 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6817", 2022-06-29T16:05:34.822 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:34.822 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.823 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.823 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.823 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:34.823 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:34.823 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.824 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:34.824 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6812", 2022-06-29T16:05:34.824 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:34.824 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.824 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.825 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:34.825 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6813", 2022-06-29T16:05:34.825 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:34.825 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.826 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.826 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.826 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6809/12891", 2022-06-29T16:05:34.826 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6811/12891", 2022-06-29T16:05:34.827 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6817/12891", 2022-06-29T16:05:34.827 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6813/12891", 2022-06-29T16:05:34.827 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:34.827 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:34.827 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:34.828 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.828 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.828 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.828 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:05:34.829 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "2581d351-973a-4b4e-85ca-3658f0068765", 2022-06-29T16:05:34.829 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:34.829 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:34.829 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:34.829 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:34.830 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:34.830 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:34.830 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:34.830 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 78, 2022-06-29T16:05:34.831 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:34.831 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:34.831 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:34.831 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:34.831 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.832 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:34.832 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6800", 2022-06-29T16:05:34.832 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:34.832 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.832 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.833 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:34.833 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6801", 2022-06-29T16:05:34.833 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:34.833 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.834 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.834 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.834 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:34.834 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:34.835 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.835 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:34.835 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6802", 2022-06-29T16:05:34.835 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:34.836 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.836 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.836 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:34.836 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6803", 2022-06-29T16:05:34.836 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:34.837 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.837 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.837 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.837 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:34.837 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:34.838 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.838 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:34.838 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6806", 2022-06-29T16:05:34.838 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:34.839 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.839 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.839 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:34.839 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6807", 2022-06-29T16:05:34.839 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:34.840 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.840 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.840 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.840 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:34.841 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:34.841 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.841 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:34.841 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6804", 2022-06-29T16:05:34.842 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:34.842 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.842 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.842 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:34.843 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6805", 2022-06-29T16:05:34.843 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:34.843 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.843 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.843 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.844 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6801/12889", 2022-06-29T16:05:34.844 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6803/12889", 2022-06-29T16:05:34.844 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6807/12889", 2022-06-29T16:05:34.844 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6805/12889", 2022-06-29T16:05:34.845 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:34.845 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:34.845 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:34.845 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.846 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.846 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.846 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:05:34.846 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "3d234db9-87cd-4fe9-814f-0240a287d8b4", 2022-06-29T16:05:34.847 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:34.847 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:34.847 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:34.847 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:34.848 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:34.848 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:34.848 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:34.848 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 78, 2022-06-29T16:05:34.848 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:34.849 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:34.849 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:34.849 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:34.849 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.850 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:34.850 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6808", 2022-06-29T16:05:34.850 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:34.850 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.851 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.851 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:34.851 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6810", 2022-06-29T16:05:34.851 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:34.851 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.852 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.852 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.852 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:34.852 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:34.853 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.853 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:34.853 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6812", 2022-06-29T16:05:34.853 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:34.854 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.854 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.854 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:34.854 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6814", 2022-06-29T16:05:34.854 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:34.855 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.855 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.855 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.855 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:34.856 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:34.856 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.856 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:34.856 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6823", 2022-06-29T16:05:34.856 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:34.857 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.857 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.857 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:34.857 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6825", 2022-06-29T16:05:34.858 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:34.858 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.858 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.858 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.859 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:34.859 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:34.859 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.859 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:34.859 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6817", 2022-06-29T16:05:34.860 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:34.860 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.860 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.860 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:34.861 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6820", 2022-06-29T16:05:34.861 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:34.861 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.861 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.862 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.862 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6810/13249", 2022-06-29T16:05:34.862 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6814/13249", 2022-06-29T16:05:34.863 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6825/13249", 2022-06-29T16:05:34.863 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6820/13249", 2022-06-29T16:05:34.863 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:34.863 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:34.864 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:34.864 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.864 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.864 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.864 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:05:34.865 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "e4bf1104-ff5c-438b-b608-8cee31c163f6", 2022-06-29T16:05:34.865 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:34.865 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:34.865 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:34.866 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:34.866 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:34.866 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:34.866 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:34.866 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 78, 2022-06-29T16:05:34.867 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:34.867 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:34.867 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:34.867 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:34.867 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.868 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:34.868 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6809", 2022-06-29T16:05:34.868 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:34.868 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.869 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.869 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:34.869 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6811", 2022-06-29T16:05:34.869 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:34.869 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.870 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.870 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.870 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:34.870 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:34.871 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.871 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:34.871 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6813", 2022-06-29T16:05:34.871 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:34.871 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.872 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.872 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:34.872 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6815", 2022-06-29T16:05:34.872 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:34.873 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.873 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.873 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.873 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:34.874 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:34.874 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.874 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:34.874 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6824", 2022-06-29T16:05:34.874 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:34.875 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.875 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.875 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:34.875 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6827", 2022-06-29T16:05:34.876 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:34.876 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.876 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.876 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.876 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:34.877 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:34.877 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.877 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:34.877 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6818", 2022-06-29T16:05:34.878 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:34.878 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.878 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.878 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:34.878 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6821", 2022-06-29T16:05:34.879 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:34.879 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.879 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.879 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.880 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6811/13248", 2022-06-29T16:05:34.880 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6815/13248", 2022-06-29T16:05:34.880 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6827/13248", 2022-06-29T16:05:34.880 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6821/13248", 2022-06-29T16:05:34.881 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:34.881 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:34.881 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:34.881 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.882 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.882 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.882 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:05:34.882 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "8564dd25-27ab-4203-b54e-af6d3aa05857", 2022-06-29T16:05:34.882 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:34.883 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:34.883 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:34.883 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:34.883 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:34.884 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:34.884 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:34.884 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 78, 2022-06-29T16:05:34.884 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:34.885 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:34.885 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:34.885 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:34.885 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.885 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:34.886 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6816", 2022-06-29T16:05:34.886 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:34.886 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.886 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.887 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:34.887 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6819", 2022-06-29T16:05:34.887 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:34.887 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.888 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.888 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.888 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:34.888 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:34.888 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.889 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:34.889 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6822", 2022-06-29T16:05:34.889 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:34.889 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.890 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.890 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:34.890 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6826", 2022-06-29T16:05:34.890 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:34.891 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.891 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.891 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.891 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:34.891 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:34.892 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.892 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:34.892 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6830", 2022-06-29T16:05:34.892 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:34.893 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.893 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.893 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:34.893 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6831", 2022-06-29T16:05:34.894 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:34.894 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.894 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.894 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.894 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:34.895 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:34.895 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.895 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:34.895 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6828", 2022-06-29T16:05:34.896 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:34.896 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.896 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.896 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:34.896 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6829", 2022-06-29T16:05:34.897 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:34.897 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.897 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.897 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.898 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6819/13250", 2022-06-29T16:05:34.898 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6826/13250", 2022-06-29T16:05:34.898 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6831/13250", 2022-06-29T16:05:34.898 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6829/13250", 2022-06-29T16:05:34.899 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:34.899 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:34.899 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:34.899 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.899 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.900 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.900 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:05:34.900 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "bd513a9c-3b19-4314-9897-d300b5c897bb", 2022-06-29T16:05:34.900 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:34.901 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:34.901 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:34.901 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:34.901 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:34.901 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:34.902 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:34.902 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 68, 2022-06-29T16:05:34.902 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:34.902 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:34.903 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:34.903 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:34.903 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.903 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:34.903 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6800", 2022-06-29T16:05:34.904 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:34.904 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.904 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.904 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:34.904 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6801", 2022-06-29T16:05:34.905 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:34.905 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.905 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.905 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.905 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:34.906 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:34.906 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.906 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:34.906 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6802", 2022-06-29T16:05:34.907 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:34.907 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.907 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.907 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:34.908 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6803", 2022-06-29T16:05:34.908 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:34.908 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.908 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.908 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.909 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:34.909 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:34.909 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.909 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:34.910 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6806", 2022-06-29T16:05:34.910 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:34.910 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.910 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.911 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:34.911 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6807", 2022-06-29T16:05:34.911 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:34.911 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.912 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.912 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.912 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:34.912 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:34.913 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.913 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:34.913 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6804", 2022-06-29T16:05:34.913 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:34.913 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.914 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.914 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:34.914 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6805", 2022-06-29T16:05:34.914 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:34.915 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.915 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.915 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.915 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6801/13247", 2022-06-29T16:05:34.915 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6803/13247", 2022-06-29T16:05:34.916 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6807/13247", 2022-06-29T16:05:34.916 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6805/13247", 2022-06-29T16:05:34.916 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:34.916 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:34.916 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:34.917 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:34.917 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.917 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:05:34.917 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_xinfo": [ 2022-06-29T16:05:34.918 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.918 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:05:34.918 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:34.918 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:34.919 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:34.919 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:34.919 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:34.919 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.891342+0000", 2022-06-29T16:05:34.919 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:34.920 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.920 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.920 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:05:34.920 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:34.920 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:34.921 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:34.921 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:34.921 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:34.921 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.900866+0000", 2022-06-29T16:05:34.921 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:34.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.922 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:05:34.922 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:34.923 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:34.923 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:34.923 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:34.923 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:34.923 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.912755+0000", 2022-06-29T16:05:34.924 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:34.924 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.924 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.924 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:05:34.925 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:34.925 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:34.925 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:34.925 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:34.925 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:34.926 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.864332+0000", 2022-06-29T16:05:34.926 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:34.926 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.926 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.927 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:05:34.927 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:34.927 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:34.927 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:34.927 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:34.928 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:34.928 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.863756+0000", 2022-06-29T16:05:34.928 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:34.928 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.929 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.929 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:05:34.929 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:34.929 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:34.929 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:34.930 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:34.930 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:34.930 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.869041+0000", 2022-06-29T16:05:34.930 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:34.931 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.931 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.931 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:05:34.931 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:34.932 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:34.932 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:34.932 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:34.932 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:34.932 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.895043+0000", 2022-06-29T16:05:34.933 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:34.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.933 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:34.933 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:05:34.934 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:34.934 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:34.934 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:34.934 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:34.934 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:34.935 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.889491+0000", 2022-06-29T16:05:34.935 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:34.935 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.935 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:05:34.936 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap": [], 2022-06-29T16:05:34.936 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap_items": [], 2022-06-29T16:05:34.936 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_temp": [], 2022-06-29T16:05:34.936 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_temp": [], 2022-06-29T16:05:34.936 INFO:teuthology.orchestra.run.smithi060.stdout: "blacklist": { 2022-06-29T16:05:34.937 INFO:teuthology.orchestra.run.smithi060.stdout: "172.21.15.92:6835/1245503283": "2022-06-30T16:05:28.480010+0000", 2022-06-29T16:05:34.937 INFO:teuthology.orchestra.run.smithi060.stdout: "172.21.15.92:6834/1245503283": "2022-06-30T16:05:28.480010+0000", 2022-06-29T16:05:34.937 INFO:teuthology.orchestra.run.smithi060.stdout: "172.21.15.92:6835/2464413229": "2022-06-30T16:05:16.667864+0000", 2022-06-29T16:05:34.937 INFO:teuthology.orchestra.run.smithi060.stdout: "172.21.15.92:6834/2464413229": "2022-06-30T16:05:16.667864+0000" 2022-06-29T16:05:34.938 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.938 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profiles": { 2022-06-29T16:05:34.938 INFO:teuthology.orchestra.run.smithi060.stdout: "default": { 2022-06-29T16:05:34.938 INFO:teuthology.orchestra.run.smithi060.stdout: "crush-failure-domain": "osd", 2022-06-29T16:05:34.938 INFO:teuthology.orchestra.run.smithi060.stdout: "k": "2", 2022-06-29T16:05:34.939 INFO:teuthology.orchestra.run.smithi060.stdout: "m": "1", 2022-06-29T16:05:34.939 INFO:teuthology.orchestra.run.smithi060.stdout: "plugin": "jerasure", 2022-06-29T16:05:34.939 INFO:teuthology.orchestra.run.smithi060.stdout: "ruleset-failure-domain": "osd", 2022-06-29T16:05:34.939 INFO:teuthology.orchestra.run.smithi060.stdout: "technique": "reed_sol_van" 2022-06-29T16:05:34.940 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:34.940 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:34.940 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps_queue": [], 2022-06-29T16:05:34.940 INFO:teuthology.orchestra.run.smithi060.stdout: "new_removed_snaps": [], 2022-06-29T16:05:34.941 INFO:teuthology.orchestra.run.smithi060.stdout: "new_purged_snaps": [], 2022-06-29T16:05:34.941 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_node_flags": {}, 2022-06-29T16:05:34.941 INFO:teuthology.orchestra.run.smithi060.stdout: "device_class_flags": {} 2022-06-29T16:05:34.941 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:05:34.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.730+0000 7fb043644700 1 -- 172.21.15.60:0/2796802139 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb01c053a70 msgr2=0x7fb01c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:34.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.730+0000 7fb043644700 1 --2- 172.21.15.60:0/2796802139 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb01c053a70 0x7fb01c055f30 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7fb03400d000 tx=0x7fb034009040).stop 2022-06-29T16:05:34.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.730+0000 7fb043644700 1 -- 172.21.15.60:0/2796802139 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb03c0b46c0 msgr2=0x7fb03c13ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:34.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.730+0000 7fb043644700 1 --2- 172.21.15.60:0/2796802139 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb03c0b46c0 0x7fb03c13ea20 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 rx=0x7fb02c007000 tx=0x7fb02c008040).stop 2022-06-29T16:05:34.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.730+0000 7fb043644700 1 -- 172.21.15.60:0/2796802139 shutdown_connections 2022-06-29T16:05:34.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.730+0000 7fb043644700 1 --2- 172.21.15.60:0/2796802139 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb01c053a70 0x7fb01c055f30 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:34.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.730+0000 7fb043644700 1 --2- 172.21.15.60:0/2796802139 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb03c0b46c0 0x7fb03c13ea20 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:34.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.730+0000 7fb043644700 1 --2- 172.21.15.60:0/2796802139 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb03c0b4010 0x7fb03c0b0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:34.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.730+0000 7fb043644700 1 --2- 172.21.15.60:0/2796802139 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb03c0ae7c0 0x7fb03c0b7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:34.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.730+0000 7fb043644700 1 -- 172.21.15.60:0/2796802139 >> 172.21.15.60:0/2796802139 conn(0x7fb03c0b9340 msgr2=0x7fb03c05cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:34.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.730+0000 7fb043644700 1 -- 172.21.15.60:0/2796802139 shutdown_connections 2022-06-29T16:05:34.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:34.730+0000 7fb043644700 1 -- 172.21.15.60:0/2796802139 wait complete. 2022-06-29T16:05:34.947 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:05:35.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.050+0000 7fc1aa698700 1 Processor -- start 2022-06-29T16:05:35.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.050+0000 7fc1aa698700 1 -- start start 2022-06-29T16:05:35.064 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.054+0000 7fc1aa698700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc1a413a920 0x7fc1a4143f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:35.064 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.054+0000 7fc1aa698700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1a4140ab0 0x7fc1a4140f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:35.064 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.054+0000 7fc1aa698700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1a413ad00 con 0x7fc1a40ae270 2022-06-29T16:05:35.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.054+0000 7fc1aa698700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1a4144420 con 0x7fc1a413a920 2022-06-29T16:05:35.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.054+0000 7fc1aa698700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a40b5d20 con 0x7fc1a4140ab0 2022-06-29T16:05:35.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.054+0000 7fc1a3fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc1a413a920 0x7fc1a4143f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:35.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.054+0000 7fc1a3fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc1a413a920 0x7fc1a4143f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50500/0 (socket says 172.21.15.60:50500) 2022-06-29T16:05:35.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.054+0000 7fc1a3fff700 1 -- 172.21.15.60:0/2244870242 learned_addr learned my addr 172.21.15.60:0/2244870242 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:35.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.054+0000 7fc1a8c35700 1 --2- 172.21.15.60:0/2244870242 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1a4140ab0 0x7fc1a4140f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:35.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.054+0000 7fc1a2ffd700 1 -- 172.21.15.60:0/2244870242 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206182213 0 0) 0x7fc1a413ad00 con 0x7fc1a40ae270 2022-06-29T16:05:35.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.054+0000 7fc1a3fff700 1 -- 172.21.15.60:0/2244870242 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1a4140ab0 msgr2=0x7fc1a4140f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:35.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.054+0000 7fc1a3fff700 1 --2- 172.21.15.60:0/2244870242 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1a4140ab0 0x7fc1a4140f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:35.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.054+0000 7fc1a3fff700 1 -- 172.21.15.60:0/2244870242 >> v1:172.21.15.92:6789/0 conn(0x7fc1a40ae270 legacy=0x7fc1a40b5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:35.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.054+0000 7fc1a3fff700 1 -- 172.21.15.60:0/2244870242 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a4144670 con 0x7fc1a413a920 2022-06-29T16:05:35.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.054+0000 7fc1a3fff700 1 --2- 172.21.15.60:0/2244870242 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1a413a920 0x7fc1a4143f30 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 rx=0x7fc18c002f70 tx=0x7fc18c006eb0).ready entity=mon.0 client_cookie=3f0f8ecf3098134 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:35.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.054+0000 7fc1a2ffd700 1 -- 172.21.15.60:0/2244870242 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc18c00d570 con 0x7fc1a413a920 2022-06-29T16:05:35.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.054+0000 7fc1a2ffd700 1 -- 172.21.15.60:0/2244870242 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc18c000e90 con 0x7fc1a413a920 2022-06-29T16:05:35.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.054+0000 7fc1a2ffd700 1 -- 172.21.15.60:0/2244870242 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc18c00d8c0 con 0x7fc1a413a920 2022-06-29T16:05:35.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.054+0000 7fc1aa698700 1 -- 172.21.15.60:0/2244870242 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1a413a920 msgr2=0x7fc1a4143f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:35.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.054+0000 7fc1aa698700 1 --2- 172.21.15.60:0/2244870242 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1a413a920 0x7fc1a4143f30 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 rx=0x7fc18c002f70 tx=0x7fc18c006eb0).stop 2022-06-29T16:05:35.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.054+0000 7fc1aa698700 1 -- 172.21.15.60:0/2244870242 shutdown_connections 2022-06-29T16:05:35.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.054+0000 7fc1aa698700 1 --2- 172.21.15.60:0/2244870242 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1a4140ab0 0x7fc1a4140f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:35.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.054+0000 7fc1aa698700 1 --2- 172.21.15.60:0/2244870242 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1a413a920 0x7fc1a4143f30 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:35.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.054+0000 7fc1aa698700 1 -- 172.21.15.60:0/2244870242 >> 172.21.15.60:0/2244870242 conn(0x7fc1a40632c0 msgr2=0x7fc1a40b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:35.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.054+0000 7fc1aa698700 1 -- 172.21.15.60:0/2244870242 shutdown_connections 2022-06-29T16:05:35.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.054+0000 7fc1aa698700 1 -- 172.21.15.60:0/2244870242 wait complete. 2022-06-29T16:05:35.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.058+0000 7fc1aa698700 1 Processor -- start 2022-06-29T16:05:35.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.058+0000 7fc1aa698700 1 -- start start 2022-06-29T16:05:35.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.058+0000 7fc1aa698700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1a40ae270 0x7fc1a41500d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:35.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.058+0000 7fc1aa698700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1a413a920 0x7fc1a40b47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:35.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.058+0000 7fc1aa698700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1a4140ab0 0x7fc1a4153a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:35.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.058+0000 7fc1aa698700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a4153620 con 0x7fc1a40ae270 2022-06-29T16:05:35.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.058+0000 7fc1aa698700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a413f180 con 0x7fc1a413a920 2022-06-29T16:05:35.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.058+0000 7fc1aa698700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a413f420 con 0x7fc1a4140ab0 2022-06-29T16:05:35.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.058+0000 7fc1a3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1a40ae270 0x7fc1a41500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:35.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.058+0000 7fc1a3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1a40ae270 0x7fc1a41500d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50504/0 (socket says 172.21.15.60:50504) 2022-06-29T16:05:35.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.058+0000 7fc1a3fff700 1 -- 172.21.15.60:0/388251458 learned_addr learned my addr 172.21.15.60:0/388251458 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:35.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.058+0000 7fc1a8c35700 1 --2- 172.21.15.60:0/388251458 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1a4140ab0 0x7fc1a4153a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:35.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.058+0000 7fc1a37fe700 1 --2- 172.21.15.60:0/388251458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1a413a920 0x7fc1a40b47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:35.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.058+0000 7fc1a3fff700 1 -- 172.21.15.60:0/388251458 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1a4140ab0 msgr2=0x7fc1a4153a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:35.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.058+0000 7fc1a3fff700 1 --2- 172.21.15.60:0/388251458 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1a4140ab0 0x7fc1a4153a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:35.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.058+0000 7fc1a3fff700 1 -- 172.21.15.60:0/388251458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1a413a920 msgr2=0x7fc1a40b47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:35.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.058+0000 7fc1a3fff700 1 --2- 172.21.15.60:0/388251458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1a413a920 0x7fc1a40b47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:35.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.058+0000 7fc1a3fff700 1 -- 172.21.15.60:0/388251458 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc18c00b1e0 con 0x7fc1a40ae270 2022-06-29T16:05:35.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.058+0000 7fc1a3fff700 1 --2- 172.21.15.60:0/388251458 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1a40ae270 0x7fc1a41500d0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 rx=0x7fc18c002f70 tx=0x7fc18c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:35.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.058+0000 7fc1a17fa700 1 -- 172.21.15.60:0/388251458 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc18c011760 con 0x7fc1a40ae270 2022-06-29T16:05:35.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.058+0000 7fc1a17fa700 1 -- 172.21.15.60:0/388251458 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc18c0118c0 con 0x7fc1a40ae270 2022-06-29T16:05:35.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.058+0000 7fc1aa698700 1 -- 172.21.15.60:0/388251458 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1a413d8e0 con 0x7fc1a40ae270 2022-06-29T16:05:35.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.058+0000 7fc1a17fa700 1 -- 172.21.15.60:0/388251458 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc18c011c20 con 0x7fc1a40ae270 2022-06-29T16:05:35.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.062+0000 7fc1aa698700 1 -- 172.21.15.60:0/388251458 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1a413e4d0 con 0x7fc1a40ae270 2022-06-29T16:05:35.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.062+0000 7fc1aa698700 1 -- 172.21.15.60:0/388251458 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1a4153790 con 0x7fc1a40ae270 2022-06-29T16:05:35.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.062+0000 7fc1a17fa700 1 -- 172.21.15.60:0/388251458 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc18c01a040 con 0x7fc1a40ae270 2022-06-29T16:05:35.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.062+0000 7fc1a17fa700 1 --2- 172.21.15.60:0/388251458 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc188053a70 0x7fc188055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:35.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.062+0000 7fc1a17fa700 1 -- 172.21.15.60:0/388251458 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5425+0+0 (secure 0 0 0) 0x7fc18c06ca50 con 0x7fc1a40ae270 2022-06-29T16:05:35.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.062+0000 7fc1a37fe700 1 --2- 172.21.15.60:0/388251458 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc188053a70 0x7fc188055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:35.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.066+0000 7fc1a37fe700 1 --2- 172.21.15.60:0/388251458 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc188053a70 0x7fc188055f30 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7fc19400d000 tx=0x7fc194007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:35.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.070+0000 7fc1a17fa700 1 -- 172.21.15.60:0/388251458 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc1a4153790 con 0x7fc1a40ae270 2022-06-29T16:05:35.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.230+0000 7fc1aa698700 1 -- 172.21.15.60:0/388251458 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1a4144910 con 0x7fc1a40ae270 2022-06-29T16:05:35.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.230+0000 7fc1a17fa700 1 -- 172.21.15.60:0/388251458 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+1209 (secure 0 0 0) 0x7fc18c023020 con 0x7fc1a40ae270 2022-06-29T16:05:35.238 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:35.238 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":38,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6657,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.60:6833/3121204394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3121204394},{"type":"v1","addr":"172.21.15.60:6833","nonce":3121204394}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":34},{"gid":6916,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/3365830848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3365830848},{"type":"v1","addr":"172.21.15.92:6835","nonce":3365830848}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":38}],"filesystems":[]} 2022-06-29T16:05:35.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.230+0000 7fc1aa698700 1 -- 172.21.15.60:0/388251458 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc188053a70 msgr2=0x7fc188055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:35.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.230+0000 7fc1aa698700 1 --2- 172.21.15.60:0/388251458 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc188053a70 0x7fc188055f30 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7fc19400d000 tx=0x7fc194007a00).stop 2022-06-29T16:05:35.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.230+0000 7fc1aa698700 1 -- 172.21.15.60:0/388251458 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1a40ae270 msgr2=0x7fc1a41500d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:35.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.230+0000 7fc1aa698700 1 --2- 172.21.15.60:0/388251458 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1a40ae270 0x7fc1a41500d0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 rx=0x7fc18c002f70 tx=0x7fc18c015000).stop 2022-06-29T16:05:35.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.230+0000 7fc1aa698700 1 -- 172.21.15.60:0/388251458 shutdown_connections 2022-06-29T16:05:35.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.230+0000 7fc1aa698700 1 --2- 172.21.15.60:0/388251458 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc188053a70 0x7fc188055f30 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:35.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.230+0000 7fc1aa698700 1 --2- 172.21.15.60:0/388251458 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1a4140ab0 0x7fc1a4153a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:35.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.230+0000 7fc1aa698700 1 --2- 172.21.15.60:0/388251458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1a413a920 0x7fc1a40b47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:35.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.230+0000 7fc1aa698700 1 --2- 172.21.15.60:0/388251458 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1a40ae270 0x7fc1a41500d0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:35.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.230+0000 7fc1aa698700 1 -- 172.21.15.60:0/388251458 >> 172.21.15.60:0/388251458 conn(0x7fc1a40632c0 msgr2=0x7fc1a40b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:35.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.230+0000 7fc1aa698700 1 -- 172.21.15.60:0/388251458 shutdown_connections 2022-06-29T16:05:35.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.230+0000 7fc1aa698700 1 -- 172.21.15.60:0/388251458 wait complete. 2022-06-29T16:05:35.243 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 38 2022-06-29T16:05:35.251 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:05:35.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.346+0000 7f1983ed4700 1 Processor -- start 2022-06-29T16:05:35.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.346+0000 7f1983ed4700 1 -- start start 2022-06-29T16:05:35.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.346+0000 7f1983ed4700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f197c0acbc0 0x7f197c14a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:35.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.346+0000 7f1983ed4700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f197c05cf40 0x7f197c065da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:35.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.346+0000 7f1983ed4700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f197c0ad9d0 con 0x7f197c0b24f0 2022-06-29T16:05:35.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.346+0000 7f1983ed4700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f197c0ad460 con 0x7f197c0acbc0 2022-06-29T16:05:35.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.346+0000 7f1983ed4700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f197c0adc00 con 0x7f197c05cf40 2022-06-29T16:05:35.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.346+0000 7f1981c70700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f197c0acbc0 0x7f197c14a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:35.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.346+0000 7f1981c70700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f197c0acbc0 0x7f197c14a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50514/0 (socket says 172.21.15.60:50514) 2022-06-29T16:05:35.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.346+0000 7f1981c70700 1 -- 172.21.15.60:0/405016718 learned_addr learned my addr 172.21.15.60:0/405016718 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:35.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.346+0000 7f1982471700 1 --2- 172.21.15.60:0/405016718 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f197c05cf40 0x7f197c065da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:35.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.346+0000 7f1981c70700 1 -- 172.21.15.60:0/405016718 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f197c05cf40 msgr2=0x7f197c065da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:35.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.346+0000 7f1981c70700 1 --2- 172.21.15.60:0/405016718 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f197c05cf40 0x7f197c065da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:35.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.346+0000 7f1981c70700 1 -- 172.21.15.60:0/405016718 >> v1:172.21.15.92:6789/0 conn(0x7f197c0b24f0 legacy=0x7f197c146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:35.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.346+0000 7f1981c70700 1 -- 172.21.15.60:0/405016718 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f197c0adea0 con 0x7f197c0acbc0 2022-06-29T16:05:35.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.346+0000 7f1980c6e700 1 -- 172.21.15.60:0/405016718 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 768600669 0 0) 0x7f197c0ad9d0 con 0x7f197c0b24f0 2022-06-29T16:05:35.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.346+0000 7f1981c70700 1 --2- 172.21.15.60:0/405016718 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f197c0acbc0 0x7f197c14a560 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 rx=0x7f1978002f70 tx=0x7f1978006eb0).ready entity=mon.0 client_cookie=301c6c5773a8cc71 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:35.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.346+0000 7f1980c6e700 1 -- 172.21.15.60:0/405016718 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f197800d570 con 0x7f197c0acbc0 2022-06-29T16:05:35.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.346+0000 7f1980c6e700 1 -- 172.21.15.60:0/405016718 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1978000e90 con 0x7f197c0acbc0 2022-06-29T16:05:35.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.346+0000 7f1980c6e700 1 -- 172.21.15.60:0/405016718 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f197800d8c0 con 0x7f197c0acbc0 2022-06-29T16:05:35.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.346+0000 7f1983ed4700 1 -- 172.21.15.60:0/405016718 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f197c0acbc0 msgr2=0x7f197c14a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:35.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.346+0000 7f1983ed4700 1 --2- 172.21.15.60:0/405016718 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f197c0acbc0 0x7f197c14a560 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 rx=0x7f1978002f70 tx=0x7f1978006eb0).stop 2022-06-29T16:05:35.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.346+0000 7f1983ed4700 1 -- 172.21.15.60:0/405016718 shutdown_connections 2022-06-29T16:05:35.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.346+0000 7f1983ed4700 1 --2- 172.21.15.60:0/405016718 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f197c05cf40 0x7f197c065da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:35.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.346+0000 7f1983ed4700 1 --2- 172.21.15.60:0/405016718 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f197c0acbc0 0x7f197c14a560 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:35.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.346+0000 7f1983ed4700 1 -- 172.21.15.60:0/405016718 >> 172.21.15.60:0/405016718 conn(0x7f197c0b7140 msgr2=0x7f197c0b7520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:35.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.346+0000 7f1983ed4700 1 -- 172.21.15.60:0/405016718 shutdown_connections 2022-06-29T16:05:35.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.346+0000 7f1983ed4700 1 -- 172.21.15.60:0/405016718 wait complete. 2022-06-29T16:05:35.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.350+0000 7f1983ed4700 1 Processor -- start 2022-06-29T16:05:35.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.350+0000 7f1983ed4700 1 -- start start 2022-06-29T16:05:35.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.350+0000 7f1983ed4700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f197c05cf40 0x7f197c0b3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:35.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.350+0000 7f1983ed4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f197c0acbc0 0x7f197c1596c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:35.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.350+0000 7f1983ed4700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f197c0b24f0 0x7f197c110960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:35.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.350+0000 7f1983ed4700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197c0ae720 con 0x7f197c0acbc0 2022-06-29T16:05:35.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.350+0000 7f1983ed4700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197c14e2a0 con 0x7f197c0b24f0 2022-06-29T16:05:35.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.350+0000 7f1983ed4700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f197c14e4d0 con 0x7f197c05cf40 2022-06-29T16:05:35.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.354+0000 7f198146f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f197c0acbc0 0x7f197c1596c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:35.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.354+0000 7f198146f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f197c0acbc0 0x7f197c1596c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50520/0 (socket says 172.21.15.60:50520) 2022-06-29T16:05:35.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.354+0000 7f198146f700 1 -- 172.21.15.60:0/3053370325 learned_addr learned my addr 172.21.15.60:0/3053370325 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:35.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.354+0000 7f1981c70700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f197c05cf40 0x7f197c0b3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:35.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.354+0000 7f1982471700 1 --2- 172.21.15.60:0/3053370325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f197c0b24f0 0x7f197c110960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:35.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.354+0000 7f198146f700 1 -- 172.21.15.60:0/3053370325 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f197c05cf40 msgr2=0x7f197c0b3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:35.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.354+0000 7f198146f700 1 --2- 172.21.15.60:0/3053370325 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f197c05cf40 0x7f197c0b3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:35.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.354+0000 7f198146f700 1 -- 172.21.15.60:0/3053370325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f197c0b24f0 msgr2=0x7f197c110960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:35.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.354+0000 7f198146f700 1 --2- 172.21.15.60:0/3053370325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f197c0b24f0 0x7f197c110960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:35.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.354+0000 7f198146f700 1 -- 172.21.15.60:0/3053370325 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f197800b1e0 con 0x7f197c0acbc0 2022-06-29T16:05:35.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.354+0000 7f198146f700 1 --2- 172.21.15.60:0/3053370325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f197c0acbc0 0x7f197c1596c0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 rx=0x7f1970000ed0 tx=0x7f1970007c20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:35.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.354+0000 7f196effd700 1 -- 172.21.15.60:0/3053370325 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1970013450 con 0x7f197c0acbc0 2022-06-29T16:05:35.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.354+0000 7f196effd700 1 -- 172.21.15.60:0/3053370325 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19700135b0 con 0x7f197c0acbc0 2022-06-29T16:05:35.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.354+0000 7f1983ed4700 1 -- 172.21.15.60:0/3053370325 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f197c14e770 con 0x7f197c0acbc0 2022-06-29T16:05:35.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.354+0000 7f196effd700 1 -- 172.21.15.60:0/3053370325 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1970013910 con 0x7f197c0acbc0 2022-06-29T16:05:35.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.354+0000 7f1983ed4700 1 -- 172.21.15.60:0/3053370325 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f197c0afe10 con 0x7f197c0acbc0 2022-06-29T16:05:35.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.354+0000 7f1983ed4700 1 -- 172.21.15.60:0/3053370325 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f197c0af170 con 0x7f197c0acbc0 2022-06-29T16:05:35.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.354+0000 7f196effd700 1 -- 172.21.15.60:0/3053370325 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f197001a060 con 0x7f197c0acbc0 2022-06-29T16:05:35.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.358+0000 7f196effd700 1 --2- 172.21.15.60:0/3053370325 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1964053b00 0x7f1964055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:35.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.358+0000 7f196effd700 1 -- 172.21.15.60:0/3053370325 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5425+0+0 (secure 0 0 0) 0x7f197006b360 con 0x7f197c0acbc0 2022-06-29T16:05:35.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.358+0000 7f1981c70700 1 --2- 172.21.15.60:0/3053370325 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1964053b00 0x7f1964055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:35.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.358+0000 7f1981c70700 1 --2- 172.21.15.60:0/3053370325 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1964053b00 0x7f1964055fc0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f197801c000 tx=0x7f1978015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:35.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.362+0000 7f196effd700 1 -- 172.21.15.60:0/3053370325 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f197c0af170 con 0x7f197c0acbc0 2022-06-29T16:05:35.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.526+0000 7f1983ed4700 1 -- 172.21.15.60:0/3053370325 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f197c0af170 con 0x7f197c0acbc0 2022-06-29T16:05:35.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.530+0000 7f196effd700 1 -- 172.21.15.60:0/3053370325 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+1209 (secure 0 0 0) 0x7f197c0af170 con 0x7f197c0acbc0 2022-06-29T16:05:35.537 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:35.537 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":38,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6657,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.60:6833/3121204394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3121204394},{"type":"v1","addr":"172.21.15.60:6833","nonce":3121204394}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":34},{"gid":6916,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/3365830848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3365830848},{"type":"v1","addr":"172.21.15.92:6835","nonce":3365830848}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":38}],"filesystems":[]} 2022-06-29T16:05:35.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.530+0000 7f1983ed4700 1 -- 172.21.15.60:0/3053370325 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1964053b00 msgr2=0x7f1964055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:35.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.530+0000 7f1983ed4700 1 --2- 172.21.15.60:0/3053370325 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1964053b00 0x7f1964055fc0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f197801c000 tx=0x7f1978015000).stop 2022-06-29T16:05:35.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.530+0000 7f1983ed4700 1 -- 172.21.15.60:0/3053370325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f197c0acbc0 msgr2=0x7f197c1596c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:35.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.530+0000 7f1983ed4700 1 --2- 172.21.15.60:0/3053370325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f197c0acbc0 0x7f197c1596c0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 rx=0x7f1970000ed0 tx=0x7f1970007c20).stop 2022-06-29T16:05:35.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.530+0000 7f1983ed4700 1 -- 172.21.15.60:0/3053370325 shutdown_connections 2022-06-29T16:05:35.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.530+0000 7f1983ed4700 1 --2- 172.21.15.60:0/3053370325 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1964053b00 0x7f1964055fc0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:35.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.530+0000 7f1983ed4700 1 --2- 172.21.15.60:0/3053370325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f197c0b24f0 0x7f197c110960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:35.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.530+0000 7f1983ed4700 1 --2- 172.21.15.60:0/3053370325 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f197c0acbc0 0x7f197c1596c0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:35.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.530+0000 7f1983ed4700 1 --2- 172.21.15.60:0/3053370325 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f197c05cf40 0x7f197c0b3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:35.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.530+0000 7f1983ed4700 1 -- 172.21.15.60:0/3053370325 >> 172.21.15.60:0/3053370325 conn(0x7f197c0b7140 msgr2=0x7f197c0b1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:35.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.530+0000 7f1983ed4700 1 -- 172.21.15.60:0/3053370325 shutdown_connections 2022-06-29T16:05:35.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.530+0000 7f1983ed4700 1 -- 172.21.15.60:0/3053370325 wait complete. 2022-06-29T16:05:35.543 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 38 2022-06-29T16:05:35.554 INFO:tasks.ceph.mds.a:Restarting daemon 2022-06-29T16:05:35.554 INFO:tasks.ceph.mds.a:Stopping old one... 2022-06-29T16:05:35.555 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-06-29T16:05:35.555 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T16:05:35.555 INFO:tasks.ceph.mds.b:Restarting daemon 2022-06-29T16:05:35.555 INFO:tasks.ceph.mds.b:Stopping old one... 2022-06-29T16:05:35.556 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-06-29T16:05:35.556 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T16:05:35.571 INFO:tasks.ceph.mds.a:Stopped 2022-06-29T16:05:35.571 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-06-29T16:05:35.573 INFO:tasks.ceph.mds.b:Stopped 2022-06-29T16:05:35.573 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-06-29T16:05:35.575 INFO:tasks.ceph.mds.a:Started 2022-06-29T16:05:35.576 DEBUG:teuthology.parallel:result is None 2022-06-29T16:05:35.576 INFO:tasks.ceph.mds.b:Started 2022-06-29T16:05:35.577 DEBUG:teuthology.parallel:result is None 2022-06-29T16:05:35.578 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-06-29T16:05:35.649 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2022-06-29T16:05:35.650 INFO:tasks.ceph.mds.b.smithi092.stdout:starting mds.b at 2022-06-29T16:05:35.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.662+0000 7f80b919d700 1 Processor -- start 2022-06-29T16:05:35.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.662+0000 7f80b919d700 1 -- start start 2022-06-29T16:05:35.673 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.662+0000 7f80b919d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f80b40b6a50 0x7f80b41461e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:35.673 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.662+0000 7f80b919d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f80b405f0b0 0x7f80b4142310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:35.674 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.666+0000 7f80b919d700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f80b4142800 0x7f80b405ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:35.674 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.666+0000 7f80b919d700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80b40b6e30 con 0x7f80b405f0b0 2022-06-29T16:05:35.674 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.666+0000 7f80b919d700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80b40b0dd0 con 0x7f80b4142800 2022-06-29T16:05:35.674 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.666+0000 7f80b919d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80b40b1070 con 0x7f80b40b6a50 2022-06-29T16:05:35.675 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.666+0000 7f80b259c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f80b405f0b0 0x7f80b4142310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:35.675 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.666+0000 7f80b259c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f80b405f0b0 0x7f80b4142310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50544/0 (socket says 172.21.15.60:50544) 2022-06-29T16:05:35.675 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.666+0000 7f80b259c700 1 -- 172.21.15.60:0/349908309 learned_addr learned my addr 172.21.15.60:0/349908309 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:35.675 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.666+0000 7f80b359e700 1 --2- 172.21.15.60:0/349908309 >> v2:172.21.15.92:3300/0 conn(0x7f80b4142800 0x7f80b405ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:35.676 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.666+0000 7f80b2d9d700 1 --2- 172.21.15.60:0/349908309 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f80b40b6a50 0x7f80b41461e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:35.676 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.666+0000 7f80b259c700 1 -- 172.21.15.60:0/349908309 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f80b40b6a50 msgr2=0x7f80b41461e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:35.676 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.666+0000 7f80b259c700 1 --2- 172.21.15.60:0/349908309 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f80b40b6a50 0x7f80b41461e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:35.676 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.666+0000 7f80b259c700 1 -- 172.21.15.60:0/349908309 >> v2:172.21.15.92:3300/0 conn(0x7f80b4142800 msgr2=0x7f80b405ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:35.677 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.666+0000 7f80b259c700 1 --2- 172.21.15.60:0/349908309 >> v2:172.21.15.92:3300/0 conn(0x7f80b4142800 0x7f80b405ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:35.677 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.666+0000 7f80b259c700 1 -- 172.21.15.60:0/349908309 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b405c5b0 con 0x7f80b405f0b0 2022-06-29T16:05:35.677 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.666+0000 7f80b359e700 1 --2- 172.21.15.60:0/349908309 >> v2:172.21.15.92:3300/0 conn(0x7f80b4142800 0x7f80b405ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:05:35.677 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.666+0000 7f80b259c700 1 --2- 172.21.15.60:0/349908309 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80b405f0b0 0x7f80b4142310 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 rx=0x7f80a8002f70 tx=0x7f80a8006eb0).ready entity=mon.0 client_cookie=c20862d6e103ad05 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:35.678 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.666+0000 7f80b1d9b700 1 -- 172.21.15.60:0/349908309 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80a800d570 con 0x7f80b405f0b0 2022-06-29T16:05:35.678 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.666+0000 7f80b1d9b700 1 -- 172.21.15.60:0/349908309 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a8000e90 con 0x7f80b405f0b0 2022-06-29T16:05:35.678 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.666+0000 7f80b1d9b700 1 -- 172.21.15.60:0/349908309 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80a800d900 con 0x7f80b405f0b0 2022-06-29T16:05:35.682 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.666+0000 7f80b919d700 1 -- 172.21.15.60:0/349908309 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80b405f0b0 msgr2=0x7f80b4142310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:35.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.666+0000 7f80b919d700 1 --2- 172.21.15.60:0/349908309 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80b405f0b0 0x7f80b4142310 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 rx=0x7f80a8002f70 tx=0x7f80a8006eb0).stop 2022-06-29T16:05:35.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.666+0000 7f80b919d700 1 -- 172.21.15.60:0/349908309 shutdown_connections 2022-06-29T16:05:35.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.666+0000 7f80b919d700 1 --2- 172.21.15.60:0/349908309 >> v2:172.21.15.92:3300/0 conn(0x7f80b4142800 0x7f80b405ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:35.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.666+0000 7f80b919d700 1 --2- 172.21.15.60:0/349908309 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80b405f0b0 0x7f80b4142310 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:35.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.666+0000 7f80b919d700 1 --2- 172.21.15.60:0/349908309 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f80b40b6a50 0x7f80b41461e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:35.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.666+0000 7f80b919d700 1 -- 172.21.15.60:0/349908309 >> 172.21.15.60:0/349908309 conn(0x7f80b40643e0 msgr2=0x7f80b40b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:35.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.666+0000 7f80b919d700 1 -- 172.21.15.60:0/349908309 shutdown_connections 2022-06-29T16:05:35.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.666+0000 7f80b919d700 1 -- 172.21.15.60:0/349908309 wait complete. 2022-06-29T16:05:35.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.670+0000 7f80b919d700 1 Processor -- start 2022-06-29T16:05:35.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.670+0000 7f80b919d700 1 -- start start 2022-06-29T16:05:35.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.670+0000 7f80b919d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f80b405f0b0 0x7f80b4066e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:35.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.670+0000 7f80b919d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80b40b6a50 0x7f80b41405d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:35.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.670+0000 7f80b919d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80b4142800 0x7f80b4099fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:35.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.670+0000 7f80b919d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b4099270 con 0x7f80b4142800 2022-06-29T16:05:35.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.670+0000 7f80b919d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b40660b0 con 0x7f80b40b6a50 2022-06-29T16:05:35.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.670+0000 7f80b919d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80b4065e10 con 0x7f80b405f0b0 2022-06-29T16:05:35.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.670+0000 7f80b359e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80b4142800 0x7f80b4099fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:35.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.670+0000 7f80b359e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80b4142800 0x7f80b4099fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50554/0 (socket says 172.21.15.60:50554) 2022-06-29T16:05:35.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.670+0000 7f80b359e700 1 -- 172.21.15.60:0/3780035828 learned_addr learned my addr 172.21.15.60:0/3780035828 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:35.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.670+0000 7f80b259c700 1 --2- 172.21.15.60:0/3780035828 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80b40b6a50 0x7f80b41405d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:35.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.670+0000 7f80b2d9d700 1 --2- 172.21.15.60:0/3780035828 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f80b405f0b0 0x7f80b4066e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:35.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.670+0000 7f80b259c700 1 -- 172.21.15.60:0/3780035828 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f80b405f0b0 msgr2=0x7f80b4066e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:35.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.670+0000 7f80b259c700 1 --2- 172.21.15.60:0/3780035828 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f80b405f0b0 0x7f80b4066e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:35.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.670+0000 7f80b259c700 1 -- 172.21.15.60:0/3780035828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80b4142800 msgr2=0x7f80b4099fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:35.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.670+0000 7f80b259c700 1 --2- 172.21.15.60:0/3780035828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80b4142800 0x7f80b4099fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:35.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.670+0000 7f80b259c700 1 -- 172.21.15.60:0/3780035828 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ac009470 con 0x7f80b40b6a50 2022-06-29T16:05:35.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.670+0000 7f80b359e700 1 --2- 172.21.15.60:0/3780035828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80b4142800 0x7f80b4099fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:05:35.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.670+0000 7f80b259c700 1 --2- 172.21.15.60:0/3780035828 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80b40b6a50 0x7f80b41405d0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f80a8002f70 tx=0x7f80a8015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:35.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.670+0000 7f80a3fff700 1 -- 172.21.15.60:0/3780035828 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80a80116e0 con 0x7f80b40b6a50 2022-06-29T16:05:35.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.670+0000 7f80a3fff700 1 -- 172.21.15.60:0/3780035828 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a8011840 con 0x7f80b40b6a50 2022-06-29T16:05:35.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.670+0000 7f80b919d700 1 -- 172.21.15.60:0/3780035828 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80a800b1e0 con 0x7f80b40b6a50 2022-06-29T16:05:35.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.670+0000 7f80a3fff700 1 -- 172.21.15.60:0/3780035828 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80a8011ba0 con 0x7f80b40b6a50 2022-06-29T16:05:35.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.670+0000 7f80b919d700 1 -- 172.21.15.60:0/3780035828 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80b40651a0 con 0x7f80b40b6a50 2022-06-29T16:05:35.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.674+0000 7f80b919d700 1 -- 172.21.15.60:0/3780035828 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80b40573d0 con 0x7f80b40b6a50 2022-06-29T16:05:35.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.674+0000 7f80a3fff700 1 -- 172.21.15.60:0/3780035828 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f80a801a040 con 0x7f80b40b6a50 2022-06-29T16:05:35.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.674+0000 7f80a3fff700 1 --2- 172.21.15.60:0/3780035828 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8090053b30 0x7f8090055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:35.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.674+0000 7f80a3fff700 1 -- 172.21.15.60:0/3780035828 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5425+0+0 (secure 0 0 0) 0x7f80a806ca60 con 0x7f80b40b6a50 2022-06-29T16:05:35.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.674+0000 7f80b2d9d700 1 --2- 172.21.15.60:0/3780035828 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8090053b30 0x7f8090055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:35.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.674+0000 7f80b2d9d700 1 --2- 172.21.15.60:0/3780035828 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8090053b30 0x7f8090055ff0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f80a400b000 tx=0x7f80a4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:35.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.682+0000 7f80a3fff700 1 -- 172.21.15.60:0/3780035828 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f80a8043b60 con 0x7f80b40b6a50 2022-06-29T16:05:35.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:35.818+0000 7f80b919d700 1 -- 172.21.15.60:0/3780035828 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f80b4067990 con 0x7f80b40b6a50 2022-06-29T16:05:36.680 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.674+0000 7f80a3fff700 1 -- 172.21.15.60:0/3780035828 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v87) v1 ==== 97+0+0 (secure 0 0 0) 0x7f80a8047180 con 0x7f80b40b6a50 2022-06-29T16:05:36.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.674+0000 7f80b919d700 1 -- 172.21.15.60:0/3780035828 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8090053b30 msgr2=0x7f8090055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:36.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.674+0000 7f80b919d700 1 --2- 172.21.15.60:0/3780035828 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8090053b30 0x7f8090055ff0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f80a400b000 tx=0x7f80a4006df0).stop 2022-06-29T16:05:36.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.674+0000 7f80b919d700 1 -- 172.21.15.60:0/3780035828 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80b40b6a50 msgr2=0x7f80b41405d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:36.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.674+0000 7f80b919d700 1 --2- 172.21.15.60:0/3780035828 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80b40b6a50 0x7f80b41405d0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f80a8002f70 tx=0x7f80a8015000).stop 2022-06-29T16:05:36.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.674+0000 7f80b919d700 1 -- 172.21.15.60:0/3780035828 shutdown_connections 2022-06-29T16:05:36.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.674+0000 7f80b919d700 1 --2- 172.21.15.60:0/3780035828 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8090053b30 0x7f8090055ff0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:36.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.674+0000 7f80b919d700 1 --2- 172.21.15.60:0/3780035828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f80b4142800 0x7f80b4099fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:36.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.674+0000 7f80b919d700 1 --2- 172.21.15.60:0/3780035828 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80b40b6a50 0x7f80b41405d0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:36.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.674+0000 7f80b919d700 1 --2- 172.21.15.60:0/3780035828 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f80b405f0b0 0x7f80b4066e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:36.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.674+0000 7f80b919d700 1 -- 172.21.15.60:0/3780035828 >> 172.21.15.60:0/3780035828 conn(0x7f80b40643e0 msgr2=0x7f80b4143370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:36.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.674+0000 7f80b919d700 1 -- 172.21.15.60:0/3780035828 shutdown_connections 2022-06-29T16:05:36.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.674+0000 7f80b919d700 1 -- 172.21.15.60:0/3780035828 wait complete. 2022-06-29T16:05:36.687 INFO:teuthology.orchestra.run.smithi060.stderr: removed all blacklist entries 2022-06-29T16:05:36.698 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-06-29T16:05:36.698 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-06-29T16:05:36.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.806+0000 7f3d20b89700 1 Processor -- start 2022-06-29T16:05:36.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.806+0000 7f3d20b89700 1 -- start start 2022-06-29T16:05:36.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.806+0000 7f3d20b89700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3d1c0b28e0 0x7f3d1c0b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:36.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.806+0000 7f3d20b89700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d1c13fa40 con 0x7f3d1c060760 2022-06-29T16:05:36.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.806+0000 7f3d20b89700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d1c13fce0 con 0x7f3d1c065450 2022-06-29T16:05:36.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.806+0000 7f3d20b89700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d1c140860 con 0x7f3d1c0b28e0 2022-06-29T16:05:36.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.806+0000 7f3d1bfff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f3d1c060760 0x7f3d1c066390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58650/0 (socket says 172.21.15.60:58650) 2022-06-29T16:05:36.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.806+0000 7f3d1bfff700 1 -- 172.21.15.60:0/2732938910 learned_addr learned my addr 172.21.15.60:0/2732938910 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:36.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.806+0000 7f3d1b7fe700 1 --2- 172.21.15.60:0/2732938910 >> v2:172.21.15.92:3300/0 conn(0x7f3d1c0b28e0 0x7f3d1c0b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:36.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.806+0000 7f3d1a7fc700 1 -- 172.21.15.60:0/2732938910 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2639919288 0 0) 0x7f3d1c13fa40 con 0x7f3d1c060760 2022-06-29T16:05:36.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.806+0000 7f3d1a7fc700 1 -- 172.21.15.60:0/2732938910 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cfc003260 con 0x7f3d1c060760 2022-06-29T16:05:36.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.806+0000 7f3d1a7fc700 1 -- 172.21.15.60:0/2732938910 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2917393709 0 0) 0x7f3cfc003260 con 0x7f3d1c060760 2022-06-29T16:05:36.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.806+0000 7f3d1a7fc700 1 -- 172.21.15.60:0/2732938910 >> v2:172.21.15.92:3300/0 conn(0x7f3d1c0b28e0 msgr2=0x7f3d1c0b2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:36.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.806+0000 7f3d1a7fc700 1 --2- 172.21.15.60:0/2732938910 >> v2:172.21.15.92:3300/0 conn(0x7f3d1c0b28e0 0x7f3d1c0b2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:36.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.806+0000 7f3d1a7fc700 1 -- 172.21.15.60:0/2732938910 >> v1:172.21.15.92:6789/0 conn(0x7f3d1c065450 legacy=0x7f3d1c139340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:36.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.806+0000 7f3d1a7fc700 1 -- 172.21.15.60:0/2732938910 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1c13f6f0 con 0x7f3d1c060760 2022-06-29T16:05:36.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.806+0000 7f3d1a7fc700 1 -- 172.21.15.60:0/2732938910 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f3d14001660 con 0x7f3d1c060760 2022-06-29T16:05:36.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.806+0000 7f3d1a7fc700 1 -- 172.21.15.60:0/2732938910 >> v1:172.21.15.60:6789/0 conn(0x7f3d1c060760 legacy=0x7f3d1c066390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:36.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.806+0000 7f3d1a7fc700 1 --2- 172.21.15.60:0/2732938910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cfc003750 0x7f3cfc003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:36.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.806+0000 7f3d1a7fc700 1 --2- 172.21.15.60:0/2732938910 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3cfc004500 0x7f3cfc004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:36.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.806+0000 7f3d1a7fc700 1 --2- 172.21.15.60:0/2732938910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cfc004e70 0x7f3cfc00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:36.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.806+0000 7f3d1a7fc700 1 -- 172.21.15.60:0/2732938910 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c140860 con 0x7f3cfc004e70 2022-06-29T16:05:36.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.806+0000 7f3d1a7fc700 1 -- 172.21.15.60:0/2732938910 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc00bd20 con 0x7f3cfc003750 2022-06-29T16:05:36.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.806+0000 7f3d1a7fc700 1 -- 172.21.15.60:0/2732938910 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc00bee0 con 0x7f3cfc004500 2022-06-29T16:05:36.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.806+0000 7f3d1bfff700 1 --2- 172.21.15.60:0/2732938910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cfc004e70 0x7f3cfc00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:36.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.806+0000 7f3d1b7fe700 1 --2- 172.21.15.60:0/2732938910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cfc003750 0x7f3cfc003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:36.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.806+0000 7f3d1affd700 1 --2- 172.21.15.60:0/2732938910 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3cfc004500 0x7f3cfc004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:36.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.806+0000 7f3d1bfff700 1 -- 172.21.15.60:0/2732938910 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3cfc004500 msgr2=0x7f3cfc004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:36.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.806+0000 7f3d1bfff700 1 --2- 172.21.15.60:0/2732938910 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3cfc004500 0x7f3cfc004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:36.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.806+0000 7f3d1bfff700 1 -- 172.21.15.60:0/2732938910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cfc003750 msgr2=0x7f3cfc003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:36.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.806+0000 7f3d1bfff700 1 --2- 172.21.15.60:0/2732938910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cfc003750 0x7f3cfc003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:36.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.806+0000 7f3d1bfff700 1 -- 172.21.15.60:0/2732938910 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cfc003260 con 0x7f3cfc004e70 2022-06-29T16:05:36.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.810+0000 7f3d1bfff700 1 -- 172.21.15.60:0/2732938910 reap_dead start 2022-06-29T16:05:36.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.810+0000 7f3d1bfff700 1 --2- 172.21.15.60:0/2732938910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cfc004e70 0x7f3cfc00b750 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 rx=0x7f3d14001370 tx=0x7f3d14001e10).ready entity=mon.0 client_cookie=b4e15820193e9ffa server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:36.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.810+0000 7f3d1a7fc700 1 -- 172.21.15.60:0/2732938910 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d140071a0 con 0x7f3cfc004e70 2022-06-29T16:05:36.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.810+0000 7f3d1a7fc700 1 -- 172.21.15.60:0/2732938910 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d14007300 con 0x7f3cfc004e70 2022-06-29T16:05:36.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.810+0000 7f3d1a7fc700 1 -- 172.21.15.60:0/2732938910 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d1400c8f0 con 0x7f3cfc004e70 2022-06-29T16:05:36.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.810+0000 7f3d20b89700 1 -- 172.21.15.60:0/2732938910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cfc004e70 msgr2=0x7f3cfc00b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:36.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.810+0000 7f3d20b89700 1 --2- 172.21.15.60:0/2732938910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cfc004e70 0x7f3cfc00b750 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 rx=0x7f3d14001370 tx=0x7f3d14001e10).stop 2022-06-29T16:05:36.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.810+0000 7f3d20b89700 1 -- 172.21.15.60:0/2732938910 shutdown_connections 2022-06-29T16:05:36.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.810+0000 7f3d20b89700 1 --2- 172.21.15.60:0/2732938910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3cfc004e70 0x7f3cfc00b750 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:36.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.810+0000 7f3d20b89700 1 -- 172.21.15.60:0/2732938910 >> 172.21.15.60:0/2732938910 conn(0x7f3d1c061c80 msgr2=0x7f3d1c0637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:36.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.810+0000 7f3d20b89700 1 -- 172.21.15.60:0/2732938910 shutdown_connections 2022-06-29T16:05:36.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.810+0000 7f3d20b89700 1 -- 172.21.15.60:0/2732938910 wait complete. 2022-06-29T16:05:36.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.810+0000 7f3d20b89700 1 Processor -- start 2022-06-29T16:05:36.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.810+0000 7f3d20b89700 1 -- start start 2022-06-29T16:05:36.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.814+0000 7f3d20b89700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cfc004e70 0x7f3d1c143b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:36.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.814+0000 7f3d20b89700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3d1c0afa60 0x7f3d1c0b58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:36.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.814+0000 7f3d20b89700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3d1c150af0 0x7f3d1c150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:36.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.814+0000 7f3d20b89700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c0afea0 con 0x7f3d1c0afa60 2022-06-29T16:05:36.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.814+0000 7f3d20b89700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c143130 con 0x7f3cfc004e70 2022-06-29T16:05:36.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.814+0000 7f3d20b89700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c1433d0 con 0x7f3d1c150af0 2022-06-29T16:05:36.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.814+0000 7f3d1affd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3d1c0afa60 0x7f3d1c0b58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:36.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.814+0000 7f3d1affd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3d1c0afa60 0x7f3d1c0b58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50574/0 (socket says 172.21.15.60:50574) 2022-06-29T16:05:36.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.814+0000 7f3d1affd700 1 -- 172.21.15.60:0/3990561677 learned_addr learned my addr 172.21.15.60:0/3990561677 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:36.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.814+0000 7f3d1b7fe700 1 --2- 172.21.15.60:0/3990561677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cfc004e70 0x7f3d1c143b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:36.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.814+0000 7f3d1bfff700 1 --2- 172.21.15.60:0/3990561677 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3d1c150af0 0x7f3d1c150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:36.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.814+0000 7f3d1affd700 1 -- 172.21.15.60:0/3990561677 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3d1c150af0 msgr2=0x7f3d1c150fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:36.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.814+0000 7f3d1affd700 1 --2- 172.21.15.60:0/3990561677 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3d1c150af0 0x7f3d1c150fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:36.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.814+0000 7f3d1affd700 1 -- 172.21.15.60:0/3990561677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cfc004e70 msgr2=0x7f3d1c143b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:36.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.814+0000 7f3d1affd700 1 --2- 172.21.15.60:0/3990561677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cfc004e70 0x7f3d1c143b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:36.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.814+0000 7f3d1affd700 1 -- 172.21.15.60:0/3990561677 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1c143670 con 0x7f3d1c0afa60 2022-06-29T16:05:36.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.814+0000 7f3d1b7fe700 1 --2- 172.21.15.60:0/3990561677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cfc004e70 0x7f3d1c143b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:36.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.814+0000 7f3d1bfff700 1 --2- 172.21.15.60:0/3990561677 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3d1c150af0 0x7f3d1c150fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:05:36.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.814+0000 7f3d1affd700 1 --2- 172.21.15.60:0/3990561677 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3d1c0afa60 0x7f3d1c0b58f0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 rx=0x7f3d0c00d000 tx=0x7f3d0c007d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:36.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.814+0000 7f3d18ff9700 1 -- 172.21.15.60:0/3990561677 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d0c012780 con 0x7f3d1c0afa60 2022-06-29T16:05:36.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.814+0000 7f3d20b89700 1 -- 172.21.15.60:0/3990561677 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d1c151490 con 0x7f3d1c0afa60 2022-06-29T16:05:36.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.814+0000 7f3d18ff9700 1 -- 172.21.15.60:0/3990561677 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d0c0128e0 con 0x7f3d1c0afa60 2022-06-29T16:05:36.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.814+0000 7f3d18ff9700 1 -- 172.21.15.60:0/3990561677 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d0c012c40 con 0x7f3d1c0afa60 2022-06-29T16:05:36.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.814+0000 7f3d20b89700 1 -- 172.21.15.60:0/3990561677 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d1c0b5de0 con 0x7f3d1c0afa60 2022-06-29T16:05:36.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.814+0000 7f3d20b89700 1 -- 172.21.15.60:0/3990561677 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d1c143800 con 0x7f3d1c0afa60 2022-06-29T16:05:36.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.818+0000 7f3d18ff9700 1 -- 172.21.15.60:0/3990561677 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3d0c01d460 con 0x7f3d1c0afa60 2022-06-29T16:05:36.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.818+0000 7f3d18ff9700 1 --2- 172.21.15.60:0/3990561677 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3d00053a70 0x7f3d00055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:36.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.818+0000 7f3d18ff9700 1 -- 172.21.15.60:0/3990561677 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f3d0c06c2e0 con 0x7f3d1c0afa60 2022-06-29T16:05:36.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.818+0000 7f3d1b7fe700 1 --2- 172.21.15.60:0/3990561677 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3d00053a70 0x7f3d00055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:36.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.818+0000 7f3d1b7fe700 1 --2- 172.21.15.60:0/3990561677 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3d00053a70 0x7f3d00055f30 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f3d0400b000 tx=0x7f3d04008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:36.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.822+0000 7f3d18ff9700 1 -- 172.21.15.60:0/3990561677 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3d0c043430 con 0x7f3d1c0afa60 2022-06-29T16:05:36.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.990+0000 7f3d20b89700 1 -- 172.21.15.60:0/3990561677 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f3d1c0573d0 con 0x7f3d1c0afa60 2022-06-29T16:05:36.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.990+0000 7f3d18ff9700 1 -- 172.21.15.60:0/3990561677 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v30) v1 ==== 80+0+4242 (secure 0 0 0) 0x7f3d0c046a50 con 0x7f3d1c0afa60 2022-06-29T16:05:37.000 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:37.000 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:05:37.000 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_dump": [ 2022-06-29T16:05:37.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:37.001 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.a", 2022-06-29T16:05:37.001 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxivv7JNhAAnA291UxlB/SfgITOipP00A==", 2022-06-29T16:05:37.001 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:05:37.001 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:05:37.002 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:05:37.002 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:05:37.002 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:05:37.002 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:37.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:37.003 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:37.003 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.b", 2022-06-29T16:05:37.003 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxiRSr/OhAAfFQTF2YHvzCv5wXBVCA5WQ==", 2022-06-29T16:05:37.004 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:05:37.004 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:05:37.004 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:05:37.004 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:05:37.005 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:05:37.005 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:37.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:37.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:37.006 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.0", 2022-06-29T16:05:37.006 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiZYPxOBAAIkweWJlTJIXgxnlagkGdXA==", 2022-06-29T16:05:37.006 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:05:37.006 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:05:37.007 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:05:37.007 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:05:37.007 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:37.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:37.007 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:37.008 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.1", 2022-06-29T16:05:37.008 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxiS2JGBhAAJn0UfAxZI3IVFyD5QBdb/Q==", 2022-06-29T16:05:37.008 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:05:37.008 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:05:37.009 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:05:37.009 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:05:37.009 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:37.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:37.010 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:37.010 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.2", 2022-06-29T16:05:37.010 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxiPhkfEhAApH7r8OUJzW7fZNM6XVv1dg==", 2022-06-29T16:05:37.010 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:05:37.011 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:05:37.011 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:05:37.011 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:05:37.011 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:37.012 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:37.012 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:37.012 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.3", 2022-06-29T16:05:37.012 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxit/47HhAA1vyU6h7hjPG+op9qP/rVYQ==", 2022-06-29T16:05:37.012 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:05:37.013 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:05:37.013 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:05:37.013 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:05:37.013 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:37.014 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:37.014 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:37.014 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.4", 2022-06-29T16:05:37.014 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxiaKzKBxAAbIwCF9ag7ezsNy11hi0BEw==", 2022-06-29T16:05:37.015 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:05:37.015 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:05:37.015 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:05:37.015 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:05:37.016 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:37.016 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:37.016 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:37.016 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.5", 2022-06-29T16:05:37.016 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxiJQV/IBAASBBDb6UFgEtXILe4fcT/xA==", 2022-06-29T16:05:37.017 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:05:37.017 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:05:37.017 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:05:37.017 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:05:37.018 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:37.018 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:37.018 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:37.018 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.6", 2022-06-29T16:05:37.019 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxi6D07OBAA2FWiZ+mNK+jFMp/kPZnbQQ==", 2022-06-29T16:05:37.019 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:05:37.019 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:05:37.019 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:05:37.020 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:05:37.020 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:37.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:37.020 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:37.021 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.7", 2022-06-29T16:05:37.021 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCMdrxinEOcEhAA562ybpzGHNQfWIq8q7zw9w==", 2022-06-29T16:05:37.021 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:05:37.021 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:05:37.021 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:05:37.022 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:05:37.022 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:37.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:37.022 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:37.023 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.0", 2022-06-29T16:05:37.023 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiHXPwBRAAkOOwlYJvIW7dqVFdlXXoqw==", 2022-06-29T16:05:37.023 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:05:37.023 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:05:37.024 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2022-06-29T16:05:37.024 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw pool=cephfs_data" 2022-06-29T16:05:37.024 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:37.024 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:37.024 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:37.025 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.1", 2022-06-29T16:05:37.025 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxibe5JCRAAGBJbi9t09w0836JqrAT7+w==", 2022-06-29T16:05:37.025 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:05:37.025 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:05:37.026 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2022-06-29T16:05:37.026 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw pool=cephfs_data" 2022-06-29T16:05:37.026 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:37.026 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:37.026 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:37.027 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.2", 2022-06-29T16:05:37.027 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiK3dBDRAA7XINPSOPQDfOrUYEpHC+gw==", 2022-06-29T16:05:37.027 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:05:37.027 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:05:37.028 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2022-06-29T16:05:37.028 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw pool=cephfs_data" 2022-06-29T16:05:37.028 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:37.028 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:37.029 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:37.029 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.admin", 2022-06-29T16:05:37.029 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxieKVcGBAAnjH80a0qWIArz2qwjYHWdQ==", 2022-06-29T16:05:37.029 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:05:37.029 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T16:05:37.030 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:05:37.030 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:05:37.030 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:05:37.031 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:37.031 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:37.031 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:37.031 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.x", 2022-06-29T16:05:37.031 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxi2LZJNBAA3MKb/pXYbQCsoRx8EQfGbw==", 2022-06-29T16:05:37.032 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:05:37.032 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T16:05:37.032 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2022-06-29T16:05:37.032 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:05:37.033 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:37.033 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:37.033 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:37.033 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.y", 2022-06-29T16:05:37.033 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxi3LmaMBAAN+0mZrdKXqSXOipVTA1drw==", 2022-06-29T16:05:37.034 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:05:37.034 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T16:05:37.034 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2022-06-29T16:05:37.034 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:05:37.035 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:37.035 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:37.035 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:37.035 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:05:37.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.994+0000 7f3d20b89700 1 -- 172.21.15.60:0/3990561677 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3d00053a70 msgr2=0x7f3d00055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:37.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.994+0000 7f3d20b89700 1 --2- 172.21.15.60:0/3990561677 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3d00053a70 0x7f3d00055f30 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f3d0400b000 tx=0x7f3d04008040).stop 2022-06-29T16:05:37.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.994+0000 7f3d20b89700 1 -- 172.21.15.60:0/3990561677 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3d1c0afa60 msgr2=0x7f3d1c0b58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:37.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.994+0000 7f3d20b89700 1 --2- 172.21.15.60:0/3990561677 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3d1c0afa60 0x7f3d1c0b58f0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 rx=0x7f3d0c00d000 tx=0x7f3d0c007d80).stop 2022-06-29T16:05:37.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.994+0000 7f3d20b89700 1 -- 172.21.15.60:0/3990561677 shutdown_connections 2022-06-29T16:05:37.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.994+0000 7f3d20b89700 1 --2- 172.21.15.60:0/3990561677 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3d00053a70 0x7f3d00055f30 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.994+0000 7f3d20b89700 1 --2- 172.21.15.60:0/3990561677 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3d1c150af0 0x7f3d1c150fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.994+0000 7f3d20b89700 1 --2- 172.21.15.60:0/3990561677 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3d1c0afa60 0x7f3d1c0b58f0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.994+0000 7f3d20b89700 1 --2- 172.21.15.60:0/3990561677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cfc004e70 0x7f3d1c143b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.994+0000 7f3d20b89700 1 -- 172.21.15.60:0/3990561677 >> 172.21.15.60:0/3990561677 conn(0x7f3d1c061c80 msgr2=0x7f3d1c0ad4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:37.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.994+0000 7f3d20b89700 1 -- 172.21.15.60:0/3990561677 shutdown_connections 2022-06-29T16:05:37.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:36.994+0000 7f3d20b89700 1 -- 172.21.15.60:0/3990561677 wait complete. 2022-06-29T16:05:37.041 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:05:37.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.138+0000 7f56353e6700 1 Processor -- start 2022-06-29T16:05:37.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.138+0000 7f56353e6700 1 -- start start 2022-06-29T16:05:37.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.142+0000 7f56353e6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5630065410 0x7f56300b4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:37.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.142+0000 7f56353e6700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5630143fd0 0x7f5630144410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:37.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.142+0000 7f56353e6700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56300657f0 con 0x7f563014a570 2022-06-29T16:05:37.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.142+0000 7f56353e6700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5630147da0 con 0x7f5630143fd0 2022-06-29T16:05:37.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.142+0000 7f56353e6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5630063b40 con 0x7f5630065410 2022-06-29T16:05:37.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.142+0000 7f5627fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5630143fd0 0x7f5630144410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:37.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.142+0000 7f5627fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5630143fd0 0x7f5630144410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50580/0 (socket says 172.21.15.60:50580) 2022-06-29T16:05:37.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.142+0000 7f5627fff700 1 -- 172.21.15.60:0/2139417571 learned_addr learned my addr 172.21.15.60:0/2139417571 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:37.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.142+0000 7f562effd700 1 --2- 172.21.15.60:0/2139417571 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5630065410 0x7f56300b4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:37.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.142+0000 7f562e7fc700 1 -- 172.21.15.60:0/2139417571 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2556804393 0 0) 0x7f56300657f0 con 0x7f563014a570 2022-06-29T16:05:37.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.142+0000 7f562e7fc700 1 -- 172.21.15.60:0/2139417571 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5610003260 con 0x7f563014a570 2022-06-29T16:05:37.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.142+0000 7f5627fff700 1 -- 172.21.15.60:0/2139417571 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5630065410 msgr2=0x7f56300b4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:37.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.142+0000 7f5627fff700 1 --2- 172.21.15.60:0/2139417571 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5630065410 0x7f56300b4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.142+0000 7f5627fff700 1 -- 172.21.15.60:0/2139417571 >> v1:172.21.15.92:6789/0 conn(0x7f563014a570 legacy=0x7f563014a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:37.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.142+0000 7f5627fff700 1 -- 172.21.15.60:0/2139417571 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5630144900 con 0x7f5630143fd0 2022-06-29T16:05:37.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.142+0000 7f562effd700 1 --2- 172.21.15.60:0/2139417571 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5630065410 0x7f56300b4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:05:37.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.142+0000 7f5627fff700 1 --2- 172.21.15.60:0/2139417571 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5630143fd0 0x7f5630144410 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 rx=0x7f5618002f70 tx=0x7f5618006eb0).ready entity=mon.0 client_cookie=b47575ab452d8c5f server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:37.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.142+0000 7f562e7fc700 1 -- 172.21.15.60:0/2139417571 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f561800d5c0 con 0x7f5630143fd0 2022-06-29T16:05:37.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.142+0000 7f562e7fc700 1 -- 172.21.15.60:0/2139417571 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5618000e90 con 0x7f5630143fd0 2022-06-29T16:05:37.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.142+0000 7f562e7fc700 1 -- 172.21.15.60:0/2139417571 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f561800d950 con 0x7f5630143fd0 2022-06-29T16:05:37.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.142+0000 7f56353e6700 1 -- 172.21.15.60:0/2139417571 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5630143fd0 msgr2=0x7f5630144410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:37.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.142+0000 7f56353e6700 1 --2- 172.21.15.60:0/2139417571 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5630143fd0 0x7f5630144410 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 rx=0x7f5618002f70 tx=0x7f5618006eb0).stop 2022-06-29T16:05:37.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.142+0000 7f56353e6700 1 -- 172.21.15.60:0/2139417571 shutdown_connections 2022-06-29T16:05:37.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.142+0000 7f56353e6700 1 --2- 172.21.15.60:0/2139417571 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5630143fd0 0x7f5630144410 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.142+0000 7f56353e6700 1 --2- 172.21.15.60:0/2139417571 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5630065410 0x7f56300b4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.142+0000 7f56353e6700 1 -- 172.21.15.60:0/2139417571 >> 172.21.15.60:0/2139417571 conn(0x7f56300b9340 msgr2=0x7f56300b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:37.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.142+0000 7f56353e6700 1 -- 172.21.15.60:0/2139417571 shutdown_connections 2022-06-29T16:05:37.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.142+0000 7f56353e6700 1 -- 172.21.15.60:0/2139417571 wait complete. 2022-06-29T16:05:37.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.146+0000 7f56353e6700 1 Processor -- start 2022-06-29T16:05:37.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.146+0000 7f56353e6700 1 -- start start 2022-06-29T16:05:37.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.146+0000 7f56353e6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5630065410 0x7f56300b8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:37.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.146+0000 7f56353e6700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5630143fd0 0x7f5630156b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:37.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.146+0000 7f56353e6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f563014a570 0x7f5630159900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:37.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.146+0000 7f56353e6700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56300b03b0 con 0x7f563014a570 2022-06-29T16:05:37.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.146+0000 7f56353e6700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5630156440 con 0x7f5630143fd0 2022-06-29T16:05:37.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.146+0000 7f56353e6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56301566e0 con 0x7f5630065410 2022-06-29T16:05:37.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.146+0000 7f5627fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5630143fd0 0x7f5630156b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:37.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.146+0000 7f562effd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5630065410 0x7f56300b8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:37.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.146+0000 7f562f7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f563014a570 0x7f5630159900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:37.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.146+0000 7f562effd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5630065410 0x7f56300b8370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:35448/0 (socket says 172.21.15.60:35448) 2022-06-29T16:05:37.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.146+0000 7f5627fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5630143fd0 0x7f5630156b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:56618/0 (socket says 172.21.15.60:56618) 2022-06-29T16:05:37.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.146+0000 7f562effd700 1 -- 172.21.15.60:0/1623354015 learned_addr learned my addr 172.21.15.60:0/1623354015 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:37.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.150+0000 7f5627fff700 1 -- 172.21.15.60:0/1623354015 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5630065410 msgr2=0x7f56300b8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:37.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.150+0000 7f5627fff700 1 --2- 172.21.15.60:0/1623354015 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5630065410 0x7f56300b8370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.150+0000 7f5627fff700 1 -- 172.21.15.60:0/1623354015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f563014a570 msgr2=0x7f5630159900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:37.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.150+0000 7f5627fff700 1 --2- 172.21.15.60:0/1623354015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f563014a570 0x7f5630159900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.150+0000 7f5627fff700 1 -- 172.21.15.60:0/1623354015 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f562000b180 con 0x7f5630143fd0 2022-06-29T16:05:37.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.150+0000 7f562effd700 1 --2- 172.21.15.60:0/1623354015 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5630065410 0x7f56300b8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:05:37.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.150+0000 7f5627fff700 1 --2- 172.21.15.60:0/1623354015 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5630143fd0 0x7f5630156b80 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f5618002f70 tx=0x7f5618015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:37.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.150+0000 7f562cff9700 1 -- 172.21.15.60:0/1623354015 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f56180116e0 con 0x7f5630143fd0 2022-06-29T16:05:37.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.150+0000 7f562cff9700 1 -- 172.21.15.60:0/1623354015 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5618011840 con 0x7f5630143fd0 2022-06-29T16:05:37.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.150+0000 7f56353e6700 1 -- 172.21.15.60:0/1623354015 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f561800b1e0 con 0x7f5630143fd0 2022-06-29T16:05:37.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.150+0000 7f562cff9700 1 -- 172.21.15.60:0/1623354015 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5618011ba0 con 0x7f5630143fd0 2022-06-29T16:05:37.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.150+0000 7f56353e6700 1 -- 172.21.15.60:0/1623354015 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56300b0010 con 0x7f5630143fd0 2022-06-29T16:05:37.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.150+0000 7f56353e6700 1 -- 172.21.15.60:0/1623354015 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56300573d0 con 0x7f5630143fd0 2022-06-29T16:05:37.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.150+0000 7f562cff9700 1 -- 172.21.15.60:0/1623354015 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5618025030 con 0x7f5630143fd0 2022-06-29T16:05:37.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.150+0000 7f562cff9700 1 --2- 172.21.15.60:0/1623354015 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5614053950 0x7f5614055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:37.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.150+0000 7f562cff9700 1 -- 172.21.15.60:0/1623354015 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f561806c9b0 con 0x7f5630143fd0 2022-06-29T16:05:37.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.150+0000 7f562effd700 1 --2- 172.21.15.60:0/1623354015 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5614053950 0x7f5614055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:37.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.154+0000 7f562effd700 1 --2- 172.21.15.60:0/1623354015 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5614053950 0x7f5614055e10 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f562000b000 tx=0x7f5620006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:37.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.158+0000 7f562cff9700 1 -- 172.21.15.60:0/1623354015 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5618043c00 con 0x7f5630143fd0 2022-06-29T16:05:37.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.286+0000 7f56353e6700 1 -- 172.21.15.60:0/1623354015 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56301568a0 con 0x7f5630143fd0 2022-06-29T16:05:37.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.290+0000 7f562cff9700 1 -- 172.21.15.60:0/1623354015 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+13182 (secure 0 0 0) 0x7f5618023090 con 0x7f5630143fd0 2022-06-29T16:05:37.298 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:37.298 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":87,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:05:36.666602+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:05:37.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.294+0000 7f56353e6700 1 -- 172.21.15.60:0/1623354015 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5614053950 msgr2=0x7f5614055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:37.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.294+0000 7f56353e6700 1 --2- 172.21.15.60:0/1623354015 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5614053950 0x7f5614055e10 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f562000b000 tx=0x7f5620006df0).stop 2022-06-29T16:05:37.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.294+0000 7f56353e6700 1 -- 172.21.15.60:0/1623354015 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5630143fd0 msgr2=0x7f5630156b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:37.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.294+0000 7f56353e6700 1 --2- 172.21.15.60:0/1623354015 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5630143fd0 0x7f5630156b80 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f5618002f70 tx=0x7f5618015000).stop 2022-06-29T16:05:37.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.294+0000 7f56353e6700 1 -- 172.21.15.60:0/1623354015 shutdown_connections 2022-06-29T16:05:37.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.294+0000 7f56353e6700 1 --2- 172.21.15.60:0/1623354015 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5614053950 0x7f5614055e10 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.294+0000 7f56353e6700 1 --2- 172.21.15.60:0/1623354015 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f563014a570 0x7f5630159900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.294+0000 7f56353e6700 1 --2- 172.21.15.60:0/1623354015 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5630143fd0 0x7f5630156b80 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.294+0000 7f56353e6700 1 --2- 172.21.15.60:0/1623354015 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5630065410 0x7f56300b8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.294+0000 7f56353e6700 1 -- 172.21.15.60:0/1623354015 >> 172.21.15.60:0/1623354015 conn(0x7f56300b9340 msgr2=0x7f56301414b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:37.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.294+0000 7f56353e6700 1 -- 172.21.15.60:0/1623354015 shutdown_connections 2022-06-29T16:05:37.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.294+0000 7f56353e6700 1 -- 172.21.15.60:0/1623354015 wait complete. 2022-06-29T16:05:37.314 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-06-29T16:05:37.315 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:05:37.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.418+0000 7f7f04b3d700 1 Processor -- start 2022-06-29T16:05:37.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.418+0000 7f7f04b3d700 1 -- start start 2022-06-29T16:05:37.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.422+0000 7f7f04b3d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f000b5990 0x7f7f00067280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:37.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.422+0000 7f7f04b3d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7f000af780 0x7f7f00067770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:37.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.422+0000 7f7f04b3d700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f000b0410 con 0x7f7f000ad080 2022-06-29T16:05:37.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.422+0000 7f7f04b3d700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f000b5d70 con 0x7f7f000af780 2022-06-29T16:05:37.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.422+0000 7f7f04b3d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f000b1730 con 0x7f7f000b5990 2022-06-29T16:05:37.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.422+0000 7f7efeffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7f000af780 0x7f7f00067770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:37.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.422+0000 7f7efeffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7f000af780 0x7f7f00067770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50598/0 (socket says 172.21.15.60:50598) 2022-06-29T16:05:37.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.422+0000 7f7efeffd700 1 -- 172.21.15.60:0/418870474 learned_addr learned my addr 172.21.15.60:0/418870474 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:37.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.422+0000 7f7eff7fe700 1 --2- 172.21.15.60:0/418870474 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f000b5990 0x7f7f00067280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:37.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.422+0000 7f7efeffd700 1 -- 172.21.15.60:0/418870474 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f000b5990 msgr2=0x7f7f00067280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:37.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.422+0000 7f7efeffd700 1 --2- 172.21.15.60:0/418870474 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f000b5990 0x7f7f00067280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.422+0000 7f7efeffd700 1 -- 172.21.15.60:0/418870474 >> v1:172.21.15.92:6789/0 conn(0x7f7f000ad080 legacy=0x7f7f000ae870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:37.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.422+0000 7f7efeffd700 1 -- 172.21.15.60:0/418870474 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f000b19d0 con 0x7f7f000af780 2022-06-29T16:05:37.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.422+0000 7f7efeffd700 1 --2- 172.21.15.60:0/418870474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f000af780 0x7f7f00067770 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 rx=0x7f7ee8002f70 tx=0x7f7ee8006eb0).ready entity=mon.0 client_cookie=cf112921a306321 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:37.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.422+0000 7f7efe7fc700 1 -- 172.21.15.60:0/418870474 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ee800d570 con 0x7f7f000af780 2022-06-29T16:05:37.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.422+0000 7f7efe7fc700 1 -- 172.21.15.60:0/418870474 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ee8000e90 con 0x7f7f000af780 2022-06-29T16:05:37.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.422+0000 7f7efe7fc700 1 -- 172.21.15.60:0/418870474 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ee800d900 con 0x7f7f000af780 2022-06-29T16:05:37.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.422+0000 7f7f04b3d700 1 -- 172.21.15.60:0/418870474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f000af780 msgr2=0x7f7f00067770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:37.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.422+0000 7f7f04b3d700 1 --2- 172.21.15.60:0/418870474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f000af780 0x7f7f00067770 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 rx=0x7f7ee8002f70 tx=0x7f7ee8006eb0).stop 2022-06-29T16:05:37.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.422+0000 7f7f04b3d700 1 -- 172.21.15.60:0/418870474 shutdown_connections 2022-06-29T16:05:37.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.422+0000 7f7f04b3d700 1 --2- 172.21.15.60:0/418870474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f000af780 0x7f7f00067770 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.422+0000 7f7f04b3d700 1 --2- 172.21.15.60:0/418870474 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f000b5990 0x7f7f00067280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.422+0000 7f7f04b3d700 1 -- 172.21.15.60:0/418870474 >> 172.21.15.60:0/418870474 conn(0x7f7f00063010 msgr2=0x7f7f000633f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:37.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.422+0000 7f7f04b3d700 1 -- 172.21.15.60:0/418870474 shutdown_connections 2022-06-29T16:05:37.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.422+0000 7f7f04b3d700 1 -- 172.21.15.60:0/418870474 wait complete. 2022-06-29T16:05:37.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.426+0000 7f7f04b3d700 1 Processor -- start 2022-06-29T16:05:37.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.426+0000 7f7f04b3d700 1 -- start start 2022-06-29T16:05:37.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.426+0000 7f7f04b3d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f000ad080 0x7f7f0013c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:37.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.426+0000 7f7f04b3d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f000af780 0x7f7f0013a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:37.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.426+0000 7f7f04b3d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f000b5990 0x7f7f001370a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:37.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.426+0000 7f7f04b3d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0013b9b0 con 0x7f7f000b5990 2022-06-29T16:05:37.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.426+0000 7f7f04b3d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0013cbc0 con 0x7f7f000ad080 2022-06-29T16:05:37.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.426+0000 7f7f04b3d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f00139de0 con 0x7f7f000af780 2022-06-29T16:05:37.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.426+0000 7f7efffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f000b5990 0x7f7f001370a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:37.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.426+0000 7f7efffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f000b5990 0x7f7f001370a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50604/0 (socket says 172.21.15.60:50604) 2022-06-29T16:05:37.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.426+0000 7f7efffff700 1 -- 172.21.15.60:0/2476258556 learned_addr learned my addr 172.21.15.60:0/2476258556 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:37.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.426+0000 7f7efeffd700 1 --2- 172.21.15.60:0/2476258556 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f000af780 0x7f7f0013a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:37.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.426+0000 7f7eff7fe700 1 --2- 172.21.15.60:0/2476258556 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f000ad080 0x7f7f0013c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:37.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.426+0000 7f7efffff700 1 -- 172.21.15.60:0/2476258556 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f000af780 msgr2=0x7f7f0013a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:37.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.426+0000 7f7efffff700 1 --2- 172.21.15.60:0/2476258556 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f000af780 0x7f7f0013a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.426+0000 7f7efffff700 1 -- 172.21.15.60:0/2476258556 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f000ad080 msgr2=0x7f7f0013c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:37.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.426+0000 7f7efffff700 1 --2- 172.21.15.60:0/2476258556 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f000ad080 0x7f7f0013c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.426+0000 7f7efffff700 1 -- 172.21.15.60:0/2476258556 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee800b1e0 con 0x7f7f000b5990 2022-06-29T16:05:37.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.426+0000 7f7efffff700 1 --2- 172.21.15.60:0/2476258556 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f000b5990 0x7f7f001370a0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 rx=0x7f7f0013b400 tx=0x7f7ef8007ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:37.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.430+0000 7f7efcff9700 1 -- 172.21.15.60:0/2476258556 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ef800e4a0 con 0x7f7f000b5990 2022-06-29T16:05:37.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.430+0000 7f7efcff9700 1 -- 172.21.15.60:0/2476258556 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ef800e600 con 0x7f7f000b5990 2022-06-29T16:05:37.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.430+0000 7f7f04b3d700 1 -- 172.21.15.60:0/2476258556 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f0013a010 con 0x7f7f000b5990 2022-06-29T16:05:37.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.430+0000 7f7efcff9700 1 -- 172.21.15.60:0/2476258556 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ef800e960 con 0x7f7f000b5990 2022-06-29T16:05:37.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.430+0000 7f7f04b3d700 1 -- 172.21.15.60:0/2476258556 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f001391a0 con 0x7f7f000b5990 2022-06-29T16:05:37.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.430+0000 7f7f04b3d700 1 -- 172.21.15.60:0/2476258556 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f000573d0 con 0x7f7f000b5990 2022-06-29T16:05:37.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.430+0000 7f7efcff9700 1 -- 172.21.15.60:0/2476258556 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7ef8010030 con 0x7f7f000b5990 2022-06-29T16:05:37.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.430+0000 7f7efcff9700 1 --2- 172.21.15.60:0/2476258556 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7ed8053aa0 0x7f7ed8055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:37.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.430+0000 7f7efcff9700 1 -- 172.21.15.60:0/2476258556 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f7ef806bd90 con 0x7f7f000b5990 2022-06-29T16:05:37.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.430+0000 7f7eff7fe700 1 --2- 172.21.15.60:0/2476258556 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7ed8053aa0 0x7f7ed8055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:37.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.434+0000 7f7eff7fe700 1 --2- 172.21.15.60:0/2476258556 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7ed8053aa0 0x7f7ed8055f60 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f7ef000b000 tx=0x7f7ef0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:37.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.438+0000 7f7efcff9700 1 -- 172.21.15.60:0/2476258556 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7ef8042fe0 con 0x7f7f000b5990 2022-06-29T16:05:37.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.562+0000 7f7f04b3d700 1 -- 172.21.15.60:0/2476258556 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f7f0013a370 con 0x7f7f000b5990 2022-06-29T16:05:37.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.562+0000 7f7efcff9700 1 -- 172.21.15.60:0/2476258556 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v87) v1 ==== 110+0+10 (secure 0 0 0) 0x7f7ef8046600 con 0x7f7f000b5990 2022-06-29T16:05:37.572 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:05:37.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.566+0000 7f7f04b3d700 1 -- 172.21.15.60:0/2476258556 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7ed8053aa0 msgr2=0x7f7ed8055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:37.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.566+0000 7f7f04b3d700 1 --2- 172.21.15.60:0/2476258556 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7ed8053aa0 0x7f7ed8055f60 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f7ef000b000 tx=0x7f7ef0008040).stop 2022-06-29T16:05:37.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.566+0000 7f7f04b3d700 1 -- 172.21.15.60:0/2476258556 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f000b5990 msgr2=0x7f7f001370a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:37.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.566+0000 7f7f04b3d700 1 --2- 172.21.15.60:0/2476258556 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f000b5990 0x7f7f001370a0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 rx=0x7f7f0013b400 tx=0x7f7ef8007ef0).stop 2022-06-29T16:05:37.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.566+0000 7f7f04b3d700 1 -- 172.21.15.60:0/2476258556 shutdown_connections 2022-06-29T16:05:37.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.566+0000 7f7f04b3d700 1 --2- 172.21.15.60:0/2476258556 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7ed8053aa0 0x7f7ed8055f60 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.566+0000 7f7f04b3d700 1 --2- 172.21.15.60:0/2476258556 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7f000b5990 0x7f7f001370a0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.566+0000 7f7f04b3d700 1 --2- 172.21.15.60:0/2476258556 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7f000af780 0x7f7f0013a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.566+0000 7f7f04b3d700 1 --2- 172.21.15.60:0/2476258556 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7f000ad080 0x7f7f0013c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.566+0000 7f7f04b3d700 1 -- 172.21.15.60:0/2476258556 >> 172.21.15.60:0/2476258556 conn(0x7f7f00063010 msgr2=0x7f7f00137e50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:37.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.566+0000 7f7f04b3d700 1 -- 172.21.15.60:0/2476258556 shutdown_connections 2022-06-29T16:05:37.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.566+0000 7f7f04b3d700 1 -- 172.21.15.60:0/2476258556 wait complete. 2022-06-29T16:05:37.585 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:05:37.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d893f700 1 Processor -- start 2022-06-29T16:05:37.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d893f700 1 -- start start 2022-06-29T16:05:37.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d893f700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f30d40b2130 0x7f30d4146100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:37.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d893f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f30d40aed00 0x7f30d41465f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:37.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d893f700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30d405d450 con 0x7f30d405d070 2022-06-29T16:05:37.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d893f700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30d40adf90 con 0x7f30d40b2130 2022-06-29T16:05:37.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d893f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30d40b25d0 con 0x7f30d40aed00 2022-06-29T16:05:37.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d37fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f30d405d070 0x7f30d414aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58694/0 (socket says 172.21.15.60:58694) 2022-06-29T16:05:37.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d37fe700 1 -- 172.21.15.60:0/1636354557 learned_addr learned my addr 172.21.15.60:0/1636354557 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:37.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d3fff700 1 --2- 172.21.15.60:0/1636354557 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f30d40aed00 0x7f30d41465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:37.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d2ffd700 1 --2- 172.21.15.60:0/1636354557 >> v2:172.21.15.92:3300/0 conn(0x7f30d40b2130 0x7f30d4146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:37.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d27fc700 1 -- 172.21.15.60:0/1636354557 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3339261759 0 0) 0x7f30d405d450 con 0x7f30d405d070 2022-06-29T16:05:37.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d27fc700 1 -- 172.21.15.60:0/1636354557 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30b4003260 con 0x7f30d405d070 2022-06-29T16:05:37.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d27fc700 1 -- 172.21.15.60:0/1636354557 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 451926223 0 0) 0x7f30b4003260 con 0x7f30d405d070 2022-06-29T16:05:37.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d27fc700 1 -- 172.21.15.60:0/1636354557 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f30d40aed00 msgr2=0x7f30d41465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:37.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d27fc700 1 --2- 172.21.15.60:0/1636354557 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f30d40aed00 0x7f30d41465f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d27fc700 1 -- 172.21.15.60:0/1636354557 >> v2:172.21.15.92:3300/0 conn(0x7f30d40b2130 msgr2=0x7f30d4146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:37.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d27fc700 1 --2- 172.21.15.60:0/1636354557 >> v2:172.21.15.92:3300/0 conn(0x7f30d40b2130 0x7f30d4146100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d27fc700 1 -- 172.21.15.60:0/1636354557 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d414a350 con 0x7f30d405d070 2022-06-29T16:05:37.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d27fc700 1 -- 172.21.15.60:0/1636354557 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f30bc001680 con 0x7f30d405d070 2022-06-29T16:05:37.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d27fc700 1 -- 172.21.15.60:0/1636354557 >> v1:172.21.15.60:6789/0 conn(0x7f30d405d070 legacy=0x7f30d414aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:37.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d27fc700 1 --2- 172.21.15.60:0/1636354557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30b4003750 0x7f30b4003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:37.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d27fc700 1 --2- 172.21.15.60:0/1636354557 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f30d40aed00 0x7f30b4004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:37.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d27fc700 1 --2- 172.21.15.60:0/1636354557 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f30b40049f0 0x7f30b4004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:37.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d27fc700 1 -- 172.21.15.60:0/1636354557 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d40adf90 con 0x7f30b4003750 2022-06-29T16:05:37.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d27fc700 1 -- 172.21.15.60:0/1636354557 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d40b25d0 con 0x7f30b40049f0 2022-06-29T16:05:37.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d27fc700 1 -- 172.21.15.60:0/1636354557 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30b400b750 con 0x7f30d40aed00 2022-06-29T16:05:37.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d37fe700 1 --2- 172.21.15.60:0/1636354557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30b4003750 0x7f30b4003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:37.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d2ffd700 1 --2- 172.21.15.60:0/1636354557 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f30d40aed00 0x7f30b4004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:37.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d3fff700 1 --2- 172.21.15.60:0/1636354557 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f30b40049f0 0x7f30b4004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:37.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d37fe700 1 -- 172.21.15.60:0/1636354557 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f30d40aed00 msgr2=0x7f30b4004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:37.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d37fe700 1 --2- 172.21.15.60:0/1636354557 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f30d40aed00 0x7f30b4004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d37fe700 1 -- 172.21.15.60:0/1636354557 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f30b40049f0 msgr2=0x7f30b4004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:37.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d37fe700 1 --2- 172.21.15.60:0/1636354557 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f30b40049f0 0x7f30b4004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d37fe700 1 -- 172.21.15.60:0/1636354557 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b4003260 con 0x7f30b4003750 2022-06-29T16:05:37.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d2ffd700 1 --2- 172.21.15.60:0/1636354557 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f30d40aed00 0x7f30b4004500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:37.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d3fff700 1 --2- 172.21.15.60:0/1636354557 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f30b40049f0 0x7f30b4004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:37.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d37fe700 1 --2- 172.21.15.60:0/1636354557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30b4003750 0x7f30b4003b70 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 rx=0x7f30bc001390 tx=0x7f30bc001e70).ready entity=mon.0 client_cookie=5a06bcb57884cfe8 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:37.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d27fc700 1 -- 172.21.15.60:0/1636354557 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30bc007170 con 0x7f30b4003750 2022-06-29T16:05:37.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d27fc700 1 -- 172.21.15.60:0/1636354557 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30bc0072d0 con 0x7f30b4003750 2022-06-29T16:05:37.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d27fc700 1 -- 172.21.15.60:0/1636354557 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30bc00c8f0 con 0x7f30b4003750 2022-06-29T16:05:37.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d893f700 1 -- 172.21.15.60:0/1636354557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30b4003750 msgr2=0x7f30b4003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:37.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.682+0000 7f30d893f700 1 --2- 172.21.15.60:0/1636354557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30b4003750 0x7f30b4003b70 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 rx=0x7f30bc001390 tx=0x7f30bc001e70).stop 2022-06-29T16:05:37.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.686+0000 7f30d3fff700 1 -- 172.21.15.60:0/1636354557 reap_dead start 2022-06-29T16:05:37.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.686+0000 7f30d893f700 1 -- 172.21.15.60:0/1636354557 shutdown_connections 2022-06-29T16:05:37.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.686+0000 7f30d893f700 1 -- 172.21.15.60:0/1636354557 >> 172.21.15.60:0/1636354557 conn(0x7f30d40b5890 msgr2=0x7f30d40b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:37.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.686+0000 7f30d893f700 1 -- 172.21.15.60:0/1636354557 shutdown_connections 2022-06-29T16:05:37.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.686+0000 7f30d893f700 1 -- 172.21.15.60:0/1636354557 wait complete. 2022-06-29T16:05:37.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.686+0000 7f30d893f700 1 Processor -- start 2022-06-29T16:05:37.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.686+0000 7f30d893f700 1 -- start start 2022-06-29T16:05:37.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.686+0000 7f30d893f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f30d4148790 0x7f30d4148b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:37.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.686+0000 7f30d893f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f30d40640d0 0x7f30d4066530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:37.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.686+0000 7f30d893f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30d4066a20 0x7f30d4156480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:37.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.686+0000 7f30d893f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d4143580 con 0x7f30d4066a20 2022-06-29T16:05:37.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.686+0000 7f30d893f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d4149110 con 0x7f30d40640d0 2022-06-29T16:05:37.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.686+0000 7f30d893f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30d4066eb0 con 0x7f30d4148790 2022-06-29T16:05:37.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.686+0000 7f30d37fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f30d4148790 0x7f30d4148b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:37.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.686+0000 7f30d3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30d4066a20 0x7f30d4156480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:37.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.686+0000 7f30d3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30d4066a20 0x7f30d4156480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50624/0 (socket says 172.21.15.60:50624) 2022-06-29T16:05:37.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.686+0000 7f30d37fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f30d4148790 0x7f30d4148b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:35482/0 (socket says 172.21.15.60:35482) 2022-06-29T16:05:37.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.686+0000 7f30d3fff700 1 -- 172.21.15.60:0/3075264494 learned_addr learned my addr 172.21.15.60:0/3075264494 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:37.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.686+0000 7f30d2ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f30d40640d0 0x7f30d4066530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:37.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.686+0000 7f30d3fff700 1 -- 172.21.15.60:0/3075264494 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f30d4148790 msgr2=0x7f30d4148b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:37.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.686+0000 7f30d3fff700 1 --2- 172.21.15.60:0/3075264494 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f30d4148790 0x7f30d4148b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.686+0000 7f30d3fff700 1 -- 172.21.15.60:0/3075264494 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f30d40640d0 msgr2=0x7f30d4066530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:37.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.686+0000 7f30d3fff700 1 --2- 172.21.15.60:0/3075264494 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f30d40640d0 0x7f30d4066530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.686+0000 7f30d3fff700 1 -- 172.21.15.60:0/3075264494 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d4148260 con 0x7f30d4066a20 2022-06-29T16:05:37.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.686+0000 7f30d3fff700 1 --2- 172.21.15.60:0/3075264494 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30d4066a20 0x7f30d4156480 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 rx=0x7f30cc00b000 tx=0x7f30cc008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:37.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.690+0000 7f30d0ff9700 1 -- 172.21.15.60:0/3075264494 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30cc015480 con 0x7f30d4066a20 2022-06-29T16:05:37.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.690+0000 7f30d0ff9700 1 -- 172.21.15.60:0/3075264494 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30cc0155e0 con 0x7f30d4066a20 2022-06-29T16:05:37.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.690+0000 7f30d893f700 1 -- 172.21.15.60:0/3075264494 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30d4143020 con 0x7f30d4066a20 2022-06-29T16:05:37.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.690+0000 7f30d0ff9700 1 -- 172.21.15.60:0/3075264494 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30cc015940 con 0x7f30d4066a20 2022-06-29T16:05:37.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.690+0000 7f30d893f700 1 -- 172.21.15.60:0/3075264494 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30d4143d60 con 0x7f30d4066a20 2022-06-29T16:05:37.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.690+0000 7f30d0ff9700 1 -- 172.21.15.60:0/3075264494 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f30cc012040 con 0x7f30d4066a20 2022-06-29T16:05:37.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.690+0000 7f30d893f700 1 -- 172.21.15.60:0/3075264494 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30d414a5f0 con 0x7f30d4066a20 2022-06-29T16:05:37.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.690+0000 7f30d0ff9700 1 --2- 172.21.15.60:0/3075264494 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f30b8053aa0 0x7f30b8055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:37.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.690+0000 7f30d0ff9700 1 -- 172.21.15.60:0/3075264494 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f30cc06bf30 con 0x7f30d4066a20 2022-06-29T16:05:37.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.690+0000 7f30d37fe700 1 --2- 172.21.15.60:0/3075264494 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f30b8053aa0 0x7f30b8055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:37.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.690+0000 7f30d37fe700 1 --2- 172.21.15.60:0/3075264494 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f30b8053aa0 0x7f30b8055f60 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f30bc01b000 tx=0x7f30bc015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:37.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.698+0000 7f30d0ff9700 1 -- 172.21.15.60:0/3075264494 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f30cc043180 con 0x7f30d4066a20 2022-06-29T16:05:37.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.826+0000 7f30d893f700 1 -- 172.21.15.60:0/3075264494 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f30d4143d60 con 0x7f30d4066a20 2022-06-29T16:05:37.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.826+0000 7f30d0ff9700 1 -- 172.21.15.60:0/3075264494 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v87) v1 ==== 92+0+10 (secure 0 0 0) 0x7f30d4143d60 con 0x7f30d4066a20 2022-06-29T16:05:37.834 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:05:37.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.826+0000 7f30d893f700 1 -- 172.21.15.60:0/3075264494 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f30b8053aa0 msgr2=0x7f30b8055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:37.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.826+0000 7f30d893f700 1 --2- 172.21.15.60:0/3075264494 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f30b8053aa0 0x7f30b8055f60 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f30bc01b000 tx=0x7f30bc015000).stop 2022-06-29T16:05:37.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.826+0000 7f30d893f700 1 -- 172.21.15.60:0/3075264494 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30d4066a20 msgr2=0x7f30d4156480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:37.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.826+0000 7f30d893f700 1 --2- 172.21.15.60:0/3075264494 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30d4066a20 0x7f30d4156480 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 rx=0x7f30cc00b000 tx=0x7f30cc008040).stop 2022-06-29T16:05:37.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.826+0000 7f30d893f700 1 -- 172.21.15.60:0/3075264494 shutdown_connections 2022-06-29T16:05:37.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.826+0000 7f30d893f700 1 --2- 172.21.15.60:0/3075264494 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f30b8053aa0 0x7f30b8055f60 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.826+0000 7f30d893f700 1 --2- 172.21.15.60:0/3075264494 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30d4066a20 0x7f30d4156480 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.826+0000 7f30d893f700 1 --2- 172.21.15.60:0/3075264494 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f30d40640d0 0x7f30d4066530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.826+0000 7f30d893f700 1 --2- 172.21.15.60:0/3075264494 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f30d4148790 0x7f30d4148b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.826+0000 7f30d893f700 1 -- 172.21.15.60:0/3075264494 >> 172.21.15.60:0/3075264494 conn(0x7f30d40b5890 msgr2=0x7f30d40b2cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:37.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.830+0000 7f30d893f700 1 -- 172.21.15.60:0/3075264494 shutdown_connections 2022-06-29T16:05:37.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.830+0000 7f30d893f700 1 -- 172.21.15.60:0/3075264494 wait complete. 2022-06-29T16:05:37.847 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-06-29T16:05:37.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.938+0000 7f5586b33700 1 Processor -- start 2022-06-29T16:05:37.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.942+0000 7f5586b33700 1 -- start start 2022-06-29T16:05:37.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.942+0000 7f5586b33700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f558005c390 0x7f55800b2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:37.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.942+0000 7f5586b33700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f55801442b0 0x7f5580144710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:37.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.942+0000 7f5586b33700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f558005c770 con 0x7f5580140a20 2022-06-29T16:05:37.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.942+0000 7f5586b33700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5580140750 con 0x7f55801442b0 2022-06-29T16:05:37.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.942+0000 7f5586b33700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5580143e80 con 0x7f558005c390 2022-06-29T16:05:37.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.942+0000 7f55850d0700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5580140a20 0x7f5580140e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58716/0 (socket says 172.21.15.60:58716) 2022-06-29T16:05:37.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.942+0000 7f55850d0700 1 -- 172.21.15.60:0/3926592947 learned_addr learned my addr 172.21.15.60:0/3926592947 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:37.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.942+0000 7f557ffff700 1 --2- 172.21.15.60:0/3926592947 >> v2:172.21.15.92:3300/0 conn(0x7f55801442b0 0x7f5580144710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:37.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.942+0000 7f55848cf700 1 --2- 172.21.15.60:0/3926592947 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f558005c390 0x7f55800b2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:37.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.942+0000 7f557f7fe700 1 -- 172.21.15.60:0/3926592947 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2306566240 0 0) 0x7f558005c770 con 0x7f5580140a20 2022-06-29T16:05:37.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.942+0000 7f557f7fe700 1 -- 172.21.15.60:0/3926592947 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f556c003260 con 0x7f5580140a20 2022-06-29T16:05:37.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.942+0000 7f55848cf700 1 -- 172.21.15.60:0/3926592947 >> v2:172.21.15.92:3300/0 conn(0x7f55801442b0 msgr2=0x7f5580144710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:37.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.942+0000 7f55848cf700 1 --2- 172.21.15.60:0/3926592947 >> v2:172.21.15.92:3300/0 conn(0x7f55801442b0 0x7f5580144710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.942+0000 7f557f7fe700 1 -- 172.21.15.60:0/3926592947 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 926911426 0 0) 0x7f556c003260 con 0x7f5580140a20 2022-06-29T16:05:37.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.942+0000 7f55848cf700 1 -- 172.21.15.60:0/3926592947 >> v1:172.21.15.60:6789/0 conn(0x7f5580140a20 legacy=0x7f5580140e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:37.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.942+0000 7f55848cf700 1 -- 172.21.15.60:0/3926592947 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5580065f40 con 0x7f558005c390 2022-06-29T16:05:37.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.942+0000 7f557ffff700 1 --2- 172.21.15.60:0/3926592947 >> v2:172.21.15.92:3300/0 conn(0x7f55801442b0 0x7f5580144710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:05:37.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.942+0000 7f55848cf700 1 --2- 172.21.15.60:0/3926592947 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f558005c390 0x7f55800b2220 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f5578001b80 tx=0x7f5570006df0).ready entity=mon.2 client_cookie=b23de1b93067ae7f server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:37.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.942+0000 7f557f7fe700 1 -- 172.21.15.60:0/3926592947 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55780019e0 con 0x7f558005c390 2022-06-29T16:05:37.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.942+0000 7f557f7fe700 1 -- 172.21.15.60:0/3926592947 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5570000e90 con 0x7f558005c390 2022-06-29T16:05:37.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.942+0000 7f557f7fe700 1 -- 172.21.15.60:0/3926592947 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5570012460 con 0x7f558005c390 2022-06-29T16:05:37.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.942+0000 7f5586b33700 1 -- 172.21.15.60:0/3926592947 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f558005c390 msgr2=0x7f55800b2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:37.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.942+0000 7f5586b33700 1 --2- 172.21.15.60:0/3926592947 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f558005c390 0x7f55800b2220 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f5578001b80 tx=0x7f5570006df0).stop 2022-06-29T16:05:37.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.942+0000 7f5586b33700 1 -- 172.21.15.60:0/3926592947 shutdown_connections 2022-06-29T16:05:37.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.942+0000 7f5586b33700 1 --2- 172.21.15.60:0/3926592947 >> v2:172.21.15.92:3300/0 conn(0x7f55801442b0 0x7f5580144710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.942+0000 7f5586b33700 1 --2- 172.21.15.60:0/3926592947 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f558005c390 0x7f55800b2220 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.942+0000 7f5586b33700 1 -- 172.21.15.60:0/3926592947 >> 172.21.15.60:0/3926592947 conn(0x7f5580064a10 msgr2=0x7f5580064df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:37.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.942+0000 7f5586b33700 1 -- 172.21.15.60:0/3926592947 shutdown_connections 2022-06-29T16:05:37.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.942+0000 7f5586b33700 1 -- 172.21.15.60:0/3926592947 wait complete. 2022-06-29T16:05:37.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.946+0000 7f5586b33700 1 Processor -- start 2022-06-29T16:05:37.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.946+0000 7f5586b33700 1 -- start start 2022-06-29T16:05:37.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.946+0000 7f5586b33700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f558005c390 0x7f558013ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:37.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.946+0000 7f5586b33700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5580140a20 0x7f5580153910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:37.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.946+0000 7f5586b33700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55801442b0 0x7f558014fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:37.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.946+0000 7f5586b33700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5580146470 con 0x7f55801442b0 2022-06-29T16:05:37.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.946+0000 7f5586b33700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55800b6b70 con 0x7f558005c390 2022-06-29T16:05:37.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.946+0000 7f5586b33700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5580146660 con 0x7f5580140a20 2022-06-29T16:05:37.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.946+0000 7f55850d0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55801442b0 0x7f558014fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:37.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.946+0000 7f55850d0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55801442b0 0x7f558014fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50638/0 (socket says 172.21.15.60:50638) 2022-06-29T16:05:37.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.946+0000 7f55850d0700 1 -- 172.21.15.60:0/465996408 learned_addr learned my addr 172.21.15.60:0/465996408 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:37.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.946+0000 7f55848cf700 1 --2- 172.21.15.60:0/465996408 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f558005c390 0x7f558013ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:37.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.946+0000 7f557ffff700 1 --2- 172.21.15.60:0/465996408 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5580140a20 0x7f5580153910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:37.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.946+0000 7f55850d0700 1 -- 172.21.15.60:0/465996408 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5580140a20 msgr2=0x7f5580153910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:37.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.946+0000 7f55850d0700 1 --2- 172.21.15.60:0/465996408 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5580140a20 0x7f5580153910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.946+0000 7f55850d0700 1 -- 172.21.15.60:0/465996408 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f558005c390 msgr2=0x7f558013ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:37.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.946+0000 7f55850d0700 1 --2- 172.21.15.60:0/465996408 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f558005c390 0x7f558013ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:37.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.946+0000 7f55850d0700 1 -- 172.21.15.60:0/465996408 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f557400b180 con 0x7f55801442b0 2022-06-29T16:05:37.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.946+0000 7f55850d0700 1 --2- 172.21.15.60:0/465996408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55801442b0 0x7f558014fa00 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 rx=0x7f5578001c40 tx=0x7f55780021a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:37.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.946+0000 7f557dffb700 1 -- 172.21.15.60:0/465996408 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5578004ab0 con 0x7f55801442b0 2022-06-29T16:05:37.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.946+0000 7f557dffb700 1 -- 172.21.15.60:0/465996408 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5578004c10 con 0x7f55801442b0 2022-06-29T16:05:37.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.946+0000 7f5586b33700 1 -- 172.21.15.60:0/465996408 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f557000b180 con 0x7f55801442b0 2022-06-29T16:05:37.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.950+0000 7f557dffb700 1 -- 172.21.15.60:0/465996408 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5578007170 con 0x7f55801442b0 2022-06-29T16:05:37.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.950+0000 7f5586b33700 1 -- 172.21.15.60:0/465996408 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55800b7e20 con 0x7f55801442b0 2022-06-29T16:05:37.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.950+0000 7f5586b33700 1 -- 172.21.15.60:0/465996408 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55800b1370 con 0x7f55801442b0 2022-06-29T16:05:37.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.950+0000 7f557dffb700 1 -- 172.21.15.60:0/465996408 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f557801b070 con 0x7f55801442b0 2022-06-29T16:05:37.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.950+0000 7f557dffb700 1 --2- 172.21.15.60:0/465996408 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5560053b00 0x7f5560055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:37.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.950+0000 7f557dffb700 1 -- 172.21.15.60:0/465996408 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f557806b640 con 0x7f55801442b0 2022-06-29T16:05:37.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.950+0000 7f55848cf700 1 --2- 172.21.15.60:0/465996408 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5560053b00 0x7f5560055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:37.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.950+0000 7f55848cf700 1 --2- 172.21.15.60:0/465996408 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5560053b00 0x7f5560055fc0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f557001a000 tx=0x7f5570014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:37.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:37.954+0000 7f557dffb700 1 -- 172.21.15.60:0/465996408 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f55780426a0 con 0x7f55801442b0 2022-06-29T16:05:38.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.086+0000 7f5586b33700 1 -- 172.21.15.60:0/465996408 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f55800573d0 con 0x7f55801442b0 2022-06-29T16:05:38.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.086+0000 7f557dffb700 1 -- 172.21.15.60:0/465996408 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v87) v1 ==== 84+0+143 (secure 0 0 0) 0x7f5578045cc0 con 0x7f55801442b0 2022-06-29T16:05:38.095 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:38.095 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2022-06-29T16:05:38.096 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:38.096 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2022-06-29T16:05:38.096 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": "device_health_metrics" 2022-06-29T16:05:38.096 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:38.096 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:38.097 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 2, 2022-06-29T16:05:38.097 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": "rbd" 2022-06-29T16:05:38.097 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:38.097 INFO:teuthology.orchestra.run.smithi060.stdout:] 2022-06-29T16:05:38.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.090+0000 7f5586b33700 1 -- 172.21.15.60:0/465996408 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5560053b00 msgr2=0x7f5560055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:38.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.090+0000 7f5586b33700 1 --2- 172.21.15.60:0/465996408 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5560053b00 0x7f5560055fc0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f557001a000 tx=0x7f5570014000).stop 2022-06-29T16:05:38.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.090+0000 7f5586b33700 1 -- 172.21.15.60:0/465996408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55801442b0 msgr2=0x7f558014fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:38.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.090+0000 7f5586b33700 1 --2- 172.21.15.60:0/465996408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55801442b0 0x7f558014fa00 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 rx=0x7f5578001c40 tx=0x7f55780021a0).stop 2022-06-29T16:05:38.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.090+0000 7f5586b33700 1 -- 172.21.15.60:0/465996408 shutdown_connections 2022-06-29T16:05:38.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.090+0000 7f5586b33700 1 --2- 172.21.15.60:0/465996408 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5560053b00 0x7f5560055fc0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:38.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.090+0000 7f5586b33700 1 --2- 172.21.15.60:0/465996408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f55801442b0 0x7f558014fa00 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:38.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.090+0000 7f5586b33700 1 --2- 172.21.15.60:0/465996408 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5580140a20 0x7f5580153910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:38.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.090+0000 7f5586b33700 1 --2- 172.21.15.60:0/465996408 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f558005c390 0x7f558013ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:38.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.090+0000 7f5586b33700 1 -- 172.21.15.60:0/465996408 >> 172.21.15.60:0/465996408 conn(0x7f5580064a10 msgr2=0x7f5580141d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:38.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.090+0000 7f5586b33700 1 -- 172.21.15.60:0/465996408 shutdown_connections 2022-06-29T16:05:38.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.090+0000 7f5586b33700 1 -- 172.21.15.60:0/465996408 wait complete. 2022-06-29T16:05:38.109 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-06-29T16:05:38.109 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-06-29T16:05:38.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.210+0000 7ff2655b4700 1 Processor -- start 2022-06-29T16:05:38.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.210+0000 7ff2655b4700 1 -- start start 2022-06-29T16:05:38.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.210+0000 7ff2655b4700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2600b5eb0 0x7ff260142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:38.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.210+0000 7ff2655b4700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff2600b48b0 0x7ff2600b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:38.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.210+0000 7ff2655b4700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2600b4d10 con 0x7ff2600b2c50 2022-06-29T16:05:38.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.210+0000 7ff2655b4700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2600b54e0 con 0x7ff2600b48b0 2022-06-29T16:05:38.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.210+0000 7ff2655b4700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2600b6290 con 0x7ff2600b5eb0 2022-06-29T16:05:38.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.210+0000 7ff2577fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff2600b48b0 0x7ff2600b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:38.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.210+0000 7ff264db3700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff2600b2c50 0x7ff2600b3090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58732/0 (socket says 172.21.15.60:58732) 2022-06-29T16:05:38.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.210+0000 7ff264db3700 1 -- 172.21.15.60:0/3698119733 learned_addr learned my addr 172.21.15.60:0/3698119733 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:38.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.214+0000 7ff25ffff700 1 --2- 172.21.15.60:0/3698119733 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2600b5eb0 0x7ff260142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:38.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.214+0000 7ff2577fe700 1 -- 172.21.15.60:0/3698119733 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2600b5eb0 msgr2=0x7ff260142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:38.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.214+0000 7ff2577fe700 1 --2- 172.21.15.60:0/3698119733 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2600b5eb0 0x7ff260142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:38.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.214+0000 7ff2577fe700 1 -- 172.21.15.60:0/3698119733 >> v1:172.21.15.60:6789/0 conn(0x7ff2600b2c50 legacy=0x7ff2600b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:38.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.214+0000 7ff2577fe700 1 -- 172.21.15.60:0/3698119733 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff260142f10 con 0x7ff2600b48b0 2022-06-29T16:05:38.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.214+0000 7ff2577fe700 1 --2- 172.21.15.60:0/3698119733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2600b48b0 0x7ff2600b05e0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 rx=0x7ff248002f70 tx=0x7ff248006df0).ready entity=mon.0 client_cookie=aa45957fd3a03133 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:38.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.214+0000 7ff25f7fe700 1 -- 172.21.15.60:0/3698119733 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff24800d570 con 0x7ff2600b48b0 2022-06-29T16:05:38.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.214+0000 7ff25f7fe700 1 -- 172.21.15.60:0/3698119733 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff248000e90 con 0x7ff2600b48b0 2022-06-29T16:05:38.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.214+0000 7ff25f7fe700 1 -- 172.21.15.60:0/3698119733 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff24800d8c0 con 0x7ff2600b48b0 2022-06-29T16:05:38.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.214+0000 7ff2655b4700 1 -- 172.21.15.60:0/3698119733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2600b48b0 msgr2=0x7ff2600b05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:38.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.214+0000 7ff2655b4700 1 --2- 172.21.15.60:0/3698119733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2600b48b0 0x7ff2600b05e0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 rx=0x7ff248002f70 tx=0x7ff248006df0).stop 2022-06-29T16:05:38.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.214+0000 7ff2655b4700 1 -- 172.21.15.60:0/3698119733 shutdown_connections 2022-06-29T16:05:38.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.214+0000 7ff2655b4700 1 --2- 172.21.15.60:0/3698119733 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2600b48b0 0x7ff2600b05e0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:38.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.214+0000 7ff2655b4700 1 --2- 172.21.15.60:0/3698119733 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2600b5eb0 0x7ff260142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:38.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.214+0000 7ff2655b4700 1 -- 172.21.15.60:0/3698119733 >> 172.21.15.60:0/3698119733 conn(0x7ff260062090 msgr2=0x7ff260062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:38.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.214+0000 7ff2655b4700 1 -- 172.21.15.60:0/3698119733 shutdown_connections 2022-06-29T16:05:38.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.214+0000 7ff2655b4700 1 -- 172.21.15.60:0/3698119733 wait complete. 2022-06-29T16:05:38.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.214+0000 7ff2655b4700 1 Processor -- start 2022-06-29T16:05:38.230 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.214+0000 7ff2655b4700 1 -- start start 2022-06-29T16:05:38.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.218+0000 7ff2655b4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2600b2c50 0x7ff260146b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:38.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.218+0000 7ff2655b4700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2600b48b0 0x7ff2600b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:38.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.218+0000 7ff2655b4700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2600b5eb0 0x7ff260141be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:38.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.218+0000 7ff2655b4700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff260145910 con 0x7ff2600b2c50 2022-06-29T16:05:38.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.218+0000 7ff2655b4700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff260146040 con 0x7ff2600b48b0 2022-06-29T16:05:38.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.218+0000 7ff2655b4700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff260145b00 con 0x7ff2600b5eb0 2022-06-29T16:05:38.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.218+0000 7ff25ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2600b2c50 0x7ff260146b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:38.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.218+0000 7ff25ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2600b2c50 0x7ff260146b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50648/0 (socket says 172.21.15.60:50648) 2022-06-29T16:05:38.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.218+0000 7ff25ffff700 1 -- 172.21.15.60:0/1231013711 learned_addr learned my addr 172.21.15.60:0/1231013711 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:38.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.218+0000 7ff2577fe700 1 --2- 172.21.15.60:0/1231013711 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2600b48b0 0x7ff2600b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:38.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.218+0000 7ff264db3700 1 --2- 172.21.15.60:0/1231013711 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2600b5eb0 0x7ff260141be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:38.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.218+0000 7ff25ffff700 1 -- 172.21.15.60:0/1231013711 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2600b5eb0 msgr2=0x7ff260141be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:38.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.218+0000 7ff25ffff700 1 --2- 172.21.15.60:0/1231013711 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2600b5eb0 0x7ff260141be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:38.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.218+0000 7ff25ffff700 1 -- 172.21.15.60:0/1231013711 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2600b48b0 msgr2=0x7ff2600b15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:38.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.218+0000 7ff25ffff700 1 --2- 172.21.15.60:0/1231013711 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2600b48b0 0x7ff2600b15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:38.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.218+0000 7ff25ffff700 1 -- 172.21.15.60:0/1231013711 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24800b180 con 0x7ff2600b2c50 2022-06-29T16:05:38.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.218+0000 7ff25ffff700 1 --2- 172.21.15.60:0/1231013711 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2600b2c50 0x7ff260146b40 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 rx=0x7ff25000c000 tx=0x7ff250006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:38.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.218+0000 7ff25dffb700 1 -- 172.21.15.60:0/1231013711 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff250011750 con 0x7ff2600b2c50 2022-06-29T16:05:38.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.218+0000 7ff25dffb700 1 -- 172.21.15.60:0/1231013711 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2500118b0 con 0x7ff2600b2c50 2022-06-29T16:05:38.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.218+0000 7ff25dffb700 1 -- 172.21.15.60:0/1231013711 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff250011c10 con 0x7ff2600b2c50 2022-06-29T16:05:38.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.218+0000 7ff2655b4700 1 -- 172.21.15.60:0/1231013711 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff260145da0 con 0x7ff2600b2c50 2022-06-29T16:05:38.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.218+0000 7ff2655b4700 1 -- 172.21.15.60:0/1231013711 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2600b1ad0 con 0x7ff2600b2c50 2022-06-29T16:05:38.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.222+0000 7ff2655b4700 1 -- 172.21.15.60:0/1231013711 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2600b4f70 con 0x7ff2600b2c50 2022-06-29T16:05:38.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.222+0000 7ff25dffb700 1 -- 172.21.15.60:0/1231013711 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff250011d70 con 0x7ff2600b2c50 2022-06-29T16:05:38.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.222+0000 7ff25dffb700 1 --2- 172.21.15.60:0/1231013711 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff244053a70 0x7ff244055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:38.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.222+0000 7ff25dffb700 1 -- 172.21.15.60:0/1231013711 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5253+0+0 (secure 0 0 0) 0x7ff25006bfa0 con 0x7ff2600b2c50 2022-06-29T16:05:38.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.222+0000 7ff2577fe700 1 --2- 172.21.15.60:0/1231013711 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff244053a70 0x7ff244055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:38.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.222+0000 7ff2577fe700 1 --2- 172.21.15.60:0/1231013711 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff244053a70 0x7ff244055f30 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7ff24801a000 tx=0x7ff248014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:38.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.226+0000 7ff25dffb700 1 -- 172.21.15.60:0/1231013711 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff250043190 con 0x7ff2600b2c50 2022-06-29T16:05:38.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:38.350+0000 7ff2655b4700 1 -- 172.21.15.60:0/1231013711 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7ff2600573d0 con 0x7ff2600b2c50 2022-06-29T16:05:39.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.046+0000 7ff25dffb700 1 -- 172.21.15.60:0/1231013711 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v88) v1 ==== 133+0+0 (secure 0 0 0) 0x7ff2500467b0 con 0x7ff2600b2c50 2022-06-29T16:05:39.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.050+0000 7ff2655b4700 1 -- 172.21.15.60:0/1231013711 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff244053a70 msgr2=0x7ff244055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:39.058 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.050+0000 7ff2655b4700 1 --2- 172.21.15.60:0/1231013711 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff244053a70 0x7ff244055f30 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7ff24801a000 tx=0x7ff248014000).stop 2022-06-29T16:05:39.058 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.050+0000 7ff2655b4700 1 -- 172.21.15.60:0/1231013711 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2600b2c50 msgr2=0x7ff260146b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:39.058 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.050+0000 7ff2655b4700 1 --2- 172.21.15.60:0/1231013711 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2600b2c50 0x7ff260146b40 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 rx=0x7ff25000c000 tx=0x7ff250006df0).stop 2022-06-29T16:05:39.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.050+0000 7ff2655b4700 1 -- 172.21.15.60:0/1231013711 shutdown_connections 2022-06-29T16:05:39.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.050+0000 7ff2655b4700 1 --2- 172.21.15.60:0/1231013711 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff244053a70 0x7ff244055f30 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:39.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.050+0000 7ff2655b4700 1 --2- 172.21.15.60:0/1231013711 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2600b5eb0 0x7ff260141be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:39.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.050+0000 7ff2655b4700 1 --2- 172.21.15.60:0/1231013711 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2600b48b0 0x7ff2600b15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:39.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.050+0000 7ff2655b4700 1 --2- 172.21.15.60:0/1231013711 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2600b2c50 0x7ff260146b40 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:39.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.050+0000 7ff2655b4700 1 -- 172.21.15.60:0/1231013711 >> 172.21.15.60:0/1231013711 conn(0x7ff260062090 msgr2=0x7ff260062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:39.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.054+0000 7ff2655b4700 1 -- 172.21.15.60:0/1231013711 shutdown_connections 2022-06-29T16:05:39.063 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.054+0000 7ff2655b4700 1 -- 172.21.15.60:0/1231013711 wait complete. 2022-06-29T16:05:39.064 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2022-06-29T16:05:39.077 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-06-29T16:05:39.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.178+0000 7ff21c8b6700 1 Processor -- start 2022-06-29T16:05:39.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.178+0000 7ff21c8b6700 1 -- start start 2022-06-29T16:05:39.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.182+0000 7ff21c8b6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff21805d070 0x7ff21813c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:39.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.182+0000 7ff21c8b6700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff2180b0830 0x7ff21813fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:39.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.182+0000 7ff21c8b6700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff21805d450 con 0x7ff2180b2130 2022-06-29T16:05:39.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.182+0000 7ff21c8b6700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff218063ba0 con 0x7ff2180b0830 2022-06-29T16:05:39.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.182+0000 7ff21c8b6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff218063e40 con 0x7ff21805d070 2022-06-29T16:05:39.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.182+0000 7ff217fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff2180b0830 0x7ff21813fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:39.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.182+0000 7ff216ffd700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff2180b2130 0x7ff21813caa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58742/0 (socket says 172.21.15.60:58742) 2022-06-29T16:05:39.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.182+0000 7ff216ffd700 1 -- 172.21.15.60:0/3032369737 learned_addr learned my addr 172.21.15.60:0/3032369737 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:39.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.182+0000 7ff2177fe700 1 --2- 172.21.15.60:0/3032369737 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff21805d070 0x7ff21813c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:39.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.182+0000 7ff2167fc700 1 -- 172.21.15.60:0/3032369737 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 290278239 0 0) 0x7ff21805d450 con 0x7ff2180b2130 2022-06-29T16:05:39.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.182+0000 7ff2167fc700 1 -- 172.21.15.60:0/3032369737 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1f8003260 con 0x7ff2180b2130 2022-06-29T16:05:39.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.182+0000 7ff217fff700 1 -- 172.21.15.60:0/3032369737 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff21805d070 msgr2=0x7ff21813c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:39.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.182+0000 7ff217fff700 1 --2- 172.21.15.60:0/3032369737 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff21805d070 0x7ff21813c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:39.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.182+0000 7ff217fff700 1 -- 172.21.15.60:0/3032369737 >> v1:172.21.15.60:6789/0 conn(0x7ff2180b2130 legacy=0x7ff21813caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:39.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.182+0000 7ff217fff700 1 -- 172.21.15.60:0/3032369737 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff218067650 con 0x7ff2180b0830 2022-06-29T16:05:39.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.182+0000 7ff2177fe700 1 --2- 172.21.15.60:0/3032369737 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff21805d070 0x7ff21813c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:39.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.182+0000 7ff217fff700 1 --2- 172.21.15.60:0/3032369737 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2180b0830 0x7ff21813fa70 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 rx=0x7ff2100032f0 tx=0x7ff2100091d0).ready entity=mon.0 client_cookie=63bbb2cfe0c38d06 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:39.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.182+0000 7ff2167fc700 1 -- 172.21.15.60:0/3032369737 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff21000b6c0 con 0x7ff2180b0830 2022-06-29T16:05:39.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.182+0000 7ff2167fc700 1 -- 172.21.15.60:0/3032369737 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff21000b820 con 0x7ff2180b0830 2022-06-29T16:05:39.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.182+0000 7ff2167fc700 1 -- 172.21.15.60:0/3032369737 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff21000b990 con 0x7ff2180b0830 2022-06-29T16:05:39.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.182+0000 7ff21c8b6700 1 -- 172.21.15.60:0/3032369737 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2180b0830 msgr2=0x7ff21813fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:39.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.182+0000 7ff21c8b6700 1 --2- 172.21.15.60:0/3032369737 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2180b0830 0x7ff21813fa70 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 rx=0x7ff2100032f0 tx=0x7ff2100091d0).stop 2022-06-29T16:05:39.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.182+0000 7ff21c8b6700 1 -- 172.21.15.60:0/3032369737 shutdown_connections 2022-06-29T16:05:39.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.182+0000 7ff21c8b6700 1 --2- 172.21.15.60:0/3032369737 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2180b0830 0x7ff21813fa70 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:39.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.182+0000 7ff21c8b6700 1 --2- 172.21.15.60:0/3032369737 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff21805d070 0x7ff21813c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:39.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.182+0000 7ff21c8b6700 1 -- 172.21.15.60:0/3032369737 >> 172.21.15.60:0/3032369737 conn(0x7ff2180b5890 msgr2=0x7ff2180b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:39.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.182+0000 7ff21c8b6700 1 -- 172.21.15.60:0/3032369737 shutdown_connections 2022-06-29T16:05:39.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.182+0000 7ff21c8b6700 1 -- 172.21.15.60:0/3032369737 wait complete. 2022-06-29T16:05:39.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.186+0000 7ff21c8b6700 1 Processor -- start 2022-06-29T16:05:39.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.186+0000 7ff21c8b6700 1 -- start start 2022-06-29T16:05:39.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.186+0000 7ff21c8b6700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff21805d070 0x7ff21812c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:39.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.186+0000 7ff21c8b6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2180b0830 0x7ff218127690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:39.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.186+0000 7ff21c8b6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2180b2130 0x7ff218127b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:39.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.186+0000 7ff21c8b6700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2180aca20 con 0x7ff2180b0830 2022-06-29T16:05:39.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.186+0000 7ff21c8b6700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2180acc50 con 0x7ff21805d070 2022-06-29T16:05:39.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.186+0000 7ff21c8b6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff21813c380 con 0x7ff2180b2130 2022-06-29T16:05:39.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.186+0000 7ff216ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2180b0830 0x7ff218127690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:39.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.186+0000 7ff216ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2180b0830 0x7ff218127690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50666/0 (socket says 172.21.15.60:50666) 2022-06-29T16:05:39.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.186+0000 7ff216ffd700 1 -- 172.21.15.60:0/1827757522 learned_addr learned my addr 172.21.15.60:0/1827757522 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:39.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.186+0000 7ff217fff700 1 --2- 172.21.15.60:0/1827757522 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2180b2130 0x7ff218127b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:39.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.186+0000 7ff2177fe700 1 --2- 172.21.15.60:0/1827757522 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff21805d070 0x7ff21812c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:39.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.186+0000 7ff216ffd700 1 -- 172.21.15.60:0/1827757522 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2180b2130 msgr2=0x7ff218127b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:39.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.186+0000 7ff216ffd700 1 --2- 172.21.15.60:0/1827757522 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2180b2130 0x7ff218127b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:39.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.186+0000 7ff216ffd700 1 -- 172.21.15.60:0/1827757522 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff21805d070 msgr2=0x7ff21812c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:39.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.186+0000 7ff216ffd700 1 --2- 172.21.15.60:0/1827757522 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff21805d070 0x7ff21812c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:39.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.186+0000 7ff216ffd700 1 -- 172.21.15.60:0/1827757522 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff210009470 con 0x7ff2180b0830 2022-06-29T16:05:39.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.186+0000 7ff2177fe700 1 --2- 172.21.15.60:0/1827757522 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff21805d070 0x7ff21812c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:39.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.186+0000 7ff216ffd700 1 --2- 172.21.15.60:0/1827757522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2180b0830 0x7ff218127690 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 rx=0x7ff208001110 tx=0x7ff208007a60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:39.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.186+0000 7ff214ff9700 1 -- 172.21.15.60:0/1827757522 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff20800ce60 con 0x7ff2180b0830 2022-06-29T16:05:39.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.186+0000 7ff214ff9700 1 -- 172.21.15.60:0/1827757522 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2080070d0 con 0x7ff2180b0830 2022-06-29T16:05:39.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.186+0000 7ff214ff9700 1 -- 172.21.15.60:0/1827757522 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff208007430 con 0x7ff2180b0830 2022-06-29T16:05:39.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.186+0000 7ff21c8b6700 1 -- 172.21.15.60:0/1827757522 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff21813c620 con 0x7ff2180b0830 2022-06-29T16:05:39.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.186+0000 7ff21c8b6700 1 -- 172.21.15.60:0/1827757522 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff218128070 con 0x7ff2180b0830 2022-06-29T16:05:39.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.190+0000 7ff21c8b6700 1 -- 172.21.15.60:0/1827757522 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2180b2b90 con 0x7ff2180b0830 2022-06-29T16:05:39.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.190+0000 7ff214ff9700 1 -- 172.21.15.60:0/1827757522 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff208010070 con 0x7ff2180b0830 2022-06-29T16:05:39.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.190+0000 7ff214ff9700 1 --2- 172.21.15.60:0/1827757522 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff1fc053b30 0x7ff1fc055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:39.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.190+0000 7ff214ff9700 1 -- 172.21.15.60:0/1827757522 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5607+0+0 (secure 0 0 0) 0x7ff20806b240 con 0x7ff2180b0830 2022-06-29T16:05:39.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.190+0000 7ff2177fe700 1 --2- 172.21.15.60:0/1827757522 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff1fc053b30 0x7ff1fc055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:39.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.190+0000 7ff2177fe700 1 --2- 172.21.15.60:0/1827757522 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff1fc053b30 0x7ff1fc055ff0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7ff20000c000 tx=0x7ff200008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:39.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.198+0000 7ff214ff9700 1 -- 172.21.15.60:0/1827757522 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff208042220 con 0x7ff2180b0830 2022-06-29T16:05:39.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:39.322+0000 7ff21c8b6700 1 -- 172.21.15.60:0/1827757522 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7ff2180573d0 con 0x7ff2180b0830 2022-06-29T16:05:40.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.046+0000 7ff214ff9700 1 -- 172.21.15.60:0/1827757522 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v89) v1 ==== 125+0+0 (secure 0 0 0) 0x7ff208018070 con 0x7ff2180b0830 2022-06-29T16:05:40.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.054+0000 7ff21c8b6700 1 -- 172.21.15.60:0/1827757522 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff1fc053b30 msgr2=0x7ff1fc055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:40.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.054+0000 7ff21c8b6700 1 --2- 172.21.15.60:0/1827757522 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff1fc053b30 0x7ff1fc055ff0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7ff20000c000 tx=0x7ff200008040).stop 2022-06-29T16:05:40.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.054+0000 7ff21c8b6700 1 -- 172.21.15.60:0/1827757522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2180b0830 msgr2=0x7ff218127690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:40.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.054+0000 7ff21c8b6700 1 --2- 172.21.15.60:0/1827757522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2180b0830 0x7ff218127690 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 rx=0x7ff208001110 tx=0x7ff208007a60).stop 2022-06-29T16:05:40.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.058+0000 7ff21c8b6700 1 -- 172.21.15.60:0/1827757522 shutdown_connections 2022-06-29T16:05:40.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.058+0000 7ff21c8b6700 1 --2- 172.21.15.60:0/1827757522 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff1fc053b30 0x7ff1fc055ff0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:40.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.058+0000 7ff21c8b6700 1 --2- 172.21.15.60:0/1827757522 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff2180b2130 0x7ff218127b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:40.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.058+0000 7ff21c8b6700 1 --2- 172.21.15.60:0/1827757522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff2180b0830 0x7ff218127690 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:40.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.058+0000 7ff21c8b6700 1 --2- 172.21.15.60:0/1827757522 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff21805d070 0x7ff21812c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:40.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.058+0000 7ff21c8b6700 1 -- 172.21.15.60:0/1827757522 >> 172.21.15.60:0/1827757522 conn(0x7ff2180b5890 msgr2=0x7ff218135500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:40.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.058+0000 7ff21c8b6700 1 -- 172.21.15.60:0/1827757522 shutdown_connections 2022-06-29T16:05:40.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.058+0000 7ff21c8b6700 1 -- 172.21.15.60:0/1827757522 wait complete. 2022-06-29T16:05:40.068 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2022-06-29T16:05:40.078 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-06-29T16:05:40.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.178+0000 7f4aac840700 1 Processor -- start 2022-06-29T16:05:40.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.178+0000 7f4aac840700 1 -- start start 2022-06-29T16:05:40.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.182+0000 7f4aac840700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4aa805c040 0x7f4aa8142670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:40.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.182+0000 7f4aac840700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4aa80ae190 0x7f4aa80ae620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:40.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.182+0000 7f4aac840700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa80afbb0 con 0x7f4aa805fb00 2022-06-29T16:05:40.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.182+0000 7f4aac840700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aa8067de0 con 0x7f4aa805c040 2022-06-29T16:05:40.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.182+0000 7f4aac840700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa805fee0 con 0x7f4aa80ae190 2022-06-29T16:05:40.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.182+0000 7f4aa6ffd700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4aa805c040 0x7f4aa8142670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:40.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.182+0000 7f4aa7fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4aa80ae190 0x7f4aa80ae620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:40.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.182+0000 7f4aa6ffd700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4aa805c040 0x7f4aa8142670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:56700/0 (socket says 172.21.15.60:56700) 2022-06-29T16:05:40.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.182+0000 7f4aa6ffd700 1 -- 172.21.15.60:0/474905774 learned_addr learned my addr 172.21.15.60:0/474905774 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:40.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.182+0000 7f4aa77fe700 1 --1- 172.21.15.60:0/474905774 >> v1:172.21.15.92:6789/0 conn(0x7f4aa805fb00 0x7f4aa8141f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:34772/0 (socket says 172.21.15.60:34772) 2022-06-29T16:05:40.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.182+0000 7f4aa6ffd700 1 -- 172.21.15.60:0/474905774 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4aa80ae190 msgr2=0x7f4aa80ae620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:40.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.182+0000 7f4aa6ffd700 1 --2- 172.21.15.60:0/474905774 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4aa80ae190 0x7f4aa80ae620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:40.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.182+0000 7f4aa6ffd700 1 -- 172.21.15.60:0/474905774 >> v1:172.21.15.92:6789/0 conn(0x7f4aa805fb00 legacy=0x7f4aa8141f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:40.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.182+0000 7f4aa67fc700 1 -- 172.21.15.60:0/474905774 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2963919175 0 0) 0x7f4aa80afbb0 con 0x7f4aa805fb00 2022-06-29T16:05:40.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.182+0000 7f4aa6ffd700 1 -- 172.21.15.60:0/474905774 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa8146c80 con 0x7f4aa805c040 2022-06-29T16:05:40.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.182+0000 7f4aa6ffd700 1 --2- 172.21.15.60:0/474905774 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4aa805c040 0x7f4aa8142670 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f4a9c002f70 tx=0x7f4a9c006eb0).ready entity=mon.1 client_cookie=c97a4c76d27e1ce8 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:40.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.182+0000 7f4aa67fc700 1 -- 172.21.15.60:0/474905774 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a9c00d570 con 0x7f4aa805c040 2022-06-29T16:05:40.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.182+0000 7f4aa67fc700 1 -- 172.21.15.60:0/474905774 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a9c000e90 con 0x7f4aa805c040 2022-06-29T16:05:40.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.182+0000 7f4aa67fc700 1 -- 172.21.15.60:0/474905774 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a9c00d8c0 con 0x7f4aa805c040 2022-06-29T16:05:40.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.182+0000 7f4aac840700 1 -- 172.21.15.60:0/474905774 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4aa805c040 msgr2=0x7f4aa8142670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:40.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.182+0000 7f4aac840700 1 --2- 172.21.15.60:0/474905774 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4aa805c040 0x7f4aa8142670 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f4a9c002f70 tx=0x7f4a9c006eb0).stop 2022-06-29T16:05:40.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.182+0000 7f4aac840700 1 -- 172.21.15.60:0/474905774 shutdown_connections 2022-06-29T16:05:40.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.182+0000 7f4aac840700 1 --2- 172.21.15.60:0/474905774 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4aa80ae190 0x7f4aa80ae620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:40.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.182+0000 7f4aac840700 1 --2- 172.21.15.60:0/474905774 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4aa805c040 0x7f4aa8142670 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:40.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.182+0000 7f4aac840700 1 -- 172.21.15.60:0/474905774 >> 172.21.15.60:0/474905774 conn(0x7f4aa8063a60 msgr2=0x7f4aa805f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:40.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.182+0000 7f4aac840700 1 -- 172.21.15.60:0/474905774 shutdown_connections 2022-06-29T16:05:40.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.182+0000 7f4aac840700 1 -- 172.21.15.60:0/474905774 wait complete. 2022-06-29T16:05:40.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.186+0000 7f4aac840700 1 Processor -- start 2022-06-29T16:05:40.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.186+0000 7f4aac840700 1 -- start start 2022-06-29T16:05:40.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.186+0000 7f4aac840700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4aa805c040 0x7f4aa8140c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:40.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.186+0000 7f4aac840700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4aa805fb00 0x7f4aa80645b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:40.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.186+0000 7f4aac840700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4aa80ae190 0x7f4aa8064aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:40.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.186+0000 7f4aac840700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa80b6480 con 0x7f4aa80ae190 2022-06-29T16:05:40.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.186+0000 7f4aac840700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa80b5900 con 0x7f4aa805c040 2022-06-29T16:05:40.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.186+0000 7f4aac840700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa8067060 con 0x7f4aa805fb00 2022-06-29T16:05:40.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.186+0000 7f4aa77fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4aa805c040 0x7f4aa8140c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:40.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.186+0000 7f4aa7fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4aa80ae190 0x7f4aa8064aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:40.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.186+0000 7f4aa6ffd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4aa805fb00 0x7f4aa80645b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:40.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.186+0000 7f4aa7fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4aa80ae190 0x7f4aa8064aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50680/0 (socket says 172.21.15.60:50680) 2022-06-29T16:05:40.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.186+0000 7f4aa6ffd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4aa805fb00 0x7f4aa80645b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:35540/0 (socket says 172.21.15.60:35540) 2022-06-29T16:05:40.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.186+0000 7f4aa6ffd700 1 -- 172.21.15.60:0/1970092066 learned_addr learned my addr 172.21.15.60:0/1970092066 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:40.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.186+0000 7f4aa77fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4aa805c040 0x7f4aa8140c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:56706/0 (socket says 172.21.15.60:56706) 2022-06-29T16:05:40.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.186+0000 7f4aa7fff700 1 -- 172.21.15.60:0/1970092066 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4aa805fb00 msgr2=0x7f4aa80645b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:40.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.186+0000 7f4aa7fff700 1 --2- 172.21.15.60:0/1970092066 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4aa805fb00 0x7f4aa80645b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:40.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.186+0000 7f4aa7fff700 1 -- 172.21.15.60:0/1970092066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4aa805c040 msgr2=0x7f4aa8140c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:40.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.186+0000 7f4aa7fff700 1 --2- 172.21.15.60:0/1970092066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4aa805c040 0x7f4aa8140c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:40.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.186+0000 7f4aa7fff700 1 -- 172.21.15.60:0/1970092066 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a9c00b1e0 con 0x7f4aa80ae190 2022-06-29T16:05:40.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.190+0000 7f4aa7fff700 1 --2- 172.21.15.60:0/1970092066 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4aa80ae190 0x7f4aa8064aa0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 rx=0x7f4aa000c000 tx=0x7f4aa0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:40.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.190+0000 7f4aa4ff9700 1 -- 172.21.15.60:0/1970092066 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4aa0016070 con 0x7f4aa80ae190 2022-06-29T16:05:40.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.190+0000 7f4aac840700 1 -- 172.21.15.60:0/1970092066 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aa8067300 con 0x7f4aa80ae190 2022-06-29T16:05:40.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.190+0000 7f4aa4ff9700 1 -- 172.21.15.60:0/1970092066 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa0011750 con 0x7f4aa80ae190 2022-06-29T16:05:40.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.190+0000 7f4aa4ff9700 1 -- 172.21.15.60:0/1970092066 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4aa0011ab0 con 0x7f4aa80ae190 2022-06-29T16:05:40.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.190+0000 7f4aac840700 1 -- 172.21.15.60:0/1970092066 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa80b5210 con 0x7f4aa80ae190 2022-06-29T16:05:40.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.190+0000 7f4aac840700 1 -- 172.21.15.60:0/1970092066 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aa80573d0 con 0x7f4aa80ae190 2022-06-29T16:05:40.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.190+0000 7f4aa4ff9700 1 -- 172.21.15.60:0/1970092066 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4aa0011c10 con 0x7f4aa80ae190 2022-06-29T16:05:40.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.190+0000 7f4aa4ff9700 1 --2- 172.21.15.60:0/1970092066 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4a88053b30 0x7f4a88055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:40.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.190+0000 7f4aa4ff9700 1 -- 172.21.15.60:0/1970092066 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f4aa006be00 con 0x7f4aa80ae190 2022-06-29T16:05:40.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.194+0000 7f4aa77fe700 1 --2- 172.21.15.60:0/1970092066 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4a88053b30 0x7f4a88055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:40.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.194+0000 7f4aa77fe700 1 --2- 172.21.15.60:0/1970092066 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4a88053b30 0x7f4a88055ff0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f4a9800d000 tx=0x7f4a98007d80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:40.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.198+0000 7f4aa4ff9700 1 -- 172.21.15.60:0/1970092066 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4aa0042cf0 con 0x7f4aa80ae190 2022-06-29T16:05:40.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.362+0000 7f4aac840700 1 -- 172.21.15.60:0/1970092066 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f4aa80685d0 con 0x7f4aa80ae190 2022-06-29T16:05:40.382 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.374+0000 7f4aa4ff9700 1 -- 172.21.15.60:0/1970092066 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 17 and data pool 18 v40) v1 ==== 189+0+0 (secure 0 0 0) 0x7f4aa0046310 con 0x7f4aa80ae190 2022-06-29T16:05:40.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.378+0000 7f4aac840700 1 -- 172.21.15.60:0/1970092066 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4a88053b30 msgr2=0x7f4a88055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:40.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.378+0000 7f4aac840700 1 --2- 172.21.15.60:0/1970092066 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4a88053b30 0x7f4a88055ff0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f4a9800d000 tx=0x7f4a98007d80).stop 2022-06-29T16:05:40.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.378+0000 7f4aac840700 1 -- 172.21.15.60:0/1970092066 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4aa80ae190 msgr2=0x7f4aa8064aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:40.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.378+0000 7f4aac840700 1 --2- 172.21.15.60:0/1970092066 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4aa80ae190 0x7f4aa8064aa0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 rx=0x7f4aa000c000 tx=0x7f4aa0006df0).stop 2022-06-29T16:05:40.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.378+0000 7f4aac840700 1 -- 172.21.15.60:0/1970092066 shutdown_connections 2022-06-29T16:05:40.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.378+0000 7f4aac840700 1 --2- 172.21.15.60:0/1970092066 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4a88053b30 0x7f4a88055ff0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:40.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.378+0000 7f4aac840700 1 --2- 172.21.15.60:0/1970092066 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4aa80ae190 0x7f4aa8064aa0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:40.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.378+0000 7f4aac840700 1 --2- 172.21.15.60:0/1970092066 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4aa805fb00 0x7f4aa80645b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:40.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.378+0000 7f4aac840700 1 --2- 172.21.15.60:0/1970092066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4aa805c040 0x7f4aa8140c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:40.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.378+0000 7f4aac840700 1 -- 172.21.15.60:0/1970092066 >> 172.21.15.60:0/1970092066 conn(0x7f4aa8063a60 msgr2=0x7f4aa80b0730 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:40.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.378+0000 7f4aac840700 1 -- 172.21.15.60:0/1970092066 shutdown_connections 2022-06-29T16:05:40.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.378+0000 7f4aac840700 1 -- 172.21.15.60:0/1970092066 wait complete. 2022-06-29T16:05:40.390 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 17 and data pool 18 2022-06-29T16:05:40.402 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:05:40.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.486+0000 7f43d4499700 1 Processor -- start 2022-06-29T16:05:40.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.486+0000 7f43d4499700 1 -- start start 2022-06-29T16:05:40.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.486+0000 7f43d4499700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f43cc13f3d0 0x7f43cc060260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:40.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.486+0000 7f43d4499700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43cc0b6e20 con 0x7f43cc143820 2022-06-29T16:05:40.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.486+0000 7f43d4499700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43cc1430e0 con 0x7f43cc13f7b0 2022-06-29T16:05:40.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.486+0000 7f43d4499700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43cc142ef0 con 0x7f43cc13f3d0 2022-06-29T16:05:40.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.486+0000 7f43d1a34700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f43cc143820 0x7f43cc0b6710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58770/0 (socket says 172.21.15.60:58770) 2022-06-29T16:05:40.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.486+0000 7f43d1a34700 1 -- 172.21.15.60:0/331563135 learned_addr learned my addr 172.21.15.60:0/331563135 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:40.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.486+0000 7f43d2235700 1 --2- 172.21.15.60:0/331563135 >> v2:172.21.15.92:3300/0 conn(0x7f43cc13f3d0 0x7f43cc060260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:40.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.486+0000 7f43d1233700 1 -- 172.21.15.60:0/331563135 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 891504782 0 0) 0x7f43cc0b6e20 con 0x7f43cc143820 2022-06-29T16:05:40.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.486+0000 7f43d1233700 1 -- 172.21.15.60:0/331563135 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43ac003260 con 0x7f43cc143820 2022-06-29T16:05:40.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.486+0000 7f43d1233700 1 -- 172.21.15.60:0/331563135 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1284195251 0 0) 0x7f43ac003260 con 0x7f43cc143820 2022-06-29T16:05:40.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.486+0000 7f43d1233700 1 -- 172.21.15.60:0/331563135 >> v2:172.21.15.92:3300/0 conn(0x7f43cc13f3d0 msgr2=0x7f43cc060260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:40.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.486+0000 7f43d1233700 1 --2- 172.21.15.60:0/331563135 >> v2:172.21.15.92:3300/0 conn(0x7f43cc13f3d0 0x7f43cc060260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:40.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.486+0000 7f43d1233700 1 -- 172.21.15.60:0/331563135 >> v1:172.21.15.92:6789/0 conn(0x7f43cc13f7b0 legacy=0x7f43cc13fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:40.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.486+0000 7f43d1233700 1 -- 172.21.15.60:0/331563135 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43cc143380 con 0x7f43cc143820 2022-06-29T16:05:40.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.486+0000 7f43d1233700 1 -- 172.21.15.60:0/331563135 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f43b4001660 con 0x7f43cc143820 2022-06-29T16:05:40.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.486+0000 7f43d1233700 1 -- 172.21.15.60:0/331563135 >> v1:172.21.15.60:6789/0 conn(0x7f43cc143820 legacy=0x7f43cc0b6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:40.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.486+0000 7f43d1233700 1 --2- 172.21.15.60:0/331563135 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43ac003750 0x7f43ac003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:40.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.486+0000 7f43d1233700 1 --2- 172.21.15.60:0/331563135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43ac004500 0x7f43ac004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:40.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.486+0000 7f43d1233700 1 --2- 172.21.15.60:0/331563135 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43ac004e70 0x7f43ac00fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:40.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.486+0000 7f43d1233700 1 -- 172.21.15.60:0/331563135 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac0101e0 con 0x7f43ac004500 2022-06-29T16:05:40.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.486+0000 7f43d1233700 1 -- 172.21.15.60:0/331563135 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac005720 con 0x7f43ac004e70 2022-06-29T16:05:40.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.486+0000 7f43d1233700 1 -- 172.21.15.60:0/331563135 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac0058b0 con 0x7f43ac003750 2022-06-29T16:05:40.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.486+0000 7f43d1a34700 1 --2- 172.21.15.60:0/331563135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43ac004500 0x7f43ac004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:40.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.486+0000 7f43d2235700 1 --2- 172.21.15.60:0/331563135 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43ac003750 0x7f43ac003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:40.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.486+0000 7f43d2a36700 1 --2- 172.21.15.60:0/331563135 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43ac004e70 0x7f43ac00fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:40.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.486+0000 7f43d1a34700 1 -- 172.21.15.60:0/331563135 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43ac003750 msgr2=0x7f43ac003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:40.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.486+0000 7f43d1a34700 1 --2- 172.21.15.60:0/331563135 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43ac003750 0x7f43ac003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:40.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.486+0000 7f43d1a34700 1 -- 172.21.15.60:0/331563135 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43ac004e70 msgr2=0x7f43ac00fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:40.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.486+0000 7f43d1a34700 1 --2- 172.21.15.60:0/331563135 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43ac004e70 0x7f43ac00fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:40.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.486+0000 7f43d1a34700 1 -- 172.21.15.60:0/331563135 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43ac003260 con 0x7f43ac004500 2022-06-29T16:05:40.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.490+0000 7f43d2a36700 1 --2- 172.21.15.60:0/331563135 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43ac004e70 0x7f43ac00fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:40.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.490+0000 7f43d2a36700 1 -- 172.21.15.60:0/331563135 reap_dead start 2022-06-29T16:05:40.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.490+0000 7f43d1a34700 1 --2- 172.21.15.60:0/331563135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43ac004500 0x7f43ac004980 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 rx=0x7f43b4001370 tx=0x7f43cc143270).ready entity=mon.0 client_cookie=10a2cdd19e6205c6 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:40.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.490+0000 7f43d1233700 1 -- 172.21.15.60:0/331563135 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43b40070d0 con 0x7f43ac004500 2022-06-29T16:05:40.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.490+0000 7f43d1233700 1 -- 172.21.15.60:0/331563135 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43b4007230 con 0x7f43ac004500 2022-06-29T16:05:40.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.490+0000 7f43d1233700 1 -- 172.21.15.60:0/331563135 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43b400c8f0 con 0x7f43ac004500 2022-06-29T16:05:40.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.490+0000 7f43d4499700 1 -- 172.21.15.60:0/331563135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43ac004500 msgr2=0x7f43ac004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:40.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.490+0000 7f43d4499700 1 --2- 172.21.15.60:0/331563135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43ac004500 0x7f43ac004980 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 rx=0x7f43b4001370 tx=0x7f43cc143270).stop 2022-06-29T16:05:40.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.490+0000 7f43d4499700 1 -- 172.21.15.60:0/331563135 shutdown_connections 2022-06-29T16:05:40.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.490+0000 7f43d4499700 1 --2- 172.21.15.60:0/331563135 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43ac004500 0x7f43ac004980 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:40.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.490+0000 7f43d4499700 1 -- 172.21.15.60:0/331563135 >> 172.21.15.60:0/331563135 conn(0x7f43cc0643e0 msgr2=0x7f43cc0b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:40.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.490+0000 7f43d4499700 1 -- 172.21.15.60:0/331563135 shutdown_connections 2022-06-29T16:05:40.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.490+0000 7f43d4499700 1 -- 172.21.15.60:0/331563135 wait complete. 2022-06-29T16:05:40.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.490+0000 7f43d4499700 1 Processor -- start 2022-06-29T16:05:40.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.490+0000 7f43d4499700 1 -- start start 2022-06-29T16:05:40.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.494+0000 7f43d4499700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43ac004500 0x7f43cc145540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:40.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.494+0000 7f43d4499700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43cc067530 0x7f43cc064920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:40.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.494+0000 7f43d4499700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43cc098fd0 0x7f43cc0afa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:40.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.494+0000 7f43d4499700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43cc0aebb0 con 0x7f43cc067530 2022-06-29T16:05:40.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.494+0000 7f43d4499700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43cc13ef00 con 0x7f43cc098fd0 2022-06-29T16:05:40.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.494+0000 7f43d4499700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43cc066890 con 0x7f43ac004500 2022-06-29T16:05:40.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.494+0000 7f43d1a34700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43cc067530 0x7f43cc064920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:40.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.494+0000 7f43d1a34700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43cc067530 0x7f43cc064920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50698/0 (socket says 172.21.15.60:50698) 2022-06-29T16:05:40.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.494+0000 7f43d1a34700 1 -- 172.21.15.60:0/3541678139 learned_addr learned my addr 172.21.15.60:0/3541678139 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:40.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.494+0000 7f43d2235700 1 --2- 172.21.15.60:0/3541678139 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43ac004500 0x7f43cc145540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:40.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.494+0000 7f43d2a36700 1 --2- 172.21.15.60:0/3541678139 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43cc098fd0 0x7f43cc0afa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:40.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.494+0000 7f43d1a34700 1 -- 172.21.15.60:0/3541678139 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43ac004500 msgr2=0x7f43cc145540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:40.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.494+0000 7f43d1a34700 1 --2- 172.21.15.60:0/3541678139 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43ac004500 0x7f43cc145540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:40.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.494+0000 7f43d1a34700 1 -- 172.21.15.60:0/3541678139 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43cc098fd0 msgr2=0x7f43cc0afa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:40.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.494+0000 7f43d1a34700 1 --2- 172.21.15.60:0/3541678139 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43cc098fd0 0x7f43cc0afa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:40.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.494+0000 7f43d1a34700 1 -- 172.21.15.60:0/3541678139 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43cc0ae320 con 0x7f43cc067530 2022-06-29T16:05:40.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.494+0000 7f43d2235700 1 --2- 172.21.15.60:0/3541678139 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43ac004500 0x7f43cc145540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:05:40.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.494+0000 7f43d1a34700 1 --2- 172.21.15.60:0/3541678139 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43cc067530 0x7f43cc064920 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 rx=0x7f43b4001370 tx=0x7f43b4013000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:40.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.494+0000 7f43cb7fe700 1 -- 172.21.15.60:0/3541678139 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43b40070d0 con 0x7f43cc067530 2022-06-29T16:05:40.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.494+0000 7f43cb7fe700 1 -- 172.21.15.60:0/3541678139 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43b4001a50 con 0x7f43cc067530 2022-06-29T16:05:40.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.494+0000 7f43d4499700 1 -- 172.21.15.60:0/3541678139 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43cc066480 con 0x7f43cc067530 2022-06-29T16:05:40.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.494+0000 7f43cb7fe700 1 -- 172.21.15.60:0/3541678139 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43b4004730 con 0x7f43cc067530 2022-06-29T16:05:40.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.494+0000 7f43d4499700 1 -- 172.21.15.60:0/3541678139 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43cc0b0110 con 0x7f43cc067530 2022-06-29T16:05:40.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.498+0000 7f43d4499700 1 -- 172.21.15.60:0/3541678139 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43cc0ae170 con 0x7f43cc067530 2022-06-29T16:05:40.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.498+0000 7f43cb7fe700 1 -- 172.21.15.60:0/3541678139 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f43b401c070 con 0x7f43cc067530 2022-06-29T16:05:40.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.498+0000 7f43cb7fe700 1 --2- 172.21.15.60:0/3541678139 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f43b0053950 0x7f43b0055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:40.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.498+0000 7f43cb7fe700 1 -- 172.21.15.60:0/3541678139 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f43b406bdf0 con 0x7f43cc067530 2022-06-29T16:05:40.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.498+0000 7f43d2235700 1 --2- 172.21.15.60:0/3541678139 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f43b0053950 0x7f43b0055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:40.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.498+0000 7f43d2235700 1 --2- 172.21.15.60:0/3541678139 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f43b0053950 0x7f43b0055e10 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f43bc00b000 tx=0x7f43bc0091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:40.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.502+0000 7f43cb7fe700 1 -- 172.21.15.60:0/3541678139 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f43b4042d10 con 0x7f43cc067530 2022-06-29T16:05:40.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.630+0000 7f43d4499700 1 -- 172.21.15.60:0/3541678139 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43cc066a50 con 0x7f43cc067530 2022-06-29T16:05:40.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.630+0000 7f43cb7fe700 1 -- 172.21.15.60:0/3541678139 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+16054 (secure 0 0 0) 0x7f43b4046330 con 0x7f43cc067530 2022-06-29T16:05:40.640 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:40.640 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":90,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:05:40.369622+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:05:38.407939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2022-06-29T16:05:39.380608+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:05:40.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.634+0000 7f43d4499700 1 -- 172.21.15.60:0/3541678139 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f43b0053950 msgr2=0x7f43b0055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:40.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.634+0000 7f43d4499700 1 --2- 172.21.15.60:0/3541678139 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f43b0053950 0x7f43b0055e10 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f43bc00b000 tx=0x7f43bc0091d0).stop 2022-06-29T16:05:40.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.634+0000 7f43d4499700 1 -- 172.21.15.60:0/3541678139 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43cc067530 msgr2=0x7f43cc064920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:40.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.634+0000 7f43d4499700 1 --2- 172.21.15.60:0/3541678139 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43cc067530 0x7f43cc064920 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 rx=0x7f43b4001370 tx=0x7f43b4013000).stop 2022-06-29T16:05:40.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.634+0000 7f43d4499700 1 -- 172.21.15.60:0/3541678139 shutdown_connections 2022-06-29T16:05:40.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.634+0000 7f43d4499700 1 --2- 172.21.15.60:0/3541678139 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f43b0053950 0x7f43b0055e10 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:40.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.634+0000 7f43d4499700 1 --2- 172.21.15.60:0/3541678139 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43cc098fd0 0x7f43cc0afa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:40.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.634+0000 7f43d4499700 1 --2- 172.21.15.60:0/3541678139 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f43cc067530 0x7f43cc064920 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:40.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.634+0000 7f43d4499700 1 --2- 172.21.15.60:0/3541678139 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f43ac004500 0x7f43cc145540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:40.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.634+0000 7f43d4499700 1 -- 172.21.15.60:0/3541678139 >> 172.21.15.60:0/3541678139 conn(0x7f43cc0643e0 msgr2=0x7f43cc0b52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:40.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.634+0000 7f43d4499700 1 -- 172.21.15.60:0/3541678139 shutdown_connections 2022-06-29T16:05:40.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.634+0000 7f43d4499700 1 -- 172.21.15.60:0/3541678139 wait complete. 2022-06-29T16:05:40.654 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:05:40.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.750+0000 7fb9a3fff700 1 Processor -- start 2022-06-29T16:05:40.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.750+0000 7fb9a3fff700 1 -- start start 2022-06-29T16:05:40.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.750+0000 7fb9a3fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb9a40b5eb0 0x7fb9a4142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:40.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.750+0000 7fb9a3fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb9a40b2c50 0x7fb9a40b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:40.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.750+0000 7fb9a3fff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a40b0cf0 con 0x7fb9a40b48b0 2022-06-29T16:05:40.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.750+0000 7fb9a3fff700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9a40b54e0 con 0x7fb9a40b5eb0 2022-06-29T16:05:40.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.750+0000 7fb9a3fff700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9a40b4d10 con 0x7fb9a40b2c50 2022-06-29T16:05:40.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.750+0000 7fb9a27fc700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb9a40b48b0 0x7fb9a40b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58790/0 (socket says 172.21.15.60:58790) 2022-06-29T16:05:40.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.750+0000 7fb9a27fc700 1 -- 172.21.15.60:0/1339214605 learned_addr learned my addr 172.21.15.60:0/1339214605 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:40.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.750+0000 7fb9a2ffd700 1 --2- 172.21.15.60:0/1339214605 >> v2:172.21.15.60:3300/0 conn(0x7fb9a40b5eb0 0x7fb9a4142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:40.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.750+0000 7fb9a37fe700 1 --2- 172.21.15.60:0/1339214605 >> v2:172.21.15.92:3300/0 conn(0x7fb9a40b2c50 0x7fb9a40b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:40.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.750+0000 7fb9a1ffb700 1 -- 172.21.15.60:0/1339214605 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 822211156 0 0) 0x7fb98c001110 con 0x7fb9a40b48b0 2022-06-29T16:05:40.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.750+0000 7fb9a1ffb700 1 -- 172.21.15.60:0/1339214605 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb984003260 con 0x7fb9a40b48b0 2022-06-29T16:05:40.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.750+0000 7fb9a2ffd700 1 -- 172.21.15.60:0/1339214605 >> v2:172.21.15.92:3300/0 conn(0x7fb9a40b2c50 msgr2=0x7fb9a40b30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:40.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.750+0000 7fb9a2ffd700 1 --2- 172.21.15.60:0/1339214605 >> v2:172.21.15.92:3300/0 conn(0x7fb9a40b2c50 0x7fb9a40b30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:40.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.750+0000 7fb9a2ffd700 1 -- 172.21.15.60:0/1339214605 >> v1:172.21.15.60:6789/0 conn(0x7fb9a40b48b0 legacy=0x7fb9a40b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:40.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.750+0000 7fb9a1ffb700 1 -- 172.21.15.60:0/1339214605 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 151872420 0 0) 0x7fb984003260 con 0x7fb9a40b48b0 2022-06-29T16:05:40.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.750+0000 7fb9a2ffd700 1 -- 172.21.15.60:0/1339214605 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a40b6290 con 0x7fb9a40b5eb0 2022-06-29T16:05:40.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.750+0000 7fb9a2ffd700 1 --2- 172.21.15.60:0/1339214605 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9a40b5eb0 0x7fb9a4142a20 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 rx=0x7fb98c001990 tx=0x7fb994006df0).ready entity=mon.0 client_cookie=c9add43cbfddae59 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:40.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.750+0000 7fb9a1ffb700 1 -- 172.21.15.60:0/1339214605 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb98c0017f0 con 0x7fb9a40b5eb0 2022-06-29T16:05:40.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.750+0000 7fb9a1ffb700 1 -- 172.21.15.60:0/1339214605 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb994000e90 con 0x7fb9a40b5eb0 2022-06-29T16:05:40.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.750+0000 7fb9a1ffb700 1 -- 172.21.15.60:0/1339214605 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb994012460 con 0x7fb9a40b5eb0 2022-06-29T16:05:40.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.750+0000 7fb9a3fff700 1 -- 172.21.15.60:0/1339214605 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9a40b5eb0 msgr2=0x7fb9a4142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:40.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.750+0000 7fb9a3fff700 1 --2- 172.21.15.60:0/1339214605 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9a40b5eb0 0x7fb9a4142a20 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 rx=0x7fb98c001990 tx=0x7fb994006df0).stop 2022-06-29T16:05:40.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.754+0000 7fb9a3fff700 1 -- 172.21.15.60:0/1339214605 shutdown_connections 2022-06-29T16:05:40.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.754+0000 7fb9a3fff700 1 --2- 172.21.15.60:0/1339214605 >> v2:172.21.15.92:3300/0 conn(0x7fb9a40b2c50 0x7fb9a40b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:40.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.754+0000 7fb9a3fff700 1 --2- 172.21.15.60:0/1339214605 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9a40b5eb0 0x7fb9a4142a20 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:40.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.754+0000 7fb9a3fff700 1 -- 172.21.15.60:0/1339214605 >> 172.21.15.60:0/1339214605 conn(0x7fb9a4062090 msgr2=0x7fb9a4062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:40.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.754+0000 7fb9a3fff700 1 -- 172.21.15.60:0/1339214605 shutdown_connections 2022-06-29T16:05:40.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.754+0000 7fb9a3fff700 1 -- 172.21.15.60:0/1339214605 wait complete. 2022-06-29T16:05:40.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.754+0000 7fb9a3fff700 1 Processor -- start 2022-06-29T16:05:40.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.754+0000 7fb9a3fff700 1 -- start start 2022-06-29T16:05:40.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.754+0000 7fb9a3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9a40b2c50 0x7fb9a4144220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:40.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.758+0000 7fb9a3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb9a40b48b0 0x7fb9a413ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:40.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.758+0000 7fb9a3fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9a40b5eb0 0x7fb9a413f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:40.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.758+0000 7fb9a3fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a40adf50 con 0x7fb9a40b2c50 2022-06-29T16:05:40.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.758+0000 7fb9a3fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a40aec50 con 0x7fb9a40b48b0 2022-06-29T16:05:40.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.758+0000 7fb9a3fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a40af330 con 0x7fb9a40b5eb0 2022-06-29T16:05:40.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.758+0000 7fb9a2ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9a40b2c50 0x7fb9a4144220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:40.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.758+0000 7fb9a2ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9a40b2c50 0x7fb9a4144220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50710/0 (socket says 172.21.15.60:50710) 2022-06-29T16:05:40.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.758+0000 7fb9a2ffd700 1 -- 172.21.15.60:0/3265989014 learned_addr learned my addr 172.21.15.60:0/3265989014 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:40.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.758+0000 7fb9a27fc700 1 --2- 172.21.15.60:0/3265989014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb9a40b48b0 0x7fb9a413ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:40.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.758+0000 7fb9a37fe700 1 --2- 172.21.15.60:0/3265989014 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9a40b5eb0 0x7fb9a413f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:40.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.758+0000 7fb9a2ffd700 1 -- 172.21.15.60:0/3265989014 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9a40b5eb0 msgr2=0x7fb9a413f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:40.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.758+0000 7fb9a2ffd700 1 --2- 172.21.15.60:0/3265989014 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9a40b5eb0 0x7fb9a413f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:40.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.758+0000 7fb9a2ffd700 1 -- 172.21.15.60:0/3265989014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb9a40b48b0 msgr2=0x7fb9a413ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:40.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.758+0000 7fb9a2ffd700 1 --2- 172.21.15.60:0/3265989014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb9a40b48b0 0x7fb9a413ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:40.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.758+0000 7fb9a2ffd700 1 -- 172.21.15.60:0/3265989014 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb99400b180 con 0x7fb9a40b2c50 2022-06-29T16:05:40.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.758+0000 7fb9a2ffd700 1 --2- 172.21.15.60:0/3265989014 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9a40b2c50 0x7fb9a4144220 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 rx=0x7fb98c001990 tx=0x7fb994014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:40.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.758+0000 7fb99bfff700 1 -- 172.21.15.60:0/3265989014 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb994016070 con 0x7fb9a40b2c50 2022-06-29T16:05:40.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.758+0000 7fb99bfff700 1 -- 172.21.15.60:0/3265989014 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb99400d6e0 con 0x7fb9a40b2c50 2022-06-29T16:05:40.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.758+0000 7fb99bfff700 1 -- 172.21.15.60:0/3265989014 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb99400da40 con 0x7fb9a40b2c50 2022-06-29T16:05:40.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.758+0000 7fb9a3fff700 1 -- 172.21.15.60:0/3265989014 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9a40ae170 con 0x7fb9a40b2c50 2022-06-29T16:05:40.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.758+0000 7fb9a3fff700 1 -- 172.21.15.60:0/3265989014 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9a413f660 con 0x7fb9a40b2c50 2022-06-29T16:05:40.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.758+0000 7fb9a3fff700 1 -- 172.21.15.60:0/3265989014 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9a4142f10 con 0x7fb9a40b2c50 2022-06-29T16:05:40.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.758+0000 7fb99bfff700 1 -- 172.21.15.60:0/3265989014 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb994025030 con 0x7fb9a40b2c50 2022-06-29T16:05:40.777 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.758+0000 7fb99bfff700 1 --2- 172.21.15.60:0/3265989014 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb988053a70 0x7fb988055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:40.777 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.758+0000 7fb99bfff700 1 -- 172.21.15.60:0/3265989014 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb99406cd60 con 0x7fb9a40b2c50 2022-06-29T16:05:40.777 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.758+0000 7fb9a27fc700 1 --2- 172.21.15.60:0/3265989014 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb988053a70 0x7fb988055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:40.777 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.762+0000 7fb9a27fc700 1 --2- 172.21.15.60:0/3265989014 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb988053a70 0x7fb988055f30 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fb98c00d000 tx=0x7fb98c000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:40.778 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.766+0000 7fb99bfff700 1 -- 172.21.15.60:0/3265989014 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb994043c80 con 0x7fb9a40b2c50 2022-06-29T16:05:40.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.894+0000 7fb9a3fff700 1 -- 172.21.15.60:0/3265989014 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9a40573d0 con 0x7fb9a40b2c50 2022-06-29T16:05:40.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.894+0000 7fb99bfff700 1 -- 172.21.15.60:0/3265989014 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+16054 (secure 0 0 0) 0x7fb994023020 con 0x7fb9a40b2c50 2022-06-29T16:05:40.903 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:40.903 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":90,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:05:40.369622+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:05:38.407939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2022-06-29T16:05:39.380608+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:05:40.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.898+0000 7fb9a3fff700 1 -- 172.21.15.60:0/3265989014 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb988053a70 msgr2=0x7fb988055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:40.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.898+0000 7fb9a3fff700 1 --2- 172.21.15.60:0/3265989014 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb988053a70 0x7fb988055f30 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fb98c00d000 tx=0x7fb98c000bc0).stop 2022-06-29T16:05:40.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.898+0000 7fb9a3fff700 1 -- 172.21.15.60:0/3265989014 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9a40b2c50 msgr2=0x7fb9a4144220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:40.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.898+0000 7fb9a3fff700 1 --2- 172.21.15.60:0/3265989014 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9a40b2c50 0x7fb9a4144220 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 rx=0x7fb98c001990 tx=0x7fb994014000).stop 2022-06-29T16:05:40.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.898+0000 7fb9a3fff700 1 -- 172.21.15.60:0/3265989014 shutdown_connections 2022-06-29T16:05:40.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.898+0000 7fb9a3fff700 1 --2- 172.21.15.60:0/3265989014 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb988053a70 0x7fb988055f30 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:40.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.898+0000 7fb9a3fff700 1 --2- 172.21.15.60:0/3265989014 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9a40b5eb0 0x7fb9a413f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:40.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.898+0000 7fb9a3fff700 1 --2- 172.21.15.60:0/3265989014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb9a40b48b0 0x7fb9a413ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:40.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.898+0000 7fb9a3fff700 1 --2- 172.21.15.60:0/3265989014 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9a40b2c50 0x7fb9a4144220 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:40.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.898+0000 7fb9a3fff700 1 -- 172.21.15.60:0/3265989014 >> 172.21.15.60:0/3265989014 conn(0x7fb9a4062090 msgr2=0x7fb9a4062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:40.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.898+0000 7fb9a3fff700 1 -- 172.21.15.60:0/3265989014 shutdown_connections 2022-06-29T16:05:40.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:40.898+0000 7fb9a3fff700 1 -- 172.21.15.60:0/3265989014 wait complete. 2022-06-29T16:05:40.917 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-06-29T16:05:41.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.022+0000 7f593b59e700 1 Processor -- start 2022-06-29T16:05:41.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.022+0000 7f593b59e700 1 -- start start 2022-06-29T16:05:41.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.022+0000 7f593b59e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f593c0b3b40 0x7f593c14d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:41.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.022+0000 7f593b59e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f593c0b9fd0 con 0x7f593c0b0890 2022-06-29T16:05:41.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.022+0000 7f593b59e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f593c0b1a50 con 0x7f593c0b9bf0 2022-06-29T16:05:41.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.022+0000 7f593b59e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f593c0b1cf0 con 0x7f593c0b3b40 2022-06-29T16:05:41.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.022+0000 7f5933fff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f593c0b0890 0x7f593c0b4540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58808/0 (socket says 172.21.15.60:58808) 2022-06-29T16:05:41.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.022+0000 7f5933fff700 1 -- 172.21.15.60:0/2893679197 learned_addr learned my addr 172.21.15.60:0/2893679197 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:41.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.022+0000 7f593ad9d700 1 --2- 172.21.15.60:0/2893679197 >> v2:172.21.15.92:3300/0 conn(0x7f593c0b3b40 0x7f593c14d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:41.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.022+0000 7f5939d9b700 1 -- 172.21.15.60:0/2893679197 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 308253449 0 0) 0x7f593c0b9fd0 con 0x7f593c0b0890 2022-06-29T16:05:41.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.022+0000 7f5939d9b700 1 -- 172.21.15.60:0/2893679197 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f591c003260 con 0x7f593c0b0890 2022-06-29T16:05:41.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.022+0000 7f593ad9d700 1 -- 172.21.15.60:0/2893679197 >> v1:172.21.15.92:6789/0 conn(0x7f593c0b9bf0 legacy=0x7f593c0b2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:41.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.022+0000 7f593ad9d700 1 -- 172.21.15.60:0/2893679197 >> v1:172.21.15.60:6789/0 conn(0x7f593c0b0890 legacy=0x7f593c0b4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:41.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.022+0000 7f593ad9d700 1 -- 172.21.15.60:0/2893679197 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f593c0b3470 con 0x7f593c0b3b40 2022-06-29T16:05:41.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.022+0000 7f593ad9d700 1 --2- 172.21.15.60:0/2893679197 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f593c0b3b40 0x7f593c14d6b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f59340032f0 tx=0x7f59340091d0).ready entity=mon.1 client_cookie=b456e24596e9bec3 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:41.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.022+0000 7f5939d9b700 1 -- 172.21.15.60:0/2893679197 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f593400b6c0 con 0x7f593c0b3b40 2022-06-29T16:05:41.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.022+0000 7f5939d9b700 1 -- 172.21.15.60:0/2893679197 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593400b820 con 0x7f593c0b3b40 2022-06-29T16:05:41.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.022+0000 7f5939d9b700 1 -- 172.21.15.60:0/2893679197 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f593400b990 con 0x7f593c0b3b40 2022-06-29T16:05:41.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.026+0000 7f593b59e700 1 -- 172.21.15.60:0/2893679197 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f593c0b3b40 msgr2=0x7f593c14d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:41.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.026+0000 7f593b59e700 1 --2- 172.21.15.60:0/2893679197 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f593c0b3b40 0x7f593c14d6b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f59340032f0 tx=0x7f59340091d0).stop 2022-06-29T16:05:41.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.026+0000 7f593b59e700 1 -- 172.21.15.60:0/2893679197 shutdown_connections 2022-06-29T16:05:41.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.026+0000 7f593b59e700 1 --2- 172.21.15.60:0/2893679197 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f593c0b3b40 0x7f593c14d6b0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:41.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.026+0000 7f593b59e700 1 -- 172.21.15.60:0/2893679197 >> 172.21.15.60:0/2893679197 conn(0x7f593c062b60 msgr2=0x7f593c0b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:41.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.026+0000 7f593b59e700 1 -- 172.21.15.60:0/2893679197 shutdown_connections 2022-06-29T16:05:41.042 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.026+0000 7f593b59e700 1 -- 172.21.15.60:0/2893679197 wait complete. 2022-06-29T16:05:41.042 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.026+0000 7f593b59e700 1 Processor -- start 2022-06-29T16:05:41.042 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.026+0000 7f593b59e700 1 -- start start 2022-06-29T16:05:41.042 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.030+0000 7f593b59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f593c0b0890 0x7f593c13d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:41.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.030+0000 7f593b59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f593c0b3b40 0x7f593c13db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:41.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.030+0000 7f593b59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f593c0b9bf0 0x7f593c146cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:41.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.030+0000 7f593b59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593c152640 con 0x7f593c0b9bf0 2022-06-29T16:05:41.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.030+0000 7f593b59e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593c152830 con 0x7f593c0b3b40 2022-06-29T16:05:41.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.030+0000 7f593b59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f593c152ad0 con 0x7f593c0b0890 2022-06-29T16:05:41.044 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.030+0000 7f593ad9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f593c0b9bf0 0x7f593c146cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:41.044 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.030+0000 7f593ad9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f593c0b9bf0 0x7f593c146cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50728/0 (socket says 172.21.15.60:50728) 2022-06-29T16:05:41.044 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.030+0000 7f593ad9d700 1 -- 172.21.15.60:0/4103811535 learned_addr learned my addr 172.21.15.60:0/4103811535 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:41.044 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.030+0000 7f5933fff700 1 --2- 172.21.15.60:0/4103811535 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f593c0b3b40 0x7f593c13db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:41.045 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.030+0000 7f593a59c700 1 --2- 172.21.15.60:0/4103811535 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f593c0b0890 0x7f593c13d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:41.045 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.030+0000 7f593ad9d700 1 -- 172.21.15.60:0/4103811535 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f593c0b0890 msgr2=0x7f593c13d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:41.045 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.030+0000 7f593ad9d700 1 --2- 172.21.15.60:0/4103811535 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f593c0b0890 0x7f593c13d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:41.045 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.030+0000 7f593ad9d700 1 -- 172.21.15.60:0/4103811535 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f593c0b3b40 msgr2=0x7f593c13db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:41.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.030+0000 7f593ad9d700 1 --2- 172.21.15.60:0/4103811535 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f593c0b3b40 0x7f593c13db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:41.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.030+0000 7f593ad9d700 1 -- 172.21.15.60:0/4103811535 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5934009470 con 0x7f593c0b9bf0 2022-06-29T16:05:41.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.030+0000 7f593a59c700 1 --2- 172.21.15.60:0/4103811535 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f593c0b0890 0x7f593c13d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:05:41.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.030+0000 7f593ad9d700 1 --2- 172.21.15.60:0/4103811535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f593c0b9bf0 0x7f593c146cc0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 rx=0x7f59340032f0 tx=0x7f5934016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:41.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.030+0000 7f59337fe700 1 -- 172.21.15.60:0/4103811535 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f593400b6c0 con 0x7f593c0b9bf0 2022-06-29T16:05:41.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.030+0000 7f59337fe700 1 -- 172.21.15.60:0/4103811535 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59340107a0 con 0x7f593c0b9bf0 2022-06-29T16:05:41.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.030+0000 7f593b59e700 1 -- 172.21.15.60:0/4103811535 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f593c0aedd0 con 0x7f593c0b9bf0 2022-06-29T16:05:41.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.030+0000 7f59337fe700 1 -- 172.21.15.60:0/4103811535 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5934010b00 con 0x7f593c0b9bf0 2022-06-29T16:05:41.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.030+0000 7f593b59e700 1 -- 172.21.15.60:0/4103811535 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f593c0af320 con 0x7f593c0b9bf0 2022-06-29T16:05:41.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.030+0000 7f593b59e700 1 -- 172.21.15.60:0/4103811535 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f593c152c90 con 0x7f593c0b9bf0 2022-06-29T16:05:41.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.030+0000 7f59337fe700 1 -- 172.21.15.60:0/4103811535 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f593400f070 con 0x7f593c0b9bf0 2022-06-29T16:05:41.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.030+0000 7f59337fe700 1 --2- 172.21.15.60:0/4103811535 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5920053b00 0x7f5920055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:41.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.034+0000 7f59337fe700 1 -- 172.21.15.60:0/4103811535 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f593406d6b0 con 0x7f593c0b9bf0 2022-06-29T16:05:41.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.034+0000 7f593a59c700 1 --2- 172.21.15.60:0/4103811535 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5920053b00 0x7f5920055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:41.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.034+0000 7f593a59c700 1 --2- 172.21.15.60:0/4103811535 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5920053b00 0x7f5920055fc0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f592c008000 tx=0x7f592c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:41.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.038+0000 7f59337fe700 1 -- 172.21.15.60:0/4103811535 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f59340444c0 con 0x7f593c0b9bf0 2022-06-29T16:05:41.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.198+0000 7f593b59e700 1 -- 172.21.15.60:0/4103811535 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f593c0573d0 con 0x7f593c0b9bf0 2022-06-29T16:05:41.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.382+0000 7f59337fe700 1 -- 172.21.15.60:0/4103811535 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v41) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5934015020 con 0x7f593c0b9bf0 2022-06-29T16:05:41.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.386+0000 7f593b59e700 1 -- 172.21.15.60:0/4103811535 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5920053b00 msgr2=0x7f5920055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:41.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.386+0000 7f593b59e700 1 --2- 172.21.15.60:0/4103811535 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5920053b00 0x7f5920055fc0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f592c008000 tx=0x7f592c009040).stop 2022-06-29T16:05:41.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.386+0000 7f593b59e700 1 -- 172.21.15.60:0/4103811535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f593c0b9bf0 msgr2=0x7f593c146cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:41.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.386+0000 7f593b59e700 1 --2- 172.21.15.60:0/4103811535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f593c0b9bf0 0x7f593c146cc0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 rx=0x7f59340032f0 tx=0x7f5934016000).stop 2022-06-29T16:05:41.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.386+0000 7f593b59e700 1 -- 172.21.15.60:0/4103811535 shutdown_connections 2022-06-29T16:05:41.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.386+0000 7f593b59e700 1 --2- 172.21.15.60:0/4103811535 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5920053b00 0x7f5920055fc0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:41.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.386+0000 7f593b59e700 1 --2- 172.21.15.60:0/4103811535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f593c0b9bf0 0x7f593c146cc0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:41.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.386+0000 7f593b59e700 1 --2- 172.21.15.60:0/4103811535 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f593c0b3b40 0x7f593c13db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:41.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.386+0000 7f593b59e700 1 --2- 172.21.15.60:0/4103811535 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f593c0b0890 0x7f593c13d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:41.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.386+0000 7f593b59e700 1 -- 172.21.15.60:0/4103811535 >> 172.21.15.60:0/4103811535 conn(0x7f593c062b60 msgr2=0x7f593c0b5bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:41.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.386+0000 7f593b59e700 1 -- 172.21.15.60:0/4103811535 shutdown_connections 2022-06-29T16:05:41.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.386+0000 7f593b59e700 1 -- 172.21.15.60:0/4103811535 wait complete. 2022-06-29T16:05:41.409 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:05:41.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.506+0000 7f54ea453700 1 Processor -- start 2022-06-29T16:05:41.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.506+0000 7f54ea453700 1 -- start start 2022-06-29T16:05:41.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.506+0000 7f54ea453700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f54e4063ad0 0x7f54e4067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:41.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.506+0000 7f54ea453700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f54e414af10 0x7f54e405c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:41.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.506+0000 7f54ea453700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e40acba0 con 0x7f54e40b5950 2022-06-29T16:05:41.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.506+0000 7f54ea453700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54e414b3a0 con 0x7f54e414af10 2022-06-29T16:05:41.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.506+0000 7f54ea453700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54e4067220 con 0x7f54e4063ad0 2022-06-29T16:05:41.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.506+0000 7f54e89f0700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f54e414af10 0x7f54e405c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:41.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.506+0000 7f54e89f0700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f54e414af10 0x7f54e405c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50736/0 (socket says 172.21.15.60:50736) 2022-06-29T16:05:41.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.506+0000 7f54e89f0700 1 -- 172.21.15.60:0/4033741824 learned_addr learned my addr 172.21.15.60:0/4033741824 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:41.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.506+0000 7f54e37fe700 1 --2- 172.21.15.60:0/4033741824 >> v2:172.21.15.92:3300/0 conn(0x7f54e4063ad0 0x7f54e4067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:41.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.506+0000 7f54e89f0700 1 -- 172.21.15.60:0/4033741824 >> v2:172.21.15.92:3300/0 conn(0x7f54e4063ad0 msgr2=0x7f54e4067680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:41.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.506+0000 7f54e89f0700 1 --2- 172.21.15.60:0/4033741824 >> v2:172.21.15.92:3300/0 conn(0x7f54e4063ad0 0x7f54e4067680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:41.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.506+0000 7f54e89f0700 1 -- 172.21.15.60:0/4033741824 >> v1:172.21.15.92:6789/0 conn(0x7f54e40b5950 legacy=0x7f54e405bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:41.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.506+0000 7f54e89f0700 1 -- 172.21.15.60:0/4033741824 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e40b5d30 con 0x7f54e414af10 2022-06-29T16:05:41.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.506+0000 7f54e89f0700 1 --2- 172.21.15.60:0/4033741824 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54e414af10 0x7f54e405c430 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 rx=0x7f54dc002f70 tx=0x7f54dc006eb0).ready entity=mon.0 client_cookie=ee26814d0b4f8dad server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:41.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.506+0000 7f54e2ffd700 1 -- 172.21.15.60:0/4033741824 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54dc00d5c0 con 0x7f54e414af10 2022-06-29T16:05:41.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.506+0000 7f54e2ffd700 1 -- 172.21.15.60:0/4033741824 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54dc000e90 con 0x7f54e414af10 2022-06-29T16:05:41.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.506+0000 7f54e2ffd700 1 -- 172.21.15.60:0/4033741824 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54dc00d950 con 0x7f54e414af10 2022-06-29T16:05:41.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.506+0000 7f54ea453700 1 -- 172.21.15.60:0/4033741824 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54e414af10 msgr2=0x7f54e405c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:41.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.506+0000 7f54ea453700 1 --2- 172.21.15.60:0/4033741824 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54e414af10 0x7f54e405c430 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 rx=0x7f54dc002f70 tx=0x7f54dc006eb0).stop 2022-06-29T16:05:41.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.506+0000 7f54ea453700 1 -- 172.21.15.60:0/4033741824 shutdown_connections 2022-06-29T16:05:41.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.506+0000 7f54ea453700 1 --2- 172.21.15.60:0/4033741824 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54e414af10 0x7f54e405c430 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:41.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.506+0000 7f54ea453700 1 --2- 172.21.15.60:0/4033741824 >> v2:172.21.15.92:3300/0 conn(0x7f54e4063ad0 0x7f54e4067680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:41.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.506+0000 7f54ea453700 1 -- 172.21.15.60:0/4033741824 >> 172.21.15.60:0/4033741824 conn(0x7f54e40632c0 msgr2=0x7f54e40b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:41.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.506+0000 7f54ea453700 1 -- 172.21.15.60:0/4033741824 shutdown_connections 2022-06-29T16:05:41.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.506+0000 7f54ea453700 1 -- 172.21.15.60:0/4033741824 wait complete. 2022-06-29T16:05:41.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.510+0000 7f54ea453700 1 Processor -- start 2022-06-29T16:05:41.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.510+0000 7f54ea453700 1 -- start start 2022-06-29T16:05:41.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.510+0000 7f54ea453700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54e4063ad0 0x7f54e414bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:41.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.510+0000 7f54ea453700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54e40b5950 0x7f54e4141f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:41.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.510+0000 7f54ea453700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54e414af10 0x7f54e4142470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:41.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.510+0000 7f54ea453700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e40af700 con 0x7f54e4063ad0 2022-06-29T16:05:41.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.510+0000 7f54ea453700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e40ad2a0 con 0x7f54e414af10 2022-06-29T16:05:41.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.510+0000 7f54ea453700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54e40aeb20 con 0x7f54e40b5950 2022-06-29T16:05:41.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.510+0000 7f54e3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54e4063ad0 0x7f54e414bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:41.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.510+0000 7f54e3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54e4063ad0 0x7f54e414bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50738/0 (socket says 172.21.15.60:50738) 2022-06-29T16:05:41.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.510+0000 7f54e3fff700 1 -- 172.21.15.60:0/3189906275 learned_addr learned my addr 172.21.15.60:0/3189906275 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:41.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.510+0000 7f54e37fe700 1 --2- 172.21.15.60:0/3189906275 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54e40b5950 0x7f54e4141f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:41.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.510+0000 7f54e89f0700 1 --2- 172.21.15.60:0/3189906275 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54e414af10 0x7f54e4142470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:41.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.510+0000 7f54e3fff700 1 -- 172.21.15.60:0/3189906275 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54e40b5950 msgr2=0x7f54e4141f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:41.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.510+0000 7f54e3fff700 1 --2- 172.21.15.60:0/3189906275 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54e40b5950 0x7f54e4141f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:41.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.510+0000 7f54e3fff700 1 -- 172.21.15.60:0/3189906275 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54e414af10 msgr2=0x7f54e4142470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:41.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.510+0000 7f54e3fff700 1 --2- 172.21.15.60:0/3189906275 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54e414af10 0x7f54e4142470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:41.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.510+0000 7f54e3fff700 1 -- 172.21.15.60:0/3189906275 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54dc00b1e0 con 0x7f54e4063ad0 2022-06-29T16:05:41.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.510+0000 7f54e89f0700 1 --2- 172.21.15.60:0/3189906275 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54e414af10 0x7f54e4142470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:41.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.510+0000 7f54e3fff700 1 --2- 172.21.15.60:0/3189906275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54e4063ad0 0x7f54e414bdb0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 rx=0x7f54e40ad0a0 tx=0x7f54cc007be0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:41.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.510+0000 7f54e17fa700 1 -- 172.21.15.60:0/3189906275 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54cc00ce00 con 0x7f54e4063ad0 2022-06-29T16:05:41.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.510+0000 7f54e17fa700 1 -- 172.21.15.60:0/3189906275 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54cc007240 con 0x7f54e4063ad0 2022-06-29T16:05:41.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.510+0000 7f54ea453700 1 -- 172.21.15.60:0/3189906275 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54e4142960 con 0x7f54e4063ad0 2022-06-29T16:05:41.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.510+0000 7f54e17fa700 1 -- 172.21.15.60:0/3189906275 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54cc006700 con 0x7f54e4063ad0 2022-06-29T16:05:41.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.510+0000 7f54ea453700 1 -- 172.21.15.60:0/3189906275 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54e4155f70 con 0x7f54e4063ad0 2022-06-29T16:05:41.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.514+0000 7f54ea453700 1 -- 172.21.15.60:0/3189906275 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54e40ad0e0 con 0x7f54e4063ad0 2022-06-29T16:05:41.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.514+0000 7f54e17fa700 1 -- 172.21.15.60:0/3189906275 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f54cc006860 con 0x7f54e4063ad0 2022-06-29T16:05:41.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.514+0000 7f54e17fa700 1 --2- 172.21.15.60:0/3189906275 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f54bc053a70 0x7f54bc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:41.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.514+0000 7f54e17fa700 1 -- 172.21.15.60:0/3189906275 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f54cc06b8d0 con 0x7f54e4063ad0 2022-06-29T16:05:41.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.514+0000 7f54e37fe700 1 --2- 172.21.15.60:0/3189906275 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f54bc053a70 0x7f54bc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:41.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.514+0000 7f54e37fe700 1 --2- 172.21.15.60:0/3189906275 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f54bc053a70 0x7f54bc055f30 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f54d400b000 tx=0x7f54d4008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:41.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.522+0000 7f54e17fa700 1 -- 172.21.15.60:0/3189906275 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f54cc0427f0 con 0x7f54e4063ad0 2022-06-29T16:05:41.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.686+0000 7f54ea453700 1 -- 172.21.15.60:0/3189906275 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54e40aece0 con 0x7f54e4063ad0 2022-06-29T16:05:41.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.686+0000 7f54e17fa700 1 -- 172.21.15.60:0/3189906275 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f54cc045e10 con 0x7f54e4063ad0 2022-06-29T16:05:41.695 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:41.695 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":41,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6182,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1306898602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1306898602},{"type":"v1","addr":"172.21.15.60:6833","nonce":1306898602}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":39}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:05:40.369985+0000","modified":"2022-06-29T16:05:41.373815+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6185":{"gid":6185,"name":"b","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/2057420852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2057420852},{"type":"v1","addr":"172.21.15.92:6835","nonce":2057420852}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-06-29T16:05:41.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.690+0000 7f54ea453700 1 -- 172.21.15.60:0/3189906275 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f54bc053a70 msgr2=0x7f54bc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:41.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.690+0000 7f54ea453700 1 --2- 172.21.15.60:0/3189906275 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f54bc053a70 0x7f54bc055f30 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f54d400b000 tx=0x7f54d4008040).stop 2022-06-29T16:05:41.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.690+0000 7f54ea453700 1 -- 172.21.15.60:0/3189906275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54e4063ad0 msgr2=0x7f54e414bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:41.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.690+0000 7f54ea453700 1 --2- 172.21.15.60:0/3189906275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54e4063ad0 0x7f54e414bdb0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 rx=0x7f54e40ad0a0 tx=0x7f54cc007be0).stop 2022-06-29T16:05:41.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.690+0000 7f54ea453700 1 -- 172.21.15.60:0/3189906275 shutdown_connections 2022-06-29T16:05:41.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.690+0000 7f54ea453700 1 --2- 172.21.15.60:0/3189906275 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f54bc053a70 0x7f54bc055f30 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:41.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.690+0000 7f54ea453700 1 --2- 172.21.15.60:0/3189906275 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54e414af10 0x7f54e4142470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:41.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.690+0000 7f54ea453700 1 --2- 172.21.15.60:0/3189906275 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54e40b5950 0x7f54e4141f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:41.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.690+0000 7f54ea453700 1 --2- 172.21.15.60:0/3189906275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54e4063ad0 0x7f54e414bdb0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:41.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.690+0000 7f54ea453700 1 -- 172.21.15.60:0/3189906275 >> 172.21.15.60:0/3189906275 conn(0x7f54e40632c0 msgr2=0x7f54e4144f10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:41.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.690+0000 7f54ea453700 1 -- 172.21.15.60:0/3189906275 shutdown_connections 2022-06-29T16:05:41.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.690+0000 7f54ea453700 1 -- 172.21.15.60:0/3189906275 wait complete. 2022-06-29T16:05:41.702 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 41 2022-06-29T16:05:41.707 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:05:41.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.802+0000 7f84bb59e700 1 Processor -- start 2022-06-29T16:05:41.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.802+0000 7f84bb59e700 1 -- start start 2022-06-29T16:05:41.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84bb59e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f84bc05fe60 0x7f84bc14a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:41.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84bb59e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84bc060240 con 0x7f84bc14af90 2022-06-29T16:05:41.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84bb59e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84bc0abbd0 con 0x7f84bc0afb20 2022-06-29T16:05:41.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84bb59e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84bc097910 con 0x7f84bc05fe60 2022-06-29T16:05:41.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84bad9d700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f84bc14af90 0x7f84bc1470b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58836/0 (socket says 172.21.15.60:58836) 2022-06-29T16:05:41.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84bad9d700 1 -- 172.21.15.60:0/212351934 learned_addr learned my addr 172.21.15.60:0/212351934 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:41.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84ba59c700 1 --2- 172.21.15.60:0/212351934 >> v2:172.21.15.92:3300/0 conn(0x7f84bc05fe60 0x7f84bc14a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:41.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84b959a700 1 -- 172.21.15.60:0/212351934 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2757780058 0 0) 0x7f84bc060240 con 0x7f84bc14af90 2022-06-29T16:05:41.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84b959a700 1 -- 172.21.15.60:0/212351934 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f849c003260 con 0x7f84bc14af90 2022-06-29T16:05:41.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84b959a700 1 -- 172.21.15.60:0/212351934 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2900825575 0 0) 0x7f849c003260 con 0x7f84bc14af90 2022-06-29T16:05:41.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84b959a700 1 -- 172.21.15.60:0/212351934 >> v2:172.21.15.92:3300/0 conn(0x7f84bc05fe60 msgr2=0x7f84bc14a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:41.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84b959a700 1 --2- 172.21.15.60:0/212351934 >> v2:172.21.15.92:3300/0 conn(0x7f84bc05fe60 0x7f84bc14a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:41.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84b959a700 1 -- 172.21.15.60:0/212351934 >> v1:172.21.15.92:6789/0 conn(0x7f84bc0afb20 legacy=0x7f84bc14a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:41.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84b959a700 1 -- 172.21.15.60:0/212351934 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84bc097bb0 con 0x7f84bc14af90 2022-06-29T16:05:41.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84b959a700 1 -- 172.21.15.60:0/212351934 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f84b40019e0 con 0x7f84bc14af90 2022-06-29T16:05:41.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84b959a700 1 -- 172.21.15.60:0/212351934 >> v1:172.21.15.60:6789/0 conn(0x7f84bc14af90 legacy=0x7f84bc1470b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:41.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84b959a700 1 --2- 172.21.15.60:0/212351934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f849c003750 0x7f849c003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:41.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84b959a700 1 --2- 172.21.15.60:0/212351934 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f849c004500 0x7f849c004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:41.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84b959a700 1 --2- 172.21.15.60:0/212351934 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f849c004e70 0x7f849c00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:41.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84b959a700 1 -- 172.21.15.60:0/212351934 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84bc097910 con 0x7f849c003750 2022-06-29T16:05:41.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84b959a700 1 -- 172.21.15.60:0/212351934 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f849c00bdd0 con 0x7f849c004500 2022-06-29T16:05:41.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84b959a700 1 -- 172.21.15.60:0/212351934 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f849c00bf60 con 0x7f849c004e70 2022-06-29T16:05:41.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84ba59c700 1 --2- 172.21.15.60:0/212351934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f849c003750 0x7f849c003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:41.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84bad9d700 1 --2- 172.21.15.60:0/212351934 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f849c004e70 0x7f849c00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:41.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84b9d9b700 1 --2- 172.21.15.60:0/212351934 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f849c004500 0x7f849c004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:41.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84ba59c700 1 -- 172.21.15.60:0/212351934 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f849c004e70 msgr2=0x7f849c00b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:41.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84ba59c700 1 --2- 172.21.15.60:0/212351934 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f849c004e70 0x7f849c00b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:41.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84ba59c700 1 -- 172.21.15.60:0/212351934 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f849c004500 msgr2=0x7f849c004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:41.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84ba59c700 1 --2- 172.21.15.60:0/212351934 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f849c004500 0x7f849c004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:41.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84bad9d700 1 -- 172.21.15.60:0/212351934 reap_dead start 2022-06-29T16:05:41.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84ba59c700 1 -- 172.21.15.60:0/212351934 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f849c003260 con 0x7f849c003750 2022-06-29T16:05:41.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84b9d9b700 1 --2- 172.21.15.60:0/212351934 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f849c004500 0x7f849c004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:05:41.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84ba59c700 1 --2- 172.21.15.60:0/212351934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f849c003750 0x7f849c003b70 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 rx=0x7f84ac00c000 tx=0x7f84ac006df0).ready entity=mon.0 client_cookie=38bf6f225d37d3e server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:41.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84b959a700 1 -- 172.21.15.60:0/212351934 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84ac00e830 con 0x7f849c003750 2022-06-29T16:05:41.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84b959a700 1 -- 172.21.15.60:0/212351934 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84ac00e990 con 0x7f849c003750 2022-06-29T16:05:41.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84b959a700 1 -- 172.21.15.60:0/212351934 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84ac00ecf0 con 0x7f849c003750 2022-06-29T16:05:41.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84bb59e700 1 -- 172.21.15.60:0/212351934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f849c003750 msgr2=0x7f849c003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:41.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84bb59e700 1 --2- 172.21.15.60:0/212351934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f849c003750 0x7f849c003b70 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 rx=0x7f84ac00c000 tx=0x7f84ac006df0).stop 2022-06-29T16:05:41.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84bb59e700 1 -- 172.21.15.60:0/212351934 shutdown_connections 2022-06-29T16:05:41.829 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84bb59e700 1 --2- 172.21.15.60:0/212351934 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f849c003750 0x7f849c003b70 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:41.829 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84bb59e700 1 -- 172.21.15.60:0/212351934 >> 172.21.15.60:0/212351934 conn(0x7f84bc0b5180 msgr2=0x7f84bc0b5560 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:41.829 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84bb59e700 1 -- 172.21.15.60:0/212351934 shutdown_connections 2022-06-29T16:05:41.829 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.806+0000 7f84bb59e700 1 -- 172.21.15.60:0/212351934 wait complete. 2022-06-29T16:05:41.829 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.810+0000 7f84bb59e700 1 Processor -- start 2022-06-29T16:05:41.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.810+0000 7f84bb59e700 1 -- start start 2022-06-29T16:05:41.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.810+0000 7f84bb59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f849c003750 0x7f84bc065420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:41.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.810+0000 7f84bb59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84bc064c40 0x7f84bc0ad180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:41.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.810+0000 7f84bb59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84bc0ad670 0x7f84bc066c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:41.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.810+0000 7f84bb59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84bc0650c0 con 0x7f84bc0ad670 2022-06-29T16:05:41.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.810+0000 7f84bb59e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84bc099320 con 0x7f84bc064c40 2022-06-29T16:05:41.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.810+0000 7f84bb59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84bc09ae70 con 0x7f849c003750 2022-06-29T16:05:41.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.810+0000 7f84bad9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84bc0ad670 0x7f84bc066c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:41.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.810+0000 7f84bad9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84bc0ad670 0x7f84bc066c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50762/0 (socket says 172.21.15.60:50762) 2022-06-29T16:05:41.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.810+0000 7f84bad9d700 1 -- 172.21.15.60:0/1521192660 learned_addr learned my addr 172.21.15.60:0/1521192660 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:41.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.810+0000 7f84b9d9b700 1 --2- 172.21.15.60:0/1521192660 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84bc064c40 0x7f84bc0ad180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:41.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.810+0000 7f84ba59c700 1 --2- 172.21.15.60:0/1521192660 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f849c003750 0x7f84bc065420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:41.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.810+0000 7f84bad9d700 1 -- 172.21.15.60:0/1521192660 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f849c003750 msgr2=0x7f84bc065420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:41.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.810+0000 7f84bad9d700 1 --2- 172.21.15.60:0/1521192660 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f849c003750 0x7f84bc065420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:41.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.810+0000 7f84bad9d700 1 -- 172.21.15.60:0/1521192660 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84bc064c40 msgr2=0x7f84bc0ad180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:41.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.810+0000 7f84bad9d700 1 --2- 172.21.15.60:0/1521192660 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84bc064c40 0x7f84bc0ad180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:41.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.810+0000 7f84bad9d700 1 -- 172.21.15.60:0/1521192660 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84bc09b110 con 0x7f84bc0ad670 2022-06-29T16:05:41.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.810+0000 7f84bad9d700 1 --2- 172.21.15.60:0/1521192660 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84bc0ad670 0x7f84bc066c50 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 rx=0x7f84b4001490 tx=0x7f84b4009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:41.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.810+0000 7f84b37fe700 1 -- 172.21.15.60:0/1521192660 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84b4007170 con 0x7f84bc0ad670 2022-06-29T16:05:41.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.810+0000 7f84bb59e700 1 -- 172.21.15.60:0/1521192660 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84bc0aef10 con 0x7f84bc0ad670 2022-06-29T16:05:41.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.810+0000 7f84b37fe700 1 -- 172.21.15.60:0/1521192660 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b40072d0 con 0x7f84bc0ad670 2022-06-29T16:05:41.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.810+0000 7f84bb59e700 1 -- 172.21.15.60:0/1521192660 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84bc098960 con 0x7f84bc0ad670 2022-06-29T16:05:41.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.810+0000 7f84b37fe700 1 -- 172.21.15.60:0/1521192660 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84b4007630 con 0x7f84bc0ad670 2022-06-29T16:05:41.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.814+0000 7f84bb59e700 1 -- 172.21.15.60:0/1521192660 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84bc065200 con 0x7f84bc0ad670 2022-06-29T16:05:41.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.814+0000 7f84b37fe700 1 -- 172.21.15.60:0/1521192660 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f84b40048a0 con 0x7f84bc0ad670 2022-06-29T16:05:41.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.814+0000 7f84b37fe700 1 --2- 172.21.15.60:0/1521192660 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f84a0053a70 0x7f84a0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:41.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.814+0000 7f84b37fe700 1 -- 172.21.15.60:0/1521192660 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f84b406b760 con 0x7f84bc0ad670 2022-06-29T16:05:41.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.814+0000 7f84ba59c700 1 --2- 172.21.15.60:0/1521192660 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f84a0053a70 0x7f84a0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:41.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.814+0000 7f84ba59c700 1 --2- 172.21.15.60:0/1521192660 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f84a0053a70 0x7f84a0055f30 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f84ac01d000 tx=0x7f84ac019000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:41.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.822+0000 7f84b37fe700 1 -- 172.21.15.60:0/1521192660 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f84b40425e0 con 0x7f84bc0ad670 2022-06-29T16:05:41.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.946+0000 7f84bb59e700 1 -- 172.21.15.60:0/1521192660 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84bc09a040 con 0x7f84bc0ad670 2022-06-29T16:05:41.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.950+0000 7f84b37fe700 1 -- 172.21.15.60:0/1521192660 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+16041 (secure 0 0 0) 0x7f84b4045c00 con 0x7f84bc0ad670 2022-06-29T16:05:41.957 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:41.958 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":91,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:05:41.373483+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:05:38.407939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2022-06-29T16:05:39.380608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:05:41.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.950+0000 7f84bb59e700 1 -- 172.21.15.60:0/1521192660 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f84a0053a70 msgr2=0x7f84a0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:41.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.950+0000 7f84bb59e700 1 --2- 172.21.15.60:0/1521192660 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f84a0053a70 0x7f84a0055f30 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f84ac01d000 tx=0x7f84ac019000).stop 2022-06-29T16:05:41.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.950+0000 7f84bb59e700 1 -- 172.21.15.60:0/1521192660 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84bc0ad670 msgr2=0x7f84bc066c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:41.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.950+0000 7f84bb59e700 1 --2- 172.21.15.60:0/1521192660 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84bc0ad670 0x7f84bc066c50 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 rx=0x7f84b4001490 tx=0x7f84b4009040).stop 2022-06-29T16:05:41.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.950+0000 7f84bb59e700 1 -- 172.21.15.60:0/1521192660 shutdown_connections 2022-06-29T16:05:41.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.950+0000 7f84bb59e700 1 --2- 172.21.15.60:0/1521192660 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f84a0053a70 0x7f84a0055f30 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:41.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.950+0000 7f84bb59e700 1 --2- 172.21.15.60:0/1521192660 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f84bc0ad670 0x7f84bc066c50 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:41.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.950+0000 7f84bb59e700 1 --2- 172.21.15.60:0/1521192660 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84bc064c40 0x7f84bc0ad180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:41.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.954+0000 7f84bb59e700 1 --2- 172.21.15.60:0/1521192660 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f849c003750 0x7f84bc065420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:41.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.954+0000 7f84bb59e700 1 -- 172.21.15.60:0/1521192660 >> 172.21.15.60:0/1521192660 conn(0x7f84bc0b5180 msgr2=0x7f84bc145c60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:41.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.954+0000 7f84bb59e700 1 -- 172.21.15.60:0/1521192660 shutdown_connections 2022-06-29T16:05:41.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:41.954+0000 7f84bb59e700 1 -- 172.21.15.60:0/1521192660 wait complete. 2022-06-29T16:05:41.975 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T16:05:42.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.078+0000 7fd40112c700 1 Processor -- start 2022-06-29T16:05:42.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.078+0000 7fd40112c700 1 -- start start 2022-06-29T16:05:42.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.078+0000 7fd40112c700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd3fc0b2040 0x7fd3fc065b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:42.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.078+0000 7fd40112c700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3fc0b4110 con 0x7fd3fc141d60 2022-06-29T16:05:42.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.078+0000 7fd40112c700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3fc0b2420 con 0x7fd3fc05bf30 2022-06-29T16:05:42.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.078+0000 7fd40112c700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3fc05d4f0 con 0x7fd3fc0b2040 2022-06-29T16:05:42.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.078+0000 7fd40092b700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd3fc141d60 0x7fd3fc1421b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58856/0 (socket says 172.21.15.60:58856) 2022-06-29T16:05:42.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.078+0000 7fd40092b700 1 -- 172.21.15.60:0/32241239 learned_addr learned my addr 172.21.15.60:0/32241239 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:42.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.078+0000 7fd3fbfff700 1 --2- 172.21.15.60:0/32241239 >> v2:172.21.15.92:3300/0 conn(0x7fd3fc0b2040 0x7fd3fc065b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:42.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.078+0000 7fd3faffd700 1 -- 172.21.15.60:0/32241239 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1715900748 0 0) 0x7fd3fc0b4110 con 0x7fd3fc141d60 2022-06-29T16:05:42.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.078+0000 7fd3faffd700 1 -- 172.21.15.60:0/32241239 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3dc003260 con 0x7fd3fc141d60 2022-06-29T16:05:42.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.078+0000 7fd3faffd700 1 -- 172.21.15.60:0/32241239 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3410961002 0 0) 0x7fd3dc003260 con 0x7fd3fc141d60 2022-06-29T16:05:42.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.078+0000 7fd3faffd700 1 -- 172.21.15.60:0/32241239 >> v2:172.21.15.92:3300/0 conn(0x7fd3fc0b2040 msgr2=0x7fd3fc065b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:42.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.078+0000 7fd3faffd700 1 --2- 172.21.15.60:0/32241239 >> v2:172.21.15.92:3300/0 conn(0x7fd3fc0b2040 0x7fd3fc065b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:42.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.078+0000 7fd3faffd700 1 -- 172.21.15.60:0/32241239 >> v1:172.21.15.92:6789/0 conn(0x7fd3fc05bf30 legacy=0x7fd3fc066020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:42.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.078+0000 7fd3faffd700 1 -- 172.21.15.60:0/32241239 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3fc0b3930 con 0x7fd3fc141d60 2022-06-29T16:05:42.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.078+0000 7fd3faffd700 1 -- 172.21.15.60:0/32241239 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7fd3f40019e0 con 0x7fd3fc141d60 2022-06-29T16:05:42.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.078+0000 7fd3faffd700 1 -- 172.21.15.60:0/32241239 >> v1:172.21.15.60:6789/0 conn(0x7fd3fc141d60 legacy=0x7fd3fc1421b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:42.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.078+0000 7fd3faffd700 1 --2- 172.21.15.60:0/32241239 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3dc003750 0x7fd3dc003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:42.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.078+0000 7fd3faffd700 1 --2- 172.21.15.60:0/32241239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3dc004500 0x7fd3dc004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:42.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.078+0000 7fd3faffd700 1 --2- 172.21.15.60:0/32241239 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3dc004e70 0x7fd3dc00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:42.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.078+0000 7fd3faffd700 1 -- 172.21.15.60:0/32241239 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc05d4f0 con 0x7fd3dc004500 2022-06-29T16:05:42.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.078+0000 7fd3faffd700 1 -- 172.21.15.60:0/32241239 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc00bd20 con 0x7fd3dc003750 2022-06-29T16:05:42.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.078+0000 7fd3faffd700 1 -- 172.21.15.60:0/32241239 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc00bee0 con 0x7fd3dc004e70 2022-06-29T16:05:42.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.082+0000 7fd3fb7fe700 1 --2- 172.21.15.60:0/32241239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3dc004500 0x7fd3dc004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:42.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.082+0000 7fd3fbfff700 1 --2- 172.21.15.60:0/32241239 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3dc003750 0x7fd3dc003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:42.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.082+0000 7fd40092b700 1 --2- 172.21.15.60:0/32241239 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3dc004e70 0x7fd3dc00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:42.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.082+0000 7fd3fb7fe700 1 -- 172.21.15.60:0/32241239 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3dc004e70 msgr2=0x7fd3dc00b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:42.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.082+0000 7fd3fb7fe700 1 --2- 172.21.15.60:0/32241239 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3dc004e70 0x7fd3dc00b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:42.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.082+0000 7fd3fb7fe700 1 -- 172.21.15.60:0/32241239 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3dc003750 msgr2=0x7fd3dc003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:42.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.082+0000 7fd3fb7fe700 1 --2- 172.21.15.60:0/32241239 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3dc003750 0x7fd3dc003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:42.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.082+0000 7fd3fb7fe700 1 -- 172.21.15.60:0/32241239 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3dc003260 con 0x7fd3dc004500 2022-06-29T16:05:42.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.082+0000 7fd40092b700 1 -- 172.21.15.60:0/32241239 reap_dead start 2022-06-29T16:05:42.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.082+0000 7fd3fb7fe700 1 --2- 172.21.15.60:0/32241239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3dc004500 0x7fd3dc004980 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 rx=0x7fd3fc065480 tx=0x7fd3ec007a00).ready entity=mon.0 client_cookie=ed292ab09ec26dec server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:42.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.082+0000 7fd3faffd700 1 -- 172.21.15.60:0/32241239 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd3ec007d10 con 0x7fd3dc004500 2022-06-29T16:05:42.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.082+0000 7fd3faffd700 1 -- 172.21.15.60:0/32241239 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3ec007030 con 0x7fd3dc004500 2022-06-29T16:05:42.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.082+0000 7fd3faffd700 1 -- 172.21.15.60:0/32241239 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd3ec007390 con 0x7fd3dc004500 2022-06-29T16:05:42.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.082+0000 7fd40112c700 1 -- 172.21.15.60:0/32241239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3dc004500 msgr2=0x7fd3dc004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:42.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.082+0000 7fd40112c700 1 --2- 172.21.15.60:0/32241239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3dc004500 0x7fd3dc004980 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 rx=0x7fd3fc065480 tx=0x7fd3ec007a00).stop 2022-06-29T16:05:42.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.082+0000 7fd40112c700 1 -- 172.21.15.60:0/32241239 shutdown_connections 2022-06-29T16:05:42.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.082+0000 7fd40112c700 1 --2- 172.21.15.60:0/32241239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3dc004500 0x7fd3dc004980 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:42.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.082+0000 7fd40112c700 1 -- 172.21.15.60:0/32241239 >> 172.21.15.60:0/32241239 conn(0x7fd3fc064810 msgr2=0x7fd3fc064bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:42.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.082+0000 7fd40112c700 1 -- 172.21.15.60:0/32241239 shutdown_connections 2022-06-29T16:05:42.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.082+0000 7fd40112c700 1 -- 172.21.15.60:0/32241239 wait complete. 2022-06-29T16:05:42.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.082+0000 7fd40112c700 1 Processor -- start 2022-06-29T16:05:42.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.082+0000 7fd40112c700 1 -- start start 2022-06-29T16:05:42.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.086+0000 7fd40112c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3dc004500 0x7fd3fc0ae2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:42.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.086+0000 7fd40112c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3fc151a70 0x7fd3fc0afe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:42.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.086+0000 7fd40112c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3fc0b0370 0x7fd3fc155070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:42.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.086+0000 7fd40112c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc148b20 con 0x7fd3fc0b0370 2022-06-29T16:05:42.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.086+0000 7fd40112c700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc13f130 con 0x7fd3fc151a70 2022-06-29T16:05:42.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.086+0000 7fd40112c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc13f9c0 con 0x7fd3dc004500 2022-06-29T16:05:42.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.086+0000 7fd40092b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3fc0b0370 0x7fd3fc155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:42.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.086+0000 7fd40092b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3fc0b0370 0x7fd3fc155070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50780/0 (socket says 172.21.15.60:50780) 2022-06-29T16:05:42.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.086+0000 7fd40092b700 1 -- 172.21.15.60:0/438274703 learned_addr learned my addr 172.21.15.60:0/438274703 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:42.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.086+0000 7fd3fbfff700 1 --2- 172.21.15.60:0/438274703 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3dc004500 0x7fd3fc0ae2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:42.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.086+0000 7fd3fb7fe700 1 --2- 172.21.15.60:0/438274703 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3fc151a70 0x7fd3fc0afe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:42.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.086+0000 7fd40092b700 1 -- 172.21.15.60:0/438274703 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3dc004500 msgr2=0x7fd3fc0ae2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:42.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.086+0000 7fd40092b700 1 --2- 172.21.15.60:0/438274703 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3dc004500 0x7fd3fc0ae2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:42.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.086+0000 7fd40092b700 1 -- 172.21.15.60:0/438274703 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3fc151a70 msgr2=0x7fd3fc0afe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:42.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.086+0000 7fd40092b700 1 --2- 172.21.15.60:0/438274703 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3fc151a70 0x7fd3fc0afe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:42.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.086+0000 7fd40092b700 1 -- 172.21.15.60:0/438274703 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3fc13fc60 con 0x7fd3fc0b0370 2022-06-29T16:05:42.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.086+0000 7fd40092b700 1 --2- 172.21.15.60:0/438274703 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3fc0b0370 0x7fd3fc155070 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0x7fd3fc0af5e0 tx=0x7fd3f4001fe0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:42.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.086+0000 7fd3f97fa700 1 -- 172.21.15.60:0/438274703 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd3f4010710 con 0x7fd3fc0b0370 2022-06-29T16:05:42.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.086+0000 7fd3f97fa700 1 -- 172.21.15.60:0/438274703 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3f4010870 con 0x7fd3fc0b0370 2022-06-29T16:05:42.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.086+0000 7fd3f97fa700 1 -- 172.21.15.60:0/438274703 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd3f4010bd0 con 0x7fd3fc0b0370 2022-06-29T16:05:42.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.086+0000 7fd40112c700 1 -- 172.21.15.60:0/438274703 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3fc13edb0 con 0x7fd3fc0b0370 2022-06-29T16:05:42.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.086+0000 7fd40112c700 1 -- 172.21.15.60:0/438274703 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3fc1403a0 con 0x7fd3fc0b0370 2022-06-29T16:05:42.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.086+0000 7fd40112c700 1 -- 172.21.15.60:0/438274703 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3fc0573d0 con 0x7fd3fc0b0370 2022-06-29T16:05:42.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.086+0000 7fd3f97fa700 1 -- 172.21.15.60:0/438274703 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd3f4010d30 con 0x7fd3fc0b0370 2022-06-29T16:05:42.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.090+0000 7fd3f97fa700 1 --2- 172.21.15.60:0/438274703 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd3e0053b00 0x7fd3e0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:42.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.090+0000 7fd3f97fa700 1 -- 172.21.15.60:0/438274703 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd3f406b700 con 0x7fd3fc0b0370 2022-06-29T16:05:42.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.090+0000 7fd3fbfff700 1 --2- 172.21.15.60:0/438274703 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd3e0053b00 0x7fd3e0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:42.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.090+0000 7fd3fbfff700 1 --2- 172.21.15.60:0/438274703 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd3e0053b00 0x7fd3e0055fc0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fd3e400b000 tx=0x7fd3e4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:42.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.094+0000 7fd3f97fa700 1 -- 172.21.15.60:0/438274703 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd3f4042620 con 0x7fd3fc0b0370 2022-06-29T16:05:42.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.258+0000 7fd40112c700 1 -- 172.21.15.60:0/438274703 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fd3fc068320 con 0x7fd3fc0b0370 2022-06-29T16:05:42.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.266+0000 7fd3f97fa700 1 -- 172.21.15.60:0/438274703 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v31) v1 ==== 184+0+0 (secure 0 0 0) 0x7fd3f4045c40 con 0x7fd3fc0b0370 2022-06-29T16:05:42.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.270+0000 7fd40112c700 1 -- 172.21.15.60:0/438274703 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd3e0053b00 msgr2=0x7fd3e0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:42.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.270+0000 7fd40112c700 1 --2- 172.21.15.60:0/438274703 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd3e0053b00 0x7fd3e0055fc0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fd3e400b000 tx=0x7fd3e4006df0).stop 2022-06-29T16:05:42.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.270+0000 7fd40112c700 1 -- 172.21.15.60:0/438274703 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3fc0b0370 msgr2=0x7fd3fc155070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:42.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.270+0000 7fd40112c700 1 --2- 172.21.15.60:0/438274703 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3fc0b0370 0x7fd3fc155070 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0x7fd3fc0af5e0 tx=0x7fd3f4001fe0).stop 2022-06-29T16:05:42.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.270+0000 7fd40112c700 1 -- 172.21.15.60:0/438274703 shutdown_connections 2022-06-29T16:05:42.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.270+0000 7fd40112c700 1 --2- 172.21.15.60:0/438274703 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd3e0053b00 0x7fd3e0055fc0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:42.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.270+0000 7fd40112c700 1 --2- 172.21.15.60:0/438274703 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3fc0b0370 0x7fd3fc155070 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:42.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.270+0000 7fd40112c700 1 --2- 172.21.15.60:0/438274703 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3fc151a70 0x7fd3fc0afe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:42.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.270+0000 7fd40112c700 1 --2- 172.21.15.60:0/438274703 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3dc004500 0x7fd3fc0ae2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:42.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.270+0000 7fd40112c700 1 -- 172.21.15.60:0/438274703 >> 172.21.15.60:0/438274703 conn(0x7fd3fc064810 msgr2=0x7fd3fc142ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:42.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.270+0000 7fd40112c700 1 -- 172.21.15.60:0/438274703 shutdown_connections 2022-06-29T16:05:42.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.270+0000 7fd40112c700 1 -- 172.21.15.60:0/438274703 wait complete. 2022-06-29T16:05:42.284 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.0 2022-06-29T16:05:42.291 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T16:05:42.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.398+0000 7f12631e4700 1 Processor -- start 2022-06-29T16:05:42.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.398+0000 7f12631e4700 1 -- start start 2022-06-29T16:05:42.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.402+0000 7f12631e4700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f125c0b9d70 0x7f125c0b3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:42.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.402+0000 7f12631e4700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f125c05d1f0 0x7f125c0b1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:42.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.402+0000 7f12631e4700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f125c0b2ce0 con 0x7f125c0b3cc0 2022-06-29T16:05:42.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.402+0000 7f12631e4700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f125c0b4d10 con 0x7f125c05d1f0 2022-06-29T16:05:42.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.402+0000 7f12631e4700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f125c0ba150 con 0x7f125c0b9d70 2022-06-29T16:05:42.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.402+0000 7f125bfff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f125c05d1f0 0x7f125c0b1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:42.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.402+0000 7f1261781700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f125c0b3cc0 0x7f125c1420b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58876/0 (socket says 172.21.15.60:58876) 2022-06-29T16:05:42.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.402+0000 7f1261781700 1 -- 172.21.15.60:0/269316887 learned_addr learned my addr 172.21.15.60:0/269316887 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:42.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.402+0000 7f1260f80700 1 --2- 172.21.15.60:0/269316887 >> v2:172.21.15.92:3300/0 conn(0x7f125c0b9d70 0x7f125c0b3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:42.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.402+0000 7f125b7fe700 1 -- 172.21.15.60:0/269316887 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 761955911 0 0) 0x7f125c0b2ce0 con 0x7f125c0b3cc0 2022-06-29T16:05:42.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.402+0000 7f125b7fe700 1 -- 172.21.15.60:0/269316887 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f123c003260 con 0x7f125c0b3cc0 2022-06-29T16:05:42.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.402+0000 7f125bfff700 1 -- 172.21.15.60:0/269316887 >> v2:172.21.15.92:3300/0 conn(0x7f125c0b9d70 msgr2=0x7f125c0b3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:42.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.402+0000 7f125bfff700 1 --2- 172.21.15.60:0/269316887 >> v2:172.21.15.92:3300/0 conn(0x7f125c0b9d70 0x7f125c0b3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:42.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.402+0000 7f125bfff700 1 -- 172.21.15.60:0/269316887 >> v1:172.21.15.60:6789/0 conn(0x7f125c0b3cc0 legacy=0x7f125c1420b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:42.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.402+0000 7f125bfff700 1 -- 172.21.15.60:0/269316887 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f125c0b4a70 con 0x7f125c05d1f0 2022-06-29T16:05:42.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.402+0000 7f1260f80700 1 --2- 172.21.15.60:0/269316887 >> v2:172.21.15.92:3300/0 conn(0x7f125c0b9d70 0x7f125c0b3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:42.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.402+0000 7f125bfff700 1 --2- 172.21.15.60:0/269316887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f125c05d1f0 0x7f125c0b1b50 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 rx=0x7f1244002f70 tx=0x7f1244006df0).ready entity=mon.0 client_cookie=4f8ad4d2890a9489 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:42.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.402+0000 7f125b7fe700 1 -- 172.21.15.60:0/269316887 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f124400d570 con 0x7f125c05d1f0 2022-06-29T16:05:42.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.402+0000 7f125b7fe700 1 -- 172.21.15.60:0/269316887 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1244000e90 con 0x7f125c05d1f0 2022-06-29T16:05:42.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.402+0000 7f125b7fe700 1 -- 172.21.15.60:0/269316887 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f124400d8c0 con 0x7f125c05d1f0 2022-06-29T16:05:42.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.402+0000 7f12631e4700 1 -- 172.21.15.60:0/269316887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f125c05d1f0 msgr2=0x7f125c0b1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:42.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.402+0000 7f12631e4700 1 --2- 172.21.15.60:0/269316887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f125c05d1f0 0x7f125c0b1b50 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 rx=0x7f1244002f70 tx=0x7f1244006df0).stop 2022-06-29T16:05:42.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.402+0000 7f12631e4700 1 -- 172.21.15.60:0/269316887 shutdown_connections 2022-06-29T16:05:42.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.402+0000 7f12631e4700 1 --2- 172.21.15.60:0/269316887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f125c05d1f0 0x7f125c0b1b50 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:42.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.402+0000 7f12631e4700 1 --2- 172.21.15.60:0/269316887 >> v2:172.21.15.92:3300/0 conn(0x7f125c0b9d70 0x7f125c0b3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:42.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.402+0000 7f12631e4700 1 -- 172.21.15.60:0/269316887 >> 172.21.15.60:0/269316887 conn(0x7f125c062ce0 msgr2=0x7f125c0b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:42.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.402+0000 7f12631e4700 1 -- 172.21.15.60:0/269316887 shutdown_connections 2022-06-29T16:05:42.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.402+0000 7f12631e4700 1 -- 172.21.15.60:0/269316887 wait complete. 2022-06-29T16:05:42.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.406+0000 7f12631e4700 1 Processor -- start 2022-06-29T16:05:42.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.406+0000 7f12631e4700 1 -- start start 2022-06-29T16:05:42.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.406+0000 7f12631e4700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f125c05d1f0 0x7f125c0accd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:42.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.406+0000 7f12631e4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f125c0b3cc0 0x7f125c140b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:42.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.406+0000 7f12631e4700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f125c0b9d70 0x7f125c1473b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:42.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.406+0000 7f12631e4700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f125c0ae710 con 0x7f125c0b3cc0 2022-06-29T16:05:42.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.406+0000 7f12631e4700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f125c0af340 con 0x7f125c0b9d70 2022-06-29T16:05:42.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.406+0000 7f12631e4700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f125c0af5e0 con 0x7f125c05d1f0 2022-06-29T16:05:42.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.406+0000 7f125bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f125c0b3cc0 0x7f125c140b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:42.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.406+0000 7f125bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f125c0b3cc0 0x7f125c140b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50792/0 (socket says 172.21.15.60:50792) 2022-06-29T16:05:42.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.406+0000 7f125bfff700 1 -- 172.21.15.60:0/1423755270 learned_addr learned my addr 172.21.15.60:0/1423755270 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:42.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.406+0000 7f1261781700 1 --2- 172.21.15.60:0/1423755270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f125c0b9d70 0x7f125c1473b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:42.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.406+0000 7f1260f80700 1 --2- 172.21.15.60:0/1423755270 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f125c05d1f0 0x7f125c0accd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:42.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.406+0000 7f125bfff700 1 -- 172.21.15.60:0/1423755270 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f125c05d1f0 msgr2=0x7f125c0accd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:42.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.406+0000 7f125bfff700 1 --2- 172.21.15.60:0/1423755270 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f125c05d1f0 0x7f125c0accd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:42.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.406+0000 7f125bfff700 1 -- 172.21.15.60:0/1423755270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f125c0b9d70 msgr2=0x7f125c1473b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:42.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.406+0000 7f125bfff700 1 --2- 172.21.15.60:0/1423755270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f125c0b9d70 0x7f125c1473b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:42.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.406+0000 7f125bfff700 1 -- 172.21.15.60:0/1423755270 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f124400b180 con 0x7f125c0b3cc0 2022-06-29T16:05:42.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.406+0000 7f125bfff700 1 --2- 172.21.15.60:0/1423755270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f125c0b3cc0 0x7f125c140b20 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 rx=0x7f1244002f70 tx=0x7f1244014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:42.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.406+0000 7f1259ffb700 1 -- 172.21.15.60:0/1423755270 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1244006b20 con 0x7f125c0b3cc0 2022-06-29T16:05:42.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.406+0000 7f1259ffb700 1 -- 172.21.15.60:0/1423755270 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f124401e400 con 0x7f125c0b3cc0 2022-06-29T16:05:42.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.406+0000 7f12631e4700 1 -- 172.21.15.60:0/1423755270 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f125c0af880 con 0x7f125c0b3cc0 2022-06-29T16:05:42.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.406+0000 7f1259ffb700 1 -- 172.21.15.60:0/1423755270 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f124400dea0 con 0x7f125c0b3cc0 2022-06-29T16:05:42.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.406+0000 7f12631e4700 1 -- 172.21.15.60:0/1423755270 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f125c141610 con 0x7f125c0b3cc0 2022-06-29T16:05:42.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.410+0000 7f12631e4700 1 -- 172.21.15.60:0/1423755270 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f125c0573d0 con 0x7f125c0b3cc0 2022-06-29T16:05:42.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.410+0000 7f1259ffb700 1 -- 172.21.15.60:0/1423755270 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f124401c070 con 0x7f125c0b3cc0 2022-06-29T16:05:42.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.410+0000 7f1259ffb700 1 --2- 172.21.15.60:0/1423755270 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1240053b00 0x7f1240055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:42.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.410+0000 7f1259ffb700 1 -- 172.21.15.60:0/1423755270 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f124406ca20 con 0x7f125c0b3cc0 2022-06-29T16:05:42.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.410+0000 7f1260f80700 1 --2- 172.21.15.60:0/1423755270 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1240053b00 0x7f1240055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:42.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.410+0000 7f1260f80700 1 --2- 172.21.15.60:0/1423755270 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1240053b00 0x7f1240055fc0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f124c00c000 tx=0x7f124c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:42.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.418+0000 7f1259ffb700 1 -- 172.21.15.60:0/1423755270 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f12440438a0 con 0x7f125c0b3cc0 2022-06-29T16:05:42.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.582+0000 7f12631e4700 1 -- 172.21.15.60:0/1423755270 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f125c0afa10 con 0x7f125c0b3cc0 2022-06-29T16:05:42.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.590+0000 7f1259ffb700 1 -- 172.21.15.60:0/1423755270 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v32) v1 ==== 184+0+0 (secure 0 0 0) 0x7f1244023020 con 0x7f125c0b3cc0 2022-06-29T16:05:42.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.590+0000 7f12631e4700 1 -- 172.21.15.60:0/1423755270 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1240053b00 msgr2=0x7f1240055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:42.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.590+0000 7f12631e4700 1 --2- 172.21.15.60:0/1423755270 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1240053b00 0x7f1240055fc0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f124c00c000 tx=0x7f124c008040).stop 2022-06-29T16:05:42.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.590+0000 7f12631e4700 1 -- 172.21.15.60:0/1423755270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f125c0b3cc0 msgr2=0x7f125c140b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:42.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.590+0000 7f12631e4700 1 --2- 172.21.15.60:0/1423755270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f125c0b3cc0 0x7f125c140b20 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 rx=0x7f1244002f70 tx=0x7f1244014000).stop 2022-06-29T16:05:42.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.590+0000 7f12631e4700 1 -- 172.21.15.60:0/1423755270 shutdown_connections 2022-06-29T16:05:42.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.590+0000 7f12631e4700 1 --2- 172.21.15.60:0/1423755270 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1240053b00 0x7f1240055fc0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:42.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.590+0000 7f12631e4700 1 --2- 172.21.15.60:0/1423755270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f125c0b9d70 0x7f125c1473b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:42.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.590+0000 7f12631e4700 1 --2- 172.21.15.60:0/1423755270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f125c0b3cc0 0x7f125c140b20 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:42.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.590+0000 7f12631e4700 1 --2- 172.21.15.60:0/1423755270 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f125c05d1f0 0x7f125c0accd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:42.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.590+0000 7f12631e4700 1 -- 172.21.15.60:0/1423755270 >> 172.21.15.60:0/1423755270 conn(0x7f125c062ce0 msgr2=0x7f125c0b7c30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:42.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.590+0000 7f12631e4700 1 -- 172.21.15.60:0/1423755270 shutdown_connections 2022-06-29T16:05:42.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.590+0000 7f12631e4700 1 -- 172.21.15.60:0/1423755270 wait complete. 2022-06-29T16:05:42.604 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.1 2022-06-29T16:05:42.612 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T16:05:42.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.722+0000 7faa83092700 1 Processor -- start 2022-06-29T16:05:42.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.722+0000 7faa83092700 1 -- start start 2022-06-29T16:05:42.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.722+0000 7faa83092700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7faa7c05fb00 0x7faa7c0b2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:42.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.722+0000 7faa83092700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7faa7c0b1500 0x7faa7c0ae040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:42.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.722+0000 7faa83092700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa7c0b2f40 con 0x7faa7c0b0e60 2022-06-29T16:05:42.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.722+0000 7faa83092700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa7c05fee0 con 0x7faa7c05fb00 2022-06-29T16:05:42.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.722+0000 7faa83092700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa7c0b0150 con 0x7faa7c0b1500 2022-06-29T16:05:42.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.722+0000 7faa82090700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7faa7c05fb00 0x7faa7c0b2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:42.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.722+0000 7faa82090700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7faa7c05fb00 0x7faa7c0b2a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50800/0 (socket says 172.21.15.60:50800) 2022-06-29T16:05:42.734 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.722+0000 7faa82090700 1 -- 172.21.15.60:0/3084010453 learned_addr learned my addr 172.21.15.60:0/3084010453 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:42.734 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.722+0000 7faa8188f700 1 --2- 172.21.15.60:0/3084010453 >> v2:172.21.15.92:3300/0 conn(0x7faa7c0b1500 0x7faa7c0ae040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:42.734 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.722+0000 7faa82090700 1 -- 172.21.15.60:0/3084010453 >> v2:172.21.15.92:3300/0 conn(0x7faa7c0b1500 msgr2=0x7faa7c0ae040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:42.734 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.722+0000 7faa82090700 1 --2- 172.21.15.60:0/3084010453 >> v2:172.21.15.92:3300/0 conn(0x7faa7c0b1500 0x7faa7c0ae040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:42.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.722+0000 7faa82090700 1 -- 172.21.15.60:0/3084010453 >> v1:172.21.15.92:6789/0 conn(0x7faa7c0b0e60 legacy=0x7faa7c146320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:42.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.722+0000 7faa82090700 1 -- 172.21.15.60:0/3084010453 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa7c0acba0 con 0x7faa7c05fb00 2022-06-29T16:05:42.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.722+0000 7faa82090700 1 --2- 172.21.15.60:0/3084010453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa7c05fb00 0x7faa7c0b2a50 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 rx=0x7faa7c062630 tx=0x7faa78006eb0).ready entity=mon.0 client_cookie=ca3c374345b2a24c server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:42.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.726+0000 7faa8108e700 1 -- 172.21.15.60:0/3084010453 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa7800d5c0 con 0x7faa7c05fb00 2022-06-29T16:05:42.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.726+0000 7faa8108e700 1 -- 172.21.15.60:0/3084010453 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa78000e90 con 0x7faa7c05fb00 2022-06-29T16:05:42.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.726+0000 7faa8108e700 1 -- 172.21.15.60:0/3084010453 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa7800d910 con 0x7faa7c05fb00 2022-06-29T16:05:42.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.726+0000 7faa83092700 1 -- 172.21.15.60:0/3084010453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa7c05fb00 msgr2=0x7faa7c0b2a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:42.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.726+0000 7faa83092700 1 --2- 172.21.15.60:0/3084010453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa7c05fb00 0x7faa7c0b2a50 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 rx=0x7faa7c062630 tx=0x7faa78006eb0).stop 2022-06-29T16:05:42.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.726+0000 7faa83092700 1 -- 172.21.15.60:0/3084010453 shutdown_connections 2022-06-29T16:05:42.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.726+0000 7faa83092700 1 --2- 172.21.15.60:0/3084010453 >> v2:172.21.15.92:3300/0 conn(0x7faa7c0b1500 0x7faa7c0ae040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:42.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.726+0000 7faa83092700 1 --2- 172.21.15.60:0/3084010453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa7c05fb00 0x7faa7c0b2a50 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:42.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.726+0000 7faa83092700 1 -- 172.21.15.60:0/3084010453 >> 172.21.15.60:0/3084010453 conn(0x7faa7c063ab0 msgr2=0x7faa7c05f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:42.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.726+0000 7faa83092700 1 -- 172.21.15.60:0/3084010453 shutdown_connections 2022-06-29T16:05:42.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.726+0000 7faa83092700 1 -- 172.21.15.60:0/3084010453 wait complete. 2022-06-29T16:05:42.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.726+0000 7faa83092700 1 Processor -- start 2022-06-29T16:05:42.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.726+0000 7faa83092700 1 -- start start 2022-06-29T16:05:42.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.726+0000 7faa83092700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa7c05fb00 0x7faa7c146920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:42.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.726+0000 7faa83092700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faa7c0b0e60 0x7faa7c147fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:42.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.726+0000 7faa83092700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa7c0b1500 0x7faa7c144220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:42.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.730+0000 7faa83092700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7c148860 con 0x7faa7c05fb00 2022-06-29T16:05:42.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.730+0000 7faa83092700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7c1529e0 con 0x7faa7c0b1500 2022-06-29T16:05:42.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.730+0000 7faa83092700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa7c152c80 con 0x7faa7c0b0e60 2022-06-29T16:05:42.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.730+0000 7faa82090700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa7c05fb00 0x7faa7c146920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:42.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.730+0000 7faa82090700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa7c05fb00 0x7faa7c146920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50806/0 (socket says 172.21.15.60:50806) 2022-06-29T16:05:42.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.730+0000 7faa82090700 1 -- 172.21.15.60:0/3035341158 learned_addr learned my addr 172.21.15.60:0/3035341158 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:42.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.730+0000 7faa8188f700 1 --2- 172.21.15.60:0/3035341158 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faa7c0b0e60 0x7faa7c147fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:42.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.730+0000 7faa82891700 1 --2- 172.21.15.60:0/3035341158 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa7c0b1500 0x7faa7c144220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:42.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.730+0000 7faa82090700 1 -- 172.21.15.60:0/3035341158 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faa7c0b0e60 msgr2=0x7faa7c147fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:42.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.730+0000 7faa82090700 1 --2- 172.21.15.60:0/3035341158 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faa7c0b0e60 0x7faa7c147fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:42.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.730+0000 7faa82090700 1 -- 172.21.15.60:0/3035341158 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa7c0b1500 msgr2=0x7faa7c144220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:42.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.730+0000 7faa82090700 1 --2- 172.21.15.60:0/3035341158 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa7c0b1500 0x7faa7c144220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:42.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.730+0000 7faa82090700 1 -- 172.21.15.60:0/3035341158 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa7800b1e0 con 0x7faa7c05fb00 2022-06-29T16:05:42.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.730+0000 7faa8188f700 1 --2- 172.21.15.60:0/3035341158 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faa7c0b0e60 0x7faa7c147fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:05:42.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.730+0000 7faa82090700 1 --2- 172.21.15.60:0/3035341158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa7c05fb00 0x7faa7c146920 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 rx=0x7faa7c062630 tx=0x7faa78015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:42.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.730+0000 7faa6b7fe700 1 -- 172.21.15.60:0/3035341158 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa78011760 con 0x7faa7c05fb00 2022-06-29T16:05:42.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.730+0000 7faa6b7fe700 1 -- 172.21.15.60:0/3035341158 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa780118c0 con 0x7faa7c05fb00 2022-06-29T16:05:42.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.730+0000 7faa83092700 1 -- 172.21.15.60:0/3035341158 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa7c148490 con 0x7faa7c05fb00 2022-06-29T16:05:42.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.730+0000 7faa6b7fe700 1 -- 172.21.15.60:0/3035341158 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa78011c20 con 0x7faa7c05fb00 2022-06-29T16:05:42.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.730+0000 7faa83092700 1 -- 172.21.15.60:0/3035341158 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa7c146570 con 0x7faa7c05fb00 2022-06-29T16:05:42.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.730+0000 7faa83092700 1 -- 172.21.15.60:0/3035341158 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa7c0573d0 con 0x7faa7c05fb00 2022-06-29T16:05:42.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.734+0000 7faa6b7fe700 1 -- 172.21.15.60:0/3035341158 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7faa78025030 con 0x7faa7c05fb00 2022-06-29T16:05:42.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.734+0000 7faa6b7fe700 1 --2- 172.21.15.60:0/3035341158 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7faa5c0539e0 0x7faa5c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:42.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.734+0000 7faa6b7fe700 1 -- 172.21.15.60:0/3035341158 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7faa7806cf00 con 0x7faa7c05fb00 2022-06-29T16:05:42.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.734+0000 7faa8188f700 1 --2- 172.21.15.60:0/3035341158 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7faa5c0539e0 0x7faa5c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:42.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.734+0000 7faa8188f700 1 --2- 172.21.15.60:0/3035341158 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7faa5c0539e0 0x7faa5c055ea0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7faa6c00c000 tx=0x7faa6c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:42.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.738+0000 7faa6b7fe700 1 -- 172.21.15.60:0/3035341158 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7faa78043d10 con 0x7faa7c05fb00 2022-06-29T16:05:42.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.914+0000 7faa83092700 1 -- 172.21.15.60:0/3035341158 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7faa7c0b2f40 con 0x7faa7c05fb00 2022-06-29T16:05:42.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.922+0000 7faa6b7fe700 1 -- 172.21.15.60:0/3035341158 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v33) v1 ==== 184+0+0 (secure 0 0 0) 0x7faa7c0b2f40 con 0x7faa7c05fb00 2022-06-29T16:05:42.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.926+0000 7faa83092700 1 -- 172.21.15.60:0/3035341158 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7faa5c0539e0 msgr2=0x7faa5c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:42.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.926+0000 7faa83092700 1 --2- 172.21.15.60:0/3035341158 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7faa5c0539e0 0x7faa5c055ea0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7faa6c00c000 tx=0x7faa6c008040).stop 2022-06-29T16:05:42.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.926+0000 7faa83092700 1 -- 172.21.15.60:0/3035341158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa7c05fb00 msgr2=0x7faa7c146920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:42.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.926+0000 7faa83092700 1 --2- 172.21.15.60:0/3035341158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa7c05fb00 0x7faa7c146920 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 rx=0x7faa7c062630 tx=0x7faa78015000).stop 2022-06-29T16:05:42.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.926+0000 7faa83092700 1 -- 172.21.15.60:0/3035341158 shutdown_connections 2022-06-29T16:05:42.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.926+0000 7faa83092700 1 --2- 172.21.15.60:0/3035341158 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7faa5c0539e0 0x7faa5c055ea0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:42.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.926+0000 7faa83092700 1 --2- 172.21.15.60:0/3035341158 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa7c0b1500 0x7faa7c144220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:42.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.926+0000 7faa83092700 1 --2- 172.21.15.60:0/3035341158 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faa7c0b0e60 0x7faa7c147fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:42.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.926+0000 7faa83092700 1 --2- 172.21.15.60:0/3035341158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa7c05fb00 0x7faa7c146920 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:42.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.926+0000 7faa83092700 1 -- 172.21.15.60:0/3035341158 >> 172.21.15.60:0/3035341158 conn(0x7faa7c063ab0 msgr2=0x7faa7c0b3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:42.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.926+0000 7faa83092700 1 -- 172.21.15.60:0/3035341158 shutdown_connections 2022-06-29T16:05:42.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:42.926+0000 7faa83092700 1 -- 172.21.15.60:0/3035341158 wait complete. 2022-06-29T16:05:42.936 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.2 2022-06-29T16:05:42.949 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:05:43.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.050+0000 7f511b59e700 1 Processor -- start 2022-06-29T16:05:43.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.050+0000 7f511b59e700 1 -- start start 2022-06-29T16:05:43.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.050+0000 7f511b59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f511c0b1160 0x7f511c0b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:43.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.050+0000 7f511b59e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f511c14b630 con 0x7f511c05fb00 2022-06-29T16:05:43.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.050+0000 7f511b59e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f511c05fee0 con 0x7f511c14aa90 2022-06-29T16:05:43.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.050+0000 7f511b59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f511c0af160 con 0x7f511c0b1160 2022-06-29T16:05:43.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.050+0000 7f511a59c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f511c05fb00 0x7f511c0aea50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58900/0 (socket says 172.21.15.60:58900) 2022-06-29T16:05:43.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.050+0000 7f511a59c700 1 -- 172.21.15.60:0/2901082820 learned_addr learned my addr 172.21.15.60:0/2901082820 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:43.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.050+0000 7f5119d9b700 1 --2- 172.21.15.60:0/2901082820 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f511c0b1160 0x7f511c0b15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:43.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.050+0000 7f511959a700 1 -- 172.21.15.60:0/2901082820 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 167020012 0 0) 0x7f511c14b630 con 0x7f511c05fb00 2022-06-29T16:05:43.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.050+0000 7f511959a700 1 -- 172.21.15.60:0/2901082820 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5100003260 con 0x7f511c05fb00 2022-06-29T16:05:43.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.050+0000 7f5119d9b700 1 -- 172.21.15.60:0/2901082820 >> v1:172.21.15.92:6789/0 conn(0x7f511c14aa90 legacy=0x7f511c14af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:43.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.050+0000 7f5119d9b700 1 -- 172.21.15.60:0/2901082820 >> v1:172.21.15.60:6789/0 conn(0x7f511c05fb00 legacy=0x7f511c0aea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:43.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.050+0000 7f5119d9b700 1 -- 172.21.15.60:0/2901082820 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f511c0b0870 con 0x7f511c0b1160 2022-06-29T16:05:43.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.050+0000 7f5119d9b700 1 --2- 172.21.15.60:0/2901082820 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f511c0b1160 0x7f511c0b15a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f510c002f70 tx=0x7f510c006df0).ready entity=mon.2 client_cookie=bded729f684fdf8 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:43.063 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.050+0000 7f511959a700 1 -- 172.21.15.60:0/2901082820 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f510c00d570 con 0x7f511c0b1160 2022-06-29T16:05:43.063 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.050+0000 7f511959a700 1 -- 172.21.15.60:0/2901082820 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f510c000e90 con 0x7f511c0b1160 2022-06-29T16:05:43.063 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.050+0000 7f511959a700 1 -- 172.21.15.60:0/2901082820 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f510c00d8c0 con 0x7f511c0b1160 2022-06-29T16:05:43.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.054+0000 7f511b59e700 1 -- 172.21.15.60:0/2901082820 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f511c0b1160 msgr2=0x7f511c0b15a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:43.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.054+0000 7f511b59e700 1 --2- 172.21.15.60:0/2901082820 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f511c0b1160 0x7f511c0b15a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f510c002f70 tx=0x7f510c006df0).stop 2022-06-29T16:05:43.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.054+0000 7f511b59e700 1 -- 172.21.15.60:0/2901082820 shutdown_connections 2022-06-29T16:05:43.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.054+0000 7f511b59e700 1 --2- 172.21.15.60:0/2901082820 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f511c0b1160 0x7f511c0b15a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:43.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.054+0000 7f511b59e700 1 -- 172.21.15.60:0/2901082820 >> 172.21.15.60:0/2901082820 conn(0x7f511c063a60 msgr2=0x7f511c05f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:43.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.054+0000 7f511b59e700 1 -- 172.21.15.60:0/2901082820 shutdown_connections 2022-06-29T16:05:43.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.054+0000 7f511b59e700 1 -- 172.21.15.60:0/2901082820 wait complete. 2022-06-29T16:05:43.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.054+0000 7f511b59e700 1 Processor -- start 2022-06-29T16:05:43.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.054+0000 7f511b59e700 1 -- start start 2022-06-29T16:05:43.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.058+0000 7f511b59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f511c05fb00 0x7f511c066b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:43.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.058+0000 7f511b59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f511c0b1160 0x7f511c1596e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:43.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.058+0000 7f511b59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f511c14aa90 0x7f511c11dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:43.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.058+0000 7f511b59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511c11da70 con 0x7f511c0b1160 2022-06-29T16:05:43.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.058+0000 7f511b59e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511c11c4f0 con 0x7f511c05fb00 2022-06-29T16:05:43.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.058+0000 7f511b59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f511c11c210 con 0x7f511c14aa90 2022-06-29T16:05:43.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.058+0000 7f5119d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f511c0b1160 0x7f511c1596e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:43.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.058+0000 7f5119d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f511c0b1160 0x7f511c1596e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50822/0 (socket says 172.21.15.60:50822) 2022-06-29T16:05:43.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.058+0000 7f5119d9b700 1 -- 172.21.15.60:0/2978438822 learned_addr learned my addr 172.21.15.60:0/2978438822 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:43.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.058+0000 7f511a59c700 1 --2- 172.21.15.60:0/2978438822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f511c05fb00 0x7f511c066b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:43.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.058+0000 7f511ad9d700 1 --2- 172.21.15.60:0/2978438822 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f511c14aa90 0x7f511c11dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:43.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.058+0000 7f5119d9b700 1 -- 172.21.15.60:0/2978438822 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f511c14aa90 msgr2=0x7f511c11dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:43.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.058+0000 7f5119d9b700 1 --2- 172.21.15.60:0/2978438822 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f511c14aa90 0x7f511c11dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:43.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.058+0000 7f5119d9b700 1 -- 172.21.15.60:0/2978438822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f511c05fb00 msgr2=0x7f511c066b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:43.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.058+0000 7f5119d9b700 1 --2- 172.21.15.60:0/2978438822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f511c05fb00 0x7f511c066b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:43.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.058+0000 7f5119d9b700 1 -- 172.21.15.60:0/2978438822 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f510c00b180 con 0x7f511c0b1160 2022-06-29T16:05:43.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.058+0000 7f5119d9b700 1 --2- 172.21.15.60:0/2978438822 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f511c0b1160 0x7f511c1596e0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 rx=0x7f510c002f70 tx=0x7f510c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:43.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.058+0000 7f510b7fe700 1 -- 172.21.15.60:0/2978438822 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f510c016070 con 0x7f511c0b1160 2022-06-29T16:05:43.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.058+0000 7f511b59e700 1 -- 172.21.15.60:0/2978438822 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f511c111470 con 0x7f511c0b1160 2022-06-29T16:05:43.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.058+0000 7f510b7fe700 1 -- 172.21.15.60:0/2978438822 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f510c0139c0 con 0x7f511c0b1160 2022-06-29T16:05:43.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.058+0000 7f510b7fe700 1 -- 172.21.15.60:0/2978438822 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f510c013d20 con 0x7f511c0b1160 2022-06-29T16:05:43.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.058+0000 7f511b59e700 1 -- 172.21.15.60:0/2978438822 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f511c1120c0 con 0x7f511c0b1160 2022-06-29T16:05:43.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.062+0000 7f511b59e700 1 -- 172.21.15.60:0/2978438822 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f511c111b20 con 0x7f511c0b1160 2022-06-29T16:05:43.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.062+0000 7f510b7fe700 1 -- 172.21.15.60:0/2978438822 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f510c00f030 con 0x7f511c0b1160 2022-06-29T16:05:43.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.062+0000 7f510b7fe700 1 --2- 172.21.15.60:0/2978438822 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f50f8053a10 0x7f50f8055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:43.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.062+0000 7f510b7fe700 1 -- 172.21.15.60:0/2978438822 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f510c06ccf0 con 0x7f511c0b1160 2022-06-29T16:05:43.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.062+0000 7f511a59c700 1 --2- 172.21.15.60:0/2978438822 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f50f8053a10 0x7f50f8055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:43.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.062+0000 7f511a59c700 1 --2- 172.21.15.60:0/2978438822 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f50f8053a10 0x7f50f8055ed0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f511000d000 tx=0x7f5110009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:43.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.066+0000 7f510b7fe700 1 -- 172.21.15.60:0/2978438822 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f510c043b70 con 0x7f511c0b1160 2022-06-29T16:05:43.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.230+0000 7f511b59e700 1 -- 172.21.15.60:0/2978438822 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f511c0573d0 con 0x7f511c0b1160 2022-06-29T16:05:43.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.230+0000 7f510b7fe700 1 -- 172.21.15.60:0/2978438822 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f510c023020 con 0x7f511c0b1160 2022-06-29T16:05:43.238 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:43.238 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":41,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6182,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1306898602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1306898602},{"type":"v1","addr":"172.21.15.60:6833","nonce":1306898602}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":39}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:05:40.369985+0000","modified":"2022-06-29T16:05:41.373815+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6185":{"gid":6185,"name":"b","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/2057420852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2057420852},{"type":"v1","addr":"172.21.15.92:6835","nonce":2057420852}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-06-29T16:05:43.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.234+0000 7f511b59e700 1 -- 172.21.15.60:0/2978438822 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f50f8053a10 msgr2=0x7f50f8055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:43.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.234+0000 7f511b59e700 1 --2- 172.21.15.60:0/2978438822 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f50f8053a10 0x7f50f8055ed0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f511000d000 tx=0x7f5110009040).stop 2022-06-29T16:05:43.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.234+0000 7f511b59e700 1 -- 172.21.15.60:0/2978438822 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f511c0b1160 msgr2=0x7f511c1596e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:43.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.234+0000 7f511b59e700 1 --2- 172.21.15.60:0/2978438822 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f511c0b1160 0x7f511c1596e0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 rx=0x7f510c002f70 tx=0x7f510c014000).stop 2022-06-29T16:05:43.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.234+0000 7f511b59e700 1 -- 172.21.15.60:0/2978438822 shutdown_connections 2022-06-29T16:05:43.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.234+0000 7f511b59e700 1 --2- 172.21.15.60:0/2978438822 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f50f8053a10 0x7f50f8055ed0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:43.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.234+0000 7f511b59e700 1 --2- 172.21.15.60:0/2978438822 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f511c14aa90 0x7f511c11dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:43.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.234+0000 7f511b59e700 1 --2- 172.21.15.60:0/2978438822 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f511c0b1160 0x7f511c1596e0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:43.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.234+0000 7f511b59e700 1 --2- 172.21.15.60:0/2978438822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f511c05fb00 0x7f511c066b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:43.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.234+0000 7f511b59e700 1 -- 172.21.15.60:0/2978438822 >> 172.21.15.60:0/2978438822 conn(0x7f511c063a60 msgr2=0x7f511c0af340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:43.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.234+0000 7f511b59e700 1 -- 172.21.15.60:0/2978438822 shutdown_connections 2022-06-29T16:05:43.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.234+0000 7f511b59e700 1 -- 172.21.15.60:0/2978438822 wait complete. 2022-06-29T16:05:43.244 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 41 2022-06-29T16:05:43.456 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 41, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:05:40.369985+0000', 'modified': '2022-06-29T16:05:41.373815+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6185}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6185': {'gid': 6185, 'name': 'b', 'rank': 0, 'incarnation': 40, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.92:6835/2057420852', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 2057420852}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 2057420852}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:05:43.456 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:05:43.457 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-06-29T16:05:43.574 INFO:teuthology.orchestra.run.smithi092.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":6185,"want_state":"up:active","state":"up:active","rank_uptime":3.1875896340000001,"mdsmap_epoch":41,"osdmap_epoch":91,"osdmap_epoch_barrier":90,"uptime":7.9114889509999999} 2022-06-29T16:05:43.574 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 6185, "mdsmap_epoch": 41, "osdmap_epoch": 91, "osdmap_epoch_barrier": 90, "rank_uptime": 3.187589634, "state": "up:active", "uptime": 7.911488951, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:05:43.575 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:05:43.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.678+0000 7f31108e0700 1 Processor -- start 2022-06-29T16:05:43.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.678+0000 7f31108e0700 1 -- start start 2022-06-29T16:05:43.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.678+0000 7f31108e0700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f310c05d070 0x7f310c0b1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:43.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.678+0000 7f31108e0700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f310c0aee30 0x7f310c141b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:43.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.678+0000 7f31108e0700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f310c05d450 con 0x7f310c13a460 2022-06-29T16:05:43.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.678+0000 7f31108e0700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f310c143a90 con 0x7f310c05d070 2022-06-29T16:05:43.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.678+0000 7f31108e0700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f310c062200 con 0x7f310c0aee30 2022-06-29T16:05:43.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.678+0000 7f310b59e700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f310c13a460 0x7f310c140bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58918/0 (socket says 172.21.15.60:58918) 2022-06-29T16:05:43.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.678+0000 7f310b59e700 1 -- 172.21.15.60:0/1420687783 learned_addr learned my addr 172.21.15.60:0/1420687783 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:43.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.678+0000 7f310ad9d700 1 --2- 172.21.15.60:0/1420687783 >> v2:172.21.15.60:3300/0 conn(0x7f310c05d070 0x7f310c0b1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:43.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.678+0000 7f310a59c700 1 --2- 172.21.15.60:0/1420687783 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f310c0aee30 0x7f310c141b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:43.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.678+0000 7f3109d9b700 1 -- 172.21.15.60:0/1420687783 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4079692518 0 0) 0x7f310c05d450 con 0x7f310c13a460 2022-06-29T16:05:43.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.678+0000 7f3109d9b700 1 -- 172.21.15.60:0/1420687783 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30ec003260 con 0x7f310c13a460 2022-06-29T16:05:43.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.678+0000 7f310ad9d700 1 -- 172.21.15.60:0/1420687783 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f310c0aee30 msgr2=0x7f310c141b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:43.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.678+0000 7f310ad9d700 1 --2- 172.21.15.60:0/1420687783 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f310c0aee30 0x7f310c141b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:43.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.678+0000 7f310ad9d700 1 -- 172.21.15.60:0/1420687783 >> v1:172.21.15.60:6789/0 conn(0x7f310c13a460 legacy=0x7f310c140bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:43.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.678+0000 7f310ad9d700 1 -- 172.21.15.60:0/1420687783 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f310c064dc0 con 0x7f310c05d070 2022-06-29T16:05:43.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.682+0000 7f310ad9d700 1 --2- 172.21.15.60:0/1420687783 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f310c05d070 0x7f310c0b1f30 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 rx=0x7f30f4002f70 tx=0x7f30f4006df0).ready entity=mon.0 client_cookie=fe18d0e7a616f1d7 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:43.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.682+0000 7f3109d9b700 1 -- 172.21.15.60:0/1420687783 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30f400d4a0 con 0x7f310c05d070 2022-06-29T16:05:43.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.682+0000 7f3109d9b700 1 -- 172.21.15.60:0/1420687783 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30f4000e90 con 0x7f310c05d070 2022-06-29T16:05:43.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.682+0000 7f3109d9b700 1 -- 172.21.15.60:0/1420687783 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30f400d7f0 con 0x7f310c05d070 2022-06-29T16:05:43.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.682+0000 7f31108e0700 1 -- 172.21.15.60:0/1420687783 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f310c05d070 msgr2=0x7f310c0b1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:43.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.682+0000 7f31108e0700 1 --2- 172.21.15.60:0/1420687783 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f310c05d070 0x7f310c0b1f30 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 rx=0x7f30f4002f70 tx=0x7f30f4006df0).stop 2022-06-29T16:05:43.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.682+0000 7f31108e0700 1 -- 172.21.15.60:0/1420687783 shutdown_connections 2022-06-29T16:05:43.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.682+0000 7f31108e0700 1 --2- 172.21.15.60:0/1420687783 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f310c0aee30 0x7f310c141b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:43.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.682+0000 7f31108e0700 1 --2- 172.21.15.60:0/1420687783 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f310c05d070 0x7f310c0b1f30 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:43.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.682+0000 7f31108e0700 1 -- 172.21.15.60:0/1420687783 >> 172.21.15.60:0/1420687783 conn(0x7f310c0b5890 msgr2=0x7f310c0b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:43.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.682+0000 7f31108e0700 1 -- 172.21.15.60:0/1420687783 shutdown_connections 2022-06-29T16:05:43.696 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.682+0000 7f31108e0700 1 -- 172.21.15.60:0/1420687783 wait complete. 2022-06-29T16:05:43.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.682+0000 7f31108e0700 1 Processor -- start 2022-06-29T16:05:43.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.682+0000 7f31108e0700 1 -- start start 2022-06-29T16:05:43.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.686+0000 7f31108e0700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f310c05d070 0x7f310c0acd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:43.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.686+0000 7f31108e0700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f310c0aee30 0x7f310c13c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:43.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.686+0000 7f31108e0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f310c13a460 0x7f310c13ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:43.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.686+0000 7f31108e0700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310c139060 con 0x7f310c13a460 2022-06-29T16:05:43.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.686+0000 7f31108e0700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310c13c4e0 con 0x7f310c0aee30 2022-06-29T16:05:43.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.686+0000 7f31108e0700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f310c155d20 con 0x7f310c05d070 2022-06-29T16:05:43.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.686+0000 7f310b59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f310c13a460 0x7f310c13ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:43.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.686+0000 7f310b59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f310c13a460 0x7f310c13ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50838/0 (socket says 172.21.15.60:50838) 2022-06-29T16:05:43.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.686+0000 7f310b59e700 1 -- 172.21.15.60:0/493011677 learned_addr learned my addr 172.21.15.60:0/493011677 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:43.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.686+0000 7f310ad9d700 1 --2- 172.21.15.60:0/493011677 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f310c05d070 0x7f310c0acd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:43.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.686+0000 7f310a59c700 1 --2- 172.21.15.60:0/493011677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f310c0aee30 0x7f310c13c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:43.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.686+0000 7f310b59e700 1 -- 172.21.15.60:0/493011677 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f310c05d070 msgr2=0x7f310c0acd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:43.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.686+0000 7f310b59e700 1 --2- 172.21.15.60:0/493011677 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f310c05d070 0x7f310c0acd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:43.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.686+0000 7f310b59e700 1 -- 172.21.15.60:0/493011677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f310c0aee30 msgr2=0x7f310c13c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:43.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.686+0000 7f310b59e700 1 --2- 172.21.15.60:0/493011677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f310c0aee30 0x7f310c13c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:43.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.686+0000 7f310b59e700 1 -- 172.21.15.60:0/493011677 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f400b180 con 0x7f310c13a460 2022-06-29T16:05:43.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.686+0000 7f310a59c700 1 --2- 172.21.15.60:0/493011677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f310c0aee30 0x7f310c13c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:43.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.686+0000 7f310b59e700 1 --2- 172.21.15.60:0/493011677 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f310c13a460 0x7f310c13ce00 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 rx=0x7f3104001490 tx=0x7f3104007da0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:43.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.686+0000 7f3103fff700 1 -- 172.21.15.60:0/493011677 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3104005c70 con 0x7f310c13a460 2022-06-29T16:05:43.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.686+0000 7f3103fff700 1 -- 172.21.15.60:0/493011677 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3104005dd0 con 0x7f310c13a460 2022-06-29T16:05:43.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.686+0000 7f31108e0700 1 -- 172.21.15.60:0/493011677 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f310c155fc0 con 0x7f310c13a460 2022-06-29T16:05:43.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.686+0000 7f3103fff700 1 -- 172.21.15.60:0/493011677 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f31040072c0 con 0x7f310c13a460 2022-06-29T16:05:43.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.686+0000 7f31108e0700 1 -- 172.21.15.60:0/493011677 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f310c0ad250 con 0x7f310c13a460 2022-06-29T16:05:43.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.686+0000 7f31108e0700 1 -- 172.21.15.60:0/493011677 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f310c13c650 con 0x7f310c13a460 2022-06-29T16:05:43.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.690+0000 7f3103fff700 1 -- 172.21.15.60:0/493011677 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3104010070 con 0x7f310c13a460 2022-06-29T16:05:43.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.690+0000 7f3103fff700 1 --2- 172.21.15.60:0/493011677 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f30f0053950 0x7f30f0055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:43.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.690+0000 7f3103fff700 1 -- 172.21.15.60:0/493011677 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f310406b7f0 con 0x7f310c13a460 2022-06-29T16:05:43.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.690+0000 7f310ad9d700 1 --2- 172.21.15.60:0/493011677 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f30f0053950 0x7f30f0055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:43.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.690+0000 7f310ad9d700 1 --2- 172.21.15.60:0/493011677 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f30f0053950 0x7f30f0055e10 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f30f401a000 tx=0x7f30f4014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:43.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.694+0000 7f3103fff700 1 -- 172.21.15.60:0/493011677 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f310c13c650 con 0x7f310c13a460 2022-06-29T16:05:43.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.822+0000 7f31108e0700 1 -- 172.21.15.60:0/493011677 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f310c13c650 con 0x7f310c13a460 2022-06-29T16:05:43.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.822+0000 7f3103fff700 1 -- 172.21.15.60:0/493011677 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+16041 (secure 0 0 0) 0x7f310c13c650 con 0x7f310c13a460 2022-06-29T16:05:43.832 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:43.833 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":91,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:05:41.373483+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:05:38.407939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2022-06-29T16:05:39.380608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:05:43.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.826+0000 7f31108e0700 1 -- 172.21.15.60:0/493011677 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f30f0053950 msgr2=0x7f30f0055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:43.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.826+0000 7f31108e0700 1 --2- 172.21.15.60:0/493011677 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f30f0053950 0x7f30f0055e10 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f30f401a000 tx=0x7f30f4014000).stop 2022-06-29T16:05:43.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.826+0000 7f31108e0700 1 -- 172.21.15.60:0/493011677 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f310c13a460 msgr2=0x7f310c13ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:43.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.826+0000 7f31108e0700 1 --2- 172.21.15.60:0/493011677 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f310c13a460 0x7f310c13ce00 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 rx=0x7f3104001490 tx=0x7f3104007da0).stop 2022-06-29T16:05:43.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.826+0000 7f31108e0700 1 -- 172.21.15.60:0/493011677 shutdown_connections 2022-06-29T16:05:43.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.826+0000 7f31108e0700 1 --2- 172.21.15.60:0/493011677 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f30f0053950 0x7f30f0055e10 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:43.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.826+0000 7f31108e0700 1 --2- 172.21.15.60:0/493011677 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f310c13a460 0x7f310c13ce00 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:43.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.826+0000 7f31108e0700 1 --2- 172.21.15.60:0/493011677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f310c0aee30 0x7f310c13c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:43.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.826+0000 7f31108e0700 1 --2- 172.21.15.60:0/493011677 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f310c05d070 0x7f310c0acd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:43.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.826+0000 7f31108e0700 1 -- 172.21.15.60:0/493011677 >> 172.21.15.60:0/493011677 conn(0x7f310c0b5890 msgr2=0x7f310c139b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:43.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.826+0000 7f31108e0700 1 -- 172.21.15.60:0/493011677 shutdown_connections 2022-06-29T16:05:43.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.826+0000 7f31108e0700 1 -- 172.21.15.60:0/493011677 wait complete. 2022-06-29T16:05:43.846 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-29T16:05:38.407939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2022-06-29T16:05:39.380608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-29T16:05:43.847 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:05:43.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.946+0000 7f20d0790700 1 Processor -- start 2022-06-29T16:05:43.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.946+0000 7f20d0790700 1 -- start start 2022-06-29T16:05:43.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.946+0000 7f20d0790700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f20c80b3d30 0x7f20c8146ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:43.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.946+0000 7f20d0790700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20c8061d90 0x7f20c8142b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:43.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.946+0000 7f20d0790700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20c80b4f50 con 0x7f20c80b9d70 2022-06-29T16:05:43.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.946+0000 7f20d0790700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20c80ba150 con 0x7f20c80b3d30 2022-06-29T16:05:43.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.946+0000 7f20d0790700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20c80b5180 con 0x7f20c8061d90 2022-06-29T16:05:43.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.946+0000 7f20cdd2b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f20c80b3d30 0x7f20c8146ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:43.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.950+0000 7f20cdd2b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f20c80b3d30 0x7f20c8146ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50846/0 (socket says 172.21.15.60:50846) 2022-06-29T16:05:43.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.950+0000 7f20cdd2b700 1 -- 172.21.15.60:0/1240433756 learned_addr learned my addr 172.21.15.60:0/1240433756 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:43.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.950+0000 7f20ced2d700 1 --2- 172.21.15.60:0/1240433756 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20c8061d90 0x7f20c8142b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:43.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.950+0000 7f20cdd2b700 1 -- 172.21.15.60:0/1240433756 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20c8061d90 msgr2=0x7f20c8142b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:43.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.950+0000 7f20cdd2b700 1 --2- 172.21.15.60:0/1240433756 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20c8061d90 0x7f20c8142b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:43.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.950+0000 7f20cdd2b700 1 -- 172.21.15.60:0/1240433756 >> v1:172.21.15.92:6789/0 conn(0x7f20c80b9d70 legacy=0x7f20c81465d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:43.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.950+0000 7f20cdd2b700 1 -- 172.21.15.60:0/1240433756 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c81461a0 con 0x7f20c80b3d30 2022-06-29T16:05:43.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.950+0000 7f20cdd2b700 1 --2- 172.21.15.60:0/1240433756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20c80b3d30 0x7f20c8146ce0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 rx=0x7f20b8002f70 tx=0x7f20b8006eb0).ready entity=mon.0 client_cookie=d099ee189158c30b server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:43.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.950+0000 7f20cd52a700 1 -- 172.21.15.60:0/1240433756 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f20b800d570 con 0x7f20c80b3d30 2022-06-29T16:05:43.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.950+0000 7f20cd52a700 1 -- 172.21.15.60:0/1240433756 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20b8000e90 con 0x7f20c80b3d30 2022-06-29T16:05:43.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.950+0000 7f20cd52a700 1 -- 172.21.15.60:0/1240433756 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f20b800d900 con 0x7f20c80b3d30 2022-06-29T16:05:43.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.950+0000 7f20d0790700 1 -- 172.21.15.60:0/1240433756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20c80b3d30 msgr2=0x7f20c8146ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:43.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.950+0000 7f20d0790700 1 --2- 172.21.15.60:0/1240433756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20c80b3d30 0x7f20c8146ce0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 rx=0x7f20b8002f70 tx=0x7f20b8006eb0).stop 2022-06-29T16:05:43.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.950+0000 7f20d0790700 1 -- 172.21.15.60:0/1240433756 shutdown_connections 2022-06-29T16:05:43.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.950+0000 7f20d0790700 1 --2- 172.21.15.60:0/1240433756 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20c8061d90 0x7f20c8142b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:43.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.950+0000 7f20d0790700 1 --2- 172.21.15.60:0/1240433756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20c80b3d30 0x7f20c8146ce0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:43.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.950+0000 7f20d0790700 1 -- 172.21.15.60:0/1240433756 >> 172.21.15.60:0/1240433756 conn(0x7f20c8062e70 msgr2=0x7f20c80b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:43.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.950+0000 7f20d0790700 1 -- 172.21.15.60:0/1240433756 shutdown_connections 2022-06-29T16:05:43.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.950+0000 7f20d0790700 1 -- 172.21.15.60:0/1240433756 wait complete. 2022-06-29T16:05:43.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.950+0000 7f20d0790700 1 Processor -- start 2022-06-29T16:05:43.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.954+0000 7f20d0790700 1 -- start start 2022-06-29T16:05:43.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.954+0000 7f20d0790700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20c8061d90 0x7f20c81139a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:43.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.954+0000 7f20d0790700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20c80b3d30 0x7f20c8140c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:43.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.954+0000 7f20d0790700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20c80b9d70 0x7f20c8141180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:43.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.954+0000 7f20d0790700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c8152c90 con 0x7f20c80b3d30 2022-06-29T16:05:43.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.954+0000 7f20d0790700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c8152780 con 0x7f20c80b9d70 2022-06-29T16:05:43.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.954+0000 7f20d0790700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20c81529f0 con 0x7f20c8061d90 2022-06-29T16:05:43.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.954+0000 7f20cdd2b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20c80b3d30 0x7f20c8140c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:43.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.954+0000 7f20cdd2b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20c80b3d30 0x7f20c8140c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50850/0 (socket says 172.21.15.60:50850) 2022-06-29T16:05:43.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.954+0000 7f20cdd2b700 1 -- 172.21.15.60:0/2253243817 learned_addr learned my addr 172.21.15.60:0/2253243817 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:43.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.954+0000 7f20ce52c700 1 --2- 172.21.15.60:0/2253243817 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20c8061d90 0x7f20c81139a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:43.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.954+0000 7f20ced2d700 1 --2- 172.21.15.60:0/2253243817 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20c80b9d70 0x7f20c8141180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:43.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.954+0000 7f20cdd2b700 1 -- 172.21.15.60:0/2253243817 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20c8061d90 msgr2=0x7f20c81139a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:43.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.954+0000 7f20cdd2b700 1 --2- 172.21.15.60:0/2253243817 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20c8061d90 0x7f20c81139a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:43.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.954+0000 7f20cdd2b700 1 -- 172.21.15.60:0/2253243817 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20c80b9d70 msgr2=0x7f20c8141180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:43.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.954+0000 7f20cdd2b700 1 --2- 172.21.15.60:0/2253243817 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20c80b9d70 0x7f20c8141180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:43.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.954+0000 7f20cdd2b700 1 -- 172.21.15.60:0/2253243817 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b800b1e0 con 0x7f20c80b3d30 2022-06-29T16:05:43.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.954+0000 7f20cdd2b700 1 --2- 172.21.15.60:0/2253243817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20c80b3d30 0x7f20c8140c90 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 rx=0x7f20b8002f70 tx=0x7f20b8015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:43.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.954+0000 7f20b77fe700 1 -- 172.21.15.60:0/2253243817 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f20b80068b0 con 0x7f20c80b3d30 2022-06-29T16:05:43.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.954+0000 7f20b77fe700 1 -- 172.21.15.60:0/2253243817 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20b800d8f0 con 0x7f20c80b3d30 2022-06-29T16:05:43.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.954+0000 7f20d0790700 1 -- 172.21.15.60:0/2253243817 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20c8141670 con 0x7f20c80b3d30 2022-06-29T16:05:43.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.954+0000 7f20b77fe700 1 -- 172.21.15.60:0/2253243817 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f20b8011d70 con 0x7f20c80b3d30 2022-06-29T16:05:43.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.954+0000 7f20d0790700 1 -- 172.21.15.60:0/2253243817 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20c80ad4d0 con 0x7f20c80b3d30 2022-06-29T16:05:43.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.958+0000 7f20d0790700 1 -- 172.21.15.60:0/2253243817 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20c80573d0 con 0x7f20c80b3d30 2022-06-29T16:05:43.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.958+0000 7f20b77fe700 1 -- 172.21.15.60:0/2253243817 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f20b8025030 con 0x7f20c80b3d30 2022-06-29T16:05:43.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.958+0000 7f20b77fe700 1 --2- 172.21.15.60:0/2253243817 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f20a8053950 0x7f20a8055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:43.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.958+0000 7f20b77fe700 1 -- 172.21.15.60:0/2253243817 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f20b806d060 con 0x7f20c80b3d30 2022-06-29T16:05:43.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.958+0000 7f20ce52c700 1 --2- 172.21.15.60:0/2253243817 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f20a8053950 0x7f20a8055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:43.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.958+0000 7f20ce52c700 1 --2- 172.21.15.60:0/2253243817 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f20a8053950 0x7f20a8055e10 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f20c400d000 tx=0x7f20c4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:43.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:43.966+0000 7f20b77fe700 1 -- 172.21.15.60:0/2253243817 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f20b8043f20 con 0x7f20c80b3d30 2022-06-29T16:05:44.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.094+0000 7f20d0790700 1 -- 172.21.15.60:0/2253243817 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f20c805da40 con 0x7f20c80b3d30 2022-06-29T16:05:44.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.094+0000 7f20b77fe700 1 -- 172.21.15.60:0/2253243817 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v91) v1 ==== 110+0+10 (secure 0 0 0) 0x7f20b8023070 con 0x7f20c80b3d30 2022-06-29T16:05:44.102 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:05:44.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.094+0000 7f20d0790700 1 -- 172.21.15.60:0/2253243817 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f20a8053950 msgr2=0x7f20a8055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:44.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.094+0000 7f20d0790700 1 --2- 172.21.15.60:0/2253243817 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f20a8053950 0x7f20a8055e10 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f20c400d000 tx=0x7f20c4009040).stop 2022-06-29T16:05:44.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.094+0000 7f20d0790700 1 -- 172.21.15.60:0/2253243817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20c80b3d30 msgr2=0x7f20c8140c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:44.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.094+0000 7f20d0790700 1 --2- 172.21.15.60:0/2253243817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20c80b3d30 0x7f20c8140c90 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 rx=0x7f20b8002f70 tx=0x7f20b8015000).stop 2022-06-29T16:05:44.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.094+0000 7f20d0790700 1 -- 172.21.15.60:0/2253243817 shutdown_connections 2022-06-29T16:05:44.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.094+0000 7f20d0790700 1 --2- 172.21.15.60:0/2253243817 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f20a8053950 0x7f20a8055e10 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:44.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.094+0000 7f20d0790700 1 --2- 172.21.15.60:0/2253243817 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20c80b9d70 0x7f20c8141180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:44.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.094+0000 7f20d0790700 1 --2- 172.21.15.60:0/2253243817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20c80b3d30 0x7f20c8140c90 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:44.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.094+0000 7f20d0790700 1 --2- 172.21.15.60:0/2253243817 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f20c8061d90 0x7f20c81139a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:44.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.094+0000 7f20d0790700 1 -- 172.21.15.60:0/2253243817 >> 172.21.15.60:0/2253243817 conn(0x7f20c8062e70 msgr2=0x7f20c80b1950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:44.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.094+0000 7f20d0790700 1 -- 172.21.15.60:0/2253243817 shutdown_connections 2022-06-29T16:05:44.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.094+0000 7f20d0790700 1 -- 172.21.15.60:0/2253243817 wait complete. 2022-06-29T16:05:44.115 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:05:44.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.214+0000 7f0e01369700 1 Processor -- start 2022-06-29T16:05:44.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.214+0000 7f0e01369700 1 -- start start 2022-06-29T16:05:44.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.214+0000 7f0e01369700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0dfc05d070 0x7f0dfc14aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:44.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.214+0000 7f0e01369700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0dfc14af80 0x7f0dfc0ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:44.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.214+0000 7f0e01369700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dfc05d450 con 0x7f0dfc0b2130 2022-06-29T16:05:44.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.214+0000 7f0e01369700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dfc0adf90 con 0x7f0dfc05d070 2022-06-29T16:05:44.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.214+0000 7f0e01369700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc0b25d0 con 0x7f0dfc14af80 2022-06-29T16:05:44.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.214+0000 7f0dfbfff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0dfc05d070 0x7f0dfc14aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:44.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.214+0000 7f0dfb7fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f0dfc0b2130 0x7f0dfc0aca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58942/0 (socket says 172.21.15.60:58942) 2022-06-29T16:05:44.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.214+0000 7f0dfb7fe700 1 -- 172.21.15.60:0/4114939533 learned_addr learned my addr 172.21.15.60:0/4114939533 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:44.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.214+0000 7f0e00b68700 1 --2- 172.21.15.60:0/4114939533 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0dfc14af80 0x7f0dfc0ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:44.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.214+0000 7f0dfbfff700 1 -- 172.21.15.60:0/4114939533 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0dfc14af80 msgr2=0x7f0dfc0ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:44.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.214+0000 7f0dfbfff700 1 --2- 172.21.15.60:0/4114939533 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0dfc14af80 0x7f0dfc0ad130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:44.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.214+0000 7f0dfbfff700 1 -- 172.21.15.60:0/4114939533 >> v1:172.21.15.60:6789/0 conn(0x7f0dfc0b2130 legacy=0x7f0dfc0aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:44.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.214+0000 7f0dfbfff700 1 -- 172.21.15.60:0/4114939533 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dfc14a350 con 0x7f0dfc05d070 2022-06-29T16:05:44.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.214+0000 7f0dfbfff700 1 --2- 172.21.15.60:0/4114939533 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0dfc05d070 0x7f0dfc14aa90 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 rx=0x7f0df0002f70 tx=0x7f0df0006df0).ready entity=mon.0 client_cookie=ad3476a4e9e3798d server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:44.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.214+0000 7f0dfaffd700 1 -- 172.21.15.60:0/4114939533 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0df000d570 con 0x7f0dfc05d070 2022-06-29T16:05:44.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.214+0000 7f0dfaffd700 1 -- 172.21.15.60:0/4114939533 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0df0000e90 con 0x7f0dfc05d070 2022-06-29T16:05:44.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.214+0000 7f0dfaffd700 1 -- 172.21.15.60:0/4114939533 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0df000d8c0 con 0x7f0dfc05d070 2022-06-29T16:05:44.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.214+0000 7f0e01369700 1 -- 172.21.15.60:0/4114939533 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0dfc05d070 msgr2=0x7f0dfc14aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:44.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.214+0000 7f0e01369700 1 --2- 172.21.15.60:0/4114939533 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0dfc05d070 0x7f0dfc14aa90 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 rx=0x7f0df0002f70 tx=0x7f0df0006df0).stop 2022-06-29T16:05:44.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.214+0000 7f0e01369700 1 -- 172.21.15.60:0/4114939533 shutdown_connections 2022-06-29T16:05:44.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.214+0000 7f0e01369700 1 --2- 172.21.15.60:0/4114939533 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0dfc14af80 0x7f0dfc0ad130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:44.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.214+0000 7f0e01369700 1 --2- 172.21.15.60:0/4114939533 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0dfc05d070 0x7f0dfc14aa90 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:44.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.214+0000 7f0e01369700 1 -- 172.21.15.60:0/4114939533 >> 172.21.15.60:0/4114939533 conn(0x7f0dfc0b5890 msgr2=0x7f0dfc0b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:44.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.214+0000 7f0e01369700 1 -- 172.21.15.60:0/4114939533 shutdown_connections 2022-06-29T16:05:44.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.214+0000 7f0e01369700 1 -- 172.21.15.60:0/4114939533 wait complete. 2022-06-29T16:05:44.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.218+0000 7f0e01369700 1 Processor -- start 2022-06-29T16:05:44.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.218+0000 7f0e01369700 1 -- start start 2022-06-29T16:05:44.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.218+0000 7f0e01369700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0dfc05d070 0x7f0dfc0668c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:44.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.218+0000 7f0e01369700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0dfc0b2130 0x7f0dfc1597f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:44.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.218+0000 7f0e01369700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0dfc14af80 0x7f0dfc159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:44.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.218+0000 7f0e01369700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc1183b0 con 0x7f0dfc05d070 2022-06-29T16:05:44.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.218+0000 7f0e01369700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc118650 con 0x7f0dfc14af80 2022-06-29T16:05:44.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.218+0000 7f0e01369700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc1188f0 con 0x7f0dfc0b2130 2022-06-29T16:05:44.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.218+0000 7f0dfbfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0dfc05d070 0x7f0dfc0668c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:44.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.218+0000 7f0dfbfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0dfc05d070 0x7f0dfc0668c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50862/0 (socket says 172.21.15.60:50862) 2022-06-29T16:05:44.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.218+0000 7f0dfbfff700 1 -- 172.21.15.60:0/2864498419 learned_addr learned my addr 172.21.15.60:0/2864498419 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:44.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.218+0000 7f0dfb7fe700 1 --2- 172.21.15.60:0/2864498419 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0dfc0b2130 0x7f0dfc1597f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:44.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.218+0000 7f0e00b68700 1 --2- 172.21.15.60:0/2864498419 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0dfc14af80 0x7f0dfc159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:44.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.218+0000 7f0dfbfff700 1 -- 172.21.15.60:0/2864498419 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0dfc0b2130 msgr2=0x7f0dfc1597f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:44.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.218+0000 7f0dfbfff700 1 --2- 172.21.15.60:0/2864498419 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0dfc0b2130 0x7f0dfc1597f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:44.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.218+0000 7f0dfbfff700 1 -- 172.21.15.60:0/2864498419 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0dfc14af80 msgr2=0x7f0dfc159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:44.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.218+0000 7f0dfbfff700 1 --2- 172.21.15.60:0/2864498419 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0dfc14af80 0x7f0dfc159ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:44.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.218+0000 7f0dfbfff700 1 -- 172.21.15.60:0/2864498419 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df000b180 con 0x7f0dfc05d070 2022-06-29T16:05:44.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.218+0000 7f0e00b68700 1 --2- 172.21.15.60:0/2864498419 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0dfc14af80 0x7f0dfc159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:44.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.218+0000 7f0dfbfff700 1 --2- 172.21.15.60:0/2864498419 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0dfc05d070 0x7f0dfc0668c0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 rx=0x7f0df0002f70 tx=0x7f0df0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:44.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.218+0000 7f0df97fa700 1 -- 172.21.15.60:0/2864498419 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0df000d8f0 con 0x7f0dfc05d070 2022-06-29T16:05:44.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.218+0000 7f0df97fa700 1 -- 172.21.15.60:0/2864498419 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0df0011bf0 con 0x7f0dfc05d070 2022-06-29T16:05:44.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.218+0000 7f0e01369700 1 -- 172.21.15.60:0/2864498419 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dfc118b90 con 0x7f0dfc05d070 2022-06-29T16:05:44.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.218+0000 7f0df97fa700 1 -- 172.21.15.60:0/2864498419 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0df001e460 con 0x7f0dfc05d070 2022-06-29T16:05:44.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.218+0000 7f0e01369700 1 -- 172.21.15.60:0/2864498419 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dfc116cd0 con 0x7f0dfc05d070 2022-06-29T16:05:44.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.222+0000 7f0e01369700 1 -- 172.21.15.60:0/2864498419 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dfc14a5f0 con 0x7f0dfc05d070 2022-06-29T16:05:44.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.222+0000 7f0df97fa700 1 -- 172.21.15.60:0/2864498419 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0df001a040 con 0x7f0dfc05d070 2022-06-29T16:05:44.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.222+0000 7f0df97fa700 1 --2- 172.21.15.60:0/2864498419 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0ddc053a10 0x7f0ddc055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:44.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.222+0000 7f0df97fa700 1 -- 172.21.15.60:0/2864498419 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0df006cfa0 con 0x7f0dfc05d070 2022-06-29T16:05:44.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.222+0000 7f0dfb7fe700 1 --2- 172.21.15.60:0/2864498419 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0ddc053a10 0x7f0ddc055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:44.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.222+0000 7f0dfb7fe700 1 --2- 172.21.15.60:0/2864498419 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0ddc053a10 0x7f0ddc055ed0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f0dec00d000 tx=0x7f0dec009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:44.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.230+0000 7f0df97fa700 1 -- 172.21.15.60:0/2864498419 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0df0043d70 con 0x7f0dfc05d070 2022-06-29T16:05:44.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.358+0000 7f0e01369700 1 -- 172.21.15.60:0/2864498419 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f0dfc0573d0 con 0x7f0dfc05d070 2022-06-29T16:05:44.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.358+0000 7f0df97fa700 1 -- 172.21.15.60:0/2864498419 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v91) v1 ==== 92+0+10 (secure 0 0 0) 0x7f0df0023020 con 0x7f0dfc05d070 2022-06-29T16:05:44.368 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:05:44.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.362+0000 7f0e01369700 1 -- 172.21.15.60:0/2864498419 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0ddc053a10 msgr2=0x7f0ddc055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:44.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.362+0000 7f0e01369700 1 --2- 172.21.15.60:0/2864498419 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0ddc053a10 0x7f0ddc055ed0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f0dec00d000 tx=0x7f0dec009040).stop 2022-06-29T16:05:44.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.362+0000 7f0e01369700 1 -- 172.21.15.60:0/2864498419 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0dfc05d070 msgr2=0x7f0dfc0668c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:44.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.362+0000 7f0e01369700 1 --2- 172.21.15.60:0/2864498419 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0dfc05d070 0x7f0dfc0668c0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 rx=0x7f0df0002f70 tx=0x7f0df0015000).stop 2022-06-29T16:05:44.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.362+0000 7f0e01369700 1 -- 172.21.15.60:0/2864498419 shutdown_connections 2022-06-29T16:05:44.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.362+0000 7f0e01369700 1 --2- 172.21.15.60:0/2864498419 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0ddc053a10 0x7f0ddc055ed0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:44.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.362+0000 7f0e01369700 1 --2- 172.21.15.60:0/2864498419 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0dfc14af80 0x7f0dfc159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:44.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.362+0000 7f0e01369700 1 --2- 172.21.15.60:0/2864498419 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0dfc0b2130 0x7f0dfc1597f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:44.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.362+0000 7f0e01369700 1 --2- 172.21.15.60:0/2864498419 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0dfc05d070 0x7f0dfc0668c0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:44.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.362+0000 7f0e01369700 1 -- 172.21.15.60:0/2864498419 >> 172.21.15.60:0/2864498419 conn(0x7f0dfc0b5890 msgr2=0x7f0dfc0af720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:44.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.362+0000 7f0e01369700 1 -- 172.21.15.60:0/2864498419 shutdown_connections 2022-06-29T16:05:44.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.362+0000 7f0e01369700 1 -- 172.21.15.60:0/2864498419 wait complete. 2022-06-29T16:05:44.381 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-29T16:05:44.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.486+0000 7f9e3e70b700 1 Processor -- start 2022-06-29T16:05:44.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.486+0000 7f9e3e70b700 1 -- start start 2022-06-29T16:05:44.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.486+0000 7f9e3e70b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9e3805d070 0x7f9e3814aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:44.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.486+0000 7f9e3e70b700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e3805d450 con 0x7f9e380b2130 2022-06-29T16:05:44.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.486+0000 7f9e3e70b700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e380ad840 con 0x7f9e3814af80 2022-06-29T16:05:44.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.486+0000 7f9e3e70b700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e380adf90 con 0x7f9e3805d070 2022-06-29T16:05:44.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.486+0000 7f9e3cf08700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9e380b2130 0x7f9e380aca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58956/0 (socket says 172.21.15.60:58956) 2022-06-29T16:05:44.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.486+0000 7f9e3d709700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9e3805d070 0x7f9e3814aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:44.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.486+0000 7f9e3cf08700 1 -- 172.21.15.60:0/3156353821 learned_addr learned my addr 172.21.15.60:0/3156353821 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:44.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.486+0000 7f9e2ffff700 1 -- 172.21.15.60:0/3156353821 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3628989738 0 0) 0x7f9e3805d450 con 0x7f9e380b2130 2022-06-29T16:05:44.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.486+0000 7f9e2ffff700 1 -- 172.21.15.60:0/3156353821 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e1c003260 con 0x7f9e380b2130 2022-06-29T16:05:44.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.486+0000 7f9e3d709700 1 -- 172.21.15.60:0/3156353821 >> v1:172.21.15.92:6789/0 conn(0x7f9e3814af80 legacy=0x7f9e380ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:44.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.486+0000 7f9e3d709700 1 -- 172.21.15.60:0/3156353821 >> v1:172.21.15.60:6789/0 conn(0x7f9e380b2130 legacy=0x7f9e380aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:44.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.486+0000 7f9e3d709700 1 -- 172.21.15.60:0/3156353821 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e380b25d0 con 0x7f9e3805d070 2022-06-29T16:05:44.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.486+0000 7f9e3d709700 1 --2- 172.21.15.60:0/3156353821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e3805d070 0x7f9e3814aa90 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 rx=0x7f9e34002f70 tx=0x7f9e34006df0).ready entity=mon.0 client_cookie=e5385afb18dc7ad0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:44.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.486+0000 7f9e2ffff700 1 -- 172.21.15.60:0/3156353821 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e3400d610 con 0x7f9e3805d070 2022-06-29T16:05:44.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.486+0000 7f9e2ffff700 1 -- 172.21.15.60:0/3156353821 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e34000e90 con 0x7f9e3805d070 2022-06-29T16:05:44.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.486+0000 7f9e2ffff700 1 -- 172.21.15.60:0/3156353821 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e3400d960 con 0x7f9e3805d070 2022-06-29T16:05:44.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.490+0000 7f9e3e70b700 1 -- 172.21.15.60:0/3156353821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e3805d070 msgr2=0x7f9e3814aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:44.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.490+0000 7f9e3e70b700 1 --2- 172.21.15.60:0/3156353821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e3805d070 0x7f9e3814aa90 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 rx=0x7f9e34002f70 tx=0x7f9e34006df0).stop 2022-06-29T16:05:44.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.490+0000 7f9e3e70b700 1 -- 172.21.15.60:0/3156353821 shutdown_connections 2022-06-29T16:05:44.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.490+0000 7f9e3e70b700 1 --2- 172.21.15.60:0/3156353821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e3805d070 0x7f9e3814aa90 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:44.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.490+0000 7f9e3e70b700 1 -- 172.21.15.60:0/3156353821 >> 172.21.15.60:0/3156353821 conn(0x7f9e380b5890 msgr2=0x7f9e380b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:44.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.490+0000 7f9e3e70b700 1 -- 172.21.15.60:0/3156353821 shutdown_connections 2022-06-29T16:05:44.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.490+0000 7f9e3e70b700 1 -- 172.21.15.60:0/3156353821 wait complete. 2022-06-29T16:05:44.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.490+0000 7f9e3e70b700 1 Processor -- start 2022-06-29T16:05:44.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.490+0000 7f9e3e70b700 1 -- start start 2022-06-29T16:05:44.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.490+0000 7f9e3e70b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e3805d070 0x7f9e38116ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:44.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.490+0000 7f9e3e70b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e380b2130 0x7f9e38119290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:44.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.490+0000 7f9e3e70b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e3814af80 0x7f9e381421d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:44.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.490+0000 7f9e3e70b700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e38117d70 con 0x7f9e3814af80 2022-06-29T16:05:44.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.490+0000 7f9e3e70b700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e38117110 con 0x7f9e380b2130 2022-06-29T16:05:44.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.490+0000 7f9e3e70b700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e381173b0 con 0x7f9e3805d070 2022-06-29T16:05:44.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.490+0000 7f9e3df0a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e3814af80 0x7f9e381421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:44.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.490+0000 7f9e3df0a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e3814af80 0x7f9e381421d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50878/0 (socket says 172.21.15.60:50878) 2022-06-29T16:05:44.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.490+0000 7f9e3df0a700 1 -- 172.21.15.60:0/3985317218 learned_addr learned my addr 172.21.15.60:0/3985317218 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:44.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.490+0000 7f9e3d709700 1 --2- 172.21.15.60:0/3985317218 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e3805d070 0x7f9e38116ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:44.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.490+0000 7f9e3cf08700 1 --2- 172.21.15.60:0/3985317218 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e380b2130 0x7f9e38119290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:44.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.490+0000 7f9e3df0a700 1 -- 172.21.15.60:0/3985317218 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e3805d070 msgr2=0x7f9e38116ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:44.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.490+0000 7f9e3df0a700 1 --2- 172.21.15.60:0/3985317218 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e3805d070 0x7f9e38116ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:44.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.490+0000 7f9e3df0a700 1 -- 172.21.15.60:0/3985317218 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e380b2130 msgr2=0x7f9e38119290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:44.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.490+0000 7f9e3df0a700 1 --2- 172.21.15.60:0/3985317218 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e380b2130 0x7f9e38119290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:44.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.490+0000 7f9e3df0a700 1 -- 172.21.15.60:0/3985317218 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e3400b180 con 0x7f9e3814af80 2022-06-29T16:05:44.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.490+0000 7f9e3df0a700 1 --2- 172.21.15.60:0/3985317218 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e3814af80 0x7f9e381421d0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 rx=0x7f9e380b5cd0 tx=0x7f9e30007f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:44.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.490+0000 7f9e2e7fc700 1 -- 172.21.15.60:0/3985317218 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e30006aa0 con 0x7f9e3814af80 2022-06-29T16:05:44.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.490+0000 7f9e2e7fc700 1 -- 172.21.15.60:0/3985317218 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e30006c00 con 0x7f9e3814af80 2022-06-29T16:05:44.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.490+0000 7f9e3e70b700 1 -- 172.21.15.60:0/3985317218 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e38065ba0 con 0x7f9e3814af80 2022-06-29T16:05:44.510 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.490+0000 7f9e2e7fc700 1 -- 172.21.15.60:0/3985317218 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e30007030 con 0x7f9e3814af80 2022-06-29T16:05:44.510 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.490+0000 7f9e3e70b700 1 -- 172.21.15.60:0/3985317218 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e38066b40 con 0x7f9e3814af80 2022-06-29T16:05:44.511 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.494+0000 7f9e3e70b700 1 -- 172.21.15.60:0/3985317218 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e3814a350 con 0x7f9e3814af80 2022-06-29T16:05:44.511 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.494+0000 7f9e2e7fc700 1 -- 172.21.15.60:0/3985317218 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9e30010030 con 0x7f9e3814af80 2022-06-29T16:05:44.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.494+0000 7f9e2e7fc700 1 --2- 172.21.15.60:0/3985317218 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9e20053a70 0x7f9e20055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:44.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.494+0000 7f9e2e7fc700 1 -- 172.21.15.60:0/3985317218 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9e3006bea0 con 0x7f9e3814af80 2022-06-29T16:05:44.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.494+0000 7f9e3d709700 1 --2- 172.21.15.60:0/3985317218 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9e20053a70 0x7f9e20055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:44.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.494+0000 7f9e3d709700 1 --2- 172.21.15.60:0/3985317218 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9e20053a70 0x7f9e20055f30 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f9e34018000 tx=0x7f9e34014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:44.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.498+0000 7f9e2e7fc700 1 -- 172.21.15.60:0/3985317218 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9e30042cb0 con 0x7f9e3814af80 2022-06-29T16:05:44.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.622+0000 7f9e3e70b700 1 -- 172.21.15.60:0/3985317218 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9e38065d30 con 0x7f9e3814af80 2022-06-29T16:05:44.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.622+0000 7f9e2e7fc700 1 -- 172.21.15.60:0/3985317218 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+10 (secure 0 0 0) 0x7f9e300462d0 con 0x7f9e3814af80 2022-06-29T16:05:44.631 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:05:44.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.626+0000 7f9e3e70b700 1 -- 172.21.15.60:0/3985317218 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9e20053a70 msgr2=0x7f9e20055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:44.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.626+0000 7f9e3e70b700 1 --2- 172.21.15.60:0/3985317218 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9e20053a70 0x7f9e20055f30 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f9e34018000 tx=0x7f9e34014000).stop 2022-06-29T16:05:44.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.626+0000 7f9e3e70b700 1 -- 172.21.15.60:0/3985317218 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e3814af80 msgr2=0x7f9e381421d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:44.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.626+0000 7f9e3e70b700 1 --2- 172.21.15.60:0/3985317218 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e3814af80 0x7f9e381421d0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 rx=0x7f9e380b5cd0 tx=0x7f9e30007f70).stop 2022-06-29T16:05:44.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.626+0000 7f9e3e70b700 1 -- 172.21.15.60:0/3985317218 shutdown_connections 2022-06-29T16:05:44.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.626+0000 7f9e3e70b700 1 --2- 172.21.15.60:0/3985317218 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9e20053a70 0x7f9e20055f30 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:44.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.626+0000 7f9e3e70b700 1 --2- 172.21.15.60:0/3985317218 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e3814af80 0x7f9e381421d0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:44.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.626+0000 7f9e3e70b700 1 --2- 172.21.15.60:0/3985317218 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e380b2130 0x7f9e38119290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:44.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.626+0000 7f9e3e70b700 1 --2- 172.21.15.60:0/3985317218 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e3805d070 0x7f9e38116ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:44.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.626+0000 7f9e3e70b700 1 -- 172.21.15.60:0/3985317218 >> 172.21.15.60:0/3985317218 conn(0x7f9e380b5890 msgr2=0x7f9e38145420 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:44.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.626+0000 7f9e3e70b700 1 -- 172.21.15.60:0/3985317218 shutdown_connections 2022-06-29T16:05:44.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.626+0000 7f9e3e70b700 1 -- 172.21.15.60:0/3985317218 wait complete. 2022-06-29T16:05:44.644 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-29T16:05:44.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.746+0000 7fe34d0cd700 1 Processor -- start 2022-06-29T16:05:44.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.746+0000 7fe34d0cd700 1 -- start start 2022-06-29T16:05:44.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.746+0000 7fe34d0cd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe3481426b0 0x7fe3480b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:44.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.746+0000 7fe34d0cd700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe348143a40 con 0x7fe3480ae7c0 2022-06-29T16:05:44.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.746+0000 7fe34d0cd700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3480aee60 con 0x7fe3480b4010 2022-06-29T16:05:44.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.746+0000 7fe34d0cd700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3480af100 con 0x7fe3481426b0 2022-06-29T16:05:44.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.746+0000 7fe346d9d700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe3480ae7c0 0x7fe348141fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58968/0 (socket says 172.21.15.60:58968) 2022-06-29T16:05:44.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.746+0000 7fe346d9d700 1 -- 172.21.15.60:0/1564522715 learned_addr learned my addr 172.21.15.60:0/1564522715 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:44.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.746+0000 7fe34759e700 1 --2- 172.21.15.60:0/1564522715 >> v2:172.21.15.60:3300/0 conn(0x7fe3481426b0 0x7fe3480b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:44.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.746+0000 7fe345d9b700 1 -- 172.21.15.60:0/1564522715 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1280710929 0 0) 0x7fe348143a40 con 0x7fe3480ae7c0 2022-06-29T16:05:44.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.746+0000 7fe345d9b700 1 -- 172.21.15.60:0/1564522715 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe32c003260 con 0x7fe3480ae7c0 2022-06-29T16:05:44.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.750+0000 7fe345d9b700 1 -- 172.21.15.60:0/1564522715 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 785904981 0 0) 0x7fe3480aee60 con 0x7fe3480b4010 2022-06-29T16:05:44.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.750+0000 7fe345d9b700 1 -- 172.21.15.60:0/1564522715 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe348143a40 con 0x7fe3480b4010 2022-06-29T16:05:44.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.750+0000 7fe34759e700 1 -- 172.21.15.60:0/1564522715 >> v1:172.21.15.92:6789/0 conn(0x7fe3480b4010 legacy=0x7fe3480b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:44.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.750+0000 7fe345d9b700 1 -- 172.21.15.60:0/1564522715 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1759619691 0 0) 0x7fe32c003260 con 0x7fe3480ae7c0 2022-06-29T16:05:44.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.750+0000 7fe34759e700 1 -- 172.21.15.60:0/1564522715 >> v1:172.21.15.60:6789/0 conn(0x7fe3480ae7c0 legacy=0x7fe348141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:44.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.750+0000 7fe34759e700 1 -- 172.21.15.60:0/1564522715 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3480aeba0 con 0x7fe3481426b0 2022-06-29T16:05:44.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.750+0000 7fe34759e700 1 --2- 172.21.15.60:0/1564522715 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3481426b0 0x7fe3480b4dd0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 rx=0x7fe340001490 tx=0x7fe338006df0).ready entity=mon.0 client_cookie=9a7d7974dd6e64d5 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:44.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.750+0000 7fe345d9b700 1 -- 172.21.15.60:0/1564522715 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3400019e0 con 0x7fe3481426b0 2022-06-29T16:05:44.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.750+0000 7fe345d9b700 1 -- 172.21.15.60:0/1564522715 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe338000e90 con 0x7fe3481426b0 2022-06-29T16:05:44.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.750+0000 7fe345d9b700 1 -- 172.21.15.60:0/1564522715 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe338012460 con 0x7fe3481426b0 2022-06-29T16:05:44.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.750+0000 7fe34d0cd700 1 -- 172.21.15.60:0/1564522715 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3481426b0 msgr2=0x7fe3480b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:44.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.750+0000 7fe34d0cd700 1 --2- 172.21.15.60:0/1564522715 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3481426b0 0x7fe3480b4dd0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 rx=0x7fe340001490 tx=0x7fe338006df0).stop 2022-06-29T16:05:44.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.750+0000 7fe34d0cd700 1 -- 172.21.15.60:0/1564522715 shutdown_connections 2022-06-29T16:05:44.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.750+0000 7fe34d0cd700 1 --2- 172.21.15.60:0/1564522715 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3481426b0 0x7fe3480b4dd0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:44.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.750+0000 7fe34d0cd700 1 -- 172.21.15.60:0/1564522715 >> 172.21.15.60:0/1564522715 conn(0x7fe3480b9340 msgr2=0x7fe3480b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:44.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.750+0000 7fe34d0cd700 1 -- 172.21.15.60:0/1564522715 shutdown_connections 2022-06-29T16:05:44.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.750+0000 7fe34d0cd700 1 -- 172.21.15.60:0/1564522715 wait complete. 2022-06-29T16:05:44.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.754+0000 7fe34d0cd700 1 Processor -- start 2022-06-29T16:05:44.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.754+0000 7fe34d0cd700 1 -- start start 2022-06-29T16:05:44.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.754+0000 7fe34d0cd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3480ae7c0 0x7fe348152c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:44.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.754+0000 7fe34d0cd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3480b4010 0x7fe348146ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:44.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.754+0000 7fe34d0cd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3481426b0 0x7fe3480af4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:44.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.754+0000 7fe34d0cd700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3480b1550 con 0x7fe3481426b0 2022-06-29T16:05:44.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.754+0000 7fe34d0cd700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe348151fe0 con 0x7fe3480ae7c0 2022-06-29T16:05:44.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.754+0000 7fe34d0cd700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe348152240 con 0x7fe3480b4010 2022-06-29T16:05:44.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.754+0000 7fe34759e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3481426b0 0x7fe3480af4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:44.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.754+0000 7fe34759e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3481426b0 0x7fe3480af4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50892/0 (socket says 172.21.15.60:50892) 2022-06-29T16:05:44.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.754+0000 7fe34759e700 1 -- 172.21.15.60:0/3372727514 learned_addr learned my addr 172.21.15.60:0/3372727514 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:44.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.754+0000 7fe34659c700 1 --2- 172.21.15.60:0/3372727514 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3480b4010 0x7fe348146ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:44.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.754+0000 7fe346d9d700 1 --2- 172.21.15.60:0/3372727514 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3480ae7c0 0x7fe348152c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:44.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.754+0000 7fe34759e700 1 -- 172.21.15.60:0/3372727514 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3480b4010 msgr2=0x7fe348146ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:44.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.754+0000 7fe34759e700 1 --2- 172.21.15.60:0/3372727514 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3480b4010 0x7fe348146ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:44.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.754+0000 7fe34759e700 1 -- 172.21.15.60:0/3372727514 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3480ae7c0 msgr2=0x7fe348152c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:44.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.754+0000 7fe34759e700 1 --2- 172.21.15.60:0/3372727514 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3480ae7c0 0x7fe348152c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:44.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.754+0000 7fe34759e700 1 -- 172.21.15.60:0/3372727514 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe33800b180 con 0x7fe3481426b0 2022-06-29T16:05:44.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.754+0000 7fe34659c700 1 --2- 172.21.15.60:0/3372727514 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3480b4010 0x7fe348146ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:44.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.754+0000 7fe34759e700 1 --2- 172.21.15.60:0/3372727514 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3481426b0 0x7fe3480af4e0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 rx=0x7fe340001490 tx=0x7fe338014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:44.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.754+0000 7fe337fff700 1 -- 172.21.15.60:0/3372727514 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe338016070 con 0x7fe3481426b0 2022-06-29T16:05:44.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.754+0000 7fe337fff700 1 -- 172.21.15.60:0/3372727514 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe33800bec0 con 0x7fe3481426b0 2022-06-29T16:05:44.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.754+0000 7fe34d0cd700 1 -- 172.21.15.60:0/3372727514 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3481524e0 con 0x7fe3481426b0 2022-06-29T16:05:44.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.754+0000 7fe337fff700 1 -- 172.21.15.60:0/3372727514 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe33800de60 con 0x7fe3481426b0 2022-06-29T16:05:44.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.754+0000 7fe34d0cd700 1 -- 172.21.15.60:0/3372727514 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe348146fa0 con 0x7fe3481426b0 2022-06-29T16:05:44.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.758+0000 7fe34d0cd700 1 -- 172.21.15.60:0/3372727514 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3480573d0 con 0x7fe3481426b0 2022-06-29T16:05:44.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.758+0000 7fe337fff700 1 -- 172.21.15.60:0/3372727514 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe33800f030 con 0x7fe3481426b0 2022-06-29T16:05:44.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.758+0000 7fe337fff700 1 --2- 172.21.15.60:0/3372727514 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe324053aa0 0x7fe324055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:44.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.758+0000 7fe337fff700 1 -- 172.21.15.60:0/3372727514 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe33806cd30 con 0x7fe3481426b0 2022-06-29T16:05:44.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.758+0000 7fe346d9d700 1 --2- 172.21.15.60:0/3372727514 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe324053aa0 0x7fe324055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:44.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.758+0000 7fe346d9d700 1 --2- 172.21.15.60:0/3372727514 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe324053aa0 0x7fe324055f60 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7fe34000b000 tx=0x7fe340000f40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:44.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.766+0000 7fe337fff700 1 -- 172.21.15.60:0/3372727514 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe338043d00 con 0x7fe3481426b0 2022-06-29T16:05:44.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.894+0000 7fe34d0cd700 1 -- 172.21.15.60:0/3372727514 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe34805d160 con 0x7fe3481426b0 2022-06-29T16:05:44.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.894+0000 7fe337fff700 1 -- 172.21.15.60:0/3372727514 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+10 (secure 0 0 0) 0x7fe338047320 con 0x7fe3481426b0 2022-06-29T16:05:44.902 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:05:44.904 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.894+0000 7fe34d0cd700 1 -- 172.21.15.60:0/3372727514 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe324053aa0 msgr2=0x7fe324055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:44.904 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.894+0000 7fe34d0cd700 1 --2- 172.21.15.60:0/3372727514 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe324053aa0 0x7fe324055f60 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7fe34000b000 tx=0x7fe340000f40).stop 2022-06-29T16:05:44.904 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.894+0000 7fe34d0cd700 1 -- 172.21.15.60:0/3372727514 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3481426b0 msgr2=0x7fe3480af4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:44.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.894+0000 7fe34d0cd700 1 --2- 172.21.15.60:0/3372727514 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3481426b0 0x7fe3480af4e0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 rx=0x7fe340001490 tx=0x7fe338014000).stop 2022-06-29T16:05:44.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.894+0000 7fe34d0cd700 1 -- 172.21.15.60:0/3372727514 shutdown_connections 2022-06-29T16:05:44.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.894+0000 7fe34d0cd700 1 --2- 172.21.15.60:0/3372727514 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe324053aa0 0x7fe324055f60 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:44.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.894+0000 7fe34d0cd700 1 --2- 172.21.15.60:0/3372727514 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3481426b0 0x7fe3480af4e0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:44.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.894+0000 7fe34d0cd700 1 --2- 172.21.15.60:0/3372727514 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3480b4010 0x7fe348146ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:44.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.894+0000 7fe34d0cd700 1 --2- 172.21.15.60:0/3372727514 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3480ae7c0 0x7fe348152c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:44.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.894+0000 7fe34d0cd700 1 -- 172.21.15.60:0/3372727514 >> 172.21.15.60:0/3372727514 conn(0x7fe3480b9340 msgr2=0x7fe3480b7240 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:44.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.894+0000 7fe34d0cd700 1 -- 172.21.15.60:0/3372727514 shutdown_connections 2022-06-29T16:05:44.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:44.894+0000 7fe34d0cd700 1 -- 172.21.15.60:0/3372727514 wait complete. 2022-06-29T16:05:44.915 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-06-29T16:05:44.916 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:05:45.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.010+0000 7f04f490b700 1 Processor -- start 2022-06-29T16:05:45.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.010+0000 7f04f490b700 1 -- start start 2022-06-29T16:05:45.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.010+0000 7f04f490b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04f005d070 0x7f04f014aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:45.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.010+0000 7f04f490b700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f005d450 con 0x7f04f00b2130 2022-06-29T16:05:45.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.010+0000 7f04f490b700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f00ad840 con 0x7f04f014af80 2022-06-29T16:05:45.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.010+0000 7f04f490b700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04f00adf90 con 0x7f04f005d070 2022-06-29T16:05:45.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.010+0000 7f04ee59c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f04f00b2130 0x7f04f00aca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:58982/0 (socket says 172.21.15.60:58982) 2022-06-29T16:05:45.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.010+0000 7f04ee59c700 1 -- 172.21.15.60:0/533866167 learned_addr learned my addr 172.21.15.60:0/533866167 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:45.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.010+0000 7f04eed9d700 1 --2- 172.21.15.60:0/533866167 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04f005d070 0x7f04f014aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:45.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.010+0000 7f04edd9b700 1 -- 172.21.15.60:0/533866167 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3458337940 0 0) 0x7f04f005d450 con 0x7f04f00b2130 2022-06-29T16:05:45.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.010+0000 7f04edd9b700 1 -- 172.21.15.60:0/533866167 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04d0003260 con 0x7f04f00b2130 2022-06-29T16:05:45.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.010+0000 7f04edd9b700 1 -- 172.21.15.60:0/533866167 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2566610603 0 0) 0x7f04d0003260 con 0x7f04f00b2130 2022-06-29T16:05:45.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.010+0000 7f04edd9b700 1 -- 172.21.15.60:0/533866167 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04f005d070 msgr2=0x7f04f014aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:45.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.010+0000 7f04edd9b700 1 --2- 172.21.15.60:0/533866167 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04f005d070 0x7f04f014aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:45.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.010+0000 7f04edd9b700 1 -- 172.21.15.60:0/533866167 >> v1:172.21.15.92:6789/0 conn(0x7f04f014af80 legacy=0x7f04f00ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:45.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.010+0000 7f04edd9b700 1 -- 172.21.15.60:0/533866167 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f00b25d0 con 0x7f04f00b2130 2022-06-29T16:05:45.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.010+0000 7f04edd9b700 1 -- 172.21.15.60:0/533866167 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f04d8001a00 con 0x7f04f00b2130 2022-06-29T16:05:45.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.010+0000 7f04edd9b700 1 -- 172.21.15.60:0/533866167 >> v1:172.21.15.60:6789/0 conn(0x7f04f00b2130 legacy=0x7f04f00aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:45.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.010+0000 7f04edd9b700 1 --2- 172.21.15.60:0/533866167 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04f005d070 0x7f04d0003780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:45.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.010+0000 7f04edd9b700 1 --2- 172.21.15.60:0/533866167 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04d0003c70 0x7f04d0004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:45.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.010+0000 7f04edd9b700 1 --2- 172.21.15.60:0/533866167 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04d00049f0 0x7f04d0004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:45.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.010+0000 7f04edd9b700 1 -- 172.21.15.60:0/533866167 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f00adf90 con 0x7f04d0003c70 2022-06-29T16:05:45.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.010+0000 7f04edd9b700 1 -- 172.21.15.60:0/533866167 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d000b7b0 con 0x7f04d00049f0 2022-06-29T16:05:45.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.010+0000 7f04edd9b700 1 -- 172.21.15.60:0/533866167 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04d000b970 con 0x7f04f005d070 2022-06-29T16:05:45.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.010+0000 7f04ee59c700 1 --2- 172.21.15.60:0/533866167 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04d0003c70 0x7f04d0004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:45.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.010+0000 7f04eed9d700 1 --2- 172.21.15.60:0/533866167 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04f005d070 0x7f04d0003780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:45.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.010+0000 7f04ef59e700 1 --2- 172.21.15.60:0/533866167 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04d00049f0 0x7f04d0004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:45.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.014+0000 7f04ee59c700 1 -- 172.21.15.60:0/533866167 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04f005d070 msgr2=0x7f04d0003780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:45.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.014+0000 7f04ee59c700 1 --2- 172.21.15.60:0/533866167 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04f005d070 0x7f04d0003780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:45.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.014+0000 7f04ee59c700 1 -- 172.21.15.60:0/533866167 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04d00049f0 msgr2=0x7f04d0004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:45.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.014+0000 7f04ee59c700 1 --2- 172.21.15.60:0/533866167 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04d00049f0 0x7f04d0004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:45.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.014+0000 7f04ee59c700 1 -- 172.21.15.60:0/533866167 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d0003260 con 0x7f04d0003c70 2022-06-29T16:05:45.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.014+0000 7f04ee59c700 1 --2- 172.21.15.60:0/533866167 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04d0003c70 0x7f04d0004500 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 rx=0x7f04d8001710 tx=0x7f04d8002100).ready entity=mon.0 client_cookie=83d99c7e78015eb server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:45.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.014+0000 7f04edd9b700 1 -- 172.21.15.60:0/533866167 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04d800cb80 con 0x7f04d0003c70 2022-06-29T16:05:45.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.014+0000 7f04edd9b700 1 -- 172.21.15.60:0/533866167 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d800cce0 con 0x7f04d0003c70 2022-06-29T16:05:45.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.014+0000 7f04edd9b700 1 -- 172.21.15.60:0/533866167 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04d8004b50 con 0x7f04d0003c70 2022-06-29T16:05:45.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.014+0000 7f04f490b700 1 -- 172.21.15.60:0/533866167 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04d0003c70 msgr2=0x7f04d0004500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:45.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.014+0000 7f04f490b700 1 --2- 172.21.15.60:0/533866167 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04d0003c70 0x7f04d0004500 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 rx=0x7f04d8001710 tx=0x7f04d8002100).stop 2022-06-29T16:05:45.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.014+0000 7f04ef59e700 1 -- 172.21.15.60:0/533866167 reap_dead start 2022-06-29T16:05:45.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.014+0000 7f04f490b700 1 -- 172.21.15.60:0/533866167 shutdown_connections 2022-06-29T16:05:45.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.014+0000 7f04f490b700 1 -- 172.21.15.60:0/533866167 >> 172.21.15.60:0/533866167 conn(0x7f04f00b5890 msgr2=0x7f04f00b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:45.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.014+0000 7f04f490b700 1 -- 172.21.15.60:0/533866167 shutdown_connections 2022-06-29T16:05:45.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.014+0000 7f04f490b700 1 -- 172.21.15.60:0/533866167 wait complete. 2022-06-29T16:05:45.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.014+0000 7f04f490b700 1 Processor -- start 2022-06-29T16:05:45.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.014+0000 7f04f490b700 1 -- start start 2022-06-29T16:05:45.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.014+0000 7f04f490b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04d0003c70 0x7f04f0118c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:45.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.014+0000 7f04f490b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04f0116ba0 0x7f04f00b4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:45.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.018+0000 7f04f490b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04f0156250 0x7f04f0156960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:45.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.018+0000 7f04f490b700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f00651a0 con 0x7f04d0003c70 2022-06-29T16:05:45.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.018+0000 7f04f490b700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f0065440 con 0x7f04f0156250 2022-06-29T16:05:45.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.018+0000 7f04f490b700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04f00656e0 con 0x7f04f0116ba0 2022-06-29T16:05:45.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.018+0000 7f04eed9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04d0003c70 0x7f04f0118c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:45.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.018+0000 7f04eed9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04d0003c70 0x7f04f0118c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50906/0 (socket says 172.21.15.60:50906) 2022-06-29T16:05:45.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.018+0000 7f04eed9d700 1 -- 172.21.15.60:0/875414296 learned_addr learned my addr 172.21.15.60:0/875414296 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:45.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.018+0000 7f04ef59e700 1 --2- 172.21.15.60:0/875414296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04f0156250 0x7f04f0156960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:45.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.018+0000 7f04ee59c700 1 --2- 172.21.15.60:0/875414296 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04f0116ba0 0x7f04f00b4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:45.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.018+0000 7f04eed9d700 1 -- 172.21.15.60:0/875414296 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04f0116ba0 msgr2=0x7f04f00b4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:45.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.018+0000 7f04eed9d700 1 --2- 172.21.15.60:0/875414296 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04f0116ba0 0x7f04f00b4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:45.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.018+0000 7f04eed9d700 1 -- 172.21.15.60:0/875414296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04f0156250 msgr2=0x7f04f0156960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:45.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.018+0000 7f04eed9d700 1 --2- 172.21.15.60:0/875414296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04f0156250 0x7f04f0156960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:45.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.018+0000 7f04eed9d700 1 -- 172.21.15.60:0/875414296 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f00b4880 con 0x7f04d0003c70 2022-06-29T16:05:45.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.018+0000 7f04ef59e700 1 --2- 172.21.15.60:0/875414296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04f0156250 0x7f04f0156960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:45.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.018+0000 7f04eed9d700 1 --2- 172.21.15.60:0/875414296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04d0003c70 0x7f04f0118c40 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 rx=0x7f04e000b000 tx=0x7f04e0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:45.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.018+0000 7f04e7fff700 1 -- 172.21.15.60:0/875414296 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04e000f8e0 con 0x7f04d0003c70 2022-06-29T16:05:45.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.018+0000 7f04e7fff700 1 -- 172.21.15.60:0/875414296 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04e000fa40 con 0x7f04d0003c70 2022-06-29T16:05:45.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.018+0000 7f04f490b700 1 -- 172.21.15.60:0/875414296 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04f0156e50 con 0x7f04d0003c70 2022-06-29T16:05:45.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.018+0000 7f04e7fff700 1 -- 172.21.15.60:0/875414296 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04e000fda0 con 0x7f04d0003c70 2022-06-29T16:05:45.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.018+0000 7f04f490b700 1 -- 172.21.15.60:0/875414296 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04f0159ba0 con 0x7f04d0003c70 2022-06-29T16:05:45.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.018+0000 7f04f490b700 1 -- 172.21.15.60:0/875414296 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04f014a350 con 0x7f04d0003c70 2022-06-29T16:05:45.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.018+0000 7f04e7fff700 1 -- 172.21.15.60:0/875414296 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f04e0014440 con 0x7f04d0003c70 2022-06-29T16:05:45.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.018+0000 7f04e7fff700 1 --2- 172.21.15.60:0/875414296 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f04d4053a70 0x7f04d4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:45.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.022+0000 7f04e7fff700 1 -- 172.21.15.60:0/875414296 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f04e006caa0 con 0x7f04d0003c70 2022-06-29T16:05:45.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.022+0000 7f04ee59c700 1 --2- 172.21.15.60:0/875414296 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f04d4053a70 0x7f04d4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:45.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.022+0000 7f04ee59c700 1 --2- 172.21.15.60:0/875414296 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f04d4053a70 0x7f04d4055f30 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7f04d801a000 tx=0x7f04d8015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:45.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.026+0000 7f04e7fff700 1 -- 172.21.15.60:0/875414296 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f04e0043850 con 0x7f04d0003c70 2022-06-29T16:05:45.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.190+0000 7f04f490b700 1 -- 172.21.15.60:0/875414296 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04f00573d0 con 0x7f04d0003c70 2022-06-29T16:05:45.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.190+0000 7f04e7fff700 1 -- 172.21.15.60:0/875414296 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f04e0018070 con 0x7f04d0003c70 2022-06-29T16:05:45.200 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:45.200 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":41,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6182,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1306898602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1306898602},{"type":"v1","addr":"172.21.15.60:6833","nonce":1306898602}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":39}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:05:40.369985+0000","modified":"2022-06-29T16:05:41.373815+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6185":{"gid":6185,"name":"b","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/2057420852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2057420852},{"type":"v1","addr":"172.21.15.92:6835","nonce":2057420852}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-06-29T16:05:45.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.194+0000 7f04f490b700 1 -- 172.21.15.60:0/875414296 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f04d4053a70 msgr2=0x7f04d4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:45.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.194+0000 7f04f490b700 1 --2- 172.21.15.60:0/875414296 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f04d4053a70 0x7f04d4055f30 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7f04d801a000 tx=0x7f04d8015000).stop 2022-06-29T16:05:45.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.194+0000 7f04f490b700 1 -- 172.21.15.60:0/875414296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04d0003c70 msgr2=0x7f04f0118c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:45.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.194+0000 7f04f490b700 1 --2- 172.21.15.60:0/875414296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04d0003c70 0x7f04f0118c40 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 rx=0x7f04e000b000 tx=0x7f04e0006df0).stop 2022-06-29T16:05:45.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.194+0000 7f04f490b700 1 -- 172.21.15.60:0/875414296 shutdown_connections 2022-06-29T16:05:45.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.194+0000 7f04f490b700 1 --2- 172.21.15.60:0/875414296 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f04d4053a70 0x7f04d4055f30 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:45.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.194+0000 7f04f490b700 1 --2- 172.21.15.60:0/875414296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04f0156250 0x7f04f0156960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:45.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.194+0000 7f04f490b700 1 --2- 172.21.15.60:0/875414296 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04f0116ba0 0x7f04f00b4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:45.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.194+0000 7f04f490b700 1 --2- 172.21.15.60:0/875414296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04d0003c70 0x7f04f0118c40 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:45.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.194+0000 7f04f490b700 1 -- 172.21.15.60:0/875414296 >> 172.21.15.60:0/875414296 conn(0x7f04f00b5890 msgr2=0x7f04f0147180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:45.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.194+0000 7f04f490b700 1 -- 172.21.15.60:0/875414296 shutdown_connections 2022-06-29T16:05:45.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:45.194+0000 7f04f490b700 1 -- 172.21.15.60:0/875414296 wait complete. 2022-06-29T16:05:45.205 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 41 2022-06-29T16:05:45.414 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 41, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:05:40.369985+0000', 'modified': '2022-06-29T16:05:41.373815+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6185}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6185': {'gid': 6185, 'name': 'b', 'rank': 0, 'incarnation': 40, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.92:6835/2057420852', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 2057420852}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 2057420852}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:05:45.415 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:05:45.415 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-06-29T16:05:45.531 INFO:teuthology.orchestra.run.smithi092.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":6185,"want_state":"up:active","state":"up:active","rank_uptime":5.1446985390000002,"mdsmap_epoch":41,"osdmap_epoch":91,"osdmap_epoch_barrier":90,"uptime":9.8685962939999996} 2022-06-29T16:05:45.532 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 6185, "mdsmap_epoch": 41, "osdmap_epoch": 91, "osdmap_epoch_barrier": 90, "rank_uptime": 5.144698539, "state": "up:active", "uptime": 9.868596294, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:05:45.532 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-06-29T16:05:45.533 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-06-29T16:05:45.533 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi121.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-06-29T16:05:45.533 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:45.534 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:05:45.542 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:05:45.559 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:05:45.559 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:05:45.570 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:05:45.571 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:05:45.572 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:45.573 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:05:45.621 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-06-29T16:05:45.622 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:05:45.624 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:05:45.641 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:05:45.641 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:05:45.651 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:05:45.652 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:05:45.653 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:45.654 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:05:45.697 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:2022-06-29T16:05:45.689+0000 7fd75858a080 -1 init, newargv = 0x555663af3f50 newargc=9 2022-06-29T16:05:45.698 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[13133]: starting ceph client 2022-06-29T16:05:45.706 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[13133]: starting fuse 2022-06-29T16:05:46.803 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:05:46.822 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:05:46.822 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:05:46.878 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:05:46.879 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:05:46.880 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:46.880 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:05:46.885 INFO:teuthology.orchestra.run.smithi121.stdout:52 2022-06-29T16:05:46.886 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-06-29T16:05:46.886 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:46.887 DEBUG:teuthology.orchestra.run.smithi121:> sudo python3 -c ' 2022-06-29T16:05:46.887 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2022-06-29T16:05:46.887 DEBUG:teuthology.orchestra.run.smithi121:> import re 2022-06-29T16:05:46.888 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-06-29T16:05:46.888 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2022-06-29T16:05:46.889 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:05:46.889 DEBUG:teuthology.orchestra.run.smithi121:> def find_socket(client_name): 2022-06-29T16:05:46.890 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-06-29T16:05:46.890 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2022-06-29T16:05:46.891 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:05:46.891 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2022-06-29T16:05:46.892 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2022-06-29T16:05:46.892 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2022-06-29T16:05:46.892 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2022-06-29T16:05:46.893 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:05:46.893 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2022-06-29T16:05:46.894 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-06-29T16:05:46.894 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2022-06-29T16:05:46.894 DEBUG:teuthology.orchestra.run.smithi121:> return f 2022-06-29T16:05:46.895 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-06-29T16:05:46.895 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:05:46.896 DEBUG:teuthology.orchestra.run.smithi121:> print(find_socket("client.0")) 2022-06-29T16:05:46.896 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-06-29T16:05:46.936 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.0.13133.asok 2022-06-29T16:05:46.941 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.13133.asok 2022-06-29T16:05:46.942 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:46.942 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.13133.asok status 2022-06-29T16:05:47.030 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2022-06-29T16:05:47.031 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2022-06-29T16:05:47.031 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "f25ffac72e5a9e63860ff7deb6767a11f0559a16", 2022-06-29T16:05:47.031 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 15.2.16-304-gf25ffac7 (f25ffac72e5a9e63860ff7deb6767a11f0559a16) octopus (stable)", 2022-06-29T16:05:47.031 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "0", 2022-06-29T16:05:47.032 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2022-06-29T16:05:47.032 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-06-29T16:05:47.032 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "13133", 2022-06-29T16:05:47.032 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2022-06-29T16:05:47.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:05:47.033 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2022-06-29T16:05:47.033 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2022-06-29T16:05:47.033 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 7159, 2022-06-29T16:05:47.034 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2022-06-29T16:05:47.034 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2022-06-29T16:05:47.034 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2022-06-29T16:05:47.034 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 7159 2022-06-29T16:05:47.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:05:47.035 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-06-29T16:05:47.035 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-06-29T16:05:47.035 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-06-29T16:05:47.035 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3577677689 2022-06-29T16:05:47.036 INFO:teuthology.orchestra.run.smithi121.stdout: } 2022-06-29T16:05:47.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:05:47.036 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-06-29T16:05:47.036 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-06-29T16:05:47.036 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-06-29T16:05:47.037 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3577677689 2022-06-29T16:05:47.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:05:47.037 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.7159 172.21.15.121:0/3577677689", 2022-06-29T16:05:47.037 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "172.21.15.121:0/3577677689", 2022-06-29T16:05:47.038 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2022-06-29T16:05:47.038 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 41, 2022-06-29T16:05:47.038 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 91, 2022-06-29T16:05:47.038 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2022-06-29T16:05:47.038 INFO:teuthology.orchestra.run.smithi121.stdout: "blacklisted": false, 2022-06-29T16:05:47.039 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2022-06-29T16:05:47.039 INFO:teuthology.orchestra.run.smithi121.stdout:} 2022-06-29T16:05:47.040 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:47.041 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:05:47.041 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:05:47.047 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T16:05:47.048 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T16:05:47.049 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:47.049 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:05:47.066 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:05:47.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.174+0000 7f14d0b0c700 1 Processor -- start 2022-06-29T16:05:47.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.174+0000 7f14d0b0c700 1 -- start start 2022-06-29T16:05:47.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.174+0000 7f14d0b0c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14cc0b5950 0x7f14cc067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:47.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.174+0000 7f14d0b0c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f14cc063ad0 0x7f14cc14af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:47.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.174+0000 7f14d0b0c700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14cc0acba0 con 0x7f14cc05bd20 2022-06-29T16:05:47.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.174+0000 7f14d0b0c700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14cc067220 con 0x7f14cc063ad0 2022-06-29T16:05:47.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.174+0000 7f14d0b0c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14cc0b5d30 con 0x7f14cc0b5950 2022-06-29T16:05:47.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.174+0000 7f14c9d9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f14cc063ad0 0x7f14cc14af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:47.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.174+0000 7f14ca59c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14cc0b5950 0x7f14cc067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:47.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.174+0000 7f14ca59c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14cc0b5950 0x7f14cc067680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:35776/0 (socket says 172.21.15.60:35776) 2022-06-29T16:05:47.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.174+0000 7f14ca59c700 1 -- 172.21.15.60:0/2346167829 learned_addr learned my addr 172.21.15.60:0/2346167829 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:47.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.174+0000 7f14c9d9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f14cc063ad0 0x7f14cc14af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50918/0 (socket says 172.21.15.60:50918) 2022-06-29T16:05:47.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.174+0000 7f14ca59c700 1 -- 172.21.15.60:0/2346167829 >> v2:172.21.15.60:3300/0 conn(0x7f14cc063ad0 msgr2=0x7f14cc14af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:47.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.174+0000 7f14ca59c700 1 --2- 172.21.15.60:0/2346167829 >> v2:172.21.15.60:3300/0 conn(0x7f14cc063ad0 0x7f14cc14af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:47.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.174+0000 7f14ca59c700 1 -- 172.21.15.60:0/2346167829 >> v1:172.21.15.92:6789/0 conn(0x7f14cc05bd20 legacy=0x7f14cc05c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:47.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.174+0000 7f14ca59c700 1 -- 172.21.15.60:0/2346167829 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14cc0ace00 con 0x7f14cc0b5950 2022-06-29T16:05:47.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.174+0000 7f14ca59c700 1 --2- 172.21.15.60:0/2346167829 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14cc0b5950 0x7f14cc067680 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f14bc002f70 tx=0x7f14bc006eb0).ready entity=mon.2 client_cookie=6f21fa5b82f0eb30 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:47.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.174+0000 7f14c959a700 1 -- 172.21.15.60:0/2346167829 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14bc00d570 con 0x7f14cc0b5950 2022-06-29T16:05:47.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.174+0000 7f14c959a700 1 -- 172.21.15.60:0/2346167829 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14bc000e90 con 0x7f14cc0b5950 2022-06-29T16:05:47.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.174+0000 7f14c959a700 1 -- 172.21.15.60:0/2346167829 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14bc00d900 con 0x7f14cc0b5950 2022-06-29T16:05:47.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.174+0000 7f14d0b0c700 1 -- 172.21.15.60:0/2346167829 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14cc0b5950 msgr2=0x7f14cc067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:47.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.174+0000 7f14d0b0c700 1 --2- 172.21.15.60:0/2346167829 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14cc0b5950 0x7f14cc067680 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f14bc002f70 tx=0x7f14bc006eb0).stop 2022-06-29T16:05:47.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.178+0000 7f14d0b0c700 1 -- 172.21.15.60:0/2346167829 shutdown_connections 2022-06-29T16:05:47.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.178+0000 7f14d0b0c700 1 --2- 172.21.15.60:0/2346167829 >> v2:172.21.15.60:3300/0 conn(0x7f14cc063ad0 0x7f14cc14af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:47.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.178+0000 7f14d0b0c700 1 --2- 172.21.15.60:0/2346167829 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14cc0b5950 0x7f14cc067680 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:47.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.178+0000 7f14d0b0c700 1 -- 172.21.15.60:0/2346167829 >> 172.21.15.60:0/2346167829 conn(0x7f14cc0632c0 msgr2=0x7f14cc0b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:47.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.178+0000 7f14d0b0c700 1 -- 172.21.15.60:0/2346167829 shutdown_connections 2022-06-29T16:05:47.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.178+0000 7f14d0b0c700 1 -- 172.21.15.60:0/2346167829 wait complete. 2022-06-29T16:05:47.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.178+0000 7f14d0b0c700 1 Processor -- start 2022-06-29T16:05:47.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.178+0000 7f14d0b0c700 1 -- start start 2022-06-29T16:05:47.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.182+0000 7f14d0b0c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14cc05bd20 0x7f14cc14cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:47.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.182+0000 7f14d0b0c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14cc063ad0 0x7f14cc144030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:47.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.182+0000 7f14d0b0c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14cc0b5950 0x7f14cc144520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:47.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.182+0000 7f14d0b0c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14cc155ea0 con 0x7f14cc063ad0 2022-06-29T16:05:47.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.182+0000 7f14d0b0c700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14cc1560d0 con 0x7f14cc05bd20 2022-06-29T16:05:47.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.182+0000 7f14d0b0c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14cc0af2e0 con 0x7f14cc0b5950 2022-06-29T16:05:47.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.182+0000 7f14c9d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14cc063ad0 0x7f14cc144030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:47.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.182+0000 7f14c9d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14cc063ad0 0x7f14cc144030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50922/0 (socket says 172.21.15.60:50922) 2022-06-29T16:05:47.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.182+0000 7f14c9d9b700 1 -- 172.21.15.60:0/189731591 learned_addr learned my addr 172.21.15.60:0/189731591 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:47.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.182+0000 7f14ca59c700 1 --2- 172.21.15.60:0/189731591 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14cc05bd20 0x7f14cc14cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:47.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.182+0000 7f14cad9d700 1 --2- 172.21.15.60:0/189731591 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14cc0b5950 0x7f14cc144520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:47.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.182+0000 7f14c9d9b700 1 -- 172.21.15.60:0/189731591 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14cc0b5950 msgr2=0x7f14cc144520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:47.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.182+0000 7f14c9d9b700 1 --2- 172.21.15.60:0/189731591 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14cc0b5950 0x7f14cc144520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:47.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.182+0000 7f14c9d9b700 1 -- 172.21.15.60:0/189731591 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14cc05bd20 msgr2=0x7f14cc14cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:47.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.182+0000 7f14c9d9b700 1 --2- 172.21.15.60:0/189731591 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14cc05bd20 0x7f14cc14cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:47.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.182+0000 7f14c9d9b700 1 -- 172.21.15.60:0/189731591 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14bc00b1e0 con 0x7f14cc063ad0 2022-06-29T16:05:47.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.182+0000 7f14c9d9b700 1 --2- 172.21.15.60:0/189731591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14cc063ad0 0x7f14cc144030 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 rx=0x7f14c000c000 tx=0x7f14c0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:47.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.182+0000 7f14bb7fe700 1 -- 172.21.15.60:0/189731591 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14c0010780 con 0x7f14cc063ad0 2022-06-29T16:05:47.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.182+0000 7f14bb7fe700 1 -- 172.21.15.60:0/189731591 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14c00108e0 con 0x7f14cc063ad0 2022-06-29T16:05:47.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.182+0000 7f14d0b0c700 1 -- 172.21.15.60:0/189731591 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14cc157080 con 0x7f14cc063ad0 2022-06-29T16:05:47.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.182+0000 7f14bb7fe700 1 -- 172.21.15.60:0/189731591 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14c0010c40 con 0x7f14cc063ad0 2022-06-29T16:05:47.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.182+0000 7f14d0b0c700 1 -- 172.21.15.60:0/189731591 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14cc14e430 con 0x7f14cc063ad0 2022-06-29T16:05:47.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.182+0000 7f14d0b0c700 1 -- 172.21.15.60:0/189731591 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14cc0af4a0 con 0x7f14cc063ad0 2022-06-29T16:05:47.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.186+0000 7f14bb7fe700 1 -- 172.21.15.60:0/189731591 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f14c001c460 con 0x7f14cc063ad0 2022-06-29T16:05:47.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.186+0000 7f14bb7fe700 1 --2- 172.21.15.60:0/189731591 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f14a80539e0 0x7f14a8055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:47.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.186+0000 7f14bb7fe700 1 -- 172.21.15.60:0/189731591 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f14c006c2a0 con 0x7f14cc063ad0 2022-06-29T16:05:47.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.186+0000 7f14ca59c700 1 --2- 172.21.15.60:0/189731591 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f14a80539e0 0x7f14a8055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:47.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.186+0000 7f14ca59c700 1 --2- 172.21.15.60:0/189731591 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f14a80539e0 0x7f14a8055ea0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f14bc01c000 tx=0x7f14bc015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:47.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.190+0000 7f14bb7fe700 1 -- 172.21.15.60:0/189731591 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f14c0043160 con 0x7f14cc063ad0 2022-06-29T16:05:47.326 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.318+0000 7f14d0b0c700 1 -- 172.21.15.60:0/189731591 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14cc0573d0 con 0x7f14cc063ad0 2022-06-29T16:05:47.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.322+0000 7f14bb7fe700 1 -- 172.21.15.60:0/189731591 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+16041 (secure 0 0 0) 0x7f14c0046780 con 0x7f14cc063ad0 2022-06-29T16:05:47.328 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:47.329 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":91,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:05:41.373483+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:05:38.407939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2022-06-29T16:05:39.380608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:05:47.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.322+0000 7f14d0b0c700 1 -- 172.21.15.60:0/189731591 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f14a80539e0 msgr2=0x7f14a8055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:47.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.322+0000 7f14d0b0c700 1 --2- 172.21.15.60:0/189731591 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f14a80539e0 0x7f14a8055ea0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f14bc01c000 tx=0x7f14bc015000).stop 2022-06-29T16:05:47.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.322+0000 7f14d0b0c700 1 -- 172.21.15.60:0/189731591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14cc063ad0 msgr2=0x7f14cc144030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:47.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.322+0000 7f14d0b0c700 1 --2- 172.21.15.60:0/189731591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14cc063ad0 0x7f14cc144030 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 rx=0x7f14c000c000 tx=0x7f14c0006df0).stop 2022-06-29T16:05:47.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.322+0000 7f14d0b0c700 1 -- 172.21.15.60:0/189731591 shutdown_connections 2022-06-29T16:05:47.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.322+0000 7f14d0b0c700 1 --2- 172.21.15.60:0/189731591 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f14a80539e0 0x7f14a8055ea0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:47.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.322+0000 7f14d0b0c700 1 --2- 172.21.15.60:0/189731591 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14cc0b5950 0x7f14cc144520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:47.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.322+0000 7f14d0b0c700 1 --2- 172.21.15.60:0/189731591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14cc063ad0 0x7f14cc144030 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:47.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.322+0000 7f14d0b0c700 1 --2- 172.21.15.60:0/189731591 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14cc05bd20 0x7f14cc14cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:47.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.322+0000 7f14d0b0c700 1 -- 172.21.15.60:0/189731591 >> 172.21.15.60:0/189731591 conn(0x7f14cc0632c0 msgr2=0x7f14cc1460e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:47.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.322+0000 7f14d0b0c700 1 -- 172.21.15.60:0/189731591 shutdown_connections 2022-06-29T16:05:47.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.322+0000 7f14d0b0c700 1 -- 172.21.15.60:0/189731591 wait complete. 2022-06-29T16:05:47.341 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-29T16:05:38.407939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2022-06-29T16:05:39.380608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-29T16:05:47.342 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:05:47.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.442+0000 7efce6743700 1 Processor -- start 2022-06-29T16:05:47.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.442+0000 7efce6743700 1 -- start start 2022-06-29T16:05:47.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.446+0000 7efce6743700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7efce00b9d70 0x7efce01465d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:47.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.446+0000 7efce6743700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efce0061d90 0x7efce0142b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:47.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.446+0000 7efce6743700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efce00b4f50 con 0x7efce00b3d30 2022-06-29T16:05:47.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.446+0000 7efce6743700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efce00ba150 con 0x7efce0061d90 2022-06-29T16:05:47.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.446+0000 7efce6743700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efce00b5180 con 0x7efce00b9d70 2022-06-29T16:05:47.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.446+0000 7efce4ce0700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efce0061d90 0x7efce0142b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:47.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.446+0000 7efce4ce0700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efce0061d90 0x7efce0142b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50932/0 (socket says 172.21.15.60:50932) 2022-06-29T16:05:47.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.446+0000 7efce4ce0700 1 -- 172.21.15.60:0/429647158 learned_addr learned my addr 172.21.15.60:0/429647158 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:47.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.446+0000 7efcdffff700 1 --2- 172.21.15.60:0/429647158 >> v2:172.21.15.92:3300/0 conn(0x7efce00b9d70 0x7efce01465d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:47.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.446+0000 7efce4ce0700 1 -- 172.21.15.60:0/429647158 >> v2:172.21.15.92:3300/0 conn(0x7efce00b9d70 msgr2=0x7efce01465d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:47.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.446+0000 7efce4ce0700 1 --2- 172.21.15.60:0/429647158 >> v2:172.21.15.92:3300/0 conn(0x7efce00b9d70 0x7efce01465d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:47.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.446+0000 7efce4ce0700 1 -- 172.21.15.60:0/429647158 >> v1:172.21.15.92:6789/0 conn(0x7efce00b3d30 legacy=0x7efce0146ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:47.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.446+0000 7efce4ce0700 1 -- 172.21.15.60:0/429647158 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce01461a0 con 0x7efce0061d90 2022-06-29T16:05:47.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.446+0000 7efce4ce0700 1 --2- 172.21.15.60:0/429647158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efce0061d90 0x7efce0142b70 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 rx=0x7efcd80032f0 tx=0x7efcd8009290).ready entity=mon.0 client_cookie=548633879b2a98f7 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:47.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.446+0000 7efcdf7fe700 1 -- 172.21.15.60:0/429647158 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efcd800b870 con 0x7efce0061d90 2022-06-29T16:05:47.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.446+0000 7efcdf7fe700 1 -- 172.21.15.60:0/429647158 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd800b9d0 con 0x7efce0061d90 2022-06-29T16:05:47.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.446+0000 7efcdf7fe700 1 -- 172.21.15.60:0/429647158 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efcd800bb40 con 0x7efce0061d90 2022-06-29T16:05:47.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.446+0000 7efce6743700 1 -- 172.21.15.60:0/429647158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efce0061d90 msgr2=0x7efce0142b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:47.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.446+0000 7efce6743700 1 --2- 172.21.15.60:0/429647158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efce0061d90 0x7efce0142b70 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 rx=0x7efcd80032f0 tx=0x7efcd8009290).stop 2022-06-29T16:05:47.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.446+0000 7efce6743700 1 -- 172.21.15.60:0/429647158 shutdown_connections 2022-06-29T16:05:47.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.446+0000 7efce6743700 1 --2- 172.21.15.60:0/429647158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efce0061d90 0x7efce0142b70 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:47.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.446+0000 7efce6743700 1 --2- 172.21.15.60:0/429647158 >> v2:172.21.15.92:3300/0 conn(0x7efce00b9d70 0x7efce01465d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:47.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.446+0000 7efce6743700 1 -- 172.21.15.60:0/429647158 >> 172.21.15.60:0/429647158 conn(0x7efce0062e70 msgr2=0x7efce00b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:47.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.446+0000 7efce6743700 1 -- 172.21.15.60:0/429647158 shutdown_connections 2022-06-29T16:05:47.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.446+0000 7efce6743700 1 -- 172.21.15.60:0/429647158 wait complete. 2022-06-29T16:05:47.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.450+0000 7efce6743700 1 Processor -- start 2022-06-29T16:05:47.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.450+0000 7efce6743700 1 -- start start 2022-06-29T16:05:47.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.450+0000 7efce6743700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efce0061d90 0x7efce0112ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:47.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.450+0000 7efce6743700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efce00b3d30 0x7efce0112fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:47.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.450+0000 7efce6743700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efce00b9d70 0x7efce013fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:47.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.450+0000 7efce6743700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce0149860 con 0x7efce0061d90 2022-06-29T16:05:47.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.450+0000 7efce6743700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce00afbd0 con 0x7efce00b9d70 2022-06-29T16:05:47.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.450+0000 7efce6743700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efce0155e10 con 0x7efce00b3d30 2022-06-29T16:05:47.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.450+0000 7efcdffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efce0061d90 0x7efce0112ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:47.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.450+0000 7efcdffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efce0061d90 0x7efce0112ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50934/0 (socket says 172.21.15.60:50934) 2022-06-29T16:05:47.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.450+0000 7efcdffff700 1 -- 172.21.15.60:0/2064876602 learned_addr learned my addr 172.21.15.60:0/2064876602 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:47.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.450+0000 7efcd77fe700 1 --2- 172.21.15.60:0/2064876602 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efce00b3d30 0x7efce0112fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:47.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.450+0000 7efce4ce0700 1 --2- 172.21.15.60:0/2064876602 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efce00b9d70 0x7efce013fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:47.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.450+0000 7efcdffff700 1 -- 172.21.15.60:0/2064876602 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efce00b3d30 msgr2=0x7efce0112fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:47.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.450+0000 7efcdffff700 1 --2- 172.21.15.60:0/2064876602 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efce00b3d30 0x7efce0112fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:47.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.450+0000 7efcdffff700 1 -- 172.21.15.60:0/2064876602 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efce00b9d70 msgr2=0x7efce013fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:47.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.450+0000 7efcdffff700 1 --2- 172.21.15.60:0/2064876602 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efce00b9d70 0x7efce013fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:47.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.450+0000 7efcdffff700 1 -- 172.21.15.60:0/2064876602 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd8009530 con 0x7efce0061d90 2022-06-29T16:05:47.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.450+0000 7efcdffff700 1 --2- 172.21.15.60:0/2064876602 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efce0061d90 0x7efce0112ac0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 rx=0x7efce00afd10 tx=0x7efcd0008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:47.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.450+0000 7efcddffb700 1 -- 172.21.15.60:0/2064876602 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efcd0011780 con 0x7efce0061d90 2022-06-29T16:05:47.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.450+0000 7efce6743700 1 -- 172.21.15.60:0/2064876602 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efce0156090 con 0x7efce0061d90 2022-06-29T16:05:47.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.450+0000 7efcddffb700 1 -- 172.21.15.60:0/2064876602 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd00118e0 con 0x7efce0061d90 2022-06-29T16:05:47.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.450+0000 7efce6743700 1 -- 172.21.15.60:0/2064876602 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efce0147350 con 0x7efce0061d90 2022-06-29T16:05:47.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.450+0000 7efcddffb700 1 -- 172.21.15.60:0/2064876602 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efcd0011c40 con 0x7efce0061d90 2022-06-29T16:05:47.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.454+0000 7efce6743700 1 -- 172.21.15.60:0/2064876602 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efce00573d0 con 0x7efce0061d90 2022-06-29T16:05:47.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.454+0000 7efcddffb700 1 -- 172.21.15.60:0/2064876602 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7efcd0012420 con 0x7efce0061d90 2022-06-29T16:05:47.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.454+0000 7efcddffb700 1 --2- 172.21.15.60:0/2064876602 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efcc40539e0 0x7efcc4055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:47.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.454+0000 7efcddffb700 1 -- 172.21.15.60:0/2064876602 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7efcd006bc70 con 0x7efce0061d90 2022-06-29T16:05:47.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.454+0000 7efcd77fe700 1 --2- 172.21.15.60:0/2064876602 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efcc40539e0 0x7efcc4055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:47.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.454+0000 7efcd77fe700 1 --2- 172.21.15.60:0/2064876602 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efcc40539e0 0x7efcc4055ea0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7efcc800d000 tx=0x7efcc8000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:47.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.462+0000 7efcddffb700 1 -- 172.21.15.60:0/2064876602 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efcd0042a90 con 0x7efce0061d90 2022-06-29T16:05:47.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.582+0000 7efce6743700 1 -- 172.21.15.60:0/2064876602 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7efce005da40 con 0x7efce0061d90 2022-06-29T16:05:47.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.582+0000 7efcddffb700 1 -- 172.21.15.60:0/2064876602 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v91) v1 ==== 110+0+10 (secure 0 0 0) 0x7efcd00460b0 con 0x7efce0061d90 2022-06-29T16:05:47.591 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:05:47.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.586+0000 7efce6743700 1 -- 172.21.15.60:0/2064876602 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efcc40539e0 msgr2=0x7efcc4055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:47.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.586+0000 7efce6743700 1 --2- 172.21.15.60:0/2064876602 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efcc40539e0 0x7efcc4055ea0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7efcc800d000 tx=0x7efcc8000bc0).stop 2022-06-29T16:05:47.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.586+0000 7efce6743700 1 -- 172.21.15.60:0/2064876602 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efce0061d90 msgr2=0x7efce0112ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:47.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.586+0000 7efce6743700 1 --2- 172.21.15.60:0/2064876602 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efce0061d90 0x7efce0112ac0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 rx=0x7efce00afd10 tx=0x7efcd0008040).stop 2022-06-29T16:05:47.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.586+0000 7efce6743700 1 -- 172.21.15.60:0/2064876602 shutdown_connections 2022-06-29T16:05:47.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.586+0000 7efce6743700 1 --2- 172.21.15.60:0/2064876602 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efcc40539e0 0x7efcc4055ea0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:47.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.586+0000 7efce6743700 1 --2- 172.21.15.60:0/2064876602 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efce00b9d70 0x7efce013fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:47.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.586+0000 7efce6743700 1 --2- 172.21.15.60:0/2064876602 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efce00b3d30 0x7efce0112fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:47.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.586+0000 7efce6743700 1 --2- 172.21.15.60:0/2064876602 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efce0061d90 0x7efce0112ac0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:47.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.586+0000 7efce6743700 1 -- 172.21.15.60:0/2064876602 >> 172.21.15.60:0/2064876602 conn(0x7efce0062e70 msgr2=0x7efce0142940 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:47.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.586+0000 7efce6743700 1 -- 172.21.15.60:0/2064876602 shutdown_connections 2022-06-29T16:05:47.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.586+0000 7efce6743700 1 -- 172.21.15.60:0/2064876602 wait complete. 2022-06-29T16:05:47.604 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:05:47.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.702+0000 7f1905159700 1 Processor -- start 2022-06-29T16:05:47.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.702+0000 7f1905159700 1 -- start start 2022-06-29T16:05:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.702+0000 7f1905159700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f19000ae7c0 0x7f1900141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.702+0000 7f1905159700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f19000b4010 0x7f1900142490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:47.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.702+0000 7f1905159700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1900143a40 con 0x7f1900146160 2022-06-29T16:05:47.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.702+0000 7f1905159700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19000b56d0 con 0x7f19000b4010 2022-06-29T16:05:47.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.702+0000 7f1905159700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19000aee60 con 0x7f19000ae7c0 2022-06-29T16:05:47.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.702+0000 7f18fe59c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f19000b4010 0x7f1900142490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:47.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.702+0000 7f18fe59c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f19000b4010 0x7f1900142490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50942/0 (socket says 172.21.15.60:50942) 2022-06-29T16:05:47.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.702+0000 7f18fe59c700 1 -- 172.21.15.60:0/3838633292 learned_addr learned my addr 172.21.15.60:0/3838633292 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:47.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.706+0000 7f18fed9d700 1 --2- 172.21.15.60:0/3838633292 >> v2:172.21.15.92:3300/0 conn(0x7f19000ae7c0 0x7f1900141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:47.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.706+0000 7f18fe59c700 1 -- 172.21.15.60:0/3838633292 >> v2:172.21.15.92:3300/0 conn(0x7f19000ae7c0 msgr2=0x7f1900141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:47.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.706+0000 7f18fe59c700 1 --2- 172.21.15.60:0/3838633292 >> v2:172.21.15.92:3300/0 conn(0x7f19000ae7c0 0x7f1900141fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:47.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.706+0000 7f18fe59c700 1 -- 172.21.15.60:0/3838633292 >> v1:172.21.15.92:6789/0 conn(0x7f1900146160 legacy=0x7f1900146590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:47.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.706+0000 7f18fe59c700 1 -- 172.21.15.60:0/3838633292 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19000af100 con 0x7f19000b4010 2022-06-29T16:05:47.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.706+0000 7f18fe59c700 1 --2- 172.21.15.60:0/3838633292 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19000b4010 0x7f1900142490 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 rx=0x7f18f4002f70 tx=0x7f18f4006eb0).ready entity=mon.0 client_cookie=e6f5e61a17d7908f server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:47.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.706+0000 7f18fdd9b700 1 -- 172.21.15.60:0/3838633292 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f18f400d610 con 0x7f19000b4010 2022-06-29T16:05:47.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.706+0000 7f18fdd9b700 1 -- 172.21.15.60:0/3838633292 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f4000e90 con 0x7f19000b4010 2022-06-29T16:05:47.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.706+0000 7f18fdd9b700 1 -- 172.21.15.60:0/3838633292 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f18f400d9a0 con 0x7f19000b4010 2022-06-29T16:05:47.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.706+0000 7f1905159700 1 -- 172.21.15.60:0/3838633292 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19000b4010 msgr2=0x7f1900142490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:47.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.706+0000 7f1905159700 1 --2- 172.21.15.60:0/3838633292 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19000b4010 0x7f1900142490 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 rx=0x7f18f4002f70 tx=0x7f18f4006eb0).stop 2022-06-29T16:05:47.721 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.706+0000 7f1905159700 1 -- 172.21.15.60:0/3838633292 shutdown_connections 2022-06-29T16:05:47.721 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.706+0000 7f1905159700 1 --2- 172.21.15.60:0/3838633292 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19000b4010 0x7f1900142490 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:47.721 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.706+0000 7f1905159700 1 --2- 172.21.15.60:0/3838633292 >> v2:172.21.15.92:3300/0 conn(0x7f19000ae7c0 0x7f1900141fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:47.721 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.706+0000 7f1905159700 1 -- 172.21.15.60:0/3838633292 >> 172.21.15.60:0/3838633292 conn(0x7f19000b9340 msgr2=0x7f19000b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:47.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.706+0000 7f1905159700 1 -- 172.21.15.60:0/3838633292 shutdown_connections 2022-06-29T16:05:47.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.706+0000 7f1905159700 1 -- 172.21.15.60:0/3838633292 wait complete. 2022-06-29T16:05:47.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.706+0000 7f1905159700 1 Processor -- start 2022-06-29T16:05:47.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.706+0000 7f1905159700 1 -- start start 2022-06-29T16:05:47.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.710+0000 7f1905159700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19000ae7c0 0x7f19001136f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:47.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.710+0000 7f1905159700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19000b4010 0x7f190013f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:47.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.710+0000 7f1905159700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1900146160 0x7f19001554d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:47.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.710+0000 7f1905159700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1900156170 con 0x7f19000b4010 2022-06-29T16:05:47.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.710+0000 7f1905159700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1900155ed0 con 0x7f1900146160 2022-06-29T16:05:47.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.710+0000 7f1905159700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19000b1e70 con 0x7f19000ae7c0 2022-06-29T16:05:47.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.710+0000 7f18fe59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19000b4010 0x7f190013f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:47.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.710+0000 7f18fe59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19000b4010 0x7f190013f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50950/0 (socket says 172.21.15.60:50950) 2022-06-29T16:05:47.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.710+0000 7f18fe59c700 1 -- 172.21.15.60:0/333037779 learned_addr learned my addr 172.21.15.60:0/333037779 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:47.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.710+0000 7f18fed9d700 1 --2- 172.21.15.60:0/333037779 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19000ae7c0 0x7f19001136f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:47.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.710+0000 7f18ff59e700 1 --2- 172.21.15.60:0/333037779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1900146160 0x7f19001554d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:47.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.710+0000 7f18fe59c700 1 -- 172.21.15.60:0/333037779 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19000ae7c0 msgr2=0x7f19001136f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:47.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.710+0000 7f18fe59c700 1 --2- 172.21.15.60:0/333037779 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19000ae7c0 0x7f19001136f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:47.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.710+0000 7f18fe59c700 1 -- 172.21.15.60:0/333037779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1900146160 msgr2=0x7f19001554d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:47.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.710+0000 7f18fe59c700 1 --2- 172.21.15.60:0/333037779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1900146160 0x7f19001554d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:47.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.710+0000 7f18fe59c700 1 -- 172.21.15.60:0/333037779 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f400b1e0 con 0x7f19000b4010 2022-06-29T16:05:47.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.710+0000 7f18ff59e700 1 --2- 172.21.15.60:0/333037779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1900146160 0x7f19001554d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:47.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.710+0000 7f18fed9d700 1 --2- 172.21.15.60:0/333037779 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19000ae7c0 0x7f19001136f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:47.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.710+0000 7f18fe59c700 1 --2- 172.21.15.60:0/333037779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19000b4010 0x7f190013f1d0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 rx=0x7f18f4002f70 tx=0x7f18f4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:47.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.710+0000 7f18effff700 1 -- 172.21.15.60:0/333037779 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f18f40118a0 con 0x7f19000b4010 2022-06-29T16:05:47.728 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.710+0000 7f18effff700 1 -- 172.21.15.60:0/333037779 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f4011a00 con 0x7f19000b4010 2022-06-29T16:05:47.728 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.710+0000 7f1905159700 1 -- 172.21.15.60:0/333037779 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19000b2d30 con 0x7f19000b4010 2022-06-29T16:05:47.728 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.710+0000 7f18effff700 1 -- 172.21.15.60:0/333037779 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f18f4011d60 con 0x7f19000b4010 2022-06-29T16:05:47.728 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.710+0000 7f1905159700 1 -- 172.21.15.60:0/333037779 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19000b3260 con 0x7f19000b4010 2022-06-29T16:05:47.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.710+0000 7f1905159700 1 -- 172.21.15.60:0/333037779 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19000573d0 con 0x7f19000b4010 2022-06-29T16:05:47.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.714+0000 7f18effff700 1 -- 172.21.15.60:0/333037779 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f18f4025030 con 0x7f19000b4010 2022-06-29T16:05:47.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.714+0000 7f18effff700 1 --2- 172.21.15.60:0/333037779 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f18dc0539e0 0x7f18dc055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:47.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.714+0000 7f18effff700 1 -- 172.21.15.60:0/333037779 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f18f406d250 con 0x7f19000b4010 2022-06-29T16:05:47.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.714+0000 7f18fed9d700 1 --2- 172.21.15.60:0/333037779 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f18dc0539e0 0x7f18dc055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:47.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.714+0000 7f18fed9d700 1 --2- 172.21.15.60:0/333037779 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f18dc0539e0 0x7f18dc055ea0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7f18f000c000 tx=0x7f18f0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:47.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.718+0000 7f18effff700 1 -- 172.21.15.60:0/333037779 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f18f4044170 con 0x7f19000b4010 2022-06-29T16:05:47.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.846+0000 7f1905159700 1 -- 172.21.15.60:0/333037779 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f190005fdb0 con 0x7f19000b4010 2022-06-29T16:05:47.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.846+0000 7f18effff700 1 -- 172.21.15.60:0/333037779 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v91) v1 ==== 92+0+10 (secure 0 0 0) 0x7f18f40230a0 con 0x7f19000b4010 2022-06-29T16:05:47.854 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:05:47.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.846+0000 7f1905159700 1 -- 172.21.15.60:0/333037779 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f18dc0539e0 msgr2=0x7f18dc055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:47.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.846+0000 7f1905159700 1 --2- 172.21.15.60:0/333037779 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f18dc0539e0 0x7f18dc055ea0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7f18f000c000 tx=0x7f18f0008040).stop 2022-06-29T16:05:47.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.846+0000 7f1905159700 1 -- 172.21.15.60:0/333037779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19000b4010 msgr2=0x7f190013f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:47.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.846+0000 7f1905159700 1 --2- 172.21.15.60:0/333037779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19000b4010 0x7f190013f1d0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 rx=0x7f18f4002f70 tx=0x7f18f4015000).stop 2022-06-29T16:05:47.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.846+0000 7f1905159700 1 -- 172.21.15.60:0/333037779 shutdown_connections 2022-06-29T16:05:47.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.846+0000 7f1905159700 1 --2- 172.21.15.60:0/333037779 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f18dc0539e0 0x7f18dc055ea0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:47.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.846+0000 7f1905159700 1 --2- 172.21.15.60:0/333037779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1900146160 0x7f19001554d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:47.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.846+0000 7f1905159700 1 --2- 172.21.15.60:0/333037779 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f19000b4010 0x7f190013f1d0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:47.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.846+0000 7f1905159700 1 --2- 172.21.15.60:0/333037779 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f19000ae7c0 0x7f19001136f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:47.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.846+0000 7f1905159700 1 -- 172.21.15.60:0/333037779 >> 172.21.15.60:0/333037779 conn(0x7f19000b9340 msgr2=0x7f19001403a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:47.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.846+0000 7f1905159700 1 -- 172.21.15.60:0/333037779 shutdown_connections 2022-06-29T16:05:47.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.846+0000 7f1905159700 1 -- 172.21.15.60:0/333037779 wait complete. 2022-06-29T16:05:47.867 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-29T16:05:47.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.970+0000 7f750d676700 1 Processor -- start 2022-06-29T16:05:47.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.970+0000 7f750d676700 1 -- start start 2022-06-29T16:05:47.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.974+0000 7f750d676700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75080ae7c0 0x7f7508141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:47.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.974+0000 7f750d676700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f75080b46c0 0x7f75080b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:47.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.974+0000 7f750d676700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7508143a40 con 0x7f75080b4010 2022-06-29T16:05:47.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.974+0000 7f750d676700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75080aee60 con 0x7f75080b46c0 2022-06-29T16:05:47.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.974+0000 7f750d676700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75080af100 con 0x7f75080ae7c0 2022-06-29T16:05:47.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.974+0000 7f75077fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f75080b46c0 0x7f75080b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:47.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.974+0000 7f75077fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f75080b46c0 0x7f75080b4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50958/0 (socket says 172.21.15.60:50958) 2022-06-29T16:05:47.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.974+0000 7f75077fe700 1 -- 172.21.15.60:0/2970536746 learned_addr learned my addr 172.21.15.60:0/2970536746 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:47.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.974+0000 7f7506ffd700 1 --2- 172.21.15.60:0/2970536746 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75080ae7c0 0x7f7508141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:47.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.974+0000 7f75077fe700 1 -- 172.21.15.60:0/2970536746 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75080ae7c0 msgr2=0x7f7508141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:47.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.974+0000 7f75077fe700 1 --2- 172.21.15.60:0/2970536746 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75080ae7c0 0x7f7508141fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:47.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.974+0000 7f75077fe700 1 -- 172.21.15.60:0/2970536746 >> v1:172.21.15.92:6789/0 conn(0x7f75080b4010 legacy=0x7f7508142490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:47.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.974+0000 7f75077fe700 1 -- 172.21.15.60:0/2970536746 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75080aeba0 con 0x7f75080b46c0 2022-06-29T16:05:47.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.974+0000 7f75077fe700 1 --2- 172.21.15.60:0/2970536746 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75080b46c0 0x7f75080b4b50 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 rx=0x7f75000032f0 tx=0x7f7500009290).ready entity=mon.0 client_cookie=7685c22b350824e5 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:47.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.974+0000 7f7505ffb700 1 -- 172.21.15.60:0/2970536746 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f750000b8a0 con 0x7f75080b46c0 2022-06-29T16:05:47.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.974+0000 7f7505ffb700 1 -- 172.21.15.60:0/2970536746 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f750000ba00 con 0x7f75080b46c0 2022-06-29T16:05:47.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.974+0000 7f7505ffb700 1 -- 172.21.15.60:0/2970536746 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f750000bb70 con 0x7f75080b46c0 2022-06-29T16:05:47.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.974+0000 7f750d676700 1 -- 172.21.15.60:0/2970536746 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75080b46c0 msgr2=0x7f75080b4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:47.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.974+0000 7f750d676700 1 --2- 172.21.15.60:0/2970536746 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75080b46c0 0x7f75080b4b50 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 rx=0x7f75000032f0 tx=0x7f7500009290).stop 2022-06-29T16:05:47.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.974+0000 7f750d676700 1 -- 172.21.15.60:0/2970536746 shutdown_connections 2022-06-29T16:05:47.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.974+0000 7f750d676700 1 --2- 172.21.15.60:0/2970536746 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75080b46c0 0x7f75080b4b50 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:47.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.974+0000 7f750d676700 1 --2- 172.21.15.60:0/2970536746 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75080ae7c0 0x7f7508141fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:47.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.974+0000 7f750d676700 1 -- 172.21.15.60:0/2970536746 >> 172.21.15.60:0/2970536746 conn(0x7f75080b9340 msgr2=0x7f75080b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:47.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.974+0000 7f750d676700 1 -- 172.21.15.60:0/2970536746 shutdown_connections 2022-06-29T16:05:47.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.974+0000 7f750d676700 1 -- 172.21.15.60:0/2970536746 wait complete. 2022-06-29T16:05:47.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.978+0000 7f750d676700 1 Processor -- start 2022-06-29T16:05:47.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.978+0000 7f750d676700 1 -- start start 2022-06-29T16:05:47.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.978+0000 7f750d676700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75080ae7c0 0x7f7508114870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:47.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.978+0000 7f750d676700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75080b4010 0x7f7508113e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:47.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.978+0000 7f750d676700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75080b46c0 0x7f75080b2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:47.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.978+0000 7f750d676700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750813e7c0 con 0x7f75080b4010 2022-06-29T16:05:47.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.978+0000 7f750d676700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75080b1890 con 0x7f75080ae7c0 2022-06-29T16:05:47.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.978+0000 7f750d676700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75080b1ac0 con 0x7f75080b46c0 2022-06-29T16:05:47.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.978+0000 7f75067fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75080b4010 0x7f7508113e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:47.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.978+0000 7f75067fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75080b4010 0x7f7508113e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50964/0 (socket says 172.21.15.60:50964) 2022-06-29T16:05:47.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.978+0000 7f75067fc700 1 -- 172.21.15.60:0/1593125854 learned_addr learned my addr 172.21.15.60:0/1593125854 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:47.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.978+0000 7f7506ffd700 1 --2- 172.21.15.60:0/1593125854 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75080ae7c0 0x7f7508114870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:47.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.978+0000 7f75077fe700 1 --2- 172.21.15.60:0/1593125854 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75080b46c0 0x7f75080b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:47.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.978+0000 7f75067fc700 1 -- 172.21.15.60:0/1593125854 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75080b46c0 msgr2=0x7f75080b2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:47.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.978+0000 7f75067fc700 1 --2- 172.21.15.60:0/1593125854 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75080b46c0 0x7f75080b2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:47.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.978+0000 7f75067fc700 1 -- 172.21.15.60:0/1593125854 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75080ae7c0 msgr2=0x7f7508114870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:47.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.978+0000 7f75067fc700 1 --2- 172.21.15.60:0/1593125854 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75080ae7c0 0x7f7508114870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:47.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.978+0000 7f75067fc700 1 -- 172.21.15.60:0/1593125854 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7500009530 con 0x7f75080b4010 2022-06-29T16:05:47.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.982+0000 7f75067fc700 1 --2- 172.21.15.60:0/1593125854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75080b4010 0x7f7508113e50 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 rx=0x7f7508062050 tx=0x7f74f0007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:47.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.982+0000 7f74fffff700 1 -- 172.21.15.60:0/1593125854 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74f0012400 con 0x7f75080b4010 2022-06-29T16:05:47.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.982+0000 7f74fffff700 1 -- 172.21.15.60:0/1593125854 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f0012560 con 0x7f75080b4010 2022-06-29T16:05:47.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.982+0000 7f74fffff700 1 -- 172.21.15.60:0/1593125854 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74f00128c0 con 0x7f75080b4010 2022-06-29T16:05:47.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.982+0000 7f750d676700 1 -- 172.21.15.60:0/1593125854 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7508112a40 con 0x7f75080b4010 2022-06-29T16:05:47.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.982+0000 7f750d676700 1 -- 172.21.15.60:0/1593125854 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75080b1490 con 0x7f75080b4010 2022-06-29T16:05:47.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.982+0000 7f750d676700 1 -- 172.21.15.60:0/1593125854 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75080573d0 con 0x7f75080b4010 2022-06-29T16:05:47.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.982+0000 7f74fffff700 1 -- 172.21.15.60:0/1593125854 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f74f0019030 con 0x7f75080b4010 2022-06-29T16:05:47.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.982+0000 7f74fffff700 1 --2- 172.21.15.60:0/1593125854 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f74ec053b30 0x7f74ec055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:47.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.982+0000 7f74fffff700 1 -- 172.21.15.60:0/1593125854 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f74f006b2b0 con 0x7f75080b4010 2022-06-29T16:05:47.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.982+0000 7f7506ffd700 1 --2- 172.21.15.60:0/1593125854 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f74ec053b30 0x7f74ec055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:47.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.986+0000 7f7506ffd700 1 --2- 172.21.15.60:0/1593125854 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f74ec053b30 0x7f74ec055ff0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7f74f800b000 tx=0x7f74f8006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:47.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:47.990+0000 7f74fffff700 1 -- 172.21.15.60:0/1593125854 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f74f00421d0 con 0x7f75080b4010 2022-06-29T16:05:48.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.118+0000 7f750d676700 1 -- 172.21.15.60:0/1593125854 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7508112bd0 con 0x7f75080b4010 2022-06-29T16:05:48.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.118+0000 7f74fffff700 1 -- 172.21.15.60:0/1593125854 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+10 (secure 0 0 0) 0x7f74f00457f0 con 0x7f75080b4010 2022-06-29T16:05:48.126 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:05:48.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.118+0000 7f750d676700 1 -- 172.21.15.60:0/1593125854 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f74ec053b30 msgr2=0x7f74ec055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:48.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.118+0000 7f750d676700 1 --2- 172.21.15.60:0/1593125854 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f74ec053b30 0x7f74ec055ff0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7f74f800b000 tx=0x7f74f8006df0).stop 2022-06-29T16:05:48.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.118+0000 7f750d676700 1 -- 172.21.15.60:0/1593125854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75080b4010 msgr2=0x7f7508113e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:48.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.118+0000 7f750d676700 1 --2- 172.21.15.60:0/1593125854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75080b4010 0x7f7508113e50 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 rx=0x7f7508062050 tx=0x7f74f0007c50).stop 2022-06-29T16:05:48.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.118+0000 7f750d676700 1 -- 172.21.15.60:0/1593125854 shutdown_connections 2022-06-29T16:05:48.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.118+0000 7f750d676700 1 --2- 172.21.15.60:0/1593125854 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f74ec053b30 0x7f74ec055ff0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:48.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.122+0000 7f750d676700 1 --2- 172.21.15.60:0/1593125854 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75080b46c0 0x7f75080b2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:48.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.122+0000 7f750d676700 1 --2- 172.21.15.60:0/1593125854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75080b4010 0x7f7508113e50 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:48.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.122+0000 7f750d676700 1 --2- 172.21.15.60:0/1593125854 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75080ae7c0 0x7f7508114870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:48.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.122+0000 7f750d676700 1 -- 172.21.15.60:0/1593125854 >> 172.21.15.60:0/1593125854 conn(0x7f75080b9340 msgr2=0x7f75080b4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:48.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.122+0000 7f750d676700 1 -- 172.21.15.60:0/1593125854 shutdown_connections 2022-06-29T16:05:48.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.122+0000 7f750d676700 1 -- 172.21.15.60:0/1593125854 wait complete. 2022-06-29T16:05:48.140 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-29T16:05:48.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.238+0000 7f72110f6700 1 Processor -- start 2022-06-29T16:05:48.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.238+0000 7f72110f6700 1 -- start start 2022-06-29T16:05:48.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.238+0000 7f72110f6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f720c05d070 0x7f720c13c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:48.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.238+0000 7f72110f6700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f720c0b0830 0x7f720c13fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:48.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.238+0000 7f72110f6700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f720c05d450 con 0x7f720c0b2130 2022-06-29T16:05:48.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.238+0000 7f72110f6700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f720c063ba0 con 0x7f720c0b0830 2022-06-29T16:05:48.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.238+0000 7f72110f6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f720c063e40 con 0x7f720c05d070 2022-06-29T16:05:48.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.238+0000 7f72108f5700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f720c0b0830 0x7f720c13fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:48.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.238+0000 7f7203fff700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f720c0b2130 0x7f720c13caa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:35072/0 (socket says 172.21.15.60:35072) 2022-06-29T16:05:48.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.238+0000 7f7203fff700 1 -- 172.21.15.60:0/4093030620 learned_addr learned my addr 172.21.15.60:0/4093030620 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:48.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.238+0000 7f720bfff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f720c05d070 0x7f720c13c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:48.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.242+0000 7f72108f5700 1 -- 172.21.15.60:0/4093030620 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f720c05d070 msgr2=0x7f720c13c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:48.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.242+0000 7f72108f5700 1 --2- 172.21.15.60:0/4093030620 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f720c05d070 0x7f720c13c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:48.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.242+0000 7f72108f5700 1 -- 172.21.15.60:0/4093030620 >> v1:172.21.15.92:6789/0 conn(0x7f720c0b2130 legacy=0x7f720c13caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:48.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.242+0000 7f72108f5700 1 -- 172.21.15.60:0/4093030620 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f720c067650 con 0x7f720c0b0830 2022-06-29T16:05:48.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.242+0000 7f72108f5700 1 --2- 172.21.15.60:0/4093030620 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f720c0b0830 0x7f720c13fa70 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f72040032f0 tx=0x7f72040091d0).ready entity=mon.1 client_cookie=9033943413c1d27d server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:48.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.242+0000 7f720b7fe700 1 -- 172.21.15.60:0/4093030620 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f720400b6c0 con 0x7f720c0b0830 2022-06-29T16:05:48.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.242+0000 7f720b7fe700 1 -- 172.21.15.60:0/4093030620 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f720400b820 con 0x7f720c0b0830 2022-06-29T16:05:48.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.242+0000 7f720b7fe700 1 -- 172.21.15.60:0/4093030620 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f720400b990 con 0x7f720c0b0830 2022-06-29T16:05:48.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.242+0000 7f72110f6700 1 -- 172.21.15.60:0/4093030620 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f720c0b0830 msgr2=0x7f720c13fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:48.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.242+0000 7f72110f6700 1 --2- 172.21.15.60:0/4093030620 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f720c0b0830 0x7f720c13fa70 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f72040032f0 tx=0x7f72040091d0).stop 2022-06-29T16:05:48.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.242+0000 7f72110f6700 1 -- 172.21.15.60:0/4093030620 shutdown_connections 2022-06-29T16:05:48.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.242+0000 7f72110f6700 1 --2- 172.21.15.60:0/4093030620 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f720c0b0830 0x7f720c13fa70 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:48.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.242+0000 7f72110f6700 1 --2- 172.21.15.60:0/4093030620 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f720c05d070 0x7f720c13c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:48.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.242+0000 7f72110f6700 1 -- 172.21.15.60:0/4093030620 >> 172.21.15.60:0/4093030620 conn(0x7f720c0b5890 msgr2=0x7f720c0b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:48.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.242+0000 7f72110f6700 1 -- 172.21.15.60:0/4093030620 shutdown_connections 2022-06-29T16:05:48.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.242+0000 7f72110f6700 1 -- 172.21.15.60:0/4093030620 wait complete. 2022-06-29T16:05:48.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.246+0000 7f72110f6700 1 Processor -- start 2022-06-29T16:05:48.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.246+0000 7f72110f6700 1 -- start start 2022-06-29T16:05:48.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.246+0000 7f72110f6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f720c05d070 0x7f720c0646a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:48.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.246+0000 7f72110f6700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f720c0b0830 0x7f720c065c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:48.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.246+0000 7f72110f6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f720c0b2130 0x7f720c0aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:48.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.246+0000 7f72110f6700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f720c0662c0 con 0x7f720c05d070 2022-06-29T16:05:48.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.246+0000 7f72110f6700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f720c066560 con 0x7f720c0b0830 2022-06-29T16:05:48.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.246+0000 7f72110f6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f720c065220 con 0x7f720c0b2130 2022-06-29T16:05:48.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.246+0000 7f720bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f720c05d070 0x7f720c0646a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:48.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.246+0000 7f7203fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f720c0b0830 0x7f720c065c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:48.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.246+0000 7f720bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f720c05d070 0x7f720c0646a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50974/0 (socket says 172.21.15.60:50974) 2022-06-29T16:05:48.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.246+0000 7f720bfff700 1 -- 172.21.15.60:0/2244457092 learned_addr learned my addr 172.21.15.60:0/2244457092 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:48.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.246+0000 7f72108f5700 1 --2- 172.21.15.60:0/2244457092 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f720c0b2130 0x7f720c0aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:48.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.246+0000 7f720bfff700 1 -- 172.21.15.60:0/2244457092 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f720c0b2130 msgr2=0x7f720c0aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:48.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.246+0000 7f720bfff700 1 --2- 172.21.15.60:0/2244457092 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f720c0b2130 0x7f720c0aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:48.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.246+0000 7f720bfff700 1 -- 172.21.15.60:0/2244457092 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f720c0b0830 msgr2=0x7f720c065c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:48.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.246+0000 7f720bfff700 1 --2- 172.21.15.60:0/2244457092 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f720c0b0830 0x7f720c065c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:48.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.246+0000 7f720bfff700 1 -- 172.21.15.60:0/2244457092 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7204009470 con 0x7f720c05d070 2022-06-29T16:05:48.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.246+0000 7f720bfff700 1 --2- 172.21.15.60:0/2244457092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f720c05d070 0x7f720c0646a0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 rx=0x7f71fc00c000 tx=0x7f71fc006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:48.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.246+0000 7f7209ffb700 1 -- 172.21.15.60:0/2244457092 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71fc010780 con 0x7f720c05d070 2022-06-29T16:05:48.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.246+0000 7f7209ffb700 1 -- 172.21.15.60:0/2244457092 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71fc0108e0 con 0x7f720c05d070 2022-06-29T16:05:48.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.246+0000 7f72110f6700 1 -- 172.21.15.60:0/2244457092 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f720c0654a0 con 0x7f720c05d070 2022-06-29T16:05:48.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.246+0000 7f7209ffb700 1 -- 172.21.15.60:0/2244457092 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71fc010c40 con 0x7f720c05d070 2022-06-29T16:05:48.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.246+0000 7f72110f6700 1 -- 172.21.15.60:0/2244457092 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f720c0ad0b0 con 0x7f720c05d070 2022-06-29T16:05:48.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.250+0000 7f72110f6700 1 -- 172.21.15.60:0/2244457092 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f720c0573d0 con 0x7f720c05d070 2022-06-29T16:05:48.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.250+0000 7f7209ffb700 1 -- 172.21.15.60:0/2244457092 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f71fc01c460 con 0x7f720c05d070 2022-06-29T16:05:48.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.250+0000 7f7209ffb700 1 --2- 172.21.15.60:0/2244457092 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f71f0053aa0 0x7f71f0055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:48.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.250+0000 7f7209ffb700 1 -- 172.21.15.60:0/2244457092 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f71fc06c310 con 0x7f720c05d070 2022-06-29T16:05:48.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.250+0000 7f7203fff700 1 --2- 172.21.15.60:0/2244457092 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f71f0053aa0 0x7f71f0055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:48.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.250+0000 7f7203fff700 1 --2- 172.21.15.60:0/2244457092 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f71f0053aa0 0x7f71f0055f60 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7f71f400d000 tx=0x7f71f4007be0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:48.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.258+0000 7f7209ffb700 1 -- 172.21.15.60:0/2244457092 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f71fc0430c0 con 0x7f720c05d070 2022-06-29T16:05:48.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.386+0000 7f72110f6700 1 -- 172.21.15.60:0/2244457092 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f720c0b4b60 con 0x7f720c05d070 2022-06-29T16:05:48.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.386+0000 7f7209ffb700 1 -- 172.21.15.60:0/2244457092 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+10 (secure 0 0 0) 0x7f71fc0466e0 con 0x7f720c05d070 2022-06-29T16:05:48.393 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:05:48.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.390+0000 7f72110f6700 1 -- 172.21.15.60:0/2244457092 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f71f0053aa0 msgr2=0x7f71f0055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:48.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.390+0000 7f72110f6700 1 --2- 172.21.15.60:0/2244457092 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f71f0053aa0 0x7f71f0055f60 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7f71f400d000 tx=0x7f71f4007be0).stop 2022-06-29T16:05:48.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.390+0000 7f72110f6700 1 -- 172.21.15.60:0/2244457092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f720c05d070 msgr2=0x7f720c0646a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:48.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.390+0000 7f72110f6700 1 --2- 172.21.15.60:0/2244457092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f720c05d070 0x7f720c0646a0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 rx=0x7f71fc00c000 tx=0x7f71fc006df0).stop 2022-06-29T16:05:48.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.390+0000 7f72110f6700 1 -- 172.21.15.60:0/2244457092 shutdown_connections 2022-06-29T16:05:48.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.390+0000 7f72110f6700 1 --2- 172.21.15.60:0/2244457092 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f71f0053aa0 0x7f71f0055f60 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:48.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.390+0000 7f72110f6700 1 --2- 172.21.15.60:0/2244457092 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f720c0b2130 0x7f720c0aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:48.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.390+0000 7f72110f6700 1 --2- 172.21.15.60:0/2244457092 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f720c0b0830 0x7f720c065c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:48.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.390+0000 7f72110f6700 1 --2- 172.21.15.60:0/2244457092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f720c05d070 0x7f720c0646a0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:48.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.390+0000 7f72110f6700 1 -- 172.21.15.60:0/2244457092 >> 172.21.15.60:0/2244457092 conn(0x7f720c0b5890 msgr2=0x7f720c0b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:48.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.390+0000 7f72110f6700 1 -- 172.21.15.60:0/2244457092 shutdown_connections 2022-06-29T16:05:48.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.390+0000 7f72110f6700 1 -- 172.21.15.60:0/2244457092 wait complete. 2022-06-29T16:05:48.413 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-06-29T16:05:48.414 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:05:48.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.510+0000 7f8244872700 1 Processor -- start 2022-06-29T16:05:48.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.510+0000 7f8244872700 1 -- start start 2022-06-29T16:05:48.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.510+0000 7f8244872700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f82400acd60 0x7f8240144900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:48.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.510+0000 7f8244872700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8240143cf0 con 0x7f82400657d0 2022-06-29T16:05:48.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.510+0000 7f8244872700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82400ad140 con 0x7f82400ad990 2022-06-29T16:05:48.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.510+0000 7f8244872700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8240064a70 con 0x7f82400acd60 2022-06-29T16:05:48.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.510+0000 7f823ffff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f82400657d0 0x7f82400b1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59072/0 (socket says 172.21.15.60:59072) 2022-06-29T16:05:48.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.510+0000 7f823ffff700 1 -- 172.21.15.60:0/80502738 learned_addr learned my addr 172.21.15.60:0/80502738 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:48.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.510+0000 7f823f7fe700 1 --2- 172.21.15.60:0/80502738 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f82400acd60 0x7f8240144900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:48.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.510+0000 7f823e7fc700 1 -- 172.21.15.60:0/80502738 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 505946221 0 0) 0x7f8240143cf0 con 0x7f82400657d0 2022-06-29T16:05:48.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.510+0000 7f823e7fc700 1 -- 172.21.15.60:0/80502738 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f822c003260 con 0x7f82400657d0 2022-06-29T16:05:48.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.510+0000 7f823e7fc700 1 -- 172.21.15.60:0/80502738 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 767657311 0 0) 0x7f82400ad140 con 0x7f82400ad990 2022-06-29T16:05:48.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.510+0000 7f823e7fc700 1 -- 172.21.15.60:0/80502738 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8240143cf0 con 0x7f82400ad990 2022-06-29T16:05:48.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.510+0000 7f823f7fe700 1 -- 172.21.15.60:0/80502738 >> v1:172.21.15.92:6789/0 conn(0x7f82400ad990 legacy=0x7f82401296f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:48.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.510+0000 7f823f7fe700 1 -- 172.21.15.60:0/80502738 >> v1:172.21.15.60:6789/0 conn(0x7f82400657d0 legacy=0x7f82400b1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:48.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.510+0000 7f823f7fe700 1 -- 172.21.15.60:0/80502738 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8240143f20 con 0x7f82400acd60 2022-06-29T16:05:48.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.510+0000 7f823f7fe700 1 --2- 172.21.15.60:0/80502738 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f82400acd60 0x7f8240144900 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f82380032f0 tx=0x7f82380091d0).ready entity=mon.2 client_cookie=30ca12a781f5d043 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:48.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.510+0000 7f823e7fc700 1 -- 172.21.15.60:0/80502738 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f823800b640 con 0x7f82400acd60 2022-06-29T16:05:48.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.510+0000 7f823e7fc700 1 -- 172.21.15.60:0/80502738 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f823800b7a0 con 0x7f82400acd60 2022-06-29T16:05:48.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.514+0000 7f823e7fc700 1 -- 172.21.15.60:0/80502738 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f823800b910 con 0x7f82400acd60 2022-06-29T16:05:48.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.514+0000 7f8244872700 1 -- 172.21.15.60:0/80502738 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f82400acd60 msgr2=0x7f8240144900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:48.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.514+0000 7f8244872700 1 --2- 172.21.15.60:0/80502738 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f82400acd60 0x7f8240144900 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f82380032f0 tx=0x7f82380091d0).stop 2022-06-29T16:05:48.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.514+0000 7f8244872700 1 -- 172.21.15.60:0/80502738 shutdown_connections 2022-06-29T16:05:48.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.514+0000 7f8244872700 1 --2- 172.21.15.60:0/80502738 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f82400acd60 0x7f8240144900 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:48.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.514+0000 7f8244872700 1 -- 172.21.15.60:0/80502738 >> 172.21.15.60:0/80502738 conn(0x7f82400b6f80 msgr2=0x7f82400b7390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:48.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.514+0000 7f8244872700 1 -- 172.21.15.60:0/80502738 shutdown_connections 2022-06-29T16:05:48.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.514+0000 7f8244872700 1 -- 172.21.15.60:0/80502738 wait complete. 2022-06-29T16:05:48.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.514+0000 7f8244872700 1 Processor -- start 2022-06-29T16:05:48.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.514+0000 7f8244872700 1 -- start start 2022-06-29T16:05:48.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.514+0000 7f8244872700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82400657d0 0x7f8240126150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:48.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.514+0000 7f8244872700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f82400ad990 0x7f8240142f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:48.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.514+0000 7f8244872700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f82400b0000 0x7f82400b0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:48.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.514+0000 7f8244872700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82400af0c0 con 0x7f82400657d0 2022-06-29T16:05:48.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.514+0000 7f8244872700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8240062450 con 0x7f82400b0000 2022-06-29T16:05:48.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.514+0000 7f8244872700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8240061590 con 0x7f82400ad990 2022-06-29T16:05:48.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.514+0000 7f823f7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82400657d0 0x7f8240126150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:48.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.514+0000 7f823f7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82400657d0 0x7f8240126150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50988/0 (socket says 172.21.15.60:50988) 2022-06-29T16:05:48.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.514+0000 7f823f7fe700 1 -- 172.21.15.60:0/3282357939 learned_addr learned my addr 172.21.15.60:0/3282357939 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:48.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.514+0000 7f823ffff700 1 --2- 172.21.15.60:0/3282357939 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f82400b0000 0x7f82400b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:48.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.514+0000 7f823effd700 1 --2- 172.21.15.60:0/3282357939 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f82400ad990 0x7f8240142f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:48.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.514+0000 7f823f7fe700 1 -- 172.21.15.60:0/3282357939 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f82400ad990 msgr2=0x7f8240142f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:48.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.514+0000 7f823f7fe700 1 --2- 172.21.15.60:0/3282357939 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f82400ad990 0x7f8240142f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:48.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.514+0000 7f823f7fe700 1 -- 172.21.15.60:0/3282357939 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f82400b0000 msgr2=0x7f82400b0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:48.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.514+0000 7f823f7fe700 1 --2- 172.21.15.60:0/3282357939 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f82400b0000 0x7f82400b0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:48.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.514+0000 7f823f7fe700 1 -- 172.21.15.60:0/3282357939 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8238009470 con 0x7f82400657d0 2022-06-29T16:05:48.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.514+0000 7f823ffff700 1 --2- 172.21.15.60:0/3282357939 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f82400b0000 0x7f82400b0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:48.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.518+0000 7f823f7fe700 1 --2- 172.21.15.60:0/3282357939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82400657d0 0x7f8240126150 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 rx=0x7f82380032f0 tx=0x7f8238016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:48.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.518+0000 7f823cff9700 1 -- 172.21.15.60:0/3282357939 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f823800b910 con 0x7f82400657d0 2022-06-29T16:05:48.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.518+0000 7f823cff9700 1 -- 172.21.15.60:0/3282357939 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f823801f3f0 con 0x7f82400657d0 2022-06-29T16:05:48.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.518+0000 7f823cff9700 1 -- 172.21.15.60:0/3282357939 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f823800b640 con 0x7f82400657d0 2022-06-29T16:05:48.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.518+0000 7f8244872700 1 -- 172.21.15.60:0/3282357939 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8240061a30 con 0x7f82400657d0 2022-06-29T16:05:48.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.518+0000 7f8244872700 1 -- 172.21.15.60:0/3282357939 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8240123c10 con 0x7f82400657d0 2022-06-29T16:05:48.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.518+0000 7f8244872700 1 -- 172.21.15.60:0/3282357939 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82400b0cb0 con 0x7f82400657d0 2022-06-29T16:05:48.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.518+0000 7f823cff9700 1 -- 172.21.15.60:0/3282357939 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f823800f070 con 0x7f82400657d0 2022-06-29T16:05:48.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.518+0000 7f823cff9700 1 --2- 172.21.15.60:0/3282357939 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8220053b00 0x7f8220055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:48.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.518+0000 7f823cff9700 1 -- 172.21.15.60:0/3282357939 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f823806c7a0 con 0x7f82400657d0 2022-06-29T16:05:48.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.518+0000 7f823effd700 1 --2- 172.21.15.60:0/3282357939 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8220053b00 0x7f8220055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:48.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.518+0000 7f823effd700 1 --2- 172.21.15.60:0/3282357939 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8220053b00 0x7f8220055fc0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7f823400b000 tx=0x7f8234009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:48.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.526+0000 7f823cff9700 1 -- 172.21.15.60:0/3282357939 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f82380436c0 con 0x7f82400657d0 2022-06-29T16:05:48.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.690+0000 7f8244872700 1 -- 172.21.15.60:0/3282357939 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82400573d0 con 0x7f82400657d0 2022-06-29T16:05:48.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.690+0000 7f823cff9700 1 -- 172.21.15.60:0/3282357939 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f8238024870 con 0x7f82400657d0 2022-06-29T16:05:48.699 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:48.699 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":41,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6182,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1306898602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1306898602},{"type":"v1","addr":"172.21.15.60:6833","nonce":1306898602}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":39}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:05:40.369985+0000","modified":"2022-06-29T16:05:41.373815+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6185":{"gid":6185,"name":"b","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/2057420852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2057420852},{"type":"v1","addr":"172.21.15.92:6835","nonce":2057420852}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-06-29T16:05:48.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.694+0000 7f8244872700 1 -- 172.21.15.60:0/3282357939 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8220053b00 msgr2=0x7f8220055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:48.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.694+0000 7f8244872700 1 --2- 172.21.15.60:0/3282357939 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8220053b00 0x7f8220055fc0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7f823400b000 tx=0x7f8234009040).stop 2022-06-29T16:05:48.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.694+0000 7f8244872700 1 -- 172.21.15.60:0/3282357939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82400657d0 msgr2=0x7f8240126150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:48.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.694+0000 7f8244872700 1 --2- 172.21.15.60:0/3282357939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82400657d0 0x7f8240126150 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 rx=0x7f82380032f0 tx=0x7f8238016000).stop 2022-06-29T16:05:48.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.694+0000 7f8244872700 1 -- 172.21.15.60:0/3282357939 shutdown_connections 2022-06-29T16:05:48.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.694+0000 7f8244872700 1 --2- 172.21.15.60:0/3282357939 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8220053b00 0x7f8220055fc0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:48.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.694+0000 7f8244872700 1 --2- 172.21.15.60:0/3282357939 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f82400b0000 0x7f82400b0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:48.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.694+0000 7f8244872700 1 --2- 172.21.15.60:0/3282357939 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f82400ad990 0x7f8240142f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:48.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.694+0000 7f8244872700 1 --2- 172.21.15.60:0/3282357939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82400657d0 0x7f8240126150 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:48.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.694+0000 7f8244872700 1 -- 172.21.15.60:0/3282357939 >> 172.21.15.60:0/3282357939 conn(0x7f82400b6f80 msgr2=0x7f8240066f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:48.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.694+0000 7f8244872700 1 -- 172.21.15.60:0/3282357939 shutdown_connections 2022-06-29T16:05:48.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:48.694+0000 7f8244872700 1 -- 172.21.15.60:0/3282357939 wait complete. 2022-06-29T16:05:48.704 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 41 2022-06-29T16:05:48.914 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 41, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:05:40.369985+0000', 'modified': '2022-06-29T16:05:41.373815+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6185}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6185': {'gid': 6185, 'name': 'b', 'rank': 0, 'incarnation': 40, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.92:6835/2057420852', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 2057420852}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 2057420852}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:05:48.914 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:05:48.915 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-06-29T16:05:49.031 INFO:teuthology.orchestra.run.smithi092.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":6185,"want_state":"up:active","state":"up:active","rank_uptime":8.6449011979999995,"mdsmap_epoch":41,"osdmap_epoch":91,"osdmap_epoch_barrier":90,"uptime":13.368802432000001} 2022-06-29T16:05:49.032 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 6185, "mdsmap_epoch": 41, "osdmap_epoch": 91, "osdmap_epoch_barrier": 90, "rank_uptime": 8.644901198, "state": "up:active", "uptime": 13.368802432, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:05:49.033 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-06-29T16:05:49.033 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-06-29T16:05:49.033 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi156.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-06-29T16:05:49.034 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:49.034 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:05:49.043 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2022-06-29T16:05:49.061 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:05:49.061 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:05:49.118 INFO:teuthology.orchestra.run.smithi156.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:05:49.119 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:05:49.120 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:49.120 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2022-06-29T16:05:49.127 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-06-29T16:05:49.128 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:05:49.130 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2022-06-29T16:05:49.145 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:05:49.145 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:05:49.158 INFO:teuthology.orchestra.run.smithi156.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:05:49.159 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:05:49.160 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:49.161 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2022-06-29T16:05:49.208 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:2022-06-29T16:05:49.203+0000 7f21105d4080 -1 init, newargv = 0x558a819983b0 newargc=9 2022-06-29T16:05:49.208 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[12571]: starting ceph client 2022-06-29T16:05:49.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[12571]: starting fuse 2022-06-29T16:05:50.311 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2022-06-29T16:05:50.330 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:05:50.331 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:05:50.386 INFO:teuthology.orchestra.run.smithi156.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:05:50.387 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:05:50.388 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:50.388 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2022-06-29T16:05:50.396 INFO:teuthology.orchestra.run.smithi156.stdout:52 2022-06-29T16:05:50.396 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-06-29T16:05:50.396 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:50.397 DEBUG:teuthology.orchestra.run.smithi156:> sudo python3 -c ' 2022-06-29T16:05:50.397 DEBUG:teuthology.orchestra.run.smithi156:> import glob 2022-06-29T16:05:50.398 DEBUG:teuthology.orchestra.run.smithi156:> import re 2022-06-29T16:05:50.398 DEBUG:teuthology.orchestra.run.smithi156:> import os 2022-06-29T16:05:50.399 DEBUG:teuthology.orchestra.run.smithi156:> import subprocess 2022-06-29T16:05:50.399 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:05:50.400 DEBUG:teuthology.orchestra.run.smithi156:> def find_socket(client_name): 2022-06-29T16:05:50.400 DEBUG:teuthology.orchestra.run.smithi156:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-06-29T16:05:50.401 DEBUG:teuthology.orchestra.run.smithi156:> files = glob.glob(asok_path) 2022-06-29T16:05:50.401 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:05:50.401 DEBUG:teuthology.orchestra.run.smithi156:> # Given a non-glob path, it better be there 2022-06-29T16:05:50.402 DEBUG:teuthology.orchestra.run.smithi156:> if "*" not in asok_path: 2022-06-29T16:05:50.402 DEBUG:teuthology.orchestra.run.smithi156:> assert(len(files) == 1) 2022-06-29T16:05:50.403 DEBUG:teuthology.orchestra.run.smithi156:> return files[0] 2022-06-29T16:05:50.403 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:05:50.404 DEBUG:teuthology.orchestra.run.smithi156:> for f in files: 2022-06-29T16:05:50.404 DEBUG:teuthology.orchestra.run.smithi156:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-06-29T16:05:50.404 DEBUG:teuthology.orchestra.run.smithi156:> if os.path.exists("/proc/{0}".format(pid)): 2022-06-29T16:05:50.405 DEBUG:teuthology.orchestra.run.smithi156:> return f 2022-06-29T16:05:50.405 DEBUG:teuthology.orchestra.run.smithi156:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-06-29T16:05:50.406 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:05:50.406 DEBUG:teuthology.orchestra.run.smithi156:> print(find_socket("client.1")) 2022-06-29T16:05:50.407 DEBUG:teuthology.orchestra.run.smithi156:> ' 2022-06-29T16:05:50.449 INFO:teuthology.orchestra.run.smithi156.stdout:/var/run/ceph/ceph-client.1.12571.asok 2022-06-29T16:05:50.453 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.12571.asok 2022-06-29T16:05:50.454 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:50.454 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.12571.asok status 2022-06-29T16:05:50.539 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2022-06-29T16:05:50.539 INFO:teuthology.orchestra.run.smithi156.stdout: "metadata": { 2022-06-29T16:05:50.540 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_sha1": "f25ffac72e5a9e63860ff7deb6767a11f0559a16", 2022-06-29T16:05:50.540 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_version": "ceph version 15.2.16-304-gf25ffac7 (f25ffac72e5a9e63860ff7deb6767a11f0559a16) octopus (stable)", 2022-06-29T16:05:50.540 INFO:teuthology.orchestra.run.smithi156.stdout: "entity_id": "1", 2022-06-29T16:05:50.540 INFO:teuthology.orchestra.run.smithi156.stdout: "hostname": "smithi156", 2022-06-29T16:05:50.541 INFO:teuthology.orchestra.run.smithi156.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-06-29T16:05:50.541 INFO:teuthology.orchestra.run.smithi156.stdout: "pid": "12571", 2022-06-29T16:05:50.541 INFO:teuthology.orchestra.run.smithi156.stdout: "root": "/" 2022-06-29T16:05:50.541 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:05:50.541 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_count": 0, 2022-06-29T16:05:50.542 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_pinned_count": 0, 2022-06-29T16:05:50.542 INFO:teuthology.orchestra.run.smithi156.stdout: "id": 7215, 2022-06-29T16:05:50.542 INFO:teuthology.orchestra.run.smithi156.stdout: "inst": { 2022-06-29T16:05:50.542 INFO:teuthology.orchestra.run.smithi156.stdout: "name": { 2022-06-29T16:05:50.543 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "client", 2022-06-29T16:05:50.543 INFO:teuthology.orchestra.run.smithi156.stdout: "num": 7215 2022-06-29T16:05:50.543 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:05:50.543 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2022-06-29T16:05:50.543 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2022-06-29T16:05:50.544 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "172.21.15.156:0", 2022-06-29T16:05:50.544 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 3013098001 2022-06-29T16:05:50.544 INFO:teuthology.orchestra.run.smithi156.stdout: } 2022-06-29T16:05:50.544 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:05:50.545 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2022-06-29T16:05:50.545 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2022-06-29T16:05:50.545 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "172.21.15.156:0", 2022-06-29T16:05:50.545 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 3013098001 2022-06-29T16:05:50.545 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:05:50.546 INFO:teuthology.orchestra.run.smithi156.stdout: "inst_str": "client.7215 172.21.15.156:0/3013098001", 2022-06-29T16:05:50.546 INFO:teuthology.orchestra.run.smithi156.stdout: "addr_str": "172.21.15.156:0/3013098001", 2022-06-29T16:05:50.546 INFO:teuthology.orchestra.run.smithi156.stdout: "inode_count": 1, 2022-06-29T16:05:50.546 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_epoch": 41, 2022-06-29T16:05:50.547 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch": 91, 2022-06-29T16:05:50.547 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch_barrier": 0, 2022-06-29T16:05:50.547 INFO:teuthology.orchestra.run.smithi156.stdout: "blacklisted": false, 2022-06-29T16:05:50.547 INFO:teuthology.orchestra.run.smithi156.stdout: "fs_name": "cephfs" 2022-06-29T16:05:50.547 INFO:teuthology.orchestra.run.smithi156.stdout:} 2022-06-29T16:05:50.549 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:50.549 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:05:50.550 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:05:50.558 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2022-06-29T16:05:50.559 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-06-29T16:05:50.560 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:50.560 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:05:50.576 INFO:teuthology.orchestra.run:Running command with timeout 10800 2022-06-29T16:05:50.576 DEBUG:teuthology.orchestra.run.smithi121:> cd /home/ubuntu/cephtest/mnt.0 && sudo mkdir -p foo 2022-06-29T16:05:50.604 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-06-29T16:05:50.605 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-06-29T16:05:50.605 DEBUG:teuthology.orchestra.run.smithi121:> import stat 2022-06-29T16:05:50.605 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:05:50.606 DEBUG:teuthology.orchestra.run.smithi121:> print(os.stat("/home/ubuntu/cephtest/mnt.0/foo").st_ino) 2022-06-29T16:05:50.606 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-06-29T16:05:50.681 INFO:teuthology.orchestra.run.smithi121.stdout:1099511627776 2022-06-29T16:05:50.874 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:05:50.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.978+0000 7f7d6759e700 1 Processor -- start 2022-06-29T16:05:50.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.978+0000 7f7d6759e700 1 -- start start 2022-06-29T16:05:50.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.978+0000 7f7d6759e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7d680b2130 0x7f7d68146100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:50.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.978+0000 7f7d6759e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7d680aed00 0x7f7d681465f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:50.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.978+0000 7f7d6759e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d6805d450 con 0x7f7d6805d070 2022-06-29T16:05:50.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.978+0000 7f7d6759e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d680adf90 con 0x7f7d680b2130 2022-06-29T16:05:50.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.978+0000 7f7d6759e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d680b25d0 con 0x7f7d680aed00 2022-06-29T16:05:50.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.978+0000 7f7d6659c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f7d6805d070 0x7f7d6814aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59082/0 (socket says 172.21.15.60:59082) 2022-06-29T16:05:50.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.978+0000 7f7d6659c700 1 -- 172.21.15.60:0/1503961149 learned_addr learned my addr 172.21.15.60:0/1503961149 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:50.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.978+0000 7f7d65d9b700 1 --2- 172.21.15.60:0/1503961149 >> v2:172.21.15.60:3300/0 conn(0x7f7d680b2130 0x7f7d68146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:50.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.982+0000 7f7d66d9d700 1 --2- 172.21.15.60:0/1503961149 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7d680aed00 0x7f7d681465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:50.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.982+0000 7f7d65d9b700 1 -- 172.21.15.60:0/1503961149 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7d680aed00 msgr2=0x7f7d681465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:50.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.982+0000 7f7d65d9b700 1 --2- 172.21.15.60:0/1503961149 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7d680aed00 0x7f7d681465f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:50.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.982+0000 7f7d65d9b700 1 -- 172.21.15.60:0/1503961149 >> v1:172.21.15.60:6789/0 conn(0x7f7d6805d070 legacy=0x7f7d6814aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:50.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.982+0000 7f7d65d9b700 1 -- 172.21.15.60:0/1503961149 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d6814a350 con 0x7f7d680b2130 2022-06-29T16:05:50.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.982+0000 7f7d66d9d700 1 --2- 172.21.15.60:0/1503961149 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7d680aed00 0x7f7d681465f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:50.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.982+0000 7f7d65d9b700 1 --2- 172.21.15.60:0/1503961149 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d680b2130 0x7f7d68146100 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 rx=0x7f7d5c002f70 tx=0x7f7d5c006df0).ready entity=mon.0 client_cookie=322b567fadcaf61d server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:50.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.982+0000 7f7d6559a700 1 -- 172.21.15.60:0/1503961149 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d5c00d570 con 0x7f7d680b2130 2022-06-29T16:05:50.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.982+0000 7f7d6559a700 1 -- 172.21.15.60:0/1503961149 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d5c000e90 con 0x7f7d680b2130 2022-06-29T16:05:50.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.982+0000 7f7d6559a700 1 -- 172.21.15.60:0/1503961149 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d5c00d8c0 con 0x7f7d680b2130 2022-06-29T16:05:50.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.982+0000 7f7d6759e700 1 -- 172.21.15.60:0/1503961149 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d680b2130 msgr2=0x7f7d68146100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:50.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.982+0000 7f7d6759e700 1 --2- 172.21.15.60:0/1503961149 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d680b2130 0x7f7d68146100 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 rx=0x7f7d5c002f70 tx=0x7f7d5c006df0).stop 2022-06-29T16:05:50.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.982+0000 7f7d6759e700 1 -- 172.21.15.60:0/1503961149 shutdown_connections 2022-06-29T16:05:50.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.982+0000 7f7d6759e700 1 --2- 172.21.15.60:0/1503961149 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7d680aed00 0x7f7d681465f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:50.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.982+0000 7f7d6759e700 1 --2- 172.21.15.60:0/1503961149 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d680b2130 0x7f7d68146100 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:50.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.982+0000 7f7d6759e700 1 -- 172.21.15.60:0/1503961149 >> 172.21.15.60:0/1503961149 conn(0x7f7d680b5890 msgr2=0x7f7d680b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:50.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.982+0000 7f7d6759e700 1 -- 172.21.15.60:0/1503961149 shutdown_connections 2022-06-29T16:05:50.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.982+0000 7f7d6759e700 1 -- 172.21.15.60:0/1503961149 wait complete. 2022-06-29T16:05:51.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.982+0000 7f7d6759e700 1 Processor -- start 2022-06-29T16:05:51.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.982+0000 7f7d6759e700 1 -- start start 2022-06-29T16:05:51.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.986+0000 7f7d6759e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d6805d070 0x7f7d6814c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:51.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.986+0000 7f7d6759e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d680aed00 0x7f7d68066a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:51.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.986+0000 7f7d6759e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7d680b2130 0x7f7d6813fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:51.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.986+0000 7f7d6759e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6814bcc0 con 0x7f7d680aed00 2022-06-29T16:05:51.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.986+0000 7f7d6759e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d681481b0 con 0x7f7d6805d070 2022-06-29T16:05:51.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.986+0000 7f7d6759e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6814b920 con 0x7f7d680b2130 2022-06-29T16:05:51.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.986+0000 7f7d65d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d680aed00 0x7f7d68066a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:51.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.986+0000 7f7d65d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d680aed00 0x7f7d68066a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51004/0 (socket says 172.21.15.60:51004) 2022-06-29T16:05:51.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.986+0000 7f7d65d9b700 1 -- 172.21.15.60:0/3776370352 learned_addr learned my addr 172.21.15.60:0/3776370352 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:51.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.986+0000 7f7d6659c700 1 --2- 172.21.15.60:0/3776370352 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d6805d070 0x7f7d6814c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:51.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.986+0000 7f7d66d9d700 1 --2- 172.21.15.60:0/3776370352 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7d680b2130 0x7f7d6813fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:51.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.986+0000 7f7d65d9b700 1 -- 172.21.15.60:0/3776370352 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7d680b2130 msgr2=0x7f7d6813fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:51.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.986+0000 7f7d65d9b700 1 --2- 172.21.15.60:0/3776370352 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7d680b2130 0x7f7d6813fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:51.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.986+0000 7f7d65d9b700 1 -- 172.21.15.60:0/3776370352 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d6805d070 msgr2=0x7f7d6814c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:51.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.986+0000 7f7d65d9b700 1 --2- 172.21.15.60:0/3776370352 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d6805d070 0x7f7d6814c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:51.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.986+0000 7f7d65d9b700 1 -- 172.21.15.60:0/3776370352 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d5c00b180 con 0x7f7d680aed00 2022-06-29T16:05:51.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.986+0000 7f7d65d9b700 1 --2- 172.21.15.60:0/3776370352 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d680aed00 0x7f7d68066a90 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 rx=0x7f7d6814bab0 tx=0x7f7d5c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:51.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.986+0000 7f7d577fe700 1 -- 172.21.15.60:0/3776370352 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d5c00bed0 con 0x7f7d680aed00 2022-06-29T16:05:51.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.986+0000 7f7d577fe700 1 -- 172.21.15.60:0/3776370352 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d5c00de30 con 0x7f7d680aed00 2022-06-29T16:05:51.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.986+0000 7f7d577fe700 1 -- 172.21.15.60:0/3776370352 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d5c011d30 con 0x7f7d680aed00 2022-06-29T16:05:51.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.986+0000 7f7d6759e700 1 -- 172.21.15.60:0/3776370352 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d6814b3e0 con 0x7f7d680aed00 2022-06-29T16:05:51.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.986+0000 7f7d6759e700 1 -- 172.21.15.60:0/3776370352 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d6814d520 con 0x7f7d680aed00 2022-06-29T16:05:51.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.990+0000 7f7d6759e700 1 -- 172.21.15.60:0/3776370352 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d6814a5f0 con 0x7f7d680aed00 2022-06-29T16:05:51.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.990+0000 7f7d577fe700 1 -- 172.21.15.60:0/3776370352 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7d5c025030 con 0x7f7d680aed00 2022-06-29T16:05:51.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.990+0000 7f7d577fe700 1 --2- 172.21.15.60:0/3776370352 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7d44053b00 0x7f7d44055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:51.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.990+0000 7f7d577fe700 1 -- 172.21.15.60:0/3776370352 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7d5c06d0d0 con 0x7f7d680aed00 2022-06-29T16:05:51.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.990+0000 7f7d6659c700 1 --2- 172.21.15.60:0/3776370352 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7d44053b00 0x7f7d44055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:51.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.990+0000 7f7d6659c700 1 --2- 172.21.15.60:0/3776370352 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7d44053b00 0x7f7d44055fc0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7f7d5800d000 tx=0x7f7d58009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:51.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:50.994+0000 7f7d577fe700 1 -- 172.21.15.60:0/3776370352 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7d5c043ee0 con 0x7f7d680aed00 2022-06-29T16:05:51.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.158+0000 7f7d6759e700 1 -- 172.21.15.60:0/3776370352 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d680573d0 con 0x7f7d680aed00 2022-06-29T16:05:51.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.162+0000 7f7d577fe700 1 -- 172.21.15.60:0/3776370352 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f7d5c023020 con 0x7f7d680aed00 2022-06-29T16:05:51.169 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:51.169 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":41,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6182,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1306898602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1306898602},{"type":"v1","addr":"172.21.15.60:6833","nonce":1306898602}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":39}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:05:40.369985+0000","modified":"2022-06-29T16:05:41.373815+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6185":{"gid":6185,"name":"b","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/2057420852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2057420852},{"type":"v1","addr":"172.21.15.92:6835","nonce":2057420852}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-06-29T16:05:51.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.162+0000 7f7d6759e700 1 -- 172.21.15.60:0/3776370352 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7d44053b00 msgr2=0x7f7d44055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:51.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.162+0000 7f7d6759e700 1 --2- 172.21.15.60:0/3776370352 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7d44053b00 0x7f7d44055fc0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7f7d5800d000 tx=0x7f7d58009040).stop 2022-06-29T16:05:51.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.162+0000 7f7d6759e700 1 -- 172.21.15.60:0/3776370352 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d680aed00 msgr2=0x7f7d68066a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:51.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.162+0000 7f7d6759e700 1 --2- 172.21.15.60:0/3776370352 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d680aed00 0x7f7d68066a90 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 rx=0x7f7d6814bab0 tx=0x7f7d5c015000).stop 2022-06-29T16:05:51.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.162+0000 7f7d6759e700 1 -- 172.21.15.60:0/3776370352 shutdown_connections 2022-06-29T16:05:51.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.162+0000 7f7d6759e700 1 --2- 172.21.15.60:0/3776370352 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7d44053b00 0x7f7d44055fc0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:51.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.162+0000 7f7d6759e700 1 --2- 172.21.15.60:0/3776370352 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7d680b2130 0x7f7d6813fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:51.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.162+0000 7f7d6759e700 1 --2- 172.21.15.60:0/3776370352 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d680aed00 0x7f7d68066a90 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:51.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.162+0000 7f7d6759e700 1 --2- 172.21.15.60:0/3776370352 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d6805d070 0x7f7d6814c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:51.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.162+0000 7f7d6759e700 1 -- 172.21.15.60:0/3776370352 >> 172.21.15.60:0/3776370352 conn(0x7f7d680b5890 msgr2=0x7f7d68144590 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:51.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.162+0000 7f7d6759e700 1 -- 172.21.15.60:0/3776370352 shutdown_connections 2022-06-29T16:05:51.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.162+0000 7f7d6759e700 1 -- 172.21.15.60:0/3776370352 wait complete. 2022-06-29T16:05:51.175 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 41 2022-06-29T16:05:51.182 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json dump inode 0x10000000000 2022-06-29T16:05:51.297 INFO:teuthology.orchestra.run.smithi092.stdout:{"path":"/foo","ino":1099511627776,"rdev":0,"ctime":"2022-06-29T16:05:50.593937+0000","btime":"2022-06-29T16:05:50.593937+0000","mode":16877,"uid":0,"gid":0,"nlink":1,"dir_layout":{"dir_hash":2,"unused1":0,"unused2":0,"unused3":0},"layout":{"stripe_unit":0,"stripe_count":0,"object_size":0,"pool_id":-1,"pool_ns":""},"old_pools":[],"size":0,"truncate_seq":1,"truncate_size":18446744073709551615,"truncate_from":0,"truncate_pending":0,"mtime":"2022-06-29T16:05:50.593937+0000","atime":"2022-06-29T16:05:50.593937+0000","time_warp_seq":0,"change_attr":0,"export_pin":-1,"export_ephemeral_random_pin":0,"export_ephemeral_distributed_pin":false,"client_ranges":[],"dirstat":{"version":0,"mtime":"0.000000","num_files":0,"num_subdirs":0,"change_attr":0},"rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":1,"rsnaps":0,"rctime":"0.000000"},"accounted_rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":1,"rsnaps":0,"rctime":"0.000000"},"version":2,"file_data_version":0,"xattr_version":1,"backtrace_version":2,"stray_prior_path":"","max_size_ever":0,"quota":{"max_bytes":0,"max_files":0},"last_scrub_stamp":"0.000000","last_scrub_version":0,"symlink":"","xattrs":[],"dirfragtree":{"splits":[]},"old_inodes":[],"oldest_snap":18446744073709551614,"damage_flags":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"request":0,"dirfrag":1,"caps":1,"dirtyparent":1,"dirty":1},"nref":4,"versionlock":{"gather_set":[],"state":"lock","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"authlock":{"gather_set":[],"state":"excl","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"linklock":{},"dirfragtreelock":{},"filelock":{"gather_set":[],"state":"excl","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"xattrlock":{"gather_set":[],"state":"excl","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"snaplock":{},"nestlock":{"gather_set":[],"state":"lock","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"flocklock":{},"policylock":{},"states":["auth","dirty","dirtyparent","dirtypool"],"client_caps":[{"client_id":7159,"pending":"pAsxLsXsxFsx","issued":"pAsxLsXsxFsx","wanted":"-","last_sent":1}],"loner":7159,"want_loner":7159,"mds_caps_wanted":[]} 2022-06-29T16:05:51.298 DEBUG:tasks.cephfs.filesystem:_json_asok output { "accounted_rstat": { "rbytes": 0, "rctime": "0.000000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "atime": "2022-06-29T16:05:50.593937+0000", "auth_pins": 0, "auth_state": { "replicas": {} }, "authlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "excl", "xlock_by": {} }, "backtrace_version": 2, "btime": "2022-06-29T16:05:50.593937+0000", "change_attr": 0, "client_caps": [ { "client_id": 7159, "issued": "pAsxLsXsxFsx", "last_sent": 1, "pending": "pAsxLsXsxFsx", "wanted": "-" } ], "client_ranges": [], "ctime": "2022-06-29T16:05:50.593937+0000", "damage_flags": 0, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "dirfragtree": { "splits": [] }, "dirfragtreelock": {}, "dirstat": { "change_attr": 0, "mtime": "0.000000", "num_files": 0, "num_subdirs": 0, "version": 0 }, "export_ephemeral_distributed_pin": false, "export_ephemeral_random_pin": 0, "export_pin": -1, "file_data_version": 0, "filelock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "excl", "xlock_by": {} }, "flocklock": {}, "gid": 0, "ino": 1099511627776, "is_auth": true, "is_freezing": false, "is_frozen": false, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "layout": { "object_size": 0, "pool_id": -1, "pool_ns": "", "stripe_count": 0, "stripe_unit": 0 }, "linklock": {}, "loner": 7159, "max_size_ever": 0, "mds_caps_wanted": [], "mode": 16877, "mtime": "2022-06-29T16:05:50.593937+0000", "nestlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "xlock_by": {} }, "nlink": 1, "nref": 4, "old_inodes": [], "old_pools": [], "oldest_snap": 18446744073709551614, "path": "/foo", "pins": { "caps": 1, "dirfrag": 1, "dirty": 1, "dirtyparent": 1, "request": 0 }, "policylock": {}, "quota": { "max_bytes": 0, "max_files": 0 }, "rdev": 0, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "rstat": { "rbytes": 0, "rctime": "0.000000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "size": 0, "snaplock": {}, "states": [ "auth", "dirty", "dirtyparent", "dirtypool" ], "stray_prior_path": "", "symlink": "", "time_warp_seq": 0, "truncate_from": 0, "truncate_pending": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "uid": 0, "version": 2, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "xlock_by": {} }, "want_loner": 7159, "xattr_version": 1, "xattrlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "excl", "xlock_by": {} }, "xattrs": [] } 2022-06-29T16:05:51.299 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-06-29T16:05:51.299 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:05:51.320 DEBUG:teuthology.orchestra.run.smithi092:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:05:51.341 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:05:51.363 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:05:51.384 DEBUG:teuthology.orchestra.run.smithi197:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:05:51.405 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-06-29T16:05:51.406 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:51.407 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:05:51.407 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:05:51.416 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T16:05:51.417 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T16:05:51.418 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2022-06-29T16:05:51.418 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-06-29T16:05:51.419 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:05:51.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[13133]: fuse finished with error 0 and tester_r 0 2022-06-29T16:05:51.476 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:51.477 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:05:51.477 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:05:51.485 INFO:teuthology.orchestra.run.smithi121.stdout:ext2/ext3 2022-06-29T16:05:51.487 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-06-29T16:05:51.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:05:51.499 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:05:51.546 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:51.547 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:05:51.547 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:05:51.556 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2022-06-29T16:05:51.557 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-06-29T16:05:51.558 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi156.front.sepia.ceph.com... 2022-06-29T16:05:51.558 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-06-29T16:05:51.559 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:05:51.614 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[12571]: fuse finished with error 0 and tester_r 0 2022-06-29T16:05:51.616 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:51.617 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:05:51.618 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:05:51.625 INFO:teuthology.orchestra.run.smithi156.stdout:ext2/ext3 2022-06-29T16:05:51.626 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-06-29T16:05:51.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:05:51.744 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:05:51.794 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:51.794 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:05:51.795 DEBUG:teuthology.orchestra.run.smithi197:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T16:05:51.802 INFO:teuthology.orchestra.run.smithi197.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-06-29T16:05:51.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:05:51.803 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-06-29T16:05:51.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:05:51.804 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T16:05:51.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:05:51.813 INFO:teuthology.orchestra.run.smithi197.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-06-29T16:05:51.814 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-06-29T16:05:51.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.918+0000 7f9353868700 1 Processor -- start 2022-06-29T16:05:51.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.918+0000 7f9353868700 1 -- start start 2022-06-29T16:05:51.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.918+0000 7f9353868700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f934c0b3cc0 0x7f934c1420b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:51.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.918+0000 7f9353868700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f934c0b2ce0 con 0x7f934c0b9d70 2022-06-29T16:05:51.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.918+0000 7f9353868700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f934c0ba150 con 0x7f934c05d1f0 2022-06-29T16:05:51.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.918+0000 7f9353868700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f934c0b4d10 con 0x7f934c0b3cc0 2022-06-29T16:05:51.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.918+0000 7f9351604700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f934c0b9d70 0x7f934c0b3140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59096/0 (socket says 172.21.15.60:59096) 2022-06-29T16:05:51.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.918+0000 7f9351604700 1 -- 172.21.15.60:0/4031893039 learned_addr learned my addr 172.21.15.60:0/4031893039 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:51.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.918+0000 7f9351e05700 1 --2- 172.21.15.60:0/4031893039 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f934c0b3cc0 0x7f934c1420b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:51.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.918+0000 7f934bfff700 1 -- 172.21.15.60:0/4031893039 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267471597 0 0) 0x7f934c0b2ce0 con 0x7f934c0b9d70 2022-06-29T16:05:51.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.918+0000 7f934bfff700 1 -- 172.21.15.60:0/4031893039 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f932c003260 con 0x7f934c0b9d70 2022-06-29T16:05:51.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.918+0000 7f934bfff700 1 -- 172.21.15.60:0/4031893039 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1741819135 0 0) 0x7f934c0ba150 con 0x7f934c05d1f0 2022-06-29T16:05:51.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.918+0000 7f934bfff700 1 -- 172.21.15.60:0/4031893039 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f934c0b2ce0 con 0x7f934c05d1f0 2022-06-29T16:05:51.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.918+0000 7f934bfff700 1 -- 172.21.15.60:0/4031893039 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 349967242 0 0) 0x7f932c003260 con 0x7f934c0b9d70 2022-06-29T16:05:51.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.918+0000 7f934bfff700 1 -- 172.21.15.60:0/4031893039 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f934c0b3cc0 msgr2=0x7f934c1420b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:51.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.918+0000 7f934bfff700 1 --2- 172.21.15.60:0/4031893039 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f934c0b3cc0 0x7f934c1420b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:51.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.918+0000 7f934bfff700 1 -- 172.21.15.60:0/4031893039 >> v1:172.21.15.92:6789/0 conn(0x7f934c05d1f0 legacy=0x7f934c0b1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:51.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.918+0000 7f934bfff700 1 -- 172.21.15.60:0/4031893039 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f934c0b4a70 con 0x7f934c0b9d70 2022-06-29T16:05:51.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.918+0000 7f934bfff700 1 -- 172.21.15.60:0/4031893039 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f9344001a00 con 0x7f934c0b9d70 2022-06-29T16:05:51.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.918+0000 7f934bfff700 1 -- 172.21.15.60:0/4031893039 >> v1:172.21.15.60:6789/0 conn(0x7f934c0b9d70 legacy=0x7f934c0b3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:51.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.918+0000 7f934bfff700 1 --2- 172.21.15.60:0/4031893039 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f932c003910 0x7f932c003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:51.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.918+0000 7f934bfff700 1 --2- 172.21.15.60:0/4031893039 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f932c004500 0x7f932c004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:51.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.918+0000 7f934bfff700 1 --2- 172.21.15.60:0/4031893039 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f934c0b3cc0 0x7f932c004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:51.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.918+0000 7f934bfff700 1 -- 172.21.15.60:0/4031893039 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f934c0b4d10 con 0x7f932c003910 2022-06-29T16:05:51.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.918+0000 7f934bfff700 1 -- 172.21.15.60:0/4031893039 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932c00b780 con 0x7f932c004500 2022-06-29T16:05:51.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.918+0000 7f934bfff700 1 -- 172.21.15.60:0/4031893039 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f932c00b940 con 0x7f934c0b3cc0 2022-06-29T16:05:51.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.918+0000 7f9350e03700 1 --2- 172.21.15.60:0/4031893039 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f932c004500 0x7f932c004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:51.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.918+0000 7f9351604700 1 --2- 172.21.15.60:0/4031893039 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f932c003910 0x7f932c003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:51.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.922+0000 7f9351e05700 1 --2- 172.21.15.60:0/4031893039 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f934c0b3cc0 0x7f932c004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:51.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.922+0000 7f9350e03700 1 -- 172.21.15.60:0/4031893039 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f934c0b3cc0 msgr2=0x7f932c004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:51.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.922+0000 7f9350e03700 1 --2- 172.21.15.60:0/4031893039 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f934c0b3cc0 0x7f932c004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:51.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.922+0000 7f9350e03700 1 -- 172.21.15.60:0/4031893039 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f932c003910 msgr2=0x7f932c003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:51.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.922+0000 7f9350e03700 1 --2- 172.21.15.60:0/4031893039 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f932c003910 0x7f932c003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:51.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.922+0000 7f9350e03700 1 -- 172.21.15.60:0/4031893039 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f932c003260 con 0x7f932c004500 2022-06-29T16:05:51.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.922+0000 7f9351604700 1 --2- 172.21.15.60:0/4031893039 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f932c003910 0x7f932c003d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:05:51.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.922+0000 7f9351e05700 1 --2- 172.21.15.60:0/4031893039 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f934c0b3cc0 0x7f932c004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:05:51.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.922+0000 7f9350e03700 1 --2- 172.21.15.60:0/4031893039 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f932c004500 0x7f932c004980 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f9334001110 tx=0x7f9334007a00).ready entity=mon.1 client_cookie=9ee512cd26d8833 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:51.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.922+0000 7f934bfff700 1 -- 172.21.15.60:0/4031893039 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9334007d10 con 0x7f932c004500 2022-06-29T16:05:51.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.922+0000 7f934bfff700 1 -- 172.21.15.60:0/4031893039 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9334007030 con 0x7f932c004500 2022-06-29T16:05:51.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.922+0000 7f934bfff700 1 -- 172.21.15.60:0/4031893039 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9334007390 con 0x7f932c004500 2022-06-29T16:05:51.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.922+0000 7f9353868700 1 -- 172.21.15.60:0/4031893039 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f932c004500 msgr2=0x7f932c004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:51.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.922+0000 7f9353868700 1 --2- 172.21.15.60:0/4031893039 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f932c004500 0x7f932c004980 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f9334001110 tx=0x7f9334007a00).stop 2022-06-29T16:05:51.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.922+0000 7f9351e05700 1 -- 172.21.15.60:0/4031893039 reap_dead start 2022-06-29T16:05:51.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.922+0000 7f9353868700 1 -- 172.21.15.60:0/4031893039 shutdown_connections 2022-06-29T16:05:51.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.922+0000 7f9353868700 1 -- 172.21.15.60:0/4031893039 >> 172.21.15.60:0/4031893039 conn(0x7f934c062ce0 msgr2=0x7f934c0b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:51.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.922+0000 7f9353868700 1 -- 172.21.15.60:0/4031893039 shutdown_connections 2022-06-29T16:05:51.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.922+0000 7f9353868700 1 -- 172.21.15.60:0/4031893039 wait complete. 2022-06-29T16:05:51.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.922+0000 7f9353868700 1 Processor -- start 2022-06-29T16:05:51.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.922+0000 7f9353868700 1 -- start start 2022-06-29T16:05:51.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.926+0000 7f9353868700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f932c004500 0x7f934c0af530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:51.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.926+0000 7f9353868700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f934c145500 0x7f934c0afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:51.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.926+0000 7f9353868700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f934c145c90 0x7f934c146140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:51.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.926+0000 7f9353868700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f934c1447e0 con 0x7f934c145c90 2022-06-29T16:05:51.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.926+0000 7f9353868700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f934c0ae000 con 0x7f934c145500 2022-06-29T16:05:51.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.926+0000 7f9353868700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f934c1440a0 con 0x7f932c004500 2022-06-29T16:05:51.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.926+0000 7f9351e05700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f934c145c90 0x7f934c146140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:51.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.926+0000 7f9351e05700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f934c145c90 0x7f934c146140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51026/0 (socket says 172.21.15.60:51026) 2022-06-29T16:05:51.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.926+0000 7f9351e05700 1 -- 172.21.15.60:0/2001334289 learned_addr learned my addr 172.21.15.60:0/2001334289 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:51.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.926+0000 7f9351604700 1 --2- 172.21.15.60:0/2001334289 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f932c004500 0x7f934c0af530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:51.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.926+0000 7f9350e03700 1 --2- 172.21.15.60:0/2001334289 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f934c145500 0x7f934c0afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:51.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.926+0000 7f9351e05700 1 -- 172.21.15.60:0/2001334289 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f932c004500 msgr2=0x7f934c0af530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:51.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.926+0000 7f9351e05700 1 --2- 172.21.15.60:0/2001334289 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f932c004500 0x7f934c0af530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:51.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.926+0000 7f9351e05700 1 -- 172.21.15.60:0/2001334289 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f934c145500 msgr2=0x7f934c0afa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:51.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.926+0000 7f9351e05700 1 --2- 172.21.15.60:0/2001334289 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f934c145500 0x7f934c0afa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:51.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.926+0000 7f9351e05700 1 -- 172.21.15.60:0/2001334289 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f934c144340 con 0x7f934c145c90 2022-06-29T16:05:51.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.926+0000 7f9351e05700 1 --2- 172.21.15.60:0/2001334289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f934c145c90 0x7f934c146140 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 rx=0x7f933c00b000 tx=0x7f933c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:51.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.926+0000 7f934a7fc700 1 -- 172.21.15.60:0/2001334289 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f933c017070 con 0x7f934c145c90 2022-06-29T16:05:51.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.926+0000 7f934a7fc700 1 -- 172.21.15.60:0/2001334289 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f933c000e90 con 0x7f934c145c90 2022-06-29T16:05:51.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.926+0000 7f9353868700 1 -- 172.21.15.60:0/2001334289 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f934c0adc60 con 0x7f934c145c90 2022-06-29T16:05:51.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.926+0000 7f934a7fc700 1 -- 172.21.15.60:0/2001334289 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f933c0135f0 con 0x7f934c145c90 2022-06-29T16:05:51.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.926+0000 7f9353868700 1 -- 172.21.15.60:0/2001334289 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f934c155470 con 0x7f934c145c90 2022-06-29T16:05:51.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.926+0000 7f9353868700 1 -- 172.21.15.60:0/2001334289 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f934c1444d0 con 0x7f934c145c90 2022-06-29T16:05:51.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.930+0000 7f934a7fc700 1 -- 172.21.15.60:0/2001334289 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f933c013750 con 0x7f934c145c90 2022-06-29T16:05:51.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.930+0000 7f934a7fc700 1 --2- 172.21.15.60:0/2001334289 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f93300539e0 0x7f9330055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:51.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.930+0000 7f934a7fc700 1 -- 172.21.15.60:0/2001334289 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f933c06ba00 con 0x7f934c145c90 2022-06-29T16:05:51.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.930+0000 7f9351604700 1 --2- 172.21.15.60:0/2001334289 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f93300539e0 0x7f9330055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:51.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.930+0000 7f9351604700 1 --2- 172.21.15.60:0/2001334289 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f93300539e0 0x7f9330055ea0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7f9344005000 tx=0x7f934c0b19c0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:51.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:51.938+0000 7f934a7fc700 1 -- 172.21.15.60:0/2001334289 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f933c042820 con 0x7f934c145c90 2022-06-29T16:05:52.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.062+0000 7f9353868700 1 -- 172.21.15.60:0/2001334289 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f934c0addf0 con 0x7f934c145c90 2022-06-29T16:05:52.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.066+0000 7f934a7fc700 1 -- 172.21.15.60:0/2001334289 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v91) v1 ==== 81+0+33648 (secure 0 0 0) 0x7f933c045e40 con 0x7f934c145c90 2022-06-29T16:05:52.075 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:52.075 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:05:52.076 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 91, 2022-06-29T16:05:52.076 INFO:teuthology.orchestra.run.smithi060.stdout: "fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", 2022-06-29T16:05:52.076 INFO:teuthology.orchestra.run.smithi060.stdout: "created": "2022-06-29T15:58:07.965719+0000", 2022-06-29T16:05:52.076 INFO:teuthology.orchestra.run.smithi060.stdout: "modified": "2022-06-29T16:05:41.373483+0000", 2022-06-29T16:05:52.077 INFO:teuthology.orchestra.run.smithi060.stdout: "last_up_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:05:52.077 INFO:teuthology.orchestra.run.smithi060.stdout: "last_in_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:05:52.077 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-06-29T16:05:52.077 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_num": 5799936, 2022-06-29T16:05:52.078 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_set": [ 2022-06-29T16:05:52.078 INFO:teuthology.orchestra.run.smithi060.stdout: "pglog_hardlimit", 2022-06-29T16:05:52.078 INFO:teuthology.orchestra.run.smithi060.stdout: "purged_snapdirs", 2022-06-29T16:05:52.078 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_deletes", 2022-06-29T16:05:52.078 INFO:teuthology.orchestra.run.smithi060.stdout: "sortbitwise" 2022-06-29T16:05:52.079 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:05:52.079 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_version": 4, 2022-06-29T16:05:52.079 INFO:teuthology.orchestra.run.smithi060.stdout: "full_ratio": 0.94999998807907104, 2022-06-29T16:05:52.079 INFO:teuthology.orchestra.run.smithi060.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-06-29T16:05:52.080 INFO:teuthology.orchestra.run.smithi060.stdout: "nearfull_ratio": 0.85000002384185791, 2022-06-29T16:05:52.080 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_snapshot": "", 2022-06-29T16:05:52.080 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_max": 18, 2022-06-29T16:05:52.080 INFO:teuthology.orchestra.run.smithi060.stdout: "max_osd": 8, 2022-06-29T16:05:52.081 INFO:teuthology.orchestra.run.smithi060.stdout: "require_min_compat_client": "jewel", 2022-06-29T16:05:52.081 INFO:teuthology.orchestra.run.smithi060.stdout: "min_compat_client": "jewel", 2022-06-29T16:05:52.081 INFO:teuthology.orchestra.run.smithi060.stdout: "require_osd_release": "octopus", 2022-06-29T16:05:52.081 INFO:teuthology.orchestra.run.smithi060.stdout: "pools": [ 2022-06-29T16:05:52.081 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.082 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2022-06-29T16:05:52.082 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "device_health_metrics", 2022-06-29T16:05:52.082 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:10.961127+0000", 2022-06-29T16:05:52.082 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:05:52.083 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:05:52.083 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:05:52.083 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:05:52.083 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:05:52.084 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:05:52.084 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:05:52.084 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:05:52.084 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 1, 2022-06-29T16:05:52.085 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 1, 2022-06-29T16:05:52.085 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 1, 2022-06-29T16:05:52.085 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 1, 2022-06-29T16:05:52.085 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 1, 2022-06-29T16:05:52.085 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:05:52.086 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:05:52.086 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:05:52.086 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:05:52.086 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:05:52.087 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:05:52.087 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:05:52.087 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.087 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "15", 2022-06-29T16:05:52.087 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:05:52.088 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:05:52.088 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:05:52.088 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:05:52.088 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:05:52.089 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:05:52.089 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:05:52.089 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:05:52.089 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:05:52.090 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:05:52.090 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:05:52.090 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:05:52.090 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:05:52.091 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:05:52.091 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:05:52.091 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:05:52.091 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:05:52.091 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:05:52.092 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:05:52.092 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:05:52.092 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:05:52.092 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:05:52.093 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:05:52.093 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:05:52.093 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:05:52.093 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:05:52.093 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.094 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:05:52.094 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:05:52.094 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:05:52.094 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:05:52.095 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:05:52.095 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:05:52.095 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:05:52.095 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:05:52.096 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:05:52.096 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:05:52.096 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:05:52.096 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:05:52.097 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 1 2022-06-29T16:05:52.097 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.097 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:05:52.097 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_devicehealth": {} 2022-06-29T16:05:52.097 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.098 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.098 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.098 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 2, 2022-06-29T16:05:52.098 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "rbd", 2022-06-29T16:05:52.099 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:19.184276+0000", 2022-06-29T16:05:52.099 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:05:52.099 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:05:52.099 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:05:52.099 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:05:52.100 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:05:52.100 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:05:52.100 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:05:52.100 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:05:52.101 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:05:52.101 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:05:52.101 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:05:52.101 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:05:52.101 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:05:52.102 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:05:52.102 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:05:52.102 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:05:52.102 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:05:52.103 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:05:52.103 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:05:52.103 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:05:52.103 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.103 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "18", 2022-06-29T16:05:52.104 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:05:52.104 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:05:52.104 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:05:52.104 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:05:52.105 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:05:52.105 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:05:52.105 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:05:52.105 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:05:52.106 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:05:52.106 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:05:52.106 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:05:52.106 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:05:52.107 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:05:52.107 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:05:52.107 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:05:52.107 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:05:52.107 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:05:52.108 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:05:52.108 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:05:52.108 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:05:52.108 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:05:52.109 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:05:52.109 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:05:52.109 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:05:52.109 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:05:52.110 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:05:52.110 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.110 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:05:52.110 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:05:52.111 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:05:52.111 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:05:52.111 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:05:52.111 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:05:52.112 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:05:52.112 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:05:52.112 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:05:52.112 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:05:52.112 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:05:52.113 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:05:52.113 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:05:52.113 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd": {} 2022-06-29T16:05:52.113 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.114 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.114 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.114 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 17, 2022-06-29T16:05:52.114 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "cephfs_metadata", 2022-06-29T16:05:52.115 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T16:05:38.407939+0000", 2022-06-29T16:05:52.115 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:05:52.115 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:05:52.115 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:05:52.116 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:05:52.116 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:05:52.116 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:05:52.116 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:05:52.116 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:05:52.117 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:05:52.117 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:05:52.117 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:05:52.117 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:05:52.118 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:05:52.118 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:05:52.118 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:05:52.118 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:05:52.119 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:05:52.119 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:05:52.119 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:05:52.119 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:05:52.119 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.120 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "90", 2022-06-29T16:05:52.120 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:05:52.120 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:05:52.120 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:05:52.121 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:05:52.121 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:05:52.121 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:05:52.121 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:05:52.121 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:05:52.122 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:05:52.122 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:05:52.122 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:05:52.122 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:05:52.123 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:05:52.123 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:05:52.123 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:05:52.123 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:05:52.123 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:05:52.124 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:05:52.124 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:05:52.124 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:05:52.124 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:05:52.125 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:05:52.125 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:05:52.125 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:05:52.125 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:05:52.125 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:05:52.126 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.126 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:05:52.126 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:05:52.126 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:05:52.127 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:05:52.127 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:05:52.127 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:05:52.127 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:05:52.128 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:05:52.128 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:05:52.128 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:05:52.128 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:05:52.128 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:05:52.129 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_bias": 4, 2022-06-29T16:05:52.129 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 16, 2022-06-29T16:05:52.129 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_priority": 5 2022-06-29T16:05:52.129 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.130 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:05:52.130 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs": { 2022-06-29T16:05:52.130 INFO:teuthology.orchestra.run.smithi060.stdout: "metadata": "cephfs" 2022-06-29T16:05:52.131 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.131 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.131 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.131 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.132 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 18, 2022-06-29T16:05:52.132 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "cephfs_data", 2022-06-29T16:05:52.132 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T16:05:39.380608+0000", 2022-06-29T16:05:52.132 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:05:52.133 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:05:52.133 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:05:52.133 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:05:52.133 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:05:52.133 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:05:52.134 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:05:52.134 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:05:52.134 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:05:52.134 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:05:52.135 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:05:52.135 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:05:52.135 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:05:52.136 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:05:52.136 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:05:52.136 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:05:52.136 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:05:52.137 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:05:52.137 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:05:52.137 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:05:52.137 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.137 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "91", 2022-06-29T16:05:52.138 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:05:52.138 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:05:52.138 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:05:52.138 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:05:52.139 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:05:52.139 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:05:52.139 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:05:52.139 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:05:52.139 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:05:52.140 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:05:52.140 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:05:52.140 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:05:52.140 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:05:52.141 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:05:52.141 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:05:52.141 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:05:52.141 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:05:52.141 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:05:52.142 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:05:52.142 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:05:52.142 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:05:52.142 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:05:52.143 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:05:52.143 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:05:52.143 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:05:52.143 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:05:52.143 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.144 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:05:52.144 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:05:52.144 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:05:52.144 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:05:52.145 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:05:52.145 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:05:52.145 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:05:52.145 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:05:52.146 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:05:52.146 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:05:52.146 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:05:52.146 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:05:52.146 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:05:52.147 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs": { 2022-06-29T16:05:52.147 INFO:teuthology.orchestra.run.smithi060.stdout: "data": "cephfs" 2022-06-29T16:05:52.147 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.147 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.148 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.148 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:05:52.148 INFO:teuthology.orchestra.run.smithi060.stdout: "osds": [ 2022-06-29T16:05:52.148 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.149 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:05:52.149 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "11e7ab9e-fce2-470f-8195-ea4c3c52df22", 2022-06-29T16:05:52.149 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:52.149 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:52.150 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:52.150 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:52.150 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:52.150 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:52.151 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:52.151 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 89, 2022-06-29T16:05:52.151 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:52.151 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:52.152 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:52.152 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:52.152 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.152 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:52.152 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6816", 2022-06-29T16:05:52.153 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:52.153 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.153 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.153 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:52.154 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6825", 2022-06-29T16:05:52.154 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:52.154 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.154 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.155 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.155 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:52.155 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:52.155 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.156 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:52.156 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6826", 2022-06-29T16:05:52.156 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:52.156 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.156 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.157 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:52.157 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6827", 2022-06-29T16:05:52.157 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:52.157 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.158 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.158 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.158 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:52.158 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:52.158 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.159 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:52.159 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6830", 2022-06-29T16:05:52.159 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:52.159 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.160 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.160 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:52.160 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6831", 2022-06-29T16:05:52.160 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:52.161 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.161 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.161 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.161 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:52.161 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:52.162 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.162 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:52.162 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6828", 2022-06-29T16:05:52.162 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:52.163 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.163 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.163 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:52.163 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6829", 2022-06-29T16:05:52.163 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:52.164 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.164 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.164 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.164 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6825/12890", 2022-06-29T16:05:52.165 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6827/12890", 2022-06-29T16:05:52.165 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6831/12890", 2022-06-29T16:05:52.165 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6829/12890", 2022-06-29T16:05:52.165 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:52.166 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:52.166 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:52.166 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.166 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.166 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.167 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:05:52.167 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "ca43f136-6f69-4431-a157-fbe073fb585c", 2022-06-29T16:05:52.167 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:52.167 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:52.168 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:52.168 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:52.168 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:52.168 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:52.168 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:52.169 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 89, 2022-06-29T16:05:52.169 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:52.169 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:52.169 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:52.170 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:52.170 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.170 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:52.170 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6815", 2022-06-29T16:05:52.170 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:52.171 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.171 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.171 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:52.171 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6818", 2022-06-29T16:05:52.172 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:52.172 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.172 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.172 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.172 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:52.173 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:52.173 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.173 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:52.173 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6819", 2022-06-29T16:05:52.174 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:52.174 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.174 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.174 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:52.175 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6820", 2022-06-29T16:05:52.175 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:52.175 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.175 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.175 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.176 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:52.176 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:52.176 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.176 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:52.177 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6823", 2022-06-29T16:05:52.177 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:52.177 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.177 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.178 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:52.178 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6824", 2022-06-29T16:05:52.178 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:52.178 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.178 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.179 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.179 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:52.179 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:52.179 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.180 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:52.180 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6821", 2022-06-29T16:05:52.180 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:52.180 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.180 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.181 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:52.181 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6822", 2022-06-29T16:05:52.181 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:52.181 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.182 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.182 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.182 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6818/12892", 2022-06-29T16:05:52.182 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6820/12892", 2022-06-29T16:05:52.182 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6824/12892", 2022-06-29T16:05:52.183 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6822/12892", 2022-06-29T16:05:52.183 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:52.183 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:52.183 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:52.184 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.184 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.184 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.184 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:05:52.185 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "74fac45f-5f07-44ec-a9cb-2644c123fd4a", 2022-06-29T16:05:52.185 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:52.185 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:52.185 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:52.185 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:52.186 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:52.186 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:52.186 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:52.186 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 89, 2022-06-29T16:05:52.187 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:52.187 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:52.187 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:52.187 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:52.187 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.188 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:52.188 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6808", 2022-06-29T16:05:52.188 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:52.188 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.189 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.189 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:52.189 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6809", 2022-06-29T16:05:52.189 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:52.189 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.190 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.190 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.190 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:52.190 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:52.191 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.191 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:52.191 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6810", 2022-06-29T16:05:52.191 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:52.192 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.192 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.192 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:52.192 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6811", 2022-06-29T16:05:52.192 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:52.193 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.193 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.193 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.193 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:52.193 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:52.194 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.194 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:52.194 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6814", 2022-06-29T16:05:52.195 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:52.195 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.195 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.195 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:52.195 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6817", 2022-06-29T16:05:52.196 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:52.196 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.196 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.196 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.197 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:52.197 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:52.197 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.197 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:52.197 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6812", 2022-06-29T16:05:52.198 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:52.198 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.198 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.198 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:52.199 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6813", 2022-06-29T16:05:52.199 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:52.199 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.199 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.199 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.200 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6809/12891", 2022-06-29T16:05:52.200 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6811/12891", 2022-06-29T16:05:52.200 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6817/12891", 2022-06-29T16:05:52.201 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6813/12891", 2022-06-29T16:05:52.201 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:52.201 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:52.201 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:52.202 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.202 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.202 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.202 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:05:52.203 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "2581d351-973a-4b4e-85ca-3658f0068765", 2022-06-29T16:05:52.203 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:52.203 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:52.203 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:52.203 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:52.204 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:52.204 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:52.204 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:52.204 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 89, 2022-06-29T16:05:52.205 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:52.205 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:52.205 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:52.205 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:52.205 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.206 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:52.206 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6800", 2022-06-29T16:05:52.206 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:52.206 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.207 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.207 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:52.207 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6801", 2022-06-29T16:05:52.207 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:52.207 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.208 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.208 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.208 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:52.208 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:52.209 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.209 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:52.209 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6802", 2022-06-29T16:05:52.209 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:52.209 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.210 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.210 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:52.210 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6803", 2022-06-29T16:05:52.210 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:52.211 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.211 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.211 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.211 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:52.212 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:52.212 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.212 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:52.212 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6806", 2022-06-29T16:05:52.212 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:52.213 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.213 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.213 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:52.213 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6807", 2022-06-29T16:05:52.213 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:52.214 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.214 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.214 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.214 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:52.215 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:52.215 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.215 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:52.215 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6804", 2022-06-29T16:05:52.216 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:52.216 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.216 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.216 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:52.216 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6805", 2022-06-29T16:05:52.217 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:05:52.217 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.217 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.217 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.218 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6801/12889", 2022-06-29T16:05:52.218 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6803/12889", 2022-06-29T16:05:52.218 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6807/12889", 2022-06-29T16:05:52.218 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6805/12889", 2022-06-29T16:05:52.219 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:52.219 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:52.219 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:52.219 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.219 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.220 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.220 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:05:52.220 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "3d234db9-87cd-4fe9-814f-0240a287d8b4", 2022-06-29T16:05:52.220 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:52.221 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:52.221 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:52.221 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:52.221 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:52.221 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:52.222 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:52.222 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 88, 2022-06-29T16:05:52.222 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:52.222 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:52.223 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:52.223 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:52.223 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.223 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:52.223 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6808", 2022-06-29T16:05:52.224 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:52.224 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.224 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.225 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:52.225 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6810", 2022-06-29T16:05:52.225 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:52.225 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.225 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.226 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.226 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:52.226 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:52.226 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.227 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:52.227 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6812", 2022-06-29T16:05:52.227 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:52.227 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.228 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.228 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:52.228 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6814", 2022-06-29T16:05:52.228 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:52.228 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.229 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.229 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.229 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:52.229 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:52.230 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.230 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:52.230 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6823", 2022-06-29T16:05:52.230 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:52.230 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.231 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.231 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:52.231 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6825", 2022-06-29T16:05:52.231 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:52.232 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.232 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.232 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.232 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:52.232 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:52.233 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.233 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:52.233 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6817", 2022-06-29T16:05:52.233 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:52.234 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.234 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.234 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:52.234 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6820", 2022-06-29T16:05:52.234 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:05:52.235 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.235 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.235 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.236 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6810/13249", 2022-06-29T16:05:52.236 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6814/13249", 2022-06-29T16:05:52.236 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6825/13249", 2022-06-29T16:05:52.236 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6820/13249", 2022-06-29T16:05:52.236 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:52.237 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:52.237 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:52.237 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.237 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.238 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.238 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:05:52.238 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "e4bf1104-ff5c-438b-b608-8cee31c163f6", 2022-06-29T16:05:52.238 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:52.239 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:52.239 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:52.239 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:52.239 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:52.240 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:52.240 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:52.240 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 78, 2022-06-29T16:05:52.240 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:52.240 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:52.241 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:52.241 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:52.241 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.241 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:52.242 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6809", 2022-06-29T16:05:52.242 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:52.242 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.242 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.242 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:52.243 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6811", 2022-06-29T16:05:52.243 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:52.243 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.243 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.244 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.244 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:52.244 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:52.244 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.245 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:52.245 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6813", 2022-06-29T16:05:52.245 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:52.245 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.245 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.246 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:52.246 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6815", 2022-06-29T16:05:52.246 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:52.246 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.247 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.247 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.247 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:52.247 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:52.248 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.248 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:52.248 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6824", 2022-06-29T16:05:52.248 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:52.249 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.249 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.249 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:52.249 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6827", 2022-06-29T16:05:52.250 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:52.250 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.250 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.250 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.250 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:52.251 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:52.251 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.251 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:52.251 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6818", 2022-06-29T16:05:52.252 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:52.252 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.252 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.252 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:52.252 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6821", 2022-06-29T16:05:52.253 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:05:52.253 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.253 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.253 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.254 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6811/13248", 2022-06-29T16:05:52.254 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6815/13248", 2022-06-29T16:05:52.254 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6827/13248", 2022-06-29T16:05:52.255 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6821/13248", 2022-06-29T16:05:52.255 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:52.255 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:52.255 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:52.255 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.256 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.256 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.256 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:05:52.256 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "8564dd25-27ab-4203-b54e-af6d3aa05857", 2022-06-29T16:05:52.257 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:52.257 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:52.257 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:52.257 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:52.257 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:52.258 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:52.258 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:52.258 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 88, 2022-06-29T16:05:52.258 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:52.259 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:52.259 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:52.259 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:52.259 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.259 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:52.260 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6816", 2022-06-29T16:05:52.260 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:52.260 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.260 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.261 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:52.261 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6819", 2022-06-29T16:05:52.261 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:52.261 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.261 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.262 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.262 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:52.262 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:52.262 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.263 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:52.263 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6822", 2022-06-29T16:05:52.263 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:52.263 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.264 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.264 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:52.264 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6826", 2022-06-29T16:05:52.264 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:52.265 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.265 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.265 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.265 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:52.265 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:52.266 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.266 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:52.266 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6830", 2022-06-29T16:05:52.266 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:52.267 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.267 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.267 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:52.267 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6831", 2022-06-29T16:05:52.267 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:52.268 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.268 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.268 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.268 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:52.269 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:52.269 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.269 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:52.269 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6828", 2022-06-29T16:05:52.269 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:52.270 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.270 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.270 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:52.270 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6829", 2022-06-29T16:05:52.271 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:05:52.271 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.271 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.271 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.272 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6819/13250", 2022-06-29T16:05:52.272 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6826/13250", 2022-06-29T16:05:52.272 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6831/13250", 2022-06-29T16:05:52.272 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6829/13250", 2022-06-29T16:05:52.272 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:52.273 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:52.273 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:52.273 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.273 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.274 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.274 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:05:52.274 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "bd513a9c-3b19-4314-9897-d300b5c897bb", 2022-06-29T16:05:52.274 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:52.275 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:52.275 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:52.275 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:52.275 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:52.275 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:52.276 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:52.276 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 89, 2022-06-29T16:05:52.276 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:52.276 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:52.276 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:52.277 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:52.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.277 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:52.277 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6800", 2022-06-29T16:05:52.278 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:52.278 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.278 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.278 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:52.278 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6801", 2022-06-29T16:05:52.279 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:52.279 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.279 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.279 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.280 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:52.280 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:52.280 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.280 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:52.280 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6802", 2022-06-29T16:05:52.281 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:52.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.281 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.281 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:52.282 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6803", 2022-06-29T16:05:52.282 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:52.282 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.282 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.282 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.283 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:52.283 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:52.283 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.283 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:52.284 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6806", 2022-06-29T16:05:52.284 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:52.284 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.285 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:52.285 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6807", 2022-06-29T16:05:52.285 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:52.286 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.286 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.286 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.286 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:52.286 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:52.287 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.287 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:52.287 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6804", 2022-06-29T16:05:52.288 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:52.288 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.288 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.289 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:52.289 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6805", 2022-06-29T16:05:52.289 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:05:52.289 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.290 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.290 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.290 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6801/13247", 2022-06-29T16:05:52.291 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6803/13247", 2022-06-29T16:05:52.291 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6807/13247", 2022-06-29T16:05:52.291 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6805/13247", 2022-06-29T16:05:52.291 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:52.292 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:52.292 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:52.292 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:52.293 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.293 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:05:52.293 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_xinfo": [ 2022-06-29T16:05:52.294 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.294 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:05:52.294 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:52.295 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:52.295 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:52.295 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:52.295 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:52.295 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.891342+0000", 2022-06-29T16:05:52.296 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:52.296 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.296 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.296 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:05:52.297 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:52.297 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:52.297 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:52.297 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:52.297 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:52.298 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.900866+0000", 2022-06-29T16:05:52.298 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:52.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.299 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:05:52.299 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:52.299 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:52.299 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:52.299 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:52.300 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:52.300 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.912755+0000", 2022-06-29T16:05:52.300 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:52.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.301 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.301 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:05:52.301 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:52.301 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:52.301 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:52.302 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:52.302 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:52.302 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.864332+0000", 2022-06-29T16:05:52.302 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:52.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.303 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:05:52.304 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:52.304 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:52.304 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:52.304 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:52.304 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:52.305 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.863756+0000", 2022-06-29T16:05:52.305 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:52.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.306 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:05:52.306 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:52.306 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:52.306 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:52.307 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:52.307 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:52.307 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.869041+0000", 2022-06-29T16:05:52.307 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:52.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.308 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:05:52.308 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:52.309 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:52.309 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:52.309 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:52.309 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:52.309 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.895043+0000", 2022-06-29T16:05:52.310 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:52.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:52.310 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:05:52.311 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:05:52.311 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:05:52.311 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:05:52.311 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:05:52.311 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:05:52.312 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.889491+0000", 2022-06-29T16:05:52.312 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:05:52.312 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.312 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:05:52.313 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap": [], 2022-06-29T16:05:52.313 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap_items": [], 2022-06-29T16:05:52.313 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_temp": [], 2022-06-29T16:05:52.313 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_temp": [], 2022-06-29T16:05:52.313 INFO:teuthology.orchestra.run.smithi060.stdout: "blacklist": {}, 2022-06-29T16:05:52.314 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profiles": { 2022-06-29T16:05:52.314 INFO:teuthology.orchestra.run.smithi060.stdout: "default": { 2022-06-29T16:05:52.314 INFO:teuthology.orchestra.run.smithi060.stdout: "crush-failure-domain": "osd", 2022-06-29T16:05:52.315 INFO:teuthology.orchestra.run.smithi060.stdout: "k": "2", 2022-06-29T16:05:52.315 INFO:teuthology.orchestra.run.smithi060.stdout: "m": "1", 2022-06-29T16:05:52.315 INFO:teuthology.orchestra.run.smithi060.stdout: "plugin": "jerasure", 2022-06-29T16:05:52.315 INFO:teuthology.orchestra.run.smithi060.stdout: "ruleset-failure-domain": "osd", 2022-06-29T16:05:52.315 INFO:teuthology.orchestra.run.smithi060.stdout: "technique": "reed_sol_van" 2022-06-29T16:05:52.316 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:52.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:52.316 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps_queue": [], 2022-06-29T16:05:52.316 INFO:teuthology.orchestra.run.smithi060.stdout: "new_removed_snaps": [], 2022-06-29T16:05:52.317 INFO:teuthology.orchestra.run.smithi060.stdout: "new_purged_snaps": [], 2022-06-29T16:05:52.317 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_node_flags": {}, 2022-06-29T16:05:52.317 INFO:teuthology.orchestra.run.smithi060.stdout: "device_class_flags": {} 2022-06-29T16:05:52.317 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:05:52.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.070+0000 7f9353868700 1 -- 172.21.15.60:0/2001334289 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f93300539e0 msgr2=0x7f9330055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:52.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.070+0000 7f9353868700 1 --2- 172.21.15.60:0/2001334289 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f93300539e0 0x7f9330055ea0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7f9344005000 tx=0x7f934c0b19c0).stop 2022-06-29T16:05:52.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.070+0000 7f9353868700 1 -- 172.21.15.60:0/2001334289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f934c145c90 msgr2=0x7f934c146140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:52.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.070+0000 7f9353868700 1 --2- 172.21.15.60:0/2001334289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f934c145c90 0x7f934c146140 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 rx=0x7f933c00b000 tx=0x7f933c008040).stop 2022-06-29T16:05:52.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.070+0000 7f9353868700 1 -- 172.21.15.60:0/2001334289 shutdown_connections 2022-06-29T16:05:52.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.070+0000 7f9353868700 1 --2- 172.21.15.60:0/2001334289 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f93300539e0 0x7f9330055ea0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:52.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.070+0000 7f9353868700 1 --2- 172.21.15.60:0/2001334289 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f934c145c90 0x7f934c146140 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:52.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.070+0000 7f9353868700 1 --2- 172.21.15.60:0/2001334289 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f934c145500 0x7f934c0afa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:52.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.070+0000 7f9353868700 1 --2- 172.21.15.60:0/2001334289 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f932c004500 0x7f934c0af530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:52.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.070+0000 7f9353868700 1 -- 172.21.15.60:0/2001334289 >> 172.21.15.60:0/2001334289 conn(0x7f934c062ce0 msgr2=0x7f934c13d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:52.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.070+0000 7f9353868700 1 -- 172.21.15.60:0/2001334289 shutdown_connections 2022-06-29T16:05:52.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.070+0000 7f9353868700 1 -- 172.21.15.60:0/2001334289 wait complete. 2022-06-29T16:05:52.323 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:05:52.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.426+0000 7fb0a5180700 1 Processor -- start 2022-06-29T16:05:52.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.426+0000 7fb0a5180700 1 -- start start 2022-06-29T16:05:52.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.426+0000 7fb0a5180700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0a005fb00 0x7fb0a0146d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:52.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.426+0000 7fb0a5180700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb0a00b0df0 0x7fb0a00ae040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:52.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.426+0000 7fb0a5180700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a00aca20 con 0x7fb0a00affd0 2022-06-29T16:05:52.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.426+0000 7fb0a5180700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0a005fee0 con 0x7fb0a00b0df0 2022-06-29T16:05:52.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.426+0000 7fb0a5180700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a00b1470 con 0x7fb0a005fb00 2022-06-29T16:05:52.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.426+0000 7fb09f7fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb0a00b0df0 0x7fb0a00ae040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:52.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.426+0000 7fb09ffff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0a005fb00 0x7fb0a0146d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:52.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.426+0000 7fb09f7fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb0a00b0df0 0x7fb0a00ae040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51034/0 (socket says 172.21.15.60:51034) 2022-06-29T16:05:52.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.426+0000 7fb09ffff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0a005fb00 0x7fb0a0146d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:35892/0 (socket says 172.21.15.60:35892) 2022-06-29T16:05:52.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.426+0000 7fb09ffff700 1 -- 172.21.15.60:0/1081069316 learned_addr learned my addr 172.21.15.60:0/1081069316 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:52.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.426+0000 7fb0a497f700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fb0a00affd0 0x7fb0a0142fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:35134/0 (socket says 172.21.15.60:35134) 2022-06-29T16:05:52.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.426+0000 7fb09ffff700 1 -- 172.21.15.60:0/1081069316 >> v2:172.21.15.60:3300/0 conn(0x7fb0a00b0df0 msgr2=0x7fb0a00ae040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:52.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.426+0000 7fb09ffff700 1 --2- 172.21.15.60:0/1081069316 >> v2:172.21.15.60:3300/0 conn(0x7fb0a00b0df0 0x7fb0a00ae040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:52.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.426+0000 7fb09ffff700 1 -- 172.21.15.60:0/1081069316 >> v1:172.21.15.92:6789/0 conn(0x7fb0a00affd0 legacy=0x7fb0a0142fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:52.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.426+0000 7fb09effd700 1 -- 172.21.15.60:0/1081069316 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 649546764 0 0) 0x7fb0a00aca20 con 0x7fb0a00affd0 2022-06-29T16:05:52.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.426+0000 7fb09ffff700 1 -- 172.21.15.60:0/1081069316 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a00b1710 con 0x7fb0a005fb00 2022-06-29T16:05:52.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.426+0000 7fb09ffff700 1 --2- 172.21.15.60:0/1081069316 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0a005fb00 0x7fb0a0146d10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fb090002f70 tx=0x7fb090006eb0).ready entity=mon.2 client_cookie=8bf085b307f399d4 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:52.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.426+0000 7fb09effd700 1 -- 172.21.15.60:0/1081069316 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb09000d570 con 0x7fb0a005fb00 2022-06-29T16:05:52.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.426+0000 7fb09effd700 1 -- 172.21.15.60:0/1081069316 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb090000e90 con 0x7fb0a005fb00 2022-06-29T16:05:52.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.426+0000 7fb09effd700 1 -- 172.21.15.60:0/1081069316 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb09000d8c0 con 0x7fb0a005fb00 2022-06-29T16:05:52.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.426+0000 7fb0a5180700 1 -- 172.21.15.60:0/1081069316 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0a005fb00 msgr2=0x7fb0a0146d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:52.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.426+0000 7fb0a5180700 1 --2- 172.21.15.60:0/1081069316 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0a005fb00 0x7fb0a0146d10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fb090002f70 tx=0x7fb090006eb0).stop 2022-06-29T16:05:52.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.426+0000 7fb0a5180700 1 -- 172.21.15.60:0/1081069316 shutdown_connections 2022-06-29T16:05:52.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.426+0000 7fb0a5180700 1 --2- 172.21.15.60:0/1081069316 >> v2:172.21.15.60:3300/0 conn(0x7fb0a00b0df0 0x7fb0a00ae040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:52.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.426+0000 7fb0a5180700 1 --2- 172.21.15.60:0/1081069316 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0a005fb00 0x7fb0a0146d10 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:52.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.426+0000 7fb0a5180700 1 -- 172.21.15.60:0/1081069316 >> 172.21.15.60:0/1081069316 conn(0x7fb0a0063a60 msgr2=0x7fb0a005f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:52.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.426+0000 7fb0a5180700 1 -- 172.21.15.60:0/1081069316 shutdown_connections 2022-06-29T16:05:52.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.426+0000 7fb0a5180700 1 -- 172.21.15.60:0/1081069316 wait complete. 2022-06-29T16:05:52.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.430+0000 7fb0a5180700 1 Processor -- start 2022-06-29T16:05:52.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.430+0000 7fb0a5180700 1 -- start start 2022-06-29T16:05:52.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.430+0000 7fb0a5180700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0a005fb00 0x7fb0a0148a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:52.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.430+0000 7fb0a5180700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb0a00affd0 0x7fb0a0144e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:52.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.430+0000 7fb0a5180700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0a00b0df0 0x7fb0a0151c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:52.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.430+0000 7fb0a5180700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a0146080 con 0x7fb0a00affd0 2022-06-29T16:05:52.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.430+0000 7fb0a5180700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a0152e60 con 0x7fb0a00b0df0 2022-06-29T16:05:52.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.430+0000 7fb0a5180700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a0112db0 con 0x7fb0a005fb00 2022-06-29T16:05:52.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.430+0000 7fb09f7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb0a00affd0 0x7fb0a0144e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:52.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.430+0000 7fb09f7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb0a00affd0 0x7fb0a0144e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51038/0 (socket says 172.21.15.60:51038) 2022-06-29T16:05:52.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.430+0000 7fb09f7fe700 1 -- 172.21.15.60:0/18199939 learned_addr learned my addr 172.21.15.60:0/18199939 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:52.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.430+0000 7fb09ffff700 1 --2- 172.21.15.60:0/18199939 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0a005fb00 0x7fb0a0148a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:52.448 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.430+0000 7fb0a497f700 1 --2- 172.21.15.60:0/18199939 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0a00b0df0 0x7fb0a0151c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:52.448 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.430+0000 7fb09f7fe700 1 -- 172.21.15.60:0/18199939 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0a005fb00 msgr2=0x7fb0a0148a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:52.448 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.430+0000 7fb09f7fe700 1 --2- 172.21.15.60:0/18199939 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0a005fb00 0x7fb0a0148a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:52.448 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.430+0000 7fb09f7fe700 1 -- 172.21.15.60:0/18199939 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0a00b0df0 msgr2=0x7fb0a0151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:52.449 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.430+0000 7fb09f7fe700 1 --2- 172.21.15.60:0/18199939 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0a00b0df0 0x7fb0a0151c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:52.449 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.430+0000 7fb09f7fe700 1 -- 172.21.15.60:0/18199939 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb09000b1e0 con 0x7fb0a00affd0 2022-06-29T16:05:52.449 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.434+0000 7fb09f7fe700 1 --2- 172.21.15.60:0/18199939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb0a00affd0 0x7fb0a0144e50 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 rx=0x7fb09400c000 tx=0x7fb094006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:52.449 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.434+0000 7fb09d7fa700 1 -- 172.21.15.60:0/18199939 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb094011750 con 0x7fb0a00affd0 2022-06-29T16:05:52.449 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.434+0000 7fb09d7fa700 1 -- 172.21.15.60:0/18199939 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0940118b0 con 0x7fb0a00affd0 2022-06-29T16:05:52.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.434+0000 7fb0a5180700 1 -- 172.21.15.60:0/18199939 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0a01462b0 con 0x7fb0a00affd0 2022-06-29T16:05:52.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.434+0000 7fb09d7fa700 1 -- 172.21.15.60:0/18199939 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb094011c10 con 0x7fb0a00affd0 2022-06-29T16:05:52.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.434+0000 7fb0a5180700 1 -- 172.21.15.60:0/18199939 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0a0147120 con 0x7fb0a00affd0 2022-06-29T16:05:52.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.434+0000 7fb0a5180700 1 -- 172.21.15.60:0/18199939 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0a0145340 con 0x7fb0a00affd0 2022-06-29T16:05:52.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.434+0000 7fb09d7fa700 1 -- 172.21.15.60:0/18199939 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb094011d70 con 0x7fb0a00affd0 2022-06-29T16:05:52.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.438+0000 7fb09d7fa700 1 --2- 172.21.15.60:0/18199939 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb080053b30 0x7fb080055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:52.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.438+0000 7fb09d7fa700 1 -- 172.21.15.60:0/18199939 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb09406c200 con 0x7fb0a00affd0 2022-06-29T16:05:52.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.438+0000 7fb09ffff700 1 --2- 172.21.15.60:0/18199939 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb080053b30 0x7fb080055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:52.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.438+0000 7fb09ffff700 1 --2- 172.21.15.60:0/18199939 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb080053b30 0x7fb080055ff0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7fb09001c000 tx=0x7fb090015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:52.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.446+0000 7fb09d7fa700 1 -- 172.21.15.60:0/18199939 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb094043010 con 0x7fb0a00affd0 2022-06-29T16:05:52.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.610+0000 7fb0a5180700 1 -- 172.21.15.60:0/18199939 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0a00573d0 con 0x7fb0a00affd0 2022-06-29T16:05:52.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.610+0000 7fb09d7fa700 1 -- 172.21.15.60:0/18199939 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2140 (secure 0 0 0) 0x7fb094046630 con 0x7fb0a00affd0 2022-06-29T16:05:52.620 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:52.620 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":41,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6182,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1306898602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1306898602},{"type":"v1","addr":"172.21.15.60:6833","nonce":1306898602}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":39}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:05:40.369985+0000","modified":"2022-06-29T16:05:41.373815+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6185":{"gid":6185,"name":"b","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/2057420852","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2057420852},{"type":"v1","addr":"172.21.15.92:6835","nonce":2057420852}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-06-29T16:05:52.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.614+0000 7fb0a5180700 1 -- 172.21.15.60:0/18199939 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb080053b30 msgr2=0x7fb080055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:52.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.614+0000 7fb0a5180700 1 --2- 172.21.15.60:0/18199939 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb080053b30 0x7fb080055ff0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7fb09001c000 tx=0x7fb090015000).stop 2022-06-29T16:05:52.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.614+0000 7fb0a5180700 1 -- 172.21.15.60:0/18199939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb0a00affd0 msgr2=0x7fb0a0144e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:52.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.614+0000 7fb0a5180700 1 --2- 172.21.15.60:0/18199939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb0a00affd0 0x7fb0a0144e50 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 rx=0x7fb09400c000 tx=0x7fb094006df0).stop 2022-06-29T16:05:52.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.614+0000 7fb0a5180700 1 -- 172.21.15.60:0/18199939 shutdown_connections 2022-06-29T16:05:52.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.614+0000 7fb0a5180700 1 --2- 172.21.15.60:0/18199939 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb080053b30 0x7fb080055ff0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:52.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.614+0000 7fb0a5180700 1 --2- 172.21.15.60:0/18199939 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0a00b0df0 0x7fb0a0151c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:52.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.614+0000 7fb0a5180700 1 --2- 172.21.15.60:0/18199939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb0a00affd0 0x7fb0a0144e50 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:52.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.614+0000 7fb0a5180700 1 --2- 172.21.15.60:0/18199939 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0a005fb00 0x7fb0a0148a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:52.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.614+0000 7fb0a5180700 1 -- 172.21.15.60:0/18199939 >> 172.21.15.60:0/18199939 conn(0x7fb0a0063a60 msgr2=0x7fb0a00b4740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:52.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.614+0000 7fb0a5180700 1 -- 172.21.15.60:0/18199939 shutdown_connections 2022-06-29T16:05:52.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.614+0000 7fb0a5180700 1 -- 172.21.15.60:0/18199939 wait complete. 2022-06-29T16:05:52.625 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 41 2022-06-29T16:05:52.633 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-06-29T16:05:52.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.730+0000 7f8d62f7e700 1 Processor -- start 2022-06-29T16:05:52.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.730+0000 7f8d62f7e700 1 -- start start 2022-06-29T16:05:52.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.730+0000 7f8d62f7e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8d5c0b5950 0x7f8d5c067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:52.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.730+0000 7f8d62f7e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8d5c063ad0 0x7f8d5c14af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:52.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.730+0000 7f8d62f7e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d5c0acba0 con 0x7f8d5c05bd20 2022-06-29T16:05:52.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.730+0000 7f8d62f7e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d5c067220 con 0x7f8d5c063ad0 2022-06-29T16:05:52.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.730+0000 7f8d62f7e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c0b5d30 con 0x7f8d5c0b5950 2022-06-29T16:05:52.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.730+0000 7f8d5bfff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8d5c063ad0 0x7f8d5c14af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:52.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.730+0000 7f8d5bfff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8d5c063ad0 0x7f8d5c14af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51044/0 (socket says 172.21.15.60:51044) 2022-06-29T16:05:52.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.730+0000 7f8d5bfff700 1 -- 172.21.15.60:0/898464249 learned_addr learned my addr 172.21.15.60:0/898464249 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:52.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.730+0000 7f8d60d1a700 1 --2- 172.21.15.60:0/898464249 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8d5c0b5950 0x7f8d5c067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:52.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.730+0000 7f8d5bfff700 1 -- 172.21.15.60:0/898464249 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8d5c0b5950 msgr2=0x7f8d5c067680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:52.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.730+0000 7f8d5bfff700 1 --2- 172.21.15.60:0/898464249 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8d5c0b5950 0x7f8d5c067680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:52.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.730+0000 7f8d5bfff700 1 -- 172.21.15.60:0/898464249 >> v1:172.21.15.92:6789/0 conn(0x7f8d5c05bd20 legacy=0x7f8d5c05c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:52.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.730+0000 7f8d5bfff700 1 -- 172.21.15.60:0/898464249 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5c0ace00 con 0x7f8d5c063ad0 2022-06-29T16:05:52.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.730+0000 7f8d5b7fe700 1 -- 172.21.15.60:0/898464249 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4201690136 0 0) 0x7f8d5c0acba0 con 0x7f8d5c05bd20 2022-06-29T16:05:52.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.734+0000 7f8d5bfff700 1 --2- 172.21.15.60:0/898464249 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d5c063ad0 0x7f8d5c14af10 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0x7f8d4c002f70 tx=0x7f8d4c006eb0).ready entity=mon.0 client_cookie=7e2e1b7091608b57 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:52.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.734+0000 7f8d5b7fe700 1 -- 172.21.15.60:0/898464249 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d4c00d570 con 0x7f8d5c063ad0 2022-06-29T16:05:52.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.734+0000 7f8d5b7fe700 1 -- 172.21.15.60:0/898464249 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d4c000e90 con 0x7f8d5c063ad0 2022-06-29T16:05:52.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.734+0000 7f8d5b7fe700 1 -- 172.21.15.60:0/898464249 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d4c00d8c0 con 0x7f8d5c063ad0 2022-06-29T16:05:52.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.734+0000 7f8d62f7e700 1 -- 172.21.15.60:0/898464249 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d5c063ad0 msgr2=0x7f8d5c14af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:52.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.734+0000 7f8d62f7e700 1 --2- 172.21.15.60:0/898464249 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d5c063ad0 0x7f8d5c14af10 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0x7f8d4c002f70 tx=0x7f8d4c006eb0).stop 2022-06-29T16:05:52.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.734+0000 7f8d62f7e700 1 -- 172.21.15.60:0/898464249 shutdown_connections 2022-06-29T16:05:52.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.734+0000 7f8d62f7e700 1 --2- 172.21.15.60:0/898464249 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d5c063ad0 0x7f8d5c14af10 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:52.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.734+0000 7f8d62f7e700 1 --2- 172.21.15.60:0/898464249 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8d5c0b5950 0x7f8d5c067680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:52.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.734+0000 7f8d62f7e700 1 -- 172.21.15.60:0/898464249 >> 172.21.15.60:0/898464249 conn(0x7f8d5c0632c0 msgr2=0x7f8d5c0b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:52.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.734+0000 7f8d62f7e700 1 -- 172.21.15.60:0/898464249 shutdown_connections 2022-06-29T16:05:52.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.734+0000 7f8d62f7e700 1 -- 172.21.15.60:0/898464249 wait complete. 2022-06-29T16:05:52.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.734+0000 7f8d62f7e700 1 Processor -- start 2022-06-29T16:05:52.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.734+0000 7f8d62f7e700 1 -- start start 2022-06-29T16:05:52.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.738+0000 7f8d62f7e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d5c05bd20 0x7f8d5c1593c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:52.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.738+0000 7f8d62f7e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8d5c063ad0 0x7f8d5c14dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:52.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.738+0000 7f8d62f7e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d5c0b5950 0x7f8d5c116bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:52.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.738+0000 7f8d62f7e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c0aec60 con 0x7f8d5c0b5950 2022-06-29T16:05:52.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.738+0000 7f8d62f7e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c0ad8c0 con 0x7f8d5c05bd20 2022-06-29T16:05:52.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.738+0000 7f8d62f7e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c0adb60 con 0x7f8d5c063ad0 2022-06-29T16:05:52.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.738+0000 7f8d60d1a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d5c05bd20 0x7f8d5c1593c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:52.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.738+0000 7f8d6151b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d5c0b5950 0x7f8d5c116bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:52.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.738+0000 7f8d5bfff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8d5c063ad0 0x7f8d5c14dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:52.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.738+0000 7f8d60d1a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d5c05bd20 0x7f8d5c1593c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:57080/0 (socket says 172.21.15.60:57080) 2022-06-29T16:05:52.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.738+0000 7f8d60d1a700 1 -- 172.21.15.60:0/1745327893 learned_addr learned my addr 172.21.15.60:0/1745327893 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:52.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.738+0000 7f8d6151b700 1 -- 172.21.15.60:0/1745327893 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8d5c063ad0 msgr2=0x7f8d5c14dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:52.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.738+0000 7f8d6151b700 1 --2- 172.21.15.60:0/1745327893 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8d5c063ad0 0x7f8d5c14dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:52.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.738+0000 7f8d6151b700 1 -- 172.21.15.60:0/1745327893 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d5c05bd20 msgr2=0x7f8d5c1593c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:52.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.738+0000 7f8d6151b700 1 --2- 172.21.15.60:0/1745327893 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d5c05bd20 0x7f8d5c1593c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:52.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.738+0000 7f8d6151b700 1 -- 172.21.15.60:0/1745327893 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d4c00b1e0 con 0x7f8d5c0b5950 2022-06-29T16:05:52.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.738+0000 7f8d6151b700 1 --2- 172.21.15.60:0/1745327893 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d5c0b5950 0x7f8d5c116bd0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 rx=0x7f8d54001250 tx=0x7f8d54007d30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:52.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.738+0000 7f8d59ffb700 1 -- 172.21.15.60:0/1745327893 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d5400f470 con 0x7f8d5c0b5950 2022-06-29T16:05:52.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.738+0000 7f8d59ffb700 1 -- 172.21.15.60:0/1745327893 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d5400f5d0 con 0x7f8d5c0b5950 2022-06-29T16:05:52.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.738+0000 7f8d59ffb700 1 -- 172.21.15.60:0/1745327893 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d5400f930 con 0x7f8d5c0b5950 2022-06-29T16:05:52.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.738+0000 7f8d62f7e700 1 -- 172.21.15.60:0/1745327893 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d5c0ae9c0 con 0x7f8d5c0b5950 2022-06-29T16:05:52.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.738+0000 7f8d62f7e700 1 -- 172.21.15.60:0/1745327893 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d5c0ad500 con 0x7f8d5c0b5950 2022-06-29T16:05:52.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.742+0000 7f8d59ffb700 1 -- 172.21.15.60:0/1745327893 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8d54019030 con 0x7f8d5c0b5950 2022-06-29T16:05:52.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.742+0000 7f8d62f7e700 1 -- 172.21.15.60:0/1745327893 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d5c0573d0 con 0x7f8d5c0b5950 2022-06-29T16:05:52.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.742+0000 7f8d59ffb700 1 --2- 172.21.15.60:0/1745327893 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8d40053a70 0x7f8d40055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:52.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.742+0000 7f8d59ffb700 1 -- 172.21.15.60:0/1745327893 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8d5406b420 con 0x7f8d5c0b5950 2022-06-29T16:05:52.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.742+0000 7f8d60d1a700 1 --2- 172.21.15.60:0/1745327893 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8d40053a70 0x7f8d40055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:52.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.742+0000 7f8d60d1a700 1 --2- 172.21.15.60:0/1745327893 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8d40053a70 0x7f8d40055f30 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f8d4400b000 tx=0x7f8d44006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:52.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.746+0000 7f8d59ffb700 1 -- 172.21.15.60:0/1745327893 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8d54042350 con 0x7f8d5c0b5950 2022-06-29T16:05:52.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:52.914+0000 7f8d62f7e700 1 -- 172.21.15.60:0/1745327893 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f8d5c116930 con 0x7f8d5c0b5950 2022-06-29T16:05:53.896 INFO:tasks.ceph.mon.a.smithi060.stderr:2022-06-29T16:05:53.886+0000 7fcc1a376700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-06-29T16:05:53.903 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:53.894+0000 7f8d59ffb700 1 -- 172.21.15.60:0/1745327893 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v42) v1 ==== 161+0+0 (secure 0 0 0) 0x7f8d54045970 con 0x7f8d5c0b5950 2022-06-29T16:05:53.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:53.898+0000 7f8d62f7e700 1 -- 172.21.15.60:0/1745327893 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8d40053a70 msgr2=0x7f8d40055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:53.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:53.898+0000 7f8d62f7e700 1 --2- 172.21.15.60:0/1745327893 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8d40053a70 0x7f8d40055f30 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f8d4400b000 tx=0x7f8d44006df0).stop 2022-06-29T16:05:53.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:53.898+0000 7f8d62f7e700 1 -- 172.21.15.60:0/1745327893 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d5c0b5950 msgr2=0x7f8d5c116bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:53.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:53.898+0000 7f8d62f7e700 1 --2- 172.21.15.60:0/1745327893 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d5c0b5950 0x7f8d5c116bd0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 rx=0x7f8d54001250 tx=0x7f8d54007d30).stop 2022-06-29T16:05:53.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:53.898+0000 7f8d62f7e700 1 -- 172.21.15.60:0/1745327893 shutdown_connections 2022-06-29T16:05:53.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:53.898+0000 7f8d62f7e700 1 --2- 172.21.15.60:0/1745327893 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8d40053a70 0x7f8d40055f30 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:53.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:53.898+0000 7f8d62f7e700 1 --2- 172.21.15.60:0/1745327893 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d5c0b5950 0x7f8d5c116bd0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:53.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:53.898+0000 7f8d62f7e700 1 --2- 172.21.15.60:0/1745327893 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8d5c063ad0 0x7f8d5c14dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:53.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:53.898+0000 7f8d62f7e700 1 --2- 172.21.15.60:0/1745327893 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d5c05bd20 0x7f8d5c1593c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:53.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:53.898+0000 7f8d62f7e700 1 -- 172.21.15.60:0/1745327893 >> 172.21.15.60:0/1745327893 conn(0x7f8d5c0632c0 msgr2=0x7f8d5c145f50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:53.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:53.898+0000 7f8d62f7e700 1 -- 172.21.15.60:0/1745327893 shutdown_connections 2022-06-29T16:05:53.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:53.898+0000 7f8d62f7e700 1 -- 172.21.15.60:0/1745327893 wait complete. 2022-06-29T16:05:53.911 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-06-29T16:05:53.922 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:05:53.936 INFO:tasks.ceph.mds.b.smithi092.stdout:starting mds.b at 2022-06-29T16:05:54.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.022+0000 7efc5150f700 1 Processor -- start 2022-06-29T16:05:54.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.026+0000 7efc5150f700 1 -- start start 2022-06-29T16:05:54.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.026+0000 7efc5150f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc4c0b3d00 0x7efc4c144580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:54.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.026+0000 7efc5150f700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7efc4c0b1f60 0x7efc4c0aca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:54.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.026+0000 7efc5150f700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc4c0b40e0 con 0x7efc4c05c470 2022-06-29T16:05:54.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.026+0000 7efc5150f700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc4c05c280 con 0x7efc4c0b1f60 2022-06-29T16:05:54.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.026+0000 7efc5150f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc4c143e40 con 0x7efc4c0b3d00 2022-06-29T16:05:54.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.026+0000 7efc4b7fe700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7efc4c0b1f60 0x7efc4c0aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:54.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.026+0000 7efc4b7fe700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7efc4c0b1f60 0x7efc4c0aca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:57088/0 (socket says 172.21.15.60:57088) 2022-06-29T16:05:54.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.026+0000 7efc4b7fe700 1 -- 172.21.15.60:0/1623326873 learned_addr learned my addr 172.21.15.60:0/1623326873 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:54.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.026+0000 7efc4bfff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc4c0b3d00 0x7efc4c144580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:54.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.026+0000 7efc4affd700 1 -- 172.21.15.60:0/1623326873 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458988106 0 0) 0x7efc4c0b40e0 con 0x7efc4c05c470 2022-06-29T16:05:54.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.026+0000 7efc4affd700 1 -- 172.21.15.60:0/1623326873 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc38003260 con 0x7efc4c05c470 2022-06-29T16:05:54.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.026+0000 7efc4b7fe700 1 -- 172.21.15.60:0/1623326873 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc4c0b3d00 msgr2=0x7efc4c144580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:54.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.026+0000 7efc4b7fe700 1 --2- 172.21.15.60:0/1623326873 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc4c0b3d00 0x7efc4c144580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:54.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.026+0000 7efc4b7fe700 1 -- 172.21.15.60:0/1623326873 >> v1:172.21.15.92:6789/0 conn(0x7efc4c05c470 legacy=0x7efc4c0acf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:54.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.026+0000 7efc4b7fe700 1 -- 172.21.15.60:0/1623326873 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc4c1440e0 con 0x7efc4c0b1f60 2022-06-29T16:05:54.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.026+0000 7efc4bfff700 1 --2- 172.21.15.60:0/1623326873 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc4c0b3d00 0x7efc4c144580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:05:54.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.026+0000 7efc4b7fe700 1 --2- 172.21.15.60:0/1623326873 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc4c0b1f60 0x7efc4c0aca20 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7efc3c002f70 tx=0x7efc3c006eb0).ready entity=mon.1 client_cookie=55336bdc7110613b server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:54.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.026+0000 7efc4affd700 1 -- 172.21.15.60:0/1623326873 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc3c00d5c0 con 0x7efc4c0b1f60 2022-06-29T16:05:54.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.026+0000 7efc4affd700 1 -- 172.21.15.60:0/1623326873 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc3c000e90 con 0x7efc4c0b1f60 2022-06-29T16:05:54.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.026+0000 7efc4affd700 1 -- 172.21.15.60:0/1623326873 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc3c00d950 con 0x7efc4c0b1f60 2022-06-29T16:05:54.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.026+0000 7efc5150f700 1 -- 172.21.15.60:0/1623326873 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc4c0b1f60 msgr2=0x7efc4c0aca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:54.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.026+0000 7efc5150f700 1 --2- 172.21.15.60:0/1623326873 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc4c0b1f60 0x7efc4c0aca20 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7efc3c002f70 tx=0x7efc3c006eb0).stop 2022-06-29T16:05:54.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.026+0000 7efc5150f700 1 -- 172.21.15.60:0/1623326873 shutdown_connections 2022-06-29T16:05:54.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.026+0000 7efc5150f700 1 --2- 172.21.15.60:0/1623326873 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc4c0b1f60 0x7efc4c0aca20 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:54.044 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.026+0000 7efc5150f700 1 --2- 172.21.15.60:0/1623326873 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc4c0b3d00 0x7efc4c144580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:54.044 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.026+0000 7efc5150f700 1 -- 172.21.15.60:0/1623326873 >> 172.21.15.60:0/1623326873 conn(0x7efc4c064810 msgr2=0x7efc4c064bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:54.045 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.026+0000 7efc5150f700 1 -- 172.21.15.60:0/1623326873 shutdown_connections 2022-06-29T16:05:54.045 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.026+0000 7efc5150f700 1 -- 172.21.15.60:0/1623326873 wait complete. 2022-06-29T16:05:54.045 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.030+0000 7efc5150f700 1 Processor -- start 2022-06-29T16:05:54.045 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.030+0000 7efc5150f700 1 -- start start 2022-06-29T16:05:54.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.030+0000 7efc5150f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc4c05c470 0x7efc4c0ae350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:54.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.030+0000 7efc5150f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc4c0b1f60 0x7efc4c13db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:54.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.030+0000 7efc5150f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc4c0b3d00 0x7efc4c13e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:54.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.030+0000 7efc5150f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc4c153d20 con 0x7efc4c0b1f60 2022-06-29T16:05:54.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.030+0000 7efc5150f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc4c0af960 con 0x7efc4c05c470 2022-06-29T16:05:54.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.030+0000 7efc5150f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc4c066910 con 0x7efc4c0b3d00 2022-06-29T16:05:54.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.030+0000 7efc4b7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc4c0b1f60 0x7efc4c13db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:54.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.030+0000 7efc4b7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc4c0b1f60 0x7efc4c13db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51064/0 (socket says 172.21.15.60:51064) 2022-06-29T16:05:54.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.030+0000 7efc4b7fe700 1 -- 172.21.15.60:0/2840146499 learned_addr learned my addr 172.21.15.60:0/2840146499 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:54.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.030+0000 7efc4bfff700 1 --2- 172.21.15.60:0/2840146499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc4c05c470 0x7efc4c0ae350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:54.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.030+0000 7efc50d0e700 1 --2- 172.21.15.60:0/2840146499 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc4c0b3d00 0x7efc4c13e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:54.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.030+0000 7efc4b7fe700 1 -- 172.21.15.60:0/2840146499 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc4c0b3d00 msgr2=0x7efc4c13e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:54.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.030+0000 7efc4b7fe700 1 --2- 172.21.15.60:0/2840146499 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc4c0b3d00 0x7efc4c13e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:54.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.030+0000 7efc4b7fe700 1 -- 172.21.15.60:0/2840146499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc4c05c470 msgr2=0x7efc4c0ae350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:54.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.030+0000 7efc4b7fe700 1 --2- 172.21.15.60:0/2840146499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc4c05c470 0x7efc4c0ae350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:54.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.030+0000 7efc4b7fe700 1 -- 172.21.15.60:0/2840146499 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc4000b180 con 0x7efc4c0b1f60 2022-06-29T16:05:54.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.030+0000 7efc4b7fe700 1 --2- 172.21.15.60:0/2840146499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc4c0b1f60 0x7efc4c13db20 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 rx=0x7efc3c002f70 tx=0x7efc3c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:54.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.030+0000 7efc497fa700 1 -- 172.21.15.60:0/2840146499 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc3c00de20 con 0x7efc4c0b1f60 2022-06-29T16:05:54.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.030+0000 7efc497fa700 1 -- 172.21.15.60:0/2840146499 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc3c011ca0 con 0x7efc4c0b1f60 2022-06-29T16:05:54.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.030+0000 7efc5150f700 1 -- 172.21.15.60:0/2840146499 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc3c00b1e0 con 0x7efc4c0b1f60 2022-06-29T16:05:54.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.030+0000 7efc5150f700 1 -- 172.21.15.60:0/2840146499 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc4c066e50 con 0x7efc4c0b1f60 2022-06-29T16:05:54.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.030+0000 7efc497fa700 1 -- 172.21.15.60:0/2840146499 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc3c01e460 con 0x7efc4c0b1f60 2022-06-29T16:05:54.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.034+0000 7efc5150f700 1 -- 172.21.15.60:0/2840146499 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc4c152f90 con 0x7efc4c0b1f60 2022-06-29T16:05:54.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.034+0000 7efc497fa700 1 -- 172.21.15.60:0/2840146499 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7efc3c01a040 con 0x7efc4c0b1f60 2022-06-29T16:05:54.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.034+0000 7efc497fa700 1 --2- 172.21.15.60:0/2840146499 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efc2c053a70 0x7efc2c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:54.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.034+0000 7efc497fa700 1 -- 172.21.15.60:0/2840146499 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6159+0+0 (secure 0 0 0) 0x7efc3c06ce20 con 0x7efc4c0b1f60 2022-06-29T16:05:54.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.034+0000 7efc4bfff700 1 --2- 172.21.15.60:0/2840146499 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efc2c053a70 0x7efc2c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:54.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.034+0000 7efc4bfff700 1 --2- 172.21.15.60:0/2840146499 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efc2c053a70 0x7efc2c055f30 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7efc4000b000 tx=0x7efc40008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:54.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.042+0000 7efc497fa700 1 -- 172.21.15.60:0/2840146499 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efc3c043d00 con 0x7efc4c0b1f60 2022-06-29T16:05:54.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.206+0000 7efc5150f700 1 -- 172.21.15.60:0/2840146499 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc4c0573d0 con 0x7efc4c0b1f60 2022-06-29T16:05:54.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.206+0000 7efc497fa700 1 -- 172.21.15.60:0/2840146499 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+1779 (secure 0 0 0) 0x7efc3c023070 con 0x7efc4c0b1f60 2022-06-29T16:05:54.215 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:54.215 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":42,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6182,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1306898602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1306898602},{"type":"v1","addr":"172.21.15.60:6833","nonce":1306898602}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":39}],"filesystems":[{"mdsmap":{"epoch":42,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:05:40.369985+0000","modified":"2022-06-29T16:05:53.893290+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-06-29T16:05:54.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.210+0000 7efc5150f700 1 -- 172.21.15.60:0/2840146499 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efc2c053a70 msgr2=0x7efc2c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:54.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.210+0000 7efc5150f700 1 --2- 172.21.15.60:0/2840146499 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efc2c053a70 0x7efc2c055f30 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7efc4000b000 tx=0x7efc40008040).stop 2022-06-29T16:05:54.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.210+0000 7efc5150f700 1 -- 172.21.15.60:0/2840146499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc4c0b1f60 msgr2=0x7efc4c13db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:54.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.210+0000 7efc5150f700 1 --2- 172.21.15.60:0/2840146499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc4c0b1f60 0x7efc4c13db20 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 rx=0x7efc3c002f70 tx=0x7efc3c015000).stop 2022-06-29T16:05:54.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.210+0000 7efc5150f700 1 -- 172.21.15.60:0/2840146499 shutdown_connections 2022-06-29T16:05:54.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.210+0000 7efc5150f700 1 --2- 172.21.15.60:0/2840146499 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efc2c053a70 0x7efc2c055f30 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:54.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.210+0000 7efc5150f700 1 --2- 172.21.15.60:0/2840146499 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efc4c0b3d00 0x7efc4c13e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:54.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.210+0000 7efc5150f700 1 --2- 172.21.15.60:0/2840146499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efc4c0b1f60 0x7efc4c13db20 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:54.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.210+0000 7efc5150f700 1 --2- 172.21.15.60:0/2840146499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc4c05c470 0x7efc4c0ae350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:54.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.210+0000 7efc5150f700 1 -- 172.21.15.60:0/2840146499 >> 172.21.15.60:0/2840146499 conn(0x7efc4c064810 msgr2=0x7efc4c13ecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:54.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.210+0000 7efc5150f700 1 -- 172.21.15.60:0/2840146499 shutdown_connections 2022-06-29T16:05:54.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.210+0000 7efc5150f700 1 -- 172.21.15.60:0/2840146499 wait complete. 2022-06-29T16:05:54.221 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 42 2022-06-29T16:05:54.228 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-06-29T16:05:54.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.326+0000 7efd38883700 1 Processor -- start 2022-06-29T16:05:54.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.326+0000 7efd38883700 1 -- start start 2022-06-29T16:05:54.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.326+0000 7efd38883700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd340b3060 0x7efd3405f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:54.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.326+0000 7efd38883700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7efd340b2b50 0x7efd34140e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:54.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.326+0000 7efd38883700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd341407e0 con 0x7efd34066b80 2022-06-29T16:05:54.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.326+0000 7efd38883700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd341441b0 con 0x7efd340b2b50 2022-06-29T16:05:54.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.326+0000 7efd38883700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd34140a20 con 0x7efd340b3060 2022-06-29T16:05:54.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.326+0000 7efd3259c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd340b3060 0x7efd3405f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:54.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.326+0000 7efd3359e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7efd340b2b50 0x7efd34140e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:54.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.326+0000 7efd3359e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7efd340b2b50 0x7efd34140e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:57104/0 (socket says 172.21.15.60:57104) 2022-06-29T16:05:54.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.326+0000 7efd3259c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd340b3060 0x7efd3405f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:35936/0 (socket says 172.21.15.60:35936) 2022-06-29T16:05:54.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.326+0000 7efd3359e700 1 -- 172.21.15.60:0/1929729112 learned_addr learned my addr 172.21.15.60:0/1929729112 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:54.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.326+0000 7efd32d9d700 1 --1- 172.21.15.60:0/1929729112 >> v1:172.21.15.92:6789/0 conn(0x7efd34066b80 0x7efd3413a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:35174/0 (socket says 172.21.15.60:35174) 2022-06-29T16:05:54.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.326+0000 7efd3259c700 1 -- 172.21.15.60:0/1929729112 >> v2:172.21.15.92:3300/0 conn(0x7efd340b2b50 msgr2=0x7efd34140e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:54.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.326+0000 7efd3259c700 1 --2- 172.21.15.60:0/1929729112 >> v2:172.21.15.92:3300/0 conn(0x7efd340b2b50 0x7efd34140e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:54.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.326+0000 7efd3259c700 1 -- 172.21.15.60:0/1929729112 >> v1:172.21.15.92:6789/0 conn(0x7efd34066b80 legacy=0x7efd3413a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:54.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.326+0000 7efd3259c700 1 -- 172.21.15.60:0/1929729112 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd3405f9e0 con 0x7efd340b3060 2022-06-29T16:05:54.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.326+0000 7efd3259c700 1 --2- 172.21.15.60:0/1929729112 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd340b3060 0x7efd3405f4f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7efd28002f70 tx=0x7efd28006df0).ready entity=mon.2 client_cookie=3258e7608c19fa56 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:54.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.326+0000 7efd31d9b700 1 -- 172.21.15.60:0/1929729112 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd2800d570 con 0x7efd340b3060 2022-06-29T16:05:54.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.326+0000 7efd31d9b700 1 -- 172.21.15.60:0/1929729112 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd28000e90 con 0x7efd340b3060 2022-06-29T16:05:54.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.326+0000 7efd31d9b700 1 -- 172.21.15.60:0/1929729112 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd2800d8c0 con 0x7efd340b3060 2022-06-29T16:05:54.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.330+0000 7efd38883700 1 -- 172.21.15.60:0/1929729112 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd340b3060 msgr2=0x7efd3405f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:54.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.330+0000 7efd38883700 1 --2- 172.21.15.60:0/1929729112 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd340b3060 0x7efd3405f4f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7efd28002f70 tx=0x7efd28006df0).stop 2022-06-29T16:05:54.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.330+0000 7efd38883700 1 -- 172.21.15.60:0/1929729112 shutdown_connections 2022-06-29T16:05:54.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.330+0000 7efd38883700 1 --2- 172.21.15.60:0/1929729112 >> v2:172.21.15.92:3300/0 conn(0x7efd340b2b50 0x7efd34140e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:54.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.330+0000 7efd38883700 1 --2- 172.21.15.60:0/1929729112 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd340b3060 0x7efd3405f4f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:54.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.330+0000 7efd38883700 1 -- 172.21.15.60:0/1929729112 >> 172.21.15.60:0/1929729112 conn(0x7efd34061c80 msgr2=0x7efd340637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:54.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.330+0000 7efd38883700 1 -- 172.21.15.60:0/1929729112 shutdown_connections 2022-06-29T16:05:54.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.330+0000 7efd38883700 1 -- 172.21.15.60:0/1929729112 wait complete. 2022-06-29T16:05:54.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.330+0000 7efd38883700 1 Processor -- start 2022-06-29T16:05:54.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.330+0000 7efd38883700 1 -- start start 2022-06-29T16:05:54.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.330+0000 7efd38883700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd34066b80 0x7efd341502c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:54.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.330+0000 7efd38883700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd340b2b50 0x7efd34146fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:54.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.330+0000 7efd38883700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd340b3060 0x7efd34141d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:54.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.330+0000 7efd38883700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3414f940 con 0x7efd340b2b50 2022-06-29T16:05:54.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.330+0000 7efd38883700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3414fb80 con 0x7efd340b3060 2022-06-29T16:05:54.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.330+0000 7efd38883700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd3414fe20 con 0x7efd34066b80 2022-06-29T16:05:54.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.330+0000 7efd3259c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd340b2b50 0x7efd34146fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:54.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.330+0000 7efd3259c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd340b2b50 0x7efd34146fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51078/0 (socket says 172.21.15.60:51078) 2022-06-29T16:05:54.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.330+0000 7efd3259c700 1 -- 172.21.15.60:0/2911508466 learned_addr learned my addr 172.21.15.60:0/2911508466 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:54.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.330+0000 7efd3359e700 1 --2- 172.21.15.60:0/2911508466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd340b3060 0x7efd34141d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:54.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.330+0000 7efd32d9d700 1 --2- 172.21.15.60:0/2911508466 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd34066b80 0x7efd341502c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:54.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.330+0000 7efd3259c700 1 -- 172.21.15.60:0/2911508466 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd34066b80 msgr2=0x7efd341502c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:54.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.330+0000 7efd3259c700 1 --2- 172.21.15.60:0/2911508466 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd34066b80 0x7efd341502c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:54.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.330+0000 7efd3259c700 1 -- 172.21.15.60:0/2911508466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd340b3060 msgr2=0x7efd34141d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:54.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.330+0000 7efd3259c700 1 --2- 172.21.15.60:0/2911508466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd340b3060 0x7efd34141d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:54.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.330+0000 7efd3259c700 1 -- 172.21.15.60:0/2911508466 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd2800b180 con 0x7efd340b2b50 2022-06-29T16:05:54.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.334+0000 7efd3259c700 1 --2- 172.21.15.60:0/2911508466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd340b2b50 0x7efd34146fb0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 rx=0x7efd28002f70 tx=0x7efd28014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:54.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.334+0000 7efd23fff700 1 -- 172.21.15.60:0/2911508466 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd2800dea0 con 0x7efd340b2b50 2022-06-29T16:05:54.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.334+0000 7efd23fff700 1 -- 172.21.15.60:0/2911508466 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd28013de0 con 0x7efd340b2b50 2022-06-29T16:05:54.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.334+0000 7efd38883700 1 -- 172.21.15.60:0/2911508466 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd341436a0 con 0x7efd340b2b50 2022-06-29T16:05:54.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.334+0000 7efd23fff700 1 -- 172.21.15.60:0/2911508466 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd2801e5f0 con 0x7efd340b2b50 2022-06-29T16:05:54.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.334+0000 7efd38883700 1 -- 172.21.15.60:0/2911508466 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd341474a0 con 0x7efd340b2b50 2022-06-29T16:05:54.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.334+0000 7efd23fff700 1 -- 172.21.15.60:0/2911508466 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7efd28025030 con 0x7efd340b2b50 2022-06-29T16:05:54.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.334+0000 7efd38883700 1 -- 172.21.15.60:0/2911508466 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd3414ffe0 con 0x7efd340b2b50 2022-06-29T16:05:54.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.334+0000 7efd23fff700 1 --2- 172.21.15.60:0/2911508466 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd10053a70 0x7efd10055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:54.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.334+0000 7efd23fff700 1 -- 172.21.15.60:0/2911508466 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6159+0+0 (secure 0 0 0) 0x7efd2806cb30 con 0x7efd340b2b50 2022-06-29T16:05:54.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.334+0000 7efd32d9d700 1 --2- 172.21.15.60:0/2911508466 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd10053a70 0x7efd10055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:54.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.334+0000 7efd32d9d700 1 --2- 172.21.15.60:0/2911508466 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd10053a70 0x7efd10055f30 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7efd2400d000 tx=0x7efd24007b80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:54.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.338+0000 7efd23fff700 1 -- 172.21.15.60:0/2911508466 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efd28043a00 con 0x7efd340b2b50 2022-06-29T16:05:54.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.502+0000 7efd38883700 1 -- 172.21.15.60:0/2911508466 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7efd340573d0 con 0x7efd340b2b50 2022-06-29T16:05:54.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.926+0000 7efd23fff700 1 -- 172.21.15.60:0/2911508466 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v43) v1 ==== 104+0+0 (secure 0 0 0) 0x7efd28023020 con 0x7efd340b2b50 2022-06-29T16:05:54.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.930+0000 7efd38883700 1 -- 172.21.15.60:0/2911508466 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd10053a70 msgr2=0x7efd10055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:54.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.930+0000 7efd38883700 1 --2- 172.21.15.60:0/2911508466 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd10053a70 0x7efd10055f30 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7efd2400d000 tx=0x7efd24007b80).stop 2022-06-29T16:05:54.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.930+0000 7efd38883700 1 -- 172.21.15.60:0/2911508466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd340b2b50 msgr2=0x7efd34146fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:54.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.930+0000 7efd38883700 1 --2- 172.21.15.60:0/2911508466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd340b2b50 0x7efd34146fb0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 rx=0x7efd28002f70 tx=0x7efd28014000).stop 2022-06-29T16:05:54.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.930+0000 7efd38883700 1 -- 172.21.15.60:0/2911508466 shutdown_connections 2022-06-29T16:05:54.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.930+0000 7efd38883700 1 --2- 172.21.15.60:0/2911508466 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd10053a70 0x7efd10055f30 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:54.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.930+0000 7efd38883700 1 --2- 172.21.15.60:0/2911508466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd340b3060 0x7efd34141d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:54.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.930+0000 7efd38883700 1 --2- 172.21.15.60:0/2911508466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd340b2b50 0x7efd34146fb0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:54.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.930+0000 7efd38883700 1 --2- 172.21.15.60:0/2911508466 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd34066b80 0x7efd341502c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:54.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.930+0000 7efd38883700 1 -- 172.21.15.60:0/2911508466 >> 172.21.15.60:0/2911508466 conn(0x7efd34061c80 msgr2=0x7efd340b3f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:54.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.930+0000 7efd38883700 1 -- 172.21.15.60:0/2911508466 shutdown_connections 2022-06-29T16:05:54.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:54.930+0000 7efd38883700 1 -- 172.21.15.60:0/2911508466 wait complete. 2022-06-29T16:05:54.957 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-06-29T16:05:55.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.066+0000 7fd24710b700 1 Processor -- start 2022-06-29T16:05:55.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.066+0000 7fd24710b700 1 -- start start 2022-06-29T16:05:55.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.066+0000 7fd24710b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd2400b6a50 0x7fd2401412a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:55.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.066+0000 7fd24710b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2400b2040 0x7fd240144000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:55.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.066+0000 7fd24710b700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2400b6e30 con 0x7fd2401444f0 2022-06-29T16:05:55.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.066+0000 7fd24710b700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd240140b60 con 0x7fd2400b6a50 2022-06-29T16:05:55.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.066+0000 7fd24710b700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd240140e00 con 0x7fd2400b2040 2022-06-29T16:05:55.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.066+0000 7fd244ea7700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd2400b6a50 0x7fd2401412a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:55.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.066+0000 7fd244ea7700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd2400b6a50 0x7fd2401412a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51086/0 (socket says 172.21.15.60:51086) 2022-06-29T16:05:55.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.066+0000 7fd244ea7700 1 -- 172.21.15.60:0/1659555329 learned_addr learned my addr 172.21.15.60:0/1659555329 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:55.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.066+0000 7fd23ffff700 1 --2- 172.21.15.60:0/1659555329 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2400b2040 0x7fd240144000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:55.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.066+0000 7fd244ea7700 1 -- 172.21.15.60:0/1659555329 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2400b2040 msgr2=0x7fd240144000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:55.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.066+0000 7fd244ea7700 1 --2- 172.21.15.60:0/1659555329 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2400b2040 0x7fd240144000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:55.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.066+0000 7fd244ea7700 1 -- 172.21.15.60:0/1659555329 >> v1:172.21.15.92:6789/0 conn(0x7fd2401444f0 legacy=0x7fd2401400a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:55.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.066+0000 7fd244ea7700 1 -- 172.21.15.60:0/1659555329 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2400b24c0 con 0x7fd2400b6a50 2022-06-29T16:05:55.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.066+0000 7fd244ea7700 1 --2- 172.21.15.60:0/1659555329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2400b6a50 0x7fd2401412a0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 rx=0x7fd234002f70 tx=0x7fd234006eb0).ready entity=mon.0 client_cookie=4aa57decc347bf07 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:55.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.066+0000 7fd23f7fe700 1 -- 172.21.15.60:0/1659555329 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd23400d5c0 con 0x7fd2400b6a50 2022-06-29T16:05:55.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.066+0000 7fd23f7fe700 1 -- 172.21.15.60:0/1659555329 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd234000e90 con 0x7fd2400b6a50 2022-06-29T16:05:55.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.066+0000 7fd23f7fe700 1 -- 172.21.15.60:0/1659555329 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd23400d950 con 0x7fd2400b6a50 2022-06-29T16:05:55.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.066+0000 7fd24710b700 1 -- 172.21.15.60:0/1659555329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2400b6a50 msgr2=0x7fd2401412a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:55.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.066+0000 7fd24710b700 1 --2- 172.21.15.60:0/1659555329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2400b6a50 0x7fd2401412a0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 rx=0x7fd234002f70 tx=0x7fd234006eb0).stop 2022-06-29T16:05:55.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.066+0000 7fd24710b700 1 -- 172.21.15.60:0/1659555329 shutdown_connections 2022-06-29T16:05:55.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.066+0000 7fd24710b700 1 --2- 172.21.15.60:0/1659555329 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2400b2040 0x7fd240144000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:55.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.066+0000 7fd24710b700 1 --2- 172.21.15.60:0/1659555329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2400b6a50 0x7fd2401412a0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:55.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.066+0000 7fd24710b700 1 -- 172.21.15.60:0/1659555329 >> 172.21.15.60:0/1659555329 conn(0x7fd240064420 msgr2=0x7fd2400b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:55.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.066+0000 7fd24710b700 1 -- 172.21.15.60:0/1659555329 shutdown_connections 2022-06-29T16:05:55.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.066+0000 7fd24710b700 1 -- 172.21.15.60:0/1659555329 wait complete. 2022-06-29T16:05:55.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.070+0000 7fd24710b700 1 Processor -- start 2022-06-29T16:05:55.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.070+0000 7fd24710b700 1 -- start start 2022-06-29T16:05:55.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.070+0000 7fd24710b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2400b2040 0x7fd2401431d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:55.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.070+0000 7fd24710b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2400b6a50 0x7fd240152790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:55.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.070+0000 7fd24710b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2401444f0 0x7fd240152c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:55.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.070+0000 7fd24710b700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2401523a0 con 0x7fd2401444f0 2022-06-29T16:05:55.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.070+0000 7fd24710b700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2400af910 con 0x7fd2400b2040 2022-06-29T16:05:55.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.070+0000 7fd24710b700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd240064920 con 0x7fd2400b6a50 2022-06-29T16:05:55.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.070+0000 7fd2456a8700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2401444f0 0x7fd240152c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:55.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.070+0000 7fd2456a8700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2401444f0 0x7fd240152c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51094/0 (socket says 172.21.15.60:51094) 2022-06-29T16:05:55.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.070+0000 7fd2456a8700 1 -- 172.21.15.60:0/3346463994 learned_addr learned my addr 172.21.15.60:0/3346463994 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:55.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.070+0000 7fd244ea7700 1 --2- 172.21.15.60:0/3346463994 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2400b2040 0x7fd2401431d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:55.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.070+0000 7fd23ffff700 1 --2- 172.21.15.60:0/3346463994 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2400b6a50 0x7fd240152790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:55.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.070+0000 7fd2456a8700 1 -- 172.21.15.60:0/3346463994 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2400b6a50 msgr2=0x7fd240152790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:55.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.070+0000 7fd2456a8700 1 --2- 172.21.15.60:0/3346463994 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2400b6a50 0x7fd240152790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:55.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.070+0000 7fd2456a8700 1 -- 172.21.15.60:0/3346463994 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2400b2040 msgr2=0x7fd2401431d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:55.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.070+0000 7fd2456a8700 1 --2- 172.21.15.60:0/3346463994 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2400b2040 0x7fd2401431d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:55.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.070+0000 7fd2456a8700 1 -- 172.21.15.60:0/3346463994 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd23400b1e0 con 0x7fd2401444f0 2022-06-29T16:05:55.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.070+0000 7fd2456a8700 1 --2- 172.21.15.60:0/3346463994 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2401444f0 0x7fd240152c80 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 rx=0x7fd2400b3d20 tx=0x7fd238007f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:55.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.070+0000 7fd23dffb700 1 -- 172.21.15.60:0/3346463994 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd23800cea0 con 0x7fd2401444f0 2022-06-29T16:05:55.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.070+0000 7fd23dffb700 1 -- 172.21.15.60:0/3346463994 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd238006aa0 con 0x7fd2401444f0 2022-06-29T16:05:55.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.070+0000 7fd24710b700 1 -- 172.21.15.60:0/3346463994 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2400aeb70 con 0x7fd2401444f0 2022-06-29T16:05:55.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.070+0000 7fd23dffb700 1 -- 172.21.15.60:0/3346463994 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd238006e00 con 0x7fd2401444f0 2022-06-29T16:05:55.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.070+0000 7fd24710b700 1 -- 172.21.15.60:0/3346463994 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2400b0180 con 0x7fd2401444f0 2022-06-29T16:05:55.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.074+0000 7fd24710b700 1 -- 172.21.15.60:0/3346463994 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2400aff20 con 0x7fd2401444f0 2022-06-29T16:05:55.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.074+0000 7fd23dffb700 1 -- 172.21.15.60:0/3346463994 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd238010030 con 0x7fd2401444f0 2022-06-29T16:05:55.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.074+0000 7fd23dffb700 1 --2- 172.21.15.60:0/3346463994 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd220053aa0 0x7fd220055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:55.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.074+0000 7fd23dffb700 1 -- 172.21.15.60:0/3346463994 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fd23806bd30 con 0x7fd2401444f0 2022-06-29T16:05:55.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.074+0000 7fd244ea7700 1 --2- 172.21.15.60:0/3346463994 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd220053aa0 0x7fd220055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:55.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.074+0000 7fd244ea7700 1 --2- 172.21.15.60:0/3346463994 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd220053aa0 0x7fd220055f60 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7fd23401c000 tx=0x7fd234015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:55.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.082+0000 7fd23dffb700 1 -- 172.21.15.60:0/3346463994 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd238042b60 con 0x7fd2401444f0 2022-06-29T16:05:55.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:55.210+0000 7fd24710b700 1 -- 172.21.15.60:0/3346463994 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd2400573d0 con 0x7fd2401444f0 2022-06-29T16:05:56.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.170+0000 7fd23dffb700 1 -- 172.21.15.60:0/3346463994 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v93) v1 ==== 185+0+0 (secure 0 0 0) 0x7fd238046180 con 0x7fd2401444f0 2022-06-29T16:05:56.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.174+0000 7fd24710b700 1 -- 172.21.15.60:0/3346463994 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd220053aa0 msgr2=0x7fd220055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:56.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.174+0000 7fd24710b700 1 --2- 172.21.15.60:0/3346463994 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd220053aa0 0x7fd220055f60 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7fd23401c000 tx=0x7fd234015000).stop 2022-06-29T16:05:56.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.174+0000 7fd24710b700 1 -- 172.21.15.60:0/3346463994 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2401444f0 msgr2=0x7fd240152c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:56.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.174+0000 7fd24710b700 1 --2- 172.21.15.60:0/3346463994 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2401444f0 0x7fd240152c80 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 rx=0x7fd2400b3d20 tx=0x7fd238007f70).stop 2022-06-29T16:05:56.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.174+0000 7fd24710b700 1 -- 172.21.15.60:0/3346463994 shutdown_connections 2022-06-29T16:05:56.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.174+0000 7fd24710b700 1 --2- 172.21.15.60:0/3346463994 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd220053aa0 0x7fd220055f60 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:56.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.174+0000 7fd24710b700 1 --2- 172.21.15.60:0/3346463994 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2401444f0 0x7fd240152c80 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:56.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.174+0000 7fd24710b700 1 --2- 172.21.15.60:0/3346463994 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2400b6a50 0x7fd240152790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:56.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.174+0000 7fd24710b700 1 --2- 172.21.15.60:0/3346463994 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2400b2040 0x7fd2401431d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:56.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.174+0000 7fd24710b700 1 -- 172.21.15.60:0/3346463994 >> 172.21.15.60:0/3346463994 conn(0x7fd240064420 msgr2=0x7fd240140120 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:56.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.174+0000 7fd24710b700 1 -- 172.21.15.60:0/3346463994 shutdown_connections 2022-06-29T16:05:56.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.174+0000 7fd24710b700 1 -- 172.21.15.60:0/3346463994 wait complete. 2022-06-29T16:05:56.187 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2022-06-29T16:05:56.196 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-06-29T16:05:56.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.302+0000 7f6daefca700 1 Processor -- start 2022-06-29T16:05:56.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.302+0000 7f6daefca700 1 -- start start 2022-06-29T16:05:56.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.302+0000 7f6daefca700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6da813c9a0 0x7f6da8062280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:56.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.302+0000 7f6daefca700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6da80b3190 0x7f6da80aca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:56.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.302+0000 7f6daefca700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6da81428a0 con 0x7f6da80acf10 2022-06-29T16:05:56.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.302+0000 7f6daefca700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6da8061530 con 0x7f6da80b3190 2022-06-29T16:05:56.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.302+0000 7f6daefca700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6da80632a0 con 0x7f6da813c9a0 2022-06-29T16:05:56.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.302+0000 7f6dae7c9700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f6da80acf10 0x7f6da8142f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:35206/0 (socket says 172.21.15.60:35206) 2022-06-29T16:05:56.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.302+0000 7f6dae7c9700 1 -- 172.21.15.60:0/2662327548 learned_addr learned my addr 172.21.15.60:0/2662327548 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:56.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.302+0000 7f6dad7c7700 1 --2- 172.21.15.60:0/2662327548 >> v2:172.21.15.92:3300/0 conn(0x7f6da80b3190 0x7f6da80aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:56.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.302+0000 7f6dadfc8700 1 --2- 172.21.15.60:0/2662327548 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6da813c9a0 0x7f6da8062280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:56.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.302+0000 7f6dacfc6700 1 -- 172.21.15.60:0/2662327548 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 32092298 0 0) 0x7f6da81428a0 con 0x7f6da80acf10 2022-06-29T16:05:56.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.302+0000 7f6dacfc6700 1 -- 172.21.15.60:0/2662327548 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d90003260 con 0x7f6da80acf10 2022-06-29T16:05:56.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.302+0000 7f6dad7c7700 1 -- 172.21.15.60:0/2662327548 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6da813c9a0 msgr2=0x7f6da8062280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:56.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.302+0000 7f6dad7c7700 1 --2- 172.21.15.60:0/2662327548 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6da813c9a0 0x7f6da8062280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:56.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.302+0000 7f6dad7c7700 1 -- 172.21.15.60:0/2662327548 >> v1:172.21.15.92:6789/0 conn(0x7f6da80acf10 legacy=0x7f6da8142f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:56.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.302+0000 7f6dad7c7700 1 -- 172.21.15.60:0/2662327548 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da813cd80 con 0x7f6da80b3190 2022-06-29T16:05:56.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.302+0000 7f6dadfc8700 1 --2- 172.21.15.60:0/2662327548 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6da813c9a0 0x7f6da8062280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:05:56.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.302+0000 7f6dad7c7700 1 --2- 172.21.15.60:0/2662327548 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6da80b3190 0x7f6da80aca20 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f6d98002f70 tx=0x7f6d98006df0).ready entity=mon.1 client_cookie=297477e01502f492 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:56.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.302+0000 7f6dacfc6700 1 -- 172.21.15.60:0/2662327548 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d9800d4a0 con 0x7f6da80b3190 2022-06-29T16:05:56.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.302+0000 7f6dacfc6700 1 -- 172.21.15.60:0/2662327548 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d98000e90 con 0x7f6da80b3190 2022-06-29T16:05:56.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.302+0000 7f6dacfc6700 1 -- 172.21.15.60:0/2662327548 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d9800d7f0 con 0x7f6da80b3190 2022-06-29T16:05:56.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.306+0000 7f6daefca700 1 -- 172.21.15.60:0/2662327548 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6da80b3190 msgr2=0x7f6da80aca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:56.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.306+0000 7f6daefca700 1 --2- 172.21.15.60:0/2662327548 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6da80b3190 0x7f6da80aca20 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f6d98002f70 tx=0x7f6d98006df0).stop 2022-06-29T16:05:56.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.306+0000 7f6daefca700 1 -- 172.21.15.60:0/2662327548 shutdown_connections 2022-06-29T16:05:56.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.306+0000 7f6daefca700 1 --2- 172.21.15.60:0/2662327548 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6da80b3190 0x7f6da80aca20 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:56.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.306+0000 7f6daefca700 1 --2- 172.21.15.60:0/2662327548 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6da813c9a0 0x7f6da8062280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:56.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.306+0000 7f6daefca700 1 -- 172.21.15.60:0/2662327548 >> 172.21.15.60:0/2662327548 conn(0x7f6da80b6f80 msgr2=0x7f6da80b7390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:56.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.306+0000 7f6daefca700 1 -- 172.21.15.60:0/2662327548 shutdown_connections 2022-06-29T16:05:56.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.306+0000 7f6daefca700 1 -- 172.21.15.60:0/2662327548 wait complete. 2022-06-29T16:05:56.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.306+0000 7f6daefca700 1 Processor -- start 2022-06-29T16:05:56.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.306+0000 7f6daefca700 1 -- start start 2022-06-29T16:05:56.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.314+0000 7f6daefca700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6da80acf10 0x7f6da8145080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:56.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.314+0000 7f6daefca700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6da80b3190 0x7f6da80ae3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:56.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.314+0000 7f6daefca700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6da813c9a0 0x7f6da80ae890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:56.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.314+0000 7f6daefca700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da81421a0 con 0x7f6da813c9a0 2022-06-29T16:05:56.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.314+0000 7f6daefca700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da81416f0 con 0x7f6da80acf10 2022-06-29T16:05:56.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.314+0000 7f6daefca700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6da8141b90 con 0x7f6da80b3190 2022-06-29T16:05:56.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.314+0000 7f6dae7c9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6da813c9a0 0x7f6da80ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:56.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.314+0000 7f6dad7c7700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6da80b3190 0x7f6da80ae3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:56.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.314+0000 7f6dae7c9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6da813c9a0 0x7f6da80ae890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51110/0 (socket says 172.21.15.60:51110) 2022-06-29T16:05:56.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.314+0000 7f6dae7c9700 1 -- 172.21.15.60:0/395227502 learned_addr learned my addr 172.21.15.60:0/395227502 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:56.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.314+0000 7f6dadfc8700 1 --2- 172.21.15.60:0/395227502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6da80acf10 0x7f6da8145080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:56.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.314+0000 7f6dae7c9700 1 -- 172.21.15.60:0/395227502 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6da80b3190 msgr2=0x7f6da80ae3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:56.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.314+0000 7f6dae7c9700 1 --2- 172.21.15.60:0/395227502 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6da80b3190 0x7f6da80ae3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:56.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.314+0000 7f6dae7c9700 1 -- 172.21.15.60:0/395227502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6da80acf10 msgr2=0x7f6da8145080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:56.326 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.314+0000 7f6dae7c9700 1 --2- 172.21.15.60:0/395227502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6da80acf10 0x7f6da8145080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:56.326 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.314+0000 7f6dae7c9700 1 -- 172.21.15.60:0/395227502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da400b180 con 0x7f6da813c9a0 2022-06-29T16:05:56.326 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.314+0000 7f6dae7c9700 1 --2- 172.21.15.60:0/395227502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6da813c9a0 0x7f6da80ae890 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 rx=0x7f6da0001490 tx=0x7f6da0007dc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:56.326 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.314+0000 7f6d96ffd700 1 -- 172.21.15.60:0/395227502 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6da0005c70 con 0x7f6da813c9a0 2022-06-29T16:05:56.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.314+0000 7f6d96ffd700 1 -- 172.21.15.60:0/395227502 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da0005dd0 con 0x7f6da813c9a0 2022-06-29T16:05:56.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.314+0000 7f6daefca700 1 -- 172.21.15.60:0/395227502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d9800b180 con 0x7f6da813c9a0 2022-06-29T16:05:56.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.314+0000 7f6d96ffd700 1 -- 172.21.15.60:0/395227502 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6da00072c0 con 0x7f6da813c9a0 2022-06-29T16:05:56.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.314+0000 7f6daefca700 1 -- 172.21.15.60:0/395227502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6da8142730 con 0x7f6da813c9a0 2022-06-29T16:05:56.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.314+0000 7f6daefca700 1 -- 172.21.15.60:0/395227502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6da81418b0 con 0x7f6da813c9a0 2022-06-29T16:05:56.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.314+0000 7f6d96ffd700 1 -- 172.21.15.60:0/395227502 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6da001a060 con 0x7f6da813c9a0 2022-06-29T16:05:56.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.314+0000 7f6d96ffd700 1 --2- 172.21.15.60:0/395227502 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6d88053b00 0x7f6d88055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:56.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.314+0000 7f6d96ffd700 1 -- 172.21.15.60:0/395227502 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f6da006b780 con 0x7f6da813c9a0 2022-06-29T16:05:56.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.314+0000 7f6dadfc8700 1 --2- 172.21.15.60:0/395227502 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6d88053b00 0x7f6d88055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:56.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.314+0000 7f6dadfc8700 1 --2- 172.21.15.60:0/395227502 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6d88053b00 0x7f6d88055fc0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7f6da400b000 tx=0x7f6da4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:56.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.326+0000 7f6d96ffd700 1 -- 172.21.15.60:0/395227502 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6da0042800 con 0x7f6da813c9a0 2022-06-29T16:05:56.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:56.450+0000 7f6daefca700 1 -- 172.21.15.60:0/395227502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6da80573d0 con 0x7f6da813c9a0 2022-06-29T16:05:57.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.181+0000 7f6d96ffd700 1 -- 172.21.15.60:0/395227502 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v94) v1 ==== 173+0+0 (secure 0 0 0) 0x7f6da0018020 con 0x7f6da813c9a0 2022-06-29T16:05:57.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.185+0000 7f6daefca700 1 -- 172.21.15.60:0/395227502 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6d88053b00 msgr2=0x7f6d88055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:57.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.185+0000 7f6daefca700 1 --2- 172.21.15.60:0/395227502 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6d88053b00 0x7f6d88055fc0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7f6da400b000 tx=0x7f6da4006df0).stop 2022-06-29T16:05:57.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.185+0000 7f6daefca700 1 -- 172.21.15.60:0/395227502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6da813c9a0 msgr2=0x7f6da80ae890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:57.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.185+0000 7f6daefca700 1 --2- 172.21.15.60:0/395227502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6da813c9a0 0x7f6da80ae890 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 rx=0x7f6da0001490 tx=0x7f6da0007dc0).stop 2022-06-29T16:05:57.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.185+0000 7f6daefca700 1 -- 172.21.15.60:0/395227502 shutdown_connections 2022-06-29T16:05:57.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.185+0000 7f6daefca700 1 --2- 172.21.15.60:0/395227502 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6d88053b00 0x7f6d88055fc0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:57.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.185+0000 7f6daefca700 1 --2- 172.21.15.60:0/395227502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6da813c9a0 0x7f6da80ae890 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:57.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.185+0000 7f6daefca700 1 --2- 172.21.15.60:0/395227502 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6da80b3190 0x7f6da80ae3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:57.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.185+0000 7f6daefca700 1 --2- 172.21.15.60:0/395227502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6da80acf10 0x7f6da8145080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:57.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.185+0000 7f6daefca700 1 -- 172.21.15.60:0/395227502 >> 172.21.15.60:0/395227502 conn(0x7f6da80b6f80 msgr2=0x7f6da80b66c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:57.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.185+0000 7f6daefca700 1 -- 172.21.15.60:0/395227502 shutdown_connections 2022-06-29T16:05:57.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.185+0000 7f6daefca700 1 -- 172.21.15.60:0/395227502 wait complete. 2022-06-29T16:05:57.195 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2022-06-29T16:05:57.205 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-06-29T16:05:57.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.313+0000 7f6038787700 1 Processor -- start 2022-06-29T16:05:57.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.313+0000 7f6038787700 1 -- start start 2022-06-29T16:05:57.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.313+0000 7f6038787700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6030140410 0x7f6030065250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:57.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.313+0000 7f6038787700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60300b5d30 con 0x7f60300b5950 2022-06-29T16:05:57.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.313+0000 7f6038787700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6030066fa0 con 0x7f6030064b50 2022-06-29T16:05:57.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.313+0000 7f6038787700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6030143960 con 0x7f6030140410 2022-06-29T16:05:57.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.313+0000 7f6035d22700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6030140410 0x7f6030065250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:57.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.313+0000 7f6036523700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f60300b5950 0x7f603013fd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59200/0 (socket says 172.21.15.60:59200) 2022-06-29T16:05:57.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.313+0000 7f6035d22700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6030140410 0x7f6030065250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51116/0 (socket says 172.21.15.60:51116) 2022-06-29T16:05:57.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.313+0000 7f6035d22700 1 -- 172.21.15.60:0/3033833467 learned_addr learned my addr 172.21.15.60:0/3033833467 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:57.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.313+0000 7f6035d22700 1 -- 172.21.15.60:0/3033833467 >> v1:172.21.15.92:6789/0 conn(0x7f6030064b50 legacy=0x7f6030128290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:57.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.313+0000 7f6035d22700 1 -- 172.21.15.60:0/3033833467 >> v1:172.21.15.60:6789/0 conn(0x7f60300b5950 legacy=0x7f603013fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:57.326 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.313+0000 7f6035d22700 1 -- 172.21.15.60:0/3033833467 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6030143c00 con 0x7f6030140410 2022-06-29T16:05:57.326 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.313+0000 7f6035d22700 1 --2- 172.21.15.60:0/3033833467 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6030140410 0x7f6030065250 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0x7f602c002f70 tx=0x7f602c006eb0).ready entity=mon.0 client_cookie=d7a090e5fe930227 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:57.326 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.313+0000 7f6035521700 1 -- 172.21.15.60:0/3033833467 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f602c00fa60 con 0x7f6030140410 2022-06-29T16:05:57.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.313+0000 7f6035521700 1 -- 172.21.15.60:0/3033833467 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f602c000e90 con 0x7f6030140410 2022-06-29T16:05:57.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.313+0000 7f6035521700 1 -- 172.21.15.60:0/3033833467 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f602c00fdf0 con 0x7f6030140410 2022-06-29T16:05:57.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.317+0000 7f6038787700 1 -- 172.21.15.60:0/3033833467 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6030140410 msgr2=0x7f6030065250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:57.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.317+0000 7f6038787700 1 --2- 172.21.15.60:0/3033833467 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6030140410 0x7f6030065250 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0x7f602c002f70 tx=0x7f602c006eb0).stop 2022-06-29T16:05:57.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.317+0000 7f6038787700 1 -- 172.21.15.60:0/3033833467 shutdown_connections 2022-06-29T16:05:57.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.317+0000 7f6038787700 1 --2- 172.21.15.60:0/3033833467 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6030140410 0x7f6030065250 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:57.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.317+0000 7f6038787700 1 -- 172.21.15.60:0/3033833467 >> 172.21.15.60:0/3033833467 conn(0x7f60300632c0 msgr2=0x7f60300b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:57.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.317+0000 7f6038787700 1 -- 172.21.15.60:0/3033833467 shutdown_connections 2022-06-29T16:05:57.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.317+0000 7f6038787700 1 -- 172.21.15.60:0/3033833467 wait complete. 2022-06-29T16:05:57.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.317+0000 7f6038787700 1 Processor -- start 2022-06-29T16:05:57.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.317+0000 7f6038787700 1 -- start start 2022-06-29T16:05:57.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.317+0000 7f6038787700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6030064b50 0x7f60300b1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:57.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.317+0000 7f6038787700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60300b5950 0x7f6030151e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:57.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.317+0000 7f6038787700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6030140410 0x7f6030155b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:57.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.317+0000 7f6038787700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60301554a0 con 0x7f6030140410 2022-06-29T16:05:57.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.317+0000 7f6038787700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6030153040 con 0x7f60300b5950 2022-06-29T16:05:57.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.317+0000 7f6038787700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603012c590 con 0x7f6030064b50 2022-06-29T16:05:57.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.317+0000 7f6036d24700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6030140410 0x7f6030155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:57.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.317+0000 7f6036d24700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6030140410 0x7f6030155b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51122/0 (socket says 172.21.15.60:51122) 2022-06-29T16:05:57.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.317+0000 7f6036d24700 1 -- 172.21.15.60:0/2735394084 learned_addr learned my addr 172.21.15.60:0/2735394084 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:57.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.317+0000 7f6035d22700 1 --2- 172.21.15.60:0/2735394084 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60300b5950 0x7f6030151e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:57.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.317+0000 7f6036523700 1 --2- 172.21.15.60:0/2735394084 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6030064b50 0x7f60300b1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:57.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.317+0000 7f6036d24700 1 -- 172.21.15.60:0/2735394084 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6030064b50 msgr2=0x7f60300b1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:57.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.317+0000 7f6036d24700 1 --2- 172.21.15.60:0/2735394084 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6030064b50 0x7f60300b1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:57.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.317+0000 7f6036d24700 1 -- 172.21.15.60:0/2735394084 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60300b5950 msgr2=0x7f6030151e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:57.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.317+0000 7f6036d24700 1 --2- 172.21.15.60:0/2735394084 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60300b5950 0x7f6030151e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:57.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.317+0000 7f6036d24700 1 -- 172.21.15.60:0/2735394084 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f602c00b1e0 con 0x7f6030140410 2022-06-29T16:05:57.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.317+0000 7f6036d24700 1 --2- 172.21.15.60:0/2735394084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6030140410 0x7f6030155b70 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 rx=0x7f60300b2a60 tx=0x7f6028007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:57.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.317+0000 7f601f7fe700 1 -- 172.21.15.60:0/2735394084 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f602800e4a0 con 0x7f6030140410 2022-06-29T16:05:57.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.317+0000 7f601f7fe700 1 -- 172.21.15.60:0/2735394084 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f602800e600 con 0x7f6030140410 2022-06-29T16:05:57.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.317+0000 7f6038787700 1 -- 172.21.15.60:0/2735394084 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60301556d0 con 0x7f6030140410 2022-06-29T16:05:57.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.317+0000 7f601f7fe700 1 -- 172.21.15.60:0/2735394084 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f602800e960 con 0x7f6030140410 2022-06-29T16:05:57.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.317+0000 7f6038787700 1 -- 172.21.15.60:0/2735394084 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f603013e270 con 0x7f6030140410 2022-06-29T16:05:57.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.321+0000 7f601f7fe700 1 -- 172.21.15.60:0/2735394084 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6028019030 con 0x7f6030140410 2022-06-29T16:05:57.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.321+0000 7f6038787700 1 -- 172.21.15.60:0/2735394084 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f603013f860 con 0x7f6030140410 2022-06-29T16:05:57.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.321+0000 7f601f7fe700 1 --2- 172.21.15.60:0/2735394084 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6010053aa0 0x7f6010055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:57.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.321+0000 7f601f7fe700 1 -- 172.21.15.60:0/2735394084 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f602806b370 con 0x7f6030140410 2022-06-29T16:05:57.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.321+0000 7f6036523700 1 --2- 172.21.15.60:0/2735394084 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6010053aa0 0x7f6010055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:57.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.321+0000 7f6036523700 1 --2- 172.21.15.60:0/2735394084 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6010053aa0 0x7f6010055f60 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7f602000b000 tx=0x7f6020007160).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:57.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.325+0000 7f601f7fe700 1 -- 172.21.15.60:0/2735394084 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f60280424d0 con 0x7f6030140410 2022-06-29T16:05:57.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.457+0000 7f6038787700 1 -- 172.21.15.60:0/2735394084 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f6030155860 con 0x7f6030140410 2022-06-29T16:05:57.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.465+0000 7f601f7fe700 1 -- 172.21.15.60:0/2735394084 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6028045af0 con 0x7f6030140410 2022-06-29T16:05:57.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.469+0000 7f6038787700 1 -- 172.21.15.60:0/2735394084 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6010053aa0 msgr2=0x7f6010055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:57.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.469+0000 7f6038787700 1 --2- 172.21.15.60:0/2735394084 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6010053aa0 0x7f6010055f60 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7f602000b000 tx=0x7f6020007160).stop 2022-06-29T16:05:57.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.469+0000 7f6038787700 1 -- 172.21.15.60:0/2735394084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6030140410 msgr2=0x7f6030155b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:57.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.469+0000 7f6038787700 1 --2- 172.21.15.60:0/2735394084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6030140410 0x7f6030155b70 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 rx=0x7f60300b2a60 tx=0x7f6028007f40).stop 2022-06-29T16:05:57.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.469+0000 7f6038787700 1 -- 172.21.15.60:0/2735394084 shutdown_connections 2022-06-29T16:05:57.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.469+0000 7f6038787700 1 --2- 172.21.15.60:0/2735394084 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6010053aa0 0x7f6010055f60 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:57.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.469+0000 7f6038787700 1 --2- 172.21.15.60:0/2735394084 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6030140410 0x7f6030155b70 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:57.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.469+0000 7f6038787700 1 --2- 172.21.15.60:0/2735394084 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60300b5950 0x7f6030151e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:57.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.469+0000 7f6038787700 1 --2- 172.21.15.60:0/2735394084 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6030064b50 0x7f60300b1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:57.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.469+0000 7f6038787700 1 -- 172.21.15.60:0/2735394084 >> 172.21.15.60:0/2735394084 conn(0x7f60300632c0 msgr2=0x7f6030142580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:57.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.469+0000 7f6038787700 1 -- 172.21.15.60:0/2735394084 shutdown_connections 2022-06-29T16:05:57.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.469+0000 7f6038787700 1 -- 172.21.15.60:0/2735394084 wait complete. 2022-06-29T16:05:57.485 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_dump_inode_hexademical' 2022-06-29T16:05:57.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.581+0000 7ff3e1f13700 1 Processor -- start 2022-06-29T16:05:57.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.581+0000 7ff3e1f13700 1 -- start start 2022-06-29T16:05:57.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.581+0000 7ff3e1f13700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff3dc0b5950 0x7ff3dc067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:57.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.581+0000 7ff3e1f13700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff3dc063ad0 0x7ff3dc14af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:57.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.581+0000 7ff3e1f13700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3dc05bd20 0x7ff3dc05c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:57.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.581+0000 7ff3e1f13700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3dc0acba0 con 0x7ff3dc0b5950 2022-06-29T16:05:57.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.581+0000 7ff3e1f13700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3dc067220 con 0x7ff3dc063ad0 2022-06-29T16:05:57.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.581+0000 7ff3e1f13700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc0b5d30 con 0x7ff3dc05bd20 2022-06-29T16:05:57.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.581+0000 7ff3db7fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff3dc0b5950 0x7ff3dc067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:57.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.581+0000 7ff3db7fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff3dc0b5950 0x7ff3dc067680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51126/0 (socket says 172.21.15.60:51126) 2022-06-29T16:05:57.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.581+0000 7ff3db7fe700 1 -- 172.21.15.60:0/2139657883 learned_addr learned my addr 172.21.15.60:0/2139657883 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:57.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.581+0000 7ff3d2ffd700 1 --2- 172.21.15.60:0/2139657883 >> v2:172.21.15.92:3300/0 conn(0x7ff3dc063ad0 0x7ff3dc14af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:57.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.581+0000 7ff3dbfff700 1 --2- 172.21.15.60:0/2139657883 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3dc05bd20 0x7ff3dc05c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:57.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.581+0000 7ff3db7fe700 1 -- 172.21.15.60:0/2139657883 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3dc05bd20 msgr2=0x7ff3dc05c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:57.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.581+0000 7ff3db7fe700 1 --2- 172.21.15.60:0/2139657883 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3dc05bd20 0x7ff3dc05c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:57.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.581+0000 7ff3db7fe700 1 -- 172.21.15.60:0/2139657883 >> v2:172.21.15.92:3300/0 conn(0x7ff3dc063ad0 msgr2=0x7ff3dc14af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:57.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.581+0000 7ff3db7fe700 1 --2- 172.21.15.60:0/2139657883 >> v2:172.21.15.92:3300/0 conn(0x7ff3dc063ad0 0x7ff3dc14af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:57.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.581+0000 7ff3db7fe700 1 -- 172.21.15.60:0/2139657883 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3dc0ace00 con 0x7ff3dc0b5950 2022-06-29T16:05:57.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.581+0000 7ff3db7fe700 1 --2- 172.21.15.60:0/2139657883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3dc0b5950 0x7ff3dc067680 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 rx=0x7ff3cc002f70 tx=0x7ff3cc006eb0).ready entity=mon.0 client_cookie=e57108ca2a2fa5e0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:57.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.581+0000 7ff3daffd700 1 -- 172.21.15.60:0/2139657883 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3cc00d570 con 0x7ff3dc0b5950 2022-06-29T16:05:57.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.581+0000 7ff3daffd700 1 -- 172.21.15.60:0/2139657883 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3cc000e90 con 0x7ff3dc0b5950 2022-06-29T16:05:57.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.581+0000 7ff3daffd700 1 -- 172.21.15.60:0/2139657883 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3cc00d900 con 0x7ff3dc0b5950 2022-06-29T16:05:57.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.581+0000 7ff3e1f13700 1 -- 172.21.15.60:0/2139657883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3dc0b5950 msgr2=0x7ff3dc067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:57.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.581+0000 7ff3e1f13700 1 --2- 172.21.15.60:0/2139657883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3dc0b5950 0x7ff3dc067680 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 rx=0x7ff3cc002f70 tx=0x7ff3cc006eb0).stop 2022-06-29T16:05:57.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.581+0000 7ff3e1f13700 1 -- 172.21.15.60:0/2139657883 shutdown_connections 2022-06-29T16:05:57.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.581+0000 7ff3e1f13700 1 --2- 172.21.15.60:0/2139657883 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3dc05bd20 0x7ff3dc05c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:57.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.581+0000 7ff3e1f13700 1 --2- 172.21.15.60:0/2139657883 >> v2:172.21.15.92:3300/0 conn(0x7ff3dc063ad0 0x7ff3dc14af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:57.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.581+0000 7ff3e1f13700 1 --2- 172.21.15.60:0/2139657883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3dc0b5950 0x7ff3dc067680 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:57.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.581+0000 7ff3e1f13700 1 -- 172.21.15.60:0/2139657883 >> 172.21.15.60:0/2139657883 conn(0x7ff3dc0632c0 msgr2=0x7ff3dc0b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:57.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.581+0000 7ff3e1f13700 1 -- 172.21.15.60:0/2139657883 shutdown_connections 2022-06-29T16:05:57.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.581+0000 7ff3e1f13700 1 -- 172.21.15.60:0/2139657883 wait complete. 2022-06-29T16:05:57.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.585+0000 7ff3e1f13700 1 Processor -- start 2022-06-29T16:05:57.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.585+0000 7ff3e1f13700 1 -- start start 2022-06-29T16:05:57.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.585+0000 7ff3e1f13700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3dc05bd20 0x7ff3dc1488e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:57.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.585+0000 7ff3e1f13700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3dc063ad0 0x7ff3dc155f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:57.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.585+0000 7ff3e1f13700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3dc0b5950 0x7ff3dc159a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:57.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.585+0000 7ff3e1f13700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc156430 con 0x7ff3dc063ad0 2022-06-29T16:05:57.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.585+0000 7ff3e1f13700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc159620 con 0x7ff3dc05bd20 2022-06-29T16:05:57.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.585+0000 7ff3e1f13700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc116c40 con 0x7ff3dc0b5950 2022-06-29T16:05:57.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.585+0000 7ff3d2ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3dc063ad0 0x7ff3dc155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:57.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.585+0000 7ff3d2ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3dc063ad0 0x7ff3dc155f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51132/0 (socket says 172.21.15.60:51132) 2022-06-29T16:05:57.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.585+0000 7ff3d2ffd700 1 -- 172.21.15.60:0/1583292194 learned_addr learned my addr 172.21.15.60:0/1583292194 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:57.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.585+0000 7ff3db7fe700 1 --2- 172.21.15.60:0/1583292194 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3dc05bd20 0x7ff3dc1488e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:57.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.585+0000 7ff3dbfff700 1 --2- 172.21.15.60:0/1583292194 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3dc0b5950 0x7ff3dc159a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:57.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.585+0000 7ff3d2ffd700 1 -- 172.21.15.60:0/1583292194 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3dc0b5950 msgr2=0x7ff3dc159a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:57.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.585+0000 7ff3d2ffd700 1 --2- 172.21.15.60:0/1583292194 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3dc0b5950 0x7ff3dc159a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:57.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.585+0000 7ff3d2ffd700 1 -- 172.21.15.60:0/1583292194 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3dc05bd20 msgr2=0x7ff3dc1488e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:57.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.585+0000 7ff3d2ffd700 1 --2- 172.21.15.60:0/1583292194 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3dc05bd20 0x7ff3dc1488e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:57.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.585+0000 7ff3d2ffd700 1 -- 172.21.15.60:0/1583292194 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3cc00b1e0 con 0x7ff3dc063ad0 2022-06-29T16:05:57.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.585+0000 7ff3db7fe700 1 --2- 172.21.15.60:0/1583292194 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3dc05bd20 0x7ff3dc1488e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:57.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.585+0000 7ff3d2ffd700 1 --2- 172.21.15.60:0/1583292194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3dc063ad0 0x7ff3dc155f40 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 rx=0x7ff3dc0ad030 tx=0x7ff3c4008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:57.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.585+0000 7ff3d97fa700 1 -- 172.21.15.60:0/1583292194 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3c4011780 con 0x7ff3dc063ad0 2022-06-29T16:05:57.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.585+0000 7ff3d97fa700 1 -- 172.21.15.60:0/1583292194 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c40118e0 con 0x7ff3dc063ad0 2022-06-29T16:05:57.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.585+0000 7ff3e1f13700 1 -- 172.21.15.60:0/1583292194 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3dc1169a0 con 0x7ff3dc063ad0 2022-06-29T16:05:57.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.585+0000 7ff3d97fa700 1 -- 172.21.15.60:0/1583292194 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3c4011c40 con 0x7ff3dc063ad0 2022-06-29T16:05:57.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.585+0000 7ff3e1f13700 1 -- 172.21.15.60:0/1583292194 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3dc0ad720 con 0x7ff3dc063ad0 2022-06-29T16:05:57.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.589+0000 7ff3d97fa700 1 -- 172.21.15.60:0/1583292194 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff3c4012420 con 0x7ff3dc063ad0 2022-06-29T16:05:57.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.589+0000 7ff3e1f13700 1 -- 172.21.15.60:0/1583292194 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3dc0573d0 con 0x7ff3dc063ad0 2022-06-29T16:05:57.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.589+0000 7ff3d97fa700 1 --2- 172.21.15.60:0/1583292194 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff3c00539e0 0x7ff3c0055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:57.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.589+0000 7ff3d97fa700 1 -- 172.21.15.60:0/1583292194 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5339+0+0 (secure 0 0 0) 0x7ff3c406bbb0 con 0x7ff3dc063ad0 2022-06-29T16:05:57.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.589+0000 7ff3db7fe700 1 --2- 172.21.15.60:0/1583292194 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff3c00539e0 0x7ff3c0055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:57.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.589+0000 7ff3db7fe700 1 --2- 172.21.15.60:0/1583292194 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff3c00539e0 0x7ff3c0055ea0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7ff3cc01a000 tx=0x7ff3cc015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:57.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.597+0000 7ff3d97fa700 1 -- 172.21.15.60:0/1583292194 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff3c4042c40 con 0x7ff3dc063ad0 2022-06-29T16:05:57.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:57.761+0000 7ff3e1f13700 1 -- 172.21.15.60:0/1583292194 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_dump_inode_hexademical"]} v 0) v1 -- 0x7ff3dc0684b0 con 0x7ff3dc063ad0 2022-06-29T16:05:58.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.185+0000 7ff3d97fa700 1 -- 172.21.15.60:0/1583292194 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_dump_inode_hexademical"]}]=0 v337) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff3c4046260 con 0x7ff3dc063ad0 2022-06-29T16:05:58.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.189+0000 7ff3e1f13700 1 -- 172.21.15.60:0/1583292194 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff3c00539e0 msgr2=0x7ff3c0055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:58.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.189+0000 7ff3e1f13700 1 --2- 172.21.15.60:0/1583292194 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff3c00539e0 0x7ff3c0055ea0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7ff3cc01a000 tx=0x7ff3cc015000).stop 2022-06-29T16:05:58.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.189+0000 7ff3e1f13700 1 -- 172.21.15.60:0/1583292194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3dc063ad0 msgr2=0x7ff3dc155f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:58.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.189+0000 7ff3e1f13700 1 --2- 172.21.15.60:0/1583292194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3dc063ad0 0x7ff3dc155f40 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 rx=0x7ff3dc0ad030 tx=0x7ff3c4008040).stop 2022-06-29T16:05:58.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.189+0000 7ff3e1f13700 1 -- 172.21.15.60:0/1583292194 shutdown_connections 2022-06-29T16:05:58.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.189+0000 7ff3e1f13700 1 --2- 172.21.15.60:0/1583292194 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff3c00539e0 0x7ff3c0055ea0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:58.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.189+0000 7ff3e1f13700 1 --2- 172.21.15.60:0/1583292194 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3dc0b5950 0x7ff3dc159a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:58.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.189+0000 7ff3e1f13700 1 --2- 172.21.15.60:0/1583292194 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3dc063ad0 0x7ff3dc155f40 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:58.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.189+0000 7ff3e1f13700 1 --2- 172.21.15.60:0/1583292194 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3dc05bd20 0x7ff3dc1488e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:58.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.189+0000 7ff3e1f13700 1 -- 172.21.15.60:0/1583292194 >> 172.21.15.60:0/1583292194 conn(0x7ff3dc0632c0 msgr2=0x7ff3dc145f50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:58.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.189+0000 7ff3e1f13700 1 -- 172.21.15.60:0/1583292194 shutdown_connections 2022-06-29T16:05:58.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.189+0000 7ff3e1f13700 1 -- 172.21.15.60:0/1583292194 wait complete. 2022-06-29T16:05:58.210 INFO:tasks.cephfs_test_runner:test_dump_inode_hexademical (tasks.cephfs.test_misc.TestMisc) ... ok 2022-06-29T16:05:58.211 INFO:tasks.cephfs_test_runner:Starting test: test_filtered_df (tasks.cephfs.test_misc.TestMisc) 2022-06-29T16:05:58.212 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_filtered_df' 2022-06-29T16:05:58.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.317+0000 7f97d759e700 1 Processor -- start 2022-06-29T16:05:58.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.317+0000 7f97d759e700 1 -- start start 2022-06-29T16:05:58.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d759e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f97d80b28b0 0x7f97d813f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:58.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d759e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97d8142d60 con 0x7f97d81438a0 2022-06-29T16:05:58.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d759e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97d80654f0 con 0x7f97d8060650 2022-06-29T16:05:58.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d759e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97d8065790 con 0x7f97d80b28b0 2022-06-29T16:05:58.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d6d9d700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f97d81438a0 0x7f97d813c300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59228/0 (socket says 172.21.15.60:59228) 2022-06-29T16:05:58.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d6d9d700 1 -- 172.21.15.60:0/152884225 learned_addr learned my addr 172.21.15.60:0/152884225 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:58.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d659c700 1 --2- 172.21.15.60:0/152884225 >> v2:172.21.15.92:3300/0 conn(0x7f97d80b28b0 0x7f97d813f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:58.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d559a700 1 -- 172.21.15.60:0/152884225 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2243176593 0 0) 0x7f97d8142d60 con 0x7f97d81438a0 2022-06-29T16:05:58.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d559a700 1 -- 172.21.15.60:0/152884225 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97b8003260 con 0x7f97d81438a0 2022-06-29T16:05:58.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d559a700 1 -- 172.21.15.60:0/152884225 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 501187208 0 0) 0x7f97d80654f0 con 0x7f97d8060650 2022-06-29T16:05:58.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d559a700 1 -- 172.21.15.60:0/152884225 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97d8142d60 con 0x7f97d8060650 2022-06-29T16:05:58.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d559a700 1 -- 172.21.15.60:0/152884225 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1641648494 0 0) 0x7f97b8003260 con 0x7f97d81438a0 2022-06-29T16:05:58.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d559a700 1 -- 172.21.15.60:0/152884225 >> v2:172.21.15.92:3300/0 conn(0x7f97d80b28b0 msgr2=0x7f97d813f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:58.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d559a700 1 --2- 172.21.15.60:0/152884225 >> v2:172.21.15.92:3300/0 conn(0x7f97d80b28b0 0x7f97d813f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:58.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d559a700 1 -- 172.21.15.60:0/152884225 >> v1:172.21.15.92:6789/0 conn(0x7f97d8060650 legacy=0x7f97d8143190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:58.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d559a700 1 -- 172.21.15.60:0/152884225 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d8060c40 con 0x7f97d81438a0 2022-06-29T16:05:58.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d559a700 1 -- 172.21.15.60:0/152884225 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f97d00019e0 con 0x7f97d81438a0 2022-06-29T16:05:58.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d559a700 1 -- 172.21.15.60:0/152884225 >> v1:172.21.15.60:6789/0 conn(0x7f97d81438a0 legacy=0x7f97d813c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:58.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d559a700 1 --2- 172.21.15.60:0/152884225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97b8003930 0x7f97b8003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:58.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d559a700 1 --2- 172.21.15.60:0/152884225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97b8004500 0x7f97b8004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:58.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d559a700 1 --2- 172.21.15.60:0/152884225 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97b8004e70 0x7f97b800b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:58.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d559a700 1 -- 172.21.15.60:0/152884225 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d8065790 con 0x7f97b8004500 2022-06-29T16:05:58.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d559a700 1 -- 172.21.15.60:0/152884225 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b800bdd0 con 0x7f97b8003930 2022-06-29T16:05:58.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d559a700 1 -- 172.21.15.60:0/152884225 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97b800bf90 con 0x7f97b8004e70 2022-06-29T16:05:58.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d5d9b700 1 --2- 172.21.15.60:0/152884225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97b8004500 0x7f97b8004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:58.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d659c700 1 --2- 172.21.15.60:0/152884225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97b8003930 0x7f97b8003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:58.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d6d9d700 1 --2- 172.21.15.60:0/152884225 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97b8004e70 0x7f97b800b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:58.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d5d9b700 1 -- 172.21.15.60:0/152884225 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97b8004e70 msgr2=0x7f97b800b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:58.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d5d9b700 1 --2- 172.21.15.60:0/152884225 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97b8004e70 0x7f97b800b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:58.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d5d9b700 1 -- 172.21.15.60:0/152884225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97b8003930 msgr2=0x7f97b8003d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:58.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d5d9b700 1 --2- 172.21.15.60:0/152884225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97b8003930 0x7f97b8003d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:58.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d5d9b700 1 -- 172.21.15.60:0/152884225 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b8003260 con 0x7f97b8004500 2022-06-29T16:05:58.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d6d9d700 1 --2- 172.21.15.60:0/152884225 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97b8004e70 0x7f97b800b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:05:58.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d6d9d700 1 -- 172.21.15.60:0/152884225 reap_dead start 2022-06-29T16:05:58.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d5d9b700 1 --2- 172.21.15.60:0/152884225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97b8004500 0x7f97b8004980 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0x7f97c0001110 tx=0x7f97c0007a00).ready entity=mon.0 client_cookie=1eb3254b0bdecb78 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:58.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d559a700 1 -- 172.21.15.60:0/152884225 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f97c0007d10 con 0x7f97b8004500 2022-06-29T16:05:58.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d559a700 1 -- 172.21.15.60:0/152884225 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97c0007030 con 0x7f97b8004500 2022-06-29T16:05:58.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d559a700 1 -- 172.21.15.60:0/152884225 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f97c0007390 con 0x7f97b8004500 2022-06-29T16:05:58.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d759e700 1 -- 172.21.15.60:0/152884225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97b8004500 msgr2=0x7f97b8004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:58.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d759e700 1 --2- 172.21.15.60:0/152884225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97b8004500 0x7f97b8004980 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0x7f97c0001110 tx=0x7f97c0007a00).stop 2022-06-29T16:05:58.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d759e700 1 -- 172.21.15.60:0/152884225 shutdown_connections 2022-06-29T16:05:58.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.321+0000 7f97d759e700 1 --2- 172.21.15.60:0/152884225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97b8004500 0x7f97b8004980 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:58.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.325+0000 7f97d759e700 1 -- 172.21.15.60:0/152884225 >> 172.21.15.60:0/152884225 conn(0x7f97d80637c0 msgr2=0x7f97d8063ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:58.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.325+0000 7f97d759e700 1 -- 172.21.15.60:0/152884225 shutdown_connections 2022-06-29T16:05:58.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.325+0000 7f97d759e700 1 -- 172.21.15.60:0/152884225 wait complete. 2022-06-29T16:05:58.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.325+0000 7f97d759e700 1 Processor -- start 2022-06-29T16:05:58.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.325+0000 7f97d759e700 1 -- start start 2022-06-29T16:05:58.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.325+0000 7f97d759e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97b8004500 0x7f97d813cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:58.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.325+0000 7f97d759e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97d8142200 0x7f97d8150d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:58.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.325+0000 7f97d759e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97d80b5720 0x7f97d80b3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:58.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.325+0000 7f97d759e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d80b5bc0 con 0x7f97b8004500 2022-06-29T16:05:58.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.325+0000 7f97d759e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d8140df0 con 0x7f97d80b5720 2022-06-29T16:05:58.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.325+0000 7f97d759e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97d81511f0 con 0x7f97d8142200 2022-06-29T16:05:58.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.325+0000 7f97d659c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97b8004500 0x7f97d813cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:58.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.325+0000 7f97d5d9b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97d8142200 0x7f97d8150d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:58.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.325+0000 7f97d659c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97b8004500 0x7f97d813cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51150/0 (socket says 172.21.15.60:51150) 2022-06-29T16:05:58.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.325+0000 7f97d659c700 1 -- 172.21.15.60:0/2768568252 learned_addr learned my addr 172.21.15.60:0/2768568252 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:58.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.325+0000 7f97d6d9d700 1 --2- 172.21.15.60:0/2768568252 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97d80b5720 0x7f97d80b3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:58.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.329+0000 7f97d5d9b700 1 -- 172.21.15.60:0/2768568252 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97d80b5720 msgr2=0x7f97d80b3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:58.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.329+0000 7f97d5d9b700 1 --2- 172.21.15.60:0/2768568252 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97d80b5720 0x7f97d80b3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:58.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.329+0000 7f97d5d9b700 1 -- 172.21.15.60:0/2768568252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97b8004500 msgr2=0x7f97d813cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:58.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.329+0000 7f97d5d9b700 1 --2- 172.21.15.60:0/2768568252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97b8004500 0x7f97d813cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:58.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.329+0000 7f97d5d9b700 1 -- 172.21.15.60:0/2768568252 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d81513b0 con 0x7f97d8142200 2022-06-29T16:05:58.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.329+0000 7f97d659c700 1 --2- 172.21.15.60:0/2768568252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97b8004500 0x7f97d813cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:05:58.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.329+0000 7f97d5d9b700 1 --2- 172.21.15.60:0/2768568252 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97d8142200 0x7f97d8150d00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f97c0001110 tx=0x7f97c0015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:58.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.329+0000 7f97cf7fe700 1 -- 172.21.15.60:0/2768568252 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f97c0007d10 con 0x7f97d8142200 2022-06-29T16:05:58.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.329+0000 7f97cf7fe700 1 -- 172.21.15.60:0/2768568252 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97c0020480 con 0x7f97d8142200 2022-06-29T16:05:58.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.329+0000 7f97d759e700 1 -- 172.21.15.60:0/2768568252 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97d80b41d0 con 0x7f97d8142200 2022-06-29T16:05:58.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.329+0000 7f97cf7fe700 1 -- 172.21.15.60:0/2768568252 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f97c00207e0 con 0x7f97d8142200 2022-06-29T16:05:58.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.329+0000 7f97d759e700 1 -- 172.21.15.60:0/2768568252 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97d8142800 con 0x7f97d8142200 2022-06-29T16:05:58.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.329+0000 7f97d759e700 1 -- 172.21.15.60:0/2768568252 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97d81511f0 con 0x7f97d8142200 2022-06-29T16:05:58.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.329+0000 7f97cf7fe700 1 -- 172.21.15.60:0/2768568252 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f97c0010070 con 0x7f97d8142200 2022-06-29T16:05:58.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.333+0000 7f97cf7fe700 1 --2- 172.21.15.60:0/2768568252 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f97bc053b30 0x7f97bc055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:58.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.333+0000 7f97cf7fe700 1 -- 172.21.15.60:0/2768568252 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f97c006c6d0 con 0x7f97d8142200 2022-06-29T16:05:58.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.333+0000 7f97d659c700 1 --2- 172.21.15.60:0/2768568252 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f97bc053b30 0x7f97bc055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:58.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.333+0000 7f97d659c700 1 --2- 172.21.15.60:0/2768568252 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f97bc053b30 0x7f97bc055ff0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7f97c800a000 tx=0x7f97c8006e80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:58.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.341+0000 7f97cf7fe700 1 -- 172.21.15.60:0/2768568252 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f97c00437c0 con 0x7f97d8142200 2022-06-29T16:05:58.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:58.517+0000 7f97d759e700 1 -- 172.21.15.60:0/2768568252 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_filtered_df"]} v 0) v1 -- 0x7f97d8151380 con 0x7f97d8142200 2022-06-29T16:05:59.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.197+0000 7f97cf7fe700 1 -- 172.21.15.60:0/2768568252 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_filtered_df"]}]=0 v338) v1 ==== 130+0+0 (secure 0 0 0) 0x7f97c0046de0 con 0x7f97d8142200 2022-06-29T16:05:59.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.201+0000 7f97d759e700 1 -- 172.21.15.60:0/2768568252 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f97bc053b30 msgr2=0x7f97bc055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:59.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.201+0000 7f97d759e700 1 --2- 172.21.15.60:0/2768568252 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f97bc053b30 0x7f97bc055ff0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7f97c800a000 tx=0x7f97c8006e80).stop 2022-06-29T16:05:59.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.201+0000 7f97d759e700 1 -- 172.21.15.60:0/2768568252 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97d8142200 msgr2=0x7f97d8150d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:59.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.201+0000 7f97d759e700 1 --2- 172.21.15.60:0/2768568252 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97d8142200 0x7f97d8150d00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f97c0001110 tx=0x7f97c0015000).stop 2022-06-29T16:05:59.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.201+0000 7f97d759e700 1 -- 172.21.15.60:0/2768568252 shutdown_connections 2022-06-29T16:05:59.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.201+0000 7f97d759e700 1 --2- 172.21.15.60:0/2768568252 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f97bc053b30 0x7f97bc055ff0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:59.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.201+0000 7f97d759e700 1 --2- 172.21.15.60:0/2768568252 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97d80b5720 0x7f97d80b3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:59.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.201+0000 7f97d759e700 1 --2- 172.21.15.60:0/2768568252 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f97d8142200 0x7f97d8150d00 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:59.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.201+0000 7f97d759e700 1 --2- 172.21.15.60:0/2768568252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97b8004500 0x7f97d813cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:59.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.201+0000 7f97d759e700 1 -- 172.21.15.60:0/2768568252 >> 172.21.15.60:0/2768568252 conn(0x7f97d80637c0 msgr2=0x7f97d805c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:59.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.201+0000 7f97d759e700 1 -- 172.21.15.60:0/2768568252 shutdown_connections 2022-06-29T16:05:59.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.201+0000 7f97d759e700 1 -- 172.21.15.60:0/2768568252 wait complete. 2022-06-29T16:05:59.221 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-06-29T16:05:59.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.329+0000 7f5f6a6c6700 1 Processor -- start 2022-06-29T16:05:59.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.329+0000 7f5f6a6c6700 1 -- start start 2022-06-29T16:05:59.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.329+0000 7f5f6a6c6700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5f640ae7c0 0x7f5f64141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:59.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.329+0000 7f5f6a6c6700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5f640b4010 0x7f5f64142490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:59.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.329+0000 7f5f6a6c6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f64146160 0x7f5f641465d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:59.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.329+0000 7f5f6a6c6700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f64143a40 con 0x7f5f640b4010 2022-06-29T16:05:59.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.329+0000 7f5f6a6c6700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f640aee60 con 0x7f5f640ae7c0 2022-06-29T16:05:59.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.329+0000 7f5f6a6c6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f640af100 con 0x7f5f64146160 2022-06-29T16:05:59.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.329+0000 7f5f637fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5f640b4010 0x7f5f64142490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:59.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.329+0000 7f5f637fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5f640b4010 0x7f5f64142490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51158/0 (socket says 172.21.15.60:51158) 2022-06-29T16:05:59.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.329+0000 7f5f637fe700 1 -- 172.21.15.60:0/2000789569 learned_addr learned my addr 172.21.15.60:0/2000789569 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:59.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.329+0000 7f5f68c63700 1 --2- 172.21.15.60:0/2000789569 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f64146160 0x7f5f641465d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:59.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.329+0000 7f5f63fff700 1 --2- 172.21.15.60:0/2000789569 >> v2:172.21.15.92:3300/0 conn(0x7f5f640ae7c0 0x7f5f64141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:59.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.329+0000 7f5f637fe700 1 -- 172.21.15.60:0/2000789569 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f64146160 msgr2=0x7f5f641465d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:59.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.329+0000 7f5f637fe700 1 --2- 172.21.15.60:0/2000789569 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f64146160 0x7f5f641465d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:59.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.329+0000 7f5f637fe700 1 -- 172.21.15.60:0/2000789569 >> v2:172.21.15.92:3300/0 conn(0x7f5f640ae7c0 msgr2=0x7f5f64141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:59.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.329+0000 7f5f637fe700 1 --2- 172.21.15.60:0/2000789569 >> v2:172.21.15.92:3300/0 conn(0x7f5f640ae7c0 0x7f5f64141fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:59.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.329+0000 7f5f637fe700 1 -- 172.21.15.60:0/2000789569 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f640aeba0 con 0x7f5f640b4010 2022-06-29T16:05:59.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.329+0000 7f5f637fe700 1 --2- 172.21.15.60:0/2000789569 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f640b4010 0x7f5f64142490 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 rx=0x7f5f58002f70 tx=0x7f5f58006eb0).ready entity=mon.0 client_cookie=95c8a04cf6082271 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:59.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.329+0000 7f5f62ffd700 1 -- 172.21.15.60:0/2000789569 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f5800d570 con 0x7f5f640b4010 2022-06-29T16:05:59.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.329+0000 7f5f62ffd700 1 -- 172.21.15.60:0/2000789569 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f58000e90 con 0x7f5f640b4010 2022-06-29T16:05:59.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.329+0000 7f5f62ffd700 1 -- 172.21.15.60:0/2000789569 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f5800d900 con 0x7f5f640b4010 2022-06-29T16:05:59.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.329+0000 7f5f6a6c6700 1 -- 172.21.15.60:0/2000789569 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f640b4010 msgr2=0x7f5f64142490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:59.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.329+0000 7f5f6a6c6700 1 --2- 172.21.15.60:0/2000789569 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f640b4010 0x7f5f64142490 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 rx=0x7f5f58002f70 tx=0x7f5f58006eb0).stop 2022-06-29T16:05:59.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.329+0000 7f5f6a6c6700 1 -- 172.21.15.60:0/2000789569 shutdown_connections 2022-06-29T16:05:59.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.329+0000 7f5f6a6c6700 1 --2- 172.21.15.60:0/2000789569 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f64146160 0x7f5f641465d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:59.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.329+0000 7f5f6a6c6700 1 --2- 172.21.15.60:0/2000789569 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f640b4010 0x7f5f64142490 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:59.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.329+0000 7f5f6a6c6700 1 --2- 172.21.15.60:0/2000789569 >> v2:172.21.15.92:3300/0 conn(0x7f5f640ae7c0 0x7f5f64141fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:59.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.329+0000 7f5f6a6c6700 1 -- 172.21.15.60:0/2000789569 >> 172.21.15.60:0/2000789569 conn(0x7f5f640b9340 msgr2=0x7f5f640b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:59.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.329+0000 7f5f6a6c6700 1 -- 172.21.15.60:0/2000789569 shutdown_connections 2022-06-29T16:05:59.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.329+0000 7f5f6a6c6700 1 -- 172.21.15.60:0/2000789569 wait complete. 2022-06-29T16:05:59.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.333+0000 7f5f6a6c6700 1 Processor -- start 2022-06-29T16:05:59.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.333+0000 7f5f6a6c6700 1 -- start start 2022-06-29T16:05:59.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.333+0000 7f5f6a6c6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f640ae7c0 0x7f5f640b3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:59.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.333+0000 7f5f6a6c6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f640b4010 0x7f5f6413f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:59.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.333+0000 7f5f6a6c6700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f64146160 0x7f5f64155300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:59.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.333+0000 7f5f6a6c6700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f64148f80 con 0x7f5f640ae7c0 2022-06-29T16:05:59.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.333+0000 7f5f6a6c6700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f640b2e40 con 0x7f5f64146160 2022-06-29T16:05:59.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.333+0000 7f5f6a6c6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f640b2b00 con 0x7f5f640b4010 2022-06-29T16:05:59.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.333+0000 7f5f63fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f640ae7c0 0x7f5f640b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:59.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.333+0000 7f5f63fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f640ae7c0 0x7f5f640b3530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51166/0 (socket says 172.21.15.60:51166) 2022-06-29T16:05:59.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.333+0000 7f5f637fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f640b4010 0x7f5f6413f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:59.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.333+0000 7f5f63fff700 1 -- 172.21.15.60:0/3568883112 learned_addr learned my addr 172.21.15.60:0/3568883112 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:59.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.333+0000 7f5f68c63700 1 --2- 172.21.15.60:0/3568883112 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f64146160 0x7f5f64155300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:59.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.333+0000 7f5f63fff700 1 -- 172.21.15.60:0/3568883112 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f640b4010 msgr2=0x7f5f6413f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:59.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.333+0000 7f5f63fff700 1 --2- 172.21.15.60:0/3568883112 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f640b4010 0x7f5f6413f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:59.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.333+0000 7f5f63fff700 1 -- 172.21.15.60:0/3568883112 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f64146160 msgr2=0x7f5f64155300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:59.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.333+0000 7f5f63fff700 1 --2- 172.21.15.60:0/3568883112 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f64146160 0x7f5f64155300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:59.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.333+0000 7f5f63fff700 1 -- 172.21.15.60:0/3568883112 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f5800b1e0 con 0x7f5f640ae7c0 2022-06-29T16:05:59.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.333+0000 7f5f63fff700 1 --2- 172.21.15.60:0/3568883112 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f640ae7c0 0x7f5f640b3530 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 rx=0x7f5f540030b0 tx=0x7f5f54008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:59.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.333+0000 7f5f617fa700 1 -- 172.21.15.60:0/3568883112 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f54011780 con 0x7f5f640ae7c0 2022-06-29T16:05:59.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.333+0000 7f5f617fa700 1 -- 172.21.15.60:0/3568883112 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f540118e0 con 0x7f5f640ae7c0 2022-06-29T16:05:59.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.333+0000 7f5f6a6c6700 1 -- 172.21.15.60:0/3568883112 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f640b3090 con 0x7f5f640ae7c0 2022-06-29T16:05:59.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.333+0000 7f5f617fa700 1 -- 172.21.15.60:0/3568883112 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f54011c40 con 0x7f5f640ae7c0 2022-06-29T16:05:59.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.333+0000 7f5f6a6c6700 1 -- 172.21.15.60:0/3568883112 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f641484a0 con 0x7f5f640ae7c0 2022-06-29T16:05:59.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.337+0000 7f5f6a6c6700 1 -- 172.21.15.60:0/3568883112 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f640573d0 con 0x7f5f640ae7c0 2022-06-29T16:05:59.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.337+0000 7f5f617fa700 1 -- 172.21.15.60:0/3568883112 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5f54012420 con 0x7f5f640ae7c0 2022-06-29T16:05:59.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.337+0000 7f5f617fa700 1 --2- 172.21.15.60:0/3568883112 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5f44053a70 0x7f5f44055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:59.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.337+0000 7f5f617fa700 1 -- 172.21.15.60:0/3568883112 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f5f5406b8f0 con 0x7f5f640ae7c0 2022-06-29T16:05:59.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.337+0000 7f5f637fe700 1 --2- 172.21.15.60:0/3568883112 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5f44053a70 0x7f5f44055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:59.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.337+0000 7f5f637fe700 1 --2- 172.21.15.60:0/3568883112 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5f44053a70 0x7f5f44055f30 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f5f5801c000 tx=0x7f5f58015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:59.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.345+0000 7f5f617fa700 1 -- 172.21.15.60:0/3568883112 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5f54042a90 con 0x7f5f640ae7c0 2022-06-29T16:05:59.400 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:05:59.404 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:05:59.407 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:05:59.410 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:05:59.413 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:05:59.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.473+0000 7f5f6a6c6700 1 -- 172.21.15.60:0/3568883112 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f5f6405fdb0 con 0x7f5f640ae7c0 2022-06-29T16:05:59.486 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.481+0000 7f5f617fa700 1 -- 172.21.15.60:0/3568883112 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5f540460b0 con 0x7f5f640ae7c0 2022-06-29T16:05:59.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.481+0000 7f5f6a6c6700 1 -- 172.21.15.60:0/3568883112 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5f44053a70 msgr2=0x7f5f44055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:59.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.481+0000 7f5f6a6c6700 1 --2- 172.21.15.60:0/3568883112 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5f44053a70 0x7f5f44055f30 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f5f5801c000 tx=0x7f5f58015000).stop 2022-06-29T16:05:59.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.481+0000 7f5f6a6c6700 1 -- 172.21.15.60:0/3568883112 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f640ae7c0 msgr2=0x7f5f640b3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:59.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.481+0000 7f5f6a6c6700 1 --2- 172.21.15.60:0/3568883112 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f640ae7c0 0x7f5f640b3530 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 rx=0x7f5f540030b0 tx=0x7f5f54008040).stop 2022-06-29T16:05:59.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.481+0000 7f5f6a6c6700 1 -- 172.21.15.60:0/3568883112 shutdown_connections 2022-06-29T16:05:59.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.481+0000 7f5f6a6c6700 1 --2- 172.21.15.60:0/3568883112 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5f44053a70 0x7f5f44055f30 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:59.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.481+0000 7f5f6a6c6700 1 --2- 172.21.15.60:0/3568883112 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f64146160 0x7f5f64155300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:59.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.485+0000 7f5f6a6c6700 1 --2- 172.21.15.60:0/3568883112 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f640b4010 0x7f5f6413f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:59.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.485+0000 7f5f6a6c6700 1 --2- 172.21.15.60:0/3568883112 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f640ae7c0 0x7f5f640b3530 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:59.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.485+0000 7f5f6a6c6700 1 -- 172.21.15.60:0/3568883112 >> 172.21.15.60:0/3568883112 conn(0x7f5f640b9340 msgr2=0x7f5f6405ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:59.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.485+0000 7f5f6a6c6700 1 -- 172.21.15.60:0/3568883112 shutdown_connections 2022-06-29T16:05:59.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.485+0000 7f5f6a6c6700 1 -- 172.21.15.60:0/3568883112 wait complete. 2022-06-29T16:05:59.501 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-06-29T16:05:59.501 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:05:59.520 DEBUG:teuthology.orchestra.run.smithi092:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:05:59.542 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:05:59.563 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:05:59.584 DEBUG:teuthology.orchestra.run.smithi197:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:05:59.606 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-06-29T16:05:59.606 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:59.607 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:05:59.607 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:05:59.617 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-06-29T16:05:59.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:05:59.618 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-06-29T16:05:59.619 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:59.620 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:05:59.620 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:05:59.637 INFO:teuthology.orchestra.run.smithi156.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-06-29T16:05:59.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:05:59.639 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-06-29T16:05:59.639 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:05:59.640 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:05:59.640 DEBUG:teuthology.orchestra.run.smithi197:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T16:05:59.648 INFO:teuthology.orchestra.run.smithi197.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-06-29T16:05:59.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:05:59.650 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-06-29T16:05:59.651 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-06-29T16:05:59.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.757+0000 7ff7cdaaf700 1 Processor -- start 2022-06-29T16:05:59.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.757+0000 7ff7cdaaf700 1 -- start start 2022-06-29T16:05:59.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.757+0000 7ff7cdaaf700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff7c813a920 0x7ff7c8143f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:59.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.757+0000 7ff7cdaaf700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff7c80ae270 0x7ff7c80b5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:59.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.757+0000 7ff7cdaaf700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7c813ad00 con 0x7ff7c80b5b00 2022-06-29T16:05:59.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.757+0000 7ff7cdaaf700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7c8144420 con 0x7ff7c813a920 2022-06-29T16:05:59.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.757+0000 7ff7cdaaf700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7c8144670 con 0x7ff7c80ae270 2022-06-29T16:05:59.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.757+0000 7ff7c77fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff7c813a920 0x7ff7c8143f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:59.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.757+0000 7ff7c7fff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff7c80b5b00 0x7ff7c8140ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59262/0 (socket says 172.21.15.60:59262) 2022-06-29T16:05:59.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.757+0000 7ff7c7fff700 1 -- 172.21.15.60:0/2890735176 learned_addr learned my addr 172.21.15.60:0/2890735176 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:05:59.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.757+0000 7ff7c6ffd700 1 --2- 172.21.15.60:0/2890735176 >> v2:172.21.15.92:3300/0 conn(0x7ff7c80ae270 0x7ff7c80b5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:59.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.757+0000 7ff7c67fc700 1 -- 172.21.15.60:0/2890735176 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2129982276 0 0) 0x7ff7c813ad00 con 0x7ff7c80b5b00 2022-06-29T16:05:59.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.757+0000 7ff7c67fc700 1 -- 172.21.15.60:0/2890735176 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7a8003260 con 0x7ff7c80b5b00 2022-06-29T16:05:59.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.757+0000 7ff7c77fe700 1 -- 172.21.15.60:0/2890735176 >> v2:172.21.15.92:3300/0 conn(0x7ff7c80ae270 msgr2=0x7ff7c80b5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:05:59.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.757+0000 7ff7c77fe700 1 --2- 172.21.15.60:0/2890735176 >> v2:172.21.15.92:3300/0 conn(0x7ff7c80ae270 0x7ff7c80b5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:59.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.757+0000 7ff7c77fe700 1 -- 172.21.15.60:0/2890735176 >> v1:172.21.15.60:6789/0 conn(0x7ff7c80b5b00 legacy=0x7ff7c8140ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:59.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.757+0000 7ff7c77fe700 1 -- 172.21.15.60:0/2890735176 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c8144910 con 0x7ff7c813a920 2022-06-29T16:05:59.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.757+0000 7ff7c77fe700 1 --2- 172.21.15.60:0/2890735176 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7c813a920 0x7ff7c8143f30 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 rx=0x7ff7b0002f70 tx=0x7ff7b0006df0).ready entity=mon.0 client_cookie=6b70ed56581994f5 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:59.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.761+0000 7ff7c67fc700 1 -- 172.21.15.60:0/2890735176 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff7b000d4f0 con 0x7ff7c813a920 2022-06-29T16:05:59.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.761+0000 7ff7c67fc700 1 -- 172.21.15.60:0/2890735176 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7b0000e90 con 0x7ff7c813a920 2022-06-29T16:05:59.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.761+0000 7ff7c67fc700 1 -- 172.21.15.60:0/2890735176 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff7b000d840 con 0x7ff7c813a920 2022-06-29T16:05:59.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.761+0000 7ff7cdaaf700 1 -- 172.21.15.60:0/2890735176 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7c813a920 msgr2=0x7ff7c8143f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:59.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.761+0000 7ff7cdaaf700 1 --2- 172.21.15.60:0/2890735176 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7c813a920 0x7ff7c8143f30 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 rx=0x7ff7b0002f70 tx=0x7ff7b0006df0).stop 2022-06-29T16:05:59.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.761+0000 7ff7cdaaf700 1 -- 172.21.15.60:0/2890735176 shutdown_connections 2022-06-29T16:05:59.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.761+0000 7ff7cdaaf700 1 --2- 172.21.15.60:0/2890735176 >> v2:172.21.15.92:3300/0 conn(0x7ff7c80ae270 0x7ff7c80b5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:59.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.761+0000 7ff7cdaaf700 1 --2- 172.21.15.60:0/2890735176 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7c813a920 0x7ff7c8143f30 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:59.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.761+0000 7ff7cdaaf700 1 -- 172.21.15.60:0/2890735176 >> 172.21.15.60:0/2890735176 conn(0x7ff7c80632c0 msgr2=0x7ff7c80b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:05:59.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.761+0000 7ff7cdaaf700 1 -- 172.21.15.60:0/2890735176 shutdown_connections 2022-06-29T16:05:59.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.761+0000 7ff7cdaaf700 1 -- 172.21.15.60:0/2890735176 wait complete. 2022-06-29T16:05:59.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.761+0000 7ff7cdaaf700 1 Processor -- start 2022-06-29T16:05:59.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.761+0000 7ff7cdaaf700 1 -- start start 2022-06-29T16:05:59.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.761+0000 7ff7cdaaf700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7c80ae270 0x7ff7c80b06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:59.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.761+0000 7ff7cdaaf700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7c80b5b00 0x7ff7c8152330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:59.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.761+0000 7ff7cdaaf700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff7c813a920 0x7ff7c8152820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:59.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.761+0000 7ff7cdaaf700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c8142730 con 0x7ff7c80ae270 2022-06-29T16:05:59.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.761+0000 7ff7cdaaf700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c81429d0 con 0x7ff7c80b5b00 2022-06-29T16:05:59.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.765+0000 7ff7cdaaf700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c8143a50 con 0x7ff7c813a920 2022-06-29T16:05:59.777 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.765+0000 7ff7c77fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7c80ae270 0x7ff7c80b06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:59.777 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.765+0000 7ff7c77fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7c80ae270 0x7ff7c80b06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51178/0 (socket says 172.21.15.60:51178) 2022-06-29T16:05:59.777 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.765+0000 7ff7c77fe700 1 -- 172.21.15.60:0/3191038005 learned_addr learned my addr 172.21.15.60:0/3191038005 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:05:59.777 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.765+0000 7ff7c6ffd700 1 --2- 172.21.15.60:0/3191038005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7c80b5b00 0x7ff7c8152330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:59.778 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.765+0000 7ff7c7fff700 1 --2- 172.21.15.60:0/3191038005 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff7c813a920 0x7ff7c8152820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:59.778 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.765+0000 7ff7c77fe700 1 -- 172.21.15.60:0/3191038005 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff7c813a920 msgr2=0x7ff7c8152820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:59.778 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.765+0000 7ff7c77fe700 1 --2- 172.21.15.60:0/3191038005 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff7c813a920 0x7ff7c8152820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:59.778 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.765+0000 7ff7c77fe700 1 -- 172.21.15.60:0/3191038005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7c80b5b00 msgr2=0x7ff7c8152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:05:59.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.765+0000 7ff7c77fe700 1 --2- 172.21.15.60:0/3191038005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7c80b5b00 0x7ff7c8152330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:05:59.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.765+0000 7ff7c77fe700 1 -- 172.21.15.60:0/3191038005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b000b180 con 0x7ff7c80ae270 2022-06-29T16:05:59.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.765+0000 7ff7c77fe700 1 --2- 172.21.15.60:0/3191038005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7c80ae270 0x7ff7c80b06f0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 rx=0x7ff7b0002f70 tx=0x7ff7b0014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:59.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.765+0000 7ff7c4ff9700 1 -- 172.21.15.60:0/3191038005 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff7b0016070 con 0x7ff7c80ae270 2022-06-29T16:05:59.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.765+0000 7ff7c4ff9700 1 -- 172.21.15.60:0/3191038005 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7b00139c0 con 0x7ff7c80ae270 2022-06-29T16:05:59.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.765+0000 7ff7cdaaf700 1 -- 172.21.15.60:0/3191038005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7c8142c70 con 0x7ff7c80ae270 2022-06-29T16:05:59.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.765+0000 7ff7c4ff9700 1 -- 172.21.15.60:0/3191038005 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff7b0013d20 con 0x7ff7c80ae270 2022-06-29T16:05:59.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.765+0000 7ff7cdaaf700 1 -- 172.21.15.60:0/3191038005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7c8152d10 con 0x7ff7c80ae270 2022-06-29T16:05:59.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.765+0000 7ff7cdaaf700 1 -- 172.21.15.60:0/3191038005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7c8144bb0 con 0x7ff7c80ae270 2022-06-29T16:05:59.781 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.769+0000 7ff7c4ff9700 1 -- 172.21.15.60:0/3191038005 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff7b0025030 con 0x7ff7c80ae270 2022-06-29T16:05:59.781 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.769+0000 7ff7c4ff9700 1 --2- 172.21.15.60:0/3191038005 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff7ac053b30 0x7ff7ac055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:05:59.781 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.769+0000 7ff7c4ff9700 1 -- 172.21.15.60:0/3191038005 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5339+0+0 (secure 0 0 0) 0x7ff7b006cc60 con 0x7ff7c80ae270 2022-06-29T16:05:59.782 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.769+0000 7ff7c6ffd700 1 --2- 172.21.15.60:0/3191038005 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff7ac053b30 0x7ff7ac055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:05:59.782 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.769+0000 7ff7c6ffd700 1 --2- 172.21.15.60:0/3191038005 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff7ac053b30 0x7ff7ac055ff0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7ff7b800b000 tx=0x7ff7b8006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:05:59.782 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.777+0000 7ff7c4ff9700 1 -- 172.21.15.60:0/3191038005 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff7b0043d50 con 0x7ff7c80ae270 2022-06-29T16:05:59.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.901+0000 7ff7cdaaf700 1 -- 172.21.15.60:0/3191038005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7ff7c80573d0 con 0x7ff7c80ae270 2022-06-29T16:05:59.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.905+0000 7ff7c4ff9700 1 -- 172.21.15.60:0/3191038005 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v94) v1 ==== 81+0+28828 (secure 0 0 0) 0x7ff7b0023020 con 0x7ff7c80ae270 2022-06-29T16:05:59.911 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:05:59.911 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:05:59.912 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 94, 2022-06-29T16:05:59.912 INFO:teuthology.orchestra.run.smithi060.stdout: "fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", 2022-06-29T16:05:59.912 INFO:teuthology.orchestra.run.smithi060.stdout: "created": "2022-06-29T15:58:07.965719+0000", 2022-06-29T16:05:59.912 INFO:teuthology.orchestra.run.smithi060.stdout: "modified": "2022-06-29T16:05:57.169634+0000", 2022-06-29T16:05:59.913 INFO:teuthology.orchestra.run.smithi060.stdout: "last_up_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:05:59.913 INFO:teuthology.orchestra.run.smithi060.stdout: "last_in_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:05:59.913 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-06-29T16:05:59.913 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_num": 5799936, 2022-06-29T16:05:59.914 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_set": [ 2022-06-29T16:05:59.914 INFO:teuthology.orchestra.run.smithi060.stdout: "pglog_hardlimit", 2022-06-29T16:05:59.914 INFO:teuthology.orchestra.run.smithi060.stdout: "purged_snapdirs", 2022-06-29T16:05:59.914 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_deletes", 2022-06-29T16:05:59.915 INFO:teuthology.orchestra.run.smithi060.stdout: "sortbitwise" 2022-06-29T16:05:59.915 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:05:59.915 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_version": 4, 2022-06-29T16:05:59.915 INFO:teuthology.orchestra.run.smithi060.stdout: "full_ratio": 0.94999998807907104, 2022-06-29T16:05:59.916 INFO:teuthology.orchestra.run.smithi060.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-06-29T16:05:59.916 INFO:teuthology.orchestra.run.smithi060.stdout: "nearfull_ratio": 0.85000002384185791, 2022-06-29T16:05:59.916 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_snapshot": "", 2022-06-29T16:05:59.916 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_max": 18, 2022-06-29T16:05:59.917 INFO:teuthology.orchestra.run.smithi060.stdout: "max_osd": 8, 2022-06-29T16:05:59.917 INFO:teuthology.orchestra.run.smithi060.stdout: "require_min_compat_client": "jewel", 2022-06-29T16:05:59.917 INFO:teuthology.orchestra.run.smithi060.stdout: "min_compat_client": "jewel", 2022-06-29T16:05:59.917 INFO:teuthology.orchestra.run.smithi060.stdout: "require_osd_release": "octopus", 2022-06-29T16:05:59.917 INFO:teuthology.orchestra.run.smithi060.stdout: "pools": [ 2022-06-29T16:05:59.918 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:59.918 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2022-06-29T16:05:59.918 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "device_health_metrics", 2022-06-29T16:05:59.918 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:10.961127+0000", 2022-06-29T16:05:59.919 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:05:59.919 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:05:59.919 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:05:59.919 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:05:59.920 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:05:59.920 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:05:59.920 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:05:59.920 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:05:59.921 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 1, 2022-06-29T16:05:59.921 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 1, 2022-06-29T16:05:59.921 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 1, 2022-06-29T16:05:59.921 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 1, 2022-06-29T16:05:59.922 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 1, 2022-06-29T16:05:59.922 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:05:59.922 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:05:59.922 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:05:59.923 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:05:59.923 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:05:59.923 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:05:59.923 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:05:59.923 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:59.924 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "15", 2022-06-29T16:05:59.924 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:05:59.924 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:05:59.924 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:05:59.925 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:05:59.925 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:05:59.925 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:05:59.925 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:05:59.926 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:05:59.926 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:05:59.926 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:05:59.926 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:05:59.927 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:05:59.927 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:05:59.927 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:05:59.927 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:05:59.927 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:05:59.928 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:05:59.928 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:05:59.928 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:05:59.928 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:05:59.929 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:05:59.929 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:05:59.929 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:05:59.929 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:05:59.930 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:05:59.930 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:05:59.930 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:59.930 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:05:59.931 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:05:59.931 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:05:59.931 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:05:59.931 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:05:59.932 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:05:59.932 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:05:59.932 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:05:59.932 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:05:59.932 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:05:59.933 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:05:59.933 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:05:59.933 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 1 2022-06-29T16:05:59.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:59.934 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:05:59.934 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_devicehealth": {} 2022-06-29T16:05:59.934 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:59.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:59.935 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:59.935 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 2, 2022-06-29T16:05:59.935 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "rbd", 2022-06-29T16:05:59.936 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:19.184276+0000", 2022-06-29T16:05:59.936 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:05:59.936 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:05:59.936 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:05:59.936 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:05:59.937 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:05:59.937 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:05:59.937 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:05:59.937 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:05:59.938 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:05:59.938 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:05:59.938 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:05:59.938 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:05:59.939 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:05:59.939 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:05:59.939 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:05:59.939 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:05:59.939 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:05:59.940 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:05:59.940 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:05:59.940 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:05:59.940 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:59.941 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "18", 2022-06-29T16:05:59.941 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:05:59.941 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:05:59.941 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:05:59.942 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:05:59.942 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:05:59.942 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:05:59.942 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:05:59.943 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:05:59.943 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:05:59.943 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:05:59.943 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:05:59.944 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:05:59.944 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:05:59.944 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:05:59.944 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:05:59.945 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:05:59.945 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:05:59.945 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:05:59.945 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:05:59.945 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:05:59.946 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:05:59.946 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:05:59.946 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:05:59.946 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:05:59.947 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:05:59.947 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:05:59.947 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:59.947 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:05:59.948 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:05:59.948 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:05:59.948 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:05:59.948 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:05:59.949 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:05:59.949 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:05:59.949 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:05:59.949 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:05:59.949 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:05:59.950 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:05:59.950 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:05:59.950 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:05:59.950 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd": {} 2022-06-29T16:05:59.951 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:59.951 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:59.951 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:05:59.951 INFO:teuthology.orchestra.run.smithi060.stdout: "osds": [ 2022-06-29T16:05:59.952 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:59.952 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:05:59.952 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "11e7ab9e-fce2-470f-8195-ea4c3c52df22", 2022-06-29T16:05:59.952 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:59.953 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:59.953 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:59.953 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:59.953 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:59.953 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:59.954 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:59.954 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 89, 2022-06-29T16:05:59.954 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:59.954 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:59.955 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:59.955 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:59.955 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:59.955 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:59.956 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6816", 2022-06-29T16:05:59.956 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:59.956 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:59.956 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:59.956 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:59.957 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6825", 2022-06-29T16:05:59.957 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:59.957 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:59.957 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:59.958 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:59.958 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:59.958 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:59.958 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:59.959 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:59.959 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6826", 2022-06-29T16:05:59.959 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:59.959 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:59.959 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:59.960 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:59.960 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6827", 2022-06-29T16:05:59.960 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:59.960 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:59.961 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:59.961 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:59.961 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:59.961 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:59.962 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:59.962 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:59.962 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6830", 2022-06-29T16:05:59.962 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:59.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:59.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:59.963 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:59.963 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6831", 2022-06-29T16:05:59.963 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:59.964 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:59.964 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:59.964 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:59.964 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:59.965 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:59.965 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:59.965 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:59.965 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6828", 2022-06-29T16:05:59.966 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:59.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:59.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:59.966 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:59.967 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6829", 2022-06-29T16:05:59.967 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:05:59.967 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:59.967 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:59.967 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:59.968 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6825/12890", 2022-06-29T16:05:59.968 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6827/12890", 2022-06-29T16:05:59.968 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6831/12890", 2022-06-29T16:05:59.968 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6829/12890", 2022-06-29T16:05:59.969 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:59.969 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:59.969 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:59.969 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:59.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:59.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:59.970 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:05:59.970 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "ca43f136-6f69-4431-a157-fbe073fb585c", 2022-06-29T16:05:59.971 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:59.971 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:59.971 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:59.971 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:59.972 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:59.972 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:59.972 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:59.972 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 89, 2022-06-29T16:05:59.973 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:59.973 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:59.973 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:59.973 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:59.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:59.974 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:59.974 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6815", 2022-06-29T16:05:59.974 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:59.974 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:59.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:59.975 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:59.975 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6818", 2022-06-29T16:05:59.975 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:59.976 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:59.976 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:59.976 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:59.976 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:59.977 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:59.977 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:59.977 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:59.977 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6819", 2022-06-29T16:05:59.977 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:59.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:59.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:59.978 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:59.979 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6820", 2022-06-29T16:05:59.979 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:59.979 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:59.979 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:59.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:59.980 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:59.980 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:59.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:59.980 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:59.981 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6823", 2022-06-29T16:05:59.981 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:59.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:59.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:59.982 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:59.982 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6824", 2022-06-29T16:05:59.982 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:59.982 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:59.983 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:59.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:59.983 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:05:59.983 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:59.984 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:59.984 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:59.984 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6821", 2022-06-29T16:05:59.984 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:59.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:59.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:59.985 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:59.985 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6822", 2022-06-29T16:05:59.985 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:05:59.986 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:59.986 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:59.986 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:59.986 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6818/12892", 2022-06-29T16:05:59.987 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6820/12892", 2022-06-29T16:05:59.987 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6824/12892", 2022-06-29T16:05:59.987 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6822/12892", 2022-06-29T16:05:59.987 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:05:59.988 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:05:59.988 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:05:59.988 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:59.988 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:59.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:59.989 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:05:59.989 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "74fac45f-5f07-44ec-a9cb-2644c123fd4a", 2022-06-29T16:05:59.989 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:05:59.990 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:05:59.990 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:05:59.990 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:05:59.990 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:05:59.991 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:05:59.991 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:05:59.991 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 89, 2022-06-29T16:05:59.991 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:05:59.992 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:05:59.992 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:05:59.992 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:59.992 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:59.993 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:59.993 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6808", 2022-06-29T16:05:59.993 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:59.993 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:59.993 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:59.994 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:59.994 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6809", 2022-06-29T16:05:59.994 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:59.994 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:59.995 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:59.995 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:59.995 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:05:59.995 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:59.996 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:59.996 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:59.996 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6810", 2022-06-29T16:05:59.996 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:59.997 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:59.997 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:59.997 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:05:59.997 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6811", 2022-06-29T16:05:59.997 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:05:59.998 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:05:59.998 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:05:59.998 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:05:59.998 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:05:59.999 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:05:59.999 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:05:59.999 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:05:59.999 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6814", 2022-06-29T16:06:00.000 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:06:00.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.000 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:00.001 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6817", 2022-06-29T16:06:00.001 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:06:00.001 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:00.001 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:00.002 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.002 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:06:00.002 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:00.002 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.002 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:00.003 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6812", 2022-06-29T16:06:00.003 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:06:00.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.003 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.004 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:00.004 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6813", 2022-06-29T16:06:00.004 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:06:00.004 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:00.005 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:00.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.005 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6809/12891", 2022-06-29T16:06:00.005 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6811/12891", 2022-06-29T16:06:00.006 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6817/12891", 2022-06-29T16:06:00.006 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6813/12891", 2022-06-29T16:06:00.006 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:06:00.006 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:06:00.007 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:06:00.007 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:00.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.007 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.008 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:06:00.008 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "2581d351-973a-4b4e-85ca-3658f0068765", 2022-06-29T16:06:00.008 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:06:00.008 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:06:00.009 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:06:00.009 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:06:00.009 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:06:00.009 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:06:00.010 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:06:00.010 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 89, 2022-06-29T16:06:00.010 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:06:00.010 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:06:00.011 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:06:00.011 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:00.011 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.011 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:00.012 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6800", 2022-06-29T16:06:00.012 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:06:00.012 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.012 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.013 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:00.013 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6801", 2022-06-29T16:06:00.013 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:06:00.013 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:00.013 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:00.014 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.014 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:06:00.014 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:00.014 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.015 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:00.015 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6802", 2022-06-29T16:06:00.015 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:06:00.015 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.016 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.016 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:00.016 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6803", 2022-06-29T16:06:00.016 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:06:00.017 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:00.017 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:00.017 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.017 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:06:00.017 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:00.018 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.018 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:00.018 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6806", 2022-06-29T16:06:00.018 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:06:00.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.019 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.019 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:00.019 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6807", 2022-06-29T16:06:00.020 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:06:00.020 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:00.020 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:00.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.021 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:06:00.021 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:00.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.021 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:00.022 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6804", 2022-06-29T16:06:00.022 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:06:00.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.022 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.023 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:00.023 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6805", 2022-06-29T16:06:00.023 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:06:00.023 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:00.024 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:00.024 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.024 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6801/12889", 2022-06-29T16:06:00.024 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6803/12889", 2022-06-29T16:06:00.024 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6807/12889", 2022-06-29T16:06:00.025 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6805/12889", 2022-06-29T16:06:00.025 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:06:00.025 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:06:00.025 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:06:00.026 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:00.026 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.026 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.027 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:06:00.027 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "3d234db9-87cd-4fe9-814f-0240a287d8b4", 2022-06-29T16:06:00.027 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:06:00.027 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:06:00.027 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:06:00.028 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:06:00.028 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:06:00.028 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:06:00.028 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:06:00.029 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 88, 2022-06-29T16:06:00.029 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:06:00.029 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:06:00.029 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:06:00.030 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:00.030 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.030 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:00.030 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6808", 2022-06-29T16:06:00.031 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:06:00.031 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.031 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.031 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:00.031 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6810", 2022-06-29T16:06:00.032 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:06:00.032 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:00.032 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:00.033 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.033 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:06:00.033 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:00.033 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.034 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:00.034 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6812", 2022-06-29T16:06:00.034 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:06:00.034 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.035 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.035 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:00.035 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6814", 2022-06-29T16:06:00.035 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:06:00.035 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:00.036 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:00.036 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.036 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:06:00.036 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:00.037 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.037 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:00.037 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6823", 2022-06-29T16:06:00.037 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:06:00.038 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.038 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.038 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:00.038 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6825", 2022-06-29T16:06:00.039 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:06:00.039 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:00.039 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:00.040 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.040 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:06:00.040 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:00.040 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.041 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:00.041 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6817", 2022-06-29T16:06:00.041 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:06:00.041 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.042 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.042 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:00.042 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6820", 2022-06-29T16:06:00.042 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:06:00.043 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:00.043 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:00.043 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.043 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6810/13249", 2022-06-29T16:06:00.044 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6814/13249", 2022-06-29T16:06:00.044 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6825/13249", 2022-06-29T16:06:00.044 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6820/13249", 2022-06-29T16:06:00.045 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:06:00.045 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:06:00.045 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:06:00.045 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:00.046 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.046 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.046 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:06:00.046 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "e4bf1104-ff5c-438b-b608-8cee31c163f6", 2022-06-29T16:06:00.047 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:06:00.047 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:06:00.048 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:06:00.048 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:06:00.049 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:06:00.049 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:06:00.049 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:06:00.050 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 78, 2022-06-29T16:06:00.050 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:06:00.051 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:06:00.051 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:06:00.052 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:00.052 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.052 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:00.053 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6809", 2022-06-29T16:06:00.053 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:06:00.054 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.054 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.055 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:00.055 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6811", 2022-06-29T16:06:00.055 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:06:00.056 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:00.056 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:00.056 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.056 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:06:00.056 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:00.057 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.057 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:00.057 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6813", 2022-06-29T16:06:00.057 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:06:00.058 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.058 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.058 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:00.058 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6815", 2022-06-29T16:06:00.059 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:06:00.059 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:00.059 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:00.059 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.060 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:06:00.060 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:00.060 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.060 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:00.061 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6824", 2022-06-29T16:06:00.061 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:06:00.061 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.061 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.062 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:00.062 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6827", 2022-06-29T16:06:00.062 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:06:00.062 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:00.063 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:00.063 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.063 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:06:00.063 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:00.063 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.064 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:00.064 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6818", 2022-06-29T16:06:00.064 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:06:00.064 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.065 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.065 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:00.065 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6821", 2022-06-29T16:06:00.065 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:06:00.066 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:00.066 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:00.066 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.066 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6811/13248", 2022-06-29T16:06:00.067 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6815/13248", 2022-06-29T16:06:00.067 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6827/13248", 2022-06-29T16:06:00.067 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6821/13248", 2022-06-29T16:06:00.067 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:06:00.068 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:06:00.068 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:06:00.068 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:00.068 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.069 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.069 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:06:00.069 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "8564dd25-27ab-4203-b54e-af6d3aa05857", 2022-06-29T16:06:00.069 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:06:00.069 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:06:00.070 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:06:00.070 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:06:00.070 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:06:00.071 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:06:00.071 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:06:00.071 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 88, 2022-06-29T16:06:00.071 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:06:00.071 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:06:00.072 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:06:00.072 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:00.072 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.072 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:00.073 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6816", 2022-06-29T16:06:00.073 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:06:00.073 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.073 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.074 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:00.074 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6819", 2022-06-29T16:06:00.074 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:06:00.074 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:00.075 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:00.075 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.075 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:06:00.075 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:00.076 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.076 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:00.076 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6822", 2022-06-29T16:06:00.076 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:06:00.076 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.077 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.077 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:00.077 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6826", 2022-06-29T16:06:00.078 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:06:00.078 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:00.078 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:00.078 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.079 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:06:00.079 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:00.079 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.079 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:00.079 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6830", 2022-06-29T16:06:00.080 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:06:00.080 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.080 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.080 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:00.081 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6831", 2022-06-29T16:06:00.081 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:06:00.081 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:00.081 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:00.082 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.082 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:06:00.082 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:00.082 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.083 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:00.083 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6828", 2022-06-29T16:06:00.083 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:06:00.083 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.084 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.084 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:00.084 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6829", 2022-06-29T16:06:00.085 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:06:00.085 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:00.085 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:00.085 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.086 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6819/13250", 2022-06-29T16:06:00.086 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6826/13250", 2022-06-29T16:06:00.086 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6831/13250", 2022-06-29T16:06:00.086 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6829/13250", 2022-06-29T16:06:00.087 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:06:00.087 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:06:00.087 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:06:00.087 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:00.087 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.088 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.088 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:06:00.088 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "bd513a9c-3b19-4314-9897-d300b5c897bb", 2022-06-29T16:06:00.088 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:06:00.089 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:06:00.089 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:06:00.089 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:06:00.090 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:06:00.090 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:06:00.091 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:06:00.091 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 89, 2022-06-29T16:06:00.092 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:06:00.092 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:06:00.093 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:06:00.093 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:00.094 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.094 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:00.095 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6800", 2022-06-29T16:06:00.095 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:06:00.095 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.096 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.096 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:00.097 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6801", 2022-06-29T16:06:00.097 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:06:00.098 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:00.098 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:00.099 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.099 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:06:00.100 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:00.100 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.100 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:00.101 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6802", 2022-06-29T16:06:00.101 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:06:00.101 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.101 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.102 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:00.102 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6803", 2022-06-29T16:06:00.102 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:06:00.102 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:00.103 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:00.103 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.103 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:06:00.103 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:00.103 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.104 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:00.104 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6806", 2022-06-29T16:06:00.104 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:06:00.104 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.105 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.105 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:00.105 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6807", 2022-06-29T16:06:00.105 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:06:00.106 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:00.106 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:00.106 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.106 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:06:00.107 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:00.107 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.107 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:00.107 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6804", 2022-06-29T16:06:00.108 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:06:00.108 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.108 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.108 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:00.109 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6805", 2022-06-29T16:06:00.109 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:06:00.109 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:00.109 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:00.109 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.110 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6801/13247", 2022-06-29T16:06:00.110 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6803/13247", 2022-06-29T16:06:00.110 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6807/13247", 2022-06-29T16:06:00.110 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6805/13247", 2022-06-29T16:06:00.111 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:06:00.111 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:06:00.111 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:06:00.111 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:00.112 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:00.112 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:06:00.112 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_xinfo": [ 2022-06-29T16:06:00.112 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.113 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:06:00.113 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:06:00.113 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:06:00.113 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:06:00.113 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:06:00.114 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:06:00.114 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.891342+0000", 2022-06-29T16:06:00.114 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:06:00.114 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.115 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.115 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:06:00.115 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:06:00.115 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:06:00.116 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:06:00.116 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:06:00.116 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:06:00.116 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.900866+0000", 2022-06-29T16:06:00.117 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:06:00.117 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.117 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.117 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:06:00.118 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:06:00.118 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:06:00.118 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:06:00.118 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:06:00.119 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:06:00.119 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.912755+0000", 2022-06-29T16:06:00.119 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:06:00.119 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.120 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.120 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:06:00.121 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:06:00.121 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:06:00.121 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:06:00.122 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:06:00.122 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:06:00.123 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.864332+0000", 2022-06-29T16:06:00.123 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:06:00.123 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.124 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.124 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:06:00.125 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:06:00.125 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:06:00.125 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:06:00.126 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:06:00.126 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:06:00.127 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.863756+0000", 2022-06-29T16:06:00.127 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:06:00.127 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.128 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.128 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:06:00.128 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:06:00.129 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:06:00.129 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:06:00.129 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:06:00.129 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:06:00.130 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.869041+0000", 2022-06-29T16:06:00.130 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:06:00.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.130 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.131 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:06:00.131 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:06:00.131 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:06:00.131 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:06:00.131 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:06:00.132 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:06:00.132 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.895043+0000", 2022-06-29T16:06:00.132 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:06:00.132 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.133 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:00.133 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:06:00.133 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:06:00.133 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:06:00.134 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:06:00.134 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:06:00.134 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:06:00.134 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.889491+0000", 2022-06-29T16:06:00.135 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:06:00.135 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:00.136 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:06:00.136 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap": [], 2022-06-29T16:06:00.137 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap_items": [], 2022-06-29T16:06:00.137 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_temp": [], 2022-06-29T16:06:00.138 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_temp": [], 2022-06-29T16:06:00.138 INFO:teuthology.orchestra.run.smithi060.stdout: "blacklist": { 2022-06-29T16:06:00.139 INFO:teuthology.orchestra.run.smithi060.stdout: "172.21.15.92:6835/2057420852": "2022-06-30T16:05:52.920809+0000", 2022-06-29T16:06:00.139 INFO:teuthology.orchestra.run.smithi060.stdout: "172.21.15.92:6834/2057420852": "2022-06-30T16:05:52.920809+0000" 2022-06-29T16:06:00.139 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.140 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profiles": { 2022-06-29T16:06:00.140 INFO:teuthology.orchestra.run.smithi060.stdout: "default": { 2022-06-29T16:06:00.141 INFO:teuthology.orchestra.run.smithi060.stdout: "crush-failure-domain": "osd", 2022-06-29T16:06:00.141 INFO:teuthology.orchestra.run.smithi060.stdout: "k": "2", 2022-06-29T16:06:00.142 INFO:teuthology.orchestra.run.smithi060.stdout: "m": "1", 2022-06-29T16:06:00.142 INFO:teuthology.orchestra.run.smithi060.stdout: "plugin": "jerasure", 2022-06-29T16:06:00.143 INFO:teuthology.orchestra.run.smithi060.stdout: "ruleset-failure-domain": "osd", 2022-06-29T16:06:00.143 INFO:teuthology.orchestra.run.smithi060.stdout: "technique": "reed_sol_van" 2022-06-29T16:06:00.144 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:00.144 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:00.144 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps_queue": [], 2022-06-29T16:06:00.145 INFO:teuthology.orchestra.run.smithi060.stdout: "new_removed_snaps": [], 2022-06-29T16:06:00.145 INFO:teuthology.orchestra.run.smithi060.stdout: "new_purged_snaps": [], 2022-06-29T16:06:00.146 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_node_flags": {}, 2022-06-29T16:06:00.146 INFO:teuthology.orchestra.run.smithi060.stdout: "device_class_flags": {} 2022-06-29T16:06:00.146 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:06:00.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.909+0000 7ff7cdaaf700 1 -- 172.21.15.60:0/3191038005 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff7ac053b30 msgr2=0x7ff7ac055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:00.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.909+0000 7ff7cdaaf700 1 --2- 172.21.15.60:0/3191038005 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff7ac053b30 0x7ff7ac055ff0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7ff7b800b000 tx=0x7ff7b8006df0).stop 2022-06-29T16:06:00.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.909+0000 7ff7cdaaf700 1 -- 172.21.15.60:0/3191038005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7c80ae270 msgr2=0x7ff7c80b06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:00.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.909+0000 7ff7cdaaf700 1 --2- 172.21.15.60:0/3191038005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7c80ae270 0x7ff7c80b06f0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 rx=0x7ff7b0002f70 tx=0x7ff7b0014000).stop 2022-06-29T16:06:00.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.909+0000 7ff7cdaaf700 1 -- 172.21.15.60:0/3191038005 shutdown_connections 2022-06-29T16:06:00.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.909+0000 7ff7cdaaf700 1 --2- 172.21.15.60:0/3191038005 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff7ac053b30 0x7ff7ac055ff0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:00.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.909+0000 7ff7cdaaf700 1 --2- 172.21.15.60:0/3191038005 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff7c813a920 0x7ff7c8152820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:00.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.909+0000 7ff7cdaaf700 1 --2- 172.21.15.60:0/3191038005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7c80b5b00 0x7ff7c8152330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:00.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.909+0000 7ff7cdaaf700 1 --2- 172.21.15.60:0/3191038005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff7c80ae270 0x7ff7c80b06f0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:00.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.909+0000 7ff7cdaaf700 1 -- 172.21.15.60:0/3191038005 >> 172.21.15.60:0/3191038005 conn(0x7ff7c80632c0 msgr2=0x7ff7c8139150 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:00.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.909+0000 7ff7cdaaf700 1 -- 172.21.15.60:0/3191038005 shutdown_connections 2022-06-29T16:06:00.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:05:59.909+0000 7ff7cdaaf700 1 -- 172.21.15.60:0/3191038005 wait complete. 2022-06-29T16:06:00.153 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:06:00.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.257+0000 7f5f67210700 1 Processor -- start 2022-06-29T16:06:00.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.257+0000 7f5f67210700 1 -- start start 2022-06-29T16:06:00.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.257+0000 7f5f67210700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f600b5950 0x7f5f60067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:00.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.257+0000 7f5f67210700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5f60063ad0 0x7f5f6014af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:00.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.257+0000 7f5f67210700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f600acba0 con 0x7f5f6005bd20 2022-06-29T16:06:00.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.257+0000 7f5f67210700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f60067220 con 0x7f5f60063ad0 2022-06-29T16:06:00.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.257+0000 7f5f67210700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f600b5d30 con 0x7f5f600b5950 2022-06-29T16:06:00.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.257+0000 7f5f657ad700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5f6005bd20 0x7f5f6005c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59276/0 (socket says 172.21.15.60:59276) 2022-06-29T16:06:00.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.257+0000 7f5f657ad700 1 -- 172.21.15.60:0/3195801551 learned_addr learned my addr 172.21.15.60:0/3195801551 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:00.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.257+0000 7f5f5ffff700 1 --2- 172.21.15.60:0/3195801551 >> v2:172.21.15.92:3300/0 conn(0x7f5f60063ad0 0x7f5f6014af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:00.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.257+0000 7f5f64fac700 1 --2- 172.21.15.60:0/3195801551 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f600b5950 0x7f5f60067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:00.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.257+0000 7f5f5f7fe700 1 -- 172.21.15.60:0/3195801551 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1751401190 0 0) 0x7f5f600acba0 con 0x7f5f6005bd20 2022-06-29T16:06:00.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.257+0000 7f5f5f7fe700 1 -- 172.21.15.60:0/3195801551 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f44003260 con 0x7f5f6005bd20 2022-06-29T16:06:00.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.257+0000 7f5f5ffff700 1 -- 172.21.15.60:0/3195801551 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f600b5950 msgr2=0x7f5f60067680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:00.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.257+0000 7f5f5ffff700 1 --2- 172.21.15.60:0/3195801551 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f600b5950 0x7f5f60067680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:00.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.257+0000 7f5f5f7fe700 1 -- 172.21.15.60:0/3195801551 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 909064764 0 0) 0x7f5f44003260 con 0x7f5f6005bd20 2022-06-29T16:06:00.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.257+0000 7f5f5ffff700 1 -- 172.21.15.60:0/3195801551 >> v1:172.21.15.60:6789/0 conn(0x7f5f6005bd20 legacy=0x7f5f6005c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:00.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.257+0000 7f5f5ffff700 1 -- 172.21.15.60:0/3195801551 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f600ace00 con 0x7f5f60063ad0 2022-06-29T16:06:00.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.257+0000 7f5f5ffff700 1 --2- 172.21.15.60:0/3195801551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f60063ad0 0x7f5f6014af10 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f5f50001800 tx=0x7f5f48006df0).ready entity=mon.1 client_cookie=ce5bea2daf6a5dd0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:00.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.257+0000 7f5f5f7fe700 1 -- 172.21.15.60:0/3195801551 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f50001660 con 0x7f5f60063ad0 2022-06-29T16:06:00.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.257+0000 7f5f5f7fe700 1 -- 172.21.15.60:0/3195801551 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f48000e90 con 0x7f5f60063ad0 2022-06-29T16:06:00.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.257+0000 7f5f5f7fe700 1 -- 172.21.15.60:0/3195801551 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f48012460 con 0x7f5f60063ad0 2022-06-29T16:06:00.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.261+0000 7f5f67210700 1 -- 172.21.15.60:0/3195801551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f60063ad0 msgr2=0x7f5f6014af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:00.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.261+0000 7f5f67210700 1 --2- 172.21.15.60:0/3195801551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f60063ad0 0x7f5f6014af10 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f5f50001800 tx=0x7f5f48006df0).stop 2022-06-29T16:06:00.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.261+0000 7f5f67210700 1 -- 172.21.15.60:0/3195801551 shutdown_connections 2022-06-29T16:06:00.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.261+0000 7f5f67210700 1 --2- 172.21.15.60:0/3195801551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f60063ad0 0x7f5f6014af10 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:00.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.261+0000 7f5f67210700 1 --2- 172.21.15.60:0/3195801551 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f600b5950 0x7f5f60067680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:00.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.261+0000 7f5f67210700 1 -- 172.21.15.60:0/3195801551 >> 172.21.15.60:0/3195801551 conn(0x7f5f600632c0 msgr2=0x7f5f600b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:00.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.261+0000 7f5f67210700 1 -- 172.21.15.60:0/3195801551 shutdown_connections 2022-06-29T16:06:00.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.261+0000 7f5f67210700 1 -- 172.21.15.60:0/3195801551 wait complete. 2022-06-29T16:06:00.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.261+0000 7f5f67210700 1 Processor -- start 2022-06-29T16:06:00.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.261+0000 7f5f67210700 1 -- start start 2022-06-29T16:06:00.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.261+0000 7f5f67210700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f6005bd20 0x7f5f60148320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:00.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.261+0000 7f5f67210700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f60063ad0 0x7f5f600b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:00.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.261+0000 7f5f67210700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f600b5950 0x7f5f600b05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:00.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.261+0000 7f5f67210700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f600ad4f0 con 0x7f5f60063ad0 2022-06-29T16:06:00.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.261+0000 7f5f67210700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f60148810 con 0x7f5f6005bd20 2022-06-29T16:06:00.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.261+0000 7f5f67210700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f600ad720 con 0x7f5f600b5950 2022-06-29T16:06:00.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.261+0000 7f5f5ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f60063ad0 0x7f5f600b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:00.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.261+0000 7f5f5ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f60063ad0 0x7f5f600b44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51192/0 (socket says 172.21.15.60:51192) 2022-06-29T16:06:00.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.261+0000 7f5f5ffff700 1 -- 172.21.15.60:0/2846499585 learned_addr learned my addr 172.21.15.60:0/2846499585 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:00.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.261+0000 7f5f64fac700 1 --2- 172.21.15.60:0/2846499585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f6005bd20 0x7f5f60148320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:00.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.261+0000 7f5f657ad700 1 --2- 172.21.15.60:0/2846499585 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f600b5950 0x7f5f600b05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:00.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.265+0000 7f5f5ffff700 1 -- 172.21.15.60:0/2846499585 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f600b5950 msgr2=0x7f5f600b05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:00.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.265+0000 7f5f5ffff700 1 --2- 172.21.15.60:0/2846499585 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f600b5950 0x7f5f600b05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:00.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.265+0000 7f5f5ffff700 1 -- 172.21.15.60:0/2846499585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f6005bd20 msgr2=0x7f5f60148320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:00.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.265+0000 7f5f5ffff700 1 --2- 172.21.15.60:0/2846499585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f6005bd20 0x7f5f60148320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:00.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.265+0000 7f5f5ffff700 1 -- 172.21.15.60:0/2846499585 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4800b180 con 0x7f5f60063ad0 2022-06-29T16:06:00.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.265+0000 7f5f64fac700 1 --2- 172.21.15.60:0/2846499585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f6005bd20 0x7f5f60148320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:06:00.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.265+0000 7f5f5ffff700 1 --2- 172.21.15.60:0/2846499585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f60063ad0 0x7f5f600b44c0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 rx=0x7f5f50001800 tx=0x7f5f48014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:00.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.265+0000 7f5f5dffb700 1 -- 172.21.15.60:0/2846499585 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f48016030 con 0x7f5f60063ad0 2022-06-29T16:06:00.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.265+0000 7f5f5dffb700 1 -- 172.21.15.60:0/2846499585 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f48013e40 con 0x7f5f60063ad0 2022-06-29T16:06:00.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.265+0000 7f5f67210700 1 -- 172.21.15.60:0/2846499585 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f60156bd0 con 0x7f5f60063ad0 2022-06-29T16:06:00.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.265+0000 7f5f5dffb700 1 -- 172.21.15.60:0/2846499585 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f4800de60 con 0x7f5f60063ad0 2022-06-29T16:06:00.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.265+0000 7f5f67210700 1 -- 172.21.15.60:0/2846499585 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f60116900 con 0x7f5f60063ad0 2022-06-29T16:06:00.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.265+0000 7f5f67210700 1 -- 172.21.15.60:0/2846499585 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f600573d0 con 0x7f5f60063ad0 2022-06-29T16:06:00.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.265+0000 7f5f5dffb700 1 -- 172.21.15.60:0/2846499585 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5f48024020 con 0x7f5f60063ad0 2022-06-29T16:06:00.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.265+0000 7f5f5dffb700 1 --2- 172.21.15.60:0/2846499585 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5f38053aa0 0x7f5f38055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:00.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.269+0000 7f5f5dffb700 1 -- 172.21.15.60:0/2846499585 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f5f48017070 con 0x7f5f60063ad0 2022-06-29T16:06:00.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.269+0000 7f5f64fac700 1 --2- 172.21.15.60:0/2846499585 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5f38053aa0 0x7f5f38055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:00.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.269+0000 7f5f64fac700 1 --2- 172.21.15.60:0/2846499585 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5f38053aa0 0x7f5f38055f60 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7f5f5800b000 tx=0x7f5f58008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:00.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.273+0000 7f5f5dffb700 1 -- 172.21.15.60:0/2846499585 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5f48043e90 con 0x7f5f60063ad0 2022-06-29T16:06:00.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.433+0000 7f5f67210700 1 -- 172.21.15.60:0/2846499585 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f600b1540 con 0x7f5f60063ad0 2022-06-29T16:06:00.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.433+0000 7f5f5dffb700 1 -- 172.21.15.60:0/2846499585 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+1209 (secure 0 0 0) 0x7f5f480474b0 con 0x7f5f60063ad0 2022-06-29T16:06:00.440 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:00.440 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":43,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6182,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1306898602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1306898602},{"type":"v1","addr":"172.21.15.60:6833","nonce":1306898602}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":39},{"gid":7240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/1867407213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1867407213},{"type":"v1","addr":"172.21.15.92:6835","nonce":1867407213}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":43}],"filesystems":[]} 2022-06-29T16:06:00.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.437+0000 7f5f67210700 1 -- 172.21.15.60:0/2846499585 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5f38053aa0 msgr2=0x7f5f38055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:00.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.437+0000 7f5f67210700 1 --2- 172.21.15.60:0/2846499585 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5f38053aa0 0x7f5f38055f60 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7f5f5800b000 tx=0x7f5f58008040).stop 2022-06-29T16:06:00.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.437+0000 7f5f67210700 1 -- 172.21.15.60:0/2846499585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f60063ad0 msgr2=0x7f5f600b44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:00.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.437+0000 7f5f67210700 1 --2- 172.21.15.60:0/2846499585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f60063ad0 0x7f5f600b44c0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 rx=0x7f5f50001800 tx=0x7f5f48014000).stop 2022-06-29T16:06:00.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.437+0000 7f5f67210700 1 -- 172.21.15.60:0/2846499585 shutdown_connections 2022-06-29T16:06:00.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.437+0000 7f5f67210700 1 --2- 172.21.15.60:0/2846499585 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5f38053aa0 0x7f5f38055f60 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:00.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.437+0000 7f5f67210700 1 --2- 172.21.15.60:0/2846499585 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f600b5950 0x7f5f600b05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:00.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.437+0000 7f5f67210700 1 --2- 172.21.15.60:0/2846499585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f60063ad0 0x7f5f600b44c0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:00.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.437+0000 7f5f67210700 1 --2- 172.21.15.60:0/2846499585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f6005bd20 0x7f5f60148320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:00.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.437+0000 7f5f67210700 1 -- 172.21.15.60:0/2846499585 >> 172.21.15.60:0/2846499585 conn(0x7f5f600632c0 msgr2=0x7f5f60142dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:00.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.437+0000 7f5f67210700 1 -- 172.21.15.60:0/2846499585 shutdown_connections 2022-06-29T16:06:00.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.437+0000 7f5f67210700 1 -- 172.21.15.60:0/2846499585 wait complete. 2022-06-29T16:06:00.447 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 43 2022-06-29T16:06:00.453 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:06:00.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.553+0000 7f5f1359e700 1 Processor -- start 2022-06-29T16:06:00.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.553+0000 7f5f1359e700 1 -- start start 2022-06-29T16:06:00.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.553+0000 7f5f1359e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5f140b2130 0x7f5f14146100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:00.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.553+0000 7f5f1359e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f140aed00 0x7f5f141465f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:00.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.553+0000 7f5f1359e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f1405d450 con 0x7f5f1405d070 2022-06-29T16:06:00.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.553+0000 7f5f1359e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f140adf90 con 0x7f5f140b2130 2022-06-29T16:06:00.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.553+0000 7f5f1359e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f140b25d0 con 0x7f5f140aed00 2022-06-29T16:06:00.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.553+0000 7f5f11d9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5f140b2130 0x7f5f14146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:00.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.553+0000 7f5f11d9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5f140b2130 0x7f5f14146100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51202/0 (socket says 172.21.15.60:51202) 2022-06-29T16:06:00.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.553+0000 7f5f11d9b700 1 -- 172.21.15.60:0/3281837337 learned_addr learned my addr 172.21.15.60:0/3281837337 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:00.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.553+0000 7f5f1259c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5f1405d070 0x7f5f1414aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59286/0 (socket says 172.21.15.60:59286) 2022-06-29T16:06:00.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.553+0000 7f5f12d9d700 1 --2- 172.21.15.60:0/3281837337 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f140aed00 0x7f5f141465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:00.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.553+0000 7f5f11d9b700 1 -- 172.21.15.60:0/3281837337 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f140aed00 msgr2=0x7f5f141465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:00.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.553+0000 7f5f11d9b700 1 --2- 172.21.15.60:0/3281837337 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f140aed00 0x7f5f141465f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:00.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.553+0000 7f5f11d9b700 1 -- 172.21.15.60:0/3281837337 >> v1:172.21.15.60:6789/0 conn(0x7f5f1405d070 legacy=0x7f5f1414aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:00.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.553+0000 7f5f11d9b700 1 -- 172.21.15.60:0/3281837337 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f1414a350 con 0x7f5f140b2130 2022-06-29T16:06:00.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.553+0000 7f5f11d9b700 1 --2- 172.21.15.60:0/3281837337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f140b2130 0x7f5f14146100 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 rx=0x7f5f040032f0 tx=0x7f5f04009290).ready entity=mon.0 client_cookie=42c88175e835d790 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:00.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.553+0000 7f5f1159a700 1 -- 172.21.15.60:0/3281837337 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f0400b8a0 con 0x7f5f140b2130 2022-06-29T16:06:00.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.553+0000 7f5f1159a700 1 -- 172.21.15.60:0/3281837337 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f0400ba00 con 0x7f5f140b2130 2022-06-29T16:06:00.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.553+0000 7f5f1159a700 1 -- 172.21.15.60:0/3281837337 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f0400bb70 con 0x7f5f140b2130 2022-06-29T16:06:00.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.553+0000 7f5f1359e700 1 -- 172.21.15.60:0/3281837337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f140b2130 msgr2=0x7f5f14146100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:00.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.553+0000 7f5f1359e700 1 --2- 172.21.15.60:0/3281837337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f140b2130 0x7f5f14146100 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 rx=0x7f5f040032f0 tx=0x7f5f04009290).stop 2022-06-29T16:06:00.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.553+0000 7f5f1359e700 1 -- 172.21.15.60:0/3281837337 shutdown_connections 2022-06-29T16:06:00.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.553+0000 7f5f1359e700 1 --2- 172.21.15.60:0/3281837337 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f140aed00 0x7f5f141465f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:00.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.553+0000 7f5f1359e700 1 --2- 172.21.15.60:0/3281837337 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f140b2130 0x7f5f14146100 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:00.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.553+0000 7f5f1359e700 1 -- 172.21.15.60:0/3281837337 >> 172.21.15.60:0/3281837337 conn(0x7f5f140b5890 msgr2=0x7f5f140b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:00.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.553+0000 7f5f1359e700 1 -- 172.21.15.60:0/3281837337 shutdown_connections 2022-06-29T16:06:00.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.553+0000 7f5f1359e700 1 -- 172.21.15.60:0/3281837337 wait complete. 2022-06-29T16:06:00.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.557+0000 7f5f1359e700 1 Processor -- start 2022-06-29T16:06:00.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.557+0000 7f5f1359e700 1 -- start start 2022-06-29T16:06:00.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.557+0000 7f5f1359e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f1405d070 0x7f5f14146d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:00.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.557+0000 7f5f1359e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f140aed00 0x7f5f14132de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:00.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.557+0000 7f5f1359e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f140b2130 0x7f5f141332d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:00.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.557+0000 7f5f1359e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1414b090 con 0x7f5f1405d070 2022-06-29T16:06:00.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.557+0000 7f5f1359e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f140641d0 con 0x7f5f140b2130 2022-06-29T16:06:00.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.557+0000 7f5f1359e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1414a9b0 con 0x7f5f140aed00 2022-06-29T16:06:00.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.557+0000 7f5f1259c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f1405d070 0x7f5f14146d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:00.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.557+0000 7f5f1259c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f1405d070 0x7f5f14146d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51208/0 (socket says 172.21.15.60:51208) 2022-06-29T16:06:00.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.557+0000 7f5f1259c700 1 -- 172.21.15.60:0/3945142929 learned_addr learned my addr 172.21.15.60:0/3945142929 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:00.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.557+0000 7f5f12d9d700 1 --2- 172.21.15.60:0/3945142929 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f140b2130 0x7f5f141332d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:00.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.557+0000 7f5f11d9b700 1 --2- 172.21.15.60:0/3945142929 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f140aed00 0x7f5f14132de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:00.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.557+0000 7f5f1259c700 1 -- 172.21.15.60:0/3945142929 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f140aed00 msgr2=0x7f5f14132de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:00.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.557+0000 7f5f1259c700 1 --2- 172.21.15.60:0/3945142929 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f140aed00 0x7f5f14132de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:00.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.557+0000 7f5f1259c700 1 -- 172.21.15.60:0/3945142929 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f140b2130 msgr2=0x7f5f141332d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:00.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.557+0000 7f5f1259c700 1 --2- 172.21.15.60:0/3945142929 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f140b2130 0x7f5f141332d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:00.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.557+0000 7f5f1259c700 1 -- 172.21.15.60:0/3945142929 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f04009530 con 0x7f5f1405d070 2022-06-29T16:06:00.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.557+0000 7f5f1259c700 1 --2- 172.21.15.60:0/3945142929 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f1405d070 0x7f5f14146d00 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 rx=0x7f5f14061990 tx=0x7f5efc007bd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:00.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.557+0000 7f5f0b7fe700 1 -- 172.21.15.60:0/3945142929 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5efc007170 con 0x7f5f1405d070 2022-06-29T16:06:00.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.557+0000 7f5f0b7fe700 1 -- 172.21.15.60:0/3945142929 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5efc0072d0 con 0x7f5f1405d070 2022-06-29T16:06:00.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.557+0000 7f5f1359e700 1 -- 172.21.15.60:0/3945142929 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f14064400 con 0x7f5f1405d070 2022-06-29T16:06:00.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.557+0000 7f5f0b7fe700 1 -- 172.21.15.60:0/3945142929 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5efc004710 con 0x7f5f1405d070 2022-06-29T16:06:00.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.557+0000 7f5f1359e700 1 -- 172.21.15.60:0/3945142929 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f141337c0 con 0x7f5f1405d070 2022-06-29T16:06:00.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.557+0000 7f5f1359e700 1 -- 172.21.15.60:0/3945142929 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f1414a5f0 con 0x7f5f1405d070 2022-06-29T16:06:00.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.561+0000 7f5f0b7fe700 1 -- 172.21.15.60:0/3945142929 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5efc004870 con 0x7f5f1405d070 2022-06-29T16:06:00.587 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.561+0000 7f5f0b7fe700 1 --2- 172.21.15.60:0/3945142929 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5ef8053a70 0x7f5ef8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:00.587 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.561+0000 7f5f11d9b700 1 --2- 172.21.15.60:0/3945142929 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5ef8053a70 0x7f5ef8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:00.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.561+0000 7f5f0b7fe700 1 -- 172.21.15.60:0/3945142929 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f5efc06b4e0 con 0x7f5f1405d070 2022-06-29T16:06:00.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.561+0000 7f5f11d9b700 1 --2- 172.21.15.60:0/3945142929 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5ef8053a70 0x7f5ef8055f30 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7f5f0401b000 tx=0x7f5f04016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:00.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.569+0000 7f5f0b7fe700 1 -- 172.21.15.60:0/3945142929 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5efc0425d0 con 0x7f5f1405d070 2022-06-29T16:06:00.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.737+0000 7f5f1359e700 1 -- 172.21.15.60:0/3945142929 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f140573d0 con 0x7f5f1405d070 2022-06-29T16:06:00.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.737+0000 7f5f0b7fe700 1 -- 172.21.15.60:0/3945142929 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+1209 (secure 0 0 0) 0x7f5efc045bf0 con 0x7f5f1405d070 2022-06-29T16:06:00.742 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:00.743 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":43,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6182,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/1306898602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1306898602},{"type":"v1","addr":"172.21.15.60:6833","nonce":1306898602}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":39},{"gid":7240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/1867407213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1867407213},{"type":"v1","addr":"172.21.15.92:6835","nonce":1867407213}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":43}],"filesystems":[]} 2022-06-29T16:06:00.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.737+0000 7f5f1359e700 1 -- 172.21.15.60:0/3945142929 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5ef8053a70 msgr2=0x7f5ef8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:00.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.737+0000 7f5f1359e700 1 --2- 172.21.15.60:0/3945142929 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5ef8053a70 0x7f5ef8055f30 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7f5f0401b000 tx=0x7f5f04016000).stop 2022-06-29T16:06:00.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.737+0000 7f5f1359e700 1 -- 172.21.15.60:0/3945142929 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f1405d070 msgr2=0x7f5f14146d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:00.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.737+0000 7f5f1359e700 1 --2- 172.21.15.60:0/3945142929 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f1405d070 0x7f5f14146d00 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 rx=0x7f5f14061990 tx=0x7f5efc007bd0).stop 2022-06-29T16:06:00.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.741+0000 7f5f1359e700 1 -- 172.21.15.60:0/3945142929 shutdown_connections 2022-06-29T16:06:00.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.741+0000 7f5f1359e700 1 --2- 172.21.15.60:0/3945142929 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5ef8053a70 0x7f5ef8055f30 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:00.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.741+0000 7f5f1359e700 1 --2- 172.21.15.60:0/3945142929 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5f140b2130 0x7f5f141332d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:00.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.741+0000 7f5f1359e700 1 --2- 172.21.15.60:0/3945142929 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5f140aed00 0x7f5f14132de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:00.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.741+0000 7f5f1359e700 1 --2- 172.21.15.60:0/3945142929 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5f1405d070 0x7f5f14146d00 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:00.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.741+0000 7f5f1359e700 1 -- 172.21.15.60:0/3945142929 >> 172.21.15.60:0/3945142929 conn(0x7f5f140b5890 msgr2=0x7f5f140b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:00.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.741+0000 7f5f1359e700 1 -- 172.21.15.60:0/3945142929 shutdown_connections 2022-06-29T16:06:00.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.741+0000 7f5f1359e700 1 -- 172.21.15.60:0/3945142929 wait complete. 2022-06-29T16:06:00.749 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 43 2022-06-29T16:06:00.756 INFO:tasks.ceph.mds.a:Restarting daemon 2022-06-29T16:06:00.757 INFO:tasks.ceph.mds.a:Stopping old one... 2022-06-29T16:06:00.757 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-06-29T16:06:00.757 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T16:06:00.758 INFO:tasks.ceph.mds.b:Restarting daemon 2022-06-29T16:06:00.758 INFO:tasks.ceph.mds.b:Stopping old one... 2022-06-29T16:06:00.758 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-06-29T16:06:00.759 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T16:06:00.771 INFO:tasks.ceph.mds.a:Stopped 2022-06-29T16:06:00.772 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-06-29T16:06:00.774 INFO:tasks.ceph.mds.b:Stopped 2022-06-29T16:06:00.775 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-06-29T16:06:00.776 INFO:tasks.ceph.mds.a:Started 2022-06-29T16:06:00.776 DEBUG:teuthology.parallel:result is None 2022-06-29T16:06:00.778 INFO:tasks.ceph.mds.b:Started 2022-06-29T16:06:00.778 DEBUG:teuthology.parallel:result is None 2022-06-29T16:06:00.779 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-06-29T16:06:00.841 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2022-06-29T16:06:00.854 INFO:tasks.ceph.mds.b.smithi092.stdout:starting mds.b at 2022-06-29T16:06:00.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.873+0000 7f17d5732700 1 Processor -- start 2022-06-29T16:06:00.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.873+0000 7f17d5732700 1 -- start start 2022-06-29T16:06:00.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.873+0000 7f17d5732700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f17d013f3d0 0x7f17d0060260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:00.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.873+0000 7f17d5732700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f17d0143820 0x7f17d00b6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:00.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.873+0000 7f17d5732700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17d00b6c00 0x7f17d013f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:00.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.873+0000 7f17d5732700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17d0142ef0 con 0x7f17d013f3d0 2022-06-29T16:06:00.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.873+0000 7f17d5732700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17d00acb90 con 0x7f17d0143820 2022-06-29T16:06:00.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.873+0000 7f17d5732700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d01430e0 con 0x7f17d00b6c00 2022-06-29T16:06:00.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.873+0000 7f17ceffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f17d013f3d0 0x7f17d0060260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:00.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.873+0000 7f17ceffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f17d013f3d0 0x7f17d0060260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51226/0 (socket says 172.21.15.60:51226) 2022-06-29T16:06:00.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.873+0000 7f17ceffd700 1 -- 172.21.15.60:0/2208524693 learned_addr learned my addr 172.21.15.60:0/2208524693 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:00.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.873+0000 7f17ce7fc700 1 --2- 172.21.15.60:0/2208524693 >> v2:172.21.15.92:3300/0 conn(0x7f17d0143820 0x7f17d00b6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:00.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.873+0000 7f17cf7fe700 1 --2- 172.21.15.60:0/2208524693 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17d00b6c00 0x7f17d013f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:00.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.873+0000 7f17ceffd700 1 -- 172.21.15.60:0/2208524693 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17d00b6c00 msgr2=0x7f17d013f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:00.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.873+0000 7f17ceffd700 1 --2- 172.21.15.60:0/2208524693 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17d00b6c00 0x7f17d013f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:00.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.873+0000 7f17ceffd700 1 -- 172.21.15.60:0/2208524693 >> v2:172.21.15.92:3300/0 conn(0x7f17d0143820 msgr2=0x7f17d00b6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:00.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.873+0000 7f17ceffd700 1 --2- 172.21.15.60:0/2208524693 >> v2:172.21.15.92:3300/0 conn(0x7f17d0143820 0x7f17d00b6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:00.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.873+0000 7f17ceffd700 1 -- 172.21.15.60:0/2208524693 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d0143380 con 0x7f17d013f3d0 2022-06-29T16:06:00.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.877+0000 7f17ceffd700 1 --2- 172.21.15.60:0/2208524693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17d013f3d0 0x7f17d0060260 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 rx=0x7f17c0002f70 tx=0x7f17c0006eb0).ready entity=mon.0 client_cookie=9130724ac27fcb27 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:00.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.877+0000 7f17cdffb700 1 -- 172.21.15.60:0/2208524693 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f17c000f970 con 0x7f17d013f3d0 2022-06-29T16:06:00.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.877+0000 7f17cdffb700 1 -- 172.21.15.60:0/2208524693 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c0000e90 con 0x7f17d013f3d0 2022-06-29T16:06:00.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.877+0000 7f17cdffb700 1 -- 172.21.15.60:0/2208524693 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f17c000fd00 con 0x7f17d013f3d0 2022-06-29T16:06:00.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.877+0000 7f17d5732700 1 -- 172.21.15.60:0/2208524693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17d013f3d0 msgr2=0x7f17d0060260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:00.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.877+0000 7f17d5732700 1 --2- 172.21.15.60:0/2208524693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17d013f3d0 0x7f17d0060260 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 rx=0x7f17c0002f70 tx=0x7f17c0006eb0).stop 2022-06-29T16:06:00.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.877+0000 7f17d5732700 1 -- 172.21.15.60:0/2208524693 shutdown_connections 2022-06-29T16:06:00.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.877+0000 7f17d5732700 1 --2- 172.21.15.60:0/2208524693 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17d00b6c00 0x7f17d013f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:00.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.877+0000 7f17d5732700 1 --2- 172.21.15.60:0/2208524693 >> v2:172.21.15.92:3300/0 conn(0x7f17d0143820 0x7f17d00b6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:00.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.877+0000 7f17d5732700 1 --2- 172.21.15.60:0/2208524693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17d013f3d0 0x7f17d0060260 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:00.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.877+0000 7f17d5732700 1 -- 172.21.15.60:0/2208524693 >> 172.21.15.60:0/2208524693 conn(0x7f17d00643e0 msgr2=0x7f17d00b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:00.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.877+0000 7f17d5732700 1 -- 172.21.15.60:0/2208524693 shutdown_connections 2022-06-29T16:06:00.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.877+0000 7f17d5732700 1 -- 172.21.15.60:0/2208524693 wait complete. 2022-06-29T16:06:00.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.877+0000 7f17d5732700 1 Processor -- start 2022-06-29T16:06:00.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.877+0000 7f17d5732700 1 -- start start 2022-06-29T16:06:00.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.877+0000 7f17d5732700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17d00b6c00 0x7f17d00b17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:00.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.877+0000 7f17d5732700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17d013f3d0 0x7f17d0064920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:00.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.877+0000 7f17d5732700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17d0143820 0x7f17d00afb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:00.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.877+0000 7f17d5732700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d00af6b0 con 0x7f17d0143820 2022-06-29T16:06:00.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.877+0000 7f17d5732700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d00adc30 con 0x7f17d00b6c00 2022-06-29T16:06:00.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.877+0000 7f17d5732700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d00aded0 con 0x7f17d013f3d0 2022-06-29T16:06:00.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.877+0000 7f17cf7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17d0143820 0x7f17d00afb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:00.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.877+0000 7f17ceffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17d00b6c00 0x7f17d00b17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:00.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.877+0000 7f17cf7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17d0143820 0x7f17d00afb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51236/0 (socket says 172.21.15.60:51236) 2022-06-29T16:06:00.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.877+0000 7f17cf7fe700 1 -- 172.21.15.60:0/1543484979 learned_addr learned my addr 172.21.15.60:0/1543484979 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:00.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.877+0000 7f17ce7fc700 1 --2- 172.21.15.60:0/1543484979 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17d013f3d0 0x7f17d0064920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:00.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.881+0000 7f17cf7fe700 1 -- 172.21.15.60:0/1543484979 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17d013f3d0 msgr2=0x7f17d0064920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:00.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.881+0000 7f17cf7fe700 1 --2- 172.21.15.60:0/1543484979 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17d013f3d0 0x7f17d0064920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:00.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.881+0000 7f17cf7fe700 1 -- 172.21.15.60:0/1543484979 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17d00b6c00 msgr2=0x7f17d00b17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:00.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.881+0000 7f17cf7fe700 1 --2- 172.21.15.60:0/1543484979 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17d00b6c00 0x7f17d00b17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:00.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.881+0000 7f17cf7fe700 1 -- 172.21.15.60:0/1543484979 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c000b1e0 con 0x7f17d0143820 2022-06-29T16:06:00.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.881+0000 7f17cf7fe700 1 --2- 172.21.15.60:0/1543484979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17d0143820 0x7f17d00afb10 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 rx=0x7f17c800c000 tx=0x7f17c80091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:00.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.881+0000 7f17bffff700 1 -- 172.21.15.60:0/1543484979 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f17c8012600 con 0x7f17d0143820 2022-06-29T16:06:00.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.881+0000 7f17bffff700 1 -- 172.21.15.60:0/1543484979 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c8012760 con 0x7f17d0143820 2022-06-29T16:06:00.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.881+0000 7f17d5732700 1 -- 172.21.15.60:0/1543484979 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17d00ae170 con 0x7f17d0143820 2022-06-29T16:06:00.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.881+0000 7f17bffff700 1 -- 172.21.15.60:0/1543484979 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f17c8012ac0 con 0x7f17d0143820 2022-06-29T16:06:00.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.881+0000 7f17d5732700 1 -- 172.21.15.60:0/1543484979 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17d00ad750 con 0x7f17d0143820 2022-06-29T16:06:00.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.881+0000 7f17d5732700 1 -- 172.21.15.60:0/1543484979 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17d00af7f0 con 0x7f17d0143820 2022-06-29T16:06:00.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.881+0000 7f17bffff700 1 -- 172.21.15.60:0/1543484979 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f17c8022070 con 0x7f17d0143820 2022-06-29T16:06:00.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.881+0000 7f17bffff700 1 --2- 172.21.15.60:0/1543484979 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f17ac053a70 0x7f17ac055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:00.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.881+0000 7f17bffff700 1 -- 172.21.15.60:0/1543484979 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f17c806c260 con 0x7f17d0143820 2022-06-29T16:06:00.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.881+0000 7f17ceffd700 1 --2- 172.21.15.60:0/1543484979 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f17ac053a70 0x7f17ac055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:00.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.885+0000 7f17ceffd700 1 --2- 172.21.15.60:0/1543484979 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f17ac053a70 0x7f17ac055f30 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f17c001e000 tx=0x7f17c0017000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:00.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:00.889+0000 7f17bffff700 1 -- 172.21.15.60:0/1543484979 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f17c8043460 con 0x7f17d0143820 2022-06-29T16:06:01.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:01.025+0000 7f17d5732700 1 -- 172.21.15.60:0/1543484979 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f17d00573d0 con 0x7f17d0143820 2022-06-29T16:06:01.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:01.881+0000 7f17bffff700 1 -- 172.21.15.60:0/1543484979 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v96) v1 ==== 97+0+0 (secure 0 0 0) 0x7f17c8016020 con 0x7f17d0143820 2022-06-29T16:06:01.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:01.885+0000 7f17d5732700 1 -- 172.21.15.60:0/1543484979 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f17ac053a70 msgr2=0x7f17ac055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:01.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:01.885+0000 7f17d5732700 1 --2- 172.21.15.60:0/1543484979 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f17ac053a70 0x7f17ac055f30 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f17c001e000 tx=0x7f17c0017000).stop 2022-06-29T16:06:01.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:01.885+0000 7f17d5732700 1 -- 172.21.15.60:0/1543484979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17d0143820 msgr2=0x7f17d00afb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:01.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:01.885+0000 7f17d5732700 1 --2- 172.21.15.60:0/1543484979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17d0143820 0x7f17d00afb10 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 rx=0x7f17c800c000 tx=0x7f17c80091d0).stop 2022-06-29T16:06:01.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:01.885+0000 7f17d5732700 1 -- 172.21.15.60:0/1543484979 shutdown_connections 2022-06-29T16:06:01.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:01.885+0000 7f17d5732700 1 --2- 172.21.15.60:0/1543484979 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f17ac053a70 0x7f17ac055f30 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:01.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:01.885+0000 7f17d5732700 1 --2- 172.21.15.60:0/1543484979 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f17d0143820 0x7f17d00afb10 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:01.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:01.885+0000 7f17d5732700 1 --2- 172.21.15.60:0/1543484979 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f17d013f3d0 0x7f17d0064920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:01.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:01.885+0000 7f17d5732700 1 --2- 172.21.15.60:0/1543484979 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17d00b6c00 0x7f17d00b17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:01.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:01.885+0000 7f17d5732700 1 -- 172.21.15.60:0/1543484979 >> 172.21.15.60:0/1543484979 conn(0x7f17d00643e0 msgr2=0x7f17d00b2990 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:01.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:01.889+0000 7f17d5732700 1 -- 172.21.15.60:0/1543484979 shutdown_connections 2022-06-29T16:06:01.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:01.889+0000 7f17d5732700 1 -- 172.21.15.60:0/1543484979 wait complete. 2022-06-29T16:06:01.895 INFO:teuthology.orchestra.run.smithi060.stderr: removed all blacklist entries 2022-06-29T16:06:01.911 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-06-29T16:06:01.911 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-06-29T16:06:02.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.017+0000 7f8c2a73a700 1 Processor -- start 2022-06-29T16:06:02.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.017+0000 7f8c2a73a700 1 -- start start 2022-06-29T16:06:02.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.017+0000 7f8c2a73a700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8c2405d070 0x7f8c2414aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:02.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.017+0000 7f8c2a73a700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c2405d450 con 0x7f8c2414af80 2022-06-29T16:06:02.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.017+0000 7f8c2a73a700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c240ad840 con 0x7f8c240b2130 2022-06-29T16:06:02.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.017+0000 7f8c2a73a700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c240adf90 con 0x7f8c2405d070 2022-06-29T16:06:02.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.017+0000 7f8c29f39700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f8c2414af80 0x7f8c240ad130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59332/0 (socket says 172.21.15.60:59332) 2022-06-29T16:06:02.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.017+0000 7f8c29f39700 1 -- 172.21.15.60:0/2322274818 learned_addr learned my addr 172.21.15.60:0/2322274818 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:02.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.017+0000 7f8c29738700 1 --2- 172.21.15.60:0/2322274818 >> v2:172.21.15.92:3300/0 conn(0x7f8c2405d070 0x7f8c2414aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:02.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.017+0000 7f8c23fff700 1 -- 172.21.15.60:0/2322274818 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1958648597 0 0) 0x7f8c2405d450 con 0x7f8c2414af80 2022-06-29T16:06:02.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.017+0000 7f8c23fff700 1 -- 172.21.15.60:0/2322274818 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c10003260 con 0x7f8c2414af80 2022-06-29T16:06:02.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.017+0000 7f8c23fff700 1 -- 172.21.15.60:0/2322274818 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3155599741 0 0) 0x7f8c240ad840 con 0x7f8c240b2130 2022-06-29T16:06:02.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.017+0000 7f8c23fff700 1 -- 172.21.15.60:0/2322274818 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c2405d450 con 0x7f8c240b2130 2022-06-29T16:06:02.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.017+0000 7f8c29738700 1 -- 172.21.15.60:0/2322274818 >> v1:172.21.15.92:6789/0 conn(0x7f8c240b2130 legacy=0x7f8c240aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:02.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.017+0000 7f8c29738700 1 -- 172.21.15.60:0/2322274818 >> v1:172.21.15.60:6789/0 conn(0x7f8c2414af80 legacy=0x7f8c240ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:02.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.017+0000 7f8c23fff700 1 -- 172.21.15.60:0/2322274818 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3577577253 0 0) 0x7f8c10003260 con 0x7f8c2414af80 2022-06-29T16:06:02.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.017+0000 7f8c29738700 1 -- 172.21.15.60:0/2322274818 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c240b25d0 con 0x7f8c2405d070 2022-06-29T16:06:02.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.017+0000 7f8c29738700 1 --2- 172.21.15.60:0/2322274818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c2405d070 0x7f8c2414aa90 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7f8c1c001110 tx=0x7f8c18006df0).ready entity=mon.1 client_cookie=76d58c9c20b03b3b server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:02.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.017+0000 7f8c23fff700 1 -- 172.21.15.60:0/2322274818 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c1c001660 con 0x7f8c2405d070 2022-06-29T16:06:02.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.017+0000 7f8c23fff700 1 -- 172.21.15.60:0/2322274818 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c18000e90 con 0x7f8c2405d070 2022-06-29T16:06:02.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.017+0000 7f8c23fff700 1 -- 172.21.15.60:0/2322274818 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c18012460 con 0x7f8c2405d070 2022-06-29T16:06:02.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.021+0000 7f8c2a73a700 1 -- 172.21.15.60:0/2322274818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c2405d070 msgr2=0x7f8c2414aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:02.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.021+0000 7f8c2a73a700 1 --2- 172.21.15.60:0/2322274818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c2405d070 0x7f8c2414aa90 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7f8c1c001110 tx=0x7f8c18006df0).stop 2022-06-29T16:06:02.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.021+0000 7f8c2a73a700 1 -- 172.21.15.60:0/2322274818 shutdown_connections 2022-06-29T16:06:02.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.021+0000 7f8c2a73a700 1 --2- 172.21.15.60:0/2322274818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c2405d070 0x7f8c2414aa90 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:02.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.021+0000 7f8c2a73a700 1 -- 172.21.15.60:0/2322274818 >> 172.21.15.60:0/2322274818 conn(0x7f8c240b5890 msgr2=0x7f8c240b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:02.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.021+0000 7f8c2a73a700 1 -- 172.21.15.60:0/2322274818 shutdown_connections 2022-06-29T16:06:02.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.021+0000 7f8c2a73a700 1 -- 172.21.15.60:0/2322274818 wait complete. 2022-06-29T16:06:02.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.021+0000 7f8c2a73a700 1 Processor -- start 2022-06-29T16:06:02.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.021+0000 7f8c2a73a700 1 -- start start 2022-06-29T16:06:02.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.021+0000 7f8c2a73a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c2405d070 0x7f8c240630c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:02.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.021+0000 7f8c2a73a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8c240b2130 0x7f8c24159ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:02.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.021+0000 7f8c2a73a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c2414af80 0x7f8c24159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:02.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.021+0000 7f8c2a73a700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c241172b0 con 0x7f8c2405d070 2022-06-29T16:06:02.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.021+0000 7f8c2a73a700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c24116d70 con 0x7f8c2414af80 2022-06-29T16:06:02.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.021+0000 7f8c2a73a700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c24156db0 con 0x7f8c240b2130 2022-06-29T16:06:02.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.021+0000 7f8c29738700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c2405d070 0x7f8c240630c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:02.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.021+0000 7f8c29738700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c2405d070 0x7f8c240630c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51248/0 (socket says 172.21.15.60:51248) 2022-06-29T16:06:02.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.021+0000 7f8c29738700 1 -- 172.21.15.60:0/1077666659 learned_addr learned my addr 172.21.15.60:0/1077666659 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:02.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.021+0000 7f8c29f39700 1 --2- 172.21.15.60:0/1077666659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c2414af80 0x7f8c24159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:02.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.021+0000 7f8c28f37700 1 --2- 172.21.15.60:0/1077666659 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8c240b2130 0x7f8c24159ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:02.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.021+0000 7f8c29738700 1 -- 172.21.15.60:0/1077666659 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8c240b2130 msgr2=0x7f8c24159ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:02.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.021+0000 7f8c29738700 1 --2- 172.21.15.60:0/1077666659 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8c240b2130 0x7f8c24159ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:02.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.021+0000 7f8c29738700 1 -- 172.21.15.60:0/1077666659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c2414af80 msgr2=0x7f8c24159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:02.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.021+0000 7f8c29738700 1 --2- 172.21.15.60:0/1077666659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c2414af80 0x7f8c24159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:02.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.021+0000 7f8c29738700 1 -- 172.21.15.60:0/1077666659 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c1800b180 con 0x7f8c2405d070 2022-06-29T16:06:02.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.021+0000 7f8c29738700 1 --2- 172.21.15.60:0/1077666659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c2405d070 0x7f8c240630c0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 rx=0x7f8c1c001110 tx=0x7f8c18014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:02.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.021+0000 7f8c227fc700 1 -- 172.21.15.60:0/1077666659 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c1800d8a0 con 0x7f8c2405d070 2022-06-29T16:06:02.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.021+0000 7f8c227fc700 1 -- 172.21.15.60:0/1077666659 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c1800da00 con 0x7f8c2405d070 2022-06-29T16:06:02.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.021+0000 7f8c2a73a700 1 -- 172.21.15.60:0/1077666659 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c24065270 con 0x7f8c2405d070 2022-06-29T16:06:02.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.021+0000 7f8c227fc700 1 -- 172.21.15.60:0/1077666659 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c1800dd60 con 0x7f8c2405d070 2022-06-29T16:06:02.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.025+0000 7f8c2a73a700 1 -- 172.21.15.60:0/1077666659 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c2415a640 con 0x7f8c2405d070 2022-06-29T16:06:02.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.025+0000 7f8c2a73a700 1 -- 172.21.15.60:0/1077666659 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c2414a350 con 0x7f8c2405d070 2022-06-29T16:06:02.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.025+0000 7f8c227fc700 1 -- 172.21.15.60:0/1077666659 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8c18025030 con 0x7f8c2405d070 2022-06-29T16:06:02.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.025+0000 7f8c227fc700 1 --2- 172.21.15.60:0/1077666659 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8c04053a10 0x7f8c04055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:02.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.025+0000 7f8c227fc700 1 -- 172.21.15.60:0/1077666659 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f8c1806c710 con 0x7f8c2405d070 2022-06-29T16:06:02.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.025+0000 7f8c28f37700 1 --2- 172.21.15.60:0/1077666659 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8c04053a10 0x7f8c04055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:02.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.029+0000 7f8c28f37700 1 --2- 172.21.15.60:0/1077666659 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8c04053a10 0x7f8c04055ed0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7f8c1400d000 tx=0x7f8c14007da0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:02.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.033+0000 7f8c227fc700 1 -- 172.21.15.60:0/1077666659 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8c18043850 con 0x7f8c2405d070 2022-06-29T16:06:02.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.201+0000 7f8c2a73a700 1 -- 172.21.15.60:0/1077666659 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f8c240573d0 con 0x7f8c2405d070 2022-06-29T16:06:02.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.201+0000 7f8c227fc700 1 -- 172.21.15.60:0/1077666659 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v33) v1 ==== 80+0+4242 (secure 0 0 0) 0x7f8c18023070 con 0x7f8c2405d070 2022-06-29T16:06:02.209 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:02.209 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:06:02.210 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_dump": [ 2022-06-29T16:06:02.210 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:02.211 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.a", 2022-06-29T16:06:02.211 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxivv7JNhAAnA291UxlB/SfgITOipP00A==", 2022-06-29T16:06:02.211 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:06:02.212 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:06:02.212 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:06:02.212 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:06:02.213 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:06:02.213 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:02.214 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:02.214 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:02.214 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.b", 2022-06-29T16:06:02.215 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxiRSr/OhAAfFQTF2YHvzCv5wXBVCA5WQ==", 2022-06-29T16:06:02.215 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:06:02.216 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:06:02.216 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:06:02.216 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:06:02.217 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:06:02.217 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:02.217 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:02.218 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:02.218 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.0", 2022-06-29T16:06:02.219 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiZYPxOBAAIkweWJlTJIXgxnlagkGdXA==", 2022-06-29T16:06:02.219 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:06:02.219 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:06:02.220 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:06:02.220 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:06:02.221 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:02.221 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:02.221 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:02.222 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.1", 2022-06-29T16:06:02.222 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxiS2JGBhAAJn0UfAxZI3IVFyD5QBdb/Q==", 2022-06-29T16:06:02.222 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:06:02.223 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:06:02.223 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:06:02.223 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:06:02.224 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:02.224 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:02.225 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:02.225 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.2", 2022-06-29T16:06:02.225 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxiPhkfEhAApH7r8OUJzW7fZNM6XVv1dg==", 2022-06-29T16:06:02.226 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:06:02.226 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:06:02.227 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:06:02.227 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:06:02.227 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:02.228 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:02.228 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:02.228 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.3", 2022-06-29T16:06:02.229 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxit/47HhAA1vyU6h7hjPG+op9qP/rVYQ==", 2022-06-29T16:06:02.229 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:06:02.230 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:06:02.230 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:06:02.230 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:06:02.231 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:02.231 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:02.232 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:02.232 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.4", 2022-06-29T16:06:02.232 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxiaKzKBxAAbIwCF9ag7ezsNy11hi0BEw==", 2022-06-29T16:06:02.233 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:06:02.233 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:06:02.233 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:06:02.234 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:06:02.234 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:02.234 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:02.235 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:02.235 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.5", 2022-06-29T16:06:02.236 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxiJQV/IBAASBBDb6UFgEtXILe4fcT/xA==", 2022-06-29T16:06:02.236 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:06:02.236 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:06:02.237 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:06:02.237 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:06:02.237 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:02.238 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:02.238 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:02.239 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.6", 2022-06-29T16:06:02.239 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxi6D07OBAA2FWiZ+mNK+jFMp/kPZnbQQ==", 2022-06-29T16:06:02.239 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:06:02.240 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:06:02.240 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:06:02.240 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:06:02.241 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:02.241 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:02.242 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:02.242 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.7", 2022-06-29T16:06:02.242 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCMdrxinEOcEhAA562ybpzGHNQfWIq8q7zw9w==", 2022-06-29T16:06:02.243 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:06:02.243 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:06:02.243 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:06:02.244 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:06:02.244 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:02.245 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:02.245 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:02.245 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.0", 2022-06-29T16:06:02.246 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiHXPwBRAAkOOwlYJvIW7dqVFdlXXoqw==", 2022-06-29T16:06:02.246 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:06:02.246 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:06:02.246 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2022-06-29T16:06:02.247 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw pool=cephfs_data" 2022-06-29T16:06:02.247 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:02.247 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:02.247 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:02.247 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.1", 2022-06-29T16:06:02.248 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxibe5JCRAAGBJbi9t09w0836JqrAT7+w==", 2022-06-29T16:06:02.248 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:06:02.248 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:06:02.248 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2022-06-29T16:06:02.249 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw pool=cephfs_data" 2022-06-29T16:06:02.249 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:02.249 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:02.249 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:02.249 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.2", 2022-06-29T16:06:02.250 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiK3dBDRAA7XINPSOPQDfOrUYEpHC+gw==", 2022-06-29T16:06:02.250 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:06:02.250 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:06:02.250 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2022-06-29T16:06:02.251 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw pool=cephfs_data" 2022-06-29T16:06:02.251 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:02.251 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:02.251 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:02.251 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.admin", 2022-06-29T16:06:02.252 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxieKVcGBAAnjH80a0qWIArz2qwjYHWdQ==", 2022-06-29T16:06:02.252 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:06:02.252 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T16:06:02.252 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:06:02.253 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:06:02.253 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:06:02.253 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:02.253 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:02.253 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:02.254 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.x", 2022-06-29T16:06:02.254 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxi2LZJNBAA3MKb/pXYbQCsoRx8EQfGbw==", 2022-06-29T16:06:02.254 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:06:02.254 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T16:06:02.255 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2022-06-29T16:06:02.255 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:06:02.255 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:02.255 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:02.255 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:02.256 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.y", 2022-06-29T16:06:02.256 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxi3LmaMBAAN+0mZrdKXqSXOipVTA1drw==", 2022-06-29T16:06:02.257 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:06:02.257 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T16:06:02.258 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2022-06-29T16:06:02.258 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:06:02.259 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:02.259 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:02.259 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:02.260 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:06:02.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.205+0000 7f8c2a73a700 1 -- 172.21.15.60:0/1077666659 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8c04053a10 msgr2=0x7f8c04055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:02.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.205+0000 7f8c2a73a700 1 --2- 172.21.15.60:0/1077666659 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8c04053a10 0x7f8c04055ed0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7f8c1400d000 tx=0x7f8c14007da0).stop 2022-06-29T16:06:02.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.205+0000 7f8c2a73a700 1 -- 172.21.15.60:0/1077666659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c2405d070 msgr2=0x7f8c240630c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:02.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.205+0000 7f8c2a73a700 1 --2- 172.21.15.60:0/1077666659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c2405d070 0x7f8c240630c0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 rx=0x7f8c1c001110 tx=0x7f8c18014000).stop 2022-06-29T16:06:02.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.205+0000 7f8c2a73a700 1 -- 172.21.15.60:0/1077666659 shutdown_connections 2022-06-29T16:06:02.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.205+0000 7f8c2a73a700 1 --2- 172.21.15.60:0/1077666659 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8c04053a10 0x7f8c04055ed0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:02.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.205+0000 7f8c2a73a700 1 --2- 172.21.15.60:0/1077666659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c2414af80 0x7f8c24159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:02.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.205+0000 7f8c2a73a700 1 --2- 172.21.15.60:0/1077666659 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8c240b2130 0x7f8c24159ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:02.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.205+0000 7f8c2a73a700 1 --2- 172.21.15.60:0/1077666659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8c2405d070 0x7f8c240630c0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:02.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.205+0000 7f8c2a73a700 1 -- 172.21.15.60:0/1077666659 >> 172.21.15.60:0/1077666659 conn(0x7f8c240b5890 msgr2=0x7f8c24143180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:02.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.205+0000 7f8c2a73a700 1 -- 172.21.15.60:0/1077666659 shutdown_connections 2022-06-29T16:06:02.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.205+0000 7f8c2a73a700 1 -- 172.21.15.60:0/1077666659 wait complete. 2022-06-29T16:06:02.270 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:06:02.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.373+0000 7fa370109700 1 Processor -- start 2022-06-29T16:06:02.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.373+0000 7fa370109700 1 -- start start 2022-06-29T16:06:02.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.377+0000 7fa370109700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa3680b5950 0x7fa368067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:02.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.377+0000 7fa370109700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa36814af10 0x7fa36805c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:02.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.377+0000 7fa370109700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3680acba0 con 0x7fa368063ad0 2022-06-29T16:06:02.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.377+0000 7fa370109700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa36814b3a0 con 0x7fa36814af10 2022-06-29T16:06:02.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.377+0000 7fa370109700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa368067220 con 0x7fa3680b5950 2022-06-29T16:06:02.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.377+0000 7fa36e6a6700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa36814af10 0x7fa36805c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:02.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.377+0000 7fa36e6a6700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa36814af10 0x7fa36805c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51260/0 (socket says 172.21.15.60:51260) 2022-06-29T16:06:02.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.377+0000 7fa36e6a6700 1 -- 172.21.15.60:0/3546966688 learned_addr learned my addr 172.21.15.60:0/3546966688 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:02.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.377+0000 7fa36dea5700 1 --2- 172.21.15.60:0/3546966688 >> v2:172.21.15.92:3300/0 conn(0x7fa3680b5950 0x7fa368067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:02.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.377+0000 7fa36e6a6700 1 -- 172.21.15.60:0/3546966688 >> v2:172.21.15.92:3300/0 conn(0x7fa3680b5950 msgr2=0x7fa368067680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:02.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.377+0000 7fa36e6a6700 1 --2- 172.21.15.60:0/3546966688 >> v2:172.21.15.92:3300/0 conn(0x7fa3680b5950 0x7fa368067680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:02.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.377+0000 7fa36e6a6700 1 -- 172.21.15.60:0/3546966688 >> v1:172.21.15.60:6789/0 conn(0x7fa368063ad0 legacy=0x7fa36805bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:02.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.377+0000 7fa36e6a6700 1 -- 172.21.15.60:0/3546966688 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3680b5d30 con 0x7fa36814af10 2022-06-29T16:06:02.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.377+0000 7fa36dea5700 1 --2- 172.21.15.60:0/3546966688 >> v2:172.21.15.92:3300/0 conn(0x7fa3680b5950 0x7fa368067680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:06:02.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.377+0000 7fa36e6a6700 1 --2- 172.21.15.60:0/3546966688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa36814af10 0x7fa36805c430 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 rx=0x7fa3680ad0a0 tx=0x7fa360006eb0).ready entity=mon.0 client_cookie=ef55253f40afc8d7 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:02.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.377+0000 7fa36cea3700 1 -- 172.21.15.60:0/3546966688 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa36000d5c0 con 0x7fa36814af10 2022-06-29T16:06:02.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.377+0000 7fa36cea3700 1 -- 172.21.15.60:0/3546966688 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa360000e90 con 0x7fa36814af10 2022-06-29T16:06:02.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.377+0000 7fa36cea3700 1 -- 172.21.15.60:0/3546966688 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa36000d910 con 0x7fa36814af10 2022-06-29T16:06:02.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.377+0000 7fa370109700 1 -- 172.21.15.60:0/3546966688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa36814af10 msgr2=0x7fa36805c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:02.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.377+0000 7fa370109700 1 --2- 172.21.15.60:0/3546966688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa36814af10 0x7fa36805c430 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 rx=0x7fa3680ad0a0 tx=0x7fa360006eb0).stop 2022-06-29T16:06:02.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.377+0000 7fa370109700 1 -- 172.21.15.60:0/3546966688 shutdown_connections 2022-06-29T16:06:02.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.377+0000 7fa370109700 1 --2- 172.21.15.60:0/3546966688 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa36814af10 0x7fa36805c430 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:02.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.377+0000 7fa370109700 1 --2- 172.21.15.60:0/3546966688 >> v2:172.21.15.92:3300/0 conn(0x7fa3680b5950 0x7fa368067680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:02.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.377+0000 7fa370109700 1 -- 172.21.15.60:0/3546966688 >> 172.21.15.60:0/3546966688 conn(0x7fa3680632c0 msgr2=0x7fa3680b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:02.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.377+0000 7fa370109700 1 -- 172.21.15.60:0/3546966688 shutdown_connections 2022-06-29T16:06:02.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.377+0000 7fa370109700 1 -- 172.21.15.60:0/3546966688 wait complete. 2022-06-29T16:06:02.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.381+0000 7fa370109700 1 Processor -- start 2022-06-29T16:06:02.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.381+0000 7fa370109700 1 -- start start 2022-06-29T16:06:02.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.381+0000 7fa370109700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa368063ad0 0x7fa368147330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:02.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.381+0000 7fa370109700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3680b5950 0x7fa3681596d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:02.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.381+0000 7fa370109700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa36814af10 0x7fa368159bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:02.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.381+0000 7fa370109700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3680af540 con 0x7fa368063ad0 2022-06-29T16:06:02.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.381+0000 7fa370109700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3680b43e0 con 0x7fa36814af10 2022-06-29T16:06:02.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.381+0000 7fa370109700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3680b48e0 con 0x7fa3680b5950 2022-06-29T16:06:02.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.381+0000 7fa36dea5700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa368063ad0 0x7fa368147330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:02.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.381+0000 7fa36dea5700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa368063ad0 0x7fa368147330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51262/0 (socket says 172.21.15.60:51262) 2022-06-29T16:06:02.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.381+0000 7fa36dea5700 1 -- 172.21.15.60:0/918441403 learned_addr learned my addr 172.21.15.60:0/918441403 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:02.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.381+0000 7fa36d6a4700 1 --2- 172.21.15.60:0/918441403 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3680b5950 0x7fa3681596d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:02.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.381+0000 7fa36e6a6700 1 --2- 172.21.15.60:0/918441403 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa36814af10 0x7fa368159bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:02.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.381+0000 7fa36dea5700 1 -- 172.21.15.60:0/918441403 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3680b5950 msgr2=0x7fa3681596d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:02.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.381+0000 7fa36dea5700 1 --2- 172.21.15.60:0/918441403 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3680b5950 0x7fa3681596d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:02.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.381+0000 7fa36dea5700 1 -- 172.21.15.60:0/918441403 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa36814af10 msgr2=0x7fa368159bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:02.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.381+0000 7fa36dea5700 1 --2- 172.21.15.60:0/918441403 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa36814af10 0x7fa368159bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:02.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.381+0000 7fa36dea5700 1 -- 172.21.15.60:0/918441403 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa36000b1e0 con 0x7fa368063ad0 2022-06-29T16:06:02.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.385+0000 7fa36dea5700 1 --2- 172.21.15.60:0/918441403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa368063ad0 0x7fa368147330 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 rx=0x7fa3680b4a70 tx=0x7fa364008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:02.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.385+0000 7fa35affd700 1 -- 172.21.15.60:0/918441403 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa364010580 con 0x7fa368063ad0 2022-06-29T16:06:02.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.385+0000 7fa35affd700 1 -- 172.21.15.60:0/918441403 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3640106e0 con 0x7fa368063ad0 2022-06-29T16:06:02.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.385+0000 7fa370109700 1 -- 172.21.15.60:0/918441403 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3680b4640 con 0x7fa368063ad0 2022-06-29T16:06:02.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.385+0000 7fa35affd700 1 -- 172.21.15.60:0/918441403 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa364010a40 con 0x7fa368063ad0 2022-06-29T16:06:02.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.385+0000 7fa370109700 1 -- 172.21.15.60:0/918441403 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa36815a280 con 0x7fa368063ad0 2022-06-29T16:06:02.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.385+0000 7fa370109700 1 -- 172.21.15.60:0/918441403 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa368159470 con 0x7fa368063ad0 2022-06-29T16:06:02.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.385+0000 7fa35affd700 1 -- 172.21.15.60:0/918441403 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa364010ba0 con 0x7fa368063ad0 2022-06-29T16:06:02.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.385+0000 7fa35affd700 1 --2- 172.21.15.60:0/918441403 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa348053b00 0x7fa348055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:02.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.385+0000 7fa35affd700 1 -- 172.21.15.60:0/918441403 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fa36406bab0 con 0x7fa368063ad0 2022-06-29T16:06:02.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.385+0000 7fa36d6a4700 1 --2- 172.21.15.60:0/918441403 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa348053b00 0x7fa348055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:02.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.389+0000 7fa36d6a4700 1 --2- 172.21.15.60:0/918441403 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa348053b00 0x7fa348055fc0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7fa35c00d000 tx=0x7fa35c007f10).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:02.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.393+0000 7fa35affd700 1 -- 172.21.15.60:0/918441403 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa364042d00 con 0x7fa368063ad0 2022-06-29T16:06:02.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.521+0000 7fa370109700 1 -- 172.21.15.60:0/918441403 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa36815a280 con 0x7fa368063ad0 2022-06-29T16:06:02.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.521+0000 7fa35affd700 1 -- 172.21.15.60:0/918441403 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+13182 (secure 0 0 0) 0x7fa36815a280 con 0x7fa368063ad0 2022-06-29T16:06:02.528 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:02.528 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":96,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:06:01.872030+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:06:02.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.525+0000 7fa370109700 1 -- 172.21.15.60:0/918441403 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa348053b00 msgr2=0x7fa348055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:02.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.525+0000 7fa370109700 1 --2- 172.21.15.60:0/918441403 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa348053b00 0x7fa348055fc0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7fa35c00d000 tx=0x7fa35c007f10).stop 2022-06-29T16:06:02.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.525+0000 7fa370109700 1 -- 172.21.15.60:0/918441403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa368063ad0 msgr2=0x7fa368147330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:02.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.525+0000 7fa370109700 1 --2- 172.21.15.60:0/918441403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa368063ad0 0x7fa368147330 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 rx=0x7fa3680b4a70 tx=0x7fa364008040).stop 2022-06-29T16:06:02.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.525+0000 7fa370109700 1 -- 172.21.15.60:0/918441403 shutdown_connections 2022-06-29T16:06:02.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.525+0000 7fa370109700 1 --2- 172.21.15.60:0/918441403 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa348053b00 0x7fa348055fc0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:02.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.525+0000 7fa370109700 1 --2- 172.21.15.60:0/918441403 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa36814af10 0x7fa368159bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:02.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.525+0000 7fa370109700 1 --2- 172.21.15.60:0/918441403 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa3680b5950 0x7fa3681596d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:02.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.525+0000 7fa370109700 1 --2- 172.21.15.60:0/918441403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa368063ad0 0x7fa368147330 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:02.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.525+0000 7fa370109700 1 -- 172.21.15.60:0/918441403 >> 172.21.15.60:0/918441403 conn(0x7fa3680632c0 msgr2=0x7fa368145f50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:02.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.525+0000 7fa370109700 1 -- 172.21.15.60:0/918441403 shutdown_connections 2022-06-29T16:06:02.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.525+0000 7fa370109700 1 -- 172.21.15.60:0/918441403 wait complete. 2022-06-29T16:06:02.541 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-06-29T16:06:02.541 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:06:02.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.645+0000 7f5bbd6ea700 1 Processor -- start 2022-06-29T16:06:02.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.645+0000 7f5bbd6ea700 1 -- start start 2022-06-29T16:06:02.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.645+0000 7f5bbd6ea700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5bb813ca00 0x7f5bb813ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:02.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.645+0000 7f5bbd6ea700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb80b2fa0 con 0x7f5bb80b07f0 2022-06-29T16:06:02.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.645+0000 7f5bbd6ea700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb80ae250 con 0x7f5bb80b2bc0 2022-06-29T16:06:02.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.645+0000 7f5bbd6ea700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bb80ae4f0 con 0x7f5bb813ca00 2022-06-29T16:06:02.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.645+0000 7f5bbcee9700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5bb813ca00 0x7f5bb813ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:02.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.645+0000 7f5bb77fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5bb80b07f0 0x7f5bb8142ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59358/0 (socket says 172.21.15.60:59358) 2022-06-29T16:06:02.660 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.645+0000 7f5bb77fe700 1 -- 172.21.15.60:0/3009788113 learned_addr learned my addr 172.21.15.60:0/3009788113 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:02.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.649+0000 7f5bb6ffd700 1 -- 172.21.15.60:0/3009788113 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 13791756 0 0) 0x7f5bb80b2fa0 con 0x7f5bb80b07f0 2022-06-29T16:06:02.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.649+0000 7f5bb6ffd700 1 -- 172.21.15.60:0/3009788113 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ba4003260 con 0x7f5bb80b07f0 2022-06-29T16:06:02.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.649+0000 7f5bbcee9700 1 -- 172.21.15.60:0/3009788113 >> v1:172.21.15.92:6789/0 conn(0x7f5bb80b2bc0 legacy=0x7f5bb80ae990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:02.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.649+0000 7f5bbcee9700 1 -- 172.21.15.60:0/3009788113 >> v1:172.21.15.60:6789/0 conn(0x7f5bb80b07f0 legacy=0x7f5bb8142ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:02.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.649+0000 7f5bbcee9700 1 -- 172.21.15.60:0/3009788113 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb81435f0 con 0x7f5bb813ca00 2022-06-29T16:06:02.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.649+0000 7f5bbcee9700 1 --2- 172.21.15.60:0/3009788113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5bb813ca00 0x7f5bb813ce50 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 rx=0x7f5bb00032f0 tx=0x7f5bb00091d0).ready entity=mon.0 client_cookie=1d21ec9dc7e39f35 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:02.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.649+0000 7f5bb6ffd700 1 -- 172.21.15.60:0/3009788113 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bb000b6c0 con 0x7f5bb813ca00 2022-06-29T16:06:02.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.649+0000 7f5bb6ffd700 1 -- 172.21.15.60:0/3009788113 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bb000b820 con 0x7f5bb813ca00 2022-06-29T16:06:02.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.649+0000 7f5bb6ffd700 1 -- 172.21.15.60:0/3009788113 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bb000b990 con 0x7f5bb813ca00 2022-06-29T16:06:02.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.649+0000 7f5bbd6ea700 1 -- 172.21.15.60:0/3009788113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5bb813ca00 msgr2=0x7f5bb813ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:02.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.649+0000 7f5bbd6ea700 1 --2- 172.21.15.60:0/3009788113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5bb813ca00 0x7f5bb813ce50 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 rx=0x7f5bb00032f0 tx=0x7f5bb00091d0).stop 2022-06-29T16:06:02.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.649+0000 7f5bbd6ea700 1 -- 172.21.15.60:0/3009788113 shutdown_connections 2022-06-29T16:06:02.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.649+0000 7f5bbd6ea700 1 --2- 172.21.15.60:0/3009788113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5bb813ca00 0x7f5bb813ce50 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:02.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.649+0000 7f5bbd6ea700 1 -- 172.21.15.60:0/3009788113 >> 172.21.15.60:0/3009788113 conn(0x7f5bb80637c0 msgr2=0x7f5bb8063ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:02.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.649+0000 7f5bbd6ea700 1 -- 172.21.15.60:0/3009788113 shutdown_connections 2022-06-29T16:06:02.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.649+0000 7f5bbd6ea700 1 -- 172.21.15.60:0/3009788113 wait complete. 2022-06-29T16:06:02.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.653+0000 7f5bbd6ea700 1 Processor -- start 2022-06-29T16:06:02.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.653+0000 7f5bbd6ea700 1 -- start start 2022-06-29T16:06:02.675 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.653+0000 7f5bbd6ea700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5bb80b07f0 0x7f5bb812ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:02.675 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.653+0000 7f5bbd6ea700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5bb80b2bc0 0x7f5bb80b5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:02.676 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.653+0000 7f5bbd6ea700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5bb813ca00 0x7f5bb8064980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:02.676 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.653+0000 7f5bbd6ea700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb812e230 con 0x7f5bb813ca00 2022-06-29T16:06:02.677 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.653+0000 7f5bbd6ea700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb814ce70 con 0x7f5bb80b07f0 2022-06-29T16:06:02.677 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.653+0000 7f5bbd6ea700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb80644e0 con 0x7f5bb80b2bc0 2022-06-29T16:06:02.678 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.653+0000 7f5bbcee9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5bb813ca00 0x7f5bb8064980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:02.678 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.653+0000 7f5bbcee9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5bb813ca00 0x7f5bb8064980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51280/0 (socket says 172.21.15.60:51280) 2022-06-29T16:06:02.678 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.653+0000 7f5bbcee9700 1 -- 172.21.15.60:0/1755145902 learned_addr learned my addr 172.21.15.60:0/1755145902 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:02.679 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.653+0000 7f5bb7fff700 1 --2- 172.21.15.60:0/1755145902 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5bb80b07f0 0x7f5bb812ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:02.679 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.653+0000 7f5bb77fe700 1 --2- 172.21.15.60:0/1755145902 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5bb80b2bc0 0x7f5bb80b5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:02.680 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.653+0000 7f5bbcee9700 1 -- 172.21.15.60:0/1755145902 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5bb80b2bc0 msgr2=0x7f5bb80b5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:02.680 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.653+0000 7f5bbcee9700 1 --2- 172.21.15.60:0/1755145902 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5bb80b2bc0 0x7f5bb80b5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:02.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.653+0000 7f5bbcee9700 1 -- 172.21.15.60:0/1755145902 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5bb80b07f0 msgr2=0x7f5bb812ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:02.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.653+0000 7f5bbcee9700 1 --2- 172.21.15.60:0/1755145902 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5bb80b07f0 0x7f5bb812ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:02.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.653+0000 7f5bbcee9700 1 -- 172.21.15.60:0/1755145902 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb0009470 con 0x7f5bb813ca00 2022-06-29T16:06:02.682 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.653+0000 7f5bbcee9700 1 --2- 172.21.15.60:0/1755145902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5bb813ca00 0x7f5bb8064980 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 rx=0x7f5bb00032f0 tx=0x7f5bb0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:02.682 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.653+0000 7f5bb57fa700 1 -- 172.21.15.60:0/1755145902 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bb000b6c0 con 0x7f5bb813ca00 2022-06-29T16:06:02.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.653+0000 7f5bb57fa700 1 -- 172.21.15.60:0/1755145902 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bb0021630 con 0x7f5bb813ca00 2022-06-29T16:06:02.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.653+0000 7f5bbd6ea700 1 -- 172.21.15.60:0/1755145902 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bb814caf0 con 0x7f5bb813ca00 2022-06-29T16:06:02.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.653+0000 7f5bb57fa700 1 -- 172.21.15.60:0/1755145902 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bb0021990 con 0x7f5bb813ca00 2022-06-29T16:06:02.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.657+0000 7f5bbd6ea700 1 -- 172.21.15.60:0/1755145902 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bb812cab0 con 0x7f5bb813ca00 2022-06-29T16:06:02.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.657+0000 7f5bbd6ea700 1 -- 172.21.15.60:0/1755145902 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bb8065cf0 con 0x7f5bb813ca00 2022-06-29T16:06:02.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.657+0000 7f5bb57fa700 1 -- 172.21.15.60:0/1755145902 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5bb0027020 con 0x7f5bb813ca00 2022-06-29T16:06:02.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.657+0000 7f5bb57fa700 1 --2- 172.21.15.60:0/1755145902 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5b98053950 0x7f5b98055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:02.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.657+0000 7f5bb57fa700 1 -- 172.21.15.60:0/1755145902 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f5bb006ce00 con 0x7f5bb813ca00 2022-06-29T16:06:02.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.657+0000 7f5bb7fff700 1 --2- 172.21.15.60:0/1755145902 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5b98053950 0x7f5b98055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:02.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.657+0000 7f5bb7fff700 1 --2- 172.21.15.60:0/1755145902 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5b98053950 0x7f5b98055e10 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7f5ba800d000 tx=0x7f5ba8007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:02.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.665+0000 7f5bb57fa700 1 -- 172.21.15.60:0/1755145902 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5bb8065cf0 con 0x7f5bb813ca00 2022-06-29T16:06:02.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.789+0000 7f5bbd6ea700 1 -- 172.21.15.60:0/1755145902 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f5bb8065cf0 con 0x7f5bb813ca00 2022-06-29T16:06:02.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.789+0000 7f5bb57fa700 1 -- 172.21.15.60:0/1755145902 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v96) v1 ==== 110+0+10 (secure 0 0 0) 0x7f5bb8065cf0 con 0x7f5bb813ca00 2022-06-29T16:06:02.796 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:06:02.797 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.793+0000 7f5bbd6ea700 1 -- 172.21.15.60:0/1755145902 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5b98053950 msgr2=0x7f5b98055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:02.798 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.793+0000 7f5bbd6ea700 1 --2- 172.21.15.60:0/1755145902 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5b98053950 0x7f5b98055e10 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7f5ba800d000 tx=0x7f5ba8007b90).stop 2022-06-29T16:06:02.798 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.793+0000 7f5bbd6ea700 1 -- 172.21.15.60:0/1755145902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5bb813ca00 msgr2=0x7f5bb8064980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:02.798 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.793+0000 7f5bbd6ea700 1 --2- 172.21.15.60:0/1755145902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5bb813ca00 0x7f5bb8064980 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 rx=0x7f5bb00032f0 tx=0x7f5bb0016000).stop 2022-06-29T16:06:02.798 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.793+0000 7f5bbd6ea700 1 -- 172.21.15.60:0/1755145902 shutdown_connections 2022-06-29T16:06:02.799 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.793+0000 7f5bbd6ea700 1 --2- 172.21.15.60:0/1755145902 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5b98053950 0x7f5b98055e10 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:02.799 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.793+0000 7f5bbd6ea700 1 --2- 172.21.15.60:0/1755145902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5bb813ca00 0x7f5bb8064980 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:02.799 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.793+0000 7f5bbd6ea700 1 --2- 172.21.15.60:0/1755145902 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5bb80b2bc0 0x7f5bb80b5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:02.799 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.793+0000 7f5bbd6ea700 1 --2- 172.21.15.60:0/1755145902 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5bb80b07f0 0x7f5bb812ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:02.800 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.793+0000 7f5bbd6ea700 1 -- 172.21.15.60:0/1755145902 >> 172.21.15.60:0/1755145902 conn(0x7f5bb80637c0 msgr2=0x7f5bb8140de0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:02.800 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.793+0000 7f5bbd6ea700 1 -- 172.21.15.60:0/1755145902 shutdown_connections 2022-06-29T16:06:02.800 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.793+0000 7f5bbd6ea700 1 -- 172.21.15.60:0/1755145902 wait complete. 2022-06-29T16:06:02.808 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:06:02.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.909+0000 7f7583fff700 1 Processor -- start 2022-06-29T16:06:02.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.909+0000 7f7583fff700 1 -- start start 2022-06-29T16:06:02.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.913+0000 7f7583fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f758412bde0 0x7f7584142ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:02.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.913+0000 7f7583fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7584066b20 0x7f7584142fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:02.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.913+0000 7f7583fff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75840ae9a0 con 0x7f758405d070 2022-06-29T16:06:02.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.913+0000 7f7583fff700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7584143b60 con 0x7f7584066b20 2022-06-29T16:06:02.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.913+0000 7f7583fff700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75840aced0 con 0x7f758412bde0 2022-06-29T16:06:02.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.913+0000 7f75837fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7584066b20 0x7f7584142fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:02.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.913+0000 7f7582ffd700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f758405d070 0x7f758412b6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59372/0 (socket says 172.21.15.60:59372) 2022-06-29T16:06:02.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.913+0000 7f7582ffd700 1 -- 172.21.15.60:0/2708733639 learned_addr learned my addr 172.21.15.60:0/2708733639 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:02.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.913+0000 7f75827fc700 1 --2- 172.21.15.60:0/2708733639 >> v2:172.21.15.92:3300/0 conn(0x7f758412bde0 0x7f7584142ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:02.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.913+0000 7f75837fe700 1 -- 172.21.15.60:0/2708733639 >> v2:172.21.15.92:3300/0 conn(0x7f758412bde0 msgr2=0x7f7584142ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:02.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.913+0000 7f75837fe700 1 --2- 172.21.15.60:0/2708733639 >> v2:172.21.15.92:3300/0 conn(0x7f758412bde0 0x7f7584142ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:02.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.913+0000 7f75837fe700 1 -- 172.21.15.60:0/2708733639 >> v1:172.21.15.60:6789/0 conn(0x7f758405d070 legacy=0x7f758412b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:02.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.913+0000 7f7581ffb700 1 -- 172.21.15.60:0/2708733639 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2206601872 0 0) 0x7f75840ae9a0 con 0x7f758405d070 2022-06-29T16:06:02.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.913+0000 7f75837fe700 1 -- 172.21.15.60:0/2708733639 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f758405d450 con 0x7f7584066b20 2022-06-29T16:06:02.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.913+0000 7f75837fe700 1 --2- 172.21.15.60:0/2708733639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7584066b20 0x7f7584142fd0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 rx=0x7f757c002f70 tx=0x7f757c006df0).ready entity=mon.0 client_cookie=f5779ba6a93cff8d server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:02.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.913+0000 7f7581ffb700 1 -- 172.21.15.60:0/2708733639 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f757c00d570 con 0x7f7584066b20 2022-06-29T16:06:02.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.913+0000 7f7581ffb700 1 -- 172.21.15.60:0/2708733639 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f757c000e90 con 0x7f7584066b20 2022-06-29T16:06:02.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.913+0000 7f7581ffb700 1 -- 172.21.15.60:0/2708733639 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f757c00d8c0 con 0x7f7584066b20 2022-06-29T16:06:02.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.913+0000 7f7583fff700 1 -- 172.21.15.60:0/2708733639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7584066b20 msgr2=0x7f7584142fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:02.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.913+0000 7f7583fff700 1 --2- 172.21.15.60:0/2708733639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7584066b20 0x7f7584142fd0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 rx=0x7f757c002f70 tx=0x7f757c006df0).stop 2022-06-29T16:06:02.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.913+0000 7f7583fff700 1 -- 172.21.15.60:0/2708733639 shutdown_connections 2022-06-29T16:06:02.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.913+0000 7f7583fff700 1 --2- 172.21.15.60:0/2708733639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7584066b20 0x7f7584142fd0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:02.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.913+0000 7f7583fff700 1 --2- 172.21.15.60:0/2708733639 >> v2:172.21.15.92:3300/0 conn(0x7f758412bde0 0x7f7584142ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:02.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.913+0000 7f7583fff700 1 -- 172.21.15.60:0/2708733639 >> 172.21.15.60:0/2708733639 conn(0x7f75840b5890 msgr2=0x7f75840b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:02.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.913+0000 7f7583fff700 1 -- 172.21.15.60:0/2708733639 shutdown_connections 2022-06-29T16:06:02.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.913+0000 7f7583fff700 1 -- 172.21.15.60:0/2708733639 wait complete. 2022-06-29T16:06:02.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.917+0000 7f7583fff700 1 Processor -- start 2022-06-29T16:06:02.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.917+0000 7f7583fff700 1 -- start start 2022-06-29T16:06:02.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.917+0000 7f7583fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f758405d070 0x7f758412dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:02.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.917+0000 7f7583fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7584066b20 0x7f7584151df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:02.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.917+0000 7f7583fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f758412bde0 0x7f75841522e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:02.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.917+0000 7f7583fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75841306f0 con 0x7f7584066b20 2022-06-29T16:06:02.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.917+0000 7f7583fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7584130450 con 0x7f758412bde0 2022-06-29T16:06:02.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.917+0000 7f7583fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f758412e360 con 0x7f758405d070 2022-06-29T16:06:02.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.917+0000 7f75827fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7584066b20 0x7f7584151df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:02.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.917+0000 7f75827fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7584066b20 0x7f7584151df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51290/0 (socket says 172.21.15.60:51290) 2022-06-29T16:06:02.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.917+0000 7f75827fc700 1 -- 172.21.15.60:0/1033082656 learned_addr learned my addr 172.21.15.60:0/1033082656 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:02.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.917+0000 7f7582ffd700 1 --2- 172.21.15.60:0/1033082656 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f758405d070 0x7f758412dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:02.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.917+0000 7f75837fe700 1 --2- 172.21.15.60:0/1033082656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f758412bde0 0x7f75841522e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:02.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.917+0000 7f75827fc700 1 -- 172.21.15.60:0/1033082656 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f758405d070 msgr2=0x7f758412dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:02.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.917+0000 7f75827fc700 1 --2- 172.21.15.60:0/1033082656 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f758405d070 0x7f758412dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:02.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.917+0000 7f75827fc700 1 -- 172.21.15.60:0/1033082656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f758412bde0 msgr2=0x7f75841522e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:02.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.917+0000 7f75827fc700 1 --2- 172.21.15.60:0/1033082656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f758412bde0 0x7f75841522e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:02.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.917+0000 7f75827fc700 1 -- 172.21.15.60:0/1033082656 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757c00b180 con 0x7f7584066b20 2022-06-29T16:06:02.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.917+0000 7f75827fc700 1 --2- 172.21.15.60:0/1033082656 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7584066b20 0x7f7584151df0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 rx=0x7f756c00c000 tx=0x7f756c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:02.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.921+0000 7f757bfff700 1 -- 172.21.15.60:0/1033082656 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f756c012600 con 0x7f7584066b20 2022-06-29T16:06:02.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.921+0000 7f757bfff700 1 -- 172.21.15.60:0/1033082656 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f756c012760 con 0x7f7584066b20 2022-06-29T16:06:02.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.921+0000 7f7583fff700 1 -- 172.21.15.60:0/1033082656 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f758412c4c0 con 0x7f7584066b20 2022-06-29T16:06:02.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.921+0000 7f757bfff700 1 -- 172.21.15.60:0/1033082656 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f756c012ac0 con 0x7f7584066b20 2022-06-29T16:06:02.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.921+0000 7f7583fff700 1 -- 172.21.15.60:0/1033082656 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75841527d0 con 0x7f7584066b20 2022-06-29T16:06:02.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.921+0000 7f7583fff700 1 -- 172.21.15.60:0/1033082656 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75840b1cb0 con 0x7f7584066b20 2022-06-29T16:06:02.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.921+0000 7f757bfff700 1 -- 172.21.15.60:0/1033082656 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f756c018020 con 0x7f7584066b20 2022-06-29T16:06:02.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.921+0000 7f757bfff700 1 --2- 172.21.15.60:0/1033082656 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7568053a70 0x7f7568055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:02.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.921+0000 7f757bfff700 1 -- 172.21.15.60:0/1033082656 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f756c06c020 con 0x7f7584066b20 2022-06-29T16:06:02.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.921+0000 7f7582ffd700 1 --2- 172.21.15.60:0/1033082656 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7568053a70 0x7f7568055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:02.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.921+0000 7f7582ffd700 1 --2- 172.21.15.60:0/1033082656 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7568053a70 0x7f7568055f30 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7f757400d000 tx=0x7f7574007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:02.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:02.929+0000 7f757bfff700 1 -- 172.21.15.60:0/1033082656 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f75840b1cb0 con 0x7f7584066b20 2022-06-29T16:06:03.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.061+0000 7f7583fff700 1 -- 172.21.15.60:0/1033082656 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f75840b1cb0 con 0x7f7584066b20 2022-06-29T16:06:03.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.061+0000 7f757bfff700 1 -- 172.21.15.60:0/1033082656 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v96) v1 ==== 92+0+10 (secure 0 0 0) 0x7f75840b1cb0 con 0x7f7584066b20 2022-06-29T16:06:03.066 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:06:03.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.065+0000 7f7583fff700 1 -- 172.21.15.60:0/1033082656 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7568053a70 msgr2=0x7f7568055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:03.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.065+0000 7f7583fff700 1 --2- 172.21.15.60:0/1033082656 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7568053a70 0x7f7568055f30 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7f757400d000 tx=0x7f7574007a00).stop 2022-06-29T16:06:03.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.065+0000 7f7583fff700 1 -- 172.21.15.60:0/1033082656 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7584066b20 msgr2=0x7f7584151df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:03.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.065+0000 7f7583fff700 1 --2- 172.21.15.60:0/1033082656 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7584066b20 0x7f7584151df0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 rx=0x7f756c00c000 tx=0x7f756c0091d0).stop 2022-06-29T16:06:03.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.065+0000 7f7583fff700 1 -- 172.21.15.60:0/1033082656 shutdown_connections 2022-06-29T16:06:03.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.065+0000 7f7583fff700 1 --2- 172.21.15.60:0/1033082656 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7568053a70 0x7f7568055f30 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:03.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.065+0000 7f7583fff700 1 --2- 172.21.15.60:0/1033082656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f758412bde0 0x7f75841522e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:03.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.065+0000 7f7583fff700 1 --2- 172.21.15.60:0/1033082656 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7584066b20 0x7f7584151df0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:03.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.065+0000 7f7583fff700 1 --2- 172.21.15.60:0/1033082656 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f758405d070 0x7f758412dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:03.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.065+0000 7f7583fff700 1 -- 172.21.15.60:0/1033082656 >> 172.21.15.60:0/1033082656 conn(0x7f75840b5890 msgr2=0x7f7584145e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:03.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.065+0000 7f7583fff700 1 -- 172.21.15.60:0/1033082656 shutdown_connections 2022-06-29T16:06:03.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.065+0000 7f7583fff700 1 -- 172.21.15.60:0/1033082656 wait complete. 2022-06-29T16:06:03.085 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-06-29T16:06:03.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.181+0000 7f39b4c3f700 1 Processor -- start 2022-06-29T16:06:03.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.181+0000 7f39b4c3f700 1 -- start start 2022-06-29T16:06:03.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.181+0000 7f39b4c3f700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f39b005d070 0x7f39b014aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:03.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.181+0000 7f39b4c3f700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f39b00b2130 0x7f39b00aca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:03.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.181+0000 7f39b4c3f700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b005d450 con 0x7f39b014af80 2022-06-29T16:06:03.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.181+0000 7f39b4c3f700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39b00adf90 con 0x7f39b00b2130 2022-06-29T16:06:03.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.181+0000 7f39b4c3f700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39b00b25d0 con 0x7f39b005d070 2022-06-29T16:06:03.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.181+0000 7f39aeffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f39b00b2130 0x7f39b00aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:03.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.181+0000 7f39aeffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f39b00b2130 0x7f39b00aca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51298/0 (socket says 172.21.15.60:51298) 2022-06-29T16:06:03.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.181+0000 7f39aeffd700 1 -- 172.21.15.60:0/3270805745 learned_addr learned my addr 172.21.15.60:0/3270805745 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:03.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.181+0000 7f39af7fe700 1 --2- 172.21.15.60:0/3270805745 >> v2:172.21.15.92:3300/0 conn(0x7f39b005d070 0x7f39b014aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:03.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.181+0000 7f39aeffd700 1 -- 172.21.15.60:0/3270805745 >> v2:172.21.15.92:3300/0 conn(0x7f39b005d070 msgr2=0x7f39b014aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:03.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.181+0000 7f39aeffd700 1 --2- 172.21.15.60:0/3270805745 >> v2:172.21.15.92:3300/0 conn(0x7f39b005d070 0x7f39b014aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:03.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.181+0000 7f39aeffd700 1 -- 172.21.15.60:0/3270805745 >> v1:172.21.15.60:6789/0 conn(0x7f39b014af80 legacy=0x7f39b00acf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:03.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.181+0000 7f39aeffd700 1 -- 172.21.15.60:0/3270805745 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b014a350 con 0x7f39b00b2130 2022-06-29T16:06:03.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.181+0000 7f39af7fe700 1 --2- 172.21.15.60:0/3270805745 >> v2:172.21.15.92:3300/0 conn(0x7f39b005d070 0x7f39b014aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:06:03.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.181+0000 7f39aeffd700 1 --2- 172.21.15.60:0/3270805745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39b00b2130 0x7f39b00aca20 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0x7f39a80032f0 tx=0x7f39a8009290).ready entity=mon.0 client_cookie=6006a8f8aa65ac13 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:03.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.181+0000 7f39ae7fc700 1 -- 172.21.15.60:0/3270805745 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f39a800b870 con 0x7f39b00b2130 2022-06-29T16:06:03.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.181+0000 7f39ae7fc700 1 -- 172.21.15.60:0/3270805745 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a800b9d0 con 0x7f39b00b2130 2022-06-29T16:06:03.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.181+0000 7f39ae7fc700 1 -- 172.21.15.60:0/3270805745 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f39a800bb40 con 0x7f39b00b2130 2022-06-29T16:06:03.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.181+0000 7f39b4c3f700 1 -- 172.21.15.60:0/3270805745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39b00b2130 msgr2=0x7f39b00aca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:03.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.181+0000 7f39b4c3f700 1 --2- 172.21.15.60:0/3270805745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39b00b2130 0x7f39b00aca20 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0x7f39a80032f0 tx=0x7f39a8009290).stop 2022-06-29T16:06:03.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.181+0000 7f39b4c3f700 1 -- 172.21.15.60:0/3270805745 shutdown_connections 2022-06-29T16:06:03.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.181+0000 7f39b4c3f700 1 --2- 172.21.15.60:0/3270805745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39b00b2130 0x7f39b00aca20 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:03.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.181+0000 7f39b4c3f700 1 --2- 172.21.15.60:0/3270805745 >> v2:172.21.15.92:3300/0 conn(0x7f39b005d070 0x7f39b014aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:03.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.181+0000 7f39b4c3f700 1 -- 172.21.15.60:0/3270805745 >> 172.21.15.60:0/3270805745 conn(0x7f39b00b5890 msgr2=0x7f39b00b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:03.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.181+0000 7f39b4c3f700 1 -- 172.21.15.60:0/3270805745 shutdown_connections 2022-06-29T16:06:03.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.181+0000 7f39b4c3f700 1 -- 172.21.15.60:0/3270805745 wait complete. 2022-06-29T16:06:03.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.185+0000 7f39b4c3f700 1 Processor -- start 2022-06-29T16:06:03.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.185+0000 7f39b4c3f700 1 -- start start 2022-06-29T16:06:03.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.185+0000 7f39b4c3f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f39b005d070 0x7f39b0066560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:03.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.185+0000 7f39b4c3f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39b00b2130 0x7f39b00b40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:03.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.185+0000 7f39b4c3f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39b014af80 0x7f39b00b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:03.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.185+0000 7f39b4c3f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b0065a50 con 0x7f39b014af80 2022-06-29T16:06:03.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.185+0000 7f39b4c3f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b0119020 con 0x7f39b005d070 2022-06-29T16:06:03.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.185+0000 7f39b4c3f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b0118d80 con 0x7f39b00b2130 2022-06-29T16:06:03.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.185+0000 7f39affff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39b014af80 0x7f39b00b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:03.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.189+0000 7f39aeffd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39b00b2130 0x7f39b00b40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:03.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.189+0000 7f39affff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39b014af80 0x7f39b00b45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51308/0 (socket says 172.21.15.60:51308) 2022-06-29T16:06:03.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.189+0000 7f39affff700 1 -- 172.21.15.60:0/2597630972 learned_addr learned my addr 172.21.15.60:0/2597630972 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:03.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.189+0000 7f39af7fe700 1 --2- 172.21.15.60:0/2597630972 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f39b005d070 0x7f39b0066560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:03.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.189+0000 7f39affff700 1 -- 172.21.15.60:0/2597630972 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39b00b2130 msgr2=0x7f39b00b40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:03.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.189+0000 7f39affff700 1 --2- 172.21.15.60:0/2597630972 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39b00b2130 0x7f39b00b40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:03.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.189+0000 7f39affff700 1 -- 172.21.15.60:0/2597630972 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f39b005d070 msgr2=0x7f39b0066560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:03.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.189+0000 7f39affff700 1 --2- 172.21.15.60:0/2597630972 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f39b005d070 0x7f39b0066560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:03.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.189+0000 7f39affff700 1 -- 172.21.15.60:0/2597630972 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a8009530 con 0x7f39b014af80 2022-06-29T16:06:03.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.189+0000 7f39affff700 1 --2- 172.21.15.60:0/2597630972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39b014af80 0x7f39b00b45d0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 rx=0x7f39b01167b0 tx=0x7f39a0007ba0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:03.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.189+0000 7f39acff9700 1 -- 172.21.15.60:0/2597630972 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f39a000ccc0 con 0x7f39b014af80 2022-06-29T16:06:03.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.189+0000 7f39b4c3f700 1 -- 172.21.15.60:0/2597630972 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b0116890 con 0x7f39b014af80 2022-06-29T16:06:03.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.189+0000 7f39acff9700 1 -- 172.21.15.60:0/2597630972 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a000ce20 con 0x7f39b014af80 2022-06-29T16:06:03.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.189+0000 7f39acff9700 1 -- 172.21.15.60:0/2597630972 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f39a00125f0 con 0x7f39b014af80 2022-06-29T16:06:03.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.189+0000 7f39b4c3f700 1 -- 172.21.15.60:0/2597630972 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b0159860 con 0x7f39b014af80 2022-06-29T16:06:03.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.189+0000 7f39b4c3f700 1 -- 172.21.15.60:0/2597630972 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b014a5f0 con 0x7f39b014af80 2022-06-29T16:06:03.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.189+0000 7f39acff9700 1 -- 172.21.15.60:0/2597630972 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f39a0019030 con 0x7f39b014af80 2022-06-29T16:06:03.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.189+0000 7f39acff9700 1 --2- 172.21.15.60:0/2597630972 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3994053b00 0x7f3994055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:03.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.189+0000 7f39acff9700 1 -- 172.21.15.60:0/2597630972 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f39a006ade0 con 0x7f39b014af80 2022-06-29T16:06:03.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.189+0000 7f39af7fe700 1 --2- 172.21.15.60:0/2597630972 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3994053b00 0x7f3994055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:03.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.193+0000 7f39af7fe700 1 --2- 172.21.15.60:0/2597630972 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3994053b00 0x7f3994055fc0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7f399800d000 tx=0x7f3998008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:03.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.197+0000 7f39acff9700 1 -- 172.21.15.60:0/2597630972 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f39a0041f90 con 0x7f39b014af80 2022-06-29T16:06:03.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.325+0000 7f39b4c3f700 1 -- 172.21.15.60:0/2597630972 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f39b00573d0 con 0x7f39b014af80 2022-06-29T16:06:03.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.325+0000 7f39acff9700 1 -- 172.21.15.60:0/2597630972 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v96) v1 ==== 84+0+143 (secure 0 0 0) 0x7f39a00455b0 con 0x7f39b014af80 2022-06-29T16:06:03.330 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:03.330 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2022-06-29T16:06:03.331 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:03.331 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2022-06-29T16:06:03.331 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": "device_health_metrics" 2022-06-29T16:06:03.332 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:03.332 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:03.333 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 2, 2022-06-29T16:06:03.333 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": "rbd" 2022-06-29T16:06:03.333 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:03.333 INFO:teuthology.orchestra.run.smithi060.stdout:] 2022-06-29T16:06:03.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.329+0000 7f39b4c3f700 1 -- 172.21.15.60:0/2597630972 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3994053b00 msgr2=0x7f3994055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:03.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.329+0000 7f39b4c3f700 1 --2- 172.21.15.60:0/2597630972 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3994053b00 0x7f3994055fc0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7f399800d000 tx=0x7f3998008040).stop 2022-06-29T16:06:03.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.329+0000 7f39b4c3f700 1 -- 172.21.15.60:0/2597630972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39b014af80 msgr2=0x7f39b00b45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:03.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.329+0000 7f39b4c3f700 1 --2- 172.21.15.60:0/2597630972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39b014af80 0x7f39b00b45d0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 rx=0x7f39b01167b0 tx=0x7f39a0007ba0).stop 2022-06-29T16:06:03.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.329+0000 7f39b4c3f700 1 -- 172.21.15.60:0/2597630972 shutdown_connections 2022-06-29T16:06:03.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.329+0000 7f39b4c3f700 1 --2- 172.21.15.60:0/2597630972 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3994053b00 0x7f3994055fc0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:03.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.329+0000 7f39b4c3f700 1 --2- 172.21.15.60:0/2597630972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39b014af80 0x7f39b00b45d0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:03.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.329+0000 7f39b4c3f700 1 --2- 172.21.15.60:0/2597630972 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f39b00b2130 0x7f39b00b40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:03.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.329+0000 7f39b4c3f700 1 --2- 172.21.15.60:0/2597630972 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f39b005d070 0x7f39b0066560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:03.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.329+0000 7f39b4c3f700 1 -- 172.21.15.60:0/2597630972 >> 172.21.15.60:0/2597630972 conn(0x7f39b00b5890 msgr2=0x7f39b0143390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:03.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.329+0000 7f39b4c3f700 1 -- 172.21.15.60:0/2597630972 shutdown_connections 2022-06-29T16:06:03.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.329+0000 7f39b4c3f700 1 -- 172.21.15.60:0/2597630972 wait complete. 2022-06-29T16:06:03.351 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-06-29T16:06:03.351 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-06-29T16:06:03.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.449+0000 7f76f61d7700 1 Processor -- start 2022-06-29T16:06:03.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.453+0000 7f76f61d7700 1 -- start start 2022-06-29T16:06:03.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.453+0000 7f76f61d7700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f76f00b5eb0 0x7f76f0142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:03.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.453+0000 7f76f61d7700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76f00b0cf0 con 0x7f76f00b2c50 2022-06-29T16:06:03.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.453+0000 7f76f61d7700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76f00b4d10 con 0x7f76f00b48b0 2022-06-29T16:06:03.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.453+0000 7f76f61d7700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76f00b54e0 con 0x7f76f00b5eb0 2022-06-29T16:06:03.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.453+0000 7f76f59d6700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f76f00b2c50 0x7f76f014d380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59402/0 (socket says 172.21.15.60:59402) 2022-06-29T16:06:03.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.453+0000 7f76f59d6700 1 -- 172.21.15.60:0/2135393866 learned_addr learned my addr 172.21.15.60:0/2135393866 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:03.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.453+0000 7f76f51d5700 1 --2- 172.21.15.60:0/2135393866 >> v2:172.21.15.92:3300/0 conn(0x7f76f00b5eb0 0x7f76f0142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:03.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.453+0000 7f76dffff700 1 -- 172.21.15.60:0/2135393866 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4000253484 0 0) 0x7f76e8001490 con 0x7f76f00b2c50 2022-06-29T16:06:03.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.453+0000 7f76dffff700 1 -- 172.21.15.60:0/2135393866 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76d8003260 con 0x7f76f00b2c50 2022-06-29T16:06:03.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.453+0000 7f76dffff700 1 -- 172.21.15.60:0/2135393866 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3873027851 0 0) 0x7f76f00b4d10 con 0x7f76f00b48b0 2022-06-29T16:06:03.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.453+0000 7f76dffff700 1 -- 172.21.15.60:0/2135393866 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76e8001490 con 0x7f76f00b48b0 2022-06-29T16:06:03.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.453+0000 7f76f51d5700 1 -- 172.21.15.60:0/2135393866 >> v1:172.21.15.92:6789/0 conn(0x7f76f00b48b0 legacy=0x7f76f00b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:03.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.453+0000 7f76dffff700 1 -- 172.21.15.60:0/2135393866 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3784285783 0 0) 0x7f76d8003260 con 0x7f76f00b2c50 2022-06-29T16:06:03.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.453+0000 7f76f51d5700 1 -- 172.21.15.60:0/2135393866 >> v1:172.21.15.60:6789/0 conn(0x7f76f00b2c50 legacy=0x7f76f014d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:03.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.453+0000 7f76f51d5700 1 -- 172.21.15.60:0/2135393866 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f00b6290 con 0x7f76f00b5eb0 2022-06-29T16:06:03.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.453+0000 7f76f51d5700 1 --2- 172.21.15.60:0/2135393866 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76f00b5eb0 0x7f76f0142a20 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f76e8001620 tx=0x7f76ec006df0).ready entity=mon.1 client_cookie=bb42fdf4e9549518 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:03.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.453+0000 7f76dffff700 1 -- 172.21.15.60:0/2135393866 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76e8001b70 con 0x7f76f00b5eb0 2022-06-29T16:06:03.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.453+0000 7f76dffff700 1 -- 172.21.15.60:0/2135393866 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76ec000e90 con 0x7f76f00b5eb0 2022-06-29T16:06:03.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.453+0000 7f76dffff700 1 -- 172.21.15.60:0/2135393866 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76ec012460 con 0x7f76f00b5eb0 2022-06-29T16:06:03.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.457+0000 7f76f61d7700 1 -- 172.21.15.60:0/2135393866 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76f00b5eb0 msgr2=0x7f76f0142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:03.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.457+0000 7f76f61d7700 1 --2- 172.21.15.60:0/2135393866 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76f00b5eb0 0x7f76f0142a20 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f76e8001620 tx=0x7f76ec006df0).stop 2022-06-29T16:06:03.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.457+0000 7f76f61d7700 1 -- 172.21.15.60:0/2135393866 shutdown_connections 2022-06-29T16:06:03.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.457+0000 7f76f61d7700 1 --2- 172.21.15.60:0/2135393866 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76f00b5eb0 0x7f76f0142a20 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:03.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.457+0000 7f76f61d7700 1 -- 172.21.15.60:0/2135393866 >> 172.21.15.60:0/2135393866 conn(0x7f76f0062090 msgr2=0x7f76f0062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:03.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.457+0000 7f76f61d7700 1 -- 172.21.15.60:0/2135393866 shutdown_connections 2022-06-29T16:06:03.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.457+0000 7f76f61d7700 1 -- 172.21.15.60:0/2135393866 wait complete. 2022-06-29T16:06:03.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.457+0000 7f76f61d7700 1 Processor -- start 2022-06-29T16:06:03.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.457+0000 7f76f61d7700 1 -- start start 2022-06-29T16:06:03.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.457+0000 7f76f61d7700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76f00b2c50 0x7f76f0147a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:03.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.457+0000 7f76f61d7700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76f00b48b0 0x7f76f0155920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:03.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.457+0000 7f76f61d7700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76f00b5eb0 0x7f76f00aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:03.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.457+0000 7f76f61d7700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f013cd00 con 0x7f76f00b5eb0 2022-06-29T16:06:03.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.457+0000 7f76f61d7700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f01551e0 con 0x7f76f00b48b0 2022-06-29T16:06:03.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.457+0000 7f76f61d7700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76f0155480 con 0x7f76f00b2c50 2022-06-29T16:06:03.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.457+0000 7f76f59d6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76f00b5eb0 0x7f76f00aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:03.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.457+0000 7f76f59d6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76f00b5eb0 0x7f76f00aca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51320/0 (socket says 172.21.15.60:51320) 2022-06-29T16:06:03.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.457+0000 7f76f59d6700 1 -- 172.21.15.60:0/3861492947 learned_addr learned my addr 172.21.15.60:0/3861492947 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:03.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.461+0000 7f76f51d5700 1 --2- 172.21.15.60:0/3861492947 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76f00b2c50 0x7f76f0147a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:03.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.461+0000 7f76f49d4700 1 --2- 172.21.15.60:0/3861492947 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76f00b48b0 0x7f76f0155920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:03.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.461+0000 7f76f59d6700 1 -- 172.21.15.60:0/3861492947 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76f00b2c50 msgr2=0x7f76f0147a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:03.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.461+0000 7f76f59d6700 1 --2- 172.21.15.60:0/3861492947 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76f00b2c50 0x7f76f0147a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:03.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.461+0000 7f76f59d6700 1 -- 172.21.15.60:0/3861492947 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76f00b48b0 msgr2=0x7f76f0155920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:03.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.461+0000 7f76f59d6700 1 --2- 172.21.15.60:0/3861492947 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76f00b48b0 0x7f76f0155920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:03.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.461+0000 7f76f59d6700 1 -- 172.21.15.60:0/3861492947 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ec00b180 con 0x7f76f00b5eb0 2022-06-29T16:06:03.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.461+0000 7f76f49d4700 1 --2- 172.21.15.60:0/3861492947 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76f00b48b0 0x7f76f0155920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:06:03.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.461+0000 7f76f59d6700 1 --2- 172.21.15.60:0/3861492947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76f00b5eb0 0x7f76f00aca20 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 rx=0x7f76e8001d50 tx=0x7f76e80017d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:03.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.461+0000 7f76de7fc700 1 -- 172.21.15.60:0/3861492947 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76e8004ab0 con 0x7f76f00b5eb0 2022-06-29T16:06:03.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.461+0000 7f76de7fc700 1 -- 172.21.15.60:0/3861492947 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76e8004c10 con 0x7f76f00b5eb0 2022-06-29T16:06:03.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.461+0000 7f76f61d7700 1 -- 172.21.15.60:0/3861492947 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76f013c9f0 con 0x7f76f00b5eb0 2022-06-29T16:06:03.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.461+0000 7f76de7fc700 1 -- 172.21.15.60:0/3861492947 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76e8007170 con 0x7f76f00b5eb0 2022-06-29T16:06:03.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.461+0000 7f76f61d7700 1 -- 172.21.15.60:0/3861492947 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76f0141300 con 0x7f76f00b5eb0 2022-06-29T16:06:03.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.461+0000 7f76f61d7700 1 -- 172.21.15.60:0/3861492947 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76f0142f10 con 0x7f76f00b5eb0 2022-06-29T16:06:03.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.461+0000 7f76de7fc700 1 -- 172.21.15.60:0/3861492947 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f76e8010070 con 0x7f76f00b5eb0 2022-06-29T16:06:03.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.461+0000 7f76de7fc700 1 --2- 172.21.15.60:0/3861492947 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f76d0053b00 0x7f76d0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:03.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.461+0000 7f76de7fc700 1 -- 172.21.15.60:0/3861492947 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f76e806be30 con 0x7f76f00b5eb0 2022-06-29T16:06:03.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.461+0000 7f76f51d5700 1 --2- 172.21.15.60:0/3861492947 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f76d0053b00 0x7f76d0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:03.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.465+0000 7f76f51d5700 1 --2- 172.21.15.60:0/3861492947 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f76d0053b00 0x7f76d0055fc0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f76ec01a000 tx=0x7f76ec014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:03.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.469+0000 7f76de7fc700 1 -- 172.21.15.60:0/3861492947 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f76e8043080 con 0x7f76f00b5eb0 2022-06-29T16:06:03.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:03.593+0000 7f76f61d7700 1 -- 172.21.15.60:0/3861492947 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f76f00b0cf0 con 0x7f76f00b5eb0 2022-06-29T16:06:04.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.217+0000 7f76de7fc700 1 -- 172.21.15.60:0/3861492947 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v97) v1 ==== 133+0+0 (secure 0 0 0) 0x7f76f00b0cf0 con 0x7f76f00b5eb0 2022-06-29T16:06:04.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.221+0000 7f76f61d7700 1 -- 172.21.15.60:0/3861492947 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f76d0053b00 msgr2=0x7f76d0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:04.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.221+0000 7f76f61d7700 1 --2- 172.21.15.60:0/3861492947 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f76d0053b00 0x7f76d0055fc0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f76ec01a000 tx=0x7f76ec014000).stop 2022-06-29T16:06:04.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.221+0000 7f76f61d7700 1 -- 172.21.15.60:0/3861492947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76f00b5eb0 msgr2=0x7f76f00aca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:04.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.221+0000 7f76f61d7700 1 --2- 172.21.15.60:0/3861492947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76f00b5eb0 0x7f76f00aca20 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 rx=0x7f76e8001d50 tx=0x7f76e80017d0).stop 2022-06-29T16:06:04.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.221+0000 7f76f61d7700 1 -- 172.21.15.60:0/3861492947 shutdown_connections 2022-06-29T16:06:04.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.221+0000 7f76f61d7700 1 --2- 172.21.15.60:0/3861492947 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f76d0053b00 0x7f76d0055fc0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:04.230 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.221+0000 7f76f61d7700 1 --2- 172.21.15.60:0/3861492947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76f00b5eb0 0x7f76f00aca20 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:04.230 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.221+0000 7f76f61d7700 1 --2- 172.21.15.60:0/3861492947 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76f00b48b0 0x7f76f0155920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:04.230 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.221+0000 7f76f61d7700 1 --2- 172.21.15.60:0/3861492947 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76f00b2c50 0x7f76f0147a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:04.230 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.221+0000 7f76f61d7700 1 -- 172.21.15.60:0/3861492947 >> 172.21.15.60:0/3861492947 conn(0x7f76f0062090 msgr2=0x7f76f00b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:04.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.225+0000 7f76f61d7700 1 -- 172.21.15.60:0/3861492947 shutdown_connections 2022-06-29T16:06:04.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.225+0000 7f76f61d7700 1 -- 172.21.15.60:0/3861492947 wait complete. 2022-06-29T16:06:04.231 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2022-06-29T16:06:04.243 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-06-29T16:06:04.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.345+0000 7f4a330d0700 1 Processor -- start 2022-06-29T16:06:04.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.345+0000 7f4a330d0700 1 -- start start 2022-06-29T16:06:04.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.349+0000 7f4a330d0700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4a2c143910 0x7f4a2c0ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:04.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.349+0000 7f4a330d0700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4a2c0ad620 0x7f4a2c13fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:04.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.349+0000 7f4a330d0700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a2c05fee0 con 0x7f4a2c05fb00 2022-06-29T16:06:04.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.349+0000 7f4a330d0700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a2c0b1f00 con 0x7f4a2c0ad620 2022-06-29T16:06:04.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.349+0000 7f4a330d0700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c0af1d0 con 0x7f4a2c143910 2022-06-29T16:06:04.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.349+0000 7f4a328cf700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4a2c0ad620 0x7f4a2c13fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:04.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.349+0000 7f4a328cf700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4a2c0ad620 0x7f4a2c13fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51328/0 (socket says 172.21.15.60:51328) 2022-06-29T16:06:04.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.349+0000 7f4a328cf700 1 -- 172.21.15.60:0/510936001 learned_addr learned my addr 172.21.15.60:0/510936001 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:04.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.349+0000 7f4a320ce700 1 --1- 172.21.15.60:0/510936001 >> v1:172.21.15.60:6789/0 conn(0x7f4a2c05fb00 0x7f4a2c0aca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59412/0 (socket says 172.21.15.60:59412) 2022-06-29T16:06:04.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.349+0000 7f4a318cd700 1 --2- 172.21.15.60:0/510936001 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4a2c143910 0x7f4a2c0ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:04.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.349+0000 7f4a328cf700 1 -- 172.21.15.60:0/510936001 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4a2c143910 msgr2=0x7f4a2c0ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:04.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.349+0000 7f4a328cf700 1 --2- 172.21.15.60:0/510936001 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4a2c143910 0x7f4a2c0ad130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:04.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.349+0000 7f4a328cf700 1 -- 172.21.15.60:0/510936001 >> v1:172.21.15.60:6789/0 conn(0x7f4a2c05fb00 legacy=0x7f4a2c0aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:04.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.349+0000 7f4a328cf700 1 -- 172.21.15.60:0/510936001 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2c0af470 con 0x7f4a2c0ad620 2022-06-29T16:06:04.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.349+0000 7f4a328cf700 1 --2- 172.21.15.60:0/510936001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a2c0ad620 0x7f4a2c13fe00 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 rx=0x7f4a240032f0 tx=0x7f4a24009290).ready entity=mon.0 client_cookie=5ddb7d0a7271b70 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:04.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.349+0000 7f4a310cc700 1 -- 172.21.15.60:0/510936001 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a2400ba90 con 0x7f4a2c0ad620 2022-06-29T16:06:04.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.349+0000 7f4a310cc700 1 -- 172.21.15.60:0/510936001 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a2400bbf0 con 0x7f4a2c0ad620 2022-06-29T16:06:04.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.349+0000 7f4a310cc700 1 -- 172.21.15.60:0/510936001 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a24010470 con 0x7f4a2c0ad620 2022-06-29T16:06:04.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.349+0000 7f4a330d0700 1 -- 172.21.15.60:0/510936001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a2c0ad620 msgr2=0x7f4a2c13fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:04.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.349+0000 7f4a330d0700 1 --2- 172.21.15.60:0/510936001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a2c0ad620 0x7f4a2c13fe00 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 rx=0x7f4a240032f0 tx=0x7f4a24009290).stop 2022-06-29T16:06:04.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.349+0000 7f4a330d0700 1 -- 172.21.15.60:0/510936001 shutdown_connections 2022-06-29T16:06:04.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.349+0000 7f4a330d0700 1 --2- 172.21.15.60:0/510936001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a2c0ad620 0x7f4a2c13fe00 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:04.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.349+0000 7f4a330d0700 1 --2- 172.21.15.60:0/510936001 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4a2c143910 0x7f4a2c0ad130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:04.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.349+0000 7f4a330d0700 1 -- 172.21.15.60:0/510936001 >> 172.21.15.60:0/510936001 conn(0x7f4a2c063a60 msgr2=0x7f4a2c05f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:04.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.349+0000 7f4a330d0700 1 -- 172.21.15.60:0/510936001 shutdown_connections 2022-06-29T16:06:04.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.349+0000 7f4a330d0700 1 -- 172.21.15.60:0/510936001 wait complete. 2022-06-29T16:06:04.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.353+0000 7f4a330d0700 1 Processor -- start 2022-06-29T16:06:04.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.353+0000 7f4a330d0700 1 -- start start 2022-06-29T16:06:04.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.353+0000 7f4a330d0700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a2c05fb00 0x7f4a2c113210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:04.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.353+0000 7f4a330d0700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4a2c0ad620 0x7f4a2c1141f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:04.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.353+0000 7f4a330d0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a2c143910 0x7f4a2c1146e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:04.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.353+0000 7f4a330d0700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c1520d0 con 0x7f4a2c143910 2022-06-29T16:06:04.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.353+0000 7f4a330d0700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c152370 con 0x7f4a2c05fb00 2022-06-29T16:06:04.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.353+0000 7f4a330d0700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c113700 con 0x7f4a2c0ad620 2022-06-29T16:06:04.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.353+0000 7f4a328cf700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a2c143910 0x7f4a2c1146e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:04.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.353+0000 7f4a318cd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4a2c0ad620 0x7f4a2c1141f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:04.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.353+0000 7f4a328cf700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a2c143910 0x7f4a2c1146e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51336/0 (socket says 172.21.15.60:51336) 2022-06-29T16:06:04.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.353+0000 7f4a328cf700 1 -- 172.21.15.60:0/900910111 learned_addr learned my addr 172.21.15.60:0/900910111 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:04.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.353+0000 7f4a320ce700 1 --2- 172.21.15.60:0/900910111 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a2c05fb00 0x7f4a2c113210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:04.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.353+0000 7f4a318cd700 1 -- 172.21.15.60:0/900910111 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a2c05fb00 msgr2=0x7f4a2c113210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:04.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.353+0000 7f4a318cd700 1 --2- 172.21.15.60:0/900910111 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a2c05fb00 0x7f4a2c113210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:04.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.353+0000 7f4a318cd700 1 -- 172.21.15.60:0/900910111 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a2c143910 msgr2=0x7f4a2c1146e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:04.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.353+0000 7f4a318cd700 1 --2- 172.21.15.60:0/900910111 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a2c143910 0x7f4a2c1146e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:04.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.353+0000 7f4a318cd700 1 -- 172.21.15.60:0/900910111 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a24009530 con 0x7f4a2c0ad620 2022-06-29T16:06:04.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.353+0000 7f4a328cf700 1 --2- 172.21.15.60:0/900910111 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a2c143910 0x7f4a2c1146e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:06:04.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.357+0000 7f4a318cd700 1 --2- 172.21.15.60:0/900910111 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4a2c0ad620 0x7f4a2c1141f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f4a1c00c000 tx=0x7f4a1c006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:04.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.357+0000 7f4a1b7fe700 1 -- 172.21.15.60:0/900910111 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a1c011750 con 0x7f4a2c0ad620 2022-06-29T16:06:04.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.357+0000 7f4a1b7fe700 1 -- 172.21.15.60:0/900910111 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a1c0118b0 con 0x7f4a2c0ad620 2022-06-29T16:06:04.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.357+0000 7f4a330d0700 1 -- 172.21.15.60:0/900910111 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a2c151cb0 con 0x7f4a2c0ad620 2022-06-29T16:06:04.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.357+0000 7f4a1b7fe700 1 -- 172.21.15.60:0/900910111 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a1c011c10 con 0x7f4a2c0ad620 2022-06-29T16:06:04.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.357+0000 7f4a330d0700 1 -- 172.21.15.60:0/900910111 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a2c152c50 con 0x7f4a2c0ad620 2022-06-29T16:06:04.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.357+0000 7f4a1b7fe700 1 -- 172.21.15.60:0/900910111 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4a1c011d70 con 0x7f4a2c0ad620 2022-06-29T16:06:04.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.357+0000 7f4a330d0700 1 -- 172.21.15.60:0/900910111 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a2c0573d0 con 0x7f4a2c0ad620 2022-06-29T16:06:04.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.357+0000 7f4a1b7fe700 1 --2- 172.21.15.60:0/900910111 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4a0c053aa0 0x7f4a0c055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:04.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.357+0000 7f4a1b7fe700 1 -- 172.21.15.60:0/900910111 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f4a1c06bf60 con 0x7f4a2c0ad620 2022-06-29T16:06:04.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.357+0000 7f4a320ce700 1 --2- 172.21.15.60:0/900910111 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4a0c053aa0 0x7f4a0c055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:04.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.361+0000 7f4a320ce700 1 --2- 172.21.15.60:0/900910111 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4a0c053aa0 0x7f4a0c055f60 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f4a2800d000 tx=0x7f4a28007b40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:04.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.369+0000 7f4a1b7fe700 1 -- 172.21.15.60:0/900910111 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4a1c042ff0 con 0x7f4a2c0ad620 2022-06-29T16:06:04.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.497+0000 7f4a330d0700 1 -- 172.21.15.60:0/900910111 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f4a2c1526f0 con 0x7f4a2c0ad620 2022-06-29T16:06:04.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.909+0000 7f4a1b7fe700 1 -- 172.21.15.60:0/900910111 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v98) v1 ==== 125+0+0 (secure 0 0 0) 0x7f4a1c046610 con 0x7f4a2c0ad620 2022-06-29T16:06:04.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.917+0000 7f4a330d0700 1 -- 172.21.15.60:0/900910111 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4a0c053aa0 msgr2=0x7f4a0c055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:04.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.917+0000 7f4a330d0700 1 --2- 172.21.15.60:0/900910111 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4a0c053aa0 0x7f4a0c055f60 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f4a2800d000 tx=0x7f4a28007b40).stop 2022-06-29T16:06:04.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.917+0000 7f4a330d0700 1 -- 172.21.15.60:0/900910111 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4a2c0ad620 msgr2=0x7f4a2c1141f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:04.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.917+0000 7f4a330d0700 1 --2- 172.21.15.60:0/900910111 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4a2c0ad620 0x7f4a2c1141f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f4a1c00c000 tx=0x7f4a1c006df0).stop 2022-06-29T16:06:04.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.917+0000 7f4a330d0700 1 -- 172.21.15.60:0/900910111 shutdown_connections 2022-06-29T16:06:04.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.917+0000 7f4a330d0700 1 --2- 172.21.15.60:0/900910111 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4a0c053aa0 0x7f4a0c055f60 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:04.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.917+0000 7f4a330d0700 1 --2- 172.21.15.60:0/900910111 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4a2c143910 0x7f4a2c1146e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:04.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.917+0000 7f4a330d0700 1 --2- 172.21.15.60:0/900910111 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4a2c0ad620 0x7f4a2c1141f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:04.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.917+0000 7f4a330d0700 1 --2- 172.21.15.60:0/900910111 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a2c05fb00 0x7f4a2c113210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:04.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.917+0000 7f4a330d0700 1 -- 172.21.15.60:0/900910111 >> 172.21.15.60:0/900910111 conn(0x7f4a2c063a60 msgr2=0x7f4a2c0b28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:04.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.917+0000 7f4a330d0700 1 -- 172.21.15.60:0/900910111 shutdown_connections 2022-06-29T16:06:04.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:04.917+0000 7f4a330d0700 1 -- 172.21.15.60:0/900910111 wait complete. 2022-06-29T16:06:04.925 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2022-06-29T16:06:04.933 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-06-29T16:06:05.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.037+0000 7fb4ff59e700 1 Processor -- start 2022-06-29T16:06:05.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.037+0000 7fb4ff59e700 1 -- start start 2022-06-29T16:06:05.045 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.037+0000 7fb4ff59e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb5000afb00 0x7fb500067980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:05.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.037+0000 7fb4ff59e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb50013f6f0 0x7fb5000ad780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:05.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.037+0000 7fb4ff59e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5000b1e00 con 0x7fb50005fb60 2022-06-29T16:06:05.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.037+0000 7fb4ff59e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb50005ff40 con 0x7fb5000afb00 2022-06-29T16:06:05.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.037+0000 7fb4ff59e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5000adc70 con 0x7fb50013f6f0 2022-06-29T16:06:05.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.037+0000 7fb4fe59c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb50005fb60 0x7fb5000ad070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59426/0 (socket says 172.21.15.60:59426) 2022-06-29T16:06:05.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.037+0000 7fb4fe59c700 1 -- 172.21.15.60:0/4209261158 learned_addr learned my addr 172.21.15.60:0/4209261158 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:05.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.037+0000 7fb4fdd9b700 1 --2- 172.21.15.60:0/4209261158 >> v2:172.21.15.60:3300/0 conn(0x7fb5000afb00 0x7fb500067980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:05.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.037+0000 7fb4fed9d700 1 --2- 172.21.15.60:0/4209261158 >> v2:172.21.15.92:3300/0 conn(0x7fb50013f6f0 0x7fb5000ad780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:05.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.037+0000 7fb4fd59a700 1 -- 172.21.15.60:0/4209261158 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1618687183 0 0) 0x7fb5000b1e00 con 0x7fb50005fb60 2022-06-29T16:06:05.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.037+0000 7fb4fd59a700 1 -- 172.21.15.60:0/4209261158 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4e4003260 con 0x7fb50005fb60 2022-06-29T16:06:05.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.037+0000 7fb4fdd9b700 1 -- 172.21.15.60:0/4209261158 >> v2:172.21.15.92:3300/0 conn(0x7fb50013f6f0 msgr2=0x7fb5000ad780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:05.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.037+0000 7fb4fdd9b700 1 --2- 172.21.15.60:0/4209261158 >> v2:172.21.15.92:3300/0 conn(0x7fb50013f6f0 0x7fb5000ad780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:05.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.037+0000 7fb4fdd9b700 1 -- 172.21.15.60:0/4209261158 >> v1:172.21.15.60:6789/0 conn(0x7fb50005fb60 legacy=0x7fb5000ad070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:05.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.037+0000 7fb4fdd9b700 1 -- 172.21.15.60:0/4209261158 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5000b1b60 con 0x7fb5000afb00 2022-06-29T16:06:05.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.037+0000 7fb4fdd9b700 1 --2- 172.21.15.60:0/4209261158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5000afb00 0x7fb500067980 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 rx=0x7fb4f4002f70 tx=0x7fb4f4006df0).ready entity=mon.0 client_cookie=77c87fad394c73c4 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:05.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.037+0000 7fb4fd59a700 1 -- 172.21.15.60:0/4209261158 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4f400d4f0 con 0x7fb5000afb00 2022-06-29T16:06:05.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.041+0000 7fb4fd59a700 1 -- 172.21.15.60:0/4209261158 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4f4000e90 con 0x7fb5000afb00 2022-06-29T16:06:05.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.041+0000 7fb4fd59a700 1 -- 172.21.15.60:0/4209261158 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4f400d840 con 0x7fb5000afb00 2022-06-29T16:06:05.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.041+0000 7fb4ff59e700 1 -- 172.21.15.60:0/4209261158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5000afb00 msgr2=0x7fb500067980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:05.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.041+0000 7fb4ff59e700 1 --2- 172.21.15.60:0/4209261158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5000afb00 0x7fb500067980 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 rx=0x7fb4f4002f70 tx=0x7fb4f4006df0).stop 2022-06-29T16:06:05.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.041+0000 7fb4ff59e700 1 -- 172.21.15.60:0/4209261158 shutdown_connections 2022-06-29T16:06:05.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.041+0000 7fb4ff59e700 1 --2- 172.21.15.60:0/4209261158 >> v2:172.21.15.92:3300/0 conn(0x7fb50013f6f0 0x7fb5000ad780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:05.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.041+0000 7fb4ff59e700 1 --2- 172.21.15.60:0/4209261158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5000afb00 0x7fb500067980 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:05.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.041+0000 7fb4ff59e700 1 -- 172.21.15.60:0/4209261158 >> 172.21.15.60:0/4209261158 conn(0x7fb500063a60 msgr2=0x7fb50005f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:05.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.041+0000 7fb4ff59e700 1 -- 172.21.15.60:0/4209261158 shutdown_connections 2022-06-29T16:06:05.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.041+0000 7fb4ff59e700 1 -- 172.21.15.60:0/4209261158 wait complete. 2022-06-29T16:06:05.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.041+0000 7fb4ff59e700 1 Processor -- start 2022-06-29T16:06:05.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.041+0000 7fb4ff59e700 1 -- start start 2022-06-29T16:06:05.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.045+0000 7fb4ff59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb50005fb60 0x7fb500152db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:05.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.045+0000 7fb4ff59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5000afb00 0x7fb5000b2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:05.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.045+0000 7fb4ff59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb50013f6f0 0x7fb500145180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:05.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.045+0000 7fb4ff59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb500141b90 con 0x7fb5000afb00 2022-06-29T16:06:05.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.045+0000 7fb4ff59e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb50013cd40 con 0x7fb50013f6f0 2022-06-29T16:06:05.058 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.045+0000 7fb4ff59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb50013cfe0 con 0x7fb50005fb60 2022-06-29T16:06:05.058 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.045+0000 7fb4fdd9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5000afb00 0x7fb5000b2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:05.058 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.045+0000 7fb4fdd9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5000afb00 0x7fb5000b2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51348/0 (socket says 172.21.15.60:51348) 2022-06-29T16:06:05.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.045+0000 7fb4fdd9b700 1 -- 172.21.15.60:0/390777862 learned_addr learned my addr 172.21.15.60:0/390777862 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:05.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.045+0000 7fb4fe59c700 1 --2- 172.21.15.60:0/390777862 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb50005fb60 0x7fb500152db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:05.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.045+0000 7fb4fed9d700 1 --2- 172.21.15.60:0/390777862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb50013f6f0 0x7fb500145180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:05.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.045+0000 7fb4fdd9b700 1 -- 172.21.15.60:0/390777862 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb50005fb60 msgr2=0x7fb500152db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:05.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.045+0000 7fb4fdd9b700 1 --2- 172.21.15.60:0/390777862 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb50005fb60 0x7fb500152db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:05.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.045+0000 7fb4fdd9b700 1 -- 172.21.15.60:0/390777862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb50013f6f0 msgr2=0x7fb500145180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:05.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.045+0000 7fb4fdd9b700 1 --2- 172.21.15.60:0/390777862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb50013f6f0 0x7fb500145180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:05.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.045+0000 7fb4fdd9b700 1 -- 172.21.15.60:0/390777862 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f400b180 con 0x7fb5000afb00 2022-06-29T16:06:05.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.045+0000 7fb4fdd9b700 1 --2- 172.21.15.60:0/390777862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5000afb00 0x7fb5000b2450 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 rx=0x7fb4f4002f70 tx=0x7fb4f4014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:05.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.045+0000 7fb4ef7fe700 1 -- 172.21.15.60:0/390777862 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4f4013de0 con 0x7fb5000afb00 2022-06-29T16:06:05.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.045+0000 7fb4ef7fe700 1 -- 172.21.15.60:0/390777862 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4f400de60 con 0x7fb5000afb00 2022-06-29T16:06:05.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.045+0000 7fb4ff59e700 1 -- 172.21.15.60:0/390777862 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb500141730 con 0x7fb5000afb00 2022-06-29T16:06:05.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.045+0000 7fb4ef7fe700 1 -- 172.21.15.60:0/390777862 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4f401e5f0 con 0x7fb5000afb00 2022-06-29T16:06:05.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.045+0000 7fb4ff59e700 1 -- 172.21.15.60:0/390777862 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb50013db40 con 0x7fb5000afb00 2022-06-29T16:06:05.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.045+0000 7fb4ff59e700 1 -- 172.21.15.60:0/390777862 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb50013d1a0 con 0x7fb5000afb00 2022-06-29T16:06:05.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.049+0000 7fb4ef7fe700 1 -- 172.21.15.60:0/390777862 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb4f4025030 con 0x7fb5000afb00 2022-06-29T16:06:05.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.049+0000 7fb4ef7fe700 1 --2- 172.21.15.60:0/390777862 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb4dc053b00 0x7fb4dc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:05.063 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.049+0000 7fb4ef7fe700 1 -- 172.21.15.60:0/390777862 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5957+0+0 (secure 0 0 0) 0x7fb4f406c930 con 0x7fb5000afb00 2022-06-29T16:06:05.063 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.049+0000 7fb4fe59c700 1 --2- 172.21.15.60:0/390777862 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb4dc053b00 0x7fb4dc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:05.063 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.049+0000 7fb4fe59c700 1 --2- 172.21.15.60:0/390777862 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb4dc053b00 0x7fb4dc055fc0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7fb4f000d000 tx=0x7fb4f0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:05.064 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.057+0000 7fb4ef7fe700 1 -- 172.21.15.60:0/390777862 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb4f40438c0 con 0x7fb5000afb00 2022-06-29T16:06:05.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.221+0000 7fb4ff59e700 1 -- 172.21.15.60:0/390777862 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7fb5000573d0 con 0x7fb5000afb00 2022-06-29T16:06:05.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.233+0000 7fb4ef7fe700 1 -- 172.21.15.60:0/390777862 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 19 and data pool 20 v45) v1 ==== 189+0+0 (secure 0 0 0) 0x7fb4f4023070 con 0x7fb5000afb00 2022-06-29T16:06:05.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.237+0000 7fb4ff59e700 1 -- 172.21.15.60:0/390777862 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb4dc053b00 msgr2=0x7fb4dc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:05.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.237+0000 7fb4ff59e700 1 --2- 172.21.15.60:0/390777862 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb4dc053b00 0x7fb4dc055fc0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7fb4f000d000 tx=0x7fb4f0009040).stop 2022-06-29T16:06:05.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.237+0000 7fb4ff59e700 1 -- 172.21.15.60:0/390777862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5000afb00 msgr2=0x7fb5000b2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:05.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.237+0000 7fb4ff59e700 1 --2- 172.21.15.60:0/390777862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5000afb00 0x7fb5000b2450 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 rx=0x7fb4f4002f70 tx=0x7fb4f4014000).stop 2022-06-29T16:06:05.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.237+0000 7fb4ff59e700 1 -- 172.21.15.60:0/390777862 shutdown_connections 2022-06-29T16:06:05.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.237+0000 7fb4ff59e700 1 --2- 172.21.15.60:0/390777862 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb4dc053b00 0x7fb4dc055fc0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:05.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.237+0000 7fb4ff59e700 1 --2- 172.21.15.60:0/390777862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb50013f6f0 0x7fb500145180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:05.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.237+0000 7fb4ff59e700 1 --2- 172.21.15.60:0/390777862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5000afb00 0x7fb5000b2450 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:05.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.237+0000 7fb4ff59e700 1 --2- 172.21.15.60:0/390777862 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb50005fb60 0x7fb500152db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:05.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.237+0000 7fb4ff59e700 1 -- 172.21.15.60:0/390777862 >> 172.21.15.60:0/390777862 conn(0x7fb500063a60 msgr2=0x7fb5000b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:05.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.237+0000 7fb4ff59e700 1 -- 172.21.15.60:0/390777862 shutdown_connections 2022-06-29T16:06:05.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.237+0000 7fb4ff59e700 1 -- 172.21.15.60:0/390777862 wait complete. 2022-06-29T16:06:05.245 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 19 and data pool 20 2022-06-29T16:06:05.257 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:06:05.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.345+0000 7f6a1d8ae700 1 Processor -- start 2022-06-29T16:06:05.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.345+0000 7f6a1d8ae700 1 -- start start 2022-06-29T16:06:05.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.345+0000 7f6a1d8ae700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6a180b48b0 0x7f6a18142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:05.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.345+0000 7f6a1d8ae700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6a180b2c50 0x7f6a180b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:05.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.345+0000 7f6a1d8ae700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a180b0cf0 con 0x7f6a180b5eb0 2022-06-29T16:06:05.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.345+0000 7f6a1d8ae700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a180b54e0 con 0x7f6a180b2c50 2022-06-29T16:06:05.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.345+0000 7f6a1d8ae700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a180b4d10 con 0x7f6a180b48b0 2022-06-29T16:06:05.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.345+0000 7f6a1d0ad700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6a180b2c50 0x7f6a180b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:05.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.345+0000 7f6a1c8ac700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6a180b5eb0 0x7f6a180b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59440/0 (socket says 172.21.15.60:59440) 2022-06-29T16:06:05.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.345+0000 7f6a1d0ad700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6a180b2c50 0x7f6a180b30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51356/0 (socket says 172.21.15.60:51356) 2022-06-29T16:06:05.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.345+0000 7f6a1c8ac700 1 -- 172.21.15.60:0/2946444240 learned_addr learned my addr 172.21.15.60:0/2946444240 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:05.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.345+0000 7f6a17fff700 1 --2- 172.21.15.60:0/2946444240 >> v2:172.21.15.92:3300/0 conn(0x7f6a180b48b0 0x7f6a18142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:05.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.349+0000 7f6a1d0ad700 1 -- 172.21.15.60:0/2946444240 >> v2:172.21.15.92:3300/0 conn(0x7f6a180b48b0 msgr2=0x7f6a18142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:05.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.349+0000 7f6a1d0ad700 1 --2- 172.21.15.60:0/2946444240 >> v2:172.21.15.92:3300/0 conn(0x7f6a180b48b0 0x7f6a18142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:05.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.349+0000 7f6a1d0ad700 1 -- 172.21.15.60:0/2946444240 >> v1:172.21.15.60:6789/0 conn(0x7f6a180b5eb0 legacy=0x7f6a180b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:05.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.349+0000 7f6a1d0ad700 1 -- 172.21.15.60:0/2946444240 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a180b6290 con 0x7f6a180b2c50 2022-06-29T16:06:05.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.349+0000 7f6a1d0ad700 1 --2- 172.21.15.60:0/2946444240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a180b2c50 0x7f6a180b30c0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 rx=0x7f6a100032f0 tx=0x7f6a100091d0).ready entity=mon.0 client_cookie=24973092f18fec53 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:05.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.349+0000 7f6a177fe700 1 -- 172.21.15.60:0/2946444240 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a1000b680 con 0x7f6a180b2c50 2022-06-29T16:06:05.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.349+0000 7f6a177fe700 1 -- 172.21.15.60:0/2946444240 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a1000b7e0 con 0x7f6a180b2c50 2022-06-29T16:06:05.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.349+0000 7f6a177fe700 1 -- 172.21.15.60:0/2946444240 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a1000b950 con 0x7f6a180b2c50 2022-06-29T16:06:05.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.349+0000 7f6a1d8ae700 1 -- 172.21.15.60:0/2946444240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a180b2c50 msgr2=0x7f6a180b30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:05.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.349+0000 7f6a1d8ae700 1 --2- 172.21.15.60:0/2946444240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a180b2c50 0x7f6a180b30c0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 rx=0x7f6a100032f0 tx=0x7f6a100091d0).stop 2022-06-29T16:06:05.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.349+0000 7f6a1d8ae700 1 -- 172.21.15.60:0/2946444240 shutdown_connections 2022-06-29T16:06:05.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.349+0000 7f6a1d8ae700 1 --2- 172.21.15.60:0/2946444240 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a180b2c50 0x7f6a180b30c0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:05.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.349+0000 7f6a1d8ae700 1 --2- 172.21.15.60:0/2946444240 >> v2:172.21.15.92:3300/0 conn(0x7f6a180b48b0 0x7f6a18142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:05.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.349+0000 7f6a1d8ae700 1 -- 172.21.15.60:0/2946444240 >> 172.21.15.60:0/2946444240 conn(0x7f6a18062090 msgr2=0x7f6a18062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:05.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.349+0000 7f6a1d8ae700 1 -- 172.21.15.60:0/2946444240 shutdown_connections 2022-06-29T16:06:05.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.349+0000 7f6a1d8ae700 1 -- 172.21.15.60:0/2946444240 wait complete. 2022-06-29T16:06:05.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.349+0000 7f6a1d8ae700 1 Processor -- start 2022-06-29T16:06:05.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.349+0000 7f6a1d8ae700 1 -- start start 2022-06-29T16:06:05.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.349+0000 7f6a1d8ae700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6a180b2c50 0x7f6a18116640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:05.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.349+0000 7f6a1d8ae700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a180b48b0 0x7f6a180aebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:05.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.349+0000 7f6a1d8ae700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6a180b5eb0 0x7f6a18155610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:05.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.349+0000 7f6a1d8ae700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a18115110 con 0x7f6a180b48b0 2022-06-29T16:06:05.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.349+0000 7f6a1d8ae700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a181126b0 con 0x7f6a180b5eb0 2022-06-29T16:06:05.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.349+0000 7f6a1d8ae700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a18112950 con 0x7f6a180b2c50 2022-06-29T16:06:05.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.349+0000 7f6a17fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a180b48b0 0x7f6a180aebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:05.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.349+0000 7f6a17fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a180b48b0 0x7f6a180aebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51362/0 (socket says 172.21.15.60:51362) 2022-06-29T16:06:05.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.349+0000 7f6a17fff700 1 -- 172.21.15.60:0/2207793250 learned_addr learned my addr 172.21.15.60:0/2207793250 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:05.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.353+0000 7f6a1c8ac700 1 --2- 172.21.15.60:0/2207793250 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6a180b2c50 0x7f6a18116640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:05.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.353+0000 7f6a1d0ad700 1 --2- 172.21.15.60:0/2207793250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6a180b5eb0 0x7f6a18155610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:05.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.353+0000 7f6a17fff700 1 -- 172.21.15.60:0/2207793250 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6a180b2c50 msgr2=0x7f6a18116640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:05.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.353+0000 7f6a17fff700 1 --2- 172.21.15.60:0/2207793250 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6a180b2c50 0x7f6a18116640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:05.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.353+0000 7f6a17fff700 1 -- 172.21.15.60:0/2207793250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6a180b5eb0 msgr2=0x7f6a18155610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:05.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.353+0000 7f6a17fff700 1 --2- 172.21.15.60:0/2207793250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6a180b5eb0 0x7f6a18155610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:05.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.353+0000 7f6a17fff700 1 -- 172.21.15.60:0/2207793250 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a10009470 con 0x7f6a180b48b0 2022-06-29T16:06:05.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.353+0000 7f6a17fff700 1 --2- 172.21.15.60:0/2207793250 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a180b48b0 0x7f6a180aebd0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 rx=0x7f6a0400c000 tx=0x7f6a04006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:05.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.353+0000 7f6a15ffb700 1 -- 172.21.15.60:0/2207793250 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a0400c830 con 0x7f6a180b48b0 2022-06-29T16:06:05.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.353+0000 7f6a15ffb700 1 -- 172.21.15.60:0/2207793250 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a0400c990 con 0x7f6a180b48b0 2022-06-29T16:06:05.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.353+0000 7f6a1d8ae700 1 -- 172.21.15.60:0/2207793250 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a18112bf0 con 0x7f6a180b48b0 2022-06-29T16:06:05.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.353+0000 7f6a15ffb700 1 -- 172.21.15.60:0/2207793250 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a0400ccf0 con 0x7f6a180b48b0 2022-06-29T16:06:05.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.353+0000 7f6a1d8ae700 1 -- 172.21.15.60:0/2207793250 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a180af0c0 con 0x7f6a180b48b0 2022-06-29T16:06:05.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.353+0000 7f6a1d8ae700 1 -- 172.21.15.60:0/2207793250 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a181124d0 con 0x7f6a180b48b0 2022-06-29T16:06:05.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.353+0000 7f6a15ffb700 1 -- 172.21.15.60:0/2207793250 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6a0401c460 con 0x7f6a180b48b0 2022-06-29T16:06:05.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.353+0000 7f6a15ffb700 1 --2- 172.21.15.60:0/2207793250 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f69f8053b30 0x7f69f8055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:05.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.357+0000 7f6a15ffb700 1 -- 172.21.15.60:0/2207793250 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6a0406c3a0 con 0x7f6a180b48b0 2022-06-29T16:06:05.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.357+0000 7f6a1c8ac700 1 --2- 172.21.15.60:0/2207793250 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f69f8053b30 0x7f69f8055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:05.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.357+0000 7f6a1c8ac700 1 --2- 172.21.15.60:0/2207793250 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f69f8053b30 0x7f69f8055ff0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7f6a0c00b000 tx=0x7f6a0c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:05.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.361+0000 7f6a15ffb700 1 -- 172.21.15.60:0/2207793250 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6a04043260 con 0x7f6a180b48b0 2022-06-29T16:06:05.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.489+0000 7f6a1d8ae700 1 -- 172.21.15.60:0/2207793250 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a180573d0 con 0x7f6a180b48b0 2022-06-29T16:06:05.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.489+0000 7f6a15ffb700 1 -- 172.21.15.60:0/2207793250 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+16054 (secure 0 0 0) 0x7f6a04046880 con 0x7f6a180b48b0 2022-06-29T16:06:05.495 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:05.495 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":99,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:06:05.226336+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:06:03.648757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2022-06-29T16:06:04.556403+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:06:05.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.493+0000 7f6a1d8ae700 1 -- 172.21.15.60:0/2207793250 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f69f8053b30 msgr2=0x7f69f8055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:05.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.493+0000 7f6a1d8ae700 1 --2- 172.21.15.60:0/2207793250 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f69f8053b30 0x7f69f8055ff0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7f6a0c00b000 tx=0x7f6a0c009040).stop 2022-06-29T16:06:05.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.493+0000 7f6a1d8ae700 1 -- 172.21.15.60:0/2207793250 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a180b48b0 msgr2=0x7f6a180aebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:05.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.493+0000 7f6a1d8ae700 1 --2- 172.21.15.60:0/2207793250 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a180b48b0 0x7f6a180aebd0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 rx=0x7f6a0400c000 tx=0x7f6a04006df0).stop 2022-06-29T16:06:05.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.493+0000 7f6a1d8ae700 1 -- 172.21.15.60:0/2207793250 shutdown_connections 2022-06-29T16:06:05.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.493+0000 7f6a1d8ae700 1 --2- 172.21.15.60:0/2207793250 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f69f8053b30 0x7f69f8055ff0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:05.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.493+0000 7f6a1d8ae700 1 --2- 172.21.15.60:0/2207793250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6a180b5eb0 0x7f6a18155610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:05.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.493+0000 7f6a1d8ae700 1 --2- 172.21.15.60:0/2207793250 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a180b48b0 0x7f6a180aebd0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:05.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.493+0000 7f6a1d8ae700 1 --2- 172.21.15.60:0/2207793250 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6a180b2c50 0x7f6a18116640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:05.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.493+0000 7f6a1d8ae700 1 -- 172.21.15.60:0/2207793250 >> 172.21.15.60:0/2207793250 conn(0x7f6a18062090 msgr2=0x7f6a18141210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:05.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.493+0000 7f6a1d8ae700 1 -- 172.21.15.60:0/2207793250 shutdown_connections 2022-06-29T16:06:05.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.493+0000 7f6a1d8ae700 1 -- 172.21.15.60:0/2207793250 wait complete. 2022-06-29T16:06:05.509 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:06:05.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.601+0000 7f0878c21700 1 Processor -- start 2022-06-29T16:06:05.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.601+0000 7f0878c21700 1 -- start start 2022-06-29T16:06:05.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.601+0000 7f0878c21700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f087413f3d0 0x7f0874060260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:05.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.601+0000 7f0878c21700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0874143820 0x7f08740b6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:05.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.601+0000 7f0878c21700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08741430e0 con 0x7f08740b6c00 2022-06-29T16:06:05.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.601+0000 7f0878c21700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0874142ef0 con 0x7f0874143820 2022-06-29T16:06:05.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.601+0000 7f0878c21700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08740acb90 con 0x7f087413f3d0 2022-06-29T16:06:05.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.601+0000 7f0871d9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0874143820 0x7f08740b6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:05.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.601+0000 7f0872d9d700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f08740b6c00 0x7f087413f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59458/0 (socket says 172.21.15.60:59458) 2022-06-29T16:06:05.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.601+0000 7f0872d9d700 1 -- 172.21.15.60:0/759728730 learned_addr learned my addr 172.21.15.60:0/759728730 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:05.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.601+0000 7f087259c700 1 --2- 172.21.15.60:0/759728730 >> v2:172.21.15.92:3300/0 conn(0x7f087413f3d0 0x7f0874060260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:05.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.601+0000 7f0871d9b700 1 -- 172.21.15.60:0/759728730 >> v2:172.21.15.92:3300/0 conn(0x7f087413f3d0 msgr2=0x7f0874060260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:05.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.601+0000 7f0871d9b700 1 --2- 172.21.15.60:0/759728730 >> v2:172.21.15.92:3300/0 conn(0x7f087413f3d0 0x7f0874060260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:05.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.601+0000 7f0871d9b700 1 -- 172.21.15.60:0/759728730 >> v1:172.21.15.60:6789/0 conn(0x7f08740b6c00 legacy=0x7f087413f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:05.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.601+0000 7f0871d9b700 1 -- 172.21.15.60:0/759728730 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0874143380 con 0x7f0874143820 2022-06-29T16:06:05.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.601+0000 7f087159a700 1 -- 172.21.15.60:0/759728730 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1105882185 0 0) 0x7f08741430e0 con 0x7f08740b6c00 2022-06-29T16:06:05.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.605+0000 7f0871d9b700 1 --2- 172.21.15.60:0/759728730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0874143820 0x7f08740b6710 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 rx=0x7f0864002f70 tx=0x7f0864006e80).ready entity=mon.0 client_cookie=17c5db979e905a54 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:05.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.605+0000 7f087159a700 1 -- 172.21.15.60:0/759728730 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f086400b980 con 0x7f0874143820 2022-06-29T16:06:05.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.605+0000 7f087159a700 1 -- 172.21.15.60:0/759728730 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0864000e90 con 0x7f0874143820 2022-06-29T16:06:05.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.605+0000 7f087159a700 1 -- 172.21.15.60:0/759728730 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f086400bcd0 con 0x7f0874143820 2022-06-29T16:06:05.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.605+0000 7f0878c21700 1 -- 172.21.15.60:0/759728730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0874143820 msgr2=0x7f08740b6710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:05.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.605+0000 7f0878c21700 1 --2- 172.21.15.60:0/759728730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0874143820 0x7f08740b6710 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 rx=0x7f0864002f70 tx=0x7f0864006e80).stop 2022-06-29T16:06:05.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.605+0000 7f0878c21700 1 -- 172.21.15.60:0/759728730 shutdown_connections 2022-06-29T16:06:05.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.605+0000 7f0878c21700 1 --2- 172.21.15.60:0/759728730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0874143820 0x7f08740b6710 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:05.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.605+0000 7f0878c21700 1 --2- 172.21.15.60:0/759728730 >> v2:172.21.15.92:3300/0 conn(0x7f087413f3d0 0x7f0874060260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:05.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.605+0000 7f0878c21700 1 -- 172.21.15.60:0/759728730 >> 172.21.15.60:0/759728730 conn(0x7f08740643e0 msgr2=0x7f08740b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:05.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.605+0000 7f0878c21700 1 -- 172.21.15.60:0/759728730 shutdown_connections 2022-06-29T16:06:05.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.605+0000 7f0878c21700 1 -- 172.21.15.60:0/759728730 wait complete. 2022-06-29T16:06:05.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.605+0000 7f0878c21700 1 Processor -- start 2022-06-29T16:06:05.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.605+0000 7f0878c21700 1 -- start start 2022-06-29T16:06:05.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.609+0000 7f0878c21700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08740b6c00 0x7f08740b57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:05.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.609+0000 7f0878c21700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f087413f3d0 0x7f0874098fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:05.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.609+0000 7f0878c21700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0874143820 0x7f0874151620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:05.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.609+0000 7f0878c21700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0874067780 con 0x7f0874143820 2022-06-29T16:06:05.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.609+0000 7f0878c21700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0874067520 con 0x7f087413f3d0 2022-06-29T16:06:05.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.609+0000 7f0878c21700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0874151180 con 0x7f08740b6c00 2022-06-29T16:06:05.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.609+0000 7f0872d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0874143820 0x7f0874151620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:05.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.609+0000 7f0872d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0874143820 0x7f0874151620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51378/0 (socket says 172.21.15.60:51378) 2022-06-29T16:06:05.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.609+0000 7f0872d9d700 1 -- 172.21.15.60:0/3174826034 learned_addr learned my addr 172.21.15.60:0/3174826034 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:05.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.609+0000 7f0871d9b700 1 --2- 172.21.15.60:0/3174826034 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f087413f3d0 0x7f0874098fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:05.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.609+0000 7f087259c700 1 --2- 172.21.15.60:0/3174826034 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08740b6c00 0x7f08740b57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:05.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.609+0000 7f0872d9d700 1 -- 172.21.15.60:0/3174826034 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08740b6c00 msgr2=0x7f08740b57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:05.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.609+0000 7f0872d9d700 1 --2- 172.21.15.60:0/3174826034 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08740b6c00 0x7f08740b57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:05.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.609+0000 7f0872d9d700 1 -- 172.21.15.60:0/3174826034 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f087413f3d0 msgr2=0x7f0874098fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:05.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.609+0000 7f0872d9d700 1 --2- 172.21.15.60:0/3174826034 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f087413f3d0 0x7f0874098fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:05.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.609+0000 7f0872d9d700 1 -- 172.21.15.60:0/3174826034 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086400d410 con 0x7f0874143820 2022-06-29T16:06:05.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.609+0000 7f0872d9d700 1 --2- 172.21.15.60:0/3174826034 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0874143820 0x7f0874151620 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 rx=0x7f086c001250 tx=0x7f086c007d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:05.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.609+0000 7f08637fe700 1 -- 172.21.15.60:0/3174826034 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f086c005ba0 con 0x7f0874143820 2022-06-29T16:06:05.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.609+0000 7f08637fe700 1 -- 172.21.15.60:0/3174826034 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f086c005d00 con 0x7f0874143820 2022-06-29T16:06:05.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.609+0000 7f0878c21700 1 -- 172.21.15.60:0/3174826034 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f087409bd70 con 0x7f0874143820 2022-06-29T16:06:05.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.609+0000 7f08637fe700 1 -- 172.21.15.60:0/3174826034 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f086c004a90 con 0x7f0874143820 2022-06-29T16:06:05.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.609+0000 7f0878c21700 1 -- 172.21.15.60:0/3174826034 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08740663f0 con 0x7f0874143820 2022-06-29T16:06:05.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.609+0000 7f0878c21700 1 -- 172.21.15.60:0/3174826034 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0874151340 con 0x7f0874143820 2022-06-29T16:06:05.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.613+0000 7f08637fe700 1 -- 172.21.15.60:0/3174826034 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f086c004bf0 con 0x7f0874143820 2022-06-29T16:06:05.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.613+0000 7f08637fe700 1 --2- 172.21.15.60:0/3174826034 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0850053b00 0x7f0850055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:05.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.613+0000 7f08637fe700 1 -- 172.21.15.60:0/3174826034 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f086c06ba70 con 0x7f0874143820 2022-06-29T16:06:05.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.613+0000 7f087259c700 1 --2- 172.21.15.60:0/3174826034 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0850053b00 0x7f0850055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:05.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.613+0000 7f087259c700 1 --2- 172.21.15.60:0/3174826034 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0850053b00 0x7f0850055fc0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7f086800b000 tx=0x7f0868006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:05.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.621+0000 7f08637fe700 1 -- 172.21.15.60:0/3174826034 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f086c042990 con 0x7f0874143820 2022-06-29T16:06:05.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.741+0000 7f0878c21700 1 -- 172.21.15.60:0/3174826034 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08740573d0 con 0x7f0874143820 2022-06-29T16:06:05.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.745+0000 7f08637fe700 1 -- 172.21.15.60:0/3174826034 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+16054 (secure 0 0 0) 0x7f086c06be60 con 0x7f0874143820 2022-06-29T16:06:05.750 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:05.750 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":99,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:06:05.226336+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:06:03.648757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2022-06-29T16:06:04.556403+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:06:05.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.745+0000 7f0878c21700 1 -- 172.21.15.60:0/3174826034 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0850053b00 msgr2=0x7f0850055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:05.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.745+0000 7f0878c21700 1 --2- 172.21.15.60:0/3174826034 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0850053b00 0x7f0850055fc0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7f086800b000 tx=0x7f0868006df0).stop 2022-06-29T16:06:05.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.745+0000 7f0878c21700 1 -- 172.21.15.60:0/3174826034 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0874143820 msgr2=0x7f0874151620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:05.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.745+0000 7f0878c21700 1 --2- 172.21.15.60:0/3174826034 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0874143820 0x7f0874151620 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 rx=0x7f086c001250 tx=0x7f086c007d80).stop 2022-06-29T16:06:05.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.745+0000 7f0878c21700 1 -- 172.21.15.60:0/3174826034 shutdown_connections 2022-06-29T16:06:05.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.745+0000 7f0878c21700 1 --2- 172.21.15.60:0/3174826034 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0850053b00 0x7f0850055fc0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:05.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.745+0000 7f0878c21700 1 --2- 172.21.15.60:0/3174826034 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0874143820 0x7f0874151620 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:05.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.745+0000 7f0878c21700 1 --2- 172.21.15.60:0/3174826034 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f087413f3d0 0x7f0874098fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:05.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.745+0000 7f0878c21700 1 --2- 172.21.15.60:0/3174826034 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08740b6c00 0x7f08740b57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:05.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.745+0000 7f0878c21700 1 -- 172.21.15.60:0/3174826034 >> 172.21.15.60:0/3174826034 conn(0x7f08740643e0 msgr2=0x7f087413f9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:05.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.745+0000 7f0878c21700 1 -- 172.21.15.60:0/3174826034 shutdown_connections 2022-06-29T16:06:05.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.745+0000 7f0878c21700 1 -- 172.21.15.60:0/3174826034 wait complete. 2022-06-29T16:06:05.763 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-06-29T16:06:05.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.869+0000 7fec372a9700 1 Processor -- start 2022-06-29T16:06:05.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.869+0000 7fec372a9700 1 -- start start 2022-06-29T16:06:05.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.869+0000 7fec372a9700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec300b0890 0x7fec300b4540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:05.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.869+0000 7fec372a9700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fec300b3b40 0x7fec300b4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:05.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.869+0000 7fec372a9700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec300b9fd0 con 0x7fec300b9bf0 2022-06-29T16:06:05.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.873+0000 7fec372a9700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec300b1a50 con 0x7fec300b3b40 2022-06-29T16:06:05.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.873+0000 7fec372a9700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec300b1cf0 con 0x7fec300b0890 2022-06-29T16:06:05.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.873+0000 7fec35aa6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec300b0890 0x7fec300b4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:05.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.873+0000 7fec36aa8700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fec300b3b40 0x7fec300b4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:05.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.873+0000 7fec35aa6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec300b0890 0x7fec300b4540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:36246/0 (socket says 172.21.15.60:36246) 2022-06-29T16:06:05.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.873+0000 7fec35aa6700 1 -- 172.21.15.60:0/506451435 learned_addr learned my addr 172.21.15.60:0/506451435 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:05.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.873+0000 7fec36aa8700 1 -- 172.21.15.60:0/506451435 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec300b0890 msgr2=0x7fec300b4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:05.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.873+0000 7fec36aa8700 1 --2- 172.21.15.60:0/506451435 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec300b0890 0x7fec300b4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:05.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.873+0000 7fec36aa8700 1 -- 172.21.15.60:0/506451435 >> v1:172.21.15.92:6789/0 conn(0x7fec300b9bf0 legacy=0x7fec300b2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:05.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.873+0000 7fec36aa8700 1 -- 172.21.15.60:0/506451435 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec300b3470 con 0x7fec300b3b40 2022-06-29T16:06:05.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.873+0000 7fec36aa8700 1 --2- 172.21.15.60:0/506451435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec300b3b40 0x7fec300b4a30 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7fec280032f0 tx=0x7fec280091d0).ready entity=mon.1 client_cookie=cef2e5571d9534ea server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:05.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.873+0000 7fec352a5700 1 -- 172.21.15.60:0/506451435 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec2800b6c0 con 0x7fec300b3b40 2022-06-29T16:06:05.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.873+0000 7fec352a5700 1 -- 172.21.15.60:0/506451435 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec2800b820 con 0x7fec300b3b40 2022-06-29T16:06:05.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.873+0000 7fec352a5700 1 -- 172.21.15.60:0/506451435 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec2800b990 con 0x7fec300b3b40 2022-06-29T16:06:05.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.873+0000 7fec372a9700 1 -- 172.21.15.60:0/506451435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec300b3b40 msgr2=0x7fec300b4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:05.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.873+0000 7fec372a9700 1 --2- 172.21.15.60:0/506451435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec300b3b40 0x7fec300b4a30 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7fec280032f0 tx=0x7fec280091d0).stop 2022-06-29T16:06:05.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.873+0000 7fec372a9700 1 -- 172.21.15.60:0/506451435 shutdown_connections 2022-06-29T16:06:05.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.873+0000 7fec372a9700 1 --2- 172.21.15.60:0/506451435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec300b3b40 0x7fec300b4a30 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:05.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.873+0000 7fec372a9700 1 --2- 172.21.15.60:0/506451435 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec300b0890 0x7fec300b4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:05.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.873+0000 7fec372a9700 1 -- 172.21.15.60:0/506451435 >> 172.21.15.60:0/506451435 conn(0x7fec30062b60 msgr2=0x7fec300b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:05.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.873+0000 7fec372a9700 1 -- 172.21.15.60:0/506451435 shutdown_connections 2022-06-29T16:06:05.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.873+0000 7fec372a9700 1 -- 172.21.15.60:0/506451435 wait complete. 2022-06-29T16:06:05.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.877+0000 7fec372a9700 1 Processor -- start 2022-06-29T16:06:05.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.877+0000 7fec372a9700 1 -- start start 2022-06-29T16:06:05.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.877+0000 7fec372a9700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec300b0890 0x7fec300aefb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:05.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.877+0000 7fec372a9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec300b3b40 0x7fec30144060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:05.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.877+0000 7fec372a9700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec300b9bf0 0x7fec30113850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:05.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.877+0000 7fec372a9700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec30113420 con 0x7fec300b3b40 2022-06-29T16:06:05.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.877+0000 7fec372a9700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec30146d30 con 0x7fec300b9bf0 2022-06-29T16:06:05.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.877+0000 7fec372a9700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec30146fd0 con 0x7fec300b0890 2022-06-29T16:06:05.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.877+0000 7fec35aa6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec300b3b40 0x7fec30144060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:05.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.877+0000 7fec35aa6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec300b3b40 0x7fec30144060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51390/0 (socket says 172.21.15.60:51390) 2022-06-29T16:06:05.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.877+0000 7fec35aa6700 1 -- 172.21.15.60:0/2021394839 learned_addr learned my addr 172.21.15.60:0/2021394839 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:05.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.877+0000 7fec36aa8700 1 --2- 172.21.15.60:0/2021394839 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec300b9bf0 0x7fec30113850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:05.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.877+0000 7fec362a7700 1 --2- 172.21.15.60:0/2021394839 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec300b0890 0x7fec300aefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:05.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.877+0000 7fec35aa6700 1 -- 172.21.15.60:0/2021394839 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec300b0890 msgr2=0x7fec300aefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:05.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.877+0000 7fec35aa6700 1 --2- 172.21.15.60:0/2021394839 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec300b0890 0x7fec300aefb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:05.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.877+0000 7fec35aa6700 1 -- 172.21.15.60:0/2021394839 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec300b9bf0 msgr2=0x7fec30113850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:05.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.877+0000 7fec35aa6700 1 --2- 172.21.15.60:0/2021394839 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec300b9bf0 0x7fec30113850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:05.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.877+0000 7fec35aa6700 1 -- 172.21.15.60:0/2021394839 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec28009470 con 0x7fec300b3b40 2022-06-29T16:06:05.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.877+0000 7fec36aa8700 1 --2- 172.21.15.60:0/2021394839 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec300b9bf0 0x7fec30113850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:06:05.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.877+0000 7fec35aa6700 1 --2- 172.21.15.60:0/2021394839 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec300b3b40 0x7fec30144060 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 rx=0x7fec2000c000 tx=0x7fec20008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:05.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.877+0000 7fec1f7fe700 1 -- 172.21.15.60:0/2021394839 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec20013610 con 0x7fec300b3b40 2022-06-29T16:06:05.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.877+0000 7fec1f7fe700 1 -- 172.21.15.60:0/2021394839 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec20013770 con 0x7fec300b3b40 2022-06-29T16:06:05.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.877+0000 7fec372a9700 1 -- 172.21.15.60:0/2021394839 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec301469b0 con 0x7fec300b3b40 2022-06-29T16:06:05.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.877+0000 7fec1f7fe700 1 -- 172.21.15.60:0/2021394839 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec20013ad0 con 0x7fec300b3b40 2022-06-29T16:06:05.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.877+0000 7fec372a9700 1 -- 172.21.15.60:0/2021394839 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec30147e40 con 0x7fec300b3b40 2022-06-29T16:06:05.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.881+0000 7fec372a9700 1 -- 172.21.15.60:0/2021394839 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec30113590 con 0x7fec300b3b40 2022-06-29T16:06:05.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.881+0000 7fec1f7fe700 1 -- 172.21.15.60:0/2021394839 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fec20023070 con 0x7fec300b3b40 2022-06-29T16:06:05.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.881+0000 7fec1f7fe700 1 --2- 172.21.15.60:0/2021394839 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fec10053b30 0x7fec10055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:05.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.881+0000 7fec1f7fe700 1 -- 172.21.15.60:0/2021394839 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fec2006c3c0 con 0x7fec300b3b40 2022-06-29T16:06:05.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.881+0000 7fec362a7700 1 --2- 172.21.15.60:0/2021394839 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fec10053b30 0x7fec10055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:05.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.881+0000 7fec362a7700 1 --2- 172.21.15.60:0/2021394839 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fec10053b30 0x7fec10055ff0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7fec2c00b000 tx=0x7fec2c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:05.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:05.889+0000 7fec1f7fe700 1 -- 172.21.15.60:0/2021394839 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fec30113590 con 0x7fec300b3b40 2022-06-29T16:06:06.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.053+0000 7fec372a9700 1 -- 172.21.15.60:0/2021394839 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fec30146ba0 con 0x7fec300b3b40 2022-06-29T16:06:06.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.253+0000 7fec1f7fe700 1 -- 172.21.15.60:0/2021394839 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v46) v1 ==== 118+0+0 (secure 0 0 0) 0x7fec20043150 con 0x7fec300b3b40 2022-06-29T16:06:06.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.253+0000 7fec372a9700 1 -- 172.21.15.60:0/2021394839 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fec10053b30 msgr2=0x7fec10055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:06.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.253+0000 7fec372a9700 1 --2- 172.21.15.60:0/2021394839 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fec10053b30 0x7fec10055ff0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7fec2c00b000 tx=0x7fec2c009040).stop 2022-06-29T16:06:06.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.253+0000 7fec372a9700 1 -- 172.21.15.60:0/2021394839 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec300b3b40 msgr2=0x7fec30144060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:06.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.253+0000 7fec372a9700 1 --2- 172.21.15.60:0/2021394839 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec300b3b40 0x7fec30144060 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 rx=0x7fec2000c000 tx=0x7fec20008040).stop 2022-06-29T16:06:06.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.253+0000 7fec372a9700 1 -- 172.21.15.60:0/2021394839 shutdown_connections 2022-06-29T16:06:06.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.253+0000 7fec372a9700 1 --2- 172.21.15.60:0/2021394839 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fec10053b30 0x7fec10055ff0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:06.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.253+0000 7fec372a9700 1 --2- 172.21.15.60:0/2021394839 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec300b9bf0 0x7fec30113850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:06.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.253+0000 7fec372a9700 1 --2- 172.21.15.60:0/2021394839 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec300b3b40 0x7fec30144060 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:06.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.253+0000 7fec372a9700 1 --2- 172.21.15.60:0/2021394839 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec300b0890 0x7fec300aefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:06.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.253+0000 7fec372a9700 1 -- 172.21.15.60:0/2021394839 >> 172.21.15.60:0/2021394839 conn(0x7fec30062b60 msgr2=0x7fec300b63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:06.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.253+0000 7fec372a9700 1 -- 172.21.15.60:0/2021394839 shutdown_connections 2022-06-29T16:06:06.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.257+0000 7fec372a9700 1 -- 172.21.15.60:0/2021394839 wait complete. 2022-06-29T16:06:06.277 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:06:06.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.385+0000 7fdceef2a700 1 Processor -- start 2022-06-29T16:06:06.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.385+0000 7fdceef2a700 1 -- start start 2022-06-29T16:06:06.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.385+0000 7fdceef2a700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdce805ff30 0x7fdce80608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:06.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.385+0000 7fdceef2a700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdce8060470 con 0x7fdce80b09f0 2022-06-29T16:06:06.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.385+0000 7fdceef2a700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdce805f880 con 0x7fdce805f4a0 2022-06-29T16:06:06.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.385+0000 7fdceef2a700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdce8060d90 con 0x7fdce805ff30 2022-06-29T16:06:06.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.385+0000 7fdced727700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fdce80b09f0 0x7fdce814aad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59482/0 (socket says 172.21.15.60:59482) 2022-06-29T16:06:06.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.385+0000 7fdced727700 1 -- 172.21.15.60:0/2542048953 learned_addr learned my addr 172.21.15.60:0/2542048953 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:06.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.385+0000 7fdcee729700 1 --2- 172.21.15.60:0/2542048953 >> v2:172.21.15.92:3300/0 conn(0x7fdce805ff30 0x7fdce80608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:06.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.385+0000 7fdcecf26700 1 -- 172.21.15.60:0/2542048953 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2031128760 0 0) 0x7fdce8060470 con 0x7fdce80b09f0 2022-06-29T16:06:06.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.385+0000 7fdcecf26700 1 -- 172.21.15.60:0/2542048953 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcd4003260 con 0x7fdce80b09f0 2022-06-29T16:06:06.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.385+0000 7fdcecf26700 1 -- 172.21.15.60:0/2542048953 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3198733425 0 0) 0x7fdce805f880 con 0x7fdce805f4a0 2022-06-29T16:06:06.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.385+0000 7fdcecf26700 1 -- 172.21.15.60:0/2542048953 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdce8060470 con 0x7fdce805f4a0 2022-06-29T16:06:06.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdcecf26700 1 -- 172.21.15.60:0/2542048953 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2680450276 0 0) 0x7fdcd4003260 con 0x7fdce80b09f0 2022-06-29T16:06:06.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdcecf26700 1 -- 172.21.15.60:0/2542048953 >> v2:172.21.15.92:3300/0 conn(0x7fdce805ff30 msgr2=0x7fdce80608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:06.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdcecf26700 1 --2- 172.21.15.60:0/2542048953 >> v2:172.21.15.92:3300/0 conn(0x7fdce805ff30 0x7fdce80608a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:06.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdcecf26700 1 -- 172.21.15.60:0/2542048953 >> v1:172.21.15.92:6789/0 conn(0x7fdce805f4a0 legacy=0x7fdce814a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:06.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdcecf26700 1 -- 172.21.15.60:0/2542048953 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce814b1e0 con 0x7fdce80b09f0 2022-06-29T16:06:06.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdcecf26700 1 -- 172.21.15.60:0/2542048953 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7fdce40019e0 con 0x7fdce80b09f0 2022-06-29T16:06:06.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdcecf26700 1 -- 172.21.15.60:0/2542048953 >> v1:172.21.15.60:6789/0 conn(0x7fdce80b09f0 legacy=0x7fdce814aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:06.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdcecf26700 1 --2- 172.21.15.60:0/2542048953 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcd40038e0 0x7fdcd4003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:06.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdcecf26700 1 --2- 172.21.15.60:0/2542048953 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdcd4004500 0x7fdcd4004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:06.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdcecf26700 1 --2- 172.21.15.60:0/2542048953 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdcd4004e70 0x7fdcd400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:06.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdcecf26700 1 -- 172.21.15.60:0/2542048953 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdce8060d90 con 0x7fdcd40038e0 2022-06-29T16:06:06.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdcecf26700 1 -- 172.21.15.60:0/2542048953 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd400bdd0 con 0x7fdcd4004500 2022-06-29T16:06:06.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdcecf26700 1 -- 172.21.15.60:0/2542048953 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd400bf40 con 0x7fdcd4004e70 2022-06-29T16:06:06.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdcedf28700 1 --2- 172.21.15.60:0/2542048953 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcd40038e0 0x7fdcd4003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:06.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdced727700 1 --2- 172.21.15.60:0/2542048953 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdcd4004500 0x7fdcd4004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:06.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdcee729700 1 --2- 172.21.15.60:0/2542048953 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdcd4004e70 0x7fdcd400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:06.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdcedf28700 1 -- 172.21.15.60:0/2542048953 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdcd4004e70 msgr2=0x7fdcd400b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:06.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdcedf28700 1 --2- 172.21.15.60:0/2542048953 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdcd4004e70 0x7fdcd400b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:06.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdcedf28700 1 -- 172.21.15.60:0/2542048953 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdcd4004500 msgr2=0x7fdcd4004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:06.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdcedf28700 1 --2- 172.21.15.60:0/2542048953 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdcd4004500 0x7fdcd4004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:06.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdcedf28700 1 -- 172.21.15.60:0/2542048953 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd4003260 con 0x7fdcd40038e0 2022-06-29T16:06:06.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdcee729700 1 --2- 172.21.15.60:0/2542048953 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdcd4004e70 0x7fdcd400b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:06:06.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdcee729700 1 -- 172.21.15.60:0/2542048953 reap_dead start 2022-06-29T16:06:06.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdcedf28700 1 --2- 172.21.15.60:0/2542048953 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcd40038e0 0x7fdcd4003ce0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 rx=0x7fdcd8001110 tx=0x7fdcd8007a00).ready entity=mon.0 client_cookie=26791f921a2b80f7 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:06.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdcecf26700 1 -- 172.21.15.60:0/2542048953 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdcd8007d10 con 0x7fdcd40038e0 2022-06-29T16:06:06.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdcecf26700 1 -- 172.21.15.60:0/2542048953 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcd8007030 con 0x7fdcd40038e0 2022-06-29T16:06:06.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdcecf26700 1 -- 172.21.15.60:0/2542048953 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdcd8007390 con 0x7fdcd40038e0 2022-06-29T16:06:06.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdceef2a700 1 -- 172.21.15.60:0/2542048953 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcd40038e0 msgr2=0x7fdcd4003ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:06.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdceef2a700 1 --2- 172.21.15.60:0/2542048953 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcd40038e0 0x7fdcd4003ce0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 rx=0x7fdcd8001110 tx=0x7fdcd8007a00).stop 2022-06-29T16:06:06.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdceef2a700 1 -- 172.21.15.60:0/2542048953 shutdown_connections 2022-06-29T16:06:06.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdceef2a700 1 --2- 172.21.15.60:0/2542048953 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcd40038e0 0x7fdcd4003ce0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:06.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdceef2a700 1 -- 172.21.15.60:0/2542048953 >> 172.21.15.60:0/2542048953 conn(0x7fdce80637c0 msgr2=0x7fdce8063ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:06.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdceef2a700 1 -- 172.21.15.60:0/2542048953 shutdown_connections 2022-06-29T16:06:06.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdceef2a700 1 -- 172.21.15.60:0/2542048953 wait complete. 2022-06-29T16:06:06.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.389+0000 7fdceef2a700 1 Processor -- start 2022-06-29T16:06:06.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.393+0000 7fdceef2a700 1 -- start start 2022-06-29T16:06:06.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.393+0000 7fdceef2a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcd40038e0 0x7fdce80ae820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:06.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.393+0000 7fdceef2a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdce80adf70 0x7fdce810fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:06.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.393+0000 7fdceef2a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdce8155f90 0x7fdce8156410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:06.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.393+0000 7fdceef2a700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdce80ad280 con 0x7fdcd40038e0 2022-06-29T16:06:06.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.393+0000 7fdceef2a700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdce814a9c0 con 0x7fdce80adf70 2022-06-29T16:06:06.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.393+0000 7fdceef2a700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdce814ac60 con 0x7fdce8155f90 2022-06-29T16:06:06.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.393+0000 7fdcedf28700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcd40038e0 0x7fdce80ae820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:06.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.393+0000 7fdcedf28700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcd40038e0 0x7fdce80ae820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51410/0 (socket says 172.21.15.60:51410) 2022-06-29T16:06:06.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.393+0000 7fdcedf28700 1 -- 172.21.15.60:0/3936406343 learned_addr learned my addr 172.21.15.60:0/3936406343 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:06.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.393+0000 7fdcee729700 1 --2- 172.21.15.60:0/3936406343 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdce8155f90 0x7fdce8156410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:06.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.393+0000 7fdced727700 1 --2- 172.21.15.60:0/3936406343 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdce80adf70 0x7fdce810fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:06.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.393+0000 7fdcedf28700 1 -- 172.21.15.60:0/3936406343 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdce8155f90 msgr2=0x7fdce8156410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:06.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.393+0000 7fdcedf28700 1 --2- 172.21.15.60:0/3936406343 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdce8155f90 0x7fdce8156410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:06.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.393+0000 7fdcedf28700 1 -- 172.21.15.60:0/3936406343 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdce80adf70 msgr2=0x7fdce810fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:06.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.393+0000 7fdcedf28700 1 --2- 172.21.15.60:0/3936406343 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdce80adf70 0x7fdce810fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:06.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.393+0000 7fdcedf28700 1 -- 172.21.15.60:0/3936406343 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce80ad6b0 con 0x7fdcd40038e0 2022-06-29T16:06:06.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.393+0000 7fdcedf28700 1 --2- 172.21.15.60:0/3936406343 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcd40038e0 0x7fdce80ae820 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 rx=0x7fdcd8001110 tx=0x7fdcd8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:06.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.393+0000 7fdcdeffd700 1 -- 172.21.15.60:0/3936406343 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdcd8007d10 con 0x7fdcd40038e0 2022-06-29T16:06:06.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.393+0000 7fdcdeffd700 1 -- 172.21.15.60:0/3936406343 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcd801f430 con 0x7fdcd40038e0 2022-06-29T16:06:06.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.393+0000 7fdcdeffd700 1 -- 172.21.15.60:0/3936406343 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdcd801f790 con 0x7fdcd40038e0 2022-06-29T16:06:06.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.393+0000 7fdceef2a700 1 -- 172.21.15.60:0/3936406343 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdce8156900 con 0x7fdcd40038e0 2022-06-29T16:06:06.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.393+0000 7fdceef2a700 1 -- 172.21.15.60:0/3936406343 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdce81594e0 con 0x7fdcd40038e0 2022-06-29T16:06:06.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.397+0000 7fdceef2a700 1 -- 172.21.15.60:0/3936406343 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdce80ad3f0 con 0x7fdcd40038e0 2022-06-29T16:06:06.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.397+0000 7fdcdeffd700 1 -- 172.21.15.60:0/3936406343 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdcd8025020 con 0x7fdcd40038e0 2022-06-29T16:06:06.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.397+0000 7fdcdeffd700 1 --2- 172.21.15.60:0/3936406343 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdcc8053950 0x7fdcc8055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:06.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.397+0000 7fdcdeffd700 1 -- 172.21.15.60:0/3936406343 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fdcd806cfb0 con 0x7fdcd40038e0 2022-06-29T16:06:06.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.397+0000 7fdced727700 1 --2- 172.21.15.60:0/3936406343 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdcc8053950 0x7fdcc8055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:06.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.397+0000 7fdced727700 1 --2- 172.21.15.60:0/3936406343 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdcc8053950 0x7fdcc8055e10 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7fdce4005000 tx=0x7fdce4002130).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:06.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.401+0000 7fdcdeffd700 1 -- 172.21.15.60:0/3936406343 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdce80ad3f0 con 0x7fdcd40038e0 2022-06-29T16:06:06.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.569+0000 7fdceef2a700 1 -- 172.21.15.60:0/3936406343 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdce80ad3c0 con 0x7fdcd40038e0 2022-06-29T16:06:06.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.569+0000 7fdcdeffd700 1 -- 172.21.15.60:0/3936406343 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+2140 (secure 0 0 0) 0x7fdcd8043ce0 con 0x7fdcd40038e0 2022-06-29T16:06:06.574 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:06.574 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":46,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7320,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/3498160620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3498160620},{"type":"v1","addr":"172.21.15.60:6833","nonce":3498160620}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:06:05.226825+0000","modified":"2022-06-29T16:06:06.241973+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7327},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7327":{"gid":7327,"name":"b","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/1551394182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1551394182},{"type":"v1","addr":"172.21.15.92:6835","nonce":1551394182}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2022-06-29T16:06:06.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.569+0000 7fdceef2a700 1 -- 172.21.15.60:0/3936406343 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdcc8053950 msgr2=0x7fdcc8055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:06.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.569+0000 7fdceef2a700 1 --2- 172.21.15.60:0/3936406343 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdcc8053950 0x7fdcc8055e10 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7fdce4005000 tx=0x7fdce4002130).stop 2022-06-29T16:06:06.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.569+0000 7fdceef2a700 1 -- 172.21.15.60:0/3936406343 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcd40038e0 msgr2=0x7fdce80ae820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:06.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.569+0000 7fdceef2a700 1 --2- 172.21.15.60:0/3936406343 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcd40038e0 0x7fdce80ae820 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 rx=0x7fdcd8001110 tx=0x7fdcd8016000).stop 2022-06-29T16:06:06.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.569+0000 7fdceef2a700 1 -- 172.21.15.60:0/3936406343 shutdown_connections 2022-06-29T16:06:06.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.569+0000 7fdceef2a700 1 --2- 172.21.15.60:0/3936406343 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdcc8053950 0x7fdcc8055e10 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:06.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.569+0000 7fdceef2a700 1 --2- 172.21.15.60:0/3936406343 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdce8155f90 0x7fdce8156410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:06.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.569+0000 7fdceef2a700 1 --2- 172.21.15.60:0/3936406343 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdce80adf70 0x7fdce810fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:06.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.569+0000 7fdceef2a700 1 --2- 172.21.15.60:0/3936406343 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcd40038e0 0x7fdce80ae820 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:06.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.569+0000 7fdceef2a700 1 -- 172.21.15.60:0/3936406343 >> 172.21.15.60:0/3936406343 conn(0x7fdce80637c0 msgr2=0x7fdce8065330 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:06.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.569+0000 7fdceef2a700 1 -- 172.21.15.60:0/3936406343 shutdown_connections 2022-06-29T16:06:06.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.569+0000 7fdceef2a700 1 -- 172.21.15.60:0/3936406343 wait complete. 2022-06-29T16:06:06.579 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 46 2022-06-29T16:06:06.588 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:06:06.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.685+0000 7f1c16ab8700 1 Processor -- start 2022-06-29T16:06:06.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.685+0000 7f1c16ab8700 1 -- start start 2022-06-29T16:06:06.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.689+0000 7f1c16ab8700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1c100b5950 0x7f1c10067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:06.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.689+0000 7f1c16ab8700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c10063ad0 0x7f1c1014af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:06.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.689+0000 7f1c16ab8700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c100acba0 con 0x7f1c1005bd20 2022-06-29T16:06:06.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.689+0000 7f1c16ab8700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c10067220 con 0x7f1c100b5950 2022-06-29T16:06:06.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.689+0000 7f1c16ab8700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c100b5d30 con 0x7f1c10063ad0 2022-06-29T16:06:06.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.689+0000 7f1c14854700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1c100b5950 0x7f1c10067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:06.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.689+0000 7f1c14854700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1c100b5950 0x7f1c10067680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51416/0 (socket says 172.21.15.60:51416) 2022-06-29T16:06:06.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.689+0000 7f1c14854700 1 -- 172.21.15.60:0/1477329964 learned_addr learned my addr 172.21.15.60:0/1477329964 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:06.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.689+0000 7f1c15055700 1 --1- 172.21.15.60:0/1477329964 >> v1:172.21.15.60:6789/0 conn(0x7f1c1005bd20 0x7f1c1005c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59506/0 (socket says 172.21.15.60:59506) 2022-06-29T16:06:06.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.689+0000 7f1c0ffff700 1 --2- 172.21.15.60:0/1477329964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c10063ad0 0x7f1c1014af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:06.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.689+0000 7f1c14854700 1 -- 172.21.15.60:0/1477329964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c10063ad0 msgr2=0x7f1c1014af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:06.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.689+0000 7f1c14854700 1 --2- 172.21.15.60:0/1477329964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c10063ad0 0x7f1c1014af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:06.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.689+0000 7f1c14854700 1 -- 172.21.15.60:0/1477329964 >> v1:172.21.15.60:6789/0 conn(0x7f1c1005bd20 legacy=0x7f1c1005c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:06.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.689+0000 7f1c14854700 1 -- 172.21.15.60:0/1477329964 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c100ace00 con 0x7f1c100b5950 2022-06-29T16:06:06.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.689+0000 7f1c14854700 1 --2- 172.21.15.60:0/1477329964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c100b5950 0x7f1c10067680 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 rx=0x7f1c00002f70 tx=0x7f1c00006eb0).ready entity=mon.0 client_cookie=db332ad30bb7cb53 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:06.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.689+0000 7f1c0f7fe700 1 -- 172.21.15.60:0/1477329964 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c0000d5c0 con 0x7f1c100b5950 2022-06-29T16:06:06.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.689+0000 7f1c0f7fe700 1 -- 172.21.15.60:0/1477329964 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c00000e90 con 0x7f1c100b5950 2022-06-29T16:06:06.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.689+0000 7f1c0f7fe700 1 -- 172.21.15.60:0/1477329964 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c0000d950 con 0x7f1c100b5950 2022-06-29T16:06:06.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.689+0000 7f1c16ab8700 1 -- 172.21.15.60:0/1477329964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c100b5950 msgr2=0x7f1c10067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:06.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.689+0000 7f1c16ab8700 1 --2- 172.21.15.60:0/1477329964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c100b5950 0x7f1c10067680 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 rx=0x7f1c00002f70 tx=0x7f1c00006eb0).stop 2022-06-29T16:06:06.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.689+0000 7f1c16ab8700 1 -- 172.21.15.60:0/1477329964 shutdown_connections 2022-06-29T16:06:06.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.689+0000 7f1c16ab8700 1 --2- 172.21.15.60:0/1477329964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c10063ad0 0x7f1c1014af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:06.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.689+0000 7f1c16ab8700 1 --2- 172.21.15.60:0/1477329964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c100b5950 0x7f1c10067680 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:06.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.689+0000 7f1c16ab8700 1 -- 172.21.15.60:0/1477329964 >> 172.21.15.60:0/1477329964 conn(0x7f1c100632c0 msgr2=0x7f1c100b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:06.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.689+0000 7f1c16ab8700 1 -- 172.21.15.60:0/1477329964 shutdown_connections 2022-06-29T16:06:06.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.689+0000 7f1c16ab8700 1 -- 172.21.15.60:0/1477329964 wait complete. 2022-06-29T16:06:06.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.693+0000 7f1c16ab8700 1 Processor -- start 2022-06-29T16:06:06.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.693+0000 7f1c16ab8700 1 -- start start 2022-06-29T16:06:06.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.693+0000 7f1c16ab8700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c1005bd20 0x7f1c10117850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:06.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.693+0000 7f1c16ab8700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c10063ad0 0x7f1c10156230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:06.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.693+0000 7f1c16ab8700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c100b5950 0x7f1c100ae5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:06.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.693+0000 7f1c16ab8700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c10157280 con 0x7f1c1005bd20 2022-06-29T16:06:06.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.693+0000 7f1c16ab8700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c100b0f80 con 0x7f1c10063ad0 2022-06-29T16:06:06.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.693+0000 7f1c16ab8700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c10159800 con 0x7f1c100b5950 2022-06-29T16:06:06.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.693+0000 7f1c14854700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c1005bd20 0x7f1c10117850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:06.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.693+0000 7f1c14854700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c1005bd20 0x7f1c10117850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51422/0 (socket says 172.21.15.60:51422) 2022-06-29T16:06:06.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.693+0000 7f1c14854700 1 -- 172.21.15.60:0/2367637251 learned_addr learned my addr 172.21.15.60:0/2367637251 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:06.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.693+0000 7f1c15055700 1 --2- 172.21.15.60:0/2367637251 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c100b5950 0x7f1c100ae5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:06.721 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.693+0000 7f1c0ffff700 1 --2- 172.21.15.60:0/2367637251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c10063ad0 0x7f1c10156230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:06.721 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.693+0000 7f1c14854700 1 -- 172.21.15.60:0/2367637251 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c100b5950 msgr2=0x7f1c100ae5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:06.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.693+0000 7f1c14854700 1 --2- 172.21.15.60:0/2367637251 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c100b5950 0x7f1c100ae5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:06.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.693+0000 7f1c14854700 1 -- 172.21.15.60:0/2367637251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c10063ad0 msgr2=0x7f1c10156230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:06.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.693+0000 7f1c14854700 1 --2- 172.21.15.60:0/2367637251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c10063ad0 0x7f1c10156230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:06.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.693+0000 7f1c14854700 1 -- 172.21.15.60:0/2367637251 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c0000b1e0 con 0x7f1c1005bd20 2022-06-29T16:06:06.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.693+0000 7f1c0ffff700 1 --2- 172.21.15.60:0/2367637251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c10063ad0 0x7f1c10156230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:06:06.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.693+0000 7f1c14854700 1 --2- 172.21.15.60:0/2367637251 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c1005bd20 0x7f1c10117850 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 rx=0x7f1c101567a0 tx=0x7f1c00015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:06.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.693+0000 7f1c0dffb700 1 -- 172.21.15.60:0/2367637251 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c0000d940 con 0x7f1c1005bd20 2022-06-29T16:06:06.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.693+0000 7f1c0dffb700 1 -- 172.21.15.60:0/2367637251 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c00011ca0 con 0x7f1c1005bd20 2022-06-29T16:06:06.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.693+0000 7f1c16ab8700 1 -- 172.21.15.60:0/2367637251 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c10148c50 con 0x7f1c1005bd20 2022-06-29T16:06:06.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.693+0000 7f1c0dffb700 1 -- 172.21.15.60:0/2367637251 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c0001e460 con 0x7f1c1005bd20 2022-06-29T16:06:06.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.693+0000 7f1c16ab8700 1 -- 172.21.15.60:0/2367637251 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c100ae0c0 con 0x7f1c1005bd20 2022-06-29T16:06:06.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.697+0000 7f1c16ab8700 1 -- 172.21.15.60:0/2367637251 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c10159940 con 0x7f1c1005bd20 2022-06-29T16:06:06.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.697+0000 7f1c0dffb700 1 -- 172.21.15.60:0/2367637251 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1c00025030 con 0x7f1c1005bd20 2022-06-29T16:06:06.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.697+0000 7f1c0dffb700 1 --2- 172.21.15.60:0/2367637251 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1bf4053a70 0x7f1bf4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:06.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.697+0000 7f1c0dffb700 1 -- 172.21.15.60:0/2367637251 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1c0006d2f0 con 0x7f1c1005bd20 2022-06-29T16:06:06.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.697+0000 7f1c0ffff700 1 --2- 172.21.15.60:0/2367637251 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1bf4053a70 0x7f1bf4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:06.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.697+0000 7f1c0ffff700 1 --2- 172.21.15.60:0/2367637251 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1bf4053a70 0x7f1bf4055f30 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7f1bf800c000 tx=0x7f1bf8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:06.728 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.701+0000 7f1c0dffb700 1 -- 172.21.15.60:0/2367637251 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1c00044210 con 0x7f1c1005bd20 2022-06-29T16:06:06.829 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.825+0000 7f1c16ab8700 1 -- 172.21.15.60:0/2367637251 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c10148a70 con 0x7f1c1005bd20 2022-06-29T16:06:06.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.825+0000 7f1c0dffb700 1 -- 172.21.15.60:0/2367637251 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+16043 (secure 0 0 0) 0x7f1c00023090 con 0x7f1c1005bd20 2022-06-29T16:06:06.831 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:06.832 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":100,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:06:06.229758+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:06:03.648757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2022-06-29T16:06:04.556403+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:06:06.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.829+0000 7f1c16ab8700 1 -- 172.21.15.60:0/2367637251 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1bf4053a70 msgr2=0x7f1bf4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:06.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.829+0000 7f1c16ab8700 1 --2- 172.21.15.60:0/2367637251 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1bf4053a70 0x7f1bf4055f30 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7f1bf800c000 tx=0x7f1bf8008040).stop 2022-06-29T16:06:06.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.829+0000 7f1c16ab8700 1 -- 172.21.15.60:0/2367637251 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c1005bd20 msgr2=0x7f1c10117850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:06.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.829+0000 7f1c16ab8700 1 --2- 172.21.15.60:0/2367637251 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c1005bd20 0x7f1c10117850 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 rx=0x7f1c101567a0 tx=0x7f1c00015000).stop 2022-06-29T16:06:06.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.829+0000 7f1c16ab8700 1 -- 172.21.15.60:0/2367637251 shutdown_connections 2022-06-29T16:06:06.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.829+0000 7f1c16ab8700 1 --2- 172.21.15.60:0/2367637251 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1bf4053a70 0x7f1bf4055f30 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:06.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.829+0000 7f1c16ab8700 1 --2- 172.21.15.60:0/2367637251 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c100b5950 0x7f1c100ae5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:06.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.829+0000 7f1c16ab8700 1 --2- 172.21.15.60:0/2367637251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c10063ad0 0x7f1c10156230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:06.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.829+0000 7f1c16ab8700 1 --2- 172.21.15.60:0/2367637251 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c1005bd20 0x7f1c10117850 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:06.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.829+0000 7f1c16ab8700 1 -- 172.21.15.60:0/2367637251 >> 172.21.15.60:0/2367637251 conn(0x7f1c100632c0 msgr2=0x7f1c10145360 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:06.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.829+0000 7f1c16ab8700 1 -- 172.21.15.60:0/2367637251 shutdown_connections 2022-06-29T16:06:06.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:06.829+0000 7f1c16ab8700 1 -- 172.21.15.60:0/2367637251 wait complete. 2022-06-29T16:06:06.846 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T16:06:07.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.045+0000 7fe66201f700 1 Processor -- start 2022-06-29T16:06:07.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.045+0000 7fe66201f700 1 -- start start 2022-06-29T16:06:07.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.045+0000 7fe66201f700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe65c0ae7c0 0x7fe65c141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:07.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.045+0000 7fe66201f700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe65c0b46c0 0x7fe65c0b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:07.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.045+0000 7fe66201f700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe65c143a40 con 0x7fe65c0b4010 2022-06-29T16:06:07.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.045+0000 7fe66201f700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe65c0aee60 con 0x7fe65c0b46c0 2022-06-29T16:06:07.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.045+0000 7fe66201f700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe65c0af100 con 0x7fe65c0ae7c0 2022-06-29T16:06:07.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.045+0000 7fe65bfff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe65c0b46c0 0x7fe65c0b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:07.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.045+0000 7fe65affd700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe65c0b4010 0x7fe65c142490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59518/0 (socket says 172.21.15.60:59518) 2022-06-29T16:06:07.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.045+0000 7fe65affd700 1 -- 172.21.15.60:0/2723758728 learned_addr learned my addr 172.21.15.60:0/2723758728 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:07.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.045+0000 7fe65b7fe700 1 --2- 172.21.15.60:0/2723758728 >> v2:172.21.15.92:3300/0 conn(0x7fe65c0ae7c0 0x7fe65c141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:07.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.045+0000 7fe65bfff700 1 -- 172.21.15.60:0/2723758728 >> v2:172.21.15.92:3300/0 conn(0x7fe65c0ae7c0 msgr2=0x7fe65c141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:07.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.045+0000 7fe65bfff700 1 --2- 172.21.15.60:0/2723758728 >> v2:172.21.15.92:3300/0 conn(0x7fe65c0ae7c0 0x7fe65c141fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:07.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.045+0000 7fe65bfff700 1 -- 172.21.15.60:0/2723758728 >> v1:172.21.15.60:6789/0 conn(0x7fe65c0b4010 legacy=0x7fe65c142490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:07.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.045+0000 7fe65bfff700 1 -- 172.21.15.60:0/2723758728 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe65c0aeba0 con 0x7fe65c0b46c0 2022-06-29T16:06:07.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.045+0000 7fe65a7fc700 1 -- 172.21.15.60:0/2723758728 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1008085488 0 0) 0x7fe65c143a40 con 0x7fe65c0b4010 2022-06-29T16:06:07.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.045+0000 7fe65bfff700 1 --2- 172.21.15.60:0/2723758728 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe65c0b46c0 0x7fe65c0b4af0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 rx=0x7fe6540032f0 tx=0x7fe6540091d0).ready entity=mon.0 client_cookie=e76f1032bb6845a0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:07.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.045+0000 7fe65a7fc700 1 -- 172.21.15.60:0/2723758728 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe65400b680 con 0x7fe65c0b46c0 2022-06-29T16:06:07.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.045+0000 7fe65a7fc700 1 -- 172.21.15.60:0/2723758728 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe65400b7e0 con 0x7fe65c0b46c0 2022-06-29T16:06:07.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.045+0000 7fe65a7fc700 1 -- 172.21.15.60:0/2723758728 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe65400b950 con 0x7fe65c0b46c0 2022-06-29T16:06:07.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.049+0000 7fe66201f700 1 -- 172.21.15.60:0/2723758728 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe65c0b46c0 msgr2=0x7fe65c0b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:07.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.049+0000 7fe66201f700 1 --2- 172.21.15.60:0/2723758728 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe65c0b46c0 0x7fe65c0b4af0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 rx=0x7fe6540032f0 tx=0x7fe6540091d0).stop 2022-06-29T16:06:07.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.049+0000 7fe66201f700 1 -- 172.21.15.60:0/2723758728 shutdown_connections 2022-06-29T16:06:07.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.049+0000 7fe66201f700 1 --2- 172.21.15.60:0/2723758728 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe65c0b46c0 0x7fe65c0b4af0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:07.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.049+0000 7fe66201f700 1 --2- 172.21.15.60:0/2723758728 >> v2:172.21.15.92:3300/0 conn(0x7fe65c0ae7c0 0x7fe65c141fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:07.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.049+0000 7fe66201f700 1 -- 172.21.15.60:0/2723758728 >> 172.21.15.60:0/2723758728 conn(0x7fe65c0b9340 msgr2=0x7fe65c0b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:07.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.049+0000 7fe66201f700 1 -- 172.21.15.60:0/2723758728 shutdown_connections 2022-06-29T16:06:07.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.049+0000 7fe66201f700 1 -- 172.21.15.60:0/2723758728 wait complete. 2022-06-29T16:06:07.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.049+0000 7fe66201f700 1 Processor -- start 2022-06-29T16:06:07.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.049+0000 7fe66201f700 1 -- start start 2022-06-29T16:06:07.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.049+0000 7fe66201f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe65c0ae7c0 0x7fe65c146300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:07.063 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.049+0000 7fe66201f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe65c0b4010 0x7fe65c155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:07.063 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.049+0000 7fe66201f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe65c0b46c0 0x7fe65c0af4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:07.063 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.049+0000 7fe66201f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65c149dd0 con 0x7fe65c0b46c0 2022-06-29T16:06:07.064 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.049+0000 7fe66201f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65c1561b0 con 0x7fe65c0ae7c0 2022-06-29T16:06:07.064 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.049+0000 7fe66201f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe65c1552e0 con 0x7fe65c0b4010 2022-06-29T16:06:07.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.049+0000 7fe65bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe65c0b46c0 0x7fe65c0af4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:07.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.049+0000 7fe65bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe65c0b46c0 0x7fe65c0af4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51440/0 (socket says 172.21.15.60:51440) 2022-06-29T16:06:07.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.049+0000 7fe65bfff700 1 -- 172.21.15.60:0/1494478227 learned_addr learned my addr 172.21.15.60:0/1494478227 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:07.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.049+0000 7fe65affd700 1 --2- 172.21.15.60:0/1494478227 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe65c0b4010 0x7fe65c155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:07.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.049+0000 7fe65b7fe700 1 --2- 172.21.15.60:0/1494478227 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe65c0ae7c0 0x7fe65c146300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:07.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.049+0000 7fe65bfff700 1 -- 172.21.15.60:0/1494478227 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe65c0b4010 msgr2=0x7fe65c155cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:07.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.049+0000 7fe65bfff700 1 --2- 172.21.15.60:0/1494478227 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe65c0b4010 0x7fe65c155cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:07.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.049+0000 7fe65bfff700 1 -- 172.21.15.60:0/1494478227 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe65c0ae7c0 msgr2=0x7fe65c146300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:07.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.049+0000 7fe65bfff700 1 --2- 172.21.15.60:0/1494478227 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe65c0ae7c0 0x7fe65c146300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:07.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.049+0000 7fe65bfff700 1 -- 172.21.15.60:0/1494478227 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe654009470 con 0x7fe65c0b46c0 2022-06-29T16:06:07.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.049+0000 7fe65b7fe700 1 --2- 172.21.15.60:0/1494478227 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe65c0ae7c0 0x7fe65c146300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:06:07.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.053+0000 7fe65bfff700 1 --2- 172.21.15.60:0/1494478227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe65c0b46c0 0x7fe65c0af4e0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 rx=0x7fe6540032f0 tx=0x7fe654016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:07.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.053+0000 7fe658ff9700 1 -- 172.21.15.60:0/1494478227 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe65400b680 con 0x7fe65c0b46c0 2022-06-29T16:06:07.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.053+0000 7fe658ff9700 1 -- 172.21.15.60:0/1494478227 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6540107a0 con 0x7fe65c0b46c0 2022-06-29T16:06:07.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.053+0000 7fe66201f700 1 -- 172.21.15.60:0/1494478227 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe65c155580 con 0x7fe65c0b46c0 2022-06-29T16:06:07.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.053+0000 7fe658ff9700 1 -- 172.21.15.60:0/1494478227 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe654010b00 con 0x7fe65c0b46c0 2022-06-29T16:06:07.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.053+0000 7fe66201f700 1 -- 172.21.15.60:0/1494478227 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe65c0b2320 con 0x7fe65c0b46c0 2022-06-29T16:06:07.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.053+0000 7fe66201f700 1 -- 172.21.15.60:0/1494478227 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe65c0573d0 con 0x7fe65c0b46c0 2022-06-29T16:06:07.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.053+0000 7fe658ff9700 1 -- 172.21.15.60:0/1494478227 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe654029070 con 0x7fe65c0b46c0 2022-06-29T16:06:07.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.053+0000 7fe658ff9700 1 --2- 172.21.15.60:0/1494478227 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe640053a10 0x7fe640055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:07.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.053+0000 7fe658ff9700 1 -- 172.21.15.60:0/1494478227 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe65406cf90 con 0x7fe65c0b46c0 2022-06-29T16:06:07.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.053+0000 7fe65b7fe700 1 --2- 172.21.15.60:0/1494478227 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe640053a10 0x7fe640055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:07.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.053+0000 7fe65b7fe700 1 --2- 172.21.15.60:0/1494478227 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe640053a10 0x7fe640055ed0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7fe64c00b000 tx=0x7fe64c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:07.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.057+0000 7fe658ff9700 1 -- 172.21.15.60:0/1494478227 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe654043e10 con 0x7fe65c0b46c0 2022-06-29T16:06:07.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.229+0000 7fe66201f700 1 -- 172.21.15.60:0/1494478227 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fe65c0b2680 con 0x7fe65c0b46c0 2022-06-29T16:06:07.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.233+0000 7fe658ff9700 1 -- 172.21.15.60:0/1494478227 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v34) v1 ==== 184+0+0 (secure 0 0 0) 0x7fe654015020 con 0x7fe65c0b46c0 2022-06-29T16:06:07.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.237+0000 7fe66201f700 1 -- 172.21.15.60:0/1494478227 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe640053a10 msgr2=0x7fe640055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:07.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.237+0000 7fe66201f700 1 --2- 172.21.15.60:0/1494478227 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe640053a10 0x7fe640055ed0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7fe64c00b000 tx=0x7fe64c008040).stop 2022-06-29T16:06:07.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.237+0000 7fe66201f700 1 -- 172.21.15.60:0/1494478227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe65c0b46c0 msgr2=0x7fe65c0af4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:07.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.237+0000 7fe66201f700 1 --2- 172.21.15.60:0/1494478227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe65c0b46c0 0x7fe65c0af4e0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 rx=0x7fe6540032f0 tx=0x7fe654016000).stop 2022-06-29T16:06:07.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.237+0000 7fe66201f700 1 -- 172.21.15.60:0/1494478227 shutdown_connections 2022-06-29T16:06:07.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.237+0000 7fe66201f700 1 --2- 172.21.15.60:0/1494478227 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe640053a10 0x7fe640055ed0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:07.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.237+0000 7fe66201f700 1 --2- 172.21.15.60:0/1494478227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe65c0b46c0 0x7fe65c0af4e0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:07.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.237+0000 7fe66201f700 1 --2- 172.21.15.60:0/1494478227 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe65c0b4010 0x7fe65c155cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:07.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.237+0000 7fe66201f700 1 --2- 172.21.15.60:0/1494478227 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe65c0ae7c0 0x7fe65c146300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:07.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.237+0000 7fe66201f700 1 -- 172.21.15.60:0/1494478227 >> 172.21.15.60:0/1494478227 conn(0x7fe65c0b9340 msgr2=0x7fe65c0b4e70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:07.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.237+0000 7fe66201f700 1 -- 172.21.15.60:0/1494478227 shutdown_connections 2022-06-29T16:06:07.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.237+0000 7fe66201f700 1 -- 172.21.15.60:0/1494478227 wait complete. 2022-06-29T16:06:07.246 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.0 2022-06-29T16:06:07.254 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T16:06:07.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.357+0000 7fcbd6ddd700 1 Processor -- start 2022-06-29T16:06:07.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.357+0000 7fcbd6ddd700 1 -- start start 2022-06-29T16:06:07.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.357+0000 7fcbd6ddd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcbd00b5eb0 0x7fcbd0142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:07.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.357+0000 7fcbd6ddd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbd00b0280 0x7fcbd0146650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:07.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.357+0000 7fcbd6ddd700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbd0146fd0 con 0x7fcbd0146b40 2022-06-29T16:06:07.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.357+0000 7fcbd6ddd700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbd00b6290 con 0x7fcbd00b5eb0 2022-06-29T16:06:07.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.357+0000 7fcbd6ddd700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd00b0ab0 con 0x7fcbd00b0280 2022-06-29T16:06:07.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.357+0000 7fcbd65dc700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcbd0146b40 0x7fcbd00b2bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59534/0 (socket says 172.21.15.60:59534) 2022-06-29T16:06:07.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.357+0000 7fcbd65dc700 1 -- 172.21.15.60:0/3627533973 learned_addr learned my addr 172.21.15.60:0/3627533973 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:07.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.357+0000 7fcbd5ddb700 1 --2- 172.21.15.60:0/3627533973 >> v2:172.21.15.60:3300/0 conn(0x7fcbd00b5eb0 0x7fcbd0142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:07.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.357+0000 7fcbd55da700 1 --2- 172.21.15.60:0/3627533973 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbd00b0280 0x7fcbd0146650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:07.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.357+0000 7fcbd4dd9700 1 -- 172.21.15.60:0/3627533973 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1138786849 0 0) 0x7fcbc8001490 con 0x7fcbd0146b40 2022-06-29T16:06:07.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.357+0000 7fcbd4dd9700 1 -- 172.21.15.60:0/3627533973 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbb8003260 con 0x7fcbd0146b40 2022-06-29T16:06:07.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.357+0000 7fcbd5ddb700 1 -- 172.21.15.60:0/3627533973 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbd00b0280 msgr2=0x7fcbd0146650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:07.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.357+0000 7fcbd5ddb700 1 --2- 172.21.15.60:0/3627533973 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbd00b0280 0x7fcbd0146650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:07.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.357+0000 7fcbd4dd9700 1 -- 172.21.15.60:0/3627533973 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1986324809 0 0) 0x7fcbb8003260 con 0x7fcbd0146b40 2022-06-29T16:06:07.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.357+0000 7fcbd5ddb700 1 -- 172.21.15.60:0/3627533973 >> v1:172.21.15.60:6789/0 conn(0x7fcbd0146b40 legacy=0x7fcbd00b2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:07.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.357+0000 7fcbd5ddb700 1 -- 172.21.15.60:0/3627533973 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd01461b0 con 0x7fcbd00b5eb0 2022-06-29T16:06:07.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.357+0000 7fcbd5ddb700 1 --2- 172.21.15.60:0/3627533973 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbd00b5eb0 0x7fcbd0142a20 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 rx=0x7fcbc8001d10 tx=0x7fcbcc006df0).ready entity=mon.0 client_cookie=909c9a4170367b50 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:07.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.361+0000 7fcbd4dd9700 1 -- 172.21.15.60:0/3627533973 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbc8001b70 con 0x7fcbd00b5eb0 2022-06-29T16:06:07.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.361+0000 7fcbd4dd9700 1 -- 172.21.15.60:0/3627533973 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbcc000e90 con 0x7fcbd00b5eb0 2022-06-29T16:06:07.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.361+0000 7fcbd4dd9700 1 -- 172.21.15.60:0/3627533973 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbcc012460 con 0x7fcbd00b5eb0 2022-06-29T16:06:07.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.361+0000 7fcbd6ddd700 1 -- 172.21.15.60:0/3627533973 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbd00b5eb0 msgr2=0x7fcbd0142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:07.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.361+0000 7fcbd6ddd700 1 --2- 172.21.15.60:0/3627533973 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbd00b5eb0 0x7fcbd0142a20 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 rx=0x7fcbc8001d10 tx=0x7fcbcc006df0).stop 2022-06-29T16:06:07.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.361+0000 7fcbd6ddd700 1 -- 172.21.15.60:0/3627533973 shutdown_connections 2022-06-29T16:06:07.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.361+0000 7fcbd6ddd700 1 --2- 172.21.15.60:0/3627533973 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbd00b0280 0x7fcbd0146650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:07.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.361+0000 7fcbd6ddd700 1 --2- 172.21.15.60:0/3627533973 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbd00b5eb0 0x7fcbd0142a20 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:07.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.361+0000 7fcbd6ddd700 1 -- 172.21.15.60:0/3627533973 >> 172.21.15.60:0/3627533973 conn(0x7fcbd0062090 msgr2=0x7fcbd0062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:07.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.361+0000 7fcbd6ddd700 1 -- 172.21.15.60:0/3627533973 shutdown_connections 2022-06-29T16:06:07.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.361+0000 7fcbd6ddd700 1 -- 172.21.15.60:0/3627533973 wait complete. 2022-06-29T16:06:07.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.361+0000 7fcbd6ddd700 1 Processor -- start 2022-06-29T16:06:07.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.365+0000 7fcbd6ddd700 1 -- start start 2022-06-29T16:06:07.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.365+0000 7fcbd6ddd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbd00b0280 0x7fcbd01459f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:07.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.365+0000 7fcbd6ddd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcbd00b5eb0 0x7fcbd0140cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:07.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.365+0000 7fcbd6ddd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbd0146b40 0x7fcbd00ae380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:07.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.365+0000 7fcbd6ddd700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd00af530 con 0x7fcbd0146b40 2022-06-29T16:06:07.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.365+0000 7fcbd6ddd700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd01454a0 con 0x7fcbd00b5eb0 2022-06-29T16:06:07.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.365+0000 7fcbd6ddd700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd01456d0 con 0x7fcbd00b0280 2022-06-29T16:06:07.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.365+0000 7fcbd65dc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbd0146b40 0x7fcbd00ae380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:07.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.365+0000 7fcbd65dc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbd0146b40 0x7fcbd00ae380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51454/0 (socket says 172.21.15.60:51454) 2022-06-29T16:06:07.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.365+0000 7fcbd65dc700 1 -- 172.21.15.60:0/2475740936 learned_addr learned my addr 172.21.15.60:0/2475740936 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:07.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.365+0000 7fcbd5ddb700 1 --2- 172.21.15.60:0/2475740936 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbd00b0280 0x7fcbd01459f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:07.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.365+0000 7fcbd55da700 1 --2- 172.21.15.60:0/2475740936 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcbd00b5eb0 0x7fcbd0140cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:07.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.365+0000 7fcbd65dc700 1 -- 172.21.15.60:0/2475740936 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbd00b0280 msgr2=0x7fcbd01459f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:07.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.365+0000 7fcbd65dc700 1 --2- 172.21.15.60:0/2475740936 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbd00b0280 0x7fcbd01459f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:07.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.365+0000 7fcbd65dc700 1 -- 172.21.15.60:0/2475740936 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcbd00b5eb0 msgr2=0x7fcbd0140cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:07.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.365+0000 7fcbd65dc700 1 --2- 172.21.15.60:0/2475740936 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcbd00b5eb0 0x7fcbd0140cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:07.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.365+0000 7fcbd65dc700 1 -- 172.21.15.60:0/2475740936 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbcc00b180 con 0x7fcbd0146b40 2022-06-29T16:06:07.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.365+0000 7fcbd55da700 1 --2- 172.21.15.60:0/2475740936 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcbd00b5eb0 0x7fcbd0140cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:06:07.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.365+0000 7fcbd65dc700 1 --2- 172.21.15.60:0/2475740936 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbd0146b40 0x7fcbd00ae380 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 rx=0x7fcbc8001dd0 tx=0x7fcbc8001690).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:07.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.365+0000 7fcbbeffd700 1 -- 172.21.15.60:0/2475740936 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbc8004ab0 con 0x7fcbd0146b40 2022-06-29T16:06:07.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.365+0000 7fcbbeffd700 1 -- 172.21.15.60:0/2475740936 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbc8004c10 con 0x7fcbd0146b40 2022-06-29T16:06:07.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.365+0000 7fcbd6ddd700 1 -- 172.21.15.60:0/2475740936 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbd00aca20 con 0x7fcbd0146b40 2022-06-29T16:06:07.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.365+0000 7fcbbeffd700 1 -- 172.21.15.60:0/2475740936 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbc8007170 con 0x7fcbd0146b40 2022-06-29T16:06:07.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.365+0000 7fcbd6ddd700 1 -- 172.21.15.60:0/2475740936 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbd00ad160 con 0x7fcbd0146b40 2022-06-29T16:06:07.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.369+0000 7fcbd6ddd700 1 -- 172.21.15.60:0/2475740936 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbd00573d0 con 0x7fcbd0146b40 2022-06-29T16:06:07.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.369+0000 7fcbbeffd700 1 -- 172.21.15.60:0/2475740936 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcbc801b060 con 0x7fcbd0146b40 2022-06-29T16:06:07.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.369+0000 7fcbbeffd700 1 --2- 172.21.15.60:0/2475740936 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcbb0053a70 0x7fcbb0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:07.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.369+0000 7fcbbeffd700 1 -- 172.21.15.60:0/2475740936 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcbc806ba10 con 0x7fcbd0146b40 2022-06-29T16:06:07.382 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.369+0000 7fcbd5ddb700 1 --2- 172.21.15.60:0/2475740936 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcbb0053a70 0x7fcbb0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:07.382 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.369+0000 7fcbd5ddb700 1 --2- 172.21.15.60:0/2475740936 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcbb0053a70 0x7fcbb0055f30 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7fcbcc01a000 tx=0x7fcbcc014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:07.382 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.373+0000 7fcbbeffd700 1 -- 172.21.15.60:0/2475740936 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcbc8042930 con 0x7fcbd0146b40 2022-06-29T16:06:07.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.545+0000 7fcbd6ddd700 1 -- 172.21.15.60:0/2475740936 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fcbd0146fd0 con 0x7fcbd0146b40 2022-06-29T16:06:07.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.549+0000 7fcbbeffd700 1 -- 172.21.15.60:0/2475740936 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v35) v1 ==== 184+0+0 (secure 0 0 0) 0x7fcbd0146fd0 con 0x7fcbd0146b40 2022-06-29T16:06:07.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.553+0000 7fcbd6ddd700 1 -- 172.21.15.60:0/2475740936 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcbb0053a70 msgr2=0x7fcbb0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:07.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.553+0000 7fcbd6ddd700 1 --2- 172.21.15.60:0/2475740936 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcbb0053a70 0x7fcbb0055f30 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7fcbcc01a000 tx=0x7fcbcc014000).stop 2022-06-29T16:06:07.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.553+0000 7fcbd6ddd700 1 -- 172.21.15.60:0/2475740936 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbd0146b40 msgr2=0x7fcbd00ae380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:07.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.553+0000 7fcbd6ddd700 1 --2- 172.21.15.60:0/2475740936 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbd0146b40 0x7fcbd00ae380 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 rx=0x7fcbc8001dd0 tx=0x7fcbc8001690).stop 2022-06-29T16:06:07.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.553+0000 7fcbd6ddd700 1 -- 172.21.15.60:0/2475740936 shutdown_connections 2022-06-29T16:06:07.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.553+0000 7fcbd6ddd700 1 --2- 172.21.15.60:0/2475740936 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcbb0053a70 0x7fcbb0055f30 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:07.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.553+0000 7fcbd6ddd700 1 --2- 172.21.15.60:0/2475740936 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbd0146b40 0x7fcbd00ae380 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:07.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.553+0000 7fcbd6ddd700 1 --2- 172.21.15.60:0/2475740936 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcbd00b5eb0 0x7fcbd0140cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:07.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.553+0000 7fcbd6ddd700 1 --2- 172.21.15.60:0/2475740936 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbd00b0280 0x7fcbd01459f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:07.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.553+0000 7fcbd6ddd700 1 -- 172.21.15.60:0/2475740936 >> 172.21.15.60:0/2475740936 conn(0x7fcbd0062090 msgr2=0x7fcbd01427f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:07.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.553+0000 7fcbd6ddd700 1 -- 172.21.15.60:0/2475740936 shutdown_connections 2022-06-29T16:06:07.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.553+0000 7fcbd6ddd700 1 -- 172.21.15.60:0/2475740936 wait complete. 2022-06-29T16:06:07.563 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.1 2022-06-29T16:06:07.571 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T16:06:07.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.681+0000 7fef8f682700 1 Processor -- start 2022-06-29T16:06:07.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.681+0000 7fef8f682700 1 -- start start 2022-06-29T16:06:07.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.681+0000 7fef8f682700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fef880b4010 0x7fef881426b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:07.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.681+0000 7fef8f682700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fef880b46c0 0x7fef880b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:07.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.681+0000 7fef8f682700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef88143a40 con 0x7fef880ae7c0 2022-06-29T16:06:07.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.681+0000 7fef8f682700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef880aee60 con 0x7fef880b46c0 2022-06-29T16:06:07.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.681+0000 7fef8f682700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef880af100 con 0x7fef880b4010 2022-06-29T16:06:07.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.681+0000 7fef8dc1f700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fef880b46c0 0x7fef880b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:07.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.681+0000 7fef8dc1f700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fef880b46c0 0x7fef880b4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51460/0 (socket says 172.21.15.60:51460) 2022-06-29T16:06:07.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.681+0000 7fef8dc1f700 1 -- 172.21.15.60:0/554225631 learned_addr learned my addr 172.21.15.60:0/554225631 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:07.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.681+0000 7fef8d41e700 1 --1- 172.21.15.60:0/554225631 >> v1:172.21.15.60:6789/0 conn(0x7fef880ae7c0 0x7fef88141fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59544/0 (socket says 172.21.15.60:59544) 2022-06-29T16:06:07.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.685+0000 7fef8cc1d700 1 --2- 172.21.15.60:0/554225631 >> v2:172.21.15.92:3300/0 conn(0x7fef880b4010 0x7fef881426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:07.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.685+0000 7fef8dc1f700 1 -- 172.21.15.60:0/554225631 >> v2:172.21.15.92:3300/0 conn(0x7fef880b4010 msgr2=0x7fef881426b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:07.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.685+0000 7fef8dc1f700 1 --2- 172.21.15.60:0/554225631 >> v2:172.21.15.92:3300/0 conn(0x7fef880b4010 0x7fef881426b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:07.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.685+0000 7fef8dc1f700 1 -- 172.21.15.60:0/554225631 >> v1:172.21.15.60:6789/0 conn(0x7fef880ae7c0 legacy=0x7fef88141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:07.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.685+0000 7fef8dc1f700 1 -- 172.21.15.60:0/554225631 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef880aeba0 con 0x7fef880b46c0 2022-06-29T16:06:07.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.685+0000 7fef77fff700 1 -- 172.21.15.60:0/554225631 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 602097246 0 0) 0x7fef88143a40 con 0x7fef880ae7c0 2022-06-29T16:06:07.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.685+0000 7fef8cc1d700 1 --2- 172.21.15.60:0/554225631 >> v2:172.21.15.92:3300/0 conn(0x7fef880b4010 0x7fef881426b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:06:07.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.685+0000 7fef8dc1f700 1 --2- 172.21.15.60:0/554225631 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef880b46c0 0x7fef880b4af0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 rx=0x7fef800032f0 tx=0x7fef80009290).ready entity=mon.0 client_cookie=c743a55037291602 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:07.696 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.685+0000 7fef77fff700 1 -- 172.21.15.60:0/554225631 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef8000b870 con 0x7fef880b46c0 2022-06-29T16:06:07.696 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.685+0000 7fef77fff700 1 -- 172.21.15.60:0/554225631 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef8000b9d0 con 0x7fef880b46c0 2022-06-29T16:06:07.696 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.685+0000 7fef77fff700 1 -- 172.21.15.60:0/554225631 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef8000bb40 con 0x7fef880b46c0 2022-06-29T16:06:07.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.685+0000 7fef8f682700 1 -- 172.21.15.60:0/554225631 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef880b46c0 msgr2=0x7fef880b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:07.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.685+0000 7fef8f682700 1 --2- 172.21.15.60:0/554225631 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef880b46c0 0x7fef880b4af0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 rx=0x7fef800032f0 tx=0x7fef80009290).stop 2022-06-29T16:06:07.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.685+0000 7fef8f682700 1 -- 172.21.15.60:0/554225631 shutdown_connections 2022-06-29T16:06:07.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.685+0000 7fef8f682700 1 --2- 172.21.15.60:0/554225631 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef880b46c0 0x7fef880b4af0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:07.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.685+0000 7fef8f682700 1 --2- 172.21.15.60:0/554225631 >> v2:172.21.15.92:3300/0 conn(0x7fef880b4010 0x7fef881426b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:07.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.685+0000 7fef8f682700 1 -- 172.21.15.60:0/554225631 >> 172.21.15.60:0/554225631 conn(0x7fef880b9340 msgr2=0x7fef880b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:07.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.685+0000 7fef8f682700 1 -- 172.21.15.60:0/554225631 shutdown_connections 2022-06-29T16:06:07.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.685+0000 7fef8f682700 1 -- 172.21.15.60:0/554225631 wait complete. 2022-06-29T16:06:07.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.685+0000 7fef8f682700 1 Processor -- start 2022-06-29T16:06:07.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.685+0000 7fef8f682700 1 -- start start 2022-06-29T16:06:07.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.685+0000 7fef8f682700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef880ae7c0 0x7fef8813ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:07.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.685+0000 7fef8f682700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef880b4010 0x7fef88146840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:07.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.685+0000 7fef8f682700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef880b46c0 0x7fef880b1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:07.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.685+0000 7fef8f682700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef88145f30 con 0x7fef880ae7c0 2022-06-29T16:06:07.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.685+0000 7fef8f682700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef880b1560 con 0x7fef880b46c0 2022-06-29T16:06:07.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.685+0000 7fef8f682700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef88146f00 con 0x7fef880b4010 2022-06-29T16:06:07.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.685+0000 7fef8d41e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef880ae7c0 0x7fef8813ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:07.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.685+0000 7fef8d41e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef880ae7c0 0x7fef8813ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51466/0 (socket says 172.21.15.60:51466) 2022-06-29T16:06:07.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.685+0000 7fef8d41e700 1 -- 172.21.15.60:0/4180868627 learned_addr learned my addr 172.21.15.60:0/4180868627 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:07.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.685+0000 7fef8cc1d700 1 --2- 172.21.15.60:0/4180868627 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef880b4010 0x7fef88146840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:07.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.685+0000 7fef8dc1f700 1 --2- 172.21.15.60:0/4180868627 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef880b46c0 0x7fef880b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:07.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.689+0000 7fef8d41e700 1 -- 172.21.15.60:0/4180868627 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef880b4010 msgr2=0x7fef88146840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:07.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.689+0000 7fef8d41e700 1 --2- 172.21.15.60:0/4180868627 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef880b4010 0x7fef88146840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:07.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.689+0000 7fef8d41e700 1 -- 172.21.15.60:0/4180868627 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef880b46c0 msgr2=0x7fef880b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:07.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.689+0000 7fef8d41e700 1 --2- 172.21.15.60:0/4180868627 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef880b46c0 0x7fef880b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:07.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.689+0000 7fef8d41e700 1 -- 172.21.15.60:0/4180868627 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef80009530 con 0x7fef880ae7c0 2022-06-29T16:06:07.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.689+0000 7fef8d41e700 1 --2- 172.21.15.60:0/4180868627 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef880ae7c0 0x7fef8813ca50 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 rx=0x7fef88146070 tx=0x7fef84007ac0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:07.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.689+0000 7fef767fc700 1 -- 172.21.15.60:0/4180868627 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef840070d0 con 0x7fef880ae7c0 2022-06-29T16:06:07.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.689+0000 7fef767fc700 1 -- 172.21.15.60:0/4180868627 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef84007230 con 0x7fef880ae7c0 2022-06-29T16:06:07.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.689+0000 7fef8f682700 1 -- 172.21.15.60:0/4180868627 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef88144ca0 con 0x7fef880ae7c0 2022-06-29T16:06:07.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.689+0000 7fef767fc700 1 -- 172.21.15.60:0/4180868627 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef84005740 con 0x7fef880ae7c0 2022-06-29T16:06:07.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.689+0000 7fef8f682700 1 -- 172.21.15.60:0/4180868627 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef881448a0 con 0x7fef880ae7c0 2022-06-29T16:06:07.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.689+0000 7fef767fc700 1 -- 172.21.15.60:0/4180868627 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fef84010070 con 0x7fef880ae7c0 2022-06-29T16:06:07.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.689+0000 7fef8f682700 1 -- 172.21.15.60:0/4180868627 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef88146d30 con 0x7fef880ae7c0 2022-06-29T16:06:07.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.689+0000 7fef767fc700 1 --2- 172.21.15.60:0/4180868627 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fef68053b00 0x7fef68055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:07.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.689+0000 7fef767fc700 1 -- 172.21.15.60:0/4180868627 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fef8406b660 con 0x7fef880ae7c0 2022-06-29T16:06:07.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.689+0000 7fef8cc1d700 1 --2- 172.21.15.60:0/4180868627 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fef68053b00 0x7fef68055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:07.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.693+0000 7fef8cc1d700 1 --2- 172.21.15.60:0/4180868627 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fef68053b00 0x7fef68055fc0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7fef7800c000 tx=0x7fef78008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:07.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.701+0000 7fef767fc700 1 -- 172.21.15.60:0/4180868627 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fef840424e0 con 0x7fef880ae7c0 2022-06-29T16:06:07.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.869+0000 7fef8f682700 1 -- 172.21.15.60:0/4180868627 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fef88145c20 con 0x7fef880ae7c0 2022-06-29T16:06:07.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.877+0000 7fef767fc700 1 -- 172.21.15.60:0/4180868627 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v36) v1 ==== 184+0+0 (secure 0 0 0) 0x7fef84045b00 con 0x7fef880ae7c0 2022-06-29T16:06:07.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.881+0000 7fef8f682700 1 -- 172.21.15.60:0/4180868627 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fef68053b00 msgr2=0x7fef68055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:07.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.881+0000 7fef8f682700 1 --2- 172.21.15.60:0/4180868627 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fef68053b00 0x7fef68055fc0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7fef7800c000 tx=0x7fef78008040).stop 2022-06-29T16:06:07.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.881+0000 7fef8f682700 1 -- 172.21.15.60:0/4180868627 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef880ae7c0 msgr2=0x7fef8813ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:07.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.881+0000 7fef8f682700 1 --2- 172.21.15.60:0/4180868627 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef880ae7c0 0x7fef8813ca50 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 rx=0x7fef88146070 tx=0x7fef84007ac0).stop 2022-06-29T16:06:07.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.881+0000 7fef8f682700 1 -- 172.21.15.60:0/4180868627 shutdown_connections 2022-06-29T16:06:07.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.881+0000 7fef8f682700 1 --2- 172.21.15.60:0/4180868627 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fef68053b00 0x7fef68055fc0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:07.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.881+0000 7fef8f682700 1 --2- 172.21.15.60:0/4180868627 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef880b46c0 0x7fef880b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:07.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.881+0000 7fef8f682700 1 --2- 172.21.15.60:0/4180868627 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef880b4010 0x7fef88146840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:07.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.881+0000 7fef8f682700 1 --2- 172.21.15.60:0/4180868627 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef880ae7c0 0x7fef8813ca50 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:07.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.881+0000 7fef8f682700 1 -- 172.21.15.60:0/4180868627 >> 172.21.15.60:0/4180868627 conn(0x7fef880b9340 msgr2=0x7fef880b9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:07.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.881+0000 7fef8f682700 1 -- 172.21.15.60:0/4180868627 shutdown_connections 2022-06-29T16:06:07.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.881+0000 7fef8f682700 1 -- 172.21.15.60:0/4180868627 wait complete. 2022-06-29T16:06:07.888 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.2 2022-06-29T16:06:07.898 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:06:07.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.993+0000 7f9701bf6700 1 Processor -- start 2022-06-29T16:06:07.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.993+0000 7f9701bf6700 1 -- start start 2022-06-29T16:06:08.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.993+0000 7f9701bf6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96fc05d070 0x7f96fc14aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:08.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.993+0000 7f9701bf6700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f96fc0b2130 0x7f96fc0aca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:08.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.993+0000 7f9701bf6700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96fc05d450 con 0x7f96fc14af80 2022-06-29T16:06:08.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.993+0000 7f9701bf6700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96fc0adf90 con 0x7f96fc0b2130 2022-06-29T16:06:08.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.993+0000 7f9701bf6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96fc0b25d0 con 0x7f96fc05d070 2022-06-29T16:06:08.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.993+0000 7f97013f5700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f96fc14af80 0x7f96fc0acf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59562/0 (socket says 172.21.15.60:59562) 2022-06-29T16:06:08.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.993+0000 7f97013f5700 1 -- 172.21.15.60:0/3334169679 learned_addr learned my addr 172.21.15.60:0/3334169679 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:08.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.993+0000 7f96fbfff700 1 --2- 172.21.15.60:0/3334169679 >> v2:172.21.15.92:3300/0 conn(0x7f96fc0b2130 0x7f96fc0aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:08.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.993+0000 7f9700bf4700 1 --2- 172.21.15.60:0/3334169679 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96fc05d070 0x7f96fc14aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:08.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.993+0000 7f96fb7fe700 1 -- 172.21.15.60:0/3334169679 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 423052716 0 0) 0x7f96fc05d450 con 0x7f96fc14af80 2022-06-29T16:06:08.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.993+0000 7f96fb7fe700 1 -- 172.21.15.60:0/3334169679 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96dc003260 con 0x7f96fc14af80 2022-06-29T16:06:08.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.997+0000 7f96fbfff700 1 -- 172.21.15.60:0/3334169679 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96fc05d070 msgr2=0x7f96fc14aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:08.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.997+0000 7f96fbfff700 1 --2- 172.21.15.60:0/3334169679 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96fc05d070 0x7f96fc14aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:08.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.997+0000 7f96fbfff700 1 -- 172.21.15.60:0/3334169679 >> v1:172.21.15.60:6789/0 conn(0x7f96fc14af80 legacy=0x7f96fc0acf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:08.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.997+0000 7f96fbfff700 1 -- 172.21.15.60:0/3334169679 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96fc14a350 con 0x7f96fc0b2130 2022-06-29T16:06:08.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.997+0000 7f96fbfff700 1 --2- 172.21.15.60:0/3334169679 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96fc0b2130 0x7f96fc0aca20 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f96e4002f70 tx=0x7f96e4006df0).ready entity=mon.1 client_cookie=3fdfd94782e56ba2 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:08.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.997+0000 7f96fb7fe700 1 -- 172.21.15.60:0/3334169679 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f96e400d4a0 con 0x7f96fc0b2130 2022-06-29T16:06:08.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.997+0000 7f96fb7fe700 1 -- 172.21.15.60:0/3334169679 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96e4000e90 con 0x7f96fc0b2130 2022-06-29T16:06:08.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.997+0000 7f96fb7fe700 1 -- 172.21.15.60:0/3334169679 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f96e400d7f0 con 0x7f96fc0b2130 2022-06-29T16:06:08.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.997+0000 7f9701bf6700 1 -- 172.21.15.60:0/3334169679 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96fc0b2130 msgr2=0x7f96fc0aca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:08.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.997+0000 7f9701bf6700 1 --2- 172.21.15.60:0/3334169679 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96fc0b2130 0x7f96fc0aca20 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f96e4002f70 tx=0x7f96e4006df0).stop 2022-06-29T16:06:08.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.997+0000 7f9701bf6700 1 -- 172.21.15.60:0/3334169679 shutdown_connections 2022-06-29T16:06:08.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.997+0000 7f9701bf6700 1 --2- 172.21.15.60:0/3334169679 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96fc0b2130 0x7f96fc0aca20 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:08.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.997+0000 7f9701bf6700 1 --2- 172.21.15.60:0/3334169679 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96fc05d070 0x7f96fc14aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:08.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.997+0000 7f9701bf6700 1 -- 172.21.15.60:0/3334169679 >> 172.21.15.60:0/3334169679 conn(0x7f96fc0b5890 msgr2=0x7f96fc0b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:08.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.997+0000 7f9701bf6700 1 -- 172.21.15.60:0/3334169679 shutdown_connections 2022-06-29T16:06:08.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.997+0000 7f9701bf6700 1 -- 172.21.15.60:0/3334169679 wait complete. 2022-06-29T16:06:08.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.997+0000 7f9701bf6700 1 Processor -- start 2022-06-29T16:06:08.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:07.997+0000 7f9701bf6700 1 -- start start 2022-06-29T16:06:08.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.001+0000 7f9701bf6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96fc05d070 0x7f96fc066660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:08.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.001+0000 7f9701bf6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96fc0b2130 0x7f96fc066b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:08.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.001+0000 7f9701bf6700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96fc14af80 0x7f96fc143420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:08.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.001+0000 7f9701bf6700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96fc064b90 con 0x7f96fc0b2130 2022-06-29T16:06:08.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.001+0000 7f9701bf6700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96fc14d180 con 0x7f96fc14af80 2022-06-29T16:06:08.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.001+0000 7f9701bf6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96fc14cee0 con 0x7f96fc05d070 2022-06-29T16:06:08.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.001+0000 7f96fbfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96fc0b2130 0x7f96fc066b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:08.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.001+0000 7f96fbfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96fc0b2130 0x7f96fc066b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51480/0 (socket says 172.21.15.60:51480) 2022-06-29T16:06:08.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.001+0000 7f96fbfff700 1 -- 172.21.15.60:0/519791909 learned_addr learned my addr 172.21.15.60:0/519791909 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:08.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.001+0000 7f9700bf4700 1 --2- 172.21.15.60:0/519791909 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96fc05d070 0x7f96fc066660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:08.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.001+0000 7f97013f5700 1 --2- 172.21.15.60:0/519791909 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96fc14af80 0x7f96fc143420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:08.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.001+0000 7f96fbfff700 1 -- 172.21.15.60:0/519791909 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96fc05d070 msgr2=0x7f96fc066660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:08.016 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.001+0000 7f96fbfff700 1 --2- 172.21.15.60:0/519791909 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96fc05d070 0x7f96fc066660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:08.016 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.001+0000 7f96fbfff700 1 -- 172.21.15.60:0/519791909 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96fc14af80 msgr2=0x7f96fc143420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:08.016 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.001+0000 7f96fbfff700 1 --2- 172.21.15.60:0/519791909 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96fc14af80 0x7f96fc143420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:08.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.001+0000 7f96fbfff700 1 -- 172.21.15.60:0/519791909 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e400b180 con 0x7f96fc0b2130 2022-06-29T16:06:08.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.001+0000 7f96fbfff700 1 --2- 172.21.15.60:0/519791909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96fc0b2130 0x7f96fc066b50 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 rx=0x7f96e4002f70 tx=0x7f96e4014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:08.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.001+0000 7f96f9ffb700 1 -- 172.21.15.60:0/519791909 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f96e4016070 con 0x7f96fc0b2130 2022-06-29T16:06:08.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.001+0000 7f96f9ffb700 1 -- 172.21.15.60:0/519791909 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96e4013a40 con 0x7f96fc0b2130 2022-06-29T16:06:08.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.001+0000 7f9701bf6700 1 -- 172.21.15.60:0/519791909 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96fc063da0 con 0x7f96fc0b2130 2022-06-29T16:06:08.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.001+0000 7f96f9ffb700 1 -- 172.21.15.60:0/519791909 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f96e4013da0 con 0x7f96fc0b2130 2022-06-29T16:06:08.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.001+0000 7f9701bf6700 1 -- 172.21.15.60:0/519791909 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96fc063260 con 0x7f96fc0b2130 2022-06-29T16:06:08.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.001+0000 7f9701bf6700 1 -- 172.21.15.60:0/519791909 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96fc14a5f0 con 0x7f96fc0b2130 2022-06-29T16:06:08.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.001+0000 7f96f9ffb700 1 -- 172.21.15.60:0/519791909 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f96e400f030 con 0x7f96fc0b2130 2022-06-29T16:06:08.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.005+0000 7f96f9ffb700 1 --2- 172.21.15.60:0/519791909 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f96e0053a70 0x7f96e0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:08.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.005+0000 7f96f9ffb700 1 -- 172.21.15.60:0/519791909 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f96e406cc50 con 0x7f96fc0b2130 2022-06-29T16:06:08.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.005+0000 7f9700bf4700 1 --2- 172.21.15.60:0/519791909 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f96e0053a70 0x7f96e0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:08.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.005+0000 7f9700bf4700 1 --2- 172.21.15.60:0/519791909 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f96e0053a70 0x7f96e0055f30 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7f96ec00b000 tx=0x7f96ec006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:08.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.009+0000 7f96f9ffb700 1 -- 172.21.15.60:0/519791909 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f96e4043ad0 con 0x7f96fc0b2130 2022-06-29T16:06:08.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.173+0000 7f9701bf6700 1 -- 172.21.15.60:0/519791909 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96fc064cd0 con 0x7f96fc0b2130 2022-06-29T16:06:08.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.173+0000 7f96f9ffb700 1 -- 172.21.15.60:0/519791909 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f96e4023020 con 0x7f96fc0b2130 2022-06-29T16:06:08.180 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:08.180 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":46,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7320,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/3498160620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3498160620},{"type":"v1","addr":"172.21.15.60:6833","nonce":3498160620}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:06:05.226825+0000","modified":"2022-06-29T16:06:06.241973+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7327},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7327":{"gid":7327,"name":"b","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/1551394182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1551394182},{"type":"v1","addr":"172.21.15.92:6835","nonce":1551394182}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2022-06-29T16:06:08.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.177+0000 7f9701bf6700 1 -- 172.21.15.60:0/519791909 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f96e0053a70 msgr2=0x7f96e0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:08.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.177+0000 7f9701bf6700 1 --2- 172.21.15.60:0/519791909 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f96e0053a70 0x7f96e0055f30 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7f96ec00b000 tx=0x7f96ec006df0).stop 2022-06-29T16:06:08.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.177+0000 7f9701bf6700 1 -- 172.21.15.60:0/519791909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96fc0b2130 msgr2=0x7f96fc066b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:08.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.177+0000 7f9701bf6700 1 --2- 172.21.15.60:0/519791909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96fc0b2130 0x7f96fc066b50 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 rx=0x7f96e4002f70 tx=0x7f96e4014000).stop 2022-06-29T16:06:08.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.177+0000 7f9701bf6700 1 -- 172.21.15.60:0/519791909 shutdown_connections 2022-06-29T16:06:08.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.177+0000 7f9701bf6700 1 --2- 172.21.15.60:0/519791909 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f96e0053a70 0x7f96e0055f30 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:08.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.177+0000 7f9701bf6700 1 --2- 172.21.15.60:0/519791909 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96fc14af80 0x7f96fc143420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:08.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.177+0000 7f9701bf6700 1 --2- 172.21.15.60:0/519791909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96fc0b2130 0x7f96fc066b50 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:08.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.177+0000 7f9701bf6700 1 --2- 172.21.15.60:0/519791909 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f96fc05d070 0x7f96fc066660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:08.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.177+0000 7f9701bf6700 1 -- 172.21.15.60:0/519791909 >> 172.21.15.60:0/519791909 conn(0x7f96fc0b5890 msgr2=0x7f96fc0af720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:08.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.177+0000 7f9701bf6700 1 -- 172.21.15.60:0/519791909 shutdown_connections 2022-06-29T16:06:08.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.177+0000 7f9701bf6700 1 -- 172.21.15.60:0/519791909 wait complete. 2022-06-29T16:06:08.185 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 46 2022-06-29T16:06:08.394 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 46, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:06:05.226825+0000', 'modified': '2022-06-29T16:06:06.241973+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7327}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7327': {'gid': 7327, 'name': 'b', 'rank': 0, 'incarnation': 45, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.92:6835/1551394182', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 1551394182}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 1551394182}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:06:08.395 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:06:08.395 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-06-29T16:06:08.511 INFO:teuthology.orchestra.run.smithi092.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":7327,"want_state":"up:active","state":"up:active","rank_uptime":3.2673410440000001,"mdsmap_epoch":46,"osdmap_epoch":100,"osdmap_epoch_barrier":100,"uptime":7.6440586789999996} 2022-06-29T16:06:08.512 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 7327, "mdsmap_epoch": 46, "osdmap_epoch": 100, "osdmap_epoch_barrier": 100, "rank_uptime": 3.267341044, "state": "up:active", "uptime": 7.644058679, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:06:08.512 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:06:08.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.617+0000 7f915a6d3700 1 Processor -- start 2022-06-29T16:06:08.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.617+0000 7f915a6d3700 1 -- start start 2022-06-29T16:06:08.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.617+0000 7f915a6d3700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9154063ad0 0x7f9154067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:08.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.617+0000 7f915a6d3700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f915414af10 0x7f915405c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:08.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.617+0000 7f915a6d3700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91540acba0 con 0x7f91540b5950 2022-06-29T16:06:08.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.617+0000 7f915a6d3700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f915414b3a0 con 0x7f915414af10 2022-06-29T16:06:08.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.617+0000 7f915a6d3700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9154067220 con 0x7f9154063ad0 2022-06-29T16:06:08.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.617+0000 7f91537fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9154063ad0 0x7f9154067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:08.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.617+0000 7f91537fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9154063ad0 0x7f9154067680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:36350/0 (socket says 172.21.15.60:36350) 2022-06-29T16:06:08.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.617+0000 7f91537fe700 1 -- 172.21.15.60:0/2372937351 learned_addr learned my addr 172.21.15.60:0/2372937351 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:08.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.617+0000 7f9158c70700 1 --2- 172.21.15.60:0/2372937351 >> v2:172.21.15.92:3300/0 conn(0x7f915414af10 0x7f915405c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:08.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.617+0000 7f91537fe700 1 -- 172.21.15.60:0/2372937351 >> v2:172.21.15.92:3300/0 conn(0x7f915414af10 msgr2=0x7f915405c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:08.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.617+0000 7f91537fe700 1 --2- 172.21.15.60:0/2372937351 >> v2:172.21.15.92:3300/0 conn(0x7f915414af10 0x7f915405c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:08.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.617+0000 7f91537fe700 1 -- 172.21.15.60:0/2372937351 >> v1:172.21.15.92:6789/0 conn(0x7f91540b5950 legacy=0x7f915405bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:08.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.617+0000 7f91537fe700 1 -- 172.21.15.60:0/2372937351 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91540b5d30 con 0x7f9154063ad0 2022-06-29T16:06:08.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.617+0000 7f91537fe700 1 --2- 172.21.15.60:0/2372937351 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9154063ad0 0x7f9154067680 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f913c002f70 tx=0x7f913c006eb0).ready entity=mon.2 client_cookie=ac5b515a8d0991a5 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:08.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.617+0000 7f9152ffd700 1 -- 172.21.15.60:0/2372937351 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f913c00d570 con 0x7f9154063ad0 2022-06-29T16:06:08.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.617+0000 7f9152ffd700 1 -- 172.21.15.60:0/2372937351 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f913c000e90 con 0x7f9154063ad0 2022-06-29T16:06:08.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.617+0000 7f9152ffd700 1 -- 172.21.15.60:0/2372937351 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f913c00d900 con 0x7f9154063ad0 2022-06-29T16:06:08.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.621+0000 7f915a6d3700 1 -- 172.21.15.60:0/2372937351 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9154063ad0 msgr2=0x7f9154067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:08.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.621+0000 7f915a6d3700 1 --2- 172.21.15.60:0/2372937351 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9154063ad0 0x7f9154067680 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f913c002f70 tx=0x7f913c006eb0).stop 2022-06-29T16:06:08.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.621+0000 7f915a6d3700 1 -- 172.21.15.60:0/2372937351 shutdown_connections 2022-06-29T16:06:08.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.621+0000 7f915a6d3700 1 --2- 172.21.15.60:0/2372937351 >> v2:172.21.15.92:3300/0 conn(0x7f915414af10 0x7f915405c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:08.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.621+0000 7f915a6d3700 1 --2- 172.21.15.60:0/2372937351 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9154063ad0 0x7f9154067680 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:08.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.621+0000 7f915a6d3700 1 -- 172.21.15.60:0/2372937351 >> 172.21.15.60:0/2372937351 conn(0x7f91540632c0 msgr2=0x7f91540b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:08.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.621+0000 7f915a6d3700 1 -- 172.21.15.60:0/2372937351 shutdown_connections 2022-06-29T16:06:08.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.621+0000 7f915a6d3700 1 -- 172.21.15.60:0/2372937351 wait complete. 2022-06-29T16:06:08.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.621+0000 7f915a6d3700 1 Processor -- start 2022-06-29T16:06:08.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.621+0000 7f915a6d3700 1 -- start start 2022-06-29T16:06:08.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.621+0000 7f915a6d3700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9154063ad0 0x7f91540b0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:08.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.621+0000 7f915a6d3700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91540b5950 0x7f91541472b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:08.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.621+0000 7f915a6d3700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f915414af10 0x7f91541423b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:08.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.621+0000 7f915a6d3700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9154117660 con 0x7f9154063ad0 2022-06-29T16:06:08.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.621+0000 7f915a6d3700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9154118290 con 0x7f915414af10 2022-06-29T16:06:08.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.621+0000 7f915a6d3700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9154118530 con 0x7f91540b5950 2022-06-29T16:06:08.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.621+0000 7f9153fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9154063ad0 0x7f91540b0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:08.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.621+0000 7f9153fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9154063ad0 0x7f91540b0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51492/0 (socket says 172.21.15.60:51492) 2022-06-29T16:06:08.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.621+0000 7f9153fff700 1 -- 172.21.15.60:0/300160257 learned_addr learned my addr 172.21.15.60:0/300160257 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:08.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.621+0000 7f91537fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91540b5950 0x7f91541472b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:08.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.621+0000 7f9158c70700 1 --2- 172.21.15.60:0/300160257 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f915414af10 0x7f91541423b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:08.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.625+0000 7f9153fff700 1 -- 172.21.15.60:0/300160257 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91540b5950 msgr2=0x7f91541472b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:08.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.625+0000 7f9153fff700 1 --2- 172.21.15.60:0/300160257 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91540b5950 0x7f91541472b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:08.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.625+0000 7f9153fff700 1 -- 172.21.15.60:0/300160257 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f915414af10 msgr2=0x7f91541423b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:08.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.625+0000 7f9153fff700 1 --2- 172.21.15.60:0/300160257 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f915414af10 0x7f91541423b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:08.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.625+0000 7f9153fff700 1 -- 172.21.15.60:0/300160257 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913c00b1e0 con 0x7f9154063ad0 2022-06-29T16:06:08.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.625+0000 7f9153fff700 1 --2- 172.21.15.60:0/300160257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9154063ad0 0x7f91540b0620 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 rx=0x7f915405ff90 tx=0x7f9144007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:08.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.625+0000 7f91517fa700 1 -- 172.21.15.60:0/300160257 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9144007030 con 0x7f9154063ad0 2022-06-29T16:06:08.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.625+0000 7f91517fa700 1 -- 172.21.15.60:0/300160257 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9144007190 con 0x7f9154063ad0 2022-06-29T16:06:08.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.625+0000 7f915a6d3700 1 -- 172.21.15.60:0/300160257 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91541187d0 con 0x7f9154063ad0 2022-06-29T16:06:08.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.625+0000 7f91517fa700 1 -- 172.21.15.60:0/300160257 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9144006510 con 0x7f9154063ad0 2022-06-29T16:06:08.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.625+0000 7f915a6d3700 1 -- 172.21.15.60:0/300160257 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9154118c70 con 0x7f9154063ad0 2022-06-29T16:06:08.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.625+0000 7f915a6d3700 1 -- 172.21.15.60:0/300160257 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9154118960 con 0x7f9154063ad0 2022-06-29T16:06:08.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.625+0000 7f91517fa700 1 -- 172.21.15.60:0/300160257 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9144010030 con 0x7f9154063ad0 2022-06-29T16:06:08.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.625+0000 7f91517fa700 1 --2- 172.21.15.60:0/300160257 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9138053aa0 0x7f9138055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:08.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.625+0000 7f91517fa700 1 -- 172.21.15.60:0/300160257 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f914406ba20 con 0x7f9154063ad0 2022-06-29T16:06:08.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.629+0000 7f91537fe700 1 --2- 172.21.15.60:0/300160257 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9138053aa0 0x7f9138055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:08.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.629+0000 7f91537fe700 1 --2- 172.21.15.60:0/300160257 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9138053aa0 0x7f9138055f60 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7f913c01c000 tx=0x7f913c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:08.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.633+0000 7f91517fa700 1 -- 172.21.15.60:0/300160257 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9144042940 con 0x7f9154063ad0 2022-06-29T16:06:08.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.765+0000 7f915a6d3700 1 -- 172.21.15.60:0/300160257 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91540573d0 con 0x7f9154063ad0 2022-06-29T16:06:08.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.765+0000 7f91517fa700 1 -- 172.21.15.60:0/300160257 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+16043 (secure 0 0 0) 0x7f9144045f60 con 0x7f9154063ad0 2022-06-29T16:06:08.771 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:08.772 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":100,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:06:06.229758+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:06:03.648757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2022-06-29T16:06:04.556403+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:06:08.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.769+0000 7f915a6d3700 1 -- 172.21.15.60:0/300160257 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9138053aa0 msgr2=0x7f9138055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:08.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.769+0000 7f915a6d3700 1 --2- 172.21.15.60:0/300160257 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9138053aa0 0x7f9138055f60 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7f913c01c000 tx=0x7f913c015000).stop 2022-06-29T16:06:08.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.769+0000 7f915a6d3700 1 -- 172.21.15.60:0/300160257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9154063ad0 msgr2=0x7f91540b0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:08.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.769+0000 7f915a6d3700 1 --2- 172.21.15.60:0/300160257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9154063ad0 0x7f91540b0620 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 rx=0x7f915405ff90 tx=0x7f9144007c10).stop 2022-06-29T16:06:08.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.769+0000 7f915a6d3700 1 -- 172.21.15.60:0/300160257 shutdown_connections 2022-06-29T16:06:08.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.769+0000 7f915a6d3700 1 --2- 172.21.15.60:0/300160257 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9138053aa0 0x7f9138055f60 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:08.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.769+0000 7f915a6d3700 1 --2- 172.21.15.60:0/300160257 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f915414af10 0x7f91541423b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:08.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.769+0000 7f915a6d3700 1 --2- 172.21.15.60:0/300160257 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91540b5950 0x7f91541472b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:08.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.769+0000 7f915a6d3700 1 --2- 172.21.15.60:0/300160257 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9154063ad0 0x7f91540b0620 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:08.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.769+0000 7f915a6d3700 1 -- 172.21.15.60:0/300160257 >> 172.21.15.60:0/300160257 conn(0x7f91540632c0 msgr2=0x7f9154063f90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:08.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.769+0000 7f915a6d3700 1 -- 172.21.15.60:0/300160257 shutdown_connections 2022-06-29T16:06:08.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.769+0000 7f915a6d3700 1 -- 172.21.15.60:0/300160257 wait complete. 2022-06-29T16:06:08.785 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-29T16:06:03.648757+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2022-06-29T16:06:04.556403+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-29T16:06:08.785 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:06:08.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.893+0000 7f04907b3700 1 Processor -- start 2022-06-29T16:06:08.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.893+0000 7f04907b3700 1 -- start start 2022-06-29T16:06:08.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.893+0000 7f04907b3700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0488061d90 0x7f0488146ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:08.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.893+0000 7f04907b3700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04880b4f50 con 0x7f04880b3d30 2022-06-29T16:06:08.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.893+0000 7f04907b3700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04881461a0 con 0x7f04880b9d70 2022-06-29T16:06:08.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.893+0000 7f04907b3700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04880ba150 con 0x7f0488061d90 2022-06-29T16:06:08.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.893+0000 7f048dd4e700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f04880b3d30 0x7f0488142b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59588/0 (socket says 172.21.15.60:59588) 2022-06-29T16:06:08.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.893+0000 7f048dd4e700 1 -- 172.21.15.60:0/817435110 learned_addr learned my addr 172.21.15.60:0/817435110 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:08.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.893+0000 7f048ed50700 1 --2- 172.21.15.60:0/817435110 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0488061d90 0x7f0488146ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:08.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.893+0000 7f048d54d700 1 -- 172.21.15.60:0/817435110 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 939294069 0 0) 0x7f04880b4f50 con 0x7f04880b3d30 2022-06-29T16:06:08.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.893+0000 7f048d54d700 1 -- 172.21.15.60:0/817435110 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0470003260 con 0x7f04880b3d30 2022-06-29T16:06:08.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.893+0000 7f048d54d700 1 -- 172.21.15.60:0/817435110 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1555594934 0 0) 0x7f04881461a0 con 0x7f04880b9d70 2022-06-29T16:06:08.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.893+0000 7f048ed50700 1 -- 172.21.15.60:0/817435110 >> v1:172.21.15.92:6789/0 conn(0x7f04880b9d70 legacy=0x7f04881465d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:08.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.893+0000 7f048ed50700 1 -- 172.21.15.60:0/817435110 >> v1:172.21.15.60:6789/0 conn(0x7f04880b3d30 legacy=0x7f0488142b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:08.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.893+0000 7f048ed50700 1 -- 172.21.15.60:0/817435110 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04880b5180 con 0x7f0488061d90 2022-06-29T16:06:08.903 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.893+0000 7f048ed50700 1 --2- 172.21.15.60:0/817435110 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0488061d90 0x7f0488146ce0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f0478001800 tx=0x7f04800091d0).ready entity=mon.2 client_cookie=d1935dc2fd974ae server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:08.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.893+0000 7f048d54d700 1 -- 172.21.15.60:0/817435110 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0478001660 con 0x7f0488061d90 2022-06-29T16:06:08.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.893+0000 7f048d54d700 1 -- 172.21.15.60:0/817435110 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f048000b580 con 0x7f0488061d90 2022-06-29T16:06:08.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.897+0000 7f048d54d700 1 -- 172.21.15.60:0/817435110 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f048000b6f0 con 0x7f0488061d90 2022-06-29T16:06:08.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.897+0000 7f04907b3700 1 -- 172.21.15.60:0/817435110 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0488061d90 msgr2=0x7f0488146ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:08.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.897+0000 7f04907b3700 1 --2- 172.21.15.60:0/817435110 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0488061d90 0x7f0488146ce0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f0478001800 tx=0x7f04800091d0).stop 2022-06-29T16:06:08.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.897+0000 7f04907b3700 1 -- 172.21.15.60:0/817435110 shutdown_connections 2022-06-29T16:06:08.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.897+0000 7f04907b3700 1 --2- 172.21.15.60:0/817435110 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0488061d90 0x7f0488146ce0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:08.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.897+0000 7f04907b3700 1 -- 172.21.15.60:0/817435110 >> 172.21.15.60:0/817435110 conn(0x7f0488062e70 msgr2=0x7f04880b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:08.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.897+0000 7f04907b3700 1 -- 172.21.15.60:0/817435110 shutdown_connections 2022-06-29T16:06:08.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.897+0000 7f04907b3700 1 -- 172.21.15.60:0/817435110 wait complete. 2022-06-29T16:06:08.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.897+0000 7f04907b3700 1 Processor -- start 2022-06-29T16:06:08.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.897+0000 7f04907b3700 1 -- start start 2022-06-29T16:06:08.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.897+0000 7f04907b3700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0488061d90 0x7f0488155a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:08.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.897+0000 7f04907b3700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04880b3d30 0x7f04880aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:08.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.897+0000 7f04907b3700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04880b9d70 0x7f048813e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:08.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.897+0000 7f04907b3700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0488145f60 con 0x7f04880b9d70 2022-06-29T16:06:08.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.897+0000 7f04907b3700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0488155310 con 0x7f04880b3d30 2022-06-29T16:06:08.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.901+0000 7f04907b3700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04881555b0 con 0x7f0488061d90 2022-06-29T16:06:08.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.901+0000 7f048ed50700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04880b9d70 0x7f048813e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:08.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.901+0000 7f048ed50700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04880b9d70 0x7f048813e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51510/0 (socket says 172.21.15.60:51510) 2022-06-29T16:06:08.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.901+0000 7f048e54f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0488061d90 0x7f0488155a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:08.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.901+0000 7f048ed50700 1 -- 172.21.15.60:0/3130973299 learned_addr learned my addr 172.21.15.60:0/3130973299 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:08.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.901+0000 7f048dd4e700 1 --2- 172.21.15.60:0/3130973299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04880b3d30 0x7f04880aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:08.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.901+0000 7f048dd4e700 1 -- 172.21.15.60:0/3130973299 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0488061d90 msgr2=0x7f0488155a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:08.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.901+0000 7f048dd4e700 1 --2- 172.21.15.60:0/3130973299 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0488061d90 0x7f0488155a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:08.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.901+0000 7f048dd4e700 1 -- 172.21.15.60:0/3130973299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04880b9d70 msgr2=0x7f048813e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:08.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.901+0000 7f048dd4e700 1 --2- 172.21.15.60:0/3130973299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04880b9d70 0x7f048813e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:08.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.901+0000 7f048dd4e700 1 -- 172.21.15.60:0/3130973299 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0480009470 con 0x7f04880b3d30 2022-06-29T16:06:08.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.901+0000 7f048dd4e700 1 --2- 172.21.15.60:0/3130973299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04880b3d30 0x7f04880aca20 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7f0478001840 tx=0x7f04780019b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:08.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.901+0000 7f04777fe700 1 -- 172.21.15.60:0/3130973299 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f047800ce00 con 0x7f04880b3d30 2022-06-29T16:06:08.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.901+0000 7f04907b3700 1 -- 172.21.15.60:0/3130973299 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04881445c0 con 0x7f04880b3d30 2022-06-29T16:06:08.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.901+0000 7f04777fe700 1 -- 172.21.15.60:0/3130973299 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0478007170 con 0x7f04880b3d30 2022-06-29T16:06:08.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.901+0000 7f04777fe700 1 -- 172.21.15.60:0/3130973299 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0478004730 con 0x7f04880b3d30 2022-06-29T16:06:08.915 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.901+0000 7f04907b3700 1 -- 172.21.15.60:0/3130973299 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04881441c0 con 0x7f04880b3d30 2022-06-29T16:06:08.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.901+0000 7f04907b3700 1 -- 172.21.15.60:0/3130973299 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04880573d0 con 0x7f04880b3d30 2022-06-29T16:06:08.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.905+0000 7f04777fe700 1 -- 172.21.15.60:0/3130973299 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f047801b070 con 0x7f04880b3d30 2022-06-29T16:06:08.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.905+0000 7f04777fe700 1 --2- 172.21.15.60:0/3130973299 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f04680539e0 0x7f0468055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:08.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.905+0000 7f04777fe700 1 -- 172.21.15.60:0/3130973299 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f047806b6f0 con 0x7f04880b3d30 2022-06-29T16:06:08.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.905+0000 7f048e54f700 1 --2- 172.21.15.60:0/3130973299 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f04680539e0 0x7f0468055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:08.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.905+0000 7f048e54f700 1 --2- 172.21.15.60:0/3130973299 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f04680539e0 0x7f0468055ea0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7f048400d000 tx=0x7f0484007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:08.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:08.913+0000 7f04777fe700 1 -- 172.21.15.60:0/3130973299 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0478042420 con 0x7f04880b3d30 2022-06-29T16:06:09.044 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.037+0000 7f04907b3700 1 -- 172.21.15.60:0/3130973299 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f0488155720 con 0x7f04880b3d30 2022-06-29T16:06:09.045 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.041+0000 7f04777fe700 1 -- 172.21.15.60:0/3130973299 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v100) v1 ==== 110+0+10 (secure 0 0 0) 0x7f0478045a40 con 0x7f04880b3d30 2022-06-29T16:06:09.045 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:06:09.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.045+0000 7f04907b3700 1 -- 172.21.15.60:0/3130973299 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f04680539e0 msgr2=0x7f0468055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:09.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.045+0000 7f04907b3700 1 --2- 172.21.15.60:0/3130973299 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f04680539e0 0x7f0468055ea0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7f048400d000 tx=0x7f0484007b90).stop 2022-06-29T16:06:09.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.045+0000 7f04907b3700 1 -- 172.21.15.60:0/3130973299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04880b3d30 msgr2=0x7f04880aca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:09.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.045+0000 7f04907b3700 1 --2- 172.21.15.60:0/3130973299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04880b3d30 0x7f04880aca20 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7f0478001840 tx=0x7f04780019b0).stop 2022-06-29T16:06:09.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.045+0000 7f04907b3700 1 -- 172.21.15.60:0/3130973299 shutdown_connections 2022-06-29T16:06:09.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.045+0000 7f04907b3700 1 --2- 172.21.15.60:0/3130973299 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f04680539e0 0x7f0468055ea0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.045+0000 7f04907b3700 1 --2- 172.21.15.60:0/3130973299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04880b9d70 0x7f048813e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.045+0000 7f04907b3700 1 --2- 172.21.15.60:0/3130973299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04880b3d30 0x7f04880aca20 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.045+0000 7f04907b3700 1 --2- 172.21.15.60:0/3130973299 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0488061d90 0x7f0488155a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.045+0000 7f04907b3700 1 -- 172.21.15.60:0/3130973299 >> 172.21.15.60:0/3130973299 conn(0x7f0488062e70 msgr2=0x7f04880b1d10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:09.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.045+0000 7f04907b3700 1 -- 172.21.15.60:0/3130973299 shutdown_connections 2022-06-29T16:06:09.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.045+0000 7f04907b3700 1 -- 172.21.15.60:0/3130973299 wait complete. 2022-06-29T16:06:09.065 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:06:09.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.169+0000 7f07ba9ca700 1 Processor -- start 2022-06-29T16:06:09.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.169+0000 7f07ba9ca700 1 -- start start 2022-06-29T16:06:09.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.169+0000 7f07ba9ca700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07b40b5eb0 0x7f07b4142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:09.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.169+0000 7f07ba9ca700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f07b40b2c50 0x7f07b40b3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:09.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.169+0000 7f07ba9ca700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b40b0cf0 con 0x7f07b40b48b0 2022-06-29T16:06:09.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.169+0000 7f07ba9ca700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07b40b54e0 con 0x7f07b40b2c50 2022-06-29T16:06:09.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.169+0000 7f07ba9ca700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07b40b4d10 con 0x7f07b40b5eb0 2022-06-29T16:06:09.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.169+0000 7f07ba1c9700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f07b40b2c50 0x7f07b40b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:09.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.169+0000 7f07ba1c9700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f07b40b2c50 0x7f07b40b3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51518/0 (socket says 172.21.15.60:51518) 2022-06-29T16:06:09.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.169+0000 7f07b99c8700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07b40b5eb0 0x7f07b4142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:09.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.169+0000 7f07ba1c9700 1 -- 172.21.15.60:0/2992568171 learned_addr learned my addr 172.21.15.60:0/2992568171 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:09.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.169+0000 7f07ba1c9700 1 -- 172.21.15.60:0/2992568171 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07b40b5eb0 msgr2=0x7f07b4142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:09.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.169+0000 7f07ba1c9700 1 --2- 172.21.15.60:0/2992568171 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07b40b5eb0 0x7f07b4142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.169+0000 7f07ba1c9700 1 -- 172.21.15.60:0/2992568171 >> v1:172.21.15.92:6789/0 conn(0x7f07b40b48b0 legacy=0x7f07b40b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:09.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.169+0000 7f07ba1c9700 1 -- 172.21.15.60:0/2992568171 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b40b6290 con 0x7f07b40b2c50 2022-06-29T16:06:09.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.169+0000 7f07ba1c9700 1 --2- 172.21.15.60:0/2992568171 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07b40b2c50 0x7f07b40b3090 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 rx=0x7f07ac002f70 tx=0x7f07ac006eb0).ready entity=mon.0 client_cookie=124d0b68b71c5bc7 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:09.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.169+0000 7f07b89c6700 1 -- 172.21.15.60:0/2992568171 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07ac00d570 con 0x7f07b40b2c50 2022-06-29T16:06:09.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.169+0000 7f07b89c6700 1 -- 172.21.15.60:0/2992568171 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07ac000e90 con 0x7f07b40b2c50 2022-06-29T16:06:09.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.169+0000 7f07b89c6700 1 -- 172.21.15.60:0/2992568171 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07ac00d900 con 0x7f07b40b2c50 2022-06-29T16:06:09.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.173+0000 7f07ba9ca700 1 -- 172.21.15.60:0/2992568171 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07b40b2c50 msgr2=0x7f07b40b3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:09.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.173+0000 7f07ba9ca700 1 --2- 172.21.15.60:0/2992568171 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07b40b2c50 0x7f07b40b3090 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 rx=0x7f07ac002f70 tx=0x7f07ac006eb0).stop 2022-06-29T16:06:09.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.173+0000 7f07ba9ca700 1 -- 172.21.15.60:0/2992568171 shutdown_connections 2022-06-29T16:06:09.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.173+0000 7f07ba9ca700 1 --2- 172.21.15.60:0/2992568171 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07b40b2c50 0x7f07b40b3090 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.173+0000 7f07ba9ca700 1 --2- 172.21.15.60:0/2992568171 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07b40b5eb0 0x7f07b4142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.173+0000 7f07ba9ca700 1 -- 172.21.15.60:0/2992568171 >> 172.21.15.60:0/2992568171 conn(0x7f07b4062090 msgr2=0x7f07b4062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:09.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.173+0000 7f07ba9ca700 1 -- 172.21.15.60:0/2992568171 shutdown_connections 2022-06-29T16:06:09.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.173+0000 7f07ba9ca700 1 -- 172.21.15.60:0/2992568171 wait complete. 2022-06-29T16:06:09.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.173+0000 7f07ba9ca700 1 Processor -- start 2022-06-29T16:06:09.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.173+0000 7f07ba9ca700 1 -- start start 2022-06-29T16:06:09.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.173+0000 7f07ba9ca700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07b40b2c50 0x7f07b4141720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:09.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.173+0000 7f07ba9ca700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07b40b48b0 0x7f07b41521f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:09.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.173+0000 7f07ba9ca700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f07b40b5eb0 0x7f07b4155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:09.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.173+0000 7f07ba9ca700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b40b1dd0 con 0x7f07b40b48b0 2022-06-29T16:06:09.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.173+0000 7f07ba9ca700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b4155860 con 0x7f07b40b5eb0 2022-06-29T16:06:09.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.173+0000 7f07ba9ca700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07b40b1ff0 con 0x7f07b40b2c50 2022-06-29T16:06:09.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.173+0000 7f07b91c7700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07b40b48b0 0x7f07b41521f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:09.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.173+0000 7f07b99c8700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07b40b2c50 0x7f07b4141720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:09.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.173+0000 7f07b91c7700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07b40b48b0 0x7f07b41521f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51522/0 (socket says 172.21.15.60:51522) 2022-06-29T16:06:09.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.173+0000 7f07b99c8700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07b40b2c50 0x7f07b4141720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:36382/0 (socket says 172.21.15.60:36382) 2022-06-29T16:06:09.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.177+0000 7f07b91c7700 1 -- 172.21.15.60:0/948666069 learned_addr learned my addr 172.21.15.60:0/948666069 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:09.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.177+0000 7f07ba1c9700 1 --2- 172.21.15.60:0/948666069 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f07b40b5eb0 0x7f07b4155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:09.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.177+0000 7f07b99c8700 1 -- 172.21.15.60:0/948666069 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f07b40b5eb0 msgr2=0x7f07b4155f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:09.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.177+0000 7f07b99c8700 1 --2- 172.21.15.60:0/948666069 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f07b40b5eb0 0x7f07b4155f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.177+0000 7f07b99c8700 1 -- 172.21.15.60:0/948666069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07b40b48b0 msgr2=0x7f07b41521f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:09.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.177+0000 7f07b99c8700 1 --2- 172.21.15.60:0/948666069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07b40b48b0 0x7f07b41521f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.177+0000 7f07b99c8700 1 -- 172.21.15.60:0/948666069 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ac00b1e0 con 0x7f07b40b2c50 2022-06-29T16:06:09.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.177+0000 7f07b91c7700 1 --2- 172.21.15.60:0/948666069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07b40b48b0 0x7f07b41521f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:06:09.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.177+0000 7f07b99c8700 1 --2- 172.21.15.60:0/948666069 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07b40b2c50 0x7f07b4141720 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f07b40af640 tx=0x7f07a40091d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:09.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.177+0000 7f07b2ffd700 1 -- 172.21.15.60:0/948666069 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07a400c610 con 0x7f07b40b2c50 2022-06-29T16:06:09.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.177+0000 7f07b2ffd700 1 -- 172.21.15.60:0/948666069 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07a400c770 con 0x7f07b40b2c50 2022-06-29T16:06:09.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.177+0000 7f07ba9ca700 1 -- 172.21.15.60:0/948666069 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07b4155a90 con 0x7f07b40b2c50 2022-06-29T16:06:09.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.177+0000 7f07b2ffd700 1 -- 172.21.15.60:0/948666069 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07a400cad0 con 0x7f07b40b2c50 2022-06-29T16:06:09.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.177+0000 7f07ba9ca700 1 -- 172.21.15.60:0/948666069 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07b4153320 con 0x7f07b40b2c50 2022-06-29T16:06:09.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.177+0000 7f07ba9ca700 1 -- 172.21.15.60:0/948666069 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b4155c20 con 0x7f07b40b2c50 2022-06-29T16:06:09.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.177+0000 7f07b2ffd700 1 -- 172.21.15.60:0/948666069 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f07a4022070 con 0x7f07b40b2c50 2022-06-29T16:06:09.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.181+0000 7f07b2ffd700 1 --2- 172.21.15.60:0/948666069 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0794053b00 0x7f0794055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:09.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.181+0000 7f07b2ffd700 1 -- 172.21.15.60:0/948666069 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f07a406c480 con 0x7f07b40b2c50 2022-06-29T16:06:09.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.181+0000 7f07b91c7700 1 --2- 172.21.15.60:0/948666069 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0794053b00 0x7f0794055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:09.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.181+0000 7f07b91c7700 1 --2- 172.21.15.60:0/948666069 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0794053b00 0x7f0794055fc0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7f07a800d000 tx=0x7f07a8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:09.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.189+0000 7f07b2ffd700 1 -- 172.21.15.60:0/948666069 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f07a40433a0 con 0x7f07b40b2c50 2022-06-29T16:06:09.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.317+0000 7f07ba9ca700 1 -- 172.21.15.60:0/948666069 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f07b40573d0 con 0x7f07b40b2c50 2022-06-29T16:06:09.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.317+0000 7f07b2ffd700 1 -- 172.21.15.60:0/948666069 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v100) v1 ==== 92+0+10 (secure 0 0 0) 0x7f07a40160a0 con 0x7f07b40b2c50 2022-06-29T16:06:09.324 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:06:09.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.321+0000 7f07ba9ca700 1 -- 172.21.15.60:0/948666069 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0794053b00 msgr2=0x7f0794055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:09.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.321+0000 7f07ba9ca700 1 --2- 172.21.15.60:0/948666069 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0794053b00 0x7f0794055fc0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7f07a800d000 tx=0x7f07a8009040).stop 2022-06-29T16:06:09.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.321+0000 7f07ba9ca700 1 -- 172.21.15.60:0/948666069 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07b40b2c50 msgr2=0x7f07b4141720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:09.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.321+0000 7f07ba9ca700 1 --2- 172.21.15.60:0/948666069 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07b40b2c50 0x7f07b4141720 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f07b40af640 tx=0x7f07a40091d0).stop 2022-06-29T16:06:09.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.321+0000 7f07ba9ca700 1 -- 172.21.15.60:0/948666069 shutdown_connections 2022-06-29T16:06:09.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.321+0000 7f07ba9ca700 1 --2- 172.21.15.60:0/948666069 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0794053b00 0x7f0794055fc0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.321+0000 7f07ba9ca700 1 --2- 172.21.15.60:0/948666069 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f07b40b5eb0 0x7f07b4155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.321+0000 7f07ba9ca700 1 --2- 172.21.15.60:0/948666069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07b40b48b0 0x7f07b41521f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.321+0000 7f07ba9ca700 1 --2- 172.21.15.60:0/948666069 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07b40b2c50 0x7f07b4141720 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.321+0000 7f07ba9ca700 1 -- 172.21.15.60:0/948666069 >> 172.21.15.60:0/948666069 conn(0x7f07b4062090 msgr2=0x7f07b41427f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:09.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.321+0000 7f07ba9ca700 1 -- 172.21.15.60:0/948666069 shutdown_connections 2022-06-29T16:06:09.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.321+0000 7f07ba9ca700 1 -- 172.21.15.60:0/948666069 wait complete. 2022-06-29T16:06:09.342 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-29T16:06:09.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.445+0000 7fa50c5ab700 1 Processor -- start 2022-06-29T16:06:09.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.445+0000 7fa50c5ab700 1 -- start start 2022-06-29T16:06:09.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.445+0000 7fa50c5ab700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa5040ae7c0 0x7fa504141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:09.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.445+0000 7fa50c5ab700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5040b4010 0x7fa504142490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:09.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.445+0000 7fa50c5ab700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa504143a40 con 0x7fa504146160 2022-06-29T16:06:09.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.445+0000 7fa50c5ab700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5040b56d0 con 0x7fa5040ae7c0 2022-06-29T16:06:09.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.445+0000 7fa50c5ab700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5040aee60 con 0x7fa5040b4010 2022-06-29T16:06:09.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.445+0000 7fa50a347700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa5040ae7c0 0x7fa504141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:09.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.445+0000 7fa50a347700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa5040ae7c0 0x7fa504141fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51528/0 (socket says 172.21.15.60:51528) 2022-06-29T16:06:09.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.445+0000 7fa50a347700 1 -- 172.21.15.60:0/2580549623 learned_addr learned my addr 172.21.15.60:0/2580549623 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:09.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.445+0000 7fa509b46700 1 --2- 172.21.15.60:0/2580549623 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5040b4010 0x7fa504142490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:09.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.445+0000 7fa50a347700 1 -- 172.21.15.60:0/2580549623 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5040b4010 msgr2=0x7fa504142490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:09.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.445+0000 7fa50a347700 1 --2- 172.21.15.60:0/2580549623 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5040b4010 0x7fa504142490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.445+0000 7fa50a347700 1 -- 172.21.15.60:0/2580549623 >> v1:172.21.15.92:6789/0 conn(0x7fa504146160 legacy=0x7fa5041465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:09.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.445+0000 7fa50a347700 1 -- 172.21.15.60:0/2580549623 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5040af100 con 0x7fa5040ae7c0 2022-06-29T16:06:09.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.445+0000 7fa50a347700 1 --2- 172.21.15.60:0/2580549623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5040ae7c0 0x7fa504141fa0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 rx=0x7fa5000032f0 tx=0x7fa500009290).ready entity=mon.0 client_cookie=757de77d63dbe38a server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:09.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.445+0000 7fa509345700 1 -- 172.21.15.60:0/2580549623 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa50000b8a0 con 0x7fa5040ae7c0 2022-06-29T16:06:09.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.445+0000 7fa509345700 1 -- 172.21.15.60:0/2580549623 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa50000ba00 con 0x7fa5040ae7c0 2022-06-29T16:06:09.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.445+0000 7fa509345700 1 -- 172.21.15.60:0/2580549623 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa50000bb70 con 0x7fa5040ae7c0 2022-06-29T16:06:09.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.449+0000 7fa50c5ab700 1 -- 172.21.15.60:0/2580549623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5040ae7c0 msgr2=0x7fa504141fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:09.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.449+0000 7fa50c5ab700 1 --2- 172.21.15.60:0/2580549623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5040ae7c0 0x7fa504141fa0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 rx=0x7fa5000032f0 tx=0x7fa500009290).stop 2022-06-29T16:06:09.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.449+0000 7fa50c5ab700 1 -- 172.21.15.60:0/2580549623 shutdown_connections 2022-06-29T16:06:09.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.449+0000 7fa50c5ab700 1 --2- 172.21.15.60:0/2580549623 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa5040b4010 0x7fa504142490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.449+0000 7fa50c5ab700 1 --2- 172.21.15.60:0/2580549623 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5040ae7c0 0x7fa504141fa0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.449+0000 7fa50c5ab700 1 -- 172.21.15.60:0/2580549623 >> 172.21.15.60:0/2580549623 conn(0x7fa5040b9340 msgr2=0x7fa5040b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:09.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.449+0000 7fa50c5ab700 1 -- 172.21.15.60:0/2580549623 shutdown_connections 2022-06-29T16:06:09.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.449+0000 7fa50c5ab700 1 -- 172.21.15.60:0/2580549623 wait complete. 2022-06-29T16:06:09.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.449+0000 7fa50c5ab700 1 Processor -- start 2022-06-29T16:06:09.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.449+0000 7fa50c5ab700 1 -- start start 2022-06-29T16:06:09.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.449+0000 7fa50c5ab700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa5040ae7c0 0x7fa50405c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:09.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.449+0000 7fa50c5ab700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5040b4010 0x7fa504155460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:09.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.449+0000 7fa50c5ab700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa504146160 0x7fa5040b2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:09.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.449+0000 7fa50c5ab700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5040b2b70 con 0x7fa5040b4010 2022-06-29T16:06:09.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.449+0000 7fa50c5ab700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5040b0400 con 0x7fa5040ae7c0 2022-06-29T16:06:09.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.449+0000 7fa50c5ab700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5040b06a0 con 0x7fa504146160 2022-06-29T16:06:09.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.449+0000 7fa509b46700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5040b4010 0x7fa504155460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:09.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.449+0000 7fa509b46700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5040b4010 0x7fa504155460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51536/0 (socket says 172.21.15.60:51536) 2022-06-29T16:06:09.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.449+0000 7fa509b46700 1 -- 172.21.15.60:0/2228979102 learned_addr learned my addr 172.21.15.60:0/2228979102 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:09.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.449+0000 7fa50a347700 1 --2- 172.21.15.60:0/2228979102 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa5040ae7c0 0x7fa50405c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:09.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.449+0000 7fa50ab48700 1 --2- 172.21.15.60:0/2228979102 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa504146160 0x7fa5040b2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:09.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.449+0000 7fa509b46700 1 -- 172.21.15.60:0/2228979102 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa504146160 msgr2=0x7fa5040b2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:09.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.449+0000 7fa509b46700 1 --2- 172.21.15.60:0/2228979102 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa504146160 0x7fa5040b2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.449+0000 7fa509b46700 1 -- 172.21.15.60:0/2228979102 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa5040ae7c0 msgr2=0x7fa50405c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:09.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.449+0000 7fa509b46700 1 --2- 172.21.15.60:0/2228979102 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa5040ae7c0 0x7fa50405c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.449+0000 7fa509b46700 1 -- 172.21.15.60:0/2228979102 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa500009530 con 0x7fa5040b4010 2022-06-29T16:06:09.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.453+0000 7fa50ab48700 1 --2- 172.21.15.60:0/2228979102 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa504146160 0x7fa5040b2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:06:09.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.453+0000 7fa509b46700 1 --2- 172.21.15.60:0/2228979102 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5040b4010 0x7fa504155460 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 rx=0x7fa4f40030b0 tx=0x7fa4f4008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:09.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.453+0000 7fa4f37fe700 1 -- 172.21.15.60:0/2228979102 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa4f4011780 con 0x7fa5040b4010 2022-06-29T16:06:09.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.453+0000 7fa4f37fe700 1 -- 172.21.15.60:0/2228979102 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4f40118e0 con 0x7fa5040b4010 2022-06-29T16:06:09.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.453+0000 7fa50c5ab700 1 -- 172.21.15.60:0/2228979102 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5040b0980 con 0x7fa5040b4010 2022-06-29T16:06:09.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.453+0000 7fa4f37fe700 1 -- 172.21.15.60:0/2228979102 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa4f4011c40 con 0x7fa5040b4010 2022-06-29T16:06:09.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.453+0000 7fa50c5ab700 1 -- 172.21.15.60:0/2228979102 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5040b8d20 con 0x7fa5040b4010 2022-06-29T16:06:09.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.453+0000 7fa4f37fe700 1 -- 172.21.15.60:0/2228979102 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa4f4012420 con 0x7fa5040b4010 2022-06-29T16:06:09.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.453+0000 7fa50c5ab700 1 -- 172.21.15.60:0/2228979102 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5040b1780 con 0x7fa5040b4010 2022-06-29T16:06:09.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.453+0000 7fa4f37fe700 1 --2- 172.21.15.60:0/2228979102 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa4e4053aa0 0x7fa4e4055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:09.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.453+0000 7fa4f37fe700 1 -- 172.21.15.60:0/2228979102 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa4f406bed0 con 0x7fa5040b4010 2022-06-29T16:06:09.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.453+0000 7fa50a347700 1 --2- 172.21.15.60:0/2228979102 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa4e4053aa0 0x7fa4e4055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:09.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.457+0000 7fa50a347700 1 --2- 172.21.15.60:0/2228979102 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa4e4053aa0 0x7fa4e4055f60 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7fa500019000 tx=0x7fa500016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:09.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.457+0000 7fa4f37fe700 1 -- 172.21.15.60:0/2228979102 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa4f4042c80 con 0x7fa5040b4010 2022-06-29T16:06:09.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.585+0000 7fa50c5ab700 1 -- 172.21.15.60:0/2228979102 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa5040573d0 con 0x7fa5040b4010 2022-06-29T16:06:09.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.585+0000 7fa4f37fe700 1 -- 172.21.15.60:0/2228979102 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v100) v1 ==== 104+0+10 (secure 0 0 0) 0x7fa4f40462a0 con 0x7fa5040b4010 2022-06-29T16:06:09.590 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:06:09.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.589+0000 7fa50c5ab700 1 -- 172.21.15.60:0/2228979102 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa4e4053aa0 msgr2=0x7fa4e4055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:09.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.589+0000 7fa50c5ab700 1 --2- 172.21.15.60:0/2228979102 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa4e4053aa0 0x7fa4e4055f60 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7fa500019000 tx=0x7fa500016000).stop 2022-06-29T16:06:09.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.589+0000 7fa50c5ab700 1 -- 172.21.15.60:0/2228979102 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5040b4010 msgr2=0x7fa504155460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:09.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.589+0000 7fa50c5ab700 1 --2- 172.21.15.60:0/2228979102 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5040b4010 0x7fa504155460 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 rx=0x7fa4f40030b0 tx=0x7fa4f4008040).stop 2022-06-29T16:06:09.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.589+0000 7fa50c5ab700 1 -- 172.21.15.60:0/2228979102 shutdown_connections 2022-06-29T16:06:09.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.589+0000 7fa50c5ab700 1 --2- 172.21.15.60:0/2228979102 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa4e4053aa0 0x7fa4e4055f60 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.589+0000 7fa50c5ab700 1 --2- 172.21.15.60:0/2228979102 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa504146160 0x7fa5040b2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.589+0000 7fa50c5ab700 1 --2- 172.21.15.60:0/2228979102 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa5040b4010 0x7fa504155460 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.589+0000 7fa50c5ab700 1 --2- 172.21.15.60:0/2228979102 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa5040ae7c0 0x7fa50405c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.589+0000 7fa50c5ab700 1 -- 172.21.15.60:0/2228979102 >> 172.21.15.60:0/2228979102 conn(0x7fa5040b9340 msgr2=0x7fa5041425b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:09.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.589+0000 7fa50c5ab700 1 -- 172.21.15.60:0/2228979102 shutdown_connections 2022-06-29T16:06:09.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.589+0000 7fa50c5ab700 1 -- 172.21.15.60:0/2228979102 wait complete. 2022-06-29T16:06:09.609 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-29T16:06:09.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.713+0000 7f5549ca3700 1 Processor -- start 2022-06-29T16:06:09.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.713+0000 7f5549ca3700 1 -- start start 2022-06-29T16:06:09.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.713+0000 7f5549ca3700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f55440b5950 0x7f5544067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:09.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.713+0000 7f5549ca3700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f554414af10 0x7f554405c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:09.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.713+0000 7f5549ca3700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55440acba0 con 0x7f5544063ad0 2022-06-29T16:06:09.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.713+0000 7f5549ca3700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f554414b3a0 con 0x7f554414af10 2022-06-29T16:06:09.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.713+0000 7f5549ca3700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5544067220 con 0x7f55440b5950 2022-06-29T16:06:09.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.713+0000 7f5543fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f554414af10 0x7f554405c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:09.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.713+0000 7f5543fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f554414af10 0x7f554405c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51546/0 (socket says 172.21.15.60:51546) 2022-06-29T16:06:09.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.713+0000 7f5543fff700 1 -- 172.21.15.60:0/2938080755 learned_addr learned my addr 172.21.15.60:0/2938080755 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:09.721 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.713+0000 7f55437fe700 1 --2- 172.21.15.60:0/2938080755 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f55440b5950 0x7f5544067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:09.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.713+0000 7f5543fff700 1 -- 172.21.15.60:0/2938080755 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f55440b5950 msgr2=0x7f5544067680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:09.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.713+0000 7f5543fff700 1 --2- 172.21.15.60:0/2938080755 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f55440b5950 0x7f5544067680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.713+0000 7f5543fff700 1 -- 172.21.15.60:0/2938080755 >> v1:172.21.15.92:6789/0 conn(0x7f5544063ad0 legacy=0x7f554405bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:09.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.713+0000 7f5543fff700 1 -- 172.21.15.60:0/2938080755 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55440b5d30 con 0x7f554414af10 2022-06-29T16:06:09.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.713+0000 7f5543fff700 1 --2- 172.21.15.60:0/2938080755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f554414af10 0x7f554405c430 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 rx=0x7f553c0032f0 tx=0x7f553c009290).ready entity=mon.0 client_cookie=3e5cd4aa07a97c0e server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:09.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.713+0000 7f5542ffd700 1 -- 172.21.15.60:0/2938080755 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f553c00b8a0 con 0x7f554414af10 2022-06-29T16:06:09.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.713+0000 7f5542ffd700 1 -- 172.21.15.60:0/2938080755 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f553c00ba00 con 0x7f554414af10 2022-06-29T16:06:09.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.713+0000 7f5542ffd700 1 -- 172.21.15.60:0/2938080755 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f553c00bb70 con 0x7f554414af10 2022-06-29T16:06:09.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.717+0000 7f5549ca3700 1 -- 172.21.15.60:0/2938080755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f554414af10 msgr2=0x7f554405c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:09.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.717+0000 7f5549ca3700 1 --2- 172.21.15.60:0/2938080755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f554414af10 0x7f554405c430 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 rx=0x7f553c0032f0 tx=0x7f553c009290).stop 2022-06-29T16:06:09.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.717+0000 7f5549ca3700 1 -- 172.21.15.60:0/2938080755 shutdown_connections 2022-06-29T16:06:09.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.717+0000 7f5549ca3700 1 --2- 172.21.15.60:0/2938080755 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f554414af10 0x7f554405c430 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.717+0000 7f5549ca3700 1 --2- 172.21.15.60:0/2938080755 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f55440b5950 0x7f5544067680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.717+0000 7f5549ca3700 1 -- 172.21.15.60:0/2938080755 >> 172.21.15.60:0/2938080755 conn(0x7f55440632c0 msgr2=0x7f55440b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:09.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.717+0000 7f5549ca3700 1 -- 172.21.15.60:0/2938080755 shutdown_connections 2022-06-29T16:06:09.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.717+0000 7f5549ca3700 1 -- 172.21.15.60:0/2938080755 wait complete. 2022-06-29T16:06:09.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.717+0000 7f5549ca3700 1 Processor -- start 2022-06-29T16:06:09.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.717+0000 7f5549ca3700 1 -- start start 2022-06-29T16:06:09.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.721+0000 7f5549ca3700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5544063ad0 0x7f55440b07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:09.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.721+0000 7f5549ca3700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f55440b5950 0x7f55441595b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:09.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.721+0000 7f5549ca3700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f554414af10 0x7f5544159aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:09.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.721+0000 7f5549ca3700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55440ad2a0 con 0x7f554414af10 2022-06-29T16:06:09.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.721+0000 7f5549ca3700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55440ad000 con 0x7f5544063ad0 2022-06-29T16:06:09.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.721+0000 7f5549ca3700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5544156160 con 0x7f55440b5950 2022-06-29T16:06:09.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.721+0000 7f5543fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f554414af10 0x7f5544159aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:09.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.721+0000 7f5543fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f554414af10 0x7f5544159aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51552/0 (socket says 172.21.15.60:51552) 2022-06-29T16:06:09.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.721+0000 7f5543fff700 1 -- 172.21.15.60:0/12006236 learned_addr learned my addr 172.21.15.60:0/12006236 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:09.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.721+0000 7f55437fe700 1 --2- 172.21.15.60:0/12006236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5544063ad0 0x7f55440b07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:09.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.721+0000 7f553affd700 1 --2- 172.21.15.60:0/12006236 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f55440b5950 0x7f55441595b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:09.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.721+0000 7f5543fff700 1 -- 172.21.15.60:0/12006236 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f55440b5950 msgr2=0x7f55441595b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:09.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.721+0000 7f5543fff700 1 --2- 172.21.15.60:0/12006236 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f55440b5950 0x7f55441595b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.721+0000 7f5543fff700 1 -- 172.21.15.60:0/12006236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5544063ad0 msgr2=0x7f55440b07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:09.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.721+0000 7f5543fff700 1 --2- 172.21.15.60:0/12006236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5544063ad0 0x7f55440b07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.721+0000 7f5543fff700 1 -- 172.21.15.60:0/12006236 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f553c009530 con 0x7f554414af10 2022-06-29T16:06:09.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.721+0000 7f5543fff700 1 --2- 172.21.15.60:0/12006236 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f554414af10 0x7f5544159aa0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 rx=0x7f553c0032f0 tx=0x7f553c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:09.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.721+0000 7f55417fa700 1 -- 172.21.15.60:0/12006236 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f553c00b8a0 con 0x7f554414af10 2022-06-29T16:06:09.734 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.721+0000 7f55417fa700 1 -- 172.21.15.60:0/12006236 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f553c021630 con 0x7f554414af10 2022-06-29T16:06:09.734 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.721+0000 7f5549ca3700 1 -- 172.21.15.60:0/12006236 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f554414a620 con 0x7f554414af10 2022-06-29T16:06:09.734 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.721+0000 7f55417fa700 1 -- 172.21.15.60:0/12006236 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f553c021990 con 0x7f554414af10 2022-06-29T16:06:09.734 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.721+0000 7f5549ca3700 1 -- 172.21.15.60:0/12006236 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55440ad730 con 0x7f554414af10 2022-06-29T16:06:09.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.725+0000 7f5549ca3700 1 -- 172.21.15.60:0/12006236 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55440b1000 con 0x7f554414af10 2022-06-29T16:06:09.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.725+0000 7f55417fa700 1 -- 172.21.15.60:0/12006236 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f553c028070 con 0x7f554414af10 2022-06-29T16:06:09.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.725+0000 7f55417fa700 1 --2- 172.21.15.60:0/12006236 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5528053a70 0x7f5528055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:09.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.725+0000 7f55417fa700 1 -- 172.21.15.60:0/12006236 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f553c06ce00 con 0x7f554414af10 2022-06-29T16:06:09.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.725+0000 7f55437fe700 1 --2- 172.21.15.60:0/12006236 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5528053a70 0x7f5528055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:09.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.725+0000 7f55437fe700 1 --2- 172.21.15.60:0/12006236 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5528053a70 0x7f5528055f30 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7f553400b000 tx=0x7f5534006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:09.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.733+0000 7f55417fa700 1 -- 172.21.15.60:0/12006236 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f553c043d60 con 0x7f554414af10 2022-06-29T16:06:09.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.857+0000 7f5549ca3700 1 -- 172.21.15.60:0/12006236 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5544156320 con 0x7f554414af10 2022-06-29T16:06:09.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.861+0000 7f55417fa700 1 -- 172.21.15.60:0/12006236 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v100) v1 ==== 100+0+10 (secure 0 0 0) 0x7f553c015030 con 0x7f554414af10 2022-06-29T16:06:09.866 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:06:09.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.861+0000 7f5549ca3700 1 -- 172.21.15.60:0/12006236 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5528053a70 msgr2=0x7f5528055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:09.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.861+0000 7f5549ca3700 1 --2- 172.21.15.60:0/12006236 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5528053a70 0x7f5528055f30 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7f553400b000 tx=0x7f5534006df0).stop 2022-06-29T16:06:09.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.861+0000 7f5549ca3700 1 -- 172.21.15.60:0/12006236 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f554414af10 msgr2=0x7f5544159aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:09.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.861+0000 7f5549ca3700 1 --2- 172.21.15.60:0/12006236 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f554414af10 0x7f5544159aa0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 rx=0x7f553c0032f0 tx=0x7f553c016000).stop 2022-06-29T16:06:09.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.861+0000 7f5549ca3700 1 -- 172.21.15.60:0/12006236 shutdown_connections 2022-06-29T16:06:09.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.861+0000 7f5549ca3700 1 --2- 172.21.15.60:0/12006236 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5528053a70 0x7f5528055f30 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.861+0000 7f5549ca3700 1 --2- 172.21.15.60:0/12006236 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f554414af10 0x7f5544159aa0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.861+0000 7f5549ca3700 1 --2- 172.21.15.60:0/12006236 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f55440b5950 0x7f55441595b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.861+0000 7f5549ca3700 1 --2- 172.21.15.60:0/12006236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5544063ad0 0x7f55440b07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.861+0000 7f5549ca3700 1 -- 172.21.15.60:0/12006236 >> 172.21.15.60:0/12006236 conn(0x7f55440632c0 msgr2=0x7f5544146ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:09.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.861+0000 7f5549ca3700 1 -- 172.21.15.60:0/12006236 shutdown_connections 2022-06-29T16:06:09.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.861+0000 7f5549ca3700 1 -- 172.21.15.60:0/12006236 wait complete. 2022-06-29T16:06:09.879 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-06-29T16:06:09.880 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:06:09.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.977+0000 7f0b154b8700 1 Processor -- start 2022-06-29T16:06:09.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.977+0000 7f0b154b8700 1 -- start start 2022-06-29T16:06:09.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.977+0000 7f0b154b8700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0b100b2130 0x7f0b10146100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:09.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.977+0000 7f0b154b8700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b1005d450 con 0x7f0b1005d070 2022-06-29T16:06:09.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.977+0000 7f0b154b8700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b100b25d0 con 0x7f0b100aed00 2022-06-29T16:06:09.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.977+0000 7f0b154b8700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b100adf90 con 0x7f0b100b2130 2022-06-29T16:06:09.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.977+0000 7f0b0f7fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0b100b2130 0x7f0b10146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:09.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.977+0000 7f0b0ffff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f0b1005d070 0x7f0b1014aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59642/0 (socket says 172.21.15.60:59642) 2022-06-29T16:06:09.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.977+0000 7f0b0ffff700 1 -- 172.21.15.60:0/349772618 learned_addr learned my addr 172.21.15.60:0/349772618 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:09.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.977+0000 7f0b0f7fe700 1 -- 172.21.15.60:0/349772618 >> v1:172.21.15.92:6789/0 conn(0x7f0b100aed00 legacy=0x7f0b101465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:09.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.977+0000 7f0b0f7fe700 1 -- 172.21.15.60:0/349772618 >> v1:172.21.15.60:6789/0 conn(0x7f0b1005d070 legacy=0x7f0b1014aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:09.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.977+0000 7f0b0f7fe700 1 -- 172.21.15.60:0/349772618 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b1014a350 con 0x7f0b100b2130 2022-06-29T16:06:09.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.977+0000 7f0b0f7fe700 1 --2- 172.21.15.60:0/349772618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b100b2130 0x7f0b10146100 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 rx=0x7f0af8002f70 tx=0x7f0af8006df0).ready entity=mon.0 client_cookie=551168787f820c6 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:09.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.977+0000 7f0b0effd700 1 -- 172.21.15.60:0/349772618 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0af800d610 con 0x7f0b100b2130 2022-06-29T16:06:09.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.977+0000 7f0b0effd700 1 -- 172.21.15.60:0/349772618 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0af8000e90 con 0x7f0b100b2130 2022-06-29T16:06:09.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.977+0000 7f0b0effd700 1 -- 172.21.15.60:0/349772618 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0af800d960 con 0x7f0b100b2130 2022-06-29T16:06:09.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.981+0000 7f0b154b8700 1 -- 172.21.15.60:0/349772618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b100b2130 msgr2=0x7f0b10146100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:09.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.981+0000 7f0b154b8700 1 --2- 172.21.15.60:0/349772618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b100b2130 0x7f0b10146100 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 rx=0x7f0af8002f70 tx=0x7f0af8006df0).stop 2022-06-29T16:06:09.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.981+0000 7f0b154b8700 1 -- 172.21.15.60:0/349772618 shutdown_connections 2022-06-29T16:06:09.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.981+0000 7f0b154b8700 1 --2- 172.21.15.60:0/349772618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b100b2130 0x7f0b10146100 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.981+0000 7f0b154b8700 1 -- 172.21.15.60:0/349772618 >> 172.21.15.60:0/349772618 conn(0x7f0b100b5890 msgr2=0x7f0b100b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:09.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.981+0000 7f0b154b8700 1 -- 172.21.15.60:0/349772618 shutdown_connections 2022-06-29T16:06:09.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.981+0000 7f0b154b8700 1 -- 172.21.15.60:0/349772618 wait complete. 2022-06-29T16:06:09.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.985+0000 7f0b154b8700 1 Processor -- start 2022-06-29T16:06:09.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.985+0000 7f0b154b8700 1 -- start start 2022-06-29T16:06:09.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.985+0000 7f0b154b8700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b1005d070 0x7f0b10159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:09.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.985+0000 7f0b154b8700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b100aed00 0x7f0b100b46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:09.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.985+0000 7f0b154b8700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b100b2130 0x7f0b10118af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:09.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.985+0000 7f0b154b8700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b10117a00 con 0x7f0b100b2130 2022-06-29T16:06:09.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.985+0000 7f0b154b8700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b10065b40 con 0x7f0b100aed00 2022-06-29T16:06:09.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.985+0000 7f0b154b8700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b10065de0 con 0x7f0b1005d070 2022-06-29T16:06:09.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.985+0000 7f0b14cb7700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b100b2130 0x7f0b10118af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:09.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.985+0000 7f0b14cb7700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b100b2130 0x7f0b10118af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51566/0 (socket says 172.21.15.60:51566) 2022-06-29T16:06:09.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.985+0000 7f0b14cb7700 1 -- 172.21.15.60:0/301540570 learned_addr learned my addr 172.21.15.60:0/301540570 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:09.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.985+0000 7f0b0f7fe700 1 --2- 172.21.15.60:0/301540570 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b100aed00 0x7f0b100b46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:09.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.985+0000 7f0b0ffff700 1 --2- 172.21.15.60:0/301540570 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b1005d070 0x7f0b10159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:09.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.985+0000 7f0b14cb7700 1 -- 172.21.15.60:0/301540570 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b1005d070 msgr2=0x7f0b10159ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:09.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.985+0000 7f0b14cb7700 1 --2- 172.21.15.60:0/301540570 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b1005d070 0x7f0b10159ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.985+0000 7f0b14cb7700 1 -- 172.21.15.60:0/301540570 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b100aed00 msgr2=0x7f0b100b46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:09.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.985+0000 7f0b14cb7700 1 --2- 172.21.15.60:0/301540570 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b100aed00 0x7f0b100b46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:09.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.985+0000 7f0b14cb7700 1 -- 172.21.15.60:0/301540570 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0af800b180 con 0x7f0b100b2130 2022-06-29T16:06:09.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.985+0000 7f0b14cb7700 1 --2- 172.21.15.60:0/301540570 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b100b2130 0x7f0b10118af0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 rx=0x7f0b100b5cd0 tx=0x7f0b08007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:09.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.985+0000 7f0b0d7fa700 1 -- 172.21.15.60:0/301540570 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b0800f470 con 0x7f0b100b2130 2022-06-29T16:06:09.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.985+0000 7f0b0d7fa700 1 -- 172.21.15.60:0/301540570 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b0800f5d0 con 0x7f0b100b2130 2022-06-29T16:06:09.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.985+0000 7f0b154b8700 1 -- 172.21.15.60:0/301540570 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b100656a0 con 0x7f0b100b2130 2022-06-29T16:06:09.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.985+0000 7f0b0d7fa700 1 -- 172.21.15.60:0/301540570 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b0800f930 con 0x7f0b100b2130 2022-06-29T16:06:09.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.985+0000 7f0b154b8700 1 -- 172.21.15.60:0/301540570 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b100669d0 con 0x7f0b100b2130 2022-06-29T16:06:09.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.989+0000 7f0b0d7fa700 1 -- 172.21.15.60:0/301540570 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0b08019030 con 0x7f0b100b2130 2022-06-29T16:06:10.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.989+0000 7f0b154b8700 1 -- 172.21.15.60:0/301540570 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b1014a5f0 con 0x7f0b100b2130 2022-06-29T16:06:10.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.989+0000 7f0b0d7fa700 1 --2- 172.21.15.60:0/301540570 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0af4053b00 0x7f0af4055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:10.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.989+0000 7f0b0d7fa700 1 -- 172.21.15.60:0/301540570 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0b0806b630 con 0x7f0b100b2130 2022-06-29T16:06:10.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.989+0000 7f0b0ffff700 1 --2- 172.21.15.60:0/301540570 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0af4053b00 0x7f0af4055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:10.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.989+0000 7f0b0ffff700 1 --2- 172.21.15.60:0/301540570 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0af4053b00 0x7f0af4055fc0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7f0b0000d000 tx=0x7f0b00009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:10.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:09.997+0000 7f0b0d7fa700 1 -- 172.21.15.60:0/301540570 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0b08042440 con 0x7f0b100b2130 2022-06-29T16:06:10.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:10.157+0000 7f0b154b8700 1 -- 172.21.15.60:0/301540570 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b10065fa0 con 0x7f0b100b2130 2022-06-29T16:06:10.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:10.161+0000 7f0b0d7fa700 1 -- 172.21.15.60:0/301540570 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f0b08045a60 con 0x7f0b100b2130 2022-06-29T16:06:10.165 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:10.165 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":46,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7320,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/3498160620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3498160620},{"type":"v1","addr":"172.21.15.60:6833","nonce":3498160620}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:06:05.226825+0000","modified":"2022-06-29T16:06:06.241973+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7327},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7327":{"gid":7327,"name":"b","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/1551394182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1551394182},{"type":"v1","addr":"172.21.15.92:6835","nonce":1551394182}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2022-06-29T16:06:10.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:10.161+0000 7f0b154b8700 1 -- 172.21.15.60:0/301540570 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0af4053b00 msgr2=0x7f0af4055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:10.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:10.161+0000 7f0b154b8700 1 --2- 172.21.15.60:0/301540570 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0af4053b00 0x7f0af4055fc0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7f0b0000d000 tx=0x7f0b00009040).stop 2022-06-29T16:06:10.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:10.161+0000 7f0b154b8700 1 -- 172.21.15.60:0/301540570 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b100b2130 msgr2=0x7f0b10118af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:10.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:10.161+0000 7f0b154b8700 1 --2- 172.21.15.60:0/301540570 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b100b2130 0x7f0b10118af0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 rx=0x7f0b100b5cd0 tx=0x7f0b08007f40).stop 2022-06-29T16:06:10.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:10.161+0000 7f0b154b8700 1 -- 172.21.15.60:0/301540570 shutdown_connections 2022-06-29T16:06:10.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:10.161+0000 7f0b154b8700 1 --2- 172.21.15.60:0/301540570 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0af4053b00 0x7f0af4055fc0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:10.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:10.161+0000 7f0b154b8700 1 --2- 172.21.15.60:0/301540570 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0b100b2130 0x7f0b10118af0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:10.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:10.161+0000 7f0b154b8700 1 --2- 172.21.15.60:0/301540570 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b100aed00 0x7f0b100b46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:10.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:10.161+0000 7f0b154b8700 1 --2- 172.21.15.60:0/301540570 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0b1005d070 0x7f0b10159ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:10.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:10.161+0000 7f0b154b8700 1 -- 172.21.15.60:0/301540570 >> 172.21.15.60:0/301540570 conn(0x7f0b100b5890 msgr2=0x7f0b100aca20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:10.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:10.161+0000 7f0b154b8700 1 -- 172.21.15.60:0/301540570 shutdown_connections 2022-06-29T16:06:10.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:10.161+0000 7f0b154b8700 1 -- 172.21.15.60:0/301540570 wait complete. 2022-06-29T16:06:10.171 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 46 2022-06-29T16:06:10.378 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 46, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:06:05.226825+0000', 'modified': '2022-06-29T16:06:06.241973+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7327}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7327': {'gid': 7327, 'name': 'b', 'rank': 0, 'incarnation': 45, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.92:6835/1551394182', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 1551394182}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 1551394182}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:06:10.379 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:06:10.379 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-06-29T16:06:10.492 INFO:teuthology.orchestra.run.smithi092.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":7327,"want_state":"up:active","state":"up:active","rank_uptime":5.2491619250000001,"mdsmap_epoch":46,"osdmap_epoch":100,"osdmap_epoch_barrier":100,"uptime":9.6258789999999994} 2022-06-29T16:06:10.493 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 7327, "mdsmap_epoch": 46, "osdmap_epoch": 100, "osdmap_epoch_barrier": 100, "rank_uptime": 5.249161925, "state": "up:active", "uptime": 9.625879, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:06:10.493 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-06-29T16:06:10.494 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-06-29T16:06:10.494 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi121.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-06-29T16:06:10.495 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:10.495 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:06:10.504 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:06:10.518 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:06:10.519 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:06:10.530 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:06:10.531 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:06:10.532 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:10.533 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:06:10.585 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-06-29T16:06:10.586 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:06:10.588 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:06:10.606 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:06:10.607 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:06:10.617 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:06:10.618 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:06:10.619 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:10.619 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:06:10.663 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:2022-06-29T16:06:10.653+0000 7fa7d09e3080 -1 init, newargv = 0x55e2da3f02a0 newargc=9 2022-06-29T16:06:10.663 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[13226]: starting ceph client 2022-06-29T16:06:10.672 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[13226]: starting fuse 2022-06-29T16:06:11.770 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:06:11.789 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:06:11.790 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:06:11.846 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:06:11.847 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:06:11.848 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:11.848 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:06:11.853 INFO:teuthology.orchestra.run.smithi121.stdout:52 2022-06-29T16:06:11.854 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-06-29T16:06:11.854 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:11.855 DEBUG:teuthology.orchestra.run.smithi121:> sudo python3 -c ' 2022-06-29T16:06:11.855 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2022-06-29T16:06:11.856 DEBUG:teuthology.orchestra.run.smithi121:> import re 2022-06-29T16:06:11.856 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-06-29T16:06:11.856 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2022-06-29T16:06:11.857 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:06:11.857 DEBUG:teuthology.orchestra.run.smithi121:> def find_socket(client_name): 2022-06-29T16:06:11.858 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-06-29T16:06:11.858 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2022-06-29T16:06:11.859 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:06:11.859 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2022-06-29T16:06:11.859 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2022-06-29T16:06:11.860 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2022-06-29T16:06:11.860 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2022-06-29T16:06:11.861 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:06:11.862 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2022-06-29T16:06:11.862 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-06-29T16:06:11.863 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2022-06-29T16:06:11.863 DEBUG:teuthology.orchestra.run.smithi121:> return f 2022-06-29T16:06:11.864 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-06-29T16:06:11.864 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:06:11.864 DEBUG:teuthology.orchestra.run.smithi121:> print(find_socket("client.0")) 2022-06-29T16:06:11.865 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-06-29T16:06:11.909 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.0.13226.asok 2022-06-29T16:06:11.914 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.13226.asok 2022-06-29T16:06:11.914 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:11.915 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.13226.asok status 2022-06-29T16:06:12.049 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2022-06-29T16:06:12.050 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2022-06-29T16:06:12.050 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "f25ffac72e5a9e63860ff7deb6767a11f0559a16", 2022-06-29T16:06:12.050 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 15.2.16-304-gf25ffac7 (f25ffac72e5a9e63860ff7deb6767a11f0559a16) octopus (stable)", 2022-06-29T16:06:12.050 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "0", 2022-06-29T16:06:12.051 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2022-06-29T16:06:12.051 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-06-29T16:06:12.051 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "13226", 2022-06-29T16:06:12.051 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2022-06-29T16:06:12.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:06:12.052 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2022-06-29T16:06:12.052 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2022-06-29T16:06:12.052 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 6527, 2022-06-29T16:06:12.052 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2022-06-29T16:06:12.053 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2022-06-29T16:06:12.053 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2022-06-29T16:06:12.053 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 6527 2022-06-29T16:06:12.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:06:12.054 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-06-29T16:06:12.054 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-06-29T16:06:12.054 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-06-29T16:06:12.054 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1414041517 2022-06-29T16:06:12.054 INFO:teuthology.orchestra.run.smithi121.stdout: } 2022-06-29T16:06:12.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:06:12.055 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-06-29T16:06:12.055 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-06-29T16:06:12.055 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-06-29T16:06:12.056 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1414041517 2022-06-29T16:06:12.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:06:12.056 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.6527 172.21.15.121:0/1414041517", 2022-06-29T16:06:12.056 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "172.21.15.121:0/1414041517", 2022-06-29T16:06:12.056 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2022-06-29T16:06:12.057 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 46, 2022-06-29T16:06:12.057 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 100, 2022-06-29T16:06:12.057 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2022-06-29T16:06:12.058 INFO:teuthology.orchestra.run.smithi121.stdout: "blacklisted": false, 2022-06-29T16:06:12.058 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2022-06-29T16:06:12.058 INFO:teuthology.orchestra.run.smithi121.stdout:} 2022-06-29T16:06:12.059 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:12.060 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:06:12.061 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:06:12.067 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T16:06:12.068 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T16:06:12.069 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:12.069 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:06:12.086 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:06:12.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.193+0000 7f9bf7fff700 1 Processor -- start 2022-06-29T16:06:12.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.193+0000 7f9bf7fff700 1 -- start start 2022-06-29T16:06:12.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.193+0000 7f9bf7fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9bf80b5eb0 0x7f9bf8142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:12.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.193+0000 7f9bf7fff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf80b0cf0 con 0x7f9bf80b2c50 2022-06-29T16:06:12.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.193+0000 7f9bf7fff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf80b4d10 con 0x7f9bf80b48b0 2022-06-29T16:06:12.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.193+0000 7f9bf7fff700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bf80b54e0 con 0x7f9bf80b5eb0 2022-06-29T16:06:12.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.193+0000 7f9bf6ffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9bf80b5eb0 0x7f9bf8142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:12.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.193+0000 7f9bf77fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9bf80b2c50 0x7f9bf814d380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59660/0 (socket says 172.21.15.60:59660) 2022-06-29T16:06:12.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.193+0000 7f9bf77fe700 1 -- 172.21.15.60:0/3949983968 learned_addr learned my addr 172.21.15.60:0/3949983968 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:12.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.193+0000 7f9bf5ffb700 1 -- 172.21.15.60:0/3949983968 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 129252490 0 0) 0x7f9bf0001110 con 0x7f9bf80b2c50 2022-06-29T16:06:12.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.193+0000 7f9bf5ffb700 1 -- 172.21.15.60:0/3949983968 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bd8003260 con 0x7f9bf80b2c50 2022-06-29T16:06:12.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.193+0000 7f9bf5ffb700 1 -- 172.21.15.60:0/3949983968 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1431473847 0 0) 0x7f9bf80b4d10 con 0x7f9bf80b48b0 2022-06-29T16:06:12.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.193+0000 7f9bf6ffd700 1 -- 172.21.15.60:0/3949983968 >> v1:172.21.15.92:6789/0 conn(0x7f9bf80b48b0 legacy=0x7f9bf80b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:12.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.197+0000 7f9bf6ffd700 1 -- 172.21.15.60:0/3949983968 >> v1:172.21.15.60:6789/0 conn(0x7f9bf80b2c50 legacy=0x7f9bf814d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:12.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.197+0000 7f9bf6ffd700 1 -- 172.21.15.60:0/3949983968 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf80b6290 con 0x7f9bf80b5eb0 2022-06-29T16:06:12.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.197+0000 7f9bf6ffd700 1 --2- 172.21.15.60:0/3949983968 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bf80b5eb0 0x7f9bf8142a20 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0x7f9bf0001990 tx=0x7f9be8006df0).ready entity=mon.0 client_cookie=1974c9199eaacb48 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:12.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.197+0000 7f9bf5ffb700 1 -- 172.21.15.60:0/3949983968 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9bf00017f0 con 0x7f9bf80b5eb0 2022-06-29T16:06:12.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.197+0000 7f9bf5ffb700 1 -- 172.21.15.60:0/3949983968 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be8000e90 con 0x7f9bf80b5eb0 2022-06-29T16:06:12.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.197+0000 7f9bf5ffb700 1 -- 172.21.15.60:0/3949983968 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9be8011400 con 0x7f9bf80b5eb0 2022-06-29T16:06:12.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.197+0000 7f9bf7fff700 1 -- 172.21.15.60:0/3949983968 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bf80b5eb0 msgr2=0x7f9bf8142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:12.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.197+0000 7f9bf7fff700 1 --2- 172.21.15.60:0/3949983968 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bf80b5eb0 0x7f9bf8142a20 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0x7f9bf0001990 tx=0x7f9be8006df0).stop 2022-06-29T16:06:12.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.197+0000 7f9bf7fff700 1 -- 172.21.15.60:0/3949983968 shutdown_connections 2022-06-29T16:06:12.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.197+0000 7f9bf7fff700 1 --2- 172.21.15.60:0/3949983968 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bf80b5eb0 0x7f9bf8142a20 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:12.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.197+0000 7f9bf7fff700 1 -- 172.21.15.60:0/3949983968 >> 172.21.15.60:0/3949983968 conn(0x7f9bf8062090 msgr2=0x7f9bf8062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:12.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.197+0000 7f9bf7fff700 1 -- 172.21.15.60:0/3949983968 shutdown_connections 2022-06-29T16:06:12.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.197+0000 7f9bf7fff700 1 -- 172.21.15.60:0/3949983968 wait complete. 2022-06-29T16:06:12.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.197+0000 7f9bf7fff700 1 Processor -- start 2022-06-29T16:06:12.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.197+0000 7f9bf7fff700 1 -- start start 2022-06-29T16:06:12.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.201+0000 7f9bf7fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bf80b2c50 0x7f9bf813edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:12.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.201+0000 7f9bf7fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bf80b48b0 0x7f9bf813bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:12.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.201+0000 7f9bf7fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bf80b5eb0 0x7f9bf813c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:12.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.201+0000 7f9bf7fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf813fe80 con 0x7f9bf80b5eb0 2022-06-29T16:06:12.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.201+0000 7f9bf7fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf813c930 con 0x7f9bf80b2c50 2022-06-29T16:06:12.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.201+0000 7f9bf7fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf8149ed0 con 0x7f9bf80b48b0 2022-06-29T16:06:12.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.201+0000 7f9bf77fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bf80b5eb0 0x7f9bf813c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:12.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.201+0000 7f9bf6ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bf80b2c50 0x7f9bf813edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:12.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.201+0000 7f9bf77fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bf80b5eb0 0x7f9bf813c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51580/0 (socket says 172.21.15.60:51580) 2022-06-29T16:06:12.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.201+0000 7f9bf77fe700 1 -- 172.21.15.60:0/3160614873 learned_addr learned my addr 172.21.15.60:0/3160614873 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:12.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.201+0000 7f9bf67fc700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bf80b48b0 0x7f9bf813bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:12.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.201+0000 7f9bf77fe700 1 -- 172.21.15.60:0/3160614873 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bf80b48b0 msgr2=0x7f9bf813bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:12.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.201+0000 7f9bf77fe700 1 --2- 172.21.15.60:0/3160614873 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bf80b48b0 0x7f9bf813bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:12.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.201+0000 7f9bf77fe700 1 -- 172.21.15.60:0/3160614873 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bf80b2c50 msgr2=0x7f9bf813edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:12.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.201+0000 7f9bf77fe700 1 --2- 172.21.15.60:0/3160614873 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bf80b2c50 0x7f9bf813edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:12.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.201+0000 7f9bf77fe700 1 -- 172.21.15.60:0/3160614873 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be800b180 con 0x7f9bf80b5eb0 2022-06-29T16:06:12.216 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.201+0000 7f9bf77fe700 1 --2- 172.21.15.60:0/3160614873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bf80b5eb0 0x7f9bf813c440 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 rx=0x7f9bf00019d0 tx=0x7f9bf000cac0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:12.216 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.201+0000 7f9beffff700 1 -- 172.21.15.60:0/3160614873 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9bf0007170 con 0x7f9bf80b5eb0 2022-06-29T16:06:12.216 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.201+0000 7f9beffff700 1 -- 172.21.15.60:0/3160614873 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bf00072d0 con 0x7f9bf80b5eb0 2022-06-29T16:06:12.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.201+0000 7f9bf7fff700 1 -- 172.21.15.60:0/3160614873 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf8146800 con 0x7f9bf80b5eb0 2022-06-29T16:06:12.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.201+0000 7f9beffff700 1 -- 172.21.15.60:0/3160614873 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9bf0004920 con 0x7f9bf80b5eb0 2022-06-29T16:06:12.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.201+0000 7f9bf7fff700 1 -- 172.21.15.60:0/3160614873 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf8149950 con 0x7f9bf80b5eb0 2022-06-29T16:06:12.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.201+0000 7f9bf7fff700 1 -- 172.21.15.60:0/3160614873 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf81469e0 con 0x7f9bf80b5eb0 2022-06-29T16:06:12.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.205+0000 7f9beffff700 1 -- 172.21.15.60:0/3160614873 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9bf0004a80 con 0x7f9bf80b5eb0 2022-06-29T16:06:12.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.205+0000 7f9beffff700 1 --2- 172.21.15.60:0/3160614873 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9bdc053b00 0x7f9bdc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:12.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.205+0000 7f9beffff700 1 -- 172.21.15.60:0/3160614873 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9bf006b7e0 con 0x7f9bf80b5eb0 2022-06-29T16:06:12.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.205+0000 7f9bf6ffd700 1 --2- 172.21.15.60:0/3160614873 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9bdc053b00 0x7f9bdc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:12.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.205+0000 7f9bf6ffd700 1 --2- 172.21.15.60:0/3160614873 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9bdc053b00 0x7f9bdc055fc0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7f9be8019000 tx=0x7f9bf80b6bd0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:12.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.209+0000 7f9beffff700 1 -- 172.21.15.60:0/3160614873 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9bf81469e0 con 0x7f9bf80b5eb0 2022-06-29T16:06:12.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.341+0000 7f9bf7fff700 1 -- 172.21.15.60:0/3160614873 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9bf80b0cf0 con 0x7f9bf80b5eb0 2022-06-29T16:06:12.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.341+0000 7f9beffff700 1 -- 172.21.15.60:0/3160614873 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+16043 (secure 0 0 0) 0x7f9bf80b0cf0 con 0x7f9bf80b5eb0 2022-06-29T16:06:12.348 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:12.348 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":100,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:06:06.229758+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:06:03.648757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2022-06-29T16:06:04.556403+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:06:12.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.345+0000 7f9bf7fff700 1 -- 172.21.15.60:0/3160614873 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9bdc053b00 msgr2=0x7f9bdc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:12.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.345+0000 7f9bf7fff700 1 --2- 172.21.15.60:0/3160614873 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9bdc053b00 0x7f9bdc055fc0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7f9be8019000 tx=0x7f9bf80b6bd0).stop 2022-06-29T16:06:12.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.345+0000 7f9bf7fff700 1 -- 172.21.15.60:0/3160614873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bf80b5eb0 msgr2=0x7f9bf813c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:12.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.345+0000 7f9bf7fff700 1 --2- 172.21.15.60:0/3160614873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bf80b5eb0 0x7f9bf813c440 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 rx=0x7f9bf00019d0 tx=0x7f9bf000cac0).stop 2022-06-29T16:06:12.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.345+0000 7f9bf7fff700 1 -- 172.21.15.60:0/3160614873 shutdown_connections 2022-06-29T16:06:12.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.345+0000 7f9bf7fff700 1 --2- 172.21.15.60:0/3160614873 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9bdc053b00 0x7f9bdc055fc0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:12.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.345+0000 7f9bf7fff700 1 --2- 172.21.15.60:0/3160614873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bf80b5eb0 0x7f9bf813c440 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:12.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.345+0000 7f9bf7fff700 1 --2- 172.21.15.60:0/3160614873 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bf80b48b0 0x7f9bf813bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:12.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.345+0000 7f9bf7fff700 1 --2- 172.21.15.60:0/3160614873 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bf80b2c50 0x7f9bf813edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:12.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.345+0000 7f9bf7fff700 1 -- 172.21.15.60:0/3160614873 >> 172.21.15.60:0/3160614873 conn(0x7f9bf8062090 msgr2=0x7f9bf8141530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:12.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.345+0000 7f9bf7fff700 1 -- 172.21.15.60:0/3160614873 shutdown_connections 2022-06-29T16:06:12.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.345+0000 7f9bf7fff700 1 -- 172.21.15.60:0/3160614873 wait complete. 2022-06-29T16:06:12.361 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-29T16:06:03.648757+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2022-06-29T16:06:04.556403+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-29T16:06:12.363 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:06:12.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.473+0000 7fd2bc122700 1 Processor -- start 2022-06-29T16:06:12.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.473+0000 7fd2bc122700 1 -- start start 2022-06-29T16:06:12.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.473+0000 7fd2bc122700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2b40b3d30 0x7fd2b4146ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:12.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.473+0000 7fd2bc122700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd2b4061d90 0x7fd2b4142b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:12.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.473+0000 7fd2bc122700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2b40b4f50 con 0x7fd2b40b9d70 2022-06-29T16:06:12.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.473+0000 7fd2bc122700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2b40ba150 con 0x7fd2b4061d90 2022-06-29T16:06:12.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.473+0000 7fd2bc122700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b40b5180 con 0x7fd2b40b3d30 2022-06-29T16:06:12.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.473+0000 7fd2ba6bf700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd2b4061d90 0x7fd2b4142b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:12.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.473+0000 7fd2b96bd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2b40b3d30 0x7fd2b4146ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:12.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.473+0000 7fd2ba6bf700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd2b4061d90 0x7fd2b4142b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51588/0 (socket says 172.21.15.60:51588) 2022-06-29T16:06:12.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.473+0000 7fd2b96bd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2b40b3d30 0x7fd2b4146ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:36448/0 (socket says 172.21.15.60:36448) 2022-06-29T16:06:12.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.473+0000 7fd2ba6bf700 1 -- 172.21.15.60:0/3700894446 learned_addr learned my addr 172.21.15.60:0/3700894446 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:12.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.473+0000 7fd2b9ebe700 1 --1- 172.21.15.60:0/3700894446 >> v1:172.21.15.92:6789/0 conn(0x7fd2b40b9d70 0x7fd2b41465d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:35686/0 (socket says 172.21.15.60:35686) 2022-06-29T16:06:12.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.473+0000 7fd2b8ebc700 1 -- 172.21.15.60:0/3700894446 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3549352604 0 0) 0x7fd2b40b4f50 con 0x7fd2b40b9d70 2022-06-29T16:06:12.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.473+0000 7fd2ba6bf700 1 -- 172.21.15.60:0/3700894446 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2b40b3d30 msgr2=0x7fd2b4146ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:12.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.473+0000 7fd2ba6bf700 1 --2- 172.21.15.60:0/3700894446 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2b40b3d30 0x7fd2b4146ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:12.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.473+0000 7fd2ba6bf700 1 -- 172.21.15.60:0/3700894446 >> v1:172.21.15.92:6789/0 conn(0x7fd2b40b9d70 legacy=0x7fd2b41465d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:12.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.473+0000 7fd2ba6bf700 1 -- 172.21.15.60:0/3700894446 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b41461a0 con 0x7fd2b4061d90 2022-06-29T16:06:12.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.473+0000 7fd2ba6bf700 1 --2- 172.21.15.60:0/3700894446 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2b4061d90 0x7fd2b4142b70 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 rx=0x7fd2ac0032f0 tx=0x7fd2ac009290).ready entity=mon.0 client_cookie=accfaf7c3ad7ec4b server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:12.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.473+0000 7fd2b8ebc700 1 -- 172.21.15.60:0/3700894446 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2ac00b8a0 con 0x7fd2b4061d90 2022-06-29T16:06:12.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.473+0000 7fd2b8ebc700 1 -- 172.21.15.60:0/3700894446 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2ac00ba00 con 0x7fd2b4061d90 2022-06-29T16:06:12.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.473+0000 7fd2b8ebc700 1 -- 172.21.15.60:0/3700894446 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2ac00bb70 con 0x7fd2b4061d90 2022-06-29T16:06:12.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.477+0000 7fd2bc122700 1 -- 172.21.15.60:0/3700894446 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2b4061d90 msgr2=0x7fd2b4142b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:12.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.477+0000 7fd2bc122700 1 --2- 172.21.15.60:0/3700894446 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2b4061d90 0x7fd2b4142b70 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 rx=0x7fd2ac0032f0 tx=0x7fd2ac009290).stop 2022-06-29T16:06:12.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.477+0000 7fd2bc122700 1 -- 172.21.15.60:0/3700894446 shutdown_connections 2022-06-29T16:06:12.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.477+0000 7fd2bc122700 1 --2- 172.21.15.60:0/3700894446 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2b4061d90 0x7fd2b4142b70 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:12.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.477+0000 7fd2bc122700 1 --2- 172.21.15.60:0/3700894446 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2b40b3d30 0x7fd2b4146ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:12.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.477+0000 7fd2bc122700 1 -- 172.21.15.60:0/3700894446 >> 172.21.15.60:0/3700894446 conn(0x7fd2b4062e70 msgr2=0x7fd2b40b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:12.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.477+0000 7fd2bc122700 1 -- 172.21.15.60:0/3700894446 shutdown_connections 2022-06-29T16:06:12.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.477+0000 7fd2bc122700 1 -- 172.21.15.60:0/3700894446 wait complete. 2022-06-29T16:06:12.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.477+0000 7fd2bc122700 1 Processor -- start 2022-06-29T16:06:12.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.477+0000 7fd2bc122700 1 -- start start 2022-06-29T16:06:12.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.477+0000 7fd2bc122700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2b4061d90 0x7fd2b40aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:12.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.477+0000 7fd2bc122700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2b40b3d30 0x7fd2b413e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:12.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.481+0000 7fd2bc122700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2b40b9d70 0x7fd2b40aeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:12.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.481+0000 7fd2bc122700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b4156050 con 0x7fd2b40b9d70 2022-06-29T16:06:12.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.481+0000 7fd2bc122700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b4144ba0 con 0x7fd2b4061d90 2022-06-29T16:06:12.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.481+0000 7fd2bc122700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b4144660 con 0x7fd2b40b3d30 2022-06-29T16:06:12.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.481+0000 7fd2ba6bf700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2b40b9d70 0x7fd2b40aeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:12.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.481+0000 7fd2ba6bf700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2b40b9d70 0x7fd2b40aeaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51594/0 (socket says 172.21.15.60:51594) 2022-06-29T16:06:12.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.481+0000 7fd2ba6bf700 1 -- 172.21.15.60:0/2030572861 learned_addr learned my addr 172.21.15.60:0/2030572861 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:12.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.481+0000 7fd2b9ebe700 1 --2- 172.21.15.60:0/2030572861 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2b4061d90 0x7fd2b40aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:12.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.481+0000 7fd2b96bd700 1 --2- 172.21.15.60:0/2030572861 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2b40b3d30 0x7fd2b413e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:12.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.481+0000 7fd2ba6bf700 1 -- 172.21.15.60:0/2030572861 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2b40b3d30 msgr2=0x7fd2b413e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:12.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.481+0000 7fd2ba6bf700 1 --2- 172.21.15.60:0/2030572861 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2b40b3d30 0x7fd2b413e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:12.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.481+0000 7fd2ba6bf700 1 -- 172.21.15.60:0/2030572861 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2b4061d90 msgr2=0x7fd2b40aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:12.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.481+0000 7fd2ba6bf700 1 --2- 172.21.15.60:0/2030572861 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2b4061d90 0x7fd2b40aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:12.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.481+0000 7fd2ba6bf700 1 -- 172.21.15.60:0/2030572861 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ac009530 con 0x7fd2b40b9d70 2022-06-29T16:06:12.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.481+0000 7fd2ba6bf700 1 --2- 172.21.15.60:0/2030572861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2b40b9d70 0x7fd2b40aeaf0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 rx=0x7fd2ac0032f0 tx=0x7fd2ac016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:12.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.481+0000 7fd2aaffd700 1 -- 172.21.15.60:0/2030572861 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2ac00b8a0 con 0x7fd2b40b9d70 2022-06-29T16:06:12.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.481+0000 7fd2aaffd700 1 -- 172.21.15.60:0/2030572861 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2ac021630 con 0x7fd2b40b9d70 2022-06-29T16:06:12.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.481+0000 7fd2bc122700 1 -- 172.21.15.60:0/2030572861 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2b4144900 con 0x7fd2b40b9d70 2022-06-29T16:06:12.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.481+0000 7fd2aaffd700 1 -- 172.21.15.60:0/2030572861 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2ac021990 con 0x7fd2b40b9d70 2022-06-29T16:06:12.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.481+0000 7fd2bc122700 1 -- 172.21.15.60:0/2030572861 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2b4145dd0 con 0x7fd2b40b9d70 2022-06-29T16:06:12.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.481+0000 7fd2aaffd700 1 -- 172.21.15.60:0/2030572861 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd2ac00f070 con 0x7fd2b40b9d70 2022-06-29T16:06:12.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.481+0000 7fd2bc122700 1 -- 172.21.15.60:0/2030572861 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2b40573d0 con 0x7fd2b40b9d70 2022-06-29T16:06:12.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.485+0000 7fd2aaffd700 1 --2- 172.21.15.60:0/2030572861 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd29c0539e0 0x7fd29c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:12.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.485+0000 7fd2aaffd700 1 -- 172.21.15.60:0/2030572861 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd2ac06d680 con 0x7fd2b40b9d70 2022-06-29T16:06:12.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.485+0000 7fd2b9ebe700 1 --2- 172.21.15.60:0/2030572861 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd29c0539e0 0x7fd29c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:12.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.485+0000 7fd2b9ebe700 1 --2- 172.21.15.60:0/2030572861 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd29c0539e0 0x7fd29c055ea0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7fd2b000b000 tx=0x7fd2b0007160).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:12.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.489+0000 7fd2aaffd700 1 -- 172.21.15.60:0/2030572861 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd2ac0445a0 con 0x7fd2b40b9d70 2022-06-29T16:06:12.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.613+0000 7fd2bc122700 1 -- 172.21.15.60:0/2030572861 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fd2b405da40 con 0x7fd2b40b9d70 2022-06-29T16:06:12.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.617+0000 7fd2aaffd700 1 -- 172.21.15.60:0/2030572861 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v100) v1 ==== 110+0+10 (secure 0 0 0) 0x7fd2ac047bc0 con 0x7fd2b40b9d70 2022-06-29T16:06:12.621 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:06:12.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.617+0000 7fd2bc122700 1 -- 172.21.15.60:0/2030572861 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd29c0539e0 msgr2=0x7fd29c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:12.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.617+0000 7fd2bc122700 1 --2- 172.21.15.60:0/2030572861 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd29c0539e0 0x7fd29c055ea0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7fd2b000b000 tx=0x7fd2b0007160).stop 2022-06-29T16:06:12.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.617+0000 7fd2bc122700 1 -- 172.21.15.60:0/2030572861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2b40b9d70 msgr2=0x7fd2b40aeaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:12.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.617+0000 7fd2bc122700 1 --2- 172.21.15.60:0/2030572861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2b40b9d70 0x7fd2b40aeaf0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 rx=0x7fd2ac0032f0 tx=0x7fd2ac016000).stop 2022-06-29T16:06:12.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.617+0000 7fd2bc122700 1 -- 172.21.15.60:0/2030572861 shutdown_connections 2022-06-29T16:06:12.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.617+0000 7fd2bc122700 1 --2- 172.21.15.60:0/2030572861 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd29c0539e0 0x7fd29c055ea0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:12.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.617+0000 7fd2bc122700 1 --2- 172.21.15.60:0/2030572861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2b40b9d70 0x7fd2b40aeaf0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:12.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.617+0000 7fd2bc122700 1 --2- 172.21.15.60:0/2030572861 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2b40b3d30 0x7fd2b413e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:12.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.617+0000 7fd2bc122700 1 --2- 172.21.15.60:0/2030572861 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2b4061d90 0x7fd2b40aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:12.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.617+0000 7fd2bc122700 1 -- 172.21.15.60:0/2030572861 >> 172.21.15.60:0/2030572861 conn(0x7fd2b4062e70 msgr2=0x7fd2b40b0760 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:12.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.617+0000 7fd2bc122700 1 -- 172.21.15.60:0/2030572861 shutdown_connections 2022-06-29T16:06:12.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.617+0000 7fd2bc122700 1 -- 172.21.15.60:0/2030572861 wait complete. 2022-06-29T16:06:12.634 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:06:12.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.741+0000 7f050eba9700 1 Processor -- start 2022-06-29T16:06:12.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.741+0000 7f050eba9700 1 -- start start 2022-06-29T16:06:12.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.741+0000 7f050eba9700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f050805d1f0 0x7f05080b1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:12.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.741+0000 7f050eba9700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05080b2ce0 con 0x7f05080b3cc0 2022-06-29T16:06:12.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.741+0000 7f050eba9700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05080ba150 con 0x7f05080b9d70 2022-06-29T16:06:12.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.741+0000 7f050eba9700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05080b4d10 con 0x7f050805d1f0 2022-06-29T16:06:12.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.741+0000 7f050d146700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f05080b3cc0 0x7f05081420b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59688/0 (socket says 172.21.15.60:59688) 2022-06-29T16:06:12.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.741+0000 7f050d146700 1 -- 172.21.15.60:0/1065060356 learned_addr learned my addr 172.21.15.60:0/1065060356 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:12.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.741+0000 7f0507fff700 1 --2- 172.21.15.60:0/1065060356 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f050805d1f0 0x7f05080b1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:12.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.741+0000 7f05077fe700 1 -- 172.21.15.60:0/1065060356 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3584992120 0 0) 0x7f05080b2ce0 con 0x7f05080b3cc0 2022-06-29T16:06:12.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.741+0000 7f05077fe700 1 -- 172.21.15.60:0/1065060356 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04f4003260 con 0x7f05080b3cc0 2022-06-29T16:06:12.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.741+0000 7f05077fe700 1 -- 172.21.15.60:0/1065060356 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 554399212 0 0) 0x7f05080ba150 con 0x7f05080b9d70 2022-06-29T16:06:12.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.741+0000 7f05077fe700 1 -- 172.21.15.60:0/1065060356 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05080b2ce0 con 0x7f05080b9d70 2022-06-29T16:06:12.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.741+0000 7f0507fff700 1 -- 172.21.15.60:0/1065060356 >> v1:172.21.15.92:6789/0 conn(0x7f05080b9d70 legacy=0x7f05080b3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:12.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.741+0000 7f0507fff700 1 -- 172.21.15.60:0/1065060356 >> v1:172.21.15.60:6789/0 conn(0x7f05080b3cc0 legacy=0x7f05081420b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:12.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.741+0000 7f0507fff700 1 -- 172.21.15.60:0/1065060356 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05080b4a70 con 0x7f050805d1f0 2022-06-29T16:06:12.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.741+0000 7f0507fff700 1 --2- 172.21.15.60:0/1065060356 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f050805d1f0 0x7f05080b1b50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f04fc002f70 tx=0x7f04fc006df0).ready entity=mon.2 client_cookie=cf439fca84543520 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:12.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.741+0000 7f05077fe700 1 -- 172.21.15.60:0/1065060356 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04fc00d5c0 con 0x7f050805d1f0 2022-06-29T16:06:12.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.741+0000 7f05077fe700 1 -- 172.21.15.60:0/1065060356 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04fc000e90 con 0x7f050805d1f0 2022-06-29T16:06:12.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.741+0000 7f05077fe700 1 -- 172.21.15.60:0/1065060356 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04fc00d910 con 0x7f050805d1f0 2022-06-29T16:06:12.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.745+0000 7f050eba9700 1 -- 172.21.15.60:0/1065060356 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f050805d1f0 msgr2=0x7f05080b1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:12.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.745+0000 7f050eba9700 1 --2- 172.21.15.60:0/1065060356 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f050805d1f0 0x7f05080b1b50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f04fc002f70 tx=0x7f04fc006df0).stop 2022-06-29T16:06:12.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.745+0000 7f050eba9700 1 -- 172.21.15.60:0/1065060356 shutdown_connections 2022-06-29T16:06:12.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.745+0000 7f050eba9700 1 --2- 172.21.15.60:0/1065060356 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f050805d1f0 0x7f05080b1b50 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:12.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.745+0000 7f050eba9700 1 -- 172.21.15.60:0/1065060356 >> 172.21.15.60:0/1065060356 conn(0x7f0508062ce0 msgr2=0x7f05080b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:12.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.745+0000 7f050eba9700 1 -- 172.21.15.60:0/1065060356 shutdown_connections 2022-06-29T16:06:12.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.745+0000 7f050eba9700 1 -- 172.21.15.60:0/1065060356 wait complete. 2022-06-29T16:06:12.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.745+0000 7f050eba9700 1 Processor -- start 2022-06-29T16:06:12.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.745+0000 7f050eba9700 1 -- start start 2022-06-29T16:06:12.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.745+0000 7f050eba9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f050805d1f0 0x7f05080af900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:12.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.745+0000 7f050eba9700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05080b3cc0 0x7f050813db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:12.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.745+0000 7f050eba9700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05080b9d70 0x7f050813e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:12.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.745+0000 7f050eba9700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0508152c90 con 0x7f050805d1f0 2022-06-29T16:06:12.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.745+0000 7f050eba9700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0508146b90 con 0x7f05080b9d70 2022-06-29T16:06:12.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.745+0000 7f050eba9700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0508146760 con 0x7f05080b3cc0 2022-06-29T16:06:12.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.745+0000 7f050c945700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f050805d1f0 0x7f05080af900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:12.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.745+0000 7f050c945700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f050805d1f0 0x7f05080af900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51604/0 (socket says 172.21.15.60:51604) 2022-06-29T16:06:12.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.745+0000 7f050c945700 1 -- 172.21.15.60:0/1877034458 learned_addr learned my addr 172.21.15.60:0/1877034458 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:12.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.745+0000 7f050d146700 1 --2- 172.21.15.60:0/1877034458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05080b9d70 0x7f050813e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:12.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.745+0000 7f0507fff700 1 --2- 172.21.15.60:0/1877034458 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05080b3cc0 0x7f050813db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:12.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.745+0000 7f050c945700 1 -- 172.21.15.60:0/1877034458 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05080b3cc0 msgr2=0x7f050813db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:12.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.745+0000 7f050c945700 1 --2- 172.21.15.60:0/1877034458 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05080b3cc0 0x7f050813db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:12.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.745+0000 7f050c945700 1 -- 172.21.15.60:0/1877034458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05080b9d70 msgr2=0x7f050813e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:12.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.745+0000 7f050c945700 1 --2- 172.21.15.60:0/1877034458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05080b9d70 0x7f050813e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:12.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.745+0000 7f050c945700 1 -- 172.21.15.60:0/1877034458 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04fc00b180 con 0x7f050805d1f0 2022-06-29T16:06:12.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.745+0000 7f050d146700 1 --2- 172.21.15.60:0/1877034458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05080b9d70 0x7f050813e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:06:12.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.749+0000 7f050c945700 1 --2- 172.21.15.60:0/1877034458 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f050805d1f0 0x7f05080af900 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 rx=0x7f04f8001110 tx=0x7f04f8007ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:12.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.749+0000 7f0505ffb700 1 -- 172.21.15.60:0/1877034458 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04f800ce10 con 0x7f050805d1f0 2022-06-29T16:06:12.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.749+0000 7f0505ffb700 1 -- 172.21.15.60:0/1877034458 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04f800f470 con 0x7f050805d1f0 2022-06-29T16:06:12.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.749+0000 7f050eba9700 1 -- 172.21.15.60:0/1877034458 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f050813e570 con 0x7f050805d1f0 2022-06-29T16:06:12.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.749+0000 7f0505ffb700 1 -- 172.21.15.60:0/1877034458 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04f800f7d0 con 0x7f050805d1f0 2022-06-29T16:06:12.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.749+0000 7f050eba9700 1 -- 172.21.15.60:0/1877034458 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0508146380 con 0x7f050805d1f0 2022-06-29T16:06:12.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.749+0000 7f050eba9700 1 -- 172.21.15.60:0/1877034458 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05080573d0 con 0x7f050805d1f0 2022-06-29T16:06:12.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.749+0000 7f0505ffb700 1 -- 172.21.15.60:0/1877034458 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f04f8019030 con 0x7f050805d1f0 2022-06-29T16:06:12.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.749+0000 7f0505ffb700 1 --2- 172.21.15.60:0/1877034458 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f04e8053a70 0x7f04e8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:12.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.749+0000 7f0505ffb700 1 -- 172.21.15.60:0/1877034458 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f04f806b2d0 con 0x7f050805d1f0 2022-06-29T16:06:12.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.749+0000 7f0507fff700 1 --2- 172.21.15.60:0/1877034458 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f04e8053a70 0x7f04e8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:12.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.753+0000 7f0507fff700 1 --2- 172.21.15.60:0/1877034458 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f04e8053a70 0x7f04e8055f30 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7f04fc018000 tx=0x7f04fc014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:12.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.757+0000 7f0505ffb700 1 -- 172.21.15.60:0/1877034458 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f04f80421f0 con 0x7f050805d1f0 2022-06-29T16:06:12.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.885+0000 7f050eba9700 1 -- 172.21.15.60:0/1877034458 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f050805d990 con 0x7f050805d1f0 2022-06-29T16:06:12.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.885+0000 7f0505ffb700 1 -- 172.21.15.60:0/1877034458 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v100) v1 ==== 92+0+10 (secure 0 0 0) 0x7f04f8045810 con 0x7f050805d1f0 2022-06-29T16:06:12.891 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:06:12.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.889+0000 7f050eba9700 1 -- 172.21.15.60:0/1877034458 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f04e8053a70 msgr2=0x7f04e8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:12.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.889+0000 7f050eba9700 1 --2- 172.21.15.60:0/1877034458 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f04e8053a70 0x7f04e8055f30 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7f04fc018000 tx=0x7f04fc014000).stop 2022-06-29T16:06:12.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.889+0000 7f050eba9700 1 -- 172.21.15.60:0/1877034458 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f050805d1f0 msgr2=0x7f05080af900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:12.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.889+0000 7f050eba9700 1 --2- 172.21.15.60:0/1877034458 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f050805d1f0 0x7f05080af900 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 rx=0x7f04f8001110 tx=0x7f04f8007ae0).stop 2022-06-29T16:06:12.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.889+0000 7f050eba9700 1 -- 172.21.15.60:0/1877034458 shutdown_connections 2022-06-29T16:06:12.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.889+0000 7f050eba9700 1 --2- 172.21.15.60:0/1877034458 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f04e8053a70 0x7f04e8055f30 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:12.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.889+0000 7f050eba9700 1 --2- 172.21.15.60:0/1877034458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05080b9d70 0x7f050813e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:12.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.889+0000 7f050eba9700 1 --2- 172.21.15.60:0/1877034458 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05080b3cc0 0x7f050813db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:12.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.889+0000 7f050eba9700 1 --2- 172.21.15.60:0/1877034458 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f050805d1f0 0x7f05080af900 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:12.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.889+0000 7f050eba9700 1 -- 172.21.15.60:0/1877034458 >> 172.21.15.60:0/1877034458 conn(0x7f0508062ce0 msgr2=0x7f05081437f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:12.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.889+0000 7f050eba9700 1 -- 172.21.15.60:0/1877034458 shutdown_connections 2022-06-29T16:06:12.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:12.889+0000 7f050eba9700 1 -- 172.21.15.60:0/1877034458 wait complete. 2022-06-29T16:06:12.905 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-29T16:06:13.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.005+0000 7f1343a62700 1 Processor -- start 2022-06-29T16:06:13.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.005+0000 7f1343a62700 1 -- start start 2022-06-29T16:06:13.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f1343a62700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f133c05d160 0x7f133c1461e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:13.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f1343a62700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f133c0b6e30 con 0x7f133c0b6a50 2022-06-29T16:06:13.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f1343a62700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f133c0b0dd0 con 0x7f133c05f0b0 2022-06-29T16:06:13.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f1343a62700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f133c0b1070 con 0x7f133c05d160 2022-06-29T16:06:13.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f13417fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f133c0b6a50 0x7f133c142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59698/0 (socket says 172.21.15.60:59698) 2022-06-29T16:06:13.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f13417fe700 1 -- 172.21.15.60:0/2229222140 learned_addr learned my addr 172.21.15.60:0/2229222140 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:13.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f1341fff700 1 --2- 172.21.15.60:0/2229222140 >> v2:172.21.15.92:3300/0 conn(0x7f133c05d160 0x7f133c1461e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:13.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f132ffff700 1 -- 172.21.15.60:0/2229222140 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1425054445 0 0) 0x7f133c0b6e30 con 0x7f133c0b6a50 2022-06-29T16:06:13.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f132ffff700 1 -- 172.21.15.60:0/2229222140 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1320003260 con 0x7f133c0b6a50 2022-06-29T16:06:13.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f132ffff700 1 -- 172.21.15.60:0/2229222140 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1449678110 0 0) 0x7f133c0b0dd0 con 0x7f133c05f0b0 2022-06-29T16:06:13.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f132ffff700 1 -- 172.21.15.60:0/2229222140 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f133c0b6e30 con 0x7f133c05f0b0 2022-06-29T16:06:13.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f132ffff700 1 -- 172.21.15.60:0/2229222140 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2227180123 0 0) 0x7f1320003260 con 0x7f133c0b6a50 2022-06-29T16:06:13.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f132ffff700 1 -- 172.21.15.60:0/2229222140 >> v2:172.21.15.92:3300/0 conn(0x7f133c05d160 msgr2=0x7f133c1461e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:13.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f132ffff700 1 --2- 172.21.15.60:0/2229222140 >> v2:172.21.15.92:3300/0 conn(0x7f133c05d160 0x7f133c1461e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:13.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f132ffff700 1 -- 172.21.15.60:0/2229222140 >> v1:172.21.15.92:6789/0 conn(0x7f133c05f0b0 legacy=0x7f133c05ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:13.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f132ffff700 1 -- 172.21.15.60:0/2229222140 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f133c05c5b0 con 0x7f133c0b6a50 2022-06-29T16:06:13.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f132ffff700 1 -- 172.21.15.60:0/2229222140 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f1330001660 con 0x7f133c0b6a50 2022-06-29T16:06:13.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f132ffff700 1 -- 172.21.15.60:0/2229222140 >> v1:172.21.15.60:6789/0 conn(0x7f133c0b6a50 legacy=0x7f133c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:13.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f132ffff700 1 --2- 172.21.15.60:0/2229222140 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13200038e0 0x7f1320003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:13.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f132ffff700 1 --2- 172.21.15.60:0/2229222140 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1320004500 0x7f1320004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:13.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f132ffff700 1 --2- 172.21.15.60:0/2229222140 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1320004e70 0x7f132000b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:13.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f132ffff700 1 -- 172.21.15.60:0/2229222140 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133c0b1070 con 0x7f1320004500 2022-06-29T16:06:13.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f132ffff700 1 -- 172.21.15.60:0/2229222140 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f132000bdd0 con 0x7f1320004e70 2022-06-29T16:06:13.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f132ffff700 1 -- 172.21.15.60:0/2229222140 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f132000bf40 con 0x7f13200038e0 2022-06-29T16:06:13.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f1341fff700 1 --2- 172.21.15.60:0/2229222140 >> v2:172.21.15.92:3300/0 conn(0x7f133c05d160 0x7f133c1461e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:06:13.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f1340ffd700 1 --2- 172.21.15.60:0/2229222140 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1320004500 0x7f1320004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:13.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f13417fe700 1 --2- 172.21.15.60:0/2229222140 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13200038e0 0x7f1320003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:13.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f1341fff700 1 --2- 172.21.15.60:0/2229222140 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1320004e70 0x7f132000b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:13.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f1340ffd700 1 -- 172.21.15.60:0/2229222140 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13200038e0 msgr2=0x7f1320003ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:13.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f1340ffd700 1 --2- 172.21.15.60:0/2229222140 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13200038e0 0x7f1320003ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:13.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f1340ffd700 1 -- 172.21.15.60:0/2229222140 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1320004e70 msgr2=0x7f132000b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:13.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f1340ffd700 1 --2- 172.21.15.60:0/2229222140 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1320004e70 0x7f132000b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:13.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f1340ffd700 1 -- 172.21.15.60:0/2229222140 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1320003260 con 0x7f1320004500 2022-06-29T16:06:13.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f1341fff700 1 --2- 172.21.15.60:0/2229222140 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1320004e70 0x7f132000b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:06:13.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f1341fff700 1 -- 172.21.15.60:0/2229222140 reap_dead start 2022-06-29T16:06:13.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f1340ffd700 1 --2- 172.21.15.60:0/2229222140 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1320004500 0x7f1320004980 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 rx=0x7f1338001110 tx=0x7f13380079c0).ready entity=mon.0 client_cookie=d573b24267acfa29 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:13.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f132ffff700 1 -- 172.21.15.60:0/2229222140 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1338007cd0 con 0x7f1320004500 2022-06-29T16:06:13.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f132ffff700 1 -- 172.21.15.60:0/2229222140 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1338007030 con 0x7f1320004500 2022-06-29T16:06:13.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.009+0000 7f132ffff700 1 -- 172.21.15.60:0/2229222140 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1338007390 con 0x7f1320004500 2022-06-29T16:06:13.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.013+0000 7f1343a62700 1 -- 172.21.15.60:0/2229222140 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1320004500 msgr2=0x7f1320004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:13.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.013+0000 7f1343a62700 1 --2- 172.21.15.60:0/2229222140 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1320004500 0x7f1320004980 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 rx=0x7f1338001110 tx=0x7f13380079c0).stop 2022-06-29T16:06:13.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.013+0000 7f1343a62700 1 -- 172.21.15.60:0/2229222140 shutdown_connections 2022-06-29T16:06:13.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.013+0000 7f1343a62700 1 --2- 172.21.15.60:0/2229222140 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1320004500 0x7f1320004980 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:13.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.013+0000 7f1343a62700 1 -- 172.21.15.60:0/2229222140 >> 172.21.15.60:0/2229222140 conn(0x7f133c0643e0 msgr2=0x7f133c0b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:13.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.013+0000 7f1343a62700 1 -- 172.21.15.60:0/2229222140 shutdown_connections 2022-06-29T16:06:13.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.013+0000 7f1343a62700 1 -- 172.21.15.60:0/2229222140 wait complete. 2022-06-29T16:06:13.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.013+0000 7f1343a62700 1 Processor -- start 2022-06-29T16:06:13.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.013+0000 7f1343a62700 1 -- start start 2022-06-29T16:06:13.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.017+0000 7f1343a62700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1320004500 0x7f133c144540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:13.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.017+0000 7f1343a62700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f133c0ad9e0 0x7f133c0654a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:13.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.017+0000 7f1343a62700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f133c0ad440 0x7f133c065990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:13.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.017+0000 7f1343a62700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133c144a30 con 0x7f1320004500 2022-06-29T16:06:13.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.017+0000 7f1343a62700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133c144c70 con 0x7f133c0ad9e0 2022-06-29T16:06:13.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.017+0000 7f1343a62700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f133c144f10 con 0x7f133c0ad440 2022-06-29T16:06:13.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.017+0000 7f13417fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1320004500 0x7f133c144540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:13.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.017+0000 7f13417fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1320004500 0x7f133c144540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51624/0 (socket says 172.21.15.60:51624) 2022-06-29T16:06:13.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.017+0000 7f13417fe700 1 -- 172.21.15.60:0/106867847 learned_addr learned my addr 172.21.15.60:0/106867847 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:13.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.017+0000 7f1340ffd700 1 --2- 172.21.15.60:0/106867847 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f133c0ad9e0 0x7f133c0654a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:13.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.017+0000 7f1341fff700 1 --2- 172.21.15.60:0/106867847 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f133c0ad440 0x7f133c065990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:13.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.017+0000 7f13417fe700 1 -- 172.21.15.60:0/106867847 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f133c0ad440 msgr2=0x7f133c065990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:13.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.017+0000 7f13417fe700 1 --2- 172.21.15.60:0/106867847 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f133c0ad440 0x7f133c065990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:13.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.017+0000 7f13417fe700 1 -- 172.21.15.60:0/106867847 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f133c0ad9e0 msgr2=0x7f133c0654a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:13.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.017+0000 7f13417fe700 1 --2- 172.21.15.60:0/106867847 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f133c0ad9e0 0x7f133c0654a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:13.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.017+0000 7f13417fe700 1 -- 172.21.15.60:0/106867847 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f133c1468a0 con 0x7f1320004500 2022-06-29T16:06:13.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.017+0000 7f1340ffd700 1 --2- 172.21.15.60:0/106867847 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f133c0ad9e0 0x7f133c0654a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:06:13.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.017+0000 7f13417fe700 1 --2- 172.21.15.60:0/106867847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1320004500 0x7f133c144540 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 rx=0x7f133000b000 tx=0x7f1330001e90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:13.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.017+0000 7f132e7fc700 1 -- 172.21.15.60:0/106867847 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1330010710 con 0x7f1320004500 2022-06-29T16:06:13.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.017+0000 7f132e7fc700 1 -- 172.21.15.60:0/106867847 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1330010870 con 0x7f1320004500 2022-06-29T16:06:13.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.017+0000 7f1343a62700 1 -- 172.21.15.60:0/106867847 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f133c065e80 con 0x7f1320004500 2022-06-29T16:06:13.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.017+0000 7f132e7fc700 1 -- 172.21.15.60:0/106867847 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1330010bd0 con 0x7f1320004500 2022-06-29T16:06:13.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.017+0000 7f1343a62700 1 -- 172.21.15.60:0/106867847 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f133c1557b0 con 0x7f1320004500 2022-06-29T16:06:13.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.017+0000 7f132e7fc700 1 -- 172.21.15.60:0/106867847 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f133001a020 con 0x7f1320004500 2022-06-29T16:06:13.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.021+0000 7f1343a62700 1 -- 172.21.15.60:0/106867847 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f133c0573d0 con 0x7f1320004500 2022-06-29T16:06:13.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.021+0000 7f132e7fc700 1 --2- 172.21.15.60:0/106867847 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1324053a70 0x7f1324055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:13.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.021+0000 7f132e7fc700 1 -- 172.21.15.60:0/106867847 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f133006bdc0 con 0x7f1320004500 2022-06-29T16:06:13.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.021+0000 7f1340ffd700 1 --2- 172.21.15.60:0/106867847 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1324053a70 0x7f1324055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:13.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.021+0000 7f1340ffd700 1 --2- 172.21.15.60:0/106867847 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1324053a70 0x7f1324055f30 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7f1338019000 tx=0x7f1338015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:13.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.025+0000 7f132e7fc700 1 -- 172.21.15.60:0/106867847 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1330042c40 con 0x7f1320004500 2022-06-29T16:06:13.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.153+0000 7f1343a62700 1 -- 172.21.15.60:0/106867847 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f133c067990 con 0x7f1320004500 2022-06-29T16:06:13.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.153+0000 7f132e7fc700 1 -- 172.21.15.60:0/106867847 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v100) v1 ==== 104+0+10 (secure 0 0 0) 0x7f1330046260 con 0x7f1320004500 2022-06-29T16:06:13.159 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:06:13.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.157+0000 7f1343a62700 1 -- 172.21.15.60:0/106867847 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1324053a70 msgr2=0x7f1324055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:13.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.157+0000 7f1343a62700 1 --2- 172.21.15.60:0/106867847 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1324053a70 0x7f1324055f30 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7f1338019000 tx=0x7f1338015000).stop 2022-06-29T16:06:13.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.157+0000 7f1343a62700 1 -- 172.21.15.60:0/106867847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1320004500 msgr2=0x7f133c144540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:13.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.157+0000 7f1343a62700 1 --2- 172.21.15.60:0/106867847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1320004500 0x7f133c144540 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 rx=0x7f133000b000 tx=0x7f1330001e90).stop 2022-06-29T16:06:13.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.157+0000 7f1343a62700 1 -- 172.21.15.60:0/106867847 shutdown_connections 2022-06-29T16:06:13.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.157+0000 7f1343a62700 1 --2- 172.21.15.60:0/106867847 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1324053a70 0x7f1324055f30 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:13.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.157+0000 7f1343a62700 1 --2- 172.21.15.60:0/106867847 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f133c0ad440 0x7f133c065990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:13.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.157+0000 7f1343a62700 1 --2- 172.21.15.60:0/106867847 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f133c0ad9e0 0x7f133c0654a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:13.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.157+0000 7f1343a62700 1 --2- 172.21.15.60:0/106867847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1320004500 0x7f133c144540 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:13.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.157+0000 7f1343a62700 1 -- 172.21.15.60:0/106867847 >> 172.21.15.60:0/106867847 conn(0x7f133c0643e0 msgr2=0x7f133c1420c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:13.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.157+0000 7f1343a62700 1 -- 172.21.15.60:0/106867847 shutdown_connections 2022-06-29T16:06:13.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.157+0000 7f1343a62700 1 -- 172.21.15.60:0/106867847 wait complete. 2022-06-29T16:06:13.178 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-29T16:06:13.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.285+0000 7f3f3b59e700 1 Processor -- start 2022-06-29T16:06:13.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.285+0000 7f3f3b59e700 1 -- start start 2022-06-29T16:06:13.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.285+0000 7f3f3b59e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3f3c0b3360 0x7f3f3c142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:13.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.285+0000 7f3f3b59e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3c0b0cf0 con 0x7f3f3c0b5eb0 2022-06-29T16:06:13.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.285+0000 7f3f3b59e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3c0b4d10 con 0x7f3f3c0b48b0 2022-06-29T16:06:13.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.285+0000 7f3f3b59e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f3c0b54e0 con 0x7f3f3c0b3360 2022-06-29T16:06:13.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.285+0000 7f3f3a59c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f3f3c0b5eb0 0x7f3f3c0b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59718/0 (socket says 172.21.15.60:59718) 2022-06-29T16:06:13.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.285+0000 7f3f3a59c700 1 -- 172.21.15.60:0/1249148176 learned_addr learned my addr 172.21.15.60:0/1249148176 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:13.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.285+0000 7f3f3ad9d700 1 --2- 172.21.15.60:0/1249148176 >> v2:172.21.15.92:3300/0 conn(0x7f3f3c0b3360 0x7f3f3c142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:13.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.289+0000 7f3f3959a700 1 -- 172.21.15.60:0/1249148176 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3456275573 0 0) 0x7f3f2c001110 con 0x7f3f3c0b5eb0 2022-06-29T16:06:13.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.289+0000 7f3f3959a700 1 -- 172.21.15.60:0/1249148176 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f1c003260 con 0x7f3f3c0b5eb0 2022-06-29T16:06:13.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.289+0000 7f3f3959a700 1 -- 172.21.15.60:0/1249148176 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1735561641 0 0) 0x7f3f3c0b4d10 con 0x7f3f3c0b48b0 2022-06-29T16:06:13.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.289+0000 7f3f3959a700 1 -- 172.21.15.60:0/1249148176 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f2c001110 con 0x7f3f3c0b48b0 2022-06-29T16:06:13.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.289+0000 7f3f3ad9d700 1 -- 172.21.15.60:0/1249148176 >> v1:172.21.15.92:6789/0 conn(0x7f3f3c0b48b0 legacy=0x7f3f3c0b2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:13.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.289+0000 7f3f3ad9d700 1 -- 172.21.15.60:0/1249148176 >> v1:172.21.15.60:6789/0 conn(0x7f3f3c0b5eb0 legacy=0x7f3f3c0b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:13.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.289+0000 7f3f3ad9d700 1 -- 172.21.15.60:0/1249148176 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3c0b6290 con 0x7f3f3c0b3360 2022-06-29T16:06:13.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.289+0000 7f3f3ad9d700 1 --2- 172.21.15.60:0/1249148176 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f3c0b3360 0x7f3f3c142a20 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7f3f340032f0 tx=0x7f3f340091d0).ready entity=mon.1 client_cookie=4043de724e73eb78 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:13.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.289+0000 7f3f3959a700 1 -- 172.21.15.60:0/1249148176 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f3400b600 con 0x7f3f3c0b3360 2022-06-29T16:06:13.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.289+0000 7f3f3959a700 1 -- 172.21.15.60:0/1249148176 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f3400b760 con 0x7f3f3c0b3360 2022-06-29T16:06:13.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.289+0000 7f3f3959a700 1 -- 172.21.15.60:0/1249148176 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f3400b8d0 con 0x7f3f3c0b3360 2022-06-29T16:06:13.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.289+0000 7f3f3b59e700 1 -- 172.21.15.60:0/1249148176 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f3c0b3360 msgr2=0x7f3f3c142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:13.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.289+0000 7f3f3b59e700 1 --2- 172.21.15.60:0/1249148176 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f3c0b3360 0x7f3f3c142a20 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7f3f340032f0 tx=0x7f3f340091d0).stop 2022-06-29T16:06:13.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.289+0000 7f3f3b59e700 1 -- 172.21.15.60:0/1249148176 shutdown_connections 2022-06-29T16:06:13.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.289+0000 7f3f3b59e700 1 --2- 172.21.15.60:0/1249148176 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f3c0b3360 0x7f3f3c142a20 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:13.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.289+0000 7f3f3b59e700 1 -- 172.21.15.60:0/1249148176 >> 172.21.15.60:0/1249148176 conn(0x7f3f3c062090 msgr2=0x7f3f3c062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:13.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.289+0000 7f3f3b59e700 1 -- 172.21.15.60:0/1249148176 shutdown_connections 2022-06-29T16:06:13.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.289+0000 7f3f3b59e700 1 -- 172.21.15.60:0/1249148176 wait complete. 2022-06-29T16:06:13.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.289+0000 7f3f3b59e700 1 Processor -- start 2022-06-29T16:06:13.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.289+0000 7f3f3b59e700 1 -- start start 2022-06-29T16:06:13.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.293+0000 7f3f3b59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f3c0b3360 0x7f3f3c152690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:13.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.293+0000 7f3f3b59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f3c0b48b0 0x7f3f3c0af250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:13.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.293+0000 7f3f3b59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f3c0b5eb0 0x7f3f3c0aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:13.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.293+0000 7f3f3b59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c0ae1e0 con 0x7f3f3c0b3360 2022-06-29T16:06:13.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.293+0000 7f3f3b59e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c13ed70 con 0x7f3f3c0b5eb0 2022-06-29T16:06:13.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.293+0000 7f3f3b59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c144060 con 0x7f3f3c0b48b0 2022-06-29T16:06:13.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.293+0000 7f3f3a59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f3c0b3360 0x7f3f3c152690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:13.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.293+0000 7f3f3a59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f3c0b3360 0x7f3f3c152690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51640/0 (socket says 172.21.15.60:51640) 2022-06-29T16:06:13.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.293+0000 7f3f3a59c700 1 -- 172.21.15.60:0/1469221408 learned_addr learned my addr 172.21.15.60:0/1469221408 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:13.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.293+0000 7f3f39d9b700 1 --2- 172.21.15.60:0/1469221408 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f3c0b48b0 0x7f3f3c0af250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:13.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.293+0000 7f3f3ad9d700 1 --2- 172.21.15.60:0/1469221408 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f3c0b5eb0 0x7f3f3c0aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:13.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.293+0000 7f3f3a59c700 1 -- 172.21.15.60:0/1469221408 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f3c0b48b0 msgr2=0x7f3f3c0af250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:13.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.293+0000 7f3f3a59c700 1 --2- 172.21.15.60:0/1469221408 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f3c0b48b0 0x7f3f3c0af250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:13.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.293+0000 7f3f3a59c700 1 -- 172.21.15.60:0/1469221408 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f3c0b5eb0 msgr2=0x7f3f3c0aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:13.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.293+0000 7f3f3a59c700 1 --2- 172.21.15.60:0/1469221408 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f3c0b5eb0 0x7f3f3c0aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:13.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.293+0000 7f3f3a59c700 1 -- 172.21.15.60:0/1469221408 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f34009470 con 0x7f3f3c0b3360 2022-06-29T16:06:13.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.293+0000 7f3f3a59c700 1 --2- 172.21.15.60:0/1469221408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f3c0b3360 0x7f3f3c152690 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 rx=0x7f3f2c001360 tx=0x7f3f2c007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:13.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.293+0000 7f3f337fe700 1 -- 172.21.15.60:0/1469221408 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f2c0070d0 con 0x7f3f3c0b3360 2022-06-29T16:06:13.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.293+0000 7f3f337fe700 1 -- 172.21.15.60:0/1469221408 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f2c007230 con 0x7f3f3c0b3360 2022-06-29T16:06:13.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.293+0000 7f3f3b59e700 1 -- 172.21.15.60:0/1469221408 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f3c152c60 con 0x7f3f3c0b3360 2022-06-29T16:06:13.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.293+0000 7f3f337fe700 1 -- 172.21.15.60:0/1469221408 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f2c005610 con 0x7f3f3c0b3360 2022-06-29T16:06:13.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.293+0000 7f3f3b59e700 1 -- 172.21.15.60:0/1469221408 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f3c13fa80 con 0x7f3f3c0b3360 2022-06-29T16:06:13.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.297+0000 7f3f337fe700 1 -- 172.21.15.60:0/1469221408 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3f2c01a060 con 0x7f3f3c0b3360 2022-06-29T16:06:13.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.297+0000 7f3f3b59e700 1 -- 172.21.15.60:0/1469221408 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f3c13ef70 con 0x7f3f3c0b3360 2022-06-29T16:06:13.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.297+0000 7f3f337fe700 1 --2- 172.21.15.60:0/1469221408 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3f20053a70 0x7f3f20055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:13.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.297+0000 7f3f337fe700 1 -- 172.21.15.60:0/1469221408 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3f2c06b8a0 con 0x7f3f3c0b3360 2022-06-29T16:06:13.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.297+0000 7f3f39d9b700 1 --2- 172.21.15.60:0/1469221408 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3f20053a70 0x7f3f20055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:13.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.297+0000 7f3f39d9b700 1 --2- 172.21.15.60:0/1469221408 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3f20053a70 0x7f3f20055f30 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7f3f2400d000 tx=0x7f3f24007a20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:13.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.305+0000 7f3f337fe700 1 -- 172.21.15.60:0/1469221408 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3f2c0426b0 con 0x7f3f3c0b3360 2022-06-29T16:06:13.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.433+0000 7f3f3b59e700 1 -- 172.21.15.60:0/1469221408 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3f3c0b0cf0 con 0x7f3f3c0b3360 2022-06-29T16:06:13.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.433+0000 7f3f337fe700 1 -- 172.21.15.60:0/1469221408 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v100) v1 ==== 100+0+10 (secure 0 0 0) 0x7f3f3c0b0cf0 con 0x7f3f3c0b3360 2022-06-29T16:06:13.439 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:06:13.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.437+0000 7f3f3b59e700 1 -- 172.21.15.60:0/1469221408 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3f20053a70 msgr2=0x7f3f20055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:13.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.437+0000 7f3f3b59e700 1 --2- 172.21.15.60:0/1469221408 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3f20053a70 0x7f3f20055f30 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7f3f2400d000 tx=0x7f3f24007a20).stop 2022-06-29T16:06:13.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.437+0000 7f3f3b59e700 1 -- 172.21.15.60:0/1469221408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f3c0b3360 msgr2=0x7f3f3c152690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:13.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.437+0000 7f3f3b59e700 1 --2- 172.21.15.60:0/1469221408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f3c0b3360 0x7f3f3c152690 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 rx=0x7f3f2c001360 tx=0x7f3f2c007b50).stop 2022-06-29T16:06:13.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.437+0000 7f3f3b59e700 1 -- 172.21.15.60:0/1469221408 shutdown_connections 2022-06-29T16:06:13.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.437+0000 7f3f3b59e700 1 --2- 172.21.15.60:0/1469221408 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3f20053a70 0x7f3f20055f30 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:13.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.437+0000 7f3f3b59e700 1 --2- 172.21.15.60:0/1469221408 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3f3c0b5eb0 0x7f3f3c0aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:13.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.437+0000 7f3f3b59e700 1 --2- 172.21.15.60:0/1469221408 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3f3c0b48b0 0x7f3f3c0af250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:13.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.437+0000 7f3f3b59e700 1 --2- 172.21.15.60:0/1469221408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f3c0b3360 0x7f3f3c152690 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:13.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.437+0000 7f3f3b59e700 1 -- 172.21.15.60:0/1469221408 >> 172.21.15.60:0/1469221408 conn(0x7f3f3c062090 msgr2=0x7f3f3c0b7dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:13.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.437+0000 7f3f3b59e700 1 -- 172.21.15.60:0/1469221408 shutdown_connections 2022-06-29T16:06:13.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.437+0000 7f3f3b59e700 1 -- 172.21.15.60:0/1469221408 wait complete. 2022-06-29T16:06:13.453 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-06-29T16:06:13.453 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:06:13.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.549+0000 7fa2370b0700 1 Processor -- start 2022-06-29T16:06:13.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.549+0000 7fa2370b0700 1 -- start start 2022-06-29T16:06:13.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.553+0000 7fa2370b0700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa2300b5950 0x7fa230067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:13.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.553+0000 7fa2370b0700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa230063ad0 0x7fa23014af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:13.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.553+0000 7fa2370b0700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2300acba0 con 0x7fa23005bd20 2022-06-29T16:06:13.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.553+0000 7fa2370b0700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa230067220 con 0x7fa230063ad0 2022-06-29T16:06:13.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.553+0000 7fa2370b0700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2300b5d30 con 0x7fa2300b5950 2022-06-29T16:06:13.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.553+0000 7fa234e4c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa2300b5950 0x7fa230067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:13.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.553+0000 7fa23564d700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fa23005bd20 0x7fa23005c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:35750/0 (socket says 172.21.15.60:35750) 2022-06-29T16:06:13.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.553+0000 7fa23564d700 1 -- 172.21.15.60:0/4203868182 learned_addr learned my addr 172.21.15.60:0/4203868182 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:13.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.553+0000 7fa22ffff700 1 --2- 172.21.15.60:0/4203868182 >> v2:172.21.15.92:3300/0 conn(0x7fa230063ad0 0x7fa23014af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:13.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.553+0000 7fa234e4c700 1 -- 172.21.15.60:0/4203868182 >> v2:172.21.15.92:3300/0 conn(0x7fa230063ad0 msgr2=0x7fa23014af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:13.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.553+0000 7fa22f7fe700 1 -- 172.21.15.60:0/4203868182 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4271362324 0 0) 0x7fa2300acba0 con 0x7fa23005bd20 2022-06-29T16:06:13.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.553+0000 7fa234e4c700 1 --2- 172.21.15.60:0/4203868182 >> v2:172.21.15.92:3300/0 conn(0x7fa230063ad0 0x7fa23014af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:13.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.553+0000 7fa234e4c700 1 -- 172.21.15.60:0/4203868182 >> v1:172.21.15.92:6789/0 conn(0x7fa23005bd20 legacy=0x7fa23005c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:13.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.553+0000 7fa234e4c700 1 -- 172.21.15.60:0/4203868182 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2300ace00 con 0x7fa2300b5950 2022-06-29T16:06:13.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.553+0000 7fa234e4c700 1 --2- 172.21.15.60:0/4203868182 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa2300b5950 0x7fa230067680 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fa2280032f0 tx=0x7fa2280091d0).ready entity=mon.2 client_cookie=255d88d42a0eb2e8 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:13.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.553+0000 7fa22f7fe700 1 -- 172.21.15.60:0/4203868182 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa22800b6c0 con 0x7fa2300b5950 2022-06-29T16:06:13.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.553+0000 7fa22f7fe700 1 -- 172.21.15.60:0/4203868182 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa22800b820 con 0x7fa2300b5950 2022-06-29T16:06:13.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.553+0000 7fa22f7fe700 1 -- 172.21.15.60:0/4203868182 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa22800b990 con 0x7fa2300b5950 2022-06-29T16:06:13.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.553+0000 7fa2370b0700 1 -- 172.21.15.60:0/4203868182 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa2300b5950 msgr2=0x7fa230067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:13.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.553+0000 7fa2370b0700 1 --2- 172.21.15.60:0/4203868182 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa2300b5950 0x7fa230067680 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fa2280032f0 tx=0x7fa2280091d0).stop 2022-06-29T16:06:13.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.553+0000 7fa2370b0700 1 -- 172.21.15.60:0/4203868182 shutdown_connections 2022-06-29T16:06:13.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.553+0000 7fa2370b0700 1 --2- 172.21.15.60:0/4203868182 >> v2:172.21.15.92:3300/0 conn(0x7fa230063ad0 0x7fa23014af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:13.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.553+0000 7fa2370b0700 1 --2- 172.21.15.60:0/4203868182 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa2300b5950 0x7fa230067680 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:13.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.553+0000 7fa2370b0700 1 -- 172.21.15.60:0/4203868182 >> 172.21.15.60:0/4203868182 conn(0x7fa2300632c0 msgr2=0x7fa2300b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:13.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.553+0000 7fa2370b0700 1 -- 172.21.15.60:0/4203868182 shutdown_connections 2022-06-29T16:06:13.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.553+0000 7fa2370b0700 1 -- 172.21.15.60:0/4203868182 wait complete. 2022-06-29T16:06:13.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.557+0000 7fa2370b0700 1 Processor -- start 2022-06-29T16:06:13.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.557+0000 7fa2370b0700 1 -- start start 2022-06-29T16:06:13.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.557+0000 7fa2370b0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa23005bd20 0x7fa2300b3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:13.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.557+0000 7fa2370b0700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa230063ad0 0x7fa2300acfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:13.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.557+0000 7fa2370b0700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa2300b5950 0x7fa2300af400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:13.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.557+0000 7fa2370b0700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2300ae2d0 con 0x7fa23005bd20 2022-06-29T16:06:13.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.557+0000 7fa2370b0700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa230157090 con 0x7fa2300b5950 2022-06-29T16:06:13.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.557+0000 7fa2370b0700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2300aecc0 con 0x7fa230063ad0 2022-06-29T16:06:13.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.557+0000 7fa234e4c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa23005bd20 0x7fa2300b3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:13.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.557+0000 7fa234e4c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa23005bd20 0x7fa2300b3070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51652/0 (socket says 172.21.15.60:51652) 2022-06-29T16:06:13.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.557+0000 7fa234e4c700 1 -- 172.21.15.60:0/3847171334 learned_addr learned my addr 172.21.15.60:0/3847171334 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:13.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.557+0000 7fa23564d700 1 --2- 172.21.15.60:0/3847171334 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa2300b5950 0x7fa2300af400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:13.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.557+0000 7fa22ffff700 1 --2- 172.21.15.60:0/3847171334 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa230063ad0 0x7fa2300acfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:13.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.561+0000 7fa234e4c700 1 -- 172.21.15.60:0/3847171334 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa230063ad0 msgr2=0x7fa2300acfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:13.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.561+0000 7fa234e4c700 1 --2- 172.21.15.60:0/3847171334 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa230063ad0 0x7fa2300acfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:13.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.561+0000 7fa234e4c700 1 -- 172.21.15.60:0/3847171334 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa2300b5950 msgr2=0x7fa2300af400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:13.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.561+0000 7fa234e4c700 1 --2- 172.21.15.60:0/3847171334 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa2300b5950 0x7fa2300af400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:13.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.561+0000 7fa234e4c700 1 -- 172.21.15.60:0/3847171334 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa228009470 con 0x7fa23005bd20 2022-06-29T16:06:13.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.561+0000 7fa234e4c700 1 --2- 172.21.15.60:0/3847171334 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa23005bd20 0x7fa2300b3070 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 rx=0x7fa2280032f0 tx=0x7fa228016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:13.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.561+0000 7fa22dffb700 1 -- 172.21.15.60:0/3847171334 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa22800b990 con 0x7fa23005bd20 2022-06-29T16:06:13.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.561+0000 7fa22dffb700 1 -- 172.21.15.60:0/3847171334 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa228021630 con 0x7fa23005bd20 2022-06-29T16:06:13.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.561+0000 7fa22dffb700 1 -- 172.21.15.60:0/3847171334 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa22800b6c0 con 0x7fa23005bd20 2022-06-29T16:06:13.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.561+0000 7fa2370b0700 1 -- 172.21.15.60:0/3847171334 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2300aef60 con 0x7fa23005bd20 2022-06-29T16:06:13.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.561+0000 7fa2370b0700 1 -- 172.21.15.60:0/3847171334 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa230156c60 con 0x7fa23005bd20 2022-06-29T16:06:13.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.561+0000 7fa2370b0700 1 -- 172.21.15.60:0/3847171334 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2300b3c30 con 0x7fa23005bd20 2022-06-29T16:06:13.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.561+0000 7fa22dffb700 1 -- 172.21.15.60:0/3847171334 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa228029070 con 0x7fa23005bd20 2022-06-29T16:06:13.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.561+0000 7fa22dffb700 1 --2- 172.21.15.60:0/3847171334 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa214053a70 0x7fa214055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:13.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.561+0000 7fa22dffb700 1 -- 172.21.15.60:0/3847171334 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa22806ce80 con 0x7fa23005bd20 2022-06-29T16:06:13.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.561+0000 7fa22ffff700 1 --2- 172.21.15.60:0/3847171334 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa214053a70 0x7fa214055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:13.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.565+0000 7fa22ffff700 1 --2- 172.21.15.60:0/3847171334 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa214053a70 0x7fa214055f30 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0x7fa21800b000 tx=0x7fa218006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:13.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.569+0000 7fa22dffb700 1 -- 172.21.15.60:0/3847171334 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa2300b3c30 con 0x7fa23005bd20 2022-06-29T16:06:13.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.733+0000 7fa2370b0700 1 -- 172.21.15.60:0/3847171334 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2300b3c30 con 0x7fa23005bd20 2022-06-29T16:06:13.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.733+0000 7fa22dffb700 1 -- 172.21.15.60:0/3847171334 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+2140 (secure 0 0 0) 0x7fa2300b3c30 con 0x7fa23005bd20 2022-06-29T16:06:13.738 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:13.738 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":46,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7320,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/3498160620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3498160620},{"type":"v1","addr":"172.21.15.60:6833","nonce":3498160620}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:06:05.226825+0000","modified":"2022-06-29T16:06:06.241973+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7327},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7327":{"gid":7327,"name":"b","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/1551394182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1551394182},{"type":"v1","addr":"172.21.15.92:6835","nonce":1551394182}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2022-06-29T16:06:13.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.733+0000 7fa2370b0700 1 -- 172.21.15.60:0/3847171334 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa214053a70 msgr2=0x7fa214055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:13.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.733+0000 7fa2370b0700 1 --2- 172.21.15.60:0/3847171334 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa214053a70 0x7fa214055f30 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0x7fa21800b000 tx=0x7fa218006df0).stop 2022-06-29T16:06:13.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.733+0000 7fa2370b0700 1 -- 172.21.15.60:0/3847171334 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa23005bd20 msgr2=0x7fa2300b3070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:13.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.733+0000 7fa2370b0700 1 --2- 172.21.15.60:0/3847171334 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa23005bd20 0x7fa2300b3070 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 rx=0x7fa2280032f0 tx=0x7fa228016000).stop 2022-06-29T16:06:13.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.733+0000 7fa2370b0700 1 -- 172.21.15.60:0/3847171334 shutdown_connections 2022-06-29T16:06:13.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.733+0000 7fa2370b0700 1 --2- 172.21.15.60:0/3847171334 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa214053a70 0x7fa214055f30 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:13.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.733+0000 7fa2370b0700 1 --2- 172.21.15.60:0/3847171334 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa2300b5950 0x7fa2300af400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:13.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.733+0000 7fa2370b0700 1 --2- 172.21.15.60:0/3847171334 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa230063ad0 0x7fa2300acfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:13.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.733+0000 7fa2370b0700 1 --2- 172.21.15.60:0/3847171334 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa23005bd20 0x7fa2300b3070 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:13.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.733+0000 7fa2370b0700 1 -- 172.21.15.60:0/3847171334 >> 172.21.15.60:0/3847171334 conn(0x7fa2300632c0 msgr2=0x7fa2300651a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:13.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.733+0000 7fa2370b0700 1 -- 172.21.15.60:0/3847171334 shutdown_connections 2022-06-29T16:06:13.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:13.733+0000 7fa2370b0700 1 -- 172.21.15.60:0/3847171334 wait complete. 2022-06-29T16:06:13.743 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 46 2022-06-29T16:06:13.952 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 46, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:06:05.226825+0000', 'modified': '2022-06-29T16:06:06.241973+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7327}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7327': {'gid': 7327, 'name': 'b', 'rank': 0, 'incarnation': 45, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.92:6835/1551394182', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 1551394182}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 1551394182}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:06:13.953 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:06:13.954 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-06-29T16:06:14.072 INFO:teuthology.orchestra.run.smithi092.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":7327,"want_state":"up:active","state":"up:active","rank_uptime":8.8282788199999995,"mdsmap_epoch":46,"osdmap_epoch":100,"osdmap_epoch_barrier":100,"uptime":13.205001518} 2022-06-29T16:06:14.073 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 7327, "mdsmap_epoch": 46, "osdmap_epoch": 100, "osdmap_epoch_barrier": 100, "rank_uptime": 8.82827882, "state": "up:active", "uptime": 13.205001518, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:06:14.074 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-06-29T16:06:14.074 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-06-29T16:06:14.075 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi156.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-06-29T16:06:14.077 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:14.077 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:06:14.087 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2022-06-29T16:06:14.145 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:06:14.146 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:06:14.160 INFO:teuthology.orchestra.run.smithi156.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:06:14.161 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:06:14.163 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:14.163 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2022-06-29T16:06:14.214 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-06-29T16:06:14.215 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:06:14.218 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2022-06-29T16:06:14.236 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:06:14.237 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:06:14.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:2022-06-29T16:06:14.282+0000 7f5e4573b080 -1 init, newargv = 0x55b39fe24170 newargc=9 2022-06-29T16:06:14.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[12654]: starting ceph client 2022-06-29T16:06:14.293 INFO:teuthology.orchestra.run.smithi156.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:06:14.295 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:06:14.296 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:14.296 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2022-06-29T16:06:14.300 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[12654]: starting fuse 2022-06-29T16:06:14.304 INFO:teuthology.orchestra.run.smithi156.stdout:52 2022-06-29T16:06:14.304 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-06-29T16:06:14.305 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:14.305 DEBUG:teuthology.orchestra.run.smithi156:> sudo python3 -c ' 2022-06-29T16:06:14.306 DEBUG:teuthology.orchestra.run.smithi156:> import glob 2022-06-29T16:06:14.306 DEBUG:teuthology.orchestra.run.smithi156:> import re 2022-06-29T16:06:14.307 DEBUG:teuthology.orchestra.run.smithi156:> import os 2022-06-29T16:06:14.307 DEBUG:teuthology.orchestra.run.smithi156:> import subprocess 2022-06-29T16:06:14.308 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:06:14.308 DEBUG:teuthology.orchestra.run.smithi156:> def find_socket(client_name): 2022-06-29T16:06:14.308 DEBUG:teuthology.orchestra.run.smithi156:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-06-29T16:06:14.309 DEBUG:teuthology.orchestra.run.smithi156:> files = glob.glob(asok_path) 2022-06-29T16:06:14.309 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:06:14.310 DEBUG:teuthology.orchestra.run.smithi156:> # Given a non-glob path, it better be there 2022-06-29T16:06:14.310 DEBUG:teuthology.orchestra.run.smithi156:> if "*" not in asok_path: 2022-06-29T16:06:14.311 DEBUG:teuthology.orchestra.run.smithi156:> assert(len(files) == 1) 2022-06-29T16:06:14.311 DEBUG:teuthology.orchestra.run.smithi156:> return files[0] 2022-06-29T16:06:14.312 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:06:14.312 DEBUG:teuthology.orchestra.run.smithi156:> for f in files: 2022-06-29T16:06:14.312 DEBUG:teuthology.orchestra.run.smithi156:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-06-29T16:06:14.313 DEBUG:teuthology.orchestra.run.smithi156:> if os.path.exists("/proc/{0}".format(pid)): 2022-06-29T16:06:14.313 DEBUG:teuthology.orchestra.run.smithi156:> return f 2022-06-29T16:06:14.314 DEBUG:teuthology.orchestra.run.smithi156:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-06-29T16:06:14.314 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:06:14.314 DEBUG:teuthology.orchestra.run.smithi156:> print(find_socket("client.1")) 2022-06-29T16:06:14.315 DEBUG:teuthology.orchestra.run.smithi156:> ' 2022-06-29T16:06:14.358 INFO:teuthology.orchestra.run.smithi156.stdout:/var/run/ceph/ceph-client.1.12654.asok 2022-06-29T16:06:14.363 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.12654.asok 2022-06-29T16:06:14.364 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:14.365 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.12654.asok status 2022-06-29T16:06:14.499 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2022-06-29T16:06:14.499 INFO:teuthology.orchestra.run.smithi156.stdout: "metadata": { 2022-06-29T16:06:14.499 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_sha1": "f25ffac72e5a9e63860ff7deb6767a11f0559a16", 2022-06-29T16:06:14.500 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_version": "ceph version 15.2.16-304-gf25ffac7 (f25ffac72e5a9e63860ff7deb6767a11f0559a16) octopus (stable)", 2022-06-29T16:06:14.500 INFO:teuthology.orchestra.run.smithi156.stdout: "entity_id": "1", 2022-06-29T16:06:14.500 INFO:teuthology.orchestra.run.smithi156.stdout: "hostname": "smithi156", 2022-06-29T16:06:14.500 INFO:teuthology.orchestra.run.smithi156.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-06-29T16:06:14.501 INFO:teuthology.orchestra.run.smithi156.stdout: "pid": "12654", 2022-06-29T16:06:14.501 INFO:teuthology.orchestra.run.smithi156.stdout: "root": "/" 2022-06-29T16:06:14.502 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:06:14.502 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_count": 0, 2022-06-29T16:06:14.502 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_pinned_count": 0, 2022-06-29T16:06:14.502 INFO:teuthology.orchestra.run.smithi156.stdout: "id": 7528, 2022-06-29T16:06:14.503 INFO:teuthology.orchestra.run.smithi156.stdout: "inst": { 2022-06-29T16:06:14.503 INFO:teuthology.orchestra.run.smithi156.stdout: "name": { 2022-06-29T16:06:14.503 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "client", 2022-06-29T16:06:14.504 INFO:teuthology.orchestra.run.smithi156.stdout: "num": 7528 2022-06-29T16:06:14.504 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:06:14.504 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2022-06-29T16:06:14.504 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2022-06-29T16:06:14.505 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "172.21.15.156:0", 2022-06-29T16:06:14.505 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 3993090708 2022-06-29T16:06:14.505 INFO:teuthology.orchestra.run.smithi156.stdout: } 2022-06-29T16:06:14.505 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:06:14.506 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2022-06-29T16:06:14.506 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2022-06-29T16:06:14.506 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "172.21.15.156:0", 2022-06-29T16:06:14.506 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 3993090708 2022-06-29T16:06:14.506 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:06:14.507 INFO:teuthology.orchestra.run.smithi156.stdout: "inst_str": "client.7528 172.21.15.156:0/3993090708", 2022-06-29T16:06:14.507 INFO:teuthology.orchestra.run.smithi156.stdout: "addr_str": "172.21.15.156:0/3993090708", 2022-06-29T16:06:14.507 INFO:teuthology.orchestra.run.smithi156.stdout: "inode_count": 1, 2022-06-29T16:06:14.507 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_epoch": 46, 2022-06-29T16:06:14.508 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch": 100, 2022-06-29T16:06:14.508 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch_barrier": 0, 2022-06-29T16:06:14.508 INFO:teuthology.orchestra.run.smithi156.stdout: "blacklisted": false, 2022-06-29T16:06:14.508 INFO:teuthology.orchestra.run.smithi156.stdout: "fs_name": "cephfs" 2022-06-29T16:06:14.509 INFO:teuthology.orchestra.run.smithi156.stdout:} 2022-06-29T16:06:14.510 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:14.511 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:06:14.511 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:06:14.519 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2022-06-29T16:06:14.521 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-06-29T16:06:14.522 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:14.522 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:06:14.541 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2022-06-29T16:06:14.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.649+0000 7fb62d68d700 1 Processor -- start 2022-06-29T16:06:14.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.649+0000 7fb62d68d700 1 -- start start 2022-06-29T16:06:14.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.649+0000 7fb62d68d700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb628063ad0 0x7fb628067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:14.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.649+0000 7fb62d68d700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6280acba0 con 0x7fb6280b5950 2022-06-29T16:06:14.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.649+0000 7fb62d68d700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb628067220 con 0x7fb62814af10 2022-06-29T16:06:14.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.649+0000 7fb62d68d700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb62814b3a0 con 0x7fb628063ad0 2022-06-29T16:06:14.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.649+0000 7fb626ffd700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb6280b5950 0x7fb62805bd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59746/0 (socket says 172.21.15.60:59746) 2022-06-29T16:06:14.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.649+0000 7fb626ffd700 1 -- 172.21.15.60:0/220493324 learned_addr learned my addr 172.21.15.60:0/220493324 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:14.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.649+0000 7fb6267fc700 1 --2- 172.21.15.60:0/220493324 >> v2:172.21.15.92:3300/0 conn(0x7fb628063ad0 0x7fb628067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:14.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.649+0000 7fb625ffb700 1 -- 172.21.15.60:0/220493324 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 687745949 0 0) 0x7fb6280acba0 con 0x7fb6280b5950 2022-06-29T16:06:14.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.649+0000 7fb625ffb700 1 -- 172.21.15.60:0/220493324 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb60c003260 con 0x7fb6280b5950 2022-06-29T16:06:14.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.649+0000 7fb6267fc700 1 -- 172.21.15.60:0/220493324 >> v1:172.21.15.92:6789/0 conn(0x7fb62814af10 legacy=0x7fb62805c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:14.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.649+0000 7fb6267fc700 1 -- 172.21.15.60:0/220493324 >> v1:172.21.15.60:6789/0 conn(0x7fb6280b5950 legacy=0x7fb62805bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:14.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.649+0000 7fb6267fc700 1 -- 172.21.15.60:0/220493324 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6280b5d30 con 0x7fb628063ad0 2022-06-29T16:06:14.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.649+0000 7fb6267fc700 1 --2- 172.21.15.60:0/220493324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb628063ad0 0x7fb628067680 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7fb6180032f0 tx=0x7fb6180091d0).ready entity=mon.1 client_cookie=8bd41e5876da6a0e server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:14.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.649+0000 7fb625ffb700 1 -- 172.21.15.60:0/220493324 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb61800b6c0 con 0x7fb628063ad0 2022-06-29T16:06:14.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.649+0000 7fb625ffb700 1 -- 172.21.15.60:0/220493324 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb61800b820 con 0x7fb628063ad0 2022-06-29T16:06:14.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.649+0000 7fb625ffb700 1 -- 172.21.15.60:0/220493324 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb61800b990 con 0x7fb628063ad0 2022-06-29T16:06:14.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.653+0000 7fb62d68d700 1 -- 172.21.15.60:0/220493324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb628063ad0 msgr2=0x7fb628067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:14.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.653+0000 7fb62d68d700 1 --2- 172.21.15.60:0/220493324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb628063ad0 0x7fb628067680 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7fb6180032f0 tx=0x7fb6180091d0).stop 2022-06-29T16:06:14.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.653+0000 7fb62d68d700 1 -- 172.21.15.60:0/220493324 shutdown_connections 2022-06-29T16:06:14.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.653+0000 7fb62d68d700 1 --2- 172.21.15.60:0/220493324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb628063ad0 0x7fb628067680 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:14.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.653+0000 7fb62d68d700 1 -- 172.21.15.60:0/220493324 >> 172.21.15.60:0/220493324 conn(0x7fb6280632c0 msgr2=0x7fb6280b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:14.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.653+0000 7fb62d68d700 1 -- 172.21.15.60:0/220493324 shutdown_connections 2022-06-29T16:06:14.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.653+0000 7fb62d68d700 1 -- 172.21.15.60:0/220493324 wait complete. 2022-06-29T16:06:14.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.653+0000 7fb62d68d700 1 Processor -- start 2022-06-29T16:06:14.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.653+0000 7fb62d68d700 1 -- start start 2022-06-29T16:06:14.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.657+0000 7fb62d68d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb628063ad0 0x7fb6280b0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:14.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.657+0000 7fb62d68d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6280b5950 0x7fb6281596a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:14.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.657+0000 7fb62d68d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb62814af10 0x7fb628159b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:14.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.657+0000 7fb62d68d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb628157360 con 0x7fb62814af10 2022-06-29T16:06:14.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.657+0000 7fb62d68d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6280af510 con 0x7fb628063ad0 2022-06-29T16:06:14.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.657+0000 7fb62d68d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6280ad800 con 0x7fb6280b5950 2022-06-29T16:06:14.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.657+0000 7fb626ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb628063ad0 0x7fb6280b0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:14.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.657+0000 7fb6277fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb62814af10 0x7fb628159b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:14.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.657+0000 7fb626ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb628063ad0 0x7fb6280b0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:57696/0 (socket says 172.21.15.60:57696) 2022-06-29T16:06:14.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.657+0000 7fb626ffd700 1 -- 172.21.15.60:0/1169088693 learned_addr learned my addr 172.21.15.60:0/1169088693 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:14.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.657+0000 7fb6277fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb62814af10 0x7fb628159b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51670/0 (socket says 172.21.15.60:51670) 2022-06-29T16:06:14.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.657+0000 7fb6267fc700 1 --2- 172.21.15.60:0/1169088693 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6280b5950 0x7fb6281596a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:14.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.657+0000 7fb626ffd700 1 -- 172.21.15.60:0/1169088693 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6280b5950 msgr2=0x7fb6281596a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:14.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.657+0000 7fb626ffd700 1 --2- 172.21.15.60:0/1169088693 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6280b5950 0x7fb6281596a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:14.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.657+0000 7fb626ffd700 1 -- 172.21.15.60:0/1169088693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb62814af10 msgr2=0x7fb628159b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:14.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.657+0000 7fb626ffd700 1 --2- 172.21.15.60:0/1169088693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb62814af10 0x7fb628159b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:14.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.657+0000 7fb626ffd700 1 -- 172.21.15.60:0/1169088693 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb618009470 con 0x7fb628063ad0 2022-06-29T16:06:14.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.657+0000 7fb6267fc700 1 --2- 172.21.15.60:0/1169088693 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6280b5950 0x7fb6281596a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:06:14.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.657+0000 7fb626ffd700 1 --2- 172.21.15.60:0/1169088693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb628063ad0 0x7fb6280b0850 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7fb61c001110 tx=0x7fb61c007aa0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:14.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.657+0000 7fb617fff700 1 -- 172.21.15.60:0/1169088693 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb61c0070d0 con 0x7fb628063ad0 2022-06-29T16:06:14.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.657+0000 7fb617fff700 1 -- 172.21.15.60:0/1169088693 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb61c007230 con 0x7fb628063ad0 2022-06-29T16:06:14.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.657+0000 7fb62d68d700 1 -- 172.21.15.60:0/1169088693 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6280ad560 con 0x7fb628063ad0 2022-06-29T16:06:14.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.657+0000 7fb617fff700 1 -- 172.21.15.60:0/1169088693 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb61c005610 con 0x7fb628063ad0 2022-06-29T16:06:14.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.657+0000 7fb62d68d700 1 -- 172.21.15.60:0/1169088693 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6280b0d40 con 0x7fb628063ad0 2022-06-29T16:06:14.672 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.661+0000 7fb62d68d700 1 -- 172.21.15.60:0/1169088693 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6280573d0 con 0x7fb628063ad0 2022-06-29T16:06:14.673 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.661+0000 7fb617fff700 1 -- 172.21.15.60:0/1169088693 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb61c010070 con 0x7fb628063ad0 2022-06-29T16:06:14.673 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.661+0000 7fb617fff700 1 --2- 172.21.15.60:0/1169088693 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb604053b00 0x7fb604055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:14.673 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.661+0000 7fb617fff700 1 -- 172.21.15.60:0/1169088693 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb61c06b700 con 0x7fb628063ad0 2022-06-29T16:06:14.674 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.661+0000 7fb6267fc700 1 --2- 172.21.15.60:0/1169088693 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb604053b00 0x7fb604055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:14.674 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.661+0000 7fb6267fc700 1 --2- 172.21.15.60:0/1169088693 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb604053b00 0x7fb604055fc0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7fb61801b000 tx=0x7fb618016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:14.676 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.669+0000 7fb617fff700 1 -- 172.21.15.60:0/1169088693 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb61c042490 con 0x7fb628063ad0 2022-06-29T16:06:14.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.841+0000 7fb62d68d700 1 -- 172.21.15.60:0/1169088693 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7fb6280684b0 con 0x7fb628063ad0 2022-06-29T16:06:14.845 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.841+0000 7fb617fff700 1 -- 172.21.15.60:0/1169088693 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1938 (secure 0 0 0) 0x7fb61c045ab0 con 0x7fb628063ad0 2022-06-29T16:06:14.846 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:14.846 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:06:14.846 INFO:teuthology.orchestra.run.smithi060.stdout: "stats": { 2022-06-29T16:06:14.846 INFO:teuthology.orchestra.run.smithi060.stdout: "total_bytes": 767585320960, 2022-06-29T16:06:14.847 INFO:teuthology.orchestra.run.smithi060.stdout: "total_avail_bytes": 761053872128, 2022-06-29T16:06:14.847 INFO:teuthology.orchestra.run.smithi060.stdout: "total_used_bytes": 6531350528, 2022-06-29T16:06:14.847 INFO:teuthology.orchestra.run.smithi060.stdout: "total_used_raw_bytes": 6531448832, 2022-06-29T16:06:14.847 INFO:teuthology.orchestra.run.smithi060.stdout: "total_used_raw_ratio": 0.008509085513651371, 2022-06-29T16:06:14.847 INFO:teuthology.orchestra.run.smithi060.stdout: "num_osds": 8, 2022-06-29T16:06:14.848 INFO:teuthology.orchestra.run.smithi060.stdout: "num_per_pool_osds": 0, 2022-06-29T16:06:14.848 INFO:teuthology.orchestra.run.smithi060.stdout: "num_per_pool_omap_osds": 0 2022-06-29T16:06:14.848 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:14.848 INFO:teuthology.orchestra.run.smithi060.stdout: "stats_by_class": { 2022-06-29T16:06:14.849 INFO:teuthology.orchestra.run.smithi060.stdout: "ssd": { 2022-06-29T16:06:14.849 INFO:teuthology.orchestra.run.smithi060.stdout: "total_bytes": 767585320960, 2022-06-29T16:06:14.849 INFO:teuthology.orchestra.run.smithi060.stdout: "total_avail_bytes": 761053872128, 2022-06-29T16:06:14.849 INFO:teuthology.orchestra.run.smithi060.stdout: "total_used_bytes": 6531350528, 2022-06-29T16:06:14.849 INFO:teuthology.orchestra.run.smithi060.stdout: "total_used_raw_bytes": 6531448832, 2022-06-29T16:06:14.850 INFO:teuthology.orchestra.run.smithi060.stdout: "total_used_raw_ratio": 0.008509085513651371 2022-06-29T16:06:14.850 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:14.850 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:14.850 INFO:teuthology.orchestra.run.smithi060.stdout: "pools": [ 2022-06-29T16:06:14.851 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:14.851 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "device_health_metrics", 2022-06-29T16:06:14.851 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1, 2022-06-29T16:06:14.851 INFO:teuthology.orchestra.run.smithi060.stdout: "stats": { 2022-06-29T16:06:14.851 INFO:teuthology.orchestra.run.smithi060.stdout: "stored": 0, 2022-06-29T16:06:14.852 INFO:teuthology.orchestra.run.smithi060.stdout: "objects": 2, 2022-06-29T16:06:14.852 INFO:teuthology.orchestra.run.smithi060.stdout: "kb_used": 0, 2022-06-29T16:06:14.852 INFO:teuthology.orchestra.run.smithi060.stdout: "bytes_used": 0, 2022-06-29T16:06:14.852 INFO:teuthology.orchestra.run.smithi060.stdout: "percent_used": 0, 2022-06-29T16:06:14.853 INFO:teuthology.orchestra.run.smithi060.stdout: "max_avail": 361337257984 2022-06-29T16:06:14.853 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:14.853 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:14.853 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:14.853 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rbd", 2022-06-29T16:06:14.854 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2, 2022-06-29T16:06:14.854 INFO:teuthology.orchestra.run.smithi060.stdout: "stats": { 2022-06-29T16:06:14.854 INFO:teuthology.orchestra.run.smithi060.stdout: "stored": 0, 2022-06-29T16:06:14.854 INFO:teuthology.orchestra.run.smithi060.stdout: "objects": 0, 2022-06-29T16:06:14.854 INFO:teuthology.orchestra.run.smithi060.stdout: "kb_used": 0, 2022-06-29T16:06:14.855 INFO:teuthology.orchestra.run.smithi060.stdout: "bytes_used": 0, 2022-06-29T16:06:14.855 INFO:teuthology.orchestra.run.smithi060.stdout: "percent_used": 0, 2022-06-29T16:06:14.855 INFO:teuthology.orchestra.run.smithi060.stdout: "max_avail": 361337257984 2022-06-29T16:06:14.855 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:14.856 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:14.856 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:14.856 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cephfs_metadata", 2022-06-29T16:06:14.856 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 19, 2022-06-29T16:06:14.856 INFO:teuthology.orchestra.run.smithi060.stdout: "stats": { 2022-06-29T16:06:14.857 INFO:teuthology.orchestra.run.smithi060.stdout: "stored": 2304, 2022-06-29T16:06:14.857 INFO:teuthology.orchestra.run.smithi060.stdout: "objects": 21, 2022-06-29T16:06:14.857 INFO:teuthology.orchestra.run.smithi060.stdout: "kb_used": 3, 2022-06-29T16:06:14.857 INFO:teuthology.orchestra.run.smithi060.stdout: "bytes_used": 2304, 2022-06-29T16:06:14.858 INFO:teuthology.orchestra.run.smithi060.stdout: "percent_used": 3.1881572937209057e-09, 2022-06-29T16:06:14.858 INFO:teuthology.orchestra.run.smithi060.stdout: "max_avail": 361337257984 2022-06-29T16:06:14.858 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:14.858 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:14.858 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:14.859 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cephfs_data", 2022-06-29T16:06:14.859 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 20, 2022-06-29T16:06:14.859 INFO:teuthology.orchestra.run.smithi060.stdout: "stats": { 2022-06-29T16:06:14.859 INFO:teuthology.orchestra.run.smithi060.stdout: "stored": 0, 2022-06-29T16:06:14.859 INFO:teuthology.orchestra.run.smithi060.stdout: "objects": 0, 2022-06-29T16:06:14.860 INFO:teuthology.orchestra.run.smithi060.stdout: "kb_used": 0, 2022-06-29T16:06:14.860 INFO:teuthology.orchestra.run.smithi060.stdout: "bytes_used": 0, 2022-06-29T16:06:14.860 INFO:teuthology.orchestra.run.smithi060.stdout: "percent_used": 0, 2022-06-29T16:06:14.860 INFO:teuthology.orchestra.run.smithi060.stdout: "max_avail": 361337257984 2022-06-29T16:06:14.861 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:14.861 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:14.861 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:14.861 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:06:14.863 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.841+0000 7fb62d68d700 1 -- 172.21.15.60:0/1169088693 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb604053b00 msgr2=0x7fb604055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:14.863 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.841+0000 7fb62d68d700 1 --2- 172.21.15.60:0/1169088693 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb604053b00 0x7fb604055fc0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7fb61801b000 tx=0x7fb618016000).stop 2022-06-29T16:06:14.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.841+0000 7fb62d68d700 1 -- 172.21.15.60:0/1169088693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb628063ad0 msgr2=0x7fb6280b0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:14.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.841+0000 7fb62d68d700 1 --2- 172.21.15.60:0/1169088693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb628063ad0 0x7fb6280b0850 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7fb61c001110 tx=0x7fb61c007aa0).stop 2022-06-29T16:06:14.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.841+0000 7fb62d68d700 1 -- 172.21.15.60:0/1169088693 shutdown_connections 2022-06-29T16:06:14.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.841+0000 7fb62d68d700 1 --2- 172.21.15.60:0/1169088693 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb604053b00 0x7fb604055fc0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:14.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.841+0000 7fb62d68d700 1 --2- 172.21.15.60:0/1169088693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb62814af10 0x7fb628159b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:14.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.841+0000 7fb62d68d700 1 --2- 172.21.15.60:0/1169088693 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb6280b5950 0x7fb6281596a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:14.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.841+0000 7fb62d68d700 1 --2- 172.21.15.60:0/1169088693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb628063ad0 0x7fb6280b0850 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:14.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.841+0000 7fb62d68d700 1 -- 172.21.15.60:0/1169088693 >> 172.21.15.60:0/1169088693 conn(0x7fb6280632c0 msgr2=0x7fb628146f10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:14.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.841+0000 7fb62d68d700 1 -- 172.21.15.60:0/1169088693 shutdown_connections 2022-06-29T16:06:14.866 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.841+0000 7fb62d68d700 1 -- 172.21.15.60:0/1169088693 wait complete. 2022-06-29T16:06:14.866 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data size -f json-pretty 2022-06-29T16:06:14.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.969+0000 7f9402f49700 1 Processor -- start 2022-06-29T16:06:14.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.969+0000 7f9402f49700 1 -- start start 2022-06-29T16:06:14.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.969+0000 7f9402f49700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93fc05d070 0x7f93fc14aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:14.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.969+0000 7f9402f49700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f93fc0b2130 0x7f93fc0aca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:14.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.969+0000 7f9402f49700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93fc05d450 con 0x7f93fc14af80 2022-06-29T16:06:14.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.969+0000 7f9402f49700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93fc0adf90 con 0x7f93fc0b2130 2022-06-29T16:06:14.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.969+0000 7f9402f49700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93fc0b25d0 con 0x7f93fc05d070 2022-06-29T16:06:14.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.969+0000 7f9402748700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f93fc14af80 0x7f93fc0acf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59760/0 (socket says 172.21.15.60:59760) 2022-06-29T16:06:14.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.969+0000 7f9402748700 1 -- 172.21.15.60:0/1962593115 learned_addr learned my addr 172.21.15.60:0/1962593115 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:14.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9401746700 1 --2- 172.21.15.60:0/1962593115 >> v2:172.21.15.92:3300/0 conn(0x7f93fc0b2130 0x7f93fc0aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:14.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9401f47700 1 --2- 172.21.15.60:0/1962593115 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93fc05d070 0x7f93fc14aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:14.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9400f45700 1 -- 172.21.15.60:0/1962593115 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1528217363 0 0) 0x7f93fc05d450 con 0x7f93fc14af80 2022-06-29T16:06:14.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9400f45700 1 -- 172.21.15.60:0/1962593115 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93e4003260 con 0x7f93fc14af80 2022-06-29T16:06:14.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9400f45700 1 -- 172.21.15.60:0/1962593115 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2964135943 0 0) 0x7f93e4003260 con 0x7f93fc14af80 2022-06-29T16:06:14.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9400f45700 1 -- 172.21.15.60:0/1962593115 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93fc05d070 msgr2=0x7f93fc14aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:14.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9400f45700 1 --2- 172.21.15.60:0/1962593115 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93fc05d070 0x7f93fc14aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:14.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9400f45700 1 -- 172.21.15.60:0/1962593115 >> v2:172.21.15.92:3300/0 conn(0x7f93fc0b2130 msgr2=0x7f93fc0aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:14.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9400f45700 1 --2- 172.21.15.60:0/1962593115 >> v2:172.21.15.92:3300/0 conn(0x7f93fc0b2130 0x7f93fc0aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:14.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9400f45700 1 -- 172.21.15.60:0/1962593115 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93fc14a350 con 0x7f93fc14af80 2022-06-29T16:06:14.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9400f45700 1 -- 172.21.15.60:0/1962593115 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f93f4001a00 con 0x7f93fc14af80 2022-06-29T16:06:14.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9400f45700 1 -- 172.21.15.60:0/1962593115 >> v1:172.21.15.60:6789/0 conn(0x7f93fc14af80 legacy=0x7f93fc0acf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:14.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9400f45700 1 --2- 172.21.15.60:0/1962593115 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93e4003750 0x7f93e4003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:14.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9400f45700 1 --2- 172.21.15.60:0/1962593115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f93e4004500 0x7f93e4004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:14.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9400f45700 1 --2- 172.21.15.60:0/1962593115 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93fc05d070 0x7f93e4004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:14.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9400f45700 1 -- 172.21.15.60:0/1962593115 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93fc0adf90 con 0x7f93e4003750 2022-06-29T16:06:14.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9400f45700 1 -- 172.21.15.60:0/1962593115 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93fc0b25d0 con 0x7f93e4004500 2022-06-29T16:06:14.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9400f45700 1 -- 172.21.15.60:0/1962593115 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93e400b750 con 0x7f93fc05d070 2022-06-29T16:06:14.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9401746700 1 --2- 172.21.15.60:0/1962593115 >> v2:172.21.15.92:3300/0 conn(0x7f93fc0b2130 0x7f93fc0aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:06:14.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9401f47700 1 --2- 172.21.15.60:0/1962593115 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93e4003750 0x7f93e4003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:14.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9402748700 1 --2- 172.21.15.60:0/1962593115 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93fc05d070 0x7f93e4004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:14.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9401746700 1 --2- 172.21.15.60:0/1962593115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f93e4004500 0x7f93e4004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:14.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9401f47700 1 -- 172.21.15.60:0/1962593115 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93fc05d070 msgr2=0x7f93e4004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:14.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9401f47700 1 --2- 172.21.15.60:0/1962593115 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93fc05d070 0x7f93e4004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:14.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9401f47700 1 -- 172.21.15.60:0/1962593115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f93e4004500 msgr2=0x7f93e4004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:14.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9401f47700 1 --2- 172.21.15.60:0/1962593115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f93e4004500 0x7f93e4004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:14.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9401f47700 1 -- 172.21.15.60:0/1962593115 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e4003260 con 0x7f93e4003750 2022-06-29T16:06:14.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9401746700 1 --2- 172.21.15.60:0/1962593115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f93e4004500 0x7f93e4004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:06:14.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9401f47700 1 --2- 172.21.15.60:0/1962593115 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93e4003750 0x7f93e4003b70 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 rx=0x7f93f800c000 tx=0x7f93f8006df0).ready entity=mon.0 client_cookie=7f570df45b4ff10a server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:14.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9400f45700 1 -- 172.21.15.60:0/1962593115 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f93f800e830 con 0x7f93e4003750 2022-06-29T16:06:14.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9400f45700 1 -- 172.21.15.60:0/1962593115 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93f800e990 con 0x7f93e4003750 2022-06-29T16:06:14.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9400f45700 1 -- 172.21.15.60:0/1962593115 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f93f800ecf0 con 0x7f93e4003750 2022-06-29T16:06:14.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9402f49700 1 -- 172.21.15.60:0/1962593115 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93e4003750 msgr2=0x7f93e4003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:14.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9402f49700 1 --2- 172.21.15.60:0/1962593115 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93e4003750 0x7f93e4003b70 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 rx=0x7f93f800c000 tx=0x7f93f8006df0).stop 2022-06-29T16:06:14.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9402748700 1 -- 172.21.15.60:0/1962593115 reap_dead start 2022-06-29T16:06:14.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9402f49700 1 -- 172.21.15.60:0/1962593115 shutdown_connections 2022-06-29T16:06:14.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9402f49700 1 -- 172.21.15.60:0/1962593115 >> 172.21.15.60:0/1962593115 conn(0x7f93fc0b5890 msgr2=0x7f93fc0b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:14.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9402f49700 1 -- 172.21.15.60:0/1962593115 shutdown_connections 2022-06-29T16:06:14.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.973+0000 7f9402f49700 1 -- 172.21.15.60:0/1962593115 wait complete. 2022-06-29T16:06:14.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.977+0000 7f9402f49700 1 Processor -- start 2022-06-29T16:06:14.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.977+0000 7f9402f49700 1 -- start start 2022-06-29T16:06:14.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.977+0000 7f9402f49700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93e4003750 0x7f93fc0b41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:14.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.977+0000 7f9402f49700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f93fc0b4690 0x7f93fc159f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:14.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.977+0000 7f9402f49700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93fc15a420 0x7f93fc1182a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:14.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.977+0000 7f9402f49700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93fc065930 con 0x7f93e4003750 2022-06-29T16:06:14.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.977+0000 7f9402f49700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93fc0667d0 con 0x7f93fc0b4690 2022-06-29T16:06:14.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.977+0000 7f9402f49700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93fc066a70 con 0x7f93fc15a420 2022-06-29T16:06:14.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.977+0000 7f9401f47700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93e4003750 0x7f93fc0b41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:14.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.977+0000 7f9401f47700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93e4003750 0x7f93fc0b41a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51686/0 (socket says 172.21.15.60:51686) 2022-06-29T16:06:14.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.977+0000 7f9401f47700 1 -- 172.21.15.60:0/947632314 learned_addr learned my addr 172.21.15.60:0/947632314 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:14.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.977+0000 7f9402748700 1 --2- 172.21.15.60:0/947632314 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93fc15a420 0x7f93fc1182a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:14.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.977+0000 7f9401746700 1 --2- 172.21.15.60:0/947632314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f93fc0b4690 0x7f93fc159f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:15.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.977+0000 7f9401f47700 1 -- 172.21.15.60:0/947632314 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93fc15a420 msgr2=0x7f93fc1182a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:15.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.977+0000 7f9401f47700 1 --2- 172.21.15.60:0/947632314 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93fc15a420 0x7f93fc1182a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:15.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.977+0000 7f9401f47700 1 -- 172.21.15.60:0/947632314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f93fc0b4690 msgr2=0x7f93fc159f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:15.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.977+0000 7f9401f47700 1 --2- 172.21.15.60:0/947632314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f93fc0b4690 0x7f93fc159f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:15.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.977+0000 7f9401f47700 1 -- 172.21.15.60:0/947632314 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93fc066d10 con 0x7f93e4003750 2022-06-29T16:06:15.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.977+0000 7f9402748700 1 --2- 172.21.15.60:0/947632314 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93fc15a420 0x7f93fc1182a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:06:15.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.977+0000 7f9401f47700 1 --2- 172.21.15.60:0/947632314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93e4003750 0x7f93fc0b41a0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 rx=0x7f93f800c000 tx=0x7f93f8019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:15.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.977+0000 7f93eeffd700 1 -- 172.21.15.60:0/947632314 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f93f801b820 con 0x7f93e4003750 2022-06-29T16:06:15.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.977+0000 7f93eeffd700 1 -- 172.21.15.60:0/947632314 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93f8000e90 con 0x7f93e4003750 2022-06-29T16:06:15.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.977+0000 7f93eeffd700 1 -- 172.21.15.60:0/947632314 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f93f801bb70 con 0x7f93e4003750 2022-06-29T16:06:15.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.977+0000 7f9402f49700 1 -- 172.21.15.60:0/947632314 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93fc1597a0 con 0x7f93e4003750 2022-06-29T16:06:15.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.977+0000 7f9402f49700 1 -- 172.21.15.60:0/947632314 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93fc0663f0 con 0x7f93e4003750 2022-06-29T16:06:15.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.981+0000 7f93eeffd700 1 -- 172.21.15.60:0/947632314 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f93f801bcd0 con 0x7f93e4003750 2022-06-29T16:06:15.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.981+0000 7f9402f49700 1 -- 172.21.15.60:0/947632314 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93fc065300 con 0x7f93e4003750 2022-06-29T16:06:15.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.981+0000 7f93eeffd700 1 --2- 172.21.15.60:0/947632314 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f93dc053a70 0x7f93dc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:15.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.981+0000 7f93eeffd700 1 -- 172.21.15.60:0/947632314 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f93f806d530 con 0x7f93e4003750 2022-06-29T16:06:15.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.981+0000 7f9401746700 1 --2- 172.21.15.60:0/947632314 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f93dc053a70 0x7f93dc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:15.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.981+0000 7f9401746700 1 --2- 172.21.15.60:0/947632314 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f93dc053a70 0x7f93dc055f30 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7f93f000b000 tx=0x7f93f0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:15.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:14.985+0000 7f93eeffd700 1 -- 172.21.15.60:0/947632314 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f93fc065300 con 0x7f93e4003750 2022-06-29T16:06:15.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.109+0000 7f9402f49700 1 -- 172.21.15.60:0/947632314 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "size", "format": "json-pretty"} v 0) v1 -- 0x7f93fc065300 con 0x7f93e4003750 2022-06-29T16:06:15.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.109+0000 7f93eeffd700 1 -- 172.21.15.60:0/947632314 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "size", "format": "json-pretty"}]=0 v100) v1 ==== 123+0+64 (secure 0 0 0) 0x7f93fc065300 con 0x7f93e4003750 2022-06-29T16:06:15.116 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:15.116 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:06:15.116 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": "cephfs_data", 2022-06-29T16:06:15.116 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_id": 20, 2022-06-29T16:06:15.116 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2 2022-06-29T16:06:15.117 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:06:15.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.113+0000 7f9402f49700 1 -- 172.21.15.60:0/947632314 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f93dc053a70 msgr2=0x7f93dc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:15.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.113+0000 7f9402f49700 1 --2- 172.21.15.60:0/947632314 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f93dc053a70 0x7f93dc055f30 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7f93f000b000 tx=0x7f93f0008040).stop 2022-06-29T16:06:15.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.113+0000 7f9402f49700 1 -- 172.21.15.60:0/947632314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93e4003750 msgr2=0x7f93fc0b41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:15.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.113+0000 7f9402f49700 1 --2- 172.21.15.60:0/947632314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93e4003750 0x7f93fc0b41a0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 rx=0x7f93f800c000 tx=0x7f93f8019000).stop 2022-06-29T16:06:15.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.113+0000 7f9402f49700 1 -- 172.21.15.60:0/947632314 shutdown_connections 2022-06-29T16:06:15.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.113+0000 7f9402f49700 1 --2- 172.21.15.60:0/947632314 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f93dc053a70 0x7f93dc055f30 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:15.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.113+0000 7f9402f49700 1 --2- 172.21.15.60:0/947632314 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93fc15a420 0x7f93fc1182a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:15.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.113+0000 7f9402f49700 1 --2- 172.21.15.60:0/947632314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f93fc0b4690 0x7f93fc159f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:15.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.113+0000 7f9402f49700 1 --2- 172.21.15.60:0/947632314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93e4003750 0x7f93fc0b41a0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:15.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.113+0000 7f9402f49700 1 -- 172.21.15.60:0/947632314 >> 172.21.15.60:0/947632314 conn(0x7f93fc0b5890 msgr2=0x7f93fc1433b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:15.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.113+0000 7f9402f49700 1 -- 172.21.15.60:0/947632314 shutdown_connections 2022-06-29T16:06:15.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.113+0000 7f9402f49700 1 -- 172.21.15.60:0/947632314 wait complete. 2022-06-29T16:06:15.129 INFO:teuthology.orchestra.run:Running command with timeout 10800 2022-06-29T16:06:15.129 DEBUG:teuthology.orchestra.run.smithi121:> cd /home/ubuntu/cephtest/mnt.0 && sudo df . 2022-06-29T16:06:15.161 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 1K-blocks Used Available Use% Mounted on 2022-06-29T16:06:15.162 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-fuse 352866304 0 352866304 0% /home/ubuntu/cephtest/mnt.0 2022-06-29T16:06:15.163 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-06-29T16:06:15.163 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:06:15.184 DEBUG:teuthology.orchestra.run.smithi092:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:06:15.206 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:06:15.228 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:06:15.249 DEBUG:teuthology.orchestra.run.smithi197:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:06:15.271 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-06-29T16:06:15.271 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:15.272 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:06:15.272 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:06:15.281 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T16:06:15.282 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T16:06:15.282 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2022-06-29T16:06:15.282 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-06-29T16:06:15.283 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:06:15.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[13226]: fuse finished with error 0 and tester_r 0 2022-06-29T16:06:15.300 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:15.301 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:06:15.301 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:06:15.349 INFO:teuthology.orchestra.run.smithi121.stdout:ext2/ext3 2022-06-29T16:06:15.351 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-06-29T16:06:15.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:06:15.465 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:06:15.473 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:15.473 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:06:15.474 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:06:15.482 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2022-06-29T16:06:15.483 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-06-29T16:06:15.483 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi156.front.sepia.ceph.com... 2022-06-29T16:06:15.484 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-06-29T16:06:15.484 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:06:15.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[12654]: fuse finished with error 0 and tester_r 0 2022-06-29T16:06:15.544 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:15.545 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:06:15.545 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:06:15.551 INFO:teuthology.orchestra.run.smithi156.stdout:ext2/ext3 2022-06-29T16:06:15.552 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-06-29T16:06:15.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:06:15.671 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:06:15.718 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:15.719 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:06:15.720 DEBUG:teuthology.orchestra.run.smithi197:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T16:06:15.728 INFO:teuthology.orchestra.run.smithi197.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-06-29T16:06:15.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:06:15.730 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-06-29T16:06:15.731 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:06:15.732 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T16:06:15.777 INFO:teuthology.orchestra.run.smithi197.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-06-29T16:06:15.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:06:15.779 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-06-29T16:06:15.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.889+0000 7f4cfe270700 1 Processor -- start 2022-06-29T16:06:15.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.889+0000 7f4cfe270700 1 -- start start 2022-06-29T16:06:15.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.889+0000 7f4cfe270700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4cf80b5950 0x7f4cf8067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:15.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.889+0000 7f4cfe270700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4cf8063ad0 0x7f4cf814af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:15.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.889+0000 7f4cfe270700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4cf805bd20 0x7f4cf805c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:15.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.889+0000 7f4cfe270700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cf80acba0 con 0x7f4cf805bd20 2022-06-29T16:06:15.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.889+0000 7f4cfe270700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cf8067220 con 0x7f4cf8063ad0 2022-06-29T16:06:15.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.889+0000 7f4cfe270700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf80b5d30 con 0x7f4cf80b5950 2022-06-29T16:06:15.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.889+0000 7f4cfc80d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4cf805bd20 0x7f4cf805c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:15.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.889+0000 7f4cfc80d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4cf805bd20 0x7f4cf805c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51698/0 (socket says 172.21.15.60:51698) 2022-06-29T16:06:15.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.889+0000 7f4cfc80d700 1 -- 172.21.15.60:0/1473319307 learned_addr learned my addr 172.21.15.60:0/1473319307 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:15.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.889+0000 7f4cf77fe700 1 --2- 172.21.15.60:0/1473319307 >> v2:172.21.15.92:3300/0 conn(0x7f4cf8063ad0 0x7f4cf814af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:15.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.889+0000 7f4cf7fff700 1 --2- 172.21.15.60:0/1473319307 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4cf80b5950 0x7f4cf8067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:15.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.889+0000 7f4cfc80d700 1 -- 172.21.15.60:0/1473319307 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4cf80b5950 msgr2=0x7f4cf8067680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:15.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.889+0000 7f4cfc80d700 1 --2- 172.21.15.60:0/1473319307 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4cf80b5950 0x7f4cf8067680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:15.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.889+0000 7f4cfc80d700 1 -- 172.21.15.60:0/1473319307 >> v2:172.21.15.92:3300/0 conn(0x7f4cf8063ad0 msgr2=0x7f4cf814af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:15.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.889+0000 7f4cfc80d700 1 --2- 172.21.15.60:0/1473319307 >> v2:172.21.15.92:3300/0 conn(0x7f4cf8063ad0 0x7f4cf814af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:15.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.889+0000 7f4cfc80d700 1 -- 172.21.15.60:0/1473319307 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf80ace00 con 0x7f4cf805bd20 2022-06-29T16:06:15.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.889+0000 7f4cfc80d700 1 --2- 172.21.15.60:0/1473319307 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cf805bd20 0x7f4cf805c1b0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 rx=0x7f4cf00032f0 tx=0x7f4cf0009290).ready entity=mon.0 client_cookie=1a2acad07627ae43 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:15.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.889+0000 7f4cf6ffd700 1 -- 172.21.15.60:0/1473319307 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4cf000b870 con 0x7f4cf805bd20 2022-06-29T16:06:15.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.889+0000 7f4cf6ffd700 1 -- 172.21.15.60:0/1473319307 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cf000b9d0 con 0x7f4cf805bd20 2022-06-29T16:06:15.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.889+0000 7f4cf6ffd700 1 -- 172.21.15.60:0/1473319307 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4cf000bb40 con 0x7f4cf805bd20 2022-06-29T16:06:15.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.889+0000 7f4cfe270700 1 -- 172.21.15.60:0/1473319307 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cf805bd20 msgr2=0x7f4cf805c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:15.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.889+0000 7f4cfe270700 1 --2- 172.21.15.60:0/1473319307 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cf805bd20 0x7f4cf805c1b0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 rx=0x7f4cf00032f0 tx=0x7f4cf0009290).stop 2022-06-29T16:06:15.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.889+0000 7f4cfe270700 1 -- 172.21.15.60:0/1473319307 shutdown_connections 2022-06-29T16:06:15.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.889+0000 7f4cfe270700 1 --2- 172.21.15.60:0/1473319307 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cf805bd20 0x7f4cf805c1b0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:15.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.889+0000 7f4cfe270700 1 --2- 172.21.15.60:0/1473319307 >> v2:172.21.15.92:3300/0 conn(0x7f4cf8063ad0 0x7f4cf814af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:15.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.889+0000 7f4cfe270700 1 --2- 172.21.15.60:0/1473319307 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4cf80b5950 0x7f4cf8067680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:15.903 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.889+0000 7f4cfe270700 1 -- 172.21.15.60:0/1473319307 >> 172.21.15.60:0/1473319307 conn(0x7f4cf80632c0 msgr2=0x7f4cf80b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:15.903 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.889+0000 7f4cfe270700 1 -- 172.21.15.60:0/1473319307 shutdown_connections 2022-06-29T16:06:15.903 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.889+0000 7f4cfe270700 1 -- 172.21.15.60:0/1473319307 wait complete. 2022-06-29T16:06:15.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.893+0000 7f4cfe270700 1 Processor -- start 2022-06-29T16:06:15.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.893+0000 7f4cfe270700 1 -- start start 2022-06-29T16:06:15.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.893+0000 7f4cfe270700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4cf805bd20 0x7f4cf8064470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:15.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.893+0000 7f4cfe270700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4cf8063ad0 0x7f4cf80ae410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:15.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.893+0000 7f4cfe270700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cf80b5950 0x7f4cf8155dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:15.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.893+0000 7f4cfe270700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf81438a0 con 0x7f4cf80b5950 2022-06-29T16:06:15.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.893+0000 7f4cfe270700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf80aeee0 con 0x7f4cf805bd20 2022-06-29T16:06:15.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.893+0000 7f4cfe270700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf80aec40 con 0x7f4cf8063ad0 2022-06-29T16:06:15.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.893+0000 7f4cfc80d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cf80b5950 0x7f4cf8155dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:15.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.897+0000 7f4cfc80d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cf80b5950 0x7f4cf8155dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51704/0 (socket says 172.21.15.60:51704) 2022-06-29T16:06:15.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.897+0000 7f4cfc80d700 1 -- 172.21.15.60:0/2899617314 learned_addr learned my addr 172.21.15.60:0/2899617314 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:15.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.897+0000 7f4cf7fff700 1 --2- 172.21.15.60:0/2899617314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4cf805bd20 0x7f4cf8064470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:15.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.897+0000 7f4cf77fe700 1 --2- 172.21.15.60:0/2899617314 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4cf8063ad0 0x7f4cf80ae410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:15.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.897+0000 7f4cfc80d700 1 -- 172.21.15.60:0/2899617314 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4cf8063ad0 msgr2=0x7f4cf80ae410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:15.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.897+0000 7f4cfc80d700 1 --2- 172.21.15.60:0/2899617314 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4cf8063ad0 0x7f4cf80ae410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:15.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.897+0000 7f4cfc80d700 1 -- 172.21.15.60:0/2899617314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4cf805bd20 msgr2=0x7f4cf8064470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:15.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.897+0000 7f4cfc80d700 1 --2- 172.21.15.60:0/2899617314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4cf805bd20 0x7f4cf8064470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:15.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.897+0000 7f4cfc80d700 1 -- 172.21.15.60:0/2899617314 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf0009530 con 0x7f4cf80b5950 2022-06-29T16:06:15.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.897+0000 7f4cfc80d700 1 --2- 172.21.15.60:0/2899617314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cf80b5950 0x7f4cf8155dc0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 rx=0x7f4cf00032f0 tx=0x7f4cf0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:15.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.897+0000 7f4cf57fa700 1 -- 172.21.15.60:0/2899617314 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4cf000b870 con 0x7f4cf80b5950 2022-06-29T16:06:15.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.897+0000 7f4cf57fa700 1 -- 172.21.15.60:0/2899617314 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cf0021630 con 0x7f4cf80b5950 2022-06-29T16:06:15.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.897+0000 7f4cfe270700 1 -- 172.21.15.60:0/2899617314 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cf8159500 con 0x7f4cf80b5950 2022-06-29T16:06:15.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.897+0000 7f4cf57fa700 1 -- 172.21.15.60:0/2899617314 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4cf0021990 con 0x7f4cf80b5950 2022-06-29T16:06:15.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.897+0000 7f4cfe270700 1 -- 172.21.15.60:0/2899617314 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cf81562b0 con 0x7f4cf80b5950 2022-06-29T16:06:15.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.897+0000 7f4cfe270700 1 -- 172.21.15.60:0/2899617314 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cf8159690 con 0x7f4cf80b5950 2022-06-29T16:06:15.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.897+0000 7f4cf57fa700 1 -- 172.21.15.60:0/2899617314 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4cf0027020 con 0x7f4cf80b5950 2022-06-29T16:06:15.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.901+0000 7f4cf57fa700 1 --2- 172.21.15.60:0/2899617314 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4cdc053a70 0x7f4cdc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:15.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.901+0000 7f4cf57fa700 1 -- 172.21.15.60:0/2899617314 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4cf006d1f0 con 0x7f4cf80b5950 2022-06-29T16:06:15.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.901+0000 7f4cf7fff700 1 --2- 172.21.15.60:0/2899617314 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4cdc053a70 0x7f4cdc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:15.915 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.901+0000 7f4cf7fff700 1 --2- 172.21.15.60:0/2899617314 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4cdc053a70 0x7f4cdc055f30 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0x7f4ce000c000 tx=0x7f4ce0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:15.915 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:15.909+0000 7f4cf57fa700 1 -- 172.21.15.60:0/2899617314 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4cf0044110 con 0x7f4cf80b5950 2022-06-29T16:06:16.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.033+0000 7f4cfe270700 1 -- 172.21.15.60:0/2899617314 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f4cf8143880 con 0x7f4cf80b5950 2022-06-29T16:06:16.042 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.037+0000 7f4cf57fa700 1 -- 172.21.15.60:0/2899617314 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v100) v1 ==== 81+0+33650 (secure 0 0 0) 0x7f4cf0015020 con 0x7f4cf80b5950 2022-06-29T16:06:16.043 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:16.044 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:06:16.044 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 100, 2022-06-29T16:06:16.044 INFO:teuthology.orchestra.run.smithi060.stdout: "fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", 2022-06-29T16:06:16.045 INFO:teuthology.orchestra.run.smithi060.stdout: "created": "2022-06-29T15:58:07.965719+0000", 2022-06-29T16:06:16.045 INFO:teuthology.orchestra.run.smithi060.stdout: "modified": "2022-06-29T16:06:06.229758+0000", 2022-06-29T16:06:16.045 INFO:teuthology.orchestra.run.smithi060.stdout: "last_up_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:06:16.046 INFO:teuthology.orchestra.run.smithi060.stdout: "last_in_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:06:16.046 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-06-29T16:06:16.046 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_num": 5799936, 2022-06-29T16:06:16.046 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_set": [ 2022-06-29T16:06:16.047 INFO:teuthology.orchestra.run.smithi060.stdout: "pglog_hardlimit", 2022-06-29T16:06:16.047 INFO:teuthology.orchestra.run.smithi060.stdout: "purged_snapdirs", 2022-06-29T16:06:16.047 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_deletes", 2022-06-29T16:06:16.048 INFO:teuthology.orchestra.run.smithi060.stdout: "sortbitwise" 2022-06-29T16:06:16.048 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:06:16.048 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_version": 4, 2022-06-29T16:06:16.049 INFO:teuthology.orchestra.run.smithi060.stdout: "full_ratio": 0.94999998807907104, 2022-06-29T16:06:16.049 INFO:teuthology.orchestra.run.smithi060.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-06-29T16:06:16.049 INFO:teuthology.orchestra.run.smithi060.stdout: "nearfull_ratio": 0.85000002384185791, 2022-06-29T16:06:16.049 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_snapshot": "", 2022-06-29T16:06:16.050 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_max": 20, 2022-06-29T16:06:16.050 INFO:teuthology.orchestra.run.smithi060.stdout: "max_osd": 8, 2022-06-29T16:06:16.050 INFO:teuthology.orchestra.run.smithi060.stdout: "require_min_compat_client": "jewel", 2022-06-29T16:06:16.050 INFO:teuthology.orchestra.run.smithi060.stdout: "min_compat_client": "jewel", 2022-06-29T16:06:16.050 INFO:teuthology.orchestra.run.smithi060.stdout: "require_osd_release": "octopus", 2022-06-29T16:06:16.051 INFO:teuthology.orchestra.run.smithi060.stdout: "pools": [ 2022-06-29T16:06:16.051 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.051 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2022-06-29T16:06:16.051 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "device_health_metrics", 2022-06-29T16:06:16.052 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:10.961127+0000", 2022-06-29T16:06:16.052 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:06:16.052 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:06:16.052 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:06:16.052 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:06:16.053 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:06:16.053 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:06:16.053 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:06:16.053 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:06:16.054 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 1, 2022-06-29T16:06:16.054 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 1, 2022-06-29T16:06:16.054 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 1, 2022-06-29T16:06:16.054 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 1, 2022-06-29T16:06:16.054 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 1, 2022-06-29T16:06:16.055 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:06:16.055 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:06:16.055 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:06:16.055 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:06:16.055 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:06:16.056 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:06:16.056 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:06:16.056 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.056 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "15", 2022-06-29T16:06:16.057 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:06:16.057 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:06:16.057 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:06:16.057 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:06:16.057 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:06:16.058 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:06:16.058 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:06:16.058 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:06:16.058 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:06:16.059 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:06:16.059 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:06:16.059 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:06:16.060 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:06:16.060 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:06:16.060 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:06:16.060 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:06:16.060 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:06:16.061 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:06:16.061 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:06:16.061 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:06:16.061 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:06:16.062 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:06:16.062 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:06:16.062 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:06:16.063 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:06:16.063 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:06:16.063 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.064 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:06:16.064 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:06:16.064 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:06:16.064 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:06:16.065 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:06:16.065 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:06:16.065 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:06:16.065 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:06:16.065 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:06:16.066 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:06:16.066 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:06:16.066 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:06:16.066 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 1 2022-06-29T16:06:16.067 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.067 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:06:16.067 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_devicehealth": {} 2022-06-29T16:06:16.067 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.067 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.068 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.068 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 2, 2022-06-29T16:06:16.068 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "rbd", 2022-06-29T16:06:16.068 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:19.184276+0000", 2022-06-29T16:06:16.069 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:06:16.069 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:06:16.069 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:06:16.069 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:06:16.069 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:06:16.070 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:06:16.070 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:06:16.070 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:06:16.070 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:06:16.071 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:06:16.071 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:06:16.071 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:06:16.071 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:06:16.071 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:06:16.072 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:06:16.072 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:06:16.072 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:06:16.072 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:06:16.073 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:06:16.073 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:06:16.073 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.073 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "18", 2022-06-29T16:06:16.073 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:06:16.074 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:06:16.074 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:06:16.074 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:06:16.074 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:06:16.074 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:06:16.075 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:06:16.075 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:06:16.075 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:06:16.075 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:06:16.076 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:06:16.076 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:06:16.076 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:06:16.076 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:06:16.076 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:06:16.077 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:06:16.077 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:06:16.077 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:06:16.077 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:06:16.077 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:06:16.078 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:06:16.078 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:06:16.078 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:06:16.078 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:06:16.079 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:06:16.079 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:06:16.079 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.079 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:06:16.079 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:06:16.080 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:06:16.080 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:06:16.080 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:06:16.080 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:06:16.081 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:06:16.081 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:06:16.081 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:06:16.081 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:06:16.081 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:06:16.082 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:06:16.082 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:06:16.082 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd": {} 2022-06-29T16:06:16.082 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.082 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.083 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.083 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 19, 2022-06-29T16:06:16.083 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "cephfs_metadata", 2022-06-29T16:06:16.083 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T16:06:03.648757+0000", 2022-06-29T16:06:16.084 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:06:16.084 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:06:16.084 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:06:16.084 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:06:16.084 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:06:16.085 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:06:16.085 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:06:16.085 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:06:16.085 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:06:16.085 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:06:16.086 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:06:16.086 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:06:16.086 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:06:16.087 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:06:16.087 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:06:16.087 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:06:16.087 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:06:16.088 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:06:16.088 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:06:16.088 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:06:16.088 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.089 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "99", 2022-06-29T16:06:16.089 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:06:16.089 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:06:16.089 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:06:16.090 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:06:16.090 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:06:16.090 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:06:16.090 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:06:16.090 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:06:16.091 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:06:16.091 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:06:16.091 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:06:16.091 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:06:16.092 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:06:16.092 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:06:16.092 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:06:16.092 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:06:16.093 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:06:16.093 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:06:16.093 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:06:16.093 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:06:16.093 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:06:16.094 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:06:16.094 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:06:16.094 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:06:16.095 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:06:16.095 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:06:16.095 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.095 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:06:16.095 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:06:16.096 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:06:16.096 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:06:16.096 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:06:16.096 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:06:16.096 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:06:16.097 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:06:16.097 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:06:16.097 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:06:16.097 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:06:16.098 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:06:16.098 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_bias": 4, 2022-06-29T16:06:16.098 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 16, 2022-06-29T16:06:16.098 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_priority": 5 2022-06-29T16:06:16.098 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.099 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:06:16.099 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs": { 2022-06-29T16:06:16.099 INFO:teuthology.orchestra.run.smithi060.stdout: "metadata": "cephfs" 2022-06-29T16:06:16.099 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.100 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.100 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.100 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.100 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 20, 2022-06-29T16:06:16.100 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "cephfs_data", 2022-06-29T16:06:16.101 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T16:06:04.556403+0000", 2022-06-29T16:06:16.101 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:06:16.101 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:06:16.101 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:06:16.102 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:06:16.102 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:06:16.102 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:06:16.102 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:06:16.102 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:06:16.103 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:06:16.103 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:06:16.103 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:06:16.103 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:06:16.104 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:06:16.104 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:06:16.104 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:06:16.104 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:06:16.104 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:06:16.105 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:06:16.105 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:06:16.105 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:06:16.105 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.106 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "100", 2022-06-29T16:06:16.106 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:06:16.106 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:06:16.106 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:06:16.107 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:06:16.107 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:06:16.107 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:06:16.107 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:06:16.107 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:06:16.108 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:06:16.108 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:06:16.108 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:06:16.108 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:06:16.109 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:06:16.109 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:06:16.109 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:06:16.109 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:06:16.109 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:06:16.110 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:06:16.110 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:06:16.110 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:06:16.110 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:06:16.111 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:06:16.111 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:06:16.111 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:06:16.111 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:06:16.111 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:06:16.112 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.112 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:06:16.112 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:06:16.112 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:06:16.112 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:06:16.113 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:06:16.113 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:06:16.113 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:06:16.113 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:06:16.113 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:06:16.114 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:06:16.114 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:06:16.114 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:06:16.114 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:06:16.115 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs": { 2022-06-29T16:06:16.115 INFO:teuthology.orchestra.run.smithi060.stdout: "data": "cephfs" 2022-06-29T16:06:16.115 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.115 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.115 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.116 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:06:16.116 INFO:teuthology.orchestra.run.smithi060.stdout: "osds": [ 2022-06-29T16:06:16.116 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.116 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:06:16.117 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "11e7ab9e-fce2-470f-8195-ea4c3c52df22", 2022-06-29T16:06:16.117 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:06:16.117 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:06:16.117 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:06:16.117 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:06:16.118 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:06:16.118 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:06:16.118 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:06:16.118 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 98, 2022-06-29T16:06:16.118 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:06:16.119 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:06:16.119 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:06:16.119 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:16.119 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.120 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:16.120 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6816", 2022-06-29T16:06:16.120 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:06:16.120 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.121 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.121 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:16.121 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6825", 2022-06-29T16:06:16.121 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:06:16.122 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.122 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.122 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.122 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:06:16.122 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:16.123 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.123 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:16.123 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6826", 2022-06-29T16:06:16.123 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:06:16.124 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.124 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.124 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:16.124 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6827", 2022-06-29T16:06:16.124 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:06:16.125 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.125 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.125 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.125 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:06:16.125 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:16.126 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.126 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:16.126 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6830", 2022-06-29T16:06:16.126 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:06:16.127 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.127 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.127 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:16.127 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6831", 2022-06-29T16:06:16.127 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:06:16.128 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.128 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.128 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.128 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:06:16.128 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:16.129 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.129 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:16.129 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6828", 2022-06-29T16:06:16.129 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:06:16.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.130 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.130 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:16.130 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6829", 2022-06-29T16:06:16.130 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:06:16.131 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.131 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.131 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.131 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6825/12890", 2022-06-29T16:06:16.131 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6827/12890", 2022-06-29T16:06:16.132 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6831/12890", 2022-06-29T16:06:16.132 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6829/12890", 2022-06-29T16:06:16.132 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:06:16.132 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:06:16.133 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:06:16.133 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.133 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.133 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:06:16.134 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "ca43f136-6f69-4431-a157-fbe073fb585c", 2022-06-29T16:06:16.134 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:06:16.134 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:06:16.134 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:06:16.134 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:06:16.135 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:06:16.135 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:06:16.135 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:06:16.135 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 98, 2022-06-29T16:06:16.136 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:06:16.136 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:06:16.136 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:06:16.136 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:16.136 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.137 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:16.137 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6815", 2022-06-29T16:06:16.137 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:06:16.137 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.138 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.138 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:16.138 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6818", 2022-06-29T16:06:16.138 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:06:16.138 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.139 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.139 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.139 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:06:16.139 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:16.140 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.140 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:16.140 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6819", 2022-06-29T16:06:16.140 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:06:16.140 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.141 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.141 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:16.141 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6820", 2022-06-29T16:06:16.141 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:06:16.142 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.142 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.142 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.142 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:06:16.142 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:16.143 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.143 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:16.143 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6823", 2022-06-29T16:06:16.143 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:06:16.144 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.144 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.144 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:16.144 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6824", 2022-06-29T16:06:16.144 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:06:16.145 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.145 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.145 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.145 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:06:16.145 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:16.146 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.146 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:16.146 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6821", 2022-06-29T16:06:16.146 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:06:16.147 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.147 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.147 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:16.147 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6822", 2022-06-29T16:06:16.147 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:06:16.148 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.148 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.148 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.148 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6818/12892", 2022-06-29T16:06:16.148 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6820/12892", 2022-06-29T16:06:16.149 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6824/12892", 2022-06-29T16:06:16.149 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6822/12892", 2022-06-29T16:06:16.149 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:06:16.149 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:06:16.150 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:06:16.150 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.150 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.150 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.150 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:06:16.151 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "74fac45f-5f07-44ec-a9cb-2644c123fd4a", 2022-06-29T16:06:16.151 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:06:16.151 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:06:16.151 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:06:16.152 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:06:16.152 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:06:16.152 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:06:16.152 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:06:16.152 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 97, 2022-06-29T16:06:16.153 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:06:16.153 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:06:16.153 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:06:16.153 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:16.153 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.154 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:16.154 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6808", 2022-06-29T16:06:16.154 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:06:16.154 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.155 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.155 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:16.155 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6809", 2022-06-29T16:06:16.155 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:06:16.155 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.156 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.156 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.156 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:06:16.156 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:16.157 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.157 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:16.157 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6810", 2022-06-29T16:06:16.157 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:06:16.157 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.158 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.158 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:16.158 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6811", 2022-06-29T16:06:16.158 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:06:16.159 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.159 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.159 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.159 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:06:16.159 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:16.160 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.160 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:16.160 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6814", 2022-06-29T16:06:16.160 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:06:16.160 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.161 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.161 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:16.161 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6817", 2022-06-29T16:06:16.161 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:06:16.162 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.162 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.162 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.162 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:06:16.162 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:16.163 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.163 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:16.163 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6812", 2022-06-29T16:06:16.163 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:06:16.164 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.164 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.164 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:16.164 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6813", 2022-06-29T16:06:16.165 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:06:16.165 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.165 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.165 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.166 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6809/12891", 2022-06-29T16:06:16.166 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6811/12891", 2022-06-29T16:06:16.166 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6817/12891", 2022-06-29T16:06:16.166 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6813/12891", 2022-06-29T16:06:16.166 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:06:16.167 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:06:16.167 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:06:16.167 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.167 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.168 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.168 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:06:16.168 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "2581d351-973a-4b4e-85ca-3658f0068765", 2022-06-29T16:06:16.168 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:06:16.169 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:06:16.169 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:06:16.169 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:06:16.169 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:06:16.169 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:06:16.170 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:06:16.170 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 98, 2022-06-29T16:06:16.170 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:06:16.170 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:06:16.171 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:06:16.171 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:16.171 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.171 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:16.171 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6800", 2022-06-29T16:06:16.172 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:06:16.172 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.172 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.172 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:16.173 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6801", 2022-06-29T16:06:16.173 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:06:16.173 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.173 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.173 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.174 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:06:16.174 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:16.174 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.174 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:16.175 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6802", 2022-06-29T16:06:16.175 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:06:16.175 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.175 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.175 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:16.176 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6803", 2022-06-29T16:06:16.176 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:06:16.176 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.176 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.177 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.177 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:06:16.177 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:16.177 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.177 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:16.178 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6806", 2022-06-29T16:06:16.178 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:06:16.178 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.178 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.178 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:16.179 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6807", 2022-06-29T16:06:16.179 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:06:16.179 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.179 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.180 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.180 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:06:16.180 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:16.180 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.180 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:16.181 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6804", 2022-06-29T16:06:16.181 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:06:16.181 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.181 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.182 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:16.182 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6805", 2022-06-29T16:06:16.182 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:06:16.182 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.183 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.183 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.183 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6801/12889", 2022-06-29T16:06:16.183 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6803/12889", 2022-06-29T16:06:16.184 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6807/12889", 2022-06-29T16:06:16.184 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6805/12889", 2022-06-29T16:06:16.184 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:06:16.184 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:06:16.185 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:06:16.185 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.185 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.185 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.185 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:06:16.186 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "3d234db9-87cd-4fe9-814f-0240a287d8b4", 2022-06-29T16:06:16.186 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:06:16.186 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:06:16.186 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:06:16.187 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:06:16.187 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:06:16.187 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:06:16.187 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:06:16.188 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 97, 2022-06-29T16:06:16.188 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:06:16.188 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:06:16.188 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:06:16.188 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:16.189 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.189 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:16.189 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6808", 2022-06-29T16:06:16.189 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:06:16.190 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.190 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.190 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:16.190 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6810", 2022-06-29T16:06:16.190 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:06:16.191 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.191 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.191 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.191 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:06:16.192 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:16.192 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.192 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:16.192 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6812", 2022-06-29T16:06:16.193 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:06:16.193 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.193 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.193 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:16.193 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6814", 2022-06-29T16:06:16.194 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:06:16.194 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.194 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.195 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.195 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:06:16.195 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:16.195 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.195 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:16.196 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6823", 2022-06-29T16:06:16.196 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:06:16.196 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.196 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.196 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:16.197 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6825", 2022-06-29T16:06:16.197 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:06:16.197 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.197 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.198 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.198 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:06:16.198 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:16.198 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.198 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:16.199 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6817", 2022-06-29T16:06:16.199 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:06:16.199 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.199 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.200 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:16.200 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6820", 2022-06-29T16:06:16.200 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:06:16.200 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.200 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.201 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.201 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6810/13249", 2022-06-29T16:06:16.201 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6814/13249", 2022-06-29T16:06:16.201 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6825/13249", 2022-06-29T16:06:16.202 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6820/13249", 2022-06-29T16:06:16.202 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:06:16.202 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:06:16.202 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:06:16.202 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.203 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.203 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.203 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:06:16.203 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "e4bf1104-ff5c-438b-b608-8cee31c163f6", 2022-06-29T16:06:16.203 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:06:16.204 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:06:16.204 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:06:16.204 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:06:16.204 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:06:16.205 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:06:16.205 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:06:16.205 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 97, 2022-06-29T16:06:16.205 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:06:16.205 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:06:16.206 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:06:16.206 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:16.206 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.207 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:16.207 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6809", 2022-06-29T16:06:16.207 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:06:16.207 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.207 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.208 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:16.208 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6811", 2022-06-29T16:06:16.208 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:06:16.208 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.209 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.209 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.209 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:06:16.209 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:16.209 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.210 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:16.210 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6813", 2022-06-29T16:06:16.210 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:06:16.210 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.210 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.211 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:16.211 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6815", 2022-06-29T16:06:16.211 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:06:16.211 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.212 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.212 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.212 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:06:16.212 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:16.212 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.213 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:16.213 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6824", 2022-06-29T16:06:16.213 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:06:16.213 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.214 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.214 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:16.214 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6827", 2022-06-29T16:06:16.214 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:06:16.214 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.215 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.215 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.215 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:06:16.215 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:16.216 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.216 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:16.216 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6818", 2022-06-29T16:06:16.216 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:06:16.216 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.217 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.217 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:16.217 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6821", 2022-06-29T16:06:16.217 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:06:16.217 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.218 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.218 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.218 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6811/13248", 2022-06-29T16:06:16.218 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6815/13248", 2022-06-29T16:06:16.219 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6827/13248", 2022-06-29T16:06:16.219 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6821/13248", 2022-06-29T16:06:16.219 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:06:16.219 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:06:16.219 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:06:16.220 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.220 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.220 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.220 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:06:16.220 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "8564dd25-27ab-4203-b54e-af6d3aa05857", 2022-06-29T16:06:16.221 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:06:16.221 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:06:16.221 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:06:16.221 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:06:16.222 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:06:16.222 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:06:16.222 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:06:16.222 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 98, 2022-06-29T16:06:16.222 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:06:16.223 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:06:16.223 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:06:16.223 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:16.223 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.224 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:16.224 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6816", 2022-06-29T16:06:16.224 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:06:16.224 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.224 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.225 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:16.225 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6819", 2022-06-29T16:06:16.225 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:06:16.225 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.225 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.226 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.226 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:06:16.226 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:16.226 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.227 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:16.227 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6822", 2022-06-29T16:06:16.227 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:06:16.227 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.227 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.228 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:16.228 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6826", 2022-06-29T16:06:16.228 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:06:16.228 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.228 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.229 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.229 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:06:16.229 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:16.229 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.230 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:16.230 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6830", 2022-06-29T16:06:16.230 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:06:16.230 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.230 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.231 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:16.231 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6831", 2022-06-29T16:06:16.231 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:06:16.231 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.232 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.232 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.232 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:06:16.232 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:16.232 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.233 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:16.233 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6828", 2022-06-29T16:06:16.233 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:06:16.233 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.233 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.234 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:16.234 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6829", 2022-06-29T16:06:16.234 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:06:16.234 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.235 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.235 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.235 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6819/13250", 2022-06-29T16:06:16.235 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6826/13250", 2022-06-29T16:06:16.235 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6831/13250", 2022-06-29T16:06:16.236 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6829/13250", 2022-06-29T16:06:16.236 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:06:16.236 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:06:16.236 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:06:16.236 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.237 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.237 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.237 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:06:16.237 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "bd513a9c-3b19-4314-9897-d300b5c897bb", 2022-06-29T16:06:16.237 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:06:16.238 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:06:16.238 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:06:16.238 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:06:16.238 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:06:16.239 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:06:16.239 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:06:16.239 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 98, 2022-06-29T16:06:16.239 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:06:16.239 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:06:16.240 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:06:16.240 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:16.240 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.240 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:16.240 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6800", 2022-06-29T16:06:16.241 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:06:16.241 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.241 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.241 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:16.242 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6801", 2022-06-29T16:06:16.242 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:06:16.242 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.242 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.242 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.243 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:06:16.243 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:16.243 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.243 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:16.243 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6802", 2022-06-29T16:06:16.244 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:06:16.244 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.244 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.244 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:16.245 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6803", 2022-06-29T16:06:16.245 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:06:16.245 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.245 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.245 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.246 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:06:16.246 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:16.246 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.246 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:16.247 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6806", 2022-06-29T16:06:16.247 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:06:16.247 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.247 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.248 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:16.248 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6807", 2022-06-29T16:06:16.248 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:06:16.248 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.248 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.249 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.249 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:06:16.249 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:16.249 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.250 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:16.250 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6804", 2022-06-29T16:06:16.250 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:06:16.250 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.250 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.251 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:16.251 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6805", 2022-06-29T16:06:16.251 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:06:16.251 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.251 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.252 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.252 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6801/13247", 2022-06-29T16:06:16.252 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6803/13247", 2022-06-29T16:06:16.252 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6807/13247", 2022-06-29T16:06:16.253 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6805/13247", 2022-06-29T16:06:16.253 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:06:16.253 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:06:16.253 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:06:16.253 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:16.254 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.254 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:06:16.254 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_xinfo": [ 2022-06-29T16:06:16.254 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.255 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:06:16.255 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:06:16.255 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:06:16.255 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:06:16.255 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:06:16.256 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:06:16.256 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.891342+0000", 2022-06-29T16:06:16.256 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:06:16.256 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.257 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.257 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:06:16.257 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:06:16.257 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:06:16.257 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:06:16.258 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:06:16.258 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:06:16.258 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.900866+0000", 2022-06-29T16:06:16.258 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:06:16.259 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.259 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.259 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:06:16.259 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:06:16.259 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:06:16.260 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:06:16.260 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:06:16.260 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:06:16.260 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.912755+0000", 2022-06-29T16:06:16.260 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:06:16.261 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.261 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.261 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:06:16.261 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:06:16.261 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:06:16.262 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:06:16.262 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:06:16.262 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:06:16.262 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.864332+0000", 2022-06-29T16:06:16.263 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:06:16.263 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.263 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.263 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:06:16.263 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:06:16.264 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:06:16.264 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:06:16.264 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:06:16.264 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:06:16.264 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.863756+0000", 2022-06-29T16:06:16.265 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:06:16.265 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.265 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.265 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:06:16.266 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:06:16.266 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:06:16.266 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:06:16.266 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:06:16.267 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:06:16.267 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.869041+0000", 2022-06-29T16:06:16.267 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:06:16.267 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.267 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.268 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:06:16.268 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:06:16.268 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:06:16.268 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:06:16.268 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:06:16.269 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:06:16.269 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.895043+0000", 2022-06-29T16:06:16.269 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:06:16.269 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.270 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:16.270 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:06:16.270 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:06:16.270 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:06:16.270 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:06:16.271 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:06:16.271 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:06:16.271 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.889491+0000", 2022-06-29T16:06:16.271 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:06:16.272 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.272 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:06:16.272 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap": [], 2022-06-29T16:06:16.272 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap_items": [], 2022-06-29T16:06:16.272 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_temp": [], 2022-06-29T16:06:16.273 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_temp": [], 2022-06-29T16:06:16.273 INFO:teuthology.orchestra.run.smithi060.stdout: "blacklist": {}, 2022-06-29T16:06:16.273 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profiles": { 2022-06-29T16:06:16.273 INFO:teuthology.orchestra.run.smithi060.stdout: "default": { 2022-06-29T16:06:16.274 INFO:teuthology.orchestra.run.smithi060.stdout: "crush-failure-domain": "osd", 2022-06-29T16:06:16.274 INFO:teuthology.orchestra.run.smithi060.stdout: "k": "2", 2022-06-29T16:06:16.274 INFO:teuthology.orchestra.run.smithi060.stdout: "m": "1", 2022-06-29T16:06:16.274 INFO:teuthology.orchestra.run.smithi060.stdout: "plugin": "jerasure", 2022-06-29T16:06:16.274 INFO:teuthology.orchestra.run.smithi060.stdout: "ruleset-failure-domain": "osd", 2022-06-29T16:06:16.275 INFO:teuthology.orchestra.run.smithi060.stdout: "technique": "reed_sol_van" 2022-06-29T16:06:16.275 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:16.275 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:16.275 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps_queue": [], 2022-06-29T16:06:16.275 INFO:teuthology.orchestra.run.smithi060.stdout: "new_removed_snaps": [], 2022-06-29T16:06:16.276 INFO:teuthology.orchestra.run.smithi060.stdout: "new_purged_snaps": [], 2022-06-29T16:06:16.276 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_node_flags": {}, 2022-06-29T16:06:16.276 INFO:teuthology.orchestra.run.smithi060.stdout: "device_class_flags": {} 2022-06-29T16:06:16.276 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:06:16.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.037+0000 7f4cfe270700 1 -- 172.21.15.60:0/2899617314 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4cdc053a70 msgr2=0x7f4cdc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:16.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.037+0000 7f4cfe270700 1 --2- 172.21.15.60:0/2899617314 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4cdc053a70 0x7f4cdc055f30 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0x7f4ce000c000 tx=0x7f4ce0008040).stop 2022-06-29T16:06:16.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.037+0000 7f4cfe270700 1 -- 172.21.15.60:0/2899617314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cf80b5950 msgr2=0x7f4cf8155dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:16.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.037+0000 7f4cfe270700 1 --2- 172.21.15.60:0/2899617314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cf80b5950 0x7f4cf8155dc0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 rx=0x7f4cf00032f0 tx=0x7f4cf0016000).stop 2022-06-29T16:06:16.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.037+0000 7f4cfe270700 1 -- 172.21.15.60:0/2899617314 shutdown_connections 2022-06-29T16:06:16.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.037+0000 7f4cfe270700 1 --2- 172.21.15.60:0/2899617314 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4cdc053a70 0x7f4cdc055f30 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:16.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.037+0000 7f4cfe270700 1 --2- 172.21.15.60:0/2899617314 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4cf80b5950 0x7f4cf8155dc0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:16.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.037+0000 7f4cfe270700 1 --2- 172.21.15.60:0/2899617314 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4cf8063ad0 0x7f4cf80ae410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:16.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.041+0000 7f4cfe270700 1 --2- 172.21.15.60:0/2899617314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4cf805bd20 0x7f4cf8064470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:16.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.041+0000 7f4cfe270700 1 -- 172.21.15.60:0/2899617314 >> 172.21.15.60:0/2899617314 conn(0x7f4cf80632c0 msgr2=0x7f4cf81460e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:16.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.041+0000 7f4cfe270700 1 -- 172.21.15.60:0/2899617314 shutdown_connections 2022-06-29T16:06:16.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.041+0000 7f4cfe270700 1 -- 172.21.15.60:0/2899617314 wait complete. 2022-06-29T16:06:16.282 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:06:16.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.393+0000 7fa073fff700 1 Processor -- start 2022-06-29T16:06:16.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.393+0000 7fa073fff700 1 -- start start 2022-06-29T16:06:16.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.393+0000 7fa073fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa07405c5b0 0x7fa07405c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:16.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.393+0000 7fa073fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa074060610 0x7fa07413c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:16.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.393+0000 7fa073fff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0740b0a40 con 0x7fa07413cc30 2022-06-29T16:06:16.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.393+0000 7fa073fff700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0740b0850 con 0x7fa074060610 2022-06-29T16:06:16.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.393+0000 7fa073fff700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0740b0ce0 con 0x7fa07405c5b0 2022-06-29T16:06:16.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.393+0000 7fa0727fc700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa074060610 0x7fa07413c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:16.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.393+0000 7fa0727fc700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fa074060610 0x7fa07413c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51708/0 (socket says 172.21.15.60:51708) 2022-06-29T16:06:16.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.393+0000 7fa0727fc700 1 -- 172.21.15.60:0/4157263587 learned_addr learned my addr 172.21.15.60:0/4157263587 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:16.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.393+0000 7fa072ffd700 1 --2- 172.21.15.60:0/4157263587 >> v2:172.21.15.92:3300/0 conn(0x7fa07405c5b0 0x7fa07405c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:16.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.393+0000 7fa0727fc700 1 -- 172.21.15.60:0/4157263587 >> v2:172.21.15.92:3300/0 conn(0x7fa07405c5b0 msgr2=0x7fa07405c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:16.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.393+0000 7fa0727fc700 1 --2- 172.21.15.60:0/4157263587 >> v2:172.21.15.92:3300/0 conn(0x7fa07405c5b0 0x7fa07405c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:16.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.393+0000 7fa0727fc700 1 -- 172.21.15.60:0/4157263587 >> v1:172.21.15.60:6789/0 conn(0x7fa07413cc30 legacy=0x7fa074142a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:16.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.393+0000 7fa0727fc700 1 -- 172.21.15.60:0/4157263587 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa07405f350 con 0x7fa074060610 2022-06-29T16:06:16.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.393+0000 7fa0727fc700 1 --2- 172.21.15.60:0/4157263587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa074060610 0x7fa07413c740 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 rx=0x7fa05c002f70 tx=0x7fa05c006eb0).ready entity=mon.0 client_cookie=9642c7471b495532 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:16.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.393+0000 7fa071ffb700 1 -- 172.21.15.60:0/4157263587 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa05c00d610 con 0x7fa074060610 2022-06-29T16:06:16.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.393+0000 7fa071ffb700 1 -- 172.21.15.60:0/4157263587 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa05c000e90 con 0x7fa074060610 2022-06-29T16:06:16.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.393+0000 7fa071ffb700 1 -- 172.21.15.60:0/4157263587 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa05c00d9a0 con 0x7fa074060610 2022-06-29T16:06:16.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.397+0000 7fa073fff700 1 -- 172.21.15.60:0/4157263587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa074060610 msgr2=0x7fa07413c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:16.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.397+0000 7fa073fff700 1 --2- 172.21.15.60:0/4157263587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa074060610 0x7fa07413c740 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 rx=0x7fa05c002f70 tx=0x7fa05c006eb0).stop 2022-06-29T16:06:16.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.397+0000 7fa073fff700 1 -- 172.21.15.60:0/4157263587 shutdown_connections 2022-06-29T16:06:16.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.397+0000 7fa073fff700 1 --2- 172.21.15.60:0/4157263587 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa074060610 0x7fa07413c740 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:16.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.397+0000 7fa073fff700 1 --2- 172.21.15.60:0/4157263587 >> v2:172.21.15.92:3300/0 conn(0x7fa07405c5b0 0x7fa07405c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:16.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.397+0000 7fa073fff700 1 -- 172.21.15.60:0/4157263587 >> 172.21.15.60:0/4157263587 conn(0x7fa074061c80 msgr2=0x7fa0740637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:16.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.397+0000 7fa073fff700 1 -- 172.21.15.60:0/4157263587 shutdown_connections 2022-06-29T16:06:16.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.397+0000 7fa073fff700 1 -- 172.21.15.60:0/4157263587 wait complete. 2022-06-29T16:06:16.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.397+0000 7fa073fff700 1 Processor -- start 2022-06-29T16:06:16.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.397+0000 7fa073fff700 1 -- start start 2022-06-29T16:06:16.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.397+0000 7fa073fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa07405c5b0 0x7fa074145150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:16.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.397+0000 7fa073fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa074060610 0x7fa074144070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:16.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.397+0000 7fa073fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa07413cc30 0x7fa074144560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:16.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.397+0000 7fa073fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0740afd20 con 0x7fa07413cc30 2022-06-29T16:06:16.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.397+0000 7fa073fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa074144cb0 con 0x7fa07405c5b0 2022-06-29T16:06:16.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.397+0000 7fa073fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0740af7a0 con 0x7fa074060610 2022-06-29T16:06:16.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.397+0000 7fa0737fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa07413cc30 0x7fa074144560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:16.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.397+0000 7fa0737fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa07413cc30 0x7fa074144560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51718/0 (socket says 172.21.15.60:51718) 2022-06-29T16:06:16.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.397+0000 7fa072ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa07405c5b0 0x7fa074145150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:16.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.397+0000 7fa0737fe700 1 -- 172.21.15.60:0/1130306628 learned_addr learned my addr 172.21.15.60:0/1130306628 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:16.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.397+0000 7fa0727fc700 1 --2- 172.21.15.60:0/1130306628 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa074060610 0x7fa074144070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:16.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.397+0000 7fa0737fe700 1 -- 172.21.15.60:0/1130306628 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa074060610 msgr2=0x7fa074144070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:16.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.397+0000 7fa0737fe700 1 --2- 172.21.15.60:0/1130306628 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa074060610 0x7fa074144070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:16.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.397+0000 7fa0737fe700 1 -- 172.21.15.60:0/1130306628 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa07405c5b0 msgr2=0x7fa074145150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:16.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.397+0000 7fa0737fe700 1 --2- 172.21.15.60:0/1130306628 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa07405c5b0 0x7fa074145150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:16.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.397+0000 7fa0737fe700 1 -- 172.21.15.60:0/1130306628 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa05c00b1e0 con 0x7fa07413cc30 2022-06-29T16:06:16.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.397+0000 7fa0737fe700 1 --2- 172.21.15.60:0/1130306628 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa07413cc30 0x7fa074144560 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 rx=0x7fa0740652a0 tx=0x7fa06c0013a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:16.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.401+0000 7fa06bfff700 1 -- 172.21.15.60:0/1130306628 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa06c006aa0 con 0x7fa07413cc30 2022-06-29T16:06:16.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.401+0000 7fa06bfff700 1 -- 172.21.15.60:0/1130306628 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa06c006c00 con 0x7fa07413cc30 2022-06-29T16:06:16.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.401+0000 7fa073fff700 1 -- 172.21.15.60:0/1130306628 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa074140e70 con 0x7fa07413cc30 2022-06-29T16:06:16.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.401+0000 7fa06bfff700 1 -- 172.21.15.60:0/1130306628 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa06c007030 con 0x7fa07413cc30 2022-06-29T16:06:16.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.401+0000 7fa073fff700 1 -- 172.21.15.60:0/1130306628 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0741457e0 con 0x7fa07413cc30 2022-06-29T16:06:16.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.401+0000 7fa073fff700 1 -- 172.21.15.60:0/1130306628 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0740afe90 con 0x7fa07413cc30 2022-06-29T16:06:16.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.401+0000 7fa06bfff700 1 -- 172.21.15.60:0/1130306628 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa06c010030 con 0x7fa07413cc30 2022-06-29T16:06:16.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.401+0000 7fa06bfff700 1 --2- 172.21.15.60:0/1130306628 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa058053b30 0x7fa058055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:16.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.401+0000 7fa06bfff700 1 -- 172.21.15.60:0/1130306628 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa06c06bd00 con 0x7fa07413cc30 2022-06-29T16:06:16.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.401+0000 7fa072ffd700 1 --2- 172.21.15.60:0/1130306628 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa058053b30 0x7fa058055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:16.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.405+0000 7fa072ffd700 1 --2- 172.21.15.60:0/1130306628 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa058053b30 0x7fa058055ff0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7fa06400c000 tx=0x7fa064008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:16.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.409+0000 7fa06bfff700 1 -- 172.21.15.60:0/1130306628 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa0740afe90 con 0x7fa07413cc30 2022-06-29T16:06:16.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.573+0000 7fa073fff700 1 -- 172.21.15.60:0/1130306628 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa074144e70 con 0x7fa07413cc30 2022-06-29T16:06:16.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.573+0000 7fa06bfff700 1 -- 172.21.15.60:0/1130306628 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+2140 (secure 0 0 0) 0x7fa06c06f030 con 0x7fa07413cc30 2022-06-29T16:06:16.579 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:16.579 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":46,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7320,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/3498160620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3498160620},{"type":"v1","addr":"172.21.15.60:6833","nonce":3498160620}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:06:05.226825+0000","modified":"2022-06-29T16:06:06.241973+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7327},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7327":{"gid":7327,"name":"b","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6835/1551394182","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":1551394182},{"type":"v1","addr":"172.21.15.92:6835","nonce":1551394182}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2022-06-29T16:06:16.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.577+0000 7fa073fff700 1 -- 172.21.15.60:0/1130306628 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa058053b30 msgr2=0x7fa058055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:16.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.577+0000 7fa073fff700 1 --2- 172.21.15.60:0/1130306628 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa058053b30 0x7fa058055ff0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7fa06400c000 tx=0x7fa064008040).stop 2022-06-29T16:06:16.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.577+0000 7fa073fff700 1 -- 172.21.15.60:0/1130306628 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa07413cc30 msgr2=0x7fa074144560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:16.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.577+0000 7fa073fff700 1 --2- 172.21.15.60:0/1130306628 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa07413cc30 0x7fa074144560 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 rx=0x7fa0740652a0 tx=0x7fa06c0013a0).stop 2022-06-29T16:06:16.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.577+0000 7fa073fff700 1 -- 172.21.15.60:0/1130306628 shutdown_connections 2022-06-29T16:06:16.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.577+0000 7fa073fff700 1 --2- 172.21.15.60:0/1130306628 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa058053b30 0x7fa058055ff0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:16.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.577+0000 7fa073fff700 1 --2- 172.21.15.60:0/1130306628 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa07413cc30 0x7fa074144560 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:16.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.577+0000 7fa073fff700 1 --2- 172.21.15.60:0/1130306628 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa074060610 0x7fa074144070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:16.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.577+0000 7fa073fff700 1 --2- 172.21.15.60:0/1130306628 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa07405c5b0 0x7fa074145150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:16.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.577+0000 7fa073fff700 1 -- 172.21.15.60:0/1130306628 >> 172.21.15.60:0/1130306628 conn(0x7fa074061c80 msgr2=0x7fa0740b2270 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:16.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.577+0000 7fa073fff700 1 -- 172.21.15.60:0/1130306628 shutdown_connections 2022-06-29T16:06:16.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.577+0000 7fa073fff700 1 -- 172.21.15.60:0/1130306628 wait complete. 2022-06-29T16:06:16.584 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 46 2022-06-29T16:06:16.592 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-06-29T16:06:16.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.693+0000 7f06d315d700 1 Processor -- start 2022-06-29T16:06:16.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.693+0000 7f06d315d700 1 -- start start 2022-06-29T16:06:16.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.693+0000 7f06d315d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f06cc0b4010 0x7f06cc1426b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:16.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.693+0000 7f06d315d700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f06cc0b46c0 0x7f06cc0b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:16.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.693+0000 7f06d315d700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06cc143a40 con 0x7f06cc0ae7c0 2022-06-29T16:06:16.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.693+0000 7f06d315d700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06cc0aee60 con 0x7f06cc0b4010 2022-06-29T16:06:16.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.693+0000 7f06d315d700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06cc0af100 con 0x7f06cc0b46c0 2022-06-29T16:06:16.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.693+0000 7f06d0ef9700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f06cc0ae7c0 0x7f06cc141fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59808/0 (socket says 172.21.15.60:59808) 2022-06-29T16:06:16.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.693+0000 7f06cbfff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f06cc0b4010 0x7f06cc1426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:16.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.693+0000 7f06d0ef9700 1 -- 172.21.15.60:0/1521431043 learned_addr learned my addr 172.21.15.60:0/1521431043 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:16.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.693+0000 7f06d16fa700 1 --2- 172.21.15.60:0/1521431043 >> v2:172.21.15.92:3300/0 conn(0x7f06cc0b46c0 0x7f06cc0b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:16.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.693+0000 7f06cb7fe700 1 -- 172.21.15.60:0/1521431043 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3467850889 0 0) 0x7f06cc143a40 con 0x7f06cc0ae7c0 2022-06-29T16:06:16.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.693+0000 7f06cb7fe700 1 -- 172.21.15.60:0/1521431043 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06b8003260 con 0x7f06cc0ae7c0 2022-06-29T16:06:16.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.693+0000 7f06cbfff700 1 -- 172.21.15.60:0/1521431043 >> v2:172.21.15.92:3300/0 conn(0x7f06cc0b46c0 msgr2=0x7f06cc0b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:16.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.693+0000 7f06cbfff700 1 --2- 172.21.15.60:0/1521431043 >> v2:172.21.15.92:3300/0 conn(0x7f06cc0b46c0 0x7f06cc0b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:16.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.693+0000 7f06cbfff700 1 -- 172.21.15.60:0/1521431043 >> v1:172.21.15.60:6789/0 conn(0x7f06cc0ae7c0 legacy=0x7f06cc141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:16.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.693+0000 7f06cbfff700 1 -- 172.21.15.60:0/1521431043 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06cc0aeba0 con 0x7f06cc0b4010 2022-06-29T16:06:16.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.693+0000 7f06d16fa700 1 --2- 172.21.15.60:0/1521431043 >> v2:172.21.15.92:3300/0 conn(0x7f06cc0b46c0 0x7f06cc0b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:06:16.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.693+0000 7f06cbfff700 1 --2- 172.21.15.60:0/1521431043 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06cc0b4010 0x7f06cc1426b0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 rx=0x7f06c0002f70 tx=0x7f06c0006df0).ready entity=mon.0 client_cookie=fa615ea5b39820f3 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:16.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.693+0000 7f06cb7fe700 1 -- 172.21.15.60:0/1521431043 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f06c000d570 con 0x7f06cc0b4010 2022-06-29T16:06:16.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.693+0000 7f06cb7fe700 1 -- 172.21.15.60:0/1521431043 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c0000e90 con 0x7f06cc0b4010 2022-06-29T16:06:16.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.693+0000 7f06cb7fe700 1 -- 172.21.15.60:0/1521431043 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f06c000d8c0 con 0x7f06cc0b4010 2022-06-29T16:06:16.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.697+0000 7f06d315d700 1 -- 172.21.15.60:0/1521431043 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06cc0b4010 msgr2=0x7f06cc1426b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:16.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.697+0000 7f06d315d700 1 --2- 172.21.15.60:0/1521431043 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06cc0b4010 0x7f06cc1426b0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 rx=0x7f06c0002f70 tx=0x7f06c0006df0).stop 2022-06-29T16:06:16.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.697+0000 7f06d315d700 1 -- 172.21.15.60:0/1521431043 shutdown_connections 2022-06-29T16:06:16.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.697+0000 7f06d315d700 1 --2- 172.21.15.60:0/1521431043 >> v2:172.21.15.92:3300/0 conn(0x7f06cc0b46c0 0x7f06cc0b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:16.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.697+0000 7f06d315d700 1 --2- 172.21.15.60:0/1521431043 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06cc0b4010 0x7f06cc1426b0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:16.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.697+0000 7f06d315d700 1 -- 172.21.15.60:0/1521431043 >> 172.21.15.60:0/1521431043 conn(0x7f06cc0b9340 msgr2=0x7f06cc0b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:16.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.697+0000 7f06d315d700 1 -- 172.21.15.60:0/1521431043 shutdown_connections 2022-06-29T16:06:16.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.697+0000 7f06d315d700 1 -- 172.21.15.60:0/1521431043 wait complete. 2022-06-29T16:06:16.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.697+0000 7f06d315d700 1 Processor -- start 2022-06-29T16:06:16.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.697+0000 7f06d315d700 1 -- start start 2022-06-29T16:06:16.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.697+0000 7f06d315d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06cc0ae7c0 0x7f06cc0b8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:16.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.697+0000 7f06d315d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06cc0b4010 0x7f06cc13fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:16.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.697+0000 7f06d315d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06cc0b46c0 0x7f06cc140130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:16.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.697+0000 7f06d315d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06cc140620 con 0x7f06cc0b4010 2022-06-29T16:06:16.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.697+0000 7f06d315d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06cc1529c0 con 0x7f06cc0b46c0 2022-06-29T16:06:16.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.697+0000 7f06d315d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06cc1526f0 con 0x7f06cc0ae7c0 2022-06-29T16:06:16.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.701+0000 7f06cbfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06cc0b4010 0x7f06cc13fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:16.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.701+0000 7f06cbfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06cc0b4010 0x7f06cc13fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51730/0 (socket says 172.21.15.60:51730) 2022-06-29T16:06:16.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.701+0000 7f06cbfff700 1 -- 172.21.15.60:0/2478847242 learned_addr learned my addr 172.21.15.60:0/2478847242 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:16.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.701+0000 7f06d16fa700 1 --2- 172.21.15.60:0/2478847242 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06cc0b46c0 0x7f06cc140130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:16.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.701+0000 7f06d0ef9700 1 --2- 172.21.15.60:0/2478847242 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06cc0ae7c0 0x7f06cc0b8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:16.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.701+0000 7f06cbfff700 1 -- 172.21.15.60:0/2478847242 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06cc0ae7c0 msgr2=0x7f06cc0b8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:16.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.701+0000 7f06cbfff700 1 --2- 172.21.15.60:0/2478847242 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06cc0ae7c0 0x7f06cc0b8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:16.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.701+0000 7f06cbfff700 1 -- 172.21.15.60:0/2478847242 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06cc0b46c0 msgr2=0x7f06cc140130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:16.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.701+0000 7f06cbfff700 1 --2- 172.21.15.60:0/2478847242 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06cc0b46c0 0x7f06cc140130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:16.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.701+0000 7f06cbfff700 1 -- 172.21.15.60:0/2478847242 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c000b180 con 0x7f06cc0b4010 2022-06-29T16:06:16.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.701+0000 7f06cbfff700 1 --2- 172.21.15.60:0/2478847242 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06cc0b4010 0x7f06cc13fc40 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 rx=0x7f06c0002f70 tx=0x7f06c0014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:16.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.701+0000 7f06c9ffb700 1 -- 172.21.15.60:0/2478847242 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f06c0016070 con 0x7f06cc0b4010 2022-06-29T16:06:16.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.701+0000 7f06c9ffb700 1 -- 172.21.15.60:0/2478847242 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c00138d0 con 0x7f06cc0b4010 2022-06-29T16:06:16.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.701+0000 7f06d315d700 1 -- 172.21.15.60:0/2478847242 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06cc147ce0 con 0x7f06cc0b4010 2022-06-29T16:06:16.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.701+0000 7f06c9ffb700 1 -- 172.21.15.60:0/2478847242 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f06c0013c30 con 0x7f06cc0b4010 2022-06-29T16:06:16.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.701+0000 7f06d315d700 1 -- 172.21.15.60:0/2478847242 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06cc1472d0 con 0x7f06cc0b4010 2022-06-29T16:06:16.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.701+0000 7f06c9ffb700 1 -- 172.21.15.60:0/2478847242 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f06c0025030 con 0x7f06cc0b4010 2022-06-29T16:06:16.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.701+0000 7f06d315d700 1 -- 172.21.15.60:0/2478847242 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06cc0573d0 con 0x7f06cc0b4010 2022-06-29T16:06:16.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.701+0000 7f06c9ffb700 1 --2- 172.21.15.60:0/2478847242 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f06ac053aa0 0x7f06ac055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:16.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.701+0000 7f06c9ffb700 1 -- 172.21.15.60:0/2478847242 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f06c006ce50 con 0x7f06cc0b4010 2022-06-29T16:06:16.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.705+0000 7f06d0ef9700 1 --2- 172.21.15.60:0/2478847242 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f06ac053aa0 0x7f06ac055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:16.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.705+0000 7f06d0ef9700 1 --2- 172.21.15.60:0/2478847242 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f06ac053aa0 0x7f06ac055f60 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7f06bc00d000 tx=0x7f06bc009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:16.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.709+0000 7f06c9ffb700 1 -- 172.21.15.60:0/2478847242 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f06c0043c60 con 0x7f06cc0b4010 2022-06-29T16:06:16.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:16.873+0000 7f06d315d700 1 -- 172.21.15.60:0/2478847242 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f06cc05fdb0 con 0x7f06cc0b4010 2022-06-29T16:06:17.418 INFO:tasks.ceph.mon.a.smithi060.stderr:2022-06-29T16:06:17.413+0000 7fcc1a376700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-06-29T16:06:17.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.421+0000 7f06c9ffb700 1 -- 172.21.15.60:0/2478847242 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v47) v1 ==== 161+0+0 (secure 0 0 0) 0x7f06c0023020 con 0x7f06cc0b4010 2022-06-29T16:06:17.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.425+0000 7f06d315d700 1 -- 172.21.15.60:0/2478847242 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f06ac053aa0 msgr2=0x7f06ac055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:17.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.425+0000 7f06d315d700 1 --2- 172.21.15.60:0/2478847242 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f06ac053aa0 0x7f06ac055f60 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7f06bc00d000 tx=0x7f06bc009040).stop 2022-06-29T16:06:17.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.425+0000 7f06d315d700 1 -- 172.21.15.60:0/2478847242 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06cc0b4010 msgr2=0x7f06cc13fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:17.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.425+0000 7f06d315d700 1 --2- 172.21.15.60:0/2478847242 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06cc0b4010 0x7f06cc13fc40 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 rx=0x7f06c0002f70 tx=0x7f06c0014000).stop 2022-06-29T16:06:17.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.425+0000 7f06d315d700 1 -- 172.21.15.60:0/2478847242 shutdown_connections 2022-06-29T16:06:17.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.425+0000 7f06d315d700 1 --2- 172.21.15.60:0/2478847242 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f06ac053aa0 0x7f06ac055f60 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:17.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.425+0000 7f06d315d700 1 --2- 172.21.15.60:0/2478847242 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06cc0b46c0 0x7f06cc140130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:17.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.425+0000 7f06d315d700 1 --2- 172.21.15.60:0/2478847242 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f06cc0b4010 0x7f06cc13fc40 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:17.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.425+0000 7f06d315d700 1 --2- 172.21.15.60:0/2478847242 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06cc0ae7c0 0x7f06cc0b8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:17.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.425+0000 7f06d315d700 1 -- 172.21.15.60:0/2478847242 >> 172.21.15.60:0/2478847242 conn(0x7f06cc0b9340 msgr2=0x7f06cc148250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:17.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.425+0000 7f06d315d700 1 -- 172.21.15.60:0/2478847242 shutdown_connections 2022-06-29T16:06:17.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.425+0000 7f06d315d700 1 -- 172.21.15.60:0/2478847242 wait complete. 2022-06-29T16:06:17.433 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-06-29T16:06:17.445 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:06:17.453 INFO:tasks.ceph.mds.b.smithi092.stdout:starting mds.b at 2022-06-29T16:06:17.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.549+0000 7fee2c978700 1 Processor -- start 2022-06-29T16:06:17.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.549+0000 7fee2c978700 1 -- start start 2022-06-29T16:06:17.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.553+0000 7fee2c978700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee280b6a50 0x7fee281461e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:17.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.553+0000 7fee2c978700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fee2805ca50 0x7fee2805cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:17.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.553+0000 7fee2c978700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee280b6e30 con 0x7fee2805f0b0 2022-06-29T16:06:17.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.553+0000 7fee2c978700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee280b0dd0 con 0x7fee2805ca50 2022-06-29T16:06:17.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.553+0000 7fee2c978700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee280b1070 con 0x7fee280b6a50 2022-06-29T16:06:17.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.553+0000 7fee26d9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fee2805ca50 0x7fee2805cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:17.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.553+0000 7fee26d9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fee2805ca50 0x7fee2805cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51740/0 (socket says 172.21.15.60:51740) 2022-06-29T16:06:17.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.553+0000 7fee26d9d700 1 -- 172.21.15.60:0/2865947910 learned_addr learned my addr 172.21.15.60:0/2865947910 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:17.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.553+0000 7fee2659c700 1 --2- 172.21.15.60:0/2865947910 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee280b6a50 0x7fee281461e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:17.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.553+0000 7fee26d9d700 1 -- 172.21.15.60:0/2865947910 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee280b6a50 msgr2=0x7fee281461e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:17.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.553+0000 7fee26d9d700 1 --2- 172.21.15.60:0/2865947910 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee280b6a50 0x7fee281461e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:17.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.553+0000 7fee26d9d700 1 -- 172.21.15.60:0/2865947910 >> v1:172.21.15.92:6789/0 conn(0x7fee2805f0b0 legacy=0x7fee28142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:17.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.553+0000 7fee26d9d700 1 -- 172.21.15.60:0/2865947910 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee2805c5b0 con 0x7fee2805ca50 2022-06-29T16:06:17.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.553+0000 7fee2659c700 1 --2- 172.21.15.60:0/2865947910 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee280b6a50 0x7fee281461e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:06:17.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.553+0000 7fee26d9d700 1 --2- 172.21.15.60:0/2865947910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee2805ca50 0x7fee2805cec0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 rx=0x7fee200032f0 tx=0x7fee20009290).ready entity=mon.0 client_cookie=40de17cc66618e1a server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:17.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.553+0000 7fee2559a700 1 -- 172.21.15.60:0/2865947910 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee2000b8a0 con 0x7fee2805ca50 2022-06-29T16:06:17.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.553+0000 7fee2559a700 1 -- 172.21.15.60:0/2865947910 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee2000ba00 con 0x7fee2805ca50 2022-06-29T16:06:17.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.553+0000 7fee2559a700 1 -- 172.21.15.60:0/2865947910 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee2000bb70 con 0x7fee2805ca50 2022-06-29T16:06:17.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.553+0000 7fee2c978700 1 -- 172.21.15.60:0/2865947910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee2805ca50 msgr2=0x7fee2805cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:17.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.553+0000 7fee2c978700 1 --2- 172.21.15.60:0/2865947910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee2805ca50 0x7fee2805cec0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 rx=0x7fee200032f0 tx=0x7fee20009290).stop 2022-06-29T16:06:17.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.553+0000 7fee2c978700 1 -- 172.21.15.60:0/2865947910 shutdown_connections 2022-06-29T16:06:17.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.553+0000 7fee2c978700 1 --2- 172.21.15.60:0/2865947910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee2805ca50 0x7fee2805cec0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:17.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.553+0000 7fee2c978700 1 --2- 172.21.15.60:0/2865947910 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee280b6a50 0x7fee281461e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:17.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.553+0000 7fee2c978700 1 -- 172.21.15.60:0/2865947910 >> 172.21.15.60:0/2865947910 conn(0x7fee280643e0 msgr2=0x7fee280b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:17.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.553+0000 7fee2c978700 1 -- 172.21.15.60:0/2865947910 shutdown_connections 2022-06-29T16:06:17.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.553+0000 7fee2c978700 1 -- 172.21.15.60:0/2865947910 wait complete. 2022-06-29T16:06:17.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.557+0000 7fee2c978700 1 Processor -- start 2022-06-29T16:06:17.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.557+0000 7fee2c978700 1 -- start start 2022-06-29T16:06:17.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.557+0000 7fee2c978700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee2805ca50 0x7fee280b2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:17.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.557+0000 7fee2c978700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee2805f0b0 0x7fee28065b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:17.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.557+0000 7fee2c978700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee280b6a50 0x7fee28066020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:17.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.557+0000 7fee2c978700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee281440e0 con 0x7fee2805ca50 2022-06-29T16:06:17.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.557+0000 7fee2c978700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2809bc30 con 0x7fee280b6a50 2022-06-29T16:06:17.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.557+0000 7fee2c978700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee28099690 con 0x7fee2805f0b0 2022-06-29T16:06:17.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.557+0000 7fee2659c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee2805ca50 0x7fee280b2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:17.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.557+0000 7fee2659c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee2805ca50 0x7fee280b2ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51742/0 (socket says 172.21.15.60:51742) 2022-06-29T16:06:17.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.557+0000 7fee2659c700 1 -- 172.21.15.60:0/1803987069 learned_addr learned my addr 172.21.15.60:0/1803987069 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:17.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.557+0000 7fee26d9d700 1 --2- 172.21.15.60:0/1803987069 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee280b6a50 0x7fee28066020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:17.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.557+0000 7fee25d9b700 1 --2- 172.21.15.60:0/1803987069 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee2805f0b0 0x7fee28065b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:17.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.557+0000 7fee2659c700 1 -- 172.21.15.60:0/1803987069 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee2805f0b0 msgr2=0x7fee28065b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:17.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.557+0000 7fee2659c700 1 --2- 172.21.15.60:0/1803987069 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee2805f0b0 0x7fee28065b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:17.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.557+0000 7fee2659c700 1 -- 172.21.15.60:0/1803987069 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee280b6a50 msgr2=0x7fee28066020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:17.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.557+0000 7fee2659c700 1 --2- 172.21.15.60:0/1803987069 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee280b6a50 0x7fee28066020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:17.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.557+0000 7fee2659c700 1 -- 172.21.15.60:0/1803987069 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee20009530 con 0x7fee2805ca50 2022-06-29T16:06:17.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.557+0000 7fee2659c700 1 --2- 172.21.15.60:0/1803987069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee2805ca50 0x7fee280b2ed0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 rx=0x7fee1c006940 tx=0x7fee1c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:17.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.557+0000 7fee177fe700 1 -- 172.21.15.60:0/1803987069 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee1c011780 con 0x7fee2805ca50 2022-06-29T16:06:17.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.557+0000 7fee177fe700 1 -- 172.21.15.60:0/1803987069 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee1c0118e0 con 0x7fee2805ca50 2022-06-29T16:06:17.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.557+0000 7fee2c978700 1 -- 172.21.15.60:0/1803987069 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee28144310 con 0x7fee2805ca50 2022-06-29T16:06:17.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.557+0000 7fee177fe700 1 -- 172.21.15.60:0/1803987069 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee1c011c40 con 0x7fee2805ca50 2022-06-29T16:06:17.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.557+0000 7fee2c978700 1 -- 172.21.15.60:0/1803987069 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee2809a340 con 0x7fee2805ca50 2022-06-29T16:06:17.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.561+0000 7fee2c978700 1 -- 172.21.15.60:0/1803987069 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee280573d0 con 0x7fee2805ca50 2022-06-29T16:06:17.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.561+0000 7fee177fe700 1 -- 172.21.15.60:0/1803987069 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fee1c012420 con 0x7fee2805ca50 2022-06-29T16:06:17.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.561+0000 7fee177fe700 1 --2- 172.21.15.60:0/1803987069 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fee04053a70 0x7fee04055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:17.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.561+0000 7fee177fe700 1 -- 172.21.15.60:0/1803987069 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fee1c06bc10 con 0x7fee2805ca50 2022-06-29T16:06:17.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.561+0000 7fee25d9b700 1 --2- 172.21.15.60:0/1803987069 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fee04053a70 0x7fee04055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:17.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.561+0000 7fee25d9b700 1 --2- 172.21.15.60:0/1803987069 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fee04053a70 0x7fee04055f30 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0x7fee1800d000 tx=0x7fee18007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:17.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.569+0000 7fee177fe700 1 -- 172.21.15.60:0/1803987069 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fee1c042a80 con 0x7fee2805ca50 2022-06-29T16:06:17.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.737+0000 7fee2c978700 1 -- 172.21.15.60:0/1803987069 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee280658b0 con 0x7fee2805ca50 2022-06-29T16:06:17.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.737+0000 7fee177fe700 1 -- 172.21.15.60:0/1803987069 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+1780 (secure 0 0 0) 0x7fee1c0460a0 con 0x7fee2805ca50 2022-06-29T16:06:17.744 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:17.745 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":47,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7320,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/3498160620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3498160620},{"type":"v1","addr":"172.21.15.60:6833","nonce":3498160620}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":47,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:06:05.226825+0000","modified":"2022-06-29T16:06:17.415733+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2022-06-29T16:06:17.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.741+0000 7fee2c978700 1 -- 172.21.15.60:0/1803987069 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fee04053a70 msgr2=0x7fee04055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:17.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.741+0000 7fee2c978700 1 --2- 172.21.15.60:0/1803987069 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fee04053a70 0x7fee04055f30 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0x7fee1800d000 tx=0x7fee18007b90).stop 2022-06-29T16:06:17.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.741+0000 7fee2c978700 1 -- 172.21.15.60:0/1803987069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee2805ca50 msgr2=0x7fee280b2ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:17.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.741+0000 7fee2c978700 1 --2- 172.21.15.60:0/1803987069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee2805ca50 0x7fee280b2ed0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 rx=0x7fee1c006940 tx=0x7fee1c008040).stop 2022-06-29T16:06:17.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.741+0000 7fee2c978700 1 -- 172.21.15.60:0/1803987069 shutdown_connections 2022-06-29T16:06:17.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.741+0000 7fee2c978700 1 --2- 172.21.15.60:0/1803987069 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fee04053a70 0x7fee04055f30 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:17.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.741+0000 7fee2c978700 1 --2- 172.21.15.60:0/1803987069 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee280b6a50 0x7fee28066020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:17.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.741+0000 7fee2c978700 1 --2- 172.21.15.60:0/1803987069 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee2805f0b0 0x7fee28065b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:17.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.741+0000 7fee2c978700 1 --2- 172.21.15.60:0/1803987069 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee2805ca50 0x7fee280b2ed0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:17.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.741+0000 7fee2c978700 1 -- 172.21.15.60:0/1803987069 >> 172.21.15.60:0/1803987069 conn(0x7fee280643e0 msgr2=0x7fee28140100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:17.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.741+0000 7fee2c978700 1 -- 172.21.15.60:0/1803987069 shutdown_connections 2022-06-29T16:06:17.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.741+0000 7fee2c978700 1 -- 172.21.15.60:0/1803987069 wait complete. 2022-06-29T16:06:17.750 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 47 2022-06-29T16:06:17.757 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-06-29T16:06:17.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.865+0000 7f54a0cfe700 1 Processor -- start 2022-06-29T16:06:17.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.865+0000 7f54a0cfe700 1 -- start start 2022-06-29T16:06:17.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.869+0000 7f54a0cfe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f549c13f3d0 0x7f549c060260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:17.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.869+0000 7f54a0cfe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f549c143820 0x7f549c0b6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:17.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.869+0000 7f54a0cfe700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f549c0b6c00 0x7f549c13f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:17.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.869+0000 7f54a0cfe700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f549c142ef0 con 0x7f549c143820 2022-06-29T16:06:17.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.869+0000 7f54a0cfe700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f549c0acb90 con 0x7f549c0b6c00 2022-06-29T16:06:17.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.869+0000 7f54a0cfe700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f549c1430e0 con 0x7f549c13f3d0 2022-06-29T16:06:17.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.869+0000 7f5499d9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f549c143820 0x7f549c0b6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:17.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.869+0000 7f5499d9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f549c143820 0x7f549c0b6710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51752/0 (socket says 172.21.15.60:51752) 2022-06-29T16:06:17.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.869+0000 7f5499d9b700 1 -- 172.21.15.60:0/1282047531 learned_addr learned my addr 172.21.15.60:0/1282047531 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:17.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.869+0000 7f549a59c700 1 --2- 172.21.15.60:0/1282047531 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f549c13f3d0 0x7f549c060260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:17.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.869+0000 7f549ad9d700 1 --2- 172.21.15.60:0/1282047531 >> v2:172.21.15.92:3300/0 conn(0x7f549c0b6c00 0x7f549c13f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:17.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.869+0000 7f5499d9b700 1 -- 172.21.15.60:0/1282047531 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f549c13f3d0 msgr2=0x7f549c060260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:17.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.869+0000 7f5499d9b700 1 --2- 172.21.15.60:0/1282047531 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f549c13f3d0 0x7f549c060260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:17.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.869+0000 7f5499d9b700 1 -- 172.21.15.60:0/1282047531 >> v2:172.21.15.92:3300/0 conn(0x7f549c0b6c00 msgr2=0x7f549c13f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:17.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.869+0000 7f5499d9b700 1 --2- 172.21.15.60:0/1282047531 >> v2:172.21.15.92:3300/0 conn(0x7f549c0b6c00 0x7f549c13f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:17.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.869+0000 7f5499d9b700 1 -- 172.21.15.60:0/1282047531 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f549c143380 con 0x7f549c143820 2022-06-29T16:06:17.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.869+0000 7f5499d9b700 1 --2- 172.21.15.60:0/1282047531 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f549c143820 0x7f549c0b6710 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 rx=0x7f5490002f70 tx=0x7f5490006eb0).ready entity=mon.0 client_cookie=d55dc159a70e1bb8 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:17.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.869+0000 7f549959a700 1 -- 172.21.15.60:0/1282047531 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f549000f970 con 0x7f549c143820 2022-06-29T16:06:17.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.869+0000 7f549959a700 1 -- 172.21.15.60:0/1282047531 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5490000e90 con 0x7f549c143820 2022-06-29T16:06:17.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.869+0000 7f549959a700 1 -- 172.21.15.60:0/1282047531 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f549000fd00 con 0x7f549c143820 2022-06-29T16:06:17.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.869+0000 7f54a0cfe700 1 -- 172.21.15.60:0/1282047531 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f549c143820 msgr2=0x7f549c0b6710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:17.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.869+0000 7f54a0cfe700 1 --2- 172.21.15.60:0/1282047531 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f549c143820 0x7f549c0b6710 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 rx=0x7f5490002f70 tx=0x7f5490006eb0).stop 2022-06-29T16:06:17.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.869+0000 7f54a0cfe700 1 -- 172.21.15.60:0/1282047531 shutdown_connections 2022-06-29T16:06:17.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.869+0000 7f54a0cfe700 1 --2- 172.21.15.60:0/1282047531 >> v2:172.21.15.92:3300/0 conn(0x7f549c0b6c00 0x7f549c13f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:17.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.869+0000 7f54a0cfe700 1 --2- 172.21.15.60:0/1282047531 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f549c143820 0x7f549c0b6710 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:17.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.869+0000 7f54a0cfe700 1 --2- 172.21.15.60:0/1282047531 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f549c13f3d0 0x7f549c060260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:17.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.869+0000 7f54a0cfe700 1 -- 172.21.15.60:0/1282047531 >> 172.21.15.60:0/1282047531 conn(0x7f549c0643e0 msgr2=0x7f549c0b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:17.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.869+0000 7f54a0cfe700 1 -- 172.21.15.60:0/1282047531 shutdown_connections 2022-06-29T16:06:17.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.869+0000 7f54a0cfe700 1 -- 172.21.15.60:0/1282047531 wait complete. 2022-06-29T16:06:17.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.873+0000 7f54a0cfe700 1 Processor -- start 2022-06-29T16:06:17.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.873+0000 7f54a0cfe700 1 -- start start 2022-06-29T16:06:17.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.873+0000 7f54a0cfe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f549c0b6c00 0x7f549c05c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:17.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.873+0000 7f54a0cfe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f549c13f3d0 0x7f549c13e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:17.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.873+0000 7f54a0cfe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f549c143820 0x7f549c13e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:17.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.873+0000 7f54a0cfe700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f549c067770 con 0x7f549c0b6c00 2022-06-29T16:06:17.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.873+0000 7f54a0cfe700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f549c066560 con 0x7f549c13f3d0 2022-06-29T16:06:17.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.873+0000 7f54a0cfe700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f549c066800 con 0x7f549c143820 2022-06-29T16:06:17.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.873+0000 7f549a59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f549c0b6c00 0x7f549c05c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:17.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.873+0000 7f549a59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f549c0b6c00 0x7f549c05c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51756/0 (socket says 172.21.15.60:51756) 2022-06-29T16:06:17.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.873+0000 7f549a59c700 1 -- 172.21.15.60:0/3701533096 learned_addr learned my addr 172.21.15.60:0/3701533096 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:17.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.873+0000 7f5499d9b700 1 --2- 172.21.15.60:0/3701533096 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f549c13f3d0 0x7f549c13e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:17.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.873+0000 7f549ad9d700 1 --2- 172.21.15.60:0/3701533096 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f549c143820 0x7f549c13e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:17.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.873+0000 7f549a59c700 1 -- 172.21.15.60:0/3701533096 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f549c143820 msgr2=0x7f549c13e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:17.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.873+0000 7f549a59c700 1 --2- 172.21.15.60:0/3701533096 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f549c143820 0x7f549c13e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:17.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.873+0000 7f549a59c700 1 -- 172.21.15.60:0/3701533096 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f549c13f3d0 msgr2=0x7f549c13e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:17.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.873+0000 7f549a59c700 1 --2- 172.21.15.60:0/3701533096 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f549c13f3d0 0x7f549c13e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:17.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.873+0000 7f549a59c700 1 -- 172.21.15.60:0/3701533096 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f549000b1e0 con 0x7f549c0b6c00 2022-06-29T16:06:17.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.873+0000 7f549a59c700 1 --2- 172.21.15.60:0/3701533096 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f549c0b6c00 0x7f549c05c8f0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 rx=0x7f548c00c000 tx=0x7f548c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:17.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.873+0000 7f548b7fe700 1 -- 172.21.15.60:0/3701533096 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f548c010780 con 0x7f549c0b6c00 2022-06-29T16:06:17.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.873+0000 7f548b7fe700 1 -- 172.21.15.60:0/3701533096 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f548c0108e0 con 0x7f549c0b6c00 2022-06-29T16:06:17.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.873+0000 7f54a0cfe700 1 -- 172.21.15.60:0/3701533096 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f549c0673f0 con 0x7f549c0b6c00 2022-06-29T16:06:17.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.873+0000 7f548b7fe700 1 -- 172.21.15.60:0/3701533096 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f548c010c40 con 0x7f549c0b6c00 2022-06-29T16:06:17.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.873+0000 7f54a0cfe700 1 -- 172.21.15.60:0/3701533096 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f549c066050 con 0x7f549c0b6c00 2022-06-29T16:06:17.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.877+0000 7f54a0cfe700 1 -- 172.21.15.60:0/3701533096 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f549c0669c0 con 0x7f549c0b6c00 2022-06-29T16:06:17.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.877+0000 7f548b7fe700 1 -- 172.21.15.60:0/3701533096 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f548c01c460 con 0x7f549c0b6c00 2022-06-29T16:06:17.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.877+0000 7f548b7fe700 1 --2- 172.21.15.60:0/3701533096 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5478053b00 0x7f5478055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:17.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.877+0000 7f548b7fe700 1 -- 172.21.15.60:0/3701533096 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f548c06c410 con 0x7f549c0b6c00 2022-06-29T16:06:17.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.877+0000 7f5499d9b700 1 --2- 172.21.15.60:0/3701533096 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5478053b00 0x7f5478055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:17.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.877+0000 7f5499d9b700 1 --2- 172.21.15.60:0/3701533096 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5478053b00 0x7f5478055fc0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0x7f549001c000 tx=0x7f5490017000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:17.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:17.885+0000 7f548b7fe700 1 -- 172.21.15.60:0/3701533096 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f548c043170 con 0x7f549c0b6c00 2022-06-29T16:06:18.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.045+0000 7f54a0cfe700 1 -- 172.21.15.60:0/3701533096 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f549c0573d0 con 0x7f549c0b6c00 2022-06-29T16:06:18.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.437+0000 7f548b7fe700 1 -- 172.21.15.60:0/3701533096 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v48) v1 ==== 104+0+0 (secure 0 0 0) 0x7f548c046790 con 0x7f549c0b6c00 2022-06-29T16:06:18.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.441+0000 7f54a0cfe700 1 -- 172.21.15.60:0/3701533096 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5478053b00 msgr2=0x7f5478055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:18.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.441+0000 7f54a0cfe700 1 --2- 172.21.15.60:0/3701533096 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5478053b00 0x7f5478055fc0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0x7f549001c000 tx=0x7f5490017000).stop 2022-06-29T16:06:18.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.441+0000 7f54a0cfe700 1 -- 172.21.15.60:0/3701533096 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f549c0b6c00 msgr2=0x7f549c05c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:18.448 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.441+0000 7f54a0cfe700 1 --2- 172.21.15.60:0/3701533096 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f549c0b6c00 0x7f549c05c8f0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 rx=0x7f548c00c000 tx=0x7f548c006df0).stop 2022-06-29T16:06:18.448 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.441+0000 7f54a0cfe700 1 -- 172.21.15.60:0/3701533096 shutdown_connections 2022-06-29T16:06:18.449 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.441+0000 7f54a0cfe700 1 --2- 172.21.15.60:0/3701533096 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5478053b00 0x7f5478055fc0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:18.449 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.441+0000 7f54a0cfe700 1 --2- 172.21.15.60:0/3701533096 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f549c143820 0x7f549c13e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:18.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.441+0000 7f54a0cfe700 1 --2- 172.21.15.60:0/3701533096 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f549c13f3d0 0x7f549c13e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:18.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.441+0000 7f54a0cfe700 1 --2- 172.21.15.60:0/3701533096 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f549c0b6c00 0x7f549c05c8f0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:18.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.441+0000 7f54a0cfe700 1 -- 172.21.15.60:0/3701533096 >> 172.21.15.60:0/3701533096 conn(0x7f549c0643e0 msgr2=0x7f549c0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:18.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.441+0000 7f54a0cfe700 1 -- 172.21.15.60:0/3701533096 shutdown_connections 2022-06-29T16:06:18.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.441+0000 7f54a0cfe700 1 -- 172.21.15.60:0/3701533096 wait complete. 2022-06-29T16:06:18.461 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-06-29T16:06:18.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.573+0000 7fe0c598a700 1 Processor -- start 2022-06-29T16:06:18.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.573+0000 7fe0c598a700 1 -- start start 2022-06-29T16:06:18.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.573+0000 7fe0c598a700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe0c00ae7c0 0x7fe0c0141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:18.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.573+0000 7fe0c598a700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe0c00b46c0 0x7fe0c00b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:18.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.573+0000 7fe0c598a700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c0143a40 con 0x7fe0c00b4010 2022-06-29T16:06:18.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.573+0000 7fe0c598a700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0c00aee60 con 0x7fe0c00ae7c0 2022-06-29T16:06:18.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.573+0000 7fe0c598a700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0c00af100 con 0x7fe0c00b46c0 2022-06-29T16:06:18.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.577+0000 7fe0beffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe0c00ae7c0 0x7fe0c0141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:18.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.577+0000 7fe0bf7fe700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe0c00b46c0 0x7fe0c00b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:18.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.577+0000 7fe0beffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe0c00ae7c0 0x7fe0c0141fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51766/0 (socket says 172.21.15.60:51766) 2022-06-29T16:06:18.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.577+0000 7fe0beffd700 1 -- 172.21.15.60:0/2607495859 learned_addr learned my addr 172.21.15.60:0/2607495859 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:18.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.577+0000 7fe0beffd700 1 -- 172.21.15.60:0/2607495859 >> v2:172.21.15.92:3300/0 conn(0x7fe0c00b46c0 msgr2=0x7fe0c00b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:18.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.577+0000 7fe0beffd700 1 --2- 172.21.15.60:0/2607495859 >> v2:172.21.15.92:3300/0 conn(0x7fe0c00b46c0 0x7fe0c00b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:18.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.577+0000 7fe0beffd700 1 -- 172.21.15.60:0/2607495859 >> v1:172.21.15.92:6789/0 conn(0x7fe0c00b4010 legacy=0x7fe0c0142490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:18.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.577+0000 7fe0beffd700 1 -- 172.21.15.60:0/2607495859 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c00aeba0 con 0x7fe0c00ae7c0 2022-06-29T16:06:18.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.577+0000 7fe0beffd700 1 --2- 172.21.15.60:0/2607495859 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0c00ae7c0 0x7fe0c0141fa0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 rx=0x7fe0b0002f70 tx=0x7fe0b0006eb0).ready entity=mon.0 client_cookie=8f7b5ff91bc887b2 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:18.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.577+0000 7fe0bdffb700 1 -- 172.21.15.60:0/2607495859 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0b000d5c0 con 0x7fe0c00ae7c0 2022-06-29T16:06:18.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.577+0000 7fe0bdffb700 1 -- 172.21.15.60:0/2607495859 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0b0000e90 con 0x7fe0c00ae7c0 2022-06-29T16:06:18.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.577+0000 7fe0bdffb700 1 -- 172.21.15.60:0/2607495859 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0b000d950 con 0x7fe0c00ae7c0 2022-06-29T16:06:18.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.577+0000 7fe0c598a700 1 -- 172.21.15.60:0/2607495859 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0c00ae7c0 msgr2=0x7fe0c0141fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:18.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.577+0000 7fe0c598a700 1 --2- 172.21.15.60:0/2607495859 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0c00ae7c0 0x7fe0c0141fa0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 rx=0x7fe0b0002f70 tx=0x7fe0b0006eb0).stop 2022-06-29T16:06:18.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.577+0000 7fe0c598a700 1 -- 172.21.15.60:0/2607495859 shutdown_connections 2022-06-29T16:06:18.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.577+0000 7fe0c598a700 1 --2- 172.21.15.60:0/2607495859 >> v2:172.21.15.92:3300/0 conn(0x7fe0c00b46c0 0x7fe0c00b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:18.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.577+0000 7fe0c598a700 1 --2- 172.21.15.60:0/2607495859 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0c00ae7c0 0x7fe0c0141fa0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:18.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.577+0000 7fe0c598a700 1 -- 172.21.15.60:0/2607495859 >> 172.21.15.60:0/2607495859 conn(0x7fe0c00b9340 msgr2=0x7fe0c00b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:18.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.577+0000 7fe0c598a700 1 -- 172.21.15.60:0/2607495859 shutdown_connections 2022-06-29T16:06:18.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.577+0000 7fe0c598a700 1 -- 172.21.15.60:0/2607495859 wait complete. 2022-06-29T16:06:18.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.581+0000 7fe0c598a700 1 Processor -- start 2022-06-29T16:06:18.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.581+0000 7fe0c598a700 1 -- start start 2022-06-29T16:06:18.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.581+0000 7fe0c598a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0c00ae7c0 0x7fe0c00b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:18.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.581+0000 7fe0c598a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0c00b4010 0x7fe0c00b05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:18.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.581+0000 7fe0c598a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0c00b46c0 0x7fe0c00b0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:18.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.581+0000 7fe0c598a700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c013e7c0 con 0x7fe0c00b46c0 2022-06-29T16:06:18.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.581+0000 7fe0c598a700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c01144b0 con 0x7fe0c00ae7c0 2022-06-29T16:06:18.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.581+0000 7fe0c598a700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c0114750 con 0x7fe0c00b4010 2022-06-29T16:06:18.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.581+0000 7fe0bf7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0c00b46c0 0x7fe0c00b0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:18.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.581+0000 7fe0bf7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0c00b46c0 0x7fe0c00b0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51772/0 (socket says 172.21.15.60:51772) 2022-06-29T16:06:18.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.581+0000 7fe0bf7fe700 1 -- 172.21.15.60:0/2223566304 learned_addr learned my addr 172.21.15.60:0/2223566304 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:18.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.581+0000 7fe0beffd700 1 --2- 172.21.15.60:0/2223566304 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0c00ae7c0 0x7fe0c00b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:18.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.581+0000 7fe0be7fc700 1 --2- 172.21.15.60:0/2223566304 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0c00b4010 0x7fe0c00b05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:18.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.581+0000 7fe0bf7fe700 1 -- 172.21.15.60:0/2223566304 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0c00b4010 msgr2=0x7fe0c00b05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:18.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.581+0000 7fe0bf7fe700 1 --2- 172.21.15.60:0/2223566304 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0c00b4010 0x7fe0c00b05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:18.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.581+0000 7fe0bf7fe700 1 -- 172.21.15.60:0/2223566304 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0c00ae7c0 msgr2=0x7fe0c00b1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:18.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.581+0000 7fe0bf7fe700 1 --2- 172.21.15.60:0/2223566304 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0c00ae7c0 0x7fe0c00b1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:18.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.581+0000 7fe0bf7fe700 1 -- 172.21.15.60:0/2223566304 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0b000b1e0 con 0x7fe0c00b46c0 2022-06-29T16:06:18.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.581+0000 7fe0bf7fe700 1 --2- 172.21.15.60:0/2223566304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0c00b46c0 0x7fe0c00b0ab0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 rx=0x7fe0b8006e00 tx=0x7fe0b80091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:18.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.581+0000 7fe0affff700 1 -- 172.21.15.60:0/2223566304 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0b8012600 con 0x7fe0c00b46c0 2022-06-29T16:06:18.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.581+0000 7fe0c598a700 1 -- 172.21.15.60:0/2223566304 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0c01149f0 con 0x7fe0c00b46c0 2022-06-29T16:06:18.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.581+0000 7fe0affff700 1 -- 172.21.15.60:0/2223566304 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0b8012760 con 0x7fe0c00b46c0 2022-06-29T16:06:18.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.581+0000 7fe0affff700 1 -- 172.21.15.60:0/2223566304 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0b8012ac0 con 0x7fe0c00b46c0 2022-06-29T16:06:18.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.581+0000 7fe0c598a700 1 -- 172.21.15.60:0/2223566304 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0c00b1020 con 0x7fe0c00b46c0 2022-06-29T16:06:18.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.585+0000 7fe0c598a700 1 -- 172.21.15.60:0/2223566304 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0c0114b80 con 0x7fe0c00b46c0 2022-06-29T16:06:18.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.585+0000 7fe0affff700 1 -- 172.21.15.60:0/2223566304 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe0b8022070 con 0x7fe0c00b46c0 2022-06-29T16:06:18.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.585+0000 7fe0affff700 1 --2- 172.21.15.60:0/2223566304 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe09c053a70 0x7fe09c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:18.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.585+0000 7fe0affff700 1 -- 172.21.15.60:0/2223566304 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fe0b806c570 con 0x7fe0c00b46c0 2022-06-29T16:06:18.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.585+0000 7fe0beffd700 1 --2- 172.21.15.60:0/2223566304 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe09c053a70 0x7fe09c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:18.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.585+0000 7fe0beffd700 1 --2- 172.21.15.60:0/2223566304 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe09c053a70 0x7fe09c055f30 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x7fe0b001c000 tx=0x7fe0b0015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:18.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.589+0000 7fe0affff700 1 -- 172.21.15.60:0/2223566304 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe0b8043450 con 0x7fe0c00b46c0 2022-06-29T16:06:18.721 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:18.717+0000 7fe0c598a700 1 -- 172.21.15.60:0/2223566304 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe0c00573d0 con 0x7fe0c00b46c0 2022-06-29T16:06:19.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.449+0000 7fe0affff700 1 -- 172.21.15.60:0/2223566304 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v102) v1 ==== 185+0+0 (secure 0 0 0) 0x7fe0b80160e0 con 0x7fe0c00b46c0 2022-06-29T16:06:19.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.449+0000 7fe0c598a700 1 -- 172.21.15.60:0/2223566304 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe09c053a70 msgr2=0x7fe09c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:19.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.449+0000 7fe0c598a700 1 --2- 172.21.15.60:0/2223566304 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe09c053a70 0x7fe09c055f30 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x7fe0b001c000 tx=0x7fe0b0015000).stop 2022-06-29T16:06:19.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.449+0000 7fe0c598a700 1 -- 172.21.15.60:0/2223566304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0c00b46c0 msgr2=0x7fe0c00b0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:19.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.449+0000 7fe0c598a700 1 --2- 172.21.15.60:0/2223566304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0c00b46c0 0x7fe0c00b0ab0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 rx=0x7fe0b8006e00 tx=0x7fe0b80091d0).stop 2022-06-29T16:06:19.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.453+0000 7fe0c598a700 1 -- 172.21.15.60:0/2223566304 shutdown_connections 2022-06-29T16:06:19.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.453+0000 7fe0c598a700 1 --2- 172.21.15.60:0/2223566304 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe09c053a70 0x7fe09c055f30 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:19.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.453+0000 7fe0c598a700 1 --2- 172.21.15.60:0/2223566304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0c00b46c0 0x7fe0c00b0ab0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:19.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.453+0000 7fe0c598a700 1 --2- 172.21.15.60:0/2223566304 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0c00b4010 0x7fe0c00b05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:19.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.453+0000 7fe0c598a700 1 --2- 172.21.15.60:0/2223566304 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0c00ae7c0 0x7fe0c00b1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:19.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.453+0000 7fe0c598a700 1 -- 172.21.15.60:0/2223566304 >> 172.21.15.60:0/2223566304 conn(0x7fe0c00b9340 msgr2=0x7fe0c013f060 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:19.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.453+0000 7fe0c598a700 1 -- 172.21.15.60:0/2223566304 shutdown_connections 2022-06-29T16:06:19.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.453+0000 7fe0c598a700 1 -- 172.21.15.60:0/2223566304 wait complete. 2022-06-29T16:06:19.460 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2022-06-29T16:06:19.472 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-06-29T16:06:19.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.581+0000 7f54ecbe6700 1 Processor -- start 2022-06-29T16:06:19.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.581+0000 7f54ecbe6700 1 -- start start 2022-06-29T16:06:19.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.581+0000 7f54ecbe6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54e80b28e0 0x7f54e805bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:19.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.581+0000 7f54ecbe6700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f54e80afee0 0x7f54e8141e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:19.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.581+0000 7f54ecbe6700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e80b1d70 con 0x7f54e80b9be0 2022-06-29T16:06:19.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.581+0000 7f54ecbe6700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54e80b9fc0 con 0x7f54e80afee0 2022-06-29T16:06:19.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.581+0000 7f54ecbe6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54e80b3c40 con 0x7f54e80b28e0 2022-06-29T16:06:19.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.581+0000 7f54e7fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f54e80afee0 0x7f54e8141e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:19.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.581+0000 7f54deffd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54e80b28e0 0x7f54e805bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:19.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.581+0000 7f54e7fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f54e80afee0 0x7f54e8141e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51782/0 (socket says 172.21.15.60:51782) 2022-06-29T16:06:19.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.581+0000 7f54deffd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54e80b28e0 0x7f54e805bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:36642/0 (socket says 172.21.15.60:36642) 2022-06-29T16:06:19.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.581+0000 7f54e7fff700 1 -- 172.21.15.60:0/2081488528 learned_addr learned my addr 172.21.15.60:0/2081488528 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:19.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.581+0000 7f54e77fe700 1 --1- 172.21.15.60:0/2081488528 >> v1:172.21.15.92:6789/0 conn(0x7f54e80b9be0 0x7f54e80b21d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:35880/0 (socket says 172.21.15.60:35880) 2022-06-29T16:06:19.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.581+0000 7f54deffd700 1 -- 172.21.15.60:0/2081488528 >> v2:172.21.15.60:3300/0 conn(0x7f54e80afee0 msgr2=0x7f54e8141e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:19.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.581+0000 7f54deffd700 1 --2- 172.21.15.60:0/2081488528 >> v2:172.21.15.60:3300/0 conn(0x7f54e80afee0 0x7f54e8141e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:19.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.581+0000 7f54e6ffd700 1 -- 172.21.15.60:0/2081488528 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1920661826 0 0) 0x7f54e80b1d70 con 0x7f54e80b9be0 2022-06-29T16:06:19.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.581+0000 7f54deffd700 1 -- 172.21.15.60:0/2081488528 >> v1:172.21.15.92:6789/0 conn(0x7f54e80b9be0 legacy=0x7f54e80b21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:19.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.581+0000 7f54deffd700 1 -- 172.21.15.60:0/2081488528 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e80b3ee0 con 0x7f54e80b28e0 2022-06-29T16:06:19.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.581+0000 7f54deffd700 1 --2- 172.21.15.60:0/2081488528 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54e80b28e0 0x7f54e805bd20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f54d0002f70 tx=0x7f54d0006df0).ready entity=mon.2 client_cookie=30d53cbfbbba433e server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:19.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.581+0000 7f54e6ffd700 1 -- 172.21.15.60:0/2081488528 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54d000d570 con 0x7f54e80b28e0 2022-06-29T16:06:19.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.585+0000 7f54e6ffd700 1 -- 172.21.15.60:0/2081488528 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d0000e90 con 0x7f54e80b28e0 2022-06-29T16:06:19.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.585+0000 7f54e6ffd700 1 -- 172.21.15.60:0/2081488528 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54d000d8c0 con 0x7f54e80b28e0 2022-06-29T16:06:19.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.585+0000 7f54ecbe6700 1 -- 172.21.15.60:0/2081488528 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54e80b28e0 msgr2=0x7f54e805bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:19.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.585+0000 7f54ecbe6700 1 --2- 172.21.15.60:0/2081488528 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54e80b28e0 0x7f54e805bd20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f54d0002f70 tx=0x7f54d0006df0).stop 2022-06-29T16:06:19.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.585+0000 7f54ecbe6700 1 -- 172.21.15.60:0/2081488528 shutdown_connections 2022-06-29T16:06:19.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.585+0000 7f54ecbe6700 1 --2- 172.21.15.60:0/2081488528 >> v2:172.21.15.60:3300/0 conn(0x7f54e80afee0 0x7f54e8141e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:19.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.585+0000 7f54ecbe6700 1 --2- 172.21.15.60:0/2081488528 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54e80b28e0 0x7f54e805bd20 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:19.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.585+0000 7f54ecbe6700 1 -- 172.21.15.60:0/2081488528 >> 172.21.15.60:0/2081488528 conn(0x7f54e8062ce0 msgr2=0x7f54e80b9190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:19.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.585+0000 7f54ecbe6700 1 -- 172.21.15.60:0/2081488528 shutdown_connections 2022-06-29T16:06:19.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.585+0000 7f54ecbe6700 1 -- 172.21.15.60:0/2081488528 wait complete. 2022-06-29T16:06:19.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.585+0000 7f54ecbe6700 1 Processor -- start 2022-06-29T16:06:19.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.585+0000 7f54ecbe6700 1 -- start start 2022-06-29T16:06:19.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.585+0000 7f54ecbe6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54e80afee0 0x7f54e8154d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:19.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.585+0000 7f54ecbe6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54e80b28e0 0x7f54e80b8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:19.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.589+0000 7f54ecbe6700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54e80b9be0 0x7f54e8144580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:19.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.589+0000 7f54ecbe6700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e813dbc0 con 0x7f54e80afee0 2022-06-29T16:06:19.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.589+0000 7f54ecbe6700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e80adbb0 con 0x7f54e80b9be0 2022-06-29T16:06:19.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.589+0000 7f54ecbe6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54e80ae5e0 con 0x7f54e80b28e0 2022-06-29T16:06:19.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.589+0000 7f54e77fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54e80afee0 0x7f54e8154d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:19.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.589+0000 7f54e77fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54e80afee0 0x7f54e8154d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51784/0 (socket says 172.21.15.60:51784) 2022-06-29T16:06:19.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.589+0000 7f54e77fe700 1 -- 172.21.15.60:0/3244043798 learned_addr learned my addr 172.21.15.60:0/3244043798 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:19.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.589+0000 7f54deffd700 1 --2- 172.21.15.60:0/3244043798 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54e80b28e0 0x7f54e80b8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:19.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.589+0000 7f54e7fff700 1 --2- 172.21.15.60:0/3244043798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54e80b9be0 0x7f54e8144580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:19.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.589+0000 7f54e77fe700 1 -- 172.21.15.60:0/3244043798 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54e80b28e0 msgr2=0x7f54e80b8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:19.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.589+0000 7f54e77fe700 1 --2- 172.21.15.60:0/3244043798 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54e80b28e0 0x7f54e80b8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:19.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.589+0000 7f54e77fe700 1 -- 172.21.15.60:0/3244043798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54e80b9be0 msgr2=0x7f54e8144580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:19.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.589+0000 7f54e77fe700 1 --2- 172.21.15.60:0/3244043798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54e80b9be0 0x7f54e8144580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:19.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.589+0000 7f54e77fe700 1 -- 172.21.15.60:0/3244043798 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d000b180 con 0x7f54e80afee0 2022-06-29T16:06:19.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.589+0000 7f54e77fe700 1 --2- 172.21.15.60:0/3244043798 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54e80afee0 0x7f54e8154d90 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 rx=0x7f54d8000bc0 tx=0x7f54d8007c80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:19.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.589+0000 7f54e57fa700 1 -- 172.21.15.60:0/3244043798 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54d800cdf0 con 0x7f54e80afee0 2022-06-29T16:06:19.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.589+0000 7f54e57fa700 1 -- 172.21.15.60:0/3244043798 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d80070d0 con 0x7f54e80afee0 2022-06-29T16:06:19.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.589+0000 7f54ecbe6700 1 -- 172.21.15.60:0/3244043798 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54e80ae880 con 0x7f54e80afee0 2022-06-29T16:06:19.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.589+0000 7f54e57fa700 1 -- 172.21.15.60:0/3244043798 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54d8007430 con 0x7f54e80afee0 2022-06-29T16:06:19.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.589+0000 7f54ecbe6700 1 -- 172.21.15.60:0/3244043798 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54e80b91a0 con 0x7f54e80afee0 2022-06-29T16:06:19.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.589+0000 7f54ecbe6700 1 -- 172.21.15.60:0/3244043798 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54e80573d0 con 0x7f54e80afee0 2022-06-29T16:06:19.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.593+0000 7f54e57fa700 1 -- 172.21.15.60:0/3244043798 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f54d801a060 con 0x7f54e80afee0 2022-06-29T16:06:19.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.593+0000 7f54e57fa700 1 --2- 172.21.15.60:0/3244043798 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f54cc053b00 0x7f54cc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:19.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.593+0000 7f54e57fa700 1 -- 172.21.15.60:0/3244043798 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f54d806b9f0 con 0x7f54e80afee0 2022-06-29T16:06:19.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.593+0000 7f54deffd700 1 --2- 172.21.15.60:0/3244043798 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f54cc053b00 0x7f54cc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:19.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.593+0000 7f54deffd700 1 --2- 172.21.15.60:0/3244043798 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f54cc053b00 0x7f54cc055fc0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7f54d001a000 tx=0x7f54d0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:19.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.597+0000 7f54e57fa700 1 -- 172.21.15.60:0/3244043798 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f54d8042a70 con 0x7f54e80afee0 2022-06-29T16:06:19.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:19.737+0000 7f54ecbe6700 1 -- 172.21.15.60:0/3244043798 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f54e8146550 con 0x7f54e80afee0 2022-06-29T16:06:20.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.461+0000 7f54e57fa700 1 -- 172.21.15.60:0/3244043798 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v103) v1 ==== 173+0+0 (secure 0 0 0) 0x7f54d8018030 con 0x7f54e80afee0 2022-06-29T16:06:20.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.465+0000 7f54ecbe6700 1 -- 172.21.15.60:0/3244043798 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f54cc053b00 msgr2=0x7f54cc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:20.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.465+0000 7f54ecbe6700 1 --2- 172.21.15.60:0/3244043798 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f54cc053b00 0x7f54cc055fc0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7f54d001a000 tx=0x7f54d0014000).stop 2022-06-29T16:06:20.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.465+0000 7f54ecbe6700 1 -- 172.21.15.60:0/3244043798 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54e80afee0 msgr2=0x7f54e8154d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:20.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.465+0000 7f54ecbe6700 1 --2- 172.21.15.60:0/3244043798 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54e80afee0 0x7f54e8154d90 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 rx=0x7f54d8000bc0 tx=0x7f54d8007c80).stop 2022-06-29T16:06:20.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.465+0000 7f54ecbe6700 1 -- 172.21.15.60:0/3244043798 shutdown_connections 2022-06-29T16:06:20.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.465+0000 7f54ecbe6700 1 --2- 172.21.15.60:0/3244043798 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f54cc053b00 0x7f54cc055fc0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:20.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.465+0000 7f54ecbe6700 1 --2- 172.21.15.60:0/3244043798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54e80b9be0 0x7f54e8144580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:20.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.465+0000 7f54ecbe6700 1 --2- 172.21.15.60:0/3244043798 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54e80b28e0 0x7f54e80b8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:20.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.465+0000 7f54ecbe6700 1 --2- 172.21.15.60:0/3244043798 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54e80afee0 0x7f54e8154d90 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:20.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.465+0000 7f54ecbe6700 1 -- 172.21.15.60:0/3244043798 >> 172.21.15.60:0/3244043798 conn(0x7f54e8062ce0 msgr2=0x7f54e8141c40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:20.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.465+0000 7f54ecbe6700 1 -- 172.21.15.60:0/3244043798 shutdown_connections 2022-06-29T16:06:20.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.465+0000 7f54ecbe6700 1 -- 172.21.15.60:0/3244043798 wait complete. 2022-06-29T16:06:20.475 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2022-06-29T16:06:20.487 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-06-29T16:06:20.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.597+0000 7fe4adb24700 1 Processor -- start 2022-06-29T16:06:20.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.597+0000 7fe4adb24700 1 -- start start 2022-06-29T16:06:20.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.597+0000 7fe4adb24700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe4a805d070 0x7fe4a812b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:20.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.597+0000 7fe4adb24700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe4a8066b20 0x7fe4a812bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:20.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.597+0000 7fe4adb24700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a80ae9a0 con 0x7fe4a8142ae0 2022-06-29T16:06:20.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.597+0000 7fe4adb24700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4a8143b60 con 0x7fe4a805d070 2022-06-29T16:06:20.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.597+0000 7fe4adb24700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4a812c0b0 con 0x7fe4a8066b20 2022-06-29T16:06:20.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.597+0000 7fe4acb22700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe4a805d070 0x7fe4a812b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:20.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.597+0000 7fe4ad323700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe4a8142ae0 0x7fe4a8142f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59882/0 (socket says 172.21.15.60:59882) 2022-06-29T16:06:20.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.597+0000 7fe4ad323700 1 -- 172.21.15.60:0/2725102312 learned_addr learned my addr 172.21.15.60:0/2725102312 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:20.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.597+0000 7fe4a7fff700 1 --2- 172.21.15.60:0/2725102312 >> v2:172.21.15.92:3300/0 conn(0x7fe4a8066b20 0x7fe4a812bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:20.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.597+0000 7fe4acb22700 1 -- 172.21.15.60:0/2725102312 >> v2:172.21.15.92:3300/0 conn(0x7fe4a8066b20 msgr2=0x7fe4a812bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:20.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.597+0000 7fe4acb22700 1 --2- 172.21.15.60:0/2725102312 >> v2:172.21.15.92:3300/0 conn(0x7fe4a8066b20 0x7fe4a812bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:20.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.597+0000 7fe4acb22700 1 -- 172.21.15.60:0/2725102312 >> v1:172.21.15.60:6789/0 conn(0x7fe4a8142ae0 legacy=0x7fe4a8142f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:20.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.597+0000 7fe4a77fe700 1 -- 172.21.15.60:0/2725102312 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 237312054 0 0) 0x7fe4a80ae9a0 con 0x7fe4a8142ae0 2022-06-29T16:06:20.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.597+0000 7fe4acb22700 1 -- 172.21.15.60:0/2725102312 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a805d450 con 0x7fe4a805d070 2022-06-29T16:06:20.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.597+0000 7fe4acb22700 1 --2- 172.21.15.60:0/2725102312 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4a805d070 0x7fe4a812b6d0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 rx=0x7fe4a00032f0 tx=0x7fe4a00091d0).ready entity=mon.0 client_cookie=6840dfa6d5348ccf server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:20.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.597+0000 7fe4a77fe700 1 -- 172.21.15.60:0/2725102312 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4a0009ea0 con 0x7fe4a805d070 2022-06-29T16:06:20.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.597+0000 7fe4a77fe700 1 -- 172.21.15.60:0/2725102312 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a000b870 con 0x7fe4a805d070 2022-06-29T16:06:20.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.601+0000 7fe4a77fe700 1 -- 172.21.15.60:0/2725102312 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4a000b9e0 con 0x7fe4a805d070 2022-06-29T16:06:20.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.601+0000 7fe4adb24700 1 -- 172.21.15.60:0/2725102312 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4a805d070 msgr2=0x7fe4a812b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:20.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.601+0000 7fe4adb24700 1 --2- 172.21.15.60:0/2725102312 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4a805d070 0x7fe4a812b6d0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 rx=0x7fe4a00032f0 tx=0x7fe4a00091d0).stop 2022-06-29T16:06:20.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.601+0000 7fe4adb24700 1 -- 172.21.15.60:0/2725102312 shutdown_connections 2022-06-29T16:06:20.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.601+0000 7fe4adb24700 1 --2- 172.21.15.60:0/2725102312 >> v2:172.21.15.92:3300/0 conn(0x7fe4a8066b20 0x7fe4a812bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:20.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.601+0000 7fe4adb24700 1 --2- 172.21.15.60:0/2725102312 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4a805d070 0x7fe4a812b6d0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:20.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.601+0000 7fe4adb24700 1 -- 172.21.15.60:0/2725102312 >> 172.21.15.60:0/2725102312 conn(0x7fe4a80b5890 msgr2=0x7fe4a80b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:20.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.601+0000 7fe4adb24700 1 -- 172.21.15.60:0/2725102312 shutdown_connections 2022-06-29T16:06:20.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.601+0000 7fe4adb24700 1 -- 172.21.15.60:0/2725102312 wait complete. 2022-06-29T16:06:20.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.601+0000 7fe4adb24700 1 Processor -- start 2022-06-29T16:06:20.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.601+0000 7fe4adb24700 1 -- start start 2022-06-29T16:06:20.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.605+0000 7fe4adb24700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe4a805d070 0x7fe4a80afe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:20.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.605+0000 7fe4adb24700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe4a8066b20 0x7fe4a81440f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:20.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.605+0000 7fe4adb24700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4a8142ae0 0x7fe4a8127f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:20.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.605+0000 7fe4adb24700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a8127b00 con 0x7fe4a8142ae0 2022-06-29T16:06:20.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.605+0000 7fe4adb24700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a80af720 con 0x7fe4a8066b20 2022-06-29T16:06:20.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.605+0000 7fe4adb24700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a80af9c0 con 0x7fe4a805d070 2022-06-29T16:06:20.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.605+0000 7fe4ad323700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4a8142ae0 0x7fe4a8127f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:20.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.605+0000 7fe4ad323700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4a8142ae0 0x7fe4a8127f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51802/0 (socket says 172.21.15.60:51802) 2022-06-29T16:06:20.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.605+0000 7fe4ad323700 1 -- 172.21.15.60:0/4068870455 learned_addr learned my addr 172.21.15.60:0/4068870455 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:20.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.605+0000 7fe4acb22700 1 --2- 172.21.15.60:0/4068870455 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe4a805d070 0x7fe4a80afe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:20.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.605+0000 7fe4a7fff700 1 --2- 172.21.15.60:0/4068870455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe4a8066b20 0x7fe4a81440f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:20.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.605+0000 7fe4ad323700 1 -- 172.21.15.60:0/4068870455 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe4a805d070 msgr2=0x7fe4a80afe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:20.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.605+0000 7fe4ad323700 1 --2- 172.21.15.60:0/4068870455 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe4a805d070 0x7fe4a80afe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:20.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.605+0000 7fe4ad323700 1 -- 172.21.15.60:0/4068870455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe4a8066b20 msgr2=0x7fe4a81440f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:20.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.605+0000 7fe4ad323700 1 --2- 172.21.15.60:0/4068870455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe4a8066b20 0x7fe4a81440f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:20.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.605+0000 7fe4ad323700 1 -- 172.21.15.60:0/4068870455 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a0009470 con 0x7fe4a8142ae0 2022-06-29T16:06:20.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.605+0000 7fe4ad323700 1 --2- 172.21.15.60:0/4068870455 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4a8142ae0 0x7fe4a8127f10 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 rx=0x7fe498000bc0 tx=0x7fe498007c20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:20.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.605+0000 7fe4a5ffb700 1 -- 172.21.15.60:0/4068870455 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe498001d30 con 0x7fe4a8142ae0 2022-06-29T16:06:20.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.605+0000 7fe4a5ffb700 1 -- 172.21.15.60:0/4068870455 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4980072e0 con 0x7fe4a8142ae0 2022-06-29T16:06:20.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.605+0000 7fe4adb24700 1 -- 172.21.15.60:0/4068870455 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4a8127830 con 0x7fe4a8142ae0 2022-06-29T16:06:20.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.605+0000 7fe4a5ffb700 1 -- 172.21.15.60:0/4068870455 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe498005800 con 0x7fe4a8142ae0 2022-06-29T16:06:20.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.605+0000 7fe4adb24700 1 -- 172.21.15.60:0/4068870455 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4a8126ce0 con 0x7fe4a8142ae0 2022-06-29T16:06:20.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.605+0000 7fe4adb24700 1 -- 172.21.15.60:0/4068870455 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4a80573d0 con 0x7fe4a8142ae0 2022-06-29T16:06:20.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.605+0000 7fe4a5ffb700 1 -- 172.21.15.60:0/4068870455 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe49801a060 con 0x7fe4a8142ae0 2022-06-29T16:06:20.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.605+0000 7fe4a5ffb700 1 --2- 172.21.15.60:0/4068870455 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe48c0539e0 0x7fe48c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:20.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.609+0000 7fe4a5ffb700 1 -- 172.21.15.60:0/4068870455 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fe49806b350 con 0x7fe4a8142ae0 2022-06-29T16:06:20.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.609+0000 7fe4acb22700 1 --2- 172.21.15.60:0/4068870455 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe48c0539e0 0x7fe48c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:20.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.609+0000 7fe4acb22700 1 --2- 172.21.15.60:0/4068870455 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe48c0539e0 0x7fe48c055ea0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7fe4a0019000 tx=0x7fe4a0016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:20.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.613+0000 7fe4a5ffb700 1 -- 172.21.15.60:0/4068870455 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe4980424f0 con 0x7fe4a8142ae0 2022-06-29T16:06:20.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.741+0000 7fe4adb24700 1 -- 172.21.15.60:0/4068870455 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fe4a8126ce0 con 0x7fe4a8142ae0 2022-06-29T16:06:20.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.749+0000 7fe4a5ffb700 1 -- 172.21.15.60:0/4068870455 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v17) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe4a8126ce0 con 0x7fe4a8142ae0 2022-06-29T16:06:20.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.749+0000 7fe4adb24700 1 -- 172.21.15.60:0/4068870455 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe48c0539e0 msgr2=0x7fe48c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:20.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.749+0000 7fe4adb24700 1 --2- 172.21.15.60:0/4068870455 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe48c0539e0 0x7fe48c055ea0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7fe4a0019000 tx=0x7fe4a0016000).stop 2022-06-29T16:06:20.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.749+0000 7fe4adb24700 1 -- 172.21.15.60:0/4068870455 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4a8142ae0 msgr2=0x7fe4a8127f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:20.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.749+0000 7fe4adb24700 1 --2- 172.21.15.60:0/4068870455 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4a8142ae0 0x7fe4a8127f10 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 rx=0x7fe498000bc0 tx=0x7fe498007c20).stop 2022-06-29T16:06:20.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.749+0000 7fe4adb24700 1 -- 172.21.15.60:0/4068870455 shutdown_connections 2022-06-29T16:06:20.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.749+0000 7fe4adb24700 1 --2- 172.21.15.60:0/4068870455 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe48c0539e0 0x7fe48c055ea0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:20.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.749+0000 7fe4adb24700 1 --2- 172.21.15.60:0/4068870455 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe4a8142ae0 0x7fe4a8127f10 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:20.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.749+0000 7fe4adb24700 1 --2- 172.21.15.60:0/4068870455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe4a8066b20 0x7fe4a81440f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:20.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.749+0000 7fe4adb24700 1 --2- 172.21.15.60:0/4068870455 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe4a805d070 0x7fe4a80afe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:20.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.749+0000 7fe4adb24700 1 -- 172.21.15.60:0/4068870455 >> 172.21.15.60:0/4068870455 conn(0x7fe4a80b5890 msgr2=0x7fe4a8146720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:20.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.749+0000 7fe4adb24700 1 -- 172.21.15.60:0/4068870455 shutdown_connections 2022-06-29T16:06:20.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.749+0000 7fe4adb24700 1 -- 172.21.15.60:0/4068870455 wait complete. 2022-06-29T16:06:20.769 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_filtered_df' 2022-06-29T16:06:20.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.869+0000 7efd57159700 1 Processor -- start 2022-06-29T16:06:20.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.869+0000 7efd57159700 1 -- start start 2022-06-29T16:06:20.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.869+0000 7efd57159700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7efd50061870 0x7efd50146240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:20.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.869+0000 7efd57159700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efd5005c2f0 0x7efd50062f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:20.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.869+0000 7efd57159700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd50142bd0 con 0x7efd5005cf60 2022-06-29T16:06:20.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.869+0000 7efd57159700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd50062050 con 0x7efd5005c2f0 2022-06-29T16:06:20.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.869+0000 7efd57159700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd500622f0 con 0x7efd50061870 2022-06-29T16:06:20.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.869+0000 7efd4ffff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efd5005c2f0 0x7efd50062f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:20.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.869+0000 7efd556f6700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7efd5005cf60 0x7efd50066240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59896/0 (socket says 172.21.15.60:59896) 2022-06-29T16:06:20.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.869+0000 7efd556f6700 1 -- 172.21.15.60:0/2128933454 learned_addr learned my addr 172.21.15.60:0/2128933454 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:20.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.873+0000 7efd54ef5700 1 --2- 172.21.15.60:0/2128933454 >> v2:172.21.15.92:3300/0 conn(0x7efd50061870 0x7efd50146240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:20.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.873+0000 7efd4f7fe700 1 -- 172.21.15.60:0/2128933454 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1230678550 0 0) 0x7efd50142bd0 con 0x7efd5005cf60 2022-06-29T16:06:20.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.873+0000 7efd4f7fe700 1 -- 172.21.15.60:0/2128933454 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd3c003260 con 0x7efd5005cf60 2022-06-29T16:06:20.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.873+0000 7efd4ffff700 1 -- 172.21.15.60:0/2128933454 >> v2:172.21.15.92:3300/0 conn(0x7efd50061870 msgr2=0x7efd50146240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:20.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.873+0000 7efd4f7fe700 1 -- 172.21.15.60:0/2128933454 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2805520696 0 0) 0x7efd3c003260 con 0x7efd5005cf60 2022-06-29T16:06:20.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.873+0000 7efd4ffff700 1 --2- 172.21.15.60:0/2128933454 >> v2:172.21.15.92:3300/0 conn(0x7efd50061870 0x7efd50146240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:20.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.873+0000 7efd4ffff700 1 -- 172.21.15.60:0/2128933454 >> v1:172.21.15.60:6789/0 conn(0x7efd5005cf60 legacy=0x7efd50066240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:20.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.873+0000 7efd4ffff700 1 -- 172.21.15.60:0/2128933454 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd50061c50 con 0x7efd5005c2f0 2022-06-29T16:06:20.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.873+0000 7efd4ffff700 1 --2- 172.21.15.60:0/2128933454 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd5005c2f0 0x7efd50062f10 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 rx=0x7efd48001b80 tx=0x7efd40006df0).ready entity=mon.0 client_cookie=29b789e27a9d7110 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:20.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.873+0000 7efd4f7fe700 1 -- 172.21.15.60:0/2128933454 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd480019e0 con 0x7efd5005c2f0 2022-06-29T16:06:20.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.873+0000 7efd4f7fe700 1 -- 172.21.15.60:0/2128933454 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd40000e90 con 0x7efd5005c2f0 2022-06-29T16:06:20.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.873+0000 7efd4f7fe700 1 -- 172.21.15.60:0/2128933454 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd40012460 con 0x7efd5005c2f0 2022-06-29T16:06:20.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.873+0000 7efd57159700 1 -- 172.21.15.60:0/2128933454 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd5005c2f0 msgr2=0x7efd50062f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:20.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.873+0000 7efd57159700 1 --2- 172.21.15.60:0/2128933454 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd5005c2f0 0x7efd50062f10 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 rx=0x7efd48001b80 tx=0x7efd40006df0).stop 2022-06-29T16:06:20.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.873+0000 7efd57159700 1 -- 172.21.15.60:0/2128933454 shutdown_connections 2022-06-29T16:06:20.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.873+0000 7efd57159700 1 --2- 172.21.15.60:0/2128933454 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd5005c2f0 0x7efd50062f10 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:20.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.873+0000 7efd57159700 1 --2- 172.21.15.60:0/2128933454 >> v2:172.21.15.92:3300/0 conn(0x7efd50061870 0x7efd50146240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:20.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.873+0000 7efd57159700 1 -- 172.21.15.60:0/2128933454 >> 172.21.15.60:0/2128933454 conn(0x7efd50064a10 msgr2=0x7efd50064df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:20.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.873+0000 7efd57159700 1 -- 172.21.15.60:0/2128933454 shutdown_connections 2022-06-29T16:06:20.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.873+0000 7efd57159700 1 -- 172.21.15.60:0/2128933454 wait complete. 2022-06-29T16:06:20.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.873+0000 7efd57159700 1 Processor -- start 2022-06-29T16:06:20.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.873+0000 7efd57159700 1 -- start start 2022-06-29T16:06:20.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.877+0000 7efd57159700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd5005c2f0 0x7efd500b72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:20.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.877+0000 7efd57159700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd5005cf60 0x7efd500af070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:20.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.877+0000 7efd57159700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd50061870 0x7efd500b6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:20.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.877+0000 7efd57159700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd500b7850 con 0x7efd5005c2f0 2022-06-29T16:06:20.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.877+0000 7efd57159700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd500b7af0 con 0x7efd50061870 2022-06-29T16:06:20.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.877+0000 7efd57159700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd500b7d90 con 0x7efd5005cf60 2022-06-29T16:06:20.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.877+0000 7efd54ef5700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd5005c2f0 0x7efd500b72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:20.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.877+0000 7efd54ef5700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd5005c2f0 0x7efd500b72b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51812/0 (socket says 172.21.15.60:51812) 2022-06-29T16:06:20.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.877+0000 7efd54ef5700 1 -- 172.21.15.60:0/2474368663 learned_addr learned my addr 172.21.15.60:0/2474368663 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:20.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.877+0000 7efd4ffff700 1 --2- 172.21.15.60:0/2474368663 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd5005cf60 0x7efd500af070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:20.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.877+0000 7efd556f6700 1 --2- 172.21.15.60:0/2474368663 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd50061870 0x7efd500b6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:20.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.877+0000 7efd54ef5700 1 -- 172.21.15.60:0/2474368663 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd5005cf60 msgr2=0x7efd500af070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:20.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.877+0000 7efd54ef5700 1 --2- 172.21.15.60:0/2474368663 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd5005cf60 0x7efd500af070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:20.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.877+0000 7efd54ef5700 1 -- 172.21.15.60:0/2474368663 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd50061870 msgr2=0x7efd500b6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:20.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.877+0000 7efd54ef5700 1 --2- 172.21.15.60:0/2474368663 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd50061870 0x7efd500b6ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:20.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.877+0000 7efd54ef5700 1 -- 172.21.15.60:0/2474368663 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd4000b180 con 0x7efd5005c2f0 2022-06-29T16:06:20.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.877+0000 7efd54ef5700 1 --2- 172.21.15.60:0/2474368663 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd5005c2f0 0x7efd500b72b0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 rx=0x7efd4400c000 tx=0x7efd44006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:20.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.877+0000 7efd4dffb700 1 -- 172.21.15.60:0/2474368663 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd44010780 con 0x7efd5005c2f0 2022-06-29T16:06:20.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.877+0000 7efd4dffb700 1 -- 172.21.15.60:0/2474368663 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd440108e0 con 0x7efd5005c2f0 2022-06-29T16:06:20.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.877+0000 7efd57159700 1 -- 172.21.15.60:0/2474368663 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd5013ef00 con 0x7efd5005c2f0 2022-06-29T16:06:20.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.877+0000 7efd4dffb700 1 -- 172.21.15.60:0/2474368663 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd44010c40 con 0x7efd5005c2f0 2022-06-29T16:06:20.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.877+0000 7efd57159700 1 -- 172.21.15.60:0/2474368663 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd5013f3d0 con 0x7efd5005c2f0 2022-06-29T16:06:20.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.877+0000 7efd57159700 1 -- 172.21.15.60:0/2474368663 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd500573d0 con 0x7efd5005c2f0 2022-06-29T16:06:20.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.877+0000 7efd4dffb700 1 -- 172.21.15.60:0/2474368663 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7efd4401c460 con 0x7efd5005c2f0 2022-06-29T16:06:20.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.877+0000 7efd4dffb700 1 --2- 172.21.15.60:0/2474368663 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd30053a10 0x7efd30055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:20.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.877+0000 7efd4dffb700 1 -- 172.21.15.60:0/2474368663 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5339+0+0 (secure 0 0 0) 0x7efd4406c0e0 con 0x7efd5005c2f0 2022-06-29T16:06:20.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.881+0000 7efd4ffff700 1 --2- 172.21.15.60:0/2474368663 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd30053a10 0x7efd30055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:20.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.881+0000 7efd4ffff700 1 --2- 172.21.15.60:0/2474368663 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd30053a10 0x7efd30055ed0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0x7efd40018000 tx=0x7efd40014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:20.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:20.881+0000 7efd4dffb700 1 -- 172.21.15.60:0/2474368663 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efd440431e0 con 0x7efd5005c2f0 2022-06-29T16:06:21.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.053+0000 7efd57159700 1 -- 172.21.15.60:0/2474368663 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_filtered_df"]} v 0) v1 -- 0x7efd500b7f50 con 0x7efd5005c2f0 2022-06-29T16:06:21.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.465+0000 7efd4dffb700 1 -- 172.21.15.60:0/2474368663 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_filtered_df"]}]=0 v360) v1 ==== 127+0+0 (secure 0 0 0) 0x7efd44046800 con 0x7efd5005c2f0 2022-06-29T16:06:21.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.469+0000 7efd57159700 1 -- 172.21.15.60:0/2474368663 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd30053a10 msgr2=0x7efd30055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:21.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.469+0000 7efd57159700 1 --2- 172.21.15.60:0/2474368663 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd30053a10 0x7efd30055ed0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0x7efd40018000 tx=0x7efd40014000).stop 2022-06-29T16:06:21.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.469+0000 7efd57159700 1 -- 172.21.15.60:0/2474368663 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd5005c2f0 msgr2=0x7efd500b72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:21.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.469+0000 7efd57159700 1 --2- 172.21.15.60:0/2474368663 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd5005c2f0 0x7efd500b72b0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 rx=0x7efd4400c000 tx=0x7efd44006df0).stop 2022-06-29T16:06:21.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.469+0000 7efd57159700 1 -- 172.21.15.60:0/2474368663 shutdown_connections 2022-06-29T16:06:21.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.469+0000 7efd57159700 1 --2- 172.21.15.60:0/2474368663 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd30053a10 0x7efd30055ed0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:21.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.469+0000 7efd57159700 1 --2- 172.21.15.60:0/2474368663 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd50061870 0x7efd500b6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:21.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.469+0000 7efd57159700 1 --2- 172.21.15.60:0/2474368663 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd5005cf60 0x7efd500af070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:21.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.469+0000 7efd57159700 1 --2- 172.21.15.60:0/2474368663 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd5005c2f0 0x7efd500b72b0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:21.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.469+0000 7efd57159700 1 -- 172.21.15.60:0/2474368663 >> 172.21.15.60:0/2474368663 conn(0x7efd50064a10 msgr2=0x7efd50064df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:21.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.469+0000 7efd57159700 1 -- 172.21.15.60:0/2474368663 shutdown_connections 2022-06-29T16:06:21.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.469+0000 7efd57159700 1 -- 172.21.15.60:0/2474368663 wait complete. 2022-06-29T16:06:21.489 INFO:tasks.cephfs_test_runner:test_filtered_df (tasks.cephfs.test_misc.TestMisc) ... ok 2022-06-29T16:06:21.490 INFO:tasks.cephfs_test_runner:Starting test: test_fs_new (tasks.cephfs.test_misc.TestMisc) 2022-06-29T16:06:21.490 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_fs_new' 2022-06-29T16:06:21.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.597+0000 7f135e090700 1 Processor -- start 2022-06-29T16:06:21.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.597+0000 7f135e090700 1 -- start start 2022-06-29T16:06:21.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.597+0000 7f135e090700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f13580b3d30 0x7f1358146ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:21.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.597+0000 7f135e090700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1358061d90 0x7f1358142b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:21.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.597+0000 7f135e090700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13580b4f50 con 0x7f13580b9d70 2022-06-29T16:06:21.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.597+0000 7f135e090700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13580ba150 con 0x7f13580b3d30 2022-06-29T16:06:21.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.597+0000 7f135e090700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13580b5180 con 0x7f1358061d90 2022-06-29T16:06:21.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.597+0000 7f13577fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f13580b9d70 0x7f13581465d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59906/0 (socket says 172.21.15.60:59906) 2022-06-29T16:06:21.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.597+0000 7f13577fe700 1 -- 172.21.15.60:0/2792794088 learned_addr learned my addr 172.21.15.60:0/2792794088 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:21.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.597+0000 7f1357fff700 1 --2- 172.21.15.60:0/2792794088 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1358061d90 0x7f1358142b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:21.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.597+0000 7f1356ffd700 1 --2- 172.21.15.60:0/2792794088 >> v2:172.21.15.92:3300/0 conn(0x7f13580b3d30 0x7f1358146ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:21.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.597+0000 7f13567fc700 1 -- 172.21.15.60:0/2792794088 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 632297889 0 0) 0x7f13580b4f50 con 0x7f13580b9d70 2022-06-29T16:06:21.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.597+0000 7f13567fc700 1 -- 172.21.15.60:0/2792794088 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1344003260 con 0x7f13580b9d70 2022-06-29T16:06:21.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.597+0000 7f1356ffd700 1 -- 172.21.15.60:0/2792794088 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1358061d90 msgr2=0x7f1358142b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:21.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.597+0000 7f1356ffd700 1 --2- 172.21.15.60:0/2792794088 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1358061d90 0x7f1358142b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:21.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.597+0000 7f1356ffd700 1 -- 172.21.15.60:0/2792794088 >> v1:172.21.15.60:6789/0 conn(0x7f13580b9d70 legacy=0x7f13581465d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:21.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.597+0000 7f1356ffd700 1 -- 172.21.15.60:0/2792794088 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13581461a0 con 0x7f13580b3d30 2022-06-29T16:06:21.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.597+0000 7f1357fff700 1 --2- 172.21.15.60:0/2792794088 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1358061d90 0x7f1358142b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:06:21.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.597+0000 7f1356ffd700 1 --2- 172.21.15.60:0/2792794088 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13580b3d30 0x7f1358146ce0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f134c002f70 tx=0x7f134c006df0).ready entity=mon.1 client_cookie=c4b206bc74ebc23d server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:21.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.601+0000 7f13567fc700 1 -- 172.21.15.60:0/2792794088 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f134c00d4a0 con 0x7f13580b3d30 2022-06-29T16:06:21.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.601+0000 7f13567fc700 1 -- 172.21.15.60:0/2792794088 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f134c000e90 con 0x7f13580b3d30 2022-06-29T16:06:21.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.601+0000 7f13567fc700 1 -- 172.21.15.60:0/2792794088 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f134c00d7f0 con 0x7f13580b3d30 2022-06-29T16:06:21.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.601+0000 7f135e090700 1 -- 172.21.15.60:0/2792794088 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13580b3d30 msgr2=0x7f1358146ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:21.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.601+0000 7f135e090700 1 --2- 172.21.15.60:0/2792794088 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13580b3d30 0x7f1358146ce0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f134c002f70 tx=0x7f134c006df0).stop 2022-06-29T16:06:21.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.601+0000 7f135e090700 1 -- 172.21.15.60:0/2792794088 shutdown_connections 2022-06-29T16:06:21.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.601+0000 7f135e090700 1 --2- 172.21.15.60:0/2792794088 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1358061d90 0x7f1358142b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:21.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.601+0000 7f135e090700 1 --2- 172.21.15.60:0/2792794088 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13580b3d30 0x7f1358146ce0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:21.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.601+0000 7f135e090700 1 -- 172.21.15.60:0/2792794088 >> 172.21.15.60:0/2792794088 conn(0x7f1358062e70 msgr2=0x7f13580b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:21.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.601+0000 7f135e090700 1 -- 172.21.15.60:0/2792794088 shutdown_connections 2022-06-29T16:06:21.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.601+0000 7f135e090700 1 -- 172.21.15.60:0/2792794088 wait complete. 2022-06-29T16:06:21.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.601+0000 7f135e090700 1 Processor -- start 2022-06-29T16:06:21.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.601+0000 7f135e090700 1 -- start start 2022-06-29T16:06:21.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.605+0000 7f135e090700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1358061d90 0x7f1358144750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:21.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.605+0000 7f135e090700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13580b3d30 0x7f1358151d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:21.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.605+0000 7f135e090700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13580b9d70 0x7f135813e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:21.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.605+0000 7f135e090700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13580af8f0 con 0x7f1358061d90 2022-06-29T16:06:21.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.605+0000 7f135e090700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1358145a80 con 0x7f13580b9d70 2022-06-29T16:06:21.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.605+0000 7f135e090700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1358145d20 con 0x7f13580b3d30 2022-06-29T16:06:21.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.605+0000 7f13577fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1358061d90 0x7f1358144750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:21.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.605+0000 7f13577fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1358061d90 0x7f1358144750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51826/0 (socket says 172.21.15.60:51826) 2022-06-29T16:06:21.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.605+0000 7f13577fe700 1 -- 172.21.15.60:0/903678330 learned_addr learned my addr 172.21.15.60:0/903678330 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:21.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.605+0000 7f1356ffd700 1 --2- 172.21.15.60:0/903678330 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13580b3d30 0x7f1358151d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:21.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.605+0000 7f1357fff700 1 --2- 172.21.15.60:0/903678330 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13580b9d70 0x7f135813e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:21.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.605+0000 7f13577fe700 1 -- 172.21.15.60:0/903678330 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13580b3d30 msgr2=0x7f1358151d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:21.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.605+0000 7f13577fe700 1 --2- 172.21.15.60:0/903678330 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13580b3d30 0x7f1358151d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:21.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.605+0000 7f13577fe700 1 -- 172.21.15.60:0/903678330 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13580b9d70 msgr2=0x7f135813e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:21.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.605+0000 7f13577fe700 1 --2- 172.21.15.60:0/903678330 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13580b9d70 0x7f135813e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:21.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.605+0000 7f13577fe700 1 -- 172.21.15.60:0/903678330 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1350009470 con 0x7f1358061d90 2022-06-29T16:06:21.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.605+0000 7f13577fe700 1 --2- 172.21.15.60:0/903678330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1358061d90 0x7f1358144750 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 rx=0x7f1348001110 tx=0x7f1348007ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:21.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.605+0000 7f1354ff9700 1 -- 172.21.15.60:0/903678330 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13480070d0 con 0x7f1358061d90 2022-06-29T16:06:21.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.605+0000 7f1354ff9700 1 -- 172.21.15.60:0/903678330 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1348007230 con 0x7f1358061d90 2022-06-29T16:06:21.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.605+0000 7f1354ff9700 1 -- 172.21.15.60:0/903678330 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1348005610 con 0x7f1358061d90 2022-06-29T16:06:21.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.605+0000 7f135e090700 1 -- 172.21.15.60:0/903678330 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f134c00b180 con 0x7f1358061d90 2022-06-29T16:06:21.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.605+0000 7f135e090700 1 -- 172.21.15.60:0/903678330 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13580afb30 con 0x7f1358061d90 2022-06-29T16:06:21.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.609+0000 7f135e090700 1 -- 172.21.15.60:0/903678330 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1358145ee0 con 0x7f1358061d90 2022-06-29T16:06:21.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.609+0000 7f1354ff9700 1 -- 172.21.15.60:0/903678330 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1348010070 con 0x7f1358061d90 2022-06-29T16:06:21.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.609+0000 7f1354ff9700 1 --2- 172.21.15.60:0/903678330 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1338053a70 0x7f1338055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:21.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.609+0000 7f1354ff9700 1 -- 172.21.15.60:0/903678330 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f134806b200 con 0x7f1358061d90 2022-06-29T16:06:21.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.609+0000 7f1356ffd700 1 --2- 172.21.15.60:0/903678330 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1338053a70 0x7f1338055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:21.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.609+0000 7f1356ffd700 1 --2- 172.21.15.60:0/903678330 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1338053a70 0x7f1338055f30 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7f134c01a000 tx=0x7f134c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:21.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.617+0000 7f1354ff9700 1 -- 172.21.15.60:0/903678330 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1348042400 con 0x7f1358061d90 2022-06-29T16:06:21.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:21.789+0000 7f135e090700 1 -- 172.21.15.60:0/903678330 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_fs_new"]} v 0) v1 -- 0x7f13580573d0 con 0x7f1358061d90 2022-06-29T16:06:22.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.469+0000 7f1354ff9700 1 -- 172.21.15.60:0/903678330 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_fs_new"]}]=0 v361) v1 ==== 125+0+0 (secure 0 0 0) 0x7f1348045a20 con 0x7f1358061d90 2022-06-29T16:06:22.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.473+0000 7f135e090700 1 -- 172.21.15.60:0/903678330 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1338053a70 msgr2=0x7f1338055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:22.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.473+0000 7f135e090700 1 --2- 172.21.15.60:0/903678330 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1338053a70 0x7f1338055f30 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7f134c01a000 tx=0x7f134c014000).stop 2022-06-29T16:06:22.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.473+0000 7f135e090700 1 -- 172.21.15.60:0/903678330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1358061d90 msgr2=0x7f1358144750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:22.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.473+0000 7f135e090700 1 --2- 172.21.15.60:0/903678330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1358061d90 0x7f1358144750 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 rx=0x7f1348001110 tx=0x7f1348007ad0).stop 2022-06-29T16:06:22.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.473+0000 7f135e090700 1 -- 172.21.15.60:0/903678330 shutdown_connections 2022-06-29T16:06:22.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.473+0000 7f135e090700 1 --2- 172.21.15.60:0/903678330 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1338053a70 0x7f1338055f30 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:22.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.473+0000 7f135e090700 1 --2- 172.21.15.60:0/903678330 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13580b9d70 0x7f135813e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:22.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.473+0000 7f135e090700 1 --2- 172.21.15.60:0/903678330 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f13580b3d30 0x7f1358151d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:22.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.473+0000 7f135e090700 1 --2- 172.21.15.60:0/903678330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1358061d90 0x7f1358144750 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:22.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.473+0000 7f135e090700 1 -- 172.21.15.60:0/903678330 >> 172.21.15.60:0/903678330 conn(0x7f1358062e70 msgr2=0x7f1358142cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:22.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.473+0000 7f135e090700 1 -- 172.21.15.60:0/903678330 shutdown_connections 2022-06-29T16:06:22.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.473+0000 7f135e090700 1 -- 172.21.15.60:0/903678330 wait complete. 2022-06-29T16:06:22.494 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-06-29T16:06:22.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.605+0000 7f007a06d700 1 Processor -- start 2022-06-29T16:06:22.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.605+0000 7f007a06d700 1 -- start start 2022-06-29T16:06:22.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.605+0000 7f007a06d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00740b6a50 0x7f00741461e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:22.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.605+0000 7f007a06d700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f007405ca50 0x7f007405cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:22.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.605+0000 7f007a06d700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00740b6e30 con 0x7f007405f0b0 2022-06-29T16:06:22.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.605+0000 7f007a06d700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00740b0dd0 con 0x7f007405ca50 2022-06-29T16:06:22.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.605+0000 7f007a06d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00740b1070 con 0x7f00740b6a50 2022-06-29T16:06:22.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.605+0000 7f00737fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00740b6a50 0x7f00741461e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:22.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.605+0000 7f0073fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f007405ca50 0x7f007405cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:22.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.609+0000 7f0073fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f007405ca50 0x7f007405cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:57868/0 (socket says 172.21.15.60:57868) 2022-06-29T16:06:22.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.609+0000 7f0073fff700 1 -- 172.21.15.60:0/3752703393 learned_addr learned my addr 172.21.15.60:0/3752703393 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:22.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.605+0000 7f00737fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00740b6a50 0x7f00741461e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:36696/0 (socket says 172.21.15.60:36696) 2022-06-29T16:06:22.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.605+0000 7f006affd700 1 --1- 172.21.15.60:0/3752703393 >> v1:172.21.15.92:6789/0 conn(0x7f007405f0b0 0x7f0074142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:35938/0 (socket says 172.21.15.60:35938) 2022-06-29T16:06:22.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.609+0000 7f00737fe700 1 -- 172.21.15.60:0/3752703393 >> v2:172.21.15.92:3300/0 conn(0x7f007405ca50 msgr2=0x7f007405cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:22.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.609+0000 7f00737fe700 1 --2- 172.21.15.60:0/3752703393 >> v2:172.21.15.92:3300/0 conn(0x7f007405ca50 0x7f007405cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:22.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.609+0000 7f00737fe700 1 -- 172.21.15.60:0/3752703393 >> v1:172.21.15.92:6789/0 conn(0x7f007405f0b0 legacy=0x7f0074142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:22.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.609+0000 7f00737fe700 1 -- 172.21.15.60:0/3752703393 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f007405c5b0 con 0x7f00740b6a50 2022-06-29T16:06:22.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.609+0000 7f00737fe700 1 --2- 172.21.15.60:0/3752703393 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00740b6a50 0x7f00741461e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f0064002f70 tx=0x7f0064006df0).ready entity=mon.2 client_cookie=77db2a654c0df507 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:22.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.609+0000 7f0072ffd700 1 -- 172.21.15.60:0/3752703393 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f006400d570 con 0x7f00740b6a50 2022-06-29T16:06:22.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.609+0000 7f0072ffd700 1 -- 172.21.15.60:0/3752703393 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0064000e90 con 0x7f00740b6a50 2022-06-29T16:06:22.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.609+0000 7f0072ffd700 1 -- 172.21.15.60:0/3752703393 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f006400d8c0 con 0x7f00740b6a50 2022-06-29T16:06:22.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.609+0000 7f007a06d700 1 -- 172.21.15.60:0/3752703393 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00740b6a50 msgr2=0x7f00741461e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:22.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.609+0000 7f007a06d700 1 --2- 172.21.15.60:0/3752703393 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00740b6a50 0x7f00741461e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f0064002f70 tx=0x7f0064006df0).stop 2022-06-29T16:06:22.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.609+0000 7f007a06d700 1 -- 172.21.15.60:0/3752703393 shutdown_connections 2022-06-29T16:06:22.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.609+0000 7f007a06d700 1 --2- 172.21.15.60:0/3752703393 >> v2:172.21.15.92:3300/0 conn(0x7f007405ca50 0x7f007405cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:22.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.609+0000 7f007a06d700 1 --2- 172.21.15.60:0/3752703393 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00740b6a50 0x7f00741461e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:22.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.609+0000 7f007a06d700 1 -- 172.21.15.60:0/3752703393 >> 172.21.15.60:0/3752703393 conn(0x7f00740643e0 msgr2=0x7f00740b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:22.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.609+0000 7f007a06d700 1 -- 172.21.15.60:0/3752703393 shutdown_connections 2022-06-29T16:06:22.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.609+0000 7f007a06d700 1 -- 172.21.15.60:0/3752703393 wait complete. 2022-06-29T16:06:22.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.613+0000 7f007a06d700 1 Processor -- start 2022-06-29T16:06:22.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.613+0000 7f007a06d700 1 -- start start 2022-06-29T16:06:22.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.613+0000 7f007a06d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f007405ca50 0x7f0074144420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:22.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.613+0000 7f007a06d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f007405f0b0 0x7f0074065ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:22.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.613+0000 7f007a06d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00740b6a50 0x7f0074145ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:22.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.613+0000 7f007a06d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0074066b10 con 0x7f007405f0b0 2022-06-29T16:06:22.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.613+0000 7f007a06d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0074066db0 con 0x7f007405ca50 2022-06-29T16:06:22.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.613+0000 7f007a06d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0074067050 con 0x7f00740b6a50 2022-06-29T16:06:22.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.613+0000 7f006affd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f007405f0b0 0x7f0074065ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:22.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.613+0000 7f00737fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f007405ca50 0x7f0074144420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:22.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.613+0000 7f0073fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00740b6a50 0x7f0074145ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:22.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.613+0000 7f006affd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f007405f0b0 0x7f0074065ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51842/0 (socket says 172.21.15.60:51842) 2022-06-29T16:06:22.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.613+0000 7f006affd700 1 -- 172.21.15.60:0/2749797507 learned_addr learned my addr 172.21.15.60:0/2749797507 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:22.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.613+0000 7f0073fff700 1 -- 172.21.15.60:0/2749797507 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f007405ca50 msgr2=0x7f0074144420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:22.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.613+0000 7f0073fff700 1 --2- 172.21.15.60:0/2749797507 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f007405ca50 0x7f0074144420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:22.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.613+0000 7f0073fff700 1 -- 172.21.15.60:0/2749797507 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f007405f0b0 msgr2=0x7f0074065ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:22.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.613+0000 7f0073fff700 1 --2- 172.21.15.60:0/2749797507 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f007405f0b0 0x7f0074065ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:22.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.613+0000 7f0073fff700 1 -- 172.21.15.60:0/2749797507 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006400b180 con 0x7f00740b6a50 2022-06-29T16:06:22.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.613+0000 7f00737fe700 1 --2- 172.21.15.60:0/2749797507 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f007405ca50 0x7f0074144420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:06:22.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.613+0000 7f0073fff700 1 --2- 172.21.15.60:0/2749797507 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00740b6a50 0x7f0074145ee0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f006c00c000 tx=0x7f006c009290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:22.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.613+0000 7f00717fa700 1 -- 172.21.15.60:0/2749797507 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f006c012600 con 0x7f00740b6a50 2022-06-29T16:06:22.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.613+0000 7f00717fa700 1 -- 172.21.15.60:0/2749797507 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f006c012760 con 0x7f00740b6a50 2022-06-29T16:06:22.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.613+0000 7f007a06d700 1 -- 172.21.15.60:0/2749797507 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0074146e70 con 0x7f00740b6a50 2022-06-29T16:06:22.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.613+0000 7f00717fa700 1 -- 172.21.15.60:0/2749797507 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f006c012ac0 con 0x7f00740b6a50 2022-06-29T16:06:22.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.613+0000 7f007a06d700 1 -- 172.21.15.60:0/2749797507 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00740674f0 con 0x7f00740b6a50 2022-06-29T16:06:22.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.617+0000 7f007a06d700 1 -- 172.21.15.60:0/2749797507 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00740573d0 con 0x7f00740b6a50 2022-06-29T16:06:22.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.617+0000 7f00717fa700 1 -- 172.21.15.60:0/2749797507 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f006c022070 con 0x7f00740b6a50 2022-06-29T16:06:22.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.617+0000 7f00717fa700 1 --2- 172.21.15.60:0/2749797507 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0058053a70 0x7f0058055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:22.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.617+0000 7f00717fa700 1 -- 172.21.15.60:0/2749797507 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f006c06c220 con 0x7f00740b6a50 2022-06-29T16:06:22.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.617+0000 7f00737fe700 1 --2- 172.21.15.60:0/2749797507 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0058053a70 0x7f0058055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:22.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.621+0000 7f00737fe700 1 --2- 172.21.15.60:0/2749797507 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0058053a70 0x7f0058055f30 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7f006401a000 tx=0x7f0064014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:22.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.629+0000 7f00717fa700 1 -- 172.21.15.60:0/2749797507 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f006c043340 con 0x7f00740b6a50 2022-06-29T16:06:22.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.761+0000 7f007a06d700 1 -- 172.21.15.60:0/2749797507 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f0074067960 con 0x7f00740b6a50 2022-06-29T16:06:22.778 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.773+0000 7f00717fa700 1 -- 172.21.15.60:0/2749797507 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v18)=0 v18) v1 ==== 122+0+0 (secure 0 0 0) 0x7f006c0160c0 con 0x7f00740b6a50 2022-06-29T16:06:22.781 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.773+0000 7f007a06d700 1 -- 172.21.15.60:0/2749797507 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0058053a70 msgr2=0x7f0058055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:22.781 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.773+0000 7f007a06d700 1 --2- 172.21.15.60:0/2749797507 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0058053a70 0x7f0058055f30 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7f006401a000 tx=0x7f0064014000).stop 2022-06-29T16:06:22.782 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.773+0000 7f007a06d700 1 -- 172.21.15.60:0/2749797507 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00740b6a50 msgr2=0x7f0074145ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:22.782 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.773+0000 7f007a06d700 1 --2- 172.21.15.60:0/2749797507 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00740b6a50 0x7f0074145ee0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f006c00c000 tx=0x7f006c009290).stop 2022-06-29T16:06:22.783 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.773+0000 7f007a06d700 1 -- 172.21.15.60:0/2749797507 shutdown_connections 2022-06-29T16:06:22.783 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.773+0000 7f007a06d700 1 --2- 172.21.15.60:0/2749797507 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0058053a70 0x7f0058055f30 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:22.783 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.773+0000 7f007a06d700 1 --2- 172.21.15.60:0/2749797507 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00740b6a50 0x7f0074145ee0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:22.784 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.773+0000 7f007a06d700 1 --2- 172.21.15.60:0/2749797507 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f007405f0b0 0x7f0074065ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:22.784 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.773+0000 7f007a06d700 1 --2- 172.21.15.60:0/2749797507 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f007405ca50 0x7f0074144420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:22.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.773+0000 7f007a06d700 1 -- 172.21.15.60:0/2749797507 >> 172.21.15.60:0/2749797507 conn(0x7f00740643e0 msgr2=0x7f007413f1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:22.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.773+0000 7f007a06d700 1 -- 172.21.15.60:0/2749797507 shutdown_connections 2022-06-29T16:06:22.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:22.773+0000 7f007a06d700 1 -- 172.21.15.60:0/2749797507 wait complete. 2022-06-29T16:06:22.793 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-06-29T16:06:22.794 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:06:22.812 DEBUG:teuthology.orchestra.run.smithi092:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:06:22.834 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:06:22.855 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:06:22.876 DEBUG:teuthology.orchestra.run.smithi197:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:06:22.898 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-06-29T16:06:22.898 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:22.899 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:06:22.899 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:06:22.907 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-06-29T16:06:22.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:06:22.910 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-06-29T16:06:22.911 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:22.911 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:06:22.912 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:06:22.920 INFO:teuthology.orchestra.run.smithi156.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-06-29T16:06:22.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:06:22.922 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-06-29T16:06:22.922 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:22.923 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:06:22.923 DEBUG:teuthology.orchestra.run.smithi197:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T16:06:22.930 INFO:teuthology.orchestra.run.smithi197.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-06-29T16:06:22.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:06:22.932 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-06-29T16:06:22.933 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-06-29T16:06:23.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.037+0000 7fa0f5054700 1 Processor -- start 2022-06-29T16:06:23.044 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.037+0000 7fa0f5054700 1 -- start start 2022-06-29T16:06:23.045 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.037+0000 7fa0f5054700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa0f0063ad0 0x7fa0f0067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:23.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.037+0000 7fa0f5054700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa0f014af10 0x7fa0f005c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:23.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.037+0000 7fa0f5054700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f00acba0 con 0x7fa0f00b5950 2022-06-29T16:06:23.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.037+0000 7fa0f5054700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0f014b3a0 con 0x7fa0f0063ad0 2022-06-29T16:06:23.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.037+0000 7fa0f5054700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f0067220 con 0x7fa0f014af10 2022-06-29T16:06:23.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.041+0000 7fa0eed9d700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa0f00b5950 0x7fa0f005bd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59934/0 (socket says 172.21.15.60:59934) 2022-06-29T16:06:23.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.041+0000 7fa0eed9d700 1 -- 172.21.15.60:0/3634053665 learned_addr learned my addr 172.21.15.60:0/3634053665 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:23.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.041+0000 7fa0ef59e700 1 --2- 172.21.15.60:0/3634053665 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa0f014af10 0x7fa0f005c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:23.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.041+0000 7fa0ee59c700 1 --2- 172.21.15.60:0/3634053665 >> v2:172.21.15.92:3300/0 conn(0x7fa0f0063ad0 0x7fa0f0067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:23.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.041+0000 7fa0ee59c700 1 -- 172.21.15.60:0/3634053665 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa0f014af10 msgr2=0x7fa0f005c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:23.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.041+0000 7fa0ee59c700 1 --2- 172.21.15.60:0/3634053665 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa0f014af10 0x7fa0f005c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:23.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.041+0000 7fa0ee59c700 1 -- 172.21.15.60:0/3634053665 >> v1:172.21.15.60:6789/0 conn(0x7fa0f00b5950 legacy=0x7fa0f005bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:23.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.041+0000 7fa0edd9b700 1 -- 172.21.15.60:0/3634053665 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3795761895 0 0) 0x7fa0f00acba0 con 0x7fa0f00b5950 2022-06-29T16:06:23.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.041+0000 7fa0ee59c700 1 -- 172.21.15.60:0/3634053665 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f00b5d30 con 0x7fa0f0063ad0 2022-06-29T16:06:23.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.041+0000 7fa0ee59c700 1 --2- 172.21.15.60:0/3634053665 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0f0063ad0 0x7fa0f0067680 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7fa0e0002f70 tx=0x7fa0e0006df0).ready entity=mon.1 client_cookie=6cb1a8d6d4c28ae3 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:23.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.041+0000 7fa0edd9b700 1 -- 172.21.15.60:0/3634053665 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0e000d570 con 0x7fa0f0063ad0 2022-06-29T16:06:23.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.041+0000 7fa0edd9b700 1 -- 172.21.15.60:0/3634053665 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0e0000e90 con 0x7fa0f0063ad0 2022-06-29T16:06:23.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.041+0000 7fa0edd9b700 1 -- 172.21.15.60:0/3634053665 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0e000d8c0 con 0x7fa0f0063ad0 2022-06-29T16:06:23.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.041+0000 7fa0f5054700 1 -- 172.21.15.60:0/3634053665 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0f0063ad0 msgr2=0x7fa0f0067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:23.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.041+0000 7fa0f5054700 1 --2- 172.21.15.60:0/3634053665 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0f0063ad0 0x7fa0f0067680 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7fa0e0002f70 tx=0x7fa0e0006df0).stop 2022-06-29T16:06:23.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.041+0000 7fa0f5054700 1 -- 172.21.15.60:0/3634053665 shutdown_connections 2022-06-29T16:06:23.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.041+0000 7fa0f5054700 1 --2- 172.21.15.60:0/3634053665 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa0f014af10 0x7fa0f005c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:23.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.041+0000 7fa0f5054700 1 --2- 172.21.15.60:0/3634053665 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0f0063ad0 0x7fa0f0067680 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:23.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.041+0000 7fa0f5054700 1 -- 172.21.15.60:0/3634053665 >> 172.21.15.60:0/3634053665 conn(0x7fa0f00632c0 msgr2=0x7fa0f00b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:23.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.041+0000 7fa0f5054700 1 -- 172.21.15.60:0/3634053665 shutdown_connections 2022-06-29T16:06:23.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.041+0000 7fa0f5054700 1 -- 172.21.15.60:0/3634053665 wait complete. 2022-06-29T16:06:23.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.045+0000 7fa0f5054700 1 Processor -- start 2022-06-29T16:06:23.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.045+0000 7fa0f5054700 1 -- start start 2022-06-29T16:06:23.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.045+0000 7fa0f5054700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0f0063ad0 0x7fa0f00aef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:23.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.045+0000 7fa0f5054700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa0f00b5950 0x7fa0f014cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:23.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.045+0000 7fa0f5054700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0f014af10 0x7fa0f00af4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:23.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.045+0000 7fa0f5054700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f00ae690 con 0x7fa0f014af10 2022-06-29T16:06:23.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.045+0000 7fa0f5054700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f00afb60 con 0x7fa0f0063ad0 2022-06-29T16:06:23.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.045+0000 7fa0f5054700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f00afe00 con 0x7fa0f00b5950 2022-06-29T16:06:23.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.045+0000 7fa0ee59c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa0f00b5950 0x7fa0f014cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:23.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.045+0000 7fa0ef59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0f014af10 0x7fa0f00af4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:23.058 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.045+0000 7fa0eed9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0f0063ad0 0x7fa0f00aef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:23.058 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.045+0000 7fa0ee59c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa0f00b5950 0x7fa0f014cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:36718/0 (socket says 172.21.15.60:36718) 2022-06-29T16:06:23.058 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.045+0000 7fa0ee59c700 1 -- 172.21.15.60:0/2372341941 learned_addr learned my addr 172.21.15.60:0/2372341941 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:23.058 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.049+0000 7fa0ee59c700 1 -- 172.21.15.60:0/2372341941 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0f0063ad0 msgr2=0x7fa0f00aef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:23.058 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.049+0000 7fa0ee59c700 1 --2- 172.21.15.60:0/2372341941 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0f0063ad0 0x7fa0f00aef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:23.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.049+0000 7fa0ee59c700 1 -- 172.21.15.60:0/2372341941 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0f014af10 msgr2=0x7fa0f00af4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:23.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.049+0000 7fa0ee59c700 1 --2- 172.21.15.60:0/2372341941 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0f014af10 0x7fa0f00af4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:23.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.049+0000 7fa0ee59c700 1 -- 172.21.15.60:0/2372341941 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e000b180 con 0x7fa0f00b5950 2022-06-29T16:06:23.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.049+0000 7fa0ee59c700 1 --2- 172.21.15.60:0/2372341941 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa0f00b5950 0x7fa0f014cc90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fa0e0002f70 tx=0x7fa0e0014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:23.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.049+0000 7fa0e7fff700 1 -- 172.21.15.60:0/2372341941 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0e0016070 con 0x7fa0f00b5950 2022-06-29T16:06:23.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.049+0000 7fa0e7fff700 1 -- 172.21.15.60:0/2372341941 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0e00138d0 con 0x7fa0f00b5950 2022-06-29T16:06:23.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.049+0000 7fa0f5054700 1 -- 172.21.15.60:0/2372341941 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0f00b00a0 con 0x7fa0f00b5950 2022-06-29T16:06:23.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.049+0000 7fa0e7fff700 1 -- 172.21.15.60:0/2372341941 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0e0013c30 con 0x7fa0f00b5950 2022-06-29T16:06:23.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.049+0000 7fa0f5054700 1 -- 172.21.15.60:0/2372341941 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f01561b0 con 0x7fa0f00b5950 2022-06-29T16:06:23.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.049+0000 7fa0f5054700 1 -- 172.21.15.60:0/2372341941 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0f00af990 con 0x7fa0f00b5950 2022-06-29T16:06:23.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.049+0000 7fa0e7fff700 1 -- 172.21.15.60:0/2372341941 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fa0e0025030 con 0x7fa0f00b5950 2022-06-29T16:06:23.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.053+0000 7fa0e7fff700 1 --2- 172.21.15.60:0/2372341941 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa0d4053a70 0x7fa0d4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:23.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.053+0000 7fa0e7fff700 1 -- 172.21.15.60:0/2372341941 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fa0e006cb60 con 0x7fa0f00b5950 2022-06-29T16:06:23.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.053+0000 7fa0eed9d700 1 --2- 172.21.15.60:0/2372341941 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa0d4053a70 0x7fa0d4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:23.063 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.053+0000 7fa0eed9d700 1 --2- 172.21.15.60:0/2372341941 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa0d4053a70 0x7fa0d4055f30 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0x7fa0d800d000 tx=0x7fa0d8007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:23.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.061+0000 7fa0e7fff700 1 -- 172.21.15.60:0/2372341941 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa0e0043d60 con 0x7fa0f00b5950 2022-06-29T16:06:23.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.189+0000 7fa0f5054700 1 -- 172.21.15.60:0/2372341941 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fa0f00573d0 con 0x7fa0f00b5950 2022-06-29T16:06:23.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.193+0000 7fa0e7fff700 1 -- 172.21.15.60:0/2372341941 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v103) v1 ==== 81+0+28829 (secure 0 0 0) 0x7fa0e0023020 con 0x7fa0f00b5950 2022-06-29T16:06:23.199 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:23.199 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:06:23.199 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 103, 2022-06-29T16:06:23.200 INFO:teuthology.orchestra.run.smithi060.stdout: "fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", 2022-06-29T16:06:23.200 INFO:teuthology.orchestra.run.smithi060.stdout: "created": "2022-06-29T15:58:07.965719+0000", 2022-06-29T16:06:23.200 INFO:teuthology.orchestra.run.smithi060.stdout: "modified": "2022-06-29T16:06:20.448382+0000", 2022-06-29T16:06:23.200 INFO:teuthology.orchestra.run.smithi060.stdout: "last_up_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:06:23.201 INFO:teuthology.orchestra.run.smithi060.stdout: "last_in_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:06:23.201 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-06-29T16:06:23.201 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_num": 5799936, 2022-06-29T16:06:23.201 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_set": [ 2022-06-29T16:06:23.201 INFO:teuthology.orchestra.run.smithi060.stdout: "pglog_hardlimit", 2022-06-29T16:06:23.202 INFO:teuthology.orchestra.run.smithi060.stdout: "purged_snapdirs", 2022-06-29T16:06:23.202 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_deletes", 2022-06-29T16:06:23.202 INFO:teuthology.orchestra.run.smithi060.stdout: "sortbitwise" 2022-06-29T16:06:23.202 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:06:23.202 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_version": 4, 2022-06-29T16:06:23.203 INFO:teuthology.orchestra.run.smithi060.stdout: "full_ratio": 0.94999998807907104, 2022-06-29T16:06:23.203 INFO:teuthology.orchestra.run.smithi060.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-06-29T16:06:23.203 INFO:teuthology.orchestra.run.smithi060.stdout: "nearfull_ratio": 0.85000002384185791, 2022-06-29T16:06:23.203 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_snapshot": "", 2022-06-29T16:06:23.204 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_max": 20, 2022-06-29T16:06:23.204 INFO:teuthology.orchestra.run.smithi060.stdout: "max_osd": 8, 2022-06-29T16:06:23.204 INFO:teuthology.orchestra.run.smithi060.stdout: "require_min_compat_client": "jewel", 2022-06-29T16:06:23.204 INFO:teuthology.orchestra.run.smithi060.stdout: "min_compat_client": "jewel", 2022-06-29T16:06:23.204 INFO:teuthology.orchestra.run.smithi060.stdout: "require_osd_release": "octopus", 2022-06-29T16:06:23.205 INFO:teuthology.orchestra.run.smithi060.stdout: "pools": [ 2022-06-29T16:06:23.205 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.205 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2022-06-29T16:06:23.205 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "device_health_metrics", 2022-06-29T16:06:23.206 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:10.961127+0000", 2022-06-29T16:06:23.206 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:06:23.206 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:06:23.206 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:06:23.206 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:06:23.207 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:06:23.207 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:06:23.207 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:06:23.207 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:06:23.207 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 1, 2022-06-29T16:06:23.208 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 1, 2022-06-29T16:06:23.208 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 1, 2022-06-29T16:06:23.208 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 1, 2022-06-29T16:06:23.208 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 1, 2022-06-29T16:06:23.209 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:06:23.209 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:06:23.209 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:06:23.209 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:06:23.210 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:06:23.210 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:06:23.210 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:06:23.210 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.210 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "15", 2022-06-29T16:06:23.211 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:06:23.211 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:06:23.211 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:06:23.211 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:06:23.212 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:06:23.212 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:06:23.212 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:06:23.212 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:06:23.212 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:06:23.213 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:06:23.213 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:06:23.213 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:06:23.213 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:06:23.213 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:06:23.214 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:06:23.214 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:06:23.214 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:06:23.214 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:06:23.215 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:06:23.215 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:06:23.215 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:06:23.215 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:06:23.215 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:06:23.216 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:06:23.216 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:06:23.216 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:06:23.216 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.217 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:06:23.217 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:06:23.217 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:06:23.217 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:06:23.217 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:06:23.218 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:06:23.218 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:06:23.218 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:06:23.218 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:06:23.218 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:06:23.219 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:06:23.219 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:06:23.219 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 1 2022-06-29T16:06:23.219 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.220 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:06:23.220 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_devicehealth": {} 2022-06-29T16:06:23.220 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.220 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.220 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.221 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 2, 2022-06-29T16:06:23.221 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "rbd", 2022-06-29T16:06:23.221 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:19.184276+0000", 2022-06-29T16:06:23.221 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:06:23.221 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:06:23.222 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:06:23.222 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:06:23.222 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:06:23.222 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:06:23.223 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:06:23.223 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:06:23.223 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:06:23.223 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:06:23.223 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:06:23.224 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:06:23.224 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:06:23.224 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:06:23.224 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:06:23.224 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:06:23.225 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:06:23.225 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:06:23.225 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:06:23.225 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:06:23.226 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.226 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "18", 2022-06-29T16:06:23.226 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:06:23.226 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:06:23.226 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:06:23.227 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:06:23.227 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:06:23.227 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:06:23.227 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:06:23.227 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:06:23.228 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:06:23.228 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:06:23.228 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:06:23.228 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:06:23.229 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:06:23.229 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:06:23.229 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:06:23.229 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:06:23.229 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:06:23.230 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:06:23.230 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:06:23.230 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:06:23.230 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:06:23.230 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:06:23.231 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:06:23.231 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:06:23.231 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:06:23.231 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:06:23.232 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.232 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:06:23.232 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:06:23.232 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:06:23.232 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:06:23.233 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:06:23.233 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:06:23.233 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:06:23.233 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:06:23.233 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:06:23.234 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:06:23.234 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:06:23.234 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:06:23.234 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:06:23.235 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd": {} 2022-06-29T16:06:23.235 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.235 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.235 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:06:23.235 INFO:teuthology.orchestra.run.smithi060.stdout: "osds": [ 2022-06-29T16:06:23.236 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.236 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:06:23.236 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "11e7ab9e-fce2-470f-8195-ea4c3c52df22", 2022-06-29T16:06:23.236 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:06:23.237 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:06:23.237 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:06:23.237 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:06:23.237 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:06:23.237 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:06:23.238 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:06:23.238 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 98, 2022-06-29T16:06:23.238 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:06:23.238 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:06:23.238 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:06:23.239 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:23.239 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.239 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:23.239 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6816", 2022-06-29T16:06:23.239 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:06:23.240 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.240 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.240 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:23.240 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6825", 2022-06-29T16:06:23.241 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:06:23.241 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.241 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.242 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.242 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:06:23.242 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:23.242 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.243 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:23.243 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6826", 2022-06-29T16:06:23.243 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:06:23.243 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.244 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.244 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:23.244 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6827", 2022-06-29T16:06:23.244 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:06:23.244 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.245 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.245 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.245 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:06:23.245 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:23.246 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.246 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:23.246 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6830", 2022-06-29T16:06:23.246 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:06:23.246 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.247 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.247 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:23.247 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6831", 2022-06-29T16:06:23.247 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:06:23.248 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.248 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.248 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.248 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:06:23.249 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:23.249 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.249 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:23.249 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6828", 2022-06-29T16:06:23.249 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:06:23.250 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.250 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.250 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:23.250 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6829", 2022-06-29T16:06:23.251 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:06:23.251 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.251 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.251 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.251 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6825/12890", 2022-06-29T16:06:23.252 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6827/12890", 2022-06-29T16:06:23.252 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6831/12890", 2022-06-29T16:06:23.252 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6829/12890", 2022-06-29T16:06:23.252 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:06:23.253 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:06:23.253 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:06:23.253 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.253 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.254 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.254 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:06:23.254 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "ca43f136-6f69-4431-a157-fbe073fb585c", 2022-06-29T16:06:23.254 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:06:23.255 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:06:23.255 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:06:23.255 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:06:23.255 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:06:23.256 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:06:23.256 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:06:23.256 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 98, 2022-06-29T16:06:23.256 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:06:23.256 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:06:23.257 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:06:23.257 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:23.257 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.257 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:23.257 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6815", 2022-06-29T16:06:23.258 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:06:23.258 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.258 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.258 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:23.259 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6818", 2022-06-29T16:06:23.259 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:06:23.259 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.260 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.260 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.260 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:06:23.260 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:23.261 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.261 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:23.261 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6819", 2022-06-29T16:06:23.261 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:06:23.262 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.262 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.262 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:23.262 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6820", 2022-06-29T16:06:23.262 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:06:23.263 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.263 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.263 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.263 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:06:23.264 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:23.264 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.264 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:23.264 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6823", 2022-06-29T16:06:23.265 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:06:23.265 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.265 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.265 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:23.265 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6824", 2022-06-29T16:06:23.266 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:06:23.266 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.266 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.267 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.267 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:06:23.267 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:23.267 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.267 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:23.268 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6821", 2022-06-29T16:06:23.268 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:06:23.268 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.268 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.269 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:23.269 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6822", 2022-06-29T16:06:23.269 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:06:23.269 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.269 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.270 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.270 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6818/12892", 2022-06-29T16:06:23.270 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6820/12892", 2022-06-29T16:06:23.270 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6824/12892", 2022-06-29T16:06:23.271 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6822/12892", 2022-06-29T16:06:23.271 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:06:23.271 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:06:23.271 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:06:23.271 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.272 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.272 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.272 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:06:23.272 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "74fac45f-5f07-44ec-a9cb-2644c123fd4a", 2022-06-29T16:06:23.272 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:06:23.273 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:06:23.273 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:06:23.273 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:06:23.273 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:06:23.273 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:06:23.274 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:06:23.274 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 97, 2022-06-29T16:06:23.274 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:06:23.274 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:06:23.275 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:06:23.275 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:23.275 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.275 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:23.275 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6808", 2022-06-29T16:06:23.276 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:06:23.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.276 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.276 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:23.276 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6809", 2022-06-29T16:06:23.277 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:06:23.277 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.277 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.278 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:06:23.278 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:23.278 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.278 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:23.278 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6810", 2022-06-29T16:06:23.279 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:06:23.279 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.279 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.279 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:23.280 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6811", 2022-06-29T16:06:23.280 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:06:23.280 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.280 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.280 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.281 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:06:23.281 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:23.281 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.281 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:23.281 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6814", 2022-06-29T16:06:23.282 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:06:23.282 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.282 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.282 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:23.282 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6817", 2022-06-29T16:06:23.283 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:06:23.283 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.283 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.283 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.284 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:06:23.284 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:23.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.284 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:23.284 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6812", 2022-06-29T16:06:23.285 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:06:23.285 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.285 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.285 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:23.286 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6813", 2022-06-29T16:06:23.286 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:06:23.286 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.286 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.286 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.287 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6809/12891", 2022-06-29T16:06:23.287 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6811/12891", 2022-06-29T16:06:23.287 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6817/12891", 2022-06-29T16:06:23.287 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6813/12891", 2022-06-29T16:06:23.287 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:06:23.288 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:06:23.288 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:06:23.288 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.288 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.289 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.289 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:06:23.289 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "2581d351-973a-4b4e-85ca-3658f0068765", 2022-06-29T16:06:23.289 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:06:23.289 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:06:23.290 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:06:23.290 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:06:23.290 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:06:23.290 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:06:23.290 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:06:23.291 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 98, 2022-06-29T16:06:23.291 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:06:23.291 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:06:23.291 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:06:23.292 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:23.292 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.292 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:23.292 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6800", 2022-06-29T16:06:23.292 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:06:23.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.293 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:23.293 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6801", 2022-06-29T16:06:23.293 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:06:23.294 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.294 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.294 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.294 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:06:23.295 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:23.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.295 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:23.295 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6802", 2022-06-29T16:06:23.295 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:06:23.296 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.296 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.296 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:23.296 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6803", 2022-06-29T16:06:23.296 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:06:23.297 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.297 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.297 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:06:23.298 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:23.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.298 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:23.298 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6806", 2022-06-29T16:06:23.298 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:06:23.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.299 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:23.299 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6807", 2022-06-29T16:06:23.299 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:06:23.300 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.300 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.300 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:06:23.301 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:23.301 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.301 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:23.301 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6804", 2022-06-29T16:06:23.301 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:06:23.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.302 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.302 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:23.302 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6805", 2022-06-29T16:06:23.302 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:06:23.303 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.303 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.303 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6801/12889", 2022-06-29T16:06:23.304 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6803/12889", 2022-06-29T16:06:23.304 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6807/12889", 2022-06-29T16:06:23.304 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6805/12889", 2022-06-29T16:06:23.304 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:06:23.304 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:06:23.305 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:06:23.305 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.305 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:06:23.306 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "3d234db9-87cd-4fe9-814f-0240a287d8b4", 2022-06-29T16:06:23.306 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:06:23.306 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:06:23.306 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:06:23.307 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:06:23.307 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:06:23.307 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:06:23.307 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:06:23.307 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 97, 2022-06-29T16:06:23.308 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:06:23.308 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:06:23.308 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:06:23.308 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:23.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.309 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:23.309 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6808", 2022-06-29T16:06:23.309 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:06:23.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.310 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:23.310 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6810", 2022-06-29T16:06:23.310 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:06:23.310 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.311 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.311 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:06:23.311 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:23.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.312 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:23.312 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6812", 2022-06-29T16:06:23.312 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:06:23.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.313 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:23.313 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6814", 2022-06-29T16:06:23.313 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:06:23.313 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.314 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.314 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:06:23.314 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:23.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.315 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:23.315 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6823", 2022-06-29T16:06:23.315 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:06:23.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.316 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:23.316 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6825", 2022-06-29T16:06:23.316 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:06:23.316 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.317 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.317 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:06:23.317 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:23.318 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.318 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:23.318 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6817", 2022-06-29T16:06:23.318 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:06:23.318 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.319 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.319 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:23.319 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6820", 2022-06-29T16:06:23.319 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:06:23.319 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.320 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.320 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6810/13249", 2022-06-29T16:06:23.320 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6814/13249", 2022-06-29T16:06:23.321 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6825/13249", 2022-06-29T16:06:23.321 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6820/13249", 2022-06-29T16:06:23.321 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:06:23.321 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:06:23.321 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:06:23.322 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.322 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.322 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:06:23.323 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "e4bf1104-ff5c-438b-b608-8cee31c163f6", 2022-06-29T16:06:23.323 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:06:23.323 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:06:23.323 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:06:23.323 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:06:23.324 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:06:23.324 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:06:23.324 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:06:23.324 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 97, 2022-06-29T16:06:23.324 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:06:23.325 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:06:23.325 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:06:23.325 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:23.325 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.325 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:23.326 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6809", 2022-06-29T16:06:23.326 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:06:23.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.327 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:23.327 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6811", 2022-06-29T16:06:23.327 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:06:23.327 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.327 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.328 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.328 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:06:23.328 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:23.328 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.328 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:23.329 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6813", 2022-06-29T16:06:23.329 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:06:23.329 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.329 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.329 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:23.330 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6815", 2022-06-29T16:06:23.330 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:06:23.330 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.330 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.331 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.331 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:06:23.331 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:23.331 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.331 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:23.332 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6824", 2022-06-29T16:06:23.332 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:06:23.332 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.332 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.332 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:23.333 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6827", 2022-06-29T16:06:23.333 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:06:23.333 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.333 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.334 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.334 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:06:23.334 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:23.334 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.334 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:23.335 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6818", 2022-06-29T16:06:23.335 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:06:23.335 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.335 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.335 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:23.336 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6821", 2022-06-29T16:06:23.336 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:06:23.336 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.336 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.336 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.337 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6811/13248", 2022-06-29T16:06:23.337 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6815/13248", 2022-06-29T16:06:23.337 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6827/13248", 2022-06-29T16:06:23.337 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6821/13248", 2022-06-29T16:06:23.338 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:06:23.338 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:06:23.338 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:06:23.338 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.338 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.339 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.339 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:06:23.339 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "8564dd25-27ab-4203-b54e-af6d3aa05857", 2022-06-29T16:06:23.339 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:06:23.339 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:06:23.340 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:06:23.340 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:06:23.340 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:06:23.340 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:06:23.341 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:06:23.341 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 98, 2022-06-29T16:06:23.341 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:06:23.341 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:06:23.341 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:06:23.342 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:23.342 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.342 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:23.342 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6816", 2022-06-29T16:06:23.342 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:06:23.343 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.343 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.343 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:23.343 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6819", 2022-06-29T16:06:23.343 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:06:23.344 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.344 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.344 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.344 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:06:23.345 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:23.345 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.345 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:23.345 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6822", 2022-06-29T16:06:23.345 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:06:23.346 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.346 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.346 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:23.346 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6826", 2022-06-29T16:06:23.346 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:06:23.347 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.347 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.347 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.347 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:06:23.347 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:23.348 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.348 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:23.348 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6830", 2022-06-29T16:06:23.348 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:06:23.349 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.349 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.349 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:23.349 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6831", 2022-06-29T16:06:23.349 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:06:23.350 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.350 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.350 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.350 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:06:23.351 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:23.351 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.351 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:23.351 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6828", 2022-06-29T16:06:23.351 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:06:23.352 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.352 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.352 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:23.352 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6829", 2022-06-29T16:06:23.352 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:06:23.353 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.353 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.353 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.353 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6819/13250", 2022-06-29T16:06:23.353 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6826/13250", 2022-06-29T16:06:23.354 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6831/13250", 2022-06-29T16:06:23.354 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6829/13250", 2022-06-29T16:06:23.354 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:06:23.354 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:06:23.355 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:06:23.355 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.355 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.355 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.355 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:06:23.356 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "bd513a9c-3b19-4314-9897-d300b5c897bb", 2022-06-29T16:06:23.356 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:06:23.356 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:06:23.356 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:06:23.357 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:06:23.357 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:06:23.357 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:06:23.357 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:06:23.358 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 98, 2022-06-29T16:06:23.358 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:06:23.358 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:06:23.358 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:06:23.358 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:23.359 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.359 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:23.359 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6800", 2022-06-29T16:06:23.359 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:06:23.359 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.360 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.360 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:23.360 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6801", 2022-06-29T16:06:23.360 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:06:23.360 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.361 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.361 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.361 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:06:23.361 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:23.362 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.362 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:23.362 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6802", 2022-06-29T16:06:23.362 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:06:23.362 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.363 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.363 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:23.363 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6803", 2022-06-29T16:06:23.363 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:06:23.363 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.364 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.364 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.364 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:06:23.364 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:23.364 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.365 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:23.365 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6806", 2022-06-29T16:06:23.365 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:06:23.365 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.366 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.366 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:23.366 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6807", 2022-06-29T16:06:23.366 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:06:23.366 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.367 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.367 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.367 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:06:23.367 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:06:23.367 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.368 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:06:23.368 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6804", 2022-06-29T16:06:23.368 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:06:23.368 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.369 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.369 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:06:23.369 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6805", 2022-06-29T16:06:23.369 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:06:23.369 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.370 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.370 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.370 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6801/13247", 2022-06-29T16:06:23.370 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6803/13247", 2022-06-29T16:06:23.371 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6807/13247", 2022-06-29T16:06:23.371 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6805/13247", 2022-06-29T16:06:23.371 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:06:23.371 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:06:23.371 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:06:23.372 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:23.372 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.372 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:06:23.372 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_xinfo": [ 2022-06-29T16:06:23.373 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.373 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:06:23.373 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:06:23.373 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:06:23.374 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:06:23.374 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:06:23.374 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:06:23.374 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.891342+0000", 2022-06-29T16:06:23.374 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:06:23.375 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.375 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.375 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:06:23.375 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:06:23.376 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:06:23.376 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:06:23.376 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:06:23.376 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:06:23.376 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.900866+0000", 2022-06-29T16:06:23.377 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:06:23.377 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.377 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.377 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:06:23.377 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:06:23.378 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:06:23.378 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:06:23.378 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:06:23.378 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:06:23.378 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.912755+0000", 2022-06-29T16:06:23.379 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:06:23.379 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.379 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.379 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:06:23.380 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:06:23.380 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:06:23.380 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:06:23.380 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:06:23.380 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:06:23.381 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.864332+0000", 2022-06-29T16:06:23.381 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:06:23.381 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.381 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.381 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:06:23.382 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:06:23.382 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:06:23.382 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:06:23.382 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:06:23.383 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:06:23.383 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.863756+0000", 2022-06-29T16:06:23.383 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:06:23.383 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.383 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.384 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:06:23.384 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:06:23.384 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:06:23.384 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:06:23.385 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:06:23.385 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:06:23.385 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.869041+0000", 2022-06-29T16:06:23.385 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:06:23.385 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.386 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.386 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:06:23.386 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:06:23.386 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:06:23.387 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:06:23.387 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:06:23.387 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:06:23.387 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.895043+0000", 2022-06-29T16:06:23.387 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:06:23.388 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.388 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:23.388 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:06:23.388 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:06:23.389 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:06:23.389 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:06:23.389 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:06:23.389 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:06:23.389 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.889491+0000", 2022-06-29T16:06:23.390 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:06:23.390 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.390 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:06:23.390 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap": [], 2022-06-29T16:06:23.390 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap_items": [], 2022-06-29T16:06:23.391 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_temp": [], 2022-06-29T16:06:23.391 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_temp": [], 2022-06-29T16:06:23.391 INFO:teuthology.orchestra.run.smithi060.stdout: "blacklist": { 2022-06-29T16:06:23.391 INFO:teuthology.orchestra.run.smithi060.stdout: "172.21.15.92:6835/1551394182": "2022-06-30T16:06:16.876469+0000", 2022-06-29T16:06:23.392 INFO:teuthology.orchestra.run.smithi060.stdout: "172.21.15.92:6834/1551394182": "2022-06-30T16:06:16.876469+0000" 2022-06-29T16:06:23.392 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.392 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profiles": { 2022-06-29T16:06:23.392 INFO:teuthology.orchestra.run.smithi060.stdout: "default": { 2022-06-29T16:06:23.392 INFO:teuthology.orchestra.run.smithi060.stdout: "crush-failure-domain": "osd", 2022-06-29T16:06:23.393 INFO:teuthology.orchestra.run.smithi060.stdout: "k": "2", 2022-06-29T16:06:23.393 INFO:teuthology.orchestra.run.smithi060.stdout: "m": "1", 2022-06-29T16:06:23.393 INFO:teuthology.orchestra.run.smithi060.stdout: "plugin": "jerasure", 2022-06-29T16:06:23.393 INFO:teuthology.orchestra.run.smithi060.stdout: "ruleset-failure-domain": "osd", 2022-06-29T16:06:23.393 INFO:teuthology.orchestra.run.smithi060.stdout: "technique": "reed_sol_van" 2022-06-29T16:06:23.394 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:23.394 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:23.394 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps_queue": [], 2022-06-29T16:06:23.394 INFO:teuthology.orchestra.run.smithi060.stdout: "new_removed_snaps": [], 2022-06-29T16:06:23.394 INFO:teuthology.orchestra.run.smithi060.stdout: "new_purged_snaps": [], 2022-06-29T16:06:23.395 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_node_flags": {}, 2022-06-29T16:06:23.395 INFO:teuthology.orchestra.run.smithi060.stdout: "device_class_flags": {} 2022-06-29T16:06:23.395 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:06:23.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.193+0000 7fa0f5054700 1 -- 172.21.15.60:0/2372341941 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa0d4053a70 msgr2=0x7fa0d4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:23.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.193+0000 7fa0f5054700 1 --2- 172.21.15.60:0/2372341941 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa0d4053a70 0x7fa0d4055f30 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0x7fa0d800d000 tx=0x7fa0d8007a00).stop 2022-06-29T16:06:23.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.193+0000 7fa0f5054700 1 -- 172.21.15.60:0/2372341941 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa0f00b5950 msgr2=0x7fa0f014cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:23.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.193+0000 7fa0f5054700 1 --2- 172.21.15.60:0/2372341941 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa0f00b5950 0x7fa0f014cc90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fa0e0002f70 tx=0x7fa0e0014000).stop 2022-06-29T16:06:23.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.197+0000 7fa0f5054700 1 -- 172.21.15.60:0/2372341941 shutdown_connections 2022-06-29T16:06:23.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.197+0000 7fa0f5054700 1 --2- 172.21.15.60:0/2372341941 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fa0d4053a70 0x7fa0d4055f30 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:23.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.197+0000 7fa0f5054700 1 --2- 172.21.15.60:0/2372341941 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa0f014af10 0x7fa0f00af4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:23.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.197+0000 7fa0f5054700 1 --2- 172.21.15.60:0/2372341941 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fa0f00b5950 0x7fa0f014cc90 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:23.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.197+0000 7fa0f5054700 1 --2- 172.21.15.60:0/2372341941 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0f0063ad0 0x7fa0f00aef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:23.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.197+0000 7fa0f5054700 1 -- 172.21.15.60:0/2372341941 >> 172.21.15.60:0/2372341941 conn(0x7fa0f00632c0 msgr2=0x7fa0f014a330 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:23.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.197+0000 7fa0f5054700 1 -- 172.21.15.60:0/2372341941 shutdown_connections 2022-06-29T16:06:23.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.197+0000 7fa0f5054700 1 -- 172.21.15.60:0/2372341941 wait complete. 2022-06-29T16:06:23.401 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:06:23.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.509+0000 7fea61d95700 1 Processor -- start 2022-06-29T16:06:23.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.509+0000 7fea61d95700 1 -- start start 2022-06-29T16:06:23.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.509+0000 7fea61d95700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fea5c05c390 0x7fea5c0b2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:23.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.509+0000 7fea61d95700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fea5c1442b0 0x7fea5c144710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:23.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.509+0000 7fea61d95700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea5c05c770 con 0x7fea5c140a20 2022-06-29T16:06:23.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.509+0000 7fea61d95700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea5c140750 con 0x7fea5c05c390 2022-06-29T16:06:23.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.509+0000 7fea61d95700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea5c143e80 con 0x7fea5c1442b0 2022-06-29T16:06:23.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.509+0000 7fea5bfff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fea5c140a20 0x7fea5c140e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59952/0 (socket says 172.21.15.60:59952) 2022-06-29T16:06:23.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.509+0000 7fea5bfff700 1 -- 172.21.15.60:0/4226976652 learned_addr learned my addr 172.21.15.60:0/4226976652 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:23.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.509+0000 7fea52ffd700 1 --2- 172.21.15.60:0/4226976652 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fea5c1442b0 0x7fea5c144710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:23.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.509+0000 7fea5b7fe700 1 --2- 172.21.15.60:0/4226976652 >> v2:172.21.15.92:3300/0 conn(0x7fea5c05c390 0x7fea5c0b2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:23.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.509+0000 7fea5affd700 1 -- 172.21.15.60:0/4226976652 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 570830232 0 0) 0x7fea5c05c770 con 0x7fea5c140a20 2022-06-29T16:06:23.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.509+0000 7fea5affd700 1 -- 172.21.15.60:0/4226976652 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea3c003260 con 0x7fea5c140a20 2022-06-29T16:06:23.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.509+0000 7fea52ffd700 1 -- 172.21.15.60:0/4226976652 >> v2:172.21.15.92:3300/0 conn(0x7fea5c05c390 msgr2=0x7fea5c0b2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:23.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.509+0000 7fea52ffd700 1 --2- 172.21.15.60:0/4226976652 >> v2:172.21.15.92:3300/0 conn(0x7fea5c05c390 0x7fea5c0b2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:23.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.509+0000 7fea52ffd700 1 -- 172.21.15.60:0/4226976652 >> v1:172.21.15.60:6789/0 conn(0x7fea5c140a20 legacy=0x7fea5c140e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:23.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.509+0000 7fea52ffd700 1 -- 172.21.15.60:0/4226976652 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea5c065f40 con 0x7fea5c1442b0 2022-06-29T16:06:23.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.509+0000 7fea5b7fe700 1 --2- 172.21.15.60:0/4226976652 >> v2:172.21.15.92:3300/0 conn(0x7fea5c05c390 0x7fea5c0b2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:06:23.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.509+0000 7fea52ffd700 1 --2- 172.21.15.60:0/4226976652 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fea5c1442b0 0x7fea5c144710 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fea44002f70 tx=0x7fea44006df0).ready entity=mon.2 client_cookie=1ebe0639201d9a4f server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:23.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.509+0000 7fea5affd700 1 -- 172.21.15.60:0/4226976652 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea4400d4a0 con 0x7fea5c1442b0 2022-06-29T16:06:23.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.509+0000 7fea5affd700 1 -- 172.21.15.60:0/4226976652 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea44000e90 con 0x7fea5c1442b0 2022-06-29T16:06:23.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.509+0000 7fea5affd700 1 -- 172.21.15.60:0/4226976652 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea4400d7f0 con 0x7fea5c1442b0 2022-06-29T16:06:23.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.509+0000 7fea61d95700 1 -- 172.21.15.60:0/4226976652 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fea5c1442b0 msgr2=0x7fea5c144710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:23.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.509+0000 7fea61d95700 1 --2- 172.21.15.60:0/4226976652 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fea5c1442b0 0x7fea5c144710 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fea44002f70 tx=0x7fea44006df0).stop 2022-06-29T16:06:23.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.509+0000 7fea61d95700 1 -- 172.21.15.60:0/4226976652 shutdown_connections 2022-06-29T16:06:23.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.509+0000 7fea61d95700 1 --2- 172.21.15.60:0/4226976652 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fea5c1442b0 0x7fea5c144710 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:23.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.509+0000 7fea61d95700 1 --2- 172.21.15.60:0/4226976652 >> v2:172.21.15.92:3300/0 conn(0x7fea5c05c390 0x7fea5c0b2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:23.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.509+0000 7fea61d95700 1 -- 172.21.15.60:0/4226976652 >> 172.21.15.60:0/4226976652 conn(0x7fea5c064a10 msgr2=0x7fea5c064df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:23.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.509+0000 7fea61d95700 1 -- 172.21.15.60:0/4226976652 shutdown_connections 2022-06-29T16:06:23.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.509+0000 7fea61d95700 1 -- 172.21.15.60:0/4226976652 wait complete. 2022-06-29T16:06:23.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.513+0000 7fea61d95700 1 Processor -- start 2022-06-29T16:06:23.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.513+0000 7fea61d95700 1 -- start start 2022-06-29T16:06:23.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.513+0000 7fea61d95700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea5c05c390 0x7fea5c0615b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:23.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.513+0000 7fea61d95700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea5c140a20 0x7fea5c145ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:23.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.513+0000 7fea61d95700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fea5c1442b0 0x7fea5c1463d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:23.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.513+0000 7fea61d95700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea5c145a00 con 0x7fea5c140a20 2022-06-29T16:06:23.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.513+0000 7fea61d95700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea5c13fcb0 con 0x7fea5c05c390 2022-06-29T16:06:23.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.513+0000 7fea61d95700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea5c146b00 con 0x7fea5c1442b0 2022-06-29T16:06:23.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.513+0000 7fea52ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea5c140a20 0x7fea5c145ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:23.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.513+0000 7fea5b7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea5c05c390 0x7fea5c0615b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:23.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.513+0000 7fea52ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea5c140a20 0x7fea5c145ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51870/0 (socket says 172.21.15.60:51870) 2022-06-29T16:06:23.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.513+0000 7fea52ffd700 1 -- 172.21.15.60:0/2100408611 learned_addr learned my addr 172.21.15.60:0/2100408611 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:23.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.513+0000 7fea5bfff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fea5c1442b0 0x7fea5c1463d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:23.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.513+0000 7fea52ffd700 1 -- 172.21.15.60:0/2100408611 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fea5c1442b0 msgr2=0x7fea5c1463d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:23.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.513+0000 7fea52ffd700 1 --2- 172.21.15.60:0/2100408611 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fea5c1442b0 0x7fea5c1463d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:23.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.513+0000 7fea52ffd700 1 -- 172.21.15.60:0/2100408611 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea5c05c390 msgr2=0x7fea5c0615b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:23.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.513+0000 7fea52ffd700 1 --2- 172.21.15.60:0/2100408611 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea5c05c390 0x7fea5c0615b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:23.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.513+0000 7fea52ffd700 1 -- 172.21.15.60:0/2100408611 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea4c00b180 con 0x7fea5c140a20 2022-06-29T16:06:23.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.513+0000 7fea52ffd700 1 --2- 172.21.15.60:0/2100408611 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea5c140a20 0x7fea5c145ee0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 rx=0x7fea44002f70 tx=0x7fea44014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:23.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.513+0000 7fea597fa700 1 -- 172.21.15.60:0/2100408611 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea44012930 con 0x7fea5c140a20 2022-06-29T16:06:23.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.513+0000 7fea597fa700 1 -- 172.21.15.60:0/2100408611 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea44012a90 con 0x7fea5c140a20 2022-06-29T16:06:23.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.513+0000 7fea61d95700 1 -- 172.21.15.60:0/2100408611 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea4400b180 con 0x7fea5c140a20 2022-06-29T16:06:23.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.513+0000 7fea597fa700 1 -- 172.21.15.60:0/2100408611 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea44012df0 con 0x7fea5c140a20 2022-06-29T16:06:23.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.513+0000 7fea61d95700 1 -- 172.21.15.60:0/2100408611 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea5c145560 con 0x7fea5c140a20 2022-06-29T16:06:23.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.517+0000 7fea61d95700 1 -- 172.21.15.60:0/2100408611 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea5c145720 con 0x7fea5c140a20 2022-06-29T16:06:23.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.517+0000 7fea597fa700 1 -- 172.21.15.60:0/2100408611 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fea4401c070 con 0x7fea5c140a20 2022-06-29T16:06:23.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.517+0000 7fea597fa700 1 --2- 172.21.15.60:0/2100408611 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fea40053a70 0x7fea40055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:23.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.517+0000 7fea597fa700 1 -- 172.21.15.60:0/2100408611 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fea4406c790 con 0x7fea5c140a20 2022-06-29T16:06:23.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.517+0000 7fea5b7fe700 1 --2- 172.21.15.60:0/2100408611 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fea40053a70 0x7fea40055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:23.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.517+0000 7fea5b7fe700 1 --2- 172.21.15.60:0/2100408611 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fea40053a70 0x7fea40055f30 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7fea4c00b000 tx=0x7fea4c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:23.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.521+0000 7fea597fa700 1 -- 172.21.15.60:0/2100408611 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fea44043930 con 0x7fea5c140a20 2022-06-29T16:06:23.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.685+0000 7fea61d95700 1 -- 172.21.15.60:0/2100408611 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea5c0573d0 con 0x7fea5c140a20 2022-06-29T16:06:23.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.685+0000 7fea597fa700 1 -- 172.21.15.60:0/2100408611 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+1206 (secure 0 0 0) 0x7fea44019030 con 0x7fea5c140a20 2022-06-29T16:06:23.691 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:23.692 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":48,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7320,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/3498160620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3498160620},{"type":"v1","addr":"172.21.15.60:6833","nonce":3498160620}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":44},{"gid":7567,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/573625482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":573625482},{"type":"v1","addr":"172.21.15.92:6835","nonce":573625482}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":48}],"filesystems":[]} 2022-06-29T16:06:23.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.689+0000 7fea61d95700 1 -- 172.21.15.60:0/2100408611 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fea40053a70 msgr2=0x7fea40055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:23.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.689+0000 7fea61d95700 1 --2- 172.21.15.60:0/2100408611 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fea40053a70 0x7fea40055f30 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7fea4c00b000 tx=0x7fea4c006df0).stop 2022-06-29T16:06:23.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.689+0000 7fea61d95700 1 -- 172.21.15.60:0/2100408611 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea5c140a20 msgr2=0x7fea5c145ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:23.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.689+0000 7fea61d95700 1 --2- 172.21.15.60:0/2100408611 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea5c140a20 0x7fea5c145ee0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 rx=0x7fea44002f70 tx=0x7fea44014000).stop 2022-06-29T16:06:23.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.689+0000 7fea61d95700 1 -- 172.21.15.60:0/2100408611 shutdown_connections 2022-06-29T16:06:23.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.689+0000 7fea61d95700 1 --2- 172.21.15.60:0/2100408611 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fea40053a70 0x7fea40055f30 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:23.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.689+0000 7fea61d95700 1 --2- 172.21.15.60:0/2100408611 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fea5c1442b0 0x7fea5c1463d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:23.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.689+0000 7fea61d95700 1 --2- 172.21.15.60:0/2100408611 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea5c140a20 0x7fea5c145ee0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:23.696 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.689+0000 7fea61d95700 1 --2- 172.21.15.60:0/2100408611 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea5c05c390 0x7fea5c0615b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:23.696 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.689+0000 7fea61d95700 1 -- 172.21.15.60:0/2100408611 >> 172.21.15.60:0/2100408611 conn(0x7fea5c064a10 msgr2=0x7fea5c0b70c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:23.696 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.689+0000 7fea61d95700 1 -- 172.21.15.60:0/2100408611 shutdown_connections 2022-06-29T16:06:23.696 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.689+0000 7fea61d95700 1 -- 172.21.15.60:0/2100408611 wait complete. 2022-06-29T16:06:23.697 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 48 2022-06-29T16:06:23.704 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:06:23.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.801+0000 7f32959b3700 1 Processor -- start 2022-06-29T16:06:23.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.801+0000 7f32959b3700 1 -- start start 2022-06-29T16:06:23.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.805+0000 7f32959b3700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3290060260 0x7f32900b6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:23.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.805+0000 7f32959b3700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f32900b6c00 0x7f329013f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:23.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.805+0000 7f32959b3700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32901430e0 con 0x7f329013f3d0 2022-06-29T16:06:23.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.805+0000 7f32959b3700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3290142ef0 con 0x7f32900b6c00 2022-06-29T16:06:23.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.805+0000 7f32959b3700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32900acb90 con 0x7f3290060260 2022-06-29T16:06:23.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.805+0000 7f328e7fc700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3290060260 0x7f32900b6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:23.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.805+0000 7f328e7fc700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3290060260 0x7f32900b6710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:36740/0 (socket says 172.21.15.60:36740) 2022-06-29T16:06:23.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.805+0000 7f328e7fc700 1 -- 172.21.15.60:0/2991043093 learned_addr learned my addr 172.21.15.60:0/2991043093 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:23.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.805+0000 7f328f7fe700 1 --2- 172.21.15.60:0/2991043093 >> v2:172.21.15.92:3300/0 conn(0x7f32900b6c00 0x7f329013f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:23.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.805+0000 7f328f7fe700 1 -- 172.21.15.60:0/2991043093 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3290060260 msgr2=0x7f32900b6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:23.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.805+0000 7f328f7fe700 1 --2- 172.21.15.60:0/2991043093 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3290060260 0x7f32900b6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:23.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.805+0000 7f328f7fe700 1 -- 172.21.15.60:0/2991043093 >> v1:172.21.15.92:6789/0 conn(0x7f329013f3d0 legacy=0x7f3290143820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:23.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.805+0000 7f328f7fe700 1 -- 172.21.15.60:0/2991043093 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3290143380 con 0x7f32900b6c00 2022-06-29T16:06:23.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.805+0000 7f328e7fc700 1 --2- 172.21.15.60:0/2991043093 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3290060260 0x7f32900b6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:06:23.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.805+0000 7f328f7fe700 1 --2- 172.21.15.60:0/2991043093 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f32900b6c00 0x7f329013f7b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7f3288002f70 tx=0x7f3288006df0).ready entity=mon.1 client_cookie=d57f86d3f8dee0c4 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:23.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.805+0000 7f328dffb700 1 -- 172.21.15.60:0/2991043093 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f328800d570 con 0x7f32900b6c00 2022-06-29T16:06:23.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.805+0000 7f328dffb700 1 -- 172.21.15.60:0/2991043093 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3288000e90 con 0x7f32900b6c00 2022-06-29T16:06:23.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.805+0000 7f328dffb700 1 -- 172.21.15.60:0/2991043093 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f328800d8c0 con 0x7f32900b6c00 2022-06-29T16:06:23.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.805+0000 7f32959b3700 1 -- 172.21.15.60:0/2991043093 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f32900b6c00 msgr2=0x7f329013f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:23.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.805+0000 7f32959b3700 1 --2- 172.21.15.60:0/2991043093 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f32900b6c00 0x7f329013f7b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7f3288002f70 tx=0x7f3288006df0).stop 2022-06-29T16:06:23.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.805+0000 7f32959b3700 1 -- 172.21.15.60:0/2991043093 shutdown_connections 2022-06-29T16:06:23.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.805+0000 7f32959b3700 1 --2- 172.21.15.60:0/2991043093 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f32900b6c00 0x7f329013f7b0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:23.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.805+0000 7f32959b3700 1 --2- 172.21.15.60:0/2991043093 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3290060260 0x7f32900b6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:23.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.805+0000 7f32959b3700 1 -- 172.21.15.60:0/2991043093 >> 172.21.15.60:0/2991043093 conn(0x7f32900643e0 msgr2=0x7f32900b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:23.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.805+0000 7f32959b3700 1 -- 172.21.15.60:0/2991043093 shutdown_connections 2022-06-29T16:06:23.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.805+0000 7f32959b3700 1 -- 172.21.15.60:0/2991043093 wait complete. 2022-06-29T16:06:23.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.805+0000 7f32959b3700 1 Processor -- start 2022-06-29T16:06:23.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.805+0000 7f32959b3700 1 -- start start 2022-06-29T16:06:23.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.809+0000 7f32959b3700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3290060260 0x7f329012f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:23.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.809+0000 7f32959b3700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f32900b6c00 0x7f3290099410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:23.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.809+0000 7f32959b3700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f329013f3d0 0x7f3290133470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:23.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.809+0000 7f32959b3700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32901429d0 con 0x7f329013f3d0 2022-06-29T16:06:23.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.809+0000 7f32959b3700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f329012f7e0 con 0x7f3290060260 2022-06-29T16:06:23.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.809+0000 7f32959b3700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f329009be50 con 0x7f32900b6c00 2022-06-29T16:06:23.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.809+0000 7f328f7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f329013f3d0 0x7f3290133470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:23.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.809+0000 7f328f7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f329013f3d0 0x7f3290133470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51886/0 (socket says 172.21.15.60:51886) 2022-06-29T16:06:23.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.809+0000 7f328f7fe700 1 -- 172.21.15.60:0/874067137 learned_addr learned my addr 172.21.15.60:0/874067137 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:23.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.809+0000 7f328e7fc700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f32900b6c00 0x7f3290099410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:23.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.809+0000 7f328effd700 1 --2- 172.21.15.60:0/874067137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3290060260 0x7f329012f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:23.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.809+0000 7f328f7fe700 1 -- 172.21.15.60:0/874067137 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f32900b6c00 msgr2=0x7f3290099410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:23.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.809+0000 7f328f7fe700 1 --2- 172.21.15.60:0/874067137 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f32900b6c00 0x7f3290099410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:23.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.809+0000 7f328f7fe700 1 -- 172.21.15.60:0/874067137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3290060260 msgr2=0x7f329012f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:23.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.809+0000 7f328f7fe700 1 --2- 172.21.15.60:0/874067137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3290060260 0x7f329012f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:23.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.809+0000 7f328f7fe700 1 -- 172.21.15.60:0/874067137 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f327800b1e0 con 0x7f329013f3d0 2022-06-29T16:06:23.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.809+0000 7f328f7fe700 1 --2- 172.21.15.60:0/874067137 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f329013f3d0 0x7f3290133470 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 rx=0x7f329009a880 tx=0x7f3288014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:23.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.809+0000 7f3287fff700 1 -- 172.21.15.60:0/874067137 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3288013bf0 con 0x7f329013f3d0 2022-06-29T16:06:23.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.809+0000 7f3287fff700 1 -- 172.21.15.60:0/874067137 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3288013d50 con 0x7f329013f3d0 2022-06-29T16:06:23.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.809+0000 7f3287fff700 1 -- 172.21.15.60:0/874067137 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f328801e5f0 con 0x7f329013f3d0 2022-06-29T16:06:23.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.809+0000 7f32959b3700 1 -- 172.21.15.60:0/874067137 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f328800b180 con 0x7f329013f3d0 2022-06-29T16:06:23.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.809+0000 7f32959b3700 1 -- 172.21.15.60:0/874067137 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3290064d00 con 0x7f329013f3d0 2022-06-29T16:06:23.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.809+0000 7f32959b3700 1 -- 172.21.15.60:0/874067137 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3290142b40 con 0x7f329013f3d0 2022-06-29T16:06:23.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.813+0000 7f3287fff700 1 -- 172.21.15.60:0/874067137 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3288025030 con 0x7f329013f3d0 2022-06-29T16:06:23.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.813+0000 7f3287fff700 1 --2- 172.21.15.60:0/874067137 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3274053a70 0x7f3274055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:23.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.813+0000 7f3287fff700 1 -- 172.21.15.60:0/874067137 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f328806cb00 con 0x7f329013f3d0 2022-06-29T16:06:23.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.813+0000 7f328effd700 1 --2- 172.21.15.60:0/874067137 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3274053a70 0x7f3274055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:23.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.813+0000 7f328effd700 1 --2- 172.21.15.60:0/874067137 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3274053a70 0x7f3274055f30 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7f328000b000 tx=0x7f3280009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:23.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.817+0000 7f3287fff700 1 -- 172.21.15.60:0/874067137 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3290142b40 con 0x7f329013f3d0 2022-06-29T16:06:23.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.981+0000 7f32959b3700 1 -- 172.21.15.60:0/874067137 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3290099aa0 con 0x7f329013f3d0 2022-06-29T16:06:23.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.985+0000 7f3287fff700 1 -- 172.21.15.60:0/874067137 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+1206 (secure 0 0 0) 0x7f32880234d0 con 0x7f329013f3d0 2022-06-29T16:06:23.989 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:23.990 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":48,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7320,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6833/3498160620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3498160620},{"type":"v1","addr":"172.21.15.60:6833","nonce":3498160620}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":44},{"gid":7567,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/573625482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":573625482},{"type":"v1","addr":"172.21.15.92:6835","nonce":573625482}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":48}],"filesystems":[]} 2022-06-29T16:06:23.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.985+0000 7f32959b3700 1 -- 172.21.15.60:0/874067137 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3274053a70 msgr2=0x7f3274055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:23.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.985+0000 7f32959b3700 1 --2- 172.21.15.60:0/874067137 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3274053a70 0x7f3274055f30 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7f328000b000 tx=0x7f3280009040).stop 2022-06-29T16:06:23.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.985+0000 7f32959b3700 1 -- 172.21.15.60:0/874067137 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f329013f3d0 msgr2=0x7f3290133470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:23.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.985+0000 7f32959b3700 1 --2- 172.21.15.60:0/874067137 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f329013f3d0 0x7f3290133470 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 rx=0x7f329009a880 tx=0x7f3288014000).stop 2022-06-29T16:06:23.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.985+0000 7f32959b3700 1 -- 172.21.15.60:0/874067137 shutdown_connections 2022-06-29T16:06:23.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.985+0000 7f32959b3700 1 --2- 172.21.15.60:0/874067137 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3274053a70 0x7f3274055f30 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:23.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.985+0000 7f32959b3700 1 --2- 172.21.15.60:0/874067137 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f329013f3d0 0x7f3290133470 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:23.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.985+0000 7f32959b3700 1 --2- 172.21.15.60:0/874067137 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f32900b6c00 0x7f3290099410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:23.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.985+0000 7f32959b3700 1 --2- 172.21.15.60:0/874067137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3290060260 0x7f329012f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:23.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.985+0000 7f32959b3700 1 -- 172.21.15.60:0/874067137 >> 172.21.15.60:0/874067137 conn(0x7f32900643e0 msgr2=0x7f32900b5f90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:23.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.985+0000 7f32959b3700 1 -- 172.21.15.60:0/874067137 shutdown_connections 2022-06-29T16:06:23.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:23.985+0000 7f32959b3700 1 -- 172.21.15.60:0/874067137 wait complete. 2022-06-29T16:06:23.995 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 48 2022-06-29T16:06:24.003 INFO:tasks.ceph.mds.a:Restarting daemon 2022-06-29T16:06:24.004 INFO:tasks.ceph.mds.a:Stopping old one... 2022-06-29T16:06:24.004 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-06-29T16:06:24.004 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T16:06:24.005 INFO:tasks.ceph.mds.b:Restarting daemon 2022-06-29T16:06:24.005 INFO:tasks.ceph.mds.b:Stopping old one... 2022-06-29T16:06:24.005 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-06-29T16:06:24.006 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T16:06:24.019 INFO:tasks.ceph.mds.a:Stopped 2022-06-29T16:06:24.020 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-06-29T16:06:24.022 INFO:tasks.ceph.mds.b:Stopped 2022-06-29T16:06:24.022 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-06-29T16:06:24.023 INFO:tasks.ceph.mds.a:Started 2022-06-29T16:06:24.024 DEBUG:teuthology.parallel:result is None 2022-06-29T16:06:24.026 INFO:tasks.ceph.mds.b:Started 2022-06-29T16:06:24.027 DEBUG:teuthology.parallel:result is None 2022-06-29T16:06:24.028 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-06-29T16:06:24.099 INFO:tasks.ceph.mds.b.smithi092.stdout:starting mds.b at 2022-06-29T16:06:24.108 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2022-06-29T16:06:24.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.129+0000 7f5a7359e700 1 Processor -- start 2022-06-29T16:06:24.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.129+0000 7f5a7359e700 1 -- start start 2022-06-29T16:06:24.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.129+0000 7f5a7359e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5a7405bf30 0x7f5a74143cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:24.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.129+0000 7f5a7359e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5a741441b0 0x7f5a7413fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:24.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.129+0000 7f5a7359e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a74065820 con 0x7f5a740b3c70 2022-06-29T16:06:24.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.129+0000 7f5a7359e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a74066600 con 0x7f5a741441b0 2022-06-29T16:06:24.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.129+0000 7f5a7359e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a740b4050 con 0x7f5a7405bf30 2022-06-29T16:06:24.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.133+0000 7f5a7259c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5a740b3c70 0x7f5a74065ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59988/0 (socket says 172.21.15.60:59988) 2022-06-29T16:06:24.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.133+0000 7f5a7259c700 1 -- 172.21.15.60:0/2778146439 learned_addr learned my addr 172.21.15.60:0/2778146439 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:24.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.133+0000 7f5a72d9d700 1 --2- 172.21.15.60:0/2778146439 >> v2:172.21.15.92:3300/0 conn(0x7f5a741441b0 0x7f5a7413fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:24.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.133+0000 7f5a71d9b700 1 --2- 172.21.15.60:0/2778146439 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5a7405bf30 0x7f5a74143cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:24.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.133+0000 7f5a7159a700 1 -- 172.21.15.60:0/2778146439 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1614450699 0 0) 0x7f5a74065820 con 0x7f5a740b3c70 2022-06-29T16:06:24.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.133+0000 7f5a7159a700 1 -- 172.21.15.60:0/2778146439 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a58003260 con 0x7f5a740b3c70 2022-06-29T16:06:24.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.133+0000 7f5a72d9d700 1 -- 172.21.15.60:0/2778146439 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5a7405bf30 msgr2=0x7f5a74143cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:24.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.133+0000 7f5a72d9d700 1 --2- 172.21.15.60:0/2778146439 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5a7405bf30 0x7f5a74143cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:24.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.133+0000 7f5a72d9d700 1 -- 172.21.15.60:0/2778146439 >> v1:172.21.15.60:6789/0 conn(0x7f5a740b3c70 legacy=0x7f5a74065ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:24.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.133+0000 7f5a7159a700 1 -- 172.21.15.60:0/2778146439 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3650854081 0 0) 0x7f5a58003260 con 0x7f5a740b3c70 2022-06-29T16:06:24.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.133+0000 7f5a72d9d700 1 -- 172.21.15.60:0/2778146439 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a7405c4e0 con 0x7f5a741441b0 2022-06-29T16:06:24.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.133+0000 7f5a72d9d700 1 --2- 172.21.15.60:0/2778146439 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a741441b0 0x7f5a7413fa70 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7f5a64001800 tx=0x7f5a6c0091d0).ready entity=mon.1 client_cookie=979f72a36bc334d5 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:24.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.133+0000 7f5a7159a700 1 -- 172.21.15.60:0/2778146439 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a64001660 con 0x7f5a741441b0 2022-06-29T16:06:24.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.133+0000 7f5a7159a700 1 -- 172.21.15.60:0/2778146439 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a6c00f510 con 0x7f5a741441b0 2022-06-29T16:06:24.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.133+0000 7f5a7159a700 1 -- 172.21.15.60:0/2778146439 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a6c00f680 con 0x7f5a741441b0 2022-06-29T16:06:24.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.133+0000 7f5a7359e700 1 -- 172.21.15.60:0/2778146439 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a741441b0 msgr2=0x7f5a7413fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:24.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.133+0000 7f5a7359e700 1 --2- 172.21.15.60:0/2778146439 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a741441b0 0x7f5a7413fa70 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7f5a64001800 tx=0x7f5a6c0091d0).stop 2022-06-29T16:06:24.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.133+0000 7f5a7359e700 1 -- 172.21.15.60:0/2778146439 shutdown_connections 2022-06-29T16:06:24.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.133+0000 7f5a7359e700 1 --2- 172.21.15.60:0/2778146439 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a741441b0 0x7f5a7413fa70 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:24.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.133+0000 7f5a7359e700 1 --2- 172.21.15.60:0/2778146439 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5a7405bf30 0x7f5a74143cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:24.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.133+0000 7f5a7359e700 1 -- 172.21.15.60:0/2778146439 >> 172.21.15.60:0/2778146439 conn(0x7f5a74064810 msgr2=0x7f5a74064bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:24.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.133+0000 7f5a7359e700 1 -- 172.21.15.60:0/2778146439 shutdown_connections 2022-06-29T16:06:24.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.133+0000 7f5a7359e700 1 -- 172.21.15.60:0/2778146439 wait complete. 2022-06-29T16:06:24.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.133+0000 7f5a7359e700 1 Processor -- start 2022-06-29T16:06:24.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.133+0000 7f5a7359e700 1 -- start start 2022-06-29T16:06:24.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.137+0000 7f5a7359e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a7405bf30 0x7f5a74061640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:24.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.137+0000 7f5a7359e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5a740b3c70 0x7f5a741564a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:24.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.137+0000 7f5a7359e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a741441b0 0x7f5a74156990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:24.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.137+0000 7f5a7359e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a740b1380 con 0x7f5a741441b0 2022-06-29T16:06:24.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.137+0000 7f5a7359e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a74156e80 con 0x7f5a7405bf30 2022-06-29T16:06:24.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.137+0000 7f5a7359e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a740b10e0 con 0x7f5a740b3c70 2022-06-29T16:06:24.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.137+0000 7f5a72d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a741441b0 0x7f5a74156990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:24.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.137+0000 7f5a72d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a741441b0 0x7f5a74156990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51912/0 (socket says 172.21.15.60:51912) 2022-06-29T16:06:24.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.137+0000 7f5a72d9d700 1 -- 172.21.15.60:0/3461630461 learned_addr learned my addr 172.21.15.60:0/3461630461 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:24.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.137+0000 7f5a71d9b700 1 --2- 172.21.15.60:0/3461630461 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5a740b3c70 0x7f5a741564a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:24.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.137+0000 7f5a7259c700 1 --2- 172.21.15.60:0/3461630461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a7405bf30 0x7f5a74061640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:24.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.137+0000 7f5a72d9d700 1 -- 172.21.15.60:0/3461630461 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5a740b3c70 msgr2=0x7f5a741564a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:24.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.137+0000 7f5a72d9d700 1 --2- 172.21.15.60:0/3461630461 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5a740b3c70 0x7f5a741564a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:24.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.137+0000 7f5a72d9d700 1 -- 172.21.15.60:0/3461630461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a7405bf30 msgr2=0x7f5a74061640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:24.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.137+0000 7f5a72d9d700 1 --2- 172.21.15.60:0/3461630461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a7405bf30 0x7f5a74061640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:24.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.137+0000 7f5a72d9d700 1 -- 172.21.15.60:0/3461630461 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a6c009470 con 0x7f5a741441b0 2022-06-29T16:06:24.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.137+0000 7f5a72d9d700 1 --2- 172.21.15.60:0/3461630461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a741441b0 0x7f5a74156990 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 rx=0x7f5a64001800 tx=0x7f5a6c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:24.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.137+0000 7f5a637fe700 1 -- 172.21.15.60:0/3461630461 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a6c0032f0 con 0x7f5a741441b0 2022-06-29T16:06:24.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.137+0000 7f5a637fe700 1 -- 172.21.15.60:0/3461630461 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a6c011c30 con 0x7f5a741441b0 2022-06-29T16:06:24.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.137+0000 7f5a7359e700 1 -- 172.21.15.60:0/3461630461 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a74061b30 con 0x7f5a741441b0 2022-06-29T16:06:24.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.137+0000 7f5a637fe700 1 -- 172.21.15.60:0/3461630461 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a6c021480 con 0x7f5a741441b0 2022-06-29T16:06:24.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.137+0000 7f5a7359e700 1 -- 172.21.15.60:0/3461630461 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a74155e60 con 0x7f5a741441b0 2022-06-29T16:06:24.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.141+0000 7f5a7359e700 1 -- 172.21.15.60:0/3461630461 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a740573d0 con 0x7f5a741441b0 2022-06-29T16:06:24.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.141+0000 7f5a637fe700 1 -- 172.21.15.60:0/3461630461 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5a6c00f070 con 0x7f5a741441b0 2022-06-29T16:06:24.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.141+0000 7f5a637fe700 1 --2- 172.21.15.60:0/3461630461 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5a50053b30 0x7f5a50055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:24.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.141+0000 7f5a637fe700 1 -- 172.21.15.60:0/3461630461 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f5a6c06cef0 con 0x7f5a741441b0 2022-06-29T16:06:24.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.141+0000 7f5a7259c700 1 --2- 172.21.15.60:0/3461630461 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5a50053b30 0x7f5a50055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:24.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.141+0000 7f5a7259c700 1 --2- 172.21.15.60:0/3461630461 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5a50053b30 0x7f5a50055ff0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x7f5a6400d000 tx=0x7f5a64001a80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:24.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.149+0000 7f5a637fe700 1 -- 172.21.15.60:0/3461630461 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5a6c043f00 con 0x7f5a741441b0 2022-06-29T16:06:24.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:24.277+0000 7f5a7359e700 1 -- 172.21.15.60:0/3461630461 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f5a7413e260 con 0x7f5a741441b0 2022-06-29T16:06:25.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.117+0000 7f5a637fe700 1 -- 172.21.15.60:0/3461630461 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v105) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5a6c015070 con 0x7f5a741441b0 2022-06-29T16:06:25.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.121+0000 7f5a7359e700 1 -- 172.21.15.60:0/3461630461 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5a50053b30 msgr2=0x7f5a50055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:25.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.121+0000 7f5a7359e700 1 --2- 172.21.15.60:0/3461630461 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5a50053b30 0x7f5a50055ff0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x7f5a6400d000 tx=0x7f5a64001a80).stop 2022-06-29T16:06:25.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.121+0000 7f5a7359e700 1 -- 172.21.15.60:0/3461630461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a741441b0 msgr2=0x7f5a74156990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:25.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.121+0000 7f5a7359e700 1 --2- 172.21.15.60:0/3461630461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a741441b0 0x7f5a74156990 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 rx=0x7f5a64001800 tx=0x7f5a6c016000).stop 2022-06-29T16:06:25.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.121+0000 7f5a7359e700 1 -- 172.21.15.60:0/3461630461 shutdown_connections 2022-06-29T16:06:25.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.121+0000 7f5a7359e700 1 --2- 172.21.15.60:0/3461630461 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5a50053b30 0x7f5a50055ff0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:25.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.121+0000 7f5a7359e700 1 --2- 172.21.15.60:0/3461630461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a741441b0 0x7f5a74156990 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:25.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.121+0000 7f5a7359e700 1 --2- 172.21.15.60:0/3461630461 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5a740b3c70 0x7f5a741564a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:25.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.121+0000 7f5a7359e700 1 --2- 172.21.15.60:0/3461630461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a7405bf30 0x7f5a74061640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:25.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.121+0000 7f5a7359e700 1 -- 172.21.15.60:0/3461630461 >> 172.21.15.60:0/3461630461 conn(0x7f5a74064810 msgr2=0x7f5a740b19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:25.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.121+0000 7f5a7359e700 1 -- 172.21.15.60:0/3461630461 shutdown_connections 2022-06-29T16:06:25.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.121+0000 7f5a7359e700 1 -- 172.21.15.60:0/3461630461 wait complete. 2022-06-29T16:06:25.129 INFO:teuthology.orchestra.run.smithi060.stderr: removed all blacklist entries 2022-06-29T16:06:25.142 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-06-29T16:06:25.142 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-06-29T16:06:25.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.245+0000 7fdc9ded3700 1 Processor -- start 2022-06-29T16:06:25.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.245+0000 7fdc9ded3700 1 -- start start 2022-06-29T16:06:25.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.245+0000 7fdc9ded3700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdc980b5950 0x7fdc98067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:25.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.245+0000 7fdc9ded3700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc98063ad0 0x7fdc9814af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:25.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.245+0000 7fdc9ded3700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdc9805bd20 0x7fdc9805c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:25.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.245+0000 7fdc9ded3700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc980acba0 con 0x7fdc980b5950 2022-06-29T16:06:25.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.245+0000 7fdc9ded3700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc98067220 con 0x7fdc9805bd20 2022-06-29T16:06:25.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.245+0000 7fdc9ded3700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc980b5d30 con 0x7fdc98063ad0 2022-06-29T16:06:25.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.245+0000 7fdc977fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdc980b5950 0x7fdc98067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:25.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.245+0000 7fdc977fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdc980b5950 0x7fdc98067680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51916/0 (socket says 172.21.15.60:51916) 2022-06-29T16:06:25.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.245+0000 7fdc977fe700 1 -- 172.21.15.60:0/642875106 learned_addr learned my addr 172.21.15.60:0/642875106 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:25.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.249+0000 7fdc96ffd700 1 --2- 172.21.15.60:0/642875106 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc98063ad0 0x7fdc9814af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:25.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.249+0000 7fdc97fff700 1 --2- 172.21.15.60:0/642875106 >> v2:172.21.15.92:3300/0 conn(0x7fdc9805bd20 0x7fdc9805c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:25.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.249+0000 7fdc977fe700 1 -- 172.21.15.60:0/642875106 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc98063ad0 msgr2=0x7fdc9814af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:25.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.249+0000 7fdc977fe700 1 --2- 172.21.15.60:0/642875106 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc98063ad0 0x7fdc9814af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:25.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.249+0000 7fdc977fe700 1 -- 172.21.15.60:0/642875106 >> v2:172.21.15.92:3300/0 conn(0x7fdc9805bd20 msgr2=0x7fdc9805c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:25.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.249+0000 7fdc977fe700 1 --2- 172.21.15.60:0/642875106 >> v2:172.21.15.92:3300/0 conn(0x7fdc9805bd20 0x7fdc9805c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:25.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.249+0000 7fdc977fe700 1 -- 172.21.15.60:0/642875106 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc980ace00 con 0x7fdc980b5950 2022-06-29T16:06:25.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.249+0000 7fdc977fe700 1 --2- 172.21.15.60:0/642875106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc980b5950 0x7fdc98067680 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 rx=0x7fdc900032f0 tx=0x7fdc90009290).ready entity=mon.0 client_cookie=8911aa96a968ab25 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:25.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.249+0000 7fdc967fc700 1 -- 172.21.15.60:0/642875106 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc9000b870 con 0x7fdc980b5950 2022-06-29T16:06:25.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.249+0000 7fdc967fc700 1 -- 172.21.15.60:0/642875106 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc9000b9d0 con 0x7fdc980b5950 2022-06-29T16:06:25.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.249+0000 7fdc967fc700 1 -- 172.21.15.60:0/642875106 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc9000bb40 con 0x7fdc980b5950 2022-06-29T16:06:25.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.249+0000 7fdc9ded3700 1 -- 172.21.15.60:0/642875106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc980b5950 msgr2=0x7fdc98067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:25.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.249+0000 7fdc9ded3700 1 --2- 172.21.15.60:0/642875106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc980b5950 0x7fdc98067680 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 rx=0x7fdc900032f0 tx=0x7fdc90009290).stop 2022-06-29T16:06:25.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.249+0000 7fdc9ded3700 1 -- 172.21.15.60:0/642875106 shutdown_connections 2022-06-29T16:06:25.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.249+0000 7fdc9ded3700 1 --2- 172.21.15.60:0/642875106 >> v2:172.21.15.92:3300/0 conn(0x7fdc9805bd20 0x7fdc9805c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:25.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.249+0000 7fdc9ded3700 1 --2- 172.21.15.60:0/642875106 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc98063ad0 0x7fdc9814af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:25.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.249+0000 7fdc9ded3700 1 --2- 172.21.15.60:0/642875106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc980b5950 0x7fdc98067680 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:25.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.249+0000 7fdc9ded3700 1 -- 172.21.15.60:0/642875106 >> 172.21.15.60:0/642875106 conn(0x7fdc980632c0 msgr2=0x7fdc980b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:25.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.249+0000 7fdc9ded3700 1 -- 172.21.15.60:0/642875106 shutdown_connections 2022-06-29T16:06:25.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.249+0000 7fdc9ded3700 1 -- 172.21.15.60:0/642875106 wait complete. 2022-06-29T16:06:25.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.253+0000 7fdc9ded3700 1 Processor -- start 2022-06-29T16:06:25.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.253+0000 7fdc9ded3700 1 -- start start 2022-06-29T16:06:25.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.253+0000 7fdc9ded3700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc9805bd20 0x7fdc98147b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:25.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.253+0000 7fdc9ded3700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc98063ad0 0x7fdc980ad990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:25.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.253+0000 7fdc9ded3700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc980b5950 0x7fdc980aeb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:25.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.253+0000 7fdc9ded3700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc980b0520 con 0x7fdc98063ad0 2022-06-29T16:06:25.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.253+0000 7fdc9ded3700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc980ade80 con 0x7fdc980b5950 2022-06-29T16:06:25.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.253+0000 7fdc9ded3700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc981594f0 con 0x7fdc9805bd20 2022-06-29T16:06:25.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.253+0000 7fdc96ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc98063ad0 0x7fdc980ad990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:25.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.253+0000 7fdc977fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc9805bd20 0x7fdc98147b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:25.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.253+0000 7fdc96ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc98063ad0 0x7fdc980ad990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51924/0 (socket says 172.21.15.60:51924) 2022-06-29T16:06:25.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.253+0000 7fdc97fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc980b5950 0x7fdc980aeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:25.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.253+0000 7fdc96ffd700 1 -- 172.21.15.60:0/1717661023 learned_addr learned my addr 172.21.15.60:0/1717661023 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:25.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.253+0000 7fdc96ffd700 1 -- 172.21.15.60:0/1717661023 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc9805bd20 msgr2=0x7fdc98147b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:25.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.253+0000 7fdc96ffd700 1 --2- 172.21.15.60:0/1717661023 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc9805bd20 0x7fdc98147b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:25.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.253+0000 7fdc96ffd700 1 -- 172.21.15.60:0/1717661023 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc980b5950 msgr2=0x7fdc980aeb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:25.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.253+0000 7fdc96ffd700 1 --2- 172.21.15.60:0/1717661023 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc980b5950 0x7fdc980aeb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:25.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.253+0000 7fdc96ffd700 1 -- 172.21.15.60:0/1717661023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc90009530 con 0x7fdc98063ad0 2022-06-29T16:06:25.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.253+0000 7fdc96ffd700 1 --2- 172.21.15.60:0/1717661023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc98063ad0 0x7fdc980ad990 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 rx=0x7fdc980adfc0 tx=0x7fdc80008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:25.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.253+0000 7fdc94ff9700 1 -- 172.21.15.60:0/1717661023 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc8000dea0 con 0x7fdc98063ad0 2022-06-29T16:06:25.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.253+0000 7fdc94ff9700 1 -- 172.21.15.60:0/1717661023 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc80011780 con 0x7fdc98063ad0 2022-06-29T16:06:25.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.253+0000 7fdc9ded3700 1 -- 172.21.15.60:0/1717661023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc98159720 con 0x7fdc98063ad0 2022-06-29T16:06:25.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.253+0000 7fdc94ff9700 1 -- 172.21.15.60:0/1717661023 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc80011ae0 con 0x7fdc98063ad0 2022-06-29T16:06:25.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.253+0000 7fdc9ded3700 1 -- 172.21.15.60:0/1717661023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc980af000 con 0x7fdc98063ad0 2022-06-29T16:06:25.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.257+0000 7fdc9ded3700 1 -- 172.21.15.60:0/1717661023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc981598b0 con 0x7fdc98063ad0 2022-06-29T16:06:25.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.257+0000 7fdc94ff9700 1 -- 172.21.15.60:0/1717661023 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdc80011c40 con 0x7fdc98063ad0 2022-06-29T16:06:25.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.257+0000 7fdc94ff9700 1 --2- 172.21.15.60:0/1717661023 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdc7c053a70 0x7fdc7c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:25.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.257+0000 7fdc94ff9700 1 -- 172.21.15.60:0/1717661023 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fdc8006b940 con 0x7fdc98063ad0 2022-06-29T16:06:25.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.257+0000 7fdc977fe700 1 --2- 172.21.15.60:0/1717661023 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdc7c053a70 0x7fdc7c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:25.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.261+0000 7fdc977fe700 1 --2- 172.21.15.60:0/1717661023 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdc7c053a70 0x7fdc7c055f30 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0x7fdc9001b000 tx=0x7fdc90016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:25.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.265+0000 7fdc94ff9700 1 -- 172.21.15.60:0/1717661023 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdc80042b90 con 0x7fdc98063ad0 2022-06-29T16:06:25.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.429+0000 7fdc9ded3700 1 -- 172.21.15.60:0/1717661023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fdc980573d0 con 0x7fdc98063ad0 2022-06-29T16:06:25.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.433+0000 7fdc94ff9700 1 -- 172.21.15.60:0/1717661023 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v37) v1 ==== 80+0+4242 (secure 0 0 0) 0x7fdc800461b0 con 0x7fdc98063ad0 2022-06-29T16:06:25.437 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:25.438 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:06:25.438 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_dump": [ 2022-06-29T16:06:25.438 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:25.439 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.a", 2022-06-29T16:06:25.439 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxivv7JNhAAnA291UxlB/SfgITOipP00A==", 2022-06-29T16:06:25.439 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:06:25.439 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:06:25.440 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:06:25.440 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:06:25.440 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:06:25.440 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:25.440 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:25.441 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:25.441 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.b", 2022-06-29T16:06:25.441 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxiRSr/OhAAfFQTF2YHvzCv5wXBVCA5WQ==", 2022-06-29T16:06:25.441 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:06:25.442 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:06:25.442 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:06:25.442 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:06:25.442 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:06:25.443 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:25.443 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:25.443 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:25.443 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.0", 2022-06-29T16:06:25.443 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiZYPxOBAAIkweWJlTJIXgxnlagkGdXA==", 2022-06-29T16:06:25.444 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:06:25.444 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:06:25.444 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:06:25.444 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:06:25.444 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:25.445 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:25.445 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:25.445 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.1", 2022-06-29T16:06:25.446 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxiS2JGBhAAJn0UfAxZI3IVFyD5QBdb/Q==", 2022-06-29T16:06:25.446 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:06:25.446 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:06:25.446 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:06:25.447 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:06:25.447 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:25.447 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:25.447 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:25.447 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.2", 2022-06-29T16:06:25.448 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxiPhkfEhAApH7r8OUJzW7fZNM6XVv1dg==", 2022-06-29T16:06:25.448 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:06:25.448 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:06:25.448 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:06:25.449 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:06:25.449 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:25.449 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:25.449 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:25.449 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.3", 2022-06-29T16:06:25.450 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxit/47HhAA1vyU6h7hjPG+op9qP/rVYQ==", 2022-06-29T16:06:25.450 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:06:25.450 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:06:25.450 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:06:25.451 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:06:25.451 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:25.451 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:25.451 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:25.452 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.4", 2022-06-29T16:06:25.452 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxiaKzKBxAAbIwCF9ag7ezsNy11hi0BEw==", 2022-06-29T16:06:25.452 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:06:25.452 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:06:25.453 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:06:25.453 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:06:25.453 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:25.453 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:25.454 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:25.454 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.5", 2022-06-29T16:06:25.454 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxiJQV/IBAASBBDb6UFgEtXILe4fcT/xA==", 2022-06-29T16:06:25.454 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:06:25.454 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:06:25.455 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:06:25.455 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:06:25.455 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:25.455 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:25.456 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:25.456 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.6", 2022-06-29T16:06:25.456 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxi6D07OBAA2FWiZ+mNK+jFMp/kPZnbQQ==", 2022-06-29T16:06:25.457 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:06:25.457 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:06:25.457 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:06:25.457 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:06:25.458 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:25.458 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:25.458 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:25.458 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.7", 2022-06-29T16:06:25.459 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCMdrxinEOcEhAA562ybpzGHNQfWIq8q7zw9w==", 2022-06-29T16:06:25.459 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:06:25.459 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:06:25.459 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:06:25.459 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:06:25.460 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:25.460 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:25.460 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:25.460 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.0", 2022-06-29T16:06:25.461 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiHXPwBRAAkOOwlYJvIW7dqVFdlXXoqw==", 2022-06-29T16:06:25.461 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:06:25.461 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:06:25.461 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2022-06-29T16:06:25.462 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw pool=cephfs_data" 2022-06-29T16:06:25.462 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:25.462 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:25.462 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:25.462 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.1", 2022-06-29T16:06:25.463 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxibe5JCRAAGBJbi9t09w0836JqrAT7+w==", 2022-06-29T16:06:25.463 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:06:25.463 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:06:25.463 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2022-06-29T16:06:25.464 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw pool=cephfs_data" 2022-06-29T16:06:25.464 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:25.464 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:25.464 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:25.464 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.2", 2022-06-29T16:06:25.465 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiK3dBDRAA7XINPSOPQDfOrUYEpHC+gw==", 2022-06-29T16:06:25.465 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:06:25.465 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:06:25.465 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2022-06-29T16:06:25.466 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw pool=cephfs_data" 2022-06-29T16:06:25.466 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:25.466 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:25.466 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:25.467 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.admin", 2022-06-29T16:06:25.467 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxieKVcGBAAnjH80a0qWIArz2qwjYHWdQ==", 2022-06-29T16:06:25.467 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:06:25.468 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T16:06:25.468 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:06:25.468 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:06:25.468 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:06:25.469 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:25.469 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:25.469 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:25.469 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.x", 2022-06-29T16:06:25.470 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxi2LZJNBAA3MKb/pXYbQCsoRx8EQfGbw==", 2022-06-29T16:06:25.470 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:06:25.470 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T16:06:25.470 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2022-06-29T16:06:25.470 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:06:25.471 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:25.471 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:25.471 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:25.471 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.y", 2022-06-29T16:06:25.472 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxi3LmaMBAAN+0mZrdKXqSXOipVTA1drw==", 2022-06-29T16:06:25.472 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:06:25.472 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T16:06:25.472 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2022-06-29T16:06:25.472 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:06:25.473 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:25.473 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:25.473 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:25.473 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:06:25.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.433+0000 7fdc9ded3700 1 -- 172.21.15.60:0/1717661023 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdc7c053a70 msgr2=0x7fdc7c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:25.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.433+0000 7fdc9ded3700 1 --2- 172.21.15.60:0/1717661023 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdc7c053a70 0x7fdc7c055f30 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0x7fdc9001b000 tx=0x7fdc90016000).stop 2022-06-29T16:06:25.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.433+0000 7fdc9ded3700 1 -- 172.21.15.60:0/1717661023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc98063ad0 msgr2=0x7fdc980ad990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:25.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.433+0000 7fdc9ded3700 1 --2- 172.21.15.60:0/1717661023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc98063ad0 0x7fdc980ad990 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 rx=0x7fdc980adfc0 tx=0x7fdc80008040).stop 2022-06-29T16:06:25.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.433+0000 7fdc9ded3700 1 -- 172.21.15.60:0/1717661023 shutdown_connections 2022-06-29T16:06:25.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.433+0000 7fdc9ded3700 1 --2- 172.21.15.60:0/1717661023 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdc7c053a70 0x7fdc7c055f30 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:25.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.433+0000 7fdc9ded3700 1 --2- 172.21.15.60:0/1717661023 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc980b5950 0x7fdc980aeb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:25.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.433+0000 7fdc9ded3700 1 --2- 172.21.15.60:0/1717661023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc98063ad0 0x7fdc980ad990 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:25.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.433+0000 7fdc9ded3700 1 --2- 172.21.15.60:0/1717661023 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc9805bd20 0x7fdc98147b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:25.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.433+0000 7fdc9ded3700 1 -- 172.21.15.60:0/1717661023 >> 172.21.15.60:0/1717661023 conn(0x7fdc980632c0 msgr2=0x7fdc981456f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:25.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.433+0000 7fdc9ded3700 1 -- 172.21.15.60:0/1717661023 shutdown_connections 2022-06-29T16:06:25.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.433+0000 7fdc9ded3700 1 -- 172.21.15.60:0/1717661023 wait complete. 2022-06-29T16:06:25.479 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:06:25.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.577+0000 7f3e19e70700 1 Processor -- start 2022-06-29T16:06:25.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.577+0000 7f3e19e70700 1 -- start start 2022-06-29T16:06:25.586 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.581+0000 7f3e19e70700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3e14067680 0x7f3e1414af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:25.586 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.581+0000 7f3e19e70700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e140acba0 con 0x7f3e140b5950 2022-06-29T16:06:25.587 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.581+0000 7f3e19e70700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e14067220 con 0x7f3e14063ad0 2022-06-29T16:06:25.587 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.581+0000 7f3e19e70700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e140b5d30 con 0x7f3e14067680 2022-06-29T16:06:25.587 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.581+0000 7f3e13fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3e14067680 0x7f3e1414af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:25.587 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.581+0000 7f3e13fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3e14067680 0x7f3e1414af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51934/0 (socket says 172.21.15.60:51934) 2022-06-29T16:06:25.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.581+0000 7f3e13fff700 1 -- 172.21.15.60:0/3793542395 learned_addr learned my addr 172.21.15.60:0/3793542395 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:25.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.581+0000 7f3e137fe700 1 --1- 172.21.15.60:0/3793542395 >> v1:172.21.15.60:6789/0 conn(0x7f3e140b5950 0x7f3e1405bd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60018/0 (socket says 172.21.15.60:60018) 2022-06-29T16:06:25.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.581+0000 7f3e13fff700 1 -- 172.21.15.60:0/3793542395 >> v1:172.21.15.92:6789/0 conn(0x7f3e14063ad0 legacy=0x7f3e1405c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:25.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.581+0000 7f3e13fff700 1 -- 172.21.15.60:0/3793542395 >> v1:172.21.15.60:6789/0 conn(0x7f3e140b5950 legacy=0x7f3e1405bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:25.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.581+0000 7f3e13fff700 1 -- 172.21.15.60:0/3793542395 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e1405cb40 con 0x7f3e14067680 2022-06-29T16:06:25.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.581+0000 7f3e13fff700 1 --2- 172.21.15.60:0/3793542395 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e14067680 0x7f3e1414af10 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 rx=0x7f3e0c0032f0 tx=0x7f3e0c009290).ready entity=mon.0 client_cookie=d8d9e658afdf8fa7 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:25.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.581+0000 7f3e127fc700 1 -- 172.21.15.60:0/3793542395 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e0c00b8a0 con 0x7f3e14067680 2022-06-29T16:06:25.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.581+0000 7f3e127fc700 1 -- 172.21.15.60:0/3793542395 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e0c00ba00 con 0x7f3e14067680 2022-06-29T16:06:25.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.581+0000 7f3e127fc700 1 -- 172.21.15.60:0/3793542395 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e0c00bb70 con 0x7f3e14067680 2022-06-29T16:06:25.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.581+0000 7f3e19e70700 1 -- 172.21.15.60:0/3793542395 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e14067680 msgr2=0x7f3e1414af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:25.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.581+0000 7f3e19e70700 1 --2- 172.21.15.60:0/3793542395 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e14067680 0x7f3e1414af10 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 rx=0x7f3e0c0032f0 tx=0x7f3e0c009290).stop 2022-06-29T16:06:25.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.581+0000 7f3e19e70700 1 -- 172.21.15.60:0/3793542395 shutdown_connections 2022-06-29T16:06:25.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.581+0000 7f3e19e70700 1 --2- 172.21.15.60:0/3793542395 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e14067680 0x7f3e1414af10 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:25.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.581+0000 7f3e19e70700 1 -- 172.21.15.60:0/3793542395 >> 172.21.15.60:0/3793542395 conn(0x7f3e140632c0 msgr2=0x7f3e140b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:25.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.581+0000 7f3e19e70700 1 -- 172.21.15.60:0/3793542395 shutdown_connections 2022-06-29T16:06:25.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.581+0000 7f3e19e70700 1 -- 172.21.15.60:0/3793542395 wait complete. 2022-06-29T16:06:25.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.585+0000 7f3e19e70700 1 Processor -- start 2022-06-29T16:06:25.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.585+0000 7f3e19e70700 1 -- start start 2022-06-29T16:06:25.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.585+0000 7f3e19e70700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e14063ad0 0x7f3e14065220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:25.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.585+0000 7f3e19e70700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e14067680 0x7f3e1413a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:25.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.585+0000 7f3e19e70700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e140b5950 0x7f3e1413a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:25.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.585+0000 7f3e19e70700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e140b0ed0 con 0x7f3e14067680 2022-06-29T16:06:25.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.585+0000 7f3e19e70700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1413ad10 con 0x7f3e140b5950 2022-06-29T16:06:25.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.585+0000 7f3e19e70700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1413aa40 con 0x7f3e14063ad0 2022-06-29T16:06:25.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.585+0000 7f3e12ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e14067680 0x7f3e1413a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:25.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.585+0000 7f3e12ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e14067680 0x7f3e1413a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51938/0 (socket says 172.21.15.60:51938) 2022-06-29T16:06:25.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.585+0000 7f3e12ffd700 1 -- 172.21.15.60:0/2792616038 learned_addr learned my addr 172.21.15.60:0/2792616038 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:25.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.585+0000 7f3e137fe700 1 --2- 172.21.15.60:0/2792616038 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e14063ad0 0x7f3e14065220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:25.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.585+0000 7f3e13fff700 1 --2- 172.21.15.60:0/2792616038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e140b5950 0x7f3e1413a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:25.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.589+0000 7f3e12ffd700 1 -- 172.21.15.60:0/2792616038 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e14063ad0 msgr2=0x7f3e14065220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:25.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.589+0000 7f3e12ffd700 1 --2- 172.21.15.60:0/2792616038 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e14063ad0 0x7f3e14065220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:25.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.589+0000 7f3e12ffd700 1 -- 172.21.15.60:0/2792616038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e140b5950 msgr2=0x7f3e1413a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:25.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.589+0000 7f3e12ffd700 1 --2- 172.21.15.60:0/2792616038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e140b5950 0x7f3e1413a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:25.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.589+0000 7f3e12ffd700 1 -- 172.21.15.60:0/2792616038 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e0c009530 con 0x7f3e14067680 2022-06-29T16:06:25.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.589+0000 7f3e12ffd700 1 --2- 172.21.15.60:0/2792616038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e14067680 0x7f3e1413a060 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 rx=0x7f3e14067560 tx=0x7f3dfc007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:25.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.589+0000 7f3e10ff9700 1 -- 172.21.15.60:0/2792616038 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3dfc007030 con 0x7f3e14067680 2022-06-29T16:06:25.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.589+0000 7f3e10ff9700 1 -- 172.21.15.60:0/2792616038 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dfc007190 con 0x7f3e14067680 2022-06-29T16:06:25.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.589+0000 7f3e19e70700 1 -- 172.21.15.60:0/2792616038 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e140b1170 con 0x7f3e14067680 2022-06-29T16:06:25.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.589+0000 7f3e10ff9700 1 -- 172.21.15.60:0/2792616038 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3dfc006510 con 0x7f3e14067680 2022-06-29T16:06:25.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.589+0000 7f3e19e70700 1 -- 172.21.15.60:0/2792616038 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e140b0a40 con 0x7f3e14067680 2022-06-29T16:06:25.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.589+0000 7f3e19e70700 1 -- 172.21.15.60:0/2792616038 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e1413af10 con 0x7f3e14067680 2022-06-29T16:06:25.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.589+0000 7f3e10ff9700 1 -- 172.21.15.60:0/2792616038 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3dfc010030 con 0x7f3e14067680 2022-06-29T16:06:25.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.593+0000 7f3e10ff9700 1 --2- 172.21.15.60:0/2792616038 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3df8053b00 0x7f3df8055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:25.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.593+0000 7f3e10ff9700 1 -- 172.21.15.60:0/2792616038 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f3dfc06b840 con 0x7f3e14067680 2022-06-29T16:06:25.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.593+0000 7f3e137fe700 1 --2- 172.21.15.60:0/2792616038 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3df8053b00 0x7f3df8055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:25.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.593+0000 7f3e137fe700 1 --2- 172.21.15.60:0/2792616038 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3df8053b00 0x7f3df8055fc0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7f3e0400d000 tx=0x7f3e04000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:25.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.597+0000 7f3e10ff9700 1 -- 172.21.15.60:0/2792616038 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3e1413af10 con 0x7f3e14067680 2022-06-29T16:06:25.728 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.721+0000 7f3e19e70700 1 -- 172.21.15.60:0/2792616038 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e1413af10 con 0x7f3e14067680 2022-06-29T16:06:25.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.725+0000 7f3e10ff9700 1 -- 172.21.15.60:0/2792616038 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) v1 ==== 74+0+13183 (secure 0 0 0) 0x7f3e1413af10 con 0x7f3e14067680 2022-06-29T16:06:25.730 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:25.730 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":105,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:06:25.112337+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:06:25.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.725+0000 7f3e19e70700 1 -- 172.21.15.60:0/2792616038 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3df8053b00 msgr2=0x7f3df8055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:25.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.725+0000 7f3e19e70700 1 --2- 172.21.15.60:0/2792616038 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3df8053b00 0x7f3df8055fc0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7f3e0400d000 tx=0x7f3e04000bc0).stop 2022-06-29T16:06:25.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.725+0000 7f3e19e70700 1 -- 172.21.15.60:0/2792616038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e14067680 msgr2=0x7f3e1413a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:25.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.725+0000 7f3e19e70700 1 --2- 172.21.15.60:0/2792616038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e14067680 0x7f3e1413a060 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 rx=0x7f3e14067560 tx=0x7f3dfc007c10).stop 2022-06-29T16:06:25.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.725+0000 7f3e19e70700 1 -- 172.21.15.60:0/2792616038 shutdown_connections 2022-06-29T16:06:25.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.725+0000 7f3e19e70700 1 --2- 172.21.15.60:0/2792616038 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3df8053b00 0x7f3df8055fc0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:25.734 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.725+0000 7f3e19e70700 1 --2- 172.21.15.60:0/2792616038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e140b5950 0x7f3e1413a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:25.734 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.725+0000 7f3e19e70700 1 --2- 172.21.15.60:0/2792616038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e14067680 0x7f3e1413a060 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:25.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.725+0000 7f3e19e70700 1 --2- 172.21.15.60:0/2792616038 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e14063ad0 0x7f3e14065220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:25.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.725+0000 7f3e19e70700 1 -- 172.21.15.60:0/2792616038 >> 172.21.15.60:0/2792616038 conn(0x7f3e140632c0 msgr2=0x7f3e14147a70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:25.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.725+0000 7f3e19e70700 1 -- 172.21.15.60:0/2792616038 shutdown_connections 2022-06-29T16:06:25.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.725+0000 7f3e19e70700 1 -- 172.21.15.60:0/2792616038 wait complete. 2022-06-29T16:06:25.743 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-06-29T16:06:25.744 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:06:25.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.849+0000 7fae0285c700 1 Processor -- start 2022-06-29T16:06:25.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.849+0000 7fae0285c700 1 -- start start 2022-06-29T16:06:25.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.849+0000 7fae0285c700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fadfc0ad520 0x7fadfc065ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:25.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.849+0000 7fae0285c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadfc065fd0 0x7fadfc13fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:25.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.849+0000 7fae0285c700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadfc0ad140 con 0x7fadfc0acd60 2022-06-29T16:06:25.859 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.849+0000 7fae0285c700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadfc0649e0 con 0x7fadfc0ad520 2022-06-29T16:06:25.859 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.849+0000 7fae0285c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadfc0b24f0 con 0x7fadfc065fd0 2022-06-29T16:06:25.859 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.849+0000 7fae0185a700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fadfc0acd60 0x7fadfc143e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60030/0 (socket says 172.21.15.60:60030) 2022-06-29T16:06:25.860 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.849+0000 7fae0185a700 1 -- 172.21.15.60:0/3848836064 learned_addr learned my addr 172.21.15.60:0/3848836064 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:25.860 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.849+0000 7fae0205b700 1 --2- 172.21.15.60:0/3848836064 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadfc065fd0 0x7fadfc13fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:25.860 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.849+0000 7fae01059700 1 --2- 172.21.15.60:0/3848836064 >> v2:172.21.15.92:3300/0 conn(0x7fadfc0ad520 0x7fadfc065ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:25.860 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.849+0000 7fae00858700 1 -- 172.21.15.60:0/3848836064 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340223301 0 0) 0x7fadfc0ad140 con 0x7fadfc0acd60 2022-06-29T16:06:25.861 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.849+0000 7fae00858700 1 -- 172.21.15.60:0/3848836064 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fade8003260 con 0x7fadfc0acd60 2022-06-29T16:06:25.861 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.849+0000 7fae0205b700 1 -- 172.21.15.60:0/3848836064 >> v2:172.21.15.92:3300/0 conn(0x7fadfc0ad520 msgr2=0x7fadfc065ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:25.861 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.849+0000 7fae0205b700 1 --2- 172.21.15.60:0/3848836064 >> v2:172.21.15.92:3300/0 conn(0x7fadfc0ad520 0x7fadfc065ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:25.862 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.849+0000 7fae0205b700 1 -- 172.21.15.60:0/3848836064 >> v1:172.21.15.60:6789/0 conn(0x7fadfc0acd60 legacy=0x7fadfc143e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:25.862 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.849+0000 7fae0205b700 1 -- 172.21.15.60:0/3848836064 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadfc065640 con 0x7fadfc065fd0 2022-06-29T16:06:25.862 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.849+0000 7fae0205b700 1 --2- 172.21.15.60:0/3848836064 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadfc065fd0 0x7fadfc13fbf0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fadf4002f70 tx=0x7fadf4006df0).ready entity=mon.2 client_cookie=d095599438060739 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:25.862 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.849+0000 7fae00858700 1 -- 172.21.15.60:0/3848836064 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fadf400d4a0 con 0x7fadfc065fd0 2022-06-29T16:06:25.863 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.853+0000 7fae00858700 1 -- 172.21.15.60:0/3848836064 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf4000e90 con 0x7fadfc065fd0 2022-06-29T16:06:25.863 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.853+0000 7fae00858700 1 -- 172.21.15.60:0/3848836064 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fadf400d7f0 con 0x7fadfc065fd0 2022-06-29T16:06:25.863 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.853+0000 7fae0285c700 1 -- 172.21.15.60:0/3848836064 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadfc065fd0 msgr2=0x7fadfc13fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:25.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.853+0000 7fae0285c700 1 --2- 172.21.15.60:0/3848836064 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadfc065fd0 0x7fadfc13fbf0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fadf4002f70 tx=0x7fadf4006df0).stop 2022-06-29T16:06:25.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.853+0000 7fae0285c700 1 -- 172.21.15.60:0/3848836064 shutdown_connections 2022-06-29T16:06:25.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.853+0000 7fae0285c700 1 --2- 172.21.15.60:0/3848836064 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadfc065fd0 0x7fadfc13fbf0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:25.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.853+0000 7fae0285c700 1 --2- 172.21.15.60:0/3848836064 >> v2:172.21.15.92:3300/0 conn(0x7fadfc0ad520 0x7fadfc065ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:25.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.853+0000 7fae0285c700 1 -- 172.21.15.60:0/3848836064 >> 172.21.15.60:0/3848836064 conn(0x7fadfc0b6fb0 msgr2=0x7fadfc0b7390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:25.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.853+0000 7fae0285c700 1 -- 172.21.15.60:0/3848836064 shutdown_connections 2022-06-29T16:06:25.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.853+0000 7fae0285c700 1 -- 172.21.15.60:0/3848836064 wait complete. 2022-06-29T16:06:25.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.853+0000 7fae0285c700 1 Processor -- start 2022-06-29T16:06:25.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.853+0000 7fae0285c700 1 -- start start 2022-06-29T16:06:25.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.857+0000 7fae0285c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadfc065fd0 0x7fadfc0ae730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:25.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.857+0000 7fae0285c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fadfc0acd60 0x7fadfc138af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:25.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.857+0000 7fae0285c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fadfc0ad520 0x7fadfc138fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:25.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.857+0000 7fae0285c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadfc140fa0 con 0x7fadfc0ad520 2022-06-29T16:06:25.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.857+0000 7fae0285c700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadfc141240 con 0x7fadfc0acd60 2022-06-29T16:06:25.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.857+0000 7fae0285c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadfc146a60 con 0x7fadfc065fd0 2022-06-29T16:06:25.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.857+0000 7fae0205b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fadfc0ad520 0x7fadfc138fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:25.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.857+0000 7fae0185a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadfc065fd0 0x7fadfc0ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:25.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.857+0000 7fae01059700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fadfc0acd60 0x7fadfc138af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:25.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.857+0000 7fae0205b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fadfc0ad520 0x7fadfc138fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51954/0 (socket says 172.21.15.60:51954) 2022-06-29T16:06:25.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.857+0000 7fae0205b700 1 -- 172.21.15.60:0/4134649635 learned_addr learned my addr 172.21.15.60:0/4134649635 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:25.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.857+0000 7fae0205b700 1 -- 172.21.15.60:0/4134649635 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadfc065fd0 msgr2=0x7fadfc0ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:25.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.857+0000 7fae0205b700 1 --2- 172.21.15.60:0/4134649635 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadfc065fd0 0x7fadfc0ae730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:25.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.857+0000 7fae0205b700 1 -- 172.21.15.60:0/4134649635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fadfc0acd60 msgr2=0x7fadfc138af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:25.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.857+0000 7fae0205b700 1 --2- 172.21.15.60:0/4134649635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fadfc0acd60 0x7fadfc138af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:25.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.857+0000 7fae0205b700 1 -- 172.21.15.60:0/4134649635 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf400b180 con 0x7fadfc0ad520 2022-06-29T16:06:25.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.857+0000 7fae0205b700 1 --2- 172.21.15.60:0/4134649635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fadfc0ad520 0x7fadfc138fe0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 rx=0x7fadf4002f70 tx=0x7fadf4014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:25.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.857+0000 7fadfaffd700 1 -- 172.21.15.60:0/4134649635 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fadf4016070 con 0x7fadfc0ad520 2022-06-29T16:06:25.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.857+0000 7fae0285c700 1 -- 172.21.15.60:0/4134649635 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadfc1394d0 con 0x7fadfc0ad520 2022-06-29T16:06:25.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.857+0000 7fadfaffd700 1 -- 172.21.15.60:0/4134649635 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf40137d0 con 0x7fadfc0ad520 2022-06-29T16:06:25.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.857+0000 7fae0285c700 1 -- 172.21.15.60:0/4134649635 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadfc142c80 con 0x7fadfc0ad520 2022-06-29T16:06:25.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.857+0000 7fadfaffd700 1 -- 172.21.15.60:0/4134649635 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fadf4013b30 con 0x7fadfc0ad520 2022-06-29T16:06:25.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.861+0000 7fae0285c700 1 -- 172.21.15.60:0/4134649635 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadfc1467a0 con 0x7fadfc0ad520 2022-06-29T16:06:25.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.861+0000 7fadfaffd700 1 -- 172.21.15.60:0/4134649635 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fadf4025030 con 0x7fadfc0ad520 2022-06-29T16:06:25.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.861+0000 7fadfaffd700 1 --2- 172.21.15.60:0/4134649635 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7faddc053a70 0x7faddc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:25.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.861+0000 7fadfaffd700 1 -- 172.21.15.60:0/4134649635 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fadf406cad0 con 0x7fadfc0ad520 2022-06-29T16:06:25.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.861+0000 7fae0185a700 1 --2- 172.21.15.60:0/4134649635 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7faddc053a70 0x7faddc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:25.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.861+0000 7fae0185a700 1 --2- 172.21.15.60:0/4134649635 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7faddc053a70 0x7faddc055f30 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7fadec00d000 tx=0x7fadec009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:25.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.869+0000 7fadfaffd700 1 -- 172.21.15.60:0/4134649635 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fadfc1467a0 con 0x7fadfc0ad520 2022-06-29T16:06:26.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:25.997+0000 7fae0285c700 1 -- 172.21.15.60:0/4134649635 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fadfc1467a0 con 0x7fadfc0ad520 2022-06-29T16:06:26.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.001+0000 7fadfaffd700 1 -- 172.21.15.60:0/4134649635 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v105) v1 ==== 110+0+10 (secure 0 0 0) 0x7fadfc1467a0 con 0x7fadfc0ad520 2022-06-29T16:06:26.006 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:06:26.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.001+0000 7fae0285c700 1 -- 172.21.15.60:0/4134649635 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7faddc053a70 msgr2=0x7faddc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:26.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.001+0000 7fae0285c700 1 --2- 172.21.15.60:0/4134649635 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7faddc053a70 0x7faddc055f30 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7fadec00d000 tx=0x7fadec009040).stop 2022-06-29T16:06:26.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.001+0000 7fae0285c700 1 -- 172.21.15.60:0/4134649635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fadfc0ad520 msgr2=0x7fadfc138fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:26.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.001+0000 7fae0285c700 1 --2- 172.21.15.60:0/4134649635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fadfc0ad520 0x7fadfc138fe0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 rx=0x7fadf4002f70 tx=0x7fadf4014000).stop 2022-06-29T16:06:26.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.001+0000 7fae0285c700 1 -- 172.21.15.60:0/4134649635 shutdown_connections 2022-06-29T16:06:26.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.001+0000 7fae0285c700 1 --2- 172.21.15.60:0/4134649635 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7faddc053a70 0x7faddc055f30 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:26.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.001+0000 7fae0285c700 1 --2- 172.21.15.60:0/4134649635 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fadfc0ad520 0x7fadfc138fe0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:26.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.001+0000 7fae0285c700 1 --2- 172.21.15.60:0/4134649635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fadfc0acd60 0x7fadfc138af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:26.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.001+0000 7fae0285c700 1 --2- 172.21.15.60:0/4134649635 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fadfc065fd0 0x7fadfc0ae730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:26.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.001+0000 7fae0285c700 1 -- 172.21.15.60:0/4134649635 >> 172.21.15.60:0/4134649635 conn(0x7fadfc0b6fb0 msgr2=0x7fadfc066a30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:26.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.001+0000 7fae0285c700 1 -- 172.21.15.60:0/4134649635 shutdown_connections 2022-06-29T16:06:26.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.001+0000 7fae0285c700 1 -- 172.21.15.60:0/4134649635 wait complete. 2022-06-29T16:06:26.018 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:06:26.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.117+0000 7fcd4887c700 1 Processor -- start 2022-06-29T16:06:26.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.117+0000 7fcd4887c700 1 -- start start 2022-06-29T16:06:26.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.117+0000 7fcd4887c700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcd440b5eb0 0x7fcd44142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:26.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.117+0000 7fcd4887c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcd440b2c50 0x7fcd440b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:26.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.117+0000 7fcd4887c700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd440b0cf0 con 0x7fcd440b48b0 2022-06-29T16:06:26.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.117+0000 7fcd4887c700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd440b54e0 con 0x7fcd440b2c50 2022-06-29T16:06:26.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.117+0000 7fcd4887c700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd440b4d10 con 0x7fcd440b5eb0 2022-06-29T16:06:26.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.117+0000 7fcd43fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcd440b2c50 0x7fcd440b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:26.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.117+0000 7fcd43fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcd440b2c50 0x7fcd440b30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51960/0 (socket says 172.21.15.60:51960) 2022-06-29T16:06:26.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.117+0000 7fcd43fff700 1 -- 172.21.15.60:0/3816524287 learned_addr learned my addr 172.21.15.60:0/3816524287 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:26.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.117+0000 7fcd437fe700 1 --2- 172.21.15.60:0/3816524287 >> v2:172.21.15.92:3300/0 conn(0x7fcd440b5eb0 0x7fcd44142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:26.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.121+0000 7fcd43fff700 1 -- 172.21.15.60:0/3816524287 >> v2:172.21.15.92:3300/0 conn(0x7fcd440b5eb0 msgr2=0x7fcd44142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:26.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.121+0000 7fcd43fff700 1 --2- 172.21.15.60:0/3816524287 >> v2:172.21.15.92:3300/0 conn(0x7fcd440b5eb0 0x7fcd44142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:26.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.121+0000 7fcd43fff700 1 -- 172.21.15.60:0/3816524287 >> v1:172.21.15.92:6789/0 conn(0x7fcd440b48b0 legacy=0x7fcd440b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:26.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.121+0000 7fcd43fff700 1 -- 172.21.15.60:0/3816524287 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd440b6290 con 0x7fcd440b2c50 2022-06-29T16:06:26.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.121+0000 7fcd43fff700 1 --2- 172.21.15.60:0/3816524287 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd440b2c50 0x7fcd440b30c0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 rx=0x7fcd3c0032f0 tx=0x7fcd3c009290).ready entity=mon.0 client_cookie=da4a211b0861d34 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:26.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.121+0000 7fcd42ffd700 1 -- 172.21.15.60:0/3816524287 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd3c00b870 con 0x7fcd440b2c50 2022-06-29T16:06:26.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.121+0000 7fcd42ffd700 1 -- 172.21.15.60:0/3816524287 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd3c00b9d0 con 0x7fcd440b2c50 2022-06-29T16:06:26.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.121+0000 7fcd42ffd700 1 -- 172.21.15.60:0/3816524287 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd3c00bb40 con 0x7fcd440b2c50 2022-06-29T16:06:26.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.121+0000 7fcd4887c700 1 -- 172.21.15.60:0/3816524287 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd440b2c50 msgr2=0x7fcd440b30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:26.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.121+0000 7fcd4887c700 1 --2- 172.21.15.60:0/3816524287 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd440b2c50 0x7fcd440b30c0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 rx=0x7fcd3c0032f0 tx=0x7fcd3c009290).stop 2022-06-29T16:06:26.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.121+0000 7fcd4887c700 1 -- 172.21.15.60:0/3816524287 shutdown_connections 2022-06-29T16:06:26.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.121+0000 7fcd4887c700 1 --2- 172.21.15.60:0/3816524287 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd440b2c50 0x7fcd440b30c0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:26.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.121+0000 7fcd4887c700 1 --2- 172.21.15.60:0/3816524287 >> v2:172.21.15.92:3300/0 conn(0x7fcd440b5eb0 0x7fcd44142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:26.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.121+0000 7fcd4887c700 1 -- 172.21.15.60:0/3816524287 >> 172.21.15.60:0/3816524287 conn(0x7fcd44062090 msgr2=0x7fcd44062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:26.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.121+0000 7fcd4887c700 1 -- 172.21.15.60:0/3816524287 shutdown_connections 2022-06-29T16:06:26.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.121+0000 7fcd4887c700 1 -- 172.21.15.60:0/3816524287 wait complete. 2022-06-29T16:06:26.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.121+0000 7fcd4887c700 1 Processor -- start 2022-06-29T16:06:26.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.121+0000 7fcd4887c700 1 -- start start 2022-06-29T16:06:26.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.125+0000 7fcd4887c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd440b2c50 0x7fcd440adee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:26.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.125+0000 7fcd4887c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd440b48b0 0x7fcd44145080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:26.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.125+0000 7fcd4887c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd440b5eb0 0x7fcd44143ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:26.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.125+0000 7fcd4887c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd441462f0 con 0x7fcd440b5eb0 2022-06-29T16:06:26.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.125+0000 7fcd4887c700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd44145db0 con 0x7fcd440b48b0 2022-06-29T16:06:26.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.125+0000 7fcd4887c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd44146050 con 0x7fcd440b2c50 2022-06-29T16:06:26.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.125+0000 7fcd43fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd440b5eb0 0x7fcd44143ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:26.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.125+0000 7fcd43fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd440b5eb0 0x7fcd44143ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51968/0 (socket says 172.21.15.60:51968) 2022-06-29T16:06:26.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.125+0000 7fcd43fff700 1 -- 172.21.15.60:0/3170846795 learned_addr learned my addr 172.21.15.60:0/3170846795 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:26.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.125+0000 7fcd437fe700 1 --2- 172.21.15.60:0/3170846795 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd440b2c50 0x7fcd440adee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:26.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.125+0000 7fcd3bfff700 1 --2- 172.21.15.60:0/3170846795 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd440b48b0 0x7fcd44145080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:26.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.125+0000 7fcd43fff700 1 -- 172.21.15.60:0/3170846795 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd440b2c50 msgr2=0x7fcd440adee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:26.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.125+0000 7fcd43fff700 1 --2- 172.21.15.60:0/3170846795 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd440b2c50 0x7fcd440adee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:26.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.125+0000 7fcd43fff700 1 -- 172.21.15.60:0/3170846795 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd440b48b0 msgr2=0x7fcd44145080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:26.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.125+0000 7fcd43fff700 1 --2- 172.21.15.60:0/3170846795 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd440b48b0 0x7fcd44145080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:26.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.125+0000 7fcd43fff700 1 -- 172.21.15.60:0/3170846795 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd3c009530 con 0x7fcd440b5eb0 2022-06-29T16:06:26.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.125+0000 7fcd43fff700 1 --2- 172.21.15.60:0/3170846795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd440b5eb0 0x7fcd44143ee0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 rx=0x7fcd3c0032f0 tx=0x7fcd3c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:26.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.125+0000 7fcd417fa700 1 -- 172.21.15.60:0/3170846795 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd3c00b870 con 0x7fcd440b5eb0 2022-06-29T16:06:26.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.125+0000 7fcd417fa700 1 -- 172.21.15.60:0/3170846795 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd3c021630 con 0x7fcd440b5eb0 2022-06-29T16:06:26.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.125+0000 7fcd4887c700 1 -- 172.21.15.60:0/3170846795 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd441133c0 con 0x7fcd440b5eb0 2022-06-29T16:06:26.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.125+0000 7fcd417fa700 1 -- 172.21.15.60:0/3170846795 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd3c021990 con 0x7fcd440b5eb0 2022-06-29T16:06:26.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.125+0000 7fcd4887c700 1 -- 172.21.15.60:0/3170846795 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd44146e90 con 0x7fcd440b5eb0 2022-06-29T16:06:26.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.125+0000 7fcd4887c700 1 -- 172.21.15.60:0/3170846795 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd44113550 con 0x7fcd440b5eb0 2022-06-29T16:06:26.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.125+0000 7fcd417fa700 1 -- 172.21.15.60:0/3170846795 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcd3c027020 con 0x7fcd440b5eb0 2022-06-29T16:06:26.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.129+0000 7fcd417fa700 1 --2- 172.21.15.60:0/3170846795 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcd1c053a70 0x7fcd1c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:26.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.129+0000 7fcd417fa700 1 -- 172.21.15.60:0/3170846795 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fcd3c06ce20 con 0x7fcd440b5eb0 2022-06-29T16:06:26.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.129+0000 7fcd437fe700 1 --2- 172.21.15.60:0/3170846795 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcd1c053a70 0x7fcd1c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:26.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.129+0000 7fcd437fe700 1 --2- 172.21.15.60:0/3170846795 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcd1c053a70 0x7fcd1c055f30 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7fcd3400b000 tx=0x7fcd34006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:26.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.133+0000 7fcd417fa700 1 -- 172.21.15.60:0/3170846795 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcd3c044120 con 0x7fcd440b5eb0 2022-06-29T16:06:26.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.265+0000 7fcd4887c700 1 -- 172.21.15.60:0/3170846795 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fcd440573d0 con 0x7fcd440b5eb0 2022-06-29T16:06:26.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.265+0000 7fcd417fa700 1 -- 172.21.15.60:0/3170846795 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v105) v1 ==== 92+0+10 (secure 0 0 0) 0x7fcd3c015070 con 0x7fcd440b5eb0 2022-06-29T16:06:26.270 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:06:26.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.269+0000 7fcd4887c700 1 -- 172.21.15.60:0/3170846795 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcd1c053a70 msgr2=0x7fcd1c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:26.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.269+0000 7fcd4887c700 1 --2- 172.21.15.60:0/3170846795 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcd1c053a70 0x7fcd1c055f30 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7fcd3400b000 tx=0x7fcd34006df0).stop 2022-06-29T16:06:26.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.269+0000 7fcd4887c700 1 -- 172.21.15.60:0/3170846795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd440b5eb0 msgr2=0x7fcd44143ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:26.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.269+0000 7fcd4887c700 1 --2- 172.21.15.60:0/3170846795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd440b5eb0 0x7fcd44143ee0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 rx=0x7fcd3c0032f0 tx=0x7fcd3c016000).stop 2022-06-29T16:06:26.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.269+0000 7fcd4887c700 1 -- 172.21.15.60:0/3170846795 shutdown_connections 2022-06-29T16:06:26.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.269+0000 7fcd4887c700 1 --2- 172.21.15.60:0/3170846795 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcd1c053a70 0x7fcd1c055f30 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:26.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.269+0000 7fcd4887c700 1 --2- 172.21.15.60:0/3170846795 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd440b5eb0 0x7fcd44143ee0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:26.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.269+0000 7fcd4887c700 1 --2- 172.21.15.60:0/3170846795 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd440b48b0 0x7fcd44145080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:26.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.269+0000 7fcd4887c700 1 --2- 172.21.15.60:0/3170846795 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd440b2c50 0x7fcd440adee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:26.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.269+0000 7fcd4887c700 1 -- 172.21.15.60:0/3170846795 >> 172.21.15.60:0/3170846795 conn(0x7fcd44062090 msgr2=0x7fcd44141ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:26.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.269+0000 7fcd4887c700 1 -- 172.21.15.60:0/3170846795 shutdown_connections 2022-06-29T16:06:26.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.269+0000 7fcd4887c700 1 -- 172.21.15.60:0/3170846795 wait complete. 2022-06-29T16:06:26.288 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-06-29T16:06:26.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.385+0000 7f069cf46700 1 Processor -- start 2022-06-29T16:06:26.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.385+0000 7f069cf46700 1 -- start start 2022-06-29T16:06:26.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.385+0000 7f069cf46700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f069805d070 0x7f069814aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:26.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.385+0000 7f069cf46700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06980b2130 0x7f06980aca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:26.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.385+0000 7f069cf46700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069805d450 con 0x7f069814af80 2022-06-29T16:06:26.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.385+0000 7f069cf46700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06980adf90 con 0x7f069805d070 2022-06-29T16:06:26.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.385+0000 7f069cf46700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06980b25d0 con 0x7f06980b2130 2022-06-29T16:06:26.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.385+0000 7f06977fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f069805d070 0x7f069814aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:26.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.385+0000 7f0697fff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f069814af80 0x7f06980acf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60062/0 (socket says 172.21.15.60:60062) 2022-06-29T16:06:26.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.385+0000 7f0697fff700 1 -- 172.21.15.60:0/346089305 learned_addr learned my addr 172.21.15.60:0/346089305 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:26.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.385+0000 7f0696ffd700 1 --2- 172.21.15.60:0/346089305 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06980b2130 0x7f06980aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:26.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.385+0000 7f06967fc700 1 -- 172.21.15.60:0/346089305 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1822994228 0 0) 0x7f069805d450 con 0x7f069814af80 2022-06-29T16:06:26.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.385+0000 7f06967fc700 1 -- 172.21.15.60:0/346089305 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0684003260 con 0x7f069814af80 2022-06-29T16:06:26.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.385+0000 7f06967fc700 1 -- 172.21.15.60:0/346089305 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2204319807 0 0) 0x7f0684003260 con 0x7f069814af80 2022-06-29T16:06:26.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.385+0000 7f06977fe700 1 -- 172.21.15.60:0/346089305 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06980b2130 msgr2=0x7f06980aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:26.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.385+0000 7f06977fe700 1 --2- 172.21.15.60:0/346089305 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06980b2130 0x7f06980aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:26.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.385+0000 7f06977fe700 1 -- 172.21.15.60:0/346089305 >> v1:172.21.15.60:6789/0 conn(0x7f069814af80 legacy=0x7f06980acf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:26.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.385+0000 7f06977fe700 1 -- 172.21.15.60:0/346089305 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f069814a350 con 0x7f069805d070 2022-06-29T16:06:26.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.389+0000 7f06977fe700 1 --2- 172.21.15.60:0/346089305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f069805d070 0x7f069814aa90 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 rx=0x7f0688001820 tx=0x7f06900091d0).ready entity=mon.0 client_cookie=a48f3dd6de0f597d server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:26.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.389+0000 7f06967fc700 1 -- 172.21.15.60:0/346089305 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0688001680 con 0x7f069805d070 2022-06-29T16:06:26.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.389+0000 7f06967fc700 1 -- 172.21.15.60:0/346089305 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f069000f510 con 0x7f069805d070 2022-06-29T16:06:26.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.389+0000 7f06967fc700 1 -- 172.21.15.60:0/346089305 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f069000f680 con 0x7f069805d070 2022-06-29T16:06:26.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.389+0000 7f069cf46700 1 -- 172.21.15.60:0/346089305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f069805d070 msgr2=0x7f069814aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:26.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.389+0000 7f069cf46700 1 --2- 172.21.15.60:0/346089305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f069805d070 0x7f069814aa90 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 rx=0x7f0688001820 tx=0x7f06900091d0).stop 2022-06-29T16:06:26.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.389+0000 7f069cf46700 1 -- 172.21.15.60:0/346089305 shutdown_connections 2022-06-29T16:06:26.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.389+0000 7f069cf46700 1 --2- 172.21.15.60:0/346089305 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f06980b2130 0x7f06980aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:26.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.389+0000 7f069cf46700 1 --2- 172.21.15.60:0/346089305 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f069805d070 0x7f069814aa90 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:26.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.389+0000 7f069cf46700 1 -- 172.21.15.60:0/346089305 >> 172.21.15.60:0/346089305 conn(0x7f06980b5890 msgr2=0x7f06980b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:26.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.389+0000 7f069cf46700 1 -- 172.21.15.60:0/346089305 shutdown_connections 2022-06-29T16:06:26.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.389+0000 7f069cf46700 1 -- 172.21.15.60:0/346089305 wait complete. 2022-06-29T16:06:26.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.393+0000 7f069cf46700 1 Processor -- start 2022-06-29T16:06:26.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.393+0000 7f069cf46700 1 -- start start 2022-06-29T16:06:26.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.393+0000 7f069cf46700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f069805d070 0x7f06980b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:26.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.393+0000 7f069cf46700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06980b2130 0x7f0698066590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:26.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.393+0000 7f069cf46700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f069814af80 0x7f0698066a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:26.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.393+0000 7f069cf46700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0698066f70 con 0x7f069805d070 2022-06-29T16:06:26.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.393+0000 7f069cf46700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0698063850 con 0x7f06980b2130 2022-06-29T16:06:26.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.393+0000 7f069cf46700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f069815a7b0 con 0x7f069814af80 2022-06-29T16:06:26.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.393+0000 7f06977fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f069805d070 0x7f06980b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:26.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.393+0000 7f06977fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f069805d070 0x7f06980b41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51978/0 (socket says 172.21.15.60:51978) 2022-06-29T16:06:26.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.393+0000 7f06977fe700 1 -- 172.21.15.60:0/2650651131 learned_addr learned my addr 172.21.15.60:0/2650651131 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:26.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.393+0000 7f0697fff700 1 --2- 172.21.15.60:0/2650651131 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f069814af80 0x7f0698066a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:26.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.393+0000 7f0696ffd700 1 --2- 172.21.15.60:0/2650651131 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06980b2130 0x7f0698066590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:26.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.393+0000 7f06977fe700 1 -- 172.21.15.60:0/2650651131 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f069814af80 msgr2=0x7f0698066a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:26.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.393+0000 7f06977fe700 1 --2- 172.21.15.60:0/2650651131 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f069814af80 0x7f0698066a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:26.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.393+0000 7f06977fe700 1 -- 172.21.15.60:0/2650651131 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06980b2130 msgr2=0x7f0698066590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:26.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.393+0000 7f06977fe700 1 --2- 172.21.15.60:0/2650651131 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06980b2130 0x7f0698066590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:26.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.393+0000 7f06977fe700 1 -- 172.21.15.60:0/2650651131 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0690009470 con 0x7f069805d070 2022-06-29T16:06:26.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.393+0000 7f0696ffd700 1 --2- 172.21.15.60:0/2650651131 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06980b2130 0x7f0698066590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:06:26.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.393+0000 7f06977fe700 1 --2- 172.21.15.60:0/2650651131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f069805d070 0x7f06980b41c0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 rx=0x7f0688001820 tx=0x7f0690016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:26.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.393+0000 7f0694ff9700 1 -- 172.21.15.60:0/2650651131 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f069000f680 con 0x7f069805d070 2022-06-29T16:06:26.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.393+0000 7f0694ff9700 1 -- 172.21.15.60:0/2650651131 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06900032f0 con 0x7f069805d070 2022-06-29T16:06:26.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.393+0000 7f069cf46700 1 -- 172.21.15.60:0/2650651131 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06980635b0 con 0x7f069805d070 2022-06-29T16:06:26.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.393+0000 7f0694ff9700 1 -- 172.21.15.60:0/2650651131 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0690021670 con 0x7f069805d070 2022-06-29T16:06:26.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.393+0000 7f069cf46700 1 -- 172.21.15.60:0/2650651131 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0698149ec0 con 0x7f069805d070 2022-06-29T16:06:26.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.397+0000 7f069cf46700 1 -- 172.21.15.60:0/2650651131 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f069814a5f0 con 0x7f069805d070 2022-06-29T16:06:26.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.397+0000 7f0694ff9700 1 -- 172.21.15.60:0/2650651131 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0690028070 con 0x7f069805d070 2022-06-29T16:06:26.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.397+0000 7f0694ff9700 1 --2- 172.21.15.60:0/2650651131 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0678053a70 0x7f0678055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:26.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.397+0000 7f0694ff9700 1 -- 172.21.15.60:0/2650651131 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f069006ccf0 con 0x7f069805d070 2022-06-29T16:06:26.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.397+0000 7f0696ffd700 1 --2- 172.21.15.60:0/2650651131 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0678053a70 0x7f0678055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:26.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.397+0000 7f0696ffd700 1 --2- 172.21.15.60:0/2650651131 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0678053a70 0x7f0678055f30 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0x7f068c00b000 tx=0x7f068c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:26.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.405+0000 7f0694ff9700 1 -- 172.21.15.60:0/2650651131 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0690043e30 con 0x7f069805d070 2022-06-29T16:06:26.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.533+0000 7f069cf46700 1 -- 172.21.15.60:0/2650651131 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f069815a4b0 con 0x7f069805d070 2022-06-29T16:06:26.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.533+0000 7f0694ff9700 1 -- 172.21.15.60:0/2650651131 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v105) v1 ==== 84+0+143 (secure 0 0 0) 0x7f0690015070 con 0x7f069805d070 2022-06-29T16:06:26.538 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:26.539 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2022-06-29T16:06:26.539 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:26.539 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2022-06-29T16:06:26.539 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": "device_health_metrics" 2022-06-29T16:06:26.540 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:26.540 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:26.540 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 2, 2022-06-29T16:06:26.540 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": "rbd" 2022-06-29T16:06:26.540 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:26.541 INFO:teuthology.orchestra.run.smithi060.stdout:] 2022-06-29T16:06:26.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.533+0000 7f069cf46700 1 -- 172.21.15.60:0/2650651131 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0678053a70 msgr2=0x7f0678055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:26.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.533+0000 7f069cf46700 1 --2- 172.21.15.60:0/2650651131 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0678053a70 0x7f0678055f30 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0x7f068c00b000 tx=0x7f068c008040).stop 2022-06-29T16:06:26.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.533+0000 7f069cf46700 1 -- 172.21.15.60:0/2650651131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f069805d070 msgr2=0x7f06980b41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:26.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.533+0000 7f069cf46700 1 --2- 172.21.15.60:0/2650651131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f069805d070 0x7f06980b41c0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 rx=0x7f0688001820 tx=0x7f0690016000).stop 2022-06-29T16:06:26.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.533+0000 7f069cf46700 1 -- 172.21.15.60:0/2650651131 shutdown_connections 2022-06-29T16:06:26.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.533+0000 7f069cf46700 1 --2- 172.21.15.60:0/2650651131 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0678053a70 0x7f0678055f30 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:26.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.533+0000 7f069cf46700 1 --2- 172.21.15.60:0/2650651131 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f069814af80 0x7f0698066a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:26.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.533+0000 7f069cf46700 1 --2- 172.21.15.60:0/2650651131 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06980b2130 0x7f0698066590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:26.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.533+0000 7f069cf46700 1 --2- 172.21.15.60:0/2650651131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f069805d070 0x7f06980b41c0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:26.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.533+0000 7f069cf46700 1 -- 172.21.15.60:0/2650651131 >> 172.21.15.60:0/2650651131 conn(0x7f06980b5890 msgr2=0x7f06981452d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:26.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.537+0000 7f069cf46700 1 -- 172.21.15.60:0/2650651131 shutdown_connections 2022-06-29T16:06:26.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.537+0000 7f069cf46700 1 -- 172.21.15.60:0/2650651131 wait complete. 2022-06-29T16:06:26.556 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-06-29T16:06:26.557 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-06-29T16:06:26.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.661+0000 7f25f347f700 1 Processor -- start 2022-06-29T16:06:26.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.661+0000 7f25f347f700 1 -- start start 2022-06-29T16:06:26.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25f347f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25ec0ae7c0 0x7f25ec141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:26.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25f347f700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ec143a40 con 0x7f25ec0b4010 2022-06-29T16:06:26.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25f347f700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ec0aee60 con 0x7f25ec0b46c0 2022-06-29T16:06:26.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25f347f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec0af100 con 0x7f25ec0ae7c0 2022-06-29T16:06:26.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25f0a1a700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f25ec0b4010 0x7f25ec142490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60076/0 (socket says 172.21.15.60:60076) 2022-06-29T16:06:26.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25f0a1a700 1 -- 172.21.15.60:0/1149659981 learned_addr learned my addr 172.21.15.60:0/1149659981 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:26.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25f121b700 1 --2- 172.21.15.60:0/1149659981 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25ec0ae7c0 0x7f25ec141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:26.672 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25e3fff700 1 -- 172.21.15.60:0/1149659981 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1578273433 0 0) 0x7f25ec143a40 con 0x7f25ec0b4010 2022-06-29T16:06:26.672 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25e3fff700 1 -- 172.21.15.60:0/1149659981 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25d0003260 con 0x7f25ec0b4010 2022-06-29T16:06:26.677 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25e3fff700 1 -- 172.21.15.60:0/1149659981 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2804280660 0 0) 0x7f25ec0aee60 con 0x7f25ec0b46c0 2022-06-29T16:06:26.677 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25e3fff700 1 -- 172.21.15.60:0/1149659981 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25ec143a40 con 0x7f25ec0b46c0 2022-06-29T16:06:26.677 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25e3fff700 1 -- 172.21.15.60:0/1149659981 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1433418784 0 0) 0x7f25d0003260 con 0x7f25ec0b4010 2022-06-29T16:06:26.677 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25e3fff700 1 -- 172.21.15.60:0/1149659981 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25ec0ae7c0 msgr2=0x7f25ec141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:26.678 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25e3fff700 1 --2- 172.21.15.60:0/1149659981 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25ec0ae7c0 0x7f25ec141fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:26.678 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25e3fff700 1 -- 172.21.15.60:0/1149659981 >> v1:172.21.15.92:6789/0 conn(0x7f25ec0b46c0 legacy=0x7f25ec0b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:26.678 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25e3fff700 1 -- 172.21.15.60:0/1149659981 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ec0aeba0 con 0x7f25ec0b4010 2022-06-29T16:06:26.678 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25e3fff700 1 -- 172.21.15.60:0/1149659981 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f25e8001660 con 0x7f25ec0b4010 2022-06-29T16:06:26.679 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25e3fff700 1 -- 172.21.15.60:0/1149659981 >> v1:172.21.15.60:6789/0 conn(0x7f25ec0b4010 legacy=0x7f25ec142490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:26.679 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25e3fff700 1 --2- 172.21.15.60:0/1149659981 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25d0003910 0x7f25d0003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:26.679 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25e3fff700 1 --2- 172.21.15.60:0/1149659981 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25ec0ae7c0 0x7f25d0004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:26.679 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25e3fff700 1 --2- 172.21.15.60:0/1149659981 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25d00049f0 0x7f25d0004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:26.680 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25e3fff700 1 -- 172.21.15.60:0/1149659981 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec0af100 con 0x7f25d00049f0 2022-06-29T16:06:26.680 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25e3fff700 1 -- 172.21.15.60:0/1149659981 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d000b800 con 0x7f25d0003910 2022-06-29T16:06:26.680 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25e3fff700 1 -- 172.21.15.60:0/1149659981 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25d000b970 con 0x7f25ec0ae7c0 2022-06-29T16:06:26.680 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25f1a1c700 1 --2- 172.21.15.60:0/1149659981 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25d00049f0 0x7f25d0004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:26.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25f0a1a700 1 --2- 172.21.15.60:0/1149659981 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25ec0ae7c0 0x7f25d0004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:26.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25f121b700 1 --2- 172.21.15.60:0/1149659981 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25d0003910 0x7f25d0003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:26.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25f1a1c700 1 -- 172.21.15.60:0/1149659981 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25ec0ae7c0 msgr2=0x7f25d0004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:26.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25f1a1c700 1 --2- 172.21.15.60:0/1149659981 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25ec0ae7c0 0x7f25d0004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:26.682 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25f1a1c700 1 -- 172.21.15.60:0/1149659981 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25d0003910 msgr2=0x7f25d0003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:26.682 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25f1a1c700 1 --2- 172.21.15.60:0/1149659981 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25d0003910 0x7f25d0003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:26.682 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25f1a1c700 1 -- 172.21.15.60:0/1149659981 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d0003260 con 0x7f25d00049f0 2022-06-29T16:06:26.682 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25f121b700 1 --2- 172.21.15.60:0/1149659981 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25d0003910 0x7f25d0003d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:06:26.682 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25f1a1c700 1 --2- 172.21.15.60:0/1149659981 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25d00049f0 0x7f25d0004e70 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 rx=0x7f25e4001490 tx=0x7f25e4007d80).ready entity=mon.0 client_cookie=52f2f3974ce24bfa server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:26.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25e3fff700 1 -- 172.21.15.60:0/1149659981 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25e4001780 con 0x7f25d00049f0 2022-06-29T16:06:26.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25e3fff700 1 -- 172.21.15.60:0/1149659981 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e4006890 con 0x7f25d00049f0 2022-06-29T16:06:26.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.665+0000 7f25e3fff700 1 -- 172.21.15.60:0/1149659981 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25e4006bf0 con 0x7f25d00049f0 2022-06-29T16:06:26.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.669+0000 7f25f347f700 1 -- 172.21.15.60:0/1149659981 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25d00049f0 msgr2=0x7f25d0004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:26.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.669+0000 7f25f347f700 1 --2- 172.21.15.60:0/1149659981 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25d00049f0 0x7f25d0004e70 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 rx=0x7f25e4001490 tx=0x7f25e4007d80).stop 2022-06-29T16:06:26.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.669+0000 7f25f1a1c700 1 -- 172.21.15.60:0/1149659981 reap_dead start 2022-06-29T16:06:26.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.669+0000 7f25f347f700 1 -- 172.21.15.60:0/1149659981 shutdown_connections 2022-06-29T16:06:26.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.669+0000 7f25f347f700 1 -- 172.21.15.60:0/1149659981 >> 172.21.15.60:0/1149659981 conn(0x7f25ec0b9340 msgr2=0x7f25ec0b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:26.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.669+0000 7f25f347f700 1 -- 172.21.15.60:0/1149659981 shutdown_connections 2022-06-29T16:06:26.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.669+0000 7f25f347f700 1 -- 172.21.15.60:0/1149659981 wait complete. 2022-06-29T16:06:26.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.669+0000 7f25f347f700 1 Processor -- start 2022-06-29T16:06:26.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.669+0000 7f25f347f700 1 -- start start 2022-06-29T16:06:26.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.673+0000 7f25f347f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25ec05c570 0x7f25ec1555c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:26.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.673+0000 7f25f347f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25ec13e150 0x7f25ec0b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:26.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.673+0000 7f25f347f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25ec0b0e30 0x7f25ec13cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:26.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.673+0000 7f25f347f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec0b12b0 con 0x7f25ec0b0e30 2022-06-29T16:06:26.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.673+0000 7f25f347f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec0b5cf0 con 0x7f25ec05c570 2022-06-29T16:06:26.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.673+0000 7f25f347f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec05c950 con 0x7f25ec13e150 2022-06-29T16:06:26.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.673+0000 7f25f1a1c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25ec0b0e30 0x7f25ec13cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:26.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.673+0000 7f25f1a1c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25ec0b0e30 0x7f25ec13cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52002/0 (socket says 172.21.15.60:52002) 2022-06-29T16:06:26.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.673+0000 7f25f1a1c700 1 -- 172.21.15.60:0/1674535310 learned_addr learned my addr 172.21.15.60:0/1674535310 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:26.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.673+0000 7f25f0a1a700 1 --2- 172.21.15.60:0/1674535310 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25ec13e150 0x7f25ec0b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:26.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.673+0000 7f25f121b700 1 --2- 172.21.15.60:0/1674535310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25ec05c570 0x7f25ec1555c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:26.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.673+0000 7f25f1a1c700 1 -- 172.21.15.60:0/1674535310 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25ec13e150 msgr2=0x7f25ec0b0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:26.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.673+0000 7f25f1a1c700 1 --2- 172.21.15.60:0/1674535310 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25ec13e150 0x7f25ec0b0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:26.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.673+0000 7f25f1a1c700 1 -- 172.21.15.60:0/1674535310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25ec05c570 msgr2=0x7f25ec1555c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:26.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.673+0000 7f25f1a1c700 1 --2- 172.21.15.60:0/1674535310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25ec05c570 0x7f25ec1555c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:26.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.673+0000 7f25f1a1c700 1 -- 172.21.15.60:0/1674535310 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ec0b04a0 con 0x7f25ec0b0e30 2022-06-29T16:06:26.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.673+0000 7f25f1a1c700 1 --2- 172.21.15.60:0/1674535310 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25ec0b0e30 0x7f25ec13cb30 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 rx=0x7f25ec0b9270 tx=0x7f25e4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:26.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.673+0000 7f25e27fc700 1 -- 172.21.15.60:0/1674535310 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25e4013810 con 0x7f25ec0b0e30 2022-06-29T16:06:26.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.673+0000 7f25e27fc700 1 -- 172.21.15.60:0/1674535310 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e4013970 con 0x7f25ec0b0e30 2022-06-29T16:06:26.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.673+0000 7f25f347f700 1 -- 172.21.15.60:0/1674535310 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25ec0b5f20 con 0x7f25ec0b0e30 2022-06-29T16:06:26.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.673+0000 7f25e27fc700 1 -- 172.21.15.60:0/1674535310 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25e4013cd0 con 0x7f25ec0b0e30 2022-06-29T16:06:26.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.673+0000 7f25f347f700 1 -- 172.21.15.60:0/1674535310 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25ec13d8b0 con 0x7f25ec0b0e30 2022-06-29T16:06:26.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.677+0000 7f25f347f700 1 -- 172.21.15.60:0/1674535310 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25ec0b0630 con 0x7f25ec0b0e30 2022-06-29T16:06:26.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.677+0000 7f25e27fc700 1 -- 172.21.15.60:0/1674535310 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f25e4025030 con 0x7f25ec0b0e30 2022-06-29T16:06:26.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.677+0000 7f25e27fc700 1 --2- 172.21.15.60:0/1674535310 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f25d4053a70 0x7f25d4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:26.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.677+0000 7f25e27fc700 1 -- 172.21.15.60:0/1674535310 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f25e406c940 con 0x7f25ec0b0e30 2022-06-29T16:06:26.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.677+0000 7f25f121b700 1 --2- 172.21.15.60:0/1674535310 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f25d4053a70 0x7f25d4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:26.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.677+0000 7f25f121b700 1 --2- 172.21.15.60:0/1674535310 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f25d4053a70 0x7f25d4055f30 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7f25dc00b000 tx=0x7f25dc008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:26.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.681+0000 7f25e27fc700 1 -- 172.21.15.60:0/1674535310 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f25e4043b90 con 0x7f25ec0b0e30 2022-06-29T16:06:26.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.809+0000 7f25f347f700 1 -- 172.21.15.60:0/1674535310 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f25ec0573d0 con 0x7f25ec0b0e30 2022-06-29T16:06:26.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.925+0000 7f25e27fc700 1 -- 172.21.15.60:0/1674535310 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v106) v1 ==== 133+0+0 (secure 0 0 0) 0x7f25e4023030 con 0x7f25ec0b0e30 2022-06-29T16:06:26.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.929+0000 7f25f347f700 1 -- 172.21.15.60:0/1674535310 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f25d4053a70 msgr2=0x7f25d4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:26.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.929+0000 7f25f347f700 1 --2- 172.21.15.60:0/1674535310 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f25d4053a70 0x7f25d4055f30 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7f25dc00b000 tx=0x7f25dc008040).stop 2022-06-29T16:06:26.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.929+0000 7f25f347f700 1 -- 172.21.15.60:0/1674535310 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25ec0b0e30 msgr2=0x7f25ec13cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:26.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.929+0000 7f25f347f700 1 --2- 172.21.15.60:0/1674535310 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25ec0b0e30 0x7f25ec13cb30 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 rx=0x7f25ec0b9270 tx=0x7f25e4016000).stop 2022-06-29T16:06:26.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.929+0000 7f25f347f700 1 -- 172.21.15.60:0/1674535310 shutdown_connections 2022-06-29T16:06:26.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.929+0000 7f25f347f700 1 --2- 172.21.15.60:0/1674535310 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f25d4053a70 0x7f25d4055f30 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:26.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.929+0000 7f25f347f700 1 --2- 172.21.15.60:0/1674535310 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25ec0b0e30 0x7f25ec13cb30 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:26.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.929+0000 7f25f347f700 1 --2- 172.21.15.60:0/1674535310 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25ec13e150 0x7f25ec0b0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:26.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.929+0000 7f25f347f700 1 --2- 172.21.15.60:0/1674535310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25ec05c570 0x7f25ec1555c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:26.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.929+0000 7f25f347f700 1 -- 172.21.15.60:0/1674535310 >> 172.21.15.60:0/1674535310 conn(0x7f25ec0b9340 msgr2=0x7f25ec0b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:26.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.929+0000 7f25f347f700 1 -- 172.21.15.60:0/1674535310 shutdown_connections 2022-06-29T16:06:26.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:26.929+0000 7f25f347f700 1 -- 172.21.15.60:0/1674535310 wait complete. 2022-06-29T16:06:26.941 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2022-06-29T16:06:26.949 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-06-29T16:06:27.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.057+0000 7fc88994c700 1 Processor -- start 2022-06-29T16:06:27.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.057+0000 7fc88994c700 1 -- start start 2022-06-29T16:06:27.064 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.057+0000 7fc88994c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc884060710 0x7fc8840b6270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:27.064 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.057+0000 7fc88994c700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc88413ec70 0x7fc88413f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:27.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.057+0000 7fc88994c700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc884060c60 con 0x7fc8840b5e90 2022-06-29T16:06:27.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.057+0000 7fc88994c700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8840af0c0 con 0x7fc884060710 2022-06-29T16:06:27.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.057+0000 7fc88994c700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8840b1d30 con 0x7fc88413ec70 2022-06-29T16:06:27.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.057+0000 7fc8827fc700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc884060710 0x7fc8840b6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:27.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.057+0000 7fc8837fe700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc88413ec70 0x7fc88413f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:27.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.057+0000 7fc882ffd700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fc8840b5e90 0x7fc8840ae9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:36108/0 (socket says 172.21.15.60:36108) 2022-06-29T16:06:27.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.057+0000 7fc882ffd700 1 -- 172.21.15.60:0/3530870080 learned_addr learned my addr 172.21.15.60:0/3530870080 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:27.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.057+0000 7fc8827fc700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc884060710 0x7fc8840b6270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52008/0 (socket says 172.21.15.60:52008) 2022-06-29T16:06:27.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.057+0000 7fc8827fc700 1 -- 172.21.15.60:0/3530870080 >> v2:172.21.15.92:3300/0 conn(0x7fc88413ec70 msgr2=0x7fc88413f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:27.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.057+0000 7fc8827fc700 1 --2- 172.21.15.60:0/3530870080 >> v2:172.21.15.92:3300/0 conn(0x7fc88413ec70 0x7fc88413f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:27.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.057+0000 7fc8827fc700 1 -- 172.21.15.60:0/3530870080 >> v1:172.21.15.92:6789/0 conn(0x7fc8840b5e90 legacy=0x7fc8840ae9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:27.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.057+0000 7fc8827fc700 1 -- 172.21.15.60:0/3530870080 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8840ae2e0 con 0x7fc884060710 2022-06-29T16:06:27.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.061+0000 7fc8827fc700 1 --2- 172.21.15.60:0/3530870080 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc884060710 0x7fc8840b6270 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 rx=0x7fc8840b20f0 tx=0x7fc878006df0).ready entity=mon.0 client_cookie=beef9ae9cd8b56af server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:27.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.061+0000 7fc881ffb700 1 -- 172.21.15.60:0/3530870080 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc87800d5c0 con 0x7fc884060710 2022-06-29T16:06:27.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.061+0000 7fc881ffb700 1 -- 172.21.15.60:0/3530870080 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc878000e90 con 0x7fc884060710 2022-06-29T16:06:27.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.061+0000 7fc881ffb700 1 -- 172.21.15.60:0/3530870080 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc87800d910 con 0x7fc884060710 2022-06-29T16:06:27.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.061+0000 7fc88994c700 1 -- 172.21.15.60:0/3530870080 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc884060710 msgr2=0x7fc8840b6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:27.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.061+0000 7fc88994c700 1 --2- 172.21.15.60:0/3530870080 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc884060710 0x7fc8840b6270 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 rx=0x7fc8840b20f0 tx=0x7fc878006df0).stop 2022-06-29T16:06:27.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.061+0000 7fc88994c700 1 -- 172.21.15.60:0/3530870080 shutdown_connections 2022-06-29T16:06:27.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.061+0000 7fc88994c700 1 --2- 172.21.15.60:0/3530870080 >> v2:172.21.15.92:3300/0 conn(0x7fc88413ec70 0x7fc88413f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:27.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.061+0000 7fc88994c700 1 --2- 172.21.15.60:0/3530870080 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc884060710 0x7fc8840b6270 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:27.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.061+0000 7fc88994c700 1 -- 172.21.15.60:0/3530870080 >> 172.21.15.60:0/3530870080 conn(0x7fc884063b50 msgr2=0x7fc8840b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:27.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.061+0000 7fc88994c700 1 -- 172.21.15.60:0/3530870080 shutdown_connections 2022-06-29T16:06:27.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.061+0000 7fc88994c700 1 -- 172.21.15.60:0/3530870080 wait complete. 2022-06-29T16:06:27.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.061+0000 7fc88994c700 1 Processor -- start 2022-06-29T16:06:27.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.061+0000 7fc88994c700 1 -- start start 2022-06-29T16:06:27.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.061+0000 7fc88994c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc884060710 0x7fc8840666c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:27.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.061+0000 7fc88994c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8840b5e90 0x7fc8840b0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:27.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.061+0000 7fc88994c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc88413ec70 0x7fc88413aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:27.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.061+0000 7fc88994c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88414ea30 con 0x7fc8840b5e90 2022-06-29T16:06:27.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.061+0000 7fc88994c700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc884065f80 con 0x7fc88413ec70 2022-06-29T16:06:27.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.061+0000 7fc88994c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc884066220 con 0x7fc884060710 2022-06-29T16:06:27.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.061+0000 7fc8827fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8840b5e90 0x7fc8840b0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:27.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.061+0000 7fc8827fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8840b5e90 0x7fc8840b0140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52014/0 (socket says 172.21.15.60:52014) 2022-06-29T16:06:27.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.061+0000 7fc8827fc700 1 -- 172.21.15.60:0/139492938 learned_addr learned my addr 172.21.15.60:0/139492938 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:27.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.065+0000 7fc8837fe700 1 --2- 172.21.15.60:0/139492938 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc88413ec70 0x7fc88413aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:27.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.065+0000 7fc882ffd700 1 --2- 172.21.15.60:0/139492938 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc884060710 0x7fc8840666c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:27.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.065+0000 7fc8827fc700 1 -- 172.21.15.60:0/139492938 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc884060710 msgr2=0x7fc8840666c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:27.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.065+0000 7fc8827fc700 1 --2- 172.21.15.60:0/139492938 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc884060710 0x7fc8840666c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:27.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.065+0000 7fc8827fc700 1 -- 172.21.15.60:0/139492938 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc88413ec70 msgr2=0x7fc88413aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:27.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.065+0000 7fc8827fc700 1 --2- 172.21.15.60:0/139492938 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc88413ec70 0x7fc88413aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:27.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.065+0000 7fc8827fc700 1 -- 172.21.15.60:0/139492938 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87800b180 con 0x7fc8840b5e90 2022-06-29T16:06:27.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.065+0000 7fc8827fc700 1 --2- 172.21.15.60:0/139492938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8840b5e90 0x7fc8840b0140 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 rx=0x7fc8840b20f0 tx=0x7fc878014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:27.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.065+0000 7fc873fff700 1 -- 172.21.15.60:0/139492938 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc878016070 con 0x7fc8840b5e90 2022-06-29T16:06:27.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.065+0000 7fc873fff700 1 -- 172.21.15.60:0/139492938 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc87800d940 con 0x7fc8840b5e90 2022-06-29T16:06:27.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.065+0000 7fc88994c700 1 -- 172.21.15.60:0/139492938 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc884065cb0 con 0x7fc8840b5e90 2022-06-29T16:06:27.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.065+0000 7fc873fff700 1 -- 172.21.15.60:0/139492938 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc878013d60 con 0x7fc8840b5e90 2022-06-29T16:06:27.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.065+0000 7fc88994c700 1 -- 172.21.15.60:0/139492938 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc884141f50 con 0x7fc8840b5e90 2022-06-29T16:06:27.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.065+0000 7fc88994c700 1 -- 172.21.15.60:0/139492938 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8840ae510 con 0x7fc8840b5e90 2022-06-29T16:06:27.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.069+0000 7fc873fff700 1 -- 172.21.15.60:0/139492938 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc878025030 con 0x7fc8840b5e90 2022-06-29T16:06:27.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.069+0000 7fc873fff700 1 --2- 172.21.15.60:0/139492938 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc860053a70 0x7fc860055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:27.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.069+0000 7fc873fff700 1 -- 172.21.15.60:0/139492938 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5607+0+0 (secure 0 0 0) 0x7fc87806cd70 con 0x7fc8840b5e90 2022-06-29T16:06:27.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.069+0000 7fc882ffd700 1 --2- 172.21.15.60:0/139492938 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc860053a70 0x7fc860055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:27.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.069+0000 7fc882ffd700 1 --2- 172.21.15.60:0/139492938 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc860053a70 0x7fc860055f30 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0x7fc87400d000 tx=0x7fc874009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:27.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.073+0000 7fc873fff700 1 -- 172.21.15.60:0/139492938 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc878043d50 con 0x7fc8840b5e90 2022-06-29T16:06:27.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.197+0000 7fc88994c700 1 -- 172.21.15.60:0/139492938 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7fc884142690 con 0x7fc8840b5e90 2022-06-29T16:06:27.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.925+0000 7fc873fff700 1 -- 172.21.15.60:0/139492938 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v107) v1 ==== 125+0+0 (secure 0 0 0) 0x7fc878023070 con 0x7fc8840b5e90 2022-06-29T16:06:27.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.937+0000 7fc88994c700 1 -- 172.21.15.60:0/139492938 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc860053a70 msgr2=0x7fc860055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:27.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.937+0000 7fc88994c700 1 --2- 172.21.15.60:0/139492938 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc860053a70 0x7fc860055f30 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0x7fc87400d000 tx=0x7fc874009040).stop 2022-06-29T16:06:27.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.937+0000 7fc88994c700 1 -- 172.21.15.60:0/139492938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8840b5e90 msgr2=0x7fc8840b0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:27.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.937+0000 7fc88994c700 1 --2- 172.21.15.60:0/139492938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8840b5e90 0x7fc8840b0140 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 rx=0x7fc8840b20f0 tx=0x7fc878014000).stop 2022-06-29T16:06:27.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.937+0000 7fc88994c700 1 -- 172.21.15.60:0/139492938 shutdown_connections 2022-06-29T16:06:27.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.937+0000 7fc88994c700 1 --2- 172.21.15.60:0/139492938 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc860053a70 0x7fc860055f30 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:27.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.937+0000 7fc88994c700 1 --2- 172.21.15.60:0/139492938 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc88413ec70 0x7fc88413aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:27.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.937+0000 7fc88994c700 1 --2- 172.21.15.60:0/139492938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8840b5e90 0x7fc8840b0140 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:27.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.937+0000 7fc88994c700 1 --2- 172.21.15.60:0/139492938 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc884060710 0x7fc8840666c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:27.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.937+0000 7fc88994c700 1 -- 172.21.15.60:0/139492938 >> 172.21.15.60:0/139492938 conn(0x7fc884063b50 msgr2=0x7fc8840aca20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:27.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.937+0000 7fc88994c700 1 -- 172.21.15.60:0/139492938 shutdown_connections 2022-06-29T16:06:27.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:27.937+0000 7fc88994c700 1 -- 172.21.15.60:0/139492938 wait complete. 2022-06-29T16:06:27.944 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2022-06-29T16:06:27.953 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-06-29T16:06:28.063 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.057+0000 7f05da34b700 1 Processor -- start 2022-06-29T16:06:28.063 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.057+0000 7f05da34b700 1 -- start start 2022-06-29T16:06:28.064 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.061+0000 7f05da34b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05d40b0cd0 0x7f05d4066b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:28.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.061+0000 7f05da34b700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05d405ccc0 con 0x7f05d4067030 2022-06-29T16:06:28.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.061+0000 7f05da34b700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05d4065660 con 0x7f05d40b1bb0 2022-06-29T16:06:28.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.061+0000 7f05da34b700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d40674a0 con 0x7f05d40b0cd0 2022-06-29T16:06:28.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.061+0000 7f05d9b4a700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f05d4067030 0x7f05d41516e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60110/0 (socket says 172.21.15.60:60110) 2022-06-29T16:06:28.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.061+0000 7f05d9b4a700 1 -- 172.21.15.60:0/2906976132 learned_addr learned my addr 172.21.15.60:0/2906976132 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:28.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.061+0000 7f05d8b48700 1 --2- 172.21.15.60:0/2906976132 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05d40b0cd0 0x7f05d4066b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:28.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.061+0000 7f05bffff700 1 -- 172.21.15.60:0/2906976132 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1567777631 0 0) 0x7f05d405ccc0 con 0x7f05d4067030 2022-06-29T16:06:28.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.061+0000 7f05bffff700 1 -- 172.21.15.60:0/2906976132 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05b8003260 con 0x7f05d4067030 2022-06-29T16:06:28.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.061+0000 7f05d8b48700 1 -- 172.21.15.60:0/2906976132 >> v1:172.21.15.92:6789/0 conn(0x7f05d40b1bb0 legacy=0x7f05d405c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:28.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.061+0000 7f05d8b48700 1 -- 172.21.15.60:0/2906976132 >> v1:172.21.15.60:6789/0 conn(0x7f05d4067030 legacy=0x7f05d41516e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:28.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.061+0000 7f05d8b48700 1 -- 172.21.15.60:0/2906976132 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d4066400 con 0x7f05d40b0cd0 2022-06-29T16:06:28.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.061+0000 7f05d8b48700 1 --2- 172.21.15.60:0/2906976132 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05d40b0cd0 0x7f05d4066b40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f05c8002f70 tx=0x7f05c8006df0).ready entity=mon.2 client_cookie=dd72080d5d18b00a server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:28.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.061+0000 7f05bffff700 1 -- 172.21.15.60:0/2906976132 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05c800d570 con 0x7f05d40b0cd0 2022-06-29T16:06:28.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.061+0000 7f05bffff700 1 -- 172.21.15.60:0/2906976132 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05c8000e90 con 0x7f05d40b0cd0 2022-06-29T16:06:28.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.061+0000 7f05bffff700 1 -- 172.21.15.60:0/2906976132 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05c800d8c0 con 0x7f05d40b0cd0 2022-06-29T16:06:28.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.061+0000 7f05da34b700 1 -- 172.21.15.60:0/2906976132 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05d40b0cd0 msgr2=0x7f05d4066b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:28.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.061+0000 7f05da34b700 1 --2- 172.21.15.60:0/2906976132 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05d40b0cd0 0x7f05d4066b40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f05c8002f70 tx=0x7f05c8006df0).stop 2022-06-29T16:06:28.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.061+0000 7f05da34b700 1 -- 172.21.15.60:0/2906976132 shutdown_connections 2022-06-29T16:06:28.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.061+0000 7f05da34b700 1 --2- 172.21.15.60:0/2906976132 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05d40b0cd0 0x7f05d4066b40 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:28.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.061+0000 7f05da34b700 1 -- 172.21.15.60:0/2906976132 >> 172.21.15.60:0/2906976132 conn(0x7f05d40b5e70 msgr2=0x7f05d40b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:28.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.061+0000 7f05da34b700 1 -- 172.21.15.60:0/2906976132 shutdown_connections 2022-06-29T16:06:28.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.061+0000 7f05da34b700 1 -- 172.21.15.60:0/2906976132 wait complete. 2022-06-29T16:06:28.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.065+0000 7f05da34b700 1 Processor -- start 2022-06-29T16:06:28.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.065+0000 7f05da34b700 1 -- start start 2022-06-29T16:06:28.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.065+0000 7f05da34b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05d4067030 0x7f05d40aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:28.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.065+0000 7f05da34b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05d40b0cd0 0x7f05d414a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:28.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.065+0000 7f05da34b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05d40b1bb0 0x7f05d4110b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:28.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.065+0000 7f05da34b700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d410f3e0 con 0x7f05d40b1bb0 2022-06-29T16:06:28.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.065+0000 7f05da34b700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d410f680 con 0x7f05d4067030 2022-06-29T16:06:28.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.065+0000 7f05da34b700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d414af90 con 0x7f05d40b0cd0 2022-06-29T16:06:28.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.065+0000 7f05d9b4a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05d40b1bb0 0x7f05d4110b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:28.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.065+0000 7f05d9b4a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05d40b1bb0 0x7f05d4110b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52030/0 (socket says 172.21.15.60:52030) 2022-06-29T16:06:28.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.065+0000 7f05d9b4a700 1 -- 172.21.15.60:0/728818824 learned_addr learned my addr 172.21.15.60:0/728818824 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:28.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.065+0000 7f05d8b48700 1 --2- 172.21.15.60:0/728818824 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05d40b0cd0 0x7f05d414a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:28.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.065+0000 7f05d9349700 1 --2- 172.21.15.60:0/728818824 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05d4067030 0x7f05d40aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:28.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.065+0000 7f05d9b4a700 1 -- 172.21.15.60:0/728818824 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05d40b0cd0 msgr2=0x7f05d414a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:28.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.065+0000 7f05d9b4a700 1 --2- 172.21.15.60:0/728818824 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05d40b0cd0 0x7f05d414a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:28.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.065+0000 7f05d9b4a700 1 -- 172.21.15.60:0/728818824 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05d4067030 msgr2=0x7f05d40aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:28.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.065+0000 7f05d9b4a700 1 --2- 172.21.15.60:0/728818824 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05d4067030 0x7f05d40aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:28.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.065+0000 7f05d9b4a700 1 -- 172.21.15.60:0/728818824 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c800b180 con 0x7f05d40b1bb0 2022-06-29T16:06:28.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.065+0000 7f05d9b4a700 1 --2- 172.21.15.60:0/728818824 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05d40b1bb0 0x7f05d4110b10 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 rx=0x7f05cc001490 tx=0x7f05cc007e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:28.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.069+0000 7f05c6ffd700 1 -- 172.21.15.60:0/728818824 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05cc005ba0 con 0x7f05d40b1bb0 2022-06-29T16:06:28.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.069+0000 7f05c6ffd700 1 -- 172.21.15.60:0/728818824 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05cc005d00 con 0x7f05d40b1bb0 2022-06-29T16:06:28.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.069+0000 7f05da34b700 1 -- 172.21.15.60:0/728818824 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05d410f060 con 0x7f05d40b1bb0 2022-06-29T16:06:28.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.069+0000 7f05c6ffd700 1 -- 172.21.15.60:0/728818824 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05cc0072c0 con 0x7f05d40b1bb0 2022-06-29T16:06:28.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.069+0000 7f05da34b700 1 -- 172.21.15.60:0/728818824 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05d40af4d0 con 0x7f05d40b1bb0 2022-06-29T16:06:28.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.069+0000 7f05da34b700 1 -- 172.21.15.60:0/728818824 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05d410f1f0 con 0x7f05d40b1bb0 2022-06-29T16:06:28.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.069+0000 7f05c6ffd700 1 -- 172.21.15.60:0/728818824 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f05cc010070 con 0x7f05d40b1bb0 2022-06-29T16:06:28.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.069+0000 7f05c6ffd700 1 --2- 172.21.15.60:0/728818824 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f05b0053a10 0x7f05b0055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:28.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.069+0000 7f05c6ffd700 1 -- 172.21.15.60:0/728818824 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f05cc06b680 con 0x7f05d40b1bb0 2022-06-29T16:06:28.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.069+0000 7f05d9349700 1 --2- 172.21.15.60:0/728818824 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f05b0053a10 0x7f05b0055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:28.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.073+0000 7f05d9349700 1 --2- 172.21.15.60:0/728818824 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f05b0053a10 0x7f05b0055ed0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0x7f05d000d000 tx=0x7f05d0007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:28.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.077+0000 7f05c6ffd700 1 -- 172.21.15.60:0/728818824 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f05cc042610 con 0x7f05d40b1bb0 2022-06-29T16:06:28.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.245+0000 7f05da34b700 1 -- 172.21.15.60:0/728818824 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f05d40573d0 con 0x7f05d40b1bb0 2022-06-29T16:06:28.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.257+0000 7f05c6ffd700 1 -- 172.21.15.60:0/728818824 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 21 and data pool 22 v50) v1 ==== 189+0+0 (secure 0 0 0) 0x7f05cc045c30 con 0x7f05d40b1bb0 2022-06-29T16:06:28.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.261+0000 7f05da34b700 1 -- 172.21.15.60:0/728818824 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f05b0053a10 msgr2=0x7f05b0055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:28.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.261+0000 7f05da34b700 1 --2- 172.21.15.60:0/728818824 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f05b0053a10 0x7f05b0055ed0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0x7f05d000d000 tx=0x7f05d0007b90).stop 2022-06-29T16:06:28.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.261+0000 7f05da34b700 1 -- 172.21.15.60:0/728818824 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05d40b1bb0 msgr2=0x7f05d4110b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:28.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.261+0000 7f05da34b700 1 --2- 172.21.15.60:0/728818824 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05d40b1bb0 0x7f05d4110b10 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 rx=0x7f05cc001490 tx=0x7f05cc007e00).stop 2022-06-29T16:06:28.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.261+0000 7f05da34b700 1 -- 172.21.15.60:0/728818824 shutdown_connections 2022-06-29T16:06:28.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.261+0000 7f05da34b700 1 --2- 172.21.15.60:0/728818824 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f05b0053a10 0x7f05b0055ed0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:28.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.261+0000 7f05da34b700 1 --2- 172.21.15.60:0/728818824 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f05d40b1bb0 0x7f05d4110b10 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:28.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.261+0000 7f05da34b700 1 --2- 172.21.15.60:0/728818824 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f05d40b0cd0 0x7f05d414a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:28.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.261+0000 7f05da34b700 1 --2- 172.21.15.60:0/728818824 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f05d4067030 0x7f05d40aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:28.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.261+0000 7f05da34b700 1 -- 172.21.15.60:0/728818824 >> 172.21.15.60:0/728818824 conn(0x7f05d40b5e70 msgr2=0x7f05d40b47f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:28.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.261+0000 7f05da34b700 1 -- 172.21.15.60:0/728818824 shutdown_connections 2022-06-29T16:06:28.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.261+0000 7f05da34b700 1 -- 172.21.15.60:0/728818824 wait complete. 2022-06-29T16:06:28.269 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 21 and data pool 22 2022-06-29T16:06:28.280 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:06:28.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.369+0000 7f083abff700 1 Processor -- start 2022-06-29T16:06:28.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.369+0000 7f083abff700 1 -- start start 2022-06-29T16:06:28.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.369+0000 7f083abff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0834060260 0x7f08340b6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:28.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.369+0000 7f083abff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08340b6c00 0x7f083413f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:28.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.369+0000 7f083abff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08341430e0 con 0x7f083413f3d0 2022-06-29T16:06:28.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.369+0000 7f083abff700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0834142ef0 con 0x7f0834060260 2022-06-29T16:06:28.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.369+0000 7f083abff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08340acb90 con 0x7f08340b6c00 2022-06-29T16:06:28.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.369+0000 7f083899b700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f083413f3d0 0x7f0834143820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60134/0 (socket says 172.21.15.60:60134) 2022-06-29T16:06:28.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.369+0000 7f083899b700 1 -- 172.21.15.60:0/2373519835 learned_addr learned my addr 172.21.15.60:0/2373519835 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:28.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.369+0000 7f083919c700 1 --2- 172.21.15.60:0/2373519835 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08340b6c00 0x7f083413f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:28.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.369+0000 7f0833fff700 1 --2- 172.21.15.60:0/2373519835 >> v2:172.21.15.92:3300/0 conn(0x7f0834060260 0x7f08340b6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:28.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.369+0000 7f08337fe700 1 -- 172.21.15.60:0/2373519835 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3510245558 0 0) 0x7f08341430e0 con 0x7f083413f3d0 2022-06-29T16:06:28.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.369+0000 7f08337fe700 1 -- 172.21.15.60:0/2373519835 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0820003260 con 0x7f083413f3d0 2022-06-29T16:06:28.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f08337fe700 1 -- 172.21.15.60:0/2373519835 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1458595931 0 0) 0x7f0820003260 con 0x7f083413f3d0 2022-06-29T16:06:28.382 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f08337fe700 1 -- 172.21.15.60:0/2373519835 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08340b6c00 msgr2=0x7f083413f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:28.382 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f08337fe700 1 --2- 172.21.15.60:0/2373519835 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08340b6c00 0x7f083413f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:28.382 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f08337fe700 1 -- 172.21.15.60:0/2373519835 >> v2:172.21.15.92:3300/0 conn(0x7f0834060260 msgr2=0x7f08340b6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:28.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f08337fe700 1 --2- 172.21.15.60:0/2373519835 >> v2:172.21.15.92:3300/0 conn(0x7f0834060260 0x7f08340b6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:28.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f08337fe700 1 -- 172.21.15.60:0/2373519835 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0834143380 con 0x7f083413f3d0 2022-06-29T16:06:28.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f08337fe700 1 -- 172.21.15.60:0/2373519835 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f0824001660 con 0x7f083413f3d0 2022-06-29T16:06:28.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f08337fe700 1 -- 172.21.15.60:0/2373519835 >> v1:172.21.15.60:6789/0 conn(0x7f083413f3d0 legacy=0x7f0834143820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:28.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f08337fe700 1 --2- 172.21.15.60:0/2373519835 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08340b6c00 0x7f0820003750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:28.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f08337fe700 1 --2- 172.21.15.60:0/2373519835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0820003c80 0x7f0820004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:28.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f08337fe700 1 --2- 172.21.15.60:0/2373519835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08200049f0 0x7f0820004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:28.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f08337fe700 1 -- 172.21.15.60:0/2373519835 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08340acb90 con 0x7f0820003c80 2022-06-29T16:06:28.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f08337fe700 1 -- 172.21.15.60:0/2373519835 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f082000b750 con 0x7f08200049f0 2022-06-29T16:06:28.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f08337fe700 1 -- 172.21.15.60:0/2373519835 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f082000b890 con 0x7f08340b6c00 2022-06-29T16:06:28.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f0833fff700 1 --2- 172.21.15.60:0/2373519835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0820003c80 0x7f0820004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:28.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f083899b700 1 --2- 172.21.15.60:0/2373519835 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08340b6c00 0x7f0820003750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:28.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f083919c700 1 --2- 172.21.15.60:0/2373519835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08200049f0 0x7f0820004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:28.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f0833fff700 1 -- 172.21.15.60:0/2373519835 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08340b6c00 msgr2=0x7f0820003750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:28.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f0833fff700 1 --2- 172.21.15.60:0/2373519835 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08340b6c00 0x7f0820003750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:28.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f0833fff700 1 -- 172.21.15.60:0/2373519835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08200049f0 msgr2=0x7f0820004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:28.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f0833fff700 1 --2- 172.21.15.60:0/2373519835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08200049f0 0x7f0820004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:28.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f0833fff700 1 -- 172.21.15.60:0/2373519835 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0820003260 con 0x7f0820003c80 2022-06-29T16:06:28.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f083919c700 1 --2- 172.21.15.60:0/2373519835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08200049f0 0x7f0820004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:06:28.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f0833fff700 1 --2- 172.21.15.60:0/2373519835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0820003c80 0x7f0820004500 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 rx=0x7f08200054d0 tx=0x7f0828006df0).ready entity=mon.0 client_cookie=ae37aab15872ebb4 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:28.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f08337fe700 1 -- 172.21.15.60:0/2373519835 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f082800e830 con 0x7f0820003c80 2022-06-29T16:06:28.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f08337fe700 1 -- 172.21.15.60:0/2373519835 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f082800be90 con 0x7f0820003c80 2022-06-29T16:06:28.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f08337fe700 1 -- 172.21.15.60:0/2373519835 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f082800eb80 con 0x7f0820003c80 2022-06-29T16:06:28.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f083abff700 1 -- 172.21.15.60:0/2373519835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0820003c80 msgr2=0x7f0820004500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:28.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f083abff700 1 --2- 172.21.15.60:0/2373519835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0820003c80 0x7f0820004500 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 rx=0x7f08200054d0 tx=0x7f0828006df0).stop 2022-06-29T16:06:28.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f083919c700 1 -- 172.21.15.60:0/2373519835 reap_dead start 2022-06-29T16:06:28.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f083abff700 1 -- 172.21.15.60:0/2373519835 shutdown_connections 2022-06-29T16:06:28.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f083abff700 1 -- 172.21.15.60:0/2373519835 >> 172.21.15.60:0/2373519835 conn(0x7f08340643e0 msgr2=0x7f08340b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:28.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f083abff700 1 -- 172.21.15.60:0/2373519835 shutdown_connections 2022-06-29T16:06:28.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.373+0000 7f083abff700 1 -- 172.21.15.60:0/2373519835 wait complete. 2022-06-29T16:06:28.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.377+0000 7f083abff700 1 Processor -- start 2022-06-29T16:06:28.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.377+0000 7f083abff700 1 -- start start 2022-06-29T16:06:28.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.377+0000 7f083abff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0820003c80 0x7f0834064eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:28.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.377+0000 7f083abff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0834099f60 0x7f0834066e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:28.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.377+0000 7f083abff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08340657c0 0x7f0834066430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:28.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.377+0000 7f083abff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0834139d00 con 0x7f0834099f60 2022-06-29T16:06:28.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.377+0000 7f083abff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08340aef20 con 0x7f0820003c80 2022-06-29T16:06:28.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.377+0000 7f083abff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0834067720 con 0x7f08340657c0 2022-06-29T16:06:28.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.377+0000 7f0833fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0834099f60 0x7f0834066e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:28.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.377+0000 7f0833fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0834099f60 0x7f0834066e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52060/0 (socket says 172.21.15.60:52060) 2022-06-29T16:06:28.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.377+0000 7f0833fff700 1 -- 172.21.15.60:0/2373879715 learned_addr learned my addr 172.21.15.60:0/2373879715 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:28.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.377+0000 7f083899b700 1 --2- 172.21.15.60:0/2373879715 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0820003c80 0x7f0834064eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:28.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.377+0000 7f083919c700 1 --2- 172.21.15.60:0/2373879715 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08340657c0 0x7f0834066430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:28.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.377+0000 7f0833fff700 1 -- 172.21.15.60:0/2373879715 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08340657c0 msgr2=0x7f0834066430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:28.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.377+0000 7f0833fff700 1 --2- 172.21.15.60:0/2373879715 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08340657c0 0x7f0834066430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:28.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.377+0000 7f0833fff700 1 -- 172.21.15.60:0/2373879715 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0820003c80 msgr2=0x7f0834064eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:28.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.377+0000 7f0833fff700 1 --2- 172.21.15.60:0/2373879715 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0820003c80 0x7f0834064eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:28.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.377+0000 7f0833fff700 1 -- 172.21.15.60:0/2373879715 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0834067480 con 0x7f0834099f60 2022-06-29T16:06:28.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.377+0000 7f083919c700 1 --2- 172.21.15.60:0/2373879715 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08340657c0 0x7f0834066430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:06:28.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.377+0000 7f083899b700 1 --2- 172.21.15.60:0/2373879715 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0820003c80 0x7f0834064eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:06:28.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.377+0000 7f0833fff700 1 --2- 172.21.15.60:0/2373879715 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0834099f60 0x7f0834066e30 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 rx=0x7f08200054d0 tx=0x7f0828019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:28.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.377+0000 7f0831ffb700 1 -- 172.21.15.60:0/2373879715 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f082800eed0 con 0x7f0834099f60 2022-06-29T16:06:28.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.377+0000 7f0831ffb700 1 -- 172.21.15.60:0/2373879715 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0828000e90 con 0x7f0834099f60 2022-06-29T16:06:28.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.381+0000 7f083abff700 1 -- 172.21.15.60:0/2373879715 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f083409ba40 con 0x7f0834099f60 2022-06-29T16:06:28.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.381+0000 7f0831ffb700 1 -- 172.21.15.60:0/2373879715 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f082801c620 con 0x7f0834099f60 2022-06-29T16:06:28.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.381+0000 7f083abff700 1 -- 172.21.15.60:0/2373879715 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f083413e320 con 0x7f0834099f60 2022-06-29T16:06:28.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.381+0000 7f0831ffb700 1 -- 172.21.15.60:0/2373879715 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f082802d070 con 0x7f0834099f60 2022-06-29T16:06:28.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.381+0000 7f0831ffb700 1 --2- 172.21.15.60:0/2373879715 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0814053a70 0x7f0814055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:28.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.381+0000 7f0831ffb700 1 -- 172.21.15.60:0/2373879715 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f082806dff0 con 0x7f0834099f60 2022-06-29T16:06:28.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.381+0000 7f083899b700 1 --2- 172.21.15.60:0/2373879715 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0814053a70 0x7f0814055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:28.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.381+0000 7f083abff700 1 -- 172.21.15.60:0/2373879715 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08340573d0 con 0x7f0834099f60 2022-06-29T16:06:28.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.381+0000 7f083899b700 1 --2- 172.21.15.60:0/2373879715 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0814053a70 0x7f0814055f30 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f0824005000 tx=0x7f0824009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:28.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.389+0000 7f0831ffb700 1 -- 172.21.15.60:0/2373879715 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0828044e70 con 0x7f0834099f60 2022-06-29T16:06:28.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.517+0000 7f083abff700 1 -- 172.21.15.60:0/2373879715 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0834067990 con 0x7f0834099f60 2022-06-29T16:06:28.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.517+0000 7f0831ffb700 1 -- 172.21.15.60:0/2373879715 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+16064 (secure 0 0 0) 0x7f0828048490 con 0x7f0834099f60 2022-06-29T16:06:28.524 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:28.524 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":109,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:06:28.249456+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:06:26.864385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2022-06-29T16:06:27.251709+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:06:28.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.521+0000 7f083abff700 1 -- 172.21.15.60:0/2373879715 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0814053a70 msgr2=0x7f0814055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:28.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.521+0000 7f083abff700 1 --2- 172.21.15.60:0/2373879715 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0814053a70 0x7f0814055f30 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f0824005000 tx=0x7f0824009040).stop 2022-06-29T16:06:28.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.521+0000 7f083abff700 1 -- 172.21.15.60:0/2373879715 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0834099f60 msgr2=0x7f0834066e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:28.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.521+0000 7f083abff700 1 --2- 172.21.15.60:0/2373879715 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0834099f60 0x7f0834066e30 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 rx=0x7f08200054d0 tx=0x7f0828019000).stop 2022-06-29T16:06:28.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.521+0000 7f083abff700 1 -- 172.21.15.60:0/2373879715 shutdown_connections 2022-06-29T16:06:28.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.521+0000 7f083abff700 1 --2- 172.21.15.60:0/2373879715 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0814053a70 0x7f0814055f30 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:28.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.521+0000 7f083abff700 1 --2- 172.21.15.60:0/2373879715 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08340657c0 0x7f0834066430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:28.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.521+0000 7f083abff700 1 --2- 172.21.15.60:0/2373879715 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0834099f60 0x7f0834066e30 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:28.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.521+0000 7f083abff700 1 --2- 172.21.15.60:0/2373879715 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0820003c80 0x7f0834064eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:28.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.521+0000 7f083abff700 1 -- 172.21.15.60:0/2373879715 >> 172.21.15.60:0/2373879715 conn(0x7f08340643e0 msgr2=0x7f083405c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:28.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.521+0000 7f083abff700 1 -- 172.21.15.60:0/2373879715 shutdown_connections 2022-06-29T16:06:28.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.521+0000 7f083abff700 1 -- 172.21.15.60:0/2373879715 wait complete. 2022-06-29T16:06:28.538 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:06:28.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.637+0000 7fb06f51f700 1 Processor -- start 2022-06-29T16:06:28.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.637+0000 7fb06f51f700 1 -- start start 2022-06-29T16:06:28.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.637+0000 7fb06f51f700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb068062230 0x7fb06805fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:28.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.637+0000 7fb06f51f700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb068146cf0 0x7fb0680aaf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:28.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.637+0000 7fb06f51f700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0680b6e30 con 0x7fb0680b6a50 2022-06-29T16:06:28.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.637+0000 7fb06f51f700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb068098970 con 0x7fb068062230 2022-06-29T16:06:28.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.637+0000 7fb06f51f700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb06805fa40 con 0x7fb068146cf0 2022-06-29T16:06:28.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.637+0000 7fb06caba700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb068062230 0x7fb06805fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:28.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.637+0000 7fb06caba700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb068062230 0x7fb06805fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52068/0 (socket says 172.21.15.60:52068) 2022-06-29T16:06:28.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.637+0000 7fb06caba700 1 -- 172.21.15.60:0/2547238723 learned_addr learned my addr 172.21.15.60:0/2547238723 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:28.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.637+0000 7fb06dabc700 1 --2- 172.21.15.60:0/2547238723 >> v2:172.21.15.92:3300/0 conn(0x7fb068146cf0 0x7fb0680aaf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:28.646 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.637+0000 7fb06caba700 1 -- 172.21.15.60:0/2547238723 >> v2:172.21.15.92:3300/0 conn(0x7fb068146cf0 msgr2=0x7fb0680aaf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:28.646 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.637+0000 7fb06caba700 1 --2- 172.21.15.60:0/2547238723 >> v2:172.21.15.92:3300/0 conn(0x7fb068146cf0 0x7fb0680aaf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:28.646 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.637+0000 7fb06caba700 1 -- 172.21.15.60:0/2547238723 >> v1:172.21.15.60:6789/0 conn(0x7fb0680b6a50 legacy=0x7fb0681465e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:28.646 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.637+0000 7fb06caba700 1 -- 172.21.15.60:0/2547238723 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb06805f0b0 con 0x7fb068062230 2022-06-29T16:06:28.646 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.637+0000 7fb06caba700 1 --2- 172.21.15.60:0/2547238723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb068062230 0x7fb06805fee0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 rx=0x7fb064002f70 tx=0x7fb064006eb0).ready entity=mon.0 client_cookie=2e8d10c1245a4dd6 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:28.647 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.637+0000 7fb05bfff700 1 -- 172.21.15.60:0/2547238723 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb06400d610 con 0x7fb068062230 2022-06-29T16:06:28.647 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.637+0000 7fb05bfff700 1 -- 172.21.15.60:0/2547238723 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb064000e90 con 0x7fb068062230 2022-06-29T16:06:28.647 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.637+0000 7fb05bfff700 1 -- 172.21.15.60:0/2547238723 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb06400d9a0 con 0x7fb068062230 2022-06-29T16:06:28.647 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.637+0000 7fb06f51f700 1 -- 172.21.15.60:0/2547238723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb068062230 msgr2=0x7fb06805fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:28.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.637+0000 7fb06f51f700 1 --2- 172.21.15.60:0/2547238723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb068062230 0x7fb06805fee0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 rx=0x7fb064002f70 tx=0x7fb064006eb0).stop 2022-06-29T16:06:28.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.637+0000 7fb06f51f700 1 -- 172.21.15.60:0/2547238723 shutdown_connections 2022-06-29T16:06:28.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.637+0000 7fb06f51f700 1 --2- 172.21.15.60:0/2547238723 >> v2:172.21.15.92:3300/0 conn(0x7fb068146cf0 0x7fb0680aaf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:28.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.637+0000 7fb06f51f700 1 --2- 172.21.15.60:0/2547238723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb068062230 0x7fb06805fee0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:28.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.637+0000 7fb06f51f700 1 -- 172.21.15.60:0/2547238723 >> 172.21.15.60:0/2547238723 conn(0x7fb0680643e0 msgr2=0x7fb0680b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:28.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.637+0000 7fb06f51f700 1 -- 172.21.15.60:0/2547238723 shutdown_connections 2022-06-29T16:06:28.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.637+0000 7fb06f51f700 1 -- 172.21.15.60:0/2547238723 wait complete. 2022-06-29T16:06:28.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.637+0000 7fb06f51f700 1 Processor -- start 2022-06-29T16:06:28.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.641+0000 7fb06f51f700 1 -- start start 2022-06-29T16:06:28.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.641+0000 7fb06f51f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb068062230 0x7fb0680af8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:28.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.641+0000 7fb06f51f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0680b6a50 0x7fb0680afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:28.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.641+0000 7fb06f51f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb068146cf0 0x7fb068099ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:28.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.641+0000 7fb06f51f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb068066800 con 0x7fb068062230 2022-06-29T16:06:28.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.641+0000 7fb06f51f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0680b02a0 con 0x7fb068146cf0 2022-06-29T16:06:28.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.641+0000 7fb06f51f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0680ae4a0 con 0x7fb0680b6a50 2022-06-29T16:06:28.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.641+0000 7fb06d2bb700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb068062230 0x7fb0680af8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:28.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.641+0000 7fb06d2bb700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb068062230 0x7fb0680af8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52074/0 (socket says 172.21.15.60:52074) 2022-06-29T16:06:28.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.641+0000 7fb06d2bb700 1 -- 172.21.15.60:0/3101023890 learned_addr learned my addr 172.21.15.60:0/3101023890 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:28.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.641+0000 7fb06caba700 1 --2- 172.21.15.60:0/3101023890 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0680b6a50 0x7fb0680afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:28.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.641+0000 7fb06dabc700 1 --2- 172.21.15.60:0/3101023890 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb068146cf0 0x7fb068099ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:28.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.641+0000 7fb06d2bb700 1 -- 172.21.15.60:0/3101023890 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0680b6a50 msgr2=0x7fb0680afdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:28.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.641+0000 7fb06d2bb700 1 --2- 172.21.15.60:0/3101023890 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0680b6a50 0x7fb0680afdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:28.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.641+0000 7fb06d2bb700 1 -- 172.21.15.60:0/3101023890 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb068146cf0 msgr2=0x7fb068099ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:28.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.641+0000 7fb06d2bb700 1 --2- 172.21.15.60:0/3101023890 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb068146cf0 0x7fb068099ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:28.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.641+0000 7fb06d2bb700 1 -- 172.21.15.60:0/3101023890 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb06400b1e0 con 0x7fb068062230 2022-06-29T16:06:28.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.641+0000 7fb06d2bb700 1 --2- 172.21.15.60:0/3101023890 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb068062230 0x7fb0680af8c0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 rx=0x7fb0680b3d60 tx=0x7fb05c007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:28.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.641+0000 7fb05a7fc700 1 -- 172.21.15.60:0/3101023890 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb05c012400 con 0x7fb068062230 2022-06-29T16:06:28.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.641+0000 7fb05a7fc700 1 -- 172.21.15.60:0/3101023890 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb05c012560 con 0x7fb068062230 2022-06-29T16:06:28.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.641+0000 7fb06f51f700 1 -- 172.21.15.60:0/3101023890 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0680ae200 con 0x7fb068062230 2022-06-29T16:06:28.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.641+0000 7fb05a7fc700 1 -- 172.21.15.60:0/3101023890 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb05c0128c0 con 0x7fb068062230 2022-06-29T16:06:28.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.641+0000 7fb06f51f700 1 -- 172.21.15.60:0/3101023890 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb068098fd0 con 0x7fb068062230 2022-06-29T16:06:28.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.645+0000 7fb06f51f700 1 -- 172.21.15.60:0/3101023890 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0680573d0 con 0x7fb068062230 2022-06-29T16:06:28.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.645+0000 7fb05a7fc700 1 -- 172.21.15.60:0/3101023890 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb05c019030 con 0x7fb068062230 2022-06-29T16:06:28.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.645+0000 7fb05a7fc700 1 --2- 172.21.15.60:0/3101023890 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb0500539e0 0x7fb050055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:28.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.645+0000 7fb05a7fc700 1 -- 172.21.15.60:0/3101023890 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb05c06b310 con 0x7fb068062230 2022-06-29T16:06:28.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.645+0000 7fb06caba700 1 --2- 172.21.15.60:0/3101023890 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb0500539e0 0x7fb050055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:28.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.645+0000 7fb06caba700 1 --2- 172.21.15.60:0/3101023890 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb0500539e0 0x7fb050055ea0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0x7fb06401c000 tx=0x7fb064015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:28.659 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.649+0000 7fb05a7fc700 1 -- 172.21.15.60:0/3101023890 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb05c042120 con 0x7fb068062230 2022-06-29T16:06:28.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.773+0000 7fb06f51f700 1 -- 172.21.15.60:0/3101023890 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb06809ac90 con 0x7fb068062230 2022-06-29T16:06:28.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.773+0000 7fb05a7fc700 1 -- 172.21.15.60:0/3101023890 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+16064 (secure 0 0 0) 0x7fb05c045740 con 0x7fb068062230 2022-06-29T16:06:28.780 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:28.780 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":109,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:06:28.249456+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:06:26.864385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2022-06-29T16:06:27.251709+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:06:28.783 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.777+0000 7fb06f51f700 1 -- 172.21.15.60:0/3101023890 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb0500539e0 msgr2=0x7fb050055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:28.783 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.777+0000 7fb06f51f700 1 --2- 172.21.15.60:0/3101023890 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb0500539e0 0x7fb050055ea0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0x7fb06401c000 tx=0x7fb064015000).stop 2022-06-29T16:06:28.784 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.777+0000 7fb06f51f700 1 -- 172.21.15.60:0/3101023890 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb068062230 msgr2=0x7fb0680af8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:28.784 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.777+0000 7fb06f51f700 1 --2- 172.21.15.60:0/3101023890 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb068062230 0x7fb0680af8c0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 rx=0x7fb0680b3d60 tx=0x7fb05c007c50).stop 2022-06-29T16:06:28.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.777+0000 7fb06f51f700 1 -- 172.21.15.60:0/3101023890 shutdown_connections 2022-06-29T16:06:28.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.777+0000 7fb06f51f700 1 --2- 172.21.15.60:0/3101023890 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb0500539e0 0x7fb050055ea0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:28.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.777+0000 7fb06f51f700 1 --2- 172.21.15.60:0/3101023890 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb068146cf0 0x7fb068099ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:28.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.777+0000 7fb06f51f700 1 --2- 172.21.15.60:0/3101023890 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb0680b6a50 0x7fb0680afdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:28.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.777+0000 7fb06f51f700 1 --2- 172.21.15.60:0/3101023890 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb068062230 0x7fb0680af8c0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:28.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.777+0000 7fb06f51f700 1 -- 172.21.15.60:0/3101023890 >> 172.21.15.60:0/3101023890 conn(0x7fb0680643e0 msgr2=0x7fb068142140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:28.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.777+0000 7fb06f51f700 1 -- 172.21.15.60:0/3101023890 shutdown_connections 2022-06-29T16:06:28.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.777+0000 7fb06f51f700 1 -- 172.21.15.60:0/3101023890 wait complete. 2022-06-29T16:06:28.794 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-06-29T16:06:28.903 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.897+0000 7fb2f4fcd700 1 Processor -- start 2022-06-29T16:06:28.904 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.897+0000 7fb2f4fcd700 1 -- start start 2022-06-29T16:06:28.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.901+0000 7fb2f4fcd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb2f00b0890 0x7fb2f00b4540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:28.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.901+0000 7fb2f4fcd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb2f00b3b40 0x7fb2f00b4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:28.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.901+0000 7fb2f4fcd700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2f00b9fd0 con 0x7fb2f00b9bf0 2022-06-29T16:06:28.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.901+0000 7fb2f4fcd700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2f00b1a50 con 0x7fb2f00b3b40 2022-06-29T16:06:28.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.901+0000 7fb2f4fcd700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f00b1cf0 con 0x7fb2f00b0890 2022-06-29T16:06:28.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.901+0000 7fb2effff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb2f00b3b40 0x7fb2f00b4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:28.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.901+0000 7fb2ef7fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb2f00b9bf0 0x7fb2f00b2190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60168/0 (socket says 172.21.15.60:60168) 2022-06-29T16:06:28.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.901+0000 7fb2ef7fe700 1 -- 172.21.15.60:0/1114032835 learned_addr learned my addr 172.21.15.60:0/1114032835 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:28.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.901+0000 7fb2eeffd700 1 --2- 172.21.15.60:0/1114032835 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb2f00b0890 0x7fb2f00b4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:28.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.901+0000 7fb2effff700 1 -- 172.21.15.60:0/1114032835 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb2f00b0890 msgr2=0x7fb2f00b4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:28.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.901+0000 7fb2effff700 1 --2- 172.21.15.60:0/1114032835 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb2f00b0890 0x7fb2f00b4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:28.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.901+0000 7fb2effff700 1 -- 172.21.15.60:0/1114032835 >> v1:172.21.15.60:6789/0 conn(0x7fb2f00b9bf0 legacy=0x7fb2f00b2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:28.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.901+0000 7fb2effff700 1 -- 172.21.15.60:0/1114032835 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f00b3470 con 0x7fb2f00b3b40 2022-06-29T16:06:28.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.901+0000 7fb2effff700 1 --2- 172.21.15.60:0/1114032835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2f00b3b40 0x7fb2f00b4a30 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 rx=0x7fb2d8002f70 tx=0x7fb2d8006df0).ready entity=mon.0 client_cookie=3357804a9df1ecff server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:28.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.901+0000 7fb2ee7fc700 1 -- 172.21.15.60:0/1114032835 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2d800d570 con 0x7fb2f00b3b40 2022-06-29T16:06:28.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.901+0000 7fb2ee7fc700 1 -- 172.21.15.60:0/1114032835 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d8000e90 con 0x7fb2f00b3b40 2022-06-29T16:06:28.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.901+0000 7fb2ee7fc700 1 -- 172.21.15.60:0/1114032835 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2d800d8c0 con 0x7fb2f00b3b40 2022-06-29T16:06:28.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.901+0000 7fb2f4fcd700 1 -- 172.21.15.60:0/1114032835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2f00b3b40 msgr2=0x7fb2f00b4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:28.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.901+0000 7fb2f4fcd700 1 --2- 172.21.15.60:0/1114032835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2f00b3b40 0x7fb2f00b4a30 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 rx=0x7fb2d8002f70 tx=0x7fb2d8006df0).stop 2022-06-29T16:06:28.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.901+0000 7fb2f4fcd700 1 -- 172.21.15.60:0/1114032835 shutdown_connections 2022-06-29T16:06:28.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.901+0000 7fb2f4fcd700 1 --2- 172.21.15.60:0/1114032835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2f00b3b40 0x7fb2f00b4a30 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:28.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.901+0000 7fb2f4fcd700 1 --2- 172.21.15.60:0/1114032835 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb2f00b0890 0x7fb2f00b4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:28.915 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.901+0000 7fb2f4fcd700 1 -- 172.21.15.60:0/1114032835 >> 172.21.15.60:0/1114032835 conn(0x7fb2f0062b60 msgr2=0x7fb2f00b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:28.915 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.901+0000 7fb2f4fcd700 1 -- 172.21.15.60:0/1114032835 shutdown_connections 2022-06-29T16:06:28.915 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.901+0000 7fb2f4fcd700 1 -- 172.21.15.60:0/1114032835 wait complete. 2022-06-29T16:06:28.915 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.901+0000 7fb2f4fcd700 1 Processor -- start 2022-06-29T16:06:28.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.905+0000 7fb2f4fcd700 1 -- start start 2022-06-29T16:06:28.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.905+0000 7fb2f4fcd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb2f00b0890 0x7fb2f0146150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:28.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.905+0000 7fb2f4fcd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2f00b3b40 0x7fb2f0149830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:28.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.905+0000 7fb2f4fcd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb2f00b9bf0 0x7fb2f013d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:28.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.905+0000 7fb2f4fcd700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f013fb30 con 0x7fb2f00b3b40 2022-06-29T16:06:28.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.905+0000 7fb2f4fcd700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f0152a90 con 0x7fb2f00b9bf0 2022-06-29T16:06:28.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.905+0000 7fb2f4fcd700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f01527f0 con 0x7fb2f00b0890 2022-06-29T16:06:28.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.905+0000 7fb2eeffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2f00b3b40 0x7fb2f0149830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:28.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.905+0000 7fb2eeffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2f00b3b40 0x7fb2f0149830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52090/0 (socket says 172.21.15.60:52090) 2022-06-29T16:06:28.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.905+0000 7fb2eeffd700 1 -- 172.21.15.60:0/962061250 learned_addr learned my addr 172.21.15.60:0/962061250 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:28.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.905+0000 7fb2ef7fe700 1 --2- 172.21.15.60:0/962061250 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb2f00b0890 0x7fb2f0146150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:28.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.905+0000 7fb2effff700 1 --2- 172.21.15.60:0/962061250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb2f00b9bf0 0x7fb2f013d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:28.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.905+0000 7fb2eeffd700 1 -- 172.21.15.60:0/962061250 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb2f00b0890 msgr2=0x7fb2f0146150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:28.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.905+0000 7fb2eeffd700 1 --2- 172.21.15.60:0/962061250 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb2f00b0890 0x7fb2f0146150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:28.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.905+0000 7fb2eeffd700 1 -- 172.21.15.60:0/962061250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb2f00b9bf0 msgr2=0x7fb2f013d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:28.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.905+0000 7fb2eeffd700 1 --2- 172.21.15.60:0/962061250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb2f00b9bf0 0x7fb2f013d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:28.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.905+0000 7fb2eeffd700 1 -- 172.21.15.60:0/962061250 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d800b180 con 0x7fb2f00b3b40 2022-06-29T16:06:28.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.905+0000 7fb2eeffd700 1 --2- 172.21.15.60:0/962061250 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2f00b3b40 0x7fb2f0149830 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 rx=0x7fb2e000c000 tx=0x7fb2e00091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:28.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.905+0000 7fb2ecff9700 1 -- 172.21.15.60:0/962061250 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2e000c850 con 0x7fb2f00b3b40 2022-06-29T16:06:28.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.905+0000 7fb2ecff9700 1 -- 172.21.15.60:0/962061250 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e000c9b0 con 0x7fb2f00b3b40 2022-06-29T16:06:28.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.905+0000 7fb2ecff9700 1 -- 172.21.15.60:0/962061250 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2e000cd10 con 0x7fb2f00b3b40 2022-06-29T16:06:28.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.905+0000 7fb2f4fcd700 1 -- 172.21.15.60:0/962061250 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2f0152500 con 0x7fb2f00b3b40 2022-06-29T16:06:28.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.905+0000 7fb2f4fcd700 1 -- 172.21.15.60:0/962061250 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2f01479f0 con 0x7fb2f00b3b40 2022-06-29T16:06:28.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.909+0000 7fb2ecff9700 1 -- 172.21.15.60:0/962061250 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb2e0022070 con 0x7fb2f00b3b40 2022-06-29T16:06:28.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.909+0000 7fb2f4fcd700 1 -- 172.21.15.60:0/962061250 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2f0141220 con 0x7fb2f00b3b40 2022-06-29T16:06:28.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.909+0000 7fb2ecff9700 1 --2- 172.21.15.60:0/962061250 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb2d4053b00 0x7fb2d4055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:28.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.909+0000 7fb2ecff9700 1 -- 172.21.15.60:0/962061250 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb2e006c7c0 con 0x7fb2f00b3b40 2022-06-29T16:06:28.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.909+0000 7fb2ef7fe700 1 --2- 172.21.15.60:0/962061250 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb2d4053b00 0x7fb2d4055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:28.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.909+0000 7fb2ef7fe700 1 --2- 172.21.15.60:0/962061250 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb2d4053b00 0x7fb2d4055fc0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7fb2e800b000 tx=0x7fb2e8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:28.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:28.913+0000 7fb2ecff9700 1 -- 172.21.15.60:0/962061250 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb2f0141220 con 0x7fb2f00b3b40 2022-06-29T16:06:29.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.077+0000 7fb2f4fcd700 1 -- 172.21.15.60:0/962061250 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fb2f0141220 con 0x7fb2f00b3b40 2022-06-29T16:06:29.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.257+0000 7fb2ecff9700 1 -- 172.21.15.60:0/962061250 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v51) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb2f0141220 con 0x7fb2f00b3b40 2022-06-29T16:06:29.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.261+0000 7fb2f4fcd700 1 -- 172.21.15.60:0/962061250 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb2d4053b00 msgr2=0x7fb2d4055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:29.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.261+0000 7fb2f4fcd700 1 --2- 172.21.15.60:0/962061250 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb2d4053b00 0x7fb2d4055fc0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7fb2e800b000 tx=0x7fb2e8009040).stop 2022-06-29T16:06:29.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.261+0000 7fb2f4fcd700 1 -- 172.21.15.60:0/962061250 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2f00b3b40 msgr2=0x7fb2f0149830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:29.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.261+0000 7fb2f4fcd700 1 --2- 172.21.15.60:0/962061250 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2f00b3b40 0x7fb2f0149830 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 rx=0x7fb2e000c000 tx=0x7fb2e00091d0).stop 2022-06-29T16:06:29.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.261+0000 7fb2f4fcd700 1 -- 172.21.15.60:0/962061250 shutdown_connections 2022-06-29T16:06:29.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.261+0000 7fb2f4fcd700 1 --2- 172.21.15.60:0/962061250 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb2d4053b00 0x7fb2d4055fc0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:29.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.261+0000 7fb2f4fcd700 1 --2- 172.21.15.60:0/962061250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb2f00b9bf0 0x7fb2f013d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:29.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.261+0000 7fb2f4fcd700 1 --2- 172.21.15.60:0/962061250 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb2f00b3b40 0x7fb2f0149830 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:29.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.261+0000 7fb2f4fcd700 1 --2- 172.21.15.60:0/962061250 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb2f00b0890 0x7fb2f0146150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:29.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.261+0000 7fb2f4fcd700 1 -- 172.21.15.60:0/962061250 >> 172.21.15.60:0/962061250 conn(0x7fb2f0062b60 msgr2=0x7fb2f00b5bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:29.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.261+0000 7fb2f4fcd700 1 -- 172.21.15.60:0/962061250 shutdown_connections 2022-06-29T16:06:29.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.261+0000 7fb2f4fcd700 1 -- 172.21.15.60:0/962061250 wait complete. 2022-06-29T16:06:29.280 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:06:29.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.381+0000 7effcaf8d700 1 Processor -- start 2022-06-29T16:06:29.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.381+0000 7effcaf8d700 1 -- start start 2022-06-29T16:06:29.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.385+0000 7effcaf8d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7effc40b0e30 0x7effc40b59f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:29.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.385+0000 7effcaf8d700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7effc4146650 0x7effc4146ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:29.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.385+0000 7effcaf8d700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc40b1340 con 0x7effc40b5610 2022-06-29T16:06:29.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.385+0000 7effcaf8d700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effc4065350 con 0x7effc4146650 2022-06-29T16:06:29.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.385+0000 7effcaf8d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc413cf50 con 0x7effc40b0e30 2022-06-29T16:06:29.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.385+0000 7effc3fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7effc40b0e30 0x7effc40b59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:29.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.385+0000 7effc952a700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7effc4146650 0x7effc4146ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:29.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.385+0000 7effc3fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7effc40b0e30 0x7effc40b59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:36958/0 (socket says 172.21.15.60:36958) 2022-06-29T16:06:29.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.385+0000 7effc3fff700 1 -- 172.21.15.60:0/1380163557 learned_addr learned my addr 172.21.15.60:0/1380163557 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:29.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.385+0000 7effc3fff700 1 -- 172.21.15.60:0/1380163557 >> v2:172.21.15.92:3300/0 conn(0x7effc4146650 msgr2=0x7effc4146ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:29.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.385+0000 7effc3fff700 1 --2- 172.21.15.60:0/1380163557 >> v2:172.21.15.92:3300/0 conn(0x7effc4146650 0x7effc4146ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:29.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.385+0000 7effc3fff700 1 -- 172.21.15.60:0/1380163557 >> v1:172.21.15.92:6789/0 conn(0x7effc40b5610 legacy=0x7effc4142a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:29.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.385+0000 7effc3fff700 1 -- 172.21.15.60:0/1380163557 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc41461b0 con 0x7effc40b0e30 2022-06-29T16:06:29.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.385+0000 7effc3fff700 1 --2- 172.21.15.60:0/1380163557 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7effc40b0e30 0x7effc40b59f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7effb4002f70 tx=0x7effb4006eb0).ready entity=mon.2 client_cookie=858fa9539f2f012e server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:29.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.385+0000 7effc37fe700 1 -- 172.21.15.60:0/1380163557 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7effb400d570 con 0x7effc40b0e30 2022-06-29T16:06:29.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.385+0000 7effc37fe700 1 -- 172.21.15.60:0/1380163557 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb4000e90 con 0x7effc40b0e30 2022-06-29T16:06:29.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.385+0000 7effc37fe700 1 -- 172.21.15.60:0/1380163557 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7effb400d900 con 0x7effc40b0e30 2022-06-29T16:06:29.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.385+0000 7effcaf8d700 1 -- 172.21.15.60:0/1380163557 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7effc40b0e30 msgr2=0x7effc40b59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:29.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.385+0000 7effcaf8d700 1 --2- 172.21.15.60:0/1380163557 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7effc40b0e30 0x7effc40b59f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7effb4002f70 tx=0x7effb4006eb0).stop 2022-06-29T16:06:29.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.385+0000 7effcaf8d700 1 -- 172.21.15.60:0/1380163557 shutdown_connections 2022-06-29T16:06:29.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.385+0000 7effcaf8d700 1 --2- 172.21.15.60:0/1380163557 >> v2:172.21.15.92:3300/0 conn(0x7effc4146650 0x7effc4146ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:29.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.385+0000 7effcaf8d700 1 --2- 172.21.15.60:0/1380163557 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7effc40b0e30 0x7effc40b59f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:29.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.385+0000 7effcaf8d700 1 -- 172.21.15.60:0/1380163557 >> 172.21.15.60:0/1380163557 conn(0x7effc40632c0 msgr2=0x7effc40b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:29.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.385+0000 7effcaf8d700 1 -- 172.21.15.60:0/1380163557 shutdown_connections 2022-06-29T16:06:29.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.385+0000 7effcaf8d700 1 -- 172.21.15.60:0/1380163557 wait complete. 2022-06-29T16:06:29.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.389+0000 7effcaf8d700 1 Processor -- start 2022-06-29T16:06:29.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.389+0000 7effcaf8d700 1 -- start start 2022-06-29T16:06:29.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.389+0000 7effcaf8d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7effc40b0e30 0x7effc4151bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:29.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.389+0000 7effcaf8d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effc40b5610 0x7effc4063ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:29.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.389+0000 7effcaf8d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7effc4146650 0x7effc4155900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:29.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.389+0000 7effcaf8d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc4152e70 con 0x7effc40b5610 2022-06-29T16:06:29.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.389+0000 7effcaf8d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc413f5d0 con 0x7effc40b0e30 2022-06-29T16:06:29.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.389+0000 7effcaf8d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc41412e0 con 0x7effc4146650 2022-06-29T16:06:29.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.389+0000 7effc3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effc40b5610 0x7effc4063ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:29.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.389+0000 7effc3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effc40b5610 0x7effc4063ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52104/0 (socket says 172.21.15.60:52104) 2022-06-29T16:06:29.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.389+0000 7effc3fff700 1 -- 172.21.15.60:0/839239832 learned_addr learned my addr 172.21.15.60:0/839239832 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:29.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.389+0000 7effc952a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7effc4146650 0x7effc4155900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:29.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.389+0000 7effc8d29700 1 --2- 172.21.15.60:0/839239832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7effc40b0e30 0x7effc4151bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:29.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.389+0000 7effc3fff700 1 -- 172.21.15.60:0/839239832 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7effc4146650 msgr2=0x7effc4155900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:29.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.389+0000 7effc3fff700 1 --2- 172.21.15.60:0/839239832 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7effc4146650 0x7effc4155900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:29.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.389+0000 7effc3fff700 1 -- 172.21.15.60:0/839239832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7effc40b0e30 msgr2=0x7effc4151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:29.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.389+0000 7effc3fff700 1 --2- 172.21.15.60:0/839239832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7effc40b0e30 0x7effc4151bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:29.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.389+0000 7effc3fff700 1 -- 172.21.15.60:0/839239832 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb400b1e0 con 0x7effc40b5610 2022-06-29T16:06:29.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.389+0000 7effc3fff700 1 --2- 172.21.15.60:0/839239832 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effc40b5610 0x7effc4063ba0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 rx=0x7effb4002f70 tx=0x7effb4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:29.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.389+0000 7effc1ffb700 1 -- 172.21.15.60:0/839239832 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7effb4011760 con 0x7effc40b5610 2022-06-29T16:06:29.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.389+0000 7effc1ffb700 1 -- 172.21.15.60:0/839239832 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb40118c0 con 0x7effc40b5610 2022-06-29T16:06:29.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.389+0000 7effcaf8d700 1 -- 172.21.15.60:0/839239832 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc4155460 con 0x7effc40b5610 2022-06-29T16:06:29.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.389+0000 7effc1ffb700 1 -- 172.21.15.60:0/839239832 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7effb4011c20 con 0x7effc40b5610 2022-06-29T16:06:29.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.389+0000 7effcaf8d700 1 -- 172.21.15.60:0/839239832 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effc4064af0 con 0x7effc40b5610 2022-06-29T16:06:29.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.393+0000 7effcaf8d700 1 -- 172.21.15.60:0/839239832 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effc41555f0 con 0x7effc40b5610 2022-06-29T16:06:29.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.393+0000 7effc1ffb700 1 -- 172.21.15.60:0/839239832 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7effb4025030 con 0x7effc40b5610 2022-06-29T16:06:29.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.393+0000 7effc1ffb700 1 --2- 172.21.15.60:0/839239832 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7effa8053b00 0x7effa8055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:29.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.393+0000 7effc1ffb700 1 -- 172.21.15.60:0/839239832 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7effb406cf80 con 0x7effc40b5610 2022-06-29T16:06:29.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.393+0000 7effc8d29700 1 --2- 172.21.15.60:0/839239832 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7effa8053b00 0x7effa8055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:29.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.393+0000 7effc8d29700 1 --2- 172.21.15.60:0/839239832 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7effa8053b00 0x7effa8055fc0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0x7effac00d000 tx=0x7effac007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:29.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.401+0000 7effc1ffb700 1 -- 172.21.15.60:0/839239832 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7effb4043ea0 con 0x7effc40b5610 2022-06-29T16:06:29.485 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:06:29.488 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:06:29.490 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:06:29.493 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:06:29.495 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:06:29.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.565+0000 7effcaf8d700 1 -- 172.21.15.60:0/839239832 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7effc40573d0 con 0x7effc40b5610 2022-06-29T16:06:29.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.565+0000 7effc1ffb700 1 -- 172.21.15.60:0/839239832 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+2142 (secure 0 0 0) 0x7effb4023070 con 0x7effc40b5610 2022-06-29T16:06:29.570 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:29.570 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":51,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7648,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/3387042465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3387042465},{"type":"v1","addr":"172.21.15.92:6835","nonce":3387042465}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:06:28.249737+0000","modified":"2022-06-29T16:06:29.252773+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7668},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7668":{"gid":7668,"name":"a","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/2749128920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2749128920},{"type":"v1","addr":"172.21.15.60:6833","nonce":2749128920}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2022-06-29T16:06:29.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.569+0000 7effcaf8d700 1 -- 172.21.15.60:0/839239832 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7effa8053b00 msgr2=0x7effa8055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:29.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.569+0000 7effcaf8d700 1 --2- 172.21.15.60:0/839239832 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7effa8053b00 0x7effa8055fc0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0x7effac00d000 tx=0x7effac007b90).stop 2022-06-29T16:06:29.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.569+0000 7effcaf8d700 1 -- 172.21.15.60:0/839239832 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effc40b5610 msgr2=0x7effc4063ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:29.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.569+0000 7effcaf8d700 1 --2- 172.21.15.60:0/839239832 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effc40b5610 0x7effc4063ba0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 rx=0x7effb4002f70 tx=0x7effb4015000).stop 2022-06-29T16:06:29.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.569+0000 7effcaf8d700 1 -- 172.21.15.60:0/839239832 shutdown_connections 2022-06-29T16:06:29.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.569+0000 7effcaf8d700 1 --2- 172.21.15.60:0/839239832 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7effa8053b00 0x7effa8055fc0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:29.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.569+0000 7effcaf8d700 1 --2- 172.21.15.60:0/839239832 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7effc4146650 0x7effc4155900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:29.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.569+0000 7effcaf8d700 1 --2- 172.21.15.60:0/839239832 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effc40b5610 0x7effc4063ba0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:29.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.569+0000 7effcaf8d700 1 --2- 172.21.15.60:0/839239832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7effc40b0e30 0x7effc4151bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:29.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.569+0000 7effcaf8d700 1 -- 172.21.15.60:0/839239832 >> 172.21.15.60:0/839239832 conn(0x7effc40632c0 msgr2=0x7effc4147170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:29.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.569+0000 7effcaf8d700 1 -- 172.21.15.60:0/839239832 shutdown_connections 2022-06-29T16:06:29.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.569+0000 7effcaf8d700 1 -- 172.21.15.60:0/839239832 wait complete. 2022-06-29T16:06:29.577 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 51 2022-06-29T16:06:29.589 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:06:29.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.689+0000 7ff56d95f700 1 Processor -- start 2022-06-29T16:06:29.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.689+0000 7ff56d95f700 1 -- start start 2022-06-29T16:06:29.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.689+0000 7ff56d95f700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff5680b1930 0x7ff5680ab2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:29.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.689+0000 7ff56d95f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff56805c1f0 0x7ff568146240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:29.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.689+0000 7ff56d95f700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff56805bdc0 con 0x7ff5680b5db0 2022-06-29T16:06:29.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.689+0000 7ff56d95f700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5680b6190 con 0x7ff5680b1930 2022-06-29T16:06:29.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.689+0000 7ff56d95f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5680aff10 con 0x7ff56805c1f0 2022-06-29T16:06:29.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.689+0000 7ff566ffd700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff5680b5db0 0x7ff5680aabe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60196/0 (socket says 172.21.15.60:60196) 2022-06-29T16:06:29.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.689+0000 7ff566ffd700 1 -- 172.21.15.60:0/755045469 learned_addr learned my addr 172.21.15.60:0/755045469 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:29.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.689+0000 7ff5667fc700 1 --2- 172.21.15.60:0/755045469 >> v2:172.21.15.60:3300/0 conn(0x7ff5680b1930 0x7ff5680ab2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:29.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.689+0000 7ff5677fe700 1 --2- 172.21.15.60:0/755045469 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff56805c1f0 0x7ff568146240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:29.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.689+0000 7ff565ffb700 1 -- 172.21.15.60:0/755045469 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1293876784 0 0) 0x7ff56805bdc0 con 0x7ff5680b5db0 2022-06-29T16:06:29.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.689+0000 7ff565ffb700 1 -- 172.21.15.60:0/755045469 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff548003260 con 0x7ff5680b5db0 2022-06-29T16:06:29.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.689+0000 7ff5667fc700 1 -- 172.21.15.60:0/755045469 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff56805c1f0 msgr2=0x7ff568146240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:29.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.689+0000 7ff5667fc700 1 --2- 172.21.15.60:0/755045469 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff56805c1f0 0x7ff568146240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:29.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.689+0000 7ff5667fc700 1 -- 172.21.15.60:0/755045469 >> v1:172.21.15.60:6789/0 conn(0x7ff5680b5db0 legacy=0x7ff5680aabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:29.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.689+0000 7ff5667fc700 1 -- 172.21.15.60:0/755045469 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5680b01b0 con 0x7ff5680b1930 2022-06-29T16:06:29.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.689+0000 7ff5667fc700 1 --2- 172.21.15.60:0/755045469 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5680b1930 0x7ff5680ab2f0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 rx=0x7ff558002f70 tx=0x7ff558006df0).ready entity=mon.0 client_cookie=9e12ff916a494c3 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:29.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.689+0000 7ff565ffb700 1 -- 172.21.15.60:0/755045469 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff55800d4a0 con 0x7ff5680b1930 2022-06-29T16:06:29.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.689+0000 7ff565ffb700 1 -- 172.21.15.60:0/755045469 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff558000e90 con 0x7ff5680b1930 2022-06-29T16:06:29.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.689+0000 7ff565ffb700 1 -- 172.21.15.60:0/755045469 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff55800d7f0 con 0x7ff5680b1930 2022-06-29T16:06:29.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.693+0000 7ff56d95f700 1 -- 172.21.15.60:0/755045469 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5680b1930 msgr2=0x7ff5680ab2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:29.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.693+0000 7ff56d95f700 1 --2- 172.21.15.60:0/755045469 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5680b1930 0x7ff5680ab2f0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 rx=0x7ff558002f70 tx=0x7ff558006df0).stop 2022-06-29T16:06:29.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.693+0000 7ff56d95f700 1 -- 172.21.15.60:0/755045469 shutdown_connections 2022-06-29T16:06:29.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.693+0000 7ff56d95f700 1 --2- 172.21.15.60:0/755045469 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff56805c1f0 0x7ff568146240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:29.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.693+0000 7ff56d95f700 1 --2- 172.21.15.60:0/755045469 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5680b1930 0x7ff5680ab2f0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:29.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.693+0000 7ff56d95f700 1 -- 172.21.15.60:0/755045469 >> 172.21.15.60:0/755045469 conn(0x7ff5680636f0 msgr2=0x7ff5680b5360 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:29.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.693+0000 7ff56d95f700 1 -- 172.21.15.60:0/755045469 shutdown_connections 2022-06-29T16:06:29.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.693+0000 7ff56d95f700 1 -- 172.21.15.60:0/755045469 wait complete. 2022-06-29T16:06:29.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.693+0000 7ff56d95f700 1 Processor -- start 2022-06-29T16:06:29.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.693+0000 7ff56d95f700 1 -- start start 2022-06-29T16:06:29.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.693+0000 7ff56d95f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff56805c1f0 0x7ff568064520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:29.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.693+0000 7ff56d95f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5680b1930 0x7ff568156250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:29.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.693+0000 7ff56d95f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5680b5db0 0x7ff568156740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:29.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.693+0000 7ff56d95f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5680671c0 con 0x7ff5680b1930 2022-06-29T16:06:29.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.693+0000 7ff56d95f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff568065e90 con 0x7ff5680b5db0 2022-06-29T16:06:29.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.693+0000 7ff56d95f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff568066130 con 0x7ff56805c1f0 2022-06-29T16:06:29.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.693+0000 7ff5667fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5680b1930 0x7ff568156250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:29.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.697+0000 7ff5667fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5680b1930 0x7ff568156250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52118/0 (socket says 172.21.15.60:52118) 2022-06-29T16:06:29.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.697+0000 7ff5667fc700 1 -- 172.21.15.60:0/782616773 learned_addr learned my addr 172.21.15.60:0/782616773 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:29.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.697+0000 7ff566ffd700 1 --2- 172.21.15.60:0/782616773 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff56805c1f0 0x7ff568064520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:29.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.697+0000 7ff5677fe700 1 --2- 172.21.15.60:0/782616773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5680b5db0 0x7ff568156740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:29.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.697+0000 7ff5667fc700 1 -- 172.21.15.60:0/782616773 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff56805c1f0 msgr2=0x7ff568064520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:29.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.697+0000 7ff5667fc700 1 --2- 172.21.15.60:0/782616773 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff56805c1f0 0x7ff568064520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:29.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.697+0000 7ff5667fc700 1 -- 172.21.15.60:0/782616773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5680b5db0 msgr2=0x7ff568156740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:29.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.697+0000 7ff5667fc700 1 --2- 172.21.15.60:0/782616773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5680b5db0 0x7ff568156740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:29.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.697+0000 7ff5667fc700 1 -- 172.21.15.60:0/782616773 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff55800b180 con 0x7ff5680b1930 2022-06-29T16:06:29.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.697+0000 7ff5667fc700 1 --2- 172.21.15.60:0/782616773 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5680b1930 0x7ff568156250 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 rx=0x7ff558002f70 tx=0x7ff558014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:29.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.697+0000 7ff55ffff700 1 -- 172.21.15.60:0/782616773 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff558012930 con 0x7ff5680b1930 2022-06-29T16:06:29.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.697+0000 7ff55ffff700 1 -- 172.21.15.60:0/782616773 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff558012a90 con 0x7ff5680b1930 2022-06-29T16:06:29.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.697+0000 7ff56d95f700 1 -- 172.21.15.60:0/782616773 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff56814a700 con 0x7ff5680b1930 2022-06-29T16:06:29.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.697+0000 7ff55ffff700 1 -- 172.21.15.60:0/782616773 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff558012df0 con 0x7ff5680b1930 2022-06-29T16:06:29.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.697+0000 7ff56d95f700 1 -- 172.21.15.60:0/782616773 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff568156dd0 con 0x7ff5680b1930 2022-06-29T16:06:29.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.697+0000 7ff56d95f700 1 -- 172.21.15.60:0/782616773 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5680573d0 con 0x7ff5680b1930 2022-06-29T16:06:29.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.697+0000 7ff55ffff700 1 -- 172.21.15.60:0/782616773 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff55801c070 con 0x7ff5680b1930 2022-06-29T16:06:29.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.697+0000 7ff55ffff700 1 --2- 172.21.15.60:0/782616773 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff54c053aa0 0x7ff54c055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:29.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.697+0000 7ff55ffff700 1 -- 172.21.15.60:0/782616773 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff55806ca10 con 0x7ff5680b1930 2022-06-29T16:06:29.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.697+0000 7ff566ffd700 1 --2- 172.21.15.60:0/782616773 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff54c053aa0 0x7ff54c055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:29.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.701+0000 7ff566ffd700 1 --2- 172.21.15.60:0/782616773 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff54c053aa0 0x7ff54c055f60 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7ff55000d000 tx=0x7ff550009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:29.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.705+0000 7ff55ffff700 1 -- 172.21.15.60:0/782616773 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff558043890 con 0x7ff5680b1930 2022-06-29T16:06:29.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.833+0000 7ff56d95f700 1 -- 172.21.15.60:0/782616773 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff568066330 con 0x7ff5680b1930 2022-06-29T16:06:29.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.833+0000 7ff55ffff700 1 -- 172.21.15.60:0/782616773 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16051 (secure 0 0 0) 0x7ff558019030 con 0x7ff5680b1930 2022-06-29T16:06:29.839 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:29.839 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":110,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:06:29.253422+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:06:26.864385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2022-06-29T16:06:27.251709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:06:29.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.837+0000 7ff56d95f700 1 -- 172.21.15.60:0/782616773 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff54c053aa0 msgr2=0x7ff54c055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:29.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.837+0000 7ff56d95f700 1 --2- 172.21.15.60:0/782616773 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff54c053aa0 0x7ff54c055f60 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7ff55000d000 tx=0x7ff550009040).stop 2022-06-29T16:06:29.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.837+0000 7ff56d95f700 1 -- 172.21.15.60:0/782616773 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5680b1930 msgr2=0x7ff568156250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:29.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.837+0000 7ff56d95f700 1 --2- 172.21.15.60:0/782616773 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5680b1930 0x7ff568156250 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 rx=0x7ff558002f70 tx=0x7ff558014000).stop 2022-06-29T16:06:29.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.837+0000 7ff56d95f700 1 -- 172.21.15.60:0/782616773 shutdown_connections 2022-06-29T16:06:29.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.837+0000 7ff56d95f700 1 --2- 172.21.15.60:0/782616773 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff54c053aa0 0x7ff54c055f60 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:29.843 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.837+0000 7ff56d95f700 1 --2- 172.21.15.60:0/782616773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5680b5db0 0x7ff568156740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:29.843 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.837+0000 7ff56d95f700 1 --2- 172.21.15.60:0/782616773 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff5680b1930 0x7ff568156250 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:29.843 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.837+0000 7ff56d95f700 1 --2- 172.21.15.60:0/782616773 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff56805c1f0 0x7ff568064520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:29.843 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.837+0000 7ff56d95f700 1 -- 172.21.15.60:0/782616773 >> 172.21.15.60:0/782616773 conn(0x7ff5680636f0 msgr2=0x7ff5680b52c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:29.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.837+0000 7ff56d95f700 1 -- 172.21.15.60:0/782616773 shutdown_connections 2022-06-29T16:06:29.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.837+0000 7ff56d95f700 1 -- 172.21.15.60:0/782616773 wait complete. 2022-06-29T16:06:29.853 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T16:06:29.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.965+0000 7f5fb66ff700 1 Processor -- start 2022-06-29T16:06:29.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.965+0000 7f5fb66ff700 1 -- start start 2022-06-29T16:06:29.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.965+0000 7f5fb66ff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fb005fb00 0x7f5fb0143900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:29.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.965+0000 7f5fb66ff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5fb013fd60 0x7f5fb00ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:29.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.965+0000 7f5fb66ff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fb00af240 con 0x7f5fb00af940 2022-06-29T16:06:29.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.965+0000 7f5fb66ff700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fb0143eb0 con 0x7f5fb013fd60 2022-06-29T16:06:29.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.965+0000 7f5fb66ff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb005fee0 con 0x7f5fb005fb00 2022-06-29T16:06:29.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.965+0000 7f5fb5efe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5fb013fd60 0x7f5fb00ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:29.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.965+0000 7f5fb4efc700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5fb00af940 0x7f5fb00aca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60212/0 (socket says 172.21.15.60:60212) 2022-06-29T16:06:29.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.965+0000 7f5fb4efc700 1 -- 172.21.15.60:0/843758033 learned_addr learned my addr 172.21.15.60:0/843758033 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:29.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.965+0000 7f5fb56fd700 1 --2- 172.21.15.60:0/843758033 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fb005fb00 0x7f5fb0143900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:29.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.965+0000 7f5fb5efe700 1 -- 172.21.15.60:0/843758033 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fb005fb00 msgr2=0x7f5fb0143900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:29.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.965+0000 7f5f9ffff700 1 -- 172.21.15.60:0/843758033 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4028129800 0 0) 0x7f5fb00af240 con 0x7f5fb00af940 2022-06-29T16:06:29.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.965+0000 7f5fb5efe700 1 --2- 172.21.15.60:0/843758033 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fb005fb00 0x7f5fb0143900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:29.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.965+0000 7f5fb5efe700 1 -- 172.21.15.60:0/843758033 >> v1:172.21.15.60:6789/0 conn(0x7f5fb00af940 legacy=0x7f5fb00aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:29.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.965+0000 7f5fb5efe700 1 -- 172.21.15.60:0/843758033 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb00af4a0 con 0x7f5fb013fd60 2022-06-29T16:06:29.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.965+0000 7f5fb56fd700 1 --2- 172.21.15.60:0/843758033 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fb005fb00 0x7f5fb0143900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:06:29.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.965+0000 7f5fb5efe700 1 --2- 172.21.15.60:0/843758033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fb013fd60 0x7f5fb00ad130 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 rx=0x7f5fa80032f0 tx=0x7f5fa80091d0).ready entity=mon.0 client_cookie=b7a154c62306de6e server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:29.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.965+0000 7f5f9ffff700 1 -- 172.21.15.60:0/843758033 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fa8009ea0 con 0x7f5fb013fd60 2022-06-29T16:06:29.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.965+0000 7f5f9ffff700 1 -- 172.21.15.60:0/843758033 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa800b870 con 0x7f5fb013fd60 2022-06-29T16:06:29.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.965+0000 7f5f9ffff700 1 -- 172.21.15.60:0/843758033 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fa800b9e0 con 0x7f5fb013fd60 2022-06-29T16:06:29.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.965+0000 7f5fb66ff700 1 -- 172.21.15.60:0/843758033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fb013fd60 msgr2=0x7f5fb00ad130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:29.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.965+0000 7f5fb66ff700 1 --2- 172.21.15.60:0/843758033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fb013fd60 0x7f5fb00ad130 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 rx=0x7f5fa80032f0 tx=0x7f5fa80091d0).stop 2022-06-29T16:06:29.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.965+0000 7f5fb66ff700 1 -- 172.21.15.60:0/843758033 shutdown_connections 2022-06-29T16:06:29.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.965+0000 7f5fb66ff700 1 --2- 172.21.15.60:0/843758033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fb013fd60 0x7f5fb00ad130 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:29.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.965+0000 7f5fb66ff700 1 --2- 172.21.15.60:0/843758033 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fb005fb00 0x7f5fb0143900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:29.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.965+0000 7f5fb66ff700 1 -- 172.21.15.60:0/843758033 >> 172.21.15.60:0/843758033 conn(0x7f5fb0063ab0 msgr2=0x7f5fb005f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:29.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.965+0000 7f5fb66ff700 1 -- 172.21.15.60:0/843758033 shutdown_connections 2022-06-29T16:06:29.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.965+0000 7f5fb66ff700 1 -- 172.21.15.60:0/843758033 wait complete. 2022-06-29T16:06:29.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.969+0000 7f5fb66ff700 1 Processor -- start 2022-06-29T16:06:29.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.969+0000 7f5fb66ff700 1 -- start start 2022-06-29T16:06:29.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.969+0000 7f5fb66ff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fb005fb00 0x7f5fb00b0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:29.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.969+0000 7f5fb66ff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fb00af940 0x7f5fb013ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:29.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.969+0000 7f5fb66ff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fb013fd60 0x7f5fb013d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:29.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.969+0000 7f5fb66ff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb0065670 con 0x7f5fb013fd60 2022-06-29T16:06:29.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.969+0000 7f5fb66ff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb0142d80 con 0x7f5fb005fb00 2022-06-29T16:06:29.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.969+0000 7f5fb66ff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb0141a50 con 0x7f5fb00af940 2022-06-29T16:06:29.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.969+0000 7f5fb5efe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fb013fd60 0x7f5fb013d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:29.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.969+0000 7f5fb5efe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fb013fd60 0x7f5fb013d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52132/0 (socket says 172.21.15.60:52132) 2022-06-29T16:06:29.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.969+0000 7f5fb5efe700 1 -- 172.21.15.60:0/1775957355 learned_addr learned my addr 172.21.15.60:0/1775957355 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:29.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.969+0000 7f5fb4efc700 1 --2- 172.21.15.60:0/1775957355 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fb00af940 0x7f5fb013ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:29.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.969+0000 7f5fb56fd700 1 --2- 172.21.15.60:0/1775957355 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fb005fb00 0x7f5fb00b0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:29.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.969+0000 7f5fb5efe700 1 -- 172.21.15.60:0/1775957355 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fb00af940 msgr2=0x7f5fb013ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:29.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.969+0000 7f5fb5efe700 1 --2- 172.21.15.60:0/1775957355 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fb00af940 0x7f5fb013ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:29.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.969+0000 7f5fb5efe700 1 -- 172.21.15.60:0/1775957355 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fb005fb00 msgr2=0x7f5fb00b0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:29.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.969+0000 7f5fb5efe700 1 --2- 172.21.15.60:0/1775957355 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fb005fb00 0x7f5fb00b0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:29.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.969+0000 7f5fb5efe700 1 -- 172.21.15.60:0/1775957355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa8009470 con 0x7f5fb013fd60 2022-06-29T16:06:29.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.969+0000 7f5fb56fd700 1 --2- 172.21.15.60:0/1775957355 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fb005fb00 0x7f5fb00b0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:06:29.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.969+0000 7f5fb5efe700 1 --2- 172.21.15.60:0/1775957355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fb013fd60 0x7f5fb013d3c0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 rx=0x7f5fb0065fd0 tx=0x7f5fa8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:29.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.969+0000 7f5f9e7fc700 1 -- 172.21.15.60:0/1775957355 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fa8006ea0 con 0x7f5fb013fd60 2022-06-29T16:06:29.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.969+0000 7f5f9e7fc700 1 -- 172.21.15.60:0/1775957355 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa8021630 con 0x7f5fb013fd60 2022-06-29T16:06:29.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.969+0000 7f5fb66ff700 1 -- 172.21.15.60:0/1775957355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fb0141cf0 con 0x7f5fb013fd60 2022-06-29T16:06:29.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.969+0000 7f5f9e7fc700 1 -- 172.21.15.60:0/1775957355 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fa800b9e0 con 0x7f5fb013fd60 2022-06-29T16:06:29.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.969+0000 7f5fb66ff700 1 -- 172.21.15.60:0/1775957355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fb013d8b0 con 0x7f5fb013fd60 2022-06-29T16:06:29.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.973+0000 7f5fb66ff700 1 -- 172.21.15.60:0/1775957355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fb00573d0 con 0x7f5fb013fd60 2022-06-29T16:06:29.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.973+0000 7f5f9e7fc700 1 -- 172.21.15.60:0/1775957355 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5fa8027020 con 0x7f5fb013fd60 2022-06-29T16:06:29.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.973+0000 7f5f9e7fc700 1 --2- 172.21.15.60:0/1775957355 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5f90053950 0x7f5f90055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:29.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.973+0000 7f5f9e7fc700 1 -- 172.21.15.60:0/1775957355 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5fa806d3d0 con 0x7f5fb013fd60 2022-06-29T16:06:29.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.973+0000 7f5fb56fd700 1 --2- 172.21.15.60:0/1775957355 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5f90053950 0x7f5f90055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:29.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.973+0000 7f5fb56fd700 1 --2- 172.21.15.60:0/1775957355 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5f90053950 0x7f5f90055e10 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0x7f5fa000c000 tx=0x7f5fa0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:29.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:29.981+0000 7f5f9e7fc700 1 -- 172.21.15.60:0/1775957355 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5fa80442f0 con 0x7f5fb013fd60 2022-06-29T16:06:30.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.149+0000 7f5fb66ff700 1 -- 172.21.15.60:0/1775957355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f5fb0067270 con 0x7f5fb013fd60 2022-06-29T16:06:30.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.157+0000 7f5f9e7fc700 1 -- 172.21.15.60:0/1775957355 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v38) v1 ==== 184+0+0 (secure 0 0 0) 0x7f5fa80150e0 con 0x7f5fb013fd60 2022-06-29T16:06:30.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.157+0000 7f5fb66ff700 1 -- 172.21.15.60:0/1775957355 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5f90053950 msgr2=0x7f5f90055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:30.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.157+0000 7f5fb66ff700 1 --2- 172.21.15.60:0/1775957355 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5f90053950 0x7f5f90055e10 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0x7f5fa000c000 tx=0x7f5fa0008040).stop 2022-06-29T16:06:30.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.157+0000 7f5fb66ff700 1 -- 172.21.15.60:0/1775957355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fb013fd60 msgr2=0x7f5fb013d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:30.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.157+0000 7f5fb66ff700 1 --2- 172.21.15.60:0/1775957355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fb013fd60 0x7f5fb013d3c0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 rx=0x7f5fb0065fd0 tx=0x7f5fa8016000).stop 2022-06-29T16:06:30.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.157+0000 7f5fb66ff700 1 -- 172.21.15.60:0/1775957355 shutdown_connections 2022-06-29T16:06:30.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.157+0000 7f5fb66ff700 1 --2- 172.21.15.60:0/1775957355 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5f90053950 0x7f5f90055e10 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:30.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.157+0000 7f5fb66ff700 1 --2- 172.21.15.60:0/1775957355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fb013fd60 0x7f5fb013d3c0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:30.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.157+0000 7f5fb66ff700 1 --2- 172.21.15.60:0/1775957355 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fb00af940 0x7f5fb013ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:30.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.157+0000 7f5fb66ff700 1 --2- 172.21.15.60:0/1775957355 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fb005fb00 0x7f5fb00b0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:30.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.157+0000 7f5fb66ff700 1 -- 172.21.15.60:0/1775957355 >> 172.21.15.60:0/1775957355 conn(0x7f5fb0063ab0 msgr2=0x7f5fb00b20d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:30.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.157+0000 7f5fb66ff700 1 -- 172.21.15.60:0/1775957355 shutdown_connections 2022-06-29T16:06:30.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.157+0000 7f5fb66ff700 1 -- 172.21.15.60:0/1775957355 wait complete. 2022-06-29T16:06:30.166 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.0 2022-06-29T16:06:30.177 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T16:06:30.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.289+0000 7ffa73fff700 1 Processor -- start 2022-06-29T16:06:30.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.289+0000 7ffa73fff700 1 -- start start 2022-06-29T16:06:30.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.289+0000 7ffa73fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ffa740b0280 0x7ffa74146650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:30.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.289+0000 7ffa73fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffa74146b40 0x7ffa740b2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:30.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.289+0000 7ffa73fff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa74146fd0 con 0x7ffa740b5eb0 2022-06-29T16:06:30.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.289+0000 7ffa73fff700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa740b6290 con 0x7ffa740b0280 2022-06-29T16:06:30.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.289+0000 7ffa73fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa740b0ab0 con 0x7ffa74146b40 2022-06-29T16:06:30.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.289+0000 7ffa72ffd700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ffa740b5eb0 0x7ffa74142a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60224/0 (socket says 172.21.15.60:60224) 2022-06-29T16:06:30.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.289+0000 7ffa72ffd700 1 -- 172.21.15.60:0/1364546954 learned_addr learned my addr 172.21.15.60:0/1364546954 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:30.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.289+0000 7ffa737fe700 1 --2- 172.21.15.60:0/1364546954 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffa74146b40 0x7ffa740b2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:30.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.289+0000 7ffa6bfff700 1 --2- 172.21.15.60:0/1364546954 >> v2:172.21.15.92:3300/0 conn(0x7ffa740b0280 0x7ffa74146650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:30.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.289+0000 7ffa727fc700 1 -- 172.21.15.60:0/1364546954 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 999249589 0 0) 0x7ffa64001110 con 0x7ffa740b5eb0 2022-06-29T16:06:30.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.289+0000 7ffa727fc700 1 -- 172.21.15.60:0/1364546954 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa54003260 con 0x7ffa740b5eb0 2022-06-29T16:06:30.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.289+0000 7ffa737fe700 1 -- 172.21.15.60:0/1364546954 >> v2:172.21.15.92:3300/0 conn(0x7ffa740b0280 msgr2=0x7ffa74146650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:30.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.289+0000 7ffa737fe700 1 --2- 172.21.15.60:0/1364546954 >> v2:172.21.15.92:3300/0 conn(0x7ffa740b0280 0x7ffa74146650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:30.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.289+0000 7ffa737fe700 1 -- 172.21.15.60:0/1364546954 >> v1:172.21.15.60:6789/0 conn(0x7ffa740b5eb0 legacy=0x7ffa74142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:30.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.289+0000 7ffa737fe700 1 -- 172.21.15.60:0/1364546954 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa741461b0 con 0x7ffa74146b40 2022-06-29T16:06:30.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.293+0000 7ffa737fe700 1 --2- 172.21.15.60:0/1364546954 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffa74146b40 0x7ffa740b2bb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7ffa74143130 tx=0x7ffa6c0091d0).ready entity=mon.2 client_cookie=cab546bbdb9c06c3 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:30.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.293+0000 7ffa727fc700 1 -- 172.21.15.60:0/1364546954 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa6c00b600 con 0x7ffa74146b40 2022-06-29T16:06:30.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.293+0000 7ffa727fc700 1 -- 172.21.15.60:0/1364546954 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa6c00b760 con 0x7ffa74146b40 2022-06-29T16:06:30.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.293+0000 7ffa727fc700 1 -- 172.21.15.60:0/1364546954 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa6c00b8d0 con 0x7ffa74146b40 2022-06-29T16:06:30.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.293+0000 7ffa73fff700 1 -- 172.21.15.60:0/1364546954 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffa74146b40 msgr2=0x7ffa740b2bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:30.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.293+0000 7ffa73fff700 1 --2- 172.21.15.60:0/1364546954 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffa74146b40 0x7ffa740b2bb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7ffa74143130 tx=0x7ffa6c0091d0).stop 2022-06-29T16:06:30.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.293+0000 7ffa73fff700 1 -- 172.21.15.60:0/1364546954 shutdown_connections 2022-06-29T16:06:30.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.293+0000 7ffa73fff700 1 --2- 172.21.15.60:0/1364546954 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffa74146b40 0x7ffa740b2bb0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:30.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.293+0000 7ffa73fff700 1 --2- 172.21.15.60:0/1364546954 >> v2:172.21.15.92:3300/0 conn(0x7ffa740b0280 0x7ffa74146650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:30.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.293+0000 7ffa73fff700 1 -- 172.21.15.60:0/1364546954 >> 172.21.15.60:0/1364546954 conn(0x7ffa74062090 msgr2=0x7ffa74062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:30.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.293+0000 7ffa73fff700 1 -- 172.21.15.60:0/1364546954 shutdown_connections 2022-06-29T16:06:30.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.293+0000 7ffa73fff700 1 -- 172.21.15.60:0/1364546954 wait complete. 2022-06-29T16:06:30.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.293+0000 7ffa73fff700 1 Processor -- start 2022-06-29T16:06:30.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.293+0000 7ffa73fff700 1 -- start start 2022-06-29T16:06:30.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.297+0000 7ffa73fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa740b0280 0x7ffa7413deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:30.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.297+0000 7ffa73fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffa740b5eb0 0x7ffa74144270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:30.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.297+0000 7ffa73fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffa74146b40 0x7ffa7413b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:30.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.297+0000 7ffa73fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa740b70f0 con 0x7ffa740b0280 2022-06-29T16:06:30.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.297+0000 7ffa73fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa740b7390 con 0x7ffa74146b40 2022-06-29T16:06:30.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.297+0000 7ffa73fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7413c600 con 0x7ffa740b5eb0 2022-06-29T16:06:30.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.297+0000 7ffa72ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa740b0280 0x7ffa7413deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:30.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.297+0000 7ffa72ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa740b0280 0x7ffa7413deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52144/0 (socket says 172.21.15.60:52144) 2022-06-29T16:06:30.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.297+0000 7ffa72ffd700 1 -- 172.21.15.60:0/582758275 learned_addr learned my addr 172.21.15.60:0/582758275 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:30.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.297+0000 7ffa6bfff700 1 --2- 172.21.15.60:0/582758275 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffa740b5eb0 0x7ffa74144270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:30.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.297+0000 7ffa737fe700 1 --2- 172.21.15.60:0/582758275 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffa74146b40 0x7ffa7413b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:30.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.297+0000 7ffa72ffd700 1 -- 172.21.15.60:0/582758275 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffa740b5eb0 msgr2=0x7ffa74144270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:30.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.297+0000 7ffa72ffd700 1 --2- 172.21.15.60:0/582758275 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffa740b5eb0 0x7ffa74144270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:30.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.297+0000 7ffa72ffd700 1 -- 172.21.15.60:0/582758275 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffa74146b40 msgr2=0x7ffa7413b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:30.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.297+0000 7ffa72ffd700 1 --2- 172.21.15.60:0/582758275 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffa74146b40 0x7ffa7413b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:30.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.297+0000 7ffa72ffd700 1 -- 172.21.15.60:0/582758275 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa6c009470 con 0x7ffa740b0280 2022-06-29T16:06:30.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.297+0000 7ffa737fe700 1 --2- 172.21.15.60:0/582758275 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffa74146b40 0x7ffa7413b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:06:30.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.297+0000 7ffa72ffd700 1 --2- 172.21.15.60:0/582758275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa740b0280 0x7ffa7413deb0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 rx=0x7ffa640012a0 tx=0x7ffa64007900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:30.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.297+0000 7ffa70ff9700 1 -- 172.21.15.60:0/582758275 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa6400ce30 con 0x7ffa740b0280 2022-06-29T16:06:30.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.297+0000 7ffa70ff9700 1 -- 172.21.15.60:0/582758275 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa640072e0 con 0x7ffa740b0280 2022-06-29T16:06:30.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.297+0000 7ffa73fff700 1 -- 172.21.15.60:0/582758275 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa740aca20 con 0x7ffa740b0280 2022-06-29T16:06:30.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.297+0000 7ffa70ff9700 1 -- 172.21.15.60:0/582758275 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa64005610 con 0x7ffa740b0280 2022-06-29T16:06:30.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.297+0000 7ffa73fff700 1 -- 172.21.15.60:0/582758275 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa7413ba90 con 0x7ffa740b0280 2022-06-29T16:06:30.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.301+0000 7ffa73fff700 1 -- 172.21.15.60:0/582758275 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa740b7550 con 0x7ffa740b0280 2022-06-29T16:06:30.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.301+0000 7ffa70ff9700 1 -- 172.21.15.60:0/582758275 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ffa6401a070 con 0x7ffa740b0280 2022-06-29T16:06:30.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.301+0000 7ffa70ff9700 1 --2- 172.21.15.60:0/582758275 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ffa58053a70 0x7ffa58055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:30.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.301+0000 7ffa70ff9700 1 -- 172.21.15.60:0/582758275 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ffa6406b140 con 0x7ffa740b0280 2022-06-29T16:06:30.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.301+0000 7ffa6bfff700 1 --2- 172.21.15.60:0/582758275 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ffa58053a70 0x7ffa58055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:30.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.301+0000 7ffa6bfff700 1 --2- 172.21.15.60:0/582758275 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ffa58053a70 0x7ffa58055f30 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0x7ffa5c00b000 tx=0x7ffa5c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:30.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.309+0000 7ffa70ff9700 1 -- 172.21.15.60:0/582758275 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ffa64041fc0 con 0x7ffa740b0280 2022-06-29T16:06:30.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.477+0000 7ffa73fff700 1 -- 172.21.15.60:0/582758275 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7ffa74146fd0 con 0x7ffa740b0280 2022-06-29T16:06:30.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.485+0000 7ffa70ff9700 1 -- 172.21.15.60:0/582758275 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v39) v1 ==== 184+0+0 (secure 0 0 0) 0x7ffa74146fd0 con 0x7ffa740b0280 2022-06-29T16:06:30.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.485+0000 7ffa73fff700 1 -- 172.21.15.60:0/582758275 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ffa58053a70 msgr2=0x7ffa58055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:30.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.485+0000 7ffa73fff700 1 --2- 172.21.15.60:0/582758275 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ffa58053a70 0x7ffa58055f30 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0x7ffa5c00b000 tx=0x7ffa5c006df0).stop 2022-06-29T16:06:30.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.485+0000 7ffa73fff700 1 -- 172.21.15.60:0/582758275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa740b0280 msgr2=0x7ffa7413deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:30.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.485+0000 7ffa73fff700 1 --2- 172.21.15.60:0/582758275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa740b0280 0x7ffa7413deb0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 rx=0x7ffa640012a0 tx=0x7ffa64007900).stop 2022-06-29T16:06:30.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.485+0000 7ffa73fff700 1 -- 172.21.15.60:0/582758275 shutdown_connections 2022-06-29T16:06:30.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.485+0000 7ffa73fff700 1 --2- 172.21.15.60:0/582758275 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ffa58053a70 0x7ffa58055f30 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:30.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.485+0000 7ffa73fff700 1 --2- 172.21.15.60:0/582758275 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffa74146b40 0x7ffa7413b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:30.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.485+0000 7ffa73fff700 1 --2- 172.21.15.60:0/582758275 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffa740b5eb0 0x7ffa74144270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:30.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.485+0000 7ffa73fff700 1 --2- 172.21.15.60:0/582758275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa740b0280 0x7ffa7413deb0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:30.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.485+0000 7ffa73fff700 1 -- 172.21.15.60:0/582758275 >> 172.21.15.60:0/582758275 conn(0x7ffa74062090 msgr2=0x7ffa740b24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:30.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.485+0000 7ffa73fff700 1 -- 172.21.15.60:0/582758275 shutdown_connections 2022-06-29T16:06:30.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.485+0000 7ffa73fff700 1 -- 172.21.15.60:0/582758275 wait complete. 2022-06-29T16:06:30.496 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.1 2022-06-29T16:06:30.503 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T16:06:30.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.609+0000 7f0bc0725700 1 Processor -- start 2022-06-29T16:06:30.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.609+0000 7f0bc0725700 1 -- start start 2022-06-29T16:06:30.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.609+0000 7f0bc0725700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0bb80b5e90 0x7f0bb80b6270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:30.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.609+0000 7f0bc0725700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bb8060710 0x7f0bb80ae9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:30.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.609+0000 7f0bc0725700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb8060c60 con 0x7f0bb8142d20 2022-06-29T16:06:30.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.609+0000 7f0bc0725700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bb80b1d30 con 0x7f0bb80b5e90 2022-06-29T16:06:30.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.609+0000 7f0bc0725700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb8060e90 con 0x7f0bb8060710 2022-06-29T16:06:30.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.609+0000 7f0bbe4c1700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0bb80b5e90 0x7f0bb80b6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:30.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.609+0000 7f0bbecc2700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f0bb8142d20 0x7f0bb8143170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:36256/0 (socket says 172.21.15.60:36256) 2022-06-29T16:06:30.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.609+0000 7f0bbe4c1700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0bb80b5e90 0x7f0bb80b6270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:58182/0 (socket says 172.21.15.60:58182) 2022-06-29T16:06:30.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.609+0000 7f0bbecc2700 1 -- 172.21.15.60:0/2488643122 learned_addr learned my addr 172.21.15.60:0/2488643122 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:30.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.609+0000 7f0bbdcc0700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bb8060710 0x7f0bb80ae9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:30.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.609+0000 7f0bbe4c1700 1 -- 172.21.15.60:0/2488643122 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bb8060710 msgr2=0x7f0bb80ae9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:30.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.609+0000 7f0bbe4c1700 1 --2- 172.21.15.60:0/2488643122 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bb8060710 0x7f0bb80ae9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:30.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.609+0000 7f0bbe4c1700 1 -- 172.21.15.60:0/2488643122 >> v1:172.21.15.92:6789/0 conn(0x7f0bb8142d20 legacy=0x7f0bb8143170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:30.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.609+0000 7f0bbd4bf700 1 -- 172.21.15.60:0/2488643122 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3375153721 0 0) 0x7f0bb8060c60 con 0x7f0bb8142d20 2022-06-29T16:06:30.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.609+0000 7f0bbe4c1700 1 -- 172.21.15.60:0/2488643122 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb80ae2e0 con 0x7f0bb80b5e90 2022-06-29T16:06:30.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.613+0000 7f0bbe4c1700 1 --2- 172.21.15.60:0/2488643122 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bb80b5e90 0x7f0bb80b6270 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7f0ba8002f70 tx=0x7f0ba8006df0).ready entity=mon.1 client_cookie=e6feafee19a963e0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:30.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.613+0000 7f0bbd4bf700 1 -- 172.21.15.60:0/2488643122 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ba800d570 con 0x7f0bb80b5e90 2022-06-29T16:06:30.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.613+0000 7f0bbd4bf700 1 -- 172.21.15.60:0/2488643122 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ba8000e90 con 0x7f0bb80b5e90 2022-06-29T16:06:30.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.613+0000 7f0bbd4bf700 1 -- 172.21.15.60:0/2488643122 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ba800d8c0 con 0x7f0bb80b5e90 2022-06-29T16:06:30.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.613+0000 7f0bc0725700 1 -- 172.21.15.60:0/2488643122 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bb80b5e90 msgr2=0x7f0bb80b6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:30.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.613+0000 7f0bc0725700 1 --2- 172.21.15.60:0/2488643122 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bb80b5e90 0x7f0bb80b6270 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7f0ba8002f70 tx=0x7f0ba8006df0).stop 2022-06-29T16:06:30.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.613+0000 7f0bc0725700 1 -- 172.21.15.60:0/2488643122 shutdown_connections 2022-06-29T16:06:30.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.613+0000 7f0bc0725700 1 --2- 172.21.15.60:0/2488643122 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bb8060710 0x7f0bb80ae9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:30.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.613+0000 7f0bc0725700 1 --2- 172.21.15.60:0/2488643122 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bb80b5e90 0x7f0bb80b6270 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:30.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.613+0000 7f0bc0725700 1 -- 172.21.15.60:0/2488643122 >> 172.21.15.60:0/2488643122 conn(0x7f0bb8063b50 msgr2=0x7f0bb80b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:30.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.613+0000 7f0bc0725700 1 -- 172.21.15.60:0/2488643122 shutdown_connections 2022-06-29T16:06:30.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.613+0000 7f0bc0725700 1 -- 172.21.15.60:0/2488643122 wait complete. 2022-06-29T16:06:30.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.613+0000 7f0bc0725700 1 Processor -- start 2022-06-29T16:06:30.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.613+0000 7f0bc0725700 1 -- start start 2022-06-29T16:06:30.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.617+0000 7f0bc0725700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bb8060710 0x7f0bb80b0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:30.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.617+0000 7f0bc0725700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bb80b5e90 0x7f0bb80661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:30.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.617+0000 7f0bc0725700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bb8142d20 0x7f0bb80666a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:30.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.617+0000 7f0bc0725700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb80651f0 con 0x7f0bb80b5e90 2022-06-29T16:06:30.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.617+0000 7f0bc0725700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb80b4300 con 0x7f0bb8142d20 2022-06-29T16:06:30.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.617+0000 7f0bc0725700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb80b3a70 con 0x7f0bb8060710 2022-06-29T16:06:30.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.617+0000 7f0bbdcc0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bb80b5e90 0x7f0bb80661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:30.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.617+0000 7f0bbdcc0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bb80b5e90 0x7f0bb80661b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52158/0 (socket says 172.21.15.60:52158) 2022-06-29T16:06:30.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.617+0000 7f0bbdcc0700 1 -- 172.21.15.60:0/1108214821 learned_addr learned my addr 172.21.15.60:0/1108214821 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:30.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.617+0000 7f0bbe4c1700 1 --2- 172.21.15.60:0/1108214821 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bb8060710 0x7f0bb80b0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:30.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.617+0000 7f0bbecc2700 1 --2- 172.21.15.60:0/1108214821 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bb8142d20 0x7f0bb80666a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:30.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.617+0000 7f0bbdcc0700 1 -- 172.21.15.60:0/1108214821 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bb8060710 msgr2=0x7f0bb80b0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:30.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.617+0000 7f0bbdcc0700 1 --2- 172.21.15.60:0/1108214821 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bb8060710 0x7f0bb80b0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:30.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.617+0000 7f0bbdcc0700 1 -- 172.21.15.60:0/1108214821 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bb8142d20 msgr2=0x7f0bb80666a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:30.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.617+0000 7f0bbdcc0700 1 --2- 172.21.15.60:0/1108214821 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bb8142d20 0x7f0bb80666a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:30.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.617+0000 7f0bbdcc0700 1 -- 172.21.15.60:0/1108214821 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba800b180 con 0x7f0bb80b5e90 2022-06-29T16:06:30.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.617+0000 7f0bbe4c1700 1 --2- 172.21.15.60:0/1108214821 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bb8060710 0x7f0bb80b0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:06:30.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.617+0000 7f0bbdcc0700 1 --2- 172.21.15.60:0/1108214821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bb80b5e90 0x7f0bb80661b0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 rx=0x7f0bb400c000 tx=0x7f0bb40091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:30.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.617+0000 7f0ba77fe700 1 -- 172.21.15.60:0/1108214821 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0bb400c6a0 con 0x7f0bb80b5e90 2022-06-29T16:06:30.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.617+0000 7f0ba77fe700 1 -- 172.21.15.60:0/1108214821 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb400c800 con 0x7f0bb80b5e90 2022-06-29T16:06:30.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.617+0000 7f0bc0725700 1 -- 172.21.15.60:0/1108214821 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bb8145970 con 0x7f0bb80b5e90 2022-06-29T16:06:30.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.617+0000 7f0ba77fe700 1 -- 172.21.15.60:0/1108214821 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0bb400cb60 con 0x7f0bb80b5e90 2022-06-29T16:06:30.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.617+0000 7f0bc0725700 1 -- 172.21.15.60:0/1108214821 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb80b3760 con 0x7f0bb80b5e90 2022-06-29T16:06:30.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.621+0000 7f0bc0725700 1 -- 172.21.15.60:0/1108214821 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bb80ae510 con 0x7f0bb80b5e90 2022-06-29T16:06:30.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.621+0000 7f0ba77fe700 1 -- 172.21.15.60:0/1108214821 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0bb4022070 con 0x7f0bb80b5e90 2022-06-29T16:06:30.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.621+0000 7f0ba77fe700 1 --2- 172.21.15.60:0/1108214821 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0b98053aa0 0x7f0b98055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:30.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.621+0000 7f0ba77fe700 1 -- 172.21.15.60:0/1108214821 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0bb406c420 con 0x7f0bb80b5e90 2022-06-29T16:06:30.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.621+0000 7f0bbe4c1700 1 --2- 172.21.15.60:0/1108214821 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0b98053aa0 0x7f0b98055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:30.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.621+0000 7f0bbe4c1700 1 --2- 172.21.15.60:0/1108214821 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0b98053aa0 0x7f0b98055f60 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7f0ba801a000 tx=0x7f0ba8014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:30.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.625+0000 7f0ba77fe700 1 -- 172.21.15.60:0/1108214821 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0bb4043340 con 0x7f0bb80b5e90 2022-06-29T16:06:30.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.789+0000 7f0bc0725700 1 -- 172.21.15.60:0/1108214821 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f0bb80573d0 con 0x7f0bb80b5e90 2022-06-29T16:06:30.803 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.797+0000 7f0ba77fe700 1 -- 172.21.15.60:0/1108214821 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v40) v1 ==== 184+0+0 (secure 0 0 0) 0x7f0bb4046960 con 0x7f0bb80b5e90 2022-06-29T16:06:30.807 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.801+0000 7f0bc0725700 1 -- 172.21.15.60:0/1108214821 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0b98053aa0 msgr2=0x7f0b98055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:30.807 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.801+0000 7f0bc0725700 1 --2- 172.21.15.60:0/1108214821 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0b98053aa0 0x7f0b98055f60 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7f0ba801a000 tx=0x7f0ba8014000).stop 2022-06-29T16:06:30.807 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.801+0000 7f0bc0725700 1 -- 172.21.15.60:0/1108214821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bb80b5e90 msgr2=0x7f0bb80661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:30.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.801+0000 7f0bc0725700 1 --2- 172.21.15.60:0/1108214821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bb80b5e90 0x7f0bb80661b0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 rx=0x7f0bb400c000 tx=0x7f0bb40091d0).stop 2022-06-29T16:06:30.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.801+0000 7f0bc0725700 1 -- 172.21.15.60:0/1108214821 shutdown_connections 2022-06-29T16:06:30.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.801+0000 7f0bc0725700 1 --2- 172.21.15.60:0/1108214821 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0b98053aa0 0x7f0b98055f60 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:30.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.801+0000 7f0bc0725700 1 --2- 172.21.15.60:0/1108214821 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bb8142d20 0x7f0bb80666a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:30.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.801+0000 7f0bc0725700 1 --2- 172.21.15.60:0/1108214821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bb80b5e90 0x7f0bb80661b0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:30.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.801+0000 7f0bc0725700 1 --2- 172.21.15.60:0/1108214821 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0bb8060710 0x7f0bb80b0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:30.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.801+0000 7f0bc0725700 1 -- 172.21.15.60:0/1108214821 >> 172.21.15.60:0/1108214821 conn(0x7f0bb8063b50 msgr2=0x7f0bb8140c20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:30.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.801+0000 7f0bc0725700 1 -- 172.21.15.60:0/1108214821 shutdown_connections 2022-06-29T16:06:30.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.801+0000 7f0bc0725700 1 -- 172.21.15.60:0/1108214821 wait complete. 2022-06-29T16:06:30.810 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.2 2022-06-29T16:06:30.824 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:06:30.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.917+0000 7fcb5b1b4700 1 Processor -- start 2022-06-29T16:06:30.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.917+0000 7fcb5b1b4700 1 -- start start 2022-06-29T16:06:30.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.917+0000 7fcb5b1b4700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcb540b28e0 0x7fcb540b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:30.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.917+0000 7fcb5b1b4700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcb54065450 0x7fcb54140180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:30.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.917+0000 7fcb5b1b4700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb54060760 0x7fcb54139340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:30.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.917+0000 7fcb5b1b4700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb54140670 con 0x7fcb54065450 2022-06-29T16:06:30.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.917+0000 7fcb5b1b4700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb5413fa40 con 0x7fcb540b28e0 2022-06-29T16:06:30.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.917+0000 7fcb5b1b4700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5413fce0 con 0x7fcb54060760 2022-06-29T16:06:30.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.917+0000 7fcb599b1700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcb54065450 0x7fcb54140180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:30.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.917+0000 7fcb599b1700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcb54065450 0x7fcb54140180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52166/0 (socket says 172.21.15.60:52166) 2022-06-29T16:06:30.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.917+0000 7fcb599b1700 1 -- 172.21.15.60:0/3420746861 learned_addr learned my addr 172.21.15.60:0/3420746861 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:30.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.917+0000 7fcb5a9b3700 1 --2- 172.21.15.60:0/3420746861 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb54060760 0x7fcb54139340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:30.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.917+0000 7fcb5a1b2700 1 --2- 172.21.15.60:0/3420746861 >> v2:172.21.15.92:3300/0 conn(0x7fcb540b28e0 0x7fcb540b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:30.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.917+0000 7fcb599b1700 1 -- 172.21.15.60:0/3420746861 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb54060760 msgr2=0x7fcb54139340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:30.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.917+0000 7fcb599b1700 1 --2- 172.21.15.60:0/3420746861 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb54060760 0x7fcb54139340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:30.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.917+0000 7fcb599b1700 1 -- 172.21.15.60:0/3420746861 >> v2:172.21.15.92:3300/0 conn(0x7fcb540b28e0 msgr2=0x7fcb540b2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:30.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.917+0000 7fcb599b1700 1 --2- 172.21.15.60:0/3420746861 >> v2:172.21.15.92:3300/0 conn(0x7fcb540b28e0 0x7fcb540b2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:30.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.917+0000 7fcb599b1700 1 -- 172.21.15.60:0/3420746861 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb54140860 con 0x7fcb54065450 2022-06-29T16:06:30.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.921+0000 7fcb599b1700 1 --2- 172.21.15.60:0/3420746861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb54065450 0x7fcb54140180 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 rx=0x7fcb48002f70 tx=0x7fcb48006eb0).ready entity=mon.0 client_cookie=afea8ee4e0ef3ed5 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:30.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.921+0000 7fcb591b0700 1 -- 172.21.15.60:0/3420746861 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb4800d570 con 0x7fcb54065450 2022-06-29T16:06:30.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.921+0000 7fcb591b0700 1 -- 172.21.15.60:0/3420746861 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb48000e90 con 0x7fcb54065450 2022-06-29T16:06:30.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.921+0000 7fcb591b0700 1 -- 172.21.15.60:0/3420746861 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb4800d900 con 0x7fcb54065450 2022-06-29T16:06:30.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.921+0000 7fcb5b1b4700 1 -- 172.21.15.60:0/3420746861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb54065450 msgr2=0x7fcb54140180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:30.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.921+0000 7fcb5b1b4700 1 --2- 172.21.15.60:0/3420746861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb54065450 0x7fcb54140180 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 rx=0x7fcb48002f70 tx=0x7fcb48006eb0).stop 2022-06-29T16:06:30.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.921+0000 7fcb5b1b4700 1 -- 172.21.15.60:0/3420746861 shutdown_connections 2022-06-29T16:06:30.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.921+0000 7fcb5b1b4700 1 --2- 172.21.15.60:0/3420746861 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb54060760 0x7fcb54139340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:30.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.921+0000 7fcb5b1b4700 1 --2- 172.21.15.60:0/3420746861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb54065450 0x7fcb54140180 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:30.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.921+0000 7fcb5b1b4700 1 --2- 172.21.15.60:0/3420746861 >> v2:172.21.15.92:3300/0 conn(0x7fcb540b28e0 0x7fcb540b2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:30.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.921+0000 7fcb5b1b4700 1 -- 172.21.15.60:0/3420746861 >> 172.21.15.60:0/3420746861 conn(0x7fcb54061c80 msgr2=0x7fcb540637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:30.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.921+0000 7fcb5b1b4700 1 -- 172.21.15.60:0/3420746861 shutdown_connections 2022-06-29T16:06:30.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.921+0000 7fcb5b1b4700 1 -- 172.21.15.60:0/3420746861 wait complete. 2022-06-29T16:06:30.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.921+0000 7fcb5b1b4700 1 Processor -- start 2022-06-29T16:06:30.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.921+0000 7fcb5b1b4700 1 -- start start 2022-06-29T16:06:30.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.925+0000 7fcb5b1b4700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb54060760 0x7fcb540b6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:30.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.925+0000 7fcb5b1b4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb54065450 0x7fcb541437b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:30.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.925+0000 7fcb5b1b4700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb540b28e0 0x7fcb5413b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:30.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.925+0000 7fcb5b1b4700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb540b5620 con 0x7fcb54065450 2022-06-29T16:06:30.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.925+0000 7fcb5b1b4700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb54151be0 con 0x7fcb540b28e0 2022-06-29T16:06:30.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.925+0000 7fcb5b1b4700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb54151e80 con 0x7fcb54060760 2022-06-29T16:06:30.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.925+0000 7fcb599b1700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb54065450 0x7fcb541437b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:30.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.925+0000 7fcb5a9b3700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb540b28e0 0x7fcb5413b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:30.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.925+0000 7fcb5a1b2700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb54060760 0x7fcb540b6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:30.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.925+0000 7fcb599b1700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb54065450 0x7fcb541437b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52174/0 (socket says 172.21.15.60:52174) 2022-06-29T16:06:30.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.925+0000 7fcb5a1b2700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb54060760 0x7fcb540b6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:37034/0 (socket says 172.21.15.60:37034) 2022-06-29T16:06:30.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.925+0000 7fcb5a9b3700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb540b28e0 0x7fcb5413b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:58206/0 (socket says 172.21.15.60:58206) 2022-06-29T16:06:30.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.925+0000 7fcb599b1700 1 -- 172.21.15.60:0/2211502433 learned_addr learned my addr 172.21.15.60:0/2211502433 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:30.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.925+0000 7fcb599b1700 1 -- 172.21.15.60:0/2211502433 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb54060760 msgr2=0x7fcb540b6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:30.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.925+0000 7fcb599b1700 1 --2- 172.21.15.60:0/2211502433 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb54060760 0x7fcb540b6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:30.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.925+0000 7fcb599b1700 1 -- 172.21.15.60:0/2211502433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb540b28e0 msgr2=0x7fcb5413b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:30.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.925+0000 7fcb599b1700 1 --2- 172.21.15.60:0/2211502433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb540b28e0 0x7fcb5413b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:30.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.925+0000 7fcb599b1700 1 -- 172.21.15.60:0/2211502433 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4800b1e0 con 0x7fcb54065450 2022-06-29T16:06:30.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.925+0000 7fcb5a1b2700 1 --2- 172.21.15.60:0/2211502433 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb54060760 0x7fcb540b6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:06:30.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.925+0000 7fcb599b1700 1 --2- 172.21.15.60:0/2211502433 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb54065450 0x7fcb541437b0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 rx=0x7fcb48002f70 tx=0x7fcb48015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:30.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.925+0000 7fcb477fe700 1 -- 172.21.15.60:0/2211502433 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb48011860 con 0x7fcb54065450 2022-06-29T16:06:30.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.925+0000 7fcb477fe700 1 -- 172.21.15.60:0/2211502433 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb480119c0 con 0x7fcb54065450 2022-06-29T16:06:30.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.925+0000 7fcb5b1b4700 1 -- 172.21.15.60:0/2211502433 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb54152120 con 0x7fcb54065450 2022-06-29T16:06:30.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.925+0000 7fcb477fe700 1 -- 172.21.15.60:0/2211502433 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb48011d20 con 0x7fcb54065450 2022-06-29T16:06:30.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.925+0000 7fcb5b1b4700 1 -- 172.21.15.60:0/2211502433 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb54142590 con 0x7fcb54065450 2022-06-29T16:06:30.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.925+0000 7fcb5b1b4700 1 -- 172.21.15.60:0/2211502433 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb540573d0 con 0x7fcb54065450 2022-06-29T16:06:30.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.929+0000 7fcb477fe700 1 -- 172.21.15.60:0/2211502433 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcb48025030 con 0x7fcb54065450 2022-06-29T16:06:30.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.929+0000 7fcb477fe700 1 --2- 172.21.15.60:0/2211502433 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcb340539e0 0x7fcb34055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:30.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.929+0000 7fcb477fe700 1 -- 172.21.15.60:0/2211502433 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcb4806d030 con 0x7fcb54065450 2022-06-29T16:06:30.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.929+0000 7fcb5a1b2700 1 --2- 172.21.15.60:0/2211502433 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcb340539e0 0x7fcb34055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:30.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.929+0000 7fcb5a1b2700 1 --2- 172.21.15.60:0/2211502433 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcb340539e0 0x7fcb34055ea0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7fcb5000b000 tx=0x7fcb50006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:30.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:30.933+0000 7fcb477fe700 1 -- 172.21.15.60:0/2211502433 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcb48043f50 con 0x7fcb54065450 2022-06-29T16:06:31.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.097+0000 7fcb5b1b4700 1 -- 172.21.15.60:0/2211502433 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb541522b0 con 0x7fcb54065450 2022-06-29T16:06:31.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.097+0000 7fcb477fe700 1 -- 172.21.15.60:0/2211502433 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+2142 (secure 0 0 0) 0x7fcb48023020 con 0x7fcb54065450 2022-06-29T16:06:31.104 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:31.104 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":51,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7648,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/3387042465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3387042465},{"type":"v1","addr":"172.21.15.92:6835","nonce":3387042465}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:06:28.249737+0000","modified":"2022-06-29T16:06:29.252773+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7668},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7668":{"gid":7668,"name":"a","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/2749128920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2749128920},{"type":"v1","addr":"172.21.15.60:6833","nonce":2749128920}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2022-06-29T16:06:31.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.101+0000 7fcb5b1b4700 1 -- 172.21.15.60:0/2211502433 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcb340539e0 msgr2=0x7fcb34055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:31.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.101+0000 7fcb5b1b4700 1 --2- 172.21.15.60:0/2211502433 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcb340539e0 0x7fcb34055ea0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7fcb5000b000 tx=0x7fcb50006df0).stop 2022-06-29T16:06:31.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.101+0000 7fcb5b1b4700 1 -- 172.21.15.60:0/2211502433 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb54065450 msgr2=0x7fcb541437b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:31.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.101+0000 7fcb5b1b4700 1 --2- 172.21.15.60:0/2211502433 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb54065450 0x7fcb541437b0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 rx=0x7fcb48002f70 tx=0x7fcb48015000).stop 2022-06-29T16:06:31.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.101+0000 7fcb5b1b4700 1 -- 172.21.15.60:0/2211502433 shutdown_connections 2022-06-29T16:06:31.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.101+0000 7fcb5b1b4700 1 --2- 172.21.15.60:0/2211502433 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcb340539e0 0x7fcb34055ea0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:31.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.101+0000 7fcb5b1b4700 1 --2- 172.21.15.60:0/2211502433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb540b28e0 0x7fcb5413b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:31.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.101+0000 7fcb5b1b4700 1 --2- 172.21.15.60:0/2211502433 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb54065450 0x7fcb541437b0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:31.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.101+0000 7fcb5b1b4700 1 --2- 172.21.15.60:0/2211502433 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb54060760 0x7fcb540b6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:31.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.101+0000 7fcb5b1b4700 1 -- 172.21.15.60:0/2211502433 >> 172.21.15.60:0/2211502433 conn(0x7fcb54061c80 msgr2=0x7fcb5413c970 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:31.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.101+0000 7fcb5b1b4700 1 -- 172.21.15.60:0/2211502433 shutdown_connections 2022-06-29T16:06:31.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.101+0000 7fcb5b1b4700 1 -- 172.21.15.60:0/2211502433 wait complete. 2022-06-29T16:06:31.113 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 51 2022-06-29T16:06:31.324 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 51, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:06:28.249737+0000', 'modified': '2022-06-29T16:06:29.252773+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7668}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7668': {'gid': 7668, 'name': 'a', 'rank': 0, 'incarnation': 50, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.60:6833/2749128920', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6832', 'nonce': 2749128920}, {'type': 'v1', 'addr': '172.21.15.60:6833', 'nonce': 2749128920}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:06:31.325 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:06:31.325 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-06-29T16:06:31.441 INFO:teuthology.orchestra.run.smithi060.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":7668,"want_state":"up:active","state":"up:active","rank_uptime":3.1757009360000001,"mdsmap_epoch":51,"osdmap_epoch":110,"osdmap_epoch_barrier":109,"uptime":7.320391807} 2022-06-29T16:06:31.442 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 7668, "mdsmap_epoch": 51, "osdmap_epoch": 110, "osdmap_epoch_barrier": 109, "rank_uptime": 3.175700936, "state": "up:active", "uptime": 7.320391807, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:06:31.442 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:06:31.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.541+0000 7f3fa5137700 1 Processor -- start 2022-06-29T16:06:31.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.541+0000 7f3fa5137700 1 -- start start 2022-06-29T16:06:31.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.541+0000 7f3fa5137700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3fa0062230 0x7f3fa005fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:31.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.541+0000 7f3fa5137700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3fa0146cf0 0x7f3fa00aaf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:31.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.541+0000 7f3fa5137700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa00b6e30 con 0x7f3fa00b6a50 2022-06-29T16:06:31.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.541+0000 7f3fa5137700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fa0098970 con 0x7f3fa0062230 2022-06-29T16:06:31.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.541+0000 7f3fa5137700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fa005fa40 con 0x7f3fa0146cf0 2022-06-29T16:06:31.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.541+0000 7f3f9e59c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3fa0062230 0x7f3fa005fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:31.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.541+0000 7f3f9e59c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3fa0062230 0x7f3fa005fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52182/0 (socket says 172.21.15.60:52182) 2022-06-29T16:06:31.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.541+0000 7f3f9e59c700 1 -- 172.21.15.60:0/395439105 learned_addr learned my addr 172.21.15.60:0/395439105 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:31.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.541+0000 7f3f9f59e700 1 --2- 172.21.15.60:0/395439105 >> v2:172.21.15.92:3300/0 conn(0x7f3fa0146cf0 0x7f3fa00aaf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:31.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.541+0000 7f3f9e59c700 1 -- 172.21.15.60:0/395439105 >> v2:172.21.15.92:3300/0 conn(0x7f3fa0146cf0 msgr2=0x7f3fa00aaf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:31.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.541+0000 7f3f9e59c700 1 --2- 172.21.15.60:0/395439105 >> v2:172.21.15.92:3300/0 conn(0x7f3fa0146cf0 0x7f3fa00aaf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:31.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.541+0000 7f3f9e59c700 1 -- 172.21.15.60:0/395439105 >> v1:172.21.15.60:6789/0 conn(0x7f3fa00b6a50 legacy=0x7f3fa01465e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:31.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.541+0000 7f3f9e59c700 1 -- 172.21.15.60:0/395439105 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa005f0b0 con 0x7f3fa0062230 2022-06-29T16:06:31.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.541+0000 7f3f9f59e700 1 --2- 172.21.15.60:0/395439105 >> v2:172.21.15.92:3300/0 conn(0x7f3fa0146cf0 0x7f3fa00aaf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:06:31.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.541+0000 7f3f9e59c700 1 --2- 172.21.15.60:0/395439105 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3fa0062230 0x7f3fa005fee0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 rx=0x7f3f980032f0 tx=0x7f3f98009290).ready entity=mon.0 client_cookie=823c3121f28ca950 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:31.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.541+0000 7f3f9dd9b700 1 -- 172.21.15.60:0/395439105 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f9800b870 con 0x7f3fa0062230 2022-06-29T16:06:31.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.541+0000 7f3f9dd9b700 1 -- 172.21.15.60:0/395439105 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f9800b9d0 con 0x7f3fa0062230 2022-06-29T16:06:31.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.541+0000 7f3f9dd9b700 1 -- 172.21.15.60:0/395439105 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f9800bb40 con 0x7f3fa0062230 2022-06-29T16:06:31.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.541+0000 7f3fa5137700 1 -- 172.21.15.60:0/395439105 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3fa0062230 msgr2=0x7f3fa005fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:31.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.541+0000 7f3fa5137700 1 --2- 172.21.15.60:0/395439105 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3fa0062230 0x7f3fa005fee0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 rx=0x7f3f980032f0 tx=0x7f3f98009290).stop 2022-06-29T16:06:31.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.541+0000 7f3fa5137700 1 -- 172.21.15.60:0/395439105 shutdown_connections 2022-06-29T16:06:31.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.541+0000 7f3fa5137700 1 --2- 172.21.15.60:0/395439105 >> v2:172.21.15.92:3300/0 conn(0x7f3fa0146cf0 0x7f3fa00aaf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:31.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.541+0000 7f3fa5137700 1 --2- 172.21.15.60:0/395439105 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3fa0062230 0x7f3fa005fee0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:31.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.541+0000 7f3fa5137700 1 -- 172.21.15.60:0/395439105 >> 172.21.15.60:0/395439105 conn(0x7f3fa00643e0 msgr2=0x7f3fa00b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:31.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.541+0000 7f3fa5137700 1 -- 172.21.15.60:0/395439105 shutdown_connections 2022-06-29T16:06:31.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.541+0000 7f3fa5137700 1 -- 172.21.15.60:0/395439105 wait complete. 2022-06-29T16:06:31.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.545+0000 7f3fa5137700 1 Processor -- start 2022-06-29T16:06:31.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.545+0000 7f3fa5137700 1 -- start start 2022-06-29T16:06:31.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.545+0000 7f3fa5137700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3fa0062230 0x7f3fa009af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:31.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.545+0000 7f3fa5137700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3fa00b6a50 0x7f3fa0065800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:31.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.549+0000 7f3fa5137700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3fa0146cf0 0x7f3fa0065cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:31.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.549+0000 7f3fa5137700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa0099510 con 0x7f3fa0062230 2022-06-29T16:06:31.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.549+0000 7f3fa5137700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa009b750 con 0x7f3fa00b6a50 2022-06-29T16:06:31.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.549+0000 7f3fa5137700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa0099270 con 0x7f3fa0146cf0 2022-06-29T16:06:31.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.549+0000 7f3f9ed9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3fa0062230 0x7f3fa009af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:31.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.549+0000 7f3f9ed9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3fa0062230 0x7f3fa009af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52186/0 (socket says 172.21.15.60:52186) 2022-06-29T16:06:31.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.549+0000 7f3f9ed9d700 1 -- 172.21.15.60:0/2012812816 learned_addr learned my addr 172.21.15.60:0/2012812816 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:31.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.549+0000 7f3f9f59e700 1 --2- 172.21.15.60:0/2012812816 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3fa0146cf0 0x7f3fa0065cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:31.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.549+0000 7f3f9e59c700 1 --2- 172.21.15.60:0/2012812816 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3fa00b6a50 0x7f3fa0065800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:31.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.549+0000 7f3f9ed9d700 1 -- 172.21.15.60:0/2012812816 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3fa0146cf0 msgr2=0x7f3fa0065cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:31.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.549+0000 7f3f9ed9d700 1 --2- 172.21.15.60:0/2012812816 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3fa0146cf0 0x7f3fa0065cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:31.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.549+0000 7f3f9ed9d700 1 -- 172.21.15.60:0/2012812816 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3fa00b6a50 msgr2=0x7f3fa0065800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:31.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.549+0000 7f3f9ed9d700 1 --2- 172.21.15.60:0/2012812816 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3fa00b6a50 0x7f3fa0065800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:31.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.549+0000 7f3f9ed9d700 1 -- 172.21.15.60:0/2012812816 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f98009530 con 0x7f3fa0062230 2022-06-29T16:06:31.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.549+0000 7f3f9ed9d700 1 --2- 172.21.15.60:0/2012812816 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3fa0062230 0x7f3fa009af00 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 rx=0x7f3fa00b3d60 tx=0x7f3f88007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:31.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.549+0000 7f3f97fff700 1 -- 172.21.15.60:0/2012812816 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f8800ce30 con 0x7f3fa0062230 2022-06-29T16:06:31.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.549+0000 7f3f97fff700 1 -- 172.21.15.60:0/2012812816 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f88007240 con 0x7f3fa0062230 2022-06-29T16:06:31.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.549+0000 7f3fa5137700 1 -- 172.21.15.60:0/2012812816 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fa00997d0 con 0x7f3fa0062230 2022-06-29T16:06:31.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.549+0000 7f3f97fff700 1 -- 172.21.15.60:0/2012812816 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f88006700 con 0x7f3fa0062230 2022-06-29T16:06:31.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.549+0000 7f3fa5137700 1 -- 172.21.15.60:0/2012812816 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fa00661e0 con 0x7f3fa0062230 2022-06-29T16:06:31.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.549+0000 7f3fa5137700 1 -- 172.21.15.60:0/2012812816 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fa00573d0 con 0x7f3fa0062230 2022-06-29T16:06:31.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.549+0000 7f3f97fff700 1 -- 172.21.15.60:0/2012812816 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3f88010030 con 0x7f3fa0062230 2022-06-29T16:06:31.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.553+0000 7f3f97fff700 1 --2- 172.21.15.60:0/2012812816 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3f84053a10 0x7f3f84055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:31.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.553+0000 7f3f97fff700 1 -- 172.21.15.60:0/2012812816 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3f8806bb40 con 0x7f3fa0062230 2022-06-29T16:06:31.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.553+0000 7f3f9e59c700 1 --2- 172.21.15.60:0/2012812816 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3f84053a10 0x7f3f84055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:31.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.553+0000 7f3f9e59c700 1 --2- 172.21.15.60:0/2012812816 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3f84053a10 0x7f3f84055ed0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7f3f98019000 tx=0x7f3f98016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:31.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.553+0000 7f3f97fff700 1 -- 172.21.15.60:0/2012812816 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3f88042a60 con 0x7f3fa0062230 2022-06-29T16:06:31.680 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.677+0000 7f3fa5137700 1 -- 172.21.15.60:0/2012812816 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3fa0067990 con 0x7f3fa0062230 2022-06-29T16:06:31.682 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.677+0000 7f3f97fff700 1 -- 172.21.15.60:0/2012812816 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16051 (secure 0 0 0) 0x7f3f88046080 con 0x7f3fa0062230 2022-06-29T16:06:31.683 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:31.683 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":110,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:06:29.253422+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:06:26.864385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2022-06-29T16:06:27.251709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:06:31.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.677+0000 7f3fa5137700 1 -- 172.21.15.60:0/2012812816 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3f84053a10 msgr2=0x7f3f84055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:31.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.677+0000 7f3fa5137700 1 --2- 172.21.15.60:0/2012812816 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3f84053a10 0x7f3f84055ed0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7f3f98019000 tx=0x7f3f98016000).stop 2022-06-29T16:06:31.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.677+0000 7f3fa5137700 1 -- 172.21.15.60:0/2012812816 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3fa0062230 msgr2=0x7f3fa009af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:31.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.677+0000 7f3fa5137700 1 --2- 172.21.15.60:0/2012812816 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3fa0062230 0x7f3fa009af00 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 rx=0x7f3fa00b3d60 tx=0x7f3f88007c10).stop 2022-06-29T16:06:31.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.677+0000 7f3fa5137700 1 -- 172.21.15.60:0/2012812816 shutdown_connections 2022-06-29T16:06:31.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.681+0000 7f3fa5137700 1 --2- 172.21.15.60:0/2012812816 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3f84053a10 0x7f3f84055ed0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:31.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.681+0000 7f3fa5137700 1 --2- 172.21.15.60:0/2012812816 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3fa0146cf0 0x7f3fa0065cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:31.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.681+0000 7f3fa5137700 1 --2- 172.21.15.60:0/2012812816 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3fa00b6a50 0x7f3fa0065800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:31.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.681+0000 7f3fa5137700 1 --2- 172.21.15.60:0/2012812816 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3fa0062230 0x7f3fa009af00 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:31.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.681+0000 7f3fa5137700 1 -- 172.21.15.60:0/2012812816 >> 172.21.15.60:0/2012812816 conn(0x7f3fa00643e0 msgr2=0x7f3fa00b2210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:31.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.681+0000 7f3fa5137700 1 -- 172.21.15.60:0/2012812816 shutdown_connections 2022-06-29T16:06:31.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.681+0000 7f3fa5137700 1 -- 172.21.15.60:0/2012812816 wait complete. 2022-06-29T16:06:31.695 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-29T16:06:26.864385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2022-06-29T16:06:27.251709+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-29T16:06:31.696 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:06:31.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.797+0000 7f7815c1d700 1 Processor -- start 2022-06-29T16:06:31.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.797+0000 7f7815c1d700 1 -- start start 2022-06-29T16:06:31.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.801+0000 7f7815c1d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f78100b2800 0x7f78100b2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:31.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.801+0000 7f7815c1d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78100afff0 0x7f78100b0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:31.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.801+0000 7f7815c1d700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78100acf00 con 0x7f781005bf60 2022-06-29T16:06:31.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.801+0000 7f7815c1d700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78101437a0 con 0x7f78100b2800 2022-06-29T16:06:31.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.801+0000 7f7815c1d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7810143a40 con 0x7f78100afff0 2022-06-29T16:06:31.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.801+0000 7f7814c1b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f78100b2800 0x7f78100b2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:31.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.801+0000 7f7814c1b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f78100b2800 0x7f78100b2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52194/0 (socket says 172.21.15.60:52194) 2022-06-29T16:06:31.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.801+0000 7f7807fff700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f781005bf60 0x7f78100af240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:36298/0 (socket says 172.21.15.60:36298) 2022-06-29T16:06:31.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.801+0000 7f7814c1b700 1 -- 172.21.15.60:0/2617730301 learned_addr learned my addr 172.21.15.60:0/2617730301 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:31.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.801+0000 7f781541c700 1 --2- 172.21.15.60:0/2617730301 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78100afff0 0x7f78100b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:31.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.801+0000 7f7814c1b700 1 -- 172.21.15.60:0/2617730301 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78100afff0 msgr2=0x7f78100b0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:31.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.801+0000 7f7814c1b700 1 --2- 172.21.15.60:0/2617730301 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78100afff0 0x7f78100b0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:31.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.801+0000 7f7814c1b700 1 -- 172.21.15.60:0/2617730301 >> v1:172.21.15.92:6789/0 conn(0x7f781005bf60 legacy=0x7f78100af240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:31.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.801+0000 7f7814c1b700 1 -- 172.21.15.60:0/2617730301 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7810067420 con 0x7f78100b2800 2022-06-29T16:06:31.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.801+0000 7f78077fe700 1 -- 172.21.15.60:0/2617730301 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3165385708 0 0) 0x7f78100acf00 con 0x7f781005bf60 2022-06-29T16:06:31.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.801+0000 7f7814c1b700 1 --2- 172.21.15.60:0/2617730301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78100b2800 0x7f78100b2be0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 rx=0x7f780c002f70 tx=0x7f780c006eb0).ready entity=mon.0 client_cookie=e1deded8d678ede7 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:31.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.801+0000 7f78077fe700 1 -- 172.21.15.60:0/2617730301 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f780c00d570 con 0x7f78100b2800 2022-06-29T16:06:31.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.801+0000 7f78077fe700 1 -- 172.21.15.60:0/2617730301 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f780c000e90 con 0x7f78100b2800 2022-06-29T16:06:31.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.801+0000 7f78077fe700 1 -- 172.21.15.60:0/2617730301 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f780c00d8c0 con 0x7f78100b2800 2022-06-29T16:06:31.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.805+0000 7f7815c1d700 1 -- 172.21.15.60:0/2617730301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78100b2800 msgr2=0x7f78100b2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:31.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.805+0000 7f7815c1d700 1 --2- 172.21.15.60:0/2617730301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78100b2800 0x7f78100b2be0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 rx=0x7f780c002f70 tx=0x7f780c006eb0).stop 2022-06-29T16:06:31.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.805+0000 7f7815c1d700 1 -- 172.21.15.60:0/2617730301 shutdown_connections 2022-06-29T16:06:31.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.805+0000 7f7815c1d700 1 --2- 172.21.15.60:0/2617730301 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78100afff0 0x7f78100b0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:31.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.805+0000 7f7815c1d700 1 --2- 172.21.15.60:0/2617730301 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78100b2800 0x7f78100b2be0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:31.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.805+0000 7f7815c1d700 1 -- 172.21.15.60:0/2617730301 >> 172.21.15.60:0/2617730301 conn(0x7f78100b6120 msgr2=0x7f78100b20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:31.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.805+0000 7f7815c1d700 1 -- 172.21.15.60:0/2617730301 shutdown_connections 2022-06-29T16:06:31.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.805+0000 7f7815c1d700 1 -- 172.21.15.60:0/2617730301 wait complete. 2022-06-29T16:06:31.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.805+0000 7f7815c1d700 1 Processor -- start 2022-06-29T16:06:31.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.805+0000 7f7815c1d700 1 -- start start 2022-06-29T16:06:31.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.809+0000 7f7815c1d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f781005bf60 0x7f7810152270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:31.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.809+0000 7f7815c1d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78100afff0 0x7f781009e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:31.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.809+0000 7f7815c1d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78100b2800 0x7f781009ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:31.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.809+0000 7f7815c1d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f781009f370 con 0x7f78100b2800 2022-06-29T16:06:31.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.809+0000 7f7815c1d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7810147db0 con 0x7f78100afff0 2022-06-29T16:06:31.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.809+0000 7f7815c1d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7810148480 con 0x7f781005bf60 2022-06-29T16:06:31.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.809+0000 7f781541c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78100b2800 0x7f781009ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:31.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.809+0000 7f781541c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78100b2800 0x7f781009ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52204/0 (socket says 172.21.15.60:52204) 2022-06-29T16:06:31.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.809+0000 7f781541c700 1 -- 172.21.15.60:0/2089876063 learned_addr learned my addr 172.21.15.60:0/2089876063 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:31.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.809+0000 7f7807fff700 1 --2- 172.21.15.60:0/2089876063 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78100afff0 0x7f781009e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:31.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.809+0000 7f7814c1b700 1 --2- 172.21.15.60:0/2089876063 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f781005bf60 0x7f7810152270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:31.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.809+0000 7f781541c700 1 -- 172.21.15.60:0/2089876063 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f781005bf60 msgr2=0x7f7810152270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:31.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.809+0000 7f781541c700 1 --2- 172.21.15.60:0/2089876063 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f781005bf60 0x7f7810152270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:31.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.809+0000 7f781541c700 1 -- 172.21.15.60:0/2089876063 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78100afff0 msgr2=0x7f781009e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:31.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.809+0000 7f781541c700 1 --2- 172.21.15.60:0/2089876063 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78100afff0 0x7f781009e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:31.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.809+0000 7f781541c700 1 -- 172.21.15.60:0/2089876063 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f780c00b1e0 con 0x7f78100b2800 2022-06-29T16:06:31.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.809+0000 7f781541c700 1 --2- 172.21.15.60:0/2089876063 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78100b2800 0x7f781009ee80 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 rx=0x7f7808006e00 tx=0x7f78080091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:31.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.809+0000 7f7805ffb700 1 -- 172.21.15.60:0/2089876063 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7808012600 con 0x7f78100b2800 2022-06-29T16:06:31.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.809+0000 7f7805ffb700 1 -- 172.21.15.60:0/2089876063 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7808012760 con 0x7f78100b2800 2022-06-29T16:06:31.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.809+0000 7f7815c1d700 1 -- 172.21.15.60:0/2089876063 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7810152cc0 con 0x7f78100b2800 2022-06-29T16:06:31.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.809+0000 7f7805ffb700 1 -- 172.21.15.60:0/2089876063 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7808012ac0 con 0x7f78100b2800 2022-06-29T16:06:31.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.809+0000 7f7815c1d700 1 -- 172.21.15.60:0/2089876063 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78100abe30 con 0x7f78100b2800 2022-06-29T16:06:31.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.809+0000 7f7815c1d700 1 -- 172.21.15.60:0/2089876063 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78100573d0 con 0x7f78100b2800 2022-06-29T16:06:31.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.809+0000 7f7805ffb700 1 -- 172.21.15.60:0/2089876063 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7808022070 con 0x7f78100b2800 2022-06-29T16:06:31.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.813+0000 7f7805ffb700 1 --2- 172.21.15.60:0/2089876063 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f77f0053aa0 0x7f77f0055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:31.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.813+0000 7f7805ffb700 1 -- 172.21.15.60:0/2089876063 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f780806c590 con 0x7f78100b2800 2022-06-29T16:06:31.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.813+0000 7f7814c1b700 1 --2- 172.21.15.60:0/2089876063 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f77f0053aa0 0x7f77f0055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:31.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.813+0000 7f7814c1b700 1 --2- 172.21.15.60:0/2089876063 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f77f0053aa0 0x7f77f0055f60 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7f780c01c000 tx=0x7f780c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:31.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.817+0000 7f7805ffb700 1 -- 172.21.15.60:0/2089876063 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f78080433d0 con 0x7f78100b2800 2022-06-29T16:06:31.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.945+0000 7f7815c1d700 1 -- 172.21.15.60:0/2089876063 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f78100ac370 con 0x7f78100b2800 2022-06-29T16:06:31.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.945+0000 7f7805ffb700 1 -- 172.21.15.60:0/2089876063 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v110) v1 ==== 110+0+10 (secure 0 0 0) 0x7f7808016020 con 0x7f78100b2800 2022-06-29T16:06:31.951 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:06:31.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.949+0000 7f7815c1d700 1 -- 172.21.15.60:0/2089876063 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f77f0053aa0 msgr2=0x7f77f0055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:31.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.949+0000 7f7815c1d700 1 --2- 172.21.15.60:0/2089876063 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f77f0053aa0 0x7f77f0055f60 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7f780c01c000 tx=0x7f780c015000).stop 2022-06-29T16:06:31.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.949+0000 7f7815c1d700 1 -- 172.21.15.60:0/2089876063 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78100b2800 msgr2=0x7f781009ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:31.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.949+0000 7f7815c1d700 1 --2- 172.21.15.60:0/2089876063 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78100b2800 0x7f781009ee80 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 rx=0x7f7808006e00 tx=0x7f78080091d0).stop 2022-06-29T16:06:31.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.949+0000 7f7815c1d700 1 -- 172.21.15.60:0/2089876063 shutdown_connections 2022-06-29T16:06:31.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.949+0000 7f7815c1d700 1 --2- 172.21.15.60:0/2089876063 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f77f0053aa0 0x7f77f0055f60 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:31.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.949+0000 7f7815c1d700 1 --2- 172.21.15.60:0/2089876063 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78100b2800 0x7f781009ee80 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:31.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.949+0000 7f7815c1d700 1 --2- 172.21.15.60:0/2089876063 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78100afff0 0x7f781009e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:31.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.949+0000 7f7815c1d700 1 --2- 172.21.15.60:0/2089876063 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f781005bf60 0x7f7810152270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:31.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.949+0000 7f7815c1d700 1 -- 172.21.15.60:0/2089876063 >> 172.21.15.60:0/2089876063 conn(0x7f78100b6120 msgr2=0x7f78100b3550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:31.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.949+0000 7f7815c1d700 1 -- 172.21.15.60:0/2089876063 shutdown_connections 2022-06-29T16:06:31.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:31.949+0000 7f7815c1d700 1 -- 172.21.15.60:0/2089876063 wait complete. 2022-06-29T16:06:31.964 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:06:32.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.073+0000 7fd6424f0700 1 Processor -- start 2022-06-29T16:06:32.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.073+0000 7fd6424f0700 1 -- start start 2022-06-29T16:06:32.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.073+0000 7fd6424f0700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd63c05fb00 0x7fd63c143430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:32.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.073+0000 7fd6424f0700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd63c13fb30 0x7fd63c143920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:32.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.073+0000 7fd6424f0700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd63c142fd0 con 0x7fd63c062aa0 2022-06-29T16:06:32.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.073+0000 7fd640ced700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd63c062aa0 0x7fd63c0aca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60296/0 (socket says 172.21.15.60:60296) 2022-06-29T16:06:32.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.073+0000 7fd6424f0700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd63c067d70 con 0x7fd63c05fb00 2022-06-29T16:06:32.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.073+0000 7fd640ced700 1 -- 172.21.15.60:0/1399538954 learned_addr learned my addr 172.21.15.60:0/1399538954 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:32.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.073+0000 7fd6424f0700 1 -- 172.21.15.60:0/1399538954 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c05fee0 con 0x7fd63c13fb30 2022-06-29T16:06:32.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd6414ee700 1 --2- 172.21.15.60:0/1399538954 >> v2:172.21.15.92:3300/0 conn(0x7fd63c05fb00 0x7fd63c143430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:32.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd641cef700 1 --2- 172.21.15.60:0/1399538954 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd63c13fb30 0x7fd63c143920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:32.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd62bfff700 1 -- 172.21.15.60:0/1399538954 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 171470640 0 0) 0x7fd63c142fd0 con 0x7fd63c062aa0 2022-06-29T16:06:32.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd62bfff700 1 -- 172.21.15.60:0/1399538954 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd624003260 con 0x7fd63c062aa0 2022-06-29T16:06:32.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd62bfff700 1 -- 172.21.15.60:0/1399538954 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 15773256 0 0) 0x7fd624003260 con 0x7fd63c062aa0 2022-06-29T16:06:32.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd62bfff700 1 -- 172.21.15.60:0/1399538954 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd63c13fb30 msgr2=0x7fd63c143920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:32.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd62bfff700 1 --2- 172.21.15.60:0/1399538954 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd63c13fb30 0x7fd63c143920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:32.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd62bfff700 1 -- 172.21.15.60:0/1399538954 >> v2:172.21.15.92:3300/0 conn(0x7fd63c05fb00 msgr2=0x7fd63c143430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:32.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd62bfff700 1 --2- 172.21.15.60:0/1399538954 >> v2:172.21.15.92:3300/0 conn(0x7fd63c05fb00 0x7fd63c143430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:32.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd62bfff700 1 -- 172.21.15.60:0/1399538954 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd63c143e10 con 0x7fd63c062aa0 2022-06-29T16:06:32.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd62bfff700 1 -- 172.21.15.60:0/1399538954 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7fd62c001660 con 0x7fd63c062aa0 2022-06-29T16:06:32.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd62bfff700 1 -- 172.21.15.60:0/1399538954 >> v1:172.21.15.60:6789/0 conn(0x7fd63c062aa0 legacy=0x7fd63c0aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:32.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd62bfff700 1 --2- 172.21.15.60:0/1399538954 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6240037d0 0x7fd624003c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:32.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd62bfff700 1 --2- 172.21.15.60:0/1399538954 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd63c13fb30 0x7fd624004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:32.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd62bfff700 1 --2- 172.21.15.60:0/1399538954 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6240049f0 0x7fd624004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:32.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd62bfff700 1 -- 172.21.15.60:0/1399538954 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c067d70 con 0x7fd6240049f0 2022-06-29T16:06:32.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd62bfff700 1 -- 172.21.15.60:0/1399538954 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c05fee0 con 0x7fd6240037d0 2022-06-29T16:06:32.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd62bfff700 1 -- 172.21.15.60:0/1399538954 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd62400b780 con 0x7fd63c13fb30 2022-06-29T16:06:32.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd641cef700 1 --2- 172.21.15.60:0/1399538954 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6240049f0 0x7fd624004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:32.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd640ced700 1 --2- 172.21.15.60:0/1399538954 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd63c13fb30 0x7fd624004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:32.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd6414ee700 1 --2- 172.21.15.60:0/1399538954 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6240037d0 0x7fd624003c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:32.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd640ced700 1 -- 172.21.15.60:0/1399538954 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6240037d0 msgr2=0x7fd624003c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:32.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd640ced700 1 --2- 172.21.15.60:0/1399538954 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6240037d0 0x7fd624003c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:32.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd640ced700 1 -- 172.21.15.60:0/1399538954 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6240049f0 msgr2=0x7fd624004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:32.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd640ced700 1 --2- 172.21.15.60:0/1399538954 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6240049f0 0x7fd624004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:32.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd640ced700 1 -- 172.21.15.60:0/1399538954 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd624003260 con 0x7fd63c13fb30 2022-06-29T16:06:32.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd640ced700 1 --2- 172.21.15.60:0/1399538954 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd63c13fb30 0x7fd624004500 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fd62c001370 tx=0x7fd62c001d80).ready entity=mon.2 client_cookie=440b2578df66cab9 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:32.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd62bfff700 1 -- 172.21.15.60:0/1399538954 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd62c0070d0 con 0x7fd63c13fb30 2022-06-29T16:06:32.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd62bfff700 1 -- 172.21.15.60:0/1399538954 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd62c007230 con 0x7fd63c13fb30 2022-06-29T16:06:32.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd62bfff700 1 -- 172.21.15.60:0/1399538954 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd62c00c8f0 con 0x7fd63c13fb30 2022-06-29T16:06:32.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd6424f0700 1 -- 172.21.15.60:0/1399538954 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd63c13fb30 msgr2=0x7fd624004500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:32.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd6424f0700 1 --2- 172.21.15.60:0/1399538954 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd63c13fb30 0x7fd624004500 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fd62c001370 tx=0x7fd62c001d80).stop 2022-06-29T16:06:32.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd641cef700 1 -- 172.21.15.60:0/1399538954 reap_dead start 2022-06-29T16:06:32.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd6424f0700 1 -- 172.21.15.60:0/1399538954 shutdown_connections 2022-06-29T16:06:32.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd6424f0700 1 -- 172.21.15.60:0/1399538954 >> 172.21.15.60:0/1399538954 conn(0x7fd63c063a60 msgr2=0x7fd63c05f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:32.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd6424f0700 1 -- 172.21.15.60:0/1399538954 shutdown_connections 2022-06-29T16:06:32.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.077+0000 7fd6424f0700 1 -- 172.21.15.60:0/1399538954 wait complete. 2022-06-29T16:06:32.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.081+0000 7fd6424f0700 1 Processor -- start 2022-06-29T16:06:32.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.081+0000 7fd6424f0700 1 -- start start 2022-06-29T16:06:32.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.081+0000 7fd6424f0700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd63c13fb30 0x7fd63c0b0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:32.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.081+0000 7fd6424f0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd63c142140 0x7fd63c1425c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:32.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.081+0000 7fd6424f0700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd63c0b20d0 0x7fd63c0b2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:32.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.081+0000 7fd6424f0700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c13aa30 con 0x7fd63c142140 2022-06-29T16:06:32.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.081+0000 7fd6424f0700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c139c60 con 0x7fd63c13fb30 2022-06-29T16:06:32.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.081+0000 7fd6424f0700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c13ac20 con 0x7fd63c0b20d0 2022-06-29T16:06:32.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.081+0000 7fd640ced700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd63c142140 0x7fd63c1425c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:32.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.081+0000 7fd640ced700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd63c142140 0x7fd63c1425c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52222/0 (socket says 172.21.15.60:52222) 2022-06-29T16:06:32.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.081+0000 7fd640ced700 1 -- 172.21.15.60:0/1515300048 learned_addr learned my addr 172.21.15.60:0/1515300048 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:32.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.081+0000 7fd641cef700 1 --2- 172.21.15.60:0/1515300048 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd63c0b20d0 0x7fd63c0b2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:32.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.081+0000 7fd6414ee700 1 --2- 172.21.15.60:0/1515300048 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd63c13fb30 0x7fd63c0b0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:32.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.081+0000 7fd640ced700 1 -- 172.21.15.60:0/1515300048 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd63c0b20d0 msgr2=0x7fd63c0b2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:32.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.081+0000 7fd640ced700 1 --2- 172.21.15.60:0/1515300048 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd63c0b20d0 0x7fd63c0b2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:32.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.081+0000 7fd640ced700 1 -- 172.21.15.60:0/1515300048 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd63c13fb30 msgr2=0x7fd63c0b0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:32.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.081+0000 7fd640ced700 1 --2- 172.21.15.60:0/1515300048 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd63c13fb30 0x7fd63c0b0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:32.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.081+0000 7fd640ced700 1 -- 172.21.15.60:0/1515300048 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd63c13b2e0 con 0x7fd63c142140 2022-06-29T16:06:32.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.081+0000 7fd640ced700 1 --2- 172.21.15.60:0/1515300048 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd63c142140 0x7fd63c1425c0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 rx=0x7fd63c141fe0 tx=0x7fd62c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:32.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.085+0000 7fd62a7fc700 1 -- 172.21.15.60:0/1515300048 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd62c007720 con 0x7fd63c142140 2022-06-29T16:06:32.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.085+0000 7fd62a7fc700 1 -- 172.21.15.60:0/1515300048 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd62c00c6d0 con 0x7fd63c142140 2022-06-29T16:06:32.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.085+0000 7fd62a7fc700 1 -- 172.21.15.60:0/1515300048 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd62c004710 con 0x7fd63c142140 2022-06-29T16:06:32.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.085+0000 7fd6424f0700 1 -- 172.21.15.60:0/1515300048 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd63c139ee0 con 0x7fd63c142140 2022-06-29T16:06:32.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.085+0000 7fd6424f0700 1 -- 172.21.15.60:0/1515300048 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd63c142ab0 con 0x7fd63c142140 2022-06-29T16:06:32.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.085+0000 7fd6424f0700 1 -- 172.21.15.60:0/1515300048 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd63c13b470 con 0x7fd63c142140 2022-06-29T16:06:32.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.085+0000 7fd62a7fc700 1 -- 172.21.15.60:0/1515300048 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd62c010070 con 0x7fd63c142140 2022-06-29T16:06:32.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.085+0000 7fd62a7fc700 1 --2- 172.21.15.60:0/1515300048 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd61c053b00 0x7fd61c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:32.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.085+0000 7fd62a7fc700 1 -- 172.21.15.60:0/1515300048 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd62c06c820 con 0x7fd63c142140 2022-06-29T16:06:32.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.085+0000 7fd6414ee700 1 --2- 172.21.15.60:0/1515300048 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd61c053b00 0x7fd61c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:32.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.089+0000 7fd6414ee700 1 --2- 172.21.15.60:0/1515300048 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd61c053b00 0x7fd61c055fc0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0x7fd63800b000 tx=0x7fd638006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:32.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.093+0000 7fd62a7fc700 1 -- 172.21.15.60:0/1515300048 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd62c0436a0 con 0x7fd63c142140 2022-06-29T16:06:32.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.213+0000 7fd6424f0700 1 -- 172.21.15.60:0/1515300048 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd63c0573d0 con 0x7fd63c142140 2022-06-29T16:06:32.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.213+0000 7fd62a7fc700 1 -- 172.21.15.60:0/1515300048 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v110) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd62c024070 con 0x7fd63c142140 2022-06-29T16:06:32.219 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:06:32.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.213+0000 7fd6424f0700 1 -- 172.21.15.60:0/1515300048 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd61c053b00 msgr2=0x7fd61c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:32.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.213+0000 7fd6424f0700 1 --2- 172.21.15.60:0/1515300048 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd61c053b00 0x7fd61c055fc0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0x7fd63800b000 tx=0x7fd638006df0).stop 2022-06-29T16:06:32.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.213+0000 7fd6424f0700 1 -- 172.21.15.60:0/1515300048 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd63c142140 msgr2=0x7fd63c1425c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:32.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.213+0000 7fd6424f0700 1 --2- 172.21.15.60:0/1515300048 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd63c142140 0x7fd63c1425c0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 rx=0x7fd63c141fe0 tx=0x7fd62c015000).stop 2022-06-29T16:06:32.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.213+0000 7fd6424f0700 1 -- 172.21.15.60:0/1515300048 shutdown_connections 2022-06-29T16:06:32.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.217+0000 7fd6424f0700 1 --2- 172.21.15.60:0/1515300048 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd61c053b00 0x7fd61c055fc0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:32.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.217+0000 7fd6424f0700 1 --2- 172.21.15.60:0/1515300048 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd63c0b20d0 0x7fd63c0b2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:32.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.217+0000 7fd6424f0700 1 --2- 172.21.15.60:0/1515300048 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd63c142140 0x7fd63c1425c0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:32.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.217+0000 7fd6424f0700 1 --2- 172.21.15.60:0/1515300048 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd63c13fb30 0x7fd63c0b0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:32.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.217+0000 7fd6424f0700 1 -- 172.21.15.60:0/1515300048 >> 172.21.15.60:0/1515300048 conn(0x7fd63c063a60 msgr2=0x7fd63c13ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:32.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.217+0000 7fd6424f0700 1 -- 172.21.15.60:0/1515300048 shutdown_connections 2022-06-29T16:06:32.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.217+0000 7fd6424f0700 1 -- 172.21.15.60:0/1515300048 wait complete. 2022-06-29T16:06:32.232 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-29T16:06:32.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.337+0000 7f9735a96700 1 Processor -- start 2022-06-29T16:06:32.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.337+0000 7f9735a96700 1 -- start start 2022-06-29T16:06:32.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.337+0000 7f9735a96700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f973005d070 0x7f973013c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:32.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.337+0000 7f9735a96700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f973005d450 con 0x7f97300b0830 2022-06-29T16:06:32.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.337+0000 7f9735a96700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9730063ba0 con 0x7f97300b2130 2022-06-29T16:06:32.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.337+0000 7f9735a96700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9730063e40 con 0x7f973005d070 2022-06-29T16:06:32.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.341+0000 7f9734a94700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f973005d070 0x7f973013c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:32.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.341+0000 7f9735295700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f97300b0830 0x7f973013fa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60318/0 (socket says 172.21.15.60:60318) 2022-06-29T16:06:32.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.341+0000 7f9735295700 1 -- 172.21.15.60:0/487368746 learned_addr learned my addr 172.21.15.60:0/487368746 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:32.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.341+0000 7f972f7fe700 1 -- 172.21.15.60:0/487368746 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4141524072 0 0) 0x7f973005d450 con 0x7f97300b0830 2022-06-29T16:06:32.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.341+0000 7f972f7fe700 1 -- 172.21.15.60:0/487368746 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f971c003260 con 0x7f97300b0830 2022-06-29T16:06:32.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.341+0000 7f9734a94700 1 -- 172.21.15.60:0/487368746 >> v1:172.21.15.92:6789/0 conn(0x7f97300b2130 legacy=0x7f973013caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:32.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.341+0000 7f972f7fe700 1 -- 172.21.15.60:0/487368746 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4293313662 0 0) 0x7f9730063ba0 con 0x7f97300b2130 2022-06-29T16:06:32.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.341+0000 7f9734a94700 1 -- 172.21.15.60:0/487368746 >> v1:172.21.15.60:6789/0 conn(0x7f97300b0830 legacy=0x7f973013fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:32.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.341+0000 7f9734a94700 1 -- 172.21.15.60:0/487368746 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9730067650 con 0x7f973005d070 2022-06-29T16:06:32.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.341+0000 7f9734a94700 1 --2- 172.21.15.60:0/487368746 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f973005d070 0x7f973013c5b0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 rx=0x7f9728001b80 tx=0x7f9724006df0).ready entity=mon.0 client_cookie=aa4b41583c33a74e server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:32.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.341+0000 7f972f7fe700 1 -- 172.21.15.60:0/487368746 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f97280019e0 con 0x7f973005d070 2022-06-29T16:06:32.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.341+0000 7f972f7fe700 1 -- 172.21.15.60:0/487368746 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9724000e90 con 0x7f973005d070 2022-06-29T16:06:32.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.341+0000 7f972f7fe700 1 -- 172.21.15.60:0/487368746 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9724011400 con 0x7f973005d070 2022-06-29T16:06:32.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.341+0000 7f9735a96700 1 -- 172.21.15.60:0/487368746 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f973005d070 msgr2=0x7f973013c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:32.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.341+0000 7f9735a96700 1 --2- 172.21.15.60:0/487368746 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f973005d070 0x7f973013c5b0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 rx=0x7f9728001b80 tx=0x7f9724006df0).stop 2022-06-29T16:06:32.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.341+0000 7f9735a96700 1 -- 172.21.15.60:0/487368746 shutdown_connections 2022-06-29T16:06:32.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.341+0000 7f9735a96700 1 --2- 172.21.15.60:0/487368746 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f973005d070 0x7f973013c5b0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:32.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.341+0000 7f9735a96700 1 -- 172.21.15.60:0/487368746 >> 172.21.15.60:0/487368746 conn(0x7f97300b5890 msgr2=0x7f97300b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:32.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.341+0000 7f9735a96700 1 -- 172.21.15.60:0/487368746 shutdown_connections 2022-06-29T16:06:32.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.341+0000 7f9735a96700 1 -- 172.21.15.60:0/487368746 wait complete. 2022-06-29T16:06:32.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.345+0000 7f9735a96700 1 Processor -- start 2022-06-29T16:06:32.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.345+0000 7f9735a96700 1 -- start start 2022-06-29T16:06:32.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.345+0000 7f9735a96700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f973005d070 0x7f97301368e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:32.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.345+0000 7f9735a96700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97300b0830 0x7f9730064cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:32.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.345+0000 7f9735a96700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97300b2130 0x7f97300651e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:32.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.345+0000 7f9735a96700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9730067100 con 0x7f97300b2130 2022-06-29T16:06:32.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.345+0000 7f9735a96700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97300673a0 con 0x7f97300b0830 2022-06-29T16:06:32.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.345+0000 7f9735a96700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9730137330 con 0x7f973005d070 2022-06-29T16:06:32.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.345+0000 7f9735295700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97300b2130 0x7f97300651e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:32.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.345+0000 7f9735295700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97300b2130 0x7f97300651e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52238/0 (socket says 172.21.15.60:52238) 2022-06-29T16:06:32.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.345+0000 7f9735295700 1 -- 172.21.15.60:0/4184826899 learned_addr learned my addr 172.21.15.60:0/4184826899 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:32.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.345+0000 7f9734a94700 1 --2- 172.21.15.60:0/4184826899 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f973005d070 0x7f97301368e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:32.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.345+0000 7f972ffff700 1 --2- 172.21.15.60:0/4184826899 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97300b0830 0x7f9730064cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:32.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.345+0000 7f9735295700 1 -- 172.21.15.60:0/4184826899 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f973005d070 msgr2=0x7f97301368e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:32.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.345+0000 7f9735295700 1 --2- 172.21.15.60:0/4184826899 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f973005d070 0x7f97301368e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:32.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.345+0000 7f9735295700 1 -- 172.21.15.60:0/4184826899 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97300b0830 msgr2=0x7f9730064cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:32.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.345+0000 7f9735295700 1 --2- 172.21.15.60:0/4184826899 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97300b0830 0x7f9730064cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:32.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.345+0000 7f9735295700 1 -- 172.21.15.60:0/4184826899 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f972400b180 con 0x7f97300b2130 2022-06-29T16:06:32.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.345+0000 7f9735295700 1 --2- 172.21.15.60:0/4184826899 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97300b2130 0x7f97300651e0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 rx=0x7f9728001bc0 tx=0x7f9728001d30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:32.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.345+0000 7f972dffb700 1 -- 172.21.15.60:0/4184826899 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9728004ab0 con 0x7f97300b2130 2022-06-29T16:06:32.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.345+0000 7f972dffb700 1 -- 172.21.15.60:0/4184826899 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9728004c10 con 0x7f97300b2130 2022-06-29T16:06:32.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.345+0000 7f972dffb700 1 -- 172.21.15.60:0/4184826899 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9728007170 con 0x7f97300b2130 2022-06-29T16:06:32.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.345+0000 7f9735a96700 1 -- 172.21.15.60:0/4184826899 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9730136dd0 con 0x7f97300b2130 2022-06-29T16:06:32.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.345+0000 7f9735a96700 1 -- 172.21.15.60:0/4184826899 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97300ad5d0 con 0x7f97300b2130 2022-06-29T16:06:32.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.349+0000 7f9735a96700 1 -- 172.21.15.60:0/4184826899 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97300573d0 con 0x7f97300b2130 2022-06-29T16:06:32.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.349+0000 7f972dffb700 1 -- 172.21.15.60:0/4184826899 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9728010070 con 0x7f97300b2130 2022-06-29T16:06:32.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.349+0000 7f972dffb700 1 --2- 172.21.15.60:0/4184826899 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9710053a70 0x7f9710055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:32.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.349+0000 7f972dffb700 1 -- 172.21.15.60:0/4184826899 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f972806bb80 con 0x7f97300b2130 2022-06-29T16:06:32.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.349+0000 7f9734a94700 1 --2- 172.21.15.60:0/4184826899 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9710053a70 0x7f9710055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:32.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.349+0000 7f9734a94700 1 --2- 172.21.15.60:0/4184826899 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9710053a70 0x7f9710055f30 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0x7f9724019000 tx=0x7f9724006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:32.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.357+0000 7f972dffb700 1 -- 172.21.15.60:0/4184826899 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9728042a00 con 0x7f97300b2130 2022-06-29T16:06:32.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.477+0000 7f9735a96700 1 -- 172.21.15.60:0/4184826899 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9730136fc0 con 0x7f97300b2130 2022-06-29T16:06:32.586 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.477+0000 7f972dffb700 1 -- 172.21.15.60:0/4184826899 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v110) v1 ==== 104+0+10 (secure 0 0 0) 0x7f9728019070 con 0x7f97300b2130 2022-06-29T16:06:32.586 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:06:32.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.481+0000 7f9735a96700 1 -- 172.21.15.60:0/4184826899 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9710053a70 msgr2=0x7f9710055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:32.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.481+0000 7f9735a96700 1 --2- 172.21.15.60:0/4184826899 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9710053a70 0x7f9710055f30 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0x7f9724019000 tx=0x7f9724006df0).stop 2022-06-29T16:06:32.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.481+0000 7f9735a96700 1 -- 172.21.15.60:0/4184826899 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97300b2130 msgr2=0x7f97300651e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:32.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.481+0000 7f9735a96700 1 --2- 172.21.15.60:0/4184826899 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97300b2130 0x7f97300651e0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 rx=0x7f9728001bc0 tx=0x7f9728001d30).stop 2022-06-29T16:06:32.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.481+0000 7f9735a96700 1 -- 172.21.15.60:0/4184826899 shutdown_connections 2022-06-29T16:06:32.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.481+0000 7f9735a96700 1 --2- 172.21.15.60:0/4184826899 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9710053a70 0x7f9710055f30 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:32.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.481+0000 7f9735a96700 1 --2- 172.21.15.60:0/4184826899 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97300b2130 0x7f97300651e0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:32.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.481+0000 7f9735a96700 1 --2- 172.21.15.60:0/4184826899 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97300b0830 0x7f9730064cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:32.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.481+0000 7f9735a96700 1 --2- 172.21.15.60:0/4184826899 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f973005d070 0x7f97301368e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:32.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.481+0000 7f9735a96700 1 -- 172.21.15.60:0/4184826899 >> 172.21.15.60:0/4184826899 conn(0x7f97300b5890 msgr2=0x7f9730140aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:32.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.481+0000 7f9735a96700 1 -- 172.21.15.60:0/4184826899 shutdown_connections 2022-06-29T16:06:32.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.481+0000 7f9735a96700 1 -- 172.21.15.60:0/4184826899 wait complete. 2022-06-29T16:06:32.591 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-29T16:06:32.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.705+0000 7f246359e700 1 Processor -- start 2022-06-29T16:06:32.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.705+0000 7f246359e700 1 -- start start 2022-06-29T16:06:32.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.705+0000 7f246359e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f24640b5eb0 0x7f2464142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:32.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.705+0000 7f246359e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f24640b2c50 0x7f24640b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:32.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.705+0000 7f246359e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24640b0cf0 con 0x7f24640b48b0 2022-06-29T16:06:32.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.705+0000 7f246359e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24640b54e0 con 0x7f24640b5eb0 2022-06-29T16:06:32.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.705+0000 7f246359e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24640b4d10 con 0x7f24640b2c50 2022-06-29T16:06:32.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.705+0000 7f2461d9b700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f24640b48b0 0x7f24640b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60328/0 (socket says 172.21.15.60:60328) 2022-06-29T16:06:32.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.705+0000 7f2461d9b700 1 -- 172.21.15.60:0/2626834768 learned_addr learned my addr 172.21.15.60:0/2626834768 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:32.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.709+0000 7f2462d9d700 1 --2- 172.21.15.60:0/2626834768 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f24640b2c50 0x7f24640b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:32.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.709+0000 7f246259c700 1 --2- 172.21.15.60:0/2626834768 >> v2:172.21.15.92:3300/0 conn(0x7f24640b5eb0 0x7f2464142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:32.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.709+0000 7f246159a700 1 -- 172.21.15.60:0/2626834768 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 586381320 0 0) 0x7f2454001110 con 0x7f24640b48b0 2022-06-29T16:06:32.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.709+0000 7f246159a700 1 -- 172.21.15.60:0/2626834768 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2444003260 con 0x7f24640b48b0 2022-06-29T16:06:32.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.709+0000 7f246259c700 1 -- 172.21.15.60:0/2626834768 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f24640b2c50 msgr2=0x7f24640b30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:32.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.709+0000 7f246259c700 1 --2- 172.21.15.60:0/2626834768 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f24640b2c50 0x7f24640b30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:32.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.709+0000 7f246259c700 1 -- 172.21.15.60:0/2626834768 >> v1:172.21.15.60:6789/0 conn(0x7f24640b48b0 legacy=0x7f24640b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:32.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.709+0000 7f246259c700 1 -- 172.21.15.60:0/2626834768 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24640b6290 con 0x7f24640b5eb0 2022-06-29T16:06:32.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.709+0000 7f2462d9d700 1 --2- 172.21.15.60:0/2626834768 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f24640b2c50 0x7f24640b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:06:32.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.709+0000 7f246259c700 1 --2- 172.21.15.60:0/2626834768 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24640b5eb0 0x7f2464142a20 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7f244c002f70 tx=0x7f244c006df0).ready entity=mon.1 client_cookie=cf7ddc6c6230a230 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:32.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.709+0000 7f246159a700 1 -- 172.21.15.60:0/2626834768 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f244c00d4a0 con 0x7f24640b5eb0 2022-06-29T16:06:32.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.709+0000 7f246159a700 1 -- 172.21.15.60:0/2626834768 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f244c000e90 con 0x7f24640b5eb0 2022-06-29T16:06:32.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.709+0000 7f246159a700 1 -- 172.21.15.60:0/2626834768 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f244c00d7f0 con 0x7f24640b5eb0 2022-06-29T16:06:32.721 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.709+0000 7f246359e700 1 -- 172.21.15.60:0/2626834768 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24640b5eb0 msgr2=0x7f2464142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:32.721 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.709+0000 7f246359e700 1 --2- 172.21.15.60:0/2626834768 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24640b5eb0 0x7f2464142a20 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7f244c002f70 tx=0x7f244c006df0).stop 2022-06-29T16:06:32.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.709+0000 7f246359e700 1 -- 172.21.15.60:0/2626834768 shutdown_connections 2022-06-29T16:06:32.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.709+0000 7f246359e700 1 --2- 172.21.15.60:0/2626834768 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f24640b2c50 0x7f24640b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:32.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.709+0000 7f246359e700 1 --2- 172.21.15.60:0/2626834768 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24640b5eb0 0x7f2464142a20 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:32.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.709+0000 7f246359e700 1 -- 172.21.15.60:0/2626834768 >> 172.21.15.60:0/2626834768 conn(0x7f2464062090 msgr2=0x7f2464062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:32.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.709+0000 7f246359e700 1 -- 172.21.15.60:0/2626834768 shutdown_connections 2022-06-29T16:06:32.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.709+0000 7f246359e700 1 -- 172.21.15.60:0/2626834768 wait complete. 2022-06-29T16:06:32.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.713+0000 7f246359e700 1 Processor -- start 2022-06-29T16:06:32.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.713+0000 7f246359e700 1 -- start start 2022-06-29T16:06:32.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.713+0000 7f246359e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f24640b2c50 0x7f246413b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:32.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.713+0000 7f246359e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f24640b48b0 0x7f246413cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:32.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.713+0000 7f246359e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24640b5eb0 0x7f24640ad430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:32.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.713+0000 7f246359e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24640ad000 con 0x7f24640b48b0 2022-06-29T16:06:32.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.713+0000 7f246359e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24640b1610 con 0x7f24640b5eb0 2022-06-29T16:06:32.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.713+0000 7f246359e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24640b18b0 con 0x7f24640b2c50 2022-06-29T16:06:32.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.713+0000 7f2461d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f24640b48b0 0x7f246413cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:32.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.713+0000 7f2461d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f24640b48b0 0x7f246413cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52248/0 (socket says 172.21.15.60:52248) 2022-06-29T16:06:32.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.713+0000 7f2461d9b700 1 -- 172.21.15.60:0/344652867 learned_addr learned my addr 172.21.15.60:0/344652867 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:32.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.713+0000 7f246259c700 1 --2- 172.21.15.60:0/344652867 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f24640b2c50 0x7f246413b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:32.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.713+0000 7f2462d9d700 1 --2- 172.21.15.60:0/344652867 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24640b5eb0 0x7f24640ad430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:32.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.713+0000 7f2461d9b700 1 -- 172.21.15.60:0/344652867 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f24640b2c50 msgr2=0x7f246413b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:32.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.713+0000 7f2461d9b700 1 --2- 172.21.15.60:0/344652867 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f24640b2c50 0x7f246413b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:32.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.713+0000 7f2461d9b700 1 -- 172.21.15.60:0/344652867 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24640b5eb0 msgr2=0x7f24640ad430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:32.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.713+0000 7f2461d9b700 1 --2- 172.21.15.60:0/344652867 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24640b5eb0 0x7f24640ad430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:32.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.713+0000 7f2461d9b700 1 -- 172.21.15.60:0/344652867 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f245c009470 con 0x7f24640b48b0 2022-06-29T16:06:32.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.713+0000 7f2461d9b700 1 --2- 172.21.15.60:0/344652867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f24640b48b0 0x7f246413cb30 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 rx=0x7f24540012a0 tx=0x7f2454007920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:32.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.713+0000 7f245b7fe700 1 -- 172.21.15.60:0/344652867 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f245400ce70 con 0x7f24640b48b0 2022-06-29T16:06:32.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.713+0000 7f245b7fe700 1 -- 172.21.15.60:0/344652867 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24540072e0 con 0x7f24640b48b0 2022-06-29T16:06:32.728 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.713+0000 7f246359e700 1 -- 172.21.15.60:0/344652867 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f244c00b180 con 0x7f24640b48b0 2022-06-29T16:06:32.728 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.713+0000 7f245b7fe700 1 -- 172.21.15.60:0/344652867 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2454005800 con 0x7f24640b48b0 2022-06-29T16:06:32.728 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.713+0000 7f246359e700 1 -- 172.21.15.60:0/344652867 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24640b1b50 con 0x7f24640b48b0 2022-06-29T16:06:32.728 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.713+0000 7f246359e700 1 -- 172.21.15.60:0/344652867 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24640b2180 con 0x7f24640b48b0 2022-06-29T16:06:32.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.717+0000 7f245b7fe700 1 -- 172.21.15.60:0/344652867 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2454010070 con 0x7f24640b48b0 2022-06-29T16:06:32.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.717+0000 7f245b7fe700 1 --2- 172.21.15.60:0/344652867 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2448053aa0 0x7f2448055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:32.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.717+0000 7f245b7fe700 1 -- 172.21.15.60:0/344652867 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f245406b510 con 0x7f24640b48b0 2022-06-29T16:06:32.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.717+0000 7f246259c700 1 --2- 172.21.15.60:0/344652867 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2448053aa0 0x7f2448055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:32.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.717+0000 7f246259c700 1 --2- 172.21.15.60:0/344652867 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2448053aa0 0x7f2448055f60 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7f244c018000 tx=0x7f244c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:32.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.721+0000 7f245b7fe700 1 -- 172.21.15.60:0/344652867 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f24640b2180 con 0x7f24640b48b0 2022-06-29T16:06:32.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.849+0000 7f246359e700 1 -- 172.21.15.60:0/344652867 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f24640b0cf0 con 0x7f24640b48b0 2022-06-29T16:06:32.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.853+0000 7f245b7fe700 1 -- 172.21.15.60:0/344652867 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v110) v1 ==== 100+0+10 (secure 0 0 0) 0x7f24640b0cf0 con 0x7f24640b48b0 2022-06-29T16:06:32.857 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:06:32.859 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.853+0000 7f246359e700 1 -- 172.21.15.60:0/344652867 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2448053aa0 msgr2=0x7f2448055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:32.859 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.853+0000 7f246359e700 1 --2- 172.21.15.60:0/344652867 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2448053aa0 0x7f2448055f60 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7f244c018000 tx=0x7f244c014000).stop 2022-06-29T16:06:32.859 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.853+0000 7f246359e700 1 -- 172.21.15.60:0/344652867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f24640b48b0 msgr2=0x7f246413cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:32.859 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.853+0000 7f246359e700 1 --2- 172.21.15.60:0/344652867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f24640b48b0 0x7f246413cb30 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 rx=0x7f24540012a0 tx=0x7f2454007920).stop 2022-06-29T16:06:32.860 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.853+0000 7f246359e700 1 -- 172.21.15.60:0/344652867 shutdown_connections 2022-06-29T16:06:32.860 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.853+0000 7f246359e700 1 --2- 172.21.15.60:0/344652867 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2448053aa0 0x7f2448055f60 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:32.860 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.853+0000 7f246359e700 1 --2- 172.21.15.60:0/344652867 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24640b5eb0 0x7f24640ad430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:32.860 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.853+0000 7f246359e700 1 --2- 172.21.15.60:0/344652867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f24640b48b0 0x7f246413cb30 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:32.861 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.853+0000 7f246359e700 1 --2- 172.21.15.60:0/344652867 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f24640b2c50 0x7f246413b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:32.861 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.853+0000 7f246359e700 1 -- 172.21.15.60:0/344652867 >> 172.21.15.60:0/344652867 conn(0x7f2464062090 msgr2=0x7f2464062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:32.861 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.853+0000 7f246359e700 1 -- 172.21.15.60:0/344652867 shutdown_connections 2022-06-29T16:06:32.861 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.853+0000 7f246359e700 1 -- 172.21.15.60:0/344652867 wait complete. 2022-06-29T16:06:32.871 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-06-29T16:06:32.871 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:06:32.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.965+0000 7f91e3fff700 1 Processor -- start 2022-06-29T16:06:32.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.965+0000 7f91e3fff700 1 -- start start 2022-06-29T16:06:32.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.965+0000 7f91e3fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f91e405d070 0x7f91e414aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:32.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.965+0000 7f91e3fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91e40b2130 0x7f91e40aca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:32.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.965+0000 7f91e3fff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91e405d450 con 0x7f91e414af80 2022-06-29T16:06:32.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.965+0000 7f91e3fff700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91e40adf90 con 0x7f91e405d070 2022-06-29T16:06:32.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.965+0000 7f91e3fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e40b25d0 con 0x7f91e40b2130 2022-06-29T16:06:32.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.965+0000 7f91e27fc700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91e40b2130 0x7f91e40aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:32.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.965+0000 7f91e27fc700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91e40b2130 0x7f91e40aca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:37118/0 (socket says 172.21.15.60:37118) 2022-06-29T16:06:32.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.965+0000 7f91e37fe700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f91e414af80 0x7f91e40acf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:36362/0 (socket says 172.21.15.60:36362) 2022-06-29T16:06:32.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.965+0000 7f91e27fc700 1 -- 172.21.15.60:0/2260088823 learned_addr learned my addr 172.21.15.60:0/2260088823 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:32.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.965+0000 7f91e2ffd700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f91e405d070 0x7f91e414aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:32.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.969+0000 7f91e1ffb700 1 -- 172.21.15.60:0/2260088823 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3355104475 0 0) 0x7f91e405d450 con 0x7f91e414af80 2022-06-29T16:06:32.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.969+0000 7f91e1ffb700 1 -- 172.21.15.60:0/2260088823 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c4003260 con 0x7f91e414af80 2022-06-29T16:06:32.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.969+0000 7f91e27fc700 1 -- 172.21.15.60:0/2260088823 >> v2:172.21.15.92:3300/0 conn(0x7f91e405d070 msgr2=0x7f91e414aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:32.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.969+0000 7f91e27fc700 1 --2- 172.21.15.60:0/2260088823 >> v2:172.21.15.92:3300/0 conn(0x7f91e405d070 0x7f91e414aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:32.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.969+0000 7f91e27fc700 1 -- 172.21.15.60:0/2260088823 >> v1:172.21.15.92:6789/0 conn(0x7f91e414af80 legacy=0x7f91e40acf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:32.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.969+0000 7f91e27fc700 1 -- 172.21.15.60:0/2260088823 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e414a350 con 0x7f91e40b2130 2022-06-29T16:06:32.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.969+0000 7f91e27fc700 1 --2- 172.21.15.60:0/2260088823 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91e40b2130 0x7f91e40aca20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f91cc0032f0 tx=0x7f91cc009290).ready entity=mon.2 client_cookie=bf0871bd0aa7aa6c server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:32.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.969+0000 7f91e1ffb700 1 -- 172.21.15.60:0/2260088823 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91cc00b6c0 con 0x7f91e40b2130 2022-06-29T16:06:32.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.969+0000 7f91e1ffb700 1 -- 172.21.15.60:0/2260088823 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91cc00b820 con 0x7f91e40b2130 2022-06-29T16:06:32.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.969+0000 7f91e1ffb700 1 -- 172.21.15.60:0/2260088823 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91cc00b990 con 0x7f91e40b2130 2022-06-29T16:06:32.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.969+0000 7f91e3fff700 1 -- 172.21.15.60:0/2260088823 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91e40b2130 msgr2=0x7f91e40aca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:32.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.969+0000 7f91e3fff700 1 --2- 172.21.15.60:0/2260088823 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91e40b2130 0x7f91e40aca20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f91cc0032f0 tx=0x7f91cc009290).stop 2022-06-29T16:06:32.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.969+0000 7f91e3fff700 1 -- 172.21.15.60:0/2260088823 shutdown_connections 2022-06-29T16:06:32.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.969+0000 7f91e3fff700 1 --2- 172.21.15.60:0/2260088823 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91e40b2130 0x7f91e40aca20 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:32.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.969+0000 7f91e3fff700 1 --2- 172.21.15.60:0/2260088823 >> v2:172.21.15.92:3300/0 conn(0x7f91e405d070 0x7f91e414aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:32.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.969+0000 7f91e3fff700 1 -- 172.21.15.60:0/2260088823 >> 172.21.15.60:0/2260088823 conn(0x7f91e40b5890 msgr2=0x7f91e40b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:32.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.969+0000 7f91e3fff700 1 -- 172.21.15.60:0/2260088823 shutdown_connections 2022-06-29T16:06:32.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.969+0000 7f91e3fff700 1 -- 172.21.15.60:0/2260088823 wait complete. 2022-06-29T16:06:32.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.969+0000 7f91e3fff700 1 Processor -- start 2022-06-29T16:06:32.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.969+0000 7f91e3fff700 1 -- start start 2022-06-29T16:06:32.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.973+0000 7f91e3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91e405d070 0x7f91e4159690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:32.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.973+0000 7f91e3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91e40b2130 0x7f91e4144870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:32.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.973+0000 7f91e3fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91e414af80 0x7f91e4143e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:32.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.973+0000 7f91e3fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e415a670 con 0x7f91e40b2130 2022-06-29T16:06:32.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.973+0000 7f91e3fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e414a0c0 con 0x7f91e405d070 2022-06-29T16:06:32.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.973+0000 7f91e3fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e4149b80 con 0x7f91e414af80 2022-06-29T16:06:32.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.973+0000 7f91e27fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91e40b2130 0x7f91e4144870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:32.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.973+0000 7f91e27fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91e40b2130 0x7f91e4144870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52264/0 (socket says 172.21.15.60:52264) 2022-06-29T16:06:32.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.973+0000 7f91e2ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91e405d070 0x7f91e4159690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:32.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.973+0000 7f91e27fc700 1 -- 172.21.15.60:0/3776729005 learned_addr learned my addr 172.21.15.60:0/3776729005 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:32.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.973+0000 7f91e27fc700 1 -- 172.21.15.60:0/3776729005 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91e414af80 msgr2=0x7f91e4143e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-06-29T16:06:32.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.973+0000 7f91e27fc700 1 --2- 172.21.15.60:0/3776729005 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91e414af80 0x7f91e4143e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-06-29T16:06:32.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.973+0000 7f91e27fc700 1 -- 172.21.15.60:0/3776729005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91e405d070 msgr2=0x7f91e4159690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:32.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.973+0000 7f91e27fc700 1 --2- 172.21.15.60:0/3776729005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91e405d070 0x7f91e4159690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:32.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.973+0000 7f91e27fc700 1 -- 172.21.15.60:0/3776729005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91cc009530 con 0x7f91e40b2130 2022-06-29T16:06:32.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.973+0000 7f91e27fc700 1 --2- 172.21.15.60:0/3776729005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91e40b2130 0x7f91e4144870 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 rx=0x7f91e414a250 tx=0x7f91cc016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:32.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.973+0000 7f91dbfff700 1 -- 172.21.15.60:0/3776729005 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91cc00b990 con 0x7f91e40b2130 2022-06-29T16:06:32.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.973+0000 7f91e3fff700 1 -- 172.21.15.60:0/3776729005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91e4149e20 con 0x7f91e40b2130 2022-06-29T16:06:32.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.973+0000 7f91dbfff700 1 -- 172.21.15.60:0/3776729005 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91cc021620 con 0x7f91e40b2130 2022-06-29T16:06:32.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.973+0000 7f91dbfff700 1 -- 172.21.15.60:0/3776729005 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91cc00b6c0 con 0x7f91e40b2130 2022-06-29T16:06:32.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.973+0000 7f91e3fff700 1 -- 172.21.15.60:0/3776729005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91e4064050 con 0x7f91e40b2130 2022-06-29T16:06:32.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.977+0000 7f91e3fff700 1 -- 172.21.15.60:0/3776729005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91e414a5f0 con 0x7f91e40b2130 2022-06-29T16:06:32.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.977+0000 7f91dbfff700 1 -- 172.21.15.60:0/3776729005 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f91cc028070 con 0x7f91e40b2130 2022-06-29T16:06:32.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.977+0000 7f91dbfff700 1 --2- 172.21.15.60:0/3776729005 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f91c8053aa0 0x7f91c8055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:32.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.977+0000 7f91dbfff700 1 -- 172.21.15.60:0/3776729005 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f91cc06cd70 con 0x7f91e40b2130 2022-06-29T16:06:32.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.977+0000 7f91e2ffd700 1 --2- 172.21.15.60:0/3776729005 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f91c8053aa0 0x7f91c8055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:32.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.977+0000 7f91e2ffd700 1 --2- 172.21.15.60:0/3776729005 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f91c8053aa0 0x7f91c8055f60 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7f91d400b000 tx=0x7f91d4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:32.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:32.985+0000 7f91dbfff700 1 -- 172.21.15.60:0/3776729005 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f91cc043b40 con 0x7f91e40b2130 2022-06-29T16:06:33.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:33.149+0000 7f91e3fff700 1 -- 172.21.15.60:0/3776729005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91e40573d0 con 0x7f91e40b2130 2022-06-29T16:06:33.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:33.149+0000 7f91dbfff700 1 -- 172.21.15.60:0/3776729005 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+2142 (secure 0 0 0) 0x7f91cc015030 con 0x7f91e40b2130 2022-06-29T16:06:33.154 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:33.154 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":51,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7648,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/3387042465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3387042465},{"type":"v1","addr":"172.21.15.92:6835","nonce":3387042465}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:06:28.249737+0000","modified":"2022-06-29T16:06:29.252773+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7668},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7668":{"gid":7668,"name":"a","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/2749128920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2749128920},{"type":"v1","addr":"172.21.15.60:6833","nonce":2749128920}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2022-06-29T16:06:33.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:33.149+0000 7f91e3fff700 1 -- 172.21.15.60:0/3776729005 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f91c8053aa0 msgr2=0x7f91c8055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:33.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:33.149+0000 7f91e3fff700 1 --2- 172.21.15.60:0/3776729005 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f91c8053aa0 0x7f91c8055f60 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7f91d400b000 tx=0x7f91d4006df0).stop 2022-06-29T16:06:33.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:33.149+0000 7f91e3fff700 1 -- 172.21.15.60:0/3776729005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91e40b2130 msgr2=0x7f91e4144870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:33.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:33.149+0000 7f91e3fff700 1 --2- 172.21.15.60:0/3776729005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91e40b2130 0x7f91e4144870 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 rx=0x7f91e414a250 tx=0x7f91cc016000).stop 2022-06-29T16:06:33.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:33.149+0000 7f91e3fff700 1 -- 172.21.15.60:0/3776729005 shutdown_connections 2022-06-29T16:06:33.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:33.149+0000 7f91e3fff700 1 --2- 172.21.15.60:0/3776729005 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f91c8053aa0 0x7f91c8055f60 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:33.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:33.149+0000 7f91e3fff700 1 --2- 172.21.15.60:0/3776729005 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f91e414af80 0x7f91e4143e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-06-29T16:06:33.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:33.149+0000 7f91e3fff700 1 --2- 172.21.15.60:0/3776729005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f91e40b2130 0x7f91e4144870 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:33.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:33.149+0000 7f91e3fff700 1 --2- 172.21.15.60:0/3776729005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91e405d070 0x7f91e4159690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:33.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:33.149+0000 7f91e3fff700 1 -- 172.21.15.60:0/3776729005 >> 172.21.15.60:0/3776729005 conn(0x7f91e40b5890 msgr2=0x7f91e4145ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:33.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:33.149+0000 7f91e3fff700 1 -- 172.21.15.60:0/3776729005 shutdown_connections 2022-06-29T16:06:33.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:33.149+0000 7f91e3fff700 1 -- 172.21.15.60:0/3776729005 wait complete. 2022-06-29T16:06:33.160 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 51 2022-06-29T16:06:33.369 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 51, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:06:28.249737+0000', 'modified': '2022-06-29T16:06:29.252773+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7668}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7668': {'gid': 7668, 'name': 'a', 'rank': 0, 'incarnation': 50, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.60:6833/2749128920', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6832', 'nonce': 2749128920}, {'type': 'v1', 'addr': '172.21.15.60:6833', 'nonce': 2749128920}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:06:33.370 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:06:33.370 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-06-29T16:06:33.487 INFO:teuthology.orchestra.run.smithi060.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":7668,"want_state":"up:active","state":"up:active","rank_uptime":5.2215937449999998,"mdsmap_epoch":51,"osdmap_epoch":110,"osdmap_epoch_barrier":109,"uptime":9.3662956620000006} 2022-06-29T16:06:33.487 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 7668, "mdsmap_epoch": 51, "osdmap_epoch": 110, "osdmap_epoch_barrier": 109, "rank_uptime": 5.221593745, "state": "up:active", "uptime": 9.366295662, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:06:33.488 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-06-29T16:06:33.488 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-06-29T16:06:33.489 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi121.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-06-29T16:06:33.489 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:33.490 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:06:33.498 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:06:33.512 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:06:33.512 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:06:33.524 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:06:33.525 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:06:33.526 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:33.527 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:06:33.532 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-06-29T16:06:33.532 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:06:33.577 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:06:33.595 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:06:33.596 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:06:33.607 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:06:33.608 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:06:33.609 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:33.610 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:06:33.653 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:2022-06-29T16:06:33.644+0000 7fe717de5080 -1 init, newargv = 0x55ad6eb002e0 newargc=9ceph-fuse[13314]: starting ceph client 2022-06-29T16:06:33.654 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr: 2022-06-29T16:06:33.663 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[13314]: starting fuse 2022-06-29T16:06:34.760 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:06:34.778 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:06:34.779 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:06:34.835 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:06:34.835 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:06:34.836 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:34.837 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:06:34.842 INFO:teuthology.orchestra.run.smithi121.stdout:52 2022-06-29T16:06:34.842 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-06-29T16:06:34.843 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:34.843 DEBUG:teuthology.orchestra.run.smithi121:> sudo python3 -c ' 2022-06-29T16:06:34.844 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2022-06-29T16:06:34.844 DEBUG:teuthology.orchestra.run.smithi121:> import re 2022-06-29T16:06:34.845 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-06-29T16:06:34.845 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2022-06-29T16:06:34.845 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:06:34.846 DEBUG:teuthology.orchestra.run.smithi121:> def find_socket(client_name): 2022-06-29T16:06:34.846 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-06-29T16:06:34.847 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2022-06-29T16:06:34.847 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:06:34.847 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2022-06-29T16:06:34.848 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2022-06-29T16:06:34.848 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2022-06-29T16:06:34.849 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2022-06-29T16:06:34.849 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:06:34.850 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2022-06-29T16:06:34.850 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-06-29T16:06:34.850 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2022-06-29T16:06:34.851 DEBUG:teuthology.orchestra.run.smithi121:> return f 2022-06-29T16:06:34.851 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-06-29T16:06:34.852 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:06:34.852 DEBUG:teuthology.orchestra.run.smithi121:> print(find_socket("client.0")) 2022-06-29T16:06:34.852 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-06-29T16:06:34.892 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.0.13314.asok 2022-06-29T16:06:34.897 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.13314.asok 2022-06-29T16:06:34.898 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:34.898 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.13314.asok status 2022-06-29T16:06:34.984 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2022-06-29T16:06:34.985 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2022-06-29T16:06:34.985 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "f25ffac72e5a9e63860ff7deb6767a11f0559a16", 2022-06-29T16:06:34.985 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 15.2.16-304-gf25ffac7 (f25ffac72e5a9e63860ff7deb6767a11f0559a16) octopus (stable)", 2022-06-29T16:06:34.985 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "0", 2022-06-29T16:06:34.986 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2022-06-29T16:06:34.986 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-06-29T16:06:34.986 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "13314", 2022-06-29T16:06:34.986 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2022-06-29T16:06:34.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:06:34.987 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2022-06-29T16:06:34.987 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2022-06-29T16:06:34.987 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 6782, 2022-06-29T16:06:34.987 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2022-06-29T16:06:34.988 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2022-06-29T16:06:34.988 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2022-06-29T16:06:34.988 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 6782 2022-06-29T16:06:34.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:06:34.989 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-06-29T16:06:34.989 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-06-29T16:06:34.989 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-06-29T16:06:34.989 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1884298070 2022-06-29T16:06:34.989 INFO:teuthology.orchestra.run.smithi121.stdout: } 2022-06-29T16:06:34.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:06:34.990 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-06-29T16:06:34.990 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-06-29T16:06:34.990 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-06-29T16:06:34.991 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1884298070 2022-06-29T16:06:34.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:06:34.991 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.6782 172.21.15.121:0/1884298070", 2022-06-29T16:06:34.991 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "172.21.15.121:0/1884298070", 2022-06-29T16:06:34.992 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2022-06-29T16:06:34.992 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 51, 2022-06-29T16:06:34.992 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 110, 2022-06-29T16:06:34.992 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2022-06-29T16:06:34.992 INFO:teuthology.orchestra.run.smithi121.stdout: "blacklisted": false, 2022-06-29T16:06:34.993 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2022-06-29T16:06:34.993 INFO:teuthology.orchestra.run.smithi121.stdout:} 2022-06-29T16:06:34.994 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:34.995 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:06:34.995 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:06:35.002 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T16:06:35.003 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T16:06:35.004 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:35.004 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:06:35.020 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:06:35.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.129+0000 7f6eb359e700 1 Processor -- start 2022-06-29T16:06:35.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.129+0000 7f6eb359e700 1 -- start start 2022-06-29T16:06:35.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.129+0000 7f6eb359e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6eb40b2130 0x7f6eb413ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:35.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.129+0000 7f6eb359e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb405d450 con 0x7f6eb405d070 2022-06-29T16:06:35.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.129+0000 7f6eb359e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb4063ba0 con 0x7f6eb40b0830 2022-06-29T16:06:35.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.129+0000 7f6eb359e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6eb4063e40 con 0x7f6eb40b2130 2022-06-29T16:06:35.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.129+0000 7f6eb259c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6eb405d070 0x7f6eb413c5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60354/0 (socket says 172.21.15.60:60354) 2022-06-29T16:06:35.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.129+0000 7f6eb259c700 1 -- 172.21.15.60:0/986650775 learned_addr learned my addr 172.21.15.60:0/986650775 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:35.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.129+0000 7f6eb1d9b700 1 --2- 172.21.15.60:0/986650775 >> v2:172.21.15.60:3300/0 conn(0x7f6eb40b2130 0x7f6eb413ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:35.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.129+0000 7f6eb159a700 1 -- 172.21.15.60:0/986650775 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3794958003 0 0) 0x7f6eb405d450 con 0x7f6eb405d070 2022-06-29T16:06:35.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.129+0000 7f6eb159a700 1 -- 172.21.15.60:0/986650775 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e98003260 con 0x7f6eb405d070 2022-06-29T16:06:35.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.133+0000 7f6eb159a700 1 -- 172.21.15.60:0/986650775 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 370791791 0 0) 0x7f6eb4063ba0 con 0x7f6eb40b0830 2022-06-29T16:06:35.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.133+0000 7f6eb1d9b700 1 -- 172.21.15.60:0/986650775 >> v1:172.21.15.92:6789/0 conn(0x7f6eb40b0830 legacy=0x7f6eb413fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:35.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.133+0000 7f6eb1d9b700 1 -- 172.21.15.60:0/986650775 >> v1:172.21.15.60:6789/0 conn(0x7f6eb405d070 legacy=0x7f6eb413c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:35.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.133+0000 7f6eb1d9b700 1 -- 172.21.15.60:0/986650775 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb4067650 con 0x7f6eb40b2130 2022-06-29T16:06:35.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.133+0000 7f6eb1d9b700 1 --2- 172.21.15.60:0/986650775 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6eb40b2130 0x7f6eb413ccc0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 rx=0x7f6ea8002f70 tx=0x7f6ea8006df0).ready entity=mon.0 client_cookie=901102f3d97015da server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:35.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.133+0000 7f6eb159a700 1 -- 172.21.15.60:0/986650775 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ea800d610 con 0x7f6eb40b2130 2022-06-29T16:06:35.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.133+0000 7f6eb159a700 1 -- 172.21.15.60:0/986650775 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ea8000e90 con 0x7f6eb40b2130 2022-06-29T16:06:35.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.133+0000 7f6eb159a700 1 -- 172.21.15.60:0/986650775 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ea800d960 con 0x7f6eb40b2130 2022-06-29T16:06:35.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.133+0000 7f6eb359e700 1 -- 172.21.15.60:0/986650775 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6eb40b2130 msgr2=0x7f6eb413ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:35.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.133+0000 7f6eb359e700 1 --2- 172.21.15.60:0/986650775 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6eb40b2130 0x7f6eb413ccc0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 rx=0x7f6ea8002f70 tx=0x7f6ea8006df0).stop 2022-06-29T16:06:35.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.133+0000 7f6eb359e700 1 -- 172.21.15.60:0/986650775 shutdown_connections 2022-06-29T16:06:35.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.133+0000 7f6eb359e700 1 --2- 172.21.15.60:0/986650775 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6eb40b2130 0x7f6eb413ccc0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:35.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.133+0000 7f6eb359e700 1 -- 172.21.15.60:0/986650775 >> 172.21.15.60:0/986650775 conn(0x7f6eb40b5890 msgr2=0x7f6eb40b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:35.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.133+0000 7f6eb359e700 1 -- 172.21.15.60:0/986650775 shutdown_connections 2022-06-29T16:06:35.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.133+0000 7f6eb359e700 1 -- 172.21.15.60:0/986650775 wait complete. 2022-06-29T16:06:35.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.133+0000 7f6eb359e700 1 Processor -- start 2022-06-29T16:06:35.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.133+0000 7f6eb359e700 1 -- start start 2022-06-29T16:06:35.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.137+0000 7f6eb359e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6eb405d070 0x7f6eb4139480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:35.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.137+0000 7f6eb359e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6eb40b0830 0x7f6eb40ad3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:35.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.137+0000 7f6eb359e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6eb40b2130 0x7f6eb4064860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:35.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.137+0000 7f6eb359e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb4138680 con 0x7f6eb40b0830 2022-06-29T16:06:35.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.137+0000 7f6eb359e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb4064d50 con 0x7f6eb405d070 2022-06-29T16:06:35.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.137+0000 7f6eb359e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb41388b0 con 0x7f6eb40b2130 2022-06-29T16:06:35.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.137+0000 7f6eb1d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6eb40b0830 0x7f6eb40ad3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:35.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.137+0000 7f6eb1d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6eb40b0830 0x7f6eb40ad3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52274/0 (socket says 172.21.15.60:52274) 2022-06-29T16:06:35.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.137+0000 7f6eb1d9b700 1 -- 172.21.15.60:0/2013064781 learned_addr learned my addr 172.21.15.60:0/2013064781 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:35.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.137+0000 7f6eb259c700 1 --2- 172.21.15.60:0/2013064781 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6eb405d070 0x7f6eb4139480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:35.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.137+0000 7f6eb2d9d700 1 --2- 172.21.15.60:0/2013064781 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6eb40b2130 0x7f6eb4064860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:35.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.137+0000 7f6eb1d9b700 1 -- 172.21.15.60:0/2013064781 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6eb40b2130 msgr2=0x7f6eb4064860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:35.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.137+0000 7f6eb1d9b700 1 --2- 172.21.15.60:0/2013064781 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6eb40b2130 0x7f6eb4064860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:35.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.137+0000 7f6eb1d9b700 1 -- 172.21.15.60:0/2013064781 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6eb405d070 msgr2=0x7f6eb4139480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:35.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.137+0000 7f6eb1d9b700 1 --2- 172.21.15.60:0/2013064781 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6eb405d070 0x7f6eb4139480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:35.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.137+0000 7f6eb1d9b700 1 -- 172.21.15.60:0/2013064781 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea800b180 con 0x7f6eb40b0830 2022-06-29T16:06:35.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.137+0000 7f6eb1d9b700 1 --2- 172.21.15.60:0/2013064781 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6eb40b0830 0x7f6eb40ad3b0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 rx=0x7f6ea8002f70 tx=0x7f6ea8014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:35.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.137+0000 7f6ea37fe700 1 -- 172.21.15.60:0/2013064781 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ea8016070 con 0x7f6eb40b0830 2022-06-29T16:06:35.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.137+0000 7f6ea37fe700 1 -- 172.21.15.60:0/2013064781 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ea800de90 con 0x7f6eb40b0830 2022-06-29T16:06:35.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.137+0000 7f6eb359e700 1 -- 172.21.15.60:0/2013064781 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eb4065190 con 0x7f6eb40b0830 2022-06-29T16:06:35.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.137+0000 7f6ea37fe700 1 -- 172.21.15.60:0/2013064781 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ea8013d60 con 0x7f6eb40b0830 2022-06-29T16:06:35.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.137+0000 7f6eb359e700 1 -- 172.21.15.60:0/2013064781 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eb4066760 con 0x7f6eb40b0830 2022-06-29T16:06:35.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.141+0000 7f6eb359e700 1 -- 172.21.15.60:0/2013064781 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eb4065320 con 0x7f6eb40b0830 2022-06-29T16:06:35.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.141+0000 7f6ea37fe700 1 -- 172.21.15.60:0/2013064781 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6ea8025030 con 0x7f6eb40b0830 2022-06-29T16:06:35.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.141+0000 7f6ea37fe700 1 --2- 172.21.15.60:0/2013064781 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6e90053a70 0x7f6e90055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:35.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.141+0000 7f6ea37fe700 1 -- 172.21.15.60:0/2013064781 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6ea806cfa0 con 0x7f6eb40b0830 2022-06-29T16:06:35.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.141+0000 7f6eb259c700 1 --2- 172.21.15.60:0/2013064781 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6e90053a70 0x7f6e90055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:35.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.141+0000 7f6eb259c700 1 --2- 172.21.15.60:0/2013064781 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6e90053a70 0x7f6e90055f30 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0x7f6ea400d000 tx=0x7f6ea4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:35.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.149+0000 7f6ea37fe700 1 -- 172.21.15.60:0/2013064781 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6eb4065320 con 0x7f6eb40b0830 2022-06-29T16:06:35.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.281+0000 7f6eb359e700 1 -- 172.21.15.60:0/2013064781 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6eb4065320 con 0x7f6eb40b0830 2022-06-29T16:06:35.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.281+0000 7f6ea37fe700 1 -- 172.21.15.60:0/2013064781 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16051 (secure 0 0 0) 0x7f6eb4065320 con 0x7f6eb40b0830 2022-06-29T16:06:35.288 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:35.288 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":110,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:06:29.253422+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:06:26.864385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2022-06-29T16:06:27.251709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:06:35.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.285+0000 7f6eb359e700 1 -- 172.21.15.60:0/2013064781 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6e90053a70 msgr2=0x7f6e90055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:35.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.285+0000 7f6eb359e700 1 --2- 172.21.15.60:0/2013064781 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6e90053a70 0x7f6e90055f30 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0x7f6ea400d000 tx=0x7f6ea4009040).stop 2022-06-29T16:06:35.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.285+0000 7f6eb359e700 1 -- 172.21.15.60:0/2013064781 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6eb40b0830 msgr2=0x7f6eb40ad3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:35.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.285+0000 7f6eb359e700 1 --2- 172.21.15.60:0/2013064781 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6eb40b0830 0x7f6eb40ad3b0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 rx=0x7f6ea8002f70 tx=0x7f6ea8014000).stop 2022-06-29T16:06:35.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.285+0000 7f6eb359e700 1 -- 172.21.15.60:0/2013064781 shutdown_connections 2022-06-29T16:06:35.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.285+0000 7f6eb359e700 1 --2- 172.21.15.60:0/2013064781 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6e90053a70 0x7f6e90055f30 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:35.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.285+0000 7f6eb359e700 1 --2- 172.21.15.60:0/2013064781 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6eb40b2130 0x7f6eb4064860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:35.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.285+0000 7f6eb359e700 1 --2- 172.21.15.60:0/2013064781 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6eb40b0830 0x7f6eb40ad3b0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:35.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.285+0000 7f6eb359e700 1 --2- 172.21.15.60:0/2013064781 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6eb405d070 0x7f6eb4139480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:35.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.285+0000 7f6eb359e700 1 -- 172.21.15.60:0/2013064781 >> 172.21.15.60:0/2013064781 conn(0x7f6eb40b5890 msgr2=0x7f6eb413afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:35.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.285+0000 7f6eb359e700 1 -- 172.21.15.60:0/2013064781 shutdown_connections 2022-06-29T16:06:35.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.285+0000 7f6eb359e700 1 -- 172.21.15.60:0/2013064781 wait complete. 2022-06-29T16:06:35.302 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-29T16:06:26.864385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2022-06-29T16:06:27.251709+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-29T16:06:35.303 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:06:35.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.413+0000 7f7001c74700 1 Processor -- start 2022-06-29T16:06:35.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.413+0000 7f7001c74700 1 -- start start 2022-06-29T16:06:35.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.417+0000 7f7001c74700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6ffc0b2790 0x7f6ffc146c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:35.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.417+0000 7f7001c74700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ffc147130 con 0x7f6ffc05f4c0 2022-06-29T16:06:35.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.417+0000 7f7001c74700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ffc146260 con 0x7f6ffc142bb0 2022-06-29T16:06:35.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.417+0000 7f7001c74700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ffc146500 con 0x7f6ffc0b2790 2022-06-29T16:06:35.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.417+0000 7f7000c72700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6ffc0b2790 0x7f6ffc146c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:35.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.417+0000 7f6ff3fff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6ffc05f4c0 0x7f6ffc0b2b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60370/0 (socket says 172.21.15.60:60370) 2022-06-29T16:06:35.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.417+0000 7f6ff3fff700 1 -- 172.21.15.60:0/86807654 learned_addr learned my addr 172.21.15.60:0/86807654 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:35.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.417+0000 7f6ff37fe700 1 -- 172.21.15.60:0/86807654 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1610054732 0 0) 0x7f6ffc147130 con 0x7f6ffc05f4c0 2022-06-29T16:06:35.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.417+0000 7f6ff37fe700 1 -- 172.21.15.60:0/86807654 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fe8003260 con 0x7f6ffc05f4c0 2022-06-29T16:06:35.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.417+0000 7f6ff37fe700 1 -- 172.21.15.60:0/86807654 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2517268253 0 0) 0x7f6ffc146260 con 0x7f6ffc142bb0 2022-06-29T16:06:35.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.417+0000 7f6ff37fe700 1 -- 172.21.15.60:0/86807654 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ffc147130 con 0x7f6ffc142bb0 2022-06-29T16:06:35.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.417+0000 7f7000c72700 1 -- 172.21.15.60:0/86807654 >> v1:172.21.15.92:6789/0 conn(0x7f6ffc142bb0 legacy=0x7f6ffc142ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:35.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.417+0000 7f7000c72700 1 -- 172.21.15.60:0/86807654 >> v1:172.21.15.60:6789/0 conn(0x7f6ffc05f4c0 legacy=0x7f6ffc0b2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:35.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.417+0000 7f7000c72700 1 -- 172.21.15.60:0/86807654 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ffc1467a0 con 0x7f6ffc0b2790 2022-06-29T16:06:35.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.417+0000 7f7000c72700 1 --2- 172.21.15.60:0/86807654 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ffc0b2790 0x7f6ffc146c40 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 rx=0x7f6fec001850 tx=0x7f6ff8006df0).ready entity=mon.0 client_cookie=17c734a3cf857d73 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:35.440 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.417+0000 7f6ff37fe700 1 -- 172.21.15.60:0/86807654 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6fec0016b0 con 0x7f6ffc0b2790 2022-06-29T16:06:35.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.417+0000 7f6ff37fe700 1 -- 172.21.15.60:0/86807654 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff8000e90 con 0x7f6ffc0b2790 2022-06-29T16:06:35.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.417+0000 7f6ff37fe700 1 -- 172.21.15.60:0/86807654 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ff8011400 con 0x7f6ffc0b2790 2022-06-29T16:06:35.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.417+0000 7f7001c74700 1 -- 172.21.15.60:0/86807654 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ffc0b2790 msgr2=0x7f6ffc146c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:35.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.417+0000 7f7001c74700 1 --2- 172.21.15.60:0/86807654 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ffc0b2790 0x7f6ffc146c40 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 rx=0x7f6fec001850 tx=0x7f6ff8006df0).stop 2022-06-29T16:06:35.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.417+0000 7f7001c74700 1 -- 172.21.15.60:0/86807654 shutdown_connections 2022-06-29T16:06:35.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.417+0000 7f7001c74700 1 --2- 172.21.15.60:0/86807654 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ffc0b2790 0x7f6ffc146c40 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:35.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.417+0000 7f7001c74700 1 -- 172.21.15.60:0/86807654 >> 172.21.15.60:0/86807654 conn(0x7f6ffc0637c0 msgr2=0x7f6ffc063ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:35.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.417+0000 7f7001c74700 1 -- 172.21.15.60:0/86807654 shutdown_connections 2022-06-29T16:06:35.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.417+0000 7f7001c74700 1 -- 172.21.15.60:0/86807654 wait complete. 2022-06-29T16:06:35.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.421+0000 7f7001c74700 1 Processor -- start 2022-06-29T16:06:35.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.421+0000 7f7001c74700 1 -- start start 2022-06-29T16:06:35.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.421+0000 7f7001c74700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ffc05f4c0 0x7f6ffc064e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:35.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.421+0000 7f7001c74700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ffc0b2790 0x7f6ffc0aef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:35.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.421+0000 7f7001c74700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6ffc142bb0 0x7f6ffc0af480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:35.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.421+0000 7f7001c74700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc13efd0 con 0x7f6ffc05f4c0 2022-06-29T16:06:35.448 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.421+0000 7f7001c74700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc152770 con 0x7f6ffc0b2790 2022-06-29T16:06:35.448 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.421+0000 7f7001c74700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc152a10 con 0x7f6ffc142bb0 2022-06-29T16:06:35.449 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.421+0000 7f7000c72700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ffc05f4c0 0x7f6ffc064e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:35.449 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.421+0000 7f7000c72700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ffc05f4c0 0x7f6ffc064e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52286/0 (socket says 172.21.15.60:52286) 2022-06-29T16:06:35.449 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.421+0000 7f7000c72700 1 -- 172.21.15.60:0/1179119104 learned_addr learned my addr 172.21.15.60:0/1179119104 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:35.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.421+0000 7f7001473700 1 --2- 172.21.15.60:0/1179119104 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6ffc142bb0 0x7f6ffc0af480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:35.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.421+0000 7f6ff3fff700 1 --2- 172.21.15.60:0/1179119104 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ffc0b2790 0x7f6ffc0aef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:35.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.421+0000 7f7000c72700 1 -- 172.21.15.60:0/1179119104 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6ffc142bb0 msgr2=0x7f6ffc0af480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:35.451 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.421+0000 7f7000c72700 1 --2- 172.21.15.60:0/1179119104 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6ffc142bb0 0x7f6ffc0af480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:35.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.421+0000 7f7000c72700 1 -- 172.21.15.60:0/1179119104 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ffc0b2790 msgr2=0x7f6ffc0aef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:35.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.421+0000 7f7000c72700 1 --2- 172.21.15.60:0/1179119104 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ffc0b2790 0x7f6ffc0aef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:35.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.421+0000 7f7000c72700 1 -- 172.21.15.60:0/1179119104 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff800b180 con 0x7f6ffc05f4c0 2022-06-29T16:06:35.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.421+0000 7f7000c72700 1 --2- 172.21.15.60:0/1179119104 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ffc05f4c0 0x7f6ffc064e80 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 rx=0x7f6fec001850 tx=0x7f6ffc0b2460).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:35.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.425+0000 7f6ff1ffb700 1 -- 172.21.15.60:0/1179119104 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ff800d9a0 con 0x7f6ffc05f4c0 2022-06-29T16:06:35.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.425+0000 7f6ff1ffb700 1 -- 172.21.15.60:0/1179119104 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff800db00 con 0x7f6ffc05f4c0 2022-06-29T16:06:35.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.425+0000 7f7001c74700 1 -- 172.21.15.60:0/1179119104 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ffc152cb0 con 0x7f6ffc05f4c0 2022-06-29T16:06:35.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.425+0000 7f6ff1ffb700 1 -- 172.21.15.60:0/1179119104 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ff800de60 con 0x7f6ffc05f4c0 2022-06-29T16:06:35.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.425+0000 7f7001c74700 1 -- 172.21.15.60:0/1179119104 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ffc0654a0 con 0x7f6ffc05f4c0 2022-06-29T16:06:35.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.425+0000 7f7001c74700 1 -- 172.21.15.60:0/1179119104 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ffc1525c0 con 0x7f6ffc05f4c0 2022-06-29T16:06:35.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.425+0000 7f6ff1ffb700 1 -- 172.21.15.60:0/1179119104 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6ff8024030 con 0x7f6ffc05f4c0 2022-06-29T16:06:35.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.425+0000 7f6ff1ffb700 1 --2- 172.21.15.60:0/1179119104 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6fdc0539e0 0x7f6fdc055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:35.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.425+0000 7f6ff1ffb700 1 -- 172.21.15.60:0/1179119104 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6ff806c8f0 con 0x7f6ffc05f4c0 2022-06-29T16:06:35.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.425+0000 7f6ff3fff700 1 --2- 172.21.15.60:0/1179119104 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6fdc0539e0 0x7f6fdc055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:35.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.425+0000 7f6ff3fff700 1 --2- 172.21.15.60:0/1179119104 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6fdc0539e0 0x7f6fdc055ea0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7f6fec008000 tx=0x7f6fec001720).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:35.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.433+0000 7f6ff1ffb700 1 -- 172.21.15.60:0/1179119104 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6ff8043700 con 0x7f6ffc05f4c0 2022-06-29T16:06:35.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.557+0000 7f7001c74700 1 -- 172.21.15.60:0/1179119104 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f6ffc0573d0 con 0x7f6ffc05f4c0 2022-06-29T16:06:35.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.561+0000 7f6ff1ffb700 1 -- 172.21.15.60:0/1179119104 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v110) v1 ==== 110+0+10 (secure 0 0 0) 0x7f6ff8022070 con 0x7f6ffc05f4c0 2022-06-29T16:06:35.565 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:06:35.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.561+0000 7f7001c74700 1 -- 172.21.15.60:0/1179119104 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6fdc0539e0 msgr2=0x7f6fdc055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:35.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.561+0000 7f7001c74700 1 --2- 172.21.15.60:0/1179119104 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6fdc0539e0 0x7f6fdc055ea0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7f6fec008000 tx=0x7f6fec001720).stop 2022-06-29T16:06:35.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.561+0000 7f7001c74700 1 -- 172.21.15.60:0/1179119104 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ffc05f4c0 msgr2=0x7f6ffc064e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:35.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.561+0000 7f7001c74700 1 --2- 172.21.15.60:0/1179119104 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ffc05f4c0 0x7f6ffc064e80 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 rx=0x7f6fec001850 tx=0x7f6ffc0b2460).stop 2022-06-29T16:06:35.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.561+0000 7f7001c74700 1 -- 172.21.15.60:0/1179119104 shutdown_connections 2022-06-29T16:06:35.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.561+0000 7f7001c74700 1 --2- 172.21.15.60:0/1179119104 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6fdc0539e0 0x7f6fdc055ea0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:35.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.561+0000 7f7001c74700 1 --2- 172.21.15.60:0/1179119104 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6ffc142bb0 0x7f6ffc0af480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:35.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.561+0000 7f7001c74700 1 --2- 172.21.15.60:0/1179119104 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6ffc0b2790 0x7f6ffc0aef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:35.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.561+0000 7f7001c74700 1 --2- 172.21.15.60:0/1179119104 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ffc05f4c0 0x7f6ffc064e80 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:35.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.561+0000 7f7001c74700 1 -- 172.21.15.60:0/1179119104 >> 172.21.15.60:0/1179119104 conn(0x7f6ffc0637c0 msgr2=0x7f6ffc142ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:35.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.561+0000 7f7001c74700 1 -- 172.21.15.60:0/1179119104 shutdown_connections 2022-06-29T16:06:35.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.561+0000 7f7001c74700 1 -- 172.21.15.60:0/1179119104 wait complete. 2022-06-29T16:06:35.578 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:06:35.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.685+0000 7f8ab2ce7700 1 Processor -- start 2022-06-29T16:06:35.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.685+0000 7f8ab2ce7700 1 -- start start 2022-06-29T16:06:35.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.685+0000 7f8ab2ce7700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8aac0b5eb0 0x7f8aac142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:35.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.685+0000 7f8ab2ce7700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8aac0b48b0 0x7f8aac0b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:35.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.685+0000 7f8ab2ce7700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8aac0b2c50 0x7f8aac0b3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:35.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.685+0000 7f8ab2ce7700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aac0b54e0 con 0x7f8aac0b2c50 2022-06-29T16:06:35.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.685+0000 7f8ab2ce7700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aac0b4d10 con 0x7f8aac0b48b0 2022-06-29T16:06:35.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.685+0000 7f8ab2ce7700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac0b6290 con 0x7f8aac0b5eb0 2022-06-29T16:06:35.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab24e6700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8aac0b2c50 0x7f8aac0b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:35.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab24e6700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8aac0b2c50 0x7f8aac0b3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52298/0 (socket says 172.21.15.60:52298) 2022-06-29T16:06:35.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab24e6700 1 -- 172.21.15.60:0/697713459 learned_addr learned my addr 172.21.15.60:0/697713459 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:35.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab1ce5700 1 --2- 172.21.15.60:0/697713459 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8aac0b5eb0 0x7f8aac142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:35.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab14e4700 1 --2- 172.21.15.60:0/697713459 >> v2:172.21.15.92:3300/0 conn(0x7f8aac0b48b0 0x7f8aac0b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:35.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab24e6700 1 -- 172.21.15.60:0/697713459 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8aac0b5eb0 msgr2=0x7f8aac142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:35.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab24e6700 1 --2- 172.21.15.60:0/697713459 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8aac0b5eb0 0x7f8aac142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:35.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab24e6700 1 -- 172.21.15.60:0/697713459 >> v2:172.21.15.92:3300/0 conn(0x7f8aac0b48b0 msgr2=0x7f8aac0b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:35.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab24e6700 1 --2- 172.21.15.60:0/697713459 >> v2:172.21.15.92:3300/0 conn(0x7f8aac0b48b0 0x7f8aac0b05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:35.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab24e6700 1 -- 172.21.15.60:0/697713459 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aac142f10 con 0x7f8aac0b2c50 2022-06-29T16:06:35.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab14e4700 1 --2- 172.21.15.60:0/697713459 >> v2:172.21.15.92:3300/0 conn(0x7f8aac0b48b0 0x7f8aac0b05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:06:35.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab24e6700 1 --2- 172.21.15.60:0/697713459 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8aac0b2c50 0x7f8aac0b3090 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 rx=0x7f8aa40032f0 tx=0x7f8aa4009290).ready entity=mon.0 client_cookie=607f4c3508ccd18c server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:35.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab0ce3700 1 -- 172.21.15.60:0/697713459 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8aa400b870 con 0x7f8aac0b2c50 2022-06-29T16:06:35.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab0ce3700 1 -- 172.21.15.60:0/697713459 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aa400b9d0 con 0x7f8aac0b2c50 2022-06-29T16:06:35.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab0ce3700 1 -- 172.21.15.60:0/697713459 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8aa400bb40 con 0x7f8aac0b2c50 2022-06-29T16:06:35.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab2ce7700 1 -- 172.21.15.60:0/697713459 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8aac0b2c50 msgr2=0x7f8aac0b3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:35.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab2ce7700 1 --2- 172.21.15.60:0/697713459 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8aac0b2c50 0x7f8aac0b3090 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 rx=0x7f8aa40032f0 tx=0x7f8aa4009290).stop 2022-06-29T16:06:35.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab2ce7700 1 -- 172.21.15.60:0/697713459 shutdown_connections 2022-06-29T16:06:35.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab2ce7700 1 --2- 172.21.15.60:0/697713459 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8aac0b2c50 0x7f8aac0b3090 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:35.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab2ce7700 1 --2- 172.21.15.60:0/697713459 >> v2:172.21.15.92:3300/0 conn(0x7f8aac0b48b0 0x7f8aac0b05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:35.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab2ce7700 1 --2- 172.21.15.60:0/697713459 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8aac0b5eb0 0x7f8aac142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:35.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab2ce7700 1 -- 172.21.15.60:0/697713459 >> 172.21.15.60:0/697713459 conn(0x7f8aac062090 msgr2=0x7f8aac062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:35.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab2ce7700 1 -- 172.21.15.60:0/697713459 shutdown_connections 2022-06-29T16:06:35.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab2ce7700 1 -- 172.21.15.60:0/697713459 wait complete. 2022-06-29T16:06:35.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab2ce7700 1 Processor -- start 2022-06-29T16:06:35.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab2ce7700 1 -- start start 2022-06-29T16:06:35.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab2ce7700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8aac0b2c50 0x7f8aac13d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:35.721 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab2ce7700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8aac0b48b0 0x7f8aac155010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:35.721 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab2ce7700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8aac0b5eb0 0x7f8aac0acef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:35.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab2ce7700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac0ade00 con 0x7f8aac0b48b0 2022-06-29T16:06:35.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab2ce7700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac0acac0 con 0x7f8aac0b5eb0 2022-06-29T16:06:35.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab2ce7700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac155500 con 0x7f8aac0b2c50 2022-06-29T16:06:35.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab14e4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8aac0b48b0 0x7f8aac155010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:35.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab14e4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8aac0b48b0 0x7f8aac155010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52300/0 (socket says 172.21.15.60:52300) 2022-06-29T16:06:35.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab14e4700 1 -- 172.21.15.60:0/3588688967 learned_addr learned my addr 172.21.15.60:0/3588688967 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:35.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab24e6700 1 --2- 172.21.15.60:0/3588688967 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8aac0b5eb0 0x7f8aac0acef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:35.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.689+0000 7f8ab1ce5700 1 --2- 172.21.15.60:0/3588688967 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8aac0b2c50 0x7f8aac13d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:35.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.693+0000 7f8ab14e4700 1 -- 172.21.15.60:0/3588688967 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8aac0b2c50 msgr2=0x7f8aac13d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:35.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.693+0000 7f8ab14e4700 1 --2- 172.21.15.60:0/3588688967 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8aac0b2c50 0x7f8aac13d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:35.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.693+0000 7f8ab14e4700 1 -- 172.21.15.60:0/3588688967 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8aac0b5eb0 msgr2=0x7f8aac0acef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:35.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.693+0000 7f8ab14e4700 1 --2- 172.21.15.60:0/3588688967 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8aac0b5eb0 0x7f8aac0acef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:35.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.693+0000 7f8ab14e4700 1 -- 172.21.15.60:0/3588688967 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa4009530 con 0x7f8aac0b48b0 2022-06-29T16:06:35.728 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.693+0000 7f8ab14e4700 1 --2- 172.21.15.60:0/3588688967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8aac0b48b0 0x7f8aac155010 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 rx=0x7f8a9c006940 tx=0x7f8a9c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:35.728 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.693+0000 7f8a9affd700 1 -- 172.21.15.60:0/3588688967 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a9c011780 con 0x7f8aac0b48b0 2022-06-29T16:06:35.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.693+0000 7f8a9affd700 1 -- 172.21.15.60:0/3588688967 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a9c0118e0 con 0x7f8aac0b48b0 2022-06-29T16:06:35.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.693+0000 7f8ab2ce7700 1 -- 172.21.15.60:0/3588688967 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aac13ea90 con 0x7f8aac0b48b0 2022-06-29T16:06:35.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.693+0000 7f8a9affd700 1 -- 172.21.15.60:0/3588688967 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a9c011c40 con 0x7f8aac0b48b0 2022-06-29T16:06:35.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.693+0000 7f8ab2ce7700 1 -- 172.21.15.60:0/3588688967 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aac13e790 con 0x7f8aac0b48b0 2022-06-29T16:06:35.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.693+0000 7f8ab2ce7700 1 -- 172.21.15.60:0/3588688967 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aac0b4f70 con 0x7f8aac0b48b0 2022-06-29T16:06:35.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.693+0000 7f8a9affd700 1 -- 172.21.15.60:0/3588688967 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8a9c012420 con 0x7f8aac0b48b0 2022-06-29T16:06:35.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.697+0000 7f8a9affd700 1 --2- 172.21.15.60:0/3588688967 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8a8c053a70 0x7f8a8c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:35.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.697+0000 7f8a9affd700 1 -- 172.21.15.60:0/3588688967 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8a9c06c6d0 con 0x7f8aac0b48b0 2022-06-29T16:06:35.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.697+0000 7f8ab1ce5700 1 --2- 172.21.15.60:0/3588688967 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8a8c053a70 0x7f8a8c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:35.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.697+0000 7f8ab1ce5700 1 --2- 172.21.15.60:0/3588688967 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8a8c053a70 0x7f8a8c055f30 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7f8aa800c000 tx=0x7f8aa8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:35.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.701+0000 7f8a9affd700 1 -- 172.21.15.60:0/3588688967 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8a9c043590 con 0x7f8aac0b48b0 2022-06-29T16:06:35.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.825+0000 7f8ab2ce7700 1 -- 172.21.15.60:0/3588688967 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f8aac0573d0 con 0x7f8aac0b48b0 2022-06-29T16:06:35.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.825+0000 7f8a9affd700 1 -- 172.21.15.60:0/3588688967 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v110) v1 ==== 92+0+10 (secure 0 0 0) 0x7f8a9c046bb0 con 0x7f8aac0b48b0 2022-06-29T16:06:35.831 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:06:35.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.829+0000 7f8ab2ce7700 1 -- 172.21.15.60:0/3588688967 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8a8c053a70 msgr2=0x7f8a8c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:35.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.829+0000 7f8ab2ce7700 1 --2- 172.21.15.60:0/3588688967 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8a8c053a70 0x7f8a8c055f30 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7f8aa800c000 tx=0x7f8aa8008040).stop 2022-06-29T16:06:35.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.829+0000 7f8ab2ce7700 1 -- 172.21.15.60:0/3588688967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8aac0b48b0 msgr2=0x7f8aac155010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:35.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.829+0000 7f8ab2ce7700 1 --2- 172.21.15.60:0/3588688967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8aac0b48b0 0x7f8aac155010 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 rx=0x7f8a9c006940 tx=0x7f8a9c008040).stop 2022-06-29T16:06:35.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.829+0000 7f8ab2ce7700 1 -- 172.21.15.60:0/3588688967 shutdown_connections 2022-06-29T16:06:35.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.829+0000 7f8ab2ce7700 1 --2- 172.21.15.60:0/3588688967 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8a8c053a70 0x7f8a8c055f30 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:35.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.829+0000 7f8ab2ce7700 1 --2- 172.21.15.60:0/3588688967 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8aac0b5eb0 0x7f8aac0acef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:35.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.829+0000 7f8ab2ce7700 1 --2- 172.21.15.60:0/3588688967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8aac0b48b0 0x7f8aac155010 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:35.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.829+0000 7f8ab2ce7700 1 --2- 172.21.15.60:0/3588688967 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8aac0b2c50 0x7f8aac13d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:35.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.829+0000 7f8ab2ce7700 1 -- 172.21.15.60:0/3588688967 >> 172.21.15.60:0/3588688967 conn(0x7f8aac062090 msgr2=0x7f8aac0b8850 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:35.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.829+0000 7f8ab2ce7700 1 -- 172.21.15.60:0/3588688967 shutdown_connections 2022-06-29T16:06:35.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.829+0000 7f8ab2ce7700 1 -- 172.21.15.60:0/3588688967 wait complete. 2022-06-29T16:06:35.849 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-29T16:06:35.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.957+0000 7f27962b3700 1 Processor -- start 2022-06-29T16:06:35.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.957+0000 7f27962b3700 1 -- start start 2022-06-29T16:06:35.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.961+0000 7f27962b3700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f279005d070 0x7f279013c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:35.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.961+0000 7f27962b3700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f27900b2130 0x7f279013caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:35.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.961+0000 7f27962b3700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f279005d450 con 0x7f27900b0830 2022-06-29T16:06:35.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.961+0000 7f27962b3700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2790063ba0 con 0x7f27900b2130 2022-06-29T16:06:35.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.961+0000 7f27962b3700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2790063e40 con 0x7f279005d070 2022-06-29T16:06:35.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.961+0000 7f2794ab0700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f27900b2130 0x7f279013caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:35.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.961+0000 7f2794ab0700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f27900b2130 0x7f279013caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52310/0 (socket says 172.21.15.60:52310) 2022-06-29T16:06:35.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.961+0000 7f2794ab0700 1 -- 172.21.15.60:0/1440465861 learned_addr learned my addr 172.21.15.60:0/1440465861 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:35.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.961+0000 7f27952b1700 1 --2- 172.21.15.60:0/1440465861 >> v2:172.21.15.92:3300/0 conn(0x7f279005d070 0x7f279013c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:35.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.961+0000 7f2794ab0700 1 -- 172.21.15.60:0/1440465861 >> v2:172.21.15.92:3300/0 conn(0x7f279005d070 msgr2=0x7f279013c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:35.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.961+0000 7f2794ab0700 1 --2- 172.21.15.60:0/1440465861 >> v2:172.21.15.92:3300/0 conn(0x7f279005d070 0x7f279013c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:35.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.961+0000 7f2794ab0700 1 -- 172.21.15.60:0/1440465861 >> v1:172.21.15.92:6789/0 conn(0x7f27900b0830 legacy=0x7f279013fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:35.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.961+0000 7f2794ab0700 1 -- 172.21.15.60:0/1440465861 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2790067650 con 0x7f27900b2130 2022-06-29T16:06:35.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.961+0000 7f2794ab0700 1 --2- 172.21.15.60:0/1440465861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27900b2130 0x7f279013caa0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 rx=0x7f2780002f70 tx=0x7f2780006eb0).ready entity=mon.0 client_cookie=19e26d67d9703444 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:35.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.961+0000 7f2787fff700 1 -- 172.21.15.60:0/1440465861 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f278000fa10 con 0x7f27900b2130 2022-06-29T16:06:35.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.961+0000 7f2787fff700 1 -- 172.21.15.60:0/1440465861 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2780000e90 con 0x7f27900b2130 2022-06-29T16:06:35.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.961+0000 7f2787fff700 1 -- 172.21.15.60:0/1440465861 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f278000fda0 con 0x7f27900b2130 2022-06-29T16:06:35.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.961+0000 7f27962b3700 1 -- 172.21.15.60:0/1440465861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27900b2130 msgr2=0x7f279013caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:35.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.961+0000 7f27962b3700 1 --2- 172.21.15.60:0/1440465861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27900b2130 0x7f279013caa0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 rx=0x7f2780002f70 tx=0x7f2780006eb0).stop 2022-06-29T16:06:35.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.961+0000 7f27962b3700 1 -- 172.21.15.60:0/1440465861 shutdown_connections 2022-06-29T16:06:35.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.961+0000 7f27962b3700 1 --2- 172.21.15.60:0/1440465861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27900b2130 0x7f279013caa0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:35.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.961+0000 7f27962b3700 1 --2- 172.21.15.60:0/1440465861 >> v2:172.21.15.92:3300/0 conn(0x7f279005d070 0x7f279013c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:35.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.961+0000 7f27962b3700 1 -- 172.21.15.60:0/1440465861 >> 172.21.15.60:0/1440465861 conn(0x7f27900b5890 msgr2=0x7f27900b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:35.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.961+0000 7f27962b3700 1 -- 172.21.15.60:0/1440465861 shutdown_connections 2022-06-29T16:06:35.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.961+0000 7f27962b3700 1 -- 172.21.15.60:0/1440465861 wait complete. 2022-06-29T16:06:35.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.965+0000 7f27962b3700 1 Processor -- start 2022-06-29T16:06:35.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.965+0000 7f27962b3700 1 -- start start 2022-06-29T16:06:35.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.965+0000 7f27962b3700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f279005d070 0x7f27900adc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:35.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.965+0000 7f27962b3700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27900b0830 0x7f279014d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:35.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.965+0000 7f27962b3700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27900b2130 0x7f27900b3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:35.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.965+0000 7f27962b3700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f279013cd50 con 0x7f27900b0830 2022-06-29T16:06:35.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.965+0000 7f27962b3700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27900b4000 con 0x7f27900b2130 2022-06-29T16:06:35.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.965+0000 7f27962b3700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27900ae2a0 con 0x7f279005d070 2022-06-29T16:06:35.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.965+0000 7f2794ab0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27900b0830 0x7f279014d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:35.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.965+0000 7f27952b1700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f279005d070 0x7f27900adc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:35.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.965+0000 7f2794ab0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27900b0830 0x7f279014d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52316/0 (socket says 172.21.15.60:52316) 2022-06-29T16:06:35.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.965+0000 7f2794ab0700 1 -- 172.21.15.60:0/1075116877 learned_addr learned my addr 172.21.15.60:0/1075116877 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:35.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.965+0000 7f2795ab2700 1 --2- 172.21.15.60:0/1075116877 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27900b2130 0x7f27900b3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:35.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.965+0000 7f27952b1700 1 -- 172.21.15.60:0/1075116877 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27900b2130 msgr2=0x7f27900b3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:35.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.965+0000 7f27952b1700 1 --2- 172.21.15.60:0/1075116877 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27900b2130 0x7f27900b3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:35.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.965+0000 7f27952b1700 1 -- 172.21.15.60:0/1075116877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27900b0830 msgr2=0x7f279014d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:35.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.965+0000 7f27952b1700 1 --2- 172.21.15.60:0/1075116877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27900b0830 0x7f279014d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:35.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.965+0000 7f27952b1700 1 -- 172.21.15.60:0/1075116877 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f278000b1e0 con 0x7f279005d070 2022-06-29T16:06:35.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.965+0000 7f2794ab0700 1 --2- 172.21.15.60:0/1075116877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27900b0830 0x7f279014d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:06:35.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.965+0000 7f27952b1700 1 --2- 172.21.15.60:0/1075116877 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f279005d070 0x7f27900adc70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f278c00a000 tx=0x7f278c008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:35.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.965+0000 7f27867fc700 1 -- 172.21.15.60:0/1075116877 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f278c011590 con 0x7f279005d070 2022-06-29T16:06:35.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.965+0000 7f27867fc700 1 -- 172.21.15.60:0/1075116877 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f278c0116f0 con 0x7f279005d070 2022-06-29T16:06:35.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.965+0000 7f27962b3700 1 -- 172.21.15.60:0/1075116877 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2790150ee0 con 0x7f279005d070 2022-06-29T16:06:35.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.965+0000 7f27867fc700 1 -- 172.21.15.60:0/1075116877 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f278c011a50 con 0x7f279005d070 2022-06-29T16:06:35.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.965+0000 7f27962b3700 1 -- 172.21.15.60:0/1075116877 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f279014dde0 con 0x7f279005d070 2022-06-29T16:06:35.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.969+0000 7f27962b3700 1 -- 172.21.15.60:0/1075116877 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27900573d0 con 0x7f279005d070 2022-06-29T16:06:35.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.969+0000 7f27867fc700 1 -- 172.21.15.60:0/1075116877 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f278c011bb0 con 0x7f279005d070 2022-06-29T16:06:35.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.969+0000 7f27867fc700 1 --2- 172.21.15.60:0/1075116877 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2778053a70 0x7f2778055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:35.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.969+0000 7f27867fc700 1 -- 172.21.15.60:0/1075116877 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f278c06bfe0 con 0x7f279005d070 2022-06-29T16:06:35.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.969+0000 7f2794ab0700 1 --2- 172.21.15.60:0/1075116877 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2778053a70 0x7f2778055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:35.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.969+0000 7f2794ab0700 1 --2- 172.21.15.60:0/1075116877 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2778053a70 0x7f2778055f30 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7f278001e000 tx=0x7f2780017000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:35.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:35.977+0000 7f27867fc700 1 -- 172.21.15.60:0/1075116877 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f278c042da0 con 0x7f279005d070 2022-06-29T16:06:36.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.105+0000 7f27962b3700 1 -- 172.21.15.60:0/1075116877 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f27901426a0 con 0x7f279005d070 2022-06-29T16:06:36.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.109+0000 7f27867fc700 1 -- 172.21.15.60:0/1075116877 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v110) v1 ==== 104+0+10 (secure 0 0 0) 0x7f278c0463c0 con 0x7f279005d070 2022-06-29T16:06:36.113 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:06:36.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.109+0000 7f27962b3700 1 -- 172.21.15.60:0/1075116877 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2778053a70 msgr2=0x7f2778055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:36.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.109+0000 7f27962b3700 1 --2- 172.21.15.60:0/1075116877 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2778053a70 0x7f2778055f30 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7f278001e000 tx=0x7f2780017000).stop 2022-06-29T16:06:36.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.109+0000 7f27962b3700 1 -- 172.21.15.60:0/1075116877 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f279005d070 msgr2=0x7f27900adc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:36.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.109+0000 7f27962b3700 1 --2- 172.21.15.60:0/1075116877 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f279005d070 0x7f27900adc70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f278c00a000 tx=0x7f278c008040).stop 2022-06-29T16:06:36.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.109+0000 7f27962b3700 1 -- 172.21.15.60:0/1075116877 shutdown_connections 2022-06-29T16:06:36.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.109+0000 7f27962b3700 1 --2- 172.21.15.60:0/1075116877 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2778053a70 0x7f2778055f30 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:36.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.109+0000 7f27962b3700 1 --2- 172.21.15.60:0/1075116877 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27900b2130 0x7f27900b3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:36.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.109+0000 7f27962b3700 1 --2- 172.21.15.60:0/1075116877 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27900b0830 0x7f279014d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:36.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.109+0000 7f27962b3700 1 --2- 172.21.15.60:0/1075116877 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f279005d070 0x7f27900adc70 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:36.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.109+0000 7f27962b3700 1 -- 172.21.15.60:0/1075116877 >> 172.21.15.60:0/1075116877 conn(0x7f27900b5890 msgr2=0x7f2790142b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:36.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.109+0000 7f27962b3700 1 -- 172.21.15.60:0/1075116877 shutdown_connections 2022-06-29T16:06:36.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.109+0000 7f27962b3700 1 -- 172.21.15.60:0/1075116877 wait complete. 2022-06-29T16:06:36.126 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-29T16:06:36.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.229+0000 7fb891526700 1 Processor -- start 2022-06-29T16:06:36.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.229+0000 7fb891526700 1 -- start start 2022-06-29T16:06:36.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.233+0000 7fb891526700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb88c0b5eb0 0x7fb88c142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:36.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.233+0000 7fb891526700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb88c0b0cf0 con 0x7fb88c0b2c50 2022-06-29T16:06:36.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.233+0000 7fb891526700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb88c0b4d10 con 0x7fb88c0b48b0 2022-06-29T16:06:36.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.233+0000 7fb891526700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb88c0b54e0 con 0x7fb88c0b5eb0 2022-06-29T16:06:36.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.233+0000 7fb88bfff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb88c0b5eb0 0x7fb88c142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:36.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.233+0000 7fb890d25700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb88c0b2c50 0x7fb88c14d380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60410/0 (socket says 172.21.15.60:60410) 2022-06-29T16:06:36.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.233+0000 7fb890d25700 1 -- 172.21.15.60:0/2732375595 learned_addr learned my addr 172.21.15.60:0/2732375595 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:36.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.233+0000 7fb88bfff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb88c0b5eb0 0x7fb88c142a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52322/0 (socket says 172.21.15.60:52322) 2022-06-29T16:06:36.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.233+0000 7fb88affd700 1 -- 172.21.15.60:0/2732375595 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2023739194 0 0) 0x7fb884001490 con 0x7fb88c0b2c50 2022-06-29T16:06:36.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.233+0000 7fb88bfff700 1 -- 172.21.15.60:0/2732375595 >> v1:172.21.15.92:6789/0 conn(0x7fb88c0b48b0 legacy=0x7fb88c0b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:36.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.233+0000 7fb88bfff700 1 -- 172.21.15.60:0/2732375595 >> v1:172.21.15.60:6789/0 conn(0x7fb88c0b2c50 legacy=0x7fb88c14d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:36.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.233+0000 7fb88bfff700 1 -- 172.21.15.60:0/2732375595 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb88c0b6290 con 0x7fb88c0b5eb0 2022-06-29T16:06:36.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.233+0000 7fb88bfff700 1 --2- 172.21.15.60:0/2732375595 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb88c0b5eb0 0x7fb88c142a20 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 rx=0x7fb874002f70 tx=0x7fb874006df0).ready entity=mon.0 client_cookie=fd813b4d68b7a312 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:36.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.233+0000 7fb88affd700 1 -- 172.21.15.60:0/2732375595 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb87400d610 con 0x7fb88c0b5eb0 2022-06-29T16:06:36.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.233+0000 7fb88affd700 1 -- 172.21.15.60:0/2732375595 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb874000e90 con 0x7fb88c0b5eb0 2022-06-29T16:06:36.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.233+0000 7fb88affd700 1 -- 172.21.15.60:0/2732375595 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb87400d960 con 0x7fb88c0b5eb0 2022-06-29T16:06:36.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.233+0000 7fb891526700 1 -- 172.21.15.60:0/2732375595 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb88c0b5eb0 msgr2=0x7fb88c142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:36.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.233+0000 7fb891526700 1 --2- 172.21.15.60:0/2732375595 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb88c0b5eb0 0x7fb88c142a20 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 rx=0x7fb874002f70 tx=0x7fb874006df0).stop 2022-06-29T16:06:36.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.233+0000 7fb891526700 1 -- 172.21.15.60:0/2732375595 shutdown_connections 2022-06-29T16:06:36.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.233+0000 7fb891526700 1 --2- 172.21.15.60:0/2732375595 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb88c0b5eb0 0x7fb88c142a20 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:36.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.233+0000 7fb891526700 1 -- 172.21.15.60:0/2732375595 >> 172.21.15.60:0/2732375595 conn(0x7fb88c062090 msgr2=0x7fb88c062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:36.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.233+0000 7fb891526700 1 -- 172.21.15.60:0/2732375595 shutdown_connections 2022-06-29T16:06:36.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.233+0000 7fb891526700 1 -- 172.21.15.60:0/2732375595 wait complete. 2022-06-29T16:06:36.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.237+0000 7fb891526700 1 Processor -- start 2022-06-29T16:06:36.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.237+0000 7fb891526700 1 -- start start 2022-06-29T16:06:36.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.237+0000 7fb891526700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb88c0b2c50 0x7fb88c0aec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:36.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.237+0000 7fb891526700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb88c0b48b0 0x7fb88c155a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:36.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.237+0000 7fb891526700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb88c0af130 0x7fb88c155f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:36.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.237+0000 7fb891526700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb88c1124b0 con 0x7fb88c0af130 2022-06-29T16:06:36.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.237+0000 7fb891526700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb88c114770 con 0x7fb88c0b48b0 2022-06-29T16:06:36.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.237+0000 7fb891526700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb88c1128b0 con 0x7fb88c0b2c50 2022-06-29T16:06:36.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.237+0000 7fb890d25700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb88c0af130 0x7fb88c155f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:36.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.237+0000 7fb890d25700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb88c0af130 0x7fb88c155f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52332/0 (socket says 172.21.15.60:52332) 2022-06-29T16:06:36.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.237+0000 7fb890d25700 1 -- 172.21.15.60:0/3186309088 learned_addr learned my addr 172.21.15.60:0/3186309088 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:36.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.237+0000 7fb88bfff700 1 --2- 172.21.15.60:0/3186309088 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb88c0b2c50 0x7fb88c0aec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:36.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.237+0000 7fb88b7fe700 1 --2- 172.21.15.60:0/3186309088 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb88c0b48b0 0x7fb88c155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:36.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.237+0000 7fb890d25700 1 -- 172.21.15.60:0/3186309088 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb88c0b2c50 msgr2=0x7fb88c0aec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:36.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.237+0000 7fb890d25700 1 --2- 172.21.15.60:0/3186309088 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb88c0b2c50 0x7fb88c0aec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:36.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.237+0000 7fb890d25700 1 -- 172.21.15.60:0/3186309088 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb88c0b48b0 msgr2=0x7fb88c155a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:36.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.237+0000 7fb890d25700 1 --2- 172.21.15.60:0/3186309088 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb88c0b48b0 0x7fb88c155a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:36.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.237+0000 7fb890d25700 1 -- 172.21.15.60:0/3186309088 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87400b180 con 0x7fb88c0af130 2022-06-29T16:06:36.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.237+0000 7fb88b7fe700 1 --2- 172.21.15.60:0/3186309088 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb88c0b48b0 0x7fb88c155a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:06:36.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.241+0000 7fb890d25700 1 --2- 172.21.15.60:0/3186309088 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb88c0af130 0x7fb88c155f00 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 rx=0x7fb884001250 tx=0x7fb884007c20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:36.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.241+0000 7fb8897fa700 1 -- 172.21.15.60:0/3186309088 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb884005ba0 con 0x7fb88c0af130 2022-06-29T16:06:36.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.241+0000 7fb8897fa700 1 -- 172.21.15.60:0/3186309088 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb884005d00 con 0x7fb88c0af130 2022-06-29T16:06:36.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.241+0000 7fb891526700 1 -- 172.21.15.60:0/3186309088 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb88c1553e0 con 0x7fb88c0af130 2022-06-29T16:06:36.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.241+0000 7fb8897fa700 1 -- 172.21.15.60:0/3186309088 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb884004a90 con 0x7fb88c0af130 2022-06-29T16:06:36.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.241+0000 7fb891526700 1 -- 172.21.15.60:0/3186309088 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb88c112d70 con 0x7fb88c0af130 2022-06-29T16:06:36.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.241+0000 7fb891526700 1 -- 172.21.15.60:0/3186309088 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb88c113a20 con 0x7fb88c0af130 2022-06-29T16:06:36.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.241+0000 7fb8897fa700 1 -- 172.21.15.60:0/3186309088 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb88401b060 con 0x7fb88c0af130 2022-06-29T16:06:36.254 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.241+0000 7fb8897fa700 1 --2- 172.21.15.60:0/3186309088 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb870053a10 0x7fb870055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:36.254 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.245+0000 7fb8897fa700 1 -- 172.21.15.60:0/3186309088 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb88406b8f0 con 0x7fb88c0af130 2022-06-29T16:06:36.255 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.245+0000 7fb88bfff700 1 --2- 172.21.15.60:0/3186309088 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb870053a10 0x7fb870055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:36.255 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.245+0000 7fb88bfff700 1 --2- 172.21.15.60:0/3186309088 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb870053a10 0x7fb870055ed0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7fb87401a000 tx=0x7fb874015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:36.255 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.249+0000 7fb8897fa700 1 -- 172.21.15.60:0/3186309088 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb884042680 con 0x7fb88c0af130 2022-06-29T16:06:36.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.377+0000 7fb891526700 1 -- 172.21.15.60:0/3186309088 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb88c0b0cf0 con 0x7fb88c0af130 2022-06-29T16:06:36.382 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.377+0000 7fb8897fa700 1 -- 172.21.15.60:0/3186309088 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v110) v1 ==== 100+0+10 (secure 0 0 0) 0x7fb88c0b0cf0 con 0x7fb88c0af130 2022-06-29T16:06:36.383 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:06:36.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.377+0000 7fb891526700 1 -- 172.21.15.60:0/3186309088 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb870053a10 msgr2=0x7fb870055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:36.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.377+0000 7fb891526700 1 --2- 172.21.15.60:0/3186309088 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb870053a10 0x7fb870055ed0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7fb87401a000 tx=0x7fb874015000).stop 2022-06-29T16:06:36.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.377+0000 7fb891526700 1 -- 172.21.15.60:0/3186309088 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb88c0af130 msgr2=0x7fb88c155f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:36.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.377+0000 7fb891526700 1 --2- 172.21.15.60:0/3186309088 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb88c0af130 0x7fb88c155f00 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 rx=0x7fb884001250 tx=0x7fb884007c20).stop 2022-06-29T16:06:36.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.381+0000 7fb891526700 1 -- 172.21.15.60:0/3186309088 shutdown_connections 2022-06-29T16:06:36.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.381+0000 7fb891526700 1 --2- 172.21.15.60:0/3186309088 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb870053a10 0x7fb870055ed0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:36.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.381+0000 7fb891526700 1 --2- 172.21.15.60:0/3186309088 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb88c0af130 0x7fb88c155f00 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:36.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.381+0000 7fb891526700 1 --2- 172.21.15.60:0/3186309088 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb88c0b48b0 0x7fb88c155a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:36.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.381+0000 7fb891526700 1 --2- 172.21.15.60:0/3186309088 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb88c0b2c50 0x7fb88c0aec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:36.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.381+0000 7fb891526700 1 -- 172.21.15.60:0/3186309088 >> 172.21.15.60:0/3186309088 conn(0x7fb88c062090 msgr2=0x7fb88c0b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:36.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.381+0000 7fb891526700 1 -- 172.21.15.60:0/3186309088 shutdown_connections 2022-06-29T16:06:36.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.381+0000 7fb891526700 1 -- 172.21.15.60:0/3186309088 wait complete. 2022-06-29T16:06:36.395 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-06-29T16:06:36.396 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:06:36.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.493+0000 7f3e4f44b700 1 Processor -- start 2022-06-29T16:06:36.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.493+0000 7f3e4f44b700 1 -- start start 2022-06-29T16:06:36.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.497+0000 7f3e4f44b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e48063ad0 0x7f3e48067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:36.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.497+0000 7f3e4f44b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3e4814af10 0x7f3e4805c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:36.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.497+0000 7f3e4f44b700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e480acba0 con 0x7f3e480b5950 2022-06-29T16:06:36.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.497+0000 7f3e4f44b700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4814b3a0 con 0x7f3e4814af10 2022-06-29T16:06:36.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.497+0000 7f3e4f44b700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e48067220 con 0x7f3e48063ad0 2022-06-29T16:06:36.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.497+0000 7f3e4d9e8700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3e4814af10 0x7f3e4805c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:36.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.497+0000 7f3e4d9e8700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3e4814af10 0x7f3e4805c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52340/0 (socket says 172.21.15.60:52340) 2022-06-29T16:06:36.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.497+0000 7f3e4d9e8700 1 -- 172.21.15.60:0/3469695160 learned_addr learned my addr 172.21.15.60:0/3469695160 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:36.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.497+0000 7f3e4d1e7700 1 --1- 172.21.15.60:0/3469695160 >> v1:172.21.15.60:6789/0 conn(0x7f3e480b5950 0x7f3e4805bd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60422/0 (socket says 172.21.15.60:60422) 2022-06-29T16:06:36.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.497+0000 7f3e4c9e6700 1 --2- 172.21.15.60:0/3469695160 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e48063ad0 0x7f3e48067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:36.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.497+0000 7f3e4d9e8700 1 -- 172.21.15.60:0/3469695160 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e48063ad0 msgr2=0x7f3e48067680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:36.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.497+0000 7f3e4d9e8700 1 --2- 172.21.15.60:0/3469695160 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e48063ad0 0x7f3e48067680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:36.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.497+0000 7f3e4d9e8700 1 -- 172.21.15.60:0/3469695160 >> v1:172.21.15.60:6789/0 conn(0x7f3e480b5950 legacy=0x7f3e4805bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:36.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.497+0000 7f3e4d9e8700 1 -- 172.21.15.60:0/3469695160 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e480b5d30 con 0x7f3e4814af10 2022-06-29T16:06:36.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.497+0000 7f3e4d9e8700 1 --2- 172.21.15.60:0/3469695160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e4814af10 0x7f3e4805c430 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 rx=0x7f3e44001110 tx=0x7f3e40009290).ready entity=mon.0 client_cookie=dcbc052084fc8fcd server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:36.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.497+0000 7f3e3bfff700 1 -- 172.21.15.60:0/3469695160 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e4000b760 con 0x7f3e4814af10 2022-06-29T16:06:36.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.497+0000 7f3e3bfff700 1 -- 172.21.15.60:0/3469695160 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e4000b8c0 con 0x7f3e4814af10 2022-06-29T16:06:36.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.497+0000 7f3e3bfff700 1 -- 172.21.15.60:0/3469695160 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e4000ba30 con 0x7f3e4814af10 2022-06-29T16:06:36.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.497+0000 7f3e4f44b700 1 -- 172.21.15.60:0/3469695160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e4814af10 msgr2=0x7f3e4805c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:36.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.497+0000 7f3e4f44b700 1 --2- 172.21.15.60:0/3469695160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e4814af10 0x7f3e4805c430 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 rx=0x7f3e44001110 tx=0x7f3e40009290).stop 2022-06-29T16:06:36.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.497+0000 7f3e4f44b700 1 -- 172.21.15.60:0/3469695160 shutdown_connections 2022-06-29T16:06:36.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.497+0000 7f3e4f44b700 1 --2- 172.21.15.60:0/3469695160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e4814af10 0x7f3e4805c430 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:36.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.497+0000 7f3e4f44b700 1 --2- 172.21.15.60:0/3469695160 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e48063ad0 0x7f3e48067680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:36.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.497+0000 7f3e4f44b700 1 -- 172.21.15.60:0/3469695160 >> 172.21.15.60:0/3469695160 conn(0x7f3e480632c0 msgr2=0x7f3e480b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:36.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.497+0000 7f3e4f44b700 1 -- 172.21.15.60:0/3469695160 shutdown_connections 2022-06-29T16:06:36.510 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.497+0000 7f3e4f44b700 1 -- 172.21.15.60:0/3469695160 wait complete. 2022-06-29T16:06:36.510 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.497+0000 7f3e4f44b700 1 Processor -- start 2022-06-29T16:06:36.510 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.497+0000 7f3e4f44b700 1 -- start start 2022-06-29T16:06:36.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.501+0000 7f3e4f44b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e48063ad0 0x7f3e48116ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:36.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.501+0000 7f3e4f44b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e480b5950 0x7f3e48117190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:36.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.501+0000 7f3e4f44b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e4814af10 0x7f3e48142b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:36.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.501+0000 7f3e4f44b700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e480ad950 con 0x7f3e48063ad0 2022-06-29T16:06:36.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.501+0000 7f3e4f44b700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e48156810 con 0x7f3e480b5950 2022-06-29T16:06:36.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.501+0000 7f3e4f44b700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e48156a80 con 0x7f3e4814af10 2022-06-29T16:06:36.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.501+0000 7f3e4d1e7700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e48063ad0 0x7f3e48116ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:36.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.501+0000 7f3e4d1e7700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e48063ad0 0x7f3e48116ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52344/0 (socket says 172.21.15.60:52344) 2022-06-29T16:06:36.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.501+0000 7f3e4d1e7700 1 -- 172.21.15.60:0/4249201888 learned_addr learned my addr 172.21.15.60:0/4249201888 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:36.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.501+0000 7f3e4c9e6700 1 --2- 172.21.15.60:0/4249201888 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e480b5950 0x7f3e48117190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:36.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.501+0000 7f3e4d9e8700 1 --2- 172.21.15.60:0/4249201888 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e4814af10 0x7f3e48142b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:36.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.501+0000 7f3e4d1e7700 1 -- 172.21.15.60:0/4249201888 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e4814af10 msgr2=0x7f3e48142b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:36.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.501+0000 7f3e4d1e7700 1 --2- 172.21.15.60:0/4249201888 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e4814af10 0x7f3e48142b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:36.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.501+0000 7f3e4d1e7700 1 -- 172.21.15.60:0/4249201888 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e480b5950 msgr2=0x7f3e48117190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:36.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.501+0000 7f3e4d1e7700 1 --2- 172.21.15.60:0/4249201888 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e480b5950 0x7f3e48117190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:36.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.501+0000 7f3e4d1e7700 1 -- 172.21.15.60:0/4249201888 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e40009530 con 0x7f3e48063ad0 2022-06-29T16:06:36.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.501+0000 7f3e4d1e7700 1 --2- 172.21.15.60:0/4249201888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e48063ad0 0x7f3e48116ca0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 rx=0x7f3e481569a0 tx=0x7f3e44007bc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:36.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.501+0000 7f3e3a7fc700 1 -- 172.21.15.60:0/4249201888 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e4400ce90 con 0x7f3e48063ad0 2022-06-29T16:06:36.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.501+0000 7f3e3a7fc700 1 -- 172.21.15.60:0/4249201888 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e440072e0 con 0x7f3e48063ad0 2022-06-29T16:06:36.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.501+0000 7f3e4f44b700 1 -- 172.21.15.60:0/4249201888 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e480add80 con 0x7f3e48063ad0 2022-06-29T16:06:36.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.501+0000 7f3e3a7fc700 1 -- 172.21.15.60:0/4249201888 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e44005740 con 0x7f3e48063ad0 2022-06-29T16:06:36.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.501+0000 7f3e4f44b700 1 -- 172.21.15.60:0/4249201888 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e480ad4b0 con 0x7f3e48063ad0 2022-06-29T16:06:36.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.501+0000 7f3e4f44b700 1 -- 172.21.15.60:0/4249201888 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e48156c40 con 0x7f3e48063ad0 2022-06-29T16:06:36.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.505+0000 7f3e3a7fc700 1 -- 172.21.15.60:0/4249201888 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3e44010070 con 0x7f3e48063ad0 2022-06-29T16:06:36.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.505+0000 7f3e3a7fc700 1 --2- 172.21.15.60:0/4249201888 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3e300539e0 0x7f3e30055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:36.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.505+0000 7f3e3a7fc700 1 -- 172.21.15.60:0/4249201888 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3e4406b7b0 con 0x7f3e48063ad0 2022-06-29T16:06:36.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.505+0000 7f3e4c9e6700 1 --2- 172.21.15.60:0/4249201888 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3e300539e0 0x7f3e30055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:36.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.505+0000 7f3e4c9e6700 1 --2- 172.21.15.60:0/4249201888 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3e300539e0 0x7f3e30055ea0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0x7f3e3c00b000 tx=0x7f3e3c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:36.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.513+0000 7f3e3a7fc700 1 -- 172.21.15.60:0/4249201888 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3e48156c40 con 0x7f3e48063ad0 2022-06-29T16:06:36.682 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.677+0000 7f3e4f44b700 1 -- 172.21.15.60:0/4249201888 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e48156c40 con 0x7f3e48063ad0 2022-06-29T16:06:36.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.677+0000 7f3e3a7fc700 1 -- 172.21.15.60:0/4249201888 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+2142 (secure 0 0 0) 0x7f3e48156c40 con 0x7f3e48063ad0 2022-06-29T16:06:36.683 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:36.684 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":51,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7648,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/3387042465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3387042465},{"type":"v1","addr":"172.21.15.92:6835","nonce":3387042465}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:06:28.249737+0000","modified":"2022-06-29T16:06:29.252773+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7668},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7668":{"gid":7668,"name":"a","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/2749128920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2749128920},{"type":"v1","addr":"172.21.15.60:6833","nonce":2749128920}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2022-06-29T16:06:36.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.681+0000 7f3e4f44b700 1 -- 172.21.15.60:0/4249201888 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3e300539e0 msgr2=0x7f3e30055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:36.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.681+0000 7f3e4f44b700 1 --2- 172.21.15.60:0/4249201888 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3e300539e0 0x7f3e30055ea0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0x7f3e3c00b000 tx=0x7f3e3c006df0).stop 2022-06-29T16:06:36.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.681+0000 7f3e4f44b700 1 -- 172.21.15.60:0/4249201888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e48063ad0 msgr2=0x7f3e48116ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:36.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.681+0000 7f3e4f44b700 1 --2- 172.21.15.60:0/4249201888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e48063ad0 0x7f3e48116ca0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 rx=0x7f3e481569a0 tx=0x7f3e44007bc0).stop 2022-06-29T16:06:36.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.681+0000 7f3e4f44b700 1 -- 172.21.15.60:0/4249201888 shutdown_connections 2022-06-29T16:06:36.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.681+0000 7f3e4f44b700 1 --2- 172.21.15.60:0/4249201888 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3e300539e0 0x7f3e30055ea0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:36.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.681+0000 7f3e4f44b700 1 --2- 172.21.15.60:0/4249201888 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e4814af10 0x7f3e48142b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:36.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.681+0000 7f3e4f44b700 1 --2- 172.21.15.60:0/4249201888 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e480b5950 0x7f3e48117190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:36.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.681+0000 7f3e4f44b700 1 --2- 172.21.15.60:0/4249201888 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e48063ad0 0x7f3e48116ca0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:36.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.681+0000 7f3e4f44b700 1 -- 172.21.15.60:0/4249201888 >> 172.21.15.60:0/4249201888 conn(0x7f3e480632c0 msgr2=0x7f3e48146f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:36.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.681+0000 7f3e4f44b700 1 -- 172.21.15.60:0/4249201888 shutdown_connections 2022-06-29T16:06:36.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:36.681+0000 7f3e4f44b700 1 -- 172.21.15.60:0/4249201888 wait complete. 2022-06-29T16:06:36.688 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 51 2022-06-29T16:06:36.898 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 51, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:06:28.249737+0000', 'modified': '2022-06-29T16:06:29.252773+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7668}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7668': {'gid': 7668, 'name': 'a', 'rank': 0, 'incarnation': 50, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.60:6833/2749128920', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6832', 'nonce': 2749128920}, {'type': 'v1', 'addr': '172.21.15.60:6833', 'nonce': 2749128920}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:06:36.899 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:06:36.899 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-06-29T16:06:37.021 INFO:teuthology.orchestra.run.smithi060.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":7668,"want_state":"up:active","state":"up:active","rank_uptime":8.7555328879999994,"mdsmap_epoch":51,"osdmap_epoch":110,"osdmap_epoch_barrier":109,"uptime":12.900222021999999} 2022-06-29T16:06:37.022 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 7668, "mdsmap_epoch": 51, "osdmap_epoch": 110, "osdmap_epoch_barrier": 109, "rank_uptime": 8.755532888, "state": "up:active", "uptime": 12.900222022, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:06:37.022 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-06-29T16:06:37.023 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-06-29T16:06:37.023 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi156.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-06-29T16:06:37.023 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:37.024 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:06:37.032 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2022-06-29T16:06:37.092 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:06:37.093 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:06:37.108 INFO:teuthology.orchestra.run.smithi156.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:06:37.109 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:06:37.110 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:37.111 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2022-06-29T16:06:37.116 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-06-29T16:06:37.116 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:06:37.119 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2022-06-29T16:06:37.133 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:06:37.134 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:06:37.147 INFO:teuthology.orchestra.run.smithi156.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:06:37.148 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:06:37.149 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:37.150 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2022-06-29T16:06:37.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:2022-06-29T16:06:37.182+0000 7fe03096f080 -1 init, newargv = 0x55ac94866cf0 newargc=9ceph-fuse[12736]: starting ceph client 2022-06-29T16:06:37.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr: 2022-06-29T16:06:37.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[12736]: starting fuse 2022-06-29T16:06:38.297 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2022-06-29T16:06:38.316 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:06:38.316 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:06:38.374 INFO:teuthology.orchestra.run.smithi156.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:06:38.375 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:06:38.376 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:38.377 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2022-06-29T16:06:38.382 INFO:teuthology.orchestra.run.smithi156.stdout:52 2022-06-29T16:06:38.382 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-06-29T16:06:38.383 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:38.383 DEBUG:teuthology.orchestra.run.smithi156:> sudo python3 -c ' 2022-06-29T16:06:38.384 DEBUG:teuthology.orchestra.run.smithi156:> import glob 2022-06-29T16:06:38.384 DEBUG:teuthology.orchestra.run.smithi156:> import re 2022-06-29T16:06:38.384 DEBUG:teuthology.orchestra.run.smithi156:> import os 2022-06-29T16:06:38.385 DEBUG:teuthology.orchestra.run.smithi156:> import subprocess 2022-06-29T16:06:38.385 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:06:38.386 DEBUG:teuthology.orchestra.run.smithi156:> def find_socket(client_name): 2022-06-29T16:06:38.386 DEBUG:teuthology.orchestra.run.smithi156:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-06-29T16:06:38.387 DEBUG:teuthology.orchestra.run.smithi156:> files = glob.glob(asok_path) 2022-06-29T16:06:38.387 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:06:38.387 DEBUG:teuthology.orchestra.run.smithi156:> # Given a non-glob path, it better be there 2022-06-29T16:06:38.388 DEBUG:teuthology.orchestra.run.smithi156:> if "*" not in asok_path: 2022-06-29T16:06:38.388 DEBUG:teuthology.orchestra.run.smithi156:> assert(len(files) == 1) 2022-06-29T16:06:38.389 DEBUG:teuthology.orchestra.run.smithi156:> return files[0] 2022-06-29T16:06:38.389 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:06:38.390 DEBUG:teuthology.orchestra.run.smithi156:> for f in files: 2022-06-29T16:06:38.391 DEBUG:teuthology.orchestra.run.smithi156:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-06-29T16:06:38.391 DEBUG:teuthology.orchestra.run.smithi156:> if os.path.exists("/proc/{0}".format(pid)): 2022-06-29T16:06:38.391 DEBUG:teuthology.orchestra.run.smithi156:> return f 2022-06-29T16:06:38.392 DEBUG:teuthology.orchestra.run.smithi156:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-06-29T16:06:38.392 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:06:38.393 DEBUG:teuthology.orchestra.run.smithi156:> print(find_socket("client.1")) 2022-06-29T16:06:38.393 DEBUG:teuthology.orchestra.run.smithi156:> ' 2022-06-29T16:06:38.438 INFO:teuthology.orchestra.run.smithi156.stdout:/var/run/ceph/ceph-client.1.12736.asok 2022-06-29T16:06:38.443 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.12736.asok 2022-06-29T16:06:38.444 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:38.444 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.12736.asok status 2022-06-29T16:06:38.581 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2022-06-29T16:06:38.581 INFO:teuthology.orchestra.run.smithi156.stdout: "metadata": { 2022-06-29T16:06:38.582 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_sha1": "f25ffac72e5a9e63860ff7deb6767a11f0559a16", 2022-06-29T16:06:38.582 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_version": "ceph version 15.2.16-304-gf25ffac7 (f25ffac72e5a9e63860ff7deb6767a11f0559a16) octopus (stable)", 2022-06-29T16:06:38.582 INFO:teuthology.orchestra.run.smithi156.stdout: "entity_id": "1", 2022-06-29T16:06:38.582 INFO:teuthology.orchestra.run.smithi156.stdout: "hostname": "smithi156", 2022-06-29T16:06:38.583 INFO:teuthology.orchestra.run.smithi156.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-06-29T16:06:38.583 INFO:teuthology.orchestra.run.smithi156.stdout: "pid": "12736", 2022-06-29T16:06:38.583 INFO:teuthology.orchestra.run.smithi156.stdout: "root": "/" 2022-06-29T16:06:38.584 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:06:38.584 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_count": 0, 2022-06-29T16:06:38.584 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_pinned_count": 0, 2022-06-29T16:06:38.584 INFO:teuthology.orchestra.run.smithi156.stdout: "id": 7890, 2022-06-29T16:06:38.585 INFO:teuthology.orchestra.run.smithi156.stdout: "inst": { 2022-06-29T16:06:38.585 INFO:teuthology.orchestra.run.smithi156.stdout: "name": { 2022-06-29T16:06:38.585 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "client", 2022-06-29T16:06:38.585 INFO:teuthology.orchestra.run.smithi156.stdout: "num": 7890 2022-06-29T16:06:38.585 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:06:38.586 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2022-06-29T16:06:38.586 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2022-06-29T16:06:38.586 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "172.21.15.156:0", 2022-06-29T16:06:38.586 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 29218178 2022-06-29T16:06:38.587 INFO:teuthology.orchestra.run.smithi156.stdout: } 2022-06-29T16:06:38.587 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:06:38.587 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2022-06-29T16:06:38.587 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2022-06-29T16:06:38.588 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "172.21.15.156:0", 2022-06-29T16:06:38.588 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 29218178 2022-06-29T16:06:38.588 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:06:38.588 INFO:teuthology.orchestra.run.smithi156.stdout: "inst_str": "client.7890 172.21.15.156:0/29218178", 2022-06-29T16:06:38.589 INFO:teuthology.orchestra.run.smithi156.stdout: "addr_str": "172.21.15.156:0/29218178", 2022-06-29T16:06:38.589 INFO:teuthology.orchestra.run.smithi156.stdout: "inode_count": 1, 2022-06-29T16:06:38.589 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_epoch": 51, 2022-06-29T16:06:38.589 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch": 110, 2022-06-29T16:06:38.590 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch_barrier": 0, 2022-06-29T16:06:38.590 INFO:teuthology.orchestra.run.smithi156.stdout: "blacklisted": false, 2022-06-29T16:06:38.590 INFO:teuthology.orchestra.run.smithi156.stdout: "fs_name": "cephfs" 2022-06-29T16:06:38.590 INFO:teuthology.orchestra.run.smithi156.stdout:} 2022-06-29T16:06:38.592 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:38.592 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:06:38.593 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:06:38.600 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2022-06-29T16:06:38.601 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-06-29T16:06:38.602 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:38.602 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:06:38.619 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:38.620 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:06:38.620 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:06:38.629 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T16:06:38.630 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T16:06:38.631 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:38.631 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:06:38.631 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:06:38.640 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T16:06:38.641 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T16:06:38.641 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2022-06-29T16:06:38.642 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-06-29T16:06:38.642 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:06:38.655 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[13314]: fuse finished with error 0 and tester_r 0 2022-06-29T16:06:38.657 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:38.657 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:06:38.658 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:06:38.705 INFO:teuthology.orchestra.run.smithi121.stdout:ext2/ext3 2022-06-29T16:06:38.707 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-06-29T16:06:38.707 INFO:teuthology.orchestra.run:waiting for 900 2022-06-29T16:06:44.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:06:44.809 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:06:44.817 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:44.818 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:06:44.818 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:06:44.826 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2022-06-29T16:06:44.828 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-06-29T16:06:44.828 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:44.829 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:06:44.829 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:06:44.836 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2022-06-29T16:06:44.837 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-06-29T16:06:44.838 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi156.front.sepia.ceph.com... 2022-06-29T16:06:44.838 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-06-29T16:06:44.838 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:06:44.853 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[12736]: fuse finished with error 0 and tester_r 0 2022-06-29T16:06:44.854 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:06:44.855 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:06:44.855 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:06:44.862 INFO:teuthology.orchestra.run.smithi156.stdout:ext2/ext3 2022-06-29T16:06:44.863 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-06-29T16:06:44.864 INFO:teuthology.orchestra.run:waiting for 900 2022-06-29T16:06:50.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:06:50.967 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:06:50.975 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-06-29T16:06:50.975 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T16:06:50.976 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-06-29T16:06:50.976 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T16:06:50.991 INFO:tasks.ceph.mds.a:Stopped 2022-06-29T16:06:50.992 DEBUG:teuthology.parallel:result is None 2022-06-29T16:06:50.993 INFO:tasks.ceph.mds.b:Stopped 2022-06-29T16:06:50.993 DEBUG:teuthology.parallel:result is None 2022-06-29T16:06:50.994 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2022-06-29T16:06:50.994 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2022-06-29T16:06:51.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.093+0000 7f519f29e700 1 Processor -- start 2022-06-29T16:06:51.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.093+0000 7f519f29e700 1 -- start start 2022-06-29T16:06:51.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.093+0000 7f519f29e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f519805f0b0 0x7f51981461e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:51.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.093+0000 7f519f29e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f519805ca50 0x7f519805ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:51.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.093+0000 7f519f29e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51980b6e30 con 0x7f51980b6a50 2022-06-29T16:06:51.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.093+0000 7f519f29e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51980b0dd0 con 0x7f519805f0b0 2022-06-29T16:06:51.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.093+0000 7f519f29e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51980b1070 con 0x7f519805ca50 2022-06-29T16:06:51.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.093+0000 7f519c839700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f519805f0b0 0x7f51981461e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:51.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.093+0000 7f519c839700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f519805f0b0 0x7f51981461e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52354/0 (socket says 172.21.15.60:52354) 2022-06-29T16:06:51.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.093+0000 7f519c839700 1 -- 172.21.15.60:0/1419917330 learned_addr learned my addr 172.21.15.60:0/1419917330 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:51.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f519d83b700 1 --2- 172.21.15.60:0/1419917330 >> v2:172.21.15.92:3300/0 conn(0x7f519805ca50 0x7f519805ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:51.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f519c839700 1 -- 172.21.15.60:0/1419917330 >> v2:172.21.15.92:3300/0 conn(0x7f519805ca50 msgr2=0x7f519805ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:51.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f519c839700 1 --2- 172.21.15.60:0/1419917330 >> v2:172.21.15.92:3300/0 conn(0x7f519805ca50 0x7f519805ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:51.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f519c839700 1 -- 172.21.15.60:0/1419917330 >> v1:172.21.15.92:6789/0 conn(0x7f51980b6a50 legacy=0x7f5198142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:51.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f519c839700 1 -- 172.21.15.60:0/1419917330 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f519805c5b0 con 0x7f519805f0b0 2022-06-29T16:06:51.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f519c839700 1 --2- 172.21.15.60:0/1419917330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f519805f0b0 0x7f51981461e0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 rx=0x7f518c002f70 tx=0x7f518c006eb0).ready entity=mon.0 client_cookie=c5b2238e4db34c52 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:51.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f5197fff700 1 -- 172.21.15.60:0/1419917330 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f518c00d610 con 0x7f519805f0b0 2022-06-29T16:06:51.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f5197fff700 1 -- 172.21.15.60:0/1419917330 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f518c000e90 con 0x7f519805f0b0 2022-06-29T16:06:51.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f5197fff700 1 -- 172.21.15.60:0/1419917330 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f518c00d9a0 con 0x7f519805f0b0 2022-06-29T16:06:51.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f38be80d700 1 Processor -- start 2022-06-29T16:06:51.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f38be80d700 1 -- start start 2022-06-29T16:06:51.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f519f29e700 1 -- 172.21.15.60:0/1419917330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f519805f0b0 msgr2=0x7f51981461e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:51.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f519f29e700 1 --2- 172.21.15.60:0/1419917330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f519805f0b0 0x7f51981461e0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 rx=0x7f518c002f70 tx=0x7f518c006eb0).stop 2022-06-29T16:06:51.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f519f29e700 1 -- 172.21.15.60:0/1419917330 shutdown_connections 2022-06-29T16:06:51.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f519f29e700 1 --2- 172.21.15.60:0/1419917330 >> v2:172.21.15.92:3300/0 conn(0x7f519805ca50 0x7f519805ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:51.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f519f29e700 1 --2- 172.21.15.60:0/1419917330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f519805f0b0 0x7f51981461e0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:51.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f519f29e700 1 -- 172.21.15.60:0/1419917330 >> 172.21.15.60:0/1419917330 conn(0x7f51980643e0 msgr2=0x7f51980b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:51.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f519f29e700 1 -- 172.21.15.60:0/1419917330 shutdown_connections 2022-06-29T16:06:51.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f519f29e700 1 -- 172.21.15.60:0/1419917330 wait complete. 2022-06-29T16:06:51.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f38be80d700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f38b813f3d0 0x7f38b8060260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:51.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f38be80d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38b813f7b0 0x7f38b813fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:51.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f38be80d700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38b80b6e20 con 0x7f38b8143820 2022-06-29T16:06:51.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f38be80d700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38b8142ef0 con 0x7f38b813f3d0 2022-06-29T16:06:51.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f38be80d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38b80acb90 con 0x7f38b813f7b0 2022-06-29T16:06:51.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f38affff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f38b8143820 0x7f38b80b6710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60446/0 (socket says 172.21.15.60:60446) 2022-06-29T16:06:51.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f38affff700 1 -- 172.21.15.60:0/393887324 learned_addr learned my addr 172.21.15.60:0/393887324 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:51.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f38b7fff700 1 --2- 172.21.15.60:0/393887324 >> v2:172.21.15.92:3300/0 conn(0x7f38b813f3d0 0x7f38b8060260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:51.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f38bcdaa700 1 --2- 172.21.15.60:0/393887324 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38b813f7b0 0x7f38b813fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:51.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f38b77fe700 1 -- 172.21.15.60:0/393887324 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4099302208 0 0) 0x7f38b80b6e20 con 0x7f38b8143820 2022-06-29T16:06:51.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f38b77fe700 1 -- 172.21.15.60:0/393887324 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3898003260 con 0x7f38b8143820 2022-06-29T16:06:51.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f38b7fff700 1 -- 172.21.15.60:0/393887324 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38b813f7b0 msgr2=0x7f38b813fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:51.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f38b7fff700 1 --2- 172.21.15.60:0/393887324 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38b813f7b0 0x7f38b813fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:51.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f38b7fff700 1 -- 172.21.15.60:0/393887324 >> v1:172.21.15.60:6789/0 conn(0x7f38b8143820 legacy=0x7f38b80b6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:51.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f38b7fff700 1 -- 172.21.15.60:0/393887324 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b81430e0 con 0x7f38b813f3d0 2022-06-29T16:06:51.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f38b7fff700 1 --2- 172.21.15.60:0/393887324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38b813f3d0 0x7f38b8060260 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7f38a8002f70 tx=0x7f38a8006df0).ready entity=mon.1 client_cookie=f1da2345891e8cce server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:51.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f519f29e700 1 Processor -- start 2022-06-29T16:06:51.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f519f29e700 1 -- start start 2022-06-29T16:06:51.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f38b77fe700 1 -- 172.21.15.60:0/393887324 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38a800d4a0 con 0x7f38b813f3d0 2022-06-29T16:06:51.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f38b77fe700 1 -- 172.21.15.60:0/393887324 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a8000e90 con 0x7f38b813f3d0 2022-06-29T16:06:51.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.097+0000 7f38b77fe700 1 -- 172.21.15.60:0/393887324 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38a800d7f0 con 0x7f38b813f3d0 2022-06-29T16:06:51.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f38be80d700 1 -- 172.21.15.60:0/393887324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38b813f3d0 msgr2=0x7f38b8060260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:51.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f38be80d700 1 --2- 172.21.15.60:0/393887324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38b813f3d0 0x7f38b8060260 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7f38a8002f70 tx=0x7f38a8006df0).stop 2022-06-29T16:06:51.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f38be80d700 1 -- 172.21.15.60:0/393887324 shutdown_connections 2022-06-29T16:06:51.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f38be80d700 1 --2- 172.21.15.60:0/393887324 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38b813f7b0 0x7f38b813fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:51.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f38be80d700 1 --2- 172.21.15.60:0/393887324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38b813f3d0 0x7f38b8060260 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:51.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f38be80d700 1 -- 172.21.15.60:0/393887324 >> 172.21.15.60:0/393887324 conn(0x7f38b80643e0 msgr2=0x7f38b80b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:51.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f38be80d700 1 -- 172.21.15.60:0/393887324 shutdown_connections 2022-06-29T16:06:51.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f38be80d700 1 -- 172.21.15.60:0/393887324 wait complete. 2022-06-29T16:06:51.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f519f29e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f519805ca50 0x7f5198144af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:51.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f519f29e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f519805f0b0 0x7f51980aed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:51.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f519f29e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f51980b6a50 0x7f5198155a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:51.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f519f29e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51980b53c0 con 0x7f519805f0b0 2022-06-29T16:06:51.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f519f29e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51980b5660 con 0x7f51980b6a50 2022-06-29T16:06:51.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f519f29e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51980b5900 con 0x7f519805ca50 2022-06-29T16:06:51.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f519c839700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f519805f0b0 0x7f51980aed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:51.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f519c839700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f519805f0b0 0x7f51980aed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52364/0 (socket says 172.21.15.60:52364) 2022-06-29T16:06:51.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f519c839700 1 -- 172.21.15.60:0/244277719 learned_addr learned my addr 172.21.15.60:0/244277719 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:51.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f519d03a700 1 --2- 172.21.15.60:0/244277719 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f519805ca50 0x7f5198144af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:51.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f519d83b700 1 --2- 172.21.15.60:0/244277719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f51980b6a50 0x7f5198155a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:51.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f519c839700 1 -- 172.21.15.60:0/244277719 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f519805ca50 msgr2=0x7f5198144af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:51.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f519c839700 1 --2- 172.21.15.60:0/244277719 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f519805ca50 0x7f5198144af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:51.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f519c839700 1 -- 172.21.15.60:0/244277719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f51980b6a50 msgr2=0x7f5198155a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:51.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f519c839700 1 --2- 172.21.15.60:0/244277719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f51980b6a50 0x7f5198155a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:51.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f519c839700 1 -- 172.21.15.60:0/244277719 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f518c00b1e0 con 0x7f519805f0b0 2022-06-29T16:06:51.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f519d83b700 1 --2- 172.21.15.60:0/244277719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f51980b6a50 0x7f5198155a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:06:51.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f519c839700 1 --2- 172.21.15.60:0/244277719 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f519805f0b0 0x7f51980aed80 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 rx=0x7f518c002f70 tx=0x7f518c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:51.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f51967fc700 1 -- 172.21.15.60:0/244277719 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f518c006b40 con 0x7f519805f0b0 2022-06-29T16:06:51.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f51967fc700 1 -- 172.21.15.60:0/244277719 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f518c0068b0 con 0x7f519805f0b0 2022-06-29T16:06:51.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f519f29e700 1 -- 172.21.15.60:0/244277719 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51980b0360 con 0x7f519805f0b0 2022-06-29T16:06:51.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f51967fc700 1 -- 172.21.15.60:0/244277719 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f518c011e20 con 0x7f519805f0b0 2022-06-29T16:06:51.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f519f29e700 1 -- 172.21.15.60:0/244277719 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5198155590 con 0x7f519805f0b0 2022-06-29T16:06:51.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f38be80d700 1 Processor -- start 2022-06-29T16:06:51.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f38be80d700 1 -- start start 2022-06-29T16:06:51.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f519f29e700 1 -- 172.21.15.60:0/244277719 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51980b0040 con 0x7f519805f0b0 2022-06-29T16:06:51.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f38be80d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38b813f3d0 0x7f38b8061750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:51.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f38be80d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38b813f7b0 0x7f38b80af7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:51.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f38be80d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38b8143820 0x7f38b80afc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:51.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f38be80d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b8066630 con 0x7f38b813f7b0 2022-06-29T16:06:51.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f38be80d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b80660f0 con 0x7f38b813f3d0 2022-06-29T16:06:51.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f38be80d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38b8066390 con 0x7f38b8143820 2022-06-29T16:06:51.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f38affff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38b813f7b0 0x7f38b80af7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:51.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f38affff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38b813f7b0 0x7f38b80af7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52370/0 (socket says 172.21.15.60:52370) 2022-06-29T16:06:51.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f38affff700 1 -- 172.21.15.60:0/1725516474 learned_addr learned my addr 172.21.15.60:0/1725516474 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:51.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f38bcdaa700 1 --2- 172.21.15.60:0/1725516474 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38b8143820 0x7f38b80afc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:51.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f38b7fff700 1 --2- 172.21.15.60:0/1725516474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38b813f3d0 0x7f38b8061750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:51.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f51967fc700 1 -- 172.21.15.60:0/244277719 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f518c025030 con 0x7f519805f0b0 2022-06-29T16:06:51.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f38affff700 1 -- 172.21.15.60:0/1725516474 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38b8143820 msgr2=0x7f38b80afc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:51.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f38affff700 1 --2- 172.21.15.60:0/1725516474 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38b8143820 0x7f38b80afc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:51.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f38affff700 1 -- 172.21.15.60:0/1725516474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38b813f3d0 msgr2=0x7f38b8061750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:51.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f38affff700 1 --2- 172.21.15.60:0/1725516474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38b813f3d0 0x7f38b8061750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:51.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f38affff700 1 -- 172.21.15.60:0/1725516474 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a800b180 con 0x7f38b813f7b0 2022-06-29T16:06:51.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.101+0000 7f38b7fff700 1 --2- 172.21.15.60:0/1725516474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38b813f3d0 0x7f38b8061750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:06:51.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.105+0000 7f38affff700 1 --2- 172.21.15.60:0/1725516474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38b813f7b0 0x7f38b80af7a0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 rx=0x7f38a0001110 tx=0x7f38a0007aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:51.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.105+0000 7f51967fc700 1 --2- 172.21.15.60:0/244277719 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5178053b00 0x7f5178055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:51.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.105+0000 7f51967fc700 1 -- 172.21.15.60:0/244277719 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f518c06d160 con 0x7f519805f0b0 2022-06-29T16:06:51.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.105+0000 7f519d03a700 1 --2- 172.21.15.60:0/244277719 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5178053b00 0x7f5178055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:51.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.105+0000 7f519d03a700 1 --2- 172.21.15.60:0/244277719 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5178053b00 0x7f5178055fc0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7f518800d000 tx=0x7f5188007f10).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:51.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.109+0000 7f38b5ffb700 1 -- 172.21.15.60:0/1725516474 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38a00070d0 con 0x7f38b813f7b0 2022-06-29T16:06:51.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.109+0000 7f38b5ffb700 1 -- 172.21.15.60:0/1725516474 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a0007230 con 0x7f38b813f7b0 2022-06-29T16:06:51.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.109+0000 7f38be80d700 1 -- 172.21.15.60:0/1725516474 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38b80ade00 con 0x7f38b813f7b0 2022-06-29T16:06:51.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.109+0000 7f38b5ffb700 1 -- 172.21.15.60:0/1725516474 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38a0005610 con 0x7f38b813f7b0 2022-06-29T16:06:51.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.109+0000 7f38be80d700 1 -- 172.21.15.60:0/1725516474 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38b80b0180 con 0x7f38b813f7b0 2022-06-29T16:06:51.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.109+0000 7f51967fc700 1 -- 172.21.15.60:0/244277719 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f518c047e60 con 0x7f519805f0b0 2022-06-29T16:06:51.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.109+0000 7f38be80d700 1 -- 172.21.15.60:0/1725516474 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38b80573d0 con 0x7f38b813f7b0 2022-06-29T16:06:51.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.109+0000 7f38b5ffb700 1 -- 172.21.15.60:0/1725516474 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f38a0010070 con 0x7f38b813f7b0 2022-06-29T16:06:51.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.113+0000 7f38b5ffb700 1 --2- 172.21.15.60:0/1725516474 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f389c053aa0 0x7f389c055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:51.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.113+0000 7f38b5ffb700 1 -- 172.21.15.60:0/1725516474 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f38a006b500 con 0x7f38b813f7b0 2022-06-29T16:06:51.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.113+0000 7f38b7fff700 1 --2- 172.21.15.60:0/1725516474 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f389c053aa0 0x7f389c055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:51.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.113+0000 7f38b7fff700 1 --2- 172.21.15.60:0/1725516474 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f389c053aa0 0x7f389c055f60 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7f38a8018000 tx=0x7f38a8014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:51.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.117+0000 7f38b5ffb700 1 -- 172.21.15.60:0/1725516474 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f38a0042380 con 0x7f38b813f7b0 2022-06-29T16:06:51.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.281+0000 7f519f29e700 1 -- 172.21.15.60:0/244277719 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7f51980b5ac0 con 0x7f519805f0b0 2022-06-29T16:06:51.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:51.285+0000 7f38be80d700 1 -- 172.21.15.60:0/1725516474 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f38b80af520 con 0x7f38b813f7b0 2022-06-29T16:06:52.161 INFO:tasks.ceph.mon.a.smithi060.stderr:2022-06-29T16:06:52.157+0000 7fcc1a376700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-06-29T16:06:52.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.169+0000 7f51967fc700 1 -- 172.21.15.60:0/244277719 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 7668 v52) v1 ==== 95+0+0 (secure 0 0 0) 0x7f518c023020 con 0x7f519805f0b0 2022-06-29T16:06:52.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.169+0000 7f38b5ffb700 1 -- 172.21.15.60:0/1725516474 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 7648 v52) v1 ==== 95+0+0 (secure 0 0 0) 0x7f38a0018020 con 0x7f38b813f7b0 2022-06-29T16:06:52.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.169+0000 7f38be80d700 1 -- 172.21.15.60:0/1725516474 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f389c053aa0 msgr2=0x7f389c055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:52.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.169+0000 7f38be80d700 1 --2- 172.21.15.60:0/1725516474 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f389c053aa0 0x7f389c055f60 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7f38a8018000 tx=0x7f38a8014000).stop 2022-06-29T16:06:52.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.169+0000 7f38be80d700 1 -- 172.21.15.60:0/1725516474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38b813f7b0 msgr2=0x7f38b80af7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:52.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.169+0000 7f38be80d700 1 --2- 172.21.15.60:0/1725516474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38b813f7b0 0x7f38b80af7a0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 rx=0x7f38a0001110 tx=0x7f38a0007aa0).stop 2022-06-29T16:06:52.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.173+0000 7f38be80d700 1 -- 172.21.15.60:0/1725516474 shutdown_connections 2022-06-29T16:06:52.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.173+0000 7f38be80d700 1 --2- 172.21.15.60:0/1725516474 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f389c053aa0 0x7f389c055f60 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:52.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.173+0000 7f38be80d700 1 --2- 172.21.15.60:0/1725516474 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38b8143820 0x7f38b80afc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:52.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.173+0000 7f38be80d700 1 --2- 172.21.15.60:0/1725516474 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38b813f7b0 0x7f38b80af7a0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:52.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.173+0000 7f38be80d700 1 --2- 172.21.15.60:0/1725516474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38b813f3d0 0x7f38b8061750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:52.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.173+0000 7f38be80d700 1 -- 172.21.15.60:0/1725516474 >> 172.21.15.60:0/1725516474 conn(0x7f38b80643e0 msgr2=0x7f38b80b3180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:52.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.173+0000 7f38be80d700 1 -- 172.21.15.60:0/1725516474 shutdown_connections 2022-06-29T16:06:52.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.173+0000 7f38be80d700 1 -- 172.21.15.60:0/1725516474 wait complete. 2022-06-29T16:06:52.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.173+0000 7f519f29e700 1 -- 172.21.15.60:0/244277719 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5178053b00 msgr2=0x7f5178055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:52.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.173+0000 7f519f29e700 1 --2- 172.21.15.60:0/244277719 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5178053b00 0x7f5178055fc0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7f518800d000 tx=0x7f5188007f10).stop 2022-06-29T16:06:52.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.173+0000 7f519f29e700 1 -- 172.21.15.60:0/244277719 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f519805f0b0 msgr2=0x7f51980aed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:52.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.173+0000 7f519f29e700 1 --2- 172.21.15.60:0/244277719 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f519805f0b0 0x7f51980aed80 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 rx=0x7f518c002f70 tx=0x7f518c015000).stop 2022-06-29T16:06:52.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.173+0000 7f519f29e700 1 -- 172.21.15.60:0/244277719 shutdown_connections 2022-06-29T16:06:52.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.173+0000 7f519f29e700 1 --2- 172.21.15.60:0/244277719 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5178053b00 0x7f5178055fc0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:52.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.173+0000 7f519f29e700 1 --2- 172.21.15.60:0/244277719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f51980b6a50 0x7f5198155a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:52.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.173+0000 7f519f29e700 1 --2- 172.21.15.60:0/244277719 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f519805f0b0 0x7f51980aed80 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:52.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.173+0000 7f519f29e700 1 --2- 172.21.15.60:0/244277719 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f519805ca50 0x7f5198144af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:52.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.173+0000 7f519f29e700 1 -- 172.21.15.60:0/244277719 >> 172.21.15.60:0/244277719 conn(0x7f51980643e0 msgr2=0x7f519813f1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:52.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.173+0000 7f519f29e700 1 -- 172.21.15.60:0/244277719 shutdown_connections 2022-06-29T16:06:52.184 INFO:teuthology.orchestra.run.smithi060.stderr:failed mds gid 7648 2022-06-29T16:06:52.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.177+0000 7f519f29e700 1 -- 172.21.15.60:0/244277719 wait complete. 2022-06-29T16:06:52.185 INFO:teuthology.orchestra.run.smithi060.stderr:failed mds gid 7668 2022-06-29T16:06:52.189 DEBUG:teuthology.parallel:result is '' 2022-06-29T16:06:52.194 DEBUG:teuthology.parallel:result is '' 2022-06-29T16:06:52.195 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-06-29T16:06:52.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.297+0000 7fb38b59e700 1 Processor -- start 2022-06-29T16:06:52.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.297+0000 7fb38b59e700 1 -- start start 2022-06-29T16:06:52.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.301+0000 7fb38b59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb38c05d070 0x7fb38c12b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:52.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.301+0000 7fb38b59e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb38c066b20 0x7fb38c12bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:52.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.301+0000 7fb38b59e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb38c0ae9a0 con 0x7fb38c142ae0 2022-06-29T16:06:52.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.301+0000 7fb38b59e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb38c143b60 con 0x7fb38c066b20 2022-06-29T16:06:52.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.301+0000 7fb38b59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb38c12c0b0 con 0x7fb38c05d070 2022-06-29T16:06:52.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.301+0000 7fb389d9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb38c066b20 0x7fb38c12bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:52.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.301+0000 7fb38ad9d700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb38c142ae0 0x7fb38c142f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60466/0 (socket says 172.21.15.60:60466) 2022-06-29T16:06:52.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.301+0000 7fb38ad9d700 1 -- 172.21.15.60:0/546193700 learned_addr learned my addr 172.21.15.60:0/546193700 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:52.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.301+0000 7fb38a59c700 1 --2- 172.21.15.60:0/546193700 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb38c05d070 0x7fb38c12b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:52.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.301+0000 7fb38959a700 1 -- 172.21.15.60:0/546193700 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169171022 0 0) 0x7fb38c0ae9a0 con 0x7fb38c142ae0 2022-06-29T16:06:52.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.301+0000 7fb38959a700 1 -- 172.21.15.60:0/546193700 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb370003260 con 0x7fb38c142ae0 2022-06-29T16:06:52.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.301+0000 7fb389d9b700 1 -- 172.21.15.60:0/546193700 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb38c05d070 msgr2=0x7fb38c12b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:52.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.301+0000 7fb389d9b700 1 --2- 172.21.15.60:0/546193700 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb38c05d070 0x7fb38c12b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:52.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.301+0000 7fb389d9b700 1 -- 172.21.15.60:0/546193700 >> v1:172.21.15.60:6789/0 conn(0x7fb38c142ae0 legacy=0x7fb38c142f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:52.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.301+0000 7fb389d9b700 1 -- 172.21.15.60:0/546193700 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb38c05d450 con 0x7fb38c066b20 2022-06-29T16:06:52.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.301+0000 7fb38a59c700 1 --2- 172.21.15.60:0/546193700 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb38c05d070 0x7fb38c12b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:06:52.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.301+0000 7fb389d9b700 1 --2- 172.21.15.60:0/546193700 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb38c066b20 0x7fb38c12bbc0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 rx=0x7fb374002f70 tx=0x7fb374006df0).ready entity=mon.0 client_cookie=5fb90a87dbe20c19 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:52.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.301+0000 7fb38959a700 1 -- 172.21.15.60:0/546193700 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb37400d4a0 con 0x7fb38c066b20 2022-06-29T16:06:52.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.301+0000 7fb38959a700 1 -- 172.21.15.60:0/546193700 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb374000e90 con 0x7fb38c066b20 2022-06-29T16:06:52.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.301+0000 7fb38959a700 1 -- 172.21.15.60:0/546193700 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb37400d7f0 con 0x7fb38c066b20 2022-06-29T16:06:52.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.301+0000 7fb38b59e700 1 -- 172.21.15.60:0/546193700 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb38c066b20 msgr2=0x7fb38c12bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:52.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.301+0000 7fb38b59e700 1 --2- 172.21.15.60:0/546193700 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb38c066b20 0x7fb38c12bbc0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 rx=0x7fb374002f70 tx=0x7fb374006df0).stop 2022-06-29T16:06:52.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.301+0000 7fb38b59e700 1 -- 172.21.15.60:0/546193700 shutdown_connections 2022-06-29T16:06:52.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.301+0000 7fb38b59e700 1 --2- 172.21.15.60:0/546193700 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb38c066b20 0x7fb38c12bbc0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:52.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.301+0000 7fb38b59e700 1 --2- 172.21.15.60:0/546193700 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb38c05d070 0x7fb38c12b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:52.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.301+0000 7fb38b59e700 1 -- 172.21.15.60:0/546193700 >> 172.21.15.60:0/546193700 conn(0x7fb38c0b5890 msgr2=0x7fb38c0b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:52.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.301+0000 7fb38b59e700 1 -- 172.21.15.60:0/546193700 shutdown_connections 2022-06-29T16:06:52.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.301+0000 7fb38b59e700 1 -- 172.21.15.60:0/546193700 wait complete. 2022-06-29T16:06:52.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.305+0000 7fb38b59e700 1 Processor -- start 2022-06-29T16:06:52.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.305+0000 7fb38b59e700 1 -- start start 2022-06-29T16:06:52.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.305+0000 7fb38b59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb38c05d070 0x7fb38c128410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:52.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.305+0000 7fb38b59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb38c066b20 0x7fb38c0afd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:52.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.305+0000 7fb38b59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb38c142ae0 0x7fb38c0b0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:52.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.305+0000 7fb38b59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38c130820 con 0x7fb38c142ae0 2022-06-29T16:06:52.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.305+0000 7fb38b59e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38c12cdd0 con 0x7fb38c066b20 2022-06-29T16:06:52.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.305+0000 7fb38b59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb38c12d070 con 0x7fb38c05d070 2022-06-29T16:06:52.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.305+0000 7fb38ad9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb38c142ae0 0x7fb38c0b0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:52.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.305+0000 7fb38a59c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb38c05d070 0x7fb38c128410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:52.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.305+0000 7fb389d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb38c066b20 0x7fb38c0afd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:52.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.305+0000 7fb38a59c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb38c05d070 0x7fb38c128410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:37246/0 (socket says 172.21.15.60:37246) 2022-06-29T16:06:52.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.305+0000 7fb38a59c700 1 -- 172.21.15.60:0/1866054677 learned_addr learned my addr 172.21.15.60:0/1866054677 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:52.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.305+0000 7fb38ad9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb38c142ae0 0x7fb38c0b0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52388/0 (socket says 172.21.15.60:52388) 2022-06-29T16:06:52.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.305+0000 7fb38a59c700 1 -- 172.21.15.60:0/1866054677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb38c066b20 msgr2=0x7fb38c0afd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:52.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.305+0000 7fb38a59c700 1 --2- 172.21.15.60:0/1866054677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb38c066b20 0x7fb38c0afd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:52.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.305+0000 7fb38a59c700 1 -- 172.21.15.60:0/1866054677 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb38c142ae0 msgr2=0x7fb38c0b0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:52.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.305+0000 7fb38a59c700 1 --2- 172.21.15.60:0/1866054677 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb38c142ae0 0x7fb38c0b0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:52.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.305+0000 7fb38a59c700 1 -- 172.21.15.60:0/1866054677 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb384009470 con 0x7fb38c05d070 2022-06-29T16:06:52.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.305+0000 7fb38a59c700 1 --2- 172.21.15.60:0/1866054677 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb38c05d070 0x7fb38c128410 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fb38400c000 tx=0x7fb3840091d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:52.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.305+0000 7fb3837fe700 1 -- 172.21.15.60:0/1866054677 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb38400c850 con 0x7fb38c05d070 2022-06-29T16:06:52.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.305+0000 7fb3837fe700 1 -- 172.21.15.60:0/1866054677 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb38400c9b0 con 0x7fb38c05d070 2022-06-29T16:06:52.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.305+0000 7fb38b59e700 1 -- 172.21.15.60:0/1866054677 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb37400b180 con 0x7fb38c05d070 2022-06-29T16:06:52.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.305+0000 7fb3837fe700 1 -- 172.21.15.60:0/1866054677 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb38400cd10 con 0x7fb38c05d070 2022-06-29T16:06:52.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.305+0000 7fb38b59e700 1 -- 172.21.15.60:0/1866054677 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb38c12d400 con 0x7fb38c05d070 2022-06-29T16:06:52.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.309+0000 7fb38b59e700 1 -- 172.21.15.60:0/1866054677 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb38c0573d0 con 0x7fb38c05d070 2022-06-29T16:06:52.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.309+0000 7fb3837fe700 1 -- 172.21.15.60:0/1866054677 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb384022070 con 0x7fb38c05d070 2022-06-29T16:06:52.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.309+0000 7fb3837fe700 1 --2- 172.21.15.60:0/1866054677 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb3640539e0 0x7fb364055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:52.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.309+0000 7fb3837fe700 1 -- 172.21.15.60:0/1866054677 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fb38406c730 con 0x7fb38c05d070 2022-06-29T16:06:52.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.309+0000 7fb389d9b700 1 --2- 172.21.15.60:0/1866054677 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb3640539e0 0x7fb364055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:52.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.309+0000 7fb389d9b700 1 --2- 172.21.15.60:0/1866054677 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb3640539e0 0x7fb364055ea0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 rx=0x7fb37401a000 tx=0x7fb374014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:52.326 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.317+0000 7fb3837fe700 1 -- 172.21.15.60:0/1866054677 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb384043600 con 0x7fb38c05d070 2022-06-29T16:06:52.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:52.485+0000 7fb38b59e700 1 -- 172.21.15.60:0/1866054677 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb38c1458f0 con 0x7fb38c05d070 2022-06-29T16:06:53.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.173+0000 7fb3837fe700 1 -- 172.21.15.60:0/1866054677 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v53) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb384046c20 con 0x7fb38c05d070 2022-06-29T16:06:53.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.177+0000 7fb38b59e700 1 -- 172.21.15.60:0/1866054677 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb3640539e0 msgr2=0x7fb364055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:53.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.177+0000 7fb38b59e700 1 --2- 172.21.15.60:0/1866054677 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb3640539e0 0x7fb364055ea0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 rx=0x7fb37401a000 tx=0x7fb374014000).stop 2022-06-29T16:06:53.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.177+0000 7fb38b59e700 1 -- 172.21.15.60:0/1866054677 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb38c05d070 msgr2=0x7fb38c128410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:53.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.177+0000 7fb38b59e700 1 --2- 172.21.15.60:0/1866054677 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb38c05d070 0x7fb38c128410 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fb38400c000 tx=0x7fb3840091d0).stop 2022-06-29T16:06:53.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.177+0000 7fb38b59e700 1 -- 172.21.15.60:0/1866054677 shutdown_connections 2022-06-29T16:06:53.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.177+0000 7fb38b59e700 1 --2- 172.21.15.60:0/1866054677 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb3640539e0 0x7fb364055ea0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:53.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.177+0000 7fb38b59e700 1 --2- 172.21.15.60:0/1866054677 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb38c142ae0 0x7fb38c0b0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:53.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.177+0000 7fb38b59e700 1 --2- 172.21.15.60:0/1866054677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb38c066b20 0x7fb38c0afd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:53.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.177+0000 7fb38b59e700 1 --2- 172.21.15.60:0/1866054677 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb38c05d070 0x7fb38c128410 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:53.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.177+0000 7fb38b59e700 1 -- 172.21.15.60:0/1866054677 >> 172.21.15.60:0/1866054677 conn(0x7fb38c0b5890 msgr2=0x7fb38c144420 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:53.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.177+0000 7fb38b59e700 1 -- 172.21.15.60:0/1866054677 shutdown_connections 2022-06-29T16:06:53.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.177+0000 7fb38b59e700 1 -- 172.21.15.60:0/1866054677 wait complete. 2022-06-29T16:06:53.199 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-06-29T16:06:53.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.313+0000 7fe0d68c7700 1 Processor -- start 2022-06-29T16:06:53.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.313+0000 7fe0d68c7700 1 -- start start 2022-06-29T16:06:53.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.313+0000 7fe0d68c7700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe0d00b9bd0 0x7fe0d00b51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:53.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.313+0000 7fe0d68c7700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe0d00b3d30 0x7fe0d0142030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:53.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.313+0000 7fe0d68c7700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d00afda0 con 0x7fe0d0142520 2022-06-29T16:06:53.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.313+0000 7fe0d68c7700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0d01429a0 con 0x7fe0d00b3d30 2022-06-29T16:06:53.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.313+0000 7fe0d68c7700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0d00b9fb0 con 0x7fe0d00b9bd0 2022-06-29T16:06:53.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.313+0000 7fe0d50c4700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe0d00b3d30 0x7fe0d0142030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:53.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.313+0000 7fe0d50c4700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe0d00b3d30 0x7fe0d0142030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52392/0 (socket says 172.21.15.60:52392) 2022-06-29T16:06:53.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.313+0000 7fe0d50c4700 1 -- 172.21.15.60:0/2135233057 learned_addr learned my addr 172.21.15.60:0/2135233057 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:53.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.313+0000 7fe0d58c5700 1 --2- 172.21.15.60:0/2135233057 >> v2:172.21.15.92:3300/0 conn(0x7fe0d00b9bd0 0x7fe0d00b51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:53.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.313+0000 7fe0d50c4700 1 -- 172.21.15.60:0/2135233057 >> v2:172.21.15.92:3300/0 conn(0x7fe0d00b9bd0 msgr2=0x7fe0d00b51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:53.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.313+0000 7fe0d50c4700 1 --2- 172.21.15.60:0/2135233057 >> v2:172.21.15.92:3300/0 conn(0x7fe0d00b9bd0 0x7fe0d00b51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:53.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.313+0000 7fe0d50c4700 1 -- 172.21.15.60:0/2135233057 >> v1:172.21.15.92:6789/0 conn(0x7fe0d0142520 legacy=0x7fe0d005bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:53.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.313+0000 7fe0d50c4700 1 -- 172.21.15.60:0/2135233057 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d00b05c0 con 0x7fe0d00b3d30 2022-06-29T16:06:53.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.313+0000 7fe0d50c4700 1 --2- 172.21.15.60:0/2135233057 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0d00b3d30 0x7fe0d0142030 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 rx=0x7fe0c0002f70 tx=0x7fe0c0006eb0).ready entity=mon.0 client_cookie=c310d7930131e212 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:53.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.313+0000 7fe0d48c3700 1 -- 172.21.15.60:0/2135233057 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0c000d5c0 con 0x7fe0d00b3d30 2022-06-29T16:06:53.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.313+0000 7fe0d48c3700 1 -- 172.21.15.60:0/2135233057 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c0000e90 con 0x7fe0d00b3d30 2022-06-29T16:06:53.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.313+0000 7fe0d48c3700 1 -- 172.21.15.60:0/2135233057 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0c000d950 con 0x7fe0d00b3d30 2022-06-29T16:06:53.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.317+0000 7fe0d68c7700 1 -- 172.21.15.60:0/2135233057 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0d00b3d30 msgr2=0x7fe0d0142030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:53.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.317+0000 7fe0d68c7700 1 --2- 172.21.15.60:0/2135233057 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0d00b3d30 0x7fe0d0142030 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 rx=0x7fe0c0002f70 tx=0x7fe0c0006eb0).stop 2022-06-29T16:06:53.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.317+0000 7fe0d68c7700 1 -- 172.21.15.60:0/2135233057 shutdown_connections 2022-06-29T16:06:53.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.317+0000 7fe0d68c7700 1 --2- 172.21.15.60:0/2135233057 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0d00b3d30 0x7fe0d0142030 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:53.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.317+0000 7fe0d68c7700 1 --2- 172.21.15.60:0/2135233057 >> v2:172.21.15.92:3300/0 conn(0x7fe0d00b9bd0 0x7fe0d00b51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:53.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.317+0000 7fe0d68c7700 1 -- 172.21.15.60:0/2135233057 >> 172.21.15.60:0/2135233057 conn(0x7fe0d0062cb0 msgr2=0x7fe0d00b9180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:53.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.317+0000 7fe0d68c7700 1 -- 172.21.15.60:0/2135233057 shutdown_connections 2022-06-29T16:06:53.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.317+0000 7fe0d68c7700 1 -- 172.21.15.60:0/2135233057 wait complete. 2022-06-29T16:06:53.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.317+0000 7fe0d68c7700 1 Processor -- start 2022-06-29T16:06:53.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.317+0000 7fe0d68c7700 1 -- start start 2022-06-29T16:06:53.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.317+0000 7fe0d68c7700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0d00b3d30 0x7fe0d013df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:53.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.317+0000 7fe0d68c7700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0d00b9bd0 0x7fe0d013eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:53.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.317+0000 7fe0d68c7700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0d0142520 0x7fe0d013f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:53.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.317+0000 7fe0d68c7700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d0152ee0 con 0x7fe0d00b3d30 2022-06-29T16:06:53.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.317+0000 7fe0d68c7700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d0112480 con 0x7fe0d00b9bd0 2022-06-29T16:06:53.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.317+0000 7fe0d68c7700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d0112720 con 0x7fe0d0142520 2022-06-29T16:06:53.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.317+0000 7fe0d58c5700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0d00b3d30 0x7fe0d013df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:53.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.317+0000 7fe0d58c5700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0d00b3d30 0x7fe0d013df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52398/0 (socket says 172.21.15.60:52398) 2022-06-29T16:06:53.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.317+0000 7fe0d58c5700 1 -- 172.21.15.60:0/3329221421 learned_addr learned my addr 172.21.15.60:0/3329221421 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:53.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.321+0000 7fe0d60c6700 1 --2- 172.21.15.60:0/3329221421 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0d0142520 0x7fe0d013f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:53.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.321+0000 7fe0d50c4700 1 --2- 172.21.15.60:0/3329221421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0d00b9bd0 0x7fe0d013eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:53.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.321+0000 7fe0d58c5700 1 -- 172.21.15.60:0/3329221421 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0d0142520 msgr2=0x7fe0d013f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:53.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.321+0000 7fe0d58c5700 1 --2- 172.21.15.60:0/3329221421 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0d0142520 0x7fe0d013f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:53.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.321+0000 7fe0d58c5700 1 -- 172.21.15.60:0/3329221421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0d00b9bd0 msgr2=0x7fe0d013eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:53.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.321+0000 7fe0d58c5700 1 --2- 172.21.15.60:0/3329221421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0d00b9bd0 0x7fe0d013eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:53.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.321+0000 7fe0d58c5700 1 -- 172.21.15.60:0/3329221421 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c000b1e0 con 0x7fe0d00b3d30 2022-06-29T16:06:53.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.321+0000 7fe0d50c4700 1 --2- 172.21.15.60:0/3329221421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0d00b9bd0 0x7fe0d013eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:06:53.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.321+0000 7fe0d60c6700 1 --2- 172.21.15.60:0/3329221421 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0d0142520 0x7fe0d013f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:06:53.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.321+0000 7fe0d58c5700 1 --2- 172.21.15.60:0/3329221421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0d00b3d30 0x7fe0d013df00 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 rx=0x7fe0cc00c000 tx=0x7fe0cc006d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:53.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.321+0000 7fe0beffd700 1 -- 172.21.15.60:0/3329221421 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0cc010590 con 0x7fe0d00b3d30 2022-06-29T16:06:53.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.321+0000 7fe0beffd700 1 -- 172.21.15.60:0/3329221421 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0cc0106f0 con 0x7fe0d00b3d30 2022-06-29T16:06:53.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.321+0000 7fe0d68c7700 1 -- 172.21.15.60:0/3329221421 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0d013e3f0 con 0x7fe0d00b3d30 2022-06-29T16:06:53.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.321+0000 7fe0beffd700 1 -- 172.21.15.60:0/3329221421 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0cc010a50 con 0x7fe0d00b3d30 2022-06-29T16:06:53.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.321+0000 7fe0d68c7700 1 -- 172.21.15.60:0/3329221421 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0d0112bc0 con 0x7fe0d00b3d30 2022-06-29T16:06:53.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.321+0000 7fe0d68c7700 1 -- 172.21.15.60:0/3329221421 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d00573d0 con 0x7fe0d00b3d30 2022-06-29T16:06:53.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.321+0000 7fe0beffd700 1 -- 172.21.15.60:0/3329221421 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe0cc01a030 con 0x7fe0d00b3d30 2022-06-29T16:06:53.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.321+0000 7fe0beffd700 1 --2- 172.21.15.60:0/3329221421 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe0b0053a70 0x7fe0b0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:53.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.321+0000 7fe0beffd700 1 -- 172.21.15.60:0/3329221421 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fe0cc06bb60 con 0x7fe0d00b3d30 2022-06-29T16:06:53.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.325+0000 7fe0d50c4700 1 --2- 172.21.15.60:0/3329221421 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe0b0053a70 0x7fe0b0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:53.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.325+0000 7fe0d50c4700 1 --2- 172.21.15.60:0/3329221421 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe0b0053a70 0x7fe0b0055f30 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7fe0c0017000 tx=0x7fe0c0015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:53.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.333+0000 7fe0beffd700 1 -- 172.21.15.60:0/3329221421 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe0cc042ae0 con 0x7fe0d00b3d30 2022-06-29T16:06:53.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:53.461+0000 7fe0d68c7700 1 -- 172.21.15.60:0/3329221421 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe0d00b45e0 con 0x7fe0d00b3d30 2022-06-29T16:06:54.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.177+0000 7fe0beffd700 1 -- 172.21.15.60:0/3329221421 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v113) v1 ==== 185+0+0 (secure 0 0 0) 0x7fe0cc046100 con 0x7fe0d00b3d30 2022-06-29T16:06:54.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.181+0000 7fe0d68c7700 1 -- 172.21.15.60:0/3329221421 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe0b0053a70 msgr2=0x7fe0b0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:54.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.181+0000 7fe0d68c7700 1 --2- 172.21.15.60:0/3329221421 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe0b0053a70 0x7fe0b0055f30 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7fe0c0017000 tx=0x7fe0c0015000).stop 2022-06-29T16:06:54.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.181+0000 7fe0d68c7700 1 -- 172.21.15.60:0/3329221421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0d00b3d30 msgr2=0x7fe0d013df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:54.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.181+0000 7fe0d68c7700 1 --2- 172.21.15.60:0/3329221421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0d00b3d30 0x7fe0d013df00 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 rx=0x7fe0cc00c000 tx=0x7fe0cc006d90).stop 2022-06-29T16:06:54.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.181+0000 7fe0d68c7700 1 -- 172.21.15.60:0/3329221421 shutdown_connections 2022-06-29T16:06:54.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.181+0000 7fe0d68c7700 1 --2- 172.21.15.60:0/3329221421 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe0b0053a70 0x7fe0b0055f30 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:54.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.181+0000 7fe0d68c7700 1 --2- 172.21.15.60:0/3329221421 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0d0142520 0x7fe0d013f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:54.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.181+0000 7fe0d68c7700 1 --2- 172.21.15.60:0/3329221421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0d00b9bd0 0x7fe0d013eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:54.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.181+0000 7fe0d68c7700 1 --2- 172.21.15.60:0/3329221421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe0d00b3d30 0x7fe0d013df00 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:54.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.181+0000 7fe0d68c7700 1 -- 172.21.15.60:0/3329221421 >> 172.21.15.60:0/3329221421 conn(0x7fe0d0062cb0 msgr2=0x7fe0d013d120 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:54.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.181+0000 7fe0d68c7700 1 -- 172.21.15.60:0/3329221421 shutdown_connections 2022-06-29T16:06:54.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.181+0000 7fe0d68c7700 1 -- 172.21.15.60:0/3329221421 wait complete. 2022-06-29T16:06:54.191 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2022-06-29T16:06:54.202 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-06-29T16:06:54.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.309+0000 7ff951a4e700 1 Processor -- start 2022-06-29T16:06:54.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.309+0000 7ff951a4e700 1 -- start start 2022-06-29T16:06:54.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.309+0000 7ff951a4e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff94c0b2130 0x7ff94c146100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:54.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.309+0000 7ff951a4e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff94c05d450 con 0x7ff94c05d070 2022-06-29T16:06:54.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.309+0000 7ff951a4e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff94c0b25d0 con 0x7ff94c0aed00 2022-06-29T16:06:54.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.309+0000 7ff951a4e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff94c0adf90 con 0x7ff94c0b2130 2022-06-29T16:06:54.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.309+0000 7ff950a4c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff94c05d070 0x7ff94c14aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60492/0 (socket says 172.21.15.60:60492) 2022-06-29T16:06:54.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.309+0000 7ff950a4c700 1 -- 172.21.15.60:0/1433347419 learned_addr learned my addr 172.21.15.60:0/1433347419 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:06:54.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.309+0000 7ff94bfff700 1 --2- 172.21.15.60:0/1433347419 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff94c0b2130 0x7ff94c146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:54.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.309+0000 7ff94b7fe700 1 -- 172.21.15.60:0/1433347419 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3749890809 0 0) 0x7ff94c05d450 con 0x7ff94c05d070 2022-06-29T16:06:54.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.309+0000 7ff94b7fe700 1 -- 172.21.15.60:0/1433347419 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff92c003260 con 0x7ff94c05d070 2022-06-29T16:06:54.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.309+0000 7ff94b7fe700 1 -- 172.21.15.60:0/1433347419 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4292115531 0 0) 0x7ff94c0b25d0 con 0x7ff94c0aed00 2022-06-29T16:06:54.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.309+0000 7ff94bfff700 1 -- 172.21.15.60:0/1433347419 >> v1:172.21.15.92:6789/0 conn(0x7ff94c0aed00 legacy=0x7ff94c1465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:54.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.309+0000 7ff94bfff700 1 -- 172.21.15.60:0/1433347419 >> v1:172.21.15.60:6789/0 conn(0x7ff94c05d070 legacy=0x7ff94c14aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:54.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.309+0000 7ff94bfff700 1 -- 172.21.15.60:0/1433347419 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff94c14a350 con 0x7ff94c0b2130 2022-06-29T16:06:54.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.313+0000 7ff94bfff700 1 --2- 172.21.15.60:0/1433347419 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff94c0b2130 0x7ff94c146100 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7ff93c002f70 tx=0x7ff93c006df0).ready entity=mon.2 client_cookie=48ddfd76a8e03918 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:54.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.313+0000 7ff94b7fe700 1 -- 172.21.15.60:0/1433347419 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff93c00d570 con 0x7ff94c0b2130 2022-06-29T16:06:54.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.313+0000 7ff94b7fe700 1 -- 172.21.15.60:0/1433347419 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff93c000e90 con 0x7ff94c0b2130 2022-06-29T16:06:54.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.313+0000 7ff94b7fe700 1 -- 172.21.15.60:0/1433347419 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff93c00d8c0 con 0x7ff94c0b2130 2022-06-29T16:06:54.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.313+0000 7ff951a4e700 1 -- 172.21.15.60:0/1433347419 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff94c0b2130 msgr2=0x7ff94c146100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:54.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.313+0000 7ff951a4e700 1 --2- 172.21.15.60:0/1433347419 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff94c0b2130 0x7ff94c146100 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7ff93c002f70 tx=0x7ff93c006df0).stop 2022-06-29T16:06:54.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.313+0000 7ff951a4e700 1 -- 172.21.15.60:0/1433347419 shutdown_connections 2022-06-29T16:06:54.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.313+0000 7ff951a4e700 1 --2- 172.21.15.60:0/1433347419 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff94c0b2130 0x7ff94c146100 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:54.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.313+0000 7ff951a4e700 1 -- 172.21.15.60:0/1433347419 >> 172.21.15.60:0/1433347419 conn(0x7ff94c0b5890 msgr2=0x7ff94c0b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:54.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.313+0000 7ff951a4e700 1 -- 172.21.15.60:0/1433347419 shutdown_connections 2022-06-29T16:06:54.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.313+0000 7ff951a4e700 1 -- 172.21.15.60:0/1433347419 wait complete. 2022-06-29T16:06:54.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.313+0000 7ff951a4e700 1 Processor -- start 2022-06-29T16:06:54.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.313+0000 7ff951a4e700 1 -- start start 2022-06-29T16:06:54.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.313+0000 7ff951a4e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff94c05d070 0x7ff94c14e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:54.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.313+0000 7ff951a4e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff94c0aed00 0x7ff94c155e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:54.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.313+0000 7ff951a4e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff94c0b2130 0x7ff94c156370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:54.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.313+0000 7ff951a4e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94c0654b0 con 0x7ff94c0aed00 2022-06-29T16:06:54.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.313+0000 7ff951a4e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94c14d4a0 con 0x7ff94c0b2130 2022-06-29T16:06:54.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.313+0000 7ff951a4e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff94c0638b0 con 0x7ff94c05d070 2022-06-29T16:06:54.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.317+0000 7ff94bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff94c0aed00 0x7ff94c155e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:54.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.317+0000 7ff94bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff94c0aed00 0x7ff94c155e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52414/0 (socket says 172.21.15.60:52414) 2022-06-29T16:06:54.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.317+0000 7ff94bfff700 1 -- 172.21.15.60:0/3854434797 learned_addr learned my addr 172.21.15.60:0/3854434797 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:54.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.317+0000 7ff950a4c700 1 --2- 172.21.15.60:0/3854434797 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff94c05d070 0x7ff94c14e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:54.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.317+0000 7ff95124d700 1 --2- 172.21.15.60:0/3854434797 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff94c0b2130 0x7ff94c156370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:54.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.317+0000 7ff94bfff700 1 -- 172.21.15.60:0/3854434797 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff94c05d070 msgr2=0x7ff94c14e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:54.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.317+0000 7ff94bfff700 1 --2- 172.21.15.60:0/3854434797 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff94c05d070 0x7ff94c14e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:54.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.317+0000 7ff94bfff700 1 -- 172.21.15.60:0/3854434797 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff94c0b2130 msgr2=0x7ff94c156370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:54.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.317+0000 7ff94bfff700 1 --2- 172.21.15.60:0/3854434797 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff94c0b2130 0x7ff94c156370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:54.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.317+0000 7ff94bfff700 1 -- 172.21.15.60:0/3854434797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff93c00b180 con 0x7ff94c0aed00 2022-06-29T16:06:54.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.317+0000 7ff94bfff700 1 --2- 172.21.15.60:0/3854434797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff94c0aed00 0x7ff94c155e80 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 rx=0x7ff94c14d5e0 tx=0x7ff93c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:54.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.317+0000 7ff949ffb700 1 -- 172.21.15.60:0/3854434797 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff93c016070 con 0x7ff94c0aed00 2022-06-29T16:06:54.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.317+0000 7ff949ffb700 1 -- 172.21.15.60:0/3854434797 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff93c013980 con 0x7ff94c0aed00 2022-06-29T16:06:54.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.317+0000 7ff951a4e700 1 -- 172.21.15.60:0/3854434797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff94c063ae0 con 0x7ff94c0aed00 2022-06-29T16:06:54.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.317+0000 7ff949ffb700 1 -- 172.21.15.60:0/3854434797 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff93c013ce0 con 0x7ff94c0aed00 2022-06-29T16:06:54.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.317+0000 7ff951a4e700 1 -- 172.21.15.60:0/3854434797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff94c14e6c0 con 0x7ff94c0aed00 2022-06-29T16:06:54.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.317+0000 7ff951a4e700 1 -- 172.21.15.60:0/3854434797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff94c14a5f0 con 0x7ff94c0aed00 2022-06-29T16:06:54.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.317+0000 7ff949ffb700 1 -- 172.21.15.60:0/3854434797 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff93c025030 con 0x7ff94c0aed00 2022-06-29T16:06:54.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.317+0000 7ff949ffb700 1 --2- 172.21.15.60:0/3854434797 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff930053b00 0x7ff930055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:54.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.317+0000 7ff949ffb700 1 -- 172.21.15.60:0/3854434797 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5721+0+0 (secure 0 0 0) 0x7ff93c06cd00 con 0x7ff94c0aed00 2022-06-29T16:06:54.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.321+0000 7ff950a4c700 1 --2- 172.21.15.60:0/3854434797 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff930053b00 0x7ff930055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:54.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.321+0000 7ff950a4c700 1 --2- 172.21.15.60:0/3854434797 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff930053b00 0x7ff930055fc0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7ff93400b000 tx=0x7ff934009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:54.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.321+0000 7ff949ffb700 1 -- 172.21.15.60:0/3854434797 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff93c070030 con 0x7ff94c0aed00 2022-06-29T16:06:54.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:54.449+0000 7ff951a4e700 1 -- 172.21.15.60:0/3854434797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7ff94c063c70 con 0x7ff94c0aed00 2022-06-29T16:06:55.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:55.189+0000 7ff949ffb700 1 -- 172.21.15.60:0/3854434797 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v114) v1 ==== 133+0+0 (secure 0 0 0) 0x7ff93c023020 con 0x7ff94c0aed00 2022-06-29T16:06:55.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:55.193+0000 7ff951a4e700 1 -- 172.21.15.60:0/3854434797 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff930053b00 msgr2=0x7ff930055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:55.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:55.193+0000 7ff951a4e700 1 --2- 172.21.15.60:0/3854434797 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff930053b00 0x7ff930055fc0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7ff93400b000 tx=0x7ff934009040).stop 2022-06-29T16:06:55.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:55.193+0000 7ff951a4e700 1 -- 172.21.15.60:0/3854434797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff94c0aed00 msgr2=0x7ff94c155e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:55.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:55.193+0000 7ff951a4e700 1 --2- 172.21.15.60:0/3854434797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff94c0aed00 0x7ff94c155e80 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 rx=0x7ff94c14d5e0 tx=0x7ff93c014000).stop 2022-06-29T16:06:55.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:55.197+0000 7ff951a4e700 1 -- 172.21.15.60:0/3854434797 shutdown_connections 2022-06-29T16:06:55.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:55.197+0000 7ff951a4e700 1 --2- 172.21.15.60:0/3854434797 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff930053b00 0x7ff930055fc0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:55.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:55.197+0000 7ff951a4e700 1 --2- 172.21.15.60:0/3854434797 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff94c0b2130 0x7ff94c156370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:55.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:55.197+0000 7ff951a4e700 1 --2- 172.21.15.60:0/3854434797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff94c0aed00 0x7ff94c155e80 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:55.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:55.197+0000 7ff951a4e700 1 --2- 172.21.15.60:0/3854434797 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff94c05d070 0x7ff94c14e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:55.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:55.197+0000 7ff951a4e700 1 -- 172.21.15.60:0/3854434797 >> 172.21.15.60:0/3854434797 conn(0x7ff94c0b5890 msgr2=0x7ff94c0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:55.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:55.197+0000 7ff951a4e700 1 -- 172.21.15.60:0/3854434797 shutdown_connections 2022-06-29T16:06:55.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:55.197+0000 7ff951a4e700 1 -- 172.21.15.60:0/3854434797 wait complete. 2022-06-29T16:06:55.204 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2022-06-29T16:06:55.216 DEBUG:teuthology.orchestra.run.smithi121:> sudo rados -p cephfs_metadata put key /etc/fstab 2022-06-29T16:06:55.259 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.248+0000 7f4ff0ebbb80 1 Processor -- start 2022-06-29T16:06:55.260 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.248+0000 7f4ff0ebbb80 1 -- start start 2022-06-29T16:06:55.260 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.248+0000 7f4ff0ebbb80 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x5585592b7650 0x55855934fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:55.260 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.248+0000 7f4ff0ebbb80 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55855934d820 0x558559350100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:55.261 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.248+0000 7f4ff0ebbb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558559351330 0x5585593505f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:55.261 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.248+0000 7f4ff0ebbb80 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x558559244540 con 0x55855934d820 2022-06-29T16:06:55.261 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.248+0000 7f4ff0ebbb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x558559234460 con 0x558559351330 2022-06-29T16:06:55.262 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.248+0000 7f4ff0ebbb80 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x55855934d5d0 con 0x5585592b7650 2022-06-29T16:06:55.262 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4fefaf9700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558559351330 0x5585593505f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:55.262 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4fef2f8700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x5585592b7650 0x55855934fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:55.262 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4feeaf7700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55855934d820 0x558559350100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:55.263 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4fefaf9700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558559351330 0x5585593505f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.121:56874/0 (socket says 172.21.15.121:56874) 2022-06-29T16:06:55.263 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4fefaf9700 1 -- 172.21.15.121:0/3371915110 learned_addr learned my addr 172.21.15.121:0/3371915110 (peer_addr_for_me v2:172.21.15.121:0/0) 2022-06-29T16:06:55.267 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4feeaf7700 1 -- 172.21.15.121:0/3371915110 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x5585592b7650 msgr2=0x55855934fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:55.267 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4feeaf7700 1 --2- 172.21.15.121:0/3371915110 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x5585592b7650 0x55855934fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:55.267 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4feeaf7700 1 -- 172.21.15.121:0/3371915110 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558559351330 msgr2=0x5585593505f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:55.268 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4feeaf7700 1 --2- 172.21.15.121:0/3371915110 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558559351330 0x5585593505f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:55.268 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4feeaf7700 1 -- 172.21.15.121:0/3371915110 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe8005cd0 con 0x55855934d820 2022-06-29T16:06:55.268 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4fefaf9700 1 --2- 172.21.15.121:0/3371915110 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558559351330 0x5585593505f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:06:55.268 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4fef2f8700 1 --2- 172.21.15.121:0/3371915110 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x5585592b7650 0x55855934fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:06:55.269 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4feeaf7700 1 --2- 172.21.15.121:0/3371915110 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55855934d820 0x558559350100 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 rx=0x5585592b72b0 tx=0x7f4fd80017c0).ready entity=mon.0 client_cookie=ac2f0a9558064d86 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:55.269 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4fed2f4700 1 -- 172.21.15.121:0/3371915110 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4fd800f450 con 0x55855934d820 2022-06-29T16:06:55.269 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4fed2f4700 1 -- 172.21.15.121:0/3371915110 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fd800f5b0 con 0x55855934d820 2022-06-29T16:06:55.269 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4fed2f4700 1 -- 172.21.15.121:0/3371915110 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4fd800f910 con 0x55855934d820 2022-06-29T16:06:55.270 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4ff0ebbb80 1 -- 172.21.15.121:0/3371915110 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55855934d820 msgr2=0x558559350100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:55.270 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4ff0ebbb80 1 --2- 172.21.15.121:0/3371915110 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55855934d820 0x558559350100 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 rx=0x5585592b72b0 tx=0x7f4fd80017c0).stop 2022-06-29T16:06:55.270 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4ff0ebbb80 1 -- 172.21.15.121:0/3371915110 shutdown_connections 2022-06-29T16:06:55.270 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4ff0ebbb80 1 --2- 172.21.15.121:0/3371915110 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558559351330 0x5585593505f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:55.271 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4ff0ebbb80 1 --2- 172.21.15.121:0/3371915110 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55855934d820 0x558559350100 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:55.271 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4ff0ebbb80 1 --2- 172.21.15.121:0/3371915110 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x5585592b7650 0x55855934fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:55.271 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4ff0ebbb80 1 -- 172.21.15.121:0/3371915110 >> 172.21.15.121:0/3371915110 conn(0x55855933e0d0 msgr2=0x558559340340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:55.271 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4ff0ebbb80 1 -- 172.21.15.121:0/3371915110 shutdown_connections 2022-06-29T16:06:55.272 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4ff0ebbb80 1 -- 172.21.15.121:0/3371915110 wait complete. 2022-06-29T16:06:55.272 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4ff0ebbb80 1 Processor -- start 2022-06-29T16:06:55.272 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4ff0ebbb80 1 -- start start 2022-06-29T16:06:55.272 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4ff0ebbb80 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5585592b7650 0x55855934b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:55.273 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4ff0ebbb80 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55855934d820 0x558559346cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:55.273 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4ff0ebbb80 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x558559351330 0x5585593471b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:55.273 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4ff0ebbb80 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x558559352ca0 con 0x55855934d820 2022-06-29T16:06:55.273 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4ff0ebbb80 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x558559347790 con 0x5585592b7650 2022-06-29T16:06:55.274 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4ff0ebbb80 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x558559347950 con 0x558559351330 2022-06-29T16:06:55.274 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4fefaf9700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x558559351330 0x5585593471b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:55.274 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4fef2f8700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5585592b7650 0x55855934b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:55.274 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4feeaf7700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55855934d820 0x558559346cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:55.274 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4fefaf9700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x558559351330 0x5585593471b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.121:42900/0 (socket says 172.21.15.121:42900) 2022-06-29T16:06:55.275 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.252+0000 7f4fefaf9700 1 -- 172.21.15.121:0/34715686 learned_addr learned my addr 172.21.15.121:0/34715686 (peer_addr_for_me v2:172.21.15.121:0/0) 2022-06-29T16:06:55.275 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.256+0000 7f4fefaf9700 1 -- 172.21.15.121:0/34715686 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5585592b7650 msgr2=0x55855934b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:55.275 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.256+0000 7f4fefaf9700 1 --2- 172.21.15.121:0/34715686 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5585592b7650 0x55855934b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:55.275 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.256+0000 7f4fefaf9700 1 -- 172.21.15.121:0/34715686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55855934d820 msgr2=0x558559346cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:55.276 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.256+0000 7f4fefaf9700 1 --2- 172.21.15.121:0/34715686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55855934d820 0x558559346cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:55.276 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.256+0000 7f4fefaf9700 1 -- 172.21.15.121:0/34715686 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fdc001e10 con 0x558559351330 2022-06-29T16:06:55.276 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.256+0000 7f4fefaf9700 1 --2- 172.21.15.121:0/34715686 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x558559351330 0x5585593471b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f4fe801a000 tx=0x7f4fe8015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:55.276 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.256+0000 7f4fecaf3700 1 -- 172.21.15.121:0/34715686 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4fe80127f0 con 0x558559351330 2022-06-29T16:06:55.277 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.256+0000 7f4fecaf3700 1 -- 172.21.15.121:0/34715686 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe8012950 con 0x558559351330 2022-06-29T16:06:55.277 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.256+0000 7f4ff0ebbb80 1 -- 172.21.15.121:0/34715686 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5585592591d0 con 0x558559351330 2022-06-29T16:06:55.277 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.256+0000 7f4fecaf3700 1 -- 172.21.15.121:0/34715686 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4fe8012cb0 con 0x558559351330 2022-06-29T16:06:55.277 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.256+0000 7f4ff0ebbb80 1 -- 172.21.15.121:0/34715686 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558559253af0 con 0x558559351330 2022-06-29T16:06:55.278 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.256+0000 7f4fecaf3700 1 -- 172.21.15.121:0/34715686 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4fe801f470 con 0x558559351330 2022-06-29T16:06:55.278 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.260+0000 7f4fecaf3700 1 --2- 172.21.15.121:0/34715686 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4fd00525f0 0x7f4fd0054ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:55.279 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.260+0000 7f4fecaf3700 1 -- 172.21.15.121:0/34715686 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6075+0+0 (secure 0 0 0) 0x7f4fe806d390 con 0x558559351330 2022-06-29T16:06:55.280 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.260+0000 7f4fef2f8700 1 --2- 172.21.15.121:0/34715686 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4fd00525f0 0x7f4fd0054ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:55.280 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.260+0000 7f4ff0ebbb80 1 --2- 172.21.15.121:0/34715686 >> [v2:172.21.15.60:6816/12890,v1:172.21.15.60:6825/12890] conn(0x558559361fd0 0x5585593643c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:55.280 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.260+0000 7f4ff0ebbb80 1 -- 172.21.15.121:0/34715686 --> [v2:172.21.15.60:6816/12890,v1:172.21.15.60:6825/12890] -- osd_op(unknown.0.0:1 23.0 23:0620e6f9:::key:head [writefull 0~617 in=617b] snapc 0=[] ondisk+write+known_if_redirected e114) v8 -- 0x558559364950 con 0x558559361fd0 2022-06-29T16:06:55.280 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.260+0000 7f4feeaf7700 1 --2- 172.21.15.121:0/34715686 >> [v2:172.21.15.60:6816/12890,v1:172.21.15.60:6825/12890] conn(0x558559361fd0 0x5585593643c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:55.280 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.260+0000 7f4fef2f8700 1 --2- 172.21.15.121:0/34715686 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4fd00525f0 0x7f4fd0054ab0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7f4fe0008000 tx=0x7f4fe0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:55.281 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:55.260+0000 7f4feeaf7700 1 --2- 172.21.15.121:0/34715686 >> [v2:172.21.15.60:6816/12890,v1:172.21.15.60:6825/12890] conn(0x558559361fd0 0x5585593643c0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:56.205 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:56.200+0000 7f4fecaf3700 1 -- 172.21.15.121:0/34715686 <== osd.0 v2:172.21.15.60:6816/12890 1 ==== osd_map(115..115 src has 1..115) v4 ==== 320+0+0 (crc 0 0 0) 0x7f4fd8008030 con 0x558559361fd0 2022-06-29T16:06:56.207 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:56.200+0000 7f4feeaf7700 1 -- 172.21.15.121:0/34715686 <== osd.0 v2:172.21.15.60:6816/12890 2 ==== osd_op_reply(1 key [writefull 0~617] v115'1 uv1 ondisk = 0) v8 ==== 147+0+0 (crc 0 0 0) 0x7f4fd8016040 con 0x558559361fd0 2022-06-29T16:06:56.207 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:56.200+0000 7f4ff0ebbb80 1 -- 172.21.15.121:0/34715686 >> [v2:172.21.15.60:6816/12890,v1:172.21.15.60:6825/12890] conn(0x558559361fd0 msgr2=0x5585593643c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:56.208 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:56.200+0000 7f4ff0ebbb80 1 --2- 172.21.15.121:0/34715686 >> [v2:172.21.15.60:6816/12890,v1:172.21.15.60:6825/12890] conn(0x558559361fd0 0x5585593643c0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:56.208 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:56.200+0000 7f4ff0ebbb80 1 -- 172.21.15.121:0/34715686 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4fd00525f0 msgr2=0x7f4fd0054ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:56.208 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:56.200+0000 7f4ff0ebbb80 1 --2- 172.21.15.121:0/34715686 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4fd00525f0 0x7f4fd0054ab0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7f4fe0008000 tx=0x7f4fe0008040).stop 2022-06-29T16:06:56.209 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:56.200+0000 7f4ff0ebbb80 1 -- 172.21.15.121:0/34715686 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x558559351330 msgr2=0x5585593471b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:56.209 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:56.200+0000 7f4ff0ebbb80 1 --2- 172.21.15.121:0/34715686 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x558559351330 0x5585593471b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f4fe801a000 tx=0x7f4fe8015000).stop 2022-06-29T16:06:56.209 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:56.200+0000 7f4fefaf9700 1 -- 172.21.15.121:0/34715686 reap_dead start 2022-06-29T16:06:56.209 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:56.200+0000 7f4ff0ebbb80 1 -- 172.21.15.121:0/34715686 shutdown_connections 2022-06-29T16:06:56.210 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:56.200+0000 7f4ff0ebbb80 1 -- 172.21.15.121:0/34715686 >> 172.21.15.121:0/34715686 conn(0x55855933e0d0 msgr2=0x5585593402e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:56.210 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:56.200+0000 7f4ff0ebbb80 1 -- 172.21.15.121:0/34715686 shutdown_connections 2022-06-29T16:06:56.210 INFO:teuthology.orchestra.run.smithi121.stderr:2022-06-29T16:06:56.200+0000 7f4ff0ebbb80 1 -- 172.21.15.121:0/34715686 wait complete. 2022-06-29T16:06:56.212 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2022-06-29T16:06:56.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.309+0000 7f9cbd0ec700 1 Processor -- start 2022-06-29T16:06:56.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.309+0000 7f9cbd0ec700 1 -- start start 2022-06-29T16:06:56.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.309+0000 7f9cbd0ec700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9cb80aba10 0x7f9cb813e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:56.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.309+0000 7f9cbd0ec700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9cb813fdb0 0x7f9cb80b0600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:56.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.309+0000 7f9cbd0ec700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cb80afb20 con 0x7f9cb8097b40 2022-06-29T16:06:56.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.309+0000 7f9cbd0ec700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cb80abdf0 con 0x7f9cb813fdb0 2022-06-29T16:06:56.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.309+0000 7f9cbd0ec700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cb813ef70 con 0x7f9cb80aba10 2022-06-29T16:06:56.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.309+0000 7f9cbc8eb700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9cb813fdb0 0x7f9cb80b0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:56.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.309+0000 7f9cbc8eb700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9cb813fdb0 0x7f9cb80b0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52424/0 (socket says 172.21.15.60:52424) 2022-06-29T16:06:56.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.309+0000 7f9cbc8eb700 1 -- 172.21.15.60:0/1075710569 learned_addr learned my addr 172.21.15.60:0/1075710569 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:56.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.309+0000 7f9cb7fff700 1 --2- 172.21.15.60:0/1075710569 >> v2:172.21.15.92:3300/0 conn(0x7f9cb80aba10 0x7f9cb813e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:56.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.313+0000 7f9cbc8eb700 1 -- 172.21.15.60:0/1075710569 >> v2:172.21.15.92:3300/0 conn(0x7f9cb80aba10 msgr2=0x7f9cb813e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:06:56.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.313+0000 7f9cbc8eb700 1 --2- 172.21.15.60:0/1075710569 >> v2:172.21.15.92:3300/0 conn(0x7f9cb80aba10 0x7f9cb813e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:56.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.313+0000 7f9cbc8eb700 1 -- 172.21.15.60:0/1075710569 >> v1:172.21.15.92:6789/0 conn(0x7f9cb8097b40 legacy=0x7f9cb813e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:56.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.313+0000 7f9cbc8eb700 1 -- 172.21.15.60:0/1075710569 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb80afd50 con 0x7f9cb813fdb0 2022-06-29T16:06:56.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.313+0000 7f9cbc8eb700 1 --2- 172.21.15.60:0/1075710569 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9cb813fdb0 0x7f9cb80b0600 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 rx=0x7f9cb00032f0 tx=0x7f9cb000b530).ready entity=mon.0 client_cookie=6cb70f0c999f76df server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:56.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.313+0000 7f9cb6ffd700 1 -- 172.21.15.60:0/1075710569 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9cb000ddf0 con 0x7f9cb813fdb0 2022-06-29T16:06:56.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.313+0000 7f9cb6ffd700 1 -- 172.21.15.60:0/1075710569 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cb0010470 con 0x7f9cb813fdb0 2022-06-29T16:06:56.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.313+0000 7f9cb6ffd700 1 -- 172.21.15.60:0/1075710569 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9cb00107d0 con 0x7f9cb813fdb0 2022-06-29T16:06:56.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.313+0000 7f9cbd0ec700 1 -- 172.21.15.60:0/1075710569 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9cb813fdb0 msgr2=0x7f9cb80b0600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:56.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.313+0000 7f9cbd0ec700 1 --2- 172.21.15.60:0/1075710569 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9cb813fdb0 0x7f9cb80b0600 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 rx=0x7f9cb00032f0 tx=0x7f9cb000b530).stop 2022-06-29T16:06:56.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.313+0000 7f9cbd0ec700 1 -- 172.21.15.60:0/1075710569 shutdown_connections 2022-06-29T16:06:56.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.313+0000 7f9cbd0ec700 1 --2- 172.21.15.60:0/1075710569 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9cb813fdb0 0x7f9cb80b0600 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:56.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.313+0000 7f9cbd0ec700 1 --2- 172.21.15.60:0/1075710569 >> v2:172.21.15.92:3300/0 conn(0x7f9cb80aba10 0x7f9cb813e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:56.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.313+0000 7f9cbd0ec700 1 -- 172.21.15.60:0/1075710569 >> 172.21.15.60:0/1075710569 conn(0x7f9cb80b5180 msgr2=0x7f9cb80b5560 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:56.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.313+0000 7f9cbd0ec700 1 -- 172.21.15.60:0/1075710569 shutdown_connections 2022-06-29T16:06:56.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.313+0000 7f9cbd0ec700 1 -- 172.21.15.60:0/1075710569 wait complete. 2022-06-29T16:06:56.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.313+0000 7f9cbd0ec700 1 Processor -- start 2022-06-29T16:06:56.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.313+0000 7f9cbd0ec700 1 -- start start 2022-06-29T16:06:56.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.313+0000 7f9cbd0ec700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9cb8097b40 0x7f9cb809a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:56.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.313+0000 7f9cbd0ec700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cb80aba10 0x7f9cb8064af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:56.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.313+0000 7f9cbd0ec700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9cb813fdb0 0x7f9cb8096d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:56.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.313+0000 7f9cbd0ec700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb813d190 con 0x7f9cb813fdb0 2022-06-29T16:06:56.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.313+0000 7f9cbd0ec700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb813d3f0 con 0x7f9cb80aba10 2022-06-29T16:06:56.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.313+0000 7f9cbd0ec700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb813d690 con 0x7f9cb8097b40 2022-06-29T16:06:56.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.313+0000 7f9cbc8eb700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9cb813fdb0 0x7f9cb8096d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:56.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.317+0000 7f9cbc8eb700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9cb813fdb0 0x7f9cb8096d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52430/0 (socket says 172.21.15.60:52430) 2022-06-29T16:06:56.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.317+0000 7f9cbc8eb700 1 -- 172.21.15.60:0/2695045451 learned_addr learned my addr 172.21.15.60:0/2695045451 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:06:56.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.317+0000 7f9cb7fff700 1 --2- 172.21.15.60:0/2695045451 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9cb8097b40 0x7f9cb809a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:56.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.317+0000 7f9cb77fe700 1 --2- 172.21.15.60:0/2695045451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cb80aba10 0x7f9cb8064af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:56.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.317+0000 7f9cbc8eb700 1 -- 172.21.15.60:0/2695045451 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9cb8097b40 msgr2=0x7f9cb809a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:56.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.317+0000 7f9cbc8eb700 1 --2- 172.21.15.60:0/2695045451 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9cb8097b40 0x7f9cb809a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:56.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.317+0000 7f9cbc8eb700 1 -- 172.21.15.60:0/2695045451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cb80aba10 msgr2=0x7f9cb8064af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:56.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.317+0000 7f9cbc8eb700 1 --2- 172.21.15.60:0/2695045451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cb80aba10 0x7f9cb8064af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:56.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.317+0000 7f9cbc8eb700 1 -- 172.21.15.60:0/2695045451 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb000b7d0 con 0x7f9cb813fdb0 2022-06-29T16:06:56.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.317+0000 7f9cbc8eb700 1 --2- 172.21.15.60:0/2695045451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9cb813fdb0 0x7f9cb8096d60 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 rx=0x7f9cb00032f0 tx=0x7f9cb80ab540).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:56.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.317+0000 7f9cb57fa700 1 -- 172.21.15.60:0/2695045451 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9cb001d070 con 0x7f9cb813fdb0 2022-06-29T16:06:56.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.317+0000 7f9cb57fa700 1 -- 172.21.15.60:0/2695045451 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cb0018800 con 0x7f9cb813fdb0 2022-06-29T16:06:56.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.317+0000 7f9cb57fa700 1 -- 172.21.15.60:0/2695045451 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9cb0018b60 con 0x7f9cb813fdb0 2022-06-29T16:06:56.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.317+0000 7f9cbd0ec700 1 -- 172.21.15.60:0/2695045451 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cb8064fe0 con 0x7f9cb813fdb0 2022-06-29T16:06:56.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.317+0000 7f9cbd0ec700 1 -- 172.21.15.60:0/2695045451 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cb8095730 con 0x7f9cb813fdb0 2022-06-29T16:06:56.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.317+0000 7f9cbd0ec700 1 -- 172.21.15.60:0/2695045451 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cb8095980 con 0x7f9cb813fdb0 2022-06-29T16:06:56.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.317+0000 7f9cb57fa700 1 -- 172.21.15.60:0/2695045451 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9cb0029070 con 0x7f9cb813fdb0 2022-06-29T16:06:56.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.317+0000 7f9cb57fa700 1 --2- 172.21.15.60:0/2695045451 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9c98053aa0 0x7f9c98055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:06:56.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.317+0000 7f9cb57fa700 1 -- 172.21.15.60:0/2695045451 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6075+0+0 (secure 0 0 0) 0x7f9cb006d570 con 0x7f9cb813fdb0 2022-06-29T16:06:56.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.317+0000 7f9cb7fff700 1 --2- 172.21.15.60:0/2695045451 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9c98053aa0 0x7f9c98055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:06:56.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.321+0000 7f9cb7fff700 1 --2- 172.21.15.60:0/2695045451 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9c98053aa0 0x7f9c98055f60 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7f9ca800b000 tx=0x7f9ca8006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:06:56.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.325+0000 7f9cb57fa700 1 -- 172.21.15.60:0/2695045451 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9cb0044490 con 0x7f9cb813fdb0 2022-06-29T16:06:56.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.489+0000 7f9cbd0ec700 1 -- 172.21.15.60:0/2695045451 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7f9cb80573d0 con 0x7f9cb813fdb0 2022-06-29T16:06:56.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.489+0000 7f9cb57fa700 1 -- 172.21.15.60:0/2695045451 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1594 (secure 0 0 0) 0x7f9cb0047ab0 con 0x7f9cb813fdb0 2022-06-29T16:06:56.497 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:06:56.497 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:06:56.497 INFO:teuthology.orchestra.run.smithi060.stdout: "stats": { 2022-06-29T16:06:56.497 INFO:teuthology.orchestra.run.smithi060.stdout: "total_bytes": 767585320960, 2022-06-29T16:06:56.498 INFO:teuthology.orchestra.run.smithi060.stdout: "total_avail_bytes": 761051815936, 2022-06-29T16:06:56.498 INFO:teuthology.orchestra.run.smithi060.stdout: "total_used_bytes": 6533193728, 2022-06-29T16:06:56.498 INFO:teuthology.orchestra.run.smithi060.stdout: "total_used_raw_bytes": 6533505024, 2022-06-29T16:06:56.498 INFO:teuthology.orchestra.run.smithi060.stdout: "total_used_raw_ratio": 0.0085117639973759651, 2022-06-29T16:06:56.499 INFO:teuthology.orchestra.run.smithi060.stdout: "num_osds": 8, 2022-06-29T16:06:56.499 INFO:teuthology.orchestra.run.smithi060.stdout: "num_per_pool_osds": 0, 2022-06-29T16:06:56.499 INFO:teuthology.orchestra.run.smithi060.stdout: "num_per_pool_omap_osds": 0 2022-06-29T16:06:56.499 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:56.500 INFO:teuthology.orchestra.run.smithi060.stdout: "stats_by_class": { 2022-06-29T16:06:56.500 INFO:teuthology.orchestra.run.smithi060.stdout: "ssd": { 2022-06-29T16:06:56.500 INFO:teuthology.orchestra.run.smithi060.stdout: "total_bytes": 767585320960, 2022-06-29T16:06:56.500 INFO:teuthology.orchestra.run.smithi060.stdout: "total_avail_bytes": 761051815936, 2022-06-29T16:06:56.501 INFO:teuthology.orchestra.run.smithi060.stdout: "total_used_bytes": 6533193728, 2022-06-29T16:06:56.501 INFO:teuthology.orchestra.run.smithi060.stdout: "total_used_raw_bytes": 6533505024, 2022-06-29T16:06:56.501 INFO:teuthology.orchestra.run.smithi060.stdout: "total_used_raw_ratio": 0.0085117639973759651 2022-06-29T16:06:56.501 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:56.501 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:56.502 INFO:teuthology.orchestra.run.smithi060.stdout: "pools": [ 2022-06-29T16:06:56.502 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:56.502 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "device_health_metrics", 2022-06-29T16:06:56.502 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1, 2022-06-29T16:06:56.503 INFO:teuthology.orchestra.run.smithi060.stdout: "stats": { 2022-06-29T16:06:56.503 INFO:teuthology.orchestra.run.smithi060.stdout: "stored": 0, 2022-06-29T16:06:56.503 INFO:teuthology.orchestra.run.smithi060.stdout: "objects": 2, 2022-06-29T16:06:56.503 INFO:teuthology.orchestra.run.smithi060.stdout: "kb_used": 0, 2022-06-29T16:06:56.504 INFO:teuthology.orchestra.run.smithi060.stdout: "bytes_used": 0, 2022-06-29T16:06:56.504 INFO:teuthology.orchestra.run.smithi060.stdout: "percent_used": 0, 2022-06-29T16:06:56.504 INFO:teuthology.orchestra.run.smithi060.stdout: "max_avail": 361335947264 2022-06-29T16:06:56.504 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:56.505 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:56.505 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:56.505 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rbd", 2022-06-29T16:06:56.505 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2, 2022-06-29T16:06:56.506 INFO:teuthology.orchestra.run.smithi060.stdout: "stats": { 2022-06-29T16:06:56.506 INFO:teuthology.orchestra.run.smithi060.stdout: "stored": 0, 2022-06-29T16:06:56.506 INFO:teuthology.orchestra.run.smithi060.stdout: "objects": 0, 2022-06-29T16:06:56.506 INFO:teuthology.orchestra.run.smithi060.stdout: "kb_used": 0, 2022-06-29T16:06:56.507 INFO:teuthology.orchestra.run.smithi060.stdout: "bytes_used": 0, 2022-06-29T16:06:56.507 INFO:teuthology.orchestra.run.smithi060.stdout: "percent_used": 0, 2022-06-29T16:06:56.507 INFO:teuthology.orchestra.run.smithi060.stdout: "max_avail": 361335947264 2022-06-29T16:06:56.507 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:56.507 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:06:56.508 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:06:56.508 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cephfs_data", 2022-06-29T16:06:56.508 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 22, 2022-06-29T16:06:56.508 INFO:teuthology.orchestra.run.smithi060.stdout: "stats": { 2022-06-29T16:06:56.509 INFO:teuthology.orchestra.run.smithi060.stdout: "stored": 0, 2022-06-29T16:06:56.509 INFO:teuthology.orchestra.run.smithi060.stdout: "objects": 0, 2022-06-29T16:06:56.509 INFO:teuthology.orchestra.run.smithi060.stdout: "kb_used": 0, 2022-06-29T16:06:56.509 INFO:teuthology.orchestra.run.smithi060.stdout: "bytes_used": 0, 2022-06-29T16:06:56.510 INFO:teuthology.orchestra.run.smithi060.stdout: "percent_used": 0, 2022-06-29T16:06:56.510 INFO:teuthology.orchestra.run.smithi060.stdout: "max_avail": 361335947264 2022-06-29T16:06:56.510 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:56.510 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:06:56.511 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:06:56.511 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:06:56.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.493+0000 7f9cbd0ec700 1 -- 172.21.15.60:0/2695045451 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9c98053aa0 msgr2=0x7f9c98055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:56.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.493+0000 7f9cbd0ec700 1 --2- 172.21.15.60:0/2695045451 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9c98053aa0 0x7f9c98055f60 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7f9ca800b000 tx=0x7f9ca8006df0).stop 2022-06-29T16:06:56.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.493+0000 7f9cbd0ec700 1 -- 172.21.15.60:0/2695045451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9cb813fdb0 msgr2=0x7f9cb8096d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:06:56.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.493+0000 7f9cbd0ec700 1 --2- 172.21.15.60:0/2695045451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9cb813fdb0 0x7f9cb8096d60 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 rx=0x7f9cb00032f0 tx=0x7f9cb80ab540).stop 2022-06-29T16:06:56.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.493+0000 7f9cbd0ec700 1 -- 172.21.15.60:0/2695045451 shutdown_connections 2022-06-29T16:06:56.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.493+0000 7f9cbd0ec700 1 --2- 172.21.15.60:0/2695045451 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9c98053aa0 0x7f9c98055f60 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:56.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.493+0000 7f9cbd0ec700 1 --2- 172.21.15.60:0/2695045451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9cb813fdb0 0x7f9cb8096d60 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:56.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.493+0000 7f9cbd0ec700 1 --2- 172.21.15.60:0/2695045451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9cb80aba10 0x7f9cb8064af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:56.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.493+0000 7f9cbd0ec700 1 --2- 172.21.15.60:0/2695045451 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9cb8097b40 0x7f9cb809a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:06:56.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.493+0000 7f9cbd0ec700 1 -- 172.21.15.60:0/2695045451 >> 172.21.15.60:0/2695045451 conn(0x7f9cb80b5180 msgr2=0x7f9cb805bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:06:56.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.493+0000 7f9cbd0ec700 1 -- 172.21.15.60:0/2695045451 shutdown_connections 2022-06-29T16:06:56.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:06:56.493+0000 7f9cbd0ec700 1 -- 172.21.15.60:0/2695045451 wait complete. 2022-06-29T16:06:56.516 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30)... 2022-06-29T16:06:59.549 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:06:59.552 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:06:59.555 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:06:59.557 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:06:59.560 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:07:01.517 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2022-06-29T16:07:01.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.621+0000 7f9593fff700 1 Processor -- start 2022-06-29T16:07:01.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.621+0000 7f9593fff700 1 -- start start 2022-06-29T16:07:01.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.625+0000 7f9593fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f95940b5eb0 0x7f9594142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:01.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.625+0000 7f9593fff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95940b0cf0 con 0x7f95940b2c50 2022-06-29T16:07:01.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.625+0000 7f9593fff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95940b4d10 con 0x7f95940b48b0 2022-06-29T16:07:01.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.625+0000 7f9593fff700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95940b54e0 con 0x7f95940b5eb0 2022-06-29T16:07:01.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.625+0000 7f9592ffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f95940b5eb0 0x7f9594142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:01.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.625+0000 7f95937fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f95940b2c50 0x7f959414d380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60522/0 (socket says 172.21.15.60:60522) 2022-06-29T16:07:01.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.625+0000 7f95937fe700 1 -- 172.21.15.60:0/1216233333 learned_addr learned my addr 172.21.15.60:0/1216233333 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:01.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.625+0000 7f9592ffd700 1 -- 172.21.15.60:0/1216233333 >> v1:172.21.15.92:6789/0 conn(0x7f95940b48b0 legacy=0x7f95940b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:01.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.625+0000 7f9592ffd700 1 -- 172.21.15.60:0/1216233333 >> v1:172.21.15.60:6789/0 conn(0x7f95940b2c50 legacy=0x7f959414d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:01.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.625+0000 7f9592ffd700 1 -- 172.21.15.60:0/1216233333 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95940b6290 con 0x7f95940b5eb0 2022-06-29T16:07:01.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.625+0000 7f9592ffd700 1 --2- 172.21.15.60:0/1216233333 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95940b5eb0 0x7f9594142a20 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 rx=0x7f9584002f70 tx=0x7f9584006df0).ready entity=mon.0 client_cookie=48c5f20bd4152b11 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:01.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.625+0000 7f95927fc700 1 -- 172.21.15.60:0/1216233333 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f958400d610 con 0x7f95940b5eb0 2022-06-29T16:07:01.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.625+0000 7f95927fc700 1 -- 172.21.15.60:0/1216233333 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9584000e90 con 0x7f95940b5eb0 2022-06-29T16:07:01.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.625+0000 7f95927fc700 1 -- 172.21.15.60:0/1216233333 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f958400d960 con 0x7f95940b5eb0 2022-06-29T16:07:01.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.625+0000 7f9593fff700 1 -- 172.21.15.60:0/1216233333 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95940b5eb0 msgr2=0x7f9594142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:01.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.625+0000 7f9593fff700 1 --2- 172.21.15.60:0/1216233333 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95940b5eb0 0x7f9594142a20 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 rx=0x7f9584002f70 tx=0x7f9584006df0).stop 2022-06-29T16:07:01.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.625+0000 7f9593fff700 1 -- 172.21.15.60:0/1216233333 shutdown_connections 2022-06-29T16:07:01.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.625+0000 7f9593fff700 1 --2- 172.21.15.60:0/1216233333 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95940b5eb0 0x7f9594142a20 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:01.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.625+0000 7f9593fff700 1 -- 172.21.15.60:0/1216233333 >> 172.21.15.60:0/1216233333 conn(0x7f9594062090 msgr2=0x7f9594062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:01.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.625+0000 7f9593fff700 1 -- 172.21.15.60:0/1216233333 shutdown_connections 2022-06-29T16:07:01.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.625+0000 7f9593fff700 1 -- 172.21.15.60:0/1216233333 wait complete. 2022-06-29T16:07:01.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.629+0000 7f9593fff700 1 Processor -- start 2022-06-29T16:07:01.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.629+0000 7f9593fff700 1 -- start start 2022-06-29T16:07:01.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.629+0000 7f9593fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95940b2c50 0x7f95940aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:01.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.629+0000 7f9593fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95940b48b0 0x7f9594155690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:01.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.629+0000 7f9593fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95940b5eb0 0x7f9594152880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:01.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.629+0000 7f9593fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95940af4a0 con 0x7f95940b2c50 2022-06-29T16:07:01.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.629+0000 7f9593fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95940af200 con 0x7f95940b5eb0 2022-06-29T16:07:01.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.629+0000 7f9593fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9594155d50 con 0x7f95940b48b0 2022-06-29T16:07:01.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.629+0000 7f9592ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95940b2c50 0x7f95940aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:01.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.629+0000 7f9592ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95940b2c50 0x7f95940aca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52438/0 (socket says 172.21.15.60:52438) 2022-06-29T16:07:01.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.629+0000 7f9592ffd700 1 -- 172.21.15.60:0/4058209197 learned_addr learned my addr 172.21.15.60:0/4058209197 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:01.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.633+0000 7f95937fe700 1 --2- 172.21.15.60:0/4058209197 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95940b5eb0 0x7f9594152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:01.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.633+0000 7f958a7fc700 1 --2- 172.21.15.60:0/4058209197 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95940b48b0 0x7f9594155690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:01.646 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.633+0000 7f9592ffd700 1 -- 172.21.15.60:0/4058209197 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95940b48b0 msgr2=0x7f9594155690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:01.646 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.633+0000 7f9592ffd700 1 --2- 172.21.15.60:0/4058209197 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95940b48b0 0x7f9594155690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:01.646 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.633+0000 7f9592ffd700 1 -- 172.21.15.60:0/4058209197 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95940b5eb0 msgr2=0x7f9594152880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:01.646 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.633+0000 7f9592ffd700 1 --2- 172.21.15.60:0/4058209197 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95940b5eb0 0x7f9594152880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:01.647 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.633+0000 7f9592ffd700 1 -- 172.21.15.60:0/4058209197 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f958400b180 con 0x7f95940b2c50 2022-06-29T16:07:01.647 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.633+0000 7f9592ffd700 1 --2- 172.21.15.60:0/4058209197 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95940b2c50 0x7f95940aca20 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 rx=0x7f9584002f70 tx=0x7f9584015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:01.647 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.633+0000 7f9590ff9700 1 -- 172.21.15.60:0/4058209197 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f958400d990 con 0x7f95940b2c50 2022-06-29T16:07:01.647 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.633+0000 7f9590ff9700 1 -- 172.21.15.60:0/4058209197 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9584011b80 con 0x7f95940b2c50 2022-06-29T16:07:01.647 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.633+0000 7f9593fff700 1 -- 172.21.15.60:0/4058209197 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95941561f0 con 0x7f95940b2c50 2022-06-29T16:07:01.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.633+0000 7f9590ff9700 1 -- 172.21.15.60:0/4058209197 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f958401e460 con 0x7f95940b2c50 2022-06-29T16:07:01.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.633+0000 7f9593fff700 1 -- 172.21.15.60:0/4058209197 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9594146d20 con 0x7f95940b2c50 2022-06-29T16:07:01.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.633+0000 7f9593fff700 1 -- 172.21.15.60:0/4058209197 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9594155b80 con 0x7f95940b2c50 2022-06-29T16:07:01.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.633+0000 7f9590ff9700 1 -- 172.21.15.60:0/4058209197 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9584025030 con 0x7f95940b2c50 2022-06-29T16:07:01.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.633+0000 7f9590ff9700 1 --2- 172.21.15.60:0/4058209197 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9578053b00 0x7f9578055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:01.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.633+0000 7f9590ff9700 1 -- 172.21.15.60:0/4058209197 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6075+0+0 (secure 0 0 0) 0x7f958406d1c0 con 0x7f95940b2c50 2022-06-29T16:07:01.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.637+0000 7f958a7fc700 1 --2- 172.21.15.60:0/4058209197 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9578053b00 0x7f9578055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:01.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.637+0000 7f958a7fc700 1 --2- 172.21.15.60:0/4058209197 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9578053b00 0x7f9578055fc0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0x7f957c00d000 tx=0x7f957c007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:01.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.641+0000 7f9590ff9700 1 -- 172.21.15.60:0/4058209197 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f95840440e0 con 0x7f95940b2c50 2022-06-29T16:07:01.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.809+0000 7f9593fff700 1 -- 172.21.15.60:0/4058209197 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7f95940573d0 con 0x7f95940b2c50 2022-06-29T16:07:01.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.813+0000 7f9590ff9700 1 -- 172.21.15.60:0/4058209197 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1937 (secure 0 0 0) 0x7f9584023090 con 0x7f95940b2c50 2022-06-29T16:07:01.816 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:01.817 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:07:01.817 INFO:teuthology.orchestra.run.smithi060.stdout: "stats": { 2022-06-29T16:07:01.817 INFO:teuthology.orchestra.run.smithi060.stdout: "total_bytes": 767585320960, 2022-06-29T16:07:01.818 INFO:teuthology.orchestra.run.smithi060.stdout: "total_avail_bytes": 761050505216, 2022-06-29T16:07:01.818 INFO:teuthology.orchestra.run.smithi060.stdout: "total_used_bytes": 6534381568, 2022-06-29T16:07:01.818 INFO:teuthology.orchestra.run.smithi060.stdout: "total_used_raw_bytes": 6534815744, 2022-06-29T16:07:01.819 INFO:teuthology.orchestra.run.smithi060.stdout: "total_used_raw_ratio": 0.0085134720429778099, 2022-06-29T16:07:01.819 INFO:teuthology.orchestra.run.smithi060.stdout: "num_osds": 8, 2022-06-29T16:07:01.819 INFO:teuthology.orchestra.run.smithi060.stdout: "num_per_pool_osds": 0, 2022-06-29T16:07:01.819 INFO:teuthology.orchestra.run.smithi060.stdout: "num_per_pool_omap_osds": 0 2022-06-29T16:07:01.819 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:01.820 INFO:teuthology.orchestra.run.smithi060.stdout: "stats_by_class": { 2022-06-29T16:07:01.820 INFO:teuthology.orchestra.run.smithi060.stdout: "ssd": { 2022-06-29T16:07:01.820 INFO:teuthology.orchestra.run.smithi060.stdout: "total_bytes": 767585320960, 2022-06-29T16:07:01.820 INFO:teuthology.orchestra.run.smithi060.stdout: "total_avail_bytes": 761050505216, 2022-06-29T16:07:01.821 INFO:teuthology.orchestra.run.smithi060.stdout: "total_used_bytes": 6534381568, 2022-06-29T16:07:01.821 INFO:teuthology.orchestra.run.smithi060.stdout: "total_used_raw_bytes": 6534815744, 2022-06-29T16:07:01.821 INFO:teuthology.orchestra.run.smithi060.stdout: "total_used_raw_ratio": 0.0085134720429778099 2022-06-29T16:07:01.821 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:01.822 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:01.822 INFO:teuthology.orchestra.run.smithi060.stdout: "pools": [ 2022-06-29T16:07:01.822 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:01.823 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "device_health_metrics", 2022-06-29T16:07:01.823 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1, 2022-06-29T16:07:01.823 INFO:teuthology.orchestra.run.smithi060.stdout: "stats": { 2022-06-29T16:07:01.823 INFO:teuthology.orchestra.run.smithi060.stdout: "stored": 0, 2022-06-29T16:07:01.823 INFO:teuthology.orchestra.run.smithi060.stdout: "objects": 2, 2022-06-29T16:07:01.824 INFO:teuthology.orchestra.run.smithi060.stdout: "kb_used": 0, 2022-06-29T16:07:01.824 INFO:teuthology.orchestra.run.smithi060.stdout: "bytes_used": 0, 2022-06-29T16:07:01.824 INFO:teuthology.orchestra.run.smithi060.stdout: "percent_used": 0, 2022-06-29T16:07:01.824 INFO:teuthology.orchestra.run.smithi060.stdout: "max_avail": 361335488512 2022-06-29T16:07:01.825 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:01.825 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:01.825 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:01.825 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rbd", 2022-06-29T16:07:01.825 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2, 2022-06-29T16:07:01.826 INFO:teuthology.orchestra.run.smithi060.stdout: "stats": { 2022-06-29T16:07:01.826 INFO:teuthology.orchestra.run.smithi060.stdout: "stored": 0, 2022-06-29T16:07:01.826 INFO:teuthology.orchestra.run.smithi060.stdout: "objects": 0, 2022-06-29T16:07:01.826 INFO:teuthology.orchestra.run.smithi060.stdout: "kb_used": 0, 2022-06-29T16:07:01.827 INFO:teuthology.orchestra.run.smithi060.stdout: "bytes_used": 0, 2022-06-29T16:07:01.827 INFO:teuthology.orchestra.run.smithi060.stdout: "percent_used": 0, 2022-06-29T16:07:01.827 INFO:teuthology.orchestra.run.smithi060.stdout: "max_avail": 361335488512 2022-06-29T16:07:01.827 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:01.828 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:01.828 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:01.828 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cephfs_data", 2022-06-29T16:07:01.828 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 22, 2022-06-29T16:07:01.828 INFO:teuthology.orchestra.run.smithi060.stdout: "stats": { 2022-06-29T16:07:01.829 INFO:teuthology.orchestra.run.smithi060.stdout: "stored": 0, 2022-06-29T16:07:01.829 INFO:teuthology.orchestra.run.smithi060.stdout: "objects": 0, 2022-06-29T16:07:01.829 INFO:teuthology.orchestra.run.smithi060.stdout: "kb_used": 0, 2022-06-29T16:07:01.830 INFO:teuthology.orchestra.run.smithi060.stdout: "bytes_used": 0, 2022-06-29T16:07:01.830 INFO:teuthology.orchestra.run.smithi060.stdout: "percent_used": 0, 2022-06-29T16:07:01.830 INFO:teuthology.orchestra.run.smithi060.stdout: "max_avail": 361335488512 2022-06-29T16:07:01.830 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:01.831 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:01.831 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:01.831 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cephfs_metadata", 2022-06-29T16:07:01.831 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 23, 2022-06-29T16:07:01.831 INFO:teuthology.orchestra.run.smithi060.stdout: "stats": { 2022-06-29T16:07:01.832 INFO:teuthology.orchestra.run.smithi060.stdout: "stored": 617, 2022-06-29T16:07:01.832 INFO:teuthology.orchestra.run.smithi060.stdout: "objects": 1, 2022-06-29T16:07:01.832 INFO:teuthology.orchestra.run.smithi060.stdout: "kb_used": 1, 2022-06-29T16:07:01.832 INFO:teuthology.orchestra.run.smithi060.stdout: "bytes_used": 617, 2022-06-29T16:07:01.833 INFO:teuthology.orchestra.run.smithi060.stdout: "percent_used": 8.5377721559254383e-10, 2022-06-29T16:07:01.833 INFO:teuthology.orchestra.run.smithi060.stdout: "max_avail": 361335488512 2022-06-29T16:07:01.833 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:01.833 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:01.833 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:01.834 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:07:01.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.813+0000 7f9593fff700 1 -- 172.21.15.60:0/4058209197 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9578053b00 msgr2=0x7f9578055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:01.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.813+0000 7f9593fff700 1 --2- 172.21.15.60:0/4058209197 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9578053b00 0x7f9578055fc0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0x7f957c00d000 tx=0x7f957c007b90).stop 2022-06-29T16:07:01.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.813+0000 7f9593fff700 1 -- 172.21.15.60:0/4058209197 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95940b2c50 msgr2=0x7f95940aca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:01.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.813+0000 7f9593fff700 1 --2- 172.21.15.60:0/4058209197 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95940b2c50 0x7f95940aca20 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 rx=0x7f9584002f70 tx=0x7f9584015000).stop 2022-06-29T16:07:01.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.813+0000 7f9593fff700 1 -- 172.21.15.60:0/4058209197 shutdown_connections 2022-06-29T16:07:01.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.813+0000 7f9593fff700 1 --2- 172.21.15.60:0/4058209197 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9578053b00 0x7f9578055fc0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:01.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.813+0000 7f9593fff700 1 --2- 172.21.15.60:0/4058209197 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95940b5eb0 0x7f9594152880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:01.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.813+0000 7f9593fff700 1 --2- 172.21.15.60:0/4058209197 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95940b48b0 0x7f9594155690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:01.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.813+0000 7f9593fff700 1 --2- 172.21.15.60:0/4058209197 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95940b2c50 0x7f95940aca20 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:01.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.813+0000 7f9593fff700 1 -- 172.21.15.60:0/4058209197 >> 172.21.15.60:0/4058209197 conn(0x7f9594062090 msgr2=0x7f9594062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:01.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.813+0000 7f9593fff700 1 -- 172.21.15.60:0/4058209197 shutdown_connections 2022-06-29T16:07:01.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.813+0000 7f9593fff700 1 -- 172.21.15.60:0/4058209197 wait complete. 2022-06-29T16:07:01.840 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s 2022-06-29T16:07:01.841 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2022-06-29T16:07:01.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.945+0000 7efdc7fff700 1 Processor -- start 2022-06-29T16:07:01.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.945+0000 7efdc7fff700 1 -- start start 2022-06-29T16:07:01.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc7fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7efdc805fb00 0x7efdc80aec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:01.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc7fff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc805fee0 con 0x7efdc8067b30 2022-06-29T16:07:01.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc7fff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc80b0df0 con 0x7efdc8140520 2022-06-29T16:07:01.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc7fff700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdc80af580 con 0x7efdc805fb00 2022-06-29T16:07:01.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc67fc700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7efdc8067b30 0x7efdc813fe10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60534/0 (socket says 172.21.15.60:60534) 2022-06-29T16:07:01.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc67fc700 1 -- 172.21.15.60:0/3078321949 learned_addr learned my addr 172.21.15.60:0/3078321949 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:01.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc6ffd700 1 --2- 172.21.15.60:0/3078321949 >> v2:172.21.15.92:3300/0 conn(0x7efdc805fb00 0x7efdc80aec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:01.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc5ffb700 1 -- 172.21.15.60:0/3078321949 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 855381171 0 0) 0x7efdc805fee0 con 0x7efdc8067b30 2022-06-29T16:07:01.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc5ffb700 1 -- 172.21.15.60:0/3078321949 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdac003260 con 0x7efdc8067b30 2022-06-29T16:07:01.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc5ffb700 1 -- 172.21.15.60:0/3078321949 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2237584313 0 0) 0x7efdac003260 con 0x7efdc8067b30 2022-06-29T16:07:01.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc5ffb700 1 -- 172.21.15.60:0/3078321949 >> v2:172.21.15.92:3300/0 conn(0x7efdc805fb00 msgr2=0x7efdc80aec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:01.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc5ffb700 1 --2- 172.21.15.60:0/3078321949 >> v2:172.21.15.92:3300/0 conn(0x7efdc805fb00 0x7efdc80aec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:01.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc5ffb700 1 -- 172.21.15.60:0/3078321949 >> v1:172.21.15.92:6789/0 conn(0x7efdc8140520 legacy=0x7efdc8143fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:01.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc5ffb700 1 -- 172.21.15.60:0/3078321949 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc80af820 con 0x7efdc8067b30 2022-06-29T16:07:01.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc5ffb700 1 -- 172.21.15.60:0/3078321949 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7efdbc001660 con 0x7efdc8067b30 2022-06-29T16:07:01.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc5ffb700 1 -- 172.21.15.60:0/3078321949 >> v1:172.21.15.60:6789/0 conn(0x7efdc8067b30 legacy=0x7efdc813fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:01.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc5ffb700 1 --2- 172.21.15.60:0/3078321949 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdac003750 0x7efdac003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:01.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc5ffb700 1 --2- 172.21.15.60:0/3078321949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdac004060 0x7efdac004520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:01.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc5ffb700 1 --2- 172.21.15.60:0/3078321949 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efdac004cd0 0x7efdac00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:01.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc5ffb700 1 -- 172.21.15.60:0/3078321949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc80af580 con 0x7efdac004060 2022-06-29T16:07:01.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc5ffb700 1 -- 172.21.15.60:0/3078321949 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdac00bc40 con 0x7efdac004cd0 2022-06-29T16:07:01.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc5ffb700 1 -- 172.21.15.60:0/3078321949 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdac00bd80 con 0x7efdac003750 2022-06-29T16:07:01.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc67fc700 1 --2- 172.21.15.60:0/3078321949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdac004060 0x7efdac004520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:01.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc6ffd700 1 --2- 172.21.15.60:0/3078321949 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdac003750 0x7efdac003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:01.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc77fe700 1 --2- 172.21.15.60:0/3078321949 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efdac004cd0 0x7efdac00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:01.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc67fc700 1 -- 172.21.15.60:0/3078321949 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdac003750 msgr2=0x7efdac003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:01.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc67fc700 1 --2- 172.21.15.60:0/3078321949 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdac003750 0x7efdac003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:01.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc67fc700 1 -- 172.21.15.60:0/3078321949 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efdac004cd0 msgr2=0x7efdac00b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:01.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc67fc700 1 --2- 172.21.15.60:0/3078321949 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efdac004cd0 0x7efdac00b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:01.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc67fc700 1 -- 172.21.15.60:0/3078321949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdac003260 con 0x7efdac004060 2022-06-29T16:07:01.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc77fe700 1 -- 172.21.15.60:0/3078321949 reap_dead start 2022-06-29T16:07:01.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc6ffd700 1 --2- 172.21.15.60:0/3078321949 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdac003750 0x7efdac003b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:01.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc67fc700 1 --2- 172.21.15.60:0/3078321949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdac004060 0x7efdac004520 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 rx=0x7efdac0054d0 tx=0x7efdc80b1580).ready entity=mon.0 client_cookie=a43a096bab9b7138 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:01.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc5ffb700 1 -- 172.21.15.60:0/3078321949 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efdbc0053d0 con 0x7efdac004060 2022-06-29T16:07:01.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc5ffb700 1 -- 172.21.15.60:0/3078321949 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdbc008170 con 0x7efdac004060 2022-06-29T16:07:01.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc5ffb700 1 -- 172.21.15.60:0/3078321949 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efdbc0084d0 con 0x7efdac004060 2022-06-29T16:07:01.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc7fff700 1 -- 172.21.15.60:0/3078321949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdac004060 msgr2=0x7efdac004520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:01.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc7fff700 1 --2- 172.21.15.60:0/3078321949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdac004060 0x7efdac004520 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 rx=0x7efdac0054d0 tx=0x7efdc80b1580).stop 2022-06-29T16:07:01.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc7fff700 1 -- 172.21.15.60:0/3078321949 shutdown_connections 2022-06-29T16:07:01.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc7fff700 1 --2- 172.21.15.60:0/3078321949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdac004060 0x7efdac004520 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:01.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.949+0000 7efdc7fff700 1 -- 172.21.15.60:0/3078321949 >> 172.21.15.60:0/3078321949 conn(0x7efdc8063a60 msgr2=0x7efdc805f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:01.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.953+0000 7efdc7fff700 1 -- 172.21.15.60:0/3078321949 shutdown_connections 2022-06-29T16:07:01.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.953+0000 7efdc7fff700 1 -- 172.21.15.60:0/3078321949 wait complete. 2022-06-29T16:07:01.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.953+0000 7efdc7fff700 1 Processor -- start 2022-06-29T16:07:01.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.953+0000 7efdc7fff700 1 -- start start 2022-06-29T16:07:01.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.953+0000 7efdc7fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdac004060 0x7efdc80acd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:01.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.953+0000 7efdc7fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdc8066570 0x7efdc81435c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:01.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.953+0000 7efdc7fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efdc8143ab0 0x7efdc8143f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:01.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.953+0000 7efdc7fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc81431f0 con 0x7efdc8066570 2022-06-29T16:07:01.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.953+0000 7efdc7fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc81420d0 con 0x7efdc8143ab0 2022-06-29T16:07:01.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.953+0000 7efdc7fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc8142570 con 0x7efdac004060 2022-06-29T16:07:01.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.953+0000 7efdc67fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdc8066570 0x7efdc81435c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:01.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.953+0000 7efdc67fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdc8066570 0x7efdc81435c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52458/0 (socket says 172.21.15.60:52458) 2022-06-29T16:07:01.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.953+0000 7efdc67fc700 1 -- 172.21.15.60:0/3254969166 learned_addr learned my addr 172.21.15.60:0/3254969166 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:01.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.953+0000 7efdc6ffd700 1 --2- 172.21.15.60:0/3254969166 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdac004060 0x7efdc80acd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:01.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.953+0000 7efdc77fe700 1 --2- 172.21.15.60:0/3254969166 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efdc8143ab0 0x7efdc8143f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:01.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.953+0000 7efdc67fc700 1 -- 172.21.15.60:0/3254969166 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdac004060 msgr2=0x7efdc80acd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:01.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.953+0000 7efdc67fc700 1 --2- 172.21.15.60:0/3254969166 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdac004060 0x7efdc80acd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:01.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.953+0000 7efdc67fc700 1 -- 172.21.15.60:0/3254969166 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efdc8143ab0 msgr2=0x7efdc8143f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:01.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.953+0000 7efdc67fc700 1 --2- 172.21.15.60:0/3254969166 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efdc8143ab0 0x7efdc8143f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:01.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.953+0000 7efdc67fc700 1 -- 172.21.15.60:0/3254969166 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc80ae9b0 con 0x7efdc8066570 2022-06-29T16:07:01.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.957+0000 7efdc67fc700 1 --2- 172.21.15.60:0/3254969166 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdc8066570 0x7efdc81435c0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 rx=0x7efdac0054d0 tx=0x7efdbc016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:01.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.957+0000 7efdb7fff700 1 -- 172.21.15.60:0/3254969166 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efdbc015810 con 0x7efdc8066570 2022-06-29T16:07:01.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.957+0000 7efdb7fff700 1 -- 172.21.15.60:0/3254969166 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdbc019070 con 0x7efdc8066570 2022-06-29T16:07:01.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.957+0000 7efdc7fff700 1 -- 172.21.15.60:0/3254969166 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc8066a00 con 0x7efdc8066570 2022-06-29T16:07:01.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.957+0000 7efdb7fff700 1 -- 172.21.15.60:0/3254969166 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efdbc015b60 con 0x7efdc8066570 2022-06-29T16:07:01.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.957+0000 7efdc7fff700 1 -- 172.21.15.60:0/3254969166 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc80ad540 con 0x7efdc8066570 2022-06-29T16:07:01.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.957+0000 7efdb7fff700 1 -- 172.21.15.60:0/3254969166 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7efdbc029070 con 0x7efdc8066570 2022-06-29T16:07:01.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.957+0000 7efdc7fff700 1 -- 172.21.15.60:0/3254969166 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdc8142290 con 0x7efdc8066570 2022-06-29T16:07:01.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.957+0000 7efdb7fff700 1 --2- 172.21.15.60:0/3254969166 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efda4053950 0x7efda4055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:01.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.957+0000 7efdb7fff700 1 -- 172.21.15.60:0/3254969166 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6075+0+0 (secure 0 0 0) 0x7efdbc06ce20 con 0x7efdc8066570 2022-06-29T16:07:01.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.957+0000 7efdc6ffd700 1 --2- 172.21.15.60:0/3254969166 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efda4053950 0x7efda4055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:01.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.957+0000 7efdc6ffd700 1 --2- 172.21.15.60:0/3254969166 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efda4053950 0x7efda4055e10 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 rx=0x7efdb800b000 tx=0x7efdb8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:01.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:01.965+0000 7efdb7fff700 1 -- 172.21.15.60:0/3254969166 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efdbc043ca0 con 0x7efdc8066570 2022-06-29T16:07:02.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.133+0000 7efdc7fff700 1 -- 172.21.15.60:0/3254969166 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7efdc8143330 con 0x7efdc8066570 2022-06-29T16:07:02.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.133+0000 7efdb7fff700 1 -- 172.21.15.60:0/3254969166 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=-22 pool 'cephfs_metadata' already contains some objects. Use an empty pool instead. v53) v1 ==== 209+0+0 (secure 0 0 0) 0x7efdbc0472c0 con 0x7efdc8066570 2022-06-29T16:07:02.139 INFO:teuthology.orchestra.run.smithi060.stderr:Error EINVAL: pool 'cephfs_metadata' already contains some objects. Use an empty pool instead. 2022-06-29T16:07:02.143 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-06-29T16:07:02.145 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-06-29T16:07:02.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.253+0000 7fc0fac5f700 1 Processor -- start 2022-06-29T16:07:02.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.253+0000 7fc0fac5f700 1 -- start start 2022-06-29T16:07:02.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.253+0000 7fc0fac5f700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc0f413a960 0x7fc0f4066190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:02.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.253+0000 7fc0fac5f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc0f405bf30 0x7fc0f40b3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:02.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.253+0000 7fc0fac5f700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc0f40b4020 0x7fc0f4143eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:02.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.257+0000 7fc0fac5f700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0f4065ac0 con 0x7fc0f413a960 2022-06-29T16:07:02.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.257+0000 7fc0fac5f700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0f405ca10 con 0x7fc0f40b4020 2022-06-29T16:07:02.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.257+0000 7fc0fac5f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f405ced0 con 0x7fc0f405bf30 2022-06-29T16:07:02.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.257+0000 7fc0f89fb700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc0f413a960 0x7fc0f4066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:02.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.257+0000 7fc0f91fc700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc0f40b4020 0x7fc0f4143eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:02.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.257+0000 7fc0ebfff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc0f405bf30 0x7fc0f40b3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:02.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.257+0000 7fc0f91fc700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc0f40b4020 0x7fc0f4143eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:58500/0 (socket says 172.21.15.60:58500) 2022-06-29T16:07:02.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.257+0000 7fc0f91fc700 1 -- 172.21.15.60:0/2003875861 learned_addr learned my addr 172.21.15.60:0/2003875861 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:02.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.257+0000 7fc0f89fb700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc0f413a960 0x7fc0f4066190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52468/0 (socket says 172.21.15.60:52468) 2022-06-29T16:07:02.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.257+0000 7fc0f91fc700 1 -- 172.21.15.60:0/2003875861 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc0f405bf30 msgr2=0x7fc0f40b3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:02.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.257+0000 7fc0f91fc700 1 --2- 172.21.15.60:0/2003875861 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc0f405bf30 0x7fc0f40b3b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:02.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.257+0000 7fc0f91fc700 1 -- 172.21.15.60:0/2003875861 >> v2:172.21.15.60:3300/0 conn(0x7fc0f413a960 msgr2=0x7fc0f4066190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:02.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.257+0000 7fc0f91fc700 1 --2- 172.21.15.60:0/2003875861 >> v2:172.21.15.60:3300/0 conn(0x7fc0f413a960 0x7fc0f4066190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:02.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.257+0000 7fc0f91fc700 1 -- 172.21.15.60:0/2003875861 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f4065cf0 con 0x7fc0f40b4020 2022-06-29T16:07:02.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.257+0000 7fc0f89fb700 1 --2- 172.21.15.60:0/2003875861 >> v2:172.21.15.60:3300/0 conn(0x7fc0f413a960 0x7fc0f4066190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:07:02.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.257+0000 7fc0f91fc700 1 --2- 172.21.15.60:0/2003875861 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0f40b4020 0x7fc0f4143eb0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0x7fc0ec0032f0 tx=0x7fc0ec009290).ready entity=mon.1 client_cookie=56135912482d95bd server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:02.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.257+0000 7fc0eb7fe700 1 -- 172.21.15.60:0/2003875861 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0ec00b680 con 0x7fc0f40b4020 2022-06-29T16:07:02.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.257+0000 7fc0eb7fe700 1 -- 172.21.15.60:0/2003875861 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0ec00b7e0 con 0x7fc0f40b4020 2022-06-29T16:07:02.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.257+0000 7fc0eb7fe700 1 -- 172.21.15.60:0/2003875861 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0ec00b950 con 0x7fc0f40b4020 2022-06-29T16:07:02.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.257+0000 7fc0fac5f700 1 -- 172.21.15.60:0/2003875861 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0f40b4020 msgr2=0x7fc0f4143eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:02.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.257+0000 7fc0fac5f700 1 --2- 172.21.15.60:0/2003875861 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0f40b4020 0x7fc0f4143eb0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0x7fc0ec0032f0 tx=0x7fc0ec009290).stop 2022-06-29T16:07:02.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.257+0000 7fc0fac5f700 1 -- 172.21.15.60:0/2003875861 shutdown_connections 2022-06-29T16:07:02.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.257+0000 7fc0fac5f700 1 --2- 172.21.15.60:0/2003875861 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0f40b4020 0x7fc0f4143eb0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:02.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.257+0000 7fc0fac5f700 1 --2- 172.21.15.60:0/2003875861 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc0f405bf30 0x7fc0f40b3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:02.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.257+0000 7fc0fac5f700 1 --2- 172.21.15.60:0/2003875861 >> v2:172.21.15.60:3300/0 conn(0x7fc0f413a960 0x7fc0f4066190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:02.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.257+0000 7fc0fac5f700 1 -- 172.21.15.60:0/2003875861 >> 172.21.15.60:0/2003875861 conn(0x7fc0f4064a10 msgr2=0x7fc0f4064df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:02.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.257+0000 7fc0fac5f700 1 -- 172.21.15.60:0/2003875861 shutdown_connections 2022-06-29T16:07:02.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.257+0000 7fc0fac5f700 1 -- 172.21.15.60:0/2003875861 wait complete. 2022-06-29T16:07:02.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.261+0000 7fc0fac5f700 1 Processor -- start 2022-06-29T16:07:02.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.261+0000 7fc0fac5f700 1 -- start start 2022-06-29T16:07:02.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.261+0000 7fc0fac5f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0f405bf30 0x7fc0f4141db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:02.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.261+0000 7fc0fac5f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc0f40b4020 0x7fc0f40ae510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:02.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.261+0000 7fc0fac5f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0f413a960 0x7fc0f40b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:02.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.261+0000 7fc0fac5f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f4135b50 con 0x7fc0f413a960 2022-06-29T16:07:02.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.261+0000 7fc0fac5f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f40afc80 con 0x7fc0f405bf30 2022-06-29T16:07:02.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.261+0000 7fc0fac5f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f4062920 con 0x7fc0f40b4020 2022-06-29T16:07:02.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.261+0000 7fc0f91fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0f413a960 0x7fc0f40b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:02.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.261+0000 7fc0f91fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0f413a960 0x7fc0f40b00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52476/0 (socket says 172.21.15.60:52476) 2022-06-29T16:07:02.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.261+0000 7fc0f91fc700 1 -- 172.21.15.60:0/549057435 learned_addr learned my addr 172.21.15.60:0/549057435 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:02.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.261+0000 7fc0ebfff700 1 --2- 172.21.15.60:0/549057435 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc0f40b4020 0x7fc0f40ae510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:02.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.261+0000 7fc0f89fb700 1 --2- 172.21.15.60:0/549057435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0f405bf30 0x7fc0f4141db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:02.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.261+0000 7fc0f91fc700 1 -- 172.21.15.60:0/549057435 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc0f40b4020 msgr2=0x7fc0f40ae510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:02.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.261+0000 7fc0f91fc700 1 --2- 172.21.15.60:0/549057435 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc0f40b4020 0x7fc0f40ae510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:02.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.261+0000 7fc0f91fc700 1 -- 172.21.15.60:0/549057435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0f405bf30 msgr2=0x7fc0f4141db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:02.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.261+0000 7fc0f91fc700 1 --2- 172.21.15.60:0/549057435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0f405bf30 0x7fc0f4141db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:02.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.261+0000 7fc0f91fc700 1 -- 172.21.15.60:0/549057435 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f000b180 con 0x7fc0f413a960 2022-06-29T16:07:02.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.265+0000 7fc0f91fc700 1 --2- 172.21.15.60:0/549057435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0f413a960 0x7fc0f40b00e0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 rx=0x7fc0ec0032f0 tx=0x7fc0ec017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:02.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.265+0000 7fc0e9ffb700 1 -- 172.21.15.60:0/549057435 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0ec00b680 con 0x7fc0f413a960 2022-06-29T16:07:02.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.265+0000 7fc0e9ffb700 1 -- 172.21.15.60:0/549057435 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0ec00bea0 con 0x7fc0f413a960 2022-06-29T16:07:02.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.265+0000 7fc0fac5f700 1 -- 172.21.15.60:0/549057435 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0ec009530 con 0x7fc0f413a960 2022-06-29T16:07:02.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.265+0000 7fc0e9ffb700 1 -- 172.21.15.60:0/549057435 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0ec01b600 con 0x7fc0f413a960 2022-06-29T16:07:02.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.265+0000 7fc0fac5f700 1 -- 172.21.15.60:0/549057435 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0f40aea00 con 0x7fc0f413a960 2022-06-29T16:07:02.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.265+0000 7fc0fac5f700 1 -- 172.21.15.60:0/549057435 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0f40afdf0 con 0x7fc0f413a960 2022-06-29T16:07:02.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.265+0000 7fc0e9ffb700 1 -- 172.21.15.60:0/549057435 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc0ec00f070 con 0x7fc0f413a960 2022-06-29T16:07:02.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.265+0000 7fc0e9ffb700 1 --2- 172.21.15.60:0/549057435 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc0d4053aa0 0x7fc0d4055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:02.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.265+0000 7fc0e9ffb700 1 -- 172.21.15.60:0/549057435 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6075+0+0 (secure 0 0 0) 0x7fc0ec06d140 con 0x7fc0f413a960 2022-06-29T16:07:02.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.265+0000 7fc0f89fb700 1 --2- 172.21.15.60:0/549057435 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc0d4053aa0 0x7fc0d4055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:02.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.269+0000 7fc0f89fb700 1 --2- 172.21.15.60:0/549057435 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc0d4053aa0 0x7fc0d4055f60 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7fc0f000b000 tx=0x7fc0f0006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:02.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.273+0000 7fc0e9ffb700 1 -- 172.21.15.60:0/549057435 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc0ec044110 con 0x7fc0f413a960 2022-06-29T16:07:02.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.437+0000 7fc0fac5f700 1 -- 172.21.15.60:0/549057435 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7fc0f40573d0 con 0x7fc0f413a960 2022-06-29T16:07:02.445 INFO:tasks.ceph.mon.a.smithi060.stderr:2022-06-29T16:07:02.441+0000 7fcc17b71700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-06-29T16:07:02.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.449+0000 7fc0e9ffb700 1 -- 172.21.15.60:0/549057435 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 23 and data pool 22 v54) v1 ==== 189+0+0 (secure 0 0 0) 0x7fc0ec01f070 con 0x7fc0f413a960 2022-06-29T16:07:02.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.449+0000 7fc0fac5f700 1 -- 172.21.15.60:0/549057435 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc0d4053aa0 msgr2=0x7fc0d4055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:02.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.449+0000 7fc0fac5f700 1 --2- 172.21.15.60:0/549057435 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc0d4053aa0 0x7fc0d4055f60 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7fc0f000b000 tx=0x7fc0f0006df0).stop 2022-06-29T16:07:02.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.449+0000 7fc0fac5f700 1 -- 172.21.15.60:0/549057435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0f413a960 msgr2=0x7fc0f40b00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:02.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.449+0000 7fc0fac5f700 1 --2- 172.21.15.60:0/549057435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0f413a960 0x7fc0f40b00e0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 rx=0x7fc0ec0032f0 tx=0x7fc0ec017000).stop 2022-06-29T16:07:02.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.449+0000 7fc0fac5f700 1 -- 172.21.15.60:0/549057435 shutdown_connections 2022-06-29T16:07:02.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.449+0000 7fc0fac5f700 1 --2- 172.21.15.60:0/549057435 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc0d4053aa0 0x7fc0d4055f60 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:02.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.449+0000 7fc0fac5f700 1 --2- 172.21.15.60:0/549057435 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc0f413a960 0x7fc0f40b00e0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:02.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.449+0000 7fc0fac5f700 1 --2- 172.21.15.60:0/549057435 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc0f40b4020 0x7fc0f40ae510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:02.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.449+0000 7fc0fac5f700 1 --2- 172.21.15.60:0/549057435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc0f405bf30 0x7fc0f4141db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:02.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.449+0000 7fc0fac5f700 1 -- 172.21.15.60:0/549057435 >> 172.21.15.60:0/549057435 conn(0x7fc0f4064a10 msgr2=0x7fc0f41396d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:02.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.449+0000 7fc0fac5f700 1 -- 172.21.15.60:0/549057435 shutdown_connections 2022-06-29T16:07:02.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.449+0000 7fc0fac5f700 1 -- 172.21.15.60:0/549057435 wait complete. 2022-06-29T16:07:02.458 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 23 and data pool 22 2022-06-29T16:07:02.466 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-06-29T16:07:02.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.573+0000 7f866359e700 1 Processor -- start 2022-06-29T16:07:02.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.573+0000 7f866359e700 1 -- start start 2022-06-29T16:07:02.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.573+0000 7f866359e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f866405fb00 0x7f8664143430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:02.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.573+0000 7f866359e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8664142fd0 con 0x7f866413fb30 2022-06-29T16:07:02.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.573+0000 7f866359e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8664067d70 con 0x7f8664062aa0 2022-06-29T16:07:02.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.573+0000 7f866359e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f866405fee0 con 0x7f866405fb00 2022-06-29T16:07:02.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.573+0000 7f866259c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f866405fb00 0x7f8664143430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:02.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.573+0000 7f8662d9d700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f866413fb30 0x7f86640ad130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60566/0 (socket says 172.21.15.60:60566) 2022-06-29T16:07:02.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.573+0000 7f8662d9d700 1 -- 172.21.15.60:0/1650321749 learned_addr learned my addr 172.21.15.60:0/1650321749 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:02.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.573+0000 7f866259c700 1 -- 172.21.15.60:0/1650321749 >> v1:172.21.15.92:6789/0 conn(0x7f8664062aa0 legacy=0x7f86640aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:02.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.573+0000 7f866259c700 1 -- 172.21.15.60:0/1650321749 >> v1:172.21.15.60:6789/0 conn(0x7f866413fb30 legacy=0x7f86640ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:02.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.573+0000 7f866259c700 1 -- 172.21.15.60:0/1650321749 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86640ad840 con 0x7f866405fb00 2022-06-29T16:07:02.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.573+0000 7f866259c700 1 --2- 172.21.15.60:0/1650321749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f866405fb00 0x7f8664143430 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 rx=0x7f8654002f70 tx=0x7f8654006df0).ready entity=mon.0 client_cookie=1092e493f56e216f server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:02.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.573+0000 7f866159a700 1 -- 172.21.15.60:0/1650321749 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f865400d610 con 0x7f866405fb00 2022-06-29T16:07:02.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.573+0000 7f866159a700 1 -- 172.21.15.60:0/1650321749 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8654000e90 con 0x7f866405fb00 2022-06-29T16:07:02.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.573+0000 7f866159a700 1 -- 172.21.15.60:0/1650321749 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f865400d960 con 0x7f866405fb00 2022-06-29T16:07:02.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.577+0000 7f866359e700 1 -- 172.21.15.60:0/1650321749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f866405fb00 msgr2=0x7f8664143430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:02.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.577+0000 7f866359e700 1 --2- 172.21.15.60:0/1650321749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f866405fb00 0x7f8664143430 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 rx=0x7f8654002f70 tx=0x7f8654006df0).stop 2022-06-29T16:07:02.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.577+0000 7f866359e700 1 -- 172.21.15.60:0/1650321749 shutdown_connections 2022-06-29T16:07:02.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.577+0000 7f866359e700 1 --2- 172.21.15.60:0/1650321749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f866405fb00 0x7f8664143430 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:02.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.577+0000 7f866359e700 1 -- 172.21.15.60:0/1650321749 >> 172.21.15.60:0/1650321749 conn(0x7f8664063a60 msgr2=0x7f866405f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:02.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.577+0000 7f866359e700 1 -- 172.21.15.60:0/1650321749 shutdown_connections 2022-06-29T16:07:02.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.577+0000 7f866359e700 1 -- 172.21.15.60:0/1650321749 wait complete. 2022-06-29T16:07:02.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.577+0000 7f866359e700 1 Processor -- start 2022-06-29T16:07:02.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.577+0000 7f866359e700 1 -- start start 2022-06-29T16:07:02.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.577+0000 7f866359e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f866405fb00 0x7f86641458a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:02.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.577+0000 7f866359e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8664062aa0 0x7f866410f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:02.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.577+0000 7f866359e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f866413fb30 0x7f8664065390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:02.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.577+0000 7f866359e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86640650f0 con 0x7f8664062aa0 2022-06-29T16:07:02.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.577+0000 7f866359e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f866412e300 con 0x7f866413fb30 2022-06-29T16:07:02.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.577+0000 7f866359e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f866410f2a0 con 0x7f866405fb00 2022-06-29T16:07:02.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.581+0000 7f8661d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8664062aa0 0x7f866410f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:02.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.581+0000 7f866259c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f866405fb00 0x7f86641458a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:02.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.581+0000 7f8661d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8664062aa0 0x7f866410f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52486/0 (socket says 172.21.15.60:52486) 2022-06-29T16:07:02.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.581+0000 7f8661d9b700 1 -- 172.21.15.60:0/788996480 learned_addr learned my addr 172.21.15.60:0/788996480 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:02.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.581+0000 7f8662d9d700 1 --2- 172.21.15.60:0/788996480 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f866413fb30 0x7f8664065390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:02.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.581+0000 7f8661d9b700 1 -- 172.21.15.60:0/788996480 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f866405fb00 msgr2=0x7f86641458a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:02.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.581+0000 7f8661d9b700 1 --2- 172.21.15.60:0/788996480 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f866405fb00 0x7f86641458a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:02.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.581+0000 7f8661d9b700 1 -- 172.21.15.60:0/788996480 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f866413fb30 msgr2=0x7f8664065390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:02.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.581+0000 7f8661d9b700 1 --2- 172.21.15.60:0/788996480 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f866413fb30 0x7f8664065390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:02.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.581+0000 7f8661d9b700 1 -- 172.21.15.60:0/788996480 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f865400b180 con 0x7f8664062aa0 2022-06-29T16:07:02.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.581+0000 7f8661d9b700 1 --2- 172.21.15.60:0/788996480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8664062aa0 0x7f866410f9e0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 rx=0x7f8664062710 tx=0x7f8658007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:02.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.581+0000 7f86537fe700 1 -- 172.21.15.60:0/788996480 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8658012400 con 0x7f8664062aa0 2022-06-29T16:07:02.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.581+0000 7f86537fe700 1 -- 172.21.15.60:0/788996480 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8658012560 con 0x7f8664062aa0 2022-06-29T16:07:02.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.581+0000 7f86537fe700 1 -- 172.21.15.60:0/788996480 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86580128c0 con 0x7f8664062aa0 2022-06-29T16:07:02.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.581+0000 7f866359e700 1 -- 172.21.15.60:0/788996480 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f866410f540 con 0x7f8664062aa0 2022-06-29T16:07:02.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.581+0000 7f866359e700 1 -- 172.21.15.60:0/788996480 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f866412d7c0 con 0x7f8664062aa0 2022-06-29T16:07:02.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.581+0000 7f866359e700 1 -- 172.21.15.60:0/788996480 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f866410f720 con 0x7f8664062aa0 2022-06-29T16:07:02.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.581+0000 7f86537fe700 1 -- 172.21.15.60:0/788996480 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8658019030 con 0x7f8664062aa0 2022-06-29T16:07:02.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.585+0000 7f86537fe700 1 --2- 172.21.15.60:0/788996480 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8640053b30 0x7f8640055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:02.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.585+0000 7f86537fe700 1 -- 172.21.15.60:0/788996480 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f865806b210 con 0x7f8664062aa0 2022-06-29T16:07:02.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.585+0000 7f866259c700 1 --2- 172.21.15.60:0/788996480 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8640053b30 0x7f8640055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:02.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.585+0000 7f866259c700 1 --2- 172.21.15.60:0/788996480 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8640053b30 0x7f8640055ff0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0x7f865401c000 tx=0x7f8654015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:02.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.589+0000 7f86537fe700 1 -- 172.21.15.60:0/788996480 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f866410f720 con 0x7f8664062aa0 2022-06-29T16:07:02.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:02.749+0000 7f866359e700 1 -- 172.21.15.60:0/788996480 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f8664067ad0 con 0x7f8664062aa0 2022-06-29T16:07:03.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.449+0000 7f86537fe700 1 -- 172.21.15.60:0/788996480 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v55) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8658042080 con 0x7f8664062aa0 2022-06-29T16:07:03.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.453+0000 7f866359e700 1 -- 172.21.15.60:0/788996480 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8640053b30 msgr2=0x7f8640055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:03.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.453+0000 7f866359e700 1 --2- 172.21.15.60:0/788996480 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8640053b30 0x7f8640055ff0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0x7f865401c000 tx=0x7f8654015000).stop 2022-06-29T16:07:03.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.453+0000 7f866359e700 1 -- 172.21.15.60:0/788996480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8664062aa0 msgr2=0x7f866410f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:03.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.453+0000 7f866359e700 1 --2- 172.21.15.60:0/788996480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8664062aa0 0x7f866410f9e0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 rx=0x7f8664062710 tx=0x7f8658007c50).stop 2022-06-29T16:07:03.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.453+0000 7f866359e700 1 -- 172.21.15.60:0/788996480 shutdown_connections 2022-06-29T16:07:03.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.453+0000 7f866359e700 1 --2- 172.21.15.60:0/788996480 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8640053b30 0x7f8640055ff0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:03.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.453+0000 7f866359e700 1 --2- 172.21.15.60:0/788996480 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f866413fb30 0x7f8664065390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:03.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.453+0000 7f866359e700 1 --2- 172.21.15.60:0/788996480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8664062aa0 0x7f866410f9e0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:03.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.453+0000 7f866359e700 1 --2- 172.21.15.60:0/788996480 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f866405fb00 0x7f86641458a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:03.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.453+0000 7f866359e700 1 -- 172.21.15.60:0/788996480 >> 172.21.15.60:0/788996480 conn(0x7f8664063a60 msgr2=0x7f866405f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:03.461 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.453+0000 7f866359e700 1 -- 172.21.15.60:0/788996480 shutdown_connections 2022-06-29T16:07:03.461 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.453+0000 7f866359e700 1 -- 172.21.15.60:0/788996480 wait complete. 2022-06-29T16:07:03.474 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-06-29T16:07:03.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.589+0000 7fb3dad5e700 1 Processor -- start 2022-06-29T16:07:03.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.589+0000 7fb3dad5e700 1 -- start start 2022-06-29T16:07:03.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.593+0000 7fb3dad5e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3d4066590 0x7fb3d40673e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:03.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.593+0000 7fb3dad5e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb3d40b1480 0x7fb3d40afa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:03.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.593+0000 7fb3dad5e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3d40b0f50 con 0x7fb3d4140300 2022-06-29T16:07:03.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.593+0000 7fb3dad5e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3d4067080 con 0x7fb3d40b1480 2022-06-29T16:07:03.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.593+0000 7fb3dad5e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d4065670 con 0x7fb3d4066590 2022-06-29T16:07:03.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.593+0000 7fb3da55d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb3d40b1480 0x7fb3d40afa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:03.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.593+0000 7fb3d9d5c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3d4066590 0x7fb3d40673e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:03.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.593+0000 7fb3da55d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb3d40b1480 0x7fb3d40afa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52496/0 (socket says 172.21.15.60:52496) 2022-06-29T16:07:03.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.593+0000 7fb3da55d700 1 -- 172.21.15.60:0/3386751363 learned_addr learned my addr 172.21.15.60:0/3386751363 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:03.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.593+0000 7fb3da55d700 1 -- 172.21.15.60:0/3386751363 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3d4066590 msgr2=0x7fb3d40673e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:03.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.593+0000 7fb3da55d700 1 --2- 172.21.15.60:0/3386751363 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3d4066590 0x7fb3d40673e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:03.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.593+0000 7fb3d8d5a700 1 -- 172.21.15.60:0/3386751363 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3766125345 0 0) 0x7fb3d0001110 con 0x7fb3d4140300 2022-06-29T16:07:03.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.593+0000 7fb3da55d700 1 -- 172.21.15.60:0/3386751363 >> v1:172.21.15.92:6789/0 conn(0x7fb3d4140300 legacy=0x7fb3d4066970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:03.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.593+0000 7fb3da55d700 1 -- 172.21.15.60:0/3386751363 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d4064db0 con 0x7fb3d40b1480 2022-06-29T16:07:03.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.593+0000 7fb3da55d700 1 --2- 172.21.15.60:0/3386751363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3d40b1480 0x7fb3d40afa50 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 rx=0x7fb3cc0032f0 tx=0x7fb3cc009290).ready entity=mon.0 client_cookie=971022ff6c9542da server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:03.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.593+0000 7fb3d8d5a700 1 -- 172.21.15.60:0/3386751363 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3cc00ba90 con 0x7fb3d40b1480 2022-06-29T16:07:03.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.593+0000 7fb3d8d5a700 1 -- 172.21.15.60:0/3386751363 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3cc00bbf0 con 0x7fb3d40b1480 2022-06-29T16:07:03.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.593+0000 7fb3d8d5a700 1 -- 172.21.15.60:0/3386751363 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3cc010470 con 0x7fb3d40b1480 2022-06-29T16:07:03.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.593+0000 7fb3dad5e700 1 -- 172.21.15.60:0/3386751363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3d40b1480 msgr2=0x7fb3d40afa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:03.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.593+0000 7fb3dad5e700 1 --2- 172.21.15.60:0/3386751363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3d40b1480 0x7fb3d40afa50 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 rx=0x7fb3cc0032f0 tx=0x7fb3cc009290).stop 2022-06-29T16:07:03.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.593+0000 7fb3dad5e700 1 -- 172.21.15.60:0/3386751363 shutdown_connections 2022-06-29T16:07:03.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.593+0000 7fb3dad5e700 1 --2- 172.21.15.60:0/3386751363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3d40b1480 0x7fb3d40afa50 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:03.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.593+0000 7fb3dad5e700 1 --2- 172.21.15.60:0/3386751363 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3d4066590 0x7fb3d40673e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:03.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.593+0000 7fb3dad5e700 1 -- 172.21.15.60:0/3386751363 >> 172.21.15.60:0/3386751363 conn(0x7fb3d40b5e70 msgr2=0x7fb3d40b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:03.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.593+0000 7fb3dad5e700 1 -- 172.21.15.60:0/3386751363 shutdown_connections 2022-06-29T16:07:03.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.593+0000 7fb3dad5e700 1 -- 172.21.15.60:0/3386751363 wait complete. 2022-06-29T16:07:03.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.597+0000 7fb3dad5e700 1 Processor -- start 2022-06-29T16:07:03.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.597+0000 7fb3dad5e700 1 -- start start 2022-06-29T16:07:03.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.597+0000 7fb3dad5e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3d4066590 0x7fb3d414e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:03.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.597+0000 7fb3dad5e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3d40b1480 0x7fb3d410f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:03.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.597+0000 7fb3dad5e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3d4140300 0x7fb3d410f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:03.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.597+0000 7fb3dad5e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d40ae720 con 0x7fb3d4140300 2022-06-29T16:07:03.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.597+0000 7fb3dad5e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d40ae950 con 0x7fb3d40b1480 2022-06-29T16:07:03.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.597+0000 7fb3dad5e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d40aebf0 con 0x7fb3d4066590 2022-06-29T16:07:03.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.597+0000 7fb3da55d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3d4140300 0x7fb3d410f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:03.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.597+0000 7fb3da55d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3d4140300 0x7fb3d410f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52502/0 (socket says 172.21.15.60:52502) 2022-06-29T16:07:03.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.597+0000 7fb3da55d700 1 -- 172.21.15.60:0/1121897847 learned_addr learned my addr 172.21.15.60:0/1121897847 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:03.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.597+0000 7fb3d955b700 1 --2- 172.21.15.60:0/1121897847 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3d40b1480 0x7fb3d410f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:03.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.597+0000 7fb3d9d5c700 1 --2- 172.21.15.60:0/1121897847 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3d4066590 0x7fb3d414e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:03.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.597+0000 7fb3d955b700 1 -- 172.21.15.60:0/1121897847 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3d4066590 msgr2=0x7fb3d414e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:03.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.597+0000 7fb3d955b700 1 --2- 172.21.15.60:0/1121897847 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3d4066590 0x7fb3d414e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:03.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.597+0000 7fb3d955b700 1 -- 172.21.15.60:0/1121897847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3d4140300 msgr2=0x7fb3d410f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:03.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.597+0000 7fb3d955b700 1 --2- 172.21.15.60:0/1121897847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3d4140300 0x7fb3d410f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:03.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.597+0000 7fb3d955b700 1 -- 172.21.15.60:0/1121897847 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3cc009530 con 0x7fb3d40b1480 2022-06-29T16:07:03.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.597+0000 7fb3d955b700 1 --2- 172.21.15.60:0/1121897847 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3d40b1480 0x7fb3d410f3a0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7fb3d0000ed0 tx=0x7fb3d0007b90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:03.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.601+0000 7fb3c6ffd700 1 -- 172.21.15.60:0/1121897847 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3d000cea0 con 0x7fb3d40b1480 2022-06-29T16:07:03.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.601+0000 7fb3c6ffd700 1 -- 172.21.15.60:0/1121897847 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3d0012400 con 0x7fb3d40b1480 2022-06-29T16:07:03.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.601+0000 7fb3dad5e700 1 -- 172.21.15.60:0/1121897847 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3d40afa00 con 0x7fb3d40b1480 2022-06-29T16:07:03.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.601+0000 7fb3c6ffd700 1 -- 172.21.15.60:0/1121897847 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3d0012760 con 0x7fb3d40b1480 2022-06-29T16:07:03.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.601+0000 7fb3dad5e700 1 -- 172.21.15.60:0/1121897847 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3d40acc40 con 0x7fb3d40b1480 2022-06-29T16:07:03.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.601+0000 7fb3dad5e700 1 -- 172.21.15.60:0/1121897847 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3d40aedb0 con 0x7fb3d40b1480 2022-06-29T16:07:03.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.601+0000 7fb3c6ffd700 1 -- 172.21.15.60:0/1121897847 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb3d0019030 con 0x7fb3d40b1480 2022-06-29T16:07:03.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.601+0000 7fb3c6ffd700 1 --2- 172.21.15.60:0/1121897847 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb3b40539e0 0x7fb3b4055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:03.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.601+0000 7fb3c6ffd700 1 -- 172.21.15.60:0/1121897847 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fb3d006b120 con 0x7fb3d40b1480 2022-06-29T16:07:03.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.601+0000 7fb3d9d5c700 1 --2- 172.21.15.60:0/1121897847 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb3b40539e0 0x7fb3b4055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:03.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.605+0000 7fb3d9d5c700 1 --2- 172.21.15.60:0/1121897847 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb3b40539e0 0x7fb3b4055ea0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0x7fb3c800b000 tx=0x7fb3c8006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:03.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.613+0000 7fb3c6ffd700 1 -- 172.21.15.60:0/1121897847 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb3d0041ff0 con 0x7fb3d40b1480 2022-06-29T16:07:03.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:03.745+0000 7fb3dad5e700 1 -- 172.21.15.60:0/1121897847 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb3d40b0f50 con 0x7fb3d40b1480 2022-06-29T16:07:04.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.457+0000 7fb3c6ffd700 1 -- 172.21.15.60:0/1121897847 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v118) v1 ==== 185+0+0 (secure 0 0 0) 0x7fb3d40b0f50 con 0x7fb3d40b1480 2022-06-29T16:07:04.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.461+0000 7fb3dad5e700 1 -- 172.21.15.60:0/1121897847 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb3b40539e0 msgr2=0x7fb3b4055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:04.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.461+0000 7fb3dad5e700 1 --2- 172.21.15.60:0/1121897847 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb3b40539e0 0x7fb3b4055ea0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0x7fb3c800b000 tx=0x7fb3c8006df0).stop 2022-06-29T16:07:04.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.461+0000 7fb3dad5e700 1 -- 172.21.15.60:0/1121897847 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3d40b1480 msgr2=0x7fb3d410f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:04.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.461+0000 7fb3dad5e700 1 --2- 172.21.15.60:0/1121897847 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3d40b1480 0x7fb3d410f3a0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7fb3d0000ed0 tx=0x7fb3d0007b90).stop 2022-06-29T16:07:04.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.461+0000 7fb3dad5e700 1 -- 172.21.15.60:0/1121897847 shutdown_connections 2022-06-29T16:07:04.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.461+0000 7fb3dad5e700 1 --2- 172.21.15.60:0/1121897847 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb3b40539e0 0x7fb3b4055ea0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:04.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.461+0000 7fb3dad5e700 1 --2- 172.21.15.60:0/1121897847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3d4140300 0x7fb3d410f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:04.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.461+0000 7fb3dad5e700 1 --2- 172.21.15.60:0/1121897847 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3d40b1480 0x7fb3d410f3a0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:04.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.461+0000 7fb3dad5e700 1 --2- 172.21.15.60:0/1121897847 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb3d4066590 0x7fb3d414e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:04.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.461+0000 7fb3dad5e700 1 -- 172.21.15.60:0/1121897847 >> 172.21.15.60:0/1121897847 conn(0x7fb3d40b5e70 msgr2=0x7fb3d413e560 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:04.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.461+0000 7fb3dad5e700 1 -- 172.21.15.60:0/1121897847 shutdown_connections 2022-06-29T16:07:04.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.461+0000 7fb3dad5e700 1 -- 172.21.15.60:0/1121897847 wait complete. 2022-06-29T16:07:04.470 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2022-06-29T16:07:04.482 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-06-29T16:07:04.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.593+0000 7f87ee82d700 1 Processor -- start 2022-06-29T16:07:04.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.593+0000 7f87ee82d700 1 -- start start 2022-06-29T16:07:04.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.593+0000 7f87ee82d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87e80b61d0 0x7f87e80b0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:04.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.593+0000 7f87ee82d700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f87e813f790 0x7f87e813bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:04.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.593+0000 7f87ee82d700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87e8142820 con 0x7f87e813fbd0 2022-06-29T16:07:04.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.593+0000 7f87ee82d700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87e8062bb0 con 0x7f87e813f790 2022-06-29T16:07:04.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.593+0000 7f87ee82d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e80b65b0 con 0x7f87e80b61d0 2022-06-29T16:07:04.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.593+0000 7f87ecdca700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f87e813fbd0 0x7f87e813c130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60594/0 (socket says 172.21.15.60:60594) 2022-06-29T16:07:04.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.593+0000 7f87ecdca700 1 -- 172.21.15.60:0/2784109182 learned_addr learned my addr 172.21.15.60:0/2784109182 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:04.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.593+0000 7f87e7fff700 1 --2- 172.21.15.60:0/2784109182 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87e80b61d0 0x7f87e80b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:04.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.593+0000 7f87e77fe700 1 --2- 172.21.15.60:0/2784109182 >> v2:172.21.15.92:3300/0 conn(0x7f87e813f790 0x7f87e813bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:04.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.593+0000 7f87e6ffd700 1 -- 172.21.15.60:0/2784109182 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2195163392 0 0) 0x7f87e8142820 con 0x7f87e813fbd0 2022-06-29T16:07:04.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.593+0000 7f87e6ffd700 1 -- 172.21.15.60:0/2784109182 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87c8003260 con 0x7f87e813fbd0 2022-06-29T16:07:04.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.593+0000 7f87e7fff700 1 -- 172.21.15.60:0/2784109182 >> v2:172.21.15.92:3300/0 conn(0x7f87e813f790 msgr2=0x7f87e813bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:04.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.593+0000 7f87e7fff700 1 --2- 172.21.15.60:0/2784109182 >> v2:172.21.15.92:3300/0 conn(0x7f87e813f790 0x7f87e813bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:04.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.593+0000 7f87e7fff700 1 -- 172.21.15.60:0/2784109182 >> v1:172.21.15.60:6789/0 conn(0x7f87e813fbd0 legacy=0x7f87e813c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:04.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.593+0000 7f87e7fff700 1 -- 172.21.15.60:0/2784109182 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e813f050 con 0x7f87e80b61d0 2022-06-29T16:07:04.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.597+0000 7f87e77fe700 1 --2- 172.21.15.60:0/2784109182 >> v2:172.21.15.92:3300/0 conn(0x7f87e813f790 0x7f87e813bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:07:04.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.597+0000 7f87e7fff700 1 --2- 172.21.15.60:0/2784109182 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87e80b61d0 0x7f87e80b0540 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f87d80032f0 tx=0x7f87d80091d0).ready entity=mon.2 client_cookie=271cdf6594aa2f42 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:04.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.597+0000 7f87e6ffd700 1 -- 172.21.15.60:0/2784109182 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f87d800b600 con 0x7f87e80b61d0 2022-06-29T16:07:04.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.597+0000 7f87e6ffd700 1 -- 172.21.15.60:0/2784109182 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d800b760 con 0x7f87e80b61d0 2022-06-29T16:07:04.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.597+0000 7f87e6ffd700 1 -- 172.21.15.60:0/2784109182 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f87d800b8d0 con 0x7f87e80b61d0 2022-06-29T16:07:04.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.597+0000 7f87ee82d700 1 -- 172.21.15.60:0/2784109182 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87e80b61d0 msgr2=0x7f87e80b0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:04.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.597+0000 7f87ee82d700 1 --2- 172.21.15.60:0/2784109182 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87e80b61d0 0x7f87e80b0540 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f87d80032f0 tx=0x7f87d80091d0).stop 2022-06-29T16:07:04.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.597+0000 7f87ee82d700 1 -- 172.21.15.60:0/2784109182 shutdown_connections 2022-06-29T16:07:04.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.597+0000 7f87ee82d700 1 --2- 172.21.15.60:0/2784109182 >> v2:172.21.15.92:3300/0 conn(0x7f87e813f790 0x7f87e813bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:04.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.597+0000 7f87ee82d700 1 --2- 172.21.15.60:0/2784109182 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87e80b61d0 0x7f87e80b0540 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:04.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.597+0000 7f87ee82d700 1 -- 172.21.15.60:0/2784109182 >> 172.21.15.60:0/2784109182 conn(0x7f87e8063b50 msgr2=0x7f87e80b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:04.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.597+0000 7f87ee82d700 1 -- 172.21.15.60:0/2784109182 shutdown_connections 2022-06-29T16:07:04.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.597+0000 7f87ee82d700 1 -- 172.21.15.60:0/2784109182 wait complete. 2022-06-29T16:07:04.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.597+0000 7f87ee82d700 1 Processor -- start 2022-06-29T16:07:04.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.597+0000 7f87ee82d700 1 -- start start 2022-06-29T16:07:04.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.601+0000 7f87ee82d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87e80b61d0 0x7f87e8065730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:04.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.601+0000 7f87ee82d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f87e813f790 0x7f87e8133f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:04.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.601+0000 7f87ee82d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87e813fbd0 0x7f87e8134410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:04.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.601+0000 7f87ee82d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e8134900 con 0x7f87e80b61d0 2022-06-29T16:07:04.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.601+0000 7f87ee82d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e8065f80 con 0x7f87e813f790 2022-06-29T16:07:04.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.601+0000 7f87ee82d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e80641f0 con 0x7f87e813fbd0 2022-06-29T16:07:04.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.601+0000 7f87e7fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87e80b61d0 0x7f87e8065730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:04.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.601+0000 7f87e7fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87e80b61d0 0x7f87e8065730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52512/0 (socket says 172.21.15.60:52512) 2022-06-29T16:07:04.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.601+0000 7f87e7fff700 1 -- 172.21.15.60:0/1960192778 learned_addr learned my addr 172.21.15.60:0/1960192778 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:04.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.601+0000 7f87e77fe700 1 --2- 172.21.15.60:0/1960192778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f87e813f790 0x7f87e8133f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:04.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.601+0000 7f87ecdca700 1 --2- 172.21.15.60:0/1960192778 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87e813fbd0 0x7f87e8134410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:04.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.601+0000 7f87e7fff700 1 -- 172.21.15.60:0/1960192778 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87e813fbd0 msgr2=0x7f87e8134410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:04.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.601+0000 7f87e7fff700 1 --2- 172.21.15.60:0/1960192778 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87e813fbd0 0x7f87e8134410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:04.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.601+0000 7f87e7fff700 1 -- 172.21.15.60:0/1960192778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f87e813f790 msgr2=0x7f87e8133f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:04.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.601+0000 7f87e7fff700 1 --2- 172.21.15.60:0/1960192778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f87e813f790 0x7f87e8133f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:04.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.601+0000 7f87e7fff700 1 -- 172.21.15.60:0/1960192778 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d000b180 con 0x7f87e80b61d0 2022-06-29T16:07:04.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.601+0000 7f87e7fff700 1 --2- 172.21.15.60:0/1960192778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87e80b61d0 0x7f87e8065730 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 rx=0x7f87d80032f0 tx=0x7f87d8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:04.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.601+0000 7f87e57fa700 1 -- 172.21.15.60:0/1960192778 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f87d8011b60 con 0x7f87e80b61d0 2022-06-29T16:07:04.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.601+0000 7f87e57fa700 1 -- 172.21.15.60:0/1960192778 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d8011cc0 con 0x7f87e80b61d0 2022-06-29T16:07:04.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.601+0000 7f87ee82d700 1 -- 172.21.15.60:0/1960192778 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87d8009470 con 0x7f87e80b61d0 2022-06-29T16:07:04.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.601+0000 7f87e57fa700 1 -- 172.21.15.60:0/1960192778 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f87d801f5e0 con 0x7f87e80b61d0 2022-06-29T16:07:04.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.601+0000 7f87ee82d700 1 -- 172.21.15.60:0/1960192778 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87e8065c20 con 0x7f87e80b61d0 2022-06-29T16:07:04.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.605+0000 7f87ee82d700 1 -- 172.21.15.60:0/1960192778 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87e8064830 con 0x7f87e80b61d0 2022-06-29T16:07:04.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.605+0000 7f87e57fa700 1 -- 172.21.15.60:0/1960192778 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f87d8026030 con 0x7f87e80b61d0 2022-06-29T16:07:04.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.605+0000 7f87e57fa700 1 --2- 172.21.15.60:0/1960192778 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f87cc053aa0 0x7f87cc055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:04.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.605+0000 7f87e57fa700 1 -- 172.21.15.60:0/1960192778 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f87d806c8b0 con 0x7f87e80b61d0 2022-06-29T16:07:04.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.605+0000 7f87e77fe700 1 --2- 172.21.15.60:0/1960192778 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f87cc053aa0 0x7f87cc055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:04.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.605+0000 7f87e77fe700 1 --2- 172.21.15.60:0/1960192778 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f87cc053aa0 0x7f87cc055f60 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7f87d000b000 tx=0x7f87d0006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:04.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.609+0000 7f87e57fa700 1 -- 172.21.15.60:0/1960192778 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f87d80438d0 con 0x7f87e80b61d0 2022-06-29T16:07:04.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:04.733+0000 7f87ee82d700 1 -- 172.21.15.60:0/1960192778 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f87e80643c0 con 0x7f87e80b61d0 2022-06-29T16:07:05.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.465+0000 7f87e57fa700 1 -- 172.21.15.60:0/1960192778 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v119) v1 ==== 133+0+0 (secure 0 0 0) 0x7f87d8024020 con 0x7f87e80b61d0 2022-06-29T16:07:05.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.469+0000 7f87ee82d700 1 -- 172.21.15.60:0/1960192778 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f87cc053aa0 msgr2=0x7f87cc055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:05.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.469+0000 7f87ee82d700 1 --2- 172.21.15.60:0/1960192778 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f87cc053aa0 0x7f87cc055f60 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7f87d000b000 tx=0x7f87d0006df0).stop 2022-06-29T16:07:05.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.469+0000 7f87ee82d700 1 -- 172.21.15.60:0/1960192778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87e80b61d0 msgr2=0x7f87e8065730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:05.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.469+0000 7f87ee82d700 1 --2- 172.21.15.60:0/1960192778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87e80b61d0 0x7f87e8065730 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 rx=0x7f87d80032f0 tx=0x7f87d8016000).stop 2022-06-29T16:07:05.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.469+0000 7f87ee82d700 1 -- 172.21.15.60:0/1960192778 shutdown_connections 2022-06-29T16:07:05.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.469+0000 7f87ee82d700 1 --2- 172.21.15.60:0/1960192778 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f87cc053aa0 0x7f87cc055f60 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:05.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.469+0000 7f87ee82d700 1 --2- 172.21.15.60:0/1960192778 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f87e813fbd0 0x7f87e8134410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:05.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.469+0000 7f87ee82d700 1 --2- 172.21.15.60:0/1960192778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f87e813f790 0x7f87e8133f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:05.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.469+0000 7f87ee82d700 1 --2- 172.21.15.60:0/1960192778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f87e80b61d0 0x7f87e8065730 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:05.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.469+0000 7f87ee82d700 1 -- 172.21.15.60:0/1960192778 >> 172.21.15.60:0/1960192778 conn(0x7f87e8063b50 msgr2=0x7f87e8141780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:05.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.469+0000 7f87ee82d700 1 -- 172.21.15.60:0/1960192778 shutdown_connections 2022-06-29T16:07:05.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.469+0000 7f87ee82d700 1 -- 172.21.15.60:0/1960192778 wait complete. 2022-06-29T16:07:05.477 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2022-06-29T16:07:05.492 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2022-06-29T16:07:05.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.589+0000 7f8deaf1b700 1 Processor -- start 2022-06-29T16:07:05.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.589+0000 7f8deaf1b700 1 -- start start 2022-06-29T16:07:05.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.589+0000 7f8deaf1b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8de40b2f70 0x7f8de4146730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:05.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.589+0000 7f8deaf1b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8de414a680 0x7f8de414aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:05.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.589+0000 7f8deaf1b700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de40ad140 con 0x7f8de40acd60 2022-06-29T16:07:05.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.589+0000 7f8deaf1b700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8de4146c20 con 0x7f8de414a680 2022-06-29T16:07:05.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.589+0000 7f8deaf1b700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8de4062440 con 0x7f8de40b2f70 2022-06-29T16:07:05.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.589+0000 7f8dea71a700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8de414a680 0x7f8de414aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:05.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.589+0000 7f8dea71a700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8de414a680 0x7f8de414aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52524/0 (socket says 172.21.15.60:52524) 2022-06-29T16:07:05.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.589+0000 7f8dea71a700 1 -- 172.21.15.60:0/1268389952 learned_addr learned my addr 172.21.15.60:0/1268389952 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:05.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.589+0000 7f8de9718700 1 --2- 172.21.15.60:0/1268389952 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8de40b2f70 0x7f8de4146730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:05.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.589+0000 7f8dea71a700 1 -- 172.21.15.60:0/1268389952 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8de40b2f70 msgr2=0x7f8de4146730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:05.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.589+0000 7f8dea71a700 1 --2- 172.21.15.60:0/1268389952 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8de40b2f70 0x7f8de4146730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:05.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.589+0000 7f8dea71a700 1 -- 172.21.15.60:0/1268389952 >> v1:172.21.15.92:6789/0 conn(0x7f8de40acd60 legacy=0x7f8de4146020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:05.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.589+0000 7f8dea71a700 1 -- 172.21.15.60:0/1268389952 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de40632d0 con 0x7f8de414a680 2022-06-29T16:07:05.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.589+0000 7f8dea71a700 1 --2- 172.21.15.60:0/1268389952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8de414a680 0x7f8de414aaf0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 rx=0x7f8ddc002f70 tx=0x7f8ddc006eb0).ready entity=mon.0 client_cookie=3a945b205984b848 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:05.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.589+0000 7f8de8f17700 1 -- 172.21.15.60:0/1268389952 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ddc00d570 con 0x7f8de414a680 2022-06-29T16:07:05.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.589+0000 7f8de8f17700 1 -- 172.21.15.60:0/1268389952 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ddc000e90 con 0x7f8de414a680 2022-06-29T16:07:05.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.589+0000 7f8de8f17700 1 -- 172.21.15.60:0/1268389952 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ddc00d900 con 0x7f8de414a680 2022-06-29T16:07:05.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.589+0000 7f8deaf1b700 1 -- 172.21.15.60:0/1268389952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8de414a680 msgr2=0x7f8de414aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:05.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.589+0000 7f8deaf1b700 1 --2- 172.21.15.60:0/1268389952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8de414a680 0x7f8de414aaf0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 rx=0x7f8ddc002f70 tx=0x7f8ddc006eb0).stop 2022-06-29T16:07:05.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.589+0000 7f8deaf1b700 1 -- 172.21.15.60:0/1268389952 shutdown_connections 2022-06-29T16:07:05.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.589+0000 7f8deaf1b700 1 --2- 172.21.15.60:0/1268389952 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8de414a680 0x7f8de414aaf0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:05.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.589+0000 7f8deaf1b700 1 --2- 172.21.15.60:0/1268389952 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8de40b2f70 0x7f8de4146730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:05.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.589+0000 7f8deaf1b700 1 -- 172.21.15.60:0/1268389952 >> 172.21.15.60:0/1268389952 conn(0x7f8de40b6f80 msgr2=0x7f8de40b7390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:05.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.589+0000 7f8deaf1b700 1 -- 172.21.15.60:0/1268389952 shutdown_connections 2022-06-29T16:07:05.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.593+0000 7f8deaf1b700 1 -- 172.21.15.60:0/1268389952 wait complete. 2022-06-29T16:07:05.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.593+0000 7f8deaf1b700 1 Processor -- start 2022-06-29T16:07:05.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.593+0000 7f8deaf1b700 1 -- start start 2022-06-29T16:07:05.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.593+0000 7f8deaf1b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8de40acd60 0x7f8de405c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:05.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.593+0000 7f8deaf1b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8de40b2f70 0x7f8de4143b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:05.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.593+0000 7f8deaf1b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8de414a680 0x7f8de4144080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:05.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.593+0000 7f8deaf1b700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de40b0a30 con 0x7f8de40acd60 2022-06-29T16:07:05.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.593+0000 7f8deaf1b700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de40af350 con 0x7f8de40b2f70 2022-06-29T16:07:05.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.593+0000 7f8deaf1b700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8de40b0c60 con 0x7f8de414a680 2022-06-29T16:07:05.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.593+0000 7f8de9f19700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8de40acd60 0x7f8de405c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:05.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.593+0000 7f8de9f19700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8de40acd60 0x7f8de405c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52526/0 (socket says 172.21.15.60:52526) 2022-06-29T16:07:05.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.593+0000 7f8de9f19700 1 -- 172.21.15.60:0/2325911122 learned_addr learned my addr 172.21.15.60:0/2325911122 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:05.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.593+0000 7f8de9718700 1 --2- 172.21.15.60:0/2325911122 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8de40b2f70 0x7f8de4143b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:05.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.593+0000 7f8dea71a700 1 --2- 172.21.15.60:0/2325911122 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8de414a680 0x7f8de4144080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:05.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.593+0000 7f8de9f19700 1 -- 172.21.15.60:0/2325911122 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8de414a680 msgr2=0x7f8de4144080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:05.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.593+0000 7f8de9f19700 1 --2- 172.21.15.60:0/2325911122 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8de414a680 0x7f8de4144080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:05.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.593+0000 7f8de9f19700 1 -- 172.21.15.60:0/2325911122 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8de40b2f70 msgr2=0x7f8de4143b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:05.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.593+0000 7f8de9f19700 1 --2- 172.21.15.60:0/2325911122 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8de40b2f70 0x7f8de4143b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:05.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.593+0000 7f8de9f19700 1 -- 172.21.15.60:0/2325911122 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ddc00b1e0 con 0x7f8de40acd60 2022-06-29T16:07:05.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.593+0000 7f8de9f19700 1 --2- 172.21.15.60:0/2325911122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8de40acd60 0x7f8de405c680 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 rx=0x7f8de40b0df0 tx=0x7f8dd4007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:05.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.593+0000 7f8ddaffd700 1 -- 172.21.15.60:0/2325911122 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8dd4007030 con 0x7f8de40acd60 2022-06-29T16:07:05.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.593+0000 7f8ddaffd700 1 -- 172.21.15.60:0/2325911122 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dd4007190 con 0x7f8de40acd60 2022-06-29T16:07:05.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.593+0000 7f8deaf1b700 1 -- 172.21.15.60:0/2325911122 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8de4144570 con 0x7f8de40acd60 2022-06-29T16:07:05.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.593+0000 7f8ddaffd700 1 -- 172.21.15.60:0/2325911122 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8dd4006510 con 0x7f8de40acd60 2022-06-29T16:07:05.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.593+0000 7f8deaf1b700 1 -- 172.21.15.60:0/2325911122 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8de40b0e80 con 0x7f8de40acd60 2022-06-29T16:07:05.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.597+0000 7f8deaf1b700 1 -- 172.21.15.60:0/2325911122 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8de40af510 con 0x7f8de40acd60 2022-06-29T16:07:05.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.597+0000 7f8ddaffd700 1 -- 172.21.15.60:0/2325911122 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8dd401b070 con 0x7f8de40acd60 2022-06-29T16:07:05.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.597+0000 7f8ddaffd700 1 --2- 172.21.15.60:0/2325911122 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8dc4053b00 0x7f8dc4055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:05.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.597+0000 7f8ddaffd700 1 -- 172.21.15.60:0/2325911122 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6075+0+0 (secure 0 0 0) 0x7f8dd406b400 con 0x7f8de40acd60 2022-06-29T16:07:05.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.597+0000 7f8de9718700 1 --2- 172.21.15.60:0/2325911122 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8dc4053b00 0x7f8dc4055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:05.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.597+0000 7f8de9718700 1 --2- 172.21.15.60:0/2325911122 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8dc4053b00 0x7f8dc4055fc0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7f8de000b000 tx=0x7f8de00091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:05.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.605+0000 7f8ddaffd700 1 -- 172.21.15.60:0/2325911122 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8dd4042320 con 0x7f8de40acd60 2022-06-29T16:07:05.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.765+0000 7f8deaf1b700 1 -- 172.21.15.60:0/2325911122 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f8de41104a0 con 0x7f8de40acd60 2022-06-29T16:07:05.769 INFO:tasks.ceph.mon.a.smithi060.stderr:2022-06-29T16:07:05.765+0000 7fcc17b71700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-06-29T16:07:05.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.773+0000 7f8ddaffd700 1 -- 172.21.15.60:0/2325911122 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 24 and data pool 22 v56) v1 ==== 174+0+0 (secure 0 0 0) 0x7f8dd4045940 con 0x7f8de40acd60 2022-06-29T16:07:05.784 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.777+0000 7f8deaf1b700 1 -- 172.21.15.60:0/2325911122 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8dc4053b00 msgr2=0x7f8dc4055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:05.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.777+0000 7f8deaf1b700 1 --2- 172.21.15.60:0/2325911122 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8dc4053b00 0x7f8dc4055fc0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7f8de000b000 tx=0x7f8de00091d0).stop 2022-06-29T16:07:05.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.777+0000 7f8deaf1b700 1 -- 172.21.15.60:0/2325911122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8de40acd60 msgr2=0x7f8de405c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:05.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.777+0000 7f8deaf1b700 1 --2- 172.21.15.60:0/2325911122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8de40acd60 0x7f8de405c680 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 rx=0x7f8de40b0df0 tx=0x7f8dd4007c00).stop 2022-06-29T16:07:05.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.777+0000 7f8deaf1b700 1 -- 172.21.15.60:0/2325911122 shutdown_connections 2022-06-29T16:07:05.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.777+0000 7f8deaf1b700 1 --2- 172.21.15.60:0/2325911122 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8dc4053b00 0x7f8dc4055fc0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:05.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.777+0000 7f8deaf1b700 1 --2- 172.21.15.60:0/2325911122 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8de414a680 0x7f8de4144080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:05.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.777+0000 7f8deaf1b700 1 --2- 172.21.15.60:0/2325911122 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8de40b2f70 0x7f8de4143b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:05.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.781+0000 7f8deaf1b700 1 --2- 172.21.15.60:0/2325911122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8de40acd60 0x7f8de405c680 secure :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 rx=0x7f8de40b0df0 tx=0x7f8dd4007c00).stop 2022-06-29T16:07:05.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.781+0000 7f8deaf1b700 1 -- 172.21.15.60:0/2325911122 >> 172.21.15.60:0/2325911122 conn(0x7f8de40b6f80 msgr2=0x7f8de40b5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:05.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.781+0000 7f8deaf1b700 1 -- 172.21.15.60:0/2325911122 shutdown_connections 2022-06-29T16:07:05.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:05.781+0000 7f8deaf1b700 1 -- 172.21.15.60:0/2325911122 wait complete. 2022-06-29T16:07:05.789 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 24 and data pool 22 2022-06-29T16:07:05.796 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-06-29T16:07:05.796 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:07:05.811 DEBUG:teuthology.orchestra.run.smithi092:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:07:05.832 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:07:05.854 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:07:05.876 DEBUG:teuthology.orchestra.run.smithi197:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:07:05.897 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-06-29T16:07:05.898 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:05.898 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:07:05.899 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:07:05.906 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-06-29T16:07:05.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:07:05.908 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-06-29T16:07:05.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:07:05.909 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:07:05.916 INFO:teuthology.orchestra.run.smithi121.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-06-29T16:07:05.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:07:05.918 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:05.918 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:07:05.919 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:07:05.926 INFO:teuthology.orchestra.run.smithi156.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-06-29T16:07:05.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:07:05.928 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-06-29T16:07:05.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:07:05.929 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:07:05.935 INFO:teuthology.orchestra.run.smithi156.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-06-29T16:07:05.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:07:05.937 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:05.937 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:07:05.938 DEBUG:teuthology.orchestra.run.smithi197:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T16:07:05.945 INFO:teuthology.orchestra.run.smithi197.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-06-29T16:07:05.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:07:05.947 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-06-29T16:07:05.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:07:05.948 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T16:07:05.954 INFO:teuthology.orchestra.run.smithi197.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-06-29T16:07:05.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:07:05.956 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-06-29T16:07:06.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.061+0000 7f429a9b6700 1 Processor -- start 2022-06-29T16:07:06.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.061+0000 7f429a9b6700 1 -- start start 2022-06-29T16:07:06.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.065+0000 7f429a9b6700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f42940aca20 0x7f42940ace00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:06.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.065+0000 7f429a9b6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42940b3270 0x7f4294062cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:06.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.065+0000 7f429a9b6700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42940adc80 con 0x7f4294143030 2022-06-29T16:07:06.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.065+0000 7f429a9b6700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42940ad740 con 0x7f42940aca20 2022-06-29T16:07:06.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.065+0000 7f429a9b6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42940ad9e0 con 0x7f42940b3270 2022-06-29T16:07:06.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.065+0000 7f429a1b5700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4294143030 0x7f42941461c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60624/0 (socket says 172.21.15.60:60624) 2022-06-29T16:07:06.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.065+0000 7f429a1b5700 1 -- 172.21.15.60:0/3939969403 learned_addr learned my addr 172.21.15.60:0/3939969403 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:06.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.065+0000 7f42991b3700 1 --2- 172.21.15.60:0/3939969403 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42940b3270 0x7f4294062cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:06.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.065+0000 7f42999b4700 1 --2- 172.21.15.60:0/3939969403 >> v2:172.21.15.92:3300/0 conn(0x7f42940aca20 0x7f42940ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:06.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.065+0000 7f42989b2700 1 -- 172.21.15.60:0/3939969403 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3329951404 0 0) 0x7f42940adc80 con 0x7f4294143030 2022-06-29T16:07:06.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.065+0000 7f42989b2700 1 -- 172.21.15.60:0/3939969403 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f427c003260 con 0x7f4294143030 2022-06-29T16:07:06.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.065+0000 7f42989b2700 1 -- 172.21.15.60:0/3939969403 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2454991259 0 0) 0x7f427c003260 con 0x7f4294143030 2022-06-29T16:07:06.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.065+0000 7f42991b3700 1 -- 172.21.15.60:0/3939969403 >> v2:172.21.15.92:3300/0 conn(0x7f42940aca20 msgr2=0x7f42940ace00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:06.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.065+0000 7f42991b3700 1 --2- 172.21.15.60:0/3939969403 >> v2:172.21.15.92:3300/0 conn(0x7f42940aca20 0x7f42940ace00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:06.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.065+0000 7f42991b3700 1 -- 172.21.15.60:0/3939969403 >> v1:172.21.15.60:6789/0 conn(0x7f4294143030 legacy=0x7f42941461c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:06.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.065+0000 7f42991b3700 1 -- 172.21.15.60:0/3939969403 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4294142920 con 0x7f42940b3270 2022-06-29T16:07:06.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.065+0000 7f42999b4700 1 --2- 172.21.15.60:0/3939969403 >> v2:172.21.15.92:3300/0 conn(0x7f42940aca20 0x7f42940ace00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:07:06.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.065+0000 7f42991b3700 1 --2- 172.21.15.60:0/3939969403 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42940b3270 0x7f4294062cd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f428c001b80 tx=0x7f4290006df0).ready entity=mon.2 client_cookie=e62bae0162ae8d85 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:06.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.065+0000 7f42989b2700 1 -- 172.21.15.60:0/3939969403 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f428c0019e0 con 0x7f42940b3270 2022-06-29T16:07:06.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.065+0000 7f42989b2700 1 -- 172.21.15.60:0/3939969403 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4290000e90 con 0x7f42940b3270 2022-06-29T16:07:06.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.065+0000 7f42989b2700 1 -- 172.21.15.60:0/3939969403 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4290012460 con 0x7f42940b3270 2022-06-29T16:07:06.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.065+0000 7f429a9b6700 1 -- 172.21.15.60:0/3939969403 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42940b3270 msgr2=0x7f4294062cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:06.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.065+0000 7f429a9b6700 1 --2- 172.21.15.60:0/3939969403 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42940b3270 0x7f4294062cd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f428c001b80 tx=0x7f4290006df0).stop 2022-06-29T16:07:06.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.065+0000 7f429a9b6700 1 -- 172.21.15.60:0/3939969403 shutdown_connections 2022-06-29T16:07:06.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.065+0000 7f429a9b6700 1 --2- 172.21.15.60:0/3939969403 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42940b3270 0x7f4294062cd0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:06.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.065+0000 7f429a9b6700 1 --2- 172.21.15.60:0/3939969403 >> v2:172.21.15.92:3300/0 conn(0x7f42940aca20 0x7f42940ace00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:06.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.065+0000 7f429a9b6700 1 -- 172.21.15.60:0/3939969403 >> 172.21.15.60:0/3939969403 conn(0x7f42940b6f80 msgr2=0x7f42940b7390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:06.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.069+0000 7f429a9b6700 1 -- 172.21.15.60:0/3939969403 shutdown_connections 2022-06-29T16:07:06.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.069+0000 7f429a9b6700 1 -- 172.21.15.60:0/3939969403 wait complete. 2022-06-29T16:07:06.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.069+0000 7f429a9b6700 1 Processor -- start 2022-06-29T16:07:06.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.069+0000 7f429a9b6700 1 -- start start 2022-06-29T16:07:06.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.069+0000 7f429a9b6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42940aca20 0x7f4294155110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:06.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.069+0000 7f429a9b6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42940b3270 0x7f42941450c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:06.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.069+0000 7f429a9b6700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4294143030 0x7f42940665e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:06.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.069+0000 7f429a9b6700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429413d3b0 con 0x7f42940aca20 2022-06-29T16:07:06.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.069+0000 7f429a9b6700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42940b0940 con 0x7f4294143030 2022-06-29T16:07:06.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.069+0000 7f429a9b6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42940b0de0 con 0x7f42940b3270 2022-06-29T16:07:06.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.069+0000 7f42999b4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42940aca20 0x7f4294155110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:06.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.069+0000 7f42999b4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42940aca20 0x7f4294155110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52540/0 (socket says 172.21.15.60:52540) 2022-06-29T16:07:06.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.069+0000 7f42999b4700 1 -- 172.21.15.60:0/1891351503 learned_addr learned my addr 172.21.15.60:0/1891351503 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:06.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.069+0000 7f42991b3700 1 --2- 172.21.15.60:0/1891351503 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42940b3270 0x7f42941450c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:06.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.069+0000 7f429a1b5700 1 --2- 172.21.15.60:0/1891351503 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4294143030 0x7f42940665e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:06.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.069+0000 7f42999b4700 1 -- 172.21.15.60:0/1891351503 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42940b3270 msgr2=0x7f42941450c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:06.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.069+0000 7f42999b4700 1 --2- 172.21.15.60:0/1891351503 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42940b3270 0x7f42941450c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:06.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.069+0000 7f42999b4700 1 -- 172.21.15.60:0/1891351503 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4294143030 msgr2=0x7f42940665e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:06.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.069+0000 7f42999b4700 1 --2- 172.21.15.60:0/1891351503 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4294143030 0x7f42940665e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:06.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.069+0000 7f42999b4700 1 -- 172.21.15.60:0/1891351503 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f428400b180 con 0x7f42940aca20 2022-06-29T16:07:06.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.069+0000 7f42991b3700 1 --2- 172.21.15.60:0/1891351503 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42940b3270 0x7f42941450c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:07:06.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.069+0000 7f429a1b5700 1 --2- 172.21.15.60:0/1891351503 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4294143030 0x7f42940665e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:07:06.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.069+0000 7f42999b4700 1 --2- 172.21.15.60:0/1891351503 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42940aca20 0x7f4294155110 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 rx=0x7f428400c000 tx=0x7f4284006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:06.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.069+0000 7f4282ffd700 1 -- 172.21.15.60:0/1891351503 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f428400c860 con 0x7f42940aca20 2022-06-29T16:07:06.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.069+0000 7f4282ffd700 1 -- 172.21.15.60:0/1891351503 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f428400c9c0 con 0x7f42940aca20 2022-06-29T16:07:06.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.073+0000 7f429a9b6700 1 -- 172.21.15.60:0/1891351503 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f429000b180 con 0x7f42940aca20 2022-06-29T16:07:06.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.073+0000 7f4282ffd700 1 -- 172.21.15.60:0/1891351503 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f428400cd20 con 0x7f42940aca20 2022-06-29T16:07:06.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.073+0000 7f429a9b6700 1 -- 172.21.15.60:0/1891351503 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42940af1c0 con 0x7f42940aca20 2022-06-29T16:07:06.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.073+0000 7f429a9b6700 1 -- 172.21.15.60:0/1891351503 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42940573d0 con 0x7f42940aca20 2022-06-29T16:07:06.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.073+0000 7f4282ffd700 1 -- 172.21.15.60:0/1891351503 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f428401c460 con 0x7f42940aca20 2022-06-29T16:07:06.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.073+0000 7f4282ffd700 1 --2- 172.21.15.60:0/1891351503 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4274053a70 0x7f4274055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:06.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.073+0000 7f4282ffd700 1 -- 172.21.15.60:0/1891351503 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f428406c5c0 con 0x7f42940aca20 2022-06-29T16:07:06.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.073+0000 7f42991b3700 1 --2- 172.21.15.60:0/1891351503 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4274053a70 0x7f4274055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:06.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.077+0000 7f42991b3700 1 --2- 172.21.15.60:0/1891351503 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4274053a70 0x7f4274055f30 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0x7f4290016000 tx=0x7f4290014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:06.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.077+0000 7f4282ffd700 1 -- 172.21.15.60:0/1891351503 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4284043430 con 0x7f42940aca20 2022-06-29T16:07:06.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.209+0000 7f429a9b6700 1 -- 172.21.15.60:0/1891351503 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f42940b2b40 con 0x7f42940aca20 2022-06-29T16:07:06.216 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.213+0000 7f4282ffd700 1 -- 172.21.15.60:0/1891351503 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v120) v1 ==== 81+0+33826 (secure 0 0 0) 0x7f4284046a50 con 0x7f42940aca20 2022-06-29T16:07:06.217 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:06.217 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:07:06.217 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 120, 2022-06-29T16:07:06.218 INFO:teuthology.orchestra.run.smithi060.stdout: "fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", 2022-06-29T16:07:06.218 INFO:teuthology.orchestra.run.smithi060.stdout: "created": "2022-06-29T15:58:07.965719+0000", 2022-06-29T16:07:06.218 INFO:teuthology.orchestra.run.smithi060.stdout: "modified": "2022-06-29T16:07:05.766915+0000", 2022-06-29T16:07:06.219 INFO:teuthology.orchestra.run.smithi060.stdout: "last_up_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:07:06.219 INFO:teuthology.orchestra.run.smithi060.stdout: "last_in_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:07:06.219 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-06-29T16:07:06.220 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_num": 5799936, 2022-06-29T16:07:06.220 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_set": [ 2022-06-29T16:07:06.220 INFO:teuthology.orchestra.run.smithi060.stdout: "pglog_hardlimit", 2022-06-29T16:07:06.220 INFO:teuthology.orchestra.run.smithi060.stdout: "purged_snapdirs", 2022-06-29T16:07:06.220 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_deletes", 2022-06-29T16:07:06.221 INFO:teuthology.orchestra.run.smithi060.stdout: "sortbitwise" 2022-06-29T16:07:06.221 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:07:06.221 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_version": 4, 2022-06-29T16:07:06.222 INFO:teuthology.orchestra.run.smithi060.stdout: "full_ratio": 0.94999998807907104, 2022-06-29T16:07:06.222 INFO:teuthology.orchestra.run.smithi060.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-06-29T16:07:06.223 INFO:teuthology.orchestra.run.smithi060.stdout: "nearfull_ratio": 0.85000002384185791, 2022-06-29T16:07:06.223 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_snapshot": "", 2022-06-29T16:07:06.223 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_max": 24, 2022-06-29T16:07:06.223 INFO:teuthology.orchestra.run.smithi060.stdout: "max_osd": 8, 2022-06-29T16:07:06.223 INFO:teuthology.orchestra.run.smithi060.stdout: "require_min_compat_client": "jewel", 2022-06-29T16:07:06.224 INFO:teuthology.orchestra.run.smithi060.stdout: "min_compat_client": "jewel", 2022-06-29T16:07:06.224 INFO:teuthology.orchestra.run.smithi060.stdout: "require_osd_release": "octopus", 2022-06-29T16:07:06.224 INFO:teuthology.orchestra.run.smithi060.stdout: "pools": [ 2022-06-29T16:07:06.224 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.225 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2022-06-29T16:07:06.225 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "device_health_metrics", 2022-06-29T16:07:06.225 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:10.961127+0000", 2022-06-29T16:07:06.225 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:07:06.226 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:07:06.226 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:07:06.226 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:07:06.227 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:07:06.227 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:07:06.227 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:07:06.228 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:07:06.229 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 1, 2022-06-29T16:07:06.229 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 1, 2022-06-29T16:07:06.229 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 1, 2022-06-29T16:07:06.230 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 1, 2022-06-29T16:07:06.230 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 1, 2022-06-29T16:07:06.231 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:07:06.231 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:07:06.232 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:07:06.232 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:07:06.233 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:07:06.233 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:07:06.233 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:07:06.234 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.234 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "15", 2022-06-29T16:07:06.235 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:07:06.235 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:07:06.236 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:07:06.236 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:07:06.236 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:07:06.236 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:07:06.237 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:07:06.237 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:07:06.237 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:07:06.237 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:07:06.238 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:07:06.238 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:07:06.238 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:07:06.238 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:07:06.239 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:07:06.239 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:07:06.239 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:07:06.239 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:07:06.239 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:07:06.240 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:07:06.240 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:07:06.240 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:07:06.240 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:07:06.241 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:07:06.241 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:07:06.241 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:07:06.241 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.241 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:07:06.242 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:07:06.242 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:07:06.242 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:07:06.242 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:07:06.243 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:07:06.243 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:07:06.243 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:07:06.243 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:07:06.244 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:07:06.244 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:07:06.244 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:07:06.244 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 1 2022-06-29T16:07:06.244 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.245 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:07:06.245 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_devicehealth": {} 2022-06-29T16:07:06.245 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.245 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.246 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.246 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 2, 2022-06-29T16:07:06.247 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "rbd", 2022-06-29T16:07:06.247 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:19.184276+0000", 2022-06-29T16:07:06.248 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:07:06.248 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:07:06.249 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:07:06.249 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:07:06.249 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:07:06.250 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:07:06.250 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:07:06.251 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:07:06.251 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:07:06.252 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:07:06.252 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:07:06.253 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:07:06.253 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:07:06.254 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:07:06.254 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:07:06.255 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:07:06.255 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:07:06.256 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:07:06.256 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:07:06.257 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:07:06.257 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.257 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "18", 2022-06-29T16:07:06.258 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:07:06.258 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:07:06.259 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:07:06.259 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:07:06.260 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:07:06.260 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:07:06.260 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:07:06.261 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:07:06.261 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:07:06.262 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:07:06.262 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:07:06.263 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:07:06.263 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:07:06.263 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:07:06.264 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:07:06.264 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:07:06.265 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:07:06.265 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:07:06.266 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:07:06.266 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:07:06.266 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:07:06.267 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:07:06.267 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:07:06.267 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:07:06.268 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:07:06.268 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:07:06.269 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.269 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:07:06.269 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:07:06.270 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:07:06.270 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:07:06.271 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:07:06.271 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:07:06.272 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:07:06.272 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:07:06.272 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:07:06.273 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:07:06.273 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:07:06.274 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:07:06.274 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:07:06.275 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd": {} 2022-06-29T16:07:06.275 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.276 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.276 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 22, 2022-06-29T16:07:06.277 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "cephfs_data", 2022-06-29T16:07:06.277 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T16:06:27.251709+0000", 2022-06-29T16:07:06.278 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:07:06.278 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:07:06.278 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:07:06.279 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:07:06.279 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:07:06.279 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:07:06.280 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:07:06.280 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:07:06.280 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:07:06.281 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:07:06.281 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:07:06.282 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:07:06.282 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:07:06.282 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:07:06.283 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:07:06.283 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:07:06.284 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:07:06.284 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:07:06.285 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:07:06.285 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:07:06.286 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.286 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "120", 2022-06-29T16:07:06.286 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:07:06.286 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:07:06.287 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:07:06.287 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:07:06.287 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:07:06.287 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:07:06.288 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:07:06.288 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:07:06.288 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:07:06.288 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:07:06.289 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:07:06.289 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:07:06.289 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:07:06.289 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:07:06.289 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:07:06.290 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:07:06.290 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:07:06.290 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:07:06.290 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:07:06.291 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:07:06.291 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:07:06.291 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:07:06.291 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:07:06.291 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:07:06.292 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:07:06.292 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:07:06.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.292 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:07:06.293 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:07:06.293 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:07:06.293 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:07:06.293 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:07:06.294 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:07:06.294 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:07:06.294 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:07:06.294 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:07:06.295 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:07:06.295 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:07:06.295 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:07:06.295 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:07:06.295 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs": { 2022-06-29T16:07:06.296 INFO:teuthology.orchestra.run.smithi060.stdout: "data": "cephfs" 2022-06-29T16:07:06.296 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.296 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.297 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.297 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 24, 2022-06-29T16:07:06.297 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "cephfs_metadata", 2022-06-29T16:07:06.298 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T16:07:04.791800+0000", 2022-06-29T16:07:06.298 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 32769, 2022-06-29T16:07:06.298 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool,creating", 2022-06-29T16:07:06.298 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:07:06.299 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:07:06.299 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:07:06.299 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:07:06.299 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:07:06.300 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:07:06.300 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:07:06.300 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:07:06.300 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:07:06.300 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:07:06.301 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:07:06.301 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:07:06.302 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:07:06.302 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:07:06.302 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:07:06.302 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:07:06.303 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:07:06.303 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:07:06.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.303 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "120", 2022-06-29T16:07:06.303 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:07:06.304 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:07:06.304 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:07:06.304 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:07:06.304 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:07:06.305 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:07:06.305 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:07:06.305 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:07:06.305 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:07:06.305 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:07:06.306 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:07:06.306 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:07:06.306 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:07:06.306 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:07:06.307 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:07:06.307 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:07:06.307 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:07:06.308 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:07:06.308 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:07:06.308 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:07:06.308 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:07:06.309 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:07:06.309 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:07:06.309 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:07:06.309 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:07:06.310 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:07:06.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.310 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:07:06.310 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:07:06.310 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:07:06.311 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:07:06.311 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:07:06.311 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:07:06.311 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:07:06.312 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:07:06.312 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:07:06.312 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:07:06.312 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:07:06.312 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:07:06.313 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_bias": 4, 2022-06-29T16:07:06.313 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 16, 2022-06-29T16:07:06.313 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_priority": 5 2022-06-29T16:07:06.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.314 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:07:06.314 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs": { 2022-06-29T16:07:06.314 INFO:teuthology.orchestra.run.smithi060.stdout: "metadata": "cephfs" 2022-06-29T16:07:06.314 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.315 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.315 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.315 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:07:06.315 INFO:teuthology.orchestra.run.smithi060.stdout: "osds": [ 2022-06-29T16:07:06.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.316 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:07:06.316 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "11e7ab9e-fce2-470f-8195-ea4c3c52df22", 2022-06-29T16:07:06.316 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:07:06.317 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:07:06.317 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:07:06.317 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:07:06.317 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:07:06.318 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:07:06.318 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:07:06.318 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 119, 2022-06-29T16:07:06.319 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:07:06.319 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:07:06.319 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:07:06.320 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:06.320 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.320 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:06.320 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6816", 2022-06-29T16:07:06.321 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:07:06.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.321 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:06.321 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6825", 2022-06-29T16:07:06.322 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:07:06.322 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.322 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.323 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:07:06.323 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:06.323 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.323 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:06.324 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6826", 2022-06-29T16:07:06.324 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:07:06.324 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.324 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.325 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:06.325 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6827", 2022-06-29T16:07:06.325 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:07:06.325 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.325 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.326 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:07:06.326 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:06.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.327 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:06.327 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6830", 2022-06-29T16:07:06.327 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:07:06.327 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.327 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.328 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:06.328 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6831", 2022-06-29T16:07:06.328 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:07:06.328 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.329 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.329 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.329 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:07:06.330 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:06.330 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.330 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:06.330 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6828", 2022-06-29T16:07:06.331 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:07:06.331 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.331 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.331 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:06.332 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6829", 2022-06-29T16:07:06.332 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:07:06.332 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.332 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.333 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.333 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6825/12890", 2022-06-29T16:07:06.333 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6827/12890", 2022-06-29T16:07:06.333 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6831/12890", 2022-06-29T16:07:06.333 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6829/12890", 2022-06-29T16:07:06.334 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:07:06.334 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:07:06.334 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:07:06.334 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.335 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.335 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.335 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:07:06.335 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "ca43f136-6f69-4431-a157-fbe073fb585c", 2022-06-29T16:07:06.336 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:07:06.336 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:07:06.336 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:07:06.336 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:07:06.336 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:07:06.337 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:07:06.337 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:07:06.337 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 119, 2022-06-29T16:07:06.337 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:07:06.338 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:07:06.338 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:07:06.338 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:06.338 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.338 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:06.339 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6815", 2022-06-29T16:07:06.339 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:07:06.339 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.339 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.340 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:06.340 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6818", 2022-06-29T16:07:06.340 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:07:06.341 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.341 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.341 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.341 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:07:06.342 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:06.342 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.342 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:06.343 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6819", 2022-06-29T16:07:06.343 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:07:06.343 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.343 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.344 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:06.344 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6820", 2022-06-29T16:07:06.344 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:07:06.344 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.344 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.345 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.345 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:07:06.345 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:06.345 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.346 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:06.346 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6823", 2022-06-29T16:07:06.346 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:07:06.346 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.347 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.347 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:06.347 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6824", 2022-06-29T16:07:06.348 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:07:06.348 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.348 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.348 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.348 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:07:06.349 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:06.349 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.349 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:06.349 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6821", 2022-06-29T16:07:06.350 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:07:06.350 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.350 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.351 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:06.351 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6822", 2022-06-29T16:07:06.351 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:07:06.351 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.352 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.352 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.352 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6818/12892", 2022-06-29T16:07:06.352 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6820/12892", 2022-06-29T16:07:06.353 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6824/12892", 2022-06-29T16:07:06.353 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6822/12892", 2022-06-29T16:07:06.353 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:07:06.353 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:07:06.353 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:07:06.354 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.354 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.354 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.354 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:07:06.355 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "74fac45f-5f07-44ec-a9cb-2644c123fd4a", 2022-06-29T16:07:06.355 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:07:06.355 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:07:06.355 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:07:06.355 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:07:06.356 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:07:06.356 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:07:06.356 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:07:06.356 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 114, 2022-06-29T16:07:06.357 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:07:06.357 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:07:06.357 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:07:06.357 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:06.357 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.358 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:06.358 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6808", 2022-06-29T16:07:06.358 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:07:06.358 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.359 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.359 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:06.359 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6809", 2022-06-29T16:07:06.359 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:07:06.360 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.360 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.360 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.360 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:07:06.361 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:06.361 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.361 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:06.362 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6810", 2022-06-29T16:07:06.362 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:07:06.362 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.362 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.363 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:06.363 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6811", 2022-06-29T16:07:06.363 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:07:06.363 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.363 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.364 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.364 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:07:06.364 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:06.364 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.365 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:06.365 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6814", 2022-06-29T16:07:06.365 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:07:06.365 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.366 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.366 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:06.366 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6817", 2022-06-29T16:07:06.367 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:07:06.367 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.367 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.368 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.368 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:07:06.369 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:06.369 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.370 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:06.370 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6812", 2022-06-29T16:07:06.371 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:07:06.371 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.371 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.372 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:06.372 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6813", 2022-06-29T16:07:06.373 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:07:06.373 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.374 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.374 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.374 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6809/12891", 2022-06-29T16:07:06.375 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6811/12891", 2022-06-29T16:07:06.375 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6817/12891", 2022-06-29T16:07:06.375 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6813/12891", 2022-06-29T16:07:06.376 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:07:06.376 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:07:06.377 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:07:06.377 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.377 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.378 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.378 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:07:06.378 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "2581d351-973a-4b4e-85ca-3658f0068765", 2022-06-29T16:07:06.379 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:07:06.379 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:07:06.380 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:07:06.380 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:07:06.380 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:07:06.380 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:07:06.381 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:07:06.381 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 119, 2022-06-29T16:07:06.381 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:07:06.381 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:07:06.382 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:07:06.382 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:06.382 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.382 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:06.383 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6800", 2022-06-29T16:07:06.383 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:07:06.383 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.383 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.383 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:06.384 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6801", 2022-06-29T16:07:06.384 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:07:06.384 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.384 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.385 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.385 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:07:06.385 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:06.385 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.385 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:06.386 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6802", 2022-06-29T16:07:06.386 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:07:06.386 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.387 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.387 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:06.388 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6803", 2022-06-29T16:07:06.388 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:07:06.389 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.389 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.390 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.390 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:07:06.391 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:06.391 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.391 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:06.392 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6806", 2022-06-29T16:07:06.392 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:07:06.392 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.393 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.393 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:06.394 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6807", 2022-06-29T16:07:06.394 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:07:06.394 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.395 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.395 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.395 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:07:06.395 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:06.395 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.396 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:06.396 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6804", 2022-06-29T16:07:06.396 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:07:06.396 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.397 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.397 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:06.397 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6805", 2022-06-29T16:07:06.397 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:07:06.397 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.398 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.398 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.403 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6801/12889", 2022-06-29T16:07:06.403 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6803/12889", 2022-06-29T16:07:06.404 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6807/12889", 2022-06-29T16:07:06.404 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6805/12889", 2022-06-29T16:07:06.404 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:07:06.404 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:07:06.405 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:07:06.405 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.405 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.405 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.405 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:07:06.406 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "3d234db9-87cd-4fe9-814f-0240a287d8b4", 2022-06-29T16:07:06.406 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:07:06.406 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:07:06.406 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:07:06.407 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:07:06.407 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:07:06.407 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:07:06.407 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:07:06.408 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 114, 2022-06-29T16:07:06.408 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:07:06.408 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:07:06.408 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:07:06.408 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:06.409 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.409 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:06.409 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6808", 2022-06-29T16:07:06.409 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:07:06.410 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.410 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.410 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:06.410 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6810", 2022-06-29T16:07:06.410 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:07:06.411 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.411 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.411 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.411 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:07:06.412 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:06.412 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.412 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:06.412 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6812", 2022-06-29T16:07:06.412 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:07:06.413 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.413 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.413 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:06.413 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6814", 2022-06-29T16:07:06.414 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:07:06.414 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.414 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.414 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.414 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:07:06.415 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:06.415 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.415 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:06.416 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6823", 2022-06-29T16:07:06.416 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:07:06.417 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.417 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.417 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:06.417 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6825", 2022-06-29T16:07:06.418 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:07:06.418 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.418 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.419 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.419 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:07:06.420 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:06.420 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.420 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:06.421 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6817", 2022-06-29T16:07:06.421 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:07:06.422 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.422 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.422 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:06.423 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6820", 2022-06-29T16:07:06.423 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:07:06.424 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.424 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.424 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.425 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6810/13249", 2022-06-29T16:07:06.425 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6814/13249", 2022-06-29T16:07:06.425 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6825/13249", 2022-06-29T16:07:06.426 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6820/13249", 2022-06-29T16:07:06.426 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:07:06.426 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:07:06.426 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:07:06.427 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.427 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.427 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.427 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:07:06.427 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "e4bf1104-ff5c-438b-b608-8cee31c163f6", 2022-06-29T16:07:06.428 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:07:06.428 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:07:06.428 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:07:06.428 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:07:06.429 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:07:06.429 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:07:06.429 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:07:06.429 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 119, 2022-06-29T16:07:06.429 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:07:06.430 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:07:06.430 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:07:06.430 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:06.430 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.431 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:06.431 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6809", 2022-06-29T16:07:06.431 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:07:06.431 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.431 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.432 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:06.432 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6811", 2022-06-29T16:07:06.432 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:07:06.432 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.432 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.433 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.433 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:07:06.433 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:06.433 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.434 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:06.434 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6813", 2022-06-29T16:07:06.434 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:07:06.434 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.434 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.435 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:06.435 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6815", 2022-06-29T16:07:06.435 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:07:06.435 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.436 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.436 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.436 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:07:06.436 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:06.436 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.437 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:06.437 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6824", 2022-06-29T16:07:06.437 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:07:06.437 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.438 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.438 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:06.438 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6827", 2022-06-29T16:07:06.438 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:07:06.438 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.439 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.439 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.439 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:07:06.439 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:06.440 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.440 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:06.440 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6818", 2022-06-29T16:07:06.440 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:07:06.440 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.441 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.441 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:06.441 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6821", 2022-06-29T16:07:06.441 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:07:06.441 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.442 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.442 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.442 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6811/13248", 2022-06-29T16:07:06.442 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6815/13248", 2022-06-29T16:07:06.443 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6827/13248", 2022-06-29T16:07:06.443 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6821/13248", 2022-06-29T16:07:06.443 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:07:06.443 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:07:06.443 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:07:06.444 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.444 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.445 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.445 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:07:06.445 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "8564dd25-27ab-4203-b54e-af6d3aa05857", 2022-06-29T16:07:06.446 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:07:06.446 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:07:06.446 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:07:06.446 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:07:06.447 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:07:06.447 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:07:06.447 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:07:06.447 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 119, 2022-06-29T16:07:06.448 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:07:06.448 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:07:06.448 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:07:06.448 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:06.448 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.449 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:06.449 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6816", 2022-06-29T16:07:06.449 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:07:06.449 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.449 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.450 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:06.450 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6819", 2022-06-29T16:07:06.450 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:07:06.450 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.451 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.451 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.451 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:07:06.451 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:06.451 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.452 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:06.452 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6822", 2022-06-29T16:07:06.452 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:07:06.452 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.453 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.453 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:06.453 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6826", 2022-06-29T16:07:06.453 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:07:06.453 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.454 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.454 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.454 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:07:06.454 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:06.455 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.455 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:06.455 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6830", 2022-06-29T16:07:06.455 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:07:06.456 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.456 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.457 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:06.457 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6831", 2022-06-29T16:07:06.458 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:07:06.458 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.459 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.459 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.459 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:07:06.460 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:06.460 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.461 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:06.461 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6828", 2022-06-29T16:07:06.461 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:07:06.462 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.462 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.463 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:06.463 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6829", 2022-06-29T16:07:06.464 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:07:06.464 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.464 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.465 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.465 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6819/13250", 2022-06-29T16:07:06.465 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6826/13250", 2022-06-29T16:07:06.466 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6831/13250", 2022-06-29T16:07:06.466 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6829/13250", 2022-06-29T16:07:06.467 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:07:06.467 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:07:06.467 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:07:06.467 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.468 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.468 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.468 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:07:06.468 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "bd513a9c-3b19-4314-9897-d300b5c897bb", 2022-06-29T16:07:06.469 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:07:06.469 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:07:06.469 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:07:06.469 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:07:06.469 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:07:06.470 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:07:06.470 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:07:06.470 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 119, 2022-06-29T16:07:06.470 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:07:06.471 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:07:06.471 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:07:06.471 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:06.471 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.471 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:06.472 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6800", 2022-06-29T16:07:06.472 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:07:06.472 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.472 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.472 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:06.473 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6801", 2022-06-29T16:07:06.473 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:07:06.473 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.473 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.474 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.474 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:07:06.474 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:06.474 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.474 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:06.475 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6802", 2022-06-29T16:07:06.475 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:07:06.475 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.475 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.476 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:06.476 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6803", 2022-06-29T16:07:06.476 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:07:06.476 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.476 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.477 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.477 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:07:06.477 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:06.477 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.478 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:06.478 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6806", 2022-06-29T16:07:06.478 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:07:06.478 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.478 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.479 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:06.479 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6807", 2022-06-29T16:07:06.479 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:07:06.479 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.480 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.480 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.480 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:07:06.480 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:06.480 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.481 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:06.481 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6804", 2022-06-29T16:07:06.481 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:07:06.481 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.482 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.482 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:06.482 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6805", 2022-06-29T16:07:06.482 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:07:06.482 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.483 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.483 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.483 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6801/13247", 2022-06-29T16:07:06.483 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6803/13247", 2022-06-29T16:07:06.484 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6807/13247", 2022-06-29T16:07:06.484 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6805/13247", 2022-06-29T16:07:06.484 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:07:06.484 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:07:06.484 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:07:06.485 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:06.485 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.485 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:07:06.485 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_xinfo": [ 2022-06-29T16:07:06.486 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.486 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:07:06.486 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:07:06.486 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:07:06.486 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:07:06.487 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:07:06.487 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:07:06.487 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.891342+0000", 2022-06-29T16:07:06.487 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:07:06.487 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.488 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.488 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:07:06.488 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:07:06.488 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:07:06.489 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:07:06.489 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:07:06.489 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:07:06.489 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.900866+0000", 2022-06-29T16:07:06.490 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:07:06.490 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.490 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.490 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:07:06.490 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:07:06.491 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:07:06.491 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:07:06.491 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:07:06.491 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:07:06.492 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.912755+0000", 2022-06-29T16:07:06.492 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:07:06.492 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.492 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.492 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:07:06.493 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:07:06.493 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:07:06.493 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:07:06.493 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:07:06.494 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:07:06.494 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.864332+0000", 2022-06-29T16:07:06.494 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:07:06.494 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.495 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.495 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:07:06.495 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:07:06.495 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:07:06.496 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:07:06.496 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:07:06.496 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:07:06.496 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.863756+0000", 2022-06-29T16:07:06.497 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:07:06.497 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.497 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.497 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:07:06.497 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:07:06.498 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:07:06.498 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:07:06.498 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:07:06.498 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:07:06.499 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.869041+0000", 2022-06-29T16:07:06.499 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:07:06.499 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.499 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.499 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:07:06.500 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:07:06.500 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:07:06.500 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:07:06.500 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:07:06.500 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:07:06.501 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.895043+0000", 2022-06-29T16:07:06.501 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:07:06.501 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.501 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:06.502 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:07:06.502 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:07:06.502 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:07:06.502 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:07:06.503 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:07:06.503 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:07:06.503 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.889491+0000", 2022-06-29T16:07:06.503 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:07:06.503 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.504 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:07:06.504 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap": [], 2022-06-29T16:07:06.504 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap_items": [], 2022-06-29T16:07:06.504 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_temp": [], 2022-06-29T16:07:06.504 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_temp": [], 2022-06-29T16:07:06.505 INFO:teuthology.orchestra.run.smithi060.stdout: "blacklist": { 2022-06-29T16:07:06.505 INFO:teuthology.orchestra.run.smithi060.stdout: "172.21.15.60:6833/2749128920": "2022-06-30T16:06:51.286403+0000", 2022-06-29T16:07:06.505 INFO:teuthology.orchestra.run.smithi060.stdout: "172.21.15.60:6832/2749128920": "2022-06-30T16:06:51.286403+0000" 2022-06-29T16:07:06.505 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.506 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profiles": { 2022-06-29T16:07:06.506 INFO:teuthology.orchestra.run.smithi060.stdout: "default": { 2022-06-29T16:07:06.506 INFO:teuthology.orchestra.run.smithi060.stdout: "crush-failure-domain": "osd", 2022-06-29T16:07:06.506 INFO:teuthology.orchestra.run.smithi060.stdout: "k": "2", 2022-06-29T16:07:06.506 INFO:teuthology.orchestra.run.smithi060.stdout: "m": "1", 2022-06-29T16:07:06.507 INFO:teuthology.orchestra.run.smithi060.stdout: "plugin": "jerasure", 2022-06-29T16:07:06.507 INFO:teuthology.orchestra.run.smithi060.stdout: "ruleset-failure-domain": "osd", 2022-06-29T16:07:06.507 INFO:teuthology.orchestra.run.smithi060.stdout: "technique": "reed_sol_van" 2022-06-29T16:07:06.507 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:06.508 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:06.508 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps_queue": [], 2022-06-29T16:07:06.508 INFO:teuthology.orchestra.run.smithi060.stdout: "new_removed_snaps": [], 2022-06-29T16:07:06.508 INFO:teuthology.orchestra.run.smithi060.stdout: "new_purged_snaps": [], 2022-06-29T16:07:06.509 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_node_flags": {}, 2022-06-29T16:07:06.509 INFO:teuthology.orchestra.run.smithi060.stdout: "device_class_flags": {} 2022-06-29T16:07:06.509 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:07:06.511 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.213+0000 7f429a9b6700 1 -- 172.21.15.60:0/1891351503 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4274053a70 msgr2=0x7f4274055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:06.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.213+0000 7f429a9b6700 1 --2- 172.21.15.60:0/1891351503 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4274053a70 0x7f4274055f30 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0x7f4290016000 tx=0x7f4290014000).stop 2022-06-29T16:07:06.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.213+0000 7f429a9b6700 1 -- 172.21.15.60:0/1891351503 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42940aca20 msgr2=0x7f4294155110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:06.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.213+0000 7f429a9b6700 1 --2- 172.21.15.60:0/1891351503 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42940aca20 0x7f4294155110 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 rx=0x7f428400c000 tx=0x7f4284006df0).stop 2022-06-29T16:07:06.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.213+0000 7f429a9b6700 1 -- 172.21.15.60:0/1891351503 shutdown_connections 2022-06-29T16:07:06.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.213+0000 7f429a9b6700 1 --2- 172.21.15.60:0/1891351503 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4274053a70 0x7f4274055f30 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:06.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.213+0000 7f429a9b6700 1 --2- 172.21.15.60:0/1891351503 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4294143030 0x7f42940665e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:06.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.213+0000 7f429a9b6700 1 --2- 172.21.15.60:0/1891351503 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42940b3270 0x7f42941450c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:06.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.213+0000 7f429a9b6700 1 --2- 172.21.15.60:0/1891351503 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42940aca20 0x7f4294155110 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:06.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.213+0000 7f429a9b6700 1 -- 172.21.15.60:0/1891351503 >> 172.21.15.60:0/1891351503 conn(0x7f42940b6f80 msgr2=0x7f42940621a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:06.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.213+0000 7f429a9b6700 1 -- 172.21.15.60:0/1891351503 shutdown_connections 2022-06-29T16:07:06.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.213+0000 7f429a9b6700 1 -- 172.21.15.60:0/1891351503 wait complete. 2022-06-29T16:07:06.515 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:07:06.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.617+0000 7efd65172700 1 Processor -- start 2022-06-29T16:07:06.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.617+0000 7efd65172700 1 -- start start 2022-06-29T16:07:06.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.617+0000 7efd65172700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7efd600b2130 0x7efd60146100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:06.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.617+0000 7efd65172700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efd600aed00 0x7efd601465f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:06.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.617+0000 7efd65172700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd6005d450 con 0x7efd6005d070 2022-06-29T16:07:06.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.617+0000 7efd65172700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd600adf90 con 0x7efd600aed00 2022-06-29T16:07:06.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.617+0000 7efd65172700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd600b25d0 con 0x7efd600b2130 2022-06-29T16:07:06.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.621+0000 7efd64971700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efd600aed00 0x7efd601465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:06.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.621+0000 7efd5ffff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7efd6005d070 0x7efd6014aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60636/0 (socket says 172.21.15.60:60636) 2022-06-29T16:07:06.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.621+0000 7efd5ffff700 1 -- 172.21.15.60:0/3392230524 learned_addr learned my addr 172.21.15.60:0/3392230524 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:06.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.621+0000 7efd5f7fe700 1 --2- 172.21.15.60:0/3392230524 >> v2:172.21.15.92:3300/0 conn(0x7efd600b2130 0x7efd60146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:06.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.621+0000 7efd64971700 1 -- 172.21.15.60:0/3392230524 >> v2:172.21.15.92:3300/0 conn(0x7efd600b2130 msgr2=0x7efd60146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:06.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.621+0000 7efd64971700 1 --2- 172.21.15.60:0/3392230524 >> v2:172.21.15.92:3300/0 conn(0x7efd600b2130 0x7efd60146100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:06.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.621+0000 7efd64971700 1 -- 172.21.15.60:0/3392230524 >> v1:172.21.15.60:6789/0 conn(0x7efd6005d070 legacy=0x7efd6014aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:06.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.621+0000 7efd5effd700 1 -- 172.21.15.60:0/3392230524 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1904473434 0 0) 0x7efd6005d450 con 0x7efd6005d070 2022-06-29T16:07:06.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.621+0000 7efd64971700 1 -- 172.21.15.60:0/3392230524 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd6014a350 con 0x7efd600aed00 2022-06-29T16:07:06.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.621+0000 7efd64971700 1 --2- 172.21.15.60:0/3392230524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd600aed00 0x7efd601465f0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 rx=0x7efd58002f70 tx=0x7efd58006df0).ready entity=mon.0 client_cookie=5118ef40b3a483ed server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:06.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.621+0000 7efd5effd700 1 -- 172.21.15.60:0/3392230524 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd5800d570 con 0x7efd600aed00 2022-06-29T16:07:06.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.621+0000 7efd5effd700 1 -- 172.21.15.60:0/3392230524 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd58000e90 con 0x7efd600aed00 2022-06-29T16:07:06.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.621+0000 7efd5effd700 1 -- 172.21.15.60:0/3392230524 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd5800d8c0 con 0x7efd600aed00 2022-06-29T16:07:06.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.621+0000 7efd65172700 1 -- 172.21.15.60:0/3392230524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd600aed00 msgr2=0x7efd601465f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:06.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.621+0000 7efd65172700 1 --2- 172.21.15.60:0/3392230524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd600aed00 0x7efd601465f0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 rx=0x7efd58002f70 tx=0x7efd58006df0).stop 2022-06-29T16:07:06.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.621+0000 7efd65172700 1 -- 172.21.15.60:0/3392230524 shutdown_connections 2022-06-29T16:07:06.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.621+0000 7efd65172700 1 --2- 172.21.15.60:0/3392230524 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd600aed00 0x7efd601465f0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:06.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.621+0000 7efd65172700 1 --2- 172.21.15.60:0/3392230524 >> v2:172.21.15.92:3300/0 conn(0x7efd600b2130 0x7efd60146100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:06.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.621+0000 7efd65172700 1 -- 172.21.15.60:0/3392230524 >> 172.21.15.60:0/3392230524 conn(0x7efd600b5890 msgr2=0x7efd600b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:06.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.621+0000 7efd65172700 1 -- 172.21.15.60:0/3392230524 shutdown_connections 2022-06-29T16:07:06.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.621+0000 7efd65172700 1 -- 172.21.15.60:0/3392230524 wait complete. 2022-06-29T16:07:06.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.625+0000 7efd65172700 1 Processor -- start 2022-06-29T16:07:06.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.625+0000 7efd65172700 1 -- start start 2022-06-29T16:07:06.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.625+0000 7efd65172700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd6005d070 0x7efd60116ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:06.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.625+0000 7efd65172700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd600aed00 0x7efd60156bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:06.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.625+0000 7efd65172700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd600b2130 0x7efd60065250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:06.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.625+0000 7efd65172700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd60066ef0 con 0x7efd6005d070 2022-06-29T16:07:06.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.625+0000 7efd65172700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd601561f0 con 0x7efd600b2130 2022-06-29T16:07:06.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.625+0000 7efd65172700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd60156490 con 0x7efd600aed00 2022-06-29T16:07:06.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.625+0000 7efd5ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd6005d070 0x7efd60116ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:06.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.625+0000 7efd5ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd6005d070 0x7efd60116ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52554/0 (socket says 172.21.15.60:52554) 2022-06-29T16:07:06.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.625+0000 7efd5ffff700 1 -- 172.21.15.60:0/163800579 learned_addr learned my addr 172.21.15.60:0/163800579 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:06.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.625+0000 7efd64971700 1 --2- 172.21.15.60:0/163800579 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd600b2130 0x7efd60065250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:06.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.625+0000 7efd5f7fe700 1 --2- 172.21.15.60:0/163800579 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd600aed00 0x7efd60156bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:06.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.625+0000 7efd5ffff700 1 -- 172.21.15.60:0/163800579 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd600aed00 msgr2=0x7efd60156bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:06.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.625+0000 7efd5ffff700 1 --2- 172.21.15.60:0/163800579 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd600aed00 0x7efd60156bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:06.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.625+0000 7efd5ffff700 1 -- 172.21.15.60:0/163800579 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd600b2130 msgr2=0x7efd60065250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:06.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.625+0000 7efd5ffff700 1 --2- 172.21.15.60:0/163800579 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd600b2130 0x7efd60065250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:06.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.625+0000 7efd5ffff700 1 -- 172.21.15.60:0/163800579 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd5800b180 con 0x7efd6005d070 2022-06-29T16:07:06.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.625+0000 7efd64971700 1 --2- 172.21.15.60:0/163800579 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd600b2130 0x7efd60065250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:06.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.625+0000 7efd5f7fe700 1 --2- 172.21.15.60:0/163800579 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd600aed00 0x7efd60156bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:06.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.625+0000 7efd5ffff700 1 --2- 172.21.15.60:0/163800579 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd6005d070 0x7efd60116ac0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 rx=0x7efd50000ed0 tx=0x7efd50007ce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:06.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.625+0000 7efd5d7fa700 1 -- 172.21.15.60:0/163800579 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd5000ce20 con 0x7efd6005d070 2022-06-29T16:07:06.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.625+0000 7efd5d7fa700 1 -- 172.21.15.60:0/163800579 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd500072e0 con 0x7efd6005d070 2022-06-29T16:07:06.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.625+0000 7efd65172700 1 -- 172.21.15.60:0/163800579 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd60156730 con 0x7efd6005d070 2022-06-29T16:07:06.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.625+0000 7efd5d7fa700 1 -- 172.21.15.60:0/163800579 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd50005610 con 0x7efd6005d070 2022-06-29T16:07:06.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.625+0000 7efd65172700 1 -- 172.21.15.60:0/163800579 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd60065740 con 0x7efd6005d070 2022-06-29T16:07:06.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.629+0000 7efd5d7fa700 1 -- 172.21.15.60:0/163800579 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7efd50010070 con 0x7efd6005d070 2022-06-29T16:07:06.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.629+0000 7efd65172700 1 -- 172.21.15.60:0/163800579 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd6014a5f0 con 0x7efd6005d070 2022-06-29T16:07:06.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.629+0000 7efd5d7fa700 1 --2- 172.21.15.60:0/163800579 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd380539e0 0x7efd38055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:06.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.629+0000 7efd5d7fa700 1 -- 172.21.15.60:0/163800579 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6159+0+0 (secure 0 0 0) 0x7efd5006b720 con 0x7efd6005d070 2022-06-29T16:07:06.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.629+0000 7efd5f7fe700 1 --2- 172.21.15.60:0/163800579 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd380539e0 0x7efd38055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:06.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.629+0000 7efd5f7fe700 1 --2- 172.21.15.60:0/163800579 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd380539e0 0x7efd38055ea0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7efd4800b000 tx=0x7efd48008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:06.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.637+0000 7efd5d7fa700 1 -- 172.21.15.60:0/163800579 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efd50042570 con 0x7efd6005d070 2022-06-29T16:07:06.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.801+0000 7efd65172700 1 -- 172.21.15.60:0/163800579 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd601568c0 con 0x7efd6005d070 2022-06-29T16:07:06.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.801+0000 7efd5d7fa700 1 -- 172.21.15.60:0/163800579 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+1426 (secure 0 0 0) 0x7efd50045b90 con 0x7efd6005d070 2022-06-29T16:07:06.806 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:06.806 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":56,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:07:05.767077+0000","modified":"2022-06-29T16:07:05.767153+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2022-06-29T16:07:06.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.801+0000 7efd65172700 1 -- 172.21.15.60:0/163800579 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd380539e0 msgr2=0x7efd38055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:06.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.801+0000 7efd65172700 1 --2- 172.21.15.60:0/163800579 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd380539e0 0x7efd38055ea0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7efd4800b000 tx=0x7efd48008040).stop 2022-06-29T16:07:06.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.801+0000 7efd65172700 1 -- 172.21.15.60:0/163800579 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd6005d070 msgr2=0x7efd60116ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:06.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.801+0000 7efd65172700 1 --2- 172.21.15.60:0/163800579 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd6005d070 0x7efd60116ac0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 rx=0x7efd50000ed0 tx=0x7efd50007ce0).stop 2022-06-29T16:07:06.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.801+0000 7efd65172700 1 -- 172.21.15.60:0/163800579 shutdown_connections 2022-06-29T16:07:06.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.801+0000 7efd65172700 1 --2- 172.21.15.60:0/163800579 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd380539e0 0x7efd38055ea0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:06.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.801+0000 7efd65172700 1 --2- 172.21.15.60:0/163800579 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd600b2130 0x7efd60065250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:06.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.801+0000 7efd65172700 1 --2- 172.21.15.60:0/163800579 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd600aed00 0x7efd60156bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:06.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.801+0000 7efd65172700 1 --2- 172.21.15.60:0/163800579 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd6005d070 0x7efd60116ac0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:06.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.801+0000 7efd65172700 1 -- 172.21.15.60:0/163800579 >> 172.21.15.60:0/163800579 conn(0x7efd600b5890 msgr2=0x7efd601433b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:06.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.801+0000 7efd65172700 1 -- 172.21.15.60:0/163800579 shutdown_connections 2022-06-29T16:07:06.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.801+0000 7efd65172700 1 -- 172.21.15.60:0/163800579 wait complete. 2022-06-29T16:07:06.811 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 56 2022-06-29T16:07:06.819 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-06-29T16:07:06.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.917+0000 7f7fc6cc5700 1 Processor -- start 2022-06-29T16:07:06.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.917+0000 7f7fc6cc5700 1 -- start start 2022-06-29T16:07:06.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.917+0000 7f7fc6cc5700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7fc00b6a50 0x7f7fc005fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:06.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.917+0000 7f7fc6cc5700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7fc0062230 0x7f7fc01465e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:06.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.917+0000 7f7fc6cc5700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7fc005f0b0 0x7f7fc0146ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:06.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.917+0000 7f7fc6cc5700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fc0146fc0 con 0x7f7fc0062230 2022-06-29T16:07:06.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.917+0000 7f7fc6cc5700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fc00b6e30 con 0x7f7fc005f0b0 2022-06-29T16:07:06.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.917+0000 7f7fc6cc5700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc0098970 con 0x7f7fc00b6a50 2022-06-29T16:07:06.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.917+0000 7f7fbffff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7fc0062230 0x7f7fc01465e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:06.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.917+0000 7f7fbffff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7fc0062230 0x7f7fc01465e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52564/0 (socket says 172.21.15.60:52564) 2022-06-29T16:07:06.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.917+0000 7f7fbffff700 1 -- 172.21.15.60:0/52853977 learned_addr learned my addr 172.21.15.60:0/52853977 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:06.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.917+0000 7f7fc4a61700 1 --2- 172.21.15.60:0/52853977 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7fc00b6a50 0x7f7fc005fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:06.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.917+0000 7f7fc5262700 1 --2- 172.21.15.60:0/52853977 >> v2:172.21.15.92:3300/0 conn(0x7f7fc005f0b0 0x7f7fc0146ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:06.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.917+0000 7f7fbffff700 1 -- 172.21.15.60:0/52853977 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7fc00b6a50 msgr2=0x7f7fc005fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:06.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.917+0000 7f7fbffff700 1 --2- 172.21.15.60:0/52853977 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7fc00b6a50 0x7f7fc005fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:06.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.917+0000 7f7fbffff700 1 -- 172.21.15.60:0/52853977 >> v2:172.21.15.92:3300/0 conn(0x7f7fc005f0b0 msgr2=0x7f7fc0146ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:06.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.917+0000 7f7fbffff700 1 --2- 172.21.15.60:0/52853977 >> v2:172.21.15.92:3300/0 conn(0x7f7fc005f0b0 0x7f7fc0146ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:06.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.917+0000 7f7fbffff700 1 -- 172.21.15.60:0/52853977 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc005fa40 con 0x7f7fc0062230 2022-06-29T16:07:06.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.917+0000 7f7fc4a61700 1 --2- 172.21.15.60:0/52853977 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7fc00b6a50 0x7f7fc005fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:06.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.917+0000 7f7fbffff700 1 --2- 172.21.15.60:0/52853977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fc0062230 0x7f7fc01465e0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 rx=0x7f7fb0002f70 tx=0x7f7fb0006eb0).ready entity=mon.0 client_cookie=b6ef10753689ab4d server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:06.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.917+0000 7f7fbf7fe700 1 -- 172.21.15.60:0/52853977 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fb000d570 con 0x7f7fc0062230 2022-06-29T16:07:06.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.917+0000 7f7fbf7fe700 1 -- 172.21.15.60:0/52853977 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fb0000e90 con 0x7f7fc0062230 2022-06-29T16:07:06.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.917+0000 7f7fbf7fe700 1 -- 172.21.15.60:0/52853977 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fb000d900 con 0x7f7fc0062230 2022-06-29T16:07:06.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.917+0000 7f7fc6cc5700 1 -- 172.21.15.60:0/52853977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fc0062230 msgr2=0x7f7fc01465e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:06.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.917+0000 7f7fc6cc5700 1 --2- 172.21.15.60:0/52853977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fc0062230 0x7f7fc01465e0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 rx=0x7f7fb0002f70 tx=0x7f7fb0006eb0).stop 2022-06-29T16:07:06.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.917+0000 7f7fc6cc5700 1 -- 172.21.15.60:0/52853977 shutdown_connections 2022-06-29T16:07:06.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.917+0000 7f7fc6cc5700 1 --2- 172.21.15.60:0/52853977 >> v2:172.21.15.92:3300/0 conn(0x7f7fc005f0b0 0x7f7fc0146ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:06.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.917+0000 7f7fc6cc5700 1 --2- 172.21.15.60:0/52853977 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fc0062230 0x7f7fc01465e0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:06.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.917+0000 7f7fc6cc5700 1 --2- 172.21.15.60:0/52853977 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7fc00b6a50 0x7f7fc005fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:06.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.917+0000 7f7fc6cc5700 1 -- 172.21.15.60:0/52853977 >> 172.21.15.60:0/52853977 conn(0x7f7fc00643e0 msgr2=0x7f7fc00b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:06.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.917+0000 7f7fc6cc5700 1 -- 172.21.15.60:0/52853977 shutdown_connections 2022-06-29T16:07:06.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.917+0000 7f7fc6cc5700 1 -- 172.21.15.60:0/52853977 wait complete. 2022-06-29T16:07:06.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.921+0000 7f7fc6cc5700 1 Processor -- start 2022-06-29T16:07:06.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.921+0000 7f7fc6cc5700 1 -- start start 2022-06-29T16:07:06.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.921+0000 7f7fc6cc5700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fc005f0b0 0x7f7fc0066610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:06.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.921+0000 7f7fc6cc5700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fc0062230 0x7f7fc014c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:06.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.921+0000 7f7fc6cc5700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7fc00b6a50 0x7f7fc00ad440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:06.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.921+0000 7f7fc6cc5700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc00afde0 con 0x7f7fc005f0b0 2022-06-29T16:07:06.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.921+0000 7f7fc6cc5700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc00ade10 con 0x7f7fc0062230 2022-06-29T16:07:06.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.921+0000 7f7fc6cc5700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc00affc0 con 0x7f7fc00b6a50 2022-06-29T16:07:06.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.921+0000 7f7fc4a61700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fc005f0b0 0x7f7fc0066610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:06.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.921+0000 7f7fc4a61700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fc005f0b0 0x7f7fc0066610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52568/0 (socket says 172.21.15.60:52568) 2022-06-29T16:07:06.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.921+0000 7f7fc4a61700 1 -- 172.21.15.60:0/1641366116 learned_addr learned my addr 172.21.15.60:0/1641366116 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:06.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.921+0000 7f7fc5262700 1 --2- 172.21.15.60:0/1641366116 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7fc00b6a50 0x7f7fc00ad440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:06.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.921+0000 7f7fbffff700 1 --2- 172.21.15.60:0/1641366116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fc0062230 0x7f7fc014c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:06.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.921+0000 7f7fc4a61700 1 -- 172.21.15.60:0/1641366116 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7fc00b6a50 msgr2=0x7f7fc00ad440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:06.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.921+0000 7f7fc4a61700 1 --2- 172.21.15.60:0/1641366116 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7fc00b6a50 0x7f7fc00ad440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:06.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.921+0000 7f7fc4a61700 1 -- 172.21.15.60:0/1641366116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fc0062230 msgr2=0x7f7fc014c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:06.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.921+0000 7f7fc4a61700 1 --2- 172.21.15.60:0/1641366116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fc0062230 0x7f7fc014c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:06.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.921+0000 7f7fc4a61700 1 -- 172.21.15.60:0/1641366116 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb000b1e0 con 0x7f7fc005f0b0 2022-06-29T16:07:06.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.921+0000 7f7fbffff700 1 --2- 172.21.15.60:0/1641366116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fc0062230 0x7f7fc014c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:06.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.921+0000 7f7fc4a61700 1 --2- 172.21.15.60:0/1641366116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fc005f0b0 0x7f7fc0066610 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 rx=0x7f7fa8006940 tx=0x7f7fa8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:06.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.921+0000 7f7fbdffb700 1 -- 172.21.15.60:0/1641366116 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fa8011780 con 0x7f7fc005f0b0 2022-06-29T16:07:06.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.921+0000 7f7fbdffb700 1 -- 172.21.15.60:0/1641366116 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa80118e0 con 0x7f7fc005f0b0 2022-06-29T16:07:06.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.921+0000 7f7fbdffb700 1 -- 172.21.15.60:0/1641366116 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fa8011c40 con 0x7f7fc005f0b0 2022-06-29T16:07:06.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.921+0000 7f7fc6cc5700 1 -- 172.21.15.60:0/1641366116 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fc00b0260 con 0x7f7fc005f0b0 2022-06-29T16:07:06.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.921+0000 7f7fc6cc5700 1 -- 172.21.15.60:0/1641366116 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fc0065f70 con 0x7f7fc005f0b0 2022-06-29T16:07:06.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.925+0000 7f7fc6cc5700 1 -- 172.21.15.60:0/1641366116 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fc00573d0 con 0x7f7fc005f0b0 2022-06-29T16:07:06.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.925+0000 7f7fbdffb700 1 -- 172.21.15.60:0/1641366116 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7fa8012420 con 0x7f7fc005f0b0 2022-06-29T16:07:06.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.925+0000 7f7fbdffb700 1 --2- 172.21.15.60:0/1641366116 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7fa4053950 0x7f7fa4055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:06.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.925+0000 7f7fbdffb700 1 -- 172.21.15.60:0/1641366116 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f7fa806be80 con 0x7f7fc005f0b0 2022-06-29T16:07:06.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.925+0000 7f7fbffff700 1 --2- 172.21.15.60:0/1641366116 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7fa4053950 0x7f7fa4055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:06.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.925+0000 7f7fbffff700 1 --2- 172.21.15.60:0/1641366116 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7fa4053950 0x7f7fa4055e10 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7f7fb001a000 tx=0x7f7fb0015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:06.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:06.933+0000 7f7fbdffb700 1 -- 172.21.15.60:0/1641366116 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7fa8042be0 con 0x7f7fc005f0b0 2022-06-29T16:07:07.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.097+0000 7f7fc6cc5700 1 -- 172.21.15.60:0/1641366116 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f7fc0067990 con 0x7f7fc005f0b0 2022-06-29T16:07:07.781 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.777+0000 7f7fbdffb700 1 -- 172.21.15.60:0/1641366116 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v57) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7fa8046200 con 0x7f7fc005f0b0 2022-06-29T16:07:07.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.781+0000 7f7fc6cc5700 1 -- 172.21.15.60:0/1641366116 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7fa4053950 msgr2=0x7f7fa4055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:07.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.781+0000 7f7fc6cc5700 1 --2- 172.21.15.60:0/1641366116 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7fa4053950 0x7f7fa4055e10 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7f7fb001a000 tx=0x7f7fb0015000).stop 2022-06-29T16:07:07.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.781+0000 7f7fc6cc5700 1 -- 172.21.15.60:0/1641366116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fc005f0b0 msgr2=0x7f7fc0066610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:07.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.781+0000 7f7fc6cc5700 1 --2- 172.21.15.60:0/1641366116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fc005f0b0 0x7f7fc0066610 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 rx=0x7f7fa8006940 tx=0x7f7fa8008040).stop 2022-06-29T16:07:07.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.781+0000 7f7fc6cc5700 1 -- 172.21.15.60:0/1641366116 shutdown_connections 2022-06-29T16:07:07.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.781+0000 7f7fc6cc5700 1 --2- 172.21.15.60:0/1641366116 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7fa4053950 0x7f7fa4055e10 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:07.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.781+0000 7f7fc6cc5700 1 --2- 172.21.15.60:0/1641366116 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7fc00b6a50 0x7f7fc00ad440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:07.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.781+0000 7f7fc6cc5700 1 --2- 172.21.15.60:0/1641366116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fc0062230 0x7f7fc014c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:07.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.781+0000 7f7fc6cc5700 1 --2- 172.21.15.60:0/1641366116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fc005f0b0 0x7f7fc0066610 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:07.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.781+0000 7f7fc6cc5700 1 -- 172.21.15.60:0/1641366116 >> 172.21.15.60:0/1641366116 conn(0x7f7fc00643e0 msgr2=0x7f7fc00ac2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:07.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.781+0000 7f7fc6cc5700 1 -- 172.21.15.60:0/1641366116 shutdown_connections 2022-06-29T16:07:07.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.781+0000 7f7fc6cc5700 1 -- 172.21.15.60:0/1641366116 wait complete. 2022-06-29T16:07:07.789 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-06-29T16:07:07.801 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:07:07.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.905+0000 7fbc50f4e700 1 Processor -- start 2022-06-29T16:07:07.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.905+0000 7fbc50f4e700 1 -- start start 2022-06-29T16:07:07.915 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.905+0000 7fbc50f4e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbc4c0b48b0 0x7fbc4c142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:07.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.905+0000 7fbc50f4e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc4c0b0cf0 con 0x7fbc4c0b2c50 2022-06-29T16:07:07.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.905+0000 7fbc50f4e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc4c0b4d10 con 0x7fbc4c0b5eb0 2022-06-29T16:07:07.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.905+0000 7fbc50f4e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc4c0b54e0 con 0x7fbc4c0b48b0 2022-06-29T16:07:07.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.905+0000 7fbc4bfff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fbc4c0b2c50 0x7fbc4c14d380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60666/0 (socket says 172.21.15.60:60666) 2022-06-29T16:07:07.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.905+0000 7fbc4bfff700 1 -- 172.21.15.60:0/2422364809 learned_addr learned my addr 172.21.15.60:0/2422364809 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:07.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.905+0000 7fbc4affd700 1 --2- 172.21.15.60:0/2422364809 >> v2:172.21.15.92:3300/0 conn(0x7fbc4c0b48b0 0x7fbc4c142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:07.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc4a7fc700 1 -- 172.21.15.60:0/2422364809 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1971154894 0 0) 0x7fbc44001110 con 0x7fbc4c0b2c50 2022-06-29T16:07:07.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc4a7fc700 1 -- 172.21.15.60:0/2422364809 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc2c003260 con 0x7fbc4c0b2c50 2022-06-29T16:07:07.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc4a7fc700 1 -- 172.21.15.60:0/2422364809 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2781019923 0 0) 0x7fbc2c003260 con 0x7fbc4c0b2c50 2022-06-29T16:07:07.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc4a7fc700 1 -- 172.21.15.60:0/2422364809 >> v2:172.21.15.92:3300/0 conn(0x7fbc4c0b48b0 msgr2=0x7fbc4c142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:07.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc4a7fc700 1 --2- 172.21.15.60:0/2422364809 >> v2:172.21.15.92:3300/0 conn(0x7fbc4c0b48b0 0x7fbc4c142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:07.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc4a7fc700 1 -- 172.21.15.60:0/2422364809 >> v1:172.21.15.92:6789/0 conn(0x7fbc4c0b5eb0 legacy=0x7fbc4c0b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:07.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc4a7fc700 1 -- 172.21.15.60:0/2422364809 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc4c0b6290 con 0x7fbc4c0b2c50 2022-06-29T16:07:07.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc4a7fc700 1 -- 172.21.15.60:0/2422364809 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7fbc440017f0 con 0x7fbc4c0b2c50 2022-06-29T16:07:07.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc4a7fc700 1 -- 172.21.15.60:0/2422364809 >> v1:172.21.15.60:6789/0 conn(0x7fbc4c0b2c50 legacy=0x7fbc4c14d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:07.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc4a7fc700 1 --2- 172.21.15.60:0/2422364809 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc2c003750 0x7fbc2c003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:07.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc4a7fc700 1 --2- 172.21.15.60:0/2422364809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc2c004500 0x7fbc2c004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:07.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc4a7fc700 1 --2- 172.21.15.60:0/2422364809 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc2c004e70 0x7fbc2c00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:07.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc4a7fc700 1 -- 172.21.15.60:0/2422364809 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4c0b54e0 con 0x7fbc2c003750 2022-06-29T16:07:07.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc4a7fc700 1 -- 172.21.15.60:0/2422364809 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2c00bdd0 con 0x7fbc2c004500 2022-06-29T16:07:07.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc4a7fc700 1 -- 172.21.15.60:0/2422364809 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2c00bf60 con 0x7fbc2c004e70 2022-06-29T16:07:07.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc4b7fe700 1 --2- 172.21.15.60:0/2422364809 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc2c003750 0x7fbc2c003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:07.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc4affd700 1 --2- 172.21.15.60:0/2422364809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc2c004500 0x7fbc2c004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:07.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc4bfff700 1 --2- 172.21.15.60:0/2422364809 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc2c004e70 0x7fbc2c00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:07.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc4bfff700 1 -- 172.21.15.60:0/2422364809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc2c004500 msgr2=0x7fbc2c004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:07.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc4bfff700 1 --2- 172.21.15.60:0/2422364809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc2c004500 0x7fbc2c004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:07.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc4bfff700 1 -- 172.21.15.60:0/2422364809 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc2c003750 msgr2=0x7fbc2c003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:07.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc4bfff700 1 --2- 172.21.15.60:0/2422364809 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc2c003750 0x7fbc2c003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:07.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc4bfff700 1 -- 172.21.15.60:0/2422364809 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc2c003260 con 0x7fbc2c004e70 2022-06-29T16:07:07.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc4b7fe700 1 --2- 172.21.15.60:0/2422364809 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc2c003750 0x7fbc2c003b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:07:07.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc4bfff700 1 -- 172.21.15.60:0/2422364809 reap_dead start 2022-06-29T16:07:07.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc4bfff700 1 --2- 172.21.15.60:0/2422364809 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc2c004e70 0x7fbc2c00b750 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fbc44001500 tx=0x7fbc4c0b6bd0).ready entity=mon.2 client_cookie=61bc0fff7c8e0ada server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:07.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc4a7fc700 1 -- 172.21.15.60:0/2422364809 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc440070d0 con 0x7fbc2c004e70 2022-06-29T16:07:07.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc4a7fc700 1 -- 172.21.15.60:0/2422364809 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc44007230 con 0x7fbc2c004e70 2022-06-29T16:07:07.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc4a7fc700 1 -- 172.21.15.60:0/2422364809 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc4400ca50 con 0x7fbc2c004e70 2022-06-29T16:07:07.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc50f4e700 1 -- 172.21.15.60:0/2422364809 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc2c004e70 msgr2=0x7fbc2c00b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:07.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc50f4e700 1 --2- 172.21.15.60:0/2422364809 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc2c004e70 0x7fbc2c00b750 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fbc44001500 tx=0x7fbc4c0b6bd0).stop 2022-06-29T16:07:07.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc50f4e700 1 -- 172.21.15.60:0/2422364809 shutdown_connections 2022-06-29T16:07:07.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc50f4e700 1 --2- 172.21.15.60:0/2422364809 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc2c004e70 0x7fbc2c00b750 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:07.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc50f4e700 1 -- 172.21.15.60:0/2422364809 >> 172.21.15.60:0/2422364809 conn(0x7fbc4c062090 msgr2=0x7fbc4c062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:07.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc50f4e700 1 -- 172.21.15.60:0/2422364809 shutdown_connections 2022-06-29T16:07:07.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.909+0000 7fbc50f4e700 1 -- 172.21.15.60:0/2422364809 wait complete. 2022-06-29T16:07:07.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.913+0000 7fbc50f4e700 1 Processor -- start 2022-06-29T16:07:07.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.913+0000 7fbc50f4e700 1 -- start start 2022-06-29T16:07:07.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.913+0000 7fbc50f4e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc2c004e70 0x7fbc4c1473e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:07.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.913+0000 7fbc50f4e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc4c146ca0 0x7fbc4c149650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:07.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.913+0000 7fbc50f4e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc4c143ee0 0x7fbc4c144370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:07.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.913+0000 7fbc50f4e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4c144860 con 0x7fbc4c143ee0 2022-06-29T16:07:07.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.913+0000 7fbc50f4e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4c148d20 con 0x7fbc2c004e70 2022-06-29T16:07:07.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.913+0000 7fbc50f4e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4c149b40 con 0x7fbc4c146ca0 2022-06-29T16:07:07.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.913+0000 7fbc4bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc4c143ee0 0x7fbc4c144370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:07.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.913+0000 7fbc4bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc4c143ee0 0x7fbc4c144370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52592/0 (socket says 172.21.15.60:52592) 2022-06-29T16:07:07.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.913+0000 7fbc4bfff700 1 -- 172.21.15.60:0/1479869360 learned_addr learned my addr 172.21.15.60:0/1479869360 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:07.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.913+0000 7fbc4b7fe700 1 --2- 172.21.15.60:0/1479869360 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc2c004e70 0x7fbc4c1473e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:07.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.913+0000 7fbc4affd700 1 --2- 172.21.15.60:0/1479869360 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc4c146ca0 0x7fbc4c149650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:07.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.913+0000 7fbc4bfff700 1 -- 172.21.15.60:0/1479869360 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc4c146ca0 msgr2=0x7fbc4c149650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:07.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.913+0000 7fbc4bfff700 1 --2- 172.21.15.60:0/1479869360 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc4c146ca0 0x7fbc4c149650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:07.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.913+0000 7fbc4bfff700 1 -- 172.21.15.60:0/1479869360 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc2c004e70 msgr2=0x7fbc4c1473e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:07.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.913+0000 7fbc4bfff700 1 --2- 172.21.15.60:0/1479869360 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc2c004e70 0x7fbc4c1473e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:07.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.913+0000 7fbc4bfff700 1 -- 172.21.15.60:0/1479869360 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc4c148f10 con 0x7fbc4c143ee0 2022-06-29T16:07:07.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.913+0000 7fbc4b7fe700 1 --2- 172.21.15.60:0/1479869360 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc2c004e70 0x7fbc4c1473e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:07:07.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.913+0000 7fbc4bfff700 1 --2- 172.21.15.60:0/1479869360 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc4c143ee0 0x7fbc4c144370 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 rx=0x7fbc44001500 tx=0x7fbc44015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:07.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.913+0000 7fbc48ff9700 1 -- 172.21.15.60:0/1479869360 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc440070d0 con 0x7fbc4c143ee0 2022-06-29T16:07:07.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.913+0000 7fbc48ff9700 1 -- 172.21.15.60:0/1479869360 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc44004710 con 0x7fbc4c143ee0 2022-06-29T16:07:07.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.913+0000 7fbc48ff9700 1 -- 172.21.15.60:0/1479869360 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc44001d60 con 0x7fbc4c143ee0 2022-06-29T16:07:07.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.913+0000 7fbc50f4e700 1 -- 172.21.15.60:0/1479869360 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc4c1491b0 con 0x7fbc4c143ee0 2022-06-29T16:07:07.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.913+0000 7fbc50f4e700 1 -- 172.21.15.60:0/1479869360 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc4c145680 con 0x7fbc4c143ee0 2022-06-29T16:07:07.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.917+0000 7fbc50f4e700 1 -- 172.21.15.60:0/1479869360 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc4c149340 con 0x7fbc4c143ee0 2022-06-29T16:07:07.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.917+0000 7fbc48ff9700 1 -- 172.21.15.60:0/1479869360 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fbc44010070 con 0x7fbc4c143ee0 2022-06-29T16:07:07.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.917+0000 7fbc48ff9700 1 --2- 172.21.15.60:0/1479869360 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbc30053b30 0x7fbc30055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:07.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.917+0000 7fbc48ff9700 1 -- 172.21.15.60:0/1479869360 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fbc4401c040 con 0x7fbc4c143ee0 2022-06-29T16:07:07.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.917+0000 7fbc4b7fe700 1 --2- 172.21.15.60:0/1479869360 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbc30053b30 0x7fbc30055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:07.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.917+0000 7fbc4b7fe700 1 --2- 172.21.15.60:0/1479869360 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbc30053b30 0x7fbc30055ff0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0x7fbc3c009000 tx=0x7fbc3c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:07.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:07.925+0000 7fbc48ff9700 1 -- 172.21.15.60:0/1479869360 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbc440430d0 con 0x7fbc4c143ee0 2022-06-29T16:07:08.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.089+0000 7fbc50f4e700 1 -- 172.21.15.60:0/1479869360 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc4c0b0cf0 con 0x7fbc4c143ee0 2022-06-29T16:07:08.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.089+0000 7fbc48ff9700 1 -- 172.21.15.60:0/1479869360 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+1426 (secure 0 0 0) 0x7fbc4c0b0cf0 con 0x7fbc4c143ee0 2022-06-29T16:07:08.094 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:08.095 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":57,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":57,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:07:05.767077+0000","modified":"2022-06-29T16:07:07.772201+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2022-06-29T16:07:08.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.093+0000 7fbc50f4e700 1 -- 172.21.15.60:0/1479869360 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbc30053b30 msgr2=0x7fbc30055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:08.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.093+0000 7fbc50f4e700 1 --2- 172.21.15.60:0/1479869360 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbc30053b30 0x7fbc30055ff0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0x7fbc3c009000 tx=0x7fbc3c009040).stop 2022-06-29T16:07:08.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.093+0000 7fbc50f4e700 1 -- 172.21.15.60:0/1479869360 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc4c143ee0 msgr2=0x7fbc4c144370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:08.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.093+0000 7fbc50f4e700 1 --2- 172.21.15.60:0/1479869360 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc4c143ee0 0x7fbc4c144370 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 rx=0x7fbc44001500 tx=0x7fbc44015000).stop 2022-06-29T16:07:08.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.093+0000 7fbc50f4e700 1 -- 172.21.15.60:0/1479869360 shutdown_connections 2022-06-29T16:07:08.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.093+0000 7fbc50f4e700 1 --2- 172.21.15.60:0/1479869360 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbc30053b30 0x7fbc30055ff0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:08.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.093+0000 7fbc50f4e700 1 --2- 172.21.15.60:0/1479869360 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc4c143ee0 0x7fbc4c144370 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:08.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.093+0000 7fbc50f4e700 1 --2- 172.21.15.60:0/1479869360 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc4c146ca0 0x7fbc4c149650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:08.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.093+0000 7fbc50f4e700 1 --2- 172.21.15.60:0/1479869360 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc2c004e70 0x7fbc4c1473e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:08.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.093+0000 7fbc50f4e700 1 -- 172.21.15.60:0/1479869360 >> 172.21.15.60:0/1479869360 conn(0x7fbc4c062090 msgr2=0x7fbc4c143550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:08.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.093+0000 7fbc50f4e700 1 -- 172.21.15.60:0/1479869360 shutdown_connections 2022-06-29T16:07:08.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.093+0000 7fbc50f4e700 1 -- 172.21.15.60:0/1479869360 wait complete. 2022-06-29T16:07:08.100 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 57 2022-06-29T16:07:08.113 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-06-29T16:07:08.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.213+0000 7fce13273700 1 Processor -- start 2022-06-29T16:07:08.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.213+0000 7fce13273700 1 -- start start 2022-06-29T16:07:08.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.217+0000 7fce13273700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fce0c0aee30 0x7fce0c0b1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:08.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.217+0000 7fce13273700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce0c05d450 con 0x7fce0c05d070 2022-06-29T16:07:08.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.217+0000 7fce13273700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce0c062200 con 0x7fce0c141b70 2022-06-29T16:07:08.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.217+0000 7fce13273700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce0c143a90 con 0x7fce0c0aee30 2022-06-29T16:07:08.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.217+0000 7fce12271700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fce0c05d070 0x7fce0c13a460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60682/0 (socket says 172.21.15.60:60682) 2022-06-29T16:07:08.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.217+0000 7fce11a70700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fce0c0aee30 0x7fce0c0b1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:08.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.217+0000 7fce12271700 1 -- 172.21.15.60:0/2266886436 learned_addr learned my addr 172.21.15.60:0/2266886436 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:08.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.217+0000 7fce1126f700 1 -- 172.21.15.60:0/2266886436 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228440790 0 0) 0x7fce0c05d450 con 0x7fce0c05d070 2022-06-29T16:07:08.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.217+0000 7fce1126f700 1 -- 172.21.15.60:0/2266886436 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdf4003260 con 0x7fce0c05d070 2022-06-29T16:07:08.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.217+0000 7fce11a70700 1 -- 172.21.15.60:0/2266886436 >> v1:172.21.15.92:6789/0 conn(0x7fce0c141b70 legacy=0x7fce0c140bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:08.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.217+0000 7fce11a70700 1 -- 172.21.15.60:0/2266886436 >> v1:172.21.15.60:6789/0 conn(0x7fce0c05d070 legacy=0x7fce0c13a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:08.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.217+0000 7fce11a70700 1 -- 172.21.15.60:0/2266886436 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce0c064dc0 con 0x7fce0c0aee30 2022-06-29T16:07:08.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.217+0000 7fce11a70700 1 --2- 172.21.15.60:0/2266886436 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce0c0aee30 0x7fce0c0b1f30 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0x7fce08002f70 tx=0x7fce08006df0).ready entity=mon.1 client_cookie=885c1b408c94398f server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:08.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.217+0000 7fce1126f700 1 -- 172.21.15.60:0/2266886436 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce0800d610 con 0x7fce0c0aee30 2022-06-29T16:07:08.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.217+0000 7fce1126f700 1 -- 172.21.15.60:0/2266886436 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce08000e90 con 0x7fce0c0aee30 2022-06-29T16:07:08.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.217+0000 7fce1126f700 1 -- 172.21.15.60:0/2266886436 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce0800d960 con 0x7fce0c0aee30 2022-06-29T16:07:08.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.217+0000 7fce13273700 1 -- 172.21.15.60:0/2266886436 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce0c0aee30 msgr2=0x7fce0c0b1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:08.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.217+0000 7fce13273700 1 --2- 172.21.15.60:0/2266886436 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce0c0aee30 0x7fce0c0b1f30 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0x7fce08002f70 tx=0x7fce08006df0).stop 2022-06-29T16:07:08.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.217+0000 7fce13273700 1 -- 172.21.15.60:0/2266886436 shutdown_connections 2022-06-29T16:07:08.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.217+0000 7fce13273700 1 --2- 172.21.15.60:0/2266886436 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce0c0aee30 0x7fce0c0b1f30 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:08.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.217+0000 7fce13273700 1 -- 172.21.15.60:0/2266886436 >> 172.21.15.60:0/2266886436 conn(0x7fce0c0b5890 msgr2=0x7fce0c0b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:08.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.217+0000 7fce13273700 1 -- 172.21.15.60:0/2266886436 shutdown_connections 2022-06-29T16:07:08.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.217+0000 7fce13273700 1 -- 172.21.15.60:0/2266886436 wait complete. 2022-06-29T16:07:08.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.221+0000 7fce13273700 1 Processor -- start 2022-06-29T16:07:08.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.221+0000 7fce13273700 1 -- start start 2022-06-29T16:07:08.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.221+0000 7fce13273700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce0c05d070 0x7fce0c05c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:08.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.221+0000 7fce13273700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce0c0aee30 0x7fce0c10fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:08.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.221+0000 7fce13273700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce0c141b70 0x7fce0c110370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:08.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.221+0000 7fce13273700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce0c138130 con 0x7fce0c0aee30 2022-06-29T16:07:08.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.221+0000 7fce13273700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce0c10e7a0 con 0x7fce0c141b70 2022-06-29T16:07:08.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.221+0000 7fce13273700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce0c0ae4c0 con 0x7fce0c05d070 2022-06-29T16:07:08.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.221+0000 7fce11a70700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce0c0aee30 0x7fce0c10fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:08.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.221+0000 7fce11a70700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce0c0aee30 0x7fce0c10fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52606/0 (socket says 172.21.15.60:52606) 2022-06-29T16:07:08.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.221+0000 7fce11a70700 1 -- 172.21.15.60:0/374518017 learned_addr learned my addr 172.21.15.60:0/374518017 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:08.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.221+0000 7fce12271700 1 --2- 172.21.15.60:0/374518017 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce0c05d070 0x7fce0c05c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:08.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.221+0000 7fce12a72700 1 --2- 172.21.15.60:0/374518017 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce0c141b70 0x7fce0c110370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:08.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.221+0000 7fce11a70700 1 -- 172.21.15.60:0/374518017 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce0c05d070 msgr2=0x7fce0c05c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:08.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.221+0000 7fce11a70700 1 --2- 172.21.15.60:0/374518017 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce0c05d070 0x7fce0c05c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:08.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.221+0000 7fce11a70700 1 -- 172.21.15.60:0/374518017 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce0c141b70 msgr2=0x7fce0c110370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:08.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.221+0000 7fce11a70700 1 --2- 172.21.15.60:0/374518017 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce0c141b70 0x7fce0c110370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:08.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.221+0000 7fce11a70700 1 -- 172.21.15.60:0/374518017 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce0800b180 con 0x7fce0c0aee30 2022-06-29T16:07:08.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.221+0000 7fce12a72700 1 --2- 172.21.15.60:0/374518017 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce0c141b70 0x7fce0c110370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:08.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.221+0000 7fce11a70700 1 --2- 172.21.15.60:0/374518017 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce0c0aee30 0x7fce0c10fe80 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 rx=0x7fce08002f70 tx=0x7fce08014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:08.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.221+0000 7fcdff7fe700 1 -- 172.21.15.60:0/374518017 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce08016070 con 0x7fce0c0aee30 2022-06-29T16:07:08.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.221+0000 7fcdff7fe700 1 -- 172.21.15.60:0/374518017 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce0800de90 con 0x7fce0c0aee30 2022-06-29T16:07:08.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.221+0000 7fce13273700 1 -- 172.21.15.60:0/374518017 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce0c0ae760 con 0x7fce0c0aee30 2022-06-29T16:07:08.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.221+0000 7fcdff7fe700 1 -- 172.21.15.60:0/374518017 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce08013e50 con 0x7fce0c0aee30 2022-06-29T16:07:08.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.221+0000 7fce13273700 1 -- 172.21.15.60:0/374518017 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce0c1108e0 con 0x7fce0c0aee30 2022-06-29T16:07:08.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.225+0000 7fce13273700 1 -- 172.21.15.60:0/374518017 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce0c0573d0 con 0x7fce0c0aee30 2022-06-29T16:07:08.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.225+0000 7fcdff7fe700 1 -- 172.21.15.60:0/374518017 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fce08024030 con 0x7fce0c0aee30 2022-06-29T16:07:08.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.225+0000 7fcdff7fe700 1 --2- 172.21.15.60:0/374518017 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcdec053950 0x7fcdec055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:08.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.225+0000 7fcdff7fe700 1 -- 172.21.15.60:0/374518017 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fce0806ccb0 con 0x7fce0c0aee30 2022-06-29T16:07:08.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.225+0000 7fce12271700 1 --2- 172.21.15.60:0/374518017 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcdec053950 0x7fcdec055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:08.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.225+0000 7fce12271700 1 --2- 172.21.15.60:0/374518017 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcdec053950 0x7fcdec055e10 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7fce0000d000 tx=0x7fce00009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:08.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.229+0000 7fcdff7fe700 1 -- 172.21.15.60:0/374518017 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fce08043b80 con 0x7fce0c0aee30 2022-06-29T16:07:08.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.401+0000 7fce13273700 1 -- 172.21.15.60:0/374518017 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fce0c13ab70 con 0x7fce0c0aee30 2022-06-29T16:07:08.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.937+0000 7fcdff7fe700 1 -- 172.21.15.60:0/374518017 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v58) v1 ==== 104+0+0 (secure 0 0 0) 0x7fce08022020 con 0x7fce0c0aee30 2022-06-29T16:07:08.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.941+0000 7fce13273700 1 -- 172.21.15.60:0/374518017 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcdec053950 msgr2=0x7fcdec055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:08.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.941+0000 7fce13273700 1 --2- 172.21.15.60:0/374518017 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcdec053950 0x7fcdec055e10 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7fce0000d000 tx=0x7fce00009040).stop 2022-06-29T16:07:08.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.941+0000 7fce13273700 1 -- 172.21.15.60:0/374518017 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce0c0aee30 msgr2=0x7fce0c10fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:08.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.941+0000 7fce13273700 1 --2- 172.21.15.60:0/374518017 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce0c0aee30 0x7fce0c10fe80 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 rx=0x7fce08002f70 tx=0x7fce08014000).stop 2022-06-29T16:07:08.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.941+0000 7fce13273700 1 -- 172.21.15.60:0/374518017 shutdown_connections 2022-06-29T16:07:08.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.941+0000 7fce13273700 1 --2- 172.21.15.60:0/374518017 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcdec053950 0x7fcdec055e10 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:08.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.941+0000 7fce13273700 1 --2- 172.21.15.60:0/374518017 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce0c141b70 0x7fce0c110370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:08.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.941+0000 7fce13273700 1 --2- 172.21.15.60:0/374518017 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fce0c0aee30 0x7fce0c10fe80 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:08.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.941+0000 7fce13273700 1 --2- 172.21.15.60:0/374518017 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fce0c05d070 0x7fce0c05c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:08.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.941+0000 7fce13273700 1 -- 172.21.15.60:0/374518017 >> 172.21.15.60:0/374518017 conn(0x7fce0c0b5890 msgr2=0x7fce0c1389e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:08.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.941+0000 7fce13273700 1 -- 172.21.15.60:0/374518017 shutdown_connections 2022-06-29T16:07:08.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:08.941+0000 7fce13273700 1 -- 172.21.15.60:0/374518017 wait complete. 2022-06-29T16:07:08.961 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-06-29T16:07:09.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.077+0000 7f229d4fe700 1 Processor -- start 2022-06-29T16:07:09.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.077+0000 7f229d4fe700 1 -- start start 2022-06-29T16:07:09.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.081+0000 7f229d4fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f229813dff0 0x7f22980b2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:09.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.081+0000 7f229d4fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f22980b6710 0x7f22981446c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:09.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.081+0000 7f229d4fe700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2298143ff0 con 0x7f2298061620 2022-06-29T16:07:09.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.081+0000 7f229d4fe700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2298061e50 con 0x7f22980b6710 2022-06-29T16:07:09.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.081+0000 7f229d4fe700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2298144220 con 0x7f229813dff0 2022-06-29T16:07:09.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.081+0000 7f22977fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f22980b6710 0x7f22981446c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:09.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.081+0000 7f228ffff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2298061620 0x7f2298140a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60698/0 (socket says 172.21.15.60:60698) 2022-06-29T16:07:09.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.081+0000 7f22977fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f22980b6710 0x7f22981446c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52612/0 (socket says 172.21.15.60:52612) 2022-06-29T16:07:09.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.081+0000 7f228ffff700 1 -- 172.21.15.60:0/3990752865 learned_addr learned my addr 172.21.15.60:0/3990752865 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:09.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.081+0000 7f2296ffd700 1 --2- 172.21.15.60:0/3990752865 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f229813dff0 0x7f22980b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:09.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.081+0000 7f22977fe700 1 -- 172.21.15.60:0/3990752865 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f229813dff0 msgr2=0x7f22980b2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:09.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.081+0000 7f22977fe700 1 --2- 172.21.15.60:0/3990752865 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f229813dff0 0x7f22980b2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:09.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.081+0000 7f22977fe700 1 -- 172.21.15.60:0/3990752865 >> v1:172.21.15.60:6789/0 conn(0x7f2298061620 legacy=0x7f2298140a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:09.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.081+0000 7f22977fe700 1 -- 172.21.15.60:0/3990752865 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f229813e3d0 con 0x7f22980b6710 2022-06-29T16:07:09.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.081+0000 7f22977fe700 1 --2- 172.21.15.60:0/3990752865 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22980b6710 0x7f22981446c0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 rx=0x7f22900032f0 tx=0x7f22900091d0).ready entity=mon.0 client_cookie=c28ffbdbeb837802 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:09.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.081+0000 7f22967fc700 1 -- 172.21.15.60:0/3990752865 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f229000b6c0 con 0x7f22980b6710 2022-06-29T16:07:09.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.081+0000 7f22967fc700 1 -- 172.21.15.60:0/3990752865 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f229000b820 con 0x7f22980b6710 2022-06-29T16:07:09.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.081+0000 7f22967fc700 1 -- 172.21.15.60:0/3990752865 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f229000b990 con 0x7f22980b6710 2022-06-29T16:07:09.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.081+0000 7f229d4fe700 1 -- 172.21.15.60:0/3990752865 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22980b6710 msgr2=0x7f22981446c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:09.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.081+0000 7f229d4fe700 1 --2- 172.21.15.60:0/3990752865 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22980b6710 0x7f22981446c0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 rx=0x7f22900032f0 tx=0x7f22900091d0).stop 2022-06-29T16:07:09.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.081+0000 7f229d4fe700 1 -- 172.21.15.60:0/3990752865 shutdown_connections 2022-06-29T16:07:09.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.081+0000 7f229d4fe700 1 --2- 172.21.15.60:0/3990752865 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f22980b6710 0x7f22981446c0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:09.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.081+0000 7f229d4fe700 1 --2- 172.21.15.60:0/3990752865 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f229813dff0 0x7f22980b2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:09.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.081+0000 7f229d4fe700 1 -- 172.21.15.60:0/3990752865 >> 172.21.15.60:0/3990752865 conn(0x7f22980643e0 msgr2=0x7f22980b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:09.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.081+0000 7f229d4fe700 1 -- 172.21.15.60:0/3990752865 shutdown_connections 2022-06-29T16:07:09.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.081+0000 7f229d4fe700 1 -- 172.21.15.60:0/3990752865 wait complete. 2022-06-29T16:07:09.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.085+0000 7f229d4fe700 1 Processor -- start 2022-06-29T16:07:09.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.085+0000 7f229d4fe700 1 -- start start 2022-06-29T16:07:09.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.085+0000 7f229d4fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2298061620 0x7f229805f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:09.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.085+0000 7f229d4fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f22980b6710 0x7f22980aedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:09.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.085+0000 7f229d4fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f229813dff0 0x7f22980af2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:09.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.085+0000 7f229d4fe700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229809a8b0 con 0x7f229813dff0 2022-06-29T16:07:09.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.085+0000 7f229d4fe700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229809ad70 con 0x7f2298061620 2022-06-29T16:07:09.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.085+0000 7f229d4fe700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f229809b010 con 0x7f22980b6710 2022-06-29T16:07:09.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.085+0000 7f22977fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f229813dff0 0x7f22980af2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:09.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.085+0000 7f22977fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f229813dff0 0x7f22980af2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52620/0 (socket says 172.21.15.60:52620) 2022-06-29T16:07:09.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.085+0000 7f22977fe700 1 -- 172.21.15.60:0/101830243 learned_addr learned my addr 172.21.15.60:0/101830243 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:09.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.085+0000 7f228ffff700 1 --2- 172.21.15.60:0/101830243 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f22980b6710 0x7f22980aedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:09.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.085+0000 7f2296ffd700 1 --2- 172.21.15.60:0/101830243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2298061620 0x7f229805f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:09.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.085+0000 7f22977fe700 1 -- 172.21.15.60:0/101830243 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f22980b6710 msgr2=0x7f22980aedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:09.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.085+0000 7f22977fe700 1 --2- 172.21.15.60:0/101830243 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f22980b6710 0x7f22980aedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:09.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.085+0000 7f22977fe700 1 -- 172.21.15.60:0/101830243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2298061620 msgr2=0x7f229805f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:09.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.085+0000 7f22977fe700 1 --2- 172.21.15.60:0/101830243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2298061620 0x7f229805f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:09.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.085+0000 7f22977fe700 1 -- 172.21.15.60:0/101830243 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2290009470 con 0x7f229813dff0 2022-06-29T16:07:09.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.085+0000 7f228ffff700 1 --2- 172.21.15.60:0/101830243 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f22980b6710 0x7f22980aedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:09.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.085+0000 7f2296ffd700 1 --2- 172.21.15.60:0/101830243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2298061620 0x7f229805f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:09.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.085+0000 7f22977fe700 1 --2- 172.21.15.60:0/101830243 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f229813dff0 0x7f22980af2c0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 rx=0x7f22900032f0 tx=0x7f2290016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:09.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.085+0000 7f2294ff9700 1 -- 172.21.15.60:0/101830243 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2290021440 con 0x7f229813dff0 2022-06-29T16:07:09.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.085+0000 7f2294ff9700 1 -- 172.21.15.60:0/101830243 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22900215a0 con 0x7f229813dff0 2022-06-29T16:07:09.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.085+0000 7f229d4fe700 1 -- 172.21.15.60:0/101830243 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2298099260 con 0x7f229813dff0 2022-06-29T16:07:09.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.085+0000 7f2294ff9700 1 -- 172.21.15.60:0/101830243 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2290021900 con 0x7f229813dff0 2022-06-29T16:07:09.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.085+0000 7f229d4fe700 1 -- 172.21.15.60:0/101830243 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22980ad5e0 con 0x7f229813dff0 2022-06-29T16:07:09.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.089+0000 7f229d4fe700 1 -- 172.21.15.60:0/101830243 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22980adbb0 con 0x7f229813dff0 2022-06-29T16:07:09.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.089+0000 7f2294ff9700 1 -- 172.21.15.60:0/101830243 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2290027020 con 0x7f229813dff0 2022-06-29T16:07:09.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.089+0000 7f2294ff9700 1 --2- 172.21.15.60:0/101830243 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f227c053980 0x7f227c055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:09.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.089+0000 7f2294ff9700 1 -- 172.21.15.60:0/101830243 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f229006d4d0 con 0x7f229813dff0 2022-06-29T16:07:09.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.089+0000 7f2296ffd700 1 --2- 172.21.15.60:0/101830243 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f227c053980 0x7f227c055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:09.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.093+0000 7f2296ffd700 1 --2- 172.21.15.60:0/101830243 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f227c053980 0x7f227c055e40 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7f228800b000 tx=0x7f2288008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:09.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.097+0000 7f2294ff9700 1 -- 172.21.15.60:0/101830243 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f22980adbb0 con 0x7f229813dff0 2022-06-29T16:07:09.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.225+0000 7f229d4fe700 1 -- 172.21.15.60:0/101830243 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f229809abd0 con 0x7f229813dff0 2022-06-29T16:07:09.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.945+0000 7f2294ff9700 1 -- 172.21.15.60:0/101830243 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v122) v1 ==== 185+0+0 (secure 0 0 0) 0x7f229809abd0 con 0x7f229813dff0 2022-06-29T16:07:09.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.949+0000 7f229d4fe700 1 -- 172.21.15.60:0/101830243 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f227c053980 msgr2=0x7f227c055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:09.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.949+0000 7f229d4fe700 1 --2- 172.21.15.60:0/101830243 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f227c053980 0x7f227c055e40 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7f228800b000 tx=0x7f2288008040).stop 2022-06-29T16:07:09.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.949+0000 7f229d4fe700 1 -- 172.21.15.60:0/101830243 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f229813dff0 msgr2=0x7f22980af2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:09.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.949+0000 7f229d4fe700 1 --2- 172.21.15.60:0/101830243 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f229813dff0 0x7f22980af2c0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 rx=0x7f22900032f0 tx=0x7f2290016000).stop 2022-06-29T16:07:09.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.949+0000 7f229d4fe700 1 -- 172.21.15.60:0/101830243 shutdown_connections 2022-06-29T16:07:09.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.949+0000 7f229d4fe700 1 --2- 172.21.15.60:0/101830243 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f227c053980 0x7f227c055e40 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:09.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.949+0000 7f229d4fe700 1 --2- 172.21.15.60:0/101830243 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f229813dff0 0x7f22980af2c0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:09.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.949+0000 7f229d4fe700 1 --2- 172.21.15.60:0/101830243 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f22980b6710 0x7f22980aedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:09.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.949+0000 7f229d4fe700 1 --2- 172.21.15.60:0/101830243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2298061620 0x7f229805f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:09.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.949+0000 7f229d4fe700 1 -- 172.21.15.60:0/101830243 >> 172.21.15.60:0/101830243 conn(0x7f22980643e0 msgr2=0x7f22980b41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:09.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.949+0000 7f229d4fe700 1 -- 172.21.15.60:0/101830243 shutdown_connections 2022-06-29T16:07:09.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:09.949+0000 7f229d4fe700 1 -- 172.21.15.60:0/101830243 wait complete. 2022-06-29T16:07:09.958 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2022-06-29T16:07:09.967 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-06-29T16:07:10.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.073+0000 7f9991773700 1 Processor -- start 2022-06-29T16:07:10.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.073+0000 7f9991773700 1 -- start start 2022-06-29T16:07:10.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.073+0000 7f9991773700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f998c0b5950 0x7f998c065250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:10.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.073+0000 7f9991773700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f998c064b50 0x7f998c13fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:10.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.073+0000 7f9991773700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f998c0b5d30 con 0x7f998c1401f0 2022-06-29T16:07:10.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.073+0000 7f9991773700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f998c066fa0 con 0x7f998c064b50 2022-06-29T16:07:10.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.073+0000 7f9991773700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f998c143960 con 0x7f998c0b5950 2022-06-29T16:07:10.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.073+0000 7f998a7fc700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f998c064b50 0x7f998c13fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:10.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.073+0000 7f998a7fc700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f998c064b50 0x7f998c13fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52626/0 (socket says 172.21.15.60:52626) 2022-06-29T16:07:10.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.073+0000 7f998a7fc700 1 -- 172.21.15.60:0/4009690761 learned_addr learned my addr 172.21.15.60:0/4009690761 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:10.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.073+0000 7f998affd700 1 --2- 172.21.15.60:0/4009690761 >> v2:172.21.15.92:3300/0 conn(0x7f998c0b5950 0x7f998c065250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:10.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.073+0000 7f998a7fc700 1 -- 172.21.15.60:0/4009690761 >> v2:172.21.15.92:3300/0 conn(0x7f998c0b5950 msgr2=0x7f998c065250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:10.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.073+0000 7f998a7fc700 1 --2- 172.21.15.60:0/4009690761 >> v2:172.21.15.92:3300/0 conn(0x7f998c0b5950 0x7f998c065250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:10.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.073+0000 7f998a7fc700 1 -- 172.21.15.60:0/4009690761 >> v1:172.21.15.92:6789/0 conn(0x7f998c1401f0 legacy=0x7f998c063810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:10.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.073+0000 7f998a7fc700 1 -- 172.21.15.60:0/4009690761 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f998c143c00 con 0x7f998c064b50 2022-06-29T16:07:10.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.073+0000 7f998a7fc700 1 --2- 172.21.15.60:0/4009690761 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f998c064b50 0x7f998c13fd00 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 rx=0x7f998c0b4e50 tx=0x7f9980006eb0).ready entity=mon.0 client_cookie=1e30c4a6aebaade5 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:10.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.073+0000 7f9989ffb700 1 -- 172.21.15.60:0/4009690761 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f998000f9c0 con 0x7f998c064b50 2022-06-29T16:07:10.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.073+0000 7f9989ffb700 1 -- 172.21.15.60:0/4009690761 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9980000e90 con 0x7f998c064b50 2022-06-29T16:07:10.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.073+0000 7f9989ffb700 1 -- 172.21.15.60:0/4009690761 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f998000fd10 con 0x7f998c064b50 2022-06-29T16:07:10.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.073+0000 7f9991773700 1 -- 172.21.15.60:0/4009690761 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f998c064b50 msgr2=0x7f998c13fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:10.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.073+0000 7f9991773700 1 --2- 172.21.15.60:0/4009690761 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f998c064b50 0x7f998c13fd00 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 rx=0x7f998c0b4e50 tx=0x7f9980006eb0).stop 2022-06-29T16:07:10.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.073+0000 7f9991773700 1 -- 172.21.15.60:0/4009690761 shutdown_connections 2022-06-29T16:07:10.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.073+0000 7f9991773700 1 --2- 172.21.15.60:0/4009690761 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f998c064b50 0x7f998c13fd00 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:10.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.073+0000 7f9991773700 1 --2- 172.21.15.60:0/4009690761 >> v2:172.21.15.92:3300/0 conn(0x7f998c0b5950 0x7f998c065250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:10.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.073+0000 7f9991773700 1 -- 172.21.15.60:0/4009690761 >> 172.21.15.60:0/4009690761 conn(0x7f998c0632c0 msgr2=0x7f998c0b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:10.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.077+0000 7f9991773700 1 -- 172.21.15.60:0/4009690761 shutdown_connections 2022-06-29T16:07:10.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.077+0000 7f9991773700 1 -- 172.21.15.60:0/4009690761 wait complete. 2022-06-29T16:07:10.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.077+0000 7f9991773700 1 Processor -- start 2022-06-29T16:07:10.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.077+0000 7f9991773700 1 -- start start 2022-06-29T16:07:10.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.077+0000 7f9991773700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f998c064b50 0x7f998c1420e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:10.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.077+0000 7f9991773700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f998c0b5950 0x7f998c1286d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:10.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.077+0000 7f9991773700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f998c1401f0 0x7f998c128bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:10.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.077+0000 7f9991773700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f998c14ea00 con 0x7f998c0b5950 2022-06-29T16:07:10.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.077+0000 7f9991773700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f998c13b6c0 con 0x7f998c064b50 2022-06-29T16:07:10.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.077+0000 7f9991773700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f998c13ae50 con 0x7f998c1401f0 2022-06-29T16:07:10.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.077+0000 7f998a7fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f998c0b5950 0x7f998c1286d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:10.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.077+0000 7f998a7fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f998c0b5950 0x7f998c1286d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52630/0 (socket says 172.21.15.60:52630) 2022-06-29T16:07:10.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.077+0000 7f998a7fc700 1 -- 172.21.15.60:0/2316309622 learned_addr learned my addr 172.21.15.60:0/2316309622 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:10.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.077+0000 7f998affd700 1 --2- 172.21.15.60:0/2316309622 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f998c064b50 0x7f998c1420e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:10.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.077+0000 7f998a7fc700 1 -- 172.21.15.60:0/2316309622 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f998c1401f0 msgr2=0x7f998c128bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:10.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.077+0000 7f998b7fe700 1 --2- 172.21.15.60:0/2316309622 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f998c1401f0 0x7f998c128bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:10.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.077+0000 7f998a7fc700 1 --2- 172.21.15.60:0/2316309622 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f998c1401f0 0x7f998c128bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:10.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.077+0000 7f998a7fc700 1 -- 172.21.15.60:0/2316309622 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f998c064b50 msgr2=0x7f998c1420e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:10.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.077+0000 7f998a7fc700 1 --2- 172.21.15.60:0/2316309622 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f998c064b50 0x7f998c1420e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:10.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.077+0000 7f998a7fc700 1 -- 172.21.15.60:0/2316309622 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f998000b1e0 con 0x7f998c0b5950 2022-06-29T16:07:10.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.077+0000 7f998affd700 1 --2- 172.21.15.60:0/2316309622 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f998c064b50 0x7f998c1420e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:07:10.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.077+0000 7f998b7fe700 1 --2- 172.21.15.60:0/2316309622 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f998c1401f0 0x7f998c128bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:07:10.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.077+0000 7f998a7fc700 1 --2- 172.21.15.60:0/2316309622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f998c0b5950 0x7f998c1286d0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 rx=0x7f998c13a4d0 tx=0x7f9980017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:10.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.077+0000 7f997bfff700 1 -- 172.21.15.60:0/2316309622 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9980020070 con 0x7f998c0b5950 2022-06-29T16:07:10.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.077+0000 7f997bfff700 1 -- 172.21.15.60:0/2316309622 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9980010e80 con 0x7f998c0b5950 2022-06-29T16:07:10.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.077+0000 7f9991773700 1 -- 172.21.15.60:0/2316309622 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f998c1290b0 con 0x7f998c0b5950 2022-06-29T16:07:10.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.077+0000 7f997bfff700 1 -- 172.21.15.60:0/2316309622 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99800197e0 con 0x7f998c0b5950 2022-06-29T16:07:10.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.077+0000 7f9991773700 1 -- 172.21.15.60:0/2316309622 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f998c13a710 con 0x7f998c0b5950 2022-06-29T16:07:10.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.081+0000 7f997bfff700 1 -- 172.21.15.60:0/2316309622 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9980019940 con 0x7f998c0b5950 2022-06-29T16:07:10.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.081+0000 7f9991773700 1 -- 172.21.15.60:0/2316309622 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f998c13f860 con 0x7f998c0b5950 2022-06-29T16:07:10.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.081+0000 7f997bfff700 1 --2- 172.21.15.60:0/2316309622 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9968053a70 0x7f9968055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:10.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.081+0000 7f997bfff700 1 -- 172.21.15.60:0/2316309622 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f998006f0a0 con 0x7f998c0b5950 2022-06-29T16:07:10.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.081+0000 7f998affd700 1 --2- 172.21.15.60:0/2316309622 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9968053a70 0x7f9968055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:10.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.081+0000 7f998affd700 1 --2- 172.21.15.60:0/2316309622 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9968053a70 0x7f9968055f30 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0x7f997c008000 tx=0x7f997c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:10.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.089+0000 7f997bfff700 1 -- 172.21.15.60:0/2316309622 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9980046010 con 0x7f998c0b5950 2022-06-29T16:07:10.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.213+0000 7f9991773700 1 -- 172.21.15.60:0/2316309622 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f998c0573d0 con 0x7f998c0b5950 2022-06-29T16:07:10.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.957+0000 7f997bfff700 1 -- 172.21.15.60:0/2316309622 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v123) v1 ==== 173+0+0 (secure 0 0 0) 0x7f9980002f50 con 0x7f998c0b5950 2022-06-29T16:07:10.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.961+0000 7f9991773700 1 -- 172.21.15.60:0/2316309622 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9968053a70 msgr2=0x7f9968055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:10.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.961+0000 7f9991773700 1 --2- 172.21.15.60:0/2316309622 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9968053a70 0x7f9968055f30 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0x7f997c008000 tx=0x7f997c008040).stop 2022-06-29T16:07:10.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.961+0000 7f9991773700 1 -- 172.21.15.60:0/2316309622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f998c0b5950 msgr2=0x7f998c1286d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:10.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.961+0000 7f9991773700 1 --2- 172.21.15.60:0/2316309622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f998c0b5950 0x7f998c1286d0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 rx=0x7f998c13a4d0 tx=0x7f9980017000).stop 2022-06-29T16:07:10.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.961+0000 7f9991773700 1 -- 172.21.15.60:0/2316309622 shutdown_connections 2022-06-29T16:07:10.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.961+0000 7f9991773700 1 --2- 172.21.15.60:0/2316309622 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9968053a70 0x7f9968055f30 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:10.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.961+0000 7f9991773700 1 --2- 172.21.15.60:0/2316309622 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f998c1401f0 0x7f998c128bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:10.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.961+0000 7f9991773700 1 --2- 172.21.15.60:0/2316309622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f998c0b5950 0x7f998c1286d0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:10.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.961+0000 7f9991773700 1 --2- 172.21.15.60:0/2316309622 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f998c064b50 0x7f998c1420e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:10.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.961+0000 7f9991773700 1 -- 172.21.15.60:0/2316309622 >> 172.21.15.60:0/2316309622 conn(0x7f998c0632c0 msgr2=0x7f998c143e00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:10.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.961+0000 7f9991773700 1 -- 172.21.15.60:0/2316309622 shutdown_connections 2022-06-29T16:07:10.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:10.961+0000 7f9991773700 1 -- 172.21.15.60:0/2316309622 wait complete. 2022-06-29T16:07:10.969 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2022-06-29T16:07:10.978 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-06-29T16:07:11.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.077+0000 7fe467fff700 1 Processor -- start 2022-06-29T16:07:11.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.077+0000 7fe467fff700 1 -- start start 2022-06-29T16:07:11.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.077+0000 7fe467fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe46805bf30 0x7fe4681428c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:11.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.077+0000 7fe467fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe468142db0 0x7fe46813fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:11.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.077+0000 7fe467fff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe468146050 con 0x7fe4680b3b60 2022-06-29T16:07:11.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.077+0000 7fe467fff700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe468146ea0 con 0x7fe468142db0 2022-06-29T16:07:11.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.077+0000 7fe467fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4681462f0 con 0x7fe46805bf30 2022-06-29T16:07:11.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.081+0000 7fe4667fc700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe46805bf30 0x7fe4681428c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:11.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.081+0000 7fe4677fe700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe468142db0 0x7fe46813fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:11.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.081+0000 7fe4667fc700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe46805bf30 0x7fe4681428c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:37502/0 (socket says 172.21.15.60:37502) 2022-06-29T16:07:11.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.081+0000 7fe4677fe700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe468142db0 0x7fe46813fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:58672/0 (socket says 172.21.15.60:58672) 2022-06-29T16:07:11.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.081+0000 7fe4667fc700 1 -- 172.21.15.60:0/1124570018 learned_addr learned my addr 172.21.15.60:0/1124570018 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:11.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.081+0000 7fe466ffd700 1 --1- 172.21.15.60:0/1124570018 >> v1:172.21.15.92:6789/0 conn(0x7fe4680b3b60 0x7fe468146790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:36740/0 (socket says 172.21.15.60:36740) 2022-06-29T16:07:11.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.081+0000 7fe4677fe700 1 -- 172.21.15.60:0/1124570018 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe46805bf30 msgr2=0x7fe4681428c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:11.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.081+0000 7fe4677fe700 1 --2- 172.21.15.60:0/1124570018 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe46805bf30 0x7fe4681428c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:11.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.081+0000 7fe4677fe700 1 -- 172.21.15.60:0/1124570018 >> v1:172.21.15.92:6789/0 conn(0x7fe4680b3b60 legacy=0x7fe468146790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:11.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.081+0000 7fe4677fe700 1 -- 172.21.15.60:0/1124570018 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4680b2320 con 0x7fe468142db0 2022-06-29T16:07:11.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.081+0000 7fe4667fc700 1 --2- 172.21.15.60:0/1124570018 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe46805bf30 0x7fe4681428c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:07:11.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.081+0000 7fe4677fe700 1 --2- 172.21.15.60:0/1124570018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe468142db0 0x7fe46813fa70 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0x7fe4600032f0 tx=0x7fe4600091d0).ready entity=mon.1 client_cookie=249af416dfb2bad9 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:11.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.081+0000 7fe465ffb700 1 -- 172.21.15.60:0/1124570018 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe46000b6c0 con 0x7fe468142db0 2022-06-29T16:07:11.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.081+0000 7fe465ffb700 1 -- 172.21.15.60:0/1124570018 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe46000b820 con 0x7fe468142db0 2022-06-29T16:07:11.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.081+0000 7fe465ffb700 1 -- 172.21.15.60:0/1124570018 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe46000b990 con 0x7fe468142db0 2022-06-29T16:07:11.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.081+0000 7fe467fff700 1 -- 172.21.15.60:0/1124570018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe468142db0 msgr2=0x7fe46813fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:11.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.081+0000 7fe467fff700 1 --2- 172.21.15.60:0/1124570018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe468142db0 0x7fe46813fa70 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0x7fe4600032f0 tx=0x7fe4600091d0).stop 2022-06-29T16:07:11.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.081+0000 7fe467fff700 1 -- 172.21.15.60:0/1124570018 shutdown_connections 2022-06-29T16:07:11.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.081+0000 7fe467fff700 1 --2- 172.21.15.60:0/1124570018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe468142db0 0x7fe46813fa70 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:11.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.081+0000 7fe467fff700 1 --2- 172.21.15.60:0/1124570018 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe46805bf30 0x7fe4681428c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:11.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.081+0000 7fe467fff700 1 -- 172.21.15.60:0/1124570018 >> 172.21.15.60:0/1124570018 conn(0x7fe468064810 msgr2=0x7fe468064bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:11.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.081+0000 7fe467fff700 1 -- 172.21.15.60:0/1124570018 shutdown_connections 2022-06-29T16:07:11.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.081+0000 7fe467fff700 1 -- 172.21.15.60:0/1124570018 wait complete. 2022-06-29T16:07:11.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.081+0000 7fe467fff700 1 Processor -- start 2022-06-29T16:07:11.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.081+0000 7fe467fff700 1 -- start start 2022-06-29T16:07:11.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.085+0000 7fe467fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe46805bf30 0x7fe4680aeb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:11.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.085+0000 7fe467fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe4680b3b60 0x7fe468151d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:11.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.085+0000 7fe467fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe468142db0 0x7fe4681411a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:11.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.085+0000 7fe467fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe468155290 con 0x7fe468142db0 2022-06-29T16:07:11.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.085+0000 7fe467fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4680621d0 con 0x7fe46805bf30 2022-06-29T16:07:11.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.085+0000 7fe467fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe468062470 con 0x7fe4680b3b60 2022-06-29T16:07:11.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.085+0000 7fe4677fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe468142db0 0x7fe4681411a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:11.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.085+0000 7fe4677fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe468142db0 0x7fe4681411a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52648/0 (socket says 172.21.15.60:52648) 2022-06-29T16:07:11.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.085+0000 7fe4677fe700 1 -- 172.21.15.60:0/660720797 learned_addr learned my addr 172.21.15.60:0/660720797 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:11.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.085+0000 7fe466ffd700 1 --2- 172.21.15.60:0/660720797 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe46805bf30 0x7fe4680aeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:11.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.085+0000 7fe4667fc700 1 --2- 172.21.15.60:0/660720797 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe4680b3b60 0x7fe468151d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:11.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.085+0000 7fe4677fe700 1 -- 172.21.15.60:0/660720797 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe4680b3b60 msgr2=0x7fe468151d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:11.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.085+0000 7fe4677fe700 1 --2- 172.21.15.60:0/660720797 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe4680b3b60 0x7fe468151d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:11.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.085+0000 7fe4677fe700 1 -- 172.21.15.60:0/660720797 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe46805bf30 msgr2=0x7fe4680aeb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:11.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.085+0000 7fe4677fe700 1 --2- 172.21.15.60:0/660720797 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe46805bf30 0x7fe4680aeb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:11.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.085+0000 7fe4677fe700 1 -- 172.21.15.60:0/660720797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe45800b1e0 con 0x7fe468142db0 2022-06-29T16:07:11.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.085+0000 7fe4677fe700 1 --2- 172.21.15.60:0/660720797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe468142db0 0x7fe4681411a0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 rx=0x7fe4600032f0 tx=0x7fe460016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:11.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.085+0000 7fe457fff700 1 -- 172.21.15.60:0/660720797 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe46000b990 con 0x7fe468142db0 2022-06-29T16:07:11.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.085+0000 7fe457fff700 1 -- 172.21.15.60:0/660720797 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe460021630 con 0x7fe468142db0 2022-06-29T16:07:11.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.085+0000 7fe467fff700 1 -- 172.21.15.60:0/660720797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe460009470 con 0x7fe468142db0 2022-06-29T16:07:11.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.085+0000 7fe457fff700 1 -- 172.21.15.60:0/660720797 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe46000b6c0 con 0x7fe468142db0 2022-06-29T16:07:11.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.085+0000 7fe467fff700 1 -- 172.21.15.60:0/660720797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe468061c50 con 0x7fe468142db0 2022-06-29T16:07:11.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.085+0000 7fe467fff700 1 -- 172.21.15.60:0/660720797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe468062040 con 0x7fe468142db0 2022-06-29T16:07:11.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.089+0000 7fe457fff700 1 -- 172.21.15.60:0/660720797 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe460027020 con 0x7fe468142db0 2022-06-29T16:07:11.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.089+0000 7fe457fff700 1 --2- 172.21.15.60:0/660720797 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe4440539e0 0x7fe444055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:11.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.089+0000 7fe457fff700 1 -- 172.21.15.60:0/660720797 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fe46006cfe0 con 0x7fe468142db0 2022-06-29T16:07:11.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.089+0000 7fe466ffd700 1 --2- 172.21.15.60:0/660720797 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe4440539e0 0x7fe444055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:11.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.089+0000 7fe466ffd700 1 --2- 172.21.15.60:0/660720797 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe4440539e0 0x7fe444055ea0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0x7fe45c00d000 tx=0x7fe45c007b40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:11.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.097+0000 7fe457fff700 1 -- 172.21.15.60:0/660720797 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe460044110 con 0x7fe468142db0 2022-06-29T16:07:11.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.225+0000 7fe467fff700 1 -- 172.21.15.60:0/660720797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fe4681553d0 con 0x7fe468142db0 2022-06-29T16:07:11.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.233+0000 7fe457fff700 1 -- 172.21.15.60:0/660720797 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe460047730 con 0x7fe468142db0 2022-06-29T16:07:11.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.233+0000 7fe467fff700 1 -- 172.21.15.60:0/660720797 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe4440539e0 msgr2=0x7fe444055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:11.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.233+0000 7fe467fff700 1 --2- 172.21.15.60:0/660720797 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe4440539e0 0x7fe444055ea0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0x7fe45c00d000 tx=0x7fe45c007b40).stop 2022-06-29T16:07:11.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.233+0000 7fe467fff700 1 -- 172.21.15.60:0/660720797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe468142db0 msgr2=0x7fe4681411a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:11.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.233+0000 7fe467fff700 1 --2- 172.21.15.60:0/660720797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe468142db0 0x7fe4681411a0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 rx=0x7fe4600032f0 tx=0x7fe460016000).stop 2022-06-29T16:07:11.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.233+0000 7fe467fff700 1 -- 172.21.15.60:0/660720797 shutdown_connections 2022-06-29T16:07:11.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.233+0000 7fe467fff700 1 --2- 172.21.15.60:0/660720797 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe4440539e0 0x7fe444055ea0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:11.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.233+0000 7fe467fff700 1 --2- 172.21.15.60:0/660720797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe468142db0 0x7fe4681411a0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:11.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.233+0000 7fe467fff700 1 --2- 172.21.15.60:0/660720797 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe4680b3b60 0x7fe468151d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:11.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.233+0000 7fe467fff700 1 --2- 172.21.15.60:0/660720797 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe46805bf30 0x7fe4680aeb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:11.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.233+0000 7fe467fff700 1 -- 172.21.15.60:0/660720797 >> 172.21.15.60:0/660720797 conn(0x7fe468064810 msgr2=0x7fe4681422f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:11.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.233+0000 7fe467fff700 1 -- 172.21.15.60:0/660720797 shutdown_connections 2022-06-29T16:07:11.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.233+0000 7fe467fff700 1 -- 172.21.15.60:0/660720797 wait complete. 2022-06-29T16:07:11.253 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_fs_new' 2022-06-29T16:07:11.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.353+0000 7f080ac3d700 1 Processor -- start 2022-06-29T16:07:11.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.353+0000 7f080ac3d700 1 -- start start 2022-06-29T16:07:11.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.353+0000 7f080ac3d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08041471a0 0x7f0804147fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:11.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.353+0000 7f080ac3d700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f08040afb10 0x7f08040b0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:11.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.353+0000 7f080ac3d700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08040ac8f0 con 0x7f08040aed60 2022-06-29T16:07:11.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.353+0000 7f080ac3d700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08040acb90 con 0x7f08040afb10 2022-06-29T16:07:11.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.353+0000 7f080ac3d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08040b02d0 con 0x7f08041471a0 2022-06-29T16:07:11.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.353+0000 7f08091da700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f08040afb10 0x7f08040b0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:11.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.353+0000 7f0803fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08041471a0 0x7f0804147fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:11.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.353+0000 7f08089d9700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f08040aed60 0x7f0804147890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:36754/0 (socket says 172.21.15.60:36754) 2022-06-29T16:07:11.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.353+0000 7f08091da700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f08040afb10 0x7f08040b0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:58684/0 (socket says 172.21.15.60:58684) 2022-06-29T16:07:11.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.353+0000 7f08089d9700 1 -- 172.21.15.60:0/1498338548 learned_addr learned my addr 172.21.15.60:0/1498338548 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:11.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.353+0000 7f08091da700 1 -- 172.21.15.60:0/1498338548 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08041471a0 msgr2=0x7f0804147fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:11.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.353+0000 7f08091da700 1 --2- 172.21.15.60:0/1498338548 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08041471a0 0x7f0804147fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:11.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.353+0000 7f08091da700 1 -- 172.21.15.60:0/1498338548 >> v1:172.21.15.92:6789/0 conn(0x7f08040aed60 legacy=0x7f0804147890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:11.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.353+0000 7f08091da700 1 -- 172.21.15.60:0/1498338548 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08040ac5b0 con 0x7f08040afb10 2022-06-29T16:07:11.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.353+0000 7f08091da700 1 --2- 172.21.15.60:0/1498338548 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08040afb10 0x7f08040b0770 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0x7f07f4002f70 tx=0x7f07f4006df0).ready entity=mon.1 client_cookie=60fddf37f29d9846 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:11.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.353+0000 7f08037fe700 1 -- 172.21.15.60:0/1498338548 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07f400d570 con 0x7f08040afb10 2022-06-29T16:07:11.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.353+0000 7f08037fe700 1 -- 172.21.15.60:0/1498338548 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07f4000e90 con 0x7f08040afb10 2022-06-29T16:07:11.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.353+0000 7f08037fe700 1 -- 172.21.15.60:0/1498338548 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07f400d8c0 con 0x7f08040afb10 2022-06-29T16:07:11.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.357+0000 7f080ac3d700 1 -- 172.21.15.60:0/1498338548 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08040afb10 msgr2=0x7f08040b0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:11.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.357+0000 7f080ac3d700 1 --2- 172.21.15.60:0/1498338548 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08040afb10 0x7f08040b0770 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0x7f07f4002f70 tx=0x7f07f4006df0).stop 2022-06-29T16:07:11.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.357+0000 7f080ac3d700 1 -- 172.21.15.60:0/1498338548 shutdown_connections 2022-06-29T16:07:11.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.357+0000 7f080ac3d700 1 --2- 172.21.15.60:0/1498338548 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08040afb10 0x7f08040b0770 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:11.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.357+0000 7f080ac3d700 1 --2- 172.21.15.60:0/1498338548 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08041471a0 0x7f0804147fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:11.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.357+0000 7f080ac3d700 1 -- 172.21.15.60:0/1498338548 >> 172.21.15.60:0/1498338548 conn(0x7f08040b4f60 msgr2=0x7f08040b5370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:11.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.357+0000 7f080ac3d700 1 -- 172.21.15.60:0/1498338548 shutdown_connections 2022-06-29T16:07:11.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.357+0000 7f080ac3d700 1 -- 172.21.15.60:0/1498338548 wait complete. 2022-06-29T16:07:11.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.357+0000 7f080ac3d700 1 Processor -- start 2022-06-29T16:07:11.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.357+0000 7f080ac3d700 1 -- start start 2022-06-29T16:07:11.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.357+0000 7f080ac3d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08040aed60 0x7f08040b0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:11.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.357+0000 7f080ac3d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08040afb10 0x7f0804066920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:11.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.357+0000 7f080ac3d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08041431c0 0x7f0804066e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:11.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.357+0000 7f080ac3d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080414c580 con 0x7f08040afb10 2022-06-29T16:07:11.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.357+0000 7f080ac3d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08040655c0 con 0x7f08041431c0 2022-06-29T16:07:11.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.357+0000 7f080ac3d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0804064770 con 0x7f08040aed60 2022-06-29T16:07:11.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.357+0000 7f0803fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08040afb10 0x7f0804066920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:11.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.357+0000 7f08089d9700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08040aed60 0x7f08040b0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:11.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.357+0000 7f0803fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08040afb10 0x7f0804066920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52660/0 (socket says 172.21.15.60:52660) 2022-06-29T16:07:11.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.357+0000 7f0803fff700 1 -- 172.21.15.60:0/3881721900 learned_addr learned my addr 172.21.15.60:0/3881721900 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:11.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.357+0000 7f08091da700 1 --2- 172.21.15.60:0/3881721900 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08041431c0 0x7f0804066e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:11.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.357+0000 7f0803fff700 1 -- 172.21.15.60:0/3881721900 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08040aed60 msgr2=0x7f08040b0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:11.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.357+0000 7f0803fff700 1 --2- 172.21.15.60:0/3881721900 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08040aed60 0x7f08040b0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:11.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.357+0000 7f0803fff700 1 -- 172.21.15.60:0/3881721900 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08041431c0 msgr2=0x7f0804066e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:11.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.357+0000 7f0803fff700 1 --2- 172.21.15.60:0/3881721900 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08041431c0 0x7f0804066e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:11.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.361+0000 7f0803fff700 1 -- 172.21.15.60:0/3881721900 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f400b180 con 0x7f08040afb10 2022-06-29T16:07:11.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.361+0000 7f0803fff700 1 --2- 172.21.15.60:0/3881721900 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08040afb10 0x7f0804066920 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 rx=0x7f07f800c000 tx=0x7f07f8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:11.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.361+0000 7f0801ffb700 1 -- 172.21.15.60:0/3881721900 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07f8011750 con 0x7f08040afb10 2022-06-29T16:07:11.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.361+0000 7f0801ffb700 1 -- 172.21.15.60:0/3881721900 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07f80118b0 con 0x7f08040afb10 2022-06-29T16:07:11.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.361+0000 7f080ac3d700 1 -- 172.21.15.60:0/3881721900 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f080414be40 con 0x7f08040afb10 2022-06-29T16:07:11.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.361+0000 7f0801ffb700 1 -- 172.21.15.60:0/3881721900 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07f8011c10 con 0x7f08040afb10 2022-06-29T16:07:11.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.361+0000 7f080ac3d700 1 -- 172.21.15.60:0/3881721900 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08040b0d20 con 0x7f08040afb10 2022-06-29T16:07:11.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.361+0000 7f0801ffb700 1 -- 172.21.15.60:0/3881721900 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f07f8011d70 con 0x7f08040afb10 2022-06-29T16:07:11.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.361+0000 7f080ac3d700 1 -- 172.21.15.60:0/3881721900 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08040573d0 con 0x7f08040afb10 2022-06-29T16:07:11.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.361+0000 7f0801ffb700 1 --2- 172.21.15.60:0/3881721900 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f07e4053a70 0x7f07e4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:11.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.361+0000 7f0801ffb700 1 -- 172.21.15.60:0/3881721900 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f07f806c0a0 con 0x7f08040afb10 2022-06-29T16:07:11.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.361+0000 7f08089d9700 1 --2- 172.21.15.60:0/3881721900 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f07e4053a70 0x7f07e4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:11.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.365+0000 7f08089d9700 1 --2- 172.21.15.60:0/3881721900 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f07e4053a70 0x7f07e4055f30 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0x7f07fc00d000 tx=0x7f07fc009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:11.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.369+0000 7f0801ffb700 1 -- 172.21.15.60:0/3881721900 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f07f80431a0 con 0x7f08040afb10 2022-06-29T16:07:11.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.537+0000 7f080ac3d700 1 -- 172.21.15.60:0/3881721900 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_fs_new"]} v 0) v1 -- 0x7f08040b5ad0 con 0x7f08040afb10 2022-06-29T16:07:11.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.965+0000 7f0801ffb700 1 -- 172.21.15.60:0/3881721900 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_fs_new"]}]=0 v401) v1 ==== 122+0+0 (secure 0 0 0) 0x7f07f80467c0 con 0x7f08040afb10 2022-06-29T16:07:11.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.969+0000 7f080ac3d700 1 -- 172.21.15.60:0/3881721900 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f07e4053a70 msgr2=0x7f07e4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:11.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.969+0000 7f080ac3d700 1 --2- 172.21.15.60:0/3881721900 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f07e4053a70 0x7f07e4055f30 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0x7f07fc00d000 tx=0x7f07fc009040).stop 2022-06-29T16:07:11.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.969+0000 7f080ac3d700 1 -- 172.21.15.60:0/3881721900 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08040afb10 msgr2=0x7f0804066920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:11.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.969+0000 7f080ac3d700 1 --2- 172.21.15.60:0/3881721900 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08040afb10 0x7f0804066920 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 rx=0x7f07f800c000 tx=0x7f07f8006df0).stop 2022-06-29T16:07:11.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.969+0000 7f080ac3d700 1 -- 172.21.15.60:0/3881721900 shutdown_connections 2022-06-29T16:07:11.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.969+0000 7f080ac3d700 1 --2- 172.21.15.60:0/3881721900 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f07e4053a70 0x7f07e4055f30 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:11.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.969+0000 7f080ac3d700 1 --2- 172.21.15.60:0/3881721900 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08041431c0 0x7f0804066e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:11.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.969+0000 7f080ac3d700 1 --2- 172.21.15.60:0/3881721900 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f08040afb10 0x7f0804066920 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:11.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.969+0000 7f080ac3d700 1 --2- 172.21.15.60:0/3881721900 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f08040aed60 0x7f08040b0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:11.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.969+0000 7f080ac3d700 1 -- 172.21.15.60:0/3881721900 >> 172.21.15.60:0/3881721900 conn(0x7f08040b4f60 msgr2=0x7f08040b4590 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:11.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.969+0000 7f080ac3d700 1 -- 172.21.15.60:0/3881721900 shutdown_connections 2022-06-29T16:07:11.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:11.969+0000 7f080ac3d700 1 -- 172.21.15.60:0/3881721900 wait complete. 2022-06-29T16:07:11.992 INFO:tasks.cephfs_test_runner:test_fs_new (tasks.cephfs.test_misc.TestMisc) ... ok 2022-06-29T16:07:11.993 INFO:tasks.cephfs_test_runner:Starting test: test_getattr_caps (tasks.cephfs.test_misc.TestMisc) 2022-06-29T16:07:11.994 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_getattr_caps' 2022-06-29T16:07:12.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.097+0000 7fac04d81700 1 Processor -- start 2022-06-29T16:07:12.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.097+0000 7fac04d81700 1 -- start start 2022-06-29T16:07:12.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.097+0000 7fac04d81700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fac000b5eb0 0x7fac00142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:12.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.097+0000 7fac04d81700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac000b2c50 0x7fac000b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:12.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.097+0000 7fac04d81700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac000b0cf0 con 0x7fac000b48b0 2022-06-29T16:07:12.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.097+0000 7fac04d81700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac000b54e0 con 0x7fac000b5eb0 2022-06-29T16:07:12.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.097+0000 7fac04d81700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac000b4d10 con 0x7fac000b2c50 2022-06-29T16:07:12.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fabfeffd700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fac000b48b0 0x7fac000b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60752/0 (socket says 172.21.15.60:60752) 2022-06-29T16:07:12.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fabfeffd700 1 -- 172.21.15.60:0/409488443 learned_addr learned my addr 172.21.15.60:0/409488443 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:12.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fabfffff700 1 --2- 172.21.15.60:0/409488443 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac000b2c50 0x7fac000b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:12.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fabff7fe700 1 --2- 172.21.15.60:0/409488443 >> v2:172.21.15.92:3300/0 conn(0x7fac000b5eb0 0x7fac00142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:12.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fabfe7fc700 1 -- 172.21.15.60:0/409488443 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 94234850 0 0) 0x7fabe8001490 con 0x7fac000b48b0 2022-06-29T16:07:12.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fabfe7fc700 1 -- 172.21.15.60:0/409488443 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe0003260 con 0x7fac000b48b0 2022-06-29T16:07:12.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fabfe7fc700 1 -- 172.21.15.60:0/409488443 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3605692456 0 0) 0x7fabe0003260 con 0x7fac000b48b0 2022-06-29T16:07:12.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fabfe7fc700 1 -- 172.21.15.60:0/409488443 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac000b2c50 msgr2=0x7fac000b30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:12.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fabfe7fc700 1 --2- 172.21.15.60:0/409488443 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac000b2c50 0x7fac000b30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:12.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fabfe7fc700 1 -- 172.21.15.60:0/409488443 >> v2:172.21.15.92:3300/0 conn(0x7fac000b5eb0 msgr2=0x7fac00142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:12.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fabfe7fc700 1 --2- 172.21.15.60:0/409488443 >> v2:172.21.15.92:3300/0 conn(0x7fac000b5eb0 0x7fac00142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:12.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fabfe7fc700 1 -- 172.21.15.60:0/409488443 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac000b6290 con 0x7fac000b48b0 2022-06-29T16:07:12.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fabfe7fc700 1 -- 172.21.15.60:0/409488443 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7fabe8001b70 con 0x7fac000b48b0 2022-06-29T16:07:12.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fabfe7fc700 1 -- 172.21.15.60:0/409488443 >> v1:172.21.15.60:6789/0 conn(0x7fac000b48b0 legacy=0x7fac000b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:12.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fabfe7fc700 1 --2- 172.21.15.60:0/409488443 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fabe0003750 0x7fabe0003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:12.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fabfe7fc700 1 --2- 172.21.15.60:0/409488443 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabe0004500 0x7fabe0004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:12.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fabfe7fc700 1 --2- 172.21.15.60:0/409488443 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac000b2c50 0x7fabe0004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:12.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fabfe7fc700 1 -- 172.21.15.60:0/409488443 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac000b54e0 con 0x7fabe0003750 2022-06-29T16:07:12.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fabfe7fc700 1 -- 172.21.15.60:0/409488443 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac000b4d10 con 0x7fabe0004500 2022-06-29T16:07:12.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fabfe7fc700 1 -- 172.21.15.60:0/409488443 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabe000b750 con 0x7fac000b2c50 2022-06-29T16:07:12.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fabfeffd700 1 --2- 172.21.15.60:0/409488443 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabe0004500 0x7fabe0004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:12.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fabff7fe700 1 --2- 172.21.15.60:0/409488443 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fabe0003750 0x7fabe0003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:12.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fabfffff700 1 --2- 172.21.15.60:0/409488443 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac000b2c50 0x7fabe0004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:12.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fabfeffd700 1 -- 172.21.15.60:0/409488443 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac000b2c50 msgr2=0x7fabe0004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:12.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fabfeffd700 1 --2- 172.21.15.60:0/409488443 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac000b2c50 0x7fabe0004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:12.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fabfeffd700 1 -- 172.21.15.60:0/409488443 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fabe0003750 msgr2=0x7fabe0003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:12.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fabfeffd700 1 --2- 172.21.15.60:0/409488443 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fabe0003750 0x7fabe0003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:12.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fabfeffd700 1 -- 172.21.15.60:0/409488443 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe0003260 con 0x7fabe0004500 2022-06-29T16:07:12.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fabfeffd700 1 --2- 172.21.15.60:0/409488443 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabe0004500 0x7fabe0004980 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7fabe8001880 tx=0x7fac000b1780).ready entity=mon.1 client_cookie=20614448220f3953 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:12.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fabfe7fc700 1 -- 172.21.15.60:0/409488443 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabe800cdd0 con 0x7fabe0004500 2022-06-29T16:07:12.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fabfe7fc700 1 -- 172.21.15.60:0/409488443 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe8005990 con 0x7fabe0004500 2022-06-29T16:07:12.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fabfe7fc700 1 -- 172.21.15.60:0/409488443 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabe8005cf0 con 0x7fabe0004500 2022-06-29T16:07:12.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fac04d81700 1 -- 172.21.15.60:0/409488443 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabe0004500 msgr2=0x7fabe0004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:12.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fac04d81700 1 --2- 172.21.15.60:0/409488443 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabe0004500 0x7fabe0004980 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7fabe8001880 tx=0x7fac000b1780).stop 2022-06-29T16:07:12.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fabfffff700 1 -- 172.21.15.60:0/409488443 reap_dead start 2022-06-29T16:07:12.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fac04d81700 1 -- 172.21.15.60:0/409488443 shutdown_connections 2022-06-29T16:07:12.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fac04d81700 1 -- 172.21.15.60:0/409488443 >> 172.21.15.60:0/409488443 conn(0x7fac00062090 msgr2=0x7fac00062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:12.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fac04d81700 1 -- 172.21.15.60:0/409488443 shutdown_connections 2022-06-29T16:07:12.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.101+0000 7fac04d81700 1 -- 172.21.15.60:0/409488443 wait complete. 2022-06-29T16:07:12.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.105+0000 7fac04d81700 1 Processor -- start 2022-06-29T16:07:12.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.105+0000 7fac04d81700 1 -- start start 2022-06-29T16:07:12.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.105+0000 7fac04d81700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabe0004500 0x7fac000af340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:12.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.105+0000 7fac04d81700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac0013d7a0 0x7fac0013c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:12.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.105+0000 7fac04d81700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac0013f4f0 0x7fac0013cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:12.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.105+0000 7fac04d81700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac000ad0a0 con 0x7fac0013f4f0 2022-06-29T16:07:12.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.105+0000 7fac04d81700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac000ad340 con 0x7fabe0004500 2022-06-29T16:07:12.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.105+0000 7fac04d81700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac000ad5e0 con 0x7fac0013d7a0 2022-06-29T16:07:12.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.105+0000 7fabfffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac0013f4f0 0x7fac0013cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:12.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.105+0000 7fabfffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac0013f4f0 0x7fac0013cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52680/0 (socket says 172.21.15.60:52680) 2022-06-29T16:07:12.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.105+0000 7fabfffff700 1 -- 172.21.15.60:0/2903621835 learned_addr learned my addr 172.21.15.60:0/2903621835 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:12.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.105+0000 7fabfeffd700 1 --2- 172.21.15.60:0/2903621835 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac0013d7a0 0x7fac0013c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:12.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.105+0000 7fabff7fe700 1 --2- 172.21.15.60:0/2903621835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabe0004500 0x7fac000af340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:12.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.109+0000 7fabfffff700 1 -- 172.21.15.60:0/2903621835 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac0013d7a0 msgr2=0x7fac0013c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:12.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.109+0000 7fabfffff700 1 --2- 172.21.15.60:0/2903621835 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac0013d7a0 0x7fac0013c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:12.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.109+0000 7fabfffff700 1 -- 172.21.15.60:0/2903621835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabe0004500 msgr2=0x7fac000af340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:12.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.109+0000 7fabfffff700 1 --2- 172.21.15.60:0/2903621835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabe0004500 0x7fac000af340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:12.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.109+0000 7fabfffff700 1 -- 172.21.15.60:0/2903621835 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0013faf0 con 0x7fac0013f4f0 2022-06-29T16:07:12.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.109+0000 7fabfffff700 1 --2- 172.21.15.60:0/2903621835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac0013f4f0 0x7fac0013cd90 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 rx=0x7fabf800b000 tx=0x7fabf8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:12.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.109+0000 7fabfcff9700 1 -- 172.21.15.60:0/2903621835 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabf8019070 con 0x7fac0013f4f0 2022-06-29T16:07:12.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.109+0000 7fac04d81700 1 -- 172.21.15.60:0/2903621835 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac000ada80 con 0x7fac0013f4f0 2022-06-29T16:07:12.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.109+0000 7fabfcff9700 1 -- 172.21.15.60:0/2903621835 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf8000e90 con 0x7fac0013f4f0 2022-06-29T16:07:12.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.109+0000 7fac04d81700 1 -- 172.21.15.60:0/2903621835 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac00151c00 con 0x7fac0013f4f0 2022-06-29T16:07:12.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.109+0000 7fabfcff9700 1 -- 172.21.15.60:0/2903621835 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabf8014840 con 0x7fac0013f4f0 2022-06-29T16:07:12.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.109+0000 7fac04d81700 1 -- 172.21.15.60:0/2903621835 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac000573d0 con 0x7fac0013f4f0 2022-06-29T16:07:12.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.109+0000 7fabfcff9700 1 -- 172.21.15.60:0/2903621835 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fabf80149a0 con 0x7fac0013f4f0 2022-06-29T16:07:12.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.109+0000 7fabfcff9700 1 --2- 172.21.15.60:0/2903621835 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fabe4053980 0x7fabe4055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:12.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.109+0000 7fabfcff9700 1 -- 172.21.15.60:0/2903621835 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fabf806b750 con 0x7fac0013f4f0 2022-06-29T16:07:12.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.109+0000 7fabff7fe700 1 --2- 172.21.15.60:0/2903621835 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fabe4053980 0x7fabe4055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:12.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.113+0000 7fabff7fe700 1 --2- 172.21.15.60:0/2903621835 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fabe4053980 0x7fabe4055e40 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0x7fabf000b000 tx=0x7fabf0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:12.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.117+0000 7fabfcff9700 1 -- 172.21.15.60:0/2903621835 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fabf80428b0 con 0x7fac0013f4f0 2022-06-29T16:07:12.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.285+0000 7fac04d81700 1 -- 172.21.15.60:0/2903621835 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_getattr_caps"]} v 0) v1 -- 0x7fac000ad7a0 con 0x7fac0013f4f0 2022-06-29T16:07:12.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.969+0000 7fabfcff9700 1 -- 172.21.15.60:0/2903621835 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_getattr_caps"]}]=0 v402) v1 ==== 131+0+0 (secure 0 0 0) 0x7fabf8045ed0 con 0x7fac0013f4f0 2022-06-29T16:07:12.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.973+0000 7fac04d81700 1 -- 172.21.15.60:0/2903621835 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fabe4053980 msgr2=0x7fabe4055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:12.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.973+0000 7fac04d81700 1 --2- 172.21.15.60:0/2903621835 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fabe4053980 0x7fabe4055e40 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0x7fabf000b000 tx=0x7fabf0008040).stop 2022-06-29T16:07:12.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.973+0000 7fac04d81700 1 -- 172.21.15.60:0/2903621835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac0013f4f0 msgr2=0x7fac0013cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:12.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.973+0000 7fac04d81700 1 --2- 172.21.15.60:0/2903621835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac0013f4f0 0x7fac0013cd90 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 rx=0x7fabf800b000 tx=0x7fabf8006df0).stop 2022-06-29T16:07:12.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.973+0000 7fac04d81700 1 -- 172.21.15.60:0/2903621835 shutdown_connections 2022-06-29T16:07:12.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.973+0000 7fac04d81700 1 --2- 172.21.15.60:0/2903621835 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fabe4053980 0x7fabe4055e40 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:12.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.973+0000 7fac04d81700 1 --2- 172.21.15.60:0/2903621835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fac0013f4f0 0x7fac0013cd90 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:12.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.973+0000 7fac04d81700 1 --2- 172.21.15.60:0/2903621835 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fac0013d7a0 0x7fac0013c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:12.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.973+0000 7fac04d81700 1 --2- 172.21.15.60:0/2903621835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabe0004500 0x7fac000af340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:12.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.973+0000 7fac04d81700 1 -- 172.21.15.60:0/2903621835 >> 172.21.15.60:0/2903621835 conn(0x7fac00062090 msgr2=0x7fac000b0280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:12.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.973+0000 7fac04d81700 1 -- 172.21.15.60:0/2903621835 shutdown_connections 2022-06-29T16:07:12.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:12.973+0000 7fac04d81700 1 -- 172.21.15.60:0/2903621835 wait complete. 2022-06-29T16:07:12.995 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-06-29T16:07:13.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.109+0000 7f432172d700 1 Processor -- start 2022-06-29T16:07:13.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.109+0000 7f432172d700 1 -- start start 2022-06-29T16:07:13.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.113+0000 7f432172d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f431c0ae7c0 0x7f431c141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:13.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.113+0000 7f432172d700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f431c143a40 con 0x7f431c0b46c0 2022-06-29T16:07:13.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.113+0000 7f432172d700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f431c0aee60 con 0x7f431c0b4010 2022-06-29T16:07:13.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.113+0000 7f432172d700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f431c0af100 con 0x7f431c0ae7c0 2022-06-29T16:07:13.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.113+0000 7f431affd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f431c0ae7c0 0x7f431c141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:13.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.113+0000 7f431b7fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f431c0b46c0 0x7f431c0b4af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60772/0 (socket says 172.21.15.60:60772) 2022-06-29T16:07:13.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.113+0000 7f431b7fe700 1 -- 172.21.15.60:0/3457765910 learned_addr learned my addr 172.21.15.60:0/3457765910 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:13.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.113+0000 7f4319ffb700 1 -- 172.21.15.60:0/3457765910 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3970100648 0 0) 0x7f431c143a40 con 0x7f431c0b46c0 2022-06-29T16:07:13.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.113+0000 7f4319ffb700 1 -- 172.21.15.60:0/3457765910 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42fc003260 con 0x7f431c0b46c0 2022-06-29T16:07:13.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.113+0000 7f431affd700 1 -- 172.21.15.60:0/3457765910 >> v1:172.21.15.92:6789/0 conn(0x7f431c0b4010 legacy=0x7f431c142490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:13.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.113+0000 7f431affd700 1 -- 172.21.15.60:0/3457765910 >> v1:172.21.15.60:6789/0 conn(0x7f431c0b46c0 legacy=0x7f431c0b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:13.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.113+0000 7f431affd700 1 -- 172.21.15.60:0/3457765910 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431c0aeba0 con 0x7f431c0ae7c0 2022-06-29T16:07:13.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.113+0000 7f431affd700 1 --2- 172.21.15.60:0/3457765910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f431c0ae7c0 0x7f431c141fa0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 rx=0x7f430c002f70 tx=0x7f430c006df0).ready entity=mon.0 client_cookie=8ac53a960a000b3f server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:13.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.113+0000 7f4319ffb700 1 -- 172.21.15.60:0/3457765910 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f430c00d610 con 0x7f431c0ae7c0 2022-06-29T16:07:13.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.113+0000 7f4319ffb700 1 -- 172.21.15.60:0/3457765910 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f430c000e90 con 0x7f431c0ae7c0 2022-06-29T16:07:13.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.113+0000 7f4319ffb700 1 -- 172.21.15.60:0/3457765910 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f430c00d960 con 0x7f431c0ae7c0 2022-06-29T16:07:13.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.113+0000 7f432172d700 1 -- 172.21.15.60:0/3457765910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f431c0ae7c0 msgr2=0x7f431c141fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:13.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.113+0000 7f432172d700 1 --2- 172.21.15.60:0/3457765910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f431c0ae7c0 0x7f431c141fa0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 rx=0x7f430c002f70 tx=0x7f430c006df0).stop 2022-06-29T16:07:13.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.113+0000 7f432172d700 1 -- 172.21.15.60:0/3457765910 shutdown_connections 2022-06-29T16:07:13.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.113+0000 7f432172d700 1 --2- 172.21.15.60:0/3457765910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f431c0ae7c0 0x7f431c141fa0 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:13.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.113+0000 7f432172d700 1 -- 172.21.15.60:0/3457765910 >> 172.21.15.60:0/3457765910 conn(0x7f431c0b9340 msgr2=0x7f431c0b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:13.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.113+0000 7f432172d700 1 -- 172.21.15.60:0/3457765910 shutdown_connections 2022-06-29T16:07:13.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.113+0000 7f432172d700 1 -- 172.21.15.60:0/3457765910 wait complete. 2022-06-29T16:07:13.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.117+0000 7f432172d700 1 Processor -- start 2022-06-29T16:07:13.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.117+0000 7f432172d700 1 -- start start 2022-06-29T16:07:13.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.117+0000 7f432172d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f431c0ae7c0 0x7f431c0b2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:13.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.117+0000 7f432172d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f431c0b4010 0x7f431c112530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:13.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.117+0000 7f432172d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f431c0b46c0 0x7f431c0b0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:13.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.117+0000 7f432172d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431c0b2a70 con 0x7f431c0ae7c0 2022-06-29T16:07:13.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.117+0000 7f432172d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431c0b1af0 con 0x7f431c0b4010 2022-06-29T16:07:13.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.117+0000 7f432172d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f431c152b00 con 0x7f431c0b46c0 2022-06-29T16:07:13.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.117+0000 7f431affd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f431c0ae7c0 0x7f431c0b2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:13.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.117+0000 7f431affd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f431c0ae7c0 0x7f431c0b2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52694/0 (socket says 172.21.15.60:52694) 2022-06-29T16:07:13.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.117+0000 7f431affd700 1 -- 172.21.15.60:0/357496839 learned_addr learned my addr 172.21.15.60:0/357496839 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:13.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.117+0000 7f431b7fe700 1 --2- 172.21.15.60:0/357496839 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f431c0b46c0 0x7f431c0b0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:13.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.117+0000 7f431a7fc700 1 --2- 172.21.15.60:0/357496839 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f431c0b4010 0x7f431c112530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:13.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.117+0000 7f431affd700 1 -- 172.21.15.60:0/357496839 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f431c0b46c0 msgr2=0x7f431c0b0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:13.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.117+0000 7f431affd700 1 --2- 172.21.15.60:0/357496839 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f431c0b46c0 0x7f431c0b0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:13.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.117+0000 7f431affd700 1 -- 172.21.15.60:0/357496839 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f431c0b4010 msgr2=0x7f431c112530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:13.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.117+0000 7f431affd700 1 --2- 172.21.15.60:0/357496839 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f431c0b4010 0x7f431c112530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:13.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.117+0000 7f431affd700 1 -- 172.21.15.60:0/357496839 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f430c00b180 con 0x7f431c0ae7c0 2022-06-29T16:07:13.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.117+0000 7f431b7fe700 1 --2- 172.21.15.60:0/357496839 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f431c0b46c0 0x7f431c0b0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:13.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.117+0000 7f431a7fc700 1 --2- 172.21.15.60:0/357496839 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f431c0b4010 0x7f431c112530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:13.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.117+0000 7f431affd700 1 --2- 172.21.15.60:0/357496839 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f431c0ae7c0 0x7f431c0b2ea0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 rx=0x7f430c002f70 tx=0x7f430c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:13.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.121+0000 7f4313fff700 1 -- 172.21.15.60:0/357496839 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f430c00d290 con 0x7f431c0ae7c0 2022-06-29T16:07:13.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.121+0000 7f432172d700 1 -- 172.21.15.60:0/357496839 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f431c152da0 con 0x7f431c0ae7c0 2022-06-29T16:07:13.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.121+0000 7f432172d700 1 -- 172.21.15.60:0/357496839 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f431c112c40 con 0x7f431c0ae7c0 2022-06-29T16:07:13.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.121+0000 7f4313fff700 1 -- 172.21.15.60:0/357496839 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f430c01e400 con 0x7f431c0ae7c0 2022-06-29T16:07:13.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.121+0000 7f4313fff700 1 -- 172.21.15.60:0/357496839 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f430c01e760 con 0x7f431c0ae7c0 2022-06-29T16:07:13.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.121+0000 7f432172d700 1 -- 172.21.15.60:0/357496839 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42f4003410 con 0x7f431c0ae7c0 2022-06-29T16:07:13.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.121+0000 7f4313fff700 1 -- 172.21.15.60:0/357496839 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f430c025030 con 0x7f431c0ae7c0 2022-06-29T16:07:13.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.121+0000 7f4313fff700 1 --2- 172.21.15.60:0/357496839 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4300053a70 0x7f4300055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:13.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.121+0000 7f4313fff700 1 -- 172.21.15.60:0/357496839 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f430c06ca90 con 0x7f431c0ae7c0 2022-06-29T16:07:13.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.121+0000 7f431a7fc700 1 --2- 172.21.15.60:0/357496839 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4300053a70 0x7f4300055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:13.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.125+0000 7f431a7fc700 1 --2- 172.21.15.60:0/357496839 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4300053a70 0x7f4300055f30 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7f430400b000 tx=0x7f4304009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:13.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.125+0000 7f4313fff700 1 -- 172.21.15.60:0/357496839 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f430c043bf0 con 0x7f431c0ae7c0 2022-06-29T16:07:13.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.253+0000 7f432172d700 1 -- 172.21.15.60:0/357496839 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f42f4003c40 con 0x7f431c0ae7c0 2022-06-29T16:07:13.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.261+0000 7f4313fff700 1 -- 172.21.15.60:0/357496839 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7f430c023070 con 0x7f431c0ae7c0 2022-06-29T16:07:13.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.261+0000 7f432172d700 1 -- 172.21.15.60:0/357496839 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4300053a70 msgr2=0x7f4300055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:13.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.261+0000 7f432172d700 1 --2- 172.21.15.60:0/357496839 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4300053a70 0x7f4300055f30 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7f430400b000 tx=0x7f4304009040).stop 2022-06-29T16:07:13.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.261+0000 7f432172d700 1 -- 172.21.15.60:0/357496839 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f431c0ae7c0 msgr2=0x7f431c0b2ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:13.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.261+0000 7f432172d700 1 --2- 172.21.15.60:0/357496839 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f431c0ae7c0 0x7f431c0b2ea0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 rx=0x7f430c002f70 tx=0x7f430c014000).stop 2022-06-29T16:07:13.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.261+0000 7f432172d700 1 -- 172.21.15.60:0/357496839 shutdown_connections 2022-06-29T16:07:13.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.261+0000 7f432172d700 1 --2- 172.21.15.60:0/357496839 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4300053a70 0x7f4300055f30 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:13.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.261+0000 7f432172d700 1 --2- 172.21.15.60:0/357496839 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f431c0b46c0 0x7f431c0b0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:13.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.261+0000 7f432172d700 1 --2- 172.21.15.60:0/357496839 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f431c0b4010 0x7f431c112530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:13.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.261+0000 7f432172d700 1 --2- 172.21.15.60:0/357496839 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f431c0ae7c0 0x7f431c0b2ea0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:13.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.261+0000 7f432172d700 1 -- 172.21.15.60:0/357496839 >> 172.21.15.60:0/357496839 conn(0x7f431c0b9340 msgr2=0x7f431c13eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:13.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.261+0000 7f432172d700 1 -- 172.21.15.60:0/357496839 shutdown_connections 2022-06-29T16:07:13.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.261+0000 7f432172d700 1 -- 172.21.15.60:0/357496839 wait complete. 2022-06-29T16:07:13.279 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-06-29T16:07:13.280 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:07:13.300 DEBUG:teuthology.orchestra.run.smithi092:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:07:13.322 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:07:13.346 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:07:13.369 DEBUG:teuthology.orchestra.run.smithi197:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:07:13.390 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-06-29T16:07:13.391 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:13.392 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:07:13.392 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:07:13.399 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-06-29T16:07:13.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:07:13.401 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-06-29T16:07:13.402 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:13.402 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:07:13.403 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:07:13.417 INFO:teuthology.orchestra.run.smithi156.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-06-29T16:07:13.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:07:13.420 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-06-29T16:07:13.421 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:13.422 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:07:13.423 DEBUG:teuthology.orchestra.run.smithi197:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T16:07:13.430 INFO:teuthology.orchestra.run.smithi197.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-06-29T16:07:13.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:07:13.433 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-06-29T16:07:13.434 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-06-29T16:07:13.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.541+0000 7f9256384700 1 Processor -- start 2022-06-29T16:07:13.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.541+0000 7f9256384700 1 -- start start 2022-06-29T16:07:13.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.541+0000 7f9256384700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f92501426b0 0x7f92500b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:13.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.541+0000 7f9256384700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9250143a40 con 0x7f92500b4010 2022-06-29T16:07:13.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.541+0000 7f9256384700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92500aee60 con 0x7f92500ae7c0 2022-06-29T16:07:13.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.541+0000 7f9256384700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92500af100 con 0x7f92501426b0 2022-06-29T16:07:13.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.541+0000 7f9254921700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f92501426b0 0x7f92500b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:13.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.541+0000 7f924f7fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f92500b4010 0x7f92500b46c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60786/0 (socket says 172.21.15.60:60786) 2022-06-29T16:07:13.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.541+0000 7f924f7fe700 1 -- 172.21.15.60:0/3168243256 learned_addr learned my addr 172.21.15.60:0/3168243256 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:13.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.541+0000 7f924effd700 1 -- 172.21.15.60:0/3168243256 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 556746069 0 0) 0x7f9250143a40 con 0x7f92500b4010 2022-06-29T16:07:13.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.541+0000 7f924effd700 1 -- 172.21.15.60:0/3168243256 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9234003260 con 0x7f92500b4010 2022-06-29T16:07:13.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.541+0000 7f9254921700 1 -- 172.21.15.60:0/3168243256 >> v1:172.21.15.92:6789/0 conn(0x7f92500ae7c0 legacy=0x7f9250141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:13.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.541+0000 7f9254921700 1 -- 172.21.15.60:0/3168243256 >> v1:172.21.15.60:6789/0 conn(0x7f92500b4010 legacy=0x7f92500b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:13.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.541+0000 7f9254921700 1 -- 172.21.15.60:0/3168243256 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92500aeba0 con 0x7f92501426b0 2022-06-29T16:07:13.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.541+0000 7f9254921700 1 --2- 172.21.15.60:0/3168243256 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92501426b0 0x7f92500b4dd0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 rx=0x7f9248002f70 tx=0x7f9248006df0).ready entity=mon.0 client_cookie=618dacbb32594b5d server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:13.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.541+0000 7f924effd700 1 -- 172.21.15.60:0/3168243256 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f924800d610 con 0x7f92501426b0 2022-06-29T16:07:13.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.541+0000 7f924effd700 1 -- 172.21.15.60:0/3168243256 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9248000e90 con 0x7f92501426b0 2022-06-29T16:07:13.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.541+0000 7f924effd700 1 -- 172.21.15.60:0/3168243256 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f924800d960 con 0x7f92501426b0 2022-06-29T16:07:13.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.545+0000 7f9256384700 1 -- 172.21.15.60:0/3168243256 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92501426b0 msgr2=0x7f92500b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:13.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.545+0000 7f9256384700 1 --2- 172.21.15.60:0/3168243256 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92501426b0 0x7f92500b4dd0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 rx=0x7f9248002f70 tx=0x7f9248006df0).stop 2022-06-29T16:07:13.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.545+0000 7f9256384700 1 -- 172.21.15.60:0/3168243256 shutdown_connections 2022-06-29T16:07:13.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.545+0000 7f9256384700 1 --2- 172.21.15.60:0/3168243256 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92501426b0 0x7f92500b4dd0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:13.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.545+0000 7f9256384700 1 -- 172.21.15.60:0/3168243256 >> 172.21.15.60:0/3168243256 conn(0x7f92500b9340 msgr2=0x7f92500b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:13.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.545+0000 7f9256384700 1 -- 172.21.15.60:0/3168243256 shutdown_connections 2022-06-29T16:07:13.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.545+0000 7f9256384700 1 -- 172.21.15.60:0/3168243256 wait complete. 2022-06-29T16:07:13.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.545+0000 7f9256384700 1 Processor -- start 2022-06-29T16:07:13.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.545+0000 7f9256384700 1 -- start start 2022-06-29T16:07:13.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.545+0000 7f9256384700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92500ae7c0 0x7f92500b2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:13.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.545+0000 7f9256384700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92500b4010 0x7f925013eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:13.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.545+0000 7f9256384700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f92501426b0 0x7f925013f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:13.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.545+0000 7f9256384700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92500b06b0 con 0x7f92500ae7c0 2022-06-29T16:07:13.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.545+0000 7f9256384700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92500b3090 con 0x7f92500b4010 2022-06-29T16:07:13.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.545+0000 7f9256384700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92500b32d0 con 0x7f92501426b0 2022-06-29T16:07:13.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.545+0000 7f924ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92500ae7c0 0x7f92500b2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:13.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.545+0000 7f924ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92500ae7c0 0x7f92500b2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52704/0 (socket says 172.21.15.60:52704) 2022-06-29T16:07:13.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.545+0000 7f924ffff700 1 -- 172.21.15.60:0/1425170296 learned_addr learned my addr 172.21.15.60:0/1425170296 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:13.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.545+0000 7f9254921700 1 --2- 172.21.15.60:0/1425170296 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f92501426b0 0x7f925013f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:13.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.545+0000 7f924f7fe700 1 --2- 172.21.15.60:0/1425170296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92500b4010 0x7f925013eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:13.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.549+0000 7f924ffff700 1 -- 172.21.15.60:0/1425170296 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f92501426b0 msgr2=0x7f925013f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:13.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.549+0000 7f924ffff700 1 --2- 172.21.15.60:0/1425170296 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f92501426b0 0x7f925013f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:13.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.549+0000 7f924ffff700 1 -- 172.21.15.60:0/1425170296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92500b4010 msgr2=0x7f925013eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:13.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.549+0000 7f924ffff700 1 --2- 172.21.15.60:0/1425170296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92500b4010 0x7f925013eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:13.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.549+0000 7f924ffff700 1 -- 172.21.15.60:0/1425170296 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f924800b180 con 0x7f92500ae7c0 2022-06-29T16:07:13.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.549+0000 7f924ffff700 1 --2- 172.21.15.60:0/1425170296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92500ae7c0 0x7f92500b2ba0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 rx=0x7f9250060a90 tx=0x7f9240007f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:13.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.549+0000 7f924d7fa700 1 -- 172.21.15.60:0/1425170296 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9240006aa0 con 0x7f92500ae7c0 2022-06-29T16:07:13.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.549+0000 7f924d7fa700 1 -- 172.21.15.60:0/1425170296 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9240006c00 con 0x7f92500ae7c0 2022-06-29T16:07:13.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.549+0000 7f9256384700 1 -- 172.21.15.60:0/1425170296 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92500af4e0 con 0x7f92500ae7c0 2022-06-29T16:07:13.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.549+0000 7f924d7fa700 1 -- 172.21.15.60:0/1425170296 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9240007030 con 0x7f92500ae7c0 2022-06-29T16:07:13.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.549+0000 7f9256384700 1 -- 172.21.15.60:0/1425170296 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9250146b50 con 0x7f92500ae7c0 2022-06-29T16:07:13.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.549+0000 7f9256384700 1 -- 172.21.15.60:0/1425170296 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92500af750 con 0x7f92500ae7c0 2022-06-29T16:07:13.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.549+0000 7f924d7fa700 1 -- 172.21.15.60:0/1425170296 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9240010030 con 0x7f92500ae7c0 2022-06-29T16:07:13.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.549+0000 7f924d7fa700 1 --2- 172.21.15.60:0/1425170296 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9228053b30 0x7f9228055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:13.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.549+0000 7f924d7fa700 1 -- 172.21.15.60:0/1425170296 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f924006bc00 con 0x7f92500ae7c0 2022-06-29T16:07:13.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.553+0000 7f924f7fe700 1 --2- 172.21.15.60:0/1425170296 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9228053b30 0x7f9228055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:13.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.553+0000 7f924f7fe700 1 --2- 172.21.15.60:0/1425170296 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9228053b30 0x7f9228055ff0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0x7f923800d000 tx=0x7f9238009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:13.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.557+0000 7f924d7fa700 1 -- 172.21.15.60:0/1425170296 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9240042c10 con 0x7f92500ae7c0 2022-06-29T16:07:13.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.685+0000 7f9256384700 1 -- 172.21.15.60:0/1425170296 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f92500573d0 con 0x7f92500ae7c0 2022-06-29T16:07:13.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.685+0000 7f924d7fa700 1 -- 172.21.15.60:0/1425170296 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v123) v1 ==== 81+0+28837 (secure 0 0 0) 0x7f9240046230 con 0x7f92500ae7c0 2022-06-29T16:07:13.693 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:13.693 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:07:13.694 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 123, 2022-06-29T16:07:13.694 INFO:teuthology.orchestra.run.smithi060.stdout: "fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", 2022-06-29T16:07:13.695 INFO:teuthology.orchestra.run.smithi060.stdout: "created": "2022-06-29T15:58:07.965719+0000", 2022-06-29T16:07:13.695 INFO:teuthology.orchestra.run.smithi060.stdout: "modified": "2022-06-29T16:07:10.945259+0000", 2022-06-29T16:07:13.695 INFO:teuthology.orchestra.run.smithi060.stdout: "last_up_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:07:13.696 INFO:teuthology.orchestra.run.smithi060.stdout: "last_in_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:07:13.696 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-06-29T16:07:13.697 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_num": 5799936, 2022-06-29T16:07:13.697 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_set": [ 2022-06-29T16:07:13.698 INFO:teuthology.orchestra.run.smithi060.stdout: "pglog_hardlimit", 2022-06-29T16:07:13.698 INFO:teuthology.orchestra.run.smithi060.stdout: "purged_snapdirs", 2022-06-29T16:07:13.699 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_deletes", 2022-06-29T16:07:13.699 INFO:teuthology.orchestra.run.smithi060.stdout: "sortbitwise" 2022-06-29T16:07:13.700 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:07:13.700 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_version": 4, 2022-06-29T16:07:13.701 INFO:teuthology.orchestra.run.smithi060.stdout: "full_ratio": 0.94999998807907104, 2022-06-29T16:07:13.701 INFO:teuthology.orchestra.run.smithi060.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-06-29T16:07:13.702 INFO:teuthology.orchestra.run.smithi060.stdout: "nearfull_ratio": 0.85000002384185791, 2022-06-29T16:07:13.702 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_snapshot": "", 2022-06-29T16:07:13.702 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_max": 24, 2022-06-29T16:07:13.703 INFO:teuthology.orchestra.run.smithi060.stdout: "max_osd": 8, 2022-06-29T16:07:13.703 INFO:teuthology.orchestra.run.smithi060.stdout: "require_min_compat_client": "jewel", 2022-06-29T16:07:13.704 INFO:teuthology.orchestra.run.smithi060.stdout: "min_compat_client": "jewel", 2022-06-29T16:07:13.704 INFO:teuthology.orchestra.run.smithi060.stdout: "require_osd_release": "octopus", 2022-06-29T16:07:13.704 INFO:teuthology.orchestra.run.smithi060.stdout: "pools": [ 2022-06-29T16:07:13.705 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.705 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2022-06-29T16:07:13.706 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "device_health_metrics", 2022-06-29T16:07:13.706 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:10.961127+0000", 2022-06-29T16:07:13.707 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:07:13.707 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:07:13.707 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:07:13.708 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:07:13.708 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:07:13.709 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:07:13.709 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:07:13.710 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:07:13.710 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 1, 2022-06-29T16:07:13.710 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 1, 2022-06-29T16:07:13.711 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 1, 2022-06-29T16:07:13.712 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 1, 2022-06-29T16:07:13.712 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 1, 2022-06-29T16:07:13.712 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:07:13.712 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:07:13.712 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:07:13.713 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:07:13.713 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:07:13.713 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:07:13.713 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:07:13.714 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.714 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "15", 2022-06-29T16:07:13.714 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:07:13.714 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:07:13.715 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:07:13.715 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:07:13.715 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:07:13.715 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:07:13.715 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:07:13.716 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:07:13.716 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:07:13.716 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:07:13.716 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:07:13.716 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:07:13.717 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:07:13.717 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:07:13.717 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:07:13.717 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:07:13.718 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:07:13.718 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:07:13.718 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:07:13.718 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:07:13.718 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:07:13.719 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:07:13.719 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:07:13.719 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:07:13.719 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:07:13.720 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:07:13.720 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.720 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:07:13.720 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:07:13.720 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:07:13.721 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:07:13.721 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:07:13.721 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:07:13.721 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:07:13.722 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:07:13.722 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:07:13.722 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:07:13.722 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:07:13.722 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:07:13.723 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 1 2022-06-29T16:07:13.723 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.723 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:07:13.723 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_devicehealth": {} 2022-06-29T16:07:13.724 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.724 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.724 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.724 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 2, 2022-06-29T16:07:13.724 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "rbd", 2022-06-29T16:07:13.725 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:19.184276+0000", 2022-06-29T16:07:13.725 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:07:13.725 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:07:13.726 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:07:13.726 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:07:13.726 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:07:13.726 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:07:13.726 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:07:13.727 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:07:13.727 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:07:13.727 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:07:13.728 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:07:13.728 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:07:13.728 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:07:13.728 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:07:13.729 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:07:13.729 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:07:13.729 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:07:13.730 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:07:13.730 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:07:13.730 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:07:13.730 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.731 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "18", 2022-06-29T16:07:13.731 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:07:13.732 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:07:13.732 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:07:13.732 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:07:13.733 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:07:13.733 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:07:13.733 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:07:13.733 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:07:13.733 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:07:13.734 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:07:13.734 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:07:13.734 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:07:13.735 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:07:13.735 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:07:13.735 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:07:13.735 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:07:13.735 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:07:13.736 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:07:13.736 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:07:13.736 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:07:13.737 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:07:13.737 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:07:13.737 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:07:13.737 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:07:13.738 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:07:13.738 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:07:13.738 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.738 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:07:13.739 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:07:13.739 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:07:13.739 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:07:13.739 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:07:13.739 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:07:13.740 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:07:13.740 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:07:13.740 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:07:13.740 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:07:13.740 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:07:13.741 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:07:13.741 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:07:13.741 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd": {} 2022-06-29T16:07:13.742 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.742 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.742 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:07:13.743 INFO:teuthology.orchestra.run.smithi060.stdout: "osds": [ 2022-06-29T16:07:13.743 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.743 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:07:13.743 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "11e7ab9e-fce2-470f-8195-ea4c3c52df22", 2022-06-29T16:07:13.744 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:07:13.744 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:07:13.744 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:07:13.744 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:07:13.744 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:07:13.745 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:07:13.745 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:07:13.745 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 119, 2022-06-29T16:07:13.745 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:07:13.746 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:07:13.746 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:07:13.746 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:13.746 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.747 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:13.747 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6816", 2022-06-29T16:07:13.747 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:07:13.747 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.747 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.748 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:13.748 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6825", 2022-06-29T16:07:13.748 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:07:13.749 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.749 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.749 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.749 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:07:13.749 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:13.750 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.750 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:13.750 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6826", 2022-06-29T16:07:13.750 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:07:13.751 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.751 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.751 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:13.752 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6827", 2022-06-29T16:07:13.752 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:07:13.752 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.753 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.753 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.753 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:07:13.753 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:13.754 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.754 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:13.754 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6830", 2022-06-29T16:07:13.754 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:07:13.755 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.755 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.755 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:13.755 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6831", 2022-06-29T16:07:13.756 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:07:13.756 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.756 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.756 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.756 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:07:13.757 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:13.757 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.757 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:13.757 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6828", 2022-06-29T16:07:13.758 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:07:13.758 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.758 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.758 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:13.759 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6829", 2022-06-29T16:07:13.759 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:07:13.759 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.759 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.760 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.760 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6825/12890", 2022-06-29T16:07:13.760 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6827/12890", 2022-06-29T16:07:13.760 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6831/12890", 2022-06-29T16:07:13.760 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6829/12890", 2022-06-29T16:07:13.761 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:07:13.761 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:07:13.761 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:07:13.761 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.762 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.762 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.762 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:07:13.762 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "ca43f136-6f69-4431-a157-fbe073fb585c", 2022-06-29T16:07:13.763 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:07:13.763 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:07:13.763 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:07:13.763 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:07:13.764 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:07:13.764 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:07:13.764 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:07:13.765 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 119, 2022-06-29T16:07:13.765 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:07:13.765 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:07:13.765 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:07:13.765 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:13.766 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.766 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:13.766 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6815", 2022-06-29T16:07:13.766 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:07:13.767 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.767 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.767 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:13.767 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6818", 2022-06-29T16:07:13.768 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:07:13.768 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.768 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.768 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.768 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:07:13.769 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:13.769 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.769 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:13.769 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6819", 2022-06-29T16:07:13.770 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:07:13.770 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.770 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.771 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:13.771 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6820", 2022-06-29T16:07:13.771 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:07:13.772 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.772 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.773 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.773 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:07:13.773 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:13.774 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.774 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:13.775 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6823", 2022-06-29T16:07:13.775 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:07:13.775 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.776 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.776 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:13.777 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6824", 2022-06-29T16:07:13.777 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:07:13.777 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.778 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.778 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.779 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:07:13.779 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:13.780 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.780 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:13.781 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6821", 2022-06-29T16:07:13.781 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:07:13.781 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.782 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.782 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:13.782 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6822", 2022-06-29T16:07:13.783 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:07:13.783 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.784 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.784 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.785 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6818/12892", 2022-06-29T16:07:13.785 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6820/12892", 2022-06-29T16:07:13.785 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6824/12892", 2022-06-29T16:07:13.786 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6822/12892", 2022-06-29T16:07:13.786 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:07:13.786 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:07:13.786 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:07:13.787 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.787 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.787 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.787 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:07:13.787 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "74fac45f-5f07-44ec-a9cb-2644c123fd4a", 2022-06-29T16:07:13.788 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:07:13.788 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:07:13.788 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:07:13.789 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:07:13.789 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:07:13.789 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:07:13.789 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:07:13.789 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 114, 2022-06-29T16:07:13.790 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:07:13.790 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:07:13.790 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:07:13.790 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:13.791 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.791 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:13.791 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6808", 2022-06-29T16:07:13.791 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:07:13.792 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.792 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.792 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:13.792 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6809", 2022-06-29T16:07:13.792 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:07:13.793 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.793 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.793 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.793 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:07:13.794 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:13.794 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.794 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:13.794 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6810", 2022-06-29T16:07:13.795 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:07:13.795 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.795 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.795 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:13.795 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6811", 2022-06-29T16:07:13.796 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:07:13.796 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.796 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.796 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.797 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:07:13.797 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:13.797 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.797 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:13.797 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6814", 2022-06-29T16:07:13.798 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:07:13.798 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.798 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.798 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:13.799 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6817", 2022-06-29T16:07:13.799 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:07:13.799 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.800 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.800 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.800 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:07:13.800 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:13.801 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.801 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:13.801 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6812", 2022-06-29T16:07:13.801 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:07:13.802 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.802 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.802 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:13.802 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6813", 2022-06-29T16:07:13.802 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:07:13.803 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.803 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.804 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.804 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6809/12891", 2022-06-29T16:07:13.804 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6811/12891", 2022-06-29T16:07:13.805 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6817/12891", 2022-06-29T16:07:13.805 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6813/12891", 2022-06-29T16:07:13.805 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:07:13.805 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:07:13.805 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:07:13.806 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.806 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.806 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.806 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:07:13.807 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "2581d351-973a-4b4e-85ca-3658f0068765", 2022-06-29T16:07:13.807 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:07:13.808 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:07:13.808 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:07:13.808 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:07:13.808 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:07:13.809 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:07:13.809 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:07:13.809 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 119, 2022-06-29T16:07:13.809 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:07:13.809 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:07:13.810 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:07:13.810 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:13.810 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.810 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:13.811 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6800", 2022-06-29T16:07:13.811 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:07:13.811 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.811 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.812 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:13.812 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6801", 2022-06-29T16:07:13.812 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:07:13.812 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.812 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.813 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.813 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:07:13.813 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:13.813 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.814 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:13.814 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6802", 2022-06-29T16:07:13.814 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:07:13.814 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.815 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.815 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:13.815 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6803", 2022-06-29T16:07:13.815 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:07:13.815 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.816 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.816 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.816 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:07:13.816 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:13.817 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.817 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:13.817 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6806", 2022-06-29T16:07:13.817 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:07:13.818 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.818 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.818 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:13.818 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6807", 2022-06-29T16:07:13.819 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:07:13.819 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.819 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.819 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.820 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:07:13.820 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:13.820 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.820 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:13.820 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6804", 2022-06-29T16:07:13.821 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:07:13.821 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.821 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.821 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:13.822 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6805", 2022-06-29T16:07:13.822 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:07:13.822 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.822 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.823 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.823 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6801/12889", 2022-06-29T16:07:13.823 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6803/12889", 2022-06-29T16:07:13.823 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6807/12889", 2022-06-29T16:07:13.824 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6805/12889", 2022-06-29T16:07:13.824 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:07:13.824 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:07:13.824 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:07:13.824 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.825 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.825 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.825 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:07:13.826 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "3d234db9-87cd-4fe9-814f-0240a287d8b4", 2022-06-29T16:07:13.826 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:07:13.826 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:07:13.826 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:07:13.827 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:07:13.827 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:07:13.827 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:07:13.827 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:07:13.827 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 114, 2022-06-29T16:07:13.828 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:07:13.828 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:07:13.828 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:07:13.829 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:13.829 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.829 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:13.829 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6808", 2022-06-29T16:07:13.829 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:07:13.830 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.830 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.830 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:13.830 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6810", 2022-06-29T16:07:13.831 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:07:13.831 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.831 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.831 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.832 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:07:13.832 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:13.832 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.832 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:13.832 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6812", 2022-06-29T16:07:13.833 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:07:13.833 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.833 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.833 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:13.834 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6814", 2022-06-29T16:07:13.834 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:07:13.834 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.834 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.834 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.835 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:07:13.835 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:13.835 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.835 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:13.836 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6823", 2022-06-29T16:07:13.836 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:07:13.836 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.836 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.837 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:13.837 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6825", 2022-06-29T16:07:13.837 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:07:13.837 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.837 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.838 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.838 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:07:13.838 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:13.838 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.839 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:13.839 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6817", 2022-06-29T16:07:13.839 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:07:13.839 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.840 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.840 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:13.840 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6820", 2022-06-29T16:07:13.840 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:07:13.840 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.841 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.841 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.841 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6810/13249", 2022-06-29T16:07:13.841 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6814/13249", 2022-06-29T16:07:13.842 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6825/13249", 2022-06-29T16:07:13.842 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6820/13249", 2022-06-29T16:07:13.842 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:07:13.842 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:07:13.843 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:07:13.843 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.843 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.843 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.844 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:07:13.844 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "e4bf1104-ff5c-438b-b608-8cee31c163f6", 2022-06-29T16:07:13.844 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:07:13.844 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:07:13.845 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:07:13.845 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:07:13.845 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:07:13.845 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:07:13.846 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:07:13.846 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 119, 2022-06-29T16:07:13.846 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:07:13.846 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:07:13.847 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:07:13.847 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:13.847 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.847 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:13.847 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6809", 2022-06-29T16:07:13.848 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:07:13.848 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.848 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.848 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:13.849 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6811", 2022-06-29T16:07:13.849 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:07:13.849 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.849 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.849 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.850 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:07:13.850 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:13.850 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.850 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:13.851 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6813", 2022-06-29T16:07:13.851 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:07:13.851 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.851 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.852 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:13.852 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6815", 2022-06-29T16:07:13.852 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:07:13.853 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.853 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.853 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.854 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:07:13.854 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:13.854 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.854 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:13.855 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6824", 2022-06-29T16:07:13.855 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:07:13.855 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.855 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.856 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:13.856 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6827", 2022-06-29T16:07:13.856 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:07:13.856 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.857 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.857 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.857 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:07:13.857 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:13.858 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.858 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:13.858 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6818", 2022-06-29T16:07:13.858 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:07:13.859 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.859 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.859 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:13.859 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6821", 2022-06-29T16:07:13.859 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:07:13.860 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.860 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.860 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.860 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6811/13248", 2022-06-29T16:07:13.861 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6815/13248", 2022-06-29T16:07:13.861 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6827/13248", 2022-06-29T16:07:13.861 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6821/13248", 2022-06-29T16:07:13.861 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:07:13.861 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:07:13.862 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:07:13.862 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.862 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.862 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.863 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:07:13.863 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "8564dd25-27ab-4203-b54e-af6d3aa05857", 2022-06-29T16:07:13.863 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:07:13.863 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:07:13.864 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:07:13.864 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:07:13.864 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:07:13.864 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:07:13.865 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:07:13.865 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 119, 2022-06-29T16:07:13.865 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:07:13.865 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:07:13.866 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:07:13.866 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:13.866 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.866 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:13.867 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6816", 2022-06-29T16:07:13.867 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:07:13.867 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.867 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.868 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:13.868 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6819", 2022-06-29T16:07:13.868 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:07:13.868 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.869 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.869 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.869 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:07:13.869 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:13.869 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.870 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:13.870 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6822", 2022-06-29T16:07:13.870 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:07:13.870 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.871 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.871 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:13.871 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6826", 2022-06-29T16:07:13.871 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:07:13.871 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.872 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.872 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.872 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:07:13.872 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:13.873 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.873 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:13.873 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6830", 2022-06-29T16:07:13.873 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:07:13.873 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.874 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.874 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:13.874 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6831", 2022-06-29T16:07:13.874 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:07:13.875 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.875 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.875 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.875 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:07:13.876 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:13.876 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.876 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:13.876 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6828", 2022-06-29T16:07:13.876 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:07:13.877 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.877 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.877 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:13.877 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6829", 2022-06-29T16:07:13.878 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:07:13.878 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.878 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.879 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.879 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6819/13250", 2022-06-29T16:07:13.879 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6826/13250", 2022-06-29T16:07:13.879 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6831/13250", 2022-06-29T16:07:13.879 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6829/13250", 2022-06-29T16:07:13.880 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:07:13.880 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:07:13.880 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:07:13.881 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.881 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.882 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.882 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:07:13.883 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "bd513a9c-3b19-4314-9897-d300b5c897bb", 2022-06-29T16:07:13.883 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:07:13.884 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:07:13.884 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:07:13.884 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:07:13.885 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:07:13.885 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:07:13.886 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:07:13.886 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 119, 2022-06-29T16:07:13.887 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:07:13.887 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:07:13.887 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:07:13.888 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:13.888 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.889 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:13.889 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6800", 2022-06-29T16:07:13.889 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:07:13.890 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.890 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.890 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:13.891 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6801", 2022-06-29T16:07:13.891 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:07:13.891 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.891 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.892 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.892 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:07:13.892 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:13.892 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.893 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:13.893 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6802", 2022-06-29T16:07:13.893 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:07:13.894 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.894 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.894 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:13.894 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6803", 2022-06-29T16:07:13.894 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:07:13.895 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.895 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.895 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.895 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:07:13.896 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:13.896 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.896 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:13.896 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6806", 2022-06-29T16:07:13.897 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:07:13.897 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.897 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.897 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:13.898 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6807", 2022-06-29T16:07:13.898 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:07:13.898 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.899 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.899 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.899 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:07:13.900 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:13.900 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.900 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:13.900 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6804", 2022-06-29T16:07:13.901 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:07:13.901 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.901 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.901 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:13.902 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6805", 2022-06-29T16:07:13.902 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:07:13.902 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.902 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.903 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.903 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6801/13247", 2022-06-29T16:07:13.903 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6803/13247", 2022-06-29T16:07:13.903 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6807/13247", 2022-06-29T16:07:13.904 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6805/13247", 2022-06-29T16:07:13.904 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:07:13.904 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:07:13.904 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:07:13.905 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:13.905 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.905 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:07:13.905 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_xinfo": [ 2022-06-29T16:07:13.905 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.906 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:07:13.906 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:07:13.906 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:07:13.906 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:07:13.907 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:07:13.907 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:07:13.907 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.891342+0000", 2022-06-29T16:07:13.907 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:07:13.908 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.908 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.908 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:07:13.908 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:07:13.908 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:07:13.909 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:07:13.909 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:07:13.909 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:07:13.909 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.900866+0000", 2022-06-29T16:07:13.910 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:07:13.910 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.910 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.910 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:07:13.911 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:07:13.911 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:07:13.911 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:07:13.911 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:07:13.912 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:07:13.912 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.912755+0000", 2022-06-29T16:07:13.912 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:07:13.912 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.913 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.913 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:07:13.913 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:07:13.913 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:07:13.914 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:07:13.914 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:07:13.914 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:07:13.914 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.864332+0000", 2022-06-29T16:07:13.915 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:07:13.915 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.915 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.915 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:07:13.915 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:07:13.916 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:07:13.916 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:07:13.916 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:07:13.916 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:07:13.917 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.863756+0000", 2022-06-29T16:07:13.917 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:07:13.917 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.917 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.918 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:07:13.918 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:07:13.918 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:07:13.918 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:07:13.919 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:07:13.919 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:07:13.919 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.869041+0000", 2022-06-29T16:07:13.919 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:07:13.919 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.920 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.920 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:07:13.920 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:07:13.920 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:07:13.921 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:07:13.921 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:07:13.921 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:07:13.921 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.895043+0000", 2022-06-29T16:07:13.922 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:07:13.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:13.923 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:07:13.923 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:07:13.923 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:07:13.923 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:07:13.924 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:07:13.924 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:07:13.924 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.889491+0000", 2022-06-29T16:07:13.924 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:07:13.925 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.925 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:07:13.925 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap": [], 2022-06-29T16:07:13.925 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap_items": [], 2022-06-29T16:07:13.926 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_temp": [], 2022-06-29T16:07:13.926 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_temp": [], 2022-06-29T16:07:13.926 INFO:teuthology.orchestra.run.smithi060.stdout: "blacklist": { 2022-06-29T16:07:13.926 INFO:teuthology.orchestra.run.smithi060.stdout: "172.21.15.60:6833/2749128920": "2022-06-30T16:06:51.286403+0000", 2022-06-29T16:07:13.927 INFO:teuthology.orchestra.run.smithi060.stdout: "172.21.15.60:6832/2749128920": "2022-06-30T16:06:51.286403+0000" 2022-06-29T16:07:13.927 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.927 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profiles": { 2022-06-29T16:07:13.927 INFO:teuthology.orchestra.run.smithi060.stdout: "default": { 2022-06-29T16:07:13.928 INFO:teuthology.orchestra.run.smithi060.stdout: "crush-failure-domain": "osd", 2022-06-29T16:07:13.928 INFO:teuthology.orchestra.run.smithi060.stdout: "k": "2", 2022-06-29T16:07:13.928 INFO:teuthology.orchestra.run.smithi060.stdout: "m": "1", 2022-06-29T16:07:13.928 INFO:teuthology.orchestra.run.smithi060.stdout: "plugin": "jerasure", 2022-06-29T16:07:13.929 INFO:teuthology.orchestra.run.smithi060.stdout: "ruleset-failure-domain": "osd", 2022-06-29T16:07:13.929 INFO:teuthology.orchestra.run.smithi060.stdout: "technique": "reed_sol_van" 2022-06-29T16:07:13.929 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:13.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:13.930 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps_queue": [], 2022-06-29T16:07:13.930 INFO:teuthology.orchestra.run.smithi060.stdout: "new_removed_snaps": [], 2022-06-29T16:07:13.930 INFO:teuthology.orchestra.run.smithi060.stdout: "new_purged_snaps": [], 2022-06-29T16:07:13.930 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_node_flags": {}, 2022-06-29T16:07:13.931 INFO:teuthology.orchestra.run.smithi060.stdout: "device_class_flags": {} 2022-06-29T16:07:13.931 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:07:13.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.689+0000 7f9256384700 1 -- 172.21.15.60:0/1425170296 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9228053b30 msgr2=0x7f9228055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:13.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.689+0000 7f9256384700 1 --2- 172.21.15.60:0/1425170296 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9228053b30 0x7f9228055ff0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0x7f923800d000 tx=0x7f9238009040).stop 2022-06-29T16:07:13.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.689+0000 7f9256384700 1 -- 172.21.15.60:0/1425170296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92500ae7c0 msgr2=0x7f92500b2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:13.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.689+0000 7f9256384700 1 --2- 172.21.15.60:0/1425170296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92500ae7c0 0x7f92500b2ba0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 rx=0x7f9250060a90 tx=0x7f9240007f90).stop 2022-06-29T16:07:13.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.689+0000 7f9256384700 1 -- 172.21.15.60:0/1425170296 shutdown_connections 2022-06-29T16:07:13.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.689+0000 7f9256384700 1 --2- 172.21.15.60:0/1425170296 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9228053b30 0x7f9228055ff0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:13.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.689+0000 7f9256384700 1 --2- 172.21.15.60:0/1425170296 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f92501426b0 0x7f925013f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:13.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.689+0000 7f9256384700 1 --2- 172.21.15.60:0/1425170296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92500b4010 0x7f925013eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:13.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.689+0000 7f9256384700 1 --2- 172.21.15.60:0/1425170296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92500ae7c0 0x7f92500b2ba0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:13.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.689+0000 7f9256384700 1 -- 172.21.15.60:0/1425170296 >> 172.21.15.60:0/1425170296 conn(0x7f92500b9340 msgr2=0x7f9250140ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:13.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.689+0000 7f9256384700 1 -- 172.21.15.60:0/1425170296 shutdown_connections 2022-06-29T16:07:13.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:13.689+0000 7f9256384700 1 -- 172.21.15.60:0/1425170296 wait complete. 2022-06-29T16:07:13.938 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:07:14.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.045+0000 7f94ead8f700 1 Processor -- start 2022-06-29T16:07:14.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.045+0000 7f94ead8f700 1 -- start start 2022-06-29T16:07:14.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.045+0000 7f94ead8f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94e413f3d0 0x7f94e4060260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:14.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.045+0000 7f94ead8f700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f94e4143820 0x7f94e40b6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:14.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.045+0000 7f94ead8f700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94e41430e0 con 0x7f94e40b6c00 2022-06-29T16:07:14.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.045+0000 7f94ead8f700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94e4142ef0 con 0x7f94e4143820 2022-06-29T16:07:14.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.045+0000 7f94ead8f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94e40acb90 con 0x7f94e413f3d0 2022-06-29T16:07:14.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.045+0000 7f94e3fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f94e4143820 0x7f94e40b6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:14.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.045+0000 7f94e932c700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f94e40b6c00 0x7f94e413f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:36816/0 (socket says 172.21.15.60:36816) 2022-06-29T16:07:14.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.045+0000 7f94e8b2b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94e413f3d0 0x7f94e4060260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:14.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.045+0000 7f94e932c700 1 -- 172.21.15.60:0/1472633672 learned_addr learned my addr 172.21.15.60:0/1472633672 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:14.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.045+0000 7f94e37fe700 1 -- 172.21.15.60:0/1472633672 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3071217706 0 0) 0x7f94e41430e0 con 0x7f94e40b6c00 2022-06-29T16:07:14.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.045+0000 7f94e37fe700 1 -- 172.21.15.60:0/1472633672 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94c8003260 con 0x7f94e40b6c00 2022-06-29T16:07:14.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.045+0000 7f94e8b2b700 1 -- 172.21.15.60:0/1472633672 >> v2:172.21.15.60:3300/0 conn(0x7f94e4143820 msgr2=0x7f94e40b6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:14.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.045+0000 7f94e8b2b700 1 --2- 172.21.15.60:0/1472633672 >> v2:172.21.15.60:3300/0 conn(0x7f94e4143820 0x7f94e40b6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:14.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.045+0000 7f94e8b2b700 1 -- 172.21.15.60:0/1472633672 >> v1:172.21.15.92:6789/0 conn(0x7f94e40b6c00 legacy=0x7f94e413f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:14.055 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.045+0000 7f94e8b2b700 1 -- 172.21.15.60:0/1472633672 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e4143380 con 0x7f94e413f3d0 2022-06-29T16:07:14.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.045+0000 7f94e3fff700 1 --2- 172.21.15.60:0/1472633672 >> v2:172.21.15.60:3300/0 conn(0x7f94e4143820 0x7f94e40b6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:07:14.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.045+0000 7f94e8b2b700 1 --2- 172.21.15.60:0/1472633672 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94e413f3d0 0x7f94e4060260 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f94d40032f0 tx=0x7f94d40091d0).ready entity=mon.2 client_cookie=4d41e3470c5df48 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:14.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.045+0000 7f94e37fe700 1 -- 172.21.15.60:0/1472633672 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94d400b600 con 0x7f94e413f3d0 2022-06-29T16:07:14.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.045+0000 7f94e37fe700 1 -- 172.21.15.60:0/1472633672 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94d400b760 con 0x7f94e413f3d0 2022-06-29T16:07:14.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.045+0000 7f94e37fe700 1 -- 172.21.15.60:0/1472633672 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94d400b8d0 con 0x7f94e413f3d0 2022-06-29T16:07:14.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.049+0000 7f94ead8f700 1 -- 172.21.15.60:0/1472633672 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94e413f3d0 msgr2=0x7f94e4060260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:14.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.049+0000 7f94ead8f700 1 --2- 172.21.15.60:0/1472633672 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94e413f3d0 0x7f94e4060260 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f94d40032f0 tx=0x7f94d40091d0).stop 2022-06-29T16:07:14.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.049+0000 7f94ead8f700 1 -- 172.21.15.60:0/1472633672 shutdown_connections 2022-06-29T16:07:14.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.049+0000 7f94ead8f700 1 --2- 172.21.15.60:0/1472633672 >> v2:172.21.15.60:3300/0 conn(0x7f94e4143820 0x7f94e40b6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:14.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.049+0000 7f94ead8f700 1 --2- 172.21.15.60:0/1472633672 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94e413f3d0 0x7f94e4060260 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:14.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.049+0000 7f94ead8f700 1 -- 172.21.15.60:0/1472633672 >> 172.21.15.60:0/1472633672 conn(0x7f94e40643e0 msgr2=0x7f94e40b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:14.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.049+0000 7f94ead8f700 1 -- 172.21.15.60:0/1472633672 shutdown_connections 2022-06-29T16:07:14.063 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.049+0000 7f94ead8f700 1 -- 172.21.15.60:0/1472633672 wait complete. 2022-06-29T16:07:14.063 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.049+0000 7f94ead8f700 1 Processor -- start 2022-06-29T16:07:14.063 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.049+0000 7f94ead8f700 1 -- start start 2022-06-29T16:07:14.064 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.049+0000 7f94ead8f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94e40b6c00 0x7f94e409b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:14.064 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.049+0000 7f94ead8f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94e413f3d0 0x7f94e413be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:14.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.049+0000 7f94ead8f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94e4143820 0x7f94e413c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:14.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.049+0000 7f94ead8f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e413c810 con 0x7f94e413f3d0 2022-06-29T16:07:14.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.049+0000 7f94ead8f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e4064e00 con 0x7f94e4143820 2022-06-29T16:07:14.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.049+0000 7f94ead8f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94e40650a0 con 0x7f94e40b6c00 2022-06-29T16:07:14.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.049+0000 7f94e3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94e413f3d0 0x7f94e413be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:14.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.049+0000 7f94e3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94e413f3d0 0x7f94e413be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52720/0 (socket says 172.21.15.60:52720) 2022-06-29T16:07:14.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.049+0000 7f94e3fff700 1 -- 172.21.15.60:0/4166115959 learned_addr learned my addr 172.21.15.60:0/4166115959 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:14.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.049+0000 7f94e932c700 1 --2- 172.21.15.60:0/4166115959 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94e4143820 0x7f94e413c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:14.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.049+0000 7f94e8b2b700 1 --2- 172.21.15.60:0/4166115959 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94e40b6c00 0x7f94e409b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:14.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.053+0000 7f94e3fff700 1 -- 172.21.15.60:0/4166115959 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94e40b6c00 msgr2=0x7f94e409b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:14.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.053+0000 7f94e3fff700 1 --2- 172.21.15.60:0/4166115959 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94e40b6c00 0x7f94e409b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:14.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.053+0000 7f94e3fff700 1 -- 172.21.15.60:0/4166115959 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94e4143820 msgr2=0x7f94e413c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:14.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.053+0000 7f94e3fff700 1 --2- 172.21.15.60:0/4166115959 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94e4143820 0x7f94e413c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:14.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.053+0000 7f94e3fff700 1 -- 172.21.15.60:0/4166115959 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94cc00b180 con 0x7f94e413f3d0 2022-06-29T16:07:14.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.053+0000 7f94e932c700 1 --2- 172.21.15.60:0/4166115959 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94e4143820 0x7f94e413c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:07:14.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.053+0000 7f94e3fff700 1 --2- 172.21.15.60:0/4166115959 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94e413f3d0 0x7f94e413be30 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 rx=0x7f94cc00c000 tx=0x7f94cc006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:14.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.053+0000 7f94e1ffb700 1 -- 172.21.15.60:0/4166115959 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94cc010780 con 0x7f94e413f3d0 2022-06-29T16:07:14.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.053+0000 7f94e1ffb700 1 -- 172.21.15.60:0/4166115959 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94cc0108e0 con 0x7f94e413f3d0 2022-06-29T16:07:14.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.053+0000 7f94ead8f700 1 -- 172.21.15.60:0/4166115959 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94d4009470 con 0x7f94e413f3d0 2022-06-29T16:07:14.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.053+0000 7f94e1ffb700 1 -- 172.21.15.60:0/4166115959 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94cc010c40 con 0x7f94e413f3d0 2022-06-29T16:07:14.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.053+0000 7f94ead8f700 1 -- 172.21.15.60:0/4166115959 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94e40b4350 con 0x7f94e413f3d0 2022-06-29T16:07:14.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.053+0000 7f94ead8f700 1 -- 172.21.15.60:0/4166115959 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94e4065260 con 0x7f94e413f3d0 2022-06-29T16:07:14.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.053+0000 7f94e1ffb700 1 -- 172.21.15.60:0/4166115959 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f94cc01c460 con 0x7f94e413f3d0 2022-06-29T16:07:14.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.053+0000 7f94e1ffb700 1 --2- 172.21.15.60:0/4166115959 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f94bc0539e0 0x7f94bc055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:14.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.053+0000 7f94e1ffb700 1 -- 172.21.15.60:0/4166115959 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f94cc06c100 con 0x7f94e413f3d0 2022-06-29T16:07:14.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.053+0000 7f94e8b2b700 1 --2- 172.21.15.60:0/4166115959 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f94bc0539e0 0x7f94bc055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:14.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.057+0000 7f94e8b2b700 1 --2- 172.21.15.60:0/4166115959 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f94bc0539e0 0x7f94bc055ea0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7f94d401b000 tx=0x7f94d4016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:14.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.057+0000 7f94e1ffb700 1 -- 172.21.15.60:0/4166115959 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f94e4065260 con 0x7f94e413f3d0 2022-06-29T16:07:14.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.225+0000 7f94ead8f700 1 -- 172.21.15.60:0/4166115959 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94e4065260 con 0x7f94e413f3d0 2022-06-29T16:07:14.230 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.225+0000 7f94e1ffb700 1 -- 172.21.15.60:0/4166115959 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+504 (secure 0 0 0) 0x7f94e4065260 con 0x7f94e413f3d0 2022-06-29T16:07:14.231 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:14.231 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":58,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[]} 2022-06-29T16:07:14.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.229+0000 7f94ead8f700 1 -- 172.21.15.60:0/4166115959 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f94bc0539e0 msgr2=0x7f94bc055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:14.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.229+0000 7f94ead8f700 1 --2- 172.21.15.60:0/4166115959 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f94bc0539e0 0x7f94bc055ea0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7f94d401b000 tx=0x7f94d4016000).stop 2022-06-29T16:07:14.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.229+0000 7f94ead8f700 1 -- 172.21.15.60:0/4166115959 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94e413f3d0 msgr2=0x7f94e413be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:14.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.229+0000 7f94ead8f700 1 --2- 172.21.15.60:0/4166115959 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94e413f3d0 0x7f94e413be30 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 rx=0x7f94cc00c000 tx=0x7f94cc006df0).stop 2022-06-29T16:07:14.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.229+0000 7f94ead8f700 1 -- 172.21.15.60:0/4166115959 shutdown_connections 2022-06-29T16:07:14.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.229+0000 7f94ead8f700 1 --2- 172.21.15.60:0/4166115959 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f94bc0539e0 0x7f94bc055ea0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:14.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.229+0000 7f94ead8f700 1 --2- 172.21.15.60:0/4166115959 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94e4143820 0x7f94e413c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:14.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.229+0000 7f94ead8f700 1 --2- 172.21.15.60:0/4166115959 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f94e413f3d0 0x7f94e413be30 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:14.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.229+0000 7f94ead8f700 1 --2- 172.21.15.60:0/4166115959 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f94e40b6c00 0x7f94e409b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:14.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.229+0000 7f94ead8f700 1 -- 172.21.15.60:0/4166115959 >> 172.21.15.60:0/4166115959 conn(0x7f94e40643e0 msgr2=0x7f94e413cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:14.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.229+0000 7f94ead8f700 1 -- 172.21.15.60:0/4166115959 shutdown_connections 2022-06-29T16:07:14.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.229+0000 7f94ead8f700 1 -- 172.21.15.60:0/4166115959 wait complete. 2022-06-29T16:07:14.236 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 58 2022-06-29T16:07:14.244 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:07:14.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.341+0000 7f0188c15700 1 Processor -- start 2022-06-29T16:07:14.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.341+0000 7f0188c15700 1 -- start start 2022-06-29T16:07:14.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.341+0000 7f0188c15700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f018405d070 0x7f018412b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:14.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.341+0000 7f0188c15700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f018412bbc0 0x7f01841431f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:14.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.341+0000 7f0188c15700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01840ae9a0 con 0x7f0184066b20 2022-06-29T16:07:14.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.341+0000 7f0188c15700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0184143b60 con 0x7f018412bbc0 2022-06-29T16:07:14.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.341+0000 7f0188c15700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01840aced0 con 0x7f018405d070 2022-06-29T16:07:14.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.341+0000 7f0183fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f018412bbc0 0x7f01841431f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:14.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.341+0000 7f01837fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f018405d070 0x7f018412b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:14.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.341+0000 7f0183fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f018412bbc0 0x7f01841431f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:58758/0 (socket says 172.21.15.60:58758) 2022-06-29T16:07:14.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.341+0000 7f0182ffd700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f0184066b20 0x7f0184142ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:36830/0 (socket says 172.21.15.60:36830) 2022-06-29T16:07:14.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.341+0000 7f0183fff700 1 -- 172.21.15.60:0/1226081024 learned_addr learned my addr 172.21.15.60:0/1226081024 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:14.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.345+0000 7f0183fff700 1 -- 172.21.15.60:0/1226081024 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f018405d070 msgr2=0x7f018412b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:14.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.345+0000 7f0183fff700 1 --2- 172.21.15.60:0/1226081024 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f018405d070 0x7f018412b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:14.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.345+0000 7f0183fff700 1 -- 172.21.15.60:0/1226081024 >> v1:172.21.15.92:6789/0 conn(0x7f0184066b20 legacy=0x7f0184142ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:14.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.345+0000 7f01827fc700 1 -- 172.21.15.60:0/1226081024 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424171968 0 0) 0x7f01840ae9a0 con 0x7f0184066b20 2022-06-29T16:07:14.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.345+0000 7f0183fff700 1 -- 172.21.15.60:0/1226081024 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f018405d450 con 0x7f018412bbc0 2022-06-29T16:07:14.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.345+0000 7f0183fff700 1 --2- 172.21.15.60:0/1226081024 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f018412bbc0 0x7f01841431f0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7f017c0032f0 tx=0x7f017c009290).ready entity=mon.1 client_cookie=e5f6b8beb9dea096 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:14.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.345+0000 7f01827fc700 1 -- 172.21.15.60:0/1226081024 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f017c00b8a0 con 0x7f018412bbc0 2022-06-29T16:07:14.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.345+0000 7f01827fc700 1 -- 172.21.15.60:0/1226081024 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f017c00ba00 con 0x7f018412bbc0 2022-06-29T16:07:14.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.345+0000 7f01827fc700 1 -- 172.21.15.60:0/1226081024 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f017c00bb70 con 0x7f018412bbc0 2022-06-29T16:07:14.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.345+0000 7f0188c15700 1 -- 172.21.15.60:0/1226081024 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f018412bbc0 msgr2=0x7f01841431f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:14.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.345+0000 7f0188c15700 1 --2- 172.21.15.60:0/1226081024 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f018412bbc0 0x7f01841431f0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7f017c0032f0 tx=0x7f017c009290).stop 2022-06-29T16:07:14.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.345+0000 7f0188c15700 1 -- 172.21.15.60:0/1226081024 shutdown_connections 2022-06-29T16:07:14.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.345+0000 7f0188c15700 1 --2- 172.21.15.60:0/1226081024 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f018412bbc0 0x7f01841431f0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:14.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.345+0000 7f0188c15700 1 --2- 172.21.15.60:0/1226081024 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f018405d070 0x7f018412b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:14.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.345+0000 7f0188c15700 1 -- 172.21.15.60:0/1226081024 >> 172.21.15.60:0/1226081024 conn(0x7f01840b5890 msgr2=0x7f01840b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:14.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.345+0000 7f0188c15700 1 -- 172.21.15.60:0/1226081024 shutdown_connections 2022-06-29T16:07:14.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.345+0000 7f0188c15700 1 -- 172.21.15.60:0/1226081024 wait complete. 2022-06-29T16:07:14.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.349+0000 7f0188c15700 1 Processor -- start 2022-06-29T16:07:14.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.349+0000 7f0188c15700 1 -- start start 2022-06-29T16:07:14.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.349+0000 7f0188c15700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f018405d070 0x7f0184146090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:14.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.349+0000 7f0188c15700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0184066b20 0x7f01840b06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:14.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.349+0000 7f0188c15700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f018412bbc0 0x7f01840b0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:14.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.349+0000 7f0188c15700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018412e250 con 0x7f0184066b20 2022-06-29T16:07:14.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.349+0000 7f0188c15700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018412ca80 con 0x7f018412bbc0 2022-06-29T16:07:14.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.349+0000 7f0188c15700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f018412cd20 con 0x7f018405d070 2022-06-29T16:07:14.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.349+0000 7f0182ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0184066b20 0x7f01840b06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:14.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.349+0000 7f01837fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f018405d070 0x7f0184146090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:14.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.349+0000 7f0182ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0184066b20 0x7f01840b06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52732/0 (socket says 172.21.15.60:52732) 2022-06-29T16:07:14.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.349+0000 7f0182ffd700 1 -- 172.21.15.60:0/3591217956 learned_addr learned my addr 172.21.15.60:0/3591217956 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:14.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.349+0000 7f0183fff700 1 --2- 172.21.15.60:0/3591217956 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f018412bbc0 0x7f01840b0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:14.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.349+0000 7f0182ffd700 1 -- 172.21.15.60:0/3591217956 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f018405d070 msgr2=0x7f0184146090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:14.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.349+0000 7f0182ffd700 1 --2- 172.21.15.60:0/3591217956 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f018405d070 0x7f0184146090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:14.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.349+0000 7f0182ffd700 1 -- 172.21.15.60:0/3591217956 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f018412bbc0 msgr2=0x7f01840b0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:14.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.349+0000 7f0182ffd700 1 --2- 172.21.15.60:0/3591217956 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f018412bbc0 0x7f01840b0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:14.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.349+0000 7f0182ffd700 1 -- 172.21.15.60:0/3591217956 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f017c009530 con 0x7f0184066b20 2022-06-29T16:07:14.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.353+0000 7f0182ffd700 1 --2- 172.21.15.60:0/3591217956 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0184066b20 0x7f01840b06b0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 rx=0x7f016c000ed0 tx=0x7f016c007a90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:14.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.353+0000 7f0180ff9700 1 -- 172.21.15.60:0/3591217956 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f016c00ce40 con 0x7f0184066b20 2022-06-29T16:07:14.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.353+0000 7f0180ff9700 1 -- 172.21.15.60:0/3591217956 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f016c0072e0 con 0x7f0184066b20 2022-06-29T16:07:14.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.353+0000 7f0188c15700 1 -- 172.21.15.60:0/3591217956 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f018412ded0 con 0x7f0184066b20 2022-06-29T16:07:14.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.353+0000 7f0180ff9700 1 -- 172.21.15.60:0/3591217956 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f016c005740 con 0x7f0184066b20 2022-06-29T16:07:14.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.353+0000 7f0188c15700 1 -- 172.21.15.60:0/3591217956 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f018412d4a0 con 0x7f0184066b20 2022-06-29T16:07:14.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.353+0000 7f0188c15700 1 -- 172.21.15.60:0/3591217956 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f018412e060 con 0x7f0184066b20 2022-06-29T16:07:14.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.353+0000 7f0180ff9700 1 -- 172.21.15.60:0/3591217956 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f016c010070 con 0x7f0184066b20 2022-06-29T16:07:14.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.353+0000 7f0180ff9700 1 --2- 172.21.15.60:0/3591217956 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f015c053b00 0x7f015c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:14.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.353+0000 7f0180ff9700 1 -- 172.21.15.60:0/3591217956 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f016c06b3d0 con 0x7f0184066b20 2022-06-29T16:07:14.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.353+0000 7f01837fe700 1 --2- 172.21.15.60:0/3591217956 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f015c053b00 0x7f015c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:14.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.353+0000 7f01837fe700 1 --2- 172.21.15.60:0/3591217956 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f015c053b00 0x7f015c055fc0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7f017400b000 tx=0x7f0174006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:14.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.357+0000 7f0180ff9700 1 -- 172.21.15.60:0/3591217956 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f016c0425d0 con 0x7f0184066b20 2022-06-29T16:07:14.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.525+0000 7f0188c15700 1 -- 172.21.15.60:0/3591217956 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01840573d0 con 0x7f0184066b20 2022-06-29T16:07:14.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.525+0000 7f0180ff9700 1 -- 172.21.15.60:0/3591217956 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+504 (secure 0 0 0) 0x7f016c045bf0 con 0x7f0184066b20 2022-06-29T16:07:14.531 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:14.531 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":58,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[]} 2022-06-29T16:07:14.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.529+0000 7f0188c15700 1 -- 172.21.15.60:0/3591217956 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f015c053b00 msgr2=0x7f015c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:14.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.529+0000 7f0188c15700 1 --2- 172.21.15.60:0/3591217956 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f015c053b00 0x7f015c055fc0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7f017400b000 tx=0x7f0174006df0).stop 2022-06-29T16:07:14.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.529+0000 7f0188c15700 1 -- 172.21.15.60:0/3591217956 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0184066b20 msgr2=0x7f01840b06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:14.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.529+0000 7f0188c15700 1 --2- 172.21.15.60:0/3591217956 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0184066b20 0x7f01840b06b0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 rx=0x7f016c000ed0 tx=0x7f016c007a90).stop 2022-06-29T16:07:14.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.529+0000 7f0188c15700 1 -- 172.21.15.60:0/3591217956 shutdown_connections 2022-06-29T16:07:14.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.529+0000 7f0188c15700 1 --2- 172.21.15.60:0/3591217956 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f015c053b00 0x7f015c055fc0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:14.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.529+0000 7f0188c15700 1 --2- 172.21.15.60:0/3591217956 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f018412bbc0 0x7f01840b0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:14.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.529+0000 7f0188c15700 1 --2- 172.21.15.60:0/3591217956 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0184066b20 0x7f01840b06b0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:14.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.529+0000 7f0188c15700 1 --2- 172.21.15.60:0/3591217956 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f018405d070 0x7f0184146090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:14.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.529+0000 7f0188c15700 1 -- 172.21.15.60:0/3591217956 >> 172.21.15.60:0/3591217956 conn(0x7f01840b5890 msgr2=0x7f018405c510 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:14.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.529+0000 7f0188c15700 1 -- 172.21.15.60:0/3591217956 shutdown_connections 2022-06-29T16:07:14.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.529+0000 7f0188c15700 1 -- 172.21.15.60:0/3591217956 wait complete. 2022-06-29T16:07:14.536 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 58 2022-06-29T16:07:14.544 INFO:tasks.ceph.mds.a:Restarting daemon 2022-06-29T16:07:14.545 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-06-29T16:07:14.545 INFO:tasks.ceph.mds.b:Restarting daemon 2022-06-29T16:07:14.546 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-06-29T16:07:14.548 INFO:tasks.ceph.mds.a:Started 2022-06-29T16:07:14.548 DEBUG:teuthology.parallel:result is None 2022-06-29T16:07:14.549 INFO:tasks.ceph.mds.b:Started 2022-06-29T16:07:14.550 DEBUG:teuthology.parallel:result is None 2022-06-29T16:07:14.550 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-06-29T16:07:14.620 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2022-06-29T16:07:14.628 INFO:tasks.ceph.mds.b.smithi092.stdout:starting mds.b at 2022-06-29T16:07:14.647 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.641+0000 7ff4c3cf7700 1 Processor -- start 2022-06-29T16:07:14.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.641+0000 7ff4c3cf7700 1 -- start start 2022-06-29T16:07:14.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.645+0000 7ff4c3cf7700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff4bc0b9d70 0x7ff4bc0b3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:14.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.645+0000 7ff4c3cf7700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff4bc05d1f0 0x7ff4bc0b1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:14.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.645+0000 7ff4c3cf7700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4bc0b2ce0 con 0x7ff4bc0b3cc0 2022-06-29T16:07:14.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.645+0000 7ff4c3cf7700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4bc0b4d10 con 0x7ff4bc05d1f0 2022-06-29T16:07:14.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.645+0000 7ff4c3cf7700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4bc0ba150 con 0x7ff4bc0b9d70 2022-06-29T16:07:14.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.645+0000 7ff4c1292700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff4bc05d1f0 0x7ff4bc0b1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:14.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.645+0000 7ff4c1a93700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff4bc0b9d70 0x7ff4bc0b3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:14.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.645+0000 7ff4c1a93700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff4bc0b9d70 0x7ff4bc0b3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:58786/0 (socket says 172.21.15.60:58786) 2022-06-29T16:07:14.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.645+0000 7ff4c1292700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff4bc05d1f0 0x7ff4bc0b1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52758/0 (socket says 172.21.15.60:52758) 2022-06-29T16:07:14.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.645+0000 7ff4c1a93700 1 -- 172.21.15.60:0/1071170198 learned_addr learned my addr 172.21.15.60:0/1071170198 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:14.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.645+0000 7ff4c2294700 1 --1- 172.21.15.60:0/1071170198 >> v1:172.21.15.92:6789/0 conn(0x7ff4bc0b3cc0 0x7ff4bc1420b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:36862/0 (socket says 172.21.15.60:36862) 2022-06-29T16:07:14.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.645+0000 7ff4c1292700 1 -- 172.21.15.60:0/1071170198 >> v2:172.21.15.92:3300/0 conn(0x7ff4bc0b9d70 msgr2=0x7ff4bc0b3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:14.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.645+0000 7ff4c1292700 1 --2- 172.21.15.60:0/1071170198 >> v2:172.21.15.92:3300/0 conn(0x7ff4bc0b9d70 0x7ff4bc0b3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:14.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.645+0000 7ff4c1292700 1 -- 172.21.15.60:0/1071170198 >> v1:172.21.15.92:6789/0 conn(0x7ff4bc0b3cc0 legacy=0x7ff4bc1420b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:14.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.645+0000 7ff4c0a91700 1 -- 172.21.15.60:0/1071170198 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3299451317 0 0) 0x7ff4bc0b2ce0 con 0x7ff4bc0b3cc0 2022-06-29T16:07:14.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.645+0000 7ff4c1292700 1 -- 172.21.15.60:0/1071170198 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4bc0b4a70 con 0x7ff4bc05d1f0 2022-06-29T16:07:14.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.645+0000 7ff4c1292700 1 --2- 172.21.15.60:0/1071170198 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4bc05d1f0 0x7ff4bc0b1b50 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 rx=0x7ff4ac002f70 tx=0x7ff4ac006df0).ready entity=mon.0 client_cookie=1d5232ff09db9a70 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:14.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.645+0000 7ff4c0a91700 1 -- 172.21.15.60:0/1071170198 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff4ac00d570 con 0x7ff4bc05d1f0 2022-06-29T16:07:14.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.645+0000 7ff4c0a91700 1 -- 172.21.15.60:0/1071170198 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4ac000e90 con 0x7ff4bc05d1f0 2022-06-29T16:07:14.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.645+0000 7ff4c0a91700 1 -- 172.21.15.60:0/1071170198 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff4ac00d8c0 con 0x7ff4bc05d1f0 2022-06-29T16:07:14.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.645+0000 7ff4c3cf7700 1 -- 172.21.15.60:0/1071170198 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4bc05d1f0 msgr2=0x7ff4bc0b1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:14.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.645+0000 7ff4c3cf7700 1 --2- 172.21.15.60:0/1071170198 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4bc05d1f0 0x7ff4bc0b1b50 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 rx=0x7ff4ac002f70 tx=0x7ff4ac006df0).stop 2022-06-29T16:07:14.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.645+0000 7ff4c3cf7700 1 -- 172.21.15.60:0/1071170198 shutdown_connections 2022-06-29T16:07:14.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.645+0000 7ff4c3cf7700 1 --2- 172.21.15.60:0/1071170198 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4bc05d1f0 0x7ff4bc0b1b50 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:14.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.645+0000 7ff4c3cf7700 1 --2- 172.21.15.60:0/1071170198 >> v2:172.21.15.92:3300/0 conn(0x7ff4bc0b9d70 0x7ff4bc0b3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:14.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.645+0000 7ff4c3cf7700 1 -- 172.21.15.60:0/1071170198 >> 172.21.15.60:0/1071170198 conn(0x7ff4bc062ce0 msgr2=0x7ff4bc0b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:14.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.645+0000 7ff4c3cf7700 1 -- 172.21.15.60:0/1071170198 shutdown_connections 2022-06-29T16:07:14.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.645+0000 7ff4c3cf7700 1 -- 172.21.15.60:0/1071170198 wait complete. 2022-06-29T16:07:14.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.649+0000 7ff4c3cf7700 1 Processor -- start 2022-06-29T16:07:14.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.649+0000 7ff4c3cf7700 1 -- start start 2022-06-29T16:07:14.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.649+0000 7ff4c3cf7700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4bc05d1f0 0x7ff4bc142710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:14.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.649+0000 7ff4c3cf7700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4bc0b3cc0 0x7ff4bc144470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:14.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.649+0000 7ff4c3cf7700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4bc0b9d70 0x7ff4bc155ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:14.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.649+0000 7ff4c3cf7700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4bc142c00 con 0x7ff4bc05d1f0 2022-06-29T16:07:14.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.649+0000 7ff4c3cf7700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4bc144960 con 0x7ff4bc0b9d70 2022-06-29T16:07:14.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.649+0000 7ff4c3cf7700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4bc155390 con 0x7ff4bc0b3cc0 2022-06-29T16:07:14.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.649+0000 7ff4c1a93700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4bc05d1f0 0x7ff4bc142710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:14.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.649+0000 7ff4c1a93700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4bc05d1f0 0x7ff4bc142710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52762/0 (socket says 172.21.15.60:52762) 2022-06-29T16:07:14.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.649+0000 7ff4c1a93700 1 -- 172.21.15.60:0/2307035019 learned_addr learned my addr 172.21.15.60:0/2307035019 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:14.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.649+0000 7ff4c2294700 1 --2- 172.21.15.60:0/2307035019 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4bc0b9d70 0x7ff4bc155ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:14.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.649+0000 7ff4c1292700 1 --2- 172.21.15.60:0/2307035019 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4bc0b3cc0 0x7ff4bc144470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:14.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.649+0000 7ff4c1a93700 1 -- 172.21.15.60:0/2307035019 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4bc0b3cc0 msgr2=0x7ff4bc144470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:14.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.649+0000 7ff4c1a93700 1 --2- 172.21.15.60:0/2307035019 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4bc0b3cc0 0x7ff4bc144470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:14.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.649+0000 7ff4c1a93700 1 -- 172.21.15.60:0/2307035019 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4bc0b9d70 msgr2=0x7ff4bc155ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:14.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.649+0000 7ff4c1a93700 1 --2- 172.21.15.60:0/2307035019 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4bc0b9d70 0x7ff4bc155ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:14.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.649+0000 7ff4c1a93700 1 -- 172.21.15.60:0/2307035019 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ac00b180 con 0x7ff4bc05d1f0 2022-06-29T16:07:14.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.649+0000 7ff4c1a93700 1 --2- 172.21.15.60:0/2307035019 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4bc05d1f0 0x7ff4bc142710 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 rx=0x7ff4b800c000 tx=0x7ff4b8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:14.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.653+0000 7ff4aaffd700 1 -- 172.21.15.60:0/2307035019 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff4b8013610 con 0x7ff4bc05d1f0 2022-06-29T16:07:14.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.653+0000 7ff4aaffd700 1 -- 172.21.15.60:0/2307035019 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b8013770 con 0x7ff4bc05d1f0 2022-06-29T16:07:14.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.653+0000 7ff4c3cf7700 1 -- 172.21.15.60:0/2307035019 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4bc155630 con 0x7ff4bc05d1f0 2022-06-29T16:07:14.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.653+0000 7ff4aaffd700 1 -- 172.21.15.60:0/2307035019 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff4b8013ad0 con 0x7ff4bc05d1f0 2022-06-29T16:07:14.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.653+0000 7ff4c3cf7700 1 -- 172.21.15.60:0/2307035019 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4bc0af4a0 con 0x7ff4bc05d1f0 2022-06-29T16:07:14.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.653+0000 7ff4c3cf7700 1 -- 172.21.15.60:0/2307035019 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4bc1557c0 con 0x7ff4bc05d1f0 2022-06-29T16:07:14.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.653+0000 7ff4aaffd700 1 -- 172.21.15.60:0/2307035019 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff4b8013c30 con 0x7ff4bc05d1f0 2022-06-29T16:07:14.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.653+0000 7ff4aaffd700 1 --2- 172.21.15.60:0/2307035019 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff49c053b00 0x7ff49c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:14.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.653+0000 7ff4aaffd700 1 -- 172.21.15.60:0/2307035019 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5339+0+0 (secure 0 0 0) 0x7ff4b806bba0 con 0x7ff4bc05d1f0 2022-06-29T16:07:14.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.653+0000 7ff4c1292700 1 --2- 172.21.15.60:0/2307035019 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff49c053b00 0x7ff49c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:14.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.657+0000 7ff4c1292700 1 --2- 172.21.15.60:0/2307035019 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff49c053b00 0x7ff49c055fc0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0x7ff4ac01a000 tx=0x7ff4ac014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:14.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.661+0000 7ff4aaffd700 1 -- 172.21.15.60:0/2307035019 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff4b8042da0 con 0x7ff4bc05d1f0 2022-06-29T16:07:14.801 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:14.797+0000 7ff4c3cf7700 1 -- 172.21.15.60:0/2307035019 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7ff4bc0573d0 con 0x7ff4bc05d1f0 2022-06-29T16:07:15.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.637+0000 7ff4aaffd700 1 -- 172.21.15.60:0/2307035019 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v124) v1 ==== 97+0+0 (secure 0 0 0) 0x7ff4b80463c0 con 0x7ff4bc05d1f0 2022-06-29T16:07:15.647 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.641+0000 7ff4c3cf7700 1 -- 172.21.15.60:0/2307035019 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff49c053b00 msgr2=0x7ff49c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:15.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.641+0000 7ff4c3cf7700 1 --2- 172.21.15.60:0/2307035019 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff49c053b00 0x7ff49c055fc0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0x7ff4ac01a000 tx=0x7ff4ac014000).stop 2022-06-29T16:07:15.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.641+0000 7ff4c3cf7700 1 -- 172.21.15.60:0/2307035019 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4bc05d1f0 msgr2=0x7ff4bc142710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:15.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.641+0000 7ff4c3cf7700 1 --2- 172.21.15.60:0/2307035019 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4bc05d1f0 0x7ff4bc142710 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 rx=0x7ff4b800c000 tx=0x7ff4b8008040).stop 2022-06-29T16:07:15.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.641+0000 7ff4c3cf7700 1 -- 172.21.15.60:0/2307035019 shutdown_connections 2022-06-29T16:07:15.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.641+0000 7ff4c3cf7700 1 --2- 172.21.15.60:0/2307035019 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff49c053b00 0x7ff49c055fc0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:15.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.641+0000 7ff4c3cf7700 1 --2- 172.21.15.60:0/2307035019 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4bc0b9d70 0x7ff4bc155ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:15.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.641+0000 7ff4c3cf7700 1 --2- 172.21.15.60:0/2307035019 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4bc0b3cc0 0x7ff4bc144470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:15.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.641+0000 7ff4c3cf7700 1 --2- 172.21.15.60:0/2307035019 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4bc05d1f0 0x7ff4bc142710 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:15.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.641+0000 7ff4c3cf7700 1 -- 172.21.15.60:0/2307035019 >> 172.21.15.60:0/2307035019 conn(0x7ff4bc062ce0 msgr2=0x7ff4bc0b8560 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:15.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.641+0000 7ff4c3cf7700 1 -- 172.21.15.60:0/2307035019 shutdown_connections 2022-06-29T16:07:15.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.641+0000 7ff4c3cf7700 1 -- 172.21.15.60:0/2307035019 wait complete. 2022-06-29T16:07:15.652 INFO:teuthology.orchestra.run.smithi060.stderr: removed all blacklist entries 2022-06-29T16:07:15.663 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-06-29T16:07:15.663 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-06-29T16:07:15.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.765+0000 7f9a4197c700 1 Processor -- start 2022-06-29T16:07:15.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.765+0000 7f9a4197c700 1 -- start start 2022-06-29T16:07:15.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.765+0000 7f9a4197c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9a3c05d070 0x7f9a3c14aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:15.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.765+0000 7f9a4197c700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a3c05d450 con 0x7f9a3c14af80 2022-06-29T16:07:15.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.765+0000 7f9a4197c700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a3c0ad840 con 0x7f9a3c0b2130 2022-06-29T16:07:15.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.765+0000 7f9a4197c700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a3c0adf90 con 0x7f9a3c05d070 2022-06-29T16:07:15.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.765+0000 7f9a4097a700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9a3c05d070 0x7f9a3c14aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:15.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.765+0000 7f9a4097a700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9a3c05d070 0x7f9a3c14aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52772/0 (socket says 172.21.15.60:52772) 2022-06-29T16:07:15.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.765+0000 7f9a4097a700 1 -- 172.21.15.60:0/3323745152 learned_addr learned my addr 172.21.15.60:0/3323745152 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:15.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.765+0000 7f9a4117b700 1 --1- 172.21.15.60:0/3323745152 >> v1:172.21.15.60:6789/0 conn(0x7f9a3c14af80 0x7f9a3c0ad130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60860/0 (socket says 172.21.15.60:60860) 2022-06-29T16:07:15.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.765+0000 7f9a4097a700 1 -- 172.21.15.60:0/3323745152 >> v1:172.21.15.92:6789/0 conn(0x7f9a3c0b2130 legacy=0x7f9a3c0aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:15.781 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.765+0000 7f9a4097a700 1 -- 172.21.15.60:0/3323745152 >> v1:172.21.15.60:6789/0 conn(0x7f9a3c14af80 legacy=0x7f9a3c0ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:15.781 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.765+0000 7f9a4097a700 1 -- 172.21.15.60:0/3323745152 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a3c0b25d0 con 0x7f9a3c05d070 2022-06-29T16:07:15.782 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.765+0000 7f9a4097a700 1 --2- 172.21.15.60:0/3323745152 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a3c05d070 0x7f9a3c14aa90 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 rx=0x7f9a38002f70 tx=0x7f9a38006eb0).ready entity=mon.0 client_cookie=ceb68961a913ff35 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:15.783 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.765+0000 7f9a337fe700 1 -- 172.21.15.60:0/3323745152 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a3800d660 con 0x7f9a3c05d070 2022-06-29T16:07:15.783 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.765+0000 7f9a337fe700 1 -- 172.21.15.60:0/3323745152 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a38000e90 con 0x7f9a3c05d070 2022-06-29T16:07:15.783 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.765+0000 7f9a337fe700 1 -- 172.21.15.60:0/3323745152 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a3800d9f0 con 0x7f9a3c05d070 2022-06-29T16:07:15.783 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.765+0000 7f9a4197c700 1 -- 172.21.15.60:0/3323745152 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a3c05d070 msgr2=0x7f9a3c14aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:15.784 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.765+0000 7f9a4197c700 1 --2- 172.21.15.60:0/3323745152 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a3c05d070 0x7f9a3c14aa90 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 rx=0x7f9a38002f70 tx=0x7f9a38006eb0).stop 2022-06-29T16:07:15.784 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.765+0000 7f9a4197c700 1 -- 172.21.15.60:0/3323745152 shutdown_connections 2022-06-29T16:07:15.784 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.765+0000 7f9a4197c700 1 --2- 172.21.15.60:0/3323745152 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a3c05d070 0x7f9a3c14aa90 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:15.784 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.765+0000 7f9a4197c700 1 -- 172.21.15.60:0/3323745152 >> 172.21.15.60:0/3323745152 conn(0x7f9a3c0b5890 msgr2=0x7f9a3c0b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:15.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.765+0000 7f9a4197c700 1 -- 172.21.15.60:0/3323745152 shutdown_connections 2022-06-29T16:07:15.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.765+0000 7f9a4197c700 1 -- 172.21.15.60:0/3323745152 wait complete. 2022-06-29T16:07:15.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.769+0000 7f9a4197c700 1 Processor -- start 2022-06-29T16:07:15.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.769+0000 7f9a4197c700 1 -- start start 2022-06-29T16:07:15.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.769+0000 7f9a4197c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a3c05d070 0x7f9a3c0663b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:15.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.769+0000 7f9a4197c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a3c0b2130 0x7f9a3c0668a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:15.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.769+0000 7f9a4197c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a3c14af80 0x7f9a3c064210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:15.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.769+0000 7f9a4197c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3c1193b0 con 0x7f9a3c05d070 2022-06-29T16:07:15.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.769+0000 7f9a4197c700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3c119110 con 0x7f9a3c0b2130 2022-06-29T16:07:15.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.769+0000 7f9a4197c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3c065430 con 0x7f9a3c14af80 2022-06-29T16:07:15.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.769+0000 7f9a4097a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a3c05d070 0x7f9a3c0663b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:15.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.769+0000 7f9a4097a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a3c05d070 0x7f9a3c0663b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52776/0 (socket says 172.21.15.60:52776) 2022-06-29T16:07:15.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.769+0000 7f9a4097a700 1 -- 172.21.15.60:0/2538116151 learned_addr learned my addr 172.21.15.60:0/2538116151 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:15.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.769+0000 7f9a4117b700 1 --2- 172.21.15.60:0/2538116151 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a3c14af80 0x7f9a3c064210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:15.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.769+0000 7f9a33fff700 1 --2- 172.21.15.60:0/2538116151 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a3c0b2130 0x7f9a3c0668a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:15.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.769+0000 7f9a4097a700 1 -- 172.21.15.60:0/2538116151 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a3c14af80 msgr2=0x7f9a3c064210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:15.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.769+0000 7f9a4097a700 1 --2- 172.21.15.60:0/2538116151 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a3c14af80 0x7f9a3c064210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:15.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.769+0000 7f9a4097a700 1 -- 172.21.15.60:0/2538116151 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a3c0b2130 msgr2=0x7f9a3c0668a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:15.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.769+0000 7f9a4097a700 1 --2- 172.21.15.60:0/2538116151 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a3c0b2130 0x7f9a3c0668a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:15.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.769+0000 7f9a4097a700 1 -- 172.21.15.60:0/2538116151 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a3800b1e0 con 0x7f9a3c05d070 2022-06-29T16:07:15.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.769+0000 7f9a4117b700 1 --2- 172.21.15.60:0/2538116151 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a3c14af80 0x7f9a3c064210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:15.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.769+0000 7f9a4097a700 1 --2- 172.21.15.60:0/2538116151 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a3c05d070 0x7f9a3c0663b0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 rx=0x7f9a38002f70 tx=0x7f9a38015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:15.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.769+0000 7f9a31ffb700 1 -- 172.21.15.60:0/2538116151 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a38006b40 con 0x7f9a3c05d070 2022-06-29T16:07:15.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.769+0000 7f9a31ffb700 1 -- 172.21.15.60:0/2538116151 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a38011c80 con 0x7f9a3c05d070 2022-06-29T16:07:15.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.769+0000 7f9a31ffb700 1 -- 172.21.15.60:0/2538116151 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a3801e460 con 0x7f9a3c05d070 2022-06-29T16:07:15.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.769+0000 7f9a4197c700 1 -- 172.21.15.60:0/2538116151 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a3c065830 con 0x7f9a3c05d070 2022-06-29T16:07:15.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.769+0000 7f9a4197c700 1 -- 172.21.15.60:0/2538116151 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a3c116f00 con 0x7f9a3c05d070 2022-06-29T16:07:15.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.773+0000 7f9a4197c700 1 -- 172.21.15.60:0/2538116151 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a3c14a350 con 0x7f9a3c05d070 2022-06-29T16:07:15.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.773+0000 7f9a31ffb700 1 -- 172.21.15.60:0/2538116151 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9a38025030 con 0x7f9a3c05d070 2022-06-29T16:07:15.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.773+0000 7f9a31ffb700 1 --2- 172.21.15.60:0/2538116151 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9a1c053aa0 0x7f9a1c055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:15.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.773+0000 7f9a31ffb700 1 -- 172.21.15.60:0/2538116151 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f9a3806cfe0 con 0x7f9a3c05d070 2022-06-29T16:07:15.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.773+0000 7f9a33fff700 1 --2- 172.21.15.60:0/2538116151 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9a1c053aa0 0x7f9a1c055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:15.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.773+0000 7f9a33fff700 1 --2- 172.21.15.60:0/2538116151 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9a1c053aa0 0x7f9a1c055f60 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0x7f9a2c00b000 tx=0x7f9a2c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:15.797 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.781+0000 7f9a31ffb700 1 -- 172.21.15.60:0/2538116151 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9a38044190 con 0x7f9a3c05d070 2022-06-29T16:07:15.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.949+0000 7f9a4197c700 1 -- 172.21.15.60:0/2538116151 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f9a3c0573d0 con 0x7f9a3c05d070 2022-06-29T16:07:15.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.949+0000 7f9a31ffb700 1 -- 172.21.15.60:0/2538116151 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v40) v1 ==== 80+0+4242 (secure 0 0 0) 0x7f9a38023070 con 0x7f9a3c05d070 2022-06-29T16:07:15.956 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:15.956 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:07:15.956 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_dump": [ 2022-06-29T16:07:15.957 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:15.957 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.a", 2022-06-29T16:07:15.957 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxivv7JNhAAnA291UxlB/SfgITOipP00A==", 2022-06-29T16:07:15.957 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:07:15.957 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:07:15.958 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:07:15.958 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:07:15.958 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:07:15.958 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:15.959 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:15.959 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:15.959 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.b", 2022-06-29T16:07:15.959 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxiRSr/OhAAfFQTF2YHvzCv5wXBVCA5WQ==", 2022-06-29T16:07:15.959 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:07:15.960 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:07:15.960 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:07:15.960 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:07:15.960 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:07:15.961 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:15.961 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:15.961 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:15.961 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.0", 2022-06-29T16:07:15.962 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiZYPxOBAAIkweWJlTJIXgxnlagkGdXA==", 2022-06-29T16:07:15.962 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:07:15.962 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:07:15.963 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:07:15.963 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:07:15.963 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:15.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:15.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:15.964 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.1", 2022-06-29T16:07:15.964 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxiS2JGBhAAJn0UfAxZI3IVFyD5QBdb/Q==", 2022-06-29T16:07:15.964 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:07:15.964 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:07:15.965 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:07:15.965 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:07:15.965 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:15.965 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:15.965 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:15.966 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.2", 2022-06-29T16:07:15.966 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxiPhkfEhAApH7r8OUJzW7fZNM6XVv1dg==", 2022-06-29T16:07:15.966 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:07:15.966 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:07:15.967 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:07:15.967 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:07:15.967 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:15.967 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:15.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:15.968 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.3", 2022-06-29T16:07:15.968 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxit/47HhAA1vyU6h7hjPG+op9qP/rVYQ==", 2022-06-29T16:07:15.968 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:07:15.968 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:07:15.969 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:07:15.969 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:07:15.969 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:15.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:15.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:15.970 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.4", 2022-06-29T16:07:15.970 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxiaKzKBxAAbIwCF9ag7ezsNy11hi0BEw==", 2022-06-29T16:07:15.970 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:07:15.970 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:07:15.971 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:07:15.971 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:07:15.971 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:15.971 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:15.971 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:15.972 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.5", 2022-06-29T16:07:15.972 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxiJQV/IBAASBBDb6UFgEtXILe4fcT/xA==", 2022-06-29T16:07:15.972 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:07:15.972 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:07:15.972 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:07:15.973 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:07:15.973 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:15.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:15.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:15.974 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.6", 2022-06-29T16:07:15.974 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxi6D07OBAA2FWiZ+mNK+jFMp/kPZnbQQ==", 2022-06-29T16:07:15.974 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:07:15.974 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:07:15.974 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:07:15.975 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:07:15.975 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:15.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:15.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:15.975 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.7", 2022-06-29T16:07:15.976 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCMdrxinEOcEhAA562ybpzGHNQfWIq8q7zw9w==", 2022-06-29T16:07:15.976 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:07:15.976 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:07:15.976 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:07:15.977 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:07:15.977 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:15.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:15.977 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:15.977 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.0", 2022-06-29T16:07:15.978 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiHXPwBRAAkOOwlYJvIW7dqVFdlXXoqw==", 2022-06-29T16:07:15.978 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:07:15.978 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:07:15.978 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2022-06-29T16:07:15.978 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw pool=cephfs_data" 2022-06-29T16:07:15.979 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:15.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:15.979 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:15.979 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.1", 2022-06-29T16:07:15.980 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxibe5JCRAAGBJbi9t09w0836JqrAT7+w==", 2022-06-29T16:07:15.980 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:07:15.980 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:07:15.980 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2022-06-29T16:07:15.980 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw pool=cephfs_data" 2022-06-29T16:07:15.981 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:15.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:15.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:15.981 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.2", 2022-06-29T16:07:15.981 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiK3dBDRAA7XINPSOPQDfOrUYEpHC+gw==", 2022-06-29T16:07:15.982 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:07:15.982 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:07:15.982 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2022-06-29T16:07:15.982 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw pool=cephfs_data" 2022-06-29T16:07:15.983 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:15.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:15.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:15.983 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.admin", 2022-06-29T16:07:15.983 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxieKVcGBAAnjH80a0qWIArz2qwjYHWdQ==", 2022-06-29T16:07:15.984 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:07:15.984 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T16:07:15.984 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:07:15.984 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:07:15.984 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:07:15.985 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:15.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:15.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:15.985 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.x", 2022-06-29T16:07:15.986 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxi2LZJNBAA3MKb/pXYbQCsoRx8EQfGbw==", 2022-06-29T16:07:15.986 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:07:15.986 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T16:07:15.987 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2022-06-29T16:07:15.987 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:07:15.987 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:15.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:15.988 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:15.988 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.y", 2022-06-29T16:07:15.988 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxi3LmaMBAAN+0mZrdKXqSXOipVTA1drw==", 2022-06-29T16:07:15.988 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:07:15.988 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T16:07:15.989 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2022-06-29T16:07:15.989 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:07:15.989 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:15.989 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:15.990 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:15.990 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:07:15.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.953+0000 7f9a4197c700 1 -- 172.21.15.60:0/2538116151 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9a1c053aa0 msgr2=0x7f9a1c055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:15.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.953+0000 7f9a4197c700 1 --2- 172.21.15.60:0/2538116151 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9a1c053aa0 0x7f9a1c055f60 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0x7f9a2c00b000 tx=0x7f9a2c009040).stop 2022-06-29T16:07:15.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.953+0000 7f9a4197c700 1 -- 172.21.15.60:0/2538116151 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a3c05d070 msgr2=0x7f9a3c0663b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:15.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.953+0000 7f9a4197c700 1 --2- 172.21.15.60:0/2538116151 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a3c05d070 0x7f9a3c0663b0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 rx=0x7f9a38002f70 tx=0x7f9a38015000).stop 2022-06-29T16:07:15.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.953+0000 7f9a4197c700 1 -- 172.21.15.60:0/2538116151 shutdown_connections 2022-06-29T16:07:15.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.953+0000 7f9a4197c700 1 --2- 172.21.15.60:0/2538116151 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9a1c053aa0 0x7f9a1c055f60 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:15.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.953+0000 7f9a4197c700 1 --2- 172.21.15.60:0/2538116151 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a3c14af80 0x7f9a3c064210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:15.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.953+0000 7f9a4197c700 1 --2- 172.21.15.60:0/2538116151 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a3c0b2130 0x7f9a3c0668a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:15.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.953+0000 7f9a4197c700 1 --2- 172.21.15.60:0/2538116151 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a3c05d070 0x7f9a3c0663b0 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:15.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.953+0000 7f9a4197c700 1 -- 172.21.15.60:0/2538116151 >> 172.21.15.60:0/2538116151 conn(0x7f9a3c0b5890 msgr2=0x7f9a3c062450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:15.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.953+0000 7f9a4197c700 1 -- 172.21.15.60:0/2538116151 shutdown_connections 2022-06-29T16:07:15.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:15.953+0000 7f9a4197c700 1 -- 172.21.15.60:0/2538116151 wait complete. 2022-06-29T16:07:15.995 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:07:16.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.097+0000 7f0404a37700 1 Processor -- start 2022-06-29T16:07:16.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.097+0000 7f0404a37700 1 -- start start 2022-06-29T16:07:16.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.101+0000 7f0404a37700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f04000b0830 0x7f040013ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:16.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.101+0000 7f0404a37700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f040005d450 con 0x7f04000b2130 2022-06-29T16:07:16.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.101+0000 7f0404a37700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0400063ba0 con 0x7f040005d070 2022-06-29T16:07:16.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.101+0000 7f0404a37700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0400063e40 con 0x7f04000b0830 2022-06-29T16:07:16.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.101+0000 7f03f6ffd700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f04000b2130 0x7f040013fa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60872/0 (socket says 172.21.15.60:60872) 2022-06-29T16:07:16.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.101+0000 7f03f6ffd700 1 -- 172.21.15.60:0/2222861706 learned_addr learned my addr 172.21.15.60:0/2222861706 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:16.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.101+0000 7f03fffff700 1 --2- 172.21.15.60:0/2222861706 >> v2:172.21.15.60:3300/0 conn(0x7f04000b0830 0x7f040013ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:16.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.101+0000 7f03feffd700 1 -- 172.21.15.60:0/2222861706 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 493449459 0 0) 0x7f040005d450 con 0x7f04000b2130 2022-06-29T16:07:16.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.101+0000 7f03feffd700 1 -- 172.21.15.60:0/2222861706 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03e0003260 con 0x7f04000b2130 2022-06-29T16:07:16.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.101+0000 7f03feffd700 1 -- 172.21.15.60:0/2222861706 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1505450832 0 0) 0x7f0400063ba0 con 0x7f040005d070 2022-06-29T16:07:16.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.101+0000 7f03feffd700 1 -- 172.21.15.60:0/2222861706 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f040005d450 con 0x7f040005d070 2022-06-29T16:07:16.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.101+0000 7f03fffff700 1 -- 172.21.15.60:0/2222861706 >> v1:172.21.15.92:6789/0 conn(0x7f040005d070 legacy=0x7f040013c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:16.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.101+0000 7f03fffff700 1 -- 172.21.15.60:0/2222861706 >> v1:172.21.15.60:6789/0 conn(0x7f04000b2130 legacy=0x7f040013fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:16.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.101+0000 7f03fffff700 1 -- 172.21.15.60:0/2222861706 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0400067650 con 0x7f04000b0830 2022-06-29T16:07:16.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.101+0000 7f03fffff700 1 --2- 172.21.15.60:0/2222861706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04000b0830 0x7f040013ccc0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 rx=0x7f03f80032f0 tx=0x7f03f80091d0).ready entity=mon.0 client_cookie=dd652e5bc5fbee2e server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:16.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.101+0000 7f03feffd700 1 -- 172.21.15.60:0/2222861706 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03f800b600 con 0x7f04000b0830 2022-06-29T16:07:16.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.101+0000 7f03feffd700 1 -- 172.21.15.60:0/2222861706 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03f800b760 con 0x7f04000b0830 2022-06-29T16:07:16.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.101+0000 7f03feffd700 1 -- 172.21.15.60:0/2222861706 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03f800b8d0 con 0x7f04000b0830 2022-06-29T16:07:16.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.101+0000 7f0404a37700 1 -- 172.21.15.60:0/2222861706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04000b0830 msgr2=0x7f040013ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:16.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.101+0000 7f0404a37700 1 --2- 172.21.15.60:0/2222861706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04000b0830 0x7f040013ccc0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 rx=0x7f03f80032f0 tx=0x7f03f80091d0).stop 2022-06-29T16:07:16.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.101+0000 7f0404a37700 1 -- 172.21.15.60:0/2222861706 shutdown_connections 2022-06-29T16:07:16.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.101+0000 7f0404a37700 1 --2- 172.21.15.60:0/2222861706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04000b0830 0x7f040013ccc0 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:16.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.101+0000 7f0404a37700 1 -- 172.21.15.60:0/2222861706 >> 172.21.15.60:0/2222861706 conn(0x7f04000b5890 msgr2=0x7f04000b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:16.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.101+0000 7f0404a37700 1 -- 172.21.15.60:0/2222861706 shutdown_connections 2022-06-29T16:07:16.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.101+0000 7f0404a37700 1 -- 172.21.15.60:0/2222861706 wait complete. 2022-06-29T16:07:16.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.105+0000 7f0404a37700 1 Processor -- start 2022-06-29T16:07:16.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.105+0000 7f0404a37700 1 -- start start 2022-06-29T16:07:16.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.105+0000 7f0404a37700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f040005d070 0x7f04000ae550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:16.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.105+0000 7f0404a37700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04000b0830 0x7f040013a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:16.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.105+0000 7f0404a37700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04000b2130 0x7f04001359e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:16.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.105+0000 7f0404a37700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04000adf00 con 0x7f04000b0830 2022-06-29T16:07:16.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.105+0000 7f0404a37700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0400141170 con 0x7f040005d070 2022-06-29T16:07:16.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.105+0000 7f0404a37700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04000ae0f0 con 0x7f04000b2130 2022-06-29T16:07:16.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.105+0000 7f03f6ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04000b0830 0x7f040013a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:16.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.105+0000 7f03f6ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04000b0830 0x7f040013a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52792/0 (socket says 172.21.15.60:52792) 2022-06-29T16:07:16.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.105+0000 7f03f6ffd700 1 -- 172.21.15.60:0/2197947993 learned_addr learned my addr 172.21.15.60:0/2197947993 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:16.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.105+0000 7f03ff7fe700 1 --2- 172.21.15.60:0/2197947993 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f040005d070 0x7f04000ae550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:16.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.105+0000 7f03fffff700 1 --2- 172.21.15.60:0/2197947993 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04000b2130 0x7f04001359e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:16.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.105+0000 7f03f6ffd700 1 -- 172.21.15.60:0/2197947993 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04000b2130 msgr2=0x7f04001359e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:16.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.109+0000 7f03f6ffd700 1 --2- 172.21.15.60:0/2197947993 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04000b2130 0x7f04001359e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:16.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.109+0000 7f03f6ffd700 1 -- 172.21.15.60:0/2197947993 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f040005d070 msgr2=0x7f04000ae550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:16.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.109+0000 7f03f6ffd700 1 --2- 172.21.15.60:0/2197947993 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f040005d070 0x7f04000ae550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:16.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.109+0000 7f03f6ffd700 1 -- 172.21.15.60:0/2197947993 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f8009470 con 0x7f04000b0830 2022-06-29T16:07:16.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.109+0000 7f03fffff700 1 --2- 172.21.15.60:0/2197947993 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04000b2130 0x7f04001359e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:16.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.109+0000 7f03f6ffd700 1 --2- 172.21.15.60:0/2197947993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04000b0830 0x7f040013a770 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 rx=0x7f03e8001110 tx=0x7f03e8007af0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:16.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.109+0000 7f03fd7fa700 1 -- 172.21.15.60:0/2197947993 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03e80070d0 con 0x7f04000b0830 2022-06-29T16:07:16.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.109+0000 7f03fd7fa700 1 -- 172.21.15.60:0/2197947993 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e8007230 con 0x7f04000b0830 2022-06-29T16:07:16.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.109+0000 7f0404a37700 1 -- 172.21.15.60:0/2197947993 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0400140ed0 con 0x7f04000b0830 2022-06-29T16:07:16.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.109+0000 7f03fd7fa700 1 -- 172.21.15.60:0/2197947993 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03e8005610 con 0x7f04000b0830 2022-06-29T16:07:16.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.109+0000 7f0404a37700 1 -- 172.21.15.60:0/2197947993 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0400140aa0 con 0x7f04000b0830 2022-06-29T16:07:16.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.109+0000 7f0404a37700 1 -- 172.21.15.60:0/2197947993 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0400141b70 con 0x7f04000b0830 2022-06-29T16:07:16.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.109+0000 7f03fd7fa700 1 -- 172.21.15.60:0/2197947993 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f03e8010070 con 0x7f04000b0830 2022-06-29T16:07:16.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.109+0000 7f03fd7fa700 1 --2- 172.21.15.60:0/2197947993 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f03e4053950 0x7f03e4055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:16.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.109+0000 7f03fd7fa700 1 -- 172.21.15.60:0/2197947993 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f03e806b250 con 0x7f04000b0830 2022-06-29T16:07:16.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.109+0000 7f03ff7fe700 1 --2- 172.21.15.60:0/2197947993 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f03e4053950 0x7f03e4055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:16.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.113+0000 7f03ff7fe700 1 --2- 172.21.15.60:0/2197947993 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f03e4053950 0x7f03e4055e10 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7f03f000d000 tx=0x7f03f0007a20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:16.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.117+0000 7f03fd7fa700 1 -- 172.21.15.60:0/2197947993 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0400141b70 con 0x7f04000b0830 2022-06-29T16:07:16.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.245+0000 7f0404a37700 1 -- 172.21.15.60:0/2197947993 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0400141b70 con 0x7f04000b0830 2022-06-29T16:07:16.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.249+0000 7f03fd7fa700 1 -- 172.21.15.60:0/2197947993 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) v1 ==== 74+0+13191 (secure 0 0 0) 0x7f0400141b70 con 0x7f04000b0830 2022-06-29T16:07:16.254 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:16.254 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":124,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:07:15.632872+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:07:16.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.249+0000 7f0404a37700 1 -- 172.21.15.60:0/2197947993 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f03e4053950 msgr2=0x7f03e4055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:16.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.249+0000 7f0404a37700 1 --2- 172.21.15.60:0/2197947993 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f03e4053950 0x7f03e4055e10 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7f03f000d000 tx=0x7f03f0007a20).stop 2022-06-29T16:07:16.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.249+0000 7f0404a37700 1 -- 172.21.15.60:0/2197947993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04000b0830 msgr2=0x7f040013a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:16.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.249+0000 7f0404a37700 1 --2- 172.21.15.60:0/2197947993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04000b0830 0x7f040013a770 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 rx=0x7f03e8001110 tx=0x7f03e8007af0).stop 2022-06-29T16:07:16.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.249+0000 7f0404a37700 1 -- 172.21.15.60:0/2197947993 shutdown_connections 2022-06-29T16:07:16.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.249+0000 7f0404a37700 1 --2- 172.21.15.60:0/2197947993 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f03e4053950 0x7f03e4055e10 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:16.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.249+0000 7f0404a37700 1 --2- 172.21.15.60:0/2197947993 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04000b2130 0x7f04001359e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:16.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.249+0000 7f0404a37700 1 --2- 172.21.15.60:0/2197947993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f04000b0830 0x7f040013a770 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:16.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.249+0000 7f0404a37700 1 --2- 172.21.15.60:0/2197947993 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f040005d070 0x7f04000ae550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:16.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.249+0000 7f0404a37700 1 -- 172.21.15.60:0/2197947993 >> 172.21.15.60:0/2197947993 conn(0x7f04000b5890 msgr2=0x7f0400063640 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:16.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.249+0000 7f0404a37700 1 -- 172.21.15.60:0/2197947993 shutdown_connections 2022-06-29T16:07:16.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.249+0000 7f0404a37700 1 -- 172.21.15.60:0/2197947993 wait complete. 2022-06-29T16:07:16.267 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-06-29T16:07:16.268 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:07:16.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.369+0000 7f4f1ca97700 1 Processor -- start 2022-06-29T16:07:16.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.373+0000 7f4f1ca97700 1 -- start start 2022-06-29T16:07:16.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.373+0000 7f4f1ca97700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4f180aed60 0x7f4f18147890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:16.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.373+0000 7f4f1ca97700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f181471a0 0x7f4f18147d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:16.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.373+0000 7f4f1ca97700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4f180afb10 0x7f4f180b0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:16.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.373+0000 7f4f1ca97700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f180ac8f0 con 0x7f4f180aed60 2022-06-29T16:07:16.382 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.373+0000 7f4f1ca97700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f180acb90 con 0x7f4f180afb10 2022-06-29T16:07:16.382 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.373+0000 7f4f1ca97700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f180b02d0 con 0x7f4f181471a0 2022-06-29T16:07:16.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.373+0000 7f4f1659c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4f180aed60 0x7f4f18147890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:16.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.373+0000 7f4f15d9b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f181471a0 0x7f4f18147d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:16.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.373+0000 7f4f1659c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4f180aed60 0x7f4f18147890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52800/0 (socket says 172.21.15.60:52800) 2022-06-29T16:07:16.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.373+0000 7f4f1659c700 1 -- 172.21.15.60:0/548770350 learned_addr learned my addr 172.21.15.60:0/548770350 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:16.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.373+0000 7f4f16d9d700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4f180afb10 0x7f4f180b0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:16.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.373+0000 7f4f15d9b700 1 -- 172.21.15.60:0/548770350 >> v2:172.21.15.92:3300/0 conn(0x7f4f180afb10 msgr2=0x7f4f180b0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:16.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.373+0000 7f4f15d9b700 1 --2- 172.21.15.60:0/548770350 >> v2:172.21.15.92:3300/0 conn(0x7f4f180afb10 0x7f4f180b0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:16.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.373+0000 7f4f15d9b700 1 -- 172.21.15.60:0/548770350 >> v2:172.21.15.60:3300/0 conn(0x7f4f180aed60 msgr2=0x7f4f18147890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:16.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.373+0000 7f4f15d9b700 1 --2- 172.21.15.60:0/548770350 >> v2:172.21.15.60:3300/0 conn(0x7f4f180aed60 0x7f4f18147890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:16.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.373+0000 7f4f15d9b700 1 -- 172.21.15.60:0/548770350 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f180ac5b0 con 0x7f4f181471a0 2022-06-29T16:07:16.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.373+0000 7f4f15d9b700 1 --2- 172.21.15.60:0/548770350 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f181471a0 0x7f4f18147d80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f4f080032f0 tx=0x7f4f080091d0).ready entity=mon.2 client_cookie=4ad79b09f6ac4f8 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:16.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.373+0000 7f4f1559a700 1 -- 172.21.15.60:0/548770350 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f0800b600 con 0x7f4f181471a0 2022-06-29T16:07:16.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.373+0000 7f4f1559a700 1 -- 172.21.15.60:0/548770350 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f0800b760 con 0x7f4f181471a0 2022-06-29T16:07:16.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.373+0000 7f4f1559a700 1 -- 172.21.15.60:0/548770350 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f0800b8d0 con 0x7f4f181471a0 2022-06-29T16:07:16.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.373+0000 7f4f1ca97700 1 -- 172.21.15.60:0/548770350 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f181471a0 msgr2=0x7f4f18147d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:16.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.373+0000 7f4f1ca97700 1 --2- 172.21.15.60:0/548770350 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f181471a0 0x7f4f18147d80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f4f080032f0 tx=0x7f4f080091d0).stop 2022-06-29T16:07:16.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.373+0000 7f4f1ca97700 1 -- 172.21.15.60:0/548770350 shutdown_connections 2022-06-29T16:07:16.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.373+0000 7f4f1ca97700 1 --2- 172.21.15.60:0/548770350 >> v2:172.21.15.92:3300/0 conn(0x7f4f180afb10 0x7f4f180b0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:16.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.373+0000 7f4f1ca97700 1 --2- 172.21.15.60:0/548770350 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f181471a0 0x7f4f18147d80 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:16.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.373+0000 7f4f1ca97700 1 --2- 172.21.15.60:0/548770350 >> v2:172.21.15.60:3300/0 conn(0x7f4f180aed60 0x7f4f18147890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:16.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.373+0000 7f4f1ca97700 1 -- 172.21.15.60:0/548770350 >> 172.21.15.60:0/548770350 conn(0x7f4f180b4f60 msgr2=0x7f4f180b5370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:16.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.373+0000 7f4f1ca97700 1 -- 172.21.15.60:0/548770350 shutdown_connections 2022-06-29T16:07:16.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.373+0000 7f4f1ca97700 1 -- 172.21.15.60:0/548770350 wait complete. 2022-06-29T16:07:16.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.377+0000 7f4f1ca97700 1 Processor -- start 2022-06-29T16:07:16.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.377+0000 7f4f1ca97700 1 -- start start 2022-06-29T16:07:16.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.377+0000 7f4f1ca97700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f180aed60 0x7f4f1814cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:16.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.377+0000 7f4f1ca97700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f180afb10 0x7f4f18159b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:16.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.377+0000 7f4f1ca97700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f181433f0 0x7f4f1815a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:16.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.377+0000 7f4f1ca97700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1814b620 con 0x7f4f180afb10 2022-06-29T16:07:16.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.377+0000 7f4f1ca97700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f18159780 con 0x7f4f181433f0 2022-06-29T16:07:16.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.377+0000 7f4f1ca97700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f18063f20 con 0x7f4f180aed60 2022-06-29T16:07:16.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.377+0000 7f4f15d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f180afb10 0x7f4f18159b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:16.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.377+0000 7f4f1659c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f180aed60 0x7f4f1814cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:16.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.377+0000 7f4f15d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f180afb10 0x7f4f18159b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52806/0 (socket says 172.21.15.60:52806) 2022-06-29T16:07:16.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.377+0000 7f4f15d9b700 1 -- 172.21.15.60:0/740462116 learned_addr learned my addr 172.21.15.60:0/740462116 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:16.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.377+0000 7f4f16d9d700 1 --2- 172.21.15.60:0/740462116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f181433f0 0x7f4f1815a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:16.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.377+0000 7f4f15d9b700 1 -- 172.21.15.60:0/740462116 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f180aed60 msgr2=0x7f4f1814cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:16.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.377+0000 7f4f15d9b700 1 --2- 172.21.15.60:0/740462116 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f180aed60 0x7f4f1814cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:16.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.377+0000 7f4f15d9b700 1 -- 172.21.15.60:0/740462116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f181433f0 msgr2=0x7f4f1815a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:16.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.377+0000 7f4f15d9b700 1 --2- 172.21.15.60:0/740462116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f181433f0 0x7f4f1815a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:16.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.377+0000 7f4f15d9b700 1 -- 172.21.15.60:0/740462116 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f08009470 con 0x7f4f180afb10 2022-06-29T16:07:16.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.377+0000 7f4f15d9b700 1 --2- 172.21.15.60:0/740462116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f180afb10 0x7f4f18159b40 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 rx=0x7f4f18063a80 tx=0x7f4f08016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:16.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.377+0000 7f4f0f7fe700 1 -- 172.21.15.60:0/740462116 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f0800b600 con 0x7f4f180afb10 2022-06-29T16:07:16.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.377+0000 7f4f0f7fe700 1 -- 172.21.15.60:0/740462116 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f0801f5e0 con 0x7f4f180afb10 2022-06-29T16:07:16.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.377+0000 7f4f1ca97700 1 -- 172.21.15.60:0/740462116 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f180637f0 con 0x7f4f180afb10 2022-06-29T16:07:16.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.377+0000 7f4f0f7fe700 1 -- 172.21.15.60:0/740462116 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f0801f940 con 0x7f4f180afb10 2022-06-29T16:07:16.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.377+0000 7f4f1ca97700 1 -- 172.21.15.60:0/740462116 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f1815a520 con 0x7f4f180afb10 2022-06-29T16:07:16.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.381+0000 7f4f1ca97700 1 -- 172.21.15.60:0/740462116 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f180573d0 con 0x7f4f180afb10 2022-06-29T16:07:16.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.381+0000 7f4f0f7fe700 1 -- 172.21.15.60:0/740462116 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4f08026030 con 0x7f4f180afb10 2022-06-29T16:07:16.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.381+0000 7f4f0f7fe700 1 --2- 172.21.15.60:0/740462116 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4efc053980 0x7f4efc055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:16.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.381+0000 7f4f0f7fe700 1 -- 172.21.15.60:0/740462116 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f4f0806c5b0 con 0x7f4f180afb10 2022-06-29T16:07:16.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.381+0000 7f4f1659c700 1 --2- 172.21.15.60:0/740462116 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4efc053980 0x7f4efc055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:16.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.385+0000 7f4f1659c700 1 --2- 172.21.15.60:0/740462116 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4efc053980 0x7f4efc055e40 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0x7f4f00008000 tx=0x7f4f00008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:16.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.389+0000 7f4f0f7fe700 1 -- 172.21.15.60:0/740462116 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4f08043840 con 0x7f4f180afb10 2022-06-29T16:07:16.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.513+0000 7f4f1ca97700 1 -- 172.21.15.60:0/740462116 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f4f180b5ad0 con 0x7f4f180afb10 2022-06-29T16:07:16.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.513+0000 7f4f0f7fe700 1 -- 172.21.15.60:0/740462116 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v124) v1 ==== 110+0+10 (secure 0 0 0) 0x7f4f080150b0 con 0x7f4f180afb10 2022-06-29T16:07:16.519 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:07:16.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.517+0000 7f4f1ca97700 1 -- 172.21.15.60:0/740462116 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4efc053980 msgr2=0x7f4efc055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:16.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.517+0000 7f4f1ca97700 1 --2- 172.21.15.60:0/740462116 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4efc053980 0x7f4efc055e40 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0x7f4f00008000 tx=0x7f4f00008040).stop 2022-06-29T16:07:16.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.517+0000 7f4f1ca97700 1 -- 172.21.15.60:0/740462116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f180afb10 msgr2=0x7f4f18159b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:16.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.517+0000 7f4f1ca97700 1 --2- 172.21.15.60:0/740462116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f180afb10 0x7f4f18159b40 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 rx=0x7f4f18063a80 tx=0x7f4f08016000).stop 2022-06-29T16:07:16.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.517+0000 7f4f1ca97700 1 -- 172.21.15.60:0/740462116 shutdown_connections 2022-06-29T16:07:16.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.517+0000 7f4f1ca97700 1 --2- 172.21.15.60:0/740462116 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4efc053980 0x7f4efc055e40 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:16.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.517+0000 7f4f1ca97700 1 --2- 172.21.15.60:0/740462116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f181433f0 0x7f4f1815a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:16.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.517+0000 7f4f1ca97700 1 --2- 172.21.15.60:0/740462116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f180afb10 0x7f4f18159b40 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:16.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.517+0000 7f4f1ca97700 1 --2- 172.21.15.60:0/740462116 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f180aed60 0x7f4f1814cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:16.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.517+0000 7f4f1ca97700 1 -- 172.21.15.60:0/740462116 >> 172.21.15.60:0/740462116 conn(0x7f4f180b4f60 msgr2=0x7f4f1805bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:16.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.517+0000 7f4f1ca97700 1 -- 172.21.15.60:0/740462116 shutdown_connections 2022-06-29T16:07:16.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.517+0000 7f4f1ca97700 1 -- 172.21.15.60:0/740462116 wait complete. 2022-06-29T16:07:16.533 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:07:16.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54effff700 1 Processor -- start 2022-06-29T16:07:16.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54effff700 1 -- start start 2022-06-29T16:07:16.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54effff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f54f00aed00 0x7f54f0146810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:16.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54effff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54f005d450 con 0x7f54f005d070 2022-06-29T16:07:16.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54effff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54f00b25d0 con 0x7f54f00b2130 2022-06-29T16:07:16.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54effff700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54f00adf90 con 0x7f54f00aed00 2022-06-29T16:07:16.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54eeffd700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f54f005d070 0x7f54f014aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60898/0 (socket says 172.21.15.60:60898) 2022-06-29T16:07:16.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54eeffd700 1 -- 172.21.15.60:0/1526244539 learned_addr learned my addr 172.21.15.60:0/1526244539 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:16.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54ef7fe700 1 --2- 172.21.15.60:0/1526244539 >> v2:172.21.15.60:3300/0 conn(0x7f54f00aed00 0x7f54f0146810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:16.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54edffb700 1 -- 172.21.15.60:0/1526244539 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 876034788 0 0) 0x7f54f005d450 con 0x7f54f005d070 2022-06-29T16:07:16.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54edffb700 1 -- 172.21.15.60:0/1526244539 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54d0003260 con 0x7f54f005d070 2022-06-29T16:07:16.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54edffb700 1 -- 172.21.15.60:0/1526244539 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2713819100 0 0) 0x7f54d0003260 con 0x7f54f005d070 2022-06-29T16:07:16.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54edffb700 1 -- 172.21.15.60:0/1526244539 >> v2:172.21.15.60:3300/0 conn(0x7f54f00aed00 msgr2=0x7f54f0146810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:16.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54edffb700 1 --2- 172.21.15.60:0/1526244539 >> v2:172.21.15.60:3300/0 conn(0x7f54f00aed00 0x7f54f0146810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:16.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54edffb700 1 -- 172.21.15.60:0/1526244539 >> v1:172.21.15.92:6789/0 conn(0x7f54f00b2130 legacy=0x7f54f0146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:16.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54edffb700 1 -- 172.21.15.60:0/1526244539 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f014a350 con 0x7f54f005d070 2022-06-29T16:07:16.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54edffb700 1 -- 172.21.15.60:0/1526244539 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f54e0001660 con 0x7f54f005d070 2022-06-29T16:07:16.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54edffb700 1 -- 172.21.15.60:0/1526244539 >> v1:172.21.15.60:6789/0 conn(0x7f54f005d070 legacy=0x7f54f014aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:16.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54edffb700 1 --2- 172.21.15.60:0/1526244539 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54d0003750 0x7f54d0003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:16.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54edffb700 1 --2- 172.21.15.60:0/1526244539 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54d0004500 0x7f54d0004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:16.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54edffb700 1 --2- 172.21.15.60:0/1526244539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54d0004e70 0x7f54d000b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:16.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54edffb700 1 -- 172.21.15.60:0/1526244539 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f00adf90 con 0x7f54d0004e70 2022-06-29T16:07:16.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54edffb700 1 -- 172.21.15.60:0/1526244539 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d000bd20 con 0x7f54d0003750 2022-06-29T16:07:16.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54edffb700 1 -- 172.21.15.60:0/1526244539 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54d000bee0 con 0x7f54d0004500 2022-06-29T16:07:16.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54ef7fe700 1 --2- 172.21.15.60:0/1526244539 >> v2:172.21.15.60:3300/0 conn(0x7f54f00aed00 0x7f54f0146810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:07:16.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54eeffd700 1 --2- 172.21.15.60:0/1526244539 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54d0003750 0x7f54d0003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:16.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54ef7fe700 1 --2- 172.21.15.60:0/1526244539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54d0004e70 0x7f54d000b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:16.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54ee7fc700 1 --2- 172.21.15.60:0/1526244539 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54d0004500 0x7f54d0004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:16.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54ef7fe700 1 -- 172.21.15.60:0/1526244539 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54d0004500 msgr2=0x7f54d0004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:16.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54ef7fe700 1 --2- 172.21.15.60:0/1526244539 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54d0004500 0x7f54d0004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:16.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54ef7fe700 1 -- 172.21.15.60:0/1526244539 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54d0003750 msgr2=0x7f54d0003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:16.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54ef7fe700 1 --2- 172.21.15.60:0/1526244539 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54d0003750 0x7f54d0003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:16.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54ef7fe700 1 -- 172.21.15.60:0/1526244539 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d0003260 con 0x7f54d0004e70 2022-06-29T16:07:16.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54ef7fe700 1 -- 172.21.15.60:0/1526244539 reap_dead start 2022-06-29T16:07:16.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54ef7fe700 1 --2- 172.21.15.60:0/1526244539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54d0004e70 0x7f54d000b750 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 rx=0x7f54e800c000 tx=0x7f54e80091d0).ready entity=mon.0 client_cookie=53f682daea394107 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:16.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54edffb700 1 -- 172.21.15.60:0/1526244539 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54e800e920 con 0x7f54d0004e70 2022-06-29T16:07:16.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54edffb700 1 -- 172.21.15.60:0/1526244539 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54e800ea80 con 0x7f54d0004e70 2022-06-29T16:07:16.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54edffb700 1 -- 172.21.15.60:0/1526244539 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54e800ede0 con 0x7f54d0004e70 2022-06-29T16:07:16.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54effff700 1 -- 172.21.15.60:0/1526244539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54d0004e70 msgr2=0x7f54d000b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:16.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54effff700 1 --2- 172.21.15.60:0/1526244539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54d0004e70 0x7f54d000b750 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 rx=0x7f54e800c000 tx=0x7f54e80091d0).stop 2022-06-29T16:07:16.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54effff700 1 -- 172.21.15.60:0/1526244539 shutdown_connections 2022-06-29T16:07:16.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54effff700 1 --2- 172.21.15.60:0/1526244539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54d0004e70 0x7f54d000b750 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:16.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54effff700 1 -- 172.21.15.60:0/1526244539 >> 172.21.15.60:0/1526244539 conn(0x7f54f00b5890 msgr2=0x7f54f00b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:16.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.641+0000 7f54effff700 1 -- 172.21.15.60:0/1526244539 shutdown_connections 2022-06-29T16:07:16.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.645+0000 7f54effff700 1 -- 172.21.15.60:0/1526244539 wait complete. 2022-06-29T16:07:16.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.645+0000 7f54effff700 1 Processor -- start 2022-06-29T16:07:16.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.645+0000 7f54effff700 1 -- start start 2022-06-29T16:07:16.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.645+0000 7f54effff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54d0004e70 0x7f54f0063c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:16.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.645+0000 7f54effff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54f0066800 0x7f54f0159450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:16.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.645+0000 7f54effff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54f0156530 0x7f54f01569b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:16.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.645+0000 7f54effff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f0156ea0 con 0x7f54d0004e70 2022-06-29T16:07:16.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.645+0000 7f54effff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f0155ea0 con 0x7f54f0066800 2022-06-29T16:07:16.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.645+0000 7f54effff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54f0159940 con 0x7f54f0156530 2022-06-29T16:07:16.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.645+0000 7f54eeffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54d0004e70 0x7f54f0063c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:16.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.645+0000 7f54eeffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54d0004e70 0x7f54f0063c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52824/0 (socket says 172.21.15.60:52824) 2022-06-29T16:07:16.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.645+0000 7f54eeffd700 1 -- 172.21.15.60:0/249675349 learned_addr learned my addr 172.21.15.60:0/249675349 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:16.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.645+0000 7f54ee7fc700 1 --2- 172.21.15.60:0/249675349 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54f0066800 0x7f54f0159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:16.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.645+0000 7f54ef7fe700 1 --2- 172.21.15.60:0/249675349 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54f0156530 0x7f54f01569b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:16.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.645+0000 7f54eeffd700 1 -- 172.21.15.60:0/249675349 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54f0156530 msgr2=0x7f54f01569b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:16.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.645+0000 7f54eeffd700 1 --2- 172.21.15.60:0/249675349 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54f0156530 0x7f54f01569b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:16.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.645+0000 7f54eeffd700 1 -- 172.21.15.60:0/249675349 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54f0066800 msgr2=0x7f54f0159450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:16.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.645+0000 7f54eeffd700 1 --2- 172.21.15.60:0/249675349 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54f0066800 0x7f54f0159450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:16.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.645+0000 7f54eeffd700 1 -- 172.21.15.60:0/249675349 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f011a1c0 con 0x7f54d0004e70 2022-06-29T16:07:16.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.645+0000 7f54ee7fc700 1 --2- 172.21.15.60:0/249675349 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54f0066800 0x7f54f0159450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:16.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.645+0000 7f54eeffd700 1 --2- 172.21.15.60:0/249675349 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54d0004e70 0x7f54f0063c80 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 rx=0x7f54e00018c0 tx=0x7f54e0001e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:16.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.645+0000 7f54e7fff700 1 -- 172.21.15.60:0/249675349 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54e0011780 con 0x7f54d0004e70 2022-06-29T16:07:16.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.645+0000 7f54e7fff700 1 -- 172.21.15.60:0/249675349 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54e00118e0 con 0x7f54d0004e70 2022-06-29T16:07:16.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.645+0000 7f54effff700 1 -- 172.21.15.60:0/249675349 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54f0156090 con 0x7f54d0004e70 2022-06-29T16:07:16.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.645+0000 7f54e7fff700 1 -- 172.21.15.60:0/249675349 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54e0011c40 con 0x7f54d0004e70 2022-06-29T16:07:16.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.645+0000 7f54effff700 1 -- 172.21.15.60:0/249675349 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54f0116780 con 0x7f54d0004e70 2022-06-29T16:07:16.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.649+0000 7f54e7fff700 1 -- 172.21.15.60:0/249675349 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f54e001b030 con 0x7f54d0004e70 2022-06-29T16:07:16.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.649+0000 7f54effff700 1 -- 172.21.15.60:0/249675349 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54f011a000 con 0x7f54d0004e70 2022-06-29T16:07:16.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.649+0000 7f54e7fff700 1 --2- 172.21.15.60:0/249675349 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f54d4053a70 0x7f54d4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:16.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.649+0000 7f54e7fff700 1 -- 172.21.15.60:0/249675349 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f54e006b730 con 0x7f54d0004e70 2022-06-29T16:07:16.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.649+0000 7f54ee7fc700 1 --2- 172.21.15.60:0/249675349 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f54d4053a70 0x7f54d4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:16.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.649+0000 7f54ee7fc700 1 --2- 172.21.15.60:0/249675349 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f54d4053a70 0x7f54d4055f30 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7f54d800d000 tx=0x7f54d8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:16.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.657+0000 7f54e7fff700 1 -- 172.21.15.60:0/249675349 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f54e0042980 con 0x7f54d0004e70 2022-06-29T16:07:16.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.785+0000 7f54effff700 1 -- 172.21.15.60:0/249675349 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f54f014a5f0 con 0x7f54d0004e70 2022-06-29T16:07:16.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.785+0000 7f54e7fff700 1 -- 172.21.15.60:0/249675349 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v124) v1 ==== 92+0+10 (secure 0 0 0) 0x7f54e0019030 con 0x7f54d0004e70 2022-06-29T16:07:16.790 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:07:16.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.789+0000 7f54effff700 1 -- 172.21.15.60:0/249675349 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f54d4053a70 msgr2=0x7f54d4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:16.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.789+0000 7f54effff700 1 --2- 172.21.15.60:0/249675349 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f54d4053a70 0x7f54d4055f30 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7f54d800d000 tx=0x7f54d8009040).stop 2022-06-29T16:07:16.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.789+0000 7f54effff700 1 -- 172.21.15.60:0/249675349 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54d0004e70 msgr2=0x7f54f0063c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:16.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.789+0000 7f54effff700 1 --2- 172.21.15.60:0/249675349 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54d0004e70 0x7f54f0063c80 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 rx=0x7f54e00018c0 tx=0x7f54e0001e30).stop 2022-06-29T16:07:16.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.789+0000 7f54effff700 1 -- 172.21.15.60:0/249675349 shutdown_connections 2022-06-29T16:07:16.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.789+0000 7f54effff700 1 --2- 172.21.15.60:0/249675349 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f54d4053a70 0x7f54d4055f30 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:16.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.789+0000 7f54effff700 1 --2- 172.21.15.60:0/249675349 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f54f0156530 0x7f54f01569b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:16.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.789+0000 7f54effff700 1 --2- 172.21.15.60:0/249675349 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f54f0066800 0x7f54f0159450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:16.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.789+0000 7f54effff700 1 --2- 172.21.15.60:0/249675349 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54d0004e70 0x7f54f0063c80 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:16.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.789+0000 7f54effff700 1 -- 172.21.15.60:0/249675349 >> 172.21.15.60:0/249675349 conn(0x7f54f00b5890 msgr2=0x7f54f0144d90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:16.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.789+0000 7f54effff700 1 -- 172.21.15.60:0/249675349 shutdown_connections 2022-06-29T16:07:16.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.789+0000 7f54effff700 1 -- 172.21.15.60:0/249675349 wait complete. 2022-06-29T16:07:16.808 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-06-29T16:07:16.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.909+0000 7f44b53e3700 1 Processor -- start 2022-06-29T16:07:16.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.909+0000 7f44b53e3700 1 -- start start 2022-06-29T16:07:16.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.909+0000 7f44b53e3700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44b013f3d0 0x7f44b0060260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:16.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.909+0000 7f44b53e3700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f44b013f7b0 0x7f44b013fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:16.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.909+0000 7f44b53e3700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44b00b6e20 con 0x7f44b0143820 2022-06-29T16:07:16.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.909+0000 7f44b53e3700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44b0142ef0 con 0x7f44b013f7b0 2022-06-29T16:07:16.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.909+0000 7f44b53e3700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44b00acb90 con 0x7f44b013f3d0 2022-06-29T16:07:16.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.909+0000 7f44af7fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f44b013f7b0 0x7f44b013fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:16.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.909+0000 7f44af7fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f44b013f7b0 0x7f44b013fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52836/0 (socket says 172.21.15.60:52836) 2022-06-29T16:07:16.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.909+0000 7f44af7fe700 1 -- 172.21.15.60:0/3058170622 learned_addr learned my addr 172.21.15.60:0/3058170622 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:16.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.909+0000 7f44aeffd700 1 --2- 172.21.15.60:0/3058170622 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44b013f3d0 0x7f44b0060260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:16.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.909+0000 7f44adffb700 1 -- 172.21.15.60:0/3058170622 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 211073357 0 0) 0x7f44b00b6e20 con 0x7f44b0143820 2022-06-29T16:07:16.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.909+0000 7f44af7fe700 1 -- 172.21.15.60:0/3058170622 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44b013f3d0 msgr2=0x7f44b0060260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:16.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.909+0000 7f44af7fe700 1 --2- 172.21.15.60:0/3058170622 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44b013f3d0 0x7f44b0060260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:16.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.909+0000 7f44af7fe700 1 -- 172.21.15.60:0/3058170622 >> v1:172.21.15.92:6789/0 conn(0x7f44b0143820 legacy=0x7f44b00b6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:16.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.909+0000 7f44af7fe700 1 -- 172.21.15.60:0/3058170622 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b01430e0 con 0x7f44b013f7b0 2022-06-29T16:07:16.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.909+0000 7f44af7fe700 1 --2- 172.21.15.60:0/3058170622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44b013f7b0 0x7f44b013fc00 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 rx=0x7f44a8002f70 tx=0x7f44a8006eb0).ready entity=mon.0 client_cookie=357cd8bdd9009b16 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:16.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.909+0000 7f44adffb700 1 -- 172.21.15.60:0/3058170622 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f44a800d570 con 0x7f44b013f7b0 2022-06-29T16:07:16.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.909+0000 7f44adffb700 1 -- 172.21.15.60:0/3058170622 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44a8000e90 con 0x7f44b013f7b0 2022-06-29T16:07:16.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.909+0000 7f44adffb700 1 -- 172.21.15.60:0/3058170622 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f44a800d8c0 con 0x7f44b013f7b0 2022-06-29T16:07:16.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.913+0000 7f44b53e3700 1 -- 172.21.15.60:0/3058170622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44b013f7b0 msgr2=0x7f44b013fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:16.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.913+0000 7f44b53e3700 1 --2- 172.21.15.60:0/3058170622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44b013f7b0 0x7f44b013fc00 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 rx=0x7f44a8002f70 tx=0x7f44a8006eb0).stop 2022-06-29T16:07:16.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.913+0000 7f44b53e3700 1 -- 172.21.15.60:0/3058170622 shutdown_connections 2022-06-29T16:07:16.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.913+0000 7f44b53e3700 1 --2- 172.21.15.60:0/3058170622 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44b013f7b0 0x7f44b013fc00 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:16.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.913+0000 7f44b53e3700 1 --2- 172.21.15.60:0/3058170622 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44b013f3d0 0x7f44b0060260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:16.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.913+0000 7f44b53e3700 1 -- 172.21.15.60:0/3058170622 >> 172.21.15.60:0/3058170622 conn(0x7f44b00643e0 msgr2=0x7f44b00b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:16.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.913+0000 7f44b53e3700 1 -- 172.21.15.60:0/3058170622 shutdown_connections 2022-06-29T16:07:16.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.913+0000 7f44b53e3700 1 -- 172.21.15.60:0/3058170622 wait complete. 2022-06-29T16:07:16.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.913+0000 7f44b53e3700 1 Processor -- start 2022-06-29T16:07:16.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.913+0000 7f44b53e3700 1 -- start start 2022-06-29T16:07:16.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.913+0000 7f44b53e3700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44b013f3d0 0x7f44b0064ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:16.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.913+0000 7f44b53e3700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44b013f7b0 0x7f44b00b4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:16.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.913+0000 7f44b53e3700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44b0143820 0x7f44b01417c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:16.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.913+0000 7f44b53e3700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44b0138a70 con 0x7f44b013f7b0 2022-06-29T16:07:16.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.913+0000 7f44b53e3700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44b0142d90 con 0x7f44b013f3d0 2022-06-29T16:07:16.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.913+0000 7f44b53e3700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44b009ba40 con 0x7f44b0143820 2022-06-29T16:07:16.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.913+0000 7f44ae7fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44b013f7b0 0x7f44b00b4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:16.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.913+0000 7f44ae7fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44b013f7b0 0x7f44b00b4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52838/0 (socket says 172.21.15.60:52838) 2022-06-29T16:07:16.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.913+0000 7f44ae7fc700 1 -- 172.21.15.60:0/1401199855 learned_addr learned my addr 172.21.15.60:0/1401199855 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:16.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.913+0000 7f44aeffd700 1 --2- 172.21.15.60:0/1401199855 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44b013f3d0 0x7f44b0064ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:16.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.913+0000 7f44af7fe700 1 --2- 172.21.15.60:0/1401199855 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44b0143820 0x7f44b01417c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:16.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.917+0000 7f44ae7fc700 1 -- 172.21.15.60:0/1401199855 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44b0143820 msgr2=0x7f44b01417c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:16.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.917+0000 7f44ae7fc700 1 --2- 172.21.15.60:0/1401199855 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44b0143820 0x7f44b01417c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:16.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.917+0000 7f44ae7fc700 1 -- 172.21.15.60:0/1401199855 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44b013f3d0 msgr2=0x7f44b0064ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:16.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.917+0000 7f44ae7fc700 1 --2- 172.21.15.60:0/1401199855 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44b013f3d0 0x7f44b0064ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:16.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.917+0000 7f44ae7fc700 1 -- 172.21.15.60:0/1401199855 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a800b1e0 con 0x7f44b013f7b0 2022-06-29T16:07:16.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.917+0000 7f44af7fe700 1 --2- 172.21.15.60:0/1401199855 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44b0143820 0x7f44b01417c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:16.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.917+0000 7f44ae7fc700 1 --2- 172.21.15.60:0/1401199855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44b013f7b0 0x7f44b00b4590 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 rx=0x7f4498001250 tx=0x7f44980013f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:16.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.917+0000 7f44a7fff700 1 -- 172.21.15.60:0/1401199855 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f449800e4a0 con 0x7f44b013f7b0 2022-06-29T16:07:16.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.917+0000 7f44a7fff700 1 -- 172.21.15.60:0/1401199855 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f449800e600 con 0x7f44b013f7b0 2022-06-29T16:07:16.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.917+0000 7f44b53e3700 1 -- 172.21.15.60:0/1401199855 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44b0142a80 con 0x7f44b013f7b0 2022-06-29T16:07:16.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.917+0000 7f44a7fff700 1 -- 172.21.15.60:0/1401199855 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f449800e9a0 con 0x7f44b013f7b0 2022-06-29T16:07:16.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.917+0000 7f44b53e3700 1 -- 172.21.15.60:0/1401199855 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44b00b4c20 con 0x7f44b013f7b0 2022-06-29T16:07:16.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.917+0000 7f44b53e3700 1 -- 172.21.15.60:0/1401199855 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44b00573d0 con 0x7f44b013f7b0 2022-06-29T16:07:16.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.917+0000 7f44a7fff700 1 -- 172.21.15.60:0/1401199855 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4498010070 con 0x7f44b013f7b0 2022-06-29T16:07:16.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.917+0000 7f44a7fff700 1 --2- 172.21.15.60:0/1401199855 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4488053a70 0x7f4488055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:16.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.917+0000 7f44a7fff700 1 -- 172.21.15.60:0/1401199855 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f449806af70 con 0x7f44b013f7b0 2022-06-29T16:07:16.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.917+0000 7f44aeffd700 1 --2- 172.21.15.60:0/1401199855 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4488053a70 0x7f4488055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:16.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.921+0000 7f44aeffd700 1 --2- 172.21.15.60:0/1401199855 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4488053a70 0x7f4488055f30 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7f44a000b000 tx=0x7f44a0006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:16.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:16.925+0000 7f44a7fff700 1 -- 172.21.15.60:0/1401199855 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f44980421c0 con 0x7f44b013f7b0 2022-06-29T16:07:17.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.057+0000 7f44b53e3700 1 -- 172.21.15.60:0/1401199855 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f44b0067990 con 0x7f44b013f7b0 2022-06-29T16:07:17.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.057+0000 7f44a7fff700 1 -- 172.21.15.60:0/1401199855 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v124) v1 ==== 84+0+143 (secure 0 0 0) 0x7f44980457e0 con 0x7f44b013f7b0 2022-06-29T16:07:17.063 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:17.063 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2022-06-29T16:07:17.063 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:17.063 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2022-06-29T16:07:17.064 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": "device_health_metrics" 2022-06-29T16:07:17.064 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:17.064 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:17.064 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 2, 2022-06-29T16:07:17.065 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": "rbd" 2022-06-29T16:07:17.065 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:17.065 INFO:teuthology.orchestra.run.smithi060.stdout:] 2022-06-29T16:07:17.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.057+0000 7f44b53e3700 1 -- 172.21.15.60:0/1401199855 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4488053a70 msgr2=0x7f4488055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:17.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.057+0000 7f44b53e3700 1 --2- 172.21.15.60:0/1401199855 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4488053a70 0x7f4488055f30 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7f44a000b000 tx=0x7f44a0006df0).stop 2022-06-29T16:07:17.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.057+0000 7f44b53e3700 1 -- 172.21.15.60:0/1401199855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44b013f7b0 msgr2=0x7f44b00b4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:17.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.057+0000 7f44b53e3700 1 --2- 172.21.15.60:0/1401199855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44b013f7b0 0x7f44b00b4590 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 rx=0x7f4498001250 tx=0x7f44980013f0).stop 2022-06-29T16:07:17.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.057+0000 7f44b53e3700 1 -- 172.21.15.60:0/1401199855 shutdown_connections 2022-06-29T16:07:17.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.057+0000 7f44b53e3700 1 --2- 172.21.15.60:0/1401199855 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4488053a70 0x7f4488055f30 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:17.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.057+0000 7f44b53e3700 1 --2- 172.21.15.60:0/1401199855 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f44b0143820 0x7f44b01417c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:17.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.057+0000 7f44b53e3700 1 --2- 172.21.15.60:0/1401199855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f44b013f7b0 0x7f44b00b4590 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:17.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.057+0000 7f44b53e3700 1 --2- 172.21.15.60:0/1401199855 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44b013f3d0 0x7f44b0064ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:17.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.057+0000 7f44b53e3700 1 -- 172.21.15.60:0/1401199855 >> 172.21.15.60:0/1401199855 conn(0x7f44b00643e0 msgr2=0x7f44b00b5f90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:17.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.061+0000 7f44b53e3700 1 -- 172.21.15.60:0/1401199855 shutdown_connections 2022-06-29T16:07:17.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.061+0000 7f44b53e3700 1 -- 172.21.15.60:0/1401199855 wait complete. 2022-06-29T16:07:17.075 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-06-29T16:07:17.076 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-06-29T16:07:17.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.173+0000 7f58fd401700 1 Processor -- start 2022-06-29T16:07:17.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.173+0000 7f58fd401700 1 -- start start 2022-06-29T16:07:17.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.177+0000 7f58fd401700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f58f80b5eb0 0x7f58f8142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:17.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.177+0000 7f58fd401700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f58f80b2c50 0x7f58f80b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:17.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.177+0000 7f58fd401700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58f80b0cf0 con 0x7f58f80b48b0 2022-06-29T16:07:17.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.177+0000 7f58fd401700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58f80b54e0 con 0x7f58f80b5eb0 2022-06-29T16:07:17.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.177+0000 7f58fd401700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58f80b4d10 con 0x7f58f80b2c50 2022-06-29T16:07:17.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.177+0000 7f58f77fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f58f80b48b0 0x7f58f80b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60934/0 (socket says 172.21.15.60:60934) 2022-06-29T16:07:17.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.177+0000 7f58f77fe700 1 -- 172.21.15.60:0/1998291891 learned_addr learned my addr 172.21.15.60:0/1998291891 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:17.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.177+0000 7f58f7fff700 1 --2- 172.21.15.60:0/1998291891 >> v2:172.21.15.92:3300/0 conn(0x7f58f80b5eb0 0x7f58f8142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:17.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.177+0000 7f58fcc00700 1 --2- 172.21.15.60:0/1998291891 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f58f80b2c50 0x7f58f80b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:17.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.177+0000 7f58f6ffd700 1 -- 172.21.15.60:0/1998291891 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 229007563 0 0) 0x7f58e8001110 con 0x7f58f80b48b0 2022-06-29T16:07:17.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.177+0000 7f58f6ffd700 1 -- 172.21.15.60:0/1998291891 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58d8003260 con 0x7f58f80b48b0 2022-06-29T16:07:17.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.177+0000 7f58f7fff700 1 -- 172.21.15.60:0/1998291891 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f58f80b2c50 msgr2=0x7f58f80b30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:17.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.177+0000 7f58f7fff700 1 --2- 172.21.15.60:0/1998291891 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f58f80b2c50 0x7f58f80b30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:17.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.177+0000 7f58f7fff700 1 -- 172.21.15.60:0/1998291891 >> v1:172.21.15.60:6789/0 conn(0x7f58f80b48b0 legacy=0x7f58f80b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:17.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.177+0000 7f58f7fff700 1 -- 172.21.15.60:0/1998291891 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f80b6290 con 0x7f58f80b5eb0 2022-06-29T16:07:17.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.177+0000 7f58fcc00700 1 --2- 172.21.15.60:0/1998291891 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f58f80b2c50 0x7f58f80b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:07:17.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.177+0000 7f58f7fff700 1 --2- 172.21.15.60:0/1998291891 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f58f80b5eb0 0x7f58f8142a20 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7f58e0002f70 tx=0x7f58e0006df0).ready entity=mon.1 client_cookie=e018d1557a4eb992 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:17.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.177+0000 7f58f6ffd700 1 -- 172.21.15.60:0/1998291891 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f58e000d4a0 con 0x7f58f80b5eb0 2022-06-29T16:07:17.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.177+0000 7f58f6ffd700 1 -- 172.21.15.60:0/1998291891 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58e0000e90 con 0x7f58f80b5eb0 2022-06-29T16:07:17.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.177+0000 7f58f6ffd700 1 -- 172.21.15.60:0/1998291891 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f58e000d7f0 con 0x7f58f80b5eb0 2022-06-29T16:07:17.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.177+0000 7f58fd401700 1 -- 172.21.15.60:0/1998291891 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f58f80b5eb0 msgr2=0x7f58f8142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:17.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.177+0000 7f58fd401700 1 --2- 172.21.15.60:0/1998291891 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f58f80b5eb0 0x7f58f8142a20 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7f58e0002f70 tx=0x7f58e0006df0).stop 2022-06-29T16:07:17.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.177+0000 7f58fd401700 1 -- 172.21.15.60:0/1998291891 shutdown_connections 2022-06-29T16:07:17.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.177+0000 7f58fd401700 1 --2- 172.21.15.60:0/1998291891 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f58f80b2c50 0x7f58f80b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:17.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.177+0000 7f58fd401700 1 --2- 172.21.15.60:0/1998291891 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f58f80b5eb0 0x7f58f8142a20 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:17.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.177+0000 7f58fd401700 1 -- 172.21.15.60:0/1998291891 >> 172.21.15.60:0/1998291891 conn(0x7f58f8062090 msgr2=0x7f58f8062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:17.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.177+0000 7f58fd401700 1 -- 172.21.15.60:0/1998291891 shutdown_connections 2022-06-29T16:07:17.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.177+0000 7f58fd401700 1 -- 172.21.15.60:0/1998291891 wait complete. 2022-06-29T16:07:17.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.181+0000 7f58fd401700 1 Processor -- start 2022-06-29T16:07:17.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.181+0000 7f58fd401700 1 -- start start 2022-06-29T16:07:17.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.181+0000 7f58fd401700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f58f80b2c50 0x7f58f8140410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:17.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.181+0000 7f58fd401700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f58f80b48b0 0x7f58f81484c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:17.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.181+0000 7f58fd401700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58f80b5eb0 0x7f58f8151f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:17.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.181+0000 7f58fd401700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f80ae370 con 0x7f58f80b5eb0 2022-06-29T16:07:17.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.181+0000 7f58fd401700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f80adf40 con 0x7f58f80b48b0 2022-06-29T16:07:17.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.181+0000 7f58fd401700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58f80af4f0 con 0x7f58f80b2c50 2022-06-29T16:07:17.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.181+0000 7f58fcc00700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58f80b5eb0 0x7f58f8151f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:17.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.181+0000 7f58fcc00700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58f80b5eb0 0x7f58f8151f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52854/0 (socket says 172.21.15.60:52854) 2022-06-29T16:07:17.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.181+0000 7f58fcc00700 1 -- 172.21.15.60:0/1203240917 learned_addr learned my addr 172.21.15.60:0/1203240917 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:17.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.181+0000 7f58f7fff700 1 --2- 172.21.15.60:0/1203240917 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f58f80b2c50 0x7f58f8140410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:17.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.181+0000 7f58f77fe700 1 --2- 172.21.15.60:0/1203240917 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f58f80b48b0 0x7f58f81484c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:17.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.181+0000 7f58fcc00700 1 -- 172.21.15.60:0/1203240917 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f58f80b2c50 msgr2=0x7f58f8140410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:17.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.181+0000 7f58fcc00700 1 --2- 172.21.15.60:0/1203240917 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f58f80b2c50 0x7f58f8140410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:17.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.181+0000 7f58fcc00700 1 -- 172.21.15.60:0/1203240917 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f58f80b48b0 msgr2=0x7f58f81484c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:17.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.181+0000 7f58fcc00700 1 --2- 172.21.15.60:0/1203240917 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f58f80b48b0 0x7f58f81484c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:17.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.181+0000 7f58fcc00700 1 -- 172.21.15.60:0/1203240917 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f0009470 con 0x7f58f80b5eb0 2022-06-29T16:07:17.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.181+0000 7f58fcc00700 1 --2- 172.21.15.60:0/1203240917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58f80b5eb0 0x7f58f8151f40 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 rx=0x7f58f80ae080 tx=0x7f58f00091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:17.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.181+0000 7f58f57fa700 1 -- 172.21.15.60:0/1203240917 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f58f000c850 con 0x7f58f80b5eb0 2022-06-29T16:07:17.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.181+0000 7f58f57fa700 1 -- 172.21.15.60:0/1203240917 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f000c9b0 con 0x7f58f80b5eb0 2022-06-29T16:07:17.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.181+0000 7f58fd401700 1 -- 172.21.15.60:0/1203240917 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58e000b180 con 0x7f58f80b5eb0 2022-06-29T16:07:17.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.181+0000 7f58f57fa700 1 -- 172.21.15.60:0/1203240917 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f58f000cd10 con 0x7f58f80b5eb0 2022-06-29T16:07:17.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.181+0000 7f58fd401700 1 -- 172.21.15.60:0/1203240917 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58f80acc50 con 0x7f58f80b5eb0 2022-06-29T16:07:17.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.185+0000 7f58fd401700 1 -- 172.21.15.60:0/1203240917 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58f8142f10 con 0x7f58f80b5eb0 2022-06-29T16:07:17.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.185+0000 7f58f57fa700 1 -- 172.21.15.60:0/1203240917 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f58f0022070 con 0x7f58f80b5eb0 2022-06-29T16:07:17.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.185+0000 7f58f57fa700 1 --2- 172.21.15.60:0/1203240917 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f58dc053a70 0x7f58dc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:17.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.185+0000 7f58f57fa700 1 -- 172.21.15.60:0/1203240917 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f58f006c410 con 0x7f58f80b5eb0 2022-06-29T16:07:17.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.185+0000 7f58f7fff700 1 --2- 172.21.15.60:0/1203240917 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f58dc053a70 0x7f58dc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:17.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.185+0000 7f58f7fff700 1 --2- 172.21.15.60:0/1203240917 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f58dc053a70 0x7f58dc055f30 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7f58e001a000 tx=0x7f58e0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:17.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.189+0000 7f58f57fa700 1 -- 172.21.15.60:0/1203240917 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f58f00435c0 con 0x7f58f80b5eb0 2022-06-29T16:07:17.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:17.313+0000 7f58fd401700 1 -- 172.21.15.60:0/1203240917 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f58f80af6b0 con 0x7f58f80b5eb0 2022-06-29T16:07:18.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.009+0000 7f58f57fa700 1 -- 172.21.15.60:0/1203240917 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v125) v1 ==== 133+0+0 (secure 0 0 0) 0x7f58f0046be0 con 0x7f58f80b5eb0 2022-06-29T16:07:18.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.017+0000 7f58fd401700 1 -- 172.21.15.60:0/1203240917 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f58dc053a70 msgr2=0x7f58dc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:18.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.017+0000 7f58fd401700 1 --2- 172.21.15.60:0/1203240917 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f58dc053a70 0x7f58dc055f30 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7f58e001a000 tx=0x7f58e0014000).stop 2022-06-29T16:07:18.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.017+0000 7f58fd401700 1 -- 172.21.15.60:0/1203240917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58f80b5eb0 msgr2=0x7f58f8151f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:18.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.017+0000 7f58fd401700 1 --2- 172.21.15.60:0/1203240917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58f80b5eb0 0x7f58f8151f40 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 rx=0x7f58f80ae080 tx=0x7f58f00091d0).stop 2022-06-29T16:07:18.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.017+0000 7f58fd401700 1 -- 172.21.15.60:0/1203240917 shutdown_connections 2022-06-29T16:07:18.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.017+0000 7f58fd401700 1 --2- 172.21.15.60:0/1203240917 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f58dc053a70 0x7f58dc055f30 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:18.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.017+0000 7f58fd401700 1 --2- 172.21.15.60:0/1203240917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58f80b5eb0 0x7f58f8151f40 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:18.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.017+0000 7f58fd401700 1 --2- 172.21.15.60:0/1203240917 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f58f80b48b0 0x7f58f81484c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:18.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.017+0000 7f58fd401700 1 --2- 172.21.15.60:0/1203240917 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f58f80b2c50 0x7f58f8140410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:18.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.017+0000 7f58fd401700 1 -- 172.21.15.60:0/1203240917 >> 172.21.15.60:0/1203240917 conn(0x7f58f8062090 msgr2=0x7f58f80b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:18.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.017+0000 7f58fd401700 1 -- 172.21.15.60:0/1203240917 shutdown_connections 2022-06-29T16:07:18.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.017+0000 7f58fd401700 1 -- 172.21.15.60:0/1203240917 wait complete. 2022-06-29T16:07:18.024 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2022-06-29T16:07:18.036 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-06-29T16:07:18.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.145+0000 7f823759e700 1 Processor -- start 2022-06-29T16:07:18.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.145+0000 7f823759e700 1 -- start start 2022-06-29T16:07:18.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.149+0000 7f823759e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f82380b5eb0 0x7f8238142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:18.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.149+0000 7f823759e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82380b0cf0 con 0x7f82380b48b0 2022-06-29T16:07:18.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.149+0000 7f823759e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82380b4d10 con 0x7f82380b2c50 2022-06-29T16:07:18.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.149+0000 7f823759e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82380b54e0 con 0x7f82380b5eb0 2022-06-29T16:07:18.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.149+0000 7f823659c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f82380b5eb0 0x7f8238142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:18.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.149+0000 7f822dd9b700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f82380b48b0 0x7f82380b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60946/0 (socket says 172.21.15.60:60946) 2022-06-29T16:07:18.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.149+0000 7f822dd9b700 1 -- 172.21.15.60:0/1547091962 learned_addr learned my addr 172.21.15.60:0/1547091962 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:18.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.149+0000 7f823659c700 1 -- 172.21.15.60:0/1547091962 >> v1:172.21.15.92:6789/0 conn(0x7f82380b2c50 legacy=0x7f823814d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:18.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.149+0000 7f823659c700 1 -- 172.21.15.60:0/1547091962 >> v1:172.21.15.60:6789/0 conn(0x7f82380b48b0 legacy=0x7f82380b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:18.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.149+0000 7f8235d9b700 1 -- 172.21.15.60:0/1547091962 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1243841720 0 0) 0x7f8220001110 con 0x7f82380b48b0 2022-06-29T16:07:18.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.149+0000 7f823659c700 1 -- 172.21.15.60:0/1547091962 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82380b6290 con 0x7f82380b5eb0 2022-06-29T16:07:18.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.149+0000 7f823659c700 1 --2- 172.21.15.60:0/1547091962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82380b5eb0 0x7f8238142a20 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 rx=0x7f8230001490 tx=0x7f8228006df0).ready entity=mon.0 client_cookie=89525b7d56a1c188 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:18.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.149+0000 7f8235d9b700 1 -- 172.21.15.60:0/1547091962 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f822800d5c0 con 0x7f82380b5eb0 2022-06-29T16:07:18.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.149+0000 7f8235d9b700 1 -- 172.21.15.60:0/1547091962 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8228000e90 con 0x7f82380b5eb0 2022-06-29T16:07:18.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.149+0000 7f8235d9b700 1 -- 172.21.15.60:0/1547091962 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f822800d910 con 0x7f82380b5eb0 2022-06-29T16:07:18.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.149+0000 7f823759e700 1 -- 172.21.15.60:0/1547091962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82380b5eb0 msgr2=0x7f8238142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:18.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.149+0000 7f823759e700 1 --2- 172.21.15.60:0/1547091962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82380b5eb0 0x7f8238142a20 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 rx=0x7f8230001490 tx=0x7f8228006df0).stop 2022-06-29T16:07:18.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.149+0000 7f823759e700 1 -- 172.21.15.60:0/1547091962 shutdown_connections 2022-06-29T16:07:18.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.149+0000 7f823759e700 1 --2- 172.21.15.60:0/1547091962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82380b5eb0 0x7f8238142a20 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:18.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.149+0000 7f823759e700 1 -- 172.21.15.60:0/1547091962 >> 172.21.15.60:0/1547091962 conn(0x7f8238062090 msgr2=0x7f8238062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:18.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.149+0000 7f823759e700 1 -- 172.21.15.60:0/1547091962 shutdown_connections 2022-06-29T16:07:18.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.149+0000 7f823759e700 1 -- 172.21.15.60:0/1547091962 wait complete. 2022-06-29T16:07:18.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.153+0000 7f823759e700 1 Processor -- start 2022-06-29T16:07:18.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.153+0000 7f823759e700 1 -- start start 2022-06-29T16:07:18.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.153+0000 7f823759e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f82380b2c50 0x7f8238112870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:18.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.153+0000 7f823759e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f82380b48b0 0x7f823813fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:18.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.153+0000 7f823759e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82380b5eb0 0x7f823813ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:18.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.153+0000 7f823759e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8238152d60 con 0x7f82380b5eb0 2022-06-29T16:07:18.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.153+0000 7f823759e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8238152850 con 0x7f82380b48b0 2022-06-29T16:07:18.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.153+0000 7f823759e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8238152ac0 con 0x7f82380b2c50 2022-06-29T16:07:18.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.153+0000 7f8236d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82380b5eb0 0x7f823813ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:18.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.153+0000 7f8236d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82380b5eb0 0x7f823813ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52870/0 (socket says 172.21.15.60:52870) 2022-06-29T16:07:18.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.153+0000 7f8236d9d700 1 -- 172.21.15.60:0/1887045789 learned_addr learned my addr 172.21.15.60:0/1887045789 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:18.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.153+0000 7f823659c700 1 --2- 172.21.15.60:0/1887045789 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f82380b2c50 0x7f8238112870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:18.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.153+0000 7f822dd9b700 1 --2- 172.21.15.60:0/1887045789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f82380b48b0 0x7f823813fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:18.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.157+0000 7f8236d9d700 1 -- 172.21.15.60:0/1887045789 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f82380b2c50 msgr2=0x7f8238112870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:18.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.157+0000 7f8236d9d700 1 --2- 172.21.15.60:0/1887045789 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f82380b2c50 0x7f8238112870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:18.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.157+0000 7f8236d9d700 1 -- 172.21.15.60:0/1887045789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f82380b48b0 msgr2=0x7f823813fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:18.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.157+0000 7f8236d9d700 1 --2- 172.21.15.60:0/1887045789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f82380b48b0 0x7f823813fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:18.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.157+0000 7f8236d9d700 1 -- 172.21.15.60:0/1887045789 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f822800b180 con 0x7f82380b5eb0 2022-06-29T16:07:18.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.157+0000 7f8236d9d700 1 --2- 172.21.15.60:0/1887045789 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82380b5eb0 0x7f823813ff20 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 rx=0x7f82300014d0 tx=0x7f8230007f10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:18.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.157+0000 7f822ffff700 1 -- 172.21.15.60:0/1887045789 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f823000e4a0 con 0x7f82380b5eb0 2022-06-29T16:07:18.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.157+0000 7f822ffff700 1 -- 172.21.15.60:0/1887045789 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f823000e600 con 0x7f82380b5eb0 2022-06-29T16:07:18.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.157+0000 7f823759e700 1 -- 172.21.15.60:0/1887045789 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8238140410 con 0x7f82380b5eb0 2022-06-29T16:07:18.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.157+0000 7f822ffff700 1 -- 172.21.15.60:0/1887045789 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f823000e960 con 0x7f82380b5eb0 2022-06-29T16:07:18.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.157+0000 7f823759e700 1 -- 172.21.15.60:0/1887045789 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82381444b0 con 0x7f82380b5eb0 2022-06-29T16:07:18.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.157+0000 7f823759e700 1 -- 172.21.15.60:0/1887045789 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82380573d0 con 0x7f82380b5eb0 2022-06-29T16:07:18.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.157+0000 7f822ffff700 1 -- 172.21.15.60:0/1887045789 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8230019030 con 0x7f82380b5eb0 2022-06-29T16:07:18.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.161+0000 7f822ffff700 1 --2- 172.21.15.60:0/1887045789 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f821c053a70 0x7f821c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:18.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.161+0000 7f822ffff700 1 -- 172.21.15.60:0/1887045789 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f823006b7a0 con 0x7f82380b5eb0 2022-06-29T16:07:18.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.161+0000 7f823659c700 1 --2- 172.21.15.60:0/1887045789 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f821c053a70 0x7f821c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:18.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.161+0000 7f823659c700 1 --2- 172.21.15.60:0/1887045789 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f821c053a70 0x7f821c055f30 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0x7f822801a000 tx=0x7f8228014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:18.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.165+0000 7f822ffff700 1 -- 172.21.15.60:0/1887045789 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8230042890 con 0x7f82380b5eb0 2022-06-29T16:07:18.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:18.293+0000 7f823759e700 1 -- 172.21.15.60:0/1887045789 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f8238112dc0 con 0x7f82380b5eb0 2022-06-29T16:07:19.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.013+0000 7f822ffff700 1 -- 172.21.15.60:0/1887045789 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v126) v1 ==== 125+0+0 (secure 0 0 0) 0x7f8230017020 con 0x7f82380b5eb0 2022-06-29T16:07:19.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.025+0000 7f823759e700 1 -- 172.21.15.60:0/1887045789 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f821c053a70 msgr2=0x7f821c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:19.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.025+0000 7f823759e700 1 --2- 172.21.15.60:0/1887045789 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f821c053a70 0x7f821c055f30 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0x7f822801a000 tx=0x7f8228014000).stop 2022-06-29T16:07:19.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.025+0000 7f823759e700 1 -- 172.21.15.60:0/1887045789 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82380b5eb0 msgr2=0x7f823813ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:19.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.025+0000 7f823759e700 1 --2- 172.21.15.60:0/1887045789 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82380b5eb0 0x7f823813ff20 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 rx=0x7f82300014d0 tx=0x7f8230007f10).stop 2022-06-29T16:07:19.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.025+0000 7f823759e700 1 -- 172.21.15.60:0/1887045789 shutdown_connections 2022-06-29T16:07:19.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.025+0000 7f823759e700 1 --2- 172.21.15.60:0/1887045789 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f821c053a70 0x7f821c055f30 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:19.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.025+0000 7f823759e700 1 --2- 172.21.15.60:0/1887045789 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f82380b5eb0 0x7f823813ff20 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:19.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.025+0000 7f823759e700 1 --2- 172.21.15.60:0/1887045789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f82380b48b0 0x7f823813fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:19.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.025+0000 7f823759e700 1 --2- 172.21.15.60:0/1887045789 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f82380b2c50 0x7f8238112870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:19.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.025+0000 7f823759e700 1 -- 172.21.15.60:0/1887045789 >> 172.21.15.60:0/1887045789 conn(0x7f8238062090 msgr2=0x7f8238141cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:19.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.025+0000 7f823759e700 1 -- 172.21.15.60:0/1887045789 shutdown_connections 2022-06-29T16:07:19.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.025+0000 7f823759e700 1 -- 172.21.15.60:0/1887045789 wait complete. 2022-06-29T16:07:19.033 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2022-06-29T16:07:19.042 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-06-29T16:07:19.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.145+0000 7f254b31a700 1 Processor -- start 2022-06-29T16:07:19.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.145+0000 7f254b31a700 1 -- start start 2022-06-29T16:07:19.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.145+0000 7f254b31a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25440b9d70 0x7f25440b3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:19.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.145+0000 7f254b31a700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f254405d1f0 0x7f25440b1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:19.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.149+0000 7f254b31a700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25440b2ce0 con 0x7f25440b3cc0 2022-06-29T16:07:19.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.149+0000 7f254b31a700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25440b4d10 con 0x7f254405d1f0 2022-06-29T16:07:19.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.149+0000 7f254b31a700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25440ba150 con 0x7f25440b9d70 2022-06-29T16:07:19.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.149+0000 7f25490b6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25440b9d70 0x7f25440b3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:19.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.149+0000 7f25488b5700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f254405d1f0 0x7f25440b1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:19.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.149+0000 7f25490b6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25440b9d70 0x7f25440b3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:37738/0 (socket says 172.21.15.60:37738) 2022-06-29T16:07:19.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.149+0000 7f25490b6700 1 -- 172.21.15.60:0/2964569433 learned_addr learned my addr 172.21.15.60:0/2964569433 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:19.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.149+0000 7f25488b5700 1 -- 172.21.15.60:0/2964569433 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25440b9d70 msgr2=0x7f25440b3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:19.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.149+0000 7f25488b5700 1 --2- 172.21.15.60:0/2964569433 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25440b9d70 0x7f25440b3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:19.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.149+0000 7f25488b5700 1 -- 172.21.15.60:0/2964569433 >> v1:172.21.15.92:6789/0 conn(0x7f25440b3cc0 legacy=0x7f25441420b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:19.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.149+0000 7f2543fff700 1 -- 172.21.15.60:0/2964569433 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 928489647 0 0) 0x7f25440b2ce0 con 0x7f25440b3cc0 2022-06-29T16:07:19.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.149+0000 7f25488b5700 1 -- 172.21.15.60:0/2964569433 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25440b4a70 con 0x7f254405d1f0 2022-06-29T16:07:19.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.149+0000 7f25490b6700 1 --2- 172.21.15.60:0/2964569433 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25440b9d70 0x7f25440b3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:07:19.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.149+0000 7f25488b5700 1 --2- 172.21.15.60:0/2964569433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f254405d1f0 0x7f25440b1b50 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0x7f2538002f70 tx=0x7f2538006df0).ready entity=mon.1 client_cookie=5b2dadfd3b3e67e7 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:19.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.149+0000 7f2543fff700 1 -- 172.21.15.60:0/2964569433 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f253800d570 con 0x7f254405d1f0 2022-06-29T16:07:19.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.149+0000 7f2543fff700 1 -- 172.21.15.60:0/2964569433 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2538000e90 con 0x7f254405d1f0 2022-06-29T16:07:19.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.149+0000 7f2543fff700 1 -- 172.21.15.60:0/2964569433 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f253800d8c0 con 0x7f254405d1f0 2022-06-29T16:07:19.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.149+0000 7f254b31a700 1 -- 172.21.15.60:0/2964569433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f254405d1f0 msgr2=0x7f25440b1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:19.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.149+0000 7f254b31a700 1 --2- 172.21.15.60:0/2964569433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f254405d1f0 0x7f25440b1b50 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0x7f2538002f70 tx=0x7f2538006df0).stop 2022-06-29T16:07:19.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.149+0000 7f254b31a700 1 -- 172.21.15.60:0/2964569433 shutdown_connections 2022-06-29T16:07:19.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.149+0000 7f254b31a700 1 --2- 172.21.15.60:0/2964569433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f254405d1f0 0x7f25440b1b50 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:19.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.149+0000 7f254b31a700 1 --2- 172.21.15.60:0/2964569433 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25440b9d70 0x7f25440b3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:19.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.149+0000 7f254b31a700 1 -- 172.21.15.60:0/2964569433 >> 172.21.15.60:0/2964569433 conn(0x7f2544062ce0 msgr2=0x7f25440b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:19.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.149+0000 7f254b31a700 1 -- 172.21.15.60:0/2964569433 shutdown_connections 2022-06-29T16:07:19.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.149+0000 7f254b31a700 1 -- 172.21.15.60:0/2964569433 wait complete. 2022-06-29T16:07:19.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.153+0000 7f254b31a700 1 Processor -- start 2022-06-29T16:07:19.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.153+0000 7f254b31a700 1 -- start start 2022-06-29T16:07:19.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.153+0000 7f254b31a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f254405d1f0 0x7f25440ae2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:19.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.153+0000 7f254b31a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25440b3cc0 0x7f25440ada70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:19.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.153+0000 7f254b31a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25440b9d70 0x7f254413c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:19.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.153+0000 7f254b31a700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25440acb60 con 0x7f25440b3cc0 2022-06-29T16:07:19.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.153+0000 7f254b31a700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2544144d20 con 0x7f254405d1f0 2022-06-29T16:07:19.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.153+0000 7f254b31a700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2544146820 con 0x7f25440b9d70 2022-06-29T16:07:19.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.153+0000 7f25488b5700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25440b3cc0 0x7f25440ada70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:19.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.153+0000 7f25490b6700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f254405d1f0 0x7f25440ae2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:19.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.153+0000 7f25498b7700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25440b9d70 0x7f254413c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:19.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.153+0000 7f25488b5700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25440b3cc0 0x7f25440ada70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52886/0 (socket says 172.21.15.60:52886) 2022-06-29T16:07:19.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.153+0000 7f25488b5700 1 -- 172.21.15.60:0/2748464313 learned_addr learned my addr 172.21.15.60:0/2748464313 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:19.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.153+0000 7f25490b6700 1 -- 172.21.15.60:0/2748464313 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25440b9d70 msgr2=0x7f254413c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:19.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.153+0000 7f25490b6700 1 --2- 172.21.15.60:0/2748464313 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25440b9d70 0x7f254413c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:19.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.153+0000 7f25490b6700 1 -- 172.21.15.60:0/2748464313 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25440b3cc0 msgr2=0x7f25440ada70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:19.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.153+0000 7f25490b6700 1 --2- 172.21.15.60:0/2748464313 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25440b3cc0 0x7f25440ada70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:19.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.153+0000 7f25490b6700 1 -- 172.21.15.60:0/2748464313 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f253400b1e0 con 0x7f254405d1f0 2022-06-29T16:07:19.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.153+0000 7f25490b6700 1 --2- 172.21.15.60:0/2748464313 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f254405d1f0 0x7f25440ae2a0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7f253400c000 tx=0x7f2534006eb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:19.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.153+0000 7f25427fc700 1 -- 172.21.15.60:0/2748464313 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2534012600 con 0x7f254405d1f0 2022-06-29T16:07:19.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.157+0000 7f25427fc700 1 -- 172.21.15.60:0/2748464313 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2534012760 con 0x7f254405d1f0 2022-06-29T16:07:19.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.157+0000 7f254b31a700 1 -- 172.21.15.60:0/2748464313 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f253800b180 con 0x7f254405d1f0 2022-06-29T16:07:19.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.157+0000 7f25427fc700 1 -- 172.21.15.60:0/2748464313 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2534012ac0 con 0x7f254405d1f0 2022-06-29T16:07:19.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.157+0000 7f254b31a700 1 -- 172.21.15.60:0/2748464313 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25440adf60 con 0x7f254405d1f0 2022-06-29T16:07:19.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.157+0000 7f25427fc700 1 -- 172.21.15.60:0/2748464313 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2534022070 con 0x7f254405d1f0 2022-06-29T16:07:19.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.157+0000 7f254b31a700 1 -- 172.21.15.60:0/2748464313 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25441447d0 con 0x7f254405d1f0 2022-06-29T16:07:19.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.157+0000 7f25427fc700 1 --2- 172.21.15.60:0/2748464313 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2524053980 0x7f2524055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:19.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.157+0000 7f25427fc700 1 -- 172.21.15.60:0/2748464313 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f253406c110 con 0x7f254405d1f0 2022-06-29T16:07:19.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.157+0000 7f25488b5700 1 --2- 172.21.15.60:0/2748464313 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2524053980 0x7f2524055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:19.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.161+0000 7f25488b5700 1 --2- 172.21.15.60:0/2748464313 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2524053980 0x7f2524055e40 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7f253801a000 tx=0x7f2538014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:19.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.169+0000 7f25427fc700 1 -- 172.21.15.60:0/2748464313 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2534043150 con 0x7f254405d1f0 2022-06-29T16:07:19.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.337+0000 7f254b31a700 1 -- 172.21.15.60:0/2748464313 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f25440573d0 con 0x7f254405d1f0 2022-06-29T16:07:19.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.349+0000 7f25427fc700 1 -- 172.21.15.60:0/2748464313 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 25 and data pool 26 v61) v1 ==== 189+0+0 (secure 0 0 0) 0x7f2534046770 con 0x7f254405d1f0 2022-06-29T16:07:19.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.353+0000 7f254b31a700 1 -- 172.21.15.60:0/2748464313 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2524053980 msgr2=0x7f2524055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:19.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.353+0000 7f254b31a700 1 --2- 172.21.15.60:0/2748464313 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2524053980 0x7f2524055e40 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7f253801a000 tx=0x7f2538014000).stop 2022-06-29T16:07:19.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.353+0000 7f254b31a700 1 -- 172.21.15.60:0/2748464313 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f254405d1f0 msgr2=0x7f25440ae2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:19.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.353+0000 7f254b31a700 1 --2- 172.21.15.60:0/2748464313 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f254405d1f0 0x7f25440ae2a0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7f253400c000 tx=0x7f2534006eb0).stop 2022-06-29T16:07:19.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.357+0000 7f254b31a700 1 -- 172.21.15.60:0/2748464313 shutdown_connections 2022-06-29T16:07:19.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.357+0000 7f254b31a700 1 --2- 172.21.15.60:0/2748464313 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2524053980 0x7f2524055e40 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:19.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.357+0000 7f254b31a700 1 --2- 172.21.15.60:0/2748464313 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25440b9d70 0x7f254413c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:19.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.357+0000 7f254b31a700 1 --2- 172.21.15.60:0/2748464313 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25440b3cc0 0x7f25440ada70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:19.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.357+0000 7f254b31a700 1 --2- 172.21.15.60:0/2748464313 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f254405d1f0 0x7f25440ae2a0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:19.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.357+0000 7f254b31a700 1 -- 172.21.15.60:0/2748464313 >> 172.21.15.60:0/2748464313 conn(0x7f2544062ce0 msgr2=0x7f25440b85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:19.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.357+0000 7f254b31a700 1 -- 172.21.15.60:0/2748464313 shutdown_connections 2022-06-29T16:07:19.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.357+0000 7f254b31a700 1 -- 172.21.15.60:0/2748464313 wait complete. 2022-06-29T16:07:19.365 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 25 and data pool 26 2022-06-29T16:07:19.375 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:07:19.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.461+0000 7f1efffff700 1 Processor -- start 2022-06-29T16:07:19.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.461+0000 7f1efffff700 1 -- start start 2022-06-29T16:07:19.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.461+0000 7f1efffff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1f000b5eb0 0x7f1f00142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:19.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.461+0000 7f1efffff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1f000b2c50 0x7f1f000b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:19.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.461+0000 7f1efffff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f000b0cf0 con 0x7f1f000b48b0 2022-06-29T16:07:19.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.461+0000 7f1efffff700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f000b54e0 con 0x7f1f000b5eb0 2022-06-29T16:07:19.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.461+0000 7f1efffff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f000b4d10 con 0x7f1f000b2c50 2022-06-29T16:07:19.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.461+0000 7f1efeffd700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1f000b5eb0 0x7f1f00142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:19.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.461+0000 7f1eff7fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1f000b2c50 0x7f1f000b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:19.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.461+0000 7f1efe7fc700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f1f000b48b0 0x7f1f000b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:37004/0 (socket says 172.21.15.60:37004) 2022-06-29T16:07:19.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.461+0000 7f1efeffd700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1f000b5eb0 0x7f1f00142a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:58930/0 (socket says 172.21.15.60:58930) 2022-06-29T16:07:19.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.461+0000 7f1efe7fc700 1 -- 172.21.15.60:0/2802759003 learned_addr learned my addr 172.21.15.60:0/2802759003 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:19.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.465+0000 7f1efdffb700 1 -- 172.21.15.60:0/2802759003 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 537662053 0 0) 0x7f1ef0001110 con 0x7f1f000b48b0 2022-06-29T16:07:19.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.465+0000 7f1efdffb700 1 -- 172.21.15.60:0/2802759003 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ee0003260 con 0x7f1f000b48b0 2022-06-29T16:07:19.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.465+0000 7f1eff7fe700 1 -- 172.21.15.60:0/2802759003 >> v2:172.21.15.92:3300/0 conn(0x7f1f000b5eb0 msgr2=0x7f1f00142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:19.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.465+0000 7f1eff7fe700 1 --2- 172.21.15.60:0/2802759003 >> v2:172.21.15.92:3300/0 conn(0x7f1f000b5eb0 0x7f1f00142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:19.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.465+0000 7f1eff7fe700 1 -- 172.21.15.60:0/2802759003 >> v1:172.21.15.92:6789/0 conn(0x7f1f000b48b0 legacy=0x7f1f000b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:19.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.465+0000 7f1eff7fe700 1 -- 172.21.15.60:0/2802759003 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f000b6290 con 0x7f1f000b2c50 2022-06-29T16:07:19.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.465+0000 7f1efeffd700 1 --2- 172.21.15.60:0/2802759003 >> v2:172.21.15.92:3300/0 conn(0x7f1f000b5eb0 0x7f1f00142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:07:19.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.465+0000 7f1eff7fe700 1 --2- 172.21.15.60:0/2802759003 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1f000b2c50 0x7f1f000b30c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f1ef80032f0 tx=0x7f1ef80091d0).ready entity=mon.2 client_cookie=118a9d23007aa8c3 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:19.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.465+0000 7f1efdffb700 1 -- 172.21.15.60:0/2802759003 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ef800b600 con 0x7f1f000b2c50 2022-06-29T16:07:19.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.465+0000 7f1efdffb700 1 -- 172.21.15.60:0/2802759003 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef800b760 con 0x7f1f000b2c50 2022-06-29T16:07:19.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.465+0000 7f1efdffb700 1 -- 172.21.15.60:0/2802759003 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ef800b8d0 con 0x7f1f000b2c50 2022-06-29T16:07:19.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.465+0000 7f1efffff700 1 -- 172.21.15.60:0/2802759003 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1f000b2c50 msgr2=0x7f1f000b30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:19.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.465+0000 7f1efffff700 1 --2- 172.21.15.60:0/2802759003 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1f000b2c50 0x7f1f000b30c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f1ef80032f0 tx=0x7f1ef80091d0).stop 2022-06-29T16:07:19.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.465+0000 7f1efffff700 1 -- 172.21.15.60:0/2802759003 shutdown_connections 2022-06-29T16:07:19.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.465+0000 7f1efffff700 1 --2- 172.21.15.60:0/2802759003 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1f000b2c50 0x7f1f000b30c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:19.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.465+0000 7f1efffff700 1 --2- 172.21.15.60:0/2802759003 >> v2:172.21.15.92:3300/0 conn(0x7f1f000b5eb0 0x7f1f00142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:19.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.465+0000 7f1efffff700 1 -- 172.21.15.60:0/2802759003 >> 172.21.15.60:0/2802759003 conn(0x7f1f00062090 msgr2=0x7f1f00062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:19.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.465+0000 7f1efffff700 1 -- 172.21.15.60:0/2802759003 shutdown_connections 2022-06-29T16:07:19.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.465+0000 7f1efffff700 1 -- 172.21.15.60:0/2802759003 wait complete. 2022-06-29T16:07:19.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.465+0000 7f1efffff700 1 Processor -- start 2022-06-29T16:07:19.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.465+0000 7f1efffff700 1 -- start start 2022-06-29T16:07:19.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.469+0000 7f1efffff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1f000b2c50 0x7f1f00151b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:19.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.469+0000 7f1efffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f000b48b0 0x7f1f00147da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:19.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.469+0000 7f1efffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1f000b5eb0 0x7f1f00152380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:19.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.469+0000 7f1efffff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f001408b0 con 0x7f1f000b48b0 2022-06-29T16:07:19.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.469+0000 7f1efffff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f00140b50 con 0x7f1f000b5eb0 2022-06-29T16:07:19.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.469+0000 7f1efffff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f00148290 con 0x7f1f000b2c50 2022-06-29T16:07:19.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.469+0000 7f1efe7fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f000b48b0 0x7f1f00147da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:19.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.469+0000 7f1efeffd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1f000b2c50 0x7f1f00151b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:19.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.469+0000 7f1efe7fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f000b48b0 0x7f1f00147da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52908/0 (socket says 172.21.15.60:52908) 2022-06-29T16:07:19.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.469+0000 7f1efe7fc700 1 -- 172.21.15.60:0/2497019407 learned_addr learned my addr 172.21.15.60:0/2497019407 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:19.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.469+0000 7f1eff7fe700 1 --2- 172.21.15.60:0/2497019407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1f000b5eb0 0x7f1f00152380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:19.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.469+0000 7f1efe7fc700 1 -- 172.21.15.60:0/2497019407 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1f000b2c50 msgr2=0x7f1f00151b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:19.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.469+0000 7f1efe7fc700 1 --2- 172.21.15.60:0/2497019407 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1f000b2c50 0x7f1f00151b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:19.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.469+0000 7f1efe7fc700 1 -- 172.21.15.60:0/2497019407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1f000b5eb0 msgr2=0x7f1f00152380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:19.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.469+0000 7f1efe7fc700 1 --2- 172.21.15.60:0/2497019407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1f000b5eb0 0x7f1f00152380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:19.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.469+0000 7f1efe7fc700 1 -- 172.21.15.60:0/2497019407 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee800b180 con 0x7f1f000b48b0 2022-06-29T16:07:19.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.469+0000 7f1efe7fc700 1 --2- 172.21.15.60:0/2497019407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f000b48b0 0x7f1f00147da0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 rx=0x7f1ef00012a0 tx=0x7f1ef00078e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:19.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.469+0000 7f1ef7fff700 1 -- 172.21.15.60:0/2497019407 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ef00070d0 con 0x7f1f000b48b0 2022-06-29T16:07:19.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.469+0000 7f1efffff700 1 -- 172.21.15.60:0/2497019407 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ef8009470 con 0x7f1f000b48b0 2022-06-29T16:07:19.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.469+0000 7f1ef7fff700 1 -- 172.21.15.60:0/2497019407 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef0007230 con 0x7f1f000b48b0 2022-06-29T16:07:19.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.469+0000 7f1ef7fff700 1 -- 172.21.15.60:0/2497019407 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ef0007590 con 0x7f1f000b48b0 2022-06-29T16:07:19.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.469+0000 7f1efffff700 1 -- 172.21.15.60:0/2497019407 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f00143ee0 con 0x7f1f000b48b0 2022-06-29T16:07:19.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.469+0000 7f1efffff700 1 -- 172.21.15.60:0/2497019407 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f00140ed0 con 0x7f1f000b48b0 2022-06-29T16:07:19.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.473+0000 7f1ef7fff700 1 -- 172.21.15.60:0/2497019407 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1ef0010070 con 0x7f1f000b48b0 2022-06-29T16:07:19.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.473+0000 7f1ef7fff700 1 --2- 172.21.15.60:0/2497019407 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1ee4053a70 0x7f1ee4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:19.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.473+0000 7f1ef7fff700 1 -- 172.21.15.60:0/2497019407 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1ef006b2e0 con 0x7f1f000b48b0 2022-06-29T16:07:19.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.473+0000 7f1efeffd700 1 --2- 172.21.15.60:0/2497019407 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1ee4053a70 0x7f1ee4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:19.485 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.473+0000 7f1efeffd700 1 --2- 172.21.15.60:0/2497019407 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1ee4053a70 0x7f1ee4055f30 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0x7f1ee800b000 tx=0x7f1ee8006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:19.486 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.481+0000 7f1ef7fff700 1 -- 172.21.15.60:0/2497019407 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1ef0042160 con 0x7f1f000b48b0 2022-06-29T16:07:19.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.609+0000 7f1efffff700 1 -- 172.21.15.60:0/2497019407 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f000b0cf0 con 0x7f1f000b48b0 2022-06-29T16:07:19.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.609+0000 7f1ef7fff700 1 -- 172.21.15.60:0/2497019407 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+16065 (secure 0 0 0) 0x7f1f000b0cf0 con 0x7f1f000b48b0 2022-06-29T16:07:19.615 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:19.615 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":127,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:07:19.341112+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:07:17.372006+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2022-06-29T16:07:18.349364+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:07:19.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.613+0000 7f1efffff700 1 -- 172.21.15.60:0/2497019407 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1ee4053a70 msgr2=0x7f1ee4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:19.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.613+0000 7f1efffff700 1 --2- 172.21.15.60:0/2497019407 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1ee4053a70 0x7f1ee4055f30 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0x7f1ee800b000 tx=0x7f1ee8006df0).stop 2022-06-29T16:07:19.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.613+0000 7f1efffff700 1 -- 172.21.15.60:0/2497019407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f000b48b0 msgr2=0x7f1f00147da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:19.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.613+0000 7f1efffff700 1 --2- 172.21.15.60:0/2497019407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f000b48b0 0x7f1f00147da0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 rx=0x7f1ef00012a0 tx=0x7f1ef00078e0).stop 2022-06-29T16:07:19.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.613+0000 7f1efffff700 1 -- 172.21.15.60:0/2497019407 shutdown_connections 2022-06-29T16:07:19.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.613+0000 7f1efffff700 1 --2- 172.21.15.60:0/2497019407 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1ee4053a70 0x7f1ee4055f30 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:19.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.613+0000 7f1efffff700 1 --2- 172.21.15.60:0/2497019407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1f000b5eb0 0x7f1f00152380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:19.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.613+0000 7f1efffff700 1 --2- 172.21.15.60:0/2497019407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1f000b48b0 0x7f1f00147da0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:19.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.613+0000 7f1efffff700 1 --2- 172.21.15.60:0/2497019407 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1f000b2c50 0x7f1f00151b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:19.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.613+0000 7f1efffff700 1 -- 172.21.15.60:0/2497019407 >> 172.21.15.60:0/2497019407 conn(0x7f1f00062090 msgr2=0x7f1f0005bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:19.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.613+0000 7f1efffff700 1 -- 172.21.15.60:0/2497019407 shutdown_connections 2022-06-29T16:07:19.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.613+0000 7f1efffff700 1 -- 172.21.15.60:0/2497019407 wait complete. 2022-06-29T16:07:19.630 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:07:19.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.733+0000 7f5a15d7d700 1 Processor -- start 2022-06-29T16:07:19.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.733+0000 7f5a15d7d700 1 -- start start 2022-06-29T16:07:19.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.733+0000 7f5a15d7d700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5a100b2130 0x7f5a10146100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:19.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.733+0000 7f5a15d7d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5a100aed00 0x7f5a101465f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:19.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.733+0000 7f5a15d7d700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a1005d450 con 0x7f5a1005d070 2022-06-29T16:07:19.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.733+0000 7f5a15d7d700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a100adf90 con 0x7f5a100b2130 2022-06-29T16:07:19.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.733+0000 7f5a15d7d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a100b25d0 con 0x7f5a100aed00 2022-06-29T16:07:19.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.733+0000 7f5a14d7b700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5a1005d070 0x7f5a1014aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:32768/0 (socket says 172.21.15.60:32768) 2022-06-29T16:07:19.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.737+0000 7f5a14d7b700 1 -- 172.21.15.60:0/3808411014 learned_addr learned my addr 172.21.15.60:0/3808411014 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:19.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.737+0000 7f5a0ffff700 1 --2- 172.21.15.60:0/3808411014 >> v2:172.21.15.92:3300/0 conn(0x7f5a100b2130 0x7f5a10146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:19.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.737+0000 7f5a1557c700 1 --2- 172.21.15.60:0/3808411014 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5a100aed00 0x7f5a101465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:19.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.737+0000 7f5a0f7fe700 1 -- 172.21.15.60:0/3808411014 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3851091868 0 0) 0x7f5a1005d450 con 0x7f5a1005d070 2022-06-29T16:07:19.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.737+0000 7f5a0f7fe700 1 -- 172.21.15.60:0/3808411014 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59f0003260 con 0x7f5a1005d070 2022-06-29T16:07:19.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.737+0000 7f5a1557c700 1 -- 172.21.15.60:0/3808411014 >> v2:172.21.15.92:3300/0 conn(0x7f5a100b2130 msgr2=0x7f5a10146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:19.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.737+0000 7f5a1557c700 1 --2- 172.21.15.60:0/3808411014 >> v2:172.21.15.92:3300/0 conn(0x7f5a100b2130 0x7f5a10146100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:19.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.737+0000 7f5a1557c700 1 -- 172.21.15.60:0/3808411014 >> v1:172.21.15.60:6789/0 conn(0x7f5a1005d070 legacy=0x7f5a1014aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:19.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.737+0000 7f5a1557c700 1 -- 172.21.15.60:0/3808411014 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a1014a350 con 0x7f5a100aed00 2022-06-29T16:07:19.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.737+0000 7f5a1557c700 1 --2- 172.21.15.60:0/3808411014 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5a100aed00 0x7f5a101465f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f5a08002f70 tx=0x7f5a08006df0).ready entity=mon.2 client_cookie=2b09b63be91010ca server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:19.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.737+0000 7f5a0f7fe700 1 -- 172.21.15.60:0/3808411014 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a0800d4a0 con 0x7f5a100aed00 2022-06-29T16:07:19.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.737+0000 7f5a0f7fe700 1 -- 172.21.15.60:0/3808411014 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a08000e90 con 0x7f5a100aed00 2022-06-29T16:07:19.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.737+0000 7f5a0f7fe700 1 -- 172.21.15.60:0/3808411014 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a0800d7f0 con 0x7f5a100aed00 2022-06-29T16:07:19.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.737+0000 7f5a15d7d700 1 -- 172.21.15.60:0/3808411014 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5a100aed00 msgr2=0x7f5a101465f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:19.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.737+0000 7f5a15d7d700 1 --2- 172.21.15.60:0/3808411014 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5a100aed00 0x7f5a101465f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f5a08002f70 tx=0x7f5a08006df0).stop 2022-06-29T16:07:19.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.737+0000 7f5a15d7d700 1 -- 172.21.15.60:0/3808411014 shutdown_connections 2022-06-29T16:07:19.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.737+0000 7f5a15d7d700 1 --2- 172.21.15.60:0/3808411014 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5a100aed00 0x7f5a101465f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:19.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.737+0000 7f5a15d7d700 1 --2- 172.21.15.60:0/3808411014 >> v2:172.21.15.92:3300/0 conn(0x7f5a100b2130 0x7f5a10146100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:19.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.737+0000 7f5a15d7d700 1 -- 172.21.15.60:0/3808411014 >> 172.21.15.60:0/3808411014 conn(0x7f5a100b5890 msgr2=0x7f5a100b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:19.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.737+0000 7f5a15d7d700 1 -- 172.21.15.60:0/3808411014 shutdown_connections 2022-06-29T16:07:19.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.737+0000 7f5a15d7d700 1 -- 172.21.15.60:0/3808411014 wait complete. 2022-06-29T16:07:19.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.737+0000 7f5a15d7d700 1 Processor -- start 2022-06-29T16:07:19.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.737+0000 7f5a15d7d700 1 -- start start 2022-06-29T16:07:19.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.741+0000 7f5a15d7d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a1005d070 0x7f5a10066410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:19.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.741+0000 7f5a15d7d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a100aed00 0x7f5a10142e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:19.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.741+0000 7f5a15d7d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5a100b2130 0x7f5a10143320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:19.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.741+0000 7f5a15d7d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a10156c40 con 0x7f5a100aed00 2022-06-29T16:07:19.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.741+0000 7f5a15d7d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a10156ee0 con 0x7f5a1005d070 2022-06-29T16:07:19.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.741+0000 7f5a15d7d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1014d520 con 0x7f5a100b2130 2022-06-29T16:07:19.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.741+0000 7f5a0ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a100aed00 0x7f5a10142e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:19.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.741+0000 7f5a0ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a100aed00 0x7f5a10142e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52922/0 (socket says 172.21.15.60:52922) 2022-06-29T16:07:19.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.741+0000 7f5a0ffff700 1 -- 172.21.15.60:0/278072198 learned_addr learned my addr 172.21.15.60:0/278072198 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:19.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.741+0000 7f5a1557c700 1 --2- 172.21.15.60:0/278072198 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5a100b2130 0x7f5a10143320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:19.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.741+0000 7f5a14d7b700 1 --2- 172.21.15.60:0/278072198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a1005d070 0x7f5a10066410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:19.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.741+0000 7f5a0ffff700 1 -- 172.21.15.60:0/278072198 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5a100b2130 msgr2=0x7f5a10143320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:19.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.741+0000 7f5a0ffff700 1 --2- 172.21.15.60:0/278072198 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5a100b2130 0x7f5a10143320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:19.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.741+0000 7f5a0ffff700 1 -- 172.21.15.60:0/278072198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a1005d070 msgr2=0x7f5a10066410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:19.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.741+0000 7f5a0ffff700 1 --2- 172.21.15.60:0/278072198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a1005d070 0x7f5a10066410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:19.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.741+0000 7f5a0ffff700 1 -- 172.21.15.60:0/278072198 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0800b180 con 0x7f5a100aed00 2022-06-29T16:07:19.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.741+0000 7f5a0ffff700 1 --2- 172.21.15.60:0/278072198 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a100aed00 0x7f5a10142e30 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 rx=0x7f59f800c000 tx=0x7f59f80091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:19.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.741+0000 7f5a0dffb700 1 -- 172.21.15.60:0/278072198 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59f8012600 con 0x7f5a100aed00 2022-06-29T16:07:19.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.741+0000 7f5a0dffb700 1 -- 172.21.15.60:0/278072198 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f8012760 con 0x7f5a100aed00 2022-06-29T16:07:19.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.741+0000 7f5a15d7d700 1 -- 172.21.15.60:0/278072198 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a10066900 con 0x7f5a100aed00 2022-06-29T16:07:19.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.741+0000 7f5a0dffb700 1 -- 172.21.15.60:0/278072198 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59f8012ac0 con 0x7f5a100aed00 2022-06-29T16:07:19.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.741+0000 7f5a15d7d700 1 -- 172.21.15.60:0/278072198 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a10143810 con 0x7f5a100aed00 2022-06-29T16:07:19.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.745+0000 7f5a15d7d700 1 -- 172.21.15.60:0/278072198 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a10156a80 con 0x7f5a100aed00 2022-06-29T16:07:19.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.745+0000 7f5a0dffb700 1 -- 172.21.15.60:0/278072198 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f59f8022070 con 0x7f5a100aed00 2022-06-29T16:07:19.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.745+0000 7f5a0dffb700 1 --2- 172.21.15.60:0/278072198 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f59f4053a70 0x7f59f4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:19.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.745+0000 7f5a0dffb700 1 -- 172.21.15.60:0/278072198 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f59f806c510 con 0x7f5a100aed00 2022-06-29T16:07:19.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.745+0000 7f5a14d7b700 1 --2- 172.21.15.60:0/278072198 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f59f4053a70 0x7f59f4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:19.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.745+0000 7f5a14d7b700 1 --2- 172.21.15.60:0/278072198 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f59f4053a70 0x7f59f4055f30 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7f5a0000d000 tx=0x7f5a00009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:19.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.749+0000 7f5a0dffb700 1 -- 172.21.15.60:0/278072198 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f59f8043390 con 0x7f5a100aed00 2022-06-29T16:07:19.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.885+0000 7f5a15d7d700 1 -- 172.21.15.60:0/278072198 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a1014a5f0 con 0x7f5a100aed00 2022-06-29T16:07:19.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.885+0000 7f5a0dffb700 1 -- 172.21.15.60:0/278072198 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+16065 (secure 0 0 0) 0x7f59f80469b0 con 0x7f5a100aed00 2022-06-29T16:07:19.891 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:19.891 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":127,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:07:19.341112+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:07:17.372006+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2022-06-29T16:07:18.349364+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:07:19.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.889+0000 7f5a15d7d700 1 -- 172.21.15.60:0/278072198 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f59f4053a70 msgr2=0x7f59f4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:19.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.889+0000 7f5a15d7d700 1 --2- 172.21.15.60:0/278072198 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f59f4053a70 0x7f59f4055f30 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7f5a0000d000 tx=0x7f5a00009040).stop 2022-06-29T16:07:19.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.889+0000 7f5a15d7d700 1 -- 172.21.15.60:0/278072198 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a100aed00 msgr2=0x7f5a10142e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:19.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.889+0000 7f5a15d7d700 1 --2- 172.21.15.60:0/278072198 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a100aed00 0x7f5a10142e30 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 rx=0x7f59f800c000 tx=0x7f59f80091d0).stop 2022-06-29T16:07:19.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.889+0000 7f5a15d7d700 1 -- 172.21.15.60:0/278072198 shutdown_connections 2022-06-29T16:07:19.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.889+0000 7f5a15d7d700 1 --2- 172.21.15.60:0/278072198 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f59f4053a70 0x7f59f4055f30 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:19.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.889+0000 7f5a15d7d700 1 --2- 172.21.15.60:0/278072198 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5a100b2130 0x7f5a10143320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:19.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.889+0000 7f5a15d7d700 1 --2- 172.21.15.60:0/278072198 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a100aed00 0x7f5a10142e30 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:19.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.889+0000 7f5a15d7d700 1 --2- 172.21.15.60:0/278072198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a1005d070 0x7f5a10066410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:19.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.889+0000 7f5a15d7d700 1 -- 172.21.15.60:0/278072198 >> 172.21.15.60:0/278072198 conn(0x7f5a100b5890 msgr2=0x7f5a100aff10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:19.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.889+0000 7f5a15d7d700 1 -- 172.21.15.60:0/278072198 shutdown_connections 2022-06-29T16:07:19.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:19.889+0000 7f5a15d7d700 1 -- 172.21.15.60:0/278072198 wait complete. 2022-06-29T16:07:19.904 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-06-29T16:07:20.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.009+0000 7f93ed876700 1 Processor -- start 2022-06-29T16:07:20.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.009+0000 7f93ed876700 1 -- start start 2022-06-29T16:07:20.016 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.009+0000 7f93ed876700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93e805c460 0x7f93e805c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:20.016 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.009+0000 7f93ed876700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f93e8064560 0x7f93e8146240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:20.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.009+0000 7f93ed876700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f93e813c920 0x7f93e813cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:20.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.009+0000 7f93ed876700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93e805f0b0 con 0x7f93e813c920 2022-06-29T16:07:20.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.009+0000 7f93ed876700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93e80b11f0 con 0x7f93e8064560 2022-06-29T16:07:20.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.009+0000 7f93ed876700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93e80b0f50 con 0x7f93e805c460 2022-06-29T16:07:20.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.009+0000 7f93e77fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f93e813c920 0x7f93e813cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:20.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.009+0000 7f93e77fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f93e813c920 0x7f93e813cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52932/0 (socket says 172.21.15.60:52932) 2022-06-29T16:07:20.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.009+0000 7f93e6ffd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93e805c460 0x7f93e805c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:20.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.009+0000 7f93e77fe700 1 -- 172.21.15.60:0/698890487 learned_addr learned my addr 172.21.15.60:0/698890487 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:20.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.009+0000 7f93e67fc700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f93e8064560 0x7f93e8146240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:20.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.013+0000 7f93e77fe700 1 -- 172.21.15.60:0/698890487 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93e805c460 msgr2=0x7f93e805c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:20.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.013+0000 7f93e77fe700 1 --2- 172.21.15.60:0/698890487 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93e805c460 0x7f93e805c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:20.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.013+0000 7f93e77fe700 1 -- 172.21.15.60:0/698890487 >> v2:172.21.15.92:3300/0 conn(0x7f93e8064560 msgr2=0x7f93e8146240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:20.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.013+0000 7f93e77fe700 1 --2- 172.21.15.60:0/698890487 >> v2:172.21.15.92:3300/0 conn(0x7f93e8064560 0x7f93e8146240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:20.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.013+0000 7f93e77fe700 1 -- 172.21.15.60:0/698890487 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e805f2e0 con 0x7f93e813c920 2022-06-29T16:07:20.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.013+0000 7f93e77fe700 1 --2- 172.21.15.60:0/698890487 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93e813c920 0x7f93e813cd40 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 rx=0x7f93e00032f0 tx=0x7f93e0009290).ready entity=mon.0 client_cookie=ee006a3002790824 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:20.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.013+0000 7f93e5ffb700 1 -- 172.21.15.60:0/698890487 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f93e000b680 con 0x7f93e813c920 2022-06-29T16:07:20.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.013+0000 7f93e5ffb700 1 -- 172.21.15.60:0/698890487 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93e000b7e0 con 0x7f93e813c920 2022-06-29T16:07:20.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.013+0000 7f93e5ffb700 1 -- 172.21.15.60:0/698890487 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f93e000b950 con 0x7f93e813c920 2022-06-29T16:07:20.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.013+0000 7f93ed876700 1 -- 172.21.15.60:0/698890487 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93e813c920 msgr2=0x7f93e813cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:20.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.013+0000 7f93ed876700 1 --2- 172.21.15.60:0/698890487 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93e813c920 0x7f93e813cd40 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 rx=0x7f93e00032f0 tx=0x7f93e0009290).stop 2022-06-29T16:07:20.025 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.013+0000 7f93ed876700 1 -- 172.21.15.60:0/698890487 shutdown_connections 2022-06-29T16:07:20.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.013+0000 7f93ed876700 1 --2- 172.21.15.60:0/698890487 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93e813c920 0x7f93e813cd40 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:20.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.013+0000 7f93ed876700 1 --2- 172.21.15.60:0/698890487 >> v2:172.21.15.92:3300/0 conn(0x7f93e8064560 0x7f93e8146240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:20.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.013+0000 7f93ed876700 1 --2- 172.21.15.60:0/698890487 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93e805c460 0x7f93e805c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:20.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.013+0000 7f93ed876700 1 -- 172.21.15.60:0/698890487 >> 172.21.15.60:0/698890487 conn(0x7f93e8061f30 msgr2=0x7f93e80639e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:20.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.013+0000 7f93ed876700 1 -- 172.21.15.60:0/698890487 shutdown_connections 2022-06-29T16:07:20.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.013+0000 7f93ed876700 1 -- 172.21.15.60:0/698890487 wait complete. 2022-06-29T16:07:20.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.013+0000 7f93ed876700 1 Processor -- start 2022-06-29T16:07:20.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.013+0000 7f93ed876700 1 -- start start 2022-06-29T16:07:20.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.017+0000 7f93ed876700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f93e805c460 0x7f93e80ae510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:20.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.017+0000 7f93ed876700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93e8064560 0x7f93e80b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:20.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.017+0000 7f93ed876700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93e813c920 0x7f93e8141800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:20.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.017+0000 7f93ed876700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e8152870 con 0x7f93e813c920 2022-06-29T16:07:20.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.017+0000 7f93ed876700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e8152b10 con 0x7f93e805c460 2022-06-29T16:07:20.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.017+0000 7f93ed876700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93e80afc40 con 0x7f93e8064560 2022-06-29T16:07:20.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.017+0000 7f93e77fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93e813c920 0x7f93e8141800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:20.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.017+0000 7f93e77fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93e813c920 0x7f93e8141800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52938/0 (socket says 172.21.15.60:52938) 2022-06-29T16:07:20.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.017+0000 7f93e6ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f93e805c460 0x7f93e80ae510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:20.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.017+0000 7f93e77fe700 1 -- 172.21.15.60:0/64862968 learned_addr learned my addr 172.21.15.60:0/64862968 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:20.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.017+0000 7f93e67fc700 1 --2- 172.21.15.60:0/64862968 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93e8064560 0x7f93e80b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:20.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.017+0000 7f93e77fe700 1 -- 172.21.15.60:0/64862968 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93e8064560 msgr2=0x7f93e80b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:20.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.017+0000 7f93e77fe700 1 --2- 172.21.15.60:0/64862968 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93e8064560 0x7f93e80b00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:20.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.017+0000 7f93e77fe700 1 -- 172.21.15.60:0/64862968 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f93e805c460 msgr2=0x7f93e80ae510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:20.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.017+0000 7f93e77fe700 1 --2- 172.21.15.60:0/64862968 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f93e805c460 0x7f93e80ae510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:20.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.017+0000 7f93e77fe700 1 -- 172.21.15.60:0/64862968 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e0009530 con 0x7f93e813c920 2022-06-29T16:07:20.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.017+0000 7f93e67fc700 1 --2- 172.21.15.60:0/64862968 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93e8064560 0x7f93e80b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:20.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.017+0000 7f93e77fe700 1 --2- 172.21.15.60:0/64862968 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93e813c920 0x7f93e8141800 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 rx=0x7f93e00032f0 tx=0x7f93e0017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:20.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.017+0000 7f93d7fff700 1 -- 172.21.15.60:0/64862968 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f93e000b680 con 0x7f93e813c920 2022-06-29T16:07:20.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.017+0000 7f93d7fff700 1 -- 172.21.15.60:0/64862968 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93e000bea0 con 0x7f93e813c920 2022-06-29T16:07:20.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.017+0000 7f93ed876700 1 -- 172.21.15.60:0/64862968 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93e81524b0 con 0x7f93e813c920 2022-06-29T16:07:20.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.017+0000 7f93d7fff700 1 -- 172.21.15.60:0/64862968 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f93e001b600 con 0x7f93e813c920 2022-06-29T16:07:20.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.017+0000 7f93ed876700 1 -- 172.21.15.60:0/64862968 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93e8064990 con 0x7f93e813c920 2022-06-29T16:07:20.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.017+0000 7f93ed876700 1 -- 172.21.15.60:0/64862968 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93e80afe00 con 0x7f93e813c920 2022-06-29T16:07:20.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.021+0000 7f93d7fff700 1 -- 172.21.15.60:0/64862968 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f93e000f070 con 0x7f93e813c920 2022-06-29T16:07:20.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.021+0000 7f93d7fff700 1 --2- 172.21.15.60:0/64862968 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f93c4053a70 0x7f93c4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:20.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.021+0000 7f93d7fff700 1 -- 172.21.15.60:0/64862968 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f93e006d640 con 0x7f93e813c920 2022-06-29T16:07:20.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.021+0000 7f93e6ffd700 1 --2- 172.21.15.60:0/64862968 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f93c4053a70 0x7f93c4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:20.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.021+0000 7f93e6ffd700 1 --2- 172.21.15.60:0/64862968 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f93c4053a70 0x7f93c4055f30 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7f93d800b000 tx=0x7f93d8006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:20.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.025+0000 7f93d7fff700 1 -- 172.21.15.60:0/64862968 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f93e0044560 con 0x7f93e813c920 2022-06-29T16:07:20.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.197+0000 7f93ed876700 1 -- 172.21.15.60:0/64862968 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f93e80573d0 con 0x7f93e813c920 2022-06-29T16:07:20.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.361+0000 7f93d7fff700 1 -- 172.21.15.60:0/64862968 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v62) v1 ==== 118+0+0 (secure 0 0 0) 0x7f93e001f020 con 0x7f93e813c920 2022-06-29T16:07:20.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.361+0000 7f93ed876700 1 -- 172.21.15.60:0/64862968 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f93c4053a70 msgr2=0x7f93c4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:20.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.361+0000 7f93ed876700 1 --2- 172.21.15.60:0/64862968 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f93c4053a70 0x7f93c4055f30 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7f93d800b000 tx=0x7f93d8006df0).stop 2022-06-29T16:07:20.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.361+0000 7f93ed876700 1 -- 172.21.15.60:0/64862968 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93e813c920 msgr2=0x7f93e8141800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:20.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.361+0000 7f93ed876700 1 --2- 172.21.15.60:0/64862968 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93e813c920 0x7f93e8141800 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 rx=0x7f93e00032f0 tx=0x7f93e0017000).stop 2022-06-29T16:07:20.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.365+0000 7f93ed876700 1 -- 172.21.15.60:0/64862968 shutdown_connections 2022-06-29T16:07:20.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.365+0000 7f93ed876700 1 --2- 172.21.15.60:0/64862968 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f93c4053a70 0x7f93c4055f30 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:20.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.365+0000 7f93ed876700 1 --2- 172.21.15.60:0/64862968 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93e813c920 0x7f93e8141800 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:20.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.365+0000 7f93ed876700 1 --2- 172.21.15.60:0/64862968 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f93e8064560 0x7f93e80b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:20.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.365+0000 7f93ed876700 1 --2- 172.21.15.60:0/64862968 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f93e805c460 0x7f93e80ae510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:20.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.365+0000 7f93ed876700 1 -- 172.21.15.60:0/64862968 >> 172.21.15.60:0/64862968 conn(0x7f93e8061f30 msgr2=0x7f93e80b40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:20.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.365+0000 7f93ed876700 1 -- 172.21.15.60:0/64862968 shutdown_connections 2022-06-29T16:07:20.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.365+0000 7f93ed876700 1 -- 172.21.15.60:0/64862968 wait complete. 2022-06-29T16:07:20.385 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:07:20.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.489+0000 7f797759e700 1 Processor -- start 2022-06-29T16:07:20.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.489+0000 7f797759e700 1 -- start start 2022-06-29T16:07:20.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.493+0000 7f797759e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f79780b3270 0x7f79780ace00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:20.511 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.493+0000 7f797759e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7978062cd0 0x7f79781461c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:20.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.493+0000 7f797759e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79780adc80 con 0x7f79780aca20 2022-06-29T16:07:20.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.493+0000 7f797759e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79780ad740 con 0x7f7978062cd0 2022-06-29T16:07:20.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.493+0000 7f797759e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79780ad9e0 con 0x7f79780b3270 2022-06-29T16:07:20.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.493+0000 7f7976d9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7978062cd0 0x7f79781461c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:20.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.493+0000 7f797659c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f79780aca20 0x7f7978143030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:32796/0 (socket says 172.21.15.60:32796) 2022-06-29T16:07:20.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.493+0000 7f7976d9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7978062cd0 0x7f79781461c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52946/0 (socket says 172.21.15.60:52946) 2022-06-29T16:07:20.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.493+0000 7f797659c700 1 -- 172.21.15.60:0/2202515449 learned_addr learned my addr 172.21.15.60:0/2202515449 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:20.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.493+0000 7f7975d9b700 1 --2- 172.21.15.60:0/2202515449 >> v2:172.21.15.92:3300/0 conn(0x7f79780b3270 0x7f79780ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:20.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.493+0000 7f7976d9d700 1 -- 172.21.15.60:0/2202515449 >> v2:172.21.15.92:3300/0 conn(0x7f79780b3270 msgr2=0x7f79780ace00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:20.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.493+0000 7f7976d9d700 1 --2- 172.21.15.60:0/2202515449 >> v2:172.21.15.92:3300/0 conn(0x7f79780b3270 0x7f79780ace00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:20.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.493+0000 7f7976d9d700 1 -- 172.21.15.60:0/2202515449 >> v1:172.21.15.60:6789/0 conn(0x7f79780aca20 legacy=0x7f7978143030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:20.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.493+0000 7f7976d9d700 1 -- 172.21.15.60:0/2202515449 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7978142920 con 0x7f7978062cd0 2022-06-29T16:07:20.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.493+0000 7f7976d9d700 1 --2- 172.21.15.60:0/2202515449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7978062cd0 0x7f79781461c0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 rx=0x7f7968002f70 tx=0x7f7968006df0).ready entity=mon.0 client_cookie=4441e2baf21287ab server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:20.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.493+0000 7f797559a700 1 -- 172.21.15.60:0/2202515449 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f796800d5c0 con 0x7f7978062cd0 2022-06-29T16:07:20.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.493+0000 7f797559a700 1 -- 172.21.15.60:0/2202515449 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7968000e90 con 0x7f7978062cd0 2022-06-29T16:07:20.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.493+0000 7f797559a700 1 -- 172.21.15.60:0/2202515449 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f796800d910 con 0x7f7978062cd0 2022-06-29T16:07:20.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.493+0000 7f797759e700 1 -- 172.21.15.60:0/2202515449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7978062cd0 msgr2=0x7f79781461c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:20.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.493+0000 7f797759e700 1 --2- 172.21.15.60:0/2202515449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7978062cd0 0x7f79781461c0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 rx=0x7f7968002f70 tx=0x7f7968006df0).stop 2022-06-29T16:07:20.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.493+0000 7f797759e700 1 -- 172.21.15.60:0/2202515449 shutdown_connections 2022-06-29T16:07:20.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.493+0000 7f797759e700 1 --2- 172.21.15.60:0/2202515449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7978062cd0 0x7f79781461c0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:20.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.493+0000 7f797759e700 1 --2- 172.21.15.60:0/2202515449 >> v2:172.21.15.92:3300/0 conn(0x7f79780b3270 0x7f79780ace00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:20.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.493+0000 7f797759e700 1 -- 172.21.15.60:0/2202515449 >> 172.21.15.60:0/2202515449 conn(0x7f79780b6f80 msgr2=0x7f79780b7390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:20.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.493+0000 7f797759e700 1 -- 172.21.15.60:0/2202515449 shutdown_connections 2022-06-29T16:07:20.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.493+0000 7f797759e700 1 -- 172.21.15.60:0/2202515449 wait complete. 2022-06-29T16:07:20.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.493+0000 7f797759e700 1 Processor -- start 2022-06-29T16:07:20.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.493+0000 7f797759e700 1 -- start start 2022-06-29T16:07:20.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.497+0000 7f797759e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7978062cd0 0x7f7978152ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:20.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.497+0000 7f797759e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f79780aca20 0x7f79780ae310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:20.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.497+0000 7f797759e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f79780b3270 0x7f7978155460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:20.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.497+0000 7f797759e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7978152300 con 0x7f7978062cd0 2022-06-29T16:07:20.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.497+0000 7f797759e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79781525a0 con 0x7f79780b3270 2022-06-29T16:07:20.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.497+0000 7f797759e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7978152840 con 0x7f79780aca20 2022-06-29T16:07:20.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.497+0000 7f797659c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7978062cd0 0x7f7978152ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:20.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.497+0000 7f797659c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7978062cd0 0x7f7978152ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52948/0 (socket says 172.21.15.60:52948) 2022-06-29T16:07:20.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.497+0000 7f797659c700 1 -- 172.21.15.60:0/4182334660 learned_addr learned my addr 172.21.15.60:0/4182334660 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:20.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.497+0000 7f7975d9b700 1 --2- 172.21.15.60:0/4182334660 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f79780aca20 0x7f79780ae310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:20.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.497+0000 7f7976d9d700 1 --2- 172.21.15.60:0/4182334660 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f79780b3270 0x7f7978155460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:20.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.497+0000 7f797659c700 1 -- 172.21.15.60:0/4182334660 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f79780aca20 msgr2=0x7f79780ae310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:20.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.497+0000 7f797659c700 1 --2- 172.21.15.60:0/4182334660 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f79780aca20 0x7f79780ae310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:20.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.497+0000 7f797659c700 1 -- 172.21.15.60:0/4182334660 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f79780b3270 msgr2=0x7f7978155460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:20.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.497+0000 7f797659c700 1 --2- 172.21.15.60:0/4182334660 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f79780b3270 0x7f7978155460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:20.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.497+0000 7f797659c700 1 -- 172.21.15.60:0/4182334660 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f796800b180 con 0x7f7978062cd0 2022-06-29T16:07:20.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.497+0000 7f7975d9b700 1 --2- 172.21.15.60:0/4182334660 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f79780aca20 0x7f79780ae310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:20.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.497+0000 7f797659c700 1 --2- 172.21.15.60:0/4182334660 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7978062cd0 0x7f7978152ce0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 rx=0x7f79780b37b0 tx=0x7f7960007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:20.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.497+0000 7f796f7fe700 1 -- 172.21.15.60:0/4182334660 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79600070d0 con 0x7f7978062cd0 2022-06-29T16:07:20.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.497+0000 7f796f7fe700 1 -- 172.21.15.60:0/4182334660 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7960007230 con 0x7f7978062cd0 2022-06-29T16:07:20.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.497+0000 7f796f7fe700 1 -- 172.21.15.60:0/4182334660 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7960005610 con 0x7f7978062cd0 2022-06-29T16:07:20.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.497+0000 7f797759e700 1 -- 172.21.15.60:0/4182334660 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79780b0190 con 0x7f7978062cd0 2022-06-29T16:07:20.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.497+0000 7f797759e700 1 -- 172.21.15.60:0/4182334660 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79781520d0 con 0x7f7978062cd0 2022-06-29T16:07:20.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.497+0000 7f797759e700 1 -- 172.21.15.60:0/4182334660 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79780573d0 con 0x7f7978062cd0 2022-06-29T16:07:20.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.497+0000 7f796f7fe700 1 -- 172.21.15.60:0/4182334660 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7960010070 con 0x7f7978062cd0 2022-06-29T16:07:20.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.501+0000 7f796f7fe700 1 --2- 172.21.15.60:0/4182334660 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7950053b00 0x7f7950055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:20.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.501+0000 7f796f7fe700 1 -- 172.21.15.60:0/4182334660 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f796006b410 con 0x7f7978062cd0 2022-06-29T16:07:20.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.501+0000 7f7975d9b700 1 --2- 172.21.15.60:0/4182334660 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7950053b00 0x7f7950055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:20.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.501+0000 7f7975d9b700 1 --2- 172.21.15.60:0/4182334660 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7950053b00 0x7f7950055fc0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x7f797000c000 tx=0x7f7970008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:20.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.505+0000 7f796f7fe700 1 -- 172.21.15.60:0/4182334660 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7960042330 con 0x7f7978062cd0 2022-06-29T16:07:20.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.665+0000 7f797759e700 1 -- 172.21.15.60:0/4182334660 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79780b5670 con 0x7f7978062cd0 2022-06-29T16:07:20.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.665+0000 7f796f7fe700 1 -- 172.21.15.60:0/4182334660 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2142 (secure 0 0 0) 0x7f7960018070 con 0x7f7978062cd0 2022-06-29T16:07:20.671 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:20.672 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":62,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6941,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/3488623391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3488623391},{"type":"v1","addr":"172.21.15.92:6835","nonce":3488623391}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":59}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:07:19.341489+0000","modified":"2022-06-29T16:07:20.345065+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8085},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8085":{"gid":8085,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/3877729627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3877729627},{"type":"v1","addr":"172.21.15.60:6833","nonce":3877729627}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-06-29T16:07:20.674 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.669+0000 7f797759e700 1 -- 172.21.15.60:0/4182334660 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7950053b00 msgr2=0x7f7950055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:20.675 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.669+0000 7f797759e700 1 --2- 172.21.15.60:0/4182334660 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7950053b00 0x7f7950055fc0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x7f797000c000 tx=0x7f7970008040).stop 2022-06-29T16:07:20.675 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.669+0000 7f797759e700 1 -- 172.21.15.60:0/4182334660 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7978062cd0 msgr2=0x7f7978152ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:20.675 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.669+0000 7f797759e700 1 --2- 172.21.15.60:0/4182334660 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7978062cd0 0x7f7978152ce0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 rx=0x7f79780b37b0 tx=0x7f7960007c00).stop 2022-06-29T16:07:20.675 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.669+0000 7f797759e700 1 -- 172.21.15.60:0/4182334660 shutdown_connections 2022-06-29T16:07:20.676 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.669+0000 7f797759e700 1 --2- 172.21.15.60:0/4182334660 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7950053b00 0x7f7950055fc0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:20.676 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.669+0000 7f797759e700 1 --2- 172.21.15.60:0/4182334660 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f79780b3270 0x7f7978155460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:20.676 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.669+0000 7f797759e700 1 --2- 172.21.15.60:0/4182334660 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f79780aca20 0x7f79780ae310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:20.676 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.669+0000 7f797759e700 1 --2- 172.21.15.60:0/4182334660 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7978062cd0 0x7f7978152ce0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:20.677 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.669+0000 7f797759e700 1 -- 172.21.15.60:0/4182334660 >> 172.21.15.60:0/4182334660 conn(0x7f79780b6f80 msgr2=0x7f7978142ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:20.677 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.669+0000 7f797759e700 1 -- 172.21.15.60:0/4182334660 shutdown_connections 2022-06-29T16:07:20.677 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.669+0000 7f797759e700 1 -- 172.21.15.60:0/4182334660 wait complete. 2022-06-29T16:07:20.677 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 62 2022-06-29T16:07:20.685 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:07:20.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.785+0000 7f5fd3fff700 1 Processor -- start 2022-06-29T16:07:20.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.785+0000 7f5fd3fff700 1 -- start start 2022-06-29T16:07:20.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.785+0000 7f5fd3fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5fd40b2130 0x7f5fd4146100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:20.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.785+0000 7f5fd3fff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fd405d450 con 0x7f5fd40aed00 2022-06-29T16:07:20.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.785+0000 7f5fd3fff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fd40b25d0 con 0x7f5fd405d070 2022-06-29T16:07:20.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.785+0000 7f5fd3fff700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fd40adf90 con 0x7f5fd40b2130 2022-06-29T16:07:20.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.785+0000 7f5fd27fc700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5fd40b2130 0x7f5fd4146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:20.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.785+0000 7f5fd27fc700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5fd40b2130 0x7f5fd4146100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52958/0 (socket says 172.21.15.60:52958) 2022-06-29T16:07:20.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.785+0000 7f5fd27fc700 1 -- 172.21.15.60:0/50574651 learned_addr learned my addr 172.21.15.60:0/50574651 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:20.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.785+0000 7f5fd37fe700 1 --1- 172.21.15.60:0/50574651 >> v1:172.21.15.60:6789/0 conn(0x7f5fd40aed00 0x7f5fd41465f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:32814/0 (socket says 172.21.15.60:32814) 2022-06-29T16:07:20.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.785+0000 7f5fd27fc700 1 -- 172.21.15.60:0/50574651 >> v1:172.21.15.92:6789/0 conn(0x7f5fd405d070 legacy=0x7f5fd414aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:20.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.785+0000 7f5fd27fc700 1 -- 172.21.15.60:0/50574651 >> v1:172.21.15.60:6789/0 conn(0x7f5fd40aed00 legacy=0x7f5fd41465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:20.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.785+0000 7f5fd27fc700 1 -- 172.21.15.60:0/50574651 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd414a350 con 0x7f5fd40b2130 2022-06-29T16:07:20.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.785+0000 7f5fd27fc700 1 --2- 172.21.15.60:0/50574651 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fd40b2130 0x7f5fd4146100 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 rx=0x7f5fd40b5cd0 tx=0x7f5fbc006eb0).ready entity=mon.0 client_cookie=aeeee1dacea31aef server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:20.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.785+0000 7f5fd1ffb700 1 -- 172.21.15.60:0/50574651 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fbc00d610 con 0x7f5fd40b2130 2022-06-29T16:07:20.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.785+0000 7f5fd1ffb700 1 -- 172.21.15.60:0/50574651 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fbc000e90 con 0x7f5fd40b2130 2022-06-29T16:07:20.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.785+0000 7f5fd1ffb700 1 -- 172.21.15.60:0/50574651 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fbc00d960 con 0x7f5fd40b2130 2022-06-29T16:07:20.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.789+0000 7f5fd3fff700 1 -- 172.21.15.60:0/50574651 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fd40b2130 msgr2=0x7f5fd4146100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:20.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.789+0000 7f5fd3fff700 1 --2- 172.21.15.60:0/50574651 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fd40b2130 0x7f5fd4146100 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 rx=0x7f5fd40b5cd0 tx=0x7f5fbc006eb0).stop 2022-06-29T16:07:20.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.789+0000 7f5fd3fff700 1 -- 172.21.15.60:0/50574651 shutdown_connections 2022-06-29T16:07:20.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.789+0000 7f5fd3fff700 1 --2- 172.21.15.60:0/50574651 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fd40b2130 0x7f5fd4146100 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:20.797 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.789+0000 7f5fd3fff700 1 -- 172.21.15.60:0/50574651 >> 172.21.15.60:0/50574651 conn(0x7f5fd40b5890 msgr2=0x7f5fd40b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:20.797 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.789+0000 7f5fd3fff700 1 -- 172.21.15.60:0/50574651 shutdown_connections 2022-06-29T16:07:20.797 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.789+0000 7f5fd3fff700 1 -- 172.21.15.60:0/50574651 wait complete. 2022-06-29T16:07:20.800 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.789+0000 7f5fd3fff700 1 Processor -- start 2022-06-29T16:07:20.800 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.789+0000 7f5fd3fff700 1 -- start start 2022-06-29T16:07:20.800 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.789+0000 7f5fd3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fd405d070 0x7f5fd4063460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:20.801 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.789+0000 7f5fd3fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fd40aed00 0x7f5fd41568f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:20.801 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.789+0000 7f5fd3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fd40b2130 0x7f5fd4116ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:20.801 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.789+0000 7f5fd3fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd41182b0 con 0x7f5fd40b2130 2022-06-29T16:07:20.801 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.789+0000 7f5fd3fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd415a8d0 con 0x7f5fd405d070 2022-06-29T16:07:20.801 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.789+0000 7f5fd3fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd415ab00 con 0x7f5fd40aed00 2022-06-29T16:07:20.802 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.789+0000 7f5fd37fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fd40b2130 0x7f5fd4116ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:20.802 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.789+0000 7f5fd37fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fd40b2130 0x7f5fd4116ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52966/0 (socket says 172.21.15.60:52966) 2022-06-29T16:07:20.802 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.789+0000 7f5fd37fe700 1 -- 172.21.15.60:0/1949277651 learned_addr learned my addr 172.21.15.60:0/1949277651 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:20.802 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.789+0000 7f5fd2ffd700 1 --2- 172.21.15.60:0/1949277651 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fd405d070 0x7f5fd4063460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:20.803 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.789+0000 7f5fd27fc700 1 --2- 172.21.15.60:0/1949277651 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fd40aed00 0x7f5fd41568f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:20.803 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.789+0000 7f5fd37fe700 1 -- 172.21.15.60:0/1949277651 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fd40aed00 msgr2=0x7f5fd41568f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:20.803 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.789+0000 7f5fd37fe700 1 --2- 172.21.15.60:0/1949277651 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fd40aed00 0x7f5fd41568f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:20.803 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.789+0000 7f5fd37fe700 1 -- 172.21.15.60:0/1949277651 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fd405d070 msgr2=0x7f5fd4063460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:20.803 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.789+0000 7f5fd37fe700 1 --2- 172.21.15.60:0/1949277651 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fd405d070 0x7f5fd4063460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:20.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.789+0000 7f5fd37fe700 1 -- 172.21.15.60:0/1949277651 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fbc00b1e0 con 0x7f5fd40b2130 2022-06-29T16:07:20.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.789+0000 7f5fd37fe700 1 --2- 172.21.15.60:0/1949277651 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fd40b2130 0x7f5fd4116ed0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 rx=0x7f5fd4061990 tx=0x7f5fcc007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:20.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.793+0000 7f5fcbfff700 1 -- 172.21.15.60:0/1949277651 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fcc005ae0 con 0x7f5fd40b2130 2022-06-29T16:07:20.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.793+0000 7f5fcbfff700 1 -- 172.21.15.60:0/1949277651 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fcc005c40 con 0x7f5fd40b2130 2022-06-29T16:07:20.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.793+0000 7f5fd3fff700 1 -- 172.21.15.60:0/1949277651 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fd415ada0 con 0x7f5fd40b2130 2022-06-29T16:07:20.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.793+0000 7f5fcbfff700 1 -- 172.21.15.60:0/1949277651 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fcc0070d0 con 0x7f5fd40b2130 2022-06-29T16:07:20.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.793+0000 7f5fd3fff700 1 -- 172.21.15.60:0/1949277651 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fd4066050 con 0x7f5fd40b2130 2022-06-29T16:07:20.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.793+0000 7f5fd3fff700 1 -- 172.21.15.60:0/1949277651 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fd4116c30 con 0x7f5fd40b2130 2022-06-29T16:07:20.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.793+0000 7f5fcbfff700 1 -- 172.21.15.60:0/1949277651 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5fcc01a060 con 0x7f5fd40b2130 2022-06-29T16:07:20.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.793+0000 7f5fcbfff700 1 --2- 172.21.15.60:0/1949277651 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5fb8053b00 0x7f5fb8055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:20.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.793+0000 7f5fcbfff700 1 -- 172.21.15.60:0/1949277651 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5fcc06b8c0 con 0x7f5fd40b2130 2022-06-29T16:07:20.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.793+0000 7f5fd2ffd700 1 --2- 172.21.15.60:0/1949277651 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5fb8053b00 0x7f5fb8055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:20.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.797+0000 7f5fd2ffd700 1 --2- 172.21.15.60:0/1949277651 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5fb8053b00 0x7f5fb8055fc0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7f5fc400d000 tx=0x7f5fc4007b70).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:20.807 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.801+0000 7f5fcbfff700 1 -- 172.21.15.60:0/1949277651 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5fd4116c30 con 0x7f5fd40b2130 2022-06-29T16:07:20.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.929+0000 7f5fd3fff700 1 -- 172.21.15.60:0/1949277651 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5fd4116c30 con 0x7f5fd40b2130 2022-06-29T16:07:20.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.929+0000 7f5fcbfff700 1 -- 172.21.15.60:0/1949277651 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) v1 ==== 74+0+16052 (secure 0 0 0) 0x7f5fd4116c30 con 0x7f5fd40b2130 2022-06-29T16:07:20.936 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:20.936 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":128,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:07:20.344096+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:07:17.372006+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2022-06-29T16:07:18.349364+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:07:20.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.933+0000 7f5fd3fff700 1 -- 172.21.15.60:0/1949277651 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5fb8053b00 msgr2=0x7f5fb8055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:20.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.933+0000 7f5fd3fff700 1 --2- 172.21.15.60:0/1949277651 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5fb8053b00 0x7f5fb8055fc0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7f5fc400d000 tx=0x7f5fc4007b70).stop 2022-06-29T16:07:20.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.933+0000 7f5fd3fff700 1 -- 172.21.15.60:0/1949277651 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fd40b2130 msgr2=0x7f5fd4116ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:20.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.933+0000 7f5fd3fff700 1 --2- 172.21.15.60:0/1949277651 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fd40b2130 0x7f5fd4116ed0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 rx=0x7f5fd4061990 tx=0x7f5fcc007f40).stop 2022-06-29T16:07:20.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.933+0000 7f5fd3fff700 1 -- 172.21.15.60:0/1949277651 shutdown_connections 2022-06-29T16:07:20.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.933+0000 7f5fd3fff700 1 --2- 172.21.15.60:0/1949277651 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5fb8053b00 0x7f5fb8055fc0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:20.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.933+0000 7f5fd3fff700 1 --2- 172.21.15.60:0/1949277651 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fd40b2130 0x7f5fd4116ed0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:20.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.933+0000 7f5fd3fff700 1 --2- 172.21.15.60:0/1949277651 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fd40aed00 0x7f5fd41568f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:20.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.933+0000 7f5fd3fff700 1 --2- 172.21.15.60:0/1949277651 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fd405d070 0x7f5fd4063460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:20.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.933+0000 7f5fd3fff700 1 -- 172.21.15.60:0/1949277651 >> 172.21.15.60:0/1949277651 conn(0x7f5fd40b5890 msgr2=0x7f5fd40aca20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:20.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.933+0000 7f5fd3fff700 1 -- 172.21.15.60:0/1949277651 shutdown_connections 2022-06-29T16:07:20.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:20.933+0000 7f5fd3fff700 1 -- 172.21.15.60:0/1949277651 wait complete. 2022-06-29T16:07:20.949 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T16:07:21.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.057+0000 7fcbe6ade700 1 Processor -- start 2022-06-29T16:07:21.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.057+0000 7fcbe6ade700 1 -- start start 2022-06-29T16:07:21.063 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.057+0000 7fcbe6ade700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbe00b9c20 0x7fcbe00b0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:21.064 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.057+0000 7fcbe6ade700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe00b1de0 con 0x7fcbe0146480 2022-06-29T16:07:21.064 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.057+0000 7fcbe6ade700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe00b2bc0 con 0x7fcbe0146040 2022-06-29T16:07:21.064 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.057+0000 7fcbe6ade700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe00b1330 con 0x7fcbe00b9c20 2022-06-29T16:07:21.064 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.057+0000 7fcbe62dd700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcbe0146480 0x7fcbe005bd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:32828/0 (socket says 172.21.15.60:32828) 2022-06-29T16:07:21.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.057+0000 7fcbe62dd700 1 -- 172.21.15.60:0/438732138 learned_addr learned my addr 172.21.15.60:0/438732138 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:21.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.057+0000 7fcbe5adc700 1 --2- 172.21.15.60:0/438732138 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbe00b9c20 0x7fcbe00b0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:21.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.057+0000 7fcbe4ada700 1 -- 172.21.15.60:0/438732138 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922438088 0 0) 0x7fcbe00b1de0 con 0x7fcbe0146480 2022-06-29T16:07:21.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.057+0000 7fcbe4ada700 1 -- 172.21.15.60:0/438732138 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbcc003260 con 0x7fcbe0146480 2022-06-29T16:07:21.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.057+0000 7fcbe4ada700 1 -- 172.21.15.60:0/438732138 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4275804883 0 0) 0x7fcbcc003260 con 0x7fcbe0146480 2022-06-29T16:07:21.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.057+0000 7fcbe4ada700 1 -- 172.21.15.60:0/438732138 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbe00b9c20 msgr2=0x7fcbe00b0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:21.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.057+0000 7fcbe4ada700 1 --2- 172.21.15.60:0/438732138 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbe00b9c20 0x7fcbe00b0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:21.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.061+0000 7fcbe4ada700 1 -- 172.21.15.60:0/438732138 >> v1:172.21.15.92:6789/0 conn(0x7fcbe0146040 legacy=0x7fcbe00b24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:21.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.061+0000 7fcbe4ada700 1 -- 172.21.15.60:0/438732138 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe00ba000 con 0x7fcbe0146480 2022-06-29T16:07:21.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.061+0000 7fcbe4ada700 1 -- 172.21.15.60:0/438732138 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7fcbd80019e0 con 0x7fcbe0146480 2022-06-29T16:07:21.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.061+0000 7fcbe4ada700 1 -- 172.21.15.60:0/438732138 >> v1:172.21.15.60:6789/0 conn(0x7fcbe0146480 legacy=0x7fcbe005bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:21.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.061+0000 7fcbe4ada700 1 --2- 172.21.15.60:0/438732138 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcbcc003780 0x7fcbcc003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:21.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.061+0000 7fcbe4ada700 1 --2- 172.21.15.60:0/438732138 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbe00b9c20 0x7fcbcc004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:21.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.061+0000 7fcbe4ada700 1 --2- 172.21.15.60:0/438732138 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbcc0049f0 0x7fcbcc004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:21.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.061+0000 7fcbe4ada700 1 -- 172.21.15.60:0/438732138 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe00b1330 con 0x7fcbcc0049f0 2022-06-29T16:07:21.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.061+0000 7fcbe4ada700 1 -- 172.21.15.60:0/438732138 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbcc00b780 con 0x7fcbcc003780 2022-06-29T16:07:21.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.061+0000 7fcbe4ada700 1 -- 172.21.15.60:0/438732138 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbcc00b940 con 0x7fcbe00b9c20 2022-06-29T16:07:21.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.061+0000 7fcbe62dd700 1 --2- 172.21.15.60:0/438732138 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbcc0049f0 0x7fcbcc004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:21.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.061+0000 7fcbe5adc700 1 --2- 172.21.15.60:0/438732138 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcbcc003780 0x7fcbcc003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:21.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.061+0000 7fcbe52db700 1 --2- 172.21.15.60:0/438732138 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbe00b9c20 0x7fcbcc004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:21.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.061+0000 7fcbe62dd700 1 -- 172.21.15.60:0/438732138 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbe00b9c20 msgr2=0x7fcbcc004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:21.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.061+0000 7fcbe62dd700 1 --2- 172.21.15.60:0/438732138 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbe00b9c20 0x7fcbcc004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:21.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.061+0000 7fcbe62dd700 1 -- 172.21.15.60:0/438732138 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcbcc003780 msgr2=0x7fcbcc003ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:21.073 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.061+0000 7fcbe62dd700 1 --2- 172.21.15.60:0/438732138 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcbcc003780 0x7fcbcc003ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:21.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.061+0000 7fcbe62dd700 1 -- 172.21.15.60:0/438732138 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbcc003260 con 0x7fcbcc0049f0 2022-06-29T16:07:21.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.061+0000 7fcbe62dd700 1 --2- 172.21.15.60:0/438732138 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbcc0049f0 0x7fcbcc004e70 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 rx=0x7fcbd80016f0 tx=0x7fcbe00b0340).ready entity=mon.0 client_cookie=1194661e85277400 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:21.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.061+0000 7fcbe4ada700 1 -- 172.21.15.60:0/438732138 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbd800cca0 con 0x7fcbcc0049f0 2022-06-29T16:07:21.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.061+0000 7fcbe4ada700 1 -- 172.21.15.60:0/438732138 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd800ce00 con 0x7fcbcc0049f0 2022-06-29T16:07:21.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.061+0000 7fcbe4ada700 1 -- 172.21.15.60:0/438732138 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbd8005b80 con 0x7fcbcc0049f0 2022-06-29T16:07:21.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.061+0000 7fcbe6ade700 1 -- 172.21.15.60:0/438732138 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbcc0049f0 msgr2=0x7fcbcc004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:21.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.061+0000 7fcbe6ade700 1 --2- 172.21.15.60:0/438732138 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbcc0049f0 0x7fcbcc004e70 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 rx=0x7fcbd80016f0 tx=0x7fcbe00b0340).stop 2022-06-29T16:07:21.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.061+0000 7fcbe62dd700 1 -- 172.21.15.60:0/438732138 reap_dead start 2022-06-29T16:07:21.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.061+0000 7fcbe6ade700 1 -- 172.21.15.60:0/438732138 shutdown_connections 2022-06-29T16:07:21.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.061+0000 7fcbe6ade700 1 -- 172.21.15.60:0/438732138 >> 172.21.15.60:0/438732138 conn(0x7fcbe0062cb0 msgr2=0x7fcbe00b91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:21.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.061+0000 7fcbe6ade700 1 -- 172.21.15.60:0/438732138 shutdown_connections 2022-06-29T16:07:21.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.061+0000 7fcbe6ade700 1 -- 172.21.15.60:0/438732138 wait complete. 2022-06-29T16:07:21.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.065+0000 7fcbe6ade700 1 Processor -- start 2022-06-29T16:07:21.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.065+0000 7fcbe6ade700 1 -- start start 2022-06-29T16:07:21.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.065+0000 7fcbe6ade700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcbcc0049f0 0x7fcbe0113570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:21.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.065+0000 7fcbe6ade700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbe0112b70 0x7fcbe013ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:21.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.065+0000 7fcbe6ade700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbe013ef40 0x7fcbe00aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:21.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.065+0000 7fcbe6ade700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe01525a0 con 0x7fcbe0112b70 2022-06-29T16:07:21.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.065+0000 7fcbe6ade700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe00aeff0 con 0x7fcbcc0049f0 2022-06-29T16:07:21.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.065+0000 7fcbe6ade700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe00af290 con 0x7fcbe013ef40 2022-06-29T16:07:21.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.065+0000 7fcbe52db700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbe0112b70 0x7fcbe013ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:21.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.065+0000 7fcbe52db700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbe0112b70 0x7fcbe013ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52984/0 (socket says 172.21.15.60:52984) 2022-06-29T16:07:21.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.065+0000 7fcbe52db700 1 -- 172.21.15.60:0/1433344984 learned_addr learned my addr 172.21.15.60:0/1433344984 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:21.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.065+0000 7fcbe62dd700 1 --2- 172.21.15.60:0/1433344984 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbe013ef40 0x7fcbe00aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:21.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.065+0000 7fcbe5adc700 1 --2- 172.21.15.60:0/1433344984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcbcc0049f0 0x7fcbe0113570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:21.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.065+0000 7fcbe52db700 1 -- 172.21.15.60:0/1433344984 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbe013ef40 msgr2=0x7fcbe00aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:21.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.065+0000 7fcbe52db700 1 --2- 172.21.15.60:0/1433344984 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbe013ef40 0x7fcbe00aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:21.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.065+0000 7fcbe52db700 1 -- 172.21.15.60:0/1433344984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcbcc0049f0 msgr2=0x7fcbe0113570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:21.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.065+0000 7fcbe52db700 1 --2- 172.21.15.60:0/1433344984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcbcc0049f0 0x7fcbe0113570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:21.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.065+0000 7fcbe52db700 1 -- 172.21.15.60:0/1433344984 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe00aec70 con 0x7fcbe0112b70 2022-06-29T16:07:21.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.065+0000 7fcbe52db700 1 --2- 172.21.15.60:0/1433344984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbe0112b70 0x7fcbe013ea50 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 rx=0x7fcbd400b000 tx=0x7fcbd4009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:21.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.065+0000 7fcbdeffd700 1 -- 172.21.15.60:0/1433344984 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbd40155f0 con 0x7fcbe0112b70 2022-06-29T16:07:21.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.065+0000 7fcbdeffd700 1 -- 172.21.15.60:0/1433344984 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd4010040 con 0x7fcbe0112b70 2022-06-29T16:07:21.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.065+0000 7fcbe6ade700 1 -- 172.21.15.60:0/1433344984 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbe013f3c0 con 0x7fcbe0112b70 2022-06-29T16:07:21.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.065+0000 7fcbdeffd700 1 -- 172.21.15.60:0/1433344984 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbd4015940 con 0x7fcbe0112b70 2022-06-29T16:07:21.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.065+0000 7fcbe6ade700 1 -- 172.21.15.60:0/1433344984 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbe0152b70 con 0x7fcbe0112b70 2022-06-29T16:07:21.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.069+0000 7fcbe6ade700 1 -- 172.21.15.60:0/1433344984 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbe00aee60 con 0x7fcbe0112b70 2022-06-29T16:07:21.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.069+0000 7fcbdeffd700 1 -- 172.21.15.60:0/1433344984 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcbd4015aa0 con 0x7fcbe0112b70 2022-06-29T16:07:21.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.069+0000 7fcbdeffd700 1 --2- 172.21.15.60:0/1433344984 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcbc0053b00 0x7fcbc0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:21.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.069+0000 7fcbdeffd700 1 -- 172.21.15.60:0/1433344984 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcbd406bfb0 con 0x7fcbe0112b70 2022-06-29T16:07:21.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.069+0000 7fcbe5adc700 1 --2- 172.21.15.60:0/1433344984 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcbc0053b00 0x7fcbc0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:21.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.069+0000 7fcbe5adc700 1 --2- 172.21.15.60:0/1433344984 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcbc0053b00 0x7fcbc0055fc0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7fcbd000b000 tx=0x7fcbd0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:21.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.073+0000 7fcbdeffd700 1 -- 172.21.15.60:0/1433344984 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcbd4042dc0 con 0x7fcbe0112b70 2022-06-29T16:07:21.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.241+0000 7fcbe6ade700 1 -- 172.21.15.60:0/1433344984 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fcbe00573d0 con 0x7fcbe0112b70 2022-06-29T16:07:21.255 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.249+0000 7fcbdeffd700 1 -- 172.21.15.60:0/1433344984 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v41) v1 ==== 184+0+0 (secure 0 0 0) 0x7fcbd40463e0 con 0x7fcbe0112b70 2022-06-29T16:07:21.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.253+0000 7fcbe6ade700 1 -- 172.21.15.60:0/1433344984 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcbc0053b00 msgr2=0x7fcbc0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:21.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.253+0000 7fcbe6ade700 1 --2- 172.21.15.60:0/1433344984 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcbc0053b00 0x7fcbc0055fc0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7fcbd000b000 tx=0x7fcbd0008040).stop 2022-06-29T16:07:21.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.253+0000 7fcbe6ade700 1 -- 172.21.15.60:0/1433344984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbe0112b70 msgr2=0x7fcbe013ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:21.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.253+0000 7fcbe6ade700 1 --2- 172.21.15.60:0/1433344984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbe0112b70 0x7fcbe013ea50 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 rx=0x7fcbd400b000 tx=0x7fcbd4009040).stop 2022-06-29T16:07:21.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.253+0000 7fcbe6ade700 1 -- 172.21.15.60:0/1433344984 shutdown_connections 2022-06-29T16:07:21.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.253+0000 7fcbe6ade700 1 --2- 172.21.15.60:0/1433344984 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcbc0053b00 0x7fcbc0055fc0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:21.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.253+0000 7fcbe6ade700 1 --2- 172.21.15.60:0/1433344984 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcbe013ef40 0x7fcbe00aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:21.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.253+0000 7fcbe6ade700 1 --2- 172.21.15.60:0/1433344984 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcbe0112b70 0x7fcbe013ea50 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:21.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.253+0000 7fcbe6ade700 1 --2- 172.21.15.60:0/1433344984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcbcc0049f0 0x7fcbe0113570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:21.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.253+0000 7fcbe6ade700 1 -- 172.21.15.60:0/1433344984 >> 172.21.15.60:0/1433344984 conn(0x7fcbe0062cb0 msgr2=0x7fcbe005c880 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:21.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.253+0000 7fcbe6ade700 1 -- 172.21.15.60:0/1433344984 shutdown_connections 2022-06-29T16:07:21.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.253+0000 7fcbe6ade700 1 -- 172.21.15.60:0/1433344984 wait complete. 2022-06-29T16:07:21.263 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.0 2022-06-29T16:07:21.270 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T16:07:21.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.381+0000 7f957b4e7700 1 Processor -- start 2022-06-29T16:07:21.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.381+0000 7f957b4e7700 1 -- start start 2022-06-29T16:07:21.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.385+0000 7f957b4e7700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9574144ad0 0x7f9574140810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:21.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.385+0000 7f957b4e7700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95740b23a0 con 0x7f9574066d20 2022-06-29T16:07:21.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.385+0000 7f957b4e7700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9574067100 con 0x7f95740b3110 2022-06-29T16:07:21.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.385+0000 7f957b4e7700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9574140d00 con 0x7f9574144ad0 2022-06-29T16:07:21.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.385+0000 7f957a4e5700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9574066d20 0x7f95741443c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:32844/0 (socket says 172.21.15.60:32844) 2022-06-29T16:07:21.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.385+0000 7f957a4e5700 1 -- 172.21.15.60:0/2012076466 learned_addr learned my addr 172.21.15.60:0/2012076466 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:21.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.385+0000 7f9579ce4700 1 --2- 172.21.15.60:0/2012076466 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9574144ad0 0x7f9574140810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:21.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.385+0000 7f95794e3700 1 -- 172.21.15.60:0/2012076466 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228261809 0 0) 0x7f95740b23a0 con 0x7f9574066d20 2022-06-29T16:07:21.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.385+0000 7f95794e3700 1 -- 172.21.15.60:0/2012076466 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f955c003260 con 0x7f9574066d20 2022-06-29T16:07:21.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.385+0000 7f95794e3700 1 -- 172.21.15.60:0/2012076466 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4185188139 0 0) 0x7f9574067100 con 0x7f95740b3110 2022-06-29T16:07:21.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.385+0000 7f95794e3700 1 -- 172.21.15.60:0/2012076466 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95740b23a0 con 0x7f95740b3110 2022-06-29T16:07:21.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.385+0000 7f95794e3700 1 -- 172.21.15.60:0/2012076466 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1122839129 0 0) 0x7f955c003260 con 0x7f9574066d20 2022-06-29T16:07:21.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.385+0000 7f9579ce4700 1 -- 172.21.15.60:0/2012076466 >> v1:172.21.15.92:6789/0 conn(0x7f95740b3110 legacy=0x7f957413d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:21.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.385+0000 7f9579ce4700 1 -- 172.21.15.60:0/2012076466 >> v1:172.21.15.60:6789/0 conn(0x7f9574066d20 legacy=0x7f95741443c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:21.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.385+0000 7f9579ce4700 1 -- 172.21.15.60:0/2012076466 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95740b2100 con 0x7f9574144ad0 2022-06-29T16:07:21.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.385+0000 7f9579ce4700 1 --2- 172.21.15.60:0/2012076466 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9574144ad0 0x7f9574140810 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f9570001110 tx=0x7f9564006df0).ready entity=mon.2 client_cookie=4a7718a4f80ae1ad server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:21.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.385+0000 7f95794e3700 1 -- 172.21.15.60:0/2012076466 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9570001680 con 0x7f9574144ad0 2022-06-29T16:07:21.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.385+0000 7f95794e3700 1 -- 172.21.15.60:0/2012076466 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9564000e90 con 0x7f9574144ad0 2022-06-29T16:07:21.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.385+0000 7f95794e3700 1 -- 172.21.15.60:0/2012076466 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9564012460 con 0x7f9574144ad0 2022-06-29T16:07:21.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.389+0000 7f957b4e7700 1 -- 172.21.15.60:0/2012076466 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9574144ad0 msgr2=0x7f9574140810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:21.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.389+0000 7f957b4e7700 1 --2- 172.21.15.60:0/2012076466 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9574144ad0 0x7f9574140810 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f9570001110 tx=0x7f9564006df0).stop 2022-06-29T16:07:21.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.389+0000 7f957b4e7700 1 -- 172.21.15.60:0/2012076466 shutdown_connections 2022-06-29T16:07:21.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.389+0000 7f957b4e7700 1 --2- 172.21.15.60:0/2012076466 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9574144ad0 0x7f9574140810 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:21.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.389+0000 7f957b4e7700 1 -- 172.21.15.60:0/2012076466 >> 172.21.15.60:0/2012076466 conn(0x7f9574061c80 msgr2=0x7f95740637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:21.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.389+0000 7f957b4e7700 1 -- 172.21.15.60:0/2012076466 shutdown_connections 2022-06-29T16:07:21.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.389+0000 7f957b4e7700 1 -- 172.21.15.60:0/2012076466 wait complete. 2022-06-29T16:07:21.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.389+0000 7f957b4e7700 1 Processor -- start 2022-06-29T16:07:21.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.389+0000 7f957b4e7700 1 -- start start 2022-06-29T16:07:21.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.393+0000 7f957b4e7700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9574066d20 0x7f95740b4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:21.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.393+0000 7f957b4e7700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95740b3110 0x7f95740b5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:21.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.393+0000 7f957b4e7700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9574144ad0 0x7f957413bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:21.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.393+0000 7f957b4e7700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f957413e670 con 0x7f9574144ad0 2022-06-29T16:07:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.393+0000 7f957b4e7700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95740b4000 con 0x7f9574066d20 2022-06-29T16:07:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.393+0000 7f957b4e7700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95740b3d60 con 0x7f95740b3110 2022-06-29T16:07:21.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.393+0000 7f957ace6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9574144ad0 0x7f957413bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:21.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.393+0000 7f9579ce4700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95740b3110 0x7f95740b5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:21.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.393+0000 7f957ace6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9574144ad0 0x7f957413bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53000/0 (socket says 172.21.15.60:53000) 2022-06-29T16:07:21.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.393+0000 7f957ace6700 1 -- 172.21.15.60:0/2231007855 learned_addr learned my addr 172.21.15.60:0/2231007855 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:21.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.393+0000 7f957a4e5700 1 --2- 172.21.15.60:0/2231007855 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9574066d20 0x7f95740b4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:21.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.393+0000 7f957ace6700 1 -- 172.21.15.60:0/2231007855 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95740b3110 msgr2=0x7f95740b5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:21.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.393+0000 7f957ace6700 1 --2- 172.21.15.60:0/2231007855 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95740b3110 0x7f95740b5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:21.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.393+0000 7f957ace6700 1 -- 172.21.15.60:0/2231007855 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9574066d20 msgr2=0x7f95740b4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:21.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.393+0000 7f957ace6700 1 --2- 172.21.15.60:0/2231007855 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9574066d20 0x7f95740b4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.393+0000 7f957ace6700 1 -- 172.21.15.60:0/2231007855 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f956400b180 con 0x7f9574144ad0 2022-06-29T16:07:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.393+0000 7f957ace6700 1 --2- 172.21.15.60:0/2231007855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9574144ad0 0x7f957413bf90 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 rx=0x7f956c001490 tx=0x7f956c007da0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.393+0000 7f95637fe700 1 -- 172.21.15.60:0/2231007855 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f956c002310 con 0x7f9574144ad0 2022-06-29T16:07:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.393+0000 7f95637fe700 1 -- 172.21.15.60:0/2231007855 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f956c002470 con 0x7f9574144ad0 2022-06-29T16:07:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.393+0000 7f957b4e7700 1 -- 172.21.15.60:0/2231007855 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95740b5140 con 0x7f9574144ad0 2022-06-29T16:07:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.393+0000 7f95637fe700 1 -- 172.21.15.60:0/2231007855 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f956c001840 con 0x7f9574144ad0 2022-06-29T16:07:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.393+0000 7f957b4e7700 1 -- 172.21.15.60:0/2231007855 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95740b02b0 con 0x7f9574144ad0 2022-06-29T16:07:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.397+0000 7f957b4e7700 1 -- 172.21.15.60:0/2231007855 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95740573d0 con 0x7f9574144ad0 2022-06-29T16:07:21.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.397+0000 7f95637fe700 1 -- 172.21.15.60:0/2231007855 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f956c010030 con 0x7f9574144ad0 2022-06-29T16:07:21.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.397+0000 7f95637fe700 1 --2- 172.21.15.60:0/2231007855 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9554053a70 0x7f9554055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:21.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.397+0000 7f95637fe700 1 -- 172.21.15.60:0/2231007855 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f956c06beb0 con 0x7f9574144ad0 2022-06-29T16:07:21.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.397+0000 7f957a4e5700 1 --2- 172.21.15.60:0/2231007855 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9554053a70 0x7f9554055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:21.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.397+0000 7f957a4e5700 1 --2- 172.21.15.60:0/2231007855 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9554053a70 0x7f9554055f30 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7f957000d000 tx=0x7f9570000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:21.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.405+0000 7f95637fe700 1 -- 172.21.15.60:0/2231007855 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f956c042cc0 con 0x7f9574144ad0 2022-06-29T16:07:21.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.569+0000 7f957b4e7700 1 -- 172.21.15.60:0/2231007855 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f9574068480 con 0x7f9574144ad0 2022-06-29T16:07:21.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.577+0000 7f95637fe700 1 -- 172.21.15.60:0/2231007855 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v42) v1 ==== 184+0+0 (secure 0 0 0) 0x7f956c0462e0 con 0x7f9574144ad0 2022-06-29T16:07:21.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.577+0000 7f957b4e7700 1 -- 172.21.15.60:0/2231007855 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9554053a70 msgr2=0x7f9554055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:21.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.577+0000 7f957b4e7700 1 --2- 172.21.15.60:0/2231007855 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9554053a70 0x7f9554055f30 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7f957000d000 tx=0x7f9570000bc0).stop 2022-06-29T16:07:21.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.577+0000 7f957b4e7700 1 -- 172.21.15.60:0/2231007855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9574144ad0 msgr2=0x7f957413bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:21.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.577+0000 7f957b4e7700 1 --2- 172.21.15.60:0/2231007855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9574144ad0 0x7f957413bf90 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 rx=0x7f956c001490 tx=0x7f956c007da0).stop 2022-06-29T16:07:21.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.577+0000 7f957b4e7700 1 -- 172.21.15.60:0/2231007855 shutdown_connections 2022-06-29T16:07:21.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.577+0000 7f957b4e7700 1 --2- 172.21.15.60:0/2231007855 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9554053a70 0x7f9554055f30 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:21.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.577+0000 7f957b4e7700 1 --2- 172.21.15.60:0/2231007855 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9574144ad0 0x7f957413bf90 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:21.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.577+0000 7f957b4e7700 1 --2- 172.21.15.60:0/2231007855 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f95740b3110 0x7f95740b5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:21.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.577+0000 7f957b4e7700 1 --2- 172.21.15.60:0/2231007855 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9574066d20 0x7f95740b4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:21.586 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.577+0000 7f957b4e7700 1 -- 172.21.15.60:0/2231007855 >> 172.21.15.60:0/2231007855 conn(0x7f9574061c80 msgr2=0x7f9574142da0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:21.586 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.577+0000 7f957b4e7700 1 -- 172.21.15.60:0/2231007855 shutdown_connections 2022-06-29T16:07:21.586 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.577+0000 7f957b4e7700 1 -- 172.21.15.60:0/2231007855 wait complete. 2022-06-29T16:07:21.586 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.1 2022-06-29T16:07:21.595 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T16:07:21.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.705+0000 7ff3ddcd2700 1 Processor -- start 2022-06-29T16:07:21.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.705+0000 7ff3ddcd2700 1 -- start start 2022-06-29T16:07:21.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.705+0000 7ff3ddcd2700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff3d805c390 0x7ff3d80b2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:21.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.705+0000 7ff3ddcd2700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff3d81442b0 0x7ff3d8144740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:21.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.705+0000 7ff3ddcd2700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3d8140a20 0x7ff3d8140e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:21.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.705+0000 7ff3ddcd2700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3d805c770 con 0x7ff3d805c390 2022-06-29T16:07:21.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.705+0000 7ff3ddcd2700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3d8140750 con 0x7ff3d81442b0 2022-06-29T16:07:21.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.705+0000 7ff3ddcd2700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d8143e80 con 0x7ff3d8140a20 2022-06-29T16:07:21.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.705+0000 7ff3d77fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff3d805c390 0x7ff3d80b2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:21.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.705+0000 7ff3d77fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff3d805c390 0x7ff3d80b2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53004/0 (socket says 172.21.15.60:53004) 2022-06-29T16:07:21.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.705+0000 7ff3d77fe700 1 -- 172.21.15.60:0/3629520418 learned_addr learned my addr 172.21.15.60:0/3629520418 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:21.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.705+0000 7ff3d7fff700 1 --2- 172.21.15.60:0/3629520418 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3d8140a20 0x7ff3d8140e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:21.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.705+0000 7ff3ceffd700 1 --2- 172.21.15.60:0/3629520418 >> v2:172.21.15.92:3300/0 conn(0x7ff3d81442b0 0x7ff3d8144740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:21.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.705+0000 7ff3d7fff700 1 -- 172.21.15.60:0/3629520418 >> v2:172.21.15.92:3300/0 conn(0x7ff3d81442b0 msgr2=0x7ff3d8144740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:21.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.705+0000 7ff3d7fff700 1 --2- 172.21.15.60:0/3629520418 >> v2:172.21.15.92:3300/0 conn(0x7ff3d81442b0 0x7ff3d8144740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:21.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.705+0000 7ff3d7fff700 1 -- 172.21.15.60:0/3629520418 >> v2:172.21.15.60:3300/0 conn(0x7ff3d805c390 msgr2=0x7ff3d80b2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:21.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.705+0000 7ff3d7fff700 1 --2- 172.21.15.60:0/3629520418 >> v2:172.21.15.60:3300/0 conn(0x7ff3d805c390 0x7ff3d80b2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:21.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.705+0000 7ff3d7fff700 1 -- 172.21.15.60:0/3629520418 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d8065f40 con 0x7ff3d8140a20 2022-06-29T16:07:21.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.705+0000 7ff3d7fff700 1 --2- 172.21.15.60:0/3629520418 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3d8140a20 0x7ff3d8140e50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7ff3d00032f0 tx=0x7ff3d00091d0).ready entity=mon.2 client_cookie=57c1d8388faeca72 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:21.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.705+0000 7ff3d6ffd700 1 -- 172.21.15.60:0/3629520418 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3d000b600 con 0x7ff3d8140a20 2022-06-29T16:07:21.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.705+0000 7ff3d6ffd700 1 -- 172.21.15.60:0/3629520418 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d000b760 con 0x7ff3d8140a20 2022-06-29T16:07:21.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.705+0000 7ff3d6ffd700 1 -- 172.21.15.60:0/3629520418 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3d000b8d0 con 0x7ff3d8140a20 2022-06-29T16:07:21.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.709+0000 7ff3ddcd2700 1 -- 172.21.15.60:0/3629520418 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3d8140a20 msgr2=0x7ff3d8140e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:21.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.709+0000 7ff3ddcd2700 1 --2- 172.21.15.60:0/3629520418 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3d8140a20 0x7ff3d8140e50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7ff3d00032f0 tx=0x7ff3d00091d0).stop 2022-06-29T16:07:21.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.709+0000 7ff3ddcd2700 1 -- 172.21.15.60:0/3629520418 shutdown_connections 2022-06-29T16:07:21.728 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.709+0000 7ff3ddcd2700 1 --2- 172.21.15.60:0/3629520418 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3d8140a20 0x7ff3d8140e50 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:21.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.709+0000 7ff3ddcd2700 1 --2- 172.21.15.60:0/3629520418 >> v2:172.21.15.92:3300/0 conn(0x7ff3d81442b0 0x7ff3d8144740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:21.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.709+0000 7ff3ddcd2700 1 --2- 172.21.15.60:0/3629520418 >> v2:172.21.15.60:3300/0 conn(0x7ff3d805c390 0x7ff3d80b2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:21.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.709+0000 7ff3ddcd2700 1 -- 172.21.15.60:0/3629520418 >> 172.21.15.60:0/3629520418 conn(0x7ff3d8064a10 msgr2=0x7ff3d8064df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:21.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.709+0000 7ff3ddcd2700 1 -- 172.21.15.60:0/3629520418 shutdown_connections 2022-06-29T16:07:21.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.709+0000 7ff3ddcd2700 1 -- 172.21.15.60:0/3629520418 wait complete. 2022-06-29T16:07:21.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.709+0000 7ff3ddcd2700 1 Processor -- start 2022-06-29T16:07:21.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.709+0000 7ff3ddcd2700 1 -- start start 2022-06-29T16:07:21.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.713+0000 7ff3ddcd2700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3d805c390 0x7ff3d80aed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:21.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.713+0000 7ff3ddcd2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3d8140a20 0x7ff3d813ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:21.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.713+0000 7ff3ddcd2700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3d81442b0 0x7ff3d813f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:21.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.713+0000 7ff3ddcd2700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d8147050 con 0x7ff3d8140a20 2022-06-29T16:07:21.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.713+0000 7ff3ddcd2700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d8146670 con 0x7ff3d805c390 2022-06-29T16:07:21.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.713+0000 7ff3ddcd2700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d8146910 con 0x7ff3d81442b0 2022-06-29T16:07:21.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.713+0000 7ff3ceffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3d8140a20 0x7ff3d813ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:21.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.713+0000 7ff3ceffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3d8140a20 0x7ff3d813ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53012/0 (socket says 172.21.15.60:53012) 2022-06-29T16:07:21.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.713+0000 7ff3ceffd700 1 -- 172.21.15.60:0/1217847480 learned_addr learned my addr 172.21.15.60:0/1217847480 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:21.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.713+0000 7ff3d7fff700 1 --2- 172.21.15.60:0/1217847480 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3d81442b0 0x7ff3d813f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:21.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.713+0000 7ff3d77fe700 1 --2- 172.21.15.60:0/1217847480 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3d805c390 0x7ff3d80aed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:21.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.713+0000 7ff3ceffd700 1 -- 172.21.15.60:0/1217847480 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3d81442b0 msgr2=0x7ff3d813f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:21.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.713+0000 7ff3ceffd700 1 --2- 172.21.15.60:0/1217847480 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3d81442b0 0x7ff3d813f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:21.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.713+0000 7ff3ceffd700 1 -- 172.21.15.60:0/1217847480 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3d805c390 msgr2=0x7ff3d80aed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:21.734 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.713+0000 7ff3ceffd700 1 --2- 172.21.15.60:0/1217847480 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3d805c390 0x7ff3d80aed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:21.734 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.713+0000 7ff3ceffd700 1 -- 172.21.15.60:0/1217847480 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d0009470 con 0x7ff3d8140a20 2022-06-29T16:07:21.734 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.713+0000 7ff3ceffd700 1 --2- 172.21.15.60:0/1217847480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3d8140a20 0x7ff3d813ed30 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 rx=0x7ff3c000c000 tx=0x7ff3c0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:21.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.713+0000 7ff3d57fa700 1 -- 172.21.15.60:0/1217847480 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3c0010780 con 0x7ff3d8140a20 2022-06-29T16:07:21.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.713+0000 7ff3d57fa700 1 -- 172.21.15.60:0/1217847480 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c00108e0 con 0x7ff3d8140a20 2022-06-29T16:07:21.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.713+0000 7ff3ddcd2700 1 -- 172.21.15.60:0/1217847480 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3d8146bb0 con 0x7ff3d8140a20 2022-06-29T16:07:21.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.713+0000 7ff3d57fa700 1 -- 172.21.15.60:0/1217847480 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3c0010c40 con 0x7ff3d8140a20 2022-06-29T16:07:21.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.713+0000 7ff3ddcd2700 1 -- 172.21.15.60:0/1217847480 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3d80af250 con 0x7ff3d8140a20 2022-06-29T16:07:21.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.717+0000 7ff3d57fa700 1 -- 172.21.15.60:0/1217847480 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff3c001c460 con 0x7ff3d8140a20 2022-06-29T16:07:21.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.717+0000 7ff3ddcd2700 1 -- 172.21.15.60:0/1217847480 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3d80573d0 con 0x7ff3d8140a20 2022-06-29T16:07:21.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.717+0000 7ff3d57fa700 1 --2- 172.21.15.60:0/1217847480 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff3bc053b00 0x7ff3bc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:21.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.717+0000 7ff3d57fa700 1 -- 172.21.15.60:0/1217847480 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff3c006c2e0 con 0x7ff3d8140a20 2022-06-29T16:07:21.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.717+0000 7ff3d77fe700 1 --2- 172.21.15.60:0/1217847480 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff3bc053b00 0x7ff3bc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:21.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.717+0000 7ff3d77fe700 1 --2- 172.21.15.60:0/1217847480 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff3bc053b00 0x7ff3bc055fc0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0x7ff3c8008000 tx=0x7ff3c8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:21.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.721+0000 7ff3d57fa700 1 -- 172.21.15.60:0/1217847480 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff3c00431a0 con 0x7ff3d8140a20 2022-06-29T16:07:21.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.889+0000 7ff3ddcd2700 1 -- 172.21.15.60:0/1217847480 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7ff3d80684b0 con 0x7ff3d8140a20 2022-06-29T16:07:21.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.897+0000 7ff3d57fa700 1 -- 172.21.15.60:0/1217847480 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v43) v1 ==== 184+0+0 (secure 0 0 0) 0x7ff3c00467c0 con 0x7ff3d8140a20 2022-06-29T16:07:21.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.901+0000 7ff3ddcd2700 1 -- 172.21.15.60:0/1217847480 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff3bc053b00 msgr2=0x7ff3bc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:21.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.901+0000 7ff3ddcd2700 1 --2- 172.21.15.60:0/1217847480 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff3bc053b00 0x7ff3bc055fc0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0x7ff3c8008000 tx=0x7ff3c8008040).stop 2022-06-29T16:07:21.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.901+0000 7ff3ddcd2700 1 -- 172.21.15.60:0/1217847480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3d8140a20 msgr2=0x7ff3d813ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:21.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.901+0000 7ff3ddcd2700 1 --2- 172.21.15.60:0/1217847480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3d8140a20 0x7ff3d813ed30 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 rx=0x7ff3c000c000 tx=0x7ff3c0006df0).stop 2022-06-29T16:07:21.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.901+0000 7ff3ddcd2700 1 -- 172.21.15.60:0/1217847480 shutdown_connections 2022-06-29T16:07:21.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.901+0000 7ff3ddcd2700 1 --2- 172.21.15.60:0/1217847480 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff3bc053b00 0x7ff3bc055fc0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:21.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.901+0000 7ff3ddcd2700 1 --2- 172.21.15.60:0/1217847480 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff3d81442b0 0x7ff3d813f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:21.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.901+0000 7ff3ddcd2700 1 --2- 172.21.15.60:0/1217847480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3d8140a20 0x7ff3d813ed30 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:21.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.901+0000 7ff3ddcd2700 1 --2- 172.21.15.60:0/1217847480 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff3d805c390 0x7ff3d80aed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:21.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.901+0000 7ff3ddcd2700 1 -- 172.21.15.60:0/1217847480 >> 172.21.15.60:0/1217847480 conn(0x7ff3d8064a10 msgr2=0x7ff3d80b64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:21.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.901+0000 7ff3ddcd2700 1 -- 172.21.15.60:0/1217847480 shutdown_connections 2022-06-29T16:07:21.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:21.901+0000 7ff3ddcd2700 1 -- 172.21.15.60:0/1217847480 wait complete. 2022-06-29T16:07:21.908 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.2 2022-06-29T16:07:21.916 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:07:22.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.009+0000 7fbef8c96700 1 Processor -- start 2022-06-29T16:07:22.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.009+0000 7fbef8c96700 1 -- start start 2022-06-29T16:07:22.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.009+0000 7fbef8c96700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbef40b61d0 0x7fbef40b0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:22.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.009+0000 7fbef8c96700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbef405bd20 0x7fbef413f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:22.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.009+0000 7fbef8c96700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbef405c160 0x7fbef413bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:22.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.009+0000 7fbef8c96700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbef4142820 con 0x7fbef405c160 2022-06-29T16:07:22.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.009+0000 7fbef8c96700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbef4062bb0 con 0x7fbef40b61d0 2022-06-29T16:07:22.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.009+0000 7fbef8c96700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbef40b65b0 con 0x7fbef405bd20 2022-06-29T16:07:22.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.009+0000 7fbef2d9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbef405c160 0x7fbef413bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:22.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.009+0000 7fbef2d9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbef405c160 0x7fbef413bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53022/0 (socket says 172.21.15.60:53022) 2022-06-29T16:07:22.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.009+0000 7fbef2d9d700 1 -- 172.21.15.60:0/2189280205 learned_addr learned my addr 172.21.15.60:0/2189280205 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:22.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.009+0000 7fbef1d9b700 1 --2- 172.21.15.60:0/2189280205 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbef405bd20 0x7fbef413f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:22.019 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.009+0000 7fbef259c700 1 --2- 172.21.15.60:0/2189280205 >> v2:172.21.15.92:3300/0 conn(0x7fbef40b61d0 0x7fbef40b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:22.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.009+0000 7fbef2d9d700 1 -- 172.21.15.60:0/2189280205 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbef405bd20 msgr2=0x7fbef413f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:22.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.009+0000 7fbef2d9d700 1 --2- 172.21.15.60:0/2189280205 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbef405bd20 0x7fbef413f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:22.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.009+0000 7fbef2d9d700 1 -- 172.21.15.60:0/2189280205 >> v2:172.21.15.92:3300/0 conn(0x7fbef40b61d0 msgr2=0x7fbef40b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:22.020 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.009+0000 7fbef2d9d700 1 --2- 172.21.15.60:0/2189280205 >> v2:172.21.15.92:3300/0 conn(0x7fbef40b61d0 0x7fbef40b0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:22.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.009+0000 7fbef2d9d700 1 -- 172.21.15.60:0/2189280205 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef413f050 con 0x7fbef405c160 2022-06-29T16:07:22.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.009+0000 7fbef259c700 1 --2- 172.21.15.60:0/2189280205 >> v2:172.21.15.92:3300/0 conn(0x7fbef40b61d0 0x7fbef40b0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:22.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.013+0000 7fbef2d9d700 1 --2- 172.21.15.60:0/2189280205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbef405c160 0x7fbef413bc40 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 rx=0x7fbef4060670 tx=0x7fbeec009290).ready entity=mon.0 client_cookie=c83ebf2efdfbae7a server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:22.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.013+0000 7fbef159a700 1 -- 172.21.15.60:0/2189280205 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbeec00b830 con 0x7fbef405c160 2022-06-29T16:07:22.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.013+0000 7fbef159a700 1 -- 172.21.15.60:0/2189280205 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeec00b990 con 0x7fbef405c160 2022-06-29T16:07:22.022 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.013+0000 7fbef159a700 1 -- 172.21.15.60:0/2189280205 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbeec00bb00 con 0x7fbef405c160 2022-06-29T16:07:22.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.013+0000 7fbef8c96700 1 -- 172.21.15.60:0/2189280205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbef405c160 msgr2=0x7fbef413bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:22.030 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.013+0000 7fbef8c96700 1 --2- 172.21.15.60:0/2189280205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbef405c160 0x7fbef413bc40 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 rx=0x7fbef4060670 tx=0x7fbeec009290).stop 2022-06-29T16:07:22.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.013+0000 7fbef8c96700 1 -- 172.21.15.60:0/2189280205 shutdown_connections 2022-06-29T16:07:22.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.013+0000 7fbef8c96700 1 --2- 172.21.15.60:0/2189280205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbef405c160 0x7fbef413bc40 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:22.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.013+0000 7fbef8c96700 1 --2- 172.21.15.60:0/2189280205 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbef405bd20 0x7fbef413f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:22.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.013+0000 7fbef8c96700 1 --2- 172.21.15.60:0/2189280205 >> v2:172.21.15.92:3300/0 conn(0x7fbef40b61d0 0x7fbef40b0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:22.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.013+0000 7fbef8c96700 1 -- 172.21.15.60:0/2189280205 >> 172.21.15.60:0/2189280205 conn(0x7fbef4063b50 msgr2=0x7fbef40b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:22.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.013+0000 7fbef8c96700 1 -- 172.21.15.60:0/2189280205 shutdown_connections 2022-06-29T16:07:22.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.013+0000 7fbef8c96700 1 -- 172.21.15.60:0/2189280205 wait complete. 2022-06-29T16:07:22.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.013+0000 7fbef8c96700 1 Processor -- start 2022-06-29T16:07:22.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.013+0000 7fbef8c96700 1 -- start start 2022-06-29T16:07:22.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.017+0000 7fbef8c96700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbef405bd20 0x7fbef4151b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:22.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.017+0000 7fbef8c96700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbef405c160 0x7fbef4141c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:22.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.017+0000 7fbef8c96700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbef40b61d0 0x7fbef4155440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:22.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.017+0000 7fbef8c96700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef4155050 con 0x7fbef40b61d0 2022-06-29T16:07:22.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.017+0000 7fbef8c96700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef40af3e0 con 0x7fbef405bd20 2022-06-29T16:07:22.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.017+0000 7fbef8c96700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbef40af680 con 0x7fbef405c160 2022-06-29T16:07:22.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.017+0000 7fbef2d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbef40b61d0 0x7fbef4155440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:22.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.017+0000 7fbef2d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbef40b61d0 0x7fbef4155440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53028/0 (socket says 172.21.15.60:53028) 2022-06-29T16:07:22.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.017+0000 7fbef2d9d700 1 -- 172.21.15.60:0/730474222 learned_addr learned my addr 172.21.15.60:0/730474222 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:22.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.017+0000 7fbef259c700 1 --2- 172.21.15.60:0/730474222 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbef405bd20 0x7fbef4151b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:22.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.017+0000 7fbef1d9b700 1 --2- 172.21.15.60:0/730474222 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbef405c160 0x7fbef4141c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:22.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.017+0000 7fbef2d9d700 1 -- 172.21.15.60:0/730474222 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbef405c160 msgr2=0x7fbef4141c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:22.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.017+0000 7fbef2d9d700 1 --2- 172.21.15.60:0/730474222 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbef405c160 0x7fbef4141c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:22.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.017+0000 7fbef2d9d700 1 -- 172.21.15.60:0/730474222 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbef405bd20 msgr2=0x7fbef4151b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:22.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.017+0000 7fbef2d9d700 1 --2- 172.21.15.60:0/730474222 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbef405bd20 0x7fbef4151b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:22.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.017+0000 7fbef2d9d700 1 -- 172.21.15.60:0/730474222 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeec009530 con 0x7fbef40b61d0 2022-06-29T16:07:22.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.017+0000 7fbef2d9d700 1 --2- 172.21.15.60:0/730474222 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbef40b61d0 0x7fbef4155440 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 rx=0x7fbef4060670 tx=0x7fbeec016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:22.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.017+0000 7fbeeb7fe700 1 -- 172.21.15.60:0/730474222 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbeec00b830 con 0x7fbef40b61d0 2022-06-29T16:07:22.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.017+0000 7fbeeb7fe700 1 -- 172.21.15.60:0/730474222 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeec0032f0 con 0x7fbef40b61d0 2022-06-29T16:07:22.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.017+0000 7fbef8c96700 1 -- 172.21.15.60:0/730474222 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbef40af920 con 0x7fbef40b61d0 2022-06-29T16:07:22.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.017+0000 7fbeeb7fe700 1 -- 172.21.15.60:0/730474222 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbeec020620 con 0x7fbef40b61d0 2022-06-29T16:07:22.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.017+0000 7fbef8c96700 1 -- 172.21.15.60:0/730474222 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbef4064290 con 0x7fbef40b61d0 2022-06-29T16:07:22.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.017+0000 7fbef8c96700 1 -- 172.21.15.60:0/730474222 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbef4064f60 con 0x7fbef40b61d0 2022-06-29T16:07:22.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.021+0000 7fbeeb7fe700 1 -- 172.21.15.60:0/730474222 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fbeec028070 con 0x7fbef40b61d0 2022-06-29T16:07:22.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.021+0000 7fbeeb7fe700 1 --2- 172.21.15.60:0/730474222 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbed8053a10 0x7fbed8055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:22.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.021+0000 7fbeeb7fe700 1 -- 172.21.15.60:0/730474222 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbeec06c5d0 con 0x7fbef40b61d0 2022-06-29T16:07:22.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.021+0000 7fbef259c700 1 --2- 172.21.15.60:0/730474222 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbed8053a10 0x7fbed8055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:22.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.021+0000 7fbef259c700 1 --2- 172.21.15.60:0/730474222 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbed8053a10 0x7fbed8055ed0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7fbee400c000 tx=0x7fbee4008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:22.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.029+0000 7fbeeb7fe700 1 -- 172.21.15.60:0/730474222 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbef4064f60 con 0x7fbef40b61d0 2022-06-29T16:07:22.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.197+0000 7fbef8c96700 1 -- 172.21.15.60:0/730474222 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbef40afab0 con 0x7fbef40b61d0 2022-06-29T16:07:22.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.197+0000 7fbeeb7fe700 1 -- 172.21.15.60:0/730474222 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2142 (secure 0 0 0) 0x7fbeec015020 con 0x7fbef40b61d0 2022-06-29T16:07:22.203 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:22.203 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":62,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6941,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/3488623391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3488623391},{"type":"v1","addr":"172.21.15.92:6835","nonce":3488623391}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":59}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:07:19.341489+0000","modified":"2022-06-29T16:07:20.345065+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8085},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8085":{"gid":8085,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/3877729627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3877729627},{"type":"v1","addr":"172.21.15.60:6833","nonce":3877729627}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-06-29T16:07:22.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.201+0000 7fbef8c96700 1 -- 172.21.15.60:0/730474222 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbed8053a10 msgr2=0x7fbed8055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:22.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.201+0000 7fbef8c96700 1 --2- 172.21.15.60:0/730474222 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbed8053a10 0x7fbed8055ed0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7fbee400c000 tx=0x7fbee4008040).stop 2022-06-29T16:07:22.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.201+0000 7fbef8c96700 1 -- 172.21.15.60:0/730474222 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbef40b61d0 msgr2=0x7fbef4155440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:22.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.201+0000 7fbef8c96700 1 --2- 172.21.15.60:0/730474222 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbef40b61d0 0x7fbef4155440 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 rx=0x7fbef4060670 tx=0x7fbeec016000).stop 2022-06-29T16:07:22.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.201+0000 7fbef8c96700 1 -- 172.21.15.60:0/730474222 shutdown_connections 2022-06-29T16:07:22.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.201+0000 7fbef8c96700 1 --2- 172.21.15.60:0/730474222 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbed8053a10 0x7fbed8055ed0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:22.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.201+0000 7fbef8c96700 1 --2- 172.21.15.60:0/730474222 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbef40b61d0 0x7fbef4155440 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:22.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.201+0000 7fbef8c96700 1 --2- 172.21.15.60:0/730474222 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbef405c160 0x7fbef4141c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:22.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.201+0000 7fbef8c96700 1 --2- 172.21.15.60:0/730474222 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbef405bd20 0x7fbef4151b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:22.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.201+0000 7fbef8c96700 1 -- 172.21.15.60:0/730474222 >> 172.21.15.60:0/730474222 conn(0x7fbef4063b50 msgr2=0x7fbef40b0540 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:22.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.201+0000 7fbef8c96700 1 -- 172.21.15.60:0/730474222 shutdown_connections 2022-06-29T16:07:22.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.201+0000 7fbef8c96700 1 -- 172.21.15.60:0/730474222 wait complete. 2022-06-29T16:07:22.208 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 62 2022-06-29T16:07:22.417 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:07:19.341489+0000', 'modified': '2022-06-29T16:07:20.345065+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8085}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8085': {'gid': 8085, 'name': 'a', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.60:6833/3877729627', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6832', 'nonce': 3877729627}, {'type': 'v1', 'addr': '172.21.15.60:6833', 'nonce': 3877729627}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:07:22.418 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:07:22.418 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-06-29T16:07:22.538 INFO:teuthology.orchestra.run.smithi060.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":8085,"want_state":"up:active","state":"up:active","rank_uptime":3.1812482260000001,"mdsmap_epoch":62,"osdmap_epoch":128,"osdmap_epoch_barrier":128,"uptime":7.9062979179999999} 2022-06-29T16:07:22.539 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 8085, "mdsmap_epoch": 62, "osdmap_epoch": 128, "osdmap_epoch_barrier": 128, "rank_uptime": 3.181248226, "state": "up:active", "uptime": 7.906297918, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:07:22.539 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:07:22.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.641+0000 7f4bebb4a700 1 Processor -- start 2022-06-29T16:07:22.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.641+0000 7f4bebb4a700 1 -- start start 2022-06-29T16:07:22.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.641+0000 7f4bebb4a700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4be40b5950 0x7f4be4067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:22.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.641+0000 7f4bebb4a700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4be4063ad0 0x7f4be414af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:22.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.641+0000 7f4bebb4a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4be405bd20 0x7f4be405c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:22.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.641+0000 7f4bebb4a700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4be40acba0 con 0x7f4be4063ad0 2022-06-29T16:07:22.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.641+0000 7f4bebb4a700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4be4067220 con 0x7f4be40b5950 2022-06-29T16:07:22.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.641+0000 7f4bebb4a700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4be40b5d30 con 0x7f4be405bd20 2022-06-29T16:07:22.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.641+0000 7f4be90e5700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4be4063ad0 0x7f4be414af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:22.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.641+0000 7f4be90e5700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4be4063ad0 0x7f4be414af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53032/0 (socket says 172.21.15.60:53032) 2022-06-29T16:07:22.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.641+0000 7f4be90e5700 1 -- 172.21.15.60:0/189867476 learned_addr learned my addr 172.21.15.60:0/189867476 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:22.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.641+0000 7f4bea0e7700 1 --2- 172.21.15.60:0/189867476 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4be405bd20 0x7f4be405c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:22.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.641+0000 7f4be98e6700 1 --2- 172.21.15.60:0/189867476 >> v2:172.21.15.92:3300/0 conn(0x7f4be40b5950 0x7f4be4067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:22.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.641+0000 7f4be90e5700 1 -- 172.21.15.60:0/189867476 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4be405bd20 msgr2=0x7f4be405c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:22.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.641+0000 7f4be90e5700 1 --2- 172.21.15.60:0/189867476 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4be405bd20 0x7f4be405c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:22.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.641+0000 7f4be90e5700 1 -- 172.21.15.60:0/189867476 >> v2:172.21.15.92:3300/0 conn(0x7f4be40b5950 msgr2=0x7f4be4067680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:22.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.641+0000 7f4be90e5700 1 --2- 172.21.15.60:0/189867476 >> v2:172.21.15.92:3300/0 conn(0x7f4be40b5950 0x7f4be4067680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:22.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.641+0000 7f4be90e5700 1 -- 172.21.15.60:0/189867476 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be40ace00 con 0x7f4be4063ad0 2022-06-29T16:07:22.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.641+0000 7f4be90e5700 1 --2- 172.21.15.60:0/189867476 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4be4063ad0 0x7f4be414af10 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 rx=0x7f4bd80032f0 tx=0x7f4bd8009290).ready entity=mon.0 client_cookie=d36684f511af3727 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:22.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.641+0000 7f4be88e4700 1 -- 172.21.15.60:0/189867476 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4bd800b870 con 0x7f4be4063ad0 2022-06-29T16:07:22.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.641+0000 7f4be88e4700 1 -- 172.21.15.60:0/189867476 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd800b9d0 con 0x7f4be4063ad0 2022-06-29T16:07:22.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.641+0000 7f4be88e4700 1 -- 172.21.15.60:0/189867476 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4bd800bb40 con 0x7f4be4063ad0 2022-06-29T16:07:22.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.645+0000 7f4bebb4a700 1 -- 172.21.15.60:0/189867476 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4be4063ad0 msgr2=0x7f4be414af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:22.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.645+0000 7f4bebb4a700 1 --2- 172.21.15.60:0/189867476 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4be4063ad0 0x7f4be414af10 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 rx=0x7f4bd80032f0 tx=0x7f4bd8009290).stop 2022-06-29T16:07:22.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.645+0000 7f4bebb4a700 1 -- 172.21.15.60:0/189867476 shutdown_connections 2022-06-29T16:07:22.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.645+0000 7f4bebb4a700 1 --2- 172.21.15.60:0/189867476 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4be405bd20 0x7f4be405c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:22.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.645+0000 7f4bebb4a700 1 --2- 172.21.15.60:0/189867476 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4be4063ad0 0x7f4be414af10 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:22.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.645+0000 7f4bebb4a700 1 --2- 172.21.15.60:0/189867476 >> v2:172.21.15.92:3300/0 conn(0x7f4be40b5950 0x7f4be4067680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:22.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.645+0000 7f4bebb4a700 1 -- 172.21.15.60:0/189867476 >> 172.21.15.60:0/189867476 conn(0x7f4be40632c0 msgr2=0x7f4be40b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:22.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.645+0000 7f4bebb4a700 1 -- 172.21.15.60:0/189867476 shutdown_connections 2022-06-29T16:07:22.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.645+0000 7f4bebb4a700 1 -- 172.21.15.60:0/189867476 wait complete. 2022-06-29T16:07:22.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.645+0000 7f4bebb4a700 1 Processor -- start 2022-06-29T16:07:22.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.645+0000 7f4bebb4a700 1 -- start start 2022-06-29T16:07:22.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.645+0000 7f4bebb4a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4be405bd20 0x7f4be4064c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:22.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.645+0000 7f4bebb4a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4be4063ad0 0x7f4be4159410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:22.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.645+0000 7f4bebb4a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4be40b5950 0x7f4be4116c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:22.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.645+0000 7f4bebb4a700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be40adfb0 con 0x7f4be405bd20 2022-06-29T16:07:22.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.645+0000 7f4bebb4a700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be40ada70 con 0x7f4be4063ad0 2022-06-29T16:07:22.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.645+0000 7f4bebb4a700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4be40add10 con 0x7f4be40b5950 2022-06-29T16:07:22.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.645+0000 7f4be98e6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4be405bd20 0x7f4be4064c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:22.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.645+0000 7f4be98e6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4be405bd20 0x7f4be4064c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53038/0 (socket says 172.21.15.60:53038) 2022-06-29T16:07:22.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.645+0000 7f4be98e6700 1 -- 172.21.15.60:0/1715545720 learned_addr learned my addr 172.21.15.60:0/1715545720 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:22.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.645+0000 7f4bea0e7700 1 --2- 172.21.15.60:0/1715545720 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4be40b5950 0x7f4be4116c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:22.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.645+0000 7f4be90e5700 1 --2- 172.21.15.60:0/1715545720 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4be4063ad0 0x7f4be4159410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:22.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.645+0000 7f4be98e6700 1 -- 172.21.15.60:0/1715545720 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4be40b5950 msgr2=0x7f4be4116c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:22.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.645+0000 7f4be98e6700 1 --2- 172.21.15.60:0/1715545720 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4be40b5950 0x7f4be4116c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:22.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.645+0000 7f4be98e6700 1 -- 172.21.15.60:0/1715545720 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4be4063ad0 msgr2=0x7f4be4159410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:22.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.645+0000 7f4be98e6700 1 --2- 172.21.15.60:0/1715545720 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4be4063ad0 0x7f4be4159410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:22.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.645+0000 7f4be98e6700 1 -- 172.21.15.60:0/1715545720 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd8009530 con 0x7f4be405bd20 2022-06-29T16:07:22.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.649+0000 7f4be98e6700 1 --2- 172.21.15.60:0/1715545720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4be405bd20 0x7f4be4064c80 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 rx=0x7f4be00030b0 tx=0x7f4be0008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:22.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.649+0000 7f4bd6ffd700 1 -- 172.21.15.60:0/1715545720 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4be0011780 con 0x7f4be405bd20 2022-06-29T16:07:22.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.649+0000 7f4bd6ffd700 1 -- 172.21.15.60:0/1715545720 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4be00118e0 con 0x7f4be405bd20 2022-06-29T16:07:22.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.649+0000 7f4bebb4a700 1 -- 172.21.15.60:0/1715545720 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4be40ad5d0 con 0x7f4be405bd20 2022-06-29T16:07:22.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.649+0000 7f4bd6ffd700 1 -- 172.21.15.60:0/1715545720 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4be0011c40 con 0x7f4be405bd20 2022-06-29T16:07:22.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.649+0000 7f4bebb4a700 1 -- 172.21.15.60:0/1715545720 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4be40ae950 con 0x7f4be405bd20 2022-06-29T16:07:22.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.649+0000 7f4bebb4a700 1 -- 172.21.15.60:0/1715545720 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4be4116950 con 0x7f4be405bd20 2022-06-29T16:07:22.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.649+0000 7f4bd6ffd700 1 -- 172.21.15.60:0/1715545720 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4be0012420 con 0x7f4be405bd20 2022-06-29T16:07:22.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.649+0000 7f4bd6ffd700 1 --2- 172.21.15.60:0/1715545720 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4bc4053a70 0x7f4bc4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:22.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.649+0000 7f4bd6ffd700 1 -- 172.21.15.60:0/1715545720 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4be006bd40 con 0x7f4be405bd20 2022-06-29T16:07:22.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.649+0000 7f4be90e5700 1 --2- 172.21.15.60:0/1715545720 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4bc4053a70 0x7f4bc4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:22.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.653+0000 7f4be90e5700 1 --2- 172.21.15.60:0/1715545720 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4bc4053a70 0x7f4bc4055f30 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7f4bd801b000 tx=0x7f4bd8016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:22.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.657+0000 7f4bd6ffd700 1 -- 172.21.15.60:0/1715545720 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4be4116950 con 0x7f4be405bd20 2022-06-29T16:07:22.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.781+0000 7f4bebb4a700 1 -- 172.21.15.60:0/1715545720 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4be4116900 con 0x7f4be405bd20 2022-06-29T16:07:22.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.785+0000 7f4bd6ffd700 1 -- 172.21.15.60:0/1715545720 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) v1 ==== 74+0+16052 (secure 0 0 0) 0x7f4be0042af0 con 0x7f4be405bd20 2022-06-29T16:07:22.790 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:22.791 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":128,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:07:20.344096+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:07:17.372006+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2022-06-29T16:07:18.349364+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:07:22.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.785+0000 7f4bebb4a700 1 -- 172.21.15.60:0/1715545720 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4bc4053a70 msgr2=0x7f4bc4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:22.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.785+0000 7f4bebb4a700 1 --2- 172.21.15.60:0/1715545720 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4bc4053a70 0x7f4bc4055f30 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7f4bd801b000 tx=0x7f4bd8016000).stop 2022-06-29T16:07:22.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.785+0000 7f4bebb4a700 1 -- 172.21.15.60:0/1715545720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4be405bd20 msgr2=0x7f4be4064c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:22.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.785+0000 7f4bebb4a700 1 --2- 172.21.15.60:0/1715545720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4be405bd20 0x7f4be4064c80 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 rx=0x7f4be00030b0 tx=0x7f4be0008040).stop 2022-06-29T16:07:22.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.785+0000 7f4bebb4a700 1 -- 172.21.15.60:0/1715545720 shutdown_connections 2022-06-29T16:07:22.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.785+0000 7f4bebb4a700 1 --2- 172.21.15.60:0/1715545720 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4bc4053a70 0x7f4bc4055f30 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:22.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.785+0000 7f4bebb4a700 1 --2- 172.21.15.60:0/1715545720 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4be40b5950 0x7f4be4116c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:22.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.785+0000 7f4bebb4a700 1 --2- 172.21.15.60:0/1715545720 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4be4063ad0 0x7f4be4159410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:22.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.785+0000 7f4bebb4a700 1 --2- 172.21.15.60:0/1715545720 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4be405bd20 0x7f4be4064c80 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:22.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.785+0000 7f4bebb4a700 1 -- 172.21.15.60:0/1715545720 >> 172.21.15.60:0/1715545720 conn(0x7f4be40632c0 msgr2=0x7f4be41456c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:22.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.785+0000 7f4bebb4a700 1 -- 172.21.15.60:0/1715545720 shutdown_connections 2022-06-29T16:07:22.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.789+0000 7f4bebb4a700 1 -- 172.21.15.60:0/1715545720 wait complete. 2022-06-29T16:07:22.803 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-29T16:07:17.372006+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2022-06-29T16:07:18.349364+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '128', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-29T16:07:22.804 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:07:22.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.913+0000 7fee4782f700 1 Processor -- start 2022-06-29T16:07:22.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.913+0000 7fee4782f700 1 -- start start 2022-06-29T16:07:22.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.913+0000 7fee4782f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee40141780 0x7fee4005cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:22.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.913+0000 7fee4782f700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fee4005f0b0 0x7fee4013fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:22.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.913+0000 7fee4782f700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee400afe00 con 0x7fee400afa20 2022-06-29T16:07:22.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.913+0000 7fee4782f700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee40140da0 con 0x7fee4005f0b0 2022-06-29T16:07:22.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.913+0000 7fee4782f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee40141040 con 0x7fee40141780 2022-06-29T16:07:22.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.913+0000 7fee455cb700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fee400afa20 0x7fee4005c460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:32900/0 (socket says 172.21.15.60:32900) 2022-06-29T16:07:22.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.913+0000 7fee455cb700 1 -- 172.21.15.60:0/2858616684 learned_addr learned my addr 172.21.15.60:0/2858616684 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:22.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.913+0000 7fee44dca700 1 --2- 172.21.15.60:0/2858616684 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee40141780 0x7fee4005cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:22.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.913+0000 7fee45dcc700 1 --2- 172.21.15.60:0/2858616684 >> v2:172.21.15.92:3300/0 conn(0x7fee4005f0b0 0x7fee4013fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:22.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.913+0000 7fee37fff700 1 -- 172.21.15.60:0/2858616684 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3548563140 0 0) 0x7fee400afe00 con 0x7fee400afa20 2022-06-29T16:07:22.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.913+0000 7fee37fff700 1 -- 172.21.15.60:0/2858616684 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee24003260 con 0x7fee400afa20 2022-06-29T16:07:22.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.913+0000 7fee45dcc700 1 -- 172.21.15.60:0/2858616684 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee40141780 msgr2=0x7fee4005cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:22.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.913+0000 7fee45dcc700 1 --2- 172.21.15.60:0/2858616684 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee40141780 0x7fee4005cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:22.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.913+0000 7fee45dcc700 1 -- 172.21.15.60:0/2858616684 >> v1:172.21.15.60:6789/0 conn(0x7fee400afa20 legacy=0x7fee4005c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:22.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.913+0000 7fee45dcc700 1 -- 172.21.15.60:0/2858616684 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee401412e0 con 0x7fee4005f0b0 2022-06-29T16:07:22.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.913+0000 7fee44dca700 1 --2- 172.21.15.60:0/2858616684 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee40141780 0x7fee4005cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:07:22.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.913+0000 7fee45dcc700 1 --2- 172.21.15.60:0/2858616684 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee4005f0b0 0x7fee4013fe10 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7fee380032f0 tx=0x7fee380091d0).ready entity=mon.1 client_cookie=bf1edb98094d54d1 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:22.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.913+0000 7fee37fff700 1 -- 172.21.15.60:0/2858616684 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee3800b600 con 0x7fee4005f0b0 2022-06-29T16:07:22.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.913+0000 7fee37fff700 1 -- 172.21.15.60:0/2858616684 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee3800b760 con 0x7fee4005f0b0 2022-06-29T16:07:22.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.913+0000 7fee37fff700 1 -- 172.21.15.60:0/2858616684 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee3800b8d0 con 0x7fee4005f0b0 2022-06-29T16:07:22.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.917+0000 7fee4782f700 1 -- 172.21.15.60:0/2858616684 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee4005f0b0 msgr2=0x7fee4013fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:22.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.917+0000 7fee4782f700 1 --2- 172.21.15.60:0/2858616684 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee4005f0b0 0x7fee4013fe10 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7fee380032f0 tx=0x7fee380091d0).stop 2022-06-29T16:07:22.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.917+0000 7fee4782f700 1 -- 172.21.15.60:0/2858616684 shutdown_connections 2022-06-29T16:07:22.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.917+0000 7fee4782f700 1 --2- 172.21.15.60:0/2858616684 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee4005f0b0 0x7fee4013fe10 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:22.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.917+0000 7fee4782f700 1 --2- 172.21.15.60:0/2858616684 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee40141780 0x7fee4005cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:22.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.917+0000 7fee4782f700 1 -- 172.21.15.60:0/2858616684 >> 172.21.15.60:0/2858616684 conn(0x7fee40061f30 msgr2=0x7fee400639e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:22.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.917+0000 7fee4782f700 1 -- 172.21.15.60:0/2858616684 shutdown_connections 2022-06-29T16:07:22.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.917+0000 7fee4782f700 1 -- 172.21.15.60:0/2858616684 wait complete. 2022-06-29T16:07:22.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.917+0000 7fee4782f700 1 Processor -- start 2022-06-29T16:07:22.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.917+0000 7fee4782f700 1 -- start start 2022-06-29T16:07:22.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.917+0000 7fee4782f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee4005f0b0 0x7fee4013d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:22.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.917+0000 7fee4782f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee400afa20 0x7fee40154fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:22.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.917+0000 7fee4782f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee40141780 0x7fee40145970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:22.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.917+0000 7fee4782f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee400ae5e0 con 0x7fee4005f0b0 2022-06-29T16:07:22.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.917+0000 7fee4782f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee40145540 con 0x7fee40141780 2022-06-29T16:07:22.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.917+0000 7fee4782f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee400ae810 con 0x7fee400afa20 2022-06-29T16:07:22.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.917+0000 7fee455cb700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee4005f0b0 0x7fee4013d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:22.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.917+0000 7fee45dcc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee40141780 0x7fee40145970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:22.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.917+0000 7fee44dca700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee400afa20 0x7fee40154fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:22.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.917+0000 7fee45dcc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee40141780 0x7fee40145970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:59082/0 (socket says 172.21.15.60:59082) 2022-06-29T16:07:22.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.917+0000 7fee45dcc700 1 -- 172.21.15.60:0/2602401545 learned_addr learned my addr 172.21.15.60:0/2602401545 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:22.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.917+0000 7fee45dcc700 1 -- 172.21.15.60:0/2602401545 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee400afa20 msgr2=0x7fee40154fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:22.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.917+0000 7fee45dcc700 1 --2- 172.21.15.60:0/2602401545 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee400afa20 0x7fee40154fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:22.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.917+0000 7fee45dcc700 1 -- 172.21.15.60:0/2602401545 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee4005f0b0 msgr2=0x7fee4013d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:22.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.917+0000 7fee45dcc700 1 --2- 172.21.15.60:0/2602401545 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee4005f0b0 0x7fee4013d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:22.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.917+0000 7fee45dcc700 1 -- 172.21.15.60:0/2602401545 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee3000b180 con 0x7fee40141780 2022-06-29T16:07:22.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.917+0000 7fee45dcc700 1 --2- 172.21.15.60:0/2602401545 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee40141780 0x7fee40145970 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7fee40064b10 tx=0x7fee38016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:22.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.917+0000 7fee367fc700 1 -- 172.21.15.60:0/2602401545 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee3800b600 con 0x7fee40141780 2022-06-29T16:07:22.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.917+0000 7fee367fc700 1 -- 172.21.15.60:0/2602401545 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee38011b50 con 0x7fee40141780 2022-06-29T16:07:22.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.917+0000 7fee4782f700 1 -- 172.21.15.60:0/2602401545 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee38009470 con 0x7fee40141780 2022-06-29T16:07:22.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.917+0000 7fee367fc700 1 -- 172.21.15.60:0/2602401545 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee3801f3f0 con 0x7fee40141780 2022-06-29T16:07:22.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.917+0000 7fee4782f700 1 -- 172.21.15.60:0/2602401545 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee40145120 con 0x7fee40141780 2022-06-29T16:07:22.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.921+0000 7fee4782f700 1 -- 172.21.15.60:0/2602401545 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee401456b0 con 0x7fee40141780 2022-06-29T16:07:22.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.921+0000 7fee367fc700 1 -- 172.21.15.60:0/2602401545 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fee38026030 con 0x7fee40141780 2022-06-29T16:07:22.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.921+0000 7fee367fc700 1 --2- 172.21.15.60:0/2602401545 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fee28053b00 0x7fee28055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:22.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.921+0000 7fee367fc700 1 -- 172.21.15.60:0/2602401545 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fee3806ca00 con 0x7fee40141780 2022-06-29T16:07:22.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.921+0000 7fee455cb700 1 --2- 172.21.15.60:0/2602401545 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fee28053b00 0x7fee28055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:22.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.921+0000 7fee455cb700 1 --2- 172.21.15.60:0/2602401545 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fee28053b00 0x7fee28055fc0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7fee3c00d000 tx=0x7fee3c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:22.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:22.929+0000 7fee367fc700 1 -- 172.21.15.60:0/2602401545 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fee401456b0 con 0x7fee40141780 2022-06-29T16:07:23.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.053+0000 7fee4782f700 1 -- 172.21.15.60:0/2602401545 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fee40145680 con 0x7fee40141780 2022-06-29T16:07:23.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.057+0000 7fee367fc700 1 -- 172.21.15.60:0/2602401545 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v128) v1 ==== 110+0+10 (secure 0 0 0) 0x7fee380240b0 con 0x7fee40141780 2022-06-29T16:07:23.062 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:07:23.064 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.057+0000 7fee4782f700 1 -- 172.21.15.60:0/2602401545 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fee28053b00 msgr2=0x7fee28055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:23.064 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.057+0000 7fee4782f700 1 --2- 172.21.15.60:0/2602401545 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fee28053b00 0x7fee28055fc0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7fee3c00d000 tx=0x7fee3c009040).stop 2022-06-29T16:07:23.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.057+0000 7fee4782f700 1 -- 172.21.15.60:0/2602401545 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee40141780 msgr2=0x7fee40145970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:23.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.057+0000 7fee4782f700 1 --2- 172.21.15.60:0/2602401545 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee40141780 0x7fee40145970 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7fee40064b10 tx=0x7fee38016000).stop 2022-06-29T16:07:23.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.061+0000 7fee4782f700 1 -- 172.21.15.60:0/2602401545 shutdown_connections 2022-06-29T16:07:23.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.061+0000 7fee4782f700 1 --2- 172.21.15.60:0/2602401545 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fee28053b00 0x7fee28055fc0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.061+0000 7fee4782f700 1 --2- 172.21.15.60:0/2602401545 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee40141780 0x7fee40145970 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.061+0000 7fee4782f700 1 --2- 172.21.15.60:0/2602401545 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fee400afa20 0x7fee40154fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.061+0000 7fee4782f700 1 --2- 172.21.15.60:0/2602401545 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee4005f0b0 0x7fee4013d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.061+0000 7fee4782f700 1 -- 172.21.15.60:0/2602401545 >> 172.21.15.60:0/2602401545 conn(0x7fee40061f30 msgr2=0x7fee4005f590 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:23.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.061+0000 7fee4782f700 1 -- 172.21.15.60:0/2602401545 shutdown_connections 2022-06-29T16:07:23.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.061+0000 7fee4782f700 1 -- 172.21.15.60:0/2602401545 wait complete. 2022-06-29T16:07:23.075 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:07:23.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f78100ef700 1 Processor -- start 2022-06-29T16:07:23.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f78100ef700 1 -- start start 2022-06-29T16:07:23.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f78100ef700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78080ae7c0 0x7f7808141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:23.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f78100ef700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7808143a40 con 0x7f78080b4010 2022-06-29T16:07:23.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f78100ef700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78080aee60 con 0x7f78080b46c0 2022-06-29T16:07:23.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f78100ef700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78080af100 con 0x7f78080ae7c0 2022-06-29T16:07:23.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f780d68a700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f78080b4010 0x7f7808142490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:32914/0 (socket says 172.21.15.60:32914) 2022-06-29T16:07:23.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f780d68a700 1 -- 172.21.15.60:0/581740681 learned_addr learned my addr 172.21.15.60:0/581740681 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:23.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f780de8b700 1 --2- 172.21.15.60:0/581740681 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78080ae7c0 0x7f7808141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:23.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f780ce89700 1 -- 172.21.15.60:0/581740681 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1104529658 0 0) 0x7f7808143a40 con 0x7f78080b4010 2022-06-29T16:07:23.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f780ce89700 1 -- 172.21.15.60:0/581740681 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77f0003260 con 0x7f78080b4010 2022-06-29T16:07:23.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f780ce89700 1 -- 172.21.15.60:0/581740681 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3594736005 0 0) 0x7f78080aee60 con 0x7f78080b46c0 2022-06-29T16:07:23.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f780ce89700 1 -- 172.21.15.60:0/581740681 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7808143a40 con 0x7f78080b46c0 2022-06-29T16:07:23.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f780ce89700 1 -- 172.21.15.60:0/581740681 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3714630873 0 0) 0x7f77f0003260 con 0x7f78080b4010 2022-06-29T16:07:23.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f780ce89700 1 -- 172.21.15.60:0/581740681 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78080ae7c0 msgr2=0x7f7808141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:23.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f780ce89700 1 --2- 172.21.15.60:0/581740681 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78080ae7c0 0x7f7808141fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f780ce89700 1 -- 172.21.15.60:0/581740681 >> v1:172.21.15.92:6789/0 conn(0x7f78080b46c0 legacy=0x7f78080b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:23.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f780ce89700 1 -- 172.21.15.60:0/581740681 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78080aeba0 con 0x7f78080b4010 2022-06-29T16:07:23.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f780ce89700 1 -- 172.21.15.60:0/581740681 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f77f8001660 con 0x7f78080b4010 2022-06-29T16:07:23.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f780ce89700 1 -- 172.21.15.60:0/581740681 >> v1:172.21.15.60:6789/0 conn(0x7f78080b4010 legacy=0x7f7808142490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:23.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f780ce89700 1 --2- 172.21.15.60:0/581740681 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77f0003910 0x7f77f0003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:23.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f780ce89700 1 --2- 172.21.15.60:0/581740681 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78080ae7c0 0x7f77f0004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:23.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f780ce89700 1 --2- 172.21.15.60:0/581740681 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77f00049f0 0x7f77f0004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:23.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f780ce89700 1 -- 172.21.15.60:0/581740681 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78080af100 con 0x7f77f0003910 2022-06-29T16:07:23.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f780ce89700 1 -- 172.21.15.60:0/581740681 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f000b780 con 0x7f77f00049f0 2022-06-29T16:07:23.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f780ce89700 1 -- 172.21.15.60:0/581740681 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77f000b940 con 0x7f78080ae7c0 2022-06-29T16:07:23.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f780de8b700 1 --2- 172.21.15.60:0/581740681 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77f0003910 0x7f77f0003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:23.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f780d68a700 1 --2- 172.21.15.60:0/581740681 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78080ae7c0 0x7f77f0004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:23.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f780e68c700 1 --2- 172.21.15.60:0/581740681 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77f00049f0 0x7f77f0004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:23.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f780de8b700 1 -- 172.21.15.60:0/581740681 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78080ae7c0 msgr2=0x7f77f0004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:23.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f780de8b700 1 --2- 172.21.15.60:0/581740681 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f78080ae7c0 0x7f77f0004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f780de8b700 1 -- 172.21.15.60:0/581740681 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77f00049f0 msgr2=0x7f77f0004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:23.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f780de8b700 1 --2- 172.21.15.60:0/581740681 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77f00049f0 0x7f77f0004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f780de8b700 1 -- 172.21.15.60:0/581740681 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f0003260 con 0x7f77f0003910 2022-06-29T16:07:23.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.181+0000 7f780e68c700 1 --2- 172.21.15.60:0/581740681 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77f00049f0 0x7f77f0004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:23.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.185+0000 7f780de8b700 1 --2- 172.21.15.60:0/581740681 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77f0003910 0x7f77f0003d10 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 rx=0x7f780400c000 tx=0x7f7804006df0).ready entity=mon.0 client_cookie=eb08d9c5f8030875 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:23.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.185+0000 7f780ce89700 1 -- 172.21.15.60:0/581740681 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f780400e830 con 0x7f77f0003910 2022-06-29T16:07:23.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.185+0000 7f780ce89700 1 -- 172.21.15.60:0/581740681 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f780400e990 con 0x7f77f0003910 2022-06-29T16:07:23.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.185+0000 7f780ce89700 1 -- 172.21.15.60:0/581740681 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f780400ecf0 con 0x7f77f0003910 2022-06-29T16:07:23.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.185+0000 7f78100ef700 1 -- 172.21.15.60:0/581740681 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77f0003910 msgr2=0x7f77f0003d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:23.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.185+0000 7f78100ef700 1 --2- 172.21.15.60:0/581740681 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f77f0003910 0x7f77f0003d10 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 rx=0x7f780400c000 tx=0x7f7804006df0).stop 2022-06-29T16:07:23.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.185+0000 7f780e68c700 1 -- 172.21.15.60:0/581740681 reap_dead start 2022-06-29T16:07:23.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.185+0000 7f78100ef700 1 -- 172.21.15.60:0/581740681 shutdown_connections 2022-06-29T16:07:23.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.185+0000 7f78100ef700 1 -- 172.21.15.60:0/581740681 >> 172.21.15.60:0/581740681 conn(0x7f78080b9340 msgr2=0x7f78080b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:23.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.185+0000 7f78100ef700 1 -- 172.21.15.60:0/581740681 shutdown_connections 2022-06-29T16:07:23.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.185+0000 7f78100ef700 1 -- 172.21.15.60:0/581740681 wait complete. 2022-06-29T16:07:23.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.185+0000 7f78100ef700 1 Processor -- start 2022-06-29T16:07:23.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.185+0000 7f78100ef700 1 -- start start 2022-06-29T16:07:23.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.185+0000 7f78100ef700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f77f0003910 0x7f78080b7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:23.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.185+0000 7f78100ef700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f780813f8d0 0x7f78080b84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:23.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.185+0000 7f78100ef700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f780813ddf0 0x7f7808137e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:23.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.185+0000 7f78100ef700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78080b89c0 con 0x7f780813f8d0 2022-06-29T16:07:23.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.185+0000 7f78100ef700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78080b8be0 con 0x7f780813ddf0 2022-06-29T16:07:23.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.185+0000 7f78100ef700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78080b8e80 con 0x7f77f0003910 2022-06-29T16:07:23.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.189+0000 7f780d68a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f780813f8d0 0x7f78080b84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:23.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.189+0000 7f780d68a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f780813f8d0 0x7f78080b84d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53074/0 (socket says 172.21.15.60:53074) 2022-06-29T16:07:23.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.189+0000 7f780d68a700 1 -- 172.21.15.60:0/263042136 learned_addr learned my addr 172.21.15.60:0/263042136 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:23.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.189+0000 7f780e68c700 1 --2- 172.21.15.60:0/263042136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f780813ddf0 0x7f7808137e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:23.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.189+0000 7f780de8b700 1 --2- 172.21.15.60:0/263042136 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f77f0003910 0x7f78080b7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:23.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.189+0000 7f780d68a700 1 -- 172.21.15.60:0/263042136 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f77f0003910 msgr2=0x7f78080b7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:23.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.189+0000 7f780d68a700 1 --2- 172.21.15.60:0/263042136 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f77f0003910 0x7f78080b7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.189+0000 7f780d68a700 1 -- 172.21.15.60:0/263042136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f780813ddf0 msgr2=0x7f7808137e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:23.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.189+0000 7f780d68a700 1 --2- 172.21.15.60:0/263042136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f780813ddf0 0x7f7808137e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.189+0000 7f780d68a700 1 -- 172.21.15.60:0/263042136 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7808138300 con 0x7f780813f8d0 2022-06-29T16:07:23.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.189+0000 7f780d68a700 1 --2- 172.21.15.60:0/263042136 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f780813f8d0 0x7f78080b84d0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 rx=0x7f77f800d000 tx=0x7f77f8001dd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:23.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.189+0000 7f77f6ffd700 1 -- 172.21.15.60:0/263042136 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77f80116b0 con 0x7f780813f8d0 2022-06-29T16:07:23.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.189+0000 7f77f6ffd700 1 -- 172.21.15.60:0/263042136 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77f8011810 con 0x7f780813f8d0 2022-06-29T16:07:23.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.189+0000 7f78100ef700 1 -- 172.21.15.60:0/263042136 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78081384f0 con 0x7f780813f8d0 2022-06-29T16:07:23.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.189+0000 7f77f6ffd700 1 -- 172.21.15.60:0/263042136 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77f8011b70 con 0x7f780813f8d0 2022-06-29T16:07:23.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.189+0000 7f78100ef700 1 -- 172.21.15.60:0/263042136 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78081557e0 con 0x7f780813f8d0 2022-06-29T16:07:23.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.189+0000 7f78100ef700 1 -- 172.21.15.60:0/263042136 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78080b9040 con 0x7f780813f8d0 2022-06-29T16:07:23.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.189+0000 7f77f6ffd700 1 -- 172.21.15.60:0/263042136 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f77f801b030 con 0x7f780813f8d0 2022-06-29T16:07:23.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.193+0000 7f77f6ffd700 1 --2- 172.21.15.60:0/263042136 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f77e8053aa0 0x7f77e8055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:23.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.193+0000 7f77f6ffd700 1 -- 172.21.15.60:0/263042136 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f77f806ba30 con 0x7f780813f8d0 2022-06-29T16:07:23.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.193+0000 7f780de8b700 1 --2- 172.21.15.60:0/263042136 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f77e8053aa0 0x7f77e8055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:23.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.193+0000 7f780de8b700 1 --2- 172.21.15.60:0/263042136 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f77e8053aa0 0x7f77e8055f60 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0x7f780401a000 tx=0x7f7804019000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:23.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.197+0000 7f77f6ffd700 1 -- 172.21.15.60:0/263042136 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f77f80428b0 con 0x7f780813f8d0 2022-06-29T16:07:23.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.321+0000 7f78100ef700 1 -- 172.21.15.60:0/263042136 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f78080573d0 con 0x7f780813f8d0 2022-06-29T16:07:23.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.325+0000 7f77f6ffd700 1 -- 172.21.15.60:0/263042136 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v128) v1 ==== 92+0+10 (secure 0 0 0) 0x7f77f8019030 con 0x7f780813f8d0 2022-06-29T16:07:23.329 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:07:23.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.325+0000 7f78100ef700 1 -- 172.21.15.60:0/263042136 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f77e8053aa0 msgr2=0x7f77e8055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:23.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.325+0000 7f78100ef700 1 --2- 172.21.15.60:0/263042136 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f77e8053aa0 0x7f77e8055f60 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0x7f780401a000 tx=0x7f7804019000).stop 2022-06-29T16:07:23.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.325+0000 7f78100ef700 1 -- 172.21.15.60:0/263042136 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f780813f8d0 msgr2=0x7f78080b84d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:23.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.325+0000 7f78100ef700 1 --2- 172.21.15.60:0/263042136 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f780813f8d0 0x7f78080b84d0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 rx=0x7f77f800d000 tx=0x7f77f8001dd0).stop 2022-06-29T16:07:23.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.325+0000 7f78100ef700 1 -- 172.21.15.60:0/263042136 shutdown_connections 2022-06-29T16:07:23.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.325+0000 7f78100ef700 1 --2- 172.21.15.60:0/263042136 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f77e8053aa0 0x7f77e8055f60 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.325+0000 7f78100ef700 1 --2- 172.21.15.60:0/263042136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f780813ddf0 0x7f7808137e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.325+0000 7f78100ef700 1 --2- 172.21.15.60:0/263042136 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f780813f8d0 0x7f78080b84d0 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.325+0000 7f78100ef700 1 --2- 172.21.15.60:0/263042136 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f77f0003910 0x7f78080b7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.325+0000 7f78100ef700 1 -- 172.21.15.60:0/263042136 >> 172.21.15.60:0/263042136 conn(0x7f78080b9340 msgr2=0x7f78081485c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:23.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.325+0000 7f78100ef700 1 -- 172.21.15.60:0/263042136 shutdown_connections 2022-06-29T16:07:23.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.325+0000 7f78100ef700 1 -- 172.21.15.60:0/263042136 wait complete. 2022-06-29T16:07:23.342 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-29T16:07:23.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.449+0000 7fd277cc2700 1 Processor -- start 2022-06-29T16:07:23.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.449+0000 7fd277cc2700 1 -- start start 2022-06-29T16:07:23.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.449+0000 7fd277cc2700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd2700ae7c0 0x7fd270141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:23.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.449+0000 7fd277cc2700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fd2700b4010 0x7fd270142490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:23.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.449+0000 7fd277cc2700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd270143a40 con 0x7fd270146160 2022-06-29T16:07:23.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.449+0000 7fd277cc2700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2700b56d0 con 0x7fd2700ae7c0 2022-06-29T16:07:23.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.449+0000 7fd277cc2700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2700aee60 con 0x7fd2700b4010 2022-06-29T16:07:23.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.449+0000 7fd275a5e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd2700ae7c0 0x7fd270141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:23.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.449+0000 7fd275a5e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd2700ae7c0 0x7fd270141fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53080/0 (socket says 172.21.15.60:53080) 2022-06-29T16:07:23.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.449+0000 7fd275a5e700 1 -- 172.21.15.60:0/287863911 learned_addr learned my addr 172.21.15.60:0/287863911 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:23.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.449+0000 7fd27525d700 1 --2- 172.21.15.60:0/287863911 >> v2:172.21.15.92:3300/0 conn(0x7fd2700b4010 0x7fd270142490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:23.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.449+0000 7fd275a5e700 1 -- 172.21.15.60:0/287863911 >> v2:172.21.15.92:3300/0 conn(0x7fd2700b4010 msgr2=0x7fd270142490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:23.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.449+0000 7fd275a5e700 1 --2- 172.21.15.60:0/287863911 >> v2:172.21.15.92:3300/0 conn(0x7fd2700b4010 0x7fd270142490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.449+0000 7fd275a5e700 1 -- 172.21.15.60:0/287863911 >> v1:172.21.15.92:6789/0 conn(0x7fd270146160 legacy=0x7fd270146590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:23.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.449+0000 7fd275a5e700 1 -- 172.21.15.60:0/287863911 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2700af100 con 0x7fd2700ae7c0 2022-06-29T16:07:23.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.449+0000 7fd275a5e700 1 --2- 172.21.15.60:0/287863911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2700ae7c0 0x7fd270141fa0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 rx=0x7fd26c002f70 tx=0x7fd26c006eb0).ready entity=mon.0 client_cookie=cf869dbc5fa47359 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:23.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.449+0000 7fd274a5c700 1 -- 172.21.15.60:0/287863911 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd26c00d5c0 con 0x7fd2700ae7c0 2022-06-29T16:07:23.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.449+0000 7fd274a5c700 1 -- 172.21.15.60:0/287863911 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd26c000e90 con 0x7fd2700ae7c0 2022-06-29T16:07:23.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.449+0000 7fd274a5c700 1 -- 172.21.15.60:0/287863911 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd26c00d950 con 0x7fd2700ae7c0 2022-06-29T16:07:23.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.449+0000 7fd277cc2700 1 -- 172.21.15.60:0/287863911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2700ae7c0 msgr2=0x7fd270141fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:23.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.453+0000 7fd277cc2700 1 --2- 172.21.15.60:0/287863911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2700ae7c0 0x7fd270141fa0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 rx=0x7fd26c002f70 tx=0x7fd26c006eb0).stop 2022-06-29T16:07:23.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.453+0000 7fd277cc2700 1 -- 172.21.15.60:0/287863911 shutdown_connections 2022-06-29T16:07:23.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.453+0000 7fd277cc2700 1 --2- 172.21.15.60:0/287863911 >> v2:172.21.15.92:3300/0 conn(0x7fd2700b4010 0x7fd270142490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.453+0000 7fd277cc2700 1 --2- 172.21.15.60:0/287863911 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2700ae7c0 0x7fd270141fa0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.453+0000 7fd277cc2700 1 -- 172.21.15.60:0/287863911 >> 172.21.15.60:0/287863911 conn(0x7fd2700b9340 msgr2=0x7fd2700b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:23.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.453+0000 7fd277cc2700 1 -- 172.21.15.60:0/287863911 shutdown_connections 2022-06-29T16:07:23.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.453+0000 7fd277cc2700 1 -- 172.21.15.60:0/287863911 wait complete. 2022-06-29T16:07:23.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.453+0000 7fd277cc2700 1 Processor -- start 2022-06-29T16:07:23.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.453+0000 7fd277cc2700 1 -- start start 2022-06-29T16:07:23.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.453+0000 7fd277cc2700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2700b4010 0x7fd270112870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:23.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.453+0000 7fd277cc2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd270146160 0x7fd2701528f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:23.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.453+0000 7fd277cc2700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2701522d0 0x7fd270152de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:23.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.453+0000 7fd277cc2700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd270130ca0 con 0x7fd270146160 2022-06-29T16:07:23.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.453+0000 7fd277cc2700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd270130f40 con 0x7fd2701522d0 2022-06-29T16:07:23.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.453+0000 7fd277cc2700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2701311e0 con 0x7fd2700b4010 2022-06-29T16:07:23.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.453+0000 7fd27525d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd270146160 0x7fd2701528f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:23.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.453+0000 7fd27525d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd270146160 0x7fd2701528f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53086/0 (socket says 172.21.15.60:53086) 2022-06-29T16:07:23.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.453+0000 7fd27525d700 1 -- 172.21.15.60:0/1376543164 learned_addr learned my addr 172.21.15.60:0/1376543164 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:23.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.453+0000 7fd27625f700 1 --2- 172.21.15.60:0/1376543164 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2701522d0 0x7fd270152de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:23.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.453+0000 7fd275a5e700 1 --2- 172.21.15.60:0/1376543164 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2700b4010 0x7fd270112870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:23.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.453+0000 7fd27525d700 1 -- 172.21.15.60:0/1376543164 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2700b4010 msgr2=0x7fd270112870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:23.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.453+0000 7fd27525d700 1 --2- 172.21.15.60:0/1376543164 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2700b4010 0x7fd270112870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.453+0000 7fd27525d700 1 -- 172.21.15.60:0/1376543164 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2701522d0 msgr2=0x7fd270152de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:23.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.453+0000 7fd27525d700 1 --2- 172.21.15.60:0/1376543164 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2701522d0 0x7fd270152de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.453+0000 7fd27525d700 1 -- 172.21.15.60:0/1376543164 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd26c00b1e0 con 0x7fd270146160 2022-06-29T16:07:23.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.453+0000 7fd27625f700 1 --2- 172.21.15.60:0/1376543164 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2701522d0 0x7fd270152de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:23.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.453+0000 7fd27525d700 1 --2- 172.21.15.60:0/1376543164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd270146160 0x7fd2701528f0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 rx=0x7fd26000c000 tx=0x7fd260006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:23.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.453+0000 7fd266ffd700 1 -- 172.21.15.60:0/1376543164 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd260010780 con 0x7fd270146160 2022-06-29T16:07:23.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.453+0000 7fd266ffd700 1 -- 172.21.15.60:0/1376543164 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2600108e0 con 0x7fd270146160 2022-06-29T16:07:23.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.453+0000 7fd277cc2700 1 -- 172.21.15.60:0/1376543164 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2700b0160 con 0x7fd270146160 2022-06-29T16:07:23.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.453+0000 7fd266ffd700 1 -- 172.21.15.60:0/1376543164 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd260010c40 con 0x7fd270146160 2022-06-29T16:07:23.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.453+0000 7fd277cc2700 1 -- 172.21.15.60:0/1376543164 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd270130800 con 0x7fd270146160 2022-06-29T16:07:23.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.457+0000 7fd277cc2700 1 -- 172.21.15.60:0/1376543164 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2700573d0 con 0x7fd270146160 2022-06-29T16:07:23.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.457+0000 7fd266ffd700 1 -- 172.21.15.60:0/1376543164 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd26001c460 con 0x7fd270146160 2022-06-29T16:07:23.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.457+0000 7fd266ffd700 1 --2- 172.21.15.60:0/1376543164 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd258053aa0 0x7fd258055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:23.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.457+0000 7fd266ffd700 1 -- 172.21.15.60:0/1376543164 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd26006c3d0 con 0x7fd270146160 2022-06-29T16:07:23.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.457+0000 7fd275a5e700 1 --2- 172.21.15.60:0/1376543164 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd258053aa0 0x7fd258055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:23.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.457+0000 7fd275a5e700 1 --2- 172.21.15.60:0/1376543164 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd258053aa0 0x7fd258055f60 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0x7fd26c01a000 tx=0x7fd26c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:23.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.461+0000 7fd266ffd700 1 -- 172.21.15.60:0/1376543164 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd260043290 con 0x7fd270146160 2022-06-29T16:07:23.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.597+0000 7fd277cc2700 1 -- 172.21.15.60:0/1376543164 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd270130a80 con 0x7fd270146160 2022-06-29T16:07:23.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.597+0000 7fd266ffd700 1 -- 172.21.15.60:0/1376543164 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v128) v1 ==== 104+0+10 (secure 0 0 0) 0x7fd2600468b0 con 0x7fd270146160 2022-06-29T16:07:23.603 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:07:23.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.601+0000 7fd277cc2700 1 -- 172.21.15.60:0/1376543164 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd258053aa0 msgr2=0x7fd258055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:23.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.601+0000 7fd277cc2700 1 --2- 172.21.15.60:0/1376543164 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd258053aa0 0x7fd258055f60 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0x7fd26c01a000 tx=0x7fd26c015000).stop 2022-06-29T16:07:23.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.601+0000 7fd277cc2700 1 -- 172.21.15.60:0/1376543164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd270146160 msgr2=0x7fd2701528f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:23.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.601+0000 7fd277cc2700 1 --2- 172.21.15.60:0/1376543164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd270146160 0x7fd2701528f0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 rx=0x7fd26000c000 tx=0x7fd260006df0).stop 2022-06-29T16:07:23.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.601+0000 7fd277cc2700 1 -- 172.21.15.60:0/1376543164 shutdown_connections 2022-06-29T16:07:23.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.601+0000 7fd277cc2700 1 --2- 172.21.15.60:0/1376543164 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd258053aa0 0x7fd258055f60 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.601+0000 7fd277cc2700 1 --2- 172.21.15.60:0/1376543164 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2701522d0 0x7fd270152de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.601+0000 7fd277cc2700 1 --2- 172.21.15.60:0/1376543164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd270146160 0x7fd2701528f0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.601+0000 7fd277cc2700 1 --2- 172.21.15.60:0/1376543164 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2700b4010 0x7fd270112870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.601+0000 7fd277cc2700 1 -- 172.21.15.60:0/1376543164 >> 172.21.15.60:0/1376543164 conn(0x7fd2700b9340 msgr2=0x7fd27013d140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:23.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.601+0000 7fd277cc2700 1 -- 172.21.15.60:0/1376543164 shutdown_connections 2022-06-29T16:07:23.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.601+0000 7fd277cc2700 1 -- 172.21.15.60:0/1376543164 wait complete. 2022-06-29T16:07:23.623 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-29T16:07:23.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.721+0000 7f5e260d4700 1 Processor -- start 2022-06-29T16:07:23.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.721+0000 7f5e260d4700 1 -- start start 2022-06-29T16:07:23.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.725+0000 7f5e260d4700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5e20145eb0 0x7f5e20066130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:23.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.725+0000 7f5e260d4700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e20141860 0x7f5e20141cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:23.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.725+0000 7f5e260d4700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2005c2b0 con 0x7f5e200b3c70 2022-06-29T16:07:23.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.725+0000 7f5e260d4700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e20066620 con 0x7f5e20145eb0 2022-06-29T16:07:23.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.725+0000 7f5e260d4700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e200b4050 con 0x7f5e20141860 2022-06-29T16:07:23.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.725+0000 7f5e248d1700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5e20145eb0 0x7f5e20066130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:23.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.725+0000 7f5e250d2700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5e200b3c70 0x7f5e20065a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:32948/0 (socket says 172.21.15.60:32948) 2022-06-29T16:07:23.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.725+0000 7f5e250d2700 1 -- 172.21.15.60:0/1513414518 learned_addr learned my addr 172.21.15.60:0/1513414518 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:23.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.725+0000 7f5e258d3700 1 --2- 172.21.15.60:0/1513414518 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e20141860 0x7f5e20141cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:23.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.725+0000 7f5e0ffff700 1 -- 172.21.15.60:0/1513414518 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1204968277 0 0) 0x7f5e2005c2b0 con 0x7f5e200b3c70 2022-06-29T16:07:23.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.725+0000 7f5e0ffff700 1 -- 172.21.15.60:0/1513414518 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e08003260 con 0x7f5e200b3c70 2022-06-29T16:07:23.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.725+0000 7f5e248d1700 1 -- 172.21.15.60:0/1513414518 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e20141860 msgr2=0x7f5e20141cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:23.734 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.725+0000 7f5e248d1700 1 --2- 172.21.15.60:0/1513414518 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e20141860 0x7f5e20141cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.734 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.725+0000 7f5e248d1700 1 -- 172.21.15.60:0/1513414518 >> v1:172.21.15.60:6789/0 conn(0x7f5e200b3c70 legacy=0x7f5e20065a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:23.734 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.725+0000 7f5e248d1700 1 -- 172.21.15.60:0/1513414518 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e2005d120 con 0x7f5e20145eb0 2022-06-29T16:07:23.734 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.725+0000 7f5e248d1700 1 --2- 172.21.15.60:0/1513414518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e20145eb0 0x7f5e20066130 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 rx=0x7f5e10002f70 tx=0x7f5e10006df0).ready entity=mon.0 client_cookie=971382ca2793d48c server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:23.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.725+0000 7f5e0ffff700 1 -- 172.21.15.60:0/1513414518 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e1000d4a0 con 0x7f5e20145eb0 2022-06-29T16:07:23.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.725+0000 7f5e0ffff700 1 -- 172.21.15.60:0/1513414518 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e10000e90 con 0x7f5e20145eb0 2022-06-29T16:07:23.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.725+0000 7f5e0ffff700 1 -- 172.21.15.60:0/1513414518 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e1000d7f0 con 0x7f5e20145eb0 2022-06-29T16:07:23.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.725+0000 7f5e260d4700 1 -- 172.21.15.60:0/1513414518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e20145eb0 msgr2=0x7f5e20066130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:23.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.725+0000 7f5e260d4700 1 --2- 172.21.15.60:0/1513414518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e20145eb0 0x7f5e20066130 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 rx=0x7f5e10002f70 tx=0x7f5e10006df0).stop 2022-06-29T16:07:23.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.725+0000 7f5e260d4700 1 -- 172.21.15.60:0/1513414518 shutdown_connections 2022-06-29T16:07:23.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.725+0000 7f5e260d4700 1 --2- 172.21.15.60:0/1513414518 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e20141860 0x7f5e20141cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.725+0000 7f5e260d4700 1 --2- 172.21.15.60:0/1513414518 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e20145eb0 0x7f5e20066130 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.725+0000 7f5e260d4700 1 -- 172.21.15.60:0/1513414518 >> 172.21.15.60:0/1513414518 conn(0x7f5e20064810 msgr2=0x7f5e20064bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:23.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.725+0000 7f5e260d4700 1 -- 172.21.15.60:0/1513414518 shutdown_connections 2022-06-29T16:07:23.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.725+0000 7f5e260d4700 1 -- 172.21.15.60:0/1513414518 wait complete. 2022-06-29T16:07:23.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.729+0000 7f5e260d4700 1 Processor -- start 2022-06-29T16:07:23.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.729+0000 7f5e260d4700 1 -- start start 2022-06-29T16:07:23.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.729+0000 7f5e260d4700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e200b3c70 0x7f5e200b6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:23.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.729+0000 7f5e260d4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e20141860 0x7f5e200acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:23.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.729+0000 7f5e260d4700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e20145eb0 0x7f5e200b5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:23.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.729+0000 7f5e260d4700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e200ad250 con 0x7f5e20141860 2022-06-29T16:07:23.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.729+0000 7f5e260d4700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e200b5650 con 0x7f5e200b3c70 2022-06-29T16:07:23.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.729+0000 7f5e260d4700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e200b58f0 con 0x7f5e20145eb0 2022-06-29T16:07:23.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.733+0000 7f5e248d1700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e20141860 0x7f5e200acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:23.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.733+0000 7f5e248d1700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e20141860 0x7f5e200acbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53102/0 (socket says 172.21.15.60:53102) 2022-06-29T16:07:23.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.733+0000 7f5e248d1700 1 -- 172.21.15.60:0/2574389792 learned_addr learned my addr 172.21.15.60:0/2574389792 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:23.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.733+0000 7f5e258d3700 1 --2- 172.21.15.60:0/2574389792 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e20145eb0 0x7f5e200b5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:23.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.733+0000 7f5e250d2700 1 --2- 172.21.15.60:0/2574389792 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e200b3c70 0x7f5e200b6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:23.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.733+0000 7f5e248d1700 1 -- 172.21.15.60:0/2574389792 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e20145eb0 msgr2=0x7f5e200b5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:23.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.733+0000 7f5e248d1700 1 --2- 172.21.15.60:0/2574389792 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e20145eb0 0x7f5e200b5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.733+0000 7f5e248d1700 1 -- 172.21.15.60:0/2574389792 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e200b3c70 msgr2=0x7f5e200b6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:23.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.733+0000 7f5e248d1700 1 --2- 172.21.15.60:0/2574389792 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e200b3c70 0x7f5e200b6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.733+0000 7f5e248d1700 1 -- 172.21.15.60:0/2574389792 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e1000b180 con 0x7f5e20141860 2022-06-29T16:07:23.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.733+0000 7f5e258d3700 1 --2- 172.21.15.60:0/2574389792 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e20145eb0 0x7f5e200b5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:23.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.733+0000 7f5e248d1700 1 --2- 172.21.15.60:0/2574389792 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e20141860 0x7f5e200acbb0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 rx=0x7f5e10002f70 tx=0x7f5e10014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:23.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.733+0000 7f5e0e7fc700 1 -- 172.21.15.60:0/2574389792 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e10013ac0 con 0x7f5e20141860 2022-06-29T16:07:23.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.733+0000 7f5e0e7fc700 1 -- 172.21.15.60:0/2574389792 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e10013c20 con 0x7f5e20141860 2022-06-29T16:07:23.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.733+0000 7f5e260d4700 1 -- 172.21.15.60:0/2574389792 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e200b6280 con 0x7f5e20141860 2022-06-29T16:07:23.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.733+0000 7f5e0e7fc700 1 -- 172.21.15.60:0/2574389792 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e1001e400 con 0x7f5e20141860 2022-06-29T16:07:23.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.733+0000 7f5e260d4700 1 -- 172.21.15.60:0/2574389792 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e200adca0 con 0x7f5e20141860 2022-06-29T16:07:23.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.733+0000 7f5e260d4700 1 -- 172.21.15.60:0/2574389792 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e200ad0a0 con 0x7f5e20141860 2022-06-29T16:07:23.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.733+0000 7f5e0e7fc700 1 -- 172.21.15.60:0/2574389792 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5e10025030 con 0x7f5e20141860 2022-06-29T16:07:23.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.733+0000 7f5e0e7fc700 1 --2- 172.21.15.60:0/2574389792 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5e00053b00 0x7f5e00055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:23.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.733+0000 7f5e0e7fc700 1 -- 172.21.15.60:0/2574389792 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5e1006cca0 con 0x7f5e20141860 2022-06-29T16:07:23.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.733+0000 7f5e250d2700 1 --2- 172.21.15.60:0/2574389792 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5e00053b00 0x7f5e00055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:23.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.737+0000 7f5e250d2700 1 --2- 172.21.15.60:0/2574389792 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5e00053b00 0x7f5e00055fc0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7f5e1c00b000 tx=0x7f5e1c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:23.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.741+0000 7f5e0e7fc700 1 -- 172.21.15.60:0/2574389792 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5e10071070 con 0x7f5e20141860 2022-06-29T16:07:23.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.861+0000 7f5e260d4700 1 -- 172.21.15.60:0/2574389792 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5e2005d3c0 con 0x7f5e20141860 2022-06-29T16:07:23.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.865+0000 7f5e0e7fc700 1 -- 172.21.15.60:0/2574389792 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v128) v1 ==== 100+0+10 (secure 0 0 0) 0x7f5e10023070 con 0x7f5e20141860 2022-06-29T16:07:23.869 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:07:23.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.865+0000 7f5e260d4700 1 -- 172.21.15.60:0/2574389792 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5e00053b00 msgr2=0x7f5e00055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:23.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.865+0000 7f5e260d4700 1 --2- 172.21.15.60:0/2574389792 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5e00053b00 0x7f5e00055fc0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7f5e1c00b000 tx=0x7f5e1c009040).stop 2022-06-29T16:07:23.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.865+0000 7f5e260d4700 1 -- 172.21.15.60:0/2574389792 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e20141860 msgr2=0x7f5e200acbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:23.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.865+0000 7f5e260d4700 1 --2- 172.21.15.60:0/2574389792 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e20141860 0x7f5e200acbb0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 rx=0x7f5e10002f70 tx=0x7f5e10014000).stop 2022-06-29T16:07:23.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.865+0000 7f5e260d4700 1 -- 172.21.15.60:0/2574389792 shutdown_connections 2022-06-29T16:07:23.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.865+0000 7f5e260d4700 1 --2- 172.21.15.60:0/2574389792 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5e00053b00 0x7f5e00055fc0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.865+0000 7f5e260d4700 1 --2- 172.21.15.60:0/2574389792 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e20145eb0 0x7f5e200b5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.865+0000 7f5e260d4700 1 --2- 172.21.15.60:0/2574389792 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e20141860 0x7f5e200acbb0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.865+0000 7f5e260d4700 1 --2- 172.21.15.60:0/2574389792 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e200b3c70 0x7f5e200b6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.865+0000 7f5e260d4700 1 -- 172.21.15.60:0/2574389792 >> 172.21.15.60:0/2574389792 conn(0x7f5e20064810 msgr2=0x7f5e20065920 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:23.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.865+0000 7f5e260d4700 1 -- 172.21.15.60:0/2574389792 shutdown_connections 2022-06-29T16:07:23.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.865+0000 7f5e260d4700 1 -- 172.21.15.60:0/2574389792 wait complete. 2022-06-29T16:07:23.882 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-06-29T16:07:23.883 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:07:23.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.977+0000 7ff77bfff700 1 Processor -- start 2022-06-29T16:07:23.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.977+0000 7ff77bfff700 1 -- start start 2022-06-29T16:07:23.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.981+0000 7ff77bfff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff77c0b5eb0 0x7ff77c142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:23.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.981+0000 7ff77bfff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff77c0b2c50 0x7ff77c0b3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:23.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.981+0000 7ff77bfff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff77c0b0cf0 con 0x7ff77c0b48b0 2022-06-29T16:07:23.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.981+0000 7ff77bfff700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff77c0b54e0 con 0x7ff77c0b2c50 2022-06-29T16:07:23.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.981+0000 7ff77bfff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c0b4d10 con 0x7ff77c0b5eb0 2022-06-29T16:07:23.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.981+0000 7ff77b7fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff77c0b2c50 0x7ff77c0b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:23.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.981+0000 7ff77b7fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff77c0b2c50 0x7ff77c0b3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53110/0 (socket says 172.21.15.60:53110) 2022-06-29T16:07:23.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.981+0000 7ff77b7fe700 1 -- 172.21.15.60:0/3346935328 learned_addr learned my addr 172.21.15.60:0/3346935328 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:23.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.981+0000 7ff77affd700 1 --2- 172.21.15.60:0/3346935328 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff77c0b5eb0 0x7ff77c142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:23.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.981+0000 7ff77b7fe700 1 -- 172.21.15.60:0/3346935328 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff77c0b5eb0 msgr2=0x7ff77c142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:23.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.981+0000 7ff77b7fe700 1 --2- 172.21.15.60:0/3346935328 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff77c0b5eb0 0x7ff77c142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.981+0000 7ff77b7fe700 1 -- 172.21.15.60:0/3346935328 >> v1:172.21.15.92:6789/0 conn(0x7ff77c0b48b0 legacy=0x7ff77c0b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:23.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.981+0000 7ff77b7fe700 1 -- 172.21.15.60:0/3346935328 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff77c0b6290 con 0x7ff77c0b2c50 2022-06-29T16:07:23.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.981+0000 7ff77b7fe700 1 --2- 172.21.15.60:0/3346935328 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff77c0b2c50 0x7ff77c0b3090 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 rx=0x7ff774002f70 tx=0x7ff774006eb0).ready entity=mon.0 client_cookie=67df19a0b09c3caf server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:23.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.981+0000 7ff779ffb700 1 -- 172.21.15.60:0/3346935328 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff77400d5c0 con 0x7ff77c0b2c50 2022-06-29T16:07:23.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.981+0000 7ff779ffb700 1 -- 172.21.15.60:0/3346935328 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff774000e90 con 0x7ff77c0b2c50 2022-06-29T16:07:23.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.981+0000 7ff779ffb700 1 -- 172.21.15.60:0/3346935328 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff77400d950 con 0x7ff77c0b2c50 2022-06-29T16:07:23.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.981+0000 7ff77bfff700 1 -- 172.21.15.60:0/3346935328 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff77c0b2c50 msgr2=0x7ff77c0b3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:23.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.981+0000 7ff77bfff700 1 --2- 172.21.15.60:0/3346935328 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff77c0b2c50 0x7ff77c0b3090 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 rx=0x7ff774002f70 tx=0x7ff774006eb0).stop 2022-06-29T16:07:23.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.981+0000 7ff77bfff700 1 -- 172.21.15.60:0/3346935328 shutdown_connections 2022-06-29T16:07:23.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.981+0000 7ff77bfff700 1 --2- 172.21.15.60:0/3346935328 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff77c0b2c50 0x7ff77c0b3090 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.981+0000 7ff77bfff700 1 --2- 172.21.15.60:0/3346935328 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff77c0b5eb0 0x7ff77c142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.981+0000 7ff77bfff700 1 -- 172.21.15.60:0/3346935328 >> 172.21.15.60:0/3346935328 conn(0x7ff77c062090 msgr2=0x7ff77c062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:23.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.981+0000 7ff77bfff700 1 -- 172.21.15.60:0/3346935328 shutdown_connections 2022-06-29T16:07:23.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.981+0000 7ff77bfff700 1 -- 172.21.15.60:0/3346935328 wait complete. 2022-06-29T16:07:23.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.981+0000 7ff77bfff700 1 Processor -- start 2022-06-29T16:07:23.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.981+0000 7ff77bfff700 1 -- start start 2022-06-29T16:07:23.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.985+0000 7ff77bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff77c0b2c50 0x7ff77c112640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:23.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.985+0000 7ff77bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff77c0b48b0 0x7ff77c114eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:23.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.985+0000 7ff77bfff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff77c0b5eb0 0x7ff77c0b8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:23.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.985+0000 7ff77bfff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c112b60 con 0x7ff77c0b2c50 2022-06-29T16:07:23.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.985+0000 7ff77bfff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c112e00 con 0x7ff77c0b48b0 2022-06-29T16:07:23.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.985+0000 7ff77bfff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c155330 con 0x7ff77c0b5eb0 2022-06-29T16:07:23.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.985+0000 7ff77affd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff77c0b2c50 0x7ff77c112640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:23.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.985+0000 7ff77affd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff77c0b2c50 0x7ff77c112640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53114/0 (socket says 172.21.15.60:53114) 2022-06-29T16:07:23.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.985+0000 7ff77affd700 1 -- 172.21.15.60:0/1315469558 learned_addr learned my addr 172.21.15.60:0/1315469558 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:23.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.985+0000 7ff77b7fe700 1 --2- 172.21.15.60:0/1315469558 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff77c0b5eb0 0x7ff77c0b8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:23.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.985+0000 7ff77a7fc700 1 --2- 172.21.15.60:0/1315469558 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff77c0b48b0 0x7ff77c114eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:23.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.985+0000 7ff77affd700 1 -- 172.21.15.60:0/1315469558 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff77c0b5eb0 msgr2=0x7ff77c0b8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:23.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.985+0000 7ff77affd700 1 --2- 172.21.15.60:0/1315469558 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff77c0b5eb0 0x7ff77c0b8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.985+0000 7ff77affd700 1 -- 172.21.15.60:0/1315469558 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff77c0b48b0 msgr2=0x7ff77c114eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:23.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.985+0000 7ff77affd700 1 --2- 172.21.15.60:0/1315469558 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff77c0b48b0 0x7ff77c114eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:23.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.985+0000 7ff77affd700 1 -- 172.21.15.60:0/1315469558 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff77400b1e0 con 0x7ff77c0b2c50 2022-06-29T16:07:23.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.985+0000 7ff77affd700 1 --2- 172.21.15.60:0/1315469558 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff77c0b2c50 0x7ff77c112640 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 rx=0x7ff76c0030b0 tx=0x7ff76c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:23.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.985+0000 7ff773fff700 1 -- 172.21.15.60:0/1315469558 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff76c011780 con 0x7ff77c0b2c50 2022-06-29T16:07:24.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.985+0000 7ff773fff700 1 -- 172.21.15.60:0/1315469558 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff76c0118e0 con 0x7ff77c0b2c50 2022-06-29T16:07:24.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.985+0000 7ff77bfff700 1 -- 172.21.15.60:0/1315469558 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff77c146a80 con 0x7ff77c0b2c50 2022-06-29T16:07:24.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.985+0000 7ff773fff700 1 -- 172.21.15.60:0/1315469558 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff76c011c40 con 0x7ff77c0b2c50 2022-06-29T16:07:24.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.985+0000 7ff77bfff700 1 -- 172.21.15.60:0/1315469558 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff77c1458d0 con 0x7ff77c0b2c50 2022-06-29T16:07:24.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.985+0000 7ff77bfff700 1 -- 172.21.15.60:0/1315469558 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff77c142f10 con 0x7ff77c0b2c50 2022-06-29T16:07:24.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.985+0000 7ff773fff700 1 -- 172.21.15.60:0/1315469558 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff76c012420 con 0x7ff77c0b2c50 2022-06-29T16:07:24.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.989+0000 7ff773fff700 1 --2- 172.21.15.60:0/1315469558 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff760053b00 0x7ff760055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:24.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.989+0000 7ff773fff700 1 -- 172.21.15.60:0/1315469558 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff76c06bc30 con 0x7ff77c0b2c50 2022-06-29T16:07:24.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.989+0000 7ff77a7fc700 1 --2- 172.21.15.60:0/1315469558 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff760053b00 0x7ff760055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:24.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.989+0000 7ff77a7fc700 1 --2- 172.21.15.60:0/1315469558 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff760053b00 0x7ff760055fc0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7ff76400d000 tx=0x7ff764009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:24.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:23.993+0000 7ff773fff700 1 -- 172.21.15.60:0/1315469558 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff76c042af0 con 0x7ff77c0b2c50 2022-06-29T16:07:24.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:24.157+0000 7ff77bfff700 1 -- 172.21.15.60:0/1315469558 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff77c0573d0 con 0x7ff77c0b2c50 2022-06-29T16:07:24.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:24.161+0000 7ff773fff700 1 -- 172.21.15.60:0/1315469558 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2142 (secure 0 0 0) 0x7ff76c046110 con 0x7ff77c0b2c50 2022-06-29T16:07:24.164 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:24.165 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":62,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6941,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/3488623391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3488623391},{"type":"v1","addr":"172.21.15.92:6835","nonce":3488623391}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":59}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:07:19.341489+0000","modified":"2022-06-29T16:07:20.345065+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8085},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8085":{"gid":8085,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/3877729627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3877729627},{"type":"v1","addr":"172.21.15.60:6833","nonce":3877729627}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-06-29T16:07:24.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:24.161+0000 7ff77bfff700 1 -- 172.21.15.60:0/1315469558 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff760053b00 msgr2=0x7ff760055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:24.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:24.161+0000 7ff77bfff700 1 --2- 172.21.15.60:0/1315469558 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff760053b00 0x7ff760055fc0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7ff76400d000 tx=0x7ff764009040).stop 2022-06-29T16:07:24.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:24.161+0000 7ff77bfff700 1 -- 172.21.15.60:0/1315469558 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff77c0b2c50 msgr2=0x7ff77c112640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:24.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:24.161+0000 7ff77bfff700 1 --2- 172.21.15.60:0/1315469558 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff77c0b2c50 0x7ff77c112640 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 rx=0x7ff76c0030b0 tx=0x7ff76c008040).stop 2022-06-29T16:07:24.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:24.161+0000 7ff77bfff700 1 -- 172.21.15.60:0/1315469558 shutdown_connections 2022-06-29T16:07:24.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:24.161+0000 7ff77bfff700 1 --2- 172.21.15.60:0/1315469558 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff760053b00 0x7ff760055fc0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:24.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:24.161+0000 7ff77bfff700 1 --2- 172.21.15.60:0/1315469558 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff77c0b5eb0 0x7ff77c0b8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:24.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:24.161+0000 7ff77bfff700 1 --2- 172.21.15.60:0/1315469558 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff77c0b48b0 0x7ff77c114eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:24.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:24.161+0000 7ff77bfff700 1 --2- 172.21.15.60:0/1315469558 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff77c0b2c50 0x7ff77c112640 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:24.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:24.161+0000 7ff77bfff700 1 -- 172.21.15.60:0/1315469558 >> 172.21.15.60:0/1315469558 conn(0x7ff77c062090 msgr2=0x7ff77c05bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:24.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:24.161+0000 7ff77bfff700 1 -- 172.21.15.60:0/1315469558 shutdown_connections 2022-06-29T16:07:24.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:24.161+0000 7ff77bfff700 1 -- 172.21.15.60:0/1315469558 wait complete. 2022-06-29T16:07:24.170 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 62 2022-06-29T16:07:24.379 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:07:19.341489+0000', 'modified': '2022-06-29T16:07:20.345065+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8085}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8085': {'gid': 8085, 'name': 'a', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.60:6833/3877729627', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6832', 'nonce': 3877729627}, {'type': 'v1', 'addr': '172.21.15.60:6833', 'nonce': 3877729627}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:07:24.380 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:07:24.381 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-06-29T16:07:24.501 INFO:teuthology.orchestra.run.smithi060.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":8085,"want_state":"up:active","state":"up:active","rank_uptime":5.1444602169999998,"mdsmap_epoch":62,"osdmap_epoch":128,"osdmap_epoch_barrier":128,"uptime":9.8695119120000001} 2022-06-29T16:07:24.502 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 8085, "mdsmap_epoch": 62, "osdmap_epoch": 128, "osdmap_epoch_barrier": 128, "rank_uptime": 5.144460217, "state": "up:active", "uptime": 9.869511912, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:07:24.502 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-06-29T16:07:24.502 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-06-29T16:07:24.503 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi121.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-06-29T16:07:24.503 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:24.504 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:07:24.513 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:07:24.530 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:07:24.531 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:07:24.544 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:07:24.545 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:07:24.546 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:24.546 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:07:24.593 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-06-29T16:07:24.594 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:07:24.597 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:07:24.613 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:07:24.614 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:07:24.624 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:07:24.625 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:07:24.626 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:24.626 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:07:24.672 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:2022-06-29T16:07:24.668+0000 7f380006d080 -1 init, newargv = 0x55ddd39cb2a0 newargc=9ceph-fuse[13443 2022-06-29T16:07:24.672 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:]: starting ceph client 2022-06-29T16:07:24.680 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[13443]: starting fuse 2022-06-29T16:07:25.776 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:07:25.795 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:07:25.796 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:07:25.807 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:07:25.808 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:07:25.809 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:25.809 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:07:25.859 INFO:teuthology.orchestra.run.smithi121.stdout:52 2022-06-29T16:07:25.859 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-06-29T16:07:25.860 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:25.861 DEBUG:teuthology.orchestra.run.smithi121:> sudo python3 -c ' 2022-06-29T16:07:25.862 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2022-06-29T16:07:25.863 DEBUG:teuthology.orchestra.run.smithi121:> import re 2022-06-29T16:07:25.863 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-06-29T16:07:25.863 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2022-06-29T16:07:25.864 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:07:25.864 DEBUG:teuthology.orchestra.run.smithi121:> def find_socket(client_name): 2022-06-29T16:07:25.865 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-06-29T16:07:25.865 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2022-06-29T16:07:25.866 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:07:25.866 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2022-06-29T16:07:25.866 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2022-06-29T16:07:25.867 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2022-06-29T16:07:25.867 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2022-06-29T16:07:25.868 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:07:25.868 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2022-06-29T16:07:25.868 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-06-29T16:07:25.869 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2022-06-29T16:07:25.869 DEBUG:teuthology.orchestra.run.smithi121:> return f 2022-06-29T16:07:25.870 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-06-29T16:07:25.870 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:07:25.871 DEBUG:teuthology.orchestra.run.smithi121:> print(find_socket("client.0")) 2022-06-29T16:07:25.871 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-06-29T16:07:25.917 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.0.13443.asok 2022-06-29T16:07:25.923 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.13443.asok 2022-06-29T16:07:25.924 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:25.925 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.13443.asok status 2022-06-29T16:07:26.016 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2022-06-29T16:07:26.016 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2022-06-29T16:07:26.016 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "f25ffac72e5a9e63860ff7deb6767a11f0559a16", 2022-06-29T16:07:26.016 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 15.2.16-304-gf25ffac7 (f25ffac72e5a9e63860ff7deb6767a11f0559a16) octopus (stable)", 2022-06-29T16:07:26.017 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "0", 2022-06-29T16:07:26.017 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2022-06-29T16:07:26.017 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-06-29T16:07:26.017 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "13443", 2022-06-29T16:07:26.018 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2022-06-29T16:07:26.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:07:26.018 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2022-06-29T16:07:26.018 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2022-06-29T16:07:26.019 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 8194, 2022-06-29T16:07:26.019 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2022-06-29T16:07:26.019 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2022-06-29T16:07:26.020 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2022-06-29T16:07:26.020 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 8194 2022-06-29T16:07:26.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:07:26.020 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-06-29T16:07:26.020 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-06-29T16:07:26.021 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-06-29T16:07:26.021 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 555664839 2022-06-29T16:07:26.021 INFO:teuthology.orchestra.run.smithi121.stdout: } 2022-06-29T16:07:26.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:07:26.022 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-06-29T16:07:26.022 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-06-29T16:07:26.023 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-06-29T16:07:26.023 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 555664839 2022-06-29T16:07:26.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:07:26.023 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.8194 172.21.15.121:0/555664839", 2022-06-29T16:07:26.024 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "172.21.15.121:0/555664839", 2022-06-29T16:07:26.024 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2022-06-29T16:07:26.025 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 62, 2022-06-29T16:07:26.025 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 128, 2022-06-29T16:07:26.025 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2022-06-29T16:07:26.025 INFO:teuthology.orchestra.run.smithi121.stdout: "blacklisted": false, 2022-06-29T16:07:26.025 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2022-06-29T16:07:26.026 INFO:teuthology.orchestra.run.smithi121.stdout:} 2022-06-29T16:07:26.027 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:26.028 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:07:26.028 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:07:26.035 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T16:07:26.036 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T16:07:26.037 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:26.037 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:07:26.055 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:07:26.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.157+0000 7f65897c1700 1 Processor -- start 2022-06-29T16:07:26.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.161+0000 7f65897c1700 1 -- start start 2022-06-29T16:07:26.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.161+0000 7f65897c1700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f658405d070 0x7f658414aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:26.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.161+0000 7f65897c1700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f658405d450 con 0x7f65840b2130 2022-06-29T16:07:26.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.161+0000 7f65897c1700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65840ad840 con 0x7f658414af80 2022-06-29T16:07:26.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.161+0000 7f65897c1700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65840adf90 con 0x7f658405d070 2022-06-29T16:07:26.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.161+0000 7f65837fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f65840b2130 0x7f65840aca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:32976/0 (socket says 172.21.15.60:32976) 2022-06-29T16:07:26.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.161+0000 7f65837fe700 1 -- 172.21.15.60:0/1549823831 learned_addr learned my addr 172.21.15.60:0/1549823831 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:26.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.161+0000 7f6583fff700 1 --2- 172.21.15.60:0/1549823831 >> v2:172.21.15.92:3300/0 conn(0x7f658405d070 0x7f658414aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:26.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.161+0000 7f6582ffd700 1 -- 172.21.15.60:0/1549823831 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3135568857 0 0) 0x7f658405d450 con 0x7f65840b2130 2022-06-29T16:07:26.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.161+0000 7f6582ffd700 1 -- 172.21.15.60:0/1549823831 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6564003260 con 0x7f65840b2130 2022-06-29T16:07:26.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.161+0000 7f6582ffd700 1 -- 172.21.15.60:0/1549823831 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3835503559 0 0) 0x7f65840ad840 con 0x7f658414af80 2022-06-29T16:07:26.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.161+0000 7f6582ffd700 1 -- 172.21.15.60:0/1549823831 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f658405d450 con 0x7f658414af80 2022-06-29T16:07:26.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.161+0000 7f6582ffd700 1 -- 172.21.15.60:0/1549823831 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2374090340 0 0) 0x7f6564003260 con 0x7f65840b2130 2022-06-29T16:07:26.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.161+0000 7f6583fff700 1 -- 172.21.15.60:0/1549823831 >> v1:172.21.15.92:6789/0 conn(0x7f658414af80 legacy=0x7f65840ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:26.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.161+0000 7f6583fff700 1 -- 172.21.15.60:0/1549823831 >> v1:172.21.15.60:6789/0 conn(0x7f65840b2130 legacy=0x7f65840aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:26.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.161+0000 7f6583fff700 1 -- 172.21.15.60:0/1549823831 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65840b25d0 con 0x7f658405d070 2022-06-29T16:07:26.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.161+0000 7f6583fff700 1 --2- 172.21.15.60:0/1549823831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f658405d070 0x7f658414aa90 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7f6574001110 tx=0x7f656c006df0).ready entity=mon.1 client_cookie=777bbb16da85f1f server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:26.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.161+0000 7f6582ffd700 1 -- 172.21.15.60:0/1549823831 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6574001660 con 0x7f658405d070 2022-06-29T16:07:26.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.161+0000 7f6582ffd700 1 -- 172.21.15.60:0/1549823831 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f656c000e90 con 0x7f658405d070 2022-06-29T16:07:26.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.161+0000 7f6582ffd700 1 -- 172.21.15.60:0/1549823831 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f656c012460 con 0x7f658405d070 2022-06-29T16:07:26.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.161+0000 7f65897c1700 1 -- 172.21.15.60:0/1549823831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f658405d070 msgr2=0x7f658414aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:26.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.165+0000 7f65897c1700 1 --2- 172.21.15.60:0/1549823831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f658405d070 0x7f658414aa90 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7f6574001110 tx=0x7f656c006df0).stop 2022-06-29T16:07:26.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.165+0000 7f65897c1700 1 -- 172.21.15.60:0/1549823831 shutdown_connections 2022-06-29T16:07:26.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.165+0000 7f65897c1700 1 --2- 172.21.15.60:0/1549823831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f658405d070 0x7f658414aa90 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:26.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.165+0000 7f65897c1700 1 -- 172.21.15.60:0/1549823831 >> 172.21.15.60:0/1549823831 conn(0x7f65840b5890 msgr2=0x7f65840b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:26.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.165+0000 7f65897c1700 1 -- 172.21.15.60:0/1549823831 shutdown_connections 2022-06-29T16:07:26.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.165+0000 7f65897c1700 1 -- 172.21.15.60:0/1549823831 wait complete. 2022-06-29T16:07:26.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.165+0000 7f65897c1700 1 Processor -- start 2022-06-29T16:07:26.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.165+0000 7f65897c1700 1 -- start start 2022-06-29T16:07:26.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.165+0000 7f65897c1700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f658405d070 0x7f658414cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:26.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.165+0000 7f65897c1700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65840b2130 0x7f65840b4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:26.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.165+0000 7f65897c1700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f658414af80 0x7f65841596f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:26.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.165+0000 7f65897c1700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6584064690 con 0x7f65840b2130 2022-06-29T16:07:26.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.165+0000 7f65897c1700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6584064b20 con 0x7f658414af80 2022-06-29T16:07:26.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.165+0000 7f65897c1700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6584064dc0 con 0x7f658405d070 2022-06-29T16:07:26.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.165+0000 7f65837fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65840b2130 0x7f65840b4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:26.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.165+0000 7f65837fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65840b2130 0x7f65840b4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53130/0 (socket says 172.21.15.60:53130) 2022-06-29T16:07:26.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.165+0000 7f65837fe700 1 -- 172.21.15.60:0/3510413769 learned_addr learned my addr 172.21.15.60:0/3510413769 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:26.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.165+0000 7f6583fff700 1 --2- 172.21.15.60:0/3510413769 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f658405d070 0x7f658414cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:26.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.165+0000 7f6588fc0700 1 --2- 172.21.15.60:0/3510413769 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f658414af80 0x7f65841596f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:26.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.165+0000 7f65837fe700 1 -- 172.21.15.60:0/3510413769 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f658405d070 msgr2=0x7f658414cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:26.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.165+0000 7f65837fe700 1 --2- 172.21.15.60:0/3510413769 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f658405d070 0x7f658414cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:26.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.165+0000 7f65837fe700 1 -- 172.21.15.60:0/3510413769 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f658414af80 msgr2=0x7f65841596f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:26.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.165+0000 7f65837fe700 1 --2- 172.21.15.60:0/3510413769 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f658414af80 0x7f65841596f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:26.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.165+0000 7f65837fe700 1 -- 172.21.15.60:0/3510413769 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f656c00b180 con 0x7f65840b2130 2022-06-29T16:07:26.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.165+0000 7f65837fe700 1 --2- 172.21.15.60:0/3510413769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65840b2130 0x7f65840b4260 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 rx=0x7f6574001840 tx=0x7f6574001720).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:26.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.165+0000 7f65817fa700 1 -- 172.21.15.60:0/3510413769 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6574007170 con 0x7f65840b2130 2022-06-29T16:07:26.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.165+0000 7f65817fa700 1 -- 172.21.15.60:0/3510413769 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65740072d0 con 0x7f65840b2130 2022-06-29T16:07:26.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.165+0000 7f65897c1700 1 -- 172.21.15.60:0/3510413769 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65840630c0 con 0x7f65840b2130 2022-06-29T16:07:26.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.165+0000 7f65817fa700 1 -- 172.21.15.60:0/3510413769 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6574004710 con 0x7f65840b2130 2022-06-29T16:07:26.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.165+0000 7f65897c1700 1 -- 172.21.15.60:0/3510413769 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6584066510 con 0x7f65840b2130 2022-06-29T16:07:26.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.169+0000 7f65817fa700 1 -- 172.21.15.60:0/3510413769 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f657401b070 con 0x7f65840b2130 2022-06-29T16:07:26.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.169+0000 7f65817fa700 1 --2- 172.21.15.60:0/3510413769 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6568053a70 0x7f6568055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:26.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.169+0000 7f65817fa700 1 -- 172.21.15.60:0/3510413769 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f657406b530 con 0x7f65840b2130 2022-06-29T16:07:26.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.169+0000 7f65897c1700 1 -- 172.21.15.60:0/3510413769 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6584066310 con 0x7f65840b2130 2022-06-29T16:07:26.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.169+0000 7f6583fff700 1 --2- 172.21.15.60:0/3510413769 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6568053a70 0x7f6568055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:26.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.169+0000 7f6583fff700 1 --2- 172.21.15.60:0/3510413769 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6568053a70 0x7f6568055f30 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7f656c01a000 tx=0x7f656c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:26.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.177+0000 7f65817fa700 1 -- 172.21.15.60:0/3510413769 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f65740423b0 con 0x7f65840b2130 2022-06-29T16:07:26.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.297+0000 7f65897c1700 1 -- 172.21.15.60:0/3510413769 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6584064980 con 0x7f65840b2130 2022-06-29T16:07:26.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.301+0000 7f65817fa700 1 -- 172.21.15.60:0/3510413769 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) v1 ==== 74+0+16052 (secure 0 0 0) 0x7f6584064980 con 0x7f65840b2130 2022-06-29T16:07:26.304 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:26.305 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":128,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:07:20.344096+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:07:17.372006+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2022-06-29T16:07:18.349364+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:07:26.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.301+0000 7f65897c1700 1 -- 172.21.15.60:0/3510413769 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6568053a70 msgr2=0x7f6568055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:26.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.301+0000 7f65897c1700 1 --2- 172.21.15.60:0/3510413769 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6568053a70 0x7f6568055f30 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7f656c01a000 tx=0x7f656c014000).stop 2022-06-29T16:07:26.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.301+0000 7f65897c1700 1 -- 172.21.15.60:0/3510413769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65840b2130 msgr2=0x7f65840b4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:26.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.301+0000 7f65897c1700 1 --2- 172.21.15.60:0/3510413769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65840b2130 0x7f65840b4260 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 rx=0x7f6574001840 tx=0x7f6574001720).stop 2022-06-29T16:07:26.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.301+0000 7f65897c1700 1 -- 172.21.15.60:0/3510413769 shutdown_connections 2022-06-29T16:07:26.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.301+0000 7f65897c1700 1 --2- 172.21.15.60:0/3510413769 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6568053a70 0x7f6568055f30 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:26.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.301+0000 7f65897c1700 1 --2- 172.21.15.60:0/3510413769 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f658414af80 0x7f65841596f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:26.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.301+0000 7f65897c1700 1 --2- 172.21.15.60:0/3510413769 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f65840b2130 0x7f65840b4260 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:26.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.301+0000 7f65897c1700 1 --2- 172.21.15.60:0/3510413769 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f658405d070 0x7f658414cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:26.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.301+0000 7f65897c1700 1 -- 172.21.15.60:0/3510413769 >> 172.21.15.60:0/3510413769 conn(0x7f65840b5890 msgr2=0x7f6584062450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:26.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.301+0000 7f65897c1700 1 -- 172.21.15.60:0/3510413769 shutdown_connections 2022-06-29T16:07:26.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.301+0000 7f65897c1700 1 -- 172.21.15.60:0/3510413769 wait complete. 2022-06-29T16:07:26.318 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-29T16:07:17.372006+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2022-06-29T16:07:18.349364+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '128', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-29T16:07:26.319 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:07:26.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.425+0000 7f3b2f59e700 1 Processor -- start 2022-06-29T16:07:26.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.425+0000 7f3b2f59e700 1 -- start start 2022-06-29T16:07:26.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.425+0000 7f3b2f59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b300b2ba0 0x7f3b300ae990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:26.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.425+0000 7f3b2f59e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3b30060770 0x7f3b3005f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:26.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.425+0000 7f3b2f59e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b300b0d80 con 0x7f3b3005f5a0 2022-06-29T16:07:26.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.425+0000 7f3b2f59e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b300b2f80 con 0x7f3b30060770 2022-06-29T16:07:26.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.425+0000 7f3b2f59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b300ae250 con 0x7f3b300b2ba0 2022-06-29T16:07:26.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.425+0000 7f3b2dd9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3b30060770 0x7f3b3005f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:26.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.425+0000 7f3b2dd9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3b30060770 0x7f3b3005f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53136/0 (socket says 172.21.15.60:53136) 2022-06-29T16:07:26.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.425+0000 7f3b2dd9b700 1 -- 172.21.15.60:0/399379220 learned_addr learned my addr 172.21.15.60:0/399379220 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:26.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.425+0000 7f3b2e59c700 1 --2- 172.21.15.60:0/399379220 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b300b2ba0 0x7f3b300ae990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:26.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.425+0000 7f3b2dd9b700 1 -- 172.21.15.60:0/399379220 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b300b2ba0 msgr2=0x7f3b300ae990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:26.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.425+0000 7f3b2dd9b700 1 --2- 172.21.15.60:0/399379220 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b300b2ba0 0x7f3b300ae990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:26.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.425+0000 7f3b2dd9b700 1 -- 172.21.15.60:0/399379220 >> v1:172.21.15.92:6789/0 conn(0x7f3b3005f5a0 legacy=0x7f3b3013ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:26.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.425+0000 7f3b2dd9b700 1 -- 172.21.15.60:0/399379220 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b300ae4f0 con 0x7f3b30060770 2022-06-29T16:07:26.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.425+0000 7f3b2dd9b700 1 --2- 172.21.15.60:0/399379220 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b30060770 0x7f3b3005f0b0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 rx=0x7f3b24002f70 tx=0x7f3b24006eb0).ready entity=mon.0 client_cookie=9b8f30923d416cf5 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:26.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.425+0000 7f3b2d59a700 1 -- 172.21.15.60:0/399379220 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b2400d570 con 0x7f3b30060770 2022-06-29T16:07:26.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.425+0000 7f3b2d59a700 1 -- 172.21.15.60:0/399379220 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b24000e90 con 0x7f3b30060770 2022-06-29T16:07:26.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.425+0000 7f3b2d59a700 1 -- 172.21.15.60:0/399379220 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b2400d900 con 0x7f3b30060770 2022-06-29T16:07:26.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.425+0000 7f3b2f59e700 1 -- 172.21.15.60:0/399379220 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b30060770 msgr2=0x7f3b3005f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:26.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.425+0000 7f3b2f59e700 1 --2- 172.21.15.60:0/399379220 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b30060770 0x7f3b3005f0b0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 rx=0x7f3b24002f70 tx=0x7f3b24006eb0).stop 2022-06-29T16:07:26.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.425+0000 7f3b2f59e700 1 -- 172.21.15.60:0/399379220 shutdown_connections 2022-06-29T16:07:26.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.425+0000 7f3b2f59e700 1 --2- 172.21.15.60:0/399379220 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b30060770 0x7f3b3005f0b0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:26.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.425+0000 7f3b2f59e700 1 --2- 172.21.15.60:0/399379220 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b300b2ba0 0x7f3b300ae990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:26.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.425+0000 7f3b2f59e700 1 -- 172.21.15.60:0/399379220 >> 172.21.15.60:0/399379220 conn(0x7f3b300637c0 msgr2=0x7f3b30063ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:26.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.425+0000 7f3b2f59e700 1 -- 172.21.15.60:0/399379220 shutdown_connections 2022-06-29T16:07:26.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.425+0000 7f3b2f59e700 1 -- 172.21.15.60:0/399379220 wait complete. 2022-06-29T16:07:26.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.429+0000 7f3b2f59e700 1 Processor -- start 2022-06-29T16:07:26.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.429+0000 7f3b2f59e700 1 -- start start 2022-06-29T16:07:26.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.429+0000 7f3b2f59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b3005f5a0 0x7f3b3013d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:26.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.429+0000 7f3b2f59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b30060770 0x7f3b30066110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:26.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.429+0000 7f3b2f59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b300b2ba0 0x7f3b30137f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:26.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.429+0000 7f3b2f59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b30143560 con 0x7f3b300b2ba0 2022-06-29T16:07:26.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.429+0000 7f3b2f59e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b30138b10 con 0x7f3b30060770 2022-06-29T16:07:26.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.429+0000 7f3b2f59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b30138db0 con 0x7f3b3005f5a0 2022-06-29T16:07:26.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.429+0000 7f3b2ed9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b300b2ba0 0x7f3b30137f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:26.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.429+0000 7f3b2e59c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b3005f5a0 0x7f3b3013d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:26.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.429+0000 7f3b2e59c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b3005f5a0 0x7f3b3013d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:38004/0 (socket says 172.21.15.60:38004) 2022-06-29T16:07:26.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.429+0000 7f3b2e59c700 1 -- 172.21.15.60:0/822984175 learned_addr learned my addr 172.21.15.60:0/822984175 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:26.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.429+0000 7f3b2dd9b700 1 --2- 172.21.15.60:0/822984175 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b30060770 0x7f3b30066110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:26.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.429+0000 7f3b2e59c700 1 -- 172.21.15.60:0/822984175 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b30060770 msgr2=0x7f3b30066110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:26.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.429+0000 7f3b2e59c700 1 --2- 172.21.15.60:0/822984175 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b30060770 0x7f3b30066110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:26.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.429+0000 7f3b2e59c700 1 -- 172.21.15.60:0/822984175 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b300b2ba0 msgr2=0x7f3b30137f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:26.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.429+0000 7f3b2e59c700 1 --2- 172.21.15.60:0/822984175 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b300b2ba0 0x7f3b30137f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:26.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.429+0000 7f3b2e59c700 1 -- 172.21.15.60:0/822984175 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2400b1e0 con 0x7f3b3005f5a0 2022-06-29T16:07:26.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.429+0000 7f3b2ed9d700 1 --2- 172.21.15.60:0/822984175 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b300b2ba0 0x7f3b30137f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:07:26.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.429+0000 7f3b2e59c700 1 --2- 172.21.15.60:0/822984175 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b3005f5a0 0x7f3b3013d710 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f3b2000c000 tx=0x7f3b20006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:26.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.429+0000 7f3b1f7fe700 1 -- 172.21.15.60:0/822984175 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b20011750 con 0x7f3b3005f5a0 2022-06-29T16:07:26.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.429+0000 7f3b1f7fe700 1 -- 172.21.15.60:0/822984175 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b200118b0 con 0x7f3b3005f5a0 2022-06-29T16:07:26.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.429+0000 7f3b2f59e700 1 -- 172.21.15.60:0/822984175 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b301431c0 con 0x7f3b3005f5a0 2022-06-29T16:07:26.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.429+0000 7f3b1f7fe700 1 -- 172.21.15.60:0/822984175 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b20011c10 con 0x7f3b3005f5a0 2022-06-29T16:07:26.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.429+0000 7f3b2f59e700 1 -- 172.21.15.60:0/822984175 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b300677d0 con 0x7f3b3005f5a0 2022-06-29T16:07:26.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.433+0000 7f3b2f59e700 1 -- 172.21.15.60:0/822984175 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b30138f70 con 0x7f3b3005f5a0 2022-06-29T16:07:26.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.433+0000 7f3b1f7fe700 1 -- 172.21.15.60:0/822984175 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3b20011d70 con 0x7f3b3005f5a0 2022-06-29T16:07:26.448 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.433+0000 7f3b1f7fe700 1 --2- 172.21.15.60:0/822984175 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3b0c053b00 0x7f3b0c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:26.448 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.433+0000 7f3b1f7fe700 1 -- 172.21.15.60:0/822984175 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3b2006c320 con 0x7f3b3005f5a0 2022-06-29T16:07:26.448 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.433+0000 7f3b2dd9b700 1 --2- 172.21.15.60:0/822984175 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3b0c053b00 0x7f3b0c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:26.448 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.433+0000 7f3b2dd9b700 1 --2- 172.21.15.60:0/822984175 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3b0c053b00 0x7f3b0c055fc0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0x7f3b2401c000 tx=0x7f3b24015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:26.449 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.441+0000 7f3b1f7fe700 1 -- 172.21.15.60:0/822984175 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3b20043240 con 0x7f3b3005f5a0 2022-06-29T16:07:26.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.569+0000 7f3b2f59e700 1 -- 172.21.15.60:0/822984175 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f3b301433b0 con 0x7f3b3005f5a0 2022-06-29T16:07:26.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.569+0000 7f3b1f7fe700 1 -- 172.21.15.60:0/822984175 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v128) v1 ==== 110+0+10 (secure 0 0 0) 0x7f3b20046860 con 0x7f3b3005f5a0 2022-06-29T16:07:26.575 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:07:26.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.573+0000 7f3b2f59e700 1 -- 172.21.15.60:0/822984175 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3b0c053b00 msgr2=0x7f3b0c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:26.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.573+0000 7f3b2f59e700 1 --2- 172.21.15.60:0/822984175 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3b0c053b00 0x7f3b0c055fc0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0x7f3b2401c000 tx=0x7f3b24015000).stop 2022-06-29T16:07:26.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.573+0000 7f3b2f59e700 1 -- 172.21.15.60:0/822984175 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b3005f5a0 msgr2=0x7f3b3013d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:26.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.573+0000 7f3b2f59e700 1 --2- 172.21.15.60:0/822984175 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b3005f5a0 0x7f3b3013d710 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f3b2000c000 tx=0x7f3b20006df0).stop 2022-06-29T16:07:26.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.573+0000 7f3b2f59e700 1 -- 172.21.15.60:0/822984175 shutdown_connections 2022-06-29T16:07:26.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.573+0000 7f3b2f59e700 1 --2- 172.21.15.60:0/822984175 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3b0c053b00 0x7f3b0c055fc0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:26.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.573+0000 7f3b2f59e700 1 --2- 172.21.15.60:0/822984175 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3b300b2ba0 0x7f3b30137f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:26.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.573+0000 7f3b2f59e700 1 --2- 172.21.15.60:0/822984175 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b30060770 0x7f3b30066110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:26.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.573+0000 7f3b2f59e700 1 --2- 172.21.15.60:0/822984175 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3b3005f5a0 0x7f3b3013d710 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:26.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.573+0000 7f3b2f59e700 1 -- 172.21.15.60:0/822984175 >> 172.21.15.60:0/822984175 conn(0x7f3b300637c0 msgr2=0x7f3b3013abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:26.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.573+0000 7f3b2f59e700 1 -- 172.21.15.60:0/822984175 shutdown_connections 2022-06-29T16:07:26.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.573+0000 7f3b2f59e700 1 -- 172.21.15.60:0/822984175 wait complete. 2022-06-29T16:07:26.594 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:07:26.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.697+0000 7f7983fff700 1 Processor -- start 2022-06-29T16:07:26.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.697+0000 7f7983fff700 1 -- start start 2022-06-29T16:07:26.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.697+0000 7f7983fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f798405fb00 0x7f798413f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:26.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.697+0000 7f7983fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f79841405a0 0x7f7984143fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:26.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.697+0000 7f7983fff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7984064e20 con 0x7f7984067a90 2022-06-29T16:07:26.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.697+0000 7f7983fff700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79840b10e0 con 0x7f798405fb00 2022-06-29T16:07:26.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.697+0000 7f7983fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f798405fee0 con 0x7f79841405a0 2022-06-29T16:07:26.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.697+0000 7f79827fc700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f7984067a90 0x7f798413fe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33006/0 (socket says 172.21.15.60:33006) 2022-06-29T16:07:26.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.697+0000 7f79827fc700 1 -- 172.21.15.60:0/2768145138 learned_addr learned my addr 172.21.15.60:0/2768145138 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:26.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.697+0000 7f7982ffd700 1 --2- 172.21.15.60:0/2768145138 >> v2:172.21.15.92:3300/0 conn(0x7f798405fb00 0x7f798413f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:26.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.697+0000 7f79837fe700 1 --2- 172.21.15.60:0/2768145138 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f79841405a0 0x7f7984143fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:26.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.697+0000 7f7981ffb700 1 -- 172.21.15.60:0/2768145138 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 496184649 0 0) 0x7f7984064e20 con 0x7f7984067a90 2022-06-29T16:07:26.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.697+0000 7f7981ffb700 1 -- 172.21.15.60:0/2768145138 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7968003260 con 0x7f7984067a90 2022-06-29T16:07:26.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.697+0000 7f7982ffd700 1 -- 172.21.15.60:0/2768145138 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f79841405a0 msgr2=0x7f7984143fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:26.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.697+0000 7f7982ffd700 1 --2- 172.21.15.60:0/2768145138 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f79841405a0 0x7f7984143fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:26.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.697+0000 7f7982ffd700 1 -- 172.21.15.60:0/2768145138 >> v1:172.21.15.60:6789/0 conn(0x7f7984067a90 legacy=0x7f798413fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:26.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.697+0000 7f7982ffd700 1 -- 172.21.15.60:0/2768145138 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7984060b20 con 0x7f798405fb00 2022-06-29T16:07:26.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.697+0000 7f7982ffd700 1 --2- 172.21.15.60:0/2768145138 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f798405fb00 0x7f798413f9a0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7f7974002f70 tx=0x7f7974006df0).ready entity=mon.1 client_cookie=1076213435238791 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:26.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.697+0000 7f7981ffb700 1 -- 172.21.15.60:0/2768145138 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f797400d4a0 con 0x7f798405fb00 2022-06-29T16:07:26.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.697+0000 7f7981ffb700 1 -- 172.21.15.60:0/2768145138 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7974000e90 con 0x7f798405fb00 2022-06-29T16:07:26.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.697+0000 7f7981ffb700 1 -- 172.21.15.60:0/2768145138 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f797400d7f0 con 0x7f798405fb00 2022-06-29T16:07:26.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.697+0000 7f7983fff700 1 -- 172.21.15.60:0/2768145138 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f798405fb00 msgr2=0x7f798413f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:26.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.697+0000 7f7983fff700 1 --2- 172.21.15.60:0/2768145138 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f798405fb00 0x7f798413f9a0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7f7974002f70 tx=0x7f7974006df0).stop 2022-06-29T16:07:26.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.697+0000 7f7983fff700 1 -- 172.21.15.60:0/2768145138 shutdown_connections 2022-06-29T16:07:26.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.697+0000 7f7983fff700 1 --2- 172.21.15.60:0/2768145138 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f79841405a0 0x7f7984143fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:26.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.697+0000 7f7983fff700 1 --2- 172.21.15.60:0/2768145138 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f798405fb00 0x7f798413f9a0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:26.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.697+0000 7f7983fff700 1 -- 172.21.15.60:0/2768145138 >> 172.21.15.60:0/2768145138 conn(0x7f7984063a60 msgr2=0x7f798405f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:26.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.701+0000 7f7983fff700 1 -- 172.21.15.60:0/2768145138 shutdown_connections 2022-06-29T16:07:26.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.701+0000 7f7983fff700 1 -- 172.21.15.60:0/2768145138 wait complete. 2022-06-29T16:07:26.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.701+0000 7f7983fff700 1 Processor -- start 2022-06-29T16:07:26.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.701+0000 7f7983fff700 1 -- start start 2022-06-29T16:07:26.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.701+0000 7f7983fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f798405fb00 0x7f7984065730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:26.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.701+0000 7f7983fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7984067a90 0x7f79841439e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:26.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.701+0000 7f7983fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79841405a0 0x7f7984143ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:26.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.701+0000 7f7983fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798413d320 con 0x7f79841405a0 2022-06-29T16:07:26.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.701+0000 7f7983fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7984153060 con 0x7f798405fb00 2022-06-29T16:07:26.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.701+0000 7f7983fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f798413d550 con 0x7f7984067a90 2022-06-29T16:07:26.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.701+0000 7f79837fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79841405a0 0x7f7984143ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:26.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.701+0000 7f79837fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79841405a0 0x7f7984143ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53158/0 (socket says 172.21.15.60:53158) 2022-06-29T16:07:26.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.701+0000 7f79837fe700 1 -- 172.21.15.60:0/1699560578 learned_addr learned my addr 172.21.15.60:0/1699560578 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:26.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.701+0000 7f79827fc700 1 --2- 172.21.15.60:0/1699560578 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7984067a90 0x7f79841439e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:26.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.701+0000 7f7982ffd700 1 --2- 172.21.15.60:0/1699560578 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f798405fb00 0x7f7984065730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:26.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.701+0000 7f79837fe700 1 -- 172.21.15.60:0/1699560578 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7984067a90 msgr2=0x7f79841439e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:26.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.701+0000 7f79837fe700 1 --2- 172.21.15.60:0/1699560578 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7984067a90 0x7f79841439e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:26.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.701+0000 7f79837fe700 1 -- 172.21.15.60:0/1699560578 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f798405fb00 msgr2=0x7f7984065730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:26.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.701+0000 7f79837fe700 1 --2- 172.21.15.60:0/1699560578 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f798405fb00 0x7f7984065730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:26.728 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.701+0000 7f79837fe700 1 -- 172.21.15.60:0/1699560578 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f797400b180 con 0x7f79841405a0 2022-06-29T16:07:26.728 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.701+0000 7f7982ffd700 1 --2- 172.21.15.60:0/1699560578 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f798405fb00 0x7f7984065730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:26.728 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.701+0000 7f79837fe700 1 --2- 172.21.15.60:0/1699560578 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79841405a0 0x7f7984143ed0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 rx=0x7f797c006e00 tx=0x7f797c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:26.728 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.705+0000 7f7973fff700 1 -- 172.21.15.60:0/1699560578 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f797c00c7f0 con 0x7f79841405a0 2022-06-29T16:07:26.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.705+0000 7f7973fff700 1 -- 172.21.15.60:0/1699560578 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f797c00c950 con 0x7f79841405a0 2022-06-29T16:07:26.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.705+0000 7f7973fff700 1 -- 172.21.15.60:0/1699560578 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f797c00ccb0 con 0x7f79841405a0 2022-06-29T16:07:26.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.705+0000 7f7983fff700 1 -- 172.21.15.60:0/1699560578 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79841429c0 con 0x7f79841405a0 2022-06-29T16:07:26.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.705+0000 7f7983fff700 1 -- 172.21.15.60:0/1699560578 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7984142760 con 0x7f79841405a0 2022-06-29T16:07:26.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.705+0000 7f7983fff700 1 -- 172.21.15.60:0/1699560578 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79841531d0 con 0x7f79841405a0 2022-06-29T16:07:26.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.705+0000 7f7973fff700 1 -- 172.21.15.60:0/1699560578 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f797c022070 con 0x7f79841405a0 2022-06-29T16:07:26.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.705+0000 7f7973fff700 1 --2- 172.21.15.60:0/1699560578 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7960053a70 0x7f7960055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:26.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.705+0000 7f7973fff700 1 -- 172.21.15.60:0/1699560578 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f797c06c580 con 0x7f79841405a0 2022-06-29T16:07:26.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.705+0000 7f7982ffd700 1 --2- 172.21.15.60:0/1699560578 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7960053a70 0x7f7960055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:26.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.709+0000 7f7982ffd700 1 --2- 172.21.15.60:0/1699560578 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7960053a70 0x7f7960055f30 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7f7974018000 tx=0x7f7974014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:26.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.713+0000 7f7973fff700 1 -- 172.21.15.60:0/1699560578 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f797c0434a0 con 0x7f79841405a0 2022-06-29T16:07:26.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.837+0000 7f7983fff700 1 -- 172.21.15.60:0/1699560578 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f79840573d0 con 0x7f79841405a0 2022-06-29T16:07:26.843 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.837+0000 7f7973fff700 1 -- 172.21.15.60:0/1699560578 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v128) v1 ==== 92+0+10 (secure 0 0 0) 0x7f797c016070 con 0x7f79841405a0 2022-06-29T16:07:26.844 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:07:26.845 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.841+0000 7f7983fff700 1 -- 172.21.15.60:0/1699560578 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7960053a70 msgr2=0x7f7960055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:26.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.841+0000 7f7983fff700 1 --2- 172.21.15.60:0/1699560578 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7960053a70 0x7f7960055f30 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7f7974018000 tx=0x7f7974014000).stop 2022-06-29T16:07:26.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.841+0000 7f7983fff700 1 -- 172.21.15.60:0/1699560578 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79841405a0 msgr2=0x7f7984143ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:26.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.841+0000 7f7983fff700 1 --2- 172.21.15.60:0/1699560578 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79841405a0 0x7f7984143ed0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 rx=0x7f797c006e00 tx=0x7f797c0091d0).stop 2022-06-29T16:07:26.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.841+0000 7f7983fff700 1 -- 172.21.15.60:0/1699560578 shutdown_connections 2022-06-29T16:07:26.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.841+0000 7f7983fff700 1 --2- 172.21.15.60:0/1699560578 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7960053a70 0x7f7960055f30 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:26.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.841+0000 7f7983fff700 1 --2- 172.21.15.60:0/1699560578 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f79841405a0 0x7f7984143ed0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:26.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.841+0000 7f7983fff700 1 --2- 172.21.15.60:0/1699560578 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7984067a90 0x7f79841439e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:26.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.841+0000 7f7983fff700 1 --2- 172.21.15.60:0/1699560578 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f798405fb00 0x7f7984065730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:26.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.841+0000 7f7983fff700 1 -- 172.21.15.60:0/1699560578 >> 172.21.15.60:0/1699560578 conn(0x7f7984063a60 msgr2=0x7f79840b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:26.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.841+0000 7f7983fff700 1 -- 172.21.15.60:0/1699560578 shutdown_connections 2022-06-29T16:07:26.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.841+0000 7f7983fff700 1 -- 172.21.15.60:0/1699560578 wait complete. 2022-06-29T16:07:26.863 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-29T16:07:26.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.965+0000 7fe2a1b01700 1 Processor -- start 2022-06-29T16:07:26.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.965+0000 7fe2a1b01700 1 -- start start 2022-06-29T16:07:26.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.965+0000 7fe2a1b01700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe29c0af4e0 0x7fe29c13fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:26.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.965+0000 7fe2a1b01700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe29c060780 con 0x7fe29c0b6030 2022-06-29T16:07:26.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.965+0000 7fe2a1b01700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe29c0604e0 con 0x7fe29c0af970 2022-06-29T16:07:26.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.965+0000 7fe2a1b01700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe29c062db0 con 0x7fe29c0af4e0 2022-06-29T16:07:26.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.965+0000 7fe29b7fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe29c0b6030 0x7fe29c143e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33018/0 (socket says 172.21.15.60:33018) 2022-06-29T16:07:26.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.965+0000 7fe29b7fe700 1 -- 172.21.15.60:0/1675006260 learned_addr learned my addr 172.21.15.60:0/1675006260 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:26.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.965+0000 7fe29affd700 1 --2- 172.21.15.60:0/1675006260 >> v2:172.21.15.60:3300/0 conn(0x7fe29c0af4e0 0x7fe29c13fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:26.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.965+0000 7fe29a7fc700 1 -- 172.21.15.60:0/1675006260 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4047770708 0 0) 0x7fe29c060780 con 0x7fe29c0b6030 2022-06-29T16:07:26.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.965+0000 7fe29a7fc700 1 -- 172.21.15.60:0/1675006260 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe288003260 con 0x7fe29c0b6030 2022-06-29T16:07:26.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.969+0000 7fe29affd700 1 -- 172.21.15.60:0/1675006260 >> v1:172.21.15.92:6789/0 conn(0x7fe29c0af970 legacy=0x7fe29c140170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:26.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.969+0000 7fe29affd700 1 -- 172.21.15.60:0/1675006260 >> v1:172.21.15.60:6789/0 conn(0x7fe29c0b6030 legacy=0x7fe29c143e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:26.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.969+0000 7fe29affd700 1 -- 172.21.15.60:0/1675006260 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe29c064130 con 0x7fe29c0af4e0 2022-06-29T16:07:26.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.969+0000 7fe29affd700 1 --2- 172.21.15.60:0/1675006260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe29c0af4e0 0x7fe29c13fc80 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 rx=0x7fe290002f70 tx=0x7fe290006df0).ready entity=mon.0 client_cookie=41c25ee678c06a88 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:26.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.969+0000 7fe29a7fc700 1 -- 172.21.15.60:0/1675006260 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe29000d610 con 0x7fe29c0af4e0 2022-06-29T16:07:26.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.969+0000 7fe29a7fc700 1 -- 172.21.15.60:0/1675006260 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe290000e90 con 0x7fe29c0af4e0 2022-06-29T16:07:26.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.969+0000 7fe29a7fc700 1 -- 172.21.15.60:0/1675006260 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe29000d960 con 0x7fe29c0af4e0 2022-06-29T16:07:26.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.969+0000 7fe2a1b01700 1 -- 172.21.15.60:0/1675006260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe29c0af4e0 msgr2=0x7fe29c13fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:26.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.969+0000 7fe2a1b01700 1 --2- 172.21.15.60:0/1675006260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe29c0af4e0 0x7fe29c13fc80 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 rx=0x7fe290002f70 tx=0x7fe290006df0).stop 2022-06-29T16:07:26.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.969+0000 7fe2a1b01700 1 -- 172.21.15.60:0/1675006260 shutdown_connections 2022-06-29T16:07:26.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.969+0000 7fe2a1b01700 1 --2- 172.21.15.60:0/1675006260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe29c0af4e0 0x7fe29c13fc80 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:26.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.969+0000 7fe2a1b01700 1 -- 172.21.15.60:0/1675006260 >> 172.21.15.60:0/1675006260 conn(0x7fe29c063b50 msgr2=0x7fe29c0b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:26.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.969+0000 7fe2a1b01700 1 -- 172.21.15.60:0/1675006260 shutdown_connections 2022-06-29T16:07:26.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.969+0000 7fe2a1b01700 1 -- 172.21.15.60:0/1675006260 wait complete. 2022-06-29T16:07:26.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.969+0000 7fe2a1b01700 1 Processor -- start 2022-06-29T16:07:26.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.969+0000 7fe2a1b01700 1 -- start start 2022-06-29T16:07:26.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.969+0000 7fe2a1b01700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe29c0af4e0 0x7fe29c065560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:26.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.969+0000 7fe2a1b01700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe29c0af970 0x7fe29c143c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:26.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.969+0000 7fe2a1b01700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe29c0b6030 0x7fe29c13b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:26.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.969+0000 7fe2a1b01700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe29c10e970 con 0x7fe29c0b6030 2022-06-29T16:07:26.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.969+0000 7fe2a1b01700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe29c142500 con 0x7fe29c0af4e0 2022-06-29T16:07:26.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.969+0000 7fe2a1b01700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe29c10eba0 con 0x7fe29c0af970 2022-06-29T16:07:26.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.969+0000 7fe29bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe29c0b6030 0x7fe29c13b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:26.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.969+0000 7fe29bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe29c0b6030 0x7fe29c13b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53174/0 (socket says 172.21.15.60:53174) 2022-06-29T16:07:26.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.969+0000 7fe29bfff700 1 -- 172.21.15.60:0/1023835051 learned_addr learned my addr 172.21.15.60:0/1023835051 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:26.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.969+0000 7fe29b7fe700 1 --2- 172.21.15.60:0/1023835051 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe29c0af4e0 0x7fe29c065560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:26.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.969+0000 7fe29affd700 1 --2- 172.21.15.60:0/1023835051 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe29c0af970 0x7fe29c143c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:26.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.973+0000 7fe29bfff700 1 -- 172.21.15.60:0/1023835051 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe29c0af970 msgr2=0x7fe29c143c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:26.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.973+0000 7fe29bfff700 1 --2- 172.21.15.60:0/1023835051 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe29c0af970 0x7fe29c143c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:26.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.973+0000 7fe29bfff700 1 -- 172.21.15.60:0/1023835051 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe29c0af4e0 msgr2=0x7fe29c065560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:26.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.973+0000 7fe29bfff700 1 --2- 172.21.15.60:0/1023835051 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe29c0af4e0 0x7fe29c065560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:26.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.973+0000 7fe29bfff700 1 -- 172.21.15.60:0/1023835051 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe29000b180 con 0x7fe29c0b6030 2022-06-29T16:07:26.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.973+0000 7fe29bfff700 1 --2- 172.21.15.60:0/1023835051 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe29c0b6030 0x7fe29c13b370 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 rx=0x7fe29c0b2170 tx=0x7fe294007bc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:26.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.973+0000 7fe298ff9700 1 -- 172.21.15.60:0/1023835051 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe294012400 con 0x7fe29c0b6030 2022-06-29T16:07:26.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.973+0000 7fe298ff9700 1 -- 172.21.15.60:0/1023835051 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe294012560 con 0x7fe29c0b6030 2022-06-29T16:07:26.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.973+0000 7fe2a1b01700 1 -- 172.21.15.60:0/1023835051 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe29c142260 con 0x7fe29c0b6030 2022-06-29T16:07:26.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.973+0000 7fe298ff9700 1 -- 172.21.15.60:0/1023835051 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe2940128c0 con 0x7fe29c0b6030 2022-06-29T16:07:26.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.973+0000 7fe2a1b01700 1 -- 172.21.15.60:0/1023835051 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe29c0668b0 con 0x7fe29c0b6030 2022-06-29T16:07:26.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.973+0000 7fe2a1b01700 1 -- 172.21.15.60:0/1023835051 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe29c0573d0 con 0x7fe29c0b6030 2022-06-29T16:07:26.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.973+0000 7fe298ff9700 1 -- 172.21.15.60:0/1023835051 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe294019030 con 0x7fe29c0b6030 2022-06-29T16:07:26.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.973+0000 7fe298ff9700 1 --2- 172.21.15.60:0/1023835051 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe27c053b00 0x7fe27c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:26.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.973+0000 7fe298ff9700 1 -- 172.21.15.60:0/1023835051 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe29406b180 con 0x7fe29c0b6030 2022-06-29T16:07:26.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.977+0000 7fe29b7fe700 1 --2- 172.21.15.60:0/1023835051 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe27c053b00 0x7fe27c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:26.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.977+0000 7fe29b7fe700 1 --2- 172.21.15.60:0/1023835051 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe27c053b00 0x7fe27c055fc0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7fe28c00d000 tx=0x7fe28c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:26.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:26.985+0000 7fe298ff9700 1 -- 172.21.15.60:0/1023835051 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe2940420a0 con 0x7fe29c0b6030 2022-06-29T16:07:27.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.105+0000 7fe2a1b01700 1 -- 172.21.15.60:0/1023835051 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe29c065290 con 0x7fe29c0b6030 2022-06-29T16:07:27.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.109+0000 7fe298ff9700 1 -- 172.21.15.60:0/1023835051 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v128) v1 ==== 104+0+10 (secure 0 0 0) 0x7fe2940456c0 con 0x7fe29c0b6030 2022-06-29T16:07:27.113 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:07:27.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.113+0000 7fe2a1b01700 1 -- 172.21.15.60:0/1023835051 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe27c053b00 msgr2=0x7fe27c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:27.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.113+0000 7fe2a1b01700 1 --2- 172.21.15.60:0/1023835051 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe27c053b00 0x7fe27c055fc0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7fe28c00d000 tx=0x7fe28c009040).stop 2022-06-29T16:07:27.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.113+0000 7fe2a1b01700 1 -- 172.21.15.60:0/1023835051 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe29c0b6030 msgr2=0x7fe29c13b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:27.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.113+0000 7fe2a1b01700 1 --2- 172.21.15.60:0/1023835051 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe29c0b6030 0x7fe29c13b370 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 rx=0x7fe29c0b2170 tx=0x7fe294007bc0).stop 2022-06-29T16:07:27.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.113+0000 7fe2a1b01700 1 -- 172.21.15.60:0/1023835051 shutdown_connections 2022-06-29T16:07:27.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.113+0000 7fe2a1b01700 1 --2- 172.21.15.60:0/1023835051 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe27c053b00 0x7fe27c055fc0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:27.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.113+0000 7fe2a1b01700 1 --2- 172.21.15.60:0/1023835051 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe29c0b6030 0x7fe29c13b370 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:27.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.113+0000 7fe2a1b01700 1 --2- 172.21.15.60:0/1023835051 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe29c0af970 0x7fe29c143c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:27.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.113+0000 7fe2a1b01700 1 --2- 172.21.15.60:0/1023835051 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe29c0af4e0 0x7fe29c065560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:27.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.113+0000 7fe2a1b01700 1 -- 172.21.15.60:0/1023835051 >> 172.21.15.60:0/1023835051 conn(0x7fe29c063b50 msgr2=0x7fe29c13f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:27.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.113+0000 7fe2a1b01700 1 -- 172.21.15.60:0/1023835051 shutdown_connections 2022-06-29T16:07:27.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.113+0000 7fe2a1b01700 1 -- 172.21.15.60:0/1023835051 wait complete. 2022-06-29T16:07:27.131 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-29T16:07:27.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.237+0000 7fdc3e183700 1 Processor -- start 2022-06-29T16:07:27.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.237+0000 7fdc3e183700 1 -- start start 2022-06-29T16:07:27.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.237+0000 7fdc3e183700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc380b4010 0x7fdc381426b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:27.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.237+0000 7fdc3e183700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fdc380b46c0 0x7fdc380b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:27.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.237+0000 7fdc3e183700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc38143a40 con 0x7fdc380ae7c0 2022-06-29T16:07:27.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.237+0000 7fdc3e183700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc380aee60 con 0x7fdc380b46c0 2022-06-29T16:07:27.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.237+0000 7fdc3e183700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc380af100 con 0x7fdc380b4010 2022-06-29T16:07:27.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.237+0000 7fdc377fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fdc380ae7c0 0x7fdc38141fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33032/0 (socket says 172.21.15.60:33032) 2022-06-29T16:07:27.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.237+0000 7fdc377fe700 1 -- 172.21.15.60:0/3671184308 learned_addr learned my addr 172.21.15.60:0/3671184308 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:27.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.237+0000 7fdc37fff700 1 --2- 172.21.15.60:0/3671184308 >> v2:172.21.15.60:3300/0 conn(0x7fdc380b46c0 0x7fdc380b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:27.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.237+0000 7fdc2ffff700 1 --2- 172.21.15.60:0/3671184308 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc380b4010 0x7fdc381426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:27.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.237+0000 7fdc36ffd700 1 -- 172.21.15.60:0/3671184308 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 901648743 0 0) 0x7fdc38143a40 con 0x7fdc380ae7c0 2022-06-29T16:07:27.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.237+0000 7fdc36ffd700 1 -- 172.21.15.60:0/3671184308 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc18003260 con 0x7fdc380ae7c0 2022-06-29T16:07:27.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.237+0000 7fdc37fff700 1 -- 172.21.15.60:0/3671184308 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc380b4010 msgr2=0x7fdc381426b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:27.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.237+0000 7fdc37fff700 1 --2- 172.21.15.60:0/3671184308 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc380b4010 0x7fdc381426b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:27.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.237+0000 7fdc37fff700 1 -- 172.21.15.60:0/3671184308 >> v1:172.21.15.60:6789/0 conn(0x7fdc380ae7c0 legacy=0x7fdc38141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:27.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.237+0000 7fdc37fff700 1 -- 172.21.15.60:0/3671184308 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc380aeba0 con 0x7fdc380b46c0 2022-06-29T16:07:27.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.237+0000 7fdc37fff700 1 --2- 172.21.15.60:0/3671184308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc380b46c0 0x7fdc380b4b50 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 rx=0x7fdc300032f0 tx=0x7fdc300091d0).ready entity=mon.0 client_cookie=768adcb25314e4f0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:27.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.237+0000 7fdc36ffd700 1 -- 172.21.15.60:0/3671184308 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc3000b600 con 0x7fdc380b46c0 2022-06-29T16:07:27.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.237+0000 7fdc36ffd700 1 -- 172.21.15.60:0/3671184308 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc3000b760 con 0x7fdc380b46c0 2022-06-29T16:07:27.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.237+0000 7fdc36ffd700 1 -- 172.21.15.60:0/3671184308 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc3000b8d0 con 0x7fdc380b46c0 2022-06-29T16:07:27.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.241+0000 7fdc3e183700 1 -- 172.21.15.60:0/3671184308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc380b46c0 msgr2=0x7fdc380b4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:27.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.241+0000 7fdc3e183700 1 --2- 172.21.15.60:0/3671184308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc380b46c0 0x7fdc380b4b50 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 rx=0x7fdc300032f0 tx=0x7fdc300091d0).stop 2022-06-29T16:07:27.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.241+0000 7fdc3e183700 1 -- 172.21.15.60:0/3671184308 shutdown_connections 2022-06-29T16:07:27.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.241+0000 7fdc3e183700 1 --2- 172.21.15.60:0/3671184308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc380b46c0 0x7fdc380b4b50 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:27.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.241+0000 7fdc3e183700 1 --2- 172.21.15.60:0/3671184308 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc380b4010 0x7fdc381426b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:27.254 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.241+0000 7fdc3e183700 1 -- 172.21.15.60:0/3671184308 >> 172.21.15.60:0/3671184308 conn(0x7fdc380b9340 msgr2=0x7fdc380b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:27.254 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.241+0000 7fdc3e183700 1 -- 172.21.15.60:0/3671184308 shutdown_connections 2022-06-29T16:07:27.254 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.241+0000 7fdc3e183700 1 -- 172.21.15.60:0/3671184308 wait complete. 2022-06-29T16:07:27.255 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.241+0000 7fdc3e183700 1 Processor -- start 2022-06-29T16:07:27.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.241+0000 7fdc3e183700 1 -- start start 2022-06-29T16:07:27.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.241+0000 7fdc3e183700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc380ae7c0 0x7fdc380b2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:27.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.241+0000 7fdc3e183700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc380b4010 0x7fdc380b2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:27.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.241+0000 7fdc3e183700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc380b46c0 0x7fdc38155250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:27.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.241+0000 7fdc3e183700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc380af790 con 0x7fdc380ae7c0 2022-06-29T16:07:27.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.241+0000 7fdc3e183700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc380b0c60 con 0x7fdc380b4010 2022-06-29T16:07:27.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.241+0000 7fdc3e183700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc380b0f00 con 0x7fdc380b46c0 2022-06-29T16:07:27.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.241+0000 7fdc377fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc380ae7c0 0x7fdc380b2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:27.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.241+0000 7fdc377fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc380ae7c0 0x7fdc380b2940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53184/0 (socket says 172.21.15.60:53184) 2022-06-29T16:07:27.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.241+0000 7fdc377fe700 1 -- 172.21.15.60:0/1705112646 learned_addr learned my addr 172.21.15.60:0/1705112646 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:27.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.241+0000 7fdc2ffff700 1 --2- 172.21.15.60:0/1705112646 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc380b4010 0x7fdc380b2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:27.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.241+0000 7fdc37fff700 1 --2- 172.21.15.60:0/1705112646 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc380b46c0 0x7fdc38155250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:27.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.241+0000 7fdc377fe700 1 -- 172.21.15.60:0/1705112646 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc380b46c0 msgr2=0x7fdc38155250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:27.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.241+0000 7fdc377fe700 1 --2- 172.21.15.60:0/1705112646 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc380b46c0 0x7fdc38155250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:27.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.241+0000 7fdc377fe700 1 -- 172.21.15.60:0/1705112646 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc380b4010 msgr2=0x7fdc380b2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:27.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.241+0000 7fdc377fe700 1 --2- 172.21.15.60:0/1705112646 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc380b4010 0x7fdc380b2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:27.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.241+0000 7fdc377fe700 1 -- 172.21.15.60:0/1705112646 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc30009470 con 0x7fdc380ae7c0 2022-06-29T16:07:27.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.241+0000 7fdc377fe700 1 --2- 172.21.15.60:0/1705112646 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc380ae7c0 0x7fdc380b2940 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 rx=0x7fdc28001110 tx=0x7fdc28007aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:27.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.241+0000 7fdc357fa700 1 -- 172.21.15.60:0/1705112646 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc280070d0 con 0x7fdc380ae7c0 2022-06-29T16:07:27.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.241+0000 7fdc357fa700 1 -- 172.21.15.60:0/1705112646 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc28007230 con 0x7fdc380ae7c0 2022-06-29T16:07:27.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.241+0000 7fdc3e183700 1 -- 172.21.15.60:0/1705112646 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc38146b40 con 0x7fdc380ae7c0 2022-06-29T16:07:27.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.241+0000 7fdc357fa700 1 -- 172.21.15.60:0/1705112646 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc28005610 con 0x7fdc380ae7c0 2022-06-29T16:07:27.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.241+0000 7fdc3e183700 1 -- 172.21.15.60:0/1705112646 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc38155740 con 0x7fdc380ae7c0 2022-06-29T16:07:27.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.245+0000 7fdc3e183700 1 -- 172.21.15.60:0/1705112646 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc380b0aa0 con 0x7fdc380ae7c0 2022-06-29T16:07:27.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.245+0000 7fdc357fa700 1 -- 172.21.15.60:0/1705112646 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdc28010070 con 0x7fdc380ae7c0 2022-06-29T16:07:27.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.245+0000 7fdc357fa700 1 --2- 172.21.15.60:0/1705112646 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdc1c0539e0 0x7fdc1c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:27.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.245+0000 7fdc357fa700 1 -- 172.21.15.60:0/1705112646 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fdc2806b600 con 0x7fdc380ae7c0 2022-06-29T16:07:27.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.245+0000 7fdc2ffff700 1 --2- 172.21.15.60:0/1705112646 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdc1c0539e0 0x7fdc1c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:27.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.249+0000 7fdc2ffff700 1 --2- 172.21.15.60:0/1705112646 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdc1c0539e0 0x7fdc1c055ea0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0x7fdc2000b000 tx=0x7fdc20006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:27.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.253+0000 7fdc357fa700 1 -- 172.21.15.60:0/1705112646 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdc28042410 con 0x7fdc380ae7c0 2022-06-29T16:07:27.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.385+0000 7fdc3e183700 1 -- 172.21.15.60:0/1705112646 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdc380b10d0 con 0x7fdc380ae7c0 2022-06-29T16:07:27.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.385+0000 7fdc357fa700 1 -- 172.21.15.60:0/1705112646 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v128) v1 ==== 100+0+10 (secure 0 0 0) 0x7fdc28018070 con 0x7fdc380ae7c0 2022-06-29T16:07:27.390 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:07:27.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.389+0000 7fdc3e183700 1 -- 172.21.15.60:0/1705112646 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdc1c0539e0 msgr2=0x7fdc1c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:27.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.389+0000 7fdc3e183700 1 --2- 172.21.15.60:0/1705112646 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdc1c0539e0 0x7fdc1c055ea0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0x7fdc2000b000 tx=0x7fdc20006df0).stop 2022-06-29T16:07:27.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.389+0000 7fdc3e183700 1 -- 172.21.15.60:0/1705112646 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc380ae7c0 msgr2=0x7fdc380b2940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:27.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.389+0000 7fdc3e183700 1 --2- 172.21.15.60:0/1705112646 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc380ae7c0 0x7fdc380b2940 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 rx=0x7fdc28001110 tx=0x7fdc28007aa0).stop 2022-06-29T16:07:27.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.389+0000 7fdc3e183700 1 -- 172.21.15.60:0/1705112646 shutdown_connections 2022-06-29T16:07:27.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.389+0000 7fdc3e183700 1 --2- 172.21.15.60:0/1705112646 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdc1c0539e0 0x7fdc1c055ea0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:27.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.389+0000 7fdc3e183700 1 --2- 172.21.15.60:0/1705112646 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdc380b46c0 0x7fdc38155250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:27.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.389+0000 7fdc3e183700 1 --2- 172.21.15.60:0/1705112646 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc380b4010 0x7fdc380b2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:27.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.389+0000 7fdc3e183700 1 --2- 172.21.15.60:0/1705112646 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc380ae7c0 0x7fdc380b2940 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:27.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.389+0000 7fdc3e183700 1 -- 172.21.15.60:0/1705112646 >> 172.21.15.60:0/1705112646 conn(0x7fdc380b9340 msgr2=0x7fdc380b8450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:27.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.389+0000 7fdc3e183700 1 -- 172.21.15.60:0/1705112646 shutdown_connections 2022-06-29T16:07:27.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.389+0000 7fdc3e183700 1 -- 172.21.15.60:0/1705112646 wait complete. 2022-06-29T16:07:27.409 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-06-29T16:07:27.410 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:07:27.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e0e796700 1 Processor -- start 2022-06-29T16:07:27.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e0e796700 1 -- start start 2022-06-29T16:07:27.517 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e0e796700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3e08063ad0 0x7f3e08067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:27.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e0e796700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e080acba0 con 0x7f3e080b5950 2022-06-29T16:07:27.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e0e796700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e08067220 con 0x7f3e0814af10 2022-06-29T16:07:27.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e0e796700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e0814b3a0 con 0x7f3e08063ad0 2022-06-29T16:07:27.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e07fff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f3e080b5950 0x7f3e0805bd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33046/0 (socket says 172.21.15.60:33046) 2022-06-29T16:07:27.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e07fff700 1 -- 172.21.15.60:0/2460802947 learned_addr learned my addr 172.21.15.60:0/2460802947 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:27.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e077fe700 1 --2- 172.21.15.60:0/2460802947 >> v2:172.21.15.92:3300/0 conn(0x7f3e08063ad0 0x7f3e08067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:27.519 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e06ffd700 1 -- 172.21.15.60:0/2460802947 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172679094 0 0) 0x7f3e080acba0 con 0x7f3e080b5950 2022-06-29T16:07:27.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e06ffd700 1 -- 172.21.15.60:0/2460802947 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dec003260 con 0x7f3e080b5950 2022-06-29T16:07:27.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e06ffd700 1 -- 172.21.15.60:0/2460802947 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 106839246 0 0) 0x7f3dec003260 con 0x7f3e080b5950 2022-06-29T16:07:27.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e06ffd700 1 -- 172.21.15.60:0/2460802947 >> v2:172.21.15.92:3300/0 conn(0x7f3e08063ad0 msgr2=0x7f3e08067680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:27.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e06ffd700 1 --2- 172.21.15.60:0/2460802947 >> v2:172.21.15.92:3300/0 conn(0x7f3e08063ad0 0x7f3e08067680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:27.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e06ffd700 1 -- 172.21.15.60:0/2460802947 >> v1:172.21.15.92:6789/0 conn(0x7f3e0814af10 legacy=0x7f3e0805c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:27.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e06ffd700 1 -- 172.21.15.60:0/2460802947 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e080b5d30 con 0x7f3e080b5950 2022-06-29T16:07:27.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e06ffd700 1 -- 172.21.15.60:0/2460802947 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f3df0001680 con 0x7f3e080b5950 2022-06-29T16:07:27.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e06ffd700 1 -- 172.21.15.60:0/2460802947 >> v1:172.21.15.60:6789/0 conn(0x7f3e080b5950 legacy=0x7f3e0805bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:27.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e06ffd700 1 --2- 172.21.15.60:0/2460802947 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3dec003750 0x7f3dec003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:27.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e06ffd700 1 --2- 172.21.15.60:0/2460802947 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3dec004500 0x7f3dec004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:27.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e06ffd700 1 --2- 172.21.15.60:0/2460802947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dec004e70 0x7f3dec00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:27.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e06ffd700 1 -- 172.21.15.60:0/2460802947 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0814b3a0 con 0x7f3dec004e70 2022-06-29T16:07:27.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e06ffd700 1 -- 172.21.15.60:0/2460802947 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dec00bd20 con 0x7f3dec003750 2022-06-29T16:07:27.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e06ffd700 1 -- 172.21.15.60:0/2460802947 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dec00bee0 con 0x7f3dec004500 2022-06-29T16:07:27.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e07fff700 1 --2- 172.21.15.60:0/2460802947 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3dec003750 0x7f3dec003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:27.524 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e077fe700 1 --2- 172.21.15.60:0/2460802947 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3dec004500 0x7f3dec004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:27.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e0cd33700 1 --2- 172.21.15.60:0/2460802947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dec004e70 0x7f3dec00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:27.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e07fff700 1 -- 172.21.15.60:0/2460802947 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3dec004500 msgr2=0x7f3dec004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:27.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e07fff700 1 --2- 172.21.15.60:0/2460802947 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3dec004500 0x7f3dec004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:27.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e07fff700 1 -- 172.21.15.60:0/2460802947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dec004e70 msgr2=0x7f3dec00b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:27.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e07fff700 1 --2- 172.21.15.60:0/2460802947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dec004e70 0x7f3dec00b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:27.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e07fff700 1 -- 172.21.15.60:0/2460802947 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dec003260 con 0x7f3dec003750 2022-06-29T16:07:27.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e0cd33700 1 -- 172.21.15.60:0/2460802947 reap_dead start 2022-06-29T16:07:27.526 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e07fff700 1 --2- 172.21.15.60:0/2460802947 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3dec003750 0x7f3dec003b70 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7f3df0001390 tx=0x7f3e0805ce50).ready entity=mon.1 client_cookie=e1c42c2ba552c0ca server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:27.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.509+0000 7f3e06ffd700 1 -- 172.21.15.60:0/2460802947 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3df0007170 con 0x7f3dec003750 2022-06-29T16:07:27.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.513+0000 7f3e06ffd700 1 -- 172.21.15.60:0/2460802947 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df00072d0 con 0x7f3dec003750 2022-06-29T16:07:27.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.513+0000 7f3e06ffd700 1 -- 172.21.15.60:0/2460802947 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3df000c9c0 con 0x7f3dec003750 2022-06-29T16:07:27.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.513+0000 7f3e0e796700 1 -- 172.21.15.60:0/2460802947 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3dec003750 msgr2=0x7f3dec003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:27.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.513+0000 7f3e0e796700 1 --2- 172.21.15.60:0/2460802947 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3dec003750 0x7f3dec003b70 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7f3df0001390 tx=0x7f3e0805ce50).stop 2022-06-29T16:07:27.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.513+0000 7f3e0e796700 1 -- 172.21.15.60:0/2460802947 shutdown_connections 2022-06-29T16:07:27.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.513+0000 7f3e0e796700 1 --2- 172.21.15.60:0/2460802947 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3dec003750 0x7f3dec003b70 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:27.528 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.513+0000 7f3e0e796700 1 -- 172.21.15.60:0/2460802947 >> 172.21.15.60:0/2460802947 conn(0x7f3e080632c0 msgr2=0x7f3e080b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:27.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.513+0000 7f3e0e796700 1 -- 172.21.15.60:0/2460802947 shutdown_connections 2022-06-29T16:07:27.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.513+0000 7f3e0e796700 1 -- 172.21.15.60:0/2460802947 wait complete. 2022-06-29T16:07:27.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.513+0000 7f3e0e796700 1 Processor -- start 2022-06-29T16:07:27.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.513+0000 7f3e0e796700 1 -- start start 2022-06-29T16:07:27.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.513+0000 7f3e0e796700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3dec003750 0x7f3e080b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:27.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.513+0000 7f3e0e796700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e08116d90 0x7f3e08142340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:27.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.513+0000 7f3e0e796700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e08142830 0x7f3e08118230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:27.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.513+0000 7f3e0e796700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e08142ce0 con 0x7f3e08142830 2022-06-29T16:07:27.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.513+0000 7f3e0e796700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e080af1f0 con 0x7f3e08116d90 2022-06-29T16:07:27.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.513+0000 7f3e0e796700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e080af420 con 0x7f3dec003750 2022-06-29T16:07:27.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.513+0000 7f3e0cd33700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e08142830 0x7f3e08118230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:27.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.513+0000 7f3e0cd33700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e08142830 0x7f3e08118230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53208/0 (socket says 172.21.15.60:53208) 2022-06-29T16:07:27.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.513+0000 7f3e0cd33700 1 -- 172.21.15.60:0/2346749401 learned_addr learned my addr 172.21.15.60:0/2346749401 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:27.539 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.517+0000 7f3e07fff700 1 --2- 172.21.15.60:0/2346749401 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3dec003750 0x7f3e080b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:27.539 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.517+0000 7f3e077fe700 1 --2- 172.21.15.60:0/2346749401 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e08116d90 0x7f3e08142340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:27.539 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.517+0000 7f3e0cd33700 1 -- 172.21.15.60:0/2346749401 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3dec003750 msgr2=0x7f3e080b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:27.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.517+0000 7f3e0cd33700 1 --2- 172.21.15.60:0/2346749401 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3dec003750 0x7f3e080b43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:27.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.517+0000 7f3e0cd33700 1 -- 172.21.15.60:0/2346749401 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e08116d90 msgr2=0x7f3e08142340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:27.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.517+0000 7f3e0cd33700 1 --2- 172.21.15.60:0/2346749401 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e08116d90 0x7f3e08142340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:27.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.517+0000 7f3e0cd33700 1 -- 172.21.15.60:0/2346749401 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e08159550 con 0x7f3e08142830 2022-06-29T16:07:27.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.517+0000 7f3e07fff700 1 --2- 172.21.15.60:0/2346749401 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3dec003750 0x7f3e080b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:27.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.517+0000 7f3e0cd33700 1 --2- 172.21.15.60:0/2346749401 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e08142830 0x7f3e08118230 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 rx=0x7f3df0001110 tx=0x7f3df8007a00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:27.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.517+0000 7f3e057fa700 1 -- 172.21.15.60:0/2346749401 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3df8002e40 con 0x7f3e08142830 2022-06-29T16:07:27.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.517+0000 7f3e057fa700 1 -- 172.21.15.60:0/2346749401 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df8007240 con 0x7f3e08142830 2022-06-29T16:07:27.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.517+0000 7f3e0e796700 1 -- 172.21.15.60:0/2346749401 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e080aedf0 con 0x7f3e08142830 2022-06-29T16:07:27.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.517+0000 7f3e057fa700 1 -- 172.21.15.60:0/2346749401 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3df8006510 con 0x7f3e08142830 2022-06-29T16:07:27.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.517+0000 7f3e0e796700 1 -- 172.21.15.60:0/2346749401 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e080b09d0 con 0x7f3e08142830 2022-06-29T16:07:27.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.517+0000 7f3e0e796700 1 -- 172.21.15.60:0/2346749401 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e08159740 con 0x7f3e08142830 2022-06-29T16:07:27.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.517+0000 7f3e057fa700 1 -- 172.21.15.60:0/2346749401 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3df8006670 con 0x7f3e08142830 2022-06-29T16:07:27.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.517+0000 7f3e057fa700 1 --2- 172.21.15.60:0/2346749401 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3de0053a70 0x7f3de0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:27.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.517+0000 7f3e057fa700 1 -- 172.21.15.60:0/2346749401 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3df806c080 con 0x7f3e08142830 2022-06-29T16:07:27.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.517+0000 7f3e07fff700 1 --2- 172.21.15.60:0/2346749401 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3de0053a70 0x7f3de0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:27.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.521+0000 7f3e07fff700 1 --2- 172.21.15.60:0/2346749401 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3de0053a70 0x7f3de0055f30 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7f3df0018000 tx=0x7f3df0013000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:27.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.525+0000 7f3e057fa700 1 -- 172.21.15.60:0/2346749401 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3df8042f40 con 0x7f3e08142830 2022-06-29T16:07:27.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.689+0000 7f3e0e796700 1 -- 172.21.15.60:0/2346749401 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e080af590 con 0x7f3e08142830 2022-06-29T16:07:27.696 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.689+0000 7f3e057fa700 1 -- 172.21.15.60:0/2346749401 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2142 (secure 0 0 0) 0x7f3e080af590 con 0x7f3e08142830 2022-06-29T16:07:27.697 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:27.697 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":62,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6941,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/3488623391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3488623391},{"type":"v1","addr":"172.21.15.92:6835","nonce":3488623391}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":59}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:07:19.341489+0000","modified":"2022-06-29T16:07:20.345065+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8085},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8085":{"gid":8085,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/3877729627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3877729627},{"type":"v1","addr":"172.21.15.60:6833","nonce":3877729627}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-06-29T16:07:27.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.693+0000 7f3e0e796700 1 -- 172.21.15.60:0/2346749401 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3de0053a70 msgr2=0x7f3de0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:27.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.693+0000 7f3e0e796700 1 --2- 172.21.15.60:0/2346749401 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3de0053a70 0x7f3de0055f30 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7f3df0018000 tx=0x7f3df0013000).stop 2022-06-29T16:07:27.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.693+0000 7f3e0e796700 1 -- 172.21.15.60:0/2346749401 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e08142830 msgr2=0x7f3e08118230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:27.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.693+0000 7f3e0e796700 1 --2- 172.21.15.60:0/2346749401 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e08142830 0x7f3e08118230 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 rx=0x7f3df0001110 tx=0x7f3df8007a00).stop 2022-06-29T16:07:27.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.693+0000 7f3e0e796700 1 -- 172.21.15.60:0/2346749401 shutdown_connections 2022-06-29T16:07:27.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.693+0000 7f3e0e796700 1 --2- 172.21.15.60:0/2346749401 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3de0053a70 0x7f3de0055f30 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:27.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.693+0000 7f3e0e796700 1 --2- 172.21.15.60:0/2346749401 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e08142830 0x7f3e08118230 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:27.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.693+0000 7f3e0e796700 1 --2- 172.21.15.60:0/2346749401 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e08116d90 0x7f3e08142340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:27.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.693+0000 7f3e0e796700 1 --2- 172.21.15.60:0/2346749401 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3dec003750 0x7f3e080b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:27.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.693+0000 7f3e0e796700 1 -- 172.21.15.60:0/2346749401 >> 172.21.15.60:0/2346749401 conn(0x7f3e080632c0 msgr2=0x7f3e081431c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:27.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.693+0000 7f3e0e796700 1 -- 172.21.15.60:0/2346749401 shutdown_connections 2022-06-29T16:07:27.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:27.693+0000 7f3e0e796700 1 -- 172.21.15.60:0/2346749401 wait complete. 2022-06-29T16:07:27.705 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 62 2022-06-29T16:07:27.910 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:07:19.341489+0000', 'modified': '2022-06-29T16:07:20.345065+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8085}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8085': {'gid': 8085, 'name': 'a', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.60:6833/3877729627', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6832', 'nonce': 3877729627}, {'type': 'v1', 'addr': '172.21.15.60:6833', 'nonce': 3877729627}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:07:27.910 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:07:27.911 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-06-29T16:07:28.028 INFO:teuthology.orchestra.run.smithi060.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":8085,"want_state":"up:active","state":"up:active","rank_uptime":8.6717072739999992,"mdsmap_epoch":62,"osdmap_epoch":128,"osdmap_epoch_barrier":128,"uptime":13.396751132} 2022-06-29T16:07:28.028 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 8085, "mdsmap_epoch": 62, "osdmap_epoch": 128, "osdmap_epoch_barrier": 128, "rank_uptime": 8.671707274, "state": "up:active", "uptime": 13.396751132, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:07:28.029 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-06-29T16:07:28.029 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-06-29T16:07:28.030 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi156.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-06-29T16:07:28.030 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:28.031 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:07:28.039 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2022-06-29T16:07:28.054 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:07:28.055 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:07:28.109 INFO:teuthology.orchestra.run.smithi156.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:07:28.110 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:07:28.111 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:28.112 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2022-06-29T16:07:28.116 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-06-29T16:07:28.117 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:07:28.161 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2022-06-29T16:07:28.179 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:07:28.180 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:07:28.190 INFO:teuthology.orchestra.run.smithi156.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:07:28.191 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:07:28.192 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:28.192 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2022-06-29T16:07:28.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:2022-06-29T16:07:28.229+0000 7fdf59bf9080 -1 init, newargv = 0x55a90bee40c0 newargc=9ceph-fuse[12901]: starting ceph client 2022-06-29T16:07:28.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr: 2022-06-29T16:07:28.245 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[12901]: starting fuse 2022-06-29T16:07:29.343 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2022-06-29T16:07:29.362 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:07:29.363 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:07:29.422 INFO:teuthology.orchestra.run.smithi156.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:07:29.423 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:07:29.424 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:29.425 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2022-06-29T16:07:29.431 INFO:teuthology.orchestra.run.smithi156.stdout:52 2022-06-29T16:07:29.431 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-06-29T16:07:29.432 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:29.432 DEBUG:teuthology.orchestra.run.smithi156:> sudo python3 -c ' 2022-06-29T16:07:29.433 DEBUG:teuthology.orchestra.run.smithi156:> import glob 2022-06-29T16:07:29.433 DEBUG:teuthology.orchestra.run.smithi156:> import re 2022-06-29T16:07:29.434 DEBUG:teuthology.orchestra.run.smithi156:> import os 2022-06-29T16:07:29.434 DEBUG:teuthology.orchestra.run.smithi156:> import subprocess 2022-06-29T16:07:29.434 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:07:29.435 DEBUG:teuthology.orchestra.run.smithi156:> def find_socket(client_name): 2022-06-29T16:07:29.435 DEBUG:teuthology.orchestra.run.smithi156:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-06-29T16:07:29.436 DEBUG:teuthology.orchestra.run.smithi156:> files = glob.glob(asok_path) 2022-06-29T16:07:29.436 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:07:29.437 DEBUG:teuthology.orchestra.run.smithi156:> # Given a non-glob path, it better be there 2022-06-29T16:07:29.437 DEBUG:teuthology.orchestra.run.smithi156:> if "*" not in asok_path: 2022-06-29T16:07:29.437 DEBUG:teuthology.orchestra.run.smithi156:> assert(len(files) == 1) 2022-06-29T16:07:29.438 DEBUG:teuthology.orchestra.run.smithi156:> return files[0] 2022-06-29T16:07:29.438 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:07:29.439 DEBUG:teuthology.orchestra.run.smithi156:> for f in files: 2022-06-29T16:07:29.439 DEBUG:teuthology.orchestra.run.smithi156:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-06-29T16:07:29.439 DEBUG:teuthology.orchestra.run.smithi156:> if os.path.exists("/proc/{0}".format(pid)): 2022-06-29T16:07:29.440 DEBUG:teuthology.orchestra.run.smithi156:> return f 2022-06-29T16:07:29.440 DEBUG:teuthology.orchestra.run.smithi156:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-06-29T16:07:29.441 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:07:29.441 DEBUG:teuthology.orchestra.run.smithi156:> print(find_socket("client.1")) 2022-06-29T16:07:29.442 DEBUG:teuthology.orchestra.run.smithi156:> ' 2022-06-29T16:07:29.485 INFO:teuthology.orchestra.run.smithi156.stdout:/var/run/ceph/ceph-client.1.12901.asok 2022-06-29T16:07:29.490 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.12901.asok 2022-06-29T16:07:29.491 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:29.491 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.12901.asok status 2022-06-29T16:07:29.626 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2022-06-29T16:07:29.626 INFO:teuthology.orchestra.run.smithi156.stdout: "metadata": { 2022-06-29T16:07:29.627 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_sha1": "f25ffac72e5a9e63860ff7deb6767a11f0559a16", 2022-06-29T16:07:29.627 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_version": "ceph version 15.2.16-304-gf25ffac7 (f25ffac72e5a9e63860ff7deb6767a11f0559a16) octopus (stable)", 2022-06-29T16:07:29.627 INFO:teuthology.orchestra.run.smithi156.stdout: "entity_id": "1", 2022-06-29T16:07:29.627 INFO:teuthology.orchestra.run.smithi156.stdout: "hostname": "smithi156", 2022-06-29T16:07:29.627 INFO:teuthology.orchestra.run.smithi156.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-06-29T16:07:29.628 INFO:teuthology.orchestra.run.smithi156.stdout: "pid": "12901", 2022-06-29T16:07:29.628 INFO:teuthology.orchestra.run.smithi156.stdout: "root": "/" 2022-06-29T16:07:29.628 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:07:29.628 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_count": 0, 2022-06-29T16:07:29.629 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_pinned_count": 0, 2022-06-29T16:07:29.629 INFO:teuthology.orchestra.run.smithi156.stdout: "id": 8307, 2022-06-29T16:07:29.629 INFO:teuthology.orchestra.run.smithi156.stdout: "inst": { 2022-06-29T16:07:29.629 INFO:teuthology.orchestra.run.smithi156.stdout: "name": { 2022-06-29T16:07:29.629 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "client", 2022-06-29T16:07:29.630 INFO:teuthology.orchestra.run.smithi156.stdout: "num": 8307 2022-06-29T16:07:29.630 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:07:29.630 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2022-06-29T16:07:29.630 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2022-06-29T16:07:29.630 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "172.21.15.156:0", 2022-06-29T16:07:29.631 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 3377529588 2022-06-29T16:07:29.631 INFO:teuthology.orchestra.run.smithi156.stdout: } 2022-06-29T16:07:29.631 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:07:29.631 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2022-06-29T16:07:29.632 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2022-06-29T16:07:29.632 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "172.21.15.156:0", 2022-06-29T16:07:29.632 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 3377529588 2022-06-29T16:07:29.632 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:07:29.632 INFO:teuthology.orchestra.run.smithi156.stdout: "inst_str": "client.8307 172.21.15.156:0/3377529588", 2022-06-29T16:07:29.633 INFO:teuthology.orchestra.run.smithi156.stdout: "addr_str": "172.21.15.156:0/3377529588", 2022-06-29T16:07:29.633 INFO:teuthology.orchestra.run.smithi156.stdout: "inode_count": 1, 2022-06-29T16:07:29.633 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_epoch": 62, 2022-06-29T16:07:29.633 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch": 128, 2022-06-29T16:07:29.634 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch_barrier": 0, 2022-06-29T16:07:29.634 INFO:teuthology.orchestra.run.smithi156.stdout: "blacklisted": false, 2022-06-29T16:07:29.634 INFO:teuthology.orchestra.run.smithi156.stdout: "fs_name": "cephfs" 2022-06-29T16:07:29.634 INFO:teuthology.orchestra.run.smithi156.stdout:} 2022-06-29T16:07:29.635 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:07:29.636 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:29.637 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:07:29.637 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:07:29.639 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:07:29.643 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:07:29.644 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2022-06-29T16:07:29.645 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-06-29T16:07:29.646 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:29.646 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:07:29.647 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:07:29.689 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:07:29.705 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:29.706 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:07:29.706 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:07:29.713 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2022-06-29T16:07:29.715 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-06-29T16:07:29.715 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:29.716 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:07:29.716 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:07:29.793 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2022-06-29T16:07:29.795 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-06-29T16:07:29.796 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi156.front.sepia.ceph.com... 2022-06-29T16:07:29.796 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-06-29T16:07:29.797 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:07:29.812 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[12901]: fuse finished with error 0 and tester_r 0 2022-06-29T16:07:29.814 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:29.815 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:07:29.815 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:07:29.823 INFO:teuthology.orchestra.run.smithi156.stdout:ext2/ext3 2022-06-29T16:07:29.824 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-06-29T16:07:29.824 INFO:teuthology.orchestra.run:waiting for 900 2022-06-29T16:07:35.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:07:35.927 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:07:35.937 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-06-29T16:07:35.941 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-06-29T16:07:35.944 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-06-29T16:07:35.947 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-06-29T16:07:35.951 DEBUG:teuthology.orchestra.run.smithi197:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-06-29T16:07:35.973 INFO:teuthology.orchestra.run.smithi060.stdout:[global] 2022-06-29T16:07:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: chdir = "" 2022-06-29T16:07:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-06-29T16:07:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: auth supported = cephx 2022-06-29T16:07:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: filestore xattr use omap = true 2022-06-29T16:07:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: mon clock drift allowed = 1.000 2022-06-29T16:07:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: osd crush chooseleaf type = 0 2022-06-29T16:07:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: auth debug = true 2022-06-29T16:07:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on old message = true 2022-06-29T16:07:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on bug = true 2022-06-29T16:07:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: mon max pg per osd = 10000 # >= luminous 2022-06-29T16:07:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: mon pg warn max object skew = 0 2022-06-29T16:07:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: # disable pg_autoscaler by default for new pools 2022-06-29T16:07:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: osd_pool_default_pg_autoscale_mode = off 2022-06-29T16:07:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default size = 2 2022-06-29T16:07:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow primary affinity = true 2022-06-29T16:07:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow pg remap = true 2022-06-29T16:07:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on legacy crush tunables = false 2022-06-29T16:07:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on crush straw calc version zero = false 2022-06-29T16:07:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on no sortbitwise = false 2022-06-29T16:07:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on osd down out interval zero = false 2022-06-29T16:07:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on too few osds = false 2022-06-29T16:07:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-06-29T16:07:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_no_redundancy = false 2022-06-29T16:07:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-06-29T16:07:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: osd default data pool replay window = 5 2022-06-29T16:07:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: mon allow pool delete = true 2022-06-29T16:07:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: mon cluster log file level = debug 2022-06-29T16:07:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: debug asserts on shutdown = true 2022-06-29T16:07:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: mon health detail to clog = false 2022-06-29T16:07:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: mon host = "172.21.15.60,172.21.15.92,[v2:172.21.15.92:3301,v1:172.21.15.92:6790]" 2022-06-29T16:07:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: fsid = e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85 2022-06-29T16:07:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:35.984 INFO:teuthology.orchestra.run.smithi060.stdout:[osd] 2022-06-29T16:07:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: osd journal size = 100 2022-06-29T16:07:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub load threshold = 5.0 2022-06-29T16:07:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub max interval = 600 2022-06-29T16:07:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: osd recover clone overlap = true 2022-06-29T16:07:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: osd recovery max chunk = 1048576 2022-06-29T16:07:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug shutdown = true 2022-06-29T16:07:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug op order = true 2022-06-29T16:07:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug verify stray on activate = true 2022-06-29T16:07:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: osd open classes on start = true 2022-06-29T16:07:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug pg log writeout = true 2022-06-29T16:07:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: osd deep scrub update digest min age = 30 2022-06-29T16:07:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: osd map max advance = 10 2022-06-29T16:07:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: journal zero on create = true 2022-06-29T16:07:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: filestore ondisk finisher threads = 3 2022-06-29T16:07:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: filestore apply finisher threads = 3 2022-06-29T16:07:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: bdev debug aio = true 2022-06-29T16:07:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug misdirected ops = true 2022-06-29T16:07:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2022-06-29T16:07:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: debug osd = 20 2022-06-29T16:07:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: osd objectstore = filestore 2022-06-29T16:07:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: osd op complaint time = 180 2022-06-29T16:07:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: osd sloppy crc = True 2022-06-29T16:07:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:35.992 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr] 2022-06-29T16:07:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2022-06-29T16:07:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: debug mgr = 20 2022-06-29T16:07:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2022-06-29T16:07:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2022-06-29T16:07:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2022-06-29T16:07:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2022-06-29T16:07:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: mgr/telemetry/nag = false 2022-06-29T16:07:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:35.994 INFO:teuthology.orchestra.run.smithi060.stdout:[mon] 2022-06-29T16:07:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2022-06-29T16:07:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2022-06-29T16:07:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: debug paxos = 20 2022-06-29T16:07:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2022-06-29T16:07:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: mon data avail warn = 5 2022-06-29T16:07:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: mon mgr mkfs grace = 240 2022-06-29T16:07:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2022-06-29T16:07:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd reporter subtree level = osd 2022-06-29T16:07:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd prime pg temp = true 2022-06-29T16:07:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2022-06-29T16:07:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-06-29T16:07:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: auth mon ticket ttl = 660 # 11m 2022-06-29T16:07:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: auth service ticket ttl = 240 # 4m 2022-06-29T16:07:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: # don't complain about insecure global_id in the test suite 2022-06-29T16:07:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-06-29T16:07:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-06-29T16:07:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: mon op complaint time = 120 2022-06-29T16:07:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:35.999 INFO:teuthology.orchestra.run.smithi060.stdout:[client] 2022-06-29T16:07:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: rgw cache enabled = true 2022-06-29T16:07:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable ops log = true 2022-06-29T16:07:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable usage log = true 2022-06-29T16:07:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-06-29T16:07:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-06-29T16:07:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: client mount timeout = 600 2022-06-29T16:07:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: debug client = 20 2022-06-29T16:07:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2022-06-29T16:07:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: rados mon op timeout = 15m 2022-06-29T16:07:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: rados osd op timeout = 15m 2022-06-29T16:07:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: client debug getattr caps = true 2022-06-29T16:07:36.002 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.a] 2022-06-29T16:07:36.002 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.b] 2022-06-29T16:07:36.002 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.c] 2022-06-29T16:07:36.002 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.a] 2022-06-29T16:07:36.003 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.b] 2022-06-29T16:07:36.003 INFO:teuthology.orchestra.run.smithi060.stdout:[mds] 2022-06-29T16:07:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: debug mds = 20 2022-06-29T16:07:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2022-06-29T16:07:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: mds bal frag = True 2022-06-29T16:07:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: mds bal fragment size max = 10000 2022-06-29T16:07:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: mds bal merge size = 5 2022-06-29T16:07:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: mds bal split bits = 3 2022-06-29T16:07:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: mds bal split size = 100 2022-06-29T16:07:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: mds debug frag = True 2022-06-29T16:07:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: mds debug scatterstat = True 2022-06-29T16:07:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: mds op complaint time = 180 2022-06-29T16:07:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: mds verify scatter = True 2022-06-29T16:07:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: osd op complaint time = 180 2022-06-29T16:07:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: rados mon op timeout = 15m 2022-06-29T16:07:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: rados osd op timeout = 15m 2022-06-29T16:07:36.010 INFO:teuthology.orchestra.run.smithi197.stdout:[global] 2022-06-29T16:07:36.010 INFO:teuthology.orchestra.run.smithi197.stdout: chdir = "" 2022-06-29T16:07:36.011 INFO:teuthology.orchestra.run.smithi197.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-06-29T16:07:36.011 INFO:teuthology.orchestra.run.smithi197.stdout: auth supported = cephx 2022-06-29T16:07:36.011 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:36.011 INFO:teuthology.orchestra.run.smithi197.stdout: filestore xattr use omap = true 2022-06-29T16:07:36.012 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:36.012 INFO:teuthology.orchestra.run.smithi197.stdout: mon clock drift allowed = 1.000 2022-06-29T16:07:36.012 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:36.012 INFO:teuthology.orchestra.run.smithi197.stdout: osd crush chooseleaf type = 0 2022-06-29T16:07:36.013 INFO:teuthology.orchestra.run.smithi197.stdout: auth debug = true 2022-06-29T16:07:36.013 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:36.013 INFO:teuthology.orchestra.run.smithi197.stdout: ms die on old message = true 2022-06-29T16:07:36.013 INFO:teuthology.orchestra.run.smithi197.stdout: ms die on bug = true 2022-06-29T16:07:36.014 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:36.014 INFO:teuthology.orchestra.run.smithi197.stdout: mon max pg per osd = 10000 # >= luminous 2022-06-29T16:07:36.014 INFO:teuthology.orchestra.run.smithi197.stdout: mon pg warn max object skew = 0 2022-06-29T16:07:36.014 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:36.014 INFO:teuthology.orchestra.run.smithi197.stdout: # disable pg_autoscaler by default for new pools 2022-06-29T16:07:36.015 INFO:teuthology.orchestra.run.smithi197.stdout: osd_pool_default_pg_autoscale_mode = off 2022-06-29T16:07:36.015 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:36.015 INFO:teuthology.orchestra.run.smithi197.stdout: osd pool default size = 2 2022-06-29T16:07:36.015 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:36.016 INFO:teuthology.orchestra.run.smithi197.stdout: mon osd allow primary affinity = true 2022-06-29T16:07:36.016 INFO:teuthology.orchestra.run.smithi197.stdout: mon osd allow pg remap = true 2022-06-29T16:07:36.016 INFO:teuthology.orchestra.run.smithi197.stdout: mon warn on legacy crush tunables = false 2022-06-29T16:07:36.016 INFO:teuthology.orchestra.run.smithi197.stdout: mon warn on crush straw calc version zero = false 2022-06-29T16:07:36.016 INFO:teuthology.orchestra.run.smithi197.stdout: mon warn on no sortbitwise = false 2022-06-29T16:07:36.017 INFO:teuthology.orchestra.run.smithi197.stdout: mon warn on osd down out interval zero = false 2022-06-29T16:07:36.017 INFO:teuthology.orchestra.run.smithi197.stdout: mon warn on too few osds = false 2022-06-29T16:07:36.017 INFO:teuthology.orchestra.run.smithi197.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-06-29T16:07:36.017 INFO:teuthology.orchestra.run.smithi197.stdout: mon_warn_on_pool_no_redundancy = false 2022-06-29T16:07:36.018 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:36.018 INFO:teuthology.orchestra.run.smithi197.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-06-29T16:07:36.018 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:36.018 INFO:teuthology.orchestra.run.smithi197.stdout: osd default data pool replay window = 5 2022-06-29T16:07:36.019 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:36.019 INFO:teuthology.orchestra.run.smithi197.stdout: mon allow pool delete = true 2022-06-29T16:07:36.019 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:36.019 INFO:teuthology.orchestra.run.smithi197.stdout: mon cluster log file level = debug 2022-06-29T16:07:36.020 INFO:teuthology.orchestra.run.smithi197.stdout: debug asserts on shutdown = true 2022-06-29T16:07:36.020 INFO:teuthology.orchestra.run.smithi197.stdout: mon health detail to clog = false 2022-06-29T16:07:36.020 INFO:teuthology.orchestra.run.smithi197.stdout: mon host = "172.21.15.60,172.21.15.92,[v2:172.21.15.92:3301,v1:172.21.15.92:6790]" 2022-06-29T16:07:36.020 INFO:teuthology.orchestra.run.smithi197.stdout: fsid = e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85 2022-06-29T16:07:36.021 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:36.021 INFO:teuthology.orchestra.run.smithi197.stdout:[osd] 2022-06-29T16:07:36.021 INFO:teuthology.orchestra.run.smithi197.stdout: osd journal size = 100 2022-06-29T16:07:36.021 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:36.021 INFO:teuthology.orchestra.run.smithi197.stdout: osd scrub load threshold = 5.0 2022-06-29T16:07:36.022 INFO:teuthology.orchestra.run.smithi197.stdout: osd scrub max interval = 600 2022-06-29T16:07:36.022 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:36.022 INFO:teuthology.orchestra.run.smithi197.stdout: osd recover clone overlap = true 2022-06-29T16:07:36.022 INFO:teuthology.orchestra.run.smithi197.stdout: osd recovery max chunk = 1048576 2022-06-29T16:07:36.023 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:36.023 INFO:teuthology.orchestra.run.smithi197.stdout: osd debug shutdown = true 2022-06-29T16:07:36.023 INFO:teuthology.orchestra.run.smithi197.stdout: osd debug op order = true 2022-06-29T16:07:36.023 INFO:teuthology.orchestra.run.smithi197.stdout: osd debug verify stray on activate = true 2022-06-29T16:07:36.023 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:36.024 INFO:teuthology.orchestra.run.smithi197.stdout: osd open classes on start = true 2022-06-29T16:07:36.024 INFO:teuthology.orchestra.run.smithi197.stdout: osd debug pg log writeout = true 2022-06-29T16:07:36.024 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:36.024 INFO:teuthology.orchestra.run.smithi197.stdout: osd deep scrub update digest min age = 30 2022-06-29T16:07:36.025 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:36.025 INFO:teuthology.orchestra.run.smithi197.stdout: osd map max advance = 10 2022-06-29T16:07:36.025 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:36.025 INFO:teuthology.orchestra.run.smithi197.stdout: journal zero on create = true 2022-06-29T16:07:36.026 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:36.026 INFO:teuthology.orchestra.run.smithi197.stdout: filestore ondisk finisher threads = 3 2022-06-29T16:07:36.026 INFO:teuthology.orchestra.run.smithi197.stdout: filestore apply finisher threads = 3 2022-06-29T16:07:36.026 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:36.027 INFO:teuthology.orchestra.run.smithi197.stdout: bdev debug aio = true 2022-06-29T16:07:36.027 INFO:teuthology.orchestra.run.smithi197.stdout: osd debug misdirected ops = true 2022-06-29T16:07:36.027 INFO:teuthology.orchestra.run.smithi197.stdout: debug ms = 1 2022-06-29T16:07:36.027 INFO:teuthology.orchestra.run.smithi197.stdout: debug osd = 20 2022-06-29T16:07:36.028 INFO:teuthology.orchestra.run.smithi197.stdout: osd objectstore = filestore 2022-06-29T16:07:36.028 INFO:teuthology.orchestra.run.smithi197.stdout: osd op complaint time = 180 2022-06-29T16:07:36.028 INFO:teuthology.orchestra.run.smithi197.stdout: osd sloppy crc = True 2022-06-29T16:07:36.029 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:36.029 INFO:teuthology.orchestra.run.smithi197.stdout:[mgr] 2022-06-29T16:07:36.029 INFO:teuthology.orchestra.run.smithi197.stdout: debug ms = 1 2022-06-29T16:07:36.029 INFO:teuthology.orchestra.run.smithi197.stdout: debug mgr = 20 2022-06-29T16:07:36.029 INFO:teuthology.orchestra.run.smithi197.stdout: debug mon = 20 2022-06-29T16:07:36.030 INFO:teuthology.orchestra.run.smithi197.stdout: debug auth = 20 2022-06-29T16:07:36.030 INFO:teuthology.orchestra.run.smithi197.stdout: mon reweight min pgs per osd = 4 2022-06-29T16:07:36.031 INFO:teuthology.orchestra.run.smithi197.stdout: mon reweight min bytes per osd = 10 2022-06-29T16:07:36.031 INFO:teuthology.orchestra.run.smithi197.stdout: mgr/telemetry/nag = false 2022-06-29T16:07:36.031 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:36.031 INFO:teuthology.orchestra.run.smithi197.stdout:[mon] 2022-06-29T16:07:36.032 INFO:teuthology.orchestra.run.smithi197.stdout: debug ms = 1 2022-06-29T16:07:36.032 INFO:teuthology.orchestra.run.smithi197.stdout: debug mon = 20 2022-06-29T16:07:36.032 INFO:teuthology.orchestra.run.smithi197.stdout: debug paxos = 20 2022-06-29T16:07:36.032 INFO:teuthology.orchestra.run.smithi197.stdout: debug auth = 20 2022-06-29T16:07:36.033 INFO:teuthology.orchestra.run.smithi197.stdout: mon data avail warn = 5 2022-06-29T16:07:36.033 INFO:teuthology.orchestra.run.smithi197.stdout: mon mgr mkfs grace = 240 2022-06-29T16:07:36.034 INFO:teuthology.orchestra.run.smithi197.stdout: mon reweight min pgs per osd = 4 2022-06-29T16:07:36.034 INFO:teuthology.orchestra.run.smithi197.stdout: mon osd reporter subtree level = osd 2022-06-29T16:07:36.034 INFO:teuthology.orchestra.run.smithi197.stdout: mon osd prime pg temp = true 2022-06-29T16:07:36.035 INFO:teuthology.orchestra.run.smithi197.stdout: mon reweight min bytes per osd = 10 2022-06-29T16:07:36.035 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:36.035 INFO:teuthology.orchestra.run.smithi197.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-06-29T16:07:36.036 INFO:teuthology.orchestra.run.smithi197.stdout: auth mon ticket ttl = 660 # 11m 2022-06-29T16:07:36.036 INFO:teuthology.orchestra.run.smithi197.stdout: auth service ticket ttl = 240 # 4m 2022-06-29T16:07:36.037 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:36.037 INFO:teuthology.orchestra.run.smithi197.stdout: # don't complain about insecure global_id in the test suite 2022-06-29T16:07:36.037 INFO:teuthology.orchestra.run.smithi197.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-06-29T16:07:36.038 INFO:teuthology.orchestra.run.smithi197.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-06-29T16:07:36.038 INFO:teuthology.orchestra.run.smithi197.stdout: mon op complaint time = 120 2022-06-29T16:07:36.038 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:36.038 INFO:teuthology.orchestra.run.smithi197.stdout:[client] 2022-06-29T16:07:36.039 INFO:teuthology.orchestra.run.smithi197.stdout: rgw cache enabled = true 2022-06-29T16:07:36.039 INFO:teuthology.orchestra.run.smithi197.stdout: rgw enable ops log = true 2022-06-29T16:07:36.039 INFO:teuthology.orchestra.run.smithi197.stdout: rgw enable usage log = true 2022-06-29T16:07:36.040 INFO:teuthology.orchestra.run.smithi197.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-06-29T16:07:36.040 INFO:teuthology.orchestra.run.smithi197.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-06-29T16:07:36.040 INFO:teuthology.orchestra.run.smithi197.stdout: client mount timeout = 600 2022-06-29T16:07:36.040 INFO:teuthology.orchestra.run.smithi197.stdout: debug client = 20 2022-06-29T16:07:36.041 INFO:teuthology.orchestra.run.smithi197.stdout: debug ms = 1 2022-06-29T16:07:36.041 INFO:teuthology.orchestra.run.smithi197.stdout: rados mon op timeout = 15m 2022-06-29T16:07:36.041 INFO:teuthology.orchestra.run.smithi197.stdout: rados osd op timeout = 15m 2022-06-29T16:07:36.041 INFO:teuthology.orchestra.run.smithi197.stdout: client debug getattr caps = true 2022-06-29T16:07:36.042 INFO:teuthology.orchestra.run.smithi197.stdout:[mon.a] 2022-06-29T16:07:36.042 INFO:teuthology.orchestra.run.smithi197.stdout:[mon.b] 2022-06-29T16:07:36.042 INFO:teuthology.orchestra.run.smithi197.stdout:[mon.c] 2022-06-29T16:07:36.042 INFO:teuthology.orchestra.run.smithi197.stdout:[mds.a] 2022-06-29T16:07:36.043 INFO:teuthology.orchestra.run.smithi197.stdout:[mds.b] 2022-06-29T16:07:36.043 INFO:teuthology.orchestra.run.smithi197.stdout:[mds] 2022-06-29T16:07:36.043 INFO:teuthology.orchestra.run.smithi197.stdout: debug mds = 20 2022-06-29T16:07:36.043 INFO:teuthology.orchestra.run.smithi197.stdout: debug ms = 1 2022-06-29T16:07:36.044 INFO:teuthology.orchestra.run.smithi197.stdout: mds bal frag = True 2022-06-29T16:07:36.044 INFO:teuthology.orchestra.run.smithi197.stdout: mds bal fragment size max = 10000 2022-06-29T16:07:36.044 INFO:teuthology.orchestra.run.smithi197.stdout: mds bal merge size = 5 2022-06-29T16:07:36.044 INFO:teuthology.orchestra.run.smithi197.stdout: mds bal split bits = 3 2022-06-29T16:07:36.045 INFO:teuthology.orchestra.run.smithi197.stdout: mds bal split size = 100 2022-06-29T16:07:36.045 INFO:teuthology.orchestra.run.smithi197.stdout: mds debug frag = True 2022-06-29T16:07:36.045 INFO:teuthology.orchestra.run.smithi197.stdout: mds debug scatterstat = True 2022-06-29T16:07:36.045 INFO:teuthology.orchestra.run.smithi197.stdout: mds op complaint time = 180 2022-06-29T16:07:36.046 INFO:teuthology.orchestra.run.smithi197.stdout: mds verify scatter = True 2022-06-29T16:07:36.046 INFO:teuthology.orchestra.run.smithi197.stdout: osd op complaint time = 180 2022-06-29T16:07:36.046 INFO:teuthology.orchestra.run.smithi197.stdout: rados mon op timeout = 15m 2022-06-29T16:07:36.046 INFO:teuthology.orchestra.run.smithi197.stdout: rados osd op timeout = 15m 2022-06-29T16:07:36.047 INFO:teuthology.orchestra.run.smithi156.stdout:[global] 2022-06-29T16:07:36.047 INFO:teuthology.orchestra.run.smithi156.stdout: chdir = "" 2022-06-29T16:07:36.047 INFO:teuthology.orchestra.run.smithi156.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-06-29T16:07:36.048 INFO:teuthology.orchestra.run.smithi156.stdout: auth supported = cephx 2022-06-29T16:07:36.048 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:36.048 INFO:teuthology.orchestra.run.smithi156.stdout: filestore xattr use omap = true 2022-06-29T16:07:36.048 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:36.049 INFO:teuthology.orchestra.run.smithi156.stdout: mon clock drift allowed = 1.000 2022-06-29T16:07:36.049 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:36.049 INFO:teuthology.orchestra.run.smithi156.stdout: osd crush chooseleaf type = 0 2022-06-29T16:07:36.049 INFO:teuthology.orchestra.run.smithi156.stdout: auth debug = true 2022-06-29T16:07:36.049 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:36.050 INFO:teuthology.orchestra.run.smithi156.stdout: ms die on old message = true 2022-06-29T16:07:36.050 INFO:teuthology.orchestra.run.smithi156.stdout: ms die on bug = true 2022-06-29T16:07:36.050 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:36.050 INFO:teuthology.orchestra.run.smithi156.stdout: mon max pg per osd = 10000 # >= luminous 2022-06-29T16:07:36.051 INFO:teuthology.orchestra.run.smithi156.stdout: mon pg warn max object skew = 0 2022-06-29T16:07:36.051 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:36.051 INFO:teuthology.orchestra.run.smithi156.stdout: # disable pg_autoscaler by default for new pools 2022-06-29T16:07:36.051 INFO:teuthology.orchestra.run.smithi156.stdout: osd_pool_default_pg_autoscale_mode = off 2022-06-29T16:07:36.052 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:36.052 INFO:teuthology.orchestra.run.smithi156.stdout: osd pool default size = 2 2022-06-29T16:07:36.052 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:36.052 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd allow primary affinity = true 2022-06-29T16:07:36.053 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd allow pg remap = true 2022-06-29T16:07:36.053 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on legacy crush tunables = false 2022-06-29T16:07:36.053 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on crush straw calc version zero = false 2022-06-29T16:07:36.053 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on no sortbitwise = false 2022-06-29T16:07:36.053 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on osd down out interval zero = false 2022-06-29T16:07:36.054 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on too few osds = false 2022-06-29T16:07:36.054 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-06-29T16:07:36.054 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_pool_no_redundancy = false 2022-06-29T16:07:36.054 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:36.055 INFO:teuthology.orchestra.run.smithi156.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-06-29T16:07:36.055 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:36.055 INFO:teuthology.orchestra.run.smithi156.stdout: osd default data pool replay window = 5 2022-06-29T16:07:36.055 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:36.056 INFO:teuthology.orchestra.run.smithi156.stdout: mon allow pool delete = true 2022-06-29T16:07:36.056 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:36.056 INFO:teuthology.orchestra.run.smithi156.stdout: mon cluster log file level = debug 2022-06-29T16:07:36.056 INFO:teuthology.orchestra.run.smithi156.stdout: debug asserts on shutdown = true 2022-06-29T16:07:36.057 INFO:teuthology.orchestra.run.smithi156.stdout: mon health detail to clog = false 2022-06-29T16:07:36.057 INFO:teuthology.orchestra.run.smithi156.stdout: mon host = "172.21.15.60,172.21.15.92,[v2:172.21.15.92:3301,v1:172.21.15.92:6790]" 2022-06-29T16:07:36.057 INFO:teuthology.orchestra.run.smithi156.stdout: fsid = e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85 2022-06-29T16:07:36.057 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:36.058 INFO:teuthology.orchestra.run.smithi156.stdout:[osd] 2022-06-29T16:07:36.058 INFO:teuthology.orchestra.run.smithi156.stdout: osd journal size = 100 2022-06-29T16:07:36.058 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:36.058 INFO:teuthology.orchestra.run.smithi156.stdout: osd scrub load threshold = 5.0 2022-06-29T16:07:36.059 INFO:teuthology.orchestra.run.smithi156.stdout: osd scrub max interval = 600 2022-06-29T16:07:36.059 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:36.059 INFO:teuthology.orchestra.run.smithi156.stdout: osd recover clone overlap = true 2022-06-29T16:07:36.059 INFO:teuthology.orchestra.run.smithi156.stdout: osd recovery max chunk = 1048576 2022-06-29T16:07:36.060 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:36.060 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug shutdown = true 2022-06-29T16:07:36.060 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug op order = true 2022-06-29T16:07:36.060 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug verify stray on activate = true 2022-06-29T16:07:36.060 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:36.061 INFO:teuthology.orchestra.run.smithi156.stdout: osd open classes on start = true 2022-06-29T16:07:36.061 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug pg log writeout = true 2022-06-29T16:07:36.061 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:36.061 INFO:teuthology.orchestra.run.smithi156.stdout: osd deep scrub update digest min age = 30 2022-06-29T16:07:36.062 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:36.062 INFO:teuthology.orchestra.run.smithi156.stdout: osd map max advance = 10 2022-06-29T16:07:36.062 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:36.062 INFO:teuthology.orchestra.run.smithi156.stdout: journal zero on create = true 2022-06-29T16:07:36.063 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:36.063 INFO:teuthology.orchestra.run.smithi156.stdout: filestore ondisk finisher threads = 3 2022-06-29T16:07:36.063 INFO:teuthology.orchestra.run.smithi156.stdout: filestore apply finisher threads = 3 2022-06-29T16:07:36.063 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:36.063 INFO:teuthology.orchestra.run.smithi156.stdout: bdev debug aio = true 2022-06-29T16:07:36.064 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug misdirected ops = true 2022-06-29T16:07:36.064 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2022-06-29T16:07:36.064 INFO:teuthology.orchestra.run.smithi156.stdout: debug osd = 20 2022-06-29T16:07:36.064 INFO:teuthology.orchestra.run.smithi156.stdout: osd objectstore = filestore 2022-06-29T16:07:36.065 INFO:teuthology.orchestra.run.smithi156.stdout: osd op complaint time = 180 2022-06-29T16:07:36.065 INFO:teuthology.orchestra.run.smithi156.stdout: osd sloppy crc = True 2022-06-29T16:07:36.065 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:36.065 INFO:teuthology.orchestra.run.smithi156.stdout:[mgr] 2022-06-29T16:07:36.066 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2022-06-29T16:07:36.066 INFO:teuthology.orchestra.run.smithi156.stdout: debug mgr = 20 2022-06-29T16:07:36.066 INFO:teuthology.orchestra.run.smithi156.stdout: debug mon = 20 2022-06-29T16:07:36.066 INFO:teuthology.orchestra.run.smithi156.stdout: debug auth = 20 2022-06-29T16:07:36.067 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min pgs per osd = 4 2022-06-29T16:07:36.067 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min bytes per osd = 10 2022-06-29T16:07:36.067 INFO:teuthology.orchestra.run.smithi156.stdout: mgr/telemetry/nag = false 2022-06-29T16:07:36.067 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:36.068 INFO:teuthology.orchestra.run.smithi156.stdout:[mon] 2022-06-29T16:07:36.068 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2022-06-29T16:07:36.068 INFO:teuthology.orchestra.run.smithi156.stdout: debug mon = 20 2022-06-29T16:07:36.069 INFO:teuthology.orchestra.run.smithi156.stdout: debug paxos = 20 2022-06-29T16:07:36.069 INFO:teuthology.orchestra.run.smithi156.stdout: debug auth = 20 2022-06-29T16:07:36.069 INFO:teuthology.orchestra.run.smithi156.stdout: mon data avail warn = 5 2022-06-29T16:07:36.069 INFO:teuthology.orchestra.run.smithi156.stdout: mon mgr mkfs grace = 240 2022-06-29T16:07:36.070 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min pgs per osd = 4 2022-06-29T16:07:36.070 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd reporter subtree level = osd 2022-06-29T16:07:36.070 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd prime pg temp = true 2022-06-29T16:07:36.071 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min bytes per osd = 10 2022-06-29T16:07:36.071 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:36.071 INFO:teuthology.orchestra.run.smithi156.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-06-29T16:07:36.071 INFO:teuthology.orchestra.run.smithi156.stdout: auth mon ticket ttl = 660 # 11m 2022-06-29T16:07:36.072 INFO:teuthology.orchestra.run.smithi156.stdout: auth service ticket ttl = 240 # 4m 2022-06-29T16:07:36.072 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:36.072 INFO:teuthology.orchestra.run.smithi156.stdout: # don't complain about insecure global_id in the test suite 2022-06-29T16:07:36.072 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-06-29T16:07:36.073 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-06-29T16:07:36.073 INFO:teuthology.orchestra.run.smithi156.stdout: mon op complaint time = 120 2022-06-29T16:07:36.073 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:36.073 INFO:teuthology.orchestra.run.smithi156.stdout:[client] 2022-06-29T16:07:36.074 INFO:teuthology.orchestra.run.smithi156.stdout: rgw cache enabled = true 2022-06-29T16:07:36.074 INFO:teuthology.orchestra.run.smithi156.stdout: rgw enable ops log = true 2022-06-29T16:07:36.074 INFO:teuthology.orchestra.run.smithi156.stdout: rgw enable usage log = true 2022-06-29T16:07:36.074 INFO:teuthology.orchestra.run.smithi156.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-06-29T16:07:36.075 INFO:teuthology.orchestra.run.smithi156.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-06-29T16:07:36.075 INFO:teuthology.orchestra.run.smithi156.stdout: client mount timeout = 600 2022-06-29T16:07:36.075 INFO:teuthology.orchestra.run.smithi156.stdout: debug client = 20 2022-06-29T16:07:36.075 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2022-06-29T16:07:36.075 INFO:teuthology.orchestra.run.smithi156.stdout: rados mon op timeout = 15m 2022-06-29T16:07:36.076 INFO:teuthology.orchestra.run.smithi156.stdout: rados osd op timeout = 15m 2022-06-29T16:07:36.076 INFO:teuthology.orchestra.run.smithi156.stdout: client debug getattr caps = true 2022-06-29T16:07:36.076 INFO:teuthology.orchestra.run.smithi156.stdout:[mon.a] 2022-06-29T16:07:36.076 INFO:teuthology.orchestra.run.smithi156.stdout:[mon.b] 2022-06-29T16:07:36.077 INFO:teuthology.orchestra.run.smithi156.stdout:[mon.c] 2022-06-29T16:07:36.077 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.a] 2022-06-29T16:07:36.077 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.b] 2022-06-29T16:07:36.077 INFO:teuthology.orchestra.run.smithi156.stdout:[mds] 2022-06-29T16:07:36.078 INFO:teuthology.orchestra.run.smithi156.stdout: debug mds = 20 2022-06-29T16:07:36.078 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2022-06-29T16:07:36.078 INFO:teuthology.orchestra.run.smithi156.stdout: mds bal frag = True 2022-06-29T16:07:36.078 INFO:teuthology.orchestra.run.smithi156.stdout: mds bal fragment size max = 10000 2022-06-29T16:07:36.079 INFO:teuthology.orchestra.run.smithi156.stdout: mds bal merge size = 5 2022-06-29T16:07:36.079 INFO:teuthology.orchestra.run.smithi156.stdout: mds bal split bits = 3 2022-06-29T16:07:36.079 INFO:teuthology.orchestra.run.smithi156.stdout: mds bal split size = 100 2022-06-29T16:07:36.079 INFO:teuthology.orchestra.run.smithi156.stdout: mds debug frag = True 2022-06-29T16:07:36.080 INFO:teuthology.orchestra.run.smithi156.stdout: mds debug scatterstat = True 2022-06-29T16:07:36.080 INFO:teuthology.orchestra.run.smithi156.stdout: mds op complaint time = 180 2022-06-29T16:07:36.080 INFO:teuthology.orchestra.run.smithi156.stdout: mds verify scatter = True 2022-06-29T16:07:36.080 INFO:teuthology.orchestra.run.smithi156.stdout: osd op complaint time = 180 2022-06-29T16:07:36.080 INFO:teuthology.orchestra.run.smithi156.stdout: rados mon op timeout = 15m 2022-06-29T16:07:36.081 INFO:teuthology.orchestra.run.smithi156.stdout: rados osd op timeout = 15m 2022-06-29T16:07:36.081 INFO:teuthology.orchestra.run.smithi121.stdout:[global] 2022-06-29T16:07:36.081 INFO:teuthology.orchestra.run.smithi121.stdout: chdir = "" 2022-06-29T16:07:36.081 INFO:teuthology.orchestra.run.smithi121.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-06-29T16:07:36.082 INFO:teuthology.orchestra.run.smithi121.stdout: auth supported = cephx 2022-06-29T16:07:36.082 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:36.082 INFO:teuthology.orchestra.run.smithi121.stdout: filestore xattr use omap = true 2022-06-29T16:07:36.082 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:36.083 INFO:teuthology.orchestra.run.smithi121.stdout: mon clock drift allowed = 1.000 2022-06-29T16:07:36.083 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:36.083 INFO:teuthology.orchestra.run.smithi121.stdout: osd crush chooseleaf type = 0 2022-06-29T16:07:36.083 INFO:teuthology.orchestra.run.smithi121.stdout: auth debug = true 2022-06-29T16:07:36.084 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:36.084 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on old message = true 2022-06-29T16:07:36.084 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on bug = true 2022-06-29T16:07:36.084 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:36.084 INFO:teuthology.orchestra.run.smithi121.stdout: mon max pg per osd = 10000 # >= luminous 2022-06-29T16:07:36.085 INFO:teuthology.orchestra.run.smithi121.stdout: mon pg warn max object skew = 0 2022-06-29T16:07:36.085 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:36.085 INFO:teuthology.orchestra.run.smithi121.stdout: # disable pg_autoscaler by default for new pools 2022-06-29T16:07:36.085 INFO:teuthology.orchestra.run.smithi121.stdout: osd_pool_default_pg_autoscale_mode = off 2022-06-29T16:07:36.086 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:36.086 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default size = 2 2022-06-29T16:07:36.086 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:36.086 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow primary affinity = true 2022-06-29T16:07:36.087 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow pg remap = true 2022-06-29T16:07:36.087 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on legacy crush tunables = false 2022-06-29T16:07:36.087 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on crush straw calc version zero = false 2022-06-29T16:07:36.087 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on no sortbitwise = false 2022-06-29T16:07:36.087 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on osd down out interval zero = false 2022-06-29T16:07:36.088 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on too few osds = false 2022-06-29T16:07:36.088 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-06-29T16:07:36.088 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_no_redundancy = false 2022-06-29T16:07:36.088 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:36.089 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-06-29T16:07:36.089 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:36.089 INFO:teuthology.orchestra.run.smithi121.stdout: osd default data pool replay window = 5 2022-06-29T16:07:36.089 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:36.089 INFO:teuthology.orchestra.run.smithi121.stdout: mon allow pool delete = true 2022-06-29T16:07:36.090 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:36.090 INFO:teuthology.orchestra.run.smithi121.stdout: mon cluster log file level = debug 2022-06-29T16:07:36.090 INFO:teuthology.orchestra.run.smithi121.stdout: debug asserts on shutdown = true 2022-06-29T16:07:36.090 INFO:teuthology.orchestra.run.smithi121.stdout: mon health detail to clog = false 2022-06-29T16:07:36.091 INFO:teuthology.orchestra.run.smithi121.stdout: mon host = "172.21.15.60,172.21.15.92,[v2:172.21.15.92:3301,v1:172.21.15.92:6790]" 2022-06-29T16:07:36.091 INFO:teuthology.orchestra.run.smithi121.stdout: fsid = e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85 2022-06-29T16:07:36.091 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:36.091 INFO:teuthology.orchestra.run.smithi121.stdout:[osd] 2022-06-29T16:07:36.092 INFO:teuthology.orchestra.run.smithi121.stdout: osd journal size = 100 2022-06-29T16:07:36.092 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:36.093 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub load threshold = 5.0 2022-06-29T16:07:36.093 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub max interval = 600 2022-06-29T16:07:36.093 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:36.094 INFO:teuthology.orchestra.run.smithi121.stdout: osd recover clone overlap = true 2022-06-29T16:07:36.094 INFO:teuthology.orchestra.run.smithi121.stdout: osd recovery max chunk = 1048576 2022-06-29T16:07:36.094 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:36.095 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug shutdown = true 2022-06-29T16:07:36.095 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug op order = true 2022-06-29T16:07:36.095 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify stray on activate = true 2022-06-29T16:07:36.096 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:36.096 INFO:teuthology.orchestra.run.smithi121.stdout: osd open classes on start = true 2022-06-29T16:07:36.097 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug pg log writeout = true 2022-06-29T16:07:36.097 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:36.097 INFO:teuthology.orchestra.run.smithi121.stdout: osd deep scrub update digest min age = 30 2022-06-29T16:07:36.098 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:36.098 INFO:teuthology.orchestra.run.smithi121.stdout: osd map max advance = 10 2022-06-29T16:07:36.098 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:36.099 INFO:teuthology.orchestra.run.smithi121.stdout: journal zero on create = true 2022-06-29T16:07:36.099 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:36.099 INFO:teuthology.orchestra.run.smithi121.stdout: filestore ondisk finisher threads = 3 2022-06-29T16:07:36.100 INFO:teuthology.orchestra.run.smithi121.stdout: filestore apply finisher threads = 3 2022-06-29T16:07:36.100 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:36.100 INFO:teuthology.orchestra.run.smithi121.stdout: bdev debug aio = true 2022-06-29T16:07:36.101 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug misdirected ops = true 2022-06-29T16:07:36.101 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2022-06-29T16:07:36.101 INFO:teuthology.orchestra.run.smithi121.stdout: debug osd = 20 2022-06-29T16:07:36.102 INFO:teuthology.orchestra.run.smithi121.stdout: osd objectstore = filestore 2022-06-29T16:07:36.102 INFO:teuthology.orchestra.run.smithi121.stdout: osd op complaint time = 180 2022-06-29T16:07:36.102 INFO:teuthology.orchestra.run.smithi121.stdout: osd sloppy crc = True 2022-06-29T16:07:36.103 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:36.103 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr] 2022-06-29T16:07:36.103 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2022-06-29T16:07:36.104 INFO:teuthology.orchestra.run.smithi121.stdout: debug mgr = 20 2022-06-29T16:07:36.104 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2022-06-29T16:07:36.104 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2022-06-29T16:07:36.104 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2022-06-29T16:07:36.105 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2022-06-29T16:07:36.105 INFO:teuthology.orchestra.run.smithi121.stdout: mgr/telemetry/nag = false 2022-06-29T16:07:36.105 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:36.105 INFO:teuthology.orchestra.run.smithi121.stdout:[mon] 2022-06-29T16:07:36.105 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2022-06-29T16:07:36.106 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2022-06-29T16:07:36.106 INFO:teuthology.orchestra.run.smithi121.stdout: debug paxos = 20 2022-06-29T16:07:36.106 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2022-06-29T16:07:36.106 INFO:teuthology.orchestra.run.smithi121.stdout: mon data avail warn = 5 2022-06-29T16:07:36.107 INFO:teuthology.orchestra.run.smithi121.stdout: mon mgr mkfs grace = 240 2022-06-29T16:07:36.107 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2022-06-29T16:07:36.107 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd reporter subtree level = osd 2022-06-29T16:07:36.107 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd prime pg temp = true 2022-06-29T16:07:36.107 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2022-06-29T16:07:36.108 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:36.108 INFO:teuthology.orchestra.run.smithi121.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-06-29T16:07:36.108 INFO:teuthology.orchestra.run.smithi121.stdout: auth mon ticket ttl = 660 # 11m 2022-06-29T16:07:36.108 INFO:teuthology.orchestra.run.smithi121.stdout: auth service ticket ttl = 240 # 4m 2022-06-29T16:07:36.109 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:36.109 INFO:teuthology.orchestra.run.smithi121.stdout: # don't complain about insecure global_id in the test suite 2022-06-29T16:07:36.109 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-06-29T16:07:36.109 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-06-29T16:07:36.109 INFO:teuthology.orchestra.run.smithi121.stdout: mon op complaint time = 120 2022-06-29T16:07:36.110 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:36.110 INFO:teuthology.orchestra.run.smithi121.stdout:[client] 2022-06-29T16:07:36.110 INFO:teuthology.orchestra.run.smithi121.stdout: rgw cache enabled = true 2022-06-29T16:07:36.110 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable ops log = true 2022-06-29T16:07:36.111 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable usage log = true 2022-06-29T16:07:36.111 INFO:teuthology.orchestra.run.smithi121.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-06-29T16:07:36.111 INFO:teuthology.orchestra.run.smithi121.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-06-29T16:07:36.111 INFO:teuthology.orchestra.run.smithi121.stdout: client mount timeout = 600 2022-06-29T16:07:36.111 INFO:teuthology.orchestra.run.smithi121.stdout: debug client = 20 2022-06-29T16:07:36.112 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2022-06-29T16:07:36.112 INFO:teuthology.orchestra.run.smithi121.stdout: rados mon op timeout = 15m 2022-06-29T16:07:36.112 INFO:teuthology.orchestra.run.smithi121.stdout: rados osd op timeout = 15m 2022-06-29T16:07:36.112 INFO:teuthology.orchestra.run.smithi121.stdout: client debug getattr caps = true 2022-06-29T16:07:36.113 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.a] 2022-06-29T16:07:36.113 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.b] 2022-06-29T16:07:36.113 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.c] 2022-06-29T16:07:36.113 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.a] 2022-06-29T16:07:36.113 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.b] 2022-06-29T16:07:36.114 INFO:teuthology.orchestra.run.smithi121.stdout:[mds] 2022-06-29T16:07:36.114 INFO:teuthology.orchestra.run.smithi121.stdout: debug mds = 20 2022-06-29T16:07:36.114 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2022-06-29T16:07:36.114 INFO:teuthology.orchestra.run.smithi121.stdout: mds bal frag = True 2022-06-29T16:07:36.115 INFO:teuthology.orchestra.run.smithi121.stdout: mds bal fragment size max = 10000 2022-06-29T16:07:36.115 INFO:teuthology.orchestra.run.smithi121.stdout: mds bal merge size = 5 2022-06-29T16:07:36.115 INFO:teuthology.orchestra.run.smithi121.stdout: mds bal split bits = 3 2022-06-29T16:07:36.115 INFO:teuthology.orchestra.run.smithi121.stdout: mds bal split size = 100 2022-06-29T16:07:36.115 INFO:teuthology.orchestra.run.smithi121.stdout: mds debug frag = True 2022-06-29T16:07:36.116 INFO:teuthology.orchestra.run.smithi121.stdout: mds debug scatterstat = True 2022-06-29T16:07:36.116 INFO:teuthology.orchestra.run.smithi121.stdout: mds op complaint time = 180 2022-06-29T16:07:36.116 INFO:teuthology.orchestra.run.smithi121.stdout: mds verify scatter = True 2022-06-29T16:07:36.116 INFO:teuthology.orchestra.run.smithi121.stdout: osd op complaint time = 180 2022-06-29T16:07:36.117 INFO:teuthology.orchestra.run.smithi121.stdout: rados mon op timeout = 15m 2022-06-29T16:07:36.117 INFO:teuthology.orchestra.run.smithi121.stdout: rados osd op timeout = 15m 2022-06-29T16:07:36.117 INFO:teuthology.orchestra.run.smithi092.stdout:[global] 2022-06-29T16:07:36.117 INFO:teuthology.orchestra.run.smithi092.stdout: chdir = "" 2022-06-29T16:07:36.118 INFO:teuthology.orchestra.run.smithi092.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-06-29T16:07:36.118 INFO:teuthology.orchestra.run.smithi092.stdout: auth supported = cephx 2022-06-29T16:07:36.118 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:36.118 INFO:teuthology.orchestra.run.smithi092.stdout: filestore xattr use omap = true 2022-06-29T16:07:36.119 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:36.119 INFO:teuthology.orchestra.run.smithi092.stdout: mon clock drift allowed = 1.000 2022-06-29T16:07:36.119 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:36.119 INFO:teuthology.orchestra.run.smithi092.stdout: osd crush chooseleaf type = 0 2022-06-29T16:07:36.120 INFO:teuthology.orchestra.run.smithi092.stdout: auth debug = true 2022-06-29T16:07:36.120 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:36.120 INFO:teuthology.orchestra.run.smithi092.stdout: ms die on old message = true 2022-06-29T16:07:36.120 INFO:teuthology.orchestra.run.smithi092.stdout: ms die on bug = true 2022-06-29T16:07:36.121 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:36.121 INFO:teuthology.orchestra.run.smithi092.stdout: mon max pg per osd = 10000 # >= luminous 2022-06-29T16:07:36.121 INFO:teuthology.orchestra.run.smithi092.stdout: mon pg warn max object skew = 0 2022-06-29T16:07:36.121 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:36.122 INFO:teuthology.orchestra.run.smithi092.stdout: # disable pg_autoscaler by default for new pools 2022-06-29T16:07:36.122 INFO:teuthology.orchestra.run.smithi092.stdout: osd_pool_default_pg_autoscale_mode = off 2022-06-29T16:07:36.122 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:36.122 INFO:teuthology.orchestra.run.smithi092.stdout: osd pool default size = 2 2022-06-29T16:07:36.123 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:36.123 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd allow primary affinity = true 2022-06-29T16:07:36.123 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd allow pg remap = true 2022-06-29T16:07:36.123 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on legacy crush tunables = false 2022-06-29T16:07:36.123 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on crush straw calc version zero = false 2022-06-29T16:07:36.124 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on no sortbitwise = false 2022-06-29T16:07:36.124 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on osd down out interval zero = false 2022-06-29T16:07:36.124 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on too few osds = false 2022-06-29T16:07:36.124 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-06-29T16:07:36.125 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_pool_no_redundancy = false 2022-06-29T16:07:36.125 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:36.125 INFO:teuthology.orchestra.run.smithi092.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-06-29T16:07:36.125 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:36.125 INFO:teuthology.orchestra.run.smithi092.stdout: osd default data pool replay window = 5 2022-06-29T16:07:36.126 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:36.126 INFO:teuthology.orchestra.run.smithi092.stdout: mon allow pool delete = true 2022-06-29T16:07:36.126 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:36.126 INFO:teuthology.orchestra.run.smithi092.stdout: mon cluster log file level = debug 2022-06-29T16:07:36.127 INFO:teuthology.orchestra.run.smithi092.stdout: debug asserts on shutdown = true 2022-06-29T16:07:36.127 INFO:teuthology.orchestra.run.smithi092.stdout: mon health detail to clog = false 2022-06-29T16:07:36.127 INFO:teuthology.orchestra.run.smithi092.stdout: mon host = "172.21.15.60,172.21.15.92,[v2:172.21.15.92:3301,v1:172.21.15.92:6790]" 2022-06-29T16:07:36.127 INFO:teuthology.orchestra.run.smithi092.stdout: fsid = e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85 2022-06-29T16:07:36.128 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:36.128 INFO:teuthology.orchestra.run.smithi092.stdout:[osd] 2022-06-29T16:07:36.128 INFO:teuthology.orchestra.run.smithi092.stdout: osd journal size = 100 2022-06-29T16:07:36.128 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:36.128 INFO:teuthology.orchestra.run.smithi092.stdout: osd scrub load threshold = 5.0 2022-06-29T16:07:36.129 INFO:teuthology.orchestra.run.smithi092.stdout: osd scrub max interval = 600 2022-06-29T16:07:36.129 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:36.129 INFO:teuthology.orchestra.run.smithi092.stdout: osd recover clone overlap = true 2022-06-29T16:07:36.129 INFO:teuthology.orchestra.run.smithi092.stdout: osd recovery max chunk = 1048576 2022-06-29T16:07:36.129 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:36.130 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug shutdown = true 2022-06-29T16:07:36.130 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug op order = true 2022-06-29T16:07:36.130 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug verify stray on activate = true 2022-06-29T16:07:36.130 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:36.131 INFO:teuthology.orchestra.run.smithi092.stdout: osd open classes on start = true 2022-06-29T16:07:36.131 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug pg log writeout = true 2022-06-29T16:07:36.131 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:36.131 INFO:teuthology.orchestra.run.smithi092.stdout: osd deep scrub update digest min age = 30 2022-06-29T16:07:36.132 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:36.132 INFO:teuthology.orchestra.run.smithi092.stdout: osd map max advance = 10 2022-06-29T16:07:36.132 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:36.132 INFO:teuthology.orchestra.run.smithi092.stdout: journal zero on create = true 2022-06-29T16:07:36.133 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:36.133 INFO:teuthology.orchestra.run.smithi092.stdout: filestore ondisk finisher threads = 3 2022-06-29T16:07:36.133 INFO:teuthology.orchestra.run.smithi092.stdout: filestore apply finisher threads = 3 2022-06-29T16:07:36.133 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:36.133 INFO:teuthology.orchestra.run.smithi092.stdout: bdev debug aio = true 2022-06-29T16:07:36.134 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug misdirected ops = true 2022-06-29T16:07:36.134 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2022-06-29T16:07:36.134 INFO:teuthology.orchestra.run.smithi092.stdout: debug osd = 20 2022-06-29T16:07:36.134 INFO:teuthology.orchestra.run.smithi092.stdout: osd objectstore = filestore 2022-06-29T16:07:36.135 INFO:teuthology.orchestra.run.smithi092.stdout: osd op complaint time = 180 2022-06-29T16:07:36.135 INFO:teuthology.orchestra.run.smithi092.stdout: osd sloppy crc = True 2022-06-29T16:07:36.135 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:36.135 INFO:teuthology.orchestra.run.smithi092.stdout:[mgr] 2022-06-29T16:07:36.135 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2022-06-29T16:07:36.136 INFO:teuthology.orchestra.run.smithi092.stdout: debug mgr = 20 2022-06-29T16:07:36.136 INFO:teuthology.orchestra.run.smithi092.stdout: debug mon = 20 2022-06-29T16:07:36.136 INFO:teuthology.orchestra.run.smithi092.stdout: debug auth = 20 2022-06-29T16:07:36.136 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min pgs per osd = 4 2022-06-29T16:07:36.137 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min bytes per osd = 10 2022-06-29T16:07:36.137 INFO:teuthology.orchestra.run.smithi092.stdout: mgr/telemetry/nag = false 2022-06-29T16:07:36.137 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:36.137 INFO:teuthology.orchestra.run.smithi092.stdout:[mon] 2022-06-29T16:07:36.137 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2022-06-29T16:07:36.138 INFO:teuthology.orchestra.run.smithi092.stdout: debug mon = 20 2022-06-29T16:07:36.138 INFO:teuthology.orchestra.run.smithi092.stdout: debug paxos = 20 2022-06-29T16:07:36.138 INFO:teuthology.orchestra.run.smithi092.stdout: debug auth = 20 2022-06-29T16:07:36.138 INFO:teuthology.orchestra.run.smithi092.stdout: mon data avail warn = 5 2022-06-29T16:07:36.139 INFO:teuthology.orchestra.run.smithi092.stdout: mon mgr mkfs grace = 240 2022-06-29T16:07:36.139 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min pgs per osd = 4 2022-06-29T16:07:36.139 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd reporter subtree level = osd 2022-06-29T16:07:36.139 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd prime pg temp = true 2022-06-29T16:07:36.140 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min bytes per osd = 10 2022-06-29T16:07:36.140 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:36.140 INFO:teuthology.orchestra.run.smithi092.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-06-29T16:07:36.140 INFO:teuthology.orchestra.run.smithi092.stdout: auth mon ticket ttl = 660 # 11m 2022-06-29T16:07:36.141 INFO:teuthology.orchestra.run.smithi092.stdout: auth service ticket ttl = 240 # 4m 2022-06-29T16:07:36.141 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:36.141 INFO:teuthology.orchestra.run.smithi092.stdout: # don't complain about insecure global_id in the test suite 2022-06-29T16:07:36.141 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-06-29T16:07:36.141 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-06-29T16:07:36.142 INFO:teuthology.orchestra.run.smithi092.stdout: mon op complaint time = 120 2022-06-29T16:07:36.142 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:36.142 INFO:teuthology.orchestra.run.smithi092.stdout:[client] 2022-06-29T16:07:36.142 INFO:teuthology.orchestra.run.smithi092.stdout: rgw cache enabled = true 2022-06-29T16:07:36.143 INFO:teuthology.orchestra.run.smithi092.stdout: rgw enable ops log = true 2022-06-29T16:07:36.143 INFO:teuthology.orchestra.run.smithi092.stdout: rgw enable usage log = true 2022-06-29T16:07:36.143 INFO:teuthology.orchestra.run.smithi092.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-06-29T16:07:36.143 INFO:teuthology.orchestra.run.smithi092.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-06-29T16:07:36.143 INFO:teuthology.orchestra.run.smithi092.stdout: client mount timeout = 600 2022-06-29T16:07:36.144 INFO:teuthology.orchestra.run.smithi092.stdout: debug client = 20 2022-06-29T16:07:36.144 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2022-06-29T16:07:36.144 INFO:teuthology.orchestra.run.smithi092.stdout: rados mon op timeout = 15m 2022-06-29T16:07:36.145 INFO:teuthology.orchestra.run.smithi092.stdout: rados osd op timeout = 15m 2022-06-29T16:07:36.145 INFO:teuthology.orchestra.run.smithi092.stdout: client debug getattr caps = true 2022-06-29T16:07:36.145 INFO:teuthology.orchestra.run.smithi092.stdout:[mon.a] 2022-06-29T16:07:36.146 INFO:teuthology.orchestra.run.smithi092.stdout:[mon.b] 2022-06-29T16:07:36.146 INFO:teuthology.orchestra.run.smithi092.stdout:[mon.c] 2022-06-29T16:07:36.146 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.a] 2022-06-29T16:07:36.146 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.b] 2022-06-29T16:07:36.147 INFO:teuthology.orchestra.run.smithi092.stdout:[mds] 2022-06-29T16:07:36.147 INFO:teuthology.orchestra.run.smithi092.stdout: debug mds = 20 2022-06-29T16:07:36.147 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2022-06-29T16:07:36.147 INFO:teuthology.orchestra.run.smithi092.stdout: mds bal frag = True 2022-06-29T16:07:36.147 INFO:teuthology.orchestra.run.smithi092.stdout: mds bal fragment size max = 10000 2022-06-29T16:07:36.148 INFO:teuthology.orchestra.run.smithi092.stdout: mds bal merge size = 5 2022-06-29T16:07:36.148 INFO:teuthology.orchestra.run.smithi092.stdout: mds bal split bits = 3 2022-06-29T16:07:36.148 INFO:teuthology.orchestra.run.smithi092.stdout: mds bal split size = 100 2022-06-29T16:07:36.148 INFO:teuthology.orchestra.run.smithi092.stdout: mds debug frag = True 2022-06-29T16:07:36.149 INFO:teuthology.orchestra.run.smithi092.stdout: mds debug scatterstat = True 2022-06-29T16:07:36.149 INFO:teuthology.orchestra.run.smithi092.stdout: mds op complaint time = 180 2022-06-29T16:07:36.149 INFO:teuthology.orchestra.run.smithi092.stdout: mds verify scatter = True 2022-06-29T16:07:36.150 INFO:teuthology.orchestra.run.smithi092.stdout: osd op complaint time = 180 2022-06-29T16:07:36.150 INFO:teuthology.orchestra.run.smithi092.stdout: rados mon op timeout = 15m 2022-06-29T16:07:36.150 INFO:teuthology.orchestra.run.smithi092.stdout: rados osd op timeout = 15m 2022-06-29T16:07:36.152 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:07:36.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.257+0000 7ff93b611700 1 Processor -- start 2022-06-29T16:07:36.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.257+0000 7ff93b611700 1 -- start start 2022-06-29T16:07:36.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.257+0000 7ff93b611700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff93405c990 0x7ff9340b2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:36.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.257+0000 7ff93b611700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff934143940 0x7ff93405fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:36.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.257+0000 7ff93b611700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9340602c0 con 0x7ff9340b10c0 2022-06-29T16:07:36.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.257+0000 7ff93b611700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9340b1540 con 0x7ff934143940 2022-06-29T16:07:36.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.257+0000 7ff93b611700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff93405cd70 con 0x7ff93405c990 2022-06-29T16:07:36.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.257+0000 7ff938bac700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff9340b10c0 0x7ff934143230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33066/0 (socket says 172.21.15.60:33066) 2022-06-29T16:07:36.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.257+0000 7ff938bac700 1 -- 172.21.15.60:0/3242653545 learned_addr learned my addr 172.21.15.60:0/3242653545 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:36.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.257+0000 7ff939bae700 1 --2- 172.21.15.60:0/3242653545 >> v2:172.21.15.60:3300/0 conn(0x7ff934143940 0x7ff93405fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:36.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.261+0000 7ff9393ad700 1 --2- 172.21.15.60:0/3242653545 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff93405c990 0x7ff9340b2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:36.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.261+0000 7ff923fff700 1 -- 172.21.15.60:0/3242653545 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3867947688 0 0) 0x7ff924001110 con 0x7ff9340b10c0 2022-06-29T16:07:36.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.261+0000 7ff923fff700 1 -- 172.21.15.60:0/3242653545 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff91c003260 con 0x7ff9340b10c0 2022-06-29T16:07:36.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.261+0000 7ff939bae700 1 -- 172.21.15.60:0/3242653545 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff93405c990 msgr2=0x7ff9340b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:36.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.261+0000 7ff939bae700 1 --2- 172.21.15.60:0/3242653545 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff93405c990 0x7ff9340b2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:36.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.261+0000 7ff939bae700 1 -- 172.21.15.60:0/3242653545 >> v1:172.21.15.60:6789/0 conn(0x7ff9340b10c0 legacy=0x7ff934143230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:36.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.261+0000 7ff939bae700 1 -- 172.21.15.60:0/3242653545 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff934060500 con 0x7ff934143940 2022-06-29T16:07:36.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.261+0000 7ff939bae700 1 --2- 172.21.15.60:0/3242653545 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff934143940 0x7ff93405fdd0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 rx=0x7ff92c0032f0 tx=0x7ff92c0091d0).ready entity=mon.0 client_cookie=82a6a39c09e670a7 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:36.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.261+0000 7ff923fff700 1 -- 172.21.15.60:0/3242653545 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff92c00b600 con 0x7ff934143940 2022-06-29T16:07:36.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.261+0000 7ff923fff700 1 -- 172.21.15.60:0/3242653545 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff92c00b760 con 0x7ff934143940 2022-06-29T16:07:36.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.261+0000 7ff923fff700 1 -- 172.21.15.60:0/3242653545 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff92c00b8d0 con 0x7ff934143940 2022-06-29T16:07:36.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.261+0000 7ff93b611700 1 -- 172.21.15.60:0/3242653545 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff934143940 msgr2=0x7ff93405fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:36.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.261+0000 7ff93b611700 1 --2- 172.21.15.60:0/3242653545 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff934143940 0x7ff93405fdd0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 rx=0x7ff92c0032f0 tx=0x7ff92c0091d0).stop 2022-06-29T16:07:36.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.261+0000 7ff93b611700 1 -- 172.21.15.60:0/3242653545 shutdown_connections 2022-06-29T16:07:36.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.261+0000 7ff93b611700 1 --2- 172.21.15.60:0/3242653545 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff934143940 0x7ff93405fdd0 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:36.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.261+0000 7ff93b611700 1 --2- 172.21.15.60:0/3242653545 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff93405c990 0x7ff9340b2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:36.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.261+0000 7ff93b611700 1 -- 172.21.15.60:0/3242653545 >> 172.21.15.60:0/3242653545 conn(0x7ff934063aa0 msgr2=0x7ff934063e80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:36.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.261+0000 7ff93b611700 1 -- 172.21.15.60:0/3242653545 shutdown_connections 2022-06-29T16:07:36.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.261+0000 7ff93b611700 1 -- 172.21.15.60:0/3242653545 wait complete. 2022-06-29T16:07:36.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.261+0000 7ff93b611700 1 Processor -- start 2022-06-29T16:07:36.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.261+0000 7ff93b611700 1 -- start start 2022-06-29T16:07:36.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.265+0000 7ff93b611700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff93405c990 0x7ff934146890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:36.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.265+0000 7ff93b611700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9340b10c0 0x7ff93413efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:36.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.265+0000 7ff93b611700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff934143940 0x7ff93413f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:36.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.265+0000 7ff93b611700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff934145f20 con 0x7ff934143940 2022-06-29T16:07:36.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.265+0000 7ff93b611700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff934146180 con 0x7ff93405c990 2022-06-29T16:07:36.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.265+0000 7ff93b611700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9340ae0c0 con 0x7ff9340b10c0 2022-06-29T16:07:36.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.265+0000 7ff939bae700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff934143940 0x7ff93413f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:36.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.265+0000 7ff939bae700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff934143940 0x7ff93413f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53222/0 (socket says 172.21.15.60:53222) 2022-06-29T16:07:36.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.265+0000 7ff939bae700 1 -- 172.21.15.60:0/2845537145 learned_addr learned my addr 172.21.15.60:0/2845537145 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:36.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.265+0000 7ff9393ad700 1 --2- 172.21.15.60:0/2845537145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff93405c990 0x7ff934146890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:36.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.265+0000 7ff938bac700 1 --2- 172.21.15.60:0/2845537145 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9340b10c0 0x7ff93413efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:36.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.265+0000 7ff939bae700 1 -- 172.21.15.60:0/2845537145 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9340b10c0 msgr2=0x7ff93413efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:36.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.265+0000 7ff939bae700 1 --2- 172.21.15.60:0/2845537145 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9340b10c0 0x7ff93413efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:36.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.265+0000 7ff939bae700 1 -- 172.21.15.60:0/2845537145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff93405c990 msgr2=0x7ff934146890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:36.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.265+0000 7ff939bae700 1 --2- 172.21.15.60:0/2845537145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff93405c990 0x7ff934146890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:36.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.265+0000 7ff939bae700 1 -- 172.21.15.60:0/2845537145 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff92c009470 con 0x7ff934143940 2022-06-29T16:07:36.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.265+0000 7ff939bae700 1 --2- 172.21.15.60:0/2845537145 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff934143940 0x7ff93413f4b0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 rx=0x7ff92c0032f0 tx=0x7ff92c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:36.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.265+0000 7ff9227fc700 1 -- 172.21.15.60:0/2845537145 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff92c00b600 con 0x7ff934143940 2022-06-29T16:07:36.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.265+0000 7ff9227fc700 1 -- 172.21.15.60:0/2845537145 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff92c011c10 con 0x7ff934143940 2022-06-29T16:07:36.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.265+0000 7ff93b611700 1 -- 172.21.15.60:0/2845537145 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9340ae560 con 0x7ff934143940 2022-06-29T16:07:36.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.265+0000 7ff9227fc700 1 -- 172.21.15.60:0/2845537145 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff92c01f3f0 con 0x7ff934143940 2022-06-29T16:07:36.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.265+0000 7ff93b611700 1 -- 172.21.15.60:0/2845537145 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9340ae850 con 0x7ff934143940 2022-06-29T16:07:36.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.269+0000 7ff93b611700 1 -- 172.21.15.60:0/2845537145 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9340ae280 con 0x7ff934143940 2022-06-29T16:07:36.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.269+0000 7ff9227fc700 1 -- 172.21.15.60:0/2845537145 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff92c026030 con 0x7ff934143940 2022-06-29T16:07:36.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.269+0000 7ff9227fc700 1 --2- 172.21.15.60:0/2845537145 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff914053b00 0x7ff914055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:36.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.269+0000 7ff9227fc700 1 -- 172.21.15.60:0/2845537145 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff92c06cbe0 con 0x7ff934143940 2022-06-29T16:07:36.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.269+0000 7ff9393ad700 1 --2- 172.21.15.60:0/2845537145 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff914053b00 0x7ff914055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:36.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.269+0000 7ff9393ad700 1 --2- 172.21.15.60:0/2845537145 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff914053b00 0x7ff914055fc0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0x7ff93000b000 tx=0x7ff930006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:36.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.277+0000 7ff9227fc700 1 -- 172.21.15.60:0/2845537145 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff92c043970 con 0x7ff934143940 2022-06-29T16:07:36.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.409+0000 7ff93b611700 1 -- 172.21.15.60:0/2845537145 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9340573d0 con 0x7ff934143940 2022-06-29T16:07:36.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.409+0000 7ff9227fc700 1 -- 172.21.15.60:0/2845537145 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) v1 ==== 74+0+16052 (secure 0 0 0) 0x7ff92c024090 con 0x7ff934143940 2022-06-29T16:07:36.414 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:36.414 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":128,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:07:20.344096+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:07:17.372006+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2022-06-29T16:07:18.349364+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:07:36.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.413+0000 7ff93b611700 1 -- 172.21.15.60:0/2845537145 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff914053b00 msgr2=0x7ff914055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:36.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.413+0000 7ff93b611700 1 --2- 172.21.15.60:0/2845537145 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff914053b00 0x7ff914055fc0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0x7ff93000b000 tx=0x7ff930006df0).stop 2022-06-29T16:07:36.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.413+0000 7ff93b611700 1 -- 172.21.15.60:0/2845537145 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff934143940 msgr2=0x7ff93413f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:36.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.413+0000 7ff93b611700 1 --2- 172.21.15.60:0/2845537145 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff934143940 0x7ff93413f4b0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 rx=0x7ff92c0032f0 tx=0x7ff92c016000).stop 2022-06-29T16:07:36.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.413+0000 7ff93b611700 1 -- 172.21.15.60:0/2845537145 shutdown_connections 2022-06-29T16:07:36.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.413+0000 7ff93b611700 1 --2- 172.21.15.60:0/2845537145 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff914053b00 0x7ff914055fc0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:36.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.413+0000 7ff93b611700 1 --2- 172.21.15.60:0/2845537145 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff934143940 0x7ff93413f4b0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:36.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.413+0000 7ff93b611700 1 --2- 172.21.15.60:0/2845537145 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff9340b10c0 0x7ff93413efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:36.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.413+0000 7ff93b611700 1 --2- 172.21.15.60:0/2845537145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff93405c990 0x7ff934146890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:36.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.413+0000 7ff93b611700 1 -- 172.21.15.60:0/2845537145 >> 172.21.15.60:0/2845537145 conn(0x7ff934063aa0 msgr2=0x7ff9340b4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:36.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.413+0000 7ff93b611700 1 -- 172.21.15.60:0/2845537145 shutdown_connections 2022-06-29T16:07:36.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.413+0000 7ff93b611700 1 -- 172.21.15.60:0/2845537145 wait complete. 2022-06-29T16:07:36.434 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-29T16:07:17.372006+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2022-06-29T16:07:18.349364+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '128', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-29T16:07:36.435 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:07:36.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.537+0000 7f2076dc9700 1 Processor -- start 2022-06-29T16:07:36.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.537+0000 7f2076dc9700 1 -- start start 2022-06-29T16:07:36.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.537+0000 7f2076dc9700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f207005f100 0x7f207013ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:36.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.537+0000 7f2076dc9700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f207005fda0 0x7f20700aef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:36.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.537+0000 7f2076dc9700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f207005f4e0 con 0x7f20700ad160 2022-06-29T16:07:36.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.537+0000 7f2076dc9700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20700b18c0 con 0x7f207005f100 2022-06-29T16:07:36.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.537+0000 7f2076dc9700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20700ad7b0 con 0x7f207005fda0 2022-06-29T16:07:36.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.537+0000 7f2074b65700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f207005f100 0x7f207013ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:36.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.537+0000 7f2074b65700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f207005f100 0x7f207013ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53226/0 (socket says 172.21.15.60:53226) 2022-06-29T16:07:36.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.537+0000 7f2074b65700 1 -- 172.21.15.60:0/3409667925 learned_addr learned my addr 172.21.15.60:0/3409667925 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:36.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.537+0000 7f2075366700 1 --2- 172.21.15.60:0/3409667925 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f207005fda0 0x7f20700aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:36.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.541+0000 7f2074b65700 1 -- 172.21.15.60:0/3409667925 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f207005fda0 msgr2=0x7f20700aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:36.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.541+0000 7f2074b65700 1 --2- 172.21.15.60:0/3409667925 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f207005fda0 0x7f20700aef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:36.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.541+0000 7f2074b65700 1 -- 172.21.15.60:0/3409667925 >> v1:172.21.15.60:6789/0 conn(0x7f20700ad160 legacy=0x7f207013f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:36.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.541+0000 7f2074b65700 1 -- 172.21.15.60:0/3409667925 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f207013fa50 con 0x7f207005f100 2022-06-29T16:07:36.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.541+0000 7f206f7fe700 1 -- 172.21.15.60:0/3409667925 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2592883082 0 0) 0x7f207005f4e0 con 0x7f20700ad160 2022-06-29T16:07:36.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.541+0000 7f2074b65700 1 --2- 172.21.15.60:0/3409667925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f207005f100 0x7f207013ee50 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 rx=0x7f2060002f70 tx=0x7f2060006f40).ready entity=mon.0 client_cookie=c83851d5322b046f server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:36.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.541+0000 7f206f7fe700 1 -- 172.21.15.60:0/3409667925 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f206000dc70 con 0x7f207005f100 2022-06-29T16:07:36.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.541+0000 7f206f7fe700 1 -- 172.21.15.60:0/3409667925 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2060000e90 con 0x7f207005f100 2022-06-29T16:07:36.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.541+0000 7f206f7fe700 1 -- 172.21.15.60:0/3409667925 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f206000be90 con 0x7f207005f100 2022-06-29T16:07:36.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.541+0000 7f2076dc9700 1 -- 172.21.15.60:0/3409667925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f207005f100 msgr2=0x7f207013ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:36.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.541+0000 7f2076dc9700 1 --2- 172.21.15.60:0/3409667925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f207005f100 0x7f207013ee50 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 rx=0x7f2060002f70 tx=0x7f2060006f40).stop 2022-06-29T16:07:36.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.541+0000 7f2076dc9700 1 -- 172.21.15.60:0/3409667925 shutdown_connections 2022-06-29T16:07:36.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.541+0000 7f2076dc9700 1 --2- 172.21.15.60:0/3409667925 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f207005fda0 0x7f20700aef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:36.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.541+0000 7f2076dc9700 1 --2- 172.21.15.60:0/3409667925 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f207005f100 0x7f207013ee50 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:36.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.541+0000 7f2076dc9700 1 -- 172.21.15.60:0/3409667925 >> 172.21.15.60:0/3409667925 conn(0x7f207005c7f0 msgr2=0x7f20700630b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:36.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.541+0000 7f2076dc9700 1 -- 172.21.15.60:0/3409667925 shutdown_connections 2022-06-29T16:07:36.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.541+0000 7f2076dc9700 1 -- 172.21.15.60:0/3409667925 wait complete. 2022-06-29T16:07:36.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.545+0000 7f2076dc9700 1 Processor -- start 2022-06-29T16:07:36.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.545+0000 7f2076dc9700 1 -- start start 2022-06-29T16:07:36.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.545+0000 7f2076dc9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f207005f100 0x7f20700b0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:36.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.545+0000 7f2076dc9700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f207005fda0 0x7f20700643c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:36.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.545+0000 7f2076dc9700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20700ad160 0x7f20701384d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:36.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.545+0000 7f2076dc9700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2070066c50 con 0x7f207005f100 2022-06-29T16:07:36.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.545+0000 7f2076dc9700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20700b03b0 con 0x7f20700ad160 2022-06-29T16:07:36.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.545+0000 7f2076dc9700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20701420a0 con 0x7f207005fda0 2022-06-29T16:07:36.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.545+0000 7f2074b65700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f207005f100 0x7f20700b0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:36.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.545+0000 7f2074b65700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f207005f100 0x7f20700b0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53232/0 (socket says 172.21.15.60:53232) 2022-06-29T16:07:36.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.545+0000 7f2074b65700 1 -- 172.21.15.60:0/2943894284 learned_addr learned my addr 172.21.15.60:0/2943894284 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:36.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.545+0000 7f2075366700 1 --2- 172.21.15.60:0/2943894284 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20700ad160 0x7f20701384d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:36.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.545+0000 7f206ffff700 1 --2- 172.21.15.60:0/2943894284 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f207005fda0 0x7f20700643c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:36.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.545+0000 7f2074b65700 1 -- 172.21.15.60:0/2943894284 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f207005fda0 msgr2=0x7f20700643c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:36.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.545+0000 7f2074b65700 1 --2- 172.21.15.60:0/2943894284 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f207005fda0 0x7f20700643c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:36.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.545+0000 7f2074b65700 1 -- 172.21.15.60:0/2943894284 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20700ad160 msgr2=0x7f20701384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:36.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.545+0000 7f2074b65700 1 --2- 172.21.15.60:0/2943894284 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20700ad160 0x7f20701384d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:36.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.545+0000 7f2074b65700 1 -- 172.21.15.60:0/2943894284 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f206000b460 con 0x7f207005f100 2022-06-29T16:07:36.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.545+0000 7f2074b65700 1 --2- 172.21.15.60:0/2943894284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f207005f100 0x7f20700b0850 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 rx=0x7f2060002f70 tx=0x7f2060006f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:36.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.549+0000 7f206dffb700 1 -- 172.21.15.60:0/2943894284 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2060010c40 con 0x7f207005f100 2022-06-29T16:07:36.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.549+0000 7f206dffb700 1 -- 172.21.15.60:0/2943894284 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2060010da0 con 0x7f207005f100 2022-06-29T16:07:36.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.549+0000 7f2076dc9700 1 -- 172.21.15.60:0/2943894284 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20701424a0 con 0x7f207005f100 2022-06-29T16:07:36.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.549+0000 7f206dffb700 1 -- 172.21.15.60:0/2943894284 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2060024640 con 0x7f207005f100 2022-06-29T16:07:36.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.549+0000 7f2076dc9700 1 -- 172.21.15.60:0/2943894284 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2070143c70 con 0x7f207005f100 2022-06-29T16:07:36.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.549+0000 7f2076dc9700 1 -- 172.21.15.60:0/2943894284 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20700b0570 con 0x7f207005f100 2022-06-29T16:07:36.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.549+0000 7f206dffb700 1 -- 172.21.15.60:0/2943894284 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2060021070 con 0x7f207005f100 2022-06-29T16:07:36.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.549+0000 7f206dffb700 1 --2- 172.21.15.60:0/2943894284 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2050053b00 0x7f2050055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:36.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.549+0000 7f206dffb700 1 -- 172.21.15.60:0/2943894284 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f206006f9b0 con 0x7f207005f100 2022-06-29T16:07:36.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.549+0000 7f206ffff700 1 --2- 172.21.15.60:0/2943894284 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2050053b00 0x7f2050055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:36.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.549+0000 7f206ffff700 1 --2- 172.21.15.60:0/2943894284 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2050053b00 0x7f2050055fc0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0x7f206400d000 tx=0x7f2064007a20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:36.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.553+0000 7f206dffb700 1 -- 172.21.15.60:0/2943894284 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2060046800 con 0x7f207005f100 2022-06-29T16:07:36.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.677+0000 7f2076dc9700 1 -- 172.21.15.60:0/2943894284 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f20700573d0 con 0x7f207005f100 2022-06-29T16:07:36.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.677+0000 7f206dffb700 1 -- 172.21.15.60:0/2943894284 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v128) v1 ==== 110+0+10 (secure 0 0 0) 0x7f2060049de0 con 0x7f207005f100 2022-06-29T16:07:36.684 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:07:36.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.681+0000 7f2076dc9700 1 -- 172.21.15.60:0/2943894284 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2050053b00 msgr2=0x7f2050055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:36.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.681+0000 7f2076dc9700 1 --2- 172.21.15.60:0/2943894284 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2050053b00 0x7f2050055fc0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0x7f206400d000 tx=0x7f2064007a20).stop 2022-06-29T16:07:36.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.681+0000 7f2076dc9700 1 -- 172.21.15.60:0/2943894284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f207005f100 msgr2=0x7f20700b0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:36.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.681+0000 7f2076dc9700 1 --2- 172.21.15.60:0/2943894284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f207005f100 0x7f20700b0850 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 rx=0x7f2060002f70 tx=0x7f2060006f40).stop 2022-06-29T16:07:36.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.681+0000 7f2076dc9700 1 -- 172.21.15.60:0/2943894284 shutdown_connections 2022-06-29T16:07:36.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.681+0000 7f2076dc9700 1 --2- 172.21.15.60:0/2943894284 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2050053b00 0x7f2050055fc0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:36.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.681+0000 7f2076dc9700 1 --2- 172.21.15.60:0/2943894284 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20700ad160 0x7f20701384d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:36.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.681+0000 7f2076dc9700 1 --2- 172.21.15.60:0/2943894284 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f207005fda0 0x7f20700643c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:36.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.681+0000 7f2076dc9700 1 --2- 172.21.15.60:0/2943894284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f207005f100 0x7f20700b0850 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:36.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.681+0000 7f2076dc9700 1 -- 172.21.15.60:0/2943894284 >> 172.21.15.60:0/2943894284 conn(0x7f207005c7f0 msgr2=0x7f20700b4580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:36.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.681+0000 7f2076dc9700 1 -- 172.21.15.60:0/2943894284 shutdown_connections 2022-06-29T16:07:36.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.681+0000 7f2076dc9700 1 -- 172.21.15.60:0/2943894284 wait complete. 2022-06-29T16:07:36.697 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:07:36.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.793+0000 7f23e0db6700 1 Processor -- start 2022-06-29T16:07:36.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.793+0000 7f23e0db6700 1 -- start start 2022-06-29T16:07:36.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.797+0000 7f23e0db6700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f23dc1466f0 0x7f23dc146ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:36.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.797+0000 7f23e0db6700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f23dc1404a0 0x7f23dc0b1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:36.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.797+0000 7f23e0db6700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23dc05ca70 con 0x7f23dc0b3ce0 2022-06-29T16:07:36.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.797+0000 7f23e0db6700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23dc1408f0 con 0x7f23dc1404a0 2022-06-29T16:07:36.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.797+0000 7f23e0db6700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23dc146fc0 con 0x7f23dc1466f0 2022-06-29T16:07:36.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.797+0000 7f23dbfff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f23dc1404a0 0x7f23dc0b1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:36.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.797+0000 7f23dbfff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f23dc1404a0 0x7f23dc0b1ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53244/0 (socket says 172.21.15.60:53244) 2022-06-29T16:07:36.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.797+0000 7f23dbfff700 1 -- 172.21.15.60:0/2458054191 learned_addr learned my addr 172.21.15.60:0/2458054191 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:36.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.797+0000 7f23db7fe700 1 --2- 172.21.15.60:0/2458054191 >> v2:172.21.15.92:3300/0 conn(0x7f23dc1466f0 0x7f23dc146ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:36.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.797+0000 7f23dbfff700 1 -- 172.21.15.60:0/2458054191 >> v2:172.21.15.92:3300/0 conn(0x7f23dc1466f0 msgr2=0x7f23dc146ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:36.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.797+0000 7f23dbfff700 1 --2- 172.21.15.60:0/2458054191 >> v2:172.21.15.92:3300/0 conn(0x7f23dc1466f0 0x7f23dc146ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:36.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.797+0000 7f23dbfff700 1 -- 172.21.15.60:0/2458054191 >> v1:172.21.15.92:6789/0 conn(0x7f23dc0b3ce0 legacy=0x7f23dc13fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:36.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.797+0000 7f23dbfff700 1 -- 172.21.15.60:0/2458054191 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23dc05cd10 con 0x7f23dc1404a0 2022-06-29T16:07:36.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.797+0000 7f23dbfff700 1 --2- 172.21.15.60:0/2458054191 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23dc1404a0 0x7f23dc0b1ef0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 rx=0x7f23d4002f70 tx=0x7f23d4006eb0).ready entity=mon.0 client_cookie=5d6b44e4e5447f9e server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:36.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.797+0000 7f23da7fc700 1 -- 172.21.15.60:0/2458054191 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f23d400d5c0 con 0x7f23dc1404a0 2022-06-29T16:07:36.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.797+0000 7f23da7fc700 1 -- 172.21.15.60:0/2458054191 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23d4000e90 con 0x7f23dc1404a0 2022-06-29T16:07:36.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.797+0000 7f23da7fc700 1 -- 172.21.15.60:0/2458054191 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f23d400d950 con 0x7f23dc1404a0 2022-06-29T16:07:36.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.797+0000 7f23e0db6700 1 -- 172.21.15.60:0/2458054191 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23dc1404a0 msgr2=0x7f23dc0b1ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:36.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.797+0000 7f23e0db6700 1 --2- 172.21.15.60:0/2458054191 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23dc1404a0 0x7f23dc0b1ef0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 rx=0x7f23d4002f70 tx=0x7f23d4006eb0).stop 2022-06-29T16:07:36.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.797+0000 7f23e0db6700 1 -- 172.21.15.60:0/2458054191 shutdown_connections 2022-06-29T16:07:36.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.797+0000 7f23e0db6700 1 --2- 172.21.15.60:0/2458054191 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23dc1404a0 0x7f23dc0b1ef0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:36.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.797+0000 7f23e0db6700 1 --2- 172.21.15.60:0/2458054191 >> v2:172.21.15.92:3300/0 conn(0x7f23dc1466f0 0x7f23dc146ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:36.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.797+0000 7f23e0db6700 1 -- 172.21.15.60:0/2458054191 >> 172.21.15.60:0/2458054191 conn(0x7f23dc066b80 msgr2=0x7f23dc066f60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:36.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.797+0000 7f23e0db6700 1 -- 172.21.15.60:0/2458054191 shutdown_connections 2022-06-29T16:07:36.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.797+0000 7f23e0db6700 1 -- 172.21.15.60:0/2458054191 wait complete. 2022-06-29T16:07:36.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.801+0000 7f23e0db6700 1 Processor -- start 2022-06-29T16:07:36.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.801+0000 7f23e0db6700 1 -- start start 2022-06-29T16:07:36.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.801+0000 7f23e0db6700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23dc0b3ce0 0x7f23dc143a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:36.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.801+0000 7f23e0db6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23dc1404a0 0x7f23dc0ade30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:36.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.801+0000 7f23e0db6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23dc1466f0 0x7f23dc0aeb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:36.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.801+0000 7f23e0db6700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23dc0ae320 con 0x7f23dc1404a0 2022-06-29T16:07:36.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.801+0000 7f23e0db6700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23dc0af630 con 0x7f23dc0b3ce0 2022-06-29T16:07:36.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.801+0000 7f23e0db6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23dc1420c0 con 0x7f23dc1466f0 2022-06-29T16:07:36.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.801+0000 7f23daffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23dc1404a0 0x7f23dc0ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:36.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.801+0000 7f23daffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23dc1404a0 0x7f23dc0ade30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53246/0 (socket says 172.21.15.60:53246) 2022-06-29T16:07:36.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.801+0000 7f23daffd700 1 -- 172.21.15.60:0/1351469255 learned_addr learned my addr 172.21.15.60:0/1351469255 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:36.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.801+0000 7f23dbfff700 1 --2- 172.21.15.60:0/1351469255 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23dc1466f0 0x7f23dc0aeb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:36.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.801+0000 7f23db7fe700 1 --2- 172.21.15.60:0/1351469255 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23dc0b3ce0 0x7f23dc143a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:36.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.801+0000 7f23daffd700 1 -- 172.21.15.60:0/1351469255 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23dc1466f0 msgr2=0x7f23dc0aeb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:36.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.801+0000 7f23daffd700 1 --2- 172.21.15.60:0/1351469255 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23dc1466f0 0x7f23dc0aeb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:36.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.801+0000 7f23daffd700 1 -- 172.21.15.60:0/1351469255 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23dc0b3ce0 msgr2=0x7f23dc143a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:36.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.801+0000 7f23daffd700 1 --2- 172.21.15.60:0/1351469255 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23dc0b3ce0 0x7f23dc143a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:36.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.801+0000 7f23daffd700 1 -- 172.21.15.60:0/1351469255 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d400b1e0 con 0x7f23dc1404a0 2022-06-29T16:07:36.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.801+0000 7f23daffd700 1 --2- 172.21.15.60:0/1351469255 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23dc1404a0 0x7f23dc0ade30 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 rx=0x7f23dc0665e0 tx=0x7f23c4007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:36.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.801+0000 7f23d8ff9700 1 -- 172.21.15.60:0/1351469255 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f23c4012400 con 0x7f23dc1404a0 2022-06-29T16:07:36.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.801+0000 7f23d8ff9700 1 -- 172.21.15.60:0/1351469255 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c4012560 con 0x7f23dc1404a0 2022-06-29T16:07:36.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.801+0000 7f23e0db6700 1 -- 172.21.15.60:0/1351469255 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23dc0af390 con 0x7f23dc1404a0 2022-06-29T16:07:36.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.801+0000 7f23d8ff9700 1 -- 172.21.15.60:0/1351469255 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f23c40128c0 con 0x7f23dc1404a0 2022-06-29T16:07:36.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.801+0000 7f23e0db6700 1 -- 172.21.15.60:0/1351469255 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23dc0ad950 con 0x7f23dc1404a0 2022-06-29T16:07:36.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.805+0000 7f23e0db6700 1 -- 172.21.15.60:0/1351469255 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23dc0af1c0 con 0x7f23dc1404a0 2022-06-29T16:07:36.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.805+0000 7f23d8ff9700 1 -- 172.21.15.60:0/1351469255 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f23c4019030 con 0x7f23dc1404a0 2022-06-29T16:07:36.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.805+0000 7f23d8ff9700 1 --2- 172.21.15.60:0/1351469255 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f23c0053b00 0x7f23c0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:36.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.805+0000 7f23d8ff9700 1 -- 172.21.15.60:0/1351469255 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f23c406b3a0 con 0x7f23dc1404a0 2022-06-29T16:07:36.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.805+0000 7f23db7fe700 1 --2- 172.21.15.60:0/1351469255 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f23c0053b00 0x7f23c0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:36.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.805+0000 7f23db7fe700 1 --2- 172.21.15.60:0/1351469255 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f23c0053b00 0x7f23c0055fc0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7f23cc00b000 tx=0x7f23cc0091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:36.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.813+0000 7f23d8ff9700 1 -- 172.21.15.60:0/1351469255 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f23c40420d0 con 0x7f23dc1404a0 2022-06-29T16:07:36.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.937+0000 7f23e0db6700 1 -- 172.21.15.60:0/1351469255 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f23dc067d20 con 0x7f23dc1404a0 2022-06-29T16:07:36.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.937+0000 7f23d8ff9700 1 -- 172.21.15.60:0/1351469255 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v128) v1 ==== 92+0+10 (secure 0 0 0) 0x7f23c40456f0 con 0x7f23dc1404a0 2022-06-29T16:07:36.944 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:07:36.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.941+0000 7f23e0db6700 1 -- 172.21.15.60:0/1351469255 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f23c0053b00 msgr2=0x7f23c0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:36.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.941+0000 7f23e0db6700 1 --2- 172.21.15.60:0/1351469255 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f23c0053b00 0x7f23c0055fc0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7f23cc00b000 tx=0x7f23cc0091d0).stop 2022-06-29T16:07:36.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.941+0000 7f23e0db6700 1 -- 172.21.15.60:0/1351469255 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23dc1404a0 msgr2=0x7f23dc0ade30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:36.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.941+0000 7f23e0db6700 1 --2- 172.21.15.60:0/1351469255 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23dc1404a0 0x7f23dc0ade30 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 rx=0x7f23dc0665e0 tx=0x7f23c4007c50).stop 2022-06-29T16:07:36.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.941+0000 7f23e0db6700 1 -- 172.21.15.60:0/1351469255 shutdown_connections 2022-06-29T16:07:36.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.941+0000 7f23e0db6700 1 --2- 172.21.15.60:0/1351469255 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f23c0053b00 0x7f23c0055fc0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:36.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.941+0000 7f23e0db6700 1 --2- 172.21.15.60:0/1351469255 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f23dc1466f0 0x7f23dc0aeb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:36.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.941+0000 7f23e0db6700 1 --2- 172.21.15.60:0/1351469255 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f23dc1404a0 0x7f23dc0ade30 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:36.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.941+0000 7f23e0db6700 1 --2- 172.21.15.60:0/1351469255 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23dc0b3ce0 0x7f23dc143a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:36.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.941+0000 7f23e0db6700 1 -- 172.21.15.60:0/1351469255 >> 172.21.15.60:0/1351469255 conn(0x7f23dc066b80 msgr2=0x7f23dc13f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:36.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.941+0000 7f23e0db6700 1 -- 172.21.15.60:0/1351469255 shutdown_connections 2022-06-29T16:07:36.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:36.941+0000 7f23e0db6700 1 -- 172.21.15.60:0/1351469255 wait complete. 2022-06-29T16:07:36.957 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-29T16:07:37.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.065+0000 7f9e2583d700 1 Processor -- start 2022-06-29T16:07:37.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.065+0000 7f9e2583d700 1 -- start start 2022-06-29T16:07:37.070 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.065+0000 7f9e2583d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e200b97a0 0x7f9e20142b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:37.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.065+0000 7f9e2583d700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e200b9b80 con 0x7f9e201468f0 2022-06-29T16:07:37.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.065+0000 7f9e2583d700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e201440e0 con 0x7f9e20143b60 2022-06-29T16:07:37.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.065+0000 7f9e2583d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e20144380 con 0x7f9e200b97a0 2022-06-29T16:07:37.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.065+0000 7f9e1f7fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9e201468f0 0x7f9e20144ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33110/0 (socket says 172.21.15.60:33110) 2022-06-29T16:07:37.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.065+0000 7f9e1f7fe700 1 -- 172.21.15.60:0/865696439 learned_addr learned my addr 172.21.15.60:0/865696439 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:37.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.065+0000 7f9e1effd700 1 --2- 172.21.15.60:0/865696439 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e200b97a0 0x7f9e20142b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:37.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.065+0000 7f9e1e7fc700 1 -- 172.21.15.60:0/865696439 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1184383490 0 0) 0x7f9e200b9b80 con 0x7f9e201468f0 2022-06-29T16:07:37.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.065+0000 7f9e1e7fc700 1 -- 172.21.15.60:0/865696439 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e04003260 con 0x7f9e201468f0 2022-06-29T16:07:37.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.065+0000 7f9e1e7fc700 1 -- 172.21.15.60:0/865696439 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036829180 0 0) 0x7f9e201440e0 con 0x7f9e20143b60 2022-06-29T16:07:37.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.065+0000 7f9e1e7fc700 1 -- 172.21.15.60:0/865696439 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e200b9b80 con 0x7f9e20143b60 2022-06-29T16:07:37.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.065+0000 7f9e1effd700 1 -- 172.21.15.60:0/865696439 >> v1:172.21.15.92:6789/0 conn(0x7f9e20143b60 legacy=0x7f9e201461e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:37.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.065+0000 7f9e1effd700 1 -- 172.21.15.60:0/865696439 >> v1:172.21.15.60:6789/0 conn(0x7f9e201468f0 legacy=0x7f9e20144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:37.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.065+0000 7f9e1effd700 1 -- 172.21.15.60:0/865696439 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e20144620 con 0x7f9e200b97a0 2022-06-29T16:07:37.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.065+0000 7f9e1effd700 1 --2- 172.21.15.60:0/865696439 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e200b97a0 0x7f9e20142b80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f9e10002f70 tx=0x7f9e10006df0).ready entity=mon.2 client_cookie=a6d2d6ab9cce8862 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:37.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.065+0000 7f9e1e7fc700 1 -- 172.21.15.60:0/865696439 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e1000d5c0 con 0x7f9e200b97a0 2022-06-29T16:07:37.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.069+0000 7f9e1e7fc700 1 -- 172.21.15.60:0/865696439 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e10000e90 con 0x7f9e200b97a0 2022-06-29T16:07:37.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.069+0000 7f9e1e7fc700 1 -- 172.21.15.60:0/865696439 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e1000d910 con 0x7f9e200b97a0 2022-06-29T16:07:37.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.069+0000 7f9e2583d700 1 -- 172.21.15.60:0/865696439 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e200b97a0 msgr2=0x7f9e20142b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:37.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.069+0000 7f9e2583d700 1 --2- 172.21.15.60:0/865696439 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e200b97a0 0x7f9e20142b80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f9e10002f70 tx=0x7f9e10006df0).stop 2022-06-29T16:07:37.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.069+0000 7f9e2583d700 1 -- 172.21.15.60:0/865696439 shutdown_connections 2022-06-29T16:07:37.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.069+0000 7f9e2583d700 1 --2- 172.21.15.60:0/865696439 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e200b97a0 0x7f9e20142b80 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:37.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.069+0000 7f9e2583d700 1 -- 172.21.15.60:0/865696439 >> 172.21.15.60:0/865696439 conn(0x7f9e2005f930 msgr2=0x7f9e20066c60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:37.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.069+0000 7f9e2583d700 1 -- 172.21.15.60:0/865696439 shutdown_connections 2022-06-29T16:07:37.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.069+0000 7f9e2583d700 1 -- 172.21.15.60:0/865696439 wait complete. 2022-06-29T16:07:37.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.069+0000 7f9e2583d700 1 Processor -- start 2022-06-29T16:07:37.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.069+0000 7f9e2583d700 1 -- start start 2022-06-29T16:07:37.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.073+0000 7f9e2583d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e200b97a0 0x7f9e200aff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:37.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.073+0000 7f9e2583d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e20143b60 0x7f9e200b4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:37.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.073+0000 7f9e2583d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e201468f0 0x7f9e200b07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:37.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.073+0000 7f9e2583d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e20155bb0 con 0x7f9e20143b60 2022-06-29T16:07:37.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.073+0000 7f9e2583d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e20155dc0 con 0x7f9e201468f0 2022-06-29T16:07:37.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.073+0000 7f9e2583d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e200aea70 con 0x7f9e200b97a0 2022-06-29T16:07:37.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.073+0000 7f9e1effd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e200b97a0 0x7f9e200aff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:37.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.073+0000 7f9e17fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e20143b60 0x7f9e200b4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:37.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.073+0000 7f9e1f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e201468f0 0x7f9e200b07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:37.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.073+0000 7f9e1effd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e200b97a0 0x7f9e200aff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:38122/0 (socket says 172.21.15.60:38122) 2022-06-29T16:07:37.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.073+0000 7f9e1effd700 1 -- 172.21.15.60:0/338517959 learned_addr learned my addr 172.21.15.60:0/338517959 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:37.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.073+0000 7f9e17fff700 1 -- 172.21.15.60:0/338517959 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e200b97a0 msgr2=0x7f9e200aff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:37.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.073+0000 7f9e17fff700 1 --2- 172.21.15.60:0/338517959 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e200b97a0 0x7f9e200aff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:37.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.073+0000 7f9e17fff700 1 -- 172.21.15.60:0/338517959 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e201468f0 msgr2=0x7f9e200b07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:37.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.073+0000 7f9e17fff700 1 --2- 172.21.15.60:0/338517959 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e201468f0 0x7f9e200b07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:37.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.073+0000 7f9e17fff700 1 -- 172.21.15.60:0/338517959 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e1000b180 con 0x7f9e20143b60 2022-06-29T16:07:37.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.073+0000 7f9e1effd700 1 --2- 172.21.15.60:0/338517959 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e200b97a0 0x7f9e200aff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:07:37.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.073+0000 7f9e1f7fe700 1 --2- 172.21.15.60:0/338517959 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e201468f0 0x7f9e200b07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:07:37.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.073+0000 7f9e17fff700 1 --2- 172.21.15.60:0/338517959 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e20143b60 0x7f9e200b4be0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 rx=0x7f9e08001110 tx=0x7f9e08007a00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:37.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.073+0000 7f9e1cff9700 1 -- 172.21.15.60:0/338517959 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e08012400 con 0x7f9e20143b60 2022-06-29T16:07:37.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.073+0000 7f9e1cff9700 1 -- 172.21.15.60:0/338517959 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e08012560 con 0x7f9e20143b60 2022-06-29T16:07:37.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.073+0000 7f9e1cff9700 1 -- 172.21.15.60:0/338517959 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e080128c0 con 0x7f9e20143b60 2022-06-29T16:07:37.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.073+0000 7f9e2583d700 1 -- 172.21.15.60:0/338517959 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e200b50d0 con 0x7f9e20143b60 2022-06-29T16:07:37.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.073+0000 7f9e2583d700 1 -- 172.21.15.60:0/338517959 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e200aef10 con 0x7f9e20143b60 2022-06-29T16:07:37.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.077+0000 7f9e2583d700 1 -- 172.21.15.60:0/338517959 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e200aec30 con 0x7f9e20143b60 2022-06-29T16:07:37.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.077+0000 7f9e1cff9700 1 -- 172.21.15.60:0/338517959 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9e08019030 con 0x7f9e20143b60 2022-06-29T16:07:37.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.077+0000 7f9e1cff9700 1 --2- 172.21.15.60:0/338517959 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9df8053a70 0x7f9df8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:37.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.077+0000 7f9e1cff9700 1 -- 172.21.15.60:0/338517959 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9e0806b100 con 0x7f9e20143b60 2022-06-29T16:07:37.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.077+0000 7f9e1effd700 1 --2- 172.21.15.60:0/338517959 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9df8053a70 0x7f9df8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:37.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.077+0000 7f9e1effd700 1 --2- 172.21.15.60:0/338517959 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9df8053a70 0x7f9df8055f30 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7f9e10018000 tx=0x7f9e10014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:37.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.081+0000 7f9e1cff9700 1 -- 172.21.15.60:0/338517959 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9e08042020 con 0x7f9e20143b60 2022-06-29T16:07:37.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.205+0000 7f9e2583d700 1 -- 172.21.15.60:0/338517959 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9e200573d0 con 0x7f9e20143b60 2022-06-29T16:07:37.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.205+0000 7f9e1cff9700 1 -- 172.21.15.60:0/338517959 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v128) v1 ==== 104+0+10 (secure 0 0 0) 0x7f9e08045640 con 0x7f9e20143b60 2022-06-29T16:07:37.212 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:07:37.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.209+0000 7f9e2583d700 1 -- 172.21.15.60:0/338517959 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9df8053a70 msgr2=0x7f9df8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:37.216 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.209+0000 7f9e2583d700 1 --2- 172.21.15.60:0/338517959 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9df8053a70 0x7f9df8055f30 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7f9e10018000 tx=0x7f9e10014000).stop 2022-06-29T16:07:37.216 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.209+0000 7f9e2583d700 1 -- 172.21.15.60:0/338517959 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e20143b60 msgr2=0x7f9e200b4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:37.216 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.209+0000 7f9e2583d700 1 --2- 172.21.15.60:0/338517959 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e20143b60 0x7f9e200b4be0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 rx=0x7f9e08001110 tx=0x7f9e08007a00).stop 2022-06-29T16:07:37.216 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.209+0000 7f9e2583d700 1 -- 172.21.15.60:0/338517959 shutdown_connections 2022-06-29T16:07:37.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.209+0000 7f9e2583d700 1 --2- 172.21.15.60:0/338517959 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9df8053a70 0x7f9df8055f30 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:37.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.209+0000 7f9e2583d700 1 --2- 172.21.15.60:0/338517959 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e201468f0 0x7f9e200b07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:37.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.209+0000 7f9e2583d700 1 --2- 172.21.15.60:0/338517959 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9e20143b60 0x7f9e200b4be0 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:37.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.209+0000 7f9e2583d700 1 --2- 172.21.15.60:0/338517959 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9e200b97a0 0x7f9e200aff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:37.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.209+0000 7f9e2583d700 1 -- 172.21.15.60:0/338517959 >> 172.21.15.60:0/338517959 conn(0x7f9e2005f930 msgr2=0x7f9e20148aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:37.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.209+0000 7f9e2583d700 1 -- 172.21.15.60:0/338517959 shutdown_connections 2022-06-29T16:07:37.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.209+0000 7f9e2583d700 1 -- 172.21.15.60:0/338517959 wait complete. 2022-06-29T16:07:37.229 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-29T16:07:37.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.337+0000 7f3ba9b6b700 1 Processor -- start 2022-06-29T16:07:37.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.337+0000 7f3ba9b6b700 1 -- start start 2022-06-29T16:07:37.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.337+0000 7f3ba9b6b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ba40b6a80 0x7f3ba4142b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:37.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.337+0000 7f3ba9b6b700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3ba405fdf0 0x7f3ba4146740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:37.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.337+0000 7f3ba9b6b700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ba4065dd0 con 0x7f3ba4146c30 2022-06-29T16:07:37.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.337+0000 7f3ba9b6b700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ba4143bb0 con 0x7f3ba405fdf0 2022-06-29T16:07:37.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.337+0000 7f3ba9b6b700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba4063450 con 0x7f3ba40b6a80 2022-06-29T16:07:37.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.341+0000 7f3ba936a700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f3ba4146c30 0x7f3ba40afb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33126/0 (socket says 172.21.15.60:33126) 2022-06-29T16:07:37.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.341+0000 7f3ba936a700 1 -- 172.21.15.60:0/4221150638 learned_addr learned my addr 172.21.15.60:0/4221150638 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:37.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.341+0000 7f3b9bfff700 1 --2- 172.21.15.60:0/4221150638 >> v2:172.21.15.92:3300/0 conn(0x7f3ba405fdf0 0x7f3ba4146740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:37.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.341+0000 7f3ba8b69700 1 --2- 172.21.15.60:0/4221150638 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ba40b6a80 0x7f3ba4142b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:37.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.341+0000 7f3b9b7fe700 1 -- 172.21.15.60:0/4221150638 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3881095776 0 0) 0x7f3ba4065dd0 con 0x7f3ba4146c30 2022-06-29T16:07:37.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.341+0000 7f3b9bfff700 1 -- 172.21.15.60:0/4221150638 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ba40b6a80 msgr2=0x7f3ba4142b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:37.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.341+0000 7f3b9bfff700 1 --2- 172.21.15.60:0/4221150638 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ba40b6a80 0x7f3ba4142b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:37.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.341+0000 7f3b9bfff700 1 -- 172.21.15.60:0/4221150638 >> v1:172.21.15.60:6789/0 conn(0x7f3ba4146c30 legacy=0x7f3ba40afb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:37.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.341+0000 7f3b9bfff700 1 -- 172.21.15.60:0/4221150638 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba41470a0 con 0x7f3ba405fdf0 2022-06-29T16:07:37.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.341+0000 7f3b9bfff700 1 --2- 172.21.15.60:0/4221150638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ba405fdf0 0x7f3ba4146740 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0x7f3ba4063590 tx=0x7f3b94006df0).ready entity=mon.1 client_cookie=e8ef581a4f488695 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:37.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.341+0000 7f3b9b7fe700 1 -- 172.21.15.60:0/4221150638 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b9400d570 con 0x7f3ba405fdf0 2022-06-29T16:07:37.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.341+0000 7f3b9b7fe700 1 -- 172.21.15.60:0/4221150638 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b94000e90 con 0x7f3ba405fdf0 2022-06-29T16:07:37.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.341+0000 7f3b9b7fe700 1 -- 172.21.15.60:0/4221150638 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b9400d8c0 con 0x7f3ba405fdf0 2022-06-29T16:07:37.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.341+0000 7f3ba9b6b700 1 -- 172.21.15.60:0/4221150638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ba405fdf0 msgr2=0x7f3ba4146740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:37.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.341+0000 7f3ba9b6b700 1 --2- 172.21.15.60:0/4221150638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ba405fdf0 0x7f3ba4146740 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0x7f3ba4063590 tx=0x7f3b94006df0).stop 2022-06-29T16:07:37.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.341+0000 7f3ba9b6b700 1 -- 172.21.15.60:0/4221150638 shutdown_connections 2022-06-29T16:07:37.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.341+0000 7f3ba9b6b700 1 --2- 172.21.15.60:0/4221150638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ba405fdf0 0x7f3ba4146740 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:37.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.341+0000 7f3ba9b6b700 1 --2- 172.21.15.60:0/4221150638 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ba40b6a80 0x7f3ba4142b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:37.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.341+0000 7f3ba9b6b700 1 -- 172.21.15.60:0/4221150638 >> 172.21.15.60:0/4221150638 conn(0x7f3ba4064240 msgr2=0x7f3ba40b6030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:37.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.341+0000 7f3ba9b6b700 1 -- 172.21.15.60:0/4221150638 shutdown_connections 2022-06-29T16:07:37.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.341+0000 7f3ba9b6b700 1 -- 172.21.15.60:0/4221150638 wait complete. 2022-06-29T16:07:37.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.345+0000 7f3ba9b6b700 1 Processor -- start 2022-06-29T16:07:37.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.345+0000 7f3ba9b6b700 1 -- start start 2022-06-29T16:07:37.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.345+0000 7f3ba9b6b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ba405fdf0 0x7f3ba40af1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:37.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.345+0000 7f3ba9b6b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ba40b6a80 0x7f3ba4141cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:37.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.345+0000 7f3ba9b6b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ba4146c30 0x7f3ba41421c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:37.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.345+0000 7f3ba9b6b700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba40ad6c0 con 0x7f3ba4146c30 2022-06-29T16:07:37.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.345+0000 7f3ba9b6b700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba4114e40 con 0x7f3ba40b6a80 2022-06-29T16:07:37.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.345+0000 7f3ba9b6b700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba41150b0 con 0x7f3ba405fdf0 2022-06-29T16:07:37.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.345+0000 7f3ba936a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ba4146c30 0x7f3ba41421c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:37.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.345+0000 7f3b9bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ba40b6a80 0x7f3ba4141cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:37.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.345+0000 7f3ba8b69700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ba405fdf0 0x7f3ba40af1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:37.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.345+0000 7f3ba936a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ba4146c30 0x7f3ba41421c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53278/0 (socket says 172.21.15.60:53278) 2022-06-29T16:07:37.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.345+0000 7f3b9bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ba40b6a80 0x7f3ba4141cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:59306/0 (socket says 172.21.15.60:59306) 2022-06-29T16:07:37.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.345+0000 7f3ba936a700 1 -- 172.21.15.60:0/1128889717 learned_addr learned my addr 172.21.15.60:0/1128889717 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:37.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.345+0000 7f3ba8b69700 1 -- 172.21.15.60:0/1128889717 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ba40b6a80 msgr2=0x7f3ba4141cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:37.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.345+0000 7f3ba8b69700 1 --2- 172.21.15.60:0/1128889717 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ba40b6a80 0x7f3ba4141cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:37.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.345+0000 7f3ba8b69700 1 -- 172.21.15.60:0/1128889717 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ba4146c30 msgr2=0x7f3ba41421c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:37.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.345+0000 7f3ba8b69700 1 --2- 172.21.15.60:0/1128889717 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ba4146c30 0x7f3ba41421c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:37.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.345+0000 7f3ba8b69700 1 -- 172.21.15.60:0/1128889717 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b9400b180 con 0x7f3ba405fdf0 2022-06-29T16:07:37.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.349+0000 7f3ba8b69700 1 --2- 172.21.15.60:0/1128889717 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ba405fdf0 0x7f3ba40af1f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f3ba000c000 tx=0x7f3ba0006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:37.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.349+0000 7f3b99ffb700 1 -- 172.21.15.60:0/1128889717 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ba0010780 con 0x7f3ba405fdf0 2022-06-29T16:07:37.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.349+0000 7f3b99ffb700 1 -- 172.21.15.60:0/1128889717 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba00108e0 con 0x7f3ba405fdf0 2022-06-29T16:07:37.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.349+0000 7f3ba9b6b700 1 -- 172.21.15.60:0/1128889717 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ba40ada50 con 0x7f3ba405fdf0 2022-06-29T16:07:37.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.349+0000 7f3b99ffb700 1 -- 172.21.15.60:0/1128889717 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ba0010c40 con 0x7f3ba405fdf0 2022-06-29T16:07:37.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.349+0000 7f3ba9b6b700 1 -- 172.21.15.60:0/1128889717 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ba41426b0 con 0x7f3ba405fdf0 2022-06-29T16:07:37.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.349+0000 7f3ba9b6b700 1 -- 172.21.15.60:0/1128889717 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ba40573d0 con 0x7f3ba405fdf0 2022-06-29T16:07:37.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.349+0000 7f3b99ffb700 1 -- 172.21.15.60:0/1128889717 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3ba001c460 con 0x7f3ba405fdf0 2022-06-29T16:07:37.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.349+0000 7f3b99ffb700 1 --2- 172.21.15.60:0/1128889717 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3b84053aa0 0x7f3b84055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:37.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.349+0000 7f3b99ffb700 1 -- 172.21.15.60:0/1128889717 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3ba006c1a0 con 0x7f3ba405fdf0 2022-06-29T16:07:37.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.349+0000 7f3b9bfff700 1 --2- 172.21.15.60:0/1128889717 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3b84053aa0 0x7f3b84055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:37.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.353+0000 7f3b9bfff700 1 --2- 172.21.15.60:0/1128889717 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3b84053aa0 0x7f3b84055f60 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0x7f3b9401a000 tx=0x7f3b94014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:37.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.357+0000 7f3b99ffb700 1 -- 172.21.15.60:0/1128889717 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3ba0042fc0 con 0x7f3ba405fdf0 2022-06-29T16:07:37.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.489+0000 7f3ba9b6b700 1 -- 172.21.15.60:0/1128889717 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3ba4068320 con 0x7f3ba405fdf0 2022-06-29T16:07:37.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.489+0000 7f3b99ffb700 1 -- 172.21.15.60:0/1128889717 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v128) v1 ==== 100+0+10 (secure 0 0 0) 0x7f3ba00465e0 con 0x7f3ba405fdf0 2022-06-29T16:07:37.494 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:07:37.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.489+0000 7f3ba9b6b700 1 -- 172.21.15.60:0/1128889717 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3b84053aa0 msgr2=0x7f3b84055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:37.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.489+0000 7f3ba9b6b700 1 --2- 172.21.15.60:0/1128889717 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3b84053aa0 0x7f3b84055f60 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0x7f3b9401a000 tx=0x7f3b94014000).stop 2022-06-29T16:07:37.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.489+0000 7f3ba9b6b700 1 -- 172.21.15.60:0/1128889717 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ba405fdf0 msgr2=0x7f3ba40af1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:37.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.489+0000 7f3ba9b6b700 1 --2- 172.21.15.60:0/1128889717 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ba405fdf0 0x7f3ba40af1f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f3ba000c000 tx=0x7f3ba0006df0).stop 2022-06-29T16:07:37.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.493+0000 7f3ba9b6b700 1 -- 172.21.15.60:0/1128889717 shutdown_connections 2022-06-29T16:07:37.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.493+0000 7f3ba9b6b700 1 --2- 172.21.15.60:0/1128889717 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3b84053aa0 0x7f3b84055f60 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:37.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.493+0000 7f3ba9b6b700 1 --2- 172.21.15.60:0/1128889717 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3ba4146c30 0x7f3ba41421c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:37.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.493+0000 7f3ba9b6b700 1 --2- 172.21.15.60:0/1128889717 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ba40b6a80 0x7f3ba4141cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:37.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.493+0000 7f3ba9b6b700 1 --2- 172.21.15.60:0/1128889717 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3ba405fdf0 0x7f3ba40af1f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:37.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.493+0000 7f3ba9b6b700 1 -- 172.21.15.60:0/1128889717 >> 172.21.15.60:0/1128889717 conn(0x7f3ba4064240 msgr2=0x7f3ba40b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:37.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.493+0000 7f3ba9b6b700 1 -- 172.21.15.60:0/1128889717 shutdown_connections 2022-06-29T16:07:37.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.493+0000 7f3ba9b6b700 1 -- 172.21.15.60:0/1128889717 wait complete. 2022-06-29T16:07:37.510 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-06-29T16:07:37.510 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:07:37.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.605+0000 7fd39d807700 1 Processor -- start 2022-06-29T16:07:37.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.605+0000 7fd39d807700 1 -- start start 2022-06-29T16:07:37.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.609+0000 7fd39d807700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd3980b5130 0x7fd3980af270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:37.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.609+0000 7fd39d807700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd398146270 0x7fd3981466d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:37.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.609+0000 7fd39d807700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3980af0b0 con 0x7fd398147200 2022-06-29T16:07:37.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.609+0000 7fd39d807700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3980b5510 con 0x7fd3980b5130 2022-06-29T16:07:37.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.609+0000 7fd39d807700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3980aee10 con 0x7fd398146270 2022-06-29T16:07:37.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.609+0000 7fd39c805700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd3980b5130 0x7fd3980af270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:37.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.609+0000 7fd39c805700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd3980b5130 0x7fd3980af270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53282/0 (socket says 172.21.15.60:53282) 2022-06-29T16:07:37.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.609+0000 7fd39c805700 1 -- 172.21.15.60:0/1718311714 learned_addr learned my addr 172.21.15.60:0/1718311714 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:37.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.609+0000 7fd397fff700 1 --2- 172.21.15.60:0/1718311714 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd398146270 0x7fd3981466d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:37.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.609+0000 7fd39c805700 1 -- 172.21.15.60:0/1718311714 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd398146270 msgr2=0x7fd3981466d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:37.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.609+0000 7fd39c805700 1 --2- 172.21.15.60:0/1718311714 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd398146270 0x7fd3981466d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:37.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.609+0000 7fd39c805700 1 -- 172.21.15.60:0/1718311714 >> v1:172.21.15.60:6789/0 conn(0x7fd398147200 legacy=0x7fd398147650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:37.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.609+0000 7fd39c805700 1 -- 172.21.15.60:0/1718311714 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd398147d60 con 0x7fd3980b5130 2022-06-29T16:07:37.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.609+0000 7fd397fff700 1 --2- 172.21.15.60:0/1718311714 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd398146270 0x7fd3981466d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:07:37.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.609+0000 7fd39c805700 1 --2- 172.21.15.60:0/1718311714 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3980b5130 0x7fd3980af270 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 rx=0x7fd38c002f70 tx=0x7fd38c006eb0).ready entity=mon.0 client_cookie=e57f337c6e215508 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:37.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.609+0000 7fd3977fe700 1 -- 172.21.15.60:0/1718311714 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd38c00d5c0 con 0x7fd3980b5130 2022-06-29T16:07:37.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.609+0000 7fd3977fe700 1 -- 172.21.15.60:0/1718311714 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd38c000e90 con 0x7fd3980b5130 2022-06-29T16:07:37.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.609+0000 7fd3977fe700 1 -- 172.21.15.60:0/1718311714 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd38c00d950 con 0x7fd3980b5130 2022-06-29T16:07:37.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.609+0000 7fd39d807700 1 -- 172.21.15.60:0/1718311714 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3980b5130 msgr2=0x7fd3980af270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:37.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.609+0000 7fd39d807700 1 --2- 172.21.15.60:0/1718311714 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3980b5130 0x7fd3980af270 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 rx=0x7fd38c002f70 tx=0x7fd38c006eb0).stop 2022-06-29T16:07:37.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.609+0000 7fd39d807700 1 -- 172.21.15.60:0/1718311714 shutdown_connections 2022-06-29T16:07:37.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.609+0000 7fd39d807700 1 --2- 172.21.15.60:0/1718311714 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd398146270 0x7fd3981466d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:37.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.609+0000 7fd39d807700 1 --2- 172.21.15.60:0/1718311714 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3980b5130 0x7fd3980af270 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:37.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.609+0000 7fd39d807700 1 -- 172.21.15.60:0/1718311714 >> 172.21.15.60:0/1718311714 conn(0x7fd3980627a0 msgr2=0x7fd398062bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:37.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.609+0000 7fd39d807700 1 -- 172.21.15.60:0/1718311714 shutdown_connections 2022-06-29T16:07:37.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.609+0000 7fd39d807700 1 -- 172.21.15.60:0/1718311714 wait complete. 2022-06-29T16:07:37.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.613+0000 7fd39d807700 1 Processor -- start 2022-06-29T16:07:37.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.613+0000 7fd39d807700 1 -- start start 2022-06-29T16:07:37.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.613+0000 7fd39d807700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3980b5130 0x7fd3980b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:37.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.613+0000 7fd39d807700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd398146270 0x7fd398156080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:37.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.613+0000 7fd39d807700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd398147200 0x7fd398159c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:37.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.613+0000 7fd39d807700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd398065720 con 0x7fd398146270 2022-06-29T16:07:37.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.613+0000 7fd39d807700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd398159800 con 0x7fd398147200 2022-06-29T16:07:37.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.613+0000 7fd39d807700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd398066310 con 0x7fd3980b5130 2022-06-29T16:07:37.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.613+0000 7fd397fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd398146270 0x7fd398156080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:37.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.613+0000 7fd397fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd398146270 0x7fd398156080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53290/0 (socket says 172.21.15.60:53290) 2022-06-29T16:07:37.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.613+0000 7fd397fff700 1 -- 172.21.15.60:0/3672186903 learned_addr learned my addr 172.21.15.60:0/3672186903 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:37.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.613+0000 7fd39c805700 1 --2- 172.21.15.60:0/3672186903 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3980b5130 0x7fd3980b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:37.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.613+0000 7fd39d006700 1 --2- 172.21.15.60:0/3672186903 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd398147200 0x7fd398159c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:37.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.613+0000 7fd397fff700 1 -- 172.21.15.60:0/3672186903 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3980b5130 msgr2=0x7fd3980b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:37.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.613+0000 7fd397fff700 1 --2- 172.21.15.60:0/3672186903 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3980b5130 0x7fd3980b1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:37.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.613+0000 7fd397fff700 1 -- 172.21.15.60:0/3672186903 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd398147200 msgr2=0x7fd398159c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:37.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.613+0000 7fd397fff700 1 --2- 172.21.15.60:0/3672186903 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd398147200 0x7fd398159c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:37.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.613+0000 7fd397fff700 1 -- 172.21.15.60:0/3672186903 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd38c00b1e0 con 0x7fd398146270 2022-06-29T16:07:37.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.613+0000 7fd39d006700 1 --2- 172.21.15.60:0/3672186903 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd398147200 0x7fd398159c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:07:37.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.613+0000 7fd397fff700 1 --2- 172.21.15.60:0/3672186903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd398146270 0x7fd398156080 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 rx=0x7fd384007000 tx=0x7fd384008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:37.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.613+0000 7fd395ffb700 1 -- 172.21.15.60:0/3672186903 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd384010580 con 0x7fd398146270 2022-06-29T16:07:37.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.613+0000 7fd395ffb700 1 -- 172.21.15.60:0/3672186903 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3840106e0 con 0x7fd398146270 2022-06-29T16:07:37.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.613+0000 7fd39d807700 1 -- 172.21.15.60:0/3672186903 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd398156570 con 0x7fd398146270 2022-06-29T16:07:37.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.613+0000 7fd395ffb700 1 -- 172.21.15.60:0/3672186903 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd384010a40 con 0x7fd398146270 2022-06-29T16:07:37.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.613+0000 7fd39d807700 1 -- 172.21.15.60:0/3672186903 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3981114d0 con 0x7fd398146270 2022-06-29T16:07:37.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.617+0000 7fd39d807700 1 -- 172.21.15.60:0/3672186903 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3980573d0 con 0x7fd398146270 2022-06-29T16:07:37.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.617+0000 7fd395ffb700 1 -- 172.21.15.60:0/3672186903 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd384010ba0 con 0x7fd398146270 2022-06-29T16:07:37.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.617+0000 7fd395ffb700 1 --2- 172.21.15.60:0/3672186903 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd378053950 0x7fd378055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:37.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.617+0000 7fd395ffb700 1 -- 172.21.15.60:0/3672186903 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd38406cc70 con 0x7fd398146270 2022-06-29T16:07:37.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.617+0000 7fd39c805700 1 --2- 172.21.15.60:0/3672186903 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd378053950 0x7fd378055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:37.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.621+0000 7fd39c805700 1 --2- 172.21.15.60:0/3672186903 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd378053950 0x7fd378055e10 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0x7fd38c01a000 tx=0x7fd38c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:37.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.625+0000 7fd395ffb700 1 -- 172.21.15.60:0/3672186903 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd384043b30 con 0x7fd398146270 2022-06-29T16:07:37.798 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.793+0000 7fd39d807700 1 -- 172.21.15.60:0/3672186903 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3980675d0 con 0x7fd398146270 2022-06-29T16:07:37.799 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.793+0000 7fd395ffb700 1 -- 172.21.15.60:0/3672186903 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2142 (secure 0 0 0) 0x7fd384047150 con 0x7fd398146270 2022-06-29T16:07:37.799 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:37.800 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":62,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6941,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/3488623391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3488623391},{"type":"v1","addr":"172.21.15.92:6835","nonce":3488623391}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":59}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:07:19.341489+0000","modified":"2022-06-29T16:07:20.345065+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8085},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8085":{"gid":8085,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/3877729627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3877729627},{"type":"v1","addr":"172.21.15.60:6833","nonce":3877729627}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-06-29T16:07:37.803 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.797+0000 7fd39d807700 1 -- 172.21.15.60:0/3672186903 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd378053950 msgr2=0x7fd378055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:37.803 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.797+0000 7fd39d807700 1 --2- 172.21.15.60:0/3672186903 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd378053950 0x7fd378055e10 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0x7fd38c01a000 tx=0x7fd38c015000).stop 2022-06-29T16:07:37.803 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.797+0000 7fd39d807700 1 -- 172.21.15.60:0/3672186903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd398146270 msgr2=0x7fd398156080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:37.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.797+0000 7fd39d807700 1 --2- 172.21.15.60:0/3672186903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd398146270 0x7fd398156080 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 rx=0x7fd384007000 tx=0x7fd384008040).stop 2022-06-29T16:07:37.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.797+0000 7fd39d807700 1 -- 172.21.15.60:0/3672186903 shutdown_connections 2022-06-29T16:07:37.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.797+0000 7fd39d807700 1 --2- 172.21.15.60:0/3672186903 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd378053950 0x7fd378055e10 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:37.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.797+0000 7fd39d807700 1 --2- 172.21.15.60:0/3672186903 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd398147200 0x7fd398159c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:37.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.797+0000 7fd39d807700 1 --2- 172.21.15.60:0/3672186903 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd398146270 0x7fd398156080 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:37.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.797+0000 7fd39d807700 1 --2- 172.21.15.60:0/3672186903 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd3980b5130 0x7fd3980b1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:37.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.797+0000 7fd39d807700 1 -- 172.21.15.60:0/3672186903 >> 172.21.15.60:0/3672186903 conn(0x7fd3980627a0 msgr2=0x7fd398144b20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:37.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.797+0000 7fd39d807700 1 -- 172.21.15.60:0/3672186903 shutdown_connections 2022-06-29T16:07:37.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:37.797+0000 7fd39d807700 1 -- 172.21.15.60:0/3672186903 wait complete. 2022-06-29T16:07:37.806 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 62 2022-06-29T16:07:38.013 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:07:19.341489+0000', 'modified': '2022-06-29T16:07:20.345065+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8085}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8085': {'gid': 8085, 'name': 'a', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.60:6833/3877729627', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6832', 'nonce': 3877729627}, {'type': 'v1', 'addr': '172.21.15.60:6833', 'nonce': 3877729627}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:07:38.014 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:07:38.015 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-06-29T16:07:38.131 INFO:teuthology.orchestra.run.smithi060.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":8085,"want_state":"up:active","state":"up:active","rank_uptime":18.774750955999998,"mdsmap_epoch":62,"osdmap_epoch":128,"osdmap_epoch_barrier":128,"uptime":23.499805551000001} 2022-06-29T16:07:38.132 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 8085, "mdsmap_epoch": 62, "osdmap_epoch": 128, "osdmap_epoch_barrier": 128, "rank_uptime": 18.774750956, "state": "up:active", "uptime": 23.499805551, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:07:38.132 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-06-29T16:07:38.133 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-06-29T16:07:38.133 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi156.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-06-29T16:07:38.134 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:38.134 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:07:38.143 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2022-06-29T16:07:38.161 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:07:38.162 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:07:38.216 INFO:teuthology.orchestra.run.smithi156.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:07:38.217 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:07:38.219 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:38.219 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2022-06-29T16:07:38.225 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-06-29T16:07:38.226 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:07:38.269 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2022-06-29T16:07:38.288 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:07:38.289 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:07:38.300 INFO:teuthology.orchestra.run.smithi156.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:07:38.301 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:07:38.302 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:38.302 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2022-06-29T16:07:38.345 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:2022-06-29T16:07:38.337+0000 7f7e69724080 -1 init, newargv = 0x555ec43c5f80 newargc=9ceph-fuse[12987]: starting ceph client 2022-06-29T16:07:38.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr: 2022-06-29T16:07:38.355 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[12987]: starting fuse 2022-06-29T16:07:39.452 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2022-06-29T16:07:39.471 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:07:39.471 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:07:39.482 INFO:teuthology.orchestra.run.smithi156.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:07:39.483 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:07:39.484 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:39.484 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2022-06-29T16:07:39.537 INFO:teuthology.orchestra.run.smithi156.stdout:52 2022-06-29T16:07:39.538 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-06-29T16:07:39.538 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:39.539 DEBUG:teuthology.orchestra.run.smithi156:> sudo python3 -c ' 2022-06-29T16:07:39.539 DEBUG:teuthology.orchestra.run.smithi156:> import glob 2022-06-29T16:07:39.540 DEBUG:teuthology.orchestra.run.smithi156:> import re 2022-06-29T16:07:39.540 DEBUG:teuthology.orchestra.run.smithi156:> import os 2022-06-29T16:07:39.541 DEBUG:teuthology.orchestra.run.smithi156:> import subprocess 2022-06-29T16:07:39.541 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:07:39.541 DEBUG:teuthology.orchestra.run.smithi156:> def find_socket(client_name): 2022-06-29T16:07:39.542 DEBUG:teuthology.orchestra.run.smithi156:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-06-29T16:07:39.542 DEBUG:teuthology.orchestra.run.smithi156:> files = glob.glob(asok_path) 2022-06-29T16:07:39.543 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:07:39.543 DEBUG:teuthology.orchestra.run.smithi156:> # Given a non-glob path, it better be there 2022-06-29T16:07:39.544 DEBUG:teuthology.orchestra.run.smithi156:> if "*" not in asok_path: 2022-06-29T16:07:39.544 DEBUG:teuthology.orchestra.run.smithi156:> assert(len(files) == 1) 2022-06-29T16:07:39.545 DEBUG:teuthology.orchestra.run.smithi156:> return files[0] 2022-06-29T16:07:39.545 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:07:39.545 DEBUG:teuthology.orchestra.run.smithi156:> for f in files: 2022-06-29T16:07:39.546 DEBUG:teuthology.orchestra.run.smithi156:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-06-29T16:07:39.546 DEBUG:teuthology.orchestra.run.smithi156:> if os.path.exists("/proc/{0}".format(pid)): 2022-06-29T16:07:39.547 DEBUG:teuthology.orchestra.run.smithi156:> return f 2022-06-29T16:07:39.547 DEBUG:teuthology.orchestra.run.smithi156:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-06-29T16:07:39.548 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:07:39.548 DEBUG:teuthology.orchestra.run.smithi156:> print(find_socket("client.1")) 2022-06-29T16:07:39.548 DEBUG:teuthology.orchestra.run.smithi156:> ' 2022-06-29T16:07:39.594 INFO:teuthology.orchestra.run.smithi156.stdout:/var/run/ceph/ceph-client.1.12987.asok 2022-06-29T16:07:39.599 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.12987.asok 2022-06-29T16:07:39.599 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:39.600 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.12987.asok status 2022-06-29T16:07:39.692 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2022-06-29T16:07:39.693 INFO:teuthology.orchestra.run.smithi156.stdout: "metadata": { 2022-06-29T16:07:39.693 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_sha1": "f25ffac72e5a9e63860ff7deb6767a11f0559a16", 2022-06-29T16:07:39.693 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_version": "ceph version 15.2.16-304-gf25ffac7 (f25ffac72e5a9e63860ff7deb6767a11f0559a16) octopus (stable)", 2022-06-29T16:07:39.694 INFO:teuthology.orchestra.run.smithi156.stdout: "entity_id": "1", 2022-06-29T16:07:39.694 INFO:teuthology.orchestra.run.smithi156.stdout: "hostname": "smithi156", 2022-06-29T16:07:39.694 INFO:teuthology.orchestra.run.smithi156.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-06-29T16:07:39.694 INFO:teuthology.orchestra.run.smithi156.stdout: "pid": "12987", 2022-06-29T16:07:39.695 INFO:teuthology.orchestra.run.smithi156.stdout: "root": "/" 2022-06-29T16:07:39.695 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:07:39.695 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_count": 0, 2022-06-29T16:07:39.695 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_pinned_count": 0, 2022-06-29T16:07:39.695 INFO:teuthology.orchestra.run.smithi156.stdout: "id": 7127, 2022-06-29T16:07:39.696 INFO:teuthology.orchestra.run.smithi156.stdout: "inst": { 2022-06-29T16:07:39.696 INFO:teuthology.orchestra.run.smithi156.stdout: "name": { 2022-06-29T16:07:39.696 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "client", 2022-06-29T16:07:39.696 INFO:teuthology.orchestra.run.smithi156.stdout: "num": 7127 2022-06-29T16:07:39.697 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:07:39.697 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2022-06-29T16:07:39.697 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2022-06-29T16:07:39.697 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "172.21.15.156:0", 2022-06-29T16:07:39.698 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 3923477140 2022-06-29T16:07:39.698 INFO:teuthology.orchestra.run.smithi156.stdout: } 2022-06-29T16:07:39.698 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:07:39.698 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2022-06-29T16:07:39.698 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2022-06-29T16:07:39.699 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "172.21.15.156:0", 2022-06-29T16:07:39.699 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 3923477140 2022-06-29T16:07:39.699 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:07:39.699 INFO:teuthology.orchestra.run.smithi156.stdout: "inst_str": "client.7127 172.21.15.156:0/3923477140", 2022-06-29T16:07:39.700 INFO:teuthology.orchestra.run.smithi156.stdout: "addr_str": "172.21.15.156:0/3923477140", 2022-06-29T16:07:39.700 INFO:teuthology.orchestra.run.smithi156.stdout: "inode_count": 1, 2022-06-29T16:07:39.700 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_epoch": 62, 2022-06-29T16:07:39.700 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch": 128, 2022-06-29T16:07:39.700 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch_barrier": 0, 2022-06-29T16:07:39.701 INFO:teuthology.orchestra.run.smithi156.stdout: "blacklisted": false, 2022-06-29T16:07:39.701 INFO:teuthology.orchestra.run.smithi156.stdout: "fs_name": "cephfs" 2022-06-29T16:07:39.701 INFO:teuthology.orchestra.run.smithi156.stdout:} 2022-06-29T16:07:39.702 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:39.703 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:07:39.703 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:07:39.712 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2022-06-29T16:07:39.713 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-06-29T16:07:39.713 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:39.714 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:07:39.730 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:39.730 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:07:39.731 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:07:39.739 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T16:07:39.740 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T16:07:39.741 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-06-29T16:07:39.741 DEBUG:teuthology.orchestra.run.smithi121:> import time 2022-06-29T16:07:39.741 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:07:39.742 DEBUG:teuthology.orchestra.run.smithi121:> with open("/home/ubuntu/cephtest/mnt.0/testfile", '"'"'w'"'"') as f: 2022-06-29T16:07:39.742 DEBUG:teuthology.orchestra.run.smithi121:> f.write('"'"'content'"'"') 2022-06-29T16:07:39.743 DEBUG:teuthology.orchestra.run.smithi121:> f.flush() 2022-06-29T16:07:39.743 DEBUG:teuthology.orchestra.run.smithi121:> f.write('"'"'content2'"'"') 2022-06-29T16:07:39.743 DEBUG:teuthology.orchestra.run.smithi121:> while True: 2022-06-29T16:07:39.744 DEBUG:teuthology.orchestra.run.smithi121:> time.sleep(1) 2022-06-29T16:07:39.744 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-06-29T16:07:39.747 DEBUG:teuthology.orchestra.run.smithi121:> sudo ls /home/ubuntu/cephtest/mnt.0/testfile 2022-06-29T16:07:39.767 INFO:teuthology.orchestra.run.smithi121.stderr:ls: cannot access '/home/ubuntu/cephtest/mnt.0/testfile': No such file or directory 2022-06-29T16:07:39.768 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-06-29T16:07:40.770 DEBUG:teuthology.orchestra.run.smithi121:> sudo ls /home/ubuntu/cephtest/mnt.0/testfile 2022-06-29T16:07:40.787 INFO:teuthology.orchestra.run.smithi121.stdout:/home/ubuntu/cephtest/mnt.0/testfile 2022-06-29T16:07:40.789 DEBUG:tasks.cephfs.mount:File testfile became visible from 0 after 1s 2022-06-29T16:07:40.789 DEBUG:teuthology.orchestra.run.smithi156:> sudo ls /home/ubuntu/cephtest/mnt.1/testfile 2022-06-29T16:07:40.826 INFO:teuthology.orchestra.run.smithi156.stdout:/home/ubuntu/cephtest/mnt.1/testfile 2022-06-29T16:07:40.828 DEBUG:tasks.cephfs.mount:File testfile became visible from 1 after 0s 2022-06-29T16:07:40.828 INFO:teuthology.orchestra.run:Running command with timeout 10800 2022-06-29T16:07:40.829 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest/mnt.1 && sudo cat testfile 2022-06-29T16:07:40.871 INFO:teuthology.orchestra.run.smithi156.stdout:content 2022-06-29T16:07:40.888 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-06-29T16:07:40.889 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:07:40.909 DEBUG:teuthology.orchestra.run.smithi092:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:07:40.928 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:07:40.950 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:07:40.971 DEBUG:teuthology.orchestra.run.smithi197:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:07:40.992 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-06-29T16:07:40.993 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:40.994 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:07:40.994 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:07:41.002 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T16:07:41.004 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T16:07:41.004 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2022-06-29T16:07:41.004 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-06-29T16:07:41.005 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:07:41.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[13443]: fuse finished with error 0 and tester_r 0 2022-06-29T16:07:41.023 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:41.023 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:07:41.024 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:07:41.031 INFO:teuthology.orchestra.run.smithi121.stdout:ext2/ext3 2022-06-29T16:07:41.032 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-06-29T16:07:41.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:07:41.089 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:07:41.098 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:41.099 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:07:41.099 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:07:41.108 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2022-06-29T16:07:41.109 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-06-29T16:07:41.110 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi156.front.sepia.ceph.com... 2022-06-29T16:07:41.110 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-06-29T16:07:41.111 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:07:41.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[12987]: fuse finished with error 0 and tester_r 0 2022-06-29T16:07:41.168 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:41.168 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:07:41.169 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:07:41.174 INFO:teuthology.orchestra.run.smithi156.stdout:ext2/ext3 2022-06-29T16:07:41.175 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-06-29T16:07:41.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:07:41.292 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:07:41.299 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:41.300 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:07:41.300 DEBUG:teuthology.orchestra.run.smithi197:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T16:07:41.308 INFO:teuthology.orchestra.run.smithi197.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-06-29T16:07:41.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:07:41.309 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-06-29T16:07:41.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:07:41.310 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T16:07:41.317 INFO:teuthology.orchestra.run.smithi197.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-06-29T16:07:41.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:07:41.318 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-06-29T16:07:41.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.421+0000 7f9870926700 1 Processor -- start 2022-06-29T16:07:41.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.421+0000 7f9870926700 1 -- start start 2022-06-29T16:07:41.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.425+0000 7f9870926700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f986c14b640 0x7f986c14da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:41.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.425+0000 7f9870926700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f986c0ae280 con 0x7f986c0b43b0 2022-06-29T16:07:41.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.425+0000 7f9870926700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f986c0b99a0 con 0x7f986c0b95c0 2022-06-29T16:07:41.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.425+0000 7f9870926700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f986c0ad690 con 0x7f986c14b640 2022-06-29T16:07:41.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.425+0000 7f986bfff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f986c14b640 0x7f986c14da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:41.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.425+0000 7f986affd700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f986c0b43b0 0x7f986c1460a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33150/0 (socket says 172.21.15.60:33150) 2022-06-29T16:07:41.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.425+0000 7f986affd700 1 -- 172.21.15.60:0/430324838 learned_addr learned my addr 172.21.15.60:0/430324838 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:41.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.425+0000 7f986bfff700 1 -- 172.21.15.60:0/430324838 >> v1:172.21.15.92:6789/0 conn(0x7f986c0b95c0 legacy=0x7f986c0ae980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:41.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.425+0000 7f986bfff700 1 -- 172.21.15.60:0/430324838 >> v1:172.21.15.60:6789/0 conn(0x7f986c0b43b0 legacy=0x7f986c1460a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:41.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.425+0000 7f986bfff700 1 -- 172.21.15.60:0/430324838 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f986c0ad350 con 0x7f986c14b640 2022-06-29T16:07:41.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.425+0000 7f986a7fc700 1 -- 172.21.15.60:0/430324838 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1115948335 0 0) 0x7f986c0ae280 con 0x7f986c0b43b0 2022-06-29T16:07:41.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.425+0000 7f986bfff700 1 --2- 172.21.15.60:0/430324838 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f986c14b640 0x7f986c14da30 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 rx=0x7f98640032f0 tx=0x7f98640091d0).ready entity=mon.0 client_cookie=f997fca6ec3c4f6e server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:41.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.425+0000 7f986a7fc700 1 -- 172.21.15.60:0/430324838 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9864009ea0 con 0x7f986c14b640 2022-06-29T16:07:41.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.425+0000 7f986a7fc700 1 -- 172.21.15.60:0/430324838 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f986400b870 con 0x7f986c14b640 2022-06-29T16:07:41.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.425+0000 7f986a7fc700 1 -- 172.21.15.60:0/430324838 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f986400b9e0 con 0x7f986c14b640 2022-06-29T16:07:41.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.425+0000 7f9870926700 1 -- 172.21.15.60:0/430324838 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f986c14b640 msgr2=0x7f986c14da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:41.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.425+0000 7f9870926700 1 --2- 172.21.15.60:0/430324838 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f986c14b640 0x7f986c14da30 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 rx=0x7f98640032f0 tx=0x7f98640091d0).stop 2022-06-29T16:07:41.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.425+0000 7f9870926700 1 -- 172.21.15.60:0/430324838 shutdown_connections 2022-06-29T16:07:41.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.425+0000 7f9870926700 1 --2- 172.21.15.60:0/430324838 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f986c14b640 0x7f986c14da30 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:41.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.425+0000 7f9870926700 1 -- 172.21.15.60:0/430324838 >> 172.21.15.60:0/430324838 conn(0x7f986c069a60 msgr2=0x7f986c066bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:41.437 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.425+0000 7f9870926700 1 -- 172.21.15.60:0/430324838 shutdown_connections 2022-06-29T16:07:41.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.425+0000 7f9870926700 1 -- 172.21.15.60:0/430324838 wait complete. 2022-06-29T16:07:41.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.429+0000 7f9870926700 1 Processor -- start 2022-06-29T16:07:41.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.429+0000 7f9870926700 1 -- start start 2022-06-29T16:07:41.441 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.429+0000 7f9870926700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f986c0b43b0 0x7f986c1409e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:41.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.429+0000 7f9870926700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f986c0b95c0 0x7f986c155360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:41.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.429+0000 7f9870926700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f986c14b640 0x7f986c0b1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:41.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.429+0000 7f9870926700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f986c145be0 con 0x7f986c0b43b0 2022-06-29T16:07:41.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.429+0000 7f9870926700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f986c0b1580 con 0x7f986c0b95c0 2022-06-29T16:07:41.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.429+0000 7f9870926700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f986c145e10 con 0x7f986c14b640 2022-06-29T16:07:41.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.429+0000 7f986b7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f986c0b43b0 0x7f986c1409e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:41.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.429+0000 7f986b7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f986c0b43b0 0x7f986c1409e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53302/0 (socket says 172.21.15.60:53302) 2022-06-29T16:07:41.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.429+0000 7f986b7fe700 1 -- 172.21.15.60:0/2383513924 learned_addr learned my addr 172.21.15.60:0/2383513924 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:41.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.429+0000 7f986affd700 1 --2- 172.21.15.60:0/2383513924 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f986c0b95c0 0x7f986c155360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:41.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.429+0000 7f986bfff700 1 --2- 172.21.15.60:0/2383513924 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f986c14b640 0x7f986c0b1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:41.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.429+0000 7f986b7fe700 1 -- 172.21.15.60:0/2383513924 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f986c14b640 msgr2=0x7f986c0b1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:41.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.429+0000 7f986b7fe700 1 --2- 172.21.15.60:0/2383513924 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f986c14b640 0x7f986c0b1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:41.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.429+0000 7f986b7fe700 1 -- 172.21.15.60:0/2383513924 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f986c0b95c0 msgr2=0x7f986c155360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:41.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.429+0000 7f986b7fe700 1 --2- 172.21.15.60:0/2383513924 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f986c0b95c0 0x7f986c155360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:41.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.429+0000 7f986b7fe700 1 -- 172.21.15.60:0/2383513924 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9864009470 con 0x7f986c0b43b0 2022-06-29T16:07:41.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.429+0000 7f986affd700 1 --2- 172.21.15.60:0/2383513924 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f986c0b95c0 0x7f986c155360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:41.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.429+0000 7f986b7fe700 1 --2- 172.21.15.60:0/2383513924 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f986c0b43b0 0x7f986c1409e0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 rx=0x7f986c0b6e90 tx=0x7f9854007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:41.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.429+0000 7f9868ff9700 1 -- 172.21.15.60:0/2383513924 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9854012400 con 0x7f986c0b43b0 2022-06-29T16:07:41.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.429+0000 7f9868ff9700 1 -- 172.21.15.60:0/2383513924 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9854012560 con 0x7f986c0b43b0 2022-06-29T16:07:41.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.429+0000 7f9870926700 1 -- 172.21.15.60:0/2383513924 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f986c0b17e0 con 0x7f986c0b43b0 2022-06-29T16:07:41.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.429+0000 7f9868ff9700 1 -- 172.21.15.60:0/2383513924 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98540128c0 con 0x7f986c0b43b0 2022-06-29T16:07:41.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.429+0000 7f9870926700 1 -- 172.21.15.60:0/2383513924 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f986c141520 con 0x7f986c0b43b0 2022-06-29T16:07:41.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.433+0000 7f9870926700 1 -- 172.21.15.60:0/2383513924 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f986c0573d0 con 0x7f986c0b43b0 2022-06-29T16:07:41.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.433+0000 7f9868ff9700 1 -- 172.21.15.60:0/2383513924 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9854019030 con 0x7f986c0b43b0 2022-06-29T16:07:41.448 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.433+0000 7f9868ff9700 1 --2- 172.21.15.60:0/2383513924 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9850053a70 0x7f9850055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:41.448 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.433+0000 7f9868ff9700 1 -- 172.21.15.60:0/2383513924 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f985406b2a0 con 0x7f986c0b43b0 2022-06-29T16:07:41.448 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.433+0000 7f986affd700 1 --2- 172.21.15.60:0/2383513924 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9850053a70 0x7f9850055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:41.448 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.433+0000 7f986affd700 1 --2- 172.21.15.60:0/2383513924 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9850053a70 0x7f9850055f30 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0x7f985c00b000 tx=0x7f985c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:41.449 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.437+0000 7f9868ff9700 1 -- 172.21.15.60:0/2383513924 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f98540421c0 con 0x7f986c0b43b0 2022-06-29T16:07:41.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.569+0000 7f9870926700 1 -- 172.21.15.60:0/2383513924 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f986c067980 con 0x7f986c0b43b0 2022-06-29T16:07:41.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.569+0000 7f9868ff9700 1 -- 172.21.15.60:0/2383513924 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v128) v1 ==== 81+0+33659 (secure 0 0 0) 0x7f98540457e0 con 0x7f986c0b43b0 2022-06-29T16:07:41.576 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:41.576 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:07:41.577 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 128, 2022-06-29T16:07:41.577 INFO:teuthology.orchestra.run.smithi060.stdout: "fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", 2022-06-29T16:07:41.577 INFO:teuthology.orchestra.run.smithi060.stdout: "created": "2022-06-29T15:58:07.965719+0000", 2022-06-29T16:07:41.577 INFO:teuthology.orchestra.run.smithi060.stdout: "modified": "2022-06-29T16:07:20.344096+0000", 2022-06-29T16:07:41.578 INFO:teuthology.orchestra.run.smithi060.stdout: "last_up_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:07:41.578 INFO:teuthology.orchestra.run.smithi060.stdout: "last_in_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:07:41.578 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-06-29T16:07:41.578 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_num": 5799936, 2022-06-29T16:07:41.579 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_set": [ 2022-06-29T16:07:41.579 INFO:teuthology.orchestra.run.smithi060.stdout: "pglog_hardlimit", 2022-06-29T16:07:41.579 INFO:teuthology.orchestra.run.smithi060.stdout: "purged_snapdirs", 2022-06-29T16:07:41.579 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_deletes", 2022-06-29T16:07:41.579 INFO:teuthology.orchestra.run.smithi060.stdout: "sortbitwise" 2022-06-29T16:07:41.580 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:07:41.580 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_version": 4, 2022-06-29T16:07:41.580 INFO:teuthology.orchestra.run.smithi060.stdout: "full_ratio": 0.94999998807907104, 2022-06-29T16:07:41.580 INFO:teuthology.orchestra.run.smithi060.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-06-29T16:07:41.581 INFO:teuthology.orchestra.run.smithi060.stdout: "nearfull_ratio": 0.85000002384185791, 2022-06-29T16:07:41.581 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_snapshot": "", 2022-06-29T16:07:41.581 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_max": 26, 2022-06-29T16:07:41.581 INFO:teuthology.orchestra.run.smithi060.stdout: "max_osd": 8, 2022-06-29T16:07:41.581 INFO:teuthology.orchestra.run.smithi060.stdout: "require_min_compat_client": "jewel", 2022-06-29T16:07:41.582 INFO:teuthology.orchestra.run.smithi060.stdout: "min_compat_client": "jewel", 2022-06-29T16:07:41.582 INFO:teuthology.orchestra.run.smithi060.stdout: "require_osd_release": "octopus", 2022-06-29T16:07:41.582 INFO:teuthology.orchestra.run.smithi060.stdout: "pools": [ 2022-06-29T16:07:41.582 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.583 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2022-06-29T16:07:41.583 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "device_health_metrics", 2022-06-29T16:07:41.583 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:10.961127+0000", 2022-06-29T16:07:41.583 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:07:41.583 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:07:41.584 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:07:41.584 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:07:41.584 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:07:41.584 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:07:41.585 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:07:41.585 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:07:41.585 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 1, 2022-06-29T16:07:41.585 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 1, 2022-06-29T16:07:41.586 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 1, 2022-06-29T16:07:41.586 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 1, 2022-06-29T16:07:41.586 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 1, 2022-06-29T16:07:41.586 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:07:41.586 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:07:41.587 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:07:41.587 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:07:41.587 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:07:41.587 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:07:41.588 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:07:41.588 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.588 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "15", 2022-06-29T16:07:41.588 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:07:41.588 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:07:41.589 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:07:41.589 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:07:41.589 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:07:41.589 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:07:41.589 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:07:41.590 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:07:41.590 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:07:41.590 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:07:41.590 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:07:41.591 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:07:41.591 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:07:41.591 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:07:41.591 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:07:41.591 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:07:41.592 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:07:41.592 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:07:41.592 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:07:41.592 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:07:41.593 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:07:41.593 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:07:41.593 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:07:41.593 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:07:41.594 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:07:41.594 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:07:41.594 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.594 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:07:41.595 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:07:41.595 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:07:41.595 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:07:41.595 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:07:41.595 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:07:41.596 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:07:41.596 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:07:41.596 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:07:41.596 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:07:41.597 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:07:41.597 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:07:41.597 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 1 2022-06-29T16:07:41.597 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.597 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:07:41.598 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_devicehealth": {} 2022-06-29T16:07:41.598 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.598 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.598 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.599 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 2, 2022-06-29T16:07:41.599 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "rbd", 2022-06-29T16:07:41.599 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:19.184276+0000", 2022-06-29T16:07:41.599 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:07:41.599 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:07:41.600 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:07:41.600 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:07:41.600 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:07:41.600 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:07:41.600 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:07:41.601 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:07:41.601 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:07:41.601 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:07:41.601 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:07:41.602 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:07:41.602 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:07:41.602 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:07:41.602 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:07:41.602 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:07:41.603 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:07:41.603 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:07:41.603 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:07:41.603 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:07:41.603 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.604 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "18", 2022-06-29T16:07:41.604 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:07:41.604 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:07:41.604 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:07:41.605 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:07:41.605 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:07:41.605 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:07:41.605 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:07:41.606 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:07:41.606 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:07:41.606 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:07:41.606 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:07:41.607 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:07:41.607 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:07:41.607 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:07:41.607 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:07:41.607 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:07:41.608 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:07:41.608 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:07:41.608 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:07:41.608 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:07:41.609 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:07:41.609 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:07:41.609 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:07:41.609 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:07:41.609 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:07:41.610 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:07:41.610 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.610 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:07:41.610 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:07:41.610 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:07:41.611 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:07:41.611 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:07:41.611 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:07:41.611 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:07:41.612 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:07:41.612 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:07:41.612 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:07:41.612 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:07:41.612 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:07:41.613 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:07:41.613 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd": {} 2022-06-29T16:07:41.613 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.613 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.614 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.614 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 25, 2022-06-29T16:07:41.614 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "cephfs_metadata", 2022-06-29T16:07:41.614 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T16:07:17.372006+0000", 2022-06-29T16:07:41.614 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:07:41.615 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:07:41.615 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:07:41.615 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:07:41.615 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:07:41.616 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:07:41.616 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:07:41.616 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:07:41.616 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:07:41.616 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:07:41.617 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:07:41.617 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:07:41.617 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:07:41.617 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:07:41.617 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:07:41.618 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:07:41.618 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:07:41.618 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:07:41.618 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:07:41.619 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:07:41.619 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.619 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "127", 2022-06-29T16:07:41.619 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:07:41.619 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:07:41.620 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:07:41.620 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:07:41.620 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:07:41.620 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:07:41.621 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:07:41.621 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:07:41.621 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:07:41.621 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:07:41.621 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:07:41.622 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:07:41.622 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:07:41.622 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:07:41.622 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:07:41.622 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:07:41.623 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:07:41.623 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:07:41.623 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:07:41.623 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:07:41.624 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:07:41.624 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:07:41.624 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:07:41.624 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:07:41.625 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:07:41.625 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:07:41.625 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.625 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:07:41.625 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:07:41.626 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:07:41.626 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:07:41.626 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:07:41.626 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:07:41.626 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:07:41.627 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:07:41.627 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:07:41.627 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:07:41.627 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:07:41.628 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:07:41.628 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_bias": 4, 2022-06-29T16:07:41.628 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 16, 2022-06-29T16:07:41.628 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_priority": 5 2022-06-29T16:07:41.628 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.629 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:07:41.629 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs": { 2022-06-29T16:07:41.629 INFO:teuthology.orchestra.run.smithi060.stdout: "metadata": "cephfs" 2022-06-29T16:07:41.629 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.630 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.630 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.630 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.630 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 26, 2022-06-29T16:07:41.631 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "cephfs_data", 2022-06-29T16:07:41.631 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T16:07:18.349364+0000", 2022-06-29T16:07:41.631 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:07:41.631 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:07:41.631 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:07:41.632 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:07:41.632 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:07:41.632 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:07:41.632 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:07:41.633 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:07:41.633 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:07:41.633 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:07:41.633 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:07:41.633 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:07:41.634 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:07:41.634 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:07:41.634 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:07:41.634 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:07:41.635 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:07:41.635 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:07:41.635 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:07:41.635 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:07:41.635 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.636 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "128", 2022-06-29T16:07:41.636 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:07:41.636 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:07:41.636 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:07:41.637 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:07:41.637 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:07:41.637 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:07:41.637 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:07:41.637 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:07:41.638 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:07:41.638 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:07:41.638 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:07:41.638 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:07:41.639 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:07:41.639 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:07:41.639 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:07:41.639 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:07:41.639 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:07:41.640 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:07:41.640 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:07:41.640 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:07:41.640 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:07:41.641 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:07:41.641 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:07:41.641 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:07:41.641 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:07:41.641 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:07:41.642 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.642 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:07:41.642 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:07:41.642 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:07:41.643 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:07:41.643 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:07:41.643 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:07:41.643 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:07:41.643 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:07:41.644 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:07:41.644 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:07:41.644 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:07:41.645 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:07:41.645 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:07:41.645 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs": { 2022-06-29T16:07:41.646 INFO:teuthology.orchestra.run.smithi060.stdout: "data": "cephfs" 2022-06-29T16:07:41.646 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.646 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.647 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.647 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:07:41.648 INFO:teuthology.orchestra.run.smithi060.stdout: "osds": [ 2022-06-29T16:07:41.648 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.648 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:07:41.649 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "11e7ab9e-fce2-470f-8195-ea4c3c52df22", 2022-06-29T16:07:41.649 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:07:41.649 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:07:41.650 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:07:41.650 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:07:41.650 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:07:41.650 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:07:41.651 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:07:41.651 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 125, 2022-06-29T16:07:41.651 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:07:41.651 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:07:41.652 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:07:41.652 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:41.652 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.652 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:41.652 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6816", 2022-06-29T16:07:41.653 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:07:41.653 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.653 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.653 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:41.654 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6825", 2022-06-29T16:07:41.654 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:07:41.654 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.654 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.654 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.655 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:07:41.655 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:41.655 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.655 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:41.656 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6826", 2022-06-29T16:07:41.656 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:07:41.656 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.656 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.656 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:41.657 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6827", 2022-06-29T16:07:41.657 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:07:41.657 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.657 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.657 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.658 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:07:41.658 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:41.658 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.658 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:41.659 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6830", 2022-06-29T16:07:41.659 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:07:41.659 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.659 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.660 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:41.660 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6831", 2022-06-29T16:07:41.660 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:07:41.660 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.660 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.661 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.661 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:07:41.661 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:41.661 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.662 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:41.662 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6828", 2022-06-29T16:07:41.662 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:07:41.662 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.662 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.663 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:41.663 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6829", 2022-06-29T16:07:41.663 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:07:41.663 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.663 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.664 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.664 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6825/12890", 2022-06-29T16:07:41.664 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6827/12890", 2022-06-29T16:07:41.665 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6831/12890", 2022-06-29T16:07:41.665 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6829/12890", 2022-06-29T16:07:41.665 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:07:41.666 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:07:41.666 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:07:41.666 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.667 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.667 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.668 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:07:41.668 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "ca43f136-6f69-4431-a157-fbe073fb585c", 2022-06-29T16:07:41.668 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:07:41.669 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:07:41.669 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:07:41.670 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:07:41.670 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:07:41.670 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:07:41.671 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:07:41.671 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 126, 2022-06-29T16:07:41.671 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:07:41.671 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:07:41.672 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:07:41.672 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:41.672 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.672 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:41.673 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6815", 2022-06-29T16:07:41.673 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:07:41.673 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.673 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.673 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:41.674 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6818", 2022-06-29T16:07:41.674 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:07:41.674 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.674 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.674 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.675 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:07:41.675 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:41.675 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.675 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:41.676 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6819", 2022-06-29T16:07:41.676 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:07:41.676 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.676 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.676 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:41.677 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6820", 2022-06-29T16:07:41.677 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:07:41.677 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.677 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.678 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:07:41.678 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:41.678 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.678 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:41.679 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6823", 2022-06-29T16:07:41.679 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:07:41.679 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.679 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.680 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:41.680 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6824", 2022-06-29T16:07:41.680 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:07:41.680 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.680 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.681 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.681 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:07:41.681 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:41.681 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.681 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:41.682 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6821", 2022-06-29T16:07:41.682 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:07:41.682 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.682 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.683 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:41.683 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6822", 2022-06-29T16:07:41.683 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:07:41.683 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.683 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.684 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.684 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6818/12892", 2022-06-29T16:07:41.684 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6820/12892", 2022-06-29T16:07:41.684 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6824/12892", 2022-06-29T16:07:41.685 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6822/12892", 2022-06-29T16:07:41.685 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:07:41.685 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:07:41.685 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:07:41.685 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.686 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.686 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.686 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:07:41.686 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "74fac45f-5f07-44ec-a9cb-2644c123fd4a", 2022-06-29T16:07:41.687 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:07:41.687 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:07:41.687 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:07:41.687 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:07:41.687 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:07:41.688 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:07:41.688 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:07:41.688 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 126, 2022-06-29T16:07:41.688 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:07:41.689 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:07:41.689 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:07:41.689 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:41.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.689 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:41.690 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6808", 2022-06-29T16:07:41.690 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:07:41.690 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.690 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.690 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:41.691 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6809", 2022-06-29T16:07:41.691 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:07:41.691 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.691 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.692 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.692 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:07:41.692 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:41.692 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.693 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:41.693 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6810", 2022-06-29T16:07:41.693 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:07:41.693 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.693 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.694 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:41.694 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6811", 2022-06-29T16:07:41.694 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:07:41.694 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.695 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.695 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:07:41.695 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:41.695 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.696 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:41.696 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6814", 2022-06-29T16:07:41.696 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:07:41.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.697 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:41.697 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6817", 2022-06-29T16:07:41.697 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:07:41.697 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.698 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.698 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.698 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:07:41.698 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:41.698 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.699 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:41.699 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6812", 2022-06-29T16:07:41.699 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:07:41.699 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.700 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.700 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:41.700 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6813", 2022-06-29T16:07:41.700 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:07:41.700 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.701 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.701 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.701 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6809/12891", 2022-06-29T16:07:41.701 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6811/12891", 2022-06-29T16:07:41.701 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6817/12891", 2022-06-29T16:07:41.702 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6813/12891", 2022-06-29T16:07:41.702 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:07:41.702 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:07:41.702 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:07:41.703 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.703 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.703 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.704 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:07:41.704 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "2581d351-973a-4b4e-85ca-3658f0068765", 2022-06-29T16:07:41.704 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:07:41.705 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:07:41.705 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:07:41.705 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:07:41.706 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:07:41.706 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:07:41.706 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:07:41.707 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 125, 2022-06-29T16:07:41.707 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:07:41.707 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:07:41.708 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:07:41.708 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:41.709 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.709 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:41.709 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6800", 2022-06-29T16:07:41.709 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:07:41.710 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.710 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.710 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:41.710 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6801", 2022-06-29T16:07:41.710 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:07:41.711 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.711 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.711 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.711 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:07:41.711 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:41.712 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.712 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:41.712 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6802", 2022-06-29T16:07:41.712 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:07:41.713 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.713 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.713 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:41.713 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6803", 2022-06-29T16:07:41.713 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:07:41.714 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.714 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.714 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.714 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:07:41.715 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:41.715 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.715 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:41.715 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6806", 2022-06-29T16:07:41.715 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:07:41.716 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.716 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.716 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:41.716 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6807", 2022-06-29T16:07:41.717 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:07:41.717 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.717 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.717 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.717 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:07:41.718 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:41.718 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.718 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:41.718 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6804", 2022-06-29T16:07:41.719 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:07:41.719 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.719 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.719 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:41.719 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6805", 2022-06-29T16:07:41.720 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:07:41.720 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.720 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.720 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.720 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6801/12889", 2022-06-29T16:07:41.721 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6803/12889", 2022-06-29T16:07:41.721 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6807/12889", 2022-06-29T16:07:41.721 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6805/12889", 2022-06-29T16:07:41.721 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:07:41.722 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:07:41.722 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:07:41.722 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.722 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.722 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.723 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:07:41.723 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "3d234db9-87cd-4fe9-814f-0240a287d8b4", 2022-06-29T16:07:41.723 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:07:41.724 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:07:41.724 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:07:41.724 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:07:41.725 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:07:41.725 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:07:41.725 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:07:41.726 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 126, 2022-06-29T16:07:41.726 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:07:41.726 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:07:41.727 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:07:41.727 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:41.727 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.728 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:41.728 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6808", 2022-06-29T16:07:41.728 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:07:41.729 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.729 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.729 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:41.729 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6810", 2022-06-29T16:07:41.730 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:07:41.730 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.730 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.730 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.731 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:07:41.731 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:41.731 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.731 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:41.732 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6812", 2022-06-29T16:07:41.732 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:07:41.732 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.732 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.732 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:41.733 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6814", 2022-06-29T16:07:41.733 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:07:41.733 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.733 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.734 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.734 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:07:41.734 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:41.735 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.735 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:41.735 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6823", 2022-06-29T16:07:41.736 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:07:41.736 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.736 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.736 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:41.737 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6825", 2022-06-29T16:07:41.737 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:07:41.737 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.737 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.738 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.738 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:07:41.738 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:41.738 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.739 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:41.739 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6817", 2022-06-29T16:07:41.739 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:07:41.739 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.740 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.740 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:41.740 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6820", 2022-06-29T16:07:41.740 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:07:41.741 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.741 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.741 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.741 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6810/13249", 2022-06-29T16:07:41.742 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6814/13249", 2022-06-29T16:07:41.742 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6825/13249", 2022-06-29T16:07:41.742 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6820/13249", 2022-06-29T16:07:41.742 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:07:41.743 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:07:41.743 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:07:41.743 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.743 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.743 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.744 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:07:41.744 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "e4bf1104-ff5c-438b-b608-8cee31c163f6", 2022-06-29T16:07:41.744 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:07:41.744 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:07:41.745 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:07:41.745 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:07:41.745 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:07:41.745 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:07:41.746 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:07:41.746 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 119, 2022-06-29T16:07:41.746 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:07:41.746 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:07:41.747 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:07:41.747 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:41.747 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.747 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:41.747 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6809", 2022-06-29T16:07:41.748 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:07:41.748 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.748 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.749 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:41.749 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6811", 2022-06-29T16:07:41.749 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:07:41.749 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.749 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.750 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.750 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:07:41.750 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:41.750 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.751 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:41.751 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6813", 2022-06-29T16:07:41.751 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:07:41.751 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.752 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.752 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:41.752 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6815", 2022-06-29T16:07:41.752 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:07:41.753 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.753 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.753 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.753 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:07:41.753 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:41.754 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.754 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:41.754 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6824", 2022-06-29T16:07:41.754 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:07:41.755 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.755 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.755 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:41.755 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6827", 2022-06-29T16:07:41.756 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:07:41.756 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.756 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.756 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.757 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:07:41.757 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:41.757 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.757 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:41.758 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6818", 2022-06-29T16:07:41.758 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:07:41.758 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.758 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.759 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:41.759 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6821", 2022-06-29T16:07:41.759 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:07:41.759 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.759 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.760 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.760 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6811/13248", 2022-06-29T16:07:41.760 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6815/13248", 2022-06-29T16:07:41.760 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6827/13248", 2022-06-29T16:07:41.761 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6821/13248", 2022-06-29T16:07:41.761 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:07:41.761 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:07:41.761 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:07:41.761 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.762 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.762 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.762 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:07:41.762 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "8564dd25-27ab-4203-b54e-af6d3aa05857", 2022-06-29T16:07:41.763 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:07:41.763 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:07:41.763 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:07:41.763 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:07:41.764 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:07:41.764 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:07:41.764 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:07:41.764 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 126, 2022-06-29T16:07:41.764 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:07:41.765 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:07:41.765 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:07:41.765 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:41.765 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.766 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:41.766 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6816", 2022-06-29T16:07:41.766 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:07:41.766 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.767 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.767 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:41.767 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6819", 2022-06-29T16:07:41.768 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:07:41.768 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.768 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.768 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.768 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:07:41.769 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:41.769 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.769 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:41.769 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6822", 2022-06-29T16:07:41.770 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:07:41.770 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.770 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.770 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:41.771 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6826", 2022-06-29T16:07:41.771 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:07:41.771 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.771 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.771 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.772 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:07:41.772 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:41.772 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.772 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:41.773 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6830", 2022-06-29T16:07:41.773 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:07:41.773 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.773 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.773 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:41.774 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6831", 2022-06-29T16:07:41.774 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:07:41.774 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.774 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.775 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.775 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:07:41.775 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:41.775 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.776 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:41.776 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6828", 2022-06-29T16:07:41.776 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:07:41.776 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.777 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.777 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:41.777 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6829", 2022-06-29T16:07:41.777 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:07:41.777 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.778 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.778 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.778 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6819/13250", 2022-06-29T16:07:41.779 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6826/13250", 2022-06-29T16:07:41.779 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6831/13250", 2022-06-29T16:07:41.779 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6829/13250", 2022-06-29T16:07:41.779 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:07:41.779 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:07:41.780 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:07:41.780 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.780 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.780 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.781 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:07:41.781 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "bd513a9c-3b19-4314-9897-d300b5c897bb", 2022-06-29T16:07:41.781 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:07:41.781 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:07:41.781 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:07:41.782 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:07:41.782 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:07:41.782 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:07:41.782 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:07:41.783 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 126, 2022-06-29T16:07:41.783 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:07:41.783 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:07:41.783 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:07:41.784 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:41.784 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.784 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:41.784 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6800", 2022-06-29T16:07:41.785 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:07:41.785 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.785 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.785 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:41.786 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6801", 2022-06-29T16:07:41.786 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:07:41.786 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.786 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.786 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.787 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:07:41.787 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:41.787 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.787 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:41.788 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6802", 2022-06-29T16:07:41.788 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:07:41.788 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.788 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.788 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:41.789 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6803", 2022-06-29T16:07:41.789 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:07:41.789 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.790 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.790 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.790 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:07:41.790 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:41.790 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.791 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:41.791 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6806", 2022-06-29T16:07:41.791 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:07:41.791 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.792 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.792 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:41.792 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6807", 2022-06-29T16:07:41.792 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:07:41.792 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.793 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.793 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.793 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:07:41.793 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:41.794 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.794 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:41.794 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6804", 2022-06-29T16:07:41.794 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:07:41.795 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.795 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.795 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:41.795 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6805", 2022-06-29T16:07:41.795 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:07:41.796 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.796 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.796 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.797 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6801/13247", 2022-06-29T16:07:41.797 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6803/13247", 2022-06-29T16:07:41.797 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6807/13247", 2022-06-29T16:07:41.797 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6805/13247", 2022-06-29T16:07:41.798 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:07:41.798 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:07:41.798 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:07:41.798 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:41.798 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.799 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:07:41.799 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_xinfo": [ 2022-06-29T16:07:41.799 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.799 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:07:41.800 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:07:41.800 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:07:41.800 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:07:41.800 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:07:41.800 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:07:41.801 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.891342+0000", 2022-06-29T16:07:41.801 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:07:41.801 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.801 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.802 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:07:41.802 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:07:41.802 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:07:41.802 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:07:41.803 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:07:41.803 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:07:41.803 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.900866+0000", 2022-06-29T16:07:41.803 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:07:41.803 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.804 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.804 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:07:41.804 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:07:41.804 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:07:41.805 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:07:41.805 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:07:41.805 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:07:41.805 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.912755+0000", 2022-06-29T16:07:41.806 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:07:41.806 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.806 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.806 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:07:41.806 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:07:41.807 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:07:41.807 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:07:41.807 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:07:41.807 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:07:41.808 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.864332+0000", 2022-06-29T16:07:41.808 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:07:41.808 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.808 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.809 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:07:41.809 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:07:41.809 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:07:41.809 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:07:41.809 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:07:41.810 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:07:41.810 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.863756+0000", 2022-06-29T16:07:41.810 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:07:41.810 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.811 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.811 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:07:41.811 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:07:41.811 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:07:41.812 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:07:41.812 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:07:41.812 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:07:41.812 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.869041+0000", 2022-06-29T16:07:41.812 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:07:41.813 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.813 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.813 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:07:41.813 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:07:41.814 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:07:41.814 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:07:41.814 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:07:41.814 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:07:41.814 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.895043+0000", 2022-06-29T16:07:41.815 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:07:41.815 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.815 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:41.815 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:07:41.816 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:07:41.816 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:07:41.816 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:07:41.816 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:07:41.817 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:07:41.817 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.889491+0000", 2022-06-29T16:07:41.817 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:07:41.817 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.817 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:07:41.818 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap": [], 2022-06-29T16:07:41.818 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap_items": [], 2022-06-29T16:07:41.818 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_temp": [], 2022-06-29T16:07:41.818 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_temp": [], 2022-06-29T16:07:41.819 INFO:teuthology.orchestra.run.smithi060.stdout: "blacklist": {}, 2022-06-29T16:07:41.819 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profiles": { 2022-06-29T16:07:41.819 INFO:teuthology.orchestra.run.smithi060.stdout: "default": { 2022-06-29T16:07:41.819 INFO:teuthology.orchestra.run.smithi060.stdout: "crush-failure-domain": "osd", 2022-06-29T16:07:41.820 INFO:teuthology.orchestra.run.smithi060.stdout: "k": "2", 2022-06-29T16:07:41.820 INFO:teuthology.orchestra.run.smithi060.stdout: "m": "1", 2022-06-29T16:07:41.820 INFO:teuthology.orchestra.run.smithi060.stdout: "plugin": "jerasure", 2022-06-29T16:07:41.820 INFO:teuthology.orchestra.run.smithi060.stdout: "ruleset-failure-domain": "osd", 2022-06-29T16:07:41.821 INFO:teuthology.orchestra.run.smithi060.stdout: "technique": "reed_sol_van" 2022-06-29T16:07:41.821 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:41.821 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:41.821 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps_queue": [], 2022-06-29T16:07:41.821 INFO:teuthology.orchestra.run.smithi060.stdout: "new_removed_snaps": [], 2022-06-29T16:07:41.822 INFO:teuthology.orchestra.run.smithi060.stdout: "new_purged_snaps": [], 2022-06-29T16:07:41.822 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_node_flags": {}, 2022-06-29T16:07:41.822 INFO:teuthology.orchestra.run.smithi060.stdout: "device_class_flags": {} 2022-06-29T16:07:41.822 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:07:41.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.573+0000 7f9870926700 1 -- 172.21.15.60:0/2383513924 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9850053a70 msgr2=0x7f9850055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:41.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.573+0000 7f9870926700 1 --2- 172.21.15.60:0/2383513924 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9850053a70 0x7f9850055f30 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0x7f985c00b000 tx=0x7f985c009040).stop 2022-06-29T16:07:41.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.573+0000 7f9870926700 1 -- 172.21.15.60:0/2383513924 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f986c0b43b0 msgr2=0x7f986c1409e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:41.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.573+0000 7f9870926700 1 --2- 172.21.15.60:0/2383513924 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f986c0b43b0 0x7f986c1409e0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 rx=0x7f986c0b6e90 tx=0x7f9854007c50).stop 2022-06-29T16:07:41.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.573+0000 7f9870926700 1 -- 172.21.15.60:0/2383513924 shutdown_connections 2022-06-29T16:07:41.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.573+0000 7f9870926700 1 --2- 172.21.15.60:0/2383513924 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9850053a70 0x7f9850055f30 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:41.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.573+0000 7f9870926700 1 --2- 172.21.15.60:0/2383513924 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f986c14b640 0x7f986c0b1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:41.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.573+0000 7f9870926700 1 --2- 172.21.15.60:0/2383513924 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f986c0b95c0 0x7f986c155360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:41.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.573+0000 7f9870926700 1 --2- 172.21.15.60:0/2383513924 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f986c0b43b0 0x7f986c1409e0 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:41.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.573+0000 7f9870926700 1 -- 172.21.15.60:0/2383513924 >> 172.21.15.60:0/2383513924 conn(0x7f986c069a60 msgr2=0x7f986c142b00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:41.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.573+0000 7f9870926700 1 -- 172.21.15.60:0/2383513924 shutdown_connections 2022-06-29T16:07:41.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.573+0000 7f9870926700 1 -- 172.21.15.60:0/2383513924 wait complete. 2022-06-29T16:07:41.828 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:07:41.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.933+0000 7f45f11aa700 1 Processor -- start 2022-06-29T16:07:41.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.933+0000 7f45f11aa700 1 -- start start 2022-06-29T16:07:41.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.933+0000 7f45f11aa700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f45ec0b5f00 0x7f45ec146950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:41.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.933+0000 7f45f11aa700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f45ec0b3ee0 0x7f45ec146e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:41.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.933+0000 7f45f11aa700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45ec0b5830 0x7f45ec05bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:41.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.933+0000 7f45f11aa700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45ec146360 con 0x7f45ec0b3ee0 2022-06-29T16:07:41.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.933+0000 7f45f11aa700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45ec0b0810 con 0x7f45ec0b5f00 2022-06-29T16:07:41.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.933+0000 7f45f11aa700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45ec0b0a70 con 0x7f45ec0b5830 2022-06-29T16:07:41.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.937+0000 7f45eb7fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f45ec0b3ee0 0x7f45ec146e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:41.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.937+0000 7f45eb7fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f45ec0b3ee0 0x7f45ec146e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53310/0 (socket says 172.21.15.60:53310) 2022-06-29T16:07:41.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.937+0000 7f45eb7fe700 1 -- 172.21.15.60:0/3541653599 learned_addr learned my addr 172.21.15.60:0/3541653599 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:41.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.937+0000 7f45f09a9700 1 --2- 172.21.15.60:0/3541653599 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45ec0b5830 0x7f45ec05bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:41.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.937+0000 7f45ebfff700 1 --2- 172.21.15.60:0/3541653599 >> v2:172.21.15.92:3300/0 conn(0x7f45ec0b5f00 0x7f45ec146950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:41.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.937+0000 7f45eb7fe700 1 -- 172.21.15.60:0/3541653599 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45ec0b5830 msgr2=0x7f45ec05bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:41.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.937+0000 7f45eb7fe700 1 --2- 172.21.15.60:0/3541653599 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45ec0b5830 0x7f45ec05bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:41.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.937+0000 7f45eb7fe700 1 -- 172.21.15.60:0/3541653599 >> v2:172.21.15.92:3300/0 conn(0x7f45ec0b5f00 msgr2=0x7f45ec146950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:41.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.937+0000 7f45eb7fe700 1 --2- 172.21.15.60:0/3541653599 >> v2:172.21.15.92:3300/0 conn(0x7f45ec0b5f00 0x7f45ec146950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:41.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.937+0000 7f45eb7fe700 1 -- 172.21.15.60:0/3541653599 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ec0b0d10 con 0x7f45ec0b3ee0 2022-06-29T16:07:41.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.937+0000 7f45ebfff700 1 --2- 172.21.15.60:0/3541653599 >> v2:172.21.15.92:3300/0 conn(0x7f45ec0b5f00 0x7f45ec146950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:41.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.937+0000 7f45eb7fe700 1 --2- 172.21.15.60:0/3541653599 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45ec0b3ee0 0x7f45ec146e40 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 rx=0x7f45e0002f70 tx=0x7f45e0006eb0).ready entity=mon.0 client_cookie=70645b33d7afb490 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:41.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.937+0000 7f45eaffd700 1 -- 172.21.15.60:0/3541653599 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45e000d570 con 0x7f45ec0b3ee0 2022-06-29T16:07:41.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.937+0000 7f45eaffd700 1 -- 172.21.15.60:0/3541653599 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45e0000e90 con 0x7f45ec0b3ee0 2022-06-29T16:07:41.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.937+0000 7f45eaffd700 1 -- 172.21.15.60:0/3541653599 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45e000d900 con 0x7f45ec0b3ee0 2022-06-29T16:07:41.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.937+0000 7f45f11aa700 1 -- 172.21.15.60:0/3541653599 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45ec0b3ee0 msgr2=0x7f45ec146e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:41.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.937+0000 7f45f11aa700 1 --2- 172.21.15.60:0/3541653599 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45ec0b3ee0 0x7f45ec146e40 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 rx=0x7f45e0002f70 tx=0x7f45e0006eb0).stop 2022-06-29T16:07:41.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.937+0000 7f45f11aa700 1 -- 172.21.15.60:0/3541653599 shutdown_connections 2022-06-29T16:07:41.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.937+0000 7f45f11aa700 1 --2- 172.21.15.60:0/3541653599 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45ec0b5830 0x7f45ec05bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:41.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.937+0000 7f45f11aa700 1 --2- 172.21.15.60:0/3541653599 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45ec0b3ee0 0x7f45ec146e40 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:41.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.937+0000 7f45f11aa700 1 --2- 172.21.15.60:0/3541653599 >> v2:172.21.15.92:3300/0 conn(0x7f45ec0b5f00 0x7f45ec146950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:41.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.937+0000 7f45f11aa700 1 -- 172.21.15.60:0/3541653599 >> 172.21.15.60:0/3541653599 conn(0x7f45ec066c60 msgr2=0x7f45ec067070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:41.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.937+0000 7f45f11aa700 1 -- 172.21.15.60:0/3541653599 shutdown_connections 2022-06-29T16:07:41.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.937+0000 7f45f11aa700 1 -- 172.21.15.60:0/3541653599 wait complete. 2022-06-29T16:07:41.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.937+0000 7f45f11aa700 1 Processor -- start 2022-06-29T16:07:41.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.937+0000 7f45f11aa700 1 -- start start 2022-06-29T16:07:41.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.941+0000 7f45f11aa700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45ec0b3ee0 0x7f45ec0ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:41.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.941+0000 7f45f11aa700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45ec0b5830 0x7f45ec0ace90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:41.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.941+0000 7f45f11aa700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45ec0b5f00 0x7f45ec1149a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:41.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.941+0000 7f45f11aa700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ec0b0170 con 0x7f45ec0b3ee0 2022-06-29T16:07:41.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.941+0000 7f45f11aa700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ec112fa0 con 0x7f45ec0b5830 2022-06-29T16:07:41.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.941+0000 7f45f11aa700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45ec0b8a90 con 0x7f45ec0b5f00 2022-06-29T16:07:41.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.941+0000 7f45ebfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45ec0b3ee0 0x7f45ec0ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:41.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.941+0000 7f45ebfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45ec0b3ee0 0x7f45ec0ad4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53316/0 (socket says 172.21.15.60:53316) 2022-06-29T16:07:41.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.941+0000 7f45ebfff700 1 -- 172.21.15.60:0/1044443178 learned_addr learned my addr 172.21.15.60:0/1044443178 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:41.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.941+0000 7f45f09a9700 1 --2- 172.21.15.60:0/1044443178 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45ec0b5f00 0x7f45ec1149a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:41.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.941+0000 7f45eb7fe700 1 --2- 172.21.15.60:0/1044443178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45ec0b5830 0x7f45ec0ace90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:41.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.941+0000 7f45f09a9700 1 -- 172.21.15.60:0/1044443178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45ec0b5830 msgr2=0x7f45ec0ace90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:41.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.941+0000 7f45f09a9700 1 --2- 172.21.15.60:0/1044443178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45ec0b5830 0x7f45ec0ace90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:41.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.941+0000 7f45f09a9700 1 -- 172.21.15.60:0/1044443178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45ec0b3ee0 msgr2=0x7f45ec0ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:41.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.941+0000 7f45f09a9700 1 --2- 172.21.15.60:0/1044443178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45ec0b3ee0 0x7f45ec0ad4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:41.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.941+0000 7f45f09a9700 1 -- 172.21.15.60:0/1044443178 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e000b1e0 con 0x7f45ec0b5f00 2022-06-29T16:07:41.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.941+0000 7f45ebfff700 1 --2- 172.21.15.60:0/1044443178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45ec0b3ee0 0x7f45ec0ad4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:07:41.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.941+0000 7f45f09a9700 1 --2- 172.21.15.60:0/1044443178 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45ec0b5f00 0x7f45ec1149a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f45e400c000 tx=0x7f45e40091d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:41.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.941+0000 7f45e97fa700 1 -- 172.21.15.60:0/1044443178 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45e4012600 con 0x7f45ec0b5f00 2022-06-29T16:07:41.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.941+0000 7f45f11aa700 1 -- 172.21.15.60:0/1044443178 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45ec0af0a0 con 0x7f45ec0b5f00 2022-06-29T16:07:41.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.941+0000 7f45f11aa700 1 -- 172.21.15.60:0/1044443178 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45ec114e90 con 0x7f45ec0b5f00 2022-06-29T16:07:41.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.941+0000 7f45e97fa700 1 -- 172.21.15.60:0/1044443178 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45e4012760 con 0x7f45ec0b5f00 2022-06-29T16:07:41.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.941+0000 7f45e97fa700 1 -- 172.21.15.60:0/1044443178 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45e4012ac0 con 0x7f45ec0b5f00 2022-06-29T16:07:41.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.945+0000 7f45e97fa700 1 -- 172.21.15.60:0/1044443178 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f45e4012c20 con 0x7f45ec0b5f00 2022-06-29T16:07:41.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.945+0000 7f45f11aa700 1 -- 172.21.15.60:0/1044443178 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45c4003410 con 0x7f45ec0b5f00 2022-06-29T16:07:41.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.945+0000 7f45e97fa700 1 --2- 172.21.15.60:0/1044443178 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f45cc053b00 0x7f45cc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:41.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.945+0000 7f45e97fa700 1 -- 172.21.15.60:0/1044443178 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f45e406c240 con 0x7f45ec0b5f00 2022-06-29T16:07:41.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.945+0000 7f45ebfff700 1 --2- 172.21.15.60:0/1044443178 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f45cc053b00 0x7f45cc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:41.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.945+0000 7f45ebfff700 1 --2- 172.21.15.60:0/1044443178 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f45cc053b00 0x7f45cc055fc0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7f45dc008000 tx=0x7f45dc008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:41.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:41.953+0000 7f45e97fa700 1 -- 172.21.15.60:0/1044443178 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f45e4042fd0 con 0x7f45ec0b5f00 2022-06-29T16:07:42.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.121+0000 7f45f11aa700 1 -- 172.21.15.60:0/1044443178 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45c4003c40 con 0x7f45ec0b5f00 2022-06-29T16:07:42.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.121+0000 7f45e97fa700 1 -- 172.21.15.60:0/1044443178 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2142 (secure 0 0 0) 0x7f45e4016090 con 0x7f45ec0b5f00 2022-06-29T16:07:42.127 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:42.127 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":62,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6941,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/3488623391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3488623391},{"type":"v1","addr":"172.21.15.92:6835","nonce":3488623391}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":59}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:07:19.341489+0000","modified":"2022-06-29T16:07:20.345065+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8085},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8085":{"gid":8085,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/3877729627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3877729627},{"type":"v1","addr":"172.21.15.60:6833","nonce":3877729627}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-06-29T16:07:42.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.121+0000 7f45f11aa700 1 -- 172.21.15.60:0/1044443178 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f45cc053b00 msgr2=0x7f45cc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:42.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.121+0000 7f45f11aa700 1 --2- 172.21.15.60:0/1044443178 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f45cc053b00 0x7f45cc055fc0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7f45dc008000 tx=0x7f45dc008040).stop 2022-06-29T16:07:42.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.121+0000 7f45f11aa700 1 -- 172.21.15.60:0/1044443178 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45ec0b5f00 msgr2=0x7f45ec1149a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:42.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.121+0000 7f45f11aa700 1 --2- 172.21.15.60:0/1044443178 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45ec0b5f00 0x7f45ec1149a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f45e400c000 tx=0x7f45e40091d0).stop 2022-06-29T16:07:42.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.125+0000 7f45f11aa700 1 -- 172.21.15.60:0/1044443178 shutdown_connections 2022-06-29T16:07:42.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.125+0000 7f45f11aa700 1 --2- 172.21.15.60:0/1044443178 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f45cc053b00 0x7f45cc055fc0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:42.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.125+0000 7f45f11aa700 1 --2- 172.21.15.60:0/1044443178 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f45ec0b5f00 0x7f45ec1149a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:42.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.125+0000 7f45f11aa700 1 --2- 172.21.15.60:0/1044443178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45ec0b5830 0x7f45ec0ace90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:42.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.125+0000 7f45f11aa700 1 --2- 172.21.15.60:0/1044443178 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f45ec0b3ee0 0x7f45ec0ad4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:42.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.125+0000 7f45f11aa700 1 -- 172.21.15.60:0/1044443178 >> 172.21.15.60:0/1044443178 conn(0x7f45ec066c60 msgr2=0x7f45ec0b4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:42.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.125+0000 7f45f11aa700 1 -- 172.21.15.60:0/1044443178 shutdown_connections 2022-06-29T16:07:42.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.125+0000 7f45f11aa700 1 -- 172.21.15.60:0/1044443178 wait complete. 2022-06-29T16:07:42.131 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 62 2022-06-29T16:07:42.140 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-06-29T16:07:42.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.237+0000 7fcc61dbf700 1 Processor -- start 2022-06-29T16:07:42.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.237+0000 7fcc61dbf700 1 -- start start 2022-06-29T16:07:42.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.237+0000 7fcc61dbf700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcc5c0b5330 0x7fcc5c147320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:42.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.237+0000 7fcc61dbf700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc5c0aad10 0x7fcc5c14aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:42.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.237+0000 7fcc61dbf700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc5c147810 con 0x7fcc5c0b5c00 2022-06-29T16:07:42.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.237+0000 7fcc61dbf700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc5c0b5710 con 0x7fcc5c0b5330 2022-06-29T16:07:42.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.237+0000 7fcc61dbf700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5c0afb80 con 0x7fcc5c0aad10 2022-06-29T16:07:42.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.237+0000 7fcc5bfff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcc5c0b5c00 0x7fcc5c0b1a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33182/0 (socket says 172.21.15.60:33182) 2022-06-29T16:07:42.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.237+0000 7fcc5bfff700 1 -- 172.21.15.60:0/1996769522 learned_addr learned my addr 172.21.15.60:0/1996769522 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:42.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.237+0000 7fcc5affd700 1 --2- 172.21.15.60:0/1996769522 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc5c0aad10 0x7fcc5c14aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:42.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.237+0000 7fcc5b7fe700 1 --2- 172.21.15.60:0/1996769522 >> v2:172.21.15.92:3300/0 conn(0x7fcc5c0b5330 0x7fcc5c147320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:42.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.237+0000 7fcc5a7fc700 1 -- 172.21.15.60:0/1996769522 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4049800767 0 0) 0x7fcc54001490 con 0x7fcc5c0b5c00 2022-06-29T16:07:42.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.237+0000 7fcc5a7fc700 1 -- 172.21.15.60:0/1996769522 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc48003260 con 0x7fcc5c0b5c00 2022-06-29T16:07:42.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.237+0000 7fcc5affd700 1 -- 172.21.15.60:0/1996769522 >> v2:172.21.15.92:3300/0 conn(0x7fcc5c0b5330 msgr2=0x7fcc5c147320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:42.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.237+0000 7fcc5affd700 1 --2- 172.21.15.60:0/1996769522 >> v2:172.21.15.92:3300/0 conn(0x7fcc5c0b5330 0x7fcc5c147320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:42.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.237+0000 7fcc5a7fc700 1 -- 172.21.15.60:0/1996769522 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1047664157 0 0) 0x7fcc48003260 con 0x7fcc5c0b5c00 2022-06-29T16:07:42.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.237+0000 7fcc5affd700 1 -- 172.21.15.60:0/1996769522 >> v1:172.21.15.60:6789/0 conn(0x7fcc5c0b5c00 legacy=0x7fcc5c0b1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:42.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.237+0000 7fcc5affd700 1 -- 172.21.15.60:0/1996769522 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc5c14aa10 con 0x7fcc5c0aad10 2022-06-29T16:07:42.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.237+0000 7fcc5affd700 1 --2- 172.21.15.60:0/1996769522 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc5c0aad10 0x7fcc5c14aeb0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fcc5c14ad90 tx=0x7fcc4c006df0).ready entity=mon.2 client_cookie=a68c6c6c801594e4 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:42.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.237+0000 7fcc5a7fc700 1 -- 172.21.15.60:0/1996769522 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc54001b70 con 0x7fcc5c0aad10 2022-06-29T16:07:42.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.237+0000 7fcc5a7fc700 1 -- 172.21.15.60:0/1996769522 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc4c000e90 con 0x7fcc5c0aad10 2022-06-29T16:07:42.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.241+0000 7fcc5a7fc700 1 -- 172.21.15.60:0/1996769522 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc4c012460 con 0x7fcc5c0aad10 2022-06-29T16:07:42.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.241+0000 7fcc61dbf700 1 -- 172.21.15.60:0/1996769522 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc5c0aad10 msgr2=0x7fcc5c14aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:42.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.241+0000 7fcc61dbf700 1 --2- 172.21.15.60:0/1996769522 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc5c0aad10 0x7fcc5c14aeb0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fcc5c14ad90 tx=0x7fcc4c006df0).stop 2022-06-29T16:07:42.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.241+0000 7fcc61dbf700 1 -- 172.21.15.60:0/1996769522 shutdown_connections 2022-06-29T16:07:42.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.241+0000 7fcc61dbf700 1 --2- 172.21.15.60:0/1996769522 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc5c0aad10 0x7fcc5c14aeb0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:42.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.241+0000 7fcc61dbf700 1 --2- 172.21.15.60:0/1996769522 >> v2:172.21.15.92:3300/0 conn(0x7fcc5c0b5330 0x7fcc5c147320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:42.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.241+0000 7fcc61dbf700 1 -- 172.21.15.60:0/1996769522 >> 172.21.15.60:0/1996769522 conn(0x7fcc5c062910 msgr2=0x7fcc5c062d20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:42.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.241+0000 7fcc61dbf700 1 -- 172.21.15.60:0/1996769522 shutdown_connections 2022-06-29T16:07:42.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.241+0000 7fcc61dbf700 1 -- 172.21.15.60:0/1996769522 wait complete. 2022-06-29T16:07:42.254 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.241+0000 7fcc61dbf700 1 Processor -- start 2022-06-29T16:07:42.254 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.241+0000 7fcc61dbf700 1 -- start start 2022-06-29T16:07:42.254 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.241+0000 7fcc61dbf700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc5c0aad10 0x7fcc5c0a1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:42.254 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.241+0000 7fcc61dbf700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc5c0b5330 0x7fcc5c09e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:42.255 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.241+0000 7fcc61dbf700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc5c0b5c00 0x7fcc5c09fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:42.255 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.241+0000 7fcc61dbf700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5c14ec90 con 0x7fcc5c0b5c00 2022-06-29T16:07:42.255 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.241+0000 7fcc61dbf700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5c14e750 con 0x7fcc5c0b5330 2022-06-29T16:07:42.255 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.241+0000 7fcc61dbf700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5c14e4b0 con 0x7fcc5c0aad10 2022-06-29T16:07:42.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.241+0000 7fcc5bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc5c0b5c00 0x7fcc5c09fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:42.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.241+0000 7fcc5bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc5c0b5c00 0x7fcc5c09fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53332/0 (socket says 172.21.15.60:53332) 2022-06-29T16:07:42.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.241+0000 7fcc5bfff700 1 -- 172.21.15.60:0/3668705122 learned_addr learned my addr 172.21.15.60:0/3668705122 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:42.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.241+0000 7fcc5affd700 1 --2- 172.21.15.60:0/3668705122 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc5c0b5330 0x7fcc5c09e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:42.256 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.241+0000 7fcc5b7fe700 1 --2- 172.21.15.60:0/3668705122 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc5c0aad10 0x7fcc5c0a1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:42.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.245+0000 7fcc5affd700 1 -- 172.21.15.60:0/3668705122 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc5c0aad10 msgr2=0x7fcc5c0a1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:42.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.245+0000 7fcc5affd700 1 --2- 172.21.15.60:0/3668705122 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc5c0aad10 0x7fcc5c0a1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:42.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.245+0000 7fcc5affd700 1 -- 172.21.15.60:0/3668705122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc5c0b5c00 msgr2=0x7fcc5c09fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:42.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.245+0000 7fcc5affd700 1 --2- 172.21.15.60:0/3668705122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc5c0b5c00 0x7fcc5c09fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:42.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.245+0000 7fcc5affd700 1 -- 172.21.15.60:0/3668705122 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4c00b180 con 0x7fcc5c0b5330 2022-06-29T16:07:42.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.245+0000 7fcc5bfff700 1 --2- 172.21.15.60:0/3668705122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc5c0b5c00 0x7fcc5c09fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:07:42.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.245+0000 7fcc5b7fe700 1 --2- 172.21.15.60:0/3668705122 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc5c0aad10 0x7fcc5c0a1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:42.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.245+0000 7fcc5affd700 1 --2- 172.21.15.60:0/3668705122 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc5c0b5330 0x7fcc5c09e430 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7fcc5c09f920 tx=0x7fcc4c014000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:42.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.245+0000 7fcc58ff9700 1 -- 172.21.15.60:0/3668705122 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc4c016070 con 0x7fcc5c0b5330 2022-06-29T16:07:42.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.245+0000 7fcc58ff9700 1 -- 172.21.15.60:0/3668705122 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc4c013e40 con 0x7fcc5c0b5330 2022-06-29T16:07:42.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.245+0000 7fcc61dbf700 1 -- 172.21.15.60:0/3668705122 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc5c149c10 con 0x7fcc5c0b5330 2022-06-29T16:07:42.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.245+0000 7fcc58ff9700 1 -- 172.21.15.60:0/3668705122 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc4c002f50 con 0x7fcc5c0b5330 2022-06-29T16:07:42.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.245+0000 7fcc61dbf700 1 -- 172.21.15.60:0/3668705122 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc5c14a250 con 0x7fcc5c0b5330 2022-06-29T16:07:42.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.245+0000 7fcc61dbf700 1 -- 172.21.15.60:0/3668705122 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc5c14d090 con 0x7fcc5c0b5330 2022-06-29T16:07:42.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.245+0000 7fcc58ff9700 1 -- 172.21.15.60:0/3668705122 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcc4c00f030 con 0x7fcc5c0b5330 2022-06-29T16:07:42.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.245+0000 7fcc58ff9700 1 --2- 172.21.15.60:0/3668705122 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcc3c053b00 0x7fcc3c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:42.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.245+0000 7fcc58ff9700 1 -- 172.21.15.60:0/3668705122 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcc4c06cc10 con 0x7fcc5c0b5330 2022-06-29T16:07:42.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.249+0000 7fcc5b7fe700 1 --2- 172.21.15.60:0/3668705122 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcc3c053b00 0x7fcc3c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:42.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.249+0000 7fcc5b7fe700 1 --2- 172.21.15.60:0/3668705122 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcc3c053b00 0x7fcc3c055fc0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0x7fcc5000b000 tx=0x7fcc50008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:42.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.253+0000 7fcc58ff9700 1 -- 172.21.15.60:0/3668705122 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcc5c14d090 con 0x7fcc5c0b5330 2022-06-29T16:07:42.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:42.417+0000 7fcc61dbf700 1 -- 172.21.15.60:0/3668705122 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fcc5c0573d0 con 0x7fcc5c0b5330 2022-06-29T16:07:43.174 INFO:tasks.ceph.mon.a.smithi060.stderr:2022-06-29T16:07:43.169+0000 7fcc1a376700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-06-29T16:07:43.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.185+0000 7fcc58ff9700 1 -- 172.21.15.60:0/3668705122 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v63) v1 ==== 161+0+0 (secure 0 0 0) 0x7fcc4c023070 con 0x7fcc5c0b5330 2022-06-29T16:07:43.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.189+0000 7fcc61dbf700 1 -- 172.21.15.60:0/3668705122 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcc3c053b00 msgr2=0x7fcc3c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:43.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.189+0000 7fcc61dbf700 1 --2- 172.21.15.60:0/3668705122 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcc3c053b00 0x7fcc3c055fc0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0x7fcc5000b000 tx=0x7fcc50008040).stop 2022-06-29T16:07:43.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.189+0000 7fcc61dbf700 1 -- 172.21.15.60:0/3668705122 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc5c0b5330 msgr2=0x7fcc5c09e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:43.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.189+0000 7fcc61dbf700 1 --2- 172.21.15.60:0/3668705122 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc5c0b5330 0x7fcc5c09e430 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7fcc5c09f920 tx=0x7fcc4c014000).stop 2022-06-29T16:07:43.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.189+0000 7fcc61dbf700 1 -- 172.21.15.60:0/3668705122 shutdown_connections 2022-06-29T16:07:43.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.189+0000 7fcc61dbf700 1 --2- 172.21.15.60:0/3668705122 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcc3c053b00 0x7fcc3c055fc0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:43.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.189+0000 7fcc61dbf700 1 --2- 172.21.15.60:0/3668705122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc5c0b5c00 0x7fcc5c09fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:43.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.189+0000 7fcc61dbf700 1 --2- 172.21.15.60:0/3668705122 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc5c0b5330 0x7fcc5c09e430 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:43.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.189+0000 7fcc61dbf700 1 --2- 172.21.15.60:0/3668705122 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcc5c0aad10 0x7fcc5c0a1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:43.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.189+0000 7fcc61dbf700 1 -- 172.21.15.60:0/3668705122 >> 172.21.15.60:0/3668705122 conn(0x7fcc5c062910 msgr2=0x7fcc5c144a80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:43.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.189+0000 7fcc61dbf700 1 -- 172.21.15.60:0/3668705122 shutdown_connections 2022-06-29T16:07:43.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.189+0000 7fcc61dbf700 1 -- 172.21.15.60:0/3668705122 wait complete. 2022-06-29T16:07:43.198 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-06-29T16:07:43.211 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:07:43.223 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2022-06-29T16:07:43.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.305+0000 7fe679824700 1 Processor -- start 2022-06-29T16:07:43.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.305+0000 7fe679824700 1 -- start start 2022-06-29T16:07:43.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.305+0000 7fe679824700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6740b5130 0x7fe6740af270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:43.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.305+0000 7fe679824700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe674146270 0x7fe6741466d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:43.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.305+0000 7fe679824700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6740af0b0 con 0x7fe674147200 2022-06-29T16:07:43.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.305+0000 7fe679824700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6740b5510 con 0x7fe674146270 2022-06-29T16:07:43.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.305+0000 7fe679824700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6740aee10 con 0x7fe6740b5130 2022-06-29T16:07:43.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.305+0000 7fe678822700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6740b5130 0x7fe6740af270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:43.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.305+0000 7fe673fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe674146270 0x7fe6741466d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:43.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.305+0000 7fe678822700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6740b5130 0x7fe6740af270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:38212/0 (socket says 172.21.15.60:38212) 2022-06-29T16:07:43.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.305+0000 7fe678822700 1 -- 172.21.15.60:0/1134751188 learned_addr learned my addr 172.21.15.60:0/1134751188 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:43.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.305+0000 7fe673fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe674146270 0x7fe6741466d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:59384/0 (socket says 172.21.15.60:59384) 2022-06-29T16:07:43.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.305+0000 7fe679023700 1 --1- 172.21.15.60:0/1134751188 >> v1:172.21.15.92:6789/0 conn(0x7fe674147200 0x7fe674147650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:37454/0 (socket says 172.21.15.60:37454) 2022-06-29T16:07:43.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.305+0000 7fe678822700 1 -- 172.21.15.60:0/1134751188 >> v2:172.21.15.92:3300/0 conn(0x7fe674146270 msgr2=0x7fe6741466d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:43.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.305+0000 7fe678822700 1 --2- 172.21.15.60:0/1134751188 >> v2:172.21.15.92:3300/0 conn(0x7fe674146270 0x7fe6741466d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:43.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.305+0000 7fe678822700 1 -- 172.21.15.60:0/1134751188 >> v1:172.21.15.92:6789/0 conn(0x7fe674147200 legacy=0x7fe674147650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:43.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.305+0000 7fe678822700 1 -- 172.21.15.60:0/1134751188 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe674147d60 con 0x7fe6740b5130 2022-06-29T16:07:43.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.305+0000 7fe678822700 1 --2- 172.21.15.60:0/1134751188 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6740b5130 0x7fe6740af270 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fe66c0032f0 tx=0x7fe66c009290).ready entity=mon.2 client_cookie=39abc79292c8664d server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:43.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.305+0000 7fe6737fe700 1 -- 172.21.15.60:0/1134751188 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe66c00b8a0 con 0x7fe6740b5130 2022-06-29T16:07:43.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.305+0000 7fe6737fe700 1 -- 172.21.15.60:0/1134751188 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe66c00ba00 con 0x7fe6740b5130 2022-06-29T16:07:43.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.305+0000 7fe6737fe700 1 -- 172.21.15.60:0/1134751188 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe66c00bb70 con 0x7fe6740b5130 2022-06-29T16:07:43.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.309+0000 7fe679824700 1 -- 172.21.15.60:0/1134751188 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6740b5130 msgr2=0x7fe6740af270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:43.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.309+0000 7fe679824700 1 --2- 172.21.15.60:0/1134751188 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6740b5130 0x7fe6740af270 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fe66c0032f0 tx=0x7fe66c009290).stop 2022-06-29T16:07:43.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.309+0000 7fe679824700 1 -- 172.21.15.60:0/1134751188 shutdown_connections 2022-06-29T16:07:43.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.309+0000 7fe679824700 1 --2- 172.21.15.60:0/1134751188 >> v2:172.21.15.92:3300/0 conn(0x7fe674146270 0x7fe6741466d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:43.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.309+0000 7fe679824700 1 --2- 172.21.15.60:0/1134751188 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6740b5130 0x7fe6740af270 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:43.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.309+0000 7fe679824700 1 -- 172.21.15.60:0/1134751188 >> 172.21.15.60:0/1134751188 conn(0x7fe6740627a0 msgr2=0x7fe674062bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:43.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.309+0000 7fe679824700 1 -- 172.21.15.60:0/1134751188 shutdown_connections 2022-06-29T16:07:43.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.309+0000 7fe679824700 1 -- 172.21.15.60:0/1134751188 wait complete. 2022-06-29T16:07:43.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.309+0000 7fe679824700 1 Processor -- start 2022-06-29T16:07:43.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.309+0000 7fe679824700 1 -- start start 2022-06-29T16:07:43.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.309+0000 7fe679824700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6740b5130 0x7fe674067050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:43.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.309+0000 7fe679824700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe674146270 0x7fe674148330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:43.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.309+0000 7fe679824700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe674147200 0x7fe674148820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:43.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.309+0000 7fe679824700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67410e4f0 con 0x7fe6740b5130 2022-06-29T16:07:43.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.309+0000 7fe679824700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe674065f90 con 0x7fe674146270 2022-06-29T16:07:43.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.309+0000 7fe679824700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe67410e250 con 0x7fe674147200 2022-06-29T16:07:43.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.309+0000 7fe678822700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6740b5130 0x7fe674067050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:43.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.309+0000 7fe678822700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6740b5130 0x7fe674067050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53356/0 (socket says 172.21.15.60:53356) 2022-06-29T16:07:43.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.309+0000 7fe678822700 1 -- 172.21.15.60:0/1521015819 learned_addr learned my addr 172.21.15.60:0/1521015819 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:43.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.309+0000 7fe673fff700 1 --2- 172.21.15.60:0/1521015819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe674146270 0x7fe674148330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:43.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.309+0000 7fe679023700 1 --2- 172.21.15.60:0/1521015819 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe674147200 0x7fe674148820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:43.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.309+0000 7fe678822700 1 -- 172.21.15.60:0/1521015819 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe674147200 msgr2=0x7fe674148820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:43.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.309+0000 7fe678822700 1 --2- 172.21.15.60:0/1521015819 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe674147200 0x7fe674148820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:43.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.309+0000 7fe678822700 1 -- 172.21.15.60:0/1521015819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe674146270 msgr2=0x7fe674148330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:43.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.309+0000 7fe678822700 1 --2- 172.21.15.60:0/1521015819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe674146270 0x7fe674148330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:43.326 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.309+0000 7fe678822700 1 -- 172.21.15.60:0/1521015819 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe66c009530 con 0x7fe6740b5130 2022-06-29T16:07:43.326 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.309+0000 7fe678822700 1 --2- 172.21.15.60:0/1521015819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6740b5130 0x7fe674067050 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 rx=0x7fe66c0032f0 tx=0x7fe66c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:43.326 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.309+0000 7fe671ffb700 1 -- 172.21.15.60:0/1521015819 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe66c021630 con 0x7fe6740b5130 2022-06-29T16:07:43.326 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.309+0000 7fe671ffb700 1 -- 172.21.15.60:0/1521015819 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe66c021790 con 0x7fe6740b5130 2022-06-29T16:07:43.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.309+0000 7fe679824700 1 -- 172.21.15.60:0/1521015819 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe674149cc0 con 0x7fe6740b5130 2022-06-29T16:07:43.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.309+0000 7fe671ffb700 1 -- 172.21.15.60:0/1521015819 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe66c021af0 con 0x7fe6740b5130 2022-06-29T16:07:43.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.309+0000 7fe679824700 1 -- 172.21.15.60:0/1521015819 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6741571d0 con 0x7fe6740b5130 2022-06-29T16:07:43.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.313+0000 7fe679824700 1 -- 172.21.15.60:0/1521015819 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6740573d0 con 0x7fe6740b5130 2022-06-29T16:07:43.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.313+0000 7fe671ffb700 1 -- 172.21.15.60:0/1521015819 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe66c027020 con 0x7fe6740b5130 2022-06-29T16:07:43.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.313+0000 7fe671ffb700 1 --2- 172.21.15.60:0/1521015819 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe658053a70 0x7fe658055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:43.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.313+0000 7fe671ffb700 1 -- 172.21.15.60:0/1521015819 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fe66c06d7c0 con 0x7fe6740b5130 2022-06-29T16:07:43.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.313+0000 7fe673fff700 1 --2- 172.21.15.60:0/1521015819 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe658053a70 0x7fe658055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:43.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.313+0000 7fe673fff700 1 --2- 172.21.15.60:0/1521015819 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe658053a70 0x7fe658055f30 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0x7fe65c00b000 tx=0x7fe65c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:43.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.321+0000 7fe671ffb700 1 -- 172.21.15.60:0/1521015819 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe66c0445f0 con 0x7fe6740b5130 2022-06-29T16:07:43.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.477+0000 7fe679824700 1 -- 172.21.15.60:0/1521015819 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe67414ad40 con 0x7fe6740b5130 2022-06-29T16:07:43.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.481+0000 7fe671ffb700 1 -- 172.21.15.60:0/1521015819 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+1782 (secure 0 0 0) 0x7fe66c015070 con 0x7fe6740b5130 2022-06-29T16:07:43.485 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:43.485 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":63,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6941,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/3488623391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3488623391},{"type":"v1","addr":"172.21.15.92:6835","nonce":3488623391}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":59}],"filesystems":[{"mdsmap":{"epoch":63,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:07:19.341489+0000","modified":"2022-06-29T16:07:43.171627+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":129,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-06-29T16:07:43.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.481+0000 7fe679824700 1 -- 172.21.15.60:0/1521015819 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe658053a70 msgr2=0x7fe658055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:43.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.481+0000 7fe679824700 1 --2- 172.21.15.60:0/1521015819 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe658053a70 0x7fe658055f30 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0x7fe65c00b000 tx=0x7fe65c008040).stop 2022-06-29T16:07:43.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.481+0000 7fe679824700 1 -- 172.21.15.60:0/1521015819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6740b5130 msgr2=0x7fe674067050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:43.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.481+0000 7fe679824700 1 --2- 172.21.15.60:0/1521015819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6740b5130 0x7fe674067050 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 rx=0x7fe66c0032f0 tx=0x7fe66c016000).stop 2022-06-29T16:07:43.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.481+0000 7fe679824700 1 -- 172.21.15.60:0/1521015819 shutdown_connections 2022-06-29T16:07:43.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.481+0000 7fe679824700 1 --2- 172.21.15.60:0/1521015819 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe658053a70 0x7fe658055f30 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:43.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.481+0000 7fe679824700 1 --2- 172.21.15.60:0/1521015819 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe674147200 0x7fe674148820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:43.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.481+0000 7fe679824700 1 --2- 172.21.15.60:0/1521015819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe674146270 0x7fe674148330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:43.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.481+0000 7fe679824700 1 --2- 172.21.15.60:0/1521015819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6740b5130 0x7fe674067050 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:43.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.481+0000 7fe679824700 1 -- 172.21.15.60:0/1521015819 >> 172.21.15.60:0/1521015819 conn(0x7fe6740627a0 msgr2=0x7fe6740ae2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:43.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.481+0000 7fe679824700 1 -- 172.21.15.60:0/1521015819 shutdown_connections 2022-06-29T16:07:43.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.481+0000 7fe679824700 1 -- 172.21.15.60:0/1521015819 wait complete. 2022-06-29T16:07:43.491 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 63 2022-06-29T16:07:43.504 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-06-29T16:07:43.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.605+0000 7f41e8e82700 1 Processor -- start 2022-06-29T16:07:43.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.605+0000 7f41e8e82700 1 -- start start 2022-06-29T16:07:43.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.605+0000 7f41e8e82700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f41e4143b60 0x7f41e4142b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:43.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.605+0000 7f41e8e82700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f41e41468f0 0x7f41e4144ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:43.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.605+0000 7f41e8e82700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41e40b9b80 con 0x7f41e40b97a0 2022-06-29T16:07:43.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.605+0000 7f41e8e82700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41e41440e0 con 0x7f41e4143b60 2022-06-29T16:07:43.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.605+0000 7f41e8e82700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41e4144380 con 0x7f41e41468f0 2022-06-29T16:07:43.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.605+0000 7f41e1d9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f41e4143b60 0x7f41e4142b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:43.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.605+0000 7f41e1d9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f41e4143b60 0x7f41e4142b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53366/0 (socket says 172.21.15.60:53366) 2022-06-29T16:07:43.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.605+0000 7f41e1d9b700 1 -- 172.21.15.60:0/1840698130 learned_addr learned my addr 172.21.15.60:0/1840698130 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:43.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.605+0000 7f41e2d9d700 1 --2- 172.21.15.60:0/1840698130 >> v2:172.21.15.92:3300/0 conn(0x7f41e41468f0 0x7f41e4144ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:43.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.605+0000 7f41e1d9b700 1 -- 172.21.15.60:0/1840698130 >> v2:172.21.15.92:3300/0 conn(0x7f41e41468f0 msgr2=0x7f41e4144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:43.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.605+0000 7f41e1d9b700 1 --2- 172.21.15.60:0/1840698130 >> v2:172.21.15.92:3300/0 conn(0x7f41e41468f0 0x7f41e4144ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:43.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.605+0000 7f41e1d9b700 1 -- 172.21.15.60:0/1840698130 >> v1:172.21.15.92:6789/0 conn(0x7f41e40b97a0 legacy=0x7f41e41461e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:43.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.605+0000 7f41e1d9b700 1 -- 172.21.15.60:0/1840698130 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e4144620 con 0x7f41e4143b60 2022-06-29T16:07:43.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.605+0000 7f41e1d9b700 1 --2- 172.21.15.60:0/1840698130 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41e4143b60 0x7f41e4142b80 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 rx=0x7f41d4002f70 tx=0x7f41d4006eb0).ready entity=mon.0 client_cookie=c78df5dad3db3dd7 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:43.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.605+0000 7f41e159a700 1 -- 172.21.15.60:0/1840698130 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f41d400d610 con 0x7f41e4143b60 2022-06-29T16:07:43.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.605+0000 7f41e159a700 1 -- 172.21.15.60:0/1840698130 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d4000e90 con 0x7f41e4143b60 2022-06-29T16:07:43.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.605+0000 7f41e159a700 1 -- 172.21.15.60:0/1840698130 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f41d400d9a0 con 0x7f41e4143b60 2022-06-29T16:07:43.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.609+0000 7f41e8e82700 1 -- 172.21.15.60:0/1840698130 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41e4143b60 msgr2=0x7f41e4142b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:43.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.609+0000 7f41e8e82700 1 --2- 172.21.15.60:0/1840698130 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41e4143b60 0x7f41e4142b80 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 rx=0x7f41d4002f70 tx=0x7f41d4006eb0).stop 2022-06-29T16:07:43.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.609+0000 7f41e8e82700 1 -- 172.21.15.60:0/1840698130 shutdown_connections 2022-06-29T16:07:43.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.609+0000 7f41e8e82700 1 --2- 172.21.15.60:0/1840698130 >> v2:172.21.15.92:3300/0 conn(0x7f41e41468f0 0x7f41e4144ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:43.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.609+0000 7f41e8e82700 1 --2- 172.21.15.60:0/1840698130 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41e4143b60 0x7f41e4142b80 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:43.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.609+0000 7f41e8e82700 1 -- 172.21.15.60:0/1840698130 >> 172.21.15.60:0/1840698130 conn(0x7f41e405f930 msgr2=0x7f41e4066c60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:43.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.609+0000 7f41e8e82700 1 -- 172.21.15.60:0/1840698130 shutdown_connections 2022-06-29T16:07:43.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.609+0000 7f41e8e82700 1 -- 172.21.15.60:0/1840698130 wait complete. 2022-06-29T16:07:43.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.609+0000 7f41e8e82700 1 Processor -- start 2022-06-29T16:07:43.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.609+0000 7f41e8e82700 1 -- start start 2022-06-29T16:07:43.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.613+0000 7f41e8e82700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f41e40b97a0 0x7f41e40b0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:43.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.613+0000 7f41e8e82700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f41e4143b60 0x7f41e41126c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:43.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.613+0000 7f41e8e82700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41e41468f0 0x7f41e4152700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:43.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.613+0000 7f41e8e82700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e4151dd0 con 0x7f41e41468f0 2022-06-29T16:07:43.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.613+0000 7f41e8e82700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e4133a60 con 0x7f41e40b97a0 2022-06-29T16:07:43.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.613+0000 7f41e8e82700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41e40b30c0 con 0x7f41e4143b60 2022-06-29T16:07:43.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.613+0000 7f41e2d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41e41468f0 0x7f41e4152700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:43.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.613+0000 7f41e259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f41e40b97a0 0x7f41e40b0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:43.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.613+0000 7f41e1d9b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f41e4143b60 0x7f41e41126c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:43.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.613+0000 7f41e259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f41e40b97a0 0x7f41e40b0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:59400/0 (socket says 172.21.15.60:59400) 2022-06-29T16:07:43.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.613+0000 7f41e259c700 1 -- 172.21.15.60:0/2777142831 learned_addr learned my addr 172.21.15.60:0/2777142831 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:43.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.613+0000 7f41e2d9d700 1 -- 172.21.15.60:0/2777142831 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f41e4143b60 msgr2=0x7f41e41126c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:43.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.613+0000 7f41e2d9d700 1 --2- 172.21.15.60:0/2777142831 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f41e4143b60 0x7f41e41126c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:43.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.613+0000 7f41e2d9d700 1 -- 172.21.15.60:0/2777142831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f41e40b97a0 msgr2=0x7f41e40b0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:43.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.613+0000 7f41e2d9d700 1 --2- 172.21.15.60:0/2777142831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f41e40b97a0 0x7f41e40b0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:43.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.613+0000 7f41e2d9d700 1 -- 172.21.15.60:0/2777142831 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d400b1e0 con 0x7f41e41468f0 2022-06-29T16:07:43.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.613+0000 7f41e2d9d700 1 --2- 172.21.15.60:0/2777142831 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41e41468f0 0x7f41e4152700 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 rx=0x7f41dc00a000 tx=0x7f41dc008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:43.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.613+0000 7f41d37fe700 1 -- 172.21.15.60:0/2777142831 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f41dc011780 con 0x7f41e41468f0 2022-06-29T16:07:43.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.613+0000 7f41d37fe700 1 -- 172.21.15.60:0/2777142831 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41dc0118e0 con 0x7f41e41468f0 2022-06-29T16:07:43.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.613+0000 7f41e8e82700 1 -- 172.21.15.60:0/2777142831 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41e4151fc0 con 0x7f41e41468f0 2022-06-29T16:07:43.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.613+0000 7f41d37fe700 1 -- 172.21.15.60:0/2777142831 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f41dc011c40 con 0x7f41e41468f0 2022-06-29T16:07:43.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.613+0000 7f41e8e82700 1 -- 172.21.15.60:0/2777142831 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41e4112bb0 con 0x7f41e41468f0 2022-06-29T16:07:43.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.617+0000 7f41e8e82700 1 -- 172.21.15.60:0/2777142831 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41e40573d0 con 0x7f41e41468f0 2022-06-29T16:07:43.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.617+0000 7f41d37fe700 1 -- 172.21.15.60:0/2777142831 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f41dc012420 con 0x7f41e41468f0 2022-06-29T16:07:43.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.617+0000 7f41d37fe700 1 --2- 172.21.15.60:0/2777142831 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f41c0053a70 0x7f41c0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:43.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.617+0000 7f41d37fe700 1 -- 172.21.15.60:0/2777142831 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f41dc06c040 con 0x7f41e41468f0 2022-06-29T16:07:43.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.617+0000 7f41e259c700 1 --2- 172.21.15.60:0/2777142831 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f41c0053a70 0x7f41c0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:43.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.617+0000 7f41e259c700 1 --2- 172.21.15.60:0/2777142831 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f41c0053a70 0x7f41c0055f30 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0x7f41d800d000 tx=0x7f41d8007c50).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:43.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.625+0000 7f41d37fe700 1 -- 172.21.15.60:0/2777142831 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f41dc042da0 con 0x7f41e41468f0 2022-06-29T16:07:43.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:43.789+0000 7f41e8e82700 1 -- 172.21.15.60:0/2777142831 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f41e4152460 con 0x7f41e41468f0 2022-06-29T16:07:44.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.189+0000 7f41d37fe700 1 -- 172.21.15.60:0/2777142831 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v64) v1 ==== 104+0+0 (secure 0 0 0) 0x7f41dc0463c0 con 0x7f41e41468f0 2022-06-29T16:07:44.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.193+0000 7f41e8e82700 1 -- 172.21.15.60:0/2777142831 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f41c0053a70 msgr2=0x7f41c0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:44.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.193+0000 7f41e8e82700 1 --2- 172.21.15.60:0/2777142831 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f41c0053a70 0x7f41c0055f30 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0x7f41d800d000 tx=0x7f41d8007c50).stop 2022-06-29T16:07:44.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.193+0000 7f41e8e82700 1 -- 172.21.15.60:0/2777142831 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41e41468f0 msgr2=0x7f41e4152700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:44.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.193+0000 7f41e8e82700 1 --2- 172.21.15.60:0/2777142831 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41e41468f0 0x7f41e4152700 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 rx=0x7f41dc00a000 tx=0x7f41dc008040).stop 2022-06-29T16:07:44.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.193+0000 7f41e8e82700 1 -- 172.21.15.60:0/2777142831 shutdown_connections 2022-06-29T16:07:44.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.193+0000 7f41e8e82700 1 --2- 172.21.15.60:0/2777142831 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f41c0053a70 0x7f41c0055f30 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:44.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.193+0000 7f41e8e82700 1 --2- 172.21.15.60:0/2777142831 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41e41468f0 0x7f41e4152700 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:44.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.193+0000 7f41e8e82700 1 --2- 172.21.15.60:0/2777142831 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f41e4143b60 0x7f41e41126c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:44.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.193+0000 7f41e8e82700 1 --2- 172.21.15.60:0/2777142831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f41e40b97a0 0x7f41e40b0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:44.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.193+0000 7f41e8e82700 1 -- 172.21.15.60:0/2777142831 >> 172.21.15.60:0/2777142831 conn(0x7f41e405f930 msgr2=0x7f41e40b80d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:44.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.193+0000 7f41e8e82700 1 -- 172.21.15.60:0/2777142831 shutdown_connections 2022-06-29T16:07:44.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.193+0000 7f41e8e82700 1 -- 172.21.15.60:0/2777142831 wait complete. 2022-06-29T16:07:44.212 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-06-29T16:07:44.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.317+0000 7f024d3a7700 1 Processor -- start 2022-06-29T16:07:44.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.317+0000 7f024d3a7700 1 -- start start 2022-06-29T16:07:44.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.321+0000 7f024d3a7700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f02480b56c0 0x7f02480b5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:44.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.321+0000 7f024d3a7700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02480b1a50 0x7f02480b0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:44.325 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.321+0000 7f024d3a7700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0248142d50 con 0x7f02480af910 2022-06-29T16:07:44.326 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.321+0000 7f024d3a7700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0248063910 con 0x7f02480b56c0 2022-06-29T16:07:44.326 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.321+0000 7f024d3a7700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02480af380 con 0x7f02480b1a50 2022-06-29T16:07:44.326 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.321+0000 7f0247fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f02480b56c0 0x7f02480b5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:44.326 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.321+0000 7f0247fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f02480b56c0 0x7f02480b5aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53378/0 (socket says 172.21.15.60:53378) 2022-06-29T16:07:44.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.321+0000 7f0247fff700 1 -- 172.21.15.60:0/3243563098 learned_addr learned my addr 172.21.15.60:0/3243563098 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:44.327 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.321+0000 7f024cba6700 1 --2- 172.21.15.60:0/3243563098 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02480b1a50 0x7f02480b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:44.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.321+0000 7f0247fff700 1 -- 172.21.15.60:0/3243563098 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02480b1a50 msgr2=0x7f02480b0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:44.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.321+0000 7f0247fff700 1 --2- 172.21.15.60:0/3243563098 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02480b1a50 0x7f02480b0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:44.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.321+0000 7f0247fff700 1 -- 172.21.15.60:0/3243563098 >> v1:172.21.15.92:6789/0 conn(0x7f02480af910 legacy=0x7f02480674c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:44.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.321+0000 7f0247fff700 1 -- 172.21.15.60:0/3243563098 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02480b12e0 con 0x7f02480b56c0 2022-06-29T16:07:44.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.321+0000 7f0247fff700 1 --2- 172.21.15.60:0/3243563098 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02480b56c0 0x7f02480b5aa0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 rx=0x7f0238002f70 tx=0x7f0238006eb0).ready entity=mon.0 client_cookie=18fbe7aa205d9211 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:44.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.321+0000 7f0246ffd700 1 -- 172.21.15.60:0/3243563098 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f023800d570 con 0x7f02480b56c0 2022-06-29T16:07:44.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.321+0000 7f0246ffd700 1 -- 172.21.15.60:0/3243563098 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0238000e90 con 0x7f02480b56c0 2022-06-29T16:07:44.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.321+0000 7f0246ffd700 1 -- 172.21.15.60:0/3243563098 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f023800d900 con 0x7f02480b56c0 2022-06-29T16:07:44.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.321+0000 7f024d3a7700 1 -- 172.21.15.60:0/3243563098 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02480b56c0 msgr2=0x7f02480b5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:44.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.321+0000 7f024d3a7700 1 --2- 172.21.15.60:0/3243563098 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02480b56c0 0x7f02480b5aa0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 rx=0x7f0238002f70 tx=0x7f0238006eb0).stop 2022-06-29T16:07:44.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.321+0000 7f024d3a7700 1 -- 172.21.15.60:0/3243563098 shutdown_connections 2022-06-29T16:07:44.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.321+0000 7f024d3a7700 1 --2- 172.21.15.60:0/3243563098 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02480b1a50 0x7f02480b0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:44.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.321+0000 7f024d3a7700 1 --2- 172.21.15.60:0/3243563098 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02480b56c0 0x7f02480b5aa0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:44.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.321+0000 7f024d3a7700 1 -- 172.21.15.60:0/3243563098 >> 172.21.15.60:0/3243563098 conn(0x7f024805c650 msgr2=0x7f0248062f50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:44.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.325+0000 7f024d3a7700 1 -- 172.21.15.60:0/3243563098 shutdown_connections 2022-06-29T16:07:44.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.325+0000 7f024d3a7700 1 -- 172.21.15.60:0/3243563098 wait complete. 2022-06-29T16:07:44.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.325+0000 7f024d3a7700 1 Processor -- start 2022-06-29T16:07:44.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.325+0000 7f024d3a7700 1 -- start start 2022-06-29T16:07:44.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.329+0000 7f024d3a7700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02480af910 0x7f0248063ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:44.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.329+0000 7f024d3a7700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02480b1a50 0x7f0248145620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:44.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.329+0000 7f024d3a7700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02480b56c0 0x7f02481442f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:44.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.329+0000 7f024d3a7700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f024813ec20 con 0x7f02480b56c0 2022-06-29T16:07:44.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.329+0000 7f024d3a7700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0248146540 con 0x7f02480af910 2022-06-29T16:07:44.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.329+0000 7f024d3a7700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02481467e0 con 0x7f02480b1a50 2022-06-29T16:07:44.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.329+0000 7f024cba6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02480b56c0 0x7f02481442f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:44.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.329+0000 7f024cba6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02480b56c0 0x7f02481442f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53388/0 (socket says 172.21.15.60:53388) 2022-06-29T16:07:44.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.329+0000 7f024cba6700 1 -- 172.21.15.60:0/773450155 learned_addr learned my addr 172.21.15.60:0/773450155 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:44.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.329+0000 7f0247fff700 1 --2- 172.21.15.60:0/773450155 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02480af910 0x7f0248063ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:44.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.329+0000 7f02477fe700 1 --2- 172.21.15.60:0/773450155 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02480b1a50 0x7f0248145620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:44.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.329+0000 7f024cba6700 1 -- 172.21.15.60:0/773450155 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02480b1a50 msgr2=0x7f0248145620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:44.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.329+0000 7f024cba6700 1 --2- 172.21.15.60:0/773450155 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02480b1a50 0x7f0248145620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:44.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.329+0000 7f024cba6700 1 -- 172.21.15.60:0/773450155 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02480af910 msgr2=0x7f0248063ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:44.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.329+0000 7f024cba6700 1 --2- 172.21.15.60:0/773450155 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02480af910 0x7f0248063ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:44.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.329+0000 7f024cba6700 1 -- 172.21.15.60:0/773450155 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f023800b1e0 con 0x7f02480b56c0 2022-06-29T16:07:44.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.329+0000 7f024cba6700 1 --2- 172.21.15.60:0/773450155 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02480b56c0 0x7f02481442f0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 rx=0x7f0240006e00 tx=0x7f02400091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:44.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.329+0000 7f02457fa700 1 -- 172.21.15.60:0/773450155 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f024000c640 con 0x7f02480b56c0 2022-06-29T16:07:44.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.329+0000 7f02457fa700 1 -- 172.21.15.60:0/773450155 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f024000c7a0 con 0x7f02480b56c0 2022-06-29T16:07:44.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.329+0000 7f024d3a7700 1 -- 172.21.15.60:0/773450155 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0248066d30 con 0x7f02480b56c0 2022-06-29T16:07:44.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.329+0000 7f02457fa700 1 -- 172.21.15.60:0/773450155 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f024000cb00 con 0x7f02480b56c0 2022-06-29T16:07:44.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.329+0000 7f024d3a7700 1 -- 172.21.15.60:0/773450155 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0248146ea0 con 0x7f02480b56c0 2022-06-29T16:07:44.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.333+0000 7f02457fa700 1 -- 172.21.15.60:0/773450155 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f024000cc60 con 0x7f02480b56c0 2022-06-29T16:07:44.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.333+0000 7f024d3a7700 1 -- 172.21.15.60:0/773450155 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02480573d0 con 0x7f02480b56c0 2022-06-29T16:07:44.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.333+0000 7f02457fa700 1 --2- 172.21.15.60:0/773450155 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0228053b30 0x7f0228055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:44.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.333+0000 7f02457fa700 1 -- 172.21.15.60:0/773450155 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f024006c1a0 con 0x7f02480b56c0 2022-06-29T16:07:44.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.333+0000 7f0247fff700 1 --2- 172.21.15.60:0/773450155 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0228053b30 0x7f0228055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:44.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.333+0000 7f0247fff700 1 --2- 172.21.15.60:0/773450155 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0228053b30 0x7f0228055ff0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7f023801c000 tx=0x7f0238015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:44.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.341+0000 7f02457fa700 1 -- 172.21.15.60:0/773450155 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0240042f60 con 0x7f02480b56c0 2022-06-29T16:07:44.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:44.465+0000 7f024d3a7700 1 -- 172.21.15.60:0/773450155 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f02480617c0 con 0x7f02480b56c0 2022-06-29T16:07:45.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.197+0000 7f02457fa700 1 -- 172.21.15.60:0/773450155 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v130) v1 ==== 185+0+0 (secure 0 0 0) 0x7f0240046580 con 0x7f02480b56c0 2022-06-29T16:07:45.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.201+0000 7f024d3a7700 1 -- 172.21.15.60:0/773450155 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0228053b30 msgr2=0x7f0228055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:45.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.201+0000 7f024d3a7700 1 --2- 172.21.15.60:0/773450155 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0228053b30 0x7f0228055ff0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7f023801c000 tx=0x7f0238015000).stop 2022-06-29T16:07:45.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.201+0000 7f024d3a7700 1 -- 172.21.15.60:0/773450155 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02480b56c0 msgr2=0x7f02481442f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:45.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.201+0000 7f024d3a7700 1 --2- 172.21.15.60:0/773450155 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02480b56c0 0x7f02481442f0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 rx=0x7f0240006e00 tx=0x7f02400091d0).stop 2022-06-29T16:07:45.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.201+0000 7f024d3a7700 1 -- 172.21.15.60:0/773450155 shutdown_connections 2022-06-29T16:07:45.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.201+0000 7f024d3a7700 1 --2- 172.21.15.60:0/773450155 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0228053b30 0x7f0228055ff0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:45.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.201+0000 7f024d3a7700 1 --2- 172.21.15.60:0/773450155 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02480b56c0 0x7f02481442f0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:45.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.201+0000 7f024d3a7700 1 --2- 172.21.15.60:0/773450155 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f02480b1a50 0x7f0248145620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:45.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.201+0000 7f024d3a7700 1 --2- 172.21.15.60:0/773450155 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f02480af910 0x7f0248063ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:45.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.201+0000 7f024d3a7700 1 -- 172.21.15.60:0/773450155 >> 172.21.15.60:0/773450155 conn(0x7f024805c650 msgr2=0x7f02480b21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:45.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.201+0000 7f024d3a7700 1 -- 172.21.15.60:0/773450155 shutdown_connections 2022-06-29T16:07:45.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.201+0000 7f024d3a7700 1 -- 172.21.15.60:0/773450155 wait complete. 2022-06-29T16:07:45.208 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2022-06-29T16:07:45.219 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-06-29T16:07:45.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.329+0000 7f3e7d705700 1 Processor -- start 2022-06-29T16:07:45.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.329+0000 7f3e7d705700 1 -- start start 2022-06-29T16:07:45.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.333+0000 7f3e7d705700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e7809b580 0x7f3e78144150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:45.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.333+0000 7f3e7d705700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3e78061450 0x7f3e7813fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:45.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.333+0000 7f3e7d705700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e78061d00 con 0x7f3e78060290 2022-06-29T16:07:45.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.333+0000 7f3e7d705700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e78141800 con 0x7f3e78061450 2022-06-29T16:07:45.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.333+0000 7f3e7d705700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e780b2590 con 0x7f3e7809b580 2022-06-29T16:07:45.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.333+0000 7f3e6f7fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f3e78060290 0x7f3e78144640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33248/0 (socket says 172.21.15.60:33248) 2022-06-29T16:07:45.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.333+0000 7f3e6f7fe700 1 -- 172.21.15.60:0/3939871007 learned_addr learned my addr 172.21.15.60:0/3939871007 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:45.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.333+0000 7f3e7cf04700 1 --2- 172.21.15.60:0/3939871007 >> v2:172.21.15.60:3300/0 conn(0x7f3e78061450 0x7f3e7813fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:45.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.333+0000 7f3e77fff700 1 --2- 172.21.15.60:0/3939871007 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e7809b580 0x7f3e78144150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:45.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.333+0000 7f3e777fe700 1 -- 172.21.15.60:0/3939871007 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3628813730 0 0) 0x7f3e78061d00 con 0x7f3e78060290 2022-06-29T16:07:45.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.333+0000 7f3e777fe700 1 -- 172.21.15.60:0/3939871007 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e5c003260 con 0x7f3e78060290 2022-06-29T16:07:45.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.333+0000 7f3e7cf04700 1 -- 172.21.15.60:0/3939871007 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e7809b580 msgr2=0x7f3e78144150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:45.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.333+0000 7f3e7cf04700 1 --2- 172.21.15.60:0/3939871007 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e7809b580 0x7f3e78144150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:45.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.333+0000 7f3e7cf04700 1 -- 172.21.15.60:0/3939871007 >> v1:172.21.15.60:6789/0 conn(0x7f3e78060290 legacy=0x7f3e78144640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:45.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.333+0000 7f3e7cf04700 1 -- 172.21.15.60:0/3939871007 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e78144d50 con 0x7f3e78061450 2022-06-29T16:07:45.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.333+0000 7f3e7cf04700 1 --2- 172.21.15.60:0/3939871007 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e78061450 0x7f3e7813fe60 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 rx=0x7f3e700032f0 tx=0x7f3e700091d0).ready entity=mon.0 client_cookie=a6281ee58f6b1677 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:45.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.333+0000 7f3e777fe700 1 -- 172.21.15.60:0/3939871007 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e7000b600 con 0x7f3e78061450 2022-06-29T16:07:45.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.333+0000 7f3e777fe700 1 -- 172.21.15.60:0/3939871007 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e7000b760 con 0x7f3e78061450 2022-06-29T16:07:45.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.333+0000 7f3e777fe700 1 -- 172.21.15.60:0/3939871007 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e7000b8d0 con 0x7f3e78061450 2022-06-29T16:07:45.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.333+0000 7f3e7d705700 1 -- 172.21.15.60:0/3939871007 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e78061450 msgr2=0x7f3e7813fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:45.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.333+0000 7f3e7d705700 1 --2- 172.21.15.60:0/3939871007 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e78061450 0x7f3e7813fe60 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 rx=0x7f3e700032f0 tx=0x7f3e700091d0).stop 2022-06-29T16:07:45.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.333+0000 7f3e7d705700 1 -- 172.21.15.60:0/3939871007 shutdown_connections 2022-06-29T16:07:45.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.333+0000 7f3e7d705700 1 --2- 172.21.15.60:0/3939871007 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e78061450 0x7f3e7813fe60 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:45.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.333+0000 7f3e7d705700 1 --2- 172.21.15.60:0/3939871007 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e7809b580 0x7f3e78144150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:45.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.333+0000 7f3e7d705700 1 -- 172.21.15.60:0/3939871007 >> 172.21.15.60:0/3939871007 conn(0x7f3e78064290 msgr2=0x7f3e780b5ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:45.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.337+0000 7f3e7d705700 1 -- 172.21.15.60:0/3939871007 shutdown_connections 2022-06-29T16:07:45.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.337+0000 7f3e7d705700 1 -- 172.21.15.60:0/3939871007 wait complete. 2022-06-29T16:07:45.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.337+0000 7f3e7d705700 1 Processor -- start 2022-06-29T16:07:45.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.337+0000 7f3e7d705700 1 -- start start 2022-06-29T16:07:45.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.337+0000 7f3e7d705700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e78060290 0x7f3e780b4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:45.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.337+0000 7f3e7d705700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e78061450 0x7f3e78150360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:45.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.337+0000 7f3e7d705700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e7809b580 0x7f3e78150850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:45.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.337+0000 7f3e7d705700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e780af410 con 0x7f3e78060290 2022-06-29T16:07:45.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.337+0000 7f3e7d705700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7814fba0 con 0x7f3e78061450 2022-06-29T16:07:45.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.337+0000 7f3e7d705700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e780aee10 con 0x7f3e7809b580 2022-06-29T16:07:45.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.337+0000 7f3e77fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e78060290 0x7f3e780b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:45.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.337+0000 7f3e77fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e78060290 0x7f3e780b4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53398/0 (socket says 172.21.15.60:53398) 2022-06-29T16:07:45.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.337+0000 7f3e77fff700 1 -- 172.21.15.60:0/2972738092 learned_addr learned my addr 172.21.15.60:0/2972738092 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:45.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.337+0000 7f3e6f7fe700 1 --2- 172.21.15.60:0/2972738092 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e78061450 0x7f3e78150360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:45.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.337+0000 7f3e7cf04700 1 --2- 172.21.15.60:0/2972738092 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e7809b580 0x7f3e78150850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:45.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.341+0000 7f3e77fff700 1 -- 172.21.15.60:0/2972738092 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e7809b580 msgr2=0x7f3e78150850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:45.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.341+0000 7f3e77fff700 1 --2- 172.21.15.60:0/2972738092 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e7809b580 0x7f3e78150850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:45.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.341+0000 7f3e77fff700 1 -- 172.21.15.60:0/2972738092 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e78061450 msgr2=0x7f3e78150360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:45.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.341+0000 7f3e77fff700 1 --2- 172.21.15.60:0/2972738092 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e78061450 0x7f3e78150360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:45.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.341+0000 7f3e77fff700 1 -- 172.21.15.60:0/2972738092 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e70009470 con 0x7f3e78060290 2022-06-29T16:07:45.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.341+0000 7f3e6f7fe700 1 --2- 172.21.15.60:0/2972738092 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e78061450 0x7f3e78150360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:07:45.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.341+0000 7f3e77fff700 1 --2- 172.21.15.60:0/2972738092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e78060290 0x7f3e780b4300 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 rx=0x7f3e6800c000 tx=0x7f3e68006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:45.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.341+0000 7f3e75ffb700 1 -- 172.21.15.60:0/2972738092 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e68011750 con 0x7f3e78060290 2022-06-29T16:07:45.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.341+0000 7f3e75ffb700 1 -- 172.21.15.60:0/2972738092 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e680118b0 con 0x7f3e78060290 2022-06-29T16:07:45.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.341+0000 7f3e7d705700 1 -- 172.21.15.60:0/2972738092 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e780b2830 con 0x7f3e78060290 2022-06-29T16:07:45.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.341+0000 7f3e75ffb700 1 -- 172.21.15.60:0/2972738092 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e68011c10 con 0x7f3e78060290 2022-06-29T16:07:45.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.341+0000 7f3e7d705700 1 -- 172.21.15.60:0/2972738092 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e780ad0a0 con 0x7f3e78060290 2022-06-29T16:07:45.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.341+0000 7f3e7d705700 1 -- 172.21.15.60:0/2972738092 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e780acc40 con 0x7f3e78060290 2022-06-29T16:07:45.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.341+0000 7f3e75ffb700 1 -- 172.21.15.60:0/2972738092 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3e68011d70 con 0x7f3e78060290 2022-06-29T16:07:45.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.341+0000 7f3e75ffb700 1 --2- 172.21.15.60:0/2972738092 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3e50053a70 0x7f3e50055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:45.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.345+0000 7f3e75ffb700 1 -- 172.21.15.60:0/2972738092 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f3e6806c230 con 0x7f3e78060290 2022-06-29T16:07:45.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.345+0000 7f3e6f7fe700 1 --2- 172.21.15.60:0/2972738092 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3e50053a70 0x7f3e50055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:45.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.345+0000 7f3e6f7fe700 1 --2- 172.21.15.60:0/2972738092 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3e50053a70 0x7f3e50055f30 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0x7f3e6000d000 tx=0x7f3e60009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:45.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.349+0000 7f3e75ffb700 1 -- 172.21.15.60:0/2972738092 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3e68043230 con 0x7f3e78060290 2022-06-29T16:07:45.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:45.477+0000 7f3e7d705700 1 -- 172.21.15.60:0/2972738092 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3e780ad230 con 0x7f3e78060290 2022-06-29T16:07:46.216 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.209+0000 7f3e75ffb700 1 -- 172.21.15.60:0/2972738092 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v131) v1 ==== 173+0+0 (secure 0 0 0) 0x7f3e68046850 con 0x7f3e78060290 2022-06-29T16:07:46.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.213+0000 7f3e7d705700 1 -- 172.21.15.60:0/2972738092 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3e50053a70 msgr2=0x7f3e50055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:46.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.213+0000 7f3e7d705700 1 --2- 172.21.15.60:0/2972738092 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3e50053a70 0x7f3e50055f30 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0x7f3e6000d000 tx=0x7f3e60009040).stop 2022-06-29T16:07:46.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.213+0000 7f3e7d705700 1 -- 172.21.15.60:0/2972738092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e78060290 msgr2=0x7f3e780b4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:46.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.213+0000 7f3e7d705700 1 --2- 172.21.15.60:0/2972738092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e78060290 0x7f3e780b4300 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 rx=0x7f3e6800c000 tx=0x7f3e68006df0).stop 2022-06-29T16:07:46.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.213+0000 7f3e7d705700 1 -- 172.21.15.60:0/2972738092 shutdown_connections 2022-06-29T16:07:46.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.213+0000 7f3e7d705700 1 --2- 172.21.15.60:0/2972738092 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3e50053a70 0x7f3e50055f30 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:46.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.213+0000 7f3e7d705700 1 --2- 172.21.15.60:0/2972738092 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3e7809b580 0x7f3e78150850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:46.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.213+0000 7f3e7d705700 1 --2- 172.21.15.60:0/2972738092 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e78061450 0x7f3e78150360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:46.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.213+0000 7f3e7d705700 1 --2- 172.21.15.60:0/2972738092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e78060290 0x7f3e780b4300 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:46.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.213+0000 7f3e7d705700 1 -- 172.21.15.60:0/2972738092 >> 172.21.15.60:0/2972738092 conn(0x7f3e78064290 msgr2=0x7f3e7805f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:46.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.213+0000 7f3e7d705700 1 -- 172.21.15.60:0/2972738092 shutdown_connections 2022-06-29T16:07:46.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.213+0000 7f3e7d705700 1 -- 172.21.15.60:0/2972738092 wait complete. 2022-06-29T16:07:46.225 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2022-06-29T16:07:46.236 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-06-29T16:07:46.239 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-06-29T16:07:46.243 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-06-29T16:07:46.246 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-06-29T16:07:46.251 DEBUG:teuthology.orchestra.run.smithi197:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-06-29T16:07:46.268 INFO:teuthology.orchestra.run.smithi060.stdout:[global] 2022-06-29T16:07:46.268 INFO:teuthology.orchestra.run.smithi060.stdout: chdir = "" 2022-06-29T16:07:46.268 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-06-29T16:07:46.268 INFO:teuthology.orchestra.run.smithi060.stdout: auth supported = cephx 2022-06-29T16:07:46.269 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:46.269 INFO:teuthology.orchestra.run.smithi060.stdout: filestore xattr use omap = true 2022-06-29T16:07:46.269 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:46.269 INFO:teuthology.orchestra.run.smithi060.stdout: mon clock drift allowed = 1.000 2022-06-29T16:07:46.270 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:46.270 INFO:teuthology.orchestra.run.smithi060.stdout: osd crush chooseleaf type = 0 2022-06-29T16:07:46.270 INFO:teuthology.orchestra.run.smithi060.stdout: auth debug = true 2022-06-29T16:07:46.270 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:46.271 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on old message = true 2022-06-29T16:07:46.271 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on bug = true 2022-06-29T16:07:46.271 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:46.271 INFO:teuthology.orchestra.run.smithi060.stdout: mon max pg per osd = 10000 # >= luminous 2022-06-29T16:07:46.271 INFO:teuthology.orchestra.run.smithi060.stdout: mon pg warn max object skew = 0 2022-06-29T16:07:46.272 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:46.272 INFO:teuthology.orchestra.run.smithi060.stdout: # disable pg_autoscaler by default for new pools 2022-06-29T16:07:46.272 INFO:teuthology.orchestra.run.smithi060.stdout: osd_pool_default_pg_autoscale_mode = off 2022-06-29T16:07:46.272 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:46.273 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default size = 2 2022-06-29T16:07:46.273 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:46.273 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow primary affinity = true 2022-06-29T16:07:46.273 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow pg remap = true 2022-06-29T16:07:46.274 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on legacy crush tunables = false 2022-06-29T16:07:46.274 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on crush straw calc version zero = false 2022-06-29T16:07:46.274 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on no sortbitwise = false 2022-06-29T16:07:46.274 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on osd down out interval zero = false 2022-06-29T16:07:46.275 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on too few osds = false 2022-06-29T16:07:46.275 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-06-29T16:07:46.275 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_no_redundancy = false 2022-06-29T16:07:46.275 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:46.275 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-06-29T16:07:46.276 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:46.276 INFO:teuthology.orchestra.run.smithi060.stdout: osd default data pool replay window = 5 2022-06-29T16:07:46.276 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:46.276 INFO:teuthology.orchestra.run.smithi060.stdout: mon allow pool delete = true 2022-06-29T16:07:46.276 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:46.277 INFO:teuthology.orchestra.run.smithi060.stdout: mon cluster log file level = debug 2022-06-29T16:07:46.277 INFO:teuthology.orchestra.run.smithi060.stdout: debug asserts on shutdown = true 2022-06-29T16:07:46.277 INFO:teuthology.orchestra.run.smithi060.stdout: mon health detail to clog = false 2022-06-29T16:07:46.277 INFO:teuthology.orchestra.run.smithi060.stdout: mon host = "172.21.15.60,172.21.15.92,[v2:172.21.15.92:3301,v1:172.21.15.92:6790]" 2022-06-29T16:07:46.278 INFO:teuthology.orchestra.run.smithi060.stdout: fsid = e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85 2022-06-29T16:07:46.278 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:46.278 INFO:teuthology.orchestra.run.smithi060.stdout:[osd] 2022-06-29T16:07:46.278 INFO:teuthology.orchestra.run.smithi060.stdout: osd journal size = 100 2022-06-29T16:07:46.278 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:46.279 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub load threshold = 5.0 2022-06-29T16:07:46.279 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub max interval = 600 2022-06-29T16:07:46.279 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:46.279 INFO:teuthology.orchestra.run.smithi060.stdout: osd recover clone overlap = true 2022-06-29T16:07:46.280 INFO:teuthology.orchestra.run.smithi060.stdout: osd recovery max chunk = 1048576 2022-06-29T16:07:46.280 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:46.280 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug shutdown = true 2022-06-29T16:07:46.280 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug op order = true 2022-06-29T16:07:46.280 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug verify stray on activate = true 2022-06-29T16:07:46.281 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:46.281 INFO:teuthology.orchestra.run.smithi060.stdout: osd open classes on start = true 2022-06-29T16:07:46.281 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug pg log writeout = true 2022-06-29T16:07:46.281 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:46.282 INFO:teuthology.orchestra.run.smithi060.stdout: osd deep scrub update digest min age = 30 2022-06-29T16:07:46.282 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:46.282 INFO:teuthology.orchestra.run.smithi060.stdout: osd map max advance = 10 2022-06-29T16:07:46.282 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:46.282 INFO:teuthology.orchestra.run.smithi060.stdout: journal zero on create = true 2022-06-29T16:07:46.283 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:46.283 INFO:teuthology.orchestra.run.smithi060.stdout: filestore ondisk finisher threads = 3 2022-06-29T16:07:46.283 INFO:teuthology.orchestra.run.smithi060.stdout: filestore apply finisher threads = 3 2022-06-29T16:07:46.283 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:46.284 INFO:teuthology.orchestra.run.smithi060.stdout: bdev debug aio = true 2022-06-29T16:07:46.284 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug misdirected ops = true 2022-06-29T16:07:46.284 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2022-06-29T16:07:46.284 INFO:teuthology.orchestra.run.smithi060.stdout: debug osd = 20 2022-06-29T16:07:46.284 INFO:teuthology.orchestra.run.smithi060.stdout: osd objectstore = filestore 2022-06-29T16:07:46.285 INFO:teuthology.orchestra.run.smithi060.stdout: osd op complaint time = 180 2022-06-29T16:07:46.285 INFO:teuthology.orchestra.run.smithi060.stdout: osd sloppy crc = True 2022-06-29T16:07:46.285 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:46.285 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr] 2022-06-29T16:07:46.286 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2022-06-29T16:07:46.286 INFO:teuthology.orchestra.run.smithi060.stdout: debug mgr = 20 2022-06-29T16:07:46.286 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2022-06-29T16:07:46.286 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2022-06-29T16:07:46.286 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2022-06-29T16:07:46.287 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2022-06-29T16:07:46.287 INFO:teuthology.orchestra.run.smithi060.stdout: mgr/telemetry/nag = false 2022-06-29T16:07:46.287 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:46.287 INFO:teuthology.orchestra.run.smithi060.stdout:[mon] 2022-06-29T16:07:46.287 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2022-06-29T16:07:46.288 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2022-06-29T16:07:46.288 INFO:teuthology.orchestra.run.smithi060.stdout: debug paxos = 20 2022-06-29T16:07:46.288 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2022-06-29T16:07:46.288 INFO:teuthology.orchestra.run.smithi060.stdout: mon data avail warn = 5 2022-06-29T16:07:46.289 INFO:teuthology.orchestra.run.smithi060.stdout: mon mgr mkfs grace = 240 2022-06-29T16:07:46.289 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2022-06-29T16:07:46.289 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd reporter subtree level = osd 2022-06-29T16:07:46.289 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd prime pg temp = true 2022-06-29T16:07:46.289 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2022-06-29T16:07:46.290 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:46.290 INFO:teuthology.orchestra.run.smithi060.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-06-29T16:07:46.290 INFO:teuthology.orchestra.run.smithi060.stdout: auth mon ticket ttl = 660 # 11m 2022-06-29T16:07:46.290 INFO:teuthology.orchestra.run.smithi060.stdout: auth service ticket ttl = 240 # 4m 2022-06-29T16:07:46.290 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:46.291 INFO:teuthology.orchestra.run.smithi060.stdout: # don't complain about insecure global_id in the test suite 2022-06-29T16:07:46.291 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-06-29T16:07:46.291 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-06-29T16:07:46.291 INFO:teuthology.orchestra.run.smithi060.stdout: mon op complaint time = 120 2022-06-29T16:07:46.291 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:46.292 INFO:teuthology.orchestra.run.smithi060.stdout:[client] 2022-06-29T16:07:46.292 INFO:teuthology.orchestra.run.smithi060.stdout: rgw cache enabled = true 2022-06-29T16:07:46.292 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable ops log = true 2022-06-29T16:07:46.292 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable usage log = true 2022-06-29T16:07:46.293 INFO:teuthology.orchestra.run.smithi060.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-06-29T16:07:46.293 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-06-29T16:07:46.293 INFO:teuthology.orchestra.run.smithi060.stdout: client mount timeout = 600 2022-06-29T16:07:46.293 INFO:teuthology.orchestra.run.smithi060.stdout: debug client = 20 2022-06-29T16:07:46.293 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2022-06-29T16:07:46.294 INFO:teuthology.orchestra.run.smithi060.stdout: rados mon op timeout = 15m 2022-06-29T16:07:46.294 INFO:teuthology.orchestra.run.smithi060.stdout: rados osd op timeout = 15m 2022-06-29T16:07:46.294 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.a] 2022-06-29T16:07:46.294 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.b] 2022-06-29T16:07:46.294 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.c] 2022-06-29T16:07:46.295 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.a] 2022-06-29T16:07:46.295 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.b] 2022-06-29T16:07:46.295 INFO:teuthology.orchestra.run.smithi060.stdout:[mds] 2022-06-29T16:07:46.295 INFO:teuthology.orchestra.run.smithi060.stdout: debug mds = 20 2022-06-29T16:07:46.296 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2022-06-29T16:07:46.296 INFO:teuthology.orchestra.run.smithi060.stdout: mds bal frag = True 2022-06-29T16:07:46.296 INFO:teuthology.orchestra.run.smithi060.stdout: mds bal fragment size max = 10000 2022-06-29T16:07:46.296 INFO:teuthology.orchestra.run.smithi060.stdout: mds bal merge size = 5 2022-06-29T16:07:46.296 INFO:teuthology.orchestra.run.smithi060.stdout: mds bal split bits = 3 2022-06-29T16:07:46.297 INFO:teuthology.orchestra.run.smithi060.stdout: mds bal split size = 100 2022-06-29T16:07:46.297 INFO:teuthology.orchestra.run.smithi060.stdout: mds debug frag = True 2022-06-29T16:07:46.297 INFO:teuthology.orchestra.run.smithi060.stdout: mds debug scatterstat = True 2022-06-29T16:07:46.297 INFO:teuthology.orchestra.run.smithi060.stdout: mds op complaint time = 180 2022-06-29T16:07:46.297 INFO:teuthology.orchestra.run.smithi060.stdout: mds verify scatter = True 2022-06-29T16:07:46.298 INFO:teuthology.orchestra.run.smithi060.stdout: osd op complaint time = 180 2022-06-29T16:07:46.298 INFO:teuthology.orchestra.run.smithi060.stdout: rados mon op timeout = 15m 2022-06-29T16:07:46.298 INFO:teuthology.orchestra.run.smithi060.stdout: rados osd op timeout = 15m 2022-06-29T16:07:46.302 INFO:teuthology.orchestra.run.smithi197.stdout:[global] 2022-06-29T16:07:46.302 INFO:teuthology.orchestra.run.smithi197.stdout: chdir = "" 2022-06-29T16:07:46.302 INFO:teuthology.orchestra.run.smithi197.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-06-29T16:07:46.303 INFO:teuthology.orchestra.run.smithi197.stdout: auth supported = cephx 2022-06-29T16:07:46.303 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:46.303 INFO:teuthology.orchestra.run.smithi197.stdout: filestore xattr use omap = true 2022-06-29T16:07:46.303 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:46.304 INFO:teuthology.orchestra.run.smithi197.stdout: mon clock drift allowed = 1.000 2022-06-29T16:07:46.304 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:46.304 INFO:teuthology.orchestra.run.smithi197.stdout: osd crush chooseleaf type = 0 2022-06-29T16:07:46.304 INFO:teuthology.orchestra.run.smithi197.stdout: auth debug = true 2022-06-29T16:07:46.304 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:46.305 INFO:teuthology.orchestra.run.smithi197.stdout: ms die on old message = true 2022-06-29T16:07:46.305 INFO:teuthology.orchestra.run.smithi197.stdout: ms die on bug = true 2022-06-29T16:07:46.305 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:46.305 INFO:teuthology.orchestra.run.smithi197.stdout: mon max pg per osd = 10000 # >= luminous 2022-06-29T16:07:46.306 INFO:teuthology.orchestra.run.smithi197.stdout: mon pg warn max object skew = 0 2022-06-29T16:07:46.306 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:46.306 INFO:teuthology.orchestra.run.smithi197.stdout: # disable pg_autoscaler by default for new pools 2022-06-29T16:07:46.306 INFO:teuthology.orchestra.run.smithi197.stdout: osd_pool_default_pg_autoscale_mode = off 2022-06-29T16:07:46.306 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:46.307 INFO:teuthology.orchestra.run.smithi197.stdout: osd pool default size = 2 2022-06-29T16:07:46.307 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:46.307 INFO:teuthology.orchestra.run.smithi197.stdout: mon osd allow primary affinity = true 2022-06-29T16:07:46.307 INFO:teuthology.orchestra.run.smithi197.stdout: mon osd allow pg remap = true 2022-06-29T16:07:46.308 INFO:teuthology.orchestra.run.smithi197.stdout: mon warn on legacy crush tunables = false 2022-06-29T16:07:46.308 INFO:teuthology.orchestra.run.smithi197.stdout: mon warn on crush straw calc version zero = false 2022-06-29T16:07:46.308 INFO:teuthology.orchestra.run.smithi197.stdout: mon warn on no sortbitwise = false 2022-06-29T16:07:46.308 INFO:teuthology.orchestra.run.smithi197.stdout: mon warn on osd down out interval zero = false 2022-06-29T16:07:46.308 INFO:teuthology.orchestra.run.smithi197.stdout: mon warn on too few osds = false 2022-06-29T16:07:46.309 INFO:teuthology.orchestra.run.smithi197.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-06-29T16:07:46.309 INFO:teuthology.orchestra.run.smithi197.stdout: mon_warn_on_pool_no_redundancy = false 2022-06-29T16:07:46.309 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:46.309 INFO:teuthology.orchestra.run.smithi197.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-06-29T16:07:46.309 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:46.310 INFO:teuthology.orchestra.run.smithi197.stdout: osd default data pool replay window = 5 2022-06-29T16:07:46.310 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:46.310 INFO:teuthology.orchestra.run.smithi197.stdout: mon allow pool delete = true 2022-06-29T16:07:46.310 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:46.311 INFO:teuthology.orchestra.run.smithi197.stdout: mon cluster log file level = debug 2022-06-29T16:07:46.311 INFO:teuthology.orchestra.run.smithi197.stdout: debug asserts on shutdown = true 2022-06-29T16:07:46.311 INFO:teuthology.orchestra.run.smithi197.stdout: mon health detail to clog = false 2022-06-29T16:07:46.311 INFO:teuthology.orchestra.run.smithi197.stdout: mon host = "172.21.15.60,172.21.15.92,[v2:172.21.15.92:3301,v1:172.21.15.92:6790]" 2022-06-29T16:07:46.311 INFO:teuthology.orchestra.run.smithi197.stdout: fsid = e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85 2022-06-29T16:07:46.312 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:46.312 INFO:teuthology.orchestra.run.smithi197.stdout:[osd] 2022-06-29T16:07:46.312 INFO:teuthology.orchestra.run.smithi197.stdout: osd journal size = 100 2022-06-29T16:07:46.312 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:46.312 INFO:teuthology.orchestra.run.smithi197.stdout: osd scrub load threshold = 5.0 2022-06-29T16:07:46.313 INFO:teuthology.orchestra.run.smithi197.stdout: osd scrub max interval = 600 2022-06-29T16:07:46.313 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:46.313 INFO:teuthology.orchestra.run.smithi197.stdout: osd recover clone overlap = true 2022-06-29T16:07:46.313 INFO:teuthology.orchestra.run.smithi197.stdout: osd recovery max chunk = 1048576 2022-06-29T16:07:46.314 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:46.314 INFO:teuthology.orchestra.run.smithi197.stdout: osd debug shutdown = true 2022-06-29T16:07:46.314 INFO:teuthology.orchestra.run.smithi197.stdout: osd debug op order = true 2022-06-29T16:07:46.314 INFO:teuthology.orchestra.run.smithi197.stdout: osd debug verify stray on activate = true 2022-06-29T16:07:46.314 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:46.315 INFO:teuthology.orchestra.run.smithi197.stdout: osd open classes on start = true 2022-06-29T16:07:46.315 INFO:teuthology.orchestra.run.smithi197.stdout: osd debug pg log writeout = true 2022-06-29T16:07:46.315 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:46.315 INFO:teuthology.orchestra.run.smithi197.stdout: osd deep scrub update digest min age = 30 2022-06-29T16:07:46.315 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:46.316 INFO:teuthology.orchestra.run.smithi197.stdout: osd map max advance = 10 2022-06-29T16:07:46.316 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:46.316 INFO:teuthology.orchestra.run.smithi197.stdout: journal zero on create = true 2022-06-29T16:07:46.316 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:46.317 INFO:teuthology.orchestra.run.smithi197.stdout: filestore ondisk finisher threads = 3 2022-06-29T16:07:46.317 INFO:teuthology.orchestra.run.smithi197.stdout: filestore apply finisher threads = 3 2022-06-29T16:07:46.317 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:46.317 INFO:teuthology.orchestra.run.smithi197.stdout: bdev debug aio = true 2022-06-29T16:07:46.317 INFO:teuthology.orchestra.run.smithi197.stdout: osd debug misdirected ops = true 2022-06-29T16:07:46.318 INFO:teuthology.orchestra.run.smithi197.stdout: debug ms = 1 2022-06-29T16:07:46.318 INFO:teuthology.orchestra.run.smithi197.stdout: debug osd = 20 2022-06-29T16:07:46.318 INFO:teuthology.orchestra.run.smithi197.stdout: osd objectstore = filestore 2022-06-29T16:07:46.318 INFO:teuthology.orchestra.run.smithi197.stdout: osd op complaint time = 180 2022-06-29T16:07:46.319 INFO:teuthology.orchestra.run.smithi197.stdout: osd sloppy crc = True 2022-06-29T16:07:46.319 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:46.319 INFO:teuthology.orchestra.run.smithi197.stdout:[mgr] 2022-06-29T16:07:46.319 INFO:teuthology.orchestra.run.smithi197.stdout: debug ms = 1 2022-06-29T16:07:46.320 INFO:teuthology.orchestra.run.smithi197.stdout: debug mgr = 20 2022-06-29T16:07:46.320 INFO:teuthology.orchestra.run.smithi197.stdout: debug mon = 20 2022-06-29T16:07:46.320 INFO:teuthology.orchestra.run.smithi197.stdout: debug auth = 20 2022-06-29T16:07:46.320 INFO:teuthology.orchestra.run.smithi197.stdout: mon reweight min pgs per osd = 4 2022-06-29T16:07:46.320 INFO:teuthology.orchestra.run.smithi197.stdout: mon reweight min bytes per osd = 10 2022-06-29T16:07:46.321 INFO:teuthology.orchestra.run.smithi197.stdout: mgr/telemetry/nag = false 2022-06-29T16:07:46.321 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:46.321 INFO:teuthology.orchestra.run.smithi197.stdout:[mon] 2022-06-29T16:07:46.321 INFO:teuthology.orchestra.run.smithi197.stdout: debug ms = 1 2022-06-29T16:07:46.322 INFO:teuthology.orchestra.run.smithi197.stdout: debug mon = 20 2022-06-29T16:07:46.322 INFO:teuthology.orchestra.run.smithi197.stdout: debug paxos = 20 2022-06-29T16:07:46.322 INFO:teuthology.orchestra.run.smithi197.stdout: debug auth = 20 2022-06-29T16:07:46.322 INFO:teuthology.orchestra.run.smithi197.stdout: mon data avail warn = 5 2022-06-29T16:07:46.322 INFO:teuthology.orchestra.run.smithi197.stdout: mon mgr mkfs grace = 240 2022-06-29T16:07:46.323 INFO:teuthology.orchestra.run.smithi197.stdout: mon reweight min pgs per osd = 4 2022-06-29T16:07:46.323 INFO:teuthology.orchestra.run.smithi197.stdout: mon osd reporter subtree level = osd 2022-06-29T16:07:46.323 INFO:teuthology.orchestra.run.smithi197.stdout: mon osd prime pg temp = true 2022-06-29T16:07:46.323 INFO:teuthology.orchestra.run.smithi197.stdout: mon reweight min bytes per osd = 10 2022-06-29T16:07:46.323 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:46.324 INFO:teuthology.orchestra.run.smithi197.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-06-29T16:07:46.324 INFO:teuthology.orchestra.run.smithi197.stdout: auth mon ticket ttl = 660 # 11m 2022-06-29T16:07:46.324 INFO:teuthology.orchestra.run.smithi197.stdout: auth service ticket ttl = 240 # 4m 2022-06-29T16:07:46.324 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:46.325 INFO:teuthology.orchestra.run.smithi197.stdout: # don't complain about insecure global_id in the test suite 2022-06-29T16:07:46.325 INFO:teuthology.orchestra.run.smithi197.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-06-29T16:07:46.325 INFO:teuthology.orchestra.run.smithi197.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-06-29T16:07:46.325 INFO:teuthology.orchestra.run.smithi197.stdout: mon op complaint time = 120 2022-06-29T16:07:46.325 INFO:teuthology.orchestra.run.smithi197.stdout: 2022-06-29T16:07:46.326 INFO:teuthology.orchestra.run.smithi197.stdout:[client] 2022-06-29T16:07:46.326 INFO:teuthology.orchestra.run.smithi197.stdout: rgw cache enabled = true 2022-06-29T16:07:46.326 INFO:teuthology.orchestra.run.smithi197.stdout: rgw enable ops log = true 2022-06-29T16:07:46.326 INFO:teuthology.orchestra.run.smithi197.stdout: rgw enable usage log = true 2022-06-29T16:07:46.326 INFO:teuthology.orchestra.run.smithi197.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-06-29T16:07:46.327 INFO:teuthology.orchestra.run.smithi197.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-06-29T16:07:46.327 INFO:teuthology.orchestra.run.smithi197.stdout: client mount timeout = 600 2022-06-29T16:07:46.327 INFO:teuthology.orchestra.run.smithi197.stdout: debug client = 20 2022-06-29T16:07:46.327 INFO:teuthology.orchestra.run.smithi197.stdout: debug ms = 1 2022-06-29T16:07:46.328 INFO:teuthology.orchestra.run.smithi197.stdout: rados mon op timeout = 15m 2022-06-29T16:07:46.328 INFO:teuthology.orchestra.run.smithi197.stdout: rados osd op timeout = 15m 2022-06-29T16:07:46.328 INFO:teuthology.orchestra.run.smithi197.stdout:[mon.a] 2022-06-29T16:07:46.328 INFO:teuthology.orchestra.run.smithi197.stdout:[mon.b] 2022-06-29T16:07:46.328 INFO:teuthology.orchestra.run.smithi197.stdout:[mon.c] 2022-06-29T16:07:46.329 INFO:teuthology.orchestra.run.smithi197.stdout:[mds.a] 2022-06-29T16:07:46.329 INFO:teuthology.orchestra.run.smithi197.stdout:[mds.b] 2022-06-29T16:07:46.329 INFO:teuthology.orchestra.run.smithi197.stdout:[mds] 2022-06-29T16:07:46.329 INFO:teuthology.orchestra.run.smithi197.stdout: debug mds = 20 2022-06-29T16:07:46.330 INFO:teuthology.orchestra.run.smithi197.stdout: debug ms = 1 2022-06-29T16:07:46.330 INFO:teuthology.orchestra.run.smithi197.stdout: mds bal frag = True 2022-06-29T16:07:46.330 INFO:teuthology.orchestra.run.smithi197.stdout: mds bal fragment size max = 10000 2022-06-29T16:07:46.330 INFO:teuthology.orchestra.run.smithi197.stdout: mds bal merge size = 5 2022-06-29T16:07:46.330 INFO:teuthology.orchestra.run.smithi197.stdout: mds bal split bits = 3 2022-06-29T16:07:46.331 INFO:teuthology.orchestra.run.smithi197.stdout: mds bal split size = 100 2022-06-29T16:07:46.331 INFO:teuthology.orchestra.run.smithi197.stdout: mds debug frag = True 2022-06-29T16:07:46.331 INFO:teuthology.orchestra.run.smithi197.stdout: mds debug scatterstat = True 2022-06-29T16:07:46.331 INFO:teuthology.orchestra.run.smithi197.stdout: mds op complaint time = 180 2022-06-29T16:07:46.332 INFO:teuthology.orchestra.run.smithi197.stdout: mds verify scatter = True 2022-06-29T16:07:46.332 INFO:teuthology.orchestra.run.smithi197.stdout: osd op complaint time = 180 2022-06-29T16:07:46.332 INFO:teuthology.orchestra.run.smithi197.stdout: rados mon op timeout = 15m 2022-06-29T16:07:46.332 INFO:teuthology.orchestra.run.smithi197.stdout: rados osd op timeout = 15m 2022-06-29T16:07:46.332 INFO:teuthology.orchestra.run.smithi156.stdout:[global] 2022-06-29T16:07:46.333 INFO:teuthology.orchestra.run.smithi156.stdout: chdir = "" 2022-06-29T16:07:46.333 INFO:teuthology.orchestra.run.smithi156.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-06-29T16:07:46.333 INFO:teuthology.orchestra.run.smithi156.stdout: auth supported = cephx 2022-06-29T16:07:46.333 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:46.334 INFO:teuthology.orchestra.run.smithi156.stdout: filestore xattr use omap = true 2022-06-29T16:07:46.334 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:46.334 INFO:teuthology.orchestra.run.smithi156.stdout: mon clock drift allowed = 1.000 2022-06-29T16:07:46.334 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:46.335 INFO:teuthology.orchestra.run.smithi156.stdout: osd crush chooseleaf type = 0 2022-06-29T16:07:46.335 INFO:teuthology.orchestra.run.smithi156.stdout: auth debug = true 2022-06-29T16:07:46.335 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:46.335 INFO:teuthology.orchestra.run.smithi156.stdout: ms die on old message = true 2022-06-29T16:07:46.335 INFO:teuthology.orchestra.run.smithi156.stdout: ms die on bug = true 2022-06-29T16:07:46.336 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:46.336 INFO:teuthology.orchestra.run.smithi156.stdout: mon max pg per osd = 10000 # >= luminous 2022-06-29T16:07:46.336 INFO:teuthology.orchestra.run.smithi156.stdout: mon pg warn max object skew = 0 2022-06-29T16:07:46.336 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:46.336 INFO:teuthology.orchestra.run.smithi156.stdout: # disable pg_autoscaler by default for new pools 2022-06-29T16:07:46.337 INFO:teuthology.orchestra.run.smithi156.stdout: osd_pool_default_pg_autoscale_mode = off 2022-06-29T16:07:46.337 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:46.337 INFO:teuthology.orchestra.run.smithi156.stdout: osd pool default size = 2 2022-06-29T16:07:46.337 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:46.338 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd allow primary affinity = true 2022-06-29T16:07:46.338 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd allow pg remap = true 2022-06-29T16:07:46.338 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on legacy crush tunables = false 2022-06-29T16:07:46.338 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on crush straw calc version zero = false 2022-06-29T16:07:46.338 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on no sortbitwise = false 2022-06-29T16:07:46.339 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on osd down out interval zero = false 2022-06-29T16:07:46.339 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on too few osds = false 2022-06-29T16:07:46.339 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-06-29T16:07:46.339 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_pool_no_redundancy = false 2022-06-29T16:07:46.339 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:46.340 INFO:teuthology.orchestra.run.smithi156.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-06-29T16:07:46.340 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:46.340 INFO:teuthology.orchestra.run.smithi156.stdout: osd default data pool replay window = 5 2022-06-29T16:07:46.340 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:46.340 INFO:teuthology.orchestra.run.smithi156.stdout: mon allow pool delete = true 2022-06-29T16:07:46.341 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:46.341 INFO:teuthology.orchestra.run.smithi156.stdout: mon cluster log file level = debug 2022-06-29T16:07:46.341 INFO:teuthology.orchestra.run.smithi156.stdout: debug asserts on shutdown = true 2022-06-29T16:07:46.341 INFO:teuthology.orchestra.run.smithi156.stdout: mon health detail to clog = false 2022-06-29T16:07:46.342 INFO:teuthology.orchestra.run.smithi156.stdout: mon host = "172.21.15.60,172.21.15.92,[v2:172.21.15.92:3301,v1:172.21.15.92:6790]" 2022-06-29T16:07:46.342 INFO:teuthology.orchestra.run.smithi156.stdout: fsid = e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85 2022-06-29T16:07:46.342 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:46.342 INFO:teuthology.orchestra.run.smithi156.stdout:[osd] 2022-06-29T16:07:46.342 INFO:teuthology.orchestra.run.smithi156.stdout: osd journal size = 100 2022-06-29T16:07:46.343 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:46.343 INFO:teuthology.orchestra.run.smithi156.stdout: osd scrub load threshold = 5.0 2022-06-29T16:07:46.343 INFO:teuthology.orchestra.run.smithi156.stdout: osd scrub max interval = 600 2022-06-29T16:07:46.343 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:46.343 INFO:teuthology.orchestra.run.smithi156.stdout: osd recover clone overlap = true 2022-06-29T16:07:46.344 INFO:teuthology.orchestra.run.smithi156.stdout: osd recovery max chunk = 1048576 2022-06-29T16:07:46.344 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:46.344 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug shutdown = true 2022-06-29T16:07:46.344 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug op order = true 2022-06-29T16:07:46.345 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug verify stray on activate = true 2022-06-29T16:07:46.345 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:46.345 INFO:teuthology.orchestra.run.smithi156.stdout: osd open classes on start = true 2022-06-29T16:07:46.345 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug pg log writeout = true 2022-06-29T16:07:46.345 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:46.346 INFO:teuthology.orchestra.run.smithi156.stdout: osd deep scrub update digest min age = 30 2022-06-29T16:07:46.346 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:46.346 INFO:teuthology.orchestra.run.smithi156.stdout: osd map max advance = 10 2022-06-29T16:07:46.346 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:46.346 INFO:teuthology.orchestra.run.smithi156.stdout: journal zero on create = true 2022-06-29T16:07:46.347 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:46.347 INFO:teuthology.orchestra.run.smithi156.stdout: filestore ondisk finisher threads = 3 2022-06-29T16:07:46.347 INFO:teuthology.orchestra.run.smithi156.stdout: filestore apply finisher threads = 3 2022-06-29T16:07:46.347 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:46.348 INFO:teuthology.orchestra.run.smithi156.stdout: bdev debug aio = true 2022-06-29T16:07:46.348 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug misdirected ops = true 2022-06-29T16:07:46.348 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2022-06-29T16:07:46.348 INFO:teuthology.orchestra.run.smithi156.stdout: debug osd = 20 2022-06-29T16:07:46.348 INFO:teuthology.orchestra.run.smithi156.stdout: osd objectstore = filestore 2022-06-29T16:07:46.349 INFO:teuthology.orchestra.run.smithi156.stdout: osd op complaint time = 180 2022-06-29T16:07:46.349 INFO:teuthology.orchestra.run.smithi156.stdout: osd sloppy crc = True 2022-06-29T16:07:46.349 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:46.349 INFO:teuthology.orchestra.run.smithi156.stdout:[mgr] 2022-06-29T16:07:46.349 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2022-06-29T16:07:46.350 INFO:teuthology.orchestra.run.smithi156.stdout: debug mgr = 20 2022-06-29T16:07:46.350 INFO:teuthology.orchestra.run.smithi156.stdout: debug mon = 20 2022-06-29T16:07:46.350 INFO:teuthology.orchestra.run.smithi156.stdout: debug auth = 20 2022-06-29T16:07:46.350 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min pgs per osd = 4 2022-06-29T16:07:46.351 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min bytes per osd = 10 2022-06-29T16:07:46.351 INFO:teuthology.orchestra.run.smithi156.stdout: mgr/telemetry/nag = false 2022-06-29T16:07:46.351 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:46.351 INFO:teuthology.orchestra.run.smithi156.stdout:[mon] 2022-06-29T16:07:46.351 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2022-06-29T16:07:46.352 INFO:teuthology.orchestra.run.smithi156.stdout: debug mon = 20 2022-06-29T16:07:46.352 INFO:teuthology.orchestra.run.smithi156.stdout: debug paxos = 20 2022-06-29T16:07:46.352 INFO:teuthology.orchestra.run.smithi156.stdout: debug auth = 20 2022-06-29T16:07:46.352 INFO:teuthology.orchestra.run.smithi156.stdout: mon data avail warn = 5 2022-06-29T16:07:46.353 INFO:teuthology.orchestra.run.smithi156.stdout: mon mgr mkfs grace = 240 2022-06-29T16:07:46.353 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min pgs per osd = 4 2022-06-29T16:07:46.353 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd reporter subtree level = osd 2022-06-29T16:07:46.353 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd prime pg temp = true 2022-06-29T16:07:46.353 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min bytes per osd = 10 2022-06-29T16:07:46.354 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:46.354 INFO:teuthology.orchestra.run.smithi156.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-06-29T16:07:46.354 INFO:teuthology.orchestra.run.smithi156.stdout: auth mon ticket ttl = 660 # 11m 2022-06-29T16:07:46.354 INFO:teuthology.orchestra.run.smithi156.stdout: auth service ticket ttl = 240 # 4m 2022-06-29T16:07:46.354 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:46.355 INFO:teuthology.orchestra.run.smithi156.stdout: # don't complain about insecure global_id in the test suite 2022-06-29T16:07:46.355 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-06-29T16:07:46.355 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-06-29T16:07:46.355 INFO:teuthology.orchestra.run.smithi156.stdout: mon op complaint time = 120 2022-06-29T16:07:46.356 INFO:teuthology.orchestra.run.smithi156.stdout: 2022-06-29T16:07:46.356 INFO:teuthology.orchestra.run.smithi156.stdout:[client] 2022-06-29T16:07:46.356 INFO:teuthology.orchestra.run.smithi156.stdout: rgw cache enabled = true 2022-06-29T16:07:46.356 INFO:teuthology.orchestra.run.smithi156.stdout: rgw enable ops log = true 2022-06-29T16:07:46.356 INFO:teuthology.orchestra.run.smithi156.stdout: rgw enable usage log = true 2022-06-29T16:07:46.357 INFO:teuthology.orchestra.run.smithi156.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-06-29T16:07:46.357 INFO:teuthology.orchestra.run.smithi156.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-06-29T16:07:46.357 INFO:teuthology.orchestra.run.smithi156.stdout: client mount timeout = 600 2022-06-29T16:07:46.357 INFO:teuthology.orchestra.run.smithi156.stdout: debug client = 20 2022-06-29T16:07:46.357 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2022-06-29T16:07:46.358 INFO:teuthology.orchestra.run.smithi156.stdout: rados mon op timeout = 15m 2022-06-29T16:07:46.358 INFO:teuthology.orchestra.run.smithi156.stdout: rados osd op timeout = 15m 2022-06-29T16:07:46.358 INFO:teuthology.orchestra.run.smithi156.stdout:[mon.a] 2022-06-29T16:07:46.358 INFO:teuthology.orchestra.run.smithi156.stdout:[mon.b] 2022-06-29T16:07:46.359 INFO:teuthology.orchestra.run.smithi156.stdout:[mon.c] 2022-06-29T16:07:46.359 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.a] 2022-06-29T16:07:46.359 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.b] 2022-06-29T16:07:46.359 INFO:teuthology.orchestra.run.smithi156.stdout:[mds] 2022-06-29T16:07:46.359 INFO:teuthology.orchestra.run.smithi156.stdout: debug mds = 20 2022-06-29T16:07:46.360 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2022-06-29T16:07:46.360 INFO:teuthology.orchestra.run.smithi156.stdout: mds bal frag = True 2022-06-29T16:07:46.360 INFO:teuthology.orchestra.run.smithi156.stdout: mds bal fragment size max = 10000 2022-06-29T16:07:46.360 INFO:teuthology.orchestra.run.smithi156.stdout: mds bal merge size = 5 2022-06-29T16:07:46.360 INFO:teuthology.orchestra.run.smithi156.stdout: mds bal split bits = 3 2022-06-29T16:07:46.361 INFO:teuthology.orchestra.run.smithi156.stdout: mds bal split size = 100 2022-06-29T16:07:46.361 INFO:teuthology.orchestra.run.smithi156.stdout: mds debug frag = True 2022-06-29T16:07:46.361 INFO:teuthology.orchestra.run.smithi156.stdout: mds debug scatterstat = True 2022-06-29T16:07:46.361 INFO:teuthology.orchestra.run.smithi156.stdout: mds op complaint time = 180 2022-06-29T16:07:46.361 INFO:teuthology.orchestra.run.smithi156.stdout: mds verify scatter = True 2022-06-29T16:07:46.362 INFO:teuthology.orchestra.run.smithi156.stdout: osd op complaint time = 180 2022-06-29T16:07:46.362 INFO:teuthology.orchestra.run.smithi156.stdout: rados mon op timeout = 15m 2022-06-29T16:07:46.362 INFO:teuthology.orchestra.run.smithi156.stdout: rados osd op timeout = 15m 2022-06-29T16:07:46.362 INFO:teuthology.orchestra.run.smithi121.stdout:[global] 2022-06-29T16:07:46.363 INFO:teuthology.orchestra.run.smithi121.stdout: chdir = "" 2022-06-29T16:07:46.363 INFO:teuthology.orchestra.run.smithi121.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-06-29T16:07:46.363 INFO:teuthology.orchestra.run.smithi121.stdout: auth supported = cephx 2022-06-29T16:07:46.363 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:46.364 INFO:teuthology.orchestra.run.smithi121.stdout: filestore xattr use omap = true 2022-06-29T16:07:46.364 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:46.364 INFO:teuthology.orchestra.run.smithi121.stdout: mon clock drift allowed = 1.000 2022-06-29T16:07:46.364 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:46.364 INFO:teuthology.orchestra.run.smithi121.stdout: osd crush chooseleaf type = 0 2022-06-29T16:07:46.365 INFO:teuthology.orchestra.run.smithi121.stdout: auth debug = true 2022-06-29T16:07:46.365 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:46.365 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on old message = true 2022-06-29T16:07:46.365 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on bug = true 2022-06-29T16:07:46.366 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:46.366 INFO:teuthology.orchestra.run.smithi121.stdout: mon max pg per osd = 10000 # >= luminous 2022-06-29T16:07:46.366 INFO:teuthology.orchestra.run.smithi121.stdout: mon pg warn max object skew = 0 2022-06-29T16:07:46.366 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:46.366 INFO:teuthology.orchestra.run.smithi121.stdout: # disable pg_autoscaler by default for new pools 2022-06-29T16:07:46.367 INFO:teuthology.orchestra.run.smithi121.stdout: osd_pool_default_pg_autoscale_mode = off 2022-06-29T16:07:46.367 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:46.367 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default size = 2 2022-06-29T16:07:46.367 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:46.368 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow primary affinity = true 2022-06-29T16:07:46.368 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow pg remap = true 2022-06-29T16:07:46.368 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on legacy crush tunables = false 2022-06-29T16:07:46.368 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on crush straw calc version zero = false 2022-06-29T16:07:46.368 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on no sortbitwise = false 2022-06-29T16:07:46.369 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on osd down out interval zero = false 2022-06-29T16:07:46.369 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on too few osds = false 2022-06-29T16:07:46.369 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-06-29T16:07:46.369 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_no_redundancy = false 2022-06-29T16:07:46.370 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:46.370 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-06-29T16:07:46.370 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:46.370 INFO:teuthology.orchestra.run.smithi121.stdout: osd default data pool replay window = 5 2022-06-29T16:07:46.370 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:46.371 INFO:teuthology.orchestra.run.smithi121.stdout: mon allow pool delete = true 2022-06-29T16:07:46.371 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:46.371 INFO:teuthology.orchestra.run.smithi121.stdout: mon cluster log file level = debug 2022-06-29T16:07:46.371 INFO:teuthology.orchestra.run.smithi121.stdout: debug asserts on shutdown = true 2022-06-29T16:07:46.372 INFO:teuthology.orchestra.run.smithi121.stdout: mon health detail to clog = false 2022-06-29T16:07:46.372 INFO:teuthology.orchestra.run.smithi121.stdout: mon host = "172.21.15.60,172.21.15.92,[v2:172.21.15.92:3301,v1:172.21.15.92:6790]" 2022-06-29T16:07:46.372 INFO:teuthology.orchestra.run.smithi121.stdout: fsid = e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85 2022-06-29T16:07:46.372 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:46.372 INFO:teuthology.orchestra.run.smithi121.stdout:[osd] 2022-06-29T16:07:46.373 INFO:teuthology.orchestra.run.smithi121.stdout: osd journal size = 100 2022-06-29T16:07:46.373 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:46.373 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub load threshold = 5.0 2022-06-29T16:07:46.373 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub max interval = 600 2022-06-29T16:07:46.373 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:46.374 INFO:teuthology.orchestra.run.smithi121.stdout: osd recover clone overlap = true 2022-06-29T16:07:46.374 INFO:teuthology.orchestra.run.smithi121.stdout: osd recovery max chunk = 1048576 2022-06-29T16:07:46.374 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:46.374 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug shutdown = true 2022-06-29T16:07:46.374 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug op order = true 2022-06-29T16:07:46.375 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify stray on activate = true 2022-06-29T16:07:46.375 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:46.375 INFO:teuthology.orchestra.run.smithi121.stdout: osd open classes on start = true 2022-06-29T16:07:46.375 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug pg log writeout = true 2022-06-29T16:07:46.376 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:46.376 INFO:teuthology.orchestra.run.smithi121.stdout: osd deep scrub update digest min age = 30 2022-06-29T16:07:46.376 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:46.376 INFO:teuthology.orchestra.run.smithi121.stdout: osd map max advance = 10 2022-06-29T16:07:46.376 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:46.377 INFO:teuthology.orchestra.run.smithi121.stdout: journal zero on create = true 2022-06-29T16:07:46.377 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:46.377 INFO:teuthology.orchestra.run.smithi121.stdout: filestore ondisk finisher threads = 3 2022-06-29T16:07:46.377 INFO:teuthology.orchestra.run.smithi121.stdout: filestore apply finisher threads = 3 2022-06-29T16:07:46.377 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:46.378 INFO:teuthology.orchestra.run.smithi121.stdout: bdev debug aio = true 2022-06-29T16:07:46.378 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug misdirected ops = true 2022-06-29T16:07:46.378 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2022-06-29T16:07:46.378 INFO:teuthology.orchestra.run.smithi121.stdout: debug osd = 20 2022-06-29T16:07:46.378 INFO:teuthology.orchestra.run.smithi121.stdout: osd objectstore = filestore 2022-06-29T16:07:46.379 INFO:teuthology.orchestra.run.smithi121.stdout: osd op complaint time = 180 2022-06-29T16:07:46.379 INFO:teuthology.orchestra.run.smithi121.stdout: osd sloppy crc = True 2022-06-29T16:07:46.379 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:46.379 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr] 2022-06-29T16:07:46.380 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2022-06-29T16:07:46.380 INFO:teuthology.orchestra.run.smithi121.stdout: debug mgr = 20 2022-06-29T16:07:46.380 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2022-06-29T16:07:46.380 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2022-06-29T16:07:46.380 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2022-06-29T16:07:46.381 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2022-06-29T16:07:46.381 INFO:teuthology.orchestra.run.smithi121.stdout: mgr/telemetry/nag = false 2022-06-29T16:07:46.381 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:46.381 INFO:teuthology.orchestra.run.smithi121.stdout:[mon] 2022-06-29T16:07:46.381 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2022-06-29T16:07:46.382 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2022-06-29T16:07:46.382 INFO:teuthology.orchestra.run.smithi121.stdout: debug paxos = 20 2022-06-29T16:07:46.382 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2022-06-29T16:07:46.382 INFO:teuthology.orchestra.run.smithi121.stdout: mon data avail warn = 5 2022-06-29T16:07:46.383 INFO:teuthology.orchestra.run.smithi121.stdout: mon mgr mkfs grace = 240 2022-06-29T16:07:46.383 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2022-06-29T16:07:46.383 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd reporter subtree level = osd 2022-06-29T16:07:46.383 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd prime pg temp = true 2022-06-29T16:07:46.383 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2022-06-29T16:07:46.384 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:46.384 INFO:teuthology.orchestra.run.smithi121.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-06-29T16:07:46.384 INFO:teuthology.orchestra.run.smithi121.stdout: auth mon ticket ttl = 660 # 11m 2022-06-29T16:07:46.384 INFO:teuthology.orchestra.run.smithi121.stdout: auth service ticket ttl = 240 # 4m 2022-06-29T16:07:46.384 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:46.385 INFO:teuthology.orchestra.run.smithi121.stdout: # don't complain about insecure global_id in the test suite 2022-06-29T16:07:46.385 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-06-29T16:07:46.385 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-06-29T16:07:46.385 INFO:teuthology.orchestra.run.smithi121.stdout: mon op complaint time = 120 2022-06-29T16:07:46.386 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-06-29T16:07:46.386 INFO:teuthology.orchestra.run.smithi121.stdout:[client] 2022-06-29T16:07:46.386 INFO:teuthology.orchestra.run.smithi121.stdout: rgw cache enabled = true 2022-06-29T16:07:46.386 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable ops log = true 2022-06-29T16:07:46.386 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable usage log = true 2022-06-29T16:07:46.387 INFO:teuthology.orchestra.run.smithi121.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-06-29T16:07:46.387 INFO:teuthology.orchestra.run.smithi121.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-06-29T16:07:46.387 INFO:teuthology.orchestra.run.smithi121.stdout: client mount timeout = 600 2022-06-29T16:07:46.387 INFO:teuthology.orchestra.run.smithi121.stdout: debug client = 20 2022-06-29T16:07:46.387 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2022-06-29T16:07:46.388 INFO:teuthology.orchestra.run.smithi121.stdout: rados mon op timeout = 15m 2022-06-29T16:07:46.388 INFO:teuthology.orchestra.run.smithi121.stdout: rados osd op timeout = 15m 2022-06-29T16:07:46.388 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.a] 2022-06-29T16:07:46.388 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.b] 2022-06-29T16:07:46.389 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.c] 2022-06-29T16:07:46.389 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.a] 2022-06-29T16:07:46.389 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.b] 2022-06-29T16:07:46.389 INFO:teuthology.orchestra.run.smithi121.stdout:[mds] 2022-06-29T16:07:46.389 INFO:teuthology.orchestra.run.smithi121.stdout: debug mds = 20 2022-06-29T16:07:46.390 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2022-06-29T16:07:46.390 INFO:teuthology.orchestra.run.smithi121.stdout: mds bal frag = True 2022-06-29T16:07:46.390 INFO:teuthology.orchestra.run.smithi121.stdout: mds bal fragment size max = 10000 2022-06-29T16:07:46.390 INFO:teuthology.orchestra.run.smithi121.stdout: mds bal merge size = 5 2022-06-29T16:07:46.390 INFO:teuthology.orchestra.run.smithi121.stdout: mds bal split bits = 3 2022-06-29T16:07:46.391 INFO:teuthology.orchestra.run.smithi121.stdout: mds bal split size = 100 2022-06-29T16:07:46.391 INFO:teuthology.orchestra.run.smithi121.stdout: mds debug frag = True 2022-06-29T16:07:46.391 INFO:teuthology.orchestra.run.smithi121.stdout: mds debug scatterstat = True 2022-06-29T16:07:46.391 INFO:teuthology.orchestra.run.smithi121.stdout: mds op complaint time = 180 2022-06-29T16:07:46.392 INFO:teuthology.orchestra.run.smithi121.stdout: mds verify scatter = True 2022-06-29T16:07:46.392 INFO:teuthology.orchestra.run.smithi121.stdout: osd op complaint time = 180 2022-06-29T16:07:46.392 INFO:teuthology.orchestra.run.smithi121.stdout: rados mon op timeout = 15m 2022-06-29T16:07:46.392 INFO:teuthology.orchestra.run.smithi121.stdout: rados osd op timeout = 15m 2022-06-29T16:07:46.392 INFO:teuthology.orchestra.run.smithi092.stdout:[global] 2022-06-29T16:07:46.393 INFO:teuthology.orchestra.run.smithi092.stdout: chdir = "" 2022-06-29T16:07:46.393 INFO:teuthology.orchestra.run.smithi092.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-06-29T16:07:46.393 INFO:teuthology.orchestra.run.smithi092.stdout: auth supported = cephx 2022-06-29T16:07:46.393 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:46.394 INFO:teuthology.orchestra.run.smithi092.stdout: filestore xattr use omap = true 2022-06-29T16:07:46.394 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:46.394 INFO:teuthology.orchestra.run.smithi092.stdout: mon clock drift allowed = 1.000 2022-06-29T16:07:46.394 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:46.394 INFO:teuthology.orchestra.run.smithi092.stdout: osd crush chooseleaf type = 0 2022-06-29T16:07:46.395 INFO:teuthology.orchestra.run.smithi092.stdout: auth debug = true 2022-06-29T16:07:46.395 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:46.395 INFO:teuthology.orchestra.run.smithi092.stdout: ms die on old message = true 2022-06-29T16:07:46.395 INFO:teuthology.orchestra.run.smithi092.stdout: ms die on bug = true 2022-06-29T16:07:46.395 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:46.396 INFO:teuthology.orchestra.run.smithi092.stdout: mon max pg per osd = 10000 # >= luminous 2022-06-29T16:07:46.396 INFO:teuthology.orchestra.run.smithi092.stdout: mon pg warn max object skew = 0 2022-06-29T16:07:46.396 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:46.396 INFO:teuthology.orchestra.run.smithi092.stdout: # disable pg_autoscaler by default for new pools 2022-06-29T16:07:46.397 INFO:teuthology.orchestra.run.smithi092.stdout: osd_pool_default_pg_autoscale_mode = off 2022-06-29T16:07:46.397 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:46.397 INFO:teuthology.orchestra.run.smithi092.stdout: osd pool default size = 2 2022-06-29T16:07:46.397 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:46.397 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd allow primary affinity = true 2022-06-29T16:07:46.398 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd allow pg remap = true 2022-06-29T16:07:46.398 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on legacy crush tunables = false 2022-06-29T16:07:46.398 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on crush straw calc version zero = false 2022-06-29T16:07:46.398 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on no sortbitwise = false 2022-06-29T16:07:46.399 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on osd down out interval zero = false 2022-06-29T16:07:46.399 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on too few osds = false 2022-06-29T16:07:46.399 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-06-29T16:07:46.399 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_pool_no_redundancy = false 2022-06-29T16:07:46.399 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:46.400 INFO:teuthology.orchestra.run.smithi092.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-06-29T16:07:46.400 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:46.400 INFO:teuthology.orchestra.run.smithi092.stdout: osd default data pool replay window = 5 2022-06-29T16:07:46.400 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:46.400 INFO:teuthology.orchestra.run.smithi092.stdout: mon allow pool delete = true 2022-06-29T16:07:46.401 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:46.401 INFO:teuthology.orchestra.run.smithi092.stdout: mon cluster log file level = debug 2022-06-29T16:07:46.401 INFO:teuthology.orchestra.run.smithi092.stdout: debug asserts on shutdown = true 2022-06-29T16:07:46.401 INFO:teuthology.orchestra.run.smithi092.stdout: mon health detail to clog = false 2022-06-29T16:07:46.401 INFO:teuthology.orchestra.run.smithi092.stdout: mon host = "172.21.15.60,172.21.15.92,[v2:172.21.15.92:3301,v1:172.21.15.92:6790]" 2022-06-29T16:07:46.402 INFO:teuthology.orchestra.run.smithi092.stdout: fsid = e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85 2022-06-29T16:07:46.402 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:46.402 INFO:teuthology.orchestra.run.smithi092.stdout:[osd] 2022-06-29T16:07:46.402 INFO:teuthology.orchestra.run.smithi092.stdout: osd journal size = 100 2022-06-29T16:07:46.403 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:46.403 INFO:teuthology.orchestra.run.smithi092.stdout: osd scrub load threshold = 5.0 2022-06-29T16:07:46.403 INFO:teuthology.orchestra.run.smithi092.stdout: osd scrub max interval = 600 2022-06-29T16:07:46.403 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:46.403 INFO:teuthology.orchestra.run.smithi092.stdout: osd recover clone overlap = true 2022-06-29T16:07:46.404 INFO:teuthology.orchestra.run.smithi092.stdout: osd recovery max chunk = 1048576 2022-06-29T16:07:46.404 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:46.404 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug shutdown = true 2022-06-29T16:07:46.404 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug op order = true 2022-06-29T16:07:46.404 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug verify stray on activate = true 2022-06-29T16:07:46.405 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:46.405 INFO:teuthology.orchestra.run.smithi092.stdout: osd open classes on start = true 2022-06-29T16:07:46.405 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug pg log writeout = true 2022-06-29T16:07:46.405 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:46.405 INFO:teuthology.orchestra.run.smithi092.stdout: osd deep scrub update digest min age = 30 2022-06-29T16:07:46.406 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:46.406 INFO:teuthology.orchestra.run.smithi092.stdout: osd map max advance = 10 2022-06-29T16:07:46.406 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:46.406 INFO:teuthology.orchestra.run.smithi092.stdout: journal zero on create = true 2022-06-29T16:07:46.406 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:46.407 INFO:teuthology.orchestra.run.smithi092.stdout: filestore ondisk finisher threads = 3 2022-06-29T16:07:46.407 INFO:teuthology.orchestra.run.smithi092.stdout: filestore apply finisher threads = 3 2022-06-29T16:07:46.407 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:46.407 INFO:teuthology.orchestra.run.smithi092.stdout: bdev debug aio = true 2022-06-29T16:07:46.407 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug misdirected ops = true 2022-06-29T16:07:46.408 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2022-06-29T16:07:46.408 INFO:teuthology.orchestra.run.smithi092.stdout: debug osd = 20 2022-06-29T16:07:46.408 INFO:teuthology.orchestra.run.smithi092.stdout: osd objectstore = filestore 2022-06-29T16:07:46.408 INFO:teuthology.orchestra.run.smithi092.stdout: osd op complaint time = 180 2022-06-29T16:07:46.409 INFO:teuthology.orchestra.run.smithi092.stdout: osd sloppy crc = True 2022-06-29T16:07:46.409 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:46.409 INFO:teuthology.orchestra.run.smithi092.stdout:[mgr] 2022-06-29T16:07:46.409 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2022-06-29T16:07:46.409 INFO:teuthology.orchestra.run.smithi092.stdout: debug mgr = 20 2022-06-29T16:07:46.410 INFO:teuthology.orchestra.run.smithi092.stdout: debug mon = 20 2022-06-29T16:07:46.410 INFO:teuthology.orchestra.run.smithi092.stdout: debug auth = 20 2022-06-29T16:07:46.410 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min pgs per osd = 4 2022-06-29T16:07:46.410 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min bytes per osd = 10 2022-06-29T16:07:46.411 INFO:teuthology.orchestra.run.smithi092.stdout: mgr/telemetry/nag = false 2022-06-29T16:07:46.411 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:46.411 INFO:teuthology.orchestra.run.smithi092.stdout:[mon] 2022-06-29T16:07:46.411 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2022-06-29T16:07:46.411 INFO:teuthology.orchestra.run.smithi092.stdout: debug mon = 20 2022-06-29T16:07:46.412 INFO:teuthology.orchestra.run.smithi092.stdout: debug paxos = 20 2022-06-29T16:07:46.412 INFO:teuthology.orchestra.run.smithi092.stdout: debug auth = 20 2022-06-29T16:07:46.412 INFO:teuthology.orchestra.run.smithi092.stdout: mon data avail warn = 5 2022-06-29T16:07:46.412 INFO:teuthology.orchestra.run.smithi092.stdout: mon mgr mkfs grace = 240 2022-06-29T16:07:46.412 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min pgs per osd = 4 2022-06-29T16:07:46.413 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd reporter subtree level = osd 2022-06-29T16:07:46.413 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd prime pg temp = true 2022-06-29T16:07:46.413 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min bytes per osd = 10 2022-06-29T16:07:46.413 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:46.413 INFO:teuthology.orchestra.run.smithi092.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-06-29T16:07:46.414 INFO:teuthology.orchestra.run.smithi092.stdout: auth mon ticket ttl = 660 # 11m 2022-06-29T16:07:46.414 INFO:teuthology.orchestra.run.smithi092.stdout: auth service ticket ttl = 240 # 4m 2022-06-29T16:07:46.414 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:46.414 INFO:teuthology.orchestra.run.smithi092.stdout: # don't complain about insecure global_id in the test suite 2022-06-29T16:07:46.415 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-06-29T16:07:46.415 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-06-29T16:07:46.415 INFO:teuthology.orchestra.run.smithi092.stdout: mon op complaint time = 120 2022-06-29T16:07:46.415 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-06-29T16:07:46.415 INFO:teuthology.orchestra.run.smithi092.stdout:[client] 2022-06-29T16:07:46.416 INFO:teuthology.orchestra.run.smithi092.stdout: rgw cache enabled = true 2022-06-29T16:07:46.416 INFO:teuthology.orchestra.run.smithi092.stdout: rgw enable ops log = true 2022-06-29T16:07:46.416 INFO:teuthology.orchestra.run.smithi092.stdout: rgw enable usage log = true 2022-06-29T16:07:46.416 INFO:teuthology.orchestra.run.smithi092.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-06-29T16:07:46.416 INFO:teuthology.orchestra.run.smithi092.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-06-29T16:07:46.417 INFO:teuthology.orchestra.run.smithi092.stdout: client mount timeout = 600 2022-06-29T16:07:46.417 INFO:teuthology.orchestra.run.smithi092.stdout: debug client = 20 2022-06-29T16:07:46.417 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2022-06-29T16:07:46.417 INFO:teuthology.orchestra.run.smithi092.stdout: rados mon op timeout = 15m 2022-06-29T16:07:46.417 INFO:teuthology.orchestra.run.smithi092.stdout: rados osd op timeout = 15m 2022-06-29T16:07:46.418 INFO:teuthology.orchestra.run.smithi092.stdout:[mon.a] 2022-06-29T16:07:46.418 INFO:teuthology.orchestra.run.smithi092.stdout:[mon.b] 2022-06-29T16:07:46.418 INFO:teuthology.orchestra.run.smithi092.stdout:[mon.c] 2022-06-29T16:07:46.418 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.a] 2022-06-29T16:07:46.419 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.b] 2022-06-29T16:07:46.419 INFO:teuthology.orchestra.run.smithi092.stdout:[mds] 2022-06-29T16:07:46.419 INFO:teuthology.orchestra.run.smithi092.stdout: debug mds = 20 2022-06-29T16:07:46.419 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2022-06-29T16:07:46.419 INFO:teuthology.orchestra.run.smithi092.stdout: mds bal frag = True 2022-06-29T16:07:46.420 INFO:teuthology.orchestra.run.smithi092.stdout: mds bal fragment size max = 10000 2022-06-29T16:07:46.420 INFO:teuthology.orchestra.run.smithi092.stdout: mds bal merge size = 5 2022-06-29T16:07:46.420 INFO:teuthology.orchestra.run.smithi092.stdout: mds bal split bits = 3 2022-06-29T16:07:46.420 INFO:teuthology.orchestra.run.smithi092.stdout: mds bal split size = 100 2022-06-29T16:07:46.420 INFO:teuthology.orchestra.run.smithi092.stdout: mds debug frag = True 2022-06-29T16:07:46.421 INFO:teuthology.orchestra.run.smithi092.stdout: mds debug scatterstat = True 2022-06-29T16:07:46.421 INFO:teuthology.orchestra.run.smithi092.stdout: mds op complaint time = 180 2022-06-29T16:07:46.421 INFO:teuthology.orchestra.run.smithi092.stdout: mds verify scatter = True 2022-06-29T16:07:46.421 INFO:teuthology.orchestra.run.smithi092.stdout: osd op complaint time = 180 2022-06-29T16:07:46.422 INFO:teuthology.orchestra.run.smithi092.stdout: rados mon op timeout = 15m 2022-06-29T16:07:46.422 INFO:teuthology.orchestra.run.smithi092.stdout: rados osd op timeout = 15m 2022-06-29T16:07:46.422 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-06-29T16:07:46.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.525+0000 7fcb23fff700 1 Processor -- start 2022-06-29T16:07:46.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.525+0000 7fcb23fff700 1 -- start start 2022-06-29T16:07:46.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.529+0000 7fcb23fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb240b48b0 0x7fcb24142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:46.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.529+0000 7fcb23fff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb240b0cf0 con 0x7fcb240b5eb0 2022-06-29T16:07:46.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.529+0000 7fcb23fff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb240b4d10 con 0x7fcb240b2c50 2022-06-29T16:07:46.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.529+0000 7fcb23fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb240b54e0 con 0x7fcb240b48b0 2022-06-29T16:07:46.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.529+0000 7fcb22ffd700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcb240b5eb0 0x7fcb240b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33260/0 (socket says 172.21.15.60:33260) 2022-06-29T16:07:46.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.529+0000 7fcb22ffd700 1 -- 172.21.15.60:0/4072285676 learned_addr learned my addr 172.21.15.60:0/4072285676 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:46.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.529+0000 7fcb227fc700 1 --2- 172.21.15.60:0/4072285676 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb240b48b0 0x7fcb24142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:46.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.529+0000 7fcb21ffb700 1 -- 172.21.15.60:0/4072285676 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3906652631 0 0) 0x7fcb14001110 con 0x7fcb240b5eb0 2022-06-29T16:07:46.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.529+0000 7fcb21ffb700 1 -- 172.21.15.60:0/4072285676 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb08003260 con 0x7fcb240b5eb0 2022-06-29T16:07:46.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.529+0000 7fcb21ffb700 1 -- 172.21.15.60:0/4072285676 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1882401761 0 0) 0x7fcb240b4d10 con 0x7fcb240b2c50 2022-06-29T16:07:46.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.529+0000 7fcb21ffb700 1 -- 172.21.15.60:0/4072285676 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb14001110 con 0x7fcb240b2c50 2022-06-29T16:07:46.539 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.529+0000 7fcb21ffb700 1 -- 172.21.15.60:0/4072285676 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3584873160 0 0) 0x7fcb08003260 con 0x7fcb240b5eb0 2022-06-29T16:07:46.539 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.529+0000 7fcb21ffb700 1 -- 172.21.15.60:0/4072285676 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb240b48b0 msgr2=0x7fcb24142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:46.539 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.529+0000 7fcb21ffb700 1 --2- 172.21.15.60:0/4072285676 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb240b48b0 0x7fcb24142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:46.539 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.529+0000 7fcb21ffb700 1 -- 172.21.15.60:0/4072285676 >> v1:172.21.15.92:6789/0 conn(0x7fcb240b2c50 legacy=0x7fcb240b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:46.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.529+0000 7fcb21ffb700 1 -- 172.21.15.60:0/4072285676 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb240b6290 con 0x7fcb240b5eb0 2022-06-29T16:07:46.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.529+0000 7fcb21ffb700 1 -- 172.21.15.60:0/4072285676 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7fcb140017f0 con 0x7fcb240b5eb0 2022-06-29T16:07:46.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.529+0000 7fcb21ffb700 1 -- 172.21.15.60:0/4072285676 >> v1:172.21.15.60:6789/0 conn(0x7fcb240b5eb0 legacy=0x7fcb240b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:46.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.529+0000 7fcb21ffb700 1 --2- 172.21.15.60:0/4072285676 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb08003910 0x7fcb08003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:46.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.529+0000 7fcb21ffb700 1 --2- 172.21.15.60:0/4072285676 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb240b48b0 0x7fcb08004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:46.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.529+0000 7fcb21ffb700 1 --2- 172.21.15.60:0/4072285676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb080049f0 0x7fcb08004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:46.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.529+0000 7fcb21ffb700 1 -- 172.21.15.60:0/4072285676 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb240b54e0 con 0x7fcb080049f0 2022-06-29T16:07:46.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.529+0000 7fcb21ffb700 1 -- 172.21.15.60:0/4072285676 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0800b780 con 0x7fcb08003910 2022-06-29T16:07:46.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.529+0000 7fcb21ffb700 1 -- 172.21.15.60:0/4072285676 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0800b940 con 0x7fcb240b48b0 2022-06-29T16:07:46.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.529+0000 7fcb237fe700 1 --2- 172.21.15.60:0/4072285676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb080049f0 0x7fcb08004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:46.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.529+0000 7fcb22ffd700 1 --2- 172.21.15.60:0/4072285676 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb08003910 0x7fcb08003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:46.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.529+0000 7fcb237fe700 1 -- 172.21.15.60:0/4072285676 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb240b48b0 msgr2=0x7fcb08004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:46.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.529+0000 7fcb237fe700 1 --2- 172.21.15.60:0/4072285676 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb240b48b0 0x7fcb08004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-06-29T16:07:46.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.529+0000 7fcb237fe700 1 -- 172.21.15.60:0/4072285676 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb08003910 msgr2=0x7fcb08003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:46.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.529+0000 7fcb237fe700 1 --2- 172.21.15.60:0/4072285676 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb08003910 0x7fcb08003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:46.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.529+0000 7fcb237fe700 1 -- 172.21.15.60:0/4072285676 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb08003260 con 0x7fcb080049f0 2022-06-29T16:07:46.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.533+0000 7fcb237fe700 1 --2- 172.21.15.60:0/4072285676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb080049f0 0x7fcb08004e70 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 rx=0x7fcb1c001490 tx=0x7fcb1c007d80).ready entity=mon.0 client_cookie=23b44754929d83d9 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:46.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.533+0000 7fcb21ffb700 1 -- 172.21.15.60:0/4072285676 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb1c001780 con 0x7fcb080049f0 2022-06-29T16:07:46.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.533+0000 7fcb21ffb700 1 -- 172.21.15.60:0/4072285676 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb1c006890 con 0x7fcb080049f0 2022-06-29T16:07:46.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.533+0000 7fcb21ffb700 1 -- 172.21.15.60:0/4072285676 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb1c006bf0 con 0x7fcb080049f0 2022-06-29T16:07:46.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.533+0000 7fcb23fff700 1 -- 172.21.15.60:0/4072285676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb080049f0 msgr2=0x7fcb08004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:46.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.533+0000 7fcb23fff700 1 --2- 172.21.15.60:0/4072285676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb080049f0 0x7fcb08004e70 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 rx=0x7fcb1c001490 tx=0x7fcb1c007d80).stop 2022-06-29T16:07:46.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.533+0000 7fcb237fe700 1 -- 172.21.15.60:0/4072285676 reap_dead start 2022-06-29T16:07:46.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.533+0000 7fcb23fff700 1 -- 172.21.15.60:0/4072285676 shutdown_connections 2022-06-29T16:07:46.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.533+0000 7fcb23fff700 1 -- 172.21.15.60:0/4072285676 >> 172.21.15.60:0/4072285676 conn(0x7fcb24062090 msgr2=0x7fcb24062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:46.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.533+0000 7fcb23fff700 1 -- 172.21.15.60:0/4072285676 shutdown_connections 2022-06-29T16:07:46.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.533+0000 7fcb23fff700 1 -- 172.21.15.60:0/4072285676 wait complete. 2022-06-29T16:07:46.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.537+0000 7fcb23fff700 1 Processor -- start 2022-06-29T16:07:46.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.537+0000 7fcb23fff700 1 -- start start 2022-06-29T16:07:46.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.537+0000 7fcb23fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb080049f0 0x7fcb24151b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:46.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.537+0000 7fcb23fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb241410a0 0x7fcb24148340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:46.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.537+0000 7fcb23fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb240aca20 0x7fcb240aef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:46.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.537+0000 7fcb23fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb240ae1f0 con 0x7fcb241410a0 2022-06-29T16:07:46.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.537+0000 7fcb23fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb24140680 con 0x7fcb080049f0 2022-06-29T16:07:46.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.537+0000 7fcb23fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb24143f80 con 0x7fcb240aca20 2022-06-29T16:07:46.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.537+0000 7fcb227fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb241410a0 0x7fcb24148340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:46.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.537+0000 7fcb227fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb241410a0 0x7fcb24148340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53416/0 (socket says 172.21.15.60:53416) 2022-06-29T16:07:46.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.537+0000 7fcb227fc700 1 -- 172.21.15.60:0/2136663423 learned_addr learned my addr 172.21.15.60:0/2136663423 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:46.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.537+0000 7fcb22ffd700 1 --2- 172.21.15.60:0/2136663423 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb080049f0 0x7fcb24151b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:46.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.537+0000 7fcb237fe700 1 --2- 172.21.15.60:0/2136663423 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb240aca20 0x7fcb240aef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:46.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.537+0000 7fcb227fc700 1 -- 172.21.15.60:0/2136663423 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb240aca20 msgr2=0x7fcb240aef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:46.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.537+0000 7fcb227fc700 1 --2- 172.21.15.60:0/2136663423 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb240aca20 0x7fcb240aef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:46.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.537+0000 7fcb227fc700 1 -- 172.21.15.60:0/2136663423 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb080049f0 msgr2=0x7fcb24151b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:46.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.537+0000 7fcb227fc700 1 --2- 172.21.15.60:0/2136663423 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb080049f0 0x7fcb24151b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:46.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.537+0000 7fcb227fc700 1 -- 172.21.15.60:0/2136663423 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb24144220 con 0x7fcb241410a0 2022-06-29T16:07:46.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.537+0000 7fcb227fc700 1 --2- 172.21.15.60:0/2136663423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb241410a0 0x7fcb24148340 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 rx=0x7fcb1800c000 tx=0x7fcb18006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:46.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.537+0000 7fcb13fff700 1 -- 172.21.15.60:0/2136663423 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb18016070 con 0x7fcb241410a0 2022-06-29T16:07:46.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.537+0000 7fcb13fff700 1 -- 172.21.15.60:0/2136663423 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb18000e90 con 0x7fcb241410a0 2022-06-29T16:07:46.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.537+0000 7fcb23fff700 1 -- 172.21.15.60:0/2136663423 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb241401e0 con 0x7fcb241410a0 2022-06-29T16:07:46.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.537+0000 7fcb13fff700 1 -- 172.21.15.60:0/2136663423 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb18010970 con 0x7fcb241410a0 2022-06-29T16:07:46.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.537+0000 7fcb23fff700 1 -- 172.21.15.60:0/2136663423 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb24142190 con 0x7fcb241410a0 2022-06-29T16:07:46.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.541+0000 7fcb23fff700 1 -- 172.21.15.60:0/2136663423 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb240af860 con 0x7fcb241410a0 2022-06-29T16:07:46.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.541+0000 7fcb13fff700 1 -- 172.21.15.60:0/2136663423 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcb18010ad0 con 0x7fcb241410a0 2022-06-29T16:07:46.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.541+0000 7fcb13fff700 1 --2- 172.21.15.60:0/2136663423 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcb00053a10 0x7fcb00055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:46.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.541+0000 7fcb13fff700 1 -- 172.21.15.60:0/2136663423 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fcb1806bc10 con 0x7fcb241410a0 2022-06-29T16:07:46.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.541+0000 7fcb22ffd700 1 --2- 172.21.15.60:0/2136663423 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcb00053a10 0x7fcb00055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:46.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.541+0000 7fcb22ffd700 1 --2- 172.21.15.60:0/2136663423 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcb00053a10 0x7fcb00055ed0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0x7fcb14005000 tx=0x7fcb240b1780).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:46.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.549+0000 7fcb13fff700 1 -- 172.21.15.60:0/2136663423 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcb18042d00 con 0x7fcb241410a0 2022-06-29T16:07:46.679 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.673+0000 7fcb23fff700 1 -- 172.21.15.60:0/2136663423 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fcb24140370 con 0x7fcb241410a0 2022-06-29T16:07:46.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.681+0000 7fcb13fff700 1 -- 172.21.15.60:0/2136663423 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7fcb18046320 con 0x7fcb241410a0 2022-06-29T16:07:46.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.685+0000 7fcb23fff700 1 -- 172.21.15.60:0/2136663423 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcb00053a10 msgr2=0x7fcb00055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:46.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.685+0000 7fcb23fff700 1 --2- 172.21.15.60:0/2136663423 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcb00053a10 0x7fcb00055ed0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0x7fcb14005000 tx=0x7fcb240b1780).stop 2022-06-29T16:07:46.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.685+0000 7fcb23fff700 1 -- 172.21.15.60:0/2136663423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb241410a0 msgr2=0x7fcb24148340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:46.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.685+0000 7fcb23fff700 1 --2- 172.21.15.60:0/2136663423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb241410a0 0x7fcb24148340 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 rx=0x7fcb1800c000 tx=0x7fcb18006df0).stop 2022-06-29T16:07:46.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.685+0000 7fcb23fff700 1 -- 172.21.15.60:0/2136663423 shutdown_connections 2022-06-29T16:07:46.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.685+0000 7fcb23fff700 1 --2- 172.21.15.60:0/2136663423 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcb00053a10 0x7fcb00055ed0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:46.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.685+0000 7fcb23fff700 1 --2- 172.21.15.60:0/2136663423 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcb240aca20 0x7fcb240aef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:46.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.685+0000 7fcb23fff700 1 --2- 172.21.15.60:0/2136663423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcb241410a0 0x7fcb24148340 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:46.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.685+0000 7fcb23fff700 1 --2- 172.21.15.60:0/2136663423 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcb080049f0 0x7fcb24151b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:46.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.685+0000 7fcb23fff700 1 -- 172.21.15.60:0/2136663423 >> 172.21.15.60:0/2136663423 conn(0x7fcb24062090 msgr2=0x7fcb240b6460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:46.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.685+0000 7fcb23fff700 1 -- 172.21.15.60:0/2136663423 shutdown_connections 2022-06-29T16:07:46.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.685+0000 7fcb23fff700 1 -- 172.21.15.60:0/2136663423 wait complete. 2022-06-29T16:07:46.702 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_getattr_caps' 2022-06-29T16:07:46.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.801+0000 7f1c548e6700 1 Processor -- start 2022-06-29T16:07:46.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.801+0000 7f1c548e6700 1 -- start start 2022-06-29T16:07:46.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.805+0000 7f1c548e6700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1c5005d070 0x7f1c5014aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:46.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.805+0000 7f1c548e6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c5014af80 0x7f1c500ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:46.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.805+0000 7f1c548e6700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c5005d450 con 0x7f1c500b2130 2022-06-29T16:07:46.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.805+0000 7f1c548e6700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c500adf90 con 0x7f1c5005d070 2022-06-29T16:07:46.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.805+0000 7f1c548e6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c500b25d0 con 0x7f1c5014af80 2022-06-29T16:07:46.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.805+0000 7f1c4ed9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1c5005d070 0x7f1c5014aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:46.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.805+0000 7f1c4ed9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1c5005d070 0x7f1c5014aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53428/0 (socket says 172.21.15.60:53428) 2022-06-29T16:07:46.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.805+0000 7f1c4ed9d700 1 -- 172.21.15.60:0/2568981797 learned_addr learned my addr 172.21.15.60:0/2568981797 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:46.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.805+0000 7f1c4f59e700 1 --2- 172.21.15.60:0/2568981797 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c5014af80 0x7f1c500ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:46.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.805+0000 7f1c4ed9d700 1 -- 172.21.15.60:0/2568981797 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c5014af80 msgr2=0x7f1c500ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:46.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.805+0000 7f1c4ed9d700 1 --2- 172.21.15.60:0/2568981797 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c5014af80 0x7f1c500ad130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:46.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.805+0000 7f1c4ed9d700 1 -- 172.21.15.60:0/2568981797 >> v1:172.21.15.92:6789/0 conn(0x7f1c500b2130 legacy=0x7f1c500aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:46.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.805+0000 7f1c4ed9d700 1 -- 172.21.15.60:0/2568981797 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c5014a350 con 0x7f1c5005d070 2022-06-29T16:07:46.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.805+0000 7f1c4ed9d700 1 --2- 172.21.15.60:0/2568981797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c5005d070 0x7f1c5014aa90 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 rx=0x7f1c40002f70 tx=0x7f1c40006eb0).ready entity=mon.0 client_cookie=322e00348b67a372 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:46.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.805+0000 7f1c4dd9b700 1 -- 172.21.15.60:0/2568981797 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c4000d570 con 0x7f1c5005d070 2022-06-29T16:07:46.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.805+0000 7f1c4dd9b700 1 -- 172.21.15.60:0/2568981797 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c40000e90 con 0x7f1c5005d070 2022-06-29T16:07:46.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.805+0000 7f1c4dd9b700 1 -- 172.21.15.60:0/2568981797 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c4000d900 con 0x7f1c5005d070 2022-06-29T16:07:46.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.805+0000 7f1c548e6700 1 -- 172.21.15.60:0/2568981797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c5005d070 msgr2=0x7f1c5014aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:46.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.805+0000 7f1c548e6700 1 --2- 172.21.15.60:0/2568981797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c5005d070 0x7f1c5014aa90 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 rx=0x7f1c40002f70 tx=0x7f1c40006eb0).stop 2022-06-29T16:07:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.805+0000 7f1c548e6700 1 -- 172.21.15.60:0/2568981797 shutdown_connections 2022-06-29T16:07:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.805+0000 7f1c548e6700 1 --2- 172.21.15.60:0/2568981797 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c5014af80 0x7f1c500ad130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.805+0000 7f1c548e6700 1 --2- 172.21.15.60:0/2568981797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c5005d070 0x7f1c5014aa90 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.805+0000 7f1c548e6700 1 -- 172.21.15.60:0/2568981797 >> 172.21.15.60:0/2568981797 conn(0x7f1c500b5890 msgr2=0x7f1c500b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.805+0000 7f1c548e6700 1 -- 172.21.15.60:0/2568981797 shutdown_connections 2022-06-29T16:07:46.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.805+0000 7f1c548e6700 1 -- 172.21.15.60:0/2568981797 wait complete. 2022-06-29T16:07:46.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.809+0000 7f1c548e6700 1 Processor -- start 2022-06-29T16:07:46.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.809+0000 7f1c548e6700 1 -- start start 2022-06-29T16:07:46.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.809+0000 7f1c548e6700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c5005d070 0x7f1c500666c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:46.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.809+0000 7f1c548e6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c500b2130 0x7f1c50141ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:46.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.809+0000 7f1c548e6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c5014af80 0x7f1c501423e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:46.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.809+0000 7f1c548e6700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c50066ed0 con 0x7f1c500b2130 2022-06-29T16:07:46.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.809+0000 7f1c548e6700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c50148f70 con 0x7f1c5005d070 2022-06-29T16:07:46.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.809+0000 7f1c548e6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c50157360 con 0x7f1c5014af80 2022-06-29T16:07:46.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.809+0000 7f1c4e59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c500b2130 0x7f1c50141ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:46.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.809+0000 7f1c4e59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c500b2130 0x7f1c50141ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53432/0 (socket says 172.21.15.60:53432) 2022-06-29T16:07:46.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.809+0000 7f1c4e59c700 1 -- 172.21.15.60:0/313797164 learned_addr learned my addr 172.21.15.60:0/313797164 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:46.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.809+0000 7f1c4ed9d700 1 --2- 172.21.15.60:0/313797164 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c5005d070 0x7f1c500666c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:46.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.809+0000 7f1c4f59e700 1 --2- 172.21.15.60:0/313797164 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c5014af80 0x7f1c501423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:46.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.809+0000 7f1c4e59c700 1 -- 172.21.15.60:0/313797164 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c5014af80 msgr2=0x7f1c501423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:46.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.809+0000 7f1c4e59c700 1 --2- 172.21.15.60:0/313797164 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c5014af80 0x7f1c501423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:46.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.809+0000 7f1c4e59c700 1 -- 172.21.15.60:0/313797164 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c5005d070 msgr2=0x7f1c500666c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:46.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.809+0000 7f1c4e59c700 1 --2- 172.21.15.60:0/313797164 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c5005d070 0x7f1c500666c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:46.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.809+0000 7f1c4e59c700 1 -- 172.21.15.60:0/313797164 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c4000b1e0 con 0x7f1c500b2130 2022-06-29T16:07:46.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.809+0000 7f1c4f59e700 1 --2- 172.21.15.60:0/313797164 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c5014af80 0x7f1c501423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:46.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.809+0000 7f1c4ed9d700 1 --2- 172.21.15.60:0/313797164 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c5005d070 0x7f1c500666c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:46.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.809+0000 7f1c4e59c700 1 --2- 172.21.15.60:0/313797164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c500b2130 0x7f1c50141ef0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 rx=0x7f1c50061990 tx=0x7f1c38007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:46.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.809+0000 7f1c47fff700 1 -- 172.21.15.60:0/313797164 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c3800ce30 con 0x7f1c500b2130 2022-06-29T16:07:46.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.809+0000 7f1c47fff700 1 -- 172.21.15.60:0/313797164 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c3800f470 con 0x7f1c500b2130 2022-06-29T16:07:46.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.809+0000 7f1c47fff700 1 -- 172.21.15.60:0/313797164 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c3800f7d0 con 0x7f1c500b2130 2022-06-29T16:07:46.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.809+0000 7f1c548e6700 1 -- 172.21.15.60:0/313797164 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c501428d0 con 0x7f1c500b2130 2022-06-29T16:07:46.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.809+0000 7f1c548e6700 1 -- 172.21.15.60:0/313797164 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c5014e620 con 0x7f1c500b2130 2022-06-29T16:07:46.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.813+0000 7f1c548e6700 1 -- 172.21.15.60:0/313797164 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c5014a5f0 con 0x7f1c500b2130 2022-06-29T16:07:46.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.813+0000 7f1c47fff700 1 -- 172.21.15.60:0/313797164 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1c38019030 con 0x7f1c500b2130 2022-06-29T16:07:46.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.813+0000 7f1c47fff700 1 --2- 172.21.15.60:0/313797164 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1c34053a10 0x7f1c34055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:46.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.813+0000 7f1c47fff700 1 -- 172.21.15.60:0/313797164 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f1c3806b150 con 0x7f1c500b2130 2022-06-29T16:07:46.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.813+0000 7f1c4ed9d700 1 --2- 172.21.15.60:0/313797164 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1c34053a10 0x7f1c34055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:46.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.817+0000 7f1c4ed9d700 1 --2- 172.21.15.60:0/313797164 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1c34053a10 0x7f1c34055ed0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7f1c4001a000 tx=0x7f1c40015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:46.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.821+0000 7f1c47fff700 1 -- 172.21.15.60:0/313797164 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1c38042240 con 0x7f1c500b2130 2022-06-29T16:07:46.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:46.989+0000 7f1c548e6700 1 -- 172.21.15.60:0/313797164 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_getattr_caps"]} v 0) v1 -- 0x7f1c500573d0 con 0x7f1c500b2130 2022-06-29T16:07:47.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.213+0000 7f1c47fff700 1 -- 172.21.15.60:0/313797164 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_getattr_caps"]}]=0 v432) v1 ==== 128+0+0 (secure 0 0 0) 0x7f1c38045860 con 0x7f1c500b2130 2022-06-29T16:07:47.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.217+0000 7f1c548e6700 1 -- 172.21.15.60:0/313797164 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1c34053a10 msgr2=0x7f1c34055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:47.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.217+0000 7f1c548e6700 1 --2- 172.21.15.60:0/313797164 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1c34053a10 0x7f1c34055ed0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7f1c4001a000 tx=0x7f1c40015000).stop 2022-06-29T16:07:47.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.217+0000 7f1c548e6700 1 -- 172.21.15.60:0/313797164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c500b2130 msgr2=0x7f1c50141ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:47.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.217+0000 7f1c548e6700 1 --2- 172.21.15.60:0/313797164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c500b2130 0x7f1c50141ef0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 rx=0x7f1c50061990 tx=0x7f1c38007c50).stop 2022-06-29T16:07:47.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.217+0000 7f1c548e6700 1 -- 172.21.15.60:0/313797164 shutdown_connections 2022-06-29T16:07:47.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.217+0000 7f1c548e6700 1 --2- 172.21.15.60:0/313797164 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1c34053a10 0x7f1c34055ed0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:47.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.217+0000 7f1c548e6700 1 --2- 172.21.15.60:0/313797164 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1c5014af80 0x7f1c501423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:47.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.217+0000 7f1c548e6700 1 --2- 172.21.15.60:0/313797164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1c500b2130 0x7f1c50141ef0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:47.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.217+0000 7f1c548e6700 1 --2- 172.21.15.60:0/313797164 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c5005d070 0x7f1c500666c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:47.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.217+0000 7f1c548e6700 1 -- 172.21.15.60:0/313797164 >> 172.21.15.60:0/313797164 conn(0x7f1c500b5890 msgr2=0x7f1c50143c70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:47.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.217+0000 7f1c548e6700 1 -- 172.21.15.60:0/313797164 shutdown_connections 2022-06-29T16:07:47.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.217+0000 7f1c548e6700 1 -- 172.21.15.60:0/313797164 wait complete. 2022-06-29T16:07:47.239 INFO:tasks.cephfs_test_runner:test_getattr_caps (tasks.cephfs.test_misc.TestMisc) ... ok 2022-06-29T16:07:47.239 INFO:tasks.cephfs_test_runner:Starting test: test_root_rctime (tasks.cephfs.test_misc.TestMisc) 2022-06-29T16:07:47.240 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_root_rctime' 2022-06-29T16:07:47.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.345+0000 7fe472c53700 1 Processor -- start 2022-06-29T16:07:47.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.345+0000 7fe472c53700 1 -- start start 2022-06-29T16:07:47.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.345+0000 7fe472c53700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe46c05d070 0x7fe46c14aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:47.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.349+0000 7fe472c53700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe46c0b2130 0x7fe46c0aca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:47.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.349+0000 7fe472c53700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe46c14af80 0x7fe46c0acf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:47.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.349+0000 7fe472c53700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe46c0adf90 con 0x7fe46c0b2130 2022-06-29T16:07:47.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.349+0000 7fe472c53700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe46c05d450 con 0x7fe46c05d070 2022-06-29T16:07:47.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.349+0000 7fe472c53700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe46c0b25d0 con 0x7fe46c14af80 2022-06-29T16:07:47.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.349+0000 7fe471450700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe46c0b2130 0x7fe46c0aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:47.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.349+0000 7fe471450700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe46c0b2130 0x7fe46c0aca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53438/0 (socket says 172.21.15.60:53438) 2022-06-29T16:07:47.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.349+0000 7fe471450700 1 -- 172.21.15.60:0/254169670 learned_addr learned my addr 172.21.15.60:0/254169670 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:47.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.349+0000 7fe471c51700 1 --2- 172.21.15.60:0/254169670 >> v2:172.21.15.92:3300/0 conn(0x7fe46c05d070 0x7fe46c14aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:47.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.349+0000 7fe472452700 1 --2- 172.21.15.60:0/254169670 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe46c14af80 0x7fe46c0acf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:47.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.349+0000 7fe471450700 1 -- 172.21.15.60:0/254169670 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe46c14af80 msgr2=0x7fe46c0acf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:47.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.349+0000 7fe471450700 1 --2- 172.21.15.60:0/254169670 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe46c14af80 0x7fe46c0acf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:47.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.349+0000 7fe471450700 1 -- 172.21.15.60:0/254169670 >> v2:172.21.15.92:3300/0 conn(0x7fe46c05d070 msgr2=0x7fe46c14aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:47.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.349+0000 7fe471450700 1 --2- 172.21.15.60:0/254169670 >> v2:172.21.15.92:3300/0 conn(0x7fe46c05d070 0x7fe46c14aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:47.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.349+0000 7fe471450700 1 -- 172.21.15.60:0/254169670 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe46c14a350 con 0x7fe46c0b2130 2022-06-29T16:07:47.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.349+0000 7fe471450700 1 --2- 172.21.15.60:0/254169670 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe46c0b2130 0x7fe46c0aca20 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 rx=0x7fe4680032f0 tx=0x7fe468009290).ready entity=mon.0 client_cookie=12b88c925947eeb4 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:47.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.349+0000 7fe470c4f700 1 -- 172.21.15.60:0/254169670 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe46800b680 con 0x7fe46c0b2130 2022-06-29T16:07:47.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.349+0000 7fe470c4f700 1 -- 172.21.15.60:0/254169670 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe46800b7e0 con 0x7fe46c0b2130 2022-06-29T16:07:47.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.349+0000 7fe470c4f700 1 -- 172.21.15.60:0/254169670 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe46800b950 con 0x7fe46c0b2130 2022-06-29T16:07:47.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.349+0000 7fe472c53700 1 -- 172.21.15.60:0/254169670 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe46c0b2130 msgr2=0x7fe46c0aca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:47.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.349+0000 7fe472c53700 1 --2- 172.21.15.60:0/254169670 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe46c0b2130 0x7fe46c0aca20 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 rx=0x7fe4680032f0 tx=0x7fe468009290).stop 2022-06-29T16:07:47.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.349+0000 7fe472c53700 1 -- 172.21.15.60:0/254169670 shutdown_connections 2022-06-29T16:07:47.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.349+0000 7fe472c53700 1 --2- 172.21.15.60:0/254169670 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe46c14af80 0x7fe46c0acf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:47.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.349+0000 7fe472c53700 1 --2- 172.21.15.60:0/254169670 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe46c0b2130 0x7fe46c0aca20 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:47.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.349+0000 7fe472c53700 1 --2- 172.21.15.60:0/254169670 >> v2:172.21.15.92:3300/0 conn(0x7fe46c05d070 0x7fe46c14aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:47.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.349+0000 7fe472c53700 1 -- 172.21.15.60:0/254169670 >> 172.21.15.60:0/254169670 conn(0x7fe46c0b5890 msgr2=0x7fe46c0b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:47.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.349+0000 7fe472c53700 1 -- 172.21.15.60:0/254169670 shutdown_connections 2022-06-29T16:07:47.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.349+0000 7fe472c53700 1 -- 172.21.15.60:0/254169670 wait complete. 2022-06-29T16:07:47.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.353+0000 7fe472c53700 1 Processor -- start 2022-06-29T16:07:47.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.353+0000 7fe472c53700 1 -- start start 2022-06-29T16:07:47.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.353+0000 7fe472c53700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe46c05d070 0x7fe46c066230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:47.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.353+0000 7fe472c53700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe46c0b2130 0x7fe46c0b41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:47.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.353+0000 7fe472c53700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe46c14af80 0x7fe46c159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:47.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.353+0000 7fe472c53700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe46c066860 con 0x7fe46c0b2130 2022-06-29T16:07:47.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.353+0000 7fe472c53700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe46c066b00 con 0x7fe46c14af80 2022-06-29T16:07:47.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.353+0000 7fe472c53700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe46c066da0 con 0x7fe46c05d070 2022-06-29T16:07:47.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.353+0000 7fe471450700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe46c0b2130 0x7fe46c0b41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:47.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.353+0000 7fe471450700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe46c0b2130 0x7fe46c0b41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53446/0 (socket says 172.21.15.60:53446) 2022-06-29T16:07:47.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.353+0000 7fe471450700 1 -- 172.21.15.60:0/1271719053 learned_addr learned my addr 172.21.15.60:0/1271719053 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:47.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.353+0000 7fe471c51700 1 --2- 172.21.15.60:0/1271719053 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe46c05d070 0x7fe46c066230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:47.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.353+0000 7fe472452700 1 --2- 172.21.15.60:0/1271719053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe46c14af80 0x7fe46c159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:47.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.353+0000 7fe471450700 1 -- 172.21.15.60:0/1271719053 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe46c05d070 msgr2=0x7fe46c066230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:47.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.353+0000 7fe471450700 1 --2- 172.21.15.60:0/1271719053 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe46c05d070 0x7fe46c066230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:47.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.353+0000 7fe471450700 1 -- 172.21.15.60:0/1271719053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe46c14af80 msgr2=0x7fe46c159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:47.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.353+0000 7fe471450700 1 --2- 172.21.15.60:0/1271719053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe46c14af80 0x7fe46c159ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:47.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.353+0000 7fe471450700 1 -- 172.21.15.60:0/1271719053 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe468009530 con 0x7fe46c0b2130 2022-06-29T16:07:47.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.353+0000 7fe471450700 1 --2- 172.21.15.60:0/1271719053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe46c0b2130 0x7fe46c0b41e0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 rx=0x7fe4680032f0 tx=0x7fe468017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:47.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.353+0000 7fe45affd700 1 -- 172.21.15.60:0/1271719053 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe46800b680 con 0x7fe46c0b2130 2022-06-29T16:07:47.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.353+0000 7fe45affd700 1 -- 172.21.15.60:0/1271719053 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe468011e50 con 0x7fe46c0b2130 2022-06-29T16:07:47.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.353+0000 7fe472c53700 1 -- 172.21.15.60:0/1271719053 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe46c159790 con 0x7fe46c0b2130 2022-06-29T16:07:47.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.353+0000 7fe45affd700 1 -- 172.21.15.60:0/1271719053 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe46801b810 con 0x7fe46c0b2130 2022-06-29T16:07:47.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.353+0000 7fe472c53700 1 -- 172.21.15.60:0/1271719053 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe46c0b48f0 con 0x7fe46c0b2130 2022-06-29T16:07:47.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.357+0000 7fe472c53700 1 -- 172.21.15.60:0/1271719053 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe46c14a5f0 con 0x7fe46c0b2130 2022-06-29T16:07:47.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.357+0000 7fe45affd700 1 -- 172.21.15.60:0/1271719053 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe46800f070 con 0x7fe46c0b2130 2022-06-29T16:07:47.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.357+0000 7fe45affd700 1 --2- 172.21.15.60:0/1271719053 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe44c053a10 0x7fe44c055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:47.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.357+0000 7fe45affd700 1 -- 172.21.15.60:0/1271719053 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fe46806d1a0 con 0x7fe46c0b2130 2022-06-29T16:07:47.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.357+0000 7fe471c51700 1 --2- 172.21.15.60:0/1271719053 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe44c053a10 0x7fe44c055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:47.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.357+0000 7fe471c51700 1 --2- 172.21.15.60:0/1271719053 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe44c053a10 0x7fe44c055ed0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0x7fe45c00b000 tx=0x7fe45c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:47.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.361+0000 7fe45affd700 1 -- 172.21.15.60:0/1271719053 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe468044290 con 0x7fe46c0b2130 2022-06-29T16:07:47.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:47.533+0000 7fe472c53700 1 -- 172.21.15.60:0/1271719053 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_root_rctime"]} v 0) v1 -- 0x7fe46c0573d0 con 0x7fe46c0b2130 2022-06-29T16:07:48.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.217+0000 7fe45affd700 1 -- 172.21.15.60:0/1271719053 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_root_rctime"]}]=0 v433) v1 ==== 130+0+0 (secure 0 0 0) 0x7fe46801f070 con 0x7fe46c0b2130 2022-06-29T16:07:48.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.221+0000 7fe472c53700 1 -- 172.21.15.60:0/1271719053 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe44c053a10 msgr2=0x7fe44c055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:48.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.221+0000 7fe472c53700 1 --2- 172.21.15.60:0/1271719053 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe44c053a10 0x7fe44c055ed0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0x7fe45c00b000 tx=0x7fe45c006df0).stop 2022-06-29T16:07:48.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.221+0000 7fe472c53700 1 -- 172.21.15.60:0/1271719053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe46c0b2130 msgr2=0x7fe46c0b41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:48.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.221+0000 7fe472c53700 1 --2- 172.21.15.60:0/1271719053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe46c0b2130 0x7fe46c0b41e0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 rx=0x7fe4680032f0 tx=0x7fe468017000).stop 2022-06-29T16:07:48.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.221+0000 7fe472c53700 1 -- 172.21.15.60:0/1271719053 shutdown_connections 2022-06-29T16:07:48.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.221+0000 7fe472c53700 1 --2- 172.21.15.60:0/1271719053 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe44c053a10 0x7fe44c055ed0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:48.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.221+0000 7fe472c53700 1 --2- 172.21.15.60:0/1271719053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe46c14af80 0x7fe46c159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:48.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.221+0000 7fe472c53700 1 --2- 172.21.15.60:0/1271719053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe46c0b2130 0x7fe46c0b41e0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:48.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.221+0000 7fe472c53700 1 --2- 172.21.15.60:0/1271719053 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe46c05d070 0x7fe46c066230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:48.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.221+0000 7fe472c53700 1 -- 172.21.15.60:0/1271719053 >> 172.21.15.60:0/1271719053 conn(0x7fe46c0b5890 msgr2=0x7fe46c143340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:48.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.221+0000 7fe472c53700 1 -- 172.21.15.60:0/1271719053 shutdown_connections 2022-06-29T16:07:48.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.221+0000 7fe472c53700 1 -- 172.21.15.60:0/1271719053 wait complete. 2022-06-29T16:07:48.243 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-06-29T16:07:48.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.349+0000 7f22937a4700 1 Processor -- start 2022-06-29T16:07:48.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.349+0000 7f22937a4700 1 -- start start 2022-06-29T16:07:48.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.349+0000 7f22937a4700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f228c0ae7c0 0x7f228c141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:48.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.349+0000 7f22937a4700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f228c0b4010 0x7f228c142490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:48.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.349+0000 7f22937a4700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f228c143a40 con 0x7f228c146160 2022-06-29T16:07:48.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.349+0000 7f22937a4700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f228c0b56d0 con 0x7f228c0ae7c0 2022-06-29T16:07:48.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.349+0000 7f22937a4700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f228c0aee60 con 0x7f228c0b4010 2022-06-29T16:07:48.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.349+0000 7f2291540700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f228c0ae7c0 0x7f228c141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:48.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.349+0000 7f2291d41700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f228c146160 0x7f228c146590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33312/0 (socket says 172.21.15.60:33312) 2022-06-29T16:07:48.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.349+0000 7f2291d41700 1 -- 172.21.15.60:0/1797189962 learned_addr learned my addr 172.21.15.60:0/1797189962 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:48.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.349+0000 7f2290d3f700 1 --2- 172.21.15.60:0/1797189962 >> v2:172.21.15.92:3300/0 conn(0x7f228c0b4010 0x7f228c142490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:48.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.353+0000 7f2277fff700 1 -- 172.21.15.60:0/1797189962 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4215265374 0 0) 0x7f228c143a40 con 0x7f228c146160 2022-06-29T16:07:48.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.353+0000 7f2277fff700 1 -- 172.21.15.60:0/1797189962 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2270003260 con 0x7f228c146160 2022-06-29T16:07:48.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.353+0000 7f2291540700 1 -- 172.21.15.60:0/1797189962 >> v2:172.21.15.92:3300/0 conn(0x7f228c0b4010 msgr2=0x7f228c142490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:48.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.353+0000 7f2291540700 1 --2- 172.21.15.60:0/1797189962 >> v2:172.21.15.92:3300/0 conn(0x7f228c0b4010 0x7f228c142490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:48.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.353+0000 7f2291540700 1 -- 172.21.15.60:0/1797189962 >> v1:172.21.15.60:6789/0 conn(0x7f228c146160 legacy=0x7f228c146590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:48.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.353+0000 7f2291540700 1 -- 172.21.15.60:0/1797189962 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f228c0af100 con 0x7f228c0ae7c0 2022-06-29T16:07:48.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.353+0000 7f2290d3f700 1 --2- 172.21.15.60:0/1797189962 >> v2:172.21.15.92:3300/0 conn(0x7f228c0b4010 0x7f228c142490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:48.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.353+0000 7f2291540700 1 --2- 172.21.15.60:0/1797189962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f228c0ae7c0 0x7f228c141fa0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 rx=0x7f2288002f70 tx=0x7f2288006df0).ready entity=mon.0 client_cookie=e6e3c6cfcc976573 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:48.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.353+0000 7f2277fff700 1 -- 172.21.15.60:0/1797189962 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f228800d570 con 0x7f228c0ae7c0 2022-06-29T16:07:48.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.353+0000 7f2277fff700 1 -- 172.21.15.60:0/1797189962 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2288000e90 con 0x7f228c0ae7c0 2022-06-29T16:07:48.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.353+0000 7f2277fff700 1 -- 172.21.15.60:0/1797189962 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f228800d8c0 con 0x7f228c0ae7c0 2022-06-29T16:07:48.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.353+0000 7f22937a4700 1 -- 172.21.15.60:0/1797189962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f228c0ae7c0 msgr2=0x7f228c141fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:48.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.353+0000 7f22937a4700 1 --2- 172.21.15.60:0/1797189962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f228c0ae7c0 0x7f228c141fa0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 rx=0x7f2288002f70 tx=0x7f2288006df0).stop 2022-06-29T16:07:48.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.353+0000 7f22937a4700 1 -- 172.21.15.60:0/1797189962 shutdown_connections 2022-06-29T16:07:48.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.353+0000 7f22937a4700 1 --2- 172.21.15.60:0/1797189962 >> v2:172.21.15.92:3300/0 conn(0x7f228c0b4010 0x7f228c142490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:48.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.353+0000 7f22937a4700 1 --2- 172.21.15.60:0/1797189962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f228c0ae7c0 0x7f228c141fa0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:48.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.353+0000 7f22937a4700 1 -- 172.21.15.60:0/1797189962 >> 172.21.15.60:0/1797189962 conn(0x7f228c0b9340 msgr2=0x7f228c0b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:48.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.353+0000 7f22937a4700 1 -- 172.21.15.60:0/1797189962 shutdown_connections 2022-06-29T16:07:48.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.353+0000 7f22937a4700 1 -- 172.21.15.60:0/1797189962 wait complete. 2022-06-29T16:07:48.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.353+0000 7f22937a4700 1 Processor -- start 2022-06-29T16:07:48.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.353+0000 7f22937a4700 1 -- start start 2022-06-29T16:07:48.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.353+0000 7f22937a4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f228c0ae7c0 0x7f228c144760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:48.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.353+0000 7f22937a4700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f228c0b4010 0x7f228c0b14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:48.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.353+0000 7f22937a4700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f228c146160 0x7f228c140930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:48.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.353+0000 7f22937a4700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228c0afcb0 con 0x7f228c0ae7c0 2022-06-29T16:07:48.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.353+0000 7f22937a4700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228c0b19b0 con 0x7f228c0b4010 2022-06-29T16:07:48.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.353+0000 7f22937a4700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f228c0b1bf0 con 0x7f228c146160 2022-06-29T16:07:48.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.353+0000 7f2291540700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f228c0ae7c0 0x7f228c144760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:48.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.353+0000 7f2291540700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f228c0ae7c0 0x7f228c144760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53460/0 (socket says 172.21.15.60:53460) 2022-06-29T16:07:48.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.353+0000 7f2291540700 1 -- 172.21.15.60:0/1560439582 learned_addr learned my addr 172.21.15.60:0/1560439582 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:48.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.353+0000 7f2290d3f700 1 --2- 172.21.15.60:0/1560439582 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f228c0b4010 0x7f228c0b14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:48.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.353+0000 7f2291d41700 1 --2- 172.21.15.60:0/1560439582 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f228c146160 0x7f228c140930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:48.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.357+0000 7f2291540700 1 -- 172.21.15.60:0/1560439582 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f228c146160 msgr2=0x7f228c140930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:48.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.357+0000 7f2291540700 1 --2- 172.21.15.60:0/1560439582 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f228c146160 0x7f228c140930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:48.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.357+0000 7f2291540700 1 -- 172.21.15.60:0/1560439582 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f228c0b4010 msgr2=0x7f228c0b14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:48.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.357+0000 7f2291540700 1 --2- 172.21.15.60:0/1560439582 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f228c0b4010 0x7f228c0b14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:48.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.357+0000 7f2291540700 1 -- 172.21.15.60:0/1560439582 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f228800b180 con 0x7f228c0ae7c0 2022-06-29T16:07:48.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.357+0000 7f2290d3f700 1 --2- 172.21.15.60:0/1560439582 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f228c0b4010 0x7f228c0b14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:07:48.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.357+0000 7f2291540700 1 --2- 172.21.15.60:0/1560439582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f228c0ae7c0 0x7f228c144760 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 rx=0x7f2288002f70 tx=0x7f2288014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:48.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.357+0000 7f227effd700 1 -- 172.21.15.60:0/1560439582 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2288016070 con 0x7f228c0ae7c0 2022-06-29T16:07:48.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.357+0000 7f227effd700 1 -- 172.21.15.60:0/1560439582 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22880138d0 con 0x7f228c0ae7c0 2022-06-29T16:07:48.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.357+0000 7f227effd700 1 -- 172.21.15.60:0/1560439582 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2288013c30 con 0x7f228c0ae7c0 2022-06-29T16:07:48.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.357+0000 7f22937a4700 1 -- 172.21.15.60:0/1560439582 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f228c0af770 con 0x7f228c0ae7c0 2022-06-29T16:07:48.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.357+0000 7f22937a4700 1 -- 172.21.15.60:0/1560439582 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f228c155fc0 con 0x7f228c0ae7c0 2022-06-29T16:07:48.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.357+0000 7f22937a4700 1 -- 172.21.15.60:0/1560439582 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f228c0afad0 con 0x7f228c0ae7c0 2022-06-29T16:07:48.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.357+0000 7f227effd700 1 -- 172.21.15.60:0/1560439582 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f228800f030 con 0x7f228c0ae7c0 2022-06-29T16:07:48.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.357+0000 7f227effd700 1 --2- 172.21.15.60:0/1560439582 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2268053a70 0x7f2268055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:48.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.357+0000 7f227effd700 1 -- 172.21.15.60:0/1560439582 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f228806c8d0 con 0x7f228c0ae7c0 2022-06-29T16:07:48.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.361+0000 7f2290d3f700 1 --2- 172.21.15.60:0/1560439582 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2268053a70 0x7f2268055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:48.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.361+0000 7f2290d3f700 1 --2- 172.21.15.60:0/1560439582 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2268053a70 0x7f2268055f30 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0x7f228000c000 tx=0x7f2280008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:48.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.365+0000 7f227effd700 1 -- 172.21.15.60:0/1560439582 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f22880439c0 con 0x7f228c0ae7c0 2022-06-29T16:07:48.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.489+0000 7f22937a4700 1 -- 172.21.15.60:0/1560439582 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f228c0573d0 con 0x7f228c0ae7c0 2022-06-29T16:07:48.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.497+0000 7f227effd700 1 -- 172.21.15.60:0/1560439582 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2288046fe0 con 0x7f228c0ae7c0 2022-06-29T16:07:48.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.501+0000 7f22937a4700 1 -- 172.21.15.60:0/1560439582 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2268053a70 msgr2=0x7f2268055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:48.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.501+0000 7f22937a4700 1 --2- 172.21.15.60:0/1560439582 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2268053a70 0x7f2268055f30 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0x7f228000c000 tx=0x7f2280008040).stop 2022-06-29T16:07:48.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.501+0000 7f22937a4700 1 -- 172.21.15.60:0/1560439582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f228c0ae7c0 msgr2=0x7f228c144760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:48.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.501+0000 7f22937a4700 1 --2- 172.21.15.60:0/1560439582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f228c0ae7c0 0x7f228c144760 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 rx=0x7f2288002f70 tx=0x7f2288014000).stop 2022-06-29T16:07:48.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.501+0000 7f22937a4700 1 -- 172.21.15.60:0/1560439582 shutdown_connections 2022-06-29T16:07:48.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.501+0000 7f22937a4700 1 --2- 172.21.15.60:0/1560439582 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2268053a70 0x7f2268055f30 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:48.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.501+0000 7f22937a4700 1 --2- 172.21.15.60:0/1560439582 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f228c146160 0x7f228c140930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:48.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.501+0000 7f22937a4700 1 --2- 172.21.15.60:0/1560439582 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f228c0b4010 0x7f228c0b14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:48.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.501+0000 7f22937a4700 1 --2- 172.21.15.60:0/1560439582 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f228c0ae7c0 0x7f228c144760 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:48.510 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.501+0000 7f22937a4700 1 -- 172.21.15.60:0/1560439582 >> 172.21.15.60:0/1560439582 conn(0x7f228c0b9340 msgr2=0x7f228c0b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:48.510 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.501+0000 7f22937a4700 1 -- 172.21.15.60:0/1560439582 shutdown_connections 2022-06-29T16:07:48.510 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.501+0000 7f22937a4700 1 -- 172.21.15.60:0/1560439582 wait complete. 2022-06-29T16:07:48.522 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-06-29T16:07:48.523 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:07:48.541 DEBUG:teuthology.orchestra.run.smithi092:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:07:48.563 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:07:48.583 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:07:48.604 DEBUG:teuthology.orchestra.run.smithi197:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:07:48.625 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-06-29T16:07:48.626 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:48.626 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:07:48.627 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:07:48.634 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-06-29T16:07:48.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:07:48.636 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-06-29T16:07:48.637 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:48.637 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:07:48.637 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:07:48.645 INFO:teuthology.orchestra.run.smithi156.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-06-29T16:07:48.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:07:48.646 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-06-29T16:07:48.647 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:48.647 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:07:48.648 DEBUG:teuthology.orchestra.run.smithi197:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T16:07:48.655 INFO:teuthology.orchestra.run.smithi197.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-06-29T16:07:48.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:07:48.657 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-06-29T16:07:48.657 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-06-29T16:07:48.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.765+0000 7fdefb9ea700 1 Processor -- start 2022-06-29T16:07:48.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.765+0000 7fdefb9ea700 1 -- start start 2022-06-29T16:07:48.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.769+0000 7fdefb9ea700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdef4063ad0 0x7fdef4067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:48.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.769+0000 7fdefb9ea700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdef414af10 0x7fdef405c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:48.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.769+0000 7fdefb9ea700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdef40acba0 con 0x7fdef40b5950 2022-06-29T16:07:48.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.769+0000 7fdefb9ea700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdef414b3a0 con 0x7fdef4063ad0 2022-06-29T16:07:48.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.769+0000 7fdefb9ea700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef4067220 con 0x7fdef414af10 2022-06-29T16:07:48.777 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.769+0000 7fdef9f87700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdef414af10 0x7fdef405c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:48.777 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.769+0000 7fdef9786700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fdef40b5950 0x7fdef405bd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:37570/0 (socket says 172.21.15.60:37570) 2022-06-29T16:07:48.777 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.769+0000 7fdef8f85700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdef4063ad0 0x7fdef4067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:48.777 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.769+0000 7fdef9786700 1 -- 172.21.15.60:0/248276800 learned_addr learned my addr 172.21.15.60:0/248276800 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:48.778 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.769+0000 7fdeebfff700 1 -- 172.21.15.60:0/248276800 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2319400842 0 0) 0x7fdef40acba0 con 0x7fdef40b5950 2022-06-29T16:07:48.778 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.769+0000 7fdeebfff700 1 -- 172.21.15.60:0/248276800 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fded8003260 con 0x7fdef40b5950 2022-06-29T16:07:48.778 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.769+0000 7fdef9f87700 1 -- 172.21.15.60:0/248276800 >> v2:172.21.15.92:3300/0 conn(0x7fdef4063ad0 msgr2=0x7fdef4067680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:48.778 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.769+0000 7fdef9f87700 1 --2- 172.21.15.60:0/248276800 >> v2:172.21.15.92:3300/0 conn(0x7fdef4063ad0 0x7fdef4067680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:48.778 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.769+0000 7fdef9f87700 1 -- 172.21.15.60:0/248276800 >> v1:172.21.15.92:6789/0 conn(0x7fdef40b5950 legacy=0x7fdef405bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:48.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.769+0000 7fdef9f87700 1 -- 172.21.15.60:0/248276800 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef40b5d30 con 0x7fdef414af10 2022-06-29T16:07:48.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.769+0000 7fdef9f87700 1 --2- 172.21.15.60:0/248276800 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdef414af10 0x7fdef405c430 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fdef0002f70 tx=0x7fdef0006df0).ready entity=mon.2 client_cookie=c6e75e7d643ea3a5 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:48.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.769+0000 7fdeebfff700 1 -- 172.21.15.60:0/248276800 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdef000d4a0 con 0x7fdef414af10 2022-06-29T16:07:48.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.769+0000 7fdeebfff700 1 -- 172.21.15.60:0/248276800 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdef0000e90 con 0x7fdef414af10 2022-06-29T16:07:48.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.769+0000 7fdeebfff700 1 -- 172.21.15.60:0/248276800 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdef000d7f0 con 0x7fdef414af10 2022-06-29T16:07:48.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.769+0000 7fdefb9ea700 1 -- 172.21.15.60:0/248276800 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdef414af10 msgr2=0x7fdef405c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:48.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.769+0000 7fdefb9ea700 1 --2- 172.21.15.60:0/248276800 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdef414af10 0x7fdef405c430 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fdef0002f70 tx=0x7fdef0006df0).stop 2022-06-29T16:07:48.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.769+0000 7fdefb9ea700 1 -- 172.21.15.60:0/248276800 shutdown_connections 2022-06-29T16:07:48.781 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.769+0000 7fdefb9ea700 1 --2- 172.21.15.60:0/248276800 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdef414af10 0x7fdef405c430 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:48.781 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.769+0000 7fdefb9ea700 1 --2- 172.21.15.60:0/248276800 >> v2:172.21.15.92:3300/0 conn(0x7fdef4063ad0 0x7fdef4067680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:48.781 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.769+0000 7fdefb9ea700 1 -- 172.21.15.60:0/248276800 >> 172.21.15.60:0/248276800 conn(0x7fdef40632c0 msgr2=0x7fdef40b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:48.781 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.769+0000 7fdefb9ea700 1 -- 172.21.15.60:0/248276800 shutdown_connections 2022-06-29T16:07:48.781 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.769+0000 7fdefb9ea700 1 -- 172.21.15.60:0/248276800 wait complete. 2022-06-29T16:07:48.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.773+0000 7fdefb9ea700 1 Processor -- start 2022-06-29T16:07:48.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.773+0000 7fdefb9ea700 1 -- start start 2022-06-29T16:07:48.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.773+0000 7fdefb9ea700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdef4063ad0 0x7fdef4159240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:48.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.773+0000 7fdefb9ea700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdef40b5950 0x7fdef4148a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:48.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.773+0000 7fdefb9ea700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdef414af10 0x7fdef40af260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:48.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.773+0000 7fdefb9ea700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef40adb10 con 0x7fdef4063ad0 2022-06-29T16:07:48.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.773+0000 7fdefb9ea700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef40addb0 con 0x7fdef40b5950 2022-06-29T16:07:48.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.773+0000 7fdefb9ea700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef40ae050 con 0x7fdef414af10 2022-06-29T16:07:48.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.773+0000 7fdef9786700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdef4063ad0 0x7fdef4159240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:48.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.773+0000 7fdef9786700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdef4063ad0 0x7fdef4159240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53474/0 (socket says 172.21.15.60:53474) 2022-06-29T16:07:48.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.773+0000 7fdef9786700 1 -- 172.21.15.60:0/4035287808 learned_addr learned my addr 172.21.15.60:0/4035287808 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:48.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.773+0000 7fdef9f87700 1 --2- 172.21.15.60:0/4035287808 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdef414af10 0x7fdef40af260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:48.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.773+0000 7fdef8f85700 1 --2- 172.21.15.60:0/4035287808 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdef40b5950 0x7fdef4148a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:48.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.773+0000 7fdef9786700 1 -- 172.21.15.60:0/4035287808 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdef414af10 msgr2=0x7fdef40af260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:48.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.773+0000 7fdef9786700 1 --2- 172.21.15.60:0/4035287808 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdef414af10 0x7fdef40af260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:48.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.773+0000 7fdef9786700 1 -- 172.21.15.60:0/4035287808 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdef40b5950 msgr2=0x7fdef4148a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:48.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.773+0000 7fdef9786700 1 --2- 172.21.15.60:0/4035287808 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdef40b5950 0x7fdef4148a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:48.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.773+0000 7fdef9786700 1 -- 172.21.15.60:0/4035287808 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef000b180 con 0x7fdef4063ad0 2022-06-29T16:07:48.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.773+0000 7fdef9786700 1 --2- 172.21.15.60:0/4035287808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdef4063ad0 0x7fdef4159240 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 rx=0x7fdeec001490 tx=0x7fdeec007e50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:48.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.773+0000 7fdeea7fc700 1 -- 172.21.15.60:0/4035287808 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdeec005ba0 con 0x7fdef4063ad0 2022-06-29T16:07:48.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.777+0000 7fdeea7fc700 1 -- 172.21.15.60:0/4035287808 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeec005d00 con 0x7fdef4063ad0 2022-06-29T16:07:48.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.777+0000 7fdefb9ea700 1 -- 172.21.15.60:0/4035287808 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdef4117310 con 0x7fdef4063ad0 2022-06-29T16:07:48.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.777+0000 7fdeea7fc700 1 -- 172.21.15.60:0/4035287808 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdeec0072c0 con 0x7fdef4063ad0 2022-06-29T16:07:48.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.777+0000 7fdefb9ea700 1 -- 172.21.15.60:0/4035287808 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef40ad890 con 0x7fdef4063ad0 2022-06-29T16:07:48.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.777+0000 7fdefb9ea700 1 -- 172.21.15.60:0/4035287808 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdef40aeea0 con 0x7fdef4063ad0 2022-06-29T16:07:48.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.777+0000 7fdeea7fc700 1 -- 172.21.15.60:0/4035287808 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdeec01a070 con 0x7fdef4063ad0 2022-06-29T16:07:48.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.777+0000 7fdeea7fc700 1 --2- 172.21.15.60:0/4035287808 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdedc0539e0 0x7fdedc055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:48.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.777+0000 7fdeea7fc700 1 -- 172.21.15.60:0/4035287808 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fdeec06afe0 con 0x7fdef4063ad0 2022-06-29T16:07:48.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.777+0000 7fdef8f85700 1 --2- 172.21.15.60:0/4035287808 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdedc0539e0 0x7fdedc055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:48.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.781+0000 7fdef8f85700 1 --2- 172.21.15.60:0/4035287808 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdedc0539e0 0x7fdedc055ea0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7fdee400b000 tx=0x7fdee4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:48.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.781+0000 7fdeea7fc700 1 -- 172.21.15.60:0/4035287808 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdef40aeea0 con 0x7fdef4063ad0 2022-06-29T16:07:48.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.909+0000 7fdefb9ea700 1 -- 172.21.15.60:0/4035287808 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fdef40aeea0 con 0x7fdef4063ad0 2022-06-29T16:07:48.915 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.909+0000 7fdeea7fc700 1 -- 172.21.15.60:0/4035287808 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v131) v1 ==== 81+0+28837 (secure 0 0 0) 0x7fdef40aeea0 con 0x7fdef4063ad0 2022-06-29T16:07:48.916 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:48.916 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:07:48.916 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 131, 2022-06-29T16:07:48.917 INFO:teuthology.orchestra.run.smithi060.stdout: "fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", 2022-06-29T16:07:48.917 INFO:teuthology.orchestra.run.smithi060.stdout: "created": "2022-06-29T15:58:07.965719+0000", 2022-06-29T16:07:48.917 INFO:teuthology.orchestra.run.smithi060.stdout: "modified": "2022-06-29T16:07:46.196471+0000", 2022-06-29T16:07:48.917 INFO:teuthology.orchestra.run.smithi060.stdout: "last_up_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:07:48.918 INFO:teuthology.orchestra.run.smithi060.stdout: "last_in_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:07:48.918 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-06-29T16:07:48.918 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_num": 5799936, 2022-06-29T16:07:48.918 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_set": [ 2022-06-29T16:07:48.918 INFO:teuthology.orchestra.run.smithi060.stdout: "pglog_hardlimit", 2022-06-29T16:07:48.919 INFO:teuthology.orchestra.run.smithi060.stdout: "purged_snapdirs", 2022-06-29T16:07:48.919 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_deletes", 2022-06-29T16:07:48.919 INFO:teuthology.orchestra.run.smithi060.stdout: "sortbitwise" 2022-06-29T16:07:48.919 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:07:48.920 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_version": 4, 2022-06-29T16:07:48.920 INFO:teuthology.orchestra.run.smithi060.stdout: "full_ratio": 0.94999998807907104, 2022-06-29T16:07:48.920 INFO:teuthology.orchestra.run.smithi060.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-06-29T16:07:48.920 INFO:teuthology.orchestra.run.smithi060.stdout: "nearfull_ratio": 0.85000002384185791, 2022-06-29T16:07:48.921 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_snapshot": "", 2022-06-29T16:07:48.921 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_max": 26, 2022-06-29T16:07:48.921 INFO:teuthology.orchestra.run.smithi060.stdout: "max_osd": 8, 2022-06-29T16:07:48.921 INFO:teuthology.orchestra.run.smithi060.stdout: "require_min_compat_client": "jewel", 2022-06-29T16:07:48.921 INFO:teuthology.orchestra.run.smithi060.stdout: "min_compat_client": "jewel", 2022-06-29T16:07:48.922 INFO:teuthology.orchestra.run.smithi060.stdout: "require_osd_release": "octopus", 2022-06-29T16:07:48.922 INFO:teuthology.orchestra.run.smithi060.stdout: "pools": [ 2022-06-29T16:07:48.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:48.922 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2022-06-29T16:07:48.923 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "device_health_metrics", 2022-06-29T16:07:48.923 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:10.961127+0000", 2022-06-29T16:07:48.923 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:07:48.923 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:07:48.924 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:07:48.924 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:07:48.924 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:07:48.924 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:07:48.924 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:07:48.925 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:07:48.925 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 1, 2022-06-29T16:07:48.925 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 1, 2022-06-29T16:07:48.925 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 1, 2022-06-29T16:07:48.926 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 1, 2022-06-29T16:07:48.926 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 1, 2022-06-29T16:07:48.926 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:07:48.926 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:07:48.927 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:07:48.927 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:07:48.927 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:07:48.927 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:07:48.928 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:07:48.928 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:48.928 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "15", 2022-06-29T16:07:48.928 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:07:48.929 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:07:48.929 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:07:48.929 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:07:48.930 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:07:48.930 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:07:48.930 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:07:48.930 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:07:48.931 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:07:48.931 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:07:48.931 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:07:48.931 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:07:48.932 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:07:48.932 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:07:48.932 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:07:48.932 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:07:48.933 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:07:48.933 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:07:48.933 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:07:48.933 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:07:48.933 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:07:48.934 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:07:48.934 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:07:48.934 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:07:48.934 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:07:48.935 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:07:48.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:48.935 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:07:48.935 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:07:48.935 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:07:48.936 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:07:48.936 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:07:48.936 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:07:48.936 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:07:48.937 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:07:48.937 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:07:48.937 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:07:48.937 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:07:48.938 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:07:48.938 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 1 2022-06-29T16:07:48.938 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:48.938 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:07:48.938 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_devicehealth": {} 2022-06-29T16:07:48.939 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:48.939 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:48.939 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:48.939 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 2, 2022-06-29T16:07:48.940 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "rbd", 2022-06-29T16:07:48.940 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:19.184276+0000", 2022-06-29T16:07:48.940 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:07:48.940 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:07:48.940 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:07:48.941 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:07:48.941 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:07:48.941 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:07:48.941 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:07:48.942 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:07:48.942 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:07:48.942 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:07:48.942 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:07:48.943 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:07:48.943 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:07:48.943 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:07:48.943 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:07:48.943 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:07:48.944 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:07:48.944 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:07:48.944 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:07:48.944 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:07:48.945 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:48.945 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "18", 2022-06-29T16:07:48.945 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:07:48.945 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:07:48.945 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:07:48.946 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:07:48.946 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:07:48.946 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:07:48.946 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:07:48.947 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:07:48.947 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:07:48.947 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:07:48.947 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:07:48.948 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:07:48.948 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:07:48.948 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:07:48.948 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:07:48.948 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:07:48.949 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:07:48.949 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:07:48.949 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:07:48.949 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:07:48.950 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:07:48.950 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:07:48.950 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:07:48.950 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:07:48.950 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:07:48.951 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:07:48.951 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:48.951 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:07:48.951 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:07:48.952 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:07:48.952 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:07:48.952 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:07:48.952 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:07:48.953 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:07:48.953 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:07:48.953 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:07:48.953 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:07:48.954 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:07:48.954 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:07:48.954 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:07:48.954 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd": {} 2022-06-29T16:07:48.954 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:48.955 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:48.955 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:07:48.955 INFO:teuthology.orchestra.run.smithi060.stdout: "osds": [ 2022-06-29T16:07:48.956 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:48.956 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:07:48.956 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "11e7ab9e-fce2-470f-8195-ea4c3c52df22", 2022-06-29T16:07:48.956 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:07:48.956 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:07:48.957 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:07:48.957 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:07:48.957 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:07:48.957 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:07:48.958 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:07:48.958 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 125, 2022-06-29T16:07:48.958 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:07:48.958 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:07:48.959 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:07:48.959 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:48.959 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:48.959 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:48.959 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6816", 2022-06-29T16:07:48.960 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:07:48.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:48.960 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:48.960 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:48.961 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6825", 2022-06-29T16:07:48.961 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:07:48.961 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:48.961 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:48.962 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:48.962 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:07:48.962 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:48.962 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:48.962 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:48.963 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6826", 2022-06-29T16:07:48.963 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:07:48.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:48.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:48.964 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:48.964 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6827", 2022-06-29T16:07:48.964 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:07:48.964 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:48.965 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:48.965 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:48.965 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:07:48.966 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:48.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:48.966 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:48.966 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6830", 2022-06-29T16:07:48.967 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:07:48.967 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:48.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:48.967 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:48.967 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6831", 2022-06-29T16:07:48.968 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:07:48.968 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:48.968 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:48.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:48.969 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:07:48.969 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:48.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:48.969 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:48.969 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6828", 2022-06-29T16:07:48.970 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:07:48.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:48.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:48.970 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:48.971 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6829", 2022-06-29T16:07:48.971 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:07:48.971 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:48.971 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:48.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:48.972 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6825/12890", 2022-06-29T16:07:48.972 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6827/12890", 2022-06-29T16:07:48.972 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6831/12890", 2022-06-29T16:07:48.972 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6829/12890", 2022-06-29T16:07:48.973 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:07:48.973 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:07:48.973 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:07:48.973 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:48.974 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:48.974 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:48.974 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:07:48.974 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "ca43f136-6f69-4431-a157-fbe073fb585c", 2022-06-29T16:07:48.975 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:07:48.975 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:07:48.975 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:07:48.975 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:07:48.976 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:07:48.976 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:07:48.976 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:07:48.976 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 126, 2022-06-29T16:07:48.976 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:07:48.977 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:07:48.977 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:07:48.977 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:48.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:48.978 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:48.978 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6815", 2022-06-29T16:07:48.978 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:07:48.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:48.979 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:48.979 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:48.979 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6818", 2022-06-29T16:07:48.980 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:07:48.980 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:48.980 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:48.980 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:48.981 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:07:48.981 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:48.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:48.981 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:48.981 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6819", 2022-06-29T16:07:48.982 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:07:48.982 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:48.982 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:48.982 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:48.983 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6820", 2022-06-29T16:07:48.983 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:07:48.983 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:48.983 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:48.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:48.984 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:07:48.984 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:48.984 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:48.984 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:48.985 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6823", 2022-06-29T16:07:48.985 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:07:48.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:48.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:48.986 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:48.986 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6824", 2022-06-29T16:07:48.986 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:07:48.986 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:48.986 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:48.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:48.987 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:07:48.987 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:48.987 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:48.988 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:48.988 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6821", 2022-06-29T16:07:48.988 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:07:48.988 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:48.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:48.989 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:48.989 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6822", 2022-06-29T16:07:48.989 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:07:48.989 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:48.990 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:48.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:48.990 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6818/12892", 2022-06-29T16:07:48.990 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6820/12892", 2022-06-29T16:07:48.991 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6824/12892", 2022-06-29T16:07:48.991 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6822/12892", 2022-06-29T16:07:48.991 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:07:48.991 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:07:48.992 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:07:48.992 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:48.992 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:48.992 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:48.992 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:07:48.993 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "74fac45f-5f07-44ec-a9cb-2644c123fd4a", 2022-06-29T16:07:48.993 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:07:48.993 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:07:48.993 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:07:48.994 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:07:48.994 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:07:48.994 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:07:48.994 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:07:48.995 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 126, 2022-06-29T16:07:48.995 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:07:48.995 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:07:48.995 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:07:48.995 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:48.996 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:48.996 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:48.996 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6808", 2022-06-29T16:07:48.996 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:07:48.997 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:48.997 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:48.997 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:48.997 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6809", 2022-06-29T16:07:48.998 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:07:48.998 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:48.998 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:48.998 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:48.999 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:07:48.999 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:48.999 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:48.999 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:48.999 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6810", 2022-06-29T16:07:49.000 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:07:49.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.000 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:49.001 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6811", 2022-06-29T16:07:49.001 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:07:49.001 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:49.001 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:49.002 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.002 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:07:49.002 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:49.002 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.003 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:49.003 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6814", 2022-06-29T16:07:49.003 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:07:49.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.004 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.004 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:49.004 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6817", 2022-06-29T16:07:49.004 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:07:49.004 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:49.005 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:49.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.005 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:07:49.005 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:49.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.006 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:49.006 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6812", 2022-06-29T16:07:49.006 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:07:49.006 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.007 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.007 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:49.007 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6813", 2022-06-29T16:07:49.007 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:07:49.008 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:49.008 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:49.008 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.008 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6809/12891", 2022-06-29T16:07:49.008 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6811/12891", 2022-06-29T16:07:49.009 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6817/12891", 2022-06-29T16:07:49.009 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6813/12891", 2022-06-29T16:07:49.009 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:07:49.009 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:07:49.010 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:07:49.010 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:49.010 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.010 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.010 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:07:49.011 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "2581d351-973a-4b4e-85ca-3658f0068765", 2022-06-29T16:07:49.011 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:07:49.011 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:07:49.011 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:07:49.012 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:07:49.012 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:07:49.012 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:07:49.012 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:07:49.012 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 125, 2022-06-29T16:07:49.013 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:07:49.013 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:07:49.013 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:07:49.013 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:49.014 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.014 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:49.014 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6800", 2022-06-29T16:07:49.014 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:07:49.014 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.015 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.015 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:49.015 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6801", 2022-06-29T16:07:49.015 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:07:49.016 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:49.016 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:49.016 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.016 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:07:49.016 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:49.017 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.017 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:49.017 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6802", 2022-06-29T16:07:49.017 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:07:49.018 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.018 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.018 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:49.018 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6803", 2022-06-29T16:07:49.018 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:07:49.019 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:49.019 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:49.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.019 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:07:49.020 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:49.020 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.020 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:49.020 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6806", 2022-06-29T16:07:49.020 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:07:49.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.021 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:49.021 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6807", 2022-06-29T16:07:49.022 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:07:49.022 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:49.022 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:49.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.022 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:07:49.023 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:49.023 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.023 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:49.023 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6804", 2022-06-29T16:07:49.024 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:07:49.024 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.024 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.024 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:49.024 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6805", 2022-06-29T16:07:49.025 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:07:49.025 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:49.025 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:49.025 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.026 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6801/12889", 2022-06-29T16:07:49.026 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6803/12889", 2022-06-29T16:07:49.026 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6807/12889", 2022-06-29T16:07:49.026 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6805/12889", 2022-06-29T16:07:49.026 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:07:49.027 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:07:49.027 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:07:49.027 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:49.027 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.028 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.028 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:07:49.028 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "3d234db9-87cd-4fe9-814f-0240a287d8b4", 2022-06-29T16:07:49.028 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:07:49.028 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:07:49.029 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:07:49.029 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:07:49.029 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:07:49.029 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:07:49.029 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:07:49.030 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 126, 2022-06-29T16:07:49.030 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:07:49.030 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:07:49.030 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:07:49.031 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:49.031 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.031 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:49.031 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6808", 2022-06-29T16:07:49.031 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:07:49.032 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.032 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.032 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:49.032 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6810", 2022-06-29T16:07:49.033 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:07:49.033 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:49.033 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:49.033 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.033 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:07:49.034 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:49.034 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.034 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:49.034 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6812", 2022-06-29T16:07:49.035 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:07:49.035 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.035 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.035 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:49.035 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6814", 2022-06-29T16:07:49.036 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:07:49.036 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:49.036 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:49.036 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.037 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:07:49.037 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:49.037 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.037 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:49.037 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6823", 2022-06-29T16:07:49.038 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:07:49.038 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.038 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.038 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:49.039 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6825", 2022-06-29T16:07:49.039 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:07:49.039 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:49.039 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:49.040 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.040 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:07:49.040 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:49.040 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.040 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:49.041 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6817", 2022-06-29T16:07:49.041 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:07:49.041 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.041 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.041 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:49.042 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6820", 2022-06-29T16:07:49.042 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:07:49.042 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:49.042 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:49.043 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.043 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6810/13249", 2022-06-29T16:07:49.043 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6814/13249", 2022-06-29T16:07:49.043 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6825/13249", 2022-06-29T16:07:49.043 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6820/13249", 2022-06-29T16:07:49.044 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:07:49.044 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:07:49.044 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:07:49.044 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:49.045 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.045 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.045 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:07:49.045 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "e4bf1104-ff5c-438b-b608-8cee31c163f6", 2022-06-29T16:07:49.045 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:07:49.046 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:07:49.046 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:07:49.046 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:07:49.046 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:07:49.047 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:07:49.047 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:07:49.047 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 119, 2022-06-29T16:07:49.047 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:07:49.047 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:07:49.048 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:07:49.048 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:49.048 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.048 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:49.049 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6809", 2022-06-29T16:07:49.049 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:07:49.049 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.049 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.049 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:49.050 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6811", 2022-06-29T16:07:49.050 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:07:49.050 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:49.050 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:49.051 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.051 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:07:49.051 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:49.051 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.052 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:49.052 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6813", 2022-06-29T16:07:49.052 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:07:49.052 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.052 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.053 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:49.053 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6815", 2022-06-29T16:07:49.053 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:07:49.053 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:49.054 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:49.054 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.054 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:07:49.054 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:49.054 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.055 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:49.055 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6824", 2022-06-29T16:07:49.055 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:07:49.055 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.056 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.056 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:49.056 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6827", 2022-06-29T16:07:49.056 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:07:49.056 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:49.057 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:49.057 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.057 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:07:49.057 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:49.058 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.058 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:49.058 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6818", 2022-06-29T16:07:49.058 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:07:49.058 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.059 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.059 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:49.059 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6821", 2022-06-29T16:07:49.059 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:07:49.060 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:49.060 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:49.060 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.060 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6811/13248", 2022-06-29T16:07:49.060 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6815/13248", 2022-06-29T16:07:49.061 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6827/13248", 2022-06-29T16:07:49.061 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6821/13248", 2022-06-29T16:07:49.061 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:07:49.061 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:07:49.062 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:07:49.062 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:49.062 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.062 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.062 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:07:49.063 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "8564dd25-27ab-4203-b54e-af6d3aa05857", 2022-06-29T16:07:49.063 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:07:49.063 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:07:49.063 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:07:49.064 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:07:49.064 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:07:49.064 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:07:49.064 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:07:49.064 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 126, 2022-06-29T16:07:49.065 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:07:49.065 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:07:49.065 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:07:49.065 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:49.066 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.066 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:49.066 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6816", 2022-06-29T16:07:49.066 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:07:49.066 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.067 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.067 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:49.067 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6819", 2022-06-29T16:07:49.067 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:07:49.068 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:49.068 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:49.068 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.068 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:07:49.068 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:49.069 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.069 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:49.069 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6822", 2022-06-29T16:07:49.069 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:07:49.070 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.070 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.070 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:49.070 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6826", 2022-06-29T16:07:49.070 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:07:49.071 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:49.071 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:49.071 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.071 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:07:49.072 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:49.072 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.072 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:49.072 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6830", 2022-06-29T16:07:49.072 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:07:49.073 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.073 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.073 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:49.073 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6831", 2022-06-29T16:07:49.074 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:07:49.074 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:49.074 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:49.074 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.074 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:07:49.075 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:49.075 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.075 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:49.075 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6828", 2022-06-29T16:07:49.075 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:07:49.076 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.076 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.076 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:49.076 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6829", 2022-06-29T16:07:49.077 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:07:49.077 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:49.077 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:49.077 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.077 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6819/13250", 2022-06-29T16:07:49.078 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6826/13250", 2022-06-29T16:07:49.078 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6831/13250", 2022-06-29T16:07:49.078 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6829/13250", 2022-06-29T16:07:49.078 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:07:49.079 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:07:49.079 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:07:49.079 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:49.079 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.080 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.080 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:07:49.080 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "bd513a9c-3b19-4314-9897-d300b5c897bb", 2022-06-29T16:07:49.080 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:07:49.080 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:07:49.081 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:07:49.081 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:07:49.081 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:07:49.081 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:07:49.081 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:07:49.082 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 126, 2022-06-29T16:07:49.082 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:07:49.082 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:07:49.082 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:07:49.083 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:49.083 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.083 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:49.083 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6800", 2022-06-29T16:07:49.083 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:07:49.084 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.084 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.084 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:49.084 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6801", 2022-06-29T16:07:49.085 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:07:49.085 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:49.085 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:49.085 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.085 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:07:49.086 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:49.086 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.086 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:49.086 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6802", 2022-06-29T16:07:49.087 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:07:49.087 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.087 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.087 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:49.087 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6803", 2022-06-29T16:07:49.088 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:07:49.088 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:49.088 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:49.088 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.089 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:07:49.089 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:49.089 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.089 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:49.089 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6806", 2022-06-29T16:07:49.090 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:07:49.090 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.090 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.090 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:49.091 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6807", 2022-06-29T16:07:49.091 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:07:49.091 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:49.091 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:49.092 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.092 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:07:49.092 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:07:49.092 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.092 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:07:49.093 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6804", 2022-06-29T16:07:49.093 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:07:49.093 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.093 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.094 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:07:49.094 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6805", 2022-06-29T16:07:49.094 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:07:49.094 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:49.094 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:49.095 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.095 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6801/13247", 2022-06-29T16:07:49.095 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6803/13247", 2022-06-29T16:07:49.095 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6807/13247", 2022-06-29T16:07:49.096 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6805/13247", 2022-06-29T16:07:49.096 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:07:49.096 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:07:49.096 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:07:49.096 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:49.097 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:49.097 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:07:49.097 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_xinfo": [ 2022-06-29T16:07:49.097 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.098 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:07:49.098 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:07:49.098 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:07:49.098 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:07:49.098 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:07:49.099 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:07:49.099 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.891342+0000", 2022-06-29T16:07:49.099 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:07:49.099 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.100 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.100 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:07:49.100 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:07:49.100 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:07:49.100 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:07:49.101 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:07:49.101 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:07:49.101 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.900866+0000", 2022-06-29T16:07:49.101 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:07:49.102 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.102 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.102 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:07:49.102 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:07:49.102 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:07:49.103 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:07:49.103 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:07:49.103 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:07:49.103 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.912755+0000", 2022-06-29T16:07:49.104 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:07:49.104 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.104 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.104 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:07:49.104 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:07:49.105 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:07:49.105 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:07:49.105 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:07:49.105 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:07:49.106 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.864332+0000", 2022-06-29T16:07:49.106 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:07:49.106 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.106 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.106 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:07:49.107 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:07:49.107 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:07:49.107 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:07:49.107 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:07:49.108 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:07:49.108 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.863756+0000", 2022-06-29T16:07:49.108 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:07:49.108 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.108 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.109 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:07:49.109 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:07:49.109 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:07:49.109 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:07:49.109 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:07:49.110 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:07:49.110 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.869041+0000", 2022-06-29T16:07:49.110 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:07:49.110 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.111 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.111 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:07:49.111 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:07:49.111 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:07:49.112 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:07:49.112 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:07:49.112 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:07:49.112 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.895043+0000", 2022-06-29T16:07:49.112 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:07:49.113 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.113 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:49.113 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:07:49.113 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:07:49.113 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:07:49.114 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:07:49.114 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:07:49.114 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:07:49.114 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.889491+0000", 2022-06-29T16:07:49.115 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:07:49.115 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:49.115 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:07:49.115 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap": [], 2022-06-29T16:07:49.115 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap_items": [], 2022-06-29T16:07:49.116 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_temp": [], 2022-06-29T16:07:49.116 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_temp": [], 2022-06-29T16:07:49.116 INFO:teuthology.orchestra.run.smithi060.stdout: "blacklist": { 2022-06-29T16:07:49.116 INFO:teuthology.orchestra.run.smithi060.stdout: "172.21.15.60:6833/3877729627": "2022-06-30T16:07:42.420283+0000", 2022-06-29T16:07:49.117 INFO:teuthology.orchestra.run.smithi060.stdout: "172.21.15.60:6832/3877729627": "2022-06-30T16:07:42.420283+0000" 2022-06-29T16:07:49.117 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.117 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profiles": { 2022-06-29T16:07:49.117 INFO:teuthology.orchestra.run.smithi060.stdout: "default": { 2022-06-29T16:07:49.117 INFO:teuthology.orchestra.run.smithi060.stdout: "crush-failure-domain": "osd", 2022-06-29T16:07:49.118 INFO:teuthology.orchestra.run.smithi060.stdout: "k": "2", 2022-06-29T16:07:49.118 INFO:teuthology.orchestra.run.smithi060.stdout: "m": "1", 2022-06-29T16:07:49.118 INFO:teuthology.orchestra.run.smithi060.stdout: "plugin": "jerasure", 2022-06-29T16:07:49.118 INFO:teuthology.orchestra.run.smithi060.stdout: "ruleset-failure-domain": "osd", 2022-06-29T16:07:49.119 INFO:teuthology.orchestra.run.smithi060.stdout: "technique": "reed_sol_van" 2022-06-29T16:07:49.119 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:49.119 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:49.119 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps_queue": [], 2022-06-29T16:07:49.119 INFO:teuthology.orchestra.run.smithi060.stdout: "new_removed_snaps": [], 2022-06-29T16:07:49.120 INFO:teuthology.orchestra.run.smithi060.stdout: "new_purged_snaps": [], 2022-06-29T16:07:49.120 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_node_flags": {}, 2022-06-29T16:07:49.120 INFO:teuthology.orchestra.run.smithi060.stdout: "device_class_flags": {} 2022-06-29T16:07:49.120 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:07:49.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.913+0000 7fdefb9ea700 1 -- 172.21.15.60:0/4035287808 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdedc0539e0 msgr2=0x7fdedc055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:49.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.913+0000 7fdefb9ea700 1 --2- 172.21.15.60:0/4035287808 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdedc0539e0 0x7fdedc055ea0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7fdee400b000 tx=0x7fdee4006df0).stop 2022-06-29T16:07:49.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.913+0000 7fdefb9ea700 1 -- 172.21.15.60:0/4035287808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdef4063ad0 msgr2=0x7fdef4159240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:49.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.913+0000 7fdefb9ea700 1 --2- 172.21.15.60:0/4035287808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdef4063ad0 0x7fdef4159240 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 rx=0x7fdeec001490 tx=0x7fdeec007e50).stop 2022-06-29T16:07:49.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.913+0000 7fdefb9ea700 1 -- 172.21.15.60:0/4035287808 shutdown_connections 2022-06-29T16:07:49.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.913+0000 7fdefb9ea700 1 --2- 172.21.15.60:0/4035287808 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdedc0539e0 0x7fdedc055ea0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:49.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.913+0000 7fdefb9ea700 1 --2- 172.21.15.60:0/4035287808 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fdef414af10 0x7fdef40af260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:49.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.913+0000 7fdefb9ea700 1 --2- 172.21.15.60:0/4035287808 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdef40b5950 0x7fdef4148a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:49.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.913+0000 7fdefb9ea700 1 --2- 172.21.15.60:0/4035287808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdef4063ad0 0x7fdef4159240 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:49.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.913+0000 7fdefb9ea700 1 -- 172.21.15.60:0/4035287808 >> 172.21.15.60:0/4035287808 conn(0x7fdef40632c0 msgr2=0x7fdef4143930 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:49.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.913+0000 7fdefb9ea700 1 -- 172.21.15.60:0/4035287808 shutdown_connections 2022-06-29T16:07:49.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:48.913+0000 7fdefb9ea700 1 -- 172.21.15.60:0/4035287808 wait complete. 2022-06-29T16:07:49.126 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:07:49.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.229+0000 7f3bd47f4700 1 Processor -- start 2022-06-29T16:07:49.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.229+0000 7f3bd47f4700 1 -- start start 2022-06-29T16:07:49.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.229+0000 7f3bd47f4700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3bcc0ae7c0 0x7f3bcc141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:49.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.229+0000 7f3bd47f4700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3bcc0b4010 0x7f3bcc142490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:49.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.229+0000 7f3bd47f4700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bcc143a40 con 0x7f3bcc146160 2022-06-29T16:07:49.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.229+0000 7f3bd47f4700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bcc0b56d0 con 0x7f3bcc0b4010 2022-06-29T16:07:49.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.229+0000 7f3bd47f4700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bcc0aee60 con 0x7f3bcc0ae7c0 2022-06-29T16:07:49.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.229+0000 7f3bd1d8f700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3bcc0b4010 0x7f3bcc142490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:49.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.229+0000 7f3bd1d8f700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f3bcc0b4010 0x7f3bcc142490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53482/0 (socket says 172.21.15.60:53482) 2022-06-29T16:07:49.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.229+0000 7f3bd1d8f700 1 -- 172.21.15.60:0/20574785 learned_addr learned my addr 172.21.15.60:0/20574785 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:49.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.229+0000 7f3bd2590700 1 --2- 172.21.15.60:0/20574785 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3bcc0ae7c0 0x7f3bcc141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:49.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.229+0000 7f3bd1d8f700 1 -- 172.21.15.60:0/20574785 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3bcc0ae7c0 msgr2=0x7f3bcc141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:49.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.229+0000 7f3bd1d8f700 1 --2- 172.21.15.60:0/20574785 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3bcc0ae7c0 0x7f3bcc141fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:49.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.229+0000 7f3bd1d8f700 1 -- 172.21.15.60:0/20574785 >> v1:172.21.15.92:6789/0 conn(0x7f3bcc146160 legacy=0x7f3bcc1465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:49.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.229+0000 7f3bd1d8f700 1 -- 172.21.15.60:0/20574785 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bcc0af100 con 0x7f3bcc0b4010 2022-06-29T16:07:49.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.229+0000 7f3bd1d8f700 1 --2- 172.21.15.60:0/20574785 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3bcc0b4010 0x7f3bcc142490 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 rx=0x7f3bc8002f70 tx=0x7f3bc8006eb0).ready entity=mon.0 client_cookie=b21355a6de6dfb20 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:49.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.229+0000 7f3bd158e700 1 -- 172.21.15.60:0/20574785 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3bc800d570 con 0x7f3bcc0b4010 2022-06-29T16:07:49.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.229+0000 7f3bd158e700 1 -- 172.21.15.60:0/20574785 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bc8000e90 con 0x7f3bcc0b4010 2022-06-29T16:07:49.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.229+0000 7f3bd158e700 1 -- 172.21.15.60:0/20574785 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3bc800d900 con 0x7f3bcc0b4010 2022-06-29T16:07:49.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.233+0000 7f3bd47f4700 1 -- 172.21.15.60:0/20574785 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3bcc0b4010 msgr2=0x7f3bcc142490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:49.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.233+0000 7f3bd47f4700 1 --2- 172.21.15.60:0/20574785 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3bcc0b4010 0x7f3bcc142490 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 rx=0x7f3bc8002f70 tx=0x7f3bc8006eb0).stop 2022-06-29T16:07:49.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.233+0000 7f3bd47f4700 1 -- 172.21.15.60:0/20574785 shutdown_connections 2022-06-29T16:07:49.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.233+0000 7f3bd47f4700 1 --2- 172.21.15.60:0/20574785 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3bcc0b4010 0x7f3bcc142490 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:49.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.233+0000 7f3bd47f4700 1 --2- 172.21.15.60:0/20574785 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3bcc0ae7c0 0x7f3bcc141fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:49.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.233+0000 7f3bd47f4700 1 -- 172.21.15.60:0/20574785 >> 172.21.15.60:0/20574785 conn(0x7f3bcc0b9340 msgr2=0x7f3bcc0b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:49.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.233+0000 7f3bd47f4700 1 -- 172.21.15.60:0/20574785 shutdown_connections 2022-06-29T16:07:49.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.233+0000 7f3bd47f4700 1 -- 172.21.15.60:0/20574785 wait complete. 2022-06-29T16:07:49.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.233+0000 7f3bd47f4700 1 Processor -- start 2022-06-29T16:07:49.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.233+0000 7f3bd47f4700 1 -- start start 2022-06-29T16:07:49.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.233+0000 7f3bd47f4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3bcc0ae7c0 0x7f3bcc140f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:49.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.233+0000 7f3bd47f4700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3bcc0b4010 0x7f3bcc1554c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:49.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.233+0000 7f3bd47f4700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3bcc146160 0x7f3bcc1559b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:49.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.233+0000 7f3bd47f4700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bcc0b3780 con 0x7f3bcc0ae7c0 2022-06-29T16:07:49.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.233+0000 7f3bd47f4700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bcc0b0740 con 0x7f3bcc0b4010 2022-06-29T16:07:49.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.233+0000 7f3bd47f4700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bcc0b0970 con 0x7f3bcc146160 2022-06-29T16:07:49.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.233+0000 7f3bd2590700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3bcc0ae7c0 0x7f3bcc140f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:49.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.233+0000 7f3bd1d8f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3bcc0b4010 0x7f3bcc1554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:49.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.233+0000 7f3bd2590700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3bcc0ae7c0 0x7f3bcc140f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53490/0 (socket says 172.21.15.60:53490) 2022-06-29T16:07:49.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.233+0000 7f3bd2d91700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3bcc146160 0x7f3bcc1559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:49.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.233+0000 7f3bd2590700 1 -- 172.21.15.60:0/3302417309 learned_addr learned my addr 172.21.15.60:0/3302417309 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:49.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.233+0000 7f3bd1d8f700 1 -- 172.21.15.60:0/3302417309 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3bcc146160 msgr2=0x7f3bcc1559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:49.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.233+0000 7f3bd1d8f700 1 --2- 172.21.15.60:0/3302417309 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3bcc146160 0x7f3bcc1559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:49.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.233+0000 7f3bd1d8f700 1 -- 172.21.15.60:0/3302417309 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3bcc0ae7c0 msgr2=0x7f3bcc140f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:49.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.233+0000 7f3bd1d8f700 1 --2- 172.21.15.60:0/3302417309 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3bcc0ae7c0 0x7f3bcc140f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:49.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.233+0000 7f3bd1d8f700 1 -- 172.21.15.60:0/3302417309 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc800b1e0 con 0x7f3bcc0b4010 2022-06-29T16:07:49.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.237+0000 7f3bd1d8f700 1 --2- 172.21.15.60:0/3302417309 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3bcc0b4010 0x7f3bcc1554c0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0x7f3bc8002f70 tx=0x7f3bc8015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:49.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.237+0000 7f3bbb7fe700 1 -- 172.21.15.60:0/3302417309 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3bc80116e0 con 0x7f3bcc0b4010 2022-06-29T16:07:49.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.237+0000 7f3bbb7fe700 1 -- 172.21.15.60:0/3302417309 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bc8011840 con 0x7f3bcc0b4010 2022-06-29T16:07:49.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.237+0000 7f3bd47f4700 1 -- 172.21.15.60:0/3302417309 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bcc146770 con 0x7f3bcc0b4010 2022-06-29T16:07:49.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.237+0000 7f3bbb7fe700 1 -- 172.21.15.60:0/3302417309 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3bc8011ba0 con 0x7f3bcc0b4010 2022-06-29T16:07:49.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.237+0000 7f3bd47f4700 1 -- 172.21.15.60:0/3302417309 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bcc13c900 con 0x7f3bcc0b4010 2022-06-29T16:07:49.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.237+0000 7f3bbb7fe700 1 -- 172.21.15.60:0/3302417309 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3bc8025030 con 0x7f3bcc0b4010 2022-06-29T16:07:49.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.237+0000 7f3bd47f4700 1 -- 172.21.15.60:0/3302417309 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bcc146930 con 0x7f3bcc0b4010 2022-06-29T16:07:49.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.237+0000 7f3bbb7fe700 1 --2- 172.21.15.60:0/3302417309 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3bac053b00 0x7f3bac055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:49.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.237+0000 7f3bbb7fe700 1 -- 172.21.15.60:0/3302417309 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f3bc806cd70 con 0x7f3bcc0b4010 2022-06-29T16:07:49.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.237+0000 7f3bd2590700 1 --2- 172.21.15.60:0/3302417309 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3bac053b00 0x7f3bac055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:49.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.241+0000 7f3bd2590700 1 --2- 172.21.15.60:0/3302417309 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3bac053b00 0x7f3bac055fc0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7f3bbc00b000 tx=0x7f3bbc006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:49.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.245+0000 7f3bbb7fe700 1 -- 172.21.15.60:0/3302417309 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3bc8043ea0 con 0x7f3bcc0b4010 2022-06-29T16:07:49.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.409+0000 7f3bd47f4700 1 -- 172.21.15.60:0/3302417309 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3bcc0573d0 con 0x7f3bcc0b4010 2022-06-29T16:07:49.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.409+0000 7f3bbb7fe700 1 -- 172.21.15.60:0/3302417309 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+1209 (secure 0 0 0) 0x7f3bc8023090 con 0x7f3bcc0b4010 2022-06-29T16:07:49.415 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:49.416 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":65,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6941,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/3488623391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3488623391},{"type":"v1","addr":"172.21.15.92:6835","nonce":3488623391}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":59},{"gid":8299,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.60:6833/4049046489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":4049046489},{"type":"v1","addr":"172.21.15.60:6833","nonce":4049046489}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":65}],"filesystems":[]} 2022-06-29T16:07:49.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.413+0000 7f3bd47f4700 1 -- 172.21.15.60:0/3302417309 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3bac053b00 msgr2=0x7f3bac055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:49.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.413+0000 7f3bd47f4700 1 --2- 172.21.15.60:0/3302417309 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3bac053b00 0x7f3bac055fc0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7f3bbc00b000 tx=0x7f3bbc006df0).stop 2022-06-29T16:07:49.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.413+0000 7f3bd47f4700 1 -- 172.21.15.60:0/3302417309 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3bcc0b4010 msgr2=0x7f3bcc1554c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:49.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.413+0000 7f3bd47f4700 1 --2- 172.21.15.60:0/3302417309 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3bcc0b4010 0x7f3bcc1554c0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0x7f3bc8002f70 tx=0x7f3bc8015000).stop 2022-06-29T16:07:49.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.413+0000 7f3bd47f4700 1 -- 172.21.15.60:0/3302417309 shutdown_connections 2022-06-29T16:07:49.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.413+0000 7f3bd47f4700 1 --2- 172.21.15.60:0/3302417309 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3bac053b00 0x7f3bac055fc0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:49.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.413+0000 7f3bd47f4700 1 --2- 172.21.15.60:0/3302417309 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3bcc146160 0x7f3bcc1559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:49.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.413+0000 7f3bd47f4700 1 --2- 172.21.15.60:0/3302417309 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3bcc0b4010 0x7f3bcc1554c0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:49.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.413+0000 7f3bd47f4700 1 --2- 172.21.15.60:0/3302417309 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3bcc0ae7c0 0x7f3bcc140f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:49.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.413+0000 7f3bd47f4700 1 -- 172.21.15.60:0/3302417309 >> 172.21.15.60:0/3302417309 conn(0x7f3bcc0b9340 msgr2=0x7f3bcc141d70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:49.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.413+0000 7f3bd47f4700 1 -- 172.21.15.60:0/3302417309 shutdown_connections 2022-06-29T16:07:49.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.413+0000 7f3bd47f4700 1 -- 172.21.15.60:0/3302417309 wait complete. 2022-06-29T16:07:49.421 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 65 2022-06-29T16:07:49.433 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:07:49.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.529+0000 7f148e688700 1 Processor -- start 2022-06-29T16:07:49.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.529+0000 7f148e688700 1 -- start start 2022-06-29T16:07:49.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.533+0000 7f148e688700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14880b5950 0x7f1488067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:49.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.533+0000 7f148e688700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1488063ad0 0x7f148814af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:49.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.533+0000 7f148e688700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f148805bd20 0x7f148805c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:49.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.533+0000 7f148e688700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14880acba0 con 0x7f1488063ad0 2022-06-29T16:07:49.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.533+0000 7f148e688700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1488067220 con 0x7f148805bd20 2022-06-29T16:07:49.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.533+0000 7f148e688700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14880b5d30 con 0x7f14880b5950 2022-06-29T16:07:49.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.533+0000 7f14877fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1488063ad0 0x7f148814af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:49.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.533+0000 7f1487fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14880b5950 0x7f1488067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:49.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.533+0000 7f14877fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f1488063ad0 0x7f148814af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53498/0 (socket says 172.21.15.60:53498) 2022-06-29T16:07:49.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.533+0000 7f14877fe700 1 -- 172.21.15.60:0/3702450877 learned_addr learned my addr 172.21.15.60:0/3702450877 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:49.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.533+0000 7f148cc25700 1 --2- 172.21.15.60:0/3702450877 >> v2:172.21.15.92:3300/0 conn(0x7f148805bd20 0x7f148805c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:49.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.533+0000 7f1487fff700 1 -- 172.21.15.60:0/3702450877 >> v2:172.21.15.92:3300/0 conn(0x7f148805bd20 msgr2=0x7f148805c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:49.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.533+0000 7f1487fff700 1 --2- 172.21.15.60:0/3702450877 >> v2:172.21.15.92:3300/0 conn(0x7f148805bd20 0x7f148805c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:49.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.533+0000 7f1487fff700 1 -- 172.21.15.60:0/3702450877 >> v2:172.21.15.60:3300/0 conn(0x7f1488063ad0 msgr2=0x7f148814af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:49.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.533+0000 7f1487fff700 1 --2- 172.21.15.60:0/3702450877 >> v2:172.21.15.60:3300/0 conn(0x7f1488063ad0 0x7f148814af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:49.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.533+0000 7f1487fff700 1 -- 172.21.15.60:0/3702450877 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14880ace00 con 0x7f14880b5950 2022-06-29T16:07:49.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.533+0000 7f14877fe700 1 --2- 172.21.15.60:0/3702450877 >> v2:172.21.15.60:3300/0 conn(0x7f1488063ad0 0x7f148814af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:07:49.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.533+0000 7f1487fff700 1 --2- 172.21.15.60:0/3702450877 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14880b5950 0x7f1488067680 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f14800032f0 tx=0x7f14800091d0).ready entity=mon.2 client_cookie=7530522d1a068875 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:49.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.533+0000 7f1486ffd700 1 -- 172.21.15.60:0/3702450877 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f148000b600 con 0x7f14880b5950 2022-06-29T16:07:49.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.533+0000 7f1486ffd700 1 -- 172.21.15.60:0/3702450877 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f148000b760 con 0x7f14880b5950 2022-06-29T16:07:49.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.533+0000 7f1486ffd700 1 -- 172.21.15.60:0/3702450877 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f148000b8d0 con 0x7f14880b5950 2022-06-29T16:07:49.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.533+0000 7f148e688700 1 -- 172.21.15.60:0/3702450877 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14880b5950 msgr2=0x7f1488067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:49.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.533+0000 7f148e688700 1 --2- 172.21.15.60:0/3702450877 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14880b5950 0x7f1488067680 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f14800032f0 tx=0x7f14800091d0).stop 2022-06-29T16:07:49.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.533+0000 7f148e688700 1 -- 172.21.15.60:0/3702450877 shutdown_connections 2022-06-29T16:07:49.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.533+0000 7f148e688700 1 --2- 172.21.15.60:0/3702450877 >> v2:172.21.15.92:3300/0 conn(0x7f148805bd20 0x7f148805c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:49.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.533+0000 7f148e688700 1 --2- 172.21.15.60:0/3702450877 >> v2:172.21.15.60:3300/0 conn(0x7f1488063ad0 0x7f148814af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:49.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.533+0000 7f148e688700 1 --2- 172.21.15.60:0/3702450877 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14880b5950 0x7f1488067680 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:49.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.533+0000 7f148e688700 1 -- 172.21.15.60:0/3702450877 >> 172.21.15.60:0/3702450877 conn(0x7f14880632c0 msgr2=0x7f14880b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:49.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.533+0000 7f148e688700 1 -- 172.21.15.60:0/3702450877 shutdown_connections 2022-06-29T16:07:49.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.533+0000 7f148e688700 1 -- 172.21.15.60:0/3702450877 wait complete. 2022-06-29T16:07:49.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.537+0000 7f148e688700 1 Processor -- start 2022-06-29T16:07:49.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.537+0000 7f148e688700 1 -- start start 2022-06-29T16:07:49.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.537+0000 7f148e688700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f148805bd20 0x7f14880af8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:49.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.537+0000 7f148e688700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1488063ad0 0x7f1488117580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:49.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.537+0000 7f148e688700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14880b5950 0x7f1488155ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:49.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.537+0000 7f148e688700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14880affc0 con 0x7f1488063ad0 2022-06-29T16:07:49.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.537+0000 7f148e688700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1488116c10 con 0x7f148805bd20 2022-06-29T16:07:49.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.537+0000 7f148e688700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1488116e40 con 0x7f14880b5950 2022-06-29T16:07:49.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.537+0000 7f14877fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1488063ad0 0x7f1488117580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:49.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.537+0000 7f14877fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1488063ad0 0x7f1488117580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53504/0 (socket says 172.21.15.60:53504) 2022-06-29T16:07:49.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.537+0000 7f14877fe700 1 -- 172.21.15.60:0/3292336245 learned_addr learned my addr 172.21.15.60:0/3292336245 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:49.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.537+0000 7f1487fff700 1 --2- 172.21.15.60:0/3292336245 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f148805bd20 0x7f14880af8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:49.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.537+0000 7f148cc25700 1 --2- 172.21.15.60:0/3292336245 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14880b5950 0x7f1488155ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:49.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.537+0000 7f14877fe700 1 -- 172.21.15.60:0/3292336245 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14880b5950 msgr2=0x7f1488155ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:49.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.537+0000 7f14877fe700 1 --2- 172.21.15.60:0/3292336245 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14880b5950 0x7f1488155ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:49.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.537+0000 7f14877fe700 1 -- 172.21.15.60:0/3292336245 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f148805bd20 msgr2=0x7f14880af8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:49.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.537+0000 7f14877fe700 1 --2- 172.21.15.60:0/3292336245 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f148805bd20 0x7f14880af8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:49.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.537+0000 7f14877fe700 1 -- 172.21.15.60:0/3292336245 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147000b1e0 con 0x7f1488063ad0 2022-06-29T16:07:49.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.537+0000 7f14877fe700 1 --2- 172.21.15.60:0/3292336245 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1488063ad0 0x7f1488117580 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 rx=0x7f147000c000 tx=0x7f1470006eb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:49.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.537+0000 7f14857fa700 1 -- 172.21.15.60:0/3292336245 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1470012600 con 0x7f1488063ad0 2022-06-29T16:07:49.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.537+0000 7f14857fa700 1 -- 172.21.15.60:0/3292336245 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1470012760 con 0x7f1488063ad0 2022-06-29T16:07:49.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.537+0000 7f148e688700 1 -- 172.21.15.60:0/3292336245 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1480009470 con 0x7f1488063ad0 2022-06-29T16:07:49.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.537+0000 7f14857fa700 1 -- 172.21.15.60:0/3292336245 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1470012ac0 con 0x7f1488063ad0 2022-06-29T16:07:49.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.537+0000 7f148e688700 1 -- 172.21.15.60:0/3292336245 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14880ad520 con 0x7f1488063ad0 2022-06-29T16:07:49.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.541+0000 7f148e688700 1 -- 172.21.15.60:0/3292336245 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14880573d0 con 0x7f1488063ad0 2022-06-29T16:07:49.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.541+0000 7f14857fa700 1 -- 172.21.15.60:0/3292336245 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1470022070 con 0x7f1488063ad0 2022-06-29T16:07:49.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.541+0000 7f14857fa700 1 --2- 172.21.15.60:0/3292336245 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f146c053aa0 0x7f146c055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:49.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.541+0000 7f14857fa700 1 -- 172.21.15.60:0/3292336245 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f147006c170 con 0x7f1488063ad0 2022-06-29T16:07:49.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.541+0000 7f1487fff700 1 --2- 172.21.15.60:0/3292336245 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f146c053aa0 0x7f146c055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:49.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.541+0000 7f1487fff700 1 --2- 172.21.15.60:0/3292336245 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f146c053aa0 0x7f146c055f60 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0x7f148001b000 tx=0x7f1480016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:49.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.549+0000 7f14857fa700 1 -- 172.21.15.60:0/3292336245 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f14700432d0 con 0x7f1488063ad0 2022-06-29T16:07:49.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.713+0000 7f148e688700 1 -- 172.21.15.60:0/3292336245 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14881172e0 con 0x7f1488063ad0 2022-06-29T16:07:49.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.713+0000 7f14857fa700 1 -- 172.21.15.60:0/3292336245 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+1209 (secure 0 0 0) 0x7f14700468f0 con 0x7f1488063ad0 2022-06-29T16:07:49.718 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:49.719 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":65,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6941,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/3488623391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3488623391},{"type":"v1","addr":"172.21.15.92:6835","nonce":3488623391}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":59},{"gid":8299,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.60:6833/4049046489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":4049046489},{"type":"v1","addr":"172.21.15.60:6833","nonce":4049046489}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":65}],"filesystems":[]} 2022-06-29T16:07:49.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.713+0000 7f148e688700 1 -- 172.21.15.60:0/3292336245 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f146c053aa0 msgr2=0x7f146c055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:49.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.713+0000 7f148e688700 1 --2- 172.21.15.60:0/3292336245 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f146c053aa0 0x7f146c055f60 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0x7f148001b000 tx=0x7f1480016000).stop 2022-06-29T16:07:49.721 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.713+0000 7f148e688700 1 -- 172.21.15.60:0/3292336245 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1488063ad0 msgr2=0x7f1488117580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:49.721 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.713+0000 7f148e688700 1 --2- 172.21.15.60:0/3292336245 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1488063ad0 0x7f1488117580 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 rx=0x7f147000c000 tx=0x7f1470006eb0).stop 2022-06-29T16:07:49.721 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.717+0000 7f148e688700 1 -- 172.21.15.60:0/3292336245 shutdown_connections 2022-06-29T16:07:49.721 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.717+0000 7f148e688700 1 --2- 172.21.15.60:0/3292336245 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f146c053aa0 0x7f146c055f60 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:49.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.717+0000 7f148e688700 1 --2- 172.21.15.60:0/3292336245 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f14880b5950 0x7f1488155ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:49.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.717+0000 7f148e688700 1 --2- 172.21.15.60:0/3292336245 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1488063ad0 0x7f1488117580 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:49.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.717+0000 7f148e688700 1 --2- 172.21.15.60:0/3292336245 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f148805bd20 0x7f14880af8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:49.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.717+0000 7f148e688700 1 -- 172.21.15.60:0/3292336245 >> 172.21.15.60:0/3292336245 conn(0x7f14880632c0 msgr2=0x7f1488145f50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:49.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.717+0000 7f148e688700 1 -- 172.21.15.60:0/3292336245 shutdown_connections 2022-06-29T16:07:49.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.717+0000 7f148e688700 1 -- 172.21.15.60:0/3292336245 wait complete. 2022-06-29T16:07:49.723 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 65 2022-06-29T16:07:49.732 INFO:tasks.ceph.mds.a:Restarting daemon 2022-06-29T16:07:49.732 INFO:tasks.ceph.mds.a:Stopping old one... 2022-06-29T16:07:49.733 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-06-29T16:07:49.733 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T16:07:49.733 INFO:tasks.ceph.mds.b:Restarting daemon 2022-06-29T16:07:49.733 INFO:tasks.ceph.mds.b:Stopping old one... 2022-06-29T16:07:49.734 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-06-29T16:07:49.734 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T16:07:49.748 INFO:tasks.ceph.mds.a:Stopped 2022-06-29T16:07:49.749 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-06-29T16:07:49.750 INFO:tasks.ceph.mds.b:Stopped 2022-06-29T16:07:49.751 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-06-29T16:07:49.753 INFO:tasks.ceph.mds.a:Started 2022-06-29T16:07:49.753 DEBUG:teuthology.parallel:result is None 2022-06-29T16:07:49.754 INFO:tasks.ceph.mds.b:Started 2022-06-29T16:07:49.754 DEBUG:teuthology.parallel:result is None 2022-06-29T16:07:49.755 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-06-29T16:07:49.823 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2022-06-29T16:07:49.825 INFO:tasks.ceph.mds.b.smithi092.stdout:starting mds.b at 2022-06-29T16:07:49.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.849+0000 7efd5eba0700 1 Processor -- start 2022-06-29T16:07:49.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.849+0000 7efd5eba0700 1 -- start start 2022-06-29T16:07:49.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.853+0000 7efd5eba0700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efd5805fb00 0x7efd58143910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:49.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.853+0000 7efd5eba0700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7efd580aca20 0x7efd580ace60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:49.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.853+0000 7efd5eba0700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd580ad350 0x7efd5813edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:49.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.853+0000 7efd5eba0700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd580b1f00 con 0x7efd5805fb00 2022-06-29T16:07:49.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.853+0000 7efd5eba0700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd5805fee0 con 0x7efd580aca20 2022-06-29T16:07:49.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.853+0000 7efd5eba0700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd58143e00 con 0x7efd580ad350 2022-06-29T16:07:49.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.853+0000 7efd5db9e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efd5805fb00 0x7efd58143910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:49.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.853+0000 7efd5db9e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efd5805fb00 0x7efd58143910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53522/0 (socket says 172.21.15.60:53522) 2022-06-29T16:07:49.859 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.853+0000 7efd5e39f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd580ad350 0x7efd5813edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:49.859 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.853+0000 7efd5db9e700 1 -- 172.21.15.60:0/3617052962 learned_addr learned my addr 172.21.15.60:0/3617052962 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:49.859 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.853+0000 7efd5d39d700 1 --2- 172.21.15.60:0/3617052962 >> v2:172.21.15.92:3300/0 conn(0x7efd580aca20 0x7efd580ace60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:49.863 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.853+0000 7efd5e39f700 1 -- 172.21.15.60:0/3617052962 >> v2:172.21.15.92:3300/0 conn(0x7efd580aca20 msgr2=0x7efd580ace60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:49.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.853+0000 7efd5e39f700 1 --2- 172.21.15.60:0/3617052962 >> v2:172.21.15.92:3300/0 conn(0x7efd580aca20 0x7efd580ace60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:49.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.853+0000 7efd5e39f700 1 -- 172.21.15.60:0/3617052962 >> v2:172.21.15.60:3300/0 conn(0x7efd5805fb00 msgr2=0x7efd58143910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:49.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.853+0000 7efd5e39f700 1 --2- 172.21.15.60:0/3617052962 >> v2:172.21.15.60:3300/0 conn(0x7efd5805fb00 0x7efd58143910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:49.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.853+0000 7efd5e39f700 1 -- 172.21.15.60:0/3617052962 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd580af1d0 con 0x7efd580ad350 2022-06-29T16:07:49.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.853+0000 7efd5db9e700 1 --2- 172.21.15.60:0/3617052962 >> v2:172.21.15.60:3300/0 conn(0x7efd5805fb00 0x7efd58143910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:07:49.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.853+0000 7efd5e39f700 1 --2- 172.21.15.60:0/3617052962 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd580ad350 0x7efd5813edf0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7efd500032f0 tx=0x7efd500091d0).ready entity=mon.2 client_cookie=d286291701d20e15 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:49.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.853+0000 7efd5cb9c700 1 -- 172.21.15.60:0/3617052962 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd5000b600 con 0x7efd580ad350 2022-06-29T16:07:49.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.853+0000 7efd5cb9c700 1 -- 172.21.15.60:0/3617052962 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd5000b760 con 0x7efd580ad350 2022-06-29T16:07:49.866 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.853+0000 7efd5cb9c700 1 -- 172.21.15.60:0/3617052962 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd5000b8d0 con 0x7efd580ad350 2022-06-29T16:07:49.866 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.853+0000 7efd5eba0700 1 -- 172.21.15.60:0/3617052962 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd580ad350 msgr2=0x7efd5813edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:49.866 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.853+0000 7efd5eba0700 1 --2- 172.21.15.60:0/3617052962 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd580ad350 0x7efd5813edf0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7efd500032f0 tx=0x7efd500091d0).stop 2022-06-29T16:07:49.866 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.853+0000 7efd5eba0700 1 -- 172.21.15.60:0/3617052962 shutdown_connections 2022-06-29T16:07:49.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.853+0000 7efd5eba0700 1 --2- 172.21.15.60:0/3617052962 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd580ad350 0x7efd5813edf0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:49.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.853+0000 7efd5eba0700 1 --2- 172.21.15.60:0/3617052962 >> v2:172.21.15.92:3300/0 conn(0x7efd580aca20 0x7efd580ace60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:49.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.853+0000 7efd5eba0700 1 --2- 172.21.15.60:0/3617052962 >> v2:172.21.15.60:3300/0 conn(0x7efd5805fb00 0x7efd58143910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:49.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.853+0000 7efd5eba0700 1 -- 172.21.15.60:0/3617052962 >> 172.21.15.60:0/3617052962 conn(0x7efd58063a60 msgr2=0x7efd5805f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:49.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.853+0000 7efd5eba0700 1 -- 172.21.15.60:0/3617052962 shutdown_connections 2022-06-29T16:07:49.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.853+0000 7efd5eba0700 1 -- 172.21.15.60:0/3617052962 wait complete. 2022-06-29T16:07:49.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.857+0000 7efd5eba0700 1 Processor -- start 2022-06-29T16:07:49.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.857+0000 7efd5eba0700 1 -- start start 2022-06-29T16:07:49.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.857+0000 7efd5eba0700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd5805fb00 0x7efd5814da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:49.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.857+0000 7efd5eba0700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd580aca20 0x7efd58145390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:49.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.857+0000 7efd5eba0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd580ad350 0x7efd5810f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:49.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.857+0000 7efd5eba0700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5810f7b0 con 0x7efd580ad350 2022-06-29T16:07:49.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.857+0000 7efd5eba0700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd58065e50 con 0x7efd580aca20 2022-06-29T16:07:49.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.857+0000 7efd5eba0700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd58066320 con 0x7efd5805fb00 2022-06-29T16:07:49.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.857+0000 7efd5db9e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd5805fb00 0x7efd5814da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:49.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.857+0000 7efd5e39f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd580ad350 0x7efd5810f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:49.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.857+0000 7efd5db9e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd5805fb00 0x7efd5814da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:38390/0 (socket says 172.21.15.60:38390) 2022-06-29T16:07:49.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.857+0000 7efd5db9e700 1 -- 172.21.15.60:0/3610447038 learned_addr learned my addr 172.21.15.60:0/3610447038 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:49.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.857+0000 7efd5d39d700 1 --2- 172.21.15.60:0/3610447038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd580aca20 0x7efd58145390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:49.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.857+0000 7efd5db9e700 1 -- 172.21.15.60:0/3610447038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd580aca20 msgr2=0x7efd58145390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:49.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.857+0000 7efd5db9e700 1 --2- 172.21.15.60:0/3610447038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd580aca20 0x7efd58145390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:49.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.857+0000 7efd5db9e700 1 -- 172.21.15.60:0/3610447038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd580ad350 msgr2=0x7efd5810f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:49.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.857+0000 7efd5db9e700 1 --2- 172.21.15.60:0/3610447038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd580ad350 0x7efd5810f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:49.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.857+0000 7efd5db9e700 1 -- 172.21.15.60:0/3610447038 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd5400b1e0 con 0x7efd5805fb00 2022-06-29T16:07:49.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.857+0000 7efd5db9e700 1 --2- 172.21.15.60:0/3610447038 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd5805fb00 0x7efd5814da00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7efd5400c000 tx=0x7efd54006eb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:49.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.857+0000 7efd46ffd700 1 -- 172.21.15.60:0/3610447038 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd5400b420 con 0x7efd5805fb00 2022-06-29T16:07:49.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.857+0000 7efd46ffd700 1 -- 172.21.15.60:0/3610447038 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd54012600 con 0x7efd5805fb00 2022-06-29T16:07:49.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.857+0000 7efd5eba0700 1 -- 172.21.15.60:0/3610447038 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd50009470 con 0x7efd5805fb00 2022-06-29T16:07:49.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.857+0000 7efd46ffd700 1 -- 172.21.15.60:0/3610447038 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd54012960 con 0x7efd5805fb00 2022-06-29T16:07:49.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.857+0000 7efd5eba0700 1 -- 172.21.15.60:0/3610447038 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd58066c90 con 0x7efd5805fb00 2022-06-29T16:07:49.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.861+0000 7efd46ffd700 1 -- 172.21.15.60:0/3610447038 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7efd54012ac0 con 0x7efd5805fb00 2022-06-29T16:07:49.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.861+0000 7efd5eba0700 1 -- 172.21.15.60:0/3610447038 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd58110b20 con 0x7efd5805fb00 2022-06-29T16:07:49.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.861+0000 7efd46ffd700 1 --2- 172.21.15.60:0/3610447038 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd38053b00 0x7efd38055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:49.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.861+0000 7efd46ffd700 1 -- 172.21.15.60:0/3610447038 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5339+0+0 (secure 0 0 0) 0x7efd5406bd50 con 0x7efd5805fb00 2022-06-29T16:07:49.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.861+0000 7efd5d39d700 1 --2- 172.21.15.60:0/3610447038 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd38053b00 0x7efd38055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:49.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.861+0000 7efd5d39d700 1 --2- 172.21.15.60:0/3610447038 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd38053b00 0x7efd38055fc0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7efd4800b000 tx=0x7efd48006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:49.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:49.869+0000 7efd46ffd700 1 -- 172.21.15.60:0/3610447038 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efd54042dc0 con 0x7efd5805fb00 2022-06-29T16:07:50.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.009+0000 7efd5eba0700 1 -- 172.21.15.60:0/3610447038 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7efd580573d0 con 0x7efd5805fb00 2022-06-29T16:07:50.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.861+0000 7efd46ffd700 1 -- 172.21.15.60:0/3610447038 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v133) v1 ==== 97+0+0 (secure 0 0 0) 0x7efd540463e0 con 0x7efd5805fb00 2022-06-29T16:07:50.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.869+0000 7efd5eba0700 1 -- 172.21.15.60:0/3610447038 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd38053b00 msgr2=0x7efd38055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:50.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.869+0000 7efd5eba0700 1 --2- 172.21.15.60:0/3610447038 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd38053b00 0x7efd38055fc0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7efd4800b000 tx=0x7efd48006df0).stop 2022-06-29T16:07:50.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.869+0000 7efd5eba0700 1 -- 172.21.15.60:0/3610447038 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd5805fb00 msgr2=0x7efd5814da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:50.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.869+0000 7efd5eba0700 1 --2- 172.21.15.60:0/3610447038 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd5805fb00 0x7efd5814da00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7efd5400c000 tx=0x7efd54006eb0).stop 2022-06-29T16:07:50.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.869+0000 7efd5eba0700 1 -- 172.21.15.60:0/3610447038 shutdown_connections 2022-06-29T16:07:50.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.869+0000 7efd5eba0700 1 --2- 172.21.15.60:0/3610447038 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efd38053b00 0x7efd38055fc0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:50.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.869+0000 7efd5eba0700 1 --2- 172.21.15.60:0/3610447038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efd580ad350 0x7efd5810f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:50.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.869+0000 7efd5eba0700 1 --2- 172.21.15.60:0/3610447038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd580aca20 0x7efd58145390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:50.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.869+0000 7efd5eba0700 1 --2- 172.21.15.60:0/3610447038 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efd5805fb00 0x7efd5814da00 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:50.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.869+0000 7efd5eba0700 1 -- 172.21.15.60:0/3610447038 >> 172.21.15.60:0/3610447038 conn(0x7efd58063a60 msgr2=0x7efd580b4500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:50.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.869+0000 7efd5eba0700 1 -- 172.21.15.60:0/3610447038 shutdown_connections 2022-06-29T16:07:50.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.869+0000 7efd5eba0700 1 -- 172.21.15.60:0/3610447038 wait complete. 2022-06-29T16:07:50.876 INFO:teuthology.orchestra.run.smithi060.stderr: removed all blacklist entries 2022-06-29T16:07:50.888 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-06-29T16:07:50.889 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-06-29T16:07:50.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.993+0000 7fb3991aa700 1 Processor -- start 2022-06-29T16:07:50.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.993+0000 7fb3991aa700 1 -- start start 2022-06-29T16:07:51.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.993+0000 7fb3991aa700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb3940b2220 0x7fb394141130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:51.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.993+0000 7fb3991aa700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3940b2630 con 0x7fb39405c390 2022-06-29T16:07:51.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.993+0000 7fb3991aa700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb39405c770 con 0x7fb3941449c0 2022-06-29T16:07:51.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.993+0000 7fb3991aa700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb394140750 con 0x7fb3940b2220 2022-06-29T16:07:51.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.993+0000 7fb392d9d700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb39405c390 0x7fb3941442b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33394/0 (socket says 172.21.15.60:33394) 2022-06-29T16:07:51.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.993+0000 7fb392d9d700 1 -- 172.21.15.60:0/3158323403 learned_addr learned my addr 172.21.15.60:0/3158323403 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:51.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.993+0000 7fb39359e700 1 --2- 172.21.15.60:0/3158323403 >> v2:172.21.15.60:3300/0 conn(0x7fb3940b2220 0x7fb394141130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:51.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.993+0000 7fb391d9b700 1 -- 172.21.15.60:0/3158323403 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1924205938 0 0) 0x7fb3940b2630 con 0x7fb39405c390 2022-06-29T16:07:51.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.993+0000 7fb391d9b700 1 -- 172.21.15.60:0/3158323403 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb374003260 con 0x7fb39405c390 2022-06-29T16:07:51.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.993+0000 7fb39359e700 1 -- 172.21.15.60:0/3158323403 >> v1:172.21.15.92:6789/0 conn(0x7fb3941449c0 legacy=0x7fb394140a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:51.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.993+0000 7fb39359e700 1 -- 172.21.15.60:0/3158323403 >> v1:172.21.15.60:6789/0 conn(0x7fb39405c390 legacy=0x7fb3941442b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:51.002 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.993+0000 7fb39359e700 1 -- 172.21.15.60:0/3158323403 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb394143e80 con 0x7fb3940b2220 2022-06-29T16:07:51.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.993+0000 7fb39359e700 1 --2- 172.21.15.60:0/3158323403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3940b2220 0x7fb394141130 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 rx=0x7fb38c002f70 tx=0x7fb38c006df0).ready entity=mon.0 client_cookie=e7e35ec6e13cfa85 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:51.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.993+0000 7fb391d9b700 1 -- 172.21.15.60:0/3158323403 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb38c00d610 con 0x7fb3940b2220 2022-06-29T16:07:51.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.993+0000 7fb391d9b700 1 -- 172.21.15.60:0/3158323403 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb38c000e90 con 0x7fb3940b2220 2022-06-29T16:07:51.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.993+0000 7fb391d9b700 1 -- 172.21.15.60:0/3158323403 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb38c00d960 con 0x7fb3940b2220 2022-06-29T16:07:51.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.993+0000 7fb3991aa700 1 -- 172.21.15.60:0/3158323403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3940b2220 msgr2=0x7fb394141130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:51.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.993+0000 7fb3991aa700 1 --2- 172.21.15.60:0/3158323403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3940b2220 0x7fb394141130 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 rx=0x7fb38c002f70 tx=0x7fb38c006df0).stop 2022-06-29T16:07:51.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.993+0000 7fb3991aa700 1 -- 172.21.15.60:0/3158323403 shutdown_connections 2022-06-29T16:07:51.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.993+0000 7fb3991aa700 1 --2- 172.21.15.60:0/3158323403 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3940b2220 0x7fb394141130 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:51.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.997+0000 7fb3991aa700 1 -- 172.21.15.60:0/3158323403 >> 172.21.15.60:0/3158323403 conn(0x7fb394064a10 msgr2=0x7fb394064df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:51.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.997+0000 7fb3991aa700 1 -- 172.21.15.60:0/3158323403 shutdown_connections 2022-06-29T16:07:51.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.997+0000 7fb3991aa700 1 -- 172.21.15.60:0/3158323403 wait complete. 2022-06-29T16:07:51.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.997+0000 7fb3991aa700 1 Processor -- start 2022-06-29T16:07:51.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:50.997+0000 7fb3991aa700 1 -- start start 2022-06-29T16:07:51.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.001+0000 7fb3991aa700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb39405c390 0x7fb3940b64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:51.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.001+0000 7fb3991aa700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3940b2220 0x7fb394146040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:51.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.001+0000 7fb3991aa700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3941449c0 0x7fb394146530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:51.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.001+0000 7fb3991aa700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3940b6c60 con 0x7fb3940b2220 2022-06-29T16:07:51.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.001+0000 7fb3991aa700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3940b6f00 con 0x7fb3941449c0 2022-06-29T16:07:51.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.001+0000 7fb3991aa700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3940b71a0 con 0x7fb39405c390 2022-06-29T16:07:51.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.001+0000 7fb39259c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3940b2220 0x7fb394146040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:51.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.001+0000 7fb39259c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3940b2220 0x7fb394146040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53546/0 (socket says 172.21.15.60:53546) 2022-06-29T16:07:51.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.001+0000 7fb39259c700 1 -- 172.21.15.60:0/2594740558 learned_addr learned my addr 172.21.15.60:0/2594740558 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:51.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.001+0000 7fb392d9d700 1 --2- 172.21.15.60:0/2594740558 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb39405c390 0x7fb3940b64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:51.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.001+0000 7fb39359e700 1 --2- 172.21.15.60:0/2594740558 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3941449c0 0x7fb394146530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:51.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.001+0000 7fb39259c700 1 -- 172.21.15.60:0/2594740558 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb39405c390 msgr2=0x7fb3940b64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:51.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.001+0000 7fb39259c700 1 --2- 172.21.15.60:0/2594740558 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb39405c390 0x7fb3940b64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:51.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.001+0000 7fb39259c700 1 -- 172.21.15.60:0/2594740558 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3941449c0 msgr2=0x7fb394146530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:51.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.001+0000 7fb39259c700 1 --2- 172.21.15.60:0/2594740558 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3941449c0 0x7fb394146530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:51.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.001+0000 7fb39259c700 1 -- 172.21.15.60:0/2594740558 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb38c00b180 con 0x7fb3940b2220 2022-06-29T16:07:51.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.001+0000 7fb39259c700 1 --2- 172.21.15.60:0/2594740558 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3940b2220 0x7fb394146040 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 rx=0x7fb394065990 tx=0x7fb37c007bd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:51.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.001+0000 7fb38bfff700 1 -- 172.21.15.60:0/2594740558 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb37c007030 con 0x7fb3940b2220 2022-06-29T16:07:51.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.001+0000 7fb38bfff700 1 -- 172.21.15.60:0/2594740558 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb37c007190 con 0x7fb3940b2220 2022-06-29T16:07:51.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.001+0000 7fb3991aa700 1 -- 172.21.15.60:0/2594740558 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3940b7440 con 0x7fb3940b2220 2022-06-29T16:07:51.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.001+0000 7fb38bfff700 1 -- 172.21.15.60:0/2594740558 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb37c006510 con 0x7fb3940b2220 2022-06-29T16:07:51.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.001+0000 7fb3991aa700 1 -- 172.21.15.60:0/2594740558 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3940b53c0 con 0x7fb3940b2220 2022-06-29T16:07:51.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.001+0000 7fb3991aa700 1 -- 172.21.15.60:0/2594740558 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3940b7620 con 0x7fb3940b2220 2022-06-29T16:07:51.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.005+0000 7fb38bfff700 1 -- 172.21.15.60:0/2594740558 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb37c006670 con 0x7fb3940b2220 2022-06-29T16:07:51.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.005+0000 7fb38bfff700 1 --2- 172.21.15.60:0/2594740558 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb3780539e0 0x7fb378055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:51.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.005+0000 7fb38bfff700 1 -- 172.21.15.60:0/2594740558 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fb37c06ba00 con 0x7fb3940b2220 2022-06-29T16:07:51.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.005+0000 7fb392d9d700 1 --2- 172.21.15.60:0/2594740558 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb3780539e0 0x7fb378055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:51.016 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.005+0000 7fb392d9d700 1 --2- 172.21.15.60:0/2594740558 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb3780539e0 0x7fb378055ea0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7fb38400d000 tx=0x7fb384009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:51.016 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.009+0000 7fb38bfff700 1 -- 172.21.15.60:0/2594740558 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb37c042ae0 con 0x7fb3940b2220 2022-06-29T16:07:51.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.177+0000 7fb3991aa700 1 -- 172.21.15.60:0/2594740558 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fb3940b75d0 con 0x7fb3940b2220 2022-06-29T16:07:51.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.177+0000 7fb38bfff700 1 -- 172.21.15.60:0/2594740558 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v43) v1 ==== 80+0+4242 (secure 0 0 0) 0x7fb37c046100 con 0x7fb3940b2220 2022-06-29T16:07:51.183 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:51.183 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:07:51.183 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_dump": [ 2022-06-29T16:07:51.184 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:51.184 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.a", 2022-06-29T16:07:51.184 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxivv7JNhAAnA291UxlB/SfgITOipP00A==", 2022-06-29T16:07:51.184 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:07:51.185 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:07:51.185 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:07:51.185 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:07:51.185 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:07:51.186 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:51.186 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:51.186 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:51.186 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.b", 2022-06-29T16:07:51.187 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxiRSr/OhAAfFQTF2YHvzCv5wXBVCA5WQ==", 2022-06-29T16:07:51.187 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:07:51.188 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:07:51.188 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:07:51.188 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:07:51.188 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:07:51.189 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:51.189 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:51.190 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:51.190 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.0", 2022-06-29T16:07:51.191 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiZYPxOBAAIkweWJlTJIXgxnlagkGdXA==", 2022-06-29T16:07:51.191 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:07:51.192 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:07:51.192 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:07:51.192 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:07:51.193 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:51.193 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:51.194 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:51.194 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.1", 2022-06-29T16:07:51.194 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxiS2JGBhAAJn0UfAxZI3IVFyD5QBdb/Q==", 2022-06-29T16:07:51.195 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:07:51.195 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:07:51.196 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:07:51.196 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:07:51.196 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:51.196 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:51.197 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:51.197 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.2", 2022-06-29T16:07:51.197 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxiPhkfEhAApH7r8OUJzW7fZNM6XVv1dg==", 2022-06-29T16:07:51.197 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:07:51.198 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:07:51.198 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:07:51.198 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:07:51.198 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:51.199 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:51.199 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:51.199 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.3", 2022-06-29T16:07:51.199 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxit/47HhAA1vyU6h7hjPG+op9qP/rVYQ==", 2022-06-29T16:07:51.199 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:07:51.200 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:07:51.200 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:07:51.200 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:07:51.200 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:51.201 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:51.201 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:51.201 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.4", 2022-06-29T16:07:51.201 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxiaKzKBxAAbIwCF9ag7ezsNy11hi0BEw==", 2022-06-29T16:07:51.202 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:07:51.202 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:07:51.202 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:07:51.202 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:07:51.202 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:51.203 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:51.203 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:51.203 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.5", 2022-06-29T16:07:51.203 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxiJQV/IBAASBBDb6UFgEtXILe4fcT/xA==", 2022-06-29T16:07:51.204 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:07:51.204 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:07:51.204 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:07:51.204 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:07:51.204 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:51.205 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:51.205 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:51.205 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.6", 2022-06-29T16:07:51.205 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxi6D07OBAA2FWiZ+mNK+jFMp/kPZnbQQ==", 2022-06-29T16:07:51.206 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:07:51.206 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:07:51.206 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:07:51.206 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:07:51.207 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:51.207 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:51.207 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:51.207 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.7", 2022-06-29T16:07:51.208 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCMdrxinEOcEhAA562ybpzGHNQfWIq8q7zw9w==", 2022-06-29T16:07:51.208 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:07:51.208 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:07:51.208 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:07:51.209 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:07:51.209 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:51.209 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:51.209 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:51.210 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.0", 2022-06-29T16:07:51.210 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiHXPwBRAAkOOwlYJvIW7dqVFdlXXoqw==", 2022-06-29T16:07:51.210 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:07:51.210 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:07:51.211 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2022-06-29T16:07:51.211 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw pool=cephfs_data" 2022-06-29T16:07:51.211 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:51.211 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:51.212 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:51.212 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.1", 2022-06-29T16:07:51.212 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxibe5JCRAAGBJbi9t09w0836JqrAT7+w==", 2022-06-29T16:07:51.212 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:07:51.213 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:07:51.213 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2022-06-29T16:07:51.213 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw pool=cephfs_data" 2022-06-29T16:07:51.213 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:51.214 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:51.214 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:51.214 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.2", 2022-06-29T16:07:51.214 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiK3dBDRAA7XINPSOPQDfOrUYEpHC+gw==", 2022-06-29T16:07:51.214 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:07:51.215 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:07:51.215 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2022-06-29T16:07:51.215 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw pool=cephfs_data" 2022-06-29T16:07:51.215 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:51.216 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:51.216 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:51.216 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.admin", 2022-06-29T16:07:51.216 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxieKVcGBAAnjH80a0qWIArz2qwjYHWdQ==", 2022-06-29T16:07:51.217 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:07:51.217 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T16:07:51.217 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:07:51.217 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:07:51.218 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:07:51.218 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:51.218 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:51.218 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:51.218 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.x", 2022-06-29T16:07:51.219 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxi2LZJNBAA3MKb/pXYbQCsoRx8EQfGbw==", 2022-06-29T16:07:51.219 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:07:51.219 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T16:07:51.219 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2022-06-29T16:07:51.220 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:07:51.220 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:51.220 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:51.220 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:51.220 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.y", 2022-06-29T16:07:51.221 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxi3LmaMBAAN+0mZrdKXqSXOipVTA1drw==", 2022-06-29T16:07:51.221 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:07:51.221 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T16:07:51.221 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2022-06-29T16:07:51.222 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:07:51.222 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:51.222 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:51.222 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:07:51.222 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:07:51.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.181+0000 7fb3991aa700 1 -- 172.21.15.60:0/2594740558 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb3780539e0 msgr2=0x7fb378055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:51.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.181+0000 7fb3991aa700 1 --2- 172.21.15.60:0/2594740558 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb3780539e0 0x7fb378055ea0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7fb38400d000 tx=0x7fb384009040).stop 2022-06-29T16:07:51.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.181+0000 7fb3991aa700 1 -- 172.21.15.60:0/2594740558 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3940b2220 msgr2=0x7fb394146040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:51.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.181+0000 7fb3991aa700 1 --2- 172.21.15.60:0/2594740558 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3940b2220 0x7fb394146040 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 rx=0x7fb394065990 tx=0x7fb37c007bd0).stop 2022-06-29T16:07:51.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.181+0000 7fb3991aa700 1 -- 172.21.15.60:0/2594740558 shutdown_connections 2022-06-29T16:07:51.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.181+0000 7fb3991aa700 1 --2- 172.21.15.60:0/2594740558 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb3780539e0 0x7fb378055ea0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:51.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.181+0000 7fb3991aa700 1 --2- 172.21.15.60:0/2594740558 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3941449c0 0x7fb394146530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:51.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.181+0000 7fb3991aa700 1 --2- 172.21.15.60:0/2594740558 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3940b2220 0x7fb394146040 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:51.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.181+0000 7fb3991aa700 1 --2- 172.21.15.60:0/2594740558 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb39405c390 0x7fb3940b64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:51.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.181+0000 7fb3991aa700 1 -- 172.21.15.60:0/2594740558 >> 172.21.15.60:0/2594740558 conn(0x7fb394064a10 msgr2=0x7fb3940b17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:51.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.181+0000 7fb3991aa700 1 -- 172.21.15.60:0/2594740558 shutdown_connections 2022-06-29T16:07:51.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.181+0000 7fb3991aa700 1 -- 172.21.15.60:0/2594740558 wait complete. 2022-06-29T16:07:51.228 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:07:51.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f309359e700 1 Processor -- start 2022-06-29T16:07:51.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f309359e700 1 -- start start 2022-06-29T16:07:51.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f309359e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f30940aed00 0x7f3094146810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:51.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f309359e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f309405d450 con 0x7f30940b2130 2022-06-29T16:07:51.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f309359e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30940b25d0 con 0x7f309405d070 2022-06-29T16:07:51.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f309359e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30940adf90 con 0x7f30940aed00 2022-06-29T16:07:51.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f3091d9b700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f30940b2130 0x7f3094146100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33406/0 (socket says 172.21.15.60:33406) 2022-06-29T16:07:51.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f3091d9b700 1 -- 172.21.15.60:0/1876320153 learned_addr learned my addr 172.21.15.60:0/1876320153 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:51.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f3092d9d700 1 --2- 172.21.15.60:0/1876320153 >> v2:172.21.15.92:3300/0 conn(0x7f30940aed00 0x7f3094146810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:51.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f309159a700 1 -- 172.21.15.60:0/1876320153 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2678621713 0 0) 0x7f309405d450 con 0x7f30940b2130 2022-06-29T16:07:51.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f309159a700 1 -- 172.21.15.60:0/1876320153 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3078003260 con 0x7f30940b2130 2022-06-29T16:07:51.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f309159a700 1 -- 172.21.15.60:0/1876320153 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2300775769 0 0) 0x7f3078003260 con 0x7f30940b2130 2022-06-29T16:07:51.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f309159a700 1 -- 172.21.15.60:0/1876320153 >> v2:172.21.15.92:3300/0 conn(0x7f30940aed00 msgr2=0x7f3094146810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:51.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f309159a700 1 --2- 172.21.15.60:0/1876320153 >> v2:172.21.15.92:3300/0 conn(0x7f30940aed00 0x7f3094146810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:51.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f309159a700 1 -- 172.21.15.60:0/1876320153 >> v1:172.21.15.92:6789/0 conn(0x7f309405d070 legacy=0x7f309414aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:51.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f309159a700 1 -- 172.21.15.60:0/1876320153 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f309414a350 con 0x7f30940b2130 2022-06-29T16:07:51.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f309159a700 1 -- 172.21.15.60:0/1876320153 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f308c0019e0 con 0x7f30940b2130 2022-06-29T16:07:51.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f309159a700 1 -- 172.21.15.60:0/1876320153 >> v1:172.21.15.60:6789/0 conn(0x7f30940b2130 legacy=0x7f3094146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:51.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f309159a700 1 --2- 172.21.15.60:0/1876320153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3078003750 0x7f3078003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:51.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f309159a700 1 --2- 172.21.15.60:0/1876320153 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3078004500 0x7f3078004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:51.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f309159a700 1 --2- 172.21.15.60:0/1876320153 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3078004e70 0x7f307800b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:51.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f309159a700 1 -- 172.21.15.60:0/1876320153 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30940adf90 con 0x7f3078003750 2022-06-29T16:07:51.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f309159a700 1 -- 172.21.15.60:0/1876320153 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307800bdd0 con 0x7f3078004e70 2022-06-29T16:07:51.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f309159a700 1 -- 172.21.15.60:0/1876320153 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f307800bf60 con 0x7f3078004500 2022-06-29T16:07:51.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f309259c700 1 --2- 172.21.15.60:0/1876320153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3078003750 0x7f3078003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:51.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f3091d9b700 1 --2- 172.21.15.60:0/1876320153 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3078004500 0x7f3078004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:51.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f3092d9d700 1 --2- 172.21.15.60:0/1876320153 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3078004e70 0x7f307800b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:51.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f309259c700 1 -- 172.21.15.60:0/1876320153 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3078004500 msgr2=0x7f3078004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:51.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f309259c700 1 --2- 172.21.15.60:0/1876320153 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3078004500 0x7f3078004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:51.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f309259c700 1 -- 172.21.15.60:0/1876320153 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3078004e70 msgr2=0x7f307800b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:51.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f309259c700 1 --2- 172.21.15.60:0/1876320153 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3078004e70 0x7f307800b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:51.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f309259c700 1 -- 172.21.15.60:0/1876320153 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3078003260 con 0x7f3078003750 2022-06-29T16:07:51.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f3092d9d700 1 --2- 172.21.15.60:0/1876320153 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3078004e70 0x7f307800b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:07:51.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f3092d9d700 1 -- 172.21.15.60:0/1876320153 reap_dead start 2022-06-29T16:07:51.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f309259c700 1 --2- 172.21.15.60:0/1876320153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3078003750 0x7f3078003b70 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 rx=0x7f30940b5cd0 tx=0x7f307c007a00).ready entity=mon.0 client_cookie=2aa1d73723ea09ed server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:51.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f309159a700 1 -- 172.21.15.60:0/1876320153 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f307c007d10 con 0x7f3078003750 2022-06-29T16:07:51.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f309159a700 1 -- 172.21.15.60:0/1876320153 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f307c007030 con 0x7f3078003750 2022-06-29T16:07:51.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.333+0000 7f309159a700 1 -- 172.21.15.60:0/1876320153 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f307c007390 con 0x7f3078003750 2022-06-29T16:07:51.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.337+0000 7f309359e700 1 -- 172.21.15.60:0/1876320153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3078003750 msgr2=0x7f3078003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:51.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.337+0000 7f309359e700 1 --2- 172.21.15.60:0/1876320153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3078003750 0x7f3078003b70 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 rx=0x7f30940b5cd0 tx=0x7f307c007a00).stop 2022-06-29T16:07:51.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.337+0000 7f309359e700 1 -- 172.21.15.60:0/1876320153 shutdown_connections 2022-06-29T16:07:51.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.337+0000 7f309359e700 1 --2- 172.21.15.60:0/1876320153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3078003750 0x7f3078003b70 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:51.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.337+0000 7f309359e700 1 -- 172.21.15.60:0/1876320153 >> 172.21.15.60:0/1876320153 conn(0x7f30940b5890 msgr2=0x7f30940b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:51.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.337+0000 7f309359e700 1 -- 172.21.15.60:0/1876320153 shutdown_connections 2022-06-29T16:07:51.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.337+0000 7f309359e700 1 -- 172.21.15.60:0/1876320153 wait complete. 2022-06-29T16:07:51.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.337+0000 7f309359e700 1 Processor -- start 2022-06-29T16:07:51.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.337+0000 7f309359e700 1 -- start start 2022-06-29T16:07:51.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.337+0000 7f309359e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3078003750 0x7f309414c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:51.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.337+0000 7f309359e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3094146410 0x7f3094146890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:51.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.337+0000 7f309359e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3094141040 0x7f30941414f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:51.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.337+0000 7f309359e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309414d840 con 0x7f3094141040 2022-06-29T16:07:51.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.337+0000 7f309359e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3094063560 con 0x7f3078003750 2022-06-29T16:07:51.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.337+0000 7f309359e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3094063800 con 0x7f3094146410 2022-06-29T16:07:51.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.337+0000 7f3092d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3094141040 0x7f30941414f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:51.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.337+0000 7f3092d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3094141040 0x7f30941414f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53566/0 (socket says 172.21.15.60:53566) 2022-06-29T16:07:51.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.337+0000 7f3092d9d700 1 -- 172.21.15.60:0/413489002 learned_addr learned my addr 172.21.15.60:0/413489002 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:51.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.337+0000 7f3091d9b700 1 --2- 172.21.15.60:0/413489002 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3094146410 0x7f3094146890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:51.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.337+0000 7f309259c700 1 --2- 172.21.15.60:0/413489002 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3078003750 0x7f309414c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:51.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.337+0000 7f3092d9d700 1 -- 172.21.15.60:0/413489002 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3094146410 msgr2=0x7f3094146890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:51.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.337+0000 7f3092d9d700 1 --2- 172.21.15.60:0/413489002 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3094146410 0x7f3094146890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:51.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.337+0000 7f3092d9d700 1 -- 172.21.15.60:0/413489002 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3078003750 msgr2=0x7f309414c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:51.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.337+0000 7f3092d9d700 1 --2- 172.21.15.60:0/413489002 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3078003750 0x7f309414c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:51.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.337+0000 7f3092d9d700 1 -- 172.21.15.60:0/413489002 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3094063aa0 con 0x7f3094141040 2022-06-29T16:07:51.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.337+0000 7f3091d9b700 1 --2- 172.21.15.60:0/413489002 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3094146410 0x7f3094146890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:51.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.337+0000 7f309259c700 1 --2- 172.21.15.60:0/413489002 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3078003750 0x7f309414c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:51.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.341+0000 7f3092d9d700 1 --2- 172.21.15.60:0/413489002 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3094141040 0x7f30941414f0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 rx=0x7f308400b000 tx=0x7f3084008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:51.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.341+0000 7f308b7fe700 1 -- 172.21.15.60:0/413489002 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3084017070 con 0x7f3094141040 2022-06-29T16:07:51.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.341+0000 7f308b7fe700 1 -- 172.21.15.60:0/413489002 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30840135f0 con 0x7f3094141040 2022-06-29T16:07:51.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.341+0000 7f308b7fe700 1 -- 172.21.15.60:0/413489002 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3084013950 con 0x7f3094141040 2022-06-29T16:07:51.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.341+0000 7f309359e700 1 -- 172.21.15.60:0/413489002 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3094063f40 con 0x7f3094141040 2022-06-29T16:07:51.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.341+0000 7f309359e700 1 -- 172.21.15.60:0/413489002 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3094141b80 con 0x7f3094141040 2022-06-29T16:07:51.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.341+0000 7f309359e700 1 -- 172.21.15.60:0/413489002 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f309414a5f0 con 0x7f3094141040 2022-06-29T16:07:51.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.341+0000 7f308b7fe700 1 -- 172.21.15.60:0/413489002 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3084013ab0 con 0x7f3094141040 2022-06-29T16:07:51.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.341+0000 7f308b7fe700 1 --2- 172.21.15.60:0/413489002 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f306c053aa0 0x7f306c055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:51.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.341+0000 7f308b7fe700 1 -- 172.21.15.60:0/413489002 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f308406b8e0 con 0x7f3094141040 2022-06-29T16:07:51.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.341+0000 7f309259c700 1 --2- 172.21.15.60:0/413489002 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f306c053aa0 0x7f306c055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:51.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.345+0000 7f309259c700 1 --2- 172.21.15.60:0/413489002 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f306c053aa0 0x7f306c055f60 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0x7f307c019000 tx=0x7f307c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:51.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.349+0000 7f308b7fe700 1 -- 172.21.15.60:0/413489002 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3084042b30 con 0x7f3094141040 2022-06-29T16:07:51.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.477+0000 7f309359e700 1 -- 172.21.15.60:0/413489002 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3094063c30 con 0x7f3094141040 2022-06-29T16:07:51.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.481+0000 7f308b7fe700 1 -- 172.21.15.60:0/413489002 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v133) v1 ==== 74+0+13191 (secure 0 0 0) 0x7f3084046150 con 0x7f3094141040 2022-06-29T16:07:51.485 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:51.485 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":133,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:07:50.856363+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:07:51.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.481+0000 7f309359e700 1 -- 172.21.15.60:0/413489002 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f306c053aa0 msgr2=0x7f306c055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:51.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.481+0000 7f309359e700 1 --2- 172.21.15.60:0/413489002 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f306c053aa0 0x7f306c055f60 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0x7f307c019000 tx=0x7f307c015000).stop 2022-06-29T16:07:51.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.481+0000 7f309359e700 1 -- 172.21.15.60:0/413489002 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3094141040 msgr2=0x7f30941414f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:51.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.481+0000 7f309359e700 1 --2- 172.21.15.60:0/413489002 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3094141040 0x7f30941414f0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 rx=0x7f308400b000 tx=0x7f3084008040).stop 2022-06-29T16:07:51.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.481+0000 7f309359e700 1 -- 172.21.15.60:0/413489002 shutdown_connections 2022-06-29T16:07:51.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.481+0000 7f309359e700 1 --2- 172.21.15.60:0/413489002 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f306c053aa0 0x7f306c055f60 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:51.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.481+0000 7f309359e700 1 --2- 172.21.15.60:0/413489002 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3094141040 0x7f30941414f0 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:51.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.481+0000 7f309359e700 1 --2- 172.21.15.60:0/413489002 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3094146410 0x7f3094146890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:51.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.481+0000 7f309359e700 1 --2- 172.21.15.60:0/413489002 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3078003750 0x7f309414c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:51.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.481+0000 7f309359e700 1 -- 172.21.15.60:0/413489002 >> 172.21.15.60:0/413489002 conn(0x7f30940b5890 msgr2=0x7f30940b4540 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:51.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.485+0000 7f309359e700 1 -- 172.21.15.60:0/413489002 shutdown_connections 2022-06-29T16:07:51.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.485+0000 7f309359e700 1 -- 172.21.15.60:0/413489002 wait complete. 2022-06-29T16:07:51.504 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-06-29T16:07:51.505 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:07:51.610 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.605+0000 7fded3fff700 1 Processor -- start 2022-06-29T16:07:51.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.605+0000 7fded3fff700 1 -- start start 2022-06-29T16:07:51.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.605+0000 7fded3fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fded405f2c0 0x7fded405f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:51.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.605+0000 7fded3fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fded405c3f0 0x7fded405c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:51.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.605+0000 7fded3fff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded405bd20 con 0x7fded405ff90 2022-06-29T16:07:51.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.605+0000 7fded3fff700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fded41438e0 con 0x7fded405f2c0 2022-06-29T16:07:51.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.605+0000 7fded3fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fded405bf50 con 0x7fded405c3f0 2022-06-29T16:07:51.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.609+0000 7fded2ffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fded405f2c0 0x7fded405f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:51.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.609+0000 7fded2ffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fded405f2c0 0x7fded405f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53574/0 (socket says 172.21.15.60:53574) 2022-06-29T16:07:51.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.609+0000 7fded27fc700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fded405ff90 0x7fded40aede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:37674/0 (socket says 172.21.15.60:37674) 2022-06-29T16:07:51.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.609+0000 7fded37fe700 1 --2- 172.21.15.60:0/3182977431 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fded405c3f0 0x7fded405c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:51.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.609+0000 7fded2ffd700 1 -- 172.21.15.60:0/3182977431 learned_addr learned my addr 172.21.15.60:0/3182977431 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:51.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.609+0000 7fded2ffd700 1 -- 172.21.15.60:0/3182977431 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fded405c3f0 msgr2=0x7fded405c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:51.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.609+0000 7fded2ffd700 1 --2- 172.21.15.60:0/3182977431 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fded405c3f0 0x7fded405c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:51.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.609+0000 7fded2ffd700 1 -- 172.21.15.60:0/3182977431 >> v1:172.21.15.92:6789/0 conn(0x7fded405ff90 legacy=0x7fded40aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:51.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.609+0000 7fded2ffd700 1 -- 172.21.15.60:0/3182977431 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded4143560 con 0x7fded405f2c0 2022-06-29T16:07:51.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.609+0000 7fded1ffb700 1 -- 172.21.15.60:0/3182977431 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3996224946 0 0) 0x7fded405bd20 con 0x7fded405ff90 2022-06-29T16:07:51.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.609+0000 7fded2ffd700 1 --2- 172.21.15.60:0/3182977431 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fded405f2c0 0x7fded405f6a0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 rx=0x7fdec4002f70 tx=0x7fdec4006eb0).ready entity=mon.0 client_cookie=58a9388038d57830 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:51.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.609+0000 7fded1ffb700 1 -- 172.21.15.60:0/3182977431 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdec400d570 con 0x7fded405f2c0 2022-06-29T16:07:51.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.609+0000 7fded1ffb700 1 -- 172.21.15.60:0/3182977431 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec4000e90 con 0x7fded405f2c0 2022-06-29T16:07:51.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.609+0000 7fded1ffb700 1 -- 172.21.15.60:0/3182977431 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdec400d8c0 con 0x7fded405f2c0 2022-06-29T16:07:51.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.609+0000 7fded3fff700 1 -- 172.21.15.60:0/3182977431 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fded405f2c0 msgr2=0x7fded405f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:51.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.609+0000 7fded3fff700 1 --2- 172.21.15.60:0/3182977431 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fded405f2c0 0x7fded405f6a0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 rx=0x7fdec4002f70 tx=0x7fdec4006eb0).stop 2022-06-29T16:07:51.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.609+0000 7fded3fff700 1 -- 172.21.15.60:0/3182977431 shutdown_connections 2022-06-29T16:07:51.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.609+0000 7fded3fff700 1 --2- 172.21.15.60:0/3182977431 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fded405c3f0 0x7fded405c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:51.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.609+0000 7fded3fff700 1 --2- 172.21.15.60:0/3182977431 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fded405f2c0 0x7fded405f6a0 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:51.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.609+0000 7fded3fff700 1 -- 172.21.15.60:0/3182977431 >> 172.21.15.60:0/3182977431 conn(0x7fded40b91e0 msgr2=0x7fded40b95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:51.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.609+0000 7fded3fff700 1 -- 172.21.15.60:0/3182977431 shutdown_connections 2022-06-29T16:07:51.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.609+0000 7fded3fff700 1 -- 172.21.15.60:0/3182977431 wait complete. 2022-06-29T16:07:51.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.613+0000 7fded3fff700 1 Processor -- start 2022-06-29T16:07:51.622 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.613+0000 7fded3fff700 1 -- start start 2022-06-29T16:07:51.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.613+0000 7fded3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fded405c3f0 0x7fded409c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:51.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.613+0000 7fded3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fded405f2c0 0x7fded4090120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:51.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.613+0000 7fded3fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fded405ff90 0x7fded4090610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:51.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.613+0000 7fded3fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded40b3040 con 0x7fded405f2c0 2022-06-29T16:07:51.623 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.613+0000 7fded3fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded4090ba0 con 0x7fded405c3f0 2022-06-29T16:07:51.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.613+0000 7fded3fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fded4152940 con 0x7fded405ff90 2022-06-29T16:07:51.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.613+0000 7fded27fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fded405f2c0 0x7fded4090120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:51.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.613+0000 7fded27fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fded405f2c0 0x7fded4090120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53580/0 (socket says 172.21.15.60:53580) 2022-06-29T16:07:51.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.613+0000 7fded2ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fded405c3f0 0x7fded409c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:51.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.613+0000 7fded27fc700 1 -- 172.21.15.60:0/430761580 learned_addr learned my addr 172.21.15.60:0/430761580 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:51.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.613+0000 7fded37fe700 1 --2- 172.21.15.60:0/430761580 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fded405ff90 0x7fded4090610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:51.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.613+0000 7fded27fc700 1 -- 172.21.15.60:0/430761580 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fded405ff90 msgr2=0x7fded4090610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:51.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.613+0000 7fded27fc700 1 --2- 172.21.15.60:0/430761580 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fded405ff90 0x7fded4090610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:51.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.613+0000 7fded27fc700 1 -- 172.21.15.60:0/430761580 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fded405c3f0 msgr2=0x7fded409c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:51.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.613+0000 7fded27fc700 1 --2- 172.21.15.60:0/430761580 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fded405c3f0 0x7fded409c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:51.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.613+0000 7fded27fc700 1 -- 172.21.15.60:0/430761580 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec400b1e0 con 0x7fded405f2c0 2022-06-29T16:07:51.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.613+0000 7fded27fc700 1 --2- 172.21.15.60:0/430761580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fded405f2c0 0x7fded4090120 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 rx=0x7fdec8000ed0 tx=0x7fdec8007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:51.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.613+0000 7fdec3fff700 1 -- 172.21.15.60:0/430761580 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdec80070d0 con 0x7fded405f2c0 2022-06-29T16:07:51.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.613+0000 7fdec3fff700 1 -- 172.21.15.60:0/430761580 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec8007230 con 0x7fded405f2c0 2022-06-29T16:07:51.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.613+0000 7fded3fff700 1 -- 172.21.15.60:0/430761580 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fded4152be0 con 0x7fded405f2c0 2022-06-29T16:07:51.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.613+0000 7fdec3fff700 1 -- 172.21.15.60:0/430761580 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdec8005550 con 0x7fded405f2c0 2022-06-29T16:07:51.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.613+0000 7fded3fff700 1 -- 172.21.15.60:0/430761580 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fded409cd20 con 0x7fded405f2c0 2022-06-29T16:07:51.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.617+0000 7fded3fff700 1 -- 172.21.15.60:0/430761580 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fded4152d70 con 0x7fded405f2c0 2022-06-29T16:07:51.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.617+0000 7fdec3fff700 1 -- 172.21.15.60:0/430761580 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fdec8010070 con 0x7fded405f2c0 2022-06-29T16:07:51.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.617+0000 7fdec3fff700 1 --2- 172.21.15.60:0/430761580 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdeb0053b30 0x7fdeb0055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:51.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.617+0000 7fdec3fff700 1 -- 172.21.15.60:0/430761580 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fdec806b1e0 con 0x7fded405f2c0 2022-06-29T16:07:51.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.617+0000 7fded2ffd700 1 --2- 172.21.15.60:0/430761580 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdeb0053b30 0x7fdeb0055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:51.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.617+0000 7fded2ffd700 1 --2- 172.21.15.60:0/430761580 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdeb0053b30 0x7fdeb0055ff0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7fdec401c000 tx=0x7fdec4015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:51.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.625+0000 7fdec3fff700 1 -- 172.21.15.60:0/430761580 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdec8042320 con 0x7fded405f2c0 2022-06-29T16:07:51.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.753+0000 7fded3fff700 1 -- 172.21.15.60:0/430761580 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fded40573d0 con 0x7fded405f2c0 2022-06-29T16:07:51.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.753+0000 7fdec3fff700 1 -- 172.21.15.60:0/430761580 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v133) v1 ==== 110+0+10 (secure 0 0 0) 0x7fdec8045940 con 0x7fded405f2c0 2022-06-29T16:07:51.758 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:07:51.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.753+0000 7fded3fff700 1 -- 172.21.15.60:0/430761580 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdeb0053b30 msgr2=0x7fdeb0055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:51.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.753+0000 7fded3fff700 1 --2- 172.21.15.60:0/430761580 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdeb0053b30 0x7fdeb0055ff0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7fdec401c000 tx=0x7fdec4015000).stop 2022-06-29T16:07:51.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.757+0000 7fded3fff700 1 -- 172.21.15.60:0/430761580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fded405f2c0 msgr2=0x7fded4090120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:51.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.757+0000 7fded3fff700 1 --2- 172.21.15.60:0/430761580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fded405f2c0 0x7fded4090120 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 rx=0x7fdec8000ed0 tx=0x7fdec8007c50).stop 2022-06-29T16:07:51.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.757+0000 7fded3fff700 1 -- 172.21.15.60:0/430761580 shutdown_connections 2022-06-29T16:07:51.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.757+0000 7fded3fff700 1 --2- 172.21.15.60:0/430761580 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fdeb0053b30 0x7fdeb0055ff0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:51.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.757+0000 7fded3fff700 1 --2- 172.21.15.60:0/430761580 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fded405ff90 0x7fded4090610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:51.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.757+0000 7fded3fff700 1 --2- 172.21.15.60:0/430761580 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fded405f2c0 0x7fded4090120 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:51.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.757+0000 7fded3fff700 1 --2- 172.21.15.60:0/430761580 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fded405c3f0 0x7fded409c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:51.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.757+0000 7fded3fff700 1 -- 172.21.15.60:0/430761580 >> 172.21.15.60:0/430761580 conn(0x7fded40b91e0 msgr2=0x7fded4140a60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:51.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.757+0000 7fded3fff700 1 -- 172.21.15.60:0/430761580 shutdown_connections 2022-06-29T16:07:51.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.757+0000 7fded3fff700 1 -- 172.21.15.60:0/430761580 wait complete. 2022-06-29T16:07:51.771 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:07:51.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.877+0000 7f8b209d4700 1 Processor -- start 2022-06-29T16:07:51.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.877+0000 7f8b209d4700 1 -- start start 2022-06-29T16:07:51.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.877+0000 7f8b209d4700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8b1c05fb00 0x7f8b1c0ae890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:51.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.877+0000 7f8b209d4700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b1c0ad240 con 0x7f8b1c0b0fa0 2022-06-29T16:07:51.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.877+0000 7f8b209d4700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b1c05fee0 con 0x7f8b1c0b1c00 2022-06-29T16:07:51.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.877+0000 7f8b209d4700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b1c13ce60 con 0x7f8b1c05fb00 2022-06-29T16:07:51.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.877+0000 7f8b1ad9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8b1c05fb00 0x7f8b1c0ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:51.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.877+0000 7f8b1ad9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8b1c05fb00 0x7f8b1c0ae890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53586/0 (socket says 172.21.15.60:53586) 2022-06-29T16:07:51.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.877+0000 7f8b1ad9d700 1 -- 172.21.15.60:0/3841487382 learned_addr learned my addr 172.21.15.60:0/3841487382 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:51.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.877+0000 7f8b1a59c700 1 --1- 172.21.15.60:0/3841487382 >> v1:172.21.15.60:6789/0 conn(0x7f8b1c0b0fa0 0x7f8b1c0b14f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33442/0 (socket says 172.21.15.60:33442) 2022-06-29T16:07:51.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.877+0000 7f8b1ad9d700 1 -- 172.21.15.60:0/3841487382 >> v1:172.21.15.92:6789/0 conn(0x7f8b1c0b1c00 legacy=0x7f8b1c067250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:51.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.877+0000 7f8b1ad9d700 1 -- 172.21.15.60:0/3841487382 >> v1:172.21.15.60:6789/0 conn(0x7f8b1c0b0fa0 legacy=0x7f8b1c0b14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:51.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.877+0000 7f8b19d9b700 1 -- 172.21.15.60:0/3841487382 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3640647289 0 0) 0x7f8b1c0ad240 con 0x7f8b1c0b0fa0 2022-06-29T16:07:51.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.877+0000 7f8b1ad9d700 1 -- 172.21.15.60:0/3841487382 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b1c143180 con 0x7f8b1c05fb00 2022-06-29T16:07:51.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.877+0000 7f8b1ad9d700 1 --2- 172.21.15.60:0/3841487382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b1c05fb00 0x7f8b1c0ae890 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 rx=0x7f8b0c002f70 tx=0x7f8b0c006f40).ready entity=mon.0 client_cookie=ca94f38ec7b3b2fa server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:51.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.881+0000 7f8b19d9b700 1 -- 172.21.15.60:0/3841487382 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b0c00c900 con 0x7f8b1c05fb00 2022-06-29T16:07:51.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.881+0000 7f8b19d9b700 1 -- 172.21.15.60:0/3841487382 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b0c000e90 con 0x7f8b1c05fb00 2022-06-29T16:07:51.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.881+0000 7f8b19d9b700 1 -- 172.21.15.60:0/3841487382 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b0c00cc50 con 0x7f8b1c05fb00 2022-06-29T16:07:51.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.881+0000 7f8b209d4700 1 -- 172.21.15.60:0/3841487382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b1c05fb00 msgr2=0x7f8b1c0ae890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:51.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.881+0000 7f8b209d4700 1 --2- 172.21.15.60:0/3841487382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b1c05fb00 0x7f8b1c0ae890 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 rx=0x7f8b0c002f70 tx=0x7f8b0c006f40).stop 2022-06-29T16:07:51.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.881+0000 7f8b209d4700 1 -- 172.21.15.60:0/3841487382 shutdown_connections 2022-06-29T16:07:51.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.881+0000 7f8b209d4700 1 --2- 172.21.15.60:0/3841487382 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b1c05fb00 0x7f8b1c0ae890 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:51.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.881+0000 7f8b209d4700 1 -- 172.21.15.60:0/3841487382 >> 172.21.15.60:0/3841487382 conn(0x7f8b1c063a60 msgr2=0x7f8b1c05f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:51.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.881+0000 7f8b209d4700 1 -- 172.21.15.60:0/3841487382 shutdown_connections 2022-06-29T16:07:51.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.881+0000 7f8b209d4700 1 -- 172.21.15.60:0/3841487382 wait complete. 2022-06-29T16:07:51.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.881+0000 7f8b209d4700 1 Processor -- start 2022-06-29T16:07:51.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.881+0000 7f8b209d4700 1 -- start start 2022-06-29T16:07:51.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.885+0000 7f8b209d4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b1c05fb00 0x7f8b1c064f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:51.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.885+0000 7f8b209d4700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b1c0b0fa0 0x7f8b1c14db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:51.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.885+0000 7f8b209d4700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8b1c0b1c00 0x7f8b1c14e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:51.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.885+0000 7f8b209d4700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1c067010 con 0x7f8b1c05fb00 2022-06-29T16:07:51.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.885+0000 7f8b209d4700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1c10e930 con 0x7f8b1c0b0fa0 2022-06-29T16:07:51.894 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.885+0000 7f8b209d4700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1c10ebd0 con 0x7f8b1c0b1c00 2022-06-29T16:07:51.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.885+0000 7f8b1ad9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b1c05fb00 0x7f8b1c064f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:51.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.885+0000 7f8b1ad9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b1c05fb00 0x7f8b1c064f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53592/0 (socket says 172.21.15.60:53592) 2022-06-29T16:07:51.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.885+0000 7f8b1ad9d700 1 -- 172.21.15.60:0/4213026619 learned_addr learned my addr 172.21.15.60:0/4213026619 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:51.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.885+0000 7f8b1a59c700 1 --2- 172.21.15.60:0/4213026619 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b1c0b0fa0 0x7f8b1c14db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:51.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.885+0000 7f8b1b59e700 1 --2- 172.21.15.60:0/4213026619 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8b1c0b1c00 0x7f8b1c14e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:51.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.885+0000 7f8b1ad9d700 1 -- 172.21.15.60:0/4213026619 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8b1c0b1c00 msgr2=0x7f8b1c14e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:51.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.885+0000 7f8b1ad9d700 1 --2- 172.21.15.60:0/4213026619 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8b1c0b1c00 0x7f8b1c14e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:51.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.885+0000 7f8b1ad9d700 1 -- 172.21.15.60:0/4213026619 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b1c0b0fa0 msgr2=0x7f8b1c14db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:51.896 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.885+0000 7f8b1ad9d700 1 --2- 172.21.15.60:0/4213026619 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b1c0b0fa0 0x7f8b1c14db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:51.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.885+0000 7f8b1ad9d700 1 -- 172.21.15.60:0/4213026619 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b0c00a360 con 0x7f8b1c05fb00 2022-06-29T16:07:51.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.885+0000 7f8b1ad9d700 1 --2- 172.21.15.60:0/4213026619 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b1c05fb00 0x7f8b1c064f10 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 rx=0x7f8b0c002f70 tx=0x7f8b1c143310).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:51.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.885+0000 7f8b0bfff700 1 -- 172.21.15.60:0/4213026619 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b0c010e20 con 0x7f8b1c05fb00 2022-06-29T16:07:51.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.885+0000 7f8b0bfff700 1 -- 172.21.15.60:0/4213026619 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b0c01a440 con 0x7f8b1c05fb00 2022-06-29T16:07:51.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.885+0000 7f8b209d4700 1 -- 172.21.15.60:0/4213026619 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b1c065400 con 0x7f8b1c05fb00 2022-06-29T16:07:51.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.885+0000 7f8b0bfff700 1 -- 172.21.15.60:0/4213026619 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b0c01a7a0 con 0x7f8b1c05fb00 2022-06-29T16:07:51.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.885+0000 7f8b209d4700 1 -- 172.21.15.60:0/4213026619 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b1c10f070 con 0x7f8b1c05fb00 2022-06-29T16:07:51.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.885+0000 7f8b209d4700 1 -- 172.21.15.60:0/4213026619 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b1c066e00 con 0x7f8b1c05fb00 2022-06-29T16:07:51.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.885+0000 7f8b0bfff700 1 -- 172.21.15.60:0/4213026619 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8b0c02a070 con 0x7f8b1c05fb00 2022-06-29T16:07:51.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.889+0000 7f8b0bfff700 1 --2- 172.21.15.60:0/4213026619 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8af8053980 0x7f8af8055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:51.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.889+0000 7f8b0bfff700 1 -- 172.21.15.60:0/4213026619 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f8b0c06d4a0 con 0x7f8b1c05fb00 2022-06-29T16:07:51.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.889+0000 7f8b1a59c700 1 --2- 172.21.15.60:0/4213026619 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8af8053980 0x7f8af8055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:51.899 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.889+0000 7f8b1a59c700 1 --2- 172.21.15.60:0/4213026619 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8af8053980 0x7f8af8055e40 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0x7f8b1000b000 tx=0x7f8b10007160).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:51.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:51.889+0000 7f8b0bfff700 1 -- 172.21.15.60:0/4213026619 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8b0c0446f0 con 0x7f8b1c05fb00 2022-06-29T16:07:52.023 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.017+0000 7f8b209d4700 1 -- 172.21.15.60:0/4213026619 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f8b1c0573d0 con 0x7f8b1c05fb00 2022-06-29T16:07:52.024 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.017+0000 7f8b0bfff700 1 -- 172.21.15.60:0/4213026619 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v133) v1 ==== 92+0+10 (secure 0 0 0) 0x7f8b0c06be60 con 0x7f8b1c05fb00 2022-06-29T16:07:52.024 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:07:52.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.021+0000 7f8b209d4700 1 -- 172.21.15.60:0/4213026619 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8af8053980 msgr2=0x7f8af8055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:52.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.021+0000 7f8b209d4700 1 --2- 172.21.15.60:0/4213026619 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8af8053980 0x7f8af8055e40 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0x7f8b1000b000 tx=0x7f8b10007160).stop 2022-06-29T16:07:52.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.021+0000 7f8b209d4700 1 -- 172.21.15.60:0/4213026619 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b1c05fb00 msgr2=0x7f8b1c064f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:52.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.021+0000 7f8b209d4700 1 --2- 172.21.15.60:0/4213026619 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b1c05fb00 0x7f8b1c064f10 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 rx=0x7f8b0c002f70 tx=0x7f8b1c143310).stop 2022-06-29T16:07:52.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.021+0000 7f8b209d4700 1 -- 172.21.15.60:0/4213026619 shutdown_connections 2022-06-29T16:07:52.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.021+0000 7f8b209d4700 1 --2- 172.21.15.60:0/4213026619 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8af8053980 0x7f8af8055e40 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:52.027 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.021+0000 7f8b209d4700 1 --2- 172.21.15.60:0/4213026619 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8b1c0b1c00 0x7f8b1c14e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:52.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.021+0000 7f8b209d4700 1 --2- 172.21.15.60:0/4213026619 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b1c0b0fa0 0x7f8b1c14db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:52.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.021+0000 7f8b209d4700 1 --2- 172.21.15.60:0/4213026619 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b1c05fb00 0x7f8b1c064f10 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:52.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.021+0000 7f8b209d4700 1 -- 172.21.15.60:0/4213026619 >> 172.21.15.60:0/4213026619 conn(0x7f8b1c063a60 msgr2=0x7f8b1c142780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:52.028 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.021+0000 7f8b209d4700 1 -- 172.21.15.60:0/4213026619 shutdown_connections 2022-06-29T16:07:52.029 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.021+0000 7f8b209d4700 1 -- 172.21.15.60:0/4213026619 wait complete. 2022-06-29T16:07:52.037 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-06-29T16:07:52.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.133+0000 7f795f59e700 1 Processor -- start 2022-06-29T16:07:52.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.133+0000 7f795f59e700 1 -- start start 2022-06-29T16:07:52.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.133+0000 7f795f59e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f796005d070 0x7f796014aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:52.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.133+0000 7f795f59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f79600b2130 0x7f79600aca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:52.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.133+0000 7f795f59e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f796005d450 con 0x7f796014af80 2022-06-29T16:07:52.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.137+0000 7f795f59e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79600adf90 con 0x7f796005d070 2022-06-29T16:07:52.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.137+0000 7f795f59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79600b25d0 con 0x7f79600b2130 2022-06-29T16:07:52.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.137+0000 7f795ed9d700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f796014af80 0x7f79600acf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33458/0 (socket says 172.21.15.60:33458) 2022-06-29T16:07:52.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.137+0000 7f795ed9d700 1 -- 172.21.15.60:0/3120632110 learned_addr learned my addr 172.21.15.60:0/3120632110 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:52.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.137+0000 7f795e59c700 1 --2- 172.21.15.60:0/3120632110 >> v2:172.21.15.92:3300/0 conn(0x7f796005d070 0x7f796014aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:52.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.137+0000 7f795dd9b700 1 --2- 172.21.15.60:0/3120632110 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f79600b2130 0x7f79600aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:52.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.137+0000 7f795d59a700 1 -- 172.21.15.60:0/3120632110 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2703840722 0 0) 0x7f796005d450 con 0x7f796014af80 2022-06-29T16:07:52.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.137+0000 7f795d59a700 1 -- 172.21.15.60:0/3120632110 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7944003260 con 0x7f796014af80 2022-06-29T16:07:52.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.137+0000 7f795e59c700 1 -- 172.21.15.60:0/3120632110 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f79600b2130 msgr2=0x7f79600aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:52.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.137+0000 7f795e59c700 1 --2- 172.21.15.60:0/3120632110 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f79600b2130 0x7f79600aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:52.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.137+0000 7f795e59c700 1 -- 172.21.15.60:0/3120632110 >> v1:172.21.15.60:6789/0 conn(0x7f796014af80 legacy=0x7f79600acf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:52.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.137+0000 7f795e59c700 1 -- 172.21.15.60:0/3120632110 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f796014a350 con 0x7f796005d070 2022-06-29T16:07:52.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.137+0000 7f795e59c700 1 --2- 172.21.15.60:0/3120632110 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f796005d070 0x7f796014aa90 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7f79580032f0 tx=0x7f79580091d0).ready entity=mon.1 client_cookie=715dc69a41be34b1 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:52.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.137+0000 7f795d59a700 1 -- 172.21.15.60:0/3120632110 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f795800b600 con 0x7f796005d070 2022-06-29T16:07:52.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.137+0000 7f795d59a700 1 -- 172.21.15.60:0/3120632110 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f795800b760 con 0x7f796005d070 2022-06-29T16:07:52.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.137+0000 7f795d59a700 1 -- 172.21.15.60:0/3120632110 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f795800b8d0 con 0x7f796005d070 2022-06-29T16:07:52.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.137+0000 7f795f59e700 1 -- 172.21.15.60:0/3120632110 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f796005d070 msgr2=0x7f796014aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:52.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.137+0000 7f795f59e700 1 --2- 172.21.15.60:0/3120632110 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f796005d070 0x7f796014aa90 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7f79580032f0 tx=0x7f79580091d0).stop 2022-06-29T16:07:52.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.137+0000 7f795f59e700 1 -- 172.21.15.60:0/3120632110 shutdown_connections 2022-06-29T16:07:52.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.137+0000 7f795f59e700 1 --2- 172.21.15.60:0/3120632110 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f79600b2130 0x7f79600aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:52.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.137+0000 7f795f59e700 1 --2- 172.21.15.60:0/3120632110 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f796005d070 0x7f796014aa90 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:52.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.137+0000 7f795f59e700 1 -- 172.21.15.60:0/3120632110 >> 172.21.15.60:0/3120632110 conn(0x7f79600b5890 msgr2=0x7f79600b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:52.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.137+0000 7f795f59e700 1 -- 172.21.15.60:0/3120632110 shutdown_connections 2022-06-29T16:07:52.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.137+0000 7f795f59e700 1 -- 172.21.15.60:0/3120632110 wait complete. 2022-06-29T16:07:52.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.137+0000 7f795f59e700 1 Processor -- start 2022-06-29T16:07:52.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.137+0000 7f795f59e700 1 -- start start 2022-06-29T16:07:52.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.141+0000 7f795f59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f796005d070 0x7f7960063400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:52.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.141+0000 7f795f59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f79600b2130 0x7f79601481b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:52.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.141+0000 7f795f59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f796014af80 0x7f79600665c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:52.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.141+0000 7f795f59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79601493f0 con 0x7f796005d070 2022-06-29T16:07:52.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.141+0000 7f795f59e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7960064fe0 con 0x7f79600b2130 2022-06-29T16:07:52.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.141+0000 7f795f59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f796014d930 con 0x7f796014af80 2022-06-29T16:07:52.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.141+0000 7f795e59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f796005d070 0x7f7960063400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:52.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.141+0000 7f795e59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f796005d070 0x7f7960063400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53606/0 (socket says 172.21.15.60:53606) 2022-06-29T16:07:52.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.141+0000 7f795e59c700 1 -- 172.21.15.60:0/4093737519 learned_addr learned my addr 172.21.15.60:0/4093737519 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:52.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.141+0000 7f795dd9b700 1 --2- 172.21.15.60:0/4093737519 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f79600b2130 0x7f79601481b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:52.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.141+0000 7f795ed9d700 1 --2- 172.21.15.60:0/4093737519 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f796014af80 0x7f79600665c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:52.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.141+0000 7f795dd9b700 1 -- 172.21.15.60:0/4093737519 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f796014af80 msgr2=0x7f79600665c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:52.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.141+0000 7f795dd9b700 1 --2- 172.21.15.60:0/4093737519 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f796014af80 0x7f79600665c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:52.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.141+0000 7f795dd9b700 1 -- 172.21.15.60:0/4093737519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f796005d070 msgr2=0x7f7960063400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:52.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.141+0000 7f795dd9b700 1 --2- 172.21.15.60:0/4093737519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f796005d070 0x7f7960063400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:52.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.141+0000 7f795dd9b700 1 -- 172.21.15.60:0/4093737519 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7958009470 con 0x7f79600b2130 2022-06-29T16:07:52.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.141+0000 7f795dd9b700 1 --2- 172.21.15.60:0/4093737519 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f79600b2130 0x7f79601481b0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7f795400c000 tx=0x7f7954006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:52.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.141+0000 7f794f7fe700 1 -- 172.21.15.60:0/4093737519 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7954010780 con 0x7f79600b2130 2022-06-29T16:07:52.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.141+0000 7f794f7fe700 1 -- 172.21.15.60:0/4093737519 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79540108e0 con 0x7f79600b2130 2022-06-29T16:07:52.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.141+0000 7f795f59e700 1 -- 172.21.15.60:0/4093737519 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f796014dbd0 con 0x7f79600b2130 2022-06-29T16:07:52.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.141+0000 7f794f7fe700 1 -- 172.21.15.60:0/4093737519 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7954010c40 con 0x7f79600b2130 2022-06-29T16:07:52.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.141+0000 7f795f59e700 1 -- 172.21.15.60:0/4093737519 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7960149c30 con 0x7f79600b2130 2022-06-29T16:07:52.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.141+0000 7f795f59e700 1 -- 172.21.15.60:0/4093737519 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f796014a5f0 con 0x7f79600b2130 2022-06-29T16:07:52.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.145+0000 7f794f7fe700 1 -- 172.21.15.60:0/4093737519 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f795401b400 con 0x7f79600b2130 2022-06-29T16:07:52.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.145+0000 7f794f7fe700 1 --2- 172.21.15.60:0/4093737519 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f793c053a10 0x7f793c055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:52.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.145+0000 7f794f7fe700 1 -- 172.21.15.60:0/4093737519 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f795406bbb0 con 0x7f79600b2130 2022-06-29T16:07:52.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.145+0000 7f795e59c700 1 --2- 172.21.15.60:0/4093737519 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f793c053a10 0x7f793c055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:52.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.145+0000 7f795e59c700 1 --2- 172.21.15.60:0/4093737519 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f793c053a10 0x7f793c055ed0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0x7f795801b000 tx=0x7f7958016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:52.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.153+0000 7f794f7fe700 1 -- 172.21.15.60:0/4093737519 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7954072080 con 0x7f79600b2130 2022-06-29T16:07:52.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.281+0000 7f795f59e700 1 -- 172.21.15.60:0/4093737519 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f79600573d0 con 0x7f79600b2130 2022-06-29T16:07:52.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.281+0000 7f794f7fe700 1 -- 172.21.15.60:0/4093737519 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v133) v1 ==== 84+0+143 (secure 0 0 0) 0x7f7954042da0 con 0x7f79600b2130 2022-06-29T16:07:52.287 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:52.287 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2022-06-29T16:07:52.287 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:52.287 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2022-06-29T16:07:52.288 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": "device_health_metrics" 2022-06-29T16:07:52.288 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:07:52.288 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:07:52.288 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 2, 2022-06-29T16:07:52.289 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": "rbd" 2022-06-29T16:07:52.289 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:07:52.289 INFO:teuthology.orchestra.run.smithi060.stdout:] 2022-06-29T16:07:52.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.285+0000 7f795f59e700 1 -- 172.21.15.60:0/4093737519 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f793c053a10 msgr2=0x7f793c055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:52.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.285+0000 7f795f59e700 1 --2- 172.21.15.60:0/4093737519 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f793c053a10 0x7f793c055ed0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0x7f795801b000 tx=0x7f7958016000).stop 2022-06-29T16:07:52.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.285+0000 7f795f59e700 1 -- 172.21.15.60:0/4093737519 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f79600b2130 msgr2=0x7f79601481b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:52.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.285+0000 7f795f59e700 1 --2- 172.21.15.60:0/4093737519 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f79600b2130 0x7f79601481b0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7f795400c000 tx=0x7f7954006df0).stop 2022-06-29T16:07:52.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.285+0000 7f795f59e700 1 -- 172.21.15.60:0/4093737519 shutdown_connections 2022-06-29T16:07:52.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.285+0000 7f795f59e700 1 --2- 172.21.15.60:0/4093737519 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f793c053a10 0x7f793c055ed0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:52.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.285+0000 7f795f59e700 1 --2- 172.21.15.60:0/4093737519 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f796014af80 0x7f79600665c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:52.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.285+0000 7f795f59e700 1 --2- 172.21.15.60:0/4093737519 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f79600b2130 0x7f79601481b0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:52.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.285+0000 7f795f59e700 1 --2- 172.21.15.60:0/4093737519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f796005d070 0x7f7960063400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:52.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.285+0000 7f795f59e700 1 -- 172.21.15.60:0/4093737519 >> 172.21.15.60:0/4093737519 conn(0x7f79600b5890 msgr2=0x7f7960145ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:52.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.285+0000 7f795f59e700 1 -- 172.21.15.60:0/4093737519 shutdown_connections 2022-06-29T16:07:52.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.285+0000 7f795f59e700 1 -- 172.21.15.60:0/4093737519 wait complete. 2022-06-29T16:07:52.300 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-06-29T16:07:52.300 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-06-29T16:07:52.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.405+0000 7fcd155e9700 1 Processor -- start 2022-06-29T16:07:52.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.405+0000 7fcd155e9700 1 -- start start 2022-06-29T16:07:52.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.409+0000 7fcd155e9700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd10063ad0 0x7fcd10067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:52.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.409+0000 7fcd155e9700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcd1014af10 0x7fcd1005c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:52.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.409+0000 7fcd155e9700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd100acba0 con 0x7fcd100b5950 2022-06-29T16:07:52.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.409+0000 7fcd155e9700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd1014b3a0 con 0x7fcd1014af10 2022-06-29T16:07:52.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.409+0000 7fcd155e9700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd10067220 con 0x7fcd10063ad0 2022-06-29T16:07:52.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.409+0000 7fcd0f7fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcd1014af10 0x7fcd1005c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:52.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.409+0000 7fcd0f7fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcd1014af10 0x7fcd1005c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53618/0 (socket says 172.21.15.60:53618) 2022-06-29T16:07:52.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.409+0000 7fcd0f7fe700 1 -- 172.21.15.60:0/578561935 learned_addr learned my addr 172.21.15.60:0/578561935 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:52.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.409+0000 7fcd0e7fc700 1 --2- 172.21.15.60:0/578561935 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd10063ad0 0x7fcd10067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:52.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.409+0000 7fcd0f7fe700 1 -- 172.21.15.60:0/578561935 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd10063ad0 msgr2=0x7fcd10067680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:52.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.409+0000 7fcd0f7fe700 1 --2- 172.21.15.60:0/578561935 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd10063ad0 0x7fcd10067680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:52.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.409+0000 7fcd0f7fe700 1 -- 172.21.15.60:0/578561935 >> v1:172.21.15.60:6789/0 conn(0x7fcd100b5950 legacy=0x7fcd1005bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:52.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.409+0000 7fcd0dffb700 1 -- 172.21.15.60:0/578561935 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 577076436 0 0) 0x7fcd100acba0 con 0x7fcd100b5950 2022-06-29T16:07:52.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.409+0000 7fcd0f7fe700 1 -- 172.21.15.60:0/578561935 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd100b5d30 con 0x7fcd1014af10 2022-06-29T16:07:52.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.409+0000 7fcd0f7fe700 1 --2- 172.21.15.60:0/578561935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd1014af10 0x7fcd1005c430 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 rx=0x7fcd080032f0 tx=0x7fcd08009290).ready entity=mon.0 client_cookie=ee7fe26ae1d7fab5 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:52.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.409+0000 7fcd0dffb700 1 -- 172.21.15.60:0/578561935 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd0800b8a0 con 0x7fcd1014af10 2022-06-29T16:07:52.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.409+0000 7fcd0dffb700 1 -- 172.21.15.60:0/578561935 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd0800ba00 con 0x7fcd1014af10 2022-06-29T16:07:52.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.409+0000 7fcd0dffb700 1 -- 172.21.15.60:0/578561935 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd0800bb70 con 0x7fcd1014af10 2022-06-29T16:07:52.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.409+0000 7fcd155e9700 1 -- 172.21.15.60:0/578561935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd1014af10 msgr2=0x7fcd1005c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:52.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.409+0000 7fcd155e9700 1 --2- 172.21.15.60:0/578561935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd1014af10 0x7fcd1005c430 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 rx=0x7fcd080032f0 tx=0x7fcd08009290).stop 2022-06-29T16:07:52.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.409+0000 7fcd155e9700 1 -- 172.21.15.60:0/578561935 shutdown_connections 2022-06-29T16:07:52.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.409+0000 7fcd155e9700 1 --2- 172.21.15.60:0/578561935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd1014af10 0x7fcd1005c430 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:52.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.409+0000 7fcd155e9700 1 --2- 172.21.15.60:0/578561935 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd10063ad0 0x7fcd10067680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:52.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.409+0000 7fcd155e9700 1 -- 172.21.15.60:0/578561935 >> 172.21.15.60:0/578561935 conn(0x7fcd100632c0 msgr2=0x7fcd100b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:52.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.409+0000 7fcd155e9700 1 -- 172.21.15.60:0/578561935 shutdown_connections 2022-06-29T16:07:52.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.409+0000 7fcd155e9700 1 -- 172.21.15.60:0/578561935 wait complete. 2022-06-29T16:07:52.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.413+0000 7fcd155e9700 1 Processor -- start 2022-06-29T16:07:52.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.413+0000 7fcd155e9700 1 -- start start 2022-06-29T16:07:52.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.413+0000 7fcd155e9700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd10063ad0 0x7fcd10149000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:52.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.413+0000 7fcd155e9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd100b5950 0x7fcd10142bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:52.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.413+0000 7fcd155e9700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd1014af10 0x7fcd101430b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:52.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.413+0000 7fcd155e9700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd100ae360 con 0x7fcd100b5950 2022-06-29T16:07:52.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.413+0000 7fcd155e9700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd10149ca0 con 0x7fcd10063ad0 2022-06-29T16:07:52.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.413+0000 7fcd155e9700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd101561d0 con 0x7fcd1014af10 2022-06-29T16:07:52.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.413+0000 7fcd0e7fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd100b5950 0x7fcd10142bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:52.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.413+0000 7fcd0e7fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd100b5950 0x7fcd10142bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53622/0 (socket says 172.21.15.60:53622) 2022-06-29T16:07:52.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.413+0000 7fcd0e7fc700 1 -- 172.21.15.60:0/1837848571 learned_addr learned my addr 172.21.15.60:0/1837848571 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:52.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.413+0000 7fcd0f7fe700 1 --2- 172.21.15.60:0/1837848571 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd1014af10 0x7fcd101430b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:52.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.413+0000 7fcd0effd700 1 --2- 172.21.15.60:0/1837848571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd10063ad0 0x7fcd10149000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:52.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.413+0000 7fcd0e7fc700 1 -- 172.21.15.60:0/1837848571 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd1014af10 msgr2=0x7fcd101430b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:52.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.413+0000 7fcd0e7fc700 1 --2- 172.21.15.60:0/1837848571 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd1014af10 0x7fcd101430b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:52.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.413+0000 7fcd0e7fc700 1 -- 172.21.15.60:0/1837848571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd10063ad0 msgr2=0x7fcd10149000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:52.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.413+0000 7fcd0e7fc700 1 --2- 172.21.15.60:0/1837848571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd10063ad0 0x7fcd10149000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:52.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.413+0000 7fcd0e7fc700 1 -- 172.21.15.60:0/1837848571 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd08009530 con 0x7fcd100b5950 2022-06-29T16:07:52.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.413+0000 7fcd0e7fc700 1 --2- 172.21.15.60:0/1837848571 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd100b5950 0x7fcd10142bc0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 rx=0x7fccf80030b0 tx=0x7fccf8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:52.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.413+0000 7fcd07fff700 1 -- 172.21.15.60:0/1837848571 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fccf8011780 con 0x7fcd100b5950 2022-06-29T16:07:52.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.413+0000 7fcd07fff700 1 -- 172.21.15.60:0/1837848571 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccf80118e0 con 0x7fcd100b5950 2022-06-29T16:07:52.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.413+0000 7fcd155e9700 1 -- 172.21.15.60:0/1837848571 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd100aeee0 con 0x7fcd100b5950 2022-06-29T16:07:52.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.413+0000 7fcd07fff700 1 -- 172.21.15.60:0/1837848571 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fccf8011c40 con 0x7fcd100b5950 2022-06-29T16:07:52.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.413+0000 7fcd155e9700 1 -- 172.21.15.60:0/1837848571 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd100ae520 con 0x7fcd100b5950 2022-06-29T16:07:52.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.413+0000 7fcd07fff700 1 -- 172.21.15.60:0/1837848571 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fccf8012420 con 0x7fcd100b5950 2022-06-29T16:07:52.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.413+0000 7fcd155e9700 1 -- 172.21.15.60:0/1837848571 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd100573d0 con 0x7fcd100b5950 2022-06-29T16:07:52.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.417+0000 7fcd07fff700 1 --2- 172.21.15.60:0/1837848571 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcce8053a70 0x7fcce8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:52.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.417+0000 7fcd07fff700 1 -- 172.21.15.60:0/1837848571 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fccf806ba70 con 0x7fcd100b5950 2022-06-29T16:07:52.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.417+0000 7fcd0effd700 1 --2- 172.21.15.60:0/1837848571 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcce8053a70 0x7fcce8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:52.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.417+0000 7fcd0effd700 1 --2- 172.21.15.60:0/1837848571 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcce8053a70 0x7fcce8055f30 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7fcd0000b000 tx=0x7fcd00009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:52.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.421+0000 7fcd07fff700 1 -- 172.21.15.60:0/1837848571 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fccf8042c60 con 0x7fcd100b5950 2022-06-29T16:07:52.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:52.549+0000 7fcd155e9700 1 -- 172.21.15.60:0/1837848571 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7fcd100684b0 con 0x7fcd100b5950 2022-06-29T16:07:53.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.521+0000 7fcd07fff700 1 -- 172.21.15.60:0/1837848571 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v134) v1 ==== 133+0+0 (secure 0 0 0) 0x7fccf806be60 con 0x7fcd100b5950 2022-06-29T16:07:53.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.525+0000 7fcd155e9700 1 -- 172.21.15.60:0/1837848571 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcce8053a70 msgr2=0x7fcce8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:53.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.525+0000 7fcd155e9700 1 --2- 172.21.15.60:0/1837848571 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcce8053a70 0x7fcce8055f30 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7fcd0000b000 tx=0x7fcd00009040).stop 2022-06-29T16:07:53.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.525+0000 7fcd155e9700 1 -- 172.21.15.60:0/1837848571 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd100b5950 msgr2=0x7fcd10142bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:53.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.525+0000 7fcd155e9700 1 --2- 172.21.15.60:0/1837848571 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd100b5950 0x7fcd10142bc0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 rx=0x7fccf80030b0 tx=0x7fccf8008040).stop 2022-06-29T16:07:53.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.525+0000 7fcd155e9700 1 -- 172.21.15.60:0/1837848571 shutdown_connections 2022-06-29T16:07:53.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.525+0000 7fcd155e9700 1 --2- 172.21.15.60:0/1837848571 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcce8053a70 0x7fcce8055f30 secure :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 rx=0x7fcd0000b000 tx=0x7fcd00009040).stop 2022-06-29T16:07:53.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.525+0000 7fcd155e9700 1 --2- 172.21.15.60:0/1837848571 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcd1014af10 0x7fcd101430b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:53.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.525+0000 7fcd155e9700 1 --2- 172.21.15.60:0/1837848571 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcd100b5950 0x7fcd10142bc0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:53.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.525+0000 7fcd155e9700 1 --2- 172.21.15.60:0/1837848571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd10063ad0 0x7fcd10149000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:53.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.525+0000 7fcd155e9700 1 -- 172.21.15.60:0/1837848571 >> 172.21.15.60:0/1837848571 conn(0x7fcd100632c0 msgr2=0x7fcd100b4ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:53.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.525+0000 7fcd155e9700 1 -- 172.21.15.60:0/1837848571 shutdown_connections 2022-06-29T16:07:53.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.525+0000 7fcd155e9700 1 -- 172.21.15.60:0/1837848571 wait complete. 2022-06-29T16:07:53.533 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2022-06-29T16:07:53.546 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-06-29T16:07:53.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.657+0000 7fec8eede700 1 Processor -- start 2022-06-29T16:07:53.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.661+0000 7fec8eede700 1 -- start start 2022-06-29T16:07:53.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.661+0000 7fec8eede700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fec880ae7c0 0x7fec88141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:53.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.661+0000 7fec8eede700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fec880b46c0 0x7fec880b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:53.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.661+0000 7fec8eede700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec88143a40 con 0x7fec880b4010 2022-06-29T16:07:53.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.661+0000 7fec8eede700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec880aee60 con 0x7fec880ae7c0 2022-06-29T16:07:53.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.661+0000 7fec8eede700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec880af100 con 0x7fec880b46c0 2022-06-29T16:07:53.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.661+0000 7fec8cc7a700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fec880ae7c0 0x7fec88141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:53.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.661+0000 7fec87fff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fec880b4010 0x7fec88142490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33484/0 (socket says 172.21.15.60:33484) 2022-06-29T16:07:53.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.661+0000 7fec87fff700 1 -- 172.21.15.60:0/530551584 learned_addr learned my addr 172.21.15.60:0/530551584 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:53.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.661+0000 7fec8d47b700 1 --2- 172.21.15.60:0/530551584 >> v2:172.21.15.92:3300/0 conn(0x7fec880b46c0 0x7fec880b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:53.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.661+0000 7fec8cc7a700 1 -- 172.21.15.60:0/530551584 >> v2:172.21.15.92:3300/0 conn(0x7fec880b46c0 msgr2=0x7fec880b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:53.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.661+0000 7fec8cc7a700 1 --2- 172.21.15.60:0/530551584 >> v2:172.21.15.92:3300/0 conn(0x7fec880b46c0 0x7fec880b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:53.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.661+0000 7fec8cc7a700 1 -- 172.21.15.60:0/530551584 >> v1:172.21.15.60:6789/0 conn(0x7fec880b4010 legacy=0x7fec88142490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:53.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.661+0000 7fec8cc7a700 1 -- 172.21.15.60:0/530551584 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec880aeba0 con 0x7fec880ae7c0 2022-06-29T16:07:53.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.661+0000 7fec8cc7a700 1 --2- 172.21.15.60:0/530551584 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec880ae7c0 0x7fec88141fa0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 rx=0x7fec780032f0 tx=0x7fec780091d0).ready entity=mon.0 client_cookie=846bc61d6d8a6ec1 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:53.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.661+0000 7fec877fe700 1 -- 172.21.15.60:0/530551584 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec7800b680 con 0x7fec880ae7c0 2022-06-29T16:07:53.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.661+0000 7fec877fe700 1 -- 172.21.15.60:0/530551584 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec7800b7e0 con 0x7fec880ae7c0 2022-06-29T16:07:53.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.661+0000 7fec877fe700 1 -- 172.21.15.60:0/530551584 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec7800b950 con 0x7fec880ae7c0 2022-06-29T16:07:53.673 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.661+0000 7fec8eede700 1 -- 172.21.15.60:0/530551584 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec880ae7c0 msgr2=0x7fec88141fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:53.674 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.661+0000 7fec8eede700 1 --2- 172.21.15.60:0/530551584 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec880ae7c0 0x7fec88141fa0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 rx=0x7fec780032f0 tx=0x7fec780091d0).stop 2022-06-29T16:07:53.674 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.665+0000 7fec8eede700 1 -- 172.21.15.60:0/530551584 shutdown_connections 2022-06-29T16:07:53.674 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.665+0000 7fec8eede700 1 --2- 172.21.15.60:0/530551584 >> v2:172.21.15.92:3300/0 conn(0x7fec880b46c0 0x7fec880b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:53.674 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.665+0000 7fec8eede700 1 --2- 172.21.15.60:0/530551584 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec880ae7c0 0x7fec88141fa0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:53.675 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.665+0000 7fec8eede700 1 -- 172.21.15.60:0/530551584 >> 172.21.15.60:0/530551584 conn(0x7fec880b9340 msgr2=0x7fec880b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:53.675 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.665+0000 7fec8eede700 1 -- 172.21.15.60:0/530551584 shutdown_connections 2022-06-29T16:07:53.675 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.665+0000 7fec8eede700 1 -- 172.21.15.60:0/530551584 wait complete. 2022-06-29T16:07:53.675 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.665+0000 7fec8eede700 1 Processor -- start 2022-06-29T16:07:53.676 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.665+0000 7fec8eede700 1 -- start start 2022-06-29T16:07:53.676 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.665+0000 7fec8eede700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec880ae7c0 0x7fec881443d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:53.676 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.665+0000 7fec8eede700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec880b4010 0x7fec88155520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:53.676 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.665+0000 7fec8eede700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec880b46c0 0x7fec880b2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:53.677 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.665+0000 7fec8eede700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec881564b0 con 0x7fec880b4010 2022-06-29T16:07:53.677 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.665+0000 7fec8eede700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec880af780 con 0x7fec880ae7c0 2022-06-29T16:07:53.677 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.665+0000 7fec8eede700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec88155a10 con 0x7fec880b46c0 2022-06-29T16:07:53.677 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.669+0000 7fec87fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec880b4010 0x7fec88155520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:53.678 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.669+0000 7fec8cc7a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec880ae7c0 0x7fec881443d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:53.678 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.669+0000 7fec8d47b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec880b46c0 0x7fec880b2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:53.678 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.669+0000 7fec87fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec880b4010 0x7fec88155520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53634/0 (socket says 172.21.15.60:53634) 2022-06-29T16:07:53.678 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.669+0000 7fec87fff700 1 -- 172.21.15.60:0/553506819 learned_addr learned my addr 172.21.15.60:0/553506819 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:53.679 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.669+0000 7fec87fff700 1 -- 172.21.15.60:0/553506819 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec880b46c0 msgr2=0x7fec880b2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:53.679 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.669+0000 7fec87fff700 1 --2- 172.21.15.60:0/553506819 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec880b46c0 0x7fec880b2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:53.679 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.669+0000 7fec87fff700 1 -- 172.21.15.60:0/553506819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec880ae7c0 msgr2=0x7fec881443d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:53.679 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.669+0000 7fec87fff700 1 --2- 172.21.15.60:0/553506819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec880ae7c0 0x7fec881443d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:53.680 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.669+0000 7fec87fff700 1 -- 172.21.15.60:0/553506819 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec78009470 con 0x7fec880b4010 2022-06-29T16:07:53.680 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.669+0000 7fec8d47b700 1 --2- 172.21.15.60:0/553506819 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec880b46c0 0x7fec880b2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:07:53.680 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.669+0000 7fec8cc7a700 1 --2- 172.21.15.60:0/553506819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec880ae7c0 0x7fec881443d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:07:53.680 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.669+0000 7fec87fff700 1 --2- 172.21.15.60:0/553506819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec880b4010 0x7fec88155520 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 rx=0x7fec881465c0 tx=0x7fec70007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:53.682 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.669+0000 7fec85ffb700 1 -- 172.21.15.60:0/553506819 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec700070d0 con 0x7fec880b4010 2022-06-29T16:07:53.682 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.669+0000 7fec85ffb700 1 -- 172.21.15.60:0/553506819 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec70007230 con 0x7fec880b4010 2022-06-29T16:07:53.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.669+0000 7fec8eede700 1 -- 172.21.15.60:0/553506819 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec88146d60 con 0x7fec880b4010 2022-06-29T16:07:53.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.669+0000 7fec85ffb700 1 -- 172.21.15.60:0/553506819 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec70005610 con 0x7fec880b4010 2022-06-29T16:07:53.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.669+0000 7fec8eede700 1 -- 172.21.15.60:0/553506819 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec880b1600 con 0x7fec880b4010 2022-06-29T16:07:53.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.669+0000 7fec8eede700 1 -- 172.21.15.60:0/553506819 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec880b1470 con 0x7fec880b4010 2022-06-29T16:07:53.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.673+0000 7fec85ffb700 1 -- 172.21.15.60:0/553506819 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fec70010070 con 0x7fec880b4010 2022-06-29T16:07:53.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.673+0000 7fec85ffb700 1 --2- 172.21.15.60:0/553506819 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fec6c0539e0 0x7fec6c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:53.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.673+0000 7fec85ffb700 1 -- 172.21.15.60:0/553506819 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5607+0+0 (secure 0 0 0) 0x7fec7006b1b0 con 0x7fec880b4010 2022-06-29T16:07:53.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.673+0000 7fec8cc7a700 1 --2- 172.21.15.60:0/553506819 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fec6c0539e0 0x7fec6c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:53.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.673+0000 7fec8cc7a700 1 --2- 172.21.15.60:0/553506819 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fec6c0539e0 0x7fec6c055ea0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0x7fec7801b000 tx=0x7fec78016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:53.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.681+0000 7fec85ffb700 1 -- 172.21.15.60:0/553506819 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fec700422a0 con 0x7fec880b4010 2022-06-29T16:07:53.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.809+0000 7fec8eede700 1 -- 172.21.15.60:0/553506819 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7fec880b2a20 con 0x7fec880b4010 2022-06-29T16:07:53.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.869+0000 7fec85ffb700 1 -- 172.21.15.60:0/553506819 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v135) v1 ==== 125+0+0 (secure 0 0 0) 0x7fec700458c0 con 0x7fec880b4010 2022-06-29T16:07:53.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.873+0000 7fec8eede700 1 -- 172.21.15.60:0/553506819 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fec6c0539e0 msgr2=0x7fec6c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:53.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.873+0000 7fec8eede700 1 --2- 172.21.15.60:0/553506819 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fec6c0539e0 0x7fec6c055ea0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0x7fec7801b000 tx=0x7fec78016000).stop 2022-06-29T16:07:53.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.873+0000 7fec8eede700 1 -- 172.21.15.60:0/553506819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec880b4010 msgr2=0x7fec88155520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:53.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.873+0000 7fec8eede700 1 --2- 172.21.15.60:0/553506819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec880b4010 0x7fec88155520 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 rx=0x7fec881465c0 tx=0x7fec70007c00).stop 2022-06-29T16:07:53.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.873+0000 7fec8eede700 1 -- 172.21.15.60:0/553506819 shutdown_connections 2022-06-29T16:07:53.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.873+0000 7fec8eede700 1 --2- 172.21.15.60:0/553506819 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fec6c0539e0 0x7fec6c055ea0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:53.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.873+0000 7fec8eede700 1 --2- 172.21.15.60:0/553506819 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fec880b46c0 0x7fec880b2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:53.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.873+0000 7fec8eede700 1 --2- 172.21.15.60:0/553506819 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec880b4010 0x7fec88155520 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:53.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.873+0000 7fec8eede700 1 --2- 172.21.15.60:0/553506819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec880ae7c0 0x7fec881443d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:53.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.873+0000 7fec8eede700 1 -- 172.21.15.60:0/553506819 >> 172.21.15.60:0/553506819 conn(0x7fec880b9340 msgr2=0x7fec881411c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:53.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.873+0000 7fec8eede700 1 -- 172.21.15.60:0/553506819 shutdown_connections 2022-06-29T16:07:53.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.873+0000 7fec8eede700 1 -- 172.21.15.60:0/553506819 wait complete. 2022-06-29T16:07:53.882 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2022-06-29T16:07:53.892 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-06-29T16:07:53.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.985+0000 7f1ed8302700 1 Processor -- start 2022-06-29T16:07:53.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.985+0000 7f1ed8302700 1 -- start start 2022-06-29T16:07:53.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.989+0000 7f1ed8302700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ed013dff0 0x7f1ed00b2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:53.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.989+0000 7f1ed8302700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1ed00b6710 0x7f1ed01446c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:53.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.989+0000 7f1ed8302700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ed0143ff0 con 0x7f1ed0061620 2022-06-29T16:07:53.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.989+0000 7f1ed8302700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ed0061e50 con 0x7f1ed00b6710 2022-06-29T16:07:53.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.989+0000 7f1ed8302700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed0144220 con 0x7f1ed013dff0 2022-06-29T16:07:53.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.989+0000 7f1ed689f700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1ed00b6710 0x7f1ed01446c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:53.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.989+0000 7f1ed609e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ed013dff0 0x7f1ed00b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:53.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.989+0000 7f1ed689f700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1ed00b6710 0x7f1ed01446c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:59672/0 (socket says 172.21.15.60:59672) 2022-06-29T16:07:53.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.989+0000 7f1ed689f700 1 -- 172.21.15.60:0/839748268 learned_addr learned my addr 172.21.15.60:0/839748268 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:53.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.989+0000 7f1ed589d700 1 --1- 172.21.15.60:0/839748268 >> v1:172.21.15.92:6789/0 conn(0x7f1ed0061620 0x7f1ed0140a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:37744/0 (socket says 172.21.15.60:37744) 2022-06-29T16:07:53.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.989+0000 7f1ed509c700 1 -- 172.21.15.60:0/839748268 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2644231495 0 0) 0x7f1ed0143ff0 con 0x7f1ed0061620 2022-06-29T16:07:53.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.989+0000 7f1ed509c700 1 -- 172.21.15.60:0/839748268 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1eb8003260 con 0x7f1ed0061620 2022-06-29T16:07:53.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.989+0000 7f1ed609e700 1 -- 172.21.15.60:0/839748268 >> v2:172.21.15.92:3300/0 conn(0x7f1ed00b6710 msgr2=0x7f1ed01446c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:53.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.989+0000 7f1ed609e700 1 --2- 172.21.15.60:0/839748268 >> v2:172.21.15.92:3300/0 conn(0x7f1ed00b6710 0x7f1ed01446c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:53.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.989+0000 7f1ed609e700 1 -- 172.21.15.60:0/839748268 >> v1:172.21.15.92:6789/0 conn(0x7f1ed0061620 legacy=0x7f1ed0140a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:53.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.989+0000 7f1ed609e700 1 -- 172.21.15.60:0/839748268 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed013e3d0 con 0x7f1ed013dff0 2022-06-29T16:07:53.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.989+0000 7f1ed689f700 1 --2- 172.21.15.60:0/839748268 >> v2:172.21.15.92:3300/0 conn(0x7f1ed00b6710 0x7f1ed01446c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:07:53.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.989+0000 7f1ed609e700 1 --2- 172.21.15.60:0/839748268 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ed013dff0 0x7f1ed00b2110 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f1ecc002f70 tx=0x7f1ecc006df0).ready entity=mon.2 client_cookie=7ec7fcc07c275b6f server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:53.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.989+0000 7f1ed509c700 1 -- 172.21.15.60:0/839748268 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ecc00d4a0 con 0x7f1ed013dff0 2022-06-29T16:07:54.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.989+0000 7f1ed509c700 1 -- 172.21.15.60:0/839748268 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ecc000e90 con 0x7f1ed013dff0 2022-06-29T16:07:54.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.989+0000 7f1ed509c700 1 -- 172.21.15.60:0/839748268 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ecc00d7f0 con 0x7f1ed013dff0 2022-06-29T16:07:54.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.993+0000 7f1ed8302700 1 -- 172.21.15.60:0/839748268 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ed013dff0 msgr2=0x7f1ed00b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:54.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.993+0000 7f1ed8302700 1 --2- 172.21.15.60:0/839748268 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ed013dff0 0x7f1ed00b2110 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f1ecc002f70 tx=0x7f1ecc006df0).stop 2022-06-29T16:07:54.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.993+0000 7f1ed8302700 1 -- 172.21.15.60:0/839748268 shutdown_connections 2022-06-29T16:07:54.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.993+0000 7f1ed8302700 1 --2- 172.21.15.60:0/839748268 >> v2:172.21.15.92:3300/0 conn(0x7f1ed00b6710 0x7f1ed01446c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:54.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.993+0000 7f1ed8302700 1 --2- 172.21.15.60:0/839748268 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ed013dff0 0x7f1ed00b2110 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:54.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.993+0000 7f1ed8302700 1 -- 172.21.15.60:0/839748268 >> 172.21.15.60:0/839748268 conn(0x7f1ed00643e0 msgr2=0x7f1ed00b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:54.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.993+0000 7f1ed8302700 1 -- 172.21.15.60:0/839748268 shutdown_connections 2022-06-29T16:07:54.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.993+0000 7f1ed8302700 1 -- 172.21.15.60:0/839748268 wait complete. 2022-06-29T16:07:54.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.993+0000 7f1ed8302700 1 Processor -- start 2022-06-29T16:07:54.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.993+0000 7f1ed8302700 1 -- start start 2022-06-29T16:07:54.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.997+0000 7f1ed8302700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ed0061620 0x7f1ed005fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:54.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.997+0000 7f1ed8302700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ed00b6710 0x7f1ed009aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:54.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.997+0000 7f1ed8302700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ed013dff0 0x7f1ed013ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:54.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.997+0000 7f1ed8302700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed0064f40 con 0x7f1ed013dff0 2022-06-29T16:07:54.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.997+0000 7f1ed8302700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed009a200 con 0x7f1ed00b6710 2022-06-29T16:07:54.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.997+0000 7f1ed8302700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed009a4a0 con 0x7f1ed0061620 2022-06-29T16:07:54.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.997+0000 7f1ed689f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ed013dff0 0x7f1ed013ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:54.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.997+0000 7f1ed689f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ed013dff0 0x7f1ed013ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53650/0 (socket says 172.21.15.60:53650) 2022-06-29T16:07:54.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.997+0000 7f1ed689f700 1 -- 172.21.15.60:0/1873690972 learned_addr learned my addr 172.21.15.60:0/1873690972 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:54.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.997+0000 7f1ed589d700 1 --2- 172.21.15.60:0/1873690972 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ed00b6710 0x7f1ed009aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:54.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.997+0000 7f1ed609e700 1 --2- 172.21.15.60:0/1873690972 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ed0061620 0x7f1ed005fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:54.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.997+0000 7f1ed689f700 1 -- 172.21.15.60:0/1873690972 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ed0061620 msgr2=0x7f1ed005fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:54.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.997+0000 7f1ed689f700 1 --2- 172.21.15.60:0/1873690972 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ed0061620 0x7f1ed005fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:54.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.997+0000 7f1ed689f700 1 -- 172.21.15.60:0/1873690972 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ed00b6710 msgr2=0x7f1ed009aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:54.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.997+0000 7f1ed689f700 1 --2- 172.21.15.60:0/1873690972 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ed00b6710 0x7f1ed009aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:54.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.997+0000 7f1ed689f700 1 -- 172.21.15.60:0/1873690972 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec8009530 con 0x7f1ed013dff0 2022-06-29T16:07:54.010 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.997+0000 7f1ed689f700 1 --2- 172.21.15.60:0/1873690972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ed013dff0 0x7f1ed013ec00 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 rx=0x7f1ec8006ec0 tx=0x7f1ec8009290).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:54.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.997+0000 7f1ebf7fe700 1 -- 172.21.15.60:0/1873690972 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ec800c850 con 0x7f1ed013dff0 2022-06-29T16:07:54.011 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.997+0000 7f1ebf7fe700 1 -- 172.21.15.60:0/1873690972 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ec800c9b0 con 0x7f1ed013dff0 2022-06-29T16:07:54.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.997+0000 7f1ed8302700 1 -- 172.21.15.60:0/1873690972 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ecc00b180 con 0x7f1ed013dff0 2022-06-29T16:07:54.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.997+0000 7f1ebf7fe700 1 -- 172.21.15.60:0/1873690972 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ec800cd10 con 0x7f1ed013dff0 2022-06-29T16:07:54.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.997+0000 7f1ed8302700 1 -- 172.21.15.60:0/1873690972 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ed009afa0 con 0x7f1ed013dff0 2022-06-29T16:07:54.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:53.997+0000 7f1ed8302700 1 -- 172.21.15.60:0/1873690972 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ed00573d0 con 0x7f1ed013dff0 2022-06-29T16:07:54.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.001+0000 7f1ebf7fe700 1 -- 172.21.15.60:0/1873690972 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1ec8022070 con 0x7f1ed013dff0 2022-06-29T16:07:54.016 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.001+0000 7f1ebf7fe700 1 --2- 172.21.15.60:0/1873690972 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1eb0053980 0x7f1eb0055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:54.016 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.001+0000 7f1ebf7fe700 1 -- 172.21.15.60:0/1873690972 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f1ec806c550 con 0x7f1ed013dff0 2022-06-29T16:07:54.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.001+0000 7f1ed609e700 1 --2- 172.21.15.60:0/1873690972 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1eb0053980 0x7f1eb0055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:54.017 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.001+0000 7f1ed609e700 1 --2- 172.21.15.60:0/1873690972 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1eb0053980 0x7f1eb0055e40 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0x7f1ecc01a000 tx=0x7f1ecc014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:54.018 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.009+0000 7f1ebf7fe700 1 -- 172.21.15.60:0/1873690972 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1ec80434e0 con 0x7f1ed013dff0 2022-06-29T16:07:54.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.169+0000 7f1ed8302700 1 -- 172.21.15.60:0/1873690972 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f1ed00685d0 con 0x7f1ed013dff0 2022-06-29T16:07:54.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.181+0000 7f1ebf7fe700 1 -- 172.21.15.60:0/1873690972 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 27 and data pool 28 v67) v1 ==== 189+0+0 (secure 0 0 0) 0x7f1ec80160d0 con 0x7f1ed013dff0 2022-06-29T16:07:54.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.185+0000 7f1ed8302700 1 -- 172.21.15.60:0/1873690972 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1eb0053980 msgr2=0x7f1eb0055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:54.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.185+0000 7f1ed8302700 1 --2- 172.21.15.60:0/1873690972 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1eb0053980 0x7f1eb0055e40 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0x7f1ecc01a000 tx=0x7f1ecc014000).stop 2022-06-29T16:07:54.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.185+0000 7f1ed8302700 1 -- 172.21.15.60:0/1873690972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ed013dff0 msgr2=0x7f1ed013ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:54.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.185+0000 7f1ed8302700 1 --2- 172.21.15.60:0/1873690972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ed013dff0 0x7f1ed013ec00 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 rx=0x7f1ec8006ec0 tx=0x7f1ec8009290).stop 2022-06-29T16:07:54.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.189+0000 7f1ed8302700 1 -- 172.21.15.60:0/1873690972 shutdown_connections 2022-06-29T16:07:54.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.189+0000 7f1ed8302700 1 --2- 172.21.15.60:0/1873690972 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1eb0053980 0x7f1eb0055e40 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:54.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.189+0000 7f1ed8302700 1 --2- 172.21.15.60:0/1873690972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ed013dff0 0x7f1ed013ec00 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:54.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.189+0000 7f1ed8302700 1 --2- 172.21.15.60:0/1873690972 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ed00b6710 0x7f1ed009aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:54.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.189+0000 7f1ed8302700 1 --2- 172.21.15.60:0/1873690972 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f1ed0061620 0x7f1ed005fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:54.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.189+0000 7f1ed8302700 1 -- 172.21.15.60:0/1873690972 >> 172.21.15.60:0/1873690972 conn(0x7f1ed00643e0 msgr2=0x7f1ed00b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:54.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.189+0000 7f1ed8302700 1 -- 172.21.15.60:0/1873690972 shutdown_connections 2022-06-29T16:07:54.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.189+0000 7f1ed8302700 1 -- 172.21.15.60:0/1873690972 wait complete. 2022-06-29T16:07:54.199 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 27 and data pool 28 2022-06-29T16:07:54.209 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:07:54.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.297+0000 7fc1565d7700 1 Processor -- start 2022-06-29T16:07:54.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.297+0000 7fc1565d7700 1 -- start start 2022-06-29T16:07:54.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.297+0000 7fc1565d7700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc1500b0830 0x7fc15013ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:54.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.297+0000 7fc1565d7700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc15005d450 con 0x7fc15005d070 2022-06-29T16:07:54.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.297+0000 7fc1565d7700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc150063ba0 con 0x7fc1500b2130 2022-06-29T16:07:54.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.297+0000 7fc1565d7700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc150063e40 con 0x7fc1500b0830 2022-06-29T16:07:54.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.297+0000 7fc155dd6700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc1500b0830 0x7fc15013ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:54.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.297+0000 7fc1555d5700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc15005d070 0x7fc15013c5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33518/0 (socket says 172.21.15.60:33518) 2022-06-29T16:07:54.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.297+0000 7fc1555d5700 1 -- 172.21.15.60:0/1200804717 learned_addr learned my addr 172.21.15.60:0/1200804717 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:54.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.297+0000 7fc155dd6700 1 -- 172.21.15.60:0/1200804717 >> v1:172.21.15.92:6789/0 conn(0x7fc1500b2130 legacy=0x7fc15013fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:54.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.297+0000 7fc155dd6700 1 -- 172.21.15.60:0/1200804717 >> v1:172.21.15.60:6789/0 conn(0x7fc15005d070 legacy=0x7fc15013c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:54.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.297+0000 7fc13ffff700 1 -- 172.21.15.60:0/1200804717 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3393580776 0 0) 0x7fc15005d450 con 0x7fc15005d070 2022-06-29T16:07:54.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.297+0000 7fc155dd6700 1 -- 172.21.15.60:0/1200804717 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc150067650 con 0x7fc1500b0830 2022-06-29T16:07:54.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.297+0000 7fc155dd6700 1 --2- 172.21.15.60:0/1200804717 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1500b0830 0x7fc15013ccc0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 rx=0x7fc1480032f0 tx=0x7fc1480091d0).ready entity=mon.0 client_cookie=323a5e8a6343394 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:54.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.297+0000 7fc13ffff700 1 -- 172.21.15.60:0/1200804717 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc148009ea0 con 0x7fc1500b0830 2022-06-29T16:07:54.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.297+0000 7fc13ffff700 1 -- 172.21.15.60:0/1200804717 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc14800b8a0 con 0x7fc1500b0830 2022-06-29T16:07:54.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.297+0000 7fc13ffff700 1 -- 172.21.15.60:0/1200804717 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc14800ba10 con 0x7fc1500b0830 2022-06-29T16:07:54.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.301+0000 7fc1565d7700 1 -- 172.21.15.60:0/1200804717 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1500b0830 msgr2=0x7fc15013ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:54.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.301+0000 7fc1565d7700 1 --2- 172.21.15.60:0/1200804717 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1500b0830 0x7fc15013ccc0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 rx=0x7fc1480032f0 tx=0x7fc1480091d0).stop 2022-06-29T16:07:54.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.301+0000 7fc1565d7700 1 -- 172.21.15.60:0/1200804717 shutdown_connections 2022-06-29T16:07:54.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.301+0000 7fc1565d7700 1 --2- 172.21.15.60:0/1200804717 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1500b0830 0x7fc15013ccc0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:54.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.301+0000 7fc1565d7700 1 -- 172.21.15.60:0/1200804717 >> 172.21.15.60:0/1200804717 conn(0x7fc1500b5890 msgr2=0x7fc1500b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:54.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.301+0000 7fc1565d7700 1 -- 172.21.15.60:0/1200804717 shutdown_connections 2022-06-29T16:07:54.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.301+0000 7fc1565d7700 1 -- 172.21.15.60:0/1200804717 wait complete. 2022-06-29T16:07:54.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.301+0000 7fc1565d7700 1 Processor -- start 2022-06-29T16:07:54.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.301+0000 7fc1565d7700 1 -- start start 2022-06-29T16:07:54.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.305+0000 7fc1565d7700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc15005d070 0x7fc1500afa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:54.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.305+0000 7fc1565d7700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1500b0830 0x7fc150134300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:54.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.305+0000 7fc1565d7700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1500b2130 0x7fc1501391e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:54.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.305+0000 7fc1565d7700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1501457f0 con 0x7fc1500b2130 2022-06-29T16:07:54.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.305+0000 7fc1565d7700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc150143e60 con 0x7fc1500b0830 2022-06-29T16:07:54.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.305+0000 7fc1565d7700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc150145a20 con 0x7fc15005d070 2022-06-29T16:07:54.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.305+0000 7fc155dd6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1500b2130 0x7fc1501391e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:54.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.305+0000 7fc154dd4700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1500b0830 0x7fc150134300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:54.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.305+0000 7fc1555d5700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc15005d070 0x7fc1500afa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:54.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.305+0000 7fc155dd6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1500b2130 0x7fc1501391e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53672/0 (socket says 172.21.15.60:53672) 2022-06-29T16:07:54.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.305+0000 7fc1555d5700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc15005d070 0x7fc1500afa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:38530/0 (socket says 172.21.15.60:38530) 2022-06-29T16:07:54.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.305+0000 7fc155dd6700 1 -- 172.21.15.60:0/2800477768 learned_addr learned my addr 172.21.15.60:0/2800477768 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:54.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.305+0000 7fc155dd6700 1 -- 172.21.15.60:0/2800477768 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc15005d070 msgr2=0x7fc1500afa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:54.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.305+0000 7fc155dd6700 1 --2- 172.21.15.60:0/2800477768 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc15005d070 0x7fc1500afa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:54.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.305+0000 7fc155dd6700 1 -- 172.21.15.60:0/2800477768 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1500b0830 msgr2=0x7fc150134300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:54.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.305+0000 7fc155dd6700 1 --2- 172.21.15.60:0/2800477768 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1500b0830 0x7fc150134300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:54.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.305+0000 7fc155dd6700 1 -- 172.21.15.60:0/2800477768 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc148009470 con 0x7fc1500b2130 2022-06-29T16:07:54.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.305+0000 7fc155dd6700 1 --2- 172.21.15.60:0/2800477768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1500b2130 0x7fc1501391e0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 rx=0x7fc1480032f0 tx=0x7fc148016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:54.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.305+0000 7fc13e7fc700 1 -- 172.21.15.60:0/2800477768 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc148009ea0 con 0x7fc1500b2130 2022-06-29T16:07:54.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.305+0000 7fc13e7fc700 1 -- 172.21.15.60:0/2800477768 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc14801f3f0 con 0x7fc1500b2130 2022-06-29T16:07:54.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.305+0000 7fc1565d7700 1 -- 172.21.15.60:0/2800477768 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1501347f0 con 0x7fc1500b2130 2022-06-29T16:07:54.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.305+0000 7fc13e7fc700 1 -- 172.21.15.60:0/2800477768 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc148011ea0 con 0x7fc1500b2130 2022-06-29T16:07:54.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.305+0000 7fc1565d7700 1 -- 172.21.15.60:0/2800477768 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc150133920 con 0x7fc1500b2130 2022-06-29T16:07:54.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.309+0000 7fc1565d7700 1 -- 172.21.15.60:0/2800477768 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc150143fd0 con 0x7fc1500b2130 2022-06-29T16:07:54.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.309+0000 7fc13e7fc700 1 -- 172.21.15.60:0/2800477768 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc148026030 con 0x7fc1500b2130 2022-06-29T16:07:54.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.309+0000 7fc13e7fc700 1 --2- 172.21.15.60:0/2800477768 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc130053a70 0x7fc130055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:54.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.309+0000 7fc13e7fc700 1 -- 172.21.15.60:0/2800477768 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc14806cbd0 con 0x7fc1500b2130 2022-06-29T16:07:54.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.309+0000 7fc1555d5700 1 --2- 172.21.15.60:0/2800477768 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc130053a70 0x7fc130055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:54.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.309+0000 7fc1555d5700 1 --2- 172.21.15.60:0/2800477768 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc130053a70 0x7fc130055f30 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7fc14c00d000 tx=0x7fc14c0011a0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:54.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.317+0000 7fc13e7fc700 1 -- 172.21.15.60:0/2800477768 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc150143fd0 con 0x7fc1500b2130 2022-06-29T16:07:54.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.445+0000 7fc1565d7700 1 -- 172.21.15.60:0/2800477768 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc150143fd0 con 0x7fc1500b2130 2022-06-29T16:07:54.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.445+0000 7fc13e7fc700 1 -- 172.21.15.60:0/2800477768 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v136) v1 ==== 74+0+16065 (secure 0 0 0) 0x7fc150143fd0 con 0x7fc1500b2130 2022-06-29T16:07:54.453 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:54.453 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":136,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:07:54.174602+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:07:52.605597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"136","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2022-06-29T16:07:53.864756+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"136","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:07:54.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.449+0000 7fc1565d7700 1 -- 172.21.15.60:0/2800477768 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc130053a70 msgr2=0x7fc130055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:54.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.449+0000 7fc1565d7700 1 --2- 172.21.15.60:0/2800477768 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc130053a70 0x7fc130055f30 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7fc14c00d000 tx=0x7fc14c0011a0).stop 2022-06-29T16:07:54.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.449+0000 7fc1565d7700 1 -- 172.21.15.60:0/2800477768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1500b2130 msgr2=0x7fc1501391e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:54.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.449+0000 7fc1565d7700 1 --2- 172.21.15.60:0/2800477768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1500b2130 0x7fc1501391e0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 rx=0x7fc1480032f0 tx=0x7fc148016000).stop 2022-06-29T16:07:54.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.449+0000 7fc1565d7700 1 -- 172.21.15.60:0/2800477768 shutdown_connections 2022-06-29T16:07:54.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.449+0000 7fc1565d7700 1 --2- 172.21.15.60:0/2800477768 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc130053a70 0x7fc130055f30 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:54.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.449+0000 7fc1565d7700 1 --2- 172.21.15.60:0/2800477768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1500b2130 0x7fc1501391e0 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:54.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.449+0000 7fc1565d7700 1 --2- 172.21.15.60:0/2800477768 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1500b0830 0x7fc150134300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:54.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.449+0000 7fc1565d7700 1 --2- 172.21.15.60:0/2800477768 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc15005d070 0x7fc1500afa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:54.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.449+0000 7fc1565d7700 1 -- 172.21.15.60:0/2800477768 >> 172.21.15.60:0/2800477768 conn(0x7fc1500b5890 msgr2=0x7fc15013a290 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:54.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.449+0000 7fc1565d7700 1 -- 172.21.15.60:0/2800477768 shutdown_connections 2022-06-29T16:07:54.461 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.449+0000 7fc1565d7700 1 -- 172.21.15.60:0/2800477768 wait complete. 2022-06-29T16:07:54.470 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:07:54.586 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.569+0000 7fcf84962700 1 Processor -- start 2022-06-29T16:07:54.587 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.569+0000 7fcf84962700 1 -- start start 2022-06-29T16:07:54.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.569+0000 7fcf84962700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcf8013f790 0x7fcf8013bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:54.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.569+0000 7fcf84962700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcf8013fc00 0x7fcf8013c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:54.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.569+0000 7fcf84962700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf80142820 con 0x7fcf800b61d0 2022-06-29T16:07:54.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.569+0000 7fcf84962700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf800b0c50 con 0x7fcf8013fc00 2022-06-29T16:07:54.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.569+0000 7fcf84962700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf80062bb0 con 0x7fcf8013f790 2022-06-29T16:07:54.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.569+0000 7fcf7ed9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcf8013fc00 0x7fcf8013c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:54.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.569+0000 7fcf7ed9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fcf8013fc00 0x7fcf8013c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53680/0 (socket says 172.21.15.60:53680) 2022-06-29T16:07:54.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.569+0000 7fcf7ed9d700 1 -- 172.21.15.60:0/1263925274 learned_addr learned my addr 172.21.15.60:0/1263925274 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:54.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.569+0000 7fcf7dd9b700 1 --2- 172.21.15.60:0/1263925274 >> v2:172.21.15.92:3300/0 conn(0x7fcf8013f790 0x7fcf8013bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:54.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.569+0000 7fcf7ed9d700 1 -- 172.21.15.60:0/1263925274 >> v2:172.21.15.92:3300/0 conn(0x7fcf8013f790 msgr2=0x7fcf8013bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:54.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.569+0000 7fcf7ed9d700 1 --2- 172.21.15.60:0/1263925274 >> v2:172.21.15.92:3300/0 conn(0x7fcf8013f790 0x7fcf8013bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:54.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.569+0000 7fcf7ed9d700 1 -- 172.21.15.60:0/1263925274 >> v1:172.21.15.92:6789/0 conn(0x7fcf800b61d0 legacy=0x7fcf800b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:54.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.569+0000 7fcf7ed9d700 1 -- 172.21.15.60:0/1263925274 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf800b65b0 con 0x7fcf8013fc00 2022-06-29T16:07:54.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.569+0000 7fcf7ed9d700 1 --2- 172.21.15.60:0/1263925274 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf8013fc00 0x7fcf8013c130 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 rx=0x7fcf800b0190 tx=0x7fcf78009290).ready entity=mon.0 client_cookie=40db3d756fd3ebf1 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:54.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.569+0000 7fcf7d59a700 1 -- 172.21.15.60:0/1263925274 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf7800b820 con 0x7fcf8013fc00 2022-06-29T16:07:54.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.569+0000 7fcf7d59a700 1 -- 172.21.15.60:0/1263925274 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf7800b980 con 0x7fcf8013fc00 2022-06-29T16:07:54.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.569+0000 7fcf7d59a700 1 -- 172.21.15.60:0/1263925274 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf7800baf0 con 0x7fcf8013fc00 2022-06-29T16:07:54.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.569+0000 7fcf84962700 1 -- 172.21.15.60:0/1263925274 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf8013fc00 msgr2=0x7fcf8013c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:54.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.569+0000 7fcf84962700 1 --2- 172.21.15.60:0/1263925274 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf8013fc00 0x7fcf8013c130 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 rx=0x7fcf800b0190 tx=0x7fcf78009290).stop 2022-06-29T16:07:54.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.569+0000 7fcf84962700 1 -- 172.21.15.60:0/1263925274 shutdown_connections 2022-06-29T16:07:54.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.569+0000 7fcf84962700 1 --2- 172.21.15.60:0/1263925274 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf8013fc00 0x7fcf8013c130 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:54.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.569+0000 7fcf84962700 1 --2- 172.21.15.60:0/1263925274 >> v2:172.21.15.92:3300/0 conn(0x7fcf8013f790 0x7fcf8013bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:54.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.569+0000 7fcf84962700 1 -- 172.21.15.60:0/1263925274 >> 172.21.15.60:0/1263925274 conn(0x7fcf80063b50 msgr2=0x7fcf800b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:54.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.569+0000 7fcf84962700 1 -- 172.21.15.60:0/1263925274 shutdown_connections 2022-06-29T16:07:54.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.569+0000 7fcf84962700 1 -- 172.21.15.60:0/1263925274 wait complete. 2022-06-29T16:07:54.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.573+0000 7fcf84962700 1 Processor -- start 2022-06-29T16:07:54.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.573+0000 7fcf84962700 1 -- start start 2022-06-29T16:07:54.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.573+0000 7fcf84962700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf800b61d0 0x7fcf80064070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:54.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.573+0000 7fcf84962700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf8013f790 0x7fcf801414d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:54.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.573+0000 7fcf84962700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf8013fc00 0x7fcf80066880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:54.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.573+0000 7fcf84962700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf80114ca0 con 0x7fcf8013fc00 2022-06-29T16:07:54.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.573+0000 7fcf84962700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf800658b0 con 0x7fcf800b61d0 2022-06-29T16:07:54.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.573+0000 7fcf84962700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf80065b50 con 0x7fcf8013f790 2022-06-29T16:07:54.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.573+0000 7fcf7ed9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf8013fc00 0x7fcf80066880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:54.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.573+0000 7fcf7ed9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf8013fc00 0x7fcf80066880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53686/0 (socket says 172.21.15.60:53686) 2022-06-29T16:07:54.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.573+0000 7fcf7ed9d700 1 -- 172.21.15.60:0/2390555274 learned_addr learned my addr 172.21.15.60:0/2390555274 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:54.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.573+0000 7fcf7e59c700 1 --2- 172.21.15.60:0/2390555274 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf800b61d0 0x7fcf80064070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:54.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.573+0000 7fcf7dd9b700 1 --2- 172.21.15.60:0/2390555274 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf8013f790 0x7fcf801414d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:54.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.577+0000 7fcf7ed9d700 1 -- 172.21.15.60:0/2390555274 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf8013f790 msgr2=0x7fcf801414d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:54.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.577+0000 7fcf7ed9d700 1 --2- 172.21.15.60:0/2390555274 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf8013f790 0x7fcf801414d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:54.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.577+0000 7fcf7ed9d700 1 -- 172.21.15.60:0/2390555274 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf800b61d0 msgr2=0x7fcf80064070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:54.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.577+0000 7fcf7ed9d700 1 --2- 172.21.15.60:0/2390555274 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf800b61d0 0x7fcf80064070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:54.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.577+0000 7fcf7ed9d700 1 -- 172.21.15.60:0/2390555274 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf78009530 con 0x7fcf8013fc00 2022-06-29T16:07:54.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.577+0000 7fcf7dd9b700 1 --2- 172.21.15.60:0/2390555274 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf8013f790 0x7fcf801414d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:54.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.577+0000 7fcf7ed9d700 1 --2- 172.21.15.60:0/2390555274 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf8013fc00 0x7fcf80066880 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 rx=0x7fcf800b0190 tx=0x7fcf78016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:54.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.577+0000 7fcf6f7fe700 1 -- 172.21.15.60:0/2390555274 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf780032f0 con 0x7fcf8013fc00 2022-06-29T16:07:54.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.577+0000 7fcf6f7fe700 1 -- 172.21.15.60:0/2390555274 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf78021440 con 0x7fcf8013fc00 2022-06-29T16:07:54.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.577+0000 7fcf84962700 1 -- 172.21.15.60:0/2390555274 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf801419c0 con 0x7fcf8013fc00 2022-06-29T16:07:54.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.577+0000 7fcf6f7fe700 1 -- 172.21.15.60:0/2390555274 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf780217a0 con 0x7fcf8013fc00 2022-06-29T16:07:54.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.577+0000 7fcf84962700 1 -- 172.21.15.60:0/2390555274 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf80066d70 con 0x7fcf8013fc00 2022-06-29T16:07:54.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.577+0000 7fcf84962700 1 -- 172.21.15.60:0/2390555274 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf80114e70 con 0x7fcf8013fc00 2022-06-29T16:07:54.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.577+0000 7fcf6f7fe700 1 -- 172.21.15.60:0/2390555274 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcf78029070 con 0x7fcf8013fc00 2022-06-29T16:07:54.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.577+0000 7fcf6f7fe700 1 --2- 172.21.15.60:0/2390555274 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcf5c0539e0 0x7fcf5c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:54.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.577+0000 7fcf6f7fe700 1 -- 172.21.15.60:0/2390555274 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcf7806cd80 con 0x7fcf8013fc00 2022-06-29T16:07:54.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.581+0000 7fcf7e59c700 1 --2- 172.21.15.60:0/2390555274 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcf5c0539e0 0x7fcf5c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:54.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.581+0000 7fcf7e59c700 1 --2- 172.21.15.60:0/2390555274 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcf5c0539e0 0x7fcf5c055ea0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0x7fcf7400d000 tx=0x7fcf74000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:54.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.585+0000 7fcf6f7fe700 1 -- 172.21.15.60:0/2390555274 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcf80114e70 con 0x7fcf8013fc00 2022-06-29T16:07:54.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.709+0000 7fcf84962700 1 -- 172.21.15.60:0/2390555274 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf80114e70 con 0x7fcf8013fc00 2022-06-29T16:07:54.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.709+0000 7fcf6f7fe700 1 -- 172.21.15.60:0/2390555274 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v136) v1 ==== 74+0+16065 (secure 0 0 0) 0x7fcf80114e70 con 0x7fcf8013fc00 2022-06-29T16:07:54.715 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:54.715 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":136,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:07:54.174602+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:07:52.605597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"136","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2022-06-29T16:07:53.864756+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"136","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:07:54.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.713+0000 7fcf84962700 1 -- 172.21.15.60:0/2390555274 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcf5c0539e0 msgr2=0x7fcf5c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:54.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.713+0000 7fcf84962700 1 --2- 172.21.15.60:0/2390555274 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcf5c0539e0 0x7fcf5c055ea0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0x7fcf7400d000 tx=0x7fcf74000bc0).stop 2022-06-29T16:07:54.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.713+0000 7fcf84962700 1 -- 172.21.15.60:0/2390555274 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf8013fc00 msgr2=0x7fcf80066880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:54.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.713+0000 7fcf84962700 1 --2- 172.21.15.60:0/2390555274 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf8013fc00 0x7fcf80066880 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 rx=0x7fcf800b0190 tx=0x7fcf78016000).stop 2022-06-29T16:07:54.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.713+0000 7fcf84962700 1 -- 172.21.15.60:0/2390555274 shutdown_connections 2022-06-29T16:07:54.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.713+0000 7fcf84962700 1 --2- 172.21.15.60:0/2390555274 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcf5c0539e0 0x7fcf5c055ea0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:54.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.713+0000 7fcf84962700 1 --2- 172.21.15.60:0/2390555274 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf8013fc00 0x7fcf80066880 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:54.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.713+0000 7fcf84962700 1 --2- 172.21.15.60:0/2390555274 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf8013f790 0x7fcf801414d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:54.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.713+0000 7fcf84962700 1 --2- 172.21.15.60:0/2390555274 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf800b61d0 0x7fcf80064070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:54.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.713+0000 7fcf84962700 1 -- 172.21.15.60:0/2390555274 >> 172.21.15.60:0/2390555274 conn(0x7fcf80063b50 msgr2=0x7fcf8005f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:54.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.713+0000 7fcf84962700 1 -- 172.21.15.60:0/2390555274 shutdown_connections 2022-06-29T16:07:54.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.713+0000 7fcf84962700 1 -- 172.21.15.60:0/2390555274 wait complete. 2022-06-29T16:07:54.729 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-06-29T16:07:54.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.833+0000 7fe07dce2700 1 Processor -- start 2022-06-29T16:07:54.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.833+0000 7fe07dce2700 1 -- start start 2022-06-29T16:07:54.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.833+0000 7fe07dce2700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0780b2120 0x7fe0780ab880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:54.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.833+0000 7fe07dce2700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe07805cf40 0x7fe07814af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:54.843 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.833+0000 7fe07dce2700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe07814a530 con 0x7fe07805bd20 2022-06-29T16:07:54.843 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.833+0000 7fe07dce2700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0780b1720 con 0x7fe07805cf40 2022-06-29T16:07:54.843 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.833+0000 7fe07dce2700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe07814a7d0 con 0x7fe0780b2120 2022-06-29T16:07:54.843 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.833+0000 7fe07d4e1700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fe07805bd20 0x7fe07805c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:37794/0 (socket says 172.21.15.60:37794) 2022-06-29T16:07:54.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.833+0000 7fe077fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe07805cf40 0x7fe07814af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:54.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.833+0000 7fe07cce0700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0780b2120 0x7fe0780ab880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:54.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.833+0000 7fe07d4e1700 1 -- 172.21.15.60:0/2691730081 learned_addr learned my addr 172.21.15.60:0/2691730081 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:54.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.837+0000 7fe077fff700 1 -- 172.21.15.60:0/2691730081 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0780b2120 msgr2=0x7fe0780ab880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:54.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.837+0000 7fe077fff700 1 --2- 172.21.15.60:0/2691730081 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0780b2120 0x7fe0780ab880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:54.845 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.837+0000 7fe077fff700 1 -- 172.21.15.60:0/2691730081 >> v1:172.21.15.92:6789/0 conn(0x7fe07805bd20 legacy=0x7fe07805c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:54.845 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.837+0000 7fe077fff700 1 -- 172.21.15.60:0/2691730081 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe07814aa70 con 0x7fe07805cf40 2022-06-29T16:07:54.845 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.837+0000 7fe077fff700 1 --2- 172.21.15.60:0/2691730081 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe07805cf40 0x7fe07814af10 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7fe068002f70 tx=0x7fe068006df0).ready entity=mon.1 client_cookie=3ba238923fd81251 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:54.845 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.837+0000 7fe0777fe700 1 -- 172.21.15.60:0/2691730081 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe06800d570 con 0x7fe07805cf40 2022-06-29T16:07:54.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.837+0000 7fe0777fe700 1 -- 172.21.15.60:0/2691730081 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe068000e90 con 0x7fe07805cf40 2022-06-29T16:07:54.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.837+0000 7fe0777fe700 1 -- 172.21.15.60:0/2691730081 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe06800d8c0 con 0x7fe07805cf40 2022-06-29T16:07:54.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.837+0000 7fe07dce2700 1 -- 172.21.15.60:0/2691730081 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe07805cf40 msgr2=0x7fe07814af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:54.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.837+0000 7fe07dce2700 1 --2- 172.21.15.60:0/2691730081 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe07805cf40 0x7fe07814af10 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7fe068002f70 tx=0x7fe068006df0).stop 2022-06-29T16:07:54.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.837+0000 7fe07dce2700 1 -- 172.21.15.60:0/2691730081 shutdown_connections 2022-06-29T16:07:54.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.837+0000 7fe07dce2700 1 --2- 172.21.15.60:0/2691730081 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe07805cf40 0x7fe07814af10 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:54.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.837+0000 7fe07dce2700 1 --2- 172.21.15.60:0/2691730081 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe0780b2120 0x7fe0780ab880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:54.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.837+0000 7fe07dce2700 1 -- 172.21.15.60:0/2691730081 >> 172.21.15.60:0/2691730081 conn(0x7fe0780b58a0 msgr2=0x7fe0780b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:54.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.837+0000 7fe07dce2700 1 -- 172.21.15.60:0/2691730081 shutdown_connections 2022-06-29T16:07:54.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.837+0000 7fe07dce2700 1 -- 172.21.15.60:0/2691730081 wait complete. 2022-06-29T16:07:54.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.837+0000 7fe07dce2700 1 Processor -- start 2022-06-29T16:07:54.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.837+0000 7fe07dce2700 1 -- start start 2022-06-29T16:07:54.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.837+0000 7fe07dce2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe07805bd20 0x7fe078148a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:54.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.837+0000 7fe07dce2700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe07805cf40 0x7fe0781448e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:54.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.837+0000 7fe07dce2700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0780b2120 0x7fe078144dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:54.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.837+0000 7fe07dce2700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0781452c0 con 0x7fe07805bd20 2022-06-29T16:07:54.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.837+0000 7fe07dce2700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0780ae590 con 0x7fe0780b2120 2022-06-29T16:07:54.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.837+0000 7fe07dce2700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0780aea90 con 0x7fe07805cf40 2022-06-29T16:07:54.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.841+0000 7fe07cce0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe07805bd20 0x7fe078148a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:54.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.841+0000 7fe07cce0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe07805bd20 0x7fe078148a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53696/0 (socket says 172.21.15.60:53696) 2022-06-29T16:07:54.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.841+0000 7fe07cce0700 1 -- 172.21.15.60:0/3350105988 learned_addr learned my addr 172.21.15.60:0/3350105988 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:54.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.841+0000 7fe07d4e1700 1 --2- 172.21.15.60:0/3350105988 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0780b2120 0x7fe078144dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:54.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.841+0000 7fe077fff700 1 --2- 172.21.15.60:0/3350105988 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe07805cf40 0x7fe0781448e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:54.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.841+0000 7fe07cce0700 1 -- 172.21.15.60:0/3350105988 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe07805cf40 msgr2=0x7fe0781448e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:54.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.841+0000 7fe07cce0700 1 --2- 172.21.15.60:0/3350105988 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe07805cf40 0x7fe0781448e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:54.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.841+0000 7fe07cce0700 1 -- 172.21.15.60:0/3350105988 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0780b2120 msgr2=0x7fe078144dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:54.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.841+0000 7fe07cce0700 1 --2- 172.21.15.60:0/3350105988 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0780b2120 0x7fe078144dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:54.859 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.841+0000 7fe07cce0700 1 -- 172.21.15.60:0/3350105988 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe06800b180 con 0x7fe07805bd20 2022-06-29T16:07:54.859 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.841+0000 7fe07cce0700 1 --2- 172.21.15.60:0/3350105988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe07805bd20 0x7fe078148a50 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 rx=0x7fe078149830 tx=0x7fe06c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:54.860 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.841+0000 7fe075ffb700 1 -- 172.21.15.60:0/3350105988 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe06c010780 con 0x7fe07805bd20 2022-06-29T16:07:54.860 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.841+0000 7fe075ffb700 1 -- 172.21.15.60:0/3350105988 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe06c0108e0 con 0x7fe07805bd20 2022-06-29T16:07:54.861 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.841+0000 7fe07dce2700 1 -- 172.21.15.60:0/3350105988 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0780ad490 con 0x7fe07805bd20 2022-06-29T16:07:54.861 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.841+0000 7fe075ffb700 1 -- 172.21.15.60:0/3350105988 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe06c010c40 con 0x7fe07805bd20 2022-06-29T16:07:54.862 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.841+0000 7fe07dce2700 1 -- 172.21.15.60:0/3350105988 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0780ad250 con 0x7fe07805bd20 2022-06-29T16:07:54.862 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.841+0000 7fe075ffb700 1 -- 172.21.15.60:0/3350105988 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe06c01c460 con 0x7fe07805bd20 2022-06-29T16:07:54.862 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.841+0000 7fe07dce2700 1 -- 172.21.15.60:0/3350105988 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0780ad700 con 0x7fe07805bd20 2022-06-29T16:07:54.863 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.845+0000 7fe075ffb700 1 --2- 172.21.15.60:0/3350105988 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe058053950 0x7fe058055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:54.863 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.845+0000 7fe075ffb700 1 -- 172.21.15.60:0/3350105988 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe06c06c230 con 0x7fe07805bd20 2022-06-29T16:07:54.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.845+0000 7fe077fff700 1 --2- 172.21.15.60:0/3350105988 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe058053950 0x7fe058055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:54.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.845+0000 7fe077fff700 1 --2- 172.21.15.60:0/3350105988 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe058053950 0x7fe058055e10 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7fe06801a000 tx=0x7fe068014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:54.866 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:54.853+0000 7fe075ffb700 1 -- 172.21.15.60:0/3350105988 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe06c0430f0 con 0x7fe07805bd20 2022-06-29T16:07:55.021 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.017+0000 7fe07dce2700 1 -- 172.21.15.60:0/3350105988 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe0780aec50 con 0x7fe07805bd20 2022-06-29T16:07:55.194 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.189+0000 7fe075ffb700 1 -- 172.21.15.60:0/3350105988 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v68) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe06c046710 con 0x7fe07805bd20 2022-06-29T16:07:55.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.193+0000 7fe07dce2700 1 -- 172.21.15.60:0/3350105988 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe058053950 msgr2=0x7fe058055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:55.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.193+0000 7fe07dce2700 1 --2- 172.21.15.60:0/3350105988 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe058053950 0x7fe058055e10 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7fe06801a000 tx=0x7fe068014000).stop 2022-06-29T16:07:55.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.193+0000 7fe07dce2700 1 -- 172.21.15.60:0/3350105988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe07805bd20 msgr2=0x7fe078148a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:55.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.193+0000 7fe07dce2700 1 --2- 172.21.15.60:0/3350105988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe07805bd20 0x7fe078148a50 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 rx=0x7fe078149830 tx=0x7fe06c006df0).stop 2022-06-29T16:07:55.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.193+0000 7fe07dce2700 1 -- 172.21.15.60:0/3350105988 shutdown_connections 2022-06-29T16:07:55.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.193+0000 7fe07dce2700 1 --2- 172.21.15.60:0/3350105988 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe058053950 0x7fe058055e10 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:55.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.193+0000 7fe07dce2700 1 --2- 172.21.15.60:0/3350105988 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe0780b2120 0x7fe078144dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:55.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.193+0000 7fe07dce2700 1 --2- 172.21.15.60:0/3350105988 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe07805cf40 0x7fe0781448e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:55.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.193+0000 7fe07dce2700 1 --2- 172.21.15.60:0/3350105988 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe07805bd20 0x7fe078148a50 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:55.202 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.193+0000 7fe07dce2700 1 -- 172.21.15.60:0/3350105988 >> 172.21.15.60:0/3350105988 conn(0x7fe0780b58a0 msgr2=0x7fe0780b3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:55.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.193+0000 7fe07dce2700 1 -- 172.21.15.60:0/3350105988 shutdown_connections 2022-06-29T16:07:55.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.193+0000 7fe07dce2700 1 -- 172.21.15.60:0/3350105988 wait complete. 2022-06-29T16:07:55.214 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:07:55.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.317+0000 7f2da13af700 1 Processor -- start 2022-06-29T16:07:55.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.317+0000 7f2da13af700 1 -- start start 2022-06-29T16:07:55.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.321+0000 7f2da13af700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2d9c0b5eb0 0x7f2d9c142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:55.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.321+0000 7f2da13af700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2d9c0b48b0 0x7f2d9c0b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:55.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.321+0000 7f2da13af700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d9c0b2c50 0x7f2d9c0b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:55.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.321+0000 7f2da13af700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d9c0b54e0 con 0x7f2d9c0b48b0 2022-06-29T16:07:55.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.321+0000 7f2da13af700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d9c0b4d10 con 0x7f2d9c0b5eb0 2022-06-29T16:07:55.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.321+0000 7f2da13af700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9c0b6290 con 0x7f2d9c0b2c50 2022-06-29T16:07:55.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.321+0000 7f2d9b7fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2d9c0b48b0 0x7f2d9c0b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:55.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.321+0000 7f2d9b7fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2d9c0b48b0 0x7f2d9c0b05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53704/0 (socket says 172.21.15.60:53704) 2022-06-29T16:07:55.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.321+0000 7f2d9b7fe700 1 -- 172.21.15.60:0/237582690 learned_addr learned my addr 172.21.15.60:0/237582690 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:55.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.321+0000 7f2d9bfff700 1 --2- 172.21.15.60:0/237582690 >> v2:172.21.15.92:3300/0 conn(0x7f2d9c0b5eb0 0x7f2d9c142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:55.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.321+0000 7f2da0bae700 1 --2- 172.21.15.60:0/237582690 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d9c0b2c50 0x7f2d9c0b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:55.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.321+0000 7f2d9b7fe700 1 -- 172.21.15.60:0/237582690 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d9c0b2c50 msgr2=0x7f2d9c0b30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:55.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.321+0000 7f2d9b7fe700 1 --2- 172.21.15.60:0/237582690 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d9c0b2c50 0x7f2d9c0b30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:55.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.321+0000 7f2d9b7fe700 1 -- 172.21.15.60:0/237582690 >> v2:172.21.15.92:3300/0 conn(0x7f2d9c0b5eb0 msgr2=0x7f2d9c142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:55.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.321+0000 7f2d9b7fe700 1 --2- 172.21.15.60:0/237582690 >> v2:172.21.15.92:3300/0 conn(0x7f2d9c0b5eb0 0x7f2d9c142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:55.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.321+0000 7f2d9b7fe700 1 -- 172.21.15.60:0/237582690 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d9c142f10 con 0x7f2d9c0b48b0 2022-06-29T16:07:55.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.321+0000 7f2d9bfff700 1 --2- 172.21.15.60:0/237582690 >> v2:172.21.15.92:3300/0 conn(0x7f2d9c0b5eb0 0x7f2d9c142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:07:55.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.321+0000 7f2d9b7fe700 1 --2- 172.21.15.60:0/237582690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d9c0b48b0 0x7f2d9c0b05e0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 rx=0x7f2d8c002f70 tx=0x7f2d8c006eb0).ready entity=mon.0 client_cookie=aceed0ed31f46a15 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:55.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.321+0000 7f2d9affd700 1 -- 172.21.15.60:0/237582690 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d8c00d570 con 0x7f2d9c0b48b0 2022-06-29T16:07:55.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.321+0000 7f2d9affd700 1 -- 172.21.15.60:0/237582690 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d8c000e90 con 0x7f2d9c0b48b0 2022-06-29T16:07:55.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.321+0000 7f2d9affd700 1 -- 172.21.15.60:0/237582690 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d8c00d900 con 0x7f2d9c0b48b0 2022-06-29T16:07:55.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.321+0000 7f2da13af700 1 -- 172.21.15.60:0/237582690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d9c0b48b0 msgr2=0x7f2d9c0b05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:55.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.321+0000 7f2da13af700 1 --2- 172.21.15.60:0/237582690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d9c0b48b0 0x7f2d9c0b05e0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 rx=0x7f2d8c002f70 tx=0x7f2d8c006eb0).stop 2022-06-29T16:07:55.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.321+0000 7f2da13af700 1 -- 172.21.15.60:0/237582690 shutdown_connections 2022-06-29T16:07:55.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.321+0000 7f2da13af700 1 --2- 172.21.15.60:0/237582690 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d9c0b2c50 0x7f2d9c0b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:55.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.321+0000 7f2da13af700 1 --2- 172.21.15.60:0/237582690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d9c0b48b0 0x7f2d9c0b05e0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:55.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.321+0000 7f2da13af700 1 --2- 172.21.15.60:0/237582690 >> v2:172.21.15.92:3300/0 conn(0x7f2d9c0b5eb0 0x7f2d9c142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:55.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.321+0000 7f2da13af700 1 -- 172.21.15.60:0/237582690 >> 172.21.15.60:0/237582690 conn(0x7f2d9c062090 msgr2=0x7f2d9c062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:55.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.321+0000 7f2da13af700 1 -- 172.21.15.60:0/237582690 shutdown_connections 2022-06-29T16:07:55.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.321+0000 7f2da13af700 1 -- 172.21.15.60:0/237582690 wait complete. 2022-06-29T16:07:55.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.325+0000 7f2da13af700 1 Processor -- start 2022-06-29T16:07:55.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.325+0000 7f2da13af700 1 -- start start 2022-06-29T16:07:55.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.325+0000 7f2da13af700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d9c0b2c50 0x7f2d9c0b1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:55.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.325+0000 7f2da13af700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d9c0b48b0 0x7f2d9c0b7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:55.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.325+0000 7f2da13af700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d9c0b5eb0 0x7f2d9c151c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:55.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.325+0000 7f2da13af700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9c155280 con 0x7f2d9c0b5eb0 2022-06-29T16:07:55.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.325+0000 7f2da13af700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9c1554b0 con 0x7f2d9c0b48b0 2022-06-29T16:07:55.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.325+0000 7f2da13af700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9c0b8460 con 0x7f2d9c0b2c50 2022-06-29T16:07:55.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.325+0000 7f2da0bae700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d9c0b5eb0 0x7f2d9c151c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:55.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.325+0000 7f2da0bae700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d9c0b5eb0 0x7f2d9c151c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53710/0 (socket says 172.21.15.60:53710) 2022-06-29T16:07:55.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.325+0000 7f2da0bae700 1 -- 172.21.15.60:0/2726890591 learned_addr learned my addr 172.21.15.60:0/2726890591 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:55.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.325+0000 7f2d9bfff700 1 --2- 172.21.15.60:0/2726890591 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d9c0b2c50 0x7f2d9c0b1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:55.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.325+0000 7f2d9b7fe700 1 --2- 172.21.15.60:0/2726890591 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d9c0b48b0 0x7f2d9c0b7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:55.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.325+0000 7f2da0bae700 1 -- 172.21.15.60:0/2726890591 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d9c0b2c50 msgr2=0x7f2d9c0b1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:55.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.325+0000 7f2da0bae700 1 --2- 172.21.15.60:0/2726890591 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d9c0b2c50 0x7f2d9c0b1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:55.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.325+0000 7f2da0bae700 1 -- 172.21.15.60:0/2726890591 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d9c0b48b0 msgr2=0x7f2d9c0b7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:55.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.325+0000 7f2da0bae700 1 --2- 172.21.15.60:0/2726890591 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d9c0b48b0 0x7f2d9c0b7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:55.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.325+0000 7f2da0bae700 1 -- 172.21.15.60:0/2726890591 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d8400b180 con 0x7f2d9c0b5eb0 2022-06-29T16:07:55.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.325+0000 7f2d9bfff700 1 --2- 172.21.15.60:0/2726890591 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d9c0b2c50 0x7f2d9c0b1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:07:55.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.325+0000 7f2da0bae700 1 --2- 172.21.15.60:0/2726890591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d9c0b5eb0 0x7f2d9c151c40 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 rx=0x7f2d94006e00 tx=0x7f2d940091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:55.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.329+0000 7f2d997fa700 1 -- 172.21.15.60:0/2726890591 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d9400c640 con 0x7f2d9c0b5eb0 2022-06-29T16:07:55.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.329+0000 7f2d997fa700 1 -- 172.21.15.60:0/2726890591 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d9400c7a0 con 0x7f2d9c0b5eb0 2022-06-29T16:07:55.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.329+0000 7f2da13af700 1 -- 172.21.15.60:0/2726890591 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d8c00b1e0 con 0x7f2d9c0b5eb0 2022-06-29T16:07:55.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.329+0000 7f2d997fa700 1 -- 172.21.15.60:0/2726890591 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d9400cb00 con 0x7f2d9c0b5eb0 2022-06-29T16:07:55.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.329+0000 7f2da13af700 1 -- 172.21.15.60:0/2726890591 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d9c152130 con 0x7f2d9c0b5eb0 2022-06-29T16:07:55.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.329+0000 7f2da13af700 1 -- 172.21.15.60:0/2726890591 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d9c0b4f70 con 0x7f2d9c0b5eb0 2022-06-29T16:07:55.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.329+0000 7f2d997fa700 1 -- 172.21.15.60:0/2726890591 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2d9400cc60 con 0x7f2d9c0b5eb0 2022-06-29T16:07:55.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.329+0000 7f2d997fa700 1 --2- 172.21.15.60:0/2726890591 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2d80053a70 0x7f2d80055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:55.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.329+0000 7f2d997fa700 1 -- 172.21.15.60:0/2726890591 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f2d9406c0d0 con 0x7f2d9c0b5eb0 2022-06-29T16:07:55.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.329+0000 7f2d9bfff700 1 --2- 172.21.15.60:0/2726890591 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2d80053a70 0x7f2d80055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:55.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.333+0000 7f2d9bfff700 1 --2- 172.21.15.60:0/2726890591 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2d80053a70 0x7f2d80055f30 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0x7f2d8400b000 tx=0x7f2d84006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:55.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.337+0000 7f2d997fa700 1 -- 172.21.15.60:0/2726890591 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2d94042ff0 con 0x7f2d9c0b5eb0 2022-06-29T16:07:55.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.501+0000 7f2da13af700 1 -- 172.21.15.60:0/2726890591 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d9c155670 con 0x7f2d9c0b5eb0 2022-06-29T16:07:55.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.501+0000 7f2d997fa700 1 -- 172.21.15.60:0/2726890591 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2142 (secure 0 0 0) 0x7f2d94046610 con 0x7f2d9c0b5eb0 2022-06-29T16:07:55.508 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:55.508 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":68,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.92:6835/2628497783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2628497783},{"type":"v1","addr":"172.21.15.92:6835","nonce":2628497783}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":67}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:07:54.174936+0000","modified":"2022-06-29T16:07:55.178863+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8463":{"gid":8463,"name":"a","rank":0,"incarnation":67,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/2222627281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2222627281},{"type":"v1","addr":"172.21.15.60:6833","nonce":2222627281}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2022-06-29T16:07:55.511 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.505+0000 7f2da13af700 1 -- 172.21.15.60:0/2726890591 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2d80053a70 msgr2=0x7f2d80055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:55.511 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.505+0000 7f2da13af700 1 --2- 172.21.15.60:0/2726890591 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2d80053a70 0x7f2d80055f30 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0x7f2d8400b000 tx=0x7f2d84006df0).stop 2022-06-29T16:07:55.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.505+0000 7f2da13af700 1 -- 172.21.15.60:0/2726890591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d9c0b5eb0 msgr2=0x7f2d9c151c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:55.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.505+0000 7f2da13af700 1 --2- 172.21.15.60:0/2726890591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d9c0b5eb0 0x7f2d9c151c40 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 rx=0x7f2d94006e00 tx=0x7f2d940091d0).stop 2022-06-29T16:07:55.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.505+0000 7f2da13af700 1 -- 172.21.15.60:0/2726890591 shutdown_connections 2022-06-29T16:07:55.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.505+0000 7f2da13af700 1 --2- 172.21.15.60:0/2726890591 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2d80053a70 0x7f2d80055f30 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:55.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.505+0000 7f2da13af700 1 --2- 172.21.15.60:0/2726890591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2d9c0b5eb0 0x7f2d9c151c40 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:55.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.505+0000 7f2da13af700 1 --2- 172.21.15.60:0/2726890591 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2d9c0b48b0 0x7f2d9c0b7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:55.514 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.505+0000 7f2da13af700 1 --2- 172.21.15.60:0/2726890591 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2d9c0b2c50 0x7f2d9c0b1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:55.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.505+0000 7f2da13af700 1 -- 172.21.15.60:0/2726890591 >> 172.21.15.60:0/2726890591 conn(0x7f2d9c062090 msgr2=0x7f2d9c062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:55.515 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.505+0000 7f2da13af700 1 -- 172.21.15.60:0/2726890591 shutdown_connections 2022-06-29T16:07:55.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.505+0000 7f2da13af700 1 -- 172.21.15.60:0/2726890591 wait complete. 2022-06-29T16:07:55.516 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 68 2022-06-29T16:07:55.525 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:07:55.626 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.621+0000 7f3dcc40c700 1 Processor -- start 2022-06-29T16:07:55.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.621+0000 7f3dcc40c700 1 -- start start 2022-06-29T16:07:55.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.621+0000 7f3dcc40c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3dc40b5950 0x7f3dc4067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:55.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.621+0000 7f3dcc40c700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3dc4063ad0 0x7f3dc414af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:55.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.621+0000 7f3dcc40c700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dc40acba0 con 0x7f3dc405bd20 2022-06-29T16:07:55.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.621+0000 7f3dcc40c700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dc4067220 con 0x7f3dc4063ad0 2022-06-29T16:07:55.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.621+0000 7f3dcc40c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc40b5d30 con 0x7f3dc40b5950 2022-06-29T16:07:55.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.621+0000 7f3dca9a9700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f3dc405bd20 0x7f3dc405c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33574/0 (socket says 172.21.15.60:33574) 2022-06-29T16:07:55.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.621+0000 7f3dca9a9700 1 -- 172.21.15.60:0/2307519211 learned_addr learned my addr 172.21.15.60:0/2307519211 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:55.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.621+0000 7f3dc99a7700 1 --2- 172.21.15.60:0/2307519211 >> v2:172.21.15.92:3300/0 conn(0x7f3dc4063ad0 0x7f3dc414af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:55.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.621+0000 7f3dca1a8700 1 --2- 172.21.15.60:0/2307519211 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3dc40b5950 0x7f3dc4067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:55.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.621+0000 7f3dc91a6700 1 -- 172.21.15.60:0/2307519211 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3819948388 0 0) 0x7f3dc40acba0 con 0x7f3dc405bd20 2022-06-29T16:07:55.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.621+0000 7f3dc91a6700 1 -- 172.21.15.60:0/2307519211 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3db0003260 con 0x7f3dc405bd20 2022-06-29T16:07:55.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.625+0000 7f3dca1a8700 1 -- 172.21.15.60:0/2307519211 >> v2:172.21.15.92:3300/0 conn(0x7f3dc4063ad0 msgr2=0x7f3dc414af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:55.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.625+0000 7f3dca1a8700 1 --2- 172.21.15.60:0/2307519211 >> v2:172.21.15.92:3300/0 conn(0x7f3dc4063ad0 0x7f3dc414af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:55.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.625+0000 7f3dca1a8700 1 -- 172.21.15.60:0/2307519211 >> v1:172.21.15.60:6789/0 conn(0x7f3dc405bd20 legacy=0x7f3dc405c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:55.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.625+0000 7f3dc91a6700 1 -- 172.21.15.60:0/2307519211 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2684351914 0 0) 0x7f3db0003260 con 0x7f3dc405bd20 2022-06-29T16:07:55.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.625+0000 7f3dca1a8700 1 -- 172.21.15.60:0/2307519211 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc40ace00 con 0x7f3dc40b5950 2022-06-29T16:07:55.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.625+0000 7f3dc99a7700 1 --2- 172.21.15.60:0/2307519211 >> v2:172.21.15.92:3300/0 conn(0x7f3dc4063ad0 0x7f3dc414af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:07:55.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.625+0000 7f3dca1a8700 1 --2- 172.21.15.60:0/2307519211 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3dc40b5950 0x7f3dc4067680 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f3db4001800 tx=0x7f3dbc0091d0).ready entity=mon.2 client_cookie=466debe19b7be286 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:55.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.625+0000 7f3dc91a6700 1 -- 172.21.15.60:0/2307519211 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3db4001660 con 0x7f3dc40b5950 2022-06-29T16:07:55.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.625+0000 7f3dc91a6700 1 -- 172.21.15.60:0/2307519211 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dbc00f510 con 0x7f3dc40b5950 2022-06-29T16:07:55.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.625+0000 7f3dc91a6700 1 -- 172.21.15.60:0/2307519211 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3dbc00f680 con 0x7f3dc40b5950 2022-06-29T16:07:55.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.625+0000 7f3dcc40c700 1 -- 172.21.15.60:0/2307519211 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3dc40b5950 msgr2=0x7f3dc4067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:55.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.625+0000 7f3dcc40c700 1 --2- 172.21.15.60:0/2307519211 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3dc40b5950 0x7f3dc4067680 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f3db4001800 tx=0x7f3dbc0091d0).stop 2022-06-29T16:07:55.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.625+0000 7f3dcc40c700 1 -- 172.21.15.60:0/2307519211 shutdown_connections 2022-06-29T16:07:55.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.625+0000 7f3dcc40c700 1 --2- 172.21.15.60:0/2307519211 >> v2:172.21.15.92:3300/0 conn(0x7f3dc4063ad0 0x7f3dc414af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:55.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.625+0000 7f3dcc40c700 1 --2- 172.21.15.60:0/2307519211 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3dc40b5950 0x7f3dc4067680 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:55.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.625+0000 7f3dcc40c700 1 -- 172.21.15.60:0/2307519211 >> 172.21.15.60:0/2307519211 conn(0x7f3dc40632c0 msgr2=0x7f3dc40b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:55.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.625+0000 7f3dcc40c700 1 -- 172.21.15.60:0/2307519211 shutdown_connections 2022-06-29T16:07:55.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.625+0000 7f3dcc40c700 1 -- 172.21.15.60:0/2307519211 wait complete. 2022-06-29T16:07:55.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.625+0000 7f3dcc40c700 1 Processor -- start 2022-06-29T16:07:55.646 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.625+0000 7f3dcc40c700 1 -- start start 2022-06-29T16:07:55.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.629+0000 7f3dcc40c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3dc405bd20 0x7f3dc4148e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:55.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.629+0000 7f3dcc40c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dc4063ad0 0x7f3dc40aec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:55.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.629+0000 7f3dcc40c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3dc40b5950 0x7f3dc4156700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:55.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.629+0000 7f3dcc40c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc40b0690 con 0x7f3dc4063ad0 2022-06-29T16:07:55.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.629+0000 7f3dcc40c700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc4157160 con 0x7f3dc405bd20 2022-06-29T16:07:55.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.629+0000 7f3dcc40c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc4116900 con 0x7f3dc40b5950 2022-06-29T16:07:55.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.629+0000 7f3dc99a7700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dc4063ad0 0x7f3dc40aec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:55.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.629+0000 7f3dc99a7700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dc4063ad0 0x7f3dc40aec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53724/0 (socket says 172.21.15.60:53724) 2022-06-29T16:07:55.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.629+0000 7f3dc99a7700 1 -- 172.21.15.60:0/4278444629 learned_addr learned my addr 172.21.15.60:0/4278444629 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:55.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.629+0000 7f3dca1a8700 1 --2- 172.21.15.60:0/4278444629 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3dc405bd20 0x7f3dc4148e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:55.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.629+0000 7f3dca9a9700 1 --2- 172.21.15.60:0/4278444629 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3dc40b5950 0x7f3dc4156700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:55.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.629+0000 7f3dc99a7700 1 -- 172.21.15.60:0/4278444629 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3dc40b5950 msgr2=0x7f3dc4156700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:55.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.629+0000 7f3dc99a7700 1 --2- 172.21.15.60:0/4278444629 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3dc40b5950 0x7f3dc4156700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:55.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.629+0000 7f3dc99a7700 1 -- 172.21.15.60:0/4278444629 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3dc405bd20 msgr2=0x7f3dc4148e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:55.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.629+0000 7f3dc99a7700 1 --2- 172.21.15.60:0/4278444629 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3dc405bd20 0x7f3dc4148e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:55.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.629+0000 7f3dc99a7700 1 -- 172.21.15.60:0/4278444629 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db800b180 con 0x7f3dc4063ad0 2022-06-29T16:07:55.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.629+0000 7f3dc99a7700 1 --2- 172.21.15.60:0/4278444629 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dc4063ad0 0x7f3dc40aec60 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 rx=0x7f3db800c000 tx=0x7f3db8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:55.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.629+0000 7f3dc37fe700 1 -- 172.21.15.60:0/4278444629 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3db800c890 con 0x7f3dc4063ad0 2022-06-29T16:07:55.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.629+0000 7f3dc37fe700 1 -- 172.21.15.60:0/4278444629 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3db800c9f0 con 0x7f3dc4063ad0 2022-06-29T16:07:55.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.629+0000 7f3dcc40c700 1 -- 172.21.15.60:0/4278444629 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dbc009470 con 0x7f3dc4063ad0 2022-06-29T16:07:55.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.629+0000 7f3dc37fe700 1 -- 172.21.15.60:0/4278444629 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3db800cd50 con 0x7f3dc4063ad0 2022-06-29T16:07:55.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.629+0000 7f3dcc40c700 1 -- 172.21.15.60:0/4278444629 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dc41492f0 con 0x7f3dc4063ad0 2022-06-29T16:07:55.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.633+0000 7f3dc37fe700 1 -- 172.21.15.60:0/4278444629 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f3db801c460 con 0x7f3dc4063ad0 2022-06-29T16:07:55.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.633+0000 7f3dcc40c700 1 -- 172.21.15.60:0/4278444629 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dc414aa10 con 0x7f3dc4063ad0 2022-06-29T16:07:55.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.633+0000 7f3dc37fe700 1 --2- 172.21.15.60:0/4278444629 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3da4053a70 0x7f3da4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:55.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.633+0000 7f3dc37fe700 1 -- 172.21.15.60:0/4278444629 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3db806c5c0 con 0x7f3dc4063ad0 2022-06-29T16:07:55.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.633+0000 7f3dca1a8700 1 --2- 172.21.15.60:0/4278444629 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3da4053a70 0x7f3da4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:55.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.633+0000 7f3dca1a8700 1 --2- 172.21.15.60:0/4278444629 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3da4053a70 0x7f3da4055f30 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0x7f3dbc01b000 tx=0x7f3dbc016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:55.658 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.637+0000 7f3dc37fe700 1 -- 172.21.15.60:0/4278444629 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3db80433e0 con 0x7f3dc4063ad0 2022-06-29T16:07:55.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.765+0000 7f3dcc40c700 1 -- 172.21.15.60:0/4278444629 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3dc40573d0 con 0x7f3dc4063ad0 2022-06-29T16:07:55.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.765+0000 7f3dc37fe700 1 -- 172.21.15.60:0/4278444629 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v137) v1 ==== 74+0+16052 (secure 0 0 0) 0x7f3db8046a00 con 0x7f3dc4063ad0 2022-06-29T16:07:55.772 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:55.772 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":137,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:07:55.177815+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:07:52.605597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"136","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2022-06-29T16:07:53.864756+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:07:55.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.769+0000 7f3dcc40c700 1 -- 172.21.15.60:0/4278444629 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3da4053a70 msgr2=0x7f3da4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:55.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.769+0000 7f3dcc40c700 1 --2- 172.21.15.60:0/4278444629 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3da4053a70 0x7f3da4055f30 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0x7f3dbc01b000 tx=0x7f3dbc016000).stop 2022-06-29T16:07:55.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.769+0000 7f3dcc40c700 1 -- 172.21.15.60:0/4278444629 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dc4063ad0 msgr2=0x7f3dc40aec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:55.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.769+0000 7f3dcc40c700 1 --2- 172.21.15.60:0/4278444629 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dc4063ad0 0x7f3dc40aec60 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 rx=0x7f3db800c000 tx=0x7f3db8006df0).stop 2022-06-29T16:07:55.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.769+0000 7f3dcc40c700 1 -- 172.21.15.60:0/4278444629 shutdown_connections 2022-06-29T16:07:55.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.769+0000 7f3dcc40c700 1 --2- 172.21.15.60:0/4278444629 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f3da4053a70 0x7f3da4055f30 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:55.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.769+0000 7f3dcc40c700 1 --2- 172.21.15.60:0/4278444629 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f3dc40b5950 0x7f3dc4156700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:55.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.769+0000 7f3dcc40c700 1 --2- 172.21.15.60:0/4278444629 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dc4063ad0 0x7f3dc40aec60 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:55.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.769+0000 7f3dcc40c700 1 --2- 172.21.15.60:0/4278444629 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3dc405bd20 0x7f3dc4148e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:55.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.769+0000 7f3dcc40c700 1 -- 172.21.15.60:0/4278444629 >> 172.21.15.60:0/4278444629 conn(0x7f3dc40632c0 msgr2=0x7f3dc40b4ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:55.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.769+0000 7f3dcc40c700 1 -- 172.21.15.60:0/4278444629 shutdown_connections 2022-06-29T16:07:55.777 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.769+0000 7f3dcc40c700 1 -- 172.21.15.60:0/4278444629 wait complete. 2022-06-29T16:07:55.786 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T16:07:55.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.897+0000 7fe7eaacf700 1 Processor -- start 2022-06-29T16:07:55.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.897+0000 7fe7eaacf700 1 -- start start 2022-06-29T16:07:55.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.897+0000 7fe7eaacf700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7e413f790 0x7fe7e413bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:55.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.897+0000 7fe7eaacf700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe7e413fbd0 0x7fe7e413c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:55.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.897+0000 7fe7eaacf700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e4142820 con 0x7fe7e40b61d0 2022-06-29T16:07:55.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.897+0000 7fe7eaacf700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7e40b0c50 con 0x7fe7e413fbd0 2022-06-29T16:07:55.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.897+0000 7fe7eaacf700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e4062bb0 con 0x7fe7e413f790 2022-06-29T16:07:55.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.897+0000 7fe7e906c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe7e413fbd0 0x7fe7e413c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:55.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.897+0000 7fe7e906c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe7e413fbd0 0x7fe7e413c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53734/0 (socket says 172.21.15.60:53734) 2022-06-29T16:07:55.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.897+0000 7fe7e906c700 1 -- 172.21.15.60:0/1667130207 learned_addr learned my addr 172.21.15.60:0/1667130207 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:55.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.897+0000 7fe7dbfff700 1 --2- 172.21.15.60:0/1667130207 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7e413f790 0x7fe7e413bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:55.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.897+0000 7fe7e906c700 1 -- 172.21.15.60:0/1667130207 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7e413f790 msgr2=0x7fe7e413bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:55.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.897+0000 7fe7e906c700 1 --2- 172.21.15.60:0/1667130207 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7e413f790 0x7fe7e413bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:55.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.897+0000 7fe7e906c700 1 -- 172.21.15.60:0/1667130207 >> v1:172.21.15.92:6789/0 conn(0x7fe7e40b61d0 legacy=0x7fe7e40b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:55.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.897+0000 7fe7e906c700 1 -- 172.21.15.60:0/1667130207 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e40b65b0 con 0x7fe7e413fbd0 2022-06-29T16:07:55.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.897+0000 7fe7dbfff700 1 --2- 172.21.15.60:0/1667130207 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7e413f790 0x7fe7e413bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:55.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.897+0000 7fe7e906c700 1 --2- 172.21.15.60:0/1667130207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7e413fbd0 0x7fe7e413c130 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 rx=0x7fe7dc0032f0 tx=0x7fe7dc009290).ready entity=mon.0 client_cookie=6c74d79d241d61e0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:55.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.897+0000 7fe7db7fe700 1 -- 172.21.15.60:0/1667130207 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe7dc00ba90 con 0x7fe7e413fbd0 2022-06-29T16:07:55.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.897+0000 7fe7db7fe700 1 -- 172.21.15.60:0/1667130207 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7dc00bbf0 con 0x7fe7e413fbd0 2022-06-29T16:07:55.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.897+0000 7fe7db7fe700 1 -- 172.21.15.60:0/1667130207 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe7dc010470 con 0x7fe7e413fbd0 2022-06-29T16:07:55.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.901+0000 7fe7eaacf700 1 -- 172.21.15.60:0/1667130207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7e413fbd0 msgr2=0x7fe7e413c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:55.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.901+0000 7fe7eaacf700 1 --2- 172.21.15.60:0/1667130207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7e413fbd0 0x7fe7e413c130 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 rx=0x7fe7dc0032f0 tx=0x7fe7dc009290).stop 2022-06-29T16:07:55.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.901+0000 7fe7eaacf700 1 -- 172.21.15.60:0/1667130207 shutdown_connections 2022-06-29T16:07:55.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.901+0000 7fe7eaacf700 1 --2- 172.21.15.60:0/1667130207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7e413fbd0 0x7fe7e413c130 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:55.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.901+0000 7fe7eaacf700 1 --2- 172.21.15.60:0/1667130207 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7e413f790 0x7fe7e413bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:55.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.901+0000 7fe7eaacf700 1 -- 172.21.15.60:0/1667130207 >> 172.21.15.60:0/1667130207 conn(0x7fe7e4063b50 msgr2=0x7fe7e40b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:55.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.901+0000 7fe7eaacf700 1 -- 172.21.15.60:0/1667130207 shutdown_connections 2022-06-29T16:07:55.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.901+0000 7fe7eaacf700 1 -- 172.21.15.60:0/1667130207 wait complete. 2022-06-29T16:07:55.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.901+0000 7fe7eaacf700 1 Processor -- start 2022-06-29T16:07:55.914 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.901+0000 7fe7eaacf700 1 -- start start 2022-06-29T16:07:55.915 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.901+0000 7fe7eaacf700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7e40b61d0 0x7fe7e4064070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:55.915 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.901+0000 7fe7eaacf700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7e413f790 0x7fe7e40b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:55.915 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.901+0000 7fe7eaacf700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7e413fbd0 0x7fe7e4110e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:55.915 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.901+0000 7fe7eaacf700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e4065a10 con 0x7fe7e40b61d0 2022-06-29T16:07:55.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.901+0000 7fe7eaacf700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e4065d90 con 0x7fe7e413fbd0 2022-06-29T16:07:55.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.901+0000 7fe7eaacf700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e4066030 con 0x7fe7e413f790 2022-06-29T16:07:55.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.905+0000 7fe7e886b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7e40b61d0 0x7fe7e4064070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:55.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.905+0000 7fe7e886b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7e40b61d0 0x7fe7e4064070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53736/0 (socket says 172.21.15.60:53736) 2022-06-29T16:07:55.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.905+0000 7fe7e886b700 1 -- 172.21.15.60:0/3629455238 learned_addr learned my addr 172.21.15.60:0/3629455238 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:55.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.905+0000 7fe7dbfff700 1 --2- 172.21.15.60:0/3629455238 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7e413f790 0x7fe7e40b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:55.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.905+0000 7fe7e906c700 1 --2- 172.21.15.60:0/3629455238 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7e413fbd0 0x7fe7e4110e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:55.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.905+0000 7fe7e886b700 1 -- 172.21.15.60:0/3629455238 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7e413f790 msgr2=0x7fe7e40b4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:55.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.905+0000 7fe7e886b700 1 --2- 172.21.15.60:0/3629455238 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7e413f790 0x7fe7e40b4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:55.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.905+0000 7fe7e886b700 1 -- 172.21.15.60:0/3629455238 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7e413fbd0 msgr2=0x7fe7e4110e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:55.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.905+0000 7fe7e886b700 1 --2- 172.21.15.60:0/3629455238 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7e413fbd0 0x7fe7e4110e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:55.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.905+0000 7fe7e886b700 1 -- 172.21.15.60:0/3629455238 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7dc009530 con 0x7fe7e40b61d0 2022-06-29T16:07:55.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.905+0000 7fe7e886b700 1 --2- 172.21.15.60:0/3629455238 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7e40b61d0 0x7fe7e4064070 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 rx=0x7fe7e4062a10 tx=0x7fe7e0007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:55.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.905+0000 7fe7d9ffb700 1 -- 172.21.15.60:0/3629455238 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe7e0012400 con 0x7fe7e40b61d0 2022-06-29T16:07:55.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.905+0000 7fe7d9ffb700 1 -- 172.21.15.60:0/3629455238 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e0012560 con 0x7fe7e40b61d0 2022-06-29T16:07:55.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.905+0000 7fe7eaacf700 1 -- 172.21.15.60:0/3629455238 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7e410f350 con 0x7fe7e40b61d0 2022-06-29T16:07:55.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.905+0000 7fe7d9ffb700 1 -- 172.21.15.60:0/3629455238 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe7e00128c0 con 0x7fe7e40b61d0 2022-06-29T16:07:55.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.905+0000 7fe7eaacf700 1 -- 172.21.15.60:0/3629455238 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7e4066bb0 con 0x7fe7e40b61d0 2022-06-29T16:07:55.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.905+0000 7fe7eaacf700 1 -- 172.21.15.60:0/3629455238 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7e413f050 con 0x7fe7e40b61d0 2022-06-29T16:07:55.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.909+0000 7fe7d9ffb700 1 -- 172.21.15.60:0/3629455238 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe7e0019030 con 0x7fe7e40b61d0 2022-06-29T16:07:55.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.909+0000 7fe7d9ffb700 1 --2- 172.21.15.60:0/3629455238 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe7c4053af0 0x7fe7c4055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:55.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.909+0000 7fe7d9ffb700 1 -- 172.21.15.60:0/3629455238 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe7e006b210 con 0x7fe7e40b61d0 2022-06-29T16:07:55.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.909+0000 7fe7dbfff700 1 --2- 172.21.15.60:0/3629455238 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe7c4053af0 0x7fe7c4055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:55.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.909+0000 7fe7dbfff700 1 --2- 172.21.15.60:0/3629455238 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe7c4053af0 0x7fe7c4055fb0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0x7fe7d400c000 tx=0x7fe7d4008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:55.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:55.913+0000 7fe7d9ffb700 1 -- 172.21.15.60:0/3629455238 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe7e0042130 con 0x7fe7e40b61d0 2022-06-29T16:07:56.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.077+0000 7fe7eaacf700 1 -- 172.21.15.60:0/3629455238 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fe7e4066200 con 0x7fe7e40b61d0 2022-06-29T16:07:56.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.085+0000 7fe7d9ffb700 1 -- 172.21.15.60:0/3629455238 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v44) v1 ==== 184+0+0 (secure 0 0 0) 0x7fe7e0045750 con 0x7fe7e40b61d0 2022-06-29T16:07:56.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.089+0000 7fe7eaacf700 1 -- 172.21.15.60:0/3629455238 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe7c4053af0 msgr2=0x7fe7c4055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:56.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.089+0000 7fe7eaacf700 1 --2- 172.21.15.60:0/3629455238 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe7c4053af0 0x7fe7c4055fb0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0x7fe7d400c000 tx=0x7fe7d4008040).stop 2022-06-29T16:07:56.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.089+0000 7fe7eaacf700 1 -- 172.21.15.60:0/3629455238 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7e40b61d0 msgr2=0x7fe7e4064070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:56.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.089+0000 7fe7eaacf700 1 --2- 172.21.15.60:0/3629455238 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7e40b61d0 0x7fe7e4064070 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 rx=0x7fe7e4062a10 tx=0x7fe7e0007c50).stop 2022-06-29T16:07:56.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.089+0000 7fe7eaacf700 1 -- 172.21.15.60:0/3629455238 shutdown_connections 2022-06-29T16:07:56.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.089+0000 7fe7eaacf700 1 --2- 172.21.15.60:0/3629455238 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe7c4053af0 0x7fe7c4055fb0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:56.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.089+0000 7fe7eaacf700 1 --2- 172.21.15.60:0/3629455238 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7e413fbd0 0x7fe7e4110e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:56.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.089+0000 7fe7eaacf700 1 --2- 172.21.15.60:0/3629455238 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe7e413f790 0x7fe7e40b4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:56.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.089+0000 7fe7eaacf700 1 --2- 172.21.15.60:0/3629455238 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe7e40b61d0 0x7fe7e4064070 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:56.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.089+0000 7fe7eaacf700 1 -- 172.21.15.60:0/3629455238 >> 172.21.15.60:0/3629455238 conn(0x7fe7e4063b50 msgr2=0x7fe7e4141dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:56.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.089+0000 7fe7eaacf700 1 -- 172.21.15.60:0/3629455238 shutdown_connections 2022-06-29T16:07:56.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.089+0000 7fe7eaacf700 1 -- 172.21.15.60:0/3629455238 wait complete. 2022-06-29T16:07:56.096 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.0 2022-06-29T16:07:56.105 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T16:07:56.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.213+0000 7f64e6650700 1 Processor -- start 2022-06-29T16:07:56.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.213+0000 7f64e6650700 1 -- start start 2022-06-29T16:07:56.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.213+0000 7f64e6650700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f64e00b5eb0 0x7f64e0142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:56.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.213+0000 7f64e6650700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f64e00b0280 0x7f64e0146650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:56.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.213+0000 7f64e6650700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e00b6290 con 0x7f64e0146b40 2022-06-29T16:07:56.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.213+0000 7f64e6650700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64e00b0ab0 con 0x7f64e00b5eb0 2022-06-29T16:07:56.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.213+0000 7f64e6650700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64e01461b0 con 0x7f64e00b0280 2022-06-29T16:07:56.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.213+0000 7f64e5e4f700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f64e0146b40 0x7f64e00b2bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33600/0 (socket says 172.21.15.60:33600) 2022-06-29T16:07:56.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.213+0000 7f64e564e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f64e00b5eb0 0x7f64e0142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:56.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.213+0000 7f64e5e4f700 1 -- 172.21.15.60:0/934353807 learned_addr learned my addr 172.21.15.60:0/934353807 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:56.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.213+0000 7f64e4e4d700 1 --2- 172.21.15.60:0/934353807 >> v2:172.21.15.92:3300/0 conn(0x7f64e00b0280 0x7f64e0146650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:56.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.213+0000 7f64d7fff700 1 -- 172.21.15.60:0/934353807 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1408434647 0 0) 0x7f64e00b6290 con 0x7f64e0146b40 2022-06-29T16:07:56.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.213+0000 7f64d7fff700 1 -- 172.21.15.60:0/934353807 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64c4003260 con 0x7f64e0146b40 2022-06-29T16:07:56.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.213+0000 7f64e564e700 1 -- 172.21.15.60:0/934353807 >> v2:172.21.15.92:3300/0 conn(0x7f64e00b0280 msgr2=0x7f64e0146650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:56.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.213+0000 7f64d7fff700 1 -- 172.21.15.60:0/934353807 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2660160805 0 0) 0x7f64c4003260 con 0x7f64e0146b40 2022-06-29T16:07:56.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.213+0000 7f64e564e700 1 --2- 172.21.15.60:0/934353807 >> v2:172.21.15.92:3300/0 conn(0x7f64e00b0280 0x7f64e0146650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:56.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.213+0000 7f64e564e700 1 -- 172.21.15.60:0/934353807 >> v1:172.21.15.60:6789/0 conn(0x7f64e0146b40 legacy=0x7f64e00b2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:56.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.213+0000 7f64e564e700 1 -- 172.21.15.60:0/934353807 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e0146f70 con 0x7f64e00b5eb0 2022-06-29T16:07:56.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.213+0000 7f64e564e700 1 --2- 172.21.15.60:0/934353807 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64e00b5eb0 0x7f64e0142a20 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 rx=0x7f64d8001b80 tx=0x7f64dc006df0).ready entity=mon.0 client_cookie=e0f3363d7b129d82 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:56.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.213+0000 7f64d7fff700 1 -- 172.21.15.60:0/934353807 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64d80019e0 con 0x7f64e00b5eb0 2022-06-29T16:07:56.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.213+0000 7f64d7fff700 1 -- 172.21.15.60:0/934353807 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64dc000e90 con 0x7f64e00b5eb0 2022-06-29T16:07:56.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.213+0000 7f64d7fff700 1 -- 172.21.15.60:0/934353807 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64dc012460 con 0x7f64e00b5eb0 2022-06-29T16:07:56.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.217+0000 7f64e6650700 1 -- 172.21.15.60:0/934353807 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64e00b5eb0 msgr2=0x7f64e0142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:56.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.217+0000 7f64e6650700 1 --2- 172.21.15.60:0/934353807 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64e00b5eb0 0x7f64e0142a20 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 rx=0x7f64d8001b80 tx=0x7f64dc006df0).stop 2022-06-29T16:07:56.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.217+0000 7f64e6650700 1 -- 172.21.15.60:0/934353807 shutdown_connections 2022-06-29T16:07:56.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.217+0000 7f64e6650700 1 --2- 172.21.15.60:0/934353807 >> v2:172.21.15.92:3300/0 conn(0x7f64e00b0280 0x7f64e0146650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:56.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.217+0000 7f64e6650700 1 --2- 172.21.15.60:0/934353807 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64e00b5eb0 0x7f64e0142a20 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:56.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.217+0000 7f64e6650700 1 -- 172.21.15.60:0/934353807 >> 172.21.15.60:0/934353807 conn(0x7f64e0062090 msgr2=0x7f64e0062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:56.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.217+0000 7f64e6650700 1 -- 172.21.15.60:0/934353807 shutdown_connections 2022-06-29T16:07:56.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.217+0000 7f64e6650700 1 -- 172.21.15.60:0/934353807 wait complete. 2022-06-29T16:07:56.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.217+0000 7f64e6650700 1 Processor -- start 2022-06-29T16:07:56.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.217+0000 7f64e6650700 1 -- start start 2022-06-29T16:07:56.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.221+0000 7f64e6650700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64e00b0280 0x7f64e00ae150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:56.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.221+0000 7f64e6650700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64e00b5eb0 0x7f64e0145250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:56.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.221+0000 7f64e6650700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64e0146b40 0x7f64e01522e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:56.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.221+0000 7f64e6650700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e0155610 con 0x7f64e0146b40 2022-06-29T16:07:56.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.221+0000 7f64e6650700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e0151c30 con 0x7f64e00b0280 2022-06-29T16:07:56.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.221+0000 7f64e6650700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e0151e80 con 0x7f64e00b5eb0 2022-06-29T16:07:56.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.221+0000 7f64e5e4f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64e0146b40 0x7f64e01522e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:56.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.221+0000 7f64e564e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64e00b0280 0x7f64e00ae150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:56.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.221+0000 7f64e564e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64e00b0280 0x7f64e00ae150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:59780/0 (socket says 172.21.15.60:59780) 2022-06-29T16:07:56.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.221+0000 7f64e5e4f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64e0146b40 0x7f64e01522e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53754/0 (socket says 172.21.15.60:53754) 2022-06-29T16:07:56.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.221+0000 7f64e564e700 1 -- 172.21.15.60:0/1297862426 learned_addr learned my addr 172.21.15.60:0/1297862426 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:56.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.221+0000 7f64e4e4d700 1 --2- 172.21.15.60:0/1297862426 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64e00b5eb0 0x7f64e0145250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:56.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.221+0000 7f64e564e700 1 -- 172.21.15.60:0/1297862426 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64e00b5eb0 msgr2=0x7f64e0145250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:56.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.221+0000 7f64e564e700 1 --2- 172.21.15.60:0/1297862426 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64e00b5eb0 0x7f64e0145250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:56.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.221+0000 7f64e564e700 1 -- 172.21.15.60:0/1297862426 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64e0146b40 msgr2=0x7f64e01522e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:56.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.221+0000 7f64e564e700 1 --2- 172.21.15.60:0/1297862426 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64e0146b40 0x7f64e01522e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:56.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.221+0000 7f64e564e700 1 -- 172.21.15.60:0/1297862426 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64dc00b180 con 0x7f64e00b0280 2022-06-29T16:07:56.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.221+0000 7f64e4e4d700 1 --2- 172.21.15.60:0/1297862426 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64e00b5eb0 0x7f64e0145250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:56.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.221+0000 7f64e564e700 1 --2- 172.21.15.60:0/1297862426 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64e00b0280 0x7f64e00ae150 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0x7f64d8001b80 tx=0x7f64dc014000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:56.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.221+0000 7f64d67fc700 1 -- 172.21.15.60:0/1297862426 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64dc00d920 con 0x7f64e00b0280 2022-06-29T16:07:56.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.221+0000 7f64d67fc700 1 -- 172.21.15.60:0/1297862426 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64dc00da80 con 0x7f64e00b0280 2022-06-29T16:07:56.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.221+0000 7f64e6650700 1 -- 172.21.15.60:0/1297862426 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64e0155370 con 0x7f64e00b0280 2022-06-29T16:07:56.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.221+0000 7f64d67fc700 1 -- 172.21.15.60:0/1297862426 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64dc00dde0 con 0x7f64e00b0280 2022-06-29T16:07:56.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.221+0000 7f64e6650700 1 -- 172.21.15.60:0/1297862426 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64e0141bf0 con 0x7f64e00b0280 2022-06-29T16:07:56.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.225+0000 7f64e6650700 1 -- 172.21.15.60:0/1297862426 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64e00573d0 con 0x7f64e00b0280 2022-06-29T16:07:56.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.225+0000 7f64d67fc700 1 -- 172.21.15.60:0/1297862426 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f64dc025030 con 0x7f64e00b0280 2022-06-29T16:07:56.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.225+0000 7f64d67fc700 1 --2- 172.21.15.60:0/1297862426 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f64c8053aa0 0x7f64c8055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:56.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.225+0000 7f64d67fc700 1 -- 172.21.15.60:0/1297862426 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f64dc06cc60 con 0x7f64e00b0280 2022-06-29T16:07:56.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.225+0000 7f64e4e4d700 1 --2- 172.21.15.60:0/1297862426 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f64c8053aa0 0x7f64c8055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:56.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.225+0000 7f64e4e4d700 1 --2- 172.21.15.60:0/1297862426 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f64c8053aa0 0x7f64c8055f60 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0x7f64d000b000 tx=0x7f64d0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:56.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.233+0000 7f64d67fc700 1 -- 172.21.15.60:0/1297862426 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f64dc043ab0 con 0x7f64e00b0280 2022-06-29T16:07:56.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.401+0000 7f64e6650700 1 -- 172.21.15.60:0/1297862426 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f64e0152040 con 0x7f64e00b0280 2022-06-29T16:07:56.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.409+0000 7f64d67fc700 1 -- 172.21.15.60:0/1297862426 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v45) v1 ==== 184+0+0 (secure 0 0 0) 0x7f64dc023020 con 0x7f64e00b0280 2022-06-29T16:07:56.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.413+0000 7f64e6650700 1 -- 172.21.15.60:0/1297862426 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f64c8053aa0 msgr2=0x7f64c8055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:56.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.413+0000 7f64e6650700 1 --2- 172.21.15.60:0/1297862426 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f64c8053aa0 0x7f64c8055f60 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0x7f64d000b000 tx=0x7f64d0008040).stop 2022-06-29T16:07:56.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.413+0000 7f64e6650700 1 -- 172.21.15.60:0/1297862426 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64e00b0280 msgr2=0x7f64e00ae150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:56.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.413+0000 7f64e6650700 1 --2- 172.21.15.60:0/1297862426 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64e00b0280 0x7f64e00ae150 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0x7f64d8001b80 tx=0x7f64dc014000).stop 2022-06-29T16:07:56.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.413+0000 7f64e6650700 1 -- 172.21.15.60:0/1297862426 shutdown_connections 2022-06-29T16:07:56.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.413+0000 7f64e6650700 1 --2- 172.21.15.60:0/1297862426 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f64c8053aa0 0x7f64c8055f60 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:56.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.413+0000 7f64e6650700 1 --2- 172.21.15.60:0/1297862426 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f64e0146b40 0x7f64e01522e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:56.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.413+0000 7f64e6650700 1 --2- 172.21.15.60:0/1297862426 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f64e00b5eb0 0x7f64e0145250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:56.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.413+0000 7f64e6650700 1 --2- 172.21.15.60:0/1297862426 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f64e00b0280 0x7f64e00ae150 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:56.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.413+0000 7f64e6650700 1 -- 172.21.15.60:0/1297862426 >> 172.21.15.60:0/1297862426 conn(0x7f64e0062090 msgr2=0x7f64e00b7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:56.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.413+0000 7f64e6650700 1 -- 172.21.15.60:0/1297862426 shutdown_connections 2022-06-29T16:07:56.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.413+0000 7f64e6650700 1 -- 172.21.15.60:0/1297862426 wait complete. 2022-06-29T16:07:56.421 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.1 2022-06-29T16:07:56.431 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T16:07:56.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.545+0000 7f42d685d700 1 Processor -- start 2022-06-29T16:07:56.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.545+0000 7f42d685d700 1 -- start start 2022-06-29T16:07:56.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.549+0000 7f42d685d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42d013f790 0x7f42d013bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:56.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.549+0000 7f42d685d700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f42d013fbd0 0x7f42d013c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:56.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.549+0000 7f42d685d700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42d0142820 con 0x7f42d00b61d0 2022-06-29T16:07:56.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.549+0000 7f42d685d700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42d00b0c50 con 0x7f42d013fbd0 2022-06-29T16:07:56.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.549+0000 7f42d685d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42d0062bb0 con 0x7f42d013f790 2022-06-29T16:07:56.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.549+0000 7f42cffff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f42d00b61d0 0x7f42d00b0540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33612/0 (socket says 172.21.15.60:33612) 2022-06-29T16:07:56.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.549+0000 7f42cffff700 1 -- 172.21.15.60:0/727776544 learned_addr learned my addr 172.21.15.60:0/727776544 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:56.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.549+0000 7f42d4dfa700 1 --2- 172.21.15.60:0/727776544 >> v2:172.21.15.92:3300/0 conn(0x7f42d013fbd0 0x7f42d013c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:56.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.549+0000 7f42c7fff700 1 --2- 172.21.15.60:0/727776544 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42d013f790 0x7f42d013bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:56.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.549+0000 7f42cf7fe700 1 -- 172.21.15.60:0/727776544 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 509217118 0 0) 0x7f42d0142820 con 0x7f42d00b61d0 2022-06-29T16:07:56.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.549+0000 7f42cf7fe700 1 -- 172.21.15.60:0/727776544 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42b0003260 con 0x7f42d00b61d0 2022-06-29T16:07:56.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.549+0000 7f42c7fff700 1 -- 172.21.15.60:0/727776544 >> v2:172.21.15.92:3300/0 conn(0x7f42d013fbd0 msgr2=0x7f42d013c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:56.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.549+0000 7f42c7fff700 1 --2- 172.21.15.60:0/727776544 >> v2:172.21.15.92:3300/0 conn(0x7f42d013fbd0 0x7f42d013c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:56.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.549+0000 7f42c7fff700 1 -- 172.21.15.60:0/727776544 >> v1:172.21.15.60:6789/0 conn(0x7f42d00b61d0 legacy=0x7f42d00b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:56.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.549+0000 7f42cf7fe700 1 -- 172.21.15.60:0/727776544 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3932024936 0 0) 0x7f42b0003260 con 0x7f42d00b61d0 2022-06-29T16:07:56.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.549+0000 7f42c7fff700 1 -- 172.21.15.60:0/727776544 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d00b65b0 con 0x7f42d013f790 2022-06-29T16:07:56.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.549+0000 7f42d4dfa700 1 --2- 172.21.15.60:0/727776544 >> v2:172.21.15.92:3300/0 conn(0x7f42d013fbd0 0x7f42d013c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:07:56.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.549+0000 7f42c7fff700 1 --2- 172.21.15.60:0/727776544 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42d013f790 0x7f42d013bc40 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f42c0001800 tx=0x7f42b8006df0).ready entity=mon.2 client_cookie=5b6af816a9e0e005 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:56.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.549+0000 7f42cf7fe700 1 -- 172.21.15.60:0/727776544 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f42c0001660 con 0x7f42d013f790 2022-06-29T16:07:56.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.549+0000 7f42cf7fe700 1 -- 172.21.15.60:0/727776544 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b8000e90 con 0x7f42d013f790 2022-06-29T16:07:56.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.549+0000 7f42cf7fe700 1 -- 172.21.15.60:0/727776544 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f42b8012460 con 0x7f42d013f790 2022-06-29T16:07:56.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.549+0000 7f42d685d700 1 -- 172.21.15.60:0/727776544 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42d013f790 msgr2=0x7f42d013bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:56.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.549+0000 7f42d685d700 1 --2- 172.21.15.60:0/727776544 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42d013f790 0x7f42d013bc40 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f42c0001800 tx=0x7f42b8006df0).stop 2022-06-29T16:07:56.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.549+0000 7f42d685d700 1 -- 172.21.15.60:0/727776544 shutdown_connections 2022-06-29T16:07:56.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.549+0000 7f42d685d700 1 --2- 172.21.15.60:0/727776544 >> v2:172.21.15.92:3300/0 conn(0x7f42d013fbd0 0x7f42d013c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:56.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.549+0000 7f42d685d700 1 --2- 172.21.15.60:0/727776544 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42d013f790 0x7f42d013bc40 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:56.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.549+0000 7f42d685d700 1 -- 172.21.15.60:0/727776544 >> 172.21.15.60:0/727776544 conn(0x7f42d0063b50 msgr2=0x7f42d00b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:56.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.549+0000 7f42d685d700 1 -- 172.21.15.60:0/727776544 shutdown_connections 2022-06-29T16:07:56.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.549+0000 7f42d685d700 1 -- 172.21.15.60:0/727776544 wait complete. 2022-06-29T16:07:56.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.553+0000 7f42d685d700 1 Processor -- start 2022-06-29T16:07:56.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.553+0000 7f42d685d700 1 -- start start 2022-06-29T16:07:56.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.553+0000 7f42d685d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42d00b61d0 0x7f42d0110c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:56.565 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.553+0000 7f42d685d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42d013f790 0x7f42d0112890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:56.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.553+0000 7f42d685d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42d013fbd0 0x7f42d014f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:56.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.553+0000 7f42d685d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d0112130 con 0x7f42d013fbd0 2022-06-29T16:07:56.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.553+0000 7f42d685d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d0114fa0 con 0x7f42d00b61d0 2022-06-29T16:07:56.566 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.553+0000 7f42d685d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42d0115240 con 0x7f42d013f790 2022-06-29T16:07:56.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.553+0000 7f42d4dfa700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42d013fbd0 0x7f42d014f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:56.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.553+0000 7f42d4dfa700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42d013fbd0 0x7f42d014f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53768/0 (socket says 172.21.15.60:53768) 2022-06-29T16:07:56.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.553+0000 7f42d4dfa700 1 -- 172.21.15.60:0/1462496377 learned_addr learned my addr 172.21.15.60:0/1462496377 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:56.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.553+0000 7f42c7fff700 1 --2- 172.21.15.60:0/1462496377 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42d013f790 0x7f42d0112890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:56.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.553+0000 7f42cffff700 1 --2- 172.21.15.60:0/1462496377 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42d00b61d0 0x7f42d0110c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:56.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.553+0000 7f42d4dfa700 1 -- 172.21.15.60:0/1462496377 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42d013f790 msgr2=0x7f42d0112890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:56.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.553+0000 7f42d4dfa700 1 --2- 172.21.15.60:0/1462496377 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42d013f790 0x7f42d0112890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:56.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.553+0000 7f42d4dfa700 1 -- 172.21.15.60:0/1462496377 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42d00b61d0 msgr2=0x7f42d0110c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:56.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.553+0000 7f42d4dfa700 1 --2- 172.21.15.60:0/1462496377 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42d00b61d0 0x7f42d0110c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:56.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.553+0000 7f42d4dfa700 1 -- 172.21.15.60:0/1462496377 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c8009470 con 0x7f42d013fbd0 2022-06-29T16:07:56.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.553+0000 7f42c7fff700 1 --2- 172.21.15.60:0/1462496377 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42d013f790 0x7f42d0112890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:07:56.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.553+0000 7f42d4dfa700 1 --2- 172.21.15.60:0/1462496377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42d013fbd0 0x7f42d014f7f0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 rx=0x7f42c800c000 tx=0x7f42c80091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:56.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.557+0000 7f42cdffb700 1 -- 172.21.15.60:0/1462496377 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f42c800c850 con 0x7f42d013fbd0 2022-06-29T16:07:56.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.557+0000 7f42cdffb700 1 -- 172.21.15.60:0/1462496377 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42c800c9b0 con 0x7f42d013fbd0 2022-06-29T16:07:56.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.557+0000 7f42d685d700 1 -- 172.21.15.60:0/1462496377 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42b800b180 con 0x7f42d013fbd0 2022-06-29T16:07:56.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.557+0000 7f42cdffb700 1 -- 172.21.15.60:0/1462496377 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f42c800cd10 con 0x7f42d013fbd0 2022-06-29T16:07:56.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.557+0000 7f42d685d700 1 -- 172.21.15.60:0/1462496377 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42d01112a0 con 0x7f42d013fbd0 2022-06-29T16:07:56.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.557+0000 7f42d685d700 1 -- 172.21.15.60:0/1462496377 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42d0110990 con 0x7f42d013fbd0 2022-06-29T16:07:56.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.557+0000 7f42cdffb700 1 -- 172.21.15.60:0/1462496377 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f42c8022070 con 0x7f42d013fbd0 2022-06-29T16:07:56.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.557+0000 7f42cdffb700 1 --2- 172.21.15.60:0/1462496377 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f42b4053b00 0x7f42b4055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:56.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.557+0000 7f42cdffb700 1 -- 172.21.15.60:0/1462496377 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f42c806c7c0 con 0x7f42d013fbd0 2022-06-29T16:07:56.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.561+0000 7f42cffff700 1 --2- 172.21.15.60:0/1462496377 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f42b4053b00 0x7f42b4055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:56.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.561+0000 7f42cffff700 1 --2- 172.21.15.60:0/1462496377 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f42b4053b00 0x7f42b4055fc0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0x7f42c000b000 tx=0x7f42c0001c60).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:56.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.565+0000 7f42cdffb700 1 -- 172.21.15.60:0/1462496377 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f42d0110990 con 0x7f42d013fbd0 2022-06-29T16:07:56.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.733+0000 7f42d685d700 1 -- 172.21.15.60:0/1462496377 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f42d0110990 con 0x7f42d013fbd0 2022-06-29T16:07:56.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.741+0000 7f42cdffb700 1 -- 172.21.15.60:0/1462496377 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v46) v1 ==== 184+0+0 (secure 0 0 0) 0x7f42d0110990 con 0x7f42d013fbd0 2022-06-29T16:07:56.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.741+0000 7f42d685d700 1 -- 172.21.15.60:0/1462496377 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f42b4053b00 msgr2=0x7f42b4055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:56.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.741+0000 7f42d685d700 1 --2- 172.21.15.60:0/1462496377 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f42b4053b00 0x7f42b4055fc0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0x7f42c000b000 tx=0x7f42c0001c60).stop 2022-06-29T16:07:56.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.741+0000 7f42d685d700 1 -- 172.21.15.60:0/1462496377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42d013fbd0 msgr2=0x7f42d014f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:56.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.741+0000 7f42d685d700 1 --2- 172.21.15.60:0/1462496377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42d013fbd0 0x7f42d014f7f0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 rx=0x7f42c800c000 tx=0x7f42c80091d0).stop 2022-06-29T16:07:56.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.741+0000 7f42d685d700 1 -- 172.21.15.60:0/1462496377 shutdown_connections 2022-06-29T16:07:56.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.741+0000 7f42d685d700 1 --2- 172.21.15.60:0/1462496377 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f42b4053b00 0x7f42b4055fc0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:56.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.741+0000 7f42d685d700 1 --2- 172.21.15.60:0/1462496377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f42d013fbd0 0x7f42d014f7f0 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:56.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.741+0000 7f42d685d700 1 --2- 172.21.15.60:0/1462496377 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f42d013f790 0x7f42d0112890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:56.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.741+0000 7f42d685d700 1 --2- 172.21.15.60:0/1462496377 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42d00b61d0 0x7f42d0110c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:56.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.741+0000 7f42d685d700 1 -- 172.21.15.60:0/1462496377 >> 172.21.15.60:0/1462496377 conn(0x7f42d0063b50 msgr2=0x7f42d00b0540 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:56.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.741+0000 7f42d685d700 1 -- 172.21.15.60:0/1462496377 shutdown_connections 2022-06-29T16:07:56.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.741+0000 7f42d685d700 1 -- 172.21.15.60:0/1462496377 wait complete. 2022-06-29T16:07:56.750 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.2 2022-06-29T16:07:56.760 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:07:56.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.861+0000 7fbfb7f20700 1 Processor -- start 2022-06-29T16:07:56.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.861+0000 7fbfb7f20700 1 -- start start 2022-06-29T16:07:56.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.861+0000 7fbfb7f20700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbfb005d1f0 0x7fbfb00b1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:56.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.861+0000 7fbfb7f20700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbfb00b3cc0 0x7fbfb00b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:56.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.861+0000 7fbfb7f20700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb00b2ce0 con 0x7fbfb00b9d70 2022-06-29T16:07:56.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.861+0000 7fbfb7f20700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfb00b4d10 con 0x7fbfb005d1f0 2022-06-29T16:07:56.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.861+0000 7fbfb7f20700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb00ba150 con 0x7fbfb00b3cc0 2022-06-29T16:07:56.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.861+0000 7fbfb64bd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbfb00b3cc0 0x7fbfb00b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:56.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.861+0000 7fbfb54bb700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbfb005d1f0 0x7fbfb00b1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:56.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.861+0000 7fbfb5cbc700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fbfb00b9d70 0x7fbfb00b3140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:37874/0 (socket says 172.21.15.60:37874) 2022-06-29T16:07:56.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.861+0000 7fbfb5cbc700 1 -- 172.21.15.60:0/3457834623 learned_addr learned my addr 172.21.15.60:0/3457834623 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:56.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.861+0000 7fbfb4cba700 1 -- 172.21.15.60:0/3457834623 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 854607325 0 0) 0x7fbfb00b2ce0 con 0x7fbfb00b9d70 2022-06-29T16:07:56.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.861+0000 7fbfb4cba700 1 -- 172.21.15.60:0/3457834623 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf90003260 con 0x7fbfb00b9d70 2022-06-29T16:07:56.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.861+0000 7fbfb54bb700 1 -- 172.21.15.60:0/3457834623 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbfb00b3cc0 msgr2=0x7fbfb00b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:56.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.861+0000 7fbfb54bb700 1 --2- 172.21.15.60:0/3457834623 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbfb00b3cc0 0x7fbfb00b2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:56.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.861+0000 7fbfb54bb700 1 -- 172.21.15.60:0/3457834623 >> v1:172.21.15.92:6789/0 conn(0x7fbfb00b9d70 legacy=0x7fbfb00b3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:56.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.861+0000 7fbfb54bb700 1 -- 172.21.15.60:0/3457834623 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb00b4a70 con 0x7fbfb005d1f0 2022-06-29T16:07:56.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.861+0000 7fbfb64bd700 1 --2- 172.21.15.60:0/3457834623 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbfb00b3cc0 0x7fbfb00b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:07:56.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.861+0000 7fbfb54bb700 1 --2- 172.21.15.60:0/3457834623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfb005d1f0 0x7fbfb00b1b50 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7fbf98002f70 tx=0x7fbf98006df0).ready entity=mon.1 client_cookie=740f887fff58b44b server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:56.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.861+0000 7fbfb4cba700 1 -- 172.21.15.60:0/3457834623 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf9800d4a0 con 0x7fbfb005d1f0 2022-06-29T16:07:56.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.861+0000 7fbfb4cba700 1 -- 172.21.15.60:0/3457834623 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf98000e90 con 0x7fbfb005d1f0 2022-06-29T16:07:56.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.861+0000 7fbfb4cba700 1 -- 172.21.15.60:0/3457834623 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf9800d7f0 con 0x7fbfb005d1f0 2022-06-29T16:07:56.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.865+0000 7fbfb7f20700 1 -- 172.21.15.60:0/3457834623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfb005d1f0 msgr2=0x7fbfb00b1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:56.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.865+0000 7fbfb7f20700 1 --2- 172.21.15.60:0/3457834623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfb005d1f0 0x7fbfb00b1b50 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7fbf98002f70 tx=0x7fbf98006df0).stop 2022-06-29T16:07:56.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.865+0000 7fbfb7f20700 1 -- 172.21.15.60:0/3457834623 shutdown_connections 2022-06-29T16:07:56.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.865+0000 7fbfb7f20700 1 --2- 172.21.15.60:0/3457834623 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbfb00b3cc0 0x7fbfb00b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:56.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.865+0000 7fbfb7f20700 1 --2- 172.21.15.60:0/3457834623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfb005d1f0 0x7fbfb00b1b50 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:56.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.865+0000 7fbfb7f20700 1 -- 172.21.15.60:0/3457834623 >> 172.21.15.60:0/3457834623 conn(0x7fbfb0062ce0 msgr2=0x7fbfb00b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:56.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.865+0000 7fbfb7f20700 1 -- 172.21.15.60:0/3457834623 shutdown_connections 2022-06-29T16:07:56.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.865+0000 7fbfb7f20700 1 -- 172.21.15.60:0/3457834623 wait complete. 2022-06-29T16:07:56.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.865+0000 7fbfb7f20700 1 Processor -- start 2022-06-29T16:07:56.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.865+0000 7fbfb7f20700 1 -- start start 2022-06-29T16:07:56.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.865+0000 7fbfb7f20700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbfb005d1f0 0x7fbfb00ad9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:56.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.865+0000 7fbfb7f20700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfb00b3cc0 0x7fbfb00ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:56.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.865+0000 7fbfb7f20700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfb00b9d70 0x7fbfb013db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:56.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.865+0000 7fbfb7f20700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb00ae9e0 con 0x7fbfb00b9d70 2022-06-29T16:07:56.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.865+0000 7fbfb7f20700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb00af5d0 con 0x7fbfb00b3cc0 2022-06-29T16:07:56.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.865+0000 7fbfb7f20700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb00af870 con 0x7fbfb005d1f0 2022-06-29T16:07:56.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.865+0000 7fbfb64bd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfb00b9d70 0x7fbfb013db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:56.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.865+0000 7fbfb5cbc700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbfb005d1f0 0x7fbfb00ad9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:56.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.865+0000 7fbfb54bb700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfb00b3cc0 0x7fbfb00ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:56.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.865+0000 7fbfb64bd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfb00b9d70 0x7fbfb013db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53782/0 (socket says 172.21.15.60:53782) 2022-06-29T16:07:56.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.865+0000 7fbfb64bd700 1 -- 172.21.15.60:0/2404792985 learned_addr learned my addr 172.21.15.60:0/2404792985 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:56.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.869+0000 7fbfb64bd700 1 -- 172.21.15.60:0/2404792985 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbfb005d1f0 msgr2=0x7fbfb00ad9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:56.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.869+0000 7fbfb64bd700 1 --2- 172.21.15.60:0/2404792985 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbfb005d1f0 0x7fbfb00ad9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:56.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.869+0000 7fbfb64bd700 1 -- 172.21.15.60:0/2404792985 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfb00b3cc0 msgr2=0x7fbfb00ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:56.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.869+0000 7fbfb64bd700 1 --2- 172.21.15.60:0/2404792985 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfb00b3cc0 0x7fbfb00ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:56.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.869+0000 7fbfb64bd700 1 -- 172.21.15.60:0/2404792985 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa800b180 con 0x7fbfb00b9d70 2022-06-29T16:07:56.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.869+0000 7fbfb64bd700 1 --2- 172.21.15.60:0/2404792985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfb00b9d70 0x7fbfb013db90 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 rx=0x7fbfa800c000 tx=0x7fbfa8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:56.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.869+0000 7fbfaeffd700 1 -- 172.21.15.60:0/2404792985 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfa8011750 con 0x7fbfb00b9d70 2022-06-29T16:07:56.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.869+0000 7fbfaeffd700 1 -- 172.21.15.60:0/2404792985 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa80118b0 con 0x7fbfb00b9d70 2022-06-29T16:07:56.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.869+0000 7fbfb7f20700 1 -- 172.21.15.60:0/2404792985 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf9800b180 con 0x7fbfb00b9d70 2022-06-29T16:07:56.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.869+0000 7fbfaeffd700 1 -- 172.21.15.60:0/2404792985 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfa8011c10 con 0x7fbfb00b9d70 2022-06-29T16:07:56.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.869+0000 7fbfb7f20700 1 -- 172.21.15.60:0/2404792985 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfb00adf60 con 0x7fbfb00b9d70 2022-06-29T16:07:56.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.869+0000 7fbfb7f20700 1 -- 172.21.15.60:0/2404792985 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfb00573d0 con 0x7fbfb00b9d70 2022-06-29T16:07:56.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.873+0000 7fbfaeffd700 1 -- 172.21.15.60:0/2404792985 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fbfa8011d70 con 0x7fbfb00b9d70 2022-06-29T16:07:56.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.873+0000 7fbfaeffd700 1 --2- 172.21.15.60:0/2404792985 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbf940539e0 0x7fbf94055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:56.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.873+0000 7fbfaeffd700 1 -- 172.21.15.60:0/2404792985 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbfa806cf10 con 0x7fbfb00b9d70 2022-06-29T16:07:56.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.873+0000 7fbfb5cbc700 1 --2- 172.21.15.60:0/2404792985 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbf940539e0 0x7fbf94055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:56.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.873+0000 7fbfb5cbc700 1 --2- 172.21.15.60:0/2404792985 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbf940539e0 0x7fbf94055ea0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7fbfa000d000 tx=0x7fbfa0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:56.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:56.877+0000 7fbfaeffd700 1 -- 172.21.15.60:0/2404792985 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbfa8043d20 con 0x7fbfb00b9d70 2022-06-29T16:07:57.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.041+0000 7fbfb7f20700 1 -- 172.21.15.60:0/2404792985 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbfb005d990 con 0x7fbfb00b9d70 2022-06-29T16:07:57.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.045+0000 7fbfaeffd700 1 -- 172.21.15.60:0/2404792985 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2142 (secure 0 0 0) 0x7fbfa8047340 con 0x7fbfb00b9d70 2022-06-29T16:07:57.049 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:57.049 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":68,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.92:6835/2628497783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2628497783},{"type":"v1","addr":"172.21.15.92:6835","nonce":2628497783}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":67}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:07:54.174936+0000","modified":"2022-06-29T16:07:55.178863+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8463":{"gid":8463,"name":"a","rank":0,"incarnation":67,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/2222627281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2222627281},{"type":"v1","addr":"172.21.15.60:6833","nonce":2222627281}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2022-06-29T16:07:57.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.045+0000 7fbfb7f20700 1 -- 172.21.15.60:0/2404792985 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbf940539e0 msgr2=0x7fbf94055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:57.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.045+0000 7fbfb7f20700 1 --2- 172.21.15.60:0/2404792985 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbf940539e0 0x7fbf94055ea0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7fbfa000d000 tx=0x7fbfa0009040).stop 2022-06-29T16:07:57.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.045+0000 7fbfb7f20700 1 -- 172.21.15.60:0/2404792985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfb00b9d70 msgr2=0x7fbfb013db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:57.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.045+0000 7fbfb7f20700 1 --2- 172.21.15.60:0/2404792985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfb00b9d70 0x7fbfb013db90 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 rx=0x7fbfa800c000 tx=0x7fbfa8006df0).stop 2022-06-29T16:07:57.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.045+0000 7fbfb7f20700 1 -- 172.21.15.60:0/2404792985 shutdown_connections 2022-06-29T16:07:57.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.045+0000 7fbfb7f20700 1 --2- 172.21.15.60:0/2404792985 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbf940539e0 0x7fbf94055ea0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:57.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.045+0000 7fbfb7f20700 1 --2- 172.21.15.60:0/2404792985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbfb00b9d70 0x7fbfb013db90 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:57.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.045+0000 7fbfb7f20700 1 --2- 172.21.15.60:0/2404792985 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfb00b3cc0 0x7fbfb00ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:57.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.045+0000 7fbfb7f20700 1 --2- 172.21.15.60:0/2404792985 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbfb005d1f0 0x7fbfb00ad9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:57.053 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.045+0000 7fbfb7f20700 1 -- 172.21.15.60:0/2404792985 >> 172.21.15.60:0/2404792985 conn(0x7fbfb0062ce0 msgr2=0x7fbfb0141400 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:57.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.045+0000 7fbfb7f20700 1 -- 172.21.15.60:0/2404792985 shutdown_connections 2022-06-29T16:07:57.054 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.045+0000 7fbfb7f20700 1 -- 172.21.15.60:0/2404792985 wait complete. 2022-06-29T16:07:57.054 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 68 2022-06-29T16:07:57.264 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 68, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:07:54.174936+0000', 'modified': '2022-06-29T16:07:55.178863+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8463': {'gid': 8463, 'name': 'a', 'rank': 0, 'incarnation': 67, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.60:6833/2222627281', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6832', 'nonce': 2222627281}, {'type': 'v1', 'addr': '172.21.15.60:6833', 'nonce': 2222627281}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:07:57.265 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:07:57.266 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-06-29T16:07:57.384 INFO:teuthology.orchestra.run.smithi060.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":8463,"want_state":"up:active","state":"up:active","rank_uptime":3.1935359499999998,"mdsmap_epoch":68,"osdmap_epoch":137,"osdmap_epoch_barrier":137,"uptime":7.548520195} 2022-06-29T16:07:57.384 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 8463, "mdsmap_epoch": 68, "osdmap_epoch": 137, "osdmap_epoch_barrier": 137, "rank_uptime": 3.19353595, "state": "up:active", "uptime": 7.548520195, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:07:57.385 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:07:57.486 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.481+0000 7f756359e700 1 Processor -- start 2022-06-29T16:07:57.486 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.481+0000 7f756359e700 1 -- start start 2022-06-29T16:07:57.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.481+0000 7f756359e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f75640ad990 0x7f7564144900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:57.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.481+0000 7f756359e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7564143cf0 con 0x7f75640acd60 2022-06-29T16:07:57.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.481+0000 7f756359e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75640ad140 con 0x7f75640657d0 2022-06-29T16:07:57.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.481+0000 7f756359e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7564064a70 con 0x7f75640ad990 2022-06-29T16:07:57.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.481+0000 7f756259c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f75640acd60 0x7f75641296f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33640/0 (socket says 172.21.15.60:33640) 2022-06-29T16:07:57.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.481+0000 7f756259c700 1 -- 172.21.15.60:0/2917371416 learned_addr learned my addr 172.21.15.60:0/2917371416 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:57.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.481+0000 7f755bfff700 1 --2- 172.21.15.60:0/2917371416 >> v2:172.21.15.92:3300/0 conn(0x7f75640ad990 0x7f7564144900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:57.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.485+0000 7f7561d9b700 1 -- 172.21.15.60:0/2917371416 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3538098168 0 0) 0x7f7564143cf0 con 0x7f75640acd60 2022-06-29T16:07:57.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.485+0000 7f7561d9b700 1 -- 172.21.15.60:0/2917371416 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7544003260 con 0x7f75640acd60 2022-06-29T16:07:57.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.485+0000 7f7561d9b700 1 -- 172.21.15.60:0/2917371416 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2586028746 0 0) 0x7f75640ad140 con 0x7f75640657d0 2022-06-29T16:07:57.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.485+0000 7f7561d9b700 1 -- 172.21.15.60:0/2917371416 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7564143cf0 con 0x7f75640657d0 2022-06-29T16:07:57.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.485+0000 7f755bfff700 1 -- 172.21.15.60:0/2917371416 >> v1:172.21.15.92:6789/0 conn(0x7f75640657d0 legacy=0x7f75640b1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:57.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.485+0000 7f755bfff700 1 -- 172.21.15.60:0/2917371416 >> v1:172.21.15.60:6789/0 conn(0x7f75640acd60 legacy=0x7f75641296f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:57.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.485+0000 7f755bfff700 1 -- 172.21.15.60:0/2917371416 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7564143f20 con 0x7f75640ad990 2022-06-29T16:07:57.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.485+0000 7f755bfff700 1 --2- 172.21.15.60:0/2917371416 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75640ad990 0x7f7564144900 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0x7f754c002f70 tx=0x7f754c006df0).ready entity=mon.1 client_cookie=bca576bbb399550a server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:57.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.485+0000 7f7561d9b700 1 -- 172.21.15.60:0/2917371416 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f754c00d540 con 0x7f75640ad990 2022-06-29T16:07:57.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.485+0000 7f7561d9b700 1 -- 172.21.15.60:0/2917371416 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f754c000e90 con 0x7f75640ad990 2022-06-29T16:07:57.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.485+0000 7f7561d9b700 1 -- 172.21.15.60:0/2917371416 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f754c00d890 con 0x7f75640ad990 2022-06-29T16:07:57.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.485+0000 7f756359e700 1 -- 172.21.15.60:0/2917371416 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75640ad990 msgr2=0x7f7564144900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:57.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.485+0000 7f756359e700 1 --2- 172.21.15.60:0/2917371416 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75640ad990 0x7f7564144900 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0x7f754c002f70 tx=0x7f754c006df0).stop 2022-06-29T16:07:57.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.485+0000 7f756359e700 1 -- 172.21.15.60:0/2917371416 shutdown_connections 2022-06-29T16:07:57.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.485+0000 7f756359e700 1 --2- 172.21.15.60:0/2917371416 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75640ad990 0x7f7564144900 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:57.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.485+0000 7f756359e700 1 -- 172.21.15.60:0/2917371416 >> 172.21.15.60:0/2917371416 conn(0x7f75640b6f80 msgr2=0x7f75640b7390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:57.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.485+0000 7f756359e700 1 -- 172.21.15.60:0/2917371416 shutdown_connections 2022-06-29T16:07:57.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.485+0000 7f756359e700 1 -- 172.21.15.60:0/2917371416 wait complete. 2022-06-29T16:07:57.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.485+0000 7f756359e700 1 Processor -- start 2022-06-29T16:07:57.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.485+0000 7f756359e700 1 -- start start 2022-06-29T16:07:57.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.489+0000 7f756359e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75640657d0 0x7f7564110990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:57.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.489+0000 7f756359e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75640acd60 0x7f75640af180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:57.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.489+0000 7f756359e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75640ad990 0x7f7564153530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:57.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.489+0000 7f756359e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7564141850 con 0x7f75640657d0 2022-06-29T16:07:57.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.489+0000 7f756359e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7564152df0 con 0x7f75640acd60 2022-06-29T16:07:57.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.489+0000 7f756359e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7564153090 con 0x7f75640ad990 2022-06-29T16:07:57.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.489+0000 7f756259c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75640657d0 0x7f7564110990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:57.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.489+0000 7f756259c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75640657d0 0x7f7564110990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53792/0 (socket says 172.21.15.60:53792) 2022-06-29T16:07:57.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.489+0000 7f756259c700 1 -- 172.21.15.60:0/3459477773 learned_addr learned my addr 172.21.15.60:0/3459477773 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:57.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.489+0000 7f755bfff700 1 --2- 172.21.15.60:0/3459477773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75640acd60 0x7f75640af180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:57.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.489+0000 7f7562d9d700 1 --2- 172.21.15.60:0/3459477773 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75640ad990 0x7f7564153530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:57.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.489+0000 7f756259c700 1 -- 172.21.15.60:0/3459477773 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75640ad990 msgr2=0x7f7564153530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:57.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.489+0000 7f756259c700 1 --2- 172.21.15.60:0/3459477773 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75640ad990 0x7f7564153530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:57.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.489+0000 7f756259c700 1 -- 172.21.15.60:0/3459477773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75640acd60 msgr2=0x7f75640af180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:57.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.489+0000 7f756259c700 1 --2- 172.21.15.60:0/3459477773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75640acd60 0x7f75640af180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:57.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.489+0000 7f756259c700 1 -- 172.21.15.60:0/3459477773 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f754c00b180 con 0x7f75640657d0 2022-06-29T16:07:57.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.489+0000 7f756259c700 1 --2- 172.21.15.60:0/3459477773 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75640657d0 0x7f7564110990 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 rx=0x7f75540011d0 tx=0x7f7554007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:57.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.489+0000 7f755b7fe700 1 -- 172.21.15.60:0/3459477773 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75540070d0 con 0x7f75640657d0 2022-06-29T16:07:57.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.489+0000 7f755b7fe700 1 -- 172.21.15.60:0/3459477773 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7554007230 con 0x7f75640657d0 2022-06-29T16:07:57.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.489+0000 7f756359e700 1 -- 172.21.15.60:0/3459477773 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7564110700 con 0x7f75640657d0 2022-06-29T16:07:57.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.489+0000 7f755b7fe700 1 -- 172.21.15.60:0/3459477773 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7554005610 con 0x7f75640657d0 2022-06-29T16:07:57.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.489+0000 7f756359e700 1 -- 172.21.15.60:0/3459477773 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f756414f6a0 con 0x7f75640657d0 2022-06-29T16:07:57.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.493+0000 7f756359e700 1 -- 172.21.15.60:0/3459477773 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7564153250 con 0x7f75640657d0 2022-06-29T16:07:57.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.493+0000 7f755b7fe700 1 -- 172.21.15.60:0/3459477773 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7554010070 con 0x7f75640657d0 2022-06-29T16:07:57.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.493+0000 7f755b7fe700 1 --2- 172.21.15.60:0/3459477773 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7548053a70 0x7f7548055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:57.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.493+0000 7f755b7fe700 1 -- 172.21.15.60:0/3459477773 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f755406b720 con 0x7f75640657d0 2022-06-29T16:07:57.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.493+0000 7f755bfff700 1 --2- 172.21.15.60:0/3459477773 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7548053a70 0x7f7548055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:57.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.493+0000 7f755bfff700 1 --2- 172.21.15.60:0/3459477773 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7548053a70 0x7f7548055f30 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7f754c01a000 tx=0x7f754c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:57.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.501+0000 7f755b7fe700 1 -- 172.21.15.60:0/3459477773 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f75540424b0 con 0x7f75640657d0 2022-06-29T16:07:57.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.633+0000 7f756359e700 1 -- 172.21.15.60:0/3459477773 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75640573d0 con 0x7f75640657d0 2022-06-29T16:07:57.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.637+0000 7f755b7fe700 1 -- 172.21.15.60:0/3459477773 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v137) v1 ==== 74+0+16052 (secure 0 0 0) 0x7f7554045ad0 con 0x7f75640657d0 2022-06-29T16:07:57.641 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:57.641 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":137,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:07:55.177815+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:07:52.605597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"136","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2022-06-29T16:07:53.864756+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:07:57.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.637+0000 7f756359e700 1 -- 172.21.15.60:0/3459477773 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7548053a70 msgr2=0x7f7548055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:57.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.637+0000 7f756359e700 1 --2- 172.21.15.60:0/3459477773 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7548053a70 0x7f7548055f30 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7f754c01a000 tx=0x7f754c014000).stop 2022-06-29T16:07:57.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.637+0000 7f756359e700 1 -- 172.21.15.60:0/3459477773 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75640657d0 msgr2=0x7f7564110990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:57.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.637+0000 7f756359e700 1 --2- 172.21.15.60:0/3459477773 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75640657d0 0x7f7564110990 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 rx=0x7f75540011d0 tx=0x7f7554007b50).stop 2022-06-29T16:07:57.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.637+0000 7f756359e700 1 -- 172.21.15.60:0/3459477773 shutdown_connections 2022-06-29T16:07:57.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.637+0000 7f756359e700 1 --2- 172.21.15.60:0/3459477773 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7548053a70 0x7f7548055f30 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:57.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.637+0000 7f756359e700 1 --2- 172.21.15.60:0/3459477773 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f75640ad990 0x7f7564153530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:57.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.637+0000 7f756359e700 1 --2- 172.21.15.60:0/3459477773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75640acd60 0x7f75640af180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:57.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.637+0000 7f756359e700 1 --2- 172.21.15.60:0/3459477773 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f75640657d0 0x7f7564110990 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:57.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.637+0000 7f756359e700 1 -- 172.21.15.60:0/3459477773 >> 172.21.15.60:0/3459477773 conn(0x7f75640b6f80 msgr2=0x7f75640b7360 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:57.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.637+0000 7f756359e700 1 -- 172.21.15.60:0/3459477773 shutdown_connections 2022-06-29T16:07:57.646 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.637+0000 7f756359e700 1 -- 172.21.15.60:0/3459477773 wait complete. 2022-06-29T16:07:57.655 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-29T16:07:52.605597+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '136', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2022-06-29T16:07:53.864756+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '137', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-29T16:07:57.657 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:07:57.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.761+0000 7f45331c2700 1 Processor -- start 2022-06-29T16:07:57.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.761+0000 7f45331c2700 1 -- start start 2022-06-29T16:07:57.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.761+0000 7f45331c2700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f452c0b2120 0x7f452c0ab880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:57.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.761+0000 7f45331c2700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f452c05cf40 0x7f452c14af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:57.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.761+0000 7f45331c2700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f452c14a530 con 0x7f452c05bd20 2022-06-29T16:07:57.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.761+0000 7f45331c2700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f452c0b1720 con 0x7f452c05cf40 2022-06-29T16:07:57.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.761+0000 7f45331c2700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f452c14a7d0 con 0x7f452c0b2120 2022-06-29T16:07:57.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.761+0000 7f45319bf700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f452c05cf40 0x7f452c14af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:57.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.761+0000 7f45319bf700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f452c05cf40 0x7f452c14af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53802/0 (socket says 172.21.15.60:53802) 2022-06-29T16:07:57.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.761+0000 7f45319bf700 1 -- 172.21.15.60:0/2219705407 learned_addr learned my addr 172.21.15.60:0/2219705407 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:57.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.761+0000 7f45321c0700 1 --2- 172.21.15.60:0/2219705407 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f452c0b2120 0x7f452c0ab880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:57.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.761+0000 7f45319bf700 1 -- 172.21.15.60:0/2219705407 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f452c0b2120 msgr2=0x7f452c0ab880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:57.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.761+0000 7f45319bf700 1 --2- 172.21.15.60:0/2219705407 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f452c0b2120 0x7f452c0ab880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:57.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.761+0000 7f45319bf700 1 -- 172.21.15.60:0/2219705407 >> v1:172.21.15.92:6789/0 conn(0x7f452c05bd20 legacy=0x7f452c05c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:57.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.761+0000 7f45319bf700 1 -- 172.21.15.60:0/2219705407 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f452c14aa70 con 0x7f452c05cf40 2022-06-29T16:07:57.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.765+0000 7f45319bf700 1 --2- 172.21.15.60:0/2219705407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f452c05cf40 0x7f452c14af10 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 rx=0x7f45280032f0 tx=0x7f4528009290).ready entity=mon.0 client_cookie=636562719cb244c server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:57.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.765+0000 7f45311be700 1 -- 172.21.15.60:0/2219705407 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f452800b8a0 con 0x7f452c05cf40 2022-06-29T16:07:57.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.765+0000 7f45311be700 1 -- 172.21.15.60:0/2219705407 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f452800ba00 con 0x7f452c05cf40 2022-06-29T16:07:57.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.765+0000 7f45311be700 1 -- 172.21.15.60:0/2219705407 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f452800bb70 con 0x7f452c05cf40 2022-06-29T16:07:57.777 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.765+0000 7f45331c2700 1 -- 172.21.15.60:0/2219705407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f452c05cf40 msgr2=0x7f452c14af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:57.778 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.765+0000 7f45331c2700 1 --2- 172.21.15.60:0/2219705407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f452c05cf40 0x7f452c14af10 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 rx=0x7f45280032f0 tx=0x7f4528009290).stop 2022-06-29T16:07:57.778 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.765+0000 7f45331c2700 1 -- 172.21.15.60:0/2219705407 shutdown_connections 2022-06-29T16:07:57.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.765+0000 7f45331c2700 1 --2- 172.21.15.60:0/2219705407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f452c05cf40 0x7f452c14af10 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:57.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.765+0000 7f45331c2700 1 --2- 172.21.15.60:0/2219705407 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f452c0b2120 0x7f452c0ab880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:57.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.765+0000 7f45331c2700 1 -- 172.21.15.60:0/2219705407 >> 172.21.15.60:0/2219705407 conn(0x7f452c0b58a0 msgr2=0x7f452c0b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:57.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.765+0000 7f45331c2700 1 -- 172.21.15.60:0/2219705407 shutdown_connections 2022-06-29T16:07:57.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.765+0000 7f45331c2700 1 -- 172.21.15.60:0/2219705407 wait complete. 2022-06-29T16:07:57.781 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.765+0000 7f45331c2700 1 Processor -- start 2022-06-29T16:07:57.781 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.765+0000 7f45331c2700 1 -- start start 2022-06-29T16:07:57.781 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.765+0000 7f45331c2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f452c05bd20 0x7f452c063c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:57.782 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.765+0000 7f45331c2700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f452c05cf40 0x7f452c1566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:57.782 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.765+0000 7f45331c2700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f452c0b2120 0x7f452c156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:57.782 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.765+0000 7f45331c2700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452c0b2f70 con 0x7f452c05bd20 2022-06-29T16:07:57.782 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.765+0000 7f45331c2700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452c0b31d0 con 0x7f452c0b2120 2022-06-29T16:07:57.783 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.765+0000 7f45331c2700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f452c156240 con 0x7f452c05cf40 2022-06-29T16:07:57.783 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.765+0000 7f45321c0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f452c05bd20 0x7f452c063c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:57.783 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.765+0000 7f45321c0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f452c05bd20 0x7f452c063c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53806/0 (socket says 172.21.15.60:53806) 2022-06-29T16:07:57.784 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.769+0000 7f45321c0700 1 -- 172.21.15.60:0/3201275852 learned_addr learned my addr 172.21.15.60:0/3201275852 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:57.784 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.769+0000 7f45319bf700 1 --2- 172.21.15.60:0/3201275852 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f452c05cf40 0x7f452c1566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:57.784 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.769+0000 7f45329c1700 1 --2- 172.21.15.60:0/3201275852 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f452c0b2120 0x7f452c156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:57.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.769+0000 7f45319bf700 1 -- 172.21.15.60:0/3201275852 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f452c0b2120 msgr2=0x7f452c156bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:57.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.769+0000 7f45319bf700 1 --2- 172.21.15.60:0/3201275852 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f452c0b2120 0x7f452c156bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:57.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.769+0000 7f45319bf700 1 -- 172.21.15.60:0/3201275852 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f452c05bd20 msgr2=0x7f452c063c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:57.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.769+0000 7f45319bf700 1 --2- 172.21.15.60:0/3201275852 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f452c05bd20 0x7f452c063c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:57.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.769+0000 7f45319bf700 1 -- 172.21.15.60:0/3201275852 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4528009530 con 0x7f452c05cf40 2022-06-29T16:07:57.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.769+0000 7f45319bf700 1 --2- 172.21.15.60:0/3201275852 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f452c05cf40 0x7f452c1566d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f45280032f0 tx=0x7f4528016000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:57.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.769+0000 7f451b7fe700 1 -- 172.21.15.60:0/3201275852 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f452800b8a0 con 0x7f452c05cf40 2022-06-29T16:07:57.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.769+0000 7f451b7fe700 1 -- 172.21.15.60:0/3201275852 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4528021440 con 0x7f452c05cf40 2022-06-29T16:07:57.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.769+0000 7f45331c2700 1 -- 172.21.15.60:0/3201275852 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f452c0b3ea0 con 0x7f452c05cf40 2022-06-29T16:07:57.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.769+0000 7f451b7fe700 1 -- 172.21.15.60:0/3201275852 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45280217a0 con 0x7f452c05cf40 2022-06-29T16:07:57.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.769+0000 7f45331c2700 1 -- 172.21.15.60:0/3201275852 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f452c157250 con 0x7f452c05cf40 2022-06-29T16:07:57.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.769+0000 7f45331c2700 1 -- 172.21.15.60:0/3201275852 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f452c0573d0 con 0x7f452c05cf40 2022-06-29T16:07:57.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.769+0000 7f451b7fe700 1 -- 172.21.15.60:0/3201275852 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4528029070 con 0x7f452c05cf40 2022-06-29T16:07:57.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.769+0000 7f451b7fe700 1 --2- 172.21.15.60:0/3201275852 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f450c0539e0 0x7f450c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:57.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.769+0000 7f451b7fe700 1 -- 172.21.15.60:0/3201275852 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f452806ce50 con 0x7f452c05cf40 2022-06-29T16:07:57.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.773+0000 7f45321c0700 1 --2- 172.21.15.60:0/3201275852 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f450c0539e0 0x7f450c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:57.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.773+0000 7f45321c0700 1 --2- 172.21.15.60:0/3201275852 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f450c0539e0 0x7f450c055ea0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0x7f451c00b000 tx=0x7f451c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:57.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.781+0000 7f451b7fe700 1 -- 172.21.15.60:0/3201275852 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4528043cd0 con 0x7f452c05cf40 2022-06-29T16:07:57.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.909+0000 7f45331c2700 1 -- 172.21.15.60:0/3201275852 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f452c157250 con 0x7f452c05cf40 2022-06-29T16:07:57.913 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.909+0000 7f451b7fe700 1 -- 172.21.15.60:0/3201275852 <== mon.2 v2:172.21.15.92:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v137) v1 ==== 110+0+10 (secure 0 0 0) 0x7f452c157250 con 0x7f452c05cf40 2022-06-29T16:07:57.914 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:07:57.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.913+0000 7f45331c2700 1 -- 172.21.15.60:0/3201275852 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f450c0539e0 msgr2=0x7f450c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:57.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.913+0000 7f45331c2700 1 --2- 172.21.15.60:0/3201275852 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f450c0539e0 0x7f450c055ea0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0x7f451c00b000 tx=0x7f451c006df0).stop 2022-06-29T16:07:57.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.913+0000 7f45331c2700 1 -- 172.21.15.60:0/3201275852 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f452c05cf40 msgr2=0x7f452c1566d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:57.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.913+0000 7f45331c2700 1 --2- 172.21.15.60:0/3201275852 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f452c05cf40 0x7f452c1566d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f45280032f0 tx=0x7f4528016000).stop 2022-06-29T16:07:57.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.913+0000 7f45331c2700 1 -- 172.21.15.60:0/3201275852 shutdown_connections 2022-06-29T16:07:57.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.913+0000 7f45331c2700 1 --2- 172.21.15.60:0/3201275852 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f450c0539e0 0x7f450c055ea0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:57.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.913+0000 7f45331c2700 1 --2- 172.21.15.60:0/3201275852 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f452c0b2120 0x7f452c156bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:57.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.913+0000 7f45331c2700 1 --2- 172.21.15.60:0/3201275852 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f452c05cf40 0x7f452c1566d0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:57.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.913+0000 7f45331c2700 1 --2- 172.21.15.60:0/3201275852 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f452c05bd20 0x7f452c063c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:57.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.913+0000 7f45331c2700 1 -- 172.21.15.60:0/3201275852 >> 172.21.15.60:0/3201275852 conn(0x7f452c0b58a0 msgr2=0x7f452c1456e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:57.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.913+0000 7f45331c2700 1 -- 172.21.15.60:0/3201275852 shutdown_connections 2022-06-29T16:07:57.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:57.913+0000 7f45331c2700 1 -- 172.21.15.60:0/3201275852 wait complete. 2022-06-29T16:07:57.933 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:07:58.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.033+0000 7ff8affff700 1 Processor -- start 2022-06-29T16:07:58.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.033+0000 7ff8affff700 1 -- start start 2022-06-29T16:07:58.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.037+0000 7ff8affff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff8b00b5eb0 0x7ff8b0142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:58.042 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.037+0000 7ff8affff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8b00b48b0 0x7ff8b00b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:58.042 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.037+0000 7ff8affff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8b00b4d10 con 0x7ff8b00b2c50 2022-06-29T16:07:58.042 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.037+0000 7ff8affff700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8b00b54e0 con 0x7ff8b00b5eb0 2022-06-29T16:07:58.042 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.037+0000 7ff8affff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b00b6290 con 0x7ff8b00b48b0 2022-06-29T16:07:58.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.037+0000 7ff8aeffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff8b00b5eb0 0x7ff8b0142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:58.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.037+0000 7ff8aeffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff8b00b5eb0 0x7ff8b0142a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53814/0 (socket says 172.21.15.60:53814) 2022-06-29T16:07:58.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.037+0000 7ff8ae7fc700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8b00b48b0 0x7ff8b00b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:58.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.037+0000 7ff8aeffd700 1 -- 172.21.15.60:0/191211609 learned_addr learned my addr 172.21.15.60:0/191211609 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:58.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.037+0000 7ff8adffb700 1 -- 172.21.15.60:0/191211609 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 842167446 0 0) 0x7ff8b00b4d10 con 0x7ff8b00b2c50 2022-06-29T16:07:58.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.037+0000 7ff8adffb700 1 -- 172.21.15.60:0/191211609 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff890003260 con 0x7ff8b00b2c50 2022-06-29T16:07:58.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.037+0000 7ff8ae7fc700 1 -- 172.21.15.60:0/191211609 >> v2:172.21.15.60:3300/0 conn(0x7ff8b00b5eb0 msgr2=0x7ff8b0142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:58.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.037+0000 7ff8ae7fc700 1 --2- 172.21.15.60:0/191211609 >> v2:172.21.15.60:3300/0 conn(0x7ff8b00b5eb0 0x7ff8b0142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:58.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.037+0000 7ff8ae7fc700 1 -- 172.21.15.60:0/191211609 >> v1:172.21.15.92:6789/0 conn(0x7ff8b00b2c50 legacy=0x7ff8b00b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:58.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.037+0000 7ff8ae7fc700 1 -- 172.21.15.60:0/191211609 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b0142f10 con 0x7ff8b00b48b0 2022-06-29T16:07:58.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.037+0000 7ff8aeffd700 1 --2- 172.21.15.60:0/191211609 >> v2:172.21.15.60:3300/0 conn(0x7ff8b00b5eb0 0x7ff8b0142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:07:58.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.037+0000 7ff8ae7fc700 1 --2- 172.21.15.60:0/191211609 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8b00b48b0 0x7ff8b00b05e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7ff898002f70 tx=0x7ff898006df0).ready entity=mon.2 client_cookie=2735998f43133bda server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:58.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.037+0000 7ff8adffb700 1 -- 172.21.15.60:0/191211609 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff89800d4a0 con 0x7ff8b00b48b0 2022-06-29T16:07:58.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.037+0000 7ff8adffb700 1 -- 172.21.15.60:0/191211609 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff898000e90 con 0x7ff8b00b48b0 2022-06-29T16:07:58.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.037+0000 7ff8adffb700 1 -- 172.21.15.60:0/191211609 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff89800d7f0 con 0x7ff8b00b48b0 2022-06-29T16:07:58.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.037+0000 7ff8affff700 1 -- 172.21.15.60:0/191211609 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8b00b48b0 msgr2=0x7ff8b00b05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:58.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.037+0000 7ff8affff700 1 --2- 172.21.15.60:0/191211609 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8b00b48b0 0x7ff8b00b05e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7ff898002f70 tx=0x7ff898006df0).stop 2022-06-29T16:07:58.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.037+0000 7ff8affff700 1 -- 172.21.15.60:0/191211609 shutdown_connections 2022-06-29T16:07:58.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.037+0000 7ff8affff700 1 --2- 172.21.15.60:0/191211609 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8b00b48b0 0x7ff8b00b05e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:58.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.037+0000 7ff8affff700 1 --2- 172.21.15.60:0/191211609 >> v2:172.21.15.60:3300/0 conn(0x7ff8b00b5eb0 0x7ff8b0142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:58.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.037+0000 7ff8affff700 1 -- 172.21.15.60:0/191211609 >> 172.21.15.60:0/191211609 conn(0x7ff8b0062090 msgr2=0x7ff8b0062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:58.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.037+0000 7ff8affff700 1 -- 172.21.15.60:0/191211609 shutdown_connections 2022-06-29T16:07:58.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.037+0000 7ff8affff700 1 -- 172.21.15.60:0/191211609 wait complete. 2022-06-29T16:07:58.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.041+0000 7ff8affff700 1 Processor -- start 2022-06-29T16:07:58.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.041+0000 7ff8affff700 1 -- start start 2022-06-29T16:07:58.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.041+0000 7ff8affff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8b00b2c50 0x7ff8b013d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:58.056 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.041+0000 7ff8affff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8b00b48b0 0x7ff8b013bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:58.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.041+0000 7ff8affff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8b00b5eb0 0x7ff8b013c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:58.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.041+0000 7ff8affff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b00af530 con 0x7ff8b00b5eb0 2022-06-29T16:07:58.057 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.041+0000 7ff8affff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b01528b0 con 0x7ff8b00b2c50 2022-06-29T16:07:58.058 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.041+0000 7ff8affff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b0152b50 con 0x7ff8b00b48b0 2022-06-29T16:07:58.058 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.041+0000 7ff8af7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8b00b5eb0 0x7ff8b013c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:58.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.041+0000 7ff8af7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8b00b5eb0 0x7ff8b013c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53824/0 (socket says 172.21.15.60:53824) 2022-06-29T16:07:58.059 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.041+0000 7ff8af7fe700 1 -- 172.21.15.60:0/1819780239 learned_addr learned my addr 172.21.15.60:0/1819780239 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:58.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.041+0000 7ff8ae7fc700 1 --2- 172.21.15.60:0/1819780239 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8b00b48b0 0x7ff8b013bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:58.060 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.041+0000 7ff8aeffd700 1 --2- 172.21.15.60:0/1819780239 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8b00b2c50 0x7ff8b013d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:58.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.041+0000 7ff8af7fe700 1 -- 172.21.15.60:0/1819780239 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8b00b48b0 msgr2=0x7ff8b013bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:58.061 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.045+0000 7ff8af7fe700 1 --2- 172.21.15.60:0/1819780239 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8b00b48b0 0x7ff8b013bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:58.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.045+0000 7ff8af7fe700 1 -- 172.21.15.60:0/1819780239 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8b00b2c50 msgr2=0x7ff8b013d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:58.062 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.045+0000 7ff8af7fe700 1 --2- 172.21.15.60:0/1819780239 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8b00b2c50 0x7ff8b013d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:58.063 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.045+0000 7ff8af7fe700 1 -- 172.21.15.60:0/1819780239 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a000b1e0 con 0x7ff8b00b5eb0 2022-06-29T16:07:58.063 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.045+0000 7ff8ae7fc700 1 --2- 172.21.15.60:0/1819780239 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8b00b48b0 0x7ff8b013bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:07:58.063 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.045+0000 7ff8af7fe700 1 --2- 172.21.15.60:0/1819780239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8b00b5eb0 0x7ff8b013c140 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 rx=0x7ff8a8001490 tx=0x7ff8a8007dd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:58.064 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.045+0000 7ff8a7fff700 1 -- 172.21.15.60:0/1819780239 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8a8002310 con 0x7ff8b00b5eb0 2022-06-29T16:07:58.064 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.045+0000 7ff8a7fff700 1 -- 172.21.15.60:0/1819780239 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a8002470 con 0x7ff8b00b5eb0 2022-06-29T16:07:58.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.045+0000 7ff8affff700 1 -- 172.21.15.60:0/1819780239 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff89800b180 con 0x7ff8b00b5eb0 2022-06-29T16:07:58.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.045+0000 7ff8a7fff700 1 -- 172.21.15.60:0/1819780239 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8a8001840 con 0x7ff8b00b5eb0 2022-06-29T16:07:58.065 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.045+0000 7ff8affff700 1 -- 172.21.15.60:0/1819780239 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8b013c630 con 0x7ff8b00b5eb0 2022-06-29T16:07:58.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.045+0000 7ff8affff700 1 -- 172.21.15.60:0/1819780239 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8b00af370 con 0x7ff8b00b5eb0 2022-06-29T16:07:58.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.045+0000 7ff8a7fff700 1 -- 172.21.15.60:0/1819780239 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff8a8010030 con 0x7ff8b00b5eb0 2022-06-29T16:07:58.066 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.045+0000 7ff8a7fff700 1 --2- 172.21.15.60:0/1819780239 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff894053b00 0x7ff894055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:58.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.045+0000 7ff8a7fff700 1 -- 172.21.15.60:0/1819780239 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff8a806bd90 con 0x7ff8b00b5eb0 2022-06-29T16:07:58.067 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.049+0000 7ff8aeffd700 1 --2- 172.21.15.60:0/1819780239 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff894053b00 0x7ff894055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:58.068 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.049+0000 7ff8aeffd700 1 --2- 172.21.15.60:0/1819780239 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff894053b00 0x7ff894055fc0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7ff8a000b000 tx=0x7ff8a0006eb0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:58.069 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.053+0000 7ff8a7fff700 1 -- 172.21.15.60:0/1819780239 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff8a8042c10 con 0x7ff8b00b5eb0 2022-06-29T16:07:58.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.177+0000 7ff8affff700 1 -- 172.21.15.60:0/1819780239 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7ff8b0152d10 con 0x7ff8b00b5eb0 2022-06-29T16:07:58.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.177+0000 7ff8a7fff700 1 -- 172.21.15.60:0/1819780239 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v137) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff8a8046230 con 0x7ff8b00b5eb0 2022-06-29T16:07:58.184 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:07:58.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.181+0000 7ff8affff700 1 -- 172.21.15.60:0/1819780239 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff894053b00 msgr2=0x7ff894055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:58.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.181+0000 7ff8affff700 1 --2- 172.21.15.60:0/1819780239 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff894053b00 0x7ff894055fc0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7ff8a000b000 tx=0x7ff8a0006eb0).stop 2022-06-29T16:07:58.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.181+0000 7ff8affff700 1 -- 172.21.15.60:0/1819780239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8b00b5eb0 msgr2=0x7ff8b013c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:58.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.181+0000 7ff8affff700 1 --2- 172.21.15.60:0/1819780239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8b00b5eb0 0x7ff8b013c140 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 rx=0x7ff8a8001490 tx=0x7ff8a8007dd0).stop 2022-06-29T16:07:58.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.181+0000 7ff8affff700 1 -- 172.21.15.60:0/1819780239 shutdown_connections 2022-06-29T16:07:58.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.181+0000 7ff8affff700 1 --2- 172.21.15.60:0/1819780239 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff894053b00 0x7ff894055fc0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:58.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.181+0000 7ff8affff700 1 --2- 172.21.15.60:0/1819780239 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff8b00b5eb0 0x7ff8b013c140 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:58.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.181+0000 7ff8affff700 1 --2- 172.21.15.60:0/1819780239 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff8b00b48b0 0x7ff8b013bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:58.190 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.181+0000 7ff8affff700 1 --2- 172.21.15.60:0/1819780239 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8b00b2c50 0x7ff8b013d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:58.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.181+0000 7ff8affff700 1 -- 172.21.15.60:0/1819780239 >> 172.21.15.60:0/1819780239 conn(0x7ff8b0062090 msgr2=0x7ff8b00b1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:58.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.181+0000 7ff8affff700 1 -- 172.21.15.60:0/1819780239 shutdown_connections 2022-06-29T16:07:58.191 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.181+0000 7ff8affff700 1 -- 172.21.15.60:0/1819780239 wait complete. 2022-06-29T16:07:58.203 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-29T16:07:58.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.309+0000 7fb9a3fff700 1 Processor -- start 2022-06-29T16:07:58.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.309+0000 7fb9a3fff700 1 -- start start 2022-06-29T16:07:58.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.309+0000 7fb9a3fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9a405bf30 0x7fb9a4142900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:58.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.309+0000 7fb9a3fff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a40b40e0 con 0x7fb9a40b3d00 2022-06-29T16:07:58.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.309+0000 7fb9a3fff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a4146050 con 0x7fb9a4142df0 2022-06-29T16:07:58.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.309+0000 7fb9a3fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a405d4d0 con 0x7fb9a405bf30 2022-06-29T16:07:58.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.309+0000 7fb9a2ffd700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb9a40b3d00 0x7fb9a4146790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33682/0 (socket says 172.21.15.60:33682) 2022-06-29T16:07:58.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.309+0000 7fb9a2ffd700 1 -- 172.21.15.60:0/1110415595 learned_addr learned my addr 172.21.15.60:0/1110415595 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:58.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.313+0000 7fb99a7fc700 1 --2- 172.21.15.60:0/1110415595 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9a405bf30 0x7fb9a4142900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:58.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.313+0000 7fb9a27fc700 1 -- 172.21.15.60:0/1110415595 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3195464819 0 0) 0x7fb9a40b40e0 con 0x7fb9a40b3d00 2022-06-29T16:07:58.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.313+0000 7fb9a27fc700 1 -- 172.21.15.60:0/1110415595 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb984003260 con 0x7fb9a40b3d00 2022-06-29T16:07:58.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.313+0000 7fb99a7fc700 1 -- 172.21.15.60:0/1110415595 >> v1:172.21.15.92:6789/0 conn(0x7fb9a4142df0 legacy=0x7fb9a413fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:58.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.313+0000 7fb9a27fc700 1 -- 172.21.15.60:0/1110415595 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3545131456 0 0) 0x7fb9a4146050 con 0x7fb9a4142df0 2022-06-29T16:07:58.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.313+0000 7fb99a7fc700 1 -- 172.21.15.60:0/1110415595 >> v1:172.21.15.60:6789/0 conn(0x7fb9a40b3d00 legacy=0x7fb9a4146790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:58.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.313+0000 7fb99a7fc700 1 -- 172.21.15.60:0/1110415595 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a41462f0 con 0x7fb9a405bf30 2022-06-29T16:07:58.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.313+0000 7fb99a7fc700 1 --2- 172.21.15.60:0/1110415595 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9a405bf30 0x7fb9a4142900 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fb98c002f70 tx=0x7fb98c006df0).ready entity=mon.2 client_cookie=5efb8fd5af537de3 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:58.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.313+0000 7fb9a27fc700 1 -- 172.21.15.60:0/1110415595 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb98c00d570 con 0x7fb9a405bf30 2022-06-29T16:07:58.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.313+0000 7fb9a27fc700 1 -- 172.21.15.60:0/1110415595 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb98c000e90 con 0x7fb9a405bf30 2022-06-29T16:07:58.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.313+0000 7fb9a27fc700 1 -- 172.21.15.60:0/1110415595 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb98c00d8c0 con 0x7fb9a405bf30 2022-06-29T16:07:58.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.313+0000 7fb9a3fff700 1 -- 172.21.15.60:0/1110415595 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9a405bf30 msgr2=0x7fb9a4142900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:58.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.313+0000 7fb9a3fff700 1 --2- 172.21.15.60:0/1110415595 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9a405bf30 0x7fb9a4142900 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fb98c002f70 tx=0x7fb98c006df0).stop 2022-06-29T16:07:58.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.313+0000 7fb9a3fff700 1 -- 172.21.15.60:0/1110415595 shutdown_connections 2022-06-29T16:07:58.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.313+0000 7fb9a3fff700 1 --2- 172.21.15.60:0/1110415595 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9a405bf30 0x7fb9a4142900 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:58.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.313+0000 7fb9a3fff700 1 -- 172.21.15.60:0/1110415595 >> 172.21.15.60:0/1110415595 conn(0x7fb9a4064810 msgr2=0x7fb9a4064bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:58.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.313+0000 7fb9a3fff700 1 -- 172.21.15.60:0/1110415595 shutdown_connections 2022-06-29T16:07:58.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.313+0000 7fb9a3fff700 1 -- 172.21.15.60:0/1110415595 wait complete. 2022-06-29T16:07:58.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.317+0000 7fb9a3fff700 1 Processor -- start 2022-06-29T16:07:58.328 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.317+0000 7fb9a3fff700 1 -- start start 2022-06-29T16:07:58.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.317+0000 7fb9a3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9a405bf30 0x7fb9a41453e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:58.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.317+0000 7fb9a3fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9a40b3d00 0x7fb9a41553a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:58.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.317+0000 7fb9a3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb9a4142df0 0x7fb9a4155890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:58.329 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.317+0000 7fb9a3fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a4062e70 con 0x7fb9a405bf30 2022-06-29T16:07:58.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.317+0000 7fb9a3fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a4063110 con 0x7fb9a4142df0 2022-06-29T16:07:58.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.317+0000 7fb9a3fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a4062400 con 0x7fb9a40b3d00 2022-06-29T16:07:58.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.317+0000 7fb9a2ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9a405bf30 0x7fb9a41453e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:58.330 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.317+0000 7fb9a2ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9a405bf30 0x7fb9a41453e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53836/0 (socket says 172.21.15.60:53836) 2022-06-29T16:07:58.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.317+0000 7fb9a2ffd700 1 -- 172.21.15.60:0/1304586315 learned_addr learned my addr 172.21.15.60:0/1304586315 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:58.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.317+0000 7fb99a7fc700 1 --2- 172.21.15.60:0/1304586315 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9a40b3d00 0x7fb9a41553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:58.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.317+0000 7fb9a37fe700 1 --2- 172.21.15.60:0/1304586315 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb9a4142df0 0x7fb9a4155890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:58.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.317+0000 7fb9a2ffd700 1 -- 172.21.15.60:0/1304586315 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9a40b3d00 msgr2=0x7fb9a41553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:58.331 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.317+0000 7fb9a2ffd700 1 --2- 172.21.15.60:0/1304586315 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9a40b3d00 0x7fb9a41553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:58.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.317+0000 7fb9a2ffd700 1 -- 172.21.15.60:0/1304586315 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb9a4142df0 msgr2=0x7fb9a4155890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:58.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.317+0000 7fb9a2ffd700 1 --2- 172.21.15.60:0/1304586315 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb9a4142df0 0x7fb9a4155890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:58.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.317+0000 7fb9a2ffd700 1 -- 172.21.15.60:0/1304586315 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb98c00b180 con 0x7fb9a405bf30 2022-06-29T16:07:58.332 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.317+0000 7fb9a2ffd700 1 --2- 172.21.15.60:0/1304586315 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9a405bf30 0x7fb9a41453e0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 rx=0x7fb994001110 tx=0x7fb994007aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:58.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.317+0000 7fb9a0ff9700 1 -- 172.21.15.60:0/1304586315 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb9940070d0 con 0x7fb9a405bf30 2022-06-29T16:07:58.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.317+0000 7fb9a0ff9700 1 -- 172.21.15.60:0/1304586315 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb994007230 con 0x7fb9a405bf30 2022-06-29T16:07:58.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.317+0000 7fb9a3fff700 1 -- 172.21.15.60:0/1304586315 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9a4155d80 con 0x7fb9a405bf30 2022-06-29T16:07:58.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.317+0000 7fb9a0ff9700 1 -- 172.21.15.60:0/1304586315 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb994005610 con 0x7fb9a405bf30 2022-06-29T16:07:58.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.317+0000 7fb9a3fff700 1 -- 172.21.15.60:0/1304586315 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9a40b13c0 con 0x7fb9a405bf30 2022-06-29T16:07:58.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.321+0000 7fb9a3fff700 1 -- 172.21.15.60:0/1304586315 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9a40632d0 con 0x7fb9a405bf30 2022-06-29T16:07:58.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.321+0000 7fb9a0ff9700 1 -- 172.21.15.60:0/1304586315 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb994010070 con 0x7fb9a405bf30 2022-06-29T16:07:58.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.321+0000 7fb9a0ff9700 1 --2- 172.21.15.60:0/1304586315 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb988053a70 0x7fb988055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:58.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.321+0000 7fb9a0ff9700 1 -- 172.21.15.60:0/1304586315 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb99406b560 con 0x7fb9a405bf30 2022-06-29T16:07:58.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.321+0000 7fb99a7fc700 1 --2- 172.21.15.60:0/1304586315 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb988053a70 0x7fb988055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:58.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.321+0000 7fb99a7fc700 1 --2- 172.21.15.60:0/1304586315 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb988053a70 0x7fb988055f30 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7fb98c01a000 tx=0x7fb98c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:58.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.329+0000 7fb9a0ff9700 1 -- 172.21.15.60:0/1304586315 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb994042480 con 0x7fb9a405bf30 2022-06-29T16:07:58.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.453+0000 7fb9a3fff700 1 -- 172.21.15.60:0/1304586315 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb9a40573d0 con 0x7fb9a405bf30 2022-06-29T16:07:58.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.457+0000 7fb9a0ff9700 1 -- 172.21.15.60:0/1304586315 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v137) v1 ==== 104+0+10 (secure 0 0 0) 0x7fb994045aa0 con 0x7fb9a405bf30 2022-06-29T16:07:58.461 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:07:58.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.457+0000 7fb9a3fff700 1 -- 172.21.15.60:0/1304586315 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb988053a70 msgr2=0x7fb988055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:58.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.457+0000 7fb9a3fff700 1 --2- 172.21.15.60:0/1304586315 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb988053a70 0x7fb988055f30 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7fb98c01a000 tx=0x7fb98c014000).stop 2022-06-29T16:07:58.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.457+0000 7fb9a3fff700 1 -- 172.21.15.60:0/1304586315 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9a405bf30 msgr2=0x7fb9a41453e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:58.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.457+0000 7fb9a3fff700 1 --2- 172.21.15.60:0/1304586315 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9a405bf30 0x7fb9a41453e0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 rx=0x7fb994001110 tx=0x7fb994007aa0).stop 2022-06-29T16:07:58.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.457+0000 7fb9a3fff700 1 -- 172.21.15.60:0/1304586315 shutdown_connections 2022-06-29T16:07:58.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.457+0000 7fb9a3fff700 1 --2- 172.21.15.60:0/1304586315 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb988053a70 0x7fb988055f30 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:58.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.457+0000 7fb9a3fff700 1 --2- 172.21.15.60:0/1304586315 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb9a4142df0 0x7fb9a4155890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:58.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.457+0000 7fb9a3fff700 1 --2- 172.21.15.60:0/1304586315 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb9a40b3d00 0x7fb9a41553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:58.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.457+0000 7fb9a3fff700 1 --2- 172.21.15.60:0/1304586315 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb9a405bf30 0x7fb9a41453e0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:58.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.457+0000 7fb9a3fff700 1 -- 172.21.15.60:0/1304586315 >> 172.21.15.60:0/1304586315 conn(0x7fb9a4064810 msgr2=0x7fb9a4142320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:58.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.457+0000 7fb9a3fff700 1 -- 172.21.15.60:0/1304586315 shutdown_connections 2022-06-29T16:07:58.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.457+0000 7fb9a3fff700 1 -- 172.21.15.60:0/1304586315 wait complete. 2022-06-29T16:07:58.474 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-29T16:07:58.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.581+0000 7f8370e31700 1 Processor -- start 2022-06-29T16:07:58.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.581+0000 7f8370e31700 1 -- start start 2022-06-29T16:07:58.586 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.581+0000 7f8370e31700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f836c0acbc0 0x7f836c14a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:58.586 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.581+0000 7f8370e31700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f836c0ad9d0 con 0x7f836c0b24f0 2022-06-29T16:07:58.587 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.581+0000 7f8370e31700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f836c0adc00 con 0x7f836c05cf40 2022-06-29T16:07:58.587 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.581+0000 7f8370e31700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f836c0ad460 con 0x7f836c0acbc0 2022-06-29T16:07:58.587 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.581+0000 7f836a59c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f836c0acbc0 0x7f836c14a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:58.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.581+0000 7f8369d9b700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f836c0b24f0 0x7f836c146410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33696/0 (socket says 172.21.15.60:33696) 2022-06-29T16:07:58.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.581+0000 7f836a59c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f836c0acbc0 0x7f836c14a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53842/0 (socket says 172.21.15.60:53842) 2022-06-29T16:07:58.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.581+0000 7f8369d9b700 1 -- 172.21.15.60:0/389022822 learned_addr learned my addr 172.21.15.60:0/389022822 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:58.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.581+0000 7f836a59c700 1 -- 172.21.15.60:0/389022822 >> v1:172.21.15.92:6789/0 conn(0x7f836c05cf40 legacy=0x7f836c065da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:58.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.581+0000 7f836a59c700 1 -- 172.21.15.60:0/389022822 >> v1:172.21.15.60:6789/0 conn(0x7f836c0b24f0 legacy=0x7f836c146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:58.590 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.581+0000 7f836a59c700 1 -- 172.21.15.60:0/389022822 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f836c0adea0 con 0x7f836c0acbc0 2022-06-29T16:07:58.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.581+0000 7f836a59c700 1 --2- 172.21.15.60:0/389022822 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f836c0acbc0 0x7f836c14a560 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 rx=0x7f835c0032f0 tx=0x7f835c0091d0).ready entity=mon.0 client_cookie=2bb863cefe6076fb server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:58.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.581+0000 7f836959a700 1 -- 172.21.15.60:0/389022822 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f835c009ea0 con 0x7f836c0acbc0 2022-06-29T16:07:58.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.581+0000 7f836959a700 1 -- 172.21.15.60:0/389022822 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f835c00b8a0 con 0x7f836c0acbc0 2022-06-29T16:07:58.591 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.581+0000 7f836959a700 1 -- 172.21.15.60:0/389022822 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f835c00ba10 con 0x7f836c0acbc0 2022-06-29T16:07:58.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.585+0000 7f8370e31700 1 -- 172.21.15.60:0/389022822 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f836c0acbc0 msgr2=0x7f836c14a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:58.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.585+0000 7f8370e31700 1 --2- 172.21.15.60:0/389022822 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f836c0acbc0 0x7f836c14a560 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 rx=0x7f835c0032f0 tx=0x7f835c0091d0).stop 2022-06-29T16:07:58.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.585+0000 7f8370e31700 1 -- 172.21.15.60:0/389022822 shutdown_connections 2022-06-29T16:07:58.592 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.585+0000 7f8370e31700 1 --2- 172.21.15.60:0/389022822 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f836c0acbc0 0x7f836c14a560 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:58.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.585+0000 7f8370e31700 1 -- 172.21.15.60:0/389022822 >> 172.21.15.60:0/389022822 conn(0x7f836c0b7140 msgr2=0x7f836c0b7520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:58.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.585+0000 7f8370e31700 1 -- 172.21.15.60:0/389022822 shutdown_connections 2022-06-29T16:07:58.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.585+0000 7f8370e31700 1 -- 172.21.15.60:0/389022822 wait complete. 2022-06-29T16:07:58.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.585+0000 7f8370e31700 1 Processor -- start 2022-06-29T16:07:58.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.585+0000 7f8370e31700 1 -- start start 2022-06-29T16:07:58.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.589+0000 7f8370e31700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f836c05cf40 0x7f836c1112a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:58.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.589+0000 7f8370e31700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f836c0acbc0 0x7f836c111790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:58.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.589+0000 7f8370e31700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f836c0b24f0 0x7f836c0b06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:58.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.589+0000 7f8370e31700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f836c1574f0 con 0x7f836c05cf40 2022-06-29T16:07:58.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.589+0000 7f8370e31700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f836c113e50 con 0x7f836c0b24f0 2022-06-29T16:07:58.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.589+0000 7f8370e31700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f836c111c80 con 0x7f836c0acbc0 2022-06-29T16:07:58.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.589+0000 7f836a59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f836c05cf40 0x7f836c1112a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:58.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.589+0000 7f836a59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f836c05cf40 0x7f836c1112a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53848/0 (socket says 172.21.15.60:53848) 2022-06-29T16:07:58.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.589+0000 7f836a59c700 1 -- 172.21.15.60:0/4231879868 learned_addr learned my addr 172.21.15.60:0/4231879868 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:58.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.589+0000 7f8369d9b700 1 --2- 172.21.15.60:0/4231879868 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f836c0acbc0 0x7f836c111790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:58.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.589+0000 7f836ad9d700 1 --2- 172.21.15.60:0/4231879868 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f836c0b24f0 0x7f836c0b06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:58.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.589+0000 7f836a59c700 1 -- 172.21.15.60:0/4231879868 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f836c0acbc0 msgr2=0x7f836c111790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:58.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.589+0000 7f836a59c700 1 --2- 172.21.15.60:0/4231879868 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f836c0acbc0 0x7f836c111790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:58.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.589+0000 7f836a59c700 1 -- 172.21.15.60:0/4231879868 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f836c0b24f0 msgr2=0x7f836c0b06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:58.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.589+0000 7f836a59c700 1 --2- 172.21.15.60:0/4231879868 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f836c0b24f0 0x7f836c0b06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:58.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.589+0000 7f836a59c700 1 -- 172.21.15.60:0/4231879868 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f835c009470 con 0x7f836c05cf40 2022-06-29T16:07:58.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.589+0000 7f836a59c700 1 --2- 172.21.15.60:0/4231879868 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f836c05cf40 0x7f836c1112a0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 rx=0x7f835c0032f0 tx=0x7f835c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:58.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.589+0000 7f835b7fe700 1 -- 172.21.15.60:0/4231879868 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f835c009ea0 con 0x7f836c05cf40 2022-06-29T16:07:58.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.589+0000 7f835b7fe700 1 -- 172.21.15.60:0/4231879868 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f835c01f3f0 con 0x7f836c05cf40 2022-06-29T16:07:58.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.589+0000 7f8370e31700 1 -- 172.21.15.60:0/4231879868 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f836c113bb0 con 0x7f836c05cf40 2022-06-29T16:07:58.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.589+0000 7f835b7fe700 1 -- 172.21.15.60:0/4231879868 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f835c01f750 con 0x7f836c05cf40 2022-06-29T16:07:58.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.589+0000 7f8370e31700 1 -- 172.21.15.60:0/4231879868 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f836c110750 con 0x7f836c05cf40 2022-06-29T16:07:58.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.593+0000 7f8370e31700 1 -- 172.21.15.60:0/4231879868 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f836c113a00 con 0x7f836c05cf40 2022-06-29T16:07:58.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.593+0000 7f835b7fe700 1 -- 172.21.15.60:0/4231879868 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f835c026030 con 0x7f836c05cf40 2022-06-29T16:07:58.608 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.593+0000 7f835b7fe700 1 --2- 172.21.15.60:0/4231879868 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8348053b00 0x7f8348055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:58.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.593+0000 7f835b7fe700 1 -- 172.21.15.60:0/4231879868 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f835c06cce0 con 0x7f836c05cf40 2022-06-29T16:07:58.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.593+0000 7f8369d9b700 1 --2- 172.21.15.60:0/4231879868 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8348053b00 0x7f8348055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:58.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.593+0000 7f8369d9b700 1 --2- 172.21.15.60:0/4231879868 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8348053b00 0x7f8348055fc0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7f836000d000 tx=0x7f8360007be0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:58.609 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.597+0000 7f835b7fe700 1 -- 172.21.15.60:0/4231879868 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f835c043af0 con 0x7f836c05cf40 2022-06-29T16:07:58.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.725+0000 7f8370e31700 1 -- 172.21.15.60:0/4231879868 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f836c0573d0 con 0x7f836c05cf40 2022-06-29T16:07:58.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.729+0000 7f835b7fe700 1 -- 172.21.15.60:0/4231879868 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v137) v1 ==== 100+0+10 (secure 0 0 0) 0x7f835c0150a0 con 0x7f836c05cf40 2022-06-29T16:07:58.733 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:07:58.734 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.729+0000 7f8370e31700 1 -- 172.21.15.60:0/4231879868 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8348053b00 msgr2=0x7f8348055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:58.734 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.729+0000 7f8370e31700 1 --2- 172.21.15.60:0/4231879868 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8348053b00 0x7f8348055fc0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7f836000d000 tx=0x7f8360007be0).stop 2022-06-29T16:07:58.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.729+0000 7f8370e31700 1 -- 172.21.15.60:0/4231879868 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f836c05cf40 msgr2=0x7f836c1112a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:58.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.729+0000 7f8370e31700 1 --2- 172.21.15.60:0/4231879868 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f836c05cf40 0x7f836c1112a0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 rx=0x7f835c0032f0 tx=0x7f835c016000).stop 2022-06-29T16:07:58.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.729+0000 7f8370e31700 1 -- 172.21.15.60:0/4231879868 shutdown_connections 2022-06-29T16:07:58.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.729+0000 7f8370e31700 1 --2- 172.21.15.60:0/4231879868 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8348053b00 0x7f8348055fc0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:58.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.729+0000 7f8370e31700 1 --2- 172.21.15.60:0/4231879868 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f836c0b24f0 0x7f836c0b06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:58.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.729+0000 7f8370e31700 1 --2- 172.21.15.60:0/4231879868 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f836c0acbc0 0x7f836c111790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:58.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.729+0000 7f8370e31700 1 --2- 172.21.15.60:0/4231879868 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f836c05cf40 0x7f836c1112a0 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:58.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.729+0000 7f8370e31700 1 -- 172.21.15.60:0/4231879868 >> 172.21.15.60:0/4231879868 conn(0x7f836c0b7140 msgr2=0x7f836c146d10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:58.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.729+0000 7f8370e31700 1 -- 172.21.15.60:0/4231879868 shutdown_connections 2022-06-29T16:07:58.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.729+0000 7f8370e31700 1 -- 172.21.15.60:0/4231879868 wait complete. 2022-06-29T16:07:58.745 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-06-29T16:07:58.746 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:07:58.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.845+0000 7f59a759e700 1 Processor -- start 2022-06-29T16:07:58.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.845+0000 7f59a759e700 1 -- start start 2022-06-29T16:07:58.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.845+0000 7f59a759e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59a80aca20 0x7f59a80ace00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:58.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.845+0000 7f59a759e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f59a80b3270 0x7f59a8062cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:58.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.845+0000 7f59a759e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a80adc80 con 0x7f59a8143030 2022-06-29T16:07:58.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.845+0000 7f59a759e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59a80ad740 con 0x7f59a80b3270 2022-06-29T16:07:58.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.845+0000 7f59a759e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59a80ad9e0 con 0x7f59a80aca20 2022-06-29T16:07:58.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.845+0000 7f59a6d9d700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f59a8143030 0x7f59a81461c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33714/0 (socket says 172.21.15.60:33714) 2022-06-29T16:07:58.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.845+0000 7f59a6d9d700 1 -- 172.21.15.60:0/4032913956 learned_addr learned my addr 172.21.15.60:0/4032913956 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:07:58.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.845+0000 7f599dd9b700 1 --2- 172.21.15.60:0/4032913956 >> v2:172.21.15.92:3300/0 conn(0x7f59a80b3270 0x7f59a8062cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:58.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.845+0000 7f59a659c700 1 --2- 172.21.15.60:0/4032913956 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59a80aca20 0x7f59a80ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:58.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.845+0000 7f59a5d9b700 1 -- 172.21.15.60:0/4032913956 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2571483274 0 0) 0x7f59a80adc80 con 0x7f59a8143030 2022-06-29T16:07:58.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.849+0000 7f59a5d9b700 1 -- 172.21.15.60:0/4032913956 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5988003260 con 0x7f59a8143030 2022-06-29T16:07:58.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.849+0000 7f59a5d9b700 1 -- 172.21.15.60:0/4032913956 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2522884883 0 0) 0x7f5988003260 con 0x7f59a8143030 2022-06-29T16:07:58.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.849+0000 7f59a659c700 1 -- 172.21.15.60:0/4032913956 >> v2:172.21.15.92:3300/0 conn(0x7f59a80b3270 msgr2=0x7f59a8062cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:07:58.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.849+0000 7f59a659c700 1 --2- 172.21.15.60:0/4032913956 >> v2:172.21.15.92:3300/0 conn(0x7f59a80b3270 0x7f59a8062cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:58.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.849+0000 7f59a659c700 1 -- 172.21.15.60:0/4032913956 >> v1:172.21.15.60:6789/0 conn(0x7f59a8143030 legacy=0x7f59a81461c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:58.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.849+0000 7f59a659c700 1 -- 172.21.15.60:0/4032913956 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a8142920 con 0x7f59a80aca20 2022-06-29T16:07:58.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.849+0000 7f59a659c700 1 --2- 172.21.15.60:0/4032913956 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59a80aca20 0x7f59a80ace00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f59a0001b80 tx=0x7f5998006df0).ready entity=mon.2 client_cookie=5996f41c248e883c server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:58.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.849+0000 7f59a5d9b700 1 -- 172.21.15.60:0/4032913956 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59a00019e0 con 0x7f59a80aca20 2022-06-29T16:07:58.859 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.849+0000 7f59a5d9b700 1 -- 172.21.15.60:0/4032913956 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5998000e90 con 0x7f59a80aca20 2022-06-29T16:07:58.859 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.849+0000 7f59a5d9b700 1 -- 172.21.15.60:0/4032913956 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5998012460 con 0x7f59a80aca20 2022-06-29T16:07:58.859 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.849+0000 7f59a759e700 1 -- 172.21.15.60:0/4032913956 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59a80aca20 msgr2=0x7f59a80ace00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:58.859 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.849+0000 7f59a759e700 1 --2- 172.21.15.60:0/4032913956 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59a80aca20 0x7f59a80ace00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f59a0001b80 tx=0x7f5998006df0).stop 2022-06-29T16:07:58.860 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.849+0000 7f59a759e700 1 -- 172.21.15.60:0/4032913956 shutdown_connections 2022-06-29T16:07:58.860 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.849+0000 7f59a759e700 1 --2- 172.21.15.60:0/4032913956 >> v2:172.21.15.92:3300/0 conn(0x7f59a80b3270 0x7f59a8062cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:58.861 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.849+0000 7f59a759e700 1 --2- 172.21.15.60:0/4032913956 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59a80aca20 0x7f59a80ace00 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:58.861 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.849+0000 7f59a759e700 1 -- 172.21.15.60:0/4032913956 >> 172.21.15.60:0/4032913956 conn(0x7f59a80b6f80 msgr2=0x7f59a80b7390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:58.861 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.849+0000 7f59a759e700 1 -- 172.21.15.60:0/4032913956 shutdown_connections 2022-06-29T16:07:58.861 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.849+0000 7f59a759e700 1 -- 172.21.15.60:0/4032913956 wait complete. 2022-06-29T16:07:58.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.849+0000 7f59a759e700 1 Processor -- start 2022-06-29T16:07:58.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.849+0000 7f59a759e700 1 -- start start 2022-06-29T16:07:58.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.853+0000 7f59a759e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59a80aca20 0x7f59a8151d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:58.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.853+0000 7f59a759e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59a80b3270 0x7f59a80653d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:58.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.853+0000 7f59a759e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59a8143030 0x7f59a80afea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:58.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.853+0000 7f59a759e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a80afa70 con 0x7f59a80b3270 2022-06-29T16:07:58.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.853+0000 7f59a759e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a8155d80 con 0x7f59a80aca20 2022-06-29T16:07:58.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.853+0000 7f59a759e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59a8155fe0 con 0x7f59a8143030 2022-06-29T16:07:58.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.853+0000 7f599dd9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59a80b3270 0x7f59a80653d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:58.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.853+0000 7f59a659c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59a80aca20 0x7f59a8151d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:58.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.853+0000 7f599dd9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59a80b3270 0x7f59a80653d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53864/0 (socket says 172.21.15.60:53864) 2022-06-29T16:07:58.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.853+0000 7f599dd9b700 1 -- 172.21.15.60:0/87965638 learned_addr learned my addr 172.21.15.60:0/87965638 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:07:58.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.853+0000 7f59a6d9d700 1 --2- 172.21.15.60:0/87965638 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59a8143030 0x7f59a80afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:58.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.853+0000 7f599dd9b700 1 -- 172.21.15.60:0/87965638 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59a8143030 msgr2=0x7f59a80afea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:58.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.853+0000 7f599dd9b700 1 --2- 172.21.15.60:0/87965638 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59a8143030 0x7f59a80afea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:58.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.853+0000 7f599dd9b700 1 -- 172.21.15.60:0/87965638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59a80aca20 msgr2=0x7f59a8151d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:58.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.853+0000 7f599dd9b700 1 --2- 172.21.15.60:0/87965638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59a80aca20 0x7f59a8151d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:58.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.853+0000 7f599dd9b700 1 -- 172.21.15.60:0/87965638 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f599800b180 con 0x7f59a80b3270 2022-06-29T16:07:58.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.853+0000 7f599dd9b700 1 --2- 172.21.15.60:0/87965638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59a80b3270 0x7f59a80653d0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 rx=0x7f599000c000 tx=0x7f5990006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:58.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.853+0000 7f599ffff700 1 -- 172.21.15.60:0/87965638 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5990010780 con 0x7f59a80b3270 2022-06-29T16:07:58.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.853+0000 7f599ffff700 1 -- 172.21.15.60:0/87965638 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59900108e0 con 0x7f59a80b3270 2022-06-29T16:07:58.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.853+0000 7f59a759e700 1 -- 172.21.15.60:0/87965638 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59a80af6f0 con 0x7f59a80b3270 2022-06-29T16:07:58.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.853+0000 7f599ffff700 1 -- 172.21.15.60:0/87965638 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5990010c40 con 0x7f59a80b3270 2022-06-29T16:07:58.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.853+0000 7f59a759e700 1 -- 172.21.15.60:0/87965638 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59a80ae560 con 0x7f59a80b3270 2022-06-29T16:07:58.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.853+0000 7f59a759e700 1 -- 172.21.15.60:0/87965638 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59a80573d0 con 0x7f59a80b3270 2022-06-29T16:07:58.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.857+0000 7f599ffff700 1 -- 172.21.15.60:0/87965638 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f599001c460 con 0x7f59a80b3270 2022-06-29T16:07:58.880 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.857+0000 7f599ffff700 1 --2- 172.21.15.60:0/87965638 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f598c053b00 0x7f598c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:07:58.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.857+0000 7f599ffff700 1 -- 172.21.15.60:0/87965638 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f599006c290 con 0x7f59a80b3270 2022-06-29T16:07:58.881 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.857+0000 7f59a659c700 1 --2- 172.21.15.60:0/87965638 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f598c053b00 0x7f598c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:07:58.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.857+0000 7f59a659c700 1 --2- 172.21.15.60:0/87965638 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f598c053b00 0x7f598c055fc0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 rx=0x7f599801b000 tx=0x7f5998014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:07:58.882 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:58.861+0000 7f599ffff700 1 -- 172.21.15.60:0/87965638 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5990042fc0 con 0x7f59a80b3270 2022-06-29T16:07:59.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:59.029+0000 7f59a759e700 1 -- 172.21.15.60:0/87965638 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59a80ae560 con 0x7f59a80b3270 2022-06-29T16:07:59.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:59.029+0000 7f599ffff700 1 -- 172.21.15.60:0/87965638 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2142 (secure 0 0 0) 0x7f59a80ae560 con 0x7f59a80b3270 2022-06-29T16:07:59.036 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:07:59.037 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":68,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.92:6835/2628497783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2628497783},{"type":"v1","addr":"172.21.15.92:6835","nonce":2628497783}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":67}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:07:54.174936+0000","modified":"2022-06-29T16:07:55.178863+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8463":{"gid":8463,"name":"a","rank":0,"incarnation":67,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/2222627281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2222627281},{"type":"v1","addr":"172.21.15.60:6833","nonce":2222627281}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2022-06-29T16:07:59.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:59.033+0000 7f59a759e700 1 -- 172.21.15.60:0/87965638 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f598c053b00 msgr2=0x7f598c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:59.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:59.033+0000 7f59a759e700 1 --2- 172.21.15.60:0/87965638 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f598c053b00 0x7f598c055fc0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 rx=0x7f599801b000 tx=0x7f5998014000).stop 2022-06-29T16:07:59.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:59.033+0000 7f59a759e700 1 -- 172.21.15.60:0/87965638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59a80b3270 msgr2=0x7f59a80653d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:07:59.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:59.033+0000 7f59a759e700 1 --2- 172.21.15.60:0/87965638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59a80b3270 0x7f59a80653d0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 rx=0x7f599000c000 tx=0x7f5990006df0).stop 2022-06-29T16:07:59.042 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:59.033+0000 7f59a759e700 1 -- 172.21.15.60:0/87965638 shutdown_connections 2022-06-29T16:07:59.042 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:59.033+0000 7f59a759e700 1 --2- 172.21.15.60:0/87965638 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f598c053b00 0x7f598c055fc0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:59.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:59.033+0000 7f59a759e700 1 --2- 172.21.15.60:0/87965638 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f59a8143030 0x7f59a80afea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:59.043 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:59.033+0000 7f59a759e700 1 --2- 172.21.15.60:0/87965638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59a80b3270 0x7f59a80653d0 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:59.044 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:59.033+0000 7f59a759e700 1 --2- 172.21.15.60:0/87965638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59a80aca20 0x7f59a8151d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:07:59.044 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:59.033+0000 7f59a759e700 1 -- 172.21.15.60:0/87965638 >> 172.21.15.60:0/87965638 conn(0x7f59a80b6f80 msgr2=0x7f59a80621a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:07:59.045 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:59.033+0000 7f59a759e700 1 -- 172.21.15.60:0/87965638 shutdown_connections 2022-06-29T16:07:59.045 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:07:59.033+0000 7f59a759e700 1 -- 172.21.15.60:0/87965638 wait complete. 2022-06-29T16:07:59.046 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 68 2022-06-29T16:07:59.251 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 68, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:07:54.174936+0000', 'modified': '2022-06-29T16:07:55.178863+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8463': {'gid': 8463, 'name': 'a', 'rank': 0, 'incarnation': 67, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.60:6833/2222627281', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6832', 'nonce': 2222627281}, {'type': 'v1', 'addr': '172.21.15.60:6833', 'nonce': 2222627281}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:07:59.252 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:07:59.252 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-06-29T16:07:59.370 INFO:teuthology.orchestra.run.smithi060.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":8463,"want_state":"up:active","state":"up:active","rank_uptime":5.1797482219999997,"mdsmap_epoch":68,"osdmap_epoch":137,"osdmap_epoch_barrier":137,"uptime":9.5347174429999999} 2022-06-29T16:07:59.370 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 8463, "mdsmap_epoch": 68, "osdmap_epoch": 137, "osdmap_epoch_barrier": 137, "rank_uptime": 5.179748222, "state": "up:active", "uptime": 9.534717443, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:07:59.371 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-06-29T16:07:59.371 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-06-29T16:07:59.372 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi121.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-06-29T16:07:59.372 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:59.373 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:07:59.387 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:07:59.404 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:07:59.405 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:07:59.462 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:07:59.463 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:07:59.464 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:59.465 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:07:59.470 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-06-29T16:07:59.471 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:07:59.473 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:07:59.490 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:07:59.491 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:07:59.548 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:07:59.549 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:07:59.550 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:07:59.551 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:07:59.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:2022-06-29T16:07:59.543+0000 7f9b0073d080 -1 init, newargv = 0x55bd0c6e1040 newargc=9ceph-fuse[13556]: starting ceph client 2022-06-29T16:07:59.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr: 2022-06-29T16:07:59.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[13556]: starting fuse 2022-06-29T16:07:59.749 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:07:59.753 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:07:59.757 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:07:59.760 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:07:59.764 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:08:00.659 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:08:00.682 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:08:00.682 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:08:00.694 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:08:00.698 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:08:00.699 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:00.700 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:08:00.706 INFO:teuthology.orchestra.run.smithi121.stdout:52 2022-06-29T16:08:00.706 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-06-29T16:08:00.706 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:00.707 DEBUG:teuthology.orchestra.run.smithi121:> sudo python3 -c ' 2022-06-29T16:08:00.708 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2022-06-29T16:08:00.708 DEBUG:teuthology.orchestra.run.smithi121:> import re 2022-06-29T16:08:00.709 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-06-29T16:08:00.709 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2022-06-29T16:08:00.710 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:08:00.711 DEBUG:teuthology.orchestra.run.smithi121:> def find_socket(client_name): 2022-06-29T16:08:00.713 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-06-29T16:08:00.714 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2022-06-29T16:08:00.714 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:08:00.715 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2022-06-29T16:08:00.716 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2022-06-29T16:08:00.717 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2022-06-29T16:08:00.718 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2022-06-29T16:08:00.718 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:08:00.720 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2022-06-29T16:08:00.720 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-06-29T16:08:00.722 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2022-06-29T16:08:00.722 DEBUG:teuthology.orchestra.run.smithi121:> return f 2022-06-29T16:08:00.723 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-06-29T16:08:00.724 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:08:00.725 DEBUG:teuthology.orchestra.run.smithi121:> print(find_socket("client.0")) 2022-06-29T16:08:00.725 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-06-29T16:08:00.793 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.0.13556.asok 2022-06-29T16:08:00.803 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.13556.asok 2022-06-29T16:08:00.807 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:00.807 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.13556.asok status 2022-06-29T16:08:00.901 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2022-06-29T16:08:00.902 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2022-06-29T16:08:00.902 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "f25ffac72e5a9e63860ff7deb6767a11f0559a16", 2022-06-29T16:08:00.902 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 15.2.16-304-gf25ffac7 (f25ffac72e5a9e63860ff7deb6767a11f0559a16) octopus (stable)", 2022-06-29T16:08:00.903 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "0", 2022-06-29T16:08:00.903 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2022-06-29T16:08:00.903 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-06-29T16:08:00.904 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "13556", 2022-06-29T16:08:00.904 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2022-06-29T16:08:00.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:08:00.905 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2022-06-29T16:08:00.905 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2022-06-29T16:08:00.905 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 8548, 2022-06-29T16:08:00.906 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2022-06-29T16:08:00.907 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2022-06-29T16:08:00.908 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2022-06-29T16:08:00.908 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 8548 2022-06-29T16:08:00.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:08:00.909 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-06-29T16:08:00.909 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-06-29T16:08:00.910 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-06-29T16:08:00.910 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2169263194 2022-06-29T16:08:00.911 INFO:teuthology.orchestra.run.smithi121.stdout: } 2022-06-29T16:08:00.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:08:00.911 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-06-29T16:08:00.912 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-06-29T16:08:00.912 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-06-29T16:08:00.913 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2169263194 2022-06-29T16:08:00.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:08:00.913 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.8548 172.21.15.121:0/2169263194", 2022-06-29T16:08:00.914 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "172.21.15.121:0/2169263194", 2022-06-29T16:08:00.914 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2022-06-29T16:08:00.915 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 68, 2022-06-29T16:08:00.915 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 137, 2022-06-29T16:08:00.916 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2022-06-29T16:08:00.916 INFO:teuthology.orchestra.run.smithi121.stdout: "blacklisted": false, 2022-06-29T16:08:00.916 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2022-06-29T16:08:00.917 INFO:teuthology.orchestra.run.smithi121.stdout:} 2022-06-29T16:08:00.919 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:00.920 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:08:00.921 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:08:00.931 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T16:08:00.931 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T16:08:00.932 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:00.933 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:08:00.994 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:08:01.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905fcca700 1 Processor -- start 2022-06-29T16:08:01.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905fcca700 1 -- start start 2022-06-29T16:08:01.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905fcca700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f90580b5950 0x7f9058067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:01.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905fcca700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9058063ad0 0x7f905814af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:01.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905fcca700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90580acba0 con 0x7f905805bd20 2022-06-29T16:08:01.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905fcca700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9058067220 con 0x7f90580b5950 2022-06-29T16:08:01.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905fcca700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90580b5d30 con 0x7f9058063ad0 2022-06-29T16:08:01.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905e267700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f905805bd20 0x7f905805c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33726/0 (socket says 172.21.15.60:33726) 2022-06-29T16:08:01.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905e267700 1 -- 172.21.15.60:0/1589860266 learned_addr learned my addr 172.21.15.60:0/1589860266 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:01.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905da66700 1 --2- 172.21.15.60:0/1589860266 >> v2:172.21.15.60:3300/0 conn(0x7f90580b5950 0x7f9058067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:01.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905d265700 1 --2- 172.21.15.60:0/1589860266 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9058063ad0 0x7f905814af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:01.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905ca64700 1 -- 172.21.15.60:0/1589860266 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4023626246 0 0) 0x7f90580acba0 con 0x7f905805bd20 2022-06-29T16:08:01.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905ca64700 1 -- 172.21.15.60:0/1589860266 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9040003260 con 0x7f905805bd20 2022-06-29T16:08:01.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905ca64700 1 -- 172.21.15.60:0/1589860266 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 350753787 0 0) 0x7f9040003260 con 0x7f905805bd20 2022-06-29T16:08:01.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905ca64700 1 -- 172.21.15.60:0/1589860266 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9058063ad0 msgr2=0x7f905814af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:01.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905ca64700 1 --2- 172.21.15.60:0/1589860266 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9058063ad0 0x7f905814af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:01.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905ca64700 1 -- 172.21.15.60:0/1589860266 >> v2:172.21.15.60:3300/0 conn(0x7f90580b5950 msgr2=0x7f9058067680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:01.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905ca64700 1 --2- 172.21.15.60:0/1589860266 >> v2:172.21.15.60:3300/0 conn(0x7f90580b5950 0x7f9058067680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:01.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905ca64700 1 -- 172.21.15.60:0/1589860266 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90580ace00 con 0x7f905805bd20 2022-06-29T16:08:01.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905ca64700 1 -- 172.21.15.60:0/1589860266 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f90500019e0 con 0x7f905805bd20 2022-06-29T16:08:01.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905da66700 1 --2- 172.21.15.60:0/1589860266 >> v2:172.21.15.60:3300/0 conn(0x7f90580b5950 0x7f9058067680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:08:01.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905ca64700 1 -- 172.21.15.60:0/1589860266 >> v1:172.21.15.60:6789/0 conn(0x7f905805bd20 legacy=0x7f905805c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:01.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905ca64700 1 --2- 172.21.15.60:0/1589860266 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9040003750 0x7f9040003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:01.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905ca64700 1 --2- 172.21.15.60:0/1589860266 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9058063ad0 0x7f9040004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:01.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905ca64700 1 --2- 172.21.15.60:0/1589860266 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90400049f0 0x7f9040004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:01.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905ca64700 1 -- 172.21.15.60:0/1589860266 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9058067220 con 0x7f9040003750 2022-06-29T16:08:01.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905ca64700 1 -- 172.21.15.60:0/1589860266 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90580b5d30 con 0x7f90400049f0 2022-06-29T16:08:01.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905ca64700 1 -- 172.21.15.60:0/1589860266 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f904000b750 con 0x7f9058063ad0 2022-06-29T16:08:01.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905d265700 1 --2- 172.21.15.60:0/1589860266 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9058063ad0 0x7f9040004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:01.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905da66700 1 --2- 172.21.15.60:0/1589860266 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9040003750 0x7f9040003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:01.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905e267700 1 --2- 172.21.15.60:0/1589860266 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90400049f0 0x7f9040004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:01.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905d265700 1 -- 172.21.15.60:0/1589860266 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90400049f0 msgr2=0x7f9040004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:01.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905d265700 1 --2- 172.21.15.60:0/1589860266 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90400049f0 0x7f9040004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:01.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905d265700 1 -- 172.21.15.60:0/1589860266 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9040003750 msgr2=0x7f9040003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:01.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905d265700 1 --2- 172.21.15.60:0/1589860266 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9040003750 0x7f9040003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:01.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905d265700 1 -- 172.21.15.60:0/1589860266 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9040003260 con 0x7f9058063ad0 2022-06-29T16:08:01.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905da66700 1 --2- 172.21.15.60:0/1589860266 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9040003750 0x7f9040003b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:08:01.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905d265700 1 --2- 172.21.15.60:0/1589860266 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9058063ad0 0x7f9040004500 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f905400c000 tx=0x7f9054006df0).ready entity=mon.2 client_cookie=388520ff765f05a1 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:01.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905ca64700 1 -- 172.21.15.60:0/1589860266 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f905400e830 con 0x7f9058063ad0 2022-06-29T16:08:01.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905ca64700 1 -- 172.21.15.60:0/1589860266 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f905400e990 con 0x7f9058063ad0 2022-06-29T16:08:01.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.105+0000 7f905ca64700 1 -- 172.21.15.60:0/1589860266 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f905400ecf0 con 0x7f9058063ad0 2022-06-29T16:08:01.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.109+0000 7f905fcca700 1 -- 172.21.15.60:0/1589860266 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9058063ad0 msgr2=0x7f9040004500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:01.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.109+0000 7f905fcca700 1 --2- 172.21.15.60:0/1589860266 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9058063ad0 0x7f9040004500 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f905400c000 tx=0x7f9054006df0).stop 2022-06-29T16:08:01.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.109+0000 7f905e267700 1 -- 172.21.15.60:0/1589860266 reap_dead start 2022-06-29T16:08:01.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.109+0000 7f905fcca700 1 -- 172.21.15.60:0/1589860266 shutdown_connections 2022-06-29T16:08:01.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.109+0000 7f905fcca700 1 -- 172.21.15.60:0/1589860266 >> 172.21.15.60:0/1589860266 conn(0x7f90580632c0 msgr2=0x7f90580b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:01.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.109+0000 7f905fcca700 1 -- 172.21.15.60:0/1589860266 shutdown_connections 2022-06-29T16:08:01.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.109+0000 7f905fcca700 1 -- 172.21.15.60:0/1589860266 wait complete. 2022-06-29T16:08:01.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.109+0000 7f905fcca700 1 Processor -- start 2022-06-29T16:08:01.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.109+0000 7f905fcca700 1 -- start start 2022-06-29T16:08:01.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.113+0000 7f905fcca700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9058063ad0 0x7f90580ad320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:01.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.113+0000 7f905fcca700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9058148e70 0x7f9058141de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:01.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.113+0000 7f905fcca700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9058149c00 0x7f90581422d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:01.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.113+0000 7f905fcca700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90580af890 con 0x7f9058149c00 2022-06-29T16:08:01.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.113+0000 7f905fcca700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90580afb30 con 0x7f9058063ad0 2022-06-29T16:08:01.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.113+0000 7f905fcca700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90581489d0 con 0x7f9058148e70 2022-06-29T16:08:01.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.113+0000 7f905e267700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9058149c00 0x7f90581422d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:01.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.113+0000 7f905d265700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9058148e70 0x7f9058141de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:01.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.113+0000 7f905e267700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9058149c00 0x7f90581422d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53886/0 (socket says 172.21.15.60:53886) 2022-06-29T16:08:01.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.113+0000 7f905da66700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9058063ad0 0x7f90580ad320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:01.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.113+0000 7f905e267700 1 -- 172.21.15.60:0/2581128023 learned_addr learned my addr 172.21.15.60:0/2581128023 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:01.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.113+0000 7f905e267700 1 -- 172.21.15.60:0/2581128023 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9058148e70 msgr2=0x7f9058141de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:01.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.113+0000 7f905e267700 1 --2- 172.21.15.60:0/2581128023 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9058148e70 0x7f9058141de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:01.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.113+0000 7f905e267700 1 -- 172.21.15.60:0/2581128023 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9058063ad0 msgr2=0x7f90580ad320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:01.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.113+0000 7f905e267700 1 --2- 172.21.15.60:0/2581128023 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9058063ad0 0x7f90580ad320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:01.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.113+0000 7f905e267700 1 -- 172.21.15.60:0/2581128023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90580b4a20 con 0x7f9058149c00 2022-06-29T16:08:01.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.113+0000 7f905e267700 1 --2- 172.21.15.60:0/2581128023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9058149c00 0x7f90581422d0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 rx=0x7f9050000b60 tx=0x7f90580663b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:01.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.113+0000 7f904affd700 1 -- 172.21.15.60:0/2581128023 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9050011780 con 0x7f9058149c00 2022-06-29T16:08:01.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.113+0000 7f904affd700 1 -- 172.21.15.60:0/2581128023 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90500118e0 con 0x7f9058149c00 2022-06-29T16:08:01.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.113+0000 7f905fcca700 1 -- 172.21.15.60:0/2581128023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f905814a2a0 con 0x7f9058149c00 2022-06-29T16:08:01.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.113+0000 7f904affd700 1 -- 172.21.15.60:0/2581128023 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9050011c40 con 0x7f9058149c00 2022-06-29T16:08:01.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.113+0000 7f905fcca700 1 -- 172.21.15.60:0/2581128023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90580aff70 con 0x7f9058149c00 2022-06-29T16:08:01.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.117+0000 7f904affd700 1 -- 172.21.15.60:0/2581128023 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f905001c360 con 0x7f9058149c00 2022-06-29T16:08:01.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.117+0000 7f905fcca700 1 -- 172.21.15.60:0/2581128023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90580af6d0 con 0x7f9058149c00 2022-06-29T16:08:01.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.117+0000 7f904affd700 1 --2- 172.21.15.60:0/2581128023 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9038053b00 0x7f9038055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:01.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.117+0000 7f904affd700 1 -- 172.21.15.60:0/2581128023 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f905006ba60 con 0x7f9058149c00 2022-06-29T16:08:01.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.117+0000 7f905da66700 1 --2- 172.21.15.60:0/2581128023 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9038053b00 0x7f9038055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:01.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.117+0000 7f905da66700 1 --2- 172.21.15.60:0/2581128023 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9038053b00 0x7f9038055fc0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7f904c00b000 tx=0x7f904c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:01.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.121+0000 7f904affd700 1 -- 172.21.15.60:0/2581128023 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f90500429d0 con 0x7f9058149c00 2022-06-29T16:08:01.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.257+0000 7f905fcca700 1 -- 172.21.15.60:0/2581128023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9058148b90 con 0x7f9058149c00 2022-06-29T16:08:01.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.257+0000 7f904affd700 1 -- 172.21.15.60:0/2581128023 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v137) v1 ==== 74+0+16052 (secure 0 0 0) 0x7f9050019020 con 0x7f9058149c00 2022-06-29T16:08:01.264 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:08:01.264 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":137,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:07:55.177815+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:07:52.605597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"136","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2022-06-29T16:07:53.864756+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:08:01.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.261+0000 7f905fcca700 1 -- 172.21.15.60:0/2581128023 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9038053b00 msgr2=0x7f9038055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:01.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.261+0000 7f905fcca700 1 --2- 172.21.15.60:0/2581128023 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9038053b00 0x7f9038055fc0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7f904c00b000 tx=0x7f904c006df0).stop 2022-06-29T16:08:01.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.261+0000 7f905fcca700 1 -- 172.21.15.60:0/2581128023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9058149c00 msgr2=0x7f90581422d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:01.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.261+0000 7f905fcca700 1 --2- 172.21.15.60:0/2581128023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9058149c00 0x7f90581422d0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 rx=0x7f9050000b60 tx=0x7f90580663b0).stop 2022-06-29T16:08:01.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.261+0000 7f905fcca700 1 -- 172.21.15.60:0/2581128023 shutdown_connections 2022-06-29T16:08:01.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.261+0000 7f905fcca700 1 --2- 172.21.15.60:0/2581128023 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9038053b00 0x7f9038055fc0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:01.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.261+0000 7f905fcca700 1 --2- 172.21.15.60:0/2581128023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9058149c00 0x7f90581422d0 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:01.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.261+0000 7f905fcca700 1 --2- 172.21.15.60:0/2581128023 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9058148e70 0x7f9058141de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:01.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.261+0000 7f905fcca700 1 --2- 172.21.15.60:0/2581128023 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9058063ad0 0x7f90580ad320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:01.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.261+0000 7f905fcca700 1 -- 172.21.15.60:0/2581128023 >> 172.21.15.60:0/2581128023 conn(0x7f90580632c0 msgr2=0x7f9058063eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:01.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.261+0000 7f905fcca700 1 -- 172.21.15.60:0/2581128023 shutdown_connections 2022-06-29T16:08:01.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.261+0000 7f905fcca700 1 -- 172.21.15.60:0/2581128023 wait complete. 2022-06-29T16:08:01.278 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-29T16:07:52.605597+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '136', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2022-06-29T16:07:53.864756+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '137', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-29T16:08:01.280 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:08:01.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.385+0000 7fbc5ffff700 1 Processor -- start 2022-06-29T16:08:01.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.385+0000 7fbc5ffff700 1 -- start start 2022-06-29T16:08:01.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.385+0000 7fbc5ffff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbc600acd60 0x7fbc600b2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:01.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.385+0000 7fbc5ffff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbc600ad520 0x7fbc6013fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:01.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.385+0000 7fbc5ffff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc60140020 0x7fbc60062090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:01.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.385+0000 7fbc5ffff700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc600b35d0 con 0x7fbc600ad520 2022-06-29T16:08:01.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.385+0000 7fbc5ffff700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc600633e0 con 0x7fbc600acd60 2022-06-29T16:08:01.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.385+0000 7fbc5ffff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc600ad140 con 0x7fbc60140020 2022-06-29T16:08:01.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.385+0000 7fbc567fc700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbc600ad520 0x7fbc6013fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:01.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.385+0000 7fbc567fc700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbc600ad520 0x7fbc6013fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53892/0 (socket says 172.21.15.60:53892) 2022-06-29T16:08:01.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.385+0000 7fbc567fc700 1 -- 172.21.15.60:0/880496366 learned_addr learned my addr 172.21.15.60:0/880496366 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:01.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.385+0000 7fbc5effd700 1 --2- 172.21.15.60:0/880496366 >> v2:172.21.15.92:3300/0 conn(0x7fbc600acd60 0x7fbc600b2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:01.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.385+0000 7fbc5f7fe700 1 --2- 172.21.15.60:0/880496366 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc60140020 0x7fbc60062090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:01.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.385+0000 7fbc567fc700 1 -- 172.21.15.60:0/880496366 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc60140020 msgr2=0x7fbc60062090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:01.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.385+0000 7fbc567fc700 1 --2- 172.21.15.60:0/880496366 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc60140020 0x7fbc60062090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:01.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.385+0000 7fbc567fc700 1 -- 172.21.15.60:0/880496366 >> v2:172.21.15.92:3300/0 conn(0x7fbc600acd60 msgr2=0x7fbc600b2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:01.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.385+0000 7fbc567fc700 1 --2- 172.21.15.60:0/880496366 >> v2:172.21.15.92:3300/0 conn(0x7fbc600acd60 0x7fbc600b2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:01.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.385+0000 7fbc567fc700 1 -- 172.21.15.60:0/880496366 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc60146030 con 0x7fbc600ad520 2022-06-29T16:08:01.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.385+0000 7fbc567fc700 1 --2- 172.21.15.60:0/880496366 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc600ad520 0x7fbc6013fb30 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 rx=0x7fbc48002f70 tx=0x7fbc48006eb0).ready entity=mon.0 client_cookie=c9c61f3fb545e650 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:01.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.385+0000 7fbc5e7fc700 1 -- 172.21.15.60:0/880496366 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc4800d570 con 0x7fbc600ad520 2022-06-29T16:08:01.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.385+0000 7fbc5e7fc700 1 -- 172.21.15.60:0/880496366 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc48000e90 con 0x7fbc600ad520 2022-06-29T16:08:01.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.385+0000 7fbc5e7fc700 1 -- 172.21.15.60:0/880496366 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc4800d900 con 0x7fbc600ad520 2022-06-29T16:08:01.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.389+0000 7fbc5ffff700 1 -- 172.21.15.60:0/880496366 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc600ad520 msgr2=0x7fbc6013fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:01.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.389+0000 7fbc5ffff700 1 --2- 172.21.15.60:0/880496366 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc600ad520 0x7fbc6013fb30 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 rx=0x7fbc48002f70 tx=0x7fbc48006eb0).stop 2022-06-29T16:08:01.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.389+0000 7fbc5ffff700 1 -- 172.21.15.60:0/880496366 shutdown_connections 2022-06-29T16:08:01.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.389+0000 7fbc5ffff700 1 --2- 172.21.15.60:0/880496366 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc60140020 0x7fbc60062090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:01.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.389+0000 7fbc5ffff700 1 --2- 172.21.15.60:0/880496366 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc600ad520 0x7fbc6013fb30 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:01.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.389+0000 7fbc5ffff700 1 --2- 172.21.15.60:0/880496366 >> v2:172.21.15.92:3300/0 conn(0x7fbc600acd60 0x7fbc600b2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:01.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.389+0000 7fbc5ffff700 1 -- 172.21.15.60:0/880496366 >> 172.21.15.60:0/880496366 conn(0x7fbc600b6fb0 msgr2=0x7fbc600b7390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:01.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.389+0000 7fbc5ffff700 1 -- 172.21.15.60:0/880496366 shutdown_connections 2022-06-29T16:08:01.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.389+0000 7fbc5ffff700 1 -- 172.21.15.60:0/880496366 wait complete. 2022-06-29T16:08:01.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.389+0000 7fbc5ffff700 1 Processor -- start 2022-06-29T16:08:01.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.389+0000 7fbc5ffff700 1 -- start start 2022-06-29T16:08:01.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.389+0000 7fbc5ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc600acd60 0x7fbc60141980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:01.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.389+0000 7fbc5ffff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc600ad520 0x7fbc6013cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:01.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.389+0000 7fbc5ffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc60140020 0x7fbc6013d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:01.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.389+0000 7fbc5ffff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc60066360 con 0x7fbc600acd60 2022-06-29T16:08:01.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.389+0000 7fbc5ffff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc600af5e0 con 0x7fbc60140020 2022-06-29T16:08:01.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.389+0000 7fbc5ffff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc60066f40 con 0x7fbc600ad520 2022-06-29T16:08:01.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.389+0000 7fbc5effd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc600acd60 0x7fbc60141980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:01.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.389+0000 7fbc5effd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc600acd60 0x7fbc60141980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53896/0 (socket says 172.21.15.60:53896) 2022-06-29T16:08:01.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.389+0000 7fbc5effd700 1 -- 172.21.15.60:0/1017404064 learned_addr learned my addr 172.21.15.60:0/1017404064 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:01.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.389+0000 7fbc567fc700 1 --2- 172.21.15.60:0/1017404064 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc600ad520 0x7fbc6013cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:01.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.389+0000 7fbc5f7fe700 1 --2- 172.21.15.60:0/1017404064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc60140020 0x7fbc6013d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:01.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.389+0000 7fbc5effd700 1 -- 172.21.15.60:0/1017404064 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc600ad520 msgr2=0x7fbc6013cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:01.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.389+0000 7fbc5effd700 1 --2- 172.21.15.60:0/1017404064 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc600ad520 0x7fbc6013cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:01.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.389+0000 7fbc5effd700 1 -- 172.21.15.60:0/1017404064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc60140020 msgr2=0x7fbc6013d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:01.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.389+0000 7fbc5effd700 1 --2- 172.21.15.60:0/1017404064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc60140020 0x7fbc6013d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:01.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.389+0000 7fbc5effd700 1 -- 172.21.15.60:0/1017404064 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc4800b1e0 con 0x7fbc600acd60 2022-06-29T16:08:01.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.393+0000 7fbc5f7fe700 1 --2- 172.21.15.60:0/1017404064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc60140020 0x7fbc6013d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:08:01.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.393+0000 7fbc5effd700 1 --2- 172.21.15.60:0/1017404064 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc600acd60 0x7fbc60141980 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 rx=0x7fbc5000c000 tx=0x7fbc50006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:01.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.393+0000 7fbc5cff9700 1 -- 172.21.15.60:0/1017404064 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc50010780 con 0x7fbc600acd60 2022-06-29T16:08:01.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.393+0000 7fbc5cff9700 1 -- 172.21.15.60:0/1017404064 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc500108e0 con 0x7fbc600acd60 2022-06-29T16:08:01.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.393+0000 7fbc5cff9700 1 -- 172.21.15.60:0/1017404064 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc50010c40 con 0x7fbc600acd60 2022-06-29T16:08:01.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.393+0000 7fbc5ffff700 1 -- 172.21.15.60:0/1017404064 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc60067170 con 0x7fbc600acd60 2022-06-29T16:08:01.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.393+0000 7fbc5ffff700 1 -- 172.21.15.60:0/1017404064 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc60065ec0 con 0x7fbc600acd60 2022-06-29T16:08:01.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.393+0000 7fbc5ffff700 1 -- 172.21.15.60:0/1017404064 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc600573d0 con 0x7fbc600acd60 2022-06-29T16:08:01.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.393+0000 7fbc5cff9700 1 -- 172.21.15.60:0/1017404064 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fbc5001c460 con 0x7fbc600acd60 2022-06-29T16:08:01.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.393+0000 7fbc5cff9700 1 --2- 172.21.15.60:0/1017404064 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbc44053aa0 0x7fbc44055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:01.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.393+0000 7fbc5cff9700 1 -- 172.21.15.60:0/1017404064 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbc5006c2b0 con 0x7fbc600acd60 2022-06-29T16:08:01.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.397+0000 7fbc567fc700 1 --2- 172.21.15.60:0/1017404064 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbc44053aa0 0x7fbc44055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:01.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.397+0000 7fbc567fc700 1 --2- 172.21.15.60:0/1017404064 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbc44053aa0 0x7fbc44055f60 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0x7fbc4801c000 tx=0x7fbc48015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:01.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.401+0000 7fbc5cff9700 1 -- 172.21.15.60:0/1017404064 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbc50043170 con 0x7fbc600acd60 2022-06-29T16:08:01.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.537+0000 7fbc5ffff700 1 -- 172.21.15.60:0/1017404064 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fbc60067b20 con 0x7fbc600acd60 2022-06-29T16:08:01.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.537+0000 7fbc5cff9700 1 -- 172.21.15.60:0/1017404064 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v137) v1 ==== 110+0+10 (secure 0 0 0) 0x7fbc50046790 con 0x7fbc600acd60 2022-06-29T16:08:01.542 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:08:01.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.537+0000 7fbc5ffff700 1 -- 172.21.15.60:0/1017404064 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbc44053aa0 msgr2=0x7fbc44055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:01.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.537+0000 7fbc5ffff700 1 --2- 172.21.15.60:0/1017404064 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbc44053aa0 0x7fbc44055f60 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0x7fbc4801c000 tx=0x7fbc48015000).stop 2022-06-29T16:08:01.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.537+0000 7fbc5ffff700 1 -- 172.21.15.60:0/1017404064 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc600acd60 msgr2=0x7fbc60141980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:01.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.537+0000 7fbc5ffff700 1 --2- 172.21.15.60:0/1017404064 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc600acd60 0x7fbc60141980 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 rx=0x7fbc5000c000 tx=0x7fbc50006df0).stop 2022-06-29T16:08:01.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.537+0000 7fbc5ffff700 1 -- 172.21.15.60:0/1017404064 shutdown_connections 2022-06-29T16:08:01.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.537+0000 7fbc5ffff700 1 --2- 172.21.15.60:0/1017404064 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbc44053aa0 0x7fbc44055f60 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:01.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.537+0000 7fbc5ffff700 1 --2- 172.21.15.60:0/1017404064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc60140020 0x7fbc6013d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:01.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.537+0000 7fbc5ffff700 1 --2- 172.21.15.60:0/1017404064 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbc600ad520 0x7fbc6013cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:01.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.537+0000 7fbc5ffff700 1 --2- 172.21.15.60:0/1017404064 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbc600acd60 0x7fbc60141980 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:01.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.537+0000 7fbc5ffff700 1 -- 172.21.15.60:0/1017404064 >> 172.21.15.60:0/1017404064 conn(0x7fbc600b6fb0 msgr2=0x7fbc6005bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:01.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.537+0000 7fbc5ffff700 1 -- 172.21.15.60:0/1017404064 shutdown_connections 2022-06-29T16:08:01.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.537+0000 7fbc5ffff700 1 -- 172.21.15.60:0/1017404064 wait complete. 2022-06-29T16:08:01.555 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:08:01.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.657+0000 7efdf6d87700 1 Processor -- start 2022-06-29T16:08:01.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.657+0000 7efdf6d87700 1 -- start start 2022-06-29T16:08:01.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.661+0000 7efdf6d87700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efdf0060650 0x7efdf01438a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:01.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.661+0000 7efdf6d87700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdf013f690 0x7efdf00b2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:01.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.661+0000 7efdf6d87700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdf0142d60 con 0x7efdf00b28b0 2022-06-29T16:08:01.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.661+0000 7efdf6d87700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdf013fb20 con 0x7efdf0060650 2022-06-29T16:08:01.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.661+0000 7efdf6d87700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf00b3180 con 0x7efdf013f690 2022-06-29T16:08:01.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.661+0000 7efdf5584700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7efdf0060650 0x7efdf01438a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:01.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.661+0000 7efdf5d85700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7efdf00b28b0 0x7efdf0143190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33758/0 (socket says 172.21.15.60:33758) 2022-06-29T16:08:01.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.661+0000 7efdf5d85700 1 -- 172.21.15.60:0/2571133173 learned_addr learned my addr 172.21.15.60:0/2571133173 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:01.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.661+0000 7efdf6586700 1 --2- 172.21.15.60:0/2571133173 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdf013f690 0x7efdf00b2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:01.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.661+0000 7efdf5584700 1 -- 172.21.15.60:0/2571133173 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdf013f690 msgr2=0x7efdf00b2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:01.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.661+0000 7efdf5584700 1 --2- 172.21.15.60:0/2571133173 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdf013f690 0x7efdf00b2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:01.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.661+0000 7efdf5584700 1 -- 172.21.15.60:0/2571133173 >> v1:172.21.15.60:6789/0 conn(0x7efdf00b28b0 legacy=0x7efdf0143190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:01.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.661+0000 7efdf4d83700 1 -- 172.21.15.60:0/2571133173 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4183434169 0 0) 0x7efdf0142d60 con 0x7efdf00b28b0 2022-06-29T16:08:01.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.661+0000 7efdf5584700 1 -- 172.21.15.60:0/2571133173 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf00654f0 con 0x7efdf0060650 2022-06-29T16:08:01.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.661+0000 7efdf5584700 1 --2- 172.21.15.60:0/2571133173 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdf0060650 0x7efdf01438a0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 rx=0x7efde0002f70 tx=0x7efde0006df0).ready entity=mon.0 client_cookie=5764636521e2ca6f server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:01.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.661+0000 7efdf4d83700 1 -- 172.21.15.60:0/2571133173 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efde000d570 con 0x7efdf0060650 2022-06-29T16:08:01.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.661+0000 7efdf4d83700 1 -- 172.21.15.60:0/2571133173 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde0000e90 con 0x7efdf0060650 2022-06-29T16:08:01.672 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.661+0000 7efdf4d83700 1 -- 172.21.15.60:0/2571133173 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efde000d8c0 con 0x7efdf0060650 2022-06-29T16:08:01.677 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.661+0000 7efdf6d87700 1 -- 172.21.15.60:0/2571133173 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdf0060650 msgr2=0x7efdf01438a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:01.677 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.661+0000 7efdf6d87700 1 --2- 172.21.15.60:0/2571133173 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdf0060650 0x7efdf01438a0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 rx=0x7efde0002f70 tx=0x7efde0006df0).stop 2022-06-29T16:08:01.678 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.665+0000 7efdf6d87700 1 -- 172.21.15.60:0/2571133173 shutdown_connections 2022-06-29T16:08:01.678 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.665+0000 7efdf6d87700 1 --2- 172.21.15.60:0/2571133173 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdf013f690 0x7efdf00b2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:01.679 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.665+0000 7efdf6d87700 1 --2- 172.21.15.60:0/2571133173 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdf0060650 0x7efdf01438a0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:01.679 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.665+0000 7efdf6d87700 1 -- 172.21.15.60:0/2571133173 >> 172.21.15.60:0/2571133173 conn(0x7efdf00637c0 msgr2=0x7efdf0063ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:01.680 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.665+0000 7efdf6d87700 1 -- 172.21.15.60:0/2571133173 shutdown_connections 2022-06-29T16:08:01.680 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.665+0000 7efdf6d87700 1 -- 172.21.15.60:0/2571133173 wait complete. 2022-06-29T16:08:01.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.665+0000 7efdf6d87700 1 Processor -- start 2022-06-29T16:08:01.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.665+0000 7efdf6d87700 1 -- start start 2022-06-29T16:08:01.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.665+0000 7efdf6d87700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efdf0060650 0x7efdf0147f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:01.682 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.665+0000 7efdf6d87700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdf00b28b0 0x7efdf00adde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:01.682 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.665+0000 7efdf6d87700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdf013f690 0x7efdf0151a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:01.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.665+0000 7efdf6d87700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf0152d50 con 0x7efdf00b28b0 2022-06-29T16:08:01.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.665+0000 7efdf6d87700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf00ad6a0 con 0x7efdf0060650 2022-06-29T16:08:01.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.665+0000 7efdf6d87700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf00ad940 con 0x7efdf013f690 2022-06-29T16:08:01.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.665+0000 7efdf5584700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdf00b28b0 0x7efdf00adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:01.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.665+0000 7efdf5584700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdf00b28b0 0x7efdf00adde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53912/0 (socket says 172.21.15.60:53912) 2022-06-29T16:08:01.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.665+0000 7efdf5584700 1 -- 172.21.15.60:0/26904964 learned_addr learned my addr 172.21.15.60:0/26904964 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:01.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.665+0000 7efdf6586700 1 --2- 172.21.15.60:0/26904964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdf013f690 0x7efdf0151a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:01.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.665+0000 7efdf5d85700 1 --2- 172.21.15.60:0/26904964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efdf0060650 0x7efdf0147f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:01.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.669+0000 7efdf5584700 1 -- 172.21.15.60:0/26904964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdf013f690 msgr2=0x7efdf0151a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:01.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.669+0000 7efdf5584700 1 --2- 172.21.15.60:0/26904964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdf013f690 0x7efdf0151a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:01.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.669+0000 7efdf5584700 1 -- 172.21.15.60:0/26904964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efdf0060650 msgr2=0x7efdf0147f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:01.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.669+0000 7efdf5584700 1 --2- 172.21.15.60:0/26904964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efdf0060650 0x7efdf0147f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:01.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.669+0000 7efdf5584700 1 -- 172.21.15.60:0/26904964 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde000b180 con 0x7efdf00b28b0 2022-06-29T16:08:01.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.669+0000 7efdf5584700 1 --2- 172.21.15.60:0/26904964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdf00b28b0 0x7efdf00adde0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 rx=0x7efde0002f70 tx=0x7efde0014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:01.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.669+0000 7efddeffd700 1 -- 172.21.15.60:0/26904964 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efde000dea0 con 0x7efdf00b28b0 2022-06-29T16:08:01.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.669+0000 7efddeffd700 1 -- 172.21.15.60:0/26904964 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde0013de0 con 0x7efdf00b28b0 2022-06-29T16:08:01.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.669+0000 7efdf6d87700 1 -- 172.21.15.60:0/26904964 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdf00ae2d0 con 0x7efdf00b28b0 2022-06-29T16:08:01.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.669+0000 7efddeffd700 1 -- 172.21.15.60:0/26904964 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efde001e5f0 con 0x7efdf00b28b0 2022-06-29T16:08:01.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.669+0000 7efdf6d87700 1 -- 172.21.15.60:0/26904964 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdf0148d30 con 0x7efdf00b28b0 2022-06-29T16:08:01.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.669+0000 7efdf6d87700 1 -- 172.21.15.60:0/26904964 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdf00adb00 con 0x7efdf00b28b0 2022-06-29T16:08:01.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.669+0000 7efddeffd700 1 -- 172.21.15.60:0/26904964 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7efde0025030 con 0x7efdf00b28b0 2022-06-29T16:08:01.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.669+0000 7efddeffd700 1 --2- 172.21.15.60:0/26904964 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efdd0053950 0x7efdd0055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:01.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.673+0000 7efddeffd700 1 -- 172.21.15.60:0/26904964 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6073+0+0 (secure 0 0 0) 0x7efde006cbc0 con 0x7efdf00b28b0 2022-06-29T16:08:01.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.673+0000 7efdf5d85700 1 --2- 172.21.15.60:0/26904964 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efdd0053950 0x7efdd0055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:01.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.673+0000 7efdf5d85700 1 --2- 172.21.15.60:0/26904964 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efdd0053950 0x7efdd0055e10 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0x7efdec00d000 tx=0x7efdec007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:01.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.677+0000 7efddeffd700 1 -- 172.21.15.60:0/26904964 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efde0043ae0 con 0x7efdf00b28b0 2022-06-29T16:08:01.807 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.801+0000 7efdf6d87700 1 -- 172.21.15.60:0/26904964 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7efdf00573d0 con 0x7efdf00b28b0 2022-06-29T16:08:01.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.801+0000 7efddeffd700 1 -- 172.21.15.60:0/26904964 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v137) v1 ==== 92+0+10 (secure 0 0 0) 0x7efde0023070 con 0x7efdf00b28b0 2022-06-29T16:08:01.808 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:08:01.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.805+0000 7efdf6d87700 1 -- 172.21.15.60:0/26904964 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efdd0053950 msgr2=0x7efdd0055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:01.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.805+0000 7efdf6d87700 1 --2- 172.21.15.60:0/26904964 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efdd0053950 0x7efdd0055e10 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0x7efdec00d000 tx=0x7efdec007a00).stop 2022-06-29T16:08:01.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.805+0000 7efdf6d87700 1 -- 172.21.15.60:0/26904964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdf00b28b0 msgr2=0x7efdf00adde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:01.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.805+0000 7efdf6d87700 1 --2- 172.21.15.60:0/26904964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdf00b28b0 0x7efdf00adde0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 rx=0x7efde0002f70 tx=0x7efde0014000).stop 2022-06-29T16:08:01.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.805+0000 7efdf6d87700 1 -- 172.21.15.60:0/26904964 shutdown_connections 2022-06-29T16:08:01.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.805+0000 7efdf6d87700 1 --2- 172.21.15.60:0/26904964 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7efdd0053950 0x7efdd0055e10 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:01.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.805+0000 7efdf6d87700 1 --2- 172.21.15.60:0/26904964 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7efdf013f690 0x7efdf0151a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:01.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.805+0000 7efdf6d87700 1 --2- 172.21.15.60:0/26904964 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdf00b28b0 0x7efdf00adde0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:01.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.805+0000 7efdf6d87700 1 --2- 172.21.15.60:0/26904964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efdf0060650 0x7efdf0147f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:01.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.805+0000 7efdf6d87700 1 -- 172.21.15.60:0/26904964 >> 172.21.15.60:0/26904964 conn(0x7efdf00637c0 msgr2=0x7efdf01430b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:01.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.805+0000 7efdf6d87700 1 -- 172.21.15.60:0/26904964 shutdown_connections 2022-06-29T16:08:01.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.805+0000 7efdf6d87700 1 -- 172.21.15.60:0/26904964 wait complete. 2022-06-29T16:08:01.826 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-29T16:08:01.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.933+0000 7f83ba8d2700 1 Processor -- start 2022-06-29T16:08:01.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.933+0000 7f83ba8d2700 1 -- start start 2022-06-29T16:08:01.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.933+0000 7f83ba8d2700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f83b40ad990 0x7f83b4144900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:01.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.933+0000 7f83ba8d2700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b4143cf0 con 0x7f83b40657d0 2022-06-29T16:08:01.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.933+0000 7f83ba8d2700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b40ad140 con 0x7f83b40acd60 2022-06-29T16:08:01.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.933+0000 7f83ba8d2700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b4064a70 con 0x7f83b40ad990 2022-06-29T16:08:01.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.933+0000 7f83b90cf700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f83b40ad990 0x7f83b4144900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:01.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.933+0000 7f83ba0d1700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f83b40657d0 0x7f83b40b1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33776/0 (socket says 172.21.15.60:33776) 2022-06-29T16:08:01.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.933+0000 7f83ba0d1700 1 -- 172.21.15.60:0/915148543 learned_addr learned my addr 172.21.15.60:0/915148543 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:01.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.933+0000 7f83b90cf700 1 -- 172.21.15.60:0/915148543 >> v1:172.21.15.92:6789/0 conn(0x7f83b40acd60 legacy=0x7f83b41296f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:01.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.933+0000 7f83b90cf700 1 -- 172.21.15.60:0/915148543 >> v1:172.21.15.60:6789/0 conn(0x7f83b40657d0 legacy=0x7f83b40b1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:01.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.933+0000 7f83b88ce700 1 -- 172.21.15.60:0/915148543 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 966634625 0 0) 0x7f83b4143cf0 con 0x7f83b40657d0 2022-06-29T16:08:01.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.933+0000 7f83b90cf700 1 -- 172.21.15.60:0/915148543 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b4143f20 con 0x7f83b40ad990 2022-06-29T16:08:01.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.933+0000 7f83b90cf700 1 --2- 172.21.15.60:0/915148543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83b40ad990 0x7f83b4144900 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 rx=0x7f83a4002f70 tx=0x7f83a4006df0).ready entity=mon.0 client_cookie=a065ca6674016533 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:01.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.933+0000 7f83b88ce700 1 -- 172.21.15.60:0/915148543 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f83a400d610 con 0x7f83b40ad990 2022-06-29T16:08:01.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.933+0000 7f83b88ce700 1 -- 172.21.15.60:0/915148543 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a4000e90 con 0x7f83b40ad990 2022-06-29T16:08:01.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.933+0000 7f83b88ce700 1 -- 172.21.15.60:0/915148543 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f83a400d960 con 0x7f83b40ad990 2022-06-29T16:08:01.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.933+0000 7f83ba8d2700 1 -- 172.21.15.60:0/915148543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83b40ad990 msgr2=0x7f83b4144900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:01.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.933+0000 7f83ba8d2700 1 --2- 172.21.15.60:0/915148543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83b40ad990 0x7f83b4144900 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 rx=0x7f83a4002f70 tx=0x7f83a4006df0).stop 2022-06-29T16:08:01.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.933+0000 7f83ba8d2700 1 -- 172.21.15.60:0/915148543 shutdown_connections 2022-06-29T16:08:01.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.933+0000 7f83ba8d2700 1 --2- 172.21.15.60:0/915148543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83b40ad990 0x7f83b4144900 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:01.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.933+0000 7f83ba8d2700 1 -- 172.21.15.60:0/915148543 >> 172.21.15.60:0/915148543 conn(0x7f83b40b6f80 msgr2=0x7f83b40b7390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:01.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.933+0000 7f83ba8d2700 1 -- 172.21.15.60:0/915148543 shutdown_connections 2022-06-29T16:08:01.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.933+0000 7f83ba8d2700 1 -- 172.21.15.60:0/915148543 wait complete. 2022-06-29T16:08:01.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.937+0000 7f83ba8d2700 1 Processor -- start 2022-06-29T16:08:01.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.937+0000 7f83ba8d2700 1 -- start start 2022-06-29T16:08:01.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.937+0000 7f83ba8d2700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83b40657d0 0x7f83b40afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:01.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.937+0000 7f83ba8d2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83b40acd60 0x7f83b4154f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:01.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.937+0000 7f83ba8d2700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83b40ad990 0x7f83b41553f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:01.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.937+0000 7f83ba8d2700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b4131650 con 0x7f83b40acd60 2022-06-29T16:08:01.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.937+0000 7f83ba8d2700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b4130c70 con 0x7f83b40657d0 2022-06-29T16:08:01.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.937+0000 7f83ba8d2700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b4130f10 con 0x7f83b40ad990 2022-06-29T16:08:01.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.937+0000 7f83b90cf700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83b40acd60 0x7f83b4154f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:01.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.937+0000 7f83b90cf700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83b40acd60 0x7f83b4154f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53926/0 (socket says 172.21.15.60:53926) 2022-06-29T16:08:01.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.937+0000 7f83b90cf700 1 -- 172.21.15.60:0/4057417650 learned_addr learned my addr 172.21.15.60:0/4057417650 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:01.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.937+0000 7f83b98d0700 1 --2- 172.21.15.60:0/4057417650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83b40657d0 0x7f83b40afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:01.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.937+0000 7f83ba0d1700 1 --2- 172.21.15.60:0/4057417650 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83b40ad990 0x7f83b41553f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:01.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.937+0000 7f83b90cf700 1 -- 172.21.15.60:0/4057417650 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83b40ad990 msgr2=0x7f83b41553f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:01.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.937+0000 7f83b90cf700 1 --2- 172.21.15.60:0/4057417650 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83b40ad990 0x7f83b41553f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:01.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.937+0000 7f83b90cf700 1 -- 172.21.15.60:0/4057417650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83b40657d0 msgr2=0x7f83b40afdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:01.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.937+0000 7f83b90cf700 1 --2- 172.21.15.60:0/4057417650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83b40657d0 0x7f83b40afdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:01.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.937+0000 7f83b90cf700 1 -- 172.21.15.60:0/4057417650 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a400b180 con 0x7f83b40acd60 2022-06-29T16:08:01.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.937+0000 7f83ba0d1700 1 --2- 172.21.15.60:0/4057417650 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83b40ad990 0x7f83b41553f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:08:01.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.941+0000 7f83b90cf700 1 --2- 172.21.15.60:0/4057417650 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83b40acd60 0x7f83b4154f00 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 rx=0x7f83a4002f70 tx=0x7f83a4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:01.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.941+0000 7f83aaffd700 1 -- 172.21.15.60:0/4057417650 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f83a400d290 con 0x7f83b40acd60 2022-06-29T16:08:01.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.941+0000 7f83aaffd700 1 -- 172.21.15.60:0/4057417650 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a4011c30 con 0x7f83b40acd60 2022-06-29T16:08:01.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.941+0000 7f83ba8d2700 1 -- 172.21.15.60:0/4057417650 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b41311b0 con 0x7f83b40acd60 2022-06-29T16:08:01.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.941+0000 7f83aaffd700 1 -- 172.21.15.60:0/4057417650 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f83a401e460 con 0x7f83b40acd60 2022-06-29T16:08:01.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.941+0000 7f83ba8d2700 1 -- 172.21.15.60:0/4057417650 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b4130840 con 0x7f83b40acd60 2022-06-29T16:08:01.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.941+0000 7f83ba8d2700 1 -- 172.21.15.60:0/4057417650 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b41441c0 con 0x7f83b40acd60 2022-06-29T16:08:01.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.941+0000 7f83aaffd700 1 -- 172.21.15.60:0/4057417650 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f83a4025030 con 0x7f83b40acd60 2022-06-29T16:08:01.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.941+0000 7f83aaffd700 1 --2- 172.21.15.60:0/4057417650 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f839c053a70 0x7f839c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:01.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.945+0000 7f83aaffd700 1 -- 172.21.15.60:0/4057417650 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f83a406d360 con 0x7f83b40acd60 2022-06-29T16:08:01.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.945+0000 7f83b98d0700 1 --2- 172.21.15.60:0/4057417650 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f839c053a70 0x7f839c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:01.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.945+0000 7f83b98d0700 1 --2- 172.21.15.60:0/4057417650 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f839c053a70 0x7f839c055f30 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7f83b000d000 tx=0x7f83b0007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:01.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:01.953+0000 7f83aaffd700 1 -- 172.21.15.60:0/4057417650 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f83a4044200 con 0x7f83b40acd60 2022-06-29T16:08:02.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.077+0000 7f83ba8d2700 1 -- 172.21.15.60:0/4057417650 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f83b4131340 con 0x7f83b40acd60 2022-06-29T16:08:02.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.077+0000 7f83aaffd700 1 -- 172.21.15.60:0/4057417650 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v137) v1 ==== 104+0+10 (secure 0 0 0) 0x7f83a4047820 con 0x7f83b40acd60 2022-06-29T16:08:02.083 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:08:02.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.081+0000 7f83ba8d2700 1 -- 172.21.15.60:0/4057417650 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f839c053a70 msgr2=0x7f839c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:02.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.081+0000 7f83ba8d2700 1 --2- 172.21.15.60:0/4057417650 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f839c053a70 0x7f839c055f30 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7f83b000d000 tx=0x7f83b0007b90).stop 2022-06-29T16:08:02.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.081+0000 7f83ba8d2700 1 -- 172.21.15.60:0/4057417650 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83b40acd60 msgr2=0x7f83b4154f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:02.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.081+0000 7f83ba8d2700 1 --2- 172.21.15.60:0/4057417650 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83b40acd60 0x7f83b4154f00 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 rx=0x7f83a4002f70 tx=0x7f83a4015000).stop 2022-06-29T16:08:02.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.081+0000 7f83ba8d2700 1 -- 172.21.15.60:0/4057417650 shutdown_connections 2022-06-29T16:08:02.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.081+0000 7f83ba8d2700 1 --2- 172.21.15.60:0/4057417650 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f839c053a70 0x7f839c055f30 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:02.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.081+0000 7f83ba8d2700 1 --2- 172.21.15.60:0/4057417650 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83b40ad990 0x7f83b41553f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:02.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.081+0000 7f83ba8d2700 1 --2- 172.21.15.60:0/4057417650 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83b40acd60 0x7f83b4154f00 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:02.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.081+0000 7f83ba8d2700 1 --2- 172.21.15.60:0/4057417650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83b40657d0 0x7f83b40afdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:02.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.081+0000 7f83ba8d2700 1 -- 172.21.15.60:0/4057417650 >> 172.21.15.60:0/4057417650 conn(0x7f83b40b6f80 msgr2=0x7f83b4127d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:02.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.081+0000 7f83ba8d2700 1 -- 172.21.15.60:0/4057417650 shutdown_connections 2022-06-29T16:08:02.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.081+0000 7f83ba8d2700 1 -- 172.21.15.60:0/4057417650 wait complete. 2022-06-29T16:08:02.096 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-29T16:08:02.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.201+0000 7fcf9d2a2700 1 Processor -- start 2022-06-29T16:08:02.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.201+0000 7fcf9d2a2700 1 -- start start 2022-06-29T16:08:02.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.201+0000 7fcf9d2a2700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf9805d070 0x7fcf9814aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:02.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.201+0000 7fcf9d2a2700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf9805d450 con 0x7fcf980b2130 2022-06-29T16:08:02.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.201+0000 7fcf9d2a2700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf980ad840 con 0x7fcf9814af80 2022-06-29T16:08:02.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.201+0000 7fcf9d2a2700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf980adf90 con 0x7fcf9805d070 2022-06-29T16:08:02.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.201+0000 7fcf977fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcf980b2130 0x7fcf980aca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33786/0 (socket says 172.21.15.60:33786) 2022-06-29T16:08:02.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.201+0000 7fcf977fe700 1 -- 172.21.15.60:0/176789909 learned_addr learned my addr 172.21.15.60:0/176789909 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:02.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.201+0000 7fcf97fff700 1 --2- 172.21.15.60:0/176789909 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf9805d070 0x7fcf9814aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:02.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.201+0000 7fcf96ffd700 1 -- 172.21.15.60:0/176789909 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2379478165 0 0) 0x7fcf9805d450 con 0x7fcf980b2130 2022-06-29T16:08:02.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.201+0000 7fcf96ffd700 1 -- 172.21.15.60:0/176789909 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf84003260 con 0x7fcf980b2130 2022-06-29T16:08:02.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.201+0000 7fcf96ffd700 1 -- 172.21.15.60:0/176789909 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1960529364 0 0) 0x7fcf980ad840 con 0x7fcf9814af80 2022-06-29T16:08:02.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.201+0000 7fcf96ffd700 1 -- 172.21.15.60:0/176789909 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf9805d450 con 0x7fcf9814af80 2022-06-29T16:08:02.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.201+0000 7fcf97fff700 1 -- 172.21.15.60:0/176789909 >> v1:172.21.15.92:6789/0 conn(0x7fcf9814af80 legacy=0x7fcf980ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:02.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.201+0000 7fcf97fff700 1 -- 172.21.15.60:0/176789909 >> v1:172.21.15.60:6789/0 conn(0x7fcf980b2130 legacy=0x7fcf980aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:02.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.201+0000 7fcf97fff700 1 -- 172.21.15.60:0/176789909 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf980b25d0 con 0x7fcf9805d070 2022-06-29T16:08:02.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.201+0000 7fcf97fff700 1 --2- 172.21.15.60:0/176789909 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf9805d070 0x7fcf9814aa90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fcf8c002f70 tx=0x7fcf8c006df0).ready entity=mon.2 client_cookie=e3b3b73cc8baa8b0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:02.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.205+0000 7fcf96ffd700 1 -- 172.21.15.60:0/176789909 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf8c00d5c0 con 0x7fcf9805d070 2022-06-29T16:08:02.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.205+0000 7fcf96ffd700 1 -- 172.21.15.60:0/176789909 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf8c000e90 con 0x7fcf9805d070 2022-06-29T16:08:02.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.205+0000 7fcf96ffd700 1 -- 172.21.15.60:0/176789909 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf8c00d910 con 0x7fcf9805d070 2022-06-29T16:08:02.216 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.205+0000 7fcf9d2a2700 1 -- 172.21.15.60:0/176789909 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf9805d070 msgr2=0x7fcf9814aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:02.216 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.205+0000 7fcf9d2a2700 1 --2- 172.21.15.60:0/176789909 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf9805d070 0x7fcf9814aa90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fcf8c002f70 tx=0x7fcf8c006df0).stop 2022-06-29T16:08:02.216 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.205+0000 7fcf9d2a2700 1 -- 172.21.15.60:0/176789909 shutdown_connections 2022-06-29T16:08:02.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.205+0000 7fcf9d2a2700 1 --2- 172.21.15.60:0/176789909 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf9805d070 0x7fcf9814aa90 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:02.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.205+0000 7fcf9d2a2700 1 -- 172.21.15.60:0/176789909 >> 172.21.15.60:0/176789909 conn(0x7fcf980b5890 msgr2=0x7fcf980b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:02.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.205+0000 7fcf9d2a2700 1 -- 172.21.15.60:0/176789909 shutdown_connections 2022-06-29T16:08:02.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.205+0000 7fcf9d2a2700 1 -- 172.21.15.60:0/176789909 wait complete. 2022-06-29T16:08:02.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.209+0000 7fcf9d2a2700 1 Processor -- start 2022-06-29T16:08:02.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.209+0000 7fcf9d2a2700 1 -- start start 2022-06-29T16:08:02.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.209+0000 7fcf9d2a2700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf9805d070 0x7fcf98063260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:02.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.209+0000 7fcf9d2a2700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf980b2130 0x7fcf98116780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:02.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.209+0000 7fcf9d2a2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf9814af80 0x7fcf98116c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:02.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.209+0000 7fcf9d2a2700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf98117160 con 0x7fcf9814af80 2022-06-29T16:08:02.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.209+0000 7fcf9d2a2700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf981598d0 con 0x7fcf9805d070 2022-06-29T16:08:02.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.209+0000 7fcf9d2a2700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf98065480 con 0x7fcf980b2130 2022-06-29T16:08:02.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.209+0000 7fcf9caa1700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf9814af80 0x7fcf98116c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:02.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.209+0000 7fcf9caa1700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf9814af80 0x7fcf98116c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53942/0 (socket says 172.21.15.60:53942) 2022-06-29T16:08:02.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.209+0000 7fcf9caa1700 1 -- 172.21.15.60:0/1277156601 learned_addr learned my addr 172.21.15.60:0/1277156601 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:02.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.209+0000 7fcf97fff700 1 --2- 172.21.15.60:0/1277156601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf9805d070 0x7fcf98063260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:02.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.209+0000 7fcf977fe700 1 --2- 172.21.15.60:0/1277156601 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf980b2130 0x7fcf98116780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:02.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.209+0000 7fcf9caa1700 1 -- 172.21.15.60:0/1277156601 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf980b2130 msgr2=0x7fcf98116780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:02.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.209+0000 7fcf9caa1700 1 --2- 172.21.15.60:0/1277156601 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf980b2130 0x7fcf98116780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:02.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.209+0000 7fcf9caa1700 1 -- 172.21.15.60:0/1277156601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf9805d070 msgr2=0x7fcf98063260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:02.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.209+0000 7fcf9caa1700 1 --2- 172.21.15.60:0/1277156601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf9805d070 0x7fcf98063260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:02.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.209+0000 7fcf9caa1700 1 -- 172.21.15.60:0/1277156601 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf8c00b180 con 0x7fcf9814af80 2022-06-29T16:08:02.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.209+0000 7fcf97fff700 1 --2- 172.21.15.60:0/1277156601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf9805d070 0x7fcf98063260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:08:02.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.209+0000 7fcf9caa1700 1 --2- 172.21.15.60:0/1277156601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf9814af80 0x7fcf98116c70 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 rx=0x7fcf90001490 tx=0x7fcf90007d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:02.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.209+0000 7fcf957fa700 1 -- 172.21.15.60:0/1277156601 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf9000cea0 con 0x7fcf9814af80 2022-06-29T16:08:02.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.209+0000 7fcf957fa700 1 -- 172.21.15.60:0/1277156601 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf90002020 con 0x7fcf9814af80 2022-06-29T16:08:02.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.209+0000 7fcf957fa700 1 -- 172.21.15.60:0/1277156601 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf90002380 con 0x7fcf9814af80 2022-06-29T16:08:02.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.209+0000 7fcf9d2a2700 1 -- 172.21.15.60:0/1277156601 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf98065720 con 0x7fcf9814af80 2022-06-29T16:08:02.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.209+0000 7fcf9d2a2700 1 -- 172.21.15.60:0/1277156601 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf980637d0 con 0x7fcf9814af80 2022-06-29T16:08:02.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.213+0000 7fcf9d2a2700 1 -- 172.21.15.60:0/1277156601 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf9814a350 con 0x7fcf9814af80 2022-06-29T16:08:02.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.213+0000 7fcf957fa700 1 -- 172.21.15.60:0/1277156601 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fcf90010070 con 0x7fcf9814af80 2022-06-29T16:08:02.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.213+0000 7fcf957fa700 1 --2- 172.21.15.60:0/1277156601 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcf78053a70 0x7fcf78055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:02.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.213+0000 7fcf957fa700 1 -- 172.21.15.60:0/1277156601 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcf9006b370 con 0x7fcf9814af80 2022-06-29T16:08:02.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.213+0000 7fcf97fff700 1 --2- 172.21.15.60:0/1277156601 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcf78053a70 0x7fcf78055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:02.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.213+0000 7fcf97fff700 1 --2- 172.21.15.60:0/1277156601 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcf78053a70 0x7fcf78055f30 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7fcf8c01a000 tx=0x7fcf8c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:02.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.221+0000 7fcf957fa700 1 -- 172.21.15.60:0/1277156601 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcf900421f0 con 0x7fcf9814af80 2022-06-29T16:08:02.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.341+0000 7fcf9d2a2700 1 -- 172.21.15.60:0/1277156601 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcf980573d0 con 0x7fcf9814af80 2022-06-29T16:08:02.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.341+0000 7fcf957fa700 1 -- 172.21.15.60:0/1277156601 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v137) v1 ==== 100+0+10 (secure 0 0 0) 0x7fcf90045810 con 0x7fcf9814af80 2022-06-29T16:08:02.348 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:08:02.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.345+0000 7fcf9d2a2700 1 -- 172.21.15.60:0/1277156601 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcf78053a70 msgr2=0x7fcf78055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:02.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.345+0000 7fcf9d2a2700 1 --2- 172.21.15.60:0/1277156601 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcf78053a70 0x7fcf78055f30 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7fcf8c01a000 tx=0x7fcf8c014000).stop 2022-06-29T16:08:02.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.345+0000 7fcf9d2a2700 1 -- 172.21.15.60:0/1277156601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf9814af80 msgr2=0x7fcf98116c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:02.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.345+0000 7fcf9d2a2700 1 --2- 172.21.15.60:0/1277156601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf9814af80 0x7fcf98116c70 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 rx=0x7fcf90001490 tx=0x7fcf90007d70).stop 2022-06-29T16:08:02.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.345+0000 7fcf9d2a2700 1 -- 172.21.15.60:0/1277156601 shutdown_connections 2022-06-29T16:08:02.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.345+0000 7fcf9d2a2700 1 --2- 172.21.15.60:0/1277156601 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fcf78053a70 0x7fcf78055f30 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:02.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.345+0000 7fcf9d2a2700 1 --2- 172.21.15.60:0/1277156601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf9814af80 0x7fcf98116c70 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:02.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.345+0000 7fcf9d2a2700 1 --2- 172.21.15.60:0/1277156601 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fcf980b2130 0x7fcf98116780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:02.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.345+0000 7fcf9d2a2700 1 --2- 172.21.15.60:0/1277156601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf9805d070 0x7fcf98063260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:02.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.345+0000 7fcf9d2a2700 1 -- 172.21.15.60:0/1277156601 >> 172.21.15.60:0/1277156601 conn(0x7fcf980b5890 msgr2=0x7fcf980b0190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:02.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.345+0000 7fcf9d2a2700 1 -- 172.21.15.60:0/1277156601 shutdown_connections 2022-06-29T16:08:02.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.345+0000 7fcf9d2a2700 1 -- 172.21.15.60:0/1277156601 wait complete. 2022-06-29T16:08:02.359 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-06-29T16:08:02.360 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:08:02.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.461+0000 7f38f309e700 1 Processor -- start 2022-06-29T16:08:02.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.461+0000 7f38f309e700 1 -- start start 2022-06-29T16:08:02.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.465+0000 7f38f309e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38ec0b2270 0x7f38ec0b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:02.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.465+0000 7f38f309e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f38ec0aca20 0x7f38ec13e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:02.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.465+0000 7f38f309e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38ec0b2650 con 0x7f38ec0634d0 2022-06-29T16:08:02.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.465+0000 7f38f309e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38ec067770 con 0x7f38ec0aca20 2022-06-29T16:08:02.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.465+0000 7f38f309e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38ec062af0 con 0x7f38ec0b2270 2022-06-29T16:08:02.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.465+0000 7f38f289d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f38ec0aca20 0x7f38ec13e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:02.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.465+0000 7f38f189b700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f38ec0634d0 0x7f38ec13e270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33800/0 (socket says 172.21.15.60:33800) 2022-06-29T16:08:02.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.465+0000 7f38f189b700 1 -- 172.21.15.60:0/4139067730 learned_addr learned my addr 172.21.15.60:0/4139067730 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:02.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.465+0000 7f38f209c700 1 --2- 172.21.15.60:0/4139067730 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38ec0b2270 0x7f38ec0b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:02.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.465+0000 7f38f289d700 1 -- 172.21.15.60:0/4139067730 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38ec0b2270 msgr2=0x7f38ec0b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:02.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.465+0000 7f38f289d700 1 --2- 172.21.15.60:0/4139067730 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38ec0b2270 0x7f38ec0b0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:02.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.465+0000 7f38f289d700 1 -- 172.21.15.60:0/4139067730 >> v1:172.21.15.60:6789/0 conn(0x7f38ec0634d0 legacy=0x7f38ec13e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:02.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.465+0000 7f38f289d700 1 -- 172.21.15.60:0/4139067730 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38ec062d90 con 0x7f38ec0aca20 2022-06-29T16:08:02.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.465+0000 7f38f289d700 1 --2- 172.21.15.60:0/4139067730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38ec0aca20 0x7f38ec13e980 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 rx=0x7f38e40032f0 tx=0x7f38e40091d0).ready entity=mon.0 client_cookie=f3cab6e9b43669d9 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:02.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.465+0000 7f38f109a700 1 -- 172.21.15.60:0/4139067730 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38e4009ea0 con 0x7f38ec0aca20 2022-06-29T16:08:02.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.465+0000 7f38f109a700 1 -- 172.21.15.60:0/4139067730 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38e400b8a0 con 0x7f38ec0aca20 2022-06-29T16:08:02.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.465+0000 7f38f109a700 1 -- 172.21.15.60:0/4139067730 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38e400ba10 con 0x7f38ec0aca20 2022-06-29T16:08:02.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.465+0000 7f38f309e700 1 -- 172.21.15.60:0/4139067730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38ec0aca20 msgr2=0x7f38ec13e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:02.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.465+0000 7f38f309e700 1 --2- 172.21.15.60:0/4139067730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38ec0aca20 0x7f38ec13e980 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 rx=0x7f38e40032f0 tx=0x7f38e40091d0).stop 2022-06-29T16:08:02.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.465+0000 7f38f309e700 1 -- 172.21.15.60:0/4139067730 shutdown_connections 2022-06-29T16:08:02.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.465+0000 7f38f309e700 1 --2- 172.21.15.60:0/4139067730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38ec0aca20 0x7f38ec13e980 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:02.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.465+0000 7f38f309e700 1 --2- 172.21.15.60:0/4139067730 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38ec0b2270 0x7f38ec0b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:02.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.465+0000 7f38f309e700 1 -- 172.21.15.60:0/4139067730 >> 172.21.15.60:0/4139067730 conn(0x7f38ec0b5890 msgr2=0x7f38ec0b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:02.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.465+0000 7f38f309e700 1 -- 172.21.15.60:0/4139067730 shutdown_connections 2022-06-29T16:08:02.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.465+0000 7f38f309e700 1 -- 172.21.15.60:0/4139067730 wait complete. 2022-06-29T16:08:02.486 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.465+0000 7f38f309e700 1 Processor -- start 2022-06-29T16:08:02.486 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.465+0000 7f38f309e700 1 -- start start 2022-06-29T16:08:02.487 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.469+0000 7f38f309e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38ec0634d0 0x7f38ec141110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:02.487 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.469+0000 7f38f309e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38ec0aca20 0x7f38ec12c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:02.487 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.469+0000 7f38f309e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38ec0b2270 0x7f38ec12caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:02.487 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.469+0000 7f38f309e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38ec12ea00 con 0x7f38ec0aca20 2022-06-29T16:08:02.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.469+0000 7f38f309e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38ec12ec60 con 0x7f38ec0634d0 2022-06-29T16:08:02.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.469+0000 7f38f309e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38ec141ca0 con 0x7f38ec0b2270 2022-06-29T16:08:02.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.469+0000 7f38f189b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38ec0aca20 0x7f38ec12c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:02.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.469+0000 7f38f189b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38ec0aca20 0x7f38ec12c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53954/0 (socket says 172.21.15.60:53954) 2022-06-29T16:08:02.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.469+0000 7f38f189b700 1 -- 172.21.15.60:0/3583875810 learned_addr learned my addr 172.21.15.60:0/3583875810 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:02.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.469+0000 7f38f209c700 1 --2- 172.21.15.60:0/3583875810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38ec0634d0 0x7f38ec141110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:02.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.469+0000 7f38f289d700 1 --2- 172.21.15.60:0/3583875810 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38ec0b2270 0x7f38ec12caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:02.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.469+0000 7f38f189b700 1 -- 172.21.15.60:0/3583875810 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38ec0b2270 msgr2=0x7f38ec12caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:02.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.469+0000 7f38f189b700 1 --2- 172.21.15.60:0/3583875810 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38ec0b2270 0x7f38ec12caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:02.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.469+0000 7f38f189b700 1 -- 172.21.15.60:0/3583875810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38ec0634d0 msgr2=0x7f38ec141110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:02.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.469+0000 7f38f189b700 1 --2- 172.21.15.60:0/3583875810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38ec0634d0 0x7f38ec141110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:02.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.469+0000 7f38f189b700 1 -- 172.21.15.60:0/3583875810 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38e4009470 con 0x7f38ec0aca20 2022-06-29T16:08:02.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.469+0000 7f38f289d700 1 --2- 172.21.15.60:0/3583875810 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38ec0b2270 0x7f38ec12caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:08:02.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.469+0000 7f38f189b700 1 --2- 172.21.15.60:0/3583875810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38ec0aca20 0x7f38ec12c5b0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 rx=0x7f38ec0b2b70 tx=0x7f38dc007d30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:02.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.469+0000 7f38db7fe700 1 -- 172.21.15.60:0/3583875810 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38dc005610 con 0x7f38ec0aca20 2022-06-29T16:08:02.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.469+0000 7f38db7fe700 1 -- 172.21.15.60:0/3583875810 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38dc005770 con 0x7f38ec0aca20 2022-06-29T16:08:02.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.469+0000 7f38f309e700 1 -- 172.21.15.60:0/3583875810 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38ec12cf90 con 0x7f38ec0aca20 2022-06-29T16:08:02.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.469+0000 7f38db7fe700 1 -- 172.21.15.60:0/3583875810 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38dc005ad0 con 0x7f38ec0aca20 2022-06-29T16:08:02.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.469+0000 7f38f309e700 1 -- 172.21.15.60:0/3583875810 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38ec12f1d0 con 0x7f38ec0aca20 2022-06-29T16:08:02.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.473+0000 7f38f309e700 1 -- 172.21.15.60:0/3583875810 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38ec0573d0 con 0x7f38ec0aca20 2022-06-29T16:08:02.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.473+0000 7f38db7fe700 1 -- 172.21.15.60:0/3583875810 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f38dc005c30 con 0x7f38ec0aca20 2022-06-29T16:08:02.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.473+0000 7f38db7fe700 1 --2- 172.21.15.60:0/3583875810 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f38cc053b00 0x7f38cc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:02.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.473+0000 7f38db7fe700 1 -- 172.21.15.60:0/3583875810 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f38dc06e4a0 con 0x7f38ec0aca20 2022-06-29T16:08:02.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.473+0000 7f38f209c700 1 --2- 172.21.15.60:0/3583875810 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f38cc053b00 0x7f38cc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:02.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.473+0000 7f38f209c700 1 --2- 172.21.15.60:0/3583875810 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f38cc053b00 0x7f38cc055fc0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0x7f38e800c000 tx=0x7f38e8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:02.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.477+0000 7f38db7fe700 1 -- 172.21.15.60:0/3583875810 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f38dc0453c0 con 0x7f38ec0aca20 2022-06-29T16:08:02.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.637+0000 7f38f309e700 1 -- 172.21.15.60:0/3583875810 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38ec0683d0 con 0x7f38ec0aca20 2022-06-29T16:08:02.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.637+0000 7f38db7fe700 1 -- 172.21.15.60:0/3583875810 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2142 (secure 0 0 0) 0x7f38dc0489e0 con 0x7f38ec0aca20 2022-06-29T16:08:02.643 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:08:02.644 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":68,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.92:6835/2628497783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2628497783},{"type":"v1","addr":"172.21.15.92:6835","nonce":2628497783}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":67}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:07:54.174936+0000","modified":"2022-06-29T16:07:55.178863+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8463":{"gid":8463,"name":"a","rank":0,"incarnation":67,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/2222627281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2222627281},{"type":"v1","addr":"172.21.15.60:6833","nonce":2222627281}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2022-06-29T16:08:02.646 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.641+0000 7f38f309e700 1 -- 172.21.15.60:0/3583875810 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f38cc053b00 msgr2=0x7f38cc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:02.647 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.641+0000 7f38f309e700 1 --2- 172.21.15.60:0/3583875810 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f38cc053b00 0x7f38cc055fc0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0x7f38e800c000 tx=0x7f38e8008040).stop 2022-06-29T16:08:02.647 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.641+0000 7f38f309e700 1 -- 172.21.15.60:0/3583875810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38ec0aca20 msgr2=0x7f38ec12c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:02.647 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.641+0000 7f38f309e700 1 --2- 172.21.15.60:0/3583875810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38ec0aca20 0x7f38ec12c5b0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 rx=0x7f38ec0b2b70 tx=0x7f38dc007d30).stop 2022-06-29T16:08:02.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.641+0000 7f38f309e700 1 -- 172.21.15.60:0/3583875810 shutdown_connections 2022-06-29T16:08:02.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.641+0000 7f38f309e700 1 --2- 172.21.15.60:0/3583875810 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f38cc053b00 0x7f38cc055fc0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:02.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.641+0000 7f38f309e700 1 --2- 172.21.15.60:0/3583875810 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f38ec0b2270 0x7f38ec12caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:02.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.641+0000 7f38f309e700 1 --2- 172.21.15.60:0/3583875810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f38ec0aca20 0x7f38ec12c5b0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:02.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.641+0000 7f38f309e700 1 --2- 172.21.15.60:0/3583875810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38ec0634d0 0x7f38ec141110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:02.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.641+0000 7f38f309e700 1 -- 172.21.15.60:0/3583875810 >> 172.21.15.60:0/3583875810 conn(0x7f38ec0b5890 msgr2=0x7f38ec13f670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:02.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.641+0000 7f38f309e700 1 -- 172.21.15.60:0/3583875810 shutdown_connections 2022-06-29T16:08:02.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:02.641+0000 7f38f309e700 1 -- 172.21.15.60:0/3583875810 wait complete. 2022-06-29T16:08:02.650 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 68 2022-06-29T16:08:02.863 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 68, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:07:54.174936+0000', 'modified': '2022-06-29T16:07:55.178863+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8463': {'gid': 8463, 'name': 'a', 'rank': 0, 'incarnation': 67, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.60:6833/2222627281', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6832', 'nonce': 2222627281}, {'type': 'v1', 'addr': '172.21.15.60:6833', 'nonce': 2222627281}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:08:02.864 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:08:02.865 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-06-29T16:08:02.982 INFO:teuthology.orchestra.run.smithi060.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":8463,"want_state":"up:active","state":"up:active","rank_uptime":8.7924281620000002,"mdsmap_epoch":68,"osdmap_epoch":137,"osdmap_epoch_barrier":137,"uptime":13.14741076} 2022-06-29T16:08:02.983 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 8463, "mdsmap_epoch": 68, "osdmap_epoch": 137, "osdmap_epoch_barrier": 137, "rank_uptime": 8.792428162, "state": "up:active", "uptime": 13.14741076, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:08:02.984 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-06-29T16:08:02.984 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-06-29T16:08:02.984 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi156.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-06-29T16:08:02.985 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:02.985 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:08:02.994 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2022-06-29T16:08:03.009 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:08:03.010 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:08:03.064 INFO:teuthology.orchestra.run.smithi156.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:08:03.065 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:08:03.066 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:03.067 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2022-06-29T16:08:03.073 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-06-29T16:08:03.074 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:08:03.076 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2022-06-29T16:08:03.093 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:08:03.094 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:08:03.149 INFO:teuthology.orchestra.run.smithi156.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:08:03.150 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:2022-06-29T16:08:03.141+0000 7f32630b7080 -1 init, newargv = 0x564022a671b0 newargc=9 2022-06-29T16:08:03.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[13084]: starting ceph client 2022-06-29T16:08:03.153 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:08:03.155 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:03.156 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2022-06-29T16:08:03.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[13084]: starting fuse 2022-06-29T16:08:03.164 INFO:teuthology.orchestra.run.smithi156.stdout:52 2022-06-29T16:08:03.165 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-06-29T16:08:03.166 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:03.167 DEBUG:teuthology.orchestra.run.smithi156:> sudo python3 -c ' 2022-06-29T16:08:03.167 DEBUG:teuthology.orchestra.run.smithi156:> import glob 2022-06-29T16:08:03.169 DEBUG:teuthology.orchestra.run.smithi156:> import re 2022-06-29T16:08:03.169 DEBUG:teuthology.orchestra.run.smithi156:> import os 2022-06-29T16:08:03.171 DEBUG:teuthology.orchestra.run.smithi156:> import subprocess 2022-06-29T16:08:03.171 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:08:03.172 DEBUG:teuthology.orchestra.run.smithi156:> def find_socket(client_name): 2022-06-29T16:08:03.172 DEBUG:teuthology.orchestra.run.smithi156:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-06-29T16:08:03.172 DEBUG:teuthology.orchestra.run.smithi156:> files = glob.glob(asok_path) 2022-06-29T16:08:03.173 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:08:03.173 DEBUG:teuthology.orchestra.run.smithi156:> # Given a non-glob path, it better be there 2022-06-29T16:08:03.174 DEBUG:teuthology.orchestra.run.smithi156:> if "*" not in asok_path: 2022-06-29T16:08:03.174 DEBUG:teuthology.orchestra.run.smithi156:> assert(len(files) == 1) 2022-06-29T16:08:03.175 DEBUG:teuthology.orchestra.run.smithi156:> return files[0] 2022-06-29T16:08:03.175 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:08:03.175 DEBUG:teuthology.orchestra.run.smithi156:> for f in files: 2022-06-29T16:08:03.176 DEBUG:teuthology.orchestra.run.smithi156:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-06-29T16:08:03.176 DEBUG:teuthology.orchestra.run.smithi156:> if os.path.exists("/proc/{0}".format(pid)): 2022-06-29T16:08:03.177 DEBUG:teuthology.orchestra.run.smithi156:> return f 2022-06-29T16:08:03.177 DEBUG:teuthology.orchestra.run.smithi156:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-06-29T16:08:03.177 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:08:03.178 DEBUG:teuthology.orchestra.run.smithi156:> print(find_socket("client.1")) 2022-06-29T16:08:03.178 DEBUG:teuthology.orchestra.run.smithi156:> ' 2022-06-29T16:08:03.250 INFO:teuthology.orchestra.run.smithi156.stdout:/var/run/ceph/ceph-client.1.13084.asok 2022-06-29T16:08:03.255 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.13084.asok 2022-06-29T16:08:03.256 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:03.257 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.13084.asok status 2022-06-29T16:08:03.349 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2022-06-29T16:08:03.350 INFO:teuthology.orchestra.run.smithi156.stdout: "metadata": { 2022-06-29T16:08:03.351 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_sha1": "f25ffac72e5a9e63860ff7deb6767a11f0559a16", 2022-06-29T16:08:03.351 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_version": "ceph version 15.2.16-304-gf25ffac7 (f25ffac72e5a9e63860ff7deb6767a11f0559a16) octopus (stable)", 2022-06-29T16:08:03.352 INFO:teuthology.orchestra.run.smithi156.stdout: "entity_id": "1", 2022-06-29T16:08:03.352 INFO:teuthology.orchestra.run.smithi156.stdout: "hostname": "smithi156", 2022-06-29T16:08:03.352 INFO:teuthology.orchestra.run.smithi156.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-06-29T16:08:03.353 INFO:teuthology.orchestra.run.smithi156.stdout: "pid": "13084", 2022-06-29T16:08:03.353 INFO:teuthology.orchestra.run.smithi156.stdout: "root": "/" 2022-06-29T16:08:03.354 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:08:03.355 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_count": 0, 2022-06-29T16:08:03.355 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_pinned_count": 0, 2022-06-29T16:08:03.355 INFO:teuthology.orchestra.run.smithi156.stdout: "id": 8584, 2022-06-29T16:08:03.356 INFO:teuthology.orchestra.run.smithi156.stdout: "inst": { 2022-06-29T16:08:03.356 INFO:teuthology.orchestra.run.smithi156.stdout: "name": { 2022-06-29T16:08:03.357 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "client", 2022-06-29T16:08:03.357 INFO:teuthology.orchestra.run.smithi156.stdout: "num": 8584 2022-06-29T16:08:03.358 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:08:03.358 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2022-06-29T16:08:03.359 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2022-06-29T16:08:03.359 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "172.21.15.156:0", 2022-06-29T16:08:03.359 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 2467615581 2022-06-29T16:08:03.360 INFO:teuthology.orchestra.run.smithi156.stdout: } 2022-06-29T16:08:03.360 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:08:03.361 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2022-06-29T16:08:03.361 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2022-06-29T16:08:03.362 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "172.21.15.156:0", 2022-06-29T16:08:03.362 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 2467615581 2022-06-29T16:08:03.363 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:08:03.363 INFO:teuthology.orchestra.run.smithi156.stdout: "inst_str": "client.8584 172.21.15.156:0/2467615581", 2022-06-29T16:08:03.364 INFO:teuthology.orchestra.run.smithi156.stdout: "addr_str": "172.21.15.156:0/2467615581", 2022-06-29T16:08:03.364 INFO:teuthology.orchestra.run.smithi156.stdout: "inode_count": 1, 2022-06-29T16:08:03.364 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_epoch": 68, 2022-06-29T16:08:03.365 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch": 137, 2022-06-29T16:08:03.365 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch_barrier": 0, 2022-06-29T16:08:03.366 INFO:teuthology.orchestra.run.smithi156.stdout: "blacklisted": false, 2022-06-29T16:08:03.366 INFO:teuthology.orchestra.run.smithi156.stdout: "fs_name": "cephfs" 2022-06-29T16:08:03.367 INFO:teuthology.orchestra.run.smithi156.stdout:} 2022-06-29T16:08:03.369 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:03.370 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:08:03.371 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:08:03.381 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2022-06-29T16:08:03.382 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-06-29T16:08:03.382 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:03.383 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:08:03.441 INFO:teuthology.orchestra.run:Running command with timeout 10800 2022-06-29T16:08:03.442 DEBUG:teuthology.orchestra.run.smithi121:> cd /home/ubuntu/cephtest/mnt.0 && sudo getfattr --only-values -n ceph.dir.rctime . 2022-06-29T16:08:03.479 INFO:teuthology.orchestra.run.smithi121.stdout:1656518883.436112786 2022-06-29T16:08:03.480 INFO:tasks.cephfs.test_misc:rctime = 1656518883.436112786 2022-06-29T16:08:03.480 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-06-29T16:08:03.481 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:08:03.503 DEBUG:teuthology.orchestra.run.smithi092:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:08:03.525 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:08:03.548 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:08:03.570 DEBUG:teuthology.orchestra.run.smithi197:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:08:03.591 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-06-29T16:08:03.592 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:03.593 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:08:03.593 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:08:03.602 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T16:08:03.603 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T16:08:03.604 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2022-06-29T16:08:03.605 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-06-29T16:08:03.605 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:08:03.620 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[13556]: fuse finished with error 0 and tester_r 0 2022-06-29T16:08:03.622 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:03.623 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:08:03.623 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:08:03.669 INFO:teuthology.orchestra.run.smithi121.stdout:ext2/ext3 2022-06-29T16:08:03.671 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-06-29T16:08:03.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:08:03.757 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:08:03.766 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:03.767 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:08:03.767 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:08:03.776 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2022-06-29T16:08:03.778 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-06-29T16:08:03.778 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi156.front.sepia.ceph.com... 2022-06-29T16:08:03.779 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-06-29T16:08:03.780 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:08:03.834 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[13084]: fuse finished with error 0 and tester_r 0 2022-06-29T16:08:03.836 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:03.836 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:08:03.837 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:08:03.843 INFO:teuthology.orchestra.run.smithi156.stdout:ext2/ext3 2022-06-29T16:08:03.845 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-06-29T16:08:03.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:08:03.945 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:08:03.956 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:03.957 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:08:03.957 DEBUG:teuthology.orchestra.run.smithi197:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T16:08:03.965 INFO:teuthology.orchestra.run.smithi197.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-06-29T16:08:03.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:08:03.967 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-06-29T16:08:03.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:08:03.968 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T16:08:03.974 INFO:teuthology.orchestra.run.smithi197.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-06-29T16:08:03.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:08:03.976 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-06-29T16:08:04.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.085+0000 7f760d112700 1 Processor -- start 2022-06-29T16:08:04.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.085+0000 7f760d112700 1 -- start start 2022-06-29T16:08:04.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.085+0000 7f760d112700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76080b5950 0x7f7608067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:04.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.085+0000 7f760d112700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f760814af10 0x7f760805c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:04.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.085+0000 7f760d112700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76080acba0 con 0x7f7608063ad0 2022-06-29T16:08:04.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.085+0000 7f760d112700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f760814b3a0 con 0x7f760814af10 2022-06-29T16:08:04.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.085+0000 7f760d112700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7608067220 con 0x7f76080b5950 2022-06-29T16:08:04.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.085+0000 7f760759e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f760814af10 0x7f760805c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:04.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.085+0000 7f760759e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f760814af10 0x7f760805c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53962/0 (socket says 172.21.15.60:53962) 2022-06-29T16:08:04.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.085+0000 7f760759e700 1 -- 172.21.15.60:0/2598631960 learned_addr learned my addr 172.21.15.60:0/2598631960 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:04.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.085+0000 7f7606d9d700 1 --2- 172.21.15.60:0/2598631960 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76080b5950 0x7f7608067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:04.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.085+0000 7f760759e700 1 -- 172.21.15.60:0/2598631960 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76080b5950 msgr2=0x7f7608067680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:04.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.085+0000 7f760759e700 1 --2- 172.21.15.60:0/2598631960 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76080b5950 0x7f7608067680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:04.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.085+0000 7f760759e700 1 -- 172.21.15.60:0/2598631960 >> v1:172.21.15.60:6789/0 conn(0x7f7608063ad0 legacy=0x7f760805bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:04.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.085+0000 7f760759e700 1 -- 172.21.15.60:0/2598631960 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76080b5d30 con 0x7f760814af10 2022-06-29T16:08:04.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.085+0000 7f760759e700 1 --2- 172.21.15.60:0/2598631960 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f760814af10 0x7f760805c430 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 rx=0x7f76000032f0 tx=0x7f7600009290).ready entity=mon.0 client_cookie=a19862deb0a722c3 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:04.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.085+0000 7f7605d9b700 1 -- 172.21.15.60:0/2598631960 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f760000b8a0 con 0x7f760814af10 2022-06-29T16:08:04.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.085+0000 7f7605d9b700 1 -- 172.21.15.60:0/2598631960 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f760000ba00 con 0x7f760814af10 2022-06-29T16:08:04.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.085+0000 7f7605d9b700 1 -- 172.21.15.60:0/2598631960 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f760000bb70 con 0x7f760814af10 2022-06-29T16:08:04.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.085+0000 7f760d112700 1 -- 172.21.15.60:0/2598631960 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f760814af10 msgr2=0x7f760805c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:04.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.085+0000 7f760d112700 1 --2- 172.21.15.60:0/2598631960 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f760814af10 0x7f760805c430 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 rx=0x7f76000032f0 tx=0x7f7600009290).stop 2022-06-29T16:08:04.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.085+0000 7f760d112700 1 -- 172.21.15.60:0/2598631960 shutdown_connections 2022-06-29T16:08:04.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.085+0000 7f760d112700 1 --2- 172.21.15.60:0/2598631960 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f760814af10 0x7f760805c430 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:04.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.085+0000 7f760d112700 1 --2- 172.21.15.60:0/2598631960 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76080b5950 0x7f7608067680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:04.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.085+0000 7f760d112700 1 -- 172.21.15.60:0/2598631960 >> 172.21.15.60:0/2598631960 conn(0x7f76080632c0 msgr2=0x7f76080b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:04.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.085+0000 7f760d112700 1 -- 172.21.15.60:0/2598631960 shutdown_connections 2022-06-29T16:08:04.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.085+0000 7f760d112700 1 -- 172.21.15.60:0/2598631960 wait complete. 2022-06-29T16:08:04.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.089+0000 7f760d112700 1 Processor -- start 2022-06-29T16:08:04.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.089+0000 7f760d112700 1 -- start start 2022-06-29T16:08:04.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.089+0000 7f760d112700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7608063ad0 0x7f76080b07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:04.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.089+0000 7f760d112700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76080b5950 0x7f76081595b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:04.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.089+0000 7f760d112700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f760814af10 0x7f7608159aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:04.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.089+0000 7f760d112700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76080ad2a0 con 0x7f76080b5950 2022-06-29T16:08:04.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.089+0000 7f760d112700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76080ad000 con 0x7f760814af10 2022-06-29T16:08:04.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.089+0000 7f760d112700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7608156160 con 0x7f7608063ad0 2022-06-29T16:08:04.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.089+0000 7f760659c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76080b5950 0x7f76081595b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:04.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.089+0000 7f7606d9d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7608063ad0 0x7f76080b07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:04.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.089+0000 7f760659c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76080b5950 0x7f76081595b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53970/0 (socket says 172.21.15.60:53970) 2022-06-29T16:08:04.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.089+0000 7f760759e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f760814af10 0x7f7608159aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:04.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.089+0000 7f7606d9d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7608063ad0 0x7f76080b07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:38828/0 (socket says 172.21.15.60:38828) 2022-06-29T16:08:04.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.089+0000 7f760659c700 1 -- 172.21.15.60:0/194244530 learned_addr learned my addr 172.21.15.60:0/194244530 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:04.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.089+0000 7f760659c700 1 -- 172.21.15.60:0/194244530 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7608063ad0 msgr2=0x7f76080b07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:04.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.089+0000 7f760659c700 1 --2- 172.21.15.60:0/194244530 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7608063ad0 0x7f76080b07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:04.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.093+0000 7f760659c700 1 -- 172.21.15.60:0/194244530 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f760814af10 msgr2=0x7f7608159aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:04.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.093+0000 7f760659c700 1 --2- 172.21.15.60:0/194244530 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f760814af10 0x7f7608159aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:04.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.093+0000 7f760659c700 1 -- 172.21.15.60:0/194244530 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7600009530 con 0x7f76080b5950 2022-06-29T16:08:04.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.093+0000 7f760659c700 1 --2- 172.21.15.60:0/194244530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76080b5950 0x7f76081595b0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 rx=0x7f760805ff90 tx=0x7f75f8007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:04.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.093+0000 7f75fffff700 1 -- 172.21.15.60:0/194244530 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75f8007030 con 0x7f76080b5950 2022-06-29T16:08:04.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.093+0000 7f75fffff700 1 -- 172.21.15.60:0/194244530 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75f8007190 con 0x7f76080b5950 2022-06-29T16:08:04.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.093+0000 7f760d112700 1 -- 172.21.15.60:0/194244530 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f760814a620 con 0x7f76080b5950 2022-06-29T16:08:04.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.093+0000 7f75fffff700 1 -- 172.21.15.60:0/194244530 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75f8006510 con 0x7f76080b5950 2022-06-29T16:08:04.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.093+0000 7f760d112700 1 -- 172.21.15.60:0/194244530 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76080ad730 con 0x7f76080b5950 2022-06-29T16:08:04.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.093+0000 7f760d112700 1 -- 172.21.15.60:0/194244530 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76080b1000 con 0x7f76080b5950 2022-06-29T16:08:04.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.093+0000 7f75fffff700 1 -- 172.21.15.60:0/194244530 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f75f8010030 con 0x7f76080b5950 2022-06-29T16:08:04.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.097+0000 7f75fffff700 1 --2- 172.21.15.60:0/194244530 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f75ec053a70 0x7f75ec055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:04.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.097+0000 7f75fffff700 1 -- 172.21.15.60:0/194244530 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f75f806ba40 con 0x7f76080b5950 2022-06-29T16:08:04.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.097+0000 7f7606d9d700 1 --2- 172.21.15.60:0/194244530 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f75ec053a70 0x7f75ec055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:04.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.097+0000 7f7606d9d700 1 --2- 172.21.15.60:0/194244530 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f75ec053a70 0x7f75ec055f30 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0x7f75f000c000 tx=0x7f75f0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:04.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.101+0000 7f75fffff700 1 -- 172.21.15.60:0/194244530 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f75f8042960 con 0x7f76080b5950 2022-06-29T16:08:04.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.229+0000 7f760d112700 1 -- 172.21.15.60:0/194244530 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f7608155f90 con 0x7f76080b5950 2022-06-29T16:08:04.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.233+0000 7f75fffff700 1 -- 172.21.15.60:0/194244530 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v137) v1 ==== 81+0+33659 (secure 0 0 0) 0x7f75f8045f80 con 0x7f76080b5950 2022-06-29T16:08:04.238 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:08:04.238 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:08:04.238 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 137, 2022-06-29T16:08:04.239 INFO:teuthology.orchestra.run.smithi060.stdout: "fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", 2022-06-29T16:08:04.239 INFO:teuthology.orchestra.run.smithi060.stdout: "created": "2022-06-29T15:58:07.965719+0000", 2022-06-29T16:08:04.239 INFO:teuthology.orchestra.run.smithi060.stdout: "modified": "2022-06-29T16:07:55.177815+0000", 2022-06-29T16:08:04.239 INFO:teuthology.orchestra.run.smithi060.stdout: "last_up_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:08:04.240 INFO:teuthology.orchestra.run.smithi060.stdout: "last_in_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:08:04.240 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-06-29T16:08:04.240 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_num": 5799936, 2022-06-29T16:08:04.240 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_set": [ 2022-06-29T16:08:04.241 INFO:teuthology.orchestra.run.smithi060.stdout: "pglog_hardlimit", 2022-06-29T16:08:04.241 INFO:teuthology.orchestra.run.smithi060.stdout: "purged_snapdirs", 2022-06-29T16:08:04.241 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_deletes", 2022-06-29T16:08:04.241 INFO:teuthology.orchestra.run.smithi060.stdout: "sortbitwise" 2022-06-29T16:08:04.242 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:08:04.242 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_version": 4, 2022-06-29T16:08:04.242 INFO:teuthology.orchestra.run.smithi060.stdout: "full_ratio": 0.94999998807907104, 2022-06-29T16:08:04.242 INFO:teuthology.orchestra.run.smithi060.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-06-29T16:08:04.242 INFO:teuthology.orchestra.run.smithi060.stdout: "nearfull_ratio": 0.85000002384185791, 2022-06-29T16:08:04.243 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_snapshot": "", 2022-06-29T16:08:04.243 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_max": 28, 2022-06-29T16:08:04.243 INFO:teuthology.orchestra.run.smithi060.stdout: "max_osd": 8, 2022-06-29T16:08:04.243 INFO:teuthology.orchestra.run.smithi060.stdout: "require_min_compat_client": "jewel", 2022-06-29T16:08:04.244 INFO:teuthology.orchestra.run.smithi060.stdout: "min_compat_client": "jewel", 2022-06-29T16:08:04.244 INFO:teuthology.orchestra.run.smithi060.stdout: "require_osd_release": "octopus", 2022-06-29T16:08:04.244 INFO:teuthology.orchestra.run.smithi060.stdout: "pools": [ 2022-06-29T16:08:04.244 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.245 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2022-06-29T16:08:04.245 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "device_health_metrics", 2022-06-29T16:08:04.245 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:10.961127+0000", 2022-06-29T16:08:04.245 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:08:04.246 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:08:04.246 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:08:04.246 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:08:04.246 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:08:04.246 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:08:04.247 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:08:04.247 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:08:04.247 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 1, 2022-06-29T16:08:04.247 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 1, 2022-06-29T16:08:04.248 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 1, 2022-06-29T16:08:04.248 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 1, 2022-06-29T16:08:04.248 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 1, 2022-06-29T16:08:04.248 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:08:04.249 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:08:04.249 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:08:04.249 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:08:04.249 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:08:04.250 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:08:04.250 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:08:04.250 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.250 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "15", 2022-06-29T16:08:04.251 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:08:04.251 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:08:04.251 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:08:04.251 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:08:04.251 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:08:04.252 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:08:04.252 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:08:04.252 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:08:04.252 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:08:04.253 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:08:04.253 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:08:04.253 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:08:04.253 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:08:04.254 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:08:04.254 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:08:04.254 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:08:04.254 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:08:04.254 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:08:04.255 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:08:04.255 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:08:04.255 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:08:04.255 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:08:04.256 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:08:04.256 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:08:04.256 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:08:04.256 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:08:04.256 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.257 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:08:04.257 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:08:04.257 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:08:04.257 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:08:04.258 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:08:04.258 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:08:04.258 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:08:04.258 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:08:04.259 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:08:04.259 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:08:04.259 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:08:04.259 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:08:04.260 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 1 2022-06-29T16:08:04.260 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.260 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:08:04.260 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_devicehealth": {} 2022-06-29T16:08:04.261 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.261 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.261 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.261 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 2, 2022-06-29T16:08:04.261 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "rbd", 2022-06-29T16:08:04.262 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:19.184276+0000", 2022-06-29T16:08:04.262 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:08:04.262 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:08:04.262 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:08:04.263 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:08:04.263 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:08:04.263 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:08:04.263 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:08:04.263 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:08:04.264 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:08:04.264 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:08:04.264 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:08:04.264 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:08:04.265 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:08:04.265 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:08:04.265 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:08:04.265 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:08:04.265 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:08:04.266 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:08:04.266 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:08:04.266 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:08:04.266 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.267 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "18", 2022-06-29T16:08:04.267 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:08:04.267 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:08:04.268 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:08:04.268 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:08:04.268 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:08:04.268 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:08:04.269 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:08:04.269 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:08:04.269 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:08:04.269 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:08:04.269 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:08:04.270 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:08:04.270 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:08:04.270 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:08:04.270 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:08:04.271 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:08:04.271 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:08:04.271 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:08:04.271 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:08:04.271 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:08:04.272 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:08:04.272 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:08:04.272 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:08:04.272 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:08:04.273 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:08:04.273 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:08:04.273 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.273 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:08:04.273 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:08:04.274 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:08:04.274 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:08:04.274 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:08:04.274 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:08:04.275 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:08:04.275 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:08:04.275 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:08:04.276 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:08:04.276 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:08:04.276 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:08:04.276 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:08:04.276 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd": {} 2022-06-29T16:08:04.277 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.277 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 27, 2022-06-29T16:08:04.278 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "cephfs_metadata", 2022-06-29T16:08:04.278 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T16:07:52.605597+0000", 2022-06-29T16:08:04.278 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:08:04.278 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:08:04.278 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:08:04.279 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:08:04.279 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:08:04.279 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:08:04.279 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:08:04.280 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:08:04.280 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:08:04.280 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:08:04.280 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:08:04.280 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:08:04.281 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:08:04.281 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:08:04.281 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:08:04.281 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:08:04.281 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:08:04.282 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:08:04.282 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:08:04.282 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:08:04.282 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.283 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "136", 2022-06-29T16:08:04.283 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:08:04.283 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:08:04.283 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:08:04.283 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:08:04.284 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:08:04.284 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:08:04.284 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:08:04.284 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:08:04.285 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:08:04.285 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:08:04.285 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:08:04.285 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:08:04.285 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:08:04.286 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:08:04.286 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:08:04.286 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:08:04.286 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:08:04.287 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:08:04.287 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:08:04.287 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:08:04.287 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:08:04.287 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:08:04.288 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:08:04.288 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:08:04.288 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:08:04.288 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:08:04.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.289 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:08:04.289 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:08:04.289 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:08:04.290 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:08:04.290 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:08:04.290 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:08:04.290 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:08:04.290 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:08:04.291 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:08:04.291 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:08:04.291 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:08:04.291 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:08:04.292 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_bias": 4, 2022-06-29T16:08:04.292 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 16, 2022-06-29T16:08:04.292 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_priority": 5 2022-06-29T16:08:04.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.292 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:08:04.293 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs": { 2022-06-29T16:08:04.293 INFO:teuthology.orchestra.run.smithi060.stdout: "metadata": "cephfs" 2022-06-29T16:08:04.293 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.293 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.294 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.294 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.294 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 28, 2022-06-29T16:08:04.294 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "cephfs_data", 2022-06-29T16:08:04.295 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T16:07:53.864756+0000", 2022-06-29T16:08:04.295 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:08:04.295 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:08:04.295 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:08:04.295 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:08:04.296 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:08:04.296 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:08:04.296 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:08:04.296 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:08:04.296 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:08:04.297 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:08:04.297 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:08:04.297 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:08:04.297 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:08:04.298 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:08:04.298 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:08:04.298 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:08:04.298 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:08:04.298 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:08:04.299 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:08:04.299 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:08:04.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.299 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "137", 2022-06-29T16:08:04.300 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:08:04.300 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:08:04.300 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:08:04.300 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:08:04.300 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:08:04.301 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:08:04.301 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:08:04.301 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:08:04.301 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:08:04.302 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:08:04.302 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:08:04.302 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:08:04.302 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:08:04.302 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:08:04.303 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:08:04.303 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:08:04.303 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:08:04.303 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:08:04.304 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:08:04.304 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:08:04.304 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:08:04.304 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:08:04.304 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:08:04.305 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:08:04.305 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:08:04.305 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:08:04.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.306 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:08:04.306 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:08:04.306 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:08:04.306 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:08:04.307 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:08:04.307 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:08:04.307 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:08:04.307 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:08:04.307 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:08:04.308 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:08:04.308 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:08:04.308 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:08:04.308 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:08:04.308 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs": { 2022-06-29T16:08:04.309 INFO:teuthology.orchestra.run.smithi060.stdout: "data": "cephfs" 2022-06-29T16:08:04.309 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.309 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.309 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.310 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:08:04.310 INFO:teuthology.orchestra.run.smithi060.stdout: "osds": [ 2022-06-29T16:08:04.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.310 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:08:04.311 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "11e7ab9e-fce2-470f-8195-ea4c3c52df22", 2022-06-29T16:08:04.311 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:08:04.311 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:08:04.311 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:08:04.311 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:08:04.312 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:08:04.312 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:08:04.312 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:08:04.312 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 135, 2022-06-29T16:08:04.313 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:08:04.313 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:08:04.313 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:08:04.313 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:04.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.314 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:04.314 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6816", 2022-06-29T16:08:04.315 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:08:04.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.315 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:04.316 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6825", 2022-06-29T16:08:04.316 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:08:04.316 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.316 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.317 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:08:04.317 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:04.318 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.318 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:04.318 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6826", 2022-06-29T16:08:04.318 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:08:04.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.319 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.319 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:04.319 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6827", 2022-06-29T16:08:04.320 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:08:04.320 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.320 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.321 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:08:04.321 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:04.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.322 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:04.322 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6830", 2022-06-29T16:08:04.322 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:08:04.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.323 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.323 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:04.323 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6831", 2022-06-29T16:08:04.323 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:08:04.324 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.324 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.324 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.325 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:08:04.325 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:04.325 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.325 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:04.326 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6828", 2022-06-29T16:08:04.326 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:08:04.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.326 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:04.327 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6829", 2022-06-29T16:08:04.327 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:08:04.327 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.327 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.328 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.328 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6825/12890", 2022-06-29T16:08:04.328 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6827/12890", 2022-06-29T16:08:04.328 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6831/12890", 2022-06-29T16:08:04.329 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6829/12890", 2022-06-29T16:08:04.329 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:08:04.329 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:08:04.329 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:08:04.329 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.330 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.330 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.330 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:08:04.330 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "ca43f136-6f69-4431-a157-fbe073fb585c", 2022-06-29T16:08:04.331 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:08:04.331 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:08:04.331 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:08:04.331 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:08:04.332 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:08:04.332 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:08:04.332 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:08:04.332 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 135, 2022-06-29T16:08:04.332 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:08:04.333 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:08:04.333 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:08:04.333 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:04.333 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.334 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:04.334 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6815", 2022-06-29T16:08:04.334 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:08:04.334 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.335 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.335 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:04.335 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6818", 2022-06-29T16:08:04.335 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:08:04.336 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.336 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.336 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.336 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:08:04.336 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:04.337 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.337 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:04.337 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6819", 2022-06-29T16:08:04.337 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:08:04.338 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.338 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.338 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:04.338 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6820", 2022-06-29T16:08:04.339 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:08:04.339 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.339 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.339 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.339 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:08:04.340 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:04.340 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.340 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:04.340 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6823", 2022-06-29T16:08:04.341 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:08:04.341 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.341 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.341 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:04.342 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6824", 2022-06-29T16:08:04.342 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:08:04.342 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.342 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.342 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.343 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:08:04.343 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:04.343 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.343 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:04.344 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6821", 2022-06-29T16:08:04.344 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:08:04.344 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.344 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.344 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:04.345 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6822", 2022-06-29T16:08:04.345 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:08:04.345 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.345 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.346 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.346 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6818/12892", 2022-06-29T16:08:04.346 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6820/12892", 2022-06-29T16:08:04.346 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6824/12892", 2022-06-29T16:08:04.346 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6822/12892", 2022-06-29T16:08:04.347 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:08:04.347 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:08:04.347 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:08:04.347 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.348 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.348 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.348 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:08:04.348 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "74fac45f-5f07-44ec-a9cb-2644c123fd4a", 2022-06-29T16:08:04.348 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:08:04.349 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:08:04.349 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:08:04.349 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:08:04.349 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:08:04.350 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:08:04.350 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:08:04.350 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 134, 2022-06-29T16:08:04.350 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:08:04.350 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:08:04.351 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:08:04.351 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:04.351 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.351 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:04.352 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6808", 2022-06-29T16:08:04.352 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:08:04.352 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.352 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.352 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:04.353 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6809", 2022-06-29T16:08:04.353 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:08:04.353 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.353 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.354 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.354 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:08:04.354 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:04.354 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.354 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:04.355 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6810", 2022-06-29T16:08:04.355 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:08:04.355 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.355 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.356 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:04.356 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6811", 2022-06-29T16:08:04.356 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:08:04.356 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.356 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.357 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.357 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:08:04.357 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:04.357 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.358 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:04.358 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6814", 2022-06-29T16:08:04.358 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:08:04.358 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.359 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.359 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:04.359 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6817", 2022-06-29T16:08:04.359 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:08:04.360 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.360 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.360 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.360 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:08:04.360 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:04.361 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.361 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:04.361 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6812", 2022-06-29T16:08:04.361 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:08:04.361 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.362 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.362 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:04.362 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6813", 2022-06-29T16:08:04.362 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:08:04.363 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.363 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.363 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.363 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6809/12891", 2022-06-29T16:08:04.363 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6811/12891", 2022-06-29T16:08:04.364 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6817/12891", 2022-06-29T16:08:04.364 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6813/12891", 2022-06-29T16:08:04.364 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:08:04.364 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:08:04.365 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:08:04.365 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.365 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.365 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.365 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:08:04.366 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "2581d351-973a-4b4e-85ca-3658f0068765", 2022-06-29T16:08:04.366 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:08:04.366 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:08:04.366 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:08:04.366 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:08:04.367 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:08:04.367 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:08:04.367 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:08:04.367 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 125, 2022-06-29T16:08:04.368 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:08:04.368 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:08:04.368 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:08:04.368 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:04.368 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.369 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:04.369 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6800", 2022-06-29T16:08:04.369 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:08:04.369 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.369 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.370 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:04.370 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6801", 2022-06-29T16:08:04.370 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:08:04.370 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.371 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.371 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.371 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:08:04.371 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:04.372 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.372 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:04.372 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6802", 2022-06-29T16:08:04.372 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:08:04.372 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.373 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.373 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:04.373 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6803", 2022-06-29T16:08:04.374 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:08:04.374 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.374 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.374 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.374 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:08:04.375 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:04.375 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.375 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:04.375 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6806", 2022-06-29T16:08:04.376 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:08:04.376 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.376 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.376 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:04.376 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6807", 2022-06-29T16:08:04.377 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:08:04.377 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.377 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.377 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.378 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:08:04.378 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:04.378 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.378 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:04.379 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6804", 2022-06-29T16:08:04.379 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:08:04.379 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.379 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.380 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:04.380 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6805", 2022-06-29T16:08:04.380 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:08:04.380 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.380 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.381 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.381 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6801/12889", 2022-06-29T16:08:04.381 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6803/12889", 2022-06-29T16:08:04.381 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6807/12889", 2022-06-29T16:08:04.382 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6805/12889", 2022-06-29T16:08:04.382 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:08:04.382 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:08:04.382 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:08:04.383 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.383 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.383 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.383 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:08:04.383 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "3d234db9-87cd-4fe9-814f-0240a287d8b4", 2022-06-29T16:08:04.384 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:08:04.384 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:08:04.384 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:08:04.384 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:08:04.385 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:08:04.385 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:08:04.385 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:08:04.385 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 134, 2022-06-29T16:08:04.385 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:08:04.386 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:08:04.386 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:08:04.391 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:04.391 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.391 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:04.392 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6808", 2022-06-29T16:08:04.392 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:08:04.392 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.392 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.393 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:04.393 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6810", 2022-06-29T16:08:04.393 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:08:04.393 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.393 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.394 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.394 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:08:04.394 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:04.394 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.395 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:04.395 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6812", 2022-06-29T16:08:04.395 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:08:04.395 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.395 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.396 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:04.396 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6814", 2022-06-29T16:08:04.396 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:08:04.396 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.396 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.397 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.397 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:08:04.397 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:04.397 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.398 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:04.398 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6823", 2022-06-29T16:08:04.398 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:08:04.398 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.398 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.399 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:04.399 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6825", 2022-06-29T16:08:04.399 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:08:04.399 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.400 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.400 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.400 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:08:04.400 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:04.400 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.401 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:04.401 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6817", 2022-06-29T16:08:04.401 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:08:04.401 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.401 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.402 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:04.402 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6820", 2022-06-29T16:08:04.402 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:08:04.402 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.403 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.403 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.403 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6810/13249", 2022-06-29T16:08:04.403 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6814/13249", 2022-06-29T16:08:04.403 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6825/13249", 2022-06-29T16:08:04.404 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6820/13249", 2022-06-29T16:08:04.404 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:08:04.404 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:08:04.404 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:08:04.405 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.405 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.405 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.405 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:08:04.405 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "e4bf1104-ff5c-438b-b608-8cee31c163f6", 2022-06-29T16:08:04.406 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:08:04.406 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:08:04.406 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:08:04.406 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:08:04.407 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:08:04.407 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:08:04.407 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:08:04.407 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 135, 2022-06-29T16:08:04.408 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:08:04.408 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:08:04.408 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:08:04.408 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:04.408 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.409 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:04.409 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6809", 2022-06-29T16:08:04.409 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:08:04.409 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.409 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.410 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:04.410 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6811", 2022-06-29T16:08:04.410 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:08:04.410 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.411 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.411 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.411 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:08:04.411 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:04.411 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.412 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:04.412 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6813", 2022-06-29T16:08:04.412 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:08:04.412 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.412 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.413 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:04.413 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6815", 2022-06-29T16:08:04.413 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:08:04.413 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.414 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.414 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.414 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:08:04.414 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:04.414 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.415 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:04.415 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6824", 2022-06-29T16:08:04.415 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:08:04.415 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.416 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.416 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:04.416 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6827", 2022-06-29T16:08:04.416 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:08:04.417 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.417 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.417 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.417 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:08:04.417 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:04.418 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.418 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:04.418 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6818", 2022-06-29T16:08:04.418 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:08:04.419 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.419 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.419 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:04.419 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6821", 2022-06-29T16:08:04.419 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:08:04.420 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.420 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.420 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.420 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6811/13248", 2022-06-29T16:08:04.420 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6815/13248", 2022-06-29T16:08:04.421 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6827/13248", 2022-06-29T16:08:04.421 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6821/13248", 2022-06-29T16:08:04.421 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:08:04.421 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:08:04.422 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:08:04.422 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.422 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.422 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.422 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:08:04.423 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "8564dd25-27ab-4203-b54e-af6d3aa05857", 2022-06-29T16:08:04.423 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:08:04.423 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:08:04.423 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:08:04.423 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:08:04.424 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:08:04.424 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:08:04.424 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:08:04.424 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 135, 2022-06-29T16:08:04.425 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:08:04.425 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:08:04.425 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:08:04.425 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:04.425 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.426 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:04.426 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6816", 2022-06-29T16:08:04.426 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:08:04.426 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.427 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.427 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:04.427 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6819", 2022-06-29T16:08:04.427 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:08:04.427 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.428 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.428 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.428 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:08:04.428 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:04.429 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.429 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:04.429 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6822", 2022-06-29T16:08:04.429 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:08:04.429 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.430 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.430 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:04.430 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6826", 2022-06-29T16:08:04.430 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:08:04.431 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.431 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.431 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.431 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:08:04.432 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:04.432 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.432 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:04.432 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6830", 2022-06-29T16:08:04.432 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:08:04.433 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.433 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.433 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:04.433 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6831", 2022-06-29T16:08:04.434 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:08:04.434 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.434 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.434 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.434 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:08:04.435 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:04.435 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.435 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:04.435 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6828", 2022-06-29T16:08:04.436 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:08:04.436 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.436 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.436 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:04.436 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6829", 2022-06-29T16:08:04.437 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:08:04.437 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.437 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.437 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.438 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6819/13250", 2022-06-29T16:08:04.438 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6826/13250", 2022-06-29T16:08:04.438 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6831/13250", 2022-06-29T16:08:04.438 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6829/13250", 2022-06-29T16:08:04.438 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:08:04.439 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:08:04.439 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:08:04.439 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.439 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.440 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.440 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:08:04.440 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "bd513a9c-3b19-4314-9897-d300b5c897bb", 2022-06-29T16:08:04.440 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:08:04.440 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:08:04.441 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:08:04.441 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:08:04.441 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:08:04.441 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:08:04.441 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:08:04.442 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 135, 2022-06-29T16:08:04.442 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:08:04.442 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:08:04.442 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:08:04.443 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:04.443 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.443 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:04.443 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6800", 2022-06-29T16:08:04.444 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:08:04.444 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.444 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.444 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:04.444 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6801", 2022-06-29T16:08:04.445 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:08:04.445 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.445 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.445 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.446 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:08:04.446 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:04.446 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.447 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:04.447 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6802", 2022-06-29T16:08:04.447 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:08:04.447 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.448 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.448 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:04.448 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6803", 2022-06-29T16:08:04.448 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:08:04.448 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.449 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.449 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.449 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:08:04.449 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:04.449 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.450 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:04.450 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6806", 2022-06-29T16:08:04.450 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:08:04.450 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.451 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.451 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:04.451 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6807", 2022-06-29T16:08:04.451 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:08:04.451 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.452 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.452 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.452 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:08:04.452 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:04.452 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.453 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:04.453 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6804", 2022-06-29T16:08:04.453 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:08:04.453 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.454 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.454 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:04.454 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6805", 2022-06-29T16:08:04.454 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:08:04.455 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.455 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.455 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.455 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6801/13247", 2022-06-29T16:08:04.455 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6803/13247", 2022-06-29T16:08:04.456 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6807/13247", 2022-06-29T16:08:04.456 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6805/13247", 2022-06-29T16:08:04.456 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:08:04.456 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:08:04.456 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:08:04.457 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:04.457 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.457 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:08:04.457 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_xinfo": [ 2022-06-29T16:08:04.458 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.458 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:08:04.458 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:08:04.458 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:08:04.458 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:08:04.459 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:08:04.459 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:08:04.459 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.891342+0000", 2022-06-29T16:08:04.459 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:08:04.459 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.460 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.460 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:08:04.460 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:08:04.460 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:08:04.461 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:08:04.461 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:08:04.461 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:08:04.461 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.900866+0000", 2022-06-29T16:08:04.461 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:08:04.462 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.462 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.462 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:08:04.462 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:08:04.463 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:08:04.463 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:08:04.463 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:08:04.463 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:08:04.463 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.912755+0000", 2022-06-29T16:08:04.464 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:08:04.464 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.464 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.464 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:08:04.464 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:08:04.465 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:08:04.465 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:08:04.465 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:08:04.465 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:08:04.466 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.864332+0000", 2022-06-29T16:08:04.466 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:08:04.466 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.466 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.466 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:08:04.467 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:08:04.467 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:08:04.467 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:08:04.467 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:08:04.468 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:08:04.468 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.863756+0000", 2022-06-29T16:08:04.468 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:08:04.468 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.468 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.469 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:08:04.469 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:08:04.469 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:08:04.469 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:08:04.469 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:08:04.470 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:08:04.470 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.869041+0000", 2022-06-29T16:08:04.470 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:08:04.470 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.471 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.471 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:08:04.471 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:08:04.471 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:08:04.471 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:08:04.472 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:08:04.472 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:08:04.472 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.895043+0000", 2022-06-29T16:08:04.472 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:08:04.472 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.473 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:04.473 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:08:04.473 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:08:04.473 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:08:04.474 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:08:04.474 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:08:04.474 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:08:04.474 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.889491+0000", 2022-06-29T16:08:04.474 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:08:04.475 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.475 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:08:04.475 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap": [], 2022-06-29T16:08:04.475 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap_items": [], 2022-06-29T16:08:04.475 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_temp": [], 2022-06-29T16:08:04.476 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_temp": [], 2022-06-29T16:08:04.476 INFO:teuthology.orchestra.run.smithi060.stdout: "blacklist": {}, 2022-06-29T16:08:04.476 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profiles": { 2022-06-29T16:08:04.476 INFO:teuthology.orchestra.run.smithi060.stdout: "default": { 2022-06-29T16:08:04.477 INFO:teuthology.orchestra.run.smithi060.stdout: "crush-failure-domain": "osd", 2022-06-29T16:08:04.477 INFO:teuthology.orchestra.run.smithi060.stdout: "k": "2", 2022-06-29T16:08:04.477 INFO:teuthology.orchestra.run.smithi060.stdout: "m": "1", 2022-06-29T16:08:04.477 INFO:teuthology.orchestra.run.smithi060.stdout: "plugin": "jerasure", 2022-06-29T16:08:04.477 INFO:teuthology.orchestra.run.smithi060.stdout: "ruleset-failure-domain": "osd", 2022-06-29T16:08:04.478 INFO:teuthology.orchestra.run.smithi060.stdout: "technique": "reed_sol_van" 2022-06-29T16:08:04.478 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:04.478 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:04.478 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps_queue": [], 2022-06-29T16:08:04.479 INFO:teuthology.orchestra.run.smithi060.stdout: "new_removed_snaps": [], 2022-06-29T16:08:04.479 INFO:teuthology.orchestra.run.smithi060.stdout: "new_purged_snaps": [], 2022-06-29T16:08:04.479 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_node_flags": {}, 2022-06-29T16:08:04.479 INFO:teuthology.orchestra.run.smithi060.stdout: "device_class_flags": {} 2022-06-29T16:08:04.479 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:08:04.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.233+0000 7f760d112700 1 -- 172.21.15.60:0/194244530 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f75ec053a70 msgr2=0x7f75ec055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:04.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.233+0000 7f760d112700 1 --2- 172.21.15.60:0/194244530 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f75ec053a70 0x7f75ec055f30 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0x7f75f000c000 tx=0x7f75f0008040).stop 2022-06-29T16:08:04.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.233+0000 7f760d112700 1 -- 172.21.15.60:0/194244530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76080b5950 msgr2=0x7f76081595b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:04.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.233+0000 7f760d112700 1 --2- 172.21.15.60:0/194244530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76080b5950 0x7f76081595b0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 rx=0x7f760805ff90 tx=0x7f75f8007c10).stop 2022-06-29T16:08:04.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.233+0000 7f760d112700 1 -- 172.21.15.60:0/194244530 shutdown_connections 2022-06-29T16:08:04.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.233+0000 7f760d112700 1 --2- 172.21.15.60:0/194244530 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f75ec053a70 0x7f75ec055f30 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:04.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.233+0000 7f760d112700 1 --2- 172.21.15.60:0/194244530 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f760814af10 0x7f7608159aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:04.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.233+0000 7f760d112700 1 --2- 172.21.15.60:0/194244530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76080b5950 0x7f76081595b0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:04.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.233+0000 7f760d112700 1 --2- 172.21.15.60:0/194244530 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7608063ad0 0x7f76080b07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:04.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.233+0000 7f760d112700 1 -- 172.21.15.60:0/194244530 >> 172.21.15.60:0/194244530 conn(0x7f76080632c0 msgr2=0x7f7608146ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:04.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.233+0000 7f760d112700 1 -- 172.21.15.60:0/194244530 shutdown_connections 2022-06-29T16:08:04.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.233+0000 7f760d112700 1 -- 172.21.15.60:0/194244530 wait complete. 2022-06-29T16:08:04.485 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:08:04.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.589+0000 7fd4d5919700 1 Processor -- start 2022-06-29T16:08:04.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.589+0000 7fd4d5919700 1 -- start start 2022-06-29T16:08:04.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.589+0000 7fd4d5919700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4d00b2130 0x7fd4d0146100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:04.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.589+0000 7fd4d5919700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd4d00aed00 0x7fd4d01465f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:04.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.589+0000 7fd4d5919700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4d005d450 con 0x7fd4d005d070 2022-06-29T16:08:04.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.589+0000 7fd4d5919700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4d00adf90 con 0x7fd4d00aed00 2022-06-29T16:08:04.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.589+0000 7fd4d5919700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d00b25d0 con 0x7fd4d00b2130 2022-06-29T16:08:04.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.589+0000 7fd4d5118700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd4d00aed00 0x7fd4d01465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:04.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.589+0000 7fd4d5118700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd4d00aed00 0x7fd4d01465f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53978/0 (socket says 172.21.15.60:53978) 2022-06-29T16:08:04.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.593+0000 7fd4d5118700 1 -- 172.21.15.60:0/370960520 learned_addr learned my addr 172.21.15.60:0/370960520 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:04.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.593+0000 7fd4cffff700 1 --2- 172.21.15.60:0/370960520 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4d00b2130 0x7fd4d0146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:04.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.593+0000 7fd4d5118700 1 -- 172.21.15.60:0/370960520 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4d00b2130 msgr2=0x7fd4d0146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:04.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.593+0000 7fd4d5118700 1 --2- 172.21.15.60:0/370960520 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4d00b2130 0x7fd4d0146100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:04.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.593+0000 7fd4d5118700 1 -- 172.21.15.60:0/370960520 >> v1:172.21.15.92:6789/0 conn(0x7fd4d005d070 legacy=0x7fd4d014aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:04.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.593+0000 7fd4d5118700 1 -- 172.21.15.60:0/370960520 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d014a350 con 0x7fd4d00aed00 2022-06-29T16:08:04.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.593+0000 7fd4d5118700 1 --2- 172.21.15.60:0/370960520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4d00aed00 0x7fd4d01465f0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 rx=0x7fd4c80032f0 tx=0x7fd4c8009290).ready entity=mon.0 client_cookie=1b66258b30bd9a95 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:04.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.593+0000 7fd4cf7fe700 1 -- 172.21.15.60:0/370960520 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4c800b8a0 con 0x7fd4d00aed00 2022-06-29T16:08:04.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.593+0000 7fd4cf7fe700 1 -- 172.21.15.60:0/370960520 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4c800ba00 con 0x7fd4d00aed00 2022-06-29T16:08:04.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.593+0000 7fd4cf7fe700 1 -- 172.21.15.60:0/370960520 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4c800bb70 con 0x7fd4d00aed00 2022-06-29T16:08:04.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.593+0000 7fd4d5919700 1 -- 172.21.15.60:0/370960520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4d00aed00 msgr2=0x7fd4d01465f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:04.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.593+0000 7fd4d5919700 1 --2- 172.21.15.60:0/370960520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4d00aed00 0x7fd4d01465f0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 rx=0x7fd4c80032f0 tx=0x7fd4c8009290).stop 2022-06-29T16:08:04.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.593+0000 7fd4d5919700 1 -- 172.21.15.60:0/370960520 shutdown_connections 2022-06-29T16:08:04.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.593+0000 7fd4d5919700 1 --2- 172.21.15.60:0/370960520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4d00aed00 0x7fd4d01465f0 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:04.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.593+0000 7fd4d5919700 1 --2- 172.21.15.60:0/370960520 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4d00b2130 0x7fd4d0146100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:04.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.593+0000 7fd4d5919700 1 -- 172.21.15.60:0/370960520 >> 172.21.15.60:0/370960520 conn(0x7fd4d00b5890 msgr2=0x7fd4d00b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:04.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.593+0000 7fd4d5919700 1 -- 172.21.15.60:0/370960520 shutdown_connections 2022-06-29T16:08:04.607 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.593+0000 7fd4d5919700 1 -- 172.21.15.60:0/370960520 wait complete. 2022-06-29T16:08:04.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.597+0000 7fd4d5919700 1 Processor -- start 2022-06-29T16:08:04.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.597+0000 7fd4d5919700 1 -- start start 2022-06-29T16:08:04.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.597+0000 7fd4d5919700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4d005d070 0x7fd4d0118e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:04.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.597+0000 7fd4d5919700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4d00aed00 0x7fd4d0116920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:04.612 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.597+0000 7fd4d5919700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4d00b2130 0x7fd4d0065210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:04.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.597+0000 7fd4d5919700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d0118420 con 0x7fd4d005d070 2022-06-29T16:08:04.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.597+0000 7fd4d5919700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d01186c0 con 0x7fd4d00aed00 2022-06-29T16:08:04.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.597+0000 7fd4d5919700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d0118960 con 0x7fd4d00b2130 2022-06-29T16:08:04.613 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.597+0000 7fd4d4917700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4d005d070 0x7fd4d0118e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:04.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.597+0000 7fd4d4917700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4d005d070 0x7fd4d0118e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53980/0 (socket says 172.21.15.60:53980) 2022-06-29T16:08:04.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.597+0000 7fd4d4917700 1 -- 172.21.15.60:0/1412534336 learned_addr learned my addr 172.21.15.60:0/1412534336 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:04.614 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.597+0000 7fd4cffff700 1 --2- 172.21.15.60:0/1412534336 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4d00aed00 0x7fd4d0116920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:04.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.597+0000 7fd4d5118700 1 --2- 172.21.15.60:0/1412534336 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4d00b2130 0x7fd4d0065210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:04.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.597+0000 7fd4d4917700 1 -- 172.21.15.60:0/1412534336 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4d00b2130 msgr2=0x7fd4d0065210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:04.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.597+0000 7fd4d4917700 1 --2- 172.21.15.60:0/1412534336 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4d00b2130 0x7fd4d0065210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:04.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.597+0000 7fd4d4917700 1 -- 172.21.15.60:0/1412534336 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4d00aed00 msgr2=0x7fd4d0116920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:04.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.597+0000 7fd4d4917700 1 --2- 172.21.15.60:0/1412534336 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4d00aed00 0x7fd4d0116920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:04.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.597+0000 7fd4d4917700 1 -- 172.21.15.60:0/1412534336 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c8009530 con 0x7fd4d005d070 2022-06-29T16:08:04.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.597+0000 7fd4cffff700 1 --2- 172.21.15.60:0/1412534336 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4d00aed00 0x7fd4d0116920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:08:04.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.597+0000 7fd4d5118700 1 --2- 172.21.15.60:0/1412534336 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4d00b2130 0x7fd4d0065210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:08:04.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.597+0000 7fd4d4917700 1 --2- 172.21.15.60:0/1412534336 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4d005d070 0x7fd4d0118e00 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 rx=0x7fd4d014a970 tx=0x7fd4c4007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:04.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.601+0000 7fd4cdffb700 1 -- 172.21.15.60:0/1412534336 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4c400ce30 con 0x7fd4d005d070 2022-06-29T16:08:04.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.601+0000 7fd4cdffb700 1 -- 172.21.15.60:0/1412534336 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4c4007240 con 0x7fd4d005d070 2022-06-29T16:08:04.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.601+0000 7fd4d5919700 1 -- 172.21.15.60:0/1412534336 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4d0116e10 con 0x7fd4d005d070 2022-06-29T16:08:04.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.601+0000 7fd4cdffb700 1 -- 172.21.15.60:0/1412534336 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4c4006510 con 0x7fd4d005d070 2022-06-29T16:08:04.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.601+0000 7fd4d5919700 1 -- 172.21.15.60:0/1412534336 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4d0065920 con 0x7fd4d005d070 2022-06-29T16:08:04.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.601+0000 7fd4d5919700 1 -- 172.21.15.60:0/1412534336 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4d014a5f0 con 0x7fd4d005d070 2022-06-29T16:08:04.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.601+0000 7fd4cdffb700 1 -- 172.21.15.60:0/1412534336 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd4c4006670 con 0x7fd4d005d070 2022-06-29T16:08:04.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.601+0000 7fd4cdffb700 1 --2- 172.21.15.60:0/1412534336 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd4b0053a70 0x7fd4b0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:04.619 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.605+0000 7fd4cdffb700 1 -- 172.21.15.60:0/1412534336 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd4c406b8b0 con 0x7fd4d005d070 2022-06-29T16:08:04.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.605+0000 7fd4cffff700 1 --2- 172.21.15.60:0/1412534336 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd4b0053a70 0x7fd4b0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:04.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.605+0000 7fd4cffff700 1 --2- 172.21.15.60:0/1412534336 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd4b0053a70 0x7fd4b0055f30 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 rx=0x7fd4bc00b000 tx=0x7fd4bc008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:04.621 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.609+0000 7fd4cdffb700 1 -- 172.21.15.60:0/1412534336 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd4c4042770 con 0x7fd4d005d070 2022-06-29T16:08:04.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.773+0000 7fd4d5919700 1 -- 172.21.15.60:0/1412534336 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4d00573d0 con 0x7fd4d005d070 2022-06-29T16:08:04.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.777+0000 7fd4cdffb700 1 -- 172.21.15.60:0/1412534336 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2142 (secure 0 0 0) 0x7fd4c4019030 con 0x7fd4d005d070 2022-06-29T16:08:04.781 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:08:04.782 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":68,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.92:6835/2628497783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2628497783},{"type":"v1","addr":"172.21.15.92:6835","nonce":2628497783}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":67}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:07:54.174936+0000","modified":"2022-06-29T16:07:55.178863+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8463":{"gid":8463,"name":"a","rank":0,"incarnation":67,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/2222627281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":2222627281},{"type":"v1","addr":"172.21.15.60:6833","nonce":2222627281}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2022-06-29T16:08:04.784 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.777+0000 7fd4d5919700 1 -- 172.21.15.60:0/1412534336 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd4b0053a70 msgr2=0x7fd4b0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:04.784 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.777+0000 7fd4d5919700 1 --2- 172.21.15.60:0/1412534336 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd4b0053a70 0x7fd4b0055f30 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 rx=0x7fd4bc00b000 tx=0x7fd4bc008040).stop 2022-06-29T16:08:04.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.777+0000 7fd4d5919700 1 -- 172.21.15.60:0/1412534336 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4d005d070 msgr2=0x7fd4d0118e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:04.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.777+0000 7fd4d5919700 1 --2- 172.21.15.60:0/1412534336 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4d005d070 0x7fd4d0118e00 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 rx=0x7fd4d014a970 tx=0x7fd4c4007c00).stop 2022-06-29T16:08:04.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.777+0000 7fd4d5919700 1 -- 172.21.15.60:0/1412534336 shutdown_connections 2022-06-29T16:08:04.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.777+0000 7fd4d5919700 1 --2- 172.21.15.60:0/1412534336 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd4b0053a70 0x7fd4b0055f30 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:04.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.777+0000 7fd4d5919700 1 --2- 172.21.15.60:0/1412534336 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd4d00b2130 0x7fd4d0065210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:04.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.777+0000 7fd4d5919700 1 --2- 172.21.15.60:0/1412534336 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4d00aed00 0x7fd4d0116920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:04.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.777+0000 7fd4d5919700 1 --2- 172.21.15.60:0/1412534336 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4d005d070 0x7fd4d0118e00 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:04.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.777+0000 7fd4d5919700 1 -- 172.21.15.60:0/1412534336 >> 172.21.15.60:0/1412534336 conn(0x7fd4d00b5890 msgr2=0x7fd4d01470c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:04.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.777+0000 7fd4d5919700 1 -- 172.21.15.60:0/1412534336 shutdown_connections 2022-06-29T16:08:04.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.777+0000 7fd4d5919700 1 -- 172.21.15.60:0/1412534336 wait complete. 2022-06-29T16:08:04.787 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 68 2022-06-29T16:08:04.794 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-06-29T16:08:04.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.893+0000 7fe806717700 1 Processor -- start 2022-06-29T16:08:04.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.893+0000 7fe806717700 1 -- start start 2022-06-29T16:08:04.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.893+0000 7fe806717700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe8000b5950 0x7fe800067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:04.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.893+0000 7fe806717700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe80014af10 0x7fe80005c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:04.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.893+0000 7fe806717700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8000acba0 con 0x7fe800063ad0 2022-06-29T16:08:04.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.893+0000 7fe806717700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe80014b3a0 con 0x7fe8000b5950 2022-06-29T16:08:04.903 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.893+0000 7fe806717700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe800067220 con 0x7fe80014af10 2022-06-29T16:08:04.903 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.893+0000 7fe7fffff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe8000b5950 0x7fe800067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:04.903 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.893+0000 7fe7fffff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe8000b5950 0x7fe800067680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53992/0 (socket says 172.21.15.60:53992) 2022-06-29T16:08:04.903 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.893+0000 7fe7fffff700 1 -- 172.21.15.60:0/2952531547 learned_addr learned my addr 172.21.15.60:0/2952531547 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:04.904 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.893+0000 7fe804cb4700 1 --2- 172.21.15.60:0/2952531547 >> v2:172.21.15.92:3300/0 conn(0x7fe80014af10 0x7fe80005c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:04.904 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.893+0000 7fe7fffff700 1 -- 172.21.15.60:0/2952531547 >> v2:172.21.15.92:3300/0 conn(0x7fe80014af10 msgr2=0x7fe80005c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:04.904 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.893+0000 7fe7fffff700 1 --2- 172.21.15.60:0/2952531547 >> v2:172.21.15.92:3300/0 conn(0x7fe80014af10 0x7fe80005c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:04.904 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.893+0000 7fe7fffff700 1 -- 172.21.15.60:0/2952531547 >> v1:172.21.15.92:6789/0 conn(0x7fe800063ad0 legacy=0x7fe80005bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:04.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.893+0000 7fe7fffff700 1 -- 172.21.15.60:0/2952531547 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8000b5d30 con 0x7fe8000b5950 2022-06-29T16:08:04.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.893+0000 7fe7fffff700 1 --2- 172.21.15.60:0/2952531547 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8000b5950 0x7fe800067680 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 rx=0x7fe7f8002f70 tx=0x7fe7f8006eb0).ready entity=mon.0 client_cookie=d772b0c2779e0d35 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:04.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.893+0000 7fe7feffd700 1 -- 172.21.15.60:0/2952531547 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe7f800d5c0 con 0x7fe8000b5950 2022-06-29T16:08:04.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.897+0000 7fe7feffd700 1 -- 172.21.15.60:0/2952531547 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f8000e90 con 0x7fe8000b5950 2022-06-29T16:08:04.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.897+0000 7fe7feffd700 1 -- 172.21.15.60:0/2952531547 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe7f800d950 con 0x7fe8000b5950 2022-06-29T16:08:04.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.897+0000 7fe806717700 1 -- 172.21.15.60:0/2952531547 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8000b5950 msgr2=0x7fe800067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:04.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.897+0000 7fe806717700 1 --2- 172.21.15.60:0/2952531547 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8000b5950 0x7fe800067680 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 rx=0x7fe7f8002f70 tx=0x7fe7f8006eb0).stop 2022-06-29T16:08:04.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.897+0000 7fe806717700 1 -- 172.21.15.60:0/2952531547 shutdown_connections 2022-06-29T16:08:04.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.897+0000 7fe806717700 1 --2- 172.21.15.60:0/2952531547 >> v2:172.21.15.92:3300/0 conn(0x7fe80014af10 0x7fe80005c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:04.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.897+0000 7fe806717700 1 --2- 172.21.15.60:0/2952531547 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8000b5950 0x7fe800067680 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:04.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.897+0000 7fe806717700 1 -- 172.21.15.60:0/2952531547 >> 172.21.15.60:0/2952531547 conn(0x7fe8000632c0 msgr2=0x7fe8000b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:04.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.897+0000 7fe806717700 1 -- 172.21.15.60:0/2952531547 shutdown_connections 2022-06-29T16:08:04.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.897+0000 7fe806717700 1 -- 172.21.15.60:0/2952531547 wait complete. 2022-06-29T16:08:04.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.897+0000 7fe806717700 1 Processor -- start 2022-06-29T16:08:04.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.897+0000 7fe806717700 1 -- start start 2022-06-29T16:08:04.915 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.897+0000 7fe806717700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe800063ad0 0x7fe8000b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:04.915 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.897+0000 7fe806717700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe8000b5950 0x7fe8000642f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:04.915 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.897+0000 7fe806717700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe80014af10 0x7fe800156590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:04.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.897+0000 7fe806717700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8000b43a0 con 0x7fe80014af10 2022-06-29T16:08:04.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.897+0000 7fe806717700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe800064c50 con 0x7fe8000b5950 2022-06-29T16:08:04.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.897+0000 7fe806717700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8000b45d0 con 0x7fe800063ad0 2022-06-29T16:08:04.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.897+0000 7fe804cb4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe80014af10 0x7fe800156590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:04.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.897+0000 7fe804cb4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe80014af10 0x7fe800156590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53998/0 (socket says 172.21.15.60:53998) 2022-06-29T16:08:04.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.897+0000 7fe804cb4700 1 -- 172.21.15.60:0/753174595 learned_addr learned my addr 172.21.15.60:0/753174595 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:04.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.897+0000 7fe7fffff700 1 --2- 172.21.15.60:0/753174595 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe800063ad0 0x7fe8000b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:04.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.897+0000 7fe7ff7fe700 1 --2- 172.21.15.60:0/753174595 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe8000b5950 0x7fe8000642f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:04.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.897+0000 7fe804cb4700 1 -- 172.21.15.60:0/753174595 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe800063ad0 msgr2=0x7fe8000b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:04.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.897+0000 7fe804cb4700 1 --2- 172.21.15.60:0/753174595 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe800063ad0 0x7fe8000b2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:04.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.897+0000 7fe804cb4700 1 -- 172.21.15.60:0/753174595 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe8000b5950 msgr2=0x7fe8000642f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:04.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.897+0000 7fe804cb4700 1 --2- 172.21.15.60:0/753174595 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe8000b5950 0x7fe8000642f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:04.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.897+0000 7fe804cb4700 1 -- 172.21.15.60:0/753174595 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f800b1e0 con 0x7fe80014af10 2022-06-29T16:08:04.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.897+0000 7fe804cb4700 1 --2- 172.21.15.60:0/753174595 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe80014af10 0x7fe800156590 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 rx=0x7fe7f000c000 tx=0x7fe7f0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:04.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.897+0000 7fe7fd7fa700 1 -- 172.21.15.60:0/753174595 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe7f0010780 con 0x7fe80014af10 2022-06-29T16:08:04.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.897+0000 7fe7fd7fa700 1 -- 172.21.15.60:0/753174595 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f00108e0 con 0x7fe80014af10 2022-06-29T16:08:04.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.897+0000 7fe806717700 1 -- 172.21.15.60:0/753174595 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8000b4870 con 0x7fe80014af10 2022-06-29T16:08:04.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.897+0000 7fe7fd7fa700 1 -- 172.21.15.60:0/753174595 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe7f0010c40 con 0x7fe80014af10 2022-06-29T16:08:04.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.901+0000 7fe806717700 1 -- 172.21.15.60:0/753174595 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8000b0d60 con 0x7fe80014af10 2022-06-29T16:08:04.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.901+0000 7fe806717700 1 -- 172.21.15.60:0/753174595 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8000ad420 con 0x7fe80014af10 2022-06-29T16:08:04.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.901+0000 7fe7fd7fa700 1 -- 172.21.15.60:0/753174595 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe7f001c460 con 0x7fe80014af10 2022-06-29T16:08:04.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.901+0000 7fe7fd7fa700 1 --2- 172.21.15.60:0/753174595 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe7d8053a70 0x7fe7d8055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:04.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.901+0000 7fe7fd7fa700 1 -- 172.21.15.60:0/753174595 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe7f006c480 con 0x7fe80014af10 2022-06-29T16:08:04.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.905+0000 7fe7fffff700 1 --2- 172.21.15.60:0/753174595 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe7d8053a70 0x7fe7d8055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:04.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.905+0000 7fe7fffff700 1 --2- 172.21.15.60:0/753174595 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe7d8053a70 0x7fe7d8055f30 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0x7fe7f801a000 tx=0x7fe7f8015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:04.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:04.909+0000 7fe7fd7fa700 1 -- 172.21.15.60:0/753174595 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe8000ad420 con 0x7fe80014af10 2022-06-29T16:08:05.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:05.073+0000 7fe806717700 1 -- 172.21.15.60:0/753174595 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe8001562f0 con 0x7fe80014af10 2022-06-29T16:08:05.960 INFO:tasks.ceph.mon.a.smithi060.stderr:2022-06-29T16:08:05.957+0000 7fcc1a376700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-06-29T16:08:05.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:05.969+0000 7fe7fd7fa700 1 -- 172.21.15.60:0/753174595 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v69) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe8001562f0 con 0x7fe80014af10 2022-06-29T16:08:05.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:05.973+0000 7fe806717700 1 -- 172.21.15.60:0/753174595 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe7d8053a70 msgr2=0x7fe7d8055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:05.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:05.973+0000 7fe806717700 1 --2- 172.21.15.60:0/753174595 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe7d8053a70 0x7fe7d8055f30 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0x7fe7f801a000 tx=0x7fe7f8015000).stop 2022-06-29T16:08:05.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:05.973+0000 7fe806717700 1 -- 172.21.15.60:0/753174595 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe80014af10 msgr2=0x7fe800156590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:05.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:05.973+0000 7fe806717700 1 --2- 172.21.15.60:0/753174595 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe80014af10 0x7fe800156590 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 rx=0x7fe7f000c000 tx=0x7fe7f0006df0).stop 2022-06-29T16:08:05.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:05.973+0000 7fe806717700 1 -- 172.21.15.60:0/753174595 shutdown_connections 2022-06-29T16:08:05.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:05.973+0000 7fe806717700 1 --2- 172.21.15.60:0/753174595 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe7d8053a70 0x7fe7d8055f30 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:05.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:05.973+0000 7fe806717700 1 --2- 172.21.15.60:0/753174595 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe80014af10 0x7fe800156590 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:05.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:05.973+0000 7fe806717700 1 --2- 172.21.15.60:0/753174595 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe8000b5950 0x7fe8000642f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:05.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:05.973+0000 7fe806717700 1 --2- 172.21.15.60:0/753174595 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe800063ad0 0x7fe8000b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:05.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:05.973+0000 7fe806717700 1 -- 172.21.15.60:0/753174595 >> 172.21.15.60:0/753174595 conn(0x7fe8000632c0 msgr2=0x7fe8000651f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:05.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:05.973+0000 7fe806717700 1 -- 172.21.15.60:0/753174595 shutdown_connections 2022-06-29T16:08:05.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:05.973+0000 7fe806717700 1 -- 172.21.15.60:0/753174595 wait complete. 2022-06-29T16:08:05.984 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-06-29T16:08:05.999 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:08:06.003 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2022-06-29T16:08:06.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.097+0000 7f718c588700 1 Processor -- start 2022-06-29T16:08:06.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.097+0000 7f718c588700 1 -- start start 2022-06-29T16:08:06.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.101+0000 7f718c588700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f718405f0b0 0x7f71841461e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:06.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.101+0000 7f718c588700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71840b6e30 con 0x7f718405ca50 2022-06-29T16:08:06.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.101+0000 7f718c588700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71840b0dd0 con 0x7f71840b6a50 2022-06-29T16:08:06.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.101+0000 7f718c588700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71840b1070 con 0x7f718405f0b0 2022-06-29T16:08:06.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.101+0000 7f718ab25700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f718405ca50 0x7f718405ce50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33872/0 (socket says 172.21.15.60:33872) 2022-06-29T16:08:06.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.101+0000 7f718ab25700 1 -- 172.21.15.60:0/2316001045 learned_addr learned my addr 172.21.15.60:0/2316001045 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:06.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.101+0000 7f7189b23700 1 --2- 172.21.15.60:0/2316001045 >> v2:172.21.15.92:3300/0 conn(0x7f718405f0b0 0x7f71841461e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:06.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.101+0000 7f7189322700 1 -- 172.21.15.60:0/2316001045 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2565720266 0 0) 0x7f71840b6e30 con 0x7f718405ca50 2022-06-29T16:08:06.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.101+0000 7f7189322700 1 -- 172.21.15.60:0/2316001045 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f716c003260 con 0x7f718405ca50 2022-06-29T16:08:06.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.101+0000 7f7189322700 1 -- 172.21.15.60:0/2316001045 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1498279059 0 0) 0x7f71840b0dd0 con 0x7f71840b6a50 2022-06-29T16:08:06.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.101+0000 7f7189322700 1 -- 172.21.15.60:0/2316001045 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71840b6e30 con 0x7f71840b6a50 2022-06-29T16:08:06.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.101+0000 7f7189b23700 1 -- 172.21.15.60:0/2316001045 >> v1:172.21.15.92:6789/0 conn(0x7f71840b6a50 legacy=0x7f7184142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:06.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.101+0000 7f7189b23700 1 -- 172.21.15.60:0/2316001045 >> v1:172.21.15.60:6789/0 conn(0x7f718405ca50 legacy=0x7f718405ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:06.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.101+0000 7f7189b23700 1 -- 172.21.15.60:0/2316001045 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f718405c5b0 con 0x7f718405f0b0 2022-06-29T16:08:06.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.101+0000 7f7189b23700 1 --2- 172.21.15.60:0/2316001045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f718405f0b0 0x7f71841461e0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0x7f7174002f70 tx=0x7f7174006df0).ready entity=mon.1 client_cookie=52d6b0114e293521 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:06.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.101+0000 7f7189322700 1 -- 172.21.15.60:0/2316001045 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f717400d540 con 0x7f718405f0b0 2022-06-29T16:08:06.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.101+0000 7f7189322700 1 -- 172.21.15.60:0/2316001045 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7174000e90 con 0x7f718405f0b0 2022-06-29T16:08:06.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.101+0000 7f7189322700 1 -- 172.21.15.60:0/2316001045 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f717400d890 con 0x7f718405f0b0 2022-06-29T16:08:06.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.105+0000 7f718c588700 1 -- 172.21.15.60:0/2316001045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f718405f0b0 msgr2=0x7f71841461e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:06.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.105+0000 7f718c588700 1 --2- 172.21.15.60:0/2316001045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f718405f0b0 0x7f71841461e0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0x7f7174002f70 tx=0x7f7174006df0).stop 2022-06-29T16:08:06.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.105+0000 7f718c588700 1 -- 172.21.15.60:0/2316001045 shutdown_connections 2022-06-29T16:08:06.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.105+0000 7f718c588700 1 --2- 172.21.15.60:0/2316001045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f718405f0b0 0x7f71841461e0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:06.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.105+0000 7f718c588700 1 -- 172.21.15.60:0/2316001045 >> 172.21.15.60:0/2316001045 conn(0x7f71840643e0 msgr2=0x7f71840b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:06.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.105+0000 7f718c588700 1 -- 172.21.15.60:0/2316001045 shutdown_connections 2022-06-29T16:08:06.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.105+0000 7f718c588700 1 -- 172.21.15.60:0/2316001045 wait complete. 2022-06-29T16:08:06.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.105+0000 7f718c588700 1 Processor -- start 2022-06-29T16:08:06.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.105+0000 7f718c588700 1 -- start start 2022-06-29T16:08:06.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.105+0000 7f718c588700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f718405ca50 0x7f71840b55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:06.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.105+0000 7f718c588700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f718405f0b0 0x7f71840af500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:06.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.105+0000 7f718c588700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71840b6a50 0x7f71840af9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:06.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.105+0000 7f718c588700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71840ae1e0 con 0x7f71840b6a50 2022-06-29T16:08:06.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.105+0000 7f718c588700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71840675d0 con 0x7f718405f0b0 2022-06-29T16:08:06.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.105+0000 7f718c588700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7184066320 con 0x7f718405ca50 2022-06-29T16:08:06.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.105+0000 7f718ab25700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71840b6a50 0x7f71840af9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:06.117 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.109+0000 7f718ab25700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71840b6a50 0x7f71840af9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54024/0 (socket says 172.21.15.60:54024) 2022-06-29T16:08:06.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.109+0000 7f718ab25700 1 -- 172.21.15.60:0/3268516053 learned_addr learned my addr 172.21.15.60:0/3268516053 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:06.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.109+0000 7f7189b23700 1 --2- 172.21.15.60:0/3268516053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f718405f0b0 0x7f71840af500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:06.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.109+0000 7f718a324700 1 --2- 172.21.15.60:0/3268516053 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f718405ca50 0x7f71840b55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:06.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.109+0000 7f718ab25700 1 -- 172.21.15.60:0/3268516053 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f718405ca50 msgr2=0x7f71840b55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:06.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.109+0000 7f718ab25700 1 --2- 172.21.15.60:0/3268516053 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f718405ca50 0x7f71840b55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:06.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.109+0000 7f718ab25700 1 -- 172.21.15.60:0/3268516053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f718405f0b0 msgr2=0x7f71840af500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:06.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.109+0000 7f718ab25700 1 --2- 172.21.15.60:0/3268516053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f718405f0b0 0x7f71840af500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:06.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.109+0000 7f718ab25700 1 -- 172.21.15.60:0/3268516053 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f717400b180 con 0x7f71840b6a50 2022-06-29T16:08:06.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.109+0000 7f718a324700 1 --2- 172.21.15.60:0/3268516053 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f718405ca50 0x7f71840b55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:08:06.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.109+0000 7f718ab25700 1 --2- 172.21.15.60:0/3268516053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71840b6a50 0x7f71840af9f0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 rx=0x7f717c001490 tx=0x7f717c007e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:06.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.109+0000 7f71737fe700 1 -- 172.21.15.60:0/3268516053 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f717c005ba0 con 0x7f71840b6a50 2022-06-29T16:08:06.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.109+0000 7f71737fe700 1 -- 172.21.15.60:0/3268516053 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f717c005d00 con 0x7f71840b6a50 2022-06-29T16:08:06.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.109+0000 7f718c588700 1 -- 172.21.15.60:0/3268516053 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71840667c0 con 0x7f71840b6a50 2022-06-29T16:08:06.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.109+0000 7f71737fe700 1 -- 172.21.15.60:0/3268516053 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f717c0072c0 con 0x7f71840b6a50 2022-06-29T16:08:06.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.109+0000 7f718c588700 1 -- 172.21.15.60:0/3268516053 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7184067130 con 0x7f71840b6a50 2022-06-29T16:08:06.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.109+0000 7f71737fe700 1 -- 172.21.15.60:0/3268516053 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f717c01a060 con 0x7f71840b6a50 2022-06-29T16:08:06.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.109+0000 7f718c588700 1 -- 172.21.15.60:0/3268516053 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71840ad660 con 0x7f71840b6a50 2022-06-29T16:08:06.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.109+0000 7f71737fe700 1 --2- 172.21.15.60:0/3268516053 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f71640539e0 0x7f7164055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:06.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.109+0000 7f71737fe700 1 -- 172.21.15.60:0/3268516053 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f717c06b6f0 con 0x7f71840b6a50 2022-06-29T16:08:06.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.109+0000 7f718a324700 1 --2- 172.21.15.60:0/3268516053 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f71640539e0 0x7f7164055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:06.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.113+0000 7f718a324700 1 --2- 172.21.15.60:0/3268516053 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f71640539e0 0x7f7164055ea0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0x7f718000b000 tx=0x7f7180009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:06.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.113+0000 7f71737fe700 1 -- 172.21.15.60:0/3268516053 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f717c042520 con 0x7f71840b6a50 2022-06-29T16:08:06.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.281+0000 7f718c588700 1 -- 172.21.15.60:0/3268516053 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71840664e0 con 0x7f71840b6a50 2022-06-29T16:08:06.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.281+0000 7f71737fe700 1 -- 172.21.15.60:0/3268516053 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+1782 (secure 0 0 0) 0x7f717c018020 con 0x7f71840b6a50 2022-06-29T16:08:06.287 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:08:06.288 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":69,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.92:6835/2628497783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2628497783},{"type":"v1","addr":"172.21.15.92:6835","nonce":2628497783}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":67}],"filesystems":[{"mdsmap":{"epoch":69,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:07:54.174936+0000","modified":"2022-06-29T16:08:05.958068+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":138,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2022-06-29T16:08:06.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.285+0000 7f718c588700 1 -- 172.21.15.60:0/3268516053 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f71640539e0 msgr2=0x7f7164055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:06.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.285+0000 7f718c588700 1 --2- 172.21.15.60:0/3268516053 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f71640539e0 0x7f7164055ea0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0x7f718000b000 tx=0x7f7180009040).stop 2022-06-29T16:08:06.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.285+0000 7f718c588700 1 -- 172.21.15.60:0/3268516053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71840b6a50 msgr2=0x7f71840af9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:06.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.285+0000 7f718c588700 1 --2- 172.21.15.60:0/3268516053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71840b6a50 0x7f71840af9f0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 rx=0x7f717c001490 tx=0x7f717c007e00).stop 2022-06-29T16:08:06.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.285+0000 7f718c588700 1 -- 172.21.15.60:0/3268516053 shutdown_connections 2022-06-29T16:08:06.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.285+0000 7f718c588700 1 --2- 172.21.15.60:0/3268516053 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f71640539e0 0x7f7164055ea0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:06.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.285+0000 7f718c588700 1 --2- 172.21.15.60:0/3268516053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f71840b6a50 0x7f71840af9f0 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:06.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.285+0000 7f718c588700 1 --2- 172.21.15.60:0/3268516053 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f718405f0b0 0x7f71840af500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:06.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.285+0000 7f718c588700 1 --2- 172.21.15.60:0/3268516053 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f718405ca50 0x7f71840b55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:06.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.285+0000 7f718c588700 1 -- 172.21.15.60:0/3268516053 >> 172.21.15.60:0/3268516053 conn(0x7f71840643e0 msgr2=0x7f71840abbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:06.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.285+0000 7f718c588700 1 -- 172.21.15.60:0/3268516053 shutdown_connections 2022-06-29T16:08:06.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.285+0000 7f718c588700 1 -- 172.21.15.60:0/3268516053 wait complete. 2022-06-29T16:08:06.292 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 69 2022-06-29T16:08:06.305 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-06-29T16:08:06.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.453+0000 7f90eefc1700 1 Processor -- start 2022-06-29T16:08:06.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.453+0000 7f90eefc1700 1 -- start start 2022-06-29T16:08:06.461 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.453+0000 7f90eefc1700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f90e8063ad0 0x7f90e8067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:06.461 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.453+0000 7f90eefc1700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f90e814af10 0x7f90e805c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:06.461 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.453+0000 7f90eefc1700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90e80acba0 con 0x7f90e80b5950 2022-06-29T16:08:06.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.453+0000 7f90eefc1700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90e814b3a0 con 0x7f90e8063ad0 2022-06-29T16:08:06.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.453+0000 7f90eefc1700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90e8067220 con 0x7f90e814af10 2022-06-29T16:08:06.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.453+0000 7f90e7fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f90e8063ad0 0x7f90e8067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:06.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.453+0000 7f90e7fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f90e8063ad0 0x7f90e8067680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54030/0 (socket says 172.21.15.60:54030) 2022-06-29T16:08:06.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.453+0000 7f90e7fff700 1 -- 172.21.15.60:0/2525645543 learned_addr learned my addr 172.21.15.60:0/2525645543 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:06.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.453+0000 7f90ed55e700 1 --2- 172.21.15.60:0/2525645543 >> v2:172.21.15.92:3300/0 conn(0x7f90e814af10 0x7f90e805c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:06.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.453+0000 7f90e7fff700 1 -- 172.21.15.60:0/2525645543 >> v2:172.21.15.92:3300/0 conn(0x7f90e814af10 msgr2=0x7f90e805c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:06.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.453+0000 7f90e7fff700 1 --2- 172.21.15.60:0/2525645543 >> v2:172.21.15.92:3300/0 conn(0x7f90e814af10 0x7f90e805c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:06.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.453+0000 7f90e7fff700 1 -- 172.21.15.60:0/2525645543 >> v1:172.21.15.60:6789/0 conn(0x7f90e80b5950 legacy=0x7f90e805bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:06.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.453+0000 7f90e77fe700 1 -- 172.21.15.60:0/2525645543 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2245214211 0 0) 0x7f90e80acba0 con 0x7f90e80b5950 2022-06-29T16:08:06.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.453+0000 7f90e7fff700 1 -- 172.21.15.60:0/2525645543 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e80b5d30 con 0x7f90e8063ad0 2022-06-29T16:08:06.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.457+0000 7f90e7fff700 1 --2- 172.21.15.60:0/2525645543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90e8063ad0 0x7f90e8067680 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 rx=0x7f90d0002f70 tx=0x7f90d0006eb0).ready entity=mon.0 client_cookie=c1c5c7023804e3d4 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:06.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.457+0000 7f90e77fe700 1 -- 172.21.15.60:0/2525645543 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90d000d5c0 con 0x7f90e8063ad0 2022-06-29T16:08:06.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.457+0000 7f90e77fe700 1 -- 172.21.15.60:0/2525645543 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90d0000e90 con 0x7f90e8063ad0 2022-06-29T16:08:06.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.457+0000 7f90e77fe700 1 -- 172.21.15.60:0/2525645543 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90d000d910 con 0x7f90e8063ad0 2022-06-29T16:08:06.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.457+0000 7f90eefc1700 1 -- 172.21.15.60:0/2525645543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90e8063ad0 msgr2=0x7f90e8067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:06.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.457+0000 7f90eefc1700 1 --2- 172.21.15.60:0/2525645543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90e8063ad0 0x7f90e8067680 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 rx=0x7f90d0002f70 tx=0x7f90d0006eb0).stop 2022-06-29T16:08:06.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.457+0000 7f90eefc1700 1 -- 172.21.15.60:0/2525645543 shutdown_connections 2022-06-29T16:08:06.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.457+0000 7f90eefc1700 1 --2- 172.21.15.60:0/2525645543 >> v2:172.21.15.92:3300/0 conn(0x7f90e814af10 0x7f90e805c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:06.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.457+0000 7f90eefc1700 1 --2- 172.21.15.60:0/2525645543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90e8063ad0 0x7f90e8067680 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:06.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.457+0000 7f90eefc1700 1 -- 172.21.15.60:0/2525645543 >> 172.21.15.60:0/2525645543 conn(0x7f90e80632c0 msgr2=0x7f90e80b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:06.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.457+0000 7f90eefc1700 1 -- 172.21.15.60:0/2525645543 shutdown_connections 2022-06-29T16:08:06.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.457+0000 7f90eefc1700 1 -- 172.21.15.60:0/2525645543 wait complete. 2022-06-29T16:08:06.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.457+0000 7f90eefc1700 1 Processor -- start 2022-06-29T16:08:06.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.457+0000 7f90eefc1700 1 -- start start 2022-06-29T16:08:06.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.457+0000 7f90eefc1700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90e8063ad0 0x7f90e8140a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:06.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.457+0000 7f90eefc1700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90e80b5950 0x7f90e8142be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:06.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.457+0000 7f90eefc1700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90e814af10 0x7f90e813d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:06.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.457+0000 7f90eefc1700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e8142550 con 0x7f90e80b5950 2022-06-29T16:08:06.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.457+0000 7f90eefc1700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e81433e0 con 0x7f90e814af10 2022-06-29T16:08:06.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.457+0000 7f90eefc1700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90e8142740 con 0x7f90e8063ad0 2022-06-29T16:08:06.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.461+0000 7f90e7fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90e80b5950 0x7f90e8142be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:06.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.461+0000 7f90e7fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90e80b5950 0x7f90e8142be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54036/0 (socket says 172.21.15.60:54036) 2022-06-29T16:08:06.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.461+0000 7f90e7fff700 1 -- 172.21.15.60:0/4251432214 learned_addr learned my addr 172.21.15.60:0/4251432214 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:06.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.461+0000 7f90ed55e700 1 --2- 172.21.15.60:0/4251432214 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90e814af10 0x7f90e813d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:06.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.461+0000 7f90ecd5d700 1 --2- 172.21.15.60:0/4251432214 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90e8063ad0 0x7f90e8140a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:06.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.461+0000 7f90e7fff700 1 -- 172.21.15.60:0/4251432214 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90e8063ad0 msgr2=0x7f90e8140a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:06.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.461+0000 7f90e7fff700 1 --2- 172.21.15.60:0/4251432214 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90e8063ad0 0x7f90e8140a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:06.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.461+0000 7f90e7fff700 1 -- 172.21.15.60:0/4251432214 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90e814af10 msgr2=0x7f90e813d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:06.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.461+0000 7f90e7fff700 1 --2- 172.21.15.60:0/4251432214 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90e814af10 0x7f90e813d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:06.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.461+0000 7f90e7fff700 1 -- 172.21.15.60:0/4251432214 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d000b1e0 con 0x7f90e80b5950 2022-06-29T16:08:06.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.461+0000 7f90e7fff700 1 --2- 172.21.15.60:0/4251432214 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90e80b5950 0x7f90e8142be0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 rx=0x7f90d0002f70 tx=0x7f90d0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:06.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.461+0000 7f90e5ffb700 1 -- 172.21.15.60:0/4251432214 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90d000d960 con 0x7f90e80b5950 2022-06-29T16:08:06.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.461+0000 7f90e5ffb700 1 -- 172.21.15.60:0/4251432214 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90d0011c30 con 0x7f90e80b5950 2022-06-29T16:08:06.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.461+0000 7f90eefc1700 1 -- 172.21.15.60:0/4251432214 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90e81430d0 con 0x7f90e80b5950 2022-06-29T16:08:06.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.461+0000 7f90e5ffb700 1 -- 172.21.15.60:0/4251432214 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90d001e460 con 0x7f90e80b5950 2022-06-29T16:08:06.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.461+0000 7f90eefc1700 1 -- 172.21.15.60:0/4251432214 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90e8149c80 con 0x7f90e80b5950 2022-06-29T16:08:06.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.461+0000 7f90eefc1700 1 -- 172.21.15.60:0/4251432214 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90e8142900 con 0x7f90e80b5950 2022-06-29T16:08:06.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.465+0000 7f90e5ffb700 1 -- 172.21.15.60:0/4251432214 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f90d0025030 con 0x7f90e80b5950 2022-06-29T16:08:06.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.465+0000 7f90e5ffb700 1 --2- 172.21.15.60:0/4251432214 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f90cc053a10 0x7f90cc055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:06.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.465+0000 7f90e5ffb700 1 -- 172.21.15.60:0/4251432214 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f90d006d010 con 0x7f90e80b5950 2022-06-29T16:08:06.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.465+0000 7f90ecd5d700 1 --2- 172.21.15.60:0/4251432214 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f90cc053a10 0x7f90cc055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:06.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.465+0000 7f90ecd5d700 1 --2- 172.21.15.60:0/4251432214 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f90cc053a10 0x7f90cc055ed0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7f90d800d000 tx=0x7f90d8007da0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:06.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.469+0000 7f90e5ffb700 1 -- 172.21.15.60:0/4251432214 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f90d0043ee0 con 0x7f90e80b5950 2022-06-29T16:08:06.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:06.633+0000 7f90eefc1700 1 -- 172.21.15.60:0/4251432214 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f90e81435e0 con 0x7f90e80b5950 2022-06-29T16:08:07.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.085+0000 7f90e5ffb700 1 -- 172.21.15.60:0/4251432214 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v70) v1 ==== 104+0+0 (secure 0 0 0) 0x7f90d0023020 con 0x7f90e80b5950 2022-06-29T16:08:07.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.089+0000 7f90eefc1700 1 -- 172.21.15.60:0/4251432214 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f90cc053a10 msgr2=0x7f90cc055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:07.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.089+0000 7f90eefc1700 1 --2- 172.21.15.60:0/4251432214 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f90cc053a10 0x7f90cc055ed0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7f90d800d000 tx=0x7f90d8007da0).stop 2022-06-29T16:08:07.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.089+0000 7f90eefc1700 1 -- 172.21.15.60:0/4251432214 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90e80b5950 msgr2=0x7f90e8142be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:07.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.089+0000 7f90eefc1700 1 --2- 172.21.15.60:0/4251432214 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90e80b5950 0x7f90e8142be0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 rx=0x7f90d0002f70 tx=0x7f90d0015000).stop 2022-06-29T16:08:07.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.089+0000 7f90eefc1700 1 -- 172.21.15.60:0/4251432214 shutdown_connections 2022-06-29T16:08:07.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.089+0000 7f90eefc1700 1 --2- 172.21.15.60:0/4251432214 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f90cc053a10 0x7f90cc055ed0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:07.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.089+0000 7f90eefc1700 1 --2- 172.21.15.60:0/4251432214 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90e814af10 0x7f90e813d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:07.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.089+0000 7f90eefc1700 1 --2- 172.21.15.60:0/4251432214 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90e80b5950 0x7f90e8142be0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:07.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.089+0000 7f90eefc1700 1 --2- 172.21.15.60:0/4251432214 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f90e8063ad0 0x7f90e8140a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:07.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.089+0000 7f90eefc1700 1 -- 172.21.15.60:0/4251432214 >> 172.21.15.60:0/4251432214 conn(0x7f90e80632c0 msgr2=0x7f90e80b20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:07.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.089+0000 7f90eefc1700 1 -- 172.21.15.60:0/4251432214 shutdown_connections 2022-06-29T16:08:07.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.089+0000 7f90eefc1700 1 -- 172.21.15.60:0/4251432214 wait complete. 2022-06-29T16:08:07.107 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-06-29T16:08:07.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.221+0000 7f2dcf2e9700 1 Processor -- start 2022-06-29T16:08:07.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.221+0000 7f2dcf2e9700 1 -- start start 2022-06-29T16:08:07.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.221+0000 7f2dcf2e9700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2dc80b4010 0x7f2dc81426b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:07.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.221+0000 7f2dcf2e9700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc8143a40 con 0x7f2dc80ae7c0 2022-06-29T16:08:07.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.221+0000 7f2dcf2e9700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc80aee60 con 0x7f2dc80b46c0 2022-06-29T16:08:07.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.221+0000 7f2dcf2e9700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dc80af100 con 0x7f2dc80b4010 2022-06-29T16:08:07.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.221+0000 7f2dcc884700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2dc80b4010 0x7f2dc81426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:07.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.221+0000 7f2dcd085700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2dc80ae7c0 0x7f2dc8141fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33896/0 (socket says 172.21.15.60:33896) 2022-06-29T16:08:07.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2dcd085700 1 -- 172.21.15.60:0/1202567102 learned_addr learned my addr 172.21.15.60:0/1202567102 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:07.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2db7fff700 1 -- 172.21.15.60:0/1202567102 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4113126383 0 0) 0x7f2dc8143a40 con 0x7f2dc80ae7c0 2022-06-29T16:08:07.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2db7fff700 1 -- 172.21.15.60:0/1202567102 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2db0003260 con 0x7f2dc80ae7c0 2022-06-29T16:08:07.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2db7fff700 1 -- 172.21.15.60:0/1202567102 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2665113110 0 0) 0x7f2dc80aee60 con 0x7f2dc80b46c0 2022-06-29T16:08:07.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2db7fff700 1 -- 172.21.15.60:0/1202567102 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dc8143a40 con 0x7f2dc80b46c0 2022-06-29T16:08:07.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2db7fff700 1 -- 172.21.15.60:0/1202567102 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3926265349 0 0) 0x7f2db0003260 con 0x7f2dc80ae7c0 2022-06-29T16:08:07.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2db7fff700 1 -- 172.21.15.60:0/1202567102 >> v2:172.21.15.60:3300/0 conn(0x7f2dc80b4010 msgr2=0x7f2dc81426b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:07.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2db7fff700 1 --2- 172.21.15.60:0/1202567102 >> v2:172.21.15.60:3300/0 conn(0x7f2dc80b4010 0x7f2dc81426b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:07.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2db7fff700 1 -- 172.21.15.60:0/1202567102 >> v1:172.21.15.92:6789/0 conn(0x7f2dc80b46c0 legacy=0x7f2dc80b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:07.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2db7fff700 1 -- 172.21.15.60:0/1202567102 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc80aeba0 con 0x7f2dc80ae7c0 2022-06-29T16:08:07.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2db7fff700 1 -- 172.21.15.60:0/1202567102 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f2db8001660 con 0x7f2dc80ae7c0 2022-06-29T16:08:07.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2dcc884700 1 --2- 172.21.15.60:0/1202567102 >> v2:172.21.15.60:3300/0 conn(0x7f2dc80b4010 0x7f2dc81426b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:08:07.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2db7fff700 1 -- 172.21.15.60:0/1202567102 >> v1:172.21.15.60:6789/0 conn(0x7f2dc80ae7c0 legacy=0x7f2dc8141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:07.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2db7fff700 1 --2- 172.21.15.60:0/1202567102 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2db00038e0 0x7f2db0003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:07.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2db7fff700 1 --2- 172.21.15.60:0/1202567102 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2db0004500 0x7f2db0004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:07.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2db7fff700 1 --2- 172.21.15.60:0/1202567102 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2db0004e70 0x7f2db000b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:07.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2db7fff700 1 -- 172.21.15.60:0/1202567102 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc80af100 con 0x7f2db0004e70 2022-06-29T16:08:07.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2db7fff700 1 -- 172.21.15.60:0/1202567102 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db000bcf0 con 0x7f2db00038e0 2022-06-29T16:08:07.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2db7fff700 1 -- 172.21.15.60:0/1202567102 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2db000beb0 con 0x7f2db0004500 2022-06-29T16:08:07.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2dcd886700 1 --2- 172.21.15.60:0/1202567102 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2db0004e70 0x7f2db000b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:07.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2dcd085700 1 --2- 172.21.15.60:0/1202567102 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2db00038e0 0x7f2db0003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:07.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2dcc884700 1 --2- 172.21.15.60:0/1202567102 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2db0004500 0x7f2db0004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:07.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2dcd886700 1 -- 172.21.15.60:0/1202567102 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2db0004500 msgr2=0x7f2db0004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:07.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2dcd886700 1 --2- 172.21.15.60:0/1202567102 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2db0004500 0x7f2db0004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:07.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2dcd886700 1 -- 172.21.15.60:0/1202567102 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2db00038e0 msgr2=0x7f2db0003ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:07.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2dcd886700 1 --2- 172.21.15.60:0/1202567102 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2db00038e0 0x7f2db0003ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:07.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2dcd886700 1 -- 172.21.15.60:0/1202567102 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db0003260 con 0x7f2db0004e70 2022-06-29T16:08:07.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2dcd886700 1 -- 172.21.15.60:0/1202567102 reap_dead start 2022-06-29T16:08:07.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2dcd886700 1 --2- 172.21.15.60:0/1202567102 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2db0004e70 0x7f2db000b750 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 rx=0x7f2dc0001490 tx=0x7f2dc0007d80).ready entity=mon.0 client_cookie=470f9ce987d0277c server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:07.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2db7fff700 1 -- 172.21.15.60:0/1202567102 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2dc0006890 con 0x7f2db0004e70 2022-06-29T16:08:07.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2db7fff700 1 -- 172.21.15.60:0/1202567102 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dc00069f0 con 0x7f2db0004e70 2022-06-29T16:08:07.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2db7fff700 1 -- 172.21.15.60:0/1202567102 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2dc0006d50 con 0x7f2db0004e70 2022-06-29T16:08:07.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2dcf2e9700 1 -- 172.21.15.60:0/1202567102 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2db0004e70 msgr2=0x7f2db000b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:07.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2dcf2e9700 1 --2- 172.21.15.60:0/1202567102 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2db0004e70 0x7f2db000b750 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 rx=0x7f2dc0001490 tx=0x7f2dc0007d80).stop 2022-06-29T16:08:07.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2dcf2e9700 1 -- 172.21.15.60:0/1202567102 shutdown_connections 2022-06-29T16:08:07.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2dcf2e9700 1 --2- 172.21.15.60:0/1202567102 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2db0004e70 0x7f2db000b750 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:07.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.225+0000 7f2dcf2e9700 1 -- 172.21.15.60:0/1202567102 >> 172.21.15.60:0/1202567102 conn(0x7f2dc80b9340 msgr2=0x7f2dc80b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:07.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.229+0000 7f2dcf2e9700 1 -- 172.21.15.60:0/1202567102 shutdown_connections 2022-06-29T16:08:07.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.229+0000 7f2dcf2e9700 1 -- 172.21.15.60:0/1202567102 wait complete. 2022-06-29T16:08:07.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.229+0000 7f2dcf2e9700 1 Processor -- start 2022-06-29T16:08:07.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.229+0000 7f2dcf2e9700 1 -- start start 2022-06-29T16:08:07.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.229+0000 7f2dcf2e9700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2db0004e70 0x7f2dc80b2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:07.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.229+0000 7f2dcf2e9700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2dc8148840 0x7f2dc813d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:07.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.229+0000 7f2dcf2e9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dc81478e0 0x7f2dc813dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:07.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.229+0000 7f2dcf2e9700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc8147e60 con 0x7f2dc81478e0 2022-06-29T16:08:07.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.229+0000 7f2dcf2e9700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc8148100 con 0x7f2db0004e70 2022-06-29T16:08:07.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.229+0000 7f2dcf2e9700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc81483a0 con 0x7f2dc8148840 2022-06-29T16:08:07.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.229+0000 7f2dcd886700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dc81478e0 0x7f2dc813dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:07.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.229+0000 7f2dcd886700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dc81478e0 0x7f2dc813dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54058/0 (socket says 172.21.15.60:54058) 2022-06-29T16:08:07.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.229+0000 7f2dcd886700 1 -- 172.21.15.60:0/1599849975 learned_addr learned my addr 172.21.15.60:0/1599849975 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:07.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.229+0000 7f2dcc884700 1 --2- 172.21.15.60:0/1599849975 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2dc8148840 0x7f2dc813d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:07.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.229+0000 7f2dcd085700 1 --2- 172.21.15.60:0/1599849975 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2db0004e70 0x7f2dc80b2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:07.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.233+0000 7f2dcd886700 1 -- 172.21.15.60:0/1599849975 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2dc8148840 msgr2=0x7f2dc813d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:07.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.233+0000 7f2dcd886700 1 --2- 172.21.15.60:0/1599849975 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2dc8148840 0x7f2dc813d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:07.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.233+0000 7f2dcd886700 1 -- 172.21.15.60:0/1599849975 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2db0004e70 msgr2=0x7f2dc80b2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:07.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.233+0000 7f2dcd886700 1 --2- 172.21.15.60:0/1599849975 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2db0004e70 0x7f2dc80b2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:07.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.233+0000 7f2dcd886700 1 -- 172.21.15.60:0/1599849975 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc80b1520 con 0x7f2dc81478e0 2022-06-29T16:08:07.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.233+0000 7f2dcd886700 1 --2- 172.21.15.60:0/1599849975 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dc81478e0 0x7f2dc813dcd0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 rx=0x7f2dc0001490 tx=0x7f2dc0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:07.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.233+0000 7f2db67fc700 1 -- 172.21.15.60:0/1599849975 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2dc0006890 con 0x7f2dc81478e0 2022-06-29T16:08:07.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.233+0000 7f2db67fc700 1 -- 172.21.15.60:0/1599849975 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dc000c9e0 con 0x7f2dc81478e0 2022-06-29T16:08:07.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.233+0000 7f2dcf2e9700 1 -- 172.21.15.60:0/1599849975 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dc80b2620 con 0x7f2dc81478e0 2022-06-29T16:08:07.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.233+0000 7f2db67fc700 1 -- 172.21.15.60:0/1599849975 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2dc0001780 con 0x7f2dc81478e0 2022-06-29T16:08:07.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.233+0000 7f2dcf2e9700 1 -- 172.21.15.60:0/1599849975 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dc8146bd0 con 0x7f2dc81478e0 2022-06-29T16:08:07.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.233+0000 7f2dcf2e9700 1 -- 172.21.15.60:0/1599849975 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dc80573d0 con 0x7f2dc81478e0 2022-06-29T16:08:07.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.233+0000 7f2db67fc700 1 -- 172.21.15.60:0/1599849975 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2dc0026030 con 0x7f2dc81478e0 2022-06-29T16:08:07.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.233+0000 7f2db67fc700 1 --2- 172.21.15.60:0/1599849975 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2da8053aa0 0x7f2da8055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:07.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.233+0000 7f2db67fc700 1 -- 172.21.15.60:0/1599849975 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f2dc006d200 con 0x7f2dc81478e0 2022-06-29T16:08:07.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.233+0000 7f2dcd085700 1 --2- 172.21.15.60:0/1599849975 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2da8053aa0 0x7f2da8055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:07.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.237+0000 7f2dcd085700 1 --2- 172.21.15.60:0/1599849975 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2da8053aa0 0x7f2da8055f60 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0x7f2db8005000 tx=0x7f2db8001db0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:07.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.245+0000 7f2db67fc700 1 -- 172.21.15.60:0/1599849975 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2dc00440d0 con 0x7f2dc81478e0 2022-06-29T16:08:07.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:07.377+0000 7f2dcf2e9700 1 -- 172.21.15.60:0/1599849975 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2dc80573d0 con 0x7f2dc81478e0 2022-06-29T16:08:08.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.097+0000 7f2db67fc700 1 -- 172.21.15.60:0/1599849975 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v139) v1 ==== 185+0+0 (secure 0 0 0) 0x7f2dc00476f0 con 0x7f2dc81478e0 2022-06-29T16:08:08.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.101+0000 7f2dcf2e9700 1 -- 172.21.15.60:0/1599849975 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2da8053aa0 msgr2=0x7f2da8055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:08.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.101+0000 7f2dcf2e9700 1 --2- 172.21.15.60:0/1599849975 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2da8053aa0 0x7f2da8055f60 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0x7f2db8005000 tx=0x7f2db8001db0).stop 2022-06-29T16:08:08.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.101+0000 7f2dcf2e9700 1 -- 172.21.15.60:0/1599849975 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dc81478e0 msgr2=0x7f2dc813dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:08.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.101+0000 7f2dcf2e9700 1 --2- 172.21.15.60:0/1599849975 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dc81478e0 0x7f2dc813dcd0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 rx=0x7f2dc0001490 tx=0x7f2dc0016000).stop 2022-06-29T16:08:08.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.101+0000 7f2dcf2e9700 1 -- 172.21.15.60:0/1599849975 shutdown_connections 2022-06-29T16:08:08.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.101+0000 7f2dcf2e9700 1 --2- 172.21.15.60:0/1599849975 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2da8053aa0 0x7f2da8055f60 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:08.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.101+0000 7f2dcf2e9700 1 --2- 172.21.15.60:0/1599849975 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dc81478e0 0x7f2dc813dcd0 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:08.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.101+0000 7f2dcf2e9700 1 --2- 172.21.15.60:0/1599849975 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2dc8148840 0x7f2dc813d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:08.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.101+0000 7f2dcf2e9700 1 --2- 172.21.15.60:0/1599849975 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2db0004e70 0x7f2dc80b2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:08.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.101+0000 7f2dcf2e9700 1 -- 172.21.15.60:0/1599849975 >> 172.21.15.60:0/1599849975 conn(0x7f2dc80b9340 msgr2=0x7f2dc80b5630 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:08.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.101+0000 7f2dcf2e9700 1 -- 172.21.15.60:0/1599849975 shutdown_connections 2022-06-29T16:08:08.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.101+0000 7f2dcf2e9700 1 -- 172.21.15.60:0/1599849975 wait complete. 2022-06-29T16:08:08.111 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2022-06-29T16:08:08.121 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-06-29T16:08:08.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.225+0000 7f25f1ae3700 1 Processor -- start 2022-06-29T16:08:08.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.225+0000 7f25f1ae3700 1 -- start start 2022-06-29T16:08:08.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.225+0000 7f25f1ae3700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f25ec05f0b0 0x7f25ec0aaf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:08.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.225+0000 7f25f1ae3700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25ec05fee0 0x7f25ec060370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:08.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.225+0000 7f25f1ae3700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ec0b6e30 con 0x7f25ec0b6a50 2022-06-29T16:08:08.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.225+0000 7f25f1ae3700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25ec0ab450 con 0x7f25ec05f0b0 2022-06-29T16:08:08.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.225+0000 7f25f1ae3700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec0ab670 con 0x7f25ec05fee0 2022-06-29T16:08:08.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.225+0000 7f25eb7fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f25ec0b6a50 0x7f25ec146260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33916/0 (socket says 172.21.15.60:33916) 2022-06-29T16:08:08.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.225+0000 7f25eb7fe700 1 -- 172.21.15.60:0/544125019 learned_addr learned my addr 172.21.15.60:0/544125019 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:08.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.225+0000 7f25eaffd700 1 --2- 172.21.15.60:0/544125019 >> v2:172.21.15.60:3300/0 conn(0x7f25ec05f0b0 0x7f25ec0aaf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:08.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.225+0000 7f25ebfff700 1 --2- 172.21.15.60:0/544125019 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25ec05fee0 0x7f25ec060370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:08.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.225+0000 7f25ea7fc700 1 -- 172.21.15.60:0/544125019 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 951652296 0 0) 0x7f25ec0b6e30 con 0x7f25ec0b6a50 2022-06-29T16:08:08.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.225+0000 7f25ea7fc700 1 -- 172.21.15.60:0/544125019 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25cc003260 con 0x7f25ec0b6a50 2022-06-29T16:08:08.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.225+0000 7f25eaffd700 1 -- 172.21.15.60:0/544125019 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25ec05fee0 msgr2=0x7f25ec060370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:08.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.225+0000 7f25eaffd700 1 --2- 172.21.15.60:0/544125019 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25ec05fee0 0x7f25ec060370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:08.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.225+0000 7f25eaffd700 1 -- 172.21.15.60:0/544125019 >> v1:172.21.15.60:6789/0 conn(0x7f25ec0b6a50 legacy=0x7f25ec146260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:08.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.225+0000 7f25eaffd700 1 -- 172.21.15.60:0/544125019 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ec0ab910 con 0x7f25ec05f0b0 2022-06-29T16:08:08.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.225+0000 7f25eaffd700 1 --2- 172.21.15.60:0/544125019 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25ec05f0b0 0x7f25ec0aaf60 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 rx=0x7f25d4002f70 tx=0x7f25d4006df0).ready entity=mon.0 client_cookie=b00debd203b5627b server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:08.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.225+0000 7f25ea7fc700 1 -- 172.21.15.60:0/544125019 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25d400d4a0 con 0x7f25ec05f0b0 2022-06-29T16:08:08.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.225+0000 7f25ea7fc700 1 -- 172.21.15.60:0/544125019 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25d4000e90 con 0x7f25ec05f0b0 2022-06-29T16:08:08.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.225+0000 7f25ea7fc700 1 -- 172.21.15.60:0/544125019 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25d400d7f0 con 0x7f25ec05f0b0 2022-06-29T16:08:08.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.225+0000 7f25f1ae3700 1 -- 172.21.15.60:0/544125019 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25ec05f0b0 msgr2=0x7f25ec0aaf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:08.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.225+0000 7f25f1ae3700 1 --2- 172.21.15.60:0/544125019 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25ec05f0b0 0x7f25ec0aaf60 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 rx=0x7f25d4002f70 tx=0x7f25d4006df0).stop 2022-06-29T16:08:08.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.225+0000 7f25f1ae3700 1 -- 172.21.15.60:0/544125019 shutdown_connections 2022-06-29T16:08:08.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.229+0000 7f25f1ae3700 1 --2- 172.21.15.60:0/544125019 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25ec05fee0 0x7f25ec060370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:08.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.229+0000 7f25f1ae3700 1 --2- 172.21.15.60:0/544125019 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25ec05f0b0 0x7f25ec0aaf60 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:08.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.229+0000 7f25f1ae3700 1 -- 172.21.15.60:0/544125019 >> 172.21.15.60:0/544125019 conn(0x7f25ec064420 msgr2=0x7f25ec0b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:08.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.229+0000 7f25f1ae3700 1 -- 172.21.15.60:0/544125019 shutdown_connections 2022-06-29T16:08:08.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.229+0000 7f25f1ae3700 1 -- 172.21.15.60:0/544125019 wait complete. 2022-06-29T16:08:08.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.229+0000 7f25f1ae3700 1 Processor -- start 2022-06-29T16:08:08.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.229+0000 7f25f1ae3700 1 -- start start 2022-06-29T16:08:08.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.233+0000 7f25f1ae3700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25ec05f0b0 0x7f25ec0ad780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:08.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.233+0000 7f25f1ae3700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25ec05fee0 0x7f25ec155c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:08.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.233+0000 7f25f1ae3700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25ec0b6a50 0x7f25ec13b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:08.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.233+0000 7f25f1ae3700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec156170 con 0x7f25ec05f0b0 2022-06-29T16:08:08.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.233+0000 7f25f1ae3700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec151cd0 con 0x7f25ec05fee0 2022-06-29T16:08:08.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.233+0000 7f25f1ae3700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec13c5d0 con 0x7f25ec0b6a50 2022-06-29T16:08:08.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.233+0000 7f25eb7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25ec05f0b0 0x7f25ec0ad780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:08.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.233+0000 7f25eb7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25ec05f0b0 0x7f25ec0ad780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54068/0 (socket says 172.21.15.60:54068) 2022-06-29T16:08:08.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.233+0000 7f25eb7fe700 1 -- 172.21.15.60:0/544387121 learned_addr learned my addr 172.21.15.60:0/544387121 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:08.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.233+0000 7f25eaffd700 1 --2- 172.21.15.60:0/544387121 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25ec05fee0 0x7f25ec155c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:08.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.233+0000 7f25ebfff700 1 --2- 172.21.15.60:0/544387121 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25ec0b6a50 0x7f25ec13b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:08.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.233+0000 7f25eb7fe700 1 -- 172.21.15.60:0/544387121 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25ec0b6a50 msgr2=0x7f25ec13b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:08.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.233+0000 7f25eb7fe700 1 --2- 172.21.15.60:0/544387121 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25ec0b6a50 0x7f25ec13b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:08.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.233+0000 7f25eb7fe700 1 -- 172.21.15.60:0/544387121 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25ec05fee0 msgr2=0x7f25ec155c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:08.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.233+0000 7f25eb7fe700 1 --2- 172.21.15.60:0/544387121 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25ec05fee0 0x7f25ec155c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:08.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.233+0000 7f25eb7fe700 1 -- 172.21.15.60:0/544387121 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d400b180 con 0x7f25ec05f0b0 2022-06-29T16:08:08.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.233+0000 7f25eb7fe700 1 --2- 172.21.15.60:0/544387121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25ec05f0b0 0x7f25ec0ad780 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 rx=0x7f25ec1562b0 tx=0x7f25dc007e50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:08.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.233+0000 7f25e8ff9700 1 -- 172.21.15.60:0/544387121 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25dc005c70 con 0x7f25ec05f0b0 2022-06-29T16:08:08.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.233+0000 7f25e8ff9700 1 -- 172.21.15.60:0/544387121 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25dc005dd0 con 0x7f25ec05f0b0 2022-06-29T16:08:08.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.233+0000 7f25f1ae3700 1 -- 172.21.15.60:0/544387121 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25ec13c870 con 0x7f25ec05f0b0 2022-06-29T16:08:08.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.233+0000 7f25e8ff9700 1 -- 172.21.15.60:0/544387121 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25dc0072c0 con 0x7f25ec05f0b0 2022-06-29T16:08:08.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.233+0000 7f25f1ae3700 1 -- 172.21.15.60:0/544387121 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25ec13c1a0 con 0x7f25ec05f0b0 2022-06-29T16:08:08.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.233+0000 7f25f1ae3700 1 -- 172.21.15.60:0/544387121 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25ec0573d0 con 0x7f25ec05f0b0 2022-06-29T16:08:08.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.233+0000 7f25e8ff9700 1 -- 172.21.15.60:0/544387121 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f25dc010070 con 0x7f25ec05f0b0 2022-06-29T16:08:08.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.237+0000 7f25e8ff9700 1 --2- 172.21.15.60:0/544387121 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f25d00539e0 0x7f25d0055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:08.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.237+0000 7f25e8ff9700 1 -- 172.21.15.60:0/544387121 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f25dc06b8d0 con 0x7f25ec05f0b0 2022-06-29T16:08:08.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.237+0000 7f25eaffd700 1 --2- 172.21.15.60:0/544387121 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f25d00539e0 0x7f25d0055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:08.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.237+0000 7f25eaffd700 1 --2- 172.21.15.60:0/544387121 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f25d00539e0 0x7f25d0055ea0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0x7f25d401a000 tx=0x7f25d4014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:08.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.245+0000 7f25e8ff9700 1 -- 172.21.15.60:0/544387121 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f25dc0427e0 con 0x7f25ec05f0b0 2022-06-29T16:08:08.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:08.369+0000 7f25f1ae3700 1 -- 172.21.15.60:0/544387121 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f25ec067990 con 0x7f25ec05f0b0 2022-06-29T16:08:09.114 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.109+0000 7f25e8ff9700 1 -- 172.21.15.60:0/544387121 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v140) v1 ==== 173+0+0 (secure 0 0 0) 0x7f25dc045e00 con 0x7f25ec05f0b0 2022-06-29T16:08:09.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.113+0000 7f25f1ae3700 1 -- 172.21.15.60:0/544387121 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f25d00539e0 msgr2=0x7f25d0055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:09.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.113+0000 7f25f1ae3700 1 --2- 172.21.15.60:0/544387121 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f25d00539e0 0x7f25d0055ea0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0x7f25d401a000 tx=0x7f25d4014000).stop 2022-06-29T16:08:09.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.113+0000 7f25f1ae3700 1 -- 172.21.15.60:0/544387121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25ec05f0b0 msgr2=0x7f25ec0ad780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:09.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.113+0000 7f25f1ae3700 1 --2- 172.21.15.60:0/544387121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25ec05f0b0 0x7f25ec0ad780 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 rx=0x7f25ec1562b0 tx=0x7f25dc007e50).stop 2022-06-29T16:08:09.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.113+0000 7f25f1ae3700 1 -- 172.21.15.60:0/544387121 shutdown_connections 2022-06-29T16:08:09.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.113+0000 7f25f1ae3700 1 --2- 172.21.15.60:0/544387121 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f25d00539e0 0x7f25d0055ea0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:09.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.113+0000 7f25f1ae3700 1 --2- 172.21.15.60:0/544387121 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f25ec0b6a50 0x7f25ec13b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:09.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.113+0000 7f25f1ae3700 1 --2- 172.21.15.60:0/544387121 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25ec05fee0 0x7f25ec155c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:09.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.113+0000 7f25f1ae3700 1 --2- 172.21.15.60:0/544387121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f25ec05f0b0 0x7f25ec0ad780 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:09.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.113+0000 7f25f1ae3700 1 -- 172.21.15.60:0/544387121 >> 172.21.15.60:0/544387121 conn(0x7f25ec064420 msgr2=0x7f25ec0ac300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:09.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.113+0000 7f25f1ae3700 1 -- 172.21.15.60:0/544387121 shutdown_connections 2022-06-29T16:08:09.122 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.113+0000 7f25f1ae3700 1 -- 172.21.15.60:0/544387121 wait complete. 2022-06-29T16:08:09.123 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2022-06-29T16:08:09.132 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-06-29T16:08:09.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.241+0000 7f9bf5652700 1 Processor -- start 2022-06-29T16:08:09.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.241+0000 7f9bf5652700 1 -- start start 2022-06-29T16:08:09.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.241+0000 7f9bf5652700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9bf00b2130 0x7f9bf0146100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:09.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.241+0000 7f9bf5652700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bf00aed00 0x7f9bf01465f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:09.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.241+0000 7f9bf5652700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf005d450 con 0x7f9bf005d070 2022-06-29T16:08:09.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.241+0000 7f9bf5652700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bf00adf90 con 0x7f9bf00b2130 2022-06-29T16:08:09.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.241+0000 7f9bf5652700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf00b25d0 con 0x7f9bf00aed00 2022-06-29T16:08:09.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.241+0000 7f9bef7fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9bf00b2130 0x7f9bf0146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:09.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.241+0000 7f9bef7fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9bf00b2130 0x7f9bf0146100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54078/0 (socket says 172.21.15.60:54078) 2022-06-29T16:08:09.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.241+0000 7f9bef7fe700 1 -- 172.21.15.60:0/3348826932 learned_addr learned my addr 172.21.15.60:0/3348826932 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:09.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.241+0000 7f9bf4e51700 1 --2- 172.21.15.60:0/3348826932 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bf00aed00 0x7f9bf01465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:09.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.241+0000 7f9bef7fe700 1 -- 172.21.15.60:0/3348826932 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bf00aed00 msgr2=0x7f9bf01465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:09.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.241+0000 7f9bef7fe700 1 --2- 172.21.15.60:0/3348826932 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bf00aed00 0x7f9bf01465f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:09.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.241+0000 7f9bef7fe700 1 -- 172.21.15.60:0/3348826932 >> v1:172.21.15.92:6789/0 conn(0x7f9bf005d070 legacy=0x7f9bf014aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:09.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.241+0000 7f9bef7fe700 1 -- 172.21.15.60:0/3348826932 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf014a350 con 0x7f9bf00b2130 2022-06-29T16:08:09.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.241+0000 7f9bef7fe700 1 --2- 172.21.15.60:0/3348826932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bf00b2130 0x7f9bf0146100 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 rx=0x7f9be00032f0 tx=0x7f9be0009290).ready entity=mon.0 client_cookie=4a352bdaa8d0cf60 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:09.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.241+0000 7f9beeffd700 1 -- 172.21.15.60:0/3348826932 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9be000b8a0 con 0x7f9bf00b2130 2022-06-29T16:08:09.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.245+0000 7f9beeffd700 1 -- 172.21.15.60:0/3348826932 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be000ba00 con 0x7f9bf00b2130 2022-06-29T16:08:09.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.245+0000 7f9beeffd700 1 -- 172.21.15.60:0/3348826932 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9be000bb70 con 0x7f9bf00b2130 2022-06-29T16:08:09.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.245+0000 7f9bf5652700 1 -- 172.21.15.60:0/3348826932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bf00b2130 msgr2=0x7f9bf0146100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:09.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.245+0000 7f9bf5652700 1 --2- 172.21.15.60:0/3348826932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bf00b2130 0x7f9bf0146100 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 rx=0x7f9be00032f0 tx=0x7f9be0009290).stop 2022-06-29T16:08:09.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.245+0000 7f9bf5652700 1 -- 172.21.15.60:0/3348826932 shutdown_connections 2022-06-29T16:08:09.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.245+0000 7f9bf5652700 1 --2- 172.21.15.60:0/3348826932 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bf00aed00 0x7f9bf01465f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:09.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.245+0000 7f9bf5652700 1 --2- 172.21.15.60:0/3348826932 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bf00b2130 0x7f9bf0146100 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:09.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.245+0000 7f9bf5652700 1 -- 172.21.15.60:0/3348826932 >> 172.21.15.60:0/3348826932 conn(0x7f9bf00b5890 msgr2=0x7f9bf00b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:09.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.245+0000 7f9bf5652700 1 -- 172.21.15.60:0/3348826932 shutdown_connections 2022-06-29T16:08:09.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.245+0000 7f9bf5652700 1 -- 172.21.15.60:0/3348826932 wait complete. 2022-06-29T16:08:09.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.245+0000 7f9bf5652700 1 Processor -- start 2022-06-29T16:08:09.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.245+0000 7f9bf5652700 1 -- start start 2022-06-29T16:08:09.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.249+0000 7f9bf5652700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bf005d070 0x7f9bf0146b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:09.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.249+0000 7f9bf5652700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bf00aed00 0x7f9bf0066970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:09.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.249+0000 7f9bf5652700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bf00b2130 0x7f9bf0065880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:09.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.249+0000 7f9bf5652700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf014c8d0 con 0x7f9bf005d070 2022-06-29T16:08:09.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.249+0000 7f9bf5652700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf014d3e0 con 0x7f9bf00aed00 2022-06-29T16:08:09.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.249+0000 7f9bf5652700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf014ccd0 con 0x7f9bf00b2130 2022-06-29T16:08:09.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.249+0000 7f9beffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bf005d070 0x7f9bf0146b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:09.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.249+0000 7f9bf4e51700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bf00b2130 0x7f9bf0065880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:09.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.249+0000 7f9bef7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bf00aed00 0x7f9bf0066970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:09.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.249+0000 7f9beffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bf005d070 0x7f9bf0146b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54082/0 (socket says 172.21.15.60:54082) 2022-06-29T16:08:09.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.249+0000 7f9beffff700 1 -- 172.21.15.60:0/1947431255 learned_addr learned my addr 172.21.15.60:0/1947431255 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:09.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.249+0000 7f9beffff700 1 -- 172.21.15.60:0/1947431255 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bf00b2130 msgr2=0x7f9bf0065880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:09.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.249+0000 7f9beffff700 1 --2- 172.21.15.60:0/1947431255 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bf00b2130 0x7f9bf0065880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:09.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.249+0000 7f9beffff700 1 -- 172.21.15.60:0/1947431255 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bf00aed00 msgr2=0x7f9bf0066970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:09.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.249+0000 7f9beffff700 1 --2- 172.21.15.60:0/1947431255 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bf00aed00 0x7f9bf0066970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:09.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.249+0000 7f9beffff700 1 -- 172.21.15.60:0/1947431255 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be0009530 con 0x7f9bf005d070 2022-06-29T16:08:09.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.249+0000 7f9beffff700 1 --2- 172.21.15.60:0/1947431255 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bf005d070 0x7f9bf0146b40 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 rx=0x7f9bf0061990 tx=0x7f9bd8007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:09.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.249+0000 7f9bed7fa700 1 -- 172.21.15.60:0/1947431255 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9bd8007030 con 0x7f9bf005d070 2022-06-29T16:08:09.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.249+0000 7f9bed7fa700 1 -- 172.21.15.60:0/1947431255 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bd8007190 con 0x7f9bf005d070 2022-06-29T16:08:09.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.249+0000 7f9bf5652700 1 -- 172.21.15.60:0/1947431255 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf014d5f0 con 0x7f9bf005d070 2022-06-29T16:08:09.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.249+0000 7f9bed7fa700 1 -- 172.21.15.60:0/1947431255 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9bd8006510 con 0x7f9bf005d070 2022-06-29T16:08:09.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.249+0000 7f9bf5652700 1 -- 172.21.15.60:0/1947431255 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf00632d0 con 0x7f9bf005d070 2022-06-29T16:08:09.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.249+0000 7f9bed7fa700 1 -- 172.21.15.60:0/1947431255 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9bd8006670 con 0x7f9bf005d070 2022-06-29T16:08:09.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.253+0000 7f9bf5652700 1 -- 172.21.15.60:0/1947431255 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf014cb10 con 0x7f9bf005d070 2022-06-29T16:08:09.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.253+0000 7f9bed7fa700 1 --2- 172.21.15.60:0/1947431255 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9bd4053a70 0x7f9bd4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:09.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.253+0000 7f9bed7fa700 1 -- 172.21.15.60:0/1947431255 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f9bd806b420 con 0x7f9bf005d070 2022-06-29T16:08:09.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.253+0000 7f9bef7fe700 1 --2- 172.21.15.60:0/1947431255 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9bd4053a70 0x7f9bd4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:09.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.253+0000 7f9bef7fe700 1 --2- 172.21.15.60:0/1947431255 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9bd4053a70 0x7f9bd4055f30 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0x7f9be001b000 tx=0x7f9be0016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:09.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.257+0000 7f9bed7fa700 1 -- 172.21.15.60:0/1947431255 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9bd80425c0 con 0x7f9bf005d070 2022-06-29T16:08:09.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.397+0000 7f9bf5652700 1 -- 172.21.15.60:0/1947431255 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f9bf014a5f0 con 0x7f9bf005d070 2022-06-29T16:08:09.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.409+0000 7f9bed7fa700 1 -- 172.21.15.60:0/1947431255 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v23) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9bd8019070 con 0x7f9bf005d070 2022-06-29T16:08:09.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.409+0000 7f9bf5652700 1 -- 172.21.15.60:0/1947431255 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9bd4053a70 msgr2=0x7f9bd4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:09.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.409+0000 7f9bf5652700 1 --2- 172.21.15.60:0/1947431255 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9bd4053a70 0x7f9bd4055f30 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0x7f9be001b000 tx=0x7f9be0016000).stop 2022-06-29T16:08:09.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.409+0000 7f9bf5652700 1 -- 172.21.15.60:0/1947431255 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bf005d070 msgr2=0x7f9bf0146b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:09.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.409+0000 7f9bf5652700 1 --2- 172.21.15.60:0/1947431255 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bf005d070 0x7f9bf0146b40 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 rx=0x7f9bf0061990 tx=0x7f9bd8007c00).stop 2022-06-29T16:08:09.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.409+0000 7f9bf5652700 1 -- 172.21.15.60:0/1947431255 shutdown_connections 2022-06-29T16:08:09.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.409+0000 7f9bf5652700 1 --2- 172.21.15.60:0/1947431255 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9bd4053a70 0x7f9bd4055f30 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:09.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.409+0000 7f9bf5652700 1 --2- 172.21.15.60:0/1947431255 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9bf00b2130 0x7f9bf0065880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:09.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.409+0000 7f9bf5652700 1 --2- 172.21.15.60:0/1947431255 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bf00aed00 0x7f9bf0066970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:09.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.409+0000 7f9bf5652700 1 --2- 172.21.15.60:0/1947431255 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9bf005d070 0x7f9bf0146b40 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:09.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.409+0000 7f9bf5652700 1 -- 172.21.15.60:0/1947431255 >> 172.21.15.60:0/1947431255 conn(0x7f9bf00b5890 msgr2=0x7f9bf00b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:09.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.409+0000 7f9bf5652700 1 -- 172.21.15.60:0/1947431255 shutdown_connections 2022-06-29T16:08:09.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.409+0000 7f9bf5652700 1 -- 172.21.15.60:0/1947431255 wait complete. 2022-06-29T16:08:09.428 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_root_rctime' 2022-06-29T16:08:09.529 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.525+0000 7f909de95700 1 Processor -- start 2022-06-29T16:08:09.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.525+0000 7f909de95700 1 -- start start 2022-06-29T16:08:09.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.525+0000 7f909de95700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f909805fb00 0x7f90980aea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:09.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.525+0000 7f909de95700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f909814aa90 0x7f909814aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:09.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.525+0000 7f909de95700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f909805fee0 con 0x7f90980aef40 2022-06-29T16:08:09.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.525+0000 7f909de95700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90980b0870 con 0x7f909805fb00 2022-06-29T16:08:09.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.525+0000 7f909de95700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90980b0b10 con 0x7f909814aa90 2022-06-29T16:08:09.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.525+0000 7f909ce93700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f909805fb00 0x7f90980aea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:09.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.525+0000 7f909d694700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f909814aa90 0x7f909814aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:09.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.525+0000 7f909ce93700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f909805fb00 0x7f90980aea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:60120/0 (socket says 172.21.15.60:60120) 2022-06-29T16:08:09.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.525+0000 7f909ce93700 1 -- 172.21.15.60:0/1930979292 learned_addr learned my addr 172.21.15.60:0/1930979292 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:09.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.525+0000 7f9097fff700 1 --1- 172.21.15.60:0/1930979292 >> v1:172.21.15.92:6789/0 conn(0x7f90980aef40 0x7f90980b1160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:38192/0 (socket says 172.21.15.60:38192) 2022-06-29T16:08:09.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.525+0000 7f909ce93700 1 -- 172.21.15.60:0/1930979292 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f909814aa90 msgr2=0x7f909814aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:09.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.525+0000 7f909ce93700 1 --2- 172.21.15.60:0/1930979292 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f909814aa90 0x7f909814aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:09.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.525+0000 7f909ce93700 1 -- 172.21.15.60:0/1930979292 >> v1:172.21.15.92:6789/0 conn(0x7f90980aef40 legacy=0x7f90980b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:09.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.525+0000 7f909ce93700 1 -- 172.21.15.60:0/1930979292 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90980b1870 con 0x7f909805fb00 2022-06-29T16:08:09.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.525+0000 7f909ce93700 1 --2- 172.21.15.60:0/1930979292 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f909805fb00 0x7f90980aea50 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0x7f9080002f70 tx=0x7f9080006eb0).ready entity=mon.1 client_cookie=921915d7916ff832 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:09.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.525+0000 7f90977fe700 1 -- 172.21.15.60:0/1930979292 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f908000d570 con 0x7f909805fb00 2022-06-29T16:08:09.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.525+0000 7f90977fe700 1 -- 172.21.15.60:0/1930979292 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9080000e90 con 0x7f909805fb00 2022-06-29T16:08:09.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.525+0000 7f90977fe700 1 -- 172.21.15.60:0/1930979292 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f908000d900 con 0x7f909805fb00 2022-06-29T16:08:09.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.529+0000 7f909de95700 1 -- 172.21.15.60:0/1930979292 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f909805fb00 msgr2=0x7f90980aea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:09.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.529+0000 7f909de95700 1 --2- 172.21.15.60:0/1930979292 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f909805fb00 0x7f90980aea50 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0x7f9080002f70 tx=0x7f9080006eb0).stop 2022-06-29T16:08:09.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.529+0000 7f909de95700 1 -- 172.21.15.60:0/1930979292 shutdown_connections 2022-06-29T16:08:09.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.529+0000 7f909de95700 1 --2- 172.21.15.60:0/1930979292 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f909814aa90 0x7f909814aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:09.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.529+0000 7f909de95700 1 --2- 172.21.15.60:0/1930979292 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f909805fb00 0x7f90980aea50 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:09.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.529+0000 7f909de95700 1 -- 172.21.15.60:0/1930979292 >> 172.21.15.60:0/1930979292 conn(0x7f9098063a60 msgr2=0x7f909805f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:09.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.529+0000 7f909de95700 1 -- 172.21.15.60:0/1930979292 shutdown_connections 2022-06-29T16:08:09.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.529+0000 7f909de95700 1 -- 172.21.15.60:0/1930979292 wait complete. 2022-06-29T16:08:09.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.529+0000 7f909de95700 1 Processor -- start 2022-06-29T16:08:09.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.529+0000 7f909de95700 1 -- start start 2022-06-29T16:08:09.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.529+0000 7f909de95700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f909805fb00 0x7f909811bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:09.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.533+0000 7f909de95700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90980aef40 0x7f9098156280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:09.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.533+0000 7f909de95700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f909814aa90 0x7f9098156770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:09.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.533+0000 7f909de95700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9098111c50 con 0x7f909814aa90 2022-06-29T16:08:09.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.533+0000 7f909de95700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909811ba00 con 0x7f90980aef40 2022-06-29T16:08:09.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.533+0000 7f909de95700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f909811b760 con 0x7f909805fb00 2022-06-29T16:08:09.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.533+0000 7f909d694700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f909814aa90 0x7f9098156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:09.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.533+0000 7f909ce93700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f909805fb00 0x7f909811bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:09.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.533+0000 7f909d694700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f909814aa90 0x7f9098156770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54100/0 (socket says 172.21.15.60:54100) 2022-06-29T16:08:09.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.533+0000 7f9097fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90980aef40 0x7f9098156280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:09.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.533+0000 7f909d694700 1 -- 172.21.15.60:0/3901507891 learned_addr learned my addr 172.21.15.60:0/3901507891 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:09.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.533+0000 7f909ce93700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f909805fb00 0x7f909811bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:38958/0 (socket says 172.21.15.60:38958) 2022-06-29T16:08:09.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.533+0000 7f909d694700 1 -- 172.21.15.60:0/3901507891 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f909805fb00 msgr2=0x7f909811bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:09.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.533+0000 7f909d694700 1 --2- 172.21.15.60:0/3901507891 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f909805fb00 0x7f909811bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:09.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.533+0000 7f909d694700 1 -- 172.21.15.60:0/3901507891 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90980aef40 msgr2=0x7f9098156280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:09.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.533+0000 7f909d694700 1 --2- 172.21.15.60:0/3901507891 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90980aef40 0x7f9098156280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:09.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.533+0000 7f909d694700 1 -- 172.21.15.60:0/3901507891 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908000b1e0 con 0x7f909814aa90 2022-06-29T16:08:09.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.533+0000 7f9097fff700 1 --2- 172.21.15.60:0/3901507891 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90980aef40 0x7f9098156280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:08:09.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.533+0000 7f909d694700 1 --2- 172.21.15.60:0/3901507891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f909814aa90 0x7f9098156770 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 rx=0x7f909811b680 tx=0x7f9088006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:09.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.533+0000 7f9095ffb700 1 -- 172.21.15.60:0/3901507891 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9088011750 con 0x7f909814aa90 2022-06-29T16:08:09.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.533+0000 7f9095ffb700 1 -- 172.21.15.60:0/3901507891 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90880118b0 con 0x7f909814aa90 2022-06-29T16:08:09.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.533+0000 7f9095ffb700 1 -- 172.21.15.60:0/3901507891 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9088011c10 con 0x7f909814aa90 2022-06-29T16:08:09.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.533+0000 7f909de95700 1 -- 172.21.15.60:0/3901507891 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90981118d0 con 0x7f909814aa90 2022-06-29T16:08:09.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.533+0000 7f909de95700 1 -- 172.21.15.60:0/3901507891 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f909811e950 con 0x7f909814aa90 2022-06-29T16:08:09.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.533+0000 7f909de95700 1 -- 172.21.15.60:0/3901507891 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90980573d0 con 0x7f909814aa90 2022-06-29T16:08:09.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.537+0000 7f9095ffb700 1 -- 172.21.15.60:0/3901507891 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9088011d70 con 0x7f909814aa90 2022-06-29T16:08:09.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.537+0000 7f9095ffb700 1 --2- 172.21.15.60:0/3901507891 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f907c0539e0 0x7f907c055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:09.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.537+0000 7f9095ffb700 1 -- 172.21.15.60:0/3901507891 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f908806be70 con 0x7f909814aa90 2022-06-29T16:08:09.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.537+0000 7f909ce93700 1 --2- 172.21.15.60:0/3901507891 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f907c0539e0 0x7f907c055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:09.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.537+0000 7f909ce93700 1 --2- 172.21.15.60:0/3901507891 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f907c0539e0 0x7f907c055ea0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0x7f908001c000 tx=0x7f9080015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:09.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.541+0000 7f9095ffb700 1 -- 172.21.15.60:0/3901507891 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9088043070 con 0x7f909814aa90 2022-06-29T16:08:09.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:09.709+0000 7f909de95700 1 -- 172.21.15.60:0/3901507891 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_root_rctime"]} v 0) v1 -- 0x7f9098068560 con 0x7f909814aa90 2022-06-29T16:08:10.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.109+0000 7f9095ffb700 1 -- 172.21.15.60:0/3901507891 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_root_rctime"]}]=0 v453) v1 ==== 127+0+0 (secure 0 0 0) 0x7f9088046690 con 0x7f909814aa90 2022-06-29T16:08:10.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.113+0000 7f909de95700 1 -- 172.21.15.60:0/3901507891 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f907c0539e0 msgr2=0x7f907c055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:10.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.113+0000 7f909de95700 1 --2- 172.21.15.60:0/3901507891 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f907c0539e0 0x7f907c055ea0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0x7f908001c000 tx=0x7f9080015000).stop 2022-06-29T16:08:10.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.113+0000 7f909de95700 1 -- 172.21.15.60:0/3901507891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f909814aa90 msgr2=0x7f9098156770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:10.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.113+0000 7f909de95700 1 --2- 172.21.15.60:0/3901507891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f909814aa90 0x7f9098156770 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 rx=0x7f909811b680 tx=0x7f9088006df0).stop 2022-06-29T16:08:10.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.113+0000 7f909de95700 1 -- 172.21.15.60:0/3901507891 shutdown_connections 2022-06-29T16:08:10.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.113+0000 7f909de95700 1 --2- 172.21.15.60:0/3901507891 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f907c0539e0 0x7f907c055ea0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:10.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.113+0000 7f909de95700 1 --2- 172.21.15.60:0/3901507891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f909814aa90 0x7f9098156770 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:10.120 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.113+0000 7f909de95700 1 --2- 172.21.15.60:0/3901507891 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90980aef40 0x7f9098156280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:10.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.113+0000 7f909de95700 1 --2- 172.21.15.60:0/3901507891 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f909805fb00 0x7f909811bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:10.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.113+0000 7f909de95700 1 -- 172.21.15.60:0/3901507891 >> 172.21.15.60:0/3901507891 conn(0x7f9098063a60 msgr2=0x7f90980b4060 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:10.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.113+0000 7f909de95700 1 -- 172.21.15.60:0/3901507891 shutdown_connections 2022-06-29T16:08:10.121 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.113+0000 7f909de95700 1 -- 172.21.15.60:0/3901507891 wait complete. 2022-06-29T16:08:10.132 INFO:tasks.cephfs_test_runner:test_root_rctime (tasks.cephfs.test_misc.TestMisc) ... ok 2022-06-29T16:08:10.132 INFO:tasks.cephfs_test_runner:Starting test: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-06-29T16:08:10.133 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_statfs_on_deleted_fs' 2022-06-29T16:08:10.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.237+0000 7f00b088a700 1 Processor -- start 2022-06-29T16:08:10.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.237+0000 7f00b088a700 1 -- start start 2022-06-29T16:08:10.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.237+0000 7f00b088a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00ac0aed00 0x7f00ac146810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:10.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.237+0000 7f00b088a700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00ac05d450 con 0x7f00ac0b2130 2022-06-29T16:08:10.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.237+0000 7f00b088a700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00ac0b25d0 con 0x7f00ac05d070 2022-06-29T16:08:10.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.237+0000 7f00b088a700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac0adf90 con 0x7f00ac0aed00 2022-06-29T16:08:10.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.237+0000 7f00aaffd700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f00ac0b2130 0x7f00ac146100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33958/0 (socket says 172.21.15.60:33958) 2022-06-29T16:08:10.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.237+0000 7f00aaffd700 1 -- 172.21.15.60:0/3400758305 learned_addr learned my addr 172.21.15.60:0/3400758305 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:10.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.237+0000 7f00abfff700 1 --2- 172.21.15.60:0/3400758305 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00ac0aed00 0x7f00ac146810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:10.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.237+0000 7f00aa7fc700 1 -- 172.21.15.60:0/3400758305 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1521066137 0 0) 0x7f00ac05d450 con 0x7f00ac0b2130 2022-06-29T16:08:10.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.237+0000 7f00aa7fc700 1 -- 172.21.15.60:0/3400758305 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0098003260 con 0x7f00ac0b2130 2022-06-29T16:08:10.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.237+0000 7f00aa7fc700 1 -- 172.21.15.60:0/3400758305 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1777909981 0 0) 0x7f00ac0b25d0 con 0x7f00ac05d070 2022-06-29T16:08:10.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.237+0000 7f00aa7fc700 1 -- 172.21.15.60:0/3400758305 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00ac05d450 con 0x7f00ac05d070 2022-06-29T16:08:10.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.237+0000 7f00abfff700 1 -- 172.21.15.60:0/3400758305 >> v1:172.21.15.92:6789/0 conn(0x7f00ac05d070 legacy=0x7f00ac14aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:10.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.237+0000 7f00aa7fc700 1 -- 172.21.15.60:0/3400758305 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1606102949 0 0) 0x7f0098003260 con 0x7f00ac0b2130 2022-06-29T16:08:10.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.237+0000 7f00abfff700 1 -- 172.21.15.60:0/3400758305 >> v1:172.21.15.60:6789/0 conn(0x7f00ac0b2130 legacy=0x7f00ac146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:10.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.237+0000 7f00abfff700 1 -- 172.21.15.60:0/3400758305 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ac14a350 con 0x7f00ac0aed00 2022-06-29T16:08:10.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.237+0000 7f00abfff700 1 --2- 172.21.15.60:0/3400758305 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00ac0aed00 0x7f00ac146810 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f009c001490 tx=0x7f00a4006df0).ready entity=mon.2 client_cookie=f224e3aa5f2dd94d server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:10.251 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.241+0000 7f00aa7fc700 1 -- 172.21.15.60:0/3400758305 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f009c001a00 con 0x7f00ac0aed00 2022-06-29T16:08:10.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.241+0000 7f00aa7fc700 1 -- 172.21.15.60:0/3400758305 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00a4000e90 con 0x7f00ac0aed00 2022-06-29T16:08:10.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.241+0000 7f00aa7fc700 1 -- 172.21.15.60:0/3400758305 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f00a4012460 con 0x7f00ac0aed00 2022-06-29T16:08:10.252 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.241+0000 7f00b088a700 1 -- 172.21.15.60:0/3400758305 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00ac0aed00 msgr2=0x7f00ac146810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:10.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.241+0000 7f00b088a700 1 --2- 172.21.15.60:0/3400758305 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00ac0aed00 0x7f00ac146810 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f009c001490 tx=0x7f00a4006df0).stop 2022-06-29T16:08:10.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.241+0000 7f00b088a700 1 -- 172.21.15.60:0/3400758305 shutdown_connections 2022-06-29T16:08:10.253 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.241+0000 7f00b088a700 1 --2- 172.21.15.60:0/3400758305 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00ac0aed00 0x7f00ac146810 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:10.254 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.241+0000 7f00b088a700 1 -- 172.21.15.60:0/3400758305 >> 172.21.15.60:0/3400758305 conn(0x7f00ac0b5890 msgr2=0x7f00ac0b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:10.254 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.241+0000 7f00b088a700 1 -- 172.21.15.60:0/3400758305 shutdown_connections 2022-06-29T16:08:10.255 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.241+0000 7f00b088a700 1 -- 172.21.15.60:0/3400758305 wait complete. 2022-06-29T16:08:10.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.241+0000 7f00b088a700 1 Processor -- start 2022-06-29T16:08:10.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.241+0000 7f00b088a700 1 -- start start 2022-06-29T16:08:10.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.245+0000 7f00b088a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00ac05d070 0x7f00ac14d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:10.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.245+0000 7f00b088a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00ac0aed00 0x7f00ac156f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:10.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.245+0000 7f00b088a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00ac0b2130 0x7f00ac1481b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:10.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.245+0000 7f00b088a700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac157440 con 0x7f00ac05d070 2022-06-29T16:08:10.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.245+0000 7f00b088a700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac156570 con 0x7f00ac0aed00 2022-06-29T16:08:10.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.245+0000 7f00b088a700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac156810 con 0x7f00ac0b2130 2022-06-29T16:08:10.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.245+0000 7f00ab7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00ac05d070 0x7f00ac14d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:10.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.245+0000 7f00ab7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00ac05d070 0x7f00ac14d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54110/0 (socket says 172.21.15.60:54110) 2022-06-29T16:08:10.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.245+0000 7f00ab7fe700 1 -- 172.21.15.60:0/3103761409 learned_addr learned my addr 172.21.15.60:0/3103761409 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:10.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.245+0000 7f00abfff700 1 --2- 172.21.15.60:0/3103761409 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00ac0b2130 0x7f00ac1481b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:10.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.245+0000 7f00aaffd700 1 --2- 172.21.15.60:0/3103761409 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00ac0aed00 0x7f00ac156f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:10.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.245+0000 7f00ab7fe700 1 -- 172.21.15.60:0/3103761409 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00ac0b2130 msgr2=0x7f00ac1481b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:10.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.245+0000 7f00ab7fe700 1 --2- 172.21.15.60:0/3103761409 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00ac0b2130 0x7f00ac1481b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:10.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.245+0000 7f00ab7fe700 1 -- 172.21.15.60:0/3103761409 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00ac0aed00 msgr2=0x7f00ac156f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:10.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.245+0000 7f00ab7fe700 1 --2- 172.21.15.60:0/3103761409 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00ac0aed00 0x7f00ac156f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:10.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.245+0000 7f00ab7fe700 1 -- 172.21.15.60:0/3103761409 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a400b180 con 0x7f00ac05d070 2022-06-29T16:08:10.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.245+0000 7f00ab7fe700 1 --2- 172.21.15.60:0/3103761409 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00ac05d070 0x7f00ac14d170 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 rx=0x7f00a0001110 tx=0x7f00a0007ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:10.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.245+0000 7f00a8ff9700 1 -- 172.21.15.60:0/3103761409 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f00a0012400 con 0x7f00ac05d070 2022-06-29T16:08:10.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.245+0000 7f00a8ff9700 1 -- 172.21.15.60:0/3103761409 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00a0012560 con 0x7f00ac05d070 2022-06-29T16:08:10.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.245+0000 7f00b088a700 1 -- 172.21.15.60:0/3103761409 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00ac156ab0 con 0x7f00ac05d070 2022-06-29T16:08:10.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.245+0000 7f00a8ff9700 1 -- 172.21.15.60:0/3103761409 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f00a00128c0 con 0x7f00ac05d070 2022-06-29T16:08:10.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.245+0000 7f00b088a700 1 -- 172.21.15.60:0/3103761409 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00ac14bbc0 con 0x7f00ac05d070 2022-06-29T16:08:10.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.245+0000 7f00b088a700 1 -- 172.21.15.60:0/3103761409 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00ac14cab0 con 0x7f00ac05d070 2022-06-29T16:08:10.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.245+0000 7f00a8ff9700 1 -- 172.21.15.60:0/3103761409 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f00a0019030 con 0x7f00ac05d070 2022-06-29T16:08:10.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.249+0000 7f00a8ff9700 1 --2- 172.21.15.60:0/3103761409 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f008c053a70 0x7f008c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:10.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.249+0000 7f00a8ff9700 1 -- 172.21.15.60:0/3103761409 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f00a006aff0 con 0x7f00ac05d070 2022-06-29T16:08:10.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.249+0000 7f00aaffd700 1 --2- 172.21.15.60:0/3103761409 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f008c053a70 0x7f008c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:10.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.249+0000 7f00aaffd700 1 --2- 172.21.15.60:0/3103761409 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f008c053a70 0x7f008c055f30 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0x7f009c00d000 tx=0x7f009c000f40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:10.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.253+0000 7f00a8ff9700 1 -- 172.21.15.60:0/3103761409 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f00ac14cab0 con 0x7f00ac05d070 2022-06-29T16:08:10.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:10.417+0000 7f00b088a700 1 -- 172.21.15.60:0/3103761409 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_statfs_on_deleted_fs"]} v 0) v1 -- 0x7f00ac14cab0 con 0x7f00ac05d070 2022-06-29T16:08:11.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.113+0000 7f00a8ff9700 1 -- 172.21.15.60:0/3103761409 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_statfs_on_deleted_fs"]}]=0 v454) v1 ==== 139+0+0 (secure 0 0 0) 0x7f00ac14cab0 con 0x7f00ac05d070 2022-06-29T16:08:11.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.117+0000 7f00b088a700 1 -- 172.21.15.60:0/3103761409 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f008c053a70 msgr2=0x7f008c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:11.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.117+0000 7f00b088a700 1 --2- 172.21.15.60:0/3103761409 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f008c053a70 0x7f008c055f30 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0x7f009c00d000 tx=0x7f009c000f40).stop 2022-06-29T16:08:11.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.117+0000 7f00b088a700 1 -- 172.21.15.60:0/3103761409 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00ac05d070 msgr2=0x7f00ac14d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:11.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.117+0000 7f00b088a700 1 --2- 172.21.15.60:0/3103761409 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00ac05d070 0x7f00ac14d170 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 rx=0x7f00a0001110 tx=0x7f00a0007ae0).stop 2022-06-29T16:08:11.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.117+0000 7f00b088a700 1 -- 172.21.15.60:0/3103761409 shutdown_connections 2022-06-29T16:08:11.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.117+0000 7f00b088a700 1 --2- 172.21.15.60:0/3103761409 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f008c053a70 0x7f008c055f30 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:11.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.117+0000 7f00b088a700 1 --2- 172.21.15.60:0/3103761409 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f00ac0b2130 0x7f00ac1481b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:11.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.117+0000 7f00b088a700 1 --2- 172.21.15.60:0/3103761409 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f00ac0aed00 0x7f00ac156f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:11.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.117+0000 7f00b088a700 1 --2- 172.21.15.60:0/3103761409 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00ac05d070 0x7f00ac14d170 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:11.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.117+0000 7f00b088a700 1 -- 172.21.15.60:0/3103761409 >> 172.21.15.60:0/3103761409 conn(0x7f00ac0b5890 msgr2=0x7f00ac0b44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:11.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.117+0000 7f00b088a700 1 -- 172.21.15.60:0/3103761409 shutdown_connections 2022-06-29T16:08:11.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.117+0000 7f00b088a700 1 -- 172.21.15.60:0/3103761409 wait complete. 2022-06-29T16:08:11.140 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-06-29T16:08:11.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.253+0000 7f89d25c4700 1 Processor -- start 2022-06-29T16:08:11.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.253+0000 7f89d25c4700 1 -- start start 2022-06-29T16:08:11.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.257+0000 7f89d25c4700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f89cc05c990 0x7f89cc0b0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:11.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.257+0000 7f89d25c4700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f89cc05f620 0x7f89cc146990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:11.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.257+0000 7f89d25c4700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89cc05fad0 con 0x7f89cc05c5b0 2022-06-29T16:08:11.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.257+0000 7f89d25c4700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89cc146e80 con 0x7f89cc05c990 2022-06-29T16:08:11.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.257+0000 7f89d25c4700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89cc145fb0 con 0x7f89cc05f620 2022-06-29T16:08:11.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.257+0000 7f89d0dc1700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f89cc05c990 0x7f89cc0b0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:11.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.257+0000 7f89d0dc1700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f89cc05c990 0x7f89cc0b0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54122/0 (socket says 172.21.15.60:54122) 2022-06-29T16:08:11.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.257+0000 7f89d0dc1700 1 -- 172.21.15.60:0/2189001902 learned_addr learned my addr 172.21.15.60:0/2189001902 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:11.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.257+0000 7f89d1dc3700 1 --2- 172.21.15.60:0/2189001902 >> v2:172.21.15.92:3300/0 conn(0x7f89cc05f620 0x7f89cc146990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:11.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.257+0000 7f89d0dc1700 1 -- 172.21.15.60:0/2189001902 >> v2:172.21.15.92:3300/0 conn(0x7f89cc05f620 msgr2=0x7f89cc146990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:11.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.257+0000 7f89d0dc1700 1 --2- 172.21.15.60:0/2189001902 >> v2:172.21.15.92:3300/0 conn(0x7f89cc05f620 0x7f89cc146990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:11.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.257+0000 7f89d0dc1700 1 -- 172.21.15.60:0/2189001902 >> v1:172.21.15.92:6789/0 conn(0x7f89cc05c5b0 legacy=0x7f89cc13c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:11.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.257+0000 7f89d0dc1700 1 -- 172.21.15.60:0/2189001902 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89cc146250 con 0x7f89cc05c990 2022-06-29T16:08:11.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.257+0000 7f89d1dc3700 1 --2- 172.21.15.60:0/2189001902 >> v2:172.21.15.92:3300/0 conn(0x7f89cc05f620 0x7f89cc146990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:08:11.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.257+0000 7f89d0dc1700 1 --2- 172.21.15.60:0/2189001902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89cc05c990 0x7f89cc0b0c80 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 rx=0x7f89c0002f70 tx=0x7f89c0006eb0).ready entity=mon.0 client_cookie=9396247b7e142121 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:11.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.257+0000 7f89b7fff700 1 -- 172.21.15.60:0/2189001902 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89c000d610 con 0x7f89cc05c990 2022-06-29T16:08:11.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.257+0000 7f89b7fff700 1 -- 172.21.15.60:0/2189001902 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89c0000e90 con 0x7f89cc05c990 2022-06-29T16:08:11.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.257+0000 7f89b7fff700 1 -- 172.21.15.60:0/2189001902 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89c000d9a0 con 0x7f89cc05c990 2022-06-29T16:08:11.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.257+0000 7f89d25c4700 1 -- 172.21.15.60:0/2189001902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89cc05c990 msgr2=0x7f89cc0b0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:11.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.257+0000 7f89d25c4700 1 --2- 172.21.15.60:0/2189001902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89cc05c990 0x7f89cc0b0c80 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 rx=0x7f89c0002f70 tx=0x7f89c0006eb0).stop 2022-06-29T16:08:11.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.261+0000 7f89d25c4700 1 -- 172.21.15.60:0/2189001902 shutdown_connections 2022-06-29T16:08:11.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.261+0000 7f89d25c4700 1 --2- 172.21.15.60:0/2189001902 >> v2:172.21.15.92:3300/0 conn(0x7f89cc05f620 0x7f89cc146990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:11.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.261+0000 7f89d25c4700 1 --2- 172.21.15.60:0/2189001902 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89cc05c990 0x7f89cc0b0c80 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:11.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.261+0000 7f89d25c4700 1 -- 172.21.15.60:0/2189001902 >> 172.21.15.60:0/2189001902 conn(0x7f89cc0637c0 msgr2=0x7f89cc063ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:11.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.261+0000 7f89d25c4700 1 -- 172.21.15.60:0/2189001902 shutdown_connections 2022-06-29T16:08:11.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.261+0000 7f89d25c4700 1 -- 172.21.15.60:0/2189001902 wait complete. 2022-06-29T16:08:11.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.261+0000 7f89d25c4700 1 Processor -- start 2022-06-29T16:08:11.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.261+0000 7f89d25c4700 1 -- start start 2022-06-29T16:08:11.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.261+0000 7f89d25c4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89cc05c5b0 0x7f89cc0b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:11.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.261+0000 7f89d25c4700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f89cc05c990 0x7f89cc1513d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:11.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.261+0000 7f89d25c4700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f89cc05f620 0x7f89cc1518c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:11.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.261+0000 7f89d25c4700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc13d590 con 0x7f89cc05c5b0 2022-06-29T16:08:11.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.261+0000 7f89d25c4700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc0ae9d0 con 0x7f89cc05c990 2022-06-29T16:08:11.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.261+0000 7f89d25c4700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc0af510 con 0x7f89cc05f620 2022-06-29T16:08:11.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.261+0000 7f89d15c2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89cc05c5b0 0x7f89cc0b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:11.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.261+0000 7f89d15c2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89cc05c5b0 0x7f89cc0b4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54126/0 (socket says 172.21.15.60:54126) 2022-06-29T16:08:11.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.261+0000 7f89d15c2700 1 -- 172.21.15.60:0/240884793 learned_addr learned my addr 172.21.15.60:0/240884793 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:11.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.261+0000 7f89d0dc1700 1 --2- 172.21.15.60:0/240884793 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f89cc05c990 0x7f89cc1513d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:11.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.261+0000 7f89d1dc3700 1 --2- 172.21.15.60:0/240884793 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f89cc05f620 0x7f89cc1518c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:11.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.261+0000 7f89d15c2700 1 -- 172.21.15.60:0/240884793 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f89cc05f620 msgr2=0x7f89cc1518c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:11.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.261+0000 7f89d15c2700 1 --2- 172.21.15.60:0/240884793 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f89cc05f620 0x7f89cc1518c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:11.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.261+0000 7f89d15c2700 1 -- 172.21.15.60:0/240884793 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f89cc05c990 msgr2=0x7f89cc1513d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:11.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.261+0000 7f89d15c2700 1 --2- 172.21.15.60:0/240884793 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f89cc05c990 0x7f89cc1513d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:11.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.261+0000 7f89d15c2700 1 -- 172.21.15.60:0/240884793 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89c000b1e0 con 0x7f89cc05c5b0 2022-06-29T16:08:11.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.261+0000 7f89d15c2700 1 --2- 172.21.15.60:0/240884793 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89cc05c5b0 0x7f89cc0b4fe0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 rx=0x7f89cc0b39e0 tx=0x7f89c8007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:11.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.261+0000 7f89beffd700 1 -- 172.21.15.60:0/240884793 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89c8007030 con 0x7f89cc05c5b0 2022-06-29T16:08:11.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.261+0000 7f89beffd700 1 -- 172.21.15.60:0/240884793 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89c8007190 con 0x7f89cc05c5b0 2022-06-29T16:08:11.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.261+0000 7f89d25c4700 1 -- 172.21.15.60:0/240884793 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89cc0af7b0 con 0x7f89cc05c5b0 2022-06-29T16:08:11.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.261+0000 7f89beffd700 1 -- 172.21.15.60:0/240884793 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89c8006510 con 0x7f89cc05c5b0 2022-06-29T16:08:11.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.261+0000 7f89d25c4700 1 -- 172.21.15.60:0/240884793 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89cc151e30 con 0x7f89cc05c5b0 2022-06-29T16:08:11.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.265+0000 7f89d25c4700 1 -- 172.21.15.60:0/240884793 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89cc0573d0 con 0x7f89cc05c5b0 2022-06-29T16:08:11.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.265+0000 7f89beffd700 1 -- 172.21.15.60:0/240884793 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f89c8010030 con 0x7f89cc05c5b0 2022-06-29T16:08:11.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.265+0000 7f89beffd700 1 --2- 172.21.15.60:0/240884793 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f89a8053980 0x7f89a8055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:11.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.265+0000 7f89beffd700 1 -- 172.21.15.60:0/240884793 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f89c806b7f0 con 0x7f89cc05c5b0 2022-06-29T16:08:11.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.265+0000 7f89d0dc1700 1 --2- 172.21.15.60:0/240884793 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f89a8053980 0x7f89a8055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:11.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.265+0000 7f89d0dc1700 1 --2- 172.21.15.60:0/240884793 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f89a8053980 0x7f89a8055e40 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0x7f89c001c000 tx=0x7f89c0015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:11.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.273+0000 7f89beffd700 1 -- 172.21.15.60:0/240884793 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f89c80428e0 con 0x7f89cc05c5b0 2022-06-29T16:08:11.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.397+0000 7f89d25c4700 1 -- 172.21.15.60:0/240884793 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f89cc067d20 con 0x7f89cc05c5b0 2022-06-29T16:08:11.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.409+0000 7f89beffd700 1 -- 172.21.15.60:0/240884793 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v24)=0 v24) v1 ==== 122+0+0 (secure 0 0 0) 0x7f89c8045f00 con 0x7f89cc05c5b0 2022-06-29T16:08:11.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.413+0000 7f89d25c4700 1 -- 172.21.15.60:0/240884793 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f89a8053980 msgr2=0x7f89a8055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:11.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.413+0000 7f89d25c4700 1 --2- 172.21.15.60:0/240884793 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f89a8053980 0x7f89a8055e40 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0x7f89c001c000 tx=0x7f89c0015000).stop 2022-06-29T16:08:11.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.413+0000 7f89d25c4700 1 -- 172.21.15.60:0/240884793 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89cc05c5b0 msgr2=0x7f89cc0b4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:11.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.413+0000 7f89d25c4700 1 --2- 172.21.15.60:0/240884793 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89cc05c5b0 0x7f89cc0b4fe0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 rx=0x7f89cc0b39e0 tx=0x7f89c8007c10).stop 2022-06-29T16:08:11.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.413+0000 7f89d25c4700 1 -- 172.21.15.60:0/240884793 shutdown_connections 2022-06-29T16:08:11.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.413+0000 7f89d25c4700 1 --2- 172.21.15.60:0/240884793 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f89a8053980 0x7f89a8055e40 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:11.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.413+0000 7f89d25c4700 1 --2- 172.21.15.60:0/240884793 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f89cc05f620 0x7f89cc1518c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:11.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.413+0000 7f89d25c4700 1 --2- 172.21.15.60:0/240884793 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f89cc05c990 0x7f89cc1513d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:11.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.413+0000 7f89d25c4700 1 --2- 172.21.15.60:0/240884793 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f89cc05c5b0 0x7f89cc0b4fe0 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:11.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.413+0000 7f89d25c4700 1 -- 172.21.15.60:0/240884793 >> 172.21.15.60:0/240884793 conn(0x7f89cc0637c0 msgr2=0x7f89cc05fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:11.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.413+0000 7f89d25c4700 1 -- 172.21.15.60:0/240884793 shutdown_connections 2022-06-29T16:08:11.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.413+0000 7f89d25c4700 1 -- 172.21.15.60:0/240884793 wait complete. 2022-06-29T16:08:11.427 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-06-29T16:08:11.428 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:08:11.445 DEBUG:teuthology.orchestra.run.smithi092:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:08:11.467 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:08:11.489 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:08:11.512 DEBUG:teuthology.orchestra.run.smithi197:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:08:11.535 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-06-29T16:08:11.536 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:11.537 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:08:11.537 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:08:11.545 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-06-29T16:08:11.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:08:11.548 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-06-29T16:08:11.549 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:11.549 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:08:11.550 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:08:11.558 INFO:teuthology.orchestra.run.smithi156.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-06-29T16:08:11.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:08:11.559 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-06-29T16:08:11.560 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:11.560 DEBUG:teuthology.orchestra.run.smithi197:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:08:11.561 DEBUG:teuthology.orchestra.run.smithi197:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-06-29T16:08:11.568 INFO:teuthology.orchestra.run.smithi197.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-06-29T16:08:11.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:08:11.570 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-06-29T16:08:11.570 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-06-29T16:08:11.680 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.677+0000 7fca487f7700 1 Processor -- start 2022-06-29T16:08:11.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.677+0000 7fca487f7700 1 -- start start 2022-06-29T16:08:11.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.677+0000 7fca487f7700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fca400b5950 0x7fca40067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:11.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.677+0000 7fca487f7700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fca40063ad0 0x7fca4014af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:11.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.677+0000 7fca487f7700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca400acba0 con 0x7fca4005bd20 2022-06-29T16:08:11.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.677+0000 7fca487f7700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca40067220 con 0x7fca400b5950 2022-06-29T16:08:11.685 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.677+0000 7fca487f7700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca400b5d30 con 0x7fca40063ad0 2022-06-29T16:08:11.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.677+0000 7fca46593700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fca400b5950 0x7fca40067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:11.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.677+0000 7fca46593700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fca400b5950 0x7fca40067680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54134/0 (socket says 172.21.15.60:54134) 2022-06-29T16:08:11.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.677+0000 7fca46593700 1 -- 172.21.15.60:0/422530192 learned_addr learned my addr 172.21.15.60:0/422530192 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:11.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.677+0000 7fca45d92700 1 --2- 172.21.15.60:0/422530192 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fca40063ad0 0x7fca4014af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:11.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.677+0000 7fca46593700 1 -- 172.21.15.60:0/422530192 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fca40063ad0 msgr2=0x7fca4014af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:11.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.677+0000 7fca46593700 1 --2- 172.21.15.60:0/422530192 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fca40063ad0 0x7fca4014af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:11.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.677+0000 7fca46593700 1 -- 172.21.15.60:0/422530192 >> v1:172.21.15.92:6789/0 conn(0x7fca4005bd20 legacy=0x7fca4005c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:11.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.677+0000 7fca46593700 1 -- 172.21.15.60:0/422530192 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca400ace00 con 0x7fca400b5950 2022-06-29T16:08:11.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.677+0000 7fca46593700 1 --2- 172.21.15.60:0/422530192 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca400b5950 0x7fca40067680 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 rx=0x7fca28002f70 tx=0x7fca28006eb0).ready entity=mon.0 client_cookie=4efa7e94032faa64 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:11.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.677+0000 7fca45591700 1 -- 172.21.15.60:0/422530192 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca2800d570 con 0x7fca400b5950 2022-06-29T16:08:11.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.677+0000 7fca45591700 1 -- 172.21.15.60:0/422530192 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca28000e90 con 0x7fca400b5950 2022-06-29T16:08:11.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.677+0000 7fca45591700 1 -- 172.21.15.60:0/422530192 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca2800d900 con 0x7fca400b5950 2022-06-29T16:08:11.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.677+0000 7fca487f7700 1 -- 172.21.15.60:0/422530192 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca400b5950 msgr2=0x7fca40067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:11.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.677+0000 7fca487f7700 1 --2- 172.21.15.60:0/422530192 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca400b5950 0x7fca40067680 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 rx=0x7fca28002f70 tx=0x7fca28006eb0).stop 2022-06-29T16:08:11.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.677+0000 7fca487f7700 1 -- 172.21.15.60:0/422530192 shutdown_connections 2022-06-29T16:08:11.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.677+0000 7fca487f7700 1 --2- 172.21.15.60:0/422530192 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fca40063ad0 0x7fca4014af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:11.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.677+0000 7fca487f7700 1 --2- 172.21.15.60:0/422530192 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca400b5950 0x7fca40067680 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:11.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.677+0000 7fca487f7700 1 -- 172.21.15.60:0/422530192 >> 172.21.15.60:0/422530192 conn(0x7fca400632c0 msgr2=0x7fca400b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:11.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.681+0000 7fca487f7700 1 -- 172.21.15.60:0/422530192 shutdown_connections 2022-06-29T16:08:11.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.681+0000 7fca487f7700 1 -- 172.21.15.60:0/422530192 wait complete. 2022-06-29T16:08:11.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.681+0000 7fca487f7700 1 Processor -- start 2022-06-29T16:08:11.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.681+0000 7fca487f7700 1 -- start start 2022-06-29T16:08:11.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.681+0000 7fca487f7700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fca4005bd20 0x7fca40116b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:11.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.681+0000 7fca487f7700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fca40063ad0 0x7fca40159c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:11.696 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.681+0000 7fca487f7700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca400b3f80 0x7fca4015a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:11.696 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.681+0000 7fca487f7700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca400af740 con 0x7fca400b3f80 2022-06-29T16:08:11.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.681+0000 7fca487f7700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca400af9e0 con 0x7fca4005bd20 2022-06-29T16:08:11.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.681+0000 7fca487f7700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca400afc80 con 0x7fca40063ad0 2022-06-29T16:08:11.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.681+0000 7fca46593700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fca4005bd20 0x7fca40116b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:11.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.681+0000 7fca46d94700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca400b3f80 0x7fca4015a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:11.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.681+0000 7fca46593700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fca4005bd20 0x7fca40116b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:60170/0 (socket says 172.21.15.60:60170) 2022-06-29T16:08:11.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.681+0000 7fca46d94700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca400b3f80 0x7fca4015a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54144/0 (socket says 172.21.15.60:54144) 2022-06-29T16:08:11.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.681+0000 7fca46593700 1 -- 172.21.15.60:0/707933705 learned_addr learned my addr 172.21.15.60:0/707933705 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:11.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.681+0000 7fca45d92700 1 --2- 172.21.15.60:0/707933705 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fca40063ad0 0x7fca40159c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:11.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.685+0000 7fca46593700 1 -- 172.21.15.60:0/707933705 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fca40063ad0 msgr2=0x7fca40159c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:11.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.685+0000 7fca46593700 1 --2- 172.21.15.60:0/707933705 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fca40063ad0 0x7fca40159c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:11.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.685+0000 7fca46593700 1 -- 172.21.15.60:0/707933705 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca400b3f80 msgr2=0x7fca4015a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:11.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.685+0000 7fca46593700 1 --2- 172.21.15.60:0/707933705 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca400b3f80 0x7fca4015a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:11.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.685+0000 7fca46593700 1 -- 172.21.15.60:0/707933705 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca2800b1e0 con 0x7fca4005bd20 2022-06-29T16:08:11.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.685+0000 7fca45d92700 1 --2- 172.21.15.60:0/707933705 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fca40063ad0 0x7fca40159c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:08:11.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.685+0000 7fca46d94700 1 --2- 172.21.15.60:0/707933705 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca400b3f80 0x7fca4015a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:08:11.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.685+0000 7fca46593700 1 --2- 172.21.15.60:0/707933705 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fca4005bd20 0x7fca40116b40 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 rx=0x7fca28002f70 tx=0x7fca28015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:11.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.685+0000 7fca3f7fe700 1 -- 172.21.15.60:0/707933705 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca28011710 con 0x7fca4005bd20 2022-06-29T16:08:11.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.685+0000 7fca3f7fe700 1 -- 172.21.15.60:0/707933705 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca28011870 con 0x7fca4005bd20 2022-06-29T16:08:11.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.685+0000 7fca487f7700 1 -- 172.21.15.60:0/707933705 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca40117030 con 0x7fca4005bd20 2022-06-29T16:08:11.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.685+0000 7fca3f7fe700 1 -- 172.21.15.60:0/707933705 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca28011bd0 con 0x7fca4005bd20 2022-06-29T16:08:11.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.685+0000 7fca487f7700 1 -- 172.21.15.60:0/707933705 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca400b4570 con 0x7fca4005bd20 2022-06-29T16:08:11.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.685+0000 7fca487f7700 1 -- 172.21.15.60:0/707933705 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca400af560 con 0x7fca4005bd20 2022-06-29T16:08:11.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.685+0000 7fca3f7fe700 1 -- 172.21.15.60:0/707933705 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fca2801a040 con 0x7fca4005bd20 2022-06-29T16:08:11.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.689+0000 7fca3f7fe700 1 --2- 172.21.15.60:0/707933705 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fca24053a70 0x7fca24055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:11.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.689+0000 7fca3f7fe700 1 -- 172.21.15.60:0/707933705 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fca2806cab0 con 0x7fca4005bd20 2022-06-29T16:08:11.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.689+0000 7fca45d92700 1 --2- 172.21.15.60:0/707933705 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fca24053a70 0x7fca24055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:11.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.689+0000 7fca45d92700 1 --2- 172.21.15.60:0/707933705 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fca24053a70 0x7fca24055f30 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 rx=0x7fca3000b000 tx=0x7fca300091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:11.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.697+0000 7fca3f7fe700 1 -- 172.21.15.60:0/707933705 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fca28043cd0 con 0x7fca4005bd20 2022-06-29T16:08:11.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.825+0000 7fca487f7700 1 -- 172.21.15.60:0/707933705 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fca400573d0 con 0x7fca4005bd20 2022-06-29T16:08:11.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.829+0000 7fca3f7fe700 1 -- 172.21.15.60:0/707933705 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v140) v1 ==== 81+0+28837 (secure 0 0 0) 0x7fca28023020 con 0x7fca4005bd20 2022-06-29T16:08:11.835 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:08:11.835 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:08:11.835 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 140, 2022-06-29T16:08:11.836 INFO:teuthology.orchestra.run.smithi060.stdout: "fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", 2022-06-29T16:08:11.836 INFO:teuthology.orchestra.run.smithi060.stdout: "created": "2022-06-29T15:58:07.965719+0000", 2022-06-29T16:08:11.836 INFO:teuthology.orchestra.run.smithi060.stdout: "modified": "2022-06-29T16:08:09.095384+0000", 2022-06-29T16:08:11.837 INFO:teuthology.orchestra.run.smithi060.stdout: "last_up_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:08:11.838 INFO:teuthology.orchestra.run.smithi060.stdout: "last_in_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:08:11.838 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-06-29T16:08:11.839 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_num": 5799936, 2022-06-29T16:08:11.839 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_set": [ 2022-06-29T16:08:11.840 INFO:teuthology.orchestra.run.smithi060.stdout: "pglog_hardlimit", 2022-06-29T16:08:11.840 INFO:teuthology.orchestra.run.smithi060.stdout: "purged_snapdirs", 2022-06-29T16:08:11.841 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_deletes", 2022-06-29T16:08:11.841 INFO:teuthology.orchestra.run.smithi060.stdout: "sortbitwise" 2022-06-29T16:08:11.841 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:08:11.842 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_version": 4, 2022-06-29T16:08:11.842 INFO:teuthology.orchestra.run.smithi060.stdout: "full_ratio": 0.94999998807907104, 2022-06-29T16:08:11.843 INFO:teuthology.orchestra.run.smithi060.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-06-29T16:08:11.843 INFO:teuthology.orchestra.run.smithi060.stdout: "nearfull_ratio": 0.85000002384185791, 2022-06-29T16:08:11.844 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_snapshot": "", 2022-06-29T16:08:11.844 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_max": 28, 2022-06-29T16:08:11.845 INFO:teuthology.orchestra.run.smithi060.stdout: "max_osd": 8, 2022-06-29T16:08:11.846 INFO:teuthology.orchestra.run.smithi060.stdout: "require_min_compat_client": "jewel", 2022-06-29T16:08:11.846 INFO:teuthology.orchestra.run.smithi060.stdout: "min_compat_client": "jewel", 2022-06-29T16:08:11.847 INFO:teuthology.orchestra.run.smithi060.stdout: "require_osd_release": "octopus", 2022-06-29T16:08:11.847 INFO:teuthology.orchestra.run.smithi060.stdout: "pools": [ 2022-06-29T16:08:11.847 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.847 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2022-06-29T16:08:11.847 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "device_health_metrics", 2022-06-29T16:08:11.848 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:10.961127+0000", 2022-06-29T16:08:11.848 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:08:11.848 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:08:11.848 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:08:11.849 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:08:11.849 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:08:11.849 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:08:11.849 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:08:11.849 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:08:11.850 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 1, 2022-06-29T16:08:11.850 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 1, 2022-06-29T16:08:11.850 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 1, 2022-06-29T16:08:11.850 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 1, 2022-06-29T16:08:11.851 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 1, 2022-06-29T16:08:11.851 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:08:11.851 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:08:11.851 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:08:11.851 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:08:11.852 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:08:11.852 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:08:11.852 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:08:11.852 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.853 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "15", 2022-06-29T16:08:11.853 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:08:11.853 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:08:11.853 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:08:11.853 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:08:11.854 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:08:11.854 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:08:11.854 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:08:11.854 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:08:11.855 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:08:11.855 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:08:11.855 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:08:11.855 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:08:11.855 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:08:11.856 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:08:11.856 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:08:11.856 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:08:11.856 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:08:11.856 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:08:11.857 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:08:11.857 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:08:11.857 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:08:11.857 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:08:11.858 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:08:11.858 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:08:11.858 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:08:11.858 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:08:11.859 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.859 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:08:11.859 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:08:11.859 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:08:11.860 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:08:11.860 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:08:11.860 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:08:11.860 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:08:11.861 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:08:11.861 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:08:11.861 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:08:11.861 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:08:11.862 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:08:11.862 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 1 2022-06-29T16:08:11.862 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.862 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:08:11.862 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_devicehealth": {} 2022-06-29T16:08:11.863 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:11.863 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.863 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.863 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 2, 2022-06-29T16:08:11.863 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "rbd", 2022-06-29T16:08:11.864 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:19.184276+0000", 2022-06-29T16:08:11.864 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:08:11.864 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:08:11.864 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:08:11.865 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:08:11.865 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:08:11.865 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:08:11.865 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:08:11.865 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:08:11.866 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:08:11.866 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:08:11.866 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:08:11.866 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:08:11.867 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:08:11.867 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:08:11.867 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:08:11.867 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:08:11.867 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:08:11.868 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:08:11.868 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:08:11.868 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:08:11.868 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.868 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "18", 2022-06-29T16:08:11.869 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:08:11.869 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:08:11.869 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:08:11.869 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:08:11.870 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:08:11.870 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:08:11.870 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:08:11.870 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:08:11.870 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:08:11.871 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:08:11.871 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:08:11.871 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:08:11.871 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:08:11.872 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:08:11.872 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:08:11.872 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:08:11.872 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:08:11.872 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:08:11.873 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:08:11.873 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:08:11.873 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:08:11.873 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:08:11.873 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:08:11.874 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:08:11.874 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:08:11.874 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:08:11.874 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.875 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:08:11.875 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:08:11.875 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:08:11.875 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:08:11.876 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:08:11.876 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:08:11.876 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:08:11.876 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:08:11.876 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:08:11.877 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:08:11.877 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:08:11.877 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:08:11.877 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:08:11.877 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd": {} 2022-06-29T16:08:11.878 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:11.878 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:11.878 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:08:11.878 INFO:teuthology.orchestra.run.smithi060.stdout: "osds": [ 2022-06-29T16:08:11.879 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.879 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:08:11.879 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "11e7ab9e-fce2-470f-8195-ea4c3c52df22", 2022-06-29T16:08:11.879 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:08:11.880 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:08:11.880 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:08:11.880 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:08:11.880 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:08:11.880 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:08:11.881 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:08:11.881 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 135, 2022-06-29T16:08:11.881 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:08:11.881 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:08:11.881 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:08:11.882 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:11.882 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.882 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:11.882 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6816", 2022-06-29T16:08:11.883 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:08:11.883 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.883 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.883 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:11.883 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6825", 2022-06-29T16:08:11.884 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:08:11.884 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:11.884 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.884 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.884 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:08:11.885 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:11.885 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.885 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:11.885 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6826", 2022-06-29T16:08:11.886 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:08:11.886 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.886 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.886 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:11.886 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6827", 2022-06-29T16:08:11.887 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:08:11.887 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:11.887 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.887 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.888 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:08:11.888 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:11.888 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.888 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:11.888 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6830", 2022-06-29T16:08:11.889 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:08:11.889 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.889 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.889 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:11.889 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6831", 2022-06-29T16:08:11.890 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:08:11.890 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:11.890 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.890 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.891 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:08:11.891 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:11.891 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.891 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:11.891 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6828", 2022-06-29T16:08:11.892 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:08:11.892 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.892 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.892 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:11.892 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6829", 2022-06-29T16:08:11.893 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:08:11.893 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:11.893 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.893 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.893 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6825/12890", 2022-06-29T16:08:11.894 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6827/12890", 2022-06-29T16:08:11.894 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6831/12890", 2022-06-29T16:08:11.894 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6829/12890", 2022-06-29T16:08:11.894 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:08:11.895 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:08:11.895 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:08:11.895 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.895 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.895 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.896 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:08:11.896 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "ca43f136-6f69-4431-a157-fbe073fb585c", 2022-06-29T16:08:11.896 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:08:11.896 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:08:11.897 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:08:11.897 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:08:11.897 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:08:11.897 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:08:11.897 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:08:11.898 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 135, 2022-06-29T16:08:11.898 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:08:11.898 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:08:11.898 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:08:11.899 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:11.899 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.899 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:11.899 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6815", 2022-06-29T16:08:11.899 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:08:11.900 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.900 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.900 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:11.900 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6818", 2022-06-29T16:08:11.900 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:08:11.901 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:11.901 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.901 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.901 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:08:11.901 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:11.902 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.902 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:11.902 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6819", 2022-06-29T16:08:11.902 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:08:11.903 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.903 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.903 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:11.903 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6820", 2022-06-29T16:08:11.903 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:08:11.904 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:11.904 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.904 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.904 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:08:11.904 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:11.905 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.905 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:11.905 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6823", 2022-06-29T16:08:11.905 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:08:11.906 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.906 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.906 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:11.906 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6824", 2022-06-29T16:08:11.906 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:08:11.907 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:11.907 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.907 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.907 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:08:11.907 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:11.908 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.908 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:11.908 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6821", 2022-06-29T16:08:11.908 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:08:11.909 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.909 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.909 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:11.909 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6822", 2022-06-29T16:08:11.909 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:08:11.910 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:11.910 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.910 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.910 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6818/12892", 2022-06-29T16:08:11.910 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6820/12892", 2022-06-29T16:08:11.911 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6824/12892", 2022-06-29T16:08:11.911 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6822/12892", 2022-06-29T16:08:11.911 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:08:11.911 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:08:11.912 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:08:11.912 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.912 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.912 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.912 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:08:11.913 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "74fac45f-5f07-44ec-a9cb-2644c123fd4a", 2022-06-29T16:08:11.913 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:08:11.913 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:08:11.913 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:08:11.913 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:08:11.914 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:08:11.914 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:08:11.914 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:08:11.914 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 134, 2022-06-29T16:08:11.915 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:08:11.915 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:08:11.915 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:08:11.915 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:11.915 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.916 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:11.916 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6808", 2022-06-29T16:08:11.916 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:08:11.916 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.916 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.917 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:11.917 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6809", 2022-06-29T16:08:11.917 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:08:11.917 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:11.918 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.918 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.918 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:08:11.918 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:11.918 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.919 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:11.919 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6810", 2022-06-29T16:08:11.919 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:08:11.919 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.920 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.920 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:11.920 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6811", 2022-06-29T16:08:11.920 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:08:11.921 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:11.921 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.921 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.921 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:08:11.922 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:11.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.922 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:11.922 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6814", 2022-06-29T16:08:11.922 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:08:11.923 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.923 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.923 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:11.923 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6817", 2022-06-29T16:08:11.924 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:08:11.924 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:11.924 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.924 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.924 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:08:11.925 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:11.925 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.925 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:11.925 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6812", 2022-06-29T16:08:11.925 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:08:11.926 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.926 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.926 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:11.926 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6813", 2022-06-29T16:08:11.927 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:08:11.927 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:11.927 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.927 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.927 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6809/12891", 2022-06-29T16:08:11.928 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6811/12891", 2022-06-29T16:08:11.928 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6817/12891", 2022-06-29T16:08:11.928 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6813/12891", 2022-06-29T16:08:11.928 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:08:11.929 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:08:11.929 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:08:11.929 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.929 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.930 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:08:11.930 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "2581d351-973a-4b4e-85ca-3658f0068765", 2022-06-29T16:08:11.930 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:08:11.930 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:08:11.930 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:08:11.931 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:08:11.931 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:08:11.931 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:08:11.931 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:08:11.932 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 125, 2022-06-29T16:08:11.932 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:08:11.932 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:08:11.932 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:08:11.932 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:11.933 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.933 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:11.933 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6800", 2022-06-29T16:08:11.933 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:08:11.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.934 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.934 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:11.934 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6801", 2022-06-29T16:08:11.934 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:08:11.935 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:11.935 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.935 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:08:11.935 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:11.936 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.936 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:11.936 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6802", 2022-06-29T16:08:11.936 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:08:11.936 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.937 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.937 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:11.937 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6803", 2022-06-29T16:08:11.937 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:08:11.938 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:11.938 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.938 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.938 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:08:11.938 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:11.939 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.939 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:11.939 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6806", 2022-06-29T16:08:11.939 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:08:11.940 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.940 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.940 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:11.940 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6807", 2022-06-29T16:08:11.940 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:08:11.941 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:11.941 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.941 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.941 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:08:11.941 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:11.942 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.942 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:11.942 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6804", 2022-06-29T16:08:11.942 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:08:11.943 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.943 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.943 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:11.943 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6805", 2022-06-29T16:08:11.943 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:08:11.944 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:11.944 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.944 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.944 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6801/12889", 2022-06-29T16:08:11.945 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6803/12889", 2022-06-29T16:08:11.945 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6807/12889", 2022-06-29T16:08:11.945 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6805/12889", 2022-06-29T16:08:11.945 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:08:11.945 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:08:11.946 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:08:11.946 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.946 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.946 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.946 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:08:11.947 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "3d234db9-87cd-4fe9-814f-0240a287d8b4", 2022-06-29T16:08:11.947 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:08:11.947 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:08:11.947 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:08:11.948 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:08:11.948 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:08:11.948 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:08:11.948 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:08:11.948 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 134, 2022-06-29T16:08:11.949 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:08:11.949 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:08:11.949 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:08:11.949 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:11.949 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.950 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:11.950 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6808", 2022-06-29T16:08:11.950 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:08:11.950 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.951 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.951 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:11.951 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6810", 2022-06-29T16:08:11.951 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:08:11.951 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:11.952 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.952 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.952 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:08:11.952 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:11.952 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.953 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:11.953 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6812", 2022-06-29T16:08:11.953 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:08:11.953 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.954 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.954 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:11.954 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6814", 2022-06-29T16:08:11.954 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:08:11.954 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:11.955 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.955 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.955 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:08:11.955 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:11.955 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.956 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:11.956 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6823", 2022-06-29T16:08:11.956 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:08:11.956 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.957 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.957 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:11.957 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6825", 2022-06-29T16:08:11.957 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:08:11.957 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:11.958 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.958 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.958 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:08:11.958 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:11.959 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.959 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:11.959 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6817", 2022-06-29T16:08:11.959 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:08:11.959 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.960 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.960 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:11.960 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6820", 2022-06-29T16:08:11.960 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:08:11.960 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:11.961 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.961 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.961 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6810/13249", 2022-06-29T16:08:11.961 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6814/13249", 2022-06-29T16:08:11.961 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6825/13249", 2022-06-29T16:08:11.962 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6820/13249", 2022-06-29T16:08:11.962 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:08:11.962 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:08:11.962 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:08:11.963 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.963 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:08:11.963 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "e4bf1104-ff5c-438b-b608-8cee31c163f6", 2022-06-29T16:08:11.964 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:08:11.964 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:08:11.964 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:08:11.964 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:08:11.965 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:08:11.965 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:08:11.965 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:08:11.965 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 135, 2022-06-29T16:08:11.965 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:08:11.966 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:08:11.966 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:08:11.966 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:11.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.966 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:11.967 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6809", 2022-06-29T16:08:11.967 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:08:11.967 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.968 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:11.968 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6811", 2022-06-29T16:08:11.968 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:08:11.968 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:11.968 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.969 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:08:11.969 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:11.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.969 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:11.970 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6813", 2022-06-29T16:08:11.970 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:08:11.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.971 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:11.971 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6815", 2022-06-29T16:08:11.971 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:08:11.971 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:11.971 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.972 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:08:11.972 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:11.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.972 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:11.973 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6824", 2022-06-29T16:08:11.973 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:08:11.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.974 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:11.974 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6827", 2022-06-29T16:08:11.974 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:08:11.974 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:11.974 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.975 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:08:11.975 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:11.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.975 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:11.976 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6818", 2022-06-29T16:08:11.976 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:08:11.976 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.976 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.977 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:11.977 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6821", 2022-06-29T16:08:11.977 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:08:11.977 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:11.977 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.978 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6811/13248", 2022-06-29T16:08:11.979 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6815/13248", 2022-06-29T16:08:11.979 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6827/13248", 2022-06-29T16:08:11.979 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6821/13248", 2022-06-29T16:08:11.979 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:08:11.980 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:08:11.980 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:08:11.980 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.980 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.981 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:08:11.981 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "8564dd25-27ab-4203-b54e-af6d3aa05857", 2022-06-29T16:08:11.981 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:08:11.981 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:08:11.982 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:08:11.982 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:08:11.982 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:08:11.982 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:08:11.983 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:08:11.983 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 135, 2022-06-29T16:08:11.983 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:08:11.983 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:08:11.984 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:08:11.984 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:11.984 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.984 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:11.984 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6816", 2022-06-29T16:08:11.985 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:08:11.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.985 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:11.986 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6819", 2022-06-29T16:08:11.986 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:08:11.986 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:11.986 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.987 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:08:11.987 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:11.988 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.988 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:11.988 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6822", 2022-06-29T16:08:11.988 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:08:11.988 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.989 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:11.989 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6826", 2022-06-29T16:08:11.989 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:08:11.990 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:11.990 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.990 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:08:11.990 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:11.991 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.991 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:11.991 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6830", 2022-06-29T16:08:11.991 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:08:11.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.992 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.992 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:11.992 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6831", 2022-06-29T16:08:11.992 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:08:11.993 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:11.993 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.993 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.993 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:08:11.993 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:11.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.994 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:11.994 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6828", 2022-06-29T16:08:11.994 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:08:11.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.995 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.995 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:11.995 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6829", 2022-06-29T16:08:11.995 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:08:11.996 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:11.996 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.996 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6819/13250", 2022-06-29T16:08:11.996 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6826/13250", 2022-06-29T16:08:11.997 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6831/13250", 2022-06-29T16:08:11.997 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6829/13250", 2022-06-29T16:08:11.997 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:08:11.997 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:08:11.997 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:08:11.998 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:11.998 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:11.998 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:11.998 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:08:11.998 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "bd513a9c-3b19-4314-9897-d300b5c897bb", 2022-06-29T16:08:11.999 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:08:11.999 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:08:11.999 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:08:11.999 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:08:12.000 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:08:12.000 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:08:12.000 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:08:12.000 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 135, 2022-06-29T16:08:12.000 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:08:12.001 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:08:12.001 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:08:12.001 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:12.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:12.002 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:12.002 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6800", 2022-06-29T16:08:12.002 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:08:12.002 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:12.002 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:12.003 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:12.003 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6801", 2022-06-29T16:08:12.003 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:08:12.003 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:12.004 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:12.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:12.004 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:08:12.004 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:12.004 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:12.005 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:12.005 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6802", 2022-06-29T16:08:12.005 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:08:12.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:12.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:12.006 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:12.006 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6803", 2022-06-29T16:08:12.006 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:08:12.006 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:12.007 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:12.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:12.007 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:08:12.007 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:12.007 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:12.008 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:12.008 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6806", 2022-06-29T16:08:12.008 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:08:12.008 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:12.009 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:12.009 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:12.009 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6807", 2022-06-29T16:08:12.009 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:08:12.009 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:12.010 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:12.010 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:12.010 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:08:12.010 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:12.010 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:12.011 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:12.011 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6804", 2022-06-29T16:08:12.011 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:08:12.011 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:12.012 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:12.012 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:12.012 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6805", 2022-06-29T16:08:12.012 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:08:12.012 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:12.013 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:12.013 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:12.013 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6801/13247", 2022-06-29T16:08:12.013 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6803/13247", 2022-06-29T16:08:12.014 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6807/13247", 2022-06-29T16:08:12.014 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6805/13247", 2022-06-29T16:08:12.014 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:08:12.014 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:08:12.015 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:08:12.015 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:12.015 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:12.015 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:08:12.016 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_xinfo": [ 2022-06-29T16:08:12.016 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:12.016 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:08:12.016 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:08:12.017 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:08:12.017 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:08:12.017 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:08:12.017 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:08:12.017 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.891342+0000", 2022-06-29T16:08:12.018 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:08:12.018 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:12.018 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:12.018 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:08:12.019 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:08:12.019 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:08:12.019 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:08:12.019 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:08:12.019 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:08:12.020 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.900866+0000", 2022-06-29T16:08:12.020 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:08:12.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:12.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:12.021 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:08:12.021 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:08:12.021 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:08:12.021 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:08:12.022 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:08:12.022 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:08:12.022 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.912755+0000", 2022-06-29T16:08:12.022 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:08:12.023 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:12.023 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:12.023 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:08:12.023 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:08:12.023 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:08:12.024 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:08:12.024 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:08:12.024 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:08:12.024 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.864332+0000", 2022-06-29T16:08:12.025 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:08:12.025 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:12.025 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:12.025 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:08:12.025 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:08:12.026 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:08:12.026 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:08:12.026 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:08:12.026 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:08:12.027 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.863756+0000", 2022-06-29T16:08:12.027 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:08:12.027 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:12.027 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:12.027 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:08:12.028 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:08:12.028 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:08:12.028 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:08:12.028 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:08:12.028 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:08:12.029 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.869041+0000", 2022-06-29T16:08:12.029 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:08:12.029 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:12.029 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:12.030 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:08:12.030 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:08:12.030 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:08:12.031 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:08:12.031 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:08:12.032 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:08:12.032 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.895043+0000", 2022-06-29T16:08:12.032 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:08:12.033 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:12.033 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:12.033 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:08:12.034 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:08:12.034 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:08:12.035 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:08:12.035 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:08:12.036 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:08:12.036 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.889491+0000", 2022-06-29T16:08:12.036 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:08:12.037 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:12.037 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:08:12.038 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap": [], 2022-06-29T16:08:12.038 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap_items": [], 2022-06-29T16:08:12.038 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_temp": [], 2022-06-29T16:08:12.039 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_temp": [], 2022-06-29T16:08:12.039 INFO:teuthology.orchestra.run.smithi060.stdout: "blacklist": { 2022-06-29T16:08:12.040 INFO:teuthology.orchestra.run.smithi060.stdout: "172.21.15.60:6833/2222627281": "2022-06-30T16:08:05.074729+0000", 2022-06-29T16:08:12.040 INFO:teuthology.orchestra.run.smithi060.stdout: "172.21.15.60:6832/2222627281": "2022-06-30T16:08:05.074729+0000" 2022-06-29T16:08:12.040 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:12.040 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profiles": { 2022-06-29T16:08:12.040 INFO:teuthology.orchestra.run.smithi060.stdout: "default": { 2022-06-29T16:08:12.041 INFO:teuthology.orchestra.run.smithi060.stdout: "crush-failure-domain": "osd", 2022-06-29T16:08:12.041 INFO:teuthology.orchestra.run.smithi060.stdout: "k": "2", 2022-06-29T16:08:12.041 INFO:teuthology.orchestra.run.smithi060.stdout: "m": "1", 2022-06-29T16:08:12.042 INFO:teuthology.orchestra.run.smithi060.stdout: "plugin": "jerasure", 2022-06-29T16:08:12.042 INFO:teuthology.orchestra.run.smithi060.stdout: "ruleset-failure-domain": "osd", 2022-06-29T16:08:12.042 INFO:teuthology.orchestra.run.smithi060.stdout: "technique": "reed_sol_van" 2022-06-29T16:08:12.042 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:12.043 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:12.043 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps_queue": [], 2022-06-29T16:08:12.043 INFO:teuthology.orchestra.run.smithi060.stdout: "new_removed_snaps": [], 2022-06-29T16:08:12.043 INFO:teuthology.orchestra.run.smithi060.stdout: "new_purged_snaps": [], 2022-06-29T16:08:12.044 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_node_flags": {}, 2022-06-29T16:08:12.044 INFO:teuthology.orchestra.run.smithi060.stdout: "device_class_flags": {} 2022-06-29T16:08:12.044 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:08:12.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.829+0000 7fca487f7700 1 -- 172.21.15.60:0/707933705 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fca24053a70 msgr2=0x7fca24055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:12.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.829+0000 7fca487f7700 1 --2- 172.21.15.60:0/707933705 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fca24053a70 0x7fca24055f30 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 rx=0x7fca3000b000 tx=0x7fca300091d0).stop 2022-06-29T16:08:12.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.833+0000 7fca487f7700 1 -- 172.21.15.60:0/707933705 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fca4005bd20 msgr2=0x7fca40116b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:12.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.833+0000 7fca487f7700 1 --2- 172.21.15.60:0/707933705 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fca4005bd20 0x7fca40116b40 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 rx=0x7fca28002f70 tx=0x7fca28015000).stop 2022-06-29T16:08:12.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.833+0000 7fca487f7700 1 -- 172.21.15.60:0/707933705 shutdown_connections 2022-06-29T16:08:12.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.833+0000 7fca487f7700 1 --2- 172.21.15.60:0/707933705 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fca24053a70 0x7fca24055f30 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:12.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.833+0000 7fca487f7700 1 --2- 172.21.15.60:0/707933705 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca400b3f80 0x7fca4015a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:12.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.833+0000 7fca487f7700 1 --2- 172.21.15.60:0/707933705 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fca40063ad0 0x7fca40159c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:12.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.833+0000 7fca487f7700 1 --2- 172.21.15.60:0/707933705 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fca4005bd20 0x7fca40116b40 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:12.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.833+0000 7fca487f7700 1 -- 172.21.15.60:0/707933705 >> 172.21.15.60:0/707933705 conn(0x7fca400632c0 msgr2=0x7fca4014a330 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:12.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.833+0000 7fca487f7700 1 -- 172.21.15.60:0/707933705 shutdown_connections 2022-06-29T16:08:12.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:11.833+0000 7fca487f7700 1 -- 172.21.15.60:0/707933705 wait complete. 2022-06-29T16:08:12.049 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:08:12.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.157+0000 7fc2c4871700 1 Processor -- start 2022-06-29T16:08:12.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.157+0000 7fc2c4871700 1 -- start start 2022-06-29T16:08:12.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.157+0000 7fc2c4871700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc2c00b28e0 0x7fc2c00b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:12.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.161+0000 7fc2c4871700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c013fa40 con 0x7fc2c0065450 2022-06-29T16:08:12.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.161+0000 7fc2c4871700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c013fce0 con 0x7fc2c0060760 2022-06-29T16:08:12.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.161+0000 7fc2c4871700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2c0140860 con 0x7fc2c00b28e0 2022-06-29T16:08:12.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.161+0000 7fc2bf7fe700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc2c00b28e0 0x7fc2c00b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:12.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.161+0000 7fc2beffd700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc2c0065450 0x7fc2c0139340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34002/0 (socket says 172.21.15.60:34002) 2022-06-29T16:08:12.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.161+0000 7fc2beffd700 1 -- 172.21.15.60:0/2435923693 learned_addr learned my addr 172.21.15.60:0/2435923693 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:12.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.161+0000 7fc2be7fc700 1 -- 172.21.15.60:0/2435923693 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870196866 0 0) 0x7fc2c013fa40 con 0x7fc2c0065450 2022-06-29T16:08:12.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.161+0000 7fc2be7fc700 1 -- 172.21.15.60:0/2435923693 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2a0003260 con 0x7fc2c0065450 2022-06-29T16:08:12.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.161+0000 7fc2be7fc700 1 -- 172.21.15.60:0/2435923693 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520002037 0 0) 0x7fc2c013fce0 con 0x7fc2c0060760 2022-06-29T16:08:12.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.161+0000 7fc2be7fc700 1 -- 172.21.15.60:0/2435923693 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2c013fa40 con 0x7fc2c0060760 2022-06-29T16:08:12.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.161+0000 7fc2bf7fe700 1 -- 172.21.15.60:0/2435923693 >> v1:172.21.15.92:6789/0 conn(0x7fc2c0060760 legacy=0x7fc2c0066390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:12.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.161+0000 7fc2bf7fe700 1 -- 172.21.15.60:0/2435923693 >> v1:172.21.15.60:6789/0 conn(0x7fc2c0065450 legacy=0x7fc2c0139340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:12.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.161+0000 7fc2be7fc700 1 -- 172.21.15.60:0/2435923693 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3864204169 0 0) 0x7fc2a0003260 con 0x7fc2c0065450 2022-06-29T16:08:12.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.161+0000 7fc2bf7fe700 1 -- 172.21.15.60:0/2435923693 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c013f6f0 con 0x7fc2c00b28e0 2022-06-29T16:08:12.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.161+0000 7fc2bf7fe700 1 --2- 172.21.15.60:0/2435923693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c00b28e0 0x7fc2c00b2cc0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 rx=0x7fc2a8001110 tx=0x7fc2b0006df0).ready entity=mon.0 client_cookie=705d0300136d9980 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:12.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.161+0000 7fc2be7fc700 1 -- 172.21.15.60:0/2435923693 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2a8001660 con 0x7fc2c00b28e0 2022-06-29T16:08:12.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.161+0000 7fc2be7fc700 1 -- 172.21.15.60:0/2435923693 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b0000e90 con 0x7fc2c00b28e0 2022-06-29T16:08:12.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.161+0000 7fc2be7fc700 1 -- 172.21.15.60:0/2435923693 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2b0012460 con 0x7fc2c00b28e0 2022-06-29T16:08:12.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.161+0000 7fc2c4871700 1 -- 172.21.15.60:0/2435923693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c00b28e0 msgr2=0x7fc2c00b2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:12.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.161+0000 7fc2c4871700 1 --2- 172.21.15.60:0/2435923693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c00b28e0 0x7fc2c00b2cc0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 rx=0x7fc2a8001110 tx=0x7fc2b0006df0).stop 2022-06-29T16:08:12.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.161+0000 7fc2c4871700 1 -- 172.21.15.60:0/2435923693 shutdown_connections 2022-06-29T16:08:12.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.161+0000 7fc2c4871700 1 --2- 172.21.15.60:0/2435923693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c00b28e0 0x7fc2c00b2cc0 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:12.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.161+0000 7fc2c4871700 1 -- 172.21.15.60:0/2435923693 >> 172.21.15.60:0/2435923693 conn(0x7fc2c0061c80 msgr2=0x7fc2c00637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:12.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.161+0000 7fc2c4871700 1 -- 172.21.15.60:0/2435923693 shutdown_connections 2022-06-29T16:08:12.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.161+0000 7fc2c4871700 1 -- 172.21.15.60:0/2435923693 wait complete. 2022-06-29T16:08:12.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.165+0000 7fc2c4871700 1 Processor -- start 2022-06-29T16:08:12.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.165+0000 7fc2c4871700 1 -- start start 2022-06-29T16:08:12.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.165+0000 7fc2c4871700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2c0060760 0x7fc2c0143210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:12.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.165+0000 7fc2c4871700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2c0065450 0x7fc2c00b6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:12.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.165+0000 7fc2c4871700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c00b28e0 0x7fc2c0155600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:12.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.165+0000 7fc2c4871700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c0140d80 con 0x7fc2c00b28e0 2022-06-29T16:08:12.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.165+0000 7fc2c4871700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c0141020 con 0x7fc2c0060760 2022-06-29T16:08:12.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.165+0000 7fc2c4871700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c01412c0 con 0x7fc2c0065450 2022-06-29T16:08:12.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.165+0000 7fc2bffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c00b28e0 0x7fc2c0155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:12.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.165+0000 7fc2beffd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2c0065450 0x7fc2c00b6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:12.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.165+0000 7fc2bf7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2c0060760 0x7fc2c0143210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:12.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.165+0000 7fc2bffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c00b28e0 0x7fc2c0155600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54158/0 (socket says 172.21.15.60:54158) 2022-06-29T16:08:12.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.165+0000 7fc2bffff700 1 -- 172.21.15.60:0/1245801790 learned_addr learned my addr 172.21.15.60:0/1245801790 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:12.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.165+0000 7fc2bffff700 1 -- 172.21.15.60:0/1245801790 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2c0065450 msgr2=0x7fc2c00b6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:12.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.165+0000 7fc2bffff700 1 --2- 172.21.15.60:0/1245801790 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2c0065450 0x7fc2c00b6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:12.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.165+0000 7fc2bffff700 1 -- 172.21.15.60:0/1245801790 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2c0060760 msgr2=0x7fc2c0143210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:12.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.165+0000 7fc2bffff700 1 --2- 172.21.15.60:0/1245801790 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2c0060760 0x7fc2c0143210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:12.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.165+0000 7fc2bffff700 1 -- 172.21.15.60:0/1245801790 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b000b180 con 0x7fc2c00b28e0 2022-06-29T16:08:12.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.165+0000 7fc2bffff700 1 --2- 172.21.15.60:0/1245801790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c00b28e0 0x7fc2c0155600 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 rx=0x7fc2b8001490 tx=0x7fc2b8007dd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:12.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.165+0000 7fc2bcff9700 1 -- 172.21.15.60:0/1245801790 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2b8002310 con 0x7fc2c00b28e0 2022-06-29T16:08:12.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.165+0000 7fc2bcff9700 1 -- 172.21.15.60:0/1245801790 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b8002470 con 0x7fc2c00b28e0 2022-06-29T16:08:12.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.165+0000 7fc2c4871700 1 -- 172.21.15.60:0/1245801790 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2c013d990 con 0x7fc2c00b28e0 2022-06-29T16:08:12.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.165+0000 7fc2bcff9700 1 -- 172.21.15.60:0/1245801790 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2b80017a0 con 0x7fc2c00b28e0 2022-06-29T16:08:12.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.165+0000 7fc2c4871700 1 -- 172.21.15.60:0/1245801790 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2c0143a30 con 0x7fc2c00b28e0 2022-06-29T16:08:12.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.169+0000 7fc2c4871700 1 -- 172.21.15.60:0/1245801790 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2c00573d0 con 0x7fc2c00b28e0 2022-06-29T16:08:12.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.169+0000 7fc2bcff9700 1 -- 172.21.15.60:0/1245801790 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc2b8010030 con 0x7fc2c00b28e0 2022-06-29T16:08:12.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.169+0000 7fc2bcff9700 1 --2- 172.21.15.60:0/1245801790 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc2a4053a70 0x7fc2a4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:12.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.169+0000 7fc2bcff9700 1 -- 172.21.15.60:0/1245801790 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fc2b806c230 con 0x7fc2c00b28e0 2022-06-29T16:08:12.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.169+0000 7fc2bf7fe700 1 --2- 172.21.15.60:0/1245801790 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc2a4053a70 0x7fc2a4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:12.188 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.173+0000 7fc2bf7fe700 1 --2- 172.21.15.60:0/1245801790 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc2a4053a70 0x7fc2a4055f30 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 rx=0x7fc2b001a000 tx=0x7fc2b0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:12.189 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.173+0000 7fc2bcff9700 1 -- 172.21.15.60:0/1245801790 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc2b8043320 con 0x7fc2c00b28e0 2022-06-29T16:08:12.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.333+0000 7fc2c4871700 1 -- 172.21.15.60:0/1245801790 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2c0139830 con 0x7fc2c00b28e0 2022-06-29T16:08:12.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.333+0000 7fc2bcff9700 1 -- 172.21.15.60:0/1245801790 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+1203 (secure 0 0 0) 0x7fc2b8046940 con 0x7fc2c00b28e0 2022-06-29T16:08:12.339 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:08:12.339 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":71,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.92:6835/2628497783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2628497783},{"type":"v1","addr":"172.21.15.92:6835","nonce":2628497783}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":67},{"gid":8706,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.60:6833/82300070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":82300070},{"type":"v1","addr":"172.21.15.60:6833","nonce":82300070}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":71}],"filesystems":[]} 2022-06-29T16:08:12.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.337+0000 7fc2c4871700 1 -- 172.21.15.60:0/1245801790 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc2a4053a70 msgr2=0x7fc2a4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:12.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.337+0000 7fc2c4871700 1 --2- 172.21.15.60:0/1245801790 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc2a4053a70 0x7fc2a4055f30 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 rx=0x7fc2b001a000 tx=0x7fc2b0014000).stop 2022-06-29T16:08:12.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.337+0000 7fc2c4871700 1 -- 172.21.15.60:0/1245801790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c00b28e0 msgr2=0x7fc2c0155600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:12.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.337+0000 7fc2c4871700 1 --2- 172.21.15.60:0/1245801790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c00b28e0 0x7fc2c0155600 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 rx=0x7fc2b8001490 tx=0x7fc2b8007dd0).stop 2022-06-29T16:08:12.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.337+0000 7fc2c4871700 1 -- 172.21.15.60:0/1245801790 shutdown_connections 2022-06-29T16:08:12.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.337+0000 7fc2c4871700 1 --2- 172.21.15.60:0/1245801790 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc2a4053a70 0x7fc2a4055f30 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:12.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.337+0000 7fc2c4871700 1 --2- 172.21.15.60:0/1245801790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc2c00b28e0 0x7fc2c0155600 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:12.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.337+0000 7fc2c4871700 1 --2- 172.21.15.60:0/1245801790 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc2c0065450 0x7fc2c00b6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:12.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.337+0000 7fc2c4871700 1 --2- 172.21.15.60:0/1245801790 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2c0060760 0x7fc2c0143210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:12.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.337+0000 7fc2c4871700 1 -- 172.21.15.60:0/1245801790 >> 172.21.15.60:0/1245801790 conn(0x7fc2c0061c80 msgr2=0x7fc2c0069a60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:12.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.337+0000 7fc2c4871700 1 -- 172.21.15.60:0/1245801790 shutdown_connections 2022-06-29T16:08:12.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.337+0000 7fc2c4871700 1 -- 172.21.15.60:0/1245801790 wait complete. 2022-06-29T16:08:12.345 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 71 2022-06-29T16:08:12.352 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:08:12.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.445+0000 7fb80bfff700 1 Processor -- start 2022-06-29T16:08:12.450 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.445+0000 7fb80bfff700 1 -- start start 2022-06-29T16:08:12.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.445+0000 7fb80bfff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb80c0b5eb0 0x7fb80c142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:12.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.445+0000 7fb80bfff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb80c0b2c50 0x7fb80c0b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:12.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.445+0000 7fb80bfff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb80c0b0cf0 con 0x7fb80c0b48b0 2022-06-29T16:08:12.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.445+0000 7fb80bfff700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb80c0b54e0 con 0x7fb80c0b5eb0 2022-06-29T16:08:12.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.445+0000 7fb80bfff700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb80c0b4d10 con 0x7fb80c0b2c50 2022-06-29T16:08:12.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.445+0000 7fb80affd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb80c0b5eb0 0x7fb80c142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:12.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.445+0000 7fb80affd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fb80c0b5eb0 0x7fb80c142a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54162/0 (socket says 172.21.15.60:54162) 2022-06-29T16:08:12.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.445+0000 7fb80affd700 1 -- 172.21.15.60:0/3453970749 learned_addr learned my addr 172.21.15.60:0/3453970749 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:12.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.445+0000 7fb80b7fe700 1 --2- 172.21.15.60:0/3453970749 >> v2:172.21.15.92:3300/0 conn(0x7fb80c0b2c50 0x7fb80c0b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:12.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.445+0000 7fb80affd700 1 -- 172.21.15.60:0/3453970749 >> v2:172.21.15.92:3300/0 conn(0x7fb80c0b2c50 msgr2=0x7fb80c0b30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:12.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.445+0000 7fb80affd700 1 --2- 172.21.15.60:0/3453970749 >> v2:172.21.15.92:3300/0 conn(0x7fb80c0b2c50 0x7fb80c0b30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:12.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.445+0000 7fb80affd700 1 -- 172.21.15.60:0/3453970749 >> v1:172.21.15.92:6789/0 conn(0x7fb80c0b48b0 legacy=0x7fb80c0b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:12.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.445+0000 7fb80affd700 1 -- 172.21.15.60:0/3453970749 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb80c0b6290 con 0x7fb80c0b5eb0 2022-06-29T16:08:12.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.445+0000 7fb80affd700 1 --2- 172.21.15.60:0/3453970749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb80c0b5eb0 0x7fb80c142a20 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 rx=0x7fb80c062f00 tx=0x7fb804009290).ready entity=mon.0 client_cookie=4890e2151d936a1a server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:12.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.449+0000 7fb809ffb700 1 -- 172.21.15.60:0/3453970749 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb80400b830 con 0x7fb80c0b5eb0 2022-06-29T16:08:12.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.449+0000 7fb809ffb700 1 -- 172.21.15.60:0/3453970749 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb80400b990 con 0x7fb80c0b5eb0 2022-06-29T16:08:12.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.449+0000 7fb809ffb700 1 -- 172.21.15.60:0/3453970749 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb80400bb40 con 0x7fb80c0b5eb0 2022-06-29T16:08:12.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.449+0000 7fb80bfff700 1 -- 172.21.15.60:0/3453970749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb80c0b5eb0 msgr2=0x7fb80c142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:12.461 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.449+0000 7fb80bfff700 1 --2- 172.21.15.60:0/3453970749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb80c0b5eb0 0x7fb80c142a20 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 rx=0x7fb80c062f00 tx=0x7fb804009290).stop 2022-06-29T16:08:12.461 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.449+0000 7fb80bfff700 1 -- 172.21.15.60:0/3453970749 shutdown_connections 2022-06-29T16:08:12.461 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.449+0000 7fb80bfff700 1 --2- 172.21.15.60:0/3453970749 >> v2:172.21.15.92:3300/0 conn(0x7fb80c0b2c50 0x7fb80c0b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:12.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.449+0000 7fb80bfff700 1 --2- 172.21.15.60:0/3453970749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb80c0b5eb0 0x7fb80c142a20 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:12.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.449+0000 7fb80bfff700 1 -- 172.21.15.60:0/3453970749 >> 172.21.15.60:0/3453970749 conn(0x7fb80c062090 msgr2=0x7fb80c062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:12.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.449+0000 7fb80bfff700 1 -- 172.21.15.60:0/3453970749 shutdown_connections 2022-06-29T16:08:12.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.449+0000 7fb80bfff700 1 -- 172.21.15.60:0/3453970749 wait complete. 2022-06-29T16:08:12.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.449+0000 7fb80bfff700 1 Processor -- start 2022-06-29T16:08:12.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.453+0000 7fb80bfff700 1 -- start start 2022-06-29T16:08:12.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.453+0000 7fb80bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb80c0b2c50 0x7fb80c155a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:12.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.453+0000 7fb80bfff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb80c0b48b0 0x7fb80c143400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:12.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.453+0000 7fb80bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb80c0b5eb0 0x7fb80c1455e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:12.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.453+0000 7fb80bfff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80c112bd0 con 0x7fb80c0b5eb0 2022-06-29T16:08:12.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.453+0000 7fb80bfff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80c146a10 con 0x7fb80c0b2c50 2022-06-29T16:08:12.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.453+0000 7fb80bfff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb80c146ca0 con 0x7fb80c0b48b0 2022-06-29T16:08:12.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.453+0000 7fb80b7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb80c0b5eb0 0x7fb80c1455e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:12.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.453+0000 7fb80b7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb80c0b5eb0 0x7fb80c1455e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54172/0 (socket says 172.21.15.60:54172) 2022-06-29T16:08:12.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.453+0000 7fb80b7fe700 1 -- 172.21.15.60:0/823665794 learned_addr learned my addr 172.21.15.60:0/823665794 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:12.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.453+0000 7fb80a7fc700 1 --2- 172.21.15.60:0/823665794 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb80c0b48b0 0x7fb80c143400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:12.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.453+0000 7fb80affd700 1 --2- 172.21.15.60:0/823665794 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb80c0b2c50 0x7fb80c155a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:12.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.453+0000 7fb80b7fe700 1 -- 172.21.15.60:0/823665794 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb80c0b48b0 msgr2=0x7fb80c143400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:12.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.453+0000 7fb80b7fe700 1 --2- 172.21.15.60:0/823665794 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb80c0b48b0 0x7fb80c143400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:12.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.453+0000 7fb80b7fe700 1 -- 172.21.15.60:0/823665794 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb80c0b2c50 msgr2=0x7fb80c155a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:12.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.453+0000 7fb80b7fe700 1 --2- 172.21.15.60:0/823665794 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb80c0b2c50 0x7fb80c155a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:12.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.453+0000 7fb80b7fe700 1 -- 172.21.15.60:0/823665794 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb804009530 con 0x7fb80c0b5eb0 2022-06-29T16:08:12.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.453+0000 7fb80b7fe700 1 --2- 172.21.15.60:0/823665794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb80c0b5eb0 0x7fb80c1455e0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0x7fb7fc00a000 tx=0x7fb7fc008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:12.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.453+0000 7fb803fff700 1 -- 172.21.15.60:0/823665794 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb7fc011780 con 0x7fb80c0b5eb0 2022-06-29T16:08:12.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.453+0000 7fb803fff700 1 -- 172.21.15.60:0/823665794 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7fc0118e0 con 0x7fb80c0b5eb0 2022-06-29T16:08:12.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.453+0000 7fb80bfff700 1 -- 172.21.15.60:0/823665794 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb80c145060 con 0x7fb80c0b5eb0 2022-06-29T16:08:12.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.453+0000 7fb803fff700 1 -- 172.21.15.60:0/823665794 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb7fc011c40 con 0x7fb80c0b5eb0 2022-06-29T16:08:12.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.453+0000 7fb80bfff700 1 -- 172.21.15.60:0/823665794 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb80c1140b0 con 0x7fb80c0b5eb0 2022-06-29T16:08:12.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.457+0000 7fb80bfff700 1 -- 172.21.15.60:0/823665794 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb80c146e60 con 0x7fb80c0b5eb0 2022-06-29T16:08:12.469 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.457+0000 7fb803fff700 1 -- 172.21.15.60:0/823665794 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb7fc012420 con 0x7fb80c0b5eb0 2022-06-29T16:08:12.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.457+0000 7fb803fff700 1 --2- 172.21.15.60:0/823665794 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb7f0053a70 0x7fb7f0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:12.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.457+0000 7fb803fff700 1 -- 172.21.15.60:0/823665794 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fb7fc06b960 con 0x7fb80c0b5eb0 2022-06-29T16:08:12.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.457+0000 7fb80affd700 1 --2- 172.21.15.60:0/823665794 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb7f0053a70 0x7fb7f0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:12.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.457+0000 7fb80affd700 1 --2- 172.21.15.60:0/823665794 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb7f0053a70 0x7fb7f0055f30 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0x7fb80401b000 tx=0x7fb804016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:12.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.461+0000 7fb803fff700 1 -- 172.21.15.60:0/823665794 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb7fc042b00 con 0x7fb80c0b5eb0 2022-06-29T16:08:12.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.621+0000 7fb80bfff700 1 -- 172.21.15.60:0/823665794 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb80c0573d0 con 0x7fb80c0b5eb0 2022-06-29T16:08:12.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.625+0000 7fb803fff700 1 -- 172.21.15.60:0/823665794 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+1203 (secure 0 0 0) 0x7fb7fc046120 con 0x7fb80c0b5eb0 2022-06-29T16:08:12.629 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:08:12.630 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":71,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.92:6835/2628497783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2628497783},{"type":"v1","addr":"172.21.15.92:6835","nonce":2628497783}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":67},{"gid":8706,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.60:6833/82300070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":82300070},{"type":"v1","addr":"172.21.15.60:6833","nonce":82300070}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":71}],"filesystems":[]} 2022-06-29T16:08:12.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.625+0000 7fb80bfff700 1 -- 172.21.15.60:0/823665794 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb7f0053a70 msgr2=0x7fb7f0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:12.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.625+0000 7fb80bfff700 1 --2- 172.21.15.60:0/823665794 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb7f0053a70 0x7fb7f0055f30 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0x7fb80401b000 tx=0x7fb804016000).stop 2022-06-29T16:08:12.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.625+0000 7fb80bfff700 1 -- 172.21.15.60:0/823665794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb80c0b5eb0 msgr2=0x7fb80c1455e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:12.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.625+0000 7fb80bfff700 1 --2- 172.21.15.60:0/823665794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb80c0b5eb0 0x7fb80c1455e0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0x7fb7fc00a000 tx=0x7fb7fc008040).stop 2022-06-29T16:08:12.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.625+0000 7fb80bfff700 1 -- 172.21.15.60:0/823665794 shutdown_connections 2022-06-29T16:08:12.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.625+0000 7fb80bfff700 1 --2- 172.21.15.60:0/823665794 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb7f0053a70 0x7fb7f0055f30 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:12.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.625+0000 7fb80bfff700 1 --2- 172.21.15.60:0/823665794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb80c0b5eb0 0x7fb80c1455e0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:12.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.625+0000 7fb80bfff700 1 --2- 172.21.15.60:0/823665794 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb80c0b48b0 0x7fb80c143400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:12.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.625+0000 7fb80bfff700 1 --2- 172.21.15.60:0/823665794 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb80c0b2c50 0x7fb80c155a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:12.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.625+0000 7fb80bfff700 1 -- 172.21.15.60:0/823665794 >> 172.21.15.60:0/823665794 conn(0x7fb80c062090 msgr2=0x7fb80c0b7040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:12.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.625+0000 7fb80bfff700 1 -- 172.21.15.60:0/823665794 shutdown_connections 2022-06-29T16:08:12.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.625+0000 7fb80bfff700 1 -- 172.21.15.60:0/823665794 wait complete. 2022-06-29T16:08:12.635 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 71 2022-06-29T16:08:12.642 INFO:tasks.ceph.mds.a:Restarting daemon 2022-06-29T16:08:12.642 INFO:tasks.ceph.mds.a:Stopping old one... 2022-06-29T16:08:12.643 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-06-29T16:08:12.643 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T16:08:12.644 INFO:tasks.ceph.mds.b:Restarting daemon 2022-06-29T16:08:12.644 INFO:tasks.ceph.mds.b:Stopping old one... 2022-06-29T16:08:12.644 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-06-29T16:08:12.644 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T16:08:12.656 INFO:tasks.ceph.mds.a:Stopped 2022-06-29T16:08:12.656 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-06-29T16:08:12.659 INFO:tasks.ceph.mds.b:Stopped 2022-06-29T16:08:12.659 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-06-29T16:08:12.661 INFO:tasks.ceph.mds.a:Started 2022-06-29T16:08:12.661 DEBUG:teuthology.parallel:result is None 2022-06-29T16:08:12.704 INFO:tasks.ceph.mds.b:Started 2022-06-29T16:08:12.705 DEBUG:teuthology.parallel:result is None 2022-06-29T16:08:12.705 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-06-29T16:08:12.729 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2022-06-29T16:08:12.784 INFO:tasks.ceph.mds.b.smithi092.stdout:starting mds.b at 2022-06-29T16:08:12.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.805+0000 7f159db8d700 1 Processor -- start 2022-06-29T16:08:12.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.805+0000 7f159db8d700 1 -- start start 2022-06-29T16:08:12.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.805+0000 7f159db8d700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f15980acd60 0x7f1598144900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:12.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.805+0000 7f159db8d700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1598143cf0 con 0x7f15980657d0 2022-06-29T16:08:12.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.805+0000 7f159db8d700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15980ad140 con 0x7f15980ad990 2022-06-29T16:08:12.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.805+0000 7f159db8d700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1598064a70 con 0x7f15980acd60 2022-06-29T16:08:12.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.805+0000 7f159d38c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f15980657d0 0x7f15980b1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34044/0 (socket says 172.21.15.60:34044) 2022-06-29T16:08:12.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.805+0000 7f159d38c700 1 -- 172.21.15.60:0/1755218253 learned_addr learned my addr 172.21.15.60:0/1755218253 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:12.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.805+0000 7f159cb8b700 1 --2- 172.21.15.60:0/1755218253 >> v2:172.21.15.92:3300/0 conn(0x7f15980acd60 0x7f1598144900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:12.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.805+0000 7f15977fe700 1 -- 172.21.15.60:0/1755218253 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1867854540 0 0) 0x7f1598143cf0 con 0x7f15980657d0 2022-06-29T16:08:12.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.805+0000 7f15977fe700 1 -- 172.21.15.60:0/1755218253 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f157c003260 con 0x7f15980657d0 2022-06-29T16:08:12.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.805+0000 7f159cb8b700 1 -- 172.21.15.60:0/1755218253 >> v1:172.21.15.92:6789/0 conn(0x7f15980ad990 legacy=0x7f15981296f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:12.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.805+0000 7f159cb8b700 1 -- 172.21.15.60:0/1755218253 >> v1:172.21.15.60:6789/0 conn(0x7f15980657d0 legacy=0x7f15980b1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:12.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.805+0000 7f159cb8b700 1 -- 172.21.15.60:0/1755218253 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1598143f20 con 0x7f15980acd60 2022-06-29T16:08:12.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.805+0000 7f159cb8b700 1 --2- 172.21.15.60:0/1755218253 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15980acd60 0x7f1598144900 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0x7f1588002f70 tx=0x7f1588006df0).ready entity=mon.1 client_cookie=47a3533048c8ebf4 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:12.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.805+0000 7f15977fe700 1 -- 172.21.15.60:0/1755218253 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f158800d610 con 0x7f15980acd60 2022-06-29T16:08:12.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.805+0000 7f15977fe700 1 -- 172.21.15.60:0/1755218253 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1588000e90 con 0x7f15980acd60 2022-06-29T16:08:12.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.805+0000 7f15977fe700 1 -- 172.21.15.60:0/1755218253 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f158800d960 con 0x7f15980acd60 2022-06-29T16:08:12.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.809+0000 7f159db8d700 1 -- 172.21.15.60:0/1755218253 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15980acd60 msgr2=0x7f1598144900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:12.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.809+0000 7f159db8d700 1 --2- 172.21.15.60:0/1755218253 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15980acd60 0x7f1598144900 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0x7f1588002f70 tx=0x7f1588006df0).stop 2022-06-29T16:08:12.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.809+0000 7f159db8d700 1 -- 172.21.15.60:0/1755218253 shutdown_connections 2022-06-29T16:08:12.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.809+0000 7f159db8d700 1 --2- 172.21.15.60:0/1755218253 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15980acd60 0x7f1598144900 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:12.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.809+0000 7f159db8d700 1 -- 172.21.15.60:0/1755218253 >> 172.21.15.60:0/1755218253 conn(0x7f15980b6f80 msgr2=0x7f15980b7390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:12.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.809+0000 7f159db8d700 1 -- 172.21.15.60:0/1755218253 shutdown_connections 2022-06-29T16:08:12.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.809+0000 7f159db8d700 1 -- 172.21.15.60:0/1755218253 wait complete. 2022-06-29T16:08:12.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.809+0000 7f159db8d700 1 Processor -- start 2022-06-29T16:08:12.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.809+0000 7f159db8d700 1 -- start start 2022-06-29T16:08:12.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.809+0000 7f159db8d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15980657d0 0x7f1598112010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:12.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.809+0000 7f159db8d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15980acd60 0x7f15980af2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:12.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.809+0000 7f159db8d700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15980ad990 0x7f1598152f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:12.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.809+0000 7f159db8d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1598114570 con 0x7f15980acd60 2022-06-29T16:08:12.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.809+0000 7f159db8d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1598110e10 con 0x7f15980657d0 2022-06-29T16:08:12.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.809+0000 7f159db8d700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15981110b0 con 0x7f15980ad990 2022-06-29T16:08:12.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.809+0000 7f1597fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15980acd60 0x7f15980af2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:12.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.809+0000 7f1597fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15980acd60 0x7f15980af2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54194/0 (socket says 172.21.15.60:54194) 2022-06-29T16:08:12.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.809+0000 7f1597fff700 1 -- 172.21.15.60:0/3963791584 learned_addr learned my addr 172.21.15.60:0/3963791584 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:12.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.809+0000 7f159cb8b700 1 --2- 172.21.15.60:0/3963791584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15980657d0 0x7f1598112010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:12.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.809+0000 7f159d38c700 1 --2- 172.21.15.60:0/3963791584 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15980ad990 0x7f1598152f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:12.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.809+0000 7f1597fff700 1 -- 172.21.15.60:0/3963791584 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15980ad990 msgr2=0x7f1598152f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:12.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.809+0000 7f1597fff700 1 --2- 172.21.15.60:0/3963791584 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15980ad990 0x7f1598152f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:12.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.809+0000 7f1597fff700 1 -- 172.21.15.60:0/3963791584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15980657d0 msgr2=0x7f1598112010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:12.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.809+0000 7f1597fff700 1 --2- 172.21.15.60:0/3963791584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15980657d0 0x7f1598112010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:12.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.809+0000 7f1597fff700 1 -- 172.21.15.60:0/3963791584 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f158800b180 con 0x7f15980acd60 2022-06-29T16:08:12.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.809+0000 7f159d38c700 1 --2- 172.21.15.60:0/3963791584 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15980ad990 0x7f1598152f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:08:12.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.809+0000 7f1597fff700 1 --2- 172.21.15.60:0/3963791584 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15980acd60 0x7f15980af2e0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 rx=0x7f15980b7c60 tx=0x7f1580007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:12.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.809+0000 7f1595ffb700 1 -- 172.21.15.60:0/3963791584 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f158000ce30 con 0x7f15980acd60 2022-06-29T16:08:12.829 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.809+0000 7f159db8d700 1 -- 172.21.15.60:0/3963791584 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1598110a90 con 0x7f15980acd60 2022-06-29T16:08:12.829 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.809+0000 7f1595ffb700 1 -- 172.21.15.60:0/3963791584 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f158000f470 con 0x7f15980acd60 2022-06-29T16:08:12.829 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.809+0000 7f1595ffb700 1 -- 172.21.15.60:0/3963791584 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f158000f7d0 con 0x7f15980acd60 2022-06-29T16:08:12.829 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.809+0000 7f159db8d700 1 -- 172.21.15.60:0/3963791584 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15981149a0 con 0x7f15980acd60 2022-06-29T16:08:12.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.813+0000 7f159db8d700 1 -- 172.21.15.60:0/3963791584 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1574003410 con 0x7f15980acd60 2022-06-29T16:08:12.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.813+0000 7f1595ffb700 1 -- 172.21.15.60:0/3963791584 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1580019030 con 0x7f15980acd60 2022-06-29T16:08:12.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.813+0000 7f1595ffb700 1 --2- 172.21.15.60:0/3963791584 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1570053a10 0x7f1570055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:12.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.813+0000 7f1595ffb700 1 -- 172.21.15.60:0/3963791584 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f158006afe0 con 0x7f15980acd60 2022-06-29T16:08:12.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.813+0000 7f159cb8b700 1 --2- 172.21.15.60:0/3963791584 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1570053a10 0x7f1570055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:12.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.813+0000 7f159cb8b700 1 --2- 172.21.15.60:0/3963791584 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1570053a10 0x7f1570055ed0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 rx=0x7f1588018000 tx=0x7f1588014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:12.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.817+0000 7f1595ffb700 1 -- 172.21.15.60:0/3963791584 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1580042140 con 0x7f15980acd60 2022-06-29T16:08:12.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:12.957+0000 7f159db8d700 1 -- 172.21.15.60:0/3963791584 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f1574003c40 con 0x7f15980acd60 2022-06-29T16:08:13.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:13.965+0000 7f1595ffb700 1 -- 172.21.15.60:0/3963791584 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v141) v1 ==== 97+0+0 (secure 0 0 0) 0x7f1580045760 con 0x7f15980acd60 2022-06-29T16:08:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:13.969+0000 7f159db8d700 1 -- 172.21.15.60:0/3963791584 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1570053a10 msgr2=0x7f1570055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:13.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:13.969+0000 7f159db8d700 1 --2- 172.21.15.60:0/3963791584 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1570053a10 0x7f1570055ed0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 rx=0x7f1588018000 tx=0x7f1588014000).stop 2022-06-29T16:08:13.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:13.969+0000 7f159db8d700 1 -- 172.21.15.60:0/3963791584 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15980acd60 msgr2=0x7f15980af2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:13.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:13.969+0000 7f159db8d700 1 --2- 172.21.15.60:0/3963791584 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15980acd60 0x7f15980af2e0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 rx=0x7f15980b7c60 tx=0x7f1580007c50).stop 2022-06-29T16:08:13.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:13.969+0000 7f159db8d700 1 -- 172.21.15.60:0/3963791584 shutdown_connections 2022-06-29T16:08:13.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:13.969+0000 7f159db8d700 1 --2- 172.21.15.60:0/3963791584 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1570053a10 0x7f1570055ed0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:13.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:13.969+0000 7f159db8d700 1 --2- 172.21.15.60:0/3963791584 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15980ad990 0x7f1598152f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:13.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:13.969+0000 7f159db8d700 1 --2- 172.21.15.60:0/3963791584 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15980acd60 0x7f15980af2e0 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:13.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:13.969+0000 7f159db8d700 1 --2- 172.21.15.60:0/3963791584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15980657d0 0x7f1598112010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:13.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:13.969+0000 7f159db8d700 1 -- 172.21.15.60:0/3963791584 >> 172.21.15.60:0/3963791584 conn(0x7f15980b6f80 msgr2=0x7f1598067170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:13.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:13.969+0000 7f159db8d700 1 -- 172.21.15.60:0/3963791584 shutdown_connections 2022-06-29T16:08:13.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:13.969+0000 7f159db8d700 1 -- 172.21.15.60:0/3963791584 wait complete. 2022-06-29T16:08:13.980 INFO:teuthology.orchestra.run.smithi060.stderr: removed all blacklist entries 2022-06-29T16:08:13.991 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-06-29T16:08:13.992 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-06-29T16:08:14.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.137+0000 7f73ccef5700 1 Processor -- start 2022-06-29T16:08:14.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.137+0000 7f73ccef5700 1 -- start start 2022-06-29T16:08:14.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.137+0000 7f73ccef5700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f73c805d070 0x7f73c813c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:14.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.137+0000 7f73ccef5700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f73c80b2130 0x7f73c813caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:14.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.137+0000 7f73ccef5700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c805d450 con 0x7f73c80b0830 2022-06-29T16:08:14.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.137+0000 7f73ccef5700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73c8063ba0 con 0x7f73c805d070 2022-06-29T16:08:14.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.137+0000 7f73ccef5700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73c8063e40 con 0x7f73c80b2130 2022-06-29T16:08:14.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.137+0000 7f73c7fff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f73c80b0830 0x7f73c813fa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34058/0 (socket says 172.21.15.60:34058) 2022-06-29T16:08:14.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.137+0000 7f73c7fff700 1 -- 172.21.15.60:0/532163145 learned_addr learned my addr 172.21.15.60:0/532163145 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:14.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.137+0000 7f73c77fe700 1 --2- 172.21.15.60:0/532163145 >> v2:172.21.15.92:3300/0 conn(0x7f73c805d070 0x7f73c813c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:14.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.137+0000 7f73c6ffd700 1 --2- 172.21.15.60:0/532163145 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f73c80b2130 0x7f73c813caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:14.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.137+0000 7f73c67fc700 1 -- 172.21.15.60:0/532163145 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2034634306 0 0) 0x7f73c805d450 con 0x7f73c80b0830 2022-06-29T16:08:14.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.137+0000 7f73c67fc700 1 -- 172.21.15.60:0/532163145 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73a8003260 con 0x7f73c80b0830 2022-06-29T16:08:14.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.141+0000 7f73c6ffd700 1 -- 172.21.15.60:0/532163145 >> v2:172.21.15.92:3300/0 conn(0x7f73c805d070 msgr2=0x7f73c813c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:14.161 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.141+0000 7f73c6ffd700 1 --2- 172.21.15.60:0/532163145 >> v2:172.21.15.92:3300/0 conn(0x7f73c805d070 0x7f73c813c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:14.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.141+0000 7f73c6ffd700 1 -- 172.21.15.60:0/532163145 >> v1:172.21.15.60:6789/0 conn(0x7f73c80b0830 legacy=0x7f73c813fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:14.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.141+0000 7f73c6ffd700 1 -- 172.21.15.60:0/532163145 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c8067650 con 0x7f73c80b2130 2022-06-29T16:08:14.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.141+0000 7f73c77fe700 1 --2- 172.21.15.60:0/532163145 >> v2:172.21.15.92:3300/0 conn(0x7f73c805d070 0x7f73c813c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:08:14.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.141+0000 7f73c6ffd700 1 --2- 172.21.15.60:0/532163145 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f73c80b2130 0x7f73c813caa0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f73b0002f70 tx=0x7f73b0006df0).ready entity=mon.2 client_cookie=977164cacd5d6a91 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:14.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.141+0000 7f73c67fc700 1 -- 172.21.15.60:0/532163145 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73b000d4a0 con 0x7f73c80b2130 2022-06-29T16:08:14.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.141+0000 7f73c67fc700 1 -- 172.21.15.60:0/532163145 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b0000e90 con 0x7f73c80b2130 2022-06-29T16:08:14.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.141+0000 7f73c67fc700 1 -- 172.21.15.60:0/532163145 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73b000d7f0 con 0x7f73c80b2130 2022-06-29T16:08:14.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.141+0000 7f73ccef5700 1 -- 172.21.15.60:0/532163145 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f73c80b2130 msgr2=0x7f73c813caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:14.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.141+0000 7f73ccef5700 1 --2- 172.21.15.60:0/532163145 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f73c80b2130 0x7f73c813caa0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f73b0002f70 tx=0x7f73b0006df0).stop 2022-06-29T16:08:14.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.141+0000 7f73ccef5700 1 -- 172.21.15.60:0/532163145 shutdown_connections 2022-06-29T16:08:14.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.141+0000 7f73ccef5700 1 --2- 172.21.15.60:0/532163145 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f73c80b2130 0x7f73c813caa0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:14.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.141+0000 7f73ccef5700 1 --2- 172.21.15.60:0/532163145 >> v2:172.21.15.92:3300/0 conn(0x7f73c805d070 0x7f73c813c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:14.167 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.141+0000 7f73ccef5700 1 -- 172.21.15.60:0/532163145 >> 172.21.15.60:0/532163145 conn(0x7f73c80b5890 msgr2=0x7f73c80b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:14.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.141+0000 7f73ccef5700 1 -- 172.21.15.60:0/532163145 shutdown_connections 2022-06-29T16:08:14.168 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.141+0000 7f73ccef5700 1 -- 172.21.15.60:0/532163145 wait complete. 2022-06-29T16:08:14.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.141+0000 7f73ccef5700 1 Processor -- start 2022-06-29T16:08:14.169 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.141+0000 7f73ccef5700 1 -- start start 2022-06-29T16:08:14.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.145+0000 7f73ccef5700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f73c805d070 0x7f73c813a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:14.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.145+0000 7f73ccef5700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f73c80b0830 0x7f73c8064ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:14.170 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.145+0000 7f73ccef5700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73c80b2130 0x7f73c8064fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:14.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.145+0000 7f73ccef5700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c80ae610 con 0x7f73c80b2130 2022-06-29T16:08:14.171 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.145+0000 7f73ccef5700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c81391a0 con 0x7f73c805d070 2022-06-29T16:08:14.172 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.145+0000 7f73ccef5700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73c813ad30 con 0x7f73c80b0830 2022-06-29T16:08:14.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.145+0000 7f73c7fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73c80b2130 0x7f73c8064fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:14.173 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.145+0000 7f73c77fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f73c805d070 0x7f73c813a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:14.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.145+0000 7f73c7fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73c80b2130 0x7f73c8064fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54210/0 (socket says 172.21.15.60:54210) 2022-06-29T16:08:14.174 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.145+0000 7f73c7fff700 1 -- 172.21.15.60:0/3994363778 learned_addr learned my addr 172.21.15.60:0/3994363778 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:14.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.145+0000 7f73c6ffd700 1 --2- 172.21.15.60:0/3994363778 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f73c80b0830 0x7f73c8064ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:14.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.145+0000 7f73c7fff700 1 -- 172.21.15.60:0/3994363778 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f73c80b0830 msgr2=0x7f73c8064ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:14.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.145+0000 7f73c7fff700 1 --2- 172.21.15.60:0/3994363778 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f73c80b0830 0x7f73c8064ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:14.176 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.145+0000 7f73c7fff700 1 -- 172.21.15.60:0/3994363778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f73c805d070 msgr2=0x7f73c813a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:14.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.145+0000 7f73c7fff700 1 --2- 172.21.15.60:0/3994363778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f73c805d070 0x7f73c813a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:14.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.145+0000 7f73c7fff700 1 -- 172.21.15.60:0/3994363778 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b800b180 con 0x7f73c80b2130 2022-06-29T16:08:14.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.145+0000 7f73c7fff700 1 --2- 172.21.15.60:0/3994363778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73c80b2130 0x7f73c8064fc0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 rx=0x7f73c0001490 tx=0x7f73c0007e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:14.178 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.145+0000 7f73c4ff9700 1 -- 172.21.15.60:0/3994363778 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73c0005ba0 con 0x7f73c80b2130 2022-06-29T16:08:14.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.145+0000 7f73c4ff9700 1 -- 172.21.15.60:0/3994363778 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c0005d00 con 0x7f73c80b2130 2022-06-29T16:08:14.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.145+0000 7f73ccef5700 1 -- 172.21.15.60:0/3994363778 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73b000b180 con 0x7f73c80b2130 2022-06-29T16:08:14.179 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.145+0000 7f73c4ff9700 1 -- 172.21.15.60:0/3994363778 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73c00072c0 con 0x7f73c80b2130 2022-06-29T16:08:14.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.145+0000 7f73ccef5700 1 -- 172.21.15.60:0/3994363778 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73c8138ae0 con 0x7f73c80b2130 2022-06-29T16:08:14.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.149+0000 7f73ccef5700 1 -- 172.21.15.60:0/3994363778 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73c8138dc0 con 0x7f73c80b2130 2022-06-29T16:08:14.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.149+0000 7f73c4ff9700 1 -- 172.21.15.60:0/3994363778 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f73c0010070 con 0x7f73c80b2130 2022-06-29T16:08:14.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.149+0000 7f73c4ff9700 1 --2- 172.21.15.60:0/3994363778 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f73ac053aa0 0x7f73ac055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:14.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.149+0000 7f73c4ff9700 1 -- 172.21.15.60:0/3994363778 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f73c006b410 con 0x7f73c80b2130 2022-06-29T16:08:14.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.149+0000 7f73c77fe700 1 --2- 172.21.15.60:0/3994363778 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f73ac053aa0 0x7f73ac055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:14.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.149+0000 7f73c77fe700 1 --2- 172.21.15.60:0/3994363778 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f73ac053aa0 0x7f73ac055f60 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 rx=0x7f73b800b000 tx=0x7f73b8006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:14.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.157+0000 7f73c4ff9700 1 -- 172.21.15.60:0/3994363778 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f73c0042660 con 0x7f73c80b2130 2022-06-29T16:08:14.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.329+0000 7f73ccef5700 1 -- 172.21.15.60:0/3994363778 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f73c80573d0 con 0x7f73c80b2130 2022-06-29T16:08:14.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.329+0000 7f73c4ff9700 1 -- 172.21.15.60:0/3994363778 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v46) v1 ==== 80+0+4242 (secure 0 0 0) 0x7f73c0018070 con 0x7f73c80b2130 2022-06-29T16:08:14.336 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:08:14.336 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:08:14.337 INFO:teuthology.orchestra.run.smithi060.stdout: "auth_dump": [ 2022-06-29T16:08:14.337 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:14.338 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.a", 2022-06-29T16:08:14.339 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxivv7JNhAAnA291UxlB/SfgITOipP00A==", 2022-06-29T16:08:14.340 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:08:14.340 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:08:14.341 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:08:14.341 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:08:14.342 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:08:14.342 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:14.343 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:14.343 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:14.343 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mds.b", 2022-06-29T16:08:14.344 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxiRSr/OhAAfFQTF2YHvzCv5wXBVCA5WQ==", 2022-06-29T16:08:14.345 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:08:14.346 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:08:14.346 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:08:14.347 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:08:14.347 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:08:14.348 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:14.348 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:14.348 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:14.349 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.0", 2022-06-29T16:08:14.349 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiZYPxOBAAIkweWJlTJIXgxnlagkGdXA==", 2022-06-29T16:08:14.350 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:08:14.350 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:08:14.351 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:08:14.351 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:08:14.352 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:14.352 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:14.353 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:14.353 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.1", 2022-06-29T16:08:14.354 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxiS2JGBhAAJn0UfAxZI3IVFyD5QBdb/Q==", 2022-06-29T16:08:14.354 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:08:14.355 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:08:14.355 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:08:14.355 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:08:14.356 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:14.356 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:14.357 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:14.357 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.2", 2022-06-29T16:08:14.358 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxiPhkfEhAApH7r8OUJzW7fZNM6XVv1dg==", 2022-06-29T16:08:14.358 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:08:14.359 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:08:14.359 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:08:14.359 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:08:14.360 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:14.360 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:14.361 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:14.361 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.3", 2022-06-29T16:08:14.361 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCIdrxit/47HhAA1vyU6h7hjPG+op9qP/rVYQ==", 2022-06-29T16:08:14.362 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:08:14.362 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:08:14.363 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:08:14.363 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:08:14.363 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:14.364 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:14.364 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:14.364 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.4", 2022-06-29T16:08:14.365 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxiaKzKBxAAbIwCF9ag7ezsNy11hi0BEw==", 2022-06-29T16:08:14.366 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:08:14.366 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:08:14.367 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:08:14.367 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:08:14.368 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:14.369 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:14.369 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:14.369 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.5", 2022-06-29T16:08:14.370 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxiJQV/IBAASBBDb6UFgEtXILe4fcT/xA==", 2022-06-29T16:08:14.370 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:08:14.371 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:08:14.371 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:08:14.372 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:08:14.372 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:14.372 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:14.373 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:14.373 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.6", 2022-06-29T16:08:14.374 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCLdrxi6D07OBAA2FWiZ+mNK+jFMp/kPZnbQQ==", 2022-06-29T16:08:14.374 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:08:14.374 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:08:14.375 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:08:14.375 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:08:14.375 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:14.376 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:14.376 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:14.376 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "osd.7", 2022-06-29T16:08:14.377 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCMdrxinEOcEhAA562ybpzGHNQfWIq8q7zw9w==", 2022-06-29T16:08:14.377 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:08:14.378 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:08:14.378 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:08:14.379 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:08:14.379 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:14.379 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:14.380 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:14.380 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.0", 2022-06-29T16:08:14.381 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiHXPwBRAAkOOwlYJvIW7dqVFdlXXoqw==", 2022-06-29T16:08:14.381 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:08:14.382 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:08:14.382 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2022-06-29T16:08:14.383 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw pool=cephfs_data" 2022-06-29T16:08:14.383 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:14.383 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:14.384 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:14.384 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.1", 2022-06-29T16:08:14.385 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxibe5JCRAAGBJbi9t09w0836JqrAT7+w==", 2022-06-29T16:08:14.385 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:08:14.385 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:08:14.386 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2022-06-29T16:08:14.386 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw pool=cephfs_data" 2022-06-29T16:08:14.386 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:14.387 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:14.387 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:14.387 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.2", 2022-06-29T16:08:14.387 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCHdrxiK3dBDRAA7XINPSOPQDfOrUYEpHC+gw==", 2022-06-29T16:08:14.387 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:08:14.388 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow", 2022-06-29T16:08:14.388 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow r", 2022-06-29T16:08:14.388 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow rw pool=cephfs_data" 2022-06-29T16:08:14.388 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:14.389 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:14.389 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:14.389 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "client.admin", 2022-06-29T16:08:14.390 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxieKVcGBAAnjH80a0qWIArz2qwjYHWdQ==", 2022-06-29T16:08:14.390 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:08:14.390 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T16:08:14.391 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr": "allow *", 2022-06-29T16:08:14.391 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow *", 2022-06-29T16:08:14.391 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:08:14.391 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:14.392 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:14.392 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:14.392 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.x", 2022-06-29T16:08:14.392 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxi2LZJNBAA3MKb/pXYbQCsoRx8EQfGbw==", 2022-06-29T16:08:14.392 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:08:14.393 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T16:08:14.393 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2022-06-29T16:08:14.393 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:08:14.393 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:14.394 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:14.394 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:14.395 INFO:teuthology.orchestra.run.smithi060.stdout: "entity": "mgr.y", 2022-06-29T16:08:14.395 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "AQCGdrxi3LmaMBAAN+0mZrdKXqSXOipVTA1drw==", 2022-06-29T16:08:14.395 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": { 2022-06-29T16:08:14.396 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": "allow *", 2022-06-29T16:08:14.396 INFO:teuthology.orchestra.run.smithi060.stdout: "mon": "allow profile mgr", 2022-06-29T16:08:14.396 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "allow *" 2022-06-29T16:08:14.397 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:14.397 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:14.398 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:14.398 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:08:14.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.333+0000 7f73ccef5700 1 -- 172.21.15.60:0/3994363778 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f73ac053aa0 msgr2=0x7f73ac055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:14.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.333+0000 7f73ccef5700 1 --2- 172.21.15.60:0/3994363778 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f73ac053aa0 0x7f73ac055f60 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 rx=0x7f73b800b000 tx=0x7f73b8006df0).stop 2022-06-29T16:08:14.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.333+0000 7f73ccef5700 1 -- 172.21.15.60:0/3994363778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73c80b2130 msgr2=0x7f73c8064fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:14.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.333+0000 7f73ccef5700 1 --2- 172.21.15.60:0/3994363778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73c80b2130 0x7f73c8064fc0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 rx=0x7f73c0001490 tx=0x7f73c0007e00).stop 2022-06-29T16:08:14.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.333+0000 7f73ccef5700 1 -- 172.21.15.60:0/3994363778 shutdown_connections 2022-06-29T16:08:14.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.333+0000 7f73ccef5700 1 --2- 172.21.15.60:0/3994363778 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f73ac053aa0 0x7f73ac055f60 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:14.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.333+0000 7f73ccef5700 1 --2- 172.21.15.60:0/3994363778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73c80b2130 0x7f73c8064fc0 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:14.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.333+0000 7f73ccef5700 1 --2- 172.21.15.60:0/3994363778 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f73c80b0830 0x7f73c8064ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:14.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.333+0000 7f73ccef5700 1 --2- 172.21.15.60:0/3994363778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f73c805d070 0x7f73c813a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:14.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.333+0000 7f73ccef5700 1 -- 172.21.15.60:0/3994363778 >> 172.21.15.60:0/3994363778 conn(0x7f73c80b5890 msgr2=0x7f73c80b18a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:14.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.333+0000 7f73ccef5700 1 -- 172.21.15.60:0/3994363778 shutdown_connections 2022-06-29T16:08:14.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.333+0000 7f73ccef5700 1 -- 172.21.15.60:0/3994363778 wait complete. 2022-06-29T16:08:14.415 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:08:14.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.521+0000 7f50f5992700 1 Processor -- start 2022-06-29T16:08:14.525 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.521+0000 7f50f5992700 1 -- start start 2022-06-29T16:08:14.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.521+0000 7f50f5992700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f50f013f790 0x7f50f013bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:14.530 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.521+0000 7f50f5992700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50f0142820 con 0x7f50f013fc00 2022-06-29T16:08:14.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.521+0000 7f50f5992700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50f00b0c50 con 0x7f50f00b61d0 2022-06-29T16:08:14.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.521+0000 7f50f5992700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50f0062bb0 con 0x7f50f013f790 2022-06-29T16:08:14.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.521+0000 7f50ef7fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f50f013fc00 0x7f50f013c130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34072/0 (socket says 172.21.15.60:34072) 2022-06-29T16:08:14.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.521+0000 7f50ef7fe700 1 -- 172.21.15.60:0/3422731151 learned_addr learned my addr 172.21.15.60:0/3422731151 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:14.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.521+0000 7f50ee7fc700 1 --2- 172.21.15.60:0/3422731151 >> v2:172.21.15.92:3300/0 conn(0x7f50f013f790 0x7f50f013bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:14.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.521+0000 7f50edffb700 1 -- 172.21.15.60:0/3422731151 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285076960 0 0) 0x7f50f0142820 con 0x7f50f013fc00 2022-06-29T16:08:14.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.521+0000 7f50edffb700 1 -- 172.21.15.60:0/3422731151 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50d4004370 con 0x7f50f013fc00 2022-06-29T16:08:14.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.521+0000 7f50ee7fc700 1 -- 172.21.15.60:0/3422731151 >> v1:172.21.15.92:6789/0 conn(0x7f50f00b61d0 legacy=0x7f50f00b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:14.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.521+0000 7f50ee7fc700 1 -- 172.21.15.60:0/3422731151 >> v1:172.21.15.60:6789/0 conn(0x7f50f013fc00 legacy=0x7f50f013c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:14.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.521+0000 7f50ee7fc700 1 -- 172.21.15.60:0/3422731151 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50f00b65b0 con 0x7f50f013f790 2022-06-29T16:08:14.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.521+0000 7f50ee7fc700 1 --2- 172.21.15.60:0/3422731151 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50f013f790 0x7f50f013bc40 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0x7f50f00b0190 tx=0x7f50e0006df0).ready entity=mon.1 client_cookie=aed0b59fd54820c2 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:14.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.521+0000 7f50edffb700 1 -- 172.21.15.60:0/3422731151 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f50e0014070 con 0x7f50f013f790 2022-06-29T16:08:14.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.521+0000 7f50edffb700 1 -- 172.21.15.60:0/3422731151 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50e0000e90 con 0x7f50f013f790 2022-06-29T16:08:14.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.521+0000 7f50edffb700 1 -- 172.21.15.60:0/3422731151 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f50e000fc00 con 0x7f50f013f790 2022-06-29T16:08:14.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.521+0000 7f50f5992700 1 -- 172.21.15.60:0/3422731151 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50f013f790 msgr2=0x7f50f013bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:14.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.521+0000 7f50f5992700 1 --2- 172.21.15.60:0/3422731151 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50f013f790 0x7f50f013bc40 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0x7f50f00b0190 tx=0x7f50e0006df0).stop 2022-06-29T16:08:14.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.521+0000 7f50f5992700 1 -- 172.21.15.60:0/3422731151 shutdown_connections 2022-06-29T16:08:14.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.521+0000 7f50f5992700 1 --2- 172.21.15.60:0/3422731151 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50f013f790 0x7f50f013bc40 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:14.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.525+0000 7f50f5992700 1 -- 172.21.15.60:0/3422731151 >> 172.21.15.60:0/3422731151 conn(0x7f50f0063b50 msgr2=0x7f50f00b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:14.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.525+0000 7f50f5992700 1 -- 172.21.15.60:0/3422731151 shutdown_connections 2022-06-29T16:08:14.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.525+0000 7f50f5992700 1 -- 172.21.15.60:0/3422731151 wait complete. 2022-06-29T16:08:14.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.525+0000 7f50f5992700 1 Processor -- start 2022-06-29T16:08:14.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.525+0000 7f50f5992700 1 -- start start 2022-06-29T16:08:14.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.525+0000 7f50f5992700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50f00b61d0 0x7f50f00b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:14.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.525+0000 7f50f5992700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50f013f790 0x7f50f0110d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:14.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.525+0000 7f50f5992700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f50f013fc00 0x7f50f01111f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:14.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.525+0000 7f50f5992700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f0065f90 con 0x7f50f00b61d0 2022-06-29T16:08:14.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.525+0000 7f50f5992700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f0065b60 con 0x7f50f013f790 2022-06-29T16:08:14.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.525+0000 7f50f5992700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50f014d900 con 0x7f50f013fc00 2022-06-29T16:08:14.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.525+0000 7f50eeffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50f00b61d0 0x7f50f00b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:14.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.525+0000 7f50eeffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50f00b61d0 0x7f50f00b45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54220/0 (socket says 172.21.15.60:54220) 2022-06-29T16:08:14.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.525+0000 7f50eeffd700 1 -- 172.21.15.60:0/200766844 learned_addr learned my addr 172.21.15.60:0/200766844 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:14.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.525+0000 7f50ee7fc700 1 --2- 172.21.15.60:0/200766844 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50f013f790 0x7f50f0110d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:14.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.525+0000 7f50ef7fe700 1 --2- 172.21.15.60:0/200766844 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f50f013fc00 0x7f50f01111f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:14.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.529+0000 7f50eeffd700 1 -- 172.21.15.60:0/200766844 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f50f013fc00 msgr2=0x7f50f01111f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:14.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.529+0000 7f50eeffd700 1 --2- 172.21.15.60:0/200766844 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f50f013fc00 0x7f50f01111f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:14.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.529+0000 7f50eeffd700 1 -- 172.21.15.60:0/200766844 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50f013f790 msgr2=0x7f50f0110d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:14.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.529+0000 7f50eeffd700 1 --2- 172.21.15.60:0/200766844 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50f013f790 0x7f50f0110d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:14.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.529+0000 7f50eeffd700 1 -- 172.21.15.60:0/200766844 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e000b180 con 0x7f50f00b61d0 2022-06-29T16:08:14.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.529+0000 7f50eeffd700 1 --2- 172.21.15.60:0/200766844 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50f00b61d0 0x7f50f00b45e0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 rx=0x7f50f0063f60 tx=0x7f50e4007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:14.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.529+0000 7f50dffff700 1 -- 172.21.15.60:0/200766844 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f50e4012400 con 0x7f50f00b61d0 2022-06-29T16:08:14.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.529+0000 7f50dffff700 1 -- 172.21.15.60:0/200766844 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50e4012560 con 0x7f50f00b61d0 2022-06-29T16:08:14.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.529+0000 7f50f5992700 1 -- 172.21.15.60:0/200766844 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50f01116e0 con 0x7f50f00b61d0 2022-06-29T16:08:14.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.529+0000 7f50dffff700 1 -- 172.21.15.60:0/200766844 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f50e40128c0 con 0x7f50f00b61d0 2022-06-29T16:08:14.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.529+0000 7f50f5992700 1 -- 172.21.15.60:0/200766844 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50f010f4f0 con 0x7f50f00b61d0 2022-06-29T16:08:14.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.529+0000 7f50f5992700 1 -- 172.21.15.60:0/200766844 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50f00670a0 con 0x7f50f00b61d0 2022-06-29T16:08:14.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.529+0000 7f50dffff700 1 -- 172.21.15.60:0/200766844 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f50e4019030 con 0x7f50f00b61d0 2022-06-29T16:08:14.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.529+0000 7f50dffff700 1 --2- 172.21.15.60:0/200766844 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f50cc053aa0 0x7f50cc055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:14.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.529+0000 7f50dffff700 1 -- 172.21.15.60:0/200766844 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f50e406aec0 con 0x7f50f00b61d0 2022-06-29T16:08:14.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.529+0000 7f50ee7fc700 1 --2- 172.21.15.60:0/200766844 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f50cc053aa0 0x7f50cc055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:14.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.533+0000 7f50ee7fc700 1 --2- 172.21.15.60:0/200766844 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f50cc053aa0 0x7f50cc055f60 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 rx=0x7f50e001d000 tx=0x7f50e0016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:14.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.537+0000 7f50dffff700 1 -- 172.21.15.60:0/200766844 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f50f00670a0 con 0x7f50f00b61d0 2022-06-29T16:08:14.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.661+0000 7f50f5992700 1 -- 172.21.15.60:0/200766844 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50f00670a0 con 0x7f50f00b61d0 2022-06-29T16:08:14.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.665+0000 7f50dffff700 1 -- 172.21.15.60:0/200766844 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v141) v1 ==== 74+0+13191 (secure 0 0 0) 0x7f50f00670a0 con 0x7f50f00b61d0 2022-06-29T16:08:14.669 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:08:14.670 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":141,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:08:13.955363+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:08:14.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.665+0000 7f50f5992700 1 -- 172.21.15.60:0/200766844 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f50cc053aa0 msgr2=0x7f50cc055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:14.672 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.665+0000 7f50f5992700 1 --2- 172.21.15.60:0/200766844 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f50cc053aa0 0x7f50cc055f60 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 rx=0x7f50e001d000 tx=0x7f50e0016000).stop 2022-06-29T16:08:14.672 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.665+0000 7f50f5992700 1 -- 172.21.15.60:0/200766844 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50f00b61d0 msgr2=0x7f50f00b45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:14.672 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.665+0000 7f50f5992700 1 --2- 172.21.15.60:0/200766844 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50f00b61d0 0x7f50f00b45e0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 rx=0x7f50f0063f60 tx=0x7f50e4007c50).stop 2022-06-29T16:08:14.673 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.665+0000 7f50f5992700 1 -- 172.21.15.60:0/200766844 shutdown_connections 2022-06-29T16:08:14.673 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.665+0000 7f50f5992700 1 --2- 172.21.15.60:0/200766844 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f50cc053aa0 0x7f50cc055f60 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:14.673 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.665+0000 7f50f5992700 1 --2- 172.21.15.60:0/200766844 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f50f013fc00 0x7f50f01111f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:14.674 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.665+0000 7f50f5992700 1 --2- 172.21.15.60:0/200766844 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50f013f790 0x7f50f0110d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:14.674 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.665+0000 7f50f5992700 1 --2- 172.21.15.60:0/200766844 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50f00b61d0 0x7f50f00b45e0 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:14.674 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.665+0000 7f50f5992700 1 -- 172.21.15.60:0/200766844 >> 172.21.15.60:0/200766844 conn(0x7f50f0063b50 msgr2=0x7f50f0141e30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:14.675 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.665+0000 7f50f5992700 1 -- 172.21.15.60:0/200766844 shutdown_connections 2022-06-29T16:08:14.675 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.665+0000 7f50f5992700 1 -- 172.21.15.60:0/200766844 wait complete. 2022-06-29T16:08:14.687 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-06-29T16:08:14.688 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:08:14.798 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.793+0000 7fe6fb116700 1 Processor -- start 2022-06-29T16:08:14.798 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.793+0000 7fe6fb116700 1 -- start start 2022-06-29T16:08:14.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.793+0000 7fe6fb116700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe6f40b5270 0x7fe6f4146340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:14.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.793+0000 7fe6fb116700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f40b3600 con 0x7fe6f40b9c50 2022-06-29T16:08:14.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.793+0000 7fe6fb116700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f40b09f0 con 0x7fe6f4146830 2022-06-29T16:08:14.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.793+0000 7fe6fb116700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6f40b10e0 con 0x7fe6f40b5270 2022-06-29T16:08:14.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.793+0000 7fe6fa114700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe6f40b9c50 0x7fe6f40b2560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34082/0 (socket says 172.21.15.60:34082) 2022-06-29T16:08:14.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.793+0000 7fe6fa114700 1 -- 172.21.15.60:0/780530349 learned_addr learned my addr 172.21.15.60:0/780530349 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:14.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.793+0000 7fe6f9913700 1 --2- 172.21.15.60:0/780530349 >> v2:172.21.15.92:3300/0 conn(0x7fe6f40b5270 0x7fe6f4146340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:14.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.793+0000 7fe6f9112700 1 -- 172.21.15.60:0/780530349 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4111052567 0 0) 0x7fe6f40b3600 con 0x7fe6f40b9c50 2022-06-29T16:08:14.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.793+0000 7fe6f9112700 1 -- 172.21.15.60:0/780530349 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6dc003260 con 0x7fe6f40b9c50 2022-06-29T16:08:14.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.793+0000 7fe6f9112700 1 -- 172.21.15.60:0/780530349 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2258805484 0 0) 0x7fe6dc003260 con 0x7fe6f40b9c50 2022-06-29T16:08:14.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.793+0000 7fe6f9112700 1 -- 172.21.15.60:0/780530349 >> v2:172.21.15.92:3300/0 conn(0x7fe6f40b5270 msgr2=0x7fe6f4146340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:14.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.793+0000 7fe6f9112700 1 --2- 172.21.15.60:0/780530349 >> v2:172.21.15.92:3300/0 conn(0x7fe6f40b5270 0x7fe6f4146340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:14.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.793+0000 7fe6f9112700 1 -- 172.21.15.60:0/780530349 >> v1:172.21.15.92:6789/0 conn(0x7fe6f4146830 legacy=0x7fe6f414d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:14.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.793+0000 7fe6f9112700 1 -- 172.21.15.60:0/780530349 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f40ba030 con 0x7fe6f40b9c50 2022-06-29T16:08:14.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.793+0000 7fe6f9112700 1 -- 172.21.15.60:0/780530349 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7fe6e4001660 con 0x7fe6f40b9c50 2022-06-29T16:08:14.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.793+0000 7fe6f9112700 1 -- 172.21.15.60:0/780530349 >> v1:172.21.15.60:6789/0 conn(0x7fe6f40b9c50 legacy=0x7fe6f40b2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:14.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.793+0000 7fe6f9112700 1 --2- 172.21.15.60:0/780530349 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6dc003750 0x7fe6dc003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:14.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.793+0000 7fe6f9112700 1 --2- 172.21.15.60:0/780530349 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6dc004500 0x7fe6dc004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:14.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.793+0000 7fe6f9112700 1 --2- 172.21.15.60:0/780530349 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6dc004e70 0x7fe6dc00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:14.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.793+0000 7fe6f9112700 1 -- 172.21.15.60:0/780530349 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f40b10e0 con 0x7fe6dc004e70 2022-06-29T16:08:14.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.793+0000 7fe6f9112700 1 -- 172.21.15.60:0/780530349 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6dc00be50 con 0x7fe6dc004500 2022-06-29T16:08:14.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.793+0000 7fe6f9112700 1 -- 172.21.15.60:0/780530349 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6dc00c010 con 0x7fe6dc003750 2022-06-29T16:08:14.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.797+0000 7fe6fa915700 1 --2- 172.21.15.60:0/780530349 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6dc004e70 0x7fe6dc00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:14.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.797+0000 7fe6f9913700 1 --2- 172.21.15.60:0/780530349 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6dc004500 0x7fe6dc004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:14.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.797+0000 7fe6fa114700 1 --2- 172.21.15.60:0/780530349 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6dc003750 0x7fe6dc003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:14.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.797+0000 7fe6fa915700 1 -- 172.21.15.60:0/780530349 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6dc003750 msgr2=0x7fe6dc003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:14.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.797+0000 7fe6fa915700 1 --2- 172.21.15.60:0/780530349 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6dc003750 0x7fe6dc003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:14.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.797+0000 7fe6fa915700 1 -- 172.21.15.60:0/780530349 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6dc004500 msgr2=0x7fe6dc004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:14.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.797+0000 7fe6fa915700 1 --2- 172.21.15.60:0/780530349 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6dc004500 0x7fe6dc004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:14.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.797+0000 7fe6fa915700 1 -- 172.21.15.60:0/780530349 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f40ba030 con 0x7fe6dc004e70 2022-06-29T16:08:14.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.797+0000 7fe6fa915700 1 -- 172.21.15.60:0/780530349 reap_dead start 2022-06-29T16:08:14.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.797+0000 7fe6fa915700 1 --2- 172.21.15.60:0/780530349 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6dc004e70 0x7fe6dc00b750 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 rx=0x7fe6f4063080 tx=0x7fe6ec007d80).ready entity=mon.0 client_cookie=e35c0f5cd47c8196 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:14.816 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.797+0000 7fe6f9112700 1 -- 172.21.15.60:0/780530349 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6ec001620 con 0x7fe6dc004e70 2022-06-29T16:08:14.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.797+0000 7fe6f9112700 1 -- 172.21.15.60:0/780530349 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6ec006890 con 0x7fe6dc004e70 2022-06-29T16:08:14.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.797+0000 7fe6f9112700 1 -- 172.21.15.60:0/780530349 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6ec006bf0 con 0x7fe6dc004e70 2022-06-29T16:08:14.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.797+0000 7fe6fb116700 1 -- 172.21.15.60:0/780530349 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6dc004e70 msgr2=0x7fe6dc00b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:14.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.797+0000 7fe6fb116700 1 --2- 172.21.15.60:0/780530349 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6dc004e70 0x7fe6dc00b750 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 rx=0x7fe6f4063080 tx=0x7fe6ec007d80).stop 2022-06-29T16:08:14.817 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.797+0000 7fe6fb116700 1 -- 172.21.15.60:0/780530349 shutdown_connections 2022-06-29T16:08:14.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.797+0000 7fe6fb116700 1 --2- 172.21.15.60:0/780530349 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6dc004e70 0x7fe6dc00b750 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:14.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.797+0000 7fe6fb116700 1 -- 172.21.15.60:0/780530349 >> 172.21.15.60:0/780530349 conn(0x7fe6f4062c00 msgr2=0x7fe6f40b9200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:14.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.797+0000 7fe6fb116700 1 -- 172.21.15.60:0/780530349 shutdown_connections 2022-06-29T16:08:14.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.797+0000 7fe6fb116700 1 -- 172.21.15.60:0/780530349 wait complete. 2022-06-29T16:08:14.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.797+0000 7fe6fb116700 1 Processor -- start 2022-06-29T16:08:14.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.801+0000 7fe6fb116700 1 -- start start 2022-06-29T16:08:14.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.801+0000 7fe6fb116700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6dc004e70 0x7fe6f4116310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:14.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.801+0000 7fe6fb116700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6f4112c50 0x7fe6f4113780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:14.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.801+0000 7fe6fb116700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6f4113c70 0x7fe6f41140f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:14.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.801+0000 7fe6fb116700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f41130b0 con 0x7fe6f4112c50 2022-06-29T16:08:14.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.801+0000 7fe6fb116700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f41145e0 con 0x7fe6f4113c70 2022-06-29T16:08:14.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.801+0000 7fe6fb116700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f41554d0 con 0x7fe6dc004e70 2022-06-29T16:08:14.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.801+0000 7fe6f9913700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6f4112c50 0x7fe6f4113780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:14.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.801+0000 7fe6f9913700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6f4112c50 0x7fe6f4113780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54242/0 (socket says 172.21.15.60:54242) 2022-06-29T16:08:14.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.801+0000 7fe6f9913700 1 -- 172.21.15.60:0/2326887462 learned_addr learned my addr 172.21.15.60:0/2326887462 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:14.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.801+0000 7fe6fa114700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6dc004e70 0x7fe6f4116310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:14.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.801+0000 7fe6fa915700 1 --2- 172.21.15.60:0/2326887462 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6f4113c70 0x7fe6f41140f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:14.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.801+0000 7fe6f9913700 1 -- 172.21.15.60:0/2326887462 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6dc004e70 msgr2=0x7fe6f4116310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:14.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.801+0000 7fe6f9913700 1 --2- 172.21.15.60:0/2326887462 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6dc004e70 0x7fe6f4116310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:14.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.801+0000 7fe6f9913700 1 -- 172.21.15.60:0/2326887462 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6f4113c70 msgr2=0x7fe6f41140f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:14.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.801+0000 7fe6f9913700 1 --2- 172.21.15.60:0/2326887462 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6f4113c70 0x7fe6f41140f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:14.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.801+0000 7fe6f9913700 1 -- 172.21.15.60:0/2326887462 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f4155770 con 0x7fe6f4112c50 2022-06-29T16:08:14.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.801+0000 7fe6f9913700 1 --2- 172.21.15.60:0/2326887462 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6f4112c50 0x7fe6f4113780 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 rx=0x7fe6f4114b90 tx=0x7fe6f0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:14.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.801+0000 7fe6e37fe700 1 -- 172.21.15.60:0/2326887462 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6f0013680 con 0x7fe6f4112c50 2022-06-29T16:08:14.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.801+0000 7fe6e37fe700 1 -- 172.21.15.60:0/2326887462 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f00137e0 con 0x7fe6f4112c50 2022-06-29T16:08:14.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.801+0000 7fe6fb116700 1 -- 172.21.15.60:0/2326887462 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6f4155a10 con 0x7fe6f4112c50 2022-06-29T16:08:14.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.801+0000 7fe6e37fe700 1 -- 172.21.15.60:0/2326887462 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6f0013b40 con 0x7fe6f4112c50 2022-06-29T16:08:14.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.801+0000 7fe6fb116700 1 -- 172.21.15.60:0/2326887462 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6f4155eb0 con 0x7fe6f4112c50 2022-06-29T16:08:14.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.805+0000 7fe6e37fe700 1 -- 172.21.15.60:0/2326887462 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe6f0013ca0 con 0x7fe6f4112c50 2022-06-29T16:08:14.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.805+0000 7fe6fb116700 1 -- 172.21.15.60:0/2326887462 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6f41134a0 con 0x7fe6f4112c50 2022-06-29T16:08:14.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.805+0000 7fe6e37fe700 1 --2- 172.21.15.60:0/2326887462 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe6d4053980 0x7fe6d4055e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:14.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.805+0000 7fe6e37fe700 1 -- 172.21.15.60:0/2326887462 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fe6f006c2e0 con 0x7fe6f4112c50 2022-06-29T16:08:14.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.805+0000 7fe6fa114700 1 --2- 172.21.15.60:0/2326887462 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe6d4053980 0x7fe6d4055e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:14.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.805+0000 7fe6fa114700 1 --2- 172.21.15.60:0/2326887462 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe6d4053980 0x7fe6d4055e40 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0x7fe6e400d000 tx=0x7fe6e4007040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:14.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.813+0000 7fe6e37fe700 1 -- 172.21.15.60:0/2326887462 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe6f0043530 con 0x7fe6f4112c50 2022-06-29T16:08:14.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.941+0000 7fe6fb116700 1 -- 172.21.15.60:0/2326887462 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fe6f41552f0 con 0x7fe6f4112c50 2022-06-29T16:08:14.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.941+0000 7fe6e37fe700 1 -- 172.21.15.60:0/2326887462 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v141) v1 ==== 110+0+10 (secure 0 0 0) 0x7fe6f0018020 con 0x7fe6f4112c50 2022-06-29T16:08:14.951 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:08:14.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.941+0000 7fe6fb116700 1 -- 172.21.15.60:0/2326887462 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe6d4053980 msgr2=0x7fe6d4055e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:14.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.941+0000 7fe6fb116700 1 --2- 172.21.15.60:0/2326887462 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe6d4053980 0x7fe6d4055e40 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0x7fe6e400d000 tx=0x7fe6e4007040).stop 2022-06-29T16:08:14.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.941+0000 7fe6fb116700 1 -- 172.21.15.60:0/2326887462 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6f4112c50 msgr2=0x7fe6f4113780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:14.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.941+0000 7fe6fb116700 1 --2- 172.21.15.60:0/2326887462 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6f4112c50 0x7fe6f4113780 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 rx=0x7fe6f4114b90 tx=0x7fe6f0006df0).stop 2022-06-29T16:08:14.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.941+0000 7fe6fb116700 1 -- 172.21.15.60:0/2326887462 shutdown_connections 2022-06-29T16:08:14.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.941+0000 7fe6fb116700 1 --2- 172.21.15.60:0/2326887462 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe6d4053980 0x7fe6d4055e40 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:14.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.941+0000 7fe6fb116700 1 --2- 172.21.15.60:0/2326887462 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6f4113c70 0x7fe6f41140f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:14.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.941+0000 7fe6fb116700 1 --2- 172.21.15.60:0/2326887462 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6f4112c50 0x7fe6f4113780 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:14.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.941+0000 7fe6fb116700 1 --2- 172.21.15.60:0/2326887462 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe6dc004e70 0x7fe6f4116310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:14.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.941+0000 7fe6fb116700 1 -- 172.21.15.60:0/2326887462 >> 172.21.15.60:0/2326887462 conn(0x7fe6f4062c00 msgr2=0x7fe6f405bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:14.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.941+0000 7fe6fb116700 1 -- 172.21.15.60:0/2326887462 shutdown_connections 2022-06-29T16:08:14.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:14.941+0000 7fe6fb116700 1 -- 172.21.15.60:0/2326887462 wait complete. 2022-06-29T16:08:14.960 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:08:15.071 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.065+0000 7ff6f723c700 1 Processor -- start 2022-06-29T16:08:15.072 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.065+0000 7ff6f723c700 1 -- start start 2022-06-29T16:08:15.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.069+0000 7ff6f723c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff6f00ae7c0 0x7ff6f0141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:15.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.069+0000 7ff6f723c700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff6f00b46c0 0x7ff6f00b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:15.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.069+0000 7ff6f723c700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f0143a40 con 0x7ff6f00b4010 2022-06-29T16:08:15.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.069+0000 7ff6f723c700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f00aee60 con 0x7ff6f00b46c0 2022-06-29T16:08:15.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.069+0000 7ff6f723c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f00af100 con 0x7ff6f00ae7c0 2022-06-29T16:08:15.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.069+0000 7ff6f57d9700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff6f00b46c0 0x7ff6f00b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:15.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.069+0000 7ff6f57d9700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff6f00b46c0 0x7ff6f00b4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54248/0 (socket says 172.21.15.60:54248) 2022-06-29T16:08:15.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.069+0000 7ff6f57d9700 1 -- 172.21.15.60:0/1476629323 learned_addr learned my addr 172.21.15.60:0/1476629323 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:15.076 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.069+0000 7ff6f4fd8700 1 --2- 172.21.15.60:0/1476629323 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff6f00ae7c0 0x7ff6f0141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:15.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.069+0000 7ff6f57d9700 1 -- 172.21.15.60:0/1476629323 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff6f00ae7c0 msgr2=0x7ff6f0141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:15.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.069+0000 7ff6f57d9700 1 --2- 172.21.15.60:0/1476629323 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff6f00ae7c0 0x7ff6f0141fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:15.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.069+0000 7ff6f57d9700 1 -- 172.21.15.60:0/1476629323 >> v1:172.21.15.92:6789/0 conn(0x7ff6f00b4010 legacy=0x7ff6f0142490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:15.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.069+0000 7ff6f57d9700 1 -- 172.21.15.60:0/1476629323 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f00aeba0 con 0x7ff6f00b46c0 2022-06-29T16:08:15.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.069+0000 7ff6f57d9700 1 --2- 172.21.15.60:0/1476629323 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f00b46c0 0x7ff6f00b4b50 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 rx=0x7ff6e80032f0 tx=0x7ff6e8009290).ready entity=mon.0 client_cookie=56e3f87cc60f2adc server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:15.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.069+0000 7ff6ef7fe700 1 -- 172.21.15.60:0/1476629323 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff6e800b8a0 con 0x7ff6f00b46c0 2022-06-29T16:08:15.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.069+0000 7ff6ef7fe700 1 -- 172.21.15.60:0/1476629323 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e800ba00 con 0x7ff6f00b46c0 2022-06-29T16:08:15.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.069+0000 7ff6ef7fe700 1 -- 172.21.15.60:0/1476629323 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff6e800bb70 con 0x7ff6f00b46c0 2022-06-29T16:08:15.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.069+0000 7ff6f723c700 1 -- 172.21.15.60:0/1476629323 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f00b46c0 msgr2=0x7ff6f00b4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:15.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.069+0000 7ff6f723c700 1 --2- 172.21.15.60:0/1476629323 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f00b46c0 0x7ff6f00b4b50 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 rx=0x7ff6e80032f0 tx=0x7ff6e8009290).stop 2022-06-29T16:08:15.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.073+0000 7ff6f723c700 1 -- 172.21.15.60:0/1476629323 shutdown_connections 2022-06-29T16:08:15.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.073+0000 7ff6f723c700 1 --2- 172.21.15.60:0/1476629323 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f00b46c0 0x7ff6f00b4b50 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:15.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.073+0000 7ff6f723c700 1 --2- 172.21.15.60:0/1476629323 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff6f00ae7c0 0x7ff6f0141fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:15.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.073+0000 7ff6f723c700 1 -- 172.21.15.60:0/1476629323 >> 172.21.15.60:0/1476629323 conn(0x7ff6f00b9340 msgr2=0x7ff6f00b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:15.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.073+0000 7ff6f723c700 1 -- 172.21.15.60:0/1476629323 shutdown_connections 2022-06-29T16:08:15.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.073+0000 7ff6f723c700 1 -- 172.21.15.60:0/1476629323 wait complete. 2022-06-29T16:08:15.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.073+0000 7ff6f723c700 1 Processor -- start 2022-06-29T16:08:15.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.073+0000 7ff6f723c700 1 -- start start 2022-06-29T16:08:15.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.073+0000 7ff6f723c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6f00ae7c0 0x7ff6f01470f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:15.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.073+0000 7ff6f723c700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff6f00b4010 0x7ff6f0151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:15.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.073+0000 7ff6f723c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f00b46c0 0x7ff6f0113d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:15.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.073+0000 7ff6f723c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f01138a0 con 0x7ff6f00b46c0 2022-06-29T16:08:15.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.073+0000 7ff6f723c700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f00b20b0 con 0x7ff6f00ae7c0 2022-06-29T16:08:15.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.073+0000 7ff6f723c700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f00b1e10 con 0x7ff6f00b4010 2022-06-29T16:08:15.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.073+0000 7ff6f57d9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f00b46c0 0x7ff6f0113d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:15.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.073+0000 7ff6f57d9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f00b46c0 0x7ff6f0113d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54258/0 (socket says 172.21.15.60:54258) 2022-06-29T16:08:15.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.073+0000 7ff6f57d9700 1 -- 172.21.15.60:0/1743747214 learned_addr learned my addr 172.21.15.60:0/1743747214 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:15.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.073+0000 7ff6f4fd8700 1 --2- 172.21.15.60:0/1743747214 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6f00ae7c0 0x7ff6f01470f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:15.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.073+0000 7ff6effff700 1 --2- 172.21.15.60:0/1743747214 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff6f00b4010 0x7ff6f0151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:15.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.077+0000 7ff6f57d9700 1 -- 172.21.15.60:0/1743747214 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff6f00b4010 msgr2=0x7ff6f0151ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:15.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.077+0000 7ff6f57d9700 1 --2- 172.21.15.60:0/1743747214 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff6f00b4010 0x7ff6f0151ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:15.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.077+0000 7ff6f57d9700 1 -- 172.21.15.60:0/1743747214 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6f00ae7c0 msgr2=0x7ff6f01470f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:15.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.077+0000 7ff6f57d9700 1 --2- 172.21.15.60:0/1743747214 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6f00ae7c0 0x7ff6f01470f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:15.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.077+0000 7ff6f57d9700 1 -- 172.21.15.60:0/1743747214 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e8009530 con 0x7ff6f00b46c0 2022-06-29T16:08:15.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.077+0000 7ff6f57d9700 1 --2- 172.21.15.60:0/1743747214 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f00b46c0 0x7ff6f0113d00 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 rx=0x7ff6e80032f0 tx=0x7ff6e8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:15.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.077+0000 7ff6edffb700 1 -- 172.21.15.60:0/1743747214 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff6e800b8a0 con 0x7ff6f00b46c0 2022-06-29T16:08:15.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.077+0000 7ff6edffb700 1 -- 172.21.15.60:0/1743747214 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e8021440 con 0x7ff6f00b46c0 2022-06-29T16:08:15.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.077+0000 7ff6f723c700 1 -- 172.21.15.60:0/1743747214 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f01523e0 con 0x7ff6f00b46c0 2022-06-29T16:08:15.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.077+0000 7ff6edffb700 1 -- 172.21.15.60:0/1743747214 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff6e80217a0 con 0x7ff6f00b46c0 2022-06-29T16:08:15.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.077+0000 7ff6f723c700 1 -- 172.21.15.60:0/1743747214 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f00b2e20 con 0x7ff6f00b46c0 2022-06-29T16:08:15.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.077+0000 7ff6f723c700 1 -- 172.21.15.60:0/1743747214 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f0113a10 con 0x7ff6f00b46c0 2022-06-29T16:08:15.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.077+0000 7ff6edffb700 1 -- 172.21.15.60:0/1743747214 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff6e800f070 con 0x7ff6f00b46c0 2022-06-29T16:08:15.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.077+0000 7ff6edffb700 1 --2- 172.21.15.60:0/1743747214 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff6d0053b00 0x7ff6d0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:15.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.077+0000 7ff6edffb700 1 -- 172.21.15.60:0/1743747214 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5253+0+0 (secure 0 0 0) 0x7ff6e806d410 con 0x7ff6f00b46c0 2022-06-29T16:08:15.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.077+0000 7ff6f4fd8700 1 --2- 172.21.15.60:0/1743747214 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff6d0053b00 0x7ff6d0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:15.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.077+0000 7ff6f4fd8700 1 --2- 172.21.15.60:0/1743747214 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff6d0053b00 0x7ff6d0055fc0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0x7ff6e000b000 tx=0x7ff6e0006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:15.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.085+0000 7ff6edffb700 1 -- 172.21.15.60:0/1743747214 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff6e8044660 con 0x7ff6f00b46c0 2022-06-29T16:08:15.216 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.213+0000 7ff6f723c700 1 -- 172.21.15.60:0/1743747214 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7ff6f00573d0 con 0x7ff6f00b46c0 2022-06-29T16:08:15.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.213+0000 7ff6edffb700 1 -- 172.21.15.60:0/1743747214 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v141) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff6e8015020 con 0x7ff6f00b46c0 2022-06-29T16:08:15.218 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:08:15.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.213+0000 7ff6f723c700 1 -- 172.21.15.60:0/1743747214 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff6d0053b00 msgr2=0x7ff6d0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:15.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.213+0000 7ff6f723c700 1 --2- 172.21.15.60:0/1743747214 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff6d0053b00 0x7ff6d0055fc0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0x7ff6e000b000 tx=0x7ff6e0006df0).stop 2022-06-29T16:08:15.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.213+0000 7ff6f723c700 1 -- 172.21.15.60:0/1743747214 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f00b46c0 msgr2=0x7ff6f0113d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:15.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.213+0000 7ff6f723c700 1 --2- 172.21.15.60:0/1743747214 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f00b46c0 0x7ff6f0113d00 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 rx=0x7ff6e80032f0 tx=0x7ff6e8016000).stop 2022-06-29T16:08:15.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.213+0000 7ff6f723c700 1 -- 172.21.15.60:0/1743747214 shutdown_connections 2022-06-29T16:08:15.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.213+0000 7ff6f723c700 1 --2- 172.21.15.60:0/1743747214 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff6d0053b00 0x7ff6d0055fc0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:15.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.213+0000 7ff6f723c700 1 --2- 172.21.15.60:0/1743747214 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6f00b46c0 0x7ff6f0113d00 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:15.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.213+0000 7ff6f723c700 1 --2- 172.21.15.60:0/1743747214 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff6f00b4010 0x7ff6f0151ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:15.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.213+0000 7ff6f723c700 1 --2- 172.21.15.60:0/1743747214 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6f00ae7c0 0x7ff6f01470f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:15.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.213+0000 7ff6f723c700 1 -- 172.21.15.60:0/1743747214 >> 172.21.15.60:0/1743747214 conn(0x7ff6f00b9340 msgr2=0x7ff6f00b6040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:15.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.213+0000 7ff6f723c700 1 -- 172.21.15.60:0/1743747214 shutdown_connections 2022-06-29T16:08:15.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.213+0000 7ff6f723c700 1 -- 172.21.15.60:0/1743747214 wait complete. 2022-06-29T16:08:15.237 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-06-29T16:08:15.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.341+0000 7f6214c0a700 1 Processor -- start 2022-06-29T16:08:15.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.341+0000 7f6214c0a700 1 -- start start 2022-06-29T16:08:15.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.341+0000 7f6214c0a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f621005d070 0x7f621014aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:15.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.341+0000 7f6214c0a700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f62100b2130 0x7f62100aca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:15.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.341+0000 7f6214c0a700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f621005d450 con 0x7f621014af80 2022-06-29T16:08:15.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.341+0000 7f6214c0a700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62100adf90 con 0x7f62100b2130 2022-06-29T16:08:15.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.341+0000 7f6214c0a700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62100b25d0 con 0x7f621005d070 2022-06-29T16:08:15.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.341+0000 7f620ffff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f621014af80 0x7f62100acf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34120/0 (socket says 172.21.15.60:34120) 2022-06-29T16:08:15.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.341+0000 7f620ffff700 1 -- 172.21.15.60:0/4162396115 learned_addr learned my addr 172.21.15.60:0/4162396115 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:15.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.341+0000 7f620effd700 1 --2- 172.21.15.60:0/4162396115 >> v2:172.21.15.92:3300/0 conn(0x7f62100b2130 0x7f62100aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:15.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.341+0000 7f620f7fe700 1 --2- 172.21.15.60:0/4162396115 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f621005d070 0x7f621014aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:15.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.341+0000 7f620e7fc700 1 -- 172.21.15.60:0/4162396115 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2554466362 0 0) 0x7f621005d450 con 0x7f621014af80 2022-06-29T16:08:15.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.341+0000 7f620e7fc700 1 -- 172.21.15.60:0/4162396115 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61f4003260 con 0x7f621014af80 2022-06-29T16:08:15.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f620e7fc700 1 -- 172.21.15.60:0/4162396115 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1506545706 0 0) 0x7f61f4003260 con 0x7f621014af80 2022-06-29T16:08:15.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f620e7fc700 1 -- 172.21.15.60:0/4162396115 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f621005d070 msgr2=0x7f621014aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:15.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f620e7fc700 1 --2- 172.21.15.60:0/4162396115 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f621005d070 0x7f621014aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:15.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f620e7fc700 1 -- 172.21.15.60:0/4162396115 >> v2:172.21.15.92:3300/0 conn(0x7f62100b2130 msgr2=0x7f62100aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:15.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f620e7fc700 1 --2- 172.21.15.60:0/4162396115 >> v2:172.21.15.92:3300/0 conn(0x7f62100b2130 0x7f62100aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:15.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f620e7fc700 1 -- 172.21.15.60:0/4162396115 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f621014a350 con 0x7f621014af80 2022-06-29T16:08:15.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f620e7fc700 1 -- 172.21.15.60:0/4162396115 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f6200001680 con 0x7f621014af80 2022-06-29T16:08:15.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f620e7fc700 1 -- 172.21.15.60:0/4162396115 >> v1:172.21.15.60:6789/0 conn(0x7f621014af80 legacy=0x7f62100acf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:15.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f620e7fc700 1 --2- 172.21.15.60:0/4162396115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61f4003750 0x7f61f4003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:15.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f620e7fc700 1 --2- 172.21.15.60:0/4162396115 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61f4004500 0x7f61f4004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:15.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f620e7fc700 1 --2- 172.21.15.60:0/4162396115 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61f4004e70 0x7f61f400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:15.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f620e7fc700 1 -- 172.21.15.60:0/4162396115 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62100adf90 con 0x7f61f4004500 2022-06-29T16:08:15.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f620e7fc700 1 -- 172.21.15.60:0/4162396115 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62100b25d0 con 0x7f61f4003750 2022-06-29T16:08:15.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f620e7fc700 1 -- 172.21.15.60:0/4162396115 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61f400bc40 con 0x7f61f4004e70 2022-06-29T16:08:15.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f620f7fe700 1 --2- 172.21.15.60:0/4162396115 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f621005d070 0x7f621014aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:08:15.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f620effd700 1 --2- 172.21.15.60:0/4162396115 >> v2:172.21.15.92:3300/0 conn(0x7f62100b2130 0x7f62100aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:08:15.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f620effd700 1 --2- 172.21.15.60:0/4162396115 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61f4004500 0x7f61f4004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:15.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f620f7fe700 1 --2- 172.21.15.60:0/4162396115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61f4003750 0x7f61f4003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:15.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f620ffff700 1 --2- 172.21.15.60:0/4162396115 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61f4004e70 0x7f61f400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:15.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f620effd700 1 -- 172.21.15.60:0/4162396115 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61f4004e70 msgr2=0x7f61f400b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:15.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f620effd700 1 --2- 172.21.15.60:0/4162396115 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61f4004e70 0x7f61f400b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:15.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f620effd700 1 -- 172.21.15.60:0/4162396115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61f4003750 msgr2=0x7f61f4003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:15.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f620effd700 1 --2- 172.21.15.60:0/4162396115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61f4003750 0x7f61f4003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:15.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f620effd700 1 -- 172.21.15.60:0/4162396115 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f4003260 con 0x7f61f4004500 2022-06-29T16:08:15.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f620effd700 1 --2- 172.21.15.60:0/4162396115 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61f4004500 0x7f61f4004980 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 rx=0x7f61f4004470 tx=0x7f61f80091d0).ready entity=mon.0 client_cookie=6b33fed1186e519f server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:15.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f620e7fc700 1 -- 172.21.15.60:0/4162396115 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61f800e920 con 0x7f61f4004500 2022-06-29T16:08:15.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f620e7fc700 1 -- 172.21.15.60:0/4162396115 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f800ea80 con 0x7f61f4004500 2022-06-29T16:08:15.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f620e7fc700 1 -- 172.21.15.60:0/4162396115 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61f800ede0 con 0x7f61f4004500 2022-06-29T16:08:15.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f6214c0a700 1 -- 172.21.15.60:0/4162396115 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61f4004500 msgr2=0x7f61f4004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:15.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f6214c0a700 1 --2- 172.21.15.60:0/4162396115 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61f4004500 0x7f61f4004980 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 rx=0x7f61f4004470 tx=0x7f61f80091d0).stop 2022-06-29T16:08:15.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f620ffff700 1 -- 172.21.15.60:0/4162396115 reap_dead start 2022-06-29T16:08:15.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f6214c0a700 1 -- 172.21.15.60:0/4162396115 shutdown_connections 2022-06-29T16:08:15.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f6214c0a700 1 -- 172.21.15.60:0/4162396115 >> 172.21.15.60:0/4162396115 conn(0x7f62100b5890 msgr2=0x7f62100b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:15.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f6214c0a700 1 -- 172.21.15.60:0/4162396115 shutdown_connections 2022-06-29T16:08:15.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f6214c0a700 1 -- 172.21.15.60:0/4162396115 wait complete. 2022-06-29T16:08:15.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f6214c0a700 1 Processor -- start 2022-06-29T16:08:15.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.345+0000 7f6214c0a700 1 -- start start 2022-06-29T16:08:15.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.349+0000 7f6214c0a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6210116bc0 0x7f6210116fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:15.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.349+0000 7f6214c0a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6210066bf0 0x7f6210159920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:15.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.349+0000 7f6214c0a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6210159e10 0x7f621015a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:15.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.349+0000 7f6214c0a700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f621015a780 con 0x7f6210116bc0 2022-06-29T16:08:15.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.349+0000 7f6214c0a700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6210119430 con 0x7f6210066bf0 2022-06-29T16:08:15.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.349+0000 7f6214c0a700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62101196d0 con 0x7f6210159e10 2022-06-29T16:08:15.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.349+0000 7f620f7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6210116bc0 0x7f6210116fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:15.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.349+0000 7f620ffff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6210159e10 0x7f621015a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:15.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.349+0000 7f620f7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6210116bc0 0x7f6210116fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54276/0 (socket says 172.21.15.60:54276) 2022-06-29T16:08:15.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.349+0000 7f620f7fe700 1 -- 172.21.15.60:0/737137098 learned_addr learned my addr 172.21.15.60:0/737137098 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:15.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.349+0000 7f620effd700 1 --2- 172.21.15.60:0/737137098 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6210066bf0 0x7f6210159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:15.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.349+0000 7f620f7fe700 1 -- 172.21.15.60:0/737137098 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6210159e10 msgr2=0x7f621015a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:15.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.349+0000 7f620f7fe700 1 --2- 172.21.15.60:0/737137098 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6210159e10 0x7f621015a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:15.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.349+0000 7f620f7fe700 1 -- 172.21.15.60:0/737137098 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6210066bf0 msgr2=0x7f6210159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:15.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.349+0000 7f620f7fe700 1 --2- 172.21.15.60:0/737137098 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6210066bf0 0x7f6210159920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:15.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.349+0000 7f620f7fe700 1 -- 172.21.15.60:0/737137098 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6210065c70 con 0x7f6210116bc0 2022-06-29T16:08:15.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.349+0000 7f620ffff700 1 --2- 172.21.15.60:0/737137098 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6210159e10 0x7f621015a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:08:15.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.349+0000 7f620f7fe700 1 --2- 172.21.15.60:0/737137098 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6210116bc0 0x7f6210116fa0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 rx=0x7f620800b000 tx=0x7f6208008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:15.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.349+0000 7f620cff9700 1 -- 172.21.15.60:0/737137098 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6208012610 con 0x7f6210116bc0 2022-06-29T16:08:15.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.349+0000 7f620cff9700 1 -- 172.21.15.60:0/737137098 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6208012770 con 0x7f6210116bc0 2022-06-29T16:08:15.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.349+0000 7f6214c0a700 1 -- 172.21.15.60:0/737137098 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6210117490 con 0x7f6210116bc0 2022-06-29T16:08:15.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.349+0000 7f620cff9700 1 -- 172.21.15.60:0/737137098 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62080128e0 con 0x7f6210116bc0 2022-06-29T16:08:15.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.349+0000 7f6214c0a700 1 -- 172.21.15.60:0/737137098 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6210066110 con 0x7f6210116bc0 2022-06-29T16:08:15.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.353+0000 7f6214c0a700 1 -- 172.21.15.60:0/737137098 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f621014a5f0 con 0x7f6210116bc0 2022-06-29T16:08:15.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.353+0000 7f620cff9700 1 -- 172.21.15.60:0/737137098 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6208012a40 con 0x7f6210116bc0 2022-06-29T16:08:15.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.353+0000 7f620cff9700 1 --2- 172.21.15.60:0/737137098 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f61e8053a10 0x7f61e8055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:15.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.353+0000 7f620cff9700 1 -- 172.21.15.60:0/737137098 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f620806bf90 con 0x7f6210116bc0 2022-06-29T16:08:15.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.353+0000 7f620effd700 1 --2- 172.21.15.60:0/737137098 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f61e8053a10 0x7f61e8055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:15.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.353+0000 7f620effd700 1 --2- 172.21.15.60:0/737137098 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f61e8053a10 0x7f61e8055ed0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7f61f801e000 tx=0x7f61f801a000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:15.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.357+0000 7f620cff9700 1 -- 172.21.15.60:0/737137098 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6208043140 con 0x7f6210116bc0 2022-06-29T16:08:15.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.489+0000 7f6214c0a700 1 -- 172.21.15.60:0/737137098 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f6210065e00 con 0x7f6210116bc0 2022-06-29T16:08:15.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.489+0000 7f620cff9700 1 -- 172.21.15.60:0/737137098 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v141) v1 ==== 84+0+143 (secure 0 0 0) 0x7f6208046760 con 0x7f6210116bc0 2022-06-29T16:08:15.495 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:08:15.495 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2022-06-29T16:08:15.495 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:15.496 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2022-06-29T16:08:15.496 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": "device_health_metrics" 2022-06-29T16:08:15.496 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:15.497 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:15.497 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 2, 2022-06-29T16:08:15.497 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": "rbd" 2022-06-29T16:08:15.497 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:15.498 INFO:teuthology.orchestra.run.smithi060.stdout:] 2022-06-29T16:08:15.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.493+0000 7f6214c0a700 1 -- 172.21.15.60:0/737137098 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f61e8053a10 msgr2=0x7f61e8055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:15.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.493+0000 7f6214c0a700 1 --2- 172.21.15.60:0/737137098 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f61e8053a10 0x7f61e8055ed0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7f61f801e000 tx=0x7f61f801a000).stop 2022-06-29T16:08:15.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.493+0000 7f6214c0a700 1 -- 172.21.15.60:0/737137098 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6210116bc0 msgr2=0x7f6210116fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:15.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.493+0000 7f6214c0a700 1 --2- 172.21.15.60:0/737137098 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6210116bc0 0x7f6210116fa0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 rx=0x7f620800b000 tx=0x7f6208008040).stop 2022-06-29T16:08:15.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.493+0000 7f6214c0a700 1 -- 172.21.15.60:0/737137098 shutdown_connections 2022-06-29T16:08:15.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.493+0000 7f6214c0a700 1 --2- 172.21.15.60:0/737137098 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f61e8053a10 0x7f61e8055ed0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:15.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.493+0000 7f6214c0a700 1 --2- 172.21.15.60:0/737137098 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6210159e10 0x7f621015a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:15.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.493+0000 7f6214c0a700 1 --2- 172.21.15.60:0/737137098 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6210066bf0 0x7f6210159920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:15.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.493+0000 7f6214c0a700 1 --2- 172.21.15.60:0/737137098 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6210116bc0 0x7f6210116fa0 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:15.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.493+0000 7f6214c0a700 1 -- 172.21.15.60:0/737137098 >> 172.21.15.60:0/737137098 conn(0x7f62100b5890 msgr2=0x7f62100af720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:15.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.493+0000 7f6214c0a700 1 -- 172.21.15.60:0/737137098 shutdown_connections 2022-06-29T16:08:15.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.493+0000 7f6214c0a700 1 -- 172.21.15.60:0/737137098 wait complete. 2022-06-29T16:08:15.513 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-06-29T16:08:15.514 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-06-29T16:08:15.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.621+0000 7f0fd72ab700 1 Processor -- start 2022-06-29T16:08:15.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.621+0000 7f0fd72ab700 1 -- start start 2022-06-29T16:08:15.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.621+0000 7f0fd72ab700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0fd01426b0 0x7f0fd00b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:15.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.621+0000 7f0fd72ab700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fd0143a40 con 0x7f0fd00ae7c0 2022-06-29T16:08:15.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.621+0000 7f0fd72ab700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fd00aee60 con 0x7f0fd00b4010 2022-06-29T16:08:15.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.621+0000 7f0fd72ab700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd00af100 con 0x7f0fd01426b0 2022-06-29T16:08:15.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.621+0000 7f0fd5047700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f0fd00ae7c0 0x7f0fd0141fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34136/0 (socket says 172.21.15.60:34136) 2022-06-29T16:08:15.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.621+0000 7f0fd5047700 1 -- 172.21.15.60:0/2193526747 learned_addr learned my addr 172.21.15.60:0/2193526747 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:15.628 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.621+0000 7f0fd5848700 1 --2- 172.21.15.60:0/2193526747 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0fd01426b0 0x7f0fd00b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:15.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.621+0000 7f0fbbfff700 1 -- 172.21.15.60:0/2193526747 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 826586017 0 0) 0x7f0fd0143a40 con 0x7f0fd00ae7c0 2022-06-29T16:08:15.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.621+0000 7f0fbbfff700 1 -- 172.21.15.60:0/2193526747 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fb4003260 con 0x7f0fd00ae7c0 2022-06-29T16:08:15.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.621+0000 7f0fd5848700 1 -- 172.21.15.60:0/2193526747 >> v1:172.21.15.92:6789/0 conn(0x7f0fd00b4010 legacy=0x7f0fd00b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:15.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.621+0000 7f0fbbfff700 1 -- 172.21.15.60:0/2193526747 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2235078981 0 0) 0x7f0fd00aee60 con 0x7f0fd00b4010 2022-06-29T16:08:15.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.621+0000 7f0fd5848700 1 -- 172.21.15.60:0/2193526747 >> v1:172.21.15.60:6789/0 conn(0x7f0fd00ae7c0 legacy=0x7f0fd0141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:15.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.621+0000 7f0fd5848700 1 -- 172.21.15.60:0/2193526747 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd00aeba0 con 0x7f0fd01426b0 2022-06-29T16:08:15.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.621+0000 7f0fd5848700 1 --2- 172.21.15.60:0/2193526747 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0fd01426b0 0x7f0fd00b4dd0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f0fc80032f0 tx=0x7f0fc80091d0).ready entity=mon.2 client_cookie=323da4ef9138f290 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:15.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.625+0000 7f0fbbfff700 1 -- 172.21.15.60:0/2193526747 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0fc800b6c0 con 0x7f0fd01426b0 2022-06-29T16:08:15.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.625+0000 7f0fbbfff700 1 -- 172.21.15.60:0/2193526747 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc800b820 con 0x7f0fd01426b0 2022-06-29T16:08:15.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.625+0000 7f0fbbfff700 1 -- 172.21.15.60:0/2193526747 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0fc800b990 con 0x7f0fd01426b0 2022-06-29T16:08:15.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.625+0000 7f0fd72ab700 1 -- 172.21.15.60:0/2193526747 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0fd01426b0 msgr2=0x7f0fd00b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:15.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.625+0000 7f0fd72ab700 1 --2- 172.21.15.60:0/2193526747 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0fd01426b0 0x7f0fd00b4dd0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f0fc80032f0 tx=0x7f0fc80091d0).stop 2022-06-29T16:08:15.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.625+0000 7f0fd72ab700 1 -- 172.21.15.60:0/2193526747 shutdown_connections 2022-06-29T16:08:15.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.625+0000 7f0fd72ab700 1 --2- 172.21.15.60:0/2193526747 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0fd01426b0 0x7f0fd00b4dd0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:15.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.625+0000 7f0fd72ab700 1 -- 172.21.15.60:0/2193526747 >> 172.21.15.60:0/2193526747 conn(0x7f0fd00b9340 msgr2=0x7f0fd00b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:15.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.625+0000 7f0fd72ab700 1 -- 172.21.15.60:0/2193526747 shutdown_connections 2022-06-29T16:08:15.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.625+0000 7f0fd72ab700 1 -- 172.21.15.60:0/2193526747 wait complete. 2022-06-29T16:08:15.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.625+0000 7f0fd72ab700 1 Processor -- start 2022-06-29T16:08:15.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.625+0000 7f0fd72ab700 1 -- start start 2022-06-29T16:08:15.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.629+0000 7f0fd72ab700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0fd00ae7c0 0x7f0fd005c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:15.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.629+0000 7f0fd72ab700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0fd00b4010 0x7f0fd0112e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:15.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.629+0000 7f0fd72ab700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fd01426b0 0x7f0fd0151c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:15.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.629+0000 7f0fd72ab700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd00b1340 con 0x7f0fd01426b0 2022-06-29T16:08:15.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.629+0000 7f0fd72ab700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd0112720 con 0x7f0fd00b4010 2022-06-29T16:08:15.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.629+0000 7f0fd72ab700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd01129c0 con 0x7f0fd00ae7c0 2022-06-29T16:08:15.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.629+0000 7f0fd5848700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fd01426b0 0x7f0fd0151c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:15.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.629+0000 7f0fd5848700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fd01426b0 0x7f0fd0151c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54292/0 (socket says 172.21.15.60:54292) 2022-06-29T16:08:15.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.629+0000 7f0fd5848700 1 -- 172.21.15.60:0/260105121 learned_addr learned my addr 172.21.15.60:0/260105121 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:15.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.629+0000 7f0fd5047700 1 --2- 172.21.15.60:0/260105121 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0fd00ae7c0 0x7f0fd005c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:15.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.629+0000 7f0fd4846700 1 --2- 172.21.15.60:0/260105121 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0fd00b4010 0x7f0fd0112e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:15.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.629+0000 7f0fd5848700 1 -- 172.21.15.60:0/260105121 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0fd00ae7c0 msgr2=0x7f0fd005c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:15.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.629+0000 7f0fd5848700 1 --2- 172.21.15.60:0/260105121 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0fd00ae7c0 0x7f0fd005c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:15.640 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.629+0000 7f0fd5848700 1 -- 172.21.15.60:0/260105121 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0fd00b4010 msgr2=0x7f0fd0112e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:15.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.629+0000 7f0fd5848700 1 --2- 172.21.15.60:0/260105121 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0fd00b4010 0x7f0fd0112e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:15.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.629+0000 7f0fd5848700 1 -- 172.21.15.60:0/260105121 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc8009470 con 0x7f0fd01426b0 2022-06-29T16:08:15.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.629+0000 7f0fd5848700 1 --2- 172.21.15.60:0/260105121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fd01426b0 0x7f0fd0151c70 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 rx=0x7f0fc80032f0 tx=0x7f0fc8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:15.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.629+0000 7f0fc2ffd700 1 -- 172.21.15.60:0/260105121 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0fc800b990 con 0x7f0fd01426b0 2022-06-29T16:08:15.641 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.629+0000 7f0fc2ffd700 1 -- 172.21.15.60:0/260105121 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc801f3f0 con 0x7f0fd01426b0 2022-06-29T16:08:15.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.629+0000 7f0fd72ab700 1 -- 172.21.15.60:0/260105121 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd0113350 con 0x7f0fd01426b0 2022-06-29T16:08:15.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.629+0000 7f0fc2ffd700 1 -- 172.21.15.60:0/260105121 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0fc800b6c0 con 0x7f0fd01426b0 2022-06-29T16:08:15.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.629+0000 7f0fd72ab700 1 -- 172.21.15.60:0/260105121 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd005cd30 con 0x7f0fd01426b0 2022-06-29T16:08:15.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.629+0000 7f0fd72ab700 1 -- 172.21.15.60:0/260105121 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fd0112b80 con 0x7f0fd01426b0 2022-06-29T16:08:15.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.629+0000 7f0fc2ffd700 1 -- 172.21.15.60:0/260105121 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0fc8026030 con 0x7f0fd01426b0 2022-06-29T16:08:15.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.633+0000 7f0fc2ffd700 1 --2- 172.21.15.60:0/260105121 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0fac053a70 0x7f0fac055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:15.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.633+0000 7f0fc2ffd700 1 -- 172.21.15.60:0/260105121 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f0fc806c950 con 0x7f0fd01426b0 2022-06-29T16:08:15.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.633+0000 7f0fd5047700 1 --2- 172.21.15.60:0/260105121 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0fac053a70 0x7f0fac055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:15.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.633+0000 7f0fd5047700 1 --2- 172.21.15.60:0/260105121 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0fac053a70 0x7f0fac055f30 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0x7f0fcc00b000 tx=0x7f0fcc009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:15.646 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.637+0000 7f0fc2ffd700 1 -- 172.21.15.60:0/260105121 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0fc80474c0 con 0x7f0fd01426b0 2022-06-29T16:08:15.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:15.765+0000 7f0fd72ab700 1 -- 172.21.15.60:0/260105121 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f0fd00b1160 con 0x7f0fd01426b0 2022-06-29T16:08:16.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.145+0000 7f0fc2ffd700 1 -- 172.21.15.60:0/260105121 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v142) v1 ==== 133+0+0 (secure 0 0 0) 0x7f0fc80240a0 con 0x7f0fd01426b0 2022-06-29T16:08:16.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.149+0000 7f0fd72ab700 1 -- 172.21.15.60:0/260105121 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0fac053a70 msgr2=0x7f0fac055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:16.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.149+0000 7f0fd72ab700 1 --2- 172.21.15.60:0/260105121 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0fac053a70 0x7f0fac055f30 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0x7f0fcc00b000 tx=0x7f0fcc009040).stop 2022-06-29T16:08:16.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.149+0000 7f0fd72ab700 1 -- 172.21.15.60:0/260105121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fd01426b0 msgr2=0x7f0fd0151c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:16.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.149+0000 7f0fd72ab700 1 --2- 172.21.15.60:0/260105121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fd01426b0 0x7f0fd0151c70 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 rx=0x7f0fc80032f0 tx=0x7f0fc8016000).stop 2022-06-29T16:08:16.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.149+0000 7f0fd72ab700 1 -- 172.21.15.60:0/260105121 shutdown_connections 2022-06-29T16:08:16.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.149+0000 7f0fd72ab700 1 --2- 172.21.15.60:0/260105121 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0fac053a70 0x7f0fac055f30 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:16.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.149+0000 7f0fd72ab700 1 --2- 172.21.15.60:0/260105121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0fd01426b0 0x7f0fd0151c70 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:16.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.149+0000 7f0fd72ab700 1 --2- 172.21.15.60:0/260105121 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0fd00b4010 0x7f0fd0112e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:16.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.149+0000 7f0fd72ab700 1 --2- 172.21.15.60:0/260105121 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0fd00ae7c0 0x7f0fd005c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:16.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.149+0000 7f0fd72ab700 1 -- 172.21.15.60:0/260105121 >> 172.21.15.60:0/260105121 conn(0x7f0fd00b9340 msgr2=0x7f0fd01482a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:16.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.149+0000 7f0fd72ab700 1 -- 172.21.15.60:0/260105121 shutdown_connections 2022-06-29T16:08:16.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.149+0000 7f0fd72ab700 1 -- 172.21.15.60:0/260105121 wait complete. 2022-06-29T16:08:16.158 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2022-06-29T16:08:16.169 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-06-29T16:08:16.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.281+0000 7f4177269700 1 Processor -- start 2022-06-29T16:08:16.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.281+0000 7f4177269700 1 -- start start 2022-06-29T16:08:16.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.281+0000 7f4177269700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4170065450 0x7f41700b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:16.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.281+0000 7f4177269700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4170060760 0x7f4170140180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:16.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.281+0000 7f4177269700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4170140670 con 0x7f41700b28e0 2022-06-29T16:08:16.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.281+0000 7f4177269700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f417013fa40 con 0x7f4170065450 2022-06-29T16:08:16.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.281+0000 7f4177269700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f417013fce0 con 0x7f4170060760 2022-06-29T16:08:16.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.285+0000 7f4176a68700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4170060760 0x7f4170140180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:16.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.285+0000 7f4175a66700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4170065450 0x7f41700b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:16.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.285+0000 7f4176a68700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4170060760 0x7f4170140180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:39162/0 (socket says 172.21.15.60:39162) 2022-06-29T16:08:16.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.285+0000 7f4176a68700 1 -- 172.21.15.60:0/2952695801 learned_addr learned my addr 172.21.15.60:0/2952695801 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:16.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.285+0000 7f4175a66700 1 -- 172.21.15.60:0/2952695801 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4170060760 msgr2=0x7f4170140180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:16.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.285+0000 7f4175a66700 1 --2- 172.21.15.60:0/2952695801 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4170060760 0x7f4170140180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:16.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.285+0000 7f4175a66700 1 -- 172.21.15.60:0/2952695801 >> v1:172.21.15.92:6789/0 conn(0x7f41700b28e0 legacy=0x7f4170139340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:16.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.285+0000 7f4175a66700 1 -- 172.21.15.60:0/2952695801 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4170140860 con 0x7f4170065450 2022-06-29T16:08:16.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.285+0000 7f4176a68700 1 --2- 172.21.15.60:0/2952695801 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4170060760 0x7f4170140180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:08:16.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.285+0000 7f4175a66700 1 --2- 172.21.15.60:0/2952695801 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4170065450 0x7f41700b2cc0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0x7f416c002f70 tx=0x7f416c006df0).ready entity=mon.1 client_cookie=253a7c493081f196 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:16.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.285+0000 7f4175265700 1 -- 172.21.15.60:0/2952695801 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f416c014070 con 0x7f4170065450 2022-06-29T16:08:16.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.285+0000 7f4175265700 1 -- 172.21.15.60:0/2952695801 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f416c000e90 con 0x7f4170065450 2022-06-29T16:08:16.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.285+0000 7f4175265700 1 -- 172.21.15.60:0/2952695801 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f416c00fb60 con 0x7f4170065450 2022-06-29T16:08:16.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.285+0000 7f4177269700 1 -- 172.21.15.60:0/2952695801 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4170065450 msgr2=0x7f41700b2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:16.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.285+0000 7f4177269700 1 --2- 172.21.15.60:0/2952695801 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4170065450 0x7f41700b2cc0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0x7f416c002f70 tx=0x7f416c006df0).stop 2022-06-29T16:08:16.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.285+0000 7f4177269700 1 -- 172.21.15.60:0/2952695801 shutdown_connections 2022-06-29T16:08:16.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.285+0000 7f4177269700 1 --2- 172.21.15.60:0/2952695801 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4170060760 0x7f4170140180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:16.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.285+0000 7f4177269700 1 --2- 172.21.15.60:0/2952695801 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4170065450 0x7f41700b2cc0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:16.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.285+0000 7f4177269700 1 -- 172.21.15.60:0/2952695801 >> 172.21.15.60:0/2952695801 conn(0x7f4170061c80 msgr2=0x7f41700637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:16.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.285+0000 7f4177269700 1 -- 172.21.15.60:0/2952695801 shutdown_connections 2022-06-29T16:08:16.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.285+0000 7f4177269700 1 -- 172.21.15.60:0/2952695801 wait complete. 2022-06-29T16:08:16.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.289+0000 7f4177269700 1 Processor -- start 2022-06-29T16:08:16.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.289+0000 7f4177269700 1 -- start start 2022-06-29T16:08:16.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.289+0000 7f4177269700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4170060760 0x7f41701421e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:16.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.289+0000 7f4177269700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4170065450 0x7f41700b45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:16.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.289+0000 7f4177269700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f41700b28e0 0x7f417012d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:16.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.289+0000 7f4177269700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41701412c0 con 0x7f4170065450 2022-06-29T16:08:16.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.289+0000 7f4177269700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4170150c40 con 0x7f4170060760 2022-06-29T16:08:16.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.289+0000 7f4177269700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4170140d80 con 0x7f41700b28e0 2022-06-29T16:08:16.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.289+0000 7f4175a66700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4170065450 0x7f41700b45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:16.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.289+0000 7f4175a66700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4170065450 0x7f41700b45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54304/0 (socket says 172.21.15.60:54304) 2022-06-29T16:08:16.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.289+0000 7f4176267700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4170060760 0x7f41701421e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:16.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.289+0000 7f4175a66700 1 -- 172.21.15.60:0/1079053610 learned_addr learned my addr 172.21.15.60:0/1079053610 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:16.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.289+0000 7f4176a68700 1 --2- 172.21.15.60:0/1079053610 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f41700b28e0 0x7f417012d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:16.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.289+0000 7f4175a66700 1 -- 172.21.15.60:0/1079053610 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f41700b28e0 msgr2=0x7f417012d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:16.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.289+0000 7f4175a66700 1 --2- 172.21.15.60:0/1079053610 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f41700b28e0 0x7f417012d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:16.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.289+0000 7f4175a66700 1 -- 172.21.15.60:0/1079053610 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4170060760 msgr2=0x7f41701421e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:16.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.289+0000 7f4175a66700 1 --2- 172.21.15.60:0/1079053610 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4170060760 0x7f41701421e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:16.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.289+0000 7f4175a66700 1 -- 172.21.15.60:0/1079053610 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4168009530 con 0x7f4170065450 2022-06-29T16:08:16.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.289+0000 7f4175a66700 1 --2- 172.21.15.60:0/1079053610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4170065450 0x7f41700b45a0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 rx=0x7f416c002f70 tx=0x7f416c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:16.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.289+0000 7f41637fe700 1 -- 172.21.15.60:0/1079053610 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f416c010de0 con 0x7f4170065450 2022-06-29T16:08:16.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.289+0000 7f41637fe700 1 -- 172.21.15.60:0/1079053610 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f416c01d450 con 0x7f4170065450 2022-06-29T16:08:16.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.289+0000 7f4177269700 1 -- 172.21.15.60:0/1079053610 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f416c00b180 con 0x7f4170065450 2022-06-29T16:08:16.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.289+0000 7f41637fe700 1 -- 172.21.15.60:0/1079053610 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f416c01d7b0 con 0x7f4170065450 2022-06-29T16:08:16.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.289+0000 7f4177269700 1 -- 172.21.15.60:0/1079053610 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41700b4a90 con 0x7f4170065450 2022-06-29T16:08:16.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.293+0000 7f4177269700 1 -- 172.21.15.60:0/1079053610 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f417013cb30 con 0x7f4170065450 2022-06-29T16:08:16.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.293+0000 7f41637fe700 1 -- 172.21.15.60:0/1079053610 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f416c01d910 con 0x7f4170065450 2022-06-29T16:08:16.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.293+0000 7f41637fe700 1 --2- 172.21.15.60:0/1079053610 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4150053a10 0x7f4150055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:16.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.293+0000 7f41637fe700 1 -- 172.21.15.60:0/1079053610 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f416c06fda0 con 0x7f4170065450 2022-06-29T16:08:16.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.293+0000 7f4176267700 1 --2- 172.21.15.60:0/1079053610 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4150053a10 0x7f4150055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:16.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.293+0000 7f4176267700 1 --2- 172.21.15.60:0/1079053610 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4150053a10 0x7f4150055ed0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0x7f416400d000 tx=0x7f4164007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:16.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.301+0000 7f41637fe700 1 -- 172.21.15.60:0/1079053610 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f416c046e90 con 0x7f4170065450 2022-06-29T16:08:16.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.429+0000 7f4177269700 1 -- 172.21.15.60:0/1079053610 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f41700573d0 con 0x7f4170065450 2022-06-29T16:08:16.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.757+0000 7f41637fe700 1 -- 172.21.15.60:0/1079053610 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v143) v1 ==== 125+0+0 (secure 0 0 0) 0x7f416c006980 con 0x7f4170065450 2022-06-29T16:08:16.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.765+0000 7f4177269700 1 -- 172.21.15.60:0/1079053610 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4150053a10 msgr2=0x7f4150055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:16.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.765+0000 7f4177269700 1 --2- 172.21.15.60:0/1079053610 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4150053a10 0x7f4150055ed0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0x7f416400d000 tx=0x7f4164007b90).stop 2022-06-29T16:08:16.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.765+0000 7f4177269700 1 -- 172.21.15.60:0/1079053610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4170065450 msgr2=0x7f41700b45a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:16.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.765+0000 7f4177269700 1 --2- 172.21.15.60:0/1079053610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4170065450 0x7f41700b45a0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 rx=0x7f416c002f70 tx=0x7f416c016000).stop 2022-06-29T16:08:16.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.765+0000 7f4177269700 1 -- 172.21.15.60:0/1079053610 shutdown_connections 2022-06-29T16:08:16.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.765+0000 7f4177269700 1 --2- 172.21.15.60:0/1079053610 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4150053a10 0x7f4150055ed0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:16.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.765+0000 7f4177269700 1 --2- 172.21.15.60:0/1079053610 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f41700b28e0 0x7f417012d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:16.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.765+0000 7f4177269700 1 --2- 172.21.15.60:0/1079053610 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4170065450 0x7f41700b45a0 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:16.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.765+0000 7f4177269700 1 --2- 172.21.15.60:0/1079053610 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4170060760 0x7f41701421e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:16.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.765+0000 7f4177269700 1 -- 172.21.15.60:0/1079053610 >> 172.21.15.60:0/1079053610 conn(0x7f4170061c80 msgr2=0x7f41700b0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:16.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.765+0000 7f4177269700 1 -- 172.21.15.60:0/1079053610 shutdown_connections 2022-06-29T16:08:16.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.765+0000 7f4177269700 1 -- 172.21.15.60:0/1079053610 wait complete. 2022-06-29T16:08:16.775 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2022-06-29T16:08:16.785 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-06-29T16:08:16.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.889+0000 7f020684b700 1 Processor -- start 2022-06-29T16:08:16.895 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.889+0000 7f020684b700 1 -- start start 2022-06-29T16:08:16.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f020684b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0200142740 0x7f02000b51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:16.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f020684b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f02000b3d30 0x7f0200146010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:16.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f020684b700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02000afda0 con 0x7f02000b9bd0 2022-06-29T16:08:16.903 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f020684b700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02000b9fb0 con 0x7f02000b3d30 2022-06-29T16:08:16.903 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f020684b700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02000b05c0 con 0x7f0200142740 2022-06-29T16:08:16.903 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f0205849700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f02000b9bd0 0x7f0200142030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34164/0 (socket says 172.21.15.60:34164) 2022-06-29T16:08:16.903 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f0205849700 1 -- 172.21.15.60:0/952519372 learned_addr learned my addr 172.21.15.60:0/952519372 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:16.904 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f020604a700 1 --2- 172.21.15.60:0/952519372 >> v2:172.21.15.60:3300/0 conn(0x7f02000b3d30 0x7f0200146010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:16.904 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f0205048700 1 --2- 172.21.15.60:0/952519372 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0200142740 0x7f02000b51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:16.904 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f0204847700 1 -- 172.21.15.60:0/952519372 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776354773 0 0) 0x7f02000afda0 con 0x7f02000b9bd0 2022-06-29T16:08:16.904 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f0204847700 1 -- 172.21.15.60:0/952519372 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01e4003260 con 0x7f02000b9bd0 2022-06-29T16:08:16.904 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f0204847700 1 -- 172.21.15.60:0/952519372 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1725956967 0 0) 0x7f01e4003260 con 0x7f02000b9bd0 2022-06-29T16:08:16.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f0204847700 1 -- 172.21.15.60:0/952519372 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0200142740 msgr2=0x7f02000b51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:16.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f0204847700 1 --2- 172.21.15.60:0/952519372 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0200142740 0x7f02000b51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:16.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f0204847700 1 -- 172.21.15.60:0/952519372 >> v2:172.21.15.60:3300/0 conn(0x7f02000b3d30 msgr2=0x7f0200146010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:16.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f0204847700 1 --2- 172.21.15.60:0/952519372 >> v2:172.21.15.60:3300/0 conn(0x7f02000b3d30 0x7f0200146010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:16.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f0204847700 1 -- 172.21.15.60:0/952519372 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02000affd0 con 0x7f02000b9bd0 2022-06-29T16:08:16.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f0204847700 1 -- 172.21.15.60:0/952519372 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f01fc0017e0 con 0x7f02000b9bd0 2022-06-29T16:08:16.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f0204847700 1 -- 172.21.15.60:0/952519372 >> v1:172.21.15.60:6789/0 conn(0x7f02000b9bd0 legacy=0x7f0200142030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:16.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f0204847700 1 --2- 172.21.15.60:0/952519372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f01e4003750 0x7f01e4003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:16.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f0204847700 1 --2- 172.21.15.60:0/952519372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f01e4004500 0x7f01e4004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:16.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f0204847700 1 --2- 172.21.15.60:0/952519372 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0200142740 0x7f01e4004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:16.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f0204847700 1 -- 172.21.15.60:0/952519372 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02000b9fb0 con 0x7f01e4003750 2022-06-29T16:08:16.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f0204847700 1 -- 172.21.15.60:0/952519372 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02000b05c0 con 0x7f01e4004500 2022-06-29T16:08:16.907 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f0204847700 1 -- 172.21.15.60:0/952519372 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01e400b750 con 0x7f0200142740 2022-06-29T16:08:16.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f020604a700 1 --2- 172.21.15.60:0/952519372 >> v2:172.21.15.60:3300/0 conn(0x7f02000b3d30 0x7f0200146010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:08:16.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f0205849700 1 --2- 172.21.15.60:0/952519372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f01e4003750 0x7f01e4003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:16.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f020604a700 1 --2- 172.21.15.60:0/952519372 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0200142740 0x7f01e4004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:16.908 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f0205048700 1 --2- 172.21.15.60:0/952519372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f01e4004500 0x7f01e4004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:16.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f0205849700 1 -- 172.21.15.60:0/952519372 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0200142740 msgr2=0x7f01e4004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:16.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f0205849700 1 --2- 172.21.15.60:0/952519372 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0200142740 0x7f01e4004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:16.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f0205849700 1 -- 172.21.15.60:0/952519372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f01e4004500 msgr2=0x7f01e4004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:16.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f0205849700 1 --2- 172.21.15.60:0/952519372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f01e4004500 0x7f01e4004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:16.909 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f0205849700 1 -- 172.21.15.60:0/952519372 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e4003260 con 0x7f01e4003750 2022-06-29T16:08:16.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f0205048700 1 --2- 172.21.15.60:0/952519372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f01e4004500 0x7f01e4004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:08:16.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f0205849700 1 --2- 172.21.15.60:0/952519372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f01e4003750 0x7f01e4003b70 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 rx=0x7f01fc0014f0 tx=0x7f02000b0160).ready entity=mon.0 client_cookie=2387c1be3d62e2aa server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:16.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f0204847700 1 -- 172.21.15.60:0/952519372 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f01fc0070d0 con 0x7f01e4003750 2022-06-29T16:08:16.910 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f0204847700 1 -- 172.21.15.60:0/952519372 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01fc007230 con 0x7f01e4003750 2022-06-29T16:08:16.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f0204847700 1 -- 172.21.15.60:0/952519372 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f01fc0073a0 con 0x7f01e4003750 2022-06-29T16:08:16.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f020684b700 1 -- 172.21.15.60:0/952519372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f01e4003750 msgr2=0x7f01e4003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:16.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f020684b700 1 --2- 172.21.15.60:0/952519372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f01e4003750 0x7f01e4003b70 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 rx=0x7f01fc0014f0 tx=0x7f02000b0160).stop 2022-06-29T16:08:16.911 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.893+0000 7f020604a700 1 -- 172.21.15.60:0/952519372 reap_dead start 2022-06-29T16:08:16.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.897+0000 7f020684b700 1 -- 172.21.15.60:0/952519372 shutdown_connections 2022-06-29T16:08:16.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.897+0000 7f020684b700 1 -- 172.21.15.60:0/952519372 >> 172.21.15.60:0/952519372 conn(0x7f0200062cb0 msgr2=0x7f02000b9180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:16.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.897+0000 7f020684b700 1 -- 172.21.15.60:0/952519372 shutdown_connections 2022-06-29T16:08:16.912 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.897+0000 7f020684b700 1 -- 172.21.15.60:0/952519372 wait complete. 2022-06-29T16:08:16.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.897+0000 7f020684b700 1 Processor -- start 2022-06-29T16:08:16.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.897+0000 7f020684b700 1 -- start start 2022-06-29T16:08:16.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.901+0000 7f020684b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f01e4003750 0x7f02000af4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:16.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.901+0000 7f020684b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0200144a20 0x7f0200152a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:16.917 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.901+0000 7f020684b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0200144ed0 0x7f02000aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:16.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.901+0000 7f020684b700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0200151e20 con 0x7f0200144a20 2022-06-29T16:08:16.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.901+0000 7f020684b700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0200152060 con 0x7f01e4003750 2022-06-29T16:08:16.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.901+0000 7f020684b700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0200152300 con 0x7f0200144ed0 2022-06-29T16:08:16.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.901+0000 7f0205048700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0200144a20 0x7f0200152a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:16.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.901+0000 7f0205048700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0200144a20 0x7f0200152a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54324/0 (socket says 172.21.15.60:54324) 2022-06-29T16:08:16.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.901+0000 7f0205048700 1 -- 172.21.15.60:0/1477111104 learned_addr learned my addr 172.21.15.60:0/1477111104 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:16.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.901+0000 7f0205849700 1 --2- 172.21.15.60:0/1477111104 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f01e4003750 0x7f02000af4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:16.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.901+0000 7f020604a700 1 --2- 172.21.15.60:0/1477111104 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0200144ed0 0x7f02000aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:16.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.901+0000 7f0205048700 1 -- 172.21.15.60:0/1477111104 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0200144ed0 msgr2=0x7f02000aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:16.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.901+0000 7f0205048700 1 --2- 172.21.15.60:0/1477111104 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0200144ed0 0x7f02000aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:16.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.901+0000 7f0205048700 1 -- 172.21.15.60:0/1477111104 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f01e4003750 msgr2=0x7f02000af4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:16.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.901+0000 7f0205048700 1 --2- 172.21.15.60:0/1477111104 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f01e4003750 0x7f02000af4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:16.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.901+0000 7f0205048700 1 -- 172.21.15.60:0/1477111104 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02001525a0 con 0x7f0200144a20 2022-06-29T16:08:16.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.901+0000 7f0205849700 1 --2- 172.21.15.60:0/1477111104 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f01e4003750 0x7f02000af4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:08:16.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.901+0000 7f0205048700 1 --2- 172.21.15.60:0/1477111104 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0200144a20 0x7f0200152a40 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 rx=0x7f01f400b000 tx=0x7f01f4008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:16.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.901+0000 7f01f37fe700 1 -- 172.21.15.60:0/1477111104 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f01f400b3e0 con 0x7f0200144a20 2022-06-29T16:08:16.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.901+0000 7f01f37fe700 1 -- 172.21.15.60:0/1477111104 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01f400e850 con 0x7f0200144a20 2022-06-29T16:08:16.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.901+0000 7f020684b700 1 -- 172.21.15.60:0/1477111104 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0200152f30 con 0x7f0200144a20 2022-06-29T16:08:16.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.901+0000 7f01f37fe700 1 -- 172.21.15.60:0/1477111104 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f01f400ebb0 con 0x7f0200144a20 2022-06-29T16:08:16.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.901+0000 7f020684b700 1 -- 172.21.15.60:0/1477111104 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02000ad130 con 0x7f0200144a20 2022-06-29T16:08:16.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.901+0000 7f020684b700 1 -- 172.21.15.60:0/1477111104 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02000573d0 con 0x7f0200144a20 2022-06-29T16:08:16.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.905+0000 7f01f37fe700 1 -- 172.21.15.60:0/1477111104 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f01f400ed10 con 0x7f0200144a20 2022-06-29T16:08:16.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.905+0000 7f01f37fe700 1 --2- 172.21.15.60:0/1477111104 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f01dc053b00 0x7f01dc055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:16.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.905+0000 7f01f37fe700 1 -- 172.21.15.60:0/1477111104 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f01f406c470 con 0x7f0200144a20 2022-06-29T16:08:16.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.905+0000 7f0205849700 1 --2- 172.21.15.60:0/1477111104 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f01dc053b00 0x7f01dc055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:16.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.905+0000 7f0205849700 1 --2- 172.21.15.60:0/1477111104 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f01dc053b00 0x7f01dc055fc0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 rx=0x7f01fc018000 tx=0x7f01fc013000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:16.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:16.909+0000 7f01f37fe700 1 -- 172.21.15.60:0/1477111104 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f01f4043360 con 0x7f0200144a20 2022-06-29T16:08:17.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.073+0000 7f020684b700 1 -- 172.21.15.60:0/1477111104 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f02000b45e0 con 0x7f0200144a20 2022-06-29T16:08:17.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.081+0000 7f01f37fe700 1 -- 172.21.15.60:0/1477111104 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 29 and data pool 30 v72) v1 ==== 189+0+0 (secure 0 0 0) 0x7f01f4046980 con 0x7f0200144a20 2022-06-29T16:08:17.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.085+0000 7f020684b700 1 -- 172.21.15.60:0/1477111104 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f01dc053b00 msgr2=0x7f01dc055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:17.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.085+0000 7f020684b700 1 --2- 172.21.15.60:0/1477111104 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f01dc053b00 0x7f01dc055fc0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 rx=0x7f01fc018000 tx=0x7f01fc013000).stop 2022-06-29T16:08:17.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.085+0000 7f020684b700 1 -- 172.21.15.60:0/1477111104 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0200144a20 msgr2=0x7f0200152a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:17.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.085+0000 7f020684b700 1 --2- 172.21.15.60:0/1477111104 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0200144a20 0x7f0200152a40 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 rx=0x7f01f400b000 tx=0x7f01f4008040).stop 2022-06-29T16:08:17.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.085+0000 7f020684b700 1 -- 172.21.15.60:0/1477111104 shutdown_connections 2022-06-29T16:08:17.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.085+0000 7f020684b700 1 --2- 172.21.15.60:0/1477111104 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f01dc053b00 0x7f01dc055fc0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:17.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.085+0000 7f020684b700 1 --2- 172.21.15.60:0/1477111104 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0200144ed0 0x7f02000aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:17.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.085+0000 7f020684b700 1 --2- 172.21.15.60:0/1477111104 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0200144a20 0x7f0200152a40 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:17.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.085+0000 7f020684b700 1 --2- 172.21.15.60:0/1477111104 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f01e4003750 0x7f02000af4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:17.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.085+0000 7f020684b700 1 -- 172.21.15.60:0/1477111104 >> 172.21.15.60:0/1477111104 conn(0x7f0200062cb0 msgr2=0x7f0200141e00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:17.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.085+0000 7f020684b700 1 -- 172.21.15.60:0/1477111104 shutdown_connections 2022-06-29T16:08:17.094 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.085+0000 7f020684b700 1 -- 172.21.15.60:0/1477111104 wait complete. 2022-06-29T16:08:17.095 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 29 and data pool 30 2022-06-29T16:08:17.105 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:08:17.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.193+0000 7fb05d110700 1 Processor -- start 2022-06-29T16:08:17.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.193+0000 7fb05d110700 1 -- start start 2022-06-29T16:08:17.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.193+0000 7fb05d110700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb058060260 0x7fb0580b6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:17.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.193+0000 7fb05d110700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0581430e0 con 0x7fb05813f3d0 2022-06-29T16:08:17.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.193+0000 7fb05d110700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb058143380 con 0x7fb0580b6c00 2022-06-29T16:08:17.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.193+0000 7fb05d110700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb058142ef0 con 0x7fb058060260 2022-06-29T16:08:17.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.193+0000 7fb056d9d700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb05813f3d0 0x7fb058143820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34192/0 (socket says 172.21.15.60:34192) 2022-06-29T16:08:17.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.193+0000 7fb056d9d700 1 -- 172.21.15.60:0/3750340288 learned_addr learned my addr 172.21.15.60:0/3750340288 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:17.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.193+0000 7fb05659c700 1 --2- 172.21.15.60:0/3750340288 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb058060260 0x7fb0580b6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:17.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.193+0000 7fb055d9b700 1 -- 172.21.15.60:0/3750340288 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3944365150 0 0) 0x7fb0581430e0 con 0x7fb05813f3d0 2022-06-29T16:08:17.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.193+0000 7fb055d9b700 1 -- 172.21.15.60:0/3750340288 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb038003260 con 0x7fb05813f3d0 2022-06-29T16:08:17.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb055d9b700 1 -- 172.21.15.60:0/3750340288 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1923881440 0 0) 0x7fb058143380 con 0x7fb0580b6c00 2022-06-29T16:08:17.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb055d9b700 1 -- 172.21.15.60:0/3750340288 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0581430e0 con 0x7fb0580b6c00 2022-06-29T16:08:17.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb055d9b700 1 -- 172.21.15.60:0/3750340288 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1045735560 0 0) 0x7fb038003260 con 0x7fb05813f3d0 2022-06-29T16:08:17.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb055d9b700 1 -- 172.21.15.60:0/3750340288 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb058060260 msgr2=0x7fb0580b6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:17.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb055d9b700 1 --2- 172.21.15.60:0/3750340288 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb058060260 0x7fb0580b6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:17.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb055d9b700 1 -- 172.21.15.60:0/3750340288 >> v1:172.21.15.92:6789/0 conn(0x7fb0580b6c00 legacy=0x7fb05813f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:17.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb055d9b700 1 -- 172.21.15.60:0/3750340288 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb05805fed0 con 0x7fb05813f3d0 2022-06-29T16:08:17.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb055d9b700 1 -- 172.21.15.60:0/3750340288 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7fb048001660 con 0x7fb05813f3d0 2022-06-29T16:08:17.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb055d9b700 1 -- 172.21.15.60:0/3750340288 >> v1:172.21.15.60:6789/0 conn(0x7fb05813f3d0 legacy=0x7fb058143820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:17.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb055d9b700 1 --2- 172.21.15.60:0/3750340288 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb038003910 0x7fb038003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:17.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb055d9b700 1 --2- 172.21.15.60:0/3750340288 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb058060260 0x7fb038004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:17.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb055d9b700 1 --2- 172.21.15.60:0/3750340288 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0380049f0 0x7fb038004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:17.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb055d9b700 1 -- 172.21.15.60:0/3750340288 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb03800d960 con 0x7fb038003910 2022-06-29T16:08:17.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb055d9b700 1 -- 172.21.15.60:0/3750340288 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb03800dad0 con 0x7fb0380049f0 2022-06-29T16:08:17.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb055d9b700 1 -- 172.21.15.60:0/3750340288 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb03800dc90 con 0x7fb058060260 2022-06-29T16:08:17.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb056d9d700 1 --2- 172.21.15.60:0/3750340288 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb038003910 0x7fb038003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:17.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb05759e700 1 --2- 172.21.15.60:0/3750340288 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0380049f0 0x7fb038004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:17.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb05659c700 1 --2- 172.21.15.60:0/3750340288 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb058060260 0x7fb038004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:17.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb056d9d700 1 -- 172.21.15.60:0/3750340288 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb058060260 msgr2=0x7fb038004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:17.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb056d9d700 1 --2- 172.21.15.60:0/3750340288 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb058060260 0x7fb038004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:17.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb056d9d700 1 -- 172.21.15.60:0/3750340288 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0380049f0 msgr2=0x7fb038004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:17.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb056d9d700 1 --2- 172.21.15.60:0/3750340288 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0380049f0 0x7fb038004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:17.216 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb056d9d700 1 -- 172.21.15.60:0/3750340288 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb038003260 con 0x7fb038003910 2022-06-29T16:08:17.216 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb056d9d700 1 --2- 172.21.15.60:0/3750340288 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb038003910 0x7fb038003d10 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 rx=0x7fb048001430 tx=0x7fb05805fc60).ready entity=mon.0 client_cookie=6afe8c1aaf47c527 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:17.216 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb055d9b700 1 -- 172.21.15.60:0/3750340288 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0480070d0 con 0x7fb038003910 2022-06-29T16:08:17.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb055d9b700 1 -- 172.21.15.60:0/3750340288 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb048007230 con 0x7fb038003910 2022-06-29T16:08:17.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb055d9b700 1 -- 172.21.15.60:0/3750340288 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb04800c8f0 con 0x7fb038003910 2022-06-29T16:08:17.217 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb05d110700 1 -- 172.21.15.60:0/3750340288 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb038003910 msgr2=0x7fb038003d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:17.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb05d110700 1 --2- 172.21.15.60:0/3750340288 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb038003910 0x7fb038003d10 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 rx=0x7fb048001430 tx=0x7fb05805fc60).stop 2022-06-29T16:08:17.218 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb05759e700 1 -- 172.21.15.60:0/3750340288 reap_dead start 2022-06-29T16:08:17.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb05d110700 1 -- 172.21.15.60:0/3750340288 shutdown_connections 2022-06-29T16:08:17.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb05d110700 1 -- 172.21.15.60:0/3750340288 >> 172.21.15.60:0/3750340288 conn(0x7fb0580643e0 msgr2=0x7fb0580b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:17.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb05d110700 1 -- 172.21.15.60:0/3750340288 shutdown_connections 2022-06-29T16:08:17.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.197+0000 7fb05d110700 1 -- 172.21.15.60:0/3750340288 wait complete. 2022-06-29T16:08:17.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.201+0000 7fb05d110700 1 Processor -- start 2022-06-29T16:08:17.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.201+0000 7fb05d110700 1 -- start start 2022-06-29T16:08:17.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.201+0000 7fb05d110700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb038003910 0x7fb0580669f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:17.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.201+0000 7fb05d110700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0581455b0 0x7fb058140df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:17.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.201+0000 7fb05d110700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb058145f70 0x7fb058146400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:17.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.201+0000 7fb05d110700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb058145e00 con 0x7fb058145f70 2022-06-29T16:08:17.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.201+0000 7fb05d110700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0580667d0 con 0x7fb0581455b0 2022-06-29T16:08:17.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.201+0000 7fb05d110700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb058097d30 con 0x7fb038003910 2022-06-29T16:08:17.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.201+0000 7fb05759e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb058145f70 0x7fb058146400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:17.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.201+0000 7fb05759e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb058145f70 0x7fb058146400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54354/0 (socket says 172.21.15.60:54354) 2022-06-29T16:08:17.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.201+0000 7fb05659c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0581455b0 0x7fb058140df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:17.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.201+0000 7fb05759e700 1 -- 172.21.15.60:0/834163706 learned_addr learned my addr 172.21.15.60:0/834163706 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:17.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.201+0000 7fb056d9d700 1 --2- 172.21.15.60:0/834163706 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb038003910 0x7fb0580669f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:17.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.201+0000 7fb05759e700 1 -- 172.21.15.60:0/834163706 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb038003910 msgr2=0x7fb0580669f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:17.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.201+0000 7fb05759e700 1 --2- 172.21.15.60:0/834163706 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb038003910 0x7fb0580669f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:17.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.201+0000 7fb05759e700 1 -- 172.21.15.60:0/834163706 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0581455b0 msgr2=0x7fb058140df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:17.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.205+0000 7fb05759e700 1 --2- 172.21.15.60:0/834163706 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0581455b0 0x7fb058140df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:17.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.205+0000 7fb05759e700 1 -- 172.21.15.60:0/834163706 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb058066ee0 con 0x7fb058145f70 2022-06-29T16:08:17.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.205+0000 7fb05759e700 1 --2- 172.21.15.60:0/834163706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb058145f70 0x7fb058146400 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 rx=0x7fb050000b60 tx=0x7fb050007d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:17.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.205+0000 7fb04ffff700 1 -- 172.21.15.60:0/834163706 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb050010710 con 0x7fb058145f70 2022-06-29T16:08:17.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.205+0000 7fb04ffff700 1 -- 172.21.15.60:0/834163706 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb050010870 con 0x7fb058145f70 2022-06-29T16:08:17.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.205+0000 7fb05d110700 1 -- 172.21.15.60:0/834163706 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb058067300 con 0x7fb058145f70 2022-06-29T16:08:17.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.205+0000 7fb04ffff700 1 -- 172.21.15.60:0/834163706 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb050010bd0 con 0x7fb058145f70 2022-06-29T16:08:17.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.205+0000 7fb05d110700 1 -- 172.21.15.60:0/834163706 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb058145bd0 con 0x7fb058145f70 2022-06-29T16:08:17.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.205+0000 7fb04ffff700 1 -- 172.21.15.60:0/834163706 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fb05001b030 con 0x7fb058145f70 2022-06-29T16:08:17.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.205+0000 7fb05d110700 1 -- 172.21.15.60:0/834163706 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb058097ec0 con 0x7fb058145f70 2022-06-29T16:08:17.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.205+0000 7fb04ffff700 1 --2- 172.21.15.60:0/834163706 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb03c053b00 0x7fb03c055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:17.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.205+0000 7fb04ffff700 1 -- 172.21.15.60:0/834163706 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb05006bc40 con 0x7fb058145f70 2022-06-29T16:08:17.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.205+0000 7fb056d9d700 1 --2- 172.21.15.60:0/834163706 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb03c053b00 0x7fb03c055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:17.230 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.209+0000 7fb056d9d700 1 --2- 172.21.15.60:0/834163706 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb03c053b00 0x7fb03c055fc0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0x7fb04801a000 tx=0x7fb048015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:17.230 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.213+0000 7fb04ffff700 1 -- 172.21.15.60:0/834163706 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb050042b60 con 0x7fb058145f70 2022-06-29T16:08:17.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.341+0000 7fb05d110700 1 -- 172.21.15.60:0/834163706 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0580573d0 con 0x7fb058145f70 2022-06-29T16:08:17.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.345+0000 7fb04ffff700 1 -- 172.21.15.60:0/834163706 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v145) v1 ==== 74+0+16078 (secure 0 0 0) 0x7fb050046180 con 0x7fb058145f70 2022-06-29T16:08:17.349 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:08:17.350 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":145,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:08:17.075612+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:08:15.820434+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":30,"pool_name":"cephfs_data","create_time":"2022-06-29T16:08:16.484716+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:08:17.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.349+0000 7fb05d110700 1 -- 172.21.15.60:0/834163706 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb03c053b00 msgr2=0x7fb03c055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:17.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.349+0000 7fb05d110700 1 --2- 172.21.15.60:0/834163706 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb03c053b00 0x7fb03c055fc0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0x7fb04801a000 tx=0x7fb048015000).stop 2022-06-29T16:08:17.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.349+0000 7fb05d110700 1 -- 172.21.15.60:0/834163706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb058145f70 msgr2=0x7fb058146400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:17.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.349+0000 7fb05d110700 1 --2- 172.21.15.60:0/834163706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb058145f70 0x7fb058146400 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 rx=0x7fb050000b60 tx=0x7fb050007d80).stop 2022-06-29T16:08:17.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.349+0000 7fb05d110700 1 -- 172.21.15.60:0/834163706 shutdown_connections 2022-06-29T16:08:17.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.349+0000 7fb05d110700 1 --2- 172.21.15.60:0/834163706 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fb03c053b00 0x7fb03c055fc0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:17.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.349+0000 7fb05d110700 1 --2- 172.21.15.60:0/834163706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb058145f70 0x7fb058146400 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:17.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.349+0000 7fb05d110700 1 --2- 172.21.15.60:0/834163706 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0581455b0 0x7fb058140df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:17.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.349+0000 7fb05d110700 1 --2- 172.21.15.60:0/834163706 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fb038003910 0x7fb0580669f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:17.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.349+0000 7fb05d110700 1 -- 172.21.15.60:0/834163706 >> 172.21.15.60:0/834163706 conn(0x7fb0580643e0 msgr2=0x7fb058061920 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:17.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.349+0000 7fb05d110700 1 -- 172.21.15.60:0/834163706 shutdown_connections 2022-06-29T16:08:17.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.349+0000 7fb05d110700 1 -- 172.21.15.60:0/834163706 wait complete. 2022-06-29T16:08:17.367 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:08:17.470 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.465+0000 7f7b10a7e700 1 Processor -- start 2022-06-29T16:08:17.471 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.465+0000 7f7b10a7e700 1 -- start start 2022-06-29T16:08:17.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.465+0000 7f7b10a7e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7b0c0b48b0 0x7f7b0c142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:17.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.465+0000 7f7b10a7e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7b0c0b2c50 0x7f7b0c0b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:17.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.465+0000 7f7b10a7e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b0c0b0cf0 con 0x7f7b0c0b5eb0 2022-06-29T16:08:17.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.465+0000 7f7b10a7e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b0c0b54e0 con 0x7f7b0c0b48b0 2022-06-29T16:08:17.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.465+0000 7f7b10a7e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b0c0b4d10 con 0x7f7b0c0b2c50 2022-06-29T16:08:17.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.465+0000 7f7b0affd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f7b0c0b48b0 0x7f7b0c142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:17.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.465+0000 7f7b0b7fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f7b0c0b5eb0 0x7f7b0c0b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34212/0 (socket says 172.21.15.60:34212) 2022-06-29T16:08:17.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.465+0000 7f7b0b7fe700 1 -- 172.21.15.60:0/852647704 learned_addr learned my addr 172.21.15.60:0/852647704 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:17.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.465+0000 7f7b0bfff700 1 --2- 172.21.15.60:0/852647704 >> v2:172.21.15.92:3300/0 conn(0x7f7b0c0b2c50 0x7f7b0c0b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:17.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.469+0000 7f7b0affd700 1 -- 172.21.15.60:0/852647704 >> v2:172.21.15.92:3300/0 conn(0x7f7b0c0b2c50 msgr2=0x7f7b0c0b30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:17.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.469+0000 7f7b0affd700 1 --2- 172.21.15.60:0/852647704 >> v2:172.21.15.92:3300/0 conn(0x7f7b0c0b2c50 0x7f7b0c0b30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:17.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.469+0000 7f7b0affd700 1 -- 172.21.15.60:0/852647704 >> v1:172.21.15.60:6789/0 conn(0x7f7b0c0b5eb0 legacy=0x7f7b0c0b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:17.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.469+0000 7f7b0affd700 1 -- 172.21.15.60:0/852647704 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b0c0b6290 con 0x7f7b0c0b48b0 2022-06-29T16:08:17.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.469+0000 7f7b0bfff700 1 --2- 172.21.15.60:0/852647704 >> v2:172.21.15.92:3300/0 conn(0x7f7b0c0b2c50 0x7f7b0c0b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:08:17.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.469+0000 7f7b0affd700 1 --2- 172.21.15.60:0/852647704 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b0c0b48b0 0x7f7b0c142a20 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 rx=0x7f7b00002f70 tx=0x7f7b00006df0).ready entity=mon.0 client_cookie=e0041a65dab549c2 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:17.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.469+0000 7f7b0a7fc700 1 -- 172.21.15.60:0/852647704 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7b0000d5c0 con 0x7f7b0c0b48b0 2022-06-29T16:08:17.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.469+0000 7f7b0a7fc700 1 -- 172.21.15.60:0/852647704 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b00000e90 con 0x7f7b0c0b48b0 2022-06-29T16:08:17.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.469+0000 7f7b0a7fc700 1 -- 172.21.15.60:0/852647704 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7b0000d910 con 0x7f7b0c0b48b0 2022-06-29T16:08:17.486 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.469+0000 7f7b10a7e700 1 -- 172.21.15.60:0/852647704 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b0c0b48b0 msgr2=0x7f7b0c142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:17.486 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.469+0000 7f7b10a7e700 1 --2- 172.21.15.60:0/852647704 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b0c0b48b0 0x7f7b0c142a20 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 rx=0x7f7b00002f70 tx=0x7f7b00006df0).stop 2022-06-29T16:08:17.487 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.469+0000 7f7b10a7e700 1 -- 172.21.15.60:0/852647704 shutdown_connections 2022-06-29T16:08:17.487 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.469+0000 7f7b10a7e700 1 --2- 172.21.15.60:0/852647704 >> v2:172.21.15.92:3300/0 conn(0x7f7b0c0b2c50 0x7f7b0c0b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:17.487 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.469+0000 7f7b10a7e700 1 --2- 172.21.15.60:0/852647704 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b0c0b48b0 0x7f7b0c142a20 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:17.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.469+0000 7f7b10a7e700 1 -- 172.21.15.60:0/852647704 >> 172.21.15.60:0/852647704 conn(0x7f7b0c062090 msgr2=0x7f7b0c062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:17.488 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.469+0000 7f7b10a7e700 1 -- 172.21.15.60:0/852647704 shutdown_connections 2022-06-29T16:08:17.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.469+0000 7f7b10a7e700 1 -- 172.21.15.60:0/852647704 wait complete. 2022-06-29T16:08:17.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.469+0000 7f7b10a7e700 1 Processor -- start 2022-06-29T16:08:17.489 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.469+0000 7f7b10a7e700 1 -- start start 2022-06-29T16:08:17.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.473+0000 7f7b10a7e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7b0c0b2c50 0x7f7b0c140440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:17.490 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.473+0000 7f7b10a7e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7b0c0b48b0 0x7f7b0c13cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:17.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.473+0000 7f7b10a7e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b0c0b5eb0 0x7f7b0c143ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:17.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.473+0000 7f7b10a7e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0c0af890 con 0x7f7b0c0b5eb0 2022-06-29T16:08:17.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.473+0000 7f7b10a7e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0c13d6c0 con 0x7f7b0c0b48b0 2022-06-29T16:08:17.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.473+0000 7f7b10a7e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0c0af5f0 con 0x7f7b0c0b2c50 2022-06-29T16:08:17.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.473+0000 7f7b0bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b0c0b5eb0 0x7f7b0c143ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:17.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.473+0000 7f7b0bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b0c0b5eb0 0x7f7b0c143ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54368/0 (socket says 172.21.15.60:54368) 2022-06-29T16:08:17.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.473+0000 7f7b0bfff700 1 -- 172.21.15.60:0/2886583094 learned_addr learned my addr 172.21.15.60:0/2886583094 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:17.492 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.473+0000 7f7b0affd700 1 --2- 172.21.15.60:0/2886583094 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7b0c0b48b0 0x7f7b0c13cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:17.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.473+0000 7f7b0b7fe700 1 --2- 172.21.15.60:0/2886583094 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7b0c0b2c50 0x7f7b0c140440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:17.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.473+0000 7f7b0bfff700 1 -- 172.21.15.60:0/2886583094 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7b0c0b2c50 msgr2=0x7f7b0c140440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:17.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.473+0000 7f7b0bfff700 1 --2- 172.21.15.60:0/2886583094 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7b0c0b2c50 0x7f7b0c140440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:17.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.473+0000 7f7b0bfff700 1 -- 172.21.15.60:0/2886583094 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7b0c0b48b0 msgr2=0x7f7b0c13cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:17.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.473+0000 7f7b0bfff700 1 --2- 172.21.15.60:0/2886583094 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7b0c0b48b0 0x7f7b0c13cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:17.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.473+0000 7f7b0bfff700 1 -- 172.21.15.60:0/2886583094 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b0000b180 con 0x7f7b0c0b5eb0 2022-06-29T16:08:17.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.473+0000 7f7b0affd700 1 --2- 172.21.15.60:0/2886583094 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7b0c0b48b0 0x7f7b0c13cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:08:17.494 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.473+0000 7f7b0bfff700 1 --2- 172.21.15.60:0/2886583094 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b0c0b5eb0 0x7f7b0c143ee0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 rx=0x7f7afc006940 tx=0x7f7afc008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:17.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.473+0000 7f7b08ff9700 1 -- 172.21.15.60:0/2886583094 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7afc011780 con 0x7f7b0c0b5eb0 2022-06-29T16:08:17.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.473+0000 7f7b08ff9700 1 -- 172.21.15.60:0/2886583094 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7afc0118e0 con 0x7f7b0c0b5eb0 2022-06-29T16:08:17.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.473+0000 7f7b10a7e700 1 -- 172.21.15.60:0/2886583094 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b0c0acb30 con 0x7f7b0c0b5eb0 2022-06-29T16:08:17.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.473+0000 7f7b08ff9700 1 -- 172.21.15.60:0/2886583094 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7afc011c40 con 0x7f7b0c0b5eb0 2022-06-29T16:08:17.495 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.473+0000 7f7b10a7e700 1 -- 172.21.15.60:0/2886583094 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b0c0acfd0 con 0x7f7b0c0b5eb0 2022-06-29T16:08:17.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.473+0000 7f7b10a7e700 1 -- 172.21.15.60:0/2886583094 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b0c0af410 con 0x7f7b0c0b5eb0 2022-06-29T16:08:17.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.477+0000 7f7b08ff9700 1 -- 172.21.15.60:0/2886583094 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f7afc01c460 con 0x7f7b0c0b5eb0 2022-06-29T16:08:17.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.477+0000 7f7b08ff9700 1 --2- 172.21.15.60:0/2886583094 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7aec053b00 0x7f7aec055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:17.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.477+0000 7f7b08ff9700 1 -- 172.21.15.60:0/2886583094 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7afc06c1a0 con 0x7f7b0c0b5eb0 2022-06-29T16:08:17.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.477+0000 7f7b0b7fe700 1 --2- 172.21.15.60:0/2886583094 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7aec053b00 0x7f7aec055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:17.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.477+0000 7f7b0b7fe700 1 --2- 172.21.15.60:0/2886583094 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7aec053b00 0x7f7aec055fc0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7f7b0400b000 tx=0x7f7b04009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:17.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.481+0000 7f7b08ff9700 1 -- 172.21.15.60:0/2886583094 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7afc043060 con 0x7f7b0c0b5eb0 2022-06-29T16:08:17.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.605+0000 7f7b10a7e700 1 -- 172.21.15.60:0/2886583094 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b0c0573d0 con 0x7f7b0c0b5eb0 2022-06-29T16:08:17.611 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.609+0000 7f7b08ff9700 1 -- 172.21.15.60:0/2886583094 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v145) v1 ==== 74+0+16078 (secure 0 0 0) 0x7f7afc046680 con 0x7f7b0c0b5eb0 2022-06-29T16:08:17.612 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:08:17.612 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":145,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:08:17.075612+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:08:15.820434+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":30,"pool_name":"cephfs_data","create_time":"2022-06-29T16:08:16.484716+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:08:17.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.609+0000 7f7b10a7e700 1 -- 172.21.15.60:0/2886583094 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7aec053b00 msgr2=0x7f7aec055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:17.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.609+0000 7f7b10a7e700 1 --2- 172.21.15.60:0/2886583094 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7aec053b00 0x7f7aec055fc0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7f7b0400b000 tx=0x7f7b04009040).stop 2022-06-29T16:08:17.615 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.609+0000 7f7b10a7e700 1 -- 172.21.15.60:0/2886583094 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b0c0b5eb0 msgr2=0x7f7b0c143ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:17.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.609+0000 7f7b10a7e700 1 --2- 172.21.15.60:0/2886583094 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b0c0b5eb0 0x7f7b0c143ee0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 rx=0x7f7afc006940 tx=0x7f7afc008040).stop 2022-06-29T16:08:17.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.609+0000 7f7b10a7e700 1 -- 172.21.15.60:0/2886583094 shutdown_connections 2022-06-29T16:08:17.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.609+0000 7f7b10a7e700 1 --2- 172.21.15.60:0/2886583094 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f7aec053b00 0x7f7aec055fc0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:17.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.609+0000 7f7b10a7e700 1 --2- 172.21.15.60:0/2886583094 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b0c0b5eb0 0x7f7b0c143ee0 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:17.616 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.609+0000 7f7b10a7e700 1 --2- 172.21.15.60:0/2886583094 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7b0c0b48b0 0x7f7b0c13cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:17.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.609+0000 7f7b10a7e700 1 --2- 172.21.15.60:0/2886583094 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f7b0c0b2c50 0x7f7b0c140440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:17.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.609+0000 7f7b10a7e700 1 -- 172.21.15.60:0/2886583094 >> 172.21.15.60:0/2886583094 conn(0x7f7b0c062090 msgr2=0x7f7b0c1435e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:17.617 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.609+0000 7f7b10a7e700 1 -- 172.21.15.60:0/2886583094 shutdown_connections 2022-06-29T16:08:17.618 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.609+0000 7f7b10a7e700 1 -- 172.21.15.60:0/2886583094 wait complete. 2022-06-29T16:08:17.626 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-06-29T16:08:17.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.733+0000 7f07a989b700 1 Processor -- start 2022-06-29T16:08:17.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.733+0000 7f07a989b700 1 -- start start 2022-06-29T16:08:17.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.733+0000 7f07a989b700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f07a40b9d70 0x7f07a41465d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:17.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.733+0000 7f07a989b700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a40b4f50 con 0x7f07a40b3d30 2022-06-29T16:08:17.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a989b700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a41461a0 con 0x7f07a4061d90 2022-06-29T16:08:17.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a989b700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07a40ba150 con 0x7f07a40b9d70 2022-06-29T16:08:17.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a27fc700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f07a40b3d30 0x7f07a4146ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34226/0 (socket says 172.21.15.60:34226) 2022-06-29T16:08:17.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a27fc700 1 -- 172.21.15.60:0/1282103887 learned_addr learned my addr 172.21.15.60:0/1282103887 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:17.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a2ffd700 1 --2- 172.21.15.60:0/1282103887 >> v2:172.21.15.92:3300/0 conn(0x7f07a40b9d70 0x7f07a41465d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:17.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a1ffb700 1 -- 172.21.15.60:0/1282103887 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1515815527 0 0) 0x7f07a40b4f50 con 0x7f07a40b3d30 2022-06-29T16:08:17.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a1ffb700 1 -- 172.21.15.60:0/1282103887 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0784003260 con 0x7f07a40b3d30 2022-06-29T16:08:17.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a1ffb700 1 -- 172.21.15.60:0/1282103887 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2448661441 0 0) 0x7f07a41461a0 con 0x7f07a4061d90 2022-06-29T16:08:17.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a1ffb700 1 -- 172.21.15.60:0/1282103887 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07a40b4f50 con 0x7f07a4061d90 2022-06-29T16:08:17.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a1ffb700 1 -- 172.21.15.60:0/1282103887 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3723866363 0 0) 0x7f0784003260 con 0x7f07a40b3d30 2022-06-29T16:08:17.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a1ffb700 1 -- 172.21.15.60:0/1282103887 >> v2:172.21.15.92:3300/0 conn(0x7f07a40b9d70 msgr2=0x7f07a41465d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:17.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a1ffb700 1 --2- 172.21.15.60:0/1282103887 >> v2:172.21.15.92:3300/0 conn(0x7f07a40b9d70 0x7f07a41465d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:17.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a1ffb700 1 -- 172.21.15.60:0/1282103887 >> v1:172.21.15.92:6789/0 conn(0x7f07a4061d90 legacy=0x7f07a4142b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:17.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a1ffb700 1 -- 172.21.15.60:0/1282103887 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a40b5180 con 0x7f07a40b3d30 2022-06-29T16:08:17.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a1ffb700 1 -- 172.21.15.60:0/1282103887 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f078c001660 con 0x7f07a40b3d30 2022-06-29T16:08:17.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a1ffb700 1 -- 172.21.15.60:0/1282103887 >> v1:172.21.15.60:6789/0 conn(0x7f07a40b3d30 legacy=0x7f07a4146ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:17.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a1ffb700 1 --2- 172.21.15.60:0/1282103887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07840038e0 0x7f0784003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:17.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a1ffb700 1 --2- 172.21.15.60:0/1282103887 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0784004500 0x7f0784004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:17.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a1ffb700 1 --2- 172.21.15.60:0/1282103887 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0784004e70 0x7f078400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:17.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a1ffb700 1 -- 172.21.15.60:0/1282103887 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a40ba150 con 0x7f07840038e0 2022-06-29T16:08:17.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a1ffb700 1 -- 172.21.15.60:0/1282103887 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078400bcf0 con 0x7f0784004500 2022-06-29T16:08:17.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a1ffb700 1 -- 172.21.15.60:0/1282103887 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f078400beb0 con 0x7f0784004e70 2022-06-29T16:08:17.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a2ffd700 1 --2- 172.21.15.60:0/1282103887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07840038e0 0x7f0784003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:17.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a27fc700 1 --2- 172.21.15.60:0/1282103887 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0784004500 0x7f0784004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:17.749 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a37fe700 1 --2- 172.21.15.60:0/1282103887 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0784004e70 0x7f078400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:17.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a2ffd700 1 -- 172.21.15.60:0/1282103887 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0784004e70 msgr2=0x7f078400b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:17.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a2ffd700 1 --2- 172.21.15.60:0/1282103887 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0784004e70 0x7f078400b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:17.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a2ffd700 1 -- 172.21.15.60:0/1282103887 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0784004500 msgr2=0x7f0784004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:17.750 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a2ffd700 1 --2- 172.21.15.60:0/1282103887 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0784004500 0x7f0784004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:17.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a37fe700 1 -- 172.21.15.60:0/1282103887 reap_dead start 2022-06-29T16:08:17.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a2ffd700 1 -- 172.21.15.60:0/1282103887 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0784003260 con 0x7f07840038e0 2022-06-29T16:08:17.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a2ffd700 1 --2- 172.21.15.60:0/1282103887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07840038e0 0x7f0784003ce0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 rx=0x7f079400c000 tx=0x7f0794006df0).ready entity=mon.0 client_cookie=88fa804e48135898 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:17.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a1ffb700 1 -- 172.21.15.60:0/1282103887 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f079400e800 con 0x7f07840038e0 2022-06-29T16:08:17.751 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a1ffb700 1 -- 172.21.15.60:0/1282103887 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f079400e960 con 0x7f07840038e0 2022-06-29T16:08:17.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a1ffb700 1 -- 172.21.15.60:0/1282103887 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f079400ecc0 con 0x7f07840038e0 2022-06-29T16:08:17.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a989b700 1 -- 172.21.15.60:0/1282103887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07840038e0 msgr2=0x7f0784003ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:17.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a989b700 1 --2- 172.21.15.60:0/1282103887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07840038e0 0x7f0784003ce0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 rx=0x7f079400c000 tx=0x7f0794006df0).stop 2022-06-29T16:08:17.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a989b700 1 -- 172.21.15.60:0/1282103887 shutdown_connections 2022-06-29T16:08:17.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a989b700 1 --2- 172.21.15.60:0/1282103887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07840038e0 0x7f0784003ce0 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:17.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a989b700 1 -- 172.21.15.60:0/1282103887 >> 172.21.15.60:0/1282103887 conn(0x7f07a4062e70 msgr2=0x7f07a40b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:17.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a989b700 1 -- 172.21.15.60:0/1282103887 shutdown_connections 2022-06-29T16:08:17.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.737+0000 7f07a989b700 1 -- 172.21.15.60:0/1282103887 wait complete. 2022-06-29T16:08:17.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.741+0000 7f07a989b700 1 Processor -- start 2022-06-29T16:08:17.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.741+0000 7f07a989b700 1 -- start start 2022-06-29T16:08:17.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.741+0000 7f07a989b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07840038e0 0x7f07a4112720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:17.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.741+0000 7f07a989b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07a40aecb0 0x7f07a40b8900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:17.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.741+0000 7f07a989b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f07a4140f60 0x7f07a4141410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:17.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.741+0000 7f07a989b700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a4141900 con 0x7f07a40aecb0 2022-06-29T16:08:17.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.741+0000 7f07a989b700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a40af890 con 0x7f07a4140f60 2022-06-29T16:08:17.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.741+0000 7f07a989b700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a4112d00 con 0x7f07840038e0 2022-06-29T16:08:17.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.741+0000 7f07a27fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07a40aecb0 0x7f07a40b8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:17.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.741+0000 7f07a27fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07a40aecb0 0x7f07a40b8900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54386/0 (socket says 172.21.15.60:54386) 2022-06-29T16:08:17.761 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.741+0000 7f07a27fc700 1 -- 172.21.15.60:0/242512818 learned_addr learned my addr 172.21.15.60:0/242512818 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:17.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.741+0000 7f07a2ffd700 1 --2- 172.21.15.60:0/242512818 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07840038e0 0x7f07a4112720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:17.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.741+0000 7f07a37fe700 1 --2- 172.21.15.60:0/242512818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f07a4140f60 0x7f07a4141410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:17.762 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.741+0000 7f07a27fc700 1 -- 172.21.15.60:0/242512818 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07840038e0 msgr2=0x7f07a4112720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:17.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.741+0000 7f07a27fc700 1 --2- 172.21.15.60:0/242512818 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07840038e0 0x7f07a4112720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:17.763 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.741+0000 7f07a27fc700 1 -- 172.21.15.60:0/242512818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f07a4140f60 msgr2=0x7f07a4141410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:17.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.741+0000 7f07a27fc700 1 --2- 172.21.15.60:0/242512818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f07a4140f60 0x7f07a4141410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:17.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.741+0000 7f07a27fc700 1 -- 172.21.15.60:0/242512818 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a40b8df0 con 0x7f07a40aecb0 2022-06-29T16:08:17.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.741+0000 7f07a2ffd700 1 --2- 172.21.15.60:0/242512818 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07840038e0 0x7f07a4112720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:08:17.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.741+0000 7f07a27fc700 1 --2- 172.21.15.60:0/242512818 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07a40aecb0 0x7f07a40b8900 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 rx=0x7f078c001040 tx=0x7f07a40b5310).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:17.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.741+0000 7f079bfff700 1 -- 172.21.15.60:0/242512818 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f078c0116b0 con 0x7f07a40aecb0 2022-06-29T16:08:17.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.741+0000 7f079bfff700 1 -- 172.21.15.60:0/242512818 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f078c011810 con 0x7f07a40aecb0 2022-06-29T16:08:17.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.741+0000 7f079bfff700 1 -- 172.21.15.60:0/242512818 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f078c011b70 con 0x7f07a40aecb0 2022-06-29T16:08:17.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.741+0000 7f07a989b700 1 -- 172.21.15.60:0/242512818 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a40ae920 con 0x7f07a40aecb0 2022-06-29T16:08:17.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.741+0000 7f07a989b700 1 -- 172.21.15.60:0/242512818 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a4114e60 con 0x7f07a40aecb0 2022-06-29T16:08:17.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.745+0000 7f07a989b700 1 -- 172.21.15.60:0/242512818 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a40573d0 con 0x7f07a40aecb0 2022-06-29T16:08:17.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.745+0000 7f079bfff700 1 -- 172.21.15.60:0/242512818 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f078c011cd0 con 0x7f07a40aecb0 2022-06-29T16:08:17.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.745+0000 7f079bfff700 1 --2- 172.21.15.60:0/242512818 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0788053aa0 0x7f0788055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:17.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.745+0000 7f079bfff700 1 -- 172.21.15.60:0/242512818 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f078c06ba70 con 0x7f07a40aecb0 2022-06-29T16:08:17.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.745+0000 7f07a2ffd700 1 --2- 172.21.15.60:0/242512818 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0788053aa0 0x7f0788055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:17.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.745+0000 7f07a2ffd700 1 --2- 172.21.15.60:0/242512818 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0788053aa0 0x7f0788055f60 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0x7f079401a000 tx=0x7f0794019000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:17.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.753+0000 7f079bfff700 1 -- 172.21.15.60:0/242512818 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f078c0428f0 con 0x7f07a40aecb0 2022-06-29T16:08:17.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:17.917+0000 7f07a989b700 1 -- 172.21.15.60:0/242512818 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f07a40afea0 con 0x7f07a40aecb0 2022-06-29T16:08:18.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.093+0000 7f079bfff700 1 -- 172.21.15.60:0/242512818 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v73) v1 ==== 118+0+0 (secure 0 0 0) 0x7f078c06be60 con 0x7f07a40aecb0 2022-06-29T16:08:18.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.097+0000 7f07a989b700 1 -- 172.21.15.60:0/242512818 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0788053aa0 msgr2=0x7f0788055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:18.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.097+0000 7f07a989b700 1 --2- 172.21.15.60:0/242512818 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0788053aa0 0x7f0788055f60 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0x7f079401a000 tx=0x7f0794019000).stop 2022-06-29T16:08:18.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.097+0000 7f07a989b700 1 -- 172.21.15.60:0/242512818 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07a40aecb0 msgr2=0x7f07a40b8900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:18.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.097+0000 7f07a989b700 1 --2- 172.21.15.60:0/242512818 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07a40aecb0 0x7f07a40b8900 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 rx=0x7f078c001040 tx=0x7f07a40b5310).stop 2022-06-29T16:08:18.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.097+0000 7f07a989b700 1 -- 172.21.15.60:0/242512818 shutdown_connections 2022-06-29T16:08:18.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.097+0000 7f07a989b700 1 --2- 172.21.15.60:0/242512818 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0788053aa0 0x7f0788055f60 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:18.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.097+0000 7f07a989b700 1 --2- 172.21.15.60:0/242512818 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f07a4140f60 0x7f07a4141410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:18.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.097+0000 7f07a989b700 1 --2- 172.21.15.60:0/242512818 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f07a40aecb0 0x7f07a40b8900 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:18.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.097+0000 7f07a989b700 1 --2- 172.21.15.60:0/242512818 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f07840038e0 0x7f07a4112720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:18.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.097+0000 7f07a989b700 1 -- 172.21.15.60:0/242512818 >> 172.21.15.60:0/242512818 conn(0x7f07a4062e70 msgr2=0x7f07a4148660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:18.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.101+0000 7f07a989b700 1 -- 172.21.15.60:0/242512818 shutdown_connections 2022-06-29T16:08:18.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.101+0000 7f07a989b700 1 -- 172.21.15.60:0/242512818 wait complete. 2022-06-29T16:08:18.118 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:08:18.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.221+0000 7f6b051eb700 1 Processor -- start 2022-06-29T16:08:18.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.221+0000 7f6b051eb700 1 -- start start 2022-06-29T16:08:18.230 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.221+0000 7f6b051eb700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6b000aee30 0x7f6b000b1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:18.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.221+0000 7f6b051eb700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6b00141b70 0x7f6b00140bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:18.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.221+0000 7f6b051eb700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b0005d450 con 0x7f6b0005d070 2022-06-29T16:08:18.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.221+0000 7f6b051eb700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b00143a90 con 0x7f6b000aee30 2022-06-29T16:08:18.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.221+0000 7f6b051eb700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b00062200 con 0x7f6b00141b70 2022-06-29T16:08:18.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.225+0000 7f6afffff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6b0005d070 0x7f6b0013a460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34248/0 (socket says 172.21.15.60:34248) 2022-06-29T16:08:18.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.225+0000 7f6afffff700 1 -- 172.21.15.60:0/4138793042 learned_addr learned my addr 172.21.15.60:0/4138793042 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:18.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.225+0000 7f6af7fff700 1 --2- 172.21.15.60:0/4138793042 >> v2:172.21.15.92:3300/0 conn(0x7f6b000aee30 0x7f6b000b1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:18.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.225+0000 7f6b049ea700 1 --2- 172.21.15.60:0/4138793042 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6b00141b70 0x7f6b00140bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:18.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.225+0000 7f6aff7fe700 1 -- 172.21.15.60:0/4138793042 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2373700384 0 0) 0x7f6b0005d450 con 0x7f6b0005d070 2022-06-29T16:08:18.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.225+0000 7f6aff7fe700 1 -- 172.21.15.60:0/4138793042 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ae0003260 con 0x7f6b0005d070 2022-06-29T16:08:18.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.225+0000 7f6af7fff700 1 -- 172.21.15.60:0/4138793042 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6b00141b70 msgr2=0x7f6b00140bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:18.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.225+0000 7f6af7fff700 1 --2- 172.21.15.60:0/4138793042 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6b00141b70 0x7f6b00140bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:18.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.225+0000 7f6af7fff700 1 -- 172.21.15.60:0/4138793042 >> v1:172.21.15.60:6789/0 conn(0x7f6b0005d070 legacy=0x7f6b0013a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:18.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.225+0000 7f6af7fff700 1 -- 172.21.15.60:0/4138793042 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b00064dc0 con 0x7f6b000aee30 2022-06-29T16:08:18.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.225+0000 7f6af7fff700 1 --2- 172.21.15.60:0/4138793042 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6b000aee30 0x7f6b000b1f30 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7f6ae8002f70 tx=0x7f6ae8006df0).ready entity=mon.1 client_cookie=c2bba93651d8e3d8 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:18.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.225+0000 7f6aff7fe700 1 -- 172.21.15.60:0/4138793042 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ae800d4a0 con 0x7f6b000aee30 2022-06-29T16:08:18.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.225+0000 7f6aff7fe700 1 -- 172.21.15.60:0/4138793042 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ae8000e90 con 0x7f6b000aee30 2022-06-29T16:08:18.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.225+0000 7f6aff7fe700 1 -- 172.21.15.60:0/4138793042 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ae800d7f0 con 0x7f6b000aee30 2022-06-29T16:08:18.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.225+0000 7f6b051eb700 1 -- 172.21.15.60:0/4138793042 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6b000aee30 msgr2=0x7f6b000b1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:18.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.225+0000 7f6b051eb700 1 --2- 172.21.15.60:0/4138793042 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6b000aee30 0x7f6b000b1f30 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7f6ae8002f70 tx=0x7f6ae8006df0).stop 2022-06-29T16:08:18.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.225+0000 7f6b051eb700 1 -- 172.21.15.60:0/4138793042 shutdown_connections 2022-06-29T16:08:18.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.225+0000 7f6b051eb700 1 --2- 172.21.15.60:0/4138793042 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6b00141b70 0x7f6b00140bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:18.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.225+0000 7f6b051eb700 1 --2- 172.21.15.60:0/4138793042 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6b000aee30 0x7f6b000b1f30 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:18.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.225+0000 7f6b051eb700 1 -- 172.21.15.60:0/4138793042 >> 172.21.15.60:0/4138793042 conn(0x7f6b000b5890 msgr2=0x7f6b000b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:18.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.225+0000 7f6b051eb700 1 -- 172.21.15.60:0/4138793042 shutdown_connections 2022-06-29T16:08:18.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.225+0000 7f6b051eb700 1 -- 172.21.15.60:0/4138793042 wait complete. 2022-06-29T16:08:18.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.229+0000 7f6b051eb700 1 Processor -- start 2022-06-29T16:08:18.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.229+0000 7f6b051eb700 1 -- start start 2022-06-29T16:08:18.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.229+0000 7f6b051eb700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6b0005d070 0x7f6b0013f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:18.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.229+0000 7f6b051eb700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6b000aee30 0x7f6b00152e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:18.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.229+0000 7f6b051eb700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b00141b70 0x7f6b00110460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:18.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.229+0000 7f6b051eb700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b00110000 con 0x7f6b00141b70 2022-06-29T16:08:18.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.229+0000 7f6b051eb700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b00146950 con 0x7f6b0005d070 2022-06-29T16:08:18.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.229+0000 7f6b051eb700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b001466b0 con 0x7f6b000aee30 2022-06-29T16:08:18.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.229+0000 7f6b049ea700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b00141b70 0x7f6b00110460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:18.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.229+0000 7f6b049ea700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b00141b70 0x7f6b00110460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54402/0 (socket says 172.21.15.60:54402) 2022-06-29T16:08:18.243 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.229+0000 7f6b049ea700 1 -- 172.21.15.60:0/3671626742 learned_addr learned my addr 172.21.15.60:0/3671626742 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:18.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.229+0000 7f6af7fff700 1 --2- 172.21.15.60:0/3671626742 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6b000aee30 0x7f6b00152e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:18.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.229+0000 7f6afffff700 1 --2- 172.21.15.60:0/3671626742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6b0005d070 0x7f6b0013f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:18.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.229+0000 7f6b049ea700 1 -- 172.21.15.60:0/3671626742 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6b000aee30 msgr2=0x7f6b00152e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:18.244 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.229+0000 7f6b049ea700 1 --2- 172.21.15.60:0/3671626742 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6b000aee30 0x7f6b00152e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:18.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.229+0000 7f6b049ea700 1 -- 172.21.15.60:0/3671626742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6b0005d070 msgr2=0x7f6b0013f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:18.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.229+0000 7f6b049ea700 1 --2- 172.21.15.60:0/3671626742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6b0005d070 0x7f6b0013f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:18.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.229+0000 7f6b049ea700 1 -- 172.21.15.60:0/3671626742 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae800b180 con 0x7f6b00141b70 2022-06-29T16:08:18.245 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.229+0000 7f6b049ea700 1 --2- 172.21.15.60:0/3671626742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b00141b70 0x7f6b00110460 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 rx=0x7f6af8006e00 tx=0x7f6af80091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:18.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.229+0000 7f6afdffb700 1 -- 172.21.15.60:0/3671626742 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6af8012600 con 0x7f6b00141b70 2022-06-29T16:08:18.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.229+0000 7f6afdffb700 1 -- 172.21.15.60:0/3671626742 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af8012760 con 0x7f6b00141b70 2022-06-29T16:08:18.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.229+0000 7f6b051eb700 1 -- 172.21.15.60:0/3671626742 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b00153360 con 0x7f6b00141b70 2022-06-29T16:08:18.246 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.229+0000 7f6afdffb700 1 -- 172.21.15.60:0/3671626742 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6af8012ac0 con 0x7f6b00141b70 2022-06-29T16:08:18.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.229+0000 7f6b051eb700 1 -- 172.21.15.60:0/3671626742 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b001461a0 con 0x7f6b00141b70 2022-06-29T16:08:18.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.233+0000 7f6afdffb700 1 -- 172.21.15.60:0/3671626742 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6af8022070 con 0x7f6b00141b70 2022-06-29T16:08:18.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.233+0000 7f6b051eb700 1 -- 172.21.15.60:0/3671626742 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b000573d0 con 0x7f6b00141b70 2022-06-29T16:08:18.247 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.233+0000 7f6afdffb700 1 --2- 172.21.15.60:0/3671626742 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6ae40539e0 0x7f6ae4055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:18.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.233+0000 7f6afdffb700 1 -- 172.21.15.60:0/3671626742 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6af806c4c0 con 0x7f6b00141b70 2022-06-29T16:08:18.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.233+0000 7f6afffff700 1 --2- 172.21.15.60:0/3671626742 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6ae40539e0 0x7f6ae4055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:18.248 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.233+0000 7f6afffff700 1 --2- 172.21.15.60:0/3671626742 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6ae40539e0 0x7f6ae4055ea0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0x7f6af000d000 tx=0x7f6af0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:18.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.237+0000 7f6afdffb700 1 -- 172.21.15.60:0/3671626742 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6af8043340 con 0x7f6b00141b70 2022-06-29T16:08:18.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.401+0000 7f6b051eb700 1 -- 172.21.15.60:0/3671626742 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b00062f00 con 0x7f6b00141b70 2022-06-29T16:08:18.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.401+0000 7f6afdffb700 1 -- 172.21.15.60:0/3671626742 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+2139 (secure 0 0 0) 0x7f6af8016090 con 0x7f6b00141b70 2022-06-29T16:08:18.407 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:08:18.408 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":73,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7457,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.92:6835/698111951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":698111951},{"type":"v1","addr":"172.21.15.92:6835","nonce":698111951}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":72}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:08:17.075855+0000","modified":"2022-06-29T16:08:18.079541+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8790},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8790":{"gid":8790,"name":"a","rank":0,"incarnation":72,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/3619856083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3619856083},{"type":"v1","addr":"172.21.15.60:6833","nonce":3619856083}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2022-06-29T16:08:18.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.405+0000 7f6b051eb700 1 -- 172.21.15.60:0/3671626742 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6ae40539e0 msgr2=0x7f6ae4055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:18.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.405+0000 7f6b051eb700 1 --2- 172.21.15.60:0/3671626742 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6ae40539e0 0x7f6ae4055ea0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0x7f6af000d000 tx=0x7f6af0009040).stop 2022-06-29T16:08:18.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.405+0000 7f6b051eb700 1 -- 172.21.15.60:0/3671626742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b00141b70 msgr2=0x7f6b00110460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:18.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.405+0000 7f6b051eb700 1 --2- 172.21.15.60:0/3671626742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b00141b70 0x7f6b00110460 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 rx=0x7f6af8006e00 tx=0x7f6af80091d0).stop 2022-06-29T16:08:18.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.405+0000 7f6b051eb700 1 -- 172.21.15.60:0/3671626742 shutdown_connections 2022-06-29T16:08:18.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.405+0000 7f6b051eb700 1 --2- 172.21.15.60:0/3671626742 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6ae40539e0 0x7f6ae4055ea0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:18.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.405+0000 7f6b051eb700 1 --2- 172.21.15.60:0/3671626742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b00141b70 0x7f6b00110460 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:18.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.405+0000 7f6b051eb700 1 --2- 172.21.15.60:0/3671626742 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6b000aee30 0x7f6b00152e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:18.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.405+0000 7f6b051eb700 1 --2- 172.21.15.60:0/3671626742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6b0005d070 0x7f6b0013f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:18.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.405+0000 7f6b051eb700 1 -- 172.21.15.60:0/3671626742 >> 172.21.15.60:0/3671626742 conn(0x7f6b000b5890 msgr2=0x7f6b0005bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:18.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.405+0000 7f6b051eb700 1 -- 172.21.15.60:0/3671626742 shutdown_connections 2022-06-29T16:08:18.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.405+0000 7f6b051eb700 1 -- 172.21.15.60:0/3671626742 wait complete. 2022-06-29T16:08:18.413 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 73 2022-06-29T16:08:18.421 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:08:18.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.517+0000 7fbfa14bb700 1 Processor -- start 2022-06-29T16:08:18.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.517+0000 7fbfa14bb700 1 -- start start 2022-06-29T16:08:18.531 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.521+0000 7fbfa14bb700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbf9c0b9c20 0x7fbf9c0b0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:18.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.521+0000 7fbfa14bb700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbf9c1464b0 0x7fbf9c05bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:18.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.521+0000 7fbfa14bb700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf9c0b1de0 con 0x7fbf9c146040 2022-06-29T16:08:18.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.521+0000 7fbfa14bb700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf9c0b1330 con 0x7fbf9c0b9c20 2022-06-29T16:08:18.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.521+0000 7fbfa14bb700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c0b2bc0 con 0x7fbf9c1464b0 2022-06-29T16:08:18.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.521+0000 7fbf9bfff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbf9c0b9c20 0x7fbf9c0b0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:18.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.521+0000 7fbf9bfff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fbf9c0b9c20 0x7fbf9c0b0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54406/0 (socket says 172.21.15.60:54406) 2022-06-29T16:08:18.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.521+0000 7fbf9bfff700 1 -- 172.21.15.60:0/1822631131 learned_addr learned my addr 172.21.15.60:0/1822631131 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:18.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.521+0000 7fbfa0cba700 1 --2- 172.21.15.60:0/1822631131 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbf9c1464b0 0x7fbf9c05bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:18.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.521+0000 7fbf9bfff700 1 -- 172.21.15.60:0/1822631131 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbf9c1464b0 msgr2=0x7fbf9c05bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:18.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.521+0000 7fbf9bfff700 1 --2- 172.21.15.60:0/1822631131 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbf9c1464b0 0x7fbf9c05bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:18.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.521+0000 7fbf9bfff700 1 -- 172.21.15.60:0/1822631131 >> v1:172.21.15.92:6789/0 conn(0x7fbf9c146040 legacy=0x7fbf9c0b24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:18.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.521+0000 7fbf9bfff700 1 -- 172.21.15.60:0/1822631131 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf9c0ba000 con 0x7fbf9c0b9c20 2022-06-29T16:08:18.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.521+0000 7fbf9bfff700 1 --2- 172.21.15.60:0/1822631131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf9c0b9c20 0x7fbf9c0b0610 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 rx=0x7fbf940032f0 tx=0x7fbf94009290).ready entity=mon.0 client_cookie=938f990b22d54115 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:18.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.521+0000 7fbf9affd700 1 -- 172.21.15.60:0/1822631131 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf9400b8a0 con 0x7fbf9c0b9c20 2022-06-29T16:08:18.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.521+0000 7fbf9affd700 1 -- 172.21.15.60:0/1822631131 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9400ba00 con 0x7fbf9c0b9c20 2022-06-29T16:08:18.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.521+0000 7fbf9affd700 1 -- 172.21.15.60:0/1822631131 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf9400bb70 con 0x7fbf9c0b9c20 2022-06-29T16:08:18.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.521+0000 7fbfa14bb700 1 -- 172.21.15.60:0/1822631131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf9c0b9c20 msgr2=0x7fbf9c0b0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:18.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.521+0000 7fbfa14bb700 1 --2- 172.21.15.60:0/1822631131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf9c0b9c20 0x7fbf9c0b0610 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 rx=0x7fbf940032f0 tx=0x7fbf94009290).stop 2022-06-29T16:08:18.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.521+0000 7fbfa14bb700 1 -- 172.21.15.60:0/1822631131 shutdown_connections 2022-06-29T16:08:18.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.521+0000 7fbfa14bb700 1 --2- 172.21.15.60:0/1822631131 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbf9c1464b0 0x7fbf9c05bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:18.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.521+0000 7fbfa14bb700 1 --2- 172.21.15.60:0/1822631131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf9c0b9c20 0x7fbf9c0b0610 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:18.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.521+0000 7fbfa14bb700 1 -- 172.21.15.60:0/1822631131 >> 172.21.15.60:0/1822631131 conn(0x7fbf9c062cb0 msgr2=0x7fbf9c0b91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:18.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.521+0000 7fbfa14bb700 1 -- 172.21.15.60:0/1822631131 shutdown_connections 2022-06-29T16:08:18.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.521+0000 7fbfa14bb700 1 -- 172.21.15.60:0/1822631131 wait complete. 2022-06-29T16:08:18.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.521+0000 7fbfa14bb700 1 Processor -- start 2022-06-29T16:08:18.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.521+0000 7fbfa14bb700 1 -- start start 2022-06-29T16:08:18.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.525+0000 7fbfa14bb700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf9c0b9c20 0x7fbf9c148c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:18.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.525+0000 7fbfa14bb700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbf9c146040 0x7fbf9c13c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:18.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.525+0000 7fbfa14bb700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf9c1464b0 0x7fbf9c144b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:18.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.525+0000 7fbfa14bb700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c1443e0 con 0x7fbf9c0b9c20 2022-06-29T16:08:18.539 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.525+0000 7fbfa14bb700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c0b8af0 con 0x7fbf9c1464b0 2022-06-29T16:08:18.539 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.525+0000 7fbfa14bb700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c0b8d90 con 0x7fbf9c146040 2022-06-29T16:08:18.539 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.525+0000 7fbf9bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf9c0b9c20 0x7fbf9c148c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:18.539 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.525+0000 7fbf9bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf9c0b9c20 0x7fbf9c148c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54412/0 (socket says 172.21.15.60:54412) 2022-06-29T16:08:18.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.525+0000 7fbf9bfff700 1 -- 172.21.15.60:0/3110245412 learned_addr learned my addr 172.21.15.60:0/3110245412 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:18.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.525+0000 7fbfa0cba700 1 --2- 172.21.15.60:0/3110245412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf9c1464b0 0x7fbf9c144b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:18.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.525+0000 7fbf9b7fe700 1 --2- 172.21.15.60:0/3110245412 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbf9c146040 0x7fbf9c13c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:18.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.525+0000 7fbf9bfff700 1 -- 172.21.15.60:0/3110245412 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbf9c146040 msgr2=0x7fbf9c13c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:18.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.525+0000 7fbf9bfff700 1 --2- 172.21.15.60:0/3110245412 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbf9c146040 0x7fbf9c13c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:18.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.525+0000 7fbf9bfff700 1 -- 172.21.15.60:0/3110245412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf9c1464b0 msgr2=0x7fbf9c144b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:18.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.525+0000 7fbf9bfff700 1 --2- 172.21.15.60:0/3110245412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf9c1464b0 0x7fbf9c144b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:18.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.525+0000 7fbf9bfff700 1 -- 172.21.15.60:0/3110245412 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf94009530 con 0x7fbf9c0b9c20 2022-06-29T16:08:18.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.525+0000 7fbfa0cba700 1 --2- 172.21.15.60:0/3110245412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf9c1464b0 0x7fbf9c144b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:08:18.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.525+0000 7fbf9bfff700 1 --2- 172.21.15.60:0/3110245412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf9c0b9c20 0x7fbf9c148c40 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 rx=0x7fbf940032f0 tx=0x7fbf94016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:18.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.525+0000 7fbf997fa700 1 -- 172.21.15.60:0/3110245412 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf94021480 con 0x7fbf9c0b9c20 2022-06-29T16:08:18.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.525+0000 7fbf997fa700 1 -- 172.21.15.60:0/3110245412 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf940215e0 con 0x7fbf9c0b9c20 2022-06-29T16:08:18.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.525+0000 7fbfa14bb700 1 -- 172.21.15.60:0/3110245412 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf9c13ac90 con 0x7fbf9c0b9c20 2022-06-29T16:08:18.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.525+0000 7fbf997fa700 1 -- 172.21.15.60:0/3110245412 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf94021940 con 0x7fbf9c0b9c20 2022-06-29T16:08:18.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.525+0000 7fbfa14bb700 1 -- 172.21.15.60:0/3110245412 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf9c143f40 con 0x7fbf9c0b9c20 2022-06-29T16:08:18.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.525+0000 7fbfa14bb700 1 -- 172.21.15.60:0/3110245412 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf9c0b8f50 con 0x7fbf9c0b9c20 2022-06-29T16:08:18.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.525+0000 7fbf997fa700 1 -- 172.21.15.60:0/3110245412 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fbf9400f070 con 0x7fbf9c0b9c20 2022-06-29T16:08:18.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.525+0000 7fbf997fa700 1 --2- 172.21.15.60:0/3110245412 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbf7c053a70 0x7fbf7c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:18.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.525+0000 7fbf997fa700 1 -- 172.21.15.60:0/3110245412 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbf9406d420 con 0x7fbf9c0b9c20 2022-06-29T16:08:18.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.525+0000 7fbf9b7fe700 1 --2- 172.21.15.60:0/3110245412 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbf7c053a70 0x7fbf7c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:18.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.529+0000 7fbf9b7fe700 1 --2- 172.21.15.60:0/3110245412 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbf7c053a70 0x7fbf7c055f30 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0x7fbf9000b000 tx=0x7fbf90009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:18.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.533+0000 7fbf997fa700 1 -- 172.21.15.60:0/3110245412 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbf940443f0 con 0x7fbf9c0b9c20 2022-06-29T16:08:18.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.661+0000 7fbfa14bb700 1 -- 172.21.15.60:0/3110245412 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf9c0573d0 con 0x7fbf9c0b9c20 2022-06-29T16:08:18.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.661+0000 7fbf997fa700 1 -- 172.21.15.60:0/3110245412 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v146) v1 ==== 74+0+16065 (secure 0 0 0) 0x7fbf94015070 con 0x7fbf9c0b9c20 2022-06-29T16:08:18.666 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:08:18.666 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":146,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:08:18.079157+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:08:15.820434+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":30,"pool_name":"cephfs_data","create_time":"2022-06-29T16:08:16.484716+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:08:18.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.661+0000 7fbfa14bb700 1 -- 172.21.15.60:0/3110245412 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbf7c053a70 msgr2=0x7fbf7c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:18.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.661+0000 7fbfa14bb700 1 --2- 172.21.15.60:0/3110245412 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbf7c053a70 0x7fbf7c055f30 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0x7fbf9000b000 tx=0x7fbf90009040).stop 2022-06-29T16:08:18.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.661+0000 7fbfa14bb700 1 -- 172.21.15.60:0/3110245412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf9c0b9c20 msgr2=0x7fbf9c148c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:18.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.661+0000 7fbfa14bb700 1 --2- 172.21.15.60:0/3110245412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf9c0b9c20 0x7fbf9c148c40 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 rx=0x7fbf940032f0 tx=0x7fbf94016000).stop 2022-06-29T16:08:18.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.661+0000 7fbfa14bb700 1 -- 172.21.15.60:0/3110245412 shutdown_connections 2022-06-29T16:08:18.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.661+0000 7fbfa14bb700 1 --2- 172.21.15.60:0/3110245412 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbf7c053a70 0x7fbf7c055f30 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:18.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.661+0000 7fbfa14bb700 1 --2- 172.21.15.60:0/3110245412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf9c1464b0 0x7fbf9c144b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:18.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.661+0000 7fbfa14bb700 1 --2- 172.21.15.60:0/3110245412 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbf9c146040 0x7fbf9c13c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:18.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.661+0000 7fbfa14bb700 1 --2- 172.21.15.60:0/3110245412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf9c0b9c20 0x7fbf9c148c40 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:18.670 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.665+0000 7fbfa14bb700 1 -- 172.21.15.60:0/3110245412 >> 172.21.15.60:0/3110245412 conn(0x7fbf9c062cb0 msgr2=0x7fbf9c0b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:18.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.665+0000 7fbfa14bb700 1 -- 172.21.15.60:0/3110245412 shutdown_connections 2022-06-29T16:08:18.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.665+0000 7fbfa14bb700 1 -- 172.21.15.60:0/3110245412 wait complete. 2022-06-29T16:08:18.680 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T16:08:18.797 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.793+0000 7fbe6a5a5700 1 Processor -- start 2022-06-29T16:08:18.798 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.793+0000 7fbe6a5a5700 1 -- start start 2022-06-29T16:08:18.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe6a5a5700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe64067830 0x7fbe641426b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:18.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe6a5a5700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbe6405f7c0 0x7fbe640aeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:18.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe6a5a5700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe6413f0b0 con 0x7fbe6405fc20 2022-06-29T16:08:18.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe6a5a5700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe640ad4e0 con 0x7fbe6405f7c0 2022-06-29T16:08:18.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe6a5a5700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe64060b20 con 0x7fbe64067830 2022-06-29T16:08:18.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe69da4700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fbe6405fc20 0x7fbe640b2030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34278/0 (socket says 172.21.15.60:34278) 2022-06-29T16:08:18.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe69da4700 1 -- 172.21.15.60:0/4265278018 learned_addr learned my addr 172.21.15.60:0/4265278018 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:18.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe695a3700 1 --2- 172.21.15.60:0/4265278018 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe64067830 0x7fbe641426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:18.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe68da2700 1 --2- 172.21.15.60:0/4265278018 >> v2:172.21.15.92:3300/0 conn(0x7fbe6405f7c0 0x7fbe640aeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:18.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe53fff700 1 -- 172.21.15.60:0/4265278018 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1758093333 0 0) 0x7fbe5c001490 con 0x7fbe6405fc20 2022-06-29T16:08:18.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe53fff700 1 -- 172.21.15.60:0/4265278018 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe4c003260 con 0x7fbe6405fc20 2022-06-29T16:08:18.807 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe53fff700 1 -- 172.21.15.60:0/4265278018 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1940936761 0 0) 0x7fbe4c003260 con 0x7fbe6405fc20 2022-06-29T16:08:18.807 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe53fff700 1 -- 172.21.15.60:0/4265278018 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe64067830 msgr2=0x7fbe641426b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:18.807 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe53fff700 1 --2- 172.21.15.60:0/4265278018 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe64067830 0x7fbe641426b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:18.807 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe53fff700 1 -- 172.21.15.60:0/4265278018 >> v2:172.21.15.92:3300/0 conn(0x7fbe6405f7c0 msgr2=0x7fbe640aeeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:18.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe53fff700 1 --2- 172.21.15.60:0/4265278018 >> v2:172.21.15.92:3300/0 conn(0x7fbe6405f7c0 0x7fbe640aeeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:18.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe53fff700 1 -- 172.21.15.60:0/4265278018 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe640b0f70 con 0x7fbe6405fc20 2022-06-29T16:08:18.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe53fff700 1 -- 172.21.15.60:0/4265278018 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7fbe5c001b70 con 0x7fbe6405fc20 2022-06-29T16:08:18.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe53fff700 1 -- 172.21.15.60:0/4265278018 >> v1:172.21.15.60:6789/0 conn(0x7fbe6405fc20 legacy=0x7fbe640b2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:18.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe53fff700 1 --2- 172.21.15.60:0/4265278018 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe4c003750 0x7fbe4c003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:18.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe53fff700 1 --2- 172.21.15.60:0/4265278018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe4c004500 0x7fbe4c004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:18.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe53fff700 1 --2- 172.21.15.60:0/4265278018 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe64067830 0x7fbe4c004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:18.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe695a3700 1 --2- 172.21.15.60:0/4265278018 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe4c003750 0x7fbe4c003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:18.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe53fff700 1 -- 172.21.15.60:0/4265278018 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe640ad4e0 con 0x7fbe4c003750 2022-06-29T16:08:18.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe53fff700 1 -- 172.21.15.60:0/4265278018 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe64060b20 con 0x7fbe4c004500 2022-06-29T16:08:18.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe53fff700 1 -- 172.21.15.60:0/4265278018 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4c00b750 con 0x7fbe64067830 2022-06-29T16:08:18.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe68da2700 1 --2- 172.21.15.60:0/4265278018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe4c004500 0x7fbe4c004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:18.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe695a3700 1 -- 172.21.15.60:0/4265278018 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe64067830 msgr2=0x7fbe4c004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:18.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe69da4700 1 --2- 172.21.15.60:0/4265278018 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe64067830 0x7fbe4c004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:18.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe695a3700 1 --2- 172.21.15.60:0/4265278018 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe64067830 0x7fbe4c004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:18.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe695a3700 1 -- 172.21.15.60:0/4265278018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe4c004500 msgr2=0x7fbe4c004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:18.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe695a3700 1 --2- 172.21.15.60:0/4265278018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe4c004500 0x7fbe4c004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:18.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe695a3700 1 -- 172.21.15.60:0/4265278018 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe4c003260 con 0x7fbe4c003750 2022-06-29T16:08:18.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe69da4700 1 --2- 172.21.15.60:0/4265278018 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe64067830 0x7fbe4c004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:08:18.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe695a3700 1 --2- 172.21.15.60:0/4265278018 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe4c003750 0x7fbe4c003b70 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 rx=0x7fbe6000c000 tx=0x7fbe60006df0).ready entity=mon.0 client_cookie=3e9a3ee906c86f83 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:18.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe53fff700 1 -- 172.21.15.60:0/4265278018 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe6000e830 con 0x7fbe4c003750 2022-06-29T16:08:18.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe53fff700 1 -- 172.21.15.60:0/4265278018 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe6000e990 con 0x7fbe4c003750 2022-06-29T16:08:18.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe53fff700 1 -- 172.21.15.60:0/4265278018 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe6000ecf0 con 0x7fbe4c003750 2022-06-29T16:08:18.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe6a5a5700 1 -- 172.21.15.60:0/4265278018 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe4c003750 msgr2=0x7fbe4c003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:18.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe6a5a5700 1 --2- 172.21.15.60:0/4265278018 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe4c003750 0x7fbe4c003b70 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 rx=0x7fbe6000c000 tx=0x7fbe60006df0).stop 2022-06-29T16:08:18.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.797+0000 7fbe69da4700 1 -- 172.21.15.60:0/4265278018 reap_dead start 2022-06-29T16:08:18.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.801+0000 7fbe6a5a5700 1 -- 172.21.15.60:0/4265278018 shutdown_connections 2022-06-29T16:08:18.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.801+0000 7fbe6a5a5700 1 -- 172.21.15.60:0/4265278018 >> 172.21.15.60:0/4265278018 conn(0x7fbe64063ab0 msgr2=0x7fbe6405f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:18.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.801+0000 7fbe6a5a5700 1 -- 172.21.15.60:0/4265278018 shutdown_connections 2022-06-29T16:08:18.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.801+0000 7fbe6a5a5700 1 -- 172.21.15.60:0/4265278018 wait complete. 2022-06-29T16:08:18.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.801+0000 7fbe6a5a5700 1 Processor -- start 2022-06-29T16:08:18.818 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.801+0000 7fbe6a5a5700 1 -- start start 2022-06-29T16:08:18.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.801+0000 7fbe6a5a5700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe64066700 0x7fbe64144440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:18.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.801+0000 7fbe6a5a5700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe64142350 0x7fbe64145db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:18.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.801+0000 7fbe6a5a5700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe64142c60 0x7fbe641462a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:18.819 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.801+0000 7fbe6a5a5700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe64066ae0 con 0x7fbe64142c60 2022-06-29T16:08:18.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.801+0000 7fbe6a5a5700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe641427f0 con 0x7fbe64142350 2022-06-29T16:08:18.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.801+0000 7fbe6a5a5700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe64143fa0 con 0x7fbe64066700 2022-06-29T16:08:18.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.801+0000 7fbe69da4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe64142c60 0x7fbe641462a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:18.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.801+0000 7fbe695a3700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe64066700 0x7fbe64144440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:18.820 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.801+0000 7fbe68da2700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe64142350 0x7fbe64145db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:18.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.801+0000 7fbe68da2700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe64142350 0x7fbe64145db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:60464/0 (socket says 172.21.15.60:60464) 2022-06-29T16:08:18.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.801+0000 7fbe69da4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe64142c60 0x7fbe641462a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54436/0 (socket says 172.21.15.60:54436) 2022-06-29T16:08:18.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.801+0000 7fbe68da2700 1 -- 172.21.15.60:0/4289470085 learned_addr learned my addr 172.21.15.60:0/4289470085 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:18.821 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.801+0000 7fbe695a3700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe64066700 0x7fbe64144440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:39294/0 (socket says 172.21.15.60:39294) 2022-06-29T16:08:18.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.801+0000 7fbe69da4700 1 -- 172.21.15.60:0/4289470085 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe64066700 msgr2=0x7fbe64144440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:18.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.801+0000 7fbe69da4700 1 --2- 172.21.15.60:0/4289470085 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe64066700 0x7fbe64144440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:18.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.801+0000 7fbe69da4700 1 -- 172.21.15.60:0/4289470085 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe64142350 msgr2=0x7fbe64145db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:18.822 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.801+0000 7fbe69da4700 1 --2- 172.21.15.60:0/4289470085 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe64142350 0x7fbe64145db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:18.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.801+0000 7fbe69da4700 1 -- 172.21.15.60:0/4289470085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe64149670 con 0x7fbe64142c60 2022-06-29T16:08:18.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.801+0000 7fbe68da2700 1 --2- 172.21.15.60:0/4289470085 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe64142350 0x7fbe64145db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:08:18.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.801+0000 7fbe69da4700 1 --2- 172.21.15.60:0/4289470085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe64142c60 0x7fbe641462a0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 rx=0x7fbe5c001cd0 tx=0x7fbe5c009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:18.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.801+0000 7fbe527fc700 1 -- 172.21.15.60:0/4289470085 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe5c004b70 con 0x7fbe64142c60 2022-06-29T16:08:18.823 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.801+0000 7fbe527fc700 1 -- 172.21.15.60:0/4289470085 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe5c004cd0 con 0x7fbe64142c60 2022-06-29T16:08:18.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.801+0000 7fbe6a5a5700 1 -- 172.21.15.60:0/4289470085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe64144930 con 0x7fbe64142c60 2022-06-29T16:08:18.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.805+0000 7fbe527fc700 1 -- 172.21.15.60:0/4289470085 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe5c0072c0 con 0x7fbe64142c60 2022-06-29T16:08:18.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.805+0000 7fbe6a5a5700 1 -- 172.21.15.60:0/4289470085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe641407d0 con 0x7fbe64142c60 2022-06-29T16:08:18.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.805+0000 7fbe6a5a5700 1 -- 172.21.15.60:0/4289470085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe64144160 con 0x7fbe64142c60 2022-06-29T16:08:18.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.805+0000 7fbe527fc700 1 -- 172.21.15.60:0/4289470085 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fbe5c01a060 con 0x7fbe64142c60 2022-06-29T16:08:18.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.805+0000 7fbe527fc700 1 --2- 172.21.15.60:0/4289470085 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbe44053b00 0x7fbe44055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:18.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.805+0000 7fbe527fc700 1 -- 172.21.15.60:0/4289470085 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbe5c06b6d0 con 0x7fbe64142c60 2022-06-29T16:08:18.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.805+0000 7fbe695a3700 1 --2- 172.21.15.60:0/4289470085 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbe44053b00 0x7fbe44055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:18.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.805+0000 7fbe695a3700 1 --2- 172.21.15.60:0/4289470085 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbe44053b00 0x7fbe44055fc0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0x7fbe60010000 tx=0x7fbe60019000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:18.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.809+0000 7fbe527fc700 1 -- 172.21.15.60:0/4289470085 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbe5c0425f0 con 0x7fbe64142c60 2022-06-29T16:08:18.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.977+0000 7fbe6a5a5700 1 -- 172.21.15.60:0/4289470085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fbe6413f0b0 con 0x7fbe64142c60 2022-06-29T16:08:18.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.985+0000 7fbe527fc700 1 -- 172.21.15.60:0/4289470085 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v47) v1 ==== 184+0+0 (secure 0 0 0) 0x7fbe6413f0b0 con 0x7fbe64142c60 2022-06-29T16:08:18.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.985+0000 7fbe6a5a5700 1 -- 172.21.15.60:0/4289470085 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbe44053b00 msgr2=0x7fbe44055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:18.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.985+0000 7fbe6a5a5700 1 --2- 172.21.15.60:0/4289470085 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbe44053b00 0x7fbe44055fc0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0x7fbe60010000 tx=0x7fbe60019000).stop 2022-06-29T16:08:18.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.985+0000 7fbe6a5a5700 1 -- 172.21.15.60:0/4289470085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe64142c60 msgr2=0x7fbe641462a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:18.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.985+0000 7fbe6a5a5700 1 --2- 172.21.15.60:0/4289470085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe64142c60 0x7fbe641462a0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 rx=0x7fbe5c001cd0 tx=0x7fbe5c009040).stop 2022-06-29T16:08:18.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.985+0000 7fbe6a5a5700 1 -- 172.21.15.60:0/4289470085 shutdown_connections 2022-06-29T16:08:18.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.985+0000 7fbe6a5a5700 1 --2- 172.21.15.60:0/4289470085 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fbe44053b00 0x7fbe44055fc0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:18.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.985+0000 7fbe6a5a5700 1 --2- 172.21.15.60:0/4289470085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe64142c60 0x7fbe641462a0 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:18.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.985+0000 7fbe6a5a5700 1 --2- 172.21.15.60:0/4289470085 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe64142350 0x7fbe64145db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:18.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.985+0000 7fbe6a5a5700 1 --2- 172.21.15.60:0/4289470085 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fbe64066700 0x7fbe64144440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:18.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.985+0000 7fbe6a5a5700 1 -- 172.21.15.60:0/4289470085 >> 172.21.15.60:0/4289470085 conn(0x7fbe64063ab0 msgr2=0x7fbe640b5f90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:18.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.985+0000 7fbe6a5a5700 1 -- 172.21.15.60:0/4289470085 shutdown_connections 2022-06-29T16:08:18.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:18.989+0000 7fbe6a5a5700 1 -- 172.21.15.60:0/4289470085 wait complete. 2022-06-29T16:08:18.996 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.0 2022-06-29T16:08:19.007 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T16:08:19.115 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.109+0000 7f36493d2700 1 Processor -- start 2022-06-29T16:08:19.116 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.109+0000 7f36493d2700 1 -- start start 2022-06-29T16:08:19.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.113+0000 7f36493d2700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f36440acfa0 0x7f36440b1f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:19.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.113+0000 7f36493d2700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36440b1ae0 con 0x7f36440b2400 2022-06-29T16:08:19.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.113+0000 7f36493d2700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36440666d0 con 0x7f36440acbc0 2022-06-29T16:08:19.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.113+0000 7f36493d2700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36440b2e00 con 0x7f36440acfa0 2022-06-29T16:08:19.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.113+0000 7f36437fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f36440b2400 0x7f364413a5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34296/0 (socket says 172.21.15.60:34296) 2022-06-29T16:08:19.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.113+0000 7f36437fe700 1 -- 172.21.15.60:0/1316397646 learned_addr learned my addr 172.21.15.60:0/1316397646 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:19.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.113+0000 7f36427fc700 1 --2- 172.21.15.60:0/1316397646 >> v2:172.21.15.92:3300/0 conn(0x7f36440acfa0 0x7f36440b1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:19.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.113+0000 7f3641ffb700 1 -- 172.21.15.60:0/1316397646 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 480717518 0 0) 0x7f36440b1ae0 con 0x7f36440b2400 2022-06-29T16:08:19.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.113+0000 7f3641ffb700 1 -- 172.21.15.60:0/1316397646 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3624003260 con 0x7f36440b2400 2022-06-29T16:08:19.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.113+0000 7f36427fc700 1 -- 172.21.15.60:0/1316397646 >> v1:172.21.15.92:6789/0 conn(0x7f36440acbc0 legacy=0x7f3644143f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:19.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.113+0000 7f36427fc700 1 -- 172.21.15.60:0/1316397646 >> v1:172.21.15.60:6789/0 conn(0x7f36440b2400 legacy=0x7f364413a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:19.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.113+0000 7f36427fc700 1 -- 172.21.15.60:0/1316397646 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36440663c0 con 0x7f36440acfa0 2022-06-29T16:08:19.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.113+0000 7f36427fc700 1 --2- 172.21.15.60:0/1316397646 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f36440acfa0 0x7f36440b1f10 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0x7f3634002f70 tx=0x7f3634006df0).ready entity=mon.1 client_cookie=a29fda01cec67cef server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:19.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.113+0000 7f3641ffb700 1 -- 172.21.15.60:0/1316397646 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f363400d610 con 0x7f36440acfa0 2022-06-29T16:08:19.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.113+0000 7f3641ffb700 1 -- 172.21.15.60:0/1316397646 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3634000e90 con 0x7f36440acfa0 2022-06-29T16:08:19.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.113+0000 7f3641ffb700 1 -- 172.21.15.60:0/1316397646 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f363400d960 con 0x7f36440acfa0 2022-06-29T16:08:19.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.113+0000 7f36493d2700 1 -- 172.21.15.60:0/1316397646 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f36440acfa0 msgr2=0x7f36440b1f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:19.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.113+0000 7f36493d2700 1 --2- 172.21.15.60:0/1316397646 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f36440acfa0 0x7f36440b1f10 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0x7f3634002f70 tx=0x7f3634006df0).stop 2022-06-29T16:08:19.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.113+0000 7f36493d2700 1 -- 172.21.15.60:0/1316397646 shutdown_connections 2022-06-29T16:08:19.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.113+0000 7f36493d2700 1 --2- 172.21.15.60:0/1316397646 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f36440acfa0 0x7f36440b1f10 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:19.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.113+0000 7f36493d2700 1 -- 172.21.15.60:0/1316397646 >> 172.21.15.60:0/1316397646 conn(0x7f36440b7140 msgr2=0x7f36440b7520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:19.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.113+0000 7f36493d2700 1 -- 172.21.15.60:0/1316397646 shutdown_connections 2022-06-29T16:08:19.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.113+0000 7f36493d2700 1 -- 172.21.15.60:0/1316397646 wait complete. 2022-06-29T16:08:19.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.117+0000 7f36493d2700 1 Processor -- start 2022-06-29T16:08:19.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.117+0000 7f36493d2700 1 -- start start 2022-06-29T16:08:19.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.117+0000 7f36493d2700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f36440acbc0 0x7f364405c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:19.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.117+0000 7f36493d2700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f36440acfa0 0x7f364413b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:19.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.117+0000 7f36493d2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36440b2400 0x7f364413b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:19.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.117+0000 7f36493d2700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36440b12a0 con 0x7f36440b2400 2022-06-29T16:08:19.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.117+0000 7f36493d2700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36440b08c0 con 0x7f36440acfa0 2022-06-29T16:08:19.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.117+0000 7f36493d2700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36440b0b60 con 0x7f36440acbc0 2022-06-29T16:08:19.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.117+0000 7f36437fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36440b2400 0x7f364413b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:19.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.117+0000 7f36427fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f36440acfa0 0x7f364413b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:19.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.117+0000 7f36427fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f36440acfa0 0x7f364413b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:60476/0 (socket says 172.21.15.60:60476) 2022-06-29T16:08:19.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.117+0000 7f3642ffd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f36440acbc0 0x7f364405c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:19.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.117+0000 7f36427fc700 1 -- 172.21.15.60:0/4147655615 learned_addr learned my addr 172.21.15.60:0/4147655615 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:19.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.117+0000 7f36437fe700 1 -- 172.21.15.60:0/4147655615 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f36440acbc0 msgr2=0x7f364405c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:19.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.117+0000 7f36437fe700 1 --2- 172.21.15.60:0/4147655615 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f36440acbc0 0x7f364405c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:19.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.117+0000 7f36437fe700 1 -- 172.21.15.60:0/4147655615 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f36440acfa0 msgr2=0x7f364413b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:19.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.117+0000 7f36437fe700 1 --2- 172.21.15.60:0/4147655615 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f36440acfa0 0x7f364413b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:19.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.117+0000 7f36437fe700 1 -- 172.21.15.60:0/4147655615 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f363400b180 con 0x7f36440b2400 2022-06-29T16:08:19.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.117+0000 7f3642ffd700 1 --2- 172.21.15.60:0/4147655615 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f36440acbc0 0x7f364405c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:08:19.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.117+0000 7f36427fc700 1 --2- 172.21.15.60:0/4147655615 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f36440acfa0 0x7f364413b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:08:19.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.117+0000 7f36437fe700 1 --2- 172.21.15.60:0/4147655615 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36440b2400 0x7f364413b580 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 rx=0x7f363c001490 tx=0x7f363c007d40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:19.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.117+0000 7f363bfff700 1 -- 172.21.15.60:0/4147655615 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f363c00ce70 con 0x7f36440b2400 2022-06-29T16:08:19.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.117+0000 7f363bfff700 1 -- 172.21.15.60:0/4147655615 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f363c005ba0 con 0x7f36440b2400 2022-06-29T16:08:19.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.117+0000 7f363bfff700 1 -- 172.21.15.60:0/4147655615 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f363c0070d0 con 0x7f36440b2400 2022-06-29T16:08:19.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.117+0000 7f36493d2700 1 -- 172.21.15.60:0/4147655615 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36440b0e00 con 0x7f36440b2400 2022-06-29T16:08:19.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.117+0000 7f36493d2700 1 -- 172.21.15.60:0/4147655615 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f364413ba70 con 0x7f36440b2400 2022-06-29T16:08:19.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.121+0000 7f36493d2700 1 -- 172.21.15.60:0/4147655615 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36440b0f90 con 0x7f36440b2400 2022-06-29T16:08:19.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.121+0000 7f363bfff700 1 -- 172.21.15.60:0/4147655615 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f363c010070 con 0x7f36440b2400 2022-06-29T16:08:19.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.121+0000 7f363bfff700 1 --2- 172.21.15.60:0/4147655615 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f36280539e0 0x7f3628055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:19.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.121+0000 7f363bfff700 1 -- 172.21.15.60:0/4147655615 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f363c06b7f0 con 0x7f36440b2400 2022-06-29T16:08:19.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.121+0000 7f3642ffd700 1 --2- 172.21.15.60:0/4147655615 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f36280539e0 0x7f3628055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:19.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.121+0000 7f3642ffd700 1 --2- 172.21.15.60:0/4147655615 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f36280539e0 0x7f3628055ea0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0x7f362c00d000 tx=0x7f362c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:19.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.129+0000 7f363bfff700 1 -- 172.21.15.60:0/4147655615 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f363c042600 con 0x7f36440b2400 2022-06-29T16:08:19.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.293+0000 7f36493d2700 1 -- 172.21.15.60:0/4147655615 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f36440573d0 con 0x7f36440b2400 2022-06-29T16:08:19.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.301+0000 7f363bfff700 1 -- 172.21.15.60:0/4147655615 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v48) v1 ==== 184+0+0 (secure 0 0 0) 0x7f363c045c20 con 0x7f36440b2400 2022-06-29T16:08:19.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.305+0000 7f36493d2700 1 -- 172.21.15.60:0/4147655615 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f36280539e0 msgr2=0x7f3628055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:19.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.305+0000 7f36493d2700 1 --2- 172.21.15.60:0/4147655615 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f36280539e0 0x7f3628055ea0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0x7f362c00d000 tx=0x7f362c009040).stop 2022-06-29T16:08:19.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.305+0000 7f36493d2700 1 -- 172.21.15.60:0/4147655615 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36440b2400 msgr2=0x7f364413b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:19.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.305+0000 7f36493d2700 1 --2- 172.21.15.60:0/4147655615 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36440b2400 0x7f364413b580 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 rx=0x7f363c001490 tx=0x7f363c007d40).stop 2022-06-29T16:08:19.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.305+0000 7f36493d2700 1 -- 172.21.15.60:0/4147655615 shutdown_connections 2022-06-29T16:08:19.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.305+0000 7f36493d2700 1 --2- 172.21.15.60:0/4147655615 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f36280539e0 0x7f3628055ea0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:19.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.305+0000 7f36493d2700 1 --2- 172.21.15.60:0/4147655615 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36440b2400 0x7f364413b580 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:19.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.305+0000 7f36493d2700 1 --2- 172.21.15.60:0/4147655615 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f36440acfa0 0x7f364413b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:19.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.305+0000 7f36493d2700 1 --2- 172.21.15.60:0/4147655615 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f36440acbc0 0x7f364405c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:19.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.305+0000 7f36493d2700 1 -- 172.21.15.60:0/4147655615 >> 172.21.15.60:0/4147655615 conn(0x7f36440b7140 msgr2=0x7f364413dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:19.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.305+0000 7f36493d2700 1 -- 172.21.15.60:0/4147655615 shutdown_connections 2022-06-29T16:08:19.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.305+0000 7f36493d2700 1 -- 172.21.15.60:0/4147655615 wait complete. 2022-06-29T16:08:19.313 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.1 2022-06-29T16:08:19.323 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-06-29T16:08:19.438 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.433+0000 7fe30a1c1700 1 Processor -- start 2022-06-29T16:08:19.439 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.433+0000 7fe30a1c1700 1 -- start start 2022-06-29T16:08:19.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.433+0000 7fe30a1c1700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3040b0280 0x7fe304146650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:19.442 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.433+0000 7fe30a1c1700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe304146b40 0x7fe3040b2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:19.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.433+0000 7fe30a1c1700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe304146fd0 con 0x7fe3040b5eb0 2022-06-29T16:08:19.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.433+0000 7fe30a1c1700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3040b6290 con 0x7fe304146b40 2022-06-29T16:08:19.443 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.433+0000 7fe30a1c1700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3040b0ab0 con 0x7fe3040b0280 2022-06-29T16:08:19.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.433+0000 7fe3091bf700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe3040b5eb0 0x7fe304142a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34306/0 (socket says 172.21.15.60:34306) 2022-06-29T16:08:19.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.433+0000 7fe3091bf700 1 -- 172.21.15.60:0/1301120634 learned_addr learned my addr 172.21.15.60:0/1301120634 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:19.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.437+0000 7fe3089be700 1 --2- 172.21.15.60:0/1301120634 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3040b0280 0x7fe304146650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:19.444 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.437+0000 7fe3099c0700 1 --2- 172.21.15.60:0/1301120634 >> v2:172.21.15.92:3300/0 conn(0x7fe304146b40 0x7fe3040b2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:19.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.437+0000 7fe2f3fff700 1 -- 172.21.15.60:0/1301120634 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 859701556 0 0) 0x7fe300001110 con 0x7fe3040b5eb0 2022-06-29T16:08:19.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.437+0000 7fe2f3fff700 1 -- 172.21.15.60:0/1301120634 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2ec003260 con 0x7fe3040b5eb0 2022-06-29T16:08:19.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.437+0000 7fe3089be700 1 -- 172.21.15.60:0/1301120634 >> v2:172.21.15.92:3300/0 conn(0x7fe304146b40 msgr2=0x7fe3040b2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:19.445 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.437+0000 7fe3089be700 1 --2- 172.21.15.60:0/1301120634 >> v2:172.21.15.92:3300/0 conn(0x7fe304146b40 0x7fe3040b2bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:19.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.437+0000 7fe3089be700 1 -- 172.21.15.60:0/1301120634 >> v1:172.21.15.60:6789/0 conn(0x7fe3040b5eb0 legacy=0x7fe304142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:19.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.437+0000 7fe3089be700 1 -- 172.21.15.60:0/1301120634 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3041461b0 con 0x7fe3040b0280 2022-06-29T16:08:19.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.437+0000 7fe3089be700 1 --2- 172.21.15.60:0/1301120634 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3040b0280 0x7fe304146650 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fe2f4002f70 tx=0x7fe2f4006df0).ready entity=mon.2 client_cookie=77089701cdc36eec server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:19.446 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.437+0000 7fe2f3fff700 1 -- 172.21.15.60:0/1301120634 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe2f400d4a0 con 0x7fe3040b0280 2022-06-29T16:08:19.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.437+0000 7fe2f3fff700 1 -- 172.21.15.60:0/1301120634 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f4000e90 con 0x7fe3040b0280 2022-06-29T16:08:19.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.437+0000 7fe2f3fff700 1 -- 172.21.15.60:0/1301120634 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe2f400d7f0 con 0x7fe3040b0280 2022-06-29T16:08:19.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.437+0000 7fe30a1c1700 1 -- 172.21.15.60:0/1301120634 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3040b0280 msgr2=0x7fe304146650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:19.447 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.437+0000 7fe30a1c1700 1 --2- 172.21.15.60:0/1301120634 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3040b0280 0x7fe304146650 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fe2f4002f70 tx=0x7fe2f4006df0).stop 2022-06-29T16:08:19.448 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.437+0000 7fe30a1c1700 1 -- 172.21.15.60:0/1301120634 shutdown_connections 2022-06-29T16:08:19.448 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.437+0000 7fe30a1c1700 1 --2- 172.21.15.60:0/1301120634 >> v2:172.21.15.92:3300/0 conn(0x7fe304146b40 0x7fe3040b2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:19.448 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.437+0000 7fe30a1c1700 1 --2- 172.21.15.60:0/1301120634 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3040b0280 0x7fe304146650 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:19.448 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.437+0000 7fe30a1c1700 1 -- 172.21.15.60:0/1301120634 >> 172.21.15.60:0/1301120634 conn(0x7fe304062090 msgr2=0x7fe304062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:19.449 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.437+0000 7fe30a1c1700 1 -- 172.21.15.60:0/1301120634 shutdown_connections 2022-06-29T16:08:19.449 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.437+0000 7fe30a1c1700 1 -- 172.21.15.60:0/1301120634 wait complete. 2022-06-29T16:08:19.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.437+0000 7fe30a1c1700 1 Processor -- start 2022-06-29T16:08:19.452 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.437+0000 7fe30a1c1700 1 -- start start 2022-06-29T16:08:19.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.441+0000 7fe30a1c1700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3040b0280 0x7fe3040af320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:19.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.441+0000 7fe30a1c1700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3040b5eb0 0x7fe304151af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:19.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.441+0000 7fe30a1c1700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe304146b40 0x7fe304140250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:19.453 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.441+0000 7fe30a1c1700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe304144da0 con 0x7fe3040b0280 2022-06-29T16:08:19.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.441+0000 7fe30a1c1700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3041445c0 con 0x7fe3040b5eb0 2022-06-29T16:08:19.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.441+0000 7fe30a1c1700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe304144f90 con 0x7fe304146b40 2022-06-29T16:08:19.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.441+0000 7fe3091bf700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3040b0280 0x7fe3040af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:19.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.441+0000 7fe3091bf700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3040b0280 0x7fe3040af320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54458/0 (socket says 172.21.15.60:54458) 2022-06-29T16:08:19.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.441+0000 7fe3091bf700 1 -- 172.21.15.60:0/1243261296 learned_addr learned my addr 172.21.15.60:0/1243261296 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:19.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.441+0000 7fe3089be700 1 --2- 172.21.15.60:0/1243261296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3040b5eb0 0x7fe304151af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:19.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.441+0000 7fe3099c0700 1 --2- 172.21.15.60:0/1243261296 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe304146b40 0x7fe304140250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:19.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.441+0000 7fe3091bf700 1 -- 172.21.15.60:0/1243261296 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe304146b40 msgr2=0x7fe304140250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:19.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.441+0000 7fe3091bf700 1 --2- 172.21.15.60:0/1243261296 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe304146b40 0x7fe304140250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:19.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.441+0000 7fe3091bf700 1 -- 172.21.15.60:0/1243261296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3040b5eb0 msgr2=0x7fe304151af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:19.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.441+0000 7fe3091bf700 1 --2- 172.21.15.60:0/1243261296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3040b5eb0 0x7fe304151af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:19.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.441+0000 7fe3091bf700 1 -- 172.21.15.60:0/1243261296 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f400b180 con 0x7fe3040b0280 2022-06-29T16:08:19.456 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.441+0000 7fe3091bf700 1 --2- 172.21.15.60:0/1243261296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3040b0280 0x7fe3040af320 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 rx=0x7fe3000012a0 tx=0x7fe300007900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:19.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.441+0000 7fe2f27fc700 1 -- 172.21.15.60:0/1243261296 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3000070d0 con 0x7fe3040b0280 2022-06-29T16:08:19.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.441+0000 7fe2f27fc700 1 -- 172.21.15.60:0/1243261296 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe300007230 con 0x7fe3040b0280 2022-06-29T16:08:19.457 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.441+0000 7fe30a1c1700 1 -- 172.21.15.60:0/1243261296 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3040af810 con 0x7fe3040b0280 2022-06-29T16:08:19.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.441+0000 7fe2f27fc700 1 -- 172.21.15.60:0/1243261296 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe300005610 con 0x7fe3040b0280 2022-06-29T16:08:19.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.441+0000 7fe30a1c1700 1 -- 172.21.15.60:0/1243261296 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe304151fe0 con 0x7fe3040b0280 2022-06-29T16:08:19.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.441+0000 7fe30a1c1700 1 -- 172.21.15.60:0/1243261296 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3040b2ad0 con 0x7fe3040b0280 2022-06-29T16:08:19.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.445+0000 7fe2f27fc700 1 -- 172.21.15.60:0/1243261296 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe300010070 con 0x7fe3040b0280 2022-06-29T16:08:19.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.445+0000 7fe2f27fc700 1 --2- 172.21.15.60:0/1243261296 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe2e4053b00 0x7fe2e4055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:19.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.445+0000 7fe2f27fc700 1 -- 172.21.15.60:0/1243261296 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe30006b2c0 con 0x7fe3040b0280 2022-06-29T16:08:19.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.445+0000 7fe3089be700 1 --2- 172.21.15.60:0/1243261296 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe2e4053b00 0x7fe2e4055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:19.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.445+0000 7fe3089be700 1 --2- 172.21.15.60:0/1243261296 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe2e4053b00 0x7fe2e4055fc0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 rx=0x7fe2f401a000 tx=0x7fe2f4014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:19.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.453+0000 7fe2f27fc700 1 -- 172.21.15.60:0/1243261296 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe300042140 con 0x7fe3040b0280 2022-06-29T16:08:19.620 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.613+0000 7fe30a1c1700 1 -- 172.21.15.60:0/1243261296 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fe304146fd0 con 0x7fe3040b0280 2022-06-29T16:08:19.627 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.621+0000 7fe2f27fc700 1 -- 172.21.15.60:0/1243261296 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v49) v1 ==== 184+0+0 (secure 0 0 0) 0x7fe304146fd0 con 0x7fe3040b0280 2022-06-29T16:08:19.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.625+0000 7fe30a1c1700 1 -- 172.21.15.60:0/1243261296 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe2e4053b00 msgr2=0x7fe2e4055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:19.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.625+0000 7fe30a1c1700 1 --2- 172.21.15.60:0/1243261296 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe2e4053b00 0x7fe2e4055fc0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 rx=0x7fe2f401a000 tx=0x7fe2f4014000).stop 2022-06-29T16:08:19.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.625+0000 7fe30a1c1700 1 -- 172.21.15.60:0/1243261296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3040b0280 msgr2=0x7fe3040af320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:19.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.625+0000 7fe30a1c1700 1 --2- 172.21.15.60:0/1243261296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3040b0280 0x7fe3040af320 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 rx=0x7fe3000012a0 tx=0x7fe300007900).stop 2022-06-29T16:08:19.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.625+0000 7fe30a1c1700 1 -- 172.21.15.60:0/1243261296 shutdown_connections 2022-06-29T16:08:19.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.625+0000 7fe30a1c1700 1 --2- 172.21.15.60:0/1243261296 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe2e4053b00 0x7fe2e4055fc0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:19.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.625+0000 7fe30a1c1700 1 --2- 172.21.15.60:0/1243261296 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe304146b40 0x7fe304140250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:19.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.625+0000 7fe30a1c1700 1 --2- 172.21.15.60:0/1243261296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3040b5eb0 0x7fe304151af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:19.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.625+0000 7fe30a1c1700 1 --2- 172.21.15.60:0/1243261296 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3040b0280 0x7fe3040af320 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:19.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.625+0000 7fe30a1c1700 1 -- 172.21.15.60:0/1243261296 >> 172.21.15.60:0/1243261296 conn(0x7fe304062090 msgr2=0x7fe3040b7350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:19.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.625+0000 7fe30a1c1700 1 -- 172.21.15.60:0/1243261296 shutdown_connections 2022-06-29T16:08:19.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.625+0000 7fe30a1c1700 1 -- 172.21.15.60:0/1243261296 wait complete. 2022-06-29T16:08:19.633 INFO:teuthology.orchestra.run.smithi060.stderr:updated caps for client.2 2022-06-29T16:08:19.643 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:08:19.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.737+0000 7f5f9ffff700 1 Processor -- start 2022-06-29T16:08:19.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.737+0000 7f5f9ffff700 1 -- start start 2022-06-29T16:08:19.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.737+0000 7f5f9ffff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5fa005d070 0x7f5fa014aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:19.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.737+0000 7f5f9ffff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fa005d450 con 0x7f5fa00b2130 2022-06-29T16:08:19.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.737+0000 7f5f9ffff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fa00ad840 con 0x7f5fa014af80 2022-06-29T16:08:19.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.737+0000 7f5f9ffff700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fa00adf90 con 0x7f5fa005d070 2022-06-29T16:08:19.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.737+0000 7f5f9effd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5fa005d070 0x7f5fa014aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:19.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.737+0000 7f5f9e7fc700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5fa00b2130 0x7f5fa00aca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34320/0 (socket says 172.21.15.60:34320) 2022-06-29T16:08:19.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.737+0000 7f5f9e7fc700 1 -- 172.21.15.60:0/1697542613 learned_addr learned my addr 172.21.15.60:0/1697542613 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:19.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.737+0000 7f5f9effd700 1 -- 172.21.15.60:0/1697542613 >> v1:172.21.15.92:6789/0 conn(0x7f5fa014af80 legacy=0x7f5fa00ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:19.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.737+0000 7f5f9effd700 1 -- 172.21.15.60:0/1697542613 >> v1:172.21.15.60:6789/0 conn(0x7f5fa00b2130 legacy=0x7f5fa00aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:19.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.737+0000 7f5f9effd700 1 -- 172.21.15.60:0/1697542613 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa00b25d0 con 0x7f5fa005d070 2022-06-29T16:08:19.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.737+0000 7f5f9effd700 1 --2- 172.21.15.60:0/1697542613 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fa005d070 0x7f5fa014aa90 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 rx=0x7f5f90002f70 tx=0x7f5f90006df0).ready entity=mon.0 client_cookie=2a0e78fb101817fe server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:19.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.737+0000 7f5f9dffb700 1 -- 172.21.15.60:0/1697542613 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f9000d610 con 0x7f5fa005d070 2022-06-29T16:08:19.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.737+0000 7f5f9dffb700 1 -- 172.21.15.60:0/1697542613 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f90000e90 con 0x7f5fa005d070 2022-06-29T16:08:19.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.737+0000 7f5f9dffb700 1 -- 172.21.15.60:0/1697542613 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f9000d960 con 0x7f5fa005d070 2022-06-29T16:08:19.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.737+0000 7f5f9ffff700 1 -- 172.21.15.60:0/1697542613 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fa005d070 msgr2=0x7f5fa014aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:19.747 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.737+0000 7f5f9ffff700 1 --2- 172.21.15.60:0/1697542613 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fa005d070 0x7f5fa014aa90 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 rx=0x7f5f90002f70 tx=0x7f5f90006df0).stop 2022-06-29T16:08:19.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.737+0000 7f5f9ffff700 1 -- 172.21.15.60:0/1697542613 shutdown_connections 2022-06-29T16:08:19.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.737+0000 7f5f9ffff700 1 --2- 172.21.15.60:0/1697542613 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fa005d070 0x7f5fa014aa90 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:19.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.737+0000 7f5f9ffff700 1 -- 172.21.15.60:0/1697542613 >> 172.21.15.60:0/1697542613 conn(0x7f5fa00b5890 msgr2=0x7f5fa00b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:19.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.737+0000 7f5f9ffff700 1 -- 172.21.15.60:0/1697542613 shutdown_connections 2022-06-29T16:08:19.748 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.737+0000 7f5f9ffff700 1 -- 172.21.15.60:0/1697542613 wait complete. 2022-06-29T16:08:19.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.741+0000 7f5f9ffff700 1 Processor -- start 2022-06-29T16:08:19.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.741+0000 7f5f9ffff700 1 -- start start 2022-06-29T16:08:19.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.741+0000 7f5f9ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fa005d070 0x7f5fa0066b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:19.752 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.741+0000 7f5f9ffff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fa00b2130 0x7f5fa0118690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:19.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.741+0000 7f5f9ffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fa014af80 0x7f5fa0141e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:19.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.741+0000 7f5f9ffff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa0118ea0 con 0x7f5fa005d070 2022-06-29T16:08:19.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.741+0000 7f5f9ffff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa0119140 con 0x7f5fa014af80 2022-06-29T16:08:19.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.741+0000 7f5f9ffff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa00656d0 con 0x7f5fa00b2130 2022-06-29T16:08:19.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.741+0000 7f5f9effd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fa005d070 0x7f5fa0066b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:19.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.741+0000 7f5f9effd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fa005d070 0x7f5fa0066b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54472/0 (socket says 172.21.15.60:54472) 2022-06-29T16:08:19.754 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.741+0000 7f5f9effd700 1 -- 172.21.15.60:0/2847427038 learned_addr learned my addr 172.21.15.60:0/2847427038 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:19.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.741+0000 7f5f9f7fe700 1 --2- 172.21.15.60:0/2847427038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fa014af80 0x7f5fa0141e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:19.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.741+0000 7f5f9e7fc700 1 --2- 172.21.15.60:0/2847427038 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fa00b2130 0x7f5fa0118690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:19.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.741+0000 7f5f9effd700 1 -- 172.21.15.60:0/2847427038 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fa00b2130 msgr2=0x7f5fa0118690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:19.755 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.741+0000 7f5f9effd700 1 --2- 172.21.15.60:0/2847427038 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fa00b2130 0x7f5fa0118690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:19.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.741+0000 7f5f9effd700 1 -- 172.21.15.60:0/2847427038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fa014af80 msgr2=0x7f5fa0141e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:19.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.741+0000 7f5f9effd700 1 --2- 172.21.15.60:0/2847427038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fa014af80 0x7f5fa0141e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:19.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.741+0000 7f5f9effd700 1 -- 172.21.15.60:0/2847427038 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f9000b180 con 0x7f5fa005d070 2022-06-29T16:08:19.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.741+0000 7f5f9e7fc700 1 --2- 172.21.15.60:0/2847427038 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fa00b2130 0x7f5fa0118690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:08:19.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.741+0000 7f5f9effd700 1 --2- 172.21.15.60:0/2847427038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fa005d070 0x7f5fa0066b20 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 rx=0x7f5f90002f70 tx=0x7f5f90015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:19.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.741+0000 7f5f8ffff700 1 -- 172.21.15.60:0/2847427038 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f9000d990 con 0x7f5fa005d070 2022-06-29T16:08:19.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.741+0000 7f5f8ffff700 1 -- 172.21.15.60:0/2847427038 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f90011bf0 con 0x7f5fa005d070 2022-06-29T16:08:19.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.741+0000 7f5f9ffff700 1 -- 172.21.15.60:0/2847427038 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa0065b70 con 0x7f5fa005d070 2022-06-29T16:08:19.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.741+0000 7f5f8ffff700 1 -- 172.21.15.60:0/2847427038 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f9001e460 con 0x7f5fa005d070 2022-06-29T16:08:19.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.741+0000 7f5f9ffff700 1 -- 172.21.15.60:0/2847427038 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa0116bf0 con 0x7f5fa005d070 2022-06-29T16:08:19.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.745+0000 7f5f9ffff700 1 -- 172.21.15.60:0/2847427038 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fa014a350 con 0x7f5fa005d070 2022-06-29T16:08:19.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.745+0000 7f5f8ffff700 1 -- 172.21.15.60:0/2847427038 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5f90025030 con 0x7f5fa005d070 2022-06-29T16:08:19.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.745+0000 7f5f8ffff700 1 --2- 172.21.15.60:0/2847427038 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5f7c053b30 0x7f5f7c055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:19.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.745+0000 7f5f8ffff700 1 -- 172.21.15.60:0/2847427038 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5f9006d330 con 0x7f5fa005d070 2022-06-29T16:08:19.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.745+0000 7f5f9e7fc700 1 --2- 172.21.15.60:0/2847427038 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5f7c053b30 0x7f5f7c055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:19.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.745+0000 7f5f9e7fc700 1 --2- 172.21.15.60:0/2847427038 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5f7c053b30 0x7f5f7c055ff0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7f5f9400b000 tx=0x7f5f94009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:19.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.749+0000 7f5f8ffff700 1 -- 172.21.15.60:0/2847427038 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5f90044140 con 0x7f5fa005d070 2022-06-29T16:08:19.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.909+0000 7f5f9ffff700 1 -- 172.21.15.60:0/2847427038 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5fa0065890 con 0x7f5fa005d070 2022-06-29T16:08:19.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.913+0000 7f5f8ffff700 1 -- 172.21.15.60:0/2847427038 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+2139 (secure 0 0 0) 0x7f5f90023090 con 0x7f5fa005d070 2022-06-29T16:08:19.917 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:08:19.917 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":73,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7457,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.92:6835/698111951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":698111951},{"type":"v1","addr":"172.21.15.92:6835","nonce":698111951}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":72}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:08:17.075855+0000","modified":"2022-06-29T16:08:18.079541+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8790},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8790":{"gid":8790,"name":"a","rank":0,"incarnation":72,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/3619856083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3619856083},{"type":"v1","addr":"172.21.15.60:6833","nonce":3619856083}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2022-06-29T16:08:19.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.913+0000 7f5f9ffff700 1 -- 172.21.15.60:0/2847427038 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5f7c053b30 msgr2=0x7f5f7c055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:19.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.913+0000 7f5f9ffff700 1 --2- 172.21.15.60:0/2847427038 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5f7c053b30 0x7f5f7c055ff0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7f5f9400b000 tx=0x7f5f94009040).stop 2022-06-29T16:08:19.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.913+0000 7f5f9ffff700 1 -- 172.21.15.60:0/2847427038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fa005d070 msgr2=0x7f5fa0066b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:19.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.913+0000 7f5f9ffff700 1 --2- 172.21.15.60:0/2847427038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fa005d070 0x7f5fa0066b20 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 rx=0x7f5f90002f70 tx=0x7f5f90015000).stop 2022-06-29T16:08:19.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.913+0000 7f5f9ffff700 1 -- 172.21.15.60:0/2847427038 shutdown_connections 2022-06-29T16:08:19.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.913+0000 7f5f9ffff700 1 --2- 172.21.15.60:0/2847427038 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5f7c053b30 0x7f5f7c055ff0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:19.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.913+0000 7f5f9ffff700 1 --2- 172.21.15.60:0/2847427038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5fa014af80 0x7f5fa0141e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:19.920 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.913+0000 7f5f9ffff700 1 --2- 172.21.15.60:0/2847427038 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5fa00b2130 0x7f5fa0118690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:19.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.913+0000 7f5f9ffff700 1 --2- 172.21.15.60:0/2847427038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fa005d070 0x7f5fa0066b20 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:19.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.913+0000 7f5f9ffff700 1 -- 172.21.15.60:0/2847427038 >> 172.21.15.60:0/2847427038 conn(0x7f5fa00b5890 msgr2=0x7f5fa01433b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:19.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.913+0000 7f5f9ffff700 1 -- 172.21.15.60:0/2847427038 shutdown_connections 2022-06-29T16:08:19.921 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:19.913+0000 7f5f9ffff700 1 -- 172.21.15.60:0/2847427038 wait complete. 2022-06-29T16:08:19.922 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 73 2022-06-29T16:08:20.132 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:08:17.075855+0000', 'modified': '2022-06-29T16:08:18.079541+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8790}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8790': {'gid': 8790, 'name': 'a', 'rank': 0, 'incarnation': 72, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.60:6833/3619856083', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6832', 'nonce': 3619856083}, {'type': 'v1', 'addr': '172.21.15.60:6833', 'nonce': 3619856083}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:08:20.133 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:08:20.134 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-06-29T16:08:20.260 INFO:teuthology.orchestra.run.smithi060.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":8790,"want_state":"up:active","state":"up:active","rank_uptime":3.168501086,"mdsmap_epoch":73,"osdmap_epoch":147,"osdmap_epoch_barrier":145,"uptime":7.5183705659999998} 2022-06-29T16:08:20.261 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 8790, "mdsmap_epoch": 73, "osdmap_epoch": 147, "osdmap_epoch_barrier": 145, "rank_uptime": 3.168501086, "state": "up:active", "uptime": 7.518370566, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:08:20.262 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:08:20.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.369+0000 7fc1c4008700 1 Processor -- start 2022-06-29T16:08:20.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.369+0000 7fc1c4008700 1 -- start start 2022-06-29T16:08:20.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.369+0000 7fc1c4008700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc1bc063ad0 0x7fc1bc067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:20.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.369+0000 7fc1c4008700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc1bc14af10 0x7fc1bc05c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:20.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.369+0000 7fc1c4008700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1bc0acba0 con 0x7fc1bc0b5950 2022-06-29T16:08:20.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.369+0000 7fc1c4008700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1bc14b3a0 con 0x7fc1bc14af10 2022-06-29T16:08:20.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.369+0000 7fc1c4008700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1bc067220 con 0x7fc1bc063ad0 2022-06-29T16:08:20.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.369+0000 7fc1c25a5700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc1bc14af10 0x7fc1bc05c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:20.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.369+0000 7fc1c15a3700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc1bc063ad0 0x7fc1bc067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:20.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.369+0000 7fc1c25a5700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fc1bc14af10 0x7fc1bc05c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54484/0 (socket says 172.21.15.60:54484) 2022-06-29T16:08:20.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.369+0000 7fc1c25a5700 1 -- 172.21.15.60:0/831710351 learned_addr learned my addr 172.21.15.60:0/831710351 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:20.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.369+0000 7fc1c15a3700 1 -- 172.21.15.60:0/831710351 >> v2:172.21.15.60:3300/0 conn(0x7fc1bc14af10 msgr2=0x7fc1bc05c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:20.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.369+0000 7fc1c15a3700 1 --2- 172.21.15.60:0/831710351 >> v2:172.21.15.60:3300/0 conn(0x7fc1bc14af10 0x7fc1bc05c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:20.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.369+0000 7fc1c15a3700 1 -- 172.21.15.60:0/831710351 >> v1:172.21.15.92:6789/0 conn(0x7fc1bc0b5950 legacy=0x7fc1bc05bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:20.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.369+0000 7fc1c15a3700 1 -- 172.21.15.60:0/831710351 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1bc0b5d30 con 0x7fc1bc063ad0 2022-06-29T16:08:20.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.369+0000 7fc1c25a5700 1 --2- 172.21.15.60:0/831710351 >> v2:172.21.15.60:3300/0 conn(0x7fc1bc14af10 0x7fc1bc05c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:08:20.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.369+0000 7fc1c15a3700 1 --2- 172.21.15.60:0/831710351 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1bc063ad0 0x7fc1bc067680 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0x7fc1ac002f70 tx=0x7fc1ac006df0).ready entity=mon.1 client_cookie=804ed812ef5e3c00 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:20.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.369+0000 7fc1c0da2700 1 -- 172.21.15.60:0/831710351 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc1ac00d570 con 0x7fc1bc063ad0 2022-06-29T16:08:20.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.369+0000 7fc1c0da2700 1 -- 172.21.15.60:0/831710351 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1ac000e90 con 0x7fc1bc063ad0 2022-06-29T16:08:20.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.369+0000 7fc1c0da2700 1 -- 172.21.15.60:0/831710351 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc1ac00d8c0 con 0x7fc1bc063ad0 2022-06-29T16:08:20.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.369+0000 7fc1c4008700 1 -- 172.21.15.60:0/831710351 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1bc063ad0 msgr2=0x7fc1bc067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:20.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.369+0000 7fc1c4008700 1 --2- 172.21.15.60:0/831710351 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1bc063ad0 0x7fc1bc067680 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0x7fc1ac002f70 tx=0x7fc1ac006df0).stop 2022-06-29T16:08:20.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.369+0000 7fc1c4008700 1 -- 172.21.15.60:0/831710351 shutdown_connections 2022-06-29T16:08:20.383 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.369+0000 7fc1c4008700 1 --2- 172.21.15.60:0/831710351 >> v2:172.21.15.60:3300/0 conn(0x7fc1bc14af10 0x7fc1bc05c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:20.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.369+0000 7fc1c4008700 1 --2- 172.21.15.60:0/831710351 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1bc063ad0 0x7fc1bc067680 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:20.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.369+0000 7fc1c4008700 1 -- 172.21.15.60:0/831710351 >> 172.21.15.60:0/831710351 conn(0x7fc1bc0632c0 msgr2=0x7fc1bc0b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:20.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.369+0000 7fc1c4008700 1 -- 172.21.15.60:0/831710351 shutdown_connections 2022-06-29T16:08:20.384 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.369+0000 7fc1c4008700 1 -- 172.21.15.60:0/831710351 wait complete. 2022-06-29T16:08:20.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.373+0000 7fc1c4008700 1 Processor -- start 2022-06-29T16:08:20.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.373+0000 7fc1c4008700 1 -- start start 2022-06-29T16:08:20.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.373+0000 7fc1c4008700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1bc063ad0 0x7fc1bc0af750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:20.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.373+0000 7fc1c4008700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1bc0b5950 0x7fc1bc0b40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:20.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.373+0000 7fc1c4008700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1bc14af10 0x7fc1bc159da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:20.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.373+0000 7fc1c4008700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1bc0b08f0 con 0x7fc1bc063ad0 2022-06-29T16:08:20.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.373+0000 7fc1c4008700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1bc142cd0 con 0x7fc1bc14af10 2022-06-29T16:08:20.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.373+0000 7fc1c4008700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1bc0b0b90 con 0x7fc1bc0b5950 2022-06-29T16:08:20.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.373+0000 7fc1c1da4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1bc063ad0 0x7fc1bc0af750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:20.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.373+0000 7fc1c25a5700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1bc14af10 0x7fc1bc159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:20.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.373+0000 7fc1c1da4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1bc063ad0 0x7fc1bc0af750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54488/0 (socket says 172.21.15.60:54488) 2022-06-29T16:08:20.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.373+0000 7fc1c1da4700 1 -- 172.21.15.60:0/4165798900 learned_addr learned my addr 172.21.15.60:0/4165798900 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:20.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.373+0000 7fc1c15a3700 1 --2- 172.21.15.60:0/4165798900 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1bc0b5950 0x7fc1bc0b40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:20.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.373+0000 7fc1c1da4700 1 -- 172.21.15.60:0/4165798900 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1bc0b5950 msgr2=0x7fc1bc0b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:20.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.373+0000 7fc1c1da4700 1 --2- 172.21.15.60:0/4165798900 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1bc0b5950 0x7fc1bc0b40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:20.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.373+0000 7fc1c1da4700 1 -- 172.21.15.60:0/4165798900 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1bc14af10 msgr2=0x7fc1bc159da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:20.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.373+0000 7fc1c1da4700 1 --2- 172.21.15.60:0/4165798900 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1bc14af10 0x7fc1bc159da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:20.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.373+0000 7fc1c1da4700 1 -- 172.21.15.60:0/4165798900 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b4009530 con 0x7fc1bc063ad0 2022-06-29T16:08:20.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.373+0000 7fc1c1da4700 1 --2- 172.21.15.60:0/4165798900 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1bc063ad0 0x7fc1bc0af750 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 rx=0x7fc1bc0ad0a0 tx=0x7fc1b8007be0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:20.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.373+0000 7fc1aaffd700 1 -- 172.21.15.60:0/4165798900 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc1b8007030 con 0x7fc1bc063ad0 2022-06-29T16:08:20.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.373+0000 7fc1aaffd700 1 -- 172.21.15.60:0/4165798900 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b8007190 con 0x7fc1bc063ad0 2022-06-29T16:08:20.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.373+0000 7fc1aaffd700 1 -- 172.21.15.60:0/4165798900 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc1b8006510 con 0x7fc1bc063ad0 2022-06-29T16:08:20.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.373+0000 7fc1c4008700 1 -- 172.21.15.60:0/4165798900 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1ac00b180 con 0x7fc1bc063ad0 2022-06-29T16:08:20.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.373+0000 7fc1c4008700 1 -- 172.21.15.60:0/4165798900 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1bc117060 con 0x7fc1bc063ad0 2022-06-29T16:08:20.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.377+0000 7fc1c4008700 1 -- 172.21.15.60:0/4165798900 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1bc0b0d50 con 0x7fc1bc063ad0 2022-06-29T16:08:20.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.377+0000 7fc1aaffd700 1 -- 172.21.15.60:0/4165798900 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc1b8006670 con 0x7fc1bc063ad0 2022-06-29T16:08:20.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.377+0000 7fc1aaffd700 1 --2- 172.21.15.60:0/4165798900 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc19c053a70 0x7fc19c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:20.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.377+0000 7fc1aaffd700 1 -- 172.21.15.60:0/4165798900 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc1b806b910 con 0x7fc1bc063ad0 2022-06-29T16:08:20.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.377+0000 7fc1c15a3700 1 --2- 172.21.15.60:0/4165798900 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc19c053a70 0x7fc19c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:20.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.377+0000 7fc1c15a3700 1 --2- 172.21.15.60:0/4165798900 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc19c053a70 0x7fc19c055f30 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0x7fc1ac01a000 tx=0x7fc1ac014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:20.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.385+0000 7fc1aaffd700 1 -- 172.21.15.60:0/4165798900 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc1b80426c0 con 0x7fc1bc063ad0 2022-06-29T16:08:20.516 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.513+0000 7fc1c4008700 1 -- 172.21.15.60:0/4165798900 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1bc159830 con 0x7fc1bc063ad0 2022-06-29T16:08:20.518 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.513+0000 7fc1aaffd700 1 -- 172.21.15.60:0/4165798900 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v147) v1 ==== 74+0+16052 (secure 0 0 0) 0x7fc1b8045ce0 con 0x7fc1bc063ad0 2022-06-29T16:08:20.518 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:08:20.519 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":147,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:08:19.091282+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:08:15.820434+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":30,"pool_name":"cephfs_data","create_time":"2022-06-29T16:08:16.484716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"147","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:08:20.520 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.517+0000 7fc1c4008700 1 -- 172.21.15.60:0/4165798900 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc19c053a70 msgr2=0x7fc19c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:20.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.517+0000 7fc1c4008700 1 --2- 172.21.15.60:0/4165798900 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc19c053a70 0x7fc19c055f30 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0x7fc1ac01a000 tx=0x7fc1ac014000).stop 2022-06-29T16:08:20.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.517+0000 7fc1c4008700 1 -- 172.21.15.60:0/4165798900 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1bc063ad0 msgr2=0x7fc1bc0af750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:20.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.517+0000 7fc1c4008700 1 --2- 172.21.15.60:0/4165798900 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1bc063ad0 0x7fc1bc0af750 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 rx=0x7fc1bc0ad0a0 tx=0x7fc1b8007be0).stop 2022-06-29T16:08:20.521 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.517+0000 7fc1c4008700 1 -- 172.21.15.60:0/4165798900 shutdown_connections 2022-06-29T16:08:20.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.517+0000 7fc1c4008700 1 --2- 172.21.15.60:0/4165798900 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc19c053a70 0x7fc19c055f30 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:20.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.517+0000 7fc1c4008700 1 --2- 172.21.15.60:0/4165798900 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1bc14af10 0x7fc1bc159da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:20.522 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.517+0000 7fc1c4008700 1 --2- 172.21.15.60:0/4165798900 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc1bc0b5950 0x7fc1bc0b40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:20.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.517+0000 7fc1c4008700 1 --2- 172.21.15.60:0/4165798900 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc1bc063ad0 0x7fc1bc0af750 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:20.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.517+0000 7fc1c4008700 1 -- 172.21.15.60:0/4165798900 >> 172.21.15.60:0/4165798900 conn(0x7fc1bc0632c0 msgr2=0x7fc1bc1433a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:20.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.517+0000 7fc1c4008700 1 -- 172.21.15.60:0/4165798900 shutdown_connections 2022-06-29T16:08:20.523 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.517+0000 7fc1c4008700 1 -- 172.21.15.60:0/4165798900 wait complete. 2022-06-29T16:08:20.532 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 29, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-29T16:08:15.820434+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '146', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 30, 'pool_name': 'cephfs_data', 'create_time': '2022-06-29T16:08:16.484716+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '147', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-29T16:08:20.533 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:08:20.646 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.641+0000 7faf51311700 1 Processor -- start 2022-06-29T16:08:20.646 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.641+0000 7faf51311700 1 -- start start 2022-06-29T16:08:20.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.645+0000 7faf51311700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faf4c146a50 0x7faf4c14e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:20.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.645+0000 7faf51311700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf4c0b3600 con 0x7faf4c0b5270 2022-06-29T16:08:20.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.645+0000 7faf51311700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf4c0b09f0 con 0x7faf4c0b9c50 2022-06-29T16:08:20.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.645+0000 7faf51311700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf4c0b10e0 con 0x7faf4c146a50 2022-06-29T16:08:20.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.645+0000 7faf437fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7faf4c0b5270 0x7faf4c146340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34350/0 (socket says 172.21.15.60:34350) 2022-06-29T16:08:20.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.645+0000 7faf437fe700 1 -- 172.21.15.60:0/585932706 learned_addr learned my addr 172.21.15.60:0/585932706 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:20.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.645+0000 7faf50b10700 1 --2- 172.21.15.60:0/585932706 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faf4c146a50 0x7faf4c14e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:20.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.645+0000 7faf4b7fe700 1 -- 172.21.15.60:0/585932706 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2181194357 0 0) 0x7faf4c0b3600 con 0x7faf4c0b5270 2022-06-29T16:08:20.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.645+0000 7faf4b7fe700 1 -- 172.21.15.60:0/585932706 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf2c003260 con 0x7faf4c0b5270 2022-06-29T16:08:20.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.645+0000 7faf4b7fe700 1 -- 172.21.15.60:0/585932706 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285062266 0 0) 0x7faf4c0b09f0 con 0x7faf4c0b9c50 2022-06-29T16:08:20.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.645+0000 7faf4b7fe700 1 -- 172.21.15.60:0/585932706 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf4c0b3600 con 0x7faf4c0b9c50 2022-06-29T16:08:20.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.645+0000 7faf50b10700 1 -- 172.21.15.60:0/585932706 >> v1:172.21.15.92:6789/0 conn(0x7faf4c0b9c50 legacy=0x7faf4c0b2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:20.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.645+0000 7faf4b7fe700 1 -- 172.21.15.60:0/585932706 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4236390954 0 0) 0x7faf2c003260 con 0x7faf4c0b5270 2022-06-29T16:08:20.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.645+0000 7faf50b10700 1 -- 172.21.15.60:0/585932706 >> v1:172.21.15.60:6789/0 conn(0x7faf4c0b5270 legacy=0x7faf4c146340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:20.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.645+0000 7faf50b10700 1 -- 172.21.15.60:0/585932706 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf4c0ba030 con 0x7faf4c146a50 2022-06-29T16:08:20.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.645+0000 7faf50b10700 1 --2- 172.21.15.60:0/585932706 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faf4c146a50 0x7faf4c14e3e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7faf34001110 tx=0x7faf440091d0).ready entity=mon.2 client_cookie=1d3e58d2db7a05a4 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:20.655 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.645+0000 7faf4b7fe700 1 -- 172.21.15.60:0/585932706 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf34001680 con 0x7faf4c146a50 2022-06-29T16:08:20.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.645+0000 7faf4b7fe700 1 -- 172.21.15.60:0/585932706 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf4400f510 con 0x7faf4c146a50 2022-06-29T16:08:20.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.645+0000 7faf4b7fe700 1 -- 172.21.15.60:0/585932706 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf4400f680 con 0x7faf4c146a50 2022-06-29T16:08:20.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.645+0000 7faf51311700 1 -- 172.21.15.60:0/585932706 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faf4c146a50 msgr2=0x7faf4c14e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:20.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.645+0000 7faf51311700 1 --2- 172.21.15.60:0/585932706 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faf4c146a50 0x7faf4c14e3e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7faf34001110 tx=0x7faf440091d0).stop 2022-06-29T16:08:20.656 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.645+0000 7faf51311700 1 -- 172.21.15.60:0/585932706 shutdown_connections 2022-06-29T16:08:20.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.645+0000 7faf51311700 1 --2- 172.21.15.60:0/585932706 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faf4c146a50 0x7faf4c14e3e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:20.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.645+0000 7faf51311700 1 -- 172.21.15.60:0/585932706 >> 172.21.15.60:0/585932706 conn(0x7faf4c062c00 msgr2=0x7faf4c0b9200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:20.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.645+0000 7faf51311700 1 -- 172.21.15.60:0/585932706 shutdown_connections 2022-06-29T16:08:20.657 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.645+0000 7faf51311700 1 -- 172.21.15.60:0/585932706 wait complete. 2022-06-29T16:08:20.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.649+0000 7faf51311700 1 Processor -- start 2022-06-29T16:08:20.661 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.649+0000 7faf51311700 1 -- start start 2022-06-29T16:08:20.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.649+0000 7faf51311700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faf4c0b5270 0x7faf4c05c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:20.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.649+0000 7faf51311700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faf4c0b9c50 0x7faf4c0ae0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:20.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.649+0000 7faf51311700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf4c146a50 0x7faf4c0aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:20.662 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.649+0000 7faf51311700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4c114520 con 0x7faf4c146a50 2022-06-29T16:08:20.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.649+0000 7faf51311700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4c0ae760 con 0x7faf4c0b9c50 2022-06-29T16:08:20.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.649+0000 7faf51311700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf4c0aea00 con 0x7faf4c0b5270 2022-06-29T16:08:20.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.649+0000 7faf50b10700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf4c146a50 0x7faf4c0aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:20.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.649+0000 7faf50b10700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf4c146a50 0x7faf4c0aca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54502/0 (socket says 172.21.15.60:54502) 2022-06-29T16:08:20.663 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.649+0000 7faf50b10700 1 -- 172.21.15.60:0/2890797225 learned_addr learned my addr 172.21.15.60:0/2890797225 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:20.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.649+0000 7faf437fe700 1 --2- 172.21.15.60:0/2890797225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faf4c0b9c50 0x7faf4c0ae0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:20.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.649+0000 7faf4bfff700 1 --2- 172.21.15.60:0/2890797225 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faf4c0b5270 0x7faf4c05c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:20.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.649+0000 7faf50b10700 1 -- 172.21.15.60:0/2890797225 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faf4c0b5270 msgr2=0x7faf4c05c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:20.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.649+0000 7faf50b10700 1 --2- 172.21.15.60:0/2890797225 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faf4c0b5270 0x7faf4c05c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:20.664 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.649+0000 7faf50b10700 1 -- 172.21.15.60:0/2890797225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faf4c0b9c50 msgr2=0x7faf4c0ae0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:20.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.649+0000 7faf50b10700 1 --2- 172.21.15.60:0/2890797225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faf4c0b9c50 0x7faf4c0ae0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:20.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.649+0000 7faf50b10700 1 -- 172.21.15.60:0/2890797225 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf44009470 con 0x7faf4c146a50 2022-06-29T16:08:20.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.649+0000 7faf437fe700 1 --2- 172.21.15.60:0/2890797225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faf4c0b9c50 0x7faf4c0ae0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:08:20.665 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.649+0000 7faf50b10700 1 --2- 172.21.15.60:0/2890797225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf4c146a50 0x7faf4c0aca20 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 rx=0x7faf34001110 tx=0x7faf44016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:20.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.649+0000 7faf49ffb700 1 -- 172.21.15.60:0/2890797225 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf440032d0 con 0x7faf4c146a50 2022-06-29T16:08:20.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.649+0000 7faf51311700 1 -- 172.21.15.60:0/2890797225 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf4c0af500 con 0x7faf4c146a50 2022-06-29T16:08:20.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.649+0000 7faf51311700 1 -- 172.21.15.60:0/2890797225 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf4c114950 con 0x7faf4c146a50 2022-06-29T16:08:20.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.649+0000 7faf49ffb700 1 -- 172.21.15.60:0/2890797225 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf44011c30 con 0x7faf4c146a50 2022-06-29T16:08:20.666 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.649+0000 7faf49ffb700 1 -- 172.21.15.60:0/2890797225 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf44020400 con 0x7faf4c146a50 2022-06-29T16:08:20.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.653+0000 7faf51311700 1 -- 172.21.15.60:0/2890797225 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf4c0ae5d0 con 0x7faf4c146a50 2022-06-29T16:08:20.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.653+0000 7faf49ffb700 1 -- 172.21.15.60:0/2890797225 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7faf44027030 con 0x7faf4c146a50 2022-06-29T16:08:20.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.653+0000 7faf49ffb700 1 --2- 172.21.15.60:0/2890797225 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7faf300539e0 0x7faf30055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:20.667 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.653+0000 7faf49ffb700 1 -- 172.21.15.60:0/2890797225 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6073+0+0 (secure 0 0 0) 0x7faf4406cb80 con 0x7faf4c146a50 2022-06-29T16:08:20.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.653+0000 7faf4bfff700 1 --2- 172.21.15.60:0/2890797225 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7faf300539e0 0x7faf30055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:20.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.657+0000 7faf4bfff700 1 --2- 172.21.15.60:0/2890797225 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7faf300539e0 0x7faf30055ea0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0x7faf3c00b000 tx=0x7faf3c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:20.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.665+0000 7faf49ffb700 1 -- 172.21.15.60:0/2890797225 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7faf44043a40 con 0x7faf4c146a50 2022-06-29T16:08:20.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.789+0000 7faf51311700 1 -- 172.21.15.60:0/2890797225 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7faf4c114660 con 0x7faf4c146a50 2022-06-29T16:08:20.796 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.789+0000 7faf49ffb700 1 -- 172.21.15.60:0/2890797225 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v147) v1 ==== 110+0+10 (secure 0 0 0) 0x7faf44047020 con 0x7faf4c146a50 2022-06-29T16:08:20.796 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:08:20.798 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.793+0000 7faf51311700 1 -- 172.21.15.60:0/2890797225 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7faf300539e0 msgr2=0x7faf30055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:20.799 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.793+0000 7faf51311700 1 --2- 172.21.15.60:0/2890797225 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7faf300539e0 0x7faf30055ea0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0x7faf3c00b000 tx=0x7faf3c009040).stop 2022-06-29T16:08:20.799 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.793+0000 7faf51311700 1 -- 172.21.15.60:0/2890797225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf4c146a50 msgr2=0x7faf4c0aca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:20.799 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.793+0000 7faf51311700 1 --2- 172.21.15.60:0/2890797225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf4c146a50 0x7faf4c0aca20 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 rx=0x7faf34001110 tx=0x7faf44016000).stop 2022-06-29T16:08:20.800 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.793+0000 7faf51311700 1 -- 172.21.15.60:0/2890797225 shutdown_connections 2022-06-29T16:08:20.800 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.793+0000 7faf51311700 1 --2- 172.21.15.60:0/2890797225 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7faf300539e0 0x7faf30055ea0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:20.800 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.793+0000 7faf51311700 1 --2- 172.21.15.60:0/2890797225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faf4c146a50 0x7faf4c0aca20 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:20.800 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.793+0000 7faf51311700 1 --2- 172.21.15.60:0/2890797225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faf4c0b9c50 0x7faf4c0ae0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:20.801 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.793+0000 7faf51311700 1 --2- 172.21.15.60:0/2890797225 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7faf4c0b5270 0x7faf4c05c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:20.801 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.793+0000 7faf51311700 1 -- 172.21.15.60:0/2890797225 >> 172.21.15.60:0/2890797225 conn(0x7faf4c062c00 msgr2=0x7faf4c0b86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:20.801 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.793+0000 7faf51311700 1 -- 172.21.15.60:0/2890797225 shutdown_connections 2022-06-29T16:08:20.801 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.793+0000 7faf51311700 1 -- 172.21.15.60:0/2890797225 wait complete. 2022-06-29T16:08:20.809 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:08:20.918 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.913+0000 7fe32359e700 1 Processor -- start 2022-06-29T16:08:20.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.913+0000 7fe32359e700 1 -- start start 2022-06-29T16:08:20.923 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.913+0000 7fe32359e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe3240b3c70 0x7fe324065a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:20.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.913+0000 7fe32359e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe324145eb0 0x7fe324065f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:20.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.917+0000 7fe32359e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe32405c2b0 con 0x7fe3240aca20 2022-06-29T16:08:20.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.917+0000 7fe32359e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3240b4050 con 0x7fe324145eb0 2022-06-29T16:08:20.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.917+0000 7fe32359e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe32405d120 con 0x7fe3240b3c70 2022-06-29T16:08:20.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.917+0000 7fe321d9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe324145eb0 0x7fe324065f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:20.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.917+0000 7fe322d9d700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fe3240aca20 0x7fe3240ace90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34366/0 (socket says 172.21.15.60:34366) 2022-06-29T16:08:20.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.917+0000 7fe322d9d700 1 -- 172.21.15.60:0/4046829063 learned_addr learned my addr 172.21.15.60:0/4046829063 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:20.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.917+0000 7fe32259c700 1 --2- 172.21.15.60:0/4046829063 >> v2:172.21.15.92:3300/0 conn(0x7fe3240b3c70 0x7fe324065a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:20.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.917+0000 7fe32159a700 1 -- 172.21.15.60:0/4046829063 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1936082913 0 0) 0x7fe32405c2b0 con 0x7fe3240aca20 2022-06-29T16:08:20.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.917+0000 7fe32159a700 1 -- 172.21.15.60:0/4046829063 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe304003260 con 0x7fe3240aca20 2022-06-29T16:08:20.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.917+0000 7fe321d9b700 1 -- 172.21.15.60:0/4046829063 >> v2:172.21.15.92:3300/0 conn(0x7fe3240b3c70 msgr2=0x7fe324065a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:20.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.917+0000 7fe321d9b700 1 --2- 172.21.15.60:0/4046829063 >> v2:172.21.15.92:3300/0 conn(0x7fe3240b3c70 0x7fe324065a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:20.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.917+0000 7fe321d9b700 1 -- 172.21.15.60:0/4046829063 >> v1:172.21.15.60:6789/0 conn(0x7fe3240aca20 legacy=0x7fe3240ace90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:20.927 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.917+0000 7fe321d9b700 1 -- 172.21.15.60:0/4046829063 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe32405d3c0 con 0x7fe324145eb0 2022-06-29T16:08:20.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.917+0000 7fe321d9b700 1 --2- 172.21.15.60:0/4046829063 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe324145eb0 0x7fe324065f10 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 rx=0x7fe30c002f70 tx=0x7fe30c006df0).ready entity=mon.0 client_cookie=628f9531f6db63cc server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:20.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.917+0000 7fe32159a700 1 -- 172.21.15.60:0/4046829063 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe30c00d4f0 con 0x7fe324145eb0 2022-06-29T16:08:20.928 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.917+0000 7fe32159a700 1 -- 172.21.15.60:0/4046829063 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe30c000e90 con 0x7fe324145eb0 2022-06-29T16:08:20.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.917+0000 7fe32159a700 1 -- 172.21.15.60:0/4046829063 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe30c00d840 con 0x7fe324145eb0 2022-06-29T16:08:20.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.917+0000 7fe32359e700 1 -- 172.21.15.60:0/4046829063 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe324145eb0 msgr2=0x7fe324065f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:20.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.917+0000 7fe32359e700 1 --2- 172.21.15.60:0/4046829063 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe324145eb0 0x7fe324065f10 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 rx=0x7fe30c002f70 tx=0x7fe30c006df0).stop 2022-06-29T16:08:20.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.917+0000 7fe32359e700 1 -- 172.21.15.60:0/4046829063 shutdown_connections 2022-06-29T16:08:20.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.917+0000 7fe32359e700 1 --2- 172.21.15.60:0/4046829063 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe324145eb0 0x7fe324065f10 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:20.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.917+0000 7fe32359e700 1 --2- 172.21.15.60:0/4046829063 >> v2:172.21.15.92:3300/0 conn(0x7fe3240b3c70 0x7fe324065a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:20.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.917+0000 7fe32359e700 1 -- 172.21.15.60:0/4046829063 >> 172.21.15.60:0/4046829063 conn(0x7fe324064810 msgr2=0x7fe324064bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:20.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.917+0000 7fe32359e700 1 -- 172.21.15.60:0/4046829063 shutdown_connections 2022-06-29T16:08:20.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.917+0000 7fe32359e700 1 -- 172.21.15.60:0/4046829063 wait complete. 2022-06-29T16:08:20.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.921+0000 7fe32359e700 1 Processor -- start 2022-06-29T16:08:20.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.921+0000 7fe32359e700 1 -- start start 2022-06-29T16:08:20.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.921+0000 7fe32359e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3240aca20 0x7fe3240af5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:20.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.921+0000 7fe32359e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3240b3c70 0x7fe3240b5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:20.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.921+0000 7fe32359e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe324145eb0 0x7fe3240b0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:20.937 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.921+0000 7fe32359e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe324145180 con 0x7fe3240aca20 2022-06-29T16:08:20.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.921+0000 7fe32359e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe324145420 con 0x7fe3240b3c70 2022-06-29T16:08:20.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.921+0000 7fe32359e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3241456c0 con 0x7fe324145eb0 2022-06-29T16:08:20.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.921+0000 7fe32259c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3240aca20 0x7fe3240af5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:20.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.921+0000 7fe32259c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3240aca20 0x7fe3240af5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54514/0 (socket says 172.21.15.60:54514) 2022-06-29T16:08:20.938 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.921+0000 7fe32259c700 1 -- 172.21.15.60:0/1989974887 learned_addr learned my addr 172.21.15.60:0/1989974887 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:20.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.921+0000 7fe321d9b700 1 --2- 172.21.15.60:0/1989974887 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3240b3c70 0x7fe3240b5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:20.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.921+0000 7fe322d9d700 1 --2- 172.21.15.60:0/1989974887 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe324145eb0 0x7fe3240b0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:20.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.925+0000 7fe321d9b700 1 -- 172.21.15.60:0/1989974887 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe324145eb0 msgr2=0x7fe3240b0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:20.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.925+0000 7fe321d9b700 1 --2- 172.21.15.60:0/1989974887 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe324145eb0 0x7fe3240b0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:20.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.925+0000 7fe321d9b700 1 -- 172.21.15.60:0/1989974887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3240aca20 msgr2=0x7fe3240af5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:20.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.925+0000 7fe321d9b700 1 --2- 172.21.15.60:0/1989974887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3240aca20 0x7fe3240af5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:20.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.925+0000 7fe321d9b700 1 -- 172.21.15.60:0/1989974887 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe30c00b180 con 0x7fe3240b3c70 2022-06-29T16:08:20.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.925+0000 7fe32259c700 1 --2- 172.21.15.60:0/1989974887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3240aca20 0x7fe3240af5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:08:20.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.925+0000 7fe321d9b700 1 --2- 172.21.15.60:0/1989974887 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3240b3c70 0x7fe3240b5f30 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7fe30c002f70 tx=0x7fe30c014000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:20.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.925+0000 7fe31b7fe700 1 -- 172.21.15.60:0/1989974887 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe30c013d80 con 0x7fe3240b3c70 2022-06-29T16:08:20.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.925+0000 7fe31b7fe700 1 -- 172.21.15.60:0/1989974887 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe30c01e400 con 0x7fe3240b3c70 2022-06-29T16:08:20.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.925+0000 7fe32359e700 1 -- 172.21.15.60:0/1989974887 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3240b6420 con 0x7fe3240b3c70 2022-06-29T16:08:20.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.925+0000 7fe31b7fe700 1 -- 172.21.15.60:0/1989974887 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe30c01e760 con 0x7fe3240b3c70 2022-06-29T16:08:20.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.925+0000 7fe32359e700 1 -- 172.21.15.60:0/1989974887 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe324140270 con 0x7fe3240b3c70 2022-06-29T16:08:20.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.925+0000 7fe32359e700 1 -- 172.21.15.60:0/1989974887 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe324154d10 con 0x7fe3240b3c70 2022-06-29T16:08:20.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.925+0000 7fe31b7fe700 1 -- 172.21.15.60:0/1989974887 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe30c01c070 con 0x7fe3240b3c70 2022-06-29T16:08:20.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.929+0000 7fe31b7fe700 1 --2- 172.21.15.60:0/1989974887 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe308053a70 0x7fe308055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:20.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.929+0000 7fe31b7fe700 1 -- 172.21.15.60:0/1989974887 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe30c06caa0 con 0x7fe3240b3c70 2022-06-29T16:08:20.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.929+0000 7fe32259c700 1 --2- 172.21.15.60:0/1989974887 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe308053a70 0x7fe308055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:20.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.929+0000 7fe32259c700 1 --2- 172.21.15.60:0/1989974887 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe308053a70 0x7fe308055f30 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0x7fe31400b000 tx=0x7fe314006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:20.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:20.937+0000 7fe31b7fe700 1 -- 172.21.15.60:0/1989974887 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe30c043920 con 0x7fe3240b3c70 2022-06-29T16:08:21.074 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.069+0000 7fe32359e700 1 -- 172.21.15.60:0/1989974887 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fe3240573d0 con 0x7fe3240b3c70 2022-06-29T16:08:21.075 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.069+0000 7fe31b7fe700 1 -- 172.21.15.60:0/1989974887 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v147) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe30c023020 con 0x7fe3240b3c70 2022-06-29T16:08:21.076 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:08:21.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.073+0000 7fe32359e700 1 -- 172.21.15.60:0/1989974887 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe308053a70 msgr2=0x7fe308055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:21.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.073+0000 7fe32359e700 1 --2- 172.21.15.60:0/1989974887 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe308053a70 0x7fe308055f30 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0x7fe31400b000 tx=0x7fe314006df0).stop 2022-06-29T16:08:21.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.073+0000 7fe32359e700 1 -- 172.21.15.60:0/1989974887 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3240b3c70 msgr2=0x7fe3240b5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:21.079 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.073+0000 7fe32359e700 1 --2- 172.21.15.60:0/1989974887 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3240b3c70 0x7fe3240b5f30 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7fe30c002f70 tx=0x7fe30c014000).stop 2022-06-29T16:08:21.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.073+0000 7fe32359e700 1 -- 172.21.15.60:0/1989974887 shutdown_connections 2022-06-29T16:08:21.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.073+0000 7fe32359e700 1 --2- 172.21.15.60:0/1989974887 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe308053a70 0x7fe308055f30 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:21.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.073+0000 7fe32359e700 1 --2- 172.21.15.60:0/1989974887 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe324145eb0 0x7fe3240b0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:21.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.073+0000 7fe32359e700 1 --2- 172.21.15.60:0/1989974887 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3240b3c70 0x7fe3240b5f30 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:21.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.073+0000 7fe32359e700 1 --2- 172.21.15.60:0/1989974887 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3240aca20 0x7fe3240af5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:21.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.073+0000 7fe32359e700 1 -- 172.21.15.60:0/1989974887 >> 172.21.15.60:0/1989974887 conn(0x7fe324064810 msgr2=0x7fe3240b6f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:21.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.073+0000 7fe32359e700 1 -- 172.21.15.60:0/1989974887 shutdown_connections 2022-06-29T16:08:21.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.073+0000 7fe32359e700 1 -- 172.21.15.60:0/1989974887 wait complete. 2022-06-29T16:08:21.089 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-29T16:08:21.192 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.189+0000 7f76d0b0f700 1 Processor -- start 2022-06-29T16:08:21.193 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.189+0000 7f76d0b0f700 1 -- start start 2022-06-29T16:08:21.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.189+0000 7f76d0b0f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76cc0b5950 0x7f76cc067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:21.195 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.189+0000 7f76d0b0f700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f76cc063ad0 0x7f76cc14af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:21.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.189+0000 7f76d0b0f700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76cc0acba0 con 0x7f76cc05bd20 2022-06-29T16:08:21.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.189+0000 7f76d0b0f700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76cc067220 con 0x7f76cc063ad0 2022-06-29T16:08:21.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.189+0000 7f76d0b0f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc0b5d30 con 0x7f76cc0b5950 2022-06-29T16:08:21.196 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.189+0000 7f76c1d9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f76cc063ad0 0x7f76cc14af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:21.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.189+0000 7f76c1d9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f76cc063ad0 0x7f76cc14af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54524/0 (socket says 172.21.15.60:54524) 2022-06-29T16:08:21.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.189+0000 7f76cad9d700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f76cc05bd20 0x7f76cc05c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34380/0 (socket says 172.21.15.60:34380) 2022-06-29T16:08:21.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.189+0000 7f76c1d9b700 1 -- 172.21.15.60:0/3254322345 learned_addr learned my addr 172.21.15.60:0/3254322345 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:21.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.189+0000 7f76ca59c700 1 --2- 172.21.15.60:0/3254322345 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76cc0b5950 0x7f76cc067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:21.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.189+0000 7f76c1d9b700 1 -- 172.21.15.60:0/3254322345 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76cc0b5950 msgr2=0x7f76cc067680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:21.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.189+0000 7f76c1d9b700 1 --2- 172.21.15.60:0/3254322345 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76cc0b5950 0x7f76cc067680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:21.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.189+0000 7f76c1d9b700 1 -- 172.21.15.60:0/3254322345 >> v1:172.21.15.60:6789/0 conn(0x7f76cc05bd20 legacy=0x7f76cc05c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:21.198 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.189+0000 7f76c9d9b700 1 -- 172.21.15.60:0/3254322345 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1578632206 0 0) 0x7f76cc0acba0 con 0x7f76cc05bd20 2022-06-29T16:08:21.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.189+0000 7f76c1d9b700 1 -- 172.21.15.60:0/3254322345 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76cc0ace00 con 0x7f76cc063ad0 2022-06-29T16:08:21.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.189+0000 7f76ca59c700 1 --2- 172.21.15.60:0/3254322345 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76cc0b5950 0x7f76cc067680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:08:21.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.189+0000 7f76c1d9b700 1 --2- 172.21.15.60:0/3254322345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76cc063ad0 0x7f76cc14af10 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 rx=0x7f76b4002f70 tx=0x7f76b4006eb0).ready entity=mon.0 client_cookie=2ab52bea8dd86064 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:21.199 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.189+0000 7f76c9d9b700 1 -- 172.21.15.60:0/3254322345 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76b400d5c0 con 0x7f76cc063ad0 2022-06-29T16:08:21.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.189+0000 7f76c9d9b700 1 -- 172.21.15.60:0/3254322345 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b4000e90 con 0x7f76cc063ad0 2022-06-29T16:08:21.200 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.189+0000 7f76c9d9b700 1 -- 172.21.15.60:0/3254322345 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76b400d910 con 0x7f76cc063ad0 2022-06-29T16:08:21.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.193+0000 7f76d0b0f700 1 -- 172.21.15.60:0/3254322345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76cc063ad0 msgr2=0x7f76cc14af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:21.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.193+0000 7f76d0b0f700 1 --2- 172.21.15.60:0/3254322345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76cc063ad0 0x7f76cc14af10 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 rx=0x7f76b4002f70 tx=0x7f76b4006eb0).stop 2022-06-29T16:08:21.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.193+0000 7f76d0b0f700 1 -- 172.21.15.60:0/3254322345 shutdown_connections 2022-06-29T16:08:21.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.193+0000 7f76d0b0f700 1 --2- 172.21.15.60:0/3254322345 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76cc063ad0 0x7f76cc14af10 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:21.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.193+0000 7f76d0b0f700 1 --2- 172.21.15.60:0/3254322345 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76cc0b5950 0x7f76cc067680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:21.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.193+0000 7f76d0b0f700 1 -- 172.21.15.60:0/3254322345 >> 172.21.15.60:0/3254322345 conn(0x7f76cc0632c0 msgr2=0x7f76cc0b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:21.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.193+0000 7f76d0b0f700 1 -- 172.21.15.60:0/3254322345 shutdown_connections 2022-06-29T16:08:21.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.193+0000 7f76d0b0f700 1 -- 172.21.15.60:0/3254322345 wait complete. 2022-06-29T16:08:21.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.193+0000 7f76d0b0f700 1 Processor -- start 2022-06-29T16:08:21.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.193+0000 7f76d0b0f700 1 -- start start 2022-06-29T16:08:21.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.197+0000 7f76d0b0f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76cc05bd20 0x7f76cc0ad530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:21.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.197+0000 7f76d0b0f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76cc063ad0 0x7f76cc14df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:21.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.197+0000 7f76d0b0f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76cc0b5950 0x7f76cc1592e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:21.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.197+0000 7f76d0b0f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc0adb60 con 0x7f76cc05bd20 2022-06-29T16:08:21.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.197+0000 7f76d0b0f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc0ade00 con 0x7f76cc0b5950 2022-06-29T16:08:21.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.197+0000 7f76d0b0f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc0af630 con 0x7f76cc063ad0 2022-06-29T16:08:21.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.197+0000 7f76ca59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76cc05bd20 0x7f76cc0ad530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:21.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.197+0000 7f76ca59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76cc05bd20 0x7f76cc0ad530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54528/0 (socket says 172.21.15.60:54528) 2022-06-29T16:08:21.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.197+0000 7f76ca59c700 1 -- 172.21.15.60:0/948643541 learned_addr learned my addr 172.21.15.60:0/948643541 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:21.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.197+0000 7f76ca59c700 1 -- 172.21.15.60:0/948643541 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76cc063ad0 msgr2=0x7f76cc14df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:21.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.197+0000 7f76c1d9b700 1 --2- 172.21.15.60:0/948643541 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76cc063ad0 0x7f76cc14df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:21.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.197+0000 7f76cad9d700 1 --2- 172.21.15.60:0/948643541 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76cc0b5950 0x7f76cc1592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:21.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.197+0000 7f76ca59c700 1 --2- 172.21.15.60:0/948643541 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76cc063ad0 0x7f76cc14df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:21.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.197+0000 7f76ca59c700 1 -- 172.21.15.60:0/948643541 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76cc0b5950 msgr2=0x7f76cc1592e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:21.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.197+0000 7f76ca59c700 1 --2- 172.21.15.60:0/948643541 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76cc0b5950 0x7f76cc1592e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:21.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.197+0000 7f76ca59c700 1 -- 172.21.15.60:0/948643541 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b400b1e0 con 0x7f76cc05bd20 2022-06-29T16:08:21.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.197+0000 7f76cad9d700 1 --2- 172.21.15.60:0/948643541 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76cc0b5950 0x7f76cc1592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:08:21.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.197+0000 7f76c1d9b700 1 --2- 172.21.15.60:0/948643541 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76cc063ad0 0x7f76cc14df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:08:21.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.197+0000 7f76ca59c700 1 --2- 172.21.15.60:0/948643541 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76cc05bd20 0x7f76cc0ad530 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 rx=0x7f76bc006940 tx=0x7f76bc008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:21.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.197+0000 7f76c3fff700 1 -- 172.21.15.60:0/948643541 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76bc011780 con 0x7f76cc05bd20 2022-06-29T16:08:21.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.197+0000 7f76c3fff700 1 -- 172.21.15.60:0/948643541 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76bc0118e0 con 0x7f76cc05bd20 2022-06-29T16:08:21.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.197+0000 7f76c3fff700 1 -- 172.21.15.60:0/948643541 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76bc011c40 con 0x7f76cc05bd20 2022-06-29T16:08:21.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.197+0000 7f76d0b0f700 1 -- 172.21.15.60:0/948643541 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76cc0af910 con 0x7f76cc05bd20 2022-06-29T16:08:21.211 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.197+0000 7f76d0b0f700 1 -- 172.21.15.60:0/948643541 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76cc116ea0 con 0x7f76cc05bd20 2022-06-29T16:08:21.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.201+0000 7f76d0b0f700 1 -- 172.21.15.60:0/948643541 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76cc0ae000 con 0x7f76cc05bd20 2022-06-29T16:08:21.212 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.201+0000 7f76c3fff700 1 -- 172.21.15.60:0/948643541 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f76bc012420 con 0x7f76cc05bd20 2022-06-29T16:08:21.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.201+0000 7f76c3fff700 1 --2- 172.21.15.60:0/948643541 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f76b0053a70 0x7f76b0055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:21.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.201+0000 7f76c3fff700 1 -- 172.21.15.60:0/948643541 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f76bc06bd90 con 0x7f76cc05bd20 2022-06-29T16:08:21.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.201+0000 7f76c1d9b700 1 --2- 172.21.15.60:0/948643541 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f76b0053a70 0x7f76b0055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:21.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.201+0000 7f76c1d9b700 1 --2- 172.21.15.60:0/948643541 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f76b0053a70 0x7f76b0055f30 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7f76b4017000 tx=0x7f76b4015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:21.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.205+0000 7f76c3fff700 1 -- 172.21.15.60:0/948643541 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f76bc042c50 con 0x7f76cc05bd20 2022-06-29T16:08:21.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.329+0000 7f76d0b0f700 1 -- 172.21.15.60:0/948643541 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f76cc0573d0 con 0x7f76cc05bd20 2022-06-29T16:08:21.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.329+0000 7f76c3fff700 1 -- 172.21.15.60:0/948643541 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v147) v1 ==== 104+0+10 (secure 0 0 0) 0x7f76bc046270 con 0x7f76cc05bd20 2022-06-29T16:08:21.336 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:08:21.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.333+0000 7f76d0b0f700 1 -- 172.21.15.60:0/948643541 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f76b0053a70 msgr2=0x7f76b0055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:21.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.333+0000 7f76d0b0f700 1 --2- 172.21.15.60:0/948643541 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f76b0053a70 0x7f76b0055f30 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7f76b4017000 tx=0x7f76b4015000).stop 2022-06-29T16:08:21.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.333+0000 7f76d0b0f700 1 -- 172.21.15.60:0/948643541 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76cc05bd20 msgr2=0x7f76cc0ad530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:21.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.333+0000 7f76d0b0f700 1 --2- 172.21.15.60:0/948643541 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76cc05bd20 0x7f76cc0ad530 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 rx=0x7f76bc006940 tx=0x7f76bc008040).stop 2022-06-29T16:08:21.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.333+0000 7f76d0b0f700 1 -- 172.21.15.60:0/948643541 shutdown_connections 2022-06-29T16:08:21.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.333+0000 7f76d0b0f700 1 --2- 172.21.15.60:0/948643541 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f76b0053a70 0x7f76b0055f30 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:21.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.333+0000 7f76d0b0f700 1 --2- 172.21.15.60:0/948643541 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f76cc0b5950 0x7f76cc1592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:21.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.333+0000 7f76d0b0f700 1 --2- 172.21.15.60:0/948643541 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f76cc063ad0 0x7f76cc14df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:21.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.333+0000 7f76d0b0f700 1 --2- 172.21.15.60:0/948643541 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f76cc05bd20 0x7f76cc0ad530 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:21.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.333+0000 7f76d0b0f700 1 -- 172.21.15.60:0/948643541 >> 172.21.15.60:0/948643541 conn(0x7f76cc0632c0 msgr2=0x7f76cc145f50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:21.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.333+0000 7f76d0b0f700 1 -- 172.21.15.60:0/948643541 shutdown_connections 2022-06-29T16:08:21.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.333+0000 7f76d0b0f700 1 -- 172.21.15.60:0/948643541 wait complete. 2022-06-29T16:08:21.349 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-29T16:08:21.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.453+0000 7f98af59e700 1 Processor -- start 2022-06-29T16:08:21.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.453+0000 7f98af59e700 1 -- start start 2022-06-29T16:08:21.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.453+0000 7f98af59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98b00b5eb0 0x7f98b0142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:21.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.453+0000 7f98af59e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f98b00b48b0 0x7f98b00b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:21.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.453+0000 7f98af59e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98b00b4d10 con 0x7f98b00b2c50 2022-06-29T16:08:21.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.453+0000 7f98af59e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98b00b54e0 con 0x7f98b00b48b0 2022-06-29T16:08:21.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.453+0000 7f98af59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b00b6290 con 0x7f98b00b5eb0 2022-06-29T16:08:21.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.453+0000 7f98add9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f98b00b48b0 0x7f98b00b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:21.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.453+0000 7f98add9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f98b00b48b0 0x7f98b00b05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54540/0 (socket says 172.21.15.60:54540) 2022-06-29T16:08:21.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.457+0000 7f98add9b700 1 -- 172.21.15.60:0/313753207 learned_addr learned my addr 172.21.15.60:0/313753207 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:21.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.457+0000 7f98add9b700 1 -- 172.21.15.60:0/313753207 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98b00b5eb0 msgr2=0x7f98b0142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:21.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.457+0000 7f98ae59c700 1 --2- 172.21.15.60:0/313753207 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98b00b5eb0 0x7f98b0142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:21.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.457+0000 7f98add9b700 1 --2- 172.21.15.60:0/313753207 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98b00b5eb0 0x7f98b0142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:21.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.457+0000 7f98add9b700 1 -- 172.21.15.60:0/313753207 >> v1:172.21.15.92:6789/0 conn(0x7f98b00b2c50 legacy=0x7f98b00b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:21.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.457+0000 7f98add9b700 1 -- 172.21.15.60:0/313753207 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b0142f10 con 0x7f98b00b48b0 2022-06-29T16:08:21.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.457+0000 7f98ae59c700 1 --2- 172.21.15.60:0/313753207 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98b00b5eb0 0x7f98b0142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T16:08:21.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.457+0000 7f98add9b700 1 --2- 172.21.15.60:0/313753207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98b00b48b0 0x7f98b00b05e0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 rx=0x7f98a0002f70 tx=0x7f98a0006eb0).ready entity=mon.0 client_cookie=48ca8fafb215a87e server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:21.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.457+0000 7f98ad59a700 1 -- 172.21.15.60:0/313753207 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98a000d5c0 con 0x7f98b00b48b0 2022-06-29T16:08:21.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.457+0000 7f98ad59a700 1 -- 172.21.15.60:0/313753207 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98a0000e90 con 0x7f98b00b48b0 2022-06-29T16:08:21.466 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.457+0000 7f98ad59a700 1 -- 172.21.15.60:0/313753207 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98a000d950 con 0x7f98b00b48b0 2022-06-29T16:08:21.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.457+0000 7f98af59e700 1 -- 172.21.15.60:0/313753207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98b00b48b0 msgr2=0x7f98b00b05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:21.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.457+0000 7f98af59e700 1 --2- 172.21.15.60:0/313753207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98b00b48b0 0x7f98b00b05e0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 rx=0x7f98a0002f70 tx=0x7f98a0006eb0).stop 2022-06-29T16:08:21.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.457+0000 7f98af59e700 1 -- 172.21.15.60:0/313753207 shutdown_connections 2022-06-29T16:08:21.467 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.457+0000 7f98af59e700 1 --2- 172.21.15.60:0/313753207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98b00b48b0 0x7f98b00b05e0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:21.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.457+0000 7f98af59e700 1 --2- 172.21.15.60:0/313753207 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98b00b5eb0 0x7f98b0142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:21.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.457+0000 7f98af59e700 1 -- 172.21.15.60:0/313753207 >> 172.21.15.60:0/313753207 conn(0x7f98b0062090 msgr2=0x7f98b0062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:21.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.457+0000 7f98af59e700 1 -- 172.21.15.60:0/313753207 shutdown_connections 2022-06-29T16:08:21.468 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.457+0000 7f98af59e700 1 -- 172.21.15.60:0/313753207 wait complete. 2022-06-29T16:08:21.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.457+0000 7f98af59e700 1 Processor -- start 2022-06-29T16:08:21.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.457+0000 7f98af59e700 1 -- start start 2022-06-29T16:08:21.472 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.457+0000 7f98af59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98b00b2c50 0x7f98b013e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:21.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.457+0000 7f98af59e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98b00b48b0 0x7f98b00b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:21.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.457+0000 7f98af59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98b00b5eb0 0x7f98b01550b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:21.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.457+0000 7f98af59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b00b1550 con 0x7f98b00b5eb0 2022-06-29T16:08:21.473 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.457+0000 7f98af59e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b013dcf0 con 0x7f98b00b2c50 2022-06-29T16:08:21.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.457+0000 7f98af59e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b013da50 con 0x7f98b00b48b0 2022-06-29T16:08:21.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.457+0000 7f98aed9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98b00b5eb0 0x7f98b01550b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:21.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.461+0000 7f98ae59c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98b00b2c50 0x7f98b013e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:21.474 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.461+0000 7f98add9b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98b00b48b0 0x7f98b00b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:21.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.461+0000 7f98aed9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98b00b5eb0 0x7f98b01550b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54546/0 (socket says 172.21.15.60:54546) 2022-06-29T16:08:21.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.461+0000 7f98aed9d700 1 -- 172.21.15.60:0/1714001177 learned_addr learned my addr 172.21.15.60:0/1714001177 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:21.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.461+0000 7f98aed9d700 1 -- 172.21.15.60:0/1714001177 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98b00b48b0 msgr2=0x7f98b00b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:21.475 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.461+0000 7f98aed9d700 1 --2- 172.21.15.60:0/1714001177 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98b00b48b0 0x7f98b00b19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:21.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.461+0000 7f98aed9d700 1 -- 172.21.15.60:0/1714001177 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98b00b2c50 msgr2=0x7f98b013e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:21.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.461+0000 7f98aed9d700 1 --2- 172.21.15.60:0/1714001177 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98b00b2c50 0x7f98b013e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:21.476 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.461+0000 7f98aed9d700 1 -- 172.21.15.60:0/1714001177 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a000b1e0 con 0x7f98b00b5eb0 2022-06-29T16:08:21.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.461+0000 7f98aed9d700 1 --2- 172.21.15.60:0/1714001177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98b00b5eb0 0x7f98b01550b0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 rx=0x7f98b0062f00 tx=0x7f98a8007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:21.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.461+0000 7f989f7fe700 1 -- 172.21.15.60:0/1714001177 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98a800f470 con 0x7f98b00b5eb0 2022-06-29T16:08:21.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.461+0000 7f989f7fe700 1 -- 172.21.15.60:0/1714001177 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98a800f5d0 con 0x7f98b00b5eb0 2022-06-29T16:08:21.477 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.461+0000 7f989f7fe700 1 -- 172.21.15.60:0/1714001177 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98a800f930 con 0x7f98b00b5eb0 2022-06-29T16:08:21.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.461+0000 7f98af59e700 1 -- 172.21.15.60:0/1714001177 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98b013d640 con 0x7f98b00b5eb0 2022-06-29T16:08:21.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.461+0000 7f98af59e700 1 -- 172.21.15.60:0/1714001177 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b01410a0 con 0x7f98b00b5eb0 2022-06-29T16:08:21.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.461+0000 7f98af59e700 1 -- 172.21.15.60:0/1714001177 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98b00b4f70 con 0x7f98b00b5eb0 2022-06-29T16:08:21.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.461+0000 7f989f7fe700 1 -- 172.21.15.60:0/1714001177 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f98a8019030 con 0x7f98b00b5eb0 2022-06-29T16:08:21.478 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.461+0000 7f989f7fe700 1 --2- 172.21.15.60:0/1714001177 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f988c053950 0x7f988c055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:21.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.465+0000 7f989f7fe700 1 -- 172.21.15.60:0/1714001177 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f98a806b530 con 0x7f98b00b5eb0 2022-06-29T16:08:21.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.465+0000 7f98ae59c700 1 --2- 172.21.15.60:0/1714001177 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f988c053950 0x7f988c055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:21.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.465+0000 7f98ae59c700 1 --2- 172.21.15.60:0/1714001177 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f988c053950 0x7f988c055e10 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7f98a400d000 tx=0x7f98a4008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:21.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.469+0000 7f989f7fe700 1 -- 172.21.15.60:0/1714001177 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f98a80423f0 con 0x7f98b00b5eb0 2022-06-29T16:08:21.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.597+0000 7f98af59e700 1 -- 172.21.15.60:0/1714001177 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f98b00573d0 con 0x7f98b00b5eb0 2022-06-29T16:08:21.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.597+0000 7f989f7fe700 1 -- 172.21.15.60:0/1714001177 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v147) v1 ==== 100+0+10 (secure 0 0 0) 0x7f98a8045a10 con 0x7f98b00b5eb0 2022-06-29T16:08:21.602 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:08:21.603 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.597+0000 7f98af59e700 1 -- 172.21.15.60:0/1714001177 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f988c053950 msgr2=0x7f988c055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:21.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.597+0000 7f98af59e700 1 --2- 172.21.15.60:0/1714001177 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f988c053950 0x7f988c055e10 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7f98a400d000 tx=0x7f98a4008040).stop 2022-06-29T16:08:21.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.597+0000 7f98af59e700 1 -- 172.21.15.60:0/1714001177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98b00b5eb0 msgr2=0x7f98b01550b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:21.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.597+0000 7f98af59e700 1 --2- 172.21.15.60:0/1714001177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98b00b5eb0 0x7f98b01550b0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 rx=0x7f98b0062f00 tx=0x7f98a8007f40).stop 2022-06-29T16:08:21.604 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.597+0000 7f98af59e700 1 -- 172.21.15.60:0/1714001177 shutdown_connections 2022-06-29T16:08:21.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.597+0000 7f98af59e700 1 --2- 172.21.15.60:0/1714001177 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f988c053950 0x7f988c055e10 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:21.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.597+0000 7f98af59e700 1 --2- 172.21.15.60:0/1714001177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98b00b5eb0 0x7f98b01550b0 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:21.605 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.597+0000 7f98af59e700 1 --2- 172.21.15.60:0/1714001177 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f98b00b48b0 0x7f98b00b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:21.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.597+0000 7f98af59e700 1 --2- 172.21.15.60:0/1714001177 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f98b00b2c50 0x7f98b013e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:21.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.597+0000 7f98af59e700 1 -- 172.21.15.60:0/1714001177 >> 172.21.15.60:0/1714001177 conn(0x7f98b0062090 msgr2=0x7f98b01431a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:21.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.597+0000 7f98af59e700 1 -- 172.21.15.60:0/1714001177 shutdown_connections 2022-06-29T16:08:21.606 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.597+0000 7f98af59e700 1 -- 172.21.15.60:0/1714001177 wait complete. 2022-06-29T16:08:21.615 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-06-29T16:08:21.616 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:08:21.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.709+0000 7f70ced90700 1 Processor -- start 2022-06-29T16:08:21.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.709+0000 7f70ced90700 1 -- start start 2022-06-29T16:08:21.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.713+0000 7f70ced90700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70c80b3360 0x7f70c8142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:21.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.713+0000 7f70ced90700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c80b0cf0 con 0x7f70c80b5eb0 2022-06-29T16:08:21.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.713+0000 7f70ced90700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c80b4d10 con 0x7f70c80b48b0 2022-06-29T16:08:21.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.713+0000 7f70ced90700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70c80b54e0 con 0x7f70c80b3360 2022-06-29T16:08:21.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.713+0000 7f70cdd8e700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f70c80b5eb0 0x7f70c80b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34404/0 (socket says 172.21.15.60:34404) 2022-06-29T16:08:21.731 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.713+0000 7f70cdd8e700 1 -- 172.21.15.60:0/726746741 learned_addr learned my addr 172.21.15.60:0/726746741 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:21.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.713+0000 7f70ce58f700 1 --2- 172.21.15.60:0/726746741 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70c80b3360 0x7f70c8142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:21.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.713+0000 7f70ccd8c700 1 -- 172.21.15.60:0/726746741 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1402112462 0 0) 0x7f70c0001490 con 0x7f70c80b5eb0 2022-06-29T16:08:21.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.713+0000 7f70ccd8c700 1 -- 172.21.15.60:0/726746741 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70b0003260 con 0x7f70c80b5eb0 2022-06-29T16:08:21.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.713+0000 7f70ccd8c700 1 -- 172.21.15.60:0/726746741 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3820099870 0 0) 0x7f70c80b4d10 con 0x7f70c80b48b0 2022-06-29T16:08:21.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.713+0000 7f70ce58f700 1 -- 172.21.15.60:0/726746741 >> v1:172.21.15.92:6789/0 conn(0x7f70c80b48b0 legacy=0x7f70c80b2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:21.734 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.713+0000 7f70ce58f700 1 -- 172.21.15.60:0/726746741 >> v1:172.21.15.60:6789/0 conn(0x7f70c80b5eb0 legacy=0x7f70c80b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:21.734 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.713+0000 7f70ce58f700 1 -- 172.21.15.60:0/726746741 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c80b6290 con 0x7f70c80b3360 2022-06-29T16:08:21.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.713+0000 7f70ce58f700 1 --2- 172.21.15.60:0/726746741 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70c80b3360 0x7f70c8142a20 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f70c4002f70 tx=0x7f70c4006df0).ready entity=mon.2 client_cookie=747eb2a1820a06ec server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:21.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.713+0000 7f70ccd8c700 1 -- 172.21.15.60:0/726746741 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70c400d570 con 0x7f70c80b3360 2022-06-29T16:08:21.735 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.713+0000 7f70ccd8c700 1 -- 172.21.15.60:0/726746741 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70c4000e90 con 0x7f70c80b3360 2022-06-29T16:08:21.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.713+0000 7f70ccd8c700 1 -- 172.21.15.60:0/726746741 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70c400d8c0 con 0x7f70c80b3360 2022-06-29T16:08:21.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.713+0000 7f70ced90700 1 -- 172.21.15.60:0/726746741 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70c80b3360 msgr2=0x7f70c8142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:21.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.713+0000 7f70ced90700 1 --2- 172.21.15.60:0/726746741 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70c80b3360 0x7f70c8142a20 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f70c4002f70 tx=0x7f70c4006df0).stop 2022-06-29T16:08:21.736 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.713+0000 7f70ced90700 1 -- 172.21.15.60:0/726746741 shutdown_connections 2022-06-29T16:08:21.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.713+0000 7f70ced90700 1 --2- 172.21.15.60:0/726746741 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70c80b3360 0x7f70c8142a20 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:21.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.713+0000 7f70ced90700 1 -- 172.21.15.60:0/726746741 >> 172.21.15.60:0/726746741 conn(0x7f70c8062090 msgr2=0x7f70c8062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:21.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.713+0000 7f70ced90700 1 -- 172.21.15.60:0/726746741 shutdown_connections 2022-06-29T16:08:21.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.713+0000 7f70ced90700 1 -- 172.21.15.60:0/726746741 wait complete. 2022-06-29T16:08:21.737 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.713+0000 7f70ced90700 1 Processor -- start 2022-06-29T16:08:21.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.713+0000 7f70ced90700 1 -- start start 2022-06-29T16:08:21.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.717+0000 7f70ced90700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70c80b3360 0x7f70c8145f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:21.738 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.717+0000 7f70ced90700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70c80b48b0 0x7f70c81458e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:21.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.717+0000 7f70ced90700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70c80b5eb0 0x7f70c80b17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:21.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.717+0000 7f70ced90700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c81464a0 con 0x7f70c80b48b0 2022-06-29T16:08:21.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.717+0000 7f70ced90700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c8146740 con 0x7f70c80b3360 2022-06-29T16:08:21.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.717+0000 7f70ced90700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70c81469e0 con 0x7f70c80b5eb0 2022-06-29T16:08:21.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.717+0000 7f70cd58d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70c80b48b0 0x7f70c81458e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:21.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.717+0000 7f70cdd8e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70c80b3360 0x7f70c8145f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:21.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.717+0000 7f70cd58d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70c80b48b0 0x7f70c81458e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54558/0 (socket says 172.21.15.60:54558) 2022-06-29T16:08:21.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.717+0000 7f70ce58f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70c80b5eb0 0x7f70c80b17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:21.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.717+0000 7f70cd58d700 1 -- 172.21.15.60:0/1308824183 learned_addr learned my addr 172.21.15.60:0/1308824183 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:21.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.717+0000 7f70cd58d700 1 -- 172.21.15.60:0/1308824183 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70c80b5eb0 msgr2=0x7f70c80b17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:21.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.717+0000 7f70cd58d700 1 --2- 172.21.15.60:0/1308824183 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70c80b5eb0 0x7f70c80b17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:21.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.717+0000 7f70cd58d700 1 -- 172.21.15.60:0/1308824183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70c80b3360 msgr2=0x7f70c8145f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:21.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.717+0000 7f70cd58d700 1 --2- 172.21.15.60:0/1308824183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70c80b3360 0x7f70c8145f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:21.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.717+0000 7f70cd58d700 1 -- 172.21.15.60:0/1308824183 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c400b180 con 0x7f70c80b48b0 2022-06-29T16:08:21.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.717+0000 7f70cd58d700 1 --2- 172.21.15.60:0/1308824183 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70c80b48b0 0x7f70c81458e0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 rx=0x7f70b8000ed0 tx=0x7f70b8007a90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:21.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.717+0000 7f70b6ffd700 1 -- 172.21.15.60:0/1308824183 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70b8012400 con 0x7f70c80b48b0 2022-06-29T16:08:21.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.717+0000 7f70b6ffd700 1 -- 172.21.15.60:0/1308824183 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70b8012560 con 0x7f70c80b48b0 2022-06-29T16:08:21.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.717+0000 7f70ced90700 1 -- 172.21.15.60:0/1308824183 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70c80b1ca0 con 0x7f70c80b48b0 2022-06-29T16:08:21.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.717+0000 7f70b6ffd700 1 -- 172.21.15.60:0/1308824183 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70b80128c0 con 0x7f70c80b48b0 2022-06-29T16:08:21.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.717+0000 7f70ced90700 1 -- 172.21.15.60:0/1308824183 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70c813f510 con 0x7f70c80b48b0 2022-06-29T16:08:21.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.721+0000 7f70ced90700 1 -- 172.21.15.60:0/1308824183 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70c8142f10 con 0x7f70c80b48b0 2022-06-29T16:08:21.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.721+0000 7f70b6ffd700 1 -- 172.21.15.60:0/1308824183 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f70b8019030 con 0x7f70c80b48b0 2022-06-29T16:08:21.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.721+0000 7f70b6ffd700 1 --2- 172.21.15.60:0/1308824183 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f70a80539e0 0x7f70a8055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:21.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.721+0000 7f70b6ffd700 1 -- 172.21.15.60:0/1308824183 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f70b806b150 con 0x7f70c80b48b0 2022-06-29T16:08:21.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.721+0000 7f70cdd8e700 1 --2- 172.21.15.60:0/1308824183 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f70a80539e0 0x7f70a8055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:21.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.721+0000 7f70cdd8e700 1 --2- 172.21.15.60:0/1308824183 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f70a80539e0 0x7f70a8055ea0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0x7f70c000d000 tx=0x7f70c0000f40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:21.746 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.729+0000 7f70b6ffd700 1 -- 172.21.15.60:0/1308824183 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f70b8042120 con 0x7f70c80b48b0 2022-06-29T16:08:21.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.893+0000 7f70ced90700 1 -- 172.21.15.60:0/1308824183 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70c80573d0 con 0x7f70c80b48b0 2022-06-29T16:08:21.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.893+0000 7f70b6ffd700 1 -- 172.21.15.60:0/1308824183 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+2139 (secure 0 0 0) 0x7f70b8045740 con 0x7f70c80b48b0 2022-06-29T16:08:21.898 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:08:21.898 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":73,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7457,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.92:6835/698111951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":698111951},{"type":"v1","addr":"172.21.15.92:6835","nonce":698111951}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":72}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:08:17.075855+0000","modified":"2022-06-29T16:08:18.079541+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8790},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8790":{"gid":8790,"name":"a","rank":0,"incarnation":72,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/3619856083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3619856083},{"type":"v1","addr":"172.21.15.60:6833","nonce":3619856083}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2022-06-29T16:08:21.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.893+0000 7f70ced90700 1 -- 172.21.15.60:0/1308824183 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f70a80539e0 msgr2=0x7f70a8055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:21.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.893+0000 7f70ced90700 1 --2- 172.21.15.60:0/1308824183 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f70a80539e0 0x7f70a8055ea0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0x7f70c000d000 tx=0x7f70c0000f40).stop 2022-06-29T16:08:21.900 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.893+0000 7f70ced90700 1 -- 172.21.15.60:0/1308824183 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70c80b48b0 msgr2=0x7f70c81458e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:21.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.893+0000 7f70ced90700 1 --2- 172.21.15.60:0/1308824183 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70c80b48b0 0x7f70c81458e0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 rx=0x7f70b8000ed0 tx=0x7f70b8007a90).stop 2022-06-29T16:08:21.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.897+0000 7f70ced90700 1 -- 172.21.15.60:0/1308824183 shutdown_connections 2022-06-29T16:08:21.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.897+0000 7f70ced90700 1 --2- 172.21.15.60:0/1308824183 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f70a80539e0 0x7f70a8055ea0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:21.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.897+0000 7f70ced90700 1 --2- 172.21.15.60:0/1308824183 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70c80b5eb0 0x7f70c80b17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:21.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.897+0000 7f70ced90700 1 --2- 172.21.15.60:0/1308824183 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70c80b48b0 0x7f70c81458e0 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:21.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.897+0000 7f70ced90700 1 --2- 172.21.15.60:0/1308824183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70c80b3360 0x7f70c8145f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:21.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.897+0000 7f70ced90700 1 -- 172.21.15.60:0/1308824183 >> 172.21.15.60:0/1308824183 conn(0x7f70c8062090 msgr2=0x7f70c805bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:21.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.897+0000 7f70ced90700 1 -- 172.21.15.60:0/1308824183 shutdown_connections 2022-06-29T16:08:21.903 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:21.897+0000 7f70ced90700 1 -- 172.21.15.60:0/1308824183 wait complete. 2022-06-29T16:08:21.903 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 73 2022-06-29T16:08:22.113 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:08:17.075855+0000', 'modified': '2022-06-29T16:08:18.079541+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8790}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8790': {'gid': 8790, 'name': 'a', 'rank': 0, 'incarnation': 72, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.60:6833/3619856083', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6832', 'nonce': 3619856083}, {'type': 'v1', 'addr': '172.21.15.60:6833', 'nonce': 3619856083}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:08:22.114 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:08:22.115 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-06-29T16:08:22.231 INFO:teuthology.orchestra.run.smithi060.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":8790,"want_state":"up:active","state":"up:active","rank_uptime":5.1399242599999999,"mdsmap_epoch":73,"osdmap_epoch":147,"osdmap_epoch_barrier":145,"uptime":9.4897936729999994} 2022-06-29T16:08:22.232 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 8790, "mdsmap_epoch": 73, "osdmap_epoch": 147, "osdmap_epoch_barrier": 145, "rank_uptime": 5.13992426, "state": "up:active", "uptime": 9.489793673, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:08:22.232 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-06-29T16:08:22.232 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-06-29T16:08:22.233 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi121.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-06-29T16:08:22.233 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:22.234 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:08:22.242 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:08:22.300 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:08:22.301 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:08:22.311 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:08:22.311 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:08:22.313 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:22.313 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:08:22.319 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-06-29T16:08:22.319 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:08:22.322 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:08:22.338 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:08:22.339 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:08:22.393 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:08:22.394 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:08:22.395 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:22.396 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:08:22.396 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[13645]: starting ceph client 2022-06-29T16:08:22.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:2022-06-29T16:08:22.387+0000 7fd91800b080 -1 init, newargv = 0x563f3ee6f080 newargc=9 2022-06-29T16:08:22.403 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[13645]: starting fuse 2022-06-29T16:08:23.503 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-06-29T16:08:23.521 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:08:23.522 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:08:23.578 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:08:23.579 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:08:23.580 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:23.581 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-06-29T16:08:23.586 INFO:teuthology.orchestra.run.smithi121.stdout:52 2022-06-29T16:08:23.586 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-06-29T16:08:23.586 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:23.587 DEBUG:teuthology.orchestra.run.smithi121:> sudo python3 -c ' 2022-06-29T16:08:23.587 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2022-06-29T16:08:23.588 DEBUG:teuthology.orchestra.run.smithi121:> import re 2022-06-29T16:08:23.588 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-06-29T16:08:23.589 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2022-06-29T16:08:23.589 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:08:23.589 DEBUG:teuthology.orchestra.run.smithi121:> def find_socket(client_name): 2022-06-29T16:08:23.590 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-06-29T16:08:23.590 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2022-06-29T16:08:23.591 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:08:23.591 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2022-06-29T16:08:23.591 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2022-06-29T16:08:23.592 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2022-06-29T16:08:23.592 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2022-06-29T16:08:23.593 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:08:23.593 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2022-06-29T16:08:23.594 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-06-29T16:08:23.594 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2022-06-29T16:08:23.594 DEBUG:teuthology.orchestra.run.smithi121:> return f 2022-06-29T16:08:23.595 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-06-29T16:08:23.595 DEBUG:teuthology.orchestra.run.smithi121:> 2022-06-29T16:08:23.596 DEBUG:teuthology.orchestra.run.smithi121:> print(find_socket("client.0")) 2022-06-29T16:08:23.596 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-06-29T16:08:23.636 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.0.13645.asok 2022-06-29T16:08:23.641 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.13645.asok 2022-06-29T16:08:23.641 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:23.642 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.13645.asok status 2022-06-29T16:08:23.774 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2022-06-29T16:08:23.775 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2022-06-29T16:08:23.775 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "f25ffac72e5a9e63860ff7deb6767a11f0559a16", 2022-06-29T16:08:23.775 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 15.2.16-304-gf25ffac7 (f25ffac72e5a9e63860ff7deb6767a11f0559a16) octopus (stable)", 2022-06-29T16:08:23.776 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "0", 2022-06-29T16:08:23.776 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2022-06-29T16:08:23.776 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-06-29T16:08:23.776 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "13645", 2022-06-29T16:08:23.777 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2022-06-29T16:08:23.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:08:23.777 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2022-06-29T16:08:23.777 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2022-06-29T16:08:23.777 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 8958, 2022-06-29T16:08:23.778 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2022-06-29T16:08:23.778 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2022-06-29T16:08:23.778 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2022-06-29T16:08:23.778 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 8958 2022-06-29T16:08:23.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:08:23.779 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-06-29T16:08:23.779 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-06-29T16:08:23.779 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-06-29T16:08:23.780 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 147368695 2022-06-29T16:08:23.780 INFO:teuthology.orchestra.run.smithi121.stdout: } 2022-06-29T16:08:23.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:08:23.780 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-06-29T16:08:23.780 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-06-29T16:08:23.781 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-06-29T16:08:23.781 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 147368695 2022-06-29T16:08:23.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-06-29T16:08:23.781 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.8958 172.21.15.121:0/147368695", 2022-06-29T16:08:23.782 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "172.21.15.121:0/147368695", 2022-06-29T16:08:23.782 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2022-06-29T16:08:23.782 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 73, 2022-06-29T16:08:23.782 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 147, 2022-06-29T16:08:23.783 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2022-06-29T16:08:23.783 INFO:teuthology.orchestra.run.smithi121.stdout: "blacklisted": false, 2022-06-29T16:08:23.783 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2022-06-29T16:08:23.783 INFO:teuthology.orchestra.run.smithi121.stdout:} 2022-06-29T16:08:23.785 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:23.785 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:08:23.786 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:08:23.793 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-06-29T16:08:23.794 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-06-29T16:08:23.795 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:23.795 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:08:23.811 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T16:08:23.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.929+0000 7f6b83fff700 1 Processor -- start 2022-06-29T16:08:23.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.929+0000 7f6b83fff700 1 -- start start 2022-06-29T16:08:23.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.929+0000 7f6b83fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6b840b5eb0 0x7f6b84142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:23.939 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.929+0000 7f6b83fff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6b840b48b0 0x7f6b840b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:23.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.929+0000 7f6b83fff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b840b4d10 con 0x7f6b840b2c50 2022-06-29T16:08:23.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.929+0000 7f6b83fff700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b840b54e0 con 0x7f6b840b48b0 2022-06-29T16:08:23.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.929+0000 7f6b83fff700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b840b6290 con 0x7f6b840b5eb0 2022-06-29T16:08:23.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.929+0000 7f6b827fc700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f6b840b48b0 0x7f6b840b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:23.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.929+0000 7f6b837fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6b840b2c50 0x7f6b84145fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34422/0 (socket says 172.21.15.60:34422) 2022-06-29T16:08:23.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.929+0000 7f6b837fe700 1 -- 172.21.15.60:0/1737850268 learned_addr learned my addr 172.21.15.60:0/1737850268 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:23.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.929+0000 7f6b82ffd700 1 --2- 172.21.15.60:0/1737850268 >> v2:172.21.15.92:3300/0 conn(0x7f6b840b5eb0 0x7f6b84142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:23.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.929+0000 7f6b827fc700 1 -- 172.21.15.60:0/1737850268 >> v2:172.21.15.92:3300/0 conn(0x7f6b840b5eb0 msgr2=0x7f6b84142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:23.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.929+0000 7f6b827fc700 1 --2- 172.21.15.60:0/1737850268 >> v2:172.21.15.92:3300/0 conn(0x7f6b840b5eb0 0x7f6b84142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:23.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.929+0000 7f6b827fc700 1 -- 172.21.15.60:0/1737850268 >> v1:172.21.15.60:6789/0 conn(0x7f6b840b2c50 legacy=0x7f6b84145fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:23.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.929+0000 7f6b827fc700 1 -- 172.21.15.60:0/1737850268 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b84142f10 con 0x7f6b840b48b0 2022-06-29T16:08:23.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.933+0000 7f6b827fc700 1 --2- 172.21.15.60:0/1737850268 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b840b48b0 0x7f6b840b05e0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 rx=0x7f6b6c002f70 tx=0x7f6b6c006df0).ready entity=mon.0 client_cookie=3e4e7f32adc8a0f4 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:23.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.933+0000 7f6b81ffb700 1 -- 172.21.15.60:0/1737850268 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b6c00d570 con 0x7f6b840b48b0 2022-06-29T16:08:23.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.933+0000 7f6b81ffb700 1 -- 172.21.15.60:0/1737850268 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b6c000e90 con 0x7f6b840b48b0 2022-06-29T16:08:23.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.933+0000 7f6b81ffb700 1 -- 172.21.15.60:0/1737850268 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b6c00d8c0 con 0x7f6b840b48b0 2022-06-29T16:08:23.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.933+0000 7f6b83fff700 1 -- 172.21.15.60:0/1737850268 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b840b48b0 msgr2=0x7f6b840b05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:23.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.933+0000 7f6b83fff700 1 --2- 172.21.15.60:0/1737850268 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b840b48b0 0x7f6b840b05e0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 rx=0x7f6b6c002f70 tx=0x7f6b6c006df0).stop 2022-06-29T16:08:23.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.933+0000 7f6b83fff700 1 -- 172.21.15.60:0/1737850268 shutdown_connections 2022-06-29T16:08:23.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.933+0000 7f6b83fff700 1 --2- 172.21.15.60:0/1737850268 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b840b48b0 0x7f6b840b05e0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:23.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.933+0000 7f6b83fff700 1 --2- 172.21.15.60:0/1737850268 >> v2:172.21.15.92:3300/0 conn(0x7f6b840b5eb0 0x7f6b84142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:23.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.933+0000 7f6b83fff700 1 -- 172.21.15.60:0/1737850268 >> 172.21.15.60:0/1737850268 conn(0x7f6b84062090 msgr2=0x7f6b84062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:23.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.933+0000 7f6b83fff700 1 -- 172.21.15.60:0/1737850268 shutdown_connections 2022-06-29T16:08:23.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.933+0000 7f6b83fff700 1 -- 172.21.15.60:0/1737850268 wait complete. 2022-06-29T16:08:23.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.937+0000 7f6b83fff700 1 Processor -- start 2022-06-29T16:08:23.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.937+0000 7f6b83fff700 1 -- start start 2022-06-29T16:08:23.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.937+0000 7f6b83fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6b840b2c50 0x7f6b84144bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:23.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.937+0000 7f6b83fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6b840b48b0 0x7f6b84149d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:23.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.937+0000 7f6b83fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b840b5eb0 0x7f6b840ae040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:23.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.937+0000 7f6b83fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b84142a70 con 0x7f6b840b5eb0 2022-06-29T16:08:23.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.937+0000 7f6b83fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b84112fe0 con 0x7f6b840b2c50 2022-06-29T16:08:23.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.937+0000 7f6b83fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b84113280 con 0x7f6b840b48b0 2022-06-29T16:08:23.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.937+0000 7f6b837fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b840b5eb0 0x7f6b840ae040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:23.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.937+0000 7f6b837fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b840b5eb0 0x7f6b840ae040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54572/0 (socket says 172.21.15.60:54572) 2022-06-29T16:08:23.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.937+0000 7f6b837fe700 1 -- 172.21.15.60:0/3032351883 learned_addr learned my addr 172.21.15.60:0/3032351883 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:23.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.937+0000 7f6b82ffd700 1 --2- 172.21.15.60:0/3032351883 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6b840b2c50 0x7f6b84144bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:23.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.937+0000 7f6b827fc700 1 --2- 172.21.15.60:0/3032351883 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6b840b48b0 0x7f6b84149d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:23.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.937+0000 7f6b837fe700 1 -- 172.21.15.60:0/3032351883 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6b840b48b0 msgr2=0x7f6b84149d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:23.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.937+0000 7f6b837fe700 1 --2- 172.21.15.60:0/3032351883 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6b840b48b0 0x7f6b84149d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:23.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.937+0000 7f6b837fe700 1 -- 172.21.15.60:0/3032351883 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6b840b2c50 msgr2=0x7f6b84144bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:23.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.937+0000 7f6b837fe700 1 --2- 172.21.15.60:0/3032351883 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6b840b2c50 0x7f6b84144bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:23.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.937+0000 7f6b837fe700 1 -- 172.21.15.60:0/3032351883 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b6c00b180 con 0x7f6b840b5eb0 2022-06-29T16:08:23.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.937+0000 7f6b837fe700 1 --2- 172.21.15.60:0/3032351883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b840b5eb0 0x7f6b840ae040 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 rx=0x7f6b84062f00 tx=0x7f6b7c007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:23.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.937+0000 7f6b7bfff700 1 -- 172.21.15.60:0/3032351883 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b7c005ba0 con 0x7f6b840b5eb0 2022-06-29T16:08:23.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.937+0000 7f6b7bfff700 1 -- 172.21.15.60:0/3032351883 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b7c005d00 con 0x7f6b840b5eb0 2022-06-29T16:08:23.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.937+0000 7f6b83fff700 1 -- 172.21.15.60:0/3032351883 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b840adba0 con 0x7f6b840b5eb0 2022-06-29T16:08:23.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.937+0000 7f6b7bfff700 1 -- 172.21.15.60:0/3032351883 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b7c0072c0 con 0x7f6b840b5eb0 2022-06-29T16:08:23.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.937+0000 7f6b83fff700 1 -- 172.21.15.60:0/3032351883 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b84144080 con 0x7f6b840b5eb0 2022-06-29T16:08:23.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.941+0000 7f6b83fff700 1 -- 172.21.15.60:0/3032351883 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b840af210 con 0x7f6b840b5eb0 2022-06-29T16:08:23.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.941+0000 7f6b7bfff700 1 -- 172.21.15.60:0/3032351883 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6b7c01a060 con 0x7f6b840b5eb0 2022-06-29T16:08:23.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.941+0000 7f6b7bfff700 1 --2- 172.21.15.60:0/3032351883 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6b68053b00 0x7f6b68055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:23.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.941+0000 7f6b7bfff700 1 -- 172.21.15.60:0/3032351883 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6b7c06b980 con 0x7f6b840b5eb0 2022-06-29T16:08:23.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.941+0000 7f6b82ffd700 1 --2- 172.21.15.60:0/3032351883 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6b68053b00 0x7f6b68055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:23.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.941+0000 7f6b82ffd700 1 --2- 172.21.15.60:0/3032351883 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6b68053b00 0x7f6b68055fc0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7f6b7400b000 tx=0x7f6b74006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:23.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:23.949+0000 7f6b7bfff700 1 -- 172.21.15.60:0/3032351883 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6b840af210 con 0x7f6b840b5eb0 2022-06-29T16:08:24.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.081+0000 7f6b83fff700 1 -- 172.21.15.60:0/3032351883 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b840af210 con 0x7f6b840b5eb0 2022-06-29T16:08:24.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.081+0000 7f6b7bfff700 1 -- 172.21.15.60:0/3032351883 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v147) v1 ==== 74+0+16052 (secure 0 0 0) 0x7f6b840af210 con 0x7f6b840b5eb0 2022-06-29T16:08:24.087 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:08:24.087 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":147,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:08:19.091282+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2022-06-29T16:08:15.820434+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":30,"pool_name":"cephfs_data","create_time":"2022-06-29T16:08:16.484716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"147","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T16:08:24.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.085+0000 7f6b83fff700 1 -- 172.21.15.60:0/3032351883 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6b68053b00 msgr2=0x7f6b68055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:24.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.085+0000 7f6b83fff700 1 --2- 172.21.15.60:0/3032351883 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6b68053b00 0x7f6b68055fc0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7f6b7400b000 tx=0x7f6b74006df0).stop 2022-06-29T16:08:24.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.085+0000 7f6b83fff700 1 -- 172.21.15.60:0/3032351883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b840b5eb0 msgr2=0x7f6b840ae040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:24.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.085+0000 7f6b83fff700 1 --2- 172.21.15.60:0/3032351883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b840b5eb0 0x7f6b840ae040 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 rx=0x7f6b84062f00 tx=0x7f6b7c007f40).stop 2022-06-29T16:08:24.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.085+0000 7f6b83fff700 1 -- 172.21.15.60:0/3032351883 shutdown_connections 2022-06-29T16:08:24.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.085+0000 7f6b83fff700 1 --2- 172.21.15.60:0/3032351883 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6b68053b00 0x7f6b68055fc0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:24.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.085+0000 7f6b83fff700 1 --2- 172.21.15.60:0/3032351883 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6b840b5eb0 0x7f6b840ae040 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:24.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.085+0000 7f6b83fff700 1 --2- 172.21.15.60:0/3032351883 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6b840b48b0 0x7f6b84149d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:24.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.085+0000 7f6b83fff700 1 --2- 172.21.15.60:0/3032351883 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6b840b2c50 0x7f6b84144bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:24.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.085+0000 7f6b83fff700 1 -- 172.21.15.60:0/3032351883 >> 172.21.15.60:0/3032351883 conn(0x7f6b84062090 msgr2=0x7f6b84141e30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:24.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.085+0000 7f6b83fff700 1 -- 172.21.15.60:0/3032351883 shutdown_connections 2022-06-29T16:08:24.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.085+0000 7f6b83fff700 1 -- 172.21.15.60:0/3032351883 wait complete. 2022-06-29T16:08:24.100 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-06-29T15:58:10.961127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-06-29T15:58:19.184276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 29, 'pool_name': 'cephfs_metadata', 'create_time': '2022-06-29T16:08:15.820434+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '146', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 30, 'pool_name': 'cephfs_data', 'create_time': '2022-06-29T16:08:16.484716+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '147', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-06-29T16:08:24.101 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-06-29T16:08:24.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.209+0000 7f6bd6dd8700 1 Processor -- start 2022-06-29T16:08:24.216 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.209+0000 7f6bd6dd8700 1 -- start start 2022-06-29T16:08:24.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bd6dd8700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6bd00b9d70 0x7f6bd01465d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:24.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bd6dd8700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6bd00b3d30 0x7f6bd0146ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:24.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bd6dd8700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd00b4f50 con 0x7f6bd0061d90 2022-06-29T16:08:24.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bd6dd8700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bd00ba150 con 0x7f6bd00b9d70 2022-06-29T16:08:24.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bd6dd8700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd00b5180 con 0x7f6bd00b3d30 2022-06-29T16:08:24.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bd5375700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f6bd0061d90 0x7f6bd005bd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34434/0 (socket says 172.21.15.60:34434) 2022-06-29T16:08:24.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bd5375700 1 -- 172.21.15.60:0/1998359506 learned_addr learned my addr 172.21.15.60:0/1998359506 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:24.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bd4b74700 1 --2- 172.21.15.60:0/1998359506 >> v2:172.21.15.92:3300/0 conn(0x7f6bd00b9d70 0x7f6bd01465d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:24.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bcffff700 1 --2- 172.21.15.60:0/1998359506 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6bd00b3d30 0x7f6bd0146ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:24.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bcf7fe700 1 -- 172.21.15.60:0/1998359506 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3764964576 0 0) 0x7f6bd00b4f50 con 0x7f6bd0061d90 2022-06-29T16:08:24.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bcf7fe700 1 -- 172.21.15.60:0/1998359506 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bb0003260 con 0x7f6bd0061d90 2022-06-29T16:08:24.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bcf7fe700 1 -- 172.21.15.60:0/1998359506 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1837253212 0 0) 0x7f6bb0003260 con 0x7f6bd0061d90 2022-06-29T16:08:24.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bcf7fe700 1 -- 172.21.15.60:0/1998359506 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6bd00b3d30 msgr2=0x7f6bd0146ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:24.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bcf7fe700 1 --2- 172.21.15.60:0/1998359506 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6bd00b3d30 0x7f6bd0146ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:24.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bcf7fe700 1 -- 172.21.15.60:0/1998359506 >> v2:172.21.15.92:3300/0 conn(0x7f6bd00b9d70 msgr2=0x7f6bd01465d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:24.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bcf7fe700 1 --2- 172.21.15.60:0/1998359506 >> v2:172.21.15.92:3300/0 conn(0x7f6bd00b9d70 0x7f6bd01465d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:24.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bcf7fe700 1 -- 172.21.15.60:0/1998359506 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd01461a0 con 0x7f6bd0061d90 2022-06-29T16:08:24.223 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bcf7fe700 1 -- 172.21.15.60:0/1998359506 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f6bc80019e0 con 0x7f6bd0061d90 2022-06-29T16:08:24.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bcf7fe700 1 -- 172.21.15.60:0/1998359506 >> v1:172.21.15.60:6789/0 conn(0x7f6bd0061d90 legacy=0x7f6bd005bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:24.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bcf7fe700 1 --2- 172.21.15.60:0/1998359506 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bb0003750 0x7f6bb0003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:24.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bcf7fe700 1 --2- 172.21.15.60:0/1998359506 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6bb0004500 0x7f6bb0004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:24.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bcf7fe700 1 --2- 172.21.15.60:0/1998359506 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6bd00b3d30 0x7f6bb0004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:24.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bcf7fe700 1 -- 172.21.15.60:0/1998359506 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd00ba150 con 0x7f6bb0004500 2022-06-29T16:08:24.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bcf7fe700 1 -- 172.21.15.60:0/1998359506 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd00b5180 con 0x7f6bb0003750 2022-06-29T16:08:24.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bcf7fe700 1 -- 172.21.15.60:0/1998359506 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb000b750 con 0x7f6bd00b3d30 2022-06-29T16:08:24.225 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bcffff700 1 --2- 172.21.15.60:0/1998359506 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6bb0004500 0x7f6bb0004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:24.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bd4b74700 1 --2- 172.21.15.60:0/1998359506 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bb0003750 0x7f6bb0003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:24.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bd5375700 1 --2- 172.21.15.60:0/1998359506 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6bd00b3d30 0x7f6bb0004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:24.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bcffff700 1 -- 172.21.15.60:0/1998359506 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6bd00b3d30 msgr2=0x7f6bb0004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:24.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bcffff700 1 --2- 172.21.15.60:0/1998359506 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6bd00b3d30 0x7f6bb0004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:24.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bcffff700 1 -- 172.21.15.60:0/1998359506 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bb0003750 msgr2=0x7f6bb0003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:24.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bcffff700 1 --2- 172.21.15.60:0/1998359506 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bb0003750 0x7f6bb0003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:24.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bcffff700 1 -- 172.21.15.60:0/1998359506 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb0003260 con 0x7f6bb0004500 2022-06-29T16:08:24.227 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bcffff700 1 --2- 172.21.15.60:0/1998359506 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6bb0004500 0x7f6bb0004980 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 rx=0x7f6bb800c000 tx=0x7f6bb8006df0).ready entity=mon.0 client_cookie=2830bd3f8eeedf61 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:24.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bcf7fe700 1 -- 172.21.15.60:0/1998359506 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bb800e830 con 0x7f6bb0004500 2022-06-29T16:08:24.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bcf7fe700 1 -- 172.21.15.60:0/1998359506 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bb800e990 con 0x7f6bb0004500 2022-06-29T16:08:24.228 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.213+0000 7f6bcf7fe700 1 -- 172.21.15.60:0/1998359506 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bb800ecf0 con 0x7f6bb0004500 2022-06-29T16:08:24.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.217+0000 7f6bd6dd8700 1 -- 172.21.15.60:0/1998359506 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6bb0004500 msgr2=0x7f6bb0004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:24.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.217+0000 7f6bd6dd8700 1 --2- 172.21.15.60:0/1998359506 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6bb0004500 0x7f6bb0004980 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 rx=0x7f6bb800c000 tx=0x7f6bb8006df0).stop 2022-06-29T16:08:24.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.217+0000 7f6bd5375700 1 -- 172.21.15.60:0/1998359506 reap_dead start 2022-06-29T16:08:24.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.217+0000 7f6bd6dd8700 1 -- 172.21.15.60:0/1998359506 shutdown_connections 2022-06-29T16:08:24.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.217+0000 7f6bd6dd8700 1 -- 172.21.15.60:0/1998359506 >> 172.21.15.60:0/1998359506 conn(0x7f6bd0062e70 msgr2=0x7f6bd00b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:24.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.217+0000 7f6bd6dd8700 1 -- 172.21.15.60:0/1998359506 shutdown_connections 2022-06-29T16:08:24.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.217+0000 7f6bd6dd8700 1 -- 172.21.15.60:0/1998359506 wait complete. 2022-06-29T16:08:24.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.217+0000 7f6bd6dd8700 1 Processor -- start 2022-06-29T16:08:24.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.217+0000 7f6bd6dd8700 1 -- start start 2022-06-29T16:08:24.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.221+0000 7f6bd6dd8700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6bb0004500 0x7f6bd00ae220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:24.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.221+0000 7f6bd6dd8700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bd01481e0 0x7f6bd0152180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:24.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.221+0000 7f6bd6dd8700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6bd0152670 0x7f6bd00af170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:24.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.221+0000 7f6bd6dd8700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd0156210 con 0x7f6bd0152670 2022-06-29T16:08:24.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.221+0000 7f6bd6dd8700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd0155280 con 0x7f6bd01481e0 2022-06-29T16:08:24.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.221+0000 7f6bd6dd8700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd01554b0 con 0x7f6bb0004500 2022-06-29T16:08:24.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.221+0000 7f6bcffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bd01481e0 0x7f6bd0152180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:24.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.221+0000 7f6bd4b74700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6bb0004500 0x7f6bd00ae220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:24.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.221+0000 7f6bcffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bd01481e0 0x7f6bd0152180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:60622/0 (socket says 172.21.15.60:60622) 2022-06-29T16:08:24.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.221+0000 7f6bcffff700 1 -- 172.21.15.60:0/2417683200 learned_addr learned my addr 172.21.15.60:0/2417683200 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:24.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.221+0000 7f6bd5375700 1 --2- 172.21.15.60:0/2417683200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6bd0152670 0x7f6bd00af170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:24.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.221+0000 7f6bd5375700 1 -- 172.21.15.60:0/2417683200 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6bb0004500 msgr2=0x7f6bd00ae220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:24.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.221+0000 7f6bd5375700 1 --2- 172.21.15.60:0/2417683200 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6bb0004500 0x7f6bd00ae220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:24.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.221+0000 7f6bd5375700 1 -- 172.21.15.60:0/2417683200 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bd01481e0 msgr2=0x7f6bd0152180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:24.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.221+0000 7f6bd5375700 1 --2- 172.21.15.60:0/2417683200 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bd01481e0 0x7f6bd0152180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:24.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.221+0000 7f6bd5375700 1 -- 172.21.15.60:0/2417683200 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd0151cb0 con 0x7f6bd0152670 2022-06-29T16:08:24.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.221+0000 7f6bcffff700 1 --2- 172.21.15.60:0/2417683200 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bd01481e0 0x7f6bd0152180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:08:24.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.221+0000 7f6bd5375700 1 --2- 172.21.15.60:0/2417683200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6bd0152670 0x7f6bd00af170 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 rx=0x7f6bc8000b60 tx=0x7f6bc80021d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:24.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.221+0000 7f6bcdffb700 1 -- 172.21.15.60:0/2417683200 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bc8011780 con 0x7f6bd0152670 2022-06-29T16:08:24.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.221+0000 7f6bcdffb700 1 -- 172.21.15.60:0/2417683200 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc80118e0 con 0x7f6bd0152670 2022-06-29T16:08:24.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.221+0000 7f6bd6dd8700 1 -- 172.21.15.60:0/2417683200 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bd0147770 con 0x7f6bd0152670 2022-06-29T16:08:24.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.221+0000 7f6bcdffb700 1 -- 172.21.15.60:0/2417683200 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bc8011c40 con 0x7f6bd0152670 2022-06-29T16:08:24.239 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.221+0000 7f6bd6dd8700 1 -- 172.21.15.60:0/2417683200 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bd0147000 con 0x7f6bd0152670 2022-06-29T16:08:24.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.221+0000 7f6bcdffb700 1 -- 172.21.15.60:0/2417683200 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f6bc801c360 con 0x7f6bd0152670 2022-06-29T16:08:24.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.225+0000 7f6bcdffb700 1 --2- 172.21.15.60:0/2417683200 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6bb4053a10 0x7f6bb4055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:24.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.225+0000 7f6bcdffb700 1 -- 172.21.15.60:0/2417683200 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6bc806bae0 con 0x7f6bd0152670 2022-06-29T16:08:24.240 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.225+0000 7f6bd6dd8700 1 -- 172.21.15.60:0/2417683200 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bd0151ea0 con 0x7f6bd0152670 2022-06-29T16:08:24.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.225+0000 7f6bd4b74700 1 --2- 172.21.15.60:0/2417683200 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6bb4053a10 0x7f6bb4055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:24.241 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.225+0000 7f6bd4b74700 1 --2- 172.21.15.60:0/2417683200 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6bb4053a10 0x7f6bb4055ed0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0x7f6bc000b000 tx=0x7f6bc0006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:24.242 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.233+0000 7f6bcdffb700 1 -- 172.21.15.60:0/2417683200 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6bc80429a0 con 0x7f6bd0152670 2022-06-29T16:08:24.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.357+0000 7f6bd6dd8700 1 -- 172.21.15.60:0/2417683200 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f6bd00573d0 con 0x7f6bd0152670 2022-06-29T16:08:24.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.361+0000 7f6bcdffb700 1 -- 172.21.15.60:0/2417683200 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v147) v1 ==== 110+0+10 (secure 0 0 0) 0x7f6bc806b020 con 0x7f6bd0152670 2022-06-29T16:08:24.364 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2022-06-29T16:08:24.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.361+0000 7f6bd6dd8700 1 -- 172.21.15.60:0/2417683200 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6bb4053a10 msgr2=0x7f6bb4055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:24.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.361+0000 7f6bd6dd8700 1 --2- 172.21.15.60:0/2417683200 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6bb4053a10 0x7f6bb4055ed0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0x7f6bc000b000 tx=0x7f6bc0006df0).stop 2022-06-29T16:08:24.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.361+0000 7f6bd6dd8700 1 -- 172.21.15.60:0/2417683200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6bd0152670 msgr2=0x7f6bd00af170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:24.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.361+0000 7f6bd6dd8700 1 --2- 172.21.15.60:0/2417683200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6bd0152670 0x7f6bd00af170 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 rx=0x7f6bc8000b60 tx=0x7f6bc80021d0).stop 2022-06-29T16:08:24.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.361+0000 7f6bd6dd8700 1 -- 172.21.15.60:0/2417683200 shutdown_connections 2022-06-29T16:08:24.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.361+0000 7f6bd6dd8700 1 --2- 172.21.15.60:0/2417683200 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6bb4053a10 0x7f6bb4055ed0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:24.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.361+0000 7f6bd6dd8700 1 --2- 172.21.15.60:0/2417683200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6bd0152670 0x7f6bd00af170 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:24.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.361+0000 7f6bd6dd8700 1 --2- 172.21.15.60:0/2417683200 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bd01481e0 0x7f6bd0152180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:24.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.361+0000 7f6bd6dd8700 1 --2- 172.21.15.60:0/2417683200 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6bb0004500 0x7f6bd00ae220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:24.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.361+0000 7f6bd6dd8700 1 -- 172.21.15.60:0/2417683200 >> 172.21.15.60:0/2417683200 conn(0x7f6bd0062e70 msgr2=0x7f6bd00b9300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:24.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.361+0000 7f6bd6dd8700 1 -- 172.21.15.60:0/2417683200 shutdown_connections 2022-06-29T16:08:24.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.361+0000 7f6bd6dd8700 1 -- 172.21.15.60:0/2417683200 wait complete. 2022-06-29T16:08:24.377 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-06-29T16:08:24.487 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.481+0000 7febe16a6700 1 Processor -- start 2022-06-29T16:08:24.487 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.481+0000 7febe16a6700 1 -- start start 2022-06-29T16:08:24.496 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.481+0000 7febe16a6700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7febdc0b5eb0 0x7febdc142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:24.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.481+0000 7febe16a6700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7febdc0b48b0 0x7febdc0b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:24.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.481+0000 7febe16a6700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febdc0b4d10 con 0x7febdc0b2c50 2022-06-29T16:08:24.497 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.481+0000 7febe16a6700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febdc0b54e0 con 0x7febdc0b5eb0 2022-06-29T16:08:24.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.481+0000 7febe16a6700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febdc0b6290 con 0x7febdc0b48b0 2022-06-29T16:08:24.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.485+0000 7febdbfff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7febdc0b5eb0 0x7febdc142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:24.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.485+0000 7febdb7fe700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7febdc0b48b0 0x7febdc0b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:24.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.485+0000 7febdb7fe700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7febdc0b48b0 0x7febdc0b05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:60628/0 (socket says 172.21.15.60:60628) 2022-06-29T16:08:24.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.485+0000 7febdb7fe700 1 -- 172.21.15.60:0/2190668258 learned_addr learned my addr 172.21.15.60:0/2190668258 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:24.499 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.485+0000 7febdb7fe700 1 -- 172.21.15.60:0/2190668258 >> v2:172.21.15.60:3300/0 conn(0x7febdc0b5eb0 msgr2=0x7febdc142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:24.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.485+0000 7febdb7fe700 1 --2- 172.21.15.60:0/2190668258 >> v2:172.21.15.60:3300/0 conn(0x7febdc0b5eb0 0x7febdc142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:24.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.485+0000 7febdb7fe700 1 -- 172.21.15.60:0/2190668258 >> v1:172.21.15.92:6789/0 conn(0x7febdc0b2c50 legacy=0x7febdc145fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:24.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.485+0000 7febdb7fe700 1 -- 172.21.15.60:0/2190668258 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febdc142f10 con 0x7febdc0b48b0 2022-06-29T16:08:24.500 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.485+0000 7febdbfff700 1 --2- 172.21.15.60:0/2190668258 >> v2:172.21.15.60:3300/0 conn(0x7febdc0b5eb0 0x7febdc142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:08:24.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.485+0000 7febdb7fe700 1 --2- 172.21.15.60:0/2190668258 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7febdc0b48b0 0x7febdc0b05e0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7febcc002f70 tx=0x7febcc006eb0).ready entity=mon.1 client_cookie=bdb80b5c05062c7 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:24.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.485+0000 7febdaffd700 1 -- 172.21.15.60:0/2190668258 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febcc00d5c0 con 0x7febdc0b48b0 2022-06-29T16:08:24.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.485+0000 7febdaffd700 1 -- 172.21.15.60:0/2190668258 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febcc000e90 con 0x7febdc0b48b0 2022-06-29T16:08:24.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.485+0000 7febdaffd700 1 -- 172.21.15.60:0/2190668258 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febcc00d950 con 0x7febdc0b48b0 2022-06-29T16:08:24.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.485+0000 7febe16a6700 1 -- 172.21.15.60:0/2190668258 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7febdc0b48b0 msgr2=0x7febdc0b05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:24.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.485+0000 7febe16a6700 1 --2- 172.21.15.60:0/2190668258 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7febdc0b48b0 0x7febdc0b05e0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7febcc002f70 tx=0x7febcc006eb0).stop 2022-06-29T16:08:24.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.485+0000 7febe16a6700 1 -- 172.21.15.60:0/2190668258 shutdown_connections 2022-06-29T16:08:24.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.485+0000 7febe16a6700 1 --2- 172.21.15.60:0/2190668258 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7febdc0b48b0 0x7febdc0b05e0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:24.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.485+0000 7febe16a6700 1 --2- 172.21.15.60:0/2190668258 >> v2:172.21.15.60:3300/0 conn(0x7febdc0b5eb0 0x7febdc142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:24.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.485+0000 7febe16a6700 1 -- 172.21.15.60:0/2190668258 >> 172.21.15.60:0/2190668258 conn(0x7febdc062090 msgr2=0x7febdc062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:24.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.485+0000 7febe16a6700 1 -- 172.21.15.60:0/2190668258 shutdown_connections 2022-06-29T16:08:24.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.485+0000 7febe16a6700 1 -- 172.21.15.60:0/2190668258 wait complete. 2022-06-29T16:08:24.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.485+0000 7febe16a6700 1 Processor -- start 2022-06-29T16:08:24.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.485+0000 7febe16a6700 1 -- start start 2022-06-29T16:08:24.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.489+0000 7febe16a6700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7febdc0b2c50 0x7febdc143440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:24.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.489+0000 7febe16a6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febdc0b48b0 0x7febdc151e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:24.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.489+0000 7febe16a6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7febdc0b5eb0 0x7febdc149ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:24.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.489+0000 7febe16a6700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febdc1129b0 con 0x7febdc0b48b0 2022-06-29T16:08:24.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.489+0000 7febe16a6700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febdc113a30 con 0x7febdc0b2c50 2022-06-29T16:08:24.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.489+0000 7febe16a6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febdc149380 con 0x7febdc0b5eb0 2022-06-29T16:08:24.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.489+0000 7febdb7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febdc0b48b0 0x7febdc151e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:24.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.489+0000 7febdb7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febdc0b48b0 0x7febdc151e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54606/0 (socket says 172.21.15.60:54606) 2022-06-29T16:08:24.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.489+0000 7febdb7fe700 1 -- 172.21.15.60:0/950149538 learned_addr learned my addr 172.21.15.60:0/950149538 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:24.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.489+0000 7febdbfff700 1 --2- 172.21.15.60:0/950149538 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7febdc0b2c50 0x7febdc143440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:24.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.489+0000 7febe0ea5700 1 --2- 172.21.15.60:0/950149538 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7febdc0b5eb0 0x7febdc149ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:24.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.489+0000 7febdb7fe700 1 -- 172.21.15.60:0/950149538 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7febdc0b5eb0 msgr2=0x7febdc149ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:24.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.489+0000 7febdb7fe700 1 --2- 172.21.15.60:0/950149538 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7febdc0b5eb0 0x7febdc149ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:24.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.489+0000 7febdb7fe700 1 -- 172.21.15.60:0/950149538 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7febdc0b2c50 msgr2=0x7febdc143440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:24.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.489+0000 7febdb7fe700 1 --2- 172.21.15.60:0/950149538 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7febdc0b2c50 0x7febdc143440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:24.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.489+0000 7febdb7fe700 1 -- 172.21.15.60:0/950149538 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd000b180 con 0x7febdc0b48b0 2022-06-29T16:08:24.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.489+0000 7febdb7fe700 1 --2- 172.21.15.60:0/950149538 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febdc0b48b0 0x7febdc151e60 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 rx=0x7febcc002f70 tx=0x7febcc015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:24.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.489+0000 7febd97fa700 1 -- 172.21.15.60:0/950149538 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febcc0116e0 con 0x7febdc0b48b0 2022-06-29T16:08:24.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.489+0000 7febd97fa700 1 -- 172.21.15.60:0/950149538 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febcc011840 con 0x7febdc0b48b0 2022-06-29T16:08:24.510 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.489+0000 7febe16a6700 1 -- 172.21.15.60:0/950149538 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febcc00b1e0 con 0x7febdc0b48b0 2022-06-29T16:08:24.510 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.489+0000 7febd97fa700 1 -- 172.21.15.60:0/950149538 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febcc011ba0 con 0x7febdc0b48b0 2022-06-29T16:08:24.510 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.489+0000 7febe16a6700 1 -- 172.21.15.60:0/950149538 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febdc1136f0 con 0x7febdc0b48b0 2022-06-29T16:08:24.511 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.489+0000 7febe16a6700 1 -- 172.21.15.60:0/950149538 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febdc0b4f70 con 0x7febdc0b48b0 2022-06-29T16:08:24.511 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.489+0000 7febd97fa700 1 -- 172.21.15.60:0/950149538 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7febcc01a040 con 0x7febdc0b48b0 2022-06-29T16:08:24.511 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.493+0000 7febd97fa700 1 --2- 172.21.15.60:0/950149538 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7febbc053a70 0x7febbc055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:24.511 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.493+0000 7febd97fa700 1 -- 172.21.15.60:0/950149538 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6073+0+0 (secure 0 0 0) 0x7febcc06cbe0 con 0x7febdc0b48b0 2022-06-29T16:08:24.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.493+0000 7febdbfff700 1 --2- 172.21.15.60:0/950149538 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7febbc053a70 0x7febbc055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:24.512 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.493+0000 7febdbfff700 1 --2- 172.21.15.60:0/950149538 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7febbc053a70 0x7febbc055f30 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0x7febd000b000 tx=0x7febd0006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:24.513 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.497+0000 7febd97fa700 1 -- 172.21.15.60:0/950149538 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7febcc043be0 con 0x7febdc0b48b0 2022-06-29T16:08:24.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.625+0000 7febe16a6700 1 -- 172.21.15.60:0/950149538 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7febdc0573d0 con 0x7febdc0b48b0 2022-06-29T16:08:24.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.625+0000 7febd97fa700 1 -- 172.21.15.60:0/950149538 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v147) v1 ==== 92+0+10 (secure 0 0 0) 0x7febcc023020 con 0x7febdc0b48b0 2022-06-29T16:08:24.630 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:08:24.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.629+0000 7febe16a6700 1 -- 172.21.15.60:0/950149538 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7febbc053a70 msgr2=0x7febbc055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:24.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.629+0000 7febe16a6700 1 --2- 172.21.15.60:0/950149538 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7febbc053a70 0x7febbc055f30 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0x7febd000b000 tx=0x7febd0006df0).stop 2022-06-29T16:08:24.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.629+0000 7febe16a6700 1 -- 172.21.15.60:0/950149538 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febdc0b48b0 msgr2=0x7febdc151e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:24.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.629+0000 7febe16a6700 1 --2- 172.21.15.60:0/950149538 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febdc0b48b0 0x7febdc151e60 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 rx=0x7febcc002f70 tx=0x7febcc015000).stop 2022-06-29T16:08:24.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.629+0000 7febe16a6700 1 -- 172.21.15.60:0/950149538 shutdown_connections 2022-06-29T16:08:24.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.629+0000 7febe16a6700 1 --2- 172.21.15.60:0/950149538 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7febbc053a70 0x7febbc055f30 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:24.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.629+0000 7febe16a6700 1 --2- 172.21.15.60:0/950149538 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7febdc0b5eb0 0x7febdc149ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:24.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.629+0000 7febe16a6700 1 --2- 172.21.15.60:0/950149538 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7febdc0b48b0 0x7febdc151e60 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:24.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.629+0000 7febe16a6700 1 --2- 172.21.15.60:0/950149538 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7febdc0b2c50 0x7febdc143440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:24.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.629+0000 7febe16a6700 1 -- 172.21.15.60:0/950149538 >> 172.21.15.60:0/950149538 conn(0x7febdc062090 msgr2=0x7febdc0b7040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:24.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.629+0000 7febe16a6700 1 -- 172.21.15.60:0/950149538 shutdown_connections 2022-06-29T16:08:24.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.629+0000 7febe16a6700 1 -- 172.21.15.60:0/950149538 wait complete. 2022-06-29T16:08:24.643 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-06-29T16:08:24.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.749+0000 7f9216323700 1 Processor -- start 2022-06-29T16:08:24.753 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.749+0000 7f9216323700 1 -- start start 2022-06-29T16:08:24.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.749+0000 7f9216323700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f921013f790 0x7f921013bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:24.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.749+0000 7f9216323700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f921013fc00 0x7f921013c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:24.756 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.749+0000 7f9216323700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9210142820 con 0x7f92100b61d0 2022-06-29T16:08:24.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.749+0000 7f9216323700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92100b0c50 con 0x7f921013f790 2022-06-29T16:08:24.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.749+0000 7f9216323700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9210062bb0 con 0x7f921013fc00 2022-06-29T16:08:24.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.749+0000 7f920ffff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f92100b61d0 0x7f92100b0540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34466/0 (socket says 172.21.15.60:34466) 2022-06-29T16:08:24.757 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.749+0000 7f920ffff700 1 -- 172.21.15.60:0/1503431322 learned_addr learned my addr 172.21.15.60:0/1503431322 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:24.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.749+0000 7f920f7fe700 1 --2- 172.21.15.60:0/1503431322 >> v2:172.21.15.92:3300/0 conn(0x7f921013f790 0x7f921013bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:24.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.749+0000 7f92148c0700 1 --2- 172.21.15.60:0/1503431322 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f921013fc00 0x7f921013c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:24.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.749+0000 7f920effd700 1 -- 172.21.15.60:0/1503431322 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3034497337 0 0) 0x7f9210142820 con 0x7f92100b61d0 2022-06-29T16:08:24.758 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.749+0000 7f920effd700 1 -- 172.21.15.60:0/1503431322 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91fc003260 con 0x7f92100b61d0 2022-06-29T16:08:24.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.749+0000 7f920effd700 1 -- 172.21.15.60:0/1503431322 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2595165431 0 0) 0x7f91fc003260 con 0x7f92100b61d0 2022-06-29T16:08:24.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.749+0000 7f920f7fe700 1 -- 172.21.15.60:0/1503431322 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f921013fc00 msgr2=0x7f921013c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:24.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.749+0000 7f920f7fe700 1 --2- 172.21.15.60:0/1503431322 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f921013fc00 0x7f921013c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:24.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.749+0000 7f920f7fe700 1 -- 172.21.15.60:0/1503431322 >> v1:172.21.15.60:6789/0 conn(0x7f92100b61d0 legacy=0x7f92100b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:24.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.749+0000 7f920f7fe700 1 -- 172.21.15.60:0/1503431322 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92100b65b0 con 0x7f921013f790 2022-06-29T16:08:24.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.749+0000 7f920f7fe700 1 --2- 172.21.15.60:0/1503431322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f921013f790 0x7f921013bc40 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7f9200001800 tx=0x7f9204006df0).ready entity=mon.1 client_cookie=d9c609bfca3f815f server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:24.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.749+0000 7f920effd700 1 -- 172.21.15.60:0/1503431322 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9200001660 con 0x7f921013f790 2022-06-29T16:08:24.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.753+0000 7f920effd700 1 -- 172.21.15.60:0/1503431322 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9204000e90 con 0x7f921013f790 2022-06-29T16:08:24.760 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.753+0000 7f920effd700 1 -- 172.21.15.60:0/1503431322 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9204012460 con 0x7f921013f790 2022-06-29T16:08:24.764 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.753+0000 7f9216323700 1 -- 172.21.15.60:0/1503431322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f921013f790 msgr2=0x7f921013bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:24.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.753+0000 7f9216323700 1 --2- 172.21.15.60:0/1503431322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f921013f790 0x7f921013bc40 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7f9200001800 tx=0x7f9204006df0).stop 2022-06-29T16:08:24.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.753+0000 7f9216323700 1 -- 172.21.15.60:0/1503431322 shutdown_connections 2022-06-29T16:08:24.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.753+0000 7f9216323700 1 --2- 172.21.15.60:0/1503431322 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f921013fc00 0x7f921013c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:24.765 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.753+0000 7f9216323700 1 --2- 172.21.15.60:0/1503431322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f921013f790 0x7f921013bc40 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:24.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.753+0000 7f9216323700 1 -- 172.21.15.60:0/1503431322 >> 172.21.15.60:0/1503431322 conn(0x7f9210063b50 msgr2=0x7f92100b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:24.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.753+0000 7f9216323700 1 -- 172.21.15.60:0/1503431322 shutdown_connections 2022-06-29T16:08:24.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.753+0000 7f9216323700 1 -- 172.21.15.60:0/1503431322 wait complete. 2022-06-29T16:08:24.766 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.753+0000 7f9216323700 1 Processor -- start 2022-06-29T16:08:24.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.753+0000 7f9216323700 1 -- start start 2022-06-29T16:08:24.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.753+0000 7f9216323700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92100b61d0 0x7f92100643b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:24.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.753+0000 7f9216323700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f921013f790 0x7f9210150c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:24.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.753+0000 7f9216323700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f921013fc00 0x7f9210151180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:24.767 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.753+0000 7f9216323700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9210064cf0 con 0x7f921013f790 2022-06-29T16:08:24.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.753+0000 7f9216323700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9210140a10 con 0x7f92100b61d0 2022-06-29T16:08:24.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.753+0000 7f9216323700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9210140c40 con 0x7f921013fc00 2022-06-29T16:08:24.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.757+0000 7f920f7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f921013f790 0x7f9210150c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:24.768 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.757+0000 7f920f7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f921013f790 0x7f9210150c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54620/0 (socket says 172.21.15.60:54620) 2022-06-29T16:08:24.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.757+0000 7f920f7fe700 1 -- 172.21.15.60:0/265933462 learned_addr learned my addr 172.21.15.60:0/265933462 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:24.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.757+0000 7f92148c0700 1 --2- 172.21.15.60:0/265933462 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f921013fc00 0x7f9210151180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:24.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.757+0000 7f920ffff700 1 --2- 172.21.15.60:0/265933462 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92100b61d0 0x7f92100643b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:24.769 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.757+0000 7f920f7fe700 1 -- 172.21.15.60:0/265933462 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f921013fc00 msgr2=0x7f9210151180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:24.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.757+0000 7f920f7fe700 1 --2- 172.21.15.60:0/265933462 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f921013fc00 0x7f9210151180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:24.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.757+0000 7f920f7fe700 1 -- 172.21.15.60:0/265933462 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92100b61d0 msgr2=0x7f92100643b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:24.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.757+0000 7f920f7fe700 1 --2- 172.21.15.60:0/265933462 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92100b61d0 0x7f92100643b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:24.770 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.757+0000 7f920f7fe700 1 -- 172.21.15.60:0/265933462 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f920400b180 con 0x7f921013f790 2022-06-29T16:08:24.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.757+0000 7f920f7fe700 1 --2- 172.21.15.60:0/265933462 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f921013f790 0x7f9210150c90 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 rx=0x7f9200001800 tx=0x7f9204014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:24.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.757+0000 7f920d7fa700 1 -- 172.21.15.60:0/265933462 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9204016070 con 0x7f921013f790 2022-06-29T16:08:24.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.757+0000 7f920d7fa700 1 -- 172.21.15.60:0/265933462 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9204013e40 con 0x7f921013f790 2022-06-29T16:08:24.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.757+0000 7f920d7fa700 1 -- 172.21.15.60:0/265933462 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f920400dd00 con 0x7f921013f790 2022-06-29T16:08:24.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.757+0000 7f9216323700 1 -- 172.21.15.60:0/265933462 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9210066e20 con 0x7f921013f790 2022-06-29T16:08:24.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.757+0000 7f9216323700 1 -- 172.21.15.60:0/265933462 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92100af370 con 0x7f921013f790 2022-06-29T16:08:24.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.757+0000 7f9216323700 1 -- 172.21.15.60:0/265933462 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92100573d0 con 0x7f921013f790 2022-06-29T16:08:24.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.757+0000 7f920d7fa700 1 -- 172.21.15.60:0/265933462 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9204024020 con 0x7f921013f790 2022-06-29T16:08:24.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.757+0000 7f920d7fa700 1 --2- 172.21.15.60:0/265933462 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f91f0053aa0 0x7f91f0055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:24.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.757+0000 7f920d7fa700 1 -- 172.21.15.60:0/265933462 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f920406ce30 con 0x7f921013f790 2022-06-29T16:08:24.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.761+0000 7f920ffff700 1 --2- 172.21.15.60:0/265933462 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f91f0053aa0 0x7f91f0055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:24.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.761+0000 7f920ffff700 1 --2- 172.21.15.60:0/265933462 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f91f0053aa0 0x7f91f0055f60 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0x7f920000d000 tx=0x7f9200001de0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:24.774 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.765+0000 7f920d7fa700 1 -- 172.21.15.60:0/265933462 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9204043cb0 con 0x7f921013f790 2022-06-29T16:08:24.897 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.893+0000 7f9216323700 1 -- 172.21.15.60:0/265933462 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f92100686b0 con 0x7f921013f790 2022-06-29T16:08:24.898 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.893+0000 7f920d7fa700 1 -- 172.21.15.60:0/265933462 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v147) v1 ==== 104+0+10 (secure 0 0 0) 0x7f92040472d0 con 0x7f921013f790 2022-06-29T16:08:24.899 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:08:24.901 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.893+0000 7f9216323700 1 -- 172.21.15.60:0/265933462 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f91f0053aa0 msgr2=0x7f91f0055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:24.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.893+0000 7f9216323700 1 --2- 172.21.15.60:0/265933462 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f91f0053aa0 0x7f91f0055f60 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0x7f920000d000 tx=0x7f9200001de0).stop 2022-06-29T16:08:24.902 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.893+0000 7f9216323700 1 -- 172.21.15.60:0/265933462 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f921013f790 msgr2=0x7f9210150c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:24.903 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.893+0000 7f9216323700 1 --2- 172.21.15.60:0/265933462 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f921013f790 0x7f9210150c90 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 rx=0x7f9200001800 tx=0x7f9204014000).stop 2022-06-29T16:08:24.903 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.893+0000 7f9216323700 1 -- 172.21.15.60:0/265933462 shutdown_connections 2022-06-29T16:08:24.903 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.893+0000 7f9216323700 1 --2- 172.21.15.60:0/265933462 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f91f0053aa0 0x7f91f0055f60 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:24.904 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.893+0000 7f9216323700 1 --2- 172.21.15.60:0/265933462 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f921013fc00 0x7f9210151180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:24.904 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.893+0000 7f9216323700 1 --2- 172.21.15.60:0/265933462 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f921013f790 0x7f9210150c90 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:24.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.893+0000 7f9216323700 1 --2- 172.21.15.60:0/265933462 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92100b61d0 0x7f92100643b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:24.905 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.897+0000 7f9216323700 1 -- 172.21.15.60:0/265933462 >> 172.21.15.60:0/265933462 conn(0x7f9210063b50 msgr2=0x7f921005f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:24.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.897+0000 7f9216323700 1 -- 172.21.15.60:0/265933462 shutdown_connections 2022-06-29T16:08:24.906 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:24.897+0000 7f9216323700 1 -- 172.21.15.60:0/265933462 wait complete. 2022-06-29T16:08:24.912 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-06-29T16:08:25.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.021+0000 7f0a1d07b700 1 Processor -- start 2022-06-29T16:08:25.026 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.021+0000 7f0a1d07b700 1 -- start start 2022-06-29T16:08:25.031 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.021+0000 7f0a1d07b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a180b2270 0x7f0a180b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:25.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.021+0000 7f0a1d07b700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a180b2650 con 0x7f0a180aca20 2022-06-29T16:08:25.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.021+0000 7f0a1d07b700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a18062af0 con 0x7f0a180634d0 2022-06-29T16:08:25.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.021+0000 7f0a1d07b700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a18067770 con 0x7f0a180b2270 2022-06-29T16:08:25.032 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.021+0000 7f0a1c87a700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f0a180aca20 0x7f0a180af720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34480/0 (socket says 172.21.15.60:34480) 2022-06-29T16:08:25.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.021+0000 7f0a1c87a700 1 -- 172.21.15.60:0/51671113 learned_addr learned my addr 172.21.15.60:0/51671113 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:25.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a17fff700 1 --2- 172.21.15.60:0/51671113 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a180b2270 0x7f0a180b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:25.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a177fe700 1 -- 172.21.15.60:0/51671113 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2117276771 0 0) 0x7f0a180b2650 con 0x7f0a180aca20 2022-06-29T16:08:25.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a177fe700 1 -- 172.21.15.60:0/51671113 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09fc003260 con 0x7f0a180aca20 2022-06-29T16:08:25.033 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a177fe700 1 -- 172.21.15.60:0/51671113 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4096204607 0 0) 0x7f0a18062af0 con 0x7f0a180634d0 2022-06-29T16:08:25.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a177fe700 1 -- 172.21.15.60:0/51671113 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a180b2650 con 0x7f0a180634d0 2022-06-29T16:08:25.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a177fe700 1 -- 172.21.15.60:0/51671113 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2168024018 0 0) 0x7f09fc003260 con 0x7f0a180aca20 2022-06-29T16:08:25.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a177fe700 1 -- 172.21.15.60:0/51671113 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a180b2270 msgr2=0x7f0a180b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:25.034 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a177fe700 1 --2- 172.21.15.60:0/51671113 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a180b2270 0x7f0a180b0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:25.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a177fe700 1 -- 172.21.15.60:0/51671113 >> v1:172.21.15.92:6789/0 conn(0x7f0a180634d0 legacy=0x7f0a1813e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:25.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a177fe700 1 -- 172.21.15.60:0/51671113 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a18062d90 con 0x7f0a180aca20 2022-06-29T16:08:25.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a177fe700 1 -- 172.21.15.60:0/51671113 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f0a100019e0 con 0x7f0a180aca20 2022-06-29T16:08:25.035 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a177fe700 1 -- 172.21.15.60:0/51671113 >> v1:172.21.15.60:6789/0 conn(0x7f0a180aca20 legacy=0x7f0a180af720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:25.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a177fe700 1 --2- 172.21.15.60:0/51671113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09fc003910 0x7f09fc003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:25.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a177fe700 1 --2- 172.21.15.60:0/51671113 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09fc004500 0x7f09fc004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:25.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a177fe700 1 --2- 172.21.15.60:0/51671113 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09fc004e70 0x7f09fc00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:25.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a177fe700 1 -- 172.21.15.60:0/51671113 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a18067770 con 0x7f09fc003910 2022-06-29T16:08:25.036 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a177fe700 1 -- 172.21.15.60:0/51671113 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc00bd70 con 0x7f09fc004e70 2022-06-29T16:08:25.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a177fe700 1 -- 172.21.15.60:0/51671113 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc00bf30 con 0x7f09fc004500 2022-06-29T16:08:25.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a17fff700 1 --2- 172.21.15.60:0/51671113 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a180b2270 0x7f0a180b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:08:25.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a1c87a700 1 --2- 172.21.15.60:0/51671113 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09fc004e70 0x7f09fc00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:25.037 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a17fff700 1 --2- 172.21.15.60:0/51671113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09fc003910 0x7f09fc003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:25.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a0ffff700 1 --2- 172.21.15.60:0/51671113 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09fc004500 0x7f09fc004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:25.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a1c87a700 1 -- 172.21.15.60:0/51671113 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09fc004500 msgr2=0x7f09fc004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:25.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a1c87a700 1 --2- 172.21.15.60:0/51671113 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f09fc004500 0x7f09fc004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:25.038 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a1c87a700 1 -- 172.21.15.60:0/51671113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09fc003910 msgr2=0x7f09fc003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:25.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a1c87a700 1 --2- 172.21.15.60:0/51671113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09fc003910 0x7f09fc003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:25.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a1c87a700 1 -- 172.21.15.60:0/51671113 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09fc003260 con 0x7f09fc004e70 2022-06-29T16:08:25.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a17fff700 1 --2- 172.21.15.60:0/51671113 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f09fc003910 0x7f09fc003d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:08:25.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a1c87a700 1 --2- 172.21.15.60:0/51671113 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09fc004e70 0x7f09fc00b750 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7f0a100017b0 tx=0x7f0a10002150).ready entity=mon.1 client_cookie=fe07631335d974d8 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:25.039 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a177fe700 1 -- 172.21.15.60:0/51671113 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a1000cbe0 con 0x7f09fc004e70 2022-06-29T16:08:25.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a177fe700 1 -- 172.21.15.60:0/51671113 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a1000cd40 con 0x7f09fc004e70 2022-06-29T16:08:25.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a177fe700 1 -- 172.21.15.60:0/51671113 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a10005b80 con 0x7f09fc004e70 2022-06-29T16:08:25.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a1d07b700 1 -- 172.21.15.60:0/51671113 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09fc004e70 msgr2=0x7f09fc00b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:25.040 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a1d07b700 1 --2- 172.21.15.60:0/51671113 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09fc004e70 0x7f09fc00b750 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7f0a100017b0 tx=0x7f0a10002150).stop 2022-06-29T16:08:25.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a1c87a700 1 -- 172.21.15.60:0/51671113 reap_dead start 2022-06-29T16:08:25.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a1d07b700 1 -- 172.21.15.60:0/51671113 shutdown_connections 2022-06-29T16:08:25.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a1d07b700 1 -- 172.21.15.60:0/51671113 >> 172.21.15.60:0/51671113 conn(0x7f0a180b5890 msgr2=0x7f0a180b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:25.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a1d07b700 1 -- 172.21.15.60:0/51671113 shutdown_connections 2022-06-29T16:08:25.041 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.025+0000 7f0a1d07b700 1 -- 172.21.15.60:0/51671113 wait complete. 2022-06-29T16:08:25.045 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.029+0000 7f0a1d07b700 1 Processor -- start 2022-06-29T16:08:25.045 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.029+0000 7f0a1d07b700 1 -- start start 2022-06-29T16:08:25.045 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.029+0000 7f0a1d07b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09fc004e70 0x7f0a180b3450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:25.046 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.029+0000 7f0a1d07b700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a1813fdc0 0x7f0a181551e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:25.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.029+0000 7f0a1d07b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a18066a80 0x7f0a1813ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:25.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.029+0000 7f0a1d07b700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a18065e50 con 0x7f0a18066a80 2022-06-29T16:08:25.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.029+0000 7f0a1d07b700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1813e570 con 0x7f09fc004e70 2022-06-29T16:08:25.047 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.029+0000 7f0a1d07b700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a18065ab0 con 0x7f0a1813fdc0 2022-06-29T16:08:25.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.029+0000 7f0a1c87a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a18066a80 0x7f0a1813ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:25.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.029+0000 7f0a1c87a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a18066a80 0x7f0a1813ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54642/0 (socket says 172.21.15.60:54642) 2022-06-29T16:08:25.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.029+0000 7f0a1c87a700 1 -- 172.21.15.60:0/994684014 learned_addr learned my addr 172.21.15.60:0/994684014 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:25.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.029+0000 7f0a17fff700 1 --2- 172.21.15.60:0/994684014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09fc004e70 0x7f0a180b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:25.048 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.029+0000 7f0a0ffff700 1 --2- 172.21.15.60:0/994684014 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a1813fdc0 0x7f0a181551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:25.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.033+0000 7f0a1c87a700 1 -- 172.21.15.60:0/994684014 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a1813fdc0 msgr2=0x7f0a181551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:25.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.033+0000 7f0a1c87a700 1 --2- 172.21.15.60:0/994684014 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a1813fdc0 0x7f0a181551e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:25.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.033+0000 7f0a1c87a700 1 -- 172.21.15.60:0/994684014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09fc004e70 msgr2=0x7f0a180b3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:25.049 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.033+0000 7f0a1c87a700 1 --2- 172.21.15.60:0/994684014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09fc004e70 0x7f0a180b3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:25.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.033+0000 7f0a1c87a700 1 -- 172.21.15.60:0/994684014 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a181556d0 con 0x7f0a18066a80 2022-06-29T16:08:25.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.033+0000 7f0a1c87a700 1 --2- 172.21.15.60:0/994684014 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a18066a80 0x7f0a1813ea10 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 rx=0x7f0a18067b60 tx=0x7f0a10015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:25.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.033+0000 7f0a15ffb700 1 -- 172.21.15.60:0/994684014 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a100072c0 con 0x7f0a18066a80 2022-06-29T16:08:25.050 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.033+0000 7f0a15ffb700 1 -- 172.21.15.60:0/994684014 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a10007420 con 0x7f0a18066a80 2022-06-29T16:08:25.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.033+0000 7f0a1d07b700 1 -- 172.21.15.60:0/994684014 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a1813ef00 con 0x7f0a18066a80 2022-06-29T16:08:25.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.033+0000 7f0a15ffb700 1 -- 172.21.15.60:0/994684014 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a10007780 con 0x7f0a18066a80 2022-06-29T16:08:25.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.033+0000 7f0a1d07b700 1 -- 172.21.15.60:0/994684014 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a18063980 con 0x7f0a18066a80 2022-06-29T16:08:25.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.033+0000 7f0a1d07b700 1 -- 172.21.15.60:0/994684014 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a1813e730 con 0x7f0a18066a80 2022-06-29T16:08:25.051 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.033+0000 7f0a15ffb700 1 -- 172.21.15.60:0/994684014 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0a10026030 con 0x7f0a18066a80 2022-06-29T16:08:25.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.033+0000 7f0a15ffb700 1 --2- 172.21.15.60:0/994684014 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f09f0053aa0 0x7f09f0055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:25.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.033+0000 7f0a15ffb700 1 -- 172.21.15.60:0/994684014 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0a1006ca20 con 0x7f0a18066a80 2022-06-29T16:08:25.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.033+0000 7f0a17fff700 1 --2- 172.21.15.60:0/994684014 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f09f0053aa0 0x7f09f0055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:25.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.037+0000 7f0a17fff700 1 --2- 172.21.15.60:0/994684014 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f09f0053aa0 0x7f09f0055f60 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0x7f0a0800b000 tx=0x7f0a08006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:25.052 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.041+0000 7f0a15ffb700 1 -- 172.21.15.60:0/994684014 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0a10043980 con 0x7f0a18066a80 2022-06-29T16:08:25.175 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.169+0000 7f0a1d07b700 1 -- 172.21.15.60:0/994684014 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0a180573d0 con 0x7f0a18066a80 2022-06-29T16:08:25.177 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.173+0000 7f0a15ffb700 1 -- 172.21.15.60:0/994684014 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v147) v1 ==== 100+0+10 (secure 0 0 0) 0x7f0a10024070 con 0x7f0a18066a80 2022-06-29T16:08:25.178 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2022-06-29T16:08:25.180 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.173+0000 7f0a1d07b700 1 -- 172.21.15.60:0/994684014 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f09f0053aa0 msgr2=0x7f09f0055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:25.181 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.173+0000 7f0a1d07b700 1 --2- 172.21.15.60:0/994684014 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f09f0053aa0 0x7f09f0055f60 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0x7f0a0800b000 tx=0x7f0a08006df0).stop 2022-06-29T16:08:25.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.173+0000 7f0a1d07b700 1 -- 172.21.15.60:0/994684014 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a18066a80 msgr2=0x7f0a1813ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:25.182 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.173+0000 7f0a1d07b700 1 --2- 172.21.15.60:0/994684014 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a18066a80 0x7f0a1813ea10 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 rx=0x7f0a18067b60 tx=0x7f0a10015000).stop 2022-06-29T16:08:25.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.173+0000 7f0a1d07b700 1 -- 172.21.15.60:0/994684014 shutdown_connections 2022-06-29T16:08:25.183 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.173+0000 7f0a1d07b700 1 --2- 172.21.15.60:0/994684014 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f09f0053aa0 0x7f09f0055f60 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:25.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.173+0000 7f0a1d07b700 1 --2- 172.21.15.60:0/994684014 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a18066a80 0x7f0a1813ea10 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:25.184 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.173+0000 7f0a1d07b700 1 --2- 172.21.15.60:0/994684014 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0a1813fdc0 0x7f0a181551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:25.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.173+0000 7f0a1d07b700 1 --2- 172.21.15.60:0/994684014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09fc004e70 0x7f0a180b3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:25.185 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.173+0000 7f0a1d07b700 1 -- 172.21.15.60:0/994684014 >> 172.21.15.60:0/994684014 conn(0x7f0a180b5890 msgr2=0x7f0a181367b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:25.186 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.173+0000 7f0a1d07b700 1 -- 172.21.15.60:0/994684014 shutdown_connections 2022-06-29T16:08:25.187 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.173+0000 7f0a1d07b700 1 -- 172.21.15.60:0/994684014 wait complete. 2022-06-29T16:08:25.191 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-06-29T16:08:25.192 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:08:25.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.297+0000 7f9a8548f700 1 Processor -- start 2022-06-29T16:08:25.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.297+0000 7f9a8548f700 1 -- start start 2022-06-29T16:08:25.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.297+0000 7f9a8548f700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9a800aed00 0x7f9a80146810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:25.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.297+0000 7f9a8548f700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a8005d450 con 0x7f9a800b2130 2022-06-29T16:08:25.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.297+0000 7f9a8548f700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a800b25d0 con 0x7f9a8005d070 2022-06-29T16:08:25.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.297+0000 7f9a8548f700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a800adf90 con 0x7f9a800aed00 2022-06-29T16:08:25.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.297+0000 7f9a84c8e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f9a800aed00 0x7f9a80146810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:25.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.297+0000 7f9a7f7fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9a800b2130 0x7f9a80146100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34502/0 (socket says 172.21.15.60:34502) 2022-06-29T16:08:25.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.297+0000 7f9a7f7fe700 1 -- 172.21.15.60:0/333985540 learned_addr learned my addr 172.21.15.60:0/333985540 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:25.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.297+0000 7f9a84c8e700 1 -- 172.21.15.60:0/333985540 >> v1:172.21.15.92:6789/0 conn(0x7f9a8005d070 legacy=0x7f9a8014aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:25.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.297+0000 7f9a84c8e700 1 -- 172.21.15.60:0/333985540 >> v1:172.21.15.60:6789/0 conn(0x7f9a800b2130 legacy=0x7f9a80146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:25.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.297+0000 7f9a7effd700 1 -- 172.21.15.60:0/333985540 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 529670930 0 0) 0x7f9a800b25d0 con 0x7f9a8005d070 2022-06-29T16:08:25.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.297+0000 7f9a84c8e700 1 -- 172.21.15.60:0/333985540 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a8014a350 con 0x7f9a800aed00 2022-06-29T16:08:25.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.297+0000 7f9a84c8e700 1 --2- 172.21.15.60:0/333985540 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a800aed00 0x7f9a80146810 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 rx=0x7f9a68001110 tx=0x7f9a70006df0).ready entity=mon.0 client_cookie=24e49b9825b46b14 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:25.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.297+0000 7f9a7effd700 1 -- 172.21.15.60:0/333985540 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a7000d5c0 con 0x7f9a800aed00 2022-06-29T16:08:25.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.297+0000 7f9a7effd700 1 -- 172.21.15.60:0/333985540 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a70000e90 con 0x7f9a800aed00 2022-06-29T16:08:25.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.297+0000 7f9a7effd700 1 -- 172.21.15.60:0/333985540 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a7000d910 con 0x7f9a800aed00 2022-06-29T16:08:25.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.301+0000 7f9a8548f700 1 -- 172.21.15.60:0/333985540 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a800aed00 msgr2=0x7f9a80146810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:25.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.301+0000 7f9a8548f700 1 --2- 172.21.15.60:0/333985540 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a800aed00 0x7f9a80146810 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 rx=0x7f9a68001110 tx=0x7f9a70006df0).stop 2022-06-29T16:08:25.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.301+0000 7f9a8548f700 1 -- 172.21.15.60:0/333985540 shutdown_connections 2022-06-29T16:08:25.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.301+0000 7f9a8548f700 1 --2- 172.21.15.60:0/333985540 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a800aed00 0x7f9a80146810 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:25.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.301+0000 7f9a8548f700 1 -- 172.21.15.60:0/333985540 >> 172.21.15.60:0/333985540 conn(0x7f9a800b5890 msgr2=0x7f9a800b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:25.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.301+0000 7f9a8548f700 1 -- 172.21.15.60:0/333985540 shutdown_connections 2022-06-29T16:08:25.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.301+0000 7f9a8548f700 1 -- 172.21.15.60:0/333985540 wait complete. 2022-06-29T16:08:25.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.301+0000 7f9a8548f700 1 Processor -- start 2022-06-29T16:08:25.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.301+0000 7f9a8548f700 1 -- start start 2022-06-29T16:08:25.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.305+0000 7f9a8548f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a8005d070 0x7f9a800b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:25.315 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.305+0000 7f9a8548f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a800aed00 0x7f9a80156210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:25.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.305+0000 7f9a8548f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a800b2130 0x7f9a80064040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:25.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.305+0000 7f9a8548f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a80066ec0 con 0x7f9a8005d070 2022-06-29T16:08:25.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.305+0000 7f9a8548f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a80066980 con 0x7f9a800b2130 2022-06-29T16:08:25.316 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.305+0000 7f9a8548f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a80066c20 con 0x7f9a800aed00 2022-06-29T16:08:25.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.305+0000 7f9a7ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a8005d070 0x7f9a800b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:25.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.305+0000 7f9a84c8e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a800b2130 0x7f9a80064040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:25.317 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.305+0000 7f9a7f7fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a800aed00 0x7f9a80156210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:25.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.305+0000 7f9a7ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a8005d070 0x7f9a800b40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54656/0 (socket says 172.21.15.60:54656) 2022-06-29T16:08:25.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.305+0000 7f9a7ffff700 1 -- 172.21.15.60:0/2723482160 learned_addr learned my addr 172.21.15.60:0/2723482160 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:25.318 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.305+0000 7f9a7ffff700 1 -- 172.21.15.60:0/2723482160 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a800aed00 msgr2=0x7f9a80156210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:25.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.305+0000 7f9a7ffff700 1 --2- 172.21.15.60:0/2723482160 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a800aed00 0x7f9a80156210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:25.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.305+0000 7f9a7ffff700 1 -- 172.21.15.60:0/2723482160 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a800b2130 msgr2=0x7f9a80064040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:25.319 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.305+0000 7f9a7ffff700 1 --2- 172.21.15.60:0/2723482160 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a800b2130 0x7f9a80064040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:25.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.305+0000 7f9a7ffff700 1 -- 172.21.15.60:0/2723482160 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a7000b180 con 0x7f9a8005d070 2022-06-29T16:08:25.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.305+0000 7f9a84c8e700 1 --2- 172.21.15.60:0/2723482160 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a800b2130 0x7f9a80064040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:08:25.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.305+0000 7f9a7ffff700 1 --2- 172.21.15.60:0/2723482160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a8005d070 0x7f9a800b40c0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 rx=0x7f9a78000f40 tx=0x7f9a78007df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:25.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.305+0000 7f9a7d7fa700 1 -- 172.21.15.60:0/2723482160 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a7800e4a0 con 0x7f9a8005d070 2022-06-29T16:08:25.320 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.305+0000 7f9a7d7fa700 1 -- 172.21.15.60:0/2723482160 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a7800e600 con 0x7f9a8005d070 2022-06-29T16:08:25.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.305+0000 7f9a8548f700 1 -- 172.21.15.60:0/2723482160 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a80116910 con 0x7f9a8005d070 2022-06-29T16:08:25.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.305+0000 7f9a7d7fa700 1 -- 172.21.15.60:0/2723482160 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a7800e960 con 0x7f9a8005d070 2022-06-29T16:08:25.321 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.305+0000 7f9a8548f700 1 -- 172.21.15.60:0/2723482160 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a80156700 con 0x7f9a8005d070 2022-06-29T16:08:25.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.309+0000 7f9a7d7fa700 1 -- 172.21.15.60:0/2723482160 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9a78019030 con 0x7f9a8005d070 2022-06-29T16:08:25.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.309+0000 7f9a8548f700 1 -- 172.21.15.60:0/2723482160 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a8014a5f0 con 0x7f9a8005d070 2022-06-29T16:08:25.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.309+0000 7f9a7d7fa700 1 --2- 172.21.15.60:0/2723482160 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9a64053b00 0x7f9a64055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:25.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.309+0000 7f9a7d7fa700 1 -- 172.21.15.60:0/2723482160 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9a7806b550 con 0x7f9a8005d070 2022-06-29T16:08:25.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.309+0000 7f9a7f7fe700 1 --2- 172.21.15.60:0/2723482160 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9a64053b00 0x7f9a64055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:25.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.309+0000 7f9a7f7fe700 1 --2- 172.21.15.60:0/2723482160 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9a64053b00 0x7f9a64055fc0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0x7f9a6800d000 tx=0x7f9a68009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:25.324 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.313+0000 7f9a7d7fa700 1 -- 172.21.15.60:0/2723482160 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9a78042470 con 0x7f9a8005d070 2022-06-29T16:08:25.479 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.473+0000 7f9a8548f700 1 -- 172.21.15.60:0/2723482160 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a800573d0 con 0x7f9a8005d070 2022-06-29T16:08:25.480 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.473+0000 7f9a7d7fa700 1 -- 172.21.15.60:0/2723482160 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+2139 (secure 0 0 0) 0x7f9a78045a90 con 0x7f9a8005d070 2022-06-29T16:08:25.480 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:08:25.480 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":73,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7457,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.92:6835/698111951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":698111951},{"type":"v1","addr":"172.21.15.92:6835","nonce":698111951}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":72}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:08:17.075855+0000","modified":"2022-06-29T16:08:18.079541+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8790},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8790":{"gid":8790,"name":"a","rank":0,"incarnation":72,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/3619856083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3619856083},{"type":"v1","addr":"172.21.15.60:6833","nonce":3619856083}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2022-06-29T16:08:25.481 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.477+0000 7f9a8548f700 1 -- 172.21.15.60:0/2723482160 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9a64053b00 msgr2=0x7f9a64055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:25.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.477+0000 7f9a8548f700 1 --2- 172.21.15.60:0/2723482160 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9a64053b00 0x7f9a64055fc0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0x7f9a6800d000 tx=0x7f9a68009040).stop 2022-06-29T16:08:25.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.477+0000 7f9a8548f700 1 -- 172.21.15.60:0/2723482160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a8005d070 msgr2=0x7f9a800b40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:25.482 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.477+0000 7f9a8548f700 1 --2- 172.21.15.60:0/2723482160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a8005d070 0x7f9a800b40c0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 rx=0x7f9a78000f40 tx=0x7f9a78007df0).stop 2022-06-29T16:08:25.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.477+0000 7f9a8548f700 1 -- 172.21.15.60:0/2723482160 shutdown_connections 2022-06-29T16:08:25.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.477+0000 7f9a8548f700 1 --2- 172.21.15.60:0/2723482160 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9a64053b00 0x7f9a64055fc0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:25.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.477+0000 7f9a8548f700 1 --2- 172.21.15.60:0/2723482160 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a800b2130 0x7f9a80064040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:25.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.477+0000 7f9a8548f700 1 --2- 172.21.15.60:0/2723482160 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f9a800aed00 0x7f9a80156210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:25.483 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.477+0000 7f9a8548f700 1 --2- 172.21.15.60:0/2723482160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a8005d070 0x7f9a800b40c0 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:25.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.477+0000 7f9a8548f700 1 -- 172.21.15.60:0/2723482160 >> 172.21.15.60:0/2723482160 conn(0x7f9a800b5890 msgr2=0x7f9a80145420 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:25.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.477+0000 7f9a8548f700 1 -- 172.21.15.60:0/2723482160 shutdown_connections 2022-06-29T16:08:25.484 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:25.477+0000 7f9a8548f700 1 -- 172.21.15.60:0/2723482160 wait complete. 2022-06-29T16:08:25.485 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 73 2022-06-29T16:08:25.694 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-06-29T16:08:17.075855+0000', 'modified': '2022-06-29T16:08:18.079541+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8790}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8790': {'gid': 8790, 'name': 'a', 'rank': 0, 'incarnation': 72, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.60:6833/3619856083', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6832', 'nonce': 3619856083}, {'type': 'v1', 'addr': '172.21.15.60:6833', 'nonce': 3619856083}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-06-29T16:08:25.695 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-06-29T16:08:25.696 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-06-29T16:08:25.813 INFO:teuthology.orchestra.run.smithi060.stdout:{"cluster_fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","whoami":0,"id":8790,"want_state":"up:active","state":"up:active","rank_uptime":8.7222478290000005,"mdsmap_epoch":73,"osdmap_epoch":147,"osdmap_epoch_barrier":145,"uptime":13.072116546} 2022-06-29T16:08:25.814 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", "id": 8790, "mdsmap_epoch": 73, "osdmap_epoch": 147, "osdmap_epoch_barrier": 145, "rank_uptime": 8.722247829, "state": "up:active", "uptime": 13.072116546, "want_state": "up:active", "whoami": 0 } 2022-06-29T16:08:25.815 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-06-29T16:08:25.815 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-06-29T16:08:25.816 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi156.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-06-29T16:08:25.817 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:25.817 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:08:25.827 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2022-06-29T16:08:25.888 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:08:25.888 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:08:25.899 INFO:teuthology.orchestra.run.smithi156.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:08:25.900 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:08:25.901 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:25.902 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2022-06-29T16:08:25.909 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-06-29T16:08:25.910 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:08:25.913 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2022-06-29T16:08:25.928 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:08:25.929 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:08:25.985 INFO:teuthology.orchestra.run.smithi156.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:08:25.986 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:08:25.987 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:25.988 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2022-06-29T16:08:25.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:2022-06-29T16:08:25.981+0000 7f4293edd080 -1 init, newargv = 0x55f728a392f0 newargc=9 2022-06-29T16:08:25.991 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[13162]: starting ceph client 2022-06-29T16:08:26.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[13162]: starting fuse 2022-06-29T16:08:27.096 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2022-06-29T16:08:27.115 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-06-29T16:08:27.116 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-06-29T16:08:27.174 INFO:teuthology.orchestra.run.smithi156.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-06-29T16:08:27.174 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-06-29T16:08:27.176 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:27.176 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2022-06-29T16:08:27.183 INFO:teuthology.orchestra.run.smithi156.stdout:52 2022-06-29T16:08:27.183 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-06-29T16:08:27.184 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:27.184 DEBUG:teuthology.orchestra.run.smithi156:> sudo python3 -c ' 2022-06-29T16:08:27.185 DEBUG:teuthology.orchestra.run.smithi156:> import glob 2022-06-29T16:08:27.185 DEBUG:teuthology.orchestra.run.smithi156:> import re 2022-06-29T16:08:27.185 DEBUG:teuthology.orchestra.run.smithi156:> import os 2022-06-29T16:08:27.186 DEBUG:teuthology.orchestra.run.smithi156:> import subprocess 2022-06-29T16:08:27.186 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:08:27.187 DEBUG:teuthology.orchestra.run.smithi156:> def find_socket(client_name): 2022-06-29T16:08:27.187 DEBUG:teuthology.orchestra.run.smithi156:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-06-29T16:08:27.187 DEBUG:teuthology.orchestra.run.smithi156:> files = glob.glob(asok_path) 2022-06-29T16:08:27.188 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:08:27.188 DEBUG:teuthology.orchestra.run.smithi156:> # Given a non-glob path, it better be there 2022-06-29T16:08:27.189 DEBUG:teuthology.orchestra.run.smithi156:> if "*" not in asok_path: 2022-06-29T16:08:27.189 DEBUG:teuthology.orchestra.run.smithi156:> assert(len(files) == 1) 2022-06-29T16:08:27.190 DEBUG:teuthology.orchestra.run.smithi156:> return files[0] 2022-06-29T16:08:27.190 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:08:27.190 DEBUG:teuthology.orchestra.run.smithi156:> for f in files: 2022-06-29T16:08:27.191 DEBUG:teuthology.orchestra.run.smithi156:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-06-29T16:08:27.192 DEBUG:teuthology.orchestra.run.smithi156:> if os.path.exists("/proc/{0}".format(pid)): 2022-06-29T16:08:27.192 DEBUG:teuthology.orchestra.run.smithi156:> return f 2022-06-29T16:08:27.193 DEBUG:teuthology.orchestra.run.smithi156:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-06-29T16:08:27.193 DEBUG:teuthology.orchestra.run.smithi156:> 2022-06-29T16:08:27.193 DEBUG:teuthology.orchestra.run.smithi156:> print(find_socket("client.1")) 2022-06-29T16:08:27.194 DEBUG:teuthology.orchestra.run.smithi156:> ' 2022-06-29T16:08:27.272 INFO:teuthology.orchestra.run.smithi156.stdout:/var/run/ceph/ceph-client.1.13162.asok 2022-06-29T16:08:27.277 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.13162.asok 2022-06-29T16:08:27.278 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:27.278 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.13162.asok status 2022-06-29T16:08:27.370 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2022-06-29T16:08:27.370 INFO:teuthology.orchestra.run.smithi156.stdout: "metadata": { 2022-06-29T16:08:27.370 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_sha1": "f25ffac72e5a9e63860ff7deb6767a11f0559a16", 2022-06-29T16:08:27.371 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_version": "ceph version 15.2.16-304-gf25ffac7 (f25ffac72e5a9e63860ff7deb6767a11f0559a16) octopus (stable)", 2022-06-29T16:08:27.371 INFO:teuthology.orchestra.run.smithi156.stdout: "entity_id": "1", 2022-06-29T16:08:27.372 INFO:teuthology.orchestra.run.smithi156.stdout: "hostname": "smithi156", 2022-06-29T16:08:27.372 INFO:teuthology.orchestra.run.smithi156.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-06-29T16:08:27.372 INFO:teuthology.orchestra.run.smithi156.stdout: "pid": "13162", 2022-06-29T16:08:27.373 INFO:teuthology.orchestra.run.smithi156.stdout: "root": "/" 2022-06-29T16:08:27.373 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:08:27.373 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_count": 0, 2022-06-29T16:08:27.374 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_pinned_count": 0, 2022-06-29T16:08:27.374 INFO:teuthology.orchestra.run.smithi156.stdout: "id": 9006, 2022-06-29T16:08:27.374 INFO:teuthology.orchestra.run.smithi156.stdout: "inst": { 2022-06-29T16:08:27.375 INFO:teuthology.orchestra.run.smithi156.stdout: "name": { 2022-06-29T16:08:27.375 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "client", 2022-06-29T16:08:27.376 INFO:teuthology.orchestra.run.smithi156.stdout: "num": 9006 2022-06-29T16:08:27.376 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:08:27.376 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2022-06-29T16:08:27.377 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2022-06-29T16:08:27.377 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "172.21.15.156:0", 2022-06-29T16:08:27.377 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 2618224861 2022-06-29T16:08:27.378 INFO:teuthology.orchestra.run.smithi156.stdout: } 2022-06-29T16:08:27.378 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:08:27.378 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2022-06-29T16:08:27.379 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2022-06-29T16:08:27.379 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "172.21.15.156:0", 2022-06-29T16:08:27.380 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 2618224861 2022-06-29T16:08:27.380 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2022-06-29T16:08:27.380 INFO:teuthology.orchestra.run.smithi156.stdout: "inst_str": "client.9006 172.21.15.156:0/2618224861", 2022-06-29T16:08:27.381 INFO:teuthology.orchestra.run.smithi156.stdout: "addr_str": "172.21.15.156:0/2618224861", 2022-06-29T16:08:27.381 INFO:teuthology.orchestra.run.smithi156.stdout: "inode_count": 1, 2022-06-29T16:08:27.381 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_epoch": 73, 2022-06-29T16:08:27.382 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch": 147, 2022-06-29T16:08:27.382 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch_barrier": 0, 2022-06-29T16:08:27.382 INFO:teuthology.orchestra.run.smithi156.stdout: "blacklisted": false, 2022-06-29T16:08:27.383 INFO:teuthology.orchestra.run.smithi156.stdout: "fs_name": "cephfs" 2022-06-29T16:08:27.383 INFO:teuthology.orchestra.run.smithi156.stdout:} 2022-06-29T16:08:27.385 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:27.386 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:08:27.386 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:08:27.394 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2022-06-29T16:08:27.395 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-06-29T16:08:27.395 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:27.396 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:08:27.456 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:27.457 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:08:27.457 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:08:27.464 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2022-06-29T16:08:27.465 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-06-29T16:08:27.465 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:27.466 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:08:27.466 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:08:27.474 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2022-06-29T16:08:27.475 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-06-29T16:08:27.476 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi156.front.sepia.ceph.com... 2022-06-29T16:08:27.476 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-06-29T16:08:27.477 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:08:27.534 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[13162]: fuse finished with error 0 and tester_r 0 2022-06-29T16:08:27.536 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:08:27.537 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:08:27.537 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:08:27.545 INFO:teuthology.orchestra.run.smithi156.stdout:ext2/ext3 2022-06-29T16:08:27.546 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-06-29T16:08:27.547 INFO:teuthology.orchestra.run:waiting for 900 2022-06-29T16:08:27.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-06-29T16:08:27.586 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-06-29T16:08:27.634 INFO:teuthology.orchestra.run:Running command with timeout 10800 2022-06-29T16:08:27.634 DEBUG:teuthology.orchestra.run.smithi121:> cd /home/ubuntu/cephtest/mnt.0 && sudo bash -c 'stat -f .' 2022-06-29T16:08:27.656 INFO:teuthology.orchestra.run.smithi121.stdout: File: "." 2022-06-29T16:08:27.656 INFO:teuthology.orchestra.run.smithi121.stdout: ID: 0 Namelen: 255 Type: fuseblk 2022-06-29T16:08:27.656 INFO:teuthology.orchestra.run.smithi121.stdout:Block size: 4194304 Fundamental block size: 4194304 2022-06-29T16:08:27.657 INFO:teuthology.orchestra.run.smithi121.stdout:Blocks: Total: 86148 Free: 86148 Available: 86148 2022-06-29T16:08:27.657 INFO:teuthology.orchestra.run.smithi121.stdout:Inodes: Total: 1 Free: 0 2022-06-29T16:08:27.658 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-06-29T16:08:27.772 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.769+0000 7f667e926700 1 Processor -- start 2022-06-29T16:08:27.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.769+0000 7f667e926700 1 -- start start 2022-06-29T16:08:27.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.769+0000 7f667e926700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f667805d070 0x7f667814aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:27.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.769+0000 7f667e926700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f667814af80 0x7f66780ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:27.775 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.769+0000 7f667e926700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f667805d450 con 0x7f66780b2130 2022-06-29T16:08:27.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.769+0000 7f667e926700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66780adf90 con 0x7f667814af80 2022-06-29T16:08:27.777 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.769+0000 7f667e926700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66780b25d0 con 0x7f667805d070 2022-06-29T16:08:27.777 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.769+0000 7f667d123700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f66780b2130 0x7f66780aca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34516/0 (socket says 172.21.15.60:34516) 2022-06-29T16:08:27.777 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.769+0000 7f667d123700 1 -- 172.21.15.60:0/4045478562 learned_addr learned my addr 172.21.15.60:0/4045478562 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:27.777 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.769+0000 7f667d924700 1 --2- 172.21.15.60:0/4045478562 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f667805d070 0x7f667814aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:27.778 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.769+0000 7f667e125700 1 --2- 172.21.15.60:0/4045478562 >> v2:172.21.15.92:3300/0 conn(0x7f667814af80 0x7f66780ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:27.778 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.769+0000 7f667c922700 1 -- 172.21.15.60:0/4045478562 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37778321 0 0) 0x7f667805d450 con 0x7f66780b2130 2022-06-29T16:08:27.778 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.769+0000 7f667c922700 1 -- 172.21.15.60:0/4045478562 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6664003260 con 0x7f66780b2130 2022-06-29T16:08:27.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.769+0000 7f667d924700 1 -- 172.21.15.60:0/4045478562 >> v2:172.21.15.92:3300/0 conn(0x7f667814af80 msgr2=0x7f66780ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:27.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.769+0000 7f667d924700 1 --2- 172.21.15.60:0/4045478562 >> v2:172.21.15.92:3300/0 conn(0x7f667814af80 0x7f66780ad130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:27.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.769+0000 7f667d924700 1 -- 172.21.15.60:0/4045478562 >> v1:172.21.15.60:6789/0 conn(0x7f66780b2130 legacy=0x7f66780aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:27.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.769+0000 7f667d924700 1 -- 172.21.15.60:0/4045478562 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f667814a350 con 0x7f667805d070 2022-06-29T16:08:27.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.769+0000 7f667d924700 1 --2- 172.21.15.60:0/4045478562 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f667805d070 0x7f667814aa90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f666c002f70 tx=0x7f666c006df0).ready entity=mon.2 client_cookie=1d722992ee469d59 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:27.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.769+0000 7f667c922700 1 -- 172.21.15.60:0/4045478562 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f666c00d4a0 con 0x7f667805d070 2022-06-29T16:08:27.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.769+0000 7f667c922700 1 -- 172.21.15.60:0/4045478562 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f666c000e90 con 0x7f667805d070 2022-06-29T16:08:27.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.769+0000 7f667c922700 1 -- 172.21.15.60:0/4045478562 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f666c00d7f0 con 0x7f667805d070 2022-06-29T16:08:27.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.769+0000 7f667e926700 1 -- 172.21.15.60:0/4045478562 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f667805d070 msgr2=0x7f667814aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:27.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.769+0000 7f667e926700 1 --2- 172.21.15.60:0/4045478562 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f667805d070 0x7f667814aa90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f666c002f70 tx=0x7f666c006df0).stop 2022-06-29T16:08:27.785 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.769+0000 7f667e926700 1 -- 172.21.15.60:0/4045478562 shutdown_connections 2022-06-29T16:08:27.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.769+0000 7f667e926700 1 --2- 172.21.15.60:0/4045478562 >> v2:172.21.15.92:3300/0 conn(0x7f667814af80 0x7f66780ad130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:27.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.769+0000 7f667e926700 1 --2- 172.21.15.60:0/4045478562 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f667805d070 0x7f667814aa90 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:27.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.773+0000 7f667e926700 1 -- 172.21.15.60:0/4045478562 >> 172.21.15.60:0/4045478562 conn(0x7f66780b5890 msgr2=0x7f66780b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:27.786 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.773+0000 7f667e926700 1 -- 172.21.15.60:0/4045478562 shutdown_connections 2022-06-29T16:08:27.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.773+0000 7f667e926700 1 -- 172.21.15.60:0/4045478562 wait complete. 2022-06-29T16:08:27.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.773+0000 7f667e926700 1 Processor -- start 2022-06-29T16:08:27.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.773+0000 7f667e926700 1 -- start start 2022-06-29T16:08:27.787 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.777+0000 7f667e926700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f667805d070 0x7f66780b3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:27.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.777+0000 7f667e926700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f66780b2130 0x7f6678156080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:27.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.777+0000 7f667e926700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f667814af80 0x7f6678159c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:27.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.777+0000 7f667e926700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6678065780 con 0x7f667805d070 2022-06-29T16:08:27.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.777+0000 7f667e926700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667814d1b0 con 0x7f66780b2130 2022-06-29T16:08:27.788 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.777+0000 7f667e926700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6678156570 con 0x7f667814af80 2022-06-29T16:08:27.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.777+0000 7f667d924700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f667805d070 0x7f66780b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:27.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.777+0000 7f667d924700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f667805d070 0x7f66780b3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54666/0 (socket says 172.21.15.60:54666) 2022-06-29T16:08:27.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.777+0000 7f667d924700 1 -- 172.21.15.60:0/4260333652 learned_addr learned my addr 172.21.15.60:0/4260333652 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:27.789 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.777+0000 7f667e125700 1 --2- 172.21.15.60:0/4260333652 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f667814af80 0x7f6678159c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:27.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.777+0000 7f667d123700 1 --2- 172.21.15.60:0/4260333652 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f66780b2130 0x7f6678156080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:27.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.777+0000 7f667d924700 1 -- 172.21.15.60:0/4260333652 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f667814af80 msgr2=0x7f6678159c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:27.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.777+0000 7f667d924700 1 --2- 172.21.15.60:0/4260333652 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f667814af80 0x7f6678159c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:27.790 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.777+0000 7f667d924700 1 -- 172.21.15.60:0/4260333652 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f66780b2130 msgr2=0x7f6678156080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:27.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.777+0000 7f667d924700 1 --2- 172.21.15.60:0/4260333652 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f66780b2130 0x7f6678156080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:27.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.777+0000 7f667d924700 1 -- 172.21.15.60:0/4260333652 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f666c00b180 con 0x7f667805d070 2022-06-29T16:08:27.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.777+0000 7f667d123700 1 --2- 172.21.15.60:0/4260333652 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f66780b2130 0x7f6678156080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:08:27.791 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.777+0000 7f667d924700 1 --2- 172.21.15.60:0/4260333652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f667805d070 0x7f66780b3320 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 rx=0x7f667814ce70 tx=0x7f666c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:27.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.777+0000 7f6676ffd700 1 -- 172.21.15.60:0/4260333652 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f666c013a40 con 0x7f667805d070 2022-06-29T16:08:27.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.777+0000 7f6676ffd700 1 -- 172.21.15.60:0/4260333652 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f666c013ba0 con 0x7f667805d070 2022-06-29T16:08:27.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.777+0000 7f667e926700 1 -- 172.21.15.60:0/4260333652 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66780b3f20 con 0x7f667805d070 2022-06-29T16:08:27.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.777+0000 7f6676ffd700 1 -- 172.21.15.60:0/4260333652 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f666c01e400 con 0x7f667805d070 2022-06-29T16:08:27.792 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.777+0000 7f667e926700 1 -- 172.21.15.60:0/4260333652 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6678159760 con 0x7f667805d070 2022-06-29T16:08:27.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.781+0000 7f667e926700 1 -- 172.21.15.60:0/4260333652 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f667814a5f0 con 0x7f667805d070 2022-06-29T16:08:27.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.781+0000 7f6676ffd700 1 -- 172.21.15.60:0/4260333652 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f666c025030 con 0x7f667805d070 2022-06-29T16:08:27.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.781+0000 7f6676ffd700 1 --2- 172.21.15.60:0/4260333652 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6658053a70 0x7f6658055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:27.793 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.781+0000 7f6676ffd700 1 -- 172.21.15.60:0/4260333652 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f666c06ccc0 con 0x7f667805d070 2022-06-29T16:08:27.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.781+0000 7f667d123700 1 --2- 172.21.15.60:0/4260333652 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6658053a70 0x7f6658055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:27.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.781+0000 7f667d123700 1 --2- 172.21.15.60:0/4260333652 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6658053a70 0x7f6658055f30 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0x7f666800b000 tx=0x7f6668009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:27.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.785+0000 7f6676ffd700 1 -- 172.21.15.60:0/4260333652 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f666c043b40 con 0x7f667805d070 2022-06-29T16:08:27.916 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.913+0000 7f667e926700 1 -- 172.21.15.60:0/4260333652 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f66780573d0 con 0x7f667805d070 2022-06-29T16:08:27.919 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.913+0000 7f6676ffd700 1 -- 172.21.15.60:0/4260333652 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v147) v1 ==== 81+0+33659 (secure 0 0 0) 0x7f666c023070 con 0x7f667805d070 2022-06-29T16:08:27.920 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:08:27.920 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T16:08:27.920 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 147, 2022-06-29T16:08:27.921 INFO:teuthology.orchestra.run.smithi060.stdout: "fsid": "e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85", 2022-06-29T16:08:27.921 INFO:teuthology.orchestra.run.smithi060.stdout: "created": "2022-06-29T15:58:07.965719+0000", 2022-06-29T16:08:27.921 INFO:teuthology.orchestra.run.smithi060.stdout: "modified": "2022-06-29T16:08:19.091282+0000", 2022-06-29T16:08:27.922 INFO:teuthology.orchestra.run.smithi060.stdout: "last_up_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:08:27.922 INFO:teuthology.orchestra.run.smithi060.stdout: "last_in_change": "2022-06-29T15:58:13.784088+0000", 2022-06-29T16:08:27.922 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-06-29T16:08:27.922 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_num": 5799936, 2022-06-29T16:08:27.923 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_set": [ 2022-06-29T16:08:27.923 INFO:teuthology.orchestra.run.smithi060.stdout: "pglog_hardlimit", 2022-06-29T16:08:27.923 INFO:teuthology.orchestra.run.smithi060.stdout: "purged_snapdirs", 2022-06-29T16:08:27.923 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_deletes", 2022-06-29T16:08:27.924 INFO:teuthology.orchestra.run.smithi060.stdout: "sortbitwise" 2022-06-29T16:08:27.924 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:08:27.924 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_version": 4, 2022-06-29T16:08:27.924 INFO:teuthology.orchestra.run.smithi060.stdout: "full_ratio": 0.94999998807907104, 2022-06-29T16:08:27.925 INFO:teuthology.orchestra.run.smithi060.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-06-29T16:08:27.925 INFO:teuthology.orchestra.run.smithi060.stdout: "nearfull_ratio": 0.85000002384185791, 2022-06-29T16:08:27.925 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_snapshot": "", 2022-06-29T16:08:27.925 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_max": 30, 2022-06-29T16:08:27.926 INFO:teuthology.orchestra.run.smithi060.stdout: "max_osd": 8, 2022-06-29T16:08:27.926 INFO:teuthology.orchestra.run.smithi060.stdout: "require_min_compat_client": "jewel", 2022-06-29T16:08:27.926 INFO:teuthology.orchestra.run.smithi060.stdout: "min_compat_client": "jewel", 2022-06-29T16:08:27.926 INFO:teuthology.orchestra.run.smithi060.stdout: "require_osd_release": "octopus", 2022-06-29T16:08:27.926 INFO:teuthology.orchestra.run.smithi060.stdout: "pools": [ 2022-06-29T16:08:27.927 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:27.927 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2022-06-29T16:08:27.927 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "device_health_metrics", 2022-06-29T16:08:27.927 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:10.961127+0000", 2022-06-29T16:08:27.928 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:08:27.928 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:08:27.928 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:08:27.928 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:08:27.929 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:08:27.929 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:08:27.929 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:08:27.929 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:08:27.929 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 1, 2022-06-29T16:08:27.930 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 1, 2022-06-29T16:08:27.930 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 1, 2022-06-29T16:08:27.930 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 1, 2022-06-29T16:08:27.930 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 1, 2022-06-29T16:08:27.930 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:08:27.931 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:08:27.931 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:08:27.931 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:08:27.931 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:08:27.932 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:08:27.932 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:08:27.932 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:27.932 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "15", 2022-06-29T16:08:27.932 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:08:27.933 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:08:27.933 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:08:27.933 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:08:27.933 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:08:27.934 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:08:27.934 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:08:27.934 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:08:27.934 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:08:27.934 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:08:27.935 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:08:27.935 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:08:27.935 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:08:27.935 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:08:27.936 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:08:27.936 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:08:27.936 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:08:27.936 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:08:27.937 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:08:27.937 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:08:27.937 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:08:27.937 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:08:27.937 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:08:27.938 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:08:27.938 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:08:27.938 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:08:27.938 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:27.939 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:08:27.939 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:08:27.939 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:08:27.939 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:08:27.939 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:08:27.940 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:08:27.940 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:08:27.940 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:08:27.940 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:08:27.941 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:08:27.941 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:08:27.941 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:08:27.941 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 1 2022-06-29T16:08:27.941 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:27.942 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:08:27.942 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_devicehealth": {} 2022-06-29T16:08:27.942 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:27.942 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:27.942 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:27.943 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 2, 2022-06-29T16:08:27.943 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "rbd", 2022-06-29T16:08:27.943 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T15:58:19.184276+0000", 2022-06-29T16:08:27.943 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:08:27.944 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:08:27.944 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:08:27.944 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:08:27.944 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:08:27.944 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:08:27.945 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:08:27.945 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:08:27.945 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:08:27.945 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:08:27.945 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:08:27.946 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:08:27.946 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:08:27.946 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:08:27.946 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:08:27.947 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:08:27.947 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:08:27.947 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:08:27.947 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:08:27.947 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:08:27.948 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:27.948 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "18", 2022-06-29T16:08:27.948 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:08:27.948 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:08:27.949 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:08:27.949 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:08:27.949 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:08:27.949 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:08:27.949 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:08:27.950 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:08:27.950 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:08:27.950 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:08:27.950 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:08:27.951 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:08:27.951 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:08:27.951 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:08:27.951 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:08:27.951 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:08:27.952 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:08:27.952 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:08:27.952 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:08:27.952 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:08:27.952 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:08:27.953 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:08:27.953 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:08:27.953 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:08:27.953 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:08:27.954 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:08:27.954 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:27.954 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:08:27.954 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:08:27.954 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:08:27.955 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:08:27.955 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:08:27.955 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:08:27.955 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:08:27.955 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:08:27.956 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:08:27.956 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:08:27.956 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:08:27.956 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:08:27.957 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:08:27.957 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd": {} 2022-06-29T16:08:27.957 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:27.957 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:27.957 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:27.958 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 29, 2022-06-29T16:08:27.958 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "cephfs_metadata", 2022-06-29T16:08:27.958 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T16:08:15.820434+0000", 2022-06-29T16:08:27.958 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:08:27.959 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:08:27.959 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:08:27.959 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:08:27.959 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:08:27.959 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:08:27.960 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:08:27.960 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:08:27.960 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:08:27.960 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:08:27.961 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:08:27.961 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:08:27.961 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:08:27.961 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:08:27.961 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:08:27.962 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:08:27.962 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:08:27.962 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:08:27.962 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:08:27.963 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:08:27.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:27.963 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "146", 2022-06-29T16:08:27.963 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:08:27.963 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:08:27.964 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:08:27.964 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:08:27.964 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:08:27.964 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:08:27.964 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:08:27.965 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:08:27.965 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:08:27.965 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:08:27.965 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:08:27.966 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:08:27.966 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:08:27.966 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:08:27.966 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:08:27.966 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:08:27.967 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:08:27.967 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:08:27.967 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:08:27.967 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:08:27.967 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:08:27.968 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:08:27.968 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:08:27.968 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:08:27.968 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:08:27.969 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:08:27.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:27.969 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:08:27.969 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:08:27.969 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:08:27.970 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:08:27.970 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:08:27.970 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:08:27.970 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:08:27.971 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:08:27.971 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:08:27.971 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:08:27.971 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:08:27.971 INFO:teuthology.orchestra.run.smithi060.stdout: "options": { 2022-06-29T16:08:27.972 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_bias": 4, 2022-06-29T16:08:27.972 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_min": 16, 2022-06-29T16:08:27.972 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_priority": 5 2022-06-29T16:08:27.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:27.973 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:08:27.973 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs": { 2022-06-29T16:08:27.973 INFO:teuthology.orchestra.run.smithi060.stdout: "metadata": "cephfs" 2022-06-29T16:08:27.973 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:27.973 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:27.974 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:27.974 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:27.974 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 30, 2022-06-29T16:08:27.974 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": "cephfs_data", 2022-06-29T16:08:27.975 INFO:teuthology.orchestra.run.smithi060.stdout: "create_time": "2022-06-29T16:08:16.484716+0000", 2022-06-29T16:08:27.975 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2022-06-29T16:08:27.975 INFO:teuthology.orchestra.run.smithi060.stdout: "flags_names": "hashpspool", 2022-06-29T16:08:27.975 INFO:teuthology.orchestra.run.smithi060.stdout: "type": 1, 2022-06-29T16:08:27.976 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 2, 2022-06-29T16:08:27.976 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": 1, 2022-06-29T16:08:27.976 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_rule": 0, 2022-06-29T16:08:27.976 INFO:teuthology.orchestra.run.smithi060.stdout: "object_hash": 2, 2022-06-29T16:08:27.976 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscale_mode": "off", 2022-06-29T16:08:27.977 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": 8, 2022-06-29T16:08:27.977 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num": 8, 2022-06-29T16:08:27.977 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_placement_num_target": 8, 2022-06-29T16:08:27.977 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_target": 8, 2022-06-29T16:08:27.977 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num_pending": 8, 2022-06-29T16:08:27.978 INFO:teuthology.orchestra.run.smithi060.stdout: "last_pg_merge_meta": { 2022-06-29T16:08:27.978 INFO:teuthology.orchestra.run.smithi060.stdout: "source_pgid": "0.0", 2022-06-29T16:08:27.978 INFO:teuthology.orchestra.run.smithi060.stdout: "ready_epoch": 0, 2022-06-29T16:08:27.978 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_started": 0, 2022-06-29T16:08:27.979 INFO:teuthology.orchestra.run.smithi060.stdout: "last_epoch_clean": 0, 2022-06-29T16:08:27.979 INFO:teuthology.orchestra.run.smithi060.stdout: "source_version": "0'0", 2022-06-29T16:08:27.979 INFO:teuthology.orchestra.run.smithi060.stdout: "target_version": "0'0" 2022-06-29T16:08:27.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:27.980 INFO:teuthology.orchestra.run.smithi060.stdout: "last_change": "147", 2022-06-29T16:08:27.980 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend": "0", 2022-06-29T16:08:27.980 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2022-06-29T16:08:27.980 INFO:teuthology.orchestra.run.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2022-06-29T16:08:27.980 INFO:teuthology.orchestra.run.smithi060.stdout: "auid": 0, 2022-06-29T16:08:27.981 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_mode": "selfmanaged", 2022-06-29T16:08:27.981 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_seq": 0, 2022-06-29T16:08:27.981 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_epoch": 0, 2022-06-29T16:08:27.981 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_snaps": [], 2022-06-29T16:08:27.981 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps": "[]", 2022-06-29T16:08:27.982 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_bytes": 0, 2022-06-29T16:08:27.982 INFO:teuthology.orchestra.run.smithi060.stdout: "quota_max_objects": 0, 2022-06-29T16:08:27.982 INFO:teuthology.orchestra.run.smithi060.stdout: "tiers": [], 2022-06-29T16:08:27.982 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_of": -1, 2022-06-29T16:08:27.983 INFO:teuthology.orchestra.run.smithi060.stdout: "read_tier": -1, 2022-06-29T16:08:27.983 INFO:teuthology.orchestra.run.smithi060.stdout: "write_tier": -1, 2022-06-29T16:08:27.983 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_mode": "none", 2022-06-29T16:08:27.983 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_bytes": 0, 2022-06-29T16:08:27.983 INFO:teuthology.orchestra.run.smithi060.stdout: "target_max_objects": 0, 2022-06-29T16:08:27.984 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-06-29T16:08:27.984 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-06-29T16:08:27.984 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2022-06-29T16:08:27.984 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_flush_age": 0, 2022-06-29T16:08:27.985 INFO:teuthology.orchestra.run.smithi060.stdout: "cache_min_evict_age": 0, 2022-06-29T16:08:27.985 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profile": "", 2022-06-29T16:08:27.985 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_params": { 2022-06-29T16:08:27.985 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "none" 2022-06-29T16:08:27.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:27.986 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_period": 0, 2022-06-29T16:08:27.986 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_count": 0, 2022-06-29T16:08:27.986 INFO:teuthology.orchestra.run.smithi060.stdout: "use_gmt_hitset": true, 2022-06-29T16:08:27.986 INFO:teuthology.orchestra.run.smithi060.stdout: "min_read_recency_for_promote": 0, 2022-06-29T16:08:27.987 INFO:teuthology.orchestra.run.smithi060.stdout: "min_write_recency_for_promote": 0, 2022-06-29T16:08:27.987 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2022-06-29T16:08:27.987 INFO:teuthology.orchestra.run.smithi060.stdout: "hit_set_search_last_n": 0, 2022-06-29T16:08:27.987 INFO:teuthology.orchestra.run.smithi060.stdout: "grade_table": [], 2022-06-29T16:08:27.987 INFO:teuthology.orchestra.run.smithi060.stdout: "stripe_width": 0, 2022-06-29T16:08:27.988 INFO:teuthology.orchestra.run.smithi060.stdout: "expected_num_objects": 0, 2022-06-29T16:08:27.988 INFO:teuthology.orchestra.run.smithi060.stdout: "fast_read": false, 2022-06-29T16:08:27.988 INFO:teuthology.orchestra.run.smithi060.stdout: "options": {}, 2022-06-29T16:08:27.988 INFO:teuthology.orchestra.run.smithi060.stdout: "application_metadata": { 2022-06-29T16:08:27.989 INFO:teuthology.orchestra.run.smithi060.stdout: "cephfs": { 2022-06-29T16:08:27.989 INFO:teuthology.orchestra.run.smithi060.stdout: "data": "cephfs" 2022-06-29T16:08:27.989 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:27.989 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:27.989 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:27.990 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:08:27.990 INFO:teuthology.orchestra.run.smithi060.stdout: "osds": [ 2022-06-29T16:08:27.990 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:27.990 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:08:27.991 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "11e7ab9e-fce2-470f-8195-ea4c3c52df22", 2022-06-29T16:08:27.991 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:08:27.991 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:08:27.991 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:08:27.991 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:08:27.992 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:08:27.992 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:08:27.992 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:08:27.992 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 143, 2022-06-29T16:08:27.992 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:08:27.993 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:08:27.993 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:08:27.993 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:27.993 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:27.994 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:27.994 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6816", 2022-06-29T16:08:27.994 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:08:27.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:27.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:27.995 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:27.995 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6825", 2022-06-29T16:08:27.995 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:08:27.995 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:27.996 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:27.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:27.996 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:08:27.996 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:27.996 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:27.997 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:27.997 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6826", 2022-06-29T16:08:27.997 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:08:27.997 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:27.997 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:27.998 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:27.998 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6827", 2022-06-29T16:08:27.998 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:08:27.998 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:27.999 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:27.999 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:27.999 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:08:27.999 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:27.999 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.000 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:28.000 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6830", 2022-06-29T16:08:28.000 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:08:28.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.001 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:28.001 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6831", 2022-06-29T16:08:28.001 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:08:28.002 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:28.002 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.003 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:08:28.003 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:28.003 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.003 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:28.004 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6828", 2022-06-29T16:08:28.004 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:08:28.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.005 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:28.005 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6829", 2022-06-29T16:08:28.005 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12890 2022-06-29T16:08:28.005 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:28.006 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.006 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.006 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6825/12890", 2022-06-29T16:08:28.006 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6827/12890", 2022-06-29T16:08:28.007 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6831/12890", 2022-06-29T16:08:28.007 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6829/12890", 2022-06-29T16:08:28.007 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:08:28.007 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:08:28.008 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:08:28.008 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.008 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.008 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.009 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:08:28.009 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "ca43f136-6f69-4431-a157-fbe073fb585c", 2022-06-29T16:08:28.009 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:08:28.009 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:08:28.009 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:08:28.010 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:08:28.010 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:08:28.010 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:08:28.010 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:08:28.010 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 144, 2022-06-29T16:08:28.011 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:08:28.011 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:08:28.011 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:08:28.011 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:28.012 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.012 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:28.012 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6815", 2022-06-29T16:08:28.012 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:08:28.012 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.013 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.013 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:28.013 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6818", 2022-06-29T16:08:28.013 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:08:28.013 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:28.014 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.014 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.014 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:08:28.014 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:28.015 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.015 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:28.015 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6819", 2022-06-29T16:08:28.015 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:08:28.015 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.016 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.016 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:28.016 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6820", 2022-06-29T16:08:28.016 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:08:28.017 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:28.017 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.017 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.017 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:08:28.017 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:28.018 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.018 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:28.018 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6823", 2022-06-29T16:08:28.018 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:08:28.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.019 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.019 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:28.019 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6824", 2022-06-29T16:08:28.020 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:08:28.020 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:28.020 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.020 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:08:28.021 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:28.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.021 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:28.021 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6821", 2022-06-29T16:08:28.021 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:08:28.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.022 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.022 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:28.022 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6822", 2022-06-29T16:08:28.023 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12892 2022-06-29T16:08:28.023 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:28.023 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.023 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.023 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6818/12892", 2022-06-29T16:08:28.024 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6820/12892", 2022-06-29T16:08:28.024 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6824/12892", 2022-06-29T16:08:28.024 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6822/12892", 2022-06-29T16:08:28.024 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:08:28.024 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:08:28.025 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:08:28.025 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.025 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.025 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.026 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:08:28.026 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "74fac45f-5f07-44ec-a9cb-2644c123fd4a", 2022-06-29T16:08:28.026 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:08:28.026 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:08:28.026 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:08:28.027 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:08:28.027 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:08:28.027 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:08:28.027 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:08:28.027 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 143, 2022-06-29T16:08:28.028 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:08:28.028 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:08:28.028 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:08:28.028 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:28.029 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.029 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:28.029 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6808", 2022-06-29T16:08:28.029 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:08:28.029 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.030 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.030 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:28.030 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6809", 2022-06-29T16:08:28.030 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:08:28.031 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:28.031 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.031 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.031 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:08:28.031 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:28.032 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.032 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:28.032 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6810", 2022-06-29T16:08:28.032 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:08:28.033 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.033 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.033 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:28.033 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6811", 2022-06-29T16:08:28.033 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:08:28.034 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:28.034 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.034 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.034 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:08:28.035 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:28.035 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.035 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:28.035 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6814", 2022-06-29T16:08:28.036 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:08:28.036 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.036 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.036 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:28.036 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6817", 2022-06-29T16:08:28.037 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:08:28.037 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:28.037 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.037 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.038 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:08:28.038 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:28.038 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.039 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:28.039 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6812", 2022-06-29T16:08:28.039 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:08:28.040 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.040 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.040 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:28.040 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6813", 2022-06-29T16:08:28.041 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12891 2022-06-29T16:08:28.041 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:28.041 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.042 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.042 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6809/12891", 2022-06-29T16:08:28.042 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6811/12891", 2022-06-29T16:08:28.042 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6817/12891", 2022-06-29T16:08:28.043 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6813/12891", 2022-06-29T16:08:28.043 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:08:28.043 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:08:28.043 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:08:28.044 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.044 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.044 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.044 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:08:28.045 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "2581d351-973a-4b4e-85ca-3658f0068765", 2022-06-29T16:08:28.045 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:08:28.045 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:08:28.045 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:08:28.045 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:08:28.046 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:08:28.046 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:08:28.046 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:08:28.047 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 145, 2022-06-29T16:08:28.047 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:08:28.047 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:08:28.047 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:08:28.047 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:28.048 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.048 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:28.048 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6800", 2022-06-29T16:08:28.048 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:08:28.048 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.049 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.049 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:28.049 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6801", 2022-06-29T16:08:28.049 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:08:28.050 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:28.050 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.050 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.050 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:08:28.051 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:28.051 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.051 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:28.051 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6802", 2022-06-29T16:08:28.051 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:08:28.052 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.052 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.052 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:28.052 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6803", 2022-06-29T16:08:28.052 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:08:28.053 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:28.053 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.053 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.053 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:08:28.054 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:28.054 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.054 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:28.054 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6806", 2022-06-29T16:08:28.054 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:08:28.055 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.055 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.055 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:28.055 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6807", 2022-06-29T16:08:28.055 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:08:28.056 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:28.056 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.056 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.056 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:08:28.057 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:28.057 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.057 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:28.057 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6804", 2022-06-29T16:08:28.057 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:08:28.058 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.058 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.058 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:28.058 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6805", 2022-06-29T16:08:28.058 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 12889 2022-06-29T16:08:28.059 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:28.059 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.059 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.059 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.60:6801/12889", 2022-06-29T16:08:28.060 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.60:6803/12889", 2022-06-29T16:08:28.060 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.60:6807/12889", 2022-06-29T16:08:28.060 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.60:6805/12889", 2022-06-29T16:08:28.060 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:08:28.060 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:08:28.061 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:08:28.061 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.061 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.061 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.062 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:08:28.062 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "3d234db9-87cd-4fe9-814f-0240a287d8b4", 2022-06-29T16:08:28.062 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:08:28.062 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:08:28.062 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:08:28.063 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:08:28.063 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:08:28.063 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:08:28.063 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:08:28.064 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 142, 2022-06-29T16:08:28.064 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:08:28.064 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:08:28.064 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:08:28.065 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:28.065 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.065 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:28.065 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6808", 2022-06-29T16:08:28.065 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:08:28.066 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.066 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.066 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:28.066 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6810", 2022-06-29T16:08:28.067 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:08:28.067 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:28.067 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.067 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.067 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:08:28.068 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:28.068 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.068 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:28.068 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6812", 2022-06-29T16:08:28.068 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:08:28.069 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.069 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.069 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:28.069 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6814", 2022-06-29T16:08:28.070 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:08:28.070 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:28.070 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.070 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.071 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:08:28.071 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:28.071 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.071 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:28.072 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6823", 2022-06-29T16:08:28.072 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:08:28.072 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.072 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.073 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:28.073 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6825", 2022-06-29T16:08:28.074 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:08:28.074 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:28.074 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.074 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.075 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:08:28.075 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:28.075 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.075 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:28.075 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6817", 2022-06-29T16:08:28.076 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:08:28.076 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.076 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.076 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:28.077 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6820", 2022-06-29T16:08:28.077 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13249 2022-06-29T16:08:28.077 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:28.077 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.078 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.078 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6810/13249", 2022-06-29T16:08:28.078 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6814/13249", 2022-06-29T16:08:28.079 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6825/13249", 2022-06-29T16:08:28.079 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6820/13249", 2022-06-29T16:08:28.079 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:08:28.079 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:08:28.080 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:08:28.080 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.080 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.080 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.080 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:08:28.081 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "e4bf1104-ff5c-438b-b608-8cee31c163f6", 2022-06-29T16:08:28.081 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:08:28.081 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:08:28.081 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:08:28.081 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:08:28.082 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:08:28.082 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:08:28.082 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:08:28.082 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 144, 2022-06-29T16:08:28.083 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:08:28.083 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:08:28.083 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:08:28.083 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:28.084 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.084 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:28.084 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6809", 2022-06-29T16:08:28.085 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:08:28.085 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.085 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.085 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:28.086 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6811", 2022-06-29T16:08:28.086 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:08:28.086 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:28.086 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.087 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.087 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:08:28.087 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:28.087 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.087 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:28.088 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6813", 2022-06-29T16:08:28.088 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:08:28.088 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.088 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.089 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:28.089 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6815", 2022-06-29T16:08:28.089 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:08:28.089 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:28.089 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.090 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.090 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:08:28.090 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:28.090 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.090 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:28.091 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6824", 2022-06-29T16:08:28.091 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:08:28.091 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.091 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.092 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:28.092 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6827", 2022-06-29T16:08:28.092 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:08:28.092 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:28.092 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.093 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.093 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:08:28.093 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:28.093 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.094 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:28.094 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6818", 2022-06-29T16:08:28.094 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:08:28.094 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.094 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.095 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:28.095 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6821", 2022-06-29T16:08:28.095 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13248 2022-06-29T16:08:28.095 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:28.095 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.096 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.096 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6811/13248", 2022-06-29T16:08:28.096 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6815/13248", 2022-06-29T16:08:28.096 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6827/13248", 2022-06-29T16:08:28.097 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6821/13248", 2022-06-29T16:08:28.097 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:08:28.097 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:08:28.097 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:08:28.097 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.098 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.098 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.098 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:08:28.098 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "8564dd25-27ab-4203-b54e-af6d3aa05857", 2022-06-29T16:08:28.098 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:08:28.099 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:08:28.099 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:08:28.099 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:08:28.099 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:08:28.100 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:08:28.100 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:08:28.100 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 143, 2022-06-29T16:08:28.100 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:08:28.100 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:08:28.101 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:08:28.101 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:28.101 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.101 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:28.101 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6816", 2022-06-29T16:08:28.102 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:08:28.102 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.102 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.103 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:28.103 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6819", 2022-06-29T16:08:28.103 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:08:28.103 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:28.104 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.104 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.104 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:08:28.104 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:28.104 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.105 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:28.105 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6822", 2022-06-29T16:08:28.105 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:08:28.105 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.105 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.106 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:28.106 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6826", 2022-06-29T16:08:28.106 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:08:28.106 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:28.107 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.107 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.107 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:08:28.107 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:28.107 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.108 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:28.108 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6830", 2022-06-29T16:08:28.108 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:08:28.108 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.109 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.109 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:28.109 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6831", 2022-06-29T16:08:28.109 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:08:28.109 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:28.110 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.110 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.110 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:08:28.110 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:28.110 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.111 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:28.111 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6828", 2022-06-29T16:08:28.111 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:08:28.111 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.112 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.112 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:28.112 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6829", 2022-06-29T16:08:28.112 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13250 2022-06-29T16:08:28.112 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:28.113 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.113 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.113 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6819/13250", 2022-06-29T16:08:28.113 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6826/13250", 2022-06-29T16:08:28.113 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6831/13250", 2022-06-29T16:08:28.114 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6829/13250", 2022-06-29T16:08:28.114 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:08:28.114 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:08:28.114 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:08:28.115 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.115 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.115 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.115 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:08:28.115 INFO:teuthology.orchestra.run.smithi060.stdout: "uuid": "bd513a9c-3b19-4314-9897-d300b5c897bb", 2022-06-29T16:08:28.116 INFO:teuthology.orchestra.run.smithi060.stdout: "up": 1, 2022-06-29T16:08:28.116 INFO:teuthology.orchestra.run.smithi060.stdout: "in": 1, 2022-06-29T16:08:28.116 INFO:teuthology.orchestra.run.smithi060.stdout: "weight": 1, 2022-06-29T16:08:28.116 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_affinity": 1, 2022-06-29T16:08:28.116 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_begin": 0, 2022-06-29T16:08:28.117 INFO:teuthology.orchestra.run.smithi060.stdout: "last_clean_end": 0, 2022-06-29T16:08:28.117 INFO:teuthology.orchestra.run.smithi060.stdout: "up_from": 13, 2022-06-29T16:08:28.117 INFO:teuthology.orchestra.run.smithi060.stdout: "up_thru": 143, 2022-06-29T16:08:28.117 INFO:teuthology.orchestra.run.smithi060.stdout: "down_at": 0, 2022-06-29T16:08:28.118 INFO:teuthology.orchestra.run.smithi060.stdout: "lost_at": 0, 2022-06-29T16:08:28.118 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addrs": { 2022-06-29T16:08:28.118 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:28.118 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.118 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:28.119 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6800", 2022-06-29T16:08:28.119 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:08:28.119 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.119 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.119 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:28.120 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6801", 2022-06-29T16:08:28.120 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:08:28.120 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:28.120 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.121 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.121 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addrs": { 2022-06-29T16:08:28.121 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:28.121 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.121 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:28.122 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6802", 2022-06-29T16:08:28.122 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:08:28.122 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.122 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.122 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:28.123 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6803", 2022-06-29T16:08:28.123 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:08:28.123 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:28.123 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.124 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.124 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addrs": { 2022-06-29T16:08:28.124 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:28.124 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.124 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:28.125 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6806", 2022-06-29T16:08:28.125 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:08:28.125 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.125 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.125 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:28.126 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6807", 2022-06-29T16:08:28.126 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:08:28.126 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:28.126 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.127 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.127 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addrs": { 2022-06-29T16:08:28.127 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2022-06-29T16:08:28.127 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.127 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2022-06-29T16:08:28.128 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6804", 2022-06-29T16:08:28.128 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:08:28.128 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.128 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.129 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2022-06-29T16:08:28.129 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.92:6805", 2022-06-29T16:08:28.129 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 13247 2022-06-29T16:08:28.129 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:28.129 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.130 INFO:teuthology.orchestra.run.smithi060.stdout: "public_addr": "172.21.15.92:6801/13247", 2022-06-29T16:08:28.130 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_addr": "172.21.15.92:6803/13247", 2022-06-29T16:08:28.130 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_back_addr": "172.21.15.92:6807/13247", 2022-06-29T16:08:28.130 INFO:teuthology.orchestra.run.smithi060.stdout: "heartbeat_front_addr": "172.21.15.92:6805/13247", 2022-06-29T16:08:28.131 INFO:teuthology.orchestra.run.smithi060.stdout: "state": [ 2022-06-29T16:08:28.131 INFO:teuthology.orchestra.run.smithi060.stdout: "exists", 2022-06-29T16:08:28.131 INFO:teuthology.orchestra.run.smithi060.stdout: "up" 2022-06-29T16:08:28.131 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2022-06-29T16:08:28.132 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:28.132 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:08:28.132 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_xinfo": [ 2022-06-29T16:08:28.132 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.132 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 0, 2022-06-29T16:08:28.133 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:08:28.133 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:08:28.133 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:08:28.133 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:08:28.134 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:08:28.134 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.891342+0000", 2022-06-29T16:08:28.134 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:08:28.134 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.134 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.135 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 1, 2022-06-29T16:08:28.135 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:08:28.135 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:08:28.135 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:08:28.136 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:08:28.136 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:08:28.136 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.900866+0000", 2022-06-29T16:08:28.136 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:08:28.136 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.137 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.137 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 2, 2022-06-29T16:08:28.137 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:08:28.137 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:08:28.137 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:08:28.138 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:08:28.138 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:08:28.138 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.912755+0000", 2022-06-29T16:08:28.138 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:08:28.139 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.139 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.139 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 3, 2022-06-29T16:08:28.139 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:08:28.140 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:08:28.140 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:08:28.140 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:08:28.140 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:08:28.140 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.864332+0000", 2022-06-29T16:08:28.141 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:08:28.141 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.141 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.141 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 4, 2022-06-29T16:08:28.142 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:08:28.142 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:08:28.142 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:08:28.142 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:08:28.143 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:08:28.143 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.863756+0000", 2022-06-29T16:08:28.143 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:08:28.143 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.143 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.144 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 5, 2022-06-29T16:08:28.144 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:08:28.144 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:08:28.144 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:08:28.145 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:08:28.145 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:08:28.145 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.869041+0000", 2022-06-29T16:08:28.145 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:08:28.145 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.146 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.146 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 6, 2022-06-29T16:08:28.146 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:08:28.146 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:08:28.146 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:08:28.147 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:08:28.147 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:08:28.147 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.895043+0000", 2022-06-29T16:08:28.147 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:08:28.148 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.148 INFO:teuthology.orchestra.run.smithi060.stdout: { 2022-06-29T16:08:28.148 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": 7, 2022-06-29T16:08:28.148 INFO:teuthology.orchestra.run.smithi060.stdout: "down_stamp": "0.000000", 2022-06-29T16:08:28.148 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_probability": 0, 2022-06-29T16:08:28.149 INFO:teuthology.orchestra.run.smithi060.stdout: "laggy_interval": 0, 2022-06-29T16:08:28.149 INFO:teuthology.orchestra.run.smithi060.stdout: "features": 4540138292840890367, 2022-06-29T16:08:28.149 INFO:teuthology.orchestra.run.smithi060.stdout: "old_weight": 0, 2022-06-29T16:08:28.149 INFO:teuthology.orchestra.run.smithi060.stdout: "last_purged_snaps_scrub": "2022-06-29T15:58:11.889491+0000", 2022-06-29T16:08:28.149 INFO:teuthology.orchestra.run.smithi060.stdout: "dead_epoch": 0 2022-06-29T16:08:28.150 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:28.150 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2022-06-29T16:08:28.150 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap": [], 2022-06-29T16:08:28.150 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_upmap_items": [], 2022-06-29T16:08:28.151 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_temp": [], 2022-06-29T16:08:28.151 INFO:teuthology.orchestra.run.smithi060.stdout: "primary_temp": [], 2022-06-29T16:08:28.151 INFO:teuthology.orchestra.run.smithi060.stdout: "blacklist": {}, 2022-06-29T16:08:28.151 INFO:teuthology.orchestra.run.smithi060.stdout: "erasure_code_profiles": { 2022-06-29T16:08:28.151 INFO:teuthology.orchestra.run.smithi060.stdout: "default": { 2022-06-29T16:08:28.152 INFO:teuthology.orchestra.run.smithi060.stdout: "crush-failure-domain": "osd", 2022-06-29T16:08:28.152 INFO:teuthology.orchestra.run.smithi060.stdout: "k": "2", 2022-06-29T16:08:28.152 INFO:teuthology.orchestra.run.smithi060.stdout: "m": "1", 2022-06-29T16:08:28.152 INFO:teuthology.orchestra.run.smithi060.stdout: "plugin": "jerasure", 2022-06-29T16:08:28.153 INFO:teuthology.orchestra.run.smithi060.stdout: "ruleset-failure-domain": "osd", 2022-06-29T16:08:28.153 INFO:teuthology.orchestra.run.smithi060.stdout: "technique": "reed_sol_van" 2022-06-29T16:08:28.153 INFO:teuthology.orchestra.run.smithi060.stdout: } 2022-06-29T16:08:28.153 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2022-06-29T16:08:28.153 INFO:teuthology.orchestra.run.smithi060.stdout: "removed_snaps_queue": [], 2022-06-29T16:08:28.154 INFO:teuthology.orchestra.run.smithi060.stdout: "new_removed_snaps": [], 2022-06-29T16:08:28.154 INFO:teuthology.orchestra.run.smithi060.stdout: "new_purged_snaps": [], 2022-06-29T16:08:28.154 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_node_flags": {}, 2022-06-29T16:08:28.154 INFO:teuthology.orchestra.run.smithi060.stdout: "device_class_flags": {} 2022-06-29T16:08:28.154 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T16:08:28.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.917+0000 7f667e926700 1 -- 172.21.15.60:0/4260333652 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6658053a70 msgr2=0x7f6658055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:28.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.917+0000 7f667e926700 1 --2- 172.21.15.60:0/4260333652 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6658053a70 0x7f6658055f30 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0x7f666800b000 tx=0x7f6668009040).stop 2022-06-29T16:08:28.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.917+0000 7f667e926700 1 -- 172.21.15.60:0/4260333652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f667805d070 msgr2=0x7f66780b3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:28.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.917+0000 7f667e926700 1 --2- 172.21.15.60:0/4260333652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f667805d070 0x7f66780b3320 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 rx=0x7f667814ce70 tx=0x7f666c014000).stop 2022-06-29T16:08:28.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.917+0000 7f667e926700 1 -- 172.21.15.60:0/4260333652 shutdown_connections 2022-06-29T16:08:28.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.917+0000 7f667e926700 1 --2- 172.21.15.60:0/4260333652 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f6658053a70 0x7f6658055f30 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:28.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.917+0000 7f667e926700 1 --2- 172.21.15.60:0/4260333652 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f667814af80 0x7f6678159c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:28.158 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.917+0000 7f667e926700 1 --2- 172.21.15.60:0/4260333652 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f66780b2130 0x7f6678156080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:28.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.917+0000 7f667e926700 1 --2- 172.21.15.60:0/4260333652 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f667805d070 0x7f66780b3320 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:28.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.917+0000 7f667e926700 1 -- 172.21.15.60:0/4260333652 >> 172.21.15.60:0/4260333652 conn(0x7f66780b5890 msgr2=0x7f66780b4390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:28.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.917+0000 7f667e926700 1 -- 172.21.15.60:0/4260333652 shutdown_connections 2022-06-29T16:08:28.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:27.917+0000 7f667e926700 1 -- 172.21.15.60:0/4260333652 wait complete. 2022-06-29T16:08:28.160 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:08:28.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f4e848700 1 Processor -- start 2022-06-29T16:08:28.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f4e848700 1 -- start start 2022-06-29T16:08:28.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f4e848700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4f480b9d70 0x7f4f480b3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:28.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f4e848700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4f4805d1f0 0x7f4f480b1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:28.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f4e848700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f480b2ce0 con 0x7f4f480b3cc0 2022-06-29T16:08:28.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f4e848700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f480b4d10 con 0x7f4f480b9d70 2022-06-29T16:08:28.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f4e848700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f480ba150 con 0x7f4f4805d1f0 2022-06-29T16:08:28.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f4cde5700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4f480b3cc0 0x7f4f481420b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34528/0 (socket says 172.21.15.60:34528) 2022-06-29T16:08:28.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f4cde5700 1 -- 172.21.15.60:0/3001272501 learned_addr learned my addr 172.21.15.60:0/3001272501 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:28.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f47fff700 1 --2- 172.21.15.60:0/3001272501 >> v2:172.21.15.60:3300/0 conn(0x7f4f480b9d70 0x7f4f480b3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:28.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f477fe700 1 --2- 172.21.15.60:0/3001272501 >> v2:172.21.15.92:3300/0 conn(0x7f4f4805d1f0 0x7f4f480b1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:28.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f46ffd700 1 -- 172.21.15.60:0/3001272501 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 851227019 0 0) 0x7f4f480b2ce0 con 0x7f4f480b3cc0 2022-06-29T16:08:28.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f46ffd700 1 -- 172.21.15.60:0/3001272501 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f34003260 con 0x7f4f480b3cc0 2022-06-29T16:08:28.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f46ffd700 1 -- 172.21.15.60:0/3001272501 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 834092168 0 0) 0x7f4f34003260 con 0x7f4f480b3cc0 2022-06-29T16:08:28.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f46ffd700 1 -- 172.21.15.60:0/3001272501 >> v2:172.21.15.92:3300/0 conn(0x7f4f4805d1f0 msgr2=0x7f4f480b1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:28.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f46ffd700 1 --2- 172.21.15.60:0/3001272501 >> v2:172.21.15.92:3300/0 conn(0x7f4f4805d1f0 0x7f4f480b1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:28.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f46ffd700 1 -- 172.21.15.60:0/3001272501 >> v2:172.21.15.60:3300/0 conn(0x7f4f480b9d70 msgr2=0x7f4f480b3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:28.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f46ffd700 1 --2- 172.21.15.60:0/3001272501 >> v2:172.21.15.60:3300/0 conn(0x7f4f480b9d70 0x7f4f480b3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:28.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f46ffd700 1 -- 172.21.15.60:0/3001272501 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f480b4a70 con 0x7f4f480b3cc0 2022-06-29T16:08:28.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f46ffd700 1 -- 172.21.15.60:0/3001272501 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f4f40001660 con 0x7f4f480b3cc0 2022-06-29T16:08:28.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f46ffd700 1 -- 172.21.15.60:0/3001272501 >> v1:172.21.15.60:6789/0 conn(0x7f4f480b3cc0 legacy=0x7f4f481420b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:28.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f46ffd700 1 --2- 172.21.15.60:0/3001272501 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f34003750 0x7f4f34003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:28.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f46ffd700 1 --2- 172.21.15.60:0/3001272501 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f34004500 0x7f4f34004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:28.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f46ffd700 1 --2- 172.21.15.60:0/3001272501 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f34004e70 0x7f4f3400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:28.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f46ffd700 1 -- 172.21.15.60:0/3001272501 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f480b4d10 con 0x7f4f34004e70 2022-06-29T16:08:28.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f46ffd700 1 -- 172.21.15.60:0/3001272501 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f480ba150 con 0x7f4f34004500 2022-06-29T16:08:28.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f46ffd700 1 -- 172.21.15.60:0/3001272501 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3400bc40 con 0x7f4f34003750 2022-06-29T16:08:28.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f47fff700 1 --2- 172.21.15.60:0/3001272501 >> v2:172.21.15.60:3300/0 conn(0x7f4f480b9d70 0x7f4f480b3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:08:28.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f4cde5700 1 --2- 172.21.15.60:0/3001272501 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f34004e70 0x7f4f3400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:28.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f477fe700 1 --2- 172.21.15.60:0/3001272501 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f34004500 0x7f4f34004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:28.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f47fff700 1 --2- 172.21.15.60:0/3001272501 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f34003750 0x7f4f34003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:28.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f477fe700 1 -- 172.21.15.60:0/3001272501 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f34003750 msgr2=0x7f4f34003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:28.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f477fe700 1 --2- 172.21.15.60:0/3001272501 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f34003750 0x7f4f34003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:28.300 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f477fe700 1 -- 172.21.15.60:0/3001272501 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f34004e70 msgr2=0x7f4f3400b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:28.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f477fe700 1 --2- 172.21.15.60:0/3001272501 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f34004e70 0x7f4f3400b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:28.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f477fe700 1 -- 172.21.15.60:0/3001272501 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f34003260 con 0x7f4f34004500 2022-06-29T16:08:28.301 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f4cde5700 1 --2- 172.21.15.60:0/3001272501 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f34004e70 0x7f4f3400b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:08:28.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f4cde5700 1 -- 172.21.15.60:0/3001272501 reap_dead start 2022-06-29T16:08:28.302 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f477fe700 1 --2- 172.21.15.60:0/3001272501 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f34004500 0x7f4f34004980 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7f4f3c00c000 tx=0x7f4f3c0091d0).ready entity=mon.1 client_cookie=ffae7b53318f390a server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:28.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f46ffd700 1 -- 172.21.15.60:0/3001272501 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f3c00e780 con 0x7f4f34004500 2022-06-29T16:08:28.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f46ffd700 1 -- 172.21.15.60:0/3001272501 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f3c00e8e0 con 0x7f4f34004500 2022-06-29T16:08:28.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.269+0000 7f4f46ffd700 1 -- 172.21.15.60:0/3001272501 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f3c00ec40 con 0x7f4f34004500 2022-06-29T16:08:28.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.273+0000 7f4f4e848700 1 -- 172.21.15.60:0/3001272501 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f34004500 msgr2=0x7f4f34004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:28.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.273+0000 7f4f4e848700 1 --2- 172.21.15.60:0/3001272501 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f34004500 0x7f4f34004980 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7f4f3c00c000 tx=0x7f4f3c0091d0).stop 2022-06-29T16:08:28.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.273+0000 7f4f4e848700 1 -- 172.21.15.60:0/3001272501 shutdown_connections 2022-06-29T16:08:28.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.273+0000 7f4f4e848700 1 --2- 172.21.15.60:0/3001272501 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f34004500 0x7f4f34004980 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:28.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.273+0000 7f4f4e848700 1 -- 172.21.15.60:0/3001272501 >> 172.21.15.60:0/3001272501 conn(0x7f4f48062ce0 msgr2=0x7f4f480b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:28.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.273+0000 7f4f4e848700 1 -- 172.21.15.60:0/3001272501 shutdown_connections 2022-06-29T16:08:28.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.273+0000 7f4f4e848700 1 -- 172.21.15.60:0/3001272501 wait complete. 2022-06-29T16:08:28.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.273+0000 7f4f4e848700 1 Processor -- start 2022-06-29T16:08:28.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.273+0000 7f4f4e848700 1 -- start start 2022-06-29T16:08:28.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.273+0000 7f4f4e848700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f34004500 0x7f4f48112510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:28.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.273+0000 7f4f4e848700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f48152a60 0x7f4f48142990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:28.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.273+0000 7f4f4e848700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f4813dd20 0x7f4f4813e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:28.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.273+0000 7f4f4e848700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4813e690 con 0x7f4f48152a60 2022-06-29T16:08:28.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.273+0000 7f4f4e848700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f48152080 con 0x7f4f34004500 2022-06-29T16:08:28.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.273+0000 7f4f4e848700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f48152320 con 0x7f4f4813dd20 2022-06-29T16:08:28.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.273+0000 7f4f477fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f48152a60 0x7f4f48142990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:28.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.273+0000 7f4f477fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f48152a60 0x7f4f48142990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54688/0 (socket says 172.21.15.60:54688) 2022-06-29T16:08:28.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.273+0000 7f4f477fe700 1 -- 172.21.15.60:0/98643208 learned_addr learned my addr 172.21.15.60:0/98643208 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:28.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.273+0000 7f4f47fff700 1 --2- 172.21.15.60:0/98643208 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f34004500 0x7f4f48112510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:28.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.273+0000 7f4f4cde5700 1 --2- 172.21.15.60:0/98643208 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f4813dd20 0x7f4f4813e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:28.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.273+0000 7f4f477fe700 1 -- 172.21.15.60:0/98643208 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f4813dd20 msgr2=0x7f4f4813e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:28.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.273+0000 7f4f477fe700 1 --2- 172.21.15.60:0/98643208 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f4813dd20 0x7f4f4813e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:28.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.273+0000 7f4f477fe700 1 -- 172.21.15.60:0/98643208 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f34004500 msgr2=0x7f4f48112510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:28.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.273+0000 7f4f477fe700 1 --2- 172.21.15.60:0/98643208 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f34004500 0x7f4f48112510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:28.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.273+0000 7f4f477fe700 1 -- 172.21.15.60:0/98643208 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f481525c0 con 0x7f4f48152a60 2022-06-29T16:08:28.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.273+0000 7f4f4cde5700 1 --2- 172.21.15.60:0/98643208 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f4813dd20 0x7f4f4813e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:08:28.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.273+0000 7f4f477fe700 1 --2- 172.21.15.60:0/98643208 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f48152a60 0x7f4f48142990 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 rx=0x7f4f3c00c000 tx=0x7f4f3c01a000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:28.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.277+0000 7f4f457fa700 1 -- 172.21.15.60:0/98643208 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f3c01dbb0 con 0x7f4f48152a60 2022-06-29T16:08:28.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.277+0000 7f4f457fa700 1 -- 172.21.15.60:0/98643208 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f3c01dd10 con 0x7f4f48152a60 2022-06-29T16:08:28.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.277+0000 7f4f4e848700 1 -- 172.21.15.60:0/98643208 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f48142e80 con 0x7f4f48152a60 2022-06-29T16:08:28.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.277+0000 7f4f457fa700 1 -- 172.21.15.60:0/98643208 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f3c01b600 con 0x7f4f48152a60 2022-06-29T16:08:28.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.277+0000 7f4f4e848700 1 -- 172.21.15.60:0/98643208 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f480b8ef0 con 0x7f4f48152a60 2022-06-29T16:08:28.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.277+0000 7f4f4e848700 1 -- 172.21.15.60:0/98643208 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f48151ed0 con 0x7f4f48152a60 2022-06-29T16:08:28.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.277+0000 7f4f457fa700 1 -- 172.21.15.60:0/98643208 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4f3c02b020 con 0x7f4f48152a60 2022-06-29T16:08:28.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.277+0000 7f4f457fa700 1 --2- 172.21.15.60:0/98643208 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4f28053b00 0x7f4f28055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:28.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.277+0000 7f4f457fa700 1 -- 172.21.15.60:0/98643208 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4f3c06da10 con 0x7f4f48152a60 2022-06-29T16:08:28.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.277+0000 7f4f47fff700 1 --2- 172.21.15.60:0/98643208 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4f28053b00 0x7f4f28055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:28.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.281+0000 7f4f47fff700 1 --2- 172.21.15.60:0/98643208 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4f28053b00 0x7f4f28055fc0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0x7f4f3800b000 tx=0x7f4f38006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:28.314 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.285+0000 7f4f457fa700 1 -- 172.21.15.60:0/98643208 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4f3c044930 con 0x7f4f48152a60 2022-06-29T16:08:28.454 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.449+0000 7f4f4e848700 1 -- 172.21.15.60:0/98643208 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f48152750 con 0x7f4f48152a60 2022-06-29T16:08:28.455 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.449+0000 7f4f457fa700 1 -- 172.21.15.60:0/98643208 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+2139 (secure 0 0 0) 0x7f4f3c016090 con 0x7f4f48152a60 2022-06-29T16:08:28.456 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:08:28.456 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":73,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7457,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.92:6835/698111951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":698111951},{"type":"v1","addr":"172.21.15.92:6835","nonce":698111951}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":72}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:08:17.075855+0000","modified":"2022-06-29T16:08:18.079541+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8790},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8790":{"gid":8790,"name":"a","rank":0,"incarnation":72,"state":"up:active","state_seq":3,"addr":"172.21.15.60:6833/3619856083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":3619856083},{"type":"v1","addr":"172.21.15.60:6833","nonce":3619856083}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2022-06-29T16:08:28.458 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.453+0000 7f4f4e848700 1 -- 172.21.15.60:0/98643208 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4f28053b00 msgr2=0x7f4f28055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:28.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.453+0000 7f4f4e848700 1 --2- 172.21.15.60:0/98643208 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4f28053b00 0x7f4f28055fc0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0x7f4f3800b000 tx=0x7f4f38006df0).stop 2022-06-29T16:08:28.459 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.453+0000 7f4f4e848700 1 -- 172.21.15.60:0/98643208 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f48152a60 msgr2=0x7f4f48142990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:28.460 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.453+0000 7f4f4e848700 1 --2- 172.21.15.60:0/98643208 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f48152a60 0x7f4f48142990 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 rx=0x7f4f3c00c000 tx=0x7f4f3c01a000).stop 2022-06-29T16:08:28.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.453+0000 7f4f4e848700 1 -- 172.21.15.60:0/98643208 shutdown_connections 2022-06-29T16:08:28.462 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.453+0000 7f4f4e848700 1 --2- 172.21.15.60:0/98643208 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4f28053b00 0x7f4f28055fc0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:28.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.453+0000 7f4f4e848700 1 --2- 172.21.15.60:0/98643208 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4f4813dd20 0x7f4f4813e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:28.463 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.453+0000 7f4f4e848700 1 --2- 172.21.15.60:0/98643208 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4f48152a60 0x7f4f48142990 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:28.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.453+0000 7f4f4e848700 1 --2- 172.21.15.60:0/98643208 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f34004500 0x7f4f48112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:28.464 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.453+0000 7f4f4e848700 1 -- 172.21.15.60:0/98643208 >> 172.21.15.60:0/98643208 conn(0x7f4f48062ce0 msgr2=0x7f4f481402d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:28.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.453+0000 7f4f4e848700 1 -- 172.21.15.60:0/98643208 shutdown_connections 2022-06-29T16:08:28.465 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.453+0000 7f4f4e848700 1 -- 172.21.15.60:0/98643208 wait complete. 2022-06-29T16:08:28.466 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 73 2022-06-29T16:08:28.474 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-06-29T16:08:28.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.569+0000 7f61cd0ae700 1 Processor -- start 2022-06-29T16:08:28.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.569+0000 7f61cd0ae700 1 -- start start 2022-06-29T16:08:28.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61cd0ae700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f61c80b5950 0x7f61c8067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:28.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61cd0ae700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c80acba0 con 0x7f61c8063ad0 2022-06-29T16:08:28.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61cd0ae700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c8067220 con 0x7f61c814af10 2022-06-29T16:08:28.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61cd0ae700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61c814b3a0 con 0x7f61c80b5950 2022-06-29T16:08:28.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c659c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f61c8063ad0 0x7f61c805bd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34548/0 (socket says 172.21.15.60:34548) 2022-06-29T16:08:28.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c659c700 1 -- 172.21.15.60:0/3394145677 learned_addr learned my addr 172.21.15.60:0/3394145677 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:28.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c6d9d700 1 --2- 172.21.15.60:0/3394145677 >> v2:172.21.15.92:3300/0 conn(0x7f61c80b5950 0x7f61c8067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:28.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c5d9b700 1 -- 172.21.15.60:0/3394145677 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3064555871 0 0) 0x7f61c80acba0 con 0x7f61c8063ad0 2022-06-29T16:08:28.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c5d9b700 1 -- 172.21.15.60:0/3394145677 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61ac003260 con 0x7f61c8063ad0 2022-06-29T16:08:28.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c5d9b700 1 -- 172.21.15.60:0/3394145677 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3724369599 0 0) 0x7f61c8067220 con 0x7f61c814af10 2022-06-29T16:08:28.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c5d9b700 1 -- 172.21.15.60:0/3394145677 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61c80acba0 con 0x7f61c814af10 2022-06-29T16:08:28.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c5d9b700 1 -- 172.21.15.60:0/3394145677 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1737257508 0 0) 0x7f61ac003260 con 0x7f61c8063ad0 2022-06-29T16:08:28.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c5d9b700 1 -- 172.21.15.60:0/3394145677 >> v2:172.21.15.92:3300/0 conn(0x7f61c80b5950 msgr2=0x7f61c8067680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:28.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c5d9b700 1 --2- 172.21.15.60:0/3394145677 >> v2:172.21.15.92:3300/0 conn(0x7f61c80b5950 0x7f61c8067680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:28.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c5d9b700 1 -- 172.21.15.60:0/3394145677 >> v1:172.21.15.92:6789/0 conn(0x7f61c814af10 legacy=0x7f61c805c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:28.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c5d9b700 1 -- 172.21.15.60:0/3394145677 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c80b5d30 con 0x7f61c8063ad0 2022-06-29T16:08:28.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c5d9b700 1 -- 172.21.15.60:0/3394145677 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f61bc001680 con 0x7f61c8063ad0 2022-06-29T16:08:28.583 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c6d9d700 1 --2- 172.21.15.60:0/3394145677 >> v2:172.21.15.92:3300/0 conn(0x7f61c80b5950 0x7f61c8067680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T16:08:28.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c5d9b700 1 -- 172.21.15.60:0/3394145677 >> v1:172.21.15.60:6789/0 conn(0x7f61c8063ad0 legacy=0x7f61c805bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:28.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c5d9b700 1 --2- 172.21.15.60:0/3394145677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61ac003930 0x7f61ac003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:28.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c5d9b700 1 --2- 172.21.15.60:0/3394145677 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61ac004500 0x7f61ac004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:28.584 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c5d9b700 1 --2- 172.21.15.60:0/3394145677 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61ac004e70 0x7f61ac00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:28.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c5d9b700 1 -- 172.21.15.60:0/3394145677 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c814b3a0 con 0x7f61ac004500 2022-06-29T16:08:28.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c5d9b700 1 -- 172.21.15.60:0/3394145677 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61ac00bd20 con 0x7f61ac003930 2022-06-29T16:08:28.586 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c5d9b700 1 -- 172.21.15.60:0/3394145677 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61ac00bee0 con 0x7f61ac004e70 2022-06-29T16:08:28.586 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c659c700 1 --2- 172.21.15.60:0/3394145677 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61ac004500 0x7f61ac004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:28.586 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c6d9d700 1 --2- 172.21.15.60:0/3394145677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61ac003930 0x7f61ac003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:28.586 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c759e700 1 --2- 172.21.15.60:0/3394145677 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61ac004e70 0x7f61ac00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:28.587 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c659c700 1 -- 172.21.15.60:0/3394145677 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61ac004e70 msgr2=0x7f61ac00b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:28.587 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c659c700 1 --2- 172.21.15.60:0/3394145677 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61ac004e70 0x7f61ac00b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:28.587 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c659c700 1 -- 172.21.15.60:0/3394145677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61ac003930 msgr2=0x7f61ac003d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:28.587 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c659c700 1 --2- 172.21.15.60:0/3394145677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61ac003930 0x7f61ac003d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:28.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c659c700 1 -- 172.21.15.60:0/3394145677 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61ac003260 con 0x7f61ac004500 2022-06-29T16:08:28.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c759e700 1 --2- 172.21.15.60:0/3394145677 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61ac004e70 0x7f61ac00b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:08:28.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c759e700 1 -- 172.21.15.60:0/3394145677 reap_dead start 2022-06-29T16:08:28.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c659c700 1 --2- 172.21.15.60:0/3394145677 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61ac004500 0x7f61ac004980 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 rx=0x7f61bc001450 tx=0x7f61c805ce50).ready entity=mon.0 client_cookie=1c4372742f1111d2 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:28.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c5d9b700 1 -- 172.21.15.60:0/3394145677 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61bc007170 con 0x7f61ac004500 2022-06-29T16:08:28.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c5d9b700 1 -- 172.21.15.60:0/3394145677 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61bc0072d0 con 0x7f61ac004500 2022-06-29T16:08:28.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.573+0000 7f61c5d9b700 1 -- 172.21.15.60:0/3394145677 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61bc00c8f0 con 0x7f61ac004500 2022-06-29T16:08:28.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.577+0000 7f61cd0ae700 1 -- 172.21.15.60:0/3394145677 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61ac004500 msgr2=0x7f61ac004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:28.593 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.577+0000 7f61cd0ae700 1 --2- 172.21.15.60:0/3394145677 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61ac004500 0x7f61ac004980 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 rx=0x7f61bc001450 tx=0x7f61c805ce50).stop 2022-06-29T16:08:28.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.577+0000 7f61cd0ae700 1 -- 172.21.15.60:0/3394145677 shutdown_connections 2022-06-29T16:08:28.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.577+0000 7f61cd0ae700 1 --2- 172.21.15.60:0/3394145677 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61ac004500 0x7f61ac004980 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:28.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.577+0000 7f61cd0ae700 1 -- 172.21.15.60:0/3394145677 >> 172.21.15.60:0/3394145677 conn(0x7f61c80632c0 msgr2=0x7f61c80b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:28.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.577+0000 7f61cd0ae700 1 -- 172.21.15.60:0/3394145677 shutdown_connections 2022-06-29T16:08:28.594 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.577+0000 7f61cd0ae700 1 -- 172.21.15.60:0/3394145677 wait complete. 2022-06-29T16:08:28.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.577+0000 7f61cd0ae700 1 Processor -- start 2022-06-29T16:08:28.595 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.577+0000 7f61cd0ae700 1 -- start start 2022-06-29T16:08:28.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.577+0000 7f61cd0ae700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61ac004500 0x7f61c8117d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:28.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.577+0000 7f61cd0ae700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61c81433a0 0x7f61c8142030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:28.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.577+0000 7f61cd0ae700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61c8142520 0x7f61c80afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:28.596 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.577+0000 7f61cd0ae700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c8143800 con 0x7f61c8142520 2022-06-29T16:08:28.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.577+0000 7f61cd0ae700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c8118c00 con 0x7f61c81433a0 2022-06-29T16:08:28.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.577+0000 7f61cd0ae700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61c8156230 con 0x7f61ac004500 2022-06-29T16:08:28.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.577+0000 7f61c759e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61c8142520 0x7f61c80afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:28.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.577+0000 7f61c759e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61c8142520 0x7f61c80afdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54708/0 (socket says 172.21.15.60:54708) 2022-06-29T16:08:28.597 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.577+0000 7f61c759e700 1 -- 172.21.15.60:0/1148486361 learned_addr learned my addr 172.21.15.60:0/1148486361 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:28.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.577+0000 7f61c6d9d700 1 --2- 172.21.15.60:0/1148486361 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61ac004500 0x7f61c8117d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:28.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.577+0000 7f61c659c700 1 --2- 172.21.15.60:0/1148486361 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61c81433a0 0x7f61c8142030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:28.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.577+0000 7f61c759e700 1 -- 172.21.15.60:0/1148486361 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61ac004500 msgr2=0x7f61c8117d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:28.598 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.577+0000 7f61c759e700 1 --2- 172.21.15.60:0/1148486361 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61ac004500 0x7f61c8117d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:28.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.577+0000 7f61c759e700 1 -- 172.21.15.60:0/1148486361 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61c81433a0 msgr2=0x7f61c8142030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:28.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.577+0000 7f61c759e700 1 --2- 172.21.15.60:0/1148486361 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61c81433a0 0x7f61c8142030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:28.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.577+0000 7f61c759e700 1 -- 172.21.15.60:0/1148486361 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c81564d0 con 0x7f61c8142520 2022-06-29T16:08:28.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.577+0000 7f61c659c700 1 --2- 172.21.15.60:0/1148486361 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61c81433a0 0x7f61c8142030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:08:28.599 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.577+0000 7f61c759e700 1 --2- 172.21.15.60:0/1148486361 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61c8142520 0x7f61c80afdb0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 rx=0x7f61c8156af0 tx=0x7f61c0009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:28.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.581+0000 7f61b7fff700 1 -- 172.21.15.60:0/1148486361 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61c0003220 con 0x7f61c8142520 2022-06-29T16:08:28.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.581+0000 7f61b7fff700 1 -- 172.21.15.60:0/1148486361 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61c0003380 con 0x7f61c8142520 2022-06-29T16:08:28.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.581+0000 7f61b7fff700 1 -- 172.21.15.60:0/1148486361 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61c000eea0 con 0x7f61c8142520 2022-06-29T16:08:28.600 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.581+0000 7f61cd0ae700 1 -- 172.21.15.60:0/1148486361 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61c8156770 con 0x7f61c8142520 2022-06-29T16:08:28.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.581+0000 7f61cd0ae700 1 -- 172.21.15.60:0/1148486361 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61c8144090 con 0x7f61c8142520 2022-06-29T16:08:28.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.581+0000 7f61cd0ae700 1 -- 172.21.15.60:0/1148486361 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61c8156900 con 0x7f61c8142520 2022-06-29T16:08:28.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.581+0000 7f61b7fff700 1 -- 172.21.15.60:0/1148486361 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f61c001e070 con 0x7f61c8142520 2022-06-29T16:08:28.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.581+0000 7f61b7fff700 1 --2- 172.21.15.60:0/1148486361 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f61a4053aa0 0x7f61a4055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:28.601 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.581+0000 7f61b7fff700 1 -- 172.21.15.60:0/1148486361 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f61c006b530 con 0x7f61c8142520 2022-06-29T16:08:28.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.581+0000 7f61c6d9d700 1 --2- 172.21.15.60:0/1148486361 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f61a4053aa0 0x7f61a4055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:28.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.585+0000 7f61c6d9d700 1 --2- 172.21.15.60:0/1148486361 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f61a4053aa0 0x7f61a4055f60 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0x7f61b800b000 tx=0x7f61b8006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:28.602 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.589+0000 7f61b7fff700 1 -- 172.21.15.60:0/1148486361 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f61c0042450 con 0x7f61c8142520 2022-06-29T16:08:28.759 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:28.753+0000 7f61cd0ae700 1 -- 172.21.15.60:0/1148486361 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f61c80573d0 con 0x7f61c8142520 2022-06-29T16:08:29.183 INFO:tasks.ceph.mon.a.smithi060.stderr:2022-06-29T16:08:29.177+0000 7fcc1a376700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-06-29T16:08:29.197 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.193+0000 7f61b7fff700 1 -- 172.21.15.60:0/1148486361 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v74) v1 ==== 161+0+0 (secure 0 0 0) 0x7f61c0045a70 con 0x7f61c8142520 2022-06-29T16:08:29.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.197+0000 7f61cd0ae700 1 -- 172.21.15.60:0/1148486361 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f61a4053aa0 msgr2=0x7f61a4055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:29.203 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.197+0000 7f61cd0ae700 1 --2- 172.21.15.60:0/1148486361 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f61a4053aa0 0x7f61a4055f60 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0x7f61b800b000 tx=0x7f61b8006df0).stop 2022-06-29T16:08:29.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.197+0000 7f61cd0ae700 1 -- 172.21.15.60:0/1148486361 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61c8142520 msgr2=0x7f61c80afdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:29.204 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.197+0000 7f61cd0ae700 1 --2- 172.21.15.60:0/1148486361 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61c8142520 0x7f61c80afdb0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 rx=0x7f61c8156af0 tx=0x7f61c0009040).stop 2022-06-29T16:08:29.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.197+0000 7f61cd0ae700 1 -- 172.21.15.60:0/1148486361 shutdown_connections 2022-06-29T16:08:29.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.197+0000 7f61cd0ae700 1 --2- 172.21.15.60:0/1148486361 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f61a4053aa0 0x7f61a4055f60 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:29.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.197+0000 7f61cd0ae700 1 --2- 172.21.15.60:0/1148486361 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61c8142520 0x7f61c80afdb0 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:29.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.197+0000 7f61cd0ae700 1 --2- 172.21.15.60:0/1148486361 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61c81433a0 0x7f61c8142030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:29.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.197+0000 7f61cd0ae700 1 --2- 172.21.15.60:0/1148486361 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f61ac004500 0x7f61c8117d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:29.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.197+0000 7f61cd0ae700 1 -- 172.21.15.60:0/1148486361 >> 172.21.15.60:0/1148486361 conn(0x7f61c80632c0 msgr2=0x7f61c814a450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:29.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.197+0000 7f61cd0ae700 1 -- 172.21.15.60:0/1148486361 shutdown_connections 2022-06-29T16:08:29.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.197+0000 7f61cd0ae700 1 -- 172.21.15.60:0/1148486361 wait complete. 2022-06-29T16:08:29.209 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-06-29T16:08:29.223 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-06-29T16:08:29.231 INFO:tasks.ceph.mds.a.smithi060.stdout:starting mds.a at 2022-06-29T16:08:29.322 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.317+0000 7f0e9d0fd700 1 Processor -- start 2022-06-29T16:08:29.323 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.317+0000 7f0e9d0fd700 1 -- start start 2022-06-29T16:08:29.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.317+0000 7f0e9d0fd700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f0e980b48b0 0x7f0e98142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:29.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.317+0000 7f0e9d0fd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0e980b2c50 0x7f0e980b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:29.333 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.317+0000 7f0e9d0fd700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e980b0cf0 con 0x7f0e980b5eb0 2022-06-29T16:08:29.334 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.317+0000 7f0e9d0fd700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e980b54e0 con 0x7f0e980b2c50 2022-06-29T16:08:29.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.317+0000 7f0e9d0fd700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e980b4d10 con 0x7f0e980b48b0 2022-06-29T16:08:29.335 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.321+0000 7f0e9c8fc700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0e980b2c50 0x7f0e980b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:29.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.321+0000 7f0e97fff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f0e980b5eb0 0x7f0e980b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34580/0 (socket says 172.21.15.60:34580) 2022-06-29T16:08:29.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.321+0000 7f0e97fff700 1 -- 172.21.15.60:0/1049763335 learned_addr learned my addr 172.21.15.60:0/1049763335 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:29.336 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.321+0000 7f0e977fe700 1 --2- 172.21.15.60:0/1049763335 >> v2:172.21.15.92:3300/0 conn(0x7f0e980b48b0 0x7f0e98142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:29.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.321+0000 7f0e9c8fc700 1 -- 172.21.15.60:0/1049763335 >> v2:172.21.15.92:3300/0 conn(0x7f0e980b48b0 msgr2=0x7f0e98142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:29.337 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.321+0000 7f0e9c8fc700 1 --2- 172.21.15.60:0/1049763335 >> v2:172.21.15.92:3300/0 conn(0x7f0e980b48b0 0x7f0e98142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:29.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.321+0000 7f0e9c8fc700 1 -- 172.21.15.60:0/1049763335 >> v1:172.21.15.60:6789/0 conn(0x7f0e980b5eb0 legacy=0x7f0e980b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:29.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.321+0000 7f0e96ffd700 1 -- 172.21.15.60:0/1049763335 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3749764802 0 0) 0x7f0e88001110 con 0x7f0e980b5eb0 2022-06-29T16:08:29.338 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.321+0000 7f0e9c8fc700 1 -- 172.21.15.60:0/1049763335 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e980b6290 con 0x7f0e980b2c50 2022-06-29T16:08:29.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.321+0000 7f0e9c8fc700 1 --2- 172.21.15.60:0/1049763335 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e980b2c50 0x7f0e980b30c0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 rx=0x7f0e900032f0 tx=0x7f0e900091d0).ready entity=mon.0 client_cookie=dbeaaf25a97a850e server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:29.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.321+0000 7f0e96ffd700 1 -- 172.21.15.60:0/1049763335 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e9000b680 con 0x7f0e980b2c50 2022-06-29T16:08:29.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.321+0000 7f0e96ffd700 1 -- 172.21.15.60:0/1049763335 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e9000b7e0 con 0x7f0e980b2c50 2022-06-29T16:08:29.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.321+0000 7f0e96ffd700 1 -- 172.21.15.60:0/1049763335 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e9000b950 con 0x7f0e980b2c50 2022-06-29T16:08:29.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.321+0000 7f0e9d0fd700 1 -- 172.21.15.60:0/1049763335 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e980b2c50 msgr2=0x7f0e980b30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:29.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.321+0000 7f0e9d0fd700 1 --2- 172.21.15.60:0/1049763335 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e980b2c50 0x7f0e980b30c0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 rx=0x7f0e900032f0 tx=0x7f0e900091d0).stop 2022-06-29T16:08:29.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.321+0000 7f0e9d0fd700 1 -- 172.21.15.60:0/1049763335 shutdown_connections 2022-06-29T16:08:29.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.321+0000 7f0e9d0fd700 1 --2- 172.21.15.60:0/1049763335 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e980b2c50 0x7f0e980b30c0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:29.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.321+0000 7f0e9d0fd700 1 --2- 172.21.15.60:0/1049763335 >> v2:172.21.15.92:3300/0 conn(0x7f0e980b48b0 0x7f0e98142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:29.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.321+0000 7f0e9d0fd700 1 -- 172.21.15.60:0/1049763335 >> 172.21.15.60:0/1049763335 conn(0x7f0e98062090 msgr2=0x7f0e98062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:29.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.321+0000 7f0e9d0fd700 1 -- 172.21.15.60:0/1049763335 shutdown_connections 2022-06-29T16:08:29.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.321+0000 7f0e9d0fd700 1 -- 172.21.15.60:0/1049763335 wait complete. 2022-06-29T16:08:29.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.325+0000 7f0e9d0fd700 1 Processor -- start 2022-06-29T16:08:29.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.325+0000 7f0e9d0fd700 1 -- start start 2022-06-29T16:08:29.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.325+0000 7f0e9d0fd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e980b2c50 0x7f0e980acd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:29.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.325+0000 7f0e9d0fd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e980b48b0 0x7f0e980adef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:29.343 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.325+0000 7f0e9d0fd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e980b5eb0 0x7f0e980ae3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:29.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.325+0000 7f0e9d0fd700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e981446c0 con 0x7f0e980b5eb0 2022-06-29T16:08:29.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.325+0000 7f0e9d0fd700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e980adac0 con 0x7f0e980b2c50 2022-06-29T16:08:29.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.325+0000 7f0e9d0fd700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e980b1c90 con 0x7f0e980b48b0 2022-06-29T16:08:29.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.325+0000 7f0e9c8fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e980b5eb0 0x7f0e980ae3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:29.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.325+0000 7f0e9c8fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e980b5eb0 0x7f0e980ae3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54736/0 (socket says 172.21.15.60:54736) 2022-06-29T16:08:29.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.325+0000 7f0e9c8fc700 1 -- 172.21.15.60:0/1085434418 learned_addr learned my addr 172.21.15.60:0/1085434418 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:29.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.325+0000 7f0e97fff700 1 --2- 172.21.15.60:0/1085434418 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e980b2c50 0x7f0e980acd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:29.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.325+0000 7f0e977fe700 1 --2- 172.21.15.60:0/1085434418 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e980b48b0 0x7f0e980adef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:29.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.325+0000 7f0e9c8fc700 1 -- 172.21.15.60:0/1085434418 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e980b48b0 msgr2=0x7f0e980adef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:29.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.325+0000 7f0e9c8fc700 1 --2- 172.21.15.60:0/1085434418 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e980b48b0 0x7f0e980adef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:29.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.325+0000 7f0e9c8fc700 1 -- 172.21.15.60:0/1085434418 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e980b2c50 msgr2=0x7f0e980acd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:29.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.325+0000 7f0e9c8fc700 1 --2- 172.21.15.60:0/1085434418 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e980b2c50 0x7f0e980acd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:29.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.325+0000 7f0e9c8fc700 1 -- 172.21.15.60:0/1085434418 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e90009470 con 0x7f0e980b5eb0 2022-06-29T16:08:29.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.325+0000 7f0e977fe700 1 --2- 172.21.15.60:0/1085434418 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e980b48b0 0x7f0e980adef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:08:29.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.325+0000 7f0e9c8fc700 1 --2- 172.21.15.60:0/1085434418 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e980b5eb0 0x7f0e980ae3e0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 rx=0x7f0e980b24a0 tx=0x7f0e90016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:29.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.325+0000 7f0e957fa700 1 -- 172.21.15.60:0/1085434418 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e9000b950 con 0x7f0e980b5eb0 2022-06-29T16:08:29.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.325+0000 7f0e957fa700 1 -- 172.21.15.60:0/1085434418 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e90021630 con 0x7f0e980b5eb0 2022-06-29T16:08:29.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.325+0000 7f0e957fa700 1 -- 172.21.15.60:0/1085434418 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e9000b680 con 0x7f0e980b5eb0 2022-06-29T16:08:29.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.325+0000 7f0e9d0fd700 1 -- 172.21.15.60:0/1085434418 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e980b1f30 con 0x7f0e980b5eb0 2022-06-29T16:08:29.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.325+0000 7f0e9d0fd700 1 -- 172.21.15.60:0/1085434418 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e980ae950 con 0x7f0e980b5eb0 2022-06-29T16:08:29.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.325+0000 7f0e9d0fd700 1 -- 172.21.15.60:0/1085434418 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e980573d0 con 0x7f0e980b5eb0 2022-06-29T16:08:29.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.329+0000 7f0e957fa700 1 -- 172.21.15.60:0/1085434418 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0e90027020 con 0x7f0e980b5eb0 2022-06-29T16:08:29.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.329+0000 7f0e957fa700 1 --2- 172.21.15.60:0/1085434418 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0e78053b00 0x7f0e78055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:29.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.329+0000 7f0e957fa700 1 -- 172.21.15.60:0/1085434418 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f0e9006d3d0 con 0x7f0e980b5eb0 2022-06-29T16:08:29.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.329+0000 7f0e97fff700 1 --2- 172.21.15.60:0/1085434418 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0e78053b00 0x7f0e78055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:29.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.329+0000 7f0e97fff700 1 --2- 172.21.15.60:0/1085434418 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0e78053b00 0x7f0e78055fc0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 rx=0x7f0e8800d000 tx=0x7f0e88007c00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:29.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.333+0000 7f0e957fa700 1 -- 172.21.15.60:0/1085434418 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0e900442a0 con 0x7f0e980b5eb0 2022-06-29T16:08:29.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.497+0000 7f0e9d0fd700 1 -- 172.21.15.60:0/1085434418 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e98067a30 con 0x7f0e980b5eb0 2022-06-29T16:08:29.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.501+0000 7f0e957fa700 1 -- 172.21.15.60:0/1085434418 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+1779 (secure 0 0 0) 0x7f0e90015090 con 0x7f0e980b5eb0 2022-06-29T16:08:29.504 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T16:08:29.505 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":74,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7457,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.92:6835/698111951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":698111951},{"type":"v1","addr":"172.21.15.92:6835","nonce":698111951}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":72}],"filesystems":[{"mdsmap":{"epoch":74,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-06-29T16:08:17.075855+0000","modified":"2022-06-29T16:08:29.181234+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":148,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2022-06-29T16:08:29.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.501+0000 7f0e9d0fd700 1 -- 172.21.15.60:0/1085434418 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0e78053b00 msgr2=0x7f0e78055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:29.506 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.501+0000 7f0e9d0fd700 1 --2- 172.21.15.60:0/1085434418 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0e78053b00 0x7f0e78055fc0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 rx=0x7f0e8800d000 tx=0x7f0e88007c00).stop 2022-06-29T16:08:29.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.501+0000 7f0e9d0fd700 1 -- 172.21.15.60:0/1085434418 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e980b5eb0 msgr2=0x7f0e980ae3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:29.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.501+0000 7f0e9d0fd700 1 --2- 172.21.15.60:0/1085434418 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e980b5eb0 0x7f0e980ae3e0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 rx=0x7f0e980b24a0 tx=0x7f0e90016000).stop 2022-06-29T16:08:29.507 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.501+0000 7f0e9d0fd700 1 -- 172.21.15.60:0/1085434418 shutdown_connections 2022-06-29T16:08:29.508 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.501+0000 7f0e9d0fd700 1 --2- 172.21.15.60:0/1085434418 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f0e78053b00 0x7f0e78055fc0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:29.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.501+0000 7f0e9d0fd700 1 --2- 172.21.15.60:0/1085434418 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e980b5eb0 0x7f0e980ae3e0 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:29.509 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.501+0000 7f0e9d0fd700 1 --2- 172.21.15.60:0/1085434418 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0e980b48b0 0x7f0e980adef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:29.510 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.501+0000 7f0e9d0fd700 1 --2- 172.21.15.60:0/1085434418 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e980b2c50 0x7f0e980acd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:29.510 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.501+0000 7f0e9d0fd700 1 -- 172.21.15.60:0/1085434418 >> 172.21.15.60:0/1085434418 conn(0x7f0e98062090 msgr2=0x7f0e9805bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:29.511 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.501+0000 7f0e9d0fd700 1 -- 172.21.15.60:0/1085434418 shutdown_connections 2022-06-29T16:08:29.511 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.501+0000 7f0e9d0fd700 1 -- 172.21.15.60:0/1085434418 wait complete. 2022-06-29T16:08:29.512 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 74 2022-06-29T16:08:29.521 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-06-29T16:08:29.678 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.673+0000 7f70e9e46700 1 Processor -- start 2022-06-29T16:08:29.678 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.673+0000 7f70e9e46700 1 -- start start 2022-06-29T16:08:29.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.673+0000 7f70e9e46700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f70e405d070 0x7f70e414aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:29.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.673+0000 7f70e9e46700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70e405d450 con 0x7f70e40b2130 2022-06-29T16:08:29.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.673+0000 7f70e9e46700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70e40ad840 con 0x7f70e414af80 2022-06-29T16:08:29.681 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.673+0000 7f70e9e46700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70e40adf90 con 0x7f70e405d070 2022-06-29T16:08:29.682 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.673+0000 7f70e8e44700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f70e405d070 0x7f70e414aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:29.682 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.673+0000 7f70e3fff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f70e40b2130 0x7f70e40aca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34596/0 (socket says 172.21.15.60:34596) 2022-06-29T16:08:29.682 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.673+0000 7f70e3fff700 1 -- 172.21.15.60:0/4049775092 learned_addr learned my addr 172.21.15.60:0/4049775092 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T16:08:29.682 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.677+0000 7f70e8e44700 1 -- 172.21.15.60:0/4049775092 >> v1:172.21.15.92:6789/0 conn(0x7f70e414af80 legacy=0x7f70e40ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:29.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.677+0000 7f70e8e44700 1 -- 172.21.15.60:0/4049775092 >> v1:172.21.15.60:6789/0 conn(0x7f70e40b2130 legacy=0x7f70e40aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:29.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.677+0000 7f70e37fe700 1 -- 172.21.15.60:0/4049775092 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2558352459 0 0) 0x7f70e405d450 con 0x7f70e40b2130 2022-06-29T16:08:29.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.677+0000 7f70e8e44700 1 -- 172.21.15.60:0/4049775092 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e40b25d0 con 0x7f70e405d070 2022-06-29T16:08:29.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.677+0000 7f70e8e44700 1 --2- 172.21.15.60:0/4049775092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70e405d070 0x7f70e414aa90 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 rx=0x7f70d8002f70 tx=0x7f70d8006df0).ready entity=mon.0 client_cookie=8f5529aa82f2a95a server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:29.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.677+0000 7f70e37fe700 1 -- 172.21.15.60:0/4049775092 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70d800d610 con 0x7f70e405d070 2022-06-29T16:08:29.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.677+0000 7f70e37fe700 1 -- 172.21.15.60:0/4049775092 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70d8000e90 con 0x7f70e405d070 2022-06-29T16:08:29.684 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.677+0000 7f70e37fe700 1 -- 172.21.15.60:0/4049775092 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70d800d960 con 0x7f70e405d070 2022-06-29T16:08:29.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.677+0000 7f70e9e46700 1 -- 172.21.15.60:0/4049775092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70e405d070 msgr2=0x7f70e414aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:29.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.677+0000 7f70e9e46700 1 --2- 172.21.15.60:0/4049775092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70e405d070 0x7f70e414aa90 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 rx=0x7f70d8002f70 tx=0x7f70d8006df0).stop 2022-06-29T16:08:29.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.677+0000 7f70e9e46700 1 -- 172.21.15.60:0/4049775092 shutdown_connections 2022-06-29T16:08:29.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.677+0000 7f70e9e46700 1 --2- 172.21.15.60:0/4049775092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70e405d070 0x7f70e414aa90 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:29.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.677+0000 7f70e9e46700 1 -- 172.21.15.60:0/4049775092 >> 172.21.15.60:0/4049775092 conn(0x7f70e40b5890 msgr2=0x7f70e40b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:29.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.677+0000 7f70e9e46700 1 -- 172.21.15.60:0/4049775092 shutdown_connections 2022-06-29T16:08:29.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.677+0000 7f70e9e46700 1 -- 172.21.15.60:0/4049775092 wait complete. 2022-06-29T16:08:29.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.677+0000 7f70e9e46700 1 Processor -- start 2022-06-29T16:08:29.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.677+0000 7f70e9e46700 1 -- start start 2022-06-29T16:08:29.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.681+0000 7f70e9e46700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70e405d070 0x7f70e40b4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:29.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.681+0000 7f70e9e46700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70e40b2130 0x7f70e40666c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:29.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.681+0000 7f70e9e46700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70e414af80 0x7f70e4066bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:29.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.681+0000 7f70e9e46700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e4116f20 con 0x7f70e414af80 2022-06-29T16:08:29.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.681+0000 7f70e9e46700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e415a250 con 0x7f70e40b2130 2022-06-29T16:08:29.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.681+0000 7f70e9e46700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70e415a4f0 con 0x7f70e405d070 2022-06-29T16:08:29.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.681+0000 7f70e9645700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70e414af80 0x7f70e4066bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:29.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.681+0000 7f70e9645700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70e414af80 0x7f70e4066bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54748/0 (socket says 172.21.15.60:54748) 2022-06-29T16:08:29.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.681+0000 7f70e9645700 1 -- 172.21.15.60:0/1876901788 learned_addr learned my addr 172.21.15.60:0/1876901788 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:29.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.681+0000 7f70e8e44700 1 --2- 172.21.15.60:0/1876901788 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70e405d070 0x7f70e40b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:29.694 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.681+0000 7f70e3fff700 1 --2- 172.21.15.60:0/1876901788 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70e40b2130 0x7f70e40666c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:29.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.681+0000 7f70e9645700 1 -- 172.21.15.60:0/1876901788 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70e405d070 msgr2=0x7f70e40b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:29.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.681+0000 7f70e9645700 1 --2- 172.21.15.60:0/1876901788 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70e405d070 0x7f70e40b4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:29.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.681+0000 7f70e9645700 1 -- 172.21.15.60:0/1876901788 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70e40b2130 msgr2=0x7f70e40666c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:29.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.681+0000 7f70e9645700 1 --2- 172.21.15.60:0/1876901788 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70e40b2130 0x7f70e40666c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:29.696 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.681+0000 7f70e9645700 1 -- 172.21.15.60:0/1876901788 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d800b180 con 0x7f70e414af80 2022-06-29T16:08:29.696 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.681+0000 7f70e3fff700 1 --2- 172.21.15.60:0/1876901788 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70e40b2130 0x7f70e40666c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:08:29.696 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.681+0000 7f70e9645700 1 --2- 172.21.15.60:0/1876901788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70e414af80 0x7f70e4066bb0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 rx=0x7f70e40b5cd0 tx=0x7f70dc007bf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:29.696 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.681+0000 7f70e1ffb700 1 -- 172.21.15.60:0/1876901788 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70dc001d50 con 0x7f70e414af80 2022-06-29T16:08:29.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.681+0000 7f70e1ffb700 1 -- 172.21.15.60:0/1876901788 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70dc007030 con 0x7f70e414af80 2022-06-29T16:08:29.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.681+0000 7f70e1ffb700 1 -- 172.21.15.60:0/1876901788 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70dc007390 con 0x7f70e414af80 2022-06-29T16:08:29.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.681+0000 7f70e9e46700 1 -- 172.21.15.60:0/1876901788 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70e4065f10 con 0x7f70e414af80 2022-06-29T16:08:29.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.681+0000 7f70e9e46700 1 -- 172.21.15.60:0/1876901788 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70e4116c10 con 0x7f70e414af80 2022-06-29T16:08:29.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.681+0000 7f70e1ffb700 1 -- 172.21.15.60:0/1876901788 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f70dc010030 con 0x7f70e414af80 2022-06-29T16:08:29.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.681+0000 7f70e9e46700 1 -- 172.21.15.60:0/1876901788 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70e40656b0 con 0x7f70e414af80 2022-06-29T16:08:29.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.685+0000 7f70e1ffb700 1 --2- 172.21.15.60:0/1876901788 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f70c4053a70 0x7f70c4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:29.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.685+0000 7f70e1ffb700 1 -- 172.21.15.60:0/1876901788 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f70dc06ba20 con 0x7f70e414af80 2022-06-29T16:08:29.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.685+0000 7f70e8e44700 1 --2- 172.21.15.60:0/1876901788 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f70c4053a70 0x7f70c4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:29.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.685+0000 7f70e8e44700 1 --2- 172.21.15.60:0/1876901788 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f70c4053a70 0x7f70c4055f30 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0x7f70d801c000 tx=0x7f70d8015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:29.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.689+0000 7f70e1ffb700 1 -- 172.21.15.60:0/1876901788 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f70dc0428f0 con 0x7f70e414af80 2022-06-29T16:08:29.836 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:08:29.840 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:08:29.843 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:08:29.846 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:08:29.848 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:08:29.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:29.853+0000 7f70e9e46700 1 -- 172.21.15.60:0/1876901788 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f70e414a350 con 0x7f70e414af80 2022-06-29T16:08:30.201 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.197+0000 7f70e1ffb700 1 -- 172.21.15.60:0/1876901788 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v75) v1 ==== 104+0+0 (secure 0 0 0) 0x7f70dc045f10 con 0x7f70e414af80 2022-06-29T16:08:30.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.201+0000 7f70e9e46700 1 -- 172.21.15.60:0/1876901788 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f70c4053a70 msgr2=0x7f70c4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:30.205 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.201+0000 7f70e9e46700 1 --2- 172.21.15.60:0/1876901788 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f70c4053a70 0x7f70c4055f30 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0x7f70d801c000 tx=0x7f70d8015000).stop 2022-06-29T16:08:30.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.201+0000 7f70e9e46700 1 -- 172.21.15.60:0/1876901788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70e414af80 msgr2=0x7f70e4066bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:30.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.201+0000 7f70e9e46700 1 --2- 172.21.15.60:0/1876901788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70e414af80 0x7f70e4066bb0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 rx=0x7f70e40b5cd0 tx=0x7f70dc007bf0).stop 2022-06-29T16:08:30.206 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.201+0000 7f70e9e46700 1 -- 172.21.15.60:0/1876901788 shutdown_connections 2022-06-29T16:08:30.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.201+0000 7f70e9e46700 1 --2- 172.21.15.60:0/1876901788 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f70c4053a70 0x7f70c4055f30 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:30.207 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.201+0000 7f70e9e46700 1 --2- 172.21.15.60:0/1876901788 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f70e414af80 0x7f70e4066bb0 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:30.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.201+0000 7f70e9e46700 1 --2- 172.21.15.60:0/1876901788 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70e40b2130 0x7f70e40666c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:30.208 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.201+0000 7f70e9e46700 1 --2- 172.21.15.60:0/1876901788 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f70e405d070 0x7f70e40b4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:30.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.201+0000 7f70e9e46700 1 -- 172.21.15.60:0/1876901788 >> 172.21.15.60:0/1876901788 conn(0x7f70e40b5890 msgr2=0x7f70e4145420 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:30.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.201+0000 7f70e9e46700 1 -- 172.21.15.60:0/1876901788 shutdown_connections 2022-06-29T16:08:30.209 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.201+0000 7f70e9e46700 1 -- 172.21.15.60:0/1876901788 wait complete. 2022-06-29T16:08:30.221 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-06-29T16:08:30.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.333+0000 7f1071b15700 1 Processor -- start 2022-06-29T16:08:30.339 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.333+0000 7f1071b15700 1 -- start start 2022-06-29T16:08:30.340 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.337+0000 7f1071b15700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f106c13a930 0x7f106c064d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:30.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.337+0000 7f1071b15700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f106c140ab0 0x7f106c140ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:30.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.337+0000 7f1071b15700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f106c0b29f0 con 0x7f106c05ced0 2022-06-29T16:08:30.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.337+0000 7f1071b15700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f106c0b24b0 con 0x7f106c140ab0 2022-06-29T16:08:30.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.337+0000 7f1071b15700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f106c0b2750 con 0x7f106c13a930 2022-06-29T16:08:30.341 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.337+0000 7f1071314700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f106c140ab0 0x7f106c140ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:30.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.337+0000 7f1071314700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f106c140ab0 0x7f106c140ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54758/0 (socket says 172.21.15.60:54758) 2022-06-29T16:08:30.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.337+0000 7f1071314700 1 -- 172.21.15.60:0/3733678927 learned_addr learned my addr 172.21.15.60:0/3733678927 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:30.342 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.337+0000 7f106bfff700 1 --2- 172.21.15.60:0/3733678927 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f106c13a930 0x7f106c064d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:30.344 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.337+0000 7f1071314700 1 -- 172.21.15.60:0/3733678927 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f106c13a930 msgr2=0x7f106c064d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:30.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.337+0000 7f1071314700 1 --2- 172.21.15.60:0/3733678927 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f106c13a930 0x7f106c064d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:30.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.337+0000 7f1071314700 1 -- 172.21.15.60:0/3733678927 >> v1:172.21.15.92:6789/0 conn(0x7f106c05ced0 legacy=0x7f106c13a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:30.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.337+0000 7f1071314700 1 -- 172.21.15.60:0/3733678927 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106c062e80 con 0x7f106c140ab0 2022-06-29T16:08:30.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.337+0000 7f1071314700 1 --2- 172.21.15.60:0/3733678927 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f106c140ab0 0x7f106c140ed0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 rx=0x7f10640032f0 tx=0x7f1064009290).ready entity=mon.0 client_cookie=b81095ddb17eb62b server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:30.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.337+0000 7f106b7fe700 1 -- 172.21.15.60:0/3733678927 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f106400b8a0 con 0x7f106c140ab0 2022-06-29T16:08:30.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.337+0000 7f106b7fe700 1 -- 172.21.15.60:0/3733678927 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106400ba00 con 0x7f106c140ab0 2022-06-29T16:08:30.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.337+0000 7f106b7fe700 1 -- 172.21.15.60:0/3733678927 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f106400bb70 con 0x7f106c140ab0 2022-06-29T16:08:30.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.337+0000 7f1071b15700 1 -- 172.21.15.60:0/3733678927 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f106c140ab0 msgr2=0x7f106c140ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:30.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.337+0000 7f1071b15700 1 --2- 172.21.15.60:0/3733678927 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f106c140ab0 0x7f106c140ed0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 rx=0x7f10640032f0 tx=0x7f1064009290).stop 2022-06-29T16:08:30.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.337+0000 7f1071b15700 1 -- 172.21.15.60:0/3733678927 shutdown_connections 2022-06-29T16:08:30.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.337+0000 7f1071b15700 1 --2- 172.21.15.60:0/3733678927 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f106c140ab0 0x7f106c140ed0 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:30.347 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.337+0000 7f1071b15700 1 --2- 172.21.15.60:0/3733678927 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f106c13a930 0x7f106c064d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:30.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.337+0000 7f1071b15700 1 -- 172.21.15.60:0/3733678927 >> 172.21.15.60:0/3733678927 conn(0x7f106c0b5840 msgr2=0x7f106c0b1810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:30.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.337+0000 7f1071b15700 1 -- 172.21.15.60:0/3733678927 shutdown_connections 2022-06-29T16:08:30.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.337+0000 7f1071b15700 1 -- 172.21.15.60:0/3733678927 wait complete. 2022-06-29T16:08:30.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.341+0000 7f1071b15700 1 Processor -- start 2022-06-29T16:08:30.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.341+0000 7f1071b15700 1 -- start start 2022-06-29T16:08:30.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.341+0000 7f1071b15700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f106c05ced0 0x7f106c13b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:30.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.341+0000 7f1071b15700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f106c13a930 0x7f106c136540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:30.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.341+0000 7f1071b15700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f106c140ab0 0x7f106c136a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:30.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.341+0000 7f1071b15700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106c13d450 con 0x7f106c140ab0 2022-06-29T16:08:30.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.341+0000 7f1071b15700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106c13d680 con 0x7f106c13a930 2022-06-29T16:08:30.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.341+0000 7f1071b15700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f106c0afdf0 con 0x7f106c05ced0 2022-06-29T16:08:30.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.341+0000 7f1071314700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f106c140ab0 0x7f106c136a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:30.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.341+0000 7f1071314700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f106c140ab0 0x7f106c136a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54764/0 (socket says 172.21.15.60:54764) 2022-06-29T16:08:30.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.341+0000 7f106bfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f106c13a930 0x7f106c136540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:30.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.341+0000 7f1071314700 1 -- 172.21.15.60:0/734063560 learned_addr learned my addr 172.21.15.60:0/734063560 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:30.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.341+0000 7f1070b13700 1 --2- 172.21.15.60:0/734063560 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f106c05ced0 0x7f106c13b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:30.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.341+0000 7f1071314700 1 -- 172.21.15.60:0/734063560 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f106c05ced0 msgr2=0x7f106c13b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:30.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.341+0000 7f1071314700 1 --2- 172.21.15.60:0/734063560 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f106c05ced0 0x7f106c13b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:30.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.341+0000 7f1071314700 1 -- 172.21.15.60:0/734063560 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f106c13a930 msgr2=0x7f106c136540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:30.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.341+0000 7f1071314700 1 --2- 172.21.15.60:0/734063560 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f106c13a930 0x7f106c136540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:30.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.341+0000 7f1071314700 1 -- 172.21.15.60:0/734063560 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1064009530 con 0x7f106c140ab0 2022-06-29T16:08:30.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.341+0000 7f1071314700 1 --2- 172.21.15.60:0/734063560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f106c140ab0 0x7f106c136a30 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 rx=0x7f10640032f0 tx=0x7f1064016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:30.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.341+0000 7f1069ffb700 1 -- 172.21.15.60:0/734063560 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f106400b8a0 con 0x7f106c140ab0 2022-06-29T16:08:30.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.341+0000 7f1069ffb700 1 -- 172.21.15.60:0/734063560 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1064021440 con 0x7f106c140ab0 2022-06-29T16:08:30.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.341+0000 7f1071b15700 1 -- 172.21.15.60:0/734063560 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f106c136f20 con 0x7f106c140ab0 2022-06-29T16:08:30.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.341+0000 7f1069ffb700 1 -- 172.21.15.60:0/734063560 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10640217a0 con 0x7f106c140ab0 2022-06-29T16:08:30.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.341+0000 7f1071b15700 1 -- 172.21.15.60:0/734063560 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f106c067510 con 0x7f106c140ab0 2022-06-29T16:08:30.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.345+0000 7f1071b15700 1 -- 172.21.15.60:0/734063560 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f106c13d840 con 0x7f106c140ab0 2022-06-29T16:08:30.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.345+0000 7f1069ffb700 1 -- 172.21.15.60:0/734063560 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f1064029070 con 0x7f106c140ab0 2022-06-29T16:08:30.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.345+0000 7f1069ffb700 1 --2- 172.21.15.60:0/734063560 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1050053b30 0x7f1050055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:30.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.345+0000 7f1069ffb700 1 -- 172.21.15.60:0/734063560 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f106406cef0 con 0x7f106c140ab0 2022-06-29T16:08:30.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.345+0000 7f1070b13700 1 --2- 172.21.15.60:0/734063560 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1050053b30 0x7f1050055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:30.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.345+0000 7f1070b13700 1 --2- 172.21.15.60:0/734063560 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1050053b30 0x7f1050055ff0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 rx=0x7f105400d000 tx=0x7f1054009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:30.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.353+0000 7f1069ffb700 1 -- 172.21.15.60:0/734063560 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1064043ec0 con 0x7f106c140ab0 2022-06-29T16:08:30.487 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:30.481+0000 7f1071b15700 1 -- 172.21.15.60:0/734063560 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f106c13e060 con 0x7f106c140ab0 2022-06-29T16:08:31.210 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.205+0000 7f1069ffb700 1 -- 172.21.15.60:0/734063560 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v149) v1 ==== 185+0+0 (secure 0 0 0) 0x7f1064015020 con 0x7f106c140ab0 2022-06-29T16:08:31.213 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.209+0000 7f1071b15700 1 -- 172.21.15.60:0/734063560 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1050053b30 msgr2=0x7f1050055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:31.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.209+0000 7f1071b15700 1 --2- 172.21.15.60:0/734063560 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1050053b30 0x7f1050055ff0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 rx=0x7f105400d000 tx=0x7f1054009040).stop 2022-06-29T16:08:31.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.209+0000 7f1071b15700 1 -- 172.21.15.60:0/734063560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f106c140ab0 msgr2=0x7f106c136a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:31.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.209+0000 7f1071b15700 1 --2- 172.21.15.60:0/734063560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f106c140ab0 0x7f106c136a30 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 rx=0x7f10640032f0 tx=0x7f1064016000).stop 2022-06-29T16:08:31.214 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.209+0000 7f1071b15700 1 -- 172.21.15.60:0/734063560 shutdown_connections 2022-06-29T16:08:31.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.209+0000 7f1071b15700 1 --2- 172.21.15.60:0/734063560 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f1050053b30 0x7f1050055ff0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:31.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.209+0000 7f1071b15700 1 --2- 172.21.15.60:0/734063560 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f106c140ab0 0x7f106c136a30 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:31.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.209+0000 7f1071b15700 1 --2- 172.21.15.60:0/734063560 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f106c13a930 0x7f106c136540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:31.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.209+0000 7f1071b15700 1 --2- 172.21.15.60:0/734063560 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f106c05ced0 0x7f106c13b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:31.216 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.209+0000 7f1071b15700 1 -- 172.21.15.60:0/734063560 >> 172.21.15.60:0/734063560 conn(0x7f106c0b5840 msgr2=0x7f106c1395f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:31.216 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.209+0000 7f1071b15700 1 -- 172.21.15.60:0/734063560 shutdown_connections 2022-06-29T16:08:31.216 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.209+0000 7f1071b15700 1 -- 172.21.15.60:0/734063560 wait complete. 2022-06-29T16:08:31.217 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2022-06-29T16:08:31.230 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-06-29T16:08:31.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.341+0000 7f5bd88fd700 1 Processor -- start 2022-06-29T16:08:31.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.341+0000 7f5bd88fd700 1 -- start start 2022-06-29T16:08:31.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.341+0000 7f5bd88fd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5bd405c040 0x7f5bd4142670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:31.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.341+0000 7f5bd88fd700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5bd40ae190 0x7f5bd40ae600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:31.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.341+0000 7f5bd88fd700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd40afbb0 con 0x7f5bd405fb00 2022-06-29T16:08:31.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.341+0000 7f5bd88fd700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bd4067de0 con 0x7f5bd405c040 2022-06-29T16:08:31.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.341+0000 7f5bd88fd700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bd405fee0 con 0x7f5bd40ae190 2022-06-29T16:08:31.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.341+0000 7f5bd2ffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5bd405c040 0x7f5bd4142670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:31.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.341+0000 7f5bd2ffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5bd405c040 0x7f5bd4142670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54770/0 (socket says 172.21.15.60:54770) 2022-06-29T16:08:31.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.341+0000 7f5bd2ffd700 1 -- 172.21.15.60:0/1338160306 learned_addr learned my addr 172.21.15.60:0/1338160306 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:31.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.341+0000 7f5bd3fff700 1 --2- 172.21.15.60:0/1338160306 >> v2:172.21.15.92:3300/0 conn(0x7f5bd40ae190 0x7f5bd40ae600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:31.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.341+0000 7f5bd2ffd700 1 -- 172.21.15.60:0/1338160306 >> v2:172.21.15.92:3300/0 conn(0x7f5bd40ae190 msgr2=0x7f5bd40ae600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T16:08:31.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.341+0000 7f5bd2ffd700 1 --2- 172.21.15.60:0/1338160306 >> v2:172.21.15.92:3300/0 conn(0x7f5bd40ae190 0x7f5bd40ae600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:31.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.341+0000 7f5bd27fc700 1 -- 172.21.15.60:0/1338160306 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2419973638 0 0) 0x7f5bd40afbb0 con 0x7f5bd405fb00 2022-06-29T16:08:31.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.345+0000 7f5bd2ffd700 1 -- 172.21.15.60:0/1338160306 >> v1:172.21.15.92:6789/0 conn(0x7f5bd405fb00 legacy=0x7f5bd4141f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:31.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.345+0000 7f5bd2ffd700 1 -- 172.21.15.60:0/1338160306 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd4146c80 con 0x7f5bd405c040 2022-06-29T16:08:31.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.345+0000 7f5bd2ffd700 1 --2- 172.21.15.60:0/1338160306 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5bd405c040 0x7f5bd4142670 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 rx=0x7f5bc8002f70 tx=0x7f5bc8006eb0).ready entity=mon.0 client_cookie=54459ba5fac64d06 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:31.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.345+0000 7f5bd27fc700 1 -- 172.21.15.60:0/1338160306 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bc800d610 con 0x7f5bd405c040 2022-06-29T16:08:31.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.345+0000 7f5bd27fc700 1 -- 172.21.15.60:0/1338160306 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc8000e90 con 0x7f5bd405c040 2022-06-29T16:08:31.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.345+0000 7f5bd27fc700 1 -- 172.21.15.60:0/1338160306 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bc800d960 con 0x7f5bd405c040 2022-06-29T16:08:31.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.345+0000 7f5bd88fd700 1 -- 172.21.15.60:0/1338160306 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5bd405c040 msgr2=0x7f5bd4142670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:31.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.345+0000 7f5bd88fd700 1 --2- 172.21.15.60:0/1338160306 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5bd405c040 0x7f5bd4142670 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 rx=0x7f5bc8002f70 tx=0x7f5bc8006eb0).stop 2022-06-29T16:08:31.356 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.345+0000 7f5bd88fd700 1 -- 172.21.15.60:0/1338160306 shutdown_connections 2022-06-29T16:08:31.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.345+0000 7f5bd88fd700 1 --2- 172.21.15.60:0/1338160306 >> v2:172.21.15.92:3300/0 conn(0x7f5bd40ae190 0x7f5bd40ae600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:31.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.345+0000 7f5bd88fd700 1 --2- 172.21.15.60:0/1338160306 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5bd405c040 0x7f5bd4142670 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:31.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.345+0000 7f5bd88fd700 1 -- 172.21.15.60:0/1338160306 >> 172.21.15.60:0/1338160306 conn(0x7f5bd4063a60 msgr2=0x7f5bd405f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:31.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.345+0000 7f5bd88fd700 1 -- 172.21.15.60:0/1338160306 shutdown_connections 2022-06-29T16:08:31.357 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.345+0000 7f5bd88fd700 1 -- 172.21.15.60:0/1338160306 wait complete. 2022-06-29T16:08:31.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.345+0000 7f5bd88fd700 1 Processor -- start 2022-06-29T16:08:31.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.349+0000 7f5bd88fd700 1 -- start start 2022-06-29T16:08:31.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.349+0000 7f5bd88fd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5bd405c040 0x7f5bd4154f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:31.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.349+0000 7f5bd88fd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5bd405fb00 0x7f5bd4145330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:31.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.349+0000 7f5bd88fd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5bd40ae190 0x7f5bd40b3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:31.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.349+0000 7f5bd88fd700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd41412a0 con 0x7f5bd405fb00 2022-06-29T16:08:31.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.349+0000 7f5bd88fd700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd4064b70 con 0x7f5bd405c040 2022-06-29T16:08:31.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.349+0000 7f5bd88fd700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd4065a70 con 0x7f5bd40ae190 2022-06-29T16:08:31.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.349+0000 7f5bd2ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5bd405fb00 0x7f5bd4145330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:31.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.349+0000 7f5bd2ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5bd405fb00 0x7f5bd4145330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54776/0 (socket says 172.21.15.60:54776) 2022-06-29T16:08:31.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.349+0000 7f5bd2ffd700 1 -- 172.21.15.60:0/732002162 learned_addr learned my addr 172.21.15.60:0/732002162 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T16:08:31.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.349+0000 7f5bd37fe700 1 --2- 172.21.15.60:0/732002162 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5bd405c040 0x7f5bd4154f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:31.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.349+0000 7f5bd3fff700 1 --2- 172.21.15.60:0/732002162 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5bd40ae190 0x7f5bd40b3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:31.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.349+0000 7f5bd2ffd700 1 -- 172.21.15.60:0/732002162 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5bd40ae190 msgr2=0x7f5bd40b3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:31.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.349+0000 7f5bd2ffd700 1 --2- 172.21.15.60:0/732002162 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5bd40ae190 0x7f5bd40b3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:31.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.349+0000 7f5bd2ffd700 1 -- 172.21.15.60:0/732002162 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5bd405c040 msgr2=0x7f5bd4154f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:31.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.349+0000 7f5bd2ffd700 1 --2- 172.21.15.60:0/732002162 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5bd405c040 0x7f5bd4154f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:31.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.349+0000 7f5bd2ffd700 1 -- 172.21.15.60:0/732002162 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc800b1e0 con 0x7f5bd405fb00 2022-06-29T16:08:31.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.349+0000 7f5bd3fff700 1 --2- 172.21.15.60:0/732002162 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5bd40ae190 0x7f5bd40b3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T16:08:31.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.349+0000 7f5bd2ffd700 1 --2- 172.21.15.60:0/732002162 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5bd405fb00 0x7f5bd4145330 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 rx=0x7f5bc8002f70 tx=0x7f5bc8015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:31.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.349+0000 7f5bd0ff9700 1 -- 172.21.15.60:0/732002162 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bc800d290 con 0x7f5bd405fb00 2022-06-29T16:08:31.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.349+0000 7f5bd0ff9700 1 -- 172.21.15.60:0/732002162 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc800de90 con 0x7f5bd405fb00 2022-06-29T16:08:31.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.349+0000 7f5bd88fd700 1 -- 172.21.15.60:0/732002162 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bd4145bc0 con 0x7f5bd405fb00 2022-06-29T16:08:31.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.349+0000 7f5bd0ff9700 1 -- 172.21.15.60:0/732002162 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bc8011e20 con 0x7f5bd405fb00 2022-06-29T16:08:31.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.349+0000 7f5bd88fd700 1 -- 172.21.15.60:0/732002162 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bd4140b20 con 0x7f5bd405fb00 2022-06-29T16:08:31.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.353+0000 7f5bd88fd700 1 -- 172.21.15.60:0/732002162 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bd41410c0 con 0x7f5bd405fb00 2022-06-29T16:08:31.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.353+0000 7f5bd0ff9700 1 -- 172.21.15.60:0/732002162 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5bc8025030 con 0x7f5bd405fb00 2022-06-29T16:08:31.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.353+0000 7f5bd0ff9700 1 --2- 172.21.15.60:0/732002162 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5bb4053a70 0x7f5bb4055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T16:08:31.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.353+0000 7f5bd0ff9700 1 -- 172.21.15.60:0/732002162 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f5bc806d0c0 con 0x7f5bd405fb00 2022-06-29T16:08:31.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.353+0000 7f5bd37fe700 1 --2- 172.21.15.60:0/732002162 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5bb4053a70 0x7f5bb4055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T16:08:31.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.357+0000 7f5bd37fe700 1 --2- 172.21.15.60:0/732002162 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5bb4053a70 0x7f5bb4055f30 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 rx=0x7f5bc400d000 tx=0x7f5bc4007d80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T16:08:31.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.361+0000 7f5bd0ff9700 1 -- 172.21.15.60:0/732002162 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5bc80440a0 con 0x7f5bd405fb00 2022-06-29T16:08:31.493 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:31.489+0000 7f5bd88fd700 1 -- 172.21.15.60:0/732002162 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5bd40573d0 con 0x7f5bd405fb00 2022-06-29T16:08:32.215 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:32.209+0000 7f5bd0ff9700 1 -- 172.21.15.60:0/732002162 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v150) v1 ==== 173+0+0 (secure 0 0 0) 0x7f5bc80230d0 con 0x7f5bd405fb00 2022-06-29T16:08:32.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:32.213+0000 7f5bd88fd700 1 -- 172.21.15.60:0/732002162 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5bb4053a70 msgr2=0x7f5bb4055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:32.219 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:32.213+0000 7f5bd88fd700 1 --2- 172.21.15.60:0/732002162 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5bb4053a70 0x7f5bb4055f30 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 rx=0x7f5bc400d000 tx=0x7f5bc4007d80).stop 2022-06-29T16:08:32.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:32.213+0000 7f5bd88fd700 1 -- 172.21.15.60:0/732002162 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5bd405fb00 msgr2=0x7f5bd4145330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T16:08:32.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:32.213+0000 7f5bd88fd700 1 --2- 172.21.15.60:0/732002162 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5bd405fb00 0x7f5bd4145330 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 rx=0x7f5bc8002f70 tx=0x7f5bc8015000).stop 2022-06-29T16:08:32.220 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:32.213+0000 7f5bd88fd700 1 -- 172.21.15.60:0/732002162 shutdown_connections 2022-06-29T16:08:32.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:32.213+0000 7f5bd88fd700 1 --2- 172.21.15.60:0/732002162 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5bb4053a70 0x7f5bb4055f30 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:32.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:32.213+0000 7f5bd88fd700 1 --2- 172.21.15.60:0/732002162 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5bd40ae190 0x7f5bd40b3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:32.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:32.213+0000 7f5bd88fd700 1 --2- 172.21.15.60:0/732002162 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5bd405fb00 0x7f5bd4145330 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:32.221 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:32.213+0000 7f5bd88fd700 1 --2- 172.21.15.60:0/732002162 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5bd405c040 0x7f5bd4154f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T16:08:32.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:32.213+0000 7f5bd88fd700 1 -- 172.21.15.60:0/732002162 >> 172.21.15.60:0/732002162 conn(0x7f5bd4063a60 msgr2=0x7f5bd4149490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T16:08:32.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:32.213+0000 7f5bd88fd700 1 -- 172.21.15.60:0/732002162 shutdown_connections 2022-06-29T16:08:32.222 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T16:08:32.213+0000 7f5bd88fd700 1 -- 172.21.15.60:0/732002162 wait complete. 2022-06-29T16:08:32.223 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2022-06-29T16:08:32.234 INFO:teuthology.orchestra.run:Running command with timeout 60 2022-06-29T16:08:32.235 DEBUG:teuthology.orchestra.run.smithi121:> cd /home/ubuntu/cephtest/mnt.0 && sudo bash -c 'stat -f .' 2022-06-29T16:08:59.901 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:08:59.904 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:08:59.907 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:08:59.910 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:08:59.913 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:09:02.338 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:09:02.339 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:09:02.339 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:09:29.964 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:09:29.967 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:09:29.970 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:09:29.972 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:09:29.975 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:09:32.241 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-06-29T16:09:32.244 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2022-06-29T16:10:00.027 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:10:00.031 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:10:00.033 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:10:00.037 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:10:00.040 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:10:30.097 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:10:30.101 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:10:30.103 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:10:30.106 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:10:30.109 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:11:00.164 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:11:00.169 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:11:00.172 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:11:00.175 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:11:00.178 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:11:30.232 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:11:30.236 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:11:30.239 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:11:30.242 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:11:30.244 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:12:00.312 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:12:00.316 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:12:00.319 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:12:00.321 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:12:00.324 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:12:30.411 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:12:30.414 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:12:30.417 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:12:30.420 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:12:30.422 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:13:00.496 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:13:00.499 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:13:00.502 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:13:00.504 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:13:00.507 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:13:30.554 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:13:30.558 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:13:30.560 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:13:30.563 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:13:30.565 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:14:00.613 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:14:00.616 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:14:00.619 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:14:00.621 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:14:00.624 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:14:30.697 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:14:30.700 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:14:30.703 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:14:30.705 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:14:30.708 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:15:00.759 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:15:00.763 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:15:00.765 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:15:00.768 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:15:00.771 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:15:30.827 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:15:30.830 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:15:30.833 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:15:30.836 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:15:30.839 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:16:00.908 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:16:00.912 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:16:00.914 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:16:00.917 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:16:00.919 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:16:31.004 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:16:31.007 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:16:31.010 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:16:31.013 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:16:31.015 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:17:01.065 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:17:01.069 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:17:01.071 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:17:01.074 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:17:01.077 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:17:31.161 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:17:31.164 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:17:31.167 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:17:31.170 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:17:31.172 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:18:01.246 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:18:01.249 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:18:01.252 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:18:01.254 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:18:01.257 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:18:31.308 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:18:31.312 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:18:31.315 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:18:31.318 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:18:31.321 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:19:01.383 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:19:01.387 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:19:01.390 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:19:01.392 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:19:01.395 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:19:31.458 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:19:31.462 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:19:31.465 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:19:31.467 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:19:31.470 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:20:01.522 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:20:01.526 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:20:01.530 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:20:01.534 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:20:01.537 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:20:31.589 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:20:31.593 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:20:31.595 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:20:31.598 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:20:31.601 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:21:01.715 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:21:01.719 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:21:01.722 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:21:01.725 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:21:01.728 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:21:31.824 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:21:31.827 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:21:31.829 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:21:31.832 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:21:31.835 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:22:01.916 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:22:01.920 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:22:01.922 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:22:01.925 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:22:01.927 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:22:32.010 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:22:32.014 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:22:32.017 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:22:32.019 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:22:32.022 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:23:02.099 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:23:02.103 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:23:02.105 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:23:02.114 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:23:02.116 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:23:32.170 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:23:32.173 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:23:32.177 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:23:32.179 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:23:32.182 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:24:02.239 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:24:02.242 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:24:02.246 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:24:02.249 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:24:02.253 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:24:02.344 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-06-29T16:24:02.347 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2022-06-29T16:24:02.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:24:02.355 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-06-29T16:24:02.356 DEBUG:teuthology.orchestra.run.smithi060:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:24:02.378 DEBUG:teuthology.orchestra.run.smithi092:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:24:02.400 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:24:02.420 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:24:02.442 DEBUG:teuthology.orchestra.run.smithi197:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-06-29T16:24:02.464 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-06-29T16:24:02.465 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:24:02.465 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:24:02.466 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:24:32.304 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:24:32.308 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:24:32.311 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:24:32.313 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:24:32.316 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:25:02.396 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:25:02.401 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:25:02.403 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:25:02.406 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:25:02.409 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:25:32.504 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:25:32.507 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:25:32.510 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:25:32.513 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:25:32.515 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:26:02.594 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:26:02.597 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:26:02.600 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:26:02.603 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:26:02.605 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:26:32.691 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:26:32.695 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:26:32.698 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:26:32.701 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:26:32.704 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:27:02.757 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:27:02.761 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:27:02.764 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:27:02.766 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:27:02.769 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:27:32.836 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:27:32.840 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:27:32.843 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:27:32.845 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:27:32.848 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:28:02.922 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:28:02.925 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:28:02.928 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:28:02.931 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:28:02.933 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:28:32.999 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:28:33.003 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:28:33.005 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:28:33.008 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:28:33.011 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:29:03.071 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:29:03.075 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:29:03.077 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:29:03.080 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:29:03.082 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:29:33.159 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:29:33.162 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:29:33.165 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:29:33.168 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:29:33.170 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:30:03.226 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:30:03.229 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:30:03.232 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:30:03.235 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:30:03.237 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:30:33.287 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:30:33.291 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:30:33.294 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:30:33.297 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:30:33.300 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:31:03.355 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:31:03.359 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:31:03.362 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:31:03.365 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:31:03.367 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:31:33.422 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:31:33.426 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:31:33.429 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:31:33.431 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:31:33.434 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:32:03.489 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:32:03.492 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:32:03.495 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:32:03.498 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:32:03.500 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:32:33.584 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:32:33.588 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:32:33.591 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:32:33.594 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:32:33.596 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:33:03.702 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:33:03.705 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:33:03.708 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:33:03.711 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:33:03.713 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:33:33.775 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:33:33.778 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:33:33.780 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:33:33.783 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:33:33.786 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:34:03.841 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:34:03.845 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:34:03.848 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:34:03.851 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:34:03.854 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:34:33.904 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:34:33.907 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:34:33.910 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:34:33.913 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:34:33.916 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:35:04.041 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:35:04.045 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:35:04.047 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:35:04.050 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:35:04.052 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:35:34.112 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:35:34.115 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:35:34.118 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:35:34.121 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:35:34.123 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:36:04.258 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:36:04.263 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:36:04.266 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:36:04.269 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:36:04.272 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:36:34.329 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:36:34.333 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:36:34.336 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:36:34.339 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:36:34.341 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:37:04.484 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:37:04.488 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:37:04.491 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:37:04.493 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:37:04.496 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:37:34.621 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:37:34.624 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:37:34.627 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:37:34.630 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:37:34.632 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:38:04.707 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:38:04.711 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:38:04.713 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:38:04.716 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:38:04.719 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:38:34.846 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:38:34.850 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:38:34.853 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:38:34.857 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:38:34.860 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:39:02.471 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-06-29T16:39:02.472 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2022-06-29T16:39:02.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:39:02.480 INFO:tasks.cephfs_test_runner:test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) ... ERROR 2022-06-29T16:39:02.480 INFO:tasks.cephfs_test_runner:ERROR 2022-06-29T16:39:02.481 INFO:tasks.cephfs_test_runner: 2022-06-29T16:39:02.481 INFO:tasks.cephfs_test_runner:====================================================================== 2022-06-29T16:39:02.482 INFO:tasks.cephfs_test_runner:ERROR: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-06-29T16:39:02.482 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2022-06-29T16:39:02.482 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-06-29T16:39:02.483 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr 2022-06-29T16:39:02.483 INFO:tasks.cephfs_test_runner: out = self.in_stderr_buffer.read(nbytes, self.timeout) 2022-06-29T16:39:02.483 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read 2022-06-29T16:39:02.484 INFO:tasks.cephfs_test_runner: raise PipeTimeout() 2022-06-29T16:39:02.484 INFO:tasks.cephfs_test_runner:paramiko.buffered_pipe.PipeTimeout 2022-06-29T16:39:02.485 INFO:tasks.cephfs_test_runner: 2022-06-29T16:39:02.485 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2022-06-29T16:39:02.486 INFO:tasks.cephfs_test_runner: 2022-06-29T16:39:02.486 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-06-29T16:39:02.486 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_f25ffac72e5a9e63860ff7deb6767a11f0559a16/qa/tasks/cephfs/test_misc.py", line 28, in test_statfs_on_deleted_fs 2022-06-29T16:39:02.486 INFO:tasks.cephfs_test_runner: self.mount_a.umount_wait(force=True) 2022-06-29T16:39:02.487 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_f25ffac72e5a9e63860ff7deb6767a11f0559a16/qa/tasks/cephfs/fuse_mount.py", line 326, in umount_wait 2022-06-29T16:39:02.487 INFO:tasks.cephfs_test_runner: if not (self.is_mounted() and self.fuse_daemon): 2022-06-29T16:39:02.487 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_f25ffac72e5a9e63860ff7deb6767a11f0559a16/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted 2022-06-29T16:39:02.488 INFO:tasks.cephfs_test_runner: proc.wait() 2022-06-29T16:39:02.488 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 148, in wait 2022-06-29T16:39:02.488 INFO:tasks.cephfs_test_runner: greenlet.get(block=True,timeout=60) 2022-06-29T16:39:02.489 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get 2022-06-29T16:39:02.489 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception 2022-06-29T16:39:02.489 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise 2022-06-29T16:39:02.490 INFO:tasks.cephfs_test_runner: raise value.with_traceback(tb) 2022-06-29T16:39:02.490 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run 2022-06-29T16:39:02.490 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 323, in copy_file_to 2022-06-29T16:39:02.491 INFO:tasks.cephfs_test_runner: copy_to_log(src, logger, capture=stream, quiet=quiet) 2022-06-29T16:39:02.491 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 276, in copy_to_log 2022-06-29T16:39:02.491 INFO:tasks.cephfs_test_runner: for line in f: 2022-06-29T16:39:02.491 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ 2022-06-29T16:39:02.492 INFO:tasks.cephfs_test_runner: line = self.readline() 2022-06-29T16:39:02.492 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline 2022-06-29T16:39:02.492 INFO:tasks.cephfs_test_runner: new_data = self._read(n) 2022-06-29T16:39:02.493 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read 2022-06-29T16:39:02.493 INFO:tasks.cephfs_test_runner: return self.channel.recv_stderr(size) 2022-06-29T16:39:02.493 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr 2022-06-29T16:39:02.494 INFO:tasks.cephfs_test_runner: raise socket.timeout() 2022-06-29T16:39:02.494 INFO:tasks.cephfs_test_runner:socket.timeout 2022-06-29T16:39:02.494 INFO:tasks.cephfs_test_runner: 2022-06-29T16:39:02.495 INFO:tasks.cephfs_test_runner:====================================================================== 2022-06-29T16:39:02.495 INFO:tasks.cephfs_test_runner:ERROR: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-06-29T16:39:02.495 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2022-06-29T16:39:02.496 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-06-29T16:39:02.496 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr 2022-06-29T16:39:02.496 INFO:tasks.cephfs_test_runner: out = self.in_stderr_buffer.read(nbytes, self.timeout) 2022-06-29T16:39:02.497 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read 2022-06-29T16:39:02.497 INFO:tasks.cephfs_test_runner: raise PipeTimeout() 2022-06-29T16:39:02.497 INFO:tasks.cephfs_test_runner:paramiko.buffered_pipe.PipeTimeout 2022-06-29T16:39:02.498 INFO:tasks.cephfs_test_runner: 2022-06-29T16:39:02.498 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2022-06-29T16:39:02.498 INFO:tasks.cephfs_test_runner: 2022-06-29T16:39:02.499 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-06-29T16:39:02.499 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_f25ffac72e5a9e63860ff7deb6767a11f0559a16/qa/tasks/cephfs/cephfs_test_case.py", line 175, in tearDown 2022-06-29T16:39:02.499 INFO:tasks.cephfs_test_runner: m.teardown() 2022-06-29T16:39:02.500 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_f25ffac72e5a9e63860ff7deb6767a11f0559a16/qa/tasks/cephfs/fuse_mount.py", line 408, in teardown 2022-06-29T16:39:02.500 INFO:tasks.cephfs_test_runner: self.umount() 2022-06-29T16:39:02.500 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_f25ffac72e5a9e63860ff7deb6767a11f0559a16/qa/tasks/cephfs/fuse_mount.py", line 261, in umount 2022-06-29T16:39:02.500 INFO:tasks.cephfs_test_runner: if not self.is_mounted(): 2022-06-29T16:39:02.501 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_f25ffac72e5a9e63860ff7deb6767a11f0559a16/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted 2022-06-29T16:39:02.501 INFO:tasks.cephfs_test_runner: proc.wait() 2022-06-29T16:39:02.501 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 148, in wait 2022-06-29T16:39:02.501 INFO:tasks.cephfs_test_runner: greenlet.get(block=True,timeout=60) 2022-06-29T16:39:02.502 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get 2022-06-29T16:39:02.502 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception 2022-06-29T16:39:02.502 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise 2022-06-29T16:39:02.503 INFO:tasks.cephfs_test_runner: raise value.with_traceback(tb) 2022-06-29T16:39:02.503 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run 2022-06-29T16:39:02.503 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 323, in copy_file_to 2022-06-29T16:39:02.504 INFO:tasks.cephfs_test_runner: copy_to_log(src, logger, capture=stream, quiet=quiet) 2022-06-29T16:39:02.504 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 276, in copy_to_log 2022-06-29T16:39:02.504 INFO:tasks.cephfs_test_runner: for line in f: 2022-06-29T16:39:02.504 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ 2022-06-29T16:39:02.505 INFO:tasks.cephfs_test_runner: line = self.readline() 2022-06-29T16:39:02.505 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline 2022-06-29T16:39:02.505 INFO:tasks.cephfs_test_runner: new_data = self._read(n) 2022-06-29T16:39:02.506 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read 2022-06-29T16:39:02.506 INFO:tasks.cephfs_test_runner: return self.channel.recv_stderr(size) 2022-06-29T16:39:02.507 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr 2022-06-29T16:39:02.507 INFO:tasks.cephfs_test_runner: raise socket.timeout() 2022-06-29T16:39:02.507 INFO:tasks.cephfs_test_runner:socket.timeout 2022-06-29T16:39:02.508 INFO:tasks.cephfs_test_runner: 2022-06-29T16:39:02.508 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2022-06-29T16:39:02.508 INFO:tasks.cephfs_test_runner:Ran 12 tests in 2410.795s 2022-06-29T16:39:02.509 INFO:tasks.cephfs_test_runner: 2022-06-29T16:39:02.509 INFO:tasks.cephfs_test_runner:FAILED (errors=2) 2022-06-29T16:39:02.509 INFO:tasks.cephfs_test_runner: 2022-06-29T16:39:02.510 INFO:tasks.cephfs_test_runner:====================================================================== 2022-06-29T16:39:02.510 INFO:tasks.cephfs_test_runner:ERROR: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-06-29T16:39:02.510 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2022-06-29T16:39:02.511 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-06-29T16:39:02.511 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr 2022-06-29T16:39:02.511 INFO:tasks.cephfs_test_runner: out = self.in_stderr_buffer.read(nbytes, self.timeout) 2022-06-29T16:39:02.512 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read 2022-06-29T16:39:02.512 INFO:tasks.cephfs_test_runner: raise PipeTimeout() 2022-06-29T16:39:02.512 INFO:tasks.cephfs_test_runner:paramiko.buffered_pipe.PipeTimeout 2022-06-29T16:39:02.513 INFO:tasks.cephfs_test_runner: 2022-06-29T16:39:02.513 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2022-06-29T16:39:02.513 INFO:tasks.cephfs_test_runner: 2022-06-29T16:39:02.514 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-06-29T16:39:02.514 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_f25ffac72e5a9e63860ff7deb6767a11f0559a16/qa/tasks/cephfs/test_misc.py", line 28, in test_statfs_on_deleted_fs 2022-06-29T16:39:02.514 INFO:tasks.cephfs_test_runner: self.mount_a.umount_wait(force=True) 2022-06-29T16:39:02.514 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_f25ffac72e5a9e63860ff7deb6767a11f0559a16/qa/tasks/cephfs/fuse_mount.py", line 326, in umount_wait 2022-06-29T16:39:02.515 INFO:tasks.cephfs_test_runner: if not (self.is_mounted() and self.fuse_daemon): 2022-06-29T16:39:02.515 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_f25ffac72e5a9e63860ff7deb6767a11f0559a16/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted 2022-06-29T16:39:02.515 INFO:tasks.cephfs_test_runner: proc.wait() 2022-06-29T16:39:02.516 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 148, in wait 2022-06-29T16:39:02.516 INFO:tasks.cephfs_test_runner: greenlet.get(block=True,timeout=60) 2022-06-29T16:39:02.516 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get 2022-06-29T16:39:02.517 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception 2022-06-29T16:39:02.517 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise 2022-06-29T16:39:02.517 INFO:tasks.cephfs_test_runner: raise value.with_traceback(tb) 2022-06-29T16:39:02.518 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run 2022-06-29T16:39:02.518 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 323, in copy_file_to 2022-06-29T16:39:02.518 INFO:tasks.cephfs_test_runner: copy_to_log(src, logger, capture=stream, quiet=quiet) 2022-06-29T16:39:02.519 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 276, in copy_to_log 2022-06-29T16:39:02.519 INFO:tasks.cephfs_test_runner: for line in f: 2022-06-29T16:39:02.519 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ 2022-06-29T16:39:02.520 INFO:tasks.cephfs_test_runner: line = self.readline() 2022-06-29T16:39:02.520 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline 2022-06-29T16:39:02.520 INFO:tasks.cephfs_test_runner: new_data = self._read(n) 2022-06-29T16:39:02.520 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read 2022-06-29T16:39:02.521 INFO:tasks.cephfs_test_runner: return self.channel.recv_stderr(size) 2022-06-29T16:39:02.521 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr 2022-06-29T16:39:02.521 INFO:tasks.cephfs_test_runner: raise socket.timeout() 2022-06-29T16:39:02.522 INFO:tasks.cephfs_test_runner:socket.timeout 2022-06-29T16:39:02.522 INFO:tasks.cephfs_test_runner: 2022-06-29T16:39:02.522 INFO:tasks.cephfs_test_runner:====================================================================== 2022-06-29T16:39:02.523 INFO:tasks.cephfs_test_runner:ERROR: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-06-29T16:39:02.523 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2022-06-29T16:39:02.523 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-06-29T16:39:02.524 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr 2022-06-29T16:39:02.524 INFO:tasks.cephfs_test_runner: out = self.in_stderr_buffer.read(nbytes, self.timeout) 2022-06-29T16:39:02.524 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read 2022-06-29T16:39:02.525 INFO:tasks.cephfs_test_runner: raise PipeTimeout() 2022-06-29T16:39:02.525 INFO:tasks.cephfs_test_runner:paramiko.buffered_pipe.PipeTimeout 2022-06-29T16:39:02.525 INFO:tasks.cephfs_test_runner: 2022-06-29T16:39:02.525 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2022-06-29T16:39:02.526 INFO:tasks.cephfs_test_runner: 2022-06-29T16:39:02.526 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-06-29T16:39:02.526 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_f25ffac72e5a9e63860ff7deb6767a11f0559a16/qa/tasks/cephfs/cephfs_test_case.py", line 175, in tearDown 2022-06-29T16:39:02.527 INFO:tasks.cephfs_test_runner: m.teardown() 2022-06-29T16:39:02.527 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_f25ffac72e5a9e63860ff7deb6767a11f0559a16/qa/tasks/cephfs/fuse_mount.py", line 408, in teardown 2022-06-29T16:39:02.527 INFO:tasks.cephfs_test_runner: self.umount() 2022-06-29T16:39:02.528 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_f25ffac72e5a9e63860ff7deb6767a11f0559a16/qa/tasks/cephfs/fuse_mount.py", line 261, in umount 2022-06-29T16:39:02.528 INFO:tasks.cephfs_test_runner: if not self.is_mounted(): 2022-06-29T16:39:02.528 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_f25ffac72e5a9e63860ff7deb6767a11f0559a16/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted 2022-06-29T16:39:02.529 INFO:tasks.cephfs_test_runner: proc.wait() 2022-06-29T16:39:02.529 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 148, in wait 2022-06-29T16:39:02.529 INFO:tasks.cephfs_test_runner: greenlet.get(block=True,timeout=60) 2022-06-29T16:39:02.530 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get 2022-06-29T16:39:02.530 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception 2022-06-29T16:39:02.530 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise 2022-06-29T16:39:02.531 INFO:tasks.cephfs_test_runner: raise value.with_traceback(tb) 2022-06-29T16:39:02.531 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run 2022-06-29T16:39:02.531 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 323, in copy_file_to 2022-06-29T16:39:02.531 INFO:tasks.cephfs_test_runner: copy_to_log(src, logger, capture=stream, quiet=quiet) 2022-06-29T16:39:02.532 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 276, in copy_to_log 2022-06-29T16:39:02.532 INFO:tasks.cephfs_test_runner: for line in f: 2022-06-29T16:39:02.532 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ 2022-06-29T16:39:02.533 INFO:tasks.cephfs_test_runner: line = self.readline() 2022-06-29T16:39:02.533 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline 2022-06-29T16:39:02.533 INFO:tasks.cephfs_test_runner: new_data = self._read(n) 2022-06-29T16:39:02.534 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read 2022-06-29T16:39:02.534 INFO:tasks.cephfs_test_runner: return self.channel.recv_stderr(size) 2022-06-29T16:39:02.534 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr 2022-06-29T16:39:02.534 INFO:tasks.cephfs_test_runner: raise socket.timeout() 2022-06-29T16:39:02.535 INFO:tasks.cephfs_test_runner:socket.timeout 2022-06-29T16:39:02.535 INFO:tasks.cephfs_test_runner: 2022-06-29T16:39:02.536 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/run_tasks.py", line 106, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_f25ffac72e5a9e63860ff7deb6767a11f0559a16/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc), test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-06-29T16:39:02.672 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=56425ddf28954ef6a8cbe8a9f85c8d76 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/run_tasks.py", line 106, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_f25ffac72e5a9e63860ff7deb6767a11f0559a16/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc), test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-06-29T16:39:02.675 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2022-06-29T16:39:02.686 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2022-06-29T16:39:02.732 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2022-06-29T16:39:02.732 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:39:02.733 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:39:02.733 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:39:04.984 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:39:04.987 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:39:04.990 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:39:04.993 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:39:04.996 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:39:35.138 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:39:35.142 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:39:35.145 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:39:35.148 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:39:35.150 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:40:05.264 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:40:05.268 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:40:05.271 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:40:05.273 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:40:05.277 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:40:35.414 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:40:35.418 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:40:35.420 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:40:35.423 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:40:35.426 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:41:05.559 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:41:05.563 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:41:05.566 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:41:05.568 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:41:05.571 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:41:35.747 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:41:35.750 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:41:35.753 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:41:35.756 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:41:35.758 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:42:05.909 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:42:05.913 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:42:05.916 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:42:05.919 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:42:05.922 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:42:36.038 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:42:36.042 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:42:36.045 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:42:36.048 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:42:36.050 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:43:06.134 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:43:06.139 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:43:06.142 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:43:06.145 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:43:06.148 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:43:36.289 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:43:36.293 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:43:36.297 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:43:36.301 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:43:36.305 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:44:06.426 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:44:06.430 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:44:06.433 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:44:06.436 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:44:06.440 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:44:36.580 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:44:36.584 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:44:36.587 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:44:36.589 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:44:36.592 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:45:06.666 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:45:06.670 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:45:06.673 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:45:06.676 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:45:06.680 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:45:36.734 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:45:36.738 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:45:36.741 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:45:36.743 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:45:36.746 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:46:06.868 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:46:06.872 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:46:06.875 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:46:06.879 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:46:06.883 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:46:36.981 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:46:36.985 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:46:36.988 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:46:36.991 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:46:36.994 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:47:07.122 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:47:07.126 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:47:07.128 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:47:07.131 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:47:07.134 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:47:37.310 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:47:37.313 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:47:37.316 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:47:37.319 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:47:37.322 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:48:07.680 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:48:07.683 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:48:07.686 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:48:07.688 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:48:07.691 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:48:37.818 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:48:37.821 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:48:37.824 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:48:37.827 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:48:37.830 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:49:07.960 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:49:07.964 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:49:07.966 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:49:07.969 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:49:07.972 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:49:38.098 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:49:38.101 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:49:38.104 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:49:38.107 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:49:38.109 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:50:08.232 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:50:08.236 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:50:08.239 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:50:08.241 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:50:08.244 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:50:38.464 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:50:38.468 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:50:38.471 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:50:38.474 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:50:38.477 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:51:08.601 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:51:08.605 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:51:08.609 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:51:08.611 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:51:08.615 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:51:38.759 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:51:38.763 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:51:38.766 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:51:38.769 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:51:38.771 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:52:08.905 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:52:08.908 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:52:08.911 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:52:08.913 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:52:08.916 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:52:39.089 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:52:39.094 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:52:39.097 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:52:39.100 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:52:39.102 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:53:09.257 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:53:09.261 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:53:09.264 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:53:09.267 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:53:09.271 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:53:39.428 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:53:39.432 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:53:39.434 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:53:39.437 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:53:39.440 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:54:02.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T16:54:02.851 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': Connection timed out 2022-06-29T16:54:02.853 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.0 2022-06-29T16:54:02.854 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-06-29T16:54:02.855 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-06-29T16:54:02.855 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-06-29T16:54:09.563 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:54:09.567 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:54:09.570 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:54:09.573 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:54:09.575 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:54:39.697 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:54:39.701 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:54:39.704 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:54:39.707 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:54:39.709 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:55:09.794 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:55:09.798 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:55:09.800 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:55:09.803 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:55:09.806 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:55:39.939 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:55:39.945 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:55:39.948 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:55:39.951 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:55:39.954 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:56:10.216 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:56:10.220 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:56:10.223 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:56:10.226 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:56:10.229 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:56:40.361 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:56:40.364 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:56:40.367 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:56:40.370 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:56:40.372 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:57:10.511 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:57:10.516 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:57:10.519 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:57:10.522 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:57:10.524 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:57:40.653 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:57:40.657 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:57:40.660 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:57:40.663 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:57:40.665 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:58:10.895 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:58:10.899 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:58:10.902 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:58:10.905 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:58:10.908 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:58:41.045 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:58:41.049 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:58:41.052 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:58:41.054 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:58:41.057 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:59:11.198 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:59:11.202 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:59:11.205 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:59:11.207 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:59:11.210 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:59:41.328 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:59:41.331 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:59:41.334 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:59:41.337 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T16:59:41.340 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:00:11.466 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:00:11.469 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:00:11.472 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:00:11.475 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:00:11.477 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:00:41.598 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:00:41.602 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:00:41.605 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:00:41.608 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:00:41.611 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:01:11.724 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:01:11.728 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:01:11.731 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:01:11.734 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:01:11.737 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:01:41.871 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:01:41.877 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:01:41.880 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:01:41.883 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:01:41.887 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:02:12.017 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:02:12.021 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:02:12.023 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:02:12.026 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:02:12.029 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:02:42.144 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:02:42.148 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:02:42.151 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:02:42.155 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:02:42.158 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:03:12.484 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:03:12.487 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:03:12.490 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:03:12.493 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:03:12.497 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:03:42.569 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:03:42.573 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:03:42.576 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:03:42.578 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:03:42.581 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:04:12.691 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:04:12.696 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:04:12.700 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:04:12.703 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:04:12.707 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:04:42.846 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:04:42.850 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:04:42.853 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:04:42.857 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:04:42.861 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:05:12.976 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:05:12.980 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:05:12.982 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:05:12.985 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:05:12.988 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:05:43.148 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:05:43.159 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:05:43.162 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:05:43.165 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:05:43.167 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:06:13.292 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:06:13.295 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:06:13.298 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:06:13.301 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:06:13.303 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:06:43.421 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:06:43.425 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:06:43.427 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:06:43.430 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:06:43.432 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:07:13.567 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:07:13.570 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:07:13.573 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:07:13.575 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:07:13.578 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:07:43.728 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:07:43.733 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:07:43.736 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:07:43.740 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:07:43.744 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:08:13.942 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:08:13.945 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:08:13.948 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:08:13.951 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:08:13.953 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:08:44.080 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:08:44.083 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:08:44.086 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:08:44.089 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:08:44.092 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:09:02.861 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-06-29T17:09:02.877 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2022-06-29T17:09:02.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T17:09:02.909 ERROR:teuthology.run_tasks:Manager failed: ceph-fuse Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/run_tasks.py", line 188, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_f25ffac72e5a9e63860ff7deb6767a11f0559a16/qa/tasks/ceph_fuse.py", line 160, in task mount.umount_wait() File "/home/teuthworker/src/github.com_ceph_ceph-c_f25ffac72e5a9e63860ff7deb6767a11f0559a16/qa/tasks/cephfs/fuse_mount.py", line 326, in umount_wait if not (self.is_mounted() and self.fuse_daemon): File "/home/teuthworker/src/github.com_ceph_ceph-c_f25ffac72e5a9e63860ff7deb6767a11f0559a16/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-06-29T17:09:02.932 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2022-06-29T17:09:02.950 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2022-06-29T17:09:02.951 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T17:09:03.077 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.071+0000 7f4e7359e700 1 Processor -- start 2022-06-29T17:09:03.078 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.071+0000 7f4e7359e700 1 -- start start 2022-06-29T17:09:03.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.071+0000 7f4e7359e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f4e7405c5b0 0x7f4e7405c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:03.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.071+0000 7f4e7359e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4e74060610 0x7f4e7413c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:03.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.071+0000 7f4e7359e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e740b0a40 con 0x7f4e7413cc30 2022-06-29T17:09:03.082 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.071+0000 7f4e7359e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e740b0850 con 0x7f4e74060610 2022-06-29T17:09:03.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.071+0000 7f4e7359e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e740b0ce0 con 0x7f4e7405c5b0 2022-06-29T17:09:03.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.071+0000 7f4e71d9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4e74060610 0x7f4e7413c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:03.083 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.071+0000 7f4e71d9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f4e74060610 0x7f4e7413c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54826/0 (socket says 172.21.15.60:54826) 2022-06-29T17:09:03.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.071+0000 7f4e71d9b700 1 -- 172.21.15.60:0/445659117 learned_addr learned my addr 172.21.15.60:0/445659117 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T17:09:03.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.071+0000 7f4e7259c700 1 --2- 172.21.15.60:0/445659117 >> v2:172.21.15.92:3300/0 conn(0x7f4e7405c5b0 0x7f4e7405c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:03.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.075+0000 7f4e71d9b700 1 -- 172.21.15.60:0/445659117 >> v2:172.21.15.92:3300/0 conn(0x7f4e7405c5b0 msgr2=0x7f4e7405c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T17:09:03.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.075+0000 7f4e71d9b700 1 --2- 172.21.15.60:0/445659117 >> v2:172.21.15.92:3300/0 conn(0x7f4e7405c5b0 0x7f4e7405c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:03.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.075+0000 7f4e71d9b700 1 -- 172.21.15.60:0/445659117 >> v1:172.21.15.60:6789/0 conn(0x7f4e7413cc30 legacy=0x7f4e74142a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:03.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.075+0000 7f4e7159a700 1 -- 172.21.15.60:0/445659117 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3218087507 0 0) 0x7f4e740b0a40 con 0x7f4e7413cc30 2022-06-29T17:09:03.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.075+0000 7f4e71d9b700 1 -- 172.21.15.60:0/445659117 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e7405f350 con 0x7f4e74060610 2022-06-29T17:09:03.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.075+0000 7f4e71d9b700 1 --2- 172.21.15.60:0/445659117 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e74060610 0x7f4e7413c740 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 rx=0x7f4e5c002f70 tx=0x7f4e5c006eb0).ready entity=mon.0 client_cookie=b20cc4d2806b602c server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:03.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.075+0000 7f4e7159a700 1 -- 172.21.15.60:0/445659117 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e5c00d5c0 con 0x7f4e74060610 2022-06-29T17:09:03.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.075+0000 7f4e7159a700 1 -- 172.21.15.60:0/445659117 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e5c000e90 con 0x7f4e74060610 2022-06-29T17:09:03.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.075+0000 7f4e7159a700 1 -- 172.21.15.60:0/445659117 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e5c00d910 con 0x7f4e74060610 2022-06-29T17:09:03.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.075+0000 7f4e7359e700 1 -- 172.21.15.60:0/445659117 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e74060610 msgr2=0x7f4e7413c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:03.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.075+0000 7f4e7359e700 1 --2- 172.21.15.60:0/445659117 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e74060610 0x7f4e7413c740 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 rx=0x7f4e5c002f70 tx=0x7f4e5c006eb0).stop 2022-06-29T17:09:03.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.075+0000 7f4e7359e700 1 -- 172.21.15.60:0/445659117 shutdown_connections 2022-06-29T17:09:03.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.075+0000 7f4e7359e700 1 --2- 172.21.15.60:0/445659117 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e74060610 0x7f4e7413c740 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:03.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.075+0000 7f4e7359e700 1 --2- 172.21.15.60:0/445659117 >> v2:172.21.15.92:3300/0 conn(0x7f4e7405c5b0 0x7f4e7405c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:03.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.075+0000 7f4e7359e700 1 -- 172.21.15.60:0/445659117 >> 172.21.15.60:0/445659117 conn(0x7f4e74061c80 msgr2=0x7f4e740637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:03.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.075+0000 7f4e7359e700 1 -- 172.21.15.60:0/445659117 shutdown_connections 2022-06-29T17:09:03.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.075+0000 7f4e7359e700 1 -- 172.21.15.60:0/445659117 wait complete. 2022-06-29T17:09:03.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.075+0000 7f4e7359e700 1 Processor -- start 2022-06-29T17:09:03.095 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.075+0000 7f4e7359e700 1 -- start start 2022-06-29T17:09:03.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.079+0000 7f4e7359e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e7405c5b0 0x7f4e7413e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:03.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.079+0000 7f4e7359e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e74060610 0x7f4e740af1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:03.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.079+0000 7f4e7359e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4e7413cc30 0x7f4e740af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:03.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.079+0000 7f4e7359e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7413f080 con 0x7f4e7405c5b0 2022-06-29T17:09:03.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.079+0000 7f4e7359e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e74145ec0 con 0x7f4e74060610 2022-06-29T17:09:03.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.079+0000 7f4e7359e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e740aea60 con 0x7f4e7413cc30 2022-06-29T17:09:03.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.079+0000 7f4e7259c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e7405c5b0 0x7f4e7413e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:03.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.079+0000 7f4e7259c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e7405c5b0 0x7f4e7413e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54832/0 (socket says 172.21.15.60:54832) 2022-06-29T17:09:03.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.079+0000 7f4e7259c700 1 -- 172.21.15.60:0/2435001195 learned_addr learned my addr 172.21.15.60:0/2435001195 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T17:09:03.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.079+0000 7f4e72d9d700 1 --2- 172.21.15.60:0/2435001195 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4e7413cc30 0x7f4e740af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:03.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.079+0000 7f4e71d9b700 1 --2- 172.21.15.60:0/2435001195 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e74060610 0x7f4e740af1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:03.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.079+0000 7f4e7259c700 1 -- 172.21.15.60:0/2435001195 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4e7413cc30 msgr2=0x7f4e740af6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:03.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.079+0000 7f4e7259c700 1 --2- 172.21.15.60:0/2435001195 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4e7413cc30 0x7f4e740af6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:03.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.079+0000 7f4e7259c700 1 -- 172.21.15.60:0/2435001195 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e74060610 msgr2=0x7f4e740af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:03.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.079+0000 7f4e7259c700 1 --2- 172.21.15.60:0/2435001195 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e74060610 0x7f4e740af1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:03.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.079+0000 7f4e7259c700 1 -- 172.21.15.60:0/2435001195 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e5c00b1e0 con 0x7f4e7405c5b0 2022-06-29T17:09:03.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.079+0000 7f4e7259c700 1 --2- 172.21.15.60:0/2435001195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e7405c5b0 0x7f4e7413e350 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 rx=0x7f4e7413f1c0 tx=0x7f4e64006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:03.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.079+0000 7f4e6b7fe700 1 -- 172.21.15.60:0/2435001195 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e64010780 con 0x7f4e7405c5b0 2022-06-29T17:09:03.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.079+0000 7f4e6b7fe700 1 -- 172.21.15.60:0/2435001195 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e640108e0 con 0x7f4e7405c5b0 2022-06-29T17:09:03.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.079+0000 7f4e7359e700 1 -- 172.21.15.60:0/2435001195 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e740aed00 con 0x7f4e7405c5b0 2022-06-29T17:09:03.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.079+0000 7f4e6b7fe700 1 -- 172.21.15.60:0/2435001195 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e64010c40 con 0x7f4e7405c5b0 2022-06-29T17:09:03.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.079+0000 7f4e7359e700 1 -- 172.21.15.60:0/2435001195 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e74146850 con 0x7f4e7405c5b0 2022-06-29T17:09:03.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.083+0000 7f4e6b7fe700 1 -- 172.21.15.60:0/2435001195 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f4e6401c460 con 0x7f4e7405c5b0 2022-06-29T17:09:03.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.083+0000 7f4e7359e700 1 -- 172.21.15.60:0/2435001195 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e740573d0 con 0x7f4e7405c5b0 2022-06-29T17:09:03.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.083+0000 7f4e6b7fe700 1 --2- 172.21.15.60:0/2435001195 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4e58053a70 0x7f4e58055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:03.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.083+0000 7f4e6b7fe700 1 -- 172.21.15.60:0/2435001195 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f4e6406c150 con 0x7f4e7405c5b0 2022-06-29T17:09:03.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.083+0000 7f4e71d9b700 1 --2- 172.21.15.60:0/2435001195 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4e58053a70 0x7f4e58055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:03.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.083+0000 7f4e71d9b700 1 --2- 172.21.15.60:0/2435001195 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4e58053a70 0x7f4e58055f30 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 rx=0x7f4e5c01c000 tx=0x7f4e5c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:03.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.087+0000 7f4e6b7fe700 1 -- 172.21.15.60:0/2435001195 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4e64043160 con 0x7f4e7405c5b0 2022-06-29T17:09:03.224 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.219+0000 7f4e7359e700 1 -- 172.21.15.60:0/2435001195 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e74146630 con 0x7f4e7405c5b0 2022-06-29T17:09:03.226 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.219+0000 7f4e6b7fe700 1 -- 172.21.15.60:0/2435001195 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v150) v1 ==== 74+0+13320 (secure 0 0 0) 0x7f4e64046780 con 0x7f4e7405c5b0 2022-06-29T17:09:03.226 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T17:09:03.227 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":150,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:08:32.204356+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{"172.21.15.60:6833/3619856083":"2022-06-30T16:08:28.758125+0000","172.21.15.60:6832/3619856083":"2022-06-30T16:08:28.758125+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T17:09:03.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.223+0000 7f4e7359e700 1 -- 172.21.15.60:0/2435001195 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4e58053a70 msgr2=0x7f4e58055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:03.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.223+0000 7f4e7359e700 1 --2- 172.21.15.60:0/2435001195 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4e58053a70 0x7f4e58055f30 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 rx=0x7f4e5c01c000 tx=0x7f4e5c015000).stop 2022-06-29T17:09:03.230 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.223+0000 7f4e7359e700 1 -- 172.21.15.60:0/2435001195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e7405c5b0 msgr2=0x7f4e7413e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:03.230 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.223+0000 7f4e7359e700 1 --2- 172.21.15.60:0/2435001195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e7405c5b0 0x7f4e7413e350 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 rx=0x7f4e7413f1c0 tx=0x7f4e64006df0).stop 2022-06-29T17:09:03.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.223+0000 7f4e7359e700 1 -- 172.21.15.60:0/2435001195 shutdown_connections 2022-06-29T17:09:03.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.223+0000 7f4e7359e700 1 --2- 172.21.15.60:0/2435001195 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f4e58053a70 0x7f4e58055f30 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:03.231 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.223+0000 7f4e7359e700 1 --2- 172.21.15.60:0/2435001195 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f4e7413cc30 0x7f4e740af6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:03.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.223+0000 7f4e7359e700 1 --2- 172.21.15.60:0/2435001195 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e74060610 0x7f4e740af1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:03.232 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.223+0000 7f4e7359e700 1 --2- 172.21.15.60:0/2435001195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e7405c5b0 0x7f4e7413e350 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:03.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.223+0000 7f4e7359e700 1 -- 172.21.15.60:0/2435001195 >> 172.21.15.60:0/2435001195 conn(0x7f4e74061c80 msgr2=0x7f4e740b3cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:03.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.223+0000 7f4e7359e700 1 -- 172.21.15.60:0/2435001195 shutdown_connections 2022-06-29T17:09:03.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.223+0000 7f4e7359e700 1 -- 172.21.15.60:0/2435001195 wait complete. 2022-06-29T17:09:03.240 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-06-29T17:09:03.345 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.339+0000 7f29d0a29700 1 Processor -- start 2022-06-29T17:09:03.346 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.339+0000 7f29d0a29700 1 -- start start 2022-06-29T17:09:03.348 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.339+0000 7f29d0a29700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f29cc0b2130 0x7f29cc146100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:03.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.339+0000 7f29d0a29700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f29cc0aed00 0x7f29cc1465f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:03.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.339+0000 7f29d0a29700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29cc05d450 con 0x7f29cc05d070 2022-06-29T17:09:03.349 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.339+0000 7f29d0a29700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29cc0adf90 con 0x7f29cc0aed00 2022-06-29T17:09:03.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.339+0000 7f29d0a29700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc0b25d0 con 0x7f29cc0b2130 2022-06-29T17:09:03.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.339+0000 7f29cad9d700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f29cc05d070 0x7f29cc14aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34694/0 (socket says 172.21.15.60:34694) 2022-06-29T17:09:03.350 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.339+0000 7f29cad9d700 1 -- 172.21.15.60:0/2239591593 learned_addr learned my addr 172.21.15.60:0/2239591593 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T17:09:03.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.339+0000 7f29ca59c700 1 --2- 172.21.15.60:0/2239591593 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f29cc0b2130 0x7f29cc146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:03.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.339+0000 7f29cb59e700 1 --2- 172.21.15.60:0/2239591593 >> v2:172.21.15.92:3300/0 conn(0x7f29cc0aed00 0x7f29cc1465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:03.351 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.343+0000 7f29c9d9b700 1 -- 172.21.15.60:0/2239591593 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 248120744 0 0) 0x7f29cc05d450 con 0x7f29cc05d070 2022-06-29T17:09:03.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.343+0000 7f29c9d9b700 1 -- 172.21.15.60:0/2239591593 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29b0003260 con 0x7f29cc05d070 2022-06-29T17:09:03.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.343+0000 7f29ca59c700 1 -- 172.21.15.60:0/2239591593 >> v2:172.21.15.92:3300/0 conn(0x7f29cc0aed00 msgr2=0x7f29cc1465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T17:09:03.352 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.343+0000 7f29ca59c700 1 --2- 172.21.15.60:0/2239591593 >> v2:172.21.15.92:3300/0 conn(0x7f29cc0aed00 0x7f29cc1465f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:03.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.343+0000 7f29c9d9b700 1 -- 172.21.15.60:0/2239591593 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1574248579 0 0) 0x7f29b0003260 con 0x7f29cc05d070 2022-06-29T17:09:03.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.343+0000 7f29ca59c700 1 -- 172.21.15.60:0/2239591593 >> v1:172.21.15.60:6789/0 conn(0x7f29cc05d070 legacy=0x7f29cc14aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:03.353 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.343+0000 7f29ca59c700 1 -- 172.21.15.60:0/2239591593 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29cc14a350 con 0x7f29cc0b2130 2022-06-29T17:09:03.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.343+0000 7f29cb59e700 1 --2- 172.21.15.60:0/2239591593 >> v2:172.21.15.92:3300/0 conn(0x7f29cc0aed00 0x7f29cc1465f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T17:09:03.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.343+0000 7f29ca59c700 1 --2- 172.21.15.60:0/2239591593 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f29cc0b2130 0x7f29cc146100 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f29bc001820 tx=0x7f29b4006df0).ready entity=mon.2 client_cookie=1f0507fc0a8ae5f6 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:03.354 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.343+0000 7f29c9d9b700 1 -- 172.21.15.60:0/2239591593 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29bc001680 con 0x7f29cc0b2130 2022-06-29T17:09:03.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.343+0000 7f29c9d9b700 1 -- 172.21.15.60:0/2239591593 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b4000e90 con 0x7f29cc0b2130 2022-06-29T17:09:03.355 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.343+0000 7f29c9d9b700 1 -- 172.21.15.60:0/2239591593 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29b4012460 con 0x7f29cc0b2130 2022-06-29T17:09:03.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.343+0000 7f29d0a29700 1 -- 172.21.15.60:0/2239591593 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f29cc0b2130 msgr2=0x7f29cc146100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:03.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.343+0000 7f29d0a29700 1 --2- 172.21.15.60:0/2239591593 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f29cc0b2130 0x7f29cc146100 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f29bc001820 tx=0x7f29b4006df0).stop 2022-06-29T17:09:03.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.343+0000 7f29d0a29700 1 -- 172.21.15.60:0/2239591593 shutdown_connections 2022-06-29T17:09:03.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.343+0000 7f29d0a29700 1 --2- 172.21.15.60:0/2239591593 >> v2:172.21.15.92:3300/0 conn(0x7f29cc0aed00 0x7f29cc1465f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:03.360 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.343+0000 7f29d0a29700 1 --2- 172.21.15.60:0/2239591593 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f29cc0b2130 0x7f29cc146100 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:03.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.343+0000 7f29d0a29700 1 -- 172.21.15.60:0/2239591593 >> 172.21.15.60:0/2239591593 conn(0x7f29cc0b5890 msgr2=0x7f29cc0b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:03.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.343+0000 7f29d0a29700 1 -- 172.21.15.60:0/2239591593 shutdown_connections 2022-06-29T17:09:03.361 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.343+0000 7f29d0a29700 1 -- 172.21.15.60:0/2239591593 wait complete. 2022-06-29T17:09:03.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.343+0000 7f29d0a29700 1 Processor -- start 2022-06-29T17:09:03.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.343+0000 7f29d0a29700 1 -- start start 2022-06-29T17:09:03.362 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.347+0000 7f29d0a29700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f29cc05d070 0x7f29cc14c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:03.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.347+0000 7f29d0a29700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29cc0aed00 0x7f29cc149480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:03.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.347+0000 7f29d0a29700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f29cc0b2130 0x7f29cc0652f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:03.363 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.347+0000 7f29d0a29700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc066ed0 con 0x7f29cc0aed00 2022-06-29T17:09:03.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.347+0000 7f29d0a29700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc14c940 con 0x7f29cc05d070 2022-06-29T17:09:03.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.347+0000 7f29d0a29700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc063240 con 0x7f29cc0b2130 2022-06-29T17:09:03.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.347+0000 7f29ca59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29cc0aed00 0x7f29cc149480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:03.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.347+0000 7f29ca59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29cc0aed00 0x7f29cc149480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54848/0 (socket says 172.21.15.60:54848) 2022-06-29T17:09:03.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.347+0000 7f29ca59c700 1 -- 172.21.15.60:0/3560526821 learned_addr learned my addr 172.21.15.60:0/3560526821 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T17:09:03.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.347+0000 7f29cb59e700 1 --2- 172.21.15.60:0/3560526821 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f29cc0b2130 0x7f29cc0652f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:03.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.347+0000 7f29cad9d700 1 --2- 172.21.15.60:0/3560526821 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f29cc05d070 0x7f29cc14c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:03.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.347+0000 7f29ca59c700 1 -- 172.21.15.60:0/3560526821 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f29cc0b2130 msgr2=0x7f29cc0652f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:03.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.347+0000 7f29ca59c700 1 --2- 172.21.15.60:0/3560526821 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f29cc0b2130 0x7f29cc0652f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:03.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.347+0000 7f29ca59c700 1 -- 172.21.15.60:0/3560526821 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f29cc05d070 msgr2=0x7f29cc14c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:03.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.347+0000 7f29ca59c700 1 --2- 172.21.15.60:0/3560526821 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f29cc05d070 0x7f29cc14c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:03.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.347+0000 7f29ca59c700 1 -- 172.21.15.60:0/3560526821 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c4009470 con 0x7f29cc0aed00 2022-06-29T17:09:03.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.347+0000 7f29cb59e700 1 --2- 172.21.15.60:0/3560526821 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f29cc0b2130 0x7f29cc0652f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T17:09:03.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.347+0000 7f29ca59c700 1 --2- 172.21.15.60:0/3560526821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29cc0aed00 0x7f29cc149480 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 rx=0x7f29bc001820 tx=0x7f29b4014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:03.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.347+0000 7f29c3fff700 1 -- 172.21.15.60:0/3560526821 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29b4016070 con 0x7f29cc0aed00 2022-06-29T17:09:03.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.347+0000 7f29c3fff700 1 -- 172.21.15.60:0/3560526821 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b4013e40 con 0x7f29cc0aed00 2022-06-29T17:09:03.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.347+0000 7f29d0a29700 1 -- 172.21.15.60:0/3560526821 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29b400b180 con 0x7f29cc0aed00 2022-06-29T17:09:03.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.347+0000 7f29c3fff700 1 -- 172.21.15.60:0/3560526821 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29b400ddb0 con 0x7f29cc0aed00 2022-06-29T17:09:03.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.347+0000 7f29d0a29700 1 -- 172.21.15.60:0/3560526821 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29cc065980 con 0x7f29cc0aed00 2022-06-29T17:09:03.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.351+0000 7f29d0a29700 1 -- 172.21.15.60:0/3560526821 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29cc14a5f0 con 0x7f29cc0aed00 2022-06-29T17:09:03.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.351+0000 7f29c3fff700 1 -- 172.21.15.60:0/3560526821 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f29b400f030 con 0x7f29cc0aed00 2022-06-29T17:09:03.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.351+0000 7f29c3fff700 1 --2- 172.21.15.60:0/3560526821 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f29a40539e0 0x7f29a4055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:03.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.351+0000 7f29c3fff700 1 -- 172.21.15.60:0/3560526821 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f29b406caa0 con 0x7f29cc0aed00 2022-06-29T17:09:03.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.351+0000 7f29cad9d700 1 --2- 172.21.15.60:0/3560526821 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f29a40539e0 0x7f29a4055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:03.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.351+0000 7f29cad9d700 1 --2- 172.21.15.60:0/3560526821 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f29a40539e0 0x7f29a4055ea0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0x7f29bc00d000 tx=0x7f29bc001de0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:03.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.359+0000 7f29c3fff700 1 -- 172.21.15.60:0/3560526821 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f29b4043b90 con 0x7f29cc0aed00 2022-06-29T17:09:03.491 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.483+0000 7f29d0a29700 1 -- 172.21.15.60:0/3560526821 --> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f29cc14d770 con 0x7f29a40539e0 2022-06-29T17:09:03.498 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.491+0000 7f29c3fff700 1 -- 172.21.15.60:0/3560526821 <== mgr.4103 v2:172.21.15.92:6832/12872 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+68052 (secure 0 0 0) 0x7f29cc14d770 con 0x7f29a40539e0 2022-06-29T17:09:03.500 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T17:09:03.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.495+0000 7f29d0a29700 1 -- 172.21.15.60:0/3560526821 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f29a40539e0 msgr2=0x7f29a4055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:03.501 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.495+0000 7f29d0a29700 1 --2- 172.21.15.60:0/3560526821 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f29a40539e0 0x7f29a4055ea0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0x7f29bc00d000 tx=0x7f29bc001de0).stop 2022-06-29T17:09:03.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.495+0000 7f29d0a29700 1 -- 172.21.15.60:0/3560526821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29cc0aed00 msgr2=0x7f29cc149480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:03.502 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.495+0000 7f29d0a29700 1 --2- 172.21.15.60:0/3560526821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29cc0aed00 0x7f29cc149480 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 rx=0x7f29bc001820 tx=0x7f29b4014000).stop 2022-06-29T17:09:03.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.495+0000 7f29d0a29700 1 -- 172.21.15.60:0/3560526821 shutdown_connections 2022-06-29T17:09:03.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.495+0000 7f29d0a29700 1 --2- 172.21.15.60:0/3560526821 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f29a40539e0 0x7f29a4055ea0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:03.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.495+0000 7f29d0a29700 1 --2- 172.21.15.60:0/3560526821 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f29cc0b2130 0x7f29cc0652f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:03.503 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.495+0000 7f29d0a29700 1 --2- 172.21.15.60:0/3560526821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f29cc0aed00 0x7f29cc149480 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:03.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.495+0000 7f29d0a29700 1 --2- 172.21.15.60:0/3560526821 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f29cc05d070 0x7f29cc14c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:03.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.495+0000 7f29d0a29700 1 -- 172.21.15.60:0/3560526821 >> 172.21.15.60:0/3560526821 conn(0x7f29cc0b5890 msgr2=0x7f29cc0ad5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:03.504 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.495+0000 7f29d0a29700 1 -- 172.21.15.60:0/3560526821 shutdown_connections 2022-06-29T17:09:03.505 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.495+0000 7f29d0a29700 1 -- 172.21.15.60:0/3560526821 wait complete. 2022-06-29T17:09:03.505 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2022-06-29T17:09:03.514 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":2267,"stamp":"2022-06-29T17:09:02.087673+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":6385024,"kb_used_data":743210016,"kb_used_omap":27916,"kb_used_meta":0,"kb_avail":743210016,"statfs":{"total":767585320960,"available":761047056384,"internally_reserved":0,"allocated":761047056384,"data_stored":761047056384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28586567,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003260"},"pg_stats":[{"pgid":"1.0","version":"14'2","reported_seq":146,"reported_epoch":149,"state":"active+clean","last_fresh":"2022-06-29T16:08:31.209595+0000","last_change":"2022-06-29T15:58:14.849417+0000","last_active":"2022-06-29T16:08:31.209595+0000","last_peered":"2022-06-29T16:08:31.209595+0000","last_clean":"2022-06-29T16:08:31.209595+0000","last_became_active":"2022-06-29T15:58:14.848454+0000","last_became_peered":"2022-06-29T15:58:14.848454+0000","last_unstale":"2022-06-29T16:08:31.209595+0000","last_undegraded":"2022-06-29T16:08:31.209595+0000","last_fullsized":"2022-06-29T16:08:31.209595+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:11.766061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:11.766061+0000","last_clean_scrub_stamp":"2022-06-29T15:58:11.766061+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":132,"reported_epoch":149,"state":"active+clean","last_fresh":"2022-06-29T16:08:31.831503+0000","last_change":"2022-06-29T15:58:21.003987+0000","last_active":"2022-06-29T16:08:31.831503+0000","last_peered":"2022-06-29T16:08:31.831503+0000","last_clean":"2022-06-29T16:08:31.831503+0000","last_became_active":"2022-06-29T15:58:21.003537+0000","last_became_peered":"2022-06-29T15:58:21.003537+0000","last_unstale":"2022-06-29T16:08:31.831503+0000","last_undegraded":"2022-06-29T16:08:31.831503+0000","last_fullsized":"2022-06-29T16:08:31.831503+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_clean_scrub_stamp":"2022-06-29T15:58:19.966881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":141,"reported_epoch":149,"state":"active+clean","last_fresh":"2022-06-29T16:08:31.209460+0000","last_change":"2022-06-29T15:58:21.001629+0000","last_active":"2022-06-29T16:08:31.209460+0000","last_peered":"2022-06-29T16:08:31.209460+0000","last_clean":"2022-06-29T16:08:31.209460+0000","last_became_active":"2022-06-29T15:58:21.000885+0000","last_became_peered":"2022-06-29T15:58:21.000885+0000","last_unstale":"2022-06-29T16:08:31.209460+0000","last_undegraded":"2022-06-29T16:08:31.209460+0000","last_fullsized":"2022-06-29T16:08:31.209460+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_clean_scrub_stamp":"2022-06-29T15:58:19.966881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":136,"reported_epoch":149,"state":"active+clean","last_fresh":"2022-06-29T16:08:31.832454+0000","last_change":"2022-06-29T15:58:20.997373+0000","last_active":"2022-06-29T16:08:31.832454+0000","last_peered":"2022-06-29T16:08:31.832454+0000","last_clean":"2022-06-29T16:08:31.832454+0000","last_became_active":"2022-06-29T15:58:20.997032+0000","last_became_peered":"2022-06-29T15:58:20.997032+0000","last_unstale":"2022-06-29T16:08:31.832454+0000","last_undegraded":"2022-06-29T16:08:31.832454+0000","last_fullsized":"2022-06-29T16:08:31.832454+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_clean_scrub_stamp":"2022-06-29T15:58:19.966881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":132,"reported_epoch":149,"state":"active+clean","last_fresh":"2022-06-29T16:08:31.831202+0000","last_change":"2022-06-29T15:58:21.005059+0000","last_active":"2022-06-29T16:08:31.831202+0000","last_peered":"2022-06-29T16:08:31.831202+0000","last_clean":"2022-06-29T16:08:31.831202+0000","last_became_active":"2022-06-29T15:58:21.004623+0000","last_became_peered":"2022-06-29T15:58:21.004623+0000","last_unstale":"2022-06-29T16:08:31.831202+0000","last_undegraded":"2022-06-29T16:08:31.831202+0000","last_fullsized":"2022-06-29T16:08:31.831202+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_clean_scrub_stamp":"2022-06-29T15:58:19.966881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":131,"reported_epoch":149,"state":"active+clean","last_fresh":"2022-06-29T16:08:31.831388+0000","last_change":"2022-06-29T15:58:21.801445+0000","last_active":"2022-06-29T16:08:31.831388+0000","last_peered":"2022-06-29T16:08:31.831388+0000","last_clean":"2022-06-29T16:08:31.831388+0000","last_became_active":"2022-06-29T15:58:21.800792+0000","last_became_peered":"2022-06-29T15:58:21.800792+0000","last_unstale":"2022-06-29T16:08:31.831388+0000","last_undegraded":"2022-06-29T16:08:31.831388+0000","last_fullsized":"2022-06-29T16:08:31.831388+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_clean_scrub_stamp":"2022-06-29T15:58:19.966881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":141,"reported_epoch":149,"state":"active+clean","last_fresh":"2022-06-29T16:08:31.209798+0000","last_change":"2022-06-29T15:58:21.807952+0000","last_active":"2022-06-29T16:08:31.209798+0000","last_peered":"2022-06-29T16:08:31.209798+0000","last_clean":"2022-06-29T16:08:31.209798+0000","last_became_active":"2022-06-29T15:58:21.807346+0000","last_became_peered":"2022-06-29T15:58:21.807346+0000","last_unstale":"2022-06-29T16:08:31.209798+0000","last_undegraded":"2022-06-29T16:08:31.209798+0000","last_fullsized":"2022-06-29T16:08:31.209798+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_clean_scrub_stamp":"2022-06-29T15:58:19.966881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":131,"reported_epoch":149,"state":"active+clean","last_fresh":"2022-06-29T16:08:31.831563+0000","last_change":"2022-06-29T15:58:20.996366+0000","last_active":"2022-06-29T16:08:31.831563+0000","last_peered":"2022-06-29T16:08:31.831563+0000","last_clean":"2022-06-29T16:08:31.831563+0000","last_became_active":"2022-06-29T15:58:20.995883+0000","last_became_peered":"2022-06-29T15:58:20.995883+0000","last_unstale":"2022-06-29T16:08:31.831563+0000","last_undegraded":"2022-06-29T16:08:31.831563+0000","last_fullsized":"2022-06-29T16:08:31.831563+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_clean_scrub_stamp":"2022-06-29T15:58:19.966881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":131,"reported_epoch":149,"state":"active+clean","last_fresh":"2022-06-29T16:08:31.833014+0000","last_change":"2022-06-29T15:58:21.003020+0000","last_active":"2022-06-29T16:08:31.833014+0000","last_peered":"2022-06-29T16:08:31.833014+0000","last_clean":"2022-06-29T16:08:31.833014+0000","last_became_active":"2022-06-29T15:58:21.002353+0000","last_became_peered":"2022-06-29T15:58:21.002353+0000","last_unstale":"2022-06-29T16:08:31.833014+0000","last_undegraded":"2022-06-29T16:08:31.833014+0000","last_fullsized":"2022-06-29T16:08:31.833014+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_clean_scrub_stamp":"2022-06-29T15:58:19.966881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575700,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798144,"kb_used_data":92901236,"kb_used_omap":3659,"kb_used_meta":0,"kb_avail":92901236,"statfs":{"total":95948165120,"available":95130865664,"internally_reserved":0,"allocated":95130865664,"data_stored":95130865664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3747326,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jun 29 17:08:21 2022","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.83699999999999997,"15min":0.82699999999999996},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.5},"max":{"1min":1.0640000000000001,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.81599999999999995,"15min":0.81499999999999995},"min":{"1min":0.61599999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.95199999999999996,"5min":1.3120000000000001,"15min":1.377},"last":0.83699999999999997}]},{"osd":1,"last update":"Wed Jun 29 17:08:21 2022","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.82399999999999995,"15min":0.83299999999999996},"min":{"1min":0.64900000000000002,"5min":0.57199999999999995,"15min":0.56599999999999995},"max":{"1min":1.125,"5min":1.125,"15min":1.1890000000000001},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.83399999999999996,"15min":0.84899999999999998},"min":{"1min":0.623,"5min":0.60599999999999998,"15min":0.59399999999999997},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1659999999999999},"last":0.67500000000000004}]},{"osd":2,"last update":"Wed Jun 29 17:08:21 2022","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.871,"15min":0.872},"min":{"1min":0.625,"5min":0.625,"15min":0.57099999999999995},"max":{"1min":1.202,"5min":1.202,"15min":1.244},"last":0.87},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.85299999999999998,"15min":0.85099999999999998},"min":{"1min":0.61899999999999999,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.097,"5min":1.371,"15min":1.371},"last":0.999}]},{"osd":3,"last update":"Wed Jun 29 17:08:21 2022","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.84299999999999997,"15min":0.84699999999999998},"min":{"1min":0.64000000000000001,"5min":0.60499999999999998,"15min":0.48999999999999999},"max":{"1min":1.0600000000000001,"5min":1.204,"15min":1.3480000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.92300000000000004,"15min":0.91500000000000004},"min":{"1min":0.69399999999999995,"5min":0.69099999999999995,"15min":0.62},"max":{"1min":1.077,"5min":1.2549999999999999,"15min":1.4399999999999999},"last":0.95499999999999996}]},{"osd":4,"last update":"Wed Jun 29 17:08:21 2022","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.79800000000000004,"15min":0.79200000000000004},"min":{"1min":0.58999999999999997,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0189999999999999,"5min":1.321,"15min":1.4890000000000001},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.86399999999999999,"15min":0.84699999999999998},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.55300000000000005},"max":{"1min":1.02,"5min":1.284,"15min":1.4239999999999999},"last":0.97799999999999998}]},{"osd":5,"last update":"Wed Jun 29 17:08:21 2022","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.91100000000000003,"15min":0.90900000000000003},"min":{"1min":0.68899999999999995,"5min":0.58299999999999996,"15min":0.55900000000000005},"max":{"1min":1.139,"5min":1.2370000000000001,"15min":1.3149999999999999},"last":1.022},{"interface":"front","average":{"1min":1.0109999999999999,"5min":0.98799999999999999,"15min":0.98899999999999999},"min":{"1min":0.90200000000000002,"5min":0.621,"15min":0.621},"max":{"1min":1.109,"5min":1.46,"15min":1.585},"last":1.113}]},{"osd":6,"last update":"Wed Jun 29 17:08:21 2022","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.93600000000000005,"15min":0.94999999999999996},"min":{"1min":0.69699999999999995,"5min":0.65500000000000003,"15min":0.63600000000000001},"max":{"1min":1.1839999999999999,"5min":1.4139999999999999,"15min":1.5449999999999999},"last":1.0669999999999999},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.94199999999999995},"min":{"1min":0.67100000000000004,"5min":0.64200000000000002,"15min":0.61199999999999999},"max":{"1min":1.109,"5min":1.353,"15min":1.55},"last":1.0529999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575701,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798144,"kb_used_data":92901236,"kb_used_omap":4008,"kb_used_meta":0,"kb_avail":92901236,"statfs":{"total":95948165120,"available":95130865664,"internally_reserved":0,"allocated":95130865664,"data_stored":95130865664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4104844,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jun 29 17:08:41 2022","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.76800000000000002,"15min":0.75800000000000001},"min":{"1min":0.59899999999999998,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.159},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.78200000000000003,"15min":0.76800000000000002},"min":{"1min":0.55300000000000005,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.089,"5min":1.089,"15min":1.177},"last":0.64700000000000002}]},{"osd":1,"last update":"Wed Jun 29 17:08:41 2022","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.86499999999999999,"15min":0.85299999999999998},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.0740000000000001,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.90900000000000003,"15min":0.91200000000000003},"min":{"1min":0.621,"5min":0.621,"15min":0.58199999999999996},"max":{"1min":1.0469999999999999,"5min":1.1399999999999999,"15min":1.3220000000000001},"last":0.78400000000000003}]},{"osd":2,"last update":"Wed Jun 29 17:08:41 2022","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.88300000000000001,"15min":0.85799999999999998},"min":{"1min":0.64700000000000002,"5min":0.60999999999999999,"15min":0.55500000000000005},"max":{"1min":1.0389999999999999,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.877,"5min":0.88900000000000001,"15min":0.89800000000000002},"min":{"1min":0.59199999999999997,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.026,"5min":1.1950000000000001,"15min":1.2090000000000001},"last":1.024}]},{"osd":3,"last update":"Wed Jun 29 17:08:41 2022","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.82899999999999996,"15min":0.83199999999999996},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.51700000000000002},"max":{"1min":1.024,"5min":1.1719999999999999,"15min":1.4139999999999999},"last":1.0529999999999999},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.82699999999999996,"15min":0.82999999999999996},"min":{"1min":0.628,"5min":0.60999999999999999,"15min":0.60799999999999998},"max":{"1min":1.032,"5min":1.0880000000000001,"15min":1.161},"last":0.89600000000000002}]},{"osd":4,"last update":"Wed Jun 29 17:08:41 2022","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.77300000000000002,"15min":0.80100000000000005},"min":{"1min":0.52800000000000002,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.0609999999999999,"5min":1.1799999999999999,"15min":1.3180000000000001},"last":0.96299999999999997},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.92900000000000005,"15min":0.93300000000000005},"min":{"1min":0.57499999999999996,"5min":0.53000000000000003,"15min":0.52700000000000002},"max":{"1min":1.1080000000000001,"5min":1.2270000000000001,"15min":1.4850000000000001},"last":1.133}]},{"osd":5,"last update":"Wed Jun 29 17:08:41 2022","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.85399999999999998,"15min":0.83999999999999997},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.48999999999999999},"max":{"1min":1.175,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.94899999999999995,"15min":0.98399999999999999},"min":{"1min":0.64300000000000002,"5min":0.59699999999999998,"15min":0.55100000000000005},"max":{"1min":1.121,"5min":1.2330000000000001,"15min":1.4079999999999999},"last":1.0820000000000001}]},{"osd":7,"last update":"Wed Jun 29 17:08:41 2022","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.92100000000000004,"15min":0.93300000000000005},"min":{"1min":0.60999999999999999,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.0269999999999999,"5min":1.3700000000000001,"15min":1.532},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.96999999999999997,"5min":0.995,"15min":1.0029999999999999},"min":{"1min":0.80900000000000005,"5min":0.66300000000000003,"15min":0.60199999999999998},"max":{"1min":1.2350000000000001,"5min":1.2430000000000001,"15min":1.319},"last":0.99399999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575699,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798144,"kb_used_data":92901236,"kb_used_omap":2743,"kb_used_meta":0,"kb_avail":92901236,"statfs":{"total":95948165120,"available":95130865664,"internally_reserved":0,"allocated":95130865664,"data_stored":95130865664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2808996,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jun 29 17:08:27 2022","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.84999999999999998,"15min":0.85399999999999998},"min":{"1min":0.68600000000000005,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.1160000000000001,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.93600000000000005,"15min":0.93500000000000005},"min":{"1min":0.76800000000000002,"5min":0.623,"15min":0.52500000000000002},"max":{"1min":1.0860000000000001,"5min":1.167,"15min":1.286},"last":1.056}]},{"osd":1,"last update":"Wed Jun 29 17:08:27 2022","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.84099999999999997,"15min":0.83199999999999996},"min":{"1min":0.66100000000000003,"5min":0.57099999999999995,"15min":0.55100000000000005},"max":{"1min":1.0700000000000001,"5min":1.1299999999999999,"15min":1.339},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.84099999999999997,"15min":0.84599999999999997},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.22},"last":0.77300000000000002}]},{"osd":2,"last update":"Wed Jun 29 17:08:27 2022","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.86499999999999999,"15min":0.86599999999999999},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1910000000000001},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.60399999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0269999999999999,"5min":1.0940000000000001,"15min":1.131},"last":0.93899999999999995}]},{"osd":3,"last update":"Wed Jun 29 17:08:27 2022","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.86599999999999999,"15min":0.86099999999999999},"min":{"1min":0.66800000000000004,"5min":0.64300000000000002,"15min":0.59399999999999997},"max":{"1min":1.131,"5min":1.131,"15min":1.379},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.94599999999999995,"15min":0.92100000000000004},"min":{"1min":0.78600000000000003,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.84299999999999997}]},{"osd":4,"last update":"Wed Jun 29 17:08:27 2022","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.78800000000000003,"15min":0.78100000000000003},"min":{"1min":0.54100000000000004,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.0509999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.872},{"interface":"front","average":{"1min":1.006,"5min":0.995,"15min":0.995},"min":{"1min":0.82599999999999996,"5min":0.71799999999999997,"15min":0.67500000000000004},"max":{"1min":1.258,"5min":1.258,"15min":1.4179999999999999},"last":1.0029999999999999}]},{"osd":6,"last update":"Wed Jun 29 17:08:27 2022","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.90200000000000002,"15min":0.876},"min":{"1min":0.58099999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.171,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.90200000000000002,"15min":0.88900000000000001},"min":{"1min":0.73099999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0600000000000001,"5min":1.4159999999999999,"15min":1.4159999999999999},"last":0.74399999999999999}]},{"osd":7,"last update":"Wed Jun 29 17:08:27 2022","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.94499999999999995},"min":{"1min":0.64500000000000002,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.212,"5min":1.3959999999999999,"15min":1.4570000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.0469999999999999,"15min":1.0069999999999999},"min":{"1min":0.85599999999999998,"5min":0.64100000000000001,"15min":0.63400000000000001},"max":{"1min":1.3300000000000001,"5min":1.6850000000000001,"15min":1.6950000000000001},"last":0.96799999999999997}]}]},{"osd":4,"up_from":13,"seq":55834575701,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798016,"kb_used_data":92901364,"kb_used_omap":3569,"kb_used_meta":0,"kb_avail":92901364,"statfs":{"total":95948165120,"available":95130996736,"internally_reserved":0,"allocated":95130996736,"data_stored":95130996736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3654826,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jun 29 17:08:23 2022","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.79500000000000004,"15min":0.79600000000000004},"min":{"1min":0.57299999999999995,"5min":0.55300000000000005,"15min":0.53100000000000003},"max":{"1min":1.083,"5min":1.117,"15min":1.1579999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.79400000000000004,"15min":0.79700000000000004},"min":{"1min":0.61099999999999999,"5min":0.58199999999999996,"15min":0.52400000000000002},"max":{"1min":0.94499999999999995,"5min":1.135,"15min":1.1739999999999999},"last":0.85799999999999998}]},{"osd":1,"last update":"Wed Jun 29 17:08:23 2022","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.84099999999999997,"15min":0.82699999999999996},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.57299999999999995},"max":{"1min":1.024,"5min":1.0800000000000001,"15min":1.1359999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81399999999999995,"15min":0.82599999999999996},"min":{"1min":0.63500000000000001,"5min":0.59899999999999998,"15min":0.57999999999999996},"max":{"1min":0.96699999999999997,"5min":1.0940000000000001,"15min":1.1930000000000001},"last":0.82799999999999996}]},{"osd":2,"last update":"Wed Jun 29 17:08:23 2022","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.90800000000000003,"15min":0.91700000000000004},"min":{"1min":0.625,"5min":0.59699999999999998,"15min":0.57699999999999996},"max":{"1min":1.0840000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.996},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.92100000000000004,"15min":0.90800000000000003},"min":{"1min":0.67200000000000004,"5min":0.61399999999999999,"15min":0.60199999999999998},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.23},"last":0.90800000000000003}]},{"osd":3,"last update":"Wed Jun 29 17:08:23 2022","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.81699999999999995,"15min":0.81100000000000005},"min":{"1min":0.64600000000000002,"5min":0.628,"15min":0.56799999999999995},"max":{"1min":0.995,"5min":1.212,"15min":1.212},"last":1.0349999999999999},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.89400000000000002,"15min":0.90000000000000002},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.60799999999999998},"max":{"1min":1.0549999999999999,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.70299999999999996}]},{"osd":5,"last update":"Wed Jun 29 17:08:23 2022","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.85099999999999998,"15min":0.84299999999999997},"min":{"1min":0.54300000000000004,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.101,"5min":1.133,"15min":1.2450000000000001},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.91200000000000003,"15min":0.90200000000000002},"min":{"1min":0.61199999999999999,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.135,"5min":1.1819999999999999,"15min":1.248},"last":1.121}]},{"osd":6,"last update":"Wed Jun 29 17:08:23 2022","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.88600000000000001,"15min":0.88500000000000001},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.54600000000000004},"max":{"1min":1.083,"5min":1.417,"15min":1.417},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.98899999999999999,"15min":0.97399999999999998},"min":{"1min":0.67400000000000004,"5min":0.60099999999999998,"15min":0.53200000000000003},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.2889999999999999},"last":0.95499999999999996}]},{"osd":7,"last update":"Wed Jun 29 17:08:23 2022","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.88600000000000001,"15min":0.874},"min":{"1min":0.61599999999999999,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.1140000000000001,"5min":1.4319999999999999,"15min":1.4950000000000001},"last":0.97499999999999998},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.97599999999999998,"15min":0.95699999999999996},"min":{"1min":0.72699999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.1519999999999999,"5min":1.466,"15min":1.488},"last":1.0940000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575700,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798144,"kb_used_data":92901236,"kb_used_omap":3345,"kb_used_meta":0,"kb_avail":92901236,"statfs":{"total":95948165120,"available":95130865664,"internally_reserved":0,"allocated":95130865664,"data_stored":95130865664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425940,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jun 29 17:08:18 2022","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.63700000000000001,"15min":0.63300000000000001},"min":{"1min":0.47199999999999998,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.95599999999999996,"5min":1.087,"15min":1.1259999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.73499999999999999,"15min":0.74099999999999999},"min":{"1min":0.63100000000000001,"5min":0.57999999999999996,"15min":0.56799999999999995},"max":{"1min":0.93400000000000005,"5min":0.96999999999999997,"15min":1.052},"last":0.72099999999999997}]},{"osd":1,"last update":"Wed Jun 29 17:08:18 2022","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.67100000000000004,"15min":0.67200000000000004},"min":{"1min":0.51100000000000001,"5min":0.49199999999999999,"15min":0.44700000000000001},"max":{"1min":0.876,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.871,"5min":0.84499999999999997,"15min":0.85199999999999998},"min":{"1min":0.626,"5min":0.58499999999999996,"15min":0.53000000000000003},"max":{"1min":1.155,"5min":1.155,"15min":1.159},"last":0.84399999999999997}]},{"osd":2,"last update":"Wed Jun 29 17:08:18 2022","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.77100000000000002,"15min":0.77700000000000002},"min":{"1min":0.56000000000000005,"5min":0.54200000000000004,"15min":0.54100000000000004},"max":{"1min":1.042,"5min":1.194,"15min":1.198},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.71599999999999997,"15min":0.70899999999999996},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.498},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.59199999999999997}]},{"osd":4,"last update":"Wed Jun 29 17:08:18 2022","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.97299999999999998,"15min":0.97899999999999998},"min":{"1min":0.73899999999999999,"5min":0.72099999999999997,"15min":0.64700000000000002},"max":{"1min":1.2030000000000001,"5min":1.2210000000000001,"15min":1.23},"last":1.042},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.90600000000000003,"15min":0.90700000000000003},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.621},"max":{"1min":1.256,"5min":1.256,"15min":1.4490000000000001},"last":0.79000000000000004}]},{"osd":5,"last update":"Wed Jun 29 17:08:18 2022","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.876,"15min":0.89600000000000002},"min":{"1min":0.67600000000000005,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.0169999999999999,"5min":1.2849999999999999,"15min":1.2889999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.038,"5min":1.02,"15min":1.04},"min":{"1min":0.87,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.284,"5min":1.45,"15min":1.5309999999999999},"last":1.083}]},{"osd":6,"last update":"Wed Jun 29 17:08:18 2022","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.96199999999999997,"15min":0.97899999999999998},"min":{"1min":0.77300000000000002,"5min":0.74399999999999999,"15min":0.72499999999999998},"max":{"1min":1.304,"5min":1.3460000000000001,"15min":1.542},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.998,"5min":1.0149999999999999,"15min":1.0249999999999999},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.75800000000000001},"max":{"1min":1.1120000000000001,"5min":1.345,"15min":1.5289999999999999},"last":0.874}]},{"osd":7,"last update":"Wed Jun 29 17:08:18 2022","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.0860000000000001,"15min":1.0880000000000001},"min":{"1min":0.84499999999999997,"5min":0.77600000000000002,"15min":0.69399999999999995},"max":{"1min":1.323,"5min":1.4730000000000001,"15min":1.5920000000000001},"last":0.96899999999999997},{"interface":"front","average":{"1min":1.016,"5min":1.0449999999999999,"15min":1.0569999999999999},"min":{"1min":0.77200000000000002,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.206,"5min":1.395,"15min":1.47},"last":0.94999999999999996}]}]},{"osd":2,"up_from":13,"seq":55834575701,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798144,"kb_used_data":92901236,"kb_used_omap":2825,"kb_used_meta":0,"kb_avail":92901236,"statfs":{"total":95948165120,"available":95130865664,"internally_reserved":0,"allocated":95130865664,"data_stored":95130865664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2893662,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jun 29 17:08:34 2022","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61899999999999999,"15min":0.62},"min":{"1min":0.48899999999999999,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.86899999999999999,"5min":1.167,"15min":1.3080000000000001},"last":0.621},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.70999999999999996,"15min":0.71099999999999997},"min":{"1min":0.51600000000000001,"5min":0.48699999999999999,"15min":0.48499999999999999},"max":{"1min":1.1699999999999999,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.52800000000000002}]},{"osd":1,"last update":"Wed Jun 29 17:08:34 2022","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66000000000000003,"15min":0.65200000000000002},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.46999999999999997},"max":{"1min":0.78300000000000003,"5min":1.6799999999999999,"15min":1.6799999999999999},"last":0.753},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.745,"15min":0.73399999999999999},"min":{"1min":0.56299999999999994,"5min":0.55300000000000005,"15min":0.52900000000000003},"max":{"1min":0.96599999999999997,"5min":1.1899999999999999,"15min":1.206},"last":0.73199999999999998}]},{"osd":3,"last update":"Wed Jun 29 17:08:34 2022","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.85499999999999998,"15min":0.84599999999999997},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.55500000000000005},"max":{"1min":0.999,"5min":1.732,"15min":1.732},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.81000000000000005,"15min":0.80600000000000005},"min":{"1min":0.60899999999999999,"5min":0.58399999999999996,"15min":0.51600000000000001},"max":{"1min":1.0469999999999999,"5min":1.0649999999999999,"15min":1.1599999999999999},"last":1.212}]},{"osd":4,"last update":"Wed Jun 29 17:08:34 2022","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.92900000000000005,"15min":0.92600000000000005},"min":{"1min":0.76300000000000001,"5min":0.71399999999999997,"15min":0.60899999999999999},"max":{"1min":1.2729999999999999,"5min":1.2749999999999999,"15min":1.421},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.95599999999999996,"15min":0.94599999999999995},"min":{"1min":0.73499999999999999,"5min":0.72299999999999998,"15min":0.66700000000000004},"max":{"1min":1.302,"5min":1.772,"15min":1.772},"last":0.83799999999999997}]},{"osd":5,"last update":"Wed Jun 29 17:08:34 2022","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":0.96899999999999997,"15min":0.96799999999999997},"min":{"1min":0.73299999999999998,"5min":0.68100000000000005,"15min":0.67000000000000004},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.4350000000000001},"last":1.1299999999999999},{"interface":"front","average":{"1min":0.997,"5min":0.997,"15min":0.98999999999999999},"min":{"1min":0.79300000000000004,"5min":0.72499999999999998,"15min":0.68999999999999995},"max":{"1min":1.2689999999999999,"5min":1.415,"15min":1.4930000000000001},"last":1.014}]},{"osd":6,"last update":"Wed Jun 29 17:08:34 2022","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.0489999999999999,"15min":1.024},"min":{"1min":0.84399999999999997,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.397,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.0640000000000001,"15min":1.0600000000000001},"min":{"1min":0.81200000000000006,"5min":0.752,"15min":0.745},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.5549999999999999},"last":1.278}]},{"osd":7,"last update":"Wed Jun 29 17:08:34 2022","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0600000000000001,"15min":1.0620000000000001},"min":{"1min":0.92200000000000004,"5min":0.75700000000000001,"15min":0.70299999999999996},"max":{"1min":1.3640000000000001,"5min":1.54,"15min":1.54},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.125,"15min":1.1000000000000001},"min":{"1min":0.98399999999999999,"5min":0.84299999999999997,"15min":0.76800000000000002},"max":{"1min":1.4550000000000001,"5min":1.845,"15min":1.845},"last":0.97299999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575699,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798144,"kb_used_data":92901236,"kb_used_omap":3969,"kb_used_meta":0,"kb_avail":92901236,"statfs":{"total":95948165120,"available":95130865664,"internally_reserved":0,"allocated":95130865664,"data_stored":95130865664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4064788,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jun 29 17:08:07 2022","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59899999999999998,"15min":0.61799999999999999},"min":{"1min":0.433,"5min":0.432,"15min":0.432},"max":{"1min":0.87,"5min":0.93100000000000005,"15min":1.04},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.72199999999999998,"15min":0.70999999999999996},"min":{"1min":0.501,"5min":0.501,"15min":0.47699999999999998},"max":{"1min":0.91000000000000003,"5min":1.278,"15min":1.278},"last":0.65800000000000003}]},{"osd":2,"last update":"Wed Jun 29 17:08:07 2022","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.64200000000000002,"15min":0.66200000000000003},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.0389999999999999,"5min":1.042,"15min":1.569},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.58899999999999997,"5min":0.52000000000000002,"15min":0.48299999999999998},"max":{"1min":1.079,"5min":1.111,"15min":1.111},"last":0.81799999999999995}]},{"osd":3,"last update":"Wed Jun 29 17:08:07 2022","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.89200000000000002,"15min":0.89600000000000002},"min":{"1min":0.70299999999999996,"5min":0.629,"15min":0.51800000000000002},"max":{"1min":1.26,"5min":1.26,"15min":1.425},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.83599999999999997,"15min":0.83399999999999996},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.0629999999999999,"5min":1.1539999999999999,"15min":1.4350000000000001},"last":0.93999999999999995}]},{"osd":4,"last update":"Wed Jun 29 17:08:07 2022","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.88200000000000001,"15min":0.89000000000000001},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.2190000000000001,"5min":1.2310000000000001,"15min":1.395},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97599999999999998,"15min":0.96999999999999997},"min":{"1min":0.72099999999999997,"5min":0.69299999999999995,"15min":0.65200000000000002},"max":{"1min":1.387,"5min":1.663,"15min":1.663},"last":1.0720000000000001}]},{"osd":5,"last update":"Wed Jun 29 17:08:07 2022","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.95299999999999996,"15min":0.96999999999999997},"min":{"1min":0.71499999999999997,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.343,"5min":1.4099999999999999,"15min":1.637},"last":1.3069999999999999},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.96099999999999997,"15min":0.98899999999999999},"min":{"1min":0.68400000000000005,"5min":0.68300000000000005,"15min":0.68200000000000005},"max":{"1min":1.3180000000000001,"5min":1.369,"15min":1.381},"last":1.0580000000000001}]},{"osd":6,"last update":"Wed Jun 29 17:08:07 2022","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.046,"15min":1.0489999999999999},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.69399999999999995},"max":{"1min":1.617,"5min":1.73,"15min":1.73},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.097,"15min":1.093},"min":{"1min":0.81899999999999995,"5min":0.81399999999999995,"15min":0.71699999999999997},"max":{"1min":1.3959999999999999,"5min":1.635,"15min":1.7470000000000001},"last":1.359}]},{"osd":7,"last update":"Wed Jun 29 17:08:07 2022","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.0980000000000001,"15min":1.1060000000000001},"min":{"1min":0.78600000000000003,"5min":0.73499999999999999,"15min":0.69099999999999995},"max":{"1min":1.454,"5min":1.5589999999999999,"15min":1.704},"last":1.222},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.0580000000000001,"15min":1.093},"min":{"1min":0.73699999999999999,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.516,"5min":1.516,"15min":1.516},"last":1.256}]}]},{"osd":0,"up_from":13,"seq":55834575700,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798144,"kb_used_data":92901236,"kb_used_omap":3795,"kb_used_meta":0,"kb_avail":92901236,"statfs":{"total":95948165120,"available":95130865664,"internally_reserved":0,"allocated":95130865664,"data_stored":95130865664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3886185,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Jun 29 17:08:25 2022","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.61699999999999999,"15min":0.60399999999999998},"min":{"1min":0.46800000000000003,"5min":0.45200000000000001,"15min":0.45100000000000001},"max":{"1min":1.1579999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.68799999999999994,"15min":0.66800000000000004},"min":{"1min":0.53000000000000003,"5min":0.48799999999999999,"15min":0.48399999999999999},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.51200000000000001}]},{"osd":2,"last update":"Wed Jun 29 17:08:25 2022","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81499999999999995,"15min":0.80500000000000005},"min":{"1min":0.60799999999999998,"5min":0.53200000000000003,"15min":0.5},"max":{"1min":1.02,"5min":1.48,"15min":1.48},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.71199999999999997,"15min":0.72099999999999997},"min":{"1min":0.505,"5min":0.505,"15min":0.46400000000000002},"max":{"1min":1.02,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":0.58699999999999997}]},{"osd":3,"last update":"Wed Jun 29 17:08:25 2022","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.751,"15min":0.746},"min":{"1min":0.56499999999999995,"5min":0.52000000000000002,"15min":0.5},"max":{"1min":1.008,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.877,"15min":0.877},"min":{"1min":0.72799999999999998,"5min":0.66000000000000003,"15min":0.64300000000000002},"max":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.333},"last":0.82199999999999995}]},{"osd":4,"last update":"Wed Jun 29 17:08:25 2022","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.86699999999999999,"15min":0.86399999999999999},"min":{"1min":0.66300000000000003,"5min":0.628,"15min":0.628},"max":{"1min":1.0349999999999999,"5min":1.173,"15min":1.3560000000000001},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.94399999999999995,"15min":0.92300000000000004},"min":{"1min":0.751,"5min":0.70599999999999996,"15min":0.66000000000000003},"max":{"1min":1.1830000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":0.86099999999999999}]},{"osd":5,"last update":"Wed Jun 29 17:08:25 2022","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.94399999999999995,"15min":0.95299999999999996},"min":{"1min":0.68899999999999995,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.252,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.97099999999999997},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.004,"15min":0.98799999999999999},"min":{"1min":0.83699999999999997,"5min":0.70699999999999996,"15min":0.68300000000000005},"max":{"1min":1.3340000000000001,"5min":1.3999999999999999,"15min":1.6140000000000001},"last":0.92300000000000004}]},{"osd":6,"last update":"Wed Jun 29 17:08:25 2022","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.0600000000000001,"15min":1.0329999999999999},"min":{"1min":0.872,"5min":0.70499999999999996,"15min":0.68100000000000005},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.554},"last":0.88200000000000001},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0069999999999999,"15min":1.0109999999999999},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.3180000000000001,"5min":1.361,"15min":1.48},"last":0.94199999999999995}]},{"osd":7,"last update":"Wed Jun 29 17:08:25 2022","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.0409999999999999,"15min":1.0229999999999999},"min":{"1min":0.84799999999999998,"5min":0.77200000000000002,"15min":0.71699999999999997},"max":{"1min":1.522,"5min":1.522,"15min":1.5429999999999999},"last":0.89700000000000002},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.083,"15min":1.075},"min":{"1min":0.88500000000000001,"5min":0.81000000000000005,"15min":0.67700000000000005},"max":{"1min":1.3140000000000001,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":1.044}]}]}]}} 2022-06-29T17:09:03.519 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-06-29T17:09:03.624 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.619+0000 7fef23fff700 1 Processor -- start 2022-06-29T17:09:03.625 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.619+0000 7fef23fff700 1 -- start start 2022-06-29T17:09:03.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.619+0000 7fef23fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef24065450 0x7fef240b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:03.629 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.619+0000 7fef23fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fef24060760 0x7fef24140180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:03.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.619+0000 7fef23fff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef24140670 con 0x7fef240b28e0 2022-06-29T17:09:03.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.619+0000 7fef23fff700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef2413fa40 con 0x7fef24060760 2022-06-29T17:09:03.630 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.619+0000 7fef23fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef2413fce0 con 0x7fef24065450 2022-06-29T17:09:03.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.619+0000 7fef237fe700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fef24060760 0x7fef24140180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:03.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.619+0000 7fef227fc700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef24065450 0x7fef240b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:03.631 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.619+0000 7fef237fe700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fef24060760 0x7fef24140180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:60888/0 (socket says 172.21.15.60:60888) 2022-06-29T17:09:03.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.619+0000 7fef227fc700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef24065450 0x7fef240b2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:39716/0 (socket says 172.21.15.60:39716) 2022-06-29T17:09:03.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.619+0000 7fef237fe700 1 -- 172.21.15.60:0/4195519446 learned_addr learned my addr 172.21.15.60:0/4195519446 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T17:09:03.632 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.619+0000 7fef22ffd700 1 --1- 172.21.15.60:0/4195519446 >> v1:172.21.15.92:6789/0 conn(0x7fef240b28e0 0x7fef24139340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.60:38956/0 (socket says 172.21.15.60:38956) 2022-06-29T17:09:03.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.619+0000 7fef227fc700 1 -- 172.21.15.60:0/4195519446 >> v2:172.21.15.92:3300/0 conn(0x7fef24060760 msgr2=0x7fef24140180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T17:09:03.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.619+0000 7fef227fc700 1 --2- 172.21.15.60:0/4195519446 >> v2:172.21.15.92:3300/0 conn(0x7fef24060760 0x7fef24140180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:03.633 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.619+0000 7fef227fc700 1 -- 172.21.15.60:0/4195519446 >> v1:172.21.15.92:6789/0 conn(0x7fef240b28e0 legacy=0x7fef24139340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:03.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.619+0000 7fef227fc700 1 -- 172.21.15.60:0/4195519446 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef24140860 con 0x7fef24065450 2022-06-29T17:09:03.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.619+0000 7fef227fc700 1 --2- 172.21.15.60:0/4195519446 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef24065450 0x7fef240b2cc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fef24064180 tx=0x7fef0c006df0).ready entity=mon.2 client_cookie=e57ab7e8a198e03b server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:03.634 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.619+0000 7fef21ffb700 1 -- 172.21.15.60:0/4195519446 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef0c00d570 con 0x7fef24065450 2022-06-29T17:09:03.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.619+0000 7fef21ffb700 1 -- 172.21.15.60:0/4195519446 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef0c000e90 con 0x7fef24065450 2022-06-29T17:09:03.635 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.619+0000 7fef21ffb700 1 -- 172.21.15.60:0/4195519446 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef0c00d8c0 con 0x7fef24065450 2022-06-29T17:09:03.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.623+0000 7fef23fff700 1 -- 172.21.15.60:0/4195519446 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef24065450 msgr2=0x7fef240b2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:03.636 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.623+0000 7fef23fff700 1 --2- 172.21.15.60:0/4195519446 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef24065450 0x7fef240b2cc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fef24064180 tx=0x7fef0c006df0).stop 2022-06-29T17:09:03.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.623+0000 7fef23fff700 1 -- 172.21.15.60:0/4195519446 shutdown_connections 2022-06-29T17:09:03.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.623+0000 7fef23fff700 1 --2- 172.21.15.60:0/4195519446 >> v2:172.21.15.92:3300/0 conn(0x7fef24060760 0x7fef24140180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:03.637 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.623+0000 7fef23fff700 1 --2- 172.21.15.60:0/4195519446 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef24065450 0x7fef240b2cc0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:03.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.623+0000 7fef23fff700 1 -- 172.21.15.60:0/4195519446 >> 172.21.15.60:0/4195519446 conn(0x7fef24061c80 msgr2=0x7fef240637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:03.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.623+0000 7fef23fff700 1 -- 172.21.15.60:0/4195519446 shutdown_connections 2022-06-29T17:09:03.638 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.623+0000 7fef23fff700 1 -- 172.21.15.60:0/4195519446 wait complete. 2022-06-29T17:09:03.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.623+0000 7fef23fff700 1 Processor -- start 2022-06-29T17:09:03.639 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.623+0000 7fef23fff700 1 -- start start 2022-06-29T17:09:03.642 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.623+0000 7fef23fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef24060760 0x7fef24142170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:03.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.623+0000 7fef23fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef24065450 0x7fef24134290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:03.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.623+0000 7fef23fff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef240b28e0 0x7fef2412a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:03.643 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.623+0000 7fef23fff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef241361a0 con 0x7fef24065450 2022-06-29T17:09:03.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.623+0000 7fef23fff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef24137690 con 0x7fef24060760 2022-06-29T17:09:03.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.623+0000 7fef23fff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef2412b040 con 0x7fef240b28e0 2022-06-29T17:09:03.644 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.623+0000 7fef227fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef24065450 0x7fef24134290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:03.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.623+0000 7fef227fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef24065450 0x7fef24134290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54862/0 (socket says 172.21.15.60:54862) 2022-06-29T17:09:03.645 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.623+0000 7fef227fc700 1 -- 172.21.15.60:0/4067366402 learned_addr learned my addr 172.21.15.60:0/4067366402 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T17:09:03.646 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.623+0000 7fef237fe700 1 --2- 172.21.15.60:0/4067366402 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef240b28e0 0x7fef2412a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:03.646 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.623+0000 7fef22ffd700 1 --2- 172.21.15.60:0/4067366402 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef24060760 0x7fef24142170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:03.647 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.627+0000 7fef22ffd700 1 -- 172.21.15.60:0/4067366402 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef240b28e0 msgr2=0x7fef2412a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:03.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.627+0000 7fef22ffd700 1 --2- 172.21.15.60:0/4067366402 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef240b28e0 0x7fef2412a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:03.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.627+0000 7fef22ffd700 1 -- 172.21.15.60:0/4067366402 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef24065450 msgr2=0x7fef24134290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:03.648 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.627+0000 7fef22ffd700 1 --2- 172.21.15.60:0/4067366402 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef24065450 0x7fef24134290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:03.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.627+0000 7fef22ffd700 1 -- 172.21.15.60:0/4067366402 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef0c00b180 con 0x7fef24060760 2022-06-29T17:09:03.649 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.627+0000 7fef227fc700 1 --2- 172.21.15.60:0/4067366402 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef24065450 0x7fef24134290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T17:09:03.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.627+0000 7fef22ffd700 1 --2- 172.21.15.60:0/4067366402 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef24060760 0x7fef24142170 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0x7fef240642d0 tx=0x7fef14007ba0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:03.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.627+0000 7fef1bfff700 1 -- 172.21.15.60:0/4067366402 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef1400ccd0 con 0x7fef24060760 2022-06-29T17:09:03.650 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.627+0000 7fef1bfff700 1 -- 172.21.15.60:0/4067366402 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef1400ce30 con 0x7fef24060760 2022-06-29T17:09:03.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.627+0000 7fef23fff700 1 -- 172.21.15.60:0/4067366402 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef24134780 con 0x7fef24060760 2022-06-29T17:09:03.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.627+0000 7fef1bfff700 1 -- 172.21.15.60:0/4067366402 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef140072c0 con 0x7fef24060760 2022-06-29T17:09:03.651 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.627+0000 7fef23fff700 1 -- 172.21.15.60:0/4067366402 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef24137310 con 0x7fef24060760 2022-06-29T17:09:03.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.627+0000 7fef23fff700 1 -- 172.21.15.60:0/4067366402 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef24136310 con 0x7fef24060760 2022-06-29T17:09:03.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.627+0000 7fef1bfff700 1 -- 172.21.15.60:0/4067366402 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fef1401a070 con 0x7fef24060760 2022-06-29T17:09:03.652 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.631+0000 7fef1bfff700 1 --2- 172.21.15.60:0/4067366402 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fef08053b30 0x7fef08055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:03.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.631+0000 7fef1bfff700 1 -- 172.21.15.60:0/4067366402 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fef1406ab50 con 0x7fef24060760 2022-06-29T17:09:03.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.631+0000 7fef227fc700 1 --2- 172.21.15.60:0/4067366402 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fef08053b30 0x7fef08055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:03.653 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.631+0000 7fef227fc700 1 --2- 172.21.15.60:0/4067366402 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fef08053b30 0x7fef08055ff0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 rx=0x7fef0c01a000 tx=0x7fef0c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:03.654 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.635+0000 7fef1bfff700 1 -- 172.21.15.60:0/4067366402 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fef24136310 con 0x7fef24060760 2022-06-29T17:09:03.771 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.763+0000 7fef23fff700 1 -- 172.21.15.60:0/4067366402 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef24136310 con 0x7fef24060760 2022-06-29T17:09:03.773 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.767+0000 7fef1bfff700 1 -- 172.21.15.60:0/4067366402 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v150) v1 ==== 74+0+13320 (secure 0 0 0) 0x7fef24136310 con 0x7fef24060760 2022-06-29T17:09:03.773 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T17:09:03.774 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":150,"fsid":"e8f11d88-dd6e-49d6-9ee8-3fe67dcd3b85","created":"2022-06-29T15:58:07.965719+0000","modified":"2022-06-29T16:08:32.204356+0000","last_up_change":"2022-06-29T15:58:13.784088+0000","last_in_change":"2022-06-29T15:58:13.784088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-06-29T15:58:10.961127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-06-29T15:58:19.184276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"11e7ab9e-fce2-470f-8195-ea4c3c52df22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":12890},{"type":"v1","addr":"172.21.15.60:6825","nonce":12890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":12890},{"type":"v1","addr":"172.21.15.60:6827","nonce":12890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":12890},{"type":"v1","addr":"172.21.15.60:6831","nonce":12890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":12890},{"type":"v1","addr":"172.21.15.60:6829","nonce":12890}]},"public_addr":"172.21.15.60:6825/12890","cluster_addr":"172.21.15.60:6827/12890","heartbeat_back_addr":"172.21.15.60:6831/12890","heartbeat_front_addr":"172.21.15.60:6829/12890","state":["exists","up"]},{"osd":1,"uuid":"ca43f136-6f69-4431-a157-fbe073fb585c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":12892},{"type":"v1","addr":"172.21.15.60:6818","nonce":12892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":12892},{"type":"v1","addr":"172.21.15.60:6820","nonce":12892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6823","nonce":12892},{"type":"v1","addr":"172.21.15.60:6824","nonce":12892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6821","nonce":12892},{"type":"v1","addr":"172.21.15.60:6822","nonce":12892}]},"public_addr":"172.21.15.60:6818/12892","cluster_addr":"172.21.15.60:6820/12892","heartbeat_back_addr":"172.21.15.60:6824/12892","heartbeat_front_addr":"172.21.15.60:6822/12892","state":["exists","up"]},{"osd":2,"uuid":"74fac45f-5f07-44ec-a9cb-2644c123fd4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":12891},{"type":"v1","addr":"172.21.15.60:6809","nonce":12891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":12891},{"type":"v1","addr":"172.21.15.60:6811","nonce":12891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":12891},{"type":"v1","addr":"172.21.15.60:6817","nonce":12891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":12891},{"type":"v1","addr":"172.21.15.60:6813","nonce":12891}]},"public_addr":"172.21.15.60:6809/12891","cluster_addr":"172.21.15.60:6811/12891","heartbeat_back_addr":"172.21.15.60:6817/12891","heartbeat_front_addr":"172.21.15.60:6813/12891","state":["exists","up"]},{"osd":3,"uuid":"2581d351-973a-4b4e-85ca-3658f0068765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":12889},{"type":"v1","addr":"172.21.15.60:6801","nonce":12889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":12889},{"type":"v1","addr":"172.21.15.60:6803","nonce":12889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":12889},{"type":"v1","addr":"172.21.15.60:6807","nonce":12889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":12889},{"type":"v1","addr":"172.21.15.60:6805","nonce":12889}]},"public_addr":"172.21.15.60:6801/12889","cluster_addr":"172.21.15.60:6803/12889","heartbeat_back_addr":"172.21.15.60:6807/12889","heartbeat_front_addr":"172.21.15.60:6805/12889","state":["exists","up"]},{"osd":4,"uuid":"3d234db9-87cd-4fe9-814f-0240a287d8b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13249},{"type":"v1","addr":"172.21.15.92:6810","nonce":13249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13249},{"type":"v1","addr":"172.21.15.92:6814","nonce":13249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13249},{"type":"v1","addr":"172.21.15.92:6825","nonce":13249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13249},{"type":"v1","addr":"172.21.15.92:6820","nonce":13249}]},"public_addr":"172.21.15.92:6810/13249","cluster_addr":"172.21.15.92:6814/13249","heartbeat_back_addr":"172.21.15.92:6825/13249","heartbeat_front_addr":"172.21.15.92:6820/13249","state":["exists","up"]},{"osd":5,"uuid":"e4bf1104-ff5c-438b-b608-8cee31c163f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":13248},{"type":"v1","addr":"172.21.15.92:6811","nonce":13248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":13248},{"type":"v1","addr":"172.21.15.92:6815","nonce":13248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":13248},{"type":"v1","addr":"172.21.15.92:6827","nonce":13248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":13248},{"type":"v1","addr":"172.21.15.92:6821","nonce":13248}]},"public_addr":"172.21.15.92:6811/13248","cluster_addr":"172.21.15.92:6815/13248","heartbeat_back_addr":"172.21.15.92:6827/13248","heartbeat_front_addr":"172.21.15.92:6821/13248","state":["exists","up"]},{"osd":6,"uuid":"8564dd25-27ab-4203-b54e-af6d3aa05857","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13250},{"type":"v1","addr":"172.21.15.92:6819","nonce":13250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13250},{"type":"v1","addr":"172.21.15.92:6826","nonce":13250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13250},{"type":"v1","addr":"172.21.15.92:6831","nonce":13250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":13250},{"type":"v1","addr":"172.21.15.92:6829","nonce":13250}]},"public_addr":"172.21.15.92:6819/13250","cluster_addr":"172.21.15.92:6826/13250","heartbeat_back_addr":"172.21.15.92:6831/13250","heartbeat_front_addr":"172.21.15.92:6829/13250","state":["exists","up"]},{"osd":7,"uuid":"bd513a9c-3b19-4314-9897-d300b5c897bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":13247},{"type":"v1","addr":"172.21.15.92:6801","nonce":13247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13247},{"type":"v1","addr":"172.21.15.92:6803","nonce":13247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13247},{"type":"v1","addr":"172.21.15.92:6807","nonce":13247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13247},{"type":"v1","addr":"172.21.15.92:6805","nonce":13247}]},"public_addr":"172.21.15.92:6801/13247","cluster_addr":"172.21.15.92:6803/13247","heartbeat_back_addr":"172.21.15.92:6807/13247","heartbeat_front_addr":"172.21.15.92:6805/13247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.891342+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.900866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.912755+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.864332+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.869041+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.895043+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-06-29T15:58:11.889491+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{"172.21.15.60:6833/3619856083":"2022-06-30T16:08:28.758125+0000","172.21.15.60:6832/3619856083":"2022-06-30T16:08:28.758125+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-06-29T17:09:03.776 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.771+0000 7fef23fff700 1 -- 172.21.15.60:0/4067366402 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fef08053b30 msgr2=0x7fef08055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:03.777 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.771+0000 7fef23fff700 1 --2- 172.21.15.60:0/4067366402 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fef08053b30 0x7fef08055ff0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 rx=0x7fef0c01a000 tx=0x7fef0c014000).stop 2022-06-29T17:09:03.777 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.771+0000 7fef23fff700 1 -- 172.21.15.60:0/4067366402 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef24060760 msgr2=0x7fef24142170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:03.777 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.771+0000 7fef23fff700 1 --2- 172.21.15.60:0/4067366402 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef24060760 0x7fef24142170 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0x7fef240642d0 tx=0x7fef14007ba0).stop 2022-06-29T17:09:03.778 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.771+0000 7fef23fff700 1 -- 172.21.15.60:0/4067366402 shutdown_connections 2022-06-29T17:09:03.778 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.771+0000 7fef23fff700 1 --2- 172.21.15.60:0/4067366402 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fef08053b30 0x7fef08055ff0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:03.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.771+0000 7fef23fff700 1 --2- 172.21.15.60:0/4067366402 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fef240b28e0 0x7fef2412a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:03.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.771+0000 7fef23fff700 1 --2- 172.21.15.60:0/4067366402 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef24065450 0x7fef24134290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:03.779 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.771+0000 7fef23fff700 1 --2- 172.21.15.60:0/4067366402 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef24060760 0x7fef24142170 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:03.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.771+0000 7fef23fff700 1 -- 172.21.15.60:0/4067366402 >> 172.21.15.60:0/4067366402 conn(0x7fef24061c80 msgr2=0x7fef240ad330 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:03.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.771+0000 7fef23fff700 1 -- 172.21.15.60:0/4067366402 shutdown_connections 2022-06-29T17:09:03.780 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:03.771+0000 7fef23fff700 1 -- 172.21.15.60:0/4067366402 wait complete. 2022-06-29T17:09:04.794 INFO:tasks.ceph:Scrubbing osd.0 2022-06-29T17:09:04.795 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2022-06-29T17:09:04.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.915+0000 7f5e1359e700 1 Processor -- start 2022-06-29T17:09:04.922 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.915+0000 7f5e1359e700 1 -- start start 2022-06-29T17:09:04.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.919+0000 7f5e1359e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5e140b3190 0x7f5e14062280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:04.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.919+0000 7f5e1359e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5e140ad930 0x7f5e14142f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:04.924 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.919+0000 7f5e1359e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e141428a0 con 0x7f5e1413c9a0 2022-06-29T17:09:04.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.919+0000 7f5e1359e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e14061530 con 0x7f5e140b3190 2022-06-29T17:09:04.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.919+0000 7f5e1359e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e140632a0 con 0x7f5e140ad930 2022-06-29T17:09:04.925 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.919+0000 7f5e11d9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5e140b3190 0x7f5e14062280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:04.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.919+0000 7f5e11d9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5e140b3190 0x7f5e14062280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54870/0 (socket says 172.21.15.60:54870) 2022-06-29T17:09:04.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.919+0000 7f5e11d9b700 1 -- 172.21.15.60:0/698491252 learned_addr learned my addr 172.21.15.60:0/698491252 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T17:09:04.926 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.919+0000 7f5e12d9d700 1 --2- 172.21.15.60:0/698491252 >> v2:172.21.15.92:3300/0 conn(0x7f5e140ad930 0x7f5e14142f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:04.929 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.919+0000 7f5e11d9b700 1 -- 172.21.15.60:0/698491252 >> v2:172.21.15.92:3300/0 conn(0x7f5e140ad930 msgr2=0x7f5e14142f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T17:09:04.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.919+0000 7f5e11d9b700 1 --2- 172.21.15.60:0/698491252 >> v2:172.21.15.92:3300/0 conn(0x7f5e140ad930 0x7f5e14142f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:04.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.919+0000 7f5e11d9b700 1 -- 172.21.15.60:0/698491252 >> v1:172.21.15.92:6789/0 conn(0x7f5e1413c9a0 legacy=0x7f5e140aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:04.930 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.919+0000 7f5e11d9b700 1 -- 172.21.15.60:0/698491252 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e1413cd80 con 0x7f5e140b3190 2022-06-29T17:09:04.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.919+0000 7f5e11d9b700 1 --2- 172.21.15.60:0/698491252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e140b3190 0x7f5e14062280 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 rx=0x7f5e08002f70 tx=0x7f5e08006eb0).ready entity=mon.0 client_cookie=5fa83d4c3e014655 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:04.931 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.919+0000 7f5e1159a700 1 -- 172.21.15.60:0/698491252 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e0800d5c0 con 0x7f5e140b3190 2022-06-29T17:09:04.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.919+0000 7f5e1159a700 1 -- 172.21.15.60:0/698491252 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e08000e90 con 0x7f5e140b3190 2022-06-29T17:09:04.932 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.919+0000 7f5e1159a700 1 -- 172.21.15.60:0/698491252 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e0800d950 con 0x7f5e140b3190 2022-06-29T17:09:04.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.919+0000 7f5e1359e700 1 -- 172.21.15.60:0/698491252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e140b3190 msgr2=0x7f5e14062280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:04.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.919+0000 7f5e1359e700 1 --2- 172.21.15.60:0/698491252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e140b3190 0x7f5e14062280 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 rx=0x7f5e08002f70 tx=0x7f5e08006eb0).stop 2022-06-29T17:09:04.933 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.919+0000 7f5e1359e700 1 -- 172.21.15.60:0/698491252 shutdown_connections 2022-06-29T17:09:04.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.919+0000 7f5e1359e700 1 --2- 172.21.15.60:0/698491252 >> v2:172.21.15.92:3300/0 conn(0x7f5e140ad930 0x7f5e14142f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:04.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.919+0000 7f5e1359e700 1 --2- 172.21.15.60:0/698491252 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e140b3190 0x7f5e14062280 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:04.934 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.919+0000 7f5e1359e700 1 -- 172.21.15.60:0/698491252 >> 172.21.15.60:0/698491252 conn(0x7f5e140b6f80 msgr2=0x7f5e140b7390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:04.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.919+0000 7f5e1359e700 1 -- 172.21.15.60:0/698491252 shutdown_connections 2022-06-29T17:09:04.935 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.919+0000 7f5e1359e700 1 -- 172.21.15.60:0/698491252 wait complete. 2022-06-29T17:09:04.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.923+0000 7f5e1359e700 1 Processor -- start 2022-06-29T17:09:04.936 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.923+0000 7f5e1359e700 1 -- start start 2022-06-29T17:09:04.940 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.923+0000 7f5e1359e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e140ad930 0x7f5e140b67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:04.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.923+0000 7f5e1359e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e140b3190 0x7f5e14066860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:04.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.923+0000 7f5e1359e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e1413c9a0 0x7f5e14066d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:04.941 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.923+0000 7f5e1359e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e140b0d40 con 0x7f5e140b3190 2022-06-29T17:09:04.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.923+0000 7f5e1359e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e14067240 con 0x7f5e140ad930 2022-06-29T17:09:04.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.923+0000 7f5e1359e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1413f8e0 con 0x7f5e1413c9a0 2022-06-29T17:09:04.942 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.923+0000 7f5e11d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e140b3190 0x7f5e14066860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:04.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.923+0000 7f5e1259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e140ad930 0x7f5e140b67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:04.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.923+0000 7f5e1259c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e140ad930 0x7f5e140b67d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:60904/0 (socket says 172.21.15.60:60904) 2022-06-29T17:09:04.943 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.923+0000 7f5e1259c700 1 -- 172.21.15.60:0/1588806201 learned_addr learned my addr 172.21.15.60:0/1588806201 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T17:09:04.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.927+0000 7f5e12d9d700 1 --2- 172.21.15.60:0/1588806201 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e1413c9a0 0x7f5e14066d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:04.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.927+0000 7f5e1259c700 1 -- 172.21.15.60:0/1588806201 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e1413c9a0 msgr2=0x7f5e14066d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:04.944 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.927+0000 7f5e1259c700 1 --2- 172.21.15.60:0/1588806201 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5e1413c9a0 0x7f5e14066d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:04.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.927+0000 7f5e1259c700 1 -- 172.21.15.60:0/1588806201 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e140b3190 msgr2=0x7f5e14066860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:04.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.927+0000 7f5e1259c700 1 --2- 172.21.15.60:0/1588806201 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e140b3190 0x7f5e14066860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:04.945 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.927+0000 7f5e1259c700 1 -- 172.21.15.60:0/1588806201 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e0800b1e0 con 0x7f5e140ad930 2022-06-29T17:09:04.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.927+0000 7f5e1259c700 1 --2- 172.21.15.60:0/1588806201 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e140ad930 0x7f5e140b67d0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0x7f5e140b7ca0 tx=0x7f5e04007c20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:04.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.927+0000 7f5e037fe700 1 -- 172.21.15.60:0/1588806201 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e04013450 con 0x7f5e140ad930 2022-06-29T17:09:04.946 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.927+0000 7f5e037fe700 1 -- 172.21.15.60:0/1588806201 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e040135b0 con 0x7f5e140ad930 2022-06-29T17:09:04.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.927+0000 7f5e1359e700 1 -- 172.21.15.60:0/1588806201 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e1413f3d0 con 0x7f5e140ad930 2022-06-29T17:09:04.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.927+0000 7f5e037fe700 1 -- 172.21.15.60:0/1588806201 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e04013910 con 0x7f5e140ad930 2022-06-29T17:09:04.947 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.927+0000 7f5e1359e700 1 -- 172.21.15.60:0/1588806201 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e1413efd0 con 0x7f5e140ad930 2022-06-29T17:09:04.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.927+0000 7f5e1359e700 1 -- 172.21.15.60:0/1588806201 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5e140b0bc0 con 0x7f5e140ad930 2022-06-29T17:09:04.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.927+0000 7f5e037fe700 1 -- 172.21.15.60:0/1588806201 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5e0401a060 con 0x7f5e140ad930 2022-06-29T17:09:04.948 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.931+0000 7f5e037fe700 1 --2- 172.21.15.60:0/1588806201 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5df0053b00 0x7f5df0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:04.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.931+0000 7f5e037fe700 1 -- 172.21.15.60:0/1588806201 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f5e0406bb20 con 0x7f5e140ad930 2022-06-29T17:09:04.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.931+0000 7f5e037fe700 1 --2- 172.21.15.60:0/1588806201 >> [v2:172.21.15.60:6816/12890,v1:172.21.15.60:6825/12890] conn(0x7f5df005a1f0 0x7f5df005c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:04.949 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.931+0000 7f5e037fe700 1 -- 172.21.15.60:0/1588806201 --> [v2:172.21.15.60:6816/12890,v1:172.21.15.60:6825/12890] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5df005cc30 con 0x7f5df005a1f0 2022-06-29T17:09:04.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.931+0000 7f5e11d9b700 1 --2- 172.21.15.60:0/1588806201 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5df0053b00 0x7f5df0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:04.950 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.931+0000 7f5e037fe700 1 -- 172.21.15.60:0/1588806201 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=150) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e0406bf00 con 0x7f5e140ad930 2022-06-29T17:09:04.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.931+0000 7f5e12d9d700 1 --2- 172.21.15.60:0/1588806201 >> [v2:172.21.15.60:6816/12890,v1:172.21.15.60:6825/12890] conn(0x7f5df005a1f0 0x7f5df005c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:04.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.931+0000 7f5e11d9b700 1 --2- 172.21.15.60:0/1588806201 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5df0053b00 0x7f5df0055fc0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 rx=0x7f5e0801c000 tx=0x7f5e08015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:04.951 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.931+0000 7f5e12d9d700 1 --2- 172.21.15.60:0/1588806201 >> [v2:172.21.15.60:6816/12890,v1:172.21.15.60:6825/12890] conn(0x7f5df005a1f0 0x7f5df005c650 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:04.952 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.931+0000 7f5e037fe700 1 -- 172.21.15.60:0/1588806201 <== osd.0 v2:172.21.15.60:6816/12890 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f5df005cc30 con 0x7f5df005a1f0 2022-06-29T17:09:04.953 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.947+0000 7f5e1359e700 1 -- 172.21.15.60:0/1588806201 --> [v2:172.21.15.60:6816/12890,v1:172.21.15.60:6825/12890] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f5e140b0f40 con 0x7f5df005a1f0 2022-06-29T17:09:04.954 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.947+0000 7f5e037fe700 1 -- 172.21.15.60:0/1588806201 <== osd.0 v2:172.21.15.60:6816/12890 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f5e0c009da0 con 0x7f5df005a1f0 2022-06-29T17:09:04.954 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T17:09:04.955 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-06-29T17:09:04.955 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T17:09:04.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.947+0000 7f5e1359e700 1 -- 172.21.15.60:0/1588806201 >> [v2:172.21.15.60:6816/12890,v1:172.21.15.60:6825/12890] conn(0x7f5df005a1f0 msgr2=0x7f5df005c650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:04.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.947+0000 7f5e1359e700 1 --2- 172.21.15.60:0/1588806201 >> [v2:172.21.15.60:6816/12890,v1:172.21.15.60:6825/12890] conn(0x7f5df005a1f0 0x7f5df005c650 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:04.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.947+0000 7f5e1359e700 1 -- 172.21.15.60:0/1588806201 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5df0053b00 msgr2=0x7f5df0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:04.957 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.947+0000 7f5e1359e700 1 --2- 172.21.15.60:0/1588806201 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5df0053b00 0x7f5df0055fc0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 rx=0x7f5e0801c000 tx=0x7f5e08015000).stop 2022-06-29T17:09:04.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.947+0000 7f5e1359e700 1 -- 172.21.15.60:0/1588806201 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e140ad930 msgr2=0x7f5e140b67d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:04.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.947+0000 7f5e1359e700 1 --2- 172.21.15.60:0/1588806201 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e140ad930 0x7f5e140b67d0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0x7f5e140b7ca0 tx=0x7f5e04007c20).stop 2022-06-29T17:09:04.958 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.947+0000 7f5e12d9d700 1 -- 172.21.15.60:0/1588806201 reap_dead start 2022-06-29T17:09:04.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.947+0000 7f5e1359e700 1 -- 172.21.15.60:0/1588806201 shutdown_connections 2022-06-29T17:09:04.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.947+0000 7f5e1359e700 1 -- 172.21.15.60:0/1588806201 >> 172.21.15.60:0/1588806201 conn(0x7f5e140b6f80 msgr2=0x7f5e14141f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:04.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.947+0000 7f5e1359e700 1 -- 172.21.15.60:0/1588806201 shutdown_connections 2022-06-29T17:09:04.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:04.947+0000 7f5e1359e700 1 -- 172.21.15.60:0/1588806201 wait complete. 2022-06-29T17:09:04.967 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2022-06-29T17:09:05.080 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.075+0000 7f15fcb39700 1 Processor -- start 2022-06-29T17:09:05.081 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.075+0000 7f15fcb39700 1 -- start start 2022-06-29T17:09:05.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.075+0000 7f15fcb39700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f15f80b48b0 0x7f15f8142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.075+0000 7f15fcb39700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15f80b2c50 0x7f15f80b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.084 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.075+0000 7f15fcb39700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f80b0cf0 con 0x7f15f80b5eb0 2022-06-29T17:09:05.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.075+0000 7f15fcb39700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15f80b54e0 con 0x7f15f80b48b0 2022-06-29T17:09:05.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.075+0000 7f15fcb39700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15f80b4d10 con 0x7f15f80b2c50 2022-06-29T17:09:05.085 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.075+0000 7f15f77fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f15f80b5eb0 0x7f15f80b05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34738/0 (socket says 172.21.15.60:34738) 2022-06-29T17:09:05.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.075+0000 7f15f77fe700 1 -- 172.21.15.60:0/2327611181 learned_addr learned my addr 172.21.15.60:0/2327611181 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T17:09:05.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.075+0000 7f15f6ffd700 1 --2- 172.21.15.60:0/2327611181 >> v2:172.21.15.60:3300/0 conn(0x7f15f80b48b0 0x7f15f8142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.086 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.075+0000 7f15f7fff700 1 --2- 172.21.15.60:0/2327611181 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15f80b2c50 0x7f15f80b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.075+0000 7f15f67fc700 1 -- 172.21.15.60:0/2327611181 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 279681124 0 0) 0x7f15e0001110 con 0x7f15f80b5eb0 2022-06-29T17:09:05.087 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.075+0000 7f15f67fc700 1 -- 172.21.15.60:0/2327611181 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15dc003260 con 0x7f15f80b5eb0 2022-06-29T17:09:05.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.075+0000 7f15f6ffd700 1 -- 172.21.15.60:0/2327611181 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15f80b2c50 msgr2=0x7f15f80b30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T17:09:05.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.075+0000 7f15f6ffd700 1 --2- 172.21.15.60:0/2327611181 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15f80b2c50 0x7f15f80b30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.088 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.075+0000 7f15f6ffd700 1 -- 172.21.15.60:0/2327611181 >> v1:172.21.15.60:6789/0 conn(0x7f15f80b5eb0 legacy=0x7f15f80b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:05.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.075+0000 7f15f6ffd700 1 -- 172.21.15.60:0/2327611181 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f80b6290 con 0x7f15f80b48b0 2022-06-29T17:09:05.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.075+0000 7f15f6ffd700 1 --2- 172.21.15.60:0/2327611181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15f80b48b0 0x7f15f8142a20 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 rx=0x7f15e8002f70 tx=0x7f15e8006df0).ready entity=mon.0 client_cookie=67114be624d85473 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:05.089 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.075+0000 7f15f67fc700 1 -- 172.21.15.60:0/2327611181 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15e800d4a0 con 0x7f15f80b48b0 2022-06-29T17:09:05.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.075+0000 7f15f67fc700 1 -- 172.21.15.60:0/2327611181 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15e8000e90 con 0x7f15f80b48b0 2022-06-29T17:09:05.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.075+0000 7f15f67fc700 1 -- 172.21.15.60:0/2327611181 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15e800d7f0 con 0x7f15f80b48b0 2022-06-29T17:09:05.090 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.079+0000 7f15fcb39700 1 -- 172.21.15.60:0/2327611181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15f80b48b0 msgr2=0x7f15f8142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:05.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.079+0000 7f15fcb39700 1 --2- 172.21.15.60:0/2327611181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15f80b48b0 0x7f15f8142a20 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 rx=0x7f15e8002f70 tx=0x7f15e8006df0).stop 2022-06-29T17:09:05.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.079+0000 7f15fcb39700 1 -- 172.21.15.60:0/2327611181 shutdown_connections 2022-06-29T17:09:05.091 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.079+0000 7f15fcb39700 1 --2- 172.21.15.60:0/2327611181 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15f80b2c50 0x7f15f80b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.079+0000 7f15fcb39700 1 --2- 172.21.15.60:0/2327611181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15f80b48b0 0x7f15f8142a20 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.092 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.079+0000 7f15fcb39700 1 -- 172.21.15.60:0/2327611181 >> 172.21.15.60:0/2327611181 conn(0x7f15f8062090 msgr2=0x7f15f8062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:05.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.079+0000 7f15fcb39700 1 -- 172.21.15.60:0/2327611181 shutdown_connections 2022-06-29T17:09:05.093 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.079+0000 7f15fcb39700 1 -- 172.21.15.60:0/2327611181 wait complete. 2022-06-29T17:09:05.096 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.079+0000 7f15fcb39700 1 Processor -- start 2022-06-29T17:09:05.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.079+0000 7f15fcb39700 1 -- start start 2022-06-29T17:09:05.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.083+0000 7f15fcb39700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15f80b2c50 0x7f15f80aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.097 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.083+0000 7f15fcb39700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15f80b48b0 0x7f15f81529f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.083+0000 7f15fcb39700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15f80b5eb0 0x7f15f80b8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.083+0000 7f15fcb39700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f80af880 con 0x7f15f80b5eb0 2022-06-29T17:09:05.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.083+0000 7f15fcb39700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f80adec0 con 0x7f15f80b48b0 2022-06-29T17:09:05.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.083+0000 7f15fcb39700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15f8140440 con 0x7f15f80b2c50 2022-06-29T17:09:05.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.083+0000 7f15f7fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15f80b5eb0 0x7f15f80b8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.099 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.083+0000 7f15f6ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15f80b48b0 0x7f15f81529f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.083+0000 7f15f77fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15f80b2c50 0x7f15f80aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.083+0000 7f15f6ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15f80b48b0 0x7f15f81529f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:60922/0 (socket says 172.21.15.60:60922) 2022-06-29T17:09:05.100 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.083+0000 7f15f6ffd700 1 -- 172.21.15.60:0/1783774809 learned_addr learned my addr 172.21.15.60:0/1783774809 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T17:09:05.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.083+0000 7f15f7fff700 1 -- 172.21.15.60:0/1783774809 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15f80b2c50 msgr2=0x7f15f80aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:05.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.083+0000 7f15f7fff700 1 --2- 172.21.15.60:0/1783774809 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15f80b2c50 0x7f15f80aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.101 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.083+0000 7f15f7fff700 1 -- 172.21.15.60:0/1783774809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15f80b48b0 msgr2=0x7f15f81529f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:05.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.083+0000 7f15f7fff700 1 --2- 172.21.15.60:0/1783774809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15f80b48b0 0x7f15f81529f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.083+0000 7f15f7fff700 1 -- 172.21.15.60:0/1783774809 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e800b180 con 0x7f15f80b5eb0 2022-06-29T17:09:05.102 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.083+0000 7f15f6ffd700 1 --2- 172.21.15.60:0/1783774809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15f80b48b0 0x7f15f81529f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T17:09:05.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.083+0000 7f15f77fe700 1 --2- 172.21.15.60:0/1783774809 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15f80b2c50 0x7f15f80aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T17:09:05.103 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.083+0000 7f15f7fff700 1 --2- 172.21.15.60:0/1783774809 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15f80b5eb0 0x7f15f80b8110 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 rx=0x7f15f000c000 tx=0x7f15f00091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:05.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.083+0000 7f15f4ff9700 1 -- 172.21.15.60:0/1783774809 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15f0012600 con 0x7f15f80b5eb0 2022-06-29T17:09:05.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.083+0000 7f15f4ff9700 1 -- 172.21.15.60:0/1783774809 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15f0012760 con 0x7f15f80b5eb0 2022-06-29T17:09:05.104 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.083+0000 7f15fcb39700 1 -- 172.21.15.60:0/1783774809 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15f81406e0 con 0x7f15f80b5eb0 2022-06-29T17:09:05.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.083+0000 7f15f4ff9700 1 -- 172.21.15.60:0/1783774809 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15f0012ac0 con 0x7f15f80b5eb0 2022-06-29T17:09:05.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.083+0000 7f15fcb39700 1 -- 172.21.15.60:0/1783774809 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15f8152ee0 con 0x7f15f80b5eb0 2022-06-29T17:09:05.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.083+0000 7f15fcb39700 1 -- 172.21.15.60:0/1783774809 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15f8142f10 con 0x7f15f80b5eb0 2022-06-29T17:09:05.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.087+0000 7f15f4ff9700 1 -- 172.21.15.60:0/1783774809 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f15f0022070 con 0x7f15f80b5eb0 2022-06-29T17:09:05.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.087+0000 7f15f4ff9700 1 --2- 172.21.15.60:0/1783774809 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f15d00539e0 0x7f15d0055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.106 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.087+0000 7f15f4ff9700 1 -- 172.21.15.60:0/1783774809 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f15f006c1f0 con 0x7f15f80b5eb0 2022-06-29T17:09:05.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.087+0000 7f15f77fe700 1 --2- 172.21.15.60:0/1783774809 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f15d00539e0 0x7f15d0055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.107 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.087+0000 7f15f77fe700 1 --2- 172.21.15.60:0/1783774809 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f15d00539e0 0x7f15d0055ea0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 rx=0x7f15e000d000 tx=0x7f15e00078a0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:05.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.095+0000 7f15f4ff9700 1 -- 172.21.15.60:0/1783774809 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f15f0043390 con 0x7f15f80b5eb0 2022-06-29T17:09:05.229 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.223+0000 7f15fcb39700 1 -- 172.21.15.60:0/1783774809 --> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f15f80573d0 con 0x7f15d00539e0 2022-06-29T17:09:05.230 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.223+0000 7f15f4ff9700 1 -- 172.21.15.60:0/1783774809 <== mgr.4103 v2:172.21.15.92:6832/12872 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f15f80573d0 con 0x7f15d00539e0 2022-06-29T17:09:05.233 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.227+0000 7f15fcb39700 1 -- 172.21.15.60:0/1783774809 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f15d00539e0 msgr2=0x7f15d0055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:05.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.227+0000 7f15fcb39700 1 --2- 172.21.15.60:0/1783774809 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f15d00539e0 0x7f15d0055ea0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 rx=0x7f15e000d000 tx=0x7f15e00078a0).stop 2022-06-29T17:09:05.234 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.227+0000 7f15fcb39700 1 -- 172.21.15.60:0/1783774809 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15f80b5eb0 msgr2=0x7f15f80b8110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:05.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.227+0000 7f15fcb39700 1 --2- 172.21.15.60:0/1783774809 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15f80b5eb0 0x7f15f80b8110 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 rx=0x7f15f000c000 tx=0x7f15f00091d0).stop 2022-06-29T17:09:05.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.227+0000 7f15fcb39700 1 -- 172.21.15.60:0/1783774809 shutdown_connections 2022-06-29T17:09:05.235 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.227+0000 7f15fcb39700 1 --2- 172.21.15.60:0/1783774809 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f15d00539e0 0x7f15d0055ea0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.227+0000 7f15fcb39700 1 --2- 172.21.15.60:0/1783774809 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15f80b5eb0 0x7f15f80b8110 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.236 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.227+0000 7f15fcb39700 1 --2- 172.21.15.60:0/1783774809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15f80b48b0 0x7f15f81529f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.237 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.227+0000 7f15fcb39700 1 --2- 172.21.15.60:0/1783774809 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f15f80b2c50 0x7f15f80aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.227+0000 7f15fcb39700 1 -- 172.21.15.60:0/1783774809 >> 172.21.15.60:0/1783774809 conn(0x7f15f8062090 msgr2=0x7f15f8143170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:05.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.227+0000 7f15fcb39700 1 -- 172.21.15.60:0/1783774809 shutdown_connections 2022-06-29T17:09:05.238 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.227+0000 7f15fcb39700 1 -- 172.21.15.60:0/1783774809 wait complete. 2022-06-29T17:09:05.239 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 0 to deep-scrub 2022-06-29T17:09:05.248 INFO:tasks.ceph:Scrubbing osd.1 2022-06-29T17:09:05.248 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2022-06-29T17:09:05.358 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.351+0000 7f563d3b0700 1 Processor -- start 2022-06-29T17:09:05.359 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.351+0000 7f563d3b0700 1 -- start start 2022-06-29T17:09:05.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f563d3b0700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5638067070 0x7f5638067450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f563d3b0700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5638141200 con 0x7f56380b2950 2022-06-29T17:09:05.364 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f563d3b0700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56381414a0 con 0x7f5638064bc0 2022-06-29T17:09:05.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f563d3b0700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56380b37f0 con 0x7f5638067070 2022-06-29T17:09:05.365 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f56377fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f56380b2950 0x7f563813a410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34754/0 (socket says 172.21.15.60:34754) 2022-06-29T17:09:05.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f56377fe700 1 -- 172.21.15.60:0/2803302157 learned_addr learned my addr 172.21.15.60:0/2803302157 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T17:09:05.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f5637fff700 1 --2- 172.21.15.60:0/2803302157 >> v2:172.21.15.92:3300/0 conn(0x7f5638067070 0x7f5638067450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.366 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f5636ffd700 1 -- 172.21.15.60:0/2803302157 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1343541708 0 0) 0x7f5638141200 con 0x7f56380b2950 2022-06-29T17:09:05.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f5636ffd700 1 -- 172.21.15.60:0/2803302157 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5624003260 con 0x7f56380b2950 2022-06-29T17:09:05.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f5636ffd700 1 -- 172.21.15.60:0/2803302157 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 158628173 0 0) 0x7f56381414a0 con 0x7f5638064bc0 2022-06-29T17:09:05.367 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f5636ffd700 1 -- 172.21.15.60:0/2803302157 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5638141200 con 0x7f5638064bc0 2022-06-29T17:09:05.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f5636ffd700 1 -- 172.21.15.60:0/2803302157 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2578522039 0 0) 0x7f5624003260 con 0x7f56380b2950 2022-06-29T17:09:05.368 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f5636ffd700 1 -- 172.21.15.60:0/2803302157 >> v2:172.21.15.92:3300/0 conn(0x7f5638067070 msgr2=0x7f5638067450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T17:09:05.369 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f5636ffd700 1 --2- 172.21.15.60:0/2803302157 >> v2:172.21.15.92:3300/0 conn(0x7f5638067070 0x7f5638067450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.370 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f5636ffd700 1 -- 172.21.15.60:0/2803302157 >> v1:172.21.15.92:6789/0 conn(0x7f5638064bc0 legacy=0x7f5638064fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:05.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f5636ffd700 1 -- 172.21.15.60:0/2803302157 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f563813ab20 con 0x7f56380b2950 2022-06-29T17:09:05.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f5636ffd700 1 -- 172.21.15.60:0/2803302157 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f562c001660 con 0x7f56380b2950 2022-06-29T17:09:05.371 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f5636ffd700 1 -- 172.21.15.60:0/2803302157 >> v1:172.21.15.60:6789/0 conn(0x7f56380b2950 legacy=0x7f563813a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:05.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f5636ffd700 1 --2- 172.21.15.60:0/2803302157 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56240038e0 0x7f5624003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f5636ffd700 1 --2- 172.21.15.60:0/2803302157 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5624004500 0x7f5624004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.372 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f5636ffd700 1 --2- 172.21.15.60:0/2803302157 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5624004e70 0x7f562400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f5636ffd700 1 -- 172.21.15.60:0/2803302157 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56380b37f0 con 0x7f56240038e0 2022-06-29T17:09:05.373 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f5636ffd700 1 -- 172.21.15.60:0/2803302157 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562400bdd0 con 0x7f5624004e70 2022-06-29T17:09:05.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f5636ffd700 1 -- 172.21.15.60:0/2803302157 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5624010350 con 0x7f5624004500 2022-06-29T17:09:05.374 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f5637fff700 1 --2- 172.21.15.60:0/2803302157 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56240038e0 0x7f5624003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f563cbaf700 1 --2- 172.21.15.60:0/2803302157 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5624004e70 0x7f562400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f56377fe700 1 --2- 172.21.15.60:0/2803302157 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5624004500 0x7f5624004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.375 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f5637fff700 1 -- 172.21.15.60:0/2803302157 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5624004500 msgr2=0x7f5624004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T17:09:05.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f5637fff700 1 --2- 172.21.15.60:0/2803302157 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5624004500 0x7f5624004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f5637fff700 1 -- 172.21.15.60:0/2803302157 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5624004e70 msgr2=0x7f562400b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T17:09:05.376 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f5637fff700 1 --2- 172.21.15.60:0/2803302157 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5624004e70 0x7f562400b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f563cbaf700 1 -- 172.21.15.60:0/2803302157 reap_dead start 2022-06-29T17:09:05.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f5637fff700 1 -- 172.21.15.60:0/2803302157 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5624003260 con 0x7f56240038e0 2022-06-29T17:09:05.377 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f5637fff700 1 --2- 172.21.15.60:0/2803302157 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56240038e0 0x7f5624003ce0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 rx=0x7f562800c000 tx=0x7f5628006df0).ready entity=mon.0 client_cookie=3f81c975459c20e8 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:05.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f5636ffd700 1 -- 172.21.15.60:0/2803302157 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f562800e830 con 0x7f56240038e0 2022-06-29T17:09:05.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f5636ffd700 1 -- 172.21.15.60:0/2803302157 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f562800e990 con 0x7f56240038e0 2022-06-29T17:09:05.378 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f5636ffd700 1 -- 172.21.15.60:0/2803302157 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f562800ecf0 con 0x7f56240038e0 2022-06-29T17:09:05.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f563d3b0700 1 -- 172.21.15.60:0/2803302157 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56240038e0 msgr2=0x7f5624003ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:05.379 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f563d3b0700 1 --2- 172.21.15.60:0/2803302157 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56240038e0 0x7f5624003ce0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 rx=0x7f562800c000 tx=0x7f5628006df0).stop 2022-06-29T17:09:05.380 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f563d3b0700 1 -- 172.21.15.60:0/2803302157 shutdown_connections 2022-06-29T17:09:05.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.355+0000 7f563d3b0700 1 --2- 172.21.15.60:0/2803302157 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f56240038e0 0x7f5624003ce0 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.359+0000 7f563d3b0700 1 -- 172.21.15.60:0/2803302157 >> 172.21.15.60:0/2803302157 conn(0x7f56380b6f80 msgr2=0x7f56380b7360 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:05.381 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.359+0000 7f563d3b0700 1 -- 172.21.15.60:0/2803302157 shutdown_connections 2022-06-29T17:09:05.382 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.359+0000 7f563d3b0700 1 -- 172.21.15.60:0/2803302157 wait complete. 2022-06-29T17:09:05.385 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.359+0000 7f563d3b0700 1 Processor -- start 2022-06-29T17:09:05.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.359+0000 7f563d3b0700 1 -- start start 2022-06-29T17:09:05.386 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.359+0000 7f563d3b0700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56240038e0 0x7f56380aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.359+0000 7f563d3b0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5638062040 0x7f563814e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.359+0000 7f563d3b0700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f563814e690 0x7f56380ad350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.387 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.359+0000 7f563d3b0700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563814db10 con 0x7f5638062040 2022-06-29T17:09:05.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.359+0000 7f563d3b0700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563813c0a0 con 0x7f56240038e0 2022-06-29T17:09:05.388 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.359+0000 7f563d3b0700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f563814dd00 con 0x7f563814e690 2022-06-29T17:09:05.389 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.359+0000 7f56377fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5638062040 0x7f563814e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.359+0000 7f56377fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5638062040 0x7f563814e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54910/0 (socket says 172.21.15.60:54910) 2022-06-29T17:09:05.390 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.359+0000 7f56377fe700 1 -- 172.21.15.60:0/511870074 learned_addr learned my addr 172.21.15.60:0/511870074 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T17:09:05.391 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.359+0000 7f5637fff700 1 --2- 172.21.15.60:0/511870074 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56240038e0 0x7f56380aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.359+0000 7f563cbaf700 1 --2- 172.21.15.60:0/511870074 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f563814e690 0x7f56380ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.359+0000 7f56377fe700 1 -- 172.21.15.60:0/511870074 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f563814e690 msgr2=0x7f56380ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:05.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.359+0000 7f56377fe700 1 --2- 172.21.15.60:0/511870074 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f563814e690 0x7f56380ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.359+0000 7f56377fe700 1 -- 172.21.15.60:0/511870074 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56240038e0 msgr2=0x7f56380aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:05.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.359+0000 7f56377fe700 1 --2- 172.21.15.60:0/511870074 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56240038e0 0x7f56380aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.394 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.359+0000 7f56377fe700 1 -- 172.21.15.60:0/511870074 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56380acf10 con 0x7f5638062040 2022-06-29T17:09:05.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.359+0000 7f5637fff700 1 --2- 172.21.15.60:0/511870074 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56240038e0 0x7f56380aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T17:09:05.395 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.359+0000 7f563cbaf700 1 --2- 172.21.15.60:0/511870074 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f563814e690 0x7f56380ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T17:09:05.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.359+0000 7f56377fe700 1 --2- 172.21.15.60:0/511870074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5638062040 0x7f563814e1a0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 rx=0x7f562c001b10 tx=0x7f562c001500).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:05.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.363+0000 7f56357fa700 1 -- 172.21.15.60:0/511870074 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f562c010710 con 0x7f5638062040 2022-06-29T17:09:05.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.363+0000 7f56357fa700 1 -- 172.21.15.60:0/511870074 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f562c010870 con 0x7f5638062040 2022-06-29T17:09:05.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.363+0000 7f563d3b0700 1 -- 172.21.15.60:0/511870074 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56380624c0 con 0x7f5638062040 2022-06-29T17:09:05.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.363+0000 7f56357fa700 1 -- 172.21.15.60:0/511870074 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f562c010bd0 con 0x7f5638062040 2022-06-29T17:09:05.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.363+0000 7f563d3b0700 1 -- 172.21.15.60:0/511870074 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56380ae0e0 con 0x7f5638062040 2022-06-29T17:09:05.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.363+0000 7f563d3b0700 1 -- 172.21.15.60:0/511870074 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f563813bf20 con 0x7f5638062040 2022-06-29T17:09:05.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.363+0000 7f56357fa700 1 -- 172.21.15.60:0/511870074 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f562c01a020 con 0x7f5638062040 2022-06-29T17:09:05.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.363+0000 7f56357fa700 1 --2- 172.21.15.60:0/511870074 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5618053950 0x7f5618055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.363+0000 7f56357fa700 1 -- 172.21.15.60:0/511870074 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f562c06c1e0 con 0x7f5638062040 2022-06-29T17:09:05.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.363+0000 7f5637fff700 1 --2- 172.21.15.60:0/511870074 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5618053950 0x7f5618055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.363+0000 7f56357fa700 1 --2- 172.21.15.60:0/511870074 >> [v2:172.21.15.60:6815/12892,v1:172.21.15.60:6818/12892] conn(0x7f561805a130 0x7f561805c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.363+0000 7f56357fa700 1 -- 172.21.15.60:0/511870074 --> [v2:172.21.15.60:6815/12892,v1:172.21.15.60:6818/12892] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f561805cc10 con 0x7f561805a130 2022-06-29T17:09:05.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.363+0000 7f56357fa700 1 -- 172.21.15.60:0/511870074 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=150) v2 ==== 24+0+0 (secure 0 0 0) 0x7f562c06c540 con 0x7f5638062040 2022-06-29T17:09:05.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.363+0000 7f563cbaf700 1 --2- 172.21.15.60:0/511870074 >> [v2:172.21.15.60:6815/12892,v1:172.21.15.60:6818/12892] conn(0x7f561805a130 0x7f561805c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.363+0000 7f5637fff700 1 --2- 172.21.15.60:0/511870074 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5618053950 0x7f5618055e10 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0x7f562801a000 tx=0x7f5628019000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:05.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.367+0000 7f563cbaf700 1 --2- 172.21.15.60:0/511870074 >> [v2:172.21.15.60:6815/12892,v1:172.21.15.60:6818/12892] conn(0x7f561805a130 0x7f561805c590 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:05.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.367+0000 7f56357fa700 1 -- 172.21.15.60:0/511870074 <== osd.1 v2:172.21.15.60:6815/12892 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f561805cc10 con 0x7f561805a130 2022-06-29T17:09:05.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.379+0000 7f563d3b0700 1 -- 172.21.15.60:0/511870074 --> [v2:172.21.15.60:6815/12892,v1:172.21.15.60:6818/12892] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f563814ea70 con 0x7f561805a130 2022-06-29T17:09:05.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.383+0000 7f56357fa700 1 -- 172.21.15.60:0/511870074 <== osd.1 v2:172.21.15.60:6815/12892 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f563814ea70 con 0x7f561805a130 2022-06-29T17:09:05.406 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T17:09:05.407 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-06-29T17:09:05.407 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T17:09:05.409 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.383+0000 7f563d3b0700 1 -- 172.21.15.60:0/511870074 >> [v2:172.21.15.60:6815/12892,v1:172.21.15.60:6818/12892] conn(0x7f561805a130 msgr2=0x7f561805c590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:05.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.383+0000 7f563d3b0700 1 --2- 172.21.15.60:0/511870074 >> [v2:172.21.15.60:6815/12892,v1:172.21.15.60:6818/12892] conn(0x7f561805a130 0x7f561805c590 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.383+0000 7f563d3b0700 1 -- 172.21.15.60:0/511870074 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5618053950 msgr2=0x7f5618055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:05.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.383+0000 7f563d3b0700 1 --2- 172.21.15.60:0/511870074 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5618053950 0x7f5618055e10 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0x7f562801a000 tx=0x7f5628019000).stop 2022-06-29T17:09:05.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.383+0000 7f563d3b0700 1 -- 172.21.15.60:0/511870074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5638062040 msgr2=0x7f563814e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:05.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.383+0000 7f563d3b0700 1 --2- 172.21.15.60:0/511870074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5638062040 0x7f563814e1a0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 rx=0x7f562c001b10 tx=0x7f562c001500).stop 2022-06-29T17:09:05.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.383+0000 7f563cbaf700 1 -- 172.21.15.60:0/511870074 reap_dead start 2022-06-29T17:09:05.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.383+0000 7f563d3b0700 1 -- 172.21.15.60:0/511870074 shutdown_connections 2022-06-29T17:09:05.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.383+0000 7f563d3b0700 1 -- 172.21.15.60:0/511870074 >> 172.21.15.60:0/511870074 conn(0x7f56380b6f80 msgr2=0x7f56381398a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:05.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.383+0000 7f563d3b0700 1 -- 172.21.15.60:0/511870074 shutdown_connections 2022-06-29T17:09:05.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.383+0000 7f563d3b0700 1 -- 172.21.15.60:0/511870074 wait complete. 2022-06-29T17:09:05.415 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2022-06-29T17:09:05.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.519+0000 7effc1380700 1 Processor -- start 2022-06-29T17:09:05.527 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.519+0000 7effc1380700 1 -- start start 2022-06-29T17:09:05.532 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.523+0000 7effc1380700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7effbc05c5b0 0x7effbc05c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.523+0000 7effc1380700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effbc0b0a40 con 0x7effbc060610 2022-06-29T17:09:05.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.523+0000 7effc1380700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effbc0b0ce0 con 0x7effbc05fee0 2022-06-29T17:09:05.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.523+0000 7effc1380700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effbc0b0850 con 0x7effbc05c5b0 2022-06-29T17:09:05.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.523+0000 7effb37fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7effbc060610 0x7effbc13c740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34774/0 (socket says 172.21.15.60:34774) 2022-06-29T17:09:05.534 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.523+0000 7effb37fe700 1 -- 172.21.15.60:0/1624958706 learned_addr learned my addr 172.21.15.60:0/1624958706 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T17:09:05.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.523+0000 7effbbfff700 1 --2- 172.21.15.60:0/1624958706 >> v2:172.21.15.60:3300/0 conn(0x7effbc05c5b0 0x7effbc05c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.535 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.523+0000 7effbb7fe700 1 -- 172.21.15.60:0/1624958706 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2905946259 0 0) 0x7effbc0b0a40 con 0x7effbc060610 2022-06-29T17:09:05.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.523+0000 7effbb7fe700 1 -- 172.21.15.60:0/1624958706 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff9c003260 con 0x7effbc060610 2022-06-29T17:09:05.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.523+0000 7effbbfff700 1 -- 172.21.15.60:0/1624958706 >> v1:172.21.15.92:6789/0 conn(0x7effbc05fee0 legacy=0x7effbc142a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:05.536 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.523+0000 7effbbfff700 1 -- 172.21.15.60:0/1624958706 >> v1:172.21.15.60:6789/0 conn(0x7effbc060610 legacy=0x7effbc13c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:05.537 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.523+0000 7effbbfff700 1 -- 172.21.15.60:0/1624958706 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effbc13ce50 con 0x7effbc05c5b0 2022-06-29T17:09:05.538 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.523+0000 7effbbfff700 1 --2- 172.21.15.60:0/1624958706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effbc05c5b0 0x7effbc05c990 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 rx=0x7effac002f70 tx=0x7effac006df0).ready entity=mon.0 client_cookie=dfaf5d40e56cadb9 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:05.539 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.523+0000 7effbb7fe700 1 -- 172.21.15.60:0/1624958706 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7effac00d610 con 0x7effbc05c5b0 2022-06-29T17:09:05.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.523+0000 7effbb7fe700 1 -- 172.21.15.60:0/1624958706 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effac000e90 con 0x7effbc05c5b0 2022-06-29T17:09:05.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.523+0000 7effbb7fe700 1 -- 172.21.15.60:0/1624958706 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7effac00d960 con 0x7effbc05c5b0 2022-06-29T17:09:05.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.523+0000 7effc1380700 1 -- 172.21.15.60:0/1624958706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effbc05c5b0 msgr2=0x7effbc05c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:05.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.523+0000 7effc1380700 1 --2- 172.21.15.60:0/1624958706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effbc05c5b0 0x7effbc05c990 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 rx=0x7effac002f70 tx=0x7effac006df0).stop 2022-06-29T17:09:05.541 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.523+0000 7effc1380700 1 -- 172.21.15.60:0/1624958706 shutdown_connections 2022-06-29T17:09:05.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.523+0000 7effc1380700 1 --2- 172.21.15.60:0/1624958706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effbc05c5b0 0x7effbc05c990 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.523+0000 7effc1380700 1 -- 172.21.15.60:0/1624958706 >> 172.21.15.60:0/1624958706 conn(0x7effbc061c80 msgr2=0x7effbc0637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:05.542 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.523+0000 7effc1380700 1 -- 172.21.15.60:0/1624958706 shutdown_connections 2022-06-29T17:09:05.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.523+0000 7effc1380700 1 -- 172.21.15.60:0/1624958706 wait complete. 2022-06-29T17:09:05.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.527+0000 7effc1380700 1 Processor -- start 2022-06-29T17:09:05.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.527+0000 7effc1380700 1 -- start start 2022-06-29T17:09:05.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.527+0000 7effc1380700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7effbc05c5b0 0x7effbc151aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.527+0000 7effc1380700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effbc05fee0 0x7effbc13cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.549 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.527+0000 7effc1380700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7effbc060610 0x7effbc152540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.527+0000 7effc1380700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effbc0b4940 con 0x7effbc05fee0 2022-06-29T17:09:05.550 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.527+0000 7effc1380700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effbc0b46a0 con 0x7effbc05c5b0 2022-06-29T17:09:05.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.527+0000 7effc1380700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effbc13efd0 con 0x7effbc060610 2022-06-29T17:09:05.551 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.527+0000 7effb37fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effbc05fee0 0x7effbc13cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.527+0000 7effb37fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effbc05fee0 0x7effbc13cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54928/0 (socket says 172.21.15.60:54928) 2022-06-29T17:09:05.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.527+0000 7effbbfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7effbc05c5b0 0x7effbc151aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.527+0000 7effb37fe700 1 -- 172.21.15.60:0/2339777127 learned_addr learned my addr 172.21.15.60:0/2339777127 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T17:09:05.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.527+0000 7effc0b7f700 1 --2- 172.21.15.60:0/2339777127 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7effbc060610 0x7effbc152540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.527+0000 7effb37fe700 1 -- 172.21.15.60:0/2339777127 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7effbc060610 msgr2=0x7effbc152540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:05.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.527+0000 7effb37fe700 1 --2- 172.21.15.60:0/2339777127 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7effbc060610 0x7effbc152540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.527+0000 7effb37fe700 1 -- 172.21.15.60:0/2339777127 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7effbc05c5b0 msgr2=0x7effbc151aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:05.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.527+0000 7effb37fe700 1 --2- 172.21.15.60:0/2339777127 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7effbc05c5b0 0x7effbc151aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.527+0000 7effb37fe700 1 -- 172.21.15.60:0/2339777127 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effac00b180 con 0x7effbc05fee0 2022-06-29T17:09:05.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.527+0000 7effb37fe700 1 --2- 172.21.15.60:0/2339777127 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effbc05fee0 0x7effbc13cfe0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 rx=0x7effa4001110 tx=0x7effa4007ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:05.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.527+0000 7effb9ffb700 1 -- 172.21.15.60:0/2339777127 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7effa40070d0 con 0x7effbc05fee0 2022-06-29T17:09:05.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.527+0000 7effb9ffb700 1 -- 172.21.15.60:0/2339777127 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effa4007230 con 0x7effbc05fee0 2022-06-29T17:09:05.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.527+0000 7effc1380700 1 -- 172.21.15.60:0/2339777127 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effbc151f90 con 0x7effbc05fee0 2022-06-29T17:09:05.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.527+0000 7effb9ffb700 1 -- 172.21.15.60:0/2339777127 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7effa4005610 con 0x7effbc05fee0 2022-06-29T17:09:05.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.527+0000 7effc1380700 1 -- 172.21.15.60:0/2339777127 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effbc140080 con 0x7effbc05fee0 2022-06-29T17:09:05.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.531+0000 7effc1380700 1 -- 172.21.15.60:0/2339777127 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effbc13ee20 con 0x7effbc05fee0 2022-06-29T17:09:05.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.531+0000 7effb9ffb700 1 -- 172.21.15.60:0/2339777127 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7effa4010070 con 0x7effbc05fee0 2022-06-29T17:09:05.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.531+0000 7effb9ffb700 1 --2- 172.21.15.60:0/2339777127 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7effa0053b00 0x7effa0055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.531+0000 7effb9ffb700 1 -- 172.21.15.60:0/2339777127 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5339+0+0 (secure 0 0 0) 0x7effa406b0c0 con 0x7effbc05fee0 2022-06-29T17:09:05.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.531+0000 7effbbfff700 1 --2- 172.21.15.60:0/2339777127 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7effa0053b00 0x7effa0055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.535+0000 7effbbfff700 1 --2- 172.21.15.60:0/2339777127 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7effa0053b00 0x7effa0055fc0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 rx=0x7effac01a000 tx=0x7effac014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:05.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.539+0000 7effb9ffb700 1 -- 172.21.15.60:0/2339777127 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7effa40421b0 con 0x7effbc05fee0 2022-06-29T17:09:05.668 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.663+0000 7effc1380700 1 -- 172.21.15.60:0/2339777127 --> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7effbc140080 con 0x7effa0053b00 2022-06-29T17:09:05.669 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.663+0000 7effb9ffb700 1 -- 172.21.15.60:0/2339777127 <== mgr.4103 v2:172.21.15.92:6832/12872 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7effbc140080 con 0x7effa0053b00 2022-06-29T17:09:05.671 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.663+0000 7effc1380700 1 -- 172.21.15.60:0/2339777127 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7effa0053b00 msgr2=0x7effa0055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:05.672 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.663+0000 7effc1380700 1 --2- 172.21.15.60:0/2339777127 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7effa0053b00 0x7effa0055fc0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 rx=0x7effac01a000 tx=0x7effac014000).stop 2022-06-29T17:09:05.672 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.663+0000 7effc1380700 1 -- 172.21.15.60:0/2339777127 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effbc05fee0 msgr2=0x7effbc13cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:05.673 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.663+0000 7effc1380700 1 --2- 172.21.15.60:0/2339777127 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effbc05fee0 0x7effbc13cfe0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 rx=0x7effa4001110 tx=0x7effa4007ad0).stop 2022-06-29T17:09:05.673 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.663+0000 7effc1380700 1 -- 172.21.15.60:0/2339777127 shutdown_connections 2022-06-29T17:09:05.674 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.663+0000 7effc1380700 1 --2- 172.21.15.60:0/2339777127 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7effa0053b00 0x7effa0055fc0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.674 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.663+0000 7effc1380700 1 --2- 172.21.15.60:0/2339777127 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7effbc060610 0x7effbc152540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.675 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.667+0000 7effc1380700 1 --2- 172.21.15.60:0/2339777127 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7effbc05fee0 0x7effbc13cfe0 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.675 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.667+0000 7effc1380700 1 --2- 172.21.15.60:0/2339777127 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7effbc05c5b0 0x7effbc151aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.676 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.667+0000 7effc1380700 1 -- 172.21.15.60:0/2339777127 >> 172.21.15.60:0/2339777127 conn(0x7effbc061c80 msgr2=0x7effbc13c200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:05.676 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.667+0000 7effc1380700 1 -- 172.21.15.60:0/2339777127 shutdown_connections 2022-06-29T17:09:05.677 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.667+0000 7effc1380700 1 -- 172.21.15.60:0/2339777127 wait complete. 2022-06-29T17:09:05.678 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 1 to deep-scrub 2022-06-29T17:09:05.683 INFO:tasks.ceph:Scrubbing osd.2 2022-06-29T17:09:05.684 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2022-06-29T17:09:05.794 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.787+0000 7f5ec7437700 1 Processor -- start 2022-06-29T17:09:05.795 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.787+0000 7f5ec7437700 1 -- start start 2022-06-29T17:09:05.799 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.787+0000 7f5ec7437700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5ec00b61d0 0x7f5ec00b0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.799 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.787+0000 7f5ec7437700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5ec013f790 0x7f5ec013bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.799 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.787+0000 7f5ec7437700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5ec013fc00 0x7f5ec013c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.800 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.787+0000 7f5ec7437700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ec0142820 con 0x7f5ec013f790 2022-06-29T17:09:05.800 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.787+0000 7f5ec7437700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ec0062bb0 con 0x7f5ec00b61d0 2022-06-29T17:09:05.800 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.791+0000 7f5ec7437700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec00b65b0 con 0x7f5ec013fc00 2022-06-29T17:09:05.801 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.791+0000 7f5ec49d2700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5ec013f790 0x7f5ec013bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.801 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.791+0000 7f5ec49d2700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5ec013f790 0x7f5ec013bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54936/0 (socket says 172.21.15.60:54936) 2022-06-29T17:09:05.802 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.791+0000 7f5ec49d2700 1 -- 172.21.15.60:0/1743909398 learned_addr learned my addr 172.21.15.60:0/1743909398 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T17:09:05.803 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.791+0000 7f5ec59d4700 1 --2- 172.21.15.60:0/1743909398 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5ec013fc00 0x7f5ec013c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.803 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.791+0000 7f5ec51d3700 1 --2- 172.21.15.60:0/1743909398 >> v2:172.21.15.92:3300/0 conn(0x7f5ec00b61d0 0x7f5ec00b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.804 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.791+0000 7f5ec49d2700 1 -- 172.21.15.60:0/1743909398 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5ec013fc00 msgr2=0x7f5ec013c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T17:09:05.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.791+0000 7f5ec49d2700 1 --2- 172.21.15.60:0/1743909398 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5ec013fc00 0x7f5ec013c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.805 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.791+0000 7f5ec49d2700 1 -- 172.21.15.60:0/1743909398 >> v2:172.21.15.92:3300/0 conn(0x7f5ec00b61d0 msgr2=0x7f5ec00b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T17:09:05.806 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.791+0000 7f5ec49d2700 1 --2- 172.21.15.60:0/1743909398 >> v2:172.21.15.92:3300/0 conn(0x7f5ec00b61d0 0x7f5ec00b0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.807 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.791+0000 7f5ec49d2700 1 -- 172.21.15.60:0/1743909398 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec013f050 con 0x7f5ec013f790 2022-06-29T17:09:05.807 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.791+0000 7f5ec51d3700 1 --2- 172.21.15.60:0/1743909398 >> v2:172.21.15.92:3300/0 conn(0x7f5ec00b61d0 0x7f5ec00b0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T17:09:05.808 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.791+0000 7f5ec49d2700 1 --2- 172.21.15.60:0/1743909398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ec013f790 0x7f5ec013bc40 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 rx=0x7f5ec00acf50 tx=0x7f5ebc006eb0).ready entity=mon.0 client_cookie=39b3d361b326fed1 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:05.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.791+0000 7f5eb7fff700 1 -- 172.21.15.60:0/1743909398 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ebc00e610 con 0x7f5ec013f790 2022-06-29T17:09:05.809 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.791+0000 7f5eb7fff700 1 -- 172.21.15.60:0/1743909398 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ebc000e90 con 0x7f5ec013f790 2022-06-29T17:09:05.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.791+0000 7f5eb7fff700 1 -- 172.21.15.60:0/1743909398 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ebc00e9a0 con 0x7f5ec013f790 2022-06-29T17:09:05.810 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.791+0000 7f5ec7437700 1 -- 172.21.15.60:0/1743909398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ec013f790 msgr2=0x7f5ec013bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:05.811 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.791+0000 7f5ec7437700 1 --2- 172.21.15.60:0/1743909398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ec013f790 0x7f5ec013bc40 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 rx=0x7f5ec00acf50 tx=0x7f5ebc006eb0).stop 2022-06-29T17:09:05.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.791+0000 7f5ec7437700 1 -- 172.21.15.60:0/1743909398 shutdown_connections 2022-06-29T17:09:05.812 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.791+0000 7f5ec7437700 1 --2- 172.21.15.60:0/1743909398 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5ec013fc00 0x7f5ec013c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.813 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.791+0000 7f5ec7437700 1 --2- 172.21.15.60:0/1743909398 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ec013f790 0x7f5ec013bc40 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.791+0000 7f5ec7437700 1 --2- 172.21.15.60:0/1743909398 >> v2:172.21.15.92:3300/0 conn(0x7f5ec00b61d0 0x7f5ec00b0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.814 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.791+0000 7f5ec7437700 1 -- 172.21.15.60:0/1743909398 >> 172.21.15.60:0/1743909398 conn(0x7f5ec0063b50 msgr2=0x7f5ec00b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:05.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.791+0000 7f5ec7437700 1 -- 172.21.15.60:0/1743909398 shutdown_connections 2022-06-29T17:09:05.815 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.791+0000 7f5ec7437700 1 -- 172.21.15.60:0/1743909398 wait complete. 2022-06-29T17:09:05.824 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.795+0000 7f5ec7437700 1 Processor -- start 2022-06-29T17:09:05.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.795+0000 7f5ec7437700 1 -- start start 2022-06-29T17:09:05.825 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.795+0000 7f5ec7437700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5ec00b61d0 0x7f5ec0145250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.826 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.795+0000 7f5ec7437700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ec013f790 0x7f5ec00b4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.795+0000 7f5ec7437700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ec013fc00 0x7f5ec01522c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.795+0000 7f5ec7437700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec0136fe0 con 0x7f5ec013f790 2022-06-29T17:09:05.828 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.795+0000 7f5ec7437700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec0148e20 con 0x7f5ec013fc00 2022-06-29T17:09:05.829 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.795+0000 7f5ec7437700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec01490c0 con 0x7f5ec00b61d0 2022-06-29T17:09:05.829 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.795+0000 7f5ec49d2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ec013f790 0x7f5ec00b4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.830 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.795+0000 7f5ec49d2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ec013f790 0x7f5ec00b4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54942/0 (socket says 172.21.15.60:54942) 2022-06-29T17:09:05.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.795+0000 7f5ec49d2700 1 -- 172.21.15.60:0/618304502 learned_addr learned my addr 172.21.15.60:0/618304502 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T17:09:05.831 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.795+0000 7f5ec59d4700 1 --2- 172.21.15.60:0/618304502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ec013fc00 0x7f5ec01522c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.795+0000 7f5ec51d3700 1 --2- 172.21.15.60:0/618304502 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5ec00b61d0 0x7f5ec0145250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.832 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.795+0000 7f5ec49d2700 1 -- 172.21.15.60:0/618304502 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5ec00b61d0 msgr2=0x7f5ec0145250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:05.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.795+0000 7f5ec49d2700 1 --2- 172.21.15.60:0/618304502 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f5ec00b61d0 0x7f5ec0145250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.795+0000 7f5ec49d2700 1 -- 172.21.15.60:0/618304502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ec013fc00 msgr2=0x7f5ec01522c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:05.834 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.795+0000 7f5ec49d2700 1 --2- 172.21.15.60:0/618304502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ec013fc00 0x7f5ec01522c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.835 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.795+0000 7f5ec49d2700 1 -- 172.21.15.60:0/618304502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb000b180 con 0x7f5ec013f790 2022-06-29T17:09:05.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.795+0000 7f5ec49d2700 1 --2- 172.21.15.60:0/618304502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ec013f790 0x7f5ec00b4950 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 rx=0x7f5ec00acf50 tx=0x7f5ebc015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:05.836 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.795+0000 7f5eb67fc700 1 -- 172.21.15.60:0/618304502 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ebc0116e0 con 0x7f5ec013f790 2022-06-29T17:09:05.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.795+0000 7f5eb67fc700 1 -- 172.21.15.60:0/618304502 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ebc011840 con 0x7f5ec013f790 2022-06-29T17:09:05.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.795+0000 7f5ec7437700 1 -- 172.21.15.60:0/618304502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ebc00b1e0 con 0x7f5ec013f790 2022-06-29T17:09:05.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.795+0000 7f5eb67fc700 1 -- 172.21.15.60:0/618304502 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ebc011ba0 con 0x7f5ec013f790 2022-06-29T17:09:05.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.795+0000 7f5ec7437700 1 -- 172.21.15.60:0/618304502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ec0066040 con 0x7f5ec013f790 2022-06-29T17:09:05.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.799+0000 7f5ec7437700 1 -- 172.21.15.60:0/618304502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5ec013f2f0 con 0x7f5ec013f790 2022-06-29T17:09:05.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.799+0000 7f5eb67fc700 1 -- 172.21.15.60:0/618304502 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5ebc025030 con 0x7f5ec013f790 2022-06-29T17:09:05.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.799+0000 7f5eb67fc700 1 --2- 172.21.15.60:0/618304502 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5ea8053aa0 0x7f5ea8055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.841 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.799+0000 7f5eb67fc700 1 -- 172.21.15.60:0/618304502 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f5ebc06d350 con 0x7f5ec013f790 2022-06-29T17:09:05.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.799+0000 7f5eb67fc700 1 --2- 172.21.15.60:0/618304502 >> [v2:172.21.15.60:6808/12891,v1:172.21.15.60:6809/12891] conn(0x7f5ea805a190 0x7f5ea805c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.842 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.799+0000 7f5eb67fc700 1 -- 172.21.15.60:0/618304502 --> [v2:172.21.15.60:6808/12891,v1:172.21.15.60:6809/12891] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5ea805cbd0 con 0x7f5ea805a190 2022-06-29T17:09:05.843 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.799+0000 7f5eb67fc700 1 -- 172.21.15.60:0/618304502 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=150) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5ebc06d730 con 0x7f5ec013f790 2022-06-29T17:09:05.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.799+0000 7f5ec51d3700 1 --2- 172.21.15.60:0/618304502 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5ea8053aa0 0x7f5ea8055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.844 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.799+0000 7f5ec59d4700 1 --2- 172.21.15.60:0/618304502 >> [v2:172.21.15.60:6808/12891,v1:172.21.15.60:6809/12891] conn(0x7f5ea805a190 0x7f5ea805c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.845 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.799+0000 7f5ec59d4700 1 --2- 172.21.15.60:0/618304502 >> [v2:172.21.15.60:6808/12891,v1:172.21.15.60:6809/12891] conn(0x7f5ea805a190 0x7f5ea805c5f0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:05.845 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.799+0000 7f5ec51d3700 1 --2- 172.21.15.60:0/618304502 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5ea8053aa0 0x7f5ea8055f60 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 rx=0x7f5eb000b000 tx=0x7f5eb0006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:05.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.799+0000 7f5eb67fc700 1 -- 172.21.15.60:0/618304502 <== osd.2 v2:172.21.15.60:6808/12891 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f5ea805cbd0 con 0x7f5ea805a190 2022-06-29T17:09:05.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.811+0000 7f5ec7437700 1 -- 172.21.15.60:0/618304502 --> [v2:172.21.15.60:6808/12891,v1:172.21.15.60:6809/12891] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f5ec0066360 con 0x7f5ea805a190 2022-06-29T17:09:05.847 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.811+0000 7f5eb67fc700 1 -- 172.21.15.60:0/618304502 <== osd.2 v2:172.21.15.60:6808/12891 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f5ec0066360 con 0x7f5ea805a190 2022-06-29T17:09:05.848 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T17:09:05.849 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-06-29T17:09:05.850 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T17:09:05.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.811+0000 7f5ec7437700 1 -- 172.21.15.60:0/618304502 >> [v2:172.21.15.60:6808/12891,v1:172.21.15.60:6809/12891] conn(0x7f5ea805a190 msgr2=0x7f5ea805c5f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:05.853 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.811+0000 7f5ec7437700 1 --2- 172.21.15.60:0/618304502 >> [v2:172.21.15.60:6808/12891,v1:172.21.15.60:6809/12891] conn(0x7f5ea805a190 0x7f5ea805c5f0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.811+0000 7f5ec7437700 1 -- 172.21.15.60:0/618304502 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5ea8053aa0 msgr2=0x7f5ea8055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:05.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.811+0000 7f5ec7437700 1 --2- 172.21.15.60:0/618304502 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f5ea8053aa0 0x7f5ea8055f60 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 rx=0x7f5eb000b000 tx=0x7f5eb0006df0).stop 2022-06-29T17:09:05.854 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.815+0000 7f5ec7437700 1 -- 172.21.15.60:0/618304502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ec013f790 msgr2=0x7f5ec00b4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:05.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.815+0000 7f5ec7437700 1 --2- 172.21.15.60:0/618304502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5ec013f790 0x7f5ec00b4950 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 rx=0x7f5ec00acf50 tx=0x7f5ebc015000).stop 2022-06-29T17:09:05.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.815+0000 7f5ec59d4700 1 -- 172.21.15.60:0/618304502 reap_dead start 2022-06-29T17:09:05.855 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.815+0000 7f5ec7437700 1 -- 172.21.15.60:0/618304502 shutdown_connections 2022-06-29T17:09:05.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.815+0000 7f5ec7437700 1 -- 172.21.15.60:0/618304502 >> 172.21.15.60:0/618304502 conn(0x7f5ec0063b50 msgr2=0x7f5ec0140f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:05.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.815+0000 7f5ec7437700 1 -- 172.21.15.60:0/618304502 shutdown_connections 2022-06-29T17:09:05.856 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.815+0000 7f5ec7437700 1 -- 172.21.15.60:0/618304502 wait complete. 2022-06-29T17:09:05.857 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2022-06-29T17:09:05.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.963+0000 7fba25a0f700 1 Processor -- start 2022-06-29T17:09:05.969 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.963+0000 7fba25a0f700 1 -- start start 2022-06-29T17:09:05.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.963+0000 7fba25a0f700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fba200b5eb0 0x7fba20142a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.963+0000 7fba25a0f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fba200b48b0 0x7fba200b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.963+0000 7fba25a0f700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba200b4d10 con 0x7fba200b2c50 2022-06-29T17:09:05.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.963+0000 7fba25a0f700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba200b54e0 con 0x7fba200b5eb0 2022-06-29T17:09:05.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.963+0000 7fba25a0f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba200b6290 con 0x7fba200b48b0 2022-06-29T17:09:05.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.963+0000 7fba24a0d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fba200b5eb0 0x7fba20142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.963+0000 7fba2520e700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fba200b2c50 0x7fba200b3090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34806/0 (socket says 172.21.15.60:34806) 2022-06-29T17:09:05.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.963+0000 7fba24a0d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fba200b5eb0 0x7fba20142a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54950/0 (socket says 172.21.15.60:54950) 2022-06-29T17:09:05.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.963+0000 7fba2520e700 1 -- 172.21.15.60:0/2041935162 learned_addr learned my addr 172.21.15.60:0/2041935162 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T17:09:05.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.963+0000 7fba1ffff700 1 --2- 172.21.15.60:0/2041935162 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fba200b48b0 0x7fba200b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.963+0000 7fba24a0d700 1 -- 172.21.15.60:0/2041935162 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fba200b48b0 msgr2=0x7fba200b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T17:09:05.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.963+0000 7fba24a0d700 1 --2- 172.21.15.60:0/2041935162 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fba200b48b0 0x7fba200b05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.963+0000 7fba24a0d700 1 -- 172.21.15.60:0/2041935162 >> v1:172.21.15.60:6789/0 conn(0x7fba200b2c50 legacy=0x7fba200b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:05.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.963+0000 7fba1f7fe700 1 -- 172.21.15.60:0/2041935162 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 346631047 0 0) 0x7fba200b4d10 con 0x7fba200b2c50 2022-06-29T17:09:05.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.963+0000 7fba24a0d700 1 -- 172.21.15.60:0/2041935162 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba20142f10 con 0x7fba200b5eb0 2022-06-29T17:09:05.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.963+0000 7fba24a0d700 1 --2- 172.21.15.60:0/2041935162 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba200b5eb0 0x7fba20142a20 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 rx=0x7fba10002f70 tx=0x7fba10006df0).ready entity=mon.0 client_cookie=312fa1de179a19ec server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:05.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.963+0000 7fba1f7fe700 1 -- 172.21.15.60:0/2041935162 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fba1000d570 con 0x7fba200b5eb0 2022-06-29T17:09:05.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.963+0000 7fba1f7fe700 1 -- 172.21.15.60:0/2041935162 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba10000e90 con 0x7fba200b5eb0 2022-06-29T17:09:05.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.963+0000 7fba1f7fe700 1 -- 172.21.15.60:0/2041935162 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fba1000d8c0 con 0x7fba200b5eb0 2022-06-29T17:09:05.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.967+0000 7fba25a0f700 1 -- 172.21.15.60:0/2041935162 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba200b5eb0 msgr2=0x7fba20142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:05.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.967+0000 7fba25a0f700 1 --2- 172.21.15.60:0/2041935162 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba200b5eb0 0x7fba20142a20 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 rx=0x7fba10002f70 tx=0x7fba10006df0).stop 2022-06-29T17:09:05.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.967+0000 7fba25a0f700 1 -- 172.21.15.60:0/2041935162 shutdown_connections 2022-06-29T17:09:05.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.967+0000 7fba25a0f700 1 --2- 172.21.15.60:0/2041935162 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fba200b48b0 0x7fba200b05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.967+0000 7fba25a0f700 1 --2- 172.21.15.60:0/2041935162 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba200b5eb0 0x7fba20142a20 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.967+0000 7fba25a0f700 1 -- 172.21.15.60:0/2041935162 >> 172.21.15.60:0/2041935162 conn(0x7fba20062090 msgr2=0x7fba20062470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:05.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.967+0000 7fba25a0f700 1 -- 172.21.15.60:0/2041935162 shutdown_connections 2022-06-29T17:09:05.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.967+0000 7fba25a0f700 1 -- 172.21.15.60:0/2041935162 wait complete. 2022-06-29T17:09:05.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.967+0000 7fba25a0f700 1 Processor -- start 2022-06-29T17:09:05.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.967+0000 7fba25a0f700 1 -- start start 2022-06-29T17:09:05.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.967+0000 7fba25a0f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fba200b2c50 0x7fba200ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.967+0000 7fba25a0f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba200b48b0 0x7fba20140600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.967+0000 7fba25a0f700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fba200b5eb0 0x7fba200aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.967+0000 7fba25a0f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba20145140 con 0x7fba200b48b0 2022-06-29T17:09:05.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.967+0000 7fba25a0f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba200ae9a0 con 0x7fba200b2c50 2022-06-29T17:09:05.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.967+0000 7fba25a0f700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba20144ea0 con 0x7fba200b5eb0 2022-06-29T17:09:05.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.967+0000 7fba1ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba200b48b0 0x7fba20140600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.967+0000 7fba2520e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fba200b5eb0 0x7fba200aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.967+0000 7fba1ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba200b48b0 0x7fba20140600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54960/0 (socket says 172.21.15.60:54960) 2022-06-29T17:09:05.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.967+0000 7fba2520e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fba200b5eb0 0x7fba200aca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:39820/0 (socket says 172.21.15.60:39820) 2022-06-29T17:09:05.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.967+0000 7fba24a0d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fba200b2c50 0x7fba200ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.967+0000 7fba1ffff700 1 -- 172.21.15.60:0/3592136329 learned_addr learned my addr 172.21.15.60:0/3592136329 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T17:09:05.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.971+0000 7fba2520e700 1 -- 172.21.15.60:0/3592136329 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fba200b2c50 msgr2=0x7fba200ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:05.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.971+0000 7fba2520e700 1 --2- 172.21.15.60:0/3592136329 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fba200b2c50 0x7fba200ad4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.971+0000 7fba2520e700 1 -- 172.21.15.60:0/3592136329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba200b48b0 msgr2=0x7fba20140600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:05.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.971+0000 7fba2520e700 1 --2- 172.21.15.60:0/3592136329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba200b48b0 0x7fba20140600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:05.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.971+0000 7fba2520e700 1 -- 172.21.15.60:0/3592136329 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba1000b180 con 0x7fba200b5eb0 2022-06-29T17:09:05.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.971+0000 7fba2520e700 1 --2- 172.21.15.60:0/3592136329 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fba200b5eb0 0x7fba200aca20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fba18001250 tx=0x7fba18007d00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:05.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.971+0000 7fba1dffb700 1 -- 172.21.15.60:0/3592136329 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fba18005ba0 con 0x7fba200b5eb0 2022-06-29T17:09:05.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.971+0000 7fba1dffb700 1 -- 172.21.15.60:0/3592136329 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba18005d00 con 0x7fba200b5eb0 2022-06-29T17:09:05.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.971+0000 7fba25a0f700 1 -- 172.21.15.60:0/3592136329 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba200adca0 con 0x7fba200b5eb0 2022-06-29T17:09:05.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.971+0000 7fba1dffb700 1 -- 172.21.15.60:0/3592136329 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fba18004a90 con 0x7fba200b5eb0 2022-06-29T17:09:05.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.971+0000 7fba25a0f700 1 -- 172.21.15.60:0/3592136329 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba200ad0b0 con 0x7fba200b5eb0 2022-06-29T17:09:05.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.971+0000 7fba25a0f700 1 -- 172.21.15.60:0/3592136329 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba20144cf0 con 0x7fba200b5eb0 2022-06-29T17:09:05.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.971+0000 7fba1dffb700 1 -- 172.21.15.60:0/3592136329 <== mon.2 v2:172.21.15.92:3301/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fba18004bf0 con 0x7fba200b5eb0 2022-06-29T17:09:05.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.971+0000 7fba1dffb700 1 --2- 172.21.15.60:0/3592136329 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fba04053a10 0x7fba04055ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:05.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.971+0000 7fba1dffb700 1 -- 172.21.15.60:0/3592136329 <== mon.2 v2:172.21.15.92:3301/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fba1806b7a0 con 0x7fba200b5eb0 2022-06-29T17:09:05.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.971+0000 7fba24a0d700 1 --2- 172.21.15.60:0/3592136329 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fba04053a10 0x7fba04055ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:05.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.975+0000 7fba24a0d700 1 --2- 172.21.15.60:0/3592136329 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fba04053a10 0x7fba04055ed0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0x7fba1001a000 tx=0x7fba10014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:05.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:05.979+0000 7fba1dffb700 1 -- 172.21.15.60:0/3592136329 <== mon.2 v2:172.21.15.92:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fba18042830 con 0x7fba200b5eb0 2022-06-29T17:09:06.118 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.111+0000 7fba25a0f700 1 -- 172.21.15.60:0/3592136329 --> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fba200b4f70 con 0x7fba04053a10 2022-06-29T17:09:06.119 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.111+0000 7fba1dffb700 1 -- 172.21.15.60:0/3592136329 <== mgr.4103 v2:172.21.15.92:6832/12872 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fba200b4f70 con 0x7fba04053a10 2022-06-29T17:09:06.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.115+0000 7fba25a0f700 1 -- 172.21.15.60:0/3592136329 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fba04053a10 msgr2=0x7fba04055ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:06.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.115+0000 7fba25a0f700 1 --2- 172.21.15.60:0/3592136329 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fba04053a10 0x7fba04055ed0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0x7fba1001a000 tx=0x7fba10014000).stop 2022-06-29T17:09:06.123 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.115+0000 7fba25a0f700 1 -- 172.21.15.60:0/3592136329 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fba200b5eb0 msgr2=0x7fba200aca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:06.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.115+0000 7fba25a0f700 1 --2- 172.21.15.60:0/3592136329 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fba200b5eb0 0x7fba200aca20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fba18001250 tx=0x7fba18007d00).stop 2022-06-29T17:09:06.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.115+0000 7fba25a0f700 1 -- 172.21.15.60:0/3592136329 shutdown_connections 2022-06-29T17:09:06.124 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.115+0000 7fba25a0f700 1 --2- 172.21.15.60:0/3592136329 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fba04053a10 0x7fba04055ed0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:06.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.115+0000 7fba25a0f700 1 --2- 172.21.15.60:0/3592136329 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fba200b5eb0 0x7fba200aca20 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:06.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.115+0000 7fba25a0f700 1 --2- 172.21.15.60:0/3592136329 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba200b48b0 0x7fba20140600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:06.125 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.115+0000 7fba25a0f700 1 --2- 172.21.15.60:0/3592136329 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fba200b2c50 0x7fba200ad4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:06.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.115+0000 7fba25a0f700 1 -- 172.21.15.60:0/3592136329 >> 172.21.15.60:0/3592136329 conn(0x7fba20062090 msgr2=0x7fba200b7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:06.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.115+0000 7fba25a0f700 1 -- 172.21.15.60:0/3592136329 shutdown_connections 2022-06-29T17:09:06.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.115+0000 7fba25a0f700 1 -- 172.21.15.60:0/3592136329 wait complete. 2022-06-29T17:09:06.127 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 2 to deep-scrub 2022-06-29T17:09:06.137 INFO:tasks.ceph:Scrubbing osd.3 2022-06-29T17:09:06.138 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2022-06-29T17:09:06.249 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.243+0000 7f8b5c760700 1 Processor -- start 2022-06-29T17:09:06.250 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.243+0000 7f8b5c760700 1 -- start start 2022-06-29T17:09:06.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.243+0000 7f8b5c760700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8b540ae7c0 0x7f8b54141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:06.257 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.243+0000 7f8b5c760700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8b540b46c0 0x7f8b540b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:06.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.243+0000 7f8b5c760700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b54143a40 con 0x7f8b540b4010 2022-06-29T17:09:06.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.243+0000 7f8b5c760700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b540aee60 con 0x7f8b540b46c0 2022-06-29T17:09:06.258 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.243+0000 7f8b5c760700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b540af100 con 0x7f8b540ae7c0 2022-06-29T17:09:06.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.243+0000 7f8b5acfd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f8b540b46c0 0x7f8b540b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:06.259 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.243+0000 7f8b59cfb700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f8b540b4010 0x7f8b54142490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34822/0 (socket says 172.21.15.60:34822) 2022-06-29T17:09:06.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.243+0000 7f8b59cfb700 1 -- 172.21.15.60:0/2306116412 learned_addr learned my addr 172.21.15.60:0/2306116412 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T17:09:06.260 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.243+0000 7f8b5a4fc700 1 --2- 172.21.15.60:0/2306116412 >> v2:172.21.15.92:3300/0 conn(0x7f8b540ae7c0 0x7f8b54141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:06.261 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.243+0000 7f8b5acfd700 1 -- 172.21.15.60:0/2306116412 >> v2:172.21.15.92:3300/0 conn(0x7f8b540ae7c0 msgr2=0x7f8b54141fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T17:09:06.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.243+0000 7f8b5acfd700 1 --2- 172.21.15.60:0/2306116412 >> v2:172.21.15.92:3300/0 conn(0x7f8b540ae7c0 0x7f8b54141fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:06.262 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.243+0000 7f8b5acfd700 1 -- 172.21.15.60:0/2306116412 >> v1:172.21.15.60:6789/0 conn(0x7f8b540b4010 legacy=0x7f8b54142490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:06.263 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b5acfd700 1 -- 172.21.15.60:0/2306116412 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b540aeba0 con 0x7f8b540b46c0 2022-06-29T17:09:06.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b5a4fc700 1 --2- 172.21.15.60:0/2306116412 >> v2:172.21.15.92:3300/0 conn(0x7f8b540ae7c0 0x7f8b54141fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T17:09:06.264 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b5acfd700 1 --2- 172.21.15.60:0/2306116412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b540b46c0 0x7f8b540b4af0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 rx=0x7f8b4c0032f0 tx=0x7f8b4c0091d0).ready entity=mon.0 client_cookie=6452f93a7586a990 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:06.265 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b594fa700 1 -- 172.21.15.60:0/2306116412 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b4c009ea0 con 0x7f8b540b46c0 2022-06-29T17:09:06.266 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b594fa700 1 -- 172.21.15.60:0/2306116412 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b4c00b870 con 0x7f8b540b46c0 2022-06-29T17:09:06.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b594fa700 1 -- 172.21.15.60:0/2306116412 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b4c00b9e0 con 0x7f8b540b46c0 2022-06-29T17:09:06.267 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b5c760700 1 -- 172.21.15.60:0/2306116412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b540b46c0 msgr2=0x7f8b540b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:06.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b5c760700 1 --2- 172.21.15.60:0/2306116412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b540b46c0 0x7f8b540b4af0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 rx=0x7f8b4c0032f0 tx=0x7f8b4c0091d0).stop 2022-06-29T17:09:06.268 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b5c760700 1 -- 172.21.15.60:0/2306116412 shutdown_connections 2022-06-29T17:09:06.269 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b5c760700 1 --2- 172.21.15.60:0/2306116412 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b540b46c0 0x7f8b540b4af0 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:06.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b5c760700 1 --2- 172.21.15.60:0/2306116412 >> v2:172.21.15.92:3300/0 conn(0x7f8b540ae7c0 0x7f8b54141fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:06.270 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b5c760700 1 -- 172.21.15.60:0/2306116412 >> 172.21.15.60:0/2306116412 conn(0x7f8b540b9340 msgr2=0x7f8b540b9750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:06.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b5c760700 1 -- 172.21.15.60:0/2306116412 shutdown_connections 2022-06-29T17:09:06.271 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b5c760700 1 -- 172.21.15.60:0/2306116412 wait complete. 2022-06-29T17:09:06.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b5c760700 1 Processor -- start 2022-06-29T17:09:06.272 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b5c760700 1 -- start start 2022-06-29T17:09:06.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b5c760700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b540ae7c0 0x7f8b540af820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:06.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b5c760700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b540b4010 0x7f8b540b1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:06.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b5c760700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8b540b46c0 0x7f8b540b1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:06.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b5c760700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b541567b0 con 0x7f8b540ae7c0 2022-06-29T17:09:06.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b5c760700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b54156a50 con 0x7f8b540b4010 2022-06-29T17:09:06.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b5c760700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b54156cf0 con 0x7f8b540b46c0 2022-06-29T17:09:06.275 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b5a4fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b540ae7c0 0x7f8b540af820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:06.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b5a4fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b540ae7c0 0x7f8b540af820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54972/0 (socket says 172.21.15.60:54972) 2022-06-29T17:09:06.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b5a4fc700 1 -- 172.21.15.60:0/1997244575 learned_addr learned my addr 172.21.15.60:0/1997244575 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T17:09:06.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b59cfb700 1 --2- 172.21.15.60:0/1997244575 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b540b4010 0x7f8b540b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:06.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b5acfd700 1 --2- 172.21.15.60:0/1997244575 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8b540b46c0 0x7f8b540b1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:06.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b5a4fc700 1 -- 172.21.15.60:0/1997244575 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8b540b46c0 msgr2=0x7f8b540b1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:06.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b5a4fc700 1 --2- 172.21.15.60:0/1997244575 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8b540b46c0 0x7f8b540b1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:06.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b5a4fc700 1 -- 172.21.15.60:0/1997244575 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b540b4010 msgr2=0x7f8b540b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:06.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b5a4fc700 1 --2- 172.21.15.60:0/1997244575 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b540b4010 0x7f8b540b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:06.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b5a4fc700 1 -- 172.21.15.60:0/1997244575 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b4c009470 con 0x7f8b540ae7c0 2022-06-29T17:09:06.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b59cfb700 1 --2- 172.21.15.60:0/1997244575 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b540b4010 0x7f8b540b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T17:09:06.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b5acfd700 1 --2- 172.21.15.60:0/1997244575 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f8b540b46c0 0x7f8b540b1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T17:09:06.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.247+0000 7f8b5a4fc700 1 --2- 172.21.15.60:0/1997244575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b540ae7c0 0x7f8b540af820 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 rx=0x7f8b48006940 tx=0x7f8b48008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:06.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.251+0000 7f8b477fe700 1 -- 172.21.15.60:0/1997244575 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b48011780 con 0x7f8b540ae7c0 2022-06-29T17:09:06.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.251+0000 7f8b477fe700 1 -- 172.21.15.60:0/1997244575 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b480118e0 con 0x7f8b540ae7c0 2022-06-29T17:09:06.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.251+0000 7f8b5c760700 1 -- 172.21.15.60:0/1997244575 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b54157190 con 0x7f8b540ae7c0 2022-06-29T17:09:06.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.251+0000 7f8b477fe700 1 -- 172.21.15.60:0/1997244575 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b48011c40 con 0x7f8b540ae7c0 2022-06-29T17:09:06.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.251+0000 7f8b5c760700 1 -- 172.21.15.60:0/1997244575 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b541564c0 con 0x7f8b540ae7c0 2022-06-29T17:09:06.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.251+0000 7f8b5c760700 1 -- 172.21.15.60:0/1997244575 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8b540b1f50 con 0x7f8b540ae7c0 2022-06-29T17:09:06.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.251+0000 7f8b477fe700 1 -- 172.21.15.60:0/1997244575 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f8b48012420 con 0x7f8b540ae7c0 2022-06-29T17:09:06.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.251+0000 7f8b477fe700 1 --2- 172.21.15.60:0/1997244575 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8b34053b00 0x7f8b34055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:06.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.251+0000 7f8b477fe700 1 -- 172.21.15.60:0/1997244575 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f8b4806ca60 con 0x7f8b540ae7c0 2022-06-29T17:09:06.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.251+0000 7f8b477fe700 1 --2- 172.21.15.60:0/1997244575 >> [v2:172.21.15.60:6800/12889,v1:172.21.15.60:6801/12889] conn(0x7f8b3405a1f0 0x7f8b3405c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:06.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.251+0000 7f8b477fe700 1 -- 172.21.15.60:0/1997244575 --> [v2:172.21.15.60:6800/12889,v1:172.21.15.60:6801/12889] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8b3405cc30 con 0x7f8b3405a1f0 2022-06-29T17:09:06.286 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.251+0000 7f8b477fe700 1 -- 172.21.15.60:0/1997244575 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=150) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b4806cd30 con 0x7f8b540ae7c0 2022-06-29T17:09:06.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.251+0000 7f8b59cfb700 1 --2- 172.21.15.60:0/1997244575 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8b34053b00 0x7f8b34055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:06.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.251+0000 7f8b5acfd700 1 --2- 172.21.15.60:0/1997244575 >> [v2:172.21.15.60:6800/12889,v1:172.21.15.60:6801/12889] conn(0x7f8b3405a1f0 0x7f8b3405c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:06.287 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.255+0000 7f8b5acfd700 1 --2- 172.21.15.60:0/1997244575 >> [v2:172.21.15.60:6800/12889,v1:172.21.15.60:6801/12889] conn(0x7f8b3405a1f0 0x7f8b3405c650 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:06.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.255+0000 7f8b59cfb700 1 --2- 172.21.15.60:0/1997244575 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8b34053b00 0x7f8b34055fc0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0x7f8b5000b000 tx=0x7f8b50009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:06.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.255+0000 7f8b477fe700 1 -- 172.21.15.60:0/1997244575 <== osd.3 v2:172.21.15.60:6800/12889 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f8b3405cc30 con 0x7f8b3405a1f0 2022-06-29T17:09:06.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.267+0000 7f8b5c760700 1 -- 172.21.15.60:0/1997244575 --> [v2:172.21.15.60:6800/12889,v1:172.21.15.60:6801/12889] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f8b540573d0 con 0x7f8b3405a1f0 2022-06-29T17:09:06.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.271+0000 7f8b477fe700 1 -- 172.21.15.60:0/1997244575 <== osd.3 v2:172.21.15.60:6800/12889 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f8b540573d0 con 0x7f8b3405a1f0 2022-06-29T17:09:06.289 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T17:09:06.290 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-06-29T17:09:06.290 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T17:09:06.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.271+0000 7f8b5c760700 1 -- 172.21.15.60:0/1997244575 >> [v2:172.21.15.60:6800/12889,v1:172.21.15.60:6801/12889] conn(0x7f8b3405a1f0 msgr2=0x7f8b3405c650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:06.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.271+0000 7f8b5c760700 1 --2- 172.21.15.60:0/1997244575 >> [v2:172.21.15.60:6800/12889,v1:172.21.15.60:6801/12889] conn(0x7f8b3405a1f0 0x7f8b3405c650 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:06.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.271+0000 7f8b5c760700 1 -- 172.21.15.60:0/1997244575 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8b34053b00 msgr2=0x7f8b34055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:06.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.271+0000 7f8b5c760700 1 --2- 172.21.15.60:0/1997244575 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8b34053b00 0x7f8b34055fc0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0x7f8b5000b000 tx=0x7f8b50009040).stop 2022-06-29T17:09:06.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.271+0000 7f8b5c760700 1 -- 172.21.15.60:0/1997244575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b540ae7c0 msgr2=0x7f8b540af820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:06.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.271+0000 7f8b5c760700 1 --2- 172.21.15.60:0/1997244575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8b540ae7c0 0x7f8b540af820 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 rx=0x7f8b48006940 tx=0x7f8b48008040).stop 2022-06-29T17:09:06.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.271+0000 7f8b5acfd700 1 -- 172.21.15.60:0/1997244575 reap_dead start 2022-06-29T17:09:06.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.271+0000 7f8b5c760700 1 -- 172.21.15.60:0/1997244575 shutdown_connections 2022-06-29T17:09:06.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.271+0000 7f8b5c760700 1 -- 172.21.15.60:0/1997244575 >> 172.21.15.60:0/1997244575 conn(0x7f8b540b9340 msgr2=0x7f8b540b8b30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:06.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.271+0000 7f8b5c760700 1 -- 172.21.15.60:0/1997244575 shutdown_connections 2022-06-29T17:09:06.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.271+0000 7f8b5c760700 1 -- 172.21.15.60:0/1997244575 wait complete. 2022-06-29T17:09:06.296 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2022-06-29T17:09:06.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.399+0000 7f561f032700 1 Processor -- start 2022-06-29T17:09:06.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.399+0000 7f561f032700 1 -- start start 2022-06-29T17:09:06.410 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.403+0000 7f561f032700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f56180b0cf0 0x7f56181429e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:06.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.403+0000 7f561f032700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5618142ed0 0x7f561813fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:06.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.403+0000 7f561f032700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f561805f0b0 con 0x7f561805c8f0 2022-06-29T17:09:06.411 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.403+0000 7f561f032700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5618060670 con 0x7f5618142ed0 2022-06-29T17:09:06.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.403+0000 7f561f032700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56180b16e0 con 0x7f56180b0cf0 2022-06-29T17:09:06.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.403+0000 7f561e831700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5618142ed0 0x7f561813fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:06.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.403+0000 7f561e030700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f561805c8f0 0x7f561805ff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34834/0 (socket says 172.21.15.60:34834) 2022-06-29T17:09:06.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.403+0000 7f561e831700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f5618142ed0 0x7f561813fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54982/0 (socket says 172.21.15.60:54982) 2022-06-29T17:09:06.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.403+0000 7f561e030700 1 -- 172.21.15.60:0/2177645421 learned_addr learned my addr 172.21.15.60:0/2177645421 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T17:09:06.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.403+0000 7f561d82f700 1 --2- 172.21.15.60:0/2177645421 >> v2:172.21.15.92:3300/0 conn(0x7f56180b0cf0 0x7f56181429e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:06.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.403+0000 7f561e831700 1 -- 172.21.15.60:0/2177645421 >> v2:172.21.15.92:3300/0 conn(0x7f56180b0cf0 msgr2=0x7f56181429e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T17:09:06.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.403+0000 7f561e831700 1 --2- 172.21.15.60:0/2177645421 >> v2:172.21.15.92:3300/0 conn(0x7f56180b0cf0 0x7f56181429e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:06.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.403+0000 7f561e831700 1 -- 172.21.15.60:0/2177645421 >> v1:172.21.15.60:6789/0 conn(0x7f561805c8f0 legacy=0x7f561805ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:06.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.403+0000 7f561d02e700 1 -- 172.21.15.60:0/2177645421 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2883492740 0 0) 0x7f561805f0b0 con 0x7f561805c8f0 2022-06-29T17:09:06.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.403+0000 7f561e831700 1 -- 172.21.15.60:0/2177645421 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f561805ccd0 con 0x7f5618142ed0 2022-06-29T17:09:06.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.403+0000 7f561e831700 1 --2- 172.21.15.60:0/2177645421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5618142ed0 0x7f561813fad0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 rx=0x7f5610002f70 tx=0x7f5610006df0).ready entity=mon.0 client_cookie=c384465d344146ec server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:06.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.403+0000 7f561d02e700 1 -- 172.21.15.60:0/2177645421 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f561000d570 con 0x7f5618142ed0 2022-06-29T17:09:06.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.403+0000 7f561d02e700 1 -- 172.21.15.60:0/2177645421 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5610000e90 con 0x7f5618142ed0 2022-06-29T17:09:06.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.403+0000 7f561d02e700 1 -- 172.21.15.60:0/2177645421 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f561000d8c0 con 0x7f5618142ed0 2022-06-29T17:09:06.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.403+0000 7f561f032700 1 -- 172.21.15.60:0/2177645421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5618142ed0 msgr2=0x7f561813fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:06.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.403+0000 7f561f032700 1 --2- 172.21.15.60:0/2177645421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5618142ed0 0x7f561813fad0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 rx=0x7f5610002f70 tx=0x7f5610006df0).stop 2022-06-29T17:09:06.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.403+0000 7f561f032700 1 -- 172.21.15.60:0/2177645421 shutdown_connections 2022-06-29T17:09:06.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.403+0000 7f561f032700 1 --2- 172.21.15.60:0/2177645421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5618142ed0 0x7f561813fad0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:06.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.403+0000 7f561f032700 1 --2- 172.21.15.60:0/2177645421 >> v2:172.21.15.92:3300/0 conn(0x7f56180b0cf0 0x7f56181429e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:06.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.403+0000 7f561f032700 1 -- 172.21.15.60:0/2177645421 >> 172.21.15.60:0/2177645421 conn(0x7f5618061c80 msgr2=0x7f56180637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:06.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.403+0000 7f561f032700 1 -- 172.21.15.60:0/2177645421 shutdown_connections 2022-06-29T17:09:06.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.403+0000 7f561f032700 1 -- 172.21.15.60:0/2177645421 wait complete. 2022-06-29T17:09:06.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.407+0000 7f561f032700 1 Processor -- start 2022-06-29T17:09:06.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.407+0000 7f561f032700 1 -- start start 2022-06-29T17:09:06.425 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.407+0000 7f561f032700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f561805c8f0 0x7f5618155010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:06.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.407+0000 7f561f032700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56180b0cf0 0x7f56181400d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:06.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.407+0000 7f561f032700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5618142ed0 0x7f5618146470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:06.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.407+0000 7f561f032700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5618146020 con 0x7f5618142ed0 2022-06-29T17:09:06.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.407+0000 7f561f032700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5618145a40 con 0x7f56180b0cf0 2022-06-29T17:09:06.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.407+0000 7f561f032700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5618145ce0 con 0x7f561805c8f0 2022-06-29T17:09:06.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.411+0000 7f561e831700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5618142ed0 0x7f5618146470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:06.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.411+0000 7f561e831700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5618142ed0 0x7f5618146470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54988/0 (socket says 172.21.15.60:54988) 2022-06-29T17:09:06.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.411+0000 7f561e831700 1 -- 172.21.15.60:0/3113922442 learned_addr learned my addr 172.21.15.60:0/3113922442 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T17:09:06.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.411+0000 7f561d82f700 1 --2- 172.21.15.60:0/3113922442 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56180b0cf0 0x7f56181400d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:06.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.411+0000 7f561e030700 1 --2- 172.21.15.60:0/3113922442 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f561805c8f0 0x7f5618155010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:06.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.411+0000 7f561e831700 1 -- 172.21.15.60:0/3113922442 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f561805c8f0 msgr2=0x7f5618155010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:06.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.411+0000 7f561e831700 1 --2- 172.21.15.60:0/3113922442 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f561805c8f0 0x7f5618155010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:06.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.411+0000 7f561e831700 1 -- 172.21.15.60:0/3113922442 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56180b0cf0 msgr2=0x7f56181400d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:06.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.411+0000 7f561e831700 1 --2- 172.21.15.60:0/3113922442 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56180b0cf0 0x7f56181400d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:06.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.411+0000 7f561e831700 1 -- 172.21.15.60:0/3113922442 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f561000b180 con 0x7f5618142ed0 2022-06-29T17:09:06.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.411+0000 7f561d82f700 1 --2- 172.21.15.60:0/3113922442 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56180b0cf0 0x7f56181400d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T17:09:06.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.411+0000 7f561e831700 1 --2- 172.21.15.60:0/3113922442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5618142ed0 0x7f5618146470 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 rx=0x7f5610002f70 tx=0x7f5610014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:06.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.411+0000 7f56177fe700 1 -- 172.21.15.60:0/3113922442 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5610013da0 con 0x7f5618142ed0 2022-06-29T17:09:06.432 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.411+0000 7f56177fe700 1 -- 172.21.15.60:0/3113922442 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f561000de60 con 0x7f5618142ed0 2022-06-29T17:09:06.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.411+0000 7f561f032700 1 -- 172.21.15.60:0/3113922442 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56180ae4b0 con 0x7f5618142ed0 2022-06-29T17:09:06.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.411+0000 7f56177fe700 1 -- 172.21.15.60:0/3113922442 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f561001e5f0 con 0x7f5618142ed0 2022-06-29T17:09:06.433 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.411+0000 7f561f032700 1 -- 172.21.15.60:0/3113922442 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56180b0160 con 0x7f5618142ed0 2022-06-29T17:09:06.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.415+0000 7f561f032700 1 -- 172.21.15.60:0/3113922442 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56180ade00 con 0x7f5618142ed0 2022-06-29T17:09:06.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.415+0000 7f56177fe700 1 -- 172.21.15.60:0/3113922442 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f5610025030 con 0x7f5618142ed0 2022-06-29T17:09:06.434 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.415+0000 7f56177fe700 1 --2- 172.21.15.60:0/3113922442 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f55fc0539e0 0x7f55fc055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:06.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.415+0000 7f56177fe700 1 -- 172.21.15.60:0/3113922442 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f561006c840 con 0x7f5618142ed0 2022-06-29T17:09:06.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.415+0000 7f561e030700 1 --2- 172.21.15.60:0/3113922442 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f55fc0539e0 0x7f55fc055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:06.435 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.415+0000 7f561e030700 1 --2- 172.21.15.60:0/3113922442 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f55fc0539e0 0x7f55fc055ea0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0x7f560800d000 tx=0x7f5608007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:06.436 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.423+0000 7f56177fe700 1 -- 172.21.15.60:0/3113922442 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5610043a40 con 0x7f5618142ed0 2022-06-29T17:09:06.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.551+0000 7f561f032700 1 -- 172.21.15.60:0/3113922442 --> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f56180573d0 con 0x7f55fc0539e0 2022-06-29T17:09:06.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.551+0000 7f56177fe700 1 -- 172.21.15.60:0/3113922442 <== mgr.4103 v2:172.21.15.92:6832/12872 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f56180573d0 con 0x7f55fc0539e0 2022-06-29T17:09:06.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.551+0000 7f561f032700 1 -- 172.21.15.60:0/3113922442 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f55fc0539e0 msgr2=0x7f55fc055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:06.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.551+0000 7f561f032700 1 --2- 172.21.15.60:0/3113922442 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f55fc0539e0 0x7f55fc055ea0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0x7f560800d000 tx=0x7f5608007a00).stop 2022-06-29T17:09:06.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.551+0000 7f561f032700 1 -- 172.21.15.60:0/3113922442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5618142ed0 msgr2=0x7f5618146470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:06.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.551+0000 7f561f032700 1 --2- 172.21.15.60:0/3113922442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5618142ed0 0x7f5618146470 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 rx=0x7f5610002f70 tx=0x7f5610014000).stop 2022-06-29T17:09:06.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.551+0000 7f561f032700 1 -- 172.21.15.60:0/3113922442 shutdown_connections 2022-06-29T17:09:06.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.551+0000 7f561f032700 1 --2- 172.21.15.60:0/3113922442 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f55fc0539e0 0x7f55fc055ea0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:06.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.551+0000 7f561f032700 1 --2- 172.21.15.60:0/3113922442 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5618142ed0 0x7f5618146470 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:06.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.551+0000 7f561f032700 1 --2- 172.21.15.60:0/3113922442 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56180b0cf0 0x7f56181400d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:06.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.551+0000 7f561f032700 1 --2- 172.21.15.60:0/3113922442 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f561805c8f0 0x7f5618155010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:06.563 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.551+0000 7f561f032700 1 -- 172.21.15.60:0/3113922442 >> 172.21.15.60:0/3113922442 conn(0x7f5618061c80 msgr2=0x7f56180636e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:06.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.551+0000 7f561f032700 1 -- 172.21.15.60:0/3113922442 shutdown_connections 2022-06-29T17:09:06.564 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.551+0000 7f561f032700 1 -- 172.21.15.60:0/3113922442 wait complete. 2022-06-29T17:09:06.565 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 3 to deep-scrub 2022-06-29T17:09:06.570 INFO:tasks.ceph:Scrubbing osd.4 2022-06-29T17:09:06.571 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2022-06-29T17:09:06.682 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.675+0000 7f2debfff700 1 Processor -- start 2022-06-29T17:09:06.683 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.675+0000 7f2debfff700 1 -- start start 2022-06-29T17:09:06.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.675+0000 7f2debfff700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2dec05d070 0x7f2dec067030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:06.695 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.675+0000 7f2debfff700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dec0aecb0 con 0x7f2dec0b0e80 2022-06-29T17:09:06.696 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.675+0000 7f2debfff700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dec146660 con 0x7f2dec0aca20 2022-06-29T17:09:06.696 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.675+0000 7f2debfff700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dec142870 con 0x7f2dec05d070 2022-06-29T17:09:06.697 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.679+0000 7f2deaffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f2dec05d070 0x7f2dec067030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:06.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.679+0000 7f2dea7fc700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2dec0b0e80 0x7f2dec145f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34848/0 (socket says 172.21.15.60:34848) 2022-06-29T17:09:06.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.679+0000 7f2dea7fc700 1 -- 172.21.15.60:0/1960726158 learned_addr learned my addr 172.21.15.60:0/1960726158 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T17:09:06.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.679+0000 7f2de9ffb700 1 -- 172.21.15.60:0/1960726158 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1960491677 0 0) 0x7f2dec0aecb0 con 0x7f2dec0b0e80 2022-06-29T17:09:06.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.679+0000 7f2de9ffb700 1 -- 172.21.15.60:0/1960726158 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dcc003260 con 0x7f2dec0b0e80 2022-06-29T17:09:06.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.679+0000 7f2deaffd700 1 -- 172.21.15.60:0/1960726158 >> v1:172.21.15.92:6789/0 conn(0x7f2dec0aca20 legacy=0x7f2dec1417c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:06.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.679+0000 7f2deaffd700 1 -- 172.21.15.60:0/1960726158 >> v1:172.21.15.60:6789/0 conn(0x7f2dec0b0e80 legacy=0x7f2dec145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:06.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.679+0000 7f2deaffd700 1 -- 172.21.15.60:0/1960726158 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dec05d450 con 0x7f2dec05d070 2022-06-29T17:09:06.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.679+0000 7f2deaffd700 1 --2- 172.21.15.60:0/1960726158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dec05d070 0x7f2dec067030 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 rx=0x7f2ddc002f70 tx=0x7f2ddc006df0).ready entity=mon.0 client_cookie=44a87dcc11150dfe server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:06.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.679+0000 7f2de9ffb700 1 -- 172.21.15.60:0/1960726158 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ddc00d610 con 0x7f2dec05d070 2022-06-29T17:09:06.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.679+0000 7f2de9ffb700 1 -- 172.21.15.60:0/1960726158 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ddc000e90 con 0x7f2dec05d070 2022-06-29T17:09:06.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.679+0000 7f2de9ffb700 1 -- 172.21.15.60:0/1960726158 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ddc00d960 con 0x7f2dec05d070 2022-06-29T17:09:06.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.679+0000 7f2debfff700 1 -- 172.21.15.60:0/1960726158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dec05d070 msgr2=0x7f2dec067030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:06.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.679+0000 7f2debfff700 1 --2- 172.21.15.60:0/1960726158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dec05d070 0x7f2dec067030 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 rx=0x7f2ddc002f70 tx=0x7f2ddc006df0).stop 2022-06-29T17:09:06.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.679+0000 7f2debfff700 1 -- 172.21.15.60:0/1960726158 shutdown_connections 2022-06-29T17:09:06.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.679+0000 7f2debfff700 1 --2- 172.21.15.60:0/1960726158 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dec05d070 0x7f2dec067030 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:06.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.679+0000 7f2debfff700 1 -- 172.21.15.60:0/1960726158 >> 172.21.15.60:0/1960726158 conn(0x7f2dec0b5840 msgr2=0x7f2dec0b1810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:06.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.679+0000 7f2debfff700 1 -- 172.21.15.60:0/1960726158 shutdown_connections 2022-06-29T17:09:06.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.679+0000 7f2debfff700 1 -- 172.21.15.60:0/1960726158 wait complete. 2022-06-29T17:09:06.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.683+0000 7f2debfff700 1 Processor -- start 2022-06-29T17:09:06.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.683+0000 7f2debfff700 1 -- start start 2022-06-29T17:09:06.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.683+0000 7f2debfff700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2dec05d070 0x7f2dec112b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:06.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.683+0000 7f2debfff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2dec0aca20 0x7f2dec10eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:06.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.683+0000 7f2debfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dec0b0e80 0x7f2dec10f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:06.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.683+0000 7f2debfff700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dec13f0e0 con 0x7f2dec0b0e80 2022-06-29T17:09:06.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.683+0000 7f2debfff700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dec111260 con 0x7f2dec0aca20 2022-06-29T17:09:06.713 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.683+0000 7f2debfff700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dec115720 con 0x7f2dec05d070 2022-06-29T17:09:06.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.683+0000 7f2deb7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dec0b0e80 0x7f2dec10f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:06.714 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.683+0000 7f2deb7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dec0b0e80 0x7f2dec10f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55000/0 (socket says 172.21.15.60:55000) 2022-06-29T17:09:06.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.683+0000 7f2deb7fe700 1 -- 172.21.15.60:0/3808229648 learned_addr learned my addr 172.21.15.60:0/3808229648 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T17:09:06.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.683+0000 7f2deaffd700 1 --2- 172.21.15.60:0/3808229648 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2dec05d070 0x7f2dec112b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:06.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.683+0000 7f2dea7fc700 1 --2- 172.21.15.60:0/3808229648 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2dec0aca20 0x7f2dec10eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:06.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.683+0000 7f2deb7fe700 1 -- 172.21.15.60:0/3808229648 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2dec05d070 msgr2=0x7f2dec112b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:06.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.683+0000 7f2deb7fe700 1 --2- 172.21.15.60:0/3808229648 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f2dec05d070 0x7f2dec112b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:06.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.683+0000 7f2deb7fe700 1 -- 172.21.15.60:0/3808229648 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2dec0aca20 msgr2=0x7f2dec10eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:06.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.683+0000 7f2deb7fe700 1 --2- 172.21.15.60:0/3808229648 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2dec0aca20 0x7f2dec10eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:06.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.683+0000 7f2deb7fe700 1 -- 172.21.15.60:0/3808229648 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ddc00b180 con 0x7f2dec0b0e80 2022-06-29T17:09:06.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.683+0000 7f2deb7fe700 1 --2- 172.21.15.60:0/3808229648 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dec0b0e80 0x7f2dec10f3e0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 rx=0x7f2dec0ad1b0 tx=0x7f2de4007f10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:06.721 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.683+0000 7f2de3fff700 1 -- 172.21.15.60:0/3808229648 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2de400f470 con 0x7f2dec0b0e80 2022-06-29T17:09:06.721 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.683+0000 7f2de3fff700 1 -- 172.21.15.60:0/3808229648 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2de400f5d0 con 0x7f2dec0b0e80 2022-06-29T17:09:06.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.683+0000 7f2debfff700 1 -- 172.21.15.60:0/3808229648 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dec1159c0 con 0x7f2dec0b0e80 2022-06-29T17:09:06.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.683+0000 7f2de3fff700 1 -- 172.21.15.60:0/3808229648 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2de400f930 con 0x7f2dec0b0e80 2022-06-29T17:09:06.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.683+0000 7f2debfff700 1 -- 172.21.15.60:0/3808229648 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dec10f8d0 con 0x7f2dec0b0e80 2022-06-29T17:09:06.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.687+0000 7f2debfff700 1 -- 172.21.15.60:0/3808229648 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2dec1113a0 con 0x7f2dec0b0e80 2022-06-29T17:09:06.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.687+0000 7f2de3fff700 1 -- 172.21.15.60:0/3808229648 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f2de4019030 con 0x7f2dec0b0e80 2022-06-29T17:09:06.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.687+0000 7f2de3fff700 1 --2- 172.21.15.60:0/3808229648 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2dd0053a60 0x7f2dd0055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:06.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.687+0000 7f2de3fff700 1 -- 172.21.15.60:0/3808229648 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f2de406b4b0 con 0x7f2dec0b0e80 2022-06-29T17:09:06.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.687+0000 7f2de3fff700 1 --2- 172.21.15.60:0/3808229648 >> [v2:172.21.15.92:6808/13249,v1:172.21.15.92:6810/13249] conn(0x7f2dd005a150 0x7f2dd005c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:06.727 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.687+0000 7f2de3fff700 1 -- 172.21.15.60:0/3808229648 --> [v2:172.21.15.92:6808/13249,v1:172.21.15.92:6810/13249] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2dd005cc30 con 0x7f2dd005a150 2022-06-29T17:09:06.728 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.687+0000 7f2de3fff700 1 -- 172.21.15.60:0/3808229648 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=150) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2de406d9b0 con 0x7f2dec0b0e80 2022-06-29T17:09:06.728 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.687+0000 7f2deaffd700 1 --2- 172.21.15.60:0/3808229648 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2dd0053a60 0x7f2dd0055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:06.729 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.687+0000 7f2dea7fc700 1 --2- 172.21.15.60:0/3808229648 >> [v2:172.21.15.92:6808/13249,v1:172.21.15.92:6810/13249] conn(0x7f2dd005a150 0x7f2dd005c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:06.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.687+0000 7f2deaffd700 1 --2- 172.21.15.60:0/3808229648 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2dd0053a60 0x7f2dd0055f20 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0x7f2ddc01a000 tx=0x7f2ddc014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:06.730 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.687+0000 7f2dea7fc700 1 --2- 172.21.15.60:0/3808229648 >> [v2:172.21.15.92:6808/13249,v1:172.21.15.92:6810/13249] conn(0x7f2dd005a150 0x7f2dd005c5b0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:06.732 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.691+0000 7f2de3fff700 1 -- 172.21.15.60:0/3808229648 <== osd.4 v2:172.21.15.92:6808/13249 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f2dd005cc30 con 0x7f2dd005a150 2022-06-29T17:09:06.733 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.707+0000 7f2debfff700 1 -- 172.21.15.60:0/3808229648 --> [v2:172.21.15.92:6808/13249,v1:172.21.15.92:6810/13249] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f2dec111e30 con 0x7f2dd005a150 2022-06-29T17:09:06.734 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.707+0000 7f2de3fff700 1 -- 172.21.15.60:0/3808229648 <== osd.4 v2:172.21.15.92:6808/13249 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f2dec111e30 con 0x7f2dd005a150 2022-06-29T17:09:06.734 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T17:09:06.735 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-06-29T17:09:06.736 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T17:09:06.739 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.707+0000 7f2debfff700 1 -- 172.21.15.60:0/3808229648 >> [v2:172.21.15.92:6808/13249,v1:172.21.15.92:6810/13249] conn(0x7f2dd005a150 msgr2=0x7f2dd005c5b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:06.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.707+0000 7f2debfff700 1 --2- 172.21.15.60:0/3808229648 >> [v2:172.21.15.92:6808/13249,v1:172.21.15.92:6810/13249] conn(0x7f2dd005a150 0x7f2dd005c5b0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:06.740 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.707+0000 7f2debfff700 1 -- 172.21.15.60:0/3808229648 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2dd0053a60 msgr2=0x7f2dd0055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:06.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.707+0000 7f2debfff700 1 --2- 172.21.15.60:0/3808229648 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f2dd0053a60 0x7f2dd0055f20 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0x7f2ddc01a000 tx=0x7f2ddc014000).stop 2022-06-29T17:09:06.741 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.707+0000 7f2debfff700 1 -- 172.21.15.60:0/3808229648 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dec0b0e80 msgr2=0x7f2dec10f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:06.742 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.707+0000 7f2debfff700 1 --2- 172.21.15.60:0/3808229648 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dec0b0e80 0x7f2dec10f3e0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 rx=0x7f2dec0ad1b0 tx=0x7f2de4007f10).stop 2022-06-29T17:09:06.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.707+0000 7f2deb7fe700 1 -- 172.21.15.60:0/3808229648 reap_dead start 2022-06-29T17:09:06.743 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.707+0000 7f2debfff700 1 -- 172.21.15.60:0/3808229648 shutdown_connections 2022-06-29T17:09:06.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.707+0000 7f2debfff700 1 -- 172.21.15.60:0/3808229648 >> 172.21.15.60:0/3808229648 conn(0x7f2dec0b5840 msgr2=0x7f2dec145e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:06.744 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.707+0000 7f2debfff700 1 -- 172.21.15.60:0/3808229648 shutdown_connections 2022-06-29T17:09:06.745 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.707+0000 7f2debfff700 1 -- 172.21.15.60:0/3808229648 wait complete. 2022-06-29T17:09:06.746 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2022-06-29T17:09:06.857 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.851+0000 7f615c9a2700 1 Processor -- start 2022-06-29T17:09:06.858 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.851+0000 7f615c9a2700 1 -- start start 2022-06-29T17:09:06.863 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.851+0000 7f615c9a2700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6158065450 0x7f61580b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:06.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.851+0000 7f615c9a2700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f6158060760 0x7f6158140180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:06.864 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.851+0000 7f615c9a2700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6158140670 con 0x7f61580b28e0 2022-06-29T17:09:06.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.851+0000 7f615c9a2700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f615813fa40 con 0x7f6158060760 2022-06-29T17:09:06.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.851+0000 7f615c9a2700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f615813fce0 con 0x7f6158065450 2022-06-29T17:09:06.865 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.851+0000 7f61577fe700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f61580b28e0 0x7f6158139340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34860/0 (socket says 172.21.15.60:34860) 2022-06-29T17:09:06.866 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.851+0000 7f61577fe700 1 -- 172.21.15.60:0/2653520577 learned_addr learned my addr 172.21.15.60:0/2653520577 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T17:09:06.866 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.851+0000 7f6157fff700 1 --2- 172.21.15.60:0/2653520577 >> v2:172.21.15.92:3300/0 conn(0x7f6158060760 0x7f6158140180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:06.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.851+0000 7f6156ffd700 1 --2- 172.21.15.60:0/2653520577 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6158065450 0x7f61580b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:06.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.851+0000 7f61567fc700 1 -- 172.21.15.60:0/2653520577 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2646537884 0 0) 0x7f6158140670 con 0x7f61580b28e0 2022-06-29T17:09:06.867 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.851+0000 7f61567fc700 1 -- 172.21.15.60:0/2653520577 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6138003260 con 0x7f61580b28e0 2022-06-29T17:09:06.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f61567fc700 1 -- 172.21.15.60:0/2653520577 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3374340990 0 0) 0x7f6138003260 con 0x7f61580b28e0 2022-06-29T17:09:06.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f61567fc700 1 -- 172.21.15.60:0/2653520577 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6158065450 msgr2=0x7f61580b2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T17:09:06.868 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f61567fc700 1 --2- 172.21.15.60:0/2653520577 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6158065450 0x7f61580b2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:06.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f61567fc700 1 -- 172.21.15.60:0/2653520577 >> v2:172.21.15.92:3300/0 conn(0x7f6158060760 msgr2=0x7f6158140180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T17:09:06.869 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f61567fc700 1 --2- 172.21.15.60:0/2653520577 >> v2:172.21.15.92:3300/0 conn(0x7f6158060760 0x7f6158140180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:06.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f61567fc700 1 -- 172.21.15.60:0/2653520577 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6158140860 con 0x7f61580b28e0 2022-06-29T17:09:06.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f61567fc700 1 -- 172.21.15.60:0/2653520577 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 2013364139 0 0) 0x7f61500019e0 con 0x7f61580b28e0 2022-06-29T17:09:06.870 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f61567fc700 1 -- 172.21.15.60:0/2653520577 >> v1:172.21.15.60:6789/0 conn(0x7f61580b28e0 legacy=0x7f6158139340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:06.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f61567fc700 1 --2- 172.21.15.60:0/2653520577 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6158065450 0x7f6138003750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:06.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f61567fc700 1 --2- 172.21.15.60:0/2653520577 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6138003c80 0x7f6138004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:06.871 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f61567fc700 1 --2- 172.21.15.60:0/2653520577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61380049f0 0x7f6138004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:06.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f61567fc700 1 -- 172.21.15.60:0/2653520577 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f615813fa40 con 0x7f61380049f0 2022-06-29T17:09:06.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f61567fc700 1 -- 172.21.15.60:0/2653520577 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f615813fce0 con 0x7f6138003c80 2022-06-29T17:09:06.872 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f61567fc700 1 -- 172.21.15.60:0/2653520577 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f613800b750 con 0x7f6158065450 2022-06-29T17:09:06.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f6157fff700 1 --2- 172.21.15.60:0/2653520577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61380049f0 0x7f6138004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:06.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f6156ffd700 1 --2- 172.21.15.60:0/2653520577 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6138003c80 0x7f6138004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:06.873 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f61577fe700 1 --2- 172.21.15.60:0/2653520577 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6158065450 0x7f6138003750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:06.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f6157fff700 1 -- 172.21.15.60:0/2653520577 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6158065450 msgr2=0x7f6138003750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T17:09:06.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f6157fff700 1 --2- 172.21.15.60:0/2653520577 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f6158065450 0x7f6138003750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:06.874 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f6157fff700 1 -- 172.21.15.60:0/2653520577 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6138003c80 msgr2=0x7f6138004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T17:09:06.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f6157fff700 1 --2- 172.21.15.60:0/2653520577 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6138003c80 0x7f6138004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:06.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f6157fff700 1 -- 172.21.15.60:0/2653520577 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6138003260 con 0x7f61380049f0 2022-06-29T17:09:06.875 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f6156ffd700 1 --2- 172.21.15.60:0/2653520577 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6138003c80 0x7f6138004500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-06-29T17:09:06.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f6157fff700 1 --2- 172.21.15.60:0/2653520577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61380049f0 0x7f6138004e70 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 rx=0x7f61380054d0 tx=0x7f6148006df0).ready entity=mon.0 client_cookie=83b68c4e7c930cd7 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:06.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f61567fc700 1 -- 172.21.15.60:0/2653520577 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f614800e830 con 0x7f61380049f0 2022-06-29T17:09:06.876 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f61567fc700 1 -- 172.21.15.60:0/2653520577 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f614800be90 con 0x7f61380049f0 2022-06-29T17:09:06.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f61567fc700 1 -- 172.21.15.60:0/2653520577 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f614800eb80 con 0x7f61380049f0 2022-06-29T17:09:06.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f615c9a2700 1 -- 172.21.15.60:0/2653520577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61380049f0 msgr2=0x7f6138004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:06.877 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f615c9a2700 1 --2- 172.21.15.60:0/2653520577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f61380049f0 0x7f6138004e70 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 rx=0x7f61380054d0 tx=0x7f6148006df0).stop 2022-06-29T17:09:06.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f6157fff700 1 -- 172.21.15.60:0/2653520577 reap_dead start 2022-06-29T17:09:06.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f615c9a2700 1 -- 172.21.15.60:0/2653520577 shutdown_connections 2022-06-29T17:09:06.878 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f615c9a2700 1 -- 172.21.15.60:0/2653520577 >> 172.21.15.60:0/2653520577 conn(0x7f6158061c80 msgr2=0x7f61580637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:06.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f615c9a2700 1 -- 172.21.15.60:0/2653520577 shutdown_connections 2022-06-29T17:09:06.879 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.855+0000 7f615c9a2700 1 -- 172.21.15.60:0/2653520577 wait complete. 2022-06-29T17:09:06.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.859+0000 7f615c9a2700 1 Processor -- start 2022-06-29T17:09:06.883 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.859+0000 7f615c9a2700 1 -- start start 2022-06-29T17:09:06.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.859+0000 7f615c9a2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f615805c8f0 0x7f615813d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:06.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.859+0000 7f615c9a2700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f615810f420 0x7f615810f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:06.884 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.859+0000 7f615c9a2700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f615813c6c0 0x7f615813cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:06.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.859+0000 7f615c9a2700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f615813ebf0 con 0x7f615805c8f0 2022-06-29T17:09:06.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.859+0000 7f615c9a2700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f615805ccd0 con 0x7f615810f420 2022-06-29T17:09:06.885 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.859+0000 7f615c9a2700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61580adf30 con 0x7f615813c6c0 2022-06-29T17:09:06.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.859+0000 7f61577fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f615805c8f0 0x7f615813d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:06.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.859+0000 7f61577fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f615805c8f0 0x7f615813d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55018/0 (socket says 172.21.15.60:55018) 2022-06-29T17:09:06.886 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.859+0000 7f61577fe700 1 -- 172.21.15.60:0/2102777891 learned_addr learned my addr 172.21.15.60:0/2102777891 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T17:09:06.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.859+0000 7f6157fff700 1 --2- 172.21.15.60:0/2102777891 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f615813c6c0 0x7f615813cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:06.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.859+0000 7f6156ffd700 1 --2- 172.21.15.60:0/2102777891 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f615810f420 0x7f615810f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:06.887 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.859+0000 7f61577fe700 1 -- 172.21.15.60:0/2102777891 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f615813c6c0 msgr2=0x7f615813cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:06.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.859+0000 7f61577fe700 1 --2- 172.21.15.60:0/2102777891 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f615813c6c0 0x7f615813cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:06.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.859+0000 7f61577fe700 1 -- 172.21.15.60:0/2102777891 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f615810f420 msgr2=0x7f615810f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:06.888 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.859+0000 7f61577fe700 1 --2- 172.21.15.60:0/2102777891 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f615810f420 0x7f615810f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:06.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.859+0000 7f61577fe700 1 -- 172.21.15.60:0/2102777891 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f615810fd90 con 0x7f615805c8f0 2022-06-29T17:09:06.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.859+0000 7f61577fe700 1 --2- 172.21.15.60:0/2102777891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f615805c8f0 0x7f615813d450 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 rx=0x7f615000b000 tx=0x7f61580b3980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:06.889 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.859+0000 7f6154ff9700 1 -- 172.21.15.60:0/2102777891 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f615000a780 con 0x7f615805c8f0 2022-06-29T17:09:06.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.859+0000 7f6154ff9700 1 -- 172.21.15.60:0/2102777891 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6150010040 con 0x7f615805c8f0 2022-06-29T17:09:06.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.859+0000 7f615c9a2700 1 -- 172.21.15.60:0/2102777891 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f615813edf0 con 0x7f615805c8f0 2022-06-29T17:09:06.890 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.859+0000 7f6154ff9700 1 -- 172.21.15.60:0/2102777891 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f615000aad0 con 0x7f615805c8f0 2022-06-29T17:09:06.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.859+0000 7f615c9a2700 1 -- 172.21.15.60:0/2102777891 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61580aeeb0 con 0x7f615805c8f0 2022-06-29T17:09:06.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.863+0000 7f6154ff9700 1 -- 172.21.15.60:0/2102777891 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f615001e430 con 0x7f615805c8f0 2022-06-29T17:09:06.891 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.863+0000 7f615c9a2700 1 -- 172.21.15.60:0/2102777891 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f615813f0d0 con 0x7f615805c8f0 2022-06-29T17:09:06.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.863+0000 7f6154ff9700 1 --2- 172.21.15.60:0/2102777891 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f613c053b30 0x7f613c055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:06.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.863+0000 7f6154ff9700 1 -- 172.21.15.60:0/2102777891 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f615006bec0 con 0x7f615805c8f0 2022-06-29T17:09:06.892 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.863+0000 7f6156ffd700 1 --2- 172.21.15.60:0/2102777891 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f613c053b30 0x7f613c055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:06.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.863+0000 7f6156ffd700 1 --2- 172.21.15.60:0/2102777891 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f613c053b30 0x7f613c055ff0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0x7f614000b000 tx=0x7f6140008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:06.893 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.867+0000 7f6154ff9700 1 -- 172.21.15.60:0/2102777891 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6150043060 con 0x7f615805c8f0 2022-06-29T17:09:07.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.995+0000 7f615c9a2700 1 -- 172.21.15.60:0/2102777891 --> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f61580addb0 con 0x7f613c053b30 2022-06-29T17:09:07.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.995+0000 7f6154ff9700 1 -- 172.21.15.60:0/2102777891 <== mgr.4103 v2:172.21.15.92:6832/12872 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6140000e90 con 0x7f613c053b30 2022-06-29T17:09:07.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.995+0000 7f615c9a2700 1 -- 172.21.15.60:0/2102777891 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f613c053b30 msgr2=0x7f613c055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:07.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.995+0000 7f615c9a2700 1 --2- 172.21.15.60:0/2102777891 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f613c053b30 0x7f613c055ff0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0x7f614000b000 tx=0x7f6140008040).stop 2022-06-29T17:09:07.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.995+0000 7f615c9a2700 1 -- 172.21.15.60:0/2102777891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f615805c8f0 msgr2=0x7f615813d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:07.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.995+0000 7f615c9a2700 1 --2- 172.21.15.60:0/2102777891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f615805c8f0 0x7f615813d450 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 rx=0x7f615000b000 tx=0x7f61580b3980).stop 2022-06-29T17:09:07.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.999+0000 7f615c9a2700 1 -- 172.21.15.60:0/2102777891 shutdown_connections 2022-06-29T17:09:07.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.999+0000 7f615c9a2700 1 --2- 172.21.15.60:0/2102777891 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f613c053b30 0x7f613c055ff0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.999+0000 7f615c9a2700 1 --2- 172.21.15.60:0/2102777891 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f615813c6c0 0x7f615813cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.999+0000 7f615c9a2700 1 --2- 172.21.15.60:0/2102777891 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f615810f420 0x7f615810f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.999+0000 7f615c9a2700 1 --2- 172.21.15.60:0/2102777891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f615805c8f0 0x7f615813d450 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.999+0000 7f615c9a2700 1 -- 172.21.15.60:0/2102777891 >> 172.21.15.60:0/2102777891 conn(0x7f6158061c80 msgr2=0x7f6158140c00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:07.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.999+0000 7f615c9a2700 1 -- 172.21.15.60:0/2102777891 shutdown_connections 2022-06-29T17:09:07.009 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:06.999+0000 7f615c9a2700 1 -- 172.21.15.60:0/2102777891 wait complete. 2022-06-29T17:09:07.010 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 4 to deep-scrub 2022-06-29T17:09:07.014 INFO:tasks.ceph:Scrubbing osd.5 2022-06-29T17:09:07.015 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2022-06-29T17:09:07.126 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.119+0000 7f832b1cd700 1 Processor -- start 2022-06-29T17:09:07.127 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.119+0000 7f832b1cd700 1 -- start start 2022-06-29T17:09:07.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.123+0000 7f832b1cd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f83240b9be0 0x7f832405bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:07.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.123+0000 7f832b1cd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83240b28e0 0x7f8324141e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:07.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.123+0000 7f832b1cd700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83240b1d70 con 0x7f83240afee0 2022-06-29T17:09:07.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.123+0000 7f832b1cd700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83240b9fc0 con 0x7f83240b9be0 2022-06-29T17:09:07.130 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.123+0000 7f832b1cd700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83240b3c40 con 0x7f83240b28e0 2022-06-29T17:09:07.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.123+0000 7f832a1cb700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f83240b9be0 0x7f832405bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:07.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.123+0000 7f832a1cb700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f83240b9be0 0x7f832405bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55028/0 (socket says 172.21.15.60:55028) 2022-06-29T17:09:07.131 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.123+0000 7f832a1cb700 1 -- 172.21.15.60:0/2867174206 learned_addr learned my addr 172.21.15.60:0/2867174206 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T17:09:07.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.123+0000 7f832a9cc700 1 --2- 172.21.15.60:0/2867174206 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83240b28e0 0x7f8324141e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:07.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.123+0000 7f832a1cb700 1 -- 172.21.15.60:0/2867174206 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83240b28e0 msgr2=0x7f8324141e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T17:09:07.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.123+0000 7f832a1cb700 1 --2- 172.21.15.60:0/2867174206 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83240b28e0 0x7f8324141e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.123+0000 7f83291c9700 1 -- 172.21.15.60:0/2867174206 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3772583012 0 0) 0x7f83240b1d70 con 0x7f83240afee0 2022-06-29T17:09:07.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.123+0000 7f832a1cb700 1 -- 172.21.15.60:0/2867174206 >> v1:172.21.15.92:6789/0 conn(0x7f83240afee0 legacy=0x7f83240b21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:07.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.123+0000 7f832a1cb700 1 -- 172.21.15.60:0/2867174206 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83240b3ee0 con 0x7f83240b9be0 2022-06-29T17:09:07.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.123+0000 7f832a1cb700 1 --2- 172.21.15.60:0/2867174206 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83240b9be0 0x7f832405bd20 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 rx=0x7f8318002f70 tx=0x7f8318006eb0).ready entity=mon.0 client_cookie=34035053bb10ecef server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:07.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.123+0000 7f83291c9700 1 -- 172.21.15.60:0/2867174206 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f831800d5c0 con 0x7f83240b9be0 2022-06-29T17:09:07.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.123+0000 7f83291c9700 1 -- 172.21.15.60:0/2867174206 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8318000e90 con 0x7f83240b9be0 2022-06-29T17:09:07.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.123+0000 7f83291c9700 1 -- 172.21.15.60:0/2867174206 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f831800d910 con 0x7f83240b9be0 2022-06-29T17:09:07.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.123+0000 7f832b1cd700 1 -- 172.21.15.60:0/2867174206 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83240b9be0 msgr2=0x7f832405bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:07.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.123+0000 7f832b1cd700 1 --2- 172.21.15.60:0/2867174206 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83240b9be0 0x7f832405bd20 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 rx=0x7f8318002f70 tx=0x7f8318006eb0).stop 2022-06-29T17:09:07.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.123+0000 7f832b1cd700 1 -- 172.21.15.60:0/2867174206 shutdown_connections 2022-06-29T17:09:07.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.123+0000 7f832b1cd700 1 --2- 172.21.15.60:0/2867174206 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83240b28e0 0x7f8324141e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.123+0000 7f832b1cd700 1 --2- 172.21.15.60:0/2867174206 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83240b9be0 0x7f832405bd20 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.123+0000 7f832b1cd700 1 -- 172.21.15.60:0/2867174206 >> 172.21.15.60:0/2867174206 conn(0x7f8324062ce0 msgr2=0x7f83240b9190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:07.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.123+0000 7f832b1cd700 1 -- 172.21.15.60:0/2867174206 shutdown_connections 2022-06-29T17:09:07.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.123+0000 7f832b1cd700 1 -- 172.21.15.60:0/2867174206 wait complete. 2022-06-29T17:09:07.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.127+0000 7f832b1cd700 1 Processor -- start 2022-06-29T17:09:07.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.127+0000 7f832b1cd700 1 -- start start 2022-06-29T17:09:07.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.127+0000 7f832b1cd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83240afee0 0x7f8324144240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:07.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.127+0000 7f832b1cd700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83240b28e0 0x7f8324112900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:07.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.127+0000 7f832b1cd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83240b9be0 0x7f8324112df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:07.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.127+0000 7f832b1cd700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8324148110 con 0x7f83240b9be0 2022-06-29T17:09:07.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.127+0000 7f832b1cd700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8324144ed0 con 0x7f83240afee0 2022-06-29T17:09:07.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.127+0000 7f832b1cd700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8324146f20 con 0x7f83240b28e0 2022-06-29T17:09:07.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.127+0000 7f832a9cc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83240b9be0 0x7f8324112df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:07.144 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.127+0000 7f832a1cb700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83240afee0 0x7f8324144240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:07.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.127+0000 7f83299ca700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83240b28e0 0x7f8324112900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:07.145 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.127+0000 7f832a9cc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83240b9be0 0x7f8324112df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55036/0 (socket says 172.21.15.60:55036) 2022-06-29T17:09:07.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.127+0000 7f832a9cc700 1 -- 172.21.15.60:0/339013260 learned_addr learned my addr 172.21.15.60:0/339013260 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T17:09:07.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.131+0000 7f832a9cc700 1 -- 172.21.15.60:0/339013260 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83240b28e0 msgr2=0x7f8324112900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:07.146 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.131+0000 7f832a9cc700 1 --2- 172.21.15.60:0/339013260 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83240b28e0 0x7f8324112900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.131+0000 7f832a9cc700 1 -- 172.21.15.60:0/339013260 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83240afee0 msgr2=0x7f8324144240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:07.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.131+0000 7f832a9cc700 1 --2- 172.21.15.60:0/339013260 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83240afee0 0x7f8324144240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.131+0000 7f832a9cc700 1 -- 172.21.15.60:0/339013260 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f831800b1e0 con 0x7f83240b9be0 2022-06-29T17:09:07.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.131+0000 7f83299ca700 1 --2- 172.21.15.60:0/339013260 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f83240b28e0 0x7f8324112900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T17:09:07.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.131+0000 7f832a9cc700 1 --2- 172.21.15.60:0/339013260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83240b9be0 0x7f8324112df0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 rx=0x7f831c00c000 tx=0x7f831c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:07.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.131+0000 7f83177fe700 1 -- 172.21.15.60:0/339013260 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f831c012600 con 0x7f83240b9be0 2022-06-29T17:09:07.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.131+0000 7f83177fe700 1 -- 172.21.15.60:0/339013260 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f831c012760 con 0x7f83240b9be0 2022-06-29T17:09:07.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.131+0000 7f832b1cd700 1 -- 172.21.15.60:0/339013260 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83241471c0 con 0x7f83240b9be0 2022-06-29T17:09:07.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.131+0000 7f83177fe700 1 -- 172.21.15.60:0/339013260 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f831c012ac0 con 0x7f83240b9be0 2022-06-29T17:09:07.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.131+0000 7f832b1cd700 1 -- 172.21.15.60:0/339013260 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8324147d90 con 0x7f83240b9be0 2022-06-29T17:09:07.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.131+0000 7f832b1cd700 1 -- 172.21.15.60:0/339013260 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8324147350 con 0x7f83240b9be0 2022-06-29T17:09:07.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.131+0000 7f83177fe700 1 -- 172.21.15.60:0/339013260 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f831c022070 con 0x7f83240b9be0 2022-06-29T17:09:07.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.135+0000 7f83177fe700 1 --2- 172.21.15.60:0/339013260 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8304053a70 0x7f8304055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:07.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.135+0000 7f83177fe700 1 -- 172.21.15.60:0/339013260 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f831c06c200 con 0x7f83240b9be0 2022-06-29T17:09:07.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.135+0000 7f83177fe700 1 --2- 172.21.15.60:0/339013260 >> [v2:172.21.15.92:6809/13248,v1:172.21.15.92:6811/13248] conn(0x7f830405a160 0x7f830405c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:07.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.135+0000 7f83177fe700 1 -- 172.21.15.60:0/339013260 --> [v2:172.21.15.92:6809/13248,v1:172.21.15.92:6811/13248] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f830405cba0 con 0x7f830405a160 2022-06-29T17:09:07.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.135+0000 7f83177fe700 1 -- 172.21.15.60:0/339013260 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=150) v2 ==== 24+0+0 (secure 0 0 0) 0x7f831c06e890 con 0x7f83240b9be0 2022-06-29T17:09:07.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.135+0000 7f832a1cb700 1 --2- 172.21.15.60:0/339013260 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8304053a70 0x7f8304055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:07.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.135+0000 7f83299ca700 1 --2- 172.21.15.60:0/339013260 >> [v2:172.21.15.92:6809/13248,v1:172.21.15.92:6811/13248] conn(0x7f830405a160 0x7f830405c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:07.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.135+0000 7f832a1cb700 1 --2- 172.21.15.60:0/339013260 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8304053a70 0x7f8304055f30 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7f831801a000 tx=0x7f8318014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:07.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.135+0000 7f83299ca700 1 --2- 172.21.15.60:0/339013260 >> [v2:172.21.15.92:6809/13248,v1:172.21.15.92:6811/13248] conn(0x7f830405a160 0x7f830405c5c0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:07.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.135+0000 7f83177fe700 1 -- 172.21.15.60:0/339013260 <== osd.5 v2:172.21.15.92:6809/13248 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f830405cba0 con 0x7f830405a160 2022-06-29T17:09:07.159 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.155+0000 7f832b1cd700 1 -- 172.21.15.60:0/339013260 --> [v2:172.21.15.92:6809/13248,v1:172.21.15.92:6811/13248] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f83240573d0 con 0x7f830405a160 2022-06-29T17:09:07.160 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.155+0000 7f83177fe700 1 -- 172.21.15.60:0/339013260 <== osd.5 v2:172.21.15.92:6809/13248 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f83240573d0 con 0x7f830405a160 2022-06-29T17:09:07.160 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T17:09:07.161 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-06-29T17:09:07.161 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T17:09:07.162 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.155+0000 7f832b1cd700 1 -- 172.21.15.60:0/339013260 >> [v2:172.21.15.92:6809/13248,v1:172.21.15.92:6811/13248] conn(0x7f830405a160 msgr2=0x7f830405c5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:07.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.155+0000 7f832b1cd700 1 --2- 172.21.15.60:0/339013260 >> [v2:172.21.15.92:6809/13248,v1:172.21.15.92:6811/13248] conn(0x7f830405a160 0x7f830405c5c0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.155+0000 7f832b1cd700 1 -- 172.21.15.60:0/339013260 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8304053a70 msgr2=0x7f8304055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:07.163 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.155+0000 7f832b1cd700 1 --2- 172.21.15.60:0/339013260 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f8304053a70 0x7f8304055f30 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7f831801a000 tx=0x7f8318014000).stop 2022-06-29T17:09:07.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.155+0000 7f832b1cd700 1 -- 172.21.15.60:0/339013260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83240b9be0 msgr2=0x7f8324112df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:07.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.155+0000 7f832b1cd700 1 --2- 172.21.15.60:0/339013260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83240b9be0 0x7f8324112df0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 rx=0x7f831c00c000 tx=0x7f831c0091d0).stop 2022-06-29T17:09:07.164 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.155+0000 7f832a9cc700 1 -- 172.21.15.60:0/339013260 reap_dead start 2022-06-29T17:09:07.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.155+0000 7f832b1cd700 1 -- 172.21.15.60:0/339013260 shutdown_connections 2022-06-29T17:09:07.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.155+0000 7f832b1cd700 1 -- 172.21.15.60:0/339013260 >> 172.21.15.60:0/339013260 conn(0x7f8324062ce0 msgr2=0x7f8324142310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:07.165 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.155+0000 7f832b1cd700 1 -- 172.21.15.60:0/339013260 shutdown_connections 2022-06-29T17:09:07.166 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.155+0000 7f832b1cd700 1 -- 172.21.15.60:0/339013260 wait complete. 2022-06-29T17:09:07.172 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2022-06-29T17:09:07.284 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.279+0000 7f044563e700 1 Processor -- start 2022-06-29T17:09:07.285 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.279+0000 7f044563e700 1 -- start start 2022-06-29T17:09:07.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.279+0000 7f044563e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f04400b28e0 0x7f04400b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:07.288 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.279+0000 7f044563e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0440065450 0x7f0440140180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:07.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.279+0000 7f044563e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0440060760 0x7f0440139340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:07.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.279+0000 7f044563e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0440140670 con 0x7f0440060760 2022-06-29T17:09:07.289 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.279+0000 7f044563e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f044013fa40 con 0x7f04400b28e0 2022-06-29T17:09:07.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.279+0000 7f044563e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f044013fce0 con 0x7f0440065450 2022-06-29T17:09:07.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.279+0000 7f0444e3d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0440060760 0x7f0440139340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:07.290 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.279+0000 7f0444e3d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f0440060760 0x7f0440139340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55042/0 (socket says 172.21.15.60:55042) 2022-06-29T17:09:07.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.279+0000 7f0444e3d700 1 -- 172.21.15.60:0/593498956 learned_addr learned my addr 172.21.15.60:0/593498956 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T17:09:07.291 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.279+0000 7f043f7fe700 1 --2- 172.21.15.60:0/593498956 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0440065450 0x7f0440140180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:07.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.279+0000 7f043ffff700 1 --2- 172.21.15.60:0/593498956 >> v2:172.21.15.92:3300/0 conn(0x7f04400b28e0 0x7f04400b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:07.292 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.279+0000 7f0444e3d700 1 -- 172.21.15.60:0/593498956 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0440065450 msgr2=0x7f0440140180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T17:09:07.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.279+0000 7f0444e3d700 1 --2- 172.21.15.60:0/593498956 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0440065450 0x7f0440140180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.279+0000 7f0444e3d700 1 -- 172.21.15.60:0/593498956 >> v2:172.21.15.92:3300/0 conn(0x7f04400b28e0 msgr2=0x7f04400b2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T17:09:07.293 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.279+0000 7f0444e3d700 1 --2- 172.21.15.60:0/593498956 >> v2:172.21.15.92:3300/0 conn(0x7f04400b28e0 0x7f04400b2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.279+0000 7f0444e3d700 1 -- 172.21.15.60:0/593498956 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0440140860 con 0x7f0440060760 2022-06-29T17:09:07.294 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.279+0000 7f0444e3d700 1 --2- 172.21.15.60:0/593498956 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0440060760 0x7f0440139340 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 rx=0x7f0440064180 tx=0x7f0438006eb0).ready entity=mon.0 client_cookie=2296cc6daa33c6ee server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:07.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.279+0000 7f043effd700 1 -- 172.21.15.60:0/593498956 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f043800d570 con 0x7f0440060760 2022-06-29T17:09:07.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.279+0000 7f043effd700 1 -- 172.21.15.60:0/593498956 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0438000e90 con 0x7f0440060760 2022-06-29T17:09:07.295 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.279+0000 7f043effd700 1 -- 172.21.15.60:0/593498956 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f043800d8c0 con 0x7f0440060760 2022-06-29T17:09:07.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.279+0000 7f044563e700 1 -- 172.21.15.60:0/593498956 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0440060760 msgr2=0x7f0440139340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:07.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.279+0000 7f044563e700 1 --2- 172.21.15.60:0/593498956 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0440060760 0x7f0440139340 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 rx=0x7f0440064180 tx=0x7f0438006eb0).stop 2022-06-29T17:09:07.296 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.279+0000 7f044563e700 1 -- 172.21.15.60:0/593498956 shutdown_connections 2022-06-29T17:09:07.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.279+0000 7f044563e700 1 --2- 172.21.15.60:0/593498956 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0440060760 0x7f0440139340 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.279+0000 7f044563e700 1 --2- 172.21.15.60:0/593498956 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f0440065450 0x7f0440140180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.297 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.279+0000 7f044563e700 1 --2- 172.21.15.60:0/593498956 >> v2:172.21.15.92:3300/0 conn(0x7f04400b28e0 0x7f04400b2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.279+0000 7f044563e700 1 -- 172.21.15.60:0/593498956 >> 172.21.15.60:0/593498956 conn(0x7f0440061c80 msgr2=0x7f04400637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:07.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.283+0000 7f044563e700 1 -- 172.21.15.60:0/593498956 shutdown_connections 2022-06-29T17:09:07.298 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.283+0000 7f044563e700 1 -- 172.21.15.60:0/593498956 wait complete. 2022-06-29T17:09:07.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.283+0000 7f044563e700 1 Processor -- start 2022-06-29T17:09:07.299 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.283+0000 7f044563e700 1 -- start start 2022-06-29T17:09:07.303 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.283+0000 7f044563e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0440060760 0x7f04400aed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:07.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.283+0000 7f044563e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0440065450 0x7f0440143790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:07.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.283+0000 7f044563e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04400b28e0 0x7f044014f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:07.304 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.283+0000 7f044563e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04400b0290 con 0x7f0440060760 2022-06-29T17:09:07.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.283+0000 7f044563e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04400af330 con 0x7f0440065450 2022-06-29T17:09:07.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.283+0000 7f044563e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0440137460 con 0x7f04400b28e0 2022-06-29T17:09:07.305 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.283+0000 7f043ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0440060760 0x7f04400aed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:07.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.283+0000 7f043ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0440060760 0x7f04400aed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55046/0 (socket says 172.21.15.60:55046) 2022-06-29T17:09:07.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.283+0000 7f043ffff700 1 -- 172.21.15.60:0/2919687005 learned_addr learned my addr 172.21.15.60:0/2919687005 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T17:09:07.306 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.283+0000 7f0444e3d700 1 --2- 172.21.15.60:0/2919687005 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04400b28e0 0x7f044014f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:07.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.283+0000 7f043f7fe700 1 --2- 172.21.15.60:0/2919687005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0440065450 0x7f0440143790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:07.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.283+0000 7f043ffff700 1 -- 172.21.15.60:0/2919687005 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04400b28e0 msgr2=0x7f044014f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:07.307 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.283+0000 7f043ffff700 1 --2- 172.21.15.60:0/2919687005 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04400b28e0 0x7f044014f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.283+0000 7f043ffff700 1 -- 172.21.15.60:0/2919687005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0440065450 msgr2=0x7f0440143790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:07.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.283+0000 7f043ffff700 1 --2- 172.21.15.60:0/2919687005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0440065450 0x7f0440143790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.308 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.283+0000 7f043ffff700 1 -- 172.21.15.60:0/2919687005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f043800b1e0 con 0x7f0440060760 2022-06-29T17:09:07.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.283+0000 7f043ffff700 1 --2- 172.21.15.60:0/2919687005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0440060760 0x7f04400aed00 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 rx=0x7f0430003430 tx=0x7f0430008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:07.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.287+0000 7f043d7fa700 1 -- 172.21.15.60:0/2919687005 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f043000c7d0 con 0x7f0440060760 2022-06-29T17:09:07.309 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.287+0000 7f043d7fa700 1 -- 172.21.15.60:0/2919687005 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043000c930 con 0x7f0440060760 2022-06-29T17:09:07.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.287+0000 7f044563e700 1 -- 172.21.15.60:0/2919687005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0440137700 con 0x7f0440060760 2022-06-29T17:09:07.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.287+0000 7f043d7fa700 1 -- 172.21.15.60:0/2919687005 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f043000cc90 con 0x7f0440060760 2022-06-29T17:09:07.310 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.287+0000 7f044563e700 1 -- 172.21.15.60:0/2919687005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04400af620 con 0x7f0440060760 2022-06-29T17:09:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.287+0000 7f044563e700 1 -- 172.21.15.60:0/2919687005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0440137890 con 0x7f0440060760 2022-06-29T17:09:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.287+0000 7f043d7fa700 1 -- 172.21.15.60:0/2919687005 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f0430013430 con 0x7f0440060760 2022-06-29T17:09:07.311 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.287+0000 7f043d7fa700 1 --2- 172.21.15.60:0/2919687005 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f04180539e0 0x7f0418055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:07.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.287+0000 7f043d7fa700 1 -- 172.21.15.60:0/2919687005 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f043006bc80 con 0x7f0440060760 2022-06-29T17:09:07.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.287+0000 7f043f7fe700 1 --2- 172.21.15.60:0/2919687005 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f04180539e0 0x7f0418055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:07.312 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.291+0000 7f043f7fe700 1 --2- 172.21.15.60:0/2919687005 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f04180539e0 0x7f0418055ea0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0x7f042800c000 tx=0x7f0428008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:07.313 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.295+0000 7f043d7fa700 1 -- 172.21.15.60:0/2919687005 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0430071020 con 0x7f0440060760 2022-06-29T17:09:07.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.419+0000 7f044563e700 1 -- 172.21.15.60:0/2919687005 --> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f04400573d0 con 0x7f04180539e0 2022-06-29T17:09:07.424 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.419+0000 7f043d7fa700 1 -- 172.21.15.60:0/2919687005 <== mgr.4103 v2:172.21.15.92:6832/12872 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f04400573d0 con 0x7f04180539e0 2022-06-29T17:09:07.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.419+0000 7f044563e700 1 -- 172.21.15.60:0/2919687005 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f04180539e0 msgr2=0x7f0418055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:07.426 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.419+0000 7f044563e700 1 --2- 172.21.15.60:0/2919687005 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f04180539e0 0x7f0418055ea0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0x7f042800c000 tx=0x7f0428008040).stop 2022-06-29T17:09:07.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.419+0000 7f044563e700 1 -- 172.21.15.60:0/2919687005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0440060760 msgr2=0x7f04400aed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:07.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.419+0000 7f044563e700 1 --2- 172.21.15.60:0/2919687005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0440060760 0x7f04400aed00 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 rx=0x7f0430003430 tx=0x7f0430008040).stop 2022-06-29T17:09:07.427 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.419+0000 7f044563e700 1 -- 172.21.15.60:0/2919687005 shutdown_connections 2022-06-29T17:09:07.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.419+0000 7f044563e700 1 --2- 172.21.15.60:0/2919687005 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f04180539e0 0x7f0418055ea0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.428 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.419+0000 7f044563e700 1 --2- 172.21.15.60:0/2919687005 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f04400b28e0 0x7f044014f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.419+0000 7f044563e700 1 --2- 172.21.15.60:0/2919687005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0440065450 0x7f0440143790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.429 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.419+0000 7f044563e700 1 --2- 172.21.15.60:0/2919687005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0440060760 0x7f04400aed00 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.419+0000 7f044563e700 1 -- 172.21.15.60:0/2919687005 >> 172.21.15.60:0/2919687005 conn(0x7f0440061c80 msgr2=0x7f0440069a60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:07.430 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.419+0000 7f044563e700 1 -- 172.21.15.60:0/2919687005 shutdown_connections 2022-06-29T17:09:07.431 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.419+0000 7f044563e700 1 -- 172.21.15.60:0/2919687005 wait complete. 2022-06-29T17:09:07.431 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 5 to deep-scrub 2022-06-29T17:09:07.437 INFO:tasks.ceph:Scrubbing osd.6 2022-06-29T17:09:07.437 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2022-06-29T17:09:07.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.539+0000 7fd2889b8700 1 Processor -- start 2022-06-29T17:09:07.548 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.543+0000 7fd2889b8700 1 -- start start 2022-06-29T17:09:07.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.543+0000 7fd2889b8700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd28405d1f0 0x7fd2840b1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:07.552 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.543+0000 7fd2889b8700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd2840b3cc0 0x7fd2840b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:07.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.543+0000 7fd2889b8700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2840b2ce0 con 0x7fd2840b9d70 2022-06-29T17:09:07.553 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.543+0000 7fd2889b8700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2840b4d10 con 0x7fd2840b3cc0 2022-06-29T17:09:07.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.543+0000 7fd2889b8700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2840ba150 con 0x7fd28405d1f0 2022-06-29T17:09:07.554 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.543+0000 7fd282d9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd2840b3cc0 0x7fd2840b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:07.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.543+0000 7fd282d9d700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fd2840b3cc0 0x7fd2840b2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55058/0 (socket says 172.21.15.60:55058) 2022-06-29T17:09:07.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.543+0000 7fd282d9d700 1 -- 172.21.15.60:0/744749676 learned_addr learned my addr 172.21.15.60:0/744749676 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T17:09:07.555 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.543+0000 7fd281d9b700 1 --2- 172.21.15.60:0/744749676 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd28405d1f0 0x7fd2840b1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:07.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.543+0000 7fd282d9d700 1 -- 172.21.15.60:0/744749676 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd28405d1f0 msgr2=0x7fd2840b1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T17:09:07.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.543+0000 7fd282d9d700 1 --2- 172.21.15.60:0/744749676 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd28405d1f0 0x7fd2840b1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.556 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.543+0000 7fd282d9d700 1 -- 172.21.15.60:0/744749676 >> v1:172.21.15.60:6789/0 conn(0x7fd2840b9d70 legacy=0x7fd2840b3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:07.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.543+0000 7fd282d9d700 1 -- 172.21.15.60:0/744749676 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2840b4a70 con 0x7fd2840b3cc0 2022-06-29T17:09:07.557 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.543+0000 7fd28159a700 1 -- 172.21.15.60:0/744749676 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2678952870 0 0) 0x7fd2840b2ce0 con 0x7fd2840b9d70 2022-06-29T17:09:07.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.543+0000 7fd282d9d700 1 --2- 172.21.15.60:0/744749676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2840b3cc0 0x7fd2840b2040 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 rx=0x7fd27c0032f0 tx=0x7fd27c009290).ready entity=mon.0 client_cookie=bf891a2b8608e661 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:07.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.543+0000 7fd28159a700 1 -- 172.21.15.60:0/744749676 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd27c00b8a0 con 0x7fd2840b3cc0 2022-06-29T17:09:07.558 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.543+0000 7fd28159a700 1 -- 172.21.15.60:0/744749676 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd27c00ba00 con 0x7fd2840b3cc0 2022-06-29T17:09:07.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.543+0000 7fd28159a700 1 -- 172.21.15.60:0/744749676 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd27c00bb70 con 0x7fd2840b3cc0 2022-06-29T17:09:07.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.543+0000 7fd2889b8700 1 -- 172.21.15.60:0/744749676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2840b3cc0 msgr2=0x7fd2840b2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:07.559 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.543+0000 7fd2889b8700 1 --2- 172.21.15.60:0/744749676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2840b3cc0 0x7fd2840b2040 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 rx=0x7fd27c0032f0 tx=0x7fd27c009290).stop 2022-06-29T17:09:07.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.543+0000 7fd2889b8700 1 -- 172.21.15.60:0/744749676 shutdown_connections 2022-06-29T17:09:07.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.543+0000 7fd2889b8700 1 --2- 172.21.15.60:0/744749676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2840b3cc0 0x7fd2840b2040 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.560 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.543+0000 7fd2889b8700 1 --2- 172.21.15.60:0/744749676 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd28405d1f0 0x7fd2840b1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.543+0000 7fd2889b8700 1 -- 172.21.15.60:0/744749676 >> 172.21.15.60:0/744749676 conn(0x7fd284062ce0 msgr2=0x7fd2840b9320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:07.561 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.543+0000 7fd2889b8700 1 -- 172.21.15.60:0/744749676 shutdown_connections 2022-06-29T17:09:07.562 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.543+0000 7fd2889b8700 1 -- 172.21.15.60:0/744749676 wait complete. 2022-06-29T17:09:07.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.547+0000 7fd2889b8700 1 Processor -- start 2022-06-29T17:09:07.567 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.547+0000 7fd2889b8700 1 -- start start 2022-06-29T17:09:07.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.547+0000 7fd2889b8700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd28405d1f0 0x7fd2840af9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:07.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.547+0000 7fd2889b8700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2840b3cc0 0x7fd28405c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:07.568 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.547+0000 7fd2889b8700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2840b9d70 0x7fd284144ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:07.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.547+0000 7fd2889b8700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd284146da0 con 0x7fd28405d1f0 2022-06-29T17:09:07.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.547+0000 7fd2889b8700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2840ada20 con 0x7fd2840b9d70 2022-06-29T17:09:07.569 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.547+0000 7fd2889b8700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2840adc80 con 0x7fd2840b3cc0 2022-06-29T17:09:07.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.547+0000 7fd28259c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd28405d1f0 0x7fd2840af9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:07.570 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.547+0000 7fd28259c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd28405d1f0 0x7fd2840af9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55060/0 (socket says 172.21.15.60:55060) 2022-06-29T17:09:07.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.547+0000 7fd28259c700 1 -- 172.21.15.60:0/714195907 learned_addr learned my addr 172.21.15.60:0/714195907 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T17:09:07.571 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.547+0000 7fd282d9d700 1 --2- 172.21.15.60:0/714195907 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2840b9d70 0x7fd284144ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:07.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.547+0000 7fd281d9b700 1 --2- 172.21.15.60:0/714195907 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2840b3cc0 0x7fd28405c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:07.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.547+0000 7fd28259c700 1 -- 172.21.15.60:0/714195907 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2840b3cc0 msgr2=0x7fd28405c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:07.572 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.547+0000 7fd28259c700 1 --2- 172.21.15.60:0/714195907 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2840b3cc0 0x7fd28405c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.547+0000 7fd28259c700 1 -- 172.21.15.60:0/714195907 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2840b9d70 msgr2=0x7fd284144ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:07.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.547+0000 7fd28259c700 1 --2- 172.21.15.60:0/714195907 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2840b9d70 0x7fd284144ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.573 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.547+0000 7fd28259c700 1 -- 172.21.15.60:0/714195907 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd27c009530 con 0x7fd28405d1f0 2022-06-29T17:09:07.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.547+0000 7fd281d9b700 1 --2- 172.21.15.60:0/714195907 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fd2840b3cc0 0x7fd28405c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T17:09:07.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.551+0000 7fd28259c700 1 --2- 172.21.15.60:0/714195907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd28405d1f0 0x7fd2840af9d0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 rx=0x7fd274000ed0 tx=0x7fd274007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:07.574 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.551+0000 7fd27b7fe700 1 -- 172.21.15.60:0/714195907 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd274012400 con 0x7fd28405d1f0 2022-06-29T17:09:07.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.551+0000 7fd27b7fe700 1 -- 172.21.15.60:0/714195907 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd274012560 con 0x7fd28405d1f0 2022-06-29T17:09:07.575 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.551+0000 7fd27b7fe700 1 -- 172.21.15.60:0/714195907 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2740128c0 con 0x7fd28405d1f0 2022-06-29T17:09:07.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.551+0000 7fd2889b8700 1 -- 172.21.15.60:0/714195907 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd284141770 con 0x7fd28405d1f0 2022-06-29T17:09:07.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.551+0000 7fd2889b8700 1 -- 172.21.15.60:0/714195907 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2841442c0 con 0x7fd28405d1f0 2022-06-29T17:09:07.576 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.551+0000 7fd2889b8700 1 -- 172.21.15.60:0/714195907 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd2840ade80 con 0x7fd28405d1f0 2022-06-29T17:09:07.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.551+0000 7fd27b7fe700 1 -- 172.21.15.60:0/714195907 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fd274019030 con 0x7fd28405d1f0 2022-06-29T17:09:07.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.551+0000 7fd27b7fe700 1 --2- 172.21.15.60:0/714195907 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd268053b00 0x7fd268055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:07.577 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.551+0000 7fd27b7fe700 1 -- 172.21.15.60:0/714195907 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fd27406afd0 con 0x7fd28405d1f0 2022-06-29T17:09:07.578 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.551+0000 7fd27b7fe700 1 --2- 172.21.15.60:0/714195907 >> [v2:172.21.15.92:6816/13250,v1:172.21.15.92:6819/13250] conn(0x7fd26805a1f0 0x7fd26805c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:07.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.551+0000 7fd281d9b700 1 --2- 172.21.15.60:0/714195907 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd268053b00 0x7fd268055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:07.579 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.551+0000 7fd27b7fe700 1 -- 172.21.15.60:0/714195907 --> [v2:172.21.15.92:6816/13250,v1:172.21.15.92:6819/13250] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd26805cc30 con 0x7fd26805a1f0 2022-06-29T17:09:07.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.551+0000 7fd27b7fe700 1 -- 172.21.15.60:0/714195907 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=150) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd27406d590 con 0x7fd28405d1f0 2022-06-29T17:09:07.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.551+0000 7fd282d9d700 1 --2- 172.21.15.60:0/714195907 >> [v2:172.21.15.92:6816/13250,v1:172.21.15.92:6819/13250] conn(0x7fd26805a1f0 0x7fd26805c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:07.580 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.551+0000 7fd281d9b700 1 --2- 172.21.15.60:0/714195907 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd268053b00 0x7fd268055fc0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0x7fd26c00c000 tx=0x7fd26c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:07.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.551+0000 7fd282d9d700 1 --2- 172.21.15.60:0/714195907 >> [v2:172.21.15.92:6816/13250,v1:172.21.15.92:6819/13250] conn(0x7fd26805a1f0 0x7fd26805c650 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:07.581 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.555+0000 7fd27b7fe700 1 -- 172.21.15.60:0/714195907 <== osd.6 v2:172.21.15.92:6816/13250 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7fd26805cc30 con 0x7fd26805a1f0 2022-06-29T17:09:07.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.567+0000 7fd2889b8700 1 -- 172.21.15.60:0/714195907 --> [v2:172.21.15.92:6816/13250,v1:172.21.15.92:6819/13250] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd2840573d0 con 0x7fd26805a1f0 2022-06-29T17:09:07.582 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.567+0000 7fd27b7fe700 1 -- 172.21.15.60:0/714195907 <== osd.6 v2:172.21.15.92:6816/13250 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fd2840573d0 con 0x7fd26805a1f0 2022-06-29T17:09:07.583 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T17:09:07.584 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-06-29T17:09:07.584 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T17:09:07.585 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.571+0000 7fd2889b8700 1 -- 172.21.15.60:0/714195907 >> [v2:172.21.15.92:6816/13250,v1:172.21.15.92:6819/13250] conn(0x7fd26805a1f0 msgr2=0x7fd26805c650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:07.586 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.571+0000 7fd2889b8700 1 --2- 172.21.15.60:0/714195907 >> [v2:172.21.15.92:6816/13250,v1:172.21.15.92:6819/13250] conn(0x7fd26805a1f0 0x7fd26805c650 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.586 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.571+0000 7fd2889b8700 1 -- 172.21.15.60:0/714195907 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd268053b00 msgr2=0x7fd268055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:07.587 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.571+0000 7fd2889b8700 1 --2- 172.21.15.60:0/714195907 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fd268053b00 0x7fd268055fc0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0x7fd26c00c000 tx=0x7fd26c008040).stop 2022-06-29T17:09:07.587 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.571+0000 7fd2889b8700 1 -- 172.21.15.60:0/714195907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd28405d1f0 msgr2=0x7fd2840af9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:07.587 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.571+0000 7fd2889b8700 1 --2- 172.21.15.60:0/714195907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd28405d1f0 0x7fd2840af9d0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 rx=0x7fd274000ed0 tx=0x7fd274007c50).stop 2022-06-29T17:09:07.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.571+0000 7fd282d9d700 1 -- 172.21.15.60:0/714195907 reap_dead start 2022-06-29T17:09:07.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.571+0000 7fd2889b8700 1 -- 172.21.15.60:0/714195907 shutdown_connections 2022-06-29T17:09:07.588 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.571+0000 7fd2889b8700 1 -- 172.21.15.60:0/714195907 >> 172.21.15.60:0/714195907 conn(0x7fd284062ce0 msgr2=0x7fd2840b7150 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:07.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.571+0000 7fd2889b8700 1 -- 172.21.15.60:0/714195907 shutdown_connections 2022-06-29T17:09:07.589 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.571+0000 7fd2889b8700 1 -- 172.21.15.60:0/714195907 wait complete. 2022-06-29T17:09:07.590 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2022-06-29T17:09:07.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.695+0000 7fc7b9007700 1 Processor -- start 2022-06-29T17:09:07.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.695+0000 7fc7b9007700 1 -- start start 2022-06-29T17:09:07.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.695+0000 7fc7b9007700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc7b40b61d0 0x7fc7b405f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:07.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.695+0000 7fc7b9007700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7b4060460 0x7fc7b40af290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:07.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.695+0000 7fc7b9007700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b40b65b0 con 0x7fc7b40b12e0 2022-06-29T17:09:07.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.695+0000 7fc7b9007700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7b40adf20 con 0x7fc7b40b61d0 2022-06-29T17:09:07.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.695+0000 7fc7b9007700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b4147110 con 0x7fc7b4060460 2022-06-29T17:09:07.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.695+0000 7fc7b259c700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc7b40b12e0 0x7fc7b41475b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34924/0 (socket says 172.21.15.60:34924) 2022-06-29T17:09:07.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.695+0000 7fc7b259c700 1 -- 172.21.15.60:0/2910273984 learned_addr learned my addr 172.21.15.60:0/2910273984 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T17:09:07.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.695+0000 7fc7b2d9d700 1 --2- 172.21.15.60:0/2910273984 >> v2:172.21.15.92:3300/0 conn(0x7fc7b40b61d0 0x7fc7b405f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:07.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.695+0000 7fc7b359e700 1 --2- 172.21.15.60:0/2910273984 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7b4060460 0x7fc7b40af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:07.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.695+0000 7fc7b1d9b700 1 -- 172.21.15.60:0/2910273984 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3148856337 0 0) 0x7fc7b40b65b0 con 0x7fc7b40b12e0 2022-06-29T17:09:07.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.695+0000 7fc7b1d9b700 1 -- 172.21.15.60:0/2910273984 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc794003260 con 0x7fc7b40b12e0 2022-06-29T17:09:07.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.695+0000 7fc7b2d9d700 1 -- 172.21.15.60:0/2910273984 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7b4060460 msgr2=0x7fc7b40af290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T17:09:07.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.695+0000 7fc7b2d9d700 1 --2- 172.21.15.60:0/2910273984 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7b4060460 0x7fc7b40af290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.695+0000 7fc7b2d9d700 1 -- 172.21.15.60:0/2910273984 >> v1:172.21.15.60:6789/0 conn(0x7fc7b40b12e0 legacy=0x7fc7b41475b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:07.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.695+0000 7fc7b1d9b700 1 -- 172.21.15.60:0/2910273984 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 648072041 0 0) 0x7fc794003260 con 0x7fc7b40b12e0 2022-06-29T17:09:07.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.695+0000 7fc7b2d9d700 1 -- 172.21.15.60:0/2910273984 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b4146e30 con 0x7fc7b40b61d0 2022-06-29T17:09:07.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.695+0000 7fc7b2d9d700 1 --2- 172.21.15.60:0/2910273984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7b40b61d0 0x7fc7b405f0b0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0x7fc7a4001ba0 tx=0x7fc79c006df0).ready entity=mon.1 client_cookie=1549c66c67dd385a server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:07.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.695+0000 7fc7b1d9b700 1 -- 172.21.15.60:0/2910273984 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc7a4001a00 con 0x7fc7b40b61d0 2022-06-29T17:09:07.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.695+0000 7fc7b1d9b700 1 -- 172.21.15.60:0/2910273984 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79c000e90 con 0x7fc7b40b61d0 2022-06-29T17:09:07.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.695+0000 7fc7b1d9b700 1 -- 172.21.15.60:0/2910273984 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc79c012460 con 0x7fc7b40b61d0 2022-06-29T17:09:07.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.695+0000 7fc7b9007700 1 -- 172.21.15.60:0/2910273984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7b40b61d0 msgr2=0x7fc7b405f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:07.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.695+0000 7fc7b9007700 1 --2- 172.21.15.60:0/2910273984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7b40b61d0 0x7fc7b405f0b0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0x7fc7a4001ba0 tx=0x7fc79c006df0).stop 2022-06-29T17:09:07.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.699+0000 7fc7b9007700 1 -- 172.21.15.60:0/2910273984 shutdown_connections 2022-06-29T17:09:07.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.699+0000 7fc7b9007700 1 --2- 172.21.15.60:0/2910273984 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7b4060460 0x7fc7b40af290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.699+0000 7fc7b9007700 1 --2- 172.21.15.60:0/2910273984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7b40b61d0 0x7fc7b405f0b0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.699+0000 7fc7b9007700 1 -- 172.21.15.60:0/2910273984 >> 172.21.15.60:0/2910273984 conn(0x7fc7b4063b50 msgr2=0x7fc7b40b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:07.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.699+0000 7fc7b9007700 1 -- 172.21.15.60:0/2910273984 shutdown_connections 2022-06-29T17:09:07.712 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.699+0000 7fc7b9007700 1 -- 172.21.15.60:0/2910273984 wait complete. 2022-06-29T17:09:07.715 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.699+0000 7fc7b9007700 1 Processor -- start 2022-06-29T17:09:07.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.699+0000 7fc7b9007700 1 -- start start 2022-06-29T17:09:07.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.699+0000 7fc7b9007700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7b4060460 0x7fc7b40ae9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:07.716 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.699+0000 7fc7b9007700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7b40b12e0 0x7fc7b40642e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:07.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.699+0000 7fc7b9007700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7b40b61d0 0x7fc7b4159620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:07.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.699+0000 7fc7b9007700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b4066070 con 0x7fc7b40b12e0 2022-06-29T17:09:07.717 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.699+0000 7fc7b9007700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b4066310 con 0x7fc7b4060460 2022-06-29T17:09:07.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.699+0000 7fc7b9007700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b40ae550 con 0x7fc7b40b61d0 2022-06-29T17:09:07.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.699+0000 7fc7b259c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7b40b12e0 0x7fc7b40642e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:07.718 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.699+0000 7fc7b259c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7b40b12e0 0x7fc7b40642e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55078/0 (socket says 172.21.15.60:55078) 2022-06-29T17:09:07.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.699+0000 7fc7b259c700 1 -- 172.21.15.60:0/2663986104 learned_addr learned my addr 172.21.15.60:0/2663986104 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T17:09:07.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.699+0000 7fc7b359e700 1 --2- 172.21.15.60:0/2663986104 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7b40b61d0 0x7fc7b4159620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:07.719 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.699+0000 7fc7b2d9d700 1 --2- 172.21.15.60:0/2663986104 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7b4060460 0x7fc7b40ae9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:07.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.699+0000 7fc7b259c700 1 -- 172.21.15.60:0/2663986104 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7b40b61d0 msgr2=0x7fc7b4159620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:07.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.699+0000 7fc7b259c700 1 --2- 172.21.15.60:0/2663986104 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7b40b61d0 0x7fc7b4159620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.720 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.699+0000 7fc7b259c700 1 -- 172.21.15.60:0/2663986104 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7b4060460 msgr2=0x7fc7b40ae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:07.721 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.699+0000 7fc7b259c700 1 --2- 172.21.15.60:0/2663986104 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7b4060460 0x7fc7b40ae9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.721 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.699+0000 7fc7b259c700 1 -- 172.21.15.60:0/2663986104 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc79c00b180 con 0x7fc7b40b12e0 2022-06-29T17:09:07.721 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.703+0000 7fc7b259c700 1 --2- 172.21.15.60:0/2663986104 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7b40b12e0 0x7fc7b40642e0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 rx=0x7fc7a4001c60 tx=0x7fc7a40015d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:07.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.703+0000 7fc7abfff700 1 -- 172.21.15.60:0/2663986104 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc7a4004ab0 con 0x7fc7b40b12e0 2022-06-29T17:09:07.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.703+0000 7fc7abfff700 1 -- 172.21.15.60:0/2663986104 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a4004c10 con 0x7fc7b40b12e0 2022-06-29T17:09:07.722 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.703+0000 7fc7b9007700 1 -- 172.21.15.60:0/2663986104 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7b414a940 con 0x7fc7b40b12e0 2022-06-29T17:09:07.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.703+0000 7fc7abfff700 1 -- 172.21.15.60:0/2663986104 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc7a4007170 con 0x7fc7b40b12e0 2022-06-29T17:09:07.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.703+0000 7fc7b9007700 1 -- 172.21.15.60:0/2663986104 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7b414a230 con 0x7fc7b40b12e0 2022-06-29T17:09:07.723 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.703+0000 7fc7b9007700 1 -- 172.21.15.60:0/2663986104 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7b40ae710 con 0x7fc7b40b12e0 2022-06-29T17:09:07.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.703+0000 7fc7abfff700 1 -- 172.21.15.60:0/2663986104 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fc7a401b060 con 0x7fc7b40b12e0 2022-06-29T17:09:07.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.703+0000 7fc7abfff700 1 --2- 172.21.15.60:0/2663986104 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc798053b30 0x7fc798055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:07.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.703+0000 7fc7abfff700 1 -- 172.21.15.60:0/2663986104 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fc7a406b8e0 con 0x7fc7b40b12e0 2022-06-29T17:09:07.724 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.703+0000 7fc7b2d9d700 1 --2- 172.21.15.60:0/2663986104 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc798053b30 0x7fc798055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:07.725 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.703+0000 7fc7b2d9d700 1 --2- 172.21.15.60:0/2663986104 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc798053b30 0x7fc798055ff0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0x7fc79c01a000 tx=0x7fc79c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:07.726 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.715+0000 7fc7abfff700 1 -- 172.21.15.60:0/2663986104 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc7a4042ae0 con 0x7fc7b40b12e0 2022-06-29T17:09:07.845 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.839+0000 7fc7b9007700 1 -- 172.21.15.60:0/2663986104 --> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7fc7b40573d0 con 0x7fc798053b30 2022-06-29T17:09:07.846 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.839+0000 7fc7abfff700 1 -- 172.21.15.60:0/2663986104 <== mgr.4103 v2:172.21.15.92:6832/12872 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc7b40573d0 con 0x7fc798053b30 2022-06-29T17:09:07.848 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.843+0000 7fc7b9007700 1 -- 172.21.15.60:0/2663986104 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc798053b30 msgr2=0x7fc798055ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:07.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.843+0000 7fc7b9007700 1 --2- 172.21.15.60:0/2663986104 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc798053b30 0x7fc798055ff0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0x7fc79c01a000 tx=0x7fc79c014000).stop 2022-06-29T17:09:07.849 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.843+0000 7fc7b9007700 1 -- 172.21.15.60:0/2663986104 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7b40b12e0 msgr2=0x7fc7b40642e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:07.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.843+0000 7fc7b9007700 1 --2- 172.21.15.60:0/2663986104 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7b40b12e0 0x7fc7b40642e0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 rx=0x7fc7a4001c60 tx=0x7fc7a40015d0).stop 2022-06-29T17:09:07.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.843+0000 7fc7b9007700 1 -- 172.21.15.60:0/2663986104 shutdown_connections 2022-06-29T17:09:07.850 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.843+0000 7fc7b9007700 1 --2- 172.21.15.60:0/2663986104 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fc798053b30 0x7fc798055ff0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.843+0000 7fc7b9007700 1 --2- 172.21.15.60:0/2663986104 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fc7b40b61d0 0x7fc7b4159620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.843+0000 7fc7b9007700 1 --2- 172.21.15.60:0/2663986104 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7b40b12e0 0x7fc7b40642e0 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.851 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.843+0000 7fc7b9007700 1 --2- 172.21.15.60:0/2663986104 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7b4060460 0x7fc7b40ae9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.843+0000 7fc7b9007700 1 -- 172.21.15.60:0/2663986104 >> 172.21.15.60:0/2663986104 conn(0x7fc7b4063b50 msgr2=0x7fc7b40b04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:07.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.843+0000 7fc7b9007700 1 -- 172.21.15.60:0/2663986104 shutdown_connections 2022-06-29T17:09:07.852 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.843+0000 7fc7b9007700 1 -- 172.21.15.60:0/2663986104 wait complete. 2022-06-29T17:09:07.853 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 6 to deep-scrub 2022-06-29T17:09:07.860 INFO:tasks.ceph:Scrubbing osd.7 2022-06-29T17:09:07.860 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2022-06-29T17:09:07.970 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.963+0000 7ffa8759e700 1 Processor -- start 2022-06-29T17:09:07.971 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.963+0000 7ffa8759e700 1 -- start start 2022-06-29T17:09:07.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.967+0000 7ffa8759e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffa880b1480 0x7ffa880b18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:07.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.967+0000 7ffa8759e700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa880b0f50 con 0x7ffa880673e0 2022-06-29T17:09:07.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.967+0000 7ffa8759e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa88064db0 con 0x7ffa88066590 2022-06-29T17:09:07.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.967+0000 7ffa8759e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa88067080 con 0x7ffa880b1480 2022-06-29T17:09:07.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.967+0000 7ffa7ffff700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ffa880673e0 0x7ffa88066970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34940/0 (socket says 172.21.15.60:34940) 2022-06-29T17:09:07.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.967+0000 7ffa7ffff700 1 -- 172.21.15.60:0/1049388451 learned_addr learned my addr 172.21.15.60:0/1049388451 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T17:09:07.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.967+0000 7ffa86d9d700 1 --2- 172.21.15.60:0/1049388451 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffa880b1480 0x7ffa880b18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:07.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.967+0000 7ffa85d9b700 1 -- 172.21.15.60:0/1049388451 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1647502951 0 0) 0x7ffa70001110 con 0x7ffa880673e0 2022-06-29T17:09:07.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.967+0000 7ffa85d9b700 1 -- 172.21.15.60:0/1049388451 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa68003260 con 0x7ffa880673e0 2022-06-29T17:09:07.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.967+0000 7ffa85d9b700 1 -- 172.21.15.60:0/1049388451 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3652419681 0 0) 0x7ffa88064db0 con 0x7ffa88066590 2022-06-29T17:09:07.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.967+0000 7ffa85d9b700 1 -- 172.21.15.60:0/1049388451 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa70001110 con 0x7ffa88066590 2022-06-29T17:09:07.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.967+0000 7ffa86d9d700 1 -- 172.21.15.60:0/1049388451 >> v1:172.21.15.92:6789/0 conn(0x7ffa88066590 legacy=0x7ffa88140300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:07.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.967+0000 7ffa86d9d700 1 -- 172.21.15.60:0/1049388451 >> v1:172.21.15.60:6789/0 conn(0x7ffa880673e0 legacy=0x7ffa88066970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:07.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.967+0000 7ffa85d9b700 1 -- 172.21.15.60:0/1049388451 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3050414570 0 0) 0x7ffa68003260 con 0x7ffa880673e0 2022-06-29T17:09:07.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.967+0000 7ffa86d9d700 1 -- 172.21.15.60:0/1049388451 --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa88065050 con 0x7ffa880b1480 2022-06-29T17:09:07.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.967+0000 7ffa86d9d700 1 --2- 172.21.15.60:0/1049388451 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffa880b1480 0x7ffa880b18c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7ffa700012a0 tx=0x7ffa800091d0).ready entity=mon.2 client_cookie=490b58a6eba666cc server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:07.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.967+0000 7ffa85d9b700 1 -- 172.21.15.60:0/1049388451 <== mon.2 v2:172.21.15.92:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa700017f0 con 0x7ffa880b1480 2022-06-29T17:09:07.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.967+0000 7ffa85d9b700 1 -- 172.21.15.60:0/1049388451 <== mon.2 v2:172.21.15.92:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa8000f510 con 0x7ffa880b1480 2022-06-29T17:09:07.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.967+0000 7ffa85d9b700 1 -- 172.21.15.60:0/1049388451 <== mon.2 v2:172.21.15.92:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa8000f680 con 0x7ffa880b1480 2022-06-29T17:09:07.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.967+0000 7ffa8759e700 1 -- 172.21.15.60:0/1049388451 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffa880b1480 msgr2=0x7ffa880b18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:07.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.967+0000 7ffa8759e700 1 --2- 172.21.15.60:0/1049388451 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffa880b1480 0x7ffa880b18c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7ffa700012a0 tx=0x7ffa800091d0).stop 2022-06-29T17:09:07.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.967+0000 7ffa8759e700 1 -- 172.21.15.60:0/1049388451 shutdown_connections 2022-06-29T17:09:07.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.967+0000 7ffa8759e700 1 --2- 172.21.15.60:0/1049388451 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffa880b1480 0x7ffa880b18c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.967+0000 7ffa8759e700 1 -- 172.21.15.60:0/1049388451 >> 172.21.15.60:0/1049388451 conn(0x7ffa880b5e70 msgr2=0x7ffa880b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:07.989 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.967+0000 7ffa8759e700 1 -- 172.21.15.60:0/1049388451 shutdown_connections 2022-06-29T17:09:07.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.967+0000 7ffa8759e700 1 -- 172.21.15.60:0/1049388451 wait complete. 2022-06-29T17:09:07.990 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.971+0000 7ffa8759e700 1 Processor -- start 2022-06-29T17:09:07.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.971+0000 7ffa8759e700 1 -- start start 2022-06-29T17:09:07.991 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.971+0000 7ffa8759e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffa88066590 0x7ffa88132210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:07.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.971+0000 7ffa8759e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffa880673e0 0x7ffa88133680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:07.992 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.971+0000 7ffa8759e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa880b1480 0x7ffa8813ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:07.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.971+0000 7ffa8759e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8813b340 con 0x7ffa880b1480 2022-06-29T17:09:07.993 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.971+0000 7ffa8759e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa880afe20 con 0x7ffa88066590 2022-06-29T17:09:07.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.971+0000 7ffa8759e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa880b00c0 con 0x7ffa880673e0 2022-06-29T17:09:07.994 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.971+0000 7ffa86d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa880b1480 0x7ffa8813ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:07.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.971+0000 7ffa86d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa880b1480 0x7ffa8813ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55094/0 (socket says 172.21.15.60:55094) 2022-06-29T17:09:07.995 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.971+0000 7ffa8659c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffa88066590 0x7ffa88132210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:07.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.971+0000 7ffa86d9d700 1 -- 172.21.15.60:0/2393136876 learned_addr learned my addr 172.21.15.60:0/2393136876 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T17:09:07.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.971+0000 7ffa7ffff700 1 --2- 172.21.15.60:0/2393136876 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffa880673e0 0x7ffa88133680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:07.996 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.971+0000 7ffa86d9d700 1 -- 172.21.15.60:0/2393136876 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffa880673e0 msgr2=0x7ffa88133680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:07.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.971+0000 7ffa86d9d700 1 --2- 172.21.15.60:0/2393136876 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ffa880673e0 0x7ffa88133680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.971+0000 7ffa86d9d700 1 -- 172.21.15.60:0/2393136876 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffa88066590 msgr2=0x7ffa88132210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:07.997 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.971+0000 7ffa86d9d700 1 --2- 172.21.15.60:0/2393136876 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffa88066590 0x7ffa88132210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:07.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.971+0000 7ffa86d9d700 1 -- 172.21.15.60:0/2393136876 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa80009470 con 0x7ffa880b1480 2022-06-29T17:09:07.998 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.975+0000 7ffa86d9d700 1 --2- 172.21.15.60:0/2393136876 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa880b1480 0x7ffa8813ae50 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 rx=0x7ffa700012a0 tx=0x7ffa80016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:07.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.975+0000 7ffa7f7fe700 1 -- 172.21.15.60:0/2393136876 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa800032f0 con 0x7ffa880b1480 2022-06-29T17:09:07.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.975+0000 7ffa7f7fe700 1 -- 172.21.15.60:0/2393136876 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa80011b80 con 0x7ffa880b1480 2022-06-29T17:09:07.999 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.975+0000 7ffa7f7fe700 1 -- 172.21.15.60:0/2393136876 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa80021440 con 0x7ffa880b1480 2022-06-29T17:09:08.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.975+0000 7ffa8759e700 1 -- 172.21.15.60:0/2393136876 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa880b0360 con 0x7ffa880b1480 2022-06-29T17:09:08.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.975+0000 7ffa8759e700 1 -- 172.21.15.60:0/2393136876 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa880ae460 con 0x7ffa880b1480 2022-06-29T17:09:08.000 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.975+0000 7ffa8759e700 1 -- 172.21.15.60:0/2393136876 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ffa880b04f0 con 0x7ffa880b1480 2022-06-29T17:09:08.001 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.975+0000 7ffa7f7fe700 1 -- 172.21.15.60:0/2393136876 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ffa80027020 con 0x7ffa880b1480 2022-06-29T17:09:08.003 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.975+0000 7ffa7f7fe700 1 --2- 172.21.15.60:0/2393136876 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ffa6c053a70 0x7ffa6c055f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:08.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.975+0000 7ffa7f7fe700 1 -- 172.21.15.60:0/2393136876 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5339+0+0 (secure 0 0 0) 0x7ffa8006cf10 con 0x7ffa880b1480 2022-06-29T17:09:08.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.975+0000 7ffa7f7fe700 1 --2- 172.21.15.60:0/2393136876 >> [v2:172.21.15.92:6800/13247,v1:172.21.15.92:6801/13247] conn(0x7ffa6c05a160 0x7ffa6c05c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:08.004 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.975+0000 7ffa7f7fe700 1 -- 172.21.15.60:0/2393136876 --> [v2:172.21.15.92:6800/13247,v1:172.21.15.92:6801/13247] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa6c05cba0 con 0x7ffa6c05a160 2022-06-29T17:09:08.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.975+0000 7ffa7f7fe700 1 -- 172.21.15.60:0/2393136876 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=150) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffa8006f490 con 0x7ffa880b1480 2022-06-29T17:09:08.005 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.975+0000 7ffa8659c700 1 --2- 172.21.15.60:0/2393136876 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ffa6c053a70 0x7ffa6c055f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:08.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.979+0000 7ffa7ffff700 1 --2- 172.21.15.60:0/2393136876 >> [v2:172.21.15.92:6800/13247,v1:172.21.15.92:6801/13247] conn(0x7ffa6c05a160 0x7ffa6c05c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:08.006 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.979+0000 7ffa8659c700 1 --2- 172.21.15.60:0/2393136876 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ffa6c053a70 0x7ffa6c055f30 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0x7ffa7800d000 tx=0x7ffa78007a20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:08.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.979+0000 7ffa7ffff700 1 --2- 172.21.15.60:0/2393136876 >> [v2:172.21.15.92:6800/13247,v1:172.21.15.92:6801/13247] conn(0x7ffa6c05a160 0x7ffa6c05c5c0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:08.007 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.979+0000 7ffa7f7fe700 1 -- 172.21.15.60:0/2393136876 <== osd.7 v2:172.21.15.92:6800/13247 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7ffa6c05cba0 con 0x7ffa6c05a160 2022-06-29T17:09:08.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.995+0000 7ffa8759e700 1 -- 172.21.15.60:0/2393136876 --> [v2:172.21.15.92:6800/13247,v1:172.21.15.92:6801/13247] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ffa880573d0 con 0x7ffa6c05a160 2022-06-29T17:09:08.008 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.995+0000 7ffa7f7fe700 1 -- 172.21.15.60:0/2393136876 <== osd.7 v2:172.21.15.92:6800/13247 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7ffa880573d0 con 0x7ffa6c05a160 2022-06-29T17:09:08.009 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2022-06-29T17:09:08.009 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-06-29T17:09:08.010 INFO:teuthology.orchestra.run.smithi060.stdout:} 2022-06-29T17:09:08.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.995+0000 7ffa8759e700 1 -- 172.21.15.60:0/2393136876 >> [v2:172.21.15.92:6800/13247,v1:172.21.15.92:6801/13247] conn(0x7ffa6c05a160 msgr2=0x7ffa6c05c5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:08.012 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.995+0000 7ffa8759e700 1 --2- 172.21.15.60:0/2393136876 >> [v2:172.21.15.92:6800/13247,v1:172.21.15.92:6801/13247] conn(0x7ffa6c05a160 0x7ffa6c05c5c0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:08.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.995+0000 7ffa8759e700 1 -- 172.21.15.60:0/2393136876 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ffa6c053a70 msgr2=0x7ffa6c055f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:08.013 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.995+0000 7ffa8759e700 1 --2- 172.21.15.60:0/2393136876 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ffa6c053a70 0x7ffa6c055f30 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0x7ffa7800d000 tx=0x7ffa78007a20).stop 2022-06-29T17:09:08.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.995+0000 7ffa8759e700 1 -- 172.21.15.60:0/2393136876 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa880b1480 msgr2=0x7ffa8813ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:08.014 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.995+0000 7ffa8759e700 1 --2- 172.21.15.60:0/2393136876 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa880b1480 0x7ffa8813ae50 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 rx=0x7ffa700012a0 tx=0x7ffa80016000).stop 2022-06-29T17:09:08.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.995+0000 7ffa86d9d700 1 -- 172.21.15.60:0/2393136876 reap_dead start 2022-06-29T17:09:08.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.995+0000 7ffa8759e700 1 -- 172.21.15.60:0/2393136876 shutdown_connections 2022-06-29T17:09:08.015 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.995+0000 7ffa8759e700 1 -- 172.21.15.60:0/2393136876 >> 172.21.15.60:0/2393136876 conn(0x7ffa880b5e70 msgr2=0x7ffa881361c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:08.016 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.995+0000 7ffa8759e700 1 -- 172.21.15.60:0/2393136876 shutdown_connections 2022-06-29T17:09:08.016 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:07.995+0000 7ffa8759e700 1 -- 172.21.15.60:0/2393136876 wait complete. 2022-06-29T17:09:08.018 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2022-06-29T17:09:08.128 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.123+0000 7f994759e700 1 Processor -- start 2022-06-29T17:09:08.129 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.123+0000 7f994759e700 1 -- start start 2022-06-29T17:09:08.132 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.123+0000 7f994759e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f994805d070 0x7f994814aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:08.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.123+0000 7f994759e700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f99480b2130 0x7f99480aca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:08.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.123+0000 7f994759e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f994814af80 0x7f99480acf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:08.133 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.123+0000 7f994759e700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99480adf90 con 0x7f99480b2130 2022-06-29T17:09:08.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.123+0000 7f994759e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f994805d450 con 0x7f994805d070 2022-06-29T17:09:08.134 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.123+0000 7f994759e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99480b25d0 con 0x7f994814af80 2022-06-29T17:09:08.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.123+0000 7f9945d9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f99480b2130 0x7f99480aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:08.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.123+0000 7f9945d9b700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f99480b2130 0x7f99480aca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55100/0 (socket says 172.21.15.60:55100) 2022-06-29T17:09:08.135 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.123+0000 7f9945d9b700 1 -- 172.21.15.60:0/642944882 learned_addr learned my addr 172.21.15.60:0/642944882 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T17:09:08.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.123+0000 7f9946d9d700 1 --2- 172.21.15.60:0/642944882 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f994814af80 0x7f99480acf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:08.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.123+0000 7f994659c700 1 --2- 172.21.15.60:0/642944882 >> v2:172.21.15.92:3300/0 conn(0x7f994805d070 0x7f994814aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:08.136 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.123+0000 7f9945d9b700 1 -- 172.21.15.60:0/642944882 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f994814af80 msgr2=0x7f99480acf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T17:09:08.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.123+0000 7f9945d9b700 1 --2- 172.21.15.60:0/642944882 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f994814af80 0x7f99480acf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:08.137 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.123+0000 7f9945d9b700 1 -- 172.21.15.60:0/642944882 >> v2:172.21.15.92:3300/0 conn(0x7f994805d070 msgr2=0x7f994814aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T17:09:08.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.123+0000 7f9945d9b700 1 --2- 172.21.15.60:0/642944882 >> v2:172.21.15.92:3300/0 conn(0x7f994805d070 0x7f994814aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:08.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.123+0000 7f9945d9b700 1 -- 172.21.15.60:0/642944882 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994814a350 con 0x7f99480b2130 2022-06-29T17:09:08.138 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.123+0000 7f994659c700 1 --2- 172.21.15.60:0/642944882 >> v2:172.21.15.92:3300/0 conn(0x7f994805d070 0x7f994814aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-06-29T17:09:08.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.123+0000 7f9945d9b700 1 --2- 172.21.15.60:0/642944882 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99480b2130 0x7f99480aca20 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 rx=0x7f9938002f70 tx=0x7f9938006eb0).ready entity=mon.0 client_cookie=464a78486d152d28 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:08.139 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.123+0000 7f994559a700 1 -- 172.21.15.60:0/642944882 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f993800d570 con 0x7f99480b2130 2022-06-29T17:09:08.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.123+0000 7f994559a700 1 -- 172.21.15.60:0/642944882 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9938000e90 con 0x7f99480b2130 2022-06-29T17:09:08.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.123+0000 7f994559a700 1 -- 172.21.15.60:0/642944882 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f993800d900 con 0x7f99480b2130 2022-06-29T17:09:08.140 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.127+0000 7f994759e700 1 -- 172.21.15.60:0/642944882 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99480b2130 msgr2=0x7f99480aca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:08.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.127+0000 7f994759e700 1 --2- 172.21.15.60:0/642944882 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99480b2130 0x7f99480aca20 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 rx=0x7f9938002f70 tx=0x7f9938006eb0).stop 2022-06-29T17:09:08.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.127+0000 7f994759e700 1 -- 172.21.15.60:0/642944882 shutdown_connections 2022-06-29T17:09:08.141 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.127+0000 7f994759e700 1 --2- 172.21.15.60:0/642944882 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f994814af80 0x7f99480acf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:08.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.127+0000 7f994759e700 1 --2- 172.21.15.60:0/642944882 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99480b2130 0x7f99480aca20 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:08.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.127+0000 7f994759e700 1 --2- 172.21.15.60:0/642944882 >> v2:172.21.15.92:3300/0 conn(0x7f994805d070 0x7f994814aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:08.142 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.127+0000 7f994759e700 1 -- 172.21.15.60:0/642944882 >> 172.21.15.60:0/642944882 conn(0x7f99480b5890 msgr2=0x7f99480b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:08.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.127+0000 7f994759e700 1 -- 172.21.15.60:0/642944882 shutdown_connections 2022-06-29T17:09:08.143 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.127+0000 7f994759e700 1 -- 172.21.15.60:0/642944882 wait complete. 2022-06-29T17:09:08.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.127+0000 7f994759e700 1 Processor -- start 2022-06-29T17:09:08.147 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.127+0000 7f994759e700 1 -- start start 2022-06-29T17:09:08.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.131+0000 7f994759e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f994805d070 0x7f99480b31f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:08.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.131+0000 7f994759e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99480b2130 0x7f9948142da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:08.148 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.131+0000 7f994759e700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f994814af80 0x7f9948143290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:08.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.131+0000 7f994759e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9948149d40 con 0x7f99480b2130 2022-06-29T17:09:08.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.131+0000 7f994759e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9948149fe0 con 0x7f994805d070 2022-06-29T17:09:08.149 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.131+0000 7f994759e700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9948063250 con 0x7f994814af80 2022-06-29T17:09:08.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.131+0000 7f9945d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99480b2130 0x7f9948142da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:08.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.131+0000 7f9945d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99480b2130 0x7f9948142da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55108/0 (socket says 172.21.15.60:55108) 2022-06-29T17:09:08.150 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.131+0000 7f9945d9b700 1 -- 172.21.15.60:0/346183809 learned_addr learned my addr 172.21.15.60:0/346183809 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T17:09:08.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.131+0000 7f9946d9d700 1 --2- 172.21.15.60:0/346183809 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f994814af80 0x7f9948143290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:08.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.131+0000 7f994659c700 1 --2- 172.21.15.60:0/346183809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f994805d070 0x7f99480b31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:08.151 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.131+0000 7f9945d9b700 1 -- 172.21.15.60:0/346183809 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f994814af80 msgr2=0x7f9948143290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:08.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.131+0000 7f9945d9b700 1 --2- 172.21.15.60:0/346183809 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f994814af80 0x7f9948143290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:08.152 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.131+0000 7f9945d9b700 1 -- 172.21.15.60:0/346183809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f994805d070 msgr2=0x7f99480b31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:08.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.131+0000 7f9945d9b700 1 --2- 172.21.15.60:0/346183809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f994805d070 0x7f99480b31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:08.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.131+0000 7f9945d9b700 1 -- 172.21.15.60:0/346183809 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f993800b1e0 con 0x7f99480b2130 2022-06-29T17:09:08.153 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.131+0000 7f9945d9b700 1 --2- 172.21.15.60:0/346183809 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99480b2130 0x7f9948142da0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 rx=0x7f9938002f70 tx=0x7f9938015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:08.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.131+0000 7f993f7fe700 1 -- 172.21.15.60:0/346183809 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9938011860 con 0x7f99480b2130 2022-06-29T17:09:08.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.131+0000 7f993f7fe700 1 -- 172.21.15.60:0/346183809 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99380119c0 con 0x7f99480b2130 2022-06-29T17:09:08.154 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.131+0000 7f994759e700 1 -- 172.21.15.60:0/346183809 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9948143780 con 0x7f99480b2130 2022-06-29T17:09:08.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.131+0000 7f993f7fe700 1 -- 172.21.15.60:0/346183809 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9938011d20 con 0x7f99480b2130 2022-06-29T17:09:08.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.131+0000 7f994759e700 1 -- 172.21.15.60:0/346183809 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9948156110 con 0x7f99480b2130 2022-06-29T17:09:08.155 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.131+0000 7f994759e700 1 -- 172.21.15.60:0/346183809 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f994814a5f0 con 0x7f99480b2130 2022-06-29T17:09:08.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.131+0000 7f993f7fe700 1 -- 172.21.15.60:0/346183809 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f9938025030 con 0x7f99480b2130 2022-06-29T17:09:08.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.135+0000 7f993f7fe700 1 --2- 172.21.15.60:0/346183809 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9924053950 0x7f9924055e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:08.156 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.135+0000 7f993f7fe700 1 -- 172.21.15.60:0/346183809 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f993806cf20 con 0x7f99480b2130 2022-06-29T17:09:08.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.135+0000 7f994659c700 1 --2- 172.21.15.60:0/346183809 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9924053950 0x7f9924055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:08.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.135+0000 7f994659c700 1 --2- 172.21.15.60:0/346183809 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9924053950 0x7f9924055e10 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0x7f993400b000 tx=0x7f9934008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:08.157 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.139+0000 7f993f7fe700 1 -- 172.21.15.60:0/346183809 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9938044120 con 0x7f99480b2130 2022-06-29T17:09:08.273 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.267+0000 7f994759e700 1 -- 172.21.15.60:0/346183809 --> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f9948149bc0 con 0x7f9924053950 2022-06-29T17:09:08.274 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.267+0000 7f993f7fe700 1 -- 172.21.15.60:0/346183809 <== mgr.4103 v2:172.21.15.92:6832/12872 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f9934000e90 con 0x7f9924053950 2022-06-29T17:09:08.276 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.271+0000 7f994759e700 1 -- 172.21.15.60:0/346183809 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9924053950 msgr2=0x7f9924055e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:08.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.271+0000 7f994759e700 1 --2- 172.21.15.60:0/346183809 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9924053950 0x7f9924055e10 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0x7f993400b000 tx=0x7f9934008040).stop 2022-06-29T17:09:08.277 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.271+0000 7f994759e700 1 -- 172.21.15.60:0/346183809 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99480b2130 msgr2=0x7f9948142da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:08.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.271+0000 7f994759e700 1 --2- 172.21.15.60:0/346183809 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99480b2130 0x7f9948142da0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 rx=0x7f9938002f70 tx=0x7f9938015000).stop 2022-06-29T17:09:08.278 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.271+0000 7f994759e700 1 -- 172.21.15.60:0/346183809 shutdown_connections 2022-06-29T17:09:08.279 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.271+0000 7f994759e700 1 --2- 172.21.15.60:0/346183809 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f9924053950 0x7f9924055e10 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:08.280 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.271+0000 7f994759e700 1 --2- 172.21.15.60:0/346183809 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f994814af80 0x7f9948143290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:08.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.271+0000 7f994759e700 1 --2- 172.21.15.60:0/346183809 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99480b2130 0x7f9948142da0 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:08.281 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.271+0000 7f994759e700 1 --2- 172.21.15.60:0/346183809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f994805d070 0x7f99480b31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:08.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.271+0000 7f994759e700 1 -- 172.21.15.60:0/346183809 >> 172.21.15.60:0/346183809 conn(0x7f99480b5890 msgr2=0x7f99480b4390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:08.282 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.271+0000 7f994759e700 1 -- 172.21.15.60:0/346183809 shutdown_connections 2022-06-29T17:09:08.283 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.271+0000 7f994759e700 1 -- 172.21.15.60:0/346183809 wait complete. 2022-06-29T17:09:08.283 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 7 to deep-scrub 2022-06-29T17:09:08.287 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-06-29T17:09:08.392 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.387+0000 7f63ddeb6700 1 Processor -- start 2022-06-29T17:09:08.393 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.387+0000 7f63ddeb6700 1 -- start start 2022-06-29T17:09:08.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.387+0000 7f63ddeb6700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f63d805d070 0x7f63d812b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:08.396 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.387+0000 7f63ddeb6700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f63d812bbc0 0x7f63d81431f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:08.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.387+0000 7f63ddeb6700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d80ae9a0 con 0x7f63d8066b20 2022-06-29T17:09:08.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.387+0000 7f63ddeb6700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63d8143b60 con 0x7f63d805d070 2022-06-29T17:09:08.397 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.387+0000 7f63ddeb6700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63d80aced0 con 0x7f63d812bbc0 2022-06-29T17:09:08.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.387+0000 7f63dceb4700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f63d805d070 0x7f63d812b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:08.398 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.387+0000 7f63dceb4700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f63d805d070 0x7f63d812b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55114/0 (socket says 172.21.15.60:55114) 2022-06-29T17:09:08.399 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.387+0000 7f63dceb4700 1 -- 172.21.15.60:0/512932410 learned_addr learned my addr 172.21.15.60:0/512932410 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T17:09:08.400 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.387+0000 7f63dd6b5700 1 --2- 172.21.15.60:0/512932410 >> v2:172.21.15.92:3300/0 conn(0x7f63d812bbc0 0x7f63d81431f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:08.401 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.387+0000 7f63dceb4700 1 -- 172.21.15.60:0/512932410 >> v2:172.21.15.92:3300/0 conn(0x7f63d812bbc0 msgr2=0x7f63d81431f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T17:09:08.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.387+0000 7f63dceb4700 1 --2- 172.21.15.60:0/512932410 >> v2:172.21.15.92:3300/0 conn(0x7f63d812bbc0 0x7f63d81431f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:08.402 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.387+0000 7f63dceb4700 1 -- 172.21.15.60:0/512932410 >> v1:172.21.15.92:6789/0 conn(0x7f63d8066b20 legacy=0x7f63d8142ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:08.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.387+0000 7f63dceb4700 1 -- 172.21.15.60:0/512932410 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d812c050 con 0x7f63d805d070 2022-06-29T17:09:08.403 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.387+0000 7f63dceb4700 1 --2- 172.21.15.60:0/512932410 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63d805d070 0x7f63d812b6d0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 rx=0x7f63d00032f0 tx=0x7f63d0009290).ready entity=mon.0 client_cookie=b70cd0c9f88e8158 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:08.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.387+0000 7f63d77fe700 1 -- 172.21.15.60:0/512932410 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63d000b870 con 0x7f63d805d070 2022-06-29T17:09:08.404 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.387+0000 7f63d77fe700 1 -- 172.21.15.60:0/512932410 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63d000b9d0 con 0x7f63d805d070 2022-06-29T17:09:08.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.387+0000 7f63d77fe700 1 -- 172.21.15.60:0/512932410 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63d000bb40 con 0x7f63d805d070 2022-06-29T17:09:08.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.391+0000 7f63ddeb6700 1 -- 172.21.15.60:0/512932410 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63d805d070 msgr2=0x7f63d812b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:08.405 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.391+0000 7f63ddeb6700 1 --2- 172.21.15.60:0/512932410 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63d805d070 0x7f63d812b6d0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 rx=0x7f63d00032f0 tx=0x7f63d0009290).stop 2022-06-29T17:09:08.406 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.391+0000 7f63ddeb6700 1 -- 172.21.15.60:0/512932410 shutdown_connections 2022-06-29T17:09:08.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.391+0000 7f63ddeb6700 1 --2- 172.21.15.60:0/512932410 >> v2:172.21.15.92:3300/0 conn(0x7f63d812bbc0 0x7f63d81431f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:08.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.391+0000 7f63ddeb6700 1 --2- 172.21.15.60:0/512932410 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63d805d070 0x7f63d812b6d0 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:08.407 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.391+0000 7f63ddeb6700 1 -- 172.21.15.60:0/512932410 >> 172.21.15.60:0/512932410 conn(0x7f63d80b5890 msgr2=0x7f63d80b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:08.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.391+0000 7f63ddeb6700 1 -- 172.21.15.60:0/512932410 shutdown_connections 2022-06-29T17:09:08.408 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.391+0000 7f63ddeb6700 1 -- 172.21.15.60:0/512932410 wait complete. 2022-06-29T17:09:08.412 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.391+0000 7f63ddeb6700 1 Processor -- start 2022-06-29T17:09:08.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.391+0000 7f63ddeb6700 1 -- start start 2022-06-29T17:09:08.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.391+0000 7f63ddeb6700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f63d805d070 0x7f63d812de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:08.413 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.391+0000 7f63ddeb6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63d8066b20 0x7f63d812e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:08.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.391+0000 7f63ddeb6700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f63d812bbc0 0x7f63d81553f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:08.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.391+0000 7f63ddeb6700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d81453c0 con 0x7f63d8066b20 2022-06-29T17:09:08.414 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.395+0000 7f63ddeb6700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d8131010 con 0x7f63d805d070 2022-06-29T17:09:08.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.395+0000 7f63ddeb6700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d812f010 con 0x7f63d812bbc0 2022-06-29T17:09:08.415 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.395+0000 7f63d7fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63d8066b20 0x7f63d812e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:08.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.395+0000 7f63d7fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63d8066b20 0x7f63d812e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55120/0 (socket says 172.21.15.60:55120) 2022-06-29T17:09:08.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.395+0000 7f63d7fff700 1 -- 172.21.15.60:0/3268682466 learned_addr learned my addr 172.21.15.60:0/3268682466 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T17:09:08.416 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.395+0000 7f63dceb4700 1 --2- 172.21.15.60:0/3268682466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f63d805d070 0x7f63d812de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:08.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.395+0000 7f63dd6b5700 1 --2- 172.21.15.60:0/3268682466 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f63d812bbc0 0x7f63d81553f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:08.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.395+0000 7f63d7fff700 1 -- 172.21.15.60:0/3268682466 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f63d812bbc0 msgr2=0x7f63d81553f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:08.417 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.395+0000 7f63d7fff700 1 --2- 172.21.15.60:0/3268682466 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f63d812bbc0 0x7f63d81553f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:08.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.395+0000 7f63d7fff700 1 -- 172.21.15.60:0/3268682466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f63d805d070 msgr2=0x7f63d812de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:08.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.395+0000 7f63d7fff700 1 --2- 172.21.15.60:0/3268682466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f63d805d070 0x7f63d812de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:08.418 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.395+0000 7f63d7fff700 1 -- 172.21.15.60:0/3268682466 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d0009530 con 0x7f63d8066b20 2022-06-29T17:09:08.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.395+0000 7f63d7fff700 1 --2- 172.21.15.60:0/3268682466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63d8066b20 0x7f63d812e7c0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 rx=0x7f63d805ff70 tx=0x7f63cc007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:08.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.395+0000 7f63d5ffb700 1 -- 172.21.15.60:0/3268682466 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63cc007030 con 0x7f63d8066b20 2022-06-29T17:09:08.419 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.395+0000 7f63d5ffb700 1 -- 172.21.15.60:0/3268682466 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63cc007190 con 0x7f63d8066b20 2022-06-29T17:09:08.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.395+0000 7f63ddeb6700 1 -- 172.21.15.60:0/3268682466 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d812ecb0 con 0x7f63d8066b20 2022-06-29T17:09:08.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.395+0000 7f63d5ffb700 1 -- 172.21.15.60:0/3268682466 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63cc006510 con 0x7f63d8066b20 2022-06-29T17:09:08.420 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.395+0000 7f63ddeb6700 1 -- 172.21.15.60:0/3268682466 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d8131440 con 0x7f63d8066b20 2022-06-29T17:09:08.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.395+0000 7f63ddeb6700 1 -- 172.21.15.60:0/3268682466 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63d80573d0 con 0x7f63d8066b20 2022-06-29T17:09:08.421 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.395+0000 7f63d5ffb700 1 -- 172.21.15.60:0/3268682466 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7f63cc010030 con 0x7f63d8066b20 2022-06-29T17:09:08.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.399+0000 7f63d5ffb700 1 --2- 172.21.15.60:0/3268682466 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f63b80539e0 0x7f63b8055ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:08.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.399+0000 7f63d5ffb700 1 -- 172.21.15.60:0/3268682466 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f63cc06b800 con 0x7f63d8066b20 2022-06-29T17:09:08.422 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.399+0000 7f63dceb4700 1 --2- 172.21.15.60:0/3268682466 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f63b80539e0 0x7f63b8055ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:08.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.399+0000 7f63dceb4700 1 --2- 172.21.15.60:0/3268682466 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f63b80539e0 0x7f63b8055ea0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0x7f63d001b000 tx=0x7f63d0016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:08.423 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.403+0000 7f63d5ffb700 1 -- 172.21.15.60:0/3268682466 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f63cc042a00 con 0x7f63d8066b20 2022-06-29T17:09:08.533 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.527+0000 7f63ddeb6700 1 -- 172.21.15.60:0/3268682466 --> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f63d81311a0 con 0x7f63b80539e0 2022-06-29T17:09:08.540 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.535+0000 7f63d5ffb700 1 -- 172.21.15.60:0/3268682466 <== mgr.4103 v2:172.21.15.92:6832/12872 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+68135 (secure 0 0 0) 0x7f63d000bbd0 con 0x7f63b80539e0 2022-06-29T17:09:08.541 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T17:09:08.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.535+0000 7f63ddeb6700 1 -- 172.21.15.60:0/3268682466 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f63b80539e0 msgr2=0x7f63b8055ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:08.543 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.535+0000 7f63ddeb6700 1 --2- 172.21.15.60:0/3268682466 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f63b80539e0 0x7f63b8055ea0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0x7f63d001b000 tx=0x7f63d0016000).stop 2022-06-29T17:09:08.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.535+0000 7f63ddeb6700 1 -- 172.21.15.60:0/3268682466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63d8066b20 msgr2=0x7f63d812e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:08.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.535+0000 7f63ddeb6700 1 --2- 172.21.15.60:0/3268682466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63d8066b20 0x7f63d812e7c0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 rx=0x7f63d805ff70 tx=0x7f63cc007c10).stop 2022-06-29T17:09:08.544 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.535+0000 7f63ddeb6700 1 -- 172.21.15.60:0/3268682466 shutdown_connections 2022-06-29T17:09:08.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.535+0000 7f63ddeb6700 1 --2- 172.21.15.60:0/3268682466 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7f63b80539e0 0x7f63b8055ea0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:08.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.535+0000 7f63ddeb6700 1 --2- 172.21.15.60:0/3268682466 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7f63d812bbc0 0x7f63d81553f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:08.545 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.535+0000 7f63ddeb6700 1 --2- 172.21.15.60:0/3268682466 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f63d8066b20 0x7f63d812e7c0 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:08.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.535+0000 7f63ddeb6700 1 --2- 172.21.15.60:0/3268682466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f63d805d070 0x7f63d812de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:08.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.535+0000 7f63ddeb6700 1 -- 172.21.15.60:0/3268682466 >> 172.21.15.60:0/3268682466 conn(0x7f63d80b5890 msgr2=0x7f63d812ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:08.546 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.535+0000 7f63ddeb6700 1 -- 172.21.15.60:0/3268682466 shutdown_connections 2022-06-29T17:09:08.547 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:08.535+0000 7f63ddeb6700 1 -- 172.21.15.60:0/3268682466 wait complete. 2022-06-29T17:09:08.547 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2022-06-29T17:09:08.554 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":2270,"stamp":"2022-06-29T17:09:08.089357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":6385024,"kb_used_data":743210016,"kb_used_omap":27916,"kb_used_meta":0,"kb_avail":743210016,"statfs":{"total":767585320960,"available":761047056384,"internally_reserved":0,"allocated":761047056384,"data_stored":761047056384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28586567,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003386"},"pg_stats":[{"pgid":"1.0","version":"14'2","reported_seq":146,"reported_epoch":149,"state":"active+clean","last_fresh":"2022-06-29T16:08:31.209595+0000","last_change":"2022-06-29T15:58:14.849417+0000","last_active":"2022-06-29T16:08:31.209595+0000","last_peered":"2022-06-29T16:08:31.209595+0000","last_clean":"2022-06-29T16:08:31.209595+0000","last_became_active":"2022-06-29T15:58:14.848454+0000","last_became_peered":"2022-06-29T15:58:14.848454+0000","last_unstale":"2022-06-29T16:08:31.209595+0000","last_undegraded":"2022-06-29T16:08:31.209595+0000","last_fullsized":"2022-06-29T16:08:31.209595+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:11.766061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:11.766061+0000","last_clean_scrub_stamp":"2022-06-29T15:58:11.766061+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":132,"reported_epoch":149,"state":"active+clean","last_fresh":"2022-06-29T16:08:31.831503+0000","last_change":"2022-06-29T15:58:21.003987+0000","last_active":"2022-06-29T16:08:31.831503+0000","last_peered":"2022-06-29T16:08:31.831503+0000","last_clean":"2022-06-29T16:08:31.831503+0000","last_became_active":"2022-06-29T15:58:21.003537+0000","last_became_peered":"2022-06-29T15:58:21.003537+0000","last_unstale":"2022-06-29T16:08:31.831503+0000","last_undegraded":"2022-06-29T16:08:31.831503+0000","last_fullsized":"2022-06-29T16:08:31.831503+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_clean_scrub_stamp":"2022-06-29T15:58:19.966881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":141,"reported_epoch":149,"state":"active+clean","last_fresh":"2022-06-29T16:08:31.209460+0000","last_change":"2022-06-29T15:58:21.001629+0000","last_active":"2022-06-29T16:08:31.209460+0000","last_peered":"2022-06-29T16:08:31.209460+0000","last_clean":"2022-06-29T16:08:31.209460+0000","last_became_active":"2022-06-29T15:58:21.000885+0000","last_became_peered":"2022-06-29T15:58:21.000885+0000","last_unstale":"2022-06-29T16:08:31.209460+0000","last_undegraded":"2022-06-29T16:08:31.209460+0000","last_fullsized":"2022-06-29T16:08:31.209460+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_clean_scrub_stamp":"2022-06-29T15:58:19.966881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":136,"reported_epoch":149,"state":"active+clean","last_fresh":"2022-06-29T16:08:31.832454+0000","last_change":"2022-06-29T15:58:20.997373+0000","last_active":"2022-06-29T16:08:31.832454+0000","last_peered":"2022-06-29T16:08:31.832454+0000","last_clean":"2022-06-29T16:08:31.832454+0000","last_became_active":"2022-06-29T15:58:20.997032+0000","last_became_peered":"2022-06-29T15:58:20.997032+0000","last_unstale":"2022-06-29T16:08:31.832454+0000","last_undegraded":"2022-06-29T16:08:31.832454+0000","last_fullsized":"2022-06-29T16:08:31.832454+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_clean_scrub_stamp":"2022-06-29T15:58:19.966881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":132,"reported_epoch":149,"state":"active+clean","last_fresh":"2022-06-29T16:08:31.831202+0000","last_change":"2022-06-29T15:58:21.005059+0000","last_active":"2022-06-29T16:08:31.831202+0000","last_peered":"2022-06-29T16:08:31.831202+0000","last_clean":"2022-06-29T16:08:31.831202+0000","last_became_active":"2022-06-29T15:58:21.004623+0000","last_became_peered":"2022-06-29T15:58:21.004623+0000","last_unstale":"2022-06-29T16:08:31.831202+0000","last_undegraded":"2022-06-29T16:08:31.831202+0000","last_fullsized":"2022-06-29T16:08:31.831202+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_clean_scrub_stamp":"2022-06-29T15:58:19.966881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":131,"reported_epoch":149,"state":"active+clean","last_fresh":"2022-06-29T16:08:31.831388+0000","last_change":"2022-06-29T15:58:21.801445+0000","last_active":"2022-06-29T16:08:31.831388+0000","last_peered":"2022-06-29T16:08:31.831388+0000","last_clean":"2022-06-29T16:08:31.831388+0000","last_became_active":"2022-06-29T15:58:21.800792+0000","last_became_peered":"2022-06-29T15:58:21.800792+0000","last_unstale":"2022-06-29T16:08:31.831388+0000","last_undegraded":"2022-06-29T16:08:31.831388+0000","last_fullsized":"2022-06-29T16:08:31.831388+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_clean_scrub_stamp":"2022-06-29T15:58:19.966881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":141,"reported_epoch":149,"state":"active+clean","last_fresh":"2022-06-29T16:08:31.209798+0000","last_change":"2022-06-29T15:58:21.807952+0000","last_active":"2022-06-29T16:08:31.209798+0000","last_peered":"2022-06-29T16:08:31.209798+0000","last_clean":"2022-06-29T16:08:31.209798+0000","last_became_active":"2022-06-29T15:58:21.807346+0000","last_became_peered":"2022-06-29T15:58:21.807346+0000","last_unstale":"2022-06-29T16:08:31.209798+0000","last_undegraded":"2022-06-29T16:08:31.209798+0000","last_fullsized":"2022-06-29T16:08:31.209798+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_clean_scrub_stamp":"2022-06-29T15:58:19.966881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":131,"reported_epoch":149,"state":"active+clean","last_fresh":"2022-06-29T16:08:31.831563+0000","last_change":"2022-06-29T15:58:20.996366+0000","last_active":"2022-06-29T16:08:31.831563+0000","last_peered":"2022-06-29T16:08:31.831563+0000","last_clean":"2022-06-29T16:08:31.831563+0000","last_became_active":"2022-06-29T15:58:20.995883+0000","last_became_peered":"2022-06-29T15:58:20.995883+0000","last_unstale":"2022-06-29T16:08:31.831563+0000","last_undegraded":"2022-06-29T16:08:31.831563+0000","last_fullsized":"2022-06-29T16:08:31.831563+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_clean_scrub_stamp":"2022-06-29T15:58:19.966881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":131,"reported_epoch":149,"state":"active+clean","last_fresh":"2022-06-29T16:08:31.833014+0000","last_change":"2022-06-29T15:58:21.003020+0000","last_active":"2022-06-29T16:08:31.833014+0000","last_peered":"2022-06-29T16:08:31.833014+0000","last_clean":"2022-06-29T16:08:31.833014+0000","last_became_active":"2022-06-29T15:58:21.002353+0000","last_became_peered":"2022-06-29T15:58:21.002353+0000","last_unstale":"2022-06-29T16:08:31.833014+0000","last_undegraded":"2022-06-29T16:08:31.833014+0000","last_fullsized":"2022-06-29T16:08:31.833014+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T15:58:19.966881+0000","last_clean_scrub_stamp":"2022-06-29T15:58:19.966881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575701,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798144,"kb_used_data":92901236,"kb_used_omap":3659,"kb_used_meta":0,"kb_avail":92901236,"statfs":{"total":95948165120,"available":95130865664,"internally_reserved":0,"allocated":95130865664,"data_stored":95130865664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3747326,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jun 29 17:08:21 2022","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.83699999999999997,"15min":0.82699999999999996},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.5},"max":{"1min":1.0640000000000001,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.81599999999999995,"15min":0.81499999999999995},"min":{"1min":0.61599999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.95199999999999996,"5min":1.3120000000000001,"15min":1.377},"last":0.77800000000000002}]},{"osd":1,"last update":"Wed Jun 29 17:08:21 2022","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.82399999999999995,"15min":0.83299999999999996},"min":{"1min":0.64900000000000002,"5min":0.57199999999999995,"15min":0.56599999999999995},"max":{"1min":1.125,"5min":1.125,"15min":1.1890000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.83399999999999996,"15min":0.84899999999999998},"min":{"1min":0.623,"5min":0.60599999999999998,"15min":0.59399999999999997},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1659999999999999},"last":0.74399999999999999}]},{"osd":2,"last update":"Wed Jun 29 17:08:21 2022","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.871,"15min":0.872},"min":{"1min":0.625,"5min":0.625,"15min":0.57099999999999995},"max":{"1min":1.202,"5min":1.202,"15min":1.244},"last":0.626},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.85299999999999998,"15min":0.85099999999999998},"min":{"1min":0.61899999999999999,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.097,"5min":1.371,"15min":1.371},"last":0.94399999999999995}]},{"osd":3,"last update":"Wed Jun 29 17:08:21 2022","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.84299999999999997,"15min":0.84699999999999998},"min":{"1min":0.64000000000000001,"5min":0.60499999999999998,"15min":0.48999999999999999},"max":{"1min":1.0600000000000001,"5min":1.204,"15min":1.3480000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.92300000000000004,"15min":0.91500000000000004},"min":{"1min":0.69399999999999995,"5min":0.69099999999999995,"15min":0.62},"max":{"1min":1.077,"5min":1.2549999999999999,"15min":1.4399999999999999},"last":0.92200000000000004}]},{"osd":4,"last update":"Wed Jun 29 17:08:21 2022","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.79800000000000004,"15min":0.79200000000000004},"min":{"1min":0.58999999999999997,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0189999999999999,"5min":1.321,"15min":1.4890000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.86399999999999999,"15min":0.84699999999999998},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.55300000000000005},"max":{"1min":1.02,"5min":1.284,"15min":1.4239999999999999},"last":0.88400000000000001}]},{"osd":5,"last update":"Wed Jun 29 17:08:21 2022","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.91100000000000003,"15min":0.90900000000000003},"min":{"1min":0.68899999999999995,"5min":0.58299999999999996,"15min":0.55900000000000005},"max":{"1min":1.139,"5min":1.2370000000000001,"15min":1.3149999999999999},"last":0.96899999999999997},{"interface":"front","average":{"1min":1.0109999999999999,"5min":0.98799999999999999,"15min":0.98899999999999999},"min":{"1min":0.90200000000000002,"5min":0.621,"15min":0.621},"max":{"1min":1.109,"5min":1.46,"15min":1.585},"last":0.86599999999999999}]},{"osd":6,"last update":"Wed Jun 29 17:08:21 2022","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.93600000000000005,"15min":0.94999999999999996},"min":{"1min":0.69699999999999995,"5min":0.65500000000000003,"15min":0.63600000000000001},"max":{"1min":1.1839999999999999,"5min":1.4139999999999999,"15min":1.5449999999999999},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.94199999999999995},"min":{"1min":0.67100000000000004,"5min":0.64200000000000002,"15min":0.61199999999999999},"max":{"1min":1.109,"5min":1.353,"15min":1.55},"last":0.72399999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575702,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798144,"kb_used_data":92901236,"kb_used_omap":4008,"kb_used_meta":0,"kb_avail":92901236,"statfs":{"total":95948165120,"available":95130865664,"internally_reserved":0,"allocated":95130865664,"data_stored":95130865664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4104844,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jun 29 17:08:41 2022","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.76800000000000002,"15min":0.75800000000000001},"min":{"1min":0.59899999999999998,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.159},"last":0.879},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.78200000000000003,"15min":0.76800000000000002},"min":{"1min":0.55300000000000005,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.089,"5min":1.089,"15min":1.177},"last":0.80800000000000005}]},{"osd":1,"last update":"Wed Jun 29 17:08:41 2022","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.86499999999999999,"15min":0.85299999999999998},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.0740000000000001,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":1.008},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.90900000000000003,"15min":0.91200000000000003},"min":{"1min":0.621,"5min":0.621,"15min":0.58199999999999996},"max":{"1min":1.0469999999999999,"5min":1.1399999999999999,"15min":1.3220000000000001},"last":0.93500000000000005}]},{"osd":2,"last update":"Wed Jun 29 17:08:41 2022","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.88300000000000001,"15min":0.85799999999999998},"min":{"1min":0.64700000000000002,"5min":0.60999999999999999,"15min":0.55500000000000005},"max":{"1min":1.0389999999999999,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.877,"5min":0.88900000000000001,"15min":0.89800000000000002},"min":{"1min":0.59199999999999997,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.026,"5min":1.1950000000000001,"15min":1.2090000000000001},"last":0.84499999999999997}]},{"osd":3,"last update":"Wed Jun 29 17:08:41 2022","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.82899999999999996,"15min":0.83199999999999996},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.51700000000000002},"max":{"1min":1.024,"5min":1.1719999999999999,"15min":1.4139999999999999},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.82699999999999996,"15min":0.82999999999999996},"min":{"1min":0.628,"5min":0.60999999999999999,"15min":0.60799999999999998},"max":{"1min":1.032,"5min":1.0880000000000001,"15min":1.161},"last":1.024}]},{"osd":4,"last update":"Wed Jun 29 17:08:41 2022","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.77300000000000002,"15min":0.80100000000000005},"min":{"1min":0.52800000000000002,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.0609999999999999,"5min":1.1799999999999999,"15min":1.3180000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.92900000000000005,"15min":0.93300000000000005},"min":{"1min":0.57499999999999996,"5min":0.53000000000000003,"15min":0.52700000000000002},"max":{"1min":1.1080000000000001,"5min":1.2270000000000001,"15min":1.4850000000000001},"last":0.98799999999999999}]},{"osd":5,"last update":"Wed Jun 29 17:08:41 2022","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.85399999999999998,"15min":0.83999999999999997},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.48999999999999999},"max":{"1min":1.175,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":1.087},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.94899999999999995,"15min":0.98399999999999999},"min":{"1min":0.64300000000000002,"5min":0.59699999999999998,"15min":0.55100000000000005},"max":{"1min":1.121,"5min":1.2330000000000001,"15min":1.4079999999999999},"last":0.94799999999999995}]},{"osd":7,"last update":"Wed Jun 29 17:08:41 2022","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.92100000000000004,"15min":0.93300000000000005},"min":{"1min":0.60999999999999999,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.0269999999999999,"5min":1.3700000000000001,"15min":1.532},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.96999999999999997,"5min":0.995,"15min":1.0029999999999999},"min":{"1min":0.80900000000000005,"5min":0.66300000000000003,"15min":0.60199999999999998},"max":{"1min":1.2350000000000001,"5min":1.2430000000000001,"15min":1.319},"last":0.89600000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575700,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798144,"kb_used_data":92901236,"kb_used_omap":2743,"kb_used_meta":0,"kb_avail":92901236,"statfs":{"total":95948165120,"available":95130865664,"internally_reserved":0,"allocated":95130865664,"data_stored":95130865664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2808996,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jun 29 17:08:27 2022","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.84999999999999998,"15min":0.85399999999999998},"min":{"1min":0.68600000000000005,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.1160000000000001,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.93600000000000005,"15min":0.93500000000000005},"min":{"1min":0.76800000000000002,"5min":0.623,"15min":0.52500000000000002},"max":{"1min":1.0860000000000001,"5min":1.167,"15min":1.286},"last":0.82199999999999995}]},{"osd":1,"last update":"Wed Jun 29 17:08:27 2022","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.84099999999999997,"15min":0.83199999999999996},"min":{"1min":0.66100000000000003,"5min":0.57099999999999995,"15min":0.55100000000000005},"max":{"1min":1.0700000000000001,"5min":1.1299999999999999,"15min":1.339},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.84099999999999997,"15min":0.84599999999999997},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.22},"last":0.79400000000000004}]},{"osd":2,"last update":"Wed Jun 29 17:08:27 2022","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.86499999999999999,"15min":0.86599999999999999},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1910000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.60399999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0269999999999999,"5min":1.0940000000000001,"15min":1.131},"last":0.92900000000000005}]},{"osd":3,"last update":"Wed Jun 29 17:08:27 2022","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.86599999999999999,"15min":0.86099999999999999},"min":{"1min":0.66800000000000004,"5min":0.64300000000000002,"15min":0.59399999999999997},"max":{"1min":1.131,"5min":1.131,"15min":1.379},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.94599999999999995,"15min":0.92100000000000004},"min":{"1min":0.78600000000000003,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.89800000000000002}]},{"osd":4,"last update":"Wed Jun 29 17:08:27 2022","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.78800000000000003,"15min":0.78100000000000003},"min":{"1min":0.54100000000000004,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.0509999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":1.006,"5min":0.995,"15min":0.995},"min":{"1min":0.82599999999999996,"5min":0.71799999999999997,"15min":0.67500000000000004},"max":{"1min":1.258,"5min":1.258,"15min":1.4179999999999999},"last":0.95299999999999996}]},{"osd":6,"last update":"Wed Jun 29 17:08:27 2022","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.90200000000000002,"15min":0.876},"min":{"1min":0.58099999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.171,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":1.0660000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.90200000000000002,"15min":0.88900000000000001},"min":{"1min":0.73099999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0600000000000001,"5min":1.4159999999999999,"15min":1.4159999999999999},"last":1.0169999999999999}]},{"osd":7,"last update":"Wed Jun 29 17:08:27 2022","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.94499999999999995},"min":{"1min":0.64500000000000002,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.212,"5min":1.3959999999999999,"15min":1.4570000000000001},"last":0.88},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.0469999999999999,"15min":1.0069999999999999},"min":{"1min":0.85599999999999998,"5min":0.64100000000000001,"15min":0.63400000000000001},"max":{"1min":1.3300000000000001,"5min":1.6850000000000001,"15min":1.6950000000000001},"last":0.99099999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575702,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798016,"kb_used_data":92901364,"kb_used_omap":3569,"kb_used_meta":0,"kb_avail":92901364,"statfs":{"total":95948165120,"available":95130996736,"internally_reserved":0,"allocated":95130996736,"data_stored":95130996736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3654826,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jun 29 17:08:23 2022","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.79500000000000004,"15min":0.79600000000000004},"min":{"1min":0.57299999999999995,"5min":0.55300000000000005,"15min":0.53100000000000003},"max":{"1min":1.083,"5min":1.117,"15min":1.1579999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.79400000000000004,"15min":0.79700000000000004},"min":{"1min":0.61099999999999999,"5min":0.58199999999999996,"15min":0.52400000000000002},"max":{"1min":0.94499999999999995,"5min":1.135,"15min":1.1739999999999999},"last":0.745}]},{"osd":1,"last update":"Wed Jun 29 17:08:23 2022","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.84099999999999997,"15min":0.82699999999999996},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.57299999999999995},"max":{"1min":1.024,"5min":1.0800000000000001,"15min":1.1359999999999999},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81399999999999995,"15min":0.82599999999999996},"min":{"1min":0.63500000000000001,"5min":0.59899999999999998,"15min":0.57999999999999996},"max":{"1min":0.96699999999999997,"5min":1.0940000000000001,"15min":1.1930000000000001},"last":0.88700000000000001}]},{"osd":2,"last update":"Wed Jun 29 17:08:23 2022","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.90800000000000003,"15min":0.91700000000000004},"min":{"1min":0.625,"5min":0.59699999999999998,"15min":0.57699999999999996},"max":{"1min":1.0840000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.92100000000000004,"15min":0.90800000000000003},"min":{"1min":0.67200000000000004,"5min":0.61399999999999999,"15min":0.60199999999999998},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.23},"last":1.0129999999999999}]},{"osd":3,"last update":"Wed Jun 29 17:08:23 2022","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.81699999999999995,"15min":0.81100000000000005},"min":{"1min":0.64600000000000002,"5min":0.628,"15min":0.56799999999999995},"max":{"1min":0.995,"5min":1.212,"15min":1.212},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.89400000000000002,"15min":0.90000000000000002},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.60799999999999998},"max":{"1min":1.0549999999999999,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":1.042}]},{"osd":5,"last update":"Wed Jun 29 17:08:23 2022","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.85099999999999998,"15min":0.84299999999999997},"min":{"1min":0.54300000000000004,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.101,"5min":1.133,"15min":1.2450000000000001},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.91200000000000003,"15min":0.90200000000000002},"min":{"1min":0.61199999999999999,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.135,"5min":1.1819999999999999,"15min":1.248},"last":0.98699999999999999}]},{"osd":6,"last update":"Wed Jun 29 17:08:23 2022","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.88600000000000001,"15min":0.88500000000000001},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.54600000000000004},"max":{"1min":1.083,"5min":1.417,"15min":1.417},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.98899999999999999,"15min":0.97399999999999998},"min":{"1min":0.67400000000000004,"5min":0.60099999999999998,"15min":0.53200000000000003},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.2889999999999999},"last":1.0669999999999999}]},{"osd":7,"last update":"Wed Jun 29 17:08:23 2022","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.88600000000000001,"15min":0.874},"min":{"1min":0.61599999999999999,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.1140000000000001,"5min":1.4319999999999999,"15min":1.4950000000000001},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.97599999999999998,"15min":0.95699999999999996},"min":{"1min":0.72699999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.1519999999999999,"5min":1.466,"15min":1.488},"last":1.101}]}]},{"osd":3,"up_from":13,"seq":55834575701,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798144,"kb_used_data":92901236,"kb_used_omap":3345,"kb_used_meta":0,"kb_avail":92901236,"statfs":{"total":95948165120,"available":95130865664,"internally_reserved":0,"allocated":95130865664,"data_stored":95130865664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3425940,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jun 29 17:08:18 2022","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.63700000000000001,"15min":0.63300000000000001},"min":{"1min":0.47199999999999998,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.95599999999999996,"5min":1.087,"15min":1.1259999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.73499999999999999,"15min":0.74099999999999999},"min":{"1min":0.63100000000000001,"5min":0.57999999999999996,"15min":0.56799999999999995},"max":{"1min":0.93400000000000005,"5min":0.96999999999999997,"15min":1.052},"last":0.67900000000000005}]},{"osd":1,"last update":"Wed Jun 29 17:08:18 2022","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.67100000000000004,"15min":0.67200000000000004},"min":{"1min":0.51100000000000001,"5min":0.49199999999999999,"15min":0.44700000000000001},"max":{"1min":0.876,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.871,"5min":0.84499999999999997,"15min":0.85199999999999998},"min":{"1min":0.626,"5min":0.58499999999999996,"15min":0.53000000000000003},"max":{"1min":1.155,"5min":1.155,"15min":1.159},"last":0.81499999999999995}]},{"osd":2,"last update":"Wed Jun 29 17:08:18 2022","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.77100000000000002,"15min":0.77700000000000002},"min":{"1min":0.56000000000000005,"5min":0.54200000000000004,"15min":0.54100000000000004},"max":{"1min":1.042,"5min":1.194,"15min":1.198},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.71599999999999997,"15min":0.70899999999999996},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.498},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.56999999999999995}]},{"osd":4,"last update":"Wed Jun 29 17:08:18 2022","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.97299999999999998,"15min":0.97899999999999998},"min":{"1min":0.73899999999999999,"5min":0.72099999999999997,"15min":0.64700000000000002},"max":{"1min":1.2030000000000001,"5min":1.2210000000000001,"15min":1.23},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.90600000000000003,"15min":0.90700000000000003},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.621},"max":{"1min":1.256,"5min":1.256,"15min":1.4490000000000001},"last":0.88300000000000001}]},{"osd":5,"last update":"Wed Jun 29 17:08:18 2022","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.876,"15min":0.89600000000000002},"min":{"1min":0.67600000000000005,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.0169999999999999,"5min":1.2849999999999999,"15min":1.2889999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":1.038,"5min":1.02,"15min":1.04},"min":{"1min":0.87,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.284,"5min":1.45,"15min":1.5309999999999999},"last":1.038}]},{"osd":6,"last update":"Wed Jun 29 17:08:18 2022","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.96199999999999997,"15min":0.97899999999999998},"min":{"1min":0.77300000000000002,"5min":0.74399999999999999,"15min":0.72499999999999998},"max":{"1min":1.304,"5min":1.3460000000000001,"15min":1.542},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.998,"5min":1.0149999999999999,"15min":1.0249999999999999},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.75800000000000001},"max":{"1min":1.1120000000000001,"5min":1.345,"15min":1.5289999999999999},"last":0.89600000000000002}]},{"osd":7,"last update":"Wed Jun 29 17:08:18 2022","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.0860000000000001,"15min":1.0880000000000001},"min":{"1min":0.84499999999999997,"5min":0.77600000000000002,"15min":0.69399999999999995},"max":{"1min":1.323,"5min":1.4730000000000001,"15min":1.5920000000000001},"last":0.999},{"interface":"front","average":{"1min":1.016,"5min":1.0449999999999999,"15min":1.0569999999999999},"min":{"1min":0.77200000000000002,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.206,"5min":1.395,"15min":1.47},"last":0.96199999999999997}]}]},{"osd":2,"up_from":13,"seq":55834575702,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798144,"kb_used_data":92901236,"kb_used_omap":2825,"kb_used_meta":0,"kb_avail":92901236,"statfs":{"total":95948165120,"available":95130865664,"internally_reserved":0,"allocated":95130865664,"data_stored":95130865664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2893662,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jun 29 17:08:34 2022","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61899999999999999,"15min":0.62},"min":{"1min":0.48899999999999999,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.86899999999999999,"5min":1.167,"15min":1.3080000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.70999999999999996,"15min":0.71099999999999997},"min":{"1min":0.51600000000000001,"5min":0.48699999999999999,"15min":0.48499999999999999},"max":{"1min":1.1699999999999999,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.627}]},{"osd":1,"last update":"Wed Jun 29 17:08:34 2022","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66000000000000003,"15min":0.65200000000000002},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.46999999999999997},"max":{"1min":0.78300000000000003,"5min":1.6799999999999999,"15min":1.6799999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.745,"15min":0.73399999999999999},"min":{"1min":0.56299999999999994,"5min":0.55300000000000005,"15min":0.52900000000000003},"max":{"1min":0.96599999999999997,"5min":1.1899999999999999,"15min":1.206},"last":0.74399999999999999}]},{"osd":3,"last update":"Wed Jun 29 17:08:34 2022","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.85499999999999998,"15min":0.84599999999999997},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.55500000000000005},"max":{"1min":0.999,"5min":1.732,"15min":1.732},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.81000000000000005,"15min":0.80600000000000005},"min":{"1min":0.60899999999999999,"5min":0.58399999999999996,"15min":0.51600000000000001},"max":{"1min":1.0469999999999999,"5min":1.0649999999999999,"15min":1.1599999999999999},"last":0.81100000000000005}]},{"osd":4,"last update":"Wed Jun 29 17:08:34 2022","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.92900000000000005,"15min":0.92600000000000005},"min":{"1min":0.76300000000000001,"5min":0.71399999999999997,"15min":0.60899999999999999},"max":{"1min":1.2729999999999999,"5min":1.2749999999999999,"15min":1.421},"last":1.0660000000000001},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.95599999999999996,"15min":0.94599999999999995},"min":{"1min":0.73499999999999999,"5min":0.72299999999999998,"15min":0.66700000000000004},"max":{"1min":1.302,"5min":1.772,"15min":1.772},"last":0.92200000000000004}]},{"osd":5,"last update":"Wed Jun 29 17:08:34 2022","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":0.96899999999999997,"15min":0.96799999999999997},"min":{"1min":0.73299999999999998,"5min":0.68100000000000005,"15min":0.67000000000000004},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.4350000000000001},"last":1.006},{"interface":"front","average":{"1min":0.997,"5min":0.997,"15min":0.98999999999999999},"min":{"1min":0.79300000000000004,"5min":0.72499999999999998,"15min":0.68999999999999995},"max":{"1min":1.2689999999999999,"5min":1.415,"15min":1.4930000000000001},"last":0.83499999999999996}]},{"osd":6,"last update":"Wed Jun 29 17:08:34 2022","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.0489999999999999,"15min":1.024},"min":{"1min":0.84399999999999997,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.397,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":0.874},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.0640000000000001,"15min":1.0600000000000001},"min":{"1min":0.81200000000000006,"5min":0.752,"15min":0.745},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.5549999999999999},"last":1.0920000000000001}]},{"osd":7,"last update":"Wed Jun 29 17:08:34 2022","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0600000000000001,"15min":1.0620000000000001},"min":{"1min":0.92200000000000004,"5min":0.75700000000000001,"15min":0.70299999999999996},"max":{"1min":1.3640000000000001,"5min":1.54,"15min":1.54},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.125,"15min":1.1000000000000001},"min":{"1min":0.98399999999999999,"5min":0.84299999999999997,"15min":0.76800000000000002},"max":{"1min":1.4550000000000001,"5min":1.845,"15min":1.845},"last":1.0389999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575700,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798144,"kb_used_data":92901236,"kb_used_omap":3969,"kb_used_meta":0,"kb_avail":92901236,"statfs":{"total":95948165120,"available":95130865664,"internally_reserved":0,"allocated":95130865664,"data_stored":95130865664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4064788,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jun 29 17:08:07 2022","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59899999999999998,"15min":0.61799999999999999},"min":{"1min":0.433,"5min":0.432,"15min":0.432},"max":{"1min":0.87,"5min":0.93100000000000005,"15min":1.04},"last":1.0209999999999999},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.72199999999999998,"15min":0.70999999999999996},"min":{"1min":0.501,"5min":0.501,"15min":0.47699999999999998},"max":{"1min":0.91000000000000003,"5min":1.278,"15min":1.278},"last":0.621}]},{"osd":2,"last update":"Wed Jun 29 17:08:07 2022","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.64200000000000002,"15min":0.66200000000000003},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.0389999999999999,"5min":1.042,"15min":1.569},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.58899999999999997,"5min":0.52000000000000002,"15min":0.48299999999999998},"max":{"1min":1.079,"5min":1.111,"15min":1.111},"last":0.82099999999999995}]},{"osd":3,"last update":"Wed Jun 29 17:08:07 2022","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.89200000000000002,"15min":0.89600000000000002},"min":{"1min":0.70299999999999996,"5min":0.629,"15min":0.51800000000000002},"max":{"1min":1.26,"5min":1.26,"15min":1.425},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.83599999999999997,"15min":0.83399999999999996},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.0629999999999999,"5min":1.1539999999999999,"15min":1.4350000000000001},"last":0.86099999999999999}]},{"osd":4,"last update":"Wed Jun 29 17:08:07 2022","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.88200000000000001,"15min":0.89000000000000001},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.2190000000000001,"5min":1.2310000000000001,"15min":1.395},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97599999999999998,"15min":0.96999999999999997},"min":{"1min":0.72099999999999997,"5min":0.69299999999999995,"15min":0.65200000000000002},"max":{"1min":1.387,"5min":1.663,"15min":1.663},"last":1.1899999999999999}]},{"osd":5,"last update":"Wed Jun 29 17:08:07 2022","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.95299999999999996,"15min":0.96999999999999997},"min":{"1min":0.71499999999999997,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.343,"5min":1.4099999999999999,"15min":1.637},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.96099999999999997,"15min":0.98899999999999999},"min":{"1min":0.68400000000000005,"5min":0.68300000000000005,"15min":0.68200000000000005},"max":{"1min":1.3180000000000001,"5min":1.369,"15min":1.381},"last":0.96599999999999997}]},{"osd":6,"last update":"Wed Jun 29 17:08:07 2022","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.046,"15min":1.0489999999999999},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.69399999999999995},"max":{"1min":1.617,"5min":1.73,"15min":1.73},"last":1.3149999999999999},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.097,"15min":1.093},"min":{"1min":0.81899999999999995,"5min":0.81399999999999995,"15min":0.71699999999999997},"max":{"1min":1.3959999999999999,"5min":1.635,"15min":1.7470000000000001},"last":1.0660000000000001}]},{"osd":7,"last update":"Wed Jun 29 17:08:07 2022","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.0980000000000001,"15min":1.1060000000000001},"min":{"1min":0.78600000000000003,"5min":0.73499999999999999,"15min":0.69099999999999995},"max":{"1min":1.454,"5min":1.5589999999999999,"15min":1.704},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.0580000000000001,"15min":1.093},"min":{"1min":0.73699999999999999,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.516,"5min":1.516,"15min":1.516},"last":1.0920000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575701,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798144,"kb_used_data":92901236,"kb_used_omap":3795,"kb_used_meta":0,"kb_avail":92901236,"statfs":{"total":95948165120,"available":95130865664,"internally_reserved":0,"allocated":95130865664,"data_stored":95130865664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3886185,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Jun 29 17:08:25 2022","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.61699999999999999,"15min":0.60399999999999998},"min":{"1min":0.46800000000000003,"5min":0.45200000000000001,"15min":0.45100000000000001},"max":{"1min":1.1579999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.68799999999999994,"15min":0.66800000000000004},"min":{"1min":0.53000000000000003,"5min":0.48799999999999999,"15min":0.48399999999999999},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.73199999999999998}]},{"osd":2,"last update":"Wed Jun 29 17:08:25 2022","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81499999999999995,"15min":0.80500000000000005},"min":{"1min":0.60799999999999998,"5min":0.53200000000000003,"15min":0.5},"max":{"1min":1.02,"5min":1.48,"15min":1.48},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.71199999999999997,"15min":0.72099999999999997},"min":{"1min":0.505,"5min":0.505,"15min":0.46400000000000002},"max":{"1min":1.02,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":0.56000000000000005}]},{"osd":3,"last update":"Wed Jun 29 17:08:25 2022","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.751,"15min":0.746},"min":{"1min":0.56499999999999995,"5min":0.52000000000000002,"15min":0.5},"max":{"1min":1.008,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.877,"15min":0.877},"min":{"1min":0.72799999999999998,"5min":0.66000000000000003,"15min":0.64300000000000002},"max":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.333},"last":0.82799999999999996}]},{"osd":4,"last update":"Wed Jun 29 17:08:25 2022","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.86699999999999999,"15min":0.86399999999999999},"min":{"1min":0.66300000000000003,"5min":0.628,"15min":0.628},"max":{"1min":1.0349999999999999,"5min":1.173,"15min":1.3560000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.94399999999999995,"15min":0.92300000000000004},"min":{"1min":0.751,"5min":0.70599999999999996,"15min":0.66000000000000003},"max":{"1min":1.1830000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":0.878}]},{"osd":5,"last update":"Wed Jun 29 17:08:25 2022","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.94399999999999995,"15min":0.95299999999999996},"min":{"1min":0.68899999999999995,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.252,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.92700000000000005},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.004,"15min":0.98799999999999999},"min":{"1min":0.83699999999999997,"5min":0.70699999999999996,"15min":0.68300000000000005},"max":{"1min":1.3340000000000001,"5min":1.3999999999999999,"15min":1.6140000000000001},"last":0.93899999999999995}]},{"osd":6,"last update":"Wed Jun 29 17:08:25 2022","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.0600000000000001,"15min":1.0329999999999999},"min":{"1min":0.872,"5min":0.70499999999999996,"15min":0.68100000000000005},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.554},"last":0.86099999999999999},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0069999999999999,"15min":1.0109999999999999},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.3180000000000001,"5min":1.361,"15min":1.48},"last":0.90200000000000002}]},{"osd":7,"last update":"Wed Jun 29 17:08:25 2022","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.0409999999999999,"15min":1.0229999999999999},"min":{"1min":0.84799999999999998,"5min":0.77200000000000002,"15min":0.71699999999999997},"max":{"1min":1.522,"5min":1.522,"15min":1.5429999999999999},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.083,"15min":1.075},"min":{"1min":0.88500000000000001,"5min":0.81000000000000005,"15min":0.67700000000000005},"max":{"1min":1.3140000000000001,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.97799999999999998}]}]}]}} 2022-06-29T17:09:08.562 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2022-06-29T15:58:11.766061+0000 time.struct_time(tm_year=2022, tm_mon=6, tm_mday=29, tm_hour=15, tm_min=58, tm_sec=11, tm_wday=2, tm_yday=180, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=6, tm_mday=29, tm_hour=17, tm_min=9, tm_sec=3, tm_wday=2, tm_yday=180, tm_isdst=0) 2022-06-29T17:09:08.562 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2022-06-29T15:58:19.966881+0000 time.struct_time(tm_year=2022, tm_mon=6, tm_mday=29, tm_hour=15, tm_min=58, tm_sec=19, tm_wday=2, tm_yday=180, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=6, tm_mday=29, tm_hour=17, tm_min=9, tm_sec=3, tm_wday=2, tm_yday=180, tm_isdst=0) 2022-06-29T17:09:08.563 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2022-06-29T15:58:19.966881+0000 time.struct_time(tm_year=2022, tm_mon=6, tm_mday=29, tm_hour=15, tm_min=58, tm_sec=19, tm_wday=2, tm_yday=180, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=6, tm_mday=29, tm_hour=17, tm_min=9, tm_sec=3, tm_wday=2, tm_yday=180, tm_isdst=0) 2022-06-29T17:09:08.564 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2022-06-29T15:58:19.966881+0000 time.struct_time(tm_year=2022, tm_mon=6, tm_mday=29, tm_hour=15, tm_min=58, tm_sec=19, tm_wday=2, tm_yday=180, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=6, tm_mday=29, tm_hour=17, tm_min=9, tm_sec=3, tm_wday=2, tm_yday=180, tm_isdst=0) 2022-06-29T17:09:08.564 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2022-06-29T15:58:19.966881+0000 time.struct_time(tm_year=2022, tm_mon=6, tm_mday=29, tm_hour=15, tm_min=58, tm_sec=19, tm_wday=2, tm_yday=180, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=6, tm_mday=29, tm_hour=17, tm_min=9, tm_sec=3, tm_wday=2, tm_yday=180, tm_isdst=0) 2022-06-29T17:09:08.565 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2022-06-29T15:58:19.966881+0000 time.struct_time(tm_year=2022, tm_mon=6, tm_mday=29, tm_hour=15, tm_min=58, tm_sec=19, tm_wday=2, tm_yday=180, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=6, tm_mday=29, tm_hour=17, tm_min=9, tm_sec=3, tm_wday=2, tm_yday=180, tm_isdst=0) 2022-06-29T17:09:08.566 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2022-06-29T15:58:19.966881+0000 time.struct_time(tm_year=2022, tm_mon=6, tm_mday=29, tm_hour=15, tm_min=58, tm_sec=19, tm_wday=2, tm_yday=180, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=6, tm_mday=29, tm_hour=17, tm_min=9, tm_sec=3, tm_wday=2, tm_yday=180, tm_isdst=0) 2022-06-29T17:09:08.566 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2022-06-29T15:58:19.966881+0000 time.struct_time(tm_year=2022, tm_mon=6, tm_mday=29, tm_hour=15, tm_min=58, tm_sec=19, tm_wday=2, tm_yday=180, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=6, tm_mday=29, tm_hour=17, tm_min=9, tm_sec=3, tm_wday=2, tm_yday=180, tm_isdst=0) 2022-06-29T17:09:08.567 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2022-06-29T15:58:19.966881+0000 time.struct_time(tm_year=2022, tm_mon=6, tm_mday=29, tm_hour=15, tm_min=58, tm_sec=19, tm_wday=2, tm_yday=180, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=6, tm_mday=29, tm_hour=17, tm_min=9, tm_sec=3, tm_wday=2, tm_yday=180, tm_isdst=0) 2022-06-29T17:09:08.568 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2022-06-29T17:09:14.225 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:09:14.230 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:09:14.234 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:09:14.239 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:09:14.244 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:09:28.569 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-06-29T17:09:28.686 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.679+0000 7ff4b404a700 1 Processor -- start 2022-06-29T17:09:28.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.679+0000 7ff4b404a700 1 -- start start 2022-06-29T17:09:28.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.679+0000 7ff4b404a700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7ff4ac0b5950 0x7ff4ac067680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:28.687 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.679+0000 7ff4b404a700 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4ac0acba0 con 0x7ff4ac063ad0 2022-06-29T17:09:28.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.679+0000 7ff4b404a700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4ac067220 con 0x7ff4ac14af10 2022-06-29T17:09:28.688 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.679+0000 7ff4b404a700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4ac14b3a0 con 0x7ff4ac0b5950 2022-06-29T17:09:28.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.679+0000 7ff4b15e5700 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff4ac063ad0 0x7ff4ac05bd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34984/0 (socket says 172.21.15.60:34984) 2022-06-29T17:09:28.689 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.679+0000 7ff4b15e5700 1 -- 172.21.15.60:0/2437970876 learned_addr learned my addr 172.21.15.60:0/2437970876 (peer_addr_for_me v1:172.21.15.60:0/0) 2022-06-29T17:09:28.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.679+0000 7ff4b1de6700 1 --2- 172.21.15.60:0/2437970876 >> v2:172.21.15.60:3300/0 conn(0x7ff4ac0b5950 0x7ff4ac067680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:28.690 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.679+0000 7ff4b0de4700 1 -- 172.21.15.60:0/2437970876 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1191540584 0 0) 0x7ff4ac0acba0 con 0x7ff4ac063ad0 2022-06-29T17:09:28.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.679+0000 7ff4b0de4700 1 -- 172.21.15.60:0/2437970876 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff490003260 con 0x7ff4ac063ad0 2022-06-29T17:09:28.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.679+0000 7ff4b1de6700 1 -- 172.21.15.60:0/2437970876 >> v1:172.21.15.92:6789/0 conn(0x7ff4ac14af10 legacy=0x7ff4ac05c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:28.691 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.679+0000 7ff4b0de4700 1 -- 172.21.15.60:0/2437970876 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2841909882 0 0) 0x7ff490003260 con 0x7ff4ac063ad0 2022-06-29T17:09:28.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.679+0000 7ff4b1de6700 1 -- 172.21.15.60:0/2437970876 >> v1:172.21.15.60:6789/0 conn(0x7ff4ac063ad0 legacy=0x7ff4ac05bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:28.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.679+0000 7ff4b1de6700 1 -- 172.21.15.60:0/2437970876 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ac0b5d30 con 0x7ff4ac0b5950 2022-06-29T17:09:28.692 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.679+0000 7ff4b1de6700 1 --2- 172.21.15.60:0/2437970876 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4ac0b5950 0x7ff4ac067680 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 rx=0x7ff4a8001820 tx=0x7ff4a0006df0).ready entity=mon.0 client_cookie=e9e1cfb7a894b395 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:28.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.679+0000 7ff4b0de4700 1 -- 172.21.15.60:0/2437970876 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff4a8001680 con 0x7ff4ac0b5950 2022-06-29T17:09:28.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.679+0000 7ff4b0de4700 1 -- 172.21.15.60:0/2437970876 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4a0000e90 con 0x7ff4ac0b5950 2022-06-29T17:09:28.693 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.679+0000 7ff4b0de4700 1 -- 172.21.15.60:0/2437970876 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff4a000d4f0 con 0x7ff4ac0b5950 2022-06-29T17:09:28.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.679+0000 7ff4b404a700 1 -- 172.21.15.60:0/2437970876 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4ac0b5950 msgr2=0x7ff4ac067680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:28.698 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.679+0000 7ff4b404a700 1 --2- 172.21.15.60:0/2437970876 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4ac0b5950 0x7ff4ac067680 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 rx=0x7ff4a8001820 tx=0x7ff4a0006df0).stop 2022-06-29T17:09:28.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.679+0000 7ff4b404a700 1 -- 172.21.15.60:0/2437970876 shutdown_connections 2022-06-29T17:09:28.699 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.679+0000 7ff4b404a700 1 --2- 172.21.15.60:0/2437970876 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4ac0b5950 0x7ff4ac067680 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:28.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.679+0000 7ff4b404a700 1 -- 172.21.15.60:0/2437970876 >> 172.21.15.60:0/2437970876 conn(0x7ff4ac0632c0 msgr2=0x7ff4ac0b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:28.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.679+0000 7ff4b404a700 1 -- 172.21.15.60:0/2437970876 shutdown_connections 2022-06-29T17:09:28.700 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.679+0000 7ff4b404a700 1 -- 172.21.15.60:0/2437970876 wait complete. 2022-06-29T17:09:28.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.683+0000 7ff4b404a700 1 Processor -- start 2022-06-29T17:09:28.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.683+0000 7ff4b404a700 1 -- start start 2022-06-29T17:09:28.701 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.683+0000 7ff4b404a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4ac063ad0 0x7ff4ac143110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:28.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.683+0000 7ff4b404a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4ac0b5950 0x7ff4ac148320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:28.702 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.683+0000 7ff4b404a700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4ac14af10 0x7ff4ac0b06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:28.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.683+0000 7ff4b404a700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac0ad6a0 con 0x7ff4ac063ad0 2022-06-29T17:09:28.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.683+0000 7ff4b404a700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac0ad160 con 0x7ff4ac0b5950 2022-06-29T17:09:28.703 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.683+0000 7ff4b404a700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac0ad400 con 0x7ff4ac14af10 2022-06-29T17:09:28.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.683+0000 7ff4b1de6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4ac063ad0 0x7ff4ac143110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:28.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.683+0000 7ff4b1de6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4ac063ad0 0x7ff4ac143110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55136/0 (socket says 172.21.15.60:55136) 2022-06-29T17:09:28.704 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.683+0000 7ff4b1de6700 1 -- 172.21.15.60:0/3711829861 learned_addr learned my addr 172.21.15.60:0/3711829861 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T17:09:28.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.683+0000 7ff4b15e5700 1 --2- 172.21.15.60:0/3711829861 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4ac0b5950 0x7ff4ac148320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:28.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.683+0000 7ff4b25e7700 1 --2- 172.21.15.60:0/3711829861 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4ac14af10 0x7ff4ac0b06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:28.705 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.683+0000 7ff4b1de6700 1 -- 172.21.15.60:0/3711829861 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4ac14af10 msgr2=0x7ff4ac0b06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:28.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.683+0000 7ff4b1de6700 1 --2- 172.21.15.60:0/3711829861 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4ac14af10 0x7ff4ac0b06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:28.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.683+0000 7ff4b1de6700 1 -- 172.21.15.60:0/3711829861 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4ac0b5950 msgr2=0x7ff4ac148320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:28.706 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.683+0000 7ff4b1de6700 1 --2- 172.21.15.60:0/3711829861 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4ac0b5950 0x7ff4ac148320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:28.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.683+0000 7ff4b1de6700 1 -- 172.21.15.60:0/3711829861 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a000b180 con 0x7ff4ac063ad0 2022-06-29T17:09:28.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.683+0000 7ff4b1de6700 1 --2- 172.21.15.60:0/3711829861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4ac063ad0 0x7ff4ac143110 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 rx=0x7ff4a8001820 tx=0x7ff4a0014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:28.707 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.683+0000 7ff49effd700 1 -- 172.21.15.60:0/3711829861 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff4a0016070 con 0x7ff4ac063ad0 2022-06-29T17:09:28.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.683+0000 7ff49effd700 1 -- 172.21.15.60:0/3711829861 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4a0002f50 con 0x7ff4ac063ad0 2022-06-29T17:09:28.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.683+0000 7ff4b404a700 1 -- 172.21.15.60:0/3711829861 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4ac0adea0 con 0x7ff4ac063ad0 2022-06-29T17:09:28.708 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.683+0000 7ff49effd700 1 -- 172.21.15.60:0/3711829861 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff4a000d4f0 con 0x7ff4ac063ad0 2022-06-29T17:09:28.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.683+0000 7ff4b404a700 1 -- 172.21.15.60:0/3711829861 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4ac149d30 con 0x7ff4ac063ad0 2022-06-29T17:09:28.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.687+0000 7ff4b404a700 1 -- 172.21.15.60:0/3711829861 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4ac0573d0 con 0x7ff4ac063ad0 2022-06-29T17:09:28.709 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.687+0000 7ff49effd700 1 -- 172.21.15.60:0/3711829861 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7ff4a0025030 con 0x7ff4ac063ad0 2022-06-29T17:09:28.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.687+0000 7ff49effd700 1 --2- 172.21.15.60:0/3711829861 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff494053b00 0x7ff494055fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:28.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.687+0000 7ff49effd700 1 -- 172.21.15.60:0/3711829861 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5339+0+0 (secure 0 0 0) 0x7ff4a006caf0 con 0x7ff4ac063ad0 2022-06-29T17:09:28.710 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.687+0000 7ff4b15e5700 1 --2- 172.21.15.60:0/3711829861 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff494053b00 0x7ff494055fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:28.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.687+0000 7ff4b15e5700 1 --2- 172.21.15.60:0/3711829861 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff494053b00 0x7ff494055fc0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0x7ff4a800d000 tx=0x7ff4a8000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:28.711 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.695+0000 7ff49effd700 1 -- 172.21.15.60:0/3711829861 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff4a0043c20 con 0x7ff4ac063ad0 2022-06-29T17:09:28.827 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.819+0000 7ff4b404a700 1 -- 172.21.15.60:0/3711829861 --> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff4ac0acfe0 con 0x7ff494053b00 2022-06-29T17:09:28.833 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.827+0000 7ff49effd700 1 -- 172.21.15.60:0/3711829861 <== mgr.4103 v2:172.21.15.92:6832/12872 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+68304 (secure 0 0 0) 0x7ff4a8015410 con 0x7ff494053b00 2022-06-29T17:09:28.834 INFO:teuthology.orchestra.run.smithi060.stdout: 2022-06-29T17:09:28.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.827+0000 7ff4b404a700 1 -- 172.21.15.60:0/3711829861 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff494053b00 msgr2=0x7ff494055fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:28.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.827+0000 7ff4b404a700 1 --2- 172.21.15.60:0/3711829861 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff494053b00 0x7ff494055fc0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0x7ff4a800d000 tx=0x7ff4a8000bc0).stop 2022-06-29T17:09:28.837 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.827+0000 7ff4b404a700 1 -- 172.21.15.60:0/3711829861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4ac063ad0 msgr2=0x7ff4ac143110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:28.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.827+0000 7ff4b404a700 1 --2- 172.21.15.60:0/3711829861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4ac063ad0 0x7ff4ac143110 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 rx=0x7ff4a8001820 tx=0x7ff4a0014000).stop 2022-06-29T17:09:28.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.831+0000 7ff4b404a700 1 -- 172.21.15.60:0/3711829861 shutdown_connections 2022-06-29T17:09:28.838 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.831+0000 7ff4b404a700 1 --2- 172.21.15.60:0/3711829861 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7ff494053b00 0x7ff494055fc0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:28.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.831+0000 7ff4b404a700 1 --2- 172.21.15.60:0/3711829861 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7ff4ac14af10 0x7ff4ac0b06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:28.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.831+0000 7ff4b404a700 1 --2- 172.21.15.60:0/3711829861 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4ac0b5950 0x7ff4ac148320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:28.839 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.831+0000 7ff4b404a700 1 --2- 172.21.15.60:0/3711829861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff4ac063ad0 0x7ff4ac143110 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:28.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.831+0000 7ff4b404a700 1 -- 172.21.15.60:0/3711829861 >> 172.21.15.60:0/3711829861 conn(0x7ff4ac0632c0 msgr2=0x7ff4ac14a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:28.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.831+0000 7ff4b404a700 1 -- 172.21.15.60:0/3711829861 shutdown_connections 2022-06-29T17:09:28.840 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.831+0000 7ff4b404a700 1 -- 172.21.15.60:0/3711829861 wait complete. 2022-06-29T17:09:28.841 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2022-06-29T17:09:28.848 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":2280,"stamp":"2022-06-29T17:09:28.095319+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9482,"num_omap_keys":2,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":6385024,"kb_used_data":743210016,"kb_used_omap":27940,"kb_used_meta":0,"kb_avail":743210016,"statfs":{"total":767585320960,"available":761047056384,"internally_reserved":0,"allocated":761047056384,"data_stored":761047056384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28611414,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003772"},"pg_stats":[{"pgid":"1.0","version":"150'4","reported_seq":151,"reported_epoch":150,"state":"active+clean","last_fresh":"2022-06-29T17:09:12.571283+0000","last_change":"2022-06-29T17:09:12.571368+0000","last_active":"2022-06-29T17:09:12.571283+0000","last_peered":"2022-06-29T17:09:12.571283+0000","last_clean":"2022-06-29T17:09:12.571283+0000","last_became_active":"2022-06-29T15:58:14.848454+0000","last_became_peered":"2022-06-29T15:58:14.848454+0000","last_unstale":"2022-06-29T17:09:12.571283+0000","last_undegraded":"2022-06-29T17:09:12.571283+0000","last_fullsized":"2022-06-29T17:09:12.571283+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'4","last_scrub_stamp":"2022-06-29T17:09:12.571283+0000","last_deep_scrub":"150'4","last_deep_scrub_stamp":"2022-06-29T17:09:12.571283+0000","last_clean_scrub_stamp":"2022-06-29T17:09:12.571283+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9482,"num_omap_keys":2,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":135,"reported_epoch":150,"state":"active+clean","last_fresh":"2022-06-29T17:09:11.212779+0000","last_change":"2022-06-29T17:09:11.212879+0000","last_active":"2022-06-29T17:09:11.212779+0000","last_peered":"2022-06-29T17:09:11.212779+0000","last_clean":"2022-06-29T17:09:11.212779+0000","last_became_active":"2022-06-29T15:58:21.003537+0000","last_became_peered":"2022-06-29T15:58:21.003537+0000","last_unstale":"2022-06-29T17:09:11.212779+0000","last_undegraded":"2022-06-29T17:09:11.212779+0000","last_fullsized":"2022-06-29T17:09:11.212779+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T17:09:11.212779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T17:09:11.212779+0000","last_clean_scrub_stamp":"2022-06-29T17:09:11.212779+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":144,"reported_epoch":150,"state":"active+clean","last_fresh":"2022-06-29T17:09:13.549774+0000","last_change":"2022-06-29T17:09:13.549877+0000","last_active":"2022-06-29T17:09:13.549774+0000","last_peered":"2022-06-29T17:09:13.549774+0000","last_clean":"2022-06-29T17:09:13.549774+0000","last_became_active":"2022-06-29T15:58:21.000885+0000","last_became_peered":"2022-06-29T15:58:21.000885+0000","last_unstale":"2022-06-29T17:09:13.549774+0000","last_undegraded":"2022-06-29T17:09:13.549774+0000","last_fullsized":"2022-06-29T17:09:13.549774+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T17:09:13.549773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T17:09:13.549773+0000","last_clean_scrub_stamp":"2022-06-29T17:09:13.549773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":139,"reported_epoch":150,"state":"active+clean","last_fresh":"2022-06-29T17:09:06.284567+0000","last_change":"2022-06-29T17:09:06.284704+0000","last_active":"2022-06-29T17:09:06.284567+0000","last_peered":"2022-06-29T17:09:06.284567+0000","last_clean":"2022-06-29T17:09:06.284567+0000","last_became_active":"2022-06-29T15:58:20.997032+0000","last_became_peered":"2022-06-29T15:58:20.997032+0000","last_unstale":"2022-06-29T17:09:06.284567+0000","last_undegraded":"2022-06-29T17:09:06.284567+0000","last_fullsized":"2022-06-29T17:09:06.284567+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T17:09:06.284566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T17:09:06.284566+0000","last_clean_scrub_stamp":"2022-06-29T17:09:06.284566+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":135,"reported_epoch":150,"state":"active+clean","last_fresh":"2022-06-29T17:09:09.129152+0000","last_change":"2022-06-29T17:09:09.129249+0000","last_active":"2022-06-29T17:09:09.129152+0000","last_peered":"2022-06-29T17:09:09.129152+0000","last_clean":"2022-06-29T17:09:09.129152+0000","last_became_active":"2022-06-29T15:58:21.004623+0000","last_became_peered":"2022-06-29T15:58:21.004623+0000","last_unstale":"2022-06-29T17:09:09.129152+0000","last_undegraded":"2022-06-29T17:09:09.129152+0000","last_fullsized":"2022-06-29T17:09:09.129152+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T17:09:09.129151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T17:09:09.129151+0000","last_clean_scrub_stamp":"2022-06-29T17:09:09.129151+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":134,"reported_epoch":150,"state":"active+clean","last_fresh":"2022-06-29T17:09:09.803387+0000","last_change":"2022-06-29T17:09:09.803487+0000","last_active":"2022-06-29T17:09:09.803387+0000","last_peered":"2022-06-29T17:09:09.803387+0000","last_clean":"2022-06-29T17:09:09.803387+0000","last_became_active":"2022-06-29T15:58:21.800792+0000","last_became_peered":"2022-06-29T15:58:21.800792+0000","last_unstale":"2022-06-29T17:09:09.803387+0000","last_undegraded":"2022-06-29T17:09:09.803387+0000","last_fullsized":"2022-06-29T17:09:09.803387+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T17:09:09.803387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T17:09:09.803387+0000","last_clean_scrub_stamp":"2022-06-29T17:09:09.803387+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":144,"reported_epoch":150,"state":"active+clean","last_fresh":"2022-06-29T17:09:15.504212+0000","last_change":"2022-06-29T17:09:15.504312+0000","last_active":"2022-06-29T17:09:15.504212+0000","last_peered":"2022-06-29T17:09:15.504212+0000","last_clean":"2022-06-29T17:09:15.504212+0000","last_became_active":"2022-06-29T15:58:21.807346+0000","last_became_peered":"2022-06-29T15:58:21.807346+0000","last_unstale":"2022-06-29T17:09:15.504212+0000","last_undegraded":"2022-06-29T17:09:15.504212+0000","last_fullsized":"2022-06-29T17:09:15.504212+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T17:09:15.504211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T17:09:15.504211+0000","last_clean_scrub_stamp":"2022-06-29T17:09:15.504211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":134,"reported_epoch":150,"state":"active+clean","last_fresh":"2022-06-29T17:09:11.773472+0000","last_change":"2022-06-29T17:09:11.773570+0000","last_active":"2022-06-29T17:09:11.773472+0000","last_peered":"2022-06-29T17:09:11.773472+0000","last_clean":"2022-06-29T17:09:11.773472+0000","last_became_active":"2022-06-29T15:58:20.995883+0000","last_became_peered":"2022-06-29T15:58:20.995883+0000","last_unstale":"2022-06-29T17:09:11.773472+0000","last_undegraded":"2022-06-29T17:09:11.773472+0000","last_fullsized":"2022-06-29T17:09:11.773472+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T17:09:11.773471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T17:09:11.773471+0000","last_clean_scrub_stamp":"2022-06-29T17:09:11.773471+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":134,"reported_epoch":150,"state":"active+clean","last_fresh":"2022-06-29T17:09:08.790207+0000","last_change":"2022-06-29T17:09:08.790308+0000","last_active":"2022-06-29T17:09:08.790207+0000","last_peered":"2022-06-29T17:09:08.790207+0000","last_clean":"2022-06-29T17:09:08.790207+0000","last_became_active":"2022-06-29T15:58:21.002353+0000","last_became_peered":"2022-06-29T15:58:21.002353+0000","last_unstale":"2022-06-29T17:09:08.790207+0000","last_undegraded":"2022-06-29T17:09:08.790207+0000","last_fullsized":"2022-06-29T17:09:08.790207+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-06-29T17:09:08.790206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-06-29T17:09:08.790206+0000","last_clean_scrub_stamp":"2022-06-29T17:09:08.790206+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9482,"num_omap_keys":2,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575705,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798144,"kb_used_data":92901236,"kb_used_omap":3665,"kb_used_meta":0,"kb_avail":92901236,"statfs":{"total":95948165120,"available":95130865664,"internally_reserved":0,"allocated":95130865664,"data_stored":95130865664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3753739,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jun 29 17:08:21 2022","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.83699999999999997,"15min":0.82699999999999996},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.5},"max":{"1min":1.0640000000000001,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.81599999999999995,"15min":0.81499999999999995},"min":{"1min":0.61599999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.95199999999999996,"5min":1.3120000000000001,"15min":1.377},"last":0.95499999999999996}]},{"osd":1,"last update":"Wed Jun 29 17:08:21 2022","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.82399999999999995,"15min":0.83299999999999996},"min":{"1min":0.64900000000000002,"5min":0.57199999999999995,"15min":0.56599999999999995},"max":{"1min":1.125,"5min":1.125,"15min":1.1890000000000001},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.83399999999999996,"15min":0.84899999999999998},"min":{"1min":0.623,"5min":0.60599999999999998,"15min":0.59399999999999997},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1659999999999999},"last":0.97799999999999998}]},{"osd":2,"last update":"Wed Jun 29 17:08:21 2022","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.871,"15min":0.872},"min":{"1min":0.625,"5min":0.625,"15min":0.57099999999999995},"max":{"1min":1.202,"5min":1.202,"15min":1.244},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.85299999999999998,"15min":0.85099999999999998},"min":{"1min":0.61899999999999999,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.097,"5min":1.371,"15min":1.371},"last":0.83399999999999996}]},{"osd":3,"last update":"Wed Jun 29 17:08:21 2022","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.84299999999999997,"15min":0.84699999999999998},"min":{"1min":0.64000000000000001,"5min":0.60499999999999998,"15min":0.48999999999999999},"max":{"1min":1.0600000000000001,"5min":1.204,"15min":1.3480000000000001},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.92300000000000004,"15min":0.91500000000000004},"min":{"1min":0.69399999999999995,"5min":0.69099999999999995,"15min":0.62},"max":{"1min":1.077,"5min":1.2549999999999999,"15min":1.4399999999999999},"last":1.095}]},{"osd":4,"last update":"Wed Jun 29 17:08:21 2022","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.79800000000000004,"15min":0.79200000000000004},"min":{"1min":0.58999999999999997,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0189999999999999,"5min":1.321,"15min":1.4890000000000001},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.86399999999999999,"15min":0.84699999999999998},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.55300000000000005},"max":{"1min":1.02,"5min":1.284,"15min":1.4239999999999999},"last":0.79100000000000004}]},{"osd":5,"last update":"Wed Jun 29 17:08:21 2022","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.91100000000000003,"15min":0.90900000000000003},"min":{"1min":0.68899999999999995,"5min":0.58299999999999996,"15min":0.55900000000000005},"max":{"1min":1.139,"5min":1.2370000000000001,"15min":1.3149999999999999},"last":1.046},{"interface":"front","average":{"1min":1.0109999999999999,"5min":0.98799999999999999,"15min":0.98899999999999999},"min":{"1min":0.90200000000000002,"5min":0.621,"15min":0.621},"max":{"1min":1.109,"5min":1.46,"15min":1.585},"last":1.081}]},{"osd":6,"last update":"Wed Jun 29 17:08:21 2022","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.93600000000000005,"15min":0.94999999999999996},"min":{"1min":0.69699999999999995,"5min":0.65500000000000003,"15min":0.63600000000000001},"max":{"1min":1.1839999999999999,"5min":1.4139999999999999,"15min":1.5449999999999999},"last":1.0129999999999999},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.94199999999999995},"min":{"1min":0.67100000000000004,"5min":0.64200000000000002,"15min":0.61199999999999999},"max":{"1min":1.109,"5min":1.353,"15min":1.55},"last":1}]}]},{"osd":6,"up_from":13,"seq":55834575706,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798144,"kb_used_data":92901236,"kb_used_omap":4010,"kb_used_meta":0,"kb_avail":92901236,"statfs":{"total":95948165120,"available":95130865664,"internally_reserved":0,"allocated":95130865664,"data_stored":95130865664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4107229,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jun 29 17:08:41 2022","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.76800000000000002,"15min":0.75800000000000001},"min":{"1min":0.59899999999999998,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.159},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.78200000000000003,"15min":0.76800000000000002},"min":{"1min":0.55300000000000005,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.089,"5min":1.089,"15min":1.177},"last":0.69799999999999995}]},{"osd":1,"last update":"Wed Jun 29 17:08:41 2022","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.86499999999999999,"15min":0.85299999999999998},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.0740000000000001,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.90900000000000003,"15min":0.91200000000000003},"min":{"1min":0.621,"5min":0.621,"15min":0.58199999999999996},"max":{"1min":1.0469999999999999,"5min":1.1399999999999999,"15min":1.3220000000000001},"last":0.94599999999999995}]},{"osd":2,"last update":"Wed Jun 29 17:08:41 2022","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.88300000000000001,"15min":0.85799999999999998},"min":{"1min":0.64700000000000002,"5min":0.60999999999999999,"15min":0.55500000000000005},"max":{"1min":1.0389999999999999,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.877,"5min":0.88900000000000001,"15min":0.89800000000000002},"min":{"1min":0.59199999999999997,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.026,"5min":1.1950000000000001,"15min":1.2090000000000001},"last":0.74399999999999999}]},{"osd":3,"last update":"Wed Jun 29 17:08:41 2022","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.82899999999999996,"15min":0.83199999999999996},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.51700000000000002},"max":{"1min":1.024,"5min":1.1719999999999999,"15min":1.4139999999999999},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.82699999999999996,"15min":0.82999999999999996},"min":{"1min":0.628,"5min":0.60999999999999999,"15min":0.60799999999999998},"max":{"1min":1.032,"5min":1.0880000000000001,"15min":1.161},"last":0.71699999999999997}]},{"osd":4,"last update":"Wed Jun 29 17:08:41 2022","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.77300000000000002,"15min":0.80100000000000005},"min":{"1min":0.52800000000000002,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.0609999999999999,"5min":1.1799999999999999,"15min":1.3180000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.92900000000000005,"15min":0.93300000000000005},"min":{"1min":0.57499999999999996,"5min":0.53000000000000003,"15min":0.52700000000000002},"max":{"1min":1.1080000000000001,"5min":1.2270000000000001,"15min":1.4850000000000001},"last":0.99299999999999999}]},{"osd":5,"last update":"Wed Jun 29 17:08:41 2022","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.85399999999999998,"15min":0.83999999999999997},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.48999999999999999},"max":{"1min":1.175,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.871},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.94899999999999995,"15min":0.98399999999999999},"min":{"1min":0.64300000000000002,"5min":0.59699999999999998,"15min":0.55100000000000005},"max":{"1min":1.121,"5min":1.2330000000000001,"15min":1.4079999999999999},"last":1.028}]},{"osd":7,"last update":"Wed Jun 29 17:08:41 2022","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.92100000000000004,"15min":0.93300000000000005},"min":{"1min":0.60999999999999999,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.0269999999999999,"5min":1.3700000000000001,"15min":1.532},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.96999999999999997,"5min":0.995,"15min":1.0029999999999999},"min":{"1min":0.80900000000000005,"5min":0.66300000000000003,"15min":0.60199999999999998},"max":{"1min":1.2350000000000001,"5min":1.2430000000000001,"15min":1.319},"last":0.92400000000000004}]}]},{"osd":5,"up_from":13,"seq":55834575704,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798144,"kb_used_data":92901236,"kb_used_omap":2745,"kb_used_meta":0,"kb_avail":92901236,"statfs":{"total":95948165120,"available":95130865664,"internally_reserved":0,"allocated":95130865664,"data_stored":95130865664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2811308,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jun 29 17:08:27 2022","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.84999999999999998,"15min":0.85399999999999998},"min":{"1min":0.68600000000000005,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.1160000000000001,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":1.0149999999999999},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.93600000000000005,"15min":0.93500000000000005},"min":{"1min":0.76800000000000002,"5min":0.623,"15min":0.52500000000000002},"max":{"1min":1.0860000000000001,"5min":1.167,"15min":1.286},"last":0.95999999999999996}]},{"osd":1,"last update":"Wed Jun 29 17:08:27 2022","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.84099999999999997,"15min":0.83199999999999996},"min":{"1min":0.66100000000000003,"5min":0.57099999999999995,"15min":0.55100000000000005},"max":{"1min":1.0700000000000001,"5min":1.1299999999999999,"15min":1.339},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.84099999999999997,"15min":0.84599999999999997},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.22},"last":0.92600000000000005}]},{"osd":2,"last update":"Wed Jun 29 17:08:27 2022","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.86499999999999999,"15min":0.86599999999999999},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1910000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.60399999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0269999999999999,"5min":1.0940000000000001,"15min":1.131},"last":0.93700000000000006}]},{"osd":3,"last update":"Wed Jun 29 17:08:27 2022","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.86599999999999999,"15min":0.86099999999999999},"min":{"1min":0.66800000000000004,"5min":0.64300000000000002,"15min":0.59399999999999997},"max":{"1min":1.131,"5min":1.131,"15min":1.379},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.94599999999999995,"15min":0.92100000000000004},"min":{"1min":0.78600000000000003,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.872}]},{"osd":4,"last update":"Wed Jun 29 17:08:27 2022","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.78800000000000003,"15min":0.78100000000000003},"min":{"1min":0.54100000000000004,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.0509999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":1.006,"5min":0.995,"15min":0.995},"min":{"1min":0.82599999999999996,"5min":0.71799999999999997,"15min":0.67500000000000004},"max":{"1min":1.258,"5min":1.258,"15min":1.4179999999999999},"last":0.89500000000000002}]},{"osd":6,"last update":"Wed Jun 29 17:08:27 2022","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.90200000000000002,"15min":0.876},"min":{"1min":0.58099999999999996,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.171,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.90200000000000002,"15min":0.88900000000000001},"min":{"1min":0.73099999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0600000000000001,"5min":1.4159999999999999,"15min":1.4159999999999999},"last":0.84199999999999997}]},{"osd":7,"last update":"Wed Jun 29 17:08:27 2022","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.94499999999999995},"min":{"1min":0.64500000000000002,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.212,"5min":1.3959999999999999,"15min":1.4570000000000001},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.0469999999999999,"15min":1.0069999999999999},"min":{"1min":0.85599999999999998,"5min":0.64100000000000001,"15min":0.63400000000000001},"max":{"1min":1.3300000000000001,"5min":1.6850000000000001,"15min":1.6950000000000001},"last":0.96899999999999997}]}]},{"osd":4,"up_from":13,"seq":55834575706,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798016,"kb_used_data":92901364,"kb_used_omap":3569,"kb_used_meta":0,"kb_avail":92901364,"statfs":{"total":95948165120,"available":95130996736,"internally_reserved":0,"allocated":95130996736,"data_stored":95130996736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3654826,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jun 29 17:08:23 2022","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.79500000000000004,"15min":0.79600000000000004},"min":{"1min":0.57299999999999995,"5min":0.55300000000000005,"15min":0.53100000000000003},"max":{"1min":1.083,"5min":1.117,"15min":1.1579999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.79400000000000004,"15min":0.79700000000000004},"min":{"1min":0.61099999999999999,"5min":0.58199999999999996,"15min":0.52400000000000002},"max":{"1min":0.94499999999999995,"5min":1.135,"15min":1.1739999999999999},"last":0.72599999999999998}]},{"osd":1,"last update":"Wed Jun 29 17:08:23 2022","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.84099999999999997,"15min":0.82699999999999996},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.57299999999999995},"max":{"1min":1.024,"5min":1.0800000000000001,"15min":1.1359999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81399999999999995,"15min":0.82599999999999996},"min":{"1min":0.63500000000000001,"5min":0.59899999999999998,"15min":0.57999999999999996},"max":{"1min":0.96699999999999997,"5min":1.0940000000000001,"15min":1.1930000000000001},"last":0.95799999999999996}]},{"osd":2,"last update":"Wed Jun 29 17:08:23 2022","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.90800000000000003,"15min":0.91700000000000004},"min":{"1min":0.625,"5min":0.59699999999999998,"15min":0.57699999999999996},"max":{"1min":1.0840000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.92100000000000004,"15min":0.90800000000000003},"min":{"1min":0.67200000000000004,"5min":0.61399999999999999,"15min":0.60199999999999998},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.23},"last":0.78500000000000003}]},{"osd":3,"last update":"Wed Jun 29 17:08:23 2022","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.81699999999999995,"15min":0.81100000000000005},"min":{"1min":0.64600000000000002,"5min":0.628,"15min":0.56799999999999995},"max":{"1min":0.995,"5min":1.212,"15min":1.212},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.89400000000000002,"15min":0.90000000000000002},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.60799999999999998},"max":{"1min":1.0549999999999999,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.82599999999999996}]},{"osd":5,"last update":"Wed Jun 29 17:08:23 2022","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.85099999999999998,"15min":0.84299999999999997},"min":{"1min":0.54300000000000004,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.101,"5min":1.133,"15min":1.2450000000000001},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.91200000000000003,"15min":0.90200000000000002},"min":{"1min":0.61199999999999999,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.135,"5min":1.1819999999999999,"15min":1.248},"last":0.52500000000000002}]},{"osd":6,"last update":"Wed Jun 29 17:08:23 2022","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.88600000000000001,"15min":0.88500000000000001},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.54600000000000004},"max":{"1min":1.083,"5min":1.417,"15min":1.417},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.98899999999999999,"15min":0.97399999999999998},"min":{"1min":0.67400000000000004,"5min":0.60099999999999998,"15min":0.53200000000000003},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.2889999999999999},"last":0.90800000000000003}]},{"osd":7,"last update":"Wed Jun 29 17:08:23 2022","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.88600000000000001,"15min":0.874},"min":{"1min":0.61599999999999999,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.1140000000000001,"5min":1.4319999999999999,"15min":1.4950000000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.97599999999999998,"15min":0.95699999999999996},"min":{"1min":0.72699999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.1519999999999999,"5min":1.466,"15min":1.488},"last":0.755}]}]},{"osd":3,"up_from":13,"seq":55834575705,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798144,"kb_used_data":92901236,"kb_used_omap":3346,"kb_used_meta":0,"kb_avail":92901236,"statfs":{"total":95948165120,"available":95130865664,"internally_reserved":0,"allocated":95130865664,"data_stored":95130865664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3427169,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jun 29 17:09:20 2022","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.64000000000000001,"15min":0.63400000000000001},"min":{"1min":0.46100000000000002,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.0309999999999999,"5min":1.087,"15min":1.1259999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.73699999999999999,"15min":0.73899999999999999},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.94099999999999995,"5min":0.96999999999999997,"15min":1.052},"last":0.70799999999999996}]},{"osd":1,"last update":"Wed Jun 29 17:09:20 2022","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67600000000000005,"15min":0.67200000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.86599999999999999,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.84599999999999997,"15min":0.85099999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.0660000000000001,"5min":1.155,"15min":1.159},"last":0.88400000000000001}]},{"osd":2,"last update":"Wed Jun 29 17:09:20 2022","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.77700000000000002,"15min":0.77400000000000002},"min":{"1min":0.60499999999999998,"5min":0.54200000000000004,"15min":0.54100000000000004},"max":{"1min":1.0800000000000001,"5min":1.194,"15min":1.198},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.70499999999999996,"15min":0.70599999999999996},"min":{"1min":0.52800000000000002,"5min":0.52700000000000002,"15min":0.498},"max":{"1min":0.92900000000000005,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.95799999999999996}]},{"osd":4,"last update":"Wed Jun 29 17:09:20 2022","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.97599999999999998,"15min":0.98099999999999998},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.64700000000000002},"max":{"1min":1.1699999999999999,"5min":1.2210000000000001,"15min":1.23},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.89300000000000002,"15min":0.90500000000000003},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.621},"max":{"1min":1.03,"5min":1.256,"15min":1.4490000000000001},"last":0.86899999999999999}]},{"osd":5,"last update":"Wed Jun 29 17:09:20 2022","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.879,"15min":0.89600000000000002},"min":{"1min":0.66800000000000004,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.1559999999999999,"5min":1.2849999999999999,"15min":1.2889999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.071,"5min":1.036,"15min":1.044},"min":{"1min":0.86799999999999999,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.363,"5min":1.45,"15min":1.5309999999999999},"last":0.98899999999999999}]},{"osd":6,"last update":"Wed Jun 29 17:09:20 2022","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.95999999999999996,"15min":0.97799999999999998},"min":{"1min":0.746,"5min":0.74399999999999999,"15min":0.72499999999999998},"max":{"1min":1.298,"5min":1.3460000000000001,"15min":1.542},"last":1.046},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.022,"15min":1.0269999999999999},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.75800000000000001},"max":{"1min":1.252,"5min":1.345,"15min":1.5289999999999999},"last":1.0049999999999999}]},{"osd":7,"last update":"Wed Jun 29 17:09:20 2022","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.0960000000000001,"15min":1.091},"min":{"1min":0.91700000000000004,"5min":0.83199999999999996,"15min":0.69399999999999995},"max":{"1min":1.411,"5min":1.4730000000000001,"15min":1.5920000000000001},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.044,"15min":1.054},"min":{"1min":0.81299999999999994,"5min":0.77200000000000002,"15min":0.70899999999999996},"max":{"1min":1.2929999999999999,"5min":1.395,"15min":1.47},"last":0.78900000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575706,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798144,"kb_used_data":92901236,"kb_used_omap":2828,"kb_used_meta":0,"kb_avail":92901236,"statfs":{"total":95948165120,"available":95130865664,"internally_reserved":0,"allocated":95130865664,"data_stored":95130865664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2896047,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jun 29 17:08:34 2022","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61899999999999999,"15min":0.62},"min":{"1min":0.48899999999999999,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.86899999999999999,"5min":1.167,"15min":1.3080000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.70999999999999996,"15min":0.71099999999999997},"min":{"1min":0.51600000000000001,"5min":0.48699999999999999,"15min":0.48499999999999999},"max":{"1min":1.1699999999999999,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.64100000000000001}]},{"osd":1,"last update":"Wed Jun 29 17:08:34 2022","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66000000000000003,"15min":0.65200000000000002},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.46999999999999997},"max":{"1min":0.78300000000000003,"5min":1.6799999999999999,"15min":1.6799999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.745,"15min":0.73399999999999999},"min":{"1min":0.56299999999999994,"5min":0.55300000000000005,"15min":0.52900000000000003},"max":{"1min":0.96599999999999997,"5min":1.1899999999999999,"15min":1.206},"last":0.73299999999999998}]},{"osd":3,"last update":"Wed Jun 29 17:08:34 2022","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.85499999999999998,"15min":0.84599999999999997},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.55500000000000005},"max":{"1min":0.999,"5min":1.732,"15min":1.732},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.81000000000000005,"15min":0.80600000000000005},"min":{"1min":0.60899999999999999,"5min":0.58399999999999996,"15min":0.51600000000000001},"max":{"1min":1.0469999999999999,"5min":1.0649999999999999,"15min":1.1599999999999999},"last":0.82899999999999996}]},{"osd":4,"last update":"Wed Jun 29 17:08:34 2022","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.92900000000000005,"15min":0.92600000000000005},"min":{"1min":0.76300000000000001,"5min":0.71399999999999997,"15min":0.60899999999999999},"max":{"1min":1.2729999999999999,"5min":1.2749999999999999,"15min":1.421},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.95599999999999996,"15min":0.94599999999999995},"min":{"1min":0.73499999999999999,"5min":0.72299999999999998,"15min":0.66700000000000004},"max":{"1min":1.302,"5min":1.772,"15min":1.772},"last":0.97199999999999998}]},{"osd":5,"last update":"Wed Jun 29 17:08:34 2022","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":0.96899999999999997,"15min":0.96799999999999997},"min":{"1min":0.73299999999999998,"5min":0.68100000000000005,"15min":0.67000000000000004},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.4350000000000001},"last":1.0129999999999999},{"interface":"front","average":{"1min":0.997,"5min":0.997,"15min":0.98999999999999999},"min":{"1min":0.79300000000000004,"5min":0.72499999999999998,"15min":0.68999999999999995},"max":{"1min":1.2689999999999999,"5min":1.415,"15min":1.4930000000000001},"last":0.86599999999999999}]},{"osd":6,"last update":"Wed Jun 29 17:08:34 2022","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.0489999999999999,"15min":1.024},"min":{"1min":0.84399999999999997,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.397,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.0640000000000001,"15min":1.0600000000000001},"min":{"1min":0.81200000000000006,"5min":0.752,"15min":0.745},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.5549999999999999},"last":1.0940000000000001}]},{"osd":7,"last update":"Wed Jun 29 17:08:34 2022","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0600000000000001,"15min":1.0620000000000001},"min":{"1min":0.92200000000000004,"5min":0.75700000000000001,"15min":0.70299999999999996},"max":{"1min":1.3640000000000001,"5min":1.54,"15min":1.54},"last":1.147},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.125,"15min":1.1000000000000001},"min":{"1min":0.98399999999999999,"5min":0.84299999999999997,"15min":0.76800000000000002},"max":{"1min":1.4550000000000001,"5min":1.845,"15min":1.845},"last":1.0349999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575704,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798144,"kb_used_data":92901236,"kb_used_omap":3974,"kb_used_meta":0,"kb_avail":92901236,"statfs":{"total":95948165120,"available":95130865664,"internally_reserved":0,"allocated":95130865664,"data_stored":95130865664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4069458,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jun 29 17:09:07 2022","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60199999999999998,"15min":0.61899999999999999},"min":{"1min":0.47899999999999998,"5min":0.433,"15min":0.432},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.04},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.71799999999999997,"15min":0.70699999999999996},"min":{"1min":0.501,"5min":0.501,"15min":0.47699999999999998},"max":{"1min":1.379,"5min":1.379,"15min":1.379},"last":0.71299999999999997}]},{"osd":2,"last update":"Wed Jun 29 17:09:07 2022","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.64200000000000002,"15min":0.65900000000000003},"min":{"1min":0.51900000000000002,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.89500000000000002,"5min":1.0389999999999999,"15min":1.569},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.74199999999999999,"15min":0.73999999999999999},"min":{"1min":0.56299999999999994,"5min":0.52000000000000002,"15min":0.48299999999999998},"max":{"1min":0.89100000000000001,"5min":1.111,"15min":1.111},"last":0.66800000000000004}]},{"osd":3,"last update":"Wed Jun 29 17:09:07 2022","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.88700000000000001,"15min":0.89300000000000002},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.0049999999999999,"5min":1.26,"15min":1.425},"last":1.0409999999999999},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.83899999999999997,"15min":0.83399999999999996},"min":{"1min":0.69399999999999995,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.95799999999999996,"5min":1.1459999999999999,"15min":1.4350000000000001},"last":0.98799999999999999}]},{"osd":4,"last update":"Wed Jun 29 17:09:07 2022","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.89300000000000002,"15min":0.89100000000000001},"min":{"1min":0.70499999999999996,"5min":0.629,"15min":0.629},"max":{"1min":1.0900000000000001,"5min":1.2310000000000001,"15min":1.395},"last":0.75},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.97499999999999998,"15min":0.97399999999999998},"min":{"1min":0.69499999999999995,"5min":0.69299999999999995,"15min":0.65200000000000002},"max":{"1min":1.1919999999999999,"5min":1.663,"15min":1.663},"last":0.93400000000000005}]},{"osd":5,"last update":"Wed Jun 29 17:09:07 2022","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.95799999999999996,"15min":0.96999999999999997},"min":{"1min":0.753,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.3959999999999999,"5min":1.4099999999999999,"15min":1.637},"last":0.82599999999999996},{"interface":"front","average":{"1min":1.022,"5min":0.97599999999999998,"15min":0.98999999999999999},"min":{"1min":0.81799999999999995,"5min":0.68300000000000005,"15min":0.68200000000000005},"max":{"1min":1.413,"5min":1.413,"15min":1.413},"last":0.90300000000000002}]},{"osd":6,"last update":"Wed Jun 29 17:09:07 2022","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.0549999999999999,"15min":1.052},"min":{"1min":0.77300000000000002,"5min":0.71199999999999997,"15min":0.69399999999999995},"max":{"1min":1.3149999999999999,"5min":1.73,"15min":1.73},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.087,"5min":1.0900000000000001,"15min":1.0940000000000001},"min":{"1min":0.89500000000000002,"5min":0.81399999999999995,"15min":0.71699999999999997},"max":{"1min":1.359,"5min":1.5109999999999999,"15min":1.7470000000000001},"last":1.151}]},{"osd":7,"last update":"Wed Jun 29 17:09:07 2022","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.105,"15min":1.109},"min":{"1min":0.85999999999999999,"5min":0.78600000000000003,"15min":0.69099999999999995},"max":{"1min":1.349,"5min":1.4890000000000001,"15min":1.704},"last":1.085},{"interface":"front","average":{"1min":1.125,"5min":1.0740000000000001,"15min":1.093},"min":{"1min":0.90600000000000003,"5min":0.73699999999999999,"15min":0.70999999999999996},"max":{"1min":1.4990000000000001,"5min":1.516,"15min":1.516},"last":0.97499999999999998}]}]},{"osd":0,"up_from":13,"seq":55834575705,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798144,"kb_used_data":92901236,"kb_used_omap":3800,"kb_used_meta":0,"kb_avail":92901236,"statfs":{"total":95948165120,"available":95130865664,"internally_reserved":0,"allocated":95130865664,"data_stored":95130865664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3891638,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Jun 29 17:08:25 2022","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.61699999999999999,"15min":0.60399999999999998},"min":{"1min":0.46800000000000003,"5min":0.45200000000000001,"15min":0.45100000000000001},"max":{"1min":1.1579999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.68799999999999994,"15min":0.66800000000000004},"min":{"1min":0.53000000000000003,"5min":0.48799999999999999,"15min":0.48399999999999999},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.59299999999999997}]},{"osd":2,"last update":"Wed Jun 29 17:08:25 2022","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81499999999999995,"15min":0.80500000000000005},"min":{"1min":0.60799999999999998,"5min":0.53200000000000003,"15min":0.5},"max":{"1min":1.02,"5min":1.48,"15min":1.48},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.71199999999999997,"15min":0.72099999999999997},"min":{"1min":0.505,"5min":0.505,"15min":0.46400000000000002},"max":{"1min":1.02,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":0.53900000000000003}]},{"osd":3,"last update":"Wed Jun 29 17:08:25 2022","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.751,"15min":0.746},"min":{"1min":0.56499999999999995,"5min":0.52000000000000002,"15min":0.5},"max":{"1min":1.008,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.877,"15min":0.877},"min":{"1min":0.72799999999999998,"5min":0.66000000000000003,"15min":0.64300000000000002},"max":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.333},"last":0.74299999999999999}]},{"osd":4,"last update":"Wed Jun 29 17:08:25 2022","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.86699999999999999,"15min":0.86399999999999999},"min":{"1min":0.66300000000000003,"5min":0.628,"15min":0.628},"max":{"1min":1.0349999999999999,"5min":1.173,"15min":1.3560000000000001},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.94399999999999995,"15min":0.92300000000000004},"min":{"1min":0.751,"5min":0.70599999999999996,"15min":0.66000000000000003},"max":{"1min":1.1830000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":0.82999999999999996}]},{"osd":5,"last update":"Wed Jun 29 17:08:25 2022","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.94399999999999995,"15min":0.95299999999999996},"min":{"1min":0.68899999999999995,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.252,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.004,"15min":0.98799999999999999},"min":{"1min":0.83699999999999997,"5min":0.70699999999999996,"15min":0.68300000000000005},"max":{"1min":1.3340000000000001,"5min":1.3999999999999999,"15min":1.6140000000000001},"last":0.93899999999999995}]},{"osd":6,"last update":"Wed Jun 29 17:08:25 2022","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.0600000000000001,"15min":1.0329999999999999},"min":{"1min":0.872,"5min":0.70499999999999996,"15min":0.68100000000000005},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.554},"last":0.90200000000000002},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0069999999999999,"15min":1.0109999999999999},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.3180000000000001,"5min":1.361,"15min":1.48},"last":0.96199999999999997}]},{"osd":7,"last update":"Wed Jun 29 17:08:25 2022","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.0409999999999999,"15min":1.0229999999999999},"min":{"1min":0.84799999999999998,"5min":0.77200000000000002,"15min":0.71699999999999997},"max":{"1min":1.522,"5min":1.522,"15min":1.5429999999999999},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.083,"15min":1.075},"min":{"1min":0.88500000000000001,"5min":0.81000000000000005,"15min":0.67700000000000005},"max":{"1min":1.3140000000000001,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.91600000000000004}]}]}]}} 2022-06-29T17:09:28.851 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2022-06-29T17:09:28.955 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.947+0000 7fe3b9316700 1 Processor -- start 2022-06-29T17:09:28.956 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.947+0000 7fe3b9316700 1 -- start start 2022-06-29T17:09:28.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.951+0000 7fe3b9316700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe3b40aed60 0x7fe3b4147890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:28.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.951+0000 7fe3b9316700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe3b40afb10 0x7fe3b40b0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:28.959 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.951+0000 7fe3b9316700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3b40ac8f0 con 0x7fe3b41471a0 2022-06-29T17:09:28.960 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.951+0000 7fe3b9316700 1 -- --> v2:172.21.15.60:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3b40acb90 con 0x7fe3b40aed60 2022-06-29T17:09:28.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.951+0000 7fe3b9316700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3b40b02d0 con 0x7fe3b40afb10 2022-06-29T17:09:28.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.951+0000 7fe3b2ffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe3b40aed60 0x7fe3b4147890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:28.961 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.951+0000 7fe3b37fe700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe3b40afb10 0x7fe3b40b0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:28.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.951+0000 7fe3b2ffd700 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7fe3b40aed60 0x7fe3b4147890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55144/0 (socket says 172.21.15.60:55144) 2022-06-29T17:09:28.962 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.951+0000 7fe3b2ffd700 1 -- 172.21.15.60:0/2555303593 learned_addr learned my addr 172.21.15.60:0/2555303593 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T17:09:28.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.951+0000 7fe3b2ffd700 1 -- 172.21.15.60:0/2555303593 >> v2:172.21.15.92:3300/0 conn(0x7fe3b40afb10 msgr2=0x7fe3b40b0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-06-29T17:09:28.963 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.951+0000 7fe3b2ffd700 1 --2- 172.21.15.60:0/2555303593 >> v2:172.21.15.92:3300/0 conn(0x7fe3b40afb10 0x7fe3b40b0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:28.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.951+0000 7fe3b2ffd700 1 -- 172.21.15.60:0/2555303593 >> v1:172.21.15.92:6789/0 conn(0x7fe3b41471a0 legacy=0x7fe3b4147d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:28.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.951+0000 7fe3b2ffd700 1 -- 172.21.15.60:0/2555303593 --> v2:172.21.15.60:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b40ac5b0 con 0x7fe3b40aed60 2022-06-29T17:09:28.964 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.951+0000 7fe3b2ffd700 1 --2- 172.21.15.60:0/2555303593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3b40aed60 0x7fe3b4147890 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 rx=0x7fe3a8002f70 tx=0x7fe3a8006eb0).ready entity=mon.0 client_cookie=ae6dd3811efce666 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:28.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.951+0000 7fe3b1ffb700 1 -- 172.21.15.60:0/2555303593 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3a800d5c0 con 0x7fe3b40aed60 2022-06-29T17:09:28.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.951+0000 7fe3b1ffb700 1 -- 172.21.15.60:0/2555303593 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a8000e90 con 0x7fe3b40aed60 2022-06-29T17:09:28.965 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.951+0000 7fe3b1ffb700 1 -- 172.21.15.60:0/2555303593 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3a800d950 con 0x7fe3b40aed60 2022-06-29T17:09:28.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.951+0000 7fe3b9316700 1 -- 172.21.15.60:0/2555303593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3b40aed60 msgr2=0x7fe3b4147890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:28.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.951+0000 7fe3b9316700 1 --2- 172.21.15.60:0/2555303593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3b40aed60 0x7fe3b4147890 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 rx=0x7fe3a8002f70 tx=0x7fe3a8006eb0).stop 2022-06-29T17:09:28.966 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.951+0000 7fe3b9316700 1 -- 172.21.15.60:0/2555303593 shutdown_connections 2022-06-29T17:09:28.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.951+0000 7fe3b9316700 1 --2- 172.21.15.60:0/2555303593 >> v2:172.21.15.92:3300/0 conn(0x7fe3b40afb10 0x7fe3b40b0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:28.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.951+0000 7fe3b9316700 1 --2- 172.21.15.60:0/2555303593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3b40aed60 0x7fe3b4147890 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:28.967 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.951+0000 7fe3b9316700 1 -- 172.21.15.60:0/2555303593 >> 172.21.15.60:0/2555303593 conn(0x7fe3b40b4f60 msgr2=0x7fe3b40b5370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:28.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.951+0000 7fe3b9316700 1 -- 172.21.15.60:0/2555303593 shutdown_connections 2022-06-29T17:09:28.968 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.951+0000 7fe3b9316700 1 -- 172.21.15.60:0/2555303593 wait complete. 2022-06-29T17:09:28.972 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.955+0000 7fe3b9316700 1 Processor -- start 2022-06-29T17:09:28.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.955+0000 7fe3b9316700 1 -- start start 2022-06-29T17:09:28.973 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.955+0000 7fe3b9316700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3b40aed60 0x7fe3b414e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:28.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.955+0000 7fe3b9316700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3b40afb10 0x7fe3b415a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:28.974 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.955+0000 7fe3b9316700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3b415a560 0x7fe3b40a7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:28.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.955+0000 7fe3b9316700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b40a7100 con 0x7fe3b40afb10 2022-06-29T17:09:28.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.955+0000 7fe3b9316700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b4157790 con 0x7fe3b40aed60 2022-06-29T17:09:28.975 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.955+0000 7fe3b9316700 1 -- --> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b415a9f0 con 0x7fe3b415a560 2022-06-29T17:09:28.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.955+0000 7fe3b27fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3b40afb10 0x7fe3b415a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:28.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.955+0000 7fe3b2ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3b40aed60 0x7fe3b414e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:28.976 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.955+0000 7fe3b37fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3b415a560 0x7fe3b40a7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:28.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.955+0000 7fe3b27fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3b40afb10 0x7fe3b415a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55152/0 (socket says 172.21.15.60:55152) 2022-06-29T17:09:28.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.955+0000 7fe3b2ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3b40aed60 0x7fe3b414e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.60:32946/0 (socket says 172.21.15.60:32946) 2022-06-29T17:09:28.977 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.955+0000 7fe3b37fe700 1 --2- >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3b415a560 0x7fe3b40a7c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.92:3301/0 says I am v2:172.21.15.60:40012/0 (socket says 172.21.15.60:40012) 2022-06-29T17:09:28.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.955+0000 7fe3b2ffd700 1 -- 172.21.15.60:0/3018618548 learned_addr learned my addr 172.21.15.60:0/3018618548 (peer_addr_for_me v2:172.21.15.60:0/0) 2022-06-29T17:09:28.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.955+0000 7fe3b27fc700 1 -- 172.21.15.60:0/3018618548 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3b415a560 msgr2=0x7fe3b40a7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:28.978 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.955+0000 7fe3b27fc700 1 --2- 172.21.15.60:0/3018618548 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3b415a560 0x7fe3b40a7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:28.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.955+0000 7fe3b27fc700 1 -- 172.21.15.60:0/3018618548 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3b40aed60 msgr2=0x7fe3b414e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:28.979 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.955+0000 7fe3b27fc700 1 --2- 172.21.15.60:0/3018618548 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3b40aed60 0x7fe3b414e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:28.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.955+0000 7fe3b27fc700 1 -- 172.21.15.60:0/3018618548 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a800b1e0 con 0x7fe3b40afb10 2022-06-29T17:09:28.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.955+0000 7fe3b37fe700 1 --2- 172.21.15.60:0/3018618548 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3b415a560 0x7fe3b40a7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-06-29T17:09:28.980 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.955+0000 7fe3b27fc700 1 --2- 172.21.15.60:0/3018618548 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3b40afb10 0x7fe3b415a070 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 rx=0x7fe3b414ea10 tx=0x7fe3a4007ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:28.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.955+0000 7fe3a3fff700 1 -- 172.21.15.60:0/3018618548 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3a4006aa0 con 0x7fe3b40afb10 2022-06-29T17:09:28.981 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.955+0000 7fe3a3fff700 1 -- 172.21.15.60:0/3018618548 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a4006c00 con 0x7fe3b40afb10 2022-06-29T17:09:28.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.955+0000 7fe3a3fff700 1 -- 172.21.15.60:0/3018618548 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3a4007030 con 0x7fe3b40afb10 2022-06-29T17:09:28.982 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.955+0000 7fe3b9316700 1 -- 172.21.15.60:0/3018618548 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3b4159ae0 con 0x7fe3b40afb10 2022-06-29T17:09:28.983 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.955+0000 7fe3b9316700 1 -- 172.21.15.60:0/3018618548 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3b4065c60 con 0x7fe3b40afb10 2022-06-29T17:09:28.984 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.959+0000 7fe3b9316700 1 -- 172.21.15.60:0/3018618548 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3b4065e20 con 0x7fe3b40afb10 2022-06-29T17:09:28.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.959+0000 7fe3a3fff700 1 -- 172.21.15.60:0/3018618548 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 66004+0+0 (secure 0 0 0) 0x7fe3a4010030 con 0x7fe3b40afb10 2022-06-29T17:09:28.985 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.959+0000 7fe3a3fff700 1 --2- 172.21.15.60:0/3018618548 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe390053aa0 0x7fe390055f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-06-29T17:09:28.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.959+0000 7fe3a3fff700 1 -- 172.21.15.60:0/3018618548 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fe3a406b990 con 0x7fe3b40afb10 2022-06-29T17:09:28.986 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.959+0000 7fe3b2ffd700 1 --2- 172.21.15.60:0/3018618548 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe390053aa0 0x7fe390055f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-06-29T17:09:28.987 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.959+0000 7fe3b2ffd700 1 --2- 172.21.15.60:0/3018618548 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe390053aa0 0x7fe390055f60 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0x7fe3a801c000 tx=0x7fe3a8015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-06-29T17:09:28.988 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:28.967+0000 7fe3a3fff700 1 -- 172.21.15.60:0/3018618548 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe3a4042b90 con 0x7fe3b40afb10 2022-06-29T17:09:29.098 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:29.091+0000 7fe3b9316700 1 -- 172.21.15.60:0/3018618548 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fe3b40573d0 con 0x7fe3b40afb10 2022-06-29T17:09:29.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:29.099+0000 7fe3a3fff700 1 -- 172.21.15.60:0/3018618548 <== mon.0 v2:172.21.15.60:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fe3a4042b90 con 0x7fe3b40afb10 2022-06-29T17:09:29.105 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:29.099+0000 7fe3a3fff700 1 -- 172.21.15.60:0/3018618548 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v25)=0 v25) v1 ==== 123+0+0 (secure 0 0 0) 0x7fe3a40461b0 con 0x7fe3b40afb10 2022-06-29T17:09:29.108 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:29.103+0000 7fe3b9316700 1 -- 172.21.15.60:0/3018618548 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe390053aa0 msgr2=0x7fe390055f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:29.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:29.103+0000 7fe3b9316700 1 --2- 172.21.15.60:0/3018618548 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe390053aa0 0x7fe390055f60 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0x7fe3a801c000 tx=0x7fe3a8015000).stop 2022-06-29T17:09:29.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:29.103+0000 7fe3b9316700 1 -- 172.21.15.60:0/3018618548 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3b40afb10 msgr2=0x7fe3b415a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-06-29T17:09:29.109 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:29.103+0000 7fe3b9316700 1 --2- 172.21.15.60:0/3018618548 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3b40afb10 0x7fe3b415a070 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 rx=0x7fe3b414ea10 tx=0x7fe3a4007ef0).stop 2022-06-29T17:09:29.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:29.103+0000 7fe3b9316700 1 -- 172.21.15.60:0/3018618548 shutdown_connections 2022-06-29T17:09:29.110 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:29.103+0000 7fe3b9316700 1 --2- 172.21.15.60:0/3018618548 >> [v2:172.21.15.92:6832/12872,v1:172.21.15.92:6833/12872] conn(0x7fe390053aa0 0x7fe390055f60 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:29.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:29.103+0000 7fe3b9316700 1 --2- 172.21.15.60:0/3018618548 >> [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] conn(0x7fe3b415a560 0x7fe3b40a7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:29.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:29.103+0000 7fe3b9316700 1 --2- 172.21.15.60:0/3018618548 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe3b40afb10 0x7fe3b415a070 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:29.111 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:29.103+0000 7fe3b9316700 1 --2- 172.21.15.60:0/3018618548 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe3b40aed60 0x7fe3b414e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-06-29T17:09:29.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:29.103+0000 7fe3b9316700 1 -- 172.21.15.60:0/3018618548 >> 172.21.15.60:0/3018618548 conn(0x7fe3b40b4f60 msgr2=0x7fe3b405c420 unknown :-1 s=STATE_NONE l=0).mark_down 2022-06-29T17:09:29.112 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:29.103+0000 7fe3b9316700 1 -- 172.21.15.60:0/3018618548 shutdown_connections 2022-06-29T17:09:29.113 INFO:teuthology.orchestra.run.smithi060.stderr:2022-06-29T17:09:29.103+0000 7fe3b9316700 1 -- 172.21.15.60:0/3018618548 wait complete. 2022-06-29T17:09:29.123 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_f25ffac72e5a9e63860ff7deb6767a11f0559a16/qa/tasks/ceph.py", line 1839, in task yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/run_tasks.py", line 188, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_f25ffac72e5a9e63860ff7deb6767a11f0559a16/qa/tasks/ceph_fuse.py", line 160, in task mount.umount_wait() File "/home/teuthworker/src/github.com_ceph_ceph-c_f25ffac72e5a9e63860ff7deb6767a11f0559a16/qa/tasks/cephfs/fuse_mount.py", line 326, in umount_wait if not (self.is_mounted() and self.fuse_daemon): File "/home/teuthworker/src/github.com_ceph_ceph-c_f25ffac72e5a9e63860ff7deb6767a11f0559a16/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-06-29T17:09:29.129 INFO:teuthology.misc:Shutting down mds daemons... 2022-06-29T17:09:29.131 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-06-29T17:09:29.132 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T17:09:29.143 INFO:tasks.ceph.mds.a:Stopped 2022-06-29T17:09:29.144 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-06-29T17:09:29.145 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T17:09:29.161 INFO:tasks.ceph.mds.b:Stopped 2022-06-29T17:09:29.163 INFO:teuthology.misc:Shutting down osd daemons... 2022-06-29T17:09:29.165 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2022-06-29T17:09:29.166 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T17:09:29.210 INFO:tasks.ceph.osd.0:Stopped 2022-06-29T17:09:29.212 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2022-06-29T17:09:29.213 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T17:09:29.238 INFO:tasks.ceph.osd.1:Stopped 2022-06-29T17:09:29.240 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2022-06-29T17:09:29.242 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T17:09:29.255 INFO:tasks.ceph.osd.2:Stopped 2022-06-29T17:09:29.257 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2022-06-29T17:09:29.258 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T17:09:29.319 INFO:tasks.ceph.osd.3:Stopped 2022-06-29T17:09:29.320 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2022-06-29T17:09:29.321 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T17:09:29.342 INFO:tasks.ceph.osd.4:Stopped 2022-06-29T17:09:29.343 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2022-06-29T17:09:29.344 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T17:09:29.363 INFO:tasks.ceph.osd.5:Stopped 2022-06-29T17:09:29.364 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2022-06-29T17:09:29.364 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T17:09:29.424 INFO:tasks.ceph.osd.6:Stopped 2022-06-29T17:09:29.427 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2022-06-29T17:09:29.427 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T17:09:29.443 INFO:tasks.ceph.osd.7:Stopped 2022-06-29T17:09:29.444 INFO:teuthology.misc:Shutting down mgr daemons... 2022-06-29T17:09:29.445 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2022-06-29T17:09:29.446 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T17:09:29.474 INFO:tasks.ceph.mgr.y:Stopped 2022-06-29T17:09:29.475 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2022-06-29T17:09:29.476 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T17:09:29.502 INFO:tasks.ceph.mgr.x:Stopped 2022-06-29T17:09:29.505 INFO:teuthology.misc:Shutting down mon daemons... 2022-06-29T17:09:29.506 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2022-06-29T17:09:29.508 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T17:09:29.534 INFO:tasks.ceph.mon.a:Stopped 2022-06-29T17:09:29.535 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2022-06-29T17:09:29.536 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T17:09:29.565 INFO:tasks.ceph.mon.b:Stopped 2022-06-29T17:09:29.566 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2022-06-29T17:09:29.567 INFO:teuthology.orchestra.run:waiting for 300 2022-06-29T17:09:29.586 INFO:tasks.ceph.mon.c:Stopped 2022-06-29T17:09:29.587 INFO:tasks.ceph:Checking cluster log for badness... 2022-06-29T17:09:29.588 DEBUG:teuthology.orchestra.run.smithi060:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'evicting unresponsive client' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'has not responded to cap revoke by MDS for over' | egrep -v MDS_CLIENT_LATE_RELEASE | egrep -v 'responding to mclientcaps' | head -n 1 2022-06-29T17:09:29.609 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi060.front.sepia.ceph.com 2022-06-29T17:09:29.610 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2022-06-29T17:09:30.125 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi060.front.sepia.ceph.com 2022-06-29T17:09:30.127 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2022-06-29T17:09:30.439 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi060.front.sepia.ceph.com 2022-06-29T17:09:30.440 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2022-06-29T17:09:30.678 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi060.front.sepia.ceph.com 2022-06-29T17:09:30.679 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2022-06-29T17:09:30.906 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi092.front.sepia.ceph.com 2022-06-29T17:09:30.907 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2022-06-29T17:09:31.141 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi092.front.sepia.ceph.com 2022-06-29T17:09:31.142 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2022-06-29T17:09:31.471 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi092.front.sepia.ceph.com 2022-06-29T17:09:31.472 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2022-06-29T17:09:31.669 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi092.front.sepia.ceph.com 2022-06-29T17:09:31.671 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2022-06-29T17:09:31.852 INFO:tasks.ceph:Archiving mon data... 2022-06-29T17:09:31.854 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2022-06-29_13:31:31-fs-wip-yuri5-testing-2022-06-27-0923-octopus-distro-default-smithi/6905536/data/mon.a.tgz 2022-06-29T17:09:31.855 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2022-06-29T17:09:31.864 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.0NanDl11zZ 2022-06-29T17:09:31.865 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.0NanDl11zZ 2022-06-29T17:09:32.120 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.0NanDl11zZ 2022-06-29T17:09:32.138 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.0NanDl11zZ is 503KB 2022-06-29T17:09:32.206 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.0NanDl11zZ 2022-06-29T17:09:32.214 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2022-06-29_13:31:31-fs-wip-yuri5-testing-2022-06-27-0923-octopus-distro-default-smithi/6905536/data/mon.b.tgz 2022-06-29T17:09:32.215 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2022-06-29T17:09:32.224 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.SepH2hGLa1 2022-06-29T17:09:32.225 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.SepH2hGLa1 2022-06-29T17:09:32.487 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.SepH2hGLa1 2022-06-29T17:09:32.509 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.SepH2hGLa1 is 619KB 2022-06-29T17:09:32.574 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.SepH2hGLa1 2022-06-29T17:09:32.581 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2022-06-29_13:31:31-fs-wip-yuri5-testing-2022-06-27-0923-octopus-distro-default-smithi/6905536/data/mon.c.tgz 2022-06-29T17:09:32.582 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2022-06-29T17:09:32.589 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.8DiV3lU7qN 2022-06-29T17:09:32.590 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.8DiV3lU7qN 2022-06-29T17:09:32.852 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.8DiV3lU7qN 2022-06-29T17:09:32.922 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.8DiV3lU7qN is 619KB 2022-06-29T17:09:32.991 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.8DiV3lU7qN 2022-06-29T17:09:33.000 INFO:tasks.ceph:Cleaning ceph cluster... 2022-06-29T17:09:33.002 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2022-06-29T17:09:33.006 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2022-06-29T17:09:33.046 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2022-06-29T17:09:33.049 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2022-06-29T17:09:33.052 DEBUG:teuthology.orchestra.run.smithi197:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2022-06-29T17:09:33.069 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2022-06-29T17:09:33.070 DEBUG:teuthology.orchestra.run.smithi060:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2022-06-29T17:09:33.073 DEBUG:teuthology.orchestra.run.smithi092:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2022-06-29T17:09:33.076 DEBUG:teuthology.orchestra.run.smithi121:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2022-06-29T17:09:33.099 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2022-06-29T17:09:33.100 INFO:teuthology.orchestra.run.smithi092.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2022-06-29T17:09:33.106 DEBUG:teuthology.orchestra.run.smithi156:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2022-06-29T17:09:33.110 DEBUG:teuthology.orchestra.run.smithi197:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2022-06-29T17:09:33.126 INFO:teuthology.orchestra.run.smithi121.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2022-06-29T17:09:33.204 INFO:teuthology.orchestra.run.smithi156.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2022-06-29T17:09:33.210 INFO:teuthology.orchestra.run.smithi197.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2022-06-29T17:09:33.217 INFO:tasks.ceph:Archiving crash dumps... 2022-06-29T17:09:33.219 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2022-06-29_13:31:31-fs-wip-yuri5-testing-2022-06-27-0923-octopus-distro-default-smithi/6905536/remote/smithi060/crash 2022-06-29T17:09:33.221 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2022-06-29T17:09:33.243 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2022-06-29_13:31:31-fs-wip-yuri5-testing-2022-06-27-0923-octopus-distro-default-smithi/6905536/remote/smithi092/crash 2022-06-29T17:09:33.245 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2022-06-29T17:09:33.268 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2022-06-29_13:31:31-fs-wip-yuri5-testing-2022-06-27-0923-octopus-distro-default-smithi/6905536/remote/smithi121/crash 2022-06-29T17:09:33.270 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2022-06-29T17:09:33.291 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2022-06-29_13:31:31-fs-wip-yuri5-testing-2022-06-27-0923-octopus-distro-default-smithi/6905536/remote/smithi156/crash 2022-06-29T17:09:33.293 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2022-06-29T17:09:33.315 DEBUG:teuthology.misc:Transferring archived files from smithi197:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2022-06-29_13:31:31-fs-wip-yuri5-testing-2022-06-27-0923-octopus-distro-default-smithi/6905536/remote/smithi197/crash 2022-06-29T17:09:33.317 DEBUG:teuthology.orchestra.run.smithi197:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2022-06-29T17:09:33.339 INFO:tasks.ceph:Shutting down logrotate 2022-06-29T17:09:33.340 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:09:33.344 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:09:33.347 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:09:33.350 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:09:33.353 DEBUG:teuthology.orchestra.run.smithi197:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-06-29T17:09:33.402 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm /etc/logrotate.d/ceph-test.conf 2022-06-29T17:09:33.419 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm /etc/logrotate.d/ceph-test.conf 2022-06-29T17:09:33.435 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm /etc/logrotate.d/ceph-test.conf 2022-06-29T17:09:33.451 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm /etc/logrotate.d/ceph-test.conf 2022-06-29T17:09:33.469 DEBUG:teuthology.orchestra.run.smithi197:> sudo rm /etc/logrotate.d/ceph-test.conf 2022-06-29T17:09:33.486 INFO:tasks.ceph:Compressing logs... 2022-06-29T17:09:33.487 DEBUG:teuthology.orchestra.run.smithi060:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-06-29T17:09:33.490 DEBUG:teuthology.orchestra.run.smithi092:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-06-29T17:09:33.493 DEBUG:teuthology.orchestra.run.smithi121:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-06-29T17:09:33.498 DEBUG:teuthology.orchestra.run.smithi156:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-06-29T17:09:33.502 DEBUG:teuthology.orchestra.run.smithi197:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-06-29T17:09:43.613 INFO:tasks.ceph:Archiving logs... 2022-06-29T17:09:43.614 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/log/ceph to /home/teuthworker/archive/yuriw-2022-06-29_13:31:31-fs-wip-yuri5-testing-2022-06-27-0923-octopus-distro-default-smithi/6905536/remote/smithi060/log 2022-06-29T17:09:43.616 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar cz -f - -C /var/log/ceph -- . 2022-06-29T17:09:45.713 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/log/ceph to /home/teuthworker/archive/yuriw-2022-06-29_13:31:31-fs-wip-yuri5-testing-2022-06-27-0923-octopus-distro-default-smithi/6905536/remote/smithi092/log 2022-06-29T17:09:45.715 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar cz -f - -C /var/log/ceph -- . 2022-06-29T17:09:47.825 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/log/ceph to /home/teuthworker/archive/yuriw-2022-06-29_13:31:31-fs-wip-yuri5-testing-2022-06-27-0923-octopus-distro-default-smithi/6905536/remote/smithi121/log 2022-06-29T17:09:47.826 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar cz -f - -C /var/log/ceph -- . 2022-06-29T17:09:48.243 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/log/ceph to /home/teuthworker/archive/yuriw-2022-06-29_13:31:31-fs-wip-yuri5-testing-2022-06-27-0923-octopus-distro-default-smithi/6905536/remote/smithi156/log 2022-06-29T17:09:48.245 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar cz -f - -C /var/log/ceph -- . 2022-06-29T17:09:48.281 DEBUG:teuthology.misc:Transferring archived files from smithi197:/var/log/ceph to /home/teuthworker/archive/yuriw-2022-06-29_13:31:31-fs-wip-yuri5-testing-2022-06-27-0923-octopus-distro-default-smithi/6905536/remote/smithi197/log 2022-06-29T17:09:48.282 DEBUG:teuthology.orchestra.run.smithi197:> sudo tar cz -f - -C /var/log/ceph -- . 2022-06-29T17:09:48.305 DEBUG:teuthology.run_tasks:Unwinding manager install 2022-06-29T17:09:48.316 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/task/install/__init__.py", line 619, in task yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/run_tasks.py", line 188, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_f25ffac72e5a9e63860ff7deb6767a11f0559a16/qa/tasks/ceph_fuse.py", line 160, in task mount.umount_wait() File "/home/teuthworker/src/github.com_ceph_ceph-c_f25ffac72e5a9e63860ff7deb6767a11f0559a16/qa/tasks/cephfs/fuse_mount.py", line 326, in umount_wait if not (self.is_mounted() and self.fuse_daemon): File "/home/teuthworker/src/github.com_ceph_ceph-c_f25ffac72e5a9e63860ff7deb6767a11f0559a16/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-06-29T17:09:48.320 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2022-06-29T17:09:48.320 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2022-06-29T17:09:48.324 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2022-06-29T17:09:48.328 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2022-06-29T17:09:48.332 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2022-06-29T17:09:48.335 DEBUG:teuthology.orchestra.run.smithi197:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2022-06-29T17:09:48.503 DEBUG:teuthology.run_tasks:Unwinding manager clock 2022-06-29T17:09:48.520 INFO:teuthology.task.clock:Checking final clock skew... 2022-06-29T17:09:48.521 DEBUG:teuthology.orchestra.run.smithi060:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-06-29T17:09:48.524 DEBUG:teuthology.orchestra.run.smithi092:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-06-29T17:09:48.527 DEBUG:teuthology.orchestra.run.smithi121:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-06-29T17:09:48.530 DEBUG:teuthology.orchestra.run.smithi156:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-06-29T17:09:48.533 DEBUG:teuthology.orchestra.run.smithi197:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-06-29T17:09:48.546 INFO:teuthology.orchestra.run.smithi060.stdout: remote refid st t when poll reach delay offset jitter 2022-06-29T17:09:48.547 INFO:teuthology.orchestra.run.smithi060.stdout:============================================================================== 2022-06-29T17:09:48.547 INFO:teuthology.orchestra.run.smithi060.stdout:+hv01.front.sepi 162.248.241.94 3 u 184 256 377 0.115 3.117 0.200 2022-06-29T17:09:48.548 INFO:teuthology.orchestra.run.smithi060.stdout:*hv02.front.sepi 163.237.218.18 2 u 52 256 377 0.176 -0.640 0.233 2022-06-29T17:09:48.548 INFO:teuthology.orchestra.run.smithi060.stdout:+hv03.front.sepi 216.229.4.69 3 u 200 256 377 0.207 -1.392 0.107 2022-06-29T17:09:48.548 INFO:teuthology.orchestra.run.smithi060.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2022-06-29T17:09:48.551 INFO:teuthology.orchestra.run.smithi092.stdout: remote refid st t when poll reach delay offset jitter 2022-06-29T17:09:48.551 INFO:teuthology.orchestra.run.smithi092.stdout:============================================================================== 2022-06-29T17:09:48.551 INFO:teuthology.orchestra.run.smithi092.stdout:+hv01.front.sepi 162.248.241.94 3 u 54 256 377 0.173 0.730 0.387 2022-06-29T17:09:48.552 INFO:teuthology.orchestra.run.smithi092.stdout:*hv02.front.sepi 163.237.218.18 2 u 57 256 377 0.209 -3.337 0.497 2022-06-29T17:09:48.552 INFO:teuthology.orchestra.run.smithi092.stdout:+hv03.front.sepi 216.229.4.69 3 u 120 128 377 0.218 -3.408 0.609 2022-06-29T17:09:48.552 INFO:teuthology.orchestra.run.smithi092.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2022-06-29T17:09:48.554 INFO:teuthology.orchestra.run.smithi121.stdout: remote refid st t when poll reach delay offset jitter 2022-06-29T17:09:48.554 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================== 2022-06-29T17:09:48.555 INFO:teuthology.orchestra.run.smithi121.stdout:+hv01.front.sepi 162.248.241.94 3 u 126 256 377 0.164 3.653 0.868 2022-06-29T17:09:48.555 INFO:teuthology.orchestra.run.smithi121.stdout:*hv02.front.sepi 163.237.218.18 2 u 149 256 377 0.181 0.341 0.558 2022-06-29T17:09:48.555 INFO:teuthology.orchestra.run.smithi121.stdout:+hv03.front.sepi 216.229.4.69 3 u 115 256 377 0.194 -0.496 0.683 2022-06-29T17:09:48.556 INFO:teuthology.orchestra.run.smithi121.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2022-06-29T17:09:48.557 INFO:teuthology.orchestra.run.smithi156.stdout: remote refid st t when poll reach delay offset jitter 2022-06-29T17:09:48.557 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================== 2022-06-29T17:09:48.558 INFO:teuthology.orchestra.run.smithi156.stdout:+hv01.front.sepi 162.248.241.94 3 u 71 128 377 0.173 2.044 0.498 2022-06-29T17:09:48.558 INFO:teuthology.orchestra.run.smithi156.stdout:*hv02.front.sepi 163.237.218.18 2 u 112 128 377 0.181 -2.023 0.374 2022-06-29T17:09:48.558 INFO:teuthology.orchestra.run.smithi156.stdout:+hv03.front.sepi 216.229.4.69 3 u 59 128 377 0.182 -2.680 0.432 2022-06-29T17:09:48.559 INFO:teuthology.orchestra.run.smithi156.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2022-06-29T17:09:48.560 INFO:teuthology.orchestra.run.smithi197.stdout: remote refid st t when poll reach delay offset jitter 2022-06-29T17:09:48.561 INFO:teuthology.orchestra.run.smithi197.stdout:============================================================================== 2022-06-29T17:09:48.561 INFO:teuthology.orchestra.run.smithi197.stdout:+hv01.front.sepi 162.248.241.94 3 u 53 128 377 0.157 2.207 0.139 2022-06-29T17:09:48.561 INFO:teuthology.orchestra.run.smithi197.stdout:*hv02.front.sepi 163.237.218.18 2 u 122 128 377 0.202 -1.632 0.057 2022-06-29T17:09:48.562 INFO:teuthology.orchestra.run.smithi197.stdout:+hv03.front.sepi 216.229.4.69 3 u 59 128 377 0.195 -2.348 0.136 2022-06-29T17:09:48.562 INFO:teuthology.orchestra.run.smithi197.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2022-06-29T17:09:48.562 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2022-06-29T17:09:48.573 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2022-06-29T17:09:48.574 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2022-06-29T17:09:48.602 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2022-06-29T17:09:48.644 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2022-06-29T17:09:48.686 INFO:teuthology.task.internal:Duration was 4637.623535 seconds 2022-06-29T17:09:48.688 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2022-06-29T17:09:48.728 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2022-06-29T17:09:48.729 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-06-29T17:09:48.732 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-06-29T17:09:48.734 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-06-29T17:09:48.738 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-06-29T17:09:48.742 DEBUG:teuthology.orchestra.run.smithi197:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-06-29T17:09:48.806 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2022-06-29T17:09:48.807 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi060.front.sepia.ceph.com 2022-06-29T17:09:48.809 DEBUG:teuthology.orchestra.run.smithi060:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-06-29T17:09:48.827 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi092.front.sepia.ceph.com 2022-06-29T17:09:48.828 DEBUG:teuthology.orchestra.run.smithi092:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-06-29T17:09:48.844 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi121.front.sepia.ceph.com 2022-06-29T17:09:48.845 DEBUG:teuthology.orchestra.run.smithi121:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-06-29T17:09:48.863 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi156.front.sepia.ceph.com 2022-06-29T17:09:48.865 DEBUG:teuthology.orchestra.run.smithi156:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-06-29T17:09:48.882 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi197.front.sepia.ceph.com 2022-06-29T17:09:48.883 DEBUG:teuthology.orchestra.run.smithi197:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-06-29T17:09:48.900 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2022-06-29T17:09:48.901 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-06-29T17:09:48.904 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-06-29T17:09:48.906 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-06-29T17:09:48.909 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-06-29T17:09:48.927 DEBUG:teuthology.orchestra.run.smithi197:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-06-29T17:09:48.954 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2022-06-29T17:09:48.955 DEBUG:teuthology.orchestra.run.smithi060:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-06-29T17:09:48.981 DEBUG:teuthology.orchestra.run.smithi092:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-06-29T17:09:48.984 DEBUG:teuthology.orchestra.run.smithi121:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-06-29T17:09:48.986 DEBUG:teuthology.orchestra.run.smithi156:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-06-29T17:09:48.989 DEBUG:teuthology.orchestra.run.smithi197:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-06-29T17:09:49.344 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2022-06-29T17:09:49.355 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2022-06-29T17:09:49.356 DEBUG:teuthology.orchestra.run.smithi060:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-06-29T17:09:49.374 DEBUG:teuthology.orchestra.run.smithi092:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-06-29T17:09:49.391 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-06-29T17:09:49.408 DEBUG:teuthology.orchestra.run.smithi156:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-06-29T17:09:49.427 DEBUG:teuthology.orchestra.run.smithi197:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-06-29T17:09:49.445 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2022-06-29T17:09:49.458 DEBUG:teuthology.orchestra.run.smithi060:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-06-29T17:09:49.461 DEBUG:teuthology.orchestra.run.smithi092:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-06-29T17:09:49.465 DEBUG:teuthology.orchestra.run.smithi121:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-06-29T17:09:49.469 DEBUG:teuthology.orchestra.run.smithi156:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-06-29T17:09:49.472 DEBUG:teuthology.orchestra.run.smithi197:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-06-29T17:09:49.474 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = core 2022-06-29T17:09:49.476 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = core 2022-06-29T17:09:49.480 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = core 2022-06-29T17:09:49.484 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = core 2022-06-29T17:09:49.486 INFO:teuthology.orchestra.run.smithi197.stdout:kernel.core_pattern = core 2022-06-29T17:09:49.506 DEBUG:teuthology.orchestra.run.smithi060:> test -e /home/ubuntu/cephtest/archive/coredump 2022-06-29T17:09:49.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T17:09:49.541 DEBUG:teuthology.orchestra.run.smithi092:> test -e /home/ubuntu/cephtest/archive/coredump 2022-06-29T17:09:49.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T17:09:49.553 DEBUG:teuthology.orchestra.run.smithi121:> test -e /home/ubuntu/cephtest/archive/coredump 2022-06-29T17:09:49.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T17:09:49.561 DEBUG:teuthology.orchestra.run.smithi156:> test -e /home/ubuntu/cephtest/archive/coredump 2022-06-29T17:09:49.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T17:09:49.567 DEBUG:teuthology.orchestra.run.smithi197:> test -e /home/ubuntu/cephtest/archive/coredump 2022-06-29T17:09:49.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-06-29T17:09:49.575 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2022-06-29T17:09:49.587 INFO:teuthology.task.internal:Transferring archived files... 2022-06-29T17:09:49.589 DEBUG:teuthology.misc:Transferring archived files from smithi060:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2022-06-29_13:31:31-fs-wip-yuri5-testing-2022-06-27-0923-octopus-distro-default-smithi/6905536/remote/smithi060 2022-06-29T17:09:49.590 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-06-29T17:09:49.626 DEBUG:teuthology.misc:Transferring archived files from smithi092:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2022-06-29_13:31:31-fs-wip-yuri5-testing-2022-06-27-0923-octopus-distro-default-smithi/6905536/remote/smithi092 2022-06-29T17:09:49.627 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-06-29T17:09:49.659 DEBUG:teuthology.misc:Transferring archived files from smithi121:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2022-06-29_13:31:31-fs-wip-yuri5-testing-2022-06-27-0923-octopus-distro-default-smithi/6905536/remote/smithi121 2022-06-29T17:09:49.660 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-06-29T17:09:49.693 DEBUG:teuthology.misc:Transferring archived files from smithi156:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2022-06-29_13:31:31-fs-wip-yuri5-testing-2022-06-27-0923-octopus-distro-default-smithi/6905536/remote/smithi156 2022-06-29T17:09:49.694 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-06-29T17:09:49.728 DEBUG:teuthology.misc:Transferring archived files from smithi197:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2022-06-29_13:31:31-fs-wip-yuri5-testing-2022-06-27-0923-octopus-distro-default-smithi/6905536/remote/smithi197 2022-06-29T17:09:49.729 DEBUG:teuthology.orchestra.run.smithi197:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-06-29T17:09:49.762 INFO:teuthology.task.internal:Removing archive directory... 2022-06-29T17:09:49.763 DEBUG:teuthology.orchestra.run.smithi060:> rm -rf -- /home/ubuntu/cephtest/archive 2022-06-29T17:09:49.768 DEBUG:teuthology.orchestra.run.smithi092:> rm -rf -- /home/ubuntu/cephtest/archive 2022-06-29T17:09:49.771 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf -- /home/ubuntu/cephtest/archive 2022-06-29T17:09:49.774 DEBUG:teuthology.orchestra.run.smithi156:> rm -rf -- /home/ubuntu/cephtest/archive 2022-06-29T17:09:49.778 DEBUG:teuthology.orchestra.run.smithi197:> rm -rf -- /home/ubuntu/cephtest/archive 2022-06-29T17:09:49.808 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2022-06-29T17:09:49.821 INFO:teuthology.task.internal:Not uploading archives. 2022-06-29T17:09:49.822 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2022-06-29T17:09:49.864 INFO:teuthology.task.internal:Tidying up after the test... 2022-06-29T17:09:49.865 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-06-29T17:09:49.868 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-06-29T17:09:49.871 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-06-29T17:09:49.873 INFO:teuthology.orchestra.run.smithi060.stdout: 524315 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 29 17:09 /home/ubuntu/cephtest 2022-06-29T17:09:49.874 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-06-29T17:09:49.877 INFO:teuthology.orchestra.run.smithi092.stdout: 524315 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 29 17:09 /home/ubuntu/cephtest 2022-06-29T17:09:49.878 DEBUG:teuthology.orchestra.run.smithi197:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-06-29T17:09:49.882 INFO:teuthology.orchestra.run.smithi156.stdout: 524315 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 29 17:09 /home/ubuntu/cephtest 2022-06-29T17:09:49.886 INFO:teuthology.orchestra.run.smithi197.stdout: 524315 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 29 17:09 /home/ubuntu/cephtest 2022-06-30T03:50:17.213 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-06-30T03:50:17.243 DEBUG:teuthology.task.console_log:Killing console logger for smithi060 2022-06-30T03:50:17.244 DEBUG:teuthology.task.console_log:Killing console logger for smithi092 2022-06-30T03:50:17.245 DEBUG:teuthology.task.console_log:Killing console logger for smithi121 2022-06-30T03:50:17.246 DEBUG:teuthology.task.console_log:Killing console logger for smithi156 2022-06-30T03:50:17.247 DEBUG:teuthology.task.console_log:Killing console logger for smithi197 2022-06-30T03:50:17.250 DEBUG:teuthology.exit:Finished running handlers